From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EFF8C761A6 for ; Mon, 3 Apr 2023 05:23:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231560AbjDCFXq (ORCPT ); Mon, 3 Apr 2023 01:23:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231509AbjDCFXd (ORCPT ); Mon, 3 Apr 2023 01:23:33 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F14EB45E for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332Msls0031744; Mon, 3 Apr 2023 05:22:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=1N6qbt+qhUVgQy2JQLXyJCdS15y0cyZtppYubbAlJkY=; b=hOua9Es/E4R9NqhhOatqoqZs0Rk2TIVxG8ieeM+AIaeT8PTbTG4f6aG5P3XYQebQckMh Q7tqwvhoRemGGUJtPBiaXpXX4V/HUdTJ+9RvIUoIYlgUMuFpztsuaRrXhrTL6sm6eK3m j4qgPoHynHSCmDO0iGTwMEh+j7jopfyWrGGuErQ9vmx7vVm1JGrQvuvLcc7LRxohv3m/ 65SDlRj3p7Xl6yO04aSSGdpOBlVU11Dchg7rBGytx0iCmLZYnsHog+6cgqKr99MpN9P8 8idfndT1TjaLUodlXT8BYpXl1BwWbJWtWqL5Z7ArLAzN2eWJz2a8/TsglHor0XvlvMkS cQ== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppb71j7h3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:30 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332Pl6K014039; Mon, 3 Apr 2023 05:22:29 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt3djufx-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cgT8+ZSkRpktst4jWPVexS1fiEbuJ1OI+a6c/rkVxC6U0BGuhmHoLA7Fy5+gxiZX1o7ove0JeJoNkyP8OSIJFmLdevl4jCBVtNUfNdDKuoj2ECLJ+6FCF3bKZ3+Kkjn1cA6ktMrwiPgCIuHj6c5y9GrpvbJ0lOouJatJ9oumUVvRQ2Cl2d2fYHH1GkWY1EF11S9apULhbUB5L9j3LKScKDPRfMg7GZhTgq4n2Gi5w/0yo1CAXMAQv/dN2peOduXGuyTxr9nvaVA8sJWFtnDAaDKnvcivTsdr8VYMDJX4VNrvIgCm6i1vQh3BDj6wC8fTHg7AwIvJcYgwEF2btEa/Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1N6qbt+qhUVgQy2JQLXyJCdS15y0cyZtppYubbAlJkY=; b=DPwVOGIGxn22GTFqi7mQAzNX0gJiWj1Ft/Ehqp5+9ArhjiEvq+iDd0yOR3XRsH+3mOZUndGU3hD9TZ6DeVst7kYoq1jIPSCyjp5YfdXxQwIqeod4lmKlXWuJrmHwFQ7O7jYdiO61UfwU4dWj5iSQYrp0YsVnl8LIErgGA7d4yHTtmxvgsPNHzMY7ruxi1HVuavqH3hkp1ESnZguUwUNaUdgIhO4VmdPwO4Kb0Ufk+7ShnYWZMB2doKdipX4gv/Hu0TRgMWXd6qYShY33iYmJsj9O7GHvTpXJxgn5WEwXDEAUdwwNLoXqPJUUDFo78opItjHn47P0p0mJW5Tyg/9z9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1N6qbt+qhUVgQy2JQLXyJCdS15y0cyZtppYubbAlJkY=; b=Hy2ugkgh/AFwOS0PC/TBdVB5Hmafo3mRQA0a9UhGsOSgIVsD5vG3FifpRaW8hATHCe1RSAJym1FN+01vEBCVZis+ayoopfIy46YB66jSKtkNYeBirdwGDgrXy5cFlkcG9cy2UMkuAD2ibjM3YYTGa4FnwNPSXPjj5oiHDmwUK3E= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SJ0PR10MB6398.namprd10.prod.outlook.com (2603:10b6:a03:44a::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:27 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:27 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 1/9] huge_pages: get rid of process_huge_page() Date: Sun, 2 Apr 2023 22:22:25 -0700 Message-Id: <20230403052233.1880567-2-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0267.namprd04.prod.outlook.com (2603:10b6:303:88::32) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SJ0PR10MB6398:EE_ X-MS-Office365-Filtering-Correlation-Id: 45791c6b-8e75-4f74-8949-08db34036a2e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YD1Wo4q6FHinxM29vI1SsUONfNcIqT9VEQ0j15jqAUFM/+cbXdp4DWdXlZHyKcLQsbrFBn5s0nqALWiE6HEEvJccQu4cfVJ5l3y5ukUiX6qYZX2rtXREyE0L9R7bNEFOAZgnEO1Py+HdkN9qTQNxxRVKGfnQJ+Eeayz+XBlv/9G2GVVdtCGeXj8TWS83KxHhuecD1l14vYQJ6hSh7+/+79sMvvdlWO+x2XqIA/yZgo7SZQva0r8lVNRbb5WDvqHhtCQUmBk1PMtTAzu5U97KHYIgTbjVfSYxOJNuPVvfGCQBji5S2+Sg7hz3g3Hgw32e58YVw8oxAAI2q/JJV8qzwJ93gb8eOpKE/2g7WGkuZFT5VrEgUffc673O3l2tOjsOrDWPGeotzZLNOynELxAk4iYb2Zlp8fCwx42Wr+gzSmIKxRQFEbJaR27Ns51bn2IXToSs/4Xruj+Jf8zqFZnTZ/1VVhPNWKUA4eDpn7xWWsJcbi6Tpu9KWgtwe5JzeWgfJZorDWt8AHOg8BBX56IyM8SGGvHqy80Bv9dXP+9lPBCBGejQeml+FfAQEHsJEOihXo06jN0B+3lkDmvlxzH+I0xhu3fZclILX9tiszBa8DFXC30ePmmkoXDpdgUUfPMH X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(136003)(376002)(346002)(396003)(39860400002)(451199021)(2906002)(8936002)(38100700002)(41300700001)(7416002)(5660300002)(103116003)(36756003)(86362001)(478600001)(316002)(6486002)(966005)(107886003)(2616005)(1076003)(6512007)(6506007)(26005)(8676002)(4326008)(83380400001)(66476007)(66556008)(66946007)(186003)(26583001)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?xLIrrbNyKGOhvDeQ72mC7nMogOYXdEbwUTXgTaGJnbwqAAdk5HV8hqRh1w1P?= =?us-ascii?Q?Yb0nFE3Z+T7Ujw2t2NcGqeW0+FTrzKQz6iv0QyDmy9MGkdmAuEbptn6Zm1zX?= =?us-ascii?Q?RXGUogXxunRpf9R6bIQB+X1jRInCrpiX7SqnozmzKL4jPFEMwl5Zd2ssoOHp?= =?us-ascii?Q?Khw82NI19yKK4irvB3oxUuPqy+3emd6igEiBGuYIG+tC/8GqHPFNLYFKSD+w?= =?us-ascii?Q?kU62LC+cNpTCXABdhMMu1jvg0o9FKaveaFFDOHpfWuFBDAwyisYaMd8I6qZ7?= =?us-ascii?Q?ZIRNS8vkHnBDmcF9aVNFcWxqYPuA5LQHb3T42kQrva+Pqy1RkIrk33VdMpcj?= =?us-ascii?Q?FADhv7IoH9Qms/iM3phL43nPvZ/kpS0H4TBLg1odXUoOqEiIpbZxcIcJcpVT?= =?us-ascii?Q?/aggxXoD+IAlBfJwzutINURmwBDdfLLesSZdxi6uEoAEzx8T/1NkPZIw5tZs?= =?us-ascii?Q?lAiphFbivg8mYnRLyCPpIBqkdebXTkKR4UtktxKaiYyta7KaWkfOv2PSCySB?= =?us-ascii?Q?5MWlYpifmgNPUza/bEBW+FeAXQlGmxXsGYI2OCYVJH5ukTXKdDnmexTJfHTP?= =?us-ascii?Q?jQARNH6pKHJVOYzO1h5sz5oHsF5dvCuOHRYgDzNZlZLYS2zy0Nc3iwn4omnD?= =?us-ascii?Q?gPyF1gHzsESHyO9GWe/h7lPpJUCIDyitwvNHSh+vZUmMNfzGAPwwowV2Y1uL?= =?us-ascii?Q?Exv/wDPsxjpyK2hDXbbqzgIUeAXaZ9A7f6CetEc++lpacTbyT15ElW0uOIc+?= =?us-ascii?Q?RG5x1WMlfs8e0qGHQKnHWliBdu60v+1o1megK83z8+blplS/H5iPK7eO49+x?= =?us-ascii?Q?wizjhOFmWxJW3FNipsrooeLm344j1xrFgLa8yJ7HMg12xbXKo6Q+TMJOmaXd?= =?us-ascii?Q?dr6i+sGXjgE9YkBsle7I4JXK3Qegg12Zpc3KFVS/ANMcfLVgRqwhZBBA8ug6?= =?us-ascii?Q?ewOZ0OD9ic2rZK7wayucB2u5gT9yvs0VCjJb74vkFVpAxquEJwJvHev9wPHA?= =?us-ascii?Q?BfrJra6spzS7xtRIF8VxxRi8eaLJIf/+w8AzemMfythu+O/Em4lw2s9HB9kL?= =?us-ascii?Q?bDVpcDqDYRouG64k/IVEzFD/S0zVUETYOunPDIoVql+I1J39pJSu39ExyQFn?= =?us-ascii?Q?HnDSRMxnkab/5UlW+eoTtRNhXcfaMASii1OFY8AKYvCXmLxAaa9wLyiVFLDv?= =?us-ascii?Q?EHVrXr4jv9RQbtyRtUNuzPTgs3Jqf71jnRI8yVHQS00iHJddYDkE2YtoeXLG?= =?us-ascii?Q?bFtY7490WjGWYMmUDwXBTmM9SWncqBi8MJp78wdH4DrMsDawTqYaZChHaUV2?= =?us-ascii?Q?u1IfSW6giRWQLFW/BvqJ4PNeSjLQTStA5Ui0DC6XNcgatxrFnY6E2DFSCx1R?= =?us-ascii?Q?HTDWdjNxgF1xwbPyPsp8zUNDcdZOzNI5r+AHQlKsB+PpCJHbl+vilfrxl3M0?= =?us-ascii?Q?xEnqGm70SRx9VSlV3hyzbgxTD3Q/bsvft307mOHHW7O3/yQdwsuOQmBfNFHs?= =?us-ascii?Q?TGmkbINSu3USC2Sv7BxXnGMmjvvKa4TL90KWKqXUg/xY4sfO80kLIwnY2ELS?= =?us-ascii?Q?mWx6vNt4MMzThOvcVdyFUP6oRVenUXaylgGgrbaO?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?V3NALSOKYpCCpvh/qyzIZhyKL78AqK4EIAxQBe3+nkeExfNerw+O8GYUl4gm?= =?us-ascii?Q?cSI9PV83/vqx0AJa1NJox8G4NXX3KDoGkbVZdvHqhxSDIb1MnXgzFDSwDCx2?= =?us-ascii?Q?7i/mQsrhqgMS/Pp3Wn2C4NcBzi8+C3Km0ZoyaYdQ00yOso539jEdaV8onZ64?= =?us-ascii?Q?oQLO3F9qWW+NCHFxkSn4hQD8s2w+F3aKJNNkbGLxoo04yVduGatvpr+AQW6s?= =?us-ascii?Q?uF1agXbBlIVflUxJF8t9x9kM5S2bEkTZfOXCg5ub2SNejfy+YPmKRsCIPAB+?= =?us-ascii?Q?03BbXliRRWnoNEwhzhaKQjPQjrKrOQVitx3UY08La1ZmEpi5+Pxfb8Mr9RLT?= =?us-ascii?Q?6+GB/1gRO4bReFNKt/pUxWjOBO4FdzdCe/zQp84bt9FquOboByLpnz8Z3LML?= =?us-ascii?Q?7vLHH3m90SsDUu7q3j0e0YFHeg5RxGxZ844qdEsiXCPUVxdH2UeCy5tJzfTD?= =?us-ascii?Q?HQolIjaeZ3Bhj3gtoSO9I88bKndBsXM6MB+t99f7EwMmgm6ocqmrL5ulDO2X?= =?us-ascii?Q?dmVpR78BfI9FhSLqaL/bRAjIoqwtGiYajhim8WBNscUbJs86bF+EznsXRjX3?= =?us-ascii?Q?w7CQ+kD6LxJSnK5krHSdnguzUYvJhIHzCsPIOt6D7P1LNPQrQIvlwTPDdW/R?= =?us-ascii?Q?SeQ6vAqmygaZup193b6vmQA+as9duYhJdtLPoZFxTfb6v36ctWnanMulzS9y?= =?us-ascii?Q?x+/k+/1Hvu3V7lUviYgIqbNtaj2M58xfTAS5Y3fLw+b2zWersQW2Ffs7QeMS?= =?us-ascii?Q?7FMkx8REuB/Xe7iC2M46I3hvjJmqctO7SUeRqkPMQujY/qhNri+MsVzZqIpB?= =?us-ascii?Q?S51xOAdqs9D95ejv0xLfqofBW64CuFI6pLIART2Y7Dtl2mnGSuvPEZKVJEL+?= =?us-ascii?Q?F4HbEKTK1FXxyg4jUvZW2bXKIzChhBMwyrm4Um+f1hfPx8WG/wOvKlDfojqG?= =?us-ascii?Q?eO6SI7lrGxh0MN2EPhg6kIjTL6IEj7zhAxsDqHPv+REBnytAU9Mk7gNzNNYH?= =?us-ascii?Q?jxYfIEg53oCgmfGsipiEdaLVukW1zXi+npMBsTtZSOqMyIZCwIoEFqgQSDTL?= =?us-ascii?Q?Ut2cy8nETOWhCNPxFJ/oB3z0M7mWJZWyphwdNpB/dG6RK5/3kl+nLJRN0JAb?= =?us-ascii?Q?Pgm6wUc2JMmAGT/Bse76zDdngZZScmgylI1m4mC4fPvuQ/3eM6YC/wU=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45791c6b-8e75-4f74-8949-08db34036a2e X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:27.5865 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QocwTopNBygm3UVFFOScbZPlsMlf7MXtjZkZkqwCu5BuBnY9oSMBt0zKthnkf45CgCdJ7F9e6AKy+TyaJX9fK1tThqOLe03lq9uozJEXAzw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB6398 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 adultscore=0 suspectscore=0 spamscore=0 malwarescore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-ORIG-GUID: qoiaRtD32JN3hRfot9KfEK_SPa1ULIEs X-Proofpoint-GUID: qoiaRtD32JN3hRfot9KfEK_SPa1ULIEs Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" process_huge_pages() (and the subsequent call to process_subpage()) translate between base-addr[index], struct page *, first when calling clear/copy_subpage() which then translates to the parameters needed in clear/copy_user_highpage(). There's no runtime cost in doing this, but it's unnecessary complexity for something that only has two users. Accordingly, fold process_huge_page() back in its callers. Link: https://lore.kernel.org/lkml/20220606202109.1306034-1-ankur.a.arora@o= racle.com/ Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- mm/memory.c | 126 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 73 insertions(+), 53 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..d54bc27a35ca 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5629,63 +5629,13 @@ EXPORT_SYMBOL(__might_fault); #endif =20 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) -/* - * Process all subpages of the specified huge page with the specified - * operation. The target subpage will be processed last to keep its - * cache lines hot. - */ -static inline void process_huge_page( - unsigned long addr_hint, unsigned int pages_per_huge_page, - void (*process_subpage)(unsigned long addr, int idx, void *arg), - void *arg) -{ - int i, n, base, l; - unsigned long addr =3D addr_hint & - ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - - /* Process target subpage last to keep its cache lines hot */ - might_sleep(); - n =3D (addr_hint - addr) / PAGE_SIZE; - if (2 * n <=3D pages_per_huge_page) { - /* If target subpage in first half of huge page */ - base =3D 0; - l =3D n; - /* Process subpages at the end of huge page */ - for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { - cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); - } - } else { - /* If target subpage in second half of huge page */ - base =3D pages_per_huge_page - 2 * (pages_per_huge_page - n); - l =3D pages_per_huge_page - n; - /* Process subpages at the begin of huge page */ - for (i =3D 0; i < base; i++) { - cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); - } - } - /* - * Process remaining subpages in left-right-left-right pattern - * towards the target subpage - */ - for (i =3D 0; i < l; i++) { - int left_idx =3D base + i; - int right_idx =3D base + 2 * l - 1 - i; - - cond_resched(); - process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); - cond_resched(); - process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); - } -} =20 static void clear_gigantic_page(struct page *page, unsigned long addr, unsigned int pages_per_huge_page) { int i; - struct page *p; + struct page *p =3D page; =20 might_sleep(); for (i =3D 0; i < pages_per_huge_page; i++) { @@ -5707,13 +5657,48 @@ void clear_huge_page(struct page *page, { unsigned long addr =3D addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + int i, n, base, l; =20 if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { clear_gigantic_page(page, addr, pages_per_huge_page); return; } =20 - process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); + /* Process target subpage last to keep its cache lines hot */ + might_sleep(); + n =3D (addr_hint - addr) / PAGE_SIZE; + if (2 * n <=3D pages_per_huge_page) { + /* If target subpage in first half of huge page */ + base =3D 0; + l =3D n; + /* Process subpages at the end of huge page */ + for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { + cond_resched(); + clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + } + } else { + /* If target subpage in second half of huge page */ + base =3D pages_per_huge_page - 2 * (pages_per_huge_page - n); + l =3D pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ + for (i =3D 0; i < base; i++) { + cond_resched(); + clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + } + } + /* + * Process remaining subpages in left-right-left-right pattern + * towards the target subpage + */ + for (i =3D 0; i < l; i++) { + int left_idx =3D base + i; + int right_idx =3D base + 2 * l - 1 - i; + + cond_resched(); + clear_subpage(addr + left_idx * PAGE_SIZE, left_idx, (void *)page); + cond_resched(); + clear_subpage(addr + right_idx * PAGE_SIZE, right_idx, (void *)page); + } } =20 static void copy_user_gigantic_page(struct page *dst, struct page *src, @@ -5759,6 +5744,7 @@ void copy_user_huge_page(struct page *dst, struct pag= e *src, .src =3D src, .vma =3D vma, }; + int i, n, base, l; =20 if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { copy_user_gigantic_page(dst, src, addr, vma, @@ -5766,7 +5752,41 @@ void copy_user_huge_page(struct page *dst, struct pa= ge *src, return; } =20 - process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg); + /* Process target subpage last to keep its cache lines hot */ + might_sleep(); + n =3D (addr_hint - addr) / PAGE_SIZE; + if (2 * n <=3D pages_per_huge_page) { + /* If target subpage in first half of huge page */ + base =3D 0; + l =3D n; + /* Process subpages at the end of huge page */ + for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { + cond_resched(); + copy_subpage(addr + i * PAGE_SIZE, i, &arg); + } + } else { + /* If target subpage in second half of huge page */ + base =3D pages_per_huge_page - 2 * (pages_per_huge_page - n); + l =3D pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ + for (i =3D 0; i < base; i++) { + cond_resched(); + copy_subpage(addr + i * PAGE_SIZE, i, &arg); + } + } + /* + * Process remaining subpages in left-right-left-right pattern + * towards the target subpage + */ + for (i =3D 0; i < l; i++) { + int left_idx =3D base + i; + int right_idx =3D base + 2 * l - 1 - i; + + cond_resched(); + copy_subpage(addr + left_idx * PAGE_SIZE, left_idx, &arg); + cond_resched(); + copy_subpage(addr + right_idx * PAGE_SIZE, right_idx, &arg); + } } =20 long copy_huge_page_from_user(struct page *dst_page, --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30E69C761A6 for ; Mon, 3 Apr 2023 05:23:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231542AbjDCFXi (ORCPT ); Mon, 3 Apr 2023 01:23:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231496AbjDCFXb (ORCPT ); Mon, 3 Apr 2023 01:23:31 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 103443C28 for ; Sun, 2 Apr 2023 22:23:13 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MwUai024750; Mon, 3 Apr 2023 05:22:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=A+gbjMRQ73fBFl4pqWuQodz1X3vjS3hPwXVwaIKhxTY=; b=x12Br4NU5hCMefRthWdz/zXcq/h36XvBZA48N62EX/+ton8v+E3dG+tJGyTl+HWC5CYa 3R/xWCSHtFlGQ0TvYHzP+HCDkeqyMDCmtxNcJQR3tjyg4OaIHP/X8EPGwKzmn589Z+kV o895JU7vuzXpwWg5ecYA+4wU6UsmXKKq+bxDCKRNHDsDFa5zq0WdqQr1LbybRdn3fvYW 4ZrSZb6OZF62Xjpl3O0pBAehieMZIDnAebyoLvZhTYbBSr2axYUAnyI4kMiwlHjZYivP S3Uv33xF03xhMj6Vm500pHUJHyY/WiCm00wEO6qKDvE0SmgD7pq4fwa+lX8OOJdeW+iB KA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppc7tt43u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:33 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332axZw001034; Mon, 3 Apr 2023 05:22:32 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2176.outbound.protection.outlook.com [104.47.57.176]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt3eu93b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:32 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PK38zv7jQIUEySx9bR3gfkcP96Y/lsJNJ/2lLjaN4SCHIAK6YNGdOCeG9fWdPGIT94L3bSiq9gEePfG9musfn7D5ep+hELd6pHwAbBa76odzDVgwkeqQ+SW8jBwEeN2xPXuDROqVs7HzPbYgJTAXX1X716tXTJ87TTOCbzj1xkD56r+Bj6Rf4GT7P5smXznd3AsW18NMzKTpOD08wzNxuq8Is97dY1m1J9zyMqz0/+9LGhnY++qe4SKzB99y0rQlLyPwkX+958qqGNm4w8+ZtPueeeg3p+8itmMZUvUTY/u2V9DMXBSuqNA6vGThZR6ZjS1+85IOD16Ac2KIdJAf9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=A+gbjMRQ73fBFl4pqWuQodz1X3vjS3hPwXVwaIKhxTY=; b=NqhwkBJ3T/wnZAMZD5mxvms9F0PXji9OnAwpss5ihUcnnVHufroYwY2SI+hRZbVsjR0jSkufPUAQvu18d4edzGm6DeZr4wBPoqxdJ5uRyNSz2FYrhHP0GBt/6gzXEu0OkH3Hxi45LqU++QB+w0LEd/7T54t0A1It0xYjpfXA1E+SUEGyiNFEQy/1mtr21GkCtsbVvldgJpwDrFdxKY4x9J+xT4PwIW8SAuJ5lXRkMfJQzQ+c3l8OsNWlzwFw3dO8g+wBMYSev8Mf/WrTRD4rMKYtNekUt89lFXejm5MlhRL3h8tbbaS5HfKCJgzs2uEa+Cn5RKsQjTQv5GdSZMwTnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A+gbjMRQ73fBFl4pqWuQodz1X3vjS3hPwXVwaIKhxTY=; b=gZnsbqt1OxjTPRecjJxa+fqCRJ3Tf4P/dVWfyJ0tpvTBW9djklyptSjocH+6t7VzT+SuNxkbV3bQtoHx7X/uusH3+bfjsYWMos/rh/KY3sd1HmkMdSoGV5K3Y9iCG4vjkt/OrGlrqY21ih96ObA3+mRhMdzUOf1nKO1Pxe0zUHU= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SJ0PR10MB6398.namprd10.prod.outlook.com (2603:10b6:a03:44a::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:30 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:30 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 2/9] huge_page: get rid of {clear,copy}_subpage() Date: Sun, 2 Apr 2023 22:22:26 -0700 Message-Id: <20230403052233.1880567-3-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0266.namprd04.prod.outlook.com (2603:10b6:303:88::31) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SJ0PR10MB6398:EE_ X-MS-Office365-Filtering-Correlation-Id: e6a7e217-89be-4ce1-1099-08db34036b92 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hyOmS+UFryA5XzOiwc0PreKDQKORHMZ/T0+Z6zKBBhDZP6xUWuulAkf4F2T7HUUdU+MbLk8dLNHOyL2/pLq4VDin1P+t99ziABeDAjdmUtX1fqbZ84ydFA8Mf1AmaTRgePtlIzAGMkOF9nWhaZWTCfBRwRn0ZxpB8BOIztkhN02ZeQSVOIAIBAqe7TRnDJMVcxrAjz2ty2RLMmro6KTgoYNlPz25jN1T5ZkSq3/1Jmt89NxonohqRT8D7thWSslUcsdVd/k7E6Obh1rRmKa8JCQgHQpgfPptdCzPNzs6eYf8YOJzQbouQi/hfGrb9KTXNQiujSt4ERvuEHxx6EnZzq7Vs2wwJjvdG/wu36MahN4XqTrsd3PwzipuzNQ/J9BLMBGwWdklHcbvi+06WDofvBWXZUvsPCgHp0qIjy2rYDsewD0JhaT+eRy0tPhDZzaft0LAIr0G4YD5vgMHIaOrCeV9vTpsxJgkUwG4OmV8w2Ui9Wed8wFtJmskmxZ0qT/W5/Qfd1plm3kBhy7AAkh23ze7XO90/lNBb1mrwVdCOg/5V2OT/NOsgN8YftfXYdoMvyFz7umRBPKisXNxQwF00zK/jVmyJcFQkDdRoOU33t84z2rmAxYdOCNndRRdyno34BdKCka0MJDv29/19dRKKQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(136003)(376002)(346002)(396003)(39860400002)(451199021)(2906002)(8936002)(38100700002)(41300700001)(7416002)(5660300002)(103116003)(36756003)(86362001)(478600001)(316002)(6486002)(107886003)(2616005)(1076003)(6512007)(6506007)(26005)(6666004)(8676002)(4326008)(83380400001)(66476007)(66556008)(66946007)(186003)(26583001)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?j0GKmuPQY6MndwSvByy3LbGQ10SgoBLdvUkrTeVi4lfI5ahCqdVRGXHtJH8I?= =?us-ascii?Q?obDGgh3plyLEKF+ZDdwqCySZRx9YvhY7H8GplmD10B/hZ0TLwggl9lu9GJ2H?= =?us-ascii?Q?UbsjD2G8mm3hSS7p0oo8cBJyInE/7qmZK8sHAgvNpeQI6tq6O5mCwa2IY/Jb?= =?us-ascii?Q?OaHxDEE+2R+In9N53UCFb81bNqT6y+SOr2+dCDCur5fbPKuiF5lXu+rnR4WH?= =?us-ascii?Q?4bHALYa4P2VLg1hTFQyN1us4l8a8ZaKL3ohobCiNL256715/gI5O4j23+k6t?= =?us-ascii?Q?yLhNfnKX9ey62NwBuBeOrt2456IcVU5bJxw+C4LHAnXuKkaA8ZKKPECZ5iVn?= =?us-ascii?Q?dpgQQwng5Zzj3c9PfUKX2Kzn9KorwsVl23SOY6i+x9TdmZmv/LDH/N1Vnt9h?= =?us-ascii?Q?rFCsJLeAX3Aq3lpeldhiYcxFSFzi7LLM1/u/CvbCgoipZFsfX5xFf08M5ipH?= =?us-ascii?Q?v5n92AvKfk18m5VWfFZ+ZsXfIatOY62ov91ggRrJElTA8YSGUUXbJMshpNJM?= =?us-ascii?Q?c7ZXzqqM9/S5xFn0BKWf3rbZghJN71HHGP8so5B9o9GnBUoIragBNNRLaQrh?= =?us-ascii?Q?XTPIyYTF7QIVcm5z0bKRyt+0/eTedLG3X2bzD6WPd4q3OKym31K5OH5gmZI2?= =?us-ascii?Q?ilgA29Q1Z+2j8WoLb+6UJvuWOfLx27HER2w6GsqM8A2ekoMsau9FXnvh7SVk?= =?us-ascii?Q?IMcyCttW521Mho5YRrv67yPdeyfV551e0cgujGmsKq//Xo12C3cixP/RFvqA?= =?us-ascii?Q?VU01WbdHvezZBOwl//2cHKyUcdqHpxhgHTMwwZDnx8eBy17+3LQjJaZ58yqZ?= =?us-ascii?Q?yCKXnj+KQOVHlVJC34cK028zD0C+fcMpEh5aY8zrkR1JpEQnul3tVkiG/wXE?= =?us-ascii?Q?alZX4CNliASaavRC3yD+pSwYnm3nLmV6bw0jykTjtTUMTSEzlxD8gnz30Gxv?= =?us-ascii?Q?VCwzt12YzVjXMoDZ/sjiRW8g53ChLGG1cSHHEznc8OyATgDQqksnyebOSGoL?= =?us-ascii?Q?2gkAymU1RX9CbuzB+sh8UGf1jwTQ27te7v15Atl+YZiX582isRJsJH3gIIEo?= =?us-ascii?Q?d2/+Cxp0H5KgOJFr9jACfb2J1Gc2LywOm+wJrsCcJ/1nAZn6dKKxlMVA9twZ?= =?us-ascii?Q?fivlJozfF8I7h10yg/agx6Nec6RO/keO/Cqw9S8bq44U+rGrL2lxzxauLFmV?= =?us-ascii?Q?JiESyGHAJAEFAXAsow/gUX+eeksXdREEPjVO9OW84aZNFr8oy6BdinpLegmw?= =?us-ascii?Q?+8ImWpjB1SLVJT500o1A4WEJzevGvv4gGyunmmVy9bH6/w909/3zgiAFY7SB?= =?us-ascii?Q?WCsqawIZyXkdnWPV2hEV0U8PSmpjp+V/qObAXsm1vrASPYO749JN13Oiot/j?= =?us-ascii?Q?2PmkzUUKXDSEXNF5gUaTE0e2YiWUgTPKbrwUzoicqKaV/yKvzNK6jAd3ivrN?= =?us-ascii?Q?LwtGCiZ1Z59XYi0DNvhuMaCg+7K5HVdDXwJzW8DCv7IbyJIxkaHJLu14MkIP?= =?us-ascii?Q?G6Ithez8PgytJ8egQZhDJJdCl1JlALSylGtm7dArE/cs1IwAWL49QbDlMnRV?= =?us-ascii?Q?u7xzae3LCyq9MavaFWIs04TvhpjnvehG1fz/qImr?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?kNquaCKMtyHag+0Yg+H3/LTeiyLlfuLyHzIqhWalNpR7d+qq3pLDIcdy/L4O?= =?us-ascii?Q?1ApYlf75EQBlhv0LfBW16Vq0bDmHma2r5EAE/LY4JOhV0xeAR6K40oMhlDyC?= =?us-ascii?Q?kw2fX0/1QaUzHBeX01GXyGMEdFtVQTXaPDnbFdY/YfTllVPDS0HlFEb/7WiU?= =?us-ascii?Q?5EKf65zhOj4mnAt4vGxZaFrz9Hx07fU0J0y/0jistZ9SglfDcPdQ+L7rjYpd?= =?us-ascii?Q?MuHi60Rl0qDbaGJ6nhU91sawHkT9oYPfvT/K5Zb2Ir6F0VQgki60jP+uc3/q?= =?us-ascii?Q?tKQfdzYwnaWJ3RfOpZPXzqQSpm8oyTv0j3Fe5jQWDuzXGbSdu/exjYDiTyyB?= =?us-ascii?Q?cjU2f2G42IvkGQLGQOJEHDfvSE1yZ2GZGedJlrJEC4FKSvMqEObOVG7HEU2r?= =?us-ascii?Q?py+pE1Ojhq5EtOMtL5uqn0Ap3oq1A9KgG2WY9LhVh3I33FKaJgklfDW8lXms?= =?us-ascii?Q?i4BVrOA/PBEVnv83h32Uhs2NNdc2ZfdfWMFf7Q1xJF6/7oUcNntsoRAvp+Gg?= =?us-ascii?Q?xYfd72CfQ+psLollm7d4C5R7Ww9Mq3lNnMML/YOjy8NDjvb0zBuGWd6WNudq?= =?us-ascii?Q?s4ZLmyYCMCXiV5o4gJjEe5LdvFZdx5rGlE9ZRMRQKKt/Q385F8NhM2pxZvI1?= =?us-ascii?Q?8kzI0XfeOZE2CKZGzfBqPWrwzocg/SZ4bK8dZos9s07uUHtd7P/BQObfob9N?= =?us-ascii?Q?XlzHW8u3/vavcoeAYH5CpQCDlqo05L/HrXdj577Kl/R8PRG7S3Y9ezxBoT0y?= =?us-ascii?Q?efaOhGdNueQ1gw+qr/aR2Uh/J9bmsAQh1TlSHjiteOJD5/sn5b8x6og8UV8o?= =?us-ascii?Q?teFFjgCgn9kc89XBkXDRBosn5M0FQV7jokEI6TUcxW0Nsf4TX/qljbhsYPlC?= =?us-ascii?Q?z1J1XSMsLXXShJdbgTlHZp0XGDGmXU4bJbzPA466Bv33aqWv+IkbkcCLF5ps?= =?us-ascii?Q?iX5y/R23HgrbBYxueagRmn816GiJcsDdy/4clV/EwsqQtCoxZ5wKRUWO8t24?= =?us-ascii?Q?PdEDIKj0l4gc1T4O5WbAo1nVkhSEbaRCR0CIeIwBbpSJ1/JaySB35lXZxjSa?= =?us-ascii?Q?0tD8t3+LI3X9pi+Dim7YbHAkiPpl+uOdpjFz/idLxPlsRHNftrImTawF0jIr?= =?us-ascii?Q?CNaAfg+vZ8dTtPR9eNTYm+fyEYMrjtmgp49NOM+ZJpQGrQu+y2DgRnc=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e6a7e217-89be-4ce1-1099-08db34036b92 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:29.9656 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SM97iqTUjbDjuwc9MF3gvVh6t3meW7PO7sazvOEB2EbZ3LrjlimRfaySfs9gjiCEeF2Ms5n40C8pz+khJZt2zTTP+KAzOJxWAbF7tcRazPM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB6398 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-ORIG-GUID: YPAMp7xC3Fv8wlTX6rHaL5Ti_UiHFRq4 X-Proofpoint-GUID: YPAMp7xC3Fv8wlTX6rHaL5Ti_UiHFRq4 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear/copy_subpage(): static void clear_subpage(unsigned long addr, int idx, void *arg) static void copy_subpage(unsigned long addr, int idx, void *arg) take as parameters: an index, a post indexing virtual address, and a base page * which is then resolved using the index. Instead just use clear/copy_user_highpage() directly. Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- mm/memory.c | 46 ++++++++++++---------------------------------- 1 file changed, 12 insertions(+), 34 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index d54bc27a35ca..6da97e6c7d21 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5645,13 +5645,6 @@ static void clear_gigantic_page(struct page *page, } } =20 -static void clear_subpage(unsigned long addr, int idx, void *arg) -{ - struct page *page =3D arg; - - clear_user_highpage(page + idx, addr); -} - void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page) { @@ -5674,7 +5667,7 @@ void clear_huge_page(struct page *page, /* Process subpages at the end of huge page */ for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { cond_resched(); - clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } else { /* If target subpage in second half of huge page */ @@ -5683,7 +5676,7 @@ void clear_huge_page(struct page *page, /* Process subpages at the begin of huge page */ for (i =3D 0; i < base; i++) { cond_resched(); - clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } /* @@ -5695,9 +5688,9 @@ void clear_huge_page(struct page *page, int right_idx =3D base + 2 * l - 1 - i; =20 cond_resched(); - clear_subpage(addr + left_idx * PAGE_SIZE, left_idx, (void *)page); + clear_user_highpage(page + left_idx, addr + left_idx * PAGE_SIZE); cond_resched(); - clear_subpage(addr + right_idx * PAGE_SIZE, right_idx, (void *)page); + clear_user_highpage(page + right_idx, addr + right_idx * PAGE_SIZE); } } =20 @@ -5719,31 +5712,12 @@ static void copy_user_gigantic_page(struct page *ds= t, struct page *src, } } =20 -struct copy_subpage_arg { - struct page *dst; - struct page *src; - struct vm_area_struct *vma; -}; - -static void copy_subpage(unsigned long addr, int idx, void *arg) -{ - struct copy_subpage_arg *copy_arg =3D arg; - - copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, - addr, copy_arg->vma); -} - void copy_user_huge_page(struct page *dst, struct page *src, unsigned long addr_hint, struct vm_area_struct *vma, unsigned int pages_per_huge_page) { unsigned long addr =3D addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - struct copy_subpage_arg arg =3D { - .dst =3D dst, - .src =3D src, - .vma =3D vma, - }; int i, n, base, l; =20 if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { @@ -5762,7 +5736,8 @@ void copy_user_huge_page(struct page *dst, struct pag= e *src, /* Process subpages at the end of huge page */ for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { cond_resched(); - copy_subpage(addr + i * PAGE_SIZE, i, &arg); + copy_user_highpage(dst + i, src + i, + addr + i*PAGE_SIZE, vma); } } else { /* If target subpage in second half of huge page */ @@ -5771,7 +5746,8 @@ void copy_user_huge_page(struct page *dst, struct pag= e *src, /* Process subpages at the begin of huge page */ for (i =3D 0; i < base; i++) { cond_resched(); - copy_subpage(addr + i * PAGE_SIZE, i, &arg); + copy_user_highpage(dst + i, src + i, + addr + i*PAGE_SIZE, vma); } } /* @@ -5783,9 +5759,11 @@ void copy_user_huge_page(struct page *dst, struct pa= ge *src, int right_idx =3D base + 2 * l - 1 - i; =20 cond_resched(); - copy_subpage(addr + left_idx * PAGE_SIZE, left_idx, &arg); + copy_user_highpage(dst + left_idx, src + left_idx, + addr + left_idx*PAGE_SIZE, vma); cond_resched(); - copy_subpage(addr + right_idx * PAGE_SIZE, right_idx, &arg); + copy_user_highpage(dst + right_idx, src + right_idx, + addr + right_idx*PAGE_SIZE, vma); } } =20 --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7320C761A6 for ; Mon, 3 Apr 2023 05:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231551AbjDCFXm (ORCPT ); Mon, 3 Apr 2023 01:23:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231520AbjDCFXc (ORCPT ); Mon, 3 Apr 2023 01:23:32 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E83FAF11 for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MU6pK007758; Mon, 3 Apr 2023 05:22:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=ohlWGuVeTFKX/ILJRWBRlxXLpQTPT1nzsZWWTdMkjxA=; b=Umr8tG/bSTqokt31/SwS8H6nVRyzLETUEd1pJhSfI38+SSBs9QM/3Kc43v9g1rLi1HWA CqJzKOSRWu7GYItYVXUu/o3vk5gcQ4/CChpYx2c1imXvcio0zt1QktC+wtdNcDRuaOAs oQaoeRmiro54Q8g/pKp2D0FzJlb4He0KZVcQppF9yJBJ/NosTL6gg4vg7ZgvMTJ6tCXF Hw+OgIiYdqoMs19OzftY2jdFYSxId9VzxdQufIPDPSdTyN5OXPY4I1WHFItiKkDanh7b eNP2QFmnc4lE0fh12qxcxnF93537u+cb2T2NEkFVQ0hjfvyKI7IAmTDgPA29v7YLqWeo wQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppb1dj5tg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:36 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332iZTo038319; Mon, 3 Apr 2023 05:22:35 GMT Received: from nam04-mw2-obe.outbound.protection.outlook.com (mail-mw2nam04lp2168.outbound.protection.outlook.com [104.47.73.168]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnh1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BTkekeb43YDGaytj8ibdOp8kbxQRmitOIpWZBjfAWkYAMpHeWKdgvyPNkf/GrJUDLUswOpW2FHWzE2Q7/RerKKsxPIqrkg+5TPC7CnkHhftMgdnCbAOcnO3RwFi3ydw5CIfap/7+1ItAVcC6bAph0XVZprTCBV0bxdfFDS9B4C7woqrB74DPbq8ISmkC+KQDiyEjMot9Ctjs4CobgYEQ2u5hsa+1H/z7CcbH5ruc7LvDU8aQRxSqc29hFxySL4o6AbolR/79vgALB9KSL/i2aMuCmfljQQUH03xKi6mqncnCJDd7cHoxh3Ktt9aS7oEOA6E1lbjFsUwFUnC0ALOBOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ohlWGuVeTFKX/ILJRWBRlxXLpQTPT1nzsZWWTdMkjxA=; b=T3+lwwtbDWIuxosZQr7mbu1drV8m+YEFobi7qFxslRFa2qOUf44H4+Vl/iTljJ0JXfVyMW5lpRDgYwLPiOjZjZAdf8ee8d2McwO+CMuTN7kg9wa2yxChqSCmgHXzvDQ3N9b7mLhPlcSct9n3zMjkHK16BWLeVZvv/4wA1jOtuAopSAln9XzGYiTJ7XLdWWNNKg4hlmI2PymAAjuuxrWkzsK3pKx5CSPofIE5w80MRZdFgV5c5bxesJIMYHhgD4/rnjR5yBInsBGMsVRNxykffG401+2Aabh1PIw1t07M2h86eT7W3mh2mKqdSdvaB0CFRayL+hRB08yL8VAxCissVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ohlWGuVeTFKX/ILJRWBRlxXLpQTPT1nzsZWWTdMkjxA=; b=gb4IdZuB+7iEcqZpYLUj+q87gpc6O7iai0Da2r+tnrXyv7oVSoGzvjnVgQ5jKwNRxG/xVXEjazyXNqvXb11o91NvB3btsaVuLTDlU+abbadzphxZ31hERaxAaftrLrjB9wR0h6TJldYpg1+49Jqm5KuPAiNlVqoxiVTsixWkI9Y= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by PH7PR10MB7106.namprd10.prod.outlook.com (2603:10b6:510:268::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:32 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:32 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 3/9] huge_page: allow arch override for clear/copy_huge_page() Date: Sun, 2 Apr 2023 22:22:27 -0700 Message-Id: <20230403052233.1880567-4-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0130.namprd03.prod.outlook.com (2603:10b6:303:8c::15) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|PH7PR10MB7106:EE_ X-MS-Office365-Filtering-Correlation-Id: 01f37a97-c6f7-4797-e175-08db34036cda X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g17nSWXUMfqWTm+Nexe+KttTvJn7LzjMkTOkxboC103/imibfWqRP+dRUAfy0MFiZ3jNho34I+Z5O8FmN3+0FXwMboZQuNpND9xd/2U0kbhed3oxKtcIp4ZFh52MnagA7VWrrRwHKIl/xpiEIZ12vK3J1DX0GnWPqWh6zgr2LMTQo9zWyWIAzJVhb12yWJytl4fhcgux6yreEErmcf8uvCi2b3niF+QVn7zLT4mduWYNbLPZzMThBR67aD2H1vR5GeQwB+RF9UbskebL7t1qCW9TWxSsMXqur96G7BDNzHVjpgLxJ7ql9od+TJP8M2VgHc3smjnMVdYu+3pZwb0bcnj5mYGillS9r3N6WudP+FX9Y2EapwbF1u1mTt8ntNF3wYUb+bjE3xD0wQC0jDYXYXkUO+zPcMMEHt5m6XQVhkika+qZR5KO0BxaPTmVMDL8wcHY+WPI19zyBmt4UNcxJV7J7jvCyoPwRw7m4a84JbiDFduLtwoamA3Yrmr0z9SNK3ZPKMSkmkvrWUQUks3/XvGAIqUwWZNVWPW4z4FKd5JguCbsoG/Z9py0JoQqvsuQ02F/29r558OAGgvOQzACYdo9GVkOh8bPQfIJmmjscAc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(366004)(136003)(346002)(376002)(396003)(451199021)(2906002)(86362001)(5660300002)(7416002)(8936002)(66476007)(66556008)(66946007)(8676002)(41300700001)(36756003)(4326008)(316002)(478600001)(103116003)(6486002)(186003)(26005)(1076003)(6506007)(6512007)(2616005)(83380400001)(6666004)(107886003)(38100700002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?wlsYQxBY0Nisi8aeBxFdIVyAH4j2k7GBLV9PHGjuPzZYKNSHkfVq6//mQkp9?= =?us-ascii?Q?vW5PW+R1Z6w/9s93iHnlLWjJAB3bQRbws17+WSQJND4Gfp3PvWRztXQWPMq8?= =?us-ascii?Q?+qUW6tyAcNjZHf1OZnH1Cy3oMSRyLxI4kYk4wfCTroMT8ze/XtV9EWfq/qp/?= =?us-ascii?Q?IjqEA/tPRrx+x5Vym+NwWEKG3UIvy5arWju7R8QKENBb2HAVdasmSZjAfAY5?= =?us-ascii?Q?vp/E0iBBZb9KoCBuMfz743vKekPiT/mBe1h7zn/nVzXjty0x5wdo+QieQVpP?= =?us-ascii?Q?TKFXKf2ha6jcwMMIpBZ5ML8pkoU+L5vXyhRMR4Izw8oNR2hqBNuuIo+4EGal?= =?us-ascii?Q?OeK4SvGqIn0xFnmsj5MKXOgJBy9nQd4HR6ZIgKYrYGvsIuE6I/WE4t2ilh3J?= =?us-ascii?Q?MBE2L/KoeJJtJ7hHaTIvLP1olsCQR14dSkpo47CoB3PbNyU8oTL0o7h0+gsG?= =?us-ascii?Q?e0GageCqBcrxaUpQn867jMFxwdAhoUtXSzX6Tr/i4+SFJ/jvd9ibHI5AT6MK?= =?us-ascii?Q?Up9K4q215mC7kKEbjB7M+jWXfQLsRLdcL/WTj037CA3mhYBi1FqRS0+gnEhO?= =?us-ascii?Q?la/JL2FtudCOIxfPHEdcZp72HWiweEY3XKlO44LqSmHue/FRTjK7ZRznmCg2?= =?us-ascii?Q?fBxFNx5xmALmkQI/wMwO1DCYPP+BGX7BCgMegE/qKeYRPpg/dNEcLJPcfGAo?= =?us-ascii?Q?nbamqehrkRW59fqafecZPLGh2F8bFotgqi1Cfp1lgojvzpTkq2XhWl+b3XRo?= =?us-ascii?Q?Dr+sfvsmBg4d/CtABr4lweIjsTgchSxPuz0Q6+tFiBuUOSoR5CFpQoLefbmr?= =?us-ascii?Q?HvJEL/j2tE5goCBcxzyVl6iChc0+nBn4xW+AQFftwloS6W1RmVuVoXGjjI6C?= =?us-ascii?Q?m5FbxWr6l2H+Y/Kn3i6iap6XDVrz7R+eUc+18PpuAY6gFYGLs19L/bSxMxwT?= =?us-ascii?Q?RVe+AWAssddLrKBWtsWpnJBawijj+p0+/dS0QIRO+VtqMue2qtYZPJhs5y07?= =?us-ascii?Q?xGAOn0fRBvQVOi3kA8aQ5+CHNZCljyl4JZFEIsP3I6UenBNVfBzNmyIJKs7S?= =?us-ascii?Q?NRoCYM9ljDzSFr/zV4xAG78VJpdWgoh0uOj+KYEkaEfqMdjpRh5bDBL4Vj0J?= =?us-ascii?Q?y/k2K2L/kPVGmJlWfqKR38q28UgPQk2Pp7CZJ81Q4Robaxou0AzCa043/vZl?= =?us-ascii?Q?9CduJgc8UzQYr0ZC8wFLxyJjCUoaJVFKo/lqyswWZKCxWHSFTlbyISZD/qkZ?= =?us-ascii?Q?NX7v6qwn3yPqRhKSKsG8ul6gfa3YnS3ZRYokuJSmRRE3iPaiuL/W7eIVC8ST?= =?us-ascii?Q?W9jiEZf4ZBFBxrEIR+jckelBBTmsGwtBSnjI+feMtbOv0OBUNa4SkCROyMCP?= =?us-ascii?Q?+DPg8eoRIm8bIKF/dZZ8HWP4OFVjoP8DSteky1ydF32A9awPOpyzk7VbIU5G?= =?us-ascii?Q?RfvESJRXBYVcuy0b46mtXVPDgUJlhnwBuiyOBrYrZdqUiJUUmSHamGjlzUTf?= =?us-ascii?Q?VfGMFdKzzkZai6tPg0GTykwCUG+8SiA6m7cNU53mVB/FpyArWZiRNS/Fh4JZ?= =?us-ascii?Q?8mZAilp+HIZZwblQlgV/RKPZ6ptPlPjkDZAlB1C3?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?6e0FLCQ+TObGhDfhFPzRPTGMNfJoow+YEhi5Vx+XU3Ub0tJ7l5c6qDftuEAO?= =?us-ascii?Q?c3SaPxFK1uzl/sz8J/g5gCAKslwZZWnpTM0IbM3ivxuPrYYrz2dv6CdfVwje?= =?us-ascii?Q?/HI9xQayT3Nh65GVA3i/QEO111D36XV0uUAvBK2rontz27fxktQMhHloAoqw?= =?us-ascii?Q?4k9on4JYju5AWM/9YzTeg0q3e6FuUgqYsYXoLmKzPZU9DavapHU71P2FczNh?= =?us-ascii?Q?LxN9veKWYAPYETmYBvhEn6YzHGnLW/U2OKUYGQaNq7oYzNYCaf42SzCTBTXi?= =?us-ascii?Q?tW8eo3suuTdjqtVLHYiRLIM15JOUYQERKF/K/6WLHLCtoWp6dhkpXUlCiRkK?= =?us-ascii?Q?igpPdSCt63i9G0BmLBQnjJ6yVXVK6BdUIZPcmg+b38aWQ6xl/u9M8jBil622?= =?us-ascii?Q?+w27l9cvmszEB7hye34l0rXilTbe7ke/JvzglVPvTBiyVIna2F7zpmovGS56?= =?us-ascii?Q?KCsJwF6ZZvnbh2qc9VnC0YNAWeoEBlz7Md9McaMTZJOHf8V9HFNqmuQfhDai?= =?us-ascii?Q?U3xXwEkcymafwQbHx+coINTBWxHUZbgDmzlNsPDVQI+yp7Cg9m2Ua6NRH6Gt?= =?us-ascii?Q?Lsorr149K81AakdEKK6c1m2oG774e2Buasg+58kl/6+sYU2MghvWMs2cDVTd?= =?us-ascii?Q?6vvT3Vwqvex70e5V9NTF5XiYV7DwmAtF8EF2/3HymWef9Fyqt3BF9gfxxz8O?= =?us-ascii?Q?ouZwxx6fvtUdUGR8FWf7nHjZvYmDGSRMe7alpM4448LGcgbL8MVIIy0Mx7P4?= =?us-ascii?Q?CQngz8vGWVPYupevmV+4A24uV2xcZLdcKLOF8Fhfhbui/E09JJUZfcVdzVhd?= =?us-ascii?Q?tFaaATrL2qwJ/bVqwvbWh21ZvQJKkmmgg9844iwM/gnrM2qYFzIUrZkuRKsW?= =?us-ascii?Q?QmfpNVbcD9Vftl2mdoAjvNHJpfo3uPHC/IHtNhPsJP9kRRwiKjbLbm1c9gon?= =?us-ascii?Q?pEplWRSwpgx34yf/0kzR9FrfcOsrhYVIvB1AjJTOqu+QOus6gPXqXa27eY/H?= =?us-ascii?Q?pu0fdo/fPTtPGGyNPf9yGRW+SZdxB4Rv60p16jWjHr0BJ0L2V3UVNjN2r9Ia?= =?us-ascii?Q?0YNO4ynT0L/749d9GhbtxDZ4JrR0ScPUYwPgShHE2rPK3cNLJNYhl5ig4z8g?= =?us-ascii?Q?ZUrzvRE08ea8tq3z9csweudX6rPwbaIN7f8Z9n8Ex8mIgccvnOPIG88=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 01f37a97-c6f7-4797-e175-08db34036cda X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:32.1264 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rNJXMkL48RPF+xFdQXcLCh0so6eMyDa5HL9gezr45m7jf5XtCTvjoS1tnyCjf4dVMTs88RSIBfpiVA+VbXMR13gPNT6KLzT7YbKEoc9Y76Y= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7106 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: ewEv5zCY3Gyz7aBBO5lxVpjQuKxkAKo8 X-Proofpoint-ORIG-GUID: ewEv5zCY3Gyz7aBBO5lxVpjQuKxkAKo8 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear/copy_huge_page() are constrained to operate on a page-at-a-time because they also handle the CONFIG_HIGHMEM case. Mark both __weak to allow for arch specific optimizations. Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- mm/memory.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6da97e6c7d21..9a6bce384616 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5645,8 +5645,12 @@ static void clear_gigantic_page(struct page *page, } } =20 -void clear_huge_page(struct page *page, - unsigned long addr_hint, unsigned int pages_per_huge_page) +/* + * clear_huge_page(): does page-at-a-time clearing to handle the + * CONFIG_HIGHMEM case. + */ +__weak void clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) { unsigned long addr =3D addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); @@ -5712,9 +5716,13 @@ static void copy_user_gigantic_page(struct page *dst= , struct page *src, } } =20 -void copy_user_huge_page(struct page *dst, struct page *src, - unsigned long addr_hint, struct vm_area_struct *vma, - unsigned int pages_per_huge_page) +/* + * copy_user_huge_page(): does page-at-a-time copying to handle + * the CONFIG_HIGHMEM case. + */ +__weak void copy_user_huge_page(struct page *dst, struct page *src, + unsigned long addr_hint, struct vm_area_struct *vma, + unsigned int pages_per_huge_page) { unsigned long addr =3D addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFC49C76196 for ; Mon, 3 Apr 2023 05:23:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231575AbjDCFXs (ORCPT ); Mon, 3 Apr 2023 01:23:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231524AbjDCFXe (ORCPT ); Mon, 3 Apr 2023 01:23:34 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 652E21987 for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MM9Mx025427; Mon, 3 Apr 2023 05:22:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=UjM07Xo3SboE2ZDIKJWFIgfW4cJRS+1QtuqFInHyTCg=; b=1WOqz4d7LZ6Zo//oL3jSMkScTNg8MkljM3a2AvUGDRL9NhXxfqwq6za20/VP8Rt+BveG WwSOv61eHLje0XWyl7g/KxyPTeLKcPmtMUcgIFEEgyiUiabzc3qyqlLA54RWZwk39pE4 Hp/F/e5emxKhiPkwlKrNUnawXgj/aT8aC+HtKvsyMyIuomzQsNKONRa7mEGhzRgaOadU YcP+bAXBKdCxZwyep90VRVBKJgcF6Q6U/6PdONgClE5F0yFTtEzrqj/zun136DGpwSdE 5iAJV/qf2UUIN2E1INTUl4BZFQcOQHy9yDRoLrY9ic6rp5Q15hQE3qIvl0mEzTGdWpYg Pw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppb1dj5th-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:38 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3333Uv6Y038324; Mon, 3 Apr 2023 05:22:37 GMT Received: from nam04-mw2-obe.outbound.protection.outlook.com (mail-mw2nam04lp2173.outbound.protection.outlook.com [104.47.73.173]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnhk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FwI3s+LsyusrVjnaFPv2Y/9rT3KrXGz3nEkFzYrxeiTsnBdSTjp/6CjUmfHcVNQieDLldaTL+BjEzNk0X9JyJ4l1YN8BII3sfw85XNRqRiWSTSmTdQhTpqngCJQ0svNGurkzEWjsQazl4ehJ3nNhHlZSXrvZ+4TdXiduLByVw03Djj815SpLdetgVS8KlFIV4MpzCDl+L18OxaR+6x6MD4oG2q2g4KF0RAh/KV0DkD6mrNJxZQ3N+uZHYvQMPOFCFBX1QpQ8X9tlxA/WJMY/laBaIzN594ktvMTX4TDTJIsqnNywPK83UCgOZBi8ImNrT7CA7Gr3hD1N8md4iNVjCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UjM07Xo3SboE2ZDIKJWFIgfW4cJRS+1QtuqFInHyTCg=; b=BB1EaJyKAaWcDrRfS5W9Dqz97Qe7zjeiSwxNWKq1MREshAbkEk7xGVK0S/gSgaoCtVtEyBg7cgi9mHSCrW6axUt/6LGd1tv0ijJVME5XVGuAaAY2UYRbY3Mp5ZvEX+zCnwdA0Z4OxXcFkn/DuNfcX7GRlMNDRBVadO8mLBNa9NabalGV7jM3wotuvGiIwdqJHRfLs35gXE//AFIgFzIby0IwPM4aNZhyM8A2Em9yICs1thqByfGQPx6I6862YADtWIG3VA1qDqNI8KZcVSlRVoFYYIrveTQAHZpIHSGU+gCz7YK+hm4QAS+XIjgSrvQ3h09H8p1RR0CGgWYEz/A+OA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UjM07Xo3SboE2ZDIKJWFIgfW4cJRS+1QtuqFInHyTCg=; b=n6sXqHMaO5hzCTlLFo8CTQFiIk8a/a/x85w1hx0YUnRws02gna4e7JWmCr7AwTxFAUcGU13b82WatIQDvsNyU1htOOfd01dEnx79PsbLdLp1N4hqBkf1NZflzM9bWMk8D+rLEwEI6MCa5uD0Ig8rZGn95VBl+udUA1Yjgc2lHxo= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by PH7PR10MB7106.namprd10.prod.outlook.com (2603:10b6:510:268::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:35 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:34 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 4/9] x86/clear_page: parameterize clear_page*() to specify length Date: Sun, 2 Apr 2023 22:22:28 -0700 Message-Id: <20230403052233.1880567-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0247.namprd03.prod.outlook.com (2603:10b6:303:b4::12) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|PH7PR10MB7106:EE_ X-MS-Office365-Filtering-Correlation-Id: 1f2f8e1c-02d5-4634-cd1d-08db34036e7e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WffUYm/xCiTWr3T6eBIlnT5ip5e6/1255qgOJCqYMQU1Dbqz8+vjgF8lRUmDYweMNPybA+AhbJC8VGgnyroj2NV+ZudsVlg0aD5+HVm6s7lY1xpFBcPx81m6eaiNCpWgYBPyc4EdRPadCeBjWWjroUUObsxOp+/UTim+ZivcayyGP3qW6bJli2hP94ymhS/DRz1dV6bhKgW9KGW3nI0oLs3E0/7sDrWUnUDHHbYDYzWssM0in0DKbX7ZULzgi+gp1nJxKR6Kvj8ne6/x3lI0WhjGpNRsILD9PXC+zG9jw4lWq6KqgQds2SjkAuttGRPx7zNe7zIX6qUF0qbYXfBgXSo0HOwYnSAGUx5TZ5DuJl04/NrIvZ3YBkjEGs3gTm4w0eM2MOSdBhjt9O7yKdSK9wS8QiDNKOjrmMWBNKGZc27eMDQHh6FhGT2AgIxcF3WLzk6Z6tbwJDFCqTYv5tGIS/jU9bPbaTzHdNwWOna37YZFx7m9uVaCjANMlLkWD5MhaFN4ammv7cxZ6nX6UaYJJH8gtEOx8fFixwGyI/ENpmt90OPowiHDnH2bAO7gIaQh X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(366004)(136003)(346002)(376002)(396003)(451199021)(2906002)(86362001)(5660300002)(7416002)(8936002)(66476007)(66556008)(66946007)(8676002)(41300700001)(36756003)(4326008)(316002)(478600001)(103116003)(6486002)(186003)(26005)(1076003)(6506007)(6512007)(2616005)(83380400001)(6666004)(107886003)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Yx05J4aPr+IqbvHOpvXk8ZEo8RheJNIi8S3M8yMX1PhG+wVxEpvFRvCkX/I0?= =?us-ascii?Q?p/aPxICamVJ4bCIPuVwNf6SmJCbioX3TtahQIlZegGNjon9cYJEP4y3rFx6d?= =?us-ascii?Q?fVYYUVxbZW3qaDH/JZmdvsHj4DRJhI+7Mxu3e5Wgyu2iTun++LGIjhljWSvP?= =?us-ascii?Q?Bv2bc+B9vk+C7s2Ncdpjkta7e6y+EG2jFAP3zCFCF7earEul2wJwUQ5neJq8?= =?us-ascii?Q?Oy3Iip89LOrkdagaocVB20+XuUsKHky4tFgm4CL3JfV1oWVKQctLP/K4Qqxc?= =?us-ascii?Q?NmyLo430hxlmso4qu+PwnhdpZmsEKV3eMA/0uPwNypAWI2K0kb5S9qVWKnU2?= =?us-ascii?Q?VuU4d/WW5/ci++QdEDYxIgh6ekqJubcNMWRhQR+hgJ+zOo9gVINXGrO4RQ46?= =?us-ascii?Q?Vqg7+7ZF3b2P12qJyJg1IiHx3E1GHDfUM1IGGc+tkTD+1XvnaPCjSYD2SIH4?= =?us-ascii?Q?m1oxNaSIcAJ6ja1/fPWbUApPJQnyjKkoUbdEHejG/p/Y8kD3OzWF//HAU5OW?= =?us-ascii?Q?LyY8cOxCRbZav7jI0QSu1HioYDP1vKwYjfCBpf88NT5yBYjWB7QRKyTlfGAz?= =?us-ascii?Q?gD5lGEn3fzJ6yK2kcK+Oony+p6KyoWg83AObNo/KEQV7/B/AIr5SY0fqlzog?= =?us-ascii?Q?76J4xT5nv250wvaCkUQ2CIAgJSgFR2SSJfHkEUzfdQ7EY5jqrP9RyZm1paIA?= =?us-ascii?Q?SiB1QTFLq21SxvQWW8dNAwuV4hyDKUcIyZpOhmbAhPg3zeHAqWBPyW439F/a?= =?us-ascii?Q?zPXSm78lCW4Ah2gGrZwCuo1dpVQbiiARmZY6gxAUpO7VE/9hbwGZJ7wcmeN2?= =?us-ascii?Q?JD3Hacr6rhlC/rie+3qnh4vDaUGPhP2iriTzArAzHCyELI4hvX2qhoBD9sih?= =?us-ascii?Q?0zEhAS0FFlhr8x7eHA1XrLX+R/IsBRDxvv1aJavyBQAgCRsOO2nLJOkODbK1?= =?us-ascii?Q?TyMKPsHfF9pNTzRSUDoEZqS2fBQ3pmstevurkwT6RaWVthxhnOQCnSFLbkVj?= =?us-ascii?Q?y24fem7oiiPRoX99LaE44zLKCFSVEA2hxnm35ls09PXIylHTs7AXWXpNogns?= =?us-ascii?Q?IGFOfXzX0RyRE4OrX0GsPhCx+OFe+zZEQH4176UFBpBhQvYfRg/WsPc2JDDk?= =?us-ascii?Q?phILFJxzkg3+fQ5m3ON1M7Qo6T1PgepMhwyBAnJnjR/y4MvJcM97VvLS0H6o?= =?us-ascii?Q?s8zSDe+BV8+ULdGtoZeDlPuv0J7v68xzD3Q5QKcG/1fw0MfWQDjbzd13jQ00?= =?us-ascii?Q?UhA5wDg+v3x22iMp37a8l8vkQ1lfljoq/P1sqB6JIzPTIcznG4wPe/yCQnV2?= =?us-ascii?Q?YXWaG1THOA0i/Wd7Kfya2vkExOQZduwOTvCEeh4GoTR/All2o/4irtvv8WP7?= =?us-ascii?Q?MLqB5YYrRN73sYR8fGz49+/+DZuWGm96U2uVK8KaI6xkubdmh/qK4k40IPZH?= =?us-ascii?Q?AjrYNpglxONXvCg+dvGugdujKHPYBqJAZRKbqmOoB1BwK57egGt4pTNmKmZl?= =?us-ascii?Q?NzXPa9peJ5w2ipXLJklEzzPljhvdet+vi5NBouyeNDhb+Xfm7+0Kk87s/WMG?= =?us-ascii?Q?AB7sdATzrEBFRdUxqADA4/Y1hMDfVcI7KgIHE5iD?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?66vfYUq6go70SfEIgOxUSAfQFij07W6OPe7FvVXIA2JL9SaJN1lsQ0P9kN6Y?= =?us-ascii?Q?4rTu7ndfTOxse2dvOohodIA0Lq1+wzaeA82Ic5CpC/V6zC9cqqwHTIzEt4EJ?= =?us-ascii?Q?Zbn4TaS1p1XavDY7doWOFz/89T5/DxsEmB68sjeWtuEl4xtfHPR1x9uUr2kT?= =?us-ascii?Q?TWu2U45MzhW92G+krl3TbfJadq3R+jiaLgtaUjmW0yCb7STlc3NJ8/LSUeAt?= =?us-ascii?Q?Qb7dcH2AtRct5XUpfIv0BLKvAtay+6yDMfkWGbB6OH/4FkSNpHGQkV/qiBV9?= =?us-ascii?Q?Ni/COzfQls5S86Jv6U9MfmZ0QV1tTVbdaBf0NSwF3dGFWH/lorLy6zGkz0kb?= =?us-ascii?Q?oKyq+NXyzTVVTboASyNFjtT3HeF07ehHnItLhREr9jw8YXBE20fRKxnf+zbj?= =?us-ascii?Q?uQuZZj6H0mgNslEyYe+5u/6UatAdn65rqksMKzGpe48rDrWOeYKHv9jNClhi?= =?us-ascii?Q?OLZicGTVdsXajQZuc8gmIgec3pTJLz5EuCxm/vHmPEPWas4J5KLpqTbEIfuq?= =?us-ascii?Q?mzl+zykPLLX7OIBWs3bWg7qQ+srAP955viutQkVSx0Pw9amMJTUR2zj0XoYF?= =?us-ascii?Q?zcBsuNXAnp1Aw1u36iCQyFV/+IPGFLFni6+RRDuQiN1MoPj5qiu39aFIiA5F?= =?us-ascii?Q?HQfo2wt122S/NbMr5U0DZg7rhhHk2s1NBg4MitwNQeXRhBhNddOm99i8Xa3/?= =?us-ascii?Q?WcyX+00JmXUs9dgY+CAPKUI/+ixckPkqcAlISt3mN58Uo2xE9w1vmCvjMplz?= =?us-ascii?Q?3v5GdYd791XzL7ykK11UZxfLU8Cu01DZiFO1wKzoR21vicK0lb57QbDmiy4i?= =?us-ascii?Q?Oyokuc+dVpijPTwZ3KI5qE7bFtmmzTetWq5oTNzhzQy3STJhv/RwZDLXSvaO?= =?us-ascii?Q?01SLpBh/9iHHZNYDPkjPnTG3O7muGBArh/ySj1HTd9Ttioqx35dy8dKNIrBb?= =?us-ascii?Q?Dl8dItR61iLzacZ6lo/AcxJjlgRNrQgc61QTKOxIu0XGjJaYaRp2SRfWlFO/?= =?us-ascii?Q?z6wJlT7z7LaQnMnm0MauNJ9rB+JoCzXJ0wD2EVopg8rkvF6aQFvbqRIsSzIL?= =?us-ascii?Q?zFk/IJjYh03ex/xil9SDxMYNTv3Yozh1NIG8UPsflZ71cNkitpIa996NfsTe?= =?us-ascii?Q?aJXGrEsBaUDU59AYXH4oJylL+KV5RxnMAschuljt4yZ0dq24P4FBUAw=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1f2f8e1c-02d5-4634-cd1d-08db34036e7e X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:34.8234 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3va0khPeFN/HQs9xoXsiigLDI9gJ2EtiUo0uLwscOI4GgY64IrUkLy63zeq0OBaKYqG3GTbBxGYyrzeqRqpovULADgJ39+1Ezviu2Oqk0yY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7106 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: rLVRmwNI7i5to-8xeaNRvZvYUnveTt8w X-Proofpoint-ORIG-GUID: rLVRmwNI7i5to-8xeaNRvZvYUnveTt8w Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Change clear_page*() to take a length parameter. Rename to clear_pages_*(). Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- clear_page() is now defined in terms of clear_pages(). This means that all clear_page() callsites -- which are more numerous -- would use an additional register now. Opinions on if that's worth optimizing? --- arch/x86/include/asm/page_64.h | 18 ++++++++------ arch/x86/lib/clear_page_64.S | 45 +++++++++++++++++++++++++--------- 2 files changed, 43 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index cc6b8e087192..7ca3bd2448c1 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -39,22 +39,24 @@ extern unsigned long __phys_addr_symbol(unsigned long); =20 #define __phys_reloc_hide(x) (x) =20 -void clear_page_orig(void *page); -void clear_page_rep(void *page); -void clear_page_erms(void *page); +void clear_pages_orig(void *page, unsigned int length); +void clear_pages_rep(void *page, unsigned int length); +void clear_pages_erms(void *page, unsigned int length); =20 static inline void clear_page(void *page) { + unsigned long length =3D PAGE_SIZE; /* * Clean up KMSAN metadata for the page being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. */ - kmsan_unpoison_memory(page, PAGE_SIZE); - alternative_call_2(clear_page_orig, - clear_page_rep, X86_FEATURE_REP_GOOD, - clear_page_erms, X86_FEATURE_ERMS, + kmsan_unpoison_memory(page, length); + + alternative_call_2(clear_pages_orig, + clear_pages_rep, X86_FEATURE_REP_GOOD, + clear_pages_erms, X86_FEATURE_ERMS, "=3DD" (page), - "0" (page) + "0" (page), "S" (length) : "cc", "memory", "rax", "rcx"); } =20 diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index ecbfb4dd3b01..6069acf6072f 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -2,6 +2,8 @@ #include #include #include +#include +#include =20 /* * Most CPUs support enhanced REP MOVSB/STOSB instructions. It is @@ -13,18 +15,30 @@ /* * Zero a page. * %rdi - page + * %esi - page-length + * + * Clobbers: %rax, %rcx */ -SYM_FUNC_START(clear_page_rep) - movl $4096/8,%ecx +SYM_FUNC_START(clear_pages_rep) + movl %esi, %ecx xorl %eax,%eax + shrl $3,%ecx rep stosq RET -SYM_FUNC_END(clear_page_rep) -EXPORT_SYMBOL_GPL(clear_page_rep) +SYM_FUNC_END(clear_pages_rep) +EXPORT_SYMBOL_GPL(clear_pages_rep) =20 -SYM_FUNC_START(clear_page_orig) +/* + * Original page zeroing loop. + * %rdi - page + * %esi - page-length + * + * Clobbers: %rax, %rcx, %rflags + */ +SYM_FUNC_START(clear_pages_orig) + movl %esi, %ecx xorl %eax,%eax - movl $4096/64,%ecx + shrl $6,%ecx .p2align 4 .Lloop: decl %ecx @@ -41,16 +55,23 @@ SYM_FUNC_START(clear_page_orig) jnz .Lloop nop RET -SYM_FUNC_END(clear_page_orig) -EXPORT_SYMBOL_GPL(clear_page_orig) +SYM_FUNC_END(clear_pages_orig) +EXPORT_SYMBOL_GPL(clear_pages_orig) =20 -SYM_FUNC_START(clear_page_erms) - movl $4096,%ecx +/* + * Zero a page. + * %rdi - page + * %esi - page-length + * + * Clobbers: %rax, %rcx + */ +SYM_FUNC_START(clear_pages_erms) + movl %esi, %ecx xorl %eax,%eax rep stosb RET -SYM_FUNC_END(clear_page_erms) -EXPORT_SYMBOL_GPL(clear_page_erms) +SYM_FUNC_END(clear_pages_erms) +EXPORT_SYMBOL_GPL(clear_pages_erms) =20 /* * Default clear user-space. --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 223F2C76196 for ; Mon, 3 Apr 2023 05:23:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231522AbjDCFXe (ORCPT ); Mon, 3 Apr 2023 01:23:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231511AbjDCFXa (ORCPT ); Mon, 3 Apr 2023 01:23:30 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10406AD04 for ; Sun, 2 Apr 2023 22:23:13 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332Le24N009559; Mon, 3 Apr 2023 05:22:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=gW/w4UBNoRyDr+fj9Sa/039P6gNwlfFSpI3amaMp3cc=; b=Dn/H6HZmJlLesKaHwksiUDif7P9mM1uzZUQbaX7ujbmfxhyibCNFgboF/dDK/CBCwqu0 h2QbvTUK+n1rQXsU6YoJwzJ1LeBU6ugQ+POVfi1CCeMxsxllbTsYWD4mCJJSd4lcySed MOhTtxMnW7vR5UWkfGDoSLAy057orTt5kyZ43n1A+GITTPyEPxd59Sn0HlHnShjkZovn CiVBxPJvoMsr/WIdfyKci22EVc/saxmgQzKI3WT4783X30rPMcYK1ZM2zE6DcY2WshJv fU/nC5drYb3LNlu8W+NyxEg94i7NPfGtMOMTBVpnPXnexiD1f9bR53BpGqZTlUJsa4ur ug== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcnct4s8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:40 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 33358gdI028272; Mon, 3 Apr 2023 05:22:39 GMT Received: from nam04-mw2-obe.outbound.protection.outlook.com (mail-mw2nam04lp2170.outbound.protection.outlook.com [104.47.73.170]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt243jdt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YgCfHXKDhVlODPaTrw5tanTx1s4DBtDjp10We2gX9y1WCAYc5JSVdLm9NcTQL/PsmaXGpDcUIeCsglKey1HO7oTR1GND7JPpIIFa3tvq28/LbNZnnAD2w5ST1cRFqRL9+z4q05CutuvCMbsqci9OFMTO+mzy8DR8C4/HgGiARxf6+ghonpShoLolKP8CbK2BTn2ineBgD0ZhNqUHTr+HQAEtjpTt0jmsyCX8fhAPPctTrLuEtOAqF4Ce1BXWAAeT7/T2jkJ4b2ZNfSc8y4yHG8frlq2YOvl3EpPNj27DeBzUyexe80aPe6hF3loFZSCwut8f52zxZ8K4PTjCgc8swg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gW/w4UBNoRyDr+fj9Sa/039P6gNwlfFSpI3amaMp3cc=; b=Y0Cm+uvbhKlFfPzOGyQzMU+r4HcyxXl57tCCL7aBTcn8OhTeR/sJeuS5jgLVAX/ydBhX8nCHgVckAb1BX4T8MC4gfJTH4xt3VMLjGeMf3oF9QOEVLjCNOW865vGKED4loLRu0s1QD/gn1yEAq0Yy3hiYts0EdHP5JzGMHhBV4Tfq+aLAMngaUo7aYlPRtY8PN6CCuCdKBDG7gZKm8uvB2q6M86Nx7RC9uZg+2SV7CDdgbqxnQp/ygjMxLakNJ9G+LHvP7EgQbRmXUA9U8B1axiCyzMvWREOi1NOvHl9jFDf9zH+KcgzWR7J8SWX0A38cH/sLqwOj+n1PBnLTCESf0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gW/w4UBNoRyDr+fj9Sa/039P6gNwlfFSpI3amaMp3cc=; b=TUfpdY2tQMSA5eHHsYl5p6bY6EywNRdeKZjsmC2fKFyqoqO+MyeqKU6E7BcHFQ0J8vgYDFx3uq9YVTH/6+vuT7DPTFuVtRW+YzLijajZ7yuiAQdxIgZWgBaHA0ydlOzWuMdtsPUYtUc4G00QHExCXVdL3+MAevsGEf4VjuIuea4= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by PH7PR10MB7106.namprd10.prod.outlook.com (2603:10b6:510:268::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:37 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:37 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 5/9] x86/clear_pages: add clear_pages() Date: Sun, 2 Apr 2023 22:22:29 -0700 Message-Id: <20230403052233.1880567-6-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0289.namprd04.prod.outlook.com (2603:10b6:303:89::24) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|PH7PR10MB7106:EE_ X-MS-Office365-Filtering-Correlation-Id: 25fc5ff5-c6ed-4530-c68f-08db34036fd5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TyFwhO7FrdtV/0UFCTJWxkzDFsU/vNJIVTciCP1I/JqKVaSZYgA+1t8RsdQr870dbd/hjMNcGFzL+hEx/Kms9eDvv2is2SneGKgPGv4fM5ZTgLXXek3jyKFXemjKOcwa7X6MZF4TuGQ+FoAJee64573fiXl0S9D6dLEK0Sxck3FKaBWVBQmgVdDc/zwTZpUj5DxafWRRedqiiOnqiLBllWozYFkmZRublZsWyfMmEKgCU38csPOW/mcNX4p46/H40SB46t5li5D8zb6kU+XFVTF3oa/G9iYuC8QhbJpY7FmvoH5us+WB2t4dphASX1sriblXzX+5Mmgd3VMLYZXeCgSjEmm74pSXHXawL8bbMHmSUsNgHA7b3Fd6nrx403dkYoKCjI1vHk8efgs3xxitry4C5vVhwpEJfX4NpTUk2ysUL89IWfVxBpHOfBideFJEN1tUnJjNZKMnSWCUVOtAVFO+Ae+DP/vg0os05zAo/C/wYnTUBsGPlHmS3HAiPaCDYW9i1mSIU4E0tSS37mOtTPWSFz1xUrtNmc0ARDWfdfGqjDd+Ws/qY5iqJk6pKo5w X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(366004)(136003)(346002)(376002)(396003)(451199021)(2906002)(86362001)(5660300002)(7416002)(8936002)(66476007)(66556008)(66946007)(8676002)(41300700001)(36756003)(4326008)(316002)(478600001)(103116003)(6486002)(186003)(26005)(1076003)(6506007)(6512007)(2616005)(83380400001)(6666004)(107886003)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?jOjgWgbM6lFjlTtRLSnNCkEMEwDZYEDYYjHrEMfAGp5ldXwtbaEOmYfQ6KGU?= =?us-ascii?Q?UJwYfijNkHMz9ie3i5R9PugOFcFzQw54MuEYUS60CXM3NreNkbxzH9Sg8u3f?= =?us-ascii?Q?SGv94wGVXne0Jpi+bixg3hl+5LTaKpY2TI7Cc4521nc+SD4X+yEuRGUkfmR4?= =?us-ascii?Q?3QuYIIjLiH8ku6FZN+V9E5jhcDY2bMk8T9MjtyNCniA0+bKZXsrqlNzJAUD8?= =?us-ascii?Q?szfVoRnBlOKAsijKYrAuauxHrwEmunX5DzYpaVyyVxYHP4FQJ8Bw35TpY1fI?= =?us-ascii?Q?Slkrpb0hA1nNtQxIedW+750L0EnAFpU6XwDFo36uN7JqX4F8G2KdKG8vbA7a?= =?us-ascii?Q?Ozp+LUbt9zTBT9MzB8NMnEwDegPJfA8ygAjxZpw0FdKL+8qyl1bsj9zu1uh4?= =?us-ascii?Q?0tRAVsuoPwBVXoqQf01/j6xhNIeDr+NIEtwm0uiTJf8WA4tlVHO/BJ7lLaGQ?= =?us-ascii?Q?38c8pk42oDtIrwUqb+cRv0W+dEnUdSWltfeeA9ql/KR2Ngl1OfaV4uWVN1y8?= =?us-ascii?Q?FcKBwDiteJpQeiMjosyTCpV4GRQPDCaptbUNfIIBjbWwH2xFrPcySmznDmQH?= =?us-ascii?Q?KVRQaAYb3Lyg/4AfSzj4nIUkrYMo5XLij4USuBPTDO6qWADgDW7eyB3fZyIw?= =?us-ascii?Q?iO6ltyUDkUrc+B5vbRzqh01ayOoWnhAStXgTP8W/kt89CczJKDnZplLkFDXP?= =?us-ascii?Q?eCMXeyCttXTLYFgrWj7zCLMTveHUBDqR66q2X5Fc97l31nyIWjfiHn4GLepB?= =?us-ascii?Q?az56KRDsHN4Aoo+Tjl4BxDmVYo9DUi1k8Hl1CvotE1aUe/7n9YBOdn6SXpDa?= =?us-ascii?Q?SpaqN/Ovjn/DslDaG76Ssl9jP1i8WexH5XzcJi0vBxP7fz0E8ncZAGfMikQE?= =?us-ascii?Q?xkRGXM+tvBzoJhmxcVaA8icU6Qn7zPguA3g9H5RukupY1w/PKUre22FvU5xC?= =?us-ascii?Q?OFY47je35d8zSlUJJ0LNpqsy7gvZcgeB3mYy5kp2JiCmjzXnt6Jw26nC2ml5?= =?us-ascii?Q?RIjtEpcIH93U0ETqIuFhbbYYDLsxQ3Edla+cQEDSIav3kG/sPBoP4tzrgHFk?= =?us-ascii?Q?PsPj0SqyXPsnrIVE0gwZAdMst/wfR2IZEixV8sHY2phU0yk5/+DCOrZdK1Cl?= =?us-ascii?Q?wCuR5Qr6xuE4RtGIcGY2MxuUl5BgIDtu9A550lCJ4/I271lt7edHM0yiP8rW?= =?us-ascii?Q?Uz2VX5pgAY+2NiIcy4UWhyQ27UplYyEAdQdNrrDeY56qS5KmA00FpclsawSo?= =?us-ascii?Q?bjfXQYnyVu67IQknsKPJvn3g8ZFRb3DKDBpt/87+w2ZQCE30uhNA81TlpjPN?= =?us-ascii?Q?OuVdOwoDiSPRqPYgRdP9pOQmBEElvfRLl27tWxCvyX8pF+iSyb2YPblJjpQY?= =?us-ascii?Q?E4EY6FSHS0BBC0ZH1DxfK89FldvQBU8LelasydHrg4vox2QiotEpZnWqQo72?= =?us-ascii?Q?cWkL52UW5oLRpT6qUOJhEhHQy0os2XU+c6iNG/lkY+72O5pcWUCYu2YmexHV?= =?us-ascii?Q?RgvbmfOLJjoHAkCAj9eU4SfWPYzQxdflbaDUGRm34eWgJ/E7ZfEJ0RkF1NUc?= =?us-ascii?Q?5WwzzzQipahcXjT5IKYOmuGBJdPzyr/mpMp1QhVc?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?YLWSoUs2F+TjUZ9vlm7/o5qTYL00NfPC7EIcgf0lJhB9bywpH/JoKBV3IssJ?= =?us-ascii?Q?WP5Hn7VtDYJlpajko/SsVKZVzs/0XlZaoET8w/Hao9RXfyqCtMjS8n1GJnIm?= =?us-ascii?Q?8vD15iC8ZoxK9GXfRptzv82puZjq6zPMP3lZlOTx0MM12sJgp4PtaxFiShTA?= =?us-ascii?Q?yPPj+aEnid9y/FBNwp1BMrbDstIeBsU0AP8i1h6eTibAKytVL/c9dQQ4V4a5?= =?us-ascii?Q?7pD8rFlReQk3Tq5YvAofAe8jCGDtdCjW9rGQs4guwb7BlCXqbKxq9UXF89si?= =?us-ascii?Q?HX4U2x5AX5qTGZPyUOPaQLzGwn+sQiyy0zzs5HuiITV4lNWEDymCUmgYBqY2?= =?us-ascii?Q?kKhSx4stWYtAf6eUNozZA9127tTrQUw/0uSr0VrzTFksLi3oUFW+nk3zIW0k?= =?us-ascii?Q?/1DVbPgVFxNGYRWiQLLT2vgptFGQxdD3iR+igqjOHLvaw+PeDmE6l42vYidp?= =?us-ascii?Q?5VMWoaEoJETr91CWiTuQ3ByvBYbR8rMjZf9bqZRVGL9XxbSgGUYzfKBniMbF?= =?us-ascii?Q?ivytFyaWOukn8IyixkI26qecaY0cyF1pKCumIGoO1+EfUGybyOvYhUVfpisb?= =?us-ascii?Q?xZG449B5s5b6PQbGu3nQtqBBh87UzOx1vfBgHWgrLzxTvUC+j6qWEoX4E5H8?= =?us-ascii?Q?4GS9rnJuFF5DxVwwYUv8PLfTMap24i6ZXoCp56Z6XYMEdwXQGXYjRwW46++/?= =?us-ascii?Q?/HL0Z4nA1GOcHvNiOI836I+rS6QzLs6RGcsGK3yK3u4TVCPMRS8LosIRI5j1?= =?us-ascii?Q?Psfe/yx29c/SlVKvkTaF3FSzIr8L+TGQSm1D9cI1Si8NcF9CpIl5CV9gMvVs?= =?us-ascii?Q?KZTHjZeurOfpJUvyv1s2DBW0XIOlzcLPOjLusnKHpzVv+97aB2lQCxhGTI2q?= =?us-ascii?Q?g/i++qYRoC8s/2cwG6gYsN/dSAGiLZYB0wQRwHvSuZcOc6kj7mBoKB6WDpv5?= =?us-ascii?Q?ObFD9MvU4Ua71KNhcP4CuV6Rbzv5H4MBCOwcMQfgS89ywpgMaNd6E6dJlCwp?= =?us-ascii?Q?G1F886oEcE6sl2n7VUtA7ZElvAWMv6dye5A6RIKrIoFV5Jp/R6WGpHOgU8DF?= =?us-ascii?Q?0Mvw/axvLFIpxoM4cfihGGRvjcm3cNFjKsV+fNre3xG3K0wJMY3bzy2OiFpW?= =?us-ascii?Q?kq75V6JUFVVzvITfLw95X/echc8gKfibkTcWanAMreL7SjEes4z+KjM=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 25fc5ff5-c6ed-4530-c68f-08db34036fd5 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:37.1043 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lFy+bbT+hkUU2yGypQNLEk/z1osGbg4aI0nNHM5E1Q1+V/ApZpOwebI0gzc9Zj7p6CDcnbTKWS1iPhvjSH61bzfhFrgiE+iSeUGwwvzo1qM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7106 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 malwarescore=0 adultscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: ajXacZ1gH2f6bdCwT9bIRPZ4J4mSVB7H X-Proofpoint-ORIG-GUID: ajXacZ1gH2f6bdCwT9bIRPZ4J4mSVB7H Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add clear_pages() and define the ancillary clear_user_pages(). Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/include/asm/page.h | 6 ++++++ arch/x86/include/asm/page_32.h | 6 ++++++ arch/x86/include/asm/page_64.h | 9 +++++++-- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index d18e5c332cb9..03e3c69fc427 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -28,6 +28,12 @@ static inline void clear_user_page(void *page, unsigned = long vaddr, clear_page(page); } =20 +static inline void clear_user_pages(void *page, unsigned long vaddr, + struct page *pg, unsigned int nsubpages) +{ + clear_pages(page, nsubpages); +} + static inline void copy_user_page(void *to, void *from, unsigned long vadd= r, struct page *topage) { diff --git a/arch/x86/include/asm/page_32.h b/arch/x86/include/asm/page_32.h index 580d71aca65a..3523d1150cfc 100644 --- a/arch/x86/include/asm/page_32.h +++ b/arch/x86/include/asm/page_32.h @@ -22,6 +22,12 @@ static inline void clear_page(void *page) memset(page, 0, PAGE_SIZE); } =20 +static inline void clear_pages(void *page, unsigned int nsubpages) +{ + for (int i =3D 0; i < nsubpages; i++) + clear_page(page + i * PAGE_SIZE); +} + static inline void copy_page(void *to, void *from) { memcpy(to, from, PAGE_SIZE); diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index 7ca3bd2448c1..42f6c45206c1 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -43,9 +43,9 @@ void clear_pages_orig(void *page, unsigned int length); void clear_pages_rep(void *page, unsigned int length); void clear_pages_erms(void *page, unsigned int length); =20 -static inline void clear_page(void *page) +static inline void clear_pages(void *page, unsigned int nsubpages) { - unsigned long length =3D PAGE_SIZE; + unsigned int length =3D nsubpages * PAGE_SIZE; /* * Clean up KMSAN metadata for the page being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. @@ -60,6 +60,11 @@ static inline void clear_page(void *page) : "cc", "memory", "rax", "rcx"); } =20 +static inline void clear_page(void *page) +{ + clear_pages(page, 1); +} + void copy_page(void *to, void *from); =20 #ifdef CONFIG_X86_5LEVEL --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91288C76196 for ; Mon, 3 Apr 2023 05:23:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231584AbjDCFXx (ORCPT ); Mon, 3 Apr 2023 01:23:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231527AbjDCFXe (ORCPT ); Mon, 3 Apr 2023 01:23:34 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F335BDF3 for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3330qGbo020222; Mon, 3 Apr 2023 05:22:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=j54SMO8aHZZZsgBi7r1t7d7dsJQz4KL14GWBankHCBc=; b=lppQNTdGZtE0ehbHr57NIjPQfjX3bwIRB/Y0grP86aC7vC6mTDe/8r5J/HiKIubZsiaK er7J1ohrWApuGNzCsHcCm+p2wIC9jAsy+Ipcih/ORquQ7dmNncTR1r9MQ0BAO6jT19Yk kTrPfLDvEwoiABFzDnh2ec1j8iagTtvpirk0VA8eO0+znd229SP31GlzZLGrgyiF36pA SME8AZFMRtq52+CgB8TYYt/VuiNJpiBtLuhAUgjj3I0r71nsO8ZKpO78InXHNBxggdmc mjVFWsF0U4Qlhl1FR15mUji7rWy6aJPYT59mNSqTrDKO9W/i+oHDvGlC+IeGxmXNX+mK VA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcnct4sa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:45 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332NM3i028114; Mon, 3 Apr 2023 05:22:44 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2106.outbound.protection.outlook.com [104.47.55.106]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt243jf8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HLx0Ec+40KiTSJJWUXk2QMZFsmqR1BwZ66ZexSTG1z+vBWlWGtzJ9Klhbi/mTiytvyOOvvReyiLgl9OPmsnFran9yjeTWVO2xjsQkx8x8SGFyfWCuz8Cm4DSJGx5alwRI1jOt7YjSTMInSZEKX2YluOuKGY5SA9nbjMkHEkcJtOp5BpbT6+QSkcyg1YJ6a6GS7nE07VpsJEwhYvmpWBzOh+TEAUZNwe71W6i3MBfA9PXmpxkbmyHvcm1jwF5TUcwZZqzd+hjA8FPGS/aRhwxOkLHHissxs4x7AR12/Y3Pl51O2c9e+qoePjWJNttCsEUAXtrx9HUZ8BFZmbVwC6sDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j54SMO8aHZZZsgBi7r1t7d7dsJQz4KL14GWBankHCBc=; b=FbVPuUSqf8qDw6A2V1Czu3Mk993iYLy/sO2KUwpSVAEaRHq38bB03fsnQMxLO8cP8PmfbSXcUb5IedaoOwP6aCeqrIx1cwm9e3O6krjRXrpIJHRjRGNUKwR4vETIReNaZg7CMjU6OueXRfLA4wx7ssR6wijmWc3FWq57cqBghdHR3U9PuP+DpqqbiZJZnlvsRa4zCoCJc2NFDPJitoA4Nvicb93XDemsVpEWMzPGRW9gV0MgfysxhlwUfIsnu+5SVX+kdLhJEAgkCHEwmQP7S+iAWDC/SC8Kyv95QNJeOGMwHOBZ6aqtWSd7EMfOJEuBsD+ZzBL55OYfp3pRXFi05w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j54SMO8aHZZZsgBi7r1t7d7dsJQz4KL14GWBankHCBc=; b=rsOWDWhCY3QbgShl49gkoaEKpfnDdPghQYRs6hg1QOE2tNKSKNifGD5K9mq78qxsJXh3IaGsm42DIfNA/YP3LcXqaMxl+SeFWbYBiBqojXbvpOq0bjVONt4oo2PINsxwN9MZDTPwscNunHaEcCRAttbma2evFksAWXu6hHLGp80= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:39 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:39 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 6/9] mm/clear_huge_page: use multi-page clearing Date: Sun, 2 Apr 2023 22:22:30 -0700 Message-Id: <20230403052233.1880567-7-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0315.namprd03.prod.outlook.com (2603:10b6:303:dd::20) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: 2abb81d4-735e-4473-77db-08db3403715f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oUltraz0xtWPZn3Uj5/EngPUTJ+kcPgAYtWU08DjJwocetxXw/OPferKiVhzJd8mOl9VLy1c1I55KK0+WcTM0ayPI4vMsv6IAEyq8PbSrSpzj19Xafahz0UQWziyt5l7mLeUrnjL7gGMZZkBM3bx0ZQsD3SqJcyr4aLpkV01fso/hA4RJrWPfUDhXFaCEf4tIYchu/z0Mpuv//pzhDlx4iIi+lL1jVTthamtqS+G0oLrSMKFmECuk+QfQpc7NG28fMv3sb5z4MVzmG1Gs/VVkvi5d5fuZPnBlUlyKjWod6Bp/ERAsiyYhvimBj8CvMx/dxGm0LlLu//SPn4dy9789PhYrjlA3dSFJPepQ+UvlvgSe3zL3jyvfRfwvatbiCEhjjo0xjjvRcXBQLwx4tdtpytvm6UVP7fYmSfuHwMEX4Kvt2NlSGv8+sKM+FPBdri4JwqoSrSOYWhoDPNULI5zj5ojVbTWyzvbIcxD4y5ts/Lzq5MSUC9u+lXgCDVKvOSHUEdI+z+jGOQd6YZV74YP2E0Q90CXeF11U8kwEzP2QadtryhUZhCPJkbfEGomI+f8GTansnfpMDpYjkguVVwnJ69BeFTNPAInqigCtIjRFe0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(30864003)(8936002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LDwZaoxviroa5oZKWcjLk4+gKaJkaTgn1Tv7iWksudtQqRwBOXztjUo6ZBsY?= =?us-ascii?Q?ygygzZLMbpeW7v0BZm9oM3CjDfo2VQzMPKO2P+Oca2nF5RU7W4jrvcgLzNAn?= =?us-ascii?Q?rWxbvzmulCvyAJUtiNSIL3EwH8hdru03fUFxhb199R6Z5KWOrU0+BZW3XbUG?= =?us-ascii?Q?isZvYsBd4waNZcrzKNq6SGaTLaKYjWFB2YTOVzQxHcgvUf4QLAjkYKWLu2Ke?= =?us-ascii?Q?2KE2219HPA1YxGgswDUHVPL+iqNpH5HPLkUfzGUUEPKgyl2b1k/9EppRYA9K?= =?us-ascii?Q?eOfb1UfnAp77+dl4ZrZ3+m5KNpqGz8TTl3BbUWvAxZHOMHuhdPtI9+cx+ruL?= =?us-ascii?Q?YcKewx7zb71lq7Mu7FN4JCfjomZIG1xuFEyk/938QyVCnOJAQYLr+dU+d/sO?= =?us-ascii?Q?qWcmX+ACIXgQDBCSvv5icmXODcTH+3GlMHjA2DqgLNG/C7VQ2HV6fmEmR45G?= =?us-ascii?Q?ljiKGctMAxW8G+C6WhEBWneoWcZfLp/lceRDNTilqpgfJ8lP1pXPFOskFsJE?= =?us-ascii?Q?3H5vQFv4mqpn43kQXNyBRRUbXkA/HNxF5m3DtRhuXqU4sFeWUH1zlz8RB4Ci?= =?us-ascii?Q?WvE1N3q6ZLfbB92N4CLw1k5AWZLZekidy73cHsNjySTyn57PCbEnR9sAJLNn?= =?us-ascii?Q?OSV0TwEYShihHJP255vNTPO+pO65Y5es5kjwMtVOaZi9mAUDF/Tuu5+w9At5?= =?us-ascii?Q?EQu846YDeKlBj6QxZmkezFr/SC3dKlp+81ovQA0z8+yyV/zj7lWwcZ9Qvutc?= =?us-ascii?Q?c/CkQOPPS5e3IwBnzWT/ehgCf5HWFgj1eIqTkvDzZHa2tX3yTaNTJ2H3CJy7?= =?us-ascii?Q?zms+YShFDm4i1NgWmfZty98kWs8kdbOvCpaivhSMgepE3mZAP1CiGF64rBWR?= =?us-ascii?Q?ZeUwZHYWD+GWXTsYXLv1RvXDJgATqEOfdcH2KKEh0EBpfaGdzLodNdiHvTUl?= =?us-ascii?Q?9iMItgj2ayB6QwFRNAF2Os9AjRbbJY3D8tZzVQeT7FBcWetQRz8MNtmEyiQP?= =?us-ascii?Q?ZbWm7n5NvVk4htMRSlZAT8GlQw9iEsEqp3Egg3Ce8Bg0HOT6wx2suTrOB/ZX?= =?us-ascii?Q?l4lOhN4DSOikKu012PW1i42wsNa3Gxr7U9aCDQXp6o6cFlN1UaTJ6XaEKJ0M?= =?us-ascii?Q?HrwWF9dj7NTVVGNbO8QktpIXMJYi6WryCOMOHydJhJBW+o4klpAyylVicmUA?= =?us-ascii?Q?v1usOu4dTOpS3W5A4dBaDSpingMzIZRsBYrFhR2VjtNBgg4OY83BzuqCWA8s?= =?us-ascii?Q?/FDwgL9qWZarYHFBtWJyGNKTq9b1cDjzaguKfkxOIp7PS6GHqDvOHQGGgyNe?= =?us-ascii?Q?C+SB5gHyKCMMFCSLKO1/l2NTtxYIbSpj/sQ6WOyhFpsTOWTgikcuAYXEMl3J?= =?us-ascii?Q?rdACXzE9KaEQ1w87IzA2wXb+3vvljBpBdSsZAqhjmUKWrwJ7HJKLixyLdhH8?= =?us-ascii?Q?UcnvyL/Q0q6MHtP5tBhpqSlrBjEiQHUFA2cxirpbGHcqtI0wYIuoJVmhszh9?= =?us-ascii?Q?COk+O9DdZ2iWlvYvKj9+bCWty6+7fihtqDo8pg6IEO9V0QRpLayLgrdM80j1?= =?us-ascii?Q?1ai7ArMrw2ld82qrvIY4PEomVxvUeMTa1U/ypICh?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?jC5u5HHWWcM+U3N7pE9x+2Vpo1XxWleztbM+IsfBIR/ncJHRueiqIggpefAN?= =?us-ascii?Q?nBAaM+E0I0ExyL2fcXtb7pVfDjwvX8r6MzYI09oCbCoeg/zDheswn5f1dHRM?= =?us-ascii?Q?0LNaxUoiMgtvQvMVnw6DF4Dt9fvATXmOq8QdBMjfVDYsrsQYl+txcqQmJCpM?= =?us-ascii?Q?TwxYT0+ZaD6P3iCSyA7vKKllzgumCTQKqAGn+dXDXUgaWn2zFN+L7FYUfke6?= =?us-ascii?Q?/NhVxH+pkpXsYToQbED9RB9j49Rz1gORr25/5ec9u2iLhTCjyvmWqay6DuNR?= =?us-ascii?Q?T15NhwOJaOltskdE4Sptn+BIfONNqQ53sNsd1smJ0jhxl8+DAgFtIg4IkE2l?= =?us-ascii?Q?8ROgznedsbqtZSWsqNGZ3gFvYLg9TaPhXJLSpxOqREkLP5exKWm3888r0qD0?= =?us-ascii?Q?Vu3lbKzF5REQKnNvAm7T/oEfwuObt/ltZLheiui6HT10q8SoHgc+5XzEi5r7?= =?us-ascii?Q?iZE6EZqFKd6zNpXHjSILMwSnFkyxjAQEwZ2UFvZWDlA9zCSK5+pRaQX36Bpl?= =?us-ascii?Q?OQLqIQzQJCA2+UA+01bCiTV7e3Afsdu0OyuGAKsGrgTXjjfhmWZZ1WTd8As1?= =?us-ascii?Q?86F9ar5yZ4f8RqDE5quMhuw7fuOhVBc8N0Za8Yi19PNdzsOiXVAqQ5OHF/qp?= =?us-ascii?Q?JnprzKbYzK1VdCgh5dxujI+0n9TeOYpZaoIGwvh4Gq7M8QWXJeXBd+9uBfh3?= =?us-ascii?Q?3aHBEoNz/wksaLqfjKsDl+ZY44Dladv3P4ag7ifiCrpV7Uo4lv4EvnffyaTu?= =?us-ascii?Q?nT4xi8gFVsAgvB9TzXQeR/w0Ln7QD5f6vYx49GflwMumf4B8ZqYYMYex15Nj?= =?us-ascii?Q?rs9V1XwErxNkpeeYzEmifaywlpGu9YZaNRRy4WQP/h2DuojZIWOmrnr/NBa9?= =?us-ascii?Q?puV3ImDG2nVQ07kb1P7D2Be5K+ckmiZlIf3QCfpf4WcE1iqHd0MIMyJ69+n0?= =?us-ascii?Q?KSAQf0HKW14xrbLwidIYIq+HyUtSJlMjr+a/SwetyznwByKFmxgYcFdTi1QD?= =?us-ascii?Q?EsYczsOd569b3oPfjZJtmpK8fbHUt0vfSGF5IMnNnHmcOrCoLAjhNuc62++z?= =?us-ascii?Q?r4480ve3vzSOLtKGfU5w7RfakQv/WMiLmE0Yw6MZafPGMCsI8h/zeNBZO95r?= =?us-ascii?Q?pEvLjZrzYWhYtpTjfEOWeMDSGWNXukl1L2G/wvTeW85leIpkhn2A07Q=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2abb81d4-735e-4473-77db-08db3403715f X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:39.7274 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2MdWZq9BlrhY8Ydt9i6R0oplwyjsNqivR+6F0aE66sPzJMWuB23Eqm7AdmO16PfdzWUtDeCbR2o1BUC6B+Hr0dFbh1MJm3y04n2f/uhib4U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 malwarescore=0 adultscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: OzYmjE-GglT-w3DRsHWC62p0xNTxrbmF X-Proofpoint-ORIG-GUID: OzYmjE-GglT-w3DRsHWC62p0xNTxrbmF Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear_pages_rep(), clear_pages_erms() use string instructions internally. These, unlike a MOV loop, allow us to explicitly advertise the region-size to the processor. Thus, clearing in multi-page chunks means we can specify the real region sizes (or close to it) which is good for two reasons: - region-size can serve as a hint to current (some AMD Zen models) and possibly future uarchs which can use this hint to avoid polluting one or more levels of the dcache. - string instructions are typically microcoded, and would be cheaper if amortized across larger regions. We also execute fewer loop iterations (ex. a cond_resched() check for each page but those instructions are likely free.) clear_huge_page() now clears in three sections: the local neighbourhood of the faulting address (faulting page, and four surrounding pages), and its left and right regions. The local neighbourhood is cleared last to keep its cachelines hot. Performance =3D=3D Use mmap(MAP_HUGETLB) to demand fault a 64GB region (on the local NUMA node): Icelakex (Platinum 8358, ucode=3D0xd0002c1, no_turbo=3D1): mm/clear_huge_page x86/clear_huge_page change =20 (GB/s) (GB/s) =20 =20 pg-sz=3D2MB 8.76 11.82 +34.93% =20 pg-sz=3D1GB 8.99 12.18 +35.48% =20 On Icelakex we continue to allocate cachelines: pg-sz=3D2MB: - 701,951,397 L1-dcache-loads # 47.985 M/sec = ( +- 19.22% ) (69.23%) - 3,239,403,770 L1-dcache-load-misses # 691.17% of all L1-dca= che accesses ( +- 19.25% ) (69.24%) + 194,318,641 L1-dcache-loads # 17.905 M/sec = ( +- 19.07% ) (69.25%) + 3,238,878,229 L1-dcache-load-misses # 2480.93% of all L1-dca= che accesses ( +- 19.25% ) (69.26%) pg-sz=3D1GB: - 532,232,051 L1-dcache-loads # 37.378 M/sec = ( +- 19.25% ) (69.23%) - 3,224,574,249 L1-dcache-load-misses # 909.02% of all L1-dca= che accesses ( +- 19.25% ) (69.24%) + 22,587,703 L1-dcache-loads # 2.150 M/sec = ( +- 19.38% ) (69.25%) + 3,223,143,697 L1-dcache-load-misses # 21478.37% of all L1-dc= ache accesses ( +- 19.25% ) (69.25%) Milan (EPYC 7J13, ucode=3D0xa0011a9, boost=3D0): mm/clear_huge_page x86/clear_huge_page change =20 (GB/s) (GB/s) =20 =20 pg-sz=3D2MB 12.24 17.54 +43.30% =20 pg-sz=3D1GB 17.98 37.24 +107.11% =20 Milan uses a threshold ~32MB for eliding cacheline allocation, so we see a dropoff in cacheline-allocations for pg-sz=3D1GB: pg-sz=3D2MB: - 2,495,566,569 L1-dcache-loads # 476.417 M/sec = ( +- 0.04% ) (33.38%) - 1,079,711,798 L1-dcache-load-misses # 43.28% of all L1-dca= che accesses ( +- 0.01% ) (33.37%) + 2,235,310,058 L1-dcache-loads # 610.770 M/sec = ( +- 0.02% ) (33.37%) + 1,089,602,355 L1-dcache-load-misses # 48.73% of all L1-dca= che accesses ( +- 0.01% ) (33.37%) pg-sz=3D1GB: - 2,417,846,489 L1-dcache-loads # 679.753 M/sec = ( +- 0.01% ) (33.38%) - 1,075,531,869 L1-dcache-load-misses # 44.49% of all L1-dca= che accesses ( +- 0.01% ) (33.35%) + 31,159,378 L1-dcache-loads # 18.119 M/sec = ( +- 3.27% ) (33.46%) + 14,692,358 L1-dcache-load-misses # 48.21% of all L1-dca= che accesses ( +- 3.12% ) (33.46%) Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- Fuller perf stats for context: # Icelakex, baseline (mm/clear_huge_page), region-sz=3D64g, pg-sz=3D2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge= 1' (3 runs): 21,945.59 msec task-clock # 2.999 CPUs ut= ilized ( +- 19.25% ) 34 context-switches # 2.324 /sec = ( +- 20.38% ) 3 cpu-migrations # 0.205 /sec = ( +- 19.25% ) 198,152 page-faults # 13.546 K/sec = ( +- 19.29% ) 56,513,364,885 cycles # 3.863 GHz = ( +- 19.25% ) (38.44%) 2,583,719,806 instructions # 0.07 insn pe= r cycle ( +- 19.24% ) (46.14%) 585,212,952 branches # 40.005 M/sec = ( +- 19.23% ) (53.83%) 562,164 branch-misses # 0.14% of all = branches ( +- 19.23% ) (61.53%) 282,621,312,162 slots # 19.320 G/sec = ( +- 19.25% ) (69.22%) 11,048,627,225 topdown-retiring # 3.8% Retiri= ng ( +- 19.22% ) (69.22%) 34,358,400,894 topdown-bad-spec # 11.5% Bad Sp= eculation ( +- 19.57% ) (69.22%) 2,231,092,499 topdown-fe-bound # 0.8% Fronte= nd Bound ( +- 19.25% ) (69.22%) 246,679,210,776 topdown-be-bound # 84.0% Backen= d Bound ( +- 19.21% ) (69.22%) 701,951,397 L1-dcache-loads # 47.985 M/sec = ( +- 19.22% ) (69.23%) 3,239,403,770 L1-dcache-load-misses # 691.17% of all = L1-dcache accesses ( +- 19.25% ) (69.24%) 11,475,685 LLC-loads # 784.475 K/sec = ( +- 19.23% ) (69.25%) 793,272 LLC-load-misses # 10.36% of all = LL-cache accesses ( +- 19.23% ) (69.25%) 17,821,045 L1-icache-load-misses # 0.00% of all = L1-icache accesses ( +- 19.51% ) (30.77%) 693,339,354 dTLB-loads # 47.397 M/sec = ( +- 19.33% ) (30.76%) 637,811 dTLB-load-misses # 0.14% of all = dTLB cache accesses ( +- 19.09% ) (30.75%) 131,922 iTLB-load-misses # 0.00% of all = iTLB cache accesses ( +- 19.59% ) (30.75%) 7.31681 +- 0.00177 seconds time elapsed ( +- 0.02% ) # Icelakex, multi-page (x86/clear_huge_page), region-sz=3D64g, pg-sz=3D2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge= 1' (3 runs): 16,276.28 msec task-clock # 2.999 CPUs ut= ilized ( +- 19.24% ) 27 context-switches # 2.488 /sec = ( +- 19.25% ) 3 cpu-migrations # 0.276 /sec = ( +- 19.25% ) 196,935 page-faults # 18.146 K/sec = ( +- 19.25% ) 41,906,597,608 cycles # 3.861 GHz = ( +- 19.24% ) (38.44%) 729,479,932 instructions # 0.03 insn pe= r cycle ( +- 19.38% ) (46.14%) 133,969,095 branches # 12.344 M/sec = ( +- 19.35% ) (53.84%) 412,818 branch-misses # 0.46% of all = branches ( +- 18.97% ) (61.54%) 209,574,316,961 slots # 19.311 G/sec = ( +- 19.24% ) (69.24%) 4,933,512,982 topdown-retiring # 2.3% Retiri= ng ( +- 19.24% ) (69.24%) 20,272,641,267 topdown-bad-spec # 9.4% Bad Sp= eculation ( +- 19.51% ) (69.24%) 837,421,487 topdown-fe-bound # 0.4% Fronte= nd Bound ( +- 19.24% ) (69.24%) 190,089,232,476 topdown-be-bound # 88.0% Backen= d Bound ( +- 19.19% ) (69.24%) 194,318,641 L1-dcache-loads # 17.905 M/sec = ( +- 19.07% ) (69.25%) 3,238,878,229 L1-dcache-load-misses # 2480.93% of all = L1-dcache accesses ( +- 19.25% ) (69.26%) 10,560,508 LLC-loads # 973.081 K/sec = ( +- 19.23% ) (69.26%) 724,884 LLC-load-misses # 10.28% of all = LL-cache accesses ( +- 17.15% ) (69.26%) 14,378,070 L1-icache-load-misses # 0.00% of all = L1-icache accesses ( +- 19.13% ) (30.75%) 185,562,230 dTLB-loads # 17.098 M/sec = ( +- 19.74% ) (30.74%) 617,978 dTLB-load-misses # 0.51% of all = dTLB cache accesses ( +- 18.72% ) (30.74%) 112,509 iTLB-load-misses # 0.00% of all = iTLB cache accesses ( +- 19.76% ) (30.74%) 5.42697 +- 0.00152 seconds time elapsed ( +- 0.03% ) # Icelakex, baseline (mm/clear_huge_page), region-sz=3D64g, pg-sz=3D1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64 --huge = 2' (3 runs): 21,361.22 msec task-clock # 2.999 CPUs ut= ilized ( +- 19.25% ) 23 context-switches # 1.615 /sec = ( +- 18.95% ) 3 cpu-migrations # 0.211 /sec = ( +- 19.25% ) 701 page-faults # 49.230 /sec = ( +- 19.27% ) 54,981,958,487 cycles # 3.861 GHz = ( +- 19.25% ) (38.44%) 2,012,625,953 instructions # 0.05 insn pe= r cycle ( +- 19.25% ) (46.14%) 470,264,509 branches # 33.026 M/sec = ( +- 19.25% ) (53.83%) 194,801 branch-misses # 0.06% of all = branches ( +- 18.88% ) (61.53%) 274,966,507,627 slots # 19.311 G/sec = ( +- 19.25% ) (69.22%) 10,555,137,650 topdown-retiring # 3.8% Retiri= ng ( +- 19.04% ) (69.22%) 21,206,785,918 topdown-bad-spec # 7.8% Bad Sp= eculation ( +- 18.13% ) (69.22%) 1,094,597,329 topdown-fe-bound # 0.4% Fronte= nd Bound ( +- 19.25% ) (69.22%) 244,462,123,545 topdown-be-bound # 88.0% Backen= d Bound ( +- 19.33% ) (69.22%) 532,232,051 L1-dcache-loads # 37.378 M/sec = ( +- 19.25% ) (69.23%) 3,224,574,249 L1-dcache-load-misses # 909.02% of all = L1-dcache accesses ( +- 19.25% ) (69.24%) 2,318,195 LLC-loads # 162.804 K/sec = ( +- 19.35% ) (69.25%) 206,737 LLC-load-misses # 13.44% of all = LL-cache accesses ( +- 18.30% ) (69.25%) 4,950,866 L1-icache-load-misses # 0.00% of all = L1-icache accesses ( +- 19.26% ) (30.77%) 531,299,560 dTLB-loads # 37.313 M/sec = ( +- 19.24% ) (30.76%) 2,811 dTLB-load-misses # 0.00% of all = dTLB cache accesses ( +- 17.25% ) (30.75%) 26,355 iTLB-load-misses # 0.00% of all = iTLB cache accesses ( +- 19.58% ) (30.75%) 7.12187 +- 0.00190 seconds time elapsed ( +- 0.03% ) # Icelakex, multi-page (x86/clear_huge_page), region-sz=3D64g, pg-sz=3D1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64 --huge = 2' (3 runs): 15,764.52 msec task-clock # 2.999 CPUs ut= ilized ( +- 19.25% ) 17 context-switches # 1.618 /sec = ( +- 20.47% ) 3 cpu-migrations # 0.285 /sec = ( +- 19.25% ) 700 page-faults # 66.614 /sec = ( +- 19.22% ) 40,560,984,582 cycles # 3.860 GHz = ( +- 19.25% ) (38.45%) 79,578,792 instructions # 0.00 insn pe= r cycle ( +- 19.24% ) (46.15%) 13,872,134 branches # 1.320 M/sec = ( +- 19.23% ) (53.85%) 119,492 branch-misses # 1.29% of all = branches ( +- 18.80% ) (61.55%) 202,854,573,160 slots # 19.304 G/sec = ( +- 19.25% ) (69.25%) 3,982,417,725 topdown-retiring # 2.0% Retiri= ng ( +- 19.25% ) (69.25%) 13,523,424,635 topdown-bad-spec # 6.8% Bad Sp= eculation ( +- 18.69% ) (69.25%) 18,661,431 topdown-fe-bound # 0.0% Fronte= nd Bound ( +- 19.28% ) (69.25%) 185,884,147,789 topdown-be-bound # 91.3% Backen= d Bound ( +- 19.28% ) (69.25%) 22,587,703 L1-dcache-loads # 2.150 M/sec = ( +- 19.38% ) (69.25%) 3,223,143,697 L1-dcache-load-misses # 21478.37% of all= L1-dcache accesses ( +- 19.25% ) (69.25%) 1,777,675 LLC-loads # 169.169 K/sec = ( +- 19.60% ) (69.25%) 126,583 LLC-load-misses # 10.77% of all = LL-cache accesses ( +- 19.82% ) (69.25%) 3,333,729 L1-icache-load-misses # 0.00% of all = L1-icache accesses ( +- 19.49% ) (30.75%) 19,999,517 dTLB-loads # 1.903 M/sec = ( +- 19.38% ) (30.75%) 1,833 dTLB-load-misses # 0.01% of all = dTLB cache accesses ( +- 17.72% ) (30.75%) 34,066 iTLB-load-misses # 0.00% of all = iTLB cache accesses ( +- 19.09% ) (30.75%) 5.25624 +- 0.00176 seconds time elapsed ( +- 0.03% ) # Milan, baseline (mm/clear_huge_page), region-sz=3D64g, pg-sz=3D2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge= 1' (3 runs): 5,241.76 msec task-clock # 1.000 CPUs utilized = ( +- 0.08% ) 10 context-switches # 1.909 /sec = ( +- 8.82% ) 1 cpu-migrations # 0.191 /sec = =20 65,636 page-faults # 12.530 K/sec = ( +- 0.00% ) 12,730,694,768 cycles # 2.430 GHz = ( +- 0.08% ) (33.31%) 36,709,243 stalled-cycles-frontend # 0.29% frontend cycle= s idle ( +- 24.07% ) (33.32%) 37,520,225 stalled-cycles-backend # 0.29% backend cycles= idle ( +- 9.87% ) (33.34%) 874,896,010 instructions # 0.07 insn per cycle= =20 # 0.05 stalled cycles= per insn ( +- 1.23% ) (33.36%) 199,308,386 branches # 38.049 M/sec = ( +- 0.84% ) (33.38%) 441,428 branch-misses # 0.22% of all branche= s ( +- 4.68% ) (33.38%) 2,495,566,569 L1-dcache-loads # 476.417 M/sec = ( +- 0.04% ) (33.38%) 1,079,711,798 L1-dcache-load-misses # 43.28% of all L1-dcac= he accesses ( +- 0.01% ) (33.37%) 50,936,391 L1-icache-loads # 9.724 M/sec = ( +- 1.29% ) (33.35%) 284,407 L1-icache-load-misses # 0.56% of all L1-icac= he accesses ( +- 4.60% ) (33.33%) 546,596 dTLB-loads # 104.348 K/sec = ( +- 6.14% ) (33.31%) 231,897 dTLB-load-misses # 42.08% of all dTLB ca= che accesses ( +- 4.27% ) (33.29%) 6 iTLB-loads # 1.145 /sec = ( +- 72.65% ) (33.29%) 34,065 iTLB-load-misses # 262038.46% of all iTLB = cache accesses ( +- 44.88% ) (33.29%) 18,237,487 L1-dcache-prefetches # 3.482 M/sec = ( +- 12.84% ) (33.29%) 5.23915 +- 0.00421 seconds time elapsed ( +- 0.08% ) # Milan, multi-page (x86/clear_huge_page), region-sz=3D64g, pg-sz=3D2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge= 1' (3 runs): 3,655.71 msec task-clock # 0.999 CPUs utilized = ( +- 0.13% ) 7 context-switches # 1.913 /sec = ( +- 8.25% ) 1 cpu-migrations # 0.273 /sec = =20 65,636 page-faults # 17.934 K/sec = ( +- 0.00% ) 8,879,727,514 cycles # 2.426 GHz = ( +- 0.13% ) (33.26%) 5,733,380 stalled-cycles-frontend # 0.06% frontend cycle= s idle ( +-170.04% ) (33.28%) 42,012,302 stalled-cycles-backend # 0.47% backend cycles= idle ( +- 24.51% ) (33.31%) 214,672,610 instructions # 0.02 insn per cycle= =20 # 0.28 stalled cycles= per insn ( +- 1.71% ) (33.34%) 42,298,268 branches # 11.557 M/sec = ( +- 1.28% ) (33.36%) 267,936 branch-misses # 0.62% of all branche= s ( +- 7.80% ) (33.37%) 2,235,310,058 L1-dcache-loads # 610.770 M/sec = ( +- 0.02% ) (33.37%) 1,089,602,355 L1-dcache-load-misses # 48.73% of all L1-dcac= he accesses ( +- 0.01% ) (33.37%) 48,725,812 L1-icache-loads # 13.314 M/sec = ( +- 0.25% ) (33.37%) 231,227 L1-icache-load-misses # 0.47% of all L1-icac= he accesses ( +- 13.20% ) (33.37%) 280,655 dTLB-loads # 76.685 K/sec = ( +- 13.33% ) (33.37%) 151,028 dTLB-load-misses # 44.02% of all dTLB ca= che accesses ( +- 6.64% ) (33.35%) 15 iTLB-loads # 4.099 /sec = ( +- 6.67% ) (33.32%) 121,208 iTLB-load-misses # 865771.43% of all iTLB = cache accesses ( +- 2.74% ) (33.29%) 18,702,209 L1-dcache-prefetches # 5.110 M/sec = ( +- 12.51% ) (33.27%) 3.66065 +- 0.00461 seconds time elapsed ( +- 0.13% ) # Milan, baseline (mm/clear_huge_page), region-sz=3D64g, pg-sz=3D1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge= 2' (3 runs): 3,544.20 msec task-clock # 0.996 CPUs utilized = ( +- 0.21% ) 5 context-switches # 1.406 /sec = ( +- 6.67% ) 1 cpu-migrations # 0.281 /sec = =20 227 page-faults # 63.819 /sec = ( +- 0.15% ) 8,609,810,964 cycles # 2.421 GHz = ( +- 0.21% ) (33.30%) 77,420,424 stalled-cycles-frontend # 0.90% frontend cycle= s idle ( +- 20.55% ) (33.33%) 25,197,541 stalled-cycles-backend # 0.29% backend cycles= idle ( +- 1.09% ) (33.35%) 658,146,061 instructions # 0.08 insn per cycle= =20 # 0.16 stalled cycles= per insn ( +- 0.04% ) (33.38%) 154,867,131 branches # 43.539 M/sec = ( +- 0.04% ) (33.41%) 167,531 branch-misses # 0.11% of all branche= s ( +- 5.19% ) (33.41%) 2,417,846,489 L1-dcache-loads # 679.753 M/sec = ( +- 0.01% ) (33.38%) 1,075,531,869 L1-dcache-load-misses # 44.49% of all L1-dcac= he accesses ( +- 0.01% ) (33.35%) 12,835,321 L1-icache-loads # 3.609 M/sec = ( +- 0.41% ) (33.33%) 55,282 L1-icache-load-misses # 0.43% of all L1-icac= he accesses ( +- 1.98% ) (33.30%) 23,287 dTLB-loads # 6.547 K/sec = ( +- 15.61% ) (33.29%) 1,333 dTLB-load-misses # 4.48% of all dTLB ca= che accesses ( +- 1.26% ) (33.29%) 3 iTLB-loads # 0.843 /sec = ( +- 33.33% ) (33.29%) 231 iTLB-load-misses # 11550.00% of all iTLB c= ache accesses ( +- 6.14% ) (33.29%) 170,608,062 L1-dcache-prefetches # 47.965 M/sec = ( +- 0.84% ) (33.29%) 3.55776 +- 0.00738 seconds time elapsed ( +- 0.21% ) # Milan, multi-page (x86/clear_huge_page), region-sz=3D64g, pg-sz=3D1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge= 2' (3 runs): 1,718.27 msec task-clock # 0.999 CPUs utilized = ( +- 0.08% ) 6 context-switches # 3.489 /sec = ( +- 14.70% ) 1 cpu-migrations # 0.581 /sec = =20 227 page-faults # 132.000 /sec = ( +- 0.15% ) 4,176,107,493 cycles # 2.428 GHz = ( +- 0.08% ) (33.19%) 2,675,797 stalled-cycles-frontend # 0.06% frontend cycle= s idle ( +- 0.34% ) (33.25%) 147,394,527 stalled-cycles-backend # 3.53% backend cycles= idle ( +- 8.80% ) (33.31%) 12,779,784 instructions # 0.00 insn per cycle= =20 # 13.14 stalled cycles= per insn ( +- 0.09% ) (33.37%) 2,428,829 branches # 1.412 M/sec = ( +- 0.08% ) (33.42%) 63,460 branch-misses # 2.61% of all branche= s ( +- 3.48% ) (33.46%) 31,159,378 L1-dcache-loads # 18.119 M/sec = ( +- 3.27% ) (33.46%) 14,692,358 L1-dcache-load-misses # 48.21% of all L1-dcac= he accesses ( +- 3.12% ) (33.46%) 2,556,688 L1-icache-loads # 1.487 M/sec = ( +- 0.89% ) (33.46%) 21,148 L1-icache-load-misses # 0.84% of all L1-icac= he accesses ( +- 0.25% ) (33.41%) 6,114 dTLB-loads # 3.555 K/sec = ( +- 12.76% ) (33.35%) 1,742 dTLB-load-misses # 33.73% of all dTLB ca= che accesses ( +- 21.79% ) (33.29%) 45 iTLB-loads # 26.167 /sec = ( +- 7.52% ) (33.23%) 90 iTLB-load-misses # 210.94% of all iTLB ca= che accesses ( +- 21.20% ) (33.17%) 257,942 L1-dcache-prefetches # 149.993 K/sec = ( +- 9.84% ) (33.17%) 1.72042 +- 0.00139 seconds time elapsed ( +- 0.08% ) --- arch/x86/mm/hugetlbpage.c | 49 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 5804bbae4f01..4294b77c4f18 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -148,6 +148,55 @@ hugetlb_get_unmapped_area(struct file *file, unsigned = long addr, return hugetlb_get_unmapped_area_topdown(file, addr, len, pgoff, flags); } + +/* + * This is used on all !CONFIG_HIGHMEM configurations. + * + * CONFIG_HIGHMEM, falls back to the __weak version. + */ +#ifndef CONFIG_HIGHMEM +static void clear_contig_region(struct page *page, unsigned long vaddr, + unsigned int npages) +{ + clear_user_pages(page_address(page), vaddr, page, npages); +} + +void clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) +{ + unsigned long addr =3D addr_hint & + ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + const long pgidx =3D (addr_hint - addr) / PAGE_SIZE; + const int first_pg =3D 0, last_pg =3D pages_per_huge_page - 1; + const int width =3D 2; /* pages cleared last on either side */ + int sidx[3], eidx[3]; + int i, n; + + if (pages_per_huge_page > MAX_ORDER_NR_PAGES) + return clear_contig_region(page, addr, pages_per_huge_page); + + /* + * Neighbourhood of the fault. Cleared at the end to ensure + * it sticks around in the cache. + */ + n =3D 2; + sidx[n] =3D (pgidx - width) < first_pg ? first_pg : (pgidx - width); + eidx[n] =3D (pgidx + width) > last_pg ? last_pg : (pgidx + width); + + sidx[0] =3D first_pg; /* Region to the left of the fault */ + eidx[0] =3D sidx[n] - 1; + + sidx[1] =3D eidx[n] + 1; /* Region to the right of the fault */ + eidx[1] =3D last_pg; + + for (i =3D 0; i <=3D 2; i++) { + if (eidx[i] >=3D sidx[i]) + clear_contig_region(page + sidx[i], + addr + sidx[i] * PAGE_SIZE, + eidx[i] - sidx[i] + 1); + } +} +#endif /* CONFIG_HIGHMEM */ #endif /* CONFIG_HUGETLB_PAGE */ =20 #ifdef CONFIG_X86_64 --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7869BC76196 for ; Mon, 3 Apr 2023 05:24:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231602AbjDCFYJ (ORCPT ); Mon, 3 Apr 2023 01:24:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231536AbjDCFXh (ORCPT ); Mon, 3 Apr 2023 01:23:37 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9D893A99 for ; Sun, 2 Apr 2023 22:23:18 -0700 (PDT) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332NIh1B013542; Mon, 3 Apr 2023 05:22:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=AWtO26WrUxcC8Dr9AfFmU8r0+3TZoPF8bVvYSgph62Q=; b=QzWiclfTa84OJnoMiqC/7ngDJfU7OilahzGorX6/iDroF+VOt9o5E7YvMa0voGN9GMGi L/T5pVXsk7jDRfAs7svSsRk7spEwhTQFau+963l3607okU487LO2MlMNcIqeid/QU+z1 JIAipmHfsYHs8URW3bpfWRiu2GNX8IT78Gb2nICU7cJLQBTPET8ijOcuhWy89YwgmqiJ GaelT/lv+yXyj+22lS1X1citlyXcHod7ULtxhQgX+WrIQVy6d75z3/GB/mMeHtkmX43g vGQd0ImiCp3A6cAsFjE7cP5HpE5c8fnv8GK7sJjkIv+N2ALnV8OES7PI1C20nQaIC4+V xQ== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcgaj5a1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:48 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332i7pt013932; Mon, 3 Apr 2023 05:22:47 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2107.outbound.protection.outlook.com [104.47.55.107]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt3djuqb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VMfTjRopugbMrhSsq/HHWJBSJZik/77H9g6QRhukxMC6LFF5yXoM6dkaDhJBX+3CkJtQ7TrzImxT6D6QeN1lPrni9zrpdXN+jbevC8kAg0jmWgw7k2X5IWjf3jnQnTGr/51Ap6iqVgEhDNnbbnhJyoSknL4UXc4UpXKOkCVe4b/eTpDtVBvqU63bhDeb9YNqMz6w5d/jcwV0vtRjlMMfnG8cKbhmmhXGQidA8vixZZJZEa+pmqBxMfWGW4H1gw0CFcJysCGvFy9E+CivKfp92KBsNZuRLVVhvS7wZKKWEJnc+EZHSgCDzvv46S5HpDPdcYxcWV/IkEyc6hqLYZ5+/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AWtO26WrUxcC8Dr9AfFmU8r0+3TZoPF8bVvYSgph62Q=; b=fpG7HBxAZfSNIkJ23oXjW8pfBDsAos6ZuomHiV3sgcsGHdvuPyqbuiVgNfFloS7Hyv34uLNInFhbYMVxYz4+DNEG4rSQrG51yPpjg20NH/6QGETJOVlL5FRIZb0kc8JFIHX78IvBNN1EpGYhq5SNyZ/sxo7GvHo/10Vn2M2S43qgsjiVc1TyCUo9XSjq9PQQzcjW6u6KC/f6PU+AGi2OB065AWblwpSmYGvpxazSFyCaJT21oCR60Bst8kWSzOSH1kTw00NVdlFAn2IiRqbIYLYTUMPX1birWvpyxi4GmEW8wG4Hsij9Nu7tto9t3bdeezeetfr8BTjIVXCqXRFyTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AWtO26WrUxcC8Dr9AfFmU8r0+3TZoPF8bVvYSgph62Q=; b=pc5h1iDO5I63MTP/JVPka1qQH1fKok4RVjVA60P+OIRd7FyTXOAlx5KDtGE1ooBdQzg880bOxouff4eRuD5IB2IdLVINdht+W/suw6OhBNdx8cckUIppCxX4ntmfeGyZJqaZR3WgeIIi9FZJ2QU13TuIJybzKbjvzEkh3dGyiQg= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:45 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:45 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 7/9] sched: define TIF_ALLOW_RESCHED Date: Sun, 2 Apr 2023 22:22:31 -0700 Message-Id: <20230403052233.1880567-8-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0166.namprd03.prod.outlook.com (2603:10b6:303:8d::21) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: 0187daa4-2b8a-47ea-8b48-08db340372f9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: n+2pEnKRx9o049wD/LZjcRBZ5dXhBClqW6Shrj5mkMgFH/k2q6ngnOXQKb7FtFuMSSNO1In8SUP693EX4t+kl/1xHdXBvGqkGrsZmlgnzA0zSm6TGbMHf/S6/SDxYHqrwoMkVwY/wx8thwDWtg1A7YZlMg818SO1ttX44/7KYfMS2YeM4uaCO+wPSVWsFegPV1rLysE4oDM4Qwsqh3qMyVZDuhy7CT3wWyRD1Hur6hb+LGo3VlZIW9ybe7GVaIs6SWXhBGr3mH32eQRDGCtQdcTbzzOJvxXFOitwQmeezuvQGoEIbOKw7QbghRm1VWU7X2vO2NPlxkDJrQCRezyjslVb/SGrxwjiTdxNsVHL2rpaovryQROCol8+K3ixyUuH1SQJh2omXaSqlyCLQSs+PY3cVvoPctamXrNZER+prq36KY2TTZhDeDnQ8WfLRZOL1QZTvb+E8Bod58GvDNhgXXSeeqsbPjAOaFNWaJQmL3qhIuK/YbgJFxoNvwfi/mrtilrhsfoP3z3bsr3eQcXxjRJnFZ4PnoL1CDLT/xeT5x1Kdje/mJHkQ9za4OHGZ8Pd X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ahJtsEbR0Pu035xSPsiACxf6wPqKzntHhxV8lwYFiKOJ5IXr7E3AwfLMJCJ+?= =?us-ascii?Q?kD5ILdKgds1RrLPz4pca2iteRIzAxmCKACDLf7bsLn74fffJqKInGBId2mmg?= =?us-ascii?Q?cXwhOK4cg6dvJ8Q1Z2Thd3x99lymP1mu3U+vRuwuzqysWUQJY8L0CFDpG9zL?= =?us-ascii?Q?MnvtHphNvXhu6lc75UmDaGBDo/h7FQeWv0OKKFp5NemLeo55/2Uhv128LExD?= =?us-ascii?Q?WyyVlKAS0JX+4DadL/3g8/ultQ5Nr80NO/I6E31N5i45QBC8KaI52Kc73Z4z?= =?us-ascii?Q?xjfyjjt5aPOS9WXusKpCiYNJmmFeZ5EXeBFARu5MNMWeKjsEpp6rnctQjsKi?= =?us-ascii?Q?xYeOSbCzsjivtv/FkcP4gf5w80BuKzGqjtL98xfTSGS1U9LXwLtq1zbnf0Cs?= =?us-ascii?Q?ivCtONCG5ckAsNtqv5dG03yJ0mZhAh2ctgUZaPTyQqqRtQNA5jhNexXOnrNS?= =?us-ascii?Q?GRx8WHx5f1930m0u0npTOAc6WnflCeCQk7pKsTZEbbzI7+v3L0sGFq/T6cOj?= =?us-ascii?Q?5MbOfRw5O567vSGWkerDpJ0VTxFYNbUEgwgr5LNnPNAVMRfZoOjA9vxwRox9?= =?us-ascii?Q?JSIFUpAaKXn1O7m+zrnuDjmU7Gj0hHs6aE7L4cYm6QHqNxaBYrz7B+aIppR/?= =?us-ascii?Q?T2nMmL/f9ZZ2ZPnm2V57d1WZLziQlqvM1izLb1pcJXGr/1YhPSWDJ9Hn6e1V?= =?us-ascii?Q?AlRua4e23/umZc9IO5WkQ+Ib0h9o13UEcqxyM28eP/q0eIGfCHK9zLR1WGsN?= =?us-ascii?Q?iqZMs0/Llb/oMoqv4UVt1CN75XS5lbbuhC+CjdgHlYvwcQ7bff7WuIBOYc48?= =?us-ascii?Q?PNJNVMt6Uv0Ir4xa288WrIqTWADdVAXqPgcgicS/6fHrRR/gwTopnxor+Whi?= =?us-ascii?Q?3+OIFdJz7shHnlnlg/erYINnKNJAR7Oc89V02mDDK7agTqFLz4vau8PCSWEV?= =?us-ascii?Q?G8oVIpLeVSY9XsB+b0M1Kd+L7y0l4B5ySMFI8ULxij1r+CUC258sXurQKqG0?= =?us-ascii?Q?/zkk2uMckRvJa4evuDFyTH1XyjAEfCLqV1eUmK19xUl2gjNarNb5I68lWpkL?= =?us-ascii?Q?hx/Y5NRjxjCbTuhTz3hb1eKB5qGZk/fsAgeW0zCkAKb9hSGXg46aV7SNKQN2?= =?us-ascii?Q?SGIMguc2kruriOcbwW2wXqGSr+jNjCSXowHvfGTeV2Hrw6wyTekFPiFwYZxB?= =?us-ascii?Q?QWnBYcDEDumJ+9WyEYl64CSAeKQqq27PX96S6necIp3E3stRrCbg6zNa6jNP?= =?us-ascii?Q?7zvQjpWXA9wYS+ur5Zge1oCzIVFabrqtzAAX2HfPW4cSW1aKk9relzp+U1Z9?= =?us-ascii?Q?zi6X+leCNqjd45IgMBal0FrOJ8vrCSXPAEZeKWccv4a8M2f7Fb9HOxawJVSH?= =?us-ascii?Q?WfYX/gVqCfauk7rdJI5ice4DEGynXw+iEscDnwPzmX9GMtYLpXpfdUI8wHG8?= =?us-ascii?Q?fvVkFajEzI5v+dDw8OLOCMhEt4uKXwRwP4sAyK6EYWL3ChofGkDrB2GBc3Vt?= =?us-ascii?Q?Fu+dlk67XuHnOI9yApRfofROHoSUJTWxtJK6ZoyQvYEqmTixpl2SQVg+dQlS?= =?us-ascii?Q?Kpa4HkId99GyWUkNlUaw+arC8biS4dIav3rKwjiL?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?3r6ZE2sDejkM8UbH77cxDVrIGoYh+cGi9iyshzrkFpfEcbAwfl9Me7pPIyWW?= =?us-ascii?Q?ZZ/I1JIHpyzvQiGnb/IturQjewasADuxN6tKFrZqhuL1rXKqYJMvvqO54Vv0?= =?us-ascii?Q?1HuYeHw9v1mS3p3CynkSAnzWjgABRIfQdm7pgLq6PboArgi9bauSun/VWrhV?= =?us-ascii?Q?k1qr4GpoGWHEg8kEo8lA3yLaWDX+Z4C2ZaPJYyHGOjXPbF3NvUrOyjk+0GSh?= =?us-ascii?Q?QN+2PsXBQVf4y4yvw7iltqT12DEb3Dp5raC4Ckzx9oOvNkK9UcVhkAWMw/A7?= =?us-ascii?Q?u2lu+sUStulM+Ij7E7F8KyOekq5Mf2+Hc/tquCL0yqx5KmmxBQmA104R63YL?= =?us-ascii?Q?inuXqe0V1EDmttVRKsTyK6+tSYSgzwlOugSxdIsOguPg6PI5bYzXO0cLN4yD?= =?us-ascii?Q?0XDubdNxta3YjJlP/y50IY1ys5Sw52oRWJK24pWlXeRlwIs3fUuHKnkTNS4s?= =?us-ascii?Q?9FD28/zb2z/VlgHI/vV0TVZELnlSxWnI89vrcbO37ZOXppDGNHBLXMwnVcBh?= =?us-ascii?Q?nVwtdhG8iZHl+t6Q5JAPpIH883OhWs+ZA6cwkvmV9g0fg1ljwz1OqN0e6EZJ?= =?us-ascii?Q?ANanebwfQe8xMG6Qt+SVLFr0WDdVV+dMSIODOdglfTAj1B4aJtQ0Pf6gAKFa?= =?us-ascii?Q?W79hxqt++xtU9w6RSNYh5626h6pZfS81ZIUnPUFf6KULjMKI9+4vGMk9zr4L?= =?us-ascii?Q?YnG8YQTLHXLJaik1EKHVLrag3iMHc1UrZMl3w48vnCAngyxljjUrMQYe6gch?= =?us-ascii?Q?cxRUgN/PhnzfilydRE2BAIENFjwKoMBl76NUM85px6K0YwIbmZFNW4p+K7dt?= =?us-ascii?Q?eu7ZV1pDaXr05PTJ3JvV75U6lywb8vxU0aGQCIdjYRUqUonXRlVkReGw/qZz?= =?us-ascii?Q?yGJhRz5TsUDb5iS3U4GbzDgIHtW3swFZ7ho9vSJy6Ls7P4dvfldvrErdXlPx?= =?us-ascii?Q?ERhsR9yBnwxklZkaCcts/KDNVs6TowdRLJgtawie3zjIO8I7dx87JNgDbm/C?= =?us-ascii?Q?ci6Ub6OtCOxDbKvP68jJx8ZVb1Dp5Sa3lm4DlMdsWE/Ny4P83lW2RAOe78xa?= =?us-ascii?Q?fB2aRvH5Jlo909rIID9EJdA8dC51AybwmDtn5VteCXJmP5ec/1O3CqJjs/xK?= =?us-ascii?Q?KvGbGkGnGmjWukJolvVpPIsopYFl9Y8fC1D78ex4+WSkQKA6eByvNTc=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0187daa4-2b8a-47ea-8b48-08db340372f9 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:45.0440 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: H02fg1kyYkymzqYus/XlB3WnUpEHO0gj3BRTGe3t9rTYEGIfgBgvzuwIjlyrWRV5JaeGqodX5vMi6AG69uauyrbUGbThIjkeWKF4QojsDlY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=627 mlxscore=0 adultscore=0 suspectscore=0 spamscore=0 malwarescore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: 3n25f6C0Vp3I69V4G6C_LKVTj8U4YkXk X-Proofpoint-ORIG-GUID: 3n25f6C0Vp3I69V4G6C_LKVTj8U4YkXk Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Define TIF_ALLOW_RESCHED to allow threads to mark themselves as allowing rescheduling in the irqexit path with PREEMPTION_NONE/_VOLUNTARY. This is meant to be used for long running tasks where it is not convenient to periodically call cond_resched(). Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/include/asm/thread_info.h | 2 ++ include/linux/sched.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thre= ad_info.h index f1cccba52eb9..8c18b9eaeec4 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -100,6 +100,7 @@ struct thread_info { #define TIF_BLOCKSTEP 25 /* set when we want DEBUGCTLMSR_BTF */ #define TIF_LAZY_MMU_UPDATES 27 /* task is updating the mmu lazily */ #define TIF_ADDR32 29 /* 32-bit address space on 64 bits */ +#define TIF_RESCHED_ALLOW 30 /* can reschedule if needed */ =20 #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) @@ -122,6 +123,7 @@ struct thread_info { #define _TIF_BLOCKSTEP (1 << TIF_BLOCKSTEP) #define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES) #define _TIF_ADDR32 (1 << TIF_ADDR32) +#define _TIF_RESCHED_ALLOW (1 << TIF_RESCHED_ALLOW) =20 /* flags to check in __switch_to() */ #define _TIF_WORK_CTXSW_BASE \ diff --git a/include/linux/sched.h b/include/linux/sched.h index 63d242164b1a..1e7536e6d9ce 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2229,6 +2229,35 @@ static __always_inline bool need_resched(void) return unlikely(tif_need_resched()); } =20 +/* + * Define this in common code to avoid include hell. + */ +static __always_inline bool resched_allowed(void) +{ +#ifndef TIF_RESCHED_ALLOW + return false; +#else + return unlikely(test_tsk_thread_flag(current, TIF_RESCHED_ALLOW)); +#endif +} + +static inline void allow_resched(void) +{ + /* + * allow_resched() allows preemption via the irqexit context. + * To ensure that we stick around on the current runqueue, + * disallow migration. + */ + migrate_disable(); + set_tsk_thread_flag(current, TIF_RESCHED_ALLOW); +} + +static inline void disallow_resched(void) +{ + clear_tsk_thread_flag(current, TIF_RESCHED_ALLOW); + migrate_enable(); +} + /* * Wrappers for p->thread_info->cpu access. No-op on UP. */ --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D227DC76196 for ; Mon, 3 Apr 2023 05:24:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231528AbjDCFYC (ORCPT ); Mon, 3 Apr 2023 01:24:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231530AbjDCFXf (ORCPT ); Mon, 3 Apr 2023 01:23:35 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43B3E19A9 for ; Sun, 2 Apr 2023 22:23:15 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MwUaj024750; Mon, 3 Apr 2023 05:22:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=ZJIcErO6gCpLr53xeyAlFo9Dh6PeuItoHvFCmPpopeE=; b=yq99iavokHx7dDj/ZMnuuCRoekfZAZl5KaNeqSlK04pxvsWqQEpkSwpc1RyYEgxtWazh Gsk6VAIjRbXsQsJvEZsBrs3oX3QqLbZ2SPxiisC8ksJVzMZVLdj58OiuSPV0AcyG/xJ8 rqB3jyVZg6gclK9PgSQCa9LOGwiC8Y8LUsYRcuzBZaKlpJKb1VOnJlK8b0hDRZLJaIGA OitEEg/3grDjfXy7YcCVOoIjOdJUjdgIm9O/Je7YwyqIi2UUoDjfmpxtgXPhcAIAGamf HbVqFcet6GodoGhyvmgvGTSjVmrrXUX6hKAdLlMKnafPhuwmDjRmCsZYNvtoHZNimaae 7w== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppc7tt446-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:56 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332iZTr038319; Mon, 3 Apr 2023 05:22:55 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2105.outbound.protection.outlook.com [104.47.55.105]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnpk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:55 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E3Fk8fHQFXbxUx4L75SPpKVNndtsTtMxpzFepb+u/SyfuZRHRuwZXA7348ZJc2ayXY4grV+WTTynN0Sj4XYwi3sUPK9ZD4pzVCX6s2T6AU7UdRJ5g70rX76EzB3deevgEXX1Y+dxPXHXv8mkF4KhzIC3cguMUrE2CeBI6WLIuOo7iBqWHTkVJxQaEGiayAkvL2z+/xhQnGcqzZf5fjUYl0m1a4CJQ2gvrZeZd1rdhSIZmqN/w0MUdZWAWFvXBKlQPXpE6PCkpt9mks9cgkcOoExZUpvmImin60rFapctfnfZ5hjZjoHggN7OYEv3JA7cEtPnZ8kBuzx+R8TtgfsAOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZJIcErO6gCpLr53xeyAlFo9Dh6PeuItoHvFCmPpopeE=; b=GYFfn4lWX/o6HACFvYBHIa6oyUp7J85S9A3QB5ybIBWriZiTwRivris9+CAD5QpEVXkXs+nF2CUhCalO0RdtUC7kdsy2lOCUFVB0goFe2CLjSGeaX5uDHAhTmc/8DpXHx8JOols44WpXEbouSu/1dSQsfmuyITMRhEWmA/jJ62qF82c18YpjkYeC2zt8FzkyMJTXdr3zT2uuy3PbOpd+7nCmbemZm62CJ5SXvEbCCDWiGjo0ZwaK1YIDkG6iJnZ/dipOsm/obuQTIqDjdXZQO77/8TMUJlcfI/ZD3EllKQPvM8AzLulCrmD9g2Rfh4eAaqWEIHBYkr79Okk1e0tPbg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZJIcErO6gCpLr53xeyAlFo9Dh6PeuItoHvFCmPpopeE=; b=oeU9coHOBgncybcyMmLuFEfDVxbWPi66XZWaOd/VM2sTKXVYT0B4NEQWTwpLvhaMMzT8CXY/COf8sInHJ24ll0CH5yYH/COMqcLNBBWT4r+Zrbw397BI1VHK3r3qrS4dFF/BcM8IQKy3xB8MD75L0wCT0j9uM6zAVVTfH6hIcHI= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:48 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:47 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 8/9] irqentry: define irqentry_exit_allow_resched() Date: Sun, 2 Apr 2023 22:22:32 -0700 Message-Id: <20230403052233.1880567-9-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0078.namprd03.prod.outlook.com (2603:10b6:303:b6::23) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: 1bae760d-8984-4fbf-13ef-08db34037643 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /JMEgRLdVtjfsAbqIduYdVtaKETVJDnsWFbwxXSnLwTHw4Yb9b8Y5huwV6UPRr8sSAwGlLLN2IdBDFqxJ7tZkqTna5OlDEkJHe0vl7anwNP2EV1gxtPGt9ywVb30esqGvOAt/Nj/AMpIWY2FTtrRUhyFTBpaAIXqtbRnH+ErC0dajgKPDAK6ED6J8LSWdAwEaYOiSNpj+FFxVNo2XMuxfBEIO7OLFhJAmXxaBCUc5THRjQHI6BNRKHRnsE3PuXLk8ZO1+0Jr/HMzPhO9FcRyn5fnDKJ/Q6o1WDFVBCusNGfY8EmY/gI27codm129pJV0jzx88rSgCCgQhe99CwI13yW3Nvi3LZOB58PIhXHYL+9NMKoTCa+IX6ShBMc12pPjI12xflIMSvl93fA9+N+sGyAe2Srkh0alfuCH3clTBtqJkQ6kvnQkcFmMYEZpaS7D6f0NQjkzUIaBCuFVOnAjm18Bb5HjKDPdm2KU44XHrchWlIRWK63hQW0b6XexPUlUYo1d7mbx/ii4AsWfoDR4eGrzg0h29rarn3Wf918MLz4uETNmLgas8eVuMxmOG6VI X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Aq/HuEd+N7dXNtUvPz7JE4EYv/PgBz8EgC+c9JG9OA57ctbp12SGWCp9Lu2d?= =?us-ascii?Q?XGfhRPb0G57ZWij0+Qrb1dIoWUT4/q7MuD8bsbPCzsBQhiJhrTaF79fvvFT5?= =?us-ascii?Q?zeBUgTvKDvCYQuxMyx4l7yw2jxullhaHVYJqhV8xZs8GgVjXUxNU+ItWlJXf?= =?us-ascii?Q?8eHW7yA9tT3T+8wVqvLNcdFtjcTYbmnmfI3tXo+K/0UqKIM+wA2ZtlDNwqir?= =?us-ascii?Q?pbB/yR/RDlSGg367GenXLCo7rD5Vr4+G5NLz3iDMitJYMGndeuG+af8XzMVi?= =?us-ascii?Q?IC2mC8pEFZ0ojColm2FxHG4gR2lrgdl0n6o5XckhLQ2SEvHhmqgCWbOeb7bL?= =?us-ascii?Q?moR4XOASez4Lp66G9OKuHu60kbEUm2bazWHXaIPuZ5RIoKHD8TFNc2LTkPvX?= =?us-ascii?Q?i3fnWIXQ8Nu+D+P8FsVPzJwedOVi/kHsJHuRQvD6es7+jq/4dBcPewO01PJ8?= =?us-ascii?Q?RIHD3czOP4D42UHDIvkM6oSj6bGKcYwdo8BdP+7qHM0V1iZLNUilYdGn4vHi?= =?us-ascii?Q?tf0szzK07yUojt0UifoAd4yFKUwUEU/dm9jyr0oWH09a4M6LPeQL6cEoOtO5?= =?us-ascii?Q?JU7V7egCWX243h2AT8qJWnkSlqYR2RMs5b5bx7nDiJYzpmW4kSMKKSBIC3iS?= =?us-ascii?Q?UhJKRJGw3yCeJtHidZHBT8Rq5KTH/5N3dfH98R3EW94tgpVdiPMnrYqABR3z?= =?us-ascii?Q?kDUZSVkt8sbUwuCqe6e5TIOgDSXO/I/J7epPO4HAP1xOhKRbIpfGjSNvEgL8?= =?us-ascii?Q?uTO4L4Eqt4beHw2LkdpunSQc2F2sLuffIgDD3T7WF3OHqjudSB50oFW/HDzZ?= =?us-ascii?Q?hbpR33lOOq4BAxAgpZTFSMhBHIRLR2H6FecQhbCFctFfvb6nWvB17O+oy5Mt?= =?us-ascii?Q?PTJ7W5zffMkZ2LTS6R9p+E+CLkF3t/xtUl4KtYrqNaW0QxGwicZH0yvIJWpy?= =?us-ascii?Q?2PBWpCdomOzqns8qd1De975GM8TFsb4u1AktJNOQnLoyKKmHZ/npviHAuPVo?= =?us-ascii?Q?OxxZcIDR7vTa9nU+LjbaAXiebwZbVYnc1tzg8OqGjptmmngOcHK0pVU3e8ua?= =?us-ascii?Q?tkEuBejWMc+FvCDARdqyHEuCKTfcGGrZoJxjE0MG6V3RIY7139eOX67xXru1?= =?us-ascii?Q?45aHIvOwk1P5FD4iYjbjPuKwaPljGnrwChNO0fUL8sA+gvDkHZSyrvNO2Y4t?= =?us-ascii?Q?DiKwWny9wpOsoSFhlogZi0yn0XjiZfsrANt6CLrq6sAX18V7dRY+C21tQU75?= =?us-ascii?Q?fPNsLMokIgbhRCickICel1TKsxVdEdx3WjrUk1CPPnR/CgtSHUn6BQiWZIzU?= =?us-ascii?Q?IRpVAzbcbw0/fD4oJQN73iwCkDCsFYm1sPkjrInKP+C/WjI4jq7ZVO4QZm79?= =?us-ascii?Q?lWvgDeaMXRd43Olz6gNkGi4mOFbNFQZpKrIeY5DYLLWtWyCIJYY2tHvOJcUt?= =?us-ascii?Q?tgg4hCByoMKKD7CSxwQ5Xent+9EwJdobL6L/iU9QI1HXkjXnv0okKV22DLe1?= =?us-ascii?Q?hrt+p8QlXc6ziSO0LQutYGsUXImbwkEpVDO22cQl9aDhj+b6AtUBh6coJXFB?= =?us-ascii?Q?ZUV0Ub20Hbuy4qhmkneP0sVy5/0qra4pfKwEkgOL?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?Cbnia/eFO/wgtFWUA7cTl+DzNH4JwlXTsRuR2KXvyvnnt0dQE++1XfH6ZEV/?= =?us-ascii?Q?uN8kNb7CLsKFmptrmLcG8pTYgvQFnmfJdrzarrtCvBVymrrLbHMVXyoN2Y0K?= =?us-ascii?Q?AXpd6+SCJubhawJcqg9jyI8z2WGrazBLD2D0jahyIwF2xVrsa5N+ZgG5XjNT?= =?us-ascii?Q?dHPY5wpUmjvR3my4u/UgK3nSp/k13wIhvpTPSxWbkUqSouirxIahMY1/sYrv?= =?us-ascii?Q?cWL9BmGXkrrlTkuelwr5hXiKGB4rcF16LY8ZRxkvaWZJSstQK9fRb2e79Hlk?= =?us-ascii?Q?BZ5O0MycnqA9bMt6tGzpcQra0lV5kpKav4/ypKgDLmK1ieQ4+pKEhIPfMSzb?= =?us-ascii?Q?L1OXbQtoAYjJgAW97bDcbhCho0U4nclIjN9uwbo/Cju3LpFm5zZxDDsVWhIq?= =?us-ascii?Q?wn2Ur4NIF/qk3uiQ1JYfU449EZkzUPQKdfPX3Q/L69NQ8jZ8pxxsDpheEUsd?= =?us-ascii?Q?zlARblva8p/rSXYr9e2oSwwZFwwR3HrmecsOJTDwzSMhMbPHRQsPwX92Tbeq?= =?us-ascii?Q?25wKh4v/SXxs44SowvUHRDLTBDVV6lTJs6dV5fkWcZf6WEPKnL1P/JTc6/62?= =?us-ascii?Q?ArsIR3ypQXrvQZMy1hMPS04o6zHqgRNlZ1Nwb0ZHTA0h+sUEAVh2BeEiUS9N?= =?us-ascii?Q?ZUx0hdwtjMEYl501cL6NafwLZTivngAMBtQvIfm8hBx2HdjbrXLhxMHYYRjm?= =?us-ascii?Q?fjswWhg9IgSdnVLbM8YBH+P242AwbOQdOOoAtaud/iI9wnqkfJkIuOEfVivm?= =?us-ascii?Q?qVnMKz1bRPwS6SldOTkY0+lnzLOWPLpssQtOEZzUDPIMb5wkBiH+weR8b1w+?= =?us-ascii?Q?Cr5eRElKpCK5CYIC4xzMxvs2s8bfezvsxQd2MACCa37ciHqOdIdJ4vyxTDWJ?= =?us-ascii?Q?4zy5toJsKc6wi1i8mR96DKWX8UUdWOD+ZWUvKE4ieR0QPq1mym6fQhGHwIaZ?= =?us-ascii?Q?d05KGAS6DSd8GgtyruE7HLufinVjrDFyD9qNnJiPnqfX4fnOyPX5hzy1Z03s?= =?us-ascii?Q?h7VJnJ8SSVxwfDHggkuoBmM+iOrTMTqk2Fx0ro17oxi5MyU7ZQrK+TYjvSqO?= =?us-ascii?Q?ASCL/eeW67R8w+CMKgot1hGzZaqr0onuvxOAYeBJEOXUj7umwLTts6xWWJWj?= =?us-ascii?Q?dBFmT+q+DiCyW04ejvSJrjzOQmEl4pPRcy55KBS5LhT0rmQgmDicDdg=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1bae760d-8984-4fbf-13ef-08db34037643 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:47.8596 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tDyBYPmBhG9E/ZhXTWDy8nevhTrDYQ2xwFEfFIflBN0TDmISNcR80y883HbWBEqB1/K9lCEqqji9S+qnPrpEqO6lNqvE6ujFguDmRECddm4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=879 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-ORIG-GUID: f-a-V6JSgdGnspaNV5qnhzyl8asrVf1g X-Proofpoint-GUID: f-a-V6JSgdGnspaNV5qnhzyl8asrVf1g Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Allow threads marked TIF_ALLOW_RESCHED to be rescheduled in irqexit. This is only necessary under !preempt_model_preemptible() for which we reuse the same logic as irqentry_exit_code_resched(). Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- kernel/entry/common.c | 8 ++++++++ kernel/sched/core.c | 36 +++++++++++++++++++++--------------- 2 files changed, 29 insertions(+), 15 deletions(-) diff --git a/kernel/entry/common.c b/kernel/entry/common.c index be61332c66b5..f1005595ebe7 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -390,6 +390,9 @@ void raw_irqentry_exit_cond_resched(void) preempt_schedule_irq(); } } + +void irqentry_exit_allow_resched(void) __alias(raw_irqentry_exit_cond_resc= hed); + #ifdef CONFIG_PREEMPT_DYNAMIC #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resc= hed); @@ -431,6 +434,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqen= try_state_t state) instrumentation_begin(); if (IS_ENABLED(CONFIG_PREEMPTION)) irqentry_exit_cond_resched(); + /* + * We care about this clause only in the dynamic !preemptible case. + */ + if (unlikely(!preempt_model_preemptible() && resched_allowed())) + irqentry_exit_allow_resched(); =20 /* Covers both tracing and lockdep */ trace_hardirqs_on(); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 0d18c3969f90..11845a91b691 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6500,6 +6500,8 @@ pick_next_task(struct rq *rq, struct task_struct *pre= v, struct rq_flags *rf) * - explicit schedule() call * - return from syscall or exception to user-space * - return from interrupt-handler to user-space + * - return from interrupt-handler with the task having set + * TIF_RESCHED_ALLOW * * WARNING: must be called with preemption disabled! */ @@ -8597,28 +8599,32 @@ EXPORT_SYMBOL(__cond_resched_rwlock_write); * SC:preempt_schedule * SC:preempt_schedule_notrace * SC:irqentry_exit_cond_resched + * SC:irqentry_exit_allow_resched * * * NONE: - * cond_resched <- __cond_resched - * might_resched <- RET0 - * preempt_schedule <- NOP - * preempt_schedule_notrace <- NOP - * irqentry_exit_cond_resched <- NOP + * cond_resched <- __cond_resched + * might_resched <- RET0 + * preempt_schedule <- NOP + * preempt_schedule_notrace <- NOP + * irqentry_exit_cond_resched <- NOP + * irqentry_exit_allow_resched <- irqentry_exit_allow_resched * * VOLUNTARY: - * cond_resched <- __cond_resched - * might_resched <- __cond_resched - * preempt_schedule <- NOP - * preempt_schedule_notrace <- NOP - * irqentry_exit_cond_resched <- NOP + * cond_resched <- __cond_resched + * might_resched <- __cond_resched + * preempt_schedule <- NOP + * preempt_schedule_notrace <- NOP + * irqentry_exit_cond_resched <- NOP + * irqentry_exit_allow_resched <- irqentry_exit_allow_resched * * FULL: - * cond_resched <- RET0 - * might_resched <- RET0 - * preempt_schedule <- preempt_schedule - * preempt_schedule_notrace <- preempt_schedule_notrace - * irqentry_exit_cond_resched <- irqentry_exit_cond_resched + * cond_resched <- RET0 + * might_resched <- RET0 + * preempt_schedule <- preempt_schedule + * preempt_schedule_notrace <- preempt_schedule_notrace + * irqentry_exit_cond_resched <- irqentry_exit_cond_resched + * irqentry_exit_allow_resched <- NOP */ =20 enum { --=20 2.31.1 From nobody Tue Feb 10 23:53:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF6B0C76196 for ; Mon, 3 Apr 2023 05:24:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231596AbjDCFYF (ORCPT ); Mon, 3 Apr 2023 01:24:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231534AbjDCFXg (ORCPT ); Mon, 3 Apr 2023 01:23:36 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FF7B5FD2 for ; Sun, 2 Apr 2023 22:23:15 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332ML952010090; Mon, 3 Apr 2023 05:22:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=4gImNjdBdcopMpLrPUI6c2EyH13eycBU+Tw9ntgOP6Q=; b=VAXqkuILTv1fDmRnW2ObaZ5Y3mpmAS5G9sXXZ18ssmMK2aENP3rweY+kmyUosGNQ91ki 9fXkBRil/1wOZlhtaU+QMCN2UYI1MgdRzJUh14rfk5kpsSE3o8ZQC/r5LW/DbF5dTqp3 nvQBeK2nINQTlk2bji6xt8VMjh6ToTPKO8lo7F5nBYswLftjl/wWPCqoJZhf4DA3xlMb Ls4UMQsaP11TpQwNUwnK9h2pPcYfvN8MkKF+pWZBd323QnZ2R3F8VsbCGpKeNsRgn/cP kSniV524vjnqeP+QpNA5PN0BivXSAqtrW7Zh+0vfk0oVeu7gfq6N2Mk/AdSb9Oj82LB1 ug== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcnct4sf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:57 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332iZTs038319; Mon, 3 Apr 2023 05:22:56 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2105.outbound.protection.outlook.com [104.47.55.105]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnpk-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n0ihLWkM/rbL+bJdATRKrvBavYnXp/Zp2f062BDmqwkm9RJckaDHIllhye01eOecD3q6fhbh2R6D96F5miAW0g59PxH2xog9kboGfq72EZm7zqynDB1SeoIeWESZ5QqFgvXn55vXSxXWvR8cQqWuGhFiGL2mtkyldqBuC5udI1X3UhT3CrUpnZ7OISs3bYNRHOwipnicgdAEsEakEEA2GZqYhNqJT6s4GQ0dmslb5ZfvitXlkPnoT4lsiFgs8kNPeAFgkGvQDcUBNkQ8W/TmXBPY9DmK4ZVbXYtXlHy7UM51xPpK/Sobsa9ILXBO0FDMV1ryteKaTvyHQQwRPAsXMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4gImNjdBdcopMpLrPUI6c2EyH13eycBU+Tw9ntgOP6Q=; b=lv91NjVV6ySkt8VungP3ENIpnfjso/xqzrEihbXLS3HQFet6kHAB34nZ+kG5h9YwT371pN/wInALslUkqbhc3VbvtUn21wZP0ABP14lki5Kh54Ig5kAkmLdkGY3sHd9gHNzcra1QUbEAvK+ugrLhim3J5JBYHPqac9+6e6i63lSvuIh649pFOF1/+QGFtf1Vxflwx0J5H/xSudzhso9QSiY4qSnBg6le3ClLoWlHBUdtW1fhwkQ0sO+jlKrp2w0OT2O0xfd94yG+wVjlGv4JNSDKy/DIU0hGntHtmCcqCDIx6FYzG2VldaZI0+K1YZbc+YC4Y987qRAgz2sfYRuO5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4gImNjdBdcopMpLrPUI6c2EyH13eycBU+Tw9ntgOP6Q=; b=hrxFOJGxUlMf3BtPCEx23f6FMeCKATpfuY9g9kFZjJ2AC7RwqC3kWnXghkSFwEme0Xfu2cYNr6aHsf3KBJHATzZsxw4BLqUbK7gwiukjVUwX9kwczch5q9mrja97kQaMFip1GOyqAbYb16i0k6T6d2psLJx7ZshYY3RuYxMyoOY= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:50 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:50 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 9/9] x86/clear_huge_page: make clear_contig_region() preemptible Date: Sun, 2 Apr 2023 22:22:33 -0700 Message-Id: <20230403052233.1880567-10-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0003.namprd03.prod.outlook.com (2603:10b6:303:8f::8) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: e955a7c0-ed1c-47a0-8465-08db340377bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EbL+kGjbKMFebho9UeIYDAfwzB792wIeFWpV1vh9sGs83OIYJEydEXYIHVSMhBvRHES+eVIMNRRr79X+Vu9k8NJR0nTHtqmipzbeIt/N45SxqXEE1HkqOpsw35vLkg0hhw4O11CfW6Tjm1Ydw0iNXnYAc9VaVf2vx31yaRT3MtC3LY+qkreaWxsx6ArAKfEdpYrr5JZ2yEYvL5X/7o7u7fAh0E307GMLx5c/ZRqoeAJxdaDo/6vIJYLIaFrciCZq3+24PUK85Ulc3aEp2TVDoaw1q5isZ1T5Ytm21/id2qmRrT0wVgQFgQB0mK4yaMtQUdiBRVyTIdujMCX4S0M2Fry1pLdqhHLMbbr7sRB9lMtrmfTdEnQbnWRXLQuKvJgoFH8i0BzHwW01jgtEW4jXvUh5PQTSAyP6JXXIaXCcSUu3UmB7yiOzj9CCqY2gCqpfQTmDM2VME3yA1oDPVWfEQWGsCCIc5MzSRdzQiK1sqASy817NjtfCyQ2s+Wt1V1vtPzH7qYYJIMTo0p58LbCJI4mCdnCVnEuhJaVOqWD3zjz2ObXyXDEGwnTeU0C7l6hFRXZiCMCHkicNe/I/DAvZp1hR7NG5/8NW8iL7xvfIW10= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(8936002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7VYrq0RaXD3bgzxE/BXFfDzqbIwOC0DYaCtJqLH5lNpBA63lLnaZWuLJ3tPi?= =?us-ascii?Q?lkdsy+ZJFMRFm14jYuksDPqGV3NhE+nFW31Gx0vaEQLgYG5DtB1/4/LWNZFK?= =?us-ascii?Q?ZGBitzCcqxJY57b/BuuxWh4BCdhvg+TjWkuLbzXdsBSWEVOqlYMYmHTLUwXS?= =?us-ascii?Q?APMPzDA/+pKObtOnrr6tHfd4lOc3IdhkyvzaW4lWQD4t/cffS/Ml1DwEx4Ur?= =?us-ascii?Q?ywlbU1Hj/kHXjuyT8VqiKXq0ai9DOg/0HYrndsVHXos9ka/sueJbJl6yD/YR?= =?us-ascii?Q?pPi6+cgLSv1v67QlGzejM5H6e0HXF83YfzsXJ96/y37iRufWvXtmOAmanWK2?= =?us-ascii?Q?iPi394NaCS3+viVN2qgCB1VFj8qX1hXLn9GWU+KcmiDpo0x/kWRg/SKal4I3?= =?us-ascii?Q?Z1vl3qSBuHi1yTPjq8RQzYQfDSwVPGAk7ZqhbIG8BZzoiKsNLiAEgN2vsScL?= =?us-ascii?Q?dmHgOiYhN2csxq3lIUNqtes7FGzCU7SkCtYlaK/ua2uzW7M+B1nwETsBpBB2?= =?us-ascii?Q?/iont+xIF0Bwbt0ze9z6YaCAjZiIsnDXlK3hZxzBYWZF03ihM20VNuhStgF0?= =?us-ascii?Q?JpzrWWKcyiR1JheGP5boZkKHOQufFHpTkjpHWmXF63eXYLTtAzXw1zdOZKpj?= =?us-ascii?Q?DGG2huHj9+xoUaM7IIbryj2eRy36xVwSOtpyLorUAztf8Apf6RVje4ZqegPn?= =?us-ascii?Q?qIop15cCy5soXu3hfWIQwzulUZCd+WMCQbyfLzHkIJ3FReZTaJablAyOkhqT?= =?us-ascii?Q?+26m4mahzcyLVz0U/Vc92pDcE11fGqxQTiVb8ThmjW33PvV/cT6wN+aTCPVY?= =?us-ascii?Q?MNzyBjkTXlGpnqsX88hI2FTB53OX1ov08ltA+WmN2SmF6Xz3DUbDRxBkyhHo?= =?us-ascii?Q?/GMufWTcWlPjKMiPap1V8VPG95D8w/WyuDEp4BXi5QgFgzd9jVjP0+2no3bS?= =?us-ascii?Q?IIXyzkmV3yRpZitM8bH0hmZVJdg5UhdZJBCmD+4poXmAvgeTTmfM0xdlLXqh?= =?us-ascii?Q?tbcVARD/MDq7/yhZ9vfbioALR7EjaILBzKIfbrgESsznglPrE5FOo624UjsA?= =?us-ascii?Q?YCSeBvyE4Oc42zVx4XZ9SQKAOGQ1BfKGEqN915UW0NpKeSAWqD4CAqif0sv/?= =?us-ascii?Q?EEHkxFa71mfJ1cyIplzf5HHgFKFAPrU/+hewPly9TKqOjQVdi+aH0hj8mW9K?= =?us-ascii?Q?P/Yc1czmdf4XlgZGp0XvDmfEa4h/SW6QlXs74DUtt1XSa3rSXbGpkUjQcl5y?= =?us-ascii?Q?p+q0GGDnI7bbrcVWvnpPcO/KHwBfrQLfxcaVMDTZtvW3EDxyYqlRG3knO5MH?= =?us-ascii?Q?mgN37sUf2t6aaEIiYjle6HGXVS35McEoImJlH/k0Vd3cJZWJUUgnb1PIbteM?= =?us-ascii?Q?0l3KFT8GAf2vbZBv8jtN/mueAIluhCi8xNFqirWbvdJc5W0oMJuB1XeHB9Lx?= =?us-ascii?Q?wU3VfZk6Zy5lj6Cm1dL2aMww+S8vQS7aBj1ylgqT02YOlm+G1ovPao3R2e+h?= =?us-ascii?Q?BgmqyseJ9pXjE3yqu9govtzFf3ANq6lsGcvx5Rt/cc0rRSGNAYPlyPN79zPX?= =?us-ascii?Q?p/56H8nI5DmgYmnRHjQMofC2oVFr6/CwF2ekJkW1?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?zr7CdQo1TrKnvgmfFDz3j1d2Anwbz6bLRpTr+VoRl9+nwyQaWpYiomWGBkNG?= =?us-ascii?Q?mt0NUftprS8cjk3KoubFfszkqIss2U3xLl9O6CpJQs0ZdlGA9yX9VHUFIgD9?= =?us-ascii?Q?1tdvLaJ+LPHJylUYK0YrB3ii/R+LmJ8XN/KssKle/J8x7VYaAvnE/nAWfVk+?= =?us-ascii?Q?aP7sLnt6DqQxvqA/ldn26Rsm11Xw8JOxrBpnO4hDetLXuztPR89cD2IttP5G?= =?us-ascii?Q?zoZaHkB6H8qQwrS5BNuLkY0543Qew2FB3sYTKsLFt6D4b69uQIDKRaE/VoTA?= =?us-ascii?Q?53ToVyf3cJjUDXeScPzVYnayT4IdHbrETHe72U3No/NUP4lSfr0GMjKiRCIh?= =?us-ascii?Q?2Q6tX1c78RLaeuwIG3tuGToMY22D1tZvgxlae5U/D22tvdBbB5pCj0zBEVSO?= =?us-ascii?Q?25PVshq+dYHgJD9dJwtVKc3cZGeKjbbh23Doj0wFK4PPJTX2InbSVk/s6TNg?= =?us-ascii?Q?qhVauNFnTIovP40U+Emme9r43WH6gc07gcyzOsx7STmKWRZdcZQ5TRJdO7Yr?= =?us-ascii?Q?2tH3iE2BEJXvT67wHGPoadidRC7Dl2zqplJa7cAu5KkVotwPFUPxNsy5EFeX?= =?us-ascii?Q?ET9/2BUGbzwTASTVVtqwruC4k7ObwzevzvVNm6Kv6+QeSjJvih1Di8YILSSF?= =?us-ascii?Q?8G0HuNGLwwWfSlzKip+PONfJp3oAMYzAITu/QxdpsbbzeAAt+4BdxoqGs02N?= =?us-ascii?Q?BfAue1MyxOheYQbZlfgaS6N6MG2006KlSFOgu+oAK5/wzsfOXxmzhmp3mSXm?= =?us-ascii?Q?9SIgqHgQ8oRs8biJ3hjoOemDPfOQqFsMqEcfv6uJP+e3Z075BjKjyCvXCnms?= =?us-ascii?Q?rPcwe5O4Wa/lQ4lNXDnrz/R35bFHcjkXtl3nQiYI8B3AZ64+aJ/bXhcC2ECr?= =?us-ascii?Q?6RwEMb4TwLSCL9OJJ90CxxTYukwfcX4Poc6ome553Z6UX/SpohcMJ/sR7t7D?= =?us-ascii?Q?5D3Qj6QlqWFVVG61hYcu/Kcanazh+NBaxSTchoElgq1ntzk5HecvAJ9Nzr3r?= =?us-ascii?Q?glZA/3QaD9CD3iOLsmMOqmXkyMdqYIhXyd9YYGo1aLYS7hsoMIXo6DwFL8Et?= =?us-ascii?Q?/B7wM2Gdukn2RONgDs8vvxKEUSvw8K5gmy6ZrqqqeQm68rykVCvGS5cvD6Hk?= =?us-ascii?Q?rHxk7d5DlhnLaV0L3FaPhzXcaJDKZDBeFyKye0K7ju3oztBdh7LZ9uI=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e955a7c0-ed1c-47a0-8465-08db340377bb X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:50.3049 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pToPetuEUS3W64nx5gmBwlvkqB4VZHXRQ5V0Ff95DELUVJ0dIXKXQ2JbahRL8UPalC6BtBeRJbS50q2c4MrN1LLDBTrGYYyWX/4rkAerLAk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: JqiodZDSs7DsfTtPyIchBXU6qHB8E9Ma X-Proofpoint-ORIG-GUID: JqiodZDSs7DsfTtPyIchBXU6qHB8E9Ma Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear_contig_region() can be used to clear up to a huge-page (2MB/1GB) chunk Allow preemption in the irqentry_exit path to make sure we don't hold on to the CPU for an arbitrarily long period. Performance: vm-scalability/case-anon-w-seq-hugetlb mmaps an anonymous hugetlb-2mb region, and then writes sequentially to the region, demand faulting pages on the way. This test, with a CONFIG_VOLUNTARY config shows the effects of this change: stime drops (~18% on Icelakex, ~5% on Milan), while the utime goes up (~15% on Icelakex, ~13% on Milan.) *Icelakex* mm/clear_huge_page x86/clear_huge_page ch= ange (mem=3D4GB/task, tasks=3D128) stime 293.02 +- .49% 239.39 +- .83% -1= 8.30% utime 440.11 +- .28% 508.74 +- .60% +1= 5.59% wall-clock 5.96 +- .33% 6.27 +-2.23% + = 5.20% *Milan* mm/clear_huge_page x86/clear_huge_page ch= ange (mem=3D1GB/task, tasks=3D512) stime 490.95 +- 3.55% 466.90 +- 4.79% - = 4.89% utime 276.43 +- 2.85% 311.97 +- 5.15% +1= 2.85% wall-clock 3.74 +- 6.41% 3.58 +- 7.82% - = 4.27% The drop in stime is due to REP; STOS being more efficient for bigger extents. The increase in utime is due to cache effects of that change: mm/clear_huge_page() clears page-at-a-time, while narrowing towards the faulting page; while x86/clear_huge_page only optimizes for cache locality in the local neighbourhood of the faulting address. This effect on utime is visible via the increased L1-dcache-load-misses and LLC-load* and an increased backend boundedness for perf user-stat --all-user on Icelakex. The effect is slight but given the heavy cache pressure generated by the test, shows up in the drop in user IPC: - 9,455,243,414,829 instructions # 2.75 i= nsn per cycle ( +- 14.14% ) (46.17%) - 2,367,920,864,112 L1-dcache-loads # 1.054 G= /sec ( +- 14.14% ) (69.24%) - 42,075,182,813 L1-dcache-load-misses # 2.96% o= f all L1-dcache accesses ( +- 14.14% ) (69.24%) - 20,365,688 LLC-loads # 9.064 K= /sec ( +- 13.98% ) (69.24%) - 890,382 LLC-load-misses # 7.18% o= f all LL-cache accesses ( +- 14.91% ) (69.24%) + 9,467,796,660,698 instructions # 2.37 i= nsn per cycle ( +- 14.14% ) (46.16%) + 2,369,973,307,561 L1-dcache-loads # 1.027 G= /sec ( +- 14.14% ) (69.24%) + 42,155,621,201 L1-dcache-load-misses # 2.96% o= f all L1-dcache accesses ( +- 14.14% ) (69.24%) + 22,116,300 LLC-loads # 9.588 K= /sec ( +- 14.20% ) (69.24%) + 1,355,607 LLC-load-misses # 10.29% o= f all LL-cache accesses ( +- 15.49% ) (69.25%) Given the fact that the stime improves for all loads using this path, while the utime drop is load dependent add this change. Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/mm/hugetlbpage.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 4294b77c4f18..c8564b0552e5 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -158,7 +158,17 @@ hugetlb_get_unmapped_area(struct file *file, unsigned = long addr, static void clear_contig_region(struct page *page, unsigned long vaddr, unsigned int npages) { + might_sleep(); + + /* + * We might be clearing a large region. + * Allow rescheduling. + */ + allow_resched(); clear_user_pages(page_address(page), vaddr, page, npages); + disallow_resched(); + + cond_resched(); } =20 void clear_huge_page(struct page *page, --=20 2.31.1