From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 991D9C83F14 for ; Wed, 30 Aug 2023 21:14:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240900AbjH3VOT (ORCPT ); Wed, 30 Aug 2023 17:14:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240913AbjH3VOQ (ORCPT ); Wed, 30 Aug 2023 17:14:16 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 578C5CE6 for ; Wed, 30 Aug 2023 14:13:42 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInfdj032402; Wed, 30 Aug 2023 18:50:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=V2qBt25bpg2D3yMTQCWte8BlIlS/6RXGVrigjE6CaKg=; b=bU3ORyE+KcJVO19uIH+FQhex8iHbNXXa5L5qX6DsfGpOb0a98DzGaoPsxfWz8hyBa5TT ZRxIFdX+oT4AkZetf5YhruZEuz53uRR0kJpr+1HEArV4zl1C5GOOSw0h3P1T9PwUIvPS ILuM3PmuPOk5+qffLuzhKQDy21VJDAyPiCUPj4QjBYalMZQBHcauY8QF3HZghSe6HrTn dNDwA/AC9kKib3K+8gi3DmWjCR/9OVQCPCAtFpv072+jTTWbUgkPj6vywlq0smh++D43 eJkZjNBzzpnD1cNi3ERUQh90Vxo3rXeHpyw8vy7tCgMSNnei0Em36DO/YxTjeax3OE/q /Q== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9k686b2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:07 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UIDu8g032772; Wed, 30 Aug 2023 18:50:06 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2049.outbound.protection.outlook.com [104.47.66.49]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sr6dqehd0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y7VkF1CkHtnnlqNDVASGby2tY35cKWDuAMphkRQ8rasG7WQhoZz6iGTwXihZlSk6vW09fWKtmC5f484TMkz07ebQ3QMkL9hOjIr/8XrQiy5UiK+EZNMoilVKK4B2fp8nHfo/6AK2yDeIs+H+WUzkJg4ahk9e7DpRo5aHcXu0cQyIu/oDwMh3BX2FvwfmVllDJzDlHufRGH9NeTLuFgXg7kuLxJnjmvy6MIMBwC01NCPR8B1LUtbfNX1IYRzJ5DNjd5IRyjAyzQVwo+u+3bxvgh80jKUSCWEBxrN5j8ns5aqBBDvra2jcfWiEmToMeDqUzopVxDrchrU8MPYfTrneMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=V2qBt25bpg2D3yMTQCWte8BlIlS/6RXGVrigjE6CaKg=; b=R+5JCffPlraidlcQKJnxugIBlmnyMJTSiDpDdKuA21vJE3ECHxGdM88dy9q5Kr6YRZ3wt/ve8pgb9fGQhjE+Q+t9MdvPr3WIDfn+hvuYacoU1Ulu3zn0ccKLyzobDkV9+14qGVmfL5Exb1oP/uXpwPUmf6vTa3Pu5RVMrAqeRIyNzOJTvqWlRbCzpkDtkZBTZDIyfgFtWW6effvN+HuoPPymXtXu92GtEPP3e0d+XO8PGr7XKqEzJ7b1RrvJsZWgzkkcna86vuq4FmGLkVRvGvfiFV1qc0EnyG5cbyfmkP8PZYv2VdAkZPJCyhm+wOh4MyK0lnsdomdDp5CtS3v1rg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V2qBt25bpg2D3yMTQCWte8BlIlS/6RXGVrigjE6CaKg=; b=XWE/IlpxOPoXMb3XV3UMEJUFNdck1zWvzDCS61Z5VksJOf49MOx6dgFpl2wSY+17KQQsJZBo5CH48ZanX0F3CdLFfOdP97E9fhqKfIDTw7aL6d1BbJr1ZgJMHt3qVBcg2gvKSt6GdSUedE/ZC+ih0JKeg909SKnyWMT4+ncOwkM= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:04 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:04 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora , Linus Torvalds Subject: [PATCH v2 1/9] mm/clear_huge_page: allow arch override for clear_huge_page() Date: Wed, 30 Aug 2023 11:49:50 -0700 Message-Id: <20230830184958.2333078-2-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ2PR07CA0015.namprd07.prod.outlook.com (2603:10b6:a03:505::28) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: 88e90712-dc34-4d15-c881-08dba989ec30 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /cOrjgqvaWjaaJy8zApf+RRQ7Z1B61halLQvLIIyugbtv1qHf7lo5xY6yweRF+i6TVqZlnGyKY6mAweZgiaF5Nkdn9VZKK/TRFByyHuXgU2NsFwzFp4bnLWt5AzHBF5Hgg5NdE6iS+7/3RTbbgyzwrkemmm/2utCYZyK0b4+zVpFkhFNeqG8qc7nAxg+SyZ913n3HRhRduwdcEpxpCQbpa2tPT2iw8xFC0NN+AL4/AYf72ZDkctpFxH3/DbFSXbI2Zy83rvZQWIU9g8I0X1DrvhZk/151dqugQWrimSJrh3HDNb3lGIqlyBHRVdrHdDj1XYXcSkDHioRM5PkT/ITsTDTepxRpUieWU1qnVPWJBiLTNZz8lYjWjTthN31EX+/fKzWq/burIvoiaaI1ZTOUIx9awSiLDmZIh2/OmAk+BeNY3bVq55PA03alBGtWiFfylAt6hJ5i/IcMZ+JSMPUn6nfQ/BlHgJajm1UjckvcHvBF+CFMu/Bnh73LCtAqbiVJSJ8fQ/gcxQv7YvGqlE8IKoyyUiEyy1pbrszCQHAXNtBM5MQpXe2+piGCuwGNYOpb9lgwt3Meqrk2kIaKfxSAz5bDWk0BenXmAQSgT81M8U= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(54906003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4744005)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?X1wyC/J05XQmV7EsgVIks1cK2v0bSV5lOJWMxfKEPc+z3SiS/df7ICAbDSjx?= =?us-ascii?Q?CeTLtEjEfbjUhUb/r3md25xeu1iq5zIT+fLmpQrk8hiAxLASmGCi0ndRhWPY?= =?us-ascii?Q?ajxmmR0pekPI/cu+WNHMR2DuSZX/BbQ/lcimEI24Zy4XAmvLRQLbxrLK8q8T?= =?us-ascii?Q?4tnAOyv1sJNW2qzVsos+5G62qLJTjj+uRlKeHPipo6BjooZ/h0bYkFQCgNGz?= =?us-ascii?Q?ngMMiZ40wT/MNx3cCFtyn7L8GJLQ8id7lNmmrIrReL38DX+WR2PcQKEhVbXp?= =?us-ascii?Q?ilP+46fbg/MIiHktJoP2JVsp4PWEfow4XuJwfSkFJzmmxJSuzzgmL8txhYsK?= =?us-ascii?Q?qitGbRLLfLsxgEXbvJNBU2LeksfL083CI/5/FkLP58EJNp+AR71Uxgvzg0vJ?= =?us-ascii?Q?m8Cx8xpUTV7XF/JqFQaBJnGzChQpfj+ZVsToFWmasiva59dUns6xcYzf6V7B?= =?us-ascii?Q?4LgI+bn7wjpiaDnTZ7pizxP1SOUnJzICLNcI7SqLzCp/fkdiCZLtYVfefGNu?= =?us-ascii?Q?jq+OUkG7uuO1f7Gp4ERHWWE/iRgX+p64E1wqwIJm0iR7Z1y+3up+NFcmoMdP?= =?us-ascii?Q?b/C9L2s+F8aHmy1Wozg1EHR1ZZ6Bje9BiKN8ZX1C9ccLAnCiTpv4VSe2ZIG+?= =?us-ascii?Q?zcd0MpC//4NSKmZL/RIzumgEZGBMz1Vm9j2aUQ2xa9hCTikhD6fhofK4wfku?= =?us-ascii?Q?/Nl1lEu+hwU9EwHwWpUi+cX98HRvSIyimET7Lf2ABQoaT/NJEat7ZpU8bpzj?= =?us-ascii?Q?Gg1mNId8odfuoWLbDblimm36zzPViINDW0omKLjenqqwsgC35DmOzbimFclx?= =?us-ascii?Q?OhSAouAy/G7tdfhH5qkqZuCyd7m+c+lxq4cYQN9gjpnpKlSCTsRnFZkZL37g?= =?us-ascii?Q?xUDc3XQ02MzapJphR7YEUGYOMpr9YnECXS4zLTiys+nXN47pgSM8BFNi5Feu?= =?us-ascii?Q?dU7baHlBirrmrtKNSTrtwT2IGGXQ6IGTLiOYXlZK8AI+OupqbFsxvLTxyOvz?= =?us-ascii?Q?Qk1XHWuy8wxOm8HHKDi/Z8XQN74b/75eyBgHHQxSV78q59rLlkrDqP1t/shF?= =?us-ascii?Q?5Q9Gg7FKyUMkkN8pi7tsjPsedT3BTMwwg8u8HG1tvlBIg66p7Q6udK8lt55y?= =?us-ascii?Q?cLOIU5CIT8/ieC0oah2acgLituTmq/zLLEOG8U4U/sdSbya5mjXwrJGhTGw8?= =?us-ascii?Q?qipBtwYaeN9/yaYo0XAPA2Kfnj1X55m0hAcPSXZUU/ypq8cFp1DUMupoj1QE?= =?us-ascii?Q?EgSehNv5tYUrmUAqiZBug3Xa5vQ21GShOIxna/pSzHuaBSz+17pP5skIaExB?= =?us-ascii?Q?zddq1biCAs+nu5USZn7sb+9ksGM60jGobnQ7HLf0s1lpeDTr/pu26rQJZQ5B?= =?us-ascii?Q?pDuEGv3prlTpeQP1XYCB/OIZKaouxcaKCIarG2BGCIk/LYT5X6M+FUJyXQ+K?= =?us-ascii?Q?n4Xl/TncRnioR9fpKr0AR2EyOE0QjO/vSrouC3jSYoN+H+4mTrDthuiSqK23?= =?us-ascii?Q?lus86PRSd0zf4yHFTRHOl8TDNxOEAXPLWzR8FsBsgvAbVpYoM5LSfcV5C52n?= =?us-ascii?Q?8Jr3q12lxIAWf9QniIvh3YCw6/zr2lITe2MdpqfRiEPoUG6AnD0PuKR3d95k?= =?us-ascii?Q?mA=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?ixRomdRqZk7RzwTLspYovQbngCVoB77Yk4o4nI8zGN6a8ctr0a7qVXOreSdQ?= =?us-ascii?Q?HBWX9+4+fuI9Z5HG0tsquKV3rLrddCyY1wSyeBOhddXE8XBc9xu7kb53uD4k?= =?us-ascii?Q?rcf6A+m55i6A5RsgAiy27sILlD7yYkLv9IgckzSLD92GK9AN1krvcHDGHg07?= =?us-ascii?Q?Gnn9d/5d5qIPEDDyEe47lbqi8QZ2Iz6OnAhXgChi47qtQPpwqAumI3rL5r8B?= =?us-ascii?Q?c2spT4z2SSIziOoPmzSIYtaa+0c5SNAJ3accaR4PA1nQuJ9hzWdf9FsIQ+H3?= =?us-ascii?Q?gtKpA6ut2OQc9ZhMgpKnSgA3jsM51tUTvj8Vbwy3+KVgoy7WrCc3nhAfNyfU?= =?us-ascii?Q?TzRlhk8zpPrGpCtl60P+M/j7ZWjcuTGAJSaAr9HuLi//eXwwq6HX6hF+0ldv?= =?us-ascii?Q?cVAJNuY7XcNqwFzDdZir8IK9Rwr6CrBVPDUcy63KaqgVrd46g9ZY8m/5L+iF?= =?us-ascii?Q?9J0koHVn+S4xGuDqBJYA0Z+suyCMvzPj8zUTIWeNhXkXCs7EY4TZ0k9kVR6y?= =?us-ascii?Q?bWTeseqIQVciLh1wPGCR745y+GMYFsPyAupNepdTgLXrvVdW7fTRZkkkcKkd?= =?us-ascii?Q?fLNFf9aAyAAg25j3rjNVBpyuZm/0dn2tYqGJFAUuf9FUxIZIBG8zRZFTKuLY?= =?us-ascii?Q?2aR/ZvqruZq3/eeIZeFLTJ+I04xn8nwJdMEhJAzm0jtpIB4TX1U03xCNp5wM?= =?us-ascii?Q?gW3MjSOOCiHHeOz5LMTdUZ8rXXd2sbajnK1K5gFg4u199X9RFHZNVkbU7BRF?= =?us-ascii?Q?++VluQxJVRg3x39q9KAs3dfA6hZRk5gbFacVxWLwI5JT55eBOFhT2HtdS7fW?= =?us-ascii?Q?nWz9LXa/GG4U60n4lICO1cevi0ht8n5dfPe3QLHTILvRcd9eXstNcC/qsbjU?= =?us-ascii?Q?rkLBY6+yvW2Cbj86ONO6n179ENeEeRPqA7y1wVN5ZhexrhX2P3MlvZKFkLy0?= =?us-ascii?Q?lZwHCwhU6IQMWBGuuZgzjpk9/xjHqtO4uPwdCTMZTD4K8wcjjXgPZdEhL4Yp?= =?us-ascii?Q?9wfn7BMVN12AL54KQq69HAxaWNRiSbCq5Tw/hMJehsLd6v94exrx6MWB79Iv?= =?us-ascii?Q?jW65OpwFS6G60/pteoLuIMrxwjqe7CF/NbKBnD2q0ZDte87tpA0ECl88Y1L5?= =?us-ascii?Q?PWLrLvp5fsezgPd1afTU6e9buSSUwnzRe1gv/LrGXn6KVQPqJxMuNP8=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 88e90712-dc34-4d15-c881-08dba989ec30 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:04.4151 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0aIn4blNgYhZpYoip8ut2asFexK4l+AbHq7Wxv4B3GndIo9D0jIoiaFUdGL7M/IHkHecZhKkre6ONocfCEyTjfKFSHmK3MCT6XvcS2Q+O34= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-GUID: Gy16ycZ5fuBeDdzttyBSLeWGicyGRc8u X-Proofpoint-ORIG-GUID: Gy16ycZ5fuBeDdzttyBSLeWGicyGRc8u Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear_huge_page() is constrained to process a page-at-a-time because it also handles the CONFIG_HIGHMEM case. Mark __weak to allow for arch specific optimizations. Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- mm/memory.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 405a483d2fd1..3854f0b9b3a9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5982,8 +5982,9 @@ static int clear_subpage(unsigned long addr, int idx,= void *arg) return 0; } =20 -void clear_huge_page(struct page *page, - unsigned long addr_hint, unsigned int pages_per_huge_page) +__weak void clear_huge_page(struct page *page, + unsigned long addr_hint, + unsigned int pages_per_huge_page) { unsigned long addr =3D addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8318C83F15 for ; Wed, 30 Aug 2023 19:49:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230330AbjH3Tto (ORCPT ); Wed, 30 Aug 2023 15:49:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241893AbjH3TrI (ORCPT ); Wed, 30 Aug 2023 15:47:08 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94D1E7D99 for ; Wed, 30 Aug 2023 11:53:01 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInbgN026408; Wed, 30 Aug 2023 18:50:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=ddZ9ZaIYlX96ohbL+L/v67ViAco4TJ28+TNGKpwzYho=; b=1wBBdaqQ5GFL5cad6b4FSqnuHoq7LDWxxhAdFA0xLN08Ix0cE05Qa/LycaHJEUCkPr/4 kkGZBFeRtK/jiB1wryS1nYLbnshB+8GUQaeKG+cSKwWO7VdIvYy0UpcoIfGSNvJ8lJpg yArtJ/7Wo6mcFZnPPZjBfZOdCpJeQQemY7oxrVr0aeZcNO3R1bELzPxOoBLUghZHG66t AyD+V9gAQMNu7YA43se0kzJuvsq61C5fQ+H4y4zGUVp8IKt5JUPX2yk1hCmno8jBaGg2 6eDys2CVLO+jjSArbsy2mgdPKuZnkXVhcc4nak4PHuOppycDtG1/mHWqcfJw5mrcu8b3 Bg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9xt86f6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:13 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UIJHaB009143; Wed, 30 Aug 2023 18:50:12 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3ssepyapk3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CZ0Js+WxKxmUEtu7mp5B2tqGO88xK1aA6IGahKpAeE+i9Qi7MkD4WdD4KN/05av/1M6JbSDE60UBb/7+EwwnWaI2uv28mQF8SEssbuzek1KjWgHNs9UyrVMeA9dsB7ARR04/VXbxPv07NS1NOtRnaUWmUtceBGLcfXe2bd505yJ9xB5ZYReBJv9VvdUbg27uG3Q+hTw3tARNeVbH9RHQGJTwahf09YMMA2LVOcEypl5IcN1fB0t0fLHE32ENhLmi2lLp7nMjUH3qLqZFGr+ppaQN80/n8QHJomGVFv/lzmuHqAKK/iluszDz9AUzmMpChyJsm5iaLgx/k+DE6LFo3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ddZ9ZaIYlX96ohbL+L/v67ViAco4TJ28+TNGKpwzYho=; b=KbFpeiIgkb/FTja0gNDh9SkAijxpzPl33gZXlRKgdPt+e1/tNHT/ef79dozHvLHx1jlkvKpUbltQ4eoAfEtl4fY7IZfDe5gGE3vIsBsIulywdvTJ4evbx0GhctITK/DbeLYKH35pE1tUaUNRiZ9uLhYDPXk4zRh3BCpGpph4XGgnh+io+yZgq1Ht4+guOwUF3uQZqGXXVOrnrFsuCsZQ5LzqmRGShvVJgT0WtwqJKRReiv53VFGhY1IlVx6gNe+ZDql8dTY7EgeLLUiSBVU9pZo/eiHQ0Doi3wiEBKDau3upIZ89I8sH0ROaj3iIcgcLLTLK+Y54AwExKrRxjYPBqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ddZ9ZaIYlX96ohbL+L/v67ViAco4TJ28+TNGKpwzYho=; b=H0hRP/gR1/js6O3uoMs7LOXOkvTHHFPmuH2GNQesKEUeMBTh+Lg4puykUncycCbdQ9US8lqMvx1y7AB+eZywuUpITwVF3tiHeoo2Fcwws/ey4Ze08BEPzOoiGdz2/yC9Bg/d3fyM0H9jWQ5RiquzcTmPAVCEsMJyzqXOiAwfhkA= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:07 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:07 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora , Linus Torvalds Subject: [PATCH v2 2/9] mm/huge_page: separate clear_huge_page() and copy_huge_page() Date: Wed, 30 Aug 2023 11:49:51 -0700 Message-Id: <20230830184958.2333078-3-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ2PR07CA0001.namprd07.prod.outlook.com (2603:10b6:a03:505::11) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: aedf43d9-4937-4626-f1ef-08dba989edd1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3dS8xJ/1R7CrPbD+dR6vZcgjWD8wA6bWGbSb2uBglyFyp+mvjvxGqq8boK3Ej9cV99fK0U3R3zr+WaCwKTwIpFuTx73pNp/zh6shFstUOvIKmKCGmKNrAmWGtT/MFxkS7qIrvnYtbfv2c/n9g3aC0GAAW+K0Uob2YETrkZ7+1rLSmp0XSw6OgSy34mnbkUvodNXUUsFeO/U+UCHjcchqTTve7v75rIiGbcfPwmeLZ7LkeYDVAmRY7v3rcBeX+nlZPeikMag795x/HG99kmWjZySPuZCDV9Qf+1oyUkEuV4qcM584edCo8pWFXab/oDXUuzybZDc7AicfTVPRkRdXAtBsNsG/6nLsputqkfOVIAt8Gj7wvb/hNbWmjBrJgreZ/LK/gb4FEp5mk8Ex8ZzMly1oPumFiS/O4osb02TQIyG1NU7y1Jk5v6n/jc2jQxJaN4jsWbByj2dNAAnx34udCgMIg6XV00yO9Wjzq5XEuRGlsnR5gdhwi6annh6oNBi5y0WqKiL5NNk61VM+dAHesTj2swToNTQCiLRH595AL8AX+0DpsmfsSg5pwYteh6Iq+jVHTE5SrpM/j/0UHtUxYi7RDB9WTWZNpTvp94oj7hE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(54906003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cdBZC8AWzkNfx1uZVvbgG6AViaUhuYurjupZDuq56u6UIvG3lRELnSJd4zv/?= =?us-ascii?Q?0zUGUcaaRErtfTsHTbDi4cARiHHmNz0/ZdZwkSn5ATfeREB7JHhS9R/qU+PX?= =?us-ascii?Q?KTJV3Z1ALQKZgv+oTpYyh6ZOa5+ZoCdumtFdtlgJZnm3EUBxogw7HNRqoU5u?= =?us-ascii?Q?Xgw1UzeyDDEdDScoWGUP+qoSRuZHxJeNuMnBjG/S5xXB0lBrTCOsWTaRvVlP?= =?us-ascii?Q?8cocyMBh5qNe0Z1oDWUR+0KqscAIjSZLgQiAaPFHf2y2RDv2i5cjL6RDKWfW?= =?us-ascii?Q?5qGoxVkSYDj+dJjEtoosDAD4MqEu6lmRvTi/ttKR35CJQaEYbrNCjDg5AQ3Z?= =?us-ascii?Q?0tP7+MYxKY2xKkUEaYYaK7XPT7AAbjUi3oVY8OUxxuyYy4s26ux8JLU8Cv3/?= =?us-ascii?Q?yTd+3QxzdAuk2uDRkncp8IeHZ/5FFp/CSie76LSRCyY5FDkRi4Fg8gyDdCm3?= =?us-ascii?Q?Aee4N8IfRNacz/7vYrHe+DWeH/TzLdD6l7i/nrk4ee5TfVvSfxOagDxn1H3O?= =?us-ascii?Q?Wh6WnloHWyV9GMF+qvc6fMZx6xsqQ4oDlASZoQJF9NCNFU+YZ+nDxuT/QMuh?= =?us-ascii?Q?H2caO4YfslHZlc9IF7QIjwFCMFaMdssV+Izt/OhD3Nq4cz5JyzbhfyR8hrOa?= =?us-ascii?Q?K3NPst490g7F6M7mVjcM7cd2bdztvPgtmrML5iZhXMwUECG2BoMxPHrPQD2J?= =?us-ascii?Q?hmb6lC49XsvUVbrsJa+f136Mcnyv2BIRxQ3GQk7hkrJuBXZHP+SNQ2Ot6/dv?= =?us-ascii?Q?/aQJaOj5ShRt2lhD8T64/vFjR9gK522XSCIdBjGoPDRbGGKKCixQUONd3vfu?= =?us-ascii?Q?dWzJXWJVk6Nvag2fob1oA146RhNZwMOalDMobQxfBng4rAUklMcOKjbGzKpO?= =?us-ascii?Q?aoVfrVK3NTM0nK0Jb0O9Yf66Vmse0+Z5BNj/A17+wZUIaDLE/FrbDpktNGRn?= =?us-ascii?Q?Owl9nxefL0QxkKaaELK9MBkVUIT5PhTPMDsKe6TAO9umYCwN5LeGo/pOKiFj?= =?us-ascii?Q?KmgmBHMH4KtJnKrhwhJTmew7ZM12oKkCvoqTWd/gAkVMR0hzrMqKYaV1JpPp?= =?us-ascii?Q?n/BlVJcEug0fkYXIi1bDd0SJ6YMOo9orc1xY8XNBOiGtHnPmQXKjaYdRVck/?= =?us-ascii?Q?rTzB+8Ly+OVLFBILLUj6EUL8ginnym4PknX0LMyaNxBOny3S7zz1wLAg1Rru?= =?us-ascii?Q?eynxB28KNXHIEMuEht/cuzSA+FKlwK1S2IZ2iIL70TELg2yEGqeELYZ6LRoo?= =?us-ascii?Q?P36zBMlIjHMrnm5/OTMeLZveb+mNgFCnBg49gD9PI/21z8aXwahn5S291aiQ?= =?us-ascii?Q?VHeqBLjKP3E1T/dvvPoH375r0xutcF+5ky1CqBlUBM+rekt8D+yCWyWVi4WH?= =?us-ascii?Q?x9SRwM/XnSbE91gy7MOuc/fxkFCm5SlIpktFH8epj/1qFwIXRyJxYW+q/YIT?= =?us-ascii?Q?EYSmIGY3zzSkWlu1AY65udPOsn/gKc7ShFNte6/HQwhQaawEWWIiDPQjjZEo?= =?us-ascii?Q?/xdGCM7X7NoWJ8fv3KJgDgDGicoDNiDyP32V9XPI49HJb4IZ/QrBdyhypNt+?= =?us-ascii?Q?REOFkDNDYYrPu9XG8wLhP3FhOZdjj0mBBeP/0IX+GLl1cBvrUzgypk6hZMcv?= =?us-ascii?Q?gg=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?Ussg6HHf/gl5Zp3u1IKZC5E8ZuIwnbTwKehXwriNa1kJuwDQ6D166GaTRhAI?= =?us-ascii?Q?0/MOvexgjuB4FrMdG77QMliB5VuPsIzoBfJzThjnlOrmrbYCQHBdXjhLFk32?= =?us-ascii?Q?duMfoAJ3DLRKY69iTmNZdYFJI+lrXLZyU2xzJ2tBrIqpFJKNpFJe/zoS5uoE?= =?us-ascii?Q?SqLQPXCKqlgYtFs7k3JkZ99u0pHs5k7xuACEQFz/aB3FmWklCqG38/TvQmqM?= =?us-ascii?Q?0ptL6ERFvqDdwinz/h4boL/w8X8wZdKACdh7u1Kr3chIof5PLMc1YuMaYab4?= =?us-ascii?Q?DpS1gWqK2FnVyCCvK4z2eg2vzP4i4iWCTjwIumjm3MW2oDjwVR2jeyc9pDgW?= =?us-ascii?Q?O577C57DlmY0dgbOw1ZxxoKOuaJIKtjqs/sf95Ktoq5rkJqIngPlh1tN52y1?= =?us-ascii?Q?XS0G8sp69ypI6JNPQWUcLZx1odYPTV9f0bBttQrcoT7i6PsmqxqZVqqTigwx?= =?us-ascii?Q?gTSUEhRBYXJegUqqa12LOPWgllEk7rCB1yol2Omo9jzmTFHgeUUWVL2/57iw?= =?us-ascii?Q?QxoWeLSp0lVjVkQNpfDvnGbq7dkUT4var22eamVoh1k7/jBK7uMHpssgpcQu?= =?us-ascii?Q?U1IsoNvNWzDtaOn6ArSfHkDxc6yZrmewWAduGtfEUTr05oAGzwxhGMNJ1BL6?= =?us-ascii?Q?ARaKc0lfe5lt827uhnJO8pK+Ybsi642728bU746OVStm7KPLhr35uVQoH5x3?= =?us-ascii?Q?5Lw//yuF3a8bJ1r0QeTj+l6/H695Oh0xbo4oJwfLlXudJfTUWGX3Uz8nTiS+?= =?us-ascii?Q?Pnj1/QNAcY+iOyGN6tMEwCNVUq/63kKiH/YeAZdAHrwY3FjmjOvF9XB9OtjD?= =?us-ascii?Q?E2yxvnuHP/u33++wH7F56u1uifaqjKTm0++Cj4qveIYbh6G4wzokfRsseeXj?= =?us-ascii?Q?Nfvnez0CEYLEQ+uY73p/M1mI9N9xaBaZg+SPVRn+F/35qWD68UhrgVJh3GBy?= =?us-ascii?Q?Di5luZmclMVimzvNyd7OTnpRp0vkvjcggcSOlQ7GlRXI0tvO1UMciQw+PAS0?= =?us-ascii?Q?fU0A7nuTvQowDoJejO+5OTgGt1fsHBeixSyyWP+B66lq/jhUZ30xnGaZn+Bn?= =?us-ascii?Q?3oAnAy2qpLig0nmovq+A6vUaGfV6MD9Le+6iVk0q7pnij960CggbOSxEqA57?= =?us-ascii?Q?Ju/JFH9Mn7oDsD9B+hxWadW9jD6KxiyHchzH5MjC6AuyPsGua6E418k=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: aedf43d9-4937-4626-f1ef-08dba989edd1 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:07.1748 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Zn+b98msFQF+j124tNb/cX4Xbugrpn3EBgr4FhjLkgZrg2y957FHm9kA2DRM3/jlKIQ3Om83d1FR5KL1BTgFsDHD7Igym/LtM1dgRsTaZH8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 spamscore=0 phishscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-ORIG-GUID: mgOw-Ck1q6pVxnALtk4nl9gdAuKwZjuY X-Proofpoint-GUID: mgOw-Ck1q6pVxnALtk4nl9gdAuKwZjuY Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Clearing and copying of huge-pages is done via process_huge_page() which subsequently calls process_subpage(). This preserves the structural similarities while processing the pages but needs some ugly casting and even uglier indexing: for instance in the call for process_subpage(), we pass the indexed address, along with the index so the handler can also do the indexing. Additionally, both of these paths have diverged since this code was written: for instance clear_subpage() cannot fail but copy_subpage() can fail in copying. Note that there's no runtime cost in having this code be common, as most current compilers inline process_huge_page() (though some older, supported ones do not), but it's unnecessary complexity for something that only has two users. Accordingly, fold process_huge_page() back in its callers. Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- mm/memory.c | 182 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 120 insertions(+), 62 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3854f0b9b3a9..6e005b787608 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5899,66 +5899,6 @@ EXPORT_SYMBOL(__might_fault); #endif =20 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) -/* - * Process all subpages of the specified huge page with the specified - * operation. The target subpage will be processed last to keep its - * cache lines hot. - */ -static inline int process_huge_page( - unsigned long addr_hint, unsigned int pages_per_huge_page, - int (*process_subpage)(unsigned long addr, int idx, void *arg), - void *arg) -{ - int i, n, base, l, ret; - unsigned long addr =3D addr_hint & - ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - - /* Process target subpage last to keep its cache lines hot */ - might_sleep(); - n =3D (addr_hint - addr) / PAGE_SIZE; - if (2 * n <=3D pages_per_huge_page) { - /* If target subpage in first half of huge page */ - base =3D 0; - l =3D n; - /* Process subpages at the end of huge page */ - for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { - cond_resched(); - ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); - if (ret) - return ret; - } - } else { - /* If target subpage in second half of huge page */ - base =3D pages_per_huge_page - 2 * (pages_per_huge_page - n); - l =3D pages_per_huge_page - n; - /* Process subpages at the begin of huge page */ - for (i =3D 0; i < base; i++) { - cond_resched(); - ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); - if (ret) - return ret; - } - } - /* - * Process remaining subpages in left-right-left-right pattern - * towards the target subpage - */ - for (i =3D 0; i < l; i++) { - int left_idx =3D base + i; - int right_idx =3D base + 2 * l - 1 - i; - - cond_resched(); - ret =3D process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); - if (ret) - return ret; - cond_resched(); - ret =3D process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); - if (ret) - return ret; - } - return 0; -} - static void clear_gigantic_page(struct page *page, unsigned long addr, unsigned int pages_per_huge_page) @@ -5982,6 +5922,65 @@ static int clear_subpage(unsigned long addr, int idx= , void *arg) return 0; } =20 +/* + * Clear subpages of the specified huge page. The target subpage will be + * processed last to keep its cache lines hot. + */ +static int __clear_huge_page( + unsigned long addr_hint, unsigned int pages_per_huge_page, + int (*process_subpage)(unsigned long addr, int idx, void *arg), + void *arg) +{ + int i, n, base, l, ret; + unsigned long addr =3D addr_hint & + ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + + /* Process target subpage last to keep its cache lines hot */ + might_sleep(); + n =3D (addr_hint - addr) / PAGE_SIZE; + if (2 * n <=3D pages_per_huge_page) { + /* If target subpage in first half of huge page */ + base =3D 0; + l =3D n; + /* Process subpages at the end of huge page */ + for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { + cond_resched(); + ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); + if (ret) + return ret; + } + } else { + /* If target subpage in second half of huge page */ + base =3D pages_per_huge_page - 2 * (pages_per_huge_page - n); + l =3D pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ + for (i =3D 0; i < base; i++) { + cond_resched(); + ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); + if (ret) + return ret; + } + } + /* + * Process remaining subpages in left-right-left-right pattern + * towards the target subpage + */ + for (i =3D 0; i < l; i++) { + int left_idx =3D base + i; + int right_idx =3D base + 2 * l - 1 - i; + + cond_resched(); + ret =3D process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); + if (ret) + return ret; + cond_resched(); + ret =3D process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); + if (ret) + return ret; + } + return 0; +} + __weak void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page) @@ -5994,7 +5993,7 @@ __weak void clear_huge_page(struct page *page, return; } =20 - process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); + __clear_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); } =20 static int copy_user_gigantic_page(struct folio *dst, struct folio *src, @@ -6038,6 +6037,65 @@ static int copy_subpage(unsigned long addr, int idx,= void *arg) return 0; } =20 +/* + * Copy subpages of the specified huge page. The target subpage will be + * processed last to keep its cache lines hot. + */ +static int __copy_huge_page( + unsigned long addr_hint, unsigned int pages_per_huge_page, + int (*process_subpage)(unsigned long addr, int idx, void *arg), + void *arg) +{ + int i, n, base, l, ret; + unsigned long addr =3D addr_hint & + ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + + /* Process target subpage last to keep its cache lines hot */ + might_sleep(); + n =3D (addr_hint - addr) / PAGE_SIZE; + if (2 * n <=3D pages_per_huge_page) { + /* If target subpage in first half of huge page */ + base =3D 0; + l =3D n; + /* Process subpages at the end of huge page */ + for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { + cond_resched(); + ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); + if (ret) + return ret; + } + } else { + /* If target subpage in second half of huge page */ + base =3D pages_per_huge_page - 2 * (pages_per_huge_page - n); + l =3D pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ + for (i =3D 0; i < base; i++) { + cond_resched(); + ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); + if (ret) + return ret; + } + } + /* + * Process remaining subpages in left-right-left-right pattern + * towards the target subpage + */ + for (i =3D 0; i < l; i++) { + int left_idx =3D base + i; + int right_idx =3D base + 2 * l - 1 - i; + + cond_resched(); + ret =3D process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); + if (ret) + return ret; + cond_resched(); + ret =3D process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); + if (ret) + return ret; + } + return 0; +} + int copy_user_large_folio(struct folio *dst, struct folio *src, unsigned long addr_hint, struct vm_area_struct *vma) { @@ -6054,7 +6112,7 @@ int copy_user_large_folio(struct folio *dst, struct f= olio *src, return copy_user_gigantic_page(dst, src, addr, vma, pages_per_huge_page); =20 - return process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &a= rg); + return __copy_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &ar= g); } =20 long copy_folio_from_user(struct folio *dst_folio, --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AA36C83F01 for ; Wed, 30 Aug 2023 21:22:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241159AbjH3VWh (ORCPT ); Wed, 30 Aug 2023 17:22:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241104AbjH3VWf (ORCPT ); Wed, 30 Aug 2023 17:22:35 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A2DFBE for ; Wed, 30 Aug 2023 14:22:00 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInXRW032362; Wed, 30 Aug 2023 18:50:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=hk33vgDddTabMGLcd8V8QAslNep2bYD+vzm1FUsH/n4=; b=SpCEOU6TcEc269289HEcDZgbfCF0wHUliDxcFSHkDFlyd7Y5oYH3GCvMyESj5adOHxeY j1BG/6+XBhYIOzs555TmkkxrNE4D2FacYvKnjPXx6UkUtTC87KvPob1xYh2NY5yfDEMJ Jmit+GgkqcXnqY5gJcofivbjI4ldMa0OH8a/uFXvTeru5dtnkpmWqOvR/oK7AhJFj9lb oLVWmLoqiJY7PmLpQO1ciq0sitW4e9IeyIazC7QM3WVGh2i38F5U72Ffx/k8xwWWkv/k pLkhVMMcBfnK7GxFbyq8LVkq9c/CoMpejtz7ZinsL5d5TraD+SDxwgI7adyqe2BNj/Ok Jw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9k686b8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:14 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UIJHaC009143; Wed, 30 Aug 2023 18:50:13 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3ssepyapk3-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q+6aOJP5dhQzl2BPX33TzPwwIAJintGXJgBxkWTDEV5XYkzY0DBGU1GaLPRwnEBE8egfuI2WecZ6XdWp9QLURnbFljetgNWb24lP2QDZTeYImyAnIoA4HERNHLjrxHo/wEJ84rr1iugvDiG9KN1v4PU6UcpjicxdxhRKlPZBkb3q4jfgCO4Kpe8rQGxhV50MOrexPgv4RqEigVYnuoqEmn9QnFnn54oataDhgpvLrv4eqzRkuMleSXmu23H1FFgltX9fqOmzzQ2LbiVk8aii95e5c3pXKP28/V91HhENUGxf1XrWfqqDDOwNDrp+8kxMVwdfgOUP3ilVVc4kDt6Jpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hk33vgDddTabMGLcd8V8QAslNep2bYD+vzm1FUsH/n4=; b=lCQLaB1h1fsZHcvJ1ASeshvGHOXknb34UftQbWrFcLx+bg1hu5hdwl34e6MXrCxhy9A2qQ5S3f9WrZHsVKmaslF8i8B4iUWi06p+2c2mofPzEgbN/AsNjHFVgG+ky1BbROq/BOR5mGKpIu52ow9h08Z3eeSszVE+Ut3m+lbGtc/q+Nqulb7ICqEGr1xYboZ58N2uOqTbCzgZ25hdzdUAuAwC4GTC0ouqOm30UCI+TquKNTtoEiMDkti3u86iGZWVWr5eYU9D9W+T0A0UmKbntXZNDdihdYhl4C+YHTN9qqoJLft0NUx0d0GXDxQag3myjCTA7HNJsDogUDU2i5long== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hk33vgDddTabMGLcd8V8QAslNep2bYD+vzm1FUsH/n4=; b=x2TmEqvpQfG5gThHqUWqHmUL1c9FNdzao25uNhKTZjZrzEDLG/KZHOSFpG5zqZDThgm3TdtqAnGE3scotGHzb4L0tnisXIx8WJi39DL3WMohRw30za0viCtX0pXXma7Bto2TzVet4k1RAOsffJ2K5FhvMqSBOFmiGqEEdDL3X3o= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:10 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:10 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora Subject: [PATCH v2 3/9] mm/huge_page: cleanup clear_/copy_subpage() Date: Wed, 30 Aug 2023 11:49:52 -0700 Message-Id: <20230830184958.2333078-4-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ2PR07CA0012.namprd07.prod.outlook.com (2603:10b6:a03:505::13) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c370bb3-20f2-41f2-ca1d-08dba989efa2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: a8CoWdwka9GKhJuUJpgCFHLAEUgkPTGWZKcdHZvoe9LaHTSqTCjzt2r+exZ6zBCnZBMeQaizSduoDNgLv7bbBbNopT2Byr1z9NKjzhYWDEpFA3DxanpBO7HjtQwPtWVS84T8EpaXoqUkMsmEYvhQYrmLItF9bjaniDoxEqHj5lo8D5P+5DDa5CjR7mtXJIjAOrThrY/EOnBvxkMeo7O1aN15a9blR6Ohp3nQzXbpWHfmVPRlz6U91ZWddu+hR5/zVYz5lSvWcvoHJr62qZjWipJ+7RRbKsA5CjsWJ/rgJhVc1xZNQaKhzggRKKUHBqIeQQbGKshdlv4TEi2HbTMOp6c2Nd66iOEYjp6ormEmXAYchWhTeLbtS6Ivj0UT1pCt/BnBpynnpLuNTj/aRlawSyIAB6qPsSNZOj8ZXxMhxaoMggKi7G4XtpLRn63Fv++JX6XX7CzQ/E3jGNg90MlMDypmtHOu6ZUjkC6/xHHel5ph1AgIkCT3vfJc1G3TaikXGV8riwRPckAQugtDoErTImgQAa3QycuILPCjufoeD65qho6WsBqlkZNm3YqC9fLd4IrcGXGbCggLPq6TTdvsMvuGrCfFd2DreTlcbqFTZVc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(107886003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3D3u1xCvM76a6dgzs/4GfgHr7ZTkz28EjRo6dySGNF1tfPBndiGsPYLiYvy1?= =?us-ascii?Q?m9n639nfWgRCs4K2yhDJFKg0PJfxpmPVXMNBew0Vthw5HVqmAaOp7i3JJZND?= =?us-ascii?Q?Rm5OLAdolre/QgouryquaYdaCQK0y5TJ78GwQc2NMVT/wVATNFdcOPY5gf34?= =?us-ascii?Q?H0hYPWZJiOvNuMP8r96RLn08iZFpSegDwrEsZrSWCmofgRGc+Eq0OyCesKWb?= =?us-ascii?Q?2cayiNKYLOUk50TX30fZ9z3aTWWRXtV/F0ip3BYCqbcwW2XM41XD6vqPvPRq?= =?us-ascii?Q?wMdjIlXQDBTWDH4bT8G6yPz+6v+yvG+SQoPsMI2geBL7+q7GYCC4KMKksSz7?= =?us-ascii?Q?u4i/9byzVWxjgtq2R8i/jW0L4Fjqufq+ONFSLszcd+RKWrFUS/kzpFwY/Z8v?= =?us-ascii?Q?EiCo1QL2xqjurfVPkohQYPaCHW+uECKjU3c5BwkUYRqbcfd5PGq5wurKW9sc?= =?us-ascii?Q?+e8+V50VNFx/gHySuJAlRtPgEMu0DpnqAMEYfJkOfjjRg0g6q3rQEku4E597?= =?us-ascii?Q?RqNioP11YPWaskoPfI3KH/LSLHGj9CPMERhjbZ1vWkNzoDChXt+O/xF0Ag8/?= =?us-ascii?Q?DRbs4Eb/keWIvK3x3EF/NQHa6Gg2PT+NiXztWSFBMkALHJseoimUlRayCc+R?= =?us-ascii?Q?pcs2N2D+2WI4hg9/SyEyFm3M5/GxVhG/ctRQvc9eANTjt7OOpaKRGlCSf2K3?= =?us-ascii?Q?uVMl6asWsSNT69UbY95HuCqW5ExTNP9wi9LgRZwh6YK+km2UR2aKaa+9aSVL?= =?us-ascii?Q?7nhuz0brrHWVMUX9/AargbOAz4qxwNjOTs3o4CWkJqt5FkmZQTPPIDKnM1ZA?= =?us-ascii?Q?c6Hrwc2bV9EZQTgi/gdGDyw982XrEZVu361tAXtH1HtemtcDioOK3BVFXf91?= =?us-ascii?Q?vbxIjlwg7jOaGed0YNMnsVbvVgsWdtrOfMa6230ItERfuKf2cRyMOVtUev7y?= =?us-ascii?Q?GuoNznF45RaqBWeq3w4WHhvwK8COTOjLlP0490YRy0ANxWJo24AbF6n6SFSe?= =?us-ascii?Q?L3dqyTJFKQGKzhuI5ZvA6eImGIZp4qZWCB8sE6mpxoVN0BtA2jzoNi7mDPKl?= =?us-ascii?Q?9vt+Iv2AFt/gb3I5to3FS8uJ7+jjvRMCNNIqM9BLQUfHF+4GTKpfJ2WVWTFg?= =?us-ascii?Q?kKxnZ5ugVMbsvTvdU+JXC0Rapc+RGSY2ZgeWR0cyYp7dYQsN1YImdy+QQ2wh?= =?us-ascii?Q?tmdiXGiYx/QdljQh4D3Ht2WeibX6ioggfPuTcjfDYJZN//KFZ1/YjO1+wXFN?= =?us-ascii?Q?VTVHAv51zkeLOsdR3gIduz6UXbuNyO8imHT9ceZ6QOXWX8DdsIHOQhDEG8B8?= =?us-ascii?Q?U3c8jrrKEmS0Px2cGLNLXmjcHdnDijbOSCXtOnWeAX7pl3+NQ6saa8qKVWrW?= =?us-ascii?Q?HRsTAF50RFmQ44Lf8SIu+XO1fJDpxEztYlkVupPR4VsgnHr0wSHoh3z3knt8?= =?us-ascii?Q?M/SEbPl9FOHE11yNYM3ZCrfZw/YJsqY1rB+PocGNKswH0SsBYUrj2xqgE22a?= =?us-ascii?Q?UYjohxg8UeCQxBQZtWopjojOPhEaBb8xhOEp4RyNR7LJDiMHypKKEqvrgVN5?= =?us-ascii?Q?Xoh6T5NrHd1ItMjbdFtsl0SrMBMasC/SQjdvKPCE/prRewPw1kELRmPn0NT+?= =?us-ascii?Q?8g=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?NvKzZ7L4jWm73LZS93EiXtnkrGxcD264ImOID4rNPFPbLjxYIs1fqaEWXMnZ?= =?us-ascii?Q?yGDZ5v/11C/2CiRyZhcVG+5c5JY2D4lgT2zFsKdaeS/WQvg8s7YKwihYcLWc?= =?us-ascii?Q?Uy0Sg5rNqfjfqPAJTK9t1GL3aYzCafvVcKtAPKixdBjkm2zMwRYa37msAGGU?= =?us-ascii?Q?l0OkXAYVUWatlQ5j0+geoTSlb/rO79lw9oHSFCo98jLcpffGF7oA6S4+BpE0?= =?us-ascii?Q?fMj1piKBF/1YjVSU3YvZNCEh75PyGN26KISXa7ktotXRiZ9+umCX6jjBenDE?= =?us-ascii?Q?waqpU0t5VauneN80C/ZgG9NpeBPMPhxFnvjrLJ31yu57fAgwfz7/UC3qFhqP?= =?us-ascii?Q?h1rvvNb3lxT2qUeyYU/FXFxBG+SJfwdHUmnKnaYuxzVaKrSbcEULUn63GOH+?= =?us-ascii?Q?wz0ethYUs8/xGivCvkNzVNGsPwXOLolUWv9DcTlEajKqibW9MqTjzEROvI4W?= =?us-ascii?Q?7Fcr2EHES32h2hIZJvWxTHURlOVaKbqFfri024x9HT8e6MEXvx8NEBFKPOml?= =?us-ascii?Q?3ITTHINjWoQt45c57DaimZvt2sk5RgYIxm10eFKsNwG0YwRcHdn4y+l7srrK?= =?us-ascii?Q?d4ND+kxo8MeCK6/matId9HWRu6eAGFoYMOKKL1zx1rwkhDletV6iE6y5JeoW?= =?us-ascii?Q?b/ZRukOgOlwAte0HLV4sEYYQVOrV8ADG1WETo0aqpj3t84lgjOiSpoKezEbT?= =?us-ascii?Q?ZlncYTcBh7Kz6z1Zidjrka96PECjIvs3bY2Zs7G+E+WHxZALAO+p3OqNsv4t?= =?us-ascii?Q?sGm74Dpfz8IOoEsRqTK+tA7uq15vQuVDqPVryFJ762ZVkcIzPMCYvJzvlj1R?= =?us-ascii?Q?zT/o5gvoBsvjMe58QftMAlhx9QXhN8dPKNgbaKVe/VpuvMkuxIsvsdtpdLH9?= =?us-ascii?Q?r/VaWLxF7w3nqHlsW0UurRZQyaC9W1xWqLvJJ+9c89v6lNZn0O7w8/he75y+?= =?us-ascii?Q?n5q44XAaOG2eU3r33EQuo7qIKi721X36ZWaREJcBDOKJ5knAc7tfu9fqFARE?= =?us-ascii?Q?aFB7IHeVGGUrUF58Qs0l6up2c3B473nhZZuyEtANlf1iGfW47SmCmD/l5W7v?= =?us-ascii?Q?Y0IheLKQhOczj9+sCSrg2XKbPeQUZic0Rnlm4RGr7wfamKnexHmgVYzoJUcA?= =?us-ascii?Q?DlnlhsP/0YKyu4Fum/brmyAMC13344kWVlW0aglbKjVILLoackoIMww=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7c370bb3-20f2-41f2-ca1d-08dba989efa2 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:10.2225 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YBjRkwE0VI9QKHvgvP4zzjSVXNqe9GBGBlJkEhr3zAy/tG79Owh+CgmVQRzKTzgtWPN3C+S+YGcPFjZiJBHplsXuIzprukCHWWNlCo4Y77Y= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 spamscore=0 phishscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-GUID: oNAlS3PA7YW6O0NhswrsjxO3qsP6U2eL X-Proofpoint-ORIG-GUID: oNAlS3PA7YW6O0NhswrsjxO3qsP6U2eL Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear_subpage(), copy_subpage(): static int clear_subpage(unsigned long addr, int idx, void *arg) static int copy_subpage(unsigned long addr, int idx, void *arg) take as parameters: an index, a post indexing virtual address, and a struct page * (clear_subpage()), or an opaque structure (copy_subpage()), both of which we would then resolve using the index. Fold clear_subpage() back into the caller and cleanup the indexing for copy_subpage(). Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- mm/memory.c | 58 +++++++++++++++++------------------------------------ 1 file changed, 18 insertions(+), 40 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6e005b787608..4f1ce3ad0af5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5914,24 +5914,14 @@ static void clear_gigantic_page(struct page *page, } } =20 -static int clear_subpage(unsigned long addr, int idx, void *arg) -{ - struct page *page =3D arg; - - clear_user_highpage(page + idx, addr); - return 0; -} - /* * Clear subpages of the specified huge page. The target subpage will be * processed last to keep its cache lines hot. */ -static int __clear_huge_page( - unsigned long addr_hint, unsigned int pages_per_huge_page, - int (*process_subpage)(unsigned long addr, int idx, void *arg), - void *arg) +static void __clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) { - int i, n, base, l, ret; + int i, n, base, l; unsigned long addr =3D addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); =20 @@ -5945,9 +5935,7 @@ static int __clear_huge_page( /* Process subpages at the end of huge page */ for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { cond_resched(); - ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); - if (ret) - return ret; + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } else { /* If target subpage in second half of huge page */ @@ -5956,9 +5944,7 @@ static int __clear_huge_page( /* Process subpages at the begin of huge page */ for (i =3D 0; i < base; i++) { cond_resched(); - ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); - if (ret) - return ret; + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } /* @@ -5970,15 +5956,11 @@ static int __clear_huge_page( int right_idx =3D base + 2 * l - 1 - i; =20 cond_resched(); - ret =3D process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); - if (ret) - return ret; + clear_user_highpage(page + left_idx, addr + left_idx * PAGE_SIZE); + cond_resched(); - ret =3D process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); - if (ret) - return ret; + clear_user_highpage(page + right_idx, addr + right_idx * PAGE_SIZE); } - return 0; } =20 __weak void clear_huge_page(struct page *page, @@ -5993,7 +5975,7 @@ __weak void clear_huge_page(struct page *page, return; } =20 - __clear_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); + __clear_huge_page(page, addr_hint, pages_per_huge_page); } =20 static int copy_user_gigantic_page(struct folio *dst, struct folio *src, @@ -6025,12 +6007,10 @@ struct copy_subpage_arg { struct vm_area_struct *vma; }; =20 -static int copy_subpage(unsigned long addr, int idx, void *arg) +static int copy_subpage(struct copy_subpage_arg *copy_arg, unsigned long a= ddr, int idx) { - struct copy_subpage_arg *copy_arg =3D arg; - if (copy_mc_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, - addr, copy_arg->vma)) { + addr + idx * PAGE_SIZE, copy_arg->vma)) { memory_failure_queue(page_to_pfn(copy_arg->src + idx), 0); return -EHWPOISON; } @@ -6041,10 +6021,8 @@ static int copy_subpage(unsigned long addr, int idx,= void *arg) * Copy subpages of the specified huge page. The target subpage will be * processed last to keep its cache lines hot. */ -static int __copy_huge_page( - unsigned long addr_hint, unsigned int pages_per_huge_page, - int (*process_subpage)(unsigned long addr, int idx, void *arg), - void *arg) +static int __copy_huge_page(struct copy_subpage_arg *arg, + unsigned long addr_hint, unsigned int pages_per_huge_page) { int i, n, base, l, ret; unsigned long addr =3D addr_hint & @@ -6060,7 +6038,7 @@ static int __copy_huge_page( /* Process subpages at the end of huge page */ for (i =3D pages_per_huge_page - 1; i >=3D 2 * n; i--) { cond_resched(); - ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); + ret =3D copy_subpage(arg, addr, i); if (ret) return ret; } @@ -6071,7 +6049,7 @@ static int __copy_huge_page( /* Process subpages at the begin of huge page */ for (i =3D 0; i < base; i++) { cond_resched(); - ret =3D process_subpage(addr + i * PAGE_SIZE, i, arg); + ret =3D copy_subpage(arg, addr, i); if (ret) return ret; } @@ -6085,11 +6063,11 @@ static int __copy_huge_page( int right_idx =3D base + 2 * l - 1 - i; =20 cond_resched(); - ret =3D process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); + ret =3D copy_subpage(arg, addr, left_idx); if (ret) return ret; cond_resched(); - ret =3D process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); + ret =3D copy_subpage(arg, addr, right_idx); if (ret) return ret; } @@ -6112,7 +6090,7 @@ int copy_user_large_folio(struct folio *dst, struct f= olio *src, return copy_user_gigantic_page(dst, src, addr, vma, pages_per_huge_page); =20 - return __copy_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &ar= g); + return __copy_huge_page(&arg, addr_hint, pages_per_huge_page); } =20 long copy_folio_from_user(struct folio *dst_folio, --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20F28C6FA8F for ; Wed, 30 Aug 2023 20:47:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239519AbjH3Urr (ORCPT ); Wed, 30 Aug 2023 16:47:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239981AbjH3Urn (ORCPT ); Wed, 30 Aug 2023 16:47:43 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA3E2E57 for ; Wed, 30 Aug 2023 11:52:40 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInrRw018041; Wed, 30 Aug 2023 18:50:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=nJloHB0zDu8LwgV39mz1tQFxCkCRoHiIuVP6I6CUHrM=; b=utW3nYS2w74/gf46FinDK3cutipHI2RgnpVqYOY+yr9oyWS5FMr4lLHsqCmAc/+jz2Pi 4Lk7psHGTVwrbJgKre76kWpUybe30K4hCZAxjMDp9lwozd9OOdpSYc/FhSLD5sIDqx0V 2Xebu6m255owgerAEkspCV2NO+V/r8Zoyb9YMqtbwYkZNZK1D4CDtru3+gi6yynVOC4F AGH742rkLmiSABi1k7fS2geLM3ThcxjvBj0ERZb5myB8NU900CtULRnJsJb3mmQCntAK xxTIzxSx56NB458CEqQDt2c3eVRqxSpaG0r79SOqlZG5+PRYLmeSRHPRUq+EZBnqoACX ew== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9j4g75r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:17 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UI0boG000447; Wed, 30 Aug 2023 18:50:16 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sr6gcyw3h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RhFpn+d6a/o/HZX341TRrp8HGqXTvGbnMX7Tix2aLT49CleuzbDxvyptntIJo9bZJ/2lVAthMG0VZ30XWAeukOC2UeC6HzL5KmGjDlmPqhphuSgL/S+n3Y9BI/pfffrsaee8XN7f9gc6mo2hTLTIxDXreuFPm3pYNsCh5b/U4Z/4/3rJUS34CaFk+wc4f34etrPHQpLNNK+/gCa14aQ4lhp+aIO8+2F3yFIoGjDwz33FgcnMNgcpepJ5bUrGiyRUfYF9Of95rbujode8admDR3l4i2Blk1dVzuHVdq7V8stIN/FBY3mCfObWjgK8PdT96ucQyNeY2aGbRjlEjbxwWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nJloHB0zDu8LwgV39mz1tQFxCkCRoHiIuVP6I6CUHrM=; b=dvL1cBjpMoD/WdCK20r/ZBrBQXj200DfwuxgYXMBttrNeWbOLQ3liFDlsxTWe1Qv9DjLYytcC94WP7BiTtLVZOd6RZsdjC0OGeWw9YFbDU+KXZ0SP1f+givpeUFmLN0/o5MlLG6gZoAX/ntXyrqi5SccPe/LCKm0wwOScQ/kkF5nR0USnO4IKvHlUQ0+Xmy/adW3fHESlEmpUGNjgl2aQmJFBQcOqw+0gS+r9CMAUwWyG8XXitZi9e3RPyJ8KscTItQ+2+Yh0AOqGfe1gXC1NZM6UkitJDQE0IG0N2Dl1AH65dsqucGygPTVA4sNvP/me/JRekT3IwEHDb+N7QoTrA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nJloHB0zDu8LwgV39mz1tQFxCkCRoHiIuVP6I6CUHrM=; b=szWPjUYGjkD7KLZalDmBB/CiCIeU6aR5dw+f3SGPm4c0CWssdaDcM9h7Gwy44jrDbr5qgt9+7LmqOMH+KXplbguy+AKta2bEM7VzEvyREHYTAXp78qrm1XmehHSOJ9lQE5BfKkYsNQSUZA/EkSuGJhgp8WuiTmSOoFbIdThInjo= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:13 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:13 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora Subject: [PATCH v2 4/9] x86/clear_page: extend clear_page*() for multi-page clearing Date: Wed, 30 Aug 2023 11:49:53 -0700 Message-Id: <20230830184958.2333078-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0025.namprd03.prod.outlook.com (2603:10b6:303:8f::30) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: f0d48c7a-c6ca-4080-3991-08dba989f190 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tWHw1Bzleq3VLt57kDBz6HOlx1Yj+hbmhix/327oZvg6ojP4A1BjeS90W+0IU+SmBdf4fRl29Who9GnxkL0oQDPEDSbAaOtvrjO6eY3195797WawwAk35zbAPVGCGtRJIZmbcqyPYtI0xpJwgVfISDF3oefcOE+MmgJmxJH9y7o2lGKfvI9i9oOCCBz83Gr/AgWftGolZQDv7ls9FhWmX74FFRiOnQ7f3cZhY5GR9q0+kjgkQ9xrmE90oCcSwAd+XjmJIe4ZRlaXHTBtbIbT8zriqgCKGkCDL4R+KxKThSXrxT7Q+qQrP2I8fEl/dt7kcvufTPIOPse+tpztACqvlvjJIcetj7T3xTwNMfnjbDLH2pyn3tbOZ1EAJByRlMClDFHSGNmdO+zpK/xXRmoBAAAjcRjI/kxKrUgxLhJ6YxWKo5iZCch/abKPm7YxQ1wew9uEmYyAmKB7bpErdTu19+jgppPXlFTHTiyf/BfvTdxNHjLLG0yq239CLfq6QtjTlT91hIbjZYQq8CKmOd3TGYhg7YocNbzNN2mvdmuQ/Lu6trhxWjlx8+cxQOh+FRM6 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(107886003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?MNGn0hz5IREwEpfgI5QtLSKwZJgZ7WZbc6CtXAeKh9zn7jqpoLbCzXvxJIQu?= =?us-ascii?Q?hEIGnSgKGNuK+fzoSUWVSipnV2OTX6GDVSdYbPjKmpJp/E0g+KJAgb/8rOxh?= =?us-ascii?Q?Nq5uxQehaGqDkmzPFy8y7/louzlS0NUFensDAE86xU1bmnhn1Qq7C4SxBfZO?= =?us-ascii?Q?CuuofGmYzz35GQlYTrV101KvR/DN19o/CbHLroAeu5TDh2z+XmtbQNxbPpxL?= =?us-ascii?Q?5o4cAPqZz15BRxEK7dnkbfk2UgrIJ3HxzkvbuGWPOOwbeydJtqXNBxFQ+orN?= =?us-ascii?Q?OMISW9fv5X81lFsGX5BObpmjbHU3BNtAL8VFTCCXuMGwVbzQFH7rKroKSPtg?= =?us-ascii?Q?MICJgI+2i3hnRPK+1ide2ChCorb5eY/+W/HaoOV6Q3jeflmtw++W9CBKyEKm?= =?us-ascii?Q?NY4FXLwCeKSvHmi9VQ+/32eofGYGeZtjYMgKkIug0kADbYpEPQUUutKwZ2zG?= =?us-ascii?Q?LkUvWL7AThKxV8vtTfhIr6GEXI1oztUkQBoGJCKmP1FFmtOwkx5AGLukbS1J?= =?us-ascii?Q?M5bCe4Z62qcAhyfwAyWo76abt1zqp7fL0L2eCVvZj0LUBX6sajKL+XFEHt/B?= =?us-ascii?Q?7zkOxzTA9MP6tN4/ycumffGpTsr1jg/GrZUARtLCMDOImqBuLVYOUIm2Rnsg?= =?us-ascii?Q?HnN76OmcYasda8XICYbiXdj/n0oNbIJwTxps2wmMRWbc/8WFBwgcc5DzwTGP?= =?us-ascii?Q?NUygLYfi8ssSWp6canJgNGMK35Tkclf2NMevWgPnBW9cuXCqOjNpjF9/7/I+?= =?us-ascii?Q?WmtaQLhkrMF5bQTC2XMJtxZHH23eW2cdsgUkwuB6RGunWg0v5sryb1rIabgQ?= =?us-ascii?Q?A1ggI19bBPYqbFQ8GXMg9WD4T+PzQ6renq5NtXDgr6xvdrt3/xZ5x1Ag3wsL?= =?us-ascii?Q?UA/R4ddId+4FK0+MmGBxkVKGVV95ElJJwr+ThUB7wic7Axw/icMyYo5sCgG3?= =?us-ascii?Q?5MdDV4BO86FLeuHnyBPhuYh9QME7xcheUsRw4gIl4Q1X1JtA+Po3Ywk78Y6J?= =?us-ascii?Q?3oFAcb1p+/oSJuM0wNZy9yERsqLtz0gXiLYlEwXEe5cc2dk4HzS/2TmIlgUl?= =?us-ascii?Q?i90tVMeixCWIJiLL6qUaSstekMJi2k9/D0SNbLdFySYjH0zTmi/4gO0czfh1?= =?us-ascii?Q?VDD7Mrr6vfj+LFqckm9SFngqvpaIjRr5DzL5XTxfkud2yHPb6DjKp5sXoCqk?= =?us-ascii?Q?uZHTxzS7PXZv+ala57oxc4Xnel9LL6Im5joH9xwqpQHkI+wvdPWXcwGIcQyM?= =?us-ascii?Q?tgEbVX1ylqlpBvEHUHI+YTzV6fuQMWsLPVKyFdp8ZT7aTsMgj+gwr7KU8QpH?= =?us-ascii?Q?/pYxXd/4r65/ZPZXiVTFy1V8NdFOC3gsnzHz8UQIug3hAosYaUludZyxr4aQ?= =?us-ascii?Q?Ks2gkbFWYkeJvVvj6FC3ZcT5UkaZ7GfGqGLLWis7UqGbMEgHur44EGs1CydK?= =?us-ascii?Q?vlpT7oKyzaXiRK85ZueNCgj/ys5t8OOmDplsUEbFQiq7fCd9IQBxvn788ebC?= =?us-ascii?Q?qUzONoY3inONMJ/bV5sv+o3hrlsHK1xNhw0Tykmuat9VF5XyCNXX5iiGYUqo?= =?us-ascii?Q?uCqMPvDjOcXlXIwpe1GuNaxWJ9wmtUJIxEhIykWj/Dm3HzRpZl6d6d/Ft8qG?= =?us-ascii?Q?0Q=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?lb1Lk5rZslOEUWrpZyof+mbkzXtSnlzf0/D4MozEDrvZSUKwg6xOPeT+XRfn?= =?us-ascii?Q?kdDidQ4rd2JmGlCoiniopkywOuVowUjIUkVb6BsMxxKgDvHS4iqj3wjT2SLU?= =?us-ascii?Q?bH5m5JHGJMVc2RkLYjw/VmlOJcGi95UOHDPFPlZJGtCYwb5I8DOw+OzDzOIz?= =?us-ascii?Q?ldapmiPIlo42zME6tbLwen9P5Q7EH6ycJ7FQ2zMnL2MpB8Z5+bamXTG0c0ML?= =?us-ascii?Q?sh1S8T9Y4XOi/sezU/loHkEWNfthcd2zao4SLcekzXxJqlmVuRBVQVBoNgMG?= =?us-ascii?Q?Fa2FsDI725WrNUE40C21uc4AAlUzzkEFReQE8rTvl6DjgHm7rYxZ3AdLfVQx?= =?us-ascii?Q?P5iZGMhdoN+fVJMmHglXmSokOJyEFGgcwyaFh1tyfEZ6CezIXSk+8qdDrEtH?= =?us-ascii?Q?aDIKnDWW67TO/feAbbuJdconfA86Hrklzc5t9vhenox+zDhUzdo0qLlF7AW6?= =?us-ascii?Q?JIUwrGAetxhuxVFn5BmYsGJpXrzhb1QVQ/n+DxepRgB5jWZwuvbnQ7SDSulv?= =?us-ascii?Q?fBNcfLkjr9rHmh6SxojM4oTpql6bQ/VmFO2Q63ezG3GfQx3pIUQviZ9x1UNI?= =?us-ascii?Q?rGOYz9Yh9vQTr5FCcEQEpcIudTJdsQWyDyW7r8cTAhjVsd0X72nAgsJUlLlY?= =?us-ascii?Q?y6/sxUNQktNuNJbXwqtWJ/sbWLqL0Q/+zv1ve7R7P3QZ46vAgTRccZ75CjzM?= =?us-ascii?Q?d6OXrxayp8PD455xWcvN3kGIqTsKidlz2MA85I5yAEY/eslCFA3Mi8BYfif3?= =?us-ascii?Q?IaI61Yp0zyuwR+NbfhG7dI96X5iTRRsdt9Az/MAdp7HfwzMU/PdgpR8WeLKe?= =?us-ascii?Q?JK9k2uA1WZoKsTO5TtprYgcD9mOX9rIjz93t+5LhuPRXcxze8UhyC5fA9FGk?= =?us-ascii?Q?2WxfAaAYaLiEKLXX6xkXrQJUzNheXA6dqwb00PZtQrM/x7ojhVoDb5DtNWJ0?= =?us-ascii?Q?T2QscCncXELyhhdw1vJnPGR+x69Gzg8oljzp0TUml2axrtZqyc1SXwC3GJ4C?= =?us-ascii?Q?GtdGRx7tT8XoxCAk7V8uB8L8/VZHt7Ynocji4X0dkyRTqLv7hR/8SZSrDqNK?= =?us-ascii?Q?4ir7PhquK9KneVSLyBHSz1th5tpQaFhFKF60PHpqo7QrsKc5z/mFOA9KagO5?= =?us-ascii?Q?0rV1EScxcWLCKtVEM0b7UKqg1r1jGREf2mo7BitqXzRqt0tYhkhLbkQ=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f0d48c7a-c6ca-4080-3991-08dba989f190 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:13.4385 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XCjLtkeS/3muNw6Sgm8Uy4PcauxCtioNilkUhMP9Ko0i24T1BXzGpDp5nxREY8oamPHbjZ5QZBLuvuMBZYAJNjJSirzMupFXjhcaC4YrHMI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 adultscore=0 bulkscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-GUID: CPje_lIh5Rw1YeWA3jtGvcbAznD8HBQm X-Proofpoint-ORIG-GUID: CPje_lIh5Rw1YeWA3jtGvcbAznD8HBQm Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear_page*() variants now take a page-aligned length parameter and clear the whole region. Rename to clear_pages*(). Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/include/asm/page_64.h | 20 +++++++------ arch/x86/lib/clear_page_64.S | 52 +++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index cc6b8e087192..56f9adf0e892 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -39,22 +39,24 @@ extern unsigned long __phys_addr_symbol(unsigned long); =20 #define __phys_reloc_hide(x) (x) =20 -void clear_page_orig(void *page); -void clear_page_rep(void *page); -void clear_page_erms(void *page); +void clear_pages_orig(void *page, unsigned int length); +void clear_pages_rep(void *page, unsigned int length); +void clear_pages_erms(void *page, unsigned int length); =20 static inline void clear_page(void *page) { + unsigned int length =3D PAGE_SIZE; /* - * Clean up KMSAN metadata for the page being cleared. The assembly call + * Clean up KMSAN metadata for the pages being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. */ - kmsan_unpoison_memory(page, PAGE_SIZE); - alternative_call_2(clear_page_orig, - clear_page_rep, X86_FEATURE_REP_GOOD, - clear_page_erms, X86_FEATURE_ERMS, + kmsan_unpoison_memory(page, length); + + alternative_call_2(clear_pages_orig, + clear_pages_rep, X86_FEATURE_REP_GOOD, + clear_pages_erms, X86_FEATURE_ERMS, "=3DD" (page), - "0" (page) + "0" (page), "S" (length) : "cc", "memory", "rax", "rcx"); } =20 diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index f74a3e704a1c..58cb1d6355a7 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -11,20 +11,35 @@ */ =20 /* - * Zero a page. - * %rdi - page + * Zero kernel page aligned region. + * + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx */ -SYM_FUNC_START(clear_page_rep) - movl $4096/8,%ecx +SYM_FUNC_START(clear_pages_rep) + movl %esi, %ecx xorl %eax,%eax + shrl $3,%ecx rep stosq RET -SYM_FUNC_END(clear_page_rep) -EXPORT_SYMBOL_GPL(clear_page_rep) +SYM_FUNC_END(clear_pages_rep) +EXPORT_SYMBOL_GPL(clear_pages_rep) =20 -SYM_FUNC_START(clear_page_orig) +/* + * Original page zeroing loop. + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx, %rflags + */ +SYM_FUNC_START(clear_pages_orig) + movl %esi, %ecx xorl %eax,%eax - movl $4096/64,%ecx + shrl $6,%ecx .p2align 4 .Lloop: decl %ecx @@ -41,16 +56,25 @@ SYM_FUNC_START(clear_page_orig) jnz .Lloop nop RET -SYM_FUNC_END(clear_page_orig) -EXPORT_SYMBOL_GPL(clear_page_orig) +SYM_FUNC_END(clear_pages_orig) +EXPORT_SYMBOL_GPL(clear_pages_orig) =20 -SYM_FUNC_START(clear_page_erms) - movl $4096,%ecx +/* + * Zero kernel page aligned region. + * + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx + */ +SYM_FUNC_START(clear_pages_erms) + movl %esi, %ecx xorl %eax,%eax rep stosb RET -SYM_FUNC_END(clear_page_erms) -EXPORT_SYMBOL_GPL(clear_page_erms) +SYM_FUNC_END(clear_pages_erms) +EXPORT_SYMBOL_GPL(clear_pages_erms) =20 /* * Default clear user-space. --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5931BC83F01 for ; Wed, 30 Aug 2023 21:18:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241081AbjH3VSx (ORCPT ); Wed, 30 Aug 2023 17:18:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241046AbjH3VSv (ORCPT ); Wed, 30 Aug 2023 17:18:51 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53F37E0 for ; Wed, 30 Aug 2023 14:18:21 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInWYr032351; Wed, 30 Aug 2023 18:50:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=f4OnxoLyqQLXegBCEBZgTprHs3FXRbEDL24PiKFwl6s=; b=RK56wnOu0QkZnqxXtEboZftEvj3JUv080fonk/+0gHTOOvTw7L/cMPx+AFJN6AdHcBil iZBWRJCt+1lJsvtm28C45xjyNhvEl6hGU//2VRSr6OP1IhkX+ZVQ5tW5p12nX4Yr60H1 WgWtAUU/om0pqn6uxoYD7qX799j0yA3SFCE31QntrW78sY1uo0kaEPvNNjRuvmjONUGv Mz9qX28sHojw6zHK9T+2wOem8eYsPufN+ZlyiGupl2ReB0/qNaggRaYBIInEYygBXGFZ 4ChqchpSHEuzwdaaNZT2bK4y0z+uH08VZPiciLbUReSjJt8CJCLzlGw+kY/pZ0N0kEE8 Sw== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9k686be-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:19 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UH8ZkU024310; Wed, 30 Aug 2023 18:50:18 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sr6dpxdpj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H8Xs569DfhBMDWIx4oYi2tvNYohrKLluzf5YBgQD6Zd8pZUJ4U7EbXLEo6Ujq8BMseL4it+qNQgluUOamBna3zsD8lMVsoXKJy5EosExOf+FT8OtFjRCcN6ut/ZdFzgua4bkDCd3suFnjhLGFjlOC95xOC7oJnqNqqELBNDnWugsVk7d/ZtrTmDAUKvk283UWPjpXcg9VKn+k0nhm+jGv97NWX0Dijn+93Wl3Y1sXIyuGPPIkpFSxguHkH+91pbFva0JrWdjmYDaBia8YKMyMkLuaJFVL9V0uOt90x67qrpAhkYdDoezUjB/N2CQ/79EBVvCXLJ3jCiKsnB1iAsoUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f4OnxoLyqQLXegBCEBZgTprHs3FXRbEDL24PiKFwl6s=; b=HCG0LwtYtWklSbdIm7GxY7e7b274OOIDlZXuNk4GeZUENMy6He0A6SsGl0ho3XwxmlzL3ms5Z64+2OFGW3fZv/lP36RqCUPj3Yb9AiSDUv5GodhlG+3BYXqc1rMzoYtn7yUTXJoGy2Z+hCDClp07Txc6JPJqHDmOutAQTY1uMLa6incS7PdMUmyQ+QJiNk3/7pW3lcoP4cvnT5hjHOiC2cDiwowRtTTBgwwaIop+A19F/EntWrXPQDoS7TQLl/1y5boJvH4KAEgMv/p9q07/ln4ode2baN0QtsohEweK2yDY0VtiiGqx0e4hh69+3AwdCjGqIytA/QFywzmxIcabqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f4OnxoLyqQLXegBCEBZgTprHs3FXRbEDL24PiKFwl6s=; b=mQSij2sFzFH8OF49dSJHo/HEyZWcj9i4r/H3CJfmd/InDcvKDv+E6araT8qKmOdpxGzrtCYWwc0DGqidrLrU+E/UZGtmXq4JrKRUoAzSmUwwDm1Ha8ycoOzYV3uOdTF38l65K71DzU9CAgUKfiLkQetx89ytqmLwJS/ZPzNChFI= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:16 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:16 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora Subject: [PATCH v2 5/9] x86/clear_page: add clear_pages() Date: Wed, 30 Aug 2023 11:49:54 -0700 Message-Id: <20230830184958.2333078-6-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0037.namprd03.prod.outlook.com (2603:10b6:303:8e::12) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: 15522295-d22b-44f4-8411-08dba989f376 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: USGohVyEfoXKqZ7N8gal4AAUWfEUluDAJW0m8qk38KHSH2bKnMBTZnyx0oq60UU8QRtun2Jj9c/YCc8I9Cu1kIgIZkcZ/X2YXNGHZpSYybk1C9EXxyu2qXXZ11wIQ/ZKEJvwR1DcR7Ed3CdM4/Lh67ckdA6XG5HraPBZdeUAP493zzB4kdZ/OIIakf9YNFLNFGcVxrLGQL/OO79lh5qPlaQdHmjt5LjosZEhR9QDDRP3t7UoqIrFcI5qR11HvVYfoCzPqM2ccxODnAAr+Z3SBDP6IabOqJ7AkFpI/LqVnUJ7qVr4Njiiksl08/ssdV8qoIeQwwITDZz7VA6eSsariHcRKasfi7ZUvtMpRNK/D8qDro5IfKjwxAj4o7ZfUlS14E+OH3YvA/jMnen96e0eRb5Q/1Vq3JRIfoDrrByeCioAW6OFAz92mKIbxZIzKs+M1OyWXLvqd5hdRxdY2ZQljrhsLqvficZFlv7DgtO5j6OaMbtCt1wv0lAUs/Z8/Bpy1YfRoFvXX2/w0QvI+su5lkkxrP43uxBYpO+ZdEFs+HatxJUjnHknpoOnQKstbAYS X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(107886003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?uTu5kpS8sz1hUDLsjANxDWQSDL1AHRQ/Lrwzo6DdSKtBV2JVi7ynx7DVfukC?= =?us-ascii?Q?haK4TFII3bjIYSQrcXYQPrejm+V2jLNbLg7hezF8EM+ireVXAB/lN78nveP/?= =?us-ascii?Q?9Iqof+6cPPxxQLQqaKXggdkhjorhd6mdcCc39SRH9KeN9/XTHX7B2voVUHL2?= =?us-ascii?Q?3H+W69vvRkLCWLHCIGLyi+QglzK4e0EJ7yRrBnPc/nVLOZwbNxFUtZ+3UyXY?= =?us-ascii?Q?aOq9hpLfvyJfNsT/aQLcnXl8XB/xo4r6x0wD4i7+JfAllEqVmtBa4Aws019O?= =?us-ascii?Q?ahdxw4Vp6jY6AtxY9cNC2LQ4euekRezAQtskAssCfDgQde4rew1QVdtRqY1k?= =?us-ascii?Q?xdPkRdJY18SYV+Lkd4sfkO57CrB42356TCZmnLw4sk9Q0BUIlZr1O73J5B1L?= =?us-ascii?Q?ISRhixMcNzueELnx9/fFIxj7Ty1wll6BcqZlK0Xp3jA5hR06d3WLJ4wTHX3z?= =?us-ascii?Q?JOsrRzmvJGVP5o0cqA+ad2DbWyGfo1MWReO0VKYPiiPewyYOFkM3UxJX4OIc?= =?us-ascii?Q?FjhMwPGoKGzygp98Yuq77TP6KCkZA1T1Ezldm40W9NnjDwgqzn7P/cHB+cOO?= =?us-ascii?Q?EI19U6C62YfkUhFlOIxvO5x2NLxLpVaz/tkQQC703wBCJ9DymIQ7RD7ep4lC?= =?us-ascii?Q?6Vt3/MjfZBfs2NUpoSKLZZaLO9778aPkrmn/9pPmcY+89RK6o/7Y3bpNFv8N?= =?us-ascii?Q?OTcl2FAyssEj3Ub0qADaGuPKh+oCUmvE5i8u5fuhVWpwFr6tQbhUDx/MLMFJ?= =?us-ascii?Q?Ka6aV0qPKuuZoj30chO7q5nW6pcu9/OYkrVmkI1rP+C4EAC8QBPGYBUJgrg2?= =?us-ascii?Q?okLTZz3QYvK7DHb53NgB9LTsV4HcyFqhCWh34bFtnoGUGD8PpvKIPmJeTPlq?= =?us-ascii?Q?EMmQ6KCyg7EDGAo6gmSBnkpU+V3ajqqvHW9ku0ilQNW4bJr94NfrWm0f5HBu?= =?us-ascii?Q?OYz0FC34/rbv85R0RzS9kVcdvBxd5Sqb5sDAEZKxiGxWvuSqxjEmOrsk4koz?= =?us-ascii?Q?TwpYfZCqU7TE3UtshBDAPUOWJ7Y9ZVIKH1yHkuvq+k/90yvfiY05yHS0DrlU?= =?us-ascii?Q?jAoY2a7FGN7azisGjGowW7dICnAixq4EL4qvikqQYNDyekAAPXpVfAH8jPBw?= =?us-ascii?Q?9zAS0Q/G+Bl4i9PvYDx8jfxoFN38XM3hNGg8LFU2HLULZlP6aBh86Ch96GNQ?= =?us-ascii?Q?YR8mTXxAGRmZJ+Qg7Optu4V8wK0MWejXw+L/SyaCyz4UcQGsGmbr4kqnA4Ru?= =?us-ascii?Q?3FrR7/SqeqaCK/+Bu9mJHDgiHkPRPfa5snuobhjzZF46DkAbtlWNyIsaOjUO?= =?us-ascii?Q?9jQtPb1ZSI1AwiYgV/0v40BTaa7vVvU7jXue8PS0JFO5f1ct6qc5tutJcLGD?= =?us-ascii?Q?dfE2/mx8RYG2uNCyOlKKWC2Lnv2IzD8wLNjRLS25sXrVYNME6QKUBtK2mfHh?= =?us-ascii?Q?xIEKRmrQ8hVsqKTaaRoA5xCfZiEcEVXYZo//yYg6/gzQyQYiPMscbCM/lrhI?= =?us-ascii?Q?TtmED4v0ly3ayu5jHHHf75BQotSPFggOMl96QSxUYxVpXgUPiXR/REOSxFl9?= =?us-ascii?Q?z0U/kIW1vOJz/LLnTGkzTbc5gQwFmhOVUYJNSGeU8godSggM301DFivu6BTy?= =?us-ascii?Q?nw=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?ehboCqBSLkxqnQy+q3oarupS1pkV6soB3y/SxDVSzvuyROsGoRfahO6VDGmz?= =?us-ascii?Q?8k8r8ISv6b7rIiU/zMq28uZ+5MmrHhMt40LeC3wlU0tfTVWea9bi3f12qVuh?= =?us-ascii?Q?EXYqNq11Gox/txzKgIw4+M6lo0X6VJ+4SFmLfoP5elSy/ka+cFwdEo7uhKVG?= =?us-ascii?Q?Y4Vr1frvd6ZAGUL2o7ZRJDnrPrpnitT6AmobFDhy6NCAei/o8Tsf6qrrEODF?= =?us-ascii?Q?2Jv2duQAjeWjb5qtXMeEUUAqjVONB0LtWW8hJ+evr7qZo5a1gESuCYUeMpX9?= =?us-ascii?Q?NXINmN3HrUYNhhA8uaeSsGHA1lppA1E+Q7LKTVqvYgAHwgelshFzhISdncwd?= =?us-ascii?Q?tKsbixVeiKfzBVp1ZXy0qn2SilIKUoLI1Lwd14n5FU0O/Bx6aa3j1lSU1YBk?= =?us-ascii?Q?1Wfr++z6j/yiEnfbTYsPkfspgm3mb5i44nsBITV23AeD12xyk1vsOXUmeAGA?= =?us-ascii?Q?rjuEct9+/899gMAynCPnkUyfZUOesWqUpfFIlyhefH4F7iq9lC2MxCRX2bTq?= =?us-ascii?Q?lI4D2/EtZ5PPqewcNGZOGaEbb5it2cLpMurR1oSG/TnF+zDopTvt3/MHsD4s?= =?us-ascii?Q?Lf8xFqUeLA/Pod3EXdmZ33zNeVWH3oR0sibjKR2bx2eXmVcU1oAKCEKyjYpg?= =?us-ascii?Q?sqZmx7IO1NK6/rKOTglF+KsldIrchm5XX6W3GLAVKxTSkqBPKyfCTuImbjI9?= =?us-ascii?Q?CdmUlH9pLf/gfjl9nBF7rNo6FWKSxj0BZ35+3V+ct+H+KrgRTP/SuI3CPOG+?= =?us-ascii?Q?FbdBtmHpXqM+7oLYETKKSNIDM9FTRby5MJbFTR6sFPN4dF5lofK4OHzS79ll?= =?us-ascii?Q?S6bkUSCHfFq6+IRMSdhIBMmvjI3zZh77usblK3w4UOcFRPtkXrTXqPFnAw1O?= =?us-ascii?Q?u8RnuE9D7LBowhf/8rK0x2HZb4o+IfWZh43a7GQD+KWFGpeui99CbLUaoe90?= =?us-ascii?Q?5Ae4L9aPdIzlo3c6VMVgbgPjDmvRD9OvBiP435+p0eH5k9Iak/roCu8S+DJe?= =?us-ascii?Q?zkbMiJIPR4XOFTvb9eqx7+TbhxfjetmB/6xfChP7TVWHISZiAGra7kfmQuby?= =?us-ascii?Q?+Ec5MBwGV8+dHFWBoig6PTDtBWfiLQK8Ony3t3HHKvke7dTYtpYxtLK6vGUh?= =?us-ascii?Q?ckd/xNsODEFc1C30La2eTNmCRJ72kVl3pDq8vwT76/0+auhaxA6XuVo=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 15522295-d22b-44f4-8411-08dba989f376 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:16.6142 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: V+i1tj9py8sU1bIso13RywBga3n8zHMg0IzyhH9xiDd2k4p1kidQEMFLFUn2Ox/b8NMg7z6xak7NeVxboIogunw420RRDm5kKUiOY+2dVmI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-GUID: HYKnwjGXzyMIkziu8CDz8UyInc4rrSH5 X-Proofpoint-ORIG-GUID: HYKnwjGXzyMIkziu8CDz8UyInc4rrSH5 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Expose multi-page clearing via clear_pages(). Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/include/asm/page_64.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index 56f9adf0e892..1e71570ea987 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -43,9 +43,9 @@ void clear_pages_orig(void *page, unsigned int length); void clear_pages_rep(void *page, unsigned int length); void clear_pages_erms(void *page, unsigned int length); =20 -static inline void clear_page(void *page) +static inline void clear_pages(void *page, unsigned int nsubpages) { - unsigned int length =3D PAGE_SIZE; + unsigned int length =3D nsubpages * PAGE_SIZE; /* * Clean up KMSAN metadata for the pages being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. @@ -60,6 +60,11 @@ static inline void clear_page(void *page) : "cc", "memory", "rax", "rcx"); } =20 +static inline void clear_page(void *page) +{ + clear_pages(page, 1); +} + void copy_page(void *to, void *from); =20 #ifdef CONFIG_X86_5LEVEL --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B29F0C6FA8F for ; Wed, 30 Aug 2023 20:24:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238274AbjH3UYB (ORCPT ); Wed, 30 Aug 2023 16:24:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237748AbjH3UXt (ORCPT ); Wed, 30 Aug 2023 16:23:49 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B39ACC99 for ; Wed, 30 Aug 2023 13:09:13 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInXEB032355; Wed, 30 Aug 2023 18:50:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=k3dumfrrLftC0rbMzucfNY2caXdgyJ52sKyZgZ2BJlk=; b=kRkiQc+x0Y00SD9IonzHq1CPYPEIVCDZq6p2fUDPausThUPSy+o3D87R0kw1Z/kchNRH IuIX2PEpmQ5Mlb3EAuQvJcFlvN0BV8o2QJ1Lo5E4aLmfWDvEbgrmMXFwx+Eh/3K5wJIa FC3nqw+bdzWFAltBHI8QeBpS4j/keHzA3lWlD1tyg3Kv6y5Nd2amOMTM2Ux5HhWaapXI JK6GwUZcfPHkzRgLr8qXhWAfM7tKowT6ufX+A3x4vPdV9DjIDppunXjYEC/iJzGv4R21 Lm4lL5+Hd05w3jl1y3YvDihiGNVxfDeLk6wJSwtVmws/+pCrGdgW1dBTSAFI2D1KGjiT Vg== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9k686bj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:22 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UIDLKC032817; Wed, 30 Aug 2023 18:50:22 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2049.outbound.protection.outlook.com [104.47.66.49]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sr6dqehre-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GmDEIYeRanHsCPQYOL3EvPSmsSrNQ6OBsA7XY2K1HvQB+rJATqe2SK5nj14mQxOhLUua7tpf338DbS0lxIFWFR2EHuges3T4CTMnjGsk2A8gwr8ZDpvBI4OGW5VR6IdWdPoqjU5q2YzwD0J3cNmqHDjEMvNiBdpiPDPXfKz1xxGymR/1J/dP/G4kFN/9l7VRxWn8LCOkwt47a4vP1l1fNaZ+jwxbQubHUgMmlO601b8ejDvMiGSYWmANC+H6gPOVwN0bA8K/iLeFoM7Klu+J4/2YmIdqmSH4w7oSj+R7OnrlxIY9GgDBUt9QNkv8UjYAzd0lXwdBBumynx49COgMgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=k3dumfrrLftC0rbMzucfNY2caXdgyJ52sKyZgZ2BJlk=; b=A+sQ06I7i6ywJholBgVjvTsEujiFiePhbQERpH3WoFwN0fLfxi/CwAqcN+E0kOJlL3/Qz6MpQxBtpZ3OFCRGFHhShQooVFEte1N33BT0eYJ/gdvhROPbhyE8g813J3ZlgvSi/sgCaSwpktlrhyTj/c9rR2gh42HtjunRUIzqyVWaYih9KliXPdX6JAF59E7H1ksYJ8I2o//cSL58z5kOEmvQcm6kpW+wax+SnB9vbioAqtzm8IcPISwbabH/P5QpipU1uHOPTzr8b1qLNFOJ3SaLSna7L6lXD3tO2v8fBxfQb8KB7MewLusMD3ZeNZCO+49r0N2ZHILwQtDP/j4rwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k3dumfrrLftC0rbMzucfNY2caXdgyJ52sKyZgZ2BJlk=; b=MCadMPk5Jw1+KGSUZOG64aA7sz7z5Jw++pZVpjBeUVKHlp9hJjGrc01Cd/pZMhxHbv+Gyg+e3RrrRd+gVeIwVKxQE7I6/yCoMF6kGIKUmDD3r62wXBGEamFliNUAonSgg5J0x92yrYWccCiF1YYW6BZhKoOIVntNPF4pifII4II= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:19 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:19 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora Subject: [PATCH v2 6/9] x86/clear_huge_page: multi-page clearing Date: Wed, 30 Aug 2023 11:49:55 -0700 Message-Id: <20230830184958.2333078-7-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0318.namprd03.prod.outlook.com (2603:10b6:303:dd::23) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: b67a77f8-6007-4ad3-5c00-08dba989f547 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3QTrGcnRQXwG4GAP4xpIQb3YPeleqOAMiPJMKgNoAMXqLkvCw2RO7wCYVSaiI4HFx4hnCEzgOKo55F1n5hKH/q5WaKQ8vHQgU2THGowMw61b1FfB7v/N+Blcvw2H7hP1J57JoIQml7AvhGSwNY7TEicTTwMOod3/MVYerl7wxXrB4Z/QW6Fdr5rKBEL4BD5ndcU4I+z+yT/t2d+5jbStrNqJWjvtG/edR8g4EON299ygEtcyTqnFgus24xextAzIIexGWEfRnE//HiCCLA3tuHYryJxqOMhE/4m7iVgNKwT48qi+117xkivnEq4QRWqNsp1tq7uCJtREPrC4nVYHc7D2EbmcvJesfZkaVxZRg3cHStDAW00so+radWdaEpmUtVPZQVsW9Hw+QOuBqHCI3/N+TxkUL8L4+dnKsLnnW79zgOPFYBMqiz5zh7n3SJ9TXoABU93Oo+rBtfeFdKW94mL5lleRSS97iUW855bQV/0WrO00X/61VC8kTwIezGnNZSgXaE7Hml+pj9U0YEvbSWE2H3imNfuCsH4TMss2L6KRsMUCuSOoZWFoIUHLTxDQPfeTSQKPivWYCS4cJblFrPLQu1mSJ58U1mhxyHKdBmY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(107886003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?DA1kNghMTEqQZ00yWf5AxqqyuX62HKg2M6s+pTR8g9PBEaehhEm/TPHHwuhg?= =?us-ascii?Q?uN8NIA0VChvBm2wrVsMk0CWa8TIlZUzdEHqjECS5td6XwH86BTbQJ3m54/kP?= =?us-ascii?Q?1fSLIrXdffLGZtYpn9krzQrSdAIhNHZfr7p/nKGGROUE6yngVbu/XKFp6xid?= =?us-ascii?Q?qPGL5hoYdbseUX0NLgL0MWlIpJmiFFtFL5jJ62f8fjqWeWRBQAOLxObHSgQP?= =?us-ascii?Q?LJwopg6GcIramk/YBQ3Tpm03gh1hYaNf3+lx5x0rD6jIxpvQgIfTYPxuvh2V?= =?us-ascii?Q?cMKn20jASFZYwqiX13Pn9xhklNsklTw3ayCtcyGE/VZMKGnrN3u1Qw/sD5U/?= =?us-ascii?Q?AwXUMC79sBGfZ7unl0WecrD2VeDPctqCb1UnyNVuY30edIY5J6eQLLkbt96Z?= =?us-ascii?Q?o6C7R0FhVAHR4I4Ixl2EQ1RV7HysIL/h9ZQ1Hz1NQOBczJc2CMr4sdgOcXee?= =?us-ascii?Q?nFkIT2/eZsCfAXIWUqi/NZk6mOOiYrqTy12dkcA0NBQ6B6laAEpRkGEGxreN?= =?us-ascii?Q?uzgX/yTDf1qjL/MCFMuuKyARsfzfJGviF/x3wLkmhMvQr0R1EympBZ1Rfqsm?= =?us-ascii?Q?XBGg2UQUMUZbNhSCAlY4cXDduyYuItqob3DyJX/TywWmSOnjoqgz5zazxzLK?= =?us-ascii?Q?Eug/jJ5z7qucnAFpmC4Mn7yeNiu42mqNAarFqM1kHLB38BvBPfw9KyBkw4Y+?= =?us-ascii?Q?7y5+/aZfbiQpVd2CJuNg1Ld1v8kpCmeMryWq/LpvQ+/cPdqpnvHnPP1DrsSY?= =?us-ascii?Q?RVQHhDbw/e+BVkP7ONC/6ZCdut3hucdHCWSf5yI7goxBZhDe7mlgC0/8tPhc?= =?us-ascii?Q?S1MEuv9yt9OESWIQ7mDJ9JpsnXIfCCKSjugJseQLw3n79DEeoj4KlKVY9EmF?= =?us-ascii?Q?cHAxH/xpxnmHA/P7D2T4oKX8N4nruYmD9/c4GlaHufEmxeCEps+AxeQZyFeO?= =?us-ascii?Q?0xGbKMnHeHq03Zacr8QBo0auEe3qnngEQCsatYex7C5VX2XkiyfYHZCmB7J6?= =?us-ascii?Q?lDHNgX9SrCi4W4svVrhUphlqHSDm3KZHQYiDP8D/A4L5sRqdFrIs62F17s70?= =?us-ascii?Q?bCDV1O7ggK1ggRClK33lnciZLsMUoX14OaZLdT4C69/ib1lPlzOLUkVfEyJy?= =?us-ascii?Q?JzV+0KfzMzLgjxfRa4WxNAJwMI1JavPY6uZN1mQkmJHRtHo4fCgjmVZHcFiZ?= =?us-ascii?Q?FblLP8K6JEW8Sse4xyEo6fN6p1F3FIaB/SzSf6h+0hVq8OJjUsllF1pRkqZ4?= =?us-ascii?Q?KHsXLYrO1zs/LzBNqz8RGuPPCptLUW8HEpkz8IezQ4VgEV03/OAt4e5lMX4S?= =?us-ascii?Q?TInH4b5WWt+cKcvKpfhNcT79T6dwtVi8An1ZBgUaaO7sl7SVcHTnaGkm9YJz?= =?us-ascii?Q?Fc5f6YtFNjtas7toFtW9VORmRgWSXHCA+xy+d1/sm2mgYDq8UNHvX+y39dZA?= =?us-ascii?Q?6pP/eQwYKXDakyzWTi5uD0Sro6EwQA6b9pEzib6uNBpFam5KTWLGGXMS+74x?= =?us-ascii?Q?3nAnKvBwbrsI1gBbTzrzpqNF9NiNMZvqo7+3Ew1Em1NO+/5+poj9BBWIvK+4?= =?us-ascii?Q?VnHqZH8IDl5v3S6NdwirTZulafcVQvyOxW+OhpdrkGfEFsVlha/Tg4DMLuwm?= =?us-ascii?Q?FA=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?yZwHKS2LEHYv7B5KTcxUQEeL7xtQCvg/wlnOxyMBuEizNSUrdqNuxZ0hmNdu?= =?us-ascii?Q?K9Nj1YaFCns8wyxrgH9pQ6NZIEXCJH2F9mon3lF9J9x88Lmj7+SRrjDsta+b?= =?us-ascii?Q?y0Yy+l3mCCaqs17jxZ2qzO21otBcevWABeViawcWZsICJYWZvnbKtem/73tj?= =?us-ascii?Q?qms5z265oH5EtDq5BulElgwPqMe/n+6JXway6IWWesRgy6dBBPNnH1U6Ilqq?= =?us-ascii?Q?CDA4/Hs9Difu8rCvWUw3AfnztifKA+8NIVTLcwfHw2eqCqHYsqcvEL8b0HkX?= =?us-ascii?Q?Z6L2DSTFGJ1wgaLB38crM75Y6xcF1iW5xaoD46lhJHovXCy91ziwauDALMyt?= =?us-ascii?Q?FAjG6jfg4oCkjuDuWJ0v4aEW7tj1SISS+h9J+41r5QfkytqTSylUHUGUi88a?= =?us-ascii?Q?txviKxPXtV+nguWKc+G1vczTBgMtB8dQcnspM0ppaX2l6/hclI4R3BYMkaba?= =?us-ascii?Q?QXlFguuVLKfRUp7aBNAKrwr3/K2zNamVx7hGRt++X0mSI7Sri8wvmxgpzjs/?= =?us-ascii?Q?/J3fS6CxhxBfd3D1wFanxlzeNYkFGdNq6utxwf+xSrXDgdR9Vz6VP6LvmCgy?= =?us-ascii?Q?eVH3va//DcX3zgicsxcyE7P7CPmF9fjBpO1YqUIHzAvo7cTJHyPy1RmOuUMp?= =?us-ascii?Q?07vN6uvGBwHlkQZC6nTCXj3u6vnNnigKMs8vFbo9UNcZtsPyZD3moSwuC7fl?= =?us-ascii?Q?+bK+yUvkFgk4rZxPBr0xJ3si1PGGX1vJ24roVrAP/OYhyRM3GCqRj7hF6EPX?= =?us-ascii?Q?2uHrGvwSXOgS+yzXUINsG0JguuQ385QW7OvA36Yjg6XOYUYeNWpz0BUhBA/E?= =?us-ascii?Q?dyK4tsG2x4DYyReqgw1y3vYRTt2N8Go2GFLIZuRo05hIkQ3TZ883cMfTYiLx?= =?us-ascii?Q?2hzEsxC+kegkQXv6Ea2jWSFVUQ3asCTjcu038fE8nWXG0CWg1omXsOt0ls9v?= =?us-ascii?Q?4IPfoh5dctDydpX60iJpWTLzACt9J8z/lb9U6rugUeIzx2t7TmB7oFBojcwv?= =?us-ascii?Q?0it8QQAdbecQyxgIx5pkM/gBBxdhCswoJP2aAn9CiMWAbXBWf0s5T9QZqDi8?= =?us-ascii?Q?5yUiYrhuBM782lecRMjMOFo/BictvYGMOqvHJa/7FW7yWnj7GAiAaJDt8KIQ?= =?us-ascii?Q?+oGRfD7HKwXGJfEnaHQB6VKH1SA6jwbSpLr+hxGd54iGz7L6nVs0aho=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b67a77f8-6007-4ad3-5c00-08dba989f547 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:19.6471 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7mmiD+O1KCRYJUxXTlN/6aitQCTNwcg2MN3nKN/vlXq06hYG2G3gE9YwMhzDiUb+Z2Ppho/MriTqBcKaa2DPpN1av9/Jb+td73gxevYq91w= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-GUID: 4__312qXJmNbYqMLkgPDDwLakmlzmzuL X-Proofpoint-ORIG-GUID: 4__312qXJmNbYqMLkgPDDwLakmlzmzuL Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear_pages_rep(), clear_pages_erms() clear using string instructions. While clearing extents of more than a single page, we can use these more effectively by explicitly advertising the region-size to the processor. This can be used as a hint by the processor-uarch to optimize the clearing (ex. to avoid polluting one or more levels of the data-cache.) As a secondary benefit, string instructions are typically microcoded, and so it's a good idea to amortize the cost of the decode across larger regions. Accordingly, clear_huge_page() now does huge-page clearing in three parts: the neighbourhood of the faulting address, the left, and the right region of the neighbourhood. The local neighbourhood is cleared last to keep its cachelines hot. Performance =3D=3D Use mmap(MAP_HUGETLB) to demand fault a 128GB region (on the local NUMA node): Milan (EPYC 7J13, boost=3D1): mm/clear_huge_page x86/clear_huge_page change (GB/s) (GB/s) pg-sz=3D2MB 14.55 19.29 +32.5% pg-sz=3D1GB 19.34 49.60 +156.4% Milan uses a threshold of LLC-size (~32MB) for eliding cacheline allocation, so we see a dropoff in cacheline-allocations for pg-sz=3D1GB: pg-sz=3D1GB: -23,088,001,347 cycles # 3.487 GHz = ( +- 0.08% ) (35.68%) - 4,680,678,939 L1-dcache-loads # 706.831 M/sec = ( +- 0.02% ) (35.74%) - 2,150,395,280 L1-dcache-load-misses # 45.93% of all L1-dca= che accesses ( +- 0.01% ) (35.74%) + 8,983,798,764 cycles # 3.489 GHz = ( +- 0.05% ) (35.59%) + 18,294,725 L1-dcache-loads # 7.104 M/sec = ( +- 18.88% ) (35.78%) + 6,677,565 L1-dcache-load-misses # 30.48% of all L1-dca= che accesses ( +- 20.72% ) (35.78%) That's not the case with pg-sz=3D2MB, where we perform better but the number of cacheline allocations remain the same: pg-sz=3D2MB: -31,087,683,852 cycles # 3.494 GHz = ( +- 0.17% ) (35.72%) - 4,898,684,886 L1-dcache-loads # 550.627 M/sec = ( +- 0.03% ) (35.71%) - 2,161,434,236 L1-dcache-load-misses # 44.11% of all L1-dca= che accesses ( +- 0.01% ) (35.71%) +23,368,914,596 cycles # 3.480 GHz = ( +- 0.27% ) (35.72%) + 4,481,808,430 L1-dcache-loads # 667.382 M/sec = ( +- 0.03% ) (35.71%) + 2,170,453,309 L1-dcache-load-misses # 48.41% of all L1-dca= che accesses ( +- 0.06% ) (35.71%) Icelakex (Platinum 8358, no_turbo=3D0): mm/clear_huge_page x86/clear_huge_page change (GB/s) (GB/s) pg-sz=3D2MB 9.19 12.94 +40.8% pg-sz=3D1GB 9.36 12.97 +38.5% For both page-sizes, Icelakex, behaves similarly to Milan pg-sz=3D2MB: we see a drop in cycles but there's no drop in cacheline allocation. Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/mm/hugetlbpage.c | 54 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 5804bbae4f01..0b9f7a6dad93 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -148,6 +148,60 @@ hugetlb_get_unmapped_area(struct file *file, unsigned = long addr, return hugetlb_get_unmapped_area_topdown(file, addr, len, pgoff, flags); } + +#ifndef CONFIG_HIGHMEM +static void clear_contig_region(struct page *page, unsigned int npages) +{ + clear_pages(page_address(page), npages); +} + +/* + * clear_huge_page(): multi-page clearing variant of clear_huge_page(). + * + * Taking inspiration from the common code variant, we split the zeroing in + * three parts: left of the fault, right of the fault, and up to 5 pages + * in the immediate neighbourhood of the target page. + * + * Cleared in that order to keep cache lines of the target region hot. + * + * For gigantic pages, there is no expectation of cache locality so we do a + * straight zeroing. + */ +void clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) +{ + unsigned long addr =3D addr_hint & + ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + const long pgidx =3D (addr_hint - addr) / PAGE_SIZE; + const int first_pg =3D 0, last_pg =3D pages_per_huge_page - 1; + const int width =3D 2; /* pages cleared last on either side */ + int sidx[3], eidx[3]; + int i, n; + + if (pages_per_huge_page > MAX_ORDER_NR_PAGES) + return clear_contig_region(page, pages_per_huge_page); + + /* + * Neighbourhood of the fault. Cleared at the end to ensure + * it sticks around in the cache. + */ + n =3D 2; + sidx[n] =3D (pgidx - width) < first_pg ? first_pg : (pgidx - width); + eidx[n] =3D (pgidx + width) > last_pg ? last_pg : (pgidx + width); + + sidx[0] =3D first_pg; /* Region to the left of the fault */ + eidx[0] =3D sidx[n] - 1; + + sidx[1] =3D eidx[n] + 1; /* Region to the right of the fault */ + eidx[1] =3D last_pg; + + for (i =3D 0; i <=3D 2; i++) { + if (eidx[i] >=3D sidx[i]) + clear_contig_region(page + sidx[i], + eidx[i] - sidx[i] + 1); + } +} +#endif /* CONFIG_HIGHMEM */ #endif /* CONFIG_HUGETLB_PAGE */ =20 #ifdef CONFIG_X86_64 --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4077C83F14 for ; Wed, 30 Aug 2023 21:45:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242382AbjH3Vpi (ORCPT ); Wed, 30 Aug 2023 17:45:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236973AbjH3Vph (ORCPT ); Wed, 30 Aug 2023 17:45:37 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73FCBCF2 for ; Wed, 30 Aug 2023 14:45:08 -0700 (PDT) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInwQp013269; Wed, 30 Aug 2023 18:50:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=0CFuIMSxAiAQM8abfHcbcFv68C8S+dyPEtQxUd/f0nY=; b=H2ddvZEBlPaaRycR2QzhBh3qDFaS0XV/Veq8js5sITuJtXFJMv+tLR1Q4X1b0F1JxR0k XQi6T/Xz6daqY3Ypk7CGPrtx2byRBOPShj+6F/gXQX0jVHVB4tbirlMdFBWxyhB1ded9 4iSLNBKOBJf07szxbD3cqIbqtB25relhYXSfNNgVePpG+uQFP2/fOOjmrUrQoyLxBmON JsZPhSXHga0GohX1OnVT+SnFRMlmzYks6tL9/styKKb8JKlNpYFIhu6Bgu5FcpBRqkAg A5/9ghV+ZNZaGGq1rWB3fwB1OSXfFq6kSJtay35Nud6a0ZKDBDyduNkfMzdv+3yoi1xr rA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9mcr5a3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:26 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInd6m014527; Wed, 30 Aug 2023 18:50:25 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sr6hpx511-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KmyRVgTje48hquCSTrT9yJY3SQkDIayafsStvK/cn70EY89gcQpoQ3Qextl2EzLGC3bzdXvY95yBKRrGmgFKZiRIYaYBAG4evVnEGX1plSx+joGl5d73mrQHOcZURkj0Xh2Guyt5gW4ZLo7dGfin8Dgl0KUyG1vL4Ki3BGEBF3+dZO1pGH7CGfeT+wsp27qnyM7+I05I0KvxlL3h9BCcgdg/QZwhzlejK8twrMZgaIgRLLISPNyenn1dBYJfJ7VrdHuq+5tStWgmOaQ/xZoX/YQQ5Qfa/z+J2zt0yZfKojOcsaQ0/m/OD6KpoFx9EfBJhFLj/BDHc8rYPkg5kqCD7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0CFuIMSxAiAQM8abfHcbcFv68C8S+dyPEtQxUd/f0nY=; b=b6KKsV/ITFpnMG4lwCd4Ie2+4FwtaX6Qf+Du/0Zea9qxHtvFIrHgV4abON7w11PBJiUPT+NdgsLeQGOKmXhMH/IccpArqS2gmWcWIRPQ3rt3smXYkJiuKuXPKRrzWKwMAXQCXL8SmmLpGWKUkhOxpGNhhmdcQXZvQN6vqfOxhUdShieZaiOIHMaj/xVsLO03dqRogsdlNJEZFPdCZecDvR89Ly/WWQ2DXl6jj8il4P0QoKUtcHgj81OtYLuQn+sFeSHv1QeQ7D1zwaw1a2+7LlKBfhH4GVnNJa1me1AkKwCKVF48o+rr2mSm1El9RjLe/JiVBesFOGYnBD+GvFS+Yw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0CFuIMSxAiAQM8abfHcbcFv68C8S+dyPEtQxUd/f0nY=; b=md5Q25oBZoE75he6N18CNHPj01scRLe9cOwxe/nE/Itipe54yN0TY91U+sDn/X+TjFKSX4iqS6xgeIK3zfXMffU8/klbHv2l+1bkx/waEYAmpmZ6TnSTXXXpGGcDjwtp8JFpxbZ3BNub+zSn/dYpLjT1IbK4ZAdn8H8QCYEKTbA= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:23 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:23 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora , Linus Torvalds Subject: [PATCH v2 7/9] sched: define TIF_ALLOW_RESCHED Date: Wed, 30 Aug 2023 11:49:56 -0700 Message-Id: <20230830184958.2333078-8-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0373.namprd04.prod.outlook.com (2603:10b6:303:81::18) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: 40b8e4c2-af52-44f7-7b46-08dba989f745 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lRl70Qk1TFTOZnxYSHzq7yJ+IdJYBeBcEtMPBK6AssB5D53UsG2GFV1MSKuCyAal6BC8F1YUxMP4VBgnno/Do5tozGABGw49zqK67ApigMH7VOUc5QY+xpfLC/2TLMsDNbQPMfh2hBoVf34xSH1CBjY5GPPV0ahO8RvtRbwNyJcEmrxO1E4OmrrYo1bmc/K1I9Zj2T0rcRfm7j+2OiYt2/QQUZHFNMppIu+4FqXWZJ9633eYXFC4LZxxaTnBpasIZoWiW01PWFndtk1hO5dIMQy6886xC9zIAnkMyRCC8/W4yYmBDS7T20RIT5ryPRu9KtVdf21qxNKhOe1r+CRyjFckD+Vai9DcVeMW38VWgmAAQudJK4pg5JIli0vSyvMUI9rukI4p1y3zr6LUx/5eyvMV85js1x44/70dV072hVp2C5MiAt1hFBi48c9+4BOQCdBTs6BWVHhBXgDdgQuqUTB2x5vYnzZiU0PTeOZZB5Upk/G0B9YHbtlHXutNyNh339MHFJPzSsVPCtbJpV/3omFdUq8zpfzcLkVHSZmblTCNb1vBevLe63s1futzZwj7 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(54906003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nmdBHd6KbfrGvPQSPYgi6HsVB3YEwN84lduLRI9ClA/+PkzFzsGcsGjtucJA?= =?us-ascii?Q?Aqg13I1xCg5VR5t9hpd2LoD1eOJm5qykKAuB2kdpIv/yWINT+8e/PkrPMhc9?= =?us-ascii?Q?1C3zR0vQHg4zwy01VjG2DDAAIil8QFrxRhX4CtTfQiPCWOzZ681W5cIF3E9H?= =?us-ascii?Q?TJEPO8heb/D65T+Xvyag0LNni7sc8nU5UkFDZReL4lL29ykz7LHEqXgjvKPf?= =?us-ascii?Q?kUrUjadm5J9yt7fF7ZThpWy4nwol+m6JaXpepLVZORGIdcnUEdFUkSma9t7S?= =?us-ascii?Q?FlsWxdA7zWV96wPPmNBjak0JtyR+zy5DzZEPbe4LKll+sYaumqbQ0idXlBpf?= =?us-ascii?Q?S7l0o/Il4KiLk8b9Jy4DHEiftxz0/8BPXpg9e4hHiRekCixCb987ygGdnmdf?= =?us-ascii?Q?bm3mZHq2tpRBGO+geZitjTD5HfnLA1rmJjeC9cmfbnSja4pH47bpKQQaldtu?= =?us-ascii?Q?0UFNkrmPHl+ibGBIRdT4dqW1s/XK0rmH9aZtHqL+oTDN/SgHecxno2g3ieFC?= =?us-ascii?Q?w6Sv6YAYazIxYIEQVbBMTWTHYa45/YTWfUthscHGQnnujD83n1uK3I0uv8TD?= =?us-ascii?Q?f1gYCGWSyQ6vSVj44jbLAFtVW/kUKcsk8qz10dL6APokMPPtAOFznNR9rZSj?= =?us-ascii?Q?Jzu0AfXsHXXOj8VdMz+Cho8lzM/JDARS7fDR425rMRbNPqdZUtO3b9HeR632?= =?us-ascii?Q?65FpZn9uGZ8Rjguu4x3Z6fJdyK9H3BN1zdCp1NGAfaLH/Yf5NwmvaUdHyvJ6?= =?us-ascii?Q?oKFZBPx1JROTfn8J2LIoAfi1DuefrWee6q4hLYQOHjU+8t0Oxs1HCYjGVqCa?= =?us-ascii?Q?Cf1/aGfGVNv+DP6CsAMwP+Vmku+4LSo6a74IQy4slYKRJhRlKyGjKFPcLMK7?= =?us-ascii?Q?I3EkqbYyFXac0MXw7bWvXjLoqC7h1vbYWuOU+smzEgyWG0rAldQZQwL2xFQp?= =?us-ascii?Q?539OIVeIG3DlMBX4CPYtg+VuA7cPJi1SjdQgYVfbzzPcbNwtrOSIiN9Gl9Kf?= =?us-ascii?Q?62QM+HNmB9R0N5WOg2lcmqHrgQIdH1R3NpFKI2pp7g+3lQosuHacxr9zBcLt?= =?us-ascii?Q?0tNgwtTO9FL2Qpns8FTCNYg/QSLSj5sMp7khTFkb7eYKN5eYNb6K45KBda77?= =?us-ascii?Q?3jNeLASlh2DiR13G4jiTiqQEJGFbE9hHLPb4eu+8idLC1M1grJ6A4r4SqThz?= =?us-ascii?Q?tmLYlc0y29+643yTHkBICMrxGRvulkHM4pfapMWrWBqUoEEqPveAXjNnHves?= =?us-ascii?Q?RekaGq4mZZfjuMNkvbt/lY1mklMZ39WaVogBsBtQTJY5qT1081YOOerIMqDx?= =?us-ascii?Q?sqeq9VyP5SMSA3dFrGdBIvhVgFAaoJDGmtu/njv6BL0Z9UpTDSAkb5nbOk/R?= =?us-ascii?Q?2JOqHDxH7wPx9t/pVcFV0SH+U+0BPVtlEoNEkPpNNQ9y4msPjbJbNFyOPdy3?= =?us-ascii?Q?pd5yLRIs9uajx+UzCkxun6owTBWYzj98I5nC9dxrCyI4Jte/1s0+OQFqIblK?= =?us-ascii?Q?Jx0FyaBSuW0c7+APIMh84L8LPSUVoT+3n0XPEm3Fo4TbYbkzEGz00Cvv1kv/?= =?us-ascii?Q?PfhI1Ne1g6ktSeATCztN3i95NUA/jQoLapLopkfbagiziSjWtbTCkiDJXFpB?= =?us-ascii?Q?Mw=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?kg6rMERIKfjGlwQm52+bAhJ6uapRFWZ8SL3ULoO1xyHhMtj3WFldD7kBxGHl?= =?us-ascii?Q?rh7f3d0C/oqPQv8EF46A73CY15xvXNjZRlq+amSGWCQ+srIy6kimGLNQpEMH?= =?us-ascii?Q?fIUxaDm3cnRbbqK7CseR5XvVBECOi0ijVsd6PtDlmRt1vuxXnzdcxeLtE842?= =?us-ascii?Q?B9sBMPgiQW63aTiIsfdQSX3/YTDJavkYwLgcKusxFjloTlCv7WDxbbi+yHla?= =?us-ascii?Q?9x1LI8BYLoAcYxbcDKaSZLCwFh/oqEK9rikAU6ERHNaN8l5B44ZJeP4B3bfu?= =?us-ascii?Q?I2XC5li0PNmFcJJSF+glmKfwFcbIig9JbFlQIRAkGF/8qZsQH43/el7CD29+?= =?us-ascii?Q?J6vsHTfRILob4Ze6d7Yq6J7CCe4uY+4NZFtVamWxUxpmLTF0mKRrcaJVAGEt?= =?us-ascii?Q?P5goSvKQiYB4RwGtgGPA6gBYl6wXplMGTYHVLIWkoptlgP2p1H1Bo72oXxne?= =?us-ascii?Q?P6eIs+BqkgOm/ANede1LB7OGNUhsSEzOZ5+z+/rD+1De8KCKPshTI+1C3E8D?= =?us-ascii?Q?J3rQnkEVYgnztEccKi7yLtJs/giD4f7n56bXzt4SB4zlHy0dwBTYsfpM22tA?= =?us-ascii?Q?MjMIHbwlnoA8TGgnTFq3T7RXWUoeIqP0mH3hKO+hfpr80KXAXRxTJPcREGf1?= =?us-ascii?Q?H7KYVpTXuJ+3kmUc2fYYs4g/f3iVC+5xfqzzpzxRstPKdrHDdphUW4uU1s0e?= =?us-ascii?Q?ZaNLhCypE/QV4GYz+NcoVeWP030TJQj8UAE1R6+pLC0tU+pUV9BAxpXejaUz?= =?us-ascii?Q?Hyd1xo1XzccWyqBnCbU0ukraQVChZiwbQnfEHRw6CmO2tHHmsybjXSrdQzeH?= =?us-ascii?Q?Nf/36vEvOsKHkjEpMUjltoQ+QsDomSutUD3gwYbr7y+CXbWIavDzYqZsoXXX?= =?us-ascii?Q?o0OGd9OeG24X8iT4jwgB3X2QqrpwARKxuWujmBdyvhWof2SNpc4/XVpyPLlT?= =?us-ascii?Q?nHx7qulbHH+xZaxRxq4hSqOMw9qKJh246Gr+xxcc67I1YC5PvfjoEb4hp9ft?= =?us-ascii?Q?4yrcvGoRt+cU8XANFW5cvpsVBqKqi2QUFHWrJMQDx5JzgO488RwOREi6qhAS?= =?us-ascii?Q?DJJkGi/O+4Ssv1CuOm5Pm18MkMZDm7kip8tAAIxnMxGu8qEIJLY7mENv3POG?= =?us-ascii?Q?JAkhh1WHShmqXzAWJWAca0Ykczvj5a7twqt+dLS2BbAJZhXw26WCbZQ=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 40b8e4c2-af52-44f7-7b46-08dba989f745 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:23.0004 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UiA87hyir+IIf3x+gRxj3JGnMFruFf/KE7nHJNJZqoHhYusqt/gmozUg9wtnurEkc5CnjNXBYnCuU5mkrwC5VASwSoX+slkEb5Ei1u5uxtM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 adultscore=0 malwarescore=0 bulkscore=0 mlxlogscore=997 mlxscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-ORIG-GUID: J0hZYB6_6RDZYMCLXaArRS4EvQSaBzX9 X-Proofpoint-GUID: J0hZYB6_6RDZYMCLXaArRS4EvQSaBzX9 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" On preempt_model_none() or preempt_model_voluntary() configurations rescheduling of kernel threads happens only when they allow it, and only at explicit preemption points, via calls to cond_resched() or similar. That leaves out contexts where it is not convenient to periodically call cond_resched() -- for instance when executing a potentially long running primitive (such as REP; STOSB.) This means that we either suffer high scheduling latency or avoid certain constructs. Define TIF_ALLOW_RESCHED to demarcate such sections. Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/include/asm/thread_info.h | 2 ++ include/linux/sched.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thre= ad_info.h index d63b02940747..fc6f4121b412 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -100,6 +100,7 @@ struct thread_info { #define TIF_BLOCKSTEP 25 /* set when we want DEBUGCTLMSR_BTF */ #define TIF_LAZY_MMU_UPDATES 27 /* task is updating the mmu lazily */ #define TIF_ADDR32 29 /* 32-bit address space on 64 bits */ +#define TIF_RESCHED_ALLOW 30 /* reschedule if needed */ =20 #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) @@ -122,6 +123,7 @@ struct thread_info { #define _TIF_BLOCKSTEP (1 << TIF_BLOCKSTEP) #define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES) #define _TIF_ADDR32 (1 << TIF_ADDR32) +#define _TIF_RESCHED_ALLOW (1 << TIF_RESCHED_ALLOW) =20 /* flags to check in __switch_to() */ #define _TIF_WORK_CTXSW_BASE \ diff --git a/include/linux/sched.h b/include/linux/sched.h index 177b3f3676ef..4dd3d91d990f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2245,6 +2245,36 @@ static __always_inline bool need_resched(void) return unlikely(tif_need_resched()); } =20 +#ifdef TIF_RESCHED_ALLOW +/* + * allow_resched() .. disallow_resched() demarcate a preemptible section. + * + * Used around primitives where it might not be convenient to periodically + * call cond_resched(). + */ +static inline void allow_resched(void) +{ + might_sleep(); + set_tsk_thread_flag(current, TIF_RESCHED_ALLOW); +} + +static inline void disallow_resched(void) +{ + clear_tsk_thread_flag(current, TIF_RESCHED_ALLOW); +} + +static __always_inline bool resched_allowed(void) +{ + return unlikely(test_tsk_thread_flag(current, TIF_RESCHED_ALLOW)); +} + +#else +static __always_inline bool resched_allowed(void) +{ + return false; +} +#endif /* TIF_RESCHED_ALLOW */ + /* * Wrappers for p->thread_info->cpu access. No-op on UP. */ --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82B39C83F14 for ; Wed, 30 Aug 2023 21:28:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241254AbjH3V2K (ORCPT ); Wed, 30 Aug 2023 17:28:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241213AbjH3V2I (ORCPT ); Wed, 30 Aug 2023 17:28:08 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9695712F for ; Wed, 30 Aug 2023 14:27:36 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInqUt018036; Wed, 30 Aug 2023 18:50:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=ZQUSDAXAq2hTKgg73tSO5DT1RApTyE+5aqfIw3lJuGs=; b=0INu6VVv0GEMTTAIEQmj3IRUbaAkWGlnNLkcMfKUsq+nqi0t28qnnoXevRoJJxEtuSyx VB2XmzK/k5I0OCTCnA3Me57Cka7WdAJGRutZJnwW/WiR8ep/mJva6fh7AmhPWxxnDeqf 4R3sdHOYUOmQfS2fiOSmp3gdkO5wi1q231JzTqome0Sa8+Um27rFZS6J3XKjM4TkhAPQ ODR/HBUeCS96Px2WcW6kD49BkGHGEaG2IqX2TVjwC/VWWtPW/pFjFO6rDOAZAR3x1EyI IM9Mhc5QM9K+c8ptIjOAzUTpIJePzO4Xrko4OPfHPltWvxN7zyvo/SN0t3DAFcm9gLN0 TA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9j4g76d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:30 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UIVmJV002385; Wed, 30 Aug 2023 18:50:28 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3ssyw43vmg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eeBUntXapP+PwJX3vGI8EfwjVIpabiZp/+hTtWwjj6YvhqCeIOKH9+EDdOFKNPZhvvhPyhLlqX/83URG+rNLJ6pPH5vwpzqmjKBxhNjZOJ9jR5oFDTNNiQqs8du2ar1ir1VXXCWVh/BEdTVTuQzY0wHTuGjDDsSaJS19+m4YtVdtbz4qxs3qhfIIvOnipjvGOncvOA05FP8MVVw6W3pF+wjzlRR/xQIVyWlQrmuePQE7TEWdWCiUHtTeKD3Vv883euo55SAqAcn90+rv6c7idsAr+Zq4Vv/27TxYTfcYJvdcSUBcErBjRg8/xpTkC4v9e3j/JIwVGKbiqvQz+TR40Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZQUSDAXAq2hTKgg73tSO5DT1RApTyE+5aqfIw3lJuGs=; b=cRU1wsqZKWUw89kkygDeCjAGloekB2TNxhYLexHSZ6+pReZGESmYXg8ZWZjAHsO2ne13cwWou3m72JKs/lgANEgemT5FtdbasxS6PGIbUO/XPQIppBfMKv/Kr6U8EItri41/ITJnUBVjJcI2Do/EnWen3OWIru/ErE18F2RYWF5EAaw67afC1OVOr+SmIFATYVcv3INTEDY78T8RDPCP77BG8cUNy9+VZRnVuB8OHSo0SgubPRMNl5crYJg6LqGMC2G7pPPVlLthDR0oxIeV2xP8f0uTHowt8CqTq2nfPMpvmPMjcEZ73WfA2f8z4rdGYDsBVcg7BLxqpcN+dAEy1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZQUSDAXAq2hTKgg73tSO5DT1RApTyE+5aqfIw3lJuGs=; b=pxcH0wXleHNC7zGYlqE6t5mRFwY6E+knl73bCte126V/QJptfWaico93rZR+/RNQy46X9ATAqCTvFFDRWFxxzydA3I7amqfiJRu7JIJ1lck9V+TO3wPD7ySx2F9blSsuCJzFZSRCMouLGRLVrWyhB2nCh24AvAfBULeThhkHs14= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:26 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:26 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora Subject: [PATCH v2 8/9] irqentry: define irqentry_exit_allow_resched() Date: Wed, 30 Aug 2023 11:49:57 -0700 Message-Id: <20230830184958.2333078-9-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0231.namprd03.prod.outlook.com (2603:10b6:303:b9::26) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: 859026f9-65e8-452f-5177-08dba989f916 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9vhy3ZQGGhzt8VqtEpcbh/jak9MnZ6SaC9nzjmiZ8LO4Y/bzDt4Uq1mdXYePczh4yeAEQbE6MvOVSo7R2vVvOF/yqtkjp2hsM1dD94+KiDZoKQKmUDlfvcI2jDjBUdFobZ+V8ekRcCbLArx7D6R74j/YhkDHzfOKg1gUZheiQ4j65IFTceCsxvJ00JppKo5S6IkYEMARGBLYLcE4mLSJnYcRlW2/w7Cfh8oHmt3LCyoC9JBfogXSc2d/klZ9b7Xh/LanPXTw16wtt7brYpgT0nG2Pqu+aKMpqeUVeF3fZFCG9SCB9RLnmi6B1Zbj21jB6j+1hXatG7zV+4ETst9NDSN+oNmE3LpihSsKvyUKdaB5a4WiSJobxHiqC9y6HA5t8SGsvfQ6Mb+5mlrhD9uBJfyvwyIOcENh59AFdYSU/vx/YHwVpnEgYxxLOrXdyq+HfJaD24ecM6ya88oAFFxZkhtOKY4RruQZky4UQHwR0KM+Ny21yizEe8pK2yCU6xOJxvoAtimOOo2gJDG3Tq1i8oVQU55rBANbeQoAtJQFFmRg29ge9XL43BirwMGK71Qy X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(83380400001)(2616005)(107886003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?yUHmZYk5hcuhGYVVWykHTIFUSr0HeoGfE3wZnab1E5X64dXBkPaWmnO08hC4?= =?us-ascii?Q?neSNrivKQ4W8WeGsa3VHAxKCXc68ht0iwP9qg/t7RDAMWIqGBTJpTCYFeWqX?= =?us-ascii?Q?8is1G4G//4GqvRd7Xz4Rni5q3cBfRSceeJndISpHelO8GKQchjpbhFViLEQL?= =?us-ascii?Q?mUCdNnKXznsHrYl3/2OuJ4s2y0tWpqUlBG9BPH9SkwyGokPIpnwATJx1wc6o?= =?us-ascii?Q?IGtMnh4xyCNltQOZkAnXZJKLRcTQHrYQDF73WPwJcBQ2dpQ/h9+I+Qxd20Un?= =?us-ascii?Q?yDnMefhtREoMk2egWjDIqGUIQfPdk5jPcUuGzD9pMXoZ0ijnAC0wPvyBl67x?= =?us-ascii?Q?q7tVpYLl58r4tGb0VFShNuWNwLyTJYwkVdXRGbpdsThlkNkz66l1/uuOF6/W?= =?us-ascii?Q?TwMbPMFl2gpxrm0X3pbIvb/PBmaIpUNwkdXA59R/Jml2QsQnQTiaoZB2F6se?= =?us-ascii?Q?QjXXbGfR4KpbqFF/q82M8Z0ysyQ/cs3xKnh/Ym4nIpvHfFJDVIYRJs2nxVeN?= =?us-ascii?Q?KZCIw0My4OhZh0qm2AnDZT8AewENwT/nMxbguxBeqSAgP5Ruxjl6lNiIRtzQ?= =?us-ascii?Q?w+EQuPAKxtenwYLqFnWmkwwqQ6Aw9X/1dfRf7VP7PiyaXQs3a1OGGbxQi0vv?= =?us-ascii?Q?C5quS9qOWuavNrjET/UTlbGgnQNCuvQ+quBMm0voIrSUSElwPLAmdIa5Pnzq?= =?us-ascii?Q?BcqQH6/3oUzrnk0FA1idBfJ4S1wSRJqBBMk9nZhFbTQHLuiOU94pjWqbeiZZ?= =?us-ascii?Q?+xhK5X61OEIUFPRgIxCiSHa+0pyzbvYSOlTDgVgm92c3ZE1yAc324byPhEhx?= =?us-ascii?Q?/PyPDPbbRX4EaMgaQxdOk/MZbH6UKsjYwh/XgUGQ16p2MdPHIgdCNv02zdWn?= =?us-ascii?Q?WJeoxrNcZgRgvfKNUxIawl26n6q6Fl/l7HbCpRGYzwVo4DNR7Cok5RHncGO0?= =?us-ascii?Q?uu4LjbUeExE2pyLGSPA44r7tCV2uHVkkdVqp/DWUjONtHVBsX1knANcWjoug?= =?us-ascii?Q?HaP0FwWlOV3UHuPhXIa9kHMUq4KbG8uHaVf7JBJy0VsrRg6Ha7d45oEFj0cq?= =?us-ascii?Q?MLAKrDZkQPqLjBQZpaSbuXaz/crV3RY1a4Lm7h8NlY3ePlffBy+BIS2YDS2S?= =?us-ascii?Q?YUYOj699ljQfcx8CYJEwAJ2Ho9IjkMmIe1K1Uda1+eWk3rza1yEl1m8ZNztj?= =?us-ascii?Q?fkcrt8n7+TsWs4cYfzDZh98k1yq3PXXmqFPTZplAEedfqUCU6ExYUyf9Dut8?= =?us-ascii?Q?fa2pRrp0JQiqDXTXcG+6BOr1ORh92dHudFo6dM34utb6hEpS8W9DzDMz75Co?= =?us-ascii?Q?dRno4vW1FoUMz2TZ6lVHspxmMmH/FVop3HNYLwzmAb/rGm/vOFmBiKSUdRpc?= =?us-ascii?Q?Z/Rf6pyhftZ+h4LnYoBuNvvZzTh8G/aSFqNckpDlSSPdljuNj5qGNq/DOXC9?= =?us-ascii?Q?37Tx6lU4V5V04ti+eqEpqaycz1i4Ov2B+9lP1znkRqXW5dzKHhnHQSQOPOB3?= =?us-ascii?Q?R+XjVtbo/v+fUem49sqAFPcx8IszQeQ13o0twcF2lnG5F6X4W04c0JKtElWP?= =?us-ascii?Q?E05pvx3F+k3VBUrYbj1IpmC0RDH17ap4ASqHrWPRGVD2mNO6rD4k29a4Zcja?= =?us-ascii?Q?fg=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?8S6aWMRaoiIqqx9xSb0a1wWIkKniSJiQeezEmMiZIutrGvTnZZiFP2wd2lno?= =?us-ascii?Q?HFJ1J47GjIORk2Wun9IRz0hbxOgazpsdJtKGCnq6obxwcAAQs0h4Qut/tPla?= =?us-ascii?Q?tr0Y9An2w8TF1mfNLaIMmzk3RfyqqqpHnYALET90lK21PD9qABUXmjxT5Ioo?= =?us-ascii?Q?5QGx626Eo7xAWtmr7mo1ziQOBNcm1vc72xIOWyZ1fSd5CfJko+K6v4ngakf7?= =?us-ascii?Q?i6dcbBPvBmkKgdS1CSk3//aqYd5f0iECo2vRQ8ZSAMNHJD/SKt1/Oek15Er1?= =?us-ascii?Q?oWlCnXMfaEVEh0ABxclbw1Jjm7Lmx4pJMW1EN2QYxvluWrFdwf7kiVrlgk7F?= =?us-ascii?Q?DJuckh+y+PcmUiR4g299RZ65xMeBOsPuvwcDfQ+oAPPxs0IOBR7JVJK32L2e?= =?us-ascii?Q?JAP54YjgLGhDw9H21+d1djLSk3rlI5f83fxT7wfcvYCwQwhFdYxEpVZ2gN9k?= =?us-ascii?Q?8RVXXx0N8IRd0ANNgXJL5ioHFgDy6RbdMQQAVabm5GG3iT/CaAHGhvWj3b0/?= =?us-ascii?Q?sSu/ic2+lYgKKrGYJUdboLkdQ2+9836dGz4u/heVY6e6zRbb8ZRpwwrKZZQ6?= =?us-ascii?Q?X1zEZfVoghrBsNwpFLN2k5UPnHn8WpPPFMBC4up59RAI+4hKtEf+zIe/KwcK?= =?us-ascii?Q?KKfBrZonJxZgPaIvsJ0eDZbHCef9lfospmFBhs59zYYH+MsNkQcNvVuV4ntx?= =?us-ascii?Q?B5XCZS6UNguOQA6fRlL+NFWOrBwUBNL396QqUp1jj0ySgpEwHwYlFaXPP7gh?= =?us-ascii?Q?QECDPdKPTosY5JhRjWyat1ydk0F26MqWNFQLYmCpWbguwUwDLhJ6UMoS41a2?= =?us-ascii?Q?dN0Owiy9uRD6lSllSh8n8RcvmChJ0l4UUvym43WUvcZ7xfH2Nn5tsjWDBylX?= =?us-ascii?Q?aEUhk6x+Vt92/2eZjOzD07jwzbYJjLhxHDcn4loy3F8wwS8hAQWX8mk0cswP?= =?us-ascii?Q?YAEgr2W6fyFeUJtLFHoNDneAQVMhPO3unfZoP6zfWc9uZRO0f23d7yN2Fi33?= =?us-ascii?Q?Vnmy7vbNYQIAw+nS+JaMIkMqaiLJhP7MUeXshVdxKF6AKKzgUkth0rNFaIPP?= =?us-ascii?Q?QnAARVDQricMk+ucCLnSjIMx3Z0VKz/FZBk5IhjLrAU/FA1f3yVm8pLCJsgL?= =?us-ascii?Q?eX22jwmo8U8sWOugbC04S+1OS+s/nh4xF6+ToZYZMsS6XMt1r7eOgCY=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 859026f9-65e8-452f-5177-08dba989f916 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:26.0338 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tlIgP7oBt4bn9kBHfx27nH4m1QBsyiqqPqvzxlkyXxrPgJo/M3YfWLTMudWUdNpf7TFhgOYfWbNjs8ZpJFvc8jUNklM2kIClMyGBnmFqRF8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 phishscore=0 mlxscore=0 malwarescore=0 spamscore=0 mlxlogscore=482 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-GUID: mb-hkt9pfv4cL6pGQ4HITQCjtSzH_TkP X-Proofpoint-ORIG-GUID: mb-hkt9pfv4cL6pGQ4HITQCjtSzH_TkP Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Threads marked with TIF_ALLOW_RESCHED are preemptible, but do not have explicit preemption points. Handle them as we do preempt_model_full(), by rescheduling in the irqentry_exit path by calling irqentry_exit_code_resched(). Co-developed-by: Peter Zijlstra Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- include/linux/entry-common.h | 13 +++++++++++++ kernel/entry/common.c | 13 ++++++++++++- kernel/sched/core.c | 32 +++++++++++++++++--------------- 3 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index d95ab85f96ba..3716c223a703 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -415,10 +415,23 @@ irqentry_state_t noinstr irqentry_enter(struct pt_reg= s *regs); * Conditional reschedule with additional sanity checks. */ void raw_irqentry_exit_cond_resched(void); + +/** + * irqentry_exit_allow_resched - Conditionally reschedule on return from i= nterrupt + * for tasks that are explicitly marked TIF_ALLOW_RESCHED. + * + * Enabled for both preempt_model_none() and preempt_model_voluntary(). + */ +void irqentry_exit_allow_resched(void); + #ifdef CONFIG_PREEMPT_DYNAMIC #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) #define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_= resched +#ifdef TIF_RESCHED_ALLOW +#define irqentry_exit_cond_resched_dynamic_disabled irqentry_exit_allow_re= sched +#else #define irqentry_exit_cond_resched_dynamic_disabled NULL +#endif DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_res= ched); #define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resche= d)() #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) diff --git a/kernel/entry/common.c b/kernel/entry/common.c index d7ee4bc3f2ba..b4cee897d6f6 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -389,6 +389,13 @@ void raw_irqentry_exit_cond_resched(void) preempt_schedule_irq(); } } + +void irqentry_exit_allow_resched(void) +{ + if (resched_allowed()) + raw_irqentry_exit_cond_resched(); +} + #ifdef CONFIG_PREEMPT_DYNAMIC #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resc= hed); @@ -396,8 +403,10 @@ DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irq= entry_exit_cond_resched); DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); void dynamic_irqentry_exit_cond_resched(void) { - if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched)) + if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched)) { + irqentry_exit_allow_resched(); return; + } raw_irqentry_exit_cond_resched(); } #endif @@ -430,6 +439,8 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqent= ry_state_t state) instrumentation_begin(); if (IS_ENABLED(CONFIG_PREEMPTION)) irqentry_exit_cond_resched(); + else + irqentry_exit_allow_resched(); =20 /* Covers both tracing and lockdep */ trace_hardirqs_on(); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2299a5cfbfb9..3c1b1b0cd575 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6570,6 +6570,8 @@ pick_next_task(struct rq *rq, struct task_struct *pre= v, struct rq_flags *rf) * - explicit schedule() call * - return from syscall or exception to user-space * - return from interrupt-handler to user-space + * - return from interrupt-handler for a task marked + * with allow_resched() * * WARNING: must be called with preemption disabled! */ @@ -8692,25 +8694,25 @@ EXPORT_SYMBOL(__cond_resched_rwlock_write); * * * NONE: - * cond_resched <- __cond_resched - * might_resched <- RET0 - * preempt_schedule <- NOP - * preempt_schedule_notrace <- NOP - * irqentry_exit_cond_resched <- NOP + * cond_resched <- __cond_resched + * might_resched <- RET0 + * preempt_schedule <- NOP + * preempt_schedule_notrace <- NOP + * irqentry_exit_cond_resched <- irqentry_exit_allow_resched * * VOLUNTARY: - * cond_resched <- __cond_resched - * might_resched <- __cond_resched - * preempt_schedule <- NOP - * preempt_schedule_notrace <- NOP - * irqentry_exit_cond_resched <- NOP + * cond_resched <- __cond_resched + * might_resched <- __cond_resched + * preempt_schedule <- NOP + * preempt_schedule_notrace <- NOP + * irqentry_exit_cond_resched <- irqentry_exit_allow_resched * * FULL: - * cond_resched <- RET0 - * might_resched <- RET0 - * preempt_schedule <- preempt_schedule - * preempt_schedule_notrace <- preempt_schedule_notrace - * irqentry_exit_cond_resched <- irqentry_exit_cond_resched + * cond_resched <- RET0 + * might_resched <- RET0 + * preempt_schedule <- preempt_schedule + * preempt_schedule_notrace <- preempt_schedule_notrace + * irqentry_exit_cond_resched <- irqentry_exit_cond_resched */ =20 enum { --=20 2.31.1 From nobody Thu Dec 18 19:12:57 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 532ACC83F15 for ; Wed, 30 Aug 2023 20:55:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240601AbjH3Uza (ORCPT ); Wed, 30 Aug 2023 16:55:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240526AbjH3Uz2 (ORCPT ); Wed, 30 Aug 2023 16:55:28 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F2B6E51 for ; Wed, 30 Aug 2023 13:54:56 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37UInXRZ032362; Wed, 30 Aug 2023 18:50:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=Rgf7D4Qv7h/ZhjnYoNWuL0UL8IKa0QgHu+CezcwKZIQ=; b=gaOyPPuh5Id9LbMf+b/4Hp+ZJ3KfRwRQqR7h/P7/6Aw8rLoM529ZJs0gr+hMQnFgtB03 GWd4FESILOaIpEx4q55qjoM8kwVrmO8eH0UtqlenXSDsHBhdJq5D1MurhiDRdeuZ5ahq +p0ji8QwDb1SCUGp/ieWGDmrPNO65/TD7BRhG14E9MPsafI5PoBZee7AC8iMte+wL6S8 7bqluobe26FHcXkCf+60itDIB4nnoZ0uYlSE9OJrrsaTkvDGeU3OvIgjQbtxIIKavzUV MEnwtBy5qjPGQSsvHD3b7sdjY0xY5s1x2EHmzmb/c8cI/UGwXCcEG6/y+HPPi4zrEEZ1 Hg== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sq9k686c4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:32 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37UIFApZ032865; Wed, 30 Aug 2023 18:50:31 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2040.outbound.protection.outlook.com [104.47.66.40]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sr6dqehx1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Aug 2023 18:50:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gHfPsuzo1xpmPOjKpCDP1E1nwy8IG6ne/9OMT+idhGRt5DETdXmshkbAjQALZnMUOga/upkt5fXV9gRDKJLyjVyqRt2dYD8Nm9hY+Yt74ytsbY7eeHPhMtzTQQl7C68rH1hj3EU2igMig9ZSbykfYb0Nm3pflYTCXdrLCcOJj2MRJoItg4BhNhCs9mmZe+fLo4eDo12e7j0Zwx0UEx1XfwqPhI/IFI6wSGdegkUYaq4d2zz7UZ8U32ukjFkqkTQTL9AsbERu+G208HFCKKyEc9jn5gQ3fDVrcGS9BP0E7uWtEXivKdXMjefyjmF4+a+BriDIGHIgro+EtcuayDLOzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Rgf7D4Qv7h/ZhjnYoNWuL0UL8IKa0QgHu+CezcwKZIQ=; b=bXQGnROMD0XnmQU1cnv/5kTsBxy6m+CKzuomgSmOFtBfvAGJiBzzZ95y6AsgwhSNp3DakwTvDd9oy3AuA+ZHxXPeO3nvt0JwsaE/SsHKqFYeRl8P29PHiaO6YdPyAJEH6E8YtantJmypSmwX3JpkYiv+SKADKJuCa6egRd9N8kPX4iFNNEH3ELF+cbkClhW/K1E6fCZomZ/e+vyoLfFNEfM5SiUvoosX2Bg3HqNchz5XR/UEbKt+W51If1nymK0IU/ubmxqyZf9VKMOU3oerf8f3FxSRTG/KGtxAcW6+e1LvXVUWdFI+wT4Pg+tAz/owdG9xwhe/dQwe8yYXPPkpFg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Rgf7D4Qv7h/ZhjnYoNWuL0UL8IKa0QgHu+CezcwKZIQ=; b=jkal+VWNR2kayRhoqHqwCWQTrJ2fj91kcWqdN+jH2qPIrKD9CvHUatIvHJrnaqgWyV3dsnrXWaHzw/Sp1zrwEHuiEVaJUqeC2g+hHPokCMVxOuGTUpsVBXCccFZADxB1LJo77YvsXX1ABpmp8zjSf3c6mTX0KrQc/md+wsiFDf8= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by CYXPR10MB7897.namprd10.prod.outlook.com (2603:10b6:930:da::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20; Wed, 30 Aug 2023 18:50:29 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::238e:5e86:cbd0:7415%4]) with mapi id 15.20.6699.035; Wed, 30 Aug 2023 18:50:29 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, Ankur Arora Subject: [PATCH v2 9/9] x86/clear_huge_page: make clear_contig_region() preemptible Date: Wed, 30 Aug 2023 11:49:58 -0700 Message-Id: <20230830184958.2333078-10-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230830184958.2333078-1-ankur.a.arora@oracle.com> References: <20230830184958.2333078-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0087.namprd04.prod.outlook.com (2603:10b6:303:6b::32) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|CYXPR10MB7897:EE_ X-MS-Office365-Filtering-Correlation-Id: 4e2b1122-bff9-4520-d771-08dba989faf7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7bpP11w80OK83GggkGbTm7S6+CWSMxPJNVAUisHDmCpSS5g47M/4O/0jnSETpPhFXf2pnkeZXZaU/muO9JkG1KqsGCaYzULaeFodt+2zKQGPM0pYFXwcrzP08HC73al74VPevPBSvRVALIgnX8waaKEyHu3kSJSSUhPz5dmUawVWP4ma7Xu+k+zWmVTCKHOKbPTJuos+7sn5ZQQbn2GC09FVozqzGApBxLHZCeLZ31Xsr14FUEzuzuLF3jmUwPqXN/DaDfvL7Lc3+orZH2BwFpC2xb3PyWlRbIKS61196ioHIk4SEfaMkrtXV1ewpjmwTcW9MxAgTiIlGmisK2vx9/skWgtwea3AeHCXeehGubqoM8Zl3MfzBHW2zrcc/Q+QF7giMq1BrzyJT9jpWJUMOU9Jhk7hNbO1JvaKQ2HNszy0x/u2O/RmboQL1TRltD06LuaDO++q+QPF9VuNIPV6WG08+1R5mFyEYmaDoHKgGJTCzwLHvn8DS3/P2f/p38LWoWl2nIyG7ivbYeArtEiJG8iJrx/sTPZlrfX0Vt82cyZow91l8JqyjF4Krn1zVcy9+wO7ldAjYhzEJh9KAzQ+W6Uvfuuh6B17Ouw233g19u8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(396003)(136003)(366004)(451199024)(186009)(1800799009)(478600001)(38100700002)(26005)(1076003)(36756003)(66946007)(41300700001)(6512007)(2616005)(107886003)(66476007)(66556008)(6486002)(6506007)(316002)(6666004)(7416002)(5660300002)(4744005)(4326008)(8676002)(103116003)(8936002)(86362001)(2906002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FA5hLm6enP0ZjokfOmXztjNjgTQCoopSfhfpFsicJlayeGXgYFjRx213wwN0?= =?us-ascii?Q?5hb90B0dAvYFFvD22GrhejjVODcOdmQjv9CN87uKeDAqUvTb3Itn5HQDALOt?= =?us-ascii?Q?OuF18YYjoU6o7cy+YLU/j/pEs5/waFwGqfUPBIvvM5+BeOkkpHijvMP+NyWI?= =?us-ascii?Q?uCL4bXx0oLWVS7vQaaExO+XjpNhJuT/QigFfWwUKXin0cwmIUhUgJODP8W2u?= =?us-ascii?Q?VQKljqS0uEAiJ+F/x2eoXLup+++fcn54iSBBz7zieyU3fUkhykvec6e3Bw21?= =?us-ascii?Q?zcjSfKDos/JXEY98AvwDbDPG6GgOK51oO2cFigk8MUc7DE9Hc9lvuq9hNhCn?= =?us-ascii?Q?y4Qbovv2jzUoB5badqhD48VIv7UxP+YN+qHTtoLfW/gDn33jRNRxSvafisuH?= =?us-ascii?Q?bM9ZKa5sz2XeYrUZZpy9sekmNtbpsy+3uZmxvcqnAp3xnxVqKc87TvHaOyf9?= =?us-ascii?Q?a7yzhYlhR1OPuXOjjmyIo4Yl61laWxHwxGUkfkkdmjdDPJAkEOLKQZV8BWXC?= =?us-ascii?Q?cqZIxxjTO1LCaa9CAyi9jW6tf9luXiTqYv7c9MX3R7zZwdaB62GGUQB7seAN?= =?us-ascii?Q?mx8kbMiV0MVhMV0XdP1m1KIDjWvdpaYjSZj3p7y3l5cbydK4NVwEGZ5y7Uo0?= =?us-ascii?Q?4QQPbdJaigBh7v9/VDw3fvgUkdxw4TM8SnAV0SWBO5Shc3CwZajDIV/704GP?= =?us-ascii?Q?37yC5oNGFPxFmo/V3Kjh2udCC+KGWEDapT/6w1PE0Pn5htfGKY8lASP2g9+x?= =?us-ascii?Q?1QrABCjB8zVIXy05F7DHSyQV/gXPJnLLNRQuTaPl8no4jB4jzu3PoyxnMFGc?= =?us-ascii?Q?DyjIQ1915HkIYKb0SJ8kvSb5EOCyGZcqL7h9ZPxrEaNH/A685ljSD2RWMjjs?= =?us-ascii?Q?TDtYkYjhq+sNkDFYdaEgvkkj0XysjN53cBLIbj8pcsVOQxq3wYgBFIoicvTT?= =?us-ascii?Q?7VVzHwzQV+XdYNTFUyrMo7TW37xTT9zciccV0XCV30Fqhv5KOPnhl14qPVTq?= =?us-ascii?Q?RBC+Kp8FuE821wxKKTizes18wlAQyu1TxaJL9r4j+d5kNz4tWyNjiPc+9UDt?= =?us-ascii?Q?v/Jtt9g7g6SiROLQR21Y+yP0PUQfeAgtUgv+CuQm07ceOdor5SkAnQkqp3vx?= =?us-ascii?Q?rzvD1TMcbc9zwO1AJEz/OU9Z4qoYxJkG4VaYK9Yp3ooDBckp+qssys7qUdDZ?= =?us-ascii?Q?TkW7RUNP1pgm++H5JS1EknHyHGOPgB+tZYMaquvpGcDZJeEvtiZUgisvSLHQ?= =?us-ascii?Q?LX7L9lDsUoX4gZDVebbMnFLcaekj52G+YSj2Z7gbTpxY6K8zIMAJw40na/HL?= =?us-ascii?Q?nO3sknMeSvZMiIdx1RpUL7hB2C/Tbvf1W0rD6HJ7WqvRBnKgY+5vBTGeWLGW?= =?us-ascii?Q?Y8oUlpteORzGIBAIlbumVkxNrUVaHRhRW/KNV9prRwwbET48+xN2E5dcIteN?= =?us-ascii?Q?ndHJTjzKgKOhtvAxgq+reayMd046YrbGh6103euaq5LQSaa9PxHjgxtP/3sx?= =?us-ascii?Q?I7EjLNRcSX23+uQStuWzWVPZK5H9qQ/gLlPNNRM4aej9ES6vH0UBbXJALt/U?= =?us-ascii?Q?Ig+l9qoG8tftAH52yHXNtb7p4HmQ1Ml2+GRRlsx/DCTuV81yWo+DZknDas+2?= =?us-ascii?Q?ag=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?HUbLtWDhK/62UC72yeOCUMOcxTVvssqJHALfhyWho9VVzaNJIJtsnG38tTqZ?= =?us-ascii?Q?0lN33urtioUO2+yYI0bs9cVy8lirRofDJ145eahdYkCeDaOsam3fYkgKA7Ug?= =?us-ascii?Q?W4xO0AZvAbTwSgCQu05qYVUcl892Q/eiSPupMw1KeDkKj5oWwTb1lejNij87?= =?us-ascii?Q?8SaCF6NFCwmvxdvyTfZpvz7Htxb9plxExTxZERjKXFQ2iYEiUlR3RK5ohKsT?= =?us-ascii?Q?DWWC4NF3MNK4WwA0/Vzh4+NT6Q8NXRLkUbqkIpCOb/BIY73c3IPWli1KDSt0?= =?us-ascii?Q?z7gWbGHRRavpOPeoroFi3/3OTm7wAur4fVyROEDDWJ6xUiuFGjN5c61N3xPf?= =?us-ascii?Q?BYE8NodAyGQ2TrfurS9u5sJ+LRM2WIelueAd3EKQZQZulrvYK0T63BjxSdxI?= =?us-ascii?Q?k0tYiUi9mQ/2TRZ5lE8hOQt2l1NIdpgLDQuQEaK5Vko8BVaFAHGuy60kcv44?= =?us-ascii?Q?rK+TlChQ4LWXwWmAT0C+iIfS6Xw0QjZlNR+z4vYXzv0C8YACCvt8Pmh2UcQR?= =?us-ascii?Q?OKaZfuywSwzm685uaMAuf/BGWPjeLaKpc1nrDUINsdc1jnDv2RFJPNOb42Xi?= =?us-ascii?Q?rcb3FdxTKixGW0Db3Ol9onhCMucXWE68MHcmHF4A5YJTjBCjcSuRC/v57B94?= =?us-ascii?Q?g57eT7EpQQxTUka1VZscuM5O/q7shlwarXhBNecUnKxl0dsN3/NR8pfILNQ9?= =?us-ascii?Q?zbdFv4VJrLa4TKPTs2Me3/5lcguTXeMcAygjy1XLY9oWLnBl2cYy1WmehWHq?= =?us-ascii?Q?nnHc9rBDydLLU3tLe9R5tkMAOSma4c139b60GtGMqHRtwfplroYU2mYPT10g?= =?us-ascii?Q?DVhJB8LS+yV9DZc8n9gFQZTiRzxOWnSkT5izV40bb/mMR88RYlaGRn5aDmCG?= =?us-ascii?Q?HLC+MWQlGkLT2BGYIDY3J1sjZnLlubBXrEoKsIgNcdwcDDLo5kGyK6DtDMtJ?= =?us-ascii?Q?OiYMQIOSsa89o8earyBo9q3yKyfdKhf/avdONH8ZGa/FDdBZT8D3y0s373nr?= =?us-ascii?Q?vyOHfnCpOcYG2MQXrbNnAZGy6zTvOuPm9ALyEssgqtr72B3XRmboyWG/1MNX?= =?us-ascii?Q?NlFYiqECsj78r+G+xfkyyYgtrgnZCXaTaNp78TnMW4/YRjLg4MPt30JAUFx5?= =?us-ascii?Q?8tsf2t/Ht7OYCZYpOgCsH/NTpdTUPlOwixTcc14mqjBFzq+bQg77qN8=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e2b1122-bff9-4520-d771-08dba989faf7 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 18:50:29.2182 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: c1jpT9NX6HKUIXnsCA+J4aa6hsaL19SCYYKBxDawvmAJXp3LITvyxtrC1sGQ1e9O92LGGqEm98mhQnhUbqFKRycem7irCnZB1bNUmbV9g2g= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYXPR10MB7897 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-30_15,2023-08-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 adultscore=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308300170 X-Proofpoint-GUID: fevrx8sv2rJAwQDtN66tvTm5CqoXCAqk X-Proofpoint-ORIG-GUID: fevrx8sv2rJAwQDtN66tvTm5CqoXCAqk Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" clear_contig_region() can be used for clearing regions as large as a gigantic page. Allow preemption in irqentry_exit to make sure we don't hold on to the CPU for an arbitrarily long period. Signed-off-by: Ankur Arora Tested-by: Raghavendra K T --- arch/x86/mm/hugetlbpage.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 0b9f7a6dad93..55d1d15ea618 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -152,7 +152,12 @@ hugetlb_get_unmapped_area(struct file *file, unsigned = long addr, #ifndef CONFIG_HIGHMEM static void clear_contig_region(struct page *page, unsigned int npages) { + /* + * We might be clearing a large region. Allow rescheduling. + */ + allow_resched(); clear_pages(page_address(page), npages); + disallow_resched(); } =20 /* --=20 2.31.1