From nobody Thu Dec 18 01:07:41 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CBCBC6787B for ; Fri, 25 Aug 2023 19:06:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229712AbjHYTGS (ORCPT ); Fri, 25 Aug 2023 15:06:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230290AbjHYTFz (ORCPT ); Fri, 25 Aug 2023 15:05:55 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A28B12689 for ; Fri, 25 Aug 2023 12:05:52 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEo51009220; Fri, 25 Aug 2023 19:05:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=IaKcsxb0rORIYwEUogPfq5hXZgITbUVS8py0jJm+tS8=; b=xGHrmpjRHuI4SZC9aEhUGA0vOyePfNynrPdGtR5O0oCvqwmF1se1t0bEu/rm77E7cJ0Q Dl2geZMi6P1Hndvel2G7wrecV1P/xxvrzo4gPLFm7d2lCjYP0O9/a5fBwl0qLzFMcnVr 46b5RqXXZvFvmjSVc2SvQcWWimQi5jC6tslA5TMFcsZEjcspRuiUWSQzWp8qhz7Ek803 MdFSacYj2DrrKolaH09A6XX82XG7eEVahUE/z3WPwmReyRy+GZSGPMPdlTWm6h3zhk0F 6HQZ1h0v5G/TVjo9Q+YnBSmRhoUNKJZl2NUox7KkpK+Jc7O6rTFhZ2Afr5cqT2ecLjx0 6Q== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20df0ss-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:01 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PINZgD002154; Fri, 25 Aug 2023 19:04:49 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2045.outbound.protection.outlook.com [104.47.66.45]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yyg87c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:04:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iW5DcikoTA78u9+ccEbN2VYVGLJbVFfbRtNzag6Bn0qxGSX0BImdca8IAyt29D3U30Bzr5QdySfm6auaFFeATCnKcixEskHtCl7WrixtdLbW+PYUo8aHl1rlZmSGQEA5Koud39YBtzRwQdOYvXiHg3m0qhOU3Yuv3QFzG67f6u8EgE4jEuc8nmDFsYVuOtMivwYibViUsPYSRupTgPAuY0VynyLnCDhXoS8de8eHxJB1SuTrLuHHZWRXNcplOStxkx9fm0a2/zECzmPbpIMkMrfABVbG99cUde/XvRz3sUMfpHUpVecvyzj24SXF+MWKj0CvJNS/eWNF6tcV9QIEVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IaKcsxb0rORIYwEUogPfq5hXZgITbUVS8py0jJm+tS8=; b=K1CkCPj7lh5kQlLA1ihj4um3wmpx9DY5RwYNs/zuxHCKE2EU4OBdr36QTgcnwghzqI1jESm0v3LE36BZkWZUrQM+h2LjIvavWiF3/iBt61voGUhvaokWvjb/UfVhWeaMESz+Irwu/2WAij4uhYPEjkHj2A9zfm9gJaPu7cyFWIfKDJ84gCPUNBkuuP/NQC66lK9C89jjRE6nxtiFuAJK0iC+hp7z9PU/DlWJVxTb2uuzCrkFdnyAgw3bG+vjwMeCycxAxCmdjxX1rHtc3ees235Dr0C71zapuWgA9UTQjfFzCDdyVQslOobyXYo/X9nQIPlLs0Um04Yi52JCg3dEmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IaKcsxb0rORIYwEUogPfq5hXZgITbUVS8py0jJm+tS8=; b=hI4Y/O0qALFameayMJqP+ScXqE2PGpVnVUhrxPui+6KD7R+U5NFoSFrBxfluErZ7Wv9si+q6ElZiT5GQU3Revv2c9wMpi6uDWbHaL2+ttIJroKfAKItWVeCysoAQW8T9B7B7Oey132R0QqPrbqPyQl2alYiQwtJl6lMPvAeR05Y= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:47 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:47 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 01/12] hugetlb: clear flags in tail pages that will be freed individually Date: Fri, 25 Aug 2023 12:04:21 -0700 Message-ID: <20230825190436.55045-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4P221CA0008.NAMP221.PROD.OUTLOOK.COM (2603:10b6:303:8b::13) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 95a40a7c-5363-4624-817b-08dba59e2671 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yP3DnUyfzMA4oTljGSq/AmrA+/3u0aNjm14qs89BR7Mz6d3/040LqmaqMne8G1kl8fqgFqFXFGbiKTEBU2QmnQigbydAc6orUkFlIpGP9EAj7l+4Iuwem3lOc4xAxxorrdQQzqJ6T5gcSdHQxSTRO7xHaK344ZN1AALSCXXz1Pw+Ei8P/uCH3sDbNX92ODvGNgFq0cp7sez75JHfB38TFvIAJIypBO3GAvbmgZFnCIyX0Hh32Ih56pJCltPKbBVACituLkmPpIU0z+DTe4rRohWVMaGXN3yvkiwhbrEoUFk4cVmbNVdefEIHAlMFm5Ycxs3s9iOUKSRjFhoO3dFTXsv/bGJJHK71UYdq+RyBdid9+mjI0kn3BeTvfjLhVefFnXoJ7j8crNsvO9OOCRb3oDGQzoQVM6aTjZEVd/ZVKyPfYor98jnpfuMqibX/DiU5qtVDfJRVgpfn/jwGYUlPsXebI6hdrxa0Q5eNcWIOfxMfUyiwfrzF+9Oi+sb2M43gV+efRSRBiCkq/wcOctc9Em2Q9xyOtxQjO5kjPh9Ww7zHsRlm5KKL5rchdcPRYNKY X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gbYYzec3acpvkvv+xt1dCMgHomCJZpfIZ9VGWdtArL61Yi6jJbneOG3USNor?= =?us-ascii?Q?KXbUK9foBJKcCUiL3nsL2AG9tTZuW/TxL/drExCxsnoD+OVtEy4sbz3FRGPd?= =?us-ascii?Q?D9fur6M4DgiX1hbtGttf30n2T/LibvzrYHzqrZsz7AfyMGI1eilOpulOFKfH?= =?us-ascii?Q?yObV4Hx11+1kJ2at6jpNBkfx2xskhHuY/ogzhGqofQz5/vpSYMrqUqU9Npdo?= =?us-ascii?Q?zrmBp5O57FvSaSdSenCQP11stXEbGgQnOiLZ7i7cOcKYsDFe4a0jant02TmL?= =?us-ascii?Q?dO6Md1Bvih3r+jtuEKPdv9hdtw3ktxut/1HtZBhdfuIEzGDeDaR6BruASUgS?= =?us-ascii?Q?z2io8peFRdzN8GL3QCDs+UDI5OP6IG5KTx6nI0h1yoc8dlOgBuBYHuM2iY0d?= =?us-ascii?Q?iGrmK1UbhbNCEcQNmDa6FkuPRA5PZuuI7BlW7dcaWGQHfJngoLUjQBbCOlzQ?= =?us-ascii?Q?bPGR5bkTjzkFWC5zb84mvBn1GoT0+CNFCgOBdRAr+LAkgc2PogTSYy7tiPJu?= =?us-ascii?Q?mPOR3t33wBUXEyDWrSi5sCm4JcIktkRf2HpcBj7r6lJoodgbabIX5wQOQ7pd?= =?us-ascii?Q?c+3oThAHCLZOVOwrCzGOMNoRxcjSidfDZNilRzdZyXQ/rkxgO/tJ10HfXQPF?= =?us-ascii?Q?bjQl10Otq/aJVWygzw/iKiSJOuAhRlX9nJhK6XpBPHiHs2PZSxAY7PrtqPzM?= =?us-ascii?Q?qkmcAb0SkTbAUuXm6XJaDeLoTdnOPtMARBqMTqtxdhIWkIf7Mf3c8xXvsIGt?= =?us-ascii?Q?NVjf8Q7ZwMPkiqqYDG9NbdtZkNldVRzgTbnS0nAia/3NWxUOchfyTr+BCWg7?= =?us-ascii?Q?0I3r0fQzAhShfO9ErJ7eLRn8H9LAaAn0K9SHXKpZHl2109AviA1MEac41D5D?= =?us-ascii?Q?mCq2obtPe1yCo+gc7rXsXEFtQSmo3j3+QM+xLUx4QHx2/Gp/x7Ez10Gf1JNl?= =?us-ascii?Q?DC2IPf3wDww/pZ537E8p29uEm4tDpNxUQkifIBHa3iE76GPuLsR6t/FGzuOq?= =?us-ascii?Q?ZwyG4jkABwZBYdJ8aCIfXQWxTDYZNieovcnhsPvsQim2G4Ano4MRcRlhvkmi?= =?us-ascii?Q?j40j2ljMmKmP1HhI7nNZhTmPmeh6uAPKqGGDhwrdH9hU2XiIZs0D8dxFvIRa?= =?us-ascii?Q?7DkqWsLZcOQsuQQTQhT0MZehkZuttIYrsLS+beZWz+szbR6hfDSxbz/dePTy?= =?us-ascii?Q?h+ry6CbrZiU9zcgeBU9W/l1DXLrHnyoMEWXDsrNrwa5NzNSzrW8Y9jSHf9LG?= =?us-ascii?Q?2d20DsgTmWOMmYgBhdaWUZVYgKkjK2JFX4xclWNIx7308X/ki6KhG7g4uzVr?= =?us-ascii?Q?YBssyJkaYUxux/QOZ7LNubBC65yKqcAtq6ANTu2DDPhLpJphMaDi/A0dG5VJ?= =?us-ascii?Q?XF8KdiA0zin8+4iPu3u0xDY9UZtlzDqNGUwqzyIj3kNwFSMwxQ6GhmSsS5MM?= =?us-ascii?Q?BkBKEotfhKwzBlwiFxPYke7urjVRQWwQLHwcBAb0e5x9vHOgiv+UOj862HHU?= =?us-ascii?Q?jbDW0D+G18ZUFXytcsHE2GmD2UVC0I0ZU222jlCCuCtM4PyUyV2yl/UeEeML?= =?us-ascii?Q?lGLk4lifIZLFcv7BYwxODEYRHTvasFRU/I7braoFRevoxFsDjT51rmB45zkK?= =?us-ascii?Q?wg=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?mzIu7qk05iTRn6vT7c6E5B1QYvfsUZIyAeq6AUzsp1A7FLoTBCy+Ci3Sdb1S?= =?us-ascii?Q?MaXhICJcm9tSUGGfsmnw8EQ0meMkoRvf6QWHSuLX66tsYjRqOgyVnT4LX+ZM?= =?us-ascii?Q?fDe+lp9FMaNlueEWRiH+yW50WRyYbj1oQaQ9ZEZLcXHiM9eB13Uw5KNkSMCm?= =?us-ascii?Q?5awsZmKnsCXJXWz8YzGn9VMdH6EPxM+Kf94+f0Lbb8MBI8r4gVKzlLwkoQ+h?= =?us-ascii?Q?m0nXwU5TNfGP2T27XHmnQe/CnaRgLOilNcmeHEpsqR7WGNYkj3VsImd9FSA2?= =?us-ascii?Q?yNMspvdkogETD0JHiVqczOdhMmYEsqQwpEZ2AcB1n6MpeEtmnbi6xV5aUTNH?= =?us-ascii?Q?LMhOobmPh0LmpM3adiGG+WI8p18J4ytuf7rTHxFQ6v71Z+F/j5Yoh6AdZl6e?= =?us-ascii?Q?I/3fYNyy2N0h4YTRt3Fc3CG9dqoKO+jtApOXQhXZYNp99zi3T3Vp/mggMn0n?= =?us-ascii?Q?U5CnBKmEdT/QdhYnVuSwliTc1G601fvgXQl7UKUpN6viZf+piVxCd5AJOrLa?= =?us-ascii?Q?VwGCIS3CIQjay4H+T2pVTVnvmTce69MbU9rO8x1+D4gAQPF3P3UEosKv03Vg?= =?us-ascii?Q?drBoPH7tv9Sf+bPJSsNur4HdKj2uvXhzovBSBauEWJC+69Pmf75i0w6unMaz?= =?us-ascii?Q?qta6ZcrTg4VtvlL4YvOporNsSDpgZonCn33eT4MPvP/LiEjNeXDPTN8RpKCq?= =?us-ascii?Q?GlEJVgdUPJMbpX2fmiEaitDFwGbAOdRsSRzPM1A7/EWy9QO+3X9SMlZBU0Ji?= =?us-ascii?Q?+deY3IuTcfQSreSB0zC8kEliyRpE7kh5hufT4dyXFXsXaxsYFQ/BdxesP6Nx?= =?us-ascii?Q?PTtmSEbHHbDr3x6ZLndzISDaDGDYaPf6CcK+RjVCrrBeJIZ/OrYg9wHmroNe?= =?us-ascii?Q?LRUhJpqurmdNUjAfCrVQLONAVqnCyp0mbMK/CdduXmCtxL31HuQYrfbRpfzN?= =?us-ascii?Q?PeQ5rW8H7VLIQ1hbFS05NtIcWq1T3yTvK5iIUoqdlaKLFyirf96nAzfv8Pff?= =?us-ascii?Q?FGEQXszv3nDrUmOuDmJlsX/uCRzX2SSd9LemsO2ETPPNoBgxNAaXvdbnsikR?= =?us-ascii?Q?xJCmkbBayfNUZ0ADrdgZISFEUS99EqW177+YUPL72RGcUFXeIq2NgS9c6xtg?= =?us-ascii?Q?n68LCMGIOZNA?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 95a40a7c-5363-4624-817b-08dba59e2671 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:47.3441 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tzXB29o68g/CAMe5RmHwZw+C5GlRk8ogcsrK+e4jDYuVSRtyyMgOp/18T1Q9LB12PbH+PndMRTKUD7AHYbqVZw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: 3i6DDmuniE7-GNmQInzbcIkT2a_yHKJ7 X-Proofpoint-ORIG-GUID: 3i6DDmuniE7-GNmQInzbcIkT2a_yHKJ7 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0bf70f27c1ba..ba6d39b71cb1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1489,6 +1489,7 @@ static void __destroy_compound_gigantic_folio(struct = folio *folio, =20 for (i =3D 1; i < nr_pages; i++) { p =3D folio_page(folio, i); + p->flags &=3D ~PAGE_FLAGS_CHECK_AT_FREE; p->mapping =3D NULL; clear_compound_head(p); if (!demote) @@ -1707,8 +1708,6 @@ static void add_hugetlb_folio(struct hstate *h, struc= t folio *folio, static void __update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio) { - int i; - struct page *subpage; bool clear_dtor =3D folio_test_hugetlb_vmemmap_optimized(folio); =20 if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) @@ -1750,14 +1749,6 @@ static void __update_and_free_hugetlb_folio(struct h= state *h, spin_unlock_irq(&hugetlb_lock); } =20 - for (i =3D 0; i < pages_per_huge_page(h); i++) { - subpage =3D folio_page(folio, i); - subpage->flags &=3D ~(1 << PG_locked | 1 << PG_error | - 1 << PG_referenced | 1 << PG_dirty | - 1 << PG_active | 1 << PG_private | - 1 << PG_writeback); - } - /* * Non-gigantic pages demoted from CMA allocated gigantic pages * need to be given back to CMA in free_gigantic_folio. --=20 2.41.0 From nobody Thu Dec 18 01:07:41 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46505C71134 for ; Fri, 25 Aug 2023 19:06:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230239AbjHYTGX (ORCPT ); Fri, 25 Aug 2023 15:06:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230422AbjHYTGA (ORCPT ); Fri, 25 Aug 2023 15:06:00 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39EA22690 for ; Fri, 25 Aug 2023 12:05:55 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEo4x009220; Fri, 25 Aug 2023 19:05:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=OlqrnYCf/SWJqiFiZ0S7D0zk7AgEaG4b4hT8Gbzo+AY=; b=ei0QlP/+liCl+Im+OJXOaqmMYdMvDMx5nDqljH96E061eLMvfbYZ/PxX+5QzzsjOgAAh JEEo+g/nFYdHKCRv+Z22gdLLEN+/CuNaLmdrvCumKWlTmBZHCl/9r/YXtNeq6LDQkShv fRoKJaWc48RsojFD1ktAWqKdr7oPkMYRjXbpdlZHLDO0k4MrY9Lgt4Wf/Uzut45LqmRs HqzaWMY5gqbVrec/PBgZ6tIhA+rBdDcl9XrfjxNS87kJuhJWYkEY8RSrdA9aGQHn0ezj evEZlbzEGTPY4KrPfZlHcIDzrpJ4kOp9QgnGImLSW/f/JdEVI27cppH2FUyV+Tojt6fF DA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20df0sx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:00 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PHGikX035838; Fri, 25 Aug 2023 19:04:54 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yr77hf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:04:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VJte0QNpO8C7kl4nhXIJxycgoNmMOs5bFcppURIfA1qq3ml2kW600BkPUL0NxquaeXoEoy0VoB3ETUT91+rvScI0R+dwvaUWrDp3YD3qLqUGhyzQcEWWvyPx9MSJg2C5eq8JZ5ApGvJSpPYy8dNYGCJgHbuJlQgnPGuIjWdW2nsIMZcvt+So/Ej8SYg6M43ELZgRev//98BK+OMuaSXFD46tEIvjxiXY1i+aFMIf8YYmO7e/fUvMzLndV+u/LbX5J2cEyURGUOyRnV47uTZ/aINPMkUtdH2vV2pTVBpDoQtVKXxG367mnMzABVne6n/hwYn4fchUDa85BNXrMJ/SIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OlqrnYCf/SWJqiFiZ0S7D0zk7AgEaG4b4hT8Gbzo+AY=; b=WO7uGu57sbCaKFXiUPYaF8rlHDkepte9SQHP+61H7kNq8E/WrSA1S384Zrc2E6nRt34jHSEblIkmtIjIVC9Y4GANvxi28QtXJoD2rJchDrQZkoo+EmZJjyzI5imFsB0Gk/Z4cFb30mifFRjgfONmPAlL4GGelD7CBV+pOjEXYcBDbG+aQdn5zfQfvshlpdHBB/tesgWiSefFMKwyLIqHUaurUt8rER94BvP2Q6X0MdfjGgr2lVTXgIQX+X4Pjv7y/7zeeQ9UJRT8Kdj1BBxCwnn5qagmNKc8x99UesmLCqtXT42JUxuLfMAQGbwnTxSjBpDVBcm+dX6ORHG822O5gg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OlqrnYCf/SWJqiFiZ0S7D0zk7AgEaG4b4hT8Gbzo+AY=; b=So5lE5/0YT4nQSyNfRykSNZfpIQqGHvkWD3KrdDYJ58r7cThwwmwY8Lz2hudWbZoKs6OMaNov3L6D3cgvKLT5+URnMv6GESizIA2osXpCsnnZSrsx+yl6YwBlKwtcSSY+LoeGqghopUYV+3MfGy4clwaXTl3858FWXARQRJLrJ8= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:52 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:50 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH 02/12] hugetlb: Use a folio in free_hpage_workfn() Date: Fri, 25 Aug 2023 12:04:22 -0700 Message-ID: <20230825190436.55045-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4P221CA0005.NAMP221.PROD.OUTLOOK.COM (2603:10b6:303:8b::10) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: c2eb6754-8598-4f6f-141e-08dba59e2866 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rFQROwA07L7FLFzZPKiExSpb8Yimk+ggo9TaCdjXGdCMBTV6yRktwGkVDOp9p6SV7QF1yc7VmAyfrYFDmXAgNyNqtZaS/Qz03ABJvxzd1puw8B9Y0hE+z4EOUWBcL9NvzMq1LtGA5SZknjUC4YfEy2eNTZQTIv6P8G4XgLJFJ9O10/x17kZdlBtjKO/KTe69ssRPebbJ5gfcOwCexcf+asUwoF/zCz9T+5g6DgoBFqOAWeueDdPK5I+SqMWqNUqv1DTwrhhlbvv5ft5pi52yfRUyOSI/afuitf2m3AR4bY3AOYvxqtuxuyhYKROYcU5jvgWJdpSmXmN1jvYm2ywU85M9P7uWyjMhaCFvIm2XPMhRDp9pbZSRB+UCmWdJ2HYMHrni/yC4S0xJ0L4eQqvtuAX/sNkiXstQp0mUi5Lato5zhpxEwPzXpmtRQftCh8v2GswbtHBgRunoYnHqxbwgOudOugJWgwhwxqODe5eMlJxo2+YESFCbIaL+547jh/i0UdCR8ChutF0zLGLTSi48sxfoz6/vmptiVDCUw0gmp79HhcJ2s3OvSEDdDd8kTPde X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?OxeMLk9UYBeWy0q4+wFMYdUexxnmDfa9GZaIKj2Qx1EJ61G6G6pMIKkDiyI4?= =?us-ascii?Q?vaTjGjAztj7EqgLQu6HnqDik7nmOH96ohcG0WTO2AZ9S/OS9L137Es4TW40/?= =?us-ascii?Q?wYc4r21pCHX2uYNqWstK5FyT/1V06eMYMFSVfoH+3/+ebpeZHpkntYRTn2CW?= =?us-ascii?Q?LhuKJVkHgTXtqKwDuIQtacaPYu9e/mynh1B142QKBBt1pdptd31UoRu906Wh?= =?us-ascii?Q?lkKqKT/BOeTbe/gjqZYup6R350SZgzNf6xYDgwcNLze89TbXoSZYo6DEunLg?= =?us-ascii?Q?Bl5vkLMRazqqrkIaElqWItKgl5QNlhAci6V7TMfa6/tjdnjBIb0J518flN0B?= =?us-ascii?Q?ev5wbGf18fAos+Lv22geWIN5KKc9/H8+fEsvL3HQU7bAuzgQ2GjTeR+/FAXn?= =?us-ascii?Q?Cix8QvvGFxlCNICmuXGrhuPGSzq9Q83ldfIvj4zp0RhPQjI2kQb2XIvKL/z3?= =?us-ascii?Q?ubTSwDAolX/VgT8WzNW95m9NUO/MvLjItliTr95FD4odvcZSWP/HnRffx3oU?= =?us-ascii?Q?OqthoAkTUB6c+7RDusVKYIP+x54FJFjzvPY5r36Qd0MnvZmSyo3bkQR9vuvM?= =?us-ascii?Q?xssAisngPn0uavhSMXTx4/yU70v4AnwFBGvpfHxpmfGh2GzZZDNJ+bzg8dEX?= =?us-ascii?Q?+9gZuv/vogdSDwXRrtqejP9y0ILUggthfIns4jGp4pV5ExFh/4uJLAjv1M70?= =?us-ascii?Q?PDs0l/8PpjmHXFbHpVxnnuOiFrCcvPN708lOcEmPR0KmuiHrrhuusnc1GDCZ?= =?us-ascii?Q?3D+RwGpdnxlAF0/orhJnWhhsfPtEvm0txP1zSs3ix1MrvJ1vBqJ8RziVxuBf?= =?us-ascii?Q?hpEbC++Je0zIzk92/17wG7FEOgMD9Qe48V/Y/+NP6Mu6Kpa3gWlItITjykmm?= =?us-ascii?Q?8I4h3uaml6u1kYTSJBvRr/iZHsfvSulP1FSdKIILZC4jwvk/I89gS2cgYDr3?= =?us-ascii?Q?SGc4chH3cCgQl6v89s0dU3X0xa782+DkRWcK89iIz5BbiCELtNRXlwzwak5Y?= =?us-ascii?Q?nWmzVFmTcP1xULD6Ni5ETOQrl4Ky8H7eQ1z+7Ru27bLxp8Q01fPXPHGVYhFa?= =?us-ascii?Q?CV/kQhdd1Dx1SUCOGGgQLYQ+MFZodkSbs6ZCity6H5eI3Et4wrDCMd5qf6m1?= =?us-ascii?Q?SYxmwOaIeQoYnh4aOaRlCLlR4ErCFGqfolGaTXbAFoM0hLpUlsH0LRGX+70x?= =?us-ascii?Q?hlJCVca60nUlzldGIBWqhQ2xFj5KFApnB6KGwuhUQdI3Yix5EkaJOp0Oxilq?= =?us-ascii?Q?8SXxcvyrXMHy/ehH350UnvSrGAGhD0MJRLFondI5RBYSpXPkSwcmx62VFz7I?= =?us-ascii?Q?L2BaQtxSrDFJKx2DMABCphB19B51kItFSK+AaOrHyTzv1cbxVY7GVIzgsPh4?= =?us-ascii?Q?/QicvifFL2O6pnlrVIW/f6k666h/IkIpl9PZsKvIvQf0cqmlO2fIc0+ClAr3?= =?us-ascii?Q?VRPXSydntt7sJlHiKif+ehlGKQ53kwgTZuSvC+5Fe5Ok7xL50X7FnFCMtq1J?= =?us-ascii?Q?47+fBGmnJdnbltjkRtDkOaoPiKp/LuqxqOgty5dOaSRVq/yQWx5fD+YjJ96c?= =?us-ascii?Q?qZH88VSOm+BE7KMLPHRm933Qhb2PABPNtnWO9ajx?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?ccOTViSzUa9HtmR/xNh4+bgnKGrvIrBz9idEXsdlqDDDxGIBzTAgFMBuS6Sq?= =?us-ascii?Q?gPPsDPUTA3M1OJZE2RTNp/SJNePEDnFAZFTFZ5xjW7EDOsyB9xCRgr7/ghYH?= =?us-ascii?Q?2DKVoJM+5hzU+tz0AxH6bTuz3wOb6qy9/A0uhGQefez7wGzWdMl40xIm7Wmy?= =?us-ascii?Q?srRgEIM1ZpvehAYSVsQxRRKkdkiLE0NdYEX6TSNF7z1PRsoRmftg+6Kj7Yuh?= =?us-ascii?Q?IA7iBmg3WCU13h7NSEKQkdz68V8nRgIW/aWwmy21lHjLzZhUemLaUQbiK4XV?= =?us-ascii?Q?SwQCHrGRCurRctI3mC0omXiv3ELoPDNMqHMHPepB4Gm0ZsCgLbXl4b5mMmHN?= =?us-ascii?Q?reFT6wKSAGyVNX7s/GyEkqZVy+pVQTsz99bq1eEzQbABPnSrxlfQou92FZnS?= =?us-ascii?Q?MrAv57a/N9sWJjgjD+UtemGs+pYNMxNZfoXzdKz9PHHPxS31uAAiMwxV8Jyc?= =?us-ascii?Q?4r75fWDajcDSMANPcusySccIu+hKTNNRlKbkL4AXcpKgRVdEsV00D1pMqMn4?= =?us-ascii?Q?+fRlLugWjRgpRSePZYltUEFy3ANMB/N0ReQzxPIttXygZefLMf7qS9a3tTJZ?= =?us-ascii?Q?+/kTsWoqKDzOAdABmLYVJMWccbLcUm1k0B2ksTKF6qxP34CNibINF1OQfJcQ?= =?us-ascii?Q?5BJHNlOvKKAaDvv57Tc8hXbJhdNaRhyewuGRzhRdVUhG7NWihOe+pp5/Y6Bn?= =?us-ascii?Q?tUpwmTOlmmq5sipcvTWcY46zQTK+Y8Z/zbdSiheLYI9PJtsd4P33aeUMh78b?= =?us-ascii?Q?hnuX4KonpJb0EpSdLlfFg7fckw6KsDxRz6w8gDD2EfyaUeJJZLiW1m89x6O5?= =?us-ascii?Q?IyOqnScoyvvY2dOrR3rnIQ2zvofhnOA65Y/BPWU9RpsV44zQMPIJt/A6/MGK?= =?us-ascii?Q?G6bFVrHoNvGh9G83p/RCOpZS6t0JJJg26EGeEr0UPkmzvGnRrU8LaEH8kxbw?= =?us-ascii?Q?NBCr+EsYjscOHip8dPjia/TuyQ4iTkzlSWY3IYAcgq+ji06Ss/IBnIPd2wEG?= =?us-ascii?Q?UjnVQuQ8qn2KqERE9HgQ9PzZTGO2IEDCAaN5QGsdSJlYtJbZm/r+FRgdQ/ue?= =?us-ascii?Q?BKK91T82BelwsspUavlcXBln/U3xWnDVzYx5SY2VW/kJTBZNjMEPFwNdfzKS?= =?us-ascii?Q?8XQsQDGTl8Zh?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c2eb6754-8598-4f6f-141e-08dba59e2866 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:50.8976 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 11krfbWWK/fRLYx4h8qG5XbsdxH8bVKXh/0SrgMH7+r5wvpd3KV2bb/vR0HEkLJxb+7rk+NMUsuNnBikrqP0Gg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0 adultscore=0 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: RjUyhRaDvbDSPDQPjxjvqQ7jGG8PRuKV X-Proofpoint-ORIG-GUID: RjUyhRaDvbDSPDQPjxjvqQ7jGG8PRuKV Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: "Matthew Wilcox (Oracle)" update_and_free_hugetlb_folio puts the memory on hpage_freelist as a folio so we can take it off the list as a folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ba6d39b71cb1..1a48a83846cb 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1782,22 +1782,22 @@ static void free_hpage_workfn(struct work_struct *w= ork) node =3D llist_del_all(&hpage_freelist); =20 while (node) { - struct page *page; + struct folio *folio; struct hstate *h; =20 - page =3D container_of((struct address_space **)node, - struct page, mapping); + folio =3D container_of((struct address_space **)node, + struct folio, mapping); node =3D node->next; - page->mapping =3D NULL; + folio->mapping =3D NULL; /* * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in * folio_hstate() is going to trigger because a previous call to * remove_hugetlb_folio() will clear the hugetlb bit, so do * not use folio_hstate() directly. */ - h =3D size_to_hstate(page_size(page)); + h =3D size_to_hstate(folio_size(folio)); =20 - __update_and_free_hugetlb_folio(h, page_folio(page)); + __update_and_free_hugetlb_folio(h, folio); =20 cond_resched(); } --=20 2.41.0 From nobody Thu Dec 18 01:07:41 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F026DC3DA66 for ; Fri, 25 Aug 2023 19:06:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229807AbjHYTGU (ORCPT ); Fri, 25 Aug 2023 15:06:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230291AbjHYTF4 (ORCPT ); Fri, 25 Aug 2023 15:05:56 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A27F42137 for ; Fri, 25 Aug 2023 12:05:52 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIESYv031124; Fri, 25 Aug 2023 19:05:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=zb6NKCA64ESqc3mqR+LOpdZU6XXh60EAfRdU796i4Ro=; b=2t+1O4bLN+k80Qt9bl8aoXFhuuMp9ZXlhGq2afXlv1gd9xeoJmb4MlG+dFYXOYomgqlh pd7rKQDWu4w9dOxZ/FZ7q27tc8uN3+WFx6gQTKk+WZYIJqtXEZQvQceCjOSwhTJqBSdR vEthCHAMekuHt6C65L48M7Qx/vFdBltTJdnkAHgwmoVYlviSM6H5EJXYJZrEMO93F6V/ MwTvuSuIFOL67XvKcpUheYgqurW3AjJ29fH8INRv0KYVxyQudGJVbvVLKLESEagaLAm2 aKkT/Atw1827JzDLQOl9FYVJ5Iriq2LnRGjLvLrzNUa6Axw6ZHPfVdvF0BsWVmRerCQG 2w== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0d3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:01 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIjPs7035753; Fri, 25 Aug 2023 19:04:56 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2041.outbound.protection.outlook.com [104.47.66.41]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yr77jj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:04:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BC1icQetrRL8CdwcQPbVd2yUqY7c8UF/mUdQ25eApLIfuiZ+U7QPW7uEUy+EPBxnVLIlZ67q0xlezRpjAPGNWS1+yuZ1eHgfiXBQGYlijdWbSc0HemBIVBGXEplqwq3fc6qR6brsdSeRiagUEU0yLXSf+nwOPyKPSfZ66p5DiY0LnW62pE+MQw7HHP3m4AIlNJubxR3PxqHnSVnX/2VfnrspbWpM2Krtw1oeCiU9J3wQA6Dd66n6BRnXdwWaQyeWic/RZ2lLSaEWp4cCBZOpbHFP/cP2IXbCcL8+P1qzAcmRf4OT2La+l4YSQSSv+BdIzh6xZKM63vZohtGKMCjBFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zb6NKCA64ESqc3mqR+LOpdZU6XXh60EAfRdU796i4Ro=; b=Dmb9c1nDCzFNX99cinMhXC1hhMxmciTIFfMr1U5AWErAdsbLIcY7C/GWMhMFHgAPrf+CtSyaxATo52ZP81fz6iYrp3TQCj4aTV/KEnSrUI3Y0uJ+N5UWlWt+SjkrO+TlFFgP2D5/1RMRsWviPPPhOkcly69E9Gt8hs4ki0LRuR99O+ywjqubgZy6CzYs75Mc0hZvUVph6FlemZD9xwGxxEwXUZJyPy/8LC6LuokKADNevspdtGXo/kZVR7i9xGIOpD+eiI65UIVW9uXwL2f5iM01fRQv1KIVO5gUymHEbhSiNxiq4rasXTJUtZhWXlmE2ZWeazYQ4QKDqepx706E2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zb6NKCA64ESqc3mqR+LOpdZU6XXh60EAfRdU796i4Ro=; b=wV+6nZbBfTvov7/U74Whvg6bWvinIk/Wi8XNXGZUKPk1zZGKnM9pgoV7x7cVIXQnmkXByBllfoQO2zxH5RtOSPyrgsfphF7+97ARnRLmkZhzLlek1me1Scoe/cMP4uxyn+AUrGqEMNeJTRgvwY2wipLqgI7LH8mxSStdWrfz9Iw= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:54 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:54 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH 03/12] hugetlb: Remove a few calls to page_folio() Date: Fri, 25 Aug 2023 12:04:23 -0700 Message-ID: <20230825190436.55045-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0087.namprd04.prod.outlook.com (2603:10b6:303:6b::32) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: f1c6fe78-d656-440d-cf30-08dba59e2a80 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 74PHp3nRoAJ6M9Fo0Pd0MN24HTz16HRlKL8Fq19+S5T8wbfui7T/x8rlt4ltVkoXMm0R5qBM9OXkIKLhHL4kRms+ltL0yg1Ds1Jm+IYFaXFBroO7dZ+AgRcaCjFWuc7va7kmcrUKi48KS6KlyrB7p5plz64QMsfdN++mIFjq7dP40Eb6Y1y4Ymx7WCJ4vNv+mYAcC3GPiAWV67la97jBaTeAreUIag6rM4hhYaqK4stz+L4SQLH+wW8TxjBY3hAIsBDYpJRjXOJjPww3U5JNcYpSBPF4eDqehcJh83FNvnb5wU7KseB9IcZWsaftf0eSlg0SPpa2TNGo6XgB5J/0bUeCHVWGrdollihAumO2iOe4IWTx+KxoddFxnpQWtJFpgRRlL2vupcDjsjyvU5A+eev6imKkwb0JVdun6yAvW7mt1O/YwKAG5iMXy8Vf4BOiKAAhS0eBZksfOlg7mwX5NU9yhNGpHly8gC/aDNHqENcJrgJuu/RkCmAvb/mSiAJIJOgTKpzanvmhNMubZbivC5KtEe8eaTw5njjGn/brCJPQTkBnP77HVoXH4bfv0yii X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?B5TgNuPOV2zkV4qXRtkdJvrtWhUYCJxqgmbDZDF15l0LJeOkCUU84H7uWfSU?= =?us-ascii?Q?iEtzqAxh+6PN1tFXeDjMeGDRJXH1yQmn61dErAzmTOgfD5PgAF+90thw4/7D?= =?us-ascii?Q?B39q0EUvu+nquUgMlV0E5YhFmNQ3vrnBibq0RNHnKQZfzcRn5Y3XKRtz+tZp?= =?us-ascii?Q?5QBqAT/vWPdWobDzVpRauIcJ5jok157r9lkxrrCNe+Ww5aoBgtSFfPsR+Z2e?= =?us-ascii?Q?ay80Wm81YXO2tNa7e0JFfmDI1wewaugPAYdV3Yn3uPnwwAKtUa52IxZoEJze?= =?us-ascii?Q?QdKzvgAMlI7uxmvdhAfwrtR4k9mTNwNx2XmzsTxSXaFOmBswLW6ePgKOq4c3?= =?us-ascii?Q?zZ5Mmm+ls2KU+ooPqB3GgzVGwizOHVuH7WupObXX/g1Cy8btpjph25PJm+PW?= =?us-ascii?Q?nSBA72rV7fMczza7VT3UCjcoJGc+3cKB/H74SW3Aa9kyma9Pmy9PyUI1Pcax?= =?us-ascii?Q?i4h+wviTLc8GSqCMSYVcIUxaNGaZAwc8YfZhEFoinMFZNP87x1v2y97k8TPg?= =?us-ascii?Q?aDoXo4NGUtbACzlfQxes93pHmnJnbLcgjOXeXMLZyhUoN3D7cyjEXbVLVUNQ?= =?us-ascii?Q?7HAEBE+yAaCDLk95zucl0/gzsOugqyD0jBp7VysehePNxYY50qAbnDzn/6k9?= =?us-ascii?Q?AVoS101gFNC+6WsO7pE/7ooGqI4FHewPyFsbyns70U9ezzPysrLfa+UkOslb?= =?us-ascii?Q?tvudgDzioTAh4XZUQuySu8Gt4hmFvo9BmUoReVn8FvU3c9d3hnFzaJWPhYYR?= =?us-ascii?Q?F4A14mqRuMhq5P2EM/MO11yRFeBuxnetZqsUQWPVtD9z/3nR+3IiHvJVBg01?= =?us-ascii?Q?ux1C97xmuDgggksJ9nfjjI+mFZr3e7KaJiKplZUN3j4+2CyOhsnuraoGiA0D?= =?us-ascii?Q?9lUWxc29oOVcevIAYH9XcLnHjkuYAJE+cCnkkEl9aB9xnJE5hIo6e8IXmmQv?= =?us-ascii?Q?VjU5z/xncWLeeXULz5hOdqOI9n96P3qXxEernNkbi8Pajes9bUFn57c38t5+?= =?us-ascii?Q?b7fdSLg9ILh31yjMvaglfNh7vjC+g6bftvivu2ISXBfvHgZxt45tLd9sng22?= =?us-ascii?Q?BAvCI7m5ThlR24KaZZKKJSw1/FDpPega3c38kmUgiVz9j+VccefXV/n4vp+B?= =?us-ascii?Q?kKS8NUoytO5ytx0HNu9df+frR9SDkQ6VFufL0Hi/Y8iQV7ZdhmlU6Qjp8i/n?= =?us-ascii?Q?GPKm9PGRgDbS9xvCJN7lR3uFHef0xhj+grXoBo4oFoS3trr5PY+VhiZSovGo?= =?us-ascii?Q?g7gVr4dFa62f/1sq/i9K7SSGQvFauUav/NLHqG8kjqW1+GyCVYHqzBF/w2l2?= =?us-ascii?Q?iVqK4KN83jHHkegM/MTgc3EPbK9sPtTjXzzeFRR2qhp2fEux5nQAn5U2JH5G?= =?us-ascii?Q?YjYQyw3fcdWj51/m4MPj/zqCTUSdrm7Yo0UCvqWNW9ZTY3rMa8RxckMuvQZs?= =?us-ascii?Q?8RtWMA2eHNTWq7Bq5f3gKBEnq5bMmxnBdP7IeANeGwDPaiK0uhKIZGLV9blw?= =?us-ascii?Q?SZoQwgx9hUqAauP8axFIgIbtWyosEc5J9Yv5v8/eiQ+E5ui5S0VFUA7xXoKA?= =?us-ascii?Q?mQ1Fd0HhocwC/m79rSq/lXaqOdNAU9H96mPffK9L?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?XYNz8A4oew/fcpsNHxzr8c46jc11uMYyjGx9dpM86agOsu5ASLmyDCTYcB/p?= =?us-ascii?Q?i9l+g/0XB9sXL4gTW8h/ee6P8zcPBUbMQgBK2S3olXI/s9QxKNoDUknXPSTn?= =?us-ascii?Q?GvuATwrGUGW9LlPtFfnPd6tpGN20GGNEaDXbBVYr+4jB9bLamqfrFUvQm/5z?= =?us-ascii?Q?fNqYiWxvzSbySjoR2GwtJBJoEg8Nx3H2+ezwYQMnE7Q2/fVfG/WoQ+PXcarB?= =?us-ascii?Q?iefDG72ieXX38PW9aVu3K9tO5Hkioz9bKQbDmYaIE3839TxVxHlrkFDPnQeR?= =?us-ascii?Q?Je0ZoXE13aZI0+W4Ur4VnfcAudNmBKRck2CyA3QpjugF9b1J1onGzWeM1W8t?= =?us-ascii?Q?SK0Td8uko7d4P1n2W8dFXpqGpqOF96X7nzJj35oRjkyS7eMkIeJpd7IPQ5Sb?= =?us-ascii?Q?2GBNiv27srNUZ9Y1nmet1VqadTT0Jo7APg3FNaqAATI32BQ1yg1jWEaoaB4A?= =?us-ascii?Q?9ouJCgVeJSLJ91z98le8SzhObtlWaW/NAQAh34WSoFMt6uY89USofQ8i7Ave?= =?us-ascii?Q?mKKVTh5AaXkL70/DOkHaTkjnoggmiTo0z1aQlqF7EwWpy+ENOEYOgYplNIFv?= =?us-ascii?Q?jomp/cG+67t4YKT1dQfRVNZ1vDbah6ufhbog79AaXnLtk72bzeQCaG1sDUa/?= =?us-ascii?Q?741z0VkjCQE/NP1AvGoKUhrBKPZ6vrjmJNPDoj72GhkVLCsNHF+aN4i4nwo+?= =?us-ascii?Q?KJSp2kS3/Mtdrsj3gcGJALuOc8VcrL/rEBuoAmX+dR7X4qD2HXWeUqAY4IyQ?= =?us-ascii?Q?dWWrYw5GeYRD78g+zn7T2JUZstBUgutEnVFGlx5ePEEPnpMKJaOAXW1WJsmM?= =?us-ascii?Q?rfm7+iCGAYbatkHfKdzor/FUZBSIiT3CuUt6KLJKzeju/wPnEnjf7JbX9eoG?= =?us-ascii?Q?QX60IFrBIc4JYfrf5tA0Jr7lyzXIS8jtLWCEwxi5HAm6i9nqzk7C6C85F6U9?= =?us-ascii?Q?yQr4JIOG/gpJKbJp7fdrfTI9TBfq5uoABCOQYFxe2jy1rv/ZqmUQlAdMin4Y?= =?us-ascii?Q?nAOptoi+hmnwz2wwoOfFve3l9d64QoVDFqpyiDrFUAFBLkXfgWMt4vFe56Pm?= =?us-ascii?Q?bvW3uVYzeGW0LM04J8AFBnRKpLxh9LULIX/oAKCcH7mzUaC8kEO7jeexHsOH?= =?us-ascii?Q?N1U7+nD+fVRB?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f1c6fe78-d656-440d-cf30-08dba59e2a80 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:54.1109 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4viqr2pRx2yxE1H0+MrvVG6QMsocN7EvySr8SP3OO1Lme1rMR87KC5iU+uskIteL3QVmMBimJyoWeqYvQuO8rg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0 adultscore=0 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: 2OKrJiPSiCvmOy9oRGA8-kJw9qhUO_Pt X-Proofpoint-GUID: 2OKrJiPSiCvmOy9oRGA8-kJw9qhUO_Pt Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: "Matthew Wilcox (Oracle)" Anything found on a linked list threaded through ->lru is guaranteed to be a folio as the compound_head found in a tail page overlaps the ->lru member of struct page. So we can pull folios directly off these lists no matter whether pages or folios were added to the list. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1a48a83846cb..a5348dfada89 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1831,11 +1831,9 @@ static void update_and_free_hugetlb_folio(struct hst= ate *h, struct folio *folio, =20 static void update_and_free_pages_bulk(struct hstate *h, struct list_head = *list) { - struct page *page, *t_page; - struct folio *folio; + struct folio *folio, *t_folio; =20 - list_for_each_entry_safe(page, t_page, list, lru) { - folio =3D page_folio(page); + list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); } @@ -2224,8 +2222,7 @@ static struct page *remove_pool_huge_page(struct hsta= te *h, bool acct_surplus) { int nr_nodes, node; - struct page *page =3D NULL; - struct folio *folio; + struct folio *folio =3D NULL; =20 lockdep_assert_held(&hugetlb_lock); for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { @@ -2235,15 +2232,14 @@ static struct page *remove_pool_huge_page(struct hs= tate *h, */ if ((!acct_surplus || h->surplus_huge_pages_node[node]) && !list_empty(&h->hugepage_freelists[node])) { - page =3D list_entry(h->hugepage_freelists[node].next, - struct page, lru); - folio =3D page_folio(page); + folio =3D list_entry(h->hugepage_freelists[node].next, + struct folio, lru); remove_hugetlb_folio(h, folio, acct_surplus); break; } } =20 - return page; + return &folio->page; } =20 /* @@ -3359,15 +3355,15 @@ static void try_to_free_low(struct hstate *h, unsig= ned long count, * Collect pages to be freed on a list, and free after dropping lock */ for_each_node_mask(i, *nodes_allowed) { - struct page *page, *next; + struct folio *folio, *next; struct list_head *freel =3D &h->hugepage_freelists[i]; - list_for_each_entry_safe(page, next, freel, lru) { + list_for_each_entry_safe(folio, next, freel, lru) { if (count >=3D h->nr_huge_pages) goto out; - if (PageHighMem(page)) + if (folio_test_highmem(folio)) continue; - remove_hugetlb_folio(h, page_folio(page), false); - list_add(&page->lru, &page_list); + remove_hugetlb_folio(h, folio, false); + list_add(&folio->lru, &page_list); } } =20 --=20 2.41.0 From nobody Thu Dec 18 01:07:41 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 878ABEE49B5 for ; Fri, 25 Aug 2023 19:06:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229484AbjHYTG3 (ORCPT ); Fri, 25 Aug 2023 15:06:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231148AbjHYTGM (ORCPT ); Fri, 25 Aug 2023 15:06:12 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D63862132 for ; Fri, 25 Aug 2023 12:06:07 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEepJ000972; Fri, 25 Aug 2023 19:05:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=9P3Cyny+mcYwz1jwFSE8K4E3sWShTXUh71C/hj9IWl4=; b=pACqANPjZGRlFsTq2VJ0FL80q/JHBrsJ+jUVMbK0ACpMXN/uDgnZeJhiGSHYWxUrJkDR ECeRIS9hkXx0lKBsRfSV5/JyzOYP1A8qeJoGwukYEewrnhQXpY81WLvebFEXjHmYjFp8 0plKVPUcYg9AWD07S+KRtz5AiKG5QBZ8iH2/esS36kT5bwkpLs4+t7pFk1I1dKViWSW1 G4ZIaE9f0LchEs7ztK5ZDEVsWutGn8/4y+o8yyjVMq97sbvjvLihwAsKH5PFM69551uL OHRYZXcD1YR8skBEqIk7SKOyjkr+QoUqPWNg5Olt1FOVYCRJdacxYMCuw0EjgapHEMTU xg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yxq01b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:05 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PI417C005759; Fri, 25 Aug 2023 19:05:00 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yuya0v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OVcPmGBj19r133Gy/VFs+hUe4ko9UR1s4FN2yjogf0RiIx8OMZDaSqBSvRrJ7ryKM5EbuLP7arZMMd7JC5RuRMkIjH0Pt44DiK4Y93qC3PA661SykBQ9yAHUyfD7iKAZ+jYvooN51mocNjeY29WPfqTYyBVLmSxWzDmQDc6hGgMpIFUnE2AugTxqwfXU03O+HCap/GXEjOsn0uxn97okBjynsqDNEzotA6VFJKVmFOA6IKNWDHRFyuc3ZpaMRbqnyMdEMFQoGLvxls6iBmDn5w1SDzCaco9V+yha3YyW413tScnN3tb9MhNILrQeIlJcgvFqDEtvcPwb8wDkpxBjxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9P3Cyny+mcYwz1jwFSE8K4E3sWShTXUh71C/hj9IWl4=; b=LVZZxllwW5bofTD1Zoh4lB9vxurlz+v0CVMuO6vlI4DP/lcWcgP01NDSdzBqwUIBLIKDJpYn6DdgAUSyU/T59xybNDwd+DPFJ36piPTdL7nL63kjv6fOUdB4D04eM2VtP+5TIYOcxGtwNcgD81Ua2eDzEVuHWbV0vG2msUnsU3c249wHKwELCt++6ToiMMm0Joh5zU56/Fh8xZa88XA/BR0qCrb31c8LhLosvi3SwMpAD2QkNifeuPq0a4mfV5kNaB8s+7zyQn45F1KMMNgt9mA843C7GtVOmnq6mlgxFJ2tQZ7HutdPv66NEF4RjuPu9ed5waT2HCS6KSdSGfBB2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9P3Cyny+mcYwz1jwFSE8K4E3sWShTXUh71C/hj9IWl4=; b=gTWTMZna3DkMP2t2BbLfW+hTipfF0H8hIZuwzCpSEIMB4JGQxdGK+fZ9cVGrrbnbP0qFfzoWFCU90ONN/7BiMzuPCmtFYjsC/NgDVtARMq2zwH1HHJCHvxrayCCoQ9hYZpuNYch5Y0aqUi0sJPgKn9W5xg2vGOUgpVDFYL4/djU= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:58 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:58 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH 04/12] hugetlb: Convert remove_pool_huge_page() to remove_pool_hugetlb_folio() Date: Fri, 25 Aug 2023 12:04:24 -0700 Message-ID: <20230825190436.55045-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4P223CA0006.NAMP223.PROD.OUTLOOK.COM (2603:10b6:303:80::11) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 0b3179b0-4cfb-44ea-5855-08dba59e2d10 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: arA0fqYiuagTDTT5usXxz3l50Irt/cQTgoVY4737hdO3oI7Txl5cHL1H2J8PEPse8WtibGhylArf+DwIibQnVqPVbXra5zFm1mCdIQ+KH/u/ZMB55XE+VH5xA9xUj/G5BLU0xx4yrMwn1tsQ5sfL3YaaRcwZPO4Giwy5H2waqlwWT5jAK1Asm6NRIE76VBSmo+j9YKR3RsSvmPNc3dbyb6Aq4w01U1uyIPs9ZtC/WJacn0wpQ1tgrbTFsZoTtIQnT8H5WobGoothiN2yWGZEEsDh0gCuL0bcpgJQEZyF8ysWZMRm+8elP1BXH4sm1Wcgx6oCqkiEtCPj3sTqPQb+tam7eQLGZxa8/yku+A8QHi/dZr1Wbafu1rBvCG54lzBoIJJ/HHKMoUQ6PcTCQzwdN4MYLJw9k/e4z71stvZ8RXXbubMl9StFt/I9TB379iwl8JzuWWJfVsRGD317IYCMmkk5+JWvXZwz/6mlw12+bNHz1UL0MIOnaQ3xndfY+88DlsGdjTlp9oQuIPAAlLZQ6pyHJlzbsONjoo5ca3I4ze2Aua20CJpQXyujSf0fXVGYgo9tylck631RNkbMQz2swK+gXY7gU82S0BUjHXxTK+k= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?sDyjABoO2OnZ1+8Bc1YJhA4ZcA6z6Vq4x9xG00tC+QGE3DtXcS6VJgStCzzB?= =?us-ascii?Q?E+2mvU8cNggcivRi00WDaQniUtM9njpPIi9LamKsoMHxZZhB1I1AntTDlHI5?= =?us-ascii?Q?T0idpdXrhicXJZTNT3GIgHHRfTlxuCy9zbYEn15UnQ0WGp8HXgKL7cuotHJc?= =?us-ascii?Q?D/QF6lIeyGAo/KE+33zLOJgV42rAgLr7XWZQDHLrhuBDyzMwF3YBUGP8G7BS?= =?us-ascii?Q?JJ6rtFT+FkPKhKGsDLfhxAlbsyxsBkuIWvC2JiYZKZQLhGjJ5c7Wgpz9/OXw?= =?us-ascii?Q?7c7MzeTpo1F5/lXbSalnrF98B1SW/+J8uVm0sklaJib37P1NJaAM4D/zaOEh?= =?us-ascii?Q?r4dt/M7B/SQ9DqUHA+HYyLf1B9QwM1rEMJImy5nEmkWJZIpZrekX4KbyNRZ6?= =?us-ascii?Q?n00hnEh+23l3KyWiWidlZx+Xr/P5zlIB007CSTpgfSlW+rHQMESlBdnEvEAA?= =?us-ascii?Q?txo3SDwqojCYHYac1pN28uVivurlp4aZRgkZgNPRkw/+/nTJswNQIsymqyqQ?= =?us-ascii?Q?FDlZsNmSiA3AqvPiGHMRUGbPAnDFORXvRSGMdmFKmQsUw06w77CM6cj3pD1U?= =?us-ascii?Q?5hRhQpDYAfwkbUBzCR9qVe/xbfJigN+10pPmK3LzxgwqjVsBdnb698tlOvxs?= =?us-ascii?Q?lm9boiiwejxfjR0aXdgJyPmnvvDswvcIzqstFlGKBDnFH612EARUnfbKM1Ym?= =?us-ascii?Q?blRGPYzr6vKIl1f3b/BsotP4QNJaOfaQPjWaLa7zBq2Tr4QA3sEdgHlA7aLP?= =?us-ascii?Q?LQZds4lm5+cjQR/joNvKgfMPL8nfVlTBkwORL4MbWzSmu+xPm6onO/YRDooS?= =?us-ascii?Q?vO2PDDAs3E8h1WfUFsnzehQ3jg33Yzm24xgvdI3u+pO9asvcZ0vIDmOvPaLn?= =?us-ascii?Q?yPDroe+dfm4DGWzEhanUwPCuysms7Gh/r2kpWm5gy4Sy8f+D7FQvVYysTqI4?= =?us-ascii?Q?W4WTMNJZYOQXfBcyhdk9SM6jS9TTB5sLJ2tOyeDgE6NWZzoiaHXX6GhHw+EV?= =?us-ascii?Q?WaQpQYrip8RI6RKE/mD2VsPkFf8DRrsE8uoo/Gj/fnwbvy9XRdb8ELgnRucp?= =?us-ascii?Q?d1sQoif3AwmzhgACQWp8s+TmTH6e8+ceMtwqv+dt9PuSULHhYe0lC+hVTNDp?= =?us-ascii?Q?ovuz3Ay9zWY+21YGFbiE/ZwUG2NlHp9kQvKl8bdOyfQGgJuus/EPEK6N166k?= =?us-ascii?Q?/nYKkXq+gMajsIXhjHs2Y9GMm/Os4y/OalltRKskNNOWr+FKHvw+Zyyjr5dV?= =?us-ascii?Q?RK8iTSPnabTHpeCO0yHhrRqzBYAXHTEESzyu3P2MrJKsmFW3kf33UcWDdaFQ?= =?us-ascii?Q?FDRk7wrt9Il4mt8fS1T/l22T09CAQZW5fcdDKhDbFVFExZx+Xdv85cNkCdtb?= =?us-ascii?Q?KOQud4PdmVJbcWPCiqoHDJjQ0x2lfP8HtdDbaFRVIziCWitvKGdH38gPpmks?= =?us-ascii?Q?1v3V9RVEhtyQEijKnuxl8SW9khBCfvONj6kDBnfvqdtAIkQpsTTTDm/kkluN?= =?us-ascii?Q?c3Vuq23WQvE0AX3D5YLTTTjsuPQTPD45S+J2lUdkZIWjycm396ueQkf7BJaU?= =?us-ascii?Q?+diTfKCi5WnlzNRHtNGFT2eo8w7thasqW3QgNUapDWqCOG4zlXmY9N4ahuxB?= =?us-ascii?Q?qQ=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?HoY4neH3eiGkzzi7sZjMmEKwYVrOQY2+tYoJdTx8GBqT3RwLeU0tZ01mbR88?= =?us-ascii?Q?VxCHxATCy68hTAe67VrlfTQDN465oezv+ujqu/EJXRhi1AjUInRUkj0iJuZb?= =?us-ascii?Q?TADUNvunp4E6cudQsuOLw/tXNUoTQCYrWSUZy58xV/KGbG5O0lNYfb67ruVp?= =?us-ascii?Q?wioa8QIaLT9PVl0ZmMI7dXPFpWRq1IXeg7opRpwmbmTlUo3OdPdNW6f1Qz0t?= =?us-ascii?Q?NUtIeV/BG7bEm3onRG4yeU9J+AWwP84wkYADdpMkIwftnpwScsNi9CHWmZUR?= =?us-ascii?Q?oydrd8WKBU39HhJ1an9S/efMsZ/I5qVMdzz/BOJNePUEwA7wXxjk5YFDBaKQ?= =?us-ascii?Q?wrgt7IBE7pWicuz3ETJVg4v5ZQGVavkA4wYKPNnuwPw0q/bTlJtoG9RYiadp?= =?us-ascii?Q?FjjcwwWHQGNOO5YnJ2EVXM9stouhDQayiIs6DbTXmCDxrva15cNWuR7cpiyr?= =?us-ascii?Q?irD4mWhyyCO3b+v8SOsHZdeL6RlJPVtogT4hVOe6+eWIKlWe8greUhFIzJul?= =?us-ascii?Q?mpazst+tm3oH63JR0igQb6z/AvhDm85bj27vfJHtPTosZ0B9gTvzWSoh4GGe?= =?us-ascii?Q?Ja8XZlf1KPp7cVtz3IEtZEttL+RJMULp2lKp+/3SimetFcqtmg+58e5pktNy?= =?us-ascii?Q?gvG0mYBcBvO/mxjIlkEO/IE//dVc9HaxTHG+TeobGpUtVMjvMUrhKgh9iTsX?= =?us-ascii?Q?nz+GSTmpZTRjCj9xL4qNUe+gOHs1sX+21mpNaH7aBwBsQ+0oeMoCGfR0V0Oe?= =?us-ascii?Q?K+Ko2qa7w94D09cm0PMV7tB1VFnfwCpyYfTUJD6Mfa7g3K7w7eKtXDFm/UgW?= =?us-ascii?Q?CcR/W3c9s/G8TlctnY4SiIye9z1KY7gtde0ALU0Zi2H4paRK5quFl9nON7Pc?= =?us-ascii?Q?O/xCnQVNfRoU/QhixG1HpNYXV5SSaQ8J1416WgY3vRblVMGQTK7gTidcrtr2?= =?us-ascii?Q?NMlmG0EkgB/4wXTYq9hEprojpE22BJ08zEMotSaaqlHgn+FHyjmp2KfTJdfU?= =?us-ascii?Q?zvxM1gvJxVkQjIp4sOfraNBnEAXBtkra9lwsDOx8Vw296wK3Mo8Oc5QaKROf?= =?us-ascii?Q?uLI1OqDWySmL629VUQk+bHuKIHMTHdHrGIQyYgnj0gHMwLnCz8a9r+FKJBCI?= =?us-ascii?Q?vKyIYl7vVIFx?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0b3179b0-4cfb-44ea-5855-08dba59e2d10 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:58.4911 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eWYBOH6ca0+EYdEoSF9dsZIJjYIR1CZqRiUJepR7Tb6DK8aaYFM/c5XwO4XAZPvTIUZ3vz9tEMi0We2WGhUXbA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxscore=0 adultscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: b0m3FrRmbHGtG0JsJ0bwj8eE5EcS9YFF X-Proofpoint-ORIG-GUID: b0m3FrRmbHGtG0JsJ0bwj8eE5EcS9YFF Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: "Matthew Wilcox (Oracle)" Convert the callers to expect a folio and remove the unnecesary conversion back to a struct page. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Cc: Sidhartha Kumar --- mm/hugetlb.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a5348dfada89..ec10e3804060 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1446,7 +1446,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, } =20 /* - * helper for remove_pool_huge_page() - return the previously saved + * helper for remove_pool_hugetlb_folio() - return the previously saved * node ["this node"] from which to free a huge page. Advance the * next node id whether or not we find a free huge page to free so * that the next attempt to free addresses the next node. @@ -2217,9 +2217,8 @@ static int alloc_pool_huge_page(struct hstate *h, nod= emask_t *nodes_allowed, * an additional call to free the page to low level allocators. * Called with hugetlb_lock locked. */ -static struct page *remove_pool_huge_page(struct hstate *h, - nodemask_t *nodes_allowed, - bool acct_surplus) +static struct folio *remove_pool_hugetlb_folio(struct hstate *h, + nodemask_t *nodes_allowed, bool acct_surplus) { int nr_nodes, node; struct folio *folio =3D NULL; @@ -2239,7 +2238,7 @@ static struct page *remove_pool_huge_page(struct hsta= te *h, } } =20 - return &folio->page; + return folio; } =20 /* @@ -2593,7 +2592,6 @@ static void return_unused_surplus_pages(struct hstate= *h, unsigned long unused_resv_pages) { unsigned long nr_pages; - struct page *page; LIST_HEAD(page_list); =20 lockdep_assert_held(&hugetlb_lock); @@ -2614,15 +2612,17 @@ static void return_unused_surplus_pages(struct hsta= te *h, * evenly across all nodes with memory. Iterate across these nodes * until we can no longer free unreserved surplus pages. This occurs * when the nodes with surplus pages have no free pages. - * remove_pool_huge_page() will balance the freed pages across the + * remove_pool_hugetlb_folio() will balance the freed pages across the * on-line nodes with memory and will handle the hstate accounting. */ while (nr_pages--) { - page =3D remove_pool_huge_page(h, &node_states[N_MEMORY], 1); - if (!page) + struct folio *folio; + + folio =3D remove_pool_hugetlb_folio(h, &node_states[N_MEMORY], 1); + if (!folio) goto out; =20 - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } =20 out: @@ -3417,7 +3417,6 @@ static int set_max_huge_pages(struct hstate *h, unsig= ned long count, int nid, nodemask_t *nodes_allowed) { unsigned long min_count, ret; - struct page *page; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); =20 @@ -3537,11 +3536,13 @@ static int set_max_huge_pages(struct hstate *h, uns= igned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - page =3D remove_pool_huge_page(h, nodes_allowed, 0); - if (!page) + struct folio *folio; + + folio =3D remove_pool_hugetlb_folio(h, nodes_allowed, 0); + if (!folio) break; =20 - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } /* free the pages after dropping lock */ spin_unlock_irq(&hugetlb_lock); --=20 2.41.0 From nobody Thu Dec 18 01:07:41 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24378C71133 for ; Fri, 25 Aug 2023 19:06:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230080AbjHYTGV (ORCPT ); Fri, 25 Aug 2023 15:06:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230318AbjHYTF5 (ORCPT ); Fri, 25 Aug 2023 15:05:57 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C067212A for ; Fri, 25 Aug 2023 12:05:53 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEOfe031083; Fri, 25 Aug 2023 19:05:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=PdmwOcXhjYj18bUXSFQSbkfcUgdNSVEPyPGP3ecmcJc=; b=JcDyRPVGUZFskL/nrSsBSF+TKgziEeoTD9ExBT20DkdP15qNym8OEoUB5dz+5tdMseIO 4eYLldh7godXnAuH/0YvNhGxX8sdWCZZtXVVOWe+sSgXrgMeGsFb58qtJOLm0aMMkf3M lL7mUtwzgLNYRr39om71T7zSbHrYHNmLe/OyPvPHEMHQee24pN0s6ZZEfP+C9Qd81pX5 ClkASfXYkg+5R6rC/n2HgZlYhqgzF2tNMCOUNEwoSR0XZjUpJAhPruZ7ev3LqHqL7bCm OxocR328SntIBj9TSHA5QXyt2A6MV3Z/54K/6Ak/AhB13jxCMBiFW+WHzPDRB9zBcHWv ag== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0dg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:07 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIUF4w002259; Fri, 25 Aug 2023 19:05:05 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2041.outbound.protection.outlook.com [104.47.66.41]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yyg8ht-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FweYzoQTcHTB8cM+kd0fH90FC8SUt5AY/KIlWnYBAQU+BreiEo+6iq+G9BTpgFBXg1MDPrz3ZnBs+gFdmvz0NGQoO/uqxg+i3nND7I8oLLxe8wJa073DpgUyyxD36cLWuHpcT16PGQ1+XN+FKftADhNcLhSkUFhGXuvSsL/6Zv8QEKcSMbMz8NiwZIZ364/bXDRrgNt/V05epfCCvukq51zS35rn5ULCl7I278cIqT/i9DCjK3Wh2ZM10EjHeLpmgJpAYIVQzWomiNvgiK9zvN15DBbVkNuzKQ2tHzDVPBCImhbb4vebFwTyqyxoXVYH+C1rAipInTjYXYtaDao1Xg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PdmwOcXhjYj18bUXSFQSbkfcUgdNSVEPyPGP3ecmcJc=; b=i52lHRk4dSBj2osfkp7MilIg2/ArAVRDXpbJGa/KttEnVaYnKzpPBhdM3lXlMNL5XU9HrvM/lBQ7+cNB+kz9QiLs97ZGUHBL8b3n5std3V2rjMEzwPoePjLBniePI1FR+TN02AFhCo02LvPL31u9SN+6l7kmdyJYNI4JI+dfP7bIGgSV/TN5lfPMfImvUXqwAz8lX+PK6/HyQaVRG27/Jctp88nqNFCjuSyUl/08DZE0UhcApTIP6mdpTfLQhsyZ3RBzu+Cj/JlRayOF/ee8kIWawtG9/S/hp6g4xx8LON6rZVnskuH5X2tP9Wy7yJ6x3rjZUtIZ9dkXOWwMBMd76w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PdmwOcXhjYj18bUXSFQSbkfcUgdNSVEPyPGP3ecmcJc=; b=KxVyXmUubOsX8Z/6xr8zo00aKswdVs/6KuXmnTK4+Nr92LZwCtjE8DIMg5yO+4ne14G9uTmVMyToCPm4JtfcY8jfUAR0k2uAxTMJ3+ude8WcOAJzgc5FJgOTK8pIfhMPVchw448vc+raFlnQKAi7JyuqllXQobICabkokb3OOa8= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:03 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:03 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 05/12] hugetlb: restructure pool allocations Date: Fri, 25 Aug 2023 12:04:25 -0700 Message-ID: <20230825190436.55045-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW3PR06CA0029.namprd06.prod.outlook.com (2603:10b6:303:2a::34) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 08201307-93da-437a-b798-08dba59e2fb7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qlXVlevVXQK8HWBR+q/VThjL4jkvUXuSA+Qz49eVpz3fllsAJY3RHgrJioA1lp7BhpTBZVQYyZd1fX1g4iHgW6xhm68Qbn4eVEMZnNkcI+4eg7Boh3nowDqtp0iS/DQB7vMydTaus81UJBMXbfWlefiF9gZXRHx/g84FAE6airfP4MTH5vj/Oey0UaQ0f/lyLdefiFhWJu7ZHbpXc08eic+BkLozjbVXgIwaV33XzQRKFrrLb5Z+6evI+pdmp2RNj5gwHibbXZWnvimlZR91mZwfPUDePCyJliEKGcGdOl4kw17Lpy8v57fHdYyo7FhXQKwXbERE6by2YdgLh3qHYiIVIPgX28KwSYVsZqIr1S7ceXhoOnApP3MlSUvNTvIjCyUf02J6ZirWlTnTzH2HOTdOxTnSzDwAPM24/Nd2ZnCBAu3EK50Cc19NmrDybinZ2EBArUefRytJHx7pBWuaWcvQJNURyeZPvGWw/MHKxwsSUGtQHRsPsQTp6/KS/8iloBVD0tbik7twRCGjd7XVNA3E9V6fwmhzbDxMFt/wSofeUwR7Co4ueyOoQKQj7aAD X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FjD5kVpzf1H3W00Xod8M32RXSH7WXsHRVzv2CcbU0z27xWCxX8hXRswQQHUG?= =?us-ascii?Q?oBluK5eGWR4lSwa09HgaH2H0V8FNMwHTVUX9tNyjgV9dRDrL97kDPQjtWwvx?= =?us-ascii?Q?CNaD4YOxYsZQMMzBFrSgxotjBvanySTwx8C2rjpg9eXmNrEmkitoEFzw882c?= =?us-ascii?Q?hlAzpBe5L1mgWDd9x9nOV4/t/1v1y9p/3EUytPWh7vVgI/WlXUT1B8iV/LxX?= =?us-ascii?Q?IjcKJN6IqNJUtEBALEhAr1Dh2rNFG29dxUzGWHuqzics8enP5J6NwursverX?= =?us-ascii?Q?35EsxVtoMXPkkkHn196vkqjdnB+fYU0CkEh+3xcRk8PqzX9whOZWcaHm2A3F?= =?us-ascii?Q?Tsze4loj43x20uTH8D/Sk+aU8aOebepN63ZZNjtvuwRrtmE2i/p8Dj1n+AO2?= =?us-ascii?Q?KUYRoZP13OYvFw7Bnm9sG/XT7ZqRxnoHIs2DYaJmD9AETuffupYrT7L08Due?= =?us-ascii?Q?+QUNsFeb1PBWmUskhmqngAEju4acTUeEY1VijVNPZmHyFY+7lCyMZCVTsvqM?= =?us-ascii?Q?1ZUPvuK2SuveIBdCzG5B5YgJ1/FOyZVXdGSCDb37RxpORfbQtBuYznnc2dCf?= =?us-ascii?Q?Yp8XVUcXWpUhnOSzbDkQ0EDqtvEl42P59BzaRkkkUnq1J9Zq/5gTcPR235t9?= =?us-ascii?Q?vMxBC/ZsRc+BvsUwVxeSAYRpTxW5ppVTDiEV7+cGijCctgplWD5vhqX+cnqs?= =?us-ascii?Q?xd/6UvvO+0A/0dFjZkhPz4nEaWnPXWcjoVmz7aTT9B1WU0kOPzYKSHhCpGA6?= =?us-ascii?Q?9j6l/KVd0hZBmdPPxOx7/zhHTBAqTsqral0Vh82xj5qWYpjPk0+WCI1X5Ddg?= =?us-ascii?Q?Y0q6+dXOkORFuj7cQT7dF/f1gkJ3jEh46uF+Q1bKvjzmtLWBM+n3OHFFTV0H?= =?us-ascii?Q?YLBBM275sQZvmsCy3z9uZAQozTeBfSIpytKSvxj9Q73/8L2sPaT7z57rP89A?= =?us-ascii?Q?LXcIo3WSBpjHqd01jBkFvIywsg42IBxlFilwApZuQhggbsC2/2DLCmSLRll6?= =?us-ascii?Q?BLU8N/doHN8Mrbc/9LyIFI3nIkXA6p0YL/AOk8xl2DnEbJxke80z1FWzSkbT?= =?us-ascii?Q?CJ6fF4zl/pBN6flb/Z604TRXk8OyWeQDFjaaW2r+wISHpHvDGh7rrj37tm46?= =?us-ascii?Q?KnatrzbqAV1qat4qyELIC11sU3zJDoq5uoyiXUUbwO5Kjk3b1tLwBz4vaZ4W?= =?us-ascii?Q?0Sl3n+yYAowf7aRxx0vrSwYQGaLm4NRvNmLqes7+DUorCyca3c0UW7qLzpq+?= =?us-ascii?Q?HYyTmstRZgzMY7j7poCe8x1IoaS9ddAnpeAbF9e+ny+JYLidhtAbwvKiYf7X?= =?us-ascii?Q?Y5pW/2kWIkJHBXKD6GEFJJyelIHnpKEkYTCYU5WtoAr4eAH8GXETvgYnTxOg?= =?us-ascii?Q?EDlY+SvfiQ7VU+mlng6DzAo7qDEBvbL42LeaFOb3OlNyvtAB33+INPwviNfX?= =?us-ascii?Q?BK2CbSsEEiILwBYOtpLkvwKDZ6kYQiftUoicbosN0rU7Sz3QknkHlUttk01q?= =?us-ascii?Q?fHUQZuXu5O6bMAF5Nx0ZXy2uYdqH01oMYyn01dizUsrx62KJTHexCR6jr3Ln?= =?us-ascii?Q?vFE6Z7nt9R8F/fuMLYYfbLr0nkA085EivhdXxSDXdSMq0lJgB94hcQ+pHv21?= =?us-ascii?Q?gw=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?tnGmb3xHPtClVn7B7yO/h0zVXfpXvaD92JMcMwbfVuJiDpsmG6/RcMKvKonf?= =?us-ascii?Q?DTmSMj5E21jEE1/jtMntlX7sRNnasY8HkpH0NqrX26YhIlJ5Zl5TtIxxGgEH?= =?us-ascii?Q?h/rIeEI+9yl+PheivwGJICbpMkl3jpo2Ucl54HE1+yH65r04yZ2T/drlmFv1?= =?us-ascii?Q?o0jx0eHwa0QW+nQbT+Qx7/LMRize21YkSFCKs9ZWZmMGCdJ9Y0s27kJ+aUP8?= =?us-ascii?Q?LQEYadwT9c1T5yJjRCRm6hQsGOM8cqIDlSCw+z9hpYuI5Gxuaorhu1Z5/mnJ?= =?us-ascii?Q?qi6pVib2k/z6z2qaw2jgEOhmqXB/CY8P4pp8LueVblek2erLVMyE5Z2LIzn3?= =?us-ascii?Q?gyGoXTue+64XdtWMxIdwnPdmxrFs29VKRFWC5AcoEO78lCMo6hlMIeLP+HXa?= =?us-ascii?Q?iEpLvmKvGxgVwDIRJ2y+nh46TzSfOtMPc98Oy6Va63UMWnCCY/xbo4rPhbb/?= =?us-ascii?Q?Ja6fqyYVW/QJIDt2Bg+vU9AuzvP3Pe9+47ScZn9fCMi2bkN3UKByeRSXcxhR?= =?us-ascii?Q?wmRKOq2HhK+iWGZ+bKzub9HPvXzz0ore7bnjr59+7AtN/jqUHmFG0jcJsPNx?= =?us-ascii?Q?52lsdGtRzc/Vm0aL3LQb1pMN/LnwPrPFI5mNYH9Q4TBoXBok0Fr1DeQhqsfs?= =?us-ascii?Q?qJllJKP7rhl1s7uj2glCYT1r30lrG1IvbD97heKMxFzySATmWiE4M/Mn0IXh?= =?us-ascii?Q?Q/A58vHS4YXucrdP4RirP5Pj+9+fYWTLYTshM/2kBUT9luNLs/YMkuTIZmzn?= =?us-ascii?Q?D2zprLpnP3hidRJyGkdnQEAI9AE4Yx6AMIp8STqOwpDD8+PoxJB9u6XnxFUq?= =?us-ascii?Q?OqGLglX7suUOREfvM2h9VD231p4ojc9uPWdyiYPC+Bv2Y/LSZYFdP2JY3b06?= =?us-ascii?Q?85tg0Adjfv1f6DLPUhrPRVSr/SRKU6gpY+FAwXZueh1eJ40JodgWr1puo9si?= =?us-ascii?Q?cby4/YWcyQXtfvOxbhM7dR3F1lzl8PW3fWTxNZI6hMfpjnjH7/ABaMVczWD4?= =?us-ascii?Q?Mm3yJ7YItOwmIAl9Oaz5UmduHgod6CIUEq4FYpFbQFSrZbYXoThr+JTDoPZx?= =?us-ascii?Q?fnWSGeu5/2kFcdcpZtRM96czouHSHD0z3svrCHZZZkdG3M2nMWSRPdWBf7qD?= =?us-ascii?Q?Cgm2BebBsoAx?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 08201307-93da-437a-b798-08dba59e2fb7 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:02.9823 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tfwoHS2vp0n6TJd2zHXisUnCQd+G+e1IHJEkDxxq+9EctD9DNN/HVzJEhLDQLNfTcuDPja+Zefzq2QeYXGN8Uw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: hA9dqyJoFfetm6zs0B3tHWZsYX3wSaHo X-Proofpoint-GUID: hA9dqyJoFfetm6zs0B3tHWZsYX3wSaHo Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Allocation of a hugetlb page for the hugetlb pool is done by the routine alloc_pool_huge_page. This routine will allocate contiguous pages from a low level allocator, prep the pages for usage as a hugetlb page and then add the resulting hugetlb page to the pool. In the 'prep' stage, optional vmemmap optimization is done. For performance reasons we want to perform vmemmap optimization on multiple hugetlb pages at once. To do this, restructure the hugetlb pool allocation code such that vmemmap optimization can be isolated and later batched. No functional changes, only code restructure. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 155 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 119 insertions(+), 36 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ec10e3804060..452e7ed6dc36 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2146,16 +2146,9 @@ static struct folio *alloc_buddy_hugetlb_folio(struc= t hstate *h, return page_folio(page); } =20 -/* - * Common helper to allocate a fresh hugetlb page. All specific allocators - * should use this function to get new hugetlb pages - * - * Note that returned page is 'frozen': ref count of head page and all ta= il - * pages is zero. - */ -static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; bool retry =3D false; @@ -2168,6 +2161,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct= hstate *h, nid, nmask, node_alloc_noretry); if (!folio) return NULL; + if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2182,32 +2176,80 @@ static struct folio *alloc_fresh_hugetlb_folio(stru= ct hstate *h, return NULL; } } - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); =20 return folio; } =20 +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) +{ + return __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); +} + /* - * Allocates a fresh page to the hugetlb allocator pool in the node interl= eaved - * manner. + * Common helper to allocate a fresh hugetlb page. All specific allocators + * should use this function to get new hugetlb pages + * + * Note that returned page is 'frozen': ref count of head page and all ta= il + * pages is zero. */ -static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowe= d, - nodemask_t *node_alloc_noretry) +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; - int nr_nodes, node; + + folio =3D __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (!folio) + return NULL; + + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + return folio; +} + +static void prep_and_add_pool_folio(struct hstate *h, struct folio *folio) +{ + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + + /* add it to the appropriate hugepage pool */ + free_huge_folio(folio); +} + +static void prep_and_add_allocated_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) + prep_and_add_pool_folio(h, folio); + + INIT_LIST_HEAD(folio_list); +} + +/* + * Allocates a fresh hugetlb page in a node interleaved manner. The page + * will later be added to the appropriate hugetlb pool. + */ +static struct folio *alloc_pool_huge_folio(struct hstate *h, + nodemask_t *nodes_allowed, + nodemask_t *node_alloc_noretry) +{ gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; + int nr_nodes, node; =20 for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - folio =3D alloc_fresh_hugetlb_folio(h, gfp_mask, node, + struct folio *folio; + + folio =3D only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); - if (folio) { - free_huge_folio(folio); /* free it into the hugepage allocator */ - return 1; - } + if (folio) + return folio; } =20 - return 0; + return NULL; } =20 /* @@ -3184,8 +3226,7 @@ static void __init gather_bootmem_prealloc(void) WARN_ON(folio_ref_count(folio) !=3D 1); if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { WARN_ON(folio_test_reserved(folio)); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); - free_huge_folio(folio); /* add to the hugepage allocator */ + prep_and_add_pool_folio(h, folio); } else { /* VERY unlikely inflated ref count on a tail page */ free_gigantic_folio(folio, huge_page_order(h)); @@ -3231,9 +3272,22 @@ static void __init hugetlb_hstate_alloc_pages_onenod= e(struct hstate *h, int nid) h->max_huge_pages_node[nid] =3D i; } =20 +/* + * NOTE: this routine is called in different contexts for gigantic and + * non-gigantic pages. + * - For gigantic pages, this is called early in the boot process and + * pages are allocated from memblock allocated or something similar. + * Gigantic pages are actually added to pools later with the routine + * gather_bootmem_prealloc. + * - For non-gigantic pages, this is called later in the boot process after + * all of mm is up and functional. Pages are allocated from buddy and + * then added to hugetlb pools. + */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; bool node_specific_alloc =3D false; =20 @@ -3275,14 +3329,25 @@ static void __init hugetlb_hstate_alloc_pages(struc= t hstate *h) =20 for (i =3D 0; i < h->max_huge_pages; ++i) { if (hstate_is_gigantic(h)) { + /* + * gigantic pages not added to list as they are not + * added to pools now. + */ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) break; - } else if (!alloc_pool_huge_page(h, - &node_states[N_MEMORY], - node_alloc_noretry)) - break; + } else { + folio =3D alloc_pool_huge_folio(h, &node_states[N_MEMORY], + node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + } cond_resched(); } + + /* list will be empty if hstate_is_gigantic */ + prep_and_add_allocated_folios(h, &folio_list); + if (i < h->max_huge_pages) { char buf[32]; =20 @@ -3416,7 +3481,9 @@ static int adjust_pool_surplus(struct hstate *h, node= mask_t *nodes_allowed, static int set_max_huge_pages(struct hstate *h, unsigned long count, int n= id, nodemask_t *nodes_allowed) { - unsigned long min_count, ret; + unsigned long min_count; + unsigned long allocated; + struct folio *folio; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); =20 @@ -3491,7 +3558,8 @@ static int set_max_huge_pages(struct hstate *h, unsig= ned long count, int nid, break; } =20 - while (count > persistent_huge_pages(h)) { + allocated =3D 0; + while (count > (persistent_huge_pages(h) + allocated)) { /* * If this allocation races such that we no longer need the * page, free_huge_folio will handle it by freeing the page @@ -3502,15 +3570,32 @@ static int set_max_huge_pages(struct hstate *h, uns= igned long count, int nid, /* yield cpu to avoid soft lockup */ cond_resched(); =20 - ret =3D alloc_pool_huge_page(h, nodes_allowed, + folio =3D alloc_pool_huge_folio(h, nodes_allowed, node_alloc_noretry); - spin_lock_irq(&hugetlb_lock); - if (!ret) + if (!folio) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + list_add(&folio->lru, &page_list); + allocated++; =20 /* Bail for signals. Probably ctrl-c from user */ - if (signal_pending(current)) + if (signal_pending(current)) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + spin_lock_irq(&hugetlb_lock); + } + + /* Add allocated pages to the pool */ + if (!list_empty(&page_list)) { + spin_unlock_irq(&hugetlb_lock); + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); } =20 /* @@ -3536,8 +3621,6 @@ static int set_max_huge_pages(struct hstate *h, unsig= ned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - struct folio *folio; - folio =3D remove_pool_hugetlb_folio(h, nodes_allowed, 0); if (!folio) break; --=20 2.41.0 From nobody Thu Dec 18 01:07:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11822C6787B for ; Fri, 25 Aug 2023 19:06:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230016AbjHYTGV (ORCPT ); Fri, 25 Aug 2023 15:06:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230293AbjHYTF4 (ORCPT ); Fri, 25 Aug 2023 15:05:56 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A294326A6 for ; Fri, 25 Aug 2023 12:05:52 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEWaI031486; Fri, 25 Aug 2023 19:05:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=hgjjLKzxatL2kLeVbyvUq9gKeYfImLdjnfaoRhKBwD0=; b=z/xyJX30XZCI2BsNidlskUASj7IscFRb1fZd8bEPimLqCabKlc9o+Z+JooI5a5wclB+n ksqugs727M3bGNT3/SfFBaKJvvC9Ne3n6IurVD+hDZjT1zpiPdssKLWX93RFuaMEcTk+ 3gIgxqngdlIPApKKck0b72HKcSK3EEwLGdtjA+Sxnrx8YQnySl55z4Jn6IZOsaH49p1u MhZlOzXbLvGnTr/aPTE2I0GQ2Qlm6Ne/X5MxSrNDWSI/LjiTEFuA9cyFaEixPifWuwJY RtAxEzeNiPPGu7SqLvmqIrnvduPDbDHOM7qj74m/KImZWaJz2l2JI//nAwM8ondUTljO Fw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0dn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:09 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIUF51002259; Fri, 25 Aug 2023 19:05:08 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yyg8k5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jWrP9Qjf7um6WvLwmSCGrDhukrrmpgI1AcZTV5KSNEaeCiVIBXbzJLNzDqKyrB5ObIzKG0ij3ekKsiUqFwxg+awarjuunvAqOOB+yUBmUwRIWQikz2T3X/OdlkemQ3XVZATgOYoPPMchuxsITL12FangHZqIEtypOrh8wAMmCeZkpwDKj+5gdG9CKtyZ/sGvlwpZCaQ/YsY2ynYxPLhu9VbJAyOErVxfcZx0lBEOuCsx4VoslrVkXSKVktXRGz4juJ5zZAS7QhrwfcBjJlYtUvmwce4PIX45AnS1+uR+a5LyTCEXjnN75WwXJOMFCAgt1t/Hq3cGc9kUUTkGaBWCJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hgjjLKzxatL2kLeVbyvUq9gKeYfImLdjnfaoRhKBwD0=; b=THr6BKYBNTfAKpinJl98k1axbv0NvwhzQq9C05NTwdWkLUrlGFRgKwp0ITJmTGYh6ZLQZmKBM8Vlz7KCWi0hsQ7sNImQGqONnl0WiWX8OqxGnMzlvuPUTfdAB1r+Ntsvm+5kJCj9MPYxo7krlJAAoWPlO1R5AkW3VWnHclz41BWRHKOqwxlqmeMwcJNgcinDslSRhJJ7I44auv+ILaKPHOpjs7rj0B/o/YzSDXJ7HNx2KgWVDLUAprIu7F2hybSUwFP24pR0oHjxo+pn1JsybFXm2ADaFlOq1xHmMXoXQLstTosm5FEqgkbc7F76p+ME64JleiNbr+7ClK5fAeqYHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hgjjLKzxatL2kLeVbyvUq9gKeYfImLdjnfaoRhKBwD0=; b=mbIFw+4FC5egAb6bkI13P7RO7pWnN315794cOe9Rp0cW06Ts7e9Vg2ukzD3OYJhVv6P6QWhPaFlFVc1Q3ULqbwqgmO0jjsXmYhT7xNPAJQrgqy9MenLTYgwlFa0w3xQgyeNDXTKy2Tmu1UVUYvQrnivZCm7hu6yNGNMomvURBao= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:06 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:06 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 06/12] hugetlb: perform vmemmap optimization on a list of pages Date: Fri, 25 Aug 2023 12:04:26 -0700 Message-ID: <20230825190436.55045-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0208.namprd04.prod.outlook.com (2603:10b6:303:86::33) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 483753e0-17ac-4bfb-518d-08dba59e318d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hnxJSfTikXoHKsRYQD9iT+K3jM6MTMtZxAy/XGK46jZ7HEIpBoWISR8uDWPFn8++wAZ8LYxx1KqAUwuCO4M23mY3qPJyLfc7Oi3/sHTjeHqVM6le0uk157eFpQ846/lBQ265kBLXJQ4Rr1qoxUJ0829D1zq8cDeDoceWXKEf0hd2DGXoVb7pHMI/UVusiF5oqsanZw1DDC6EZiNVkthRH+Yhr7L6hDf8oqPkKrgOtaXdsrGkyV9UjgLiptoVARfZqC4/4luEPBoAy4UVUrqwD32bVV95LQLCmthoq+k9MM3iJTQUkKbzaFUMR/ycdjDB1hX+oBnsMgJiY6r+syOmHquyysvm40VQkiR8aVV17QG6niCc5PQ+0nTtC1k+ujXc9HiBKf5bip+GKVGrbW7Njt4VrXEdqWezAqfGD/gXfBMukZU4MH2zpbiEGN148iE8xEBPIAqiJvmeDvAysspYSvptZ8WeEcLVTxhW9arqqPV+e6bnaTHsHhBTIv1yZBS4eksQPzoN3IKsPAp+iNnDl0IRUnvMlZsmRU61oVryVR4Kbxvll27+FcTruszQFnoj X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?5rB4xzcCPyDfCyc5Hc3/aQ6oM98DT7rh2cRVOm85p7CdUdnjDz+cfVDhft0w?= =?us-ascii?Q?93PBR9ANwIf6U1rK8D279FsQBRfgcPktOYnnY+1GcfXGNXBSk+27LRfc9MG/?= =?us-ascii?Q?wCyskxJM4CDLcEN7OYNPlUQP6vOkPp3mpnOxRsvppWIrQvkdwFAhT7+TBo36?= =?us-ascii?Q?5HbOlEi3v3fh+piTzyR42VfuF0Pt4W3wWJC1HzvqgiVfkrNmzkUPF0jnb3EX?= =?us-ascii?Q?O21nta2qItfQVCTrL+7WLnj9NuTsF3p6HNjMFu6vPz3yY4DyU2k0Cv8724Xr?= =?us-ascii?Q?1iqs1572ItAygEOOGSeGAgmdWTwIWtD+KZrJsdw7uqG1Y8iFPrldBd1UZvKN?= =?us-ascii?Q?kRdrcuYHzOhOmKlaA9RFqjPRI1PbjWCh7vpbfpAMYFxKHIwDd3X5QhMJR8xd?= =?us-ascii?Q?XY7Fza0s1PGMrn7aF1YDntUmbEpFOEy7SBFo8I0fuyqG/BHV8fS34mQ41oCD?= =?us-ascii?Q?xK1278cTXJkMBThIpK3to1IY7CaxNvOfkEB+CA0lYHudcyH2T4Duwq9DQJei?= =?us-ascii?Q?9gS42KQrK/K3AwtwsS2fLHd8rxXcYzIanZ9ox2IaYhHj2mcHDgmaP8eAlm+V?= =?us-ascii?Q?d9525N5WTESOn9t0B3i9KaGopioDym1yidAuB38lIcijOIRKG3U85xH5gAId?= =?us-ascii?Q?QAtcrVHhJvEGngPN8SL0DJNNFbE0jiG6sUhqph6QsBi47twaxJOn2cPD0xEJ?= =?us-ascii?Q?AnSqtFIqHA0Yl0W/Jp4edzzYsYyHEKPbXy4o1YVdtaZfH2RsQ0eUHgrLsHuJ?= =?us-ascii?Q?oce2sj+IgVbXCHiQJdhOIlCm9cnkjOfFaOkigKxv4glwDInvMQyX3xqmg3Ut?= =?us-ascii?Q?BXpth+vXoub759RsC129ZwxKsEiLm/SWBfZqQxYMQXCqYitrH96XO8Tlp/hl?= =?us-ascii?Q?XjfQAOnoFaaenHCIxQP5xRtSjoNt2okG/EUHIGaUo00At50r+aKhz2IZrVEE?= =?us-ascii?Q?IfPXvEiE/6PQWAmtLAps2ynfcPgpMUMjbWom5EnEe7AO2xXzgJOTiZiQnp4Q?= =?us-ascii?Q?eCtCtbtUb98k+cmvdgVBzSsi6i1Eo1txescs+o3CGw6vJhgCXEzKZc5k8RDZ?= =?us-ascii?Q?fcf5Hzll5Cn3jh5qEqxQO9c/ki35QYxpKlzriVbvfySk9oI5rqrLQWLdZEg6?= =?us-ascii?Q?gql3Cw8K1PRXIDQOUlNCCtR5EngpRsrWk/6NpNegLFCDCp5cDskXsTNTfgTJ?= =?us-ascii?Q?e+dEx552XulxmSZXpSaNWUcbVRIFr7Wh9fQGjdOhw9RWdB33WjgxssFERqtv?= =?us-ascii?Q?ai8iOwIKCy1pkhQaozTPsPoaVg8hXPoukDfK4YEGJD6va6dNqguQ+yeCNR2p?= =?us-ascii?Q?DKexJ4Pvdv2USJwCdnxQ1j5Z5WHlvQtaHY2EVeJ1SF9UTRNYxhqsN4RjpPLR?= =?us-ascii?Q?XQUFHslkNw2m3C2eDCBS9r1hGyaedAAv4DzeapPqrD5dUauSAzVo4O30Yq0D?= =?us-ascii?Q?caOmwrhMD9cUZKpQf2jMzPEn+x7UNHgSNNujXIWdwXLiVwURNLgaepwuluyM?= =?us-ascii?Q?ULG+N1vfdBiTxc/xfodwmpHAbXqDAXEy/sPFc8cbG2tDEl6N+KJselax2lK9?= =?us-ascii?Q?bcf0+Jewv0rSvW9Zs7wOiBlb8YRGVMPDEdwGd4HWWrjj/Uh6TsedaJkB3RGR?= =?us-ascii?Q?+w=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?wspp+Rq31laP+o+2y8YLGL0kHUIVl3+YPry7w5JVbxpZN+C4QHqWuvYAHYBw?= =?us-ascii?Q?xlaqWbMULS8XxOTX/PTsu3rR4+KPRiCE/Mw8ZZdCdFqoXPdvLz9lnBqXT6Us?= =?us-ascii?Q?1W11VBdkABKQ9wTfJRJnfUuIUtmiQ6hFj6MaycrBhg7s5c8uuwD39eO2pDgB?= =?us-ascii?Q?drioz/u2YKBkntQBsz+6DlUGSCigvxL8ZwfLKNGKxdQdbZs/npko62ns8Xps?= =?us-ascii?Q?zzr+7twyPHHbho3PaUY0UAwiPHrTLne+Vkjpwe40wtBTNyRa6qt8tYdFzh4J?= =?us-ascii?Q?5/6XJDGwe8GyoDBuo91YMvr7QO90Wxuw39JsoAD4PO47Ptp4yeIKlKiCM4bG?= =?us-ascii?Q?U55l+U1QAFBjOibmsqaYlE6O5qiXovxEjNWj0hKA7cYKMYYxCCB+oJ3FJhk5?= =?us-ascii?Q?128GI95mewG5mynlgR7sOl9PzWFiquA4UWUt3qZIB4C8TXqY8ojOiUWkmSJf?= =?us-ascii?Q?0IC2wYuSRBIt+PEhpTFEIYehXsyk7CdRz4vnj0Z9mMpbrtI+xvrasYPzk8JN?= =?us-ascii?Q?5D2QWGtDyFqpLc89EQOf2Ogf6aSKZUHreCkLJH7HH5pKq2izIGCfQGDj0gQr?= =?us-ascii?Q?hrrt8e6QCTp1UDixYd/6LHkgfbQZXPLHeeruAsuGdN7plhfFwZCKUUtzLF6o?= =?us-ascii?Q?OdXhWsy94fD5uWcRxxX77fNwX/ptIgylzdZ74K5oVElI/wIfWPd7oYwV0Rng?= =?us-ascii?Q?DfYQqHOaD4gOipnV7ILCGL2n5ZKYtm96aFjK/3ZInw7boeLaf3U2224uu3Ov?= =?us-ascii?Q?e6/TLxiacrU80NSpcuKTAm8TDYb4n10BwNjSVJr22PuIivRY4jWp/GJt1JZ5?= =?us-ascii?Q?fLe2PWVkATOtjiZkrTREv6dsFn2saRHyXc9ea0lOivoxHThl4kXUafhBbTBc?= =?us-ascii?Q?AxibRCuRYb/I4s/KWOUXo0oz5+Eoor3Luow+ecUFXpH0LdSZ6RGfi5mqkLGa?= =?us-ascii?Q?CKQ1HroxFUr6EjFgxME/7hJ2FYlAOntvAKOoPCOG5n5TbvZEoXIB2mybVpbH?= =?us-ascii?Q?GyOCk78At6aZb8FPjRaLuoNFps+gwCLVIXVclMNyB0WPmdHZLqqUY5ZUuf/l?= =?us-ascii?Q?Z6OMe++KzIRgMsSF07fFKrPE0QRCrh3MBGmcPlgr3MRDUyADdkXV37XOuXqq?= =?us-ascii?Q?GOtYiotx6LPQ?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 483753e0-17ac-4bfb-518d-08dba59e318d X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:05.9692 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: dX+NjcmF7KoWhK8wcA4xqAm5kCIA1KOFBpz4srQmQ5MoLXa43OPjiibUGHBzumcX5NvziaD4/g+aYTJPpR3Ykw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: f6SQZLyQGTPDeSD3UkeDiDl_BnIG-U0o X-Proofpoint-GUID: f6SQZLyQGTPDeSD3UkeDiDl_BnIG-U0o Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 8 ++++++++ mm/hugetlb_vmemmap.c | 11 +++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 24 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 452e7ed6dc36..3133dbd89696 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2223,6 +2223,14 @@ static void prep_and_add_allocated_folios(struct hst= ate *h, { struct folio *folio, *tmp_f; =20 + /* + * Send list for bulk vmemmap optimization processing. + * prep_new_hugetlb_folio (called from prep_and_add_pool_folio) + * will also attempt to perform vmemmap optimization, but this + * is a NOOP if already done. + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) prep_and_add_pool_folio(h, folio); =20 diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4b9734777f69..147018a504a6 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -482,6 +482,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, str= uct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct p= age *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; =20 @@ -570,6 +573,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, = struct page *head) SetHPageVmemmapOptimized(head); } =20 +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] =3D { { .procname =3D "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..036494e040ca 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -47,6 +48,10 @@ static inline void hugetlb_vmemmap_optimize(const struct= hstate *h, struct page { } =20 +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struc= t list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct h= state *h) { return 0; --=20 2.41.0 From nobody Thu Dec 18 01:07:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35551EE49AD for ; Fri, 25 Aug 2023 19:06:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230322AbjHYTGZ (ORCPT ); Fri, 25 Aug 2023 15:06:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230457AbjHYTGD (ORCPT ); Fri, 25 Aug 2023 15:06:03 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DE4C212D for ; Fri, 25 Aug 2023 12:05:58 -0700 (PDT) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEIrP010977; Fri, 25 Aug 2023 19:05:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=3yqqBf3/EUJhnisgK8UhLekUfEAaatv4D5tHikWfcW4=; b=LCV6H6GrK2qc8JPE7ZlMcsbBOPJrAw86cJ/5kRz2NgMYJ4wqofJhFm3JgM+zqxkBUD95 mThA4kFvetNXiUU3Am4VI21L4HsyyHNDcJJaNzCHE9789qiJ/1NsjYq4kWFENyRz+03W F8nhvEZbzPiSyk6XV9uMirZR9ITMHo6LLmNGFUmIODC1SijZGyX4jIiNA/YqxDfHj155 0sfGKJTJw7JV/yBzxE9D7Qa97852G3FFF2TUJi6MhOXUsyBpbwMMEWCZ/LgpPzE8zB15 FnRTXSDgbxoJ2BwRgiG40iJw7cjUzeV6+V/AWT5uhDuE5wHNxZJMt8ibs4i5WyQn1d0d VQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yv74ss-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:17 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIhRpw035774; Fri, 25 Aug 2023 19:05:16 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yr77vn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AaMBVRSljPbIc0qe77chXvcGvfvJ0PYtMqt/JLEph5RfXd13Ue7f7WZq/ScN8NtIM9hRu0ifD/j8YUj8fte9Dsv0nt0hjISNhX8kNs0wS0tA415lwEpOw5NBlMRd/PRxM1ACRqwcR4EW8RvA02a7XpGoiwsHYwIVCiMasJgWBmDlLOwTYlQrr7uQM+ZN4nBShtAOuTjviVhqJF55lG45GJ3DsmudoL4/DtmqTiie8/9vMnZOZLTu3LkgExtT3FEGVaHsnknb9ytvl0s36jrADqznz2ziQxlZGXRNoNbfkO6PkKFcux966qCXgpzVmfDlmhKkB4vWBxlgSBmMGnD2Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3yqqBf3/EUJhnisgK8UhLekUfEAaatv4D5tHikWfcW4=; b=fxCkksHL0lvJwPGyBk3+6WT3mcB8zB3OXQHqPVLo3i2dG8b7ZJGEIQ8KZrDzZ4G9e86TIXeY2DOrOlnXfn4Pgk+cmG+6BtnhRxOXmjduQ5bp/0ybnGawAuL3AG2q5YA5+8TWSFXcEgOz2/8+HdG//aBZEG9Ue210WSDC0oDYFSt8XPuNzE5RdZ6Kl251nQC8U7RvnNiu5L7LJm9lW2EZpvS6bF/vdSEK92R/OFE3i5j5VDxoOarHC54tRq3Z8Xow1mPnqw95oYYr539TFZLWLB2Bea8hpKoPVEH/n+ZHdeTcbaiRiWw9kZXN024uENNbwcYE0H+Lj2W7PQSqEKs4DQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3yqqBf3/EUJhnisgK8UhLekUfEAaatv4D5tHikWfcW4=; b=OAmZc/4ORS3/ItrMeAzf5qWLA2nzpPHtj7R/GXXIf8LoO9VvP3b/0hAA7MBwwg16bBBQkJCJRczUQIfD4uiZ2YfLaTR6ATG0d5EBrkhno/iy33b8UR2yCjOipmOjJm07zOjnNSBg1ZxHGR5Wdq5hP6vNWp8SmmthCxE0hPhVjtI= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:09 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:09 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 07/12] hugetlb: perform vmemmap restoration on a list of pages Date: Fri, 25 Aug 2023 12:04:27 -0700 Message-ID: <20230825190436.55045-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0179.namprd03.prod.outlook.com (2603:10b6:303:8d::34) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: cd8336fd-8c41-481d-5fbf-08dba59e3378 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gWBpvmOUKAZohKU7SVHXIsS6RXZ7jKx3FM9PbvhIn1rtfZEmLlhqKogWoDYG+tXqYki57aQpJa/ffxF0lig8Xb8zsk7EzRjqFEhiVg7wq6eq1Zrzn9qiAedNq89U9vUQYzJ1KCWk3Smn2pafZv/sXNakSs0Xp/L9Uy5pgf1Z0zDJoMI3YRvjhMO08hddkgyp2/5VJd4aaDNquAiMO4dsOYdKCf+cc5oNa2h7bi/pTPBRvxHTRZRy9RZTdCDVka6Xhwp0QgAcvGpd98wrOGQYUgsT04nb8DkKoY9yi5B7B+XwkBRUSNdHNyG00g1PRLQapyCWJCuigykNqs28l3QAhZKIjrUkT5FvXcMTJhB7daZt0zN4MNyU0xazVx2KiesPIH5IABnp/rDET9szruFhoAkMxgRkcAsKeEgsC10DpEInD3sA9+0jCf+TQgBukpHTyA1sSrMOOPKWZ//qdYovj+eqSu+x6JBRXRUrKsmIwHqzsjn86i48rF/lxHr8znEoAFtAS+cTnIJMWEEH/5U40SVOkTPK/JEEOhaoCTGxgHKXNxZP7eTObPvwcgxjtj0B X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Guh3FDSOHYCmwWMJCnG0kxAbT7HhTPqf+R5FXkuhb0UULlrui9ro4MYLF5ss?= =?us-ascii?Q?npx+FKEH+Hqv4+/7b188P3yxY3kkFJ+RenF+PqSQRZxafL/AbvbWXgqp26BY?= =?us-ascii?Q?jvxO26soLFmtX2VxICHEQJCDr54h/jlrvAHvlbNqVMhUoE+FaXK0TMvjdt/m?= =?us-ascii?Q?2brGcdmHn0+3p6MLf2NQcmvDTe35206L1jcTNw09/FS1BmoiropyJORluh6C?= =?us-ascii?Q?xlc+trZXHncknypW6meGIXgI7tUhgL9vRwUU3fopFhAoKMYEL6nUfMPokqVW?= =?us-ascii?Q?tZLfRwh8mZKXM3CBBU1SWNOIkJiwiz+QS7BY9AOPJJgSOdxtf99jnbswPDGM?= =?us-ascii?Q?m7h1/oW384UmFvRSSbSNAk35cCgQ6n5VlFWd5QnRxlw+m0P7ySu9WJb74911?= =?us-ascii?Q?6QovDDh8IsQ0QBi9yz8f1+YG5xv/9Ru527+ZMUGXqAZgMtIwNgXBQXB0EOSd?= =?us-ascii?Q?nxrlkTwp6+uSvpnkhKMdar9kEZJcNgLBFdh2TiNu3xzyFLMfGU8VI4i6LCzi?= =?us-ascii?Q?dDLKqMeiO3WR43DFp3QSjihOGo4GU69F8g7M1wnClAeZh7nQbOtpVQhc/r5z?= =?us-ascii?Q?9NbedPz6XgZl035ih2tHmHt7cA16baM98dUGniB61T/kC6tuauveyESuqgle?= =?us-ascii?Q?VpTKpQCTbIkYPKmrtnqJ6ecUXM41hOvVFEnep2GEZNep4X7r/QZ/mIYPC1KV?= =?us-ascii?Q?H2EODr83B/bSk3M+GgHSKHL6Hjwo3RUVEcWTOAdObFk04q3cwggw/lV/ZiU8?= =?us-ascii?Q?X8OIYLZeALlJ2u6Qsf3YXRNaPYh86sR+t6/FmYhlortrcQzzAakwCUZplm4Z?= =?us-ascii?Q?pxH9snnQ2Y/ze7KNra+l6av5FbaibqeJ5c4k8LbISEi941NqvvsMote8Q+hG?= =?us-ascii?Q?2q6FFs7Yyd/6xXzwGFhiTrr4oUy52ZmLvQYb109/THbLqyoci8ytif7jYStn?= =?us-ascii?Q?RbMDFkHS7S8FLRd7C04lsbF6asMR3v/AbWPIKvEcgOzYT3Sq7d7Y2L8jVzjo?= =?us-ascii?Q?7VVfXPwvtMqXzw47fHLT5pdfkFNXooSTIP8UEPDKH5SiaQnmvShnIPf8L+bY?= =?us-ascii?Q?qr0zWPo/8m+VByAwDlNpAUKhCsuqYTKW9VYEpfSP/9Kigf6NlXvaut92n6sF?= =?us-ascii?Q?MoUyvm9I8TBkcTKI/t6WaJPn9GWm7kaPadJ8Z0nV6itKT0Bp05HgbtEVSeoQ?= =?us-ascii?Q?ofymR4mT/zAnDAX5CqbemGS1EMzAdmXfHaDooh129E75++r8Iz3vrgq9PiO7?= =?us-ascii?Q?cMd9Q717RNT+TH/sdDzqop9Z/ut8+PBbqYpAg5c1di3x5B+gqvTUXf55brLK?= =?us-ascii?Q?DRtfLB7S/8WDGvLXrWsH2r2az6YzH6N7qsJUEKyrTpPIEyTtAJ88RfMxjnn6?= =?us-ascii?Q?XsuxOXib9td1i1oIUD73/Nqs8yx1gl+ROlOGrvHD0rVPajFxE8q9Q8BQX5A1?= =?us-ascii?Q?kUdhmljcpW/ZhRCb5AvoLvUPQnZRqEaIK+4EYSzPexqep54NuG0DnXSGd+QG?= =?us-ascii?Q?Rhsdi2FLoKi/mERJlGm64EuPMLZDszdaJuJ2QBJKlCzVX/sDxw1m95pUYGzw?= =?us-ascii?Q?8tygFZbNXuswqVdcJr+DxW+YCI+EETWrqyVcqLy960JVbLewAzWjtcDc3uG8?= =?us-ascii?Q?tQ=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?DUXQQzwraCkXPAGLcbEU47Jvz7FPADnT6AKCtWvOgL8KBVKzlb+cLNA3E4qy?= =?us-ascii?Q?KakqfCNRqnk30S46M+iAwjQxs4Lkx/SuKczsVpsOq9RBuxfQ2vhu7BRMMYiC?= =?us-ascii?Q?3itjI3z7AuseH1QIabA+dznq6hz0QEh7c3YSNSyOQcI5mN8lw0NUjLLSKFF+?= =?us-ascii?Q?MqXruYoqBji2xI9WRB8MdBBXAuPJRMcTEJO6UYsqTjT/5JHRP4Y11HLhgUBS?= =?us-ascii?Q?Zm6cOHyx+psl5qYa+Dh8sjkzr58x2iRXV5FUcrmmszSRvhZ9rfomC2XuYHul?= =?us-ascii?Q?f+smiwxgQozGjJytA5awnumNoSPDPNtkSFZ+QOO4FvbZsop6+gsT0UKMeDlX?= =?us-ascii?Q?nYE5/IvikS+BoVnaWbPdE+DpuwAhLNyhcRqBWo/vMhKezLlywH0q8FQfRWg0?= =?us-ascii?Q?6WnqI/1+pw0h9JCyxxOVajOmQbGc8VDwJ619RNNtBHzIydPx/5fRMyIsHbje?= =?us-ascii?Q?yecC6tHX6oRGEC18HPMcW5/J7m/nVGtZ7aO0xZWyK95CaPPubjIIQxjrVMYk?= =?us-ascii?Q?4Q3lURg5ymGGHnqSh7AFppJ41kTOF5B42YO3I7Xmc+7c+YM1CQKWi5vEvsx0?= =?us-ascii?Q?7RMXIhMQszBBdc94KeBhsRv+NE2nVvXlsLjE9tKXsM5bnnx9BQwbK1Sswz+G?= =?us-ascii?Q?TgpOPQMfBqxU3u8YJPxjFlOS4OGK0ekBYYq5mFCy1edvOovsknGrz17i8NEb?= =?us-ascii?Q?AOif+oapbrAn7QRu6HvesDdehHQKFiPJBRWquD4XiF/uwJ9mYZTZ2t2jdu4z?= =?us-ascii?Q?XiyvDAlCwbr5QVWlAFVepjHEuSRioGtwCnrDWvllLvjX7NRp3lzrxL+x60Qg?= =?us-ascii?Q?KDWvK8/sWrWKhrCwhBcT3bEnmkIrtcH/CZjWllIOtKyQrSA9Gxx4FQYNLkTI?= =?us-ascii?Q?p47imrYzv/rV5rDTU4mjvaRi0FH30/Lpap91ijkzQPqFHxiNwDekXGER3zvW?= =?us-ascii?Q?4oqriH4u8XeKJAd31EWe9WUf980ZyScRNa7FD9PDaCC+K53KeycvI3LUooMg?= =?us-ascii?Q?mvdaNbkSFMs8/9u9nwbyCYp17CZnLXszQKgttRUNoHJ118i+lXL/xX2c4qHj?= =?us-ascii?Q?q3u/VbQYcvcV47YWOww1ODsSwhvhYlikzcgCsZ4NixHBdbmY1rMIqhYCTsMM?= =?us-ascii?Q?qnrZW2rIdtIg?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cd8336fd-8c41-481d-5fbf-08dba59e3378 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:09.1211 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: x1VXRNSRdiJYI2eVKdL8GpoYyZ2GyfvXKMfWooyCQRAkeKdmZdD7gJb/3pMYFP/9uPxXHri+Q/HKH0bmgmchVw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0 adultscore=0 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: r70cOVOm7hPTyVVRs7esswLDi7Jym02f X-Proofpoint-ORIG-GUID: r70cOVOm7hPTyVVRs7esswLDi7Jym02f Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When removing hugetlb pages from the pool, we first create a list of removed pages and then free those pages back to low level allocators. Part of the 'freeing process' is to restore vmemmap for all base pages if necessary. Pass this list of pages to a new routine hugetlb_vmemmap_restore_folios() so that vmemmap restoration can be performed in bulk. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 3 +++ mm/hugetlb_vmemmap.c | 8 ++++++++ mm/hugetlb_vmemmap.h | 6 ++++++ 3 files changed, 17 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3133dbd89696..1bde5e234d5c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1833,6 +1833,9 @@ static void update_and_free_pages_bulk(struct hstate = *h, struct list_head *list) { struct folio *folio, *t_folio; =20 + /* First restore vmemmap for all pages on list. */ + hugetlb_vmemmap_restore_folios(h, list); + list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 147018a504a6..d5e6b6c76dce 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -479,6 +479,14 @@ int hugetlb_vmemmap_restore(const struct hstate *h, st= ruct page *head) return ret; } =20 +void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_he= ad *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_restore(h, &folio->page); +} + /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct p= age *head) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 036494e040ca..b7074672ceb2 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -12,6 +12,7 @@ =20 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_he= ad *folio_list); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); =20 @@ -44,6 +45,11 @@ static inline int hugetlb_vmemmap_restore(const struct h= state *h, struct page *h return 0; } =20 +static inline void hugetlb_vmemmap_restore_folios(const struct hstate *h, = struct list_head *folio_list) +{ + return 0; +} + static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct= page *head) { } --=20 2.41.0 From nobody Thu Dec 18 01:07:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55034EE49A5 for ; Fri, 25 Aug 2023 19:06:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230288AbjHYTGY (ORCPT ); Fri, 25 Aug 2023 15:06:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230458AbjHYTGE (ORCPT ); Fri, 25 Aug 2023 15:06:04 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D606B2133 for ; Fri, 25 Aug 2023 12:05:58 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEPRi031105; Fri, 25 Aug 2023 19:05:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=JDqlank9XT0iTHl7uJPws1K8LVzypgRE+WIqntsMg+Y=; b=IhTUKsod5lykf5PsPX8PHVHBhgVkcRSXqe2IOxauGL1eW0e8iSLRsAMbWAmgk0pAYx7r P4DcvlpS1Um9Q5m31XR6xL9roKH0blg4Ul30NanmL2hKbA/Uem9pHJUULDR/OCuBsJmN HTwLRWLuR50eJc6K7+iu+Py/KvNfLfVfb9ookm38g5dMJDOIk51OSebsxG0EuH3djnmE 3+RC8zVwzU6o2hBYb5Cm09QEWD2atrSXcK4PZK9KI3uHyMLVHnKDhSKmaIJaTffeXcMH q1trYqQGGa+pBP+3rzUVg+ayRfmIAky3DmTEnVmjYmDN4ofb5l1vseYJ1KYkYVblIYLL Eg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0e0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:18 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PINm49036073; Fri, 25 Aug 2023 19:05:17 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yxqsmg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JwPgDGm+y4wgSJF6v1OViwtjzGfy0sZG9nhPBOZ2sSo1Vq0BCa1w8Q7tCBurLdXNdIJZQB9yCoIJdgT719z/82coz3hjW0mx1mru/Pr1cUw7RQsKA5bGpZIBNqsbzP2gdg/DP0SCUyJxsxeDdOPzwTa5e+pE1pgXLzYXJJAgzyhwBEUiZmNX0HPP4tgLIi4WfaPJzw9MtxyYlGlsE7Fl+ShuWR6bMVGcwi22McLbud3afMvZY5YZg81Rfsd/zX7dqoKV97rZ7v91iRA9uTgNjTAdqLnMNrg4Fa4Uiy5d1qhe0w8fSzqiTnkR4bc/548MOzGy+Ho97U28XqrCNmOdqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JDqlank9XT0iTHl7uJPws1K8LVzypgRE+WIqntsMg+Y=; b=E87cFLV7YQF957Afw5VM7jXbV+xKZD4a60FnVUZrYXWDLeH9AcQbrzkqxaroARh0wJT3yZQ2PZpxK9VQ13VIB+QgKn8Ng5KFTen1aRnw0TmEN53pL0k4/qNvS+gOwH+chCB9ptVG3dwpd+X7f+sYUIzX2VFnpw4O7SOIcuHSQvhZ55HdtWKlP05KZGApNFC3Ug6yAHKLQbjsCgMHshJ5swQQEH2qTFDWVsgZhXdgIDV38EXNJMnU11wYX1GRkqx+THwXK6+RqSwd/lXo/+WvBs/juv+Trhz5TnHiilYlKWA48BMbkoOVatoPGGwwzLjtyQAn2b5sbXoks8vKm3IGvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JDqlank9XT0iTHl7uJPws1K8LVzypgRE+WIqntsMg+Y=; b=odZEN5/LwlJEIqxrW8QHKwNdm9YxXZKCR10GiklbtNjI1LEbAx+cJ0Txr32B3JUqnsuUmo6yceLD7dSuElqKCTOi6jMGwEHxoLt0/vTpdLvbrrS6AW+xotN0zpuLoKIa+yK6LFdF3QGQ57agnOm0iwNO0go/thg15JHedfl1gIA= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:13 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:12 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 08/12] hugetlb: batch freeing of vmemmap pages Date: Fri, 25 Aug 2023 12:04:28 -0700 Message-ID: <20230825190436.55045-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0103.namprd03.prod.outlook.com (2603:10b6:303:b7::18) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: faa1ee75-cf92-461f-6b21-08dba59e3582 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1G07+/q13KIfsIP4PVmRWyQ/Blx7hgWZDi4K0J8+tpF56E3KB3oLL0A13Znha3YDvnL9FqPNL91GtSxm/i95g22jfnawDeSwx4wDzoXP3SMpxnPerEN7IWh6545wXA0nCK1fTYTGnzELhjWYooacVKh3ldjbn3WbKoKKWMaPGyPpMW22+er79BosrZ7j0UG+AzBUjSj9xrtZQj26L049ekJjIBHAseqlf/8RgHFsabelj/+YZSkAmPW99qlQnbGLAECXnd3c8/ZXq/QkAJdK53MWmnb0XwEjZRd5tGT7Lyp/3LN0VUYN8rp4nfAXsY7HF/T9qF7HXIeni73y0QhcZQKyEJtI4wCEUOwoTCi2FiwTiknMfcEx1TsUuMbSsb+dOjIj1rf4eyL0PL4AfZxicZz3b5Deaot4EhOJjOKANFeakhBxuyg0AQlu4R9L5+j0mEMDdS8oYD1pP1e/JBRYf1XdnaTu5JPZHz/BmNpA3RAGzWcFvgoFDulNLLfmqqRdQDlWMJ/Ue7VmZEYNHmAxuplGjtXdTwTL25Rg7PhbqrWd1h0iGaxJrmrGN/Q9pOnQ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Ew38Yk2Fh01yvjBlSgvmdHu6QAmLurm7gQXxTBKvYoLxUoObZx2Xgo9ReNtr?= =?us-ascii?Q?/XVAfH+YZMDXRyOMaSFH8Y8VaI64x35cBMu33HVrGNKtViE0gieDHYrs9N9y?= =?us-ascii?Q?1tv2ORfnOP8r0j0Rz6BroxDWU+BKLusiFA55gTlQTHDdmH8ZCKgg4udyBY2T?= =?us-ascii?Q?cQySMNpV4NvzkFm0wCEQ2iUGUi+n/2VnLYtQsrZ2JPkjrz9IntHVpUtknvov?= =?us-ascii?Q?iIa6QGLxCUgRtwMB3kmPQMBJhPa7wLuOIkEgOafm6A/l2GwzJVSljedCna1j?= =?us-ascii?Q?pB5vVHz81VCEdYsReC5zfWmSopyI40TevfJIOWqdQcieMIGjnq4dvrx7QHuL?= =?us-ascii?Q?icEX7lS9t7AH0z0GkjyqxjbXHgaimxL0JOBJ5/NAr3YW72Mt6agU+NKPUhl/?= =?us-ascii?Q?HMbIDEQX53jlnJ/EzrzH5pn3dSqOmXZlfiydS+pDjsCUvxa6HVRzf5zy7/8p?= =?us-ascii?Q?Fia/52PSJ689rGZyM70yX4beg3YB+Mc0+lO8eJcosJw5+kJT62nl4/6dVDwi?= =?us-ascii?Q?3KuorjZZYt2NNPGf9RVL7rO0BCMx4fDZaX/zmbc6C1rJwfloAIuJYUo5EK3u?= =?us-ascii?Q?BzCVAyVxAL7ypzNEhmUfqUGbtLksHbu6hi6bu05x0kObpoVUF09tKvpF1BaR?= =?us-ascii?Q?RbkfCB1qemALGzbyASUhYSFwskZoH5clKCCMqvjbAdbtE5S1PKY7NT/ucPih?= =?us-ascii?Q?SOkpI07ZTAg4OBw0zH5fRCncMVqbagMM+k49kyaxwpIvWje70urCQzh1b6hq?= =?us-ascii?Q?QWvcHf1VLEHj5IWzaRZ/d4LK0+0z5sXO7LtxPlHHcwvleM23SXnQTtshI2GZ?= =?us-ascii?Q?MuawA1G1JB/ECyT+XHM4n1zEwJ2vqJI8p407mchr9ztx6/GslXFpLzRFSMB+?= =?us-ascii?Q?m+P7MwXa2gU/wUQSMVouPssE2TlTAYZ49f685sqOm/l5Xx1ZKaiAfUd0+kLO?= =?us-ascii?Q?Wr9A8d1aTXEll4HlfNLMmXWB74trGgeWWNNtniwJ8EKBMEkk1voXwC94mweE?= =?us-ascii?Q?hYhjiyWKOUEdWF3bLb5w3N9xcyoH9m2uT20Y8gEyqsce38E/MBzpPy+xmaab?= =?us-ascii?Q?I5Qmg502bPwP2dSyqOZjxqVBORAwyAMMvTxQBi4SbL+fXNADJzZVQgv80PT/?= =?us-ascii?Q?TyNKstYLTl1p+rbhgZEthmoHcRj6TqRyiWlnkE+OqOmaTzPDI2XGHqWoSQ6r?= =?us-ascii?Q?N4CHRaKUiduJKzsst0+UeVbGwNoRR8bN6i9NL7murJU16BmhllKbdS16HHgk?= =?us-ascii?Q?m7Le2WOhkmv6uGtoaQ6ZzQOhDdyI1IEw6i+Lxd9hqQ3cbdceapdqqjEKwsOI?= =?us-ascii?Q?wTzoKCCrrjXskgQzMhz263Kqprc584yGRL0CH+v3s8Qs7S1hOr7VBAkAw/Ou?= =?us-ascii?Q?6RD/qtI3OJg4yffCM8y4oM2V00IdhYp9uVKdOW5NHk7TF+g4WKBM3nmO0OgA?= =?us-ascii?Q?PdQOEueVaBywQvcnoYg8zNJQ0JdLPrZi5dCbpdP/cbYhHhuUnBu5nC5Pq5JR?= =?us-ascii?Q?wchTF+0nGJ5OvZQFXHkRLBP67w5VXQTgSiUvITUsuHUnHO8qNxZwMWqcZMh5?= =?us-ascii?Q?BJEJOrW8sl8qvHuWe6WI5kWHEeGDMgILPyn93kN+NT6GaRTunrs2fPS5V2tB?= =?us-ascii?Q?8Q=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?u37lmZ0ynFClxHAdrhqbcrbYcI7Yg+/TTocAIB8ZgyRFwGg/yWXsMEEh5lqr?= =?us-ascii?Q?l0Njbjxa/trFX6ehFhlgB4Z1UeEPgCX4rFtN3gR9uB1GCQEJ0mu3C0s8c0j6?= =?us-ascii?Q?ETzs42IqHWOpyOnsbAxHWv+YOB3aD2/CmPDf25h7IEvSKv/ul2TrUZ3JPm7J?= =?us-ascii?Q?YPBmG+Zfjz2PdTA/frwFuFmqRe8kRiKYr1UwUPUaWcai/v2dC6YGrDijWsyW?= =?us-ascii?Q?q+59o4KkLPqcSnQdDhotTJ5i+7jIGZ2NeytMv3LA5V4JOw725tSReMIO8J3X?= =?us-ascii?Q?+rm/+nk5dSycLBzZlyw8BASoZP0uXZ8NPzXrBxAXKhP+3BJnA0VnRBv5NuvA?= =?us-ascii?Q?gr3ktSbePplFxDaEKjpbK7x7ZFNvIvgFE0/cH2Fwq+h3X0JMxjB/7PrOUhAD?= =?us-ascii?Q?gtYN2US7s0Pjno4+qMbZrKHcM+M8tEV4a/IOefUo570n6xYKzUTuvysAu50T?= =?us-ascii?Q?NRyefBdrjZ07wGs8k2KDwCKs52I70j5qQI+NooNqMIuYn3GkphBR+FHIhSjo?= =?us-ascii?Q?ZFD1fngU92LaKT+fC1wdAf78tJJszEhpdd3RJ8FKwDiV1UaoLSz4a260nkvA?= =?us-ascii?Q?JQcSP4zZ4ERqThbSU03z5pIq/SLESarsOC/7EEv97sXlQZYgy65X9QQK9OGw?= =?us-ascii?Q?z7uBEZqMzX32ttw4eJkoSWsC/fjamMfT3PEwZWh60rN/rdTK9v9of56RycGZ?= =?us-ascii?Q?KRZHwpjeMCYm5ulJzFu8zpJBqZSBwk8/ZlXO6uK3pIQNH+rA7vNpLcdoqXQN?= =?us-ascii?Q?486/O7f+aWlthiYCRlw60s3chQ09hdqVcolLjWmoYA8q/5ifbL2sB/l2Guow?= =?us-ascii?Q?qSspEFICT5dKRdta7FU+BljOg6b0xw83KKcOZ9iE/gSGTwJWRwvnNnx/N+sQ?= =?us-ascii?Q?bhS36/yxvTLYNCnS+Mj/oNSelHRE/rdq5Puf6lC7aowPixsABuAXl8FIeoJO?= =?us-ascii?Q?DwJcINZ/rq/ReUnHd67RXuwnktVkrm1/sA99+ZHBvWPwOXSCbLD6KA/I+Apq?= =?us-ascii?Q?9TGOyxcOmJi42Uqar7/ETQcIMr+Crntg/qhN2uqgdvCNADEgrwgL7mb+8XTo?= =?us-ascii?Q?VSCdI5COfdR6T5aLXKhS7EdOnO6pUGSjCpKAlzcXKw3iLQCDZ5LusWPhf7AU?= =?us-ascii?Q?tV6DaIfYxcRJ?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: faa1ee75-cf92-461f-6b21-08dba59e3582 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:12.6238 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: n+8AA02CWG5p98oyE4adDLPTmNWr1BYcj6+itQsiN7REg0kxRt62bHIn8YVjDIIfcDTake+T13/OKoAGNgIZSQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: Ws-EtQvUChEPHTSo6S-b3jXxBZCOCRNT X-Proofpoint-GUID: Ws-EtQvUChEPHTSo6S-b3jXxBZCOCRNT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now that batching of hugetlb vmemmap optimization processing is possible, batch the freeing of vmemmap pages. When freeing vmemmap pages for a hugetlb page, we add them to a list that is freed after the entire batch has been processed. This enhances the ability to return contiguous ranges of memory to the low level allocators. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 56 ++++++++++++++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index d5e6b6c76dce..e390170c0887 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -305,11 +305,14 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned = long addr, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @bulk_pages: list to deposit vmemmap pages to be freed in bulk operatio= ns + * or NULL in non-bulk case; * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, + struct list_head *bulk_pages) { int ret; LIST_HEAD(vmemmap_pages); @@ -372,7 +375,14 @@ static int vmemmap_remap_free(unsigned long start, uns= igned long end, } mmap_read_unlock(&init_mm); =20 - free_vmemmap_page_list(&vmemmap_pages); + /* + * if performing bulk operation, do not free pages here. + * rather add them to the bulk list + */ + if (!bulk_pages) + free_vmemmap_page_list(&vmemmap_pages); + else + list_splice(&vmemmap_pages, bulk_pages); =20 return ret; } @@ -546,17 +556,9 @@ static bool vmemmap_should_optimize(const struct hstat= e *h, const struct page *h return true; } =20 -/** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. - * - * This function only tries to optimize @head's vmemmap pages and does not - * guarantee that the optimization will succeed after it returns. The call= er - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. - */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static void __hugetlb_vmemmap_optimize(const struct hstate *h, + struct page *head, + struct list_head *bulk_pages) { unsigned long vmemmap_start =3D (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; @@ -575,18 +577,42 @@ void hugetlb_vmemmap_optimize(const struct hstate *h,= struct page *head) * to the page which @vmemmap_reuse is mapped to, then free the pages * which the range [@vmemmap_start, @vmemmap_end] is mapped to. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) + if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, bulk_pa= ges)) static_branch_dec(&hugetlb_optimize_vmemmap_key); else SetHPageVmemmapOptimized(head); } =20 +/** + * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be optimized. + * + * This function only tries to optimize @head's vmemmap pages and does not + * guarantee that the optimization will succeed after it returns. The call= er + * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages + * have been optimized. + */ +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +{ + __hugetlb_vmemmap_optimize(h, head, NULL); +} + +void hugetlb_vmemmap_optimize_bulk(const struct hstate *h, struct page *he= ad, + struct list_head *bulk_pages) +{ + __hugetlb_vmemmap_optimize(h, head, bulk_pages); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list) { struct folio *folio; + LIST_HEAD(vmemmap_pages); =20 list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_optimize(h, &folio->page); + hugetlb_vmemmap_optimize_bulk(h, &folio->page, &vmemmap_pages); + + free_vmemmap_page_list(&vmemmap_pages); } =20 static struct ctl_table hugetlb_vmemmap_sysctls[] =3D { --=20 2.41.0 From nobody Thu Dec 18 01:07:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D603EE49AB for ; Fri, 25 Aug 2023 19:06:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230370AbjHYTG0 (ORCPT ); Fri, 25 Aug 2023 15:06:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230488AbjHYTGG (ORCPT ); Fri, 25 Aug 2023 15:06:06 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8399F2690 for ; Fri, 25 Aug 2023 12:06:00 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIESZ0031124; Fri, 25 Aug 2023 19:05:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=0ahRRIn0ZNn8lfsDqjwet4c+B7wLB9LxrxIa3Q88Osc=; b=bWEj0ckdoCz9QYuzTiKhsyrwYkxPiRSZKmSCHTi+WDNv5P08bLhaNoeXFnR8oGQPVGjC Th43vBH5n+6Cmej3jr2Em274tWau3cMeFolIULCUo7V/EbrIH/lY/GAKIEzcprJNRU7B 0QhOkS1PUxwsCQ+s2ECIYLNvbiCDR0bjpCirh0eBo9QPpBBTQuVGCSghoNDWQNO5lVmq y2BT9DEFKRxeyMom9I19z7ecyQPPZd5I3aiUMV7h09QbBfvjMuVpb//+O/yo0+1gEKwP VBum4OnalPNBmB73AMpJdzFfu39gP458rJxkQpunG8AsOctY7gdXpRmwrpXiyd5QkVHj 9Q== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0e1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:19 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PINm4A036073; Fri, 25 Aug 2023 19:05:18 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yxqsmg-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jcn6NGYkcVYZpeKZjTGReRDxBUoRAXWqkxWZMQf6EmwXrs53KjnINPbuiuSJUH8AEZcEjpjbDnN5ICRu7OVei6MLoBs1bSjtWPdHKAZpcugfXIMw/tJ1I5GMKjA33YjRrOR5KPV+0XZHgWH94zlKuLHCWRBxlZogF3L/wv175wZo+XYR+I1iTsNVpj9nTFRPis1YYkbCLyP/HGB+aCSVY42fP0gk1KjpLRp69VCK6gRaU+LG8w0rzfHkW5U6p+HeeYTiiIxbGtvj5opSt1Y3yd+S0QJW9isqJUQ0EjK1MLhk2jNINg9XkD0KC24dYqUm2lo+vX26wAi0+I3cl0eXpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0ahRRIn0ZNn8lfsDqjwet4c+B7wLB9LxrxIa3Q88Osc=; b=gwKLHoQTXle6UGGbWPTFAN1uzJECRXyO9A54NsSPw9kk77ZUJaai2wEhXmmnJB35YwLRhbitFofkniTcERBmuyPUy/NQTrHrXzvIE8Wub0im9DtKfDuDChfpa01qqiu9OHaAb2celMPAEjnUeRT5/U3pQqFSi9erani0iYf0y1EGKNyHTS0kpT5avt9MLPuSip4pReHBZiPG75ww5eleExjBn+5ZTBy4WVoPFGoj2WKWaJEmlWIjW/yP3Zyn1UqZMQ09yMHNgX0AA4QLilFUzuScn11W//6geuKuWeE8xrun1xC9s79bPbJskLSsPADQqH37j3gY2cu9/iQkkgjzxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0ahRRIn0ZNn8lfsDqjwet4c+B7wLB9LxrxIa3Q88Osc=; b=Ryzbc2cmjPfEqj1AbgMeVVvNOga7q7UjZerj0WKU+0OBq0Klr6U8eW9f2DRqQbAr+D6APIKBdcj5HPho7muw2yNMH6/7awft6U5Gi5B9uQ3GLRWZZ7lLQR9MCnuTveiSVHosC6cgkkw0wNg3KXu1lys7xYW9twU4wjv8bj7I2Sw= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:16 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:16 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 09/12] hugetlb_vmemmap: Optimistically set Optimized flag Date: Fri, 25 Aug 2023 12:04:29 -0700 Message-ID: <20230825190436.55045-10-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR03CA0100.namprd03.prod.outlook.com (2603:10b6:303:b7::15) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: abedd79b-f5f6-4695-4fc5-08dba59e3785 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MAh1hHtCXEfiHswV0+4QMFeu7TMqA/19oyNn6ihMXY8ghVo8ZirDBie4BJ2ywESa4c1KaVct4cog5qWZ2sHtBLmhGD56cfC7At01ATEFtCWLHYWvINeVHIVLVe4T5HBeeIR7qRo6XgIUPHUJ2EnrGmTl4aTRljVlJLNOUTz0byfBEnF3gBJkuEkiv0D+73GbXJaZFXTCXtD5HeAlbIKIZiWYu5LtmRXMdC4BvV56yhHHxTHTXvnf6IIPXrHrKbSRZ81NA+xj2wjLeoGaCv/ovmVeCTRRAKYlyrHV1LikT9wbAR56O15o9vUhYwgHtBHXtCxB34jghBsz0I1ayhXcSbZbACiWUIUUcXFgrews5EATKbHSMAEqtMsf0x/TWtJQfdlx40YDPNxBxKzhWyOQbMORb6/RQQG9rjY0Xn8EXyqz/5YJ1BJOU8r+oXyH38WWJwTVQHj2xZ3c9mtGNZH5j+iHF7pNuix+/b661w26m5va1tUVIBYLb6s1k+L++T9ejycbrM2nOE5uujIoMxkvQuDhVxBkKZTORCLzYPkcJsA05VMb6TOZDEjonTxxasPL X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?keS5tgsj+yVcQgguZwbppEJXgXrGrv7II7O3LKR/Ul7jgok9W6lAvPshLHmN?= =?us-ascii?Q?HVN1CgeUrlN62LxhoLJpFH1qXaRDbV7owVxXnjpsi3sNNjP5yF3LS/D3TUux?= =?us-ascii?Q?UmoXcnjRb82Oqu3tQvIFhkczhdAqesIeqJAxaa9aibzJRtcL5CTapOo5ipWh?= =?us-ascii?Q?jVjqxJMWYD6ukkjOLu11hAicWB331q3Xb6r3eVat6u4ToxN6gIhBs2aXM48w?= =?us-ascii?Q?VyHPKgdhDt+FUbdUOkjuQ+TLccA5QzQPl6/eYhnsQvTHu0U9ikxF15P7NxyY?= =?us-ascii?Q?u+2jOvVjI07/kAdEuOSjxJF77dsjfpH+gZmcNpz5LdkANqfjvWCVHFo+bQmy?= =?us-ascii?Q?VD2gIU8Cg6XY5kFGAqAUMpfJEOaNFp2CBTMOPvVsYKqeXt1o+ZPzcz4j+z3k?= =?us-ascii?Q?fDHS6LWNx2IEMEw9K7Vv2U4rvGQrsqZL1rmBbOq5YQyeMk9x30QZ16T9itDb?= =?us-ascii?Q?geoDbeMhGSMF7GXm+5D16MuMzRUkGo0pT9DCJQkOxKO2CkTLt1pKGujihzxb?= =?us-ascii?Q?2eY3yOdQczvoXROR14UgGon4hVnr1HHyFNzXZWBpYwibNKvSf6AqyfBN/jxW?= =?us-ascii?Q?VLtCqQwC5ofM/hced/m/WAsYAEHqV/DqeP34tXYg6SIPj5ASuQg+xRdunrGf?= =?us-ascii?Q?0Pvw+XGfPOJYi5XKQNLjQeU6+5nrV5BrMyGMd5Fs8OlErbO/iuRiO6GmWR+t?= =?us-ascii?Q?M6blwPq1IG7bSFN78tZQiUdHBiPkgxfkIVMoUIFqkwrAmKwEKKc4TiJAcXVN?= =?us-ascii?Q?ynf2KmJU2H79s95/9pweAA3StJvVq8HEfHw3Z9O4pdH4YQregZESoId7ccz2?= =?us-ascii?Q?/UjW2ZYy9HrrpFRd8EtcMZXtpb5OxKWxLKgn5Nq6+HcTMZB0VvwjjC5jfJ9g?= =?us-ascii?Q?l+5avBTc+LGJY5Yum1GXZgMOHrk5t5Oqa7CLjpjLRW0NK1Y2jSgs3Wr8PD1c?= =?us-ascii?Q?bniqeOBzqvXjlqDGo8Xe377K10BWPgdkTwptcb1+yVQ9Aw0CXai1F2w/850i?= =?us-ascii?Q?NopxTlvB7CnPeZO/jZta9Ns+s2G+0ves3PtF3C8FuHofK3VgSrjs3LVRUojZ?= =?us-ascii?Q?UFP6tz3XdbpknNlKgvRkTkREPD6DsGc5aiWWcti0dGR+i8O3zseVC1L02Il7?= =?us-ascii?Q?91KOIsgfuhli3lxZpX1Qw0L6N+UyX8OD+NgFoRmXzINPxzx7ibgov4m9DaOI?= =?us-ascii?Q?ojbOPt77a42V/1oTPfE4rvtMURUPdeDxMBTTqgGcbgn9vFuoqoV5HTDyYiJW?= =?us-ascii?Q?h3SsKIRcG9eBiAq5AE25Ye52Lsm/3CJp3xSYJXa78nI3uk1koQ3tQnsLk4bM?= =?us-ascii?Q?0InDBDEjIiIDb5WZxebnFpCcIvsLrTfmsLpBABMGOpJSax8dX2m8us5kv4Pu?= =?us-ascii?Q?MxHkoNjXkG77OP8t5whuOa0HgDrCSteLkX/A8CNgz7GO6VIIHXLFOmaLnGur?= =?us-ascii?Q?o6VUePHbaY5/+hz8nu51xdd90AAgOmNdIJuUxTPFvYyLDxdaAoZ2kTKvGR34?= =?us-ascii?Q?785ZOZX83nOzYK7cZ9E/OBzLy4J8CH916onCskK1hUlYrgdESBRd31tPBXTy?= =?us-ascii?Q?UnTKV8m/x7s9ZAdbeZll5tqnzuXgLd6AUCevT6+EopYqUOdLyZlhlocC15IU?= =?us-ascii?Q?UA=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?EQogqAEtPqEPm1I6NwYwTlnZY/r0cinLfqoD1Ul2d5RDByWzbPJ2z7SyjhwD?= =?us-ascii?Q?JqGDQVBczusYTbWXU/DU8DxCV+fOPk848vyu+B+kZpRwUfJ4wz8Sa7Wn2rMy?= =?us-ascii?Q?pJEf1PRsYjmt6djK2n1QLoDJ1opfRNes0cfmJ/cFZK1UVlFqcWbQ8rK8wivg?= =?us-ascii?Q?cwSflqg5TeCNjwofFz/UNU8vLh6OHqVPA3a+sB/YnU1nlhXSdxT8NB/Mra2v?= =?us-ascii?Q?oZ50lNrHk2PHALTkNEwxNINtpqHPWEninTUAOSHiEQxQfMI/RPFsNUlMMwy0?= =?us-ascii?Q?rM1rM76uPEhEuItWGVjOidbVdsrf609dh0sfgE15/EeUvDHy7Od7IStOC8vv?= =?us-ascii?Q?B7GEFvBBxaoRkgS7iWoaTuUx9AqkQBHCGRuRcLq8rd39zDX+K+suAA0n3izV?= =?us-ascii?Q?FS3SzeRpxWs3kndSas7HlVCl54RkpWV2Ysu/+YhpXGtBMOJtdGKDae5NuGcn?= =?us-ascii?Q?RFPYzxJRi2X71UZf1nU8d4IonkSSJINcMG1775Z/xWNEmqNVwR7AI/91sBMz?= =?us-ascii?Q?25DNjDTQlO7sdoWWDstsC03cUEGrAxPxgABQkZLCbylVqROvoPo3mUmOkehO?= =?us-ascii?Q?omO5tborDX6i3x5lbBMc5//IV0+HdC+0j8WYfCn2F8nJKcNRBimBeDWptSIL?= =?us-ascii?Q?3nzm+RBkERfiPGtl5Rq4Q9V3pxmvPSqo4WI5RYp5UtUhrJdoSWIQtMM+hje5?= =?us-ascii?Q?3btVCq6dmOMjivOL7Vl8dOc7QI5DkXX2EtdORO9c7pD8JjWJyP3C1taFoGPP?= =?us-ascii?Q?fKpyzCjj7BcWYw0J2sdoE8wAJBDwG1OEnbXaSkg7a9uP9nMTx5aQjTEJoFBS?= =?us-ascii?Q?WH/xDKufFsAQub5lm7WWxT3Q0tuuL1V+jQThoc19hvZ9dx8Rb1o14nUdwNdL?= =?us-ascii?Q?EQFpzu8TDYWeGcv5beY4f9viAvd9TOZTe9zDe50zUxau89J62TVwdKdLSkzx?= =?us-ascii?Q?7+dm9WESFaKy1jvVOYLEbEsPDywmof2gIQj1YJ7k+mpyhwNhlOZiiw/0aAsQ?= =?us-ascii?Q?EnPlfo2u6oWjeXBhORQH5bFjoSNVq0UbLixfHRGMou8X79zQOMFqaJjR9jDh?= =?us-ascii?Q?ay45tyYIwjH93pzgtLld/uoa7r+/nGbrbGkdWL0vBRsr0Ap0VPYHpbGSWVyA?= =?us-ascii?Q?osTOtBu5fW8o?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: abedd79b-f5f6-4695-4fc5-08dba59e3785 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:15.9920 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yiVlx8AVcL/crN6CQLHEuuXU578cJnz5XmyNZCXeAtewUYg0erIie81AWy0fSOL2jDY0T3ZDNGaWSV0hYjus4g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: _6aczlNKQatDKqsWD-YfwE6U4edUiDxf X-Proofpoint-GUID: _6aczlNKQatDKqsWD-YfwE6U4edUiDxf Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" At the beginning of hugetlb_vmemmap_optimize, optimistically set the HPageVmemmapOptimized flag in the head page. Clear the flag if the operation fails. No change in behavior. However, this will become important in subsequent patches where we batch delay TLB flushing. We need to make sure the content in the old and new vmemmap pages are the same. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index e390170c0887..500a118915ff 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -566,7 +566,9 @@ static void __hugetlb_vmemmap_optimize(const struct hst= ate *h, if (!vmemmap_should_optimize(h, head)) return; =20 + /* Optimistically assume success */ static_branch_inc(&hugetlb_optimize_vmemmap_key); + SetHPageVmemmapOptimized(head); =20 vmemmap_end =3D vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse =3D vmemmap_start; @@ -577,10 +579,10 @@ static void __hugetlb_vmemmap_optimize(const struct h= state *h, * to the page which @vmemmap_reuse is mapped to, then free the pages * which the range [@vmemmap_start, @vmemmap_end] is mapped to. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, bulk_pa= ges)) + if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, bulk_pa= ges)) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - SetHPageVmemmapOptimized(head); + ClearHPageVmemmapOptimized(head); + } } =20 /** --=20 2.41.0 From nobody Thu Dec 18 01:07:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 646E7EE49B3 for ; Fri, 25 Aug 2023 19:06:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230410AbjHYTG1 (ORCPT ); Fri, 25 Aug 2023 15:06:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230498AbjHYTGG (ORCPT ); Fri, 25 Aug 2023 15:06:06 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB3222139 for ; Fri, 25 Aug 2023 12:05:59 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEMlu020647; Fri, 25 Aug 2023 19:05:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=1D7XqVWhwIqkbRqvMvx7UBrASxcP9XTXO88KGrxsyNw=; b=XGZnny1mk5Tp15Iqn/7NPm/bRsrvoj4w77pjtk8OTaNoExvFo8z6QgX77mKB96+PEJBX LG0zMeEwQTx92cGeDv/zn5EgswruNEn/lVUYhYsrcSRHOnHI+kIZfTl1H38tsYZEABEa jcO/Pbf47Phjr/aFWKjU1BXzSn/4jIxEY3qc2F6Z6FtOye6c3gYMkezHMgZSt56xXJ5h FIThXNkX6WkuM0nrLEXgEZSrBC2qXnANgJBdV9XmfGzvSEI/qF3ZSNDa67mff69krNF2 Ytlp8UZ965QKnH3mJn618QvWcxUxzua9RJ00qASMyrtw7GWvZXcIVvifn9J5l9DyESfi 0Q== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yu779r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:23 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIH5Qe000942; Fri, 25 Aug 2023 19:05:22 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2103.outbound.protection.outlook.com [104.47.55.103]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yv79m6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Cp46wsg6ySSmD8YL63Gb8ywxwTJGrLF9+PtVB8V4KNNT6TtPVewRflutN5DVyCZW+q28Mb5ucZu5SrGQAnBxXttoDXl4XmBQJWPPzAV1SCWrcJkL/BMc5owWlOhGFaWezM1l1ekKLtHx1HPe//neeF3yEEO0rZ441PPt/IqoQDhzkfxtQ8PmMTJBaL5KU+S2lzE5I2HCBIfl6gyT3kldYZvG2U0ukfbdH1Xw7ccn85vDJugyJ7ElxDZn8Sh1mYs3GonbKUVQQI/7sR8he8s8gwC4xgJ48eEwb6Ulne1Tm1odvHjB4jXaufdJrCHugxg/a0GyTaj3UBKWEi7t0f/bWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1D7XqVWhwIqkbRqvMvx7UBrASxcP9XTXO88KGrxsyNw=; b=RfL6tOD5anylMFpNV7vG0zezUTUxLHBHtfkNO7NAFwgDy3x3m2mznNiiD37Nu428m1LGEar/nso4wt74tZ8FPicRbZP51zw5080JqdT6NtogdYVaTSdRQpxrL366FisJg+NfARf/ToFXpDezAXyZQMpo0L8D6jLVw9Vi6ZOMMc5NB2brgru2X9QJSSTcVfrPDJ897Xbp1hITMWbZjd4CWpY0LixfX6Lo9C8YcOlVa2+/yKWsBo2OoEngavumQ7s+daated3YUcxDVba7KibsIl8UzsOcx1Ftrwjp/ymX3A6VBT74gcwlaU3/FZrJYL23fpA1VcmBeIkbV4bMNvSxEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1D7XqVWhwIqkbRqvMvx7UBrASxcP9XTXO88KGrxsyNw=; b=fN13S1eMfLh5RgTzNRCpLLiYU4avKxx22pJjV6GsnorZx4jQxgvBsbHhEs5ljBKKtWI4qijuctpPmdd00kbw7DzFPRA3FqLDvgiASVcIu7IfA8mkg45PGvEidsHA/MbIbm6mz5af+05DFByq6Psz5vw/WWrZqIIQ6R36rdlYhHc= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:19 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:19 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 10/12] hugetlb: batch PMD split for bulk vmemmap dedup Date: Fri, 25 Aug 2023 12:04:30 -0700 Message-ID: <20230825190436.55045-11-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0226.namprd04.prod.outlook.com (2603:10b6:303:87::21) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: e7cc4ce3-bc06-4cb6-c1d6-08dba59e3972 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TH/mtV/sqimst11xZ31aIsahAJp+tuvMJLVDbkjRPL0pQ5JDV0GvQp6crCngSlJU43k+zJBqKukAxIMuk05Y/0gOzUrr9DlYv5LM9X6Ep3ILkikSshc6Ng1FtVdTFLAtaqvQ4/ND95dnThTnjyGMwkrdnNaFXonY+MmlFUPhs20nkBKF14N6k/O/jUmNIBtc5AxS3GKPSKGj/qwi3J6N6pjr4CuWF39m0nGS6BMZ25jef4Katzebih82o/mxDTPtjHAQOePbjmcBA+WeiK6mZOQUy2OdH6YxphjaE0KQ+8YkI6luexdNzShGBhx6KjmHo4UiCGdtcmQtuofN/2qROC5kDEKKV0sIxlewNIGUyp8JENypVmemKwEhAT18EcnaxWleu6UI+dZlQnNfyaQJx8LGT93NmOUL0uAYlQBqwWd/eapVqygOGUcQ9WjO/Vw5GKaGABS12EnRnPUB8TqgHXRf54AJeF9x2GMBsQ5ERpQ/Tnva2FNQeXx5OfX+n+QJ+INVbvUb9/Mo1jQH/ligPr4VA7xUDdVzrB4xxgHfTHBr9TYWO16Pa+Rq+plI3j4I X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2qGuJyf9zoUbjmBndJ8sXknO5gcBXuP/SyL4yqQLkSDL3zXSEik7+We06wWc?= =?us-ascii?Q?pTwRGbXYL4t0weeTLEsJrhs7LfhJTZkFjT6achXCMml8FZIs0SXdli5gZU3x?= =?us-ascii?Q?rJqHsmpGfpFGLLOwctOM/xDliS323CmEOku5aEBCbgtB1lJKRsdu9sG0jy/I?= =?us-ascii?Q?yPg4Rt4mfOYff0QLq4I4naYhdQQ584xoadEU5Sl2/siXSyZ0YqwBHAo2Lefm?= =?us-ascii?Q?Bhg2zGbeiH4ZUIJA5apUKmRQp2MpO15XE0QrxK84nSmqxCxcTGWwUQT4d5H+?= =?us-ascii?Q?VHY1fiY6xhll182NzWfFmbMku0sTQe8HLmRY653RuOe6fhkczxD/19qaQTXJ?= =?us-ascii?Q?UFRvRuZcYDojgW6V2ZkkO5fv5I3kywqTu31UbEsRXqh6/jDj1UA7/9WY+qDv?= =?us-ascii?Q?buu0b68lBVpDMtqm/lYOdB9mwN4/XbioQYdMJHpRBqLp334qFKARQQaZj0Ve?= =?us-ascii?Q?2UyvOArEveJDKRe0yqTMCQ555y8zsfENrOtPKPw4yP2q5jsUXqtgI5bopWIi?= =?us-ascii?Q?FUt221zgZn4X3DaYKXqkM7LYdacKDtEFLck3VdNxLCxwWu0huiCEG/AobZ4s?= =?us-ascii?Q?6JU5oATsmV7fAMT40WpzTK0IWJ/9i6f9TuwUYYRQIwaObq6PMuD9Ohb1cv2Y?= =?us-ascii?Q?ypCgt/kcL03W/nAOyoekn8TKNDjbKWWuLItwMgOIsdnb8OczJyZ2/n64DtWT?= =?us-ascii?Q?rSpov/BB5hp+X0SBmASF865ct/xGXXHoCzZzcDK6lP5JzbrnQbZRbpRsziuP?= =?us-ascii?Q?zuB9NR6Y8C7D5strDPZs2VhE+2rpvQjhFi0oxe3a3IR/wal2Lzca5sxNMIc+?= =?us-ascii?Q?8DN+1HUzCWT1Zj7Vqq6dnOz0voohZ/PLbcA4hTwu0L0/ZYgyKIUsFCdn2OfM?= =?us-ascii?Q?rzfDE59PPZO5x0HVqriOvm6v5R4WsWMIRDE+sPiiZJxwrLTvPdgr8bVC93mz?= =?us-ascii?Q?BPmzukFB5a6s7L1n68ZOryPmxkJWBeA7Op/8LOCVvew+3LN7EiRa2kbyw7W5?= =?us-ascii?Q?LJA6sGZb+A9Wa/NjX9O+bS58qPsEqYGs7cg2ug3MQjrqdZSLYZbheEhvV1SZ?= =?us-ascii?Q?10fuW3lxNGRKabUmsY+sgXnYkjfWS6fWlOnBdQ3VIdtLBGQ4otPajOvd7nvn?= =?us-ascii?Q?Q8bJYdzAsWAelYlnqlwywNMyo7kKbxrbaQj9AMzu8TRkBrP3jJCfEW1hE+bF?= =?us-ascii?Q?8kx+valFj+107MV/YIsl5vrchITOtJ/eypnEhWyO+JUBOm8HAWqLug3zeBz8?= =?us-ascii?Q?TfXBKD+eu2X3hDGMnuOGXR0RQJgM2UYCn+Fyd8hvghCvWtBG0V5Xp68sEZ1u?= =?us-ascii?Q?tOxGVQiTyJpayWKBvZDNtuGFK2Pc26pzbHr8Mk9htIyji+mlGXXlvwBqaSmH?= =?us-ascii?Q?qoT/QFmdDjnTsdHK5MZQoQ4ndTR7tnmEY4ve66q4Ode8ETO5edy8BulUiM6Y?= =?us-ascii?Q?/re44gx5EYi0GgDWe2xn3zZWI/Nd9YQG5LEmfYRqvKMU0QGqYcbWILvqDhMn?= =?us-ascii?Q?gIWUzvrBzlN0ma9DlMquHu8sF6kP2RFqC7MU+ObWK5lHgUTUCHKhZmYK2Fj3?= =?us-ascii?Q?7fCD7b3cA/fUGzgWfIQNIF8eoPOUq3hhMCHzlJjckEHQm40kPASqpk8xKM2x?= =?us-ascii?Q?xQ=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?5Na0018J4TK22mjBFP1Awvmq1mHd8jb9g5N0RX+OzRPqVLhba/COFv4Eqd52?= =?us-ascii?Q?fw7Zqa05B5IoV4hcnIKjumNvTjarLbBqM1OSkCEUHtmNdXx5dv7F2/w8T2AK?= =?us-ascii?Q?doIGXULOW3qD/TLfkfZdS1el05GJDoMwLDBqUkElpaSkKj0XOZsaf2fVSFDz?= =?us-ascii?Q?UJowAquvRvGRHFeCgVGJAjlmZSvJEZca8mFzD8OCCaJQCbB4jqdlyaWNHvmq?= =?us-ascii?Q?nn8fLsytev29Y2Arr6yPSbOd6qdWKFuZ8Q1hvZ+YOxd9Ad32WacTGO1D3ygo?= =?us-ascii?Q?T4ozSlqSjOWZAyefFbBT8ILf4L+RShhWtcbBvYip0rtPEUKO6mu56qpWioJt?= =?us-ascii?Q?xrssFj1lhPDkBCxT/B6S0EwmsWx6PrV79k4Vi21A6Oqfq8xyZThqKzD2a4ps?= =?us-ascii?Q?l60G/zRA7smYdfNW/Y0v9IMIof5pBE7t7fI+IFr9Vk5G+/C9Te1uvbaDjXWb?= =?us-ascii?Q?JxoyqJcz0agtAYGhLn+4/gFLW3aa+nB6wbIMt+thHAKzYqvIUdOClXDi4bx0?= =?us-ascii?Q?CFhjYc68RiqinU4HXXbw3XJvuqGr0oI/GFUDsWh1HgbkxRkkmVNb3AsPJqcM?= =?us-ascii?Q?9n3QKldUdl1DaEpUczFLfBZZYvLg4MzqTDSkKIsF48M488ylQsrCCN8IPS2N?= =?us-ascii?Q?kJ60QwOQVYQeQBiwfJLtYvvPTTI5Bc9r6VWUuSB8As3oMIjQhTdBi/hVfU/7?= =?us-ascii?Q?/piR0w4cNh5gPam8n1zLRRHHr0EzLx7jgA7Fws7E80ktOqiTneLmZJF14y3A?= =?us-ascii?Q?GTLxF6I+MNsVSnT9zMGoHPYjQzN9msq7Eei5Mrz67FOpnO8HseuJ/oDwhK0D?= =?us-ascii?Q?a5u2dPdI341Mfx3kXA+kzQwZJIvS+9ufNQXYZLxYqBghfzQFUfwwBI2rErGU?= =?us-ascii?Q?SPe7oze8q81bFJnisO/lefo31414hKQzLQoQ5qBMRpU2k+AfYJxR9kJlte5r?= =?us-ascii?Q?oBP3h2V/PK/AIec1swaEZb2OBf7nHLy3p2BT8ebY/feIHSCs5Q4cbAQB8zdF?= =?us-ascii?Q?qMeoC3OWY/FzcyyEucJEXT/ELl6ZUvT1pwUD6AD0wK3spYHU8uyVLI998EVH?= =?us-ascii?Q?bYUzIsGBEcFDxGGH7hTPKfym3+hO0ZUX/z1FWkwYzXiCnwmFlWoRuloOI9S2?= =?us-ascii?Q?OZgqNUcvXVi3?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7cc4ce3-bc06-4cb6-c1d6-08dba59e3972 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:19.5089 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: e4NhHKJIv27o2YjGCnDkACiRmC91W8gezH2F/jZ0WqLFSi+QvNizHYenh3TzGddgjbyT3jDTlzjtrKmemz6eaA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 bulkscore=0 malwarescore=0 phishscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: H-ODefkFRKhDbLIp3WcUJaGTuSa0E1dP X-Proofpoint-ORIG-GUID: H-ODefkFRKhDbLIp3WcUJaGTuSa0E1dP Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. This brings down from 14.2secs into 7.9secs a 1T hugetlb allocation. Rebased by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 94 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 90 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 500a118915ff..904a64fe5669 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -26,6 +26,7 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags used to modify behavior in bulk operations */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -34,9 +35,11 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; +#define VMEMMAP_REMAP_ONLY_SPLIT BIT(0) + unsigned long flags; }; =20 -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool bu= lk) { pmd_t __pmd; int i; @@ -79,7 +82,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned lo= ng start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (!bulk) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -119,18 +123,28 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned lon= g addr, unsigned long end, struct vmemmap_remap_walk *walk) { + bool bulk; pmd_t *pmd; unsigned long next; =20 + bulk =3D walk->flags & VMEMMAP_REMAP_ONLY_SPLIT; pmd =3D pmd_offset(pud, addr); do { int ret; =20 - ret =3D split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret =3D split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, bulk); if (ret) return ret; =20 next =3D pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (bulk) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr =3D next, addr !=3D end); =20 @@ -197,7 +211,8 @@ static int vmemmap_remap_range(unsigned long start, uns= igned long end, return ret; } while (pgd++, addr =3D next, addr !=3D end); =20 - flush_tlb_kernel_range(start, end); + if (!(walk->flags & VMEMMAP_REMAP_ONLY_SPLIT)) + flush_tlb_kernel_range(start, end); =20 return 0; } @@ -296,6 +311,48 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned l= ong addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } =20 +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, = @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we = want + * to remap. + * @end: end address of the vmemmap virtual address range that we wa= nt to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + LIST_HEAD(vmemmap_pages); + struct vmemmap_remap_walk walk =3D { + .flags =3D VMEMMAP_REMAP_ONLY_SPLIT, + }; + + /* + * In order to make remapping routine most efficient for the huge pages, + * the routine of vmemmap page table walking has the following rules + * (see more details from the vmemmap_pte_range()): + * + * - The range [@start, @end) and the range [@reuse, @reuse + PAGE_SIZE) + * should be continuous. + * - The @reuse address is part of the range [@reuse, @end) that we are + * walking which is passed to vmemmap_remap_range(). + * - The @reuse address is the first in the complete range. + * + * So we need to make sure that @start and @reuse meet the above rules. + */ + BUG_ON(start - reuse !=3D PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret =3D vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @= end) * to the page which @reuse is mapped to, then free vmemmap @@ -320,6 +377,7 @@ static int vmemmap_remap_free(unsigned long start, unsi= gned long end, .remap_pte =3D vmemmap_remap_pte, .reuse_addr =3D reuse, .vmemmap_pages =3D &vmemmap_pages, + .flags =3D 0, }; int nid =3D page_to_nid((struct page *)start); gfp_t gfp_mask =3D GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | @@ -606,11 +664,39 @@ void hugetlb_vmemmap_optimize_bulk(const struct hstat= e *h, struct page *head, __hugetlb_vmemmap_optimize(h, head, bulk_pages); } =20 +void hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start =3D (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return; + + static_branch_inc(&hugetlb_optimize_vmemmap_key); + + vmemmap_end =3D vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse =3D vmemmap_start; + vmemmap_start +=3D HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) + * to the page which @vmemmap_reuse is mapped to, then free the pages + * which the range [@vmemmap_start, @vmemmap_end] is mapped to. + */ + if (vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse)) + static_branch_dec(&hugetlb_optimize_vmemmap_key); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); =20 + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_split(h, &folio->page); + + flush_tlb_kernel_range(0, TLB_FLUSH_ALL); + list_for_each_entry(folio, folio_list, lru) hugetlb_vmemmap_optimize_bulk(h, &folio->page, &vmemmap_pages); =20 --=20 2.41.0 From nobody Thu Dec 18 01:07:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8310EE49B7 for ; Fri, 25 Aug 2023 19:06:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230487AbjHYTGd (ORCPT ); Fri, 25 Aug 2023 15:06:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231172AbjHYTGP (ORCPT ); Fri, 25 Aug 2023 15:06:15 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE0902127 for ; Fri, 25 Aug 2023 12:06:12 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIERgZ031110; Fri, 25 Aug 2023 19:05:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=xPMxkRieDfK6Y92JL1GY6hu7Bt94+EbF/fBFQYFUe/8=; b=Ta2/LmhI3sn4f9S6TyPqeAD5fEekhWKQKEl6FmJs/iYJoPJBVK7j1OtsjOKIDU4KSpiw bOLVEc7TzNTOd3p3cUmyq9m5Ct5hHLWwQuB1cUJuqMcePCqIAasKmXYQiZ1jmPcPJFzE iSsWNXWZ0rMIOxZpOTjmItnlb6qx9AU9xH4kUZ668wGeRLUViC3KCHho+AN9fHp+RyYi mM+O9E1CJocPtJJWntxNSCHrXLUDOOCjtpyblcHFhBpYKRypcJgW4tgWDe46x3WcczWl Y5Yzs/35QGGoFn3r/4gEP09JudIxo36CEBd/7pLbj13vMN/81MWfGf6ecjhw5GNtMzuI oA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0e8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:26 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PHPJ4x036043; Fri, 25 Aug 2023 19:05:25 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yxqsqs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FpWvEvbLYHXqObCK+tL2P6G+Ts4HUTvWOv0XGyahYl3/w78al/+YazILl3vn5pBCg3ZYDor88Dr0qlaeSXIVAMM+T9OvUdcc3J+Z8Y+q9MAOxvjGjFjeFOZisOQP+MLIGA/GIcFUiDj6dZLuY56tShtwXw5DyPiVc9+D8VWaR/xcX25Ca+PJMOtPHDhP6sMwKxs+j6rPDdm6EozrarrIWDn2qBOOz5Z8oIYGXWS16/lwVgP9ZRalkDvpxpzMhKiX3g34jy5bWUhsnW7LzZ9Yb4ppS8Yr/bdIOlZDa5Ggk7BaIcMLyg41enACt8W1CCVXHGA7ev4xkg88000ZWnZepg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xPMxkRieDfK6Y92JL1GY6hu7Bt94+EbF/fBFQYFUe/8=; b=Xk+UzjkhgGWBGV6lkx1DniVwrpBAw/wZ/rqYVsNs9EzhhI155Z5t00Oma1M+hWm3tOY2fsZIbN9g/DTJVPdJsrAMwywCKSdyb5zy8VO5Pj72cREqzZS7ezE9/7+CiJ1gkswFNvscBzM3ovO6jTsksWOZVyr7gQMiPDIfzi3Dt2aUgSBCQKGMCjrlv0fyNd/5lNGsLZ4jrX9uQUX7uDXR3uwO4HhK4H05dH8Cgh4Ho/MoZh1gLE5n0O7E7P2C0+HsH8yfkNj4xCmXBKVdjU2Dq+cYCYY/LPNU+83pkHJfS1OGWoiknzEDRQTAoQz//SV5Jb8A4m8GSkOAzPYtSk3F8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xPMxkRieDfK6Y92JL1GY6hu7Bt94+EbF/fBFQYFUe/8=; b=RUgChCt7lPSGEwF3bSt2CCms9fWEcMVOynloujqyct8TdngGhvXZ+GCfZaQGCLT9MK73qdEGNGVZd1boPkwyDYSfptkhS2eOFg6aMcrz20wqsdln7hBglSUZXUdMdJPv0DWyLcxDoUKxGUAOXICLG3OTgMDRnYiWGVuGFKB93dA= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:22 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:22 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 11/12] hugetlb: batch TLB flushes when freeing vmemmap Date: Fri, 25 Aug 2023 12:04:31 -0700 Message-ID: <20230825190436.55045-12-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0212.namprd04.prod.outlook.com (2603:10b6:303:87::7) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: ece5d3c1-753a-4f34-2ddb-08dba59e3b7f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WmUYLsQACR4xEZw2u+vRfLmnQobbCY9MVl4cusmasSibJ3YqlToRBBzNpvY8aVLmdvwTXyxfUOu0yufC0ASC0xCXwgOed6AFGjWgPWWDlv3hvLcB2orS8361MUMyn9d2+y0gEuLmJHb7Dt6rz/21nH5CanQguWJDIsHn9sVae99cwUt+viP3nr746rG3dtrzdVi5d+rM6akEMHKtgANI5003ZXq9+bqXmubQrsIgeFW0hBlNthyQBuOkN2OvHT+u11TZTIhI4Zk5olD9OBHqNgld98/GP9lQIAMzTUS2OQ9+aKBjjtdaqY6QoD8q9Hekv2W3JKcXknjCj2Wxo1B2jZ3AM4fd6w+QV8K1BL78bShBqcGo4UaWXuGao556J8VK2uQVGvLuc0FX5xlOQ+5BVnEQp5QgcjPTfHiiCVrROjUNZb8eWqs7t1WWHZWLPvfdBIKX5lIvFIaPubM3e1OGjmWt1hTOpU/mZvNVxLByISwXKB0W7XdoeVVh2nbCVcILVEV/361ANfHuwfkCZZ7gbp4NhN3OBKhYDRziz17AAwMwJLw5ZjaN+1w7aLEuyfZC X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?niBV8kWv0/DMoVpb6An6P+LVzGdCMb33+emMW1PmjKQZtEEj1wtaI1LfTN/E?= =?us-ascii?Q?dleBnJhICX746oHocnmcyCXTWULUJk9sspItLG///47soMpOwD8G1wlaAoaj?= =?us-ascii?Q?kyzWsetB9aE5/3mDXtY4bp73OdAfnNHksKx9n8yXep6x59AgjZo0Dbu7MTOV?= =?us-ascii?Q?CalylJybIeMzJrJ3QANKUxv7gRByf+EQ2bwn0USxCbcxqQfmVaGChFiyGrjE?= =?us-ascii?Q?gcfTW/l8v5cDxHhCBb4VAHBjA9YsyijfVmXWyMtbKVrRebo07g+UjRX0YgZA?= =?us-ascii?Q?+fCxoPIm01Lhuo79McQADfqi0+oxCmRqnvE5yVGnxMaTseH6RawNz4liWxX6?= =?us-ascii?Q?qailXROvwc2PI6MhPeHu6eKiB3FZWSFdxU0QI0tCawDAYD7wViCDi78fbCcY?= =?us-ascii?Q?7G5r4AaP0ijPVlvqM4CskOXWbGVjPPEPkK0s0IncnI/CCH3ScEuh7/XEU7xw?= =?us-ascii?Q?CDRE8XhQLfvpMuZAh+fzwpoObb+2aRY/Linoc/D460omhsQXxhqj7VyYENeY?= =?us-ascii?Q?QNhhoygc1lqwCZ7Du7ERoDL/aR/bA3N8MB0mpLOV6IZZJTj1Ztq40k8RtkAZ?= =?us-ascii?Q?sXb+lCtJEHciZ8Ra7L33lWTSv0Qt+RRk3CDgrrh2E7zqprnvftcB5PoeGGHC?= =?us-ascii?Q?YDmJhzO9MZ+hMtOSzRFszUqIBDp5J1dZ/RUFfdYYqPIjg67b4U+Qb+eETSv6?= =?us-ascii?Q?h5byKRJBBJiZPLjLD0/mABDhm6ANVNJCSxur1FtQnbBhNECb6oveAVkFGLJO?= =?us-ascii?Q?B5/iKLOsWRfkzIYgtMz7HPP99Wa5wDi06EaRiMsZop3nNSlVNXqwaZyqz8GW?= =?us-ascii?Q?pHIWX94tOfcfkwDyWPLnxpSWK0gFficb/bzrzeVTkmhwOP489uqhf/PZMWrA?= =?us-ascii?Q?P8fyXdgGJqfHOfA70MGTFEcAjjiJSySUAFNe9AkwdJVNlVRzysIVXzkVz2IO?= =?us-ascii?Q?9lUAUqF2BqsrA995GG7NufIqkfkjGBzsk/IA2q2yZQ9IZJJGtiLrNrMSaiBZ?= =?us-ascii?Q?2YqacBZXRZtnH85nxIKwttGV4RxIHTQutFa5r/IyA+mcXj6LA2FfYaSHZIs1?= =?us-ascii?Q?4U6xZy/G4VH+ueMaqA3S0kDlQjM0wHYrDAvTDDAIDWO3vqH/8UErXSKKcUP/?= =?us-ascii?Q?G6zBFEr6v2xW6GkJF4wppHBh0ziQNige05re5RlULdzwTX8Hh8RAM/gqyG0r?= =?us-ascii?Q?0fdjDY1ZZ7s8ilQ/83l7mfDC6Igw4fUFi9NwXs4UChe3WJ4muszUvmfs+c1e?= =?us-ascii?Q?jn6zLZ4WpJ7qGPPQ2vD8IaV5EdyMwiu8cl+BTrgpSYRAbcFKGi7SatYGaqCN?= =?us-ascii?Q?1wXFOL1iTkuMNViSTjRKHCLAqAP8isjwJEprkab+xgY6QMBzu1YWBgDe7cmf?= =?us-ascii?Q?kdBFxCp8ENi1NftzTObGPg6K5kDs9ABbUSiPQW0YqB8KFQseQnYxEijRG7sJ?= =?us-ascii?Q?N6mDiaJ82UlTixJQtGvVyl68HnuDy0K3SZbFUdAMte4nnvGwBCZi63BYjeNN?= =?us-ascii?Q?cgo3zUjyZyVvcdrpzPCVclqq4gA9PBs3aHk2SB9GWkDmIwzFiRxGz7gQSgfq?= =?us-ascii?Q?abAHZurf311NdGJnjht5Vtd7yudsltzUXqvTFEhIFX/Dllbu0cOWhT+UAM9p?= =?us-ascii?Q?vA=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?U3uketIAvmWiYK2sHUD32dH+JcqP5a8nkG40h4BKlpF3LHoiq1EHMIEcrvFm?= =?us-ascii?Q?i0EEoS47XljvK0H69Xy0s7+yamxZnv4LNGhto1It1HBcefNV1AhNXMhAa+79?= =?us-ascii?Q?43JvOrs3s5NMcgWZRdGUkSso7iYdDlHR77i4cqPEx72wDUh6WTpbmOe3KPdf?= =?us-ascii?Q?NHWvr7U+D6FzmKlM4bodBkyJbvtp/VsCUCyTH6gzJmPPArUSqtHU8Cxr3Y5x?= =?us-ascii?Q?twKR6Pp9kN/C33C9hkYWP9wMs9ecio8q59J9fcbi+GgLEm3bv3IL3IE+OKxI?= =?us-ascii?Q?ydBzokdHCbzYinXFr+XBYMMEg8oGCN7biJNo5TtHuIMPeGBsvyoCbKT08FWS?= =?us-ascii?Q?JzpEZV4ra6pcUNeUk3ihM0GMoKLTN5oW5+zFgi5ZTpOAkD1saswvE4Go2cJI?= =?us-ascii?Q?idkmCzeCNwt3S1k0vB9cToON091h9MJvj1PMG4V74Qac3gWWasHSsvqd9yml?= =?us-ascii?Q?2RMkbNVgiV8K+zcCakuO8XWIAmcb/c6l9dUjCSan8pYyByNbHytt3d4xlRfM?= =?us-ascii?Q?BoGcBh1q1j0i/HxIdMoCRNToh9ROFE9idDFmwO89ukedTxRJ/5OfbgSYsKBQ?= =?us-ascii?Q?iVDiqEEmFw+FzpmdZnA4p+oAdQmiuaMDB8b1eulAyy2VbWtcaj/nNDQYQn52?= =?us-ascii?Q?kiwinJ+FFtJu2B4l7PAeLCI+Vct8ilLuA7ootm5iIXyr3wJeIlZidXc9xZ2e?= =?us-ascii?Q?VsQuNEuM2UPEZ3l2xb6hYAnuuLcyNL8lEKTqU96V6T3kmxAeUxegRoDaMIuJ?= =?us-ascii?Q?dsrRSalYIDyvc/TE7HCDQ7QC43FZiiaQwLPZH5jD2cMM25Rzvk83p/ggA2qC?= =?us-ascii?Q?bFnrKnqIZTjZ2fw7x6Lb4IG1ozM4iQJrIJd0KufL2mTOlqBV18jHaenBawRp?= =?us-ascii?Q?yUbM5BQuInDxpCbUHJGBc5v0jRQcoz5z2MPWQyY4uJ/k9PeCTnP4HpONN4DG?= =?us-ascii?Q?MxgYdjmaB+XL6+hh75oOvvZ/7GSf+woZ6aTQm3nmmJOAT6ty7y7K17nK7d/U?= =?us-ascii?Q?xo0kCcRZwbgm2LRK+0qxc5//4rrxFbdoK8uAgHBwy5tPqvIpNYSC+k2RD41S?= =?us-ascii?Q?NoQYwQ6ib9kx6M1DjHigu/buAfNoAuAFhn97gLG7HCTzfk860AJpDuH9kp/P?= =?us-ascii?Q?yzk1lT6ChMTI?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ece5d3c1-753a-4f34-2ddb-08dba59e3b7f X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:22.6665 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UVKqhjj+Bfedxr1RlOSpOMV1U+XzrmeQXG8xQ8jiDMfcelCeaBIu0Zn4Ug3ByBw+FD0sgDjPJSwnL3SxqI1Sxw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: DUWcYynt5oije6WGFCohS6jt-asJt9Pu X-Proofpoint-GUID: DUWcYynt5oije6WGFCohS6jt-asJt9Pu Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Joao Martins Now that a list of pages is deduplicated at once, the TLB flush can be batched for all vmemmap pages that got remapped. Add a flags field and pass whether it's a bulk allocation or just a single page to decide to remap. The TLB flush is global as we don't have guarantees from caller that the set of folios is contiguous, or to add complexity in composing a list of kVAs to flush. Modified by Mike Kravetz to perform TLB flush on single folio if an error is encountered. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 904a64fe5669..a2fc7b03ac6b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -36,6 +36,7 @@ struct vmemmap_remap_walk { unsigned long reuse_addr; struct list_head *vmemmap_pages; #define VMEMMAP_REMAP_ONLY_SPLIT BIT(0) +#define VMEMMAP_REMAP_BULK_PAGES BIT(1) unsigned long flags; }; =20 @@ -211,7 +212,8 @@ static int vmemmap_remap_range(unsigned long start, uns= igned long end, return ret; } while (pgd++, addr =3D next, addr !=3D end); =20 - if (!(walk->flags & VMEMMAP_REMAP_ONLY_SPLIT)) + if (!(walk->flags & + (VMEMMAP_REMAP_ONLY_SPLIT | VMEMMAP_REMAP_BULK_PAGES))) flush_tlb_kernel_range(start, end); =20 return 0; @@ -377,7 +379,7 @@ static int vmemmap_remap_free(unsigned long start, unsi= gned long end, .remap_pte =3D vmemmap_remap_pte, .reuse_addr =3D reuse, .vmemmap_pages =3D &vmemmap_pages, - .flags =3D 0, + .flags =3D !bulk_pages ? 0 : VMEMMAP_REMAP_BULK_PAGES, }; int nid =3D page_to_nid((struct page *)start); gfp_t gfp_mask =3D GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | @@ -427,6 +429,7 @@ static int vmemmap_remap_free(unsigned long start, unsi= gned long end, .remap_pte =3D vmemmap_restore_pte, .reuse_addr =3D reuse, .vmemmap_pages =3D &vmemmap_pages, + .flags =3D 0, }; =20 vmemmap_remap_range(reuse, end, &walk); @@ -700,6 +703,8 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, = struct list_head *folio_l list_for_each_entry(folio, folio_list, lru) hugetlb_vmemmap_optimize_bulk(h, &folio->page, &vmemmap_pages); =20 + flush_tlb_kernel_range(0, TLB_FLUSH_ALL); + free_vmemmap_page_list(&vmemmap_pages); } =20 --=20 2.41.0 From nobody Thu Dec 18 01:07:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97D1EEE49B4 for ; Fri, 25 Aug 2023 19:06:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230466AbjHYTGc (ORCPT ); Fri, 25 Aug 2023 15:06:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230519AbjHYTGL (ORCPT ); Fri, 25 Aug 2023 15:06:11 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0CACD3 for ; Fri, 25 Aug 2023 12:06:05 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEQpJ020689; Fri, 25 Aug 2023 19:05:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=mxd9Mb3jlleYqr6X0jDbIBqXQT27sF2i7EgotUHdG6A=; b=xOwe74aAcnCzf64hyZnqxAjcg+AXqq+OeJG+CoDtor7TYnSlOIkkwbPlkuZi0QJaaexe 3ObSk9ugi9fjyv/O7vLmS4NLyMAckNMbk/8GglSwIZ7Y+tJOS/x8r2NGZy+QgLay1jHi cEU2GsYzMfRwaY1ygJnyILz/VB5tG44cmfipLHFUTx/EwF3MNNYt1bF1ApYDfHt1I6H/ nJYBC3zSr42g3Zn1egdRKSs1a1aFO90bnLUjybpPjhQ7Q4GntzIBMtR8i9HUSWm5Lujn o/THO8g888hVHQofzCQcIE26p12XPTK3r478i3JoPYh7tYorIPhySidaz6wcojf468O+ tw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yu779x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:29 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PHRtnk005844; Fri, 25 Aug 2023 19:05:28 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yuyadu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iaN2eP/2WLk7YjpoOSctnwyKaeNH3sVW1dKcbcqQ2vSkklHDW59kcsVgoSpDlOoQDooZ/J4ZPLBDGcW8ucmDOctjUJDyqnOHwAkH3duRDFTMIwBVAm/zm0hzxG14037EY05jDLJ5txCH0iPGvKx5wFYb9Vs2NVr/oIMIuU9on1vN+ZXxP1aGeUPx2mXOlooaheynVA4gidNJSsG1P0JsQA21z6HAdZKY+0YNmaCHlg3s1EFaquZoCkf5tRvx3YIZG3EfY0zLud8DTxePUHDvy3ZzmBGhW9wQT1WxbdlZw8dZ8BrzY1Xapv3tZOqxWDYarYv7lpmXTjWpTiWhuolZ7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mxd9Mb3jlleYqr6X0jDbIBqXQT27sF2i7EgotUHdG6A=; b=C1R9+vDAF0EdvTdlYZCjHQ2wTx/XUpmspAV5fcu13zbgE8MOTac/uQsUxLPkic26crE7YwKVDC71Cka1xlO2knnJrLqDz/ugviO0Rvnjf9dpSQ0C4SoUPwQ2D1NhuCoUQvoNjm5tABbCQkZoaBmmXhj4VseUkrHRuuE/HTAuWJHZAwpsmTDxrdwKsLV8Vv9icQWEcZd4DQctvkXSFtN5fFL9MBIotB3nfOycf/CQEctpfM1ZyQwjkbxUiul3pJceLkeIQUXPoVU/aFjZIxmNXvmr5gaNgMIdShh28ichaBtaQzOjG4Evejwz93kZTWp9T8p1tvcxOseJOLPclbC0dQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mxd9Mb3jlleYqr6X0jDbIBqXQT27sF2i7EgotUHdG6A=; b=LneynFCxOxCJeCTJiCzU05FedLicIJO/YFxNxo4sYKsLMpvhZzb/MAyZ+PHPqIInactm1SWmevBqwW3vuaXF0SikiPOcH0FbNvwsdzbbaC0dSWSNeCoueSSkdnNvUvJlFaFQA39Hcst8tSzihXSaHlPqU5kRWHQEZZJFuwWWIJk= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:26 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:26 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 12/12] hugetlb: batch TLB flushes when restoring vmemmap Date: Fri, 25 Aug 2023 12:04:32 -0700 Message-ID: <20230825190436.55045-13-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0240.namprd04.prod.outlook.com (2603:10b6:303:87::35) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: c4099dca-1ec6-45a7-88ba-08dba59e3d97 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QaIVMGTQ5ax/Ee5aBYf85w5lv3QoggNv3UMcKCtofmGzdZwkQZ3BKwjMh7dBkuRdh/yS8qmOFim8OsdsvcYVXsm4A3TrZHQ21zlPuq80nk8SuxPHTaAv4JJqSnJjs/DIrYFU/xpzqH6mhKqcPo2NPqDu9tTEk/Y1jVgvjzpNpeKZWUAx0iUGhZk1P6IMc84koDgnTC6dQOjHGRakZ4G/09APNvA8kctk5mMtFnYpJHq7gWJIHLEIcdOGAjq2TAAQtYFnywrfowaAgTKebXv3vwnxBO3VP1QlI/Va58Dw4VPDQKB9tfeFY6hfcZCdhgqnBk+smqmMfKjwLxd3EZ2QGroVzFqlvadKLh3EoJ6X17t7l7Os4l2sVHPtZbZHByyEbLql9Q0RGhMQVJeOGlfq4+UCzKCSxgwKwmB27hZ3uvx8J8ZUZAogVM+VHdijj7Kqy4lor3JfP9vMyFGH+V0d28KZuc859sgEhD9tTMR1riW2JqzcZfk7XDcY/osiC460qG4B0euxYW02H91gj2Jc64gLDzu1MrAHtTp82l5I/s+eoG/MShngRk8CLlOKLA9G X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gfHHhv5xJqa+q012OG1KsLGGkJslK5l8SHXdXHzDmBHuIy1BXxs9wm7hyBXL?= =?us-ascii?Q?LtPxfQBo5dy1YMVGqnm4qAWsDq73YsZ8iLoUoFpZA/Ce1/LBI9q+jQrtfdFm?= =?us-ascii?Q?J4/gb0yTfWVyR9D1wYK889KaoYqZDz2kvf/SZLvIO33OuvBmarf0PzNTqbMb?= =?us-ascii?Q?gSoAS0A48qC5oH79xhOOI3iKqVPPLdaKrFuyqPfKKfx21sgQlW15aGSgNSci?= =?us-ascii?Q?QlXMGGcRjF1WXjlOlBWY+IsVwyj1FiYu4E0oKPiuz8W6FXoXKpZN8NsKY2jb?= =?us-ascii?Q?Ieyh7D1Ikmh/LH5hC/TP1OzVinoqIS2VklfzFcbJtDTYysVLuhZb0GHPpIUR?= =?us-ascii?Q?wMjXRP+pb0Z/y1E8zHuw6xDetLrs3zjB//VsQf2mpMKphUjEC5J8YIhBngHo?= =?us-ascii?Q?ODWsBmKn6ywGoGXkL0UM8t2fm5yBWXS3wvl0ATAmhEWe6CbM6HaEaqnN28N+?= =?us-ascii?Q?Oop4GNXgJi6N5v6FjLJrSNMiB9hj55AV30jqYK3HsiZziK6bAm6HOZ9vWgQY?= =?us-ascii?Q?9NueMq8y/dE5RJIvkD6sKcpfGrBPm7HWTbFnLxY+QFoyrIukt9Jk8MdtrbBi?= =?us-ascii?Q?jeNB3RjjrKIv13gnzYbqMi2L4n7ZophVg7xH1jkAfTxv2Qndw1g9ZAZEankK?= =?us-ascii?Q?bx/xH+QLmrZAdkjUSp20V6lM9JGtnVUQuOg/ibAeYmeh1KKIyeZ/YQnLEmFL?= =?us-ascii?Q?5HZbvw5rGPBkEsVjbQAFIB0aNS9VV8A4aLH/irrvJTW7smrnDQF852dFyY7r?= =?us-ascii?Q?Sr5hA6V5avwidixT9T7DL0AaDrpyEQ9SHlICsUlCF+rC59Lfe3Lnh+H9hFu5?= =?us-ascii?Q?7hhpVZHaJnWQAYZuzaYX6rjEB4b/9YbJKolSD4rZ5rk+Va1HV/MSMeatQPsM?= =?us-ascii?Q?bw6F7MWRFsWTleMk8nDTuVrP8X/mlZyvfvRhvLKNi7Fp1dXXUOWf4QP83+dk?= =?us-ascii?Q?9uUCAb3KcNZuswlpPwlCP0SAfrFW8EcrdnS/MT0QQZlA8WMe5A2z7a9mGvgf?= =?us-ascii?Q?egXp2iMbJzCCBSd4bdVf60EiAwNdGrYH2cSNZXprIwXSNsYMm1wITJzzVy6a?= =?us-ascii?Q?mdgdIhxFvIbPUoO72ULs1ur1M8iuJ4NdiI7Tjp9kJBuG01Du+/xyRo0tNHZz?= =?us-ascii?Q?nLJSuSRzwMbnDO/xcTnud6WRMksbApEosROUoV7teSJX2OIBiUbzfvBW3JGB?= =?us-ascii?Q?3ONu2Md3FbzdlYIiPAWtZBlqOvod/fsXk6jzGsQE1u+GlLAL2geXiaaBfD8I?= =?us-ascii?Q?CmgKmhgCwg693ZAU7n7s2jI4KgL6r4yID8UF6dlqdU2UeD8Cz4Qkm77q3Am4?= =?us-ascii?Q?+7L21mvk3j44YmXmAnnS8j7HpQGimqIEy1sGXhAjrDoWhv23deXJDLZhZBtm?= =?us-ascii?Q?pgAhL4V+LN0kwETD9syf5dNykZmkZT0Jfao24uJ76irjxIuiA1rtQbGn+yU8?= =?us-ascii?Q?rMSxXKHRmM+lfhbl8qh5fmYNZWXN6y3BrhD6b/xrejPMeQs7ZfzkvBBjXI/W?= =?us-ascii?Q?jb+E8kEfEGSnJz+GBiMoH4mXXC4BfvGGoydL1I5gcWrKV6yyugWwS03WiRmU?= =?us-ascii?Q?frDRDg09kMMmDXmZX3+eKCnzZgKvedbBjcaJGIf+CiEevBNC4AlqBFrtK8wz?= =?us-ascii?Q?Lg=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?/Eg/HPGvi5G2PVpkRR1okJp4pbbIMNOyL37Y6Li75xS0dicYwxkHafTFXTvK?= =?us-ascii?Q?6+ek27KKdVv+3Ho5Kj4lpmiS3C/7crzzwUIxC6IL8lUx8thK7etIKYrs0kJV?= =?us-ascii?Q?Q4dSVfIpn2UwR67rFTBpzdRm1rAUkPrTIDw6CUtbprbkowH+j6eC1ZJJ7ewS?= =?us-ascii?Q?vGR8XcuDHoxC8fmJ8UzUoPYcANkIgvaV0AH1gGVqNmi+5LGuy2PuaE4IKqvY?= =?us-ascii?Q?bgRLGk8obrR/3X+Jw8vBpJMVJoZcXS/uNgDgTKWQN0pjXkYsdYfYGiLELTj5?= =?us-ascii?Q?bs9/TA9AgpU4zS1Rybc2sRTPnX76A+QszmcHBH5uuq3Ur5WGVS2yjOyf+ULr?= =?us-ascii?Q?oaebh5VMUfUidIKf5vtb4McKUBM2E1k0vTEwO26lie4/J/QPcj9ac0Eb20yO?= =?us-ascii?Q?KEhcw+U7MYceNSH7EQutTwWSyhGHNaqKiMkZK+d4rA/28HMkw4ug2RqRCFcB?= =?us-ascii?Q?iaIMZh4GF78vkkR3tRPBX9944qWH9tIiKc9tSx8WEdlUzYT5X83XeWw+m4sz?= =?us-ascii?Q?r7Lh+rq1Oek5C/7Fr0QuBKftrYUHk54YSz5QdOz4U9Czk1ELT1zhl4zysiz2?= =?us-ascii?Q?AK6Qzs6fnc5RruKLWK6sNDAXpqJF05yuzfbSjh9rHMo4q/VV73EQZ9mSxaZs?= =?us-ascii?Q?aV8zbMArQEhp1cqcw7WYwJUfLdvjJNKaxG18Wu2wICfo74ClZ4UxcK2LyPXX?= =?us-ascii?Q?bNm1oEFFG1EUGnGdfGXvvWAnvatgqwAsTqtgnJp9T0m3CnUvx3Qb6EwsotR/?= =?us-ascii?Q?RmPe3giHoY9DUGYfid75xmYO3jmj3D9IcDDrxPLtMyEdAKmDXf50gEu7LMWl?= =?us-ascii?Q?NXWZZeeSnK254u6kK7xZQOIVlFu1mJlyozSWCZPZRijKTY1yDWQoZ9c/bekL?= =?us-ascii?Q?31GflhMj+39UfL5lFVtgbGBaVcHkVwTG1sct2Jwo+G0cORV2vaVQg3dqF00r?= =?us-ascii?Q?bE7/2QvodOM36jy3iVtveZnuGzQ/NeVgGYCJOsHIvxwdgj7DuA1CQ/EMDwsi?= =?us-ascii?Q?9lrLAwMLijtyOAITcsuCRIk3eupt/4eAFo6gXEyJ0CfzSXBbxYrJzaQJqopx?= =?us-ascii?Q?KHkW8eqEciX58a/SN4Y/8gpCdvwP3YX3zxeUW/AmBrFvaKFhA9ew6BHDVYD5?= =?us-ascii?Q?4Oh8GsrW2KB0?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c4099dca-1ec6-45a7-88ba-08dba59e3d97 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:26.4302 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CYn6XHIEJMu4Sx/+dRtxXfeaG9eYX68aEHz+Z4dGfFqXhG3lPnWcZ6LI/UY6BFbiQp4Ci++WmXFmUGUdvr132A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxscore=0 adultscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: 857681GrhwUbjGYotcHlduuiTyZWE8zi X-Proofpoint-ORIG-GUID: 857681GrhwUbjGYotcHlduuiTyZWE8zi Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Update the hugetlb_vmemmap_restore path to take a 'batch' parameter that indicates restoration is happening on a batch of pages. When set, use the existing mechanism (VMEMMAP_REMAP_BULK_PAGES) to delay TLB flushing. The routine hugetlb_vmemmap_restore_folios is the only user of this new batch parameter and it will perform a global flush after all vmemmap is restored. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index a2fc7b03ac6b..d6e7440b9507 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -479,17 +479,19 @@ static int alloc_vmemmap_page_list(unsigned long star= t, unsigned long end, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @bulk: bulk operation, batch TLB flushes * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, bool bulk) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk =3D { .remap_pte =3D vmemmap_restore_pte, .reuse_addr =3D reuse, .vmemmap_pages =3D &vmemmap_pages, + .flags =3D !bulk ? 0 : VMEMMAP_REMAP_BULK_PAGES, }; =20 /* See the comment in the vmemmap_remap_free(). */ @@ -511,17 +513,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled =3D IS_ENABLED(CONFIG_HUGETLB_PAGE_OP= TIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); =20 -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, b= ool bulk) { int ret; unsigned long vmemmap_start =3D (unsigned long)head, vmemmap_end; @@ -541,7 +533,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, str= uct page *head) * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret =3D vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret =3D vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, bu= lk); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -550,12 +542,29 @@ int hugetlb_vmemmap_restore(const struct hstate *h, s= truct page *head) return ret; } =20 +/** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, false); +} + void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_he= ad *folio_list) { struct folio *folio; =20 list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_restore(h, &folio->page); + (void)__hugetlb_vmemmap_restore(h, &folio->page, true); + + flush_tlb_kernel_range(0, TLB_FLUSH_ALL); } =20 /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ --=20 2.41.0