From nobody Mon Oct 6 19:09:20 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D26F1BD01F for ; Fri, 18 Jul 2025 02:30:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805816; cv=fail; b=IhDlMadOLzPIIXyBW2SlnVbNMqd9Edb6GXjyXx31S4NBj2V4kf0pgU57tBL3T93cOKOWsemCoahTRmH9yRo0bDfcYzqy9s73jQqNpl/3lf7VsHkp0T4OxFShyQW3M+LJykIGlYmManYAaEpdwvfDmKNx7RN9y/E7ay6m8TAtehE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805816; c=relaxed/simple; bh=uUOfqsBXLrpg3PLRoeM2u1AR7ZBedex6AWOaBPB0Rnw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=kZ/q7T3vcd3HT1Ljz09D9v8wxKpQKkbu2Fx6c9dc/GGOi/MAF62hElUhs8If6QzAU6IQRxN51N5uiNR0qV9T0MYwkuyfKZfTXVJ9KUlOhbikp0oR333rfFH7ynpdq/gR10Jms8QuuP8gsJhf0qOaFAj8P5RmafFS2SmJ0MVSLns= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=hrXikbZA; arc=fail smtp.client-ip=40.107.243.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hrXikbZA" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Lbfe0TTjQiKH8ILuBEkhLe7Icd9n1q/0KPPEbajYPuT5e499ST5XjrHTNSleWoE1YOaTiSy29tDmqt91Gu2et+4QnQYoR1n4SK0O4isjntHqOczITnU4PWH/DwVKrYAgrOvhsjrd0XCtWnyTIlGIBzF2VWzQ2BtYAWPLcI3fNVxxtW8u2HV4JbJIOikWzEra25xdMbub4q1K1Ggh4RYmsZ6OovW5DK/sRx54B8K5hQhQ09K8kDgKUCihnt6Oz0sjnS/d7zD4iZV6bVGuu/mufhiCJ3uOeT8fK+JCEgz/UL+yMoU0lTYy4lagfINbKj2uxB0CbW1mmC1Jgyshgne/Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Xjun7XvdLoafkm/qATsHjnvpS3DLiaaE2BVgzS2Giqc=; b=NXTY4AZ0euAfecQiwxqu4PnrgNRb8Oatv0JZtJXeghza1Oj9MRH56SV+bmCw7LaqLrGRBCCQSaxQglfUd6j8Tb4mnMvCOw4w7GP2+1vKMVzd6Rp+eN9Ep81mhVvYVJfXKhZ9FxpuC4KGTXqfGCb9RWd1bN3MlibiJ2wJcExrrxCRo2dCb5GcOc/Yx/nYVqjyFImFtfulqU9L/I69V8GVzPh6/NlmsRbVOBWV2fUu5fzHwZrHIJ8KNBPtDoaX1uFO/OPLOnjDI1DvT157v6TmLL53mhJ763vkjOpGkzMBDxo01jx1zI+8tO7UDgCcxEgYJd9WHJB7tWEg+B+rklVE7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Xjun7XvdLoafkm/qATsHjnvpS3DLiaaE2BVgzS2Giqc=; b=hrXikbZAHkfaxw7Bv4WjbyGQFwPBAzNSa13jAb3i3Bsq+TMKZqgbDkMOFNtCr5BE7DmTApVQLV4/s8CSWgkhy7xXc7ImL2PFdhIAqtYNK9vWI3/j8ZLjscRV7NrjPpb11hptS2H96WEHe5mjErO1MO6jbvTS2ZDvTuM7fT0BfS1ia4mROX58INuwA5Uyc96P6Dz8gLOINm5G9rsidkR1zLNmAXophbKNgTd5Md4/oapCuzokiyMmCgvJDZDmKFf40rwxObVrOMhFNw1ORcOpVbGMPcFWxZ+E60WfA9pT8+ZlFb9FAqDafsmqT6RnPqMRV4erPWxDE3EoVHtH2HQkVQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by SN7PR12MB6766.namprd12.prod.outlook.com (2603:10b6:806:26a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.35; Fri, 18 Jul 2025 02:30:10 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 02:30:10 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v4 1/6] mm/huge_memory: move unrelated code out of __split_unmapped_folio() Date: Thu, 17 Jul 2025 22:29:55 -0400 Message-ID: <20250718023000.4044406-2-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718023000.4044406-1-ziy@nvidia.com> References: <20250718023000.4044406-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR22CA0007.namprd22.prod.outlook.com (2603:10b6:208:238::12) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|SN7PR12MB6766:EE_ X-MS-Office365-Filtering-Correlation-Id: 05af4313-0d72-455f-6e82-08ddc5a30485 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?H2uTUKNxl6C88x8dUlDKScDKiSDLYdNY/PrPIL9S/6wnLEervKPQAvyC6lTx?= =?us-ascii?Q?rG5v3YcmavZZynXXZ9KHSHHThLqmHwzQ9KppUNS9VGMf2b/DgVmNoQkDLCsb?= =?us-ascii?Q?+YFwOnTPAztOCzTjigaNgECOymLQw3AaBlr5L6ir4Scpc5h6lseDx9M2fsu7?= =?us-ascii?Q?wtpBo8GBJ9paOjAZMkUxz9B6ZvBIaynWVovIpfrDDDSQOiPuCsfQO6lOx8wU?= =?us-ascii?Q?mYmf4ttiHirSQp6C3pAJGDAxjw2AGU3m2gQLRCPBGbRrGWEEZfdJyGT97SOY?= =?us-ascii?Q?kmM2sKIw1Q+NnTxh4Vuk1EljZ93Dkw2B4E5R7c3SRz1Xi7A2mv38PDdvSzVD?= =?us-ascii?Q?l32LH2MvQ7AAtr0Iyw/ZcXnYrS3IzPni+b0t8/eWa8AJ7syDrhMvJgfXKhQd?= =?us-ascii?Q?n2JQWpg4x368T9aEA98FU3odEHBjlDAQf4xZd+NwkXysqSy4h9vJGTCoaozZ?= =?us-ascii?Q?pm0PZnXmCj0C/01hw4tMM0wW8SY2qusR6Wfcba3suu8viaS0iT0Sxc/HRXKQ?= =?us-ascii?Q?jMP3/wgI/+htSEfFx+/1j6pWXNO9kGjNEvS2XYRoJ0+3obnkpjbvBREPU97X?= =?us-ascii?Q?SFuAYBdudvTR00hBckgEbX2H5T5KvfL9uxV8c2zRSRm8Whao3GgLY7vx5JX0?= =?us-ascii?Q?9joOzMv9v12SOcjW+YWd5WM5uT0OIJq2/9OdRGBRjwbMaYw20YVWekf2+Mor?= =?us-ascii?Q?q9iJtOT+5yfmkSHcYb1DKwrlQSYWx3A5+CGKISi7ShLfbjoFGrK6emxpjmo8?= =?us-ascii?Q?05brT5/7LgHYvuA7Z6ydmxb+Z7S60WOtuepm4eY/ciKqV/GSbfP2x/UEeCdK?= =?us-ascii?Q?I6KD/57n1h26R/HaQBsy6U1Jp0yD6KeKWysRhauyCH38jIbSFixMkn/BAsgm?= =?us-ascii?Q?BvaV3WNVDmMUqKAEL+Rl6Q7se/8sHYEVUvWMc0U9q8UQPUXGquuWXf5TAANP?= =?us-ascii?Q?VCXP0AOBnMw7cr+goUjTOXLWpAAsazRaC4Oebnn28pCXU+sx0ESCxqUnJu0O?= =?us-ascii?Q?hUhBzuL9mz/fEHqdiIXUyNcV5lYI4br2tAw2Y1FN4cUZXs3Bxiu94Ek8WkJ8?= =?us-ascii?Q?bqh61fnzWWa69yfB9rfhBgtwC1sU7C5QN+Txkl19e+42uS8XvlFID1shuPmn?= =?us-ascii?Q?dKlvV0BSSGOWxLgVTw4Ea9HHzG1jzZ/cDeEeh+HE5DMP4mjLLN4vf/ElC/uT?= =?us-ascii?Q?XcyFK+H3FMO9Z9vOxndokf4oVLUnlTqWmSyX/ayOWdTeckfLrWw3734Ogvsp?= =?us-ascii?Q?GIhhEQCsiVjC2tneQuOsJSUnaZonw7POb4MgndVCUeQBkkCWTrorfGvlxEKv?= =?us-ascii?Q?5SCPUjL5ZG0/hyRVSxppmudcdWeXoJEsOK31XyJQw36IWSw00SPjwRGYJFrh?= =?us-ascii?Q?dFTr94jDD2g7Oxw+sATH400gg23Hc3wadEITYQtMi4nnSoimVJs5eduVwSlI?= =?us-ascii?Q?UpeUa+kIdrQ=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?6R2KDgep5Wja+Y7EAfQpTfqOSsYNz3Lack9XAsASn9mSKz4aK73kbJ/+exnf?= =?us-ascii?Q?ikYU0CBZJ7U9EkqYVRCdk2auZPX0OzrZfu+xDc3Zh+bWtaCzTnEzfWmoDkjt?= =?us-ascii?Q?7AnqAWwJLm5p2IZbmiD4zUPM2vGiw/P4E6niQ+UVzc8e8dhCYHjr9cIfMEgB?= =?us-ascii?Q?8Mat745pNP7V+EV14q6gpqGHe89V/5nzc3Vu4Kj/kdc381wShqJc5tytfqmJ?= =?us-ascii?Q?AwZqgE9yx+iJSCga9fHoXw4dGfYbcxGCUD8EK/THM84iclRBPsxR7lM6tUDi?= =?us-ascii?Q?0CtQpSgmdgsAz4BZ1fLud+vjZLePeZDRV1EQ2pmDkeCQxCra8Bpp/BolJXDv?= =?us-ascii?Q?MSldGms3uYwhjuu0PGBLAbIzRwowg2CNeclQN53makIRTED+a9qTHZmRJH58?= =?us-ascii?Q?06b1kd2k0KjQH6vWNBTfiJnwo4uMPd75Vqxtp3KfJRpQPs34irrJvZ+fMoGA?= =?us-ascii?Q?QInIKI+zc4+nP2C6ei/60cK2wYmUR3QTrS+gmBnamRsYdihHSDWMGbsLzW51?= =?us-ascii?Q?5z2gCqw2q7RbI+RJ+3Rm4MbtvYIO2zCskInhejWwUO8nKzh6bauUwapdHnru?= =?us-ascii?Q?gYjrKCSp7hBKHlwziaM3wvp9Lsa8O26JD77xCeUlVPWZN6z3tzkgND7UwynK?= =?us-ascii?Q?iX4peDvH1pCL67ZlPRMAZ6qhDWUfmqnUkhDg5qxAew7ISdSePo3ZkJn3Psbi?= =?us-ascii?Q?f3Ovjq/FIK8X198xQe+WzVioN2ZMsF1SSyFGhFhNSHIuQxhxIZIx75zTDj0b?= =?us-ascii?Q?8bqi4qSPnhFV5SlPGQ4jJ1GG65elRYmjpMDnAN++nPpX8JGhGBBhxiSk6toR?= =?us-ascii?Q?VTXGPTci9sy459TmsFmPIC87+RVlpOJfeFx+G8Qwy9fNf8FswbRafWHSbHZv?= =?us-ascii?Q?b6hiR+GKH5qbbsmosiJvtnnGETqJzE1Tk4ZWhax3Bk0mnGEcjSOKBp+718du?= =?us-ascii?Q?qqEwUkMm0TALpEDKTfBn2H3M7u5Vv9o3z0VSsDh8H7sg79LHODrMpjDIv4cV?= =?us-ascii?Q?yCtW6lFAVWeMVTmk4AHXe0vaACpQDKkpJEhhC090VJ9m0rFKBxHQpRhD9Tlx?= =?us-ascii?Q?ywZzYitPNfAcUslIXOxc06DMQt/BXlaEPVhjcgVe/jen4unFtDGh/1vbm6kl?= =?us-ascii?Q?7RvbmvZ2PbYF05cZTxcu7D1KpUoFXfAL1IZzmHYyhpdoHk/huKh2oHHGiPj9?= =?us-ascii?Q?9obz41ylU6rOGjr/dYEJFjntJ1MYOjMVBjehV+Vt6P0I9Sj1lWnkfpOFBytC?= =?us-ascii?Q?3vaIVpbdbqnsurnMPh44IATdM8GUY7S0puoF5NGPhG17vpUwn2K8quTQL9Zh?= =?us-ascii?Q?cALyhpcwLIINKsdeR4xgHxnGNc5bkkFs3KsOxHSqsR0juWzmtBs7rRyDAA+v?= =?us-ascii?Q?syV/n4RzNY+T9sKPCYo6K8bOTBDSVgXUAyFamnlvY+wbs0tPuOcwr/2idiu4?= =?us-ascii?Q?R93kmYCkQaMt/p7FohYKitNT3Kdb9i5IGjRhrTlWXUA03QJVxhIE/rbiCNqi?= =?us-ascii?Q?8LUXMqSxumU7wgvUlptRcHW35mUnfDrqm59Y3t6uFA3SJlN81K/gnm1FdjUq?= =?us-ascii?Q?PhWMh0/5q/Ped54+z/XUGRq+O/Vjao4ZSYXoZ28U?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 05af4313-0d72-455f-6e82-08ddc5a30485 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 02:30:10.3891 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Hs4TyigPJwHCX7YGW8J5D9z5L40LRysGRAnvBPvgsfFlpAyUg4WLG1f43q/O8uPh X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6766 Content-Type: text/plain; charset="utf-8" remap(), folio_ref_unfreeze(), lru_add_split_folio() are not relevant to splitting unmapped folio operations. Move them out to __folio_split() so that __split_unmapped_folio() only handles unmapped folio splits. This makes __split_unmapped_folio() reusable. Remove the swapcache folio split check code before __split_unmapped_folio() call, since it is already checked at the beginning of __folio_split() in uniform_split_supported() and non_uniform_split_supported(). Along with the code move, there are some variable renames: 1. release is renamed to new_folio, 2. origin_folio is now folio, since __folio_split() has folio pointing to the original folio already. Signed-off-by: Zi Yan Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 270 +++++++++++++++++++++++------------------------ 1 file changed, 133 insertions(+), 137 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ce130225a8e5..63eebca07628 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3385,10 +3385,6 @@ static void __split_folio_to_order(struct folio *fol= io, int old_order, * order - 1 to new_order). * @split_at: in buddy allocator like split, the folio containing @split_at * will be split until its order becomes @new_order. - * @lock_at: the folio containing @lock_at is left locked for caller. - * @list: the after split folios will be added to @list if it is not NULL, - * otherwise to LRU lists. - * @end: the end of the file @folio maps to. -1 if @folio is anonymous mem= ory. * @xas: xa_state pointing to folio->mapping->i_pages and locked by caller * @mapping: @folio->mapping * @uniform_split: if the split is uniform or not (buddy allocator like sp= lit) @@ -3414,52 +3410,26 @@ static void __split_folio_to_order(struct folio *fo= lio, int old_order, * @page, which is split in next for loop. * * After splitting, the caller's folio reference will be transferred to the - * folio containing @page. The other folios may be freed if they are not m= apped. - * - * In terms of locking, after splitting, - * 1. uniform split leaves @page (or the folio contains it) locked; - * 2. buddy allocator like (non-uniform) split leaves @folio locked. - * + * folio containing @page. The caller needs to unlock and/or free after-sp= lit + * folios if necessary. * * For !uniform_split, when -ENOMEM is returned, the original folio might = be * split. The caller needs to check the input folio. */ static int __split_unmapped_folio(struct folio *folio, int new_order, - struct page *split_at, struct page *lock_at, - struct list_head *list, pgoff_t end, - struct xa_state *xas, struct address_space *mapping, - bool uniform_split) + struct page *split_at, struct xa_state *xas, + struct address_space *mapping, bool uniform_split) { - struct lruvec *lruvec; - struct address_space *swap_cache =3D NULL; - struct folio *origin_folio =3D folio; - struct folio *next_folio =3D folio_next(folio); - struct folio *new_folio; - struct folio *next; int order =3D folio_order(folio); - int split_order; int start_order =3D uniform_split ? new_order : order - 1; - int nr_dropped =3D 0; - int ret =3D 0; bool stop_split =3D false; - - if (folio_test_swapcache(folio)) { - VM_BUG_ON(mapping); - - /* a swapcache folio can only be uniformly split to order-0 */ - if (!uniform_split || new_order !=3D 0) - return -EINVAL; - - swap_cache =3D swap_address_space(folio->swap); - xa_lock(&swap_cache->i_pages); - } + struct folio *next; + int split_order; + int ret =3D 0; =20 if (folio_test_anon(folio)) mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); =20 - /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - lruvec =3D folio_lruvec_lock(folio); - folio_clear_has_hwpoisoned(folio); =20 /* @@ -3469,9 +3439,9 @@ static int __split_unmapped_folio(struct folio *folio= , int new_order, for (split_order =3D start_order; split_order >=3D new_order && !stop_split; split_order--) { - int old_order =3D folio_order(folio); - struct folio *release; struct folio *end_folio =3D folio_next(folio); + int old_order =3D folio_order(folio); + struct folio *new_folio; =20 /* order-1 anonymous folio is not supported */ if (folio_test_anon(folio) && split_order =3D=3D 1) @@ -3506,113 +3476,32 @@ static int __split_unmapped_folio(struct folio *fo= lio, int new_order, =20 after_split: /* - * Iterate through after-split folios and perform related - * operations. But in buddy allocator like split, the folio + * Iterate through after-split folios and update folio stats. + * But in buddy allocator like split, the folio * containing the specified page is skipped until its order * is new_order, since the folio will be worked on in next * iteration. */ - for (release =3D folio; release !=3D end_folio; release =3D next) { - next =3D folio_next(release); + for (new_folio =3D folio; new_folio !=3D end_folio; new_folio =3D next) { + next =3D folio_next(new_folio); /* - * for buddy allocator like split, the folio containing - * page will be split next and should not be released, - * until the folio's order is new_order or stop_split - * is set to true by the above xas_split() failure. + * for buddy allocator like split, new_folio containing + * @split_at page could be split again, thus do not + * change stats yet. Wait until new_folio's order is + * @new_order or stop_split is set to true by the above + * xas_split() failure. */ - if (release =3D=3D page_folio(split_at)) { - folio =3D release; + if (new_folio =3D=3D page_folio(split_at)) { + folio =3D new_folio; if (split_order !=3D new_order && !stop_split) continue; } - if (folio_test_anon(release)) { - mod_mthp_stat(folio_order(release), - MTHP_STAT_NR_ANON, 1); - } - - /* - * origin_folio should be kept frozon until page cache - * entries are updated with all the other after-split - * folios to prevent others seeing stale page cache - * entries. - */ - if (release =3D=3D origin_folio) - continue; - - folio_ref_unfreeze(release, 1 + - ((mapping || swap_cache) ? - folio_nr_pages(release) : 0)); - - lru_add_split_folio(origin_folio, release, lruvec, - list); - - /* Some pages can be beyond EOF: drop them from cache */ - if (release->index >=3D end) { - if (shmem_mapping(mapping)) - nr_dropped +=3D folio_nr_pages(release); - else if (folio_test_clear_dirty(release)) - folio_account_cleaned(release, - inode_to_wb(mapping->host)); - __filemap_remove_folio(release, NULL); - folio_put_refs(release, folio_nr_pages(release)); - } else if (mapping) { - __xa_store(&mapping->i_pages, - release->index, release, 0); - } else if (swap_cache) { - __xa_store(&swap_cache->i_pages, - swap_cache_index(release->swap), - release, 0); - } + if (folio_test_anon(new_folio)) + mod_mthp_stat(folio_order(new_folio), + MTHP_STAT_NR_ANON, 1); } } =20 - /* - * Unfreeze origin_folio only after all page cache entries, which used - * to point to it, have been updated with new folios. Otherwise, - * a parallel folio_try_get() can grab origin_folio and its caller can - * see stale page cache entries. - */ - folio_ref_unfreeze(origin_folio, 1 + - ((mapping || swap_cache) ? folio_nr_pages(origin_folio) : 0)); - - unlock_page_lruvec(lruvec); - - if (swap_cache) - xa_unlock(&swap_cache->i_pages); - if (mapping) - xa_unlock(&mapping->i_pages); - - /* Caller disabled irqs, so they are still disabled here */ - local_irq_enable(); - - if (nr_dropped) - shmem_uncharge(mapping->host, nr_dropped); - - remap_page(origin_folio, 1 << order, - folio_test_anon(origin_folio) ? - RMP_USE_SHARED_ZEROPAGE : 0); - - /* - * At this point, folio should contain the specified page. - * For uniform split, it is left for caller to unlock. - * For buddy allocator like split, the first after-split folio is left - * for caller to unlock. - */ - for (new_folio =3D origin_folio; new_folio !=3D next_folio; new_folio =3D= next) { - next =3D folio_next(new_folio); - if (new_folio =3D=3D page_folio(lock_at)) - continue; - - folio_unlock(new_folio); - /* - * Subpages may be freed if there wasn't any mapping - * like if add_to_swap() is running on a lru page that - * had its mapping zapped. And freeing these pages - * requires taking the lru_lock so we do the put_page - * of the tail pages after the split is complete. - */ - free_folio_and_swap_cache(new_folio); - } return ret; } =20 @@ -3686,6 +3575,11 @@ bool uniform_split_supported(struct folio *folio, un= signed int new_order, * It is in charge of checking whether the split is supported or not and * preparing @folio for __split_unmapped_folio(). * + * After splitting, the after-split folio containing @lock_at remains lock= ed + * and others are unlocked: + * 1. for uniform split, @lock_at points to one of @folio's subpages; + * 2. for buddy allocator like (non-uniform) split, @lock_at points to @fo= lio. + * * return: 0: successful, <0 failed (if -ENOMEM is returned, @folio might = be * split but not to @new_order, the caller needs to check) */ @@ -3695,10 +3589,12 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, { struct deferred_split *ds_queue =3D get_deferred_split_queue(folio); XA_STATE(xas, &folio->mapping->i_pages, folio->index); + struct folio *end_folio =3D folio_next(folio); bool is_anon =3D folio_test_anon(folio); struct address_space *mapping =3D NULL; struct anon_vma *anon_vma =3D NULL; int order =3D folio_order(folio); + struct folio *new_folio, *next; int extra_pins, ret; pgoff_t end; bool is_hzp; @@ -3829,6 +3725,10 @@ static int __folio_split(struct folio *folio, unsign= ed int new_order, /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); if (folio_ref_freeze(folio, 1 + extra_pins)) { + struct address_space *swap_cache =3D NULL; + int nr_dropped =3D 0; + struct lruvec *lruvec; + if (folio_order(folio) > 1 && !list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; @@ -3862,9 +3762,105 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, } } =20 - ret =3D __split_unmapped_folio(folio, new_order, - split_at, lock_at, list, end, &xas, mapping, - uniform_split); + if (folio_test_swapcache(folio)) { + VM_BUG_ON(mapping); + + swap_cache =3D swap_address_space(folio->swap); + xa_lock(&swap_cache->i_pages); + } + + /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ + lruvec =3D folio_lruvec_lock(folio); + + ret =3D __split_unmapped_folio(folio, new_order, split_at, &xas, + mapping, uniform_split); + + /* + * Unfreeze after-split folios and put them back to the right + * list. @folio should be kept frozon until page cache + * entries are updated with all the other after-split folios + * to prevent others seeing stale page cache entries. + * As a result, new_folio starts from the next folio of + * @folio. + */ + for (new_folio =3D folio_next(folio); new_folio !=3D end_folio; + new_folio =3D next) { + next =3D folio_next(new_folio); + + folio_ref_unfreeze( + new_folio, + 1 + ((mapping || swap_cache) ? + folio_nr_pages(new_folio) : + 0)); + + lru_add_split_folio(folio, new_folio, lruvec, list); + + /* Some pages can be beyond EOF: drop them from cache */ + if (new_folio->index >=3D end) { + if (shmem_mapping(mapping)) + nr_dropped +=3D folio_nr_pages(new_folio); + else if (folio_test_clear_dirty(new_folio)) + folio_account_cleaned( + new_folio, + inode_to_wb(mapping->host)); + __filemap_remove_folio(new_folio, NULL); + folio_put_refs(new_folio, + folio_nr_pages(new_folio)); + } else if (mapping) { + __xa_store(&mapping->i_pages, new_folio->index, + new_folio, 0); + } else if (swap_cache) { + __xa_store(&swap_cache->i_pages, + swap_cache_index(new_folio->swap), + new_folio, 0); + } + } + /* + * Unfreeze @folio only after all page cache entries, which + * used to point to it, have been updated with new folios. + * Otherwise, a parallel folio_try_get() can grab @folio + * and its caller can see stale page cache entries. + */ + folio_ref_unfreeze(folio, 1 + + ((mapping || swap_cache) ? folio_nr_pages(folio) : 0)); + + unlock_page_lruvec(lruvec); + + if (swap_cache) + xa_unlock(&swap_cache->i_pages); + if (mapping) + xas_unlock(&xas); + + local_irq_enable(); + + if (nr_dropped) + shmem_uncharge(mapping->host, nr_dropped); + + remap_page(folio, 1 << order, + !ret && folio_test_anon(folio) ? + RMP_USE_SHARED_ZEROPAGE : + 0); + + /* + * Unlock all after-split folios except the one containing + * @lock_at page. If @folio is not split, it will be kept locked. + */ + for (new_folio =3D folio; new_folio !=3D end_folio; + new_folio =3D next) { + next =3D folio_next(new_folio); + if (new_folio =3D=3D page_folio(lock_at)) + continue; + + folio_unlock(new_folio); + /* + * Subpages may be freed if there wasn't any mapping + * like if add_to_swap() is running on a lru page that + * had its mapping zapped. And freeing these pages + * requires taking the lru_lock so we do the put_page + * of the tail pages after the split is complete. + */ + free_folio_and_swap_cache(new_folio); + } } else { spin_unlock(&ds_queue->split_queue_lock); fail: --=20 2.47.2 From nobody Mon Oct 6 19:09:20 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2DDFB1C5F14 for ; Fri, 18 Jul 2025 02:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805817; cv=fail; b=R4aVpntxfVB3hXNVe/FnYbpHqQ8tvXtdhpwFv/DCIkBA1dOUiVLF3gbst16xOgymea11oaB3ys50b9to2CToeHUqpThV5lXIWpVhnTTsJDOOj9MJ2Z2fYR5KnVRFUVx22Jv63m2veNwCruDCYuXPvowFUPdlEMq9FOiaG84tzCo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805817; c=relaxed/simple; bh=/RPj+VjAsKvhcC0KsLMNimAlAKZ5wv0JS8GnSnumIQY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=XQK9TP1+z4eOlrvi8rkOW9pXS6GiyX5Q6PYDuvTt1PFj/mB12hnZggYt3g6buZOLW5rx7XQV6i2SIqq4W7z7YpE4GgpfW5xU0EKtnGnvjzm8vae/eEkStBr8NM9zfuZxSGx2ZlZzxEQVRrIcwFcs1PZ/1W4yLRcGN4al5VW03w4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=OvpsTCQM; arc=fail smtp.client-ip=40.107.243.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="OvpsTCQM" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ODErQmLWvSefS9yiAQ6khx3cybhMtpn87Gu5mDN2cd15z27OjI/ZCCEqOqvx3KhHFPuCUzBduEnKmYCWqOI/ZNPu4YFmXiGqIvkbU9Z43nrkRwGhsFU7Zabf9vwFwCo4X2ov6FqnvXnzh82VW0t7P9oHlpC2XIxCDXuV1SMy9I9jaV4up7iqm6YdlYSGTl0Si6u3UyJJcHo3zI9lgsSKes74KbkSPMS395KdcVObirJP2QC9iEJC0ffANXKNTK/TX4MU+bnTQDV5hbBYL+pzxbqax4GWltCcE7iWKFKaRsLRYF9+pfmuMgFXPPRnuXCmNVq8bND7tvL/pF2XhX3wVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PpAZTCpYv8UkzzYWrqRdpBMkYGcfQpkVIu4VPemzwSA=; b=VNvh/ylJBNwTrb6zW5QyNG6HIbhP/babdeu9kNUiCpiQNO30dsSrGiLzmLNdA/TeDT7ORbUIpFhJBcCU0nWnk/4laEOTIYdSPqMkMqEnRfpxcUwChVsNr6koFyiPG98BYQ9xmdcrAtCoxXOOOkC4/SkEKOqYBKv/hktGLS55i9fn8L8HNTw0r53vRTuP1Vq34ABSRNFoNWKB2lYGwfJSiuBL/bp7L3xgXSE+2kU+yPpatidQ0uTdpymNakRkRzp7/TPlfAiNQYjGkzOFiXwPUzzVuzsjbGf00AZ2i6Pnjs4vthFWmQbG5u6Lr7gmiPefA9cWyvbpmPC+/ak1Jp+TjQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PpAZTCpYv8UkzzYWrqRdpBMkYGcfQpkVIu4VPemzwSA=; b=OvpsTCQM+97fO/wbIDN6Oy/iIQHtpWjjfMQ1A5eRevgpMcvOVnKTmfQkMI0TURD1Fjii6/ZFyPy02ijRqR7C5oriAAoxEY/HJWIS4aKZJZMaythkijEGlqtetOJQs49jiHqMwZKOUNRQ2gbO2ZQicwbbOcpGt9jqciyC5EYvux0Jd0RYsRtU3k9dJzDhkdSreS+5eHuLa2dH4xs09mfkEAX4XtecVYUR75gtED28dDWfaRgk+l3ejoIwhr/7IL6AbslN0Qz1h5PdSINv9IVon8CMwAz4Y0kFoskQse0pnos4OMKRWolZsVG/rR9Z/DGUSOdYxodCDdz934QwwQrwXw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by SN7PR12MB6766.namprd12.prod.outlook.com (2603:10b6:806:26a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.35; Fri, 18 Jul 2025 02:30:12 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 02:30:12 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v4 2/6] mm/huge_memory: remove after_split label in __split_unmapped_folio(). Date: Thu, 17 Jul 2025 22:29:56 -0400 Message-ID: <20250718023000.4044406-3-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718023000.4044406-1-ziy@nvidia.com> References: <20250718023000.4044406-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR22CA0012.namprd22.prod.outlook.com (2603:10b6:208:238::17) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|SN7PR12MB6766:EE_ X-MS-Office365-Filtering-Correlation-Id: 4a540b10-ded7-4d77-e3da-08ddc5a30570 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?St5mYxgFPH0RUi9v+gh02FSDkD1GU0Py7Cn8K0j2yih7xeFOEOaua0IwwyV+?= =?us-ascii?Q?Kzq3dDgT+PyYQsS6Nl2DK8qD4DW12pwDjdZleDOBndIR+bpZLp7c1dTXjbGD?= =?us-ascii?Q?vkiC3jBTFC1URzw6EaMm3aZdSXsvaStRwzuMSe09vfuE9ePNXBmxDswgN+Ek?= =?us-ascii?Q?dR2Dadc0PfCPd/CCnrjowp6+YE6QZo3mGJcMB2ApAewEijpjrU3Hv/MP9XZE?= =?us-ascii?Q?Wg3JyBIBAFzfhPwbB0eulfaFZM42pH0rdriW1GblYBwufTU97SlO5tAW2Fxm?= =?us-ascii?Q?O3Mr4a+Ff9VODfUapBoxDfAZUeY4rx1z5lMCh8LHhTc65YKJ9tIwifJsz4tX?= =?us-ascii?Q?OfONgdAQKH2kgk2lxK2wvq5W2hRDU1ngJfMXfSXlrSRdwnm8X8Kwk5gV2UmT?= =?us-ascii?Q?d9RGJWFVkNzdcQqKKwawjYvcT3q9BCxuHHdfQ/RNO8YILLNksVofUttLV2Sg?= =?us-ascii?Q?uwyxwnMeeV1eSSUJex/7UsmIpd1b1nHw1GqAeYR+rLmZjqRuzNUkIojr4t8/?= =?us-ascii?Q?Y3coNFHDoz2VzwOLF9IQu/NjMifUUPKB7HhDj8OdirGuJnY6/X/06d+GCerg?= =?us-ascii?Q?lNFC7+qKyWbZDxN03fym5wbsrlm8RUeoY58/lS5N1PCXlv5xq7yHVsDk5BfK?= =?us-ascii?Q?rzPJmE10Adi4rQwtoKRywVIEn1FEnZlKRW91Hu+buG+y1vT05Mo6g/ghuVkS?= =?us-ascii?Q?EEXelncT3ZvgDtdpsHmJr6Vw2l+QoxQHkOU5jPJZu8xeIQvgBehi0A4o0VTB?= =?us-ascii?Q?WHVrDZWk7F6K6dCzqhmkjit0y0+iuS/xc6QnVKNYYTtdaNY7l+qT5Llhx8rv?= =?us-ascii?Q?3sROajCfwM03dPkOHh99OKYDteM8WNa5ifHfyXgRV6PjcrABvEPqGRBKGlVv?= =?us-ascii?Q?n8R5edyt/UgpPH6JbcBCshyxP49/Xb0KfhKRkjrjDyhEOoxrv15TvoYPvJxe?= =?us-ascii?Q?50xQEFknwqAAVXwZ+gwbigNE/7ppj+eyndhtVn1OoG4AB7V25NuaKHMuDMwa?= =?us-ascii?Q?ECZ641sqjEvOrEa+vNsJGv88QIgR79egz+PELwqSjVCQuefkx3ZDfHY5MESI?= =?us-ascii?Q?QwYljnU7ePfgvB46xyYX9p2chMnPBMqp56T4NPSkVN210kHVb6ZvliSSf2Qe?= =?us-ascii?Q?VF2oRWaxviydhZs4p7Zp5UIaIfN0UZkwjVXoN0yx02xlF+Vznk9i9rmWOGxt?= =?us-ascii?Q?wSuR4hUGkg2NwM5A8iHECbjsOdYTw5EeIBHww66SawVZGDPFv2YqecBbrtLI?= =?us-ascii?Q?1X1HwNHkqJS2BJCoJ5fZjTJaZ5lcj8xsvcnsiy4OK4EELXQxrf1fKEQ4CWim?= =?us-ascii?Q?I3XI0lxv+hD/nQ3147aortlOeqSqUeqE1xe2stMzikNdT1YguQ7dLdYZnk8C?= =?us-ascii?Q?DCnaJVA5vS5cQpxyMqQDrcXzGBZyOnfqYrvRW1sYxmkUyyKEgJiMCPYDxy5B?= =?us-ascii?Q?8cmAmbGx4xs=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?U2PMjWHttEXKrOnfZkEIbWLImEaC1KCKRUISzHJcfKnBzbf7NZ0iqCdwR4Vv?= =?us-ascii?Q?/nlKurgVBCIwcMdXsMH0Noc+faPkjPPQhV/RtJM++9jdKGXpnFBe6iczG4CX?= =?us-ascii?Q?VfDK+vVDtbE5EqkLK+0iSSse6iSbB9CKOxTB9dL8qBWslZL5LCNZbxa9Pj7U?= =?us-ascii?Q?J4BlLAEvL0+Tqj2fuwrek6AU7txkKLqsbBFFZQoSdGYiD8ajuS0mjcAvbTnr?= =?us-ascii?Q?P6rkHN8Fh11QLP0210AAnEguNlHRrUULNp5eWWaYPDlZETM+dWg+1q/5WImQ?= =?us-ascii?Q?ULeEORM90G8YqSwZY+0fxBOu33Cyd+6II80EGbyGprmGsHatgLyP1ygrDy0G?= =?us-ascii?Q?9ZYflBzrLro5LDP+8i6PHjsm1FJ6GjQY5AOatnRqrbBAZMhDKfPLXez+BiRn?= =?us-ascii?Q?7I0I/PWMkVA/2YhSEWeLIpjvEYjhw7IoiGzzvYl04OuB8lnwMYrt1WfFcKoo?= =?us-ascii?Q?9Fe1JM5Ntct3XIbl7ST/KsNoHdrqGG1b5F79u8bm9bXLtokmSWJbYOBYNrd1?= =?us-ascii?Q?vYoHfc3B4d7wWE+fExUgkHiShRMWOrEXxvv7Sx4GntoycWhjfOcG0hdJHkTp?= =?us-ascii?Q?1yRSLdb7jkInTbqXESwX0Oc6VuUtCzqXB9n0ZlxxFzS+P45NuR1IomvJ/JpI?= =?us-ascii?Q?aJiEw+aleosEQ/ixFDE56h9vW4V2l98MCV9b1cpYHurUJL1ylb2goVNe9hRe?= =?us-ascii?Q?9mScfUzViKioyjAUxxDHc+7Hesl6PFs7kh5L5ZjGOSwUOIWARPAePRTyzWtb?= =?us-ascii?Q?5PsceriwR7Cp1RwLSioOiAnfbsqAFT47tjYqQRA6EEzK+UtIifhAqsM+nHBB?= =?us-ascii?Q?ifW2ROITEmN00RnOY6V2qXJrVn8qopr3qXOKnFxp2r1vsEPw2AgjVNdx/wQK?= =?us-ascii?Q?rlHqRHPN3m9PBVee97HKmAR6jsjILuM0OFHmPzJyRZQBvFMSnZGDecFw7Fbw?= =?us-ascii?Q?O7lrEPN7OouuUf/wHgX4PfcbXuBZ9x9Dqg4O9N7eH9zTsJtpirrgeO/UA0V3?= =?us-ascii?Q?bDmXY4LZxcl2mtIbWbJhkaY9Bb6lcV6dGQpeE//KO6y1+PpcjL19WV/tBG9B?= =?us-ascii?Q?FBUI8nhMmaZeN6QADKWOVXGXYhXXXh94kvQkmdQWh58pTCkpQGLA6obrhDDl?= =?us-ascii?Q?oPhq6NgDeMZ6Ks7+tF4UncZVCrpK9cKe/cajg+Sr+wMz6rbuclsGtWAGg83f?= =?us-ascii?Q?BSIlk7MHJEdH2jBHyQNDlCtRDr9NKXLggueekmSci9EHNRsTznQPgH7FyIjt?= =?us-ascii?Q?OiAkU5oKYo6s7bxNiDm1m59B8ow7/ssp3zdgfgmbYQ/kCjhfqudotF3TEFZC?= =?us-ascii?Q?+6KvAHjBfM+uabRzo9khQPxk49ua4CS8TZHLhTFx5URyYWu81J7obxPkB2Xi?= =?us-ascii?Q?z7D+QQlbnWIl48u0qa7zOgdj6onEdFKhh0mgiwTLPafuLm1SwL836ZZ9wTDr?= =?us-ascii?Q?2Rqc3mgrI+wFxRlfanNzDSGmYh+L8bll34c+d0K9zJLj91GZ9ifJUzapDkd+?= =?us-ascii?Q?WCrJhtXjVSOyqLA1/unTLXiN+QMm1u0MN60Padeyc/vrBbRPZ9CjD1XkKddE?= =?us-ascii?Q?votXkMjwa5hJkHizPzRihIkChW80DCsG24jA5oXw?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4a540b10-ded7-4d77-e3da-08ddc5a30570 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 02:30:11.9578 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pBFPJ0g8KIi8aKRGWr2tvj0D9VGjjNVd2PSN/9YolYHUWgR6dLQm9lYBkAiR8Urh X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6766 Content-Type: text/plain; charset="utf-8" Checking stop_split instead to avoid the goto statement. Signed-off-by: Zi Yan Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 63eebca07628..e01359008b13 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3463,18 +3463,18 @@ static int __split_unmapped_folio(struct folio *fol= io, int new_order, if (xas_error(xas)) { ret =3D xas_error(xas); stop_split =3D true; - goto after_split; } } } =20 - folio_split_memcg_refs(folio, old_order, split_order); - split_page_owner(&folio->page, old_order, split_order); - pgalloc_tag_split(folio, old_order, split_order); + if (!stop_split) { + folio_split_memcg_refs(folio, old_order, split_order); + split_page_owner(&folio->page, old_order, split_order); + pgalloc_tag_split(folio, old_order, split_order); =20 - __split_folio_to_order(folio, old_order, split_order); + __split_folio_to_order(folio, old_order, split_order); + } =20 -after_split: /* * Iterate through after-split folios and update folio stats. * But in buddy allocator like split, the folio --=20 2.47.2 From nobody Mon Oct 6 19:09:20 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA8631CAA6D for ; Fri, 18 Jul 2025 02:30:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805819; cv=fail; b=sCDHLTDbEv3ii42gcwAnD0SzPYnL0dN1DkRVPx8SalM7U4+bqA/u6k9YVcgXd9BhvSJW4iNaP5whKeTE/4gJ7zwQDqHBi6IS9x/mMFDxpc1unNHr7iFKmPIRmv1r+DcjHU/5Or8XHCMYFKrs+gu0s6Q6uawUG1Yx0BSjPIoPqJg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805819; c=relaxed/simple; bh=QLisbG7f/IEM903OnTG7gX3EDa1+hlyNWVbgn9YngHo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Sk11NOIp4SH6tuaMGUiBB+Kvl+eoOGFfFfayhKMz+7ssYTlaZp5aybyrvNVs0sLaKVlLqZPIPFOvmpCuVDUni8h7aLZLJZ07j8g/GhYKq0W/2/i5tqWRsKp5cZiYrz7iJls97guJkS69KalqDD1wb86ojsA2PfDvhLak8AMxEyk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=BufeaRM3; arc=fail smtp.client-ip=40.107.243.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="BufeaRM3" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=K6V9OtsxMy13MeWy6HanT7enn65xD37u4JiCD5wCPm5wN3+dxvK+97K7zlv7vXQyEC3DCwgG2AwcbpB5XTmv9WtO1kG4wprguAvni8np6MJwWKHVljKm1ccZ4nDBixmpJ+C8qAXdYdX9wPXmGJoIWZxAssKaHZ6xSAK8Djw3mmGQyTNY15PVYAjw1sEv+xw1v1cgWc0K5eRscHDEi2ZF1sggEPzgKTPeuS5dLEcSNLzrAwibJ7mwjGjt19mmsYJcyDyFsk2zL7gQqfR1qYmAKGqjU/4yRUWZe0W4nnnZ2g00FMeQEr0TYAauN5hzX9kC9pIzdMjJkyeh/8azERgpLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TzFgC5lkrzXX8HmuEsW6EyeoXTEH5WAMP9CM6DwhNlI=; b=skWqOg53LYfcLaYZB0aBaz8RQyhHBP8Y8eeuSIPxZZBDt53naHhTey9Hlo+R7Ilkqu9SSaoLMCL3khjz5wgBZE68eEFzlEOaksmZdjkNQWXTN6zAcHsL6VDaSwAKJfFJh3FPzY09KY8VqHjQdSdnXE+fHsmftBo57gxHZbTxWg1x9CBBV/cx/p7JB1g5UE/JPJMjBvjItlSPZUP4UUtU/CLyhYBPwtFFM9iB7vGPieKzV3tnT3Zk1l6au1UfMVbE6TvWydYyXPkLDrMPV/o58d/Li8f4nhxKppCbJeh6TWofbJMvm1RzSZAtaE/5F9dZR4dP7YY+rkLaqPqq6gOkcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TzFgC5lkrzXX8HmuEsW6EyeoXTEH5WAMP9CM6DwhNlI=; b=BufeaRM362pKTZinsJ52VaagHRNmJC0DqP30HGqdQGDuvKL+Cm+w/Yr1skkmd/D5fxryAevpTxSovlL4iGPYRqPXKWP1KylQAS0Csb9NnP0vLFR+SIuU6+fwIcmZm3mxl1KLEuVG+e2Zgfw2FbKWRfFy3wPnNlX7+wETf40u/IzmlIgSXBfzomI/AVGj02UdBYIp3ssQstffXSLqRK/0ap/fCf7M4ZlOWZbBIRbClAdsHaUVTXj0MkbPSI7rh7w2wWloV6TTpRTdZ2CokTRGO92weg407V4XPyWrFavoeJK8tNh6nWQ2ML7NtBa/qL9Jh7SBvfUirqEZaqFzP/O4UQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by SN7PR12MB6766.namprd12.prod.outlook.com (2603:10b6:806:26a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.35; Fri, 18 Jul 2025 02:30:13 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 02:30:13 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v4 3/6] mm/huge_memory: deduplicate code in __folio_split(). Date: Thu, 17 Jul 2025 22:29:57 -0400 Message-ID: <20250718023000.4044406-4-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718023000.4044406-1-ziy@nvidia.com> References: <20250718023000.4044406-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR22CA0005.namprd22.prod.outlook.com (2603:10b6:208:238::10) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|SN7PR12MB6766:EE_ X-MS-Office365-Filtering-Correlation-Id: 2eae94e3-e0d8-4a4d-4dec-08ddc5a30641 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ms8T+julsVBtK3pR8i4q/NxG0404hbESl+MgMvSz69LRrcv6knJkaja3C8nh?= =?us-ascii?Q?qLQIEAyJNH+y+HyB79sftvukD5zV3MQQyXgVVV13Pr2IjsjudWIisP2FN9c8?= =?us-ascii?Q?GGGNQJYOb5GdHt1M21aVCi/or6sZfnnDRFM6tZ9zZ2H3ajxjgPTnB0AaF9T2?= =?us-ascii?Q?KjcYycIcb1r8VFhBG/oLW1u06No6+wGIUCcpY16pQzf506v5+SLZpllTkQAa?= =?us-ascii?Q?MatpsYnaPK/+dqIpRJ2WZirIUS4Lb5QCuPUvLCynQuJyu7FMVruq62X3oSg4?= =?us-ascii?Q?NO4BytgkyB8hcKHiZcJIPkOkUGUCgudX10ofm3d+QJ+UAxrVt9Qy+mtoKPeE?= =?us-ascii?Q?DJ1EKRpc/iQ9YSgz2WvU4ZugJM+IKdmLp/De4mHR/Uwfqjh1tx8JedDjprh/?= =?us-ascii?Q?e+z82GI9M4KSfwbf9oCYJWBHOGcg0hU0F7MCSOEPra1nKU66Rf+xSHEHNSKu?= =?us-ascii?Q?9ALCzXkQs30r/oeSTNAoid6bLop20cPb5/HqoOGYJGJlDhhAVu1pblgyLDQW?= =?us-ascii?Q?9KvWPKDEOmGwRb1nLXLZUb/4V3rsvHyt1pLYpek04dCC3U0mspeqe+ssA5oD?= =?us-ascii?Q?TeCOCgqlPUTn4YxsJxwLsEGF/FmoBHWqFBdwFD/Q7rIHbLnHuiduOocvx7rV?= =?us-ascii?Q?Ca8MFJbsXOUWFQMo7kidSqqLVympTmPnJqSxxMVt7auSYrESIiiUOGnTijTd?= =?us-ascii?Q?VuQ0Z9DTzn/8T/d108lFPRJkWtCvBmFNRKrveb9oCfbTQakB3sxxi1ijute1?= =?us-ascii?Q?Bxhv+VmE0JIilWg4ppicNcHgGPr8KqASm2NKMDE/Mr+QiJDV5UAx7DRnfwKq?= =?us-ascii?Q?EEXi3Wqu4AvCq77lyVffAJZv1mxXXVoasFbjhME74xADMv4bbIW8q/nk3piB?= =?us-ascii?Q?snrgrucMC5rJ811DcrK55PzOfj18v/GfoSW7/sRMXEWzUQGH3RbtD882VeQn?= =?us-ascii?Q?+m5iBLJZzckdWIxS3tPTsvzazvK90vPnRj46wxH8d/pkbPzi9jCA7WeYcp+P?= =?us-ascii?Q?e6UOkshQ8u2UB2unSkzqeRlxd4T6RHMt7X9VNo3j3qafXreXDrzBCZyhtXqA?= =?us-ascii?Q?FH/keqYEN7vOskCkcL+zpjYzWJ/q7Lad7Vexb2Wy0emgQf8cmPtHJpil6zVd?= =?us-ascii?Q?OnU4lD1JSE2k69DttYpj/TOI1+K/xZxhbLzbo8UYGuQnwE5pQ3x5dAWt+LaJ?= =?us-ascii?Q?dTUfTy2Lx7itU1RphgPZg+VPsWGj4+bW2PuY/cZn/OA8cg4rPJ34iBfqpzSt?= =?us-ascii?Q?85JYUCmV7+1XYjo5Ore0YA4YTDOWHLGC4Oz/PRQ4SdBLjfnn1Rle5Z0UL6+f?= =?us-ascii?Q?6AVRYAkeQT9fjiLn4hJqiAHKejTVyILk/6r+qPkhyDyUuoUfMUD1LcdAV/LB?= =?us-ascii?Q?a3Yd2UFBg7Lv/hWAvCcnkh/NKg6oqcWwS27noLy0GQHaLreZ0onK/lkC+4x+?= =?us-ascii?Q?EpAyec7ITlM=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BkrfSLMQ0jxPxADDS+dFmHbPDvMC6gGhFmbwTREqTes7vNQ5azhJDZGLXCFl?= =?us-ascii?Q?fKLdP4FPDTkZQO18LN1uOSzTJSx1JZpsCNwySnuaF90C1KOZMdDbOTBLyFZ6?= =?us-ascii?Q?hmIc899v/Oviynh2+EzJe9Ki88ALR/NfEiSxkkd5wOmDrhczviuY+Wq6eafq?= =?us-ascii?Q?6JlRqyeVXqIWfl4apQBYjiA5R87txphAyCjBBS+HFoKOvTnWkQcjo0vlK0bo?= =?us-ascii?Q?im5Evee5mgsbhvnBsVYTMfN3cLhrLFXr4lGsc4BS8jFp3itR5RyD+ehFliNy?= =?us-ascii?Q?JNEbe0Cf5essRQbSJvUWylxQ9KWQ2cIwB3SpkHMhH/lKtqT4GmqpF2VEl3Sh?= =?us-ascii?Q?k1xBPTQSNoA3485LjKcCiQQiB1ZcEk7QUkKA3zMC0RLfuDntNRQ8pPdFPfoO?= =?us-ascii?Q?N6+h237wCxFFadZe+iS+U1af1i3iuIkufPUM3NI3x7JxOXOPHqVX2HoyO8Ka?= =?us-ascii?Q?Yd7ZqkZyWXlYsGHlWoVy744YvzhwTS7OLbc4GJmxF1KtApHf7FGt+N9hpvtr?= =?us-ascii?Q?ToClWuuZG7jJjGatM15gRGyvf+jl7REnua1rZO5NjqWDFXubPfZnXHrUBsQl?= =?us-ascii?Q?Odu7MwI5rAuYiIk0qPAEMvL+TgedQBr8KbkTg9NdG7EYqaN4RVYH7kWTURx3?= =?us-ascii?Q?X97hdMKjH9gnsEtrTdwW6jO729vCEwy0jfs2p9e+XQYihRY7/3YSKNSwHnzK?= =?us-ascii?Q?LBa4DPTfHHOyr4IBKyLFMx/754GbpqXqY7e60S+/wTYWZWXDLsTXjht8PhWl?= =?us-ascii?Q?eAi1IaR/lRzZRCkwH5NMSRIMQ/b+YqeDjlJb93p0TvKZDnc4Dt/9tZVHsJO+?= =?us-ascii?Q?DR+ARMvOw9hznQWy/s4dDkFU29/Bg+fldKoa+pTrUVQ8mEhELE4C07rv+UPG?= =?us-ascii?Q?OKTWDkOURLOyS1yWKe1o6KNdUBIdTA+SYx268n7Y0szEcicoR0Ymm6C/Sc2d?= =?us-ascii?Q?zNNoSQFOjsrso7LCUKAO7Xjrat9OgMgqeHJ9bqL9S+hL3zbe14uB6HcDjBVR?= =?us-ascii?Q?LfrGBJhAHG4qlDdDVKKkENuJeU0sPg1cBzVKZN+ipDYnlRjQVjvcUqWfRLBt?= =?us-ascii?Q?1I9i1M48e6UGpqNuCbsKPbAc5WwFLWI/9nPon9kkWhR4IX92C16fBiPeEngI?= =?us-ascii?Q?wCCGHUKyH4utEAVMQw2mZ1+1yTOdnTSawo8LyKbw2kbShSICgjqBFPQwwRco?= =?us-ascii?Q?t/Lny9sztu+V1VpyRPxqfJHh+NEu7cFjCJdvk6Pr7Z44lCDgKGXjbueOGBNx?= =?us-ascii?Q?wb2dp1qqaqF9yncE1ZSfbKGH1PcXfWuCv4RMhSIqLPrZIve23SYwI+TD9O31?= =?us-ascii?Q?JOY36dYaNnYFUbYycDVyYcvn69kCvgowVT8sxuy7eEw0gZ2uweqagyUFtp5L?= =?us-ascii?Q?X3Iwz3RNOSUVnNI5Ck+D4r9v5nzCfp/ib0mANq9vpgr16Ab8b1DPbvb/AabJ?= =?us-ascii?Q?A+IaC3nnePx9rHzSAooj7RQO47E1N4gg8LoKWlO9fc+q0IUEsqSWQ2+9Bb+W?= =?us-ascii?Q?h4AYTI9FnoavUGpN6UkPiRAm3IiIP/gSeB3MAtaW0xVin3ceUs+pRdVFVoTM?= =?us-ascii?Q?2mgYmkLwfxdQ1Bx+yuB0g06QV0bb76FhVrx1usH0?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2eae94e3-e0d8-4a4d-4dec-08ddc5a30641 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 02:30:13.3256 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Q1bT+5eHBisbbjm4T/oWLLaD7B05A/VyY9zM85Xm4L1OvA1ehK8blhOOFdG7Ad2B X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6766 Content-Type: text/plain; charset="utf-8" xas unlock, remap_page(), local_irq_enable() are moved out of if branches to deduplicate the code. While at it, add remap_flags to clean up remap_page() call site. nr_dropped is renamed to nr_shmem_dropped, as it becomes a variable at __folio_split() scope. Signed-off-by: Zi Yan Acked-by: David Hildenbrand --- mm/huge_memory.c | 73 +++++++++++++++++++++++------------------------- 1 file changed, 35 insertions(+), 38 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e01359008b13..d36f7bdaeb38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3595,6 +3595,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, struct anon_vma *anon_vma =3D NULL; int order =3D folio_order(folio); struct folio *new_folio, *next; + int nr_shmem_dropped =3D 0; + int remap_flags =3D 0; int extra_pins, ret; pgoff_t end; bool is_hzp; @@ -3718,15 +3720,16 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, */ xas_lock(&xas); xas_reset(&xas); - if (xas_load(&xas) !=3D folio) + if (xas_load(&xas) !=3D folio) { + ret =3D -EAGAIN; goto fail; + } } =20 /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); if (folio_ref_freeze(folio, 1 + extra_pins)) { struct address_space *swap_cache =3D NULL; - int nr_dropped =3D 0; struct lruvec *lruvec; =20 if (folio_order(folio) > 1 && @@ -3798,7 +3801,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, /* Some pages can be beyond EOF: drop them from cache */ if (new_folio->index >=3D end) { if (shmem_mapping(mapping)) - nr_dropped +=3D folio_nr_pages(new_folio); + nr_shmem_dropped +=3D folio_nr_pages(new_folio); else if (folio_test_clear_dirty(new_folio)) folio_account_cleaned( new_folio, @@ -3828,47 +3831,41 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, =20 if (swap_cache) xa_unlock(&swap_cache->i_pages); - if (mapping) - xas_unlock(&xas); + } else { + spin_unlock(&ds_queue->split_queue_lock); + ret =3D -EAGAIN; + } +fail: + if (mapping) + xas_unlock(&xas); + + local_irq_enable(); =20 - local_irq_enable(); + if (nr_shmem_dropped) + shmem_uncharge(mapping->host, nr_shmem_dropped); =20 - if (nr_dropped) - shmem_uncharge(mapping->host, nr_dropped); + if (!ret && is_anon) + remap_flags =3D RMP_USE_SHARED_ZEROPAGE; + remap_page(folio, 1 << order, remap_flags); =20 - remap_page(folio, 1 << order, - !ret && folio_test_anon(folio) ? - RMP_USE_SHARED_ZEROPAGE : - 0); + /* + * Unlock all after-split folios except the one containing + * @lock_at page. If @folio is not split, it will be kept locked. + */ + for (new_folio =3D folio; new_folio !=3D end_folio; new_folio =3D next) { + next =3D folio_next(new_folio); + if (new_folio =3D=3D page_folio(lock_at)) + continue; =20 + folio_unlock(new_folio); /* - * Unlock all after-split folios except the one containing - * @lock_at page. If @folio is not split, it will be kept locked. + * Subpages may be freed if there wasn't any mapping + * like if add_to_swap() is running on a lru page that + * had its mapping zapped. And freeing these pages + * requires taking the lru_lock so we do the put_page + * of the tail pages after the split is complete. */ - for (new_folio =3D folio; new_folio !=3D end_folio; - new_folio =3D next) { - next =3D folio_next(new_folio); - if (new_folio =3D=3D page_folio(lock_at)) - continue; - - folio_unlock(new_folio); - /* - * Subpages may be freed if there wasn't any mapping - * like if add_to_swap() is running on a lru page that - * had its mapping zapped. And freeing these pages - * requires taking the lru_lock so we do the put_page - * of the tail pages after the split is complete. - */ - free_folio_and_swap_cache(new_folio); - } - } else { - spin_unlock(&ds_queue->split_queue_lock); -fail: - if (mapping) - xas_unlock(&xas); - local_irq_enable(); - remap_page(folio, folio_nr_pages(folio), 0); - ret =3D -EAGAIN; + free_folio_and_swap_cache(new_folio); } =20 out_unlock: --=20 2.47.2 From nobody Mon Oct 6 19:09:20 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 850EF1D5CD1 for ; Fri, 18 Jul 2025 02:30:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805820; cv=fail; b=VWEvfwab67jUQqX400kCequ54pa4hRvXvbdCIaltllrEjO9KxaoxDbagD5mNPQ3yJ96xpjFYg+sHr+SdeAaGgjjVJL05DslUy/d1q9YbL2huYe9UdCoxKSJaPNqJ9gfsG4CKmxvaA2zDsaRdZkDdCqj6URAVLdAD5gXLG1wRJLM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805820; c=relaxed/simple; bh=ZAW5VIPQqh5H9dr2HY6waz2TTPeS1pq4vRCWakAVAXc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=PwKbEXKEu6KzkKww22BkRm2CWRAwunCgwlbGLuqBs+VDB5HZLpyUXZhbPU/7V0QIeMB80qBMCfrPrpbS2GcWsCkICcuNpTngrX0rI0D/Vlz2kYrzJGeldiLEcdAup7t8qCVd6E56FfaKeqsLTLiBa+wnoW91bqgbLDEK4T2oXro= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=An4UEKgX; arc=fail smtp.client-ip=40.107.243.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="An4UEKgX" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=w6w7De75Z7iYljLrLR/RHWrcWceXRQlWI7BCM2i4pzbW6q0ZE9u9gYkUniCGVZwSNxLfWHXTQ9dKtdha69jueFVhBxUQnx9MVMtrwFjNCThx/EbjtVpRj+j5HVaRFHllscgbHmWQQaY4fb28zg+O/bxqsZ8bCYzsqpm0q/97pITtkurG1JohSqe+I+ogqaTr8Dous1ADjfjb1Q+SaK1ETfKGE+EP8V6NNFRxJ6Yne/fOHAj+IwxGqD2VrhEfkpTvyWZ3SucrJZU+cFRIko3lhJf+i2JSCAaasCuDs+HTPJPjnVMCsyTOIOUR84NPRLmTXZzjhJ2+MqbX37vwZHzIYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=05nwIUeb0py2hT/mK+lSNFkh29LCuwDfA7U2QUefuDI=; b=DdyBf+F5DMKJCNW/dlsZs1aVg8KdD+B+ffygQpLHVLigIi0m+MffQDZypA6h7I+TGm6AXCI0tMyQDUdDbcKiyKliKuFtL85cd2fQY3cLlMoM+XYpMqKbVgNmW1V2eYB75Qnc+qV4yf+LdwtTUjKqT3QXlug7VFHDdbYC9ZyA8HYAEifnSioz+CEr9jxqziauktxMp7JxhNfUIfa/CYAsuzNoEoBUfvPbCqabbw7UMFnpPwsosQcHXrWPppFmvaGcinf81rA0iMsN/ZYWIOTddZn3emd3LOuhDoLZpHKQUyaw8jeUfZcB3FrdcnKyeRjyYAV2OMlUhwEGK1T8K3NhXg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=05nwIUeb0py2hT/mK+lSNFkh29LCuwDfA7U2QUefuDI=; b=An4UEKgXA3HJUhB69MVcSgZhhXUdSnsnkvW4Afdmv48LH+3CD9u7hP9CtplbAzsg1/fi18lqvT2L33yx47RwOqTeN/Jg9URpxif3U8+pkDzqiFL+c7/WaOC0r2tHL4lvyEs+BenSRKF+C6EO9wZd+HPMhdk3FryfpY37VORdjQepeayZ9EyQDeRVCS+CjgRYrHP/HhaKb2bDX0TtBqtMnSVdrN3rCkURNTf47veoNObR2ePKcMuyvwPv1t9RYUHpRLuK8Ovj/CaJb9J04SuTIEv4pW4Sw/KQvjLpIHJwz9UKonJ4eZEI7nIPFyXWH+aIvXnrf98SUzJf9v8y01LE7Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by SN7PR12MB6766.namprd12.prod.outlook.com (2603:10b6:806:26a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.35; Fri, 18 Jul 2025 02:30:14 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 02:30:14 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v4 4/6] mm/huge_memory: convert VM_BUG* to VM_WARN* in __folio_split. Date: Thu, 17 Jul 2025 22:29:58 -0400 Message-ID: <20250718023000.4044406-5-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718023000.4044406-1-ziy@nvidia.com> References: <20250718023000.4044406-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN0P223CA0029.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:52b::18) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|SN7PR12MB6766:EE_ X-MS-Office365-Filtering-Correlation-Id: 5511ae66-bf90-4d9e-7adc-08ddc5a30731 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?WiiCkXnoCQVoLG8xtsIf/qZVNkvhtSzrHnaoqHAlWRNZRawLGTyhI1/U0UE2?= =?us-ascii?Q?6J6Y8Yc18fTOCc/U4pwdFbDjvybHn5QH5fHx216NQ5LUhrWdk45YPeVeHezg?= =?us-ascii?Q?O3HlFrChR+ECVxSI2mKGBH+7r598DWCntQBdNaK5PuOAdkPbkE0fHfzSpnV0?= =?us-ascii?Q?k0fgZwWRwVb3RivGDfuxh3KKOKweqnBWpVGpjAc+UIdxvp4gzPVqAjlAAOgy?= =?us-ascii?Q?+T6vIFFRXEiW/c7eEMmcXTMWOLYW5+yR1IumC/pEvrQFBWwcowPkkuRFILCi?= =?us-ascii?Q?87fjckBTPeJKRaZiUzC3o1ust2844QXydziHrrMJDmJ9cyl8ovRXneTMGLjH?= =?us-ascii?Q?0EDIwZETdLXq3+qeNKoD0g3Quq7oi/eXTYAKheMeQtYUxc+oEbd+LjOKIZxz?= =?us-ascii?Q?rV6IXlMcVO2f032iyUruZ6Btd4Gdz7DxkVq2vPQSDessTTcWBumAYf79aI3T?= =?us-ascii?Q?kYzNSXBG1GDrox/d1dBIB+NOuVltQ6MzRD2NuhVY3ugNkW7bF77UJZhaTO1w?= =?us-ascii?Q?CrBwxkbmptM7uwb0298UI6Iw3MNl4ClWGS9fwRV8+H5V0waA5DboHv2VnVWJ?= =?us-ascii?Q?XEKHTrNIhp4dUSk2ORv/GnTKXA/TvGqv4R5d0LVJisSUJqTm2zUa0WrQQnmb?= =?us-ascii?Q?bXS/J4MYZBRbAgzptmXxu7Bn6wbIk5IN/mcYsOpaWofEURme9RY5qu4ePG80?= =?us-ascii?Q?5eMAqRPLntKuCWxUi08SrA5kcEHudILetuXr3fP4/Y+g+jIBTzw8cozJXD41?= =?us-ascii?Q?LL+7oCUDXMm3yxY2EzjEWT5nsdG2FTlGGhqruIZDvGSMPxzXT6qOD3zVyAMr?= =?us-ascii?Q?571bgLU9oP2NNG4VhAi6tf/5r7Vo+RNpr5M6Xpjm40QCelb5H/3w+ljuTxaB?= =?us-ascii?Q?I4sBZJ9hYZRkM18Cf1aA5GvlSi1/f3acGlN+Sfggyex+kkQyxLqzNOrh61Sb?= =?us-ascii?Q?T5fsvEvFIuKn356v/Gu/iW0OXw3svIKK3DF/RxFOezeBmOwWfc3Z9Uombs2f?= =?us-ascii?Q?p897H00sNtDZrZNrnbh89nOklDu/t9b2DWZhi8c0tK6Z6wx/d8y2c50xgoot?= =?us-ascii?Q?MX8u3QT88YA3we0ggphK3G2eNkW9Mxvw4pS/9SA2b9vCPkFMwmmYkoNWmLVt?= =?us-ascii?Q?91A3jaY0qFXfuiRrNXFj79pkTOd4FGA2Kc4kmug2yql+S+HndhHICnlPo4oQ?= =?us-ascii?Q?J6IR+S61GlFj2eWcrOttkcNSbmSOD/2Pw34DJcHCEfF8JRhxM0fAVmWUBPUm?= =?us-ascii?Q?ntEvs5NGenaYYWGxihzCBy5CrMV2xn0wZMbCZ9zdfwooqc6euvJV12KqdWCr?= =?us-ascii?Q?6k3fqsuHyVil2PLyYYMcZX79+U36/3sxJJJo2Sb8QAcsx+lzQFbFUpANXibC?= =?us-ascii?Q?2c9UwwiG+lhqcS6e7r6SfoUb44A/9d+lUDx2vdR8FhvXJkmj5dz5EiPkSA7n?= =?us-ascii?Q?py9uZ97JeX0=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YMd6mbWW/RH0VH0GkcinDnswY3Z+Eofc/mIMKaUdLq6QpefwlhZixsFv9tp/?= =?us-ascii?Q?MTV4G3jkz3YKfnVtVHaBptMKe1b+yolHVUIbxGrBmEIgr9odigcKc39TEtYx?= =?us-ascii?Q?sbPfO3Nc92I0OHY3yqPCIKqKKT9E0U9l6qAxwGpL3Vg9ZpIrimbxLQcGMpDO?= =?us-ascii?Q?1aNWWHwM/0J3MmcRzJokVspl0Dgb3OFxFT+52PPZpV/J5NldDr4AiO8XhUB4?= =?us-ascii?Q?C5Fwr8oPmewevzMXDaV2jFoQxcmF59uIo9okwHjkZq/jdmrUGjK7QrxsIi2I?= =?us-ascii?Q?BOI6WwIgpOj8nigBnh/PfpaS6pALTscd0nsIW5PykGZqbR7vJLYj/BSn9Jvn?= =?us-ascii?Q?AdWrdHxf0ZeIMJUp/0Mu2gjImLWxBbJuEchtLOw2cUXmPwV4MlR03zJ0rY4K?= =?us-ascii?Q?MRF4UtMGwYAQTHF0kEIoP6RuOnC30hS5FxVa50PZxqrcPbDRcQttX3xzatWU?= =?us-ascii?Q?bDSiZrk4CVyaaTokXle1N4NNrFZJIRRPh8Ybqb+yxuC5jqEq0xkVT0IRZVtT?= =?us-ascii?Q?0sS/DtQshv1K4oayKn9s1r9gBwAw8QBcwzfDRR/OC7Gw6xGrbMR/okqX849d?= =?us-ascii?Q?cGL/O2hZ/K6+j0fj4u2OymIbmZoZq2Io91m1aIwsxLZ/61s7wU61uAEy0kKd?= =?us-ascii?Q?s81z1l7dGQhpIh1jv402xR/po2qg4S6c9knMo8ydlYirIanYxHSDRpuKWKKO?= =?us-ascii?Q?FR+iHJK82v899uZ449r4y+d2dwmarcudn4/UVwVapsxaEkJEkPwcPrcxd11z?= =?us-ascii?Q?MjKdFa8rPHLodDqzUJ00FOkbpfVUH+BYPS/SPZ3BX/padxBih6x6TsFCvkEi?= =?us-ascii?Q?A1hHJLO5l0beXnDJ9aeo9rDjtSfGoVEhHp2t0VQnLXwAXngCagCZUvTT5zwo?= =?us-ascii?Q?jSF7uZmvlN1RXtlEWU7VKkJEgjwfYqi67Doof+Jqv3mkVHqowBO8WXPZwTG+?= =?us-ascii?Q?6FxXmqx4CML6RuG+I7J6XpKxZbNpteHgi0v3ltPuoZGytpAK6GPyL3fVnxM7?= =?us-ascii?Q?p7AizkBJ8HR4YukiVxNjiKf6HfmchPOBl4O7JR9bmDYvR7oo77EHlzblb4K6?= =?us-ascii?Q?OKNeJB7ylAyWM1aB6chwc4TB+6D12ybYBnVAUHRvIuPh3upugHhyrcrdPSaV?= =?us-ascii?Q?5wUwmRrLnKcYFODMYLTFn/f9DDeMJs4zU5PBSG/qDpDsmgFK8couB453aUOT?= =?us-ascii?Q?msaqg4NTu7ecIEfUBwTGB1L7S7JgfFEoh3Dj9/xImUhejfruLSfytYxteh6V?= =?us-ascii?Q?NKFknZmS7gOJ80BXJHlNgOgHYrXkt9bARa4l297Qm5fzthA04S/mJIUjUB1u?= =?us-ascii?Q?LyZAXogqiYH0Igznf/H2H5e5iPXFH7/y2eL/CY3HNtvyDbqL7FFNAOJyShGq?= =?us-ascii?Q?w/xzM67IgG3qu7iY7V6hI8MnpVwbue1iO0aeeQqjJS8YVHHlxbYSY+eeWaqM?= =?us-ascii?Q?E+WYDXAGpuEe1QVTppyaIXJYdf+zisLXvlvwRFcKxTQm0jgHh4Zw57Su46aI?= =?us-ascii?Q?rwsf/dKREeuErsalHUW3Q1AIer5qX/RWP/5pgr97l+FZ7PkvGngHxb1obgHv?= =?us-ascii?Q?SN04yhp+hLr2a96rPojRvjDo7Y/vtQh2e/GLe3MD?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5511ae66-bf90-4d9e-7adc-08ddc5a30731 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 02:30:14.8646 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8FZsRTATIgOaPRufKAvW+N+ErxvGr49A1L2ppT/1MRTS5NjguWbsOawIEC4DziQ2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6766 Content-Type: text/plain; charset="utf-8" These VM_BUG* can be handled gracefully without crashing kernel. Signed-off-by: Zi Yan Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d36f7bdaeb38..d6ff5e8c89d7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3601,8 +3601,14 @@ static int __folio_split(struct folio *folio, unsign= ed int new_order, pgoff_t end; bool is_hzp; =20 - VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + if (!folio_test_locked(folio)) { + VM_WARN_ON_ONCE_FOLIO(1, folio); + return -EINVAL; + } + if (!folio_test_large(folio)) { + VM_WARN_ON_ONCE_FOLIO(1, folio); + return -EINVAL; + } =20 if (folio !=3D page_folio(split_at) || folio !=3D page_folio(lock_at)) return -EINVAL; @@ -3766,7 +3772,11 @@ static int __folio_split(struct folio *folio, unsign= ed int new_order, } =20 if (folio_test_swapcache(folio)) { - VM_BUG_ON(mapping); + if (mapping) { + VM_WARN_ON_ONCE_FOLIO(mapping, folio); + ret =3D -EINVAL; + goto fail; + } =20 swap_cache =3D swap_address_space(folio->swap); xa_lock(&swap_cache->i_pages); --=20 2.47.2 From nobody Mon Oct 6 19:09:20 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15AA21DB54C for ; Fri, 18 Jul 2025 02:30:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805822; cv=fail; b=DnMJ95PtN6bgkJG9SVnErOIeq5dFK9LZC9xwQ3GWu6CU59tM6Per31jlTQcksaztkgIDiQ3EM9INyGixT0E9uzYLIFr/8JGVEqVULVDfvtHAakkkK4oY1EyEOqVrHVdVewE5sy6zT8HhcDkNyOTl5fuAy9D0FI+YBL0vkmYkYcI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805822; c=relaxed/simple; bh=aDVjL6HmjfVmYts9JIEDa+GF1jyAcfZDulZKqhArZtw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=i+EU5UGeshFJ2H9w+iT4ugfCgC741Quu1rK86RQSH6KTwl8yfB5Y2cZjz1vOO3vSUUv6vO70o38vrF/cQJs1XwRksVFeVpFp/sETsmZT+VlZYyr58C2cPCingtexCsvOd/krUB9q1qRhgG/qWUB4+Ba3OqDTxCwD56DT0EYfuqk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=scjscGpf; arc=fail smtp.client-ip=40.107.243.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="scjscGpf" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=sA62JYxi/PIKYM3OZadRwcbg/ZUbgEXLeLew6p5PsrKFgSZlW3paWfsHR1lYMgZP5Iz/+ZaRiTPI5jPOiD4twyAAfjsi7iPff1MXgk1nhZthvKR5lu6GSUtxCILJfuZhlCk7prGmNOXlRJc/ytKgBpBSWZhMXvefEcaokBa945UCVbuUs4U9kP/paYm8cUj/j8/gIcpraPqpLJWJ3nb97pNcDk/i8SYFJLKno87l7M7YXRREqZryHgxoRPuoesb2GEP0qs7CvRHQOhTckTHcDecvYU19Yvv/brddgFLLQ+4FT2lukljjVsLfAFUA0M7S565ul/jELMFkvePyy7T1kg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=StO/GyGqr0tgAmqrJOOwBQstPMqGzWFSG01C/Em4QKE=; b=YaIScnL4G+swTANxJIW0lT+658u+bYjyAoP9JuGYDMlkIJWNRg5YoXX/LOWa0WQ155yv74c7GHgqNsQba+McFYEMjW5rTC39EZ59quhkRmM8nOg8BrBOza7BlxMrCofMG4hvmeEiNL1SobJL2yY+JPsw0o6HiMGuH16Lq89ZQgv9GumiNsSZiT4G3XBKvlJvN74o4hFXLnTQzVJX3yhmn0qU//nZdUl40tjG2i/70pGy64yUcmr8tqsVAbZz2RKNBmM4hD8UKRxojSZluE0COg73hDPPxb160f4iKxFh5BNAAwszF8BXK0TOINv9tI8OmfgPNc+3bXB7im+1Xh40yg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=StO/GyGqr0tgAmqrJOOwBQstPMqGzWFSG01C/Em4QKE=; b=scjscGpfxr0EcShSDxnPTfxvybXkanEv1GeJlhZ2imLirejNJStN/qwVThsUmBgdtZ4AL/dlSg/thyB5ncrU4g2Hh4M4GAQSdMvfX8XkG9YEkMKABMgGMtse7Ym4GIHh2nBf7Y98aGwTEWjiJjkN9hEa+msQRWiQxIaGt8rRS2n5VBLdgBdiiFo0+nwfDCKyKOUXx/faN87+4LqUgC56NkkpAPhp1juFdeh7fQkhcPZLd+CgNEoNPh0aVss58/9sQPgxS37hHFDYssK5gKw88Va+F1U0PPJXHexY6R7cxzZ5Tgt4MQR7EhrtgUBdZplZBokgH4BF2c98QGzVl9se1Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by SN7PR12MB6766.namprd12.prod.outlook.com (2603:10b6:806:26a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.35; Fri, 18 Jul 2025 02:30:16 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 02:30:16 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v4 5/6] mm/huge_memory: get frozen folio refcount with folio_expected_ref_count() Date: Thu, 17 Jul 2025 22:29:59 -0400 Message-ID: <20250718023000.4044406-6-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718023000.4044406-1-ziy@nvidia.com> References: <20250718023000.4044406-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN0P223CA0004.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:52b::14) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|SN7PR12MB6766:EE_ X-MS-Office365-Filtering-Correlation-Id: 414d551a-4d04-4df6-07aa-08ddc5a3080b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Z92jP8DkzelE8NvmVhLLRRZ216ac2JPwq4Cg9TUneI86c/JdQGjFWxVdo15d?= =?us-ascii?Q?cGyQVCcwddceCTtd1IW/NA0KbiEFU9rlAPcEvqWq+CjcHqj38AK/06L6qGFc?= =?us-ascii?Q?0ZQydKNDotvP/K+/4fkp1tlReb9QqLW0OLgcDCKg8FNrXBf+gqI5FSNIwOeq?= =?us-ascii?Q?W/DWVO4Rtq/JQx0ND5wfSRy9c95T+sFxpvpdnp4QjQvo9PheGwPwMHUJ/AKe?= =?us-ascii?Q?MnPiLDsgFGxo6dw8/ufzkAdCMh0sWZWYKdx2bBktjaebW4zrPWH6nJ4HzalN?= =?us-ascii?Q?Uu8u/WnsGDkNXJgR5bon0fLClr8Wa+WwyshD4Zw2a5BNjJdOtAnuF5AgLZWm?= =?us-ascii?Q?BkLK6+ZlEeZLO4MdT37rx+8I+vOA4NlOgQiXNN3I7pDK5XstQgejEYEtkabE?= =?us-ascii?Q?vgGf787watASB6591NWL+SrCITkOO99TYaoiyGBmXDMGaIyu5lOeoVOUk97n?= =?us-ascii?Q?yiWYtSlyptk2iivKBRyosBcJhX7D+Iifgf0Ay1Pr6qWPkhdmr53Br36Ncv8p?= =?us-ascii?Q?7K6Lp4AyI2/LpDMfMBlURI95ILCPPc0t7EwXDBjikOzRBKUwU6ftnBKsomAz?= =?us-ascii?Q?UHmLmWdI28vlv4sU1H9hUbcdHUdVkHKqrWRDEcfiRG1tUcmQoYowJqeUcqlT?= =?us-ascii?Q?SoLlZP/a0oYxRgUNSQGv0HK/pGmtEodFfygX0wKhs35J+s8CsaZfDg/JA0ME?= =?us-ascii?Q?ioK0srCQ9XJyRopMeiPVOHn3dw+mlBWccxgmU0tgbwdAcPRq7POn1ENsv6qo?= =?us-ascii?Q?JTkpTQs4V6wbdOqMEs5go6tLQyydw+FK4McXok+apivqK69xXQ5Rgk6QxhSa?= =?us-ascii?Q?uoCVN5U1HsDAuy9IANVDcdWpk9ubuq85NpI3pcCWGbgoa3cqP/S5cEqEgA8R?= =?us-ascii?Q?6Tfpy8i/JUFbea8HokClSyr6o6y3RxVEJya3xYBnNHEjzPR61ioWaDd/7g8f?= =?us-ascii?Q?U8/vqD0qq3GqyAZ91SJYjn3qvrTPpecc9p/xyjlWKv4mZEkIbR8pcASwHNEV?= =?us-ascii?Q?mdDuPNtbfHrTS/mutssPqJyFKXGqy6pYh8fnXUZjRh5GQMgZGUoxo9/+9jO5?= =?us-ascii?Q?FBB19BEMEXMwPZladQ6GbJ7AdVmKCyVQ0acE9tnSDXAqmhcOpCgsLw4HxxSd?= =?us-ascii?Q?UPrVrvzf1HdClHszrVDctNltxDXPJ8Tr05NfTHO9F/DYi9IxqKDffcGI9IP3?= =?us-ascii?Q?6fL21ksEkPwQRbn+8fKMaFiDR2Yzh/zKHwCd4+NCksqlTfJtdPY4S+i4InS0?= =?us-ascii?Q?QsojrF6Kz3PPFcmqEjEeZvm5xYuliuJLfLBPjuDWO4e4CS/cpl6S22Es+LWe?= =?us-ascii?Q?vRhMLmH9ER3g0zkt9jwhuiwWBQ+5+4KntU8asRvyv9fgOviocRgXct0ie1WL?= =?us-ascii?Q?Uvuer96pnid1+Ja45nAgEBoJpojj9RKExP6SRx2x6cOU9h9VfU0JacAtrJXz?= =?us-ascii?Q?hvgk2YpmMqQ=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?day7Xl7nhi58Q1jOAKIwgmw1UfwmRXL9Ki6wnxw3k5BXrSNNDGPnkMyAey8+?= =?us-ascii?Q?Og9IHvXYep8+xZd+Kr4Y0Unq0nMhW1HG1JCbq0dV9RLL8RCYEJS+PRzdymvu?= =?us-ascii?Q?i27rID/5zur2/X9YC14cRpb3va/vfJeCibv3j/sUMLGeSyFM1cHofB/RPDJ1?= =?us-ascii?Q?ykxFQyliZZ8yfm+na3WVAbMjJ5uMohlDobMVwTWVnU8kItVflQP2gJPemJuE?= =?us-ascii?Q?L9WZ8S/TBoBAcUs68y6SGS6XQcKntsJHgwb8Oa3jx2Tavu6IriHI1Ku7/KXg?= =?us-ascii?Q?orPW7BbQErUZZWBNA2bWyz4pW8hrLxbgzJQTuBbhobCOzG8PwfA/7+FNiN6O?= =?us-ascii?Q?moObPjn3G2MwoLYJ6ZkBaTlmHIVnN0pVMmO3R9Vme9iLoHukZY3ZMvu3chI0?= =?us-ascii?Q?iTnu+3mWqeXPyBz+hLA0bmhCT45WQdgztvE4VOIWMXMPiihrywFQajyfCvxv?= =?us-ascii?Q?msCDuF7kNPc5jAVzdFibIl3Ze5P3D8lHc5xzduV+CzP5wuZ63t7rxAk3eZQ8?= =?us-ascii?Q?RcXtRI8cdzKod/0HWXpVY9G2ekcQdiN61ZMWYeDT8Wbw4zG0zN2gAMlYHpXR?= =?us-ascii?Q?ESaCe3BSmmEY3fUk/O6RJ5nhwlAeGtewFfXn1RgQvhHYszu76v5C0DIsuJpJ?= =?us-ascii?Q?WZKRkwSx5XT+jOx4wdDn+6Vgl5l89x1PaOtrv0AaOAwxa0dIYj0YfG1j/OHx?= =?us-ascii?Q?Xu1oF0c7skgByGpvC03tLVQVgTxZ/HU73GTDY1yRQ6UyfgKeP8GSwopAxP7e?= =?us-ascii?Q?UdyGp9l3NIo12qP3WXhsgBi3CQOhc5Wg7E0RP/zSkQSY8liDij+mIuUyShtA?= =?us-ascii?Q?vPaDLhRujJtGARWjrM4Rv6oqxwDqtHqjRw3Kq7HsQcMTbyiL34Desg6WQEK3?= =?us-ascii?Q?SYjDGwZudsf2uwh8zirI6BQB2InX3OOPe68gCFk8Xc4zJ3rLtZA7eJWZkHuJ?= =?us-ascii?Q?SBkHh/1pfSm2Le2ATndVuuZ6tYAZAEhnmi5g/FwW1RPPcmCiZqQZKvqJY4iP?= =?us-ascii?Q?I5QqYG7fPsjGGl5p79uY2e5jEBIoPVt9TnhxIWntiN03Pl9YNDL8iW7YOWrU?= =?us-ascii?Q?+as9G8wFPQLnS5rryu2UuN4u38e09gICG2gQ73WCLXnCGwIlX/6TZVLrNe+g?= =?us-ascii?Q?TOtLC1bwHJLl0uMOXCPSGGYxVnA3V/1YsEOHH+mt+wIw+rfcep2lkvdQXzGj?= =?us-ascii?Q?SLsUoL4DYkNKzKIuDGKy34D0ZLZ9KZ04uAKKTe3DkE+5LMY3+RkiaoATZlnQ?= =?us-ascii?Q?WTihHy9qNCuq/zOj+sdKj36/LxYli6u4HTJsvm/h/S7Z+FLKnNqdBro7hMPg?= =?us-ascii?Q?zFBR9Lk+kkywt1Kh31S0AkN3DF958WjMEIGYCiXq8fYunsnWQ6hnfw3WmEiw?= =?us-ascii?Q?T0F6itHQJu7ei3Rpj8WoKi7jDkWznhwfZCpAwy214toYO9lMPJmojVADBAVo?= =?us-ascii?Q?z2tOaAIqAzMl7FS6pCLve8H9sA6dnp7Vh7Ro/AeqGknbPrzGhmvc7AKJqBHZ?= =?us-ascii?Q?vxFer3YAQNeRzunqG6/hyQHQT13A/oPIDY9o6HCGs3qCskqSxmVfVyy0RcaO?= =?us-ascii?Q?heiFfFkOcK/3JXScX1FrGTBrwlwn40iu6QkWIuPY?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 414d551a-4d04-4df6-07aa-08ddc5a3080b X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 02:30:16.3188 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: a+sGCPLR4D5OaSnJxOf7kvoY44XkaQBHy0LRbr9OvmZZrLeA5ljAfp+gDxZdJ+As X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6766 Content-Type: text/plain; charset="utf-8" Instead of open coding the refcount calculation, use folio_expected_ref_count() to calculate frozen folio refcount. Because: 1. __folio_split() does not split a folio with PG_private, so no elevated refcount from PG_private; 2. a frozen folio in __folio_split() is fully unmapped, so folio_mapcount() in folio_expected_ref_count() is always 0; 3. (mapping || swap_cache) ? folio_nr_pages(folio) is taken care of by folio_expected_ref_count() too. Suggested-by: David Hildenbrand Signed-off-by: Zi Yan Acked-by: Balbir Singh Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d6ff5e8c89d7..4db67970ae69 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3737,6 +3737,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, if (folio_ref_freeze(folio, 1 + extra_pins)) { struct address_space *swap_cache =3D NULL; struct lruvec *lruvec; + int expected_refs; =20 if (folio_order(folio) > 1 && !list_empty(&folio->_deferred_list)) { @@ -3800,11 +3801,8 @@ static int __folio_split(struct folio *folio, unsign= ed int new_order, new_folio =3D next) { next =3D folio_next(new_folio); =20 - folio_ref_unfreeze( - new_folio, - 1 + ((mapping || swap_cache) ? - folio_nr_pages(new_folio) : - 0)); + expected_refs =3D folio_expected_ref_count(new_folio) + 1; + folio_ref_unfreeze(new_folio, expected_refs); =20 lru_add_split_folio(folio, new_folio, lruvec, list); =20 @@ -3834,8 +3832,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, * Otherwise, a parallel folio_try_get() can grab @folio * and its caller can see stale page cache entries. */ - folio_ref_unfreeze(folio, 1 + - ((mapping || swap_cache) ? folio_nr_pages(folio) : 0)); + expected_refs =3D folio_expected_ref_count(folio) + 1; + folio_ref_unfreeze(folio, expected_refs); =20 unlock_page_lruvec(lruvec); =20 --=20 2.47.2 From nobody Mon Oct 6 19:09:20 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F4A51E520A for ; Fri, 18 Jul 2025 02:30:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805826; cv=fail; b=hCHA5oRpuh3fsqw6HOe8F12PjloPux63CMgr3RwC9UXfqJ9a+jutUFcUwGGSJ2RoQeffYuQkxcU9nFa2uhZ4XXDnjHm91/iSqdfXLcRrVY6i7O0SvrMtx5g61NUPog5BYq0WtqBJQSeQxH4I4W9QDcPTkV0oxN3UBXxN2dABeaU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752805826; c=relaxed/simple; bh=LsQxspP907f8yNgC1Kp9lmU0/35KW/XQw4G1IMucWtg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=EvAOx8GBRKodccTfpX24ZLINaESHPJweTP+T6zYY6pNawQbnJRsH/dnlUhYlNdGsKU8CBs49EtP90cLmbxTrV6FFBsurphzfytoYYqTdye6AzwZDgAGicrS49feVcToNIrQ0jWlwUdydFnSl0GwhqLo0OumxG40SQFLwoR/vNpw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=mKDaED5O; arc=fail smtp.client-ip=40.107.243.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="mKDaED5O" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=IvseIP0ysuM8VHxCklfGVKdWrktI6k++uy1aAez/PHpVeImgImu7h5CkglHO98qX0g6A6nFINtGarawwaVf9LqI1yiwWPBckte2B7qLilqUP45HV+ph/AOB+/FRkQcE6ddUsf/XsZtW0wiVWeWZv7X6GTV9dgHVgsc5ocrfx/TRIfOq+LcGTwxhFsPoxiqFyMjrmsJtc/HlbXqgktMRsi4Or7AOETltRbFx+qfkjCO9cFPS/bVeYukhkSaenLBKi9KxAyJrV03gGIYB0unvKVGuPfjwd9Cc+J4+qgN1NfHcyjtqLPrf+zDgRbLr/GL60X031+jpvCOsTTdDisjYeAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vnk5p6QXythh1Tl1yugCWGAHylMIbaGnTn/bEYc3BZY=; b=wjdnqCGvEAGoPB6iPoo0PqZgwgtkLmrU9cxGyivIDIIb95A52DfM5crdCTHSblU2uo1Z6B6iqi6pgfyMhE3OwzuAoeqVZNPkPUCztSWdRu14XjlHIpYRGD+Eha07ObD2q1MLPnLoTqH+nxS5Fsm+mL794BHrwOjt/X+vntAs4wMoE2lXpDBcfaaeoJyKCS22NyieskhtkirtHccqAhl6bQHI41leGWsQdbqyypcWg+11tWPh2vDotObB905IWSmXbf8qU2gbMOns/+4tn6zC9Yh1diZAxClLZBwuidvjVYHs6PKjXmwjqadKusvxMoASStmmnizetdUpE7vPgpcEGg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vnk5p6QXythh1Tl1yugCWGAHylMIbaGnTn/bEYc3BZY=; b=mKDaED5OsLPvo/RWpxkbiOMI6mukp1Jl4FriiZHbDn6MdJNhVbREZe0twW53pyCPQLnE9esEXIU6A1DqZIkBV6Xw4lRvivvQ49OMfP9HOFQmiBUq4OUjvCSvTWw3szfrAoMozWLR6yU0iO5gQrnrc1RDj5cCijVmAdkpo30IriQKIwP99LiV/iGxE3BvsoHLnIzqjL4CTPLoI/JQACbJpMT/0Fu2g41GcfQXV81RZDE9O13T7jy6uGqrhNfirqp3p2HgJ42EXOvHPiWqddAkayBhCwFxdh/2exd2BmvnTC07/V9db3CCIL+DYVpvYVorvB3OGxv55YlintZ9JJxkxw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by SN7PR12MB6766.namprd12.prod.outlook.com (2603:10b6:806:26a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.35; Fri, 18 Jul 2025 02:30:18 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 02:30:18 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v4 6/6] mm/huge_memory: refactor after-split (page) cache code. Date: Thu, 17 Jul 2025 22:30:00 -0400 Message-ID: <20250718023000.4044406-7-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718023000.4044406-1-ziy@nvidia.com> References: <20250718023000.4044406-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN0P223CA0023.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:52b::10) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|SN7PR12MB6766:EE_ X-MS-Office365-Filtering-Correlation-Id: e8c54f50-f7b7-440d-78e1-08ddc5a30926 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?qhXYnQ1v8bf8AlF6ncVq6kXxmyn5ulF+64aqvAxo12aocM88Z964bmmcza7C?= =?us-ascii?Q?AM58Mb3ZyhZHPdEnt73C7IQzB7YtMvVJtEJx4VJ61+Qximk2MwKQXpmSCFA3?= =?us-ascii?Q?k5wLJvs0QKKzilAu63u4kHS6+7uC/7iS71Qq0zLtPfpG8bMFn7luxaJ/XF2A?= =?us-ascii?Q?6UecPn0v9ZxdeRm0yivgGK8SzCPBsIasyvb0aKUHEhS7DZhdqFhlaPcw+JAa?= =?us-ascii?Q?ntQsc1gRLkZDHSE+fNXCXdGuOlEtJ3zIArwiPG1X0r1soANYWMIsmBqSOuIL?= =?us-ascii?Q?O7ijd7YWALDN9JuLzpyIUyL3koX8sQVt1S9Jl77r69neuhCaBZDsD1DjUKZ2?= =?us-ascii?Q?YuhsTalL+CqCeghUDclas9y7+yq+ZVUzWiwdDoM/1B1CcGHVQgcM3Iw9Rgbk?= =?us-ascii?Q?BecSN4gGuxshfecTjhwpuslsJDqcuzUsyWewAhFodrrQQPa44VR8PJ8Rwd9O?= =?us-ascii?Q?bVJnvdY+K1bPDAdOtWHEjxIm5L4K1JhfSzBD4NtejUlmzCvYxs4Crfl6lJnB?= =?us-ascii?Q?c+NDnuG8rvMrUWRQuCWeGvP+9r8xKll/dg/TDPaIGpqzzwA9C0Q4NlqCjIjt?= =?us-ascii?Q?4jbYEHJ6mmNOzQt+HH+Zku4a6o3IC57xLAh1fV/ZFOToVbTKD0Lfav5N0yrd?= =?us-ascii?Q?pG2rNAPwthHqcxfJ11ygawcMsaVejgaCcT54ugM5BcDBGC5JtV655GWVB5JA?= =?us-ascii?Q?HlrFSZL+UPmPbRDxIM8ND9GO4jOmX8ryzq6Dt7/pRS44hLLNY08QBBM0oHKA?= =?us-ascii?Q?P0Ojkx3bNmUI+nDKG1kdgBoF21VHUG+xVYoZjLQ3Nv11p875p7FHZA45wWjE?= =?us-ascii?Q?v/gHv5Gl0vnQOb0WlJ+v54t38LCr4DPQrJo8LH+4EOrUWtustFiRL8GetG7V?= =?us-ascii?Q?Cq5wYroYAj+9R3BwU7Z0BifRbHMoPrhUO/vozjc2BiBmt7oGovGcOEDaJSjq?= =?us-ascii?Q?33Htzfe1ROo1xk+enYXLRvJiV/ZWveRoO7wcjxThPSS1HVp+ddDaeBPnGZRJ?= =?us-ascii?Q?Nu2/Lz2lezl85slkr+DEdAzhV/0pj25zt/5vRajmbTInquunPlZZd1QnsBuT?= =?us-ascii?Q?08ObcVbK9r+dJAWBv13HdTQfs3yYPDL9zumk7dZFEZu4MpRmIv+naEzTUGKa?= =?us-ascii?Q?XfkUBML/r/xEwrSiH+VJhlebjwW/sGIpp8IuGd7qiaTDBlVKWEIn4hBmCSCi?= =?us-ascii?Q?tkOfb+Aoor1fUHm/3ZVJ4y34clJ3fDprOTpqZgbIvLpjSK4djsLdTfBaWA3W?= =?us-ascii?Q?qDmtYj/Xg7LjSzoL5nPe5pUVodn9bB/JlAX5JvdjQMmaPSI9Pg50ZcVqobq4?= =?us-ascii?Q?48GYdslX4Ch9jXoFrZIh2fQQgVXU9gz3iwSa16NseXotcsAe4c7az09VxGHG?= =?us-ascii?Q?bjzMbXnV5Gw8L2NOtCM/I8g63BNHj8D9sTuuTVAEPsHcvn2fG9jbotGbKMd0?= =?us-ascii?Q?34m6GIFBKh8=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BlKYpfjysTJZHO1O7vt76b7wqaHpQB3fK2t0X7CJFlvn66cm0blvKP5kamKa?= =?us-ascii?Q?lc/L3Qp76YfmDTulKIXvyjvQ/Q9i4nWEQhbJtkl56fGef3Uplc8eqUfprZbR?= =?us-ascii?Q?0s7YC/y2huVnWkoSDn+zqC4pIoPRJk3m2omKs46R/6Rm/TNwA75WT8S9HyGs?= =?us-ascii?Q?M20dhXLo4gYia5w7kE9ZnLrmmlhP3nu4iFQW6nl0ECLY41xgm7xmxPoBxpnV?= =?us-ascii?Q?GFTuVhXBxez1sJt4HoNWH24rrpmQP+K+Ge1NfT8kTT81RDF4MJ2BvNrn1RY/?= =?us-ascii?Q?fqu5Dfni+C2J+c0lKnZ64D4OZkl14qKR5qWuM8C16WGt0e5BCkmzZXxu2vnB?= =?us-ascii?Q?v8G82yI0HJhPLzLMRx+pExxyLv4Tq7djBWBC0BnFuyCLQbsr/5FxI8nIo9qh?= =?us-ascii?Q?3WWf9HSaj4FYFAaOBnCg3OzZbt8WVT6HztHoUTzgrYxMW19hD/8O+Lbg1btY?= =?us-ascii?Q?Op4B3ScP5XLf1iXeSZxxUWNsGfe9EdKq8zaXQXoFWrzZMWHHogvjs9386E1O?= =?us-ascii?Q?Yp2l0c4wv03tjyIG2fxlAXJknwnffFAngVnFxy8viaLpXszbo9Kaen5RrdqN?= =?us-ascii?Q?GpA3oqOb6cm5PhRP9P0F5r+weMKrNolGI6BjuB+xEsEphn6ITEYmqqfwrCUd?= =?us-ascii?Q?/yMte62F2It/+RtPEVVG1F0ypvXxgeLxFP/BCJAUUDpyjvUwmqsWcjG60OnE?= =?us-ascii?Q?XuGFau2widUzGdwjWoNfahQeBEkbXVZgjK3ukPMp9LVIyV90YQE0lJ5/N0pA?= =?us-ascii?Q?jSdE2f2fi19u00L25mgBWz299pOweNwLz770JIzUveMlnWMmNWguftH8nkn4?= =?us-ascii?Q?295Z/GWhPVDWvknS7NfUm5+l+j1j7TGfmEZBAiUi54WVbaKMdCwgP4wu63/N?= =?us-ascii?Q?sEUp4Xvd2QKg30gUDWXmq25Cw/jXh3KKARbhNEYT/ErtOKOsYTyeWVAdaLYx?= =?us-ascii?Q?CWNnCqSjccl97esPkmCmwuhGJ0ybO9vp+DleAfPO+f5LqpY6j069IKpMKA/H?= =?us-ascii?Q?2CpIFGpPh4iRWqRj13jr/u6A+fCImHGlGZBMszK5O3R1SWCftX59gRHSZf8Q?= =?us-ascii?Q?/ybuIiLtHGvkrps/CrzuwB8RVOKbVdqUxOsyqcozB5P8rgFvpFSV9ju0/+N/?= =?us-ascii?Q?YQNUIhEk83nS07bEnUwxttNSQdnnzfGrXeVODXqrA+mcd+fXHLeRqckjat0g?= =?us-ascii?Q?G8oMlTcUpp5Dst4IPGyUxw+f9rcPCYRk5VDZsyx6F7B7ADJ5nJtT/nlHfska?= =?us-ascii?Q?vXifRru/Oc390e9bBH157bqMObYZIKguxgpkAiK8tAGSlqdDCY0ZKHNp+ZRt?= =?us-ascii?Q?T2DHuc5HXRpINYj73PpKmlXj6Cp2Qtud0WwZk0aSMR5iwhX8VuzDu/JJntx6?= =?us-ascii?Q?vIs+ZQPtXRNDGWltbdJ0ra1YIHja9N13fNmnuF53gNOI2M1gE5ptQMki/yov?= =?us-ascii?Q?ibD7Y5SwzmtgMqVOOxj6Ay3jrx55MbFeCVvUN7F2QlN5QAXX3NGC0PJfryX5?= =?us-ascii?Q?xJsdc7sPD9EPK/N3i2xRjSD8OA1bvRO27P52YAw/YmlAC+i3KvIghtuFmIu1?= =?us-ascii?Q?pPbTOSAg6MkMKhM6hVD4jvJOhwV/BrVMvAByW2o4?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e8c54f50-f7b7-440d-78e1-08ddc5a30926 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 02:30:18.1564 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KHwGFBXIqF0WG0sqd89jkGmhKneHqYEDQDM7B7KfZoCM+kS2ks0D5KwDHgV3h6BK X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6766 Content-Type: text/plain; charset="utf-8" Smatch/coverity checkers report NULL mapping referencing issues[1][2][3] every time the code is modified, because they do not understand that mapping cannot be NULL when a folio is in page cache in the code. Refactor the code to make it explicit. Remove "end =3D -1" for anonymous folios, since after code refactoring, end is no longer used by anonymous folio handling code. No functional change is intended. [1]https://lore.kernel.org/linux-mm/2afe3d59-aca5-40f7-82a3-a6d976fb0f4f@st= anley.mountain/ [2]https://lore.kernel.org/oe-kbuild/64b54034-f311-4e7d-b935-c16775dbb642@s= uswa.mountain/ [3]https://lore.kernel.org/linux-mm/20250716145804.4836-1-antonio@mandelbit= .com/ Suggested-by: David Hildenbrand Signed-off-by: Zi Yan Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4db67970ae69..19342660739b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3646,7 +3646,6 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, ret =3D -EBUSY; goto out; } - end =3D -1; mapping =3D NULL; anon_vma_lock_write(anon_vma); } else { @@ -3799,6 +3798,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, */ for (new_folio =3D folio_next(folio); new_folio !=3D end_folio; new_folio =3D next) { + unsigned long nr_pages =3D folio_nr_pages(new_folio); + next =3D folio_next(new_folio); =20 expected_refs =3D folio_expected_ref_count(new_folio) + 1; @@ -3806,25 +3807,36 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, =20 lru_add_split_folio(folio, new_folio, lruvec, list); =20 - /* Some pages can be beyond EOF: drop them from cache */ - if (new_folio->index >=3D end) { - if (shmem_mapping(mapping)) - nr_shmem_dropped +=3D folio_nr_pages(new_folio); - else if (folio_test_clear_dirty(new_folio)) - folio_account_cleaned( - new_folio, - inode_to_wb(mapping->host)); - __filemap_remove_folio(new_folio, NULL); - folio_put_refs(new_folio, - folio_nr_pages(new_folio)); - } else if (mapping) { - __xa_store(&mapping->i_pages, new_folio->index, - new_folio, 0); - } else if (swap_cache) { + /* + * Anonymous folio with swap cache. + * NOTE: shmem in swap cache is not supported yet. + */ + if (swap_cache) { __xa_store(&swap_cache->i_pages, swap_cache_index(new_folio->swap), new_folio, 0); + continue; + } + + /* Anonymous folio without swap cache */ + if (!mapping) + continue; + + /* Add the new folio to the page cache. */ + if (new_folio->index < end) { + __xa_store(&mapping->i_pages, new_folio->index, + new_folio, 0); + continue; } + + /* Drop folio beyond EOF: ->index >=3D end */ + if (shmem_mapping(mapping)) + nr_shmem_dropped +=3D nr_pages; + else if (folio_test_clear_dirty(new_folio)) + folio_account_cleaned( + new_folio, inode_to_wb(mapping->host)); + __filemap_remove_folio(new_folio, NULL); + folio_put_refs(new_folio, nr_pages); } /* * Unfreeze @folio only after all page cache entries, which --=20 2.47.2