From nobody Mon Oct 6 17:06:23 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2057.outbound.protection.outlook.com [40.107.236.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 801D82E7F31 for ; Fri, 18 Jul 2025 18:37:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863878; cv=fail; b=gJyfLwa+3aH6+wj9fnJtU6Op5o28ic/6wxsL1S5+tIDQqUBgseB/UIppGDsWO+oD+Wecjb9ey9tNDqTYwe9yRb8ioVscY0I6fHwo/3N1YH2s1mAfXDtiW2IFrLKIf8mYu7jVvSNGZRjnSKlrRw50kH3XUe6Ra0X/Mg7wvBfUd1A= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863878; c=relaxed/simple; bh=mkcLkrv/jkP/eoWbkDK4dGL+Xxn658ZIOKNa/dYgefc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=X72zkF1POeHSWY1gD/LyrvapddU0nvqIST2ADJa25fZn/3FUSF3QpH6P5EtzXjKG7vFv3fseDG+fFTBxnb4jeOEfgN80VH/Izwr2GtnJNRw8g89YE4dUIXHOo7/sWuDWpvCBlb0yTg4SvsllHVGjgvICXCcf15gx4fAmJA1T18U= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Fao+ZwZ2; arc=fail smtp.client-ip=40.107.236.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Fao+ZwZ2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CCrnICmzbakS5IcwAerNrZ6ShTcA8tWmjv/lpsZXMBD4GOy6V8uaT/WNsiYPzcMnLY8ukMvOqt6uz/eQMTiR8jL4tO+uRusAL1+CmW8lkshcWiyW5ED5JpzAFx98SrDPBAX/fsrjeBgUHQDGAJzJKYwGsgGRILF0gzON02AMcZv1LpZJ+Vdrn2knGO7tknXqiMKwtLd0uONZj8vv7NPlAY77GaXiDTOcAf3yLrJf1h2hzR0+c7fy+S8Xro35koeqm8Y5XLJbCY3yNGK94Wp4L+rl9hKuSn8+p8Qp1okfhnUYhvmnp523I/z+SfEyn46O3CI/gZZ9hZmcXoQ2S14Ygw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=egPl9g9InP7chuEeEkMYM8eNExLny2cXFxlMLTs/SEI=; b=GjFlbTCeeJsz3u2BIlmukIS7CDaS7Zkwt45KSVJdFm5i78Cd1bqJoQQHnZ0kf/KcU8vE0FexhGwZw00sx8DgifCkJgpmPpe4hzgWxAdMAIps8W/H4V/9TNVJi+6XPcVLWuEVgUenAFVtt1rvBJ4Q8iYIrKYwRAKUCi9CX2I38hdVzFmGZYGSJOTmk9KISG3X3DDBMI6KrWMUinIxVcb4JFRzGcZyTZNJbWRwCDN8s9DOK3bc+dsZAOH17sXahwELJkfwjJiZkV4ZyaFkmZBtiphlzp1oH0dWq5Gn4SFAorH5rI/QoAqlnP/+NMw3H3Z0VPhw6VYV0Qgbiwg/hyWgPQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=egPl9g9InP7chuEeEkMYM8eNExLny2cXFxlMLTs/SEI=; b=Fao+ZwZ2EoMhp8XBNAC3n3WHM44Q6w7uyD2SEnqZJwyzP0m7RYtEG5GgFXNF1nLmlHhxZYrkLUKaBVCT/bJenEGKqP4x2TmkIZrncktVZwmGiqWHmKFBZGCSzL/jn92QU+/0arpFGS/Rs6HxdvvSlTgEHRRU5d4xjJ5RnD8FbMbo+y/MojO12t2bGHkI33UOGEvmZu06HjtKp+vEao7Lc7+0qvSwLLJDnaIlTiD0YwE8Rrgn/cJ//nIcr3RoiLZ6e566XnmH1kG7q4+xXoz4TM7+VjN71UMocAfXZcqMjqqx6saIAbvboYTYKlogjsfQZ7b6zyFxJawiBTe+O2kIkg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY5PR12MB6621.namprd12.prod.outlook.com (2603:10b6:930:43::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.39; Fri, 18 Jul 2025 18:37:50 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 18:37:50 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v5 1/6] mm/huge_memory: move unrelated code out of __split_unmapped_folio() Date: Fri, 18 Jul 2025 14:37:15 -0400 Message-ID: <20250718183720.4054515-2-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718183720.4054515-1-ziy@nvidia.com> References: <20250718183720.4054515-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR16CA0059.namprd16.prod.outlook.com (2603:10b6:208:234::28) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY5PR12MB6621:EE_ X-MS-Office365-Filtering-Correlation-Id: 816d8c4a-49f6-48f7-6376-08ddc62a3301 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?nRstsZ4L/Db/IeRX52SvK9Ue08Yb2mfqiHRe9eA3Sm+CYIfbicXMVC/yUvXA?= =?us-ascii?Q?fJDwXODlbB65N86wQHv+rjKbMlILut/8xSTFJk3jgIhrVlXQT3lo6rSKyeiE?= =?us-ascii?Q?G5K+1Nq4F0DaKKM/xUucQiVR0jPz3IfAPdgQO0ZYti9nlxjnWTmroqdUz5zM?= =?us-ascii?Q?7UtXNrrqeiUjz4QXuVQbCDkbJuLL0IXwsv8njnKbylyVEMfwYBxQi13xU8SV?= =?us-ascii?Q?TDP1umZaoIMXyBpPMu2orXL3yIoEJKDjYoGbJrRijJG0bee17Y+SbWKC2zfN?= =?us-ascii?Q?rsbjvqi9lxDTukDEQLUL9Q10ZcaIhir9ajD1VRahPVRXlEgfhVMBFzMUvEFQ?= =?us-ascii?Q?BiUVpcf9BLA+YAOjpeGCBQO1NLMKoQDytkLAyg5Lk+YraSJaJx0XRZtNDbB7?= =?us-ascii?Q?i9OfGnx3L80v5+/P+Tj21pJfP+884IH4XluhBud4rOqQNfOi8k83N08a6XWk?= =?us-ascii?Q?7F5sWvn6GxnSN+hxslwqUW5lU52xrb/q7Ol2xLqDxs8TR4mefOdSeCgOokCO?= =?us-ascii?Q?WECXcWyRm1bWABDD7AxkSnZjn7AHf46X6uXsqDCQsYje18Z6G64B+wI6XOQe?= =?us-ascii?Q?2R+5gn45aXWNzuF/WMB3iG4/BcQ4xbxythdtAG/ynhc6plMrtb5mnaRkb2AH?= =?us-ascii?Q?QL9lc5rq3PT8UA4ukc0f/s3Ca8cqbv5xvh3OEW9LTHUm7BVbfcM0DyaRPIxi?= =?us-ascii?Q?ltXK3aXGN0JP6Mki3Jh2hhJk/5Pl+kRQeDNvBMIvZNOi0jIiYnWhAMdlNteP?= =?us-ascii?Q?L8YBKC3m7Xl4SDRSqDUAPTtPW1SsfCO/OnovMzfmrVeBA48ahdYGCUMDC+LO?= =?us-ascii?Q?Pgv+ckeOhvQy8jc8VT0LYC6dcvEh9xt2m1vdS4ytNI0ROG+gfdMuI/z7VH0g?= =?us-ascii?Q?4cb3CftqIWfNtFNCgy76Xohw6m+DXRYVADnHtAqcBwaEmso1zEqYs2O9KJVf?= =?us-ascii?Q?HZ4g5PLV4GDVZySobXiAmOt2vHno2gGxCKsgu4KFAhlhsXeKoCkJYrPd8w3R?= =?us-ascii?Q?G/hAOy0HOdW/sCVFG8tV8PmI5YJnbtsA7uIkASc8TKoF8FapqUlCfNBVqBAc?= =?us-ascii?Q?yyaZdrckNmai4zizsFZrl4mEwy+dYs3xGvMBGZmoYyS0rwPuyO6uJQhFSZPi?= =?us-ascii?Q?IRJ22XcDK8zRQEa72Wm13nPGhOOCqSp7eETisT9JMpK4lRtTj+WRCAWDacbq?= =?us-ascii?Q?07C4FRGKLc6La3KGJsIT1uG9tPmeqqnwKPinWwd2P9eWglGyHuP22Hgt/omf?= =?us-ascii?Q?yrYXrJ0ESYDZ/I/07KUEDTe8mMIwwUV5MNJTozjn3ArdGplKLn5MH2r1s3lY?= =?us-ascii?Q?PaThFKINglaEjlZ/N+IAuQauJqp15Tu21AmGVMbpSwgy6cRxgpqieAquVCOI?= =?us-ascii?Q?lKIh66WcembpuvVCL3T18bLFGUt+RsWD0WjBb6pEYmW2OcwWa38zwKnGSaSv?= =?us-ascii?Q?FZx7pr2SnAk=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cD2Fq300Ryemb5a7WEbyuIslb2kYcyyljWmfXMBIwg6Rr6kPYv5LoAsQeYO3?= =?us-ascii?Q?TH0qWU3JT8VYK6MGdfyhDlWFjd7JHP078BV9jb+xvDTHBe7xstB8w0WPDPIh?= =?us-ascii?Q?iTLossGx4D03mSfJObbrta0soWllYpkA4SFF+3pkgi5FQiYOjV91mqC/Te7k?= =?us-ascii?Q?8eM79ZLMRe4IT9fe/DwOSkpVC79U65yo4EE9WJMIRmd3uIxyXRVGJ6WN6m6n?= =?us-ascii?Q?tSGPMEv6gMuGfmiJWeBHm/DWx3peD2uhi3VfVU421b7USevUCe2HkSk0fpZ1?= =?us-ascii?Q?fePinNygtkatmxkeNrjSBUrdKoQi9K7yWugESbeG0p7Yo007I6LxUKQJ4a/w?= =?us-ascii?Q?/n9KErwAkiv6/vS41yNDe/skQN3AZQ+Q0h5mOLF++ey+MxGpI3O/nm2Xvoga?= =?us-ascii?Q?3lB1v5x++PaeFuxC4TBUXV3PF/I2e9vr84l2bfjImh1C5uC/zapNIUmhg3VW?= =?us-ascii?Q?YUPP4jKWwazVaHhWnK7vMC0n7kDRIiLc9hdfGCJRPsNytIky80D2h3+lDriq?= =?us-ascii?Q?RtF3kT4koITW2lKPk9re0rB5KLulIgmB/UbTUikv3ImWE9cRFejjBvRm+USm?= =?us-ascii?Q?99AcNPgnoeJxzP6yOuzhSE5/2iqQTZHcCI39wb/Rh9tgeCe0Q3THK1lfLIdP?= =?us-ascii?Q?JUe4s/N4PwFd5zt+Z9FgUCZeg1XiMH6J6CGt8J7DsdzuQZPeHBWDQ6kU6VVf?= =?us-ascii?Q?fT8sAwLmw1VpY79LPWuyVd3wG5X+6UaCD/bg8fXy9J7HtCviQsrdUvnQRnA2?= =?us-ascii?Q?BVxr62jTvHePvOK3KOz4c6/BSQ+hG7smCThYI+SFWKFM2rXV1EnptwcEkF6V?= =?us-ascii?Q?mPTZE+VVmXxv01CTp8V8g+no09Al9jigZzCM5CAjWLkexHAQ2tx0xgE84yo7?= =?us-ascii?Q?+w6i+rj46041HeCF6tU93nIBWImhPYgnXT9ZtFT9RH0W04Lr51/duIzIjbYa?= =?us-ascii?Q?wWJOE+n27D5dAIOXvQQOPeTN9R/R3wkKgbOSkPdvpmzaMQ0wt4j56vRUDnbE?= =?us-ascii?Q?H0nNGAi8gFYgVERjNj00y8gcKLlVSnYWMNn6Ejru4YycS26KiAzMxzDvC0o7?= =?us-ascii?Q?uqM6VBtEWiomqLnCSOOku+MJx5yVNfV5ddXgZkPjTbSwVpsF1+FuQAsrHqHu?= =?us-ascii?Q?OWg8CZgHHTJmuiTEAWdTPCAda7kNiXFqR+gZB2l/7wCNFurPgupg3Bt+ZI5y?= =?us-ascii?Q?e6h7MofBTAFOkbOeXuPPaV6RD+UjJB4ayA9y1MHNV6e8S1ZI7yC/g3ydvHV/?= =?us-ascii?Q?EzY/zBgwCKnOGVmP7pWFO+UvN8Xrq8sQR51+GZmw2DhSjvhlZmwlgBJi6Twg?= =?us-ascii?Q?kZQUHhLJ+ynzZvjVk7yOJAcoIsNYs8+l9HH5K084uhiCKQgn3WVacc1YmX0I?= =?us-ascii?Q?arXMEqmWONJ7oMd7oovl4TwYO9Nu4n5NkzxTiUG6aSuyAi2E+8DYTN6SG6K6?= =?us-ascii?Q?Bq/vG2SWYPBnZAa5GihNXMMKzF1ZyzrZoA8G8S9GqBJ0aZBDMrhKSN+NfPAP?= =?us-ascii?Q?yVAYiXv4F99L3U4E2JW2sjrazDHFGtiHrNb7z0kN6qQ7ziVh0tgz9CAH7Jt0?= =?us-ascii?Q?wya+L0+CfQWx/8n2l8WKZamxnn9N3IV7o5VRWWxT?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 816d8c4a-49f6-48f7-6376-08ddc62a3301 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 18:37:50.4290 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gxvtOjjBb6YbQNJkY+TMtHVzAU5V9XAl2CyfwO282ami2B0+02chdbuEjL7t90cY X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6621 Content-Type: text/plain; charset="utf-8" remap(), folio_ref_unfreeze(), lru_add_split_folio() are not relevant to splitting unmapped folio operations. Move them out to __folio_split() so that __split_unmapped_folio() only handles unmapped folio splits. This makes __split_unmapped_folio() reusable. Remove the swapcache folio split check code before __split_unmapped_folio() call, since it is already checked at the beginning of __folio_split() in uniform_split_supported() and non_uniform_split_supported(). Along with the code move, there are some variable renames: 1. release is renamed to new_folio, 2. origin_folio is now folio, since __folio_split() has folio pointing to the original folio already. Signed-off-by: Zi Yan Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 270 +++++++++++++++++++++++------------------------ 1 file changed, 133 insertions(+), 137 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ce130225a8e5..63eebca07628 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3385,10 +3385,6 @@ static void __split_folio_to_order(struct folio *fol= io, int old_order, * order - 1 to new_order). * @split_at: in buddy allocator like split, the folio containing @split_at * will be split until its order becomes @new_order. - * @lock_at: the folio containing @lock_at is left locked for caller. - * @list: the after split folios will be added to @list if it is not NULL, - * otherwise to LRU lists. - * @end: the end of the file @folio maps to. -1 if @folio is anonymous mem= ory. * @xas: xa_state pointing to folio->mapping->i_pages and locked by caller * @mapping: @folio->mapping * @uniform_split: if the split is uniform or not (buddy allocator like sp= lit) @@ -3414,52 +3410,26 @@ static void __split_folio_to_order(struct folio *fo= lio, int old_order, * @page, which is split in next for loop. * * After splitting, the caller's folio reference will be transferred to the - * folio containing @page. The other folios may be freed if they are not m= apped. - * - * In terms of locking, after splitting, - * 1. uniform split leaves @page (or the folio contains it) locked; - * 2. buddy allocator like (non-uniform) split leaves @folio locked. - * + * folio containing @page. The caller needs to unlock and/or free after-sp= lit + * folios if necessary. * * For !uniform_split, when -ENOMEM is returned, the original folio might = be * split. The caller needs to check the input folio. */ static int __split_unmapped_folio(struct folio *folio, int new_order, - struct page *split_at, struct page *lock_at, - struct list_head *list, pgoff_t end, - struct xa_state *xas, struct address_space *mapping, - bool uniform_split) + struct page *split_at, struct xa_state *xas, + struct address_space *mapping, bool uniform_split) { - struct lruvec *lruvec; - struct address_space *swap_cache =3D NULL; - struct folio *origin_folio =3D folio; - struct folio *next_folio =3D folio_next(folio); - struct folio *new_folio; - struct folio *next; int order =3D folio_order(folio); - int split_order; int start_order =3D uniform_split ? new_order : order - 1; - int nr_dropped =3D 0; - int ret =3D 0; bool stop_split =3D false; - - if (folio_test_swapcache(folio)) { - VM_BUG_ON(mapping); - - /* a swapcache folio can only be uniformly split to order-0 */ - if (!uniform_split || new_order !=3D 0) - return -EINVAL; - - swap_cache =3D swap_address_space(folio->swap); - xa_lock(&swap_cache->i_pages); - } + struct folio *next; + int split_order; + int ret =3D 0; =20 if (folio_test_anon(folio)) mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); =20 - /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - lruvec =3D folio_lruvec_lock(folio); - folio_clear_has_hwpoisoned(folio); =20 /* @@ -3469,9 +3439,9 @@ static int __split_unmapped_folio(struct folio *folio= , int new_order, for (split_order =3D start_order; split_order >=3D new_order && !stop_split; split_order--) { - int old_order =3D folio_order(folio); - struct folio *release; struct folio *end_folio =3D folio_next(folio); + int old_order =3D folio_order(folio); + struct folio *new_folio; =20 /* order-1 anonymous folio is not supported */ if (folio_test_anon(folio) && split_order =3D=3D 1) @@ -3506,113 +3476,32 @@ static int __split_unmapped_folio(struct folio *fo= lio, int new_order, =20 after_split: /* - * Iterate through after-split folios and perform related - * operations. But in buddy allocator like split, the folio + * Iterate through after-split folios and update folio stats. + * But in buddy allocator like split, the folio * containing the specified page is skipped until its order * is new_order, since the folio will be worked on in next * iteration. */ - for (release =3D folio; release !=3D end_folio; release =3D next) { - next =3D folio_next(release); + for (new_folio =3D folio; new_folio !=3D end_folio; new_folio =3D next) { + next =3D folio_next(new_folio); /* - * for buddy allocator like split, the folio containing - * page will be split next and should not be released, - * until the folio's order is new_order or stop_split - * is set to true by the above xas_split() failure. + * for buddy allocator like split, new_folio containing + * @split_at page could be split again, thus do not + * change stats yet. Wait until new_folio's order is + * @new_order or stop_split is set to true by the above + * xas_split() failure. */ - if (release =3D=3D page_folio(split_at)) { - folio =3D release; + if (new_folio =3D=3D page_folio(split_at)) { + folio =3D new_folio; if (split_order !=3D new_order && !stop_split) continue; } - if (folio_test_anon(release)) { - mod_mthp_stat(folio_order(release), - MTHP_STAT_NR_ANON, 1); - } - - /* - * origin_folio should be kept frozon until page cache - * entries are updated with all the other after-split - * folios to prevent others seeing stale page cache - * entries. - */ - if (release =3D=3D origin_folio) - continue; - - folio_ref_unfreeze(release, 1 + - ((mapping || swap_cache) ? - folio_nr_pages(release) : 0)); - - lru_add_split_folio(origin_folio, release, lruvec, - list); - - /* Some pages can be beyond EOF: drop them from cache */ - if (release->index >=3D end) { - if (shmem_mapping(mapping)) - nr_dropped +=3D folio_nr_pages(release); - else if (folio_test_clear_dirty(release)) - folio_account_cleaned(release, - inode_to_wb(mapping->host)); - __filemap_remove_folio(release, NULL); - folio_put_refs(release, folio_nr_pages(release)); - } else if (mapping) { - __xa_store(&mapping->i_pages, - release->index, release, 0); - } else if (swap_cache) { - __xa_store(&swap_cache->i_pages, - swap_cache_index(release->swap), - release, 0); - } + if (folio_test_anon(new_folio)) + mod_mthp_stat(folio_order(new_folio), + MTHP_STAT_NR_ANON, 1); } } =20 - /* - * Unfreeze origin_folio only after all page cache entries, which used - * to point to it, have been updated with new folios. Otherwise, - * a parallel folio_try_get() can grab origin_folio and its caller can - * see stale page cache entries. - */ - folio_ref_unfreeze(origin_folio, 1 + - ((mapping || swap_cache) ? folio_nr_pages(origin_folio) : 0)); - - unlock_page_lruvec(lruvec); - - if (swap_cache) - xa_unlock(&swap_cache->i_pages); - if (mapping) - xa_unlock(&mapping->i_pages); - - /* Caller disabled irqs, so they are still disabled here */ - local_irq_enable(); - - if (nr_dropped) - shmem_uncharge(mapping->host, nr_dropped); - - remap_page(origin_folio, 1 << order, - folio_test_anon(origin_folio) ? - RMP_USE_SHARED_ZEROPAGE : 0); - - /* - * At this point, folio should contain the specified page. - * For uniform split, it is left for caller to unlock. - * For buddy allocator like split, the first after-split folio is left - * for caller to unlock. - */ - for (new_folio =3D origin_folio; new_folio !=3D next_folio; new_folio =3D= next) { - next =3D folio_next(new_folio); - if (new_folio =3D=3D page_folio(lock_at)) - continue; - - folio_unlock(new_folio); - /* - * Subpages may be freed if there wasn't any mapping - * like if add_to_swap() is running on a lru page that - * had its mapping zapped. And freeing these pages - * requires taking the lru_lock so we do the put_page - * of the tail pages after the split is complete. - */ - free_folio_and_swap_cache(new_folio); - } return ret; } =20 @@ -3686,6 +3575,11 @@ bool uniform_split_supported(struct folio *folio, un= signed int new_order, * It is in charge of checking whether the split is supported or not and * preparing @folio for __split_unmapped_folio(). * + * After splitting, the after-split folio containing @lock_at remains lock= ed + * and others are unlocked: + * 1. for uniform split, @lock_at points to one of @folio's subpages; + * 2. for buddy allocator like (non-uniform) split, @lock_at points to @fo= lio. + * * return: 0: successful, <0 failed (if -ENOMEM is returned, @folio might = be * split but not to @new_order, the caller needs to check) */ @@ -3695,10 +3589,12 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, { struct deferred_split *ds_queue =3D get_deferred_split_queue(folio); XA_STATE(xas, &folio->mapping->i_pages, folio->index); + struct folio *end_folio =3D folio_next(folio); bool is_anon =3D folio_test_anon(folio); struct address_space *mapping =3D NULL; struct anon_vma *anon_vma =3D NULL; int order =3D folio_order(folio); + struct folio *new_folio, *next; int extra_pins, ret; pgoff_t end; bool is_hzp; @@ -3829,6 +3725,10 @@ static int __folio_split(struct folio *folio, unsign= ed int new_order, /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); if (folio_ref_freeze(folio, 1 + extra_pins)) { + struct address_space *swap_cache =3D NULL; + int nr_dropped =3D 0; + struct lruvec *lruvec; + if (folio_order(folio) > 1 && !list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; @@ -3862,9 +3762,105 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, } } =20 - ret =3D __split_unmapped_folio(folio, new_order, - split_at, lock_at, list, end, &xas, mapping, - uniform_split); + if (folio_test_swapcache(folio)) { + VM_BUG_ON(mapping); + + swap_cache =3D swap_address_space(folio->swap); + xa_lock(&swap_cache->i_pages); + } + + /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ + lruvec =3D folio_lruvec_lock(folio); + + ret =3D __split_unmapped_folio(folio, new_order, split_at, &xas, + mapping, uniform_split); + + /* + * Unfreeze after-split folios and put them back to the right + * list. @folio should be kept frozon until page cache + * entries are updated with all the other after-split folios + * to prevent others seeing stale page cache entries. + * As a result, new_folio starts from the next folio of + * @folio. + */ + for (new_folio =3D folio_next(folio); new_folio !=3D end_folio; + new_folio =3D next) { + next =3D folio_next(new_folio); + + folio_ref_unfreeze( + new_folio, + 1 + ((mapping || swap_cache) ? + folio_nr_pages(new_folio) : + 0)); + + lru_add_split_folio(folio, new_folio, lruvec, list); + + /* Some pages can be beyond EOF: drop them from cache */ + if (new_folio->index >=3D end) { + if (shmem_mapping(mapping)) + nr_dropped +=3D folio_nr_pages(new_folio); + else if (folio_test_clear_dirty(new_folio)) + folio_account_cleaned( + new_folio, + inode_to_wb(mapping->host)); + __filemap_remove_folio(new_folio, NULL); + folio_put_refs(new_folio, + folio_nr_pages(new_folio)); + } else if (mapping) { + __xa_store(&mapping->i_pages, new_folio->index, + new_folio, 0); + } else if (swap_cache) { + __xa_store(&swap_cache->i_pages, + swap_cache_index(new_folio->swap), + new_folio, 0); + } + } + /* + * Unfreeze @folio only after all page cache entries, which + * used to point to it, have been updated with new folios. + * Otherwise, a parallel folio_try_get() can grab @folio + * and its caller can see stale page cache entries. + */ + folio_ref_unfreeze(folio, 1 + + ((mapping || swap_cache) ? folio_nr_pages(folio) : 0)); + + unlock_page_lruvec(lruvec); + + if (swap_cache) + xa_unlock(&swap_cache->i_pages); + if (mapping) + xas_unlock(&xas); + + local_irq_enable(); + + if (nr_dropped) + shmem_uncharge(mapping->host, nr_dropped); + + remap_page(folio, 1 << order, + !ret && folio_test_anon(folio) ? + RMP_USE_SHARED_ZEROPAGE : + 0); + + /* + * Unlock all after-split folios except the one containing + * @lock_at page. If @folio is not split, it will be kept locked. + */ + for (new_folio =3D folio; new_folio !=3D end_folio; + new_folio =3D next) { + next =3D folio_next(new_folio); + if (new_folio =3D=3D page_folio(lock_at)) + continue; + + folio_unlock(new_folio); + /* + * Subpages may be freed if there wasn't any mapping + * like if add_to_swap() is running on a lru page that + * had its mapping zapped. And freeing these pages + * requires taking the lru_lock so we do the put_page + * of the tail pages after the split is complete. + */ + free_folio_and_swap_cache(new_folio); + } } else { spin_unlock(&ds_queue->split_queue_lock); fail: --=20 2.47.2 From nobody Mon Oct 6 17:06:23 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2057.outbound.protection.outlook.com [40.107.236.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDA422E7178 for ; Fri, 18 Jul 2025 18:37:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863876; cv=fail; b=pcrWcK8pT5bwuHpCPK/KSre4T7bAbfYcOXiUI8AZYiNec8D86t3HQyikf8n2jX6XUUSIU2Al/Zxn8DEL8f6v7zlWt0Ol2sAU3+pxdjMamb+InWayeFFFaO0jyaBImmk9fYTZtQTEYaM9LulyReG7dZQj0S7YR98gQc6xtMCcX1Y= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863876; c=relaxed/simple; bh=r+BFGTV4FfkHPXI283gW4swMxOG4mTXrn2Td2lg8LIM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=eNJBL8IeoH+JxgfsoC2VL0ae4sZFvopM/1QTT6JD9UFOH6+s80a8t/0UHZPcPP3vG8RoCCDSGKzLNcIkTKq1/T2ulFLTppyIK2vsq4QmBuHMAkiE5uy6ZFMiZqpthPryNK9kOMy8V2ToDT2+pDnBZ6m1XbdF7gFDpDq7NDUNGPM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=YGd7efLQ; arc=fail smtp.client-ip=40.107.236.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="YGd7efLQ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Y4AXXItcMt3VK+REGcwxdGY582hBFEi7GtZYxgAqZSOusSZGLeBJWtCC5RwCJF11u0M8JcBFZyxRPRvAbGDSPV6wuOQ94viUdyDU61WwDUIpB/sLxKdGxPpi4Y0N+tdbtmK2tRqG/tPoIP9xBGvC+X1w1xYQ2owkni0SmvsF2aNk2bPKw2M8Kh1mCZOHyPyEvuQAdC6oQshM49lMBOfvvhv6bzeEFplfbL6aZPkptHfPAGy1a2+JrZxeCsAzk1KywQirrUqqQ9tSuX8nkvxMvSft057iAcKMFOkAZ0Z4+7Z1e8qTB+/pliJkQhB5rdoU7R+QQQ9nf5rqrfPy/2KL2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yOsB92v0D1dqhgoA8iNwuO32oHHq3oxXDI3YyeRap/E=; b=OV5qDi7ftui3LrVSpzVhLaQyDkTQjpDLmF157E75+Cyc9n0/nFaJTybk/xnt6iYMOPgsbU+3EfDK9UH4ESZOAEe/g5lJ9ZpuwCPglFwFtzQYCdkwockjXthTv1Sdj/ewB46KxbXwOtPBaqxFJVkgX24ZPHB78fIs63MwESPcQQ+9eIhnRubxQH/YqFKvNNnTkb4/v9JI6HQtNWyVtvNjOr2r63pm1PriRGZOJa4w1a0yyGOUO5m/YciPvVfkbuXEJwyogVV84uEI6Mq9INbb/88rX56mZxG8yNEaFo6lqJgvjRj/JzXuzXoV8OIEqHMK5hfZsEsKgPI7+BSZAGqfjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yOsB92v0D1dqhgoA8iNwuO32oHHq3oxXDI3YyeRap/E=; b=YGd7efLQT30usPFl+ilxmLlIvNJx62EAtk31MJwQT/pG/+ukv2NiDWDifQ1/JS7P61khY5wNhodwSXysVu3DQLZPwzTh4zfrYq4pm800P55B7O03NY7BJwdzV+5dgzfryy63bryXaPVGa0h50/tiNnDPbpoTh+Mr9jO3QKEhHwFym27oN57E1DkJ43IJNPt+LQY6Jg+0NrkyHAn20M/CkXkxOzc6ZN5rD1IQgyYupahPL1zGayRyOB5rZE6MPEi9SFmC5fvHVhg9cqi8CuAg7iZnRtCxcGaeGSc1L5tl66IIIe3eiVmKRlIG/1bLtaNxqBhnroVRaHuePt/WC5HAaQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY5PR12MB6621.namprd12.prod.outlook.com (2603:10b6:930:43::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.39; Fri, 18 Jul 2025 18:37:52 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 18:37:52 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v5 2/6] mm/huge_memory: remove after_split label in __split_unmapped_folio(). Date: Fri, 18 Jul 2025 14:37:16 -0400 Message-ID: <20250718183720.4054515-3-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718183720.4054515-1-ziy@nvidia.com> References: <20250718183720.4054515-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR16CA0043.namprd16.prod.outlook.com (2603:10b6:208:234::12) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY5PR12MB6621:EE_ X-MS-Office365-Filtering-Correlation-Id: f880c08a-c10e-4e4a-36ec-08ddc62a3403 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Yk5mIG2qePbe9BvX/w5s1T2UZjG7AZwM6De5yY2QG7Rkjg0ViicXU8CNnDmU?= =?us-ascii?Q?vR7Rpd37HRqhukCOoDDrG1zeFYG2H2YXGEVBllPgGo3inPgu8nOqh3686IXX?= =?us-ascii?Q?tciahJAQvJ30nQjCS7b60PuSOin3wvRdqBMDs+8JqNoVR1WJ3Z9khQroY4A0?= =?us-ascii?Q?qvHRFIN/XM8voRGgp/BMbj3iaLk7hoWt4OIS+KmxYMi42+YELEwQi4/qnmCM?= =?us-ascii?Q?GzULuGTSOTMtDQRQg+v3wHL6hTGe7hkNN4bupV0vjL10KCHaqtUN2gs90dc2?= =?us-ascii?Q?8+x7MGFj/9gyb4S+eqpgflXm+yIVcgf1ldVtfZbZ6GzDPIHWEqDK/SDIsz5k?= =?us-ascii?Q?C5y5LHI+DQ7kduUfFIdfIIExh+BEGMeqCSGQVNqUR5kbLFdmIm9AGoKzuXtR?= =?us-ascii?Q?Gmkep7eqse/Dnj2b6rdqvqWuZ2g0UlzEOppSEZT/1FkGl0PshCXcTate34+o?= =?us-ascii?Q?5BJKIXN/ducr9cru+y/Xpv2+YPxcVAe6vcFcnoJNKjSQGuH9QRZrRrmd1A01?= =?us-ascii?Q?ZoHSqe1xcGQd4X9JoCgqis0ld9ZO170hYbmlB3uLo1z5llZGpTjM8JxqyoZh?= =?us-ascii?Q?6sc147TK07+be395LifNBLjCdI3wBTBITNkPSGMgmho024s6wujaIltlrWoA?= =?us-ascii?Q?t9Dmp2DvN0BZdEyU9MAQjW4aVkuZuvlwaHylink2sNG2TM0uXQmopEpDe1lV?= =?us-ascii?Q?VRimTNy7z3o4BR9dPz9AktAMSJbPLrgfuxZzsWF+TSKs6VOYsHipx117zaMz?= =?us-ascii?Q?CjIENVBTMcE9dJftcoCtfrWSweIl7mEjkJvxhUOiH03JbHdwGlczrEBGVBHZ?= =?us-ascii?Q?6Ei95cp5Eu6Q0nTt6pQesnWUv4OZ5r93+Vzm2bqRPZ3szzpS+K94Oyy0bB5X?= =?us-ascii?Q?svVzaHYpqQ6V5esb2+CArNHkAAfXwB2EvfSyQiwtYKqOBwf3C8zrsqLYnZtw?= =?us-ascii?Q?VsqAPB4f2a44mFUmBqdX5NvkBIQPLzNxXUELGmk/RTWVUjw17KALUe4dZ6iM?= =?us-ascii?Q?MS7UOsqcWST3xAODlD3LsTJBXmjhxeW1Ij2pKgcd5+Ex5mSqzj9+sjJNbw3h?= =?us-ascii?Q?tXopmzJRhdUItE1MmYgmrF8svOvXGk8/+7ZAwurAY4/e5R8ErAr5Sp0n2oY/?= =?us-ascii?Q?++URVe95CKxFr7iuO2uo5u4oHhKLdXHJsTmf62hlkfwaVz0BS2u+NEKvWX62?= =?us-ascii?Q?7pONNj4PFYic1gqByEwSVtHJV4AMBB7HtvJ1Yef0BZdkrN8tuZ934cdWLkrI?= =?us-ascii?Q?l2KjFPc6UqkP4DR8t/poQ0J7MpbUQcr0Dc//58LVn7/iTSVVjJ7HmvNaxBM7?= =?us-ascii?Q?N2fAnQ8N2pML0X4mcLsb4idiV3XoKsgv//pbpA8y084Sm09pnd0I5GuePEt7?= =?us-ascii?Q?JKo5kh9Uuz9wj6sdl0up7iA49EB6XzC26e3J8ZadbH7Ynn1rvNoqapCpjYzO?= =?us-ascii?Q?V4yAp1gxgGo=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fIiuGzo378f/YuIkCViBDwSBPqZsAByaIlOIVzZ9ObV+514lYorCTKnlV0ek?= =?us-ascii?Q?Q2sxKuTEWaLv5wagCXYj1z7DlB+2VKcBz6upfX4HVrlukJ60Sy9fjY62lGIo?= =?us-ascii?Q?JkLxsGcyG8aYVQAvcCE0F3PJ6OsNM1g/0+1hOW2wnwSZ40RGqdNtb00ocCPD?= =?us-ascii?Q?aMWWuh78T+YBwsGObhmGpef7lCKKfQE5KG58hGDvddUANsWh2hUYB8tpDV59?= =?us-ascii?Q?G1OA0r9Hcr8VssnPB247qr+n1axvhBc/l4KFYXDafl1v6zNblR7/Kcz4qZeu?= =?us-ascii?Q?X+TyXFpNcOtui7y7lG4hFobj2lhXn/GNPmqMx7QRZ1F4yuXuhkDWuiYVdYvp?= =?us-ascii?Q?UmqObt4bMoCFHMTtwz3ZfRV9r3O3JLOVeReBpjKUckPWbPO/MlWyluPWtsQ6?= =?us-ascii?Q?WGxK2GfHRUVPr9KxXHNbYdqfLn5Nns2PdDQ7pg5S/SXTs+TSaaVhTva9xvzW?= =?us-ascii?Q?G4gi8QWT5Di6DMTFVLjkn3h56hIDtk09Yw9hoaSstmJ7RfmO6FojpWDU7YNF?= =?us-ascii?Q?a88lf+TaJufgTB7+fXWPjUJf7/uZDxFQFa9R7wE+zhk44fSUPtNtnhFrKnMO?= =?us-ascii?Q?vxqXowSs5UxiX4th+YUb3MkVML2gmdvorDw+KGpZVwV8VTQR5+5DgJNUj4Un?= =?us-ascii?Q?d0cNyfmG52fqk2tI+VsPkRxZPmxk9EOO5u6TSO431NHi21FiwDb8ze3sMu0E?= =?us-ascii?Q?3XbZs1cbonNeTitXauIInfeDrg6erh/VDzG1bcWzTNNXYJFJD1IR0uIo8BSw?= =?us-ascii?Q?jkOEgY6uXRFpYLbi7oCZ0ut/KFhGmrLblze/sylbqW6KD4UMqwbHOfM/Ye/D?= =?us-ascii?Q?nISaPwU1c4rzIPcXdFTi1VnDANWin7qufrMnqX0pRqBu7QmdARCE0vlvtY1x?= =?us-ascii?Q?+0wTJ4++HauIGuroDR1BxBorY6SeOxyOTDHlwbcJdMmylMi/cvDxQWv0mVgp?= =?us-ascii?Q?ELBuLCyEJpaQFJPkRAQzPdMGj7Rtn4/WeaVqcbgqs9P5YqcgdUkltE1ZQ7Ph?= =?us-ascii?Q?t9opKhToEd7AXRajlfoCZKEMd6YoAtvZZPLrfHaaFuJSFY+7JaE7vXJ8dRzc?= =?us-ascii?Q?A426IqnhDaLoTdE0rge9T6ouUDbjLN5GLCyDfad/Dkjzy9WhwNc0yOOWsMUd?= =?us-ascii?Q?K6qpN0TwREE4U6MQYZKDXniM8+0XGiMMRm6cZzpTPUJoDZJb7yILYsDzVvEf?= =?us-ascii?Q?kc8fLiuFUfIoZRntfjmj1z4YA/BpTdOcmFaMZCBjUxozgBQDBebyrcVQ/Byj?= =?us-ascii?Q?Ll5prQlYT3TSG1daxoE6vWw5F0HNn0t2350w1elyCoMkWwfCmBlf/tWYo1tg?= =?us-ascii?Q?2Li2W9Ha7vNIacAOXL5kg5SMsfC8VmCjdfGKxVlTzKML6CWgVxcUVZ3oqMXY?= =?us-ascii?Q?Rdw7Ent765Xz89DLzNtNWxGNY2ET0KBb02enVILEk32V/lR8eZyNE4QQjSVu?= =?us-ascii?Q?U/VHeRFoOc4N4KGK74SXdxcnlL8nPwxQOGsX8bCkpROkKT2aGdNRPAKsFCdD?= =?us-ascii?Q?TYuGIjBjof4v5dlilnyZmoippvD7ubOO2W9LRCun+aTgTAfEkXhc/A6Kxjss?= =?us-ascii?Q?+KsE+7cuhqLdHS6y47lX3FKDZL7GYemSrmvlEc3O?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f880c08a-c10e-4e4a-36ec-08ddc62a3403 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 18:37:52.1082 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yF5Ovdd7wHIdtiiLG6jPUAmm6y+WoaxQLM0CzEMe2kiIJIrAUivFS/pioqyUvJCQ X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6621 Content-Type: text/plain; charset="utf-8" Checking stop_split instead to avoid the goto statement. Signed-off-by: Zi Yan Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 63eebca07628..e01359008b13 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3463,18 +3463,18 @@ static int __split_unmapped_folio(struct folio *fol= io, int new_order, if (xas_error(xas)) { ret =3D xas_error(xas); stop_split =3D true; - goto after_split; } } } =20 - folio_split_memcg_refs(folio, old_order, split_order); - split_page_owner(&folio->page, old_order, split_order); - pgalloc_tag_split(folio, old_order, split_order); + if (!stop_split) { + folio_split_memcg_refs(folio, old_order, split_order); + split_page_owner(&folio->page, old_order, split_order); + pgalloc_tag_split(folio, old_order, split_order); =20 - __split_folio_to_order(folio, old_order, split_order); + __split_folio_to_order(folio, old_order, split_order); + } =20 -after_split: /* * Iterate through after-split folios and update folio stats. * But in buddy allocator like split, the folio --=20 2.47.2 From nobody Mon Oct 6 17:06:23 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2057.outbound.protection.outlook.com [40.107.236.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A18EC2E88A9 for ; Fri, 18 Jul 2025 18:37:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863880; cv=fail; b=g2XnwVtT7KWo8uVtfJuVf8R5AiVjkpVIojXHCYfKP8FVoOGSPMlInfndcVm8xbXs7pnXTuUrUKmlDvQgxAbZ+xZ9qcxKTOvA+1wRT0hQrpNJHY8lWLZ/CuvQaPlWU1TnLjqMedpdMJMv/MeOaLGCSd3cDAYGNk7NT/w5BkkHTLc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863880; c=relaxed/simple; bh=RSjz5ibPdeALF7k3vxsR2jhTkaTQAf1IHnmzyYRfW6Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=gmvUf3WO2CqgjhDFCnd2RcPoxR4UGfVdHagN8HMoBEZXKuRgCfIZulchReyHl532cgr47RtSHmMlCpAoiEscv7uPqbCwmP5NrAYhU0oHrL6F23BlV+UwkRtCBIt+kZIcsrmXjCm7C2HuVqDF2uc0XIfKCni2bzwfpuXbU4ZL+XE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=UqgiKzVZ; arc=fail smtp.client-ip=40.107.236.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="UqgiKzVZ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BVxSX9+QVm4Onn+wUARR27dA/BCkpfQ8FmoKDuEjZxXrlMfu/FuNj2xNVEnekCek7eRi+BfyruQHVNJQFllpYIyQ21nM/Ap4WzPblP7SkfTm1bqsMTCYVRdRt79yLM3NghRltrHXtBZgYWr63PNeRapnApsZpiPLqwrqxX+GpI1JH4EPe4/CRbYMUW+jg55nYl3PnCLBesWTMGhbVjyhQFBHS84hngAU+huhG7YZYsGpe3S2HadeAbl2l5cmrCUKgdeJCk71t+rR/REfMx0y3lVlAu/rIA8iUwDPz2vo8JgFptzRt3uqfpEGCwtMn+5L7exEj+fi4Y/fmcKBGzARfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l79qGOhvqKbUJA41BF4RwtaZz7sWuH+Izb86my/dEw8=; b=l21swLRius3G9BTYvTuYh2WhzBpgxt/367Wq9jLNuSVjXjLRvKolQ7mNhQl8GPSSoTf1SOC3oqTqadaPW53qZOd9WbLKBp9BbYsoyd/aD+OFqh4KdQV4X+CWx6P3fLT7d5/FJwy7Vk1ZzIqdbFlnEFuMFpQjjd7tcJZtW9ZWqkNWIbU2Udhh4Q+/iFYjhaMdU1oCPHeM6a2/70ilHAwqWAWPHQMjzrEW//hA48oGU4/8jZSbI6IR5TF491NiQgXngDud/UsM6CLeuufOZTweLolGQmbj/floG019cBBuRawCMSKsXwjAU2R22/3jDJyHcYhQvRIt7aa4H1uhe09emw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l79qGOhvqKbUJA41BF4RwtaZz7sWuH+Izb86my/dEw8=; b=UqgiKzVZjSfpK39usc1HPVHF5rdTQupPEdPrulVYl0c6BToG2M8Ak8P7URNw+O3xXw8YEPmoV/6tOlMENxkMQ13kwZWQ+z8P4MDupuO5UJz9b/bDzX9wQup1N4FF+r9OEtO82Y8ZZfvWTjy62HIEPD9yiE444no2e6HA93Kn4hSARqojhc7enzE5yFEhxFG+to3yQZlbZ3RPS9qhabifjXZaPehOyc+cuk/t2u2hBD47NUmWSw+EgLLhIHYDbFvFHE21ASTk6OTrxSUH+eOgeM+glMBMVca1xE5Jg1tnMKjSLQ27hiUMvC6R4+KVT7QjeKHHrHx3CRUYElpiSGPaug== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY5PR12MB6621.namprd12.prod.outlook.com (2603:10b6:930:43::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.39; Fri, 18 Jul 2025 18:37:53 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 18:37:53 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v5 3/6] mm/huge_memory: deduplicate code in __folio_split(). Date: Fri, 18 Jul 2025 14:37:17 -0400 Message-ID: <20250718183720.4054515-4-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718183720.4054515-1-ziy@nvidia.com> References: <20250718183720.4054515-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR16CA0065.namprd16.prod.outlook.com (2603:10b6:208:234::34) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY5PR12MB6621:EE_ X-MS-Office365-Filtering-Correlation-Id: cea87a10-081c-4701-b1e3-08ddc62a34d7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?DS2+wfka33urjFMKE8gIu13MhWFBofEra0qQSswd38jZjZ8t3vzQXuO0YaNu?= =?us-ascii?Q?PHQ9Q1j54Zc6Pgz3qKThjFgeLNILwvOq8UWWs2m5yBfarfp0t0T5RAgZb0VS?= =?us-ascii?Q?lxtlZusDEk2s01vU1iq6GtvogH89vdTJJvY27brEyxV1togEeHBlS9lD+Gv6?= =?us-ascii?Q?3PoKHQ/bzAVs8ZTag7veIMtH6t0oUIqQ3blLMw2wLbPDZLUfGU/dsNXKQKKA?= =?us-ascii?Q?kKXxtbRj/nxN5O9SDfKKmMbOrgL46dkv00cLFaudMoIbjnau1hmkkxftNwVP?= =?us-ascii?Q?GfLA2PUuXuY+Kx5RDRmHhzDlZ5Afd7ODz2LyJEkLaZkYcW34dBsWQXTH4Q2p?= =?us-ascii?Q?gMRMb887qLHB6kZNeRAojav0M9WnewQDK1ywwZi5mqzzo452w80s7uYyGZNq?= =?us-ascii?Q?hgptjk5mNrSFATBNnZCWp0jgPh2Dcu5JTedPuI7avLc3sXQD3UhtjkkT17om?= =?us-ascii?Q?LU74oFCbLT4WhNbp57XM3bqBqDX9/vnKg23b9mkEXahd1ippLZSR9ywFFcvc?= =?us-ascii?Q?mKsW7pA470//VsDADuiZfQMnLH8CD2Z/K+Kv9/KSjaNctWC5AEE4wQ41e6Mp?= =?us-ascii?Q?s/XcYIrsZS9OgFm9JVruKCPnlQvuSlygL3+JjIkiGeQ47MTSHESvy8ZtBwId?= =?us-ascii?Q?RVV21RZA7ptZyEA8yHvVkA1bLELXZ+CTkdU7r5W0TsLdGToD9/aWwJ9nJT/c?= =?us-ascii?Q?DSSaih6x9qXtk+866MDBBooGY+oDNkTTyySMmJI4v54v1a9tD36djBjA/DHm?= =?us-ascii?Q?HyHJaY6wRJXWlEMBL1x/b/NbxYBxQ9U8EqeZdemczrMBnZfiv9ySrHa9xONE?= =?us-ascii?Q?EdiJd7tQM5cY+xQJevzFHh8jbF5cq/X6JwbHcpHk0qHuQTr4yw+Fpbu2A+dw?= =?us-ascii?Q?+x2DfYvcHRdz8VV7OwdNk3XrIsZQQW7/kqy9FGAmTOfTl+Pi8UdWZ7dAWIJB?= =?us-ascii?Q?HFC10kmPze6SFvac/CQ4VuGr5qgvoWshluGRgzZeKWIsY7njJXI9qHJBd6eJ?= =?us-ascii?Q?RuigDvLCk+WvICqDBQnnqwUvCqOrmc4d4x7wRtxMN6peGGY2Du/ouMRaaDBS?= =?us-ascii?Q?Wf3NorArJaCvY7VjaeeRpqli3rEjdgBv/eCVBmLlk887i7dkzsjii8C3UNLM?= =?us-ascii?Q?IqfjYhBiKcw+Wz5joDUs86bCWBfTDCiOMRN0Us/UGL1ZCP6CksajT31XIcWr?= =?us-ascii?Q?bI9/RNuqUUDRCo2ZMDaOngzZnvszWqQ6S8Sdm6nPrnRWnFyjIknCsIRJe3MK?= =?us-ascii?Q?ElZ+W9kfEzO3UHjQYyA7ZWUQXMsRE0Sp1W1ZRQXM0/VSORYbu0/HOCWdlANP?= =?us-ascii?Q?Ebw4JE+g4owOdZ9YVXaUYKFS50vBmb4MnnR7U+xssYvS060jMN53Nz0FvSMv?= =?us-ascii?Q?6eqpJlPVWPgmqwfxwaAlJmPiea2Ycj8bMt+OknDMmQyg2+aTvnf+oRT3V0oS?= =?us-ascii?Q?BnJj4N4/PnU=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NNQv0Kwe3csay/ATnYABClRjMoINCJoAz843VFs3st4OcekLtDMIDd9hRbxZ?= =?us-ascii?Q?YdPm/NWfkfZv8KtZqwO4aTP0GXBfHC2P7GSD7cN9XDz2fUbb4U1iaTnYnxNS?= =?us-ascii?Q?zsxvzs+c6bFNLY0ejmb3li8UEeAlAuA7B5bLuOl1YH4U+oSrjYnBbMl/P3FN?= =?us-ascii?Q?cIsSTBK6/SH21uci7T8KZfRsItsjccwIyDrL4FgArPTGsh+ITW/4mYi6+0OB?= =?us-ascii?Q?gyXLplvcre1XFa8IrKKTKedUhghIFaEolzKOTpx38/CG4+HgtUdqRuakUg/0?= =?us-ascii?Q?xd50X+mq7hYsshXe+BBZIkK8CFVz7H1uWhy43BMHm8yu3RxGCZRinxQzJSuS?= =?us-ascii?Q?54r1mFnhaU/JoTO2UPFevFTb+DLZP9Ksk7VXabTgUn3ZQPhTz9GFpiGc7XAs?= =?us-ascii?Q?o0YPkF50FVYM5N8Ll84pgLxJ7V3lOTRIzaexFDyU1VpZoSCFUr6kMvSvMqby?= =?us-ascii?Q?xzuG9mMVXqxc3J+TYjsOm7LczW5fga3HJUr4GPt0wi9ocmaJc8wJKPwGikJ6?= =?us-ascii?Q?1mRbqpIT5gK9rpZUnrM/Vck5DxpxCR8EkBdbMKHgq+mcnO9f9SNvdhh2edX7?= =?us-ascii?Q?aOrvAye1yaOtJmMIFTAyQZeky++OAjikEqHEG7ywAqIVrgfjEdf5QrVFzh8+?= =?us-ascii?Q?ziXl9VALXFCqBdu44wk5rkM2ogA9wi/ueWnvIPoWQCoqBCQkAlPy1Bwc+xj0?= =?us-ascii?Q?cMZaq3EcGT7Mzjp+LMoLFeuShg8UoB2qDz4HVlbnvM9iHqFDwMPURjJSehoH?= =?us-ascii?Q?UsQdeB2pcgE865ANeEk3x6Yi1fwsSSyhfAWleqZyCW1VGr8n7CF+DVCjMzGo?= =?us-ascii?Q?tcoEL8GxFnyUfxKBNTAxTOTiSlUBW/qjqTxgT5Ni5I9iQnAYagm7b/+V6MFS?= =?us-ascii?Q?PdpnIycFifec+ZcnLrjPT1+nNiakxR089AbXEWFyvAmXNBudVMfMBnYk9MXz?= =?us-ascii?Q?zzP8Hy2e8NbJShiFWDTAPClL9oZVGAdUwJhx5YoTJjACD0EGKpNTEPkkvwGq?= =?us-ascii?Q?olBnj62O1zTRbBPc9t0jzuZyc6RZSRVXvGNIxt7vYpAFlt3QFyi7LEZZ7XJU?= =?us-ascii?Q?mjNtbX789w5MstW5bY/hIBR8dKyprOuzfdRc0iR3tRVVRMqju3bWkPUiHG95?= =?us-ascii?Q?ikbZ+IOkDXKKQp1xbR1QA+wHlRi9TQ0p1H7XGaNfOFrZIfpYU6zddWbsz/UH?= =?us-ascii?Q?xAF4ITsqRzZ3bG1S1sR782suafKwfGQ5bv/5RSL+sS7Umn8wRlUerucEIymg?= =?us-ascii?Q?w1IMlkLpjq/dshAf0PiBv2Fx/88eigyVB20olgOLi0f+JcBae1YZYdGvXP0j?= =?us-ascii?Q?Ld8Jvb+eaqAIw1cKYCIACHpi0qbLTBho/wR8OtjeJQERFazM1CDb4piCn7kb?= =?us-ascii?Q?COB61SCxZ0bi7nQFQiqWr60rEaqgpAhLI1LFH78LhodrkoGTrVaRN932dRcR?= =?us-ascii?Q?6SM13aJWcIWobU+lUHtmKjy9LWD0X4529aeoGs0l7q7cntaU078PXOg4WNdY?= =?us-ascii?Q?N8rK2foikboJvmEjLFWd97+1ySgJG9WeI3dM1sjhBW79jrW+xO+iH8O6bxcJ?= =?us-ascii?Q?F0SVLkEaq/oDUUg900V+Ovlj7pDrpAdKPtTOZufY?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: cea87a10-081c-4701-b1e3-08ddc62a34d7 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 18:37:53.5025 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: URk+AAZvxekPVvcEqIml/23Ti/C/kQ2xwKtIErd1oeUzcxaMc08bjZkzhAZ16bx9 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6621 Content-Type: text/plain; charset="utf-8" xas unlock, remap_page(), local_irq_enable() are moved out of if branches to deduplicate the code. While at it, add remap_flags to clean up remap_page() call site. nr_dropped is renamed to nr_shmem_dropped, as it becomes a variable at __folio_split() scope. Signed-off-by: Zi Yan Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 73 +++++++++++++++++++++++------------------------- 1 file changed, 35 insertions(+), 38 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e01359008b13..d36f7bdaeb38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3595,6 +3595,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, struct anon_vma *anon_vma =3D NULL; int order =3D folio_order(folio); struct folio *new_folio, *next; + int nr_shmem_dropped =3D 0; + int remap_flags =3D 0; int extra_pins, ret; pgoff_t end; bool is_hzp; @@ -3718,15 +3720,16 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, */ xas_lock(&xas); xas_reset(&xas); - if (xas_load(&xas) !=3D folio) + if (xas_load(&xas) !=3D folio) { + ret =3D -EAGAIN; goto fail; + } } =20 /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); if (folio_ref_freeze(folio, 1 + extra_pins)) { struct address_space *swap_cache =3D NULL; - int nr_dropped =3D 0; struct lruvec *lruvec; =20 if (folio_order(folio) > 1 && @@ -3798,7 +3801,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, /* Some pages can be beyond EOF: drop them from cache */ if (new_folio->index >=3D end) { if (shmem_mapping(mapping)) - nr_dropped +=3D folio_nr_pages(new_folio); + nr_shmem_dropped +=3D folio_nr_pages(new_folio); else if (folio_test_clear_dirty(new_folio)) folio_account_cleaned( new_folio, @@ -3828,47 +3831,41 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, =20 if (swap_cache) xa_unlock(&swap_cache->i_pages); - if (mapping) - xas_unlock(&xas); + } else { + spin_unlock(&ds_queue->split_queue_lock); + ret =3D -EAGAIN; + } +fail: + if (mapping) + xas_unlock(&xas); + + local_irq_enable(); =20 - local_irq_enable(); + if (nr_shmem_dropped) + shmem_uncharge(mapping->host, nr_shmem_dropped); =20 - if (nr_dropped) - shmem_uncharge(mapping->host, nr_dropped); + if (!ret && is_anon) + remap_flags =3D RMP_USE_SHARED_ZEROPAGE; + remap_page(folio, 1 << order, remap_flags); =20 - remap_page(folio, 1 << order, - !ret && folio_test_anon(folio) ? - RMP_USE_SHARED_ZEROPAGE : - 0); + /* + * Unlock all after-split folios except the one containing + * @lock_at page. If @folio is not split, it will be kept locked. + */ + for (new_folio =3D folio; new_folio !=3D end_folio; new_folio =3D next) { + next =3D folio_next(new_folio); + if (new_folio =3D=3D page_folio(lock_at)) + continue; =20 + folio_unlock(new_folio); /* - * Unlock all after-split folios except the one containing - * @lock_at page. If @folio is not split, it will be kept locked. + * Subpages may be freed if there wasn't any mapping + * like if add_to_swap() is running on a lru page that + * had its mapping zapped. And freeing these pages + * requires taking the lru_lock so we do the put_page + * of the tail pages after the split is complete. */ - for (new_folio =3D folio; new_folio !=3D end_folio; - new_folio =3D next) { - next =3D folio_next(new_folio); - if (new_folio =3D=3D page_folio(lock_at)) - continue; - - folio_unlock(new_folio); - /* - * Subpages may be freed if there wasn't any mapping - * like if add_to_swap() is running on a lru page that - * had its mapping zapped. And freeing these pages - * requires taking the lru_lock so we do the put_page - * of the tail pages after the split is complete. - */ - free_folio_and_swap_cache(new_folio); - } - } else { - spin_unlock(&ds_queue->split_queue_lock); -fail: - if (mapping) - xas_unlock(&xas); - local_irq_enable(); - remap_page(folio, folio_nr_pages(folio), 0); - ret =3D -EAGAIN; + free_folio_and_swap_cache(new_folio); } =20 out_unlock: --=20 2.47.2 From nobody Mon Oct 6 17:06:23 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2057.outbound.protection.outlook.com [40.107.236.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 832042E8DF2 for ; Fri, 18 Jul 2025 18:38:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863883; cv=fail; b=krFWcpfdw6KLZupsYPa1q0vOZ/OoWTwobBrZX7lvdD+3Vus7aTphcr5tKF0Eu+o90JtQt5dNOFSA/Z+XoEseF3L1buJqlEi6ovgO5jOzbpteoGW/4vjyYZGADWMzq0gZfOcWysgN/NWpAFDJOBZO3AAKhtwvSzjpF25AR6oLQ2Y= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863883; c=relaxed/simple; bh=n4ldQO6IC3XLk2s6ig0BblQTTZRXGMMAaRHzk9SPGOI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=BEUl+IA5uAg6p09uQ28H7szhRS3raEHw73dApGabvM44kQTC/17+PzOyZP0TqJXshVzCxdxKpEr2LA0h2DDWrV3xJQUCuESiofKFrncLu0IYqXFHXlY1jiWibUInhJRIoz3/+Y2XufjURZWTt4pESf4fPUjtsrrExjNOA2bruTg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=MxP1oNth; arc=fail smtp.client-ip=40.107.236.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="MxP1oNth" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UfBuNk6IYkcHhurFLlQb4efbdHdyVcLsYcncvfQXabT7/bqqy30goVuR4ZtpRjIFJ/YMBIc4zeEUcLUre/CNrzjO4BFk6dqupfIPHrXzJQ6aqz4l4uvnbLRPr8lScAKtxwlcs0wL7WvAFXO4Ad9kkV/C9u+ysII9NHziqpzX82roqQx3TJAxrkbtxXkg+Ib53TPCJFU3r7S+lSp1ZeWyGFQuYzIC4ABHW5+yGmfW1VGe3g8db+gacpc8YG+GIS7jNaZaOaqdp8AzTJj34U86PioGwOwEQNYNUMw1/GdB5wN/1Fh8eOjv/dLRvS2xoC5PV7AETe6v7mohsm1LXk4j8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=txUXqNC2N8SSNuJurX1gkuXaP7+FtImoaZJkxMD7Li0=; b=Yp+pX+jBco6Z+42+dtw/QCzAzNbL6MKnecCorYAJ4QNNjlrReoBa7OpU8Tq2XCuziu250wd2WE2mQy2MVYQcUw1porqYJlEJquDlrlt0Rk90U5eSive9I3oM/EbBivDyn/8WLRRnq3xRcUOdNTG5aRzLoW2XgBZgvuoquW32/fHPQwODZ0MXni+XFI2Q+qpz6fY6Dt1ZPhvhAQnxEGG6Wnce3ksmPHo9U8uybB8sNMUTm1jFoj2hcz1ve3wlqVv+x/OrQ3/d6iNGX58gG+m0LH2pH6pt1AaTlcDsdUnTzmaGhkjEmEeB4tjuCIwbR5wLAJKS1qNSCqN9NthzONJR3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=txUXqNC2N8SSNuJurX1gkuXaP7+FtImoaZJkxMD7Li0=; b=MxP1oNthQAt6ZLAyWm62PWwAmDo0qBkRGCgdgffdw7t8Ts1H58ZkOEkg/0HuTDkVk2MPF88HKdZle+l4xcHebXo03Lpuf96j/irRHGJJbDQTEfNI7FtpYGFZyW2pBoQpzM9HpRPbQ/QtHY9/GmliWN3qBQD/Wo9uqWCwXyPYAmO0EOmAgWsuxTBC1yhzwNQMomjknei0ZQLecMpZPzro6rX7a6BbWMIPrV8MrBnlEaWyWIk6X25EbLA9gbX+j1RO6gFVp+LIjmknPe5OqHFAWwWis5i5gJ4D0vzL8qPgG1mkwBBNgja1j6J9WjuuF/u9q+56bFEP+AEi9W63vTgvGg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY5PR12MB6621.namprd12.prod.outlook.com (2603:10b6:930:43::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.39; Fri, 18 Jul 2025 18:37:55 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 18:37:55 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v5 4/6] mm/huge_memory: convert VM_BUG* to VM_WARN* in __folio_split. Date: Fri, 18 Jul 2025 14:37:18 -0400 Message-ID: <20250718183720.4054515-5-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718183720.4054515-1-ziy@nvidia.com> References: <20250718183720.4054515-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR16CA0064.namprd16.prod.outlook.com (2603:10b6:208:234::33) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY5PR12MB6621:EE_ X-MS-Office365-Filtering-Correlation-Id: 08d56db4-72cd-4099-29ba-08ddc62a35ba X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?kcz1kfwzwP/gpaar+hMr1Vk0O+KrwmmXl9TM7Ol5KWTY0dcrzidHTq+YpHI+?= =?us-ascii?Q?v+0rKndQfglbfKKkuPAS/32vVIix4yDOjUpKQDCdd3+eJvFkHLFhUQZODysn?= =?us-ascii?Q?fznHhluyULmY8JrBqK0uy1r0UndLq3+p0lPcgftjGbimnnxtme4tBQQsH2yX?= =?us-ascii?Q?XH5aUnBBBDXKH4y3Hi5w7wHZrT49u9GIT9FRHH7nRegIaytDam83ZGwDVoZM?= =?us-ascii?Q?EuN4wTi5K2SObZPnmfcEGSmMehsZs/MBYwJcAN65P6Omv1ffDq/vzNFmEPvR?= =?us-ascii?Q?zX1XKH7hVZC3BoKcAOo8hNLiHKdUu6zYkaG+SxH5sfzC4cdsuu6/7cY9SlCv?= =?us-ascii?Q?rPdsn9upC9n+qGtjKfzpISxsyV15Z37MkC3zemJ1C5iucgLcPXtnIcrh0li9?= =?us-ascii?Q?G8lBqfwhtc1w1YHhCHFbVlux9mNIJ8HN7HHiCBMmIzj5zHB5mUuEWxSYPIVt?= =?us-ascii?Q?MExBa/7QYnefVEfl/lgNSYCVO5cw9eABcd1bTA3IR4EQxKJ3gcBKgVgo4GPt?= =?us-ascii?Q?eSqImhP++EUsbI1o2TseT5mUemUEnSUgfWEsRJwxuBZrsN6ouUbnJ65FpaRt?= =?us-ascii?Q?YNwYQftJJAfD3OxdDfbDdWzYB0/HKhjctm7NhGEzqgofgyLNPRautKuyyrit?= =?us-ascii?Q?EvO+5Yj1gLGRAt6ZOhJgIPOjVmsSCFbswwwBHXZWnRQ1NsUFVa4q4Yhkqcw2?= =?us-ascii?Q?SagGP+Cs63U7JNNEgVFD0WU5MKiDaBVaBIbxHQMN7IBwcqw+ALQvkjAwL1J/?= =?us-ascii?Q?qx0TeEfXmXSd2pkY0H/dNXWa8+Dm4F+DYH8X8etEqWg4wPhyZeqn9h4x0JXc?= =?us-ascii?Q?1OfQrxdNlqr+/UprhffiYHE6iSrzFc+CgBBHGxjqX2oqKnOrWIQvQ5VpfIQ2?= =?us-ascii?Q?01VYg5cb5KE3VrTyP0Ky5HNw27s9JRNzjg343L9oF2xCM+uXjFjpqPe7yOvl?= =?us-ascii?Q?A8hMh8LM6tPGMmTRpbUKaI4PZMz2bY0DNFhTWqiKCqWal1WwaV/EoDKpKFep?= =?us-ascii?Q?QqY/TJhaTeqvJ+YnLAApwtvxSNk+fQ4KpDwBUZeQ37DY6kL7F6+5WL2YI+uK?= =?us-ascii?Q?WuGjyMP/oNQ0W1T/AZda2flqJCKMoD8VhP5/1h140gHw8FSoC0hG92g5T5+R?= =?us-ascii?Q?MjlUFQezIBhn5pyCQdRUfYWKBiUNK43YK222bevwYcgRd1GV2nRiKl9yZ3Ov?= =?us-ascii?Q?ie9NMGdIxIwZN1tk5uWoCiP6Jg075ECh46ZdtbNAdxwy1nmjTX/YwofNTXcn?= =?us-ascii?Q?ZVA3R+jkVRYT4F6VBezv5jKlyN9N2BBY0vn+Zx3tJOQ7xZcrhxpe7skkD9rN?= =?us-ascii?Q?VAynb7NALZBR4f/z4kqe8sVkAtMCalIJYfLRY4F84cZ6GQcZBahq9UqLJnd6?= =?us-ascii?Q?fjFMM/RssRNN3azJM5cr3F0Sa9JXrRMI8zc8YPoGDlyxzNfD/gyFl0BszqQv?= =?us-ascii?Q?VWjrWkzoBhE=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?4Ew9pkcMcEBU58cLK41qjqo0lalNXU6okq6KXoQFC5tqXSGhXhgWCCt8R33s?= =?us-ascii?Q?MANN6+LnAQD+goyuCn0g9pQaELiPA0RGAaUaddXlFxJmBxSPUFHiOYpeucok?= =?us-ascii?Q?SVF383wmzFFVJOvZTIFBmJL/yUa+nsCV1xX37VMhs6pPOXHDYpj7SF2n1c2N?= =?us-ascii?Q?/c58o72nUGYdISFm5F7ghSWm78EzyEUfd2K5kFIq4QScTKk1HzUDIY/JeOrM?= =?us-ascii?Q?IzFfqsOj9AllfRDJ2ufnPh7HCNfModw7iUiIccw2OUz2Ce5L1qRP2nIoNkR2?= =?us-ascii?Q?sOX1+MdYaq4mKcpub0Fy17tdYJoEoMoTz1XDrGM+JScCf8gxpuwwkoj2oK9f?= =?us-ascii?Q?mi4THuNIu79Zmc9jkEI6dZwic1Ub7NnVtImoBh44+jnJiVEfN/LkpPvmK5Kp?= =?us-ascii?Q?X/YvCm8ZXrT1g1x+a1+KwVKs08pU+WOD1XkpeEe5GuTNDy9D4xfLGGokQXiI?= =?us-ascii?Q?MRmb7bIXBl3WNo8eRBxqDGoPvRwu13mphs/mKzB0eXlRKMvgQet/qdnXcp7g?= =?us-ascii?Q?Qt+gfcmjAKZBECLM0wh8IwkNp7N/0hrbzJ8AjthhRt5fKFrximA1Dcv6rcak?= =?us-ascii?Q?gt3PLP4K3ESciF+8hHFLJr2PEnMpYujWEn3OAjgROnePhOGCt5xb5UvGeU6z?= =?us-ascii?Q?OB0PotrFdeLkAAjKIh4ZdwqYA/WS1E57Rudu8JeHaUy4uwvqGqj7PnOsPzwJ?= =?us-ascii?Q?dqhqOZ6jU77mNKnFkFw2WEa0DCTi0E7BrI+mhQB8mcL1mv82LvW+A28S0K7S?= =?us-ascii?Q?QCiDjIuwPgr7q3IDsTn1Ln967NlKVk2DOR75mkZb9M7jVAV3pn5nW65xmNTo?= =?us-ascii?Q?1uccb4C0FNGd8p33C5tTUsEZtGsI9GyMekE4SQIysOCQRSOwPzcLzDtUXNF3?= =?us-ascii?Q?UjsVUzteEe56Z+Z1QdQc8Cen9Z2PEAwsTxgA+4dn7kaLLlUHM/7cy1xfu4tS?= =?us-ascii?Q?Gr5IMn7+ailREtN1aFMoFAZGGPDHKE6SQvF0l3uwX15sfEXoOPG42jh7rlbb?= =?us-ascii?Q?4q3DL3CtngxNNqnKqjyGC5BvfRu0LMjeF6SzGrf0qBsLACgInEQk5zrD9DUI?= =?us-ascii?Q?rEeYxxegA5aMa47IrrIhoRYagmvLkrLlL6pBMOfcfRZGaAwDHzP67qUA0Ojo?= =?us-ascii?Q?jyYXfNIvZwxVWOBIaz+YdWpMpugL9ARvQvGzJzTuAyvXNZfOqg6DiPI5mhkE?= =?us-ascii?Q?xsOw4e1qKPEsNC8g7x8rns/4qMLPfd5AkV4bFcgcS3yx4+mszlxBvQWfeCsF?= =?us-ascii?Q?xezlfnvOR8dsaIDh8TdXsDI4+GYdNRfT9jkY2iKeD4SB95KXN3DYviN7lODW?= =?us-ascii?Q?uzFKLCgcSLtixGukR0PwUiMyG6iDVR5GKjEGb3ox5Ip106MizuH3j6yda/Fd?= =?us-ascii?Q?KekT4+cafOgxg8Kg3sXHOV3C8m2RHOdeDrA/Wv8re/gI1Lv9LfA7PmV94Nkv?= =?us-ascii?Q?e/HFY/ukpU7piAsj0dEFtcF6r1Y8iyYSnamlMVR2hUY1q0XzyP4LTOqP66Yv?= =?us-ascii?Q?kCkcQ2xvVosG4xNPi4UlxHOoPKVf0vE9Q/AzCkwORFVzcA5zH9Q7wVk6db4L?= =?us-ascii?Q?naQPmOi1ImEl5nab+iRXZLhypCLBw97KaWGnAByM?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 08d56db4-72cd-4099-29ba-08ddc62a35ba X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 18:37:54.9859 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KJr8AwuDxDGhjHvnuwqBBU73hrpZdaGNV8nHaOIjKAb3yUPpI08QvpW8DWLkTQev X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6621 Content-Type: text/plain; charset="utf-8" These VM_BUG* can be handled gracefully without crashing kernel. Signed-off-by: Zi Yan Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d36f7bdaeb38..d98283164eda 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3601,8 +3601,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, pgoff_t end; bool is_hzp; =20 - VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio); =20 if (folio !=3D page_folio(split_at) || folio !=3D page_folio(lock_at)) return -EINVAL; @@ -3766,7 +3766,11 @@ static int __folio_split(struct folio *folio, unsign= ed int new_order, } =20 if (folio_test_swapcache(folio)) { - VM_BUG_ON(mapping); + if (mapping) { + VM_WARN_ON_ONCE_FOLIO(mapping, folio); + ret =3D -EINVAL; + goto fail; + } =20 swap_cache =3D swap_address_space(folio->swap); xa_lock(&swap_cache->i_pages); --=20 2.47.2 From nobody Mon Oct 6 17:06:23 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2057.outbound.protection.outlook.com [40.107.236.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3338F2E8E02 for ; Fri, 18 Jul 2025 18:38:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863884; cv=fail; b=WpNNe/3lWmgSxKUi33MHrTVnri+quJu/zANdVoYbpv0+fo6h/uZc6kqkkd1sloqJWvNXGXWqXFRhdCK/P+YhJdIfy153SFsHZnnhx1Zpr16xkXCailtmACAcqhSyPhHgeHF4dx7x0nKNi0Z9jspEja4sv2QTv80M7NRjqBJyB1o= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863884; c=relaxed/simple; bh=iPualLxNhLiS1fxxzOfpM1GnXojzTc/ROt/68dfH3fI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=gr17siDQfWk9NQzlPs6G2yl4JRJWz8oJevx75XwvTE6N7upQj2JHNI7SGgVh7V+Imnw822UAMl5KDSaIFv6UDZ6Q1/OjnUyPRJySfEqUAK45ldmzLoTQrS5aDscUtd0ywITFYty6l+q1DLV6ZXQZU+idpixFGGRK6gFEVWa3a5Y= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JP25p0Xc; arc=fail smtp.client-ip=40.107.236.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JP25p0Xc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RW06I9fOH259Z/M8j+XgNK8z3EN5QFED5urEheA3NMkEwKRvFzBilnfBZbwW8kbY9V0IXgY0bph3bDEUFhM+sp3zMQT5doq8S9LIfn4h/J1Rlce2UWZgNShjxq693bT3B3RU9sS+BGju2K+Xw1qBHrXtg2uWGXUKFuMyjOHMMoFfUy1rU4/gqwIw1ZbsviE6HeJF5hG2HvkMiK/NTjw9eWpwDm1TaI3cEXAipEhFnUqxpFM31PKRMZhVISn55cHDpm49xyYKjDn8K6by1c1KoSAJUwwAZQ08FQ5uzJ7DOBohHYc9jxkpGjL9kuJY4X5jez37KYXNhAGBBVOFYjZ+MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PvBRIIPiylHOXYtdkVDILflGtLijXwd7kLOSaLQi/fo=; b=WldZPJVPizbGUi0A1Qe+aYztyxpO4sX3vE2mvdAoJGquVH8eo9cL2LT/ADLB6AYoDTAiZhvIEhJnMFnAb5L1cdPhX7MBZogpv+oyFFMkASy4ao/vkA6GZhyhso7C98jz8gkXer2oiRMO58zPATRnEoEK6VWGEWuyPGuv+QgeTME9Ow/DzXbYcazt40+D8ge4z7TzAFMYkaUCKr19CJyqPHVUQ/HdBQG2SKGTvFp+wUBgDgjipq50mCEF9bUCa4WixqGb80nb8sUf3ebbqqAqsm8fe3CMGkUb7hAu2PLAeqAPpaHcIbxxpKv7Y8yYB0vXYdFP+j0xEizltwYBL8hfnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PvBRIIPiylHOXYtdkVDILflGtLijXwd7kLOSaLQi/fo=; b=JP25p0Xc+barOuUijJFyyORRllOf90DcsHuladyPptzu3IE0BFs+oHzXv65X3SuG+YntpsJX/bJjHsq+oK//XcfH2YCv+WVxJOvSCIstKs6PTqGH9apDpYjla3PG/MBMUn6fIWx399paYrCHMRkM3JRIKLjcGuUyNy0vG5le1yMtTOk+je137yNJhggtuISkmB06Y4HmUmdJz2uBHrd7C6ezn/dteRoWeLLCjcYbOv89HyyzMN+PEq3WEjFBmddiuww5eoZ8XMZQe8D/DP5bRqC4oPAzCIzCNN+jGWtdtGcP8KZE5uDaGkNHVR2ScqImYWf7aMvSNbW4xHjFy3hiYQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY5PR12MB6621.namprd12.prod.outlook.com (2603:10b6:930:43::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.39; Fri, 18 Jul 2025 18:37:56 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 18:37:56 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v5 5/6] mm/huge_memory: get frozen folio refcount with folio_expected_ref_count() Date: Fri, 18 Jul 2025 14:37:19 -0400 Message-ID: <20250718183720.4054515-6-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718183720.4054515-1-ziy@nvidia.com> References: <20250718183720.4054515-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR16CA0041.namprd16.prod.outlook.com (2603:10b6:208:234::10) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY5PR12MB6621:EE_ X-MS-Office365-Filtering-Correlation-Id: 109c8a01-600f-4226-d766-08ddc62a3684 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?28wwmuisq1MNmbTGogbY1NNxrGG7Z6WGmwd747LGovAaTrGcyA63n3nofaiI?= =?us-ascii?Q?0G65Deg+sE1/0w+X2sdWS0l+Jfmpf02knoc3y7GyJUJB42MiaQtDhBaGDUIk?= =?us-ascii?Q?wEXdl3/9UXZBWa+GYHl9hEAe9e9edIWv9ITgY+ckt/ceCN6v0bR+uMoJWsKr?= =?us-ascii?Q?vKfCAMNY9mSOsAH2oiSF74H2CKqWAHYeY1Q1XDm5blfIfhu8gZAQSOTPfwRL?= =?us-ascii?Q?DjBgAiDBe7W5bHgDx1ZSI6wdygFa0x092R2LbSFw0Qiamzjhi+TJ9RaMZuat?= =?us-ascii?Q?lvG9HLrQ55CtF4NbceEIZY3btwU2GQdXV2YCYVz7+7lkCvb2BvnlLHxQ1SPq?= =?us-ascii?Q?4LoYmJ3v59iYNsasKh55Pf82LUjRWv4ajLOg1yathwBOVLqPeGkNyRsHYWyE?= =?us-ascii?Q?3aSRTe7Id8vz6BU56RvrDP2uMK4lfOCP/TkNGBto8gkfO94itRDqQeFlkOWR?= =?us-ascii?Q?IzXvhM+Ju8xLPrUGUf03tfZwai4khHfAn44Y2FVizIt7zMqBBx9vy+IW5qSW?= =?us-ascii?Q?G+xIIMd2avGWGpvRBWiyWaJM6Wmocc1M0qLNRNa3qQo9t1WKDtCC191ni7R4?= =?us-ascii?Q?vcnMvyjL1ijw4FIxZaFI1opmKpLHKHGjsgkwXeDHxe9kq+it4Ijp1tO9F2rx?= =?us-ascii?Q?dryACq/3p1alQTEN5UF8OaTpUG1Vap/ojb4wP2FJHWhyTkJPjORoLrGvsjo6?= =?us-ascii?Q?cuuZ2rFDjuQm+XKBZ6fn/5LQ0dB9bO6ROm/em1Wgk7M4jHjU7CbDWPmYVzqD?= =?us-ascii?Q?y5btBLmS4fo3Dvyzhr4XcyHZH2VZa5SXalZEW38SovvYnJ0WH+U2ZFX+4klU?= =?us-ascii?Q?Qwj5lmV6sIWSFIIB2SDEqE2IOMynwCjhXlw3XTeQIb/Q3n+3NJSCv/BBlPE6?= =?us-ascii?Q?lFf8W4/oG8n/BXMVuX6zvbp6lLC46f9QXJG9ibs5mDSXFGcLG18JzI4qz5Xh?= =?us-ascii?Q?r93wcAt+MWSqVz2+cXK8z4+RXccGypNqyTWtjtYHBWsNVmBYXGuTn9Jza9pV?= =?us-ascii?Q?wix7dbQ9J7pHBdMpt2R/tBV2HWH71Bres5ElaxnYrcJVkdyT6xJ0xGXPY01B?= =?us-ascii?Q?n/3qE53ifyWlxFJsdQt8HmEdp1015zW6iwTeXImhbVDllNTErh+0Co+zCM+N?= =?us-ascii?Q?avQX/yvlWEk/KU60zzDsmZUF5vbS8FqjjvP7al6C6btrh5IwtCWRCo4P72Ma?= =?us-ascii?Q?+t1Vd+luROuWTkrj15RgzL7bJbZlxrRJqaPB5Zb1hctqVq8HmqVMU2iqAAMU?= =?us-ascii?Q?9VeEID8xz1eie75uP7M8toGKmLwNL9qhnUgW8QK24YYoljB97KT5T8uwX+4F?= =?us-ascii?Q?e5WguxQcQ0ylJICqrVKXcGGeNOHY/ANoVE2inWlakO2UQIsLBFIoUHdEchuz?= =?us-ascii?Q?SOWVImAlqqBQYZvazkNKp9wVoAIHQpb9EsRm8VKcxcuHKUERXimiYQoL/smN?= =?us-ascii?Q?UfzBxW3l3Is=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ILFtWkENXU6H2wjFIfI9olYB+5X8X9SpxWem9Mjro0AC9qlib01mhn47CoQy?= =?us-ascii?Q?2Wp09hjlNVLSfgNIT1CgdFuRTOFf9CweG/8Re43pZRvjk+A6v1ZthRd0Acme?= =?us-ascii?Q?EUWVD5QgzActfMwwODqFk6UgVCAgQRynjIogFPAlc2Cg8ASxCEtMoL/bS0kA?= =?us-ascii?Q?vEpUoGTRMJerPJdKBz2M8pMFcJ8dmuyacQgQBKrk1cKHjsp5+k5u/0WxnfVv?= =?us-ascii?Q?8JCkLdF8s9gRTPBKsQ+bF6IL9BP5aWMZS1PxKBeLlZLBVZtbjryKyF4X4GbD?= =?us-ascii?Q?WxLFuOqgX4VHnG7F5QUrZMM5EQhxlu2drUBFD4eepA+DDyACF6uP5LzaHg6B?= =?us-ascii?Q?kSetUsnxHIapR7mZd8plPq1sk2zgGhJUwpED0prIBD/T4UHhbHSXVGiYfLhl?= =?us-ascii?Q?tbFZvTUNnbMTiawVaGme2yPYIi5PVxzvb7MeZsYgBjg3a721w5oO869ZzvTU?= =?us-ascii?Q?utGsKWvCziyDqI2A9J7XnJs//lt1UQUYtK2NIhiV/cSlx1/QTFONAlCRbJqs?= =?us-ascii?Q?7Ot+3P5R+ct61PHXW6wRCifiBM9kyUvZBdBdB4fVZCKdDdzyEjRlXSg43iXe?= =?us-ascii?Q?t3gdj2I5e0EP8Bm0EvMGrJVwv7CbA+cHa+I8vawyc1tOQgz/kthPojWbCnUj?= =?us-ascii?Q?yW56h2EQv4ZboFSgOqi0Hlxtt2ocKeFTcYsQdBVKIY3ZafJ1DyASEQ7Z6DbY?= =?us-ascii?Q?BKyi5iDuueHbp2v/yGxgRJMZcDPzd98DazQZhuPluO/WdpyTHYzIAMufbo1E?= =?us-ascii?Q?C02qXK3Pft02jqEikLw/U+paPqKaXQh2axsbY90CXqN3n3lqlpJ2ioRQulAi?= =?us-ascii?Q?itLGADrkxscqn2Ji+dRF8Zt1+UIdr6mj4ssRyVW9CZFoi6Vy1TpmEJTl3CPs?= =?us-ascii?Q?OrNvZxqXt90X745xXulj3lKLtIaUTcSCPAGvZ4ddVeO/Wy7eg8qKN/XGYqwL?= =?us-ascii?Q?PejFrmEOmZQ/Pjs3dYM3YDUUuRxEGuJNQV4IKmn6m2uvOHoTy8Y4xQWPeyKY?= =?us-ascii?Q?qsa2X8oV6qsBDSABp+3U+d2AbAQ927uDObY1Ow8yoO25rI2CTNZhx2MisuDK?= =?us-ascii?Q?ii4/R+90n9fIZ8UY8eynmOoI562gB1REhS5lsS/gOvqxmddBMibQiWIh7NPG?= =?us-ascii?Q?och+fgxVT3l6Uh3ooQ3gEkbVbBgxJylAS+nFFTtUMuQtq7CRc3bxyCR3+8z8?= =?us-ascii?Q?0QC3t4IkGq0q1lo8BCF2R5t2lYL/uPnE0gEBvdK7DEyhPm4vwVndM34vIpQ/?= =?us-ascii?Q?qfYVfqJoTZoVQwHrBmz6YTtYbqX2jOYk3LayYxuolDTnwfL3tK6ZB0Oq31O4?= =?us-ascii?Q?sOv/aTrU0MwBy6i21OS1NeYhKB0DPRwKq4v1lI6SXVL5UA8ucolvQedfGZXn?= =?us-ascii?Q?3yQOp3zUEin+OHlX34fUwWKRntwc0GqkrO843KyocPCUpzzExwXQBDP45Vp2?= =?us-ascii?Q?az4AOMTXZ/LXaiziVakNbK8E5i6HdZMrAYpE96ai5LDpPGtrL2dxvQUJIi4I?= =?us-ascii?Q?kcld+gvPOnofqyOIa29tqV7mSrwRFnmeGBqip0X3PKwabBmBRxH6Y1TwxFaM?= =?us-ascii?Q?H7JSSt3y+f/PxKJA4Hh+CCbNmrEmowouycJBJLYt?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 109c8a01-600f-4226-d766-08ddc62a3684 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 18:37:56.3277 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XhDFxVsJ+qC1MaNPhWu/Gk9/BUYkKlI266Rhbd9IfVju0WES82cjteABBw8kwZhV X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6621 Content-Type: text/plain; charset="utf-8" Instead of open coding the refcount calculation, use folio_expected_ref_count() to calculate frozen folio refcount. Because: 1. __folio_split() does not split a folio with PG_private, so no elevated refcount from PG_private; 2. a frozen folio in __folio_split() is fully unmapped, so folio_mapcount() in folio_expected_ref_count() is always 0; 3. (mapping || swap_cache) ? folio_nr_pages(folio) is taken care of by folio_expected_ref_count() too. Suggested-by: David Hildenbrand Signed-off-by: Zi Yan Acked-by: Balbir Singh Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d98283164eda..19e69704fcff 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3731,6 +3731,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, if (folio_ref_freeze(folio, 1 + extra_pins)) { struct address_space *swap_cache =3D NULL; struct lruvec *lruvec; + int expected_refs; =20 if (folio_order(folio) > 1 && !list_empty(&folio->_deferred_list)) { @@ -3794,11 +3795,8 @@ static int __folio_split(struct folio *folio, unsign= ed int new_order, new_folio =3D next) { next =3D folio_next(new_folio); =20 - folio_ref_unfreeze( - new_folio, - 1 + ((mapping || swap_cache) ? - folio_nr_pages(new_folio) : - 0)); + expected_refs =3D folio_expected_ref_count(new_folio) + 1; + folio_ref_unfreeze(new_folio, expected_refs); =20 lru_add_split_folio(folio, new_folio, lruvec, list); =20 @@ -3828,8 +3826,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, * Otherwise, a parallel folio_try_get() can grab @folio * and its caller can see stale page cache entries. */ - folio_ref_unfreeze(folio, 1 + - ((mapping || swap_cache) ? folio_nr_pages(folio) : 0)); + expected_refs =3D folio_expected_ref_count(folio) + 1; + folio_ref_unfreeze(folio, expected_refs); =20 unlock_page_lruvec(lruvec); =20 --=20 2.47.2 From nobody Mon Oct 6 17:06:23 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2067.outbound.protection.outlook.com [40.107.236.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDB132E88A9 for ; Fri, 18 Jul 2025 18:38:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863887; cv=fail; b=Vy+0w/pPh+fgAhI9Q4ATR+sPi/7uGJIeRGOz+cciueAbhEm+khD2tsVAMZ1bULn/HhnE10PVJYR04bsjFeI49aUWh+rKyISlMJF+15NFLJTf378aAB97og/XtA084bA6at3Ch1ulVYkVVhKhSuz04+26XFUoybsOql0Q+/3RkPI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752863887; c=relaxed/simple; bh=pk/mnSJQEYJZHBsav+n/KOHbr1Oi0mE7l85CBouVU6k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=FLMbt9wYiBqnMTE7/G2rAJHUOoq/az+QGO62OQ6ckXDb/m1IXQvWS11n1kmNFJ4Tr7eCefPiZEFQb5KqhE7pLdLPYQYX2G98yOKwGKxSBTJYFzOcQaIlZBT4CBOD5JeKa/Bz8YhT8rHiMpU0WP8gBmYiYWzysFHUQAgax2enRwE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=oKLKsdzI; arc=fail smtp.client-ip=40.107.236.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="oKLKsdzI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rPlcgiFWgMpAPi8H1NuVhlwADtOr3r3EkztWbaIEzIoW6gsTxcajff7INYirsyzRm12F9R0+WDzRA9/m/H4qHk79oZs4PH9MDc22tMzPes1zwtMQYVv+Oz6N3uoSED1ZP7isSZYKrhRcF0tAPYCcMNO8t9pyToPZ3Q8WHhu1HuPNpzFuAbwHrlHakiFwJ67Uk6XFv5SzavGTtgXiEODSl9jAYUGQPFzMReoByh0yd3NJpmILn8Ya+H+FjHW83TvesRevREvcGca+0Wt0tQmXABkijzQTovC4RYqOjtIuFnNFBCkN2tOItregrOAA3pUib7YS/Oc7O/od4ZCSUxatuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zuNfEGnE/pkxmtI3d2K1g+qHeSUNpMH17TcrWnnOs1E=; b=mCzk42Ku8wHIpak5Yfm5PhXyq7DuG154myrbF3TwFyhACuiCbDBy6jtdNUQbXfoPG8rkHpYcn8fk2dBUZSImfymCIOIY0uPRr+6MDxj8yB11yrP7k7Pc855595CKI5GCafBYmaZLeXnCdiakAZ2mjw0Pjliw48kmGEXDFHCurUtkMhJe3OKuRIwoaKnRMojVdNllD0VyhoXpYBNhiuoT3OBexmTijxYAGOX6OaK4hYdMZJiUT4xMF7bVVYy0UJaHbkh/3EIJwJWh4uG98r1W5jkRU0uXTCgZ+wyjBdhYtLnrlWu6k9ajBsg2dSE/DiAdr7XA6H59AgAy361Q+7uaaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zuNfEGnE/pkxmtI3d2K1g+qHeSUNpMH17TcrWnnOs1E=; b=oKLKsdzIQ+GPeIWxcGuEBIR+Ny/YwbQ1/lKv+q2LyOjUvrWC2wyfzfEK7qpiv1BSxXSHwC29ys8B1Con7PdbX6JfcWqj6h5fDsdHsWrLvt2sD/awz5gAylfFxY0BE1xL6jniH8OzuhbCT6Ne7kf5uPS9HWXQB7IodssA6lmLbLoZ/C4SQVBpBjzKO3+3zPiAdoX+po1nUx3+EJrSy81Oj9VvOawrdjm2w4VZR43IfTkSZa4Yn/2M62brDxSj8JFy4vqK/vK4lEyBoFuTY8BjuH4f1zv/d8lyLWayhcyRLVwOAAgerU7ho4e/2uxKDgxEDGfjhu6+JPUMNbxnCClXrA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY5PR12MB6621.namprd12.prod.outlook.com (2603:10b6:930:43::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.39; Fri, 18 Jul 2025 18:38:03 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%6]) with mapi id 15.20.8922.037; Fri, 18 Jul 2025 18:38:03 +0000 From: Zi Yan To: David Hildenbrand , Lorenzo Stoakes , linux-mm@kvack.org Cc: Andrew Morton , Dan Carpenter , Antonio Quartulli , Hugh Dickins , Kirill Shutemov , Zi Yan , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Balbir Singh , Matthew Brost , linux-kernel@vger.kernel.org Subject: [PATCH v5 6/6] mm/huge_memory: refactor after-split (page) cache code. Date: Fri, 18 Jul 2025 14:37:20 -0400 Message-ID: <20250718183720.4054515-7-ziy@nvidia.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250718183720.4054515-1-ziy@nvidia.com> References: <20250718183720.4054515-1-ziy@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: BL0PR05CA0011.namprd05.prod.outlook.com (2603:10b6:208:91::21) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY5PR12MB6621:EE_ X-MS-Office365-Filtering-Correlation-Id: 6d1258dd-0141-47d9-03f6-08ddc62a3a66 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024|10070799003|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?zWwDQI65wRgemN9FKfnTaGWhAS+vuvtBFNmGOYgz2NjRZIj46mus48/wgEMv?= =?us-ascii?Q?JQVcCYLOA/zsBsNGIjs4SQaHX68dLylp2R3JqnItCC1lbYoufBOjetEUKTpS?= =?us-ascii?Q?1QnklZypDSKYHC07cGxKMcu8iNVhxJN1s3UDRcbYRA8rq9lqNRfinFcAT8DZ?= =?us-ascii?Q?6koccdzy8CjHhZr+DKQlIkgYYk6QR9l7pMKtX7nUo0zpUUCL84WJ0W2dHreu?= =?us-ascii?Q?m+vwnm80EpmUexGdNJxW/SVBgEp8mUzErJJLQOzDXi9HZmw9Zlqr9ebPzSlr?= =?us-ascii?Q?7l7qpLkU438zfXuhpgEuF9MusOYVLzXxVx893LpQS52gF57jAkdnCVLTxXLU?= =?us-ascii?Q?X9GXtPunvuQXSOEppzcLCVX2WQZkFoFMHP6h3a+VZH4cPE9UsKiLsQfjrwEq?= =?us-ascii?Q?oY+y6j9PxctvxtHWDMvyDYmk6eDO1RZ/3oG83j88gDfXFiJLwt29hrz10nKB?= =?us-ascii?Q?LqfZ3KRYVl0IqpP1fHvK4BuG9Zj1l0qYycyZTgVV72MHMq/HUuGiWASidcvE?= =?us-ascii?Q?SwQ6UfI2XfEoqA0Wy9XH3dq5FXNZvLeDNacWgU7/I2aQHSTbkDKkB9jOvF8E?= =?us-ascii?Q?y/JoeMC81lrcbiaIPu7/IZi9QkmVoRdwcXQrGTVyhnMay+YQVM0Kr1x/m0dl?= =?us-ascii?Q?vf8pPHKgVfnPtXcSeAXe0dnn7ockiYgTb0Ll5/U10Y/TS9cDuotNGbKPd3wy?= =?us-ascii?Q?DPNzPxOU6i6qiQdvKhjcRoZqUwk40ztFzrd1+p/zokB2jxzwvQCVlrqxPk/4?= =?us-ascii?Q?QBoJJ1hSVew/drQxx2fvQZV+ygbwAuVLxOoVZBFGeRz0OAyI3hP0u5nvSYW5?= =?us-ascii?Q?JgY693rnaLO1wihSHN4q3V7nNPUBUp8ZKM02Qc01OdCckf8yW7XzBk+FkPD5?= =?us-ascii?Q?6+v3jox2zbYuHs+T2yjSlm0RTYicoCihSdVkt8vllYdnJUfkY7N8iaXh0RxA?= =?us-ascii?Q?WvTXmD7umFO/xen/7PWek7DVkCbMRlVx/GMRg0EHnOYwKPR+wbeyFJiVgYVr?= =?us-ascii?Q?XZ1V4XG1CYJz8wTdBOWMl5cUIcNuWs9RKdzG/yd8ST4ybZq7cxwKuUhP9pNp?= =?us-ascii?Q?I44L5cyUWosJl6Z5mJX4xkgSEMxqA8L5eN9SI4SCzyoCboRtEdJkMq3ikGa6?= =?us-ascii?Q?pWkBLkmb4bUlTWqhl6xOs07M21XPZetOuITRvVGFN6Tjf60QSQ9gSbQKl1GZ?= =?us-ascii?Q?xnWqa4MBU5D2ag/UoHJ6dUCsw8Pfmw8wAdAQCM7maSH1vsH7DF0Z2BclMHWg?= =?us-ascii?Q?jAJHXSdC+4tcEDbiy6gc1Oiv2qylPhWc6tGtuDJIukrFu0Lp7QV4ftc3giGL?= =?us-ascii?Q?Lt68igYfqmpPCiz+wpqCYRaiyesg6iORV1LJzzzkBT59K6fIv13nCt1W51eD?= =?us-ascii?Q?5ukBOcCU7duZrCG4vqtZMowPoa2jfIgVXu/XOlh4n+6Y2kxoBIr4wC9xpfFH?= =?us-ascii?Q?Y6S0I8mpGUI=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024)(10070799003)(7053199007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?p69zV0cMd3WZ/ds8SEslS71q6NdZN7Cl9n/fo2tgsKlHRBdS8/wJk66b/F4+?= =?us-ascii?Q?w4fRO93wzgG/xG3jeT777qNZy1mbNwIH6mxVQfTTbBb+G/Kpng6883WRPGbW?= =?us-ascii?Q?N0Hoe6+upiVcEIRjGcBvvlfsaN71y9I4OmlERxHup769l8MbcRcmqLYYfNtb?= =?us-ascii?Q?hs85J4LBmog4GcorOxg92L2XxTnRtg8WnYnbcayiSHwZNs52wgYlYggw5AL0?= =?us-ascii?Q?dQUPxXeF3UkOW3KnAs6/xr8GJmziYHDcMs+ksGb9T/x43bzrlSbrPR1Z5Mc3?= =?us-ascii?Q?EbhIkLdliSpPaq96U0/CQfYTmyL5wlzvRmhOzxtHUnjSLRCFaaoQIZhq7TzA?= =?us-ascii?Q?Noshuz06NRpCt6udLppZwN11mnV8ddqcFWBxY2vDXmjuw9m2Wgh4qOgwS7ka?= =?us-ascii?Q?Is0X0AOtG7Gzi9jfu5rUPK4fxhBeZm9tFUcTCOe0s1nQvCKYAl6FMJUxfuhH?= =?us-ascii?Q?OUlazEytJVY+Mmi15zXkDg5EQPRoHpqqBr6141rtT/6EAOi/9Vnd7zUiKrZq?= =?us-ascii?Q?Nu6tHdBHav2ZurZzOnmhCq4AZSeJfkMTwZXIZVczaAFK8X7es7k7TvG7z/US?= =?us-ascii?Q?nTrzXPin7iZX2Y4PYfUeryequu2izyfZChUMSkZznFCeYk9LZ5JzkiqHmFx8?= =?us-ascii?Q?nfU0BeLoYEr2LJAaNx1rfyIcDcqhLeomWiHQJoNwVp9MBwK+F6kTHv82rdlx?= =?us-ascii?Q?wC8fD28zvDFirsO2pHaJdZcafy976WXDIE3yNvucVfyqevMJWtwcJ71FW8jU?= =?us-ascii?Q?CyObno6jkVhbF7oFe4u5O6x4L1DXfX+r6VRCBDVEthf2rIIi0micSEVlO0mu?= =?us-ascii?Q?s1lUxbKcFDTA0PW3vJc7GCi3xVOjaeA8ii9AGhk7YnnDIDaRB5rUKGCUW9oh?= =?us-ascii?Q?/cbCJTTGOOB2JBSWM+n3CqqReY4+JT/lnchxb9re4J3iI7xbRuaWfHUkA+tb?= =?us-ascii?Q?30ZZhBCHgHtTxVous8DkMkCIAi59TX/tt7Cy6vys6dmrRe0FinCwYWbdQfuX?= =?us-ascii?Q?F8ZPalmb7/lRdTUZVdqoRDzsgnJySbe5Is0Xj7slAiFg/0/QRb39bwMMW1XF?= =?us-ascii?Q?n9irnvSV9MtRFFfIqvymmwmD3ka+nciLAmNvYK0BFzg7sg0P2rB6NApD2F/a?= =?us-ascii?Q?16FVrkKOtx69Js9UHuljsy7gLFbGnKJXkFLmcdJxhzL9cwUW1RFU7iO+wg0K?= =?us-ascii?Q?hP4VBoPLjXlxSUGL1ugaMGHzKUXzsjX08rS3dB2/QTYhRS+5GsBVces7d9zY?= =?us-ascii?Q?rJtmvEeJdP5p0ENPqV4X9bw1IN8RkYhhupPalE2qXgHyVd8bU3DAa6dkf32e?= =?us-ascii?Q?u/ignRGQ37UWOn6sBI2BKWKbAIUNdv+IJWNZip1HHQmqZ2Ue4K1kP572avr6?= =?us-ascii?Q?VGjPrLQfrkkj4HsTx0b96oFv+EgeU/2ydOHH8jKANUsXXLsKHYUM2DIYoHxl?= =?us-ascii?Q?4jU7cQ8Jbb7zAut8SUPfp52h9nnUHwns1DFFn24YNC6KAoRBDswJJgVvKBid?= =?us-ascii?Q?6lJwoN6nJZPv4oSv5tdwFup9aJgfN1ZDi1316OkgXitaHqF7e9HTeIU/pY47?= =?us-ascii?Q?HGHCwozT/LCPYx8U5jPD5n/0XyMtqGOVwuEmYlCgVMwIXCu0v1KPHHwedyMW?= =?us-ascii?Q?0jQlvERtmXl1JCT5XIYW1L0zAjleefTVDHZLfuNd3gfJ?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6d1258dd-0141-47d9-03f6-08ddc62a3a66 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2025 18:38:02.9076 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qa0Mcipsl12RKr33yeete5RX9QFpLmAeLJdVBoukTNWxq374Rkgl/XLd0s4K/zCM X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6621 Content-Type: text/plain; charset="utf-8" Smatch/coverity checkers report NULL mapping referencing issues[1][2][3] every time the code is modified, because they do not understand that mapping cannot be NULL when a folio is in page cache in the code. Refactor the code to make it explicit. Remove "end =3D -1" for anonymous folios, since after code refactoring, end is no longer used by anonymous folio handling code. No functional change is intended. [1]https://lore.kernel.org/linux-mm/2afe3d59-aca5-40f7-82a3-a6d976fb0f4f@st= anley.mountain/ [2]https://lore.kernel.org/oe-kbuild/64b54034-f311-4e7d-b935-c16775dbb642@s= uswa.mountain/ [3]https://lore.kernel.org/linux-mm/20250716145804.4836-1-antonio@mandelbit= .com/ Suggested-by: David Hildenbrand Signed-off-by: Zi Yan Acked-by: David Hildenbrand Reviewed-by: Lorenzo Stoakes --- mm/huge_memory.c | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 19e69704fcff..9c38a95e9f09 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3640,7 +3640,6 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, ret =3D -EBUSY; goto out; } - end =3D -1; mapping =3D NULL; anon_vma_lock_write(anon_vma); } else { @@ -3793,6 +3792,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, */ for (new_folio =3D folio_next(folio); new_folio !=3D end_folio; new_folio =3D next) { + unsigned long nr_pages =3D folio_nr_pages(new_folio); + next =3D folio_next(new_folio); =20 expected_refs =3D folio_expected_ref_count(new_folio) + 1; @@ -3800,25 +3801,36 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, =20 lru_add_split_folio(folio, new_folio, lruvec, list); =20 - /* Some pages can be beyond EOF: drop them from cache */ - if (new_folio->index >=3D end) { - if (shmem_mapping(mapping)) - nr_shmem_dropped +=3D folio_nr_pages(new_folio); - else if (folio_test_clear_dirty(new_folio)) - folio_account_cleaned( - new_folio, - inode_to_wb(mapping->host)); - __filemap_remove_folio(new_folio, NULL); - folio_put_refs(new_folio, - folio_nr_pages(new_folio)); - } else if (mapping) { - __xa_store(&mapping->i_pages, new_folio->index, - new_folio, 0); - } else if (swap_cache) { + /* + * Anonymous folio with swap cache. + * NOTE: shmem in swap cache is not supported yet. + */ + if (swap_cache) { __xa_store(&swap_cache->i_pages, swap_cache_index(new_folio->swap), new_folio, 0); + continue; + } + + /* Anonymous folio without swap cache */ + if (!mapping) + continue; + + /* Add the new folio to the page cache. */ + if (new_folio->index < end) { + __xa_store(&mapping->i_pages, new_folio->index, + new_folio, 0); + continue; } + + /* Drop folio beyond EOF: ->index >=3D end */ + if (shmem_mapping(mapping)) + nr_shmem_dropped +=3D nr_pages; + else if (folio_test_clear_dirty(new_folio)) + folio_account_cleaned( + new_folio, inode_to_wb(mapping->host)); + __filemap_remove_folio(new_folio, NULL); + folio_put_refs(new_folio, nr_pages); } /* * Unfreeze @folio only after all page cache entries, which --=20 2.47.2