From nobody Sat Jun 20 20:53:30 2026 Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010063.outbound.protection.outlook.com [52.101.201.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10F0030BF5C for ; Fri, 10 Apr 2026 03:23:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.201.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775791425; cv=fail; b=PuRbx2NNavHLZVsf/Uek715o7n0sdJPUzPlBpDkVhuteOaekga9Y1PW3VII38ls9O9xTunD588VlzUtaVQstZcqyfytb4+eUHw++7Lzh9rJWeUnGZRKx2GCLvV8b4pefdINkJ95wHOAdNxJuJwRrSW86RetvG8nzWVwVGtHOKVc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775791425; c=relaxed/simple; bh=wRwqRkt/MiY4TWJGQpSrgIekhXtoNHuchtTzJdIT3Z8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=igEIzi8B4YvOwYaWYH/rVA1KOIbl2ScHxAVtkfdFV9qKjUfaRkNqoNcvNo3BMbbPwdwaa2R4qR2k60vd6YxfM4ZLUw71K4eUW0ldHoVmQWblqM4CtzPM4HQ9zhcU9DNgO5CWmdu8TO414P1wbqvWPB329Nsw0XczJ8Gmq0n10Lg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Q1QnBc6t; arc=fail smtp.client-ip=52.101.201.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Q1QnBc6t" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bTq2C/ZwiygfjPgdCVQPx/HKUyylAIUM8zeSGWvzl7obvo1ehDZ21C281wYHjH/7197GXizg5ZFcihNaVfHhk70QDyFHp+rpry7HI6d/T5dHRf5pA9hKAPNiruChVj4P2KFTAz6sfE2IBote03F5t4kr0s9fS+HGMfcguvRlDIP7v+n2M5+OuTljgw+PUEr0m7h49aVnvcFMlxxTqPrTxHOcYkSGbQ+HpMX0SodajN9p31kllrTC/TnSKWchHd+Iplt2oJ0lWSJj+WsEv4GXSAbH+GluuDpLXlNzcy8+JTNlPjJmPrVrHRLJew4qRp1SPbJkNrMCfjp6FU7gDhndOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=on70YThFz3lsPi13m35IjH9Ox2PXOcmEWPPzgJfDh0Y=; b=BjpR//Bi6vnGtB0hjy6B1ksTk67jENmX592VRM5YPl19JPnxtUSt80s6HqWsfqyj6TxHIfCH9sXJz98Pu4ZW3hKtuIAPpf1iiERjT7bPSfvDnMGhbmXgtvI2Rb5pfE02pf6hHjpNWmyzMv1zaMkpzJ3QJ+tG/8t6At79H2UBoIl5wAj+/EK0sw4eI+Jpwo8X0nIM/Fbu4/eZSxkXHjTmJm1bpYQsDNmnCPKsRPXklt+ZgEzSAkURbusbvAQ4DHd14dLBLUyN3SBye889Sf6myNq/Gt+iXBDqDtWMrgJDIPNiRCDM+XDp8NCaC8Q2AbdrzU5VGfQPa5ZqHtbpji6jXg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=on70YThFz3lsPi13m35IjH9Ox2PXOcmEWPPzgJfDh0Y=; b=Q1QnBc6tU6HgpMKBN2B4+SlcrFAO0sAmqP5sg2XprTu/caIuWGh/wU8pYObZS2JBnz2wOnB+EVQSMYqSjMn1jymYYRXjNnMYo2V3vN8FREq+TdMWxNdLTUAFaV6XGjAA2GVQbQF4Gak9/4Rk0UxTFKD03uEdtTWqpFfWM1gSfYfKzhyrwO4EdiWfUojB/dSNCuK9FBczHoz8ajKw29PFXMHyHyY4MlnhoJMmbqyHnUcPn9IH4IZEmjA7weSriwqFVS8D8baGTq9as/JXkV9ktZI8mR1XTmR8/qOpk4rdyiSiGjK+xf5yMLFvVG8w9o+dmdujlpo34W9PBZ8aHcUUhQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM3PR12MB9416.namprd12.prod.outlook.com (2603:10b6:0:4b::8) by MN6PR12MB8472.namprd12.prod.outlook.com (2603:10b6:208:46c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.21; Fri, 10 Apr 2026 03:23:36 +0000 Received: from DM3PR12MB9416.namprd12.prod.outlook.com ([fe80::8cdd:504c:7d2a:59c8]) by DM3PR12MB9416.namprd12.prod.outlook.com ([fe80::8cdd:504c:7d2a:59c8%5]) with mapi id 15.20.9769.020; Fri, 10 Apr 2026 03:23:36 +0000 From: John Hubbard To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Axel Rasmussen , Yuanchu Xie , Wei Xu , Chris Li , Kairui Song , Kemeng Shi , Nhat Pham , Baoquan He , Barry Song , LKML , linux-mm@kvack.org, John Hubbard Subject: [RFC PATCH 1/2] mm: wake up folio refcount waiters on folio_put() Date: Thu, 9 Apr 2026 20:23:32 -0700 Message-ID: <20260410032333.400406-2-jhubbard@nvidia.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260410032333.400406-1-jhubbard@nvidia.com> References: <20260410032333.400406-1-jhubbard@nvidia.com> X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ2PR07CA0003.namprd07.prod.outlook.com (2603:10b6:a03:505::26) To DM3PR12MB9416.namprd12.prod.outlook.com (2603:10b6:0:4b::8) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM3PR12MB9416:EE_|MN6PR12MB8472:EE_ X-MS-Office365-Filtering-Correlation-Id: 2ab9a553-c821-44c1-61ae-08de96b08d8b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|7416014|18002099003|22082099003|56012099003; X-Microsoft-Antispam-Message-Info: pTsN19CR2gMrv1FFOLr3VldKvCc+8tPLeZQPFsajqSZ01Y92BmvDu3IIpYzObUMfQA6s57NkywtjGVL6AVz8NAwkhB118QyrxM8YGMyJsSm08D4O/VYJ+eRPd+CpWgbov+yiJ0bGyuxqIy2CJpd1MrF7CvMgeLvYGa8jxGhSqEDiLacyrb9HNtVOwOywVPWOXmJVeeGEaoZ9i8h4m4wLUWHAm1uAr/rssJlYRETtAd+olz3/2RnkSxstje+F/rMUcMBkNDYPckardd6wrJwrdzwvZ3uXN3HJYqsa8biHemQPalOWkvNyRfpvsBEOCxswUCTS8H7z3cPyV2mZkKxUszKQwkCZamdTZRrvIx+cbvfLsRY9Dl7tEgeOx0wUi4Mr8UMNH/trXsQ+eLhL9Da8vv6yllj7PyCcLnMt3J8hMa1MpLO1Xtwct6ikVnQIgYKnPlqZPXaEM5lQwXMp5WYNh1ECJIAjrb7WdCYU68FtF2OCNe1u9fn/hM6ax6OGx6B9Qq+7KYtxpory6Fur8J5LN35myhmufgijq9hKZR/cs0FtFPwlyumMKLwduAkviYa2+Mc4rkWyRpNfEjtonqyDyV4w54/JFZsuSjl2eiuJ5QHAZuvKJvy6BkY/xPOEgPaXDggkGn6J6eUT+5QsKJETLM5GAcr96UZpVPxMuLpLFcNI5sGEUzo7NmGwqNO70tmzHMtWy+nPYrB4C7pMXv23pzBffA4Kk3piHgdeFhUl9HM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM3PR12MB9416.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(7416014)(18002099003)(22082099003)(56012099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?m7z8Y/F91uKVX9pJ5oaE9KYkTxaunpLSq3oBnAdsxxohzXwX8GA+cwEn7/fr?= =?us-ascii?Q?Zt1FhB9Uf5XjLej8wnUXE8g87sPVyAljatNIdVEaZgl9KQ04ETzNX7K6CUIf?= =?us-ascii?Q?iM3FME4HKCX7qJLkMggqjoRY59T2Ws8u+DE1ja0qysJh+nMmR1r9lQ+81Td3?= =?us-ascii?Q?jKX36uk98ml9kNUfPEibMm2C08OCGztpZIC87uSOm8gyT0yOLiId51gcHmoO?= =?us-ascii?Q?Nj+vT+cMnI8470Bo5yGMYjqqAkggt9tldpRdEG+jGu7ZYgM0rATch5/sROdM?= =?us-ascii?Q?VLwAVyKFWVIbwKXeK0Dv+hPOMGmPTBNtWE2wss7miNQMONxMOOYRSRlzPoHt?= =?us-ascii?Q?ucG17VsDsKDwgDwnDJ3rACpj56KGI1Qbxx8dWzyMkLTMIwASKDEd1KEovp4Q?= =?us-ascii?Q?awWHQXv0mhaVwbaFocyBe3nnB53uVWhGzFTjE8xp8ScVW6Xo4/OLSzaWI0i8?= =?us-ascii?Q?QuKt7OaOVE/MvH3qVOL/PqCxa0MDWrXWwKfQoLGzfwkkAcv11g8GpG9sicuV?= =?us-ascii?Q?VC3K2KI6xnM6qURrb8AAKfgcXXJQDvUYBy9WUlJN7FSmi+rqsRr9UfeK3LWh?= =?us-ascii?Q?uBWUQULPH9Rmaj9a1EeudPJ5MLBNw6htLLDL0O4XrAu2Zl/UWuO7NevJTh3y?= =?us-ascii?Q?gHi5gYA6w2bHi7Q2mTFknt+U/Zz4Xl33PBi3vxbaq6eCbsMGlnZuKxkfJIMl?= =?us-ascii?Q?Lvj2QhzRdNXleZAI99HUEAN9mnfnlUT3/GdQQMgmKWlNJN7eSSYuLg5nuQAc?= =?us-ascii?Q?aJiS3j9bC6j9o2G6mhveuKRPqy6g7g/a8TE2SIEalgOFwEDKEO0eOSkecKvK?= =?us-ascii?Q?eg1EddQKBK1Xho0twsqv2vmpd/36k5pEFTcDXELBxBsmkBhpHQLYnmuFpPIO?= =?us-ascii?Q?j0SaxXojNgNY8022lfsquzeTEhCdgKaPTt0U+NiTIF4QtscBuAsPljnCCy+3?= =?us-ascii?Q?KEdxDWblHMO5rz2qc+7GkZKSBaph2zRfZpnyp603Om4MxoKASmiA5RkiBjsX?= =?us-ascii?Q?bmAU2woDpP52L+jWo5nqtqgs094USnzt9hQjXYb8MuxbBflSgVfM6903p65b?= =?us-ascii?Q?e5udrO+VMabF5jEklFupOWJYmmQZSevx+usNP9lN7N+NbCC+BmFPkk+lRC4m?= =?us-ascii?Q?VPZ9Ad4S8+6+IZW+WAKnRf1CccLVC5/WSJIreIbDRTuKTHP5DWQG2lojYa4z?= =?us-ascii?Q?/bvb+gjynOTuuOaiRfUp8EVyeNTm6uYXR974WzJHbVDRkYP0MT9tLseWhHiW?= =?us-ascii?Q?YtzQtYVo0fzfHb0qhGolJP48vAzixM5qX+OVOnKOKP8unjitAsKROR+8CfLf?= =?us-ascii?Q?WVD94G3paTcLB+DMZgd3iykkcZQK6FKD53Nb9/vcPBqbpClW58/rmc/zL6JO?= =?us-ascii?Q?TUaAf4oDql2pUrPlS0EtP2G+vR1dPgr6DtLQJRKZv4Qvnk46YD6A7/qx3QF/?= =?us-ascii?Q?9flmaSGQ3GpY28jMIo12iL2BfcNbZ3tTs/uTmLguXbvkrK4/LvqCjpkVgTmG?= =?us-ascii?Q?LVsSF5GygkQ7IzJkXi9BXKadHR/TtWnMcaV6yC+JIvqlQm2iQ5MfvBgLmx54?= =?us-ascii?Q?c+S5X0fPJ4Y/pMR48mFa67I1ThABCIf1PIAValfElQqb5y8cM42cbvLSuEBL?= =?us-ascii?Q?x0v278kZX6hoCqA+htdrfGOWVeGu6ukqdFPhjdMldNWRm9yYXYlps3Cv0+Tu?= =?us-ascii?Q?//WvpAsdJIGGkzcD/iks1z/b/XaGus22JBIUGpfgI4xIyM0Y5hf0tnEOFoRN?= =?us-ascii?Q?oQhSdk2vdg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2ab9a553-c821-44c1-61ae-08de96b08d8b X-MS-Exchange-CrossTenant-AuthSource: DM3PR12MB9416.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2026 03:23:36.7446 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rKRzoThXGvj/AeUVLhiQ1iHW03uhY7fZB/EvzJizJhlHRBbAAw7xtbmkTqTjHD64C5dGGKtaOTiXayTx+RQe3Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR12MB8472 Content-Type: text/plain; charset="utf-8" When a folio's reference count is decremented but doesn't reach zero, wake up any waiters that might be waiting for the refcount to drop. This enables migration code to wait for transient references to be released instead of busy-retrying. The wake_up_var() calls are gated behind a static key that is disabled by default, so folio_put() compiles to a NOP on the wakeup path when no migration is waiting. The static key is enabled by the migration code in a subsequent commit. Signed-off-by: John Hubbard --- include/linux/mm.h | 8 ++++++++ mm/swap.c | 10 +++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index abb4963c1f06..ccb723412c07 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -38,6 +38,8 @@ #include #include #include +#include +#include =20 struct mempolicy; struct anon_vma; @@ -1798,6 +1800,8 @@ static inline __must_check bool try_get_page(struct p= age *page) return true; } =20 +DECLARE_STATIC_KEY_FALSE(folio_put_wakeup_key); + /** * folio_put - Decrement the reference count on a folio. * @folio: The folio. @@ -1815,6 +1819,8 @@ static inline void folio_put(struct folio *folio) { if (folio_put_testzero(folio)) __folio_put(folio); + else if (static_branch_unlikely(&folio_put_wakeup_key)) + wake_up_var(&folio->_refcount); } =20 /** @@ -1835,6 +1841,8 @@ static inline void folio_put_refs(struct folio *folio= , int refs) { if (folio_ref_sub_and_test(folio, refs)) __folio_put(folio); + else if (static_branch_unlikely(&folio_put_wakeup_key)) + wake_up_var(&folio->_refcount); } =20 void folios_put_refs(struct folio_batch *folios, unsigned int *refs); diff --git a/mm/swap.c b/mm/swap.c index bb19ccbece46..e57baa40129c 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -43,6 +43,9 @@ #define CREATE_TRACE_POINTS #include =20 +DEFINE_STATIC_KEY_FALSE(folio_put_wakeup_key); +EXPORT_SYMBOL(folio_put_wakeup_key); + /* How many pages do we try to swap or page in/out together? As a power of= 2 */ int page_cluster; static const int page_cluster_max =3D 31; @@ -968,11 +971,16 @@ void folios_put_refs(struct folio_batch *folios, unsi= gned int *refs) } if (folio_ref_sub_and_test(folio, nr_refs)) free_zone_device_folio(folio); + else if (static_branch_unlikely(&folio_put_wakeup_key)) + wake_up_var(&folio->_refcount); continue; } =20 - if (!folio_ref_sub_and_test(folio, nr_refs)) + if (!folio_ref_sub_and_test(folio, nr_refs)) { + if (static_branch_unlikely(&folio_put_wakeup_key)) + wake_up_var(&folio->_refcount); continue; + } =20 /* hugetlb has its own memcg */ if (folio_test_hugetlb(folio)) { --=20 2.53.0 From nobody Sat Jun 20 20:53:30 2026 Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010063.outbound.protection.outlook.com [52.101.201.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 876A030DD2F for ; Fri, 10 Apr 2026 03:23:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.201.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775791426; cv=fail; b=iNWccdub2L59DuzP8AONVNXu0BI0PsEzU135Tq/jqipyLhkUMmtBTSbAlXxs8XqO5ZefdvVH1ZRFFyPrYjNUKpVsoR0aAAT9zFodl/3yZetb2vc5QgwWGYMVYO8FWq1kQoiGT+iIjRZPaAnkJq2r5PJl5Tr/sWl5jnsY2z/U3hE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775791426; c=relaxed/simple; bh=2FoXkhPiPVSBi++MinXCWwEW+ANiRymryCOh8jQhYfI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=fL+qnvxkqHQiGEoNOVCMSywTUNSkAg+dwHVPTQw30NQkTOHjG/eAPSj5FJMJxtbsKfUJH4pYMuCgVmrwz0kJw+UhW1G9mimZic7QKx4OEkkwzVjK4SynrG684lSGw6x3BB2wtYZSK4jrbbn4fUF873j1Qe+b+Xv5rD98HNK24QQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=N2eBtbQc; arc=fail smtp.client-ip=52.101.201.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="N2eBtbQc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vLLZGrkqpboDqWigmIBtQaDeg6AhiM+dV0URSpGZsUpMKVlfY04vOppnz5B7MR3+wjHC3D8GnYRWHOVBTb4txSKLD1sswiHIwujngDCLuFmTj2Qcz+J0QxdcDWlQF/SpbLR1EHosPuNmqdYB4nCoh0D0Y9koYmM8X7UbEzt1rcBwQw1qZaQ0oqVYceouvqGYyMs799frMJjBKdsbAluc/t7xFEjWEoLm3xx/QAKwI7YREgR29kYcmItEPcnhNMT/TybBj9xshATykZoSfVEJzK50Vdp23hQoFXIRqRO+hnFeoCjvZp67lCUgmrp94sbeAQUnisNqNN5ltrt9Ns29PA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eG4BPIT3Iq3yNkbWj35bBFQlENiBQkEWmiN3/iiLSdA=; b=tPTYGwBB+tWj5Eo1xmAgGRx2Wu9BOl2QR7KPwK9lxZ9A11Thuvl0gX1VabxdETHDdybxzxeX+mJ7vhFuCAvg8mecIxYnmXm7je6PqlLkRqLr5kJSecYWcHOf+MVZ1jnVdW4WUgP1R/6bjRuOBVH+kCTQIj/JVKbljIEQo2mkfYSoF3i/cgdVIOdlEbfWpQB/mg3mRdrjTMOe32kZn7kSgIuvnnA1PCEhnFB+D6cYSFAxAIwG34g0WpQTtuxEy0ZHWazgPJ2K+1DLPcfR++fKZQaV2rTKJ+jCdQRZXNhugHsI/IEzqSxrErPo+y/GYO5wvTsohVCbj0Z8OtS7nxz1PA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eG4BPIT3Iq3yNkbWj35bBFQlENiBQkEWmiN3/iiLSdA=; b=N2eBtbQcqFDjLaWumAlryAXNZ2ZPMhGVwmv37PQNRH9U+KuiGUUx7EuQD52/irBrAzvTQ9bE+4zXcKt3HKkR/TRelt1wkW6I9MULFLLfLT4Wd0857lFGYDYtpLVaI+/dPDX0JFpdINnfF96pA2XyU1EA6RvudIaZasIagtTE8doULrfdVDdttR/PhwI5vHcXyP0yLuDpn2ev4cYr4A9sclAYKXTig5qOOMgldPJlsw465C5biBfMLwWwRJRbRPNUeqtAuYLVfcfTaRNcDYeEHQlwR/7ISB/Po8noPlUB0JRN7aUsePrspAkVvpQegHtgB57L6LKvUScGAS7YisWfIw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM3PR12MB9416.namprd12.prod.outlook.com (2603:10b6:0:4b::8) by MN6PR12MB8472.namprd12.prod.outlook.com (2603:10b6:208:46c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.21; Fri, 10 Apr 2026 03:23:38 +0000 Received: from DM3PR12MB9416.namprd12.prod.outlook.com ([fe80::8cdd:504c:7d2a:59c8]) by DM3PR12MB9416.namprd12.prod.outlook.com ([fe80::8cdd:504c:7d2a:59c8%5]) with mapi id 15.20.9769.020; Fri, 10 Apr 2026 03:23:38 +0000 From: John Hubbard To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Axel Rasmussen , Yuanchu Xie , Wei Xu , Chris Li , Kairui Song , Kemeng Shi , Nhat Pham , Baoquan He , Barry Song , LKML , linux-mm@kvack.org, John Hubbard Subject: [RFC PATCH 2/2] mm/migrate: wait for folio refcount during longterm pin migration Date: Thu, 9 Apr 2026 20:23:33 -0700 Message-ID: <20260410032333.400406-3-jhubbard@nvidia.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260410032333.400406-1-jhubbard@nvidia.com> References: <20260410032333.400406-1-jhubbard@nvidia.com> X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SJ0PR13CA0090.namprd13.prod.outlook.com (2603:10b6:a03:2c4::35) To DM3PR12MB9416.namprd12.prod.outlook.com (2603:10b6:0:4b::8) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM3PR12MB9416:EE_|MN6PR12MB8472:EE_ X-MS-Office365-Filtering-Correlation-Id: 50a36020-959c-4860-5d2c-08de96b08e49 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|7416014|18002099003|22082099003|56012099003; X-Microsoft-Antispam-Message-Info: hVJxPm3NIRELMuD5P9gwBAGUBt6fUUtfXJODxv8lqUmHRjoy4HUt9MDR3ksXtLyYlCAfBdQMQXtm6ApRMIsF55RiubCKZlnc6s/tCwmuL3IYNYWb9LzQrYMA4oaacwOyPGVbKmYW1g/LBg4UcVrTmRDwhB2RlORLga6Iajiy91wgmguQ5mXke3+FBS2vNoMCz7Ep7wYQSu27Cscg7fLxydgueJRTc96Bpsp2S1BPT3XgZIo2DragI7PtIzzja1XK/el0grkwpHXi1fNoHQV6m3wKqICHgUcebc6QETAF39T5jf2j0ebQO9pKOPRTvHsG0tXjhtDiz7OZyYu772GWLBwuK4+0MTEjDaRuluVweJy2OG0hSXJA/qHOLoQPwN5NvqYGLlo66Ev6WPZ2/uC6WBUWXwPDlOVkvLi543uMmvbu/b1SLAUkFAnZhU9F8YMhSTyhiigOcH65QpRGHs/1kxTXJBD+951yCqMcw/odR4wqDIk/R1PifIjj/EQo47VJIjiHzTbdHINDkUq3+Pp0UNKUnC3qqCLscCNlIXvOywvfvlAj3tWUwkfnyLJAlcYt8JT/eG7oimYfd1x87zOWyh3WCjRwuL8FDh4rt7w0N8+FB4L6lJOa5+FU5LkjxTzOFsd5z1I6leu56XfMwoIL2esM2fNhzIYVAW032X3MV70QUhXtJkWvoCrAizJL2IvXF9HKBN8H+PngFuyAYjEkOUUiQNe91LJ3T1q0pxpsp5M= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM3PR12MB9416.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(7416014)(18002099003)(22082099003)(56012099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?sYOUECQvK3Cfz+3cdSnPUVdhyEg/R5cQ589SnbRTKOqUX5dy1DQHbnANeBH2?= =?us-ascii?Q?48DvsNV2ooZoY7nQjbmeojh0NZMSoQzyUUJnmFtWKgQNLrr/Ld5XK85krtVp?= =?us-ascii?Q?SCMM82mwrAe3xs5QZq1giumB9VqiastarKadIL7FM2frT9V3IjTzS7smxBDY?= =?us-ascii?Q?LwMsrBGY0o3q7TDYXoY/XBZ9khHS+smR73YBq8FNWWuES/VcmBrUa+KblIZs?= =?us-ascii?Q?WYcnblhS+DDrLp1pDKt2XeJ3eDvPikDcBulIxx6fe6sj2TNTVLh3v0kxjFxk?= =?us-ascii?Q?0NS73wUbcs7ODBBsKhwtTuMm5CA0DRaNdAUQnKy0fTJ1w8JGt5hns8YlgSSc?= =?us-ascii?Q?eDb8lqC4QgXeas1TSNllVLlSGQ7Qx84P3vKhymlpYykGWbg2+I5xC9P7ucnB?= =?us-ascii?Q?D7r8l9mHQG+LDcQ7WF4da/lt5agMTp+rtSKh/kTMM8uUURP6pp9+qipI0j11?= =?us-ascii?Q?PcHoFrBgV9sNCAXg6U1TEE627b75Mn2mnJDwQ5yCs9yfIr3ZeYbpj14d8cIG?= =?us-ascii?Q?xOS9KHFAhMgUAnEV2ZzdqJB/gc5a5PQzWS7Ig89sOx5hQvKSDruJpp6VJh82?= =?us-ascii?Q?OLgFxQ5u6mQq1auspkLWZJe5R75fM5OKXUfv1vDa2BUgNL2L/hdORw6T8JQg?= =?us-ascii?Q?0PacoXTs347cJGmgr6CUx0l1FzzdsHYvwpeYgWDzUqeMt1MqiyL4ypidzruC?= =?us-ascii?Q?KL2mH5f0Sow2pq1hH3WVMuQTJve5WruFL3cgf8zEQvSUvgWSbagnOY2drV1s?= =?us-ascii?Q?kd432FK6tfCqURXpOoITIqswGi6hG54xf96xOj+eHfcxSj6Lhf+YZXJ9qChW?= =?us-ascii?Q?ZLkDDem4QeznBSDQI2o/7qq1ijmQahNK40Mhds/nKffv0G34WJ+PN5zfqCwX?= =?us-ascii?Q?s9UeQIrfdMgxMWvXr+oFB0quuYOs3O30KgjIWZZjBAe15BDU1FekI4woujyj?= =?us-ascii?Q?ddSJ+SFHrNK1PHzHIuhUDjfcyjFAVrq5IoDeurKKXOGvQFQXB/zHdY/hzsEr?= =?us-ascii?Q?CRReFQCL8cdITZUjUSds/fxVTOgtlb/jRtSdyf3P+n48jwTw3cQrvt1vzb7g?= =?us-ascii?Q?769Ir7XTPLrraKjn41ri0iSj3IfepkR1A4l34kUVPVeQgSAi3PrIq7FX0hOT?= =?us-ascii?Q?9VmFlDOZjkdajAvJ0q3KXbJUeWI0NWHPqZeLT9C0eREpUvj4pEEh3TOXk89P?= =?us-ascii?Q?/uOY9TvpJQ5+xuBd1k48PCUuSoqI+wJlngzuczMP2Ied8dON60N+qOgI9GC5?= =?us-ascii?Q?wEkTzmjp4Og48GvikpDvz2llWSzJI70XLt+OJw+s+pNo6FeNzYdwxoz5Y1zd?= =?us-ascii?Q?U7fCL2pUPK2VWkiuzOWUtKLtuikWHhElkGTForwJoLdc3+VZ1QZ1HgXcSsMZ?= =?us-ascii?Q?jCPSktVM8r9AuPLT8yEiapoJIvIBsmTgkta0iLofUcCUOBNCV6nKWtdlbC3Y?= =?us-ascii?Q?PQZWOec3h9YzKyBrKedtjQIq4qAu+TYogUpLFfhVD6Mt4cNccmaco4BclL4M?= =?us-ascii?Q?mH5/Z/IS/lYOJLuGh+ttZM5p7SfSHCGZd4O6O3PLutAlok3ZgUm+kinjQnsg?= =?us-ascii?Q?tix0cwVt5kHGuvoPBgcm+qsZw+B0pwcdGsHYXn+q1wrnmPm7/qVIt39LWS4B?= =?us-ascii?Q?+JEaK0hG0UWS7J8AR4bEsYnKgr6pQ+qNPMqcbVN1zEeFKKTC9n5x70xKfRBb?= =?us-ascii?Q?k86KW/PNutqphyDX8weHaSFezqBlzTnmuXFkqgoSvUpk/L8mwZlg5u7gjnCb?= =?us-ascii?Q?c5yQhikvRg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 50a36020-959c-4860-5d2c-08de96b08e49 X-MS-Exchange-CrossTenant-AuthSource: DM3PR12MB9416.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2026 03:23:38.0382 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IK9dNgRD4Ueti3sZVwdxyJOgQvEFBZqlnVfOZ5z7X6XXsmWT+/bDkqUQh48Oop602ha2KO22O7mZ1FNpuWjgYw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR12MB8472 Content-Type: text/plain; charset="utf-8" When migrating pages for FOLL_LONGTERM pinning (MR_LONGTERM_PIN), the migration can fail with -EAGAIN if the folio has unexpected references. These references are often transient (e.g., from GPU operations like cuMemset that will complete shortly). Previously, the migration code would retry up to 10 times (NR_MAX_MIGRATE_PAGES_RETRY), but this busy-retry approach failed when the transient reference holder needed more time than the retry loop provides. Fix this by waiting up to one second for the folio's refcount to drop to the expected value before retrying migration. The wait uses wait_var_event_timeout() paired with the wake_up_var() calls added to folio_put() in the previous commit. If the timeout expires, the existing retry loop continues as before. The folio_put_wakeup_key static key is enabled for the duration of migrate_pages() so that folio_put() only wakes waiters when migration is active. Signed-off-by: John Hubbard --- mm/migrate.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/mm/migrate.c b/mm/migrate.c index 2c3d489ecf51..a5d9f85aa376 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -47,6 +47,8 @@ #include =20 #include +#include +#include =20 #include "internal.h" #include "swap.h" @@ -1732,6 +1734,17 @@ static void migrate_folios_move(struct list_head *sr= c_folios, *retry +=3D 1; *thp_retry +=3D is_thp; *nr_retry_pages +=3D nr_pages; + /* + * For longterm pinning, wait for references + * to be released before retrying. + */ + if (reason =3D=3D MR_LONGTERM_PIN) { + int expected =3D folio_expected_ref_count(folio) + 1; + + wait_var_event_timeout(&folio->_refcount, + folio_ref_count(folio) <=3D expected, + HZ); + } break; case 0: stats->nr_succeeded +=3D nr_pages; @@ -1941,6 +1954,17 @@ static int migrate_pages_batch(struct list_head *fro= m, retry++; thp_retry +=3D is_thp; nr_retry_pages +=3D nr_pages; + /* + * For longterm pinning, wait for references + * to be released. + */ + if (reason =3D=3D MR_LONGTERM_PIN) { + int expected =3D folio_expected_ref_count(folio) + 1; + + wait_var_event_timeout(&folio->_refcount, + folio_ref_count(folio) <=3D expected, + HZ); + } break; case 0: list_move_tail(&folio->lru, &unmap_folios); @@ -2085,6 +2109,9 @@ int migrate_pages(struct list_head *from, new_folio_t= get_new_folio, =20 memset(&stats, 0, sizeof(stats)); =20 + if (reason =3D=3D MR_LONGTERM_PIN) + static_branch_inc(&folio_put_wakeup_key); + rc_gather =3D migrate_hugetlbs(from, get_new_folio, put_new_folio, privat= e, mode, reason, &stats, &ret_folios); if (rc_gather < 0) @@ -2137,6 +2164,9 @@ int migrate_pages(struct list_head *from, new_folio_t= get_new_folio, if (!list_empty(from)) goto again; out: + if (reason =3D=3D MR_LONGTERM_PIN) + static_branch_dec(&folio_put_wakeup_key); + /* * Put the permanent failure folio back to migration list, they * will be put back to the right list by the caller. --=20 2.53.0