From nobody Sat Jul 25 23:03:58 2026 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010070.outbound.protection.outlook.com [52.101.56.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04F242561A7 for ; Sun, 12 Jul 2026 07:41:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.70 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842071; cv=fail; b=Ddo4Qrf0Bomgc4gps/Hx35Uu+IwYO/OcQhnMk6nBtsVJCHrhv05xOWYz+efLonavN914K0dryaaKWjlvKzAkUFj+g9vrdcOhTv1qU7Sch6Nn6UhrYopJSr6rvYGGCGrhTmwJ/ptl7gcVXJO85xw0P+PywFZYm/OtJbw0mKgynMk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842071; c=relaxed/simple; bh=srTISDEaD/Ktq0rhvvy6JqlSSASppyptfsmb53TFtgU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=PWfaIbKeeRsGB9Y0M4/+OAYujWesaMjs5P5cCKnUjT/Z6dsDVYtTSN1BPtKdst9dC7tgH8O9xxNvYkzMkrKdylyMo15ROb/mW91qcH6NUYMqvzZzRWQgP4LMRAskDyaBaiJddT92IwjHi5RsHkoZeAqWUVeK5AgCzWPHlgcMwf4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=yXilX6+s; arc=fail smtp.client-ip=52.101.56.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="yXilX6+s" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RKhNOd20jRrZjwqUQcabPs4Jp0c4YV7QJkesXM7tf6FKEo4GTutdrQMh21D5vPYYGjDo6YzitTcmx2azB3tQJt8E7j4WoYi7tEHiy7pdZ0sMCq/KjUuuwufnHvO1zAZzkfDpxpQ3RGwb+ZHttyWvJy+ntMUkE3SIY337yHKWghQ904FP2hIMj0Z3BTnLbkyGDI/WXcAqYVU1ZDhnpBQzMNca2fvMhrcOOVzt1jn9OvlREiIlapcpBwgr/U+K6xP/ux72sJJcIjaJpydsXKkL4dd6ZeumarsTxaRONTwig0CvcPjBlJMBnGrr63Jr+rpzBNSAGwZfyZWxcxhj/9oFTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+pnx6PsVHcrvSxWAaNCSG/M3DsulMI24w6J6EwBrV9o=; b=MgeFI/DR8l0v245pUQFiHDxf2SHnWDwblR9om8qQGi7U5IBrUAXvmlkYMZk3kEnHC6UopB3lG7OPrV2/QjiVVXitPQVmFkutF1IQUuhRISDH2VZ8xMjdPR489oH6hOnt38DBgcnrcVKugYKH1xNFUXMbheck9+UnTyeJwDNeHISONQm00REpC17XGzThhRPF/CNkpTT19eoo/cX5ak5oQwySiCZFJxYFVI3AZS/m4IfH4+olti8oXrONm+WdiK5lZPWhz/c81c9s49Uv32NfY97OR3nmVZ9wTh2241niQFkaCkaW2P9b90z+YJiWsVAj2/zmt/5FG2iwiIqzN5pgsw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+pnx6PsVHcrvSxWAaNCSG/M3DsulMI24w6J6EwBrV9o=; b=yXilX6+sCKtSqFgF5iwqJIiozRhLtV2nLwFHNolNY3L7IPia5Ua6Unng90Og40Bihwp1f4rFLfbfvGIsXSO05uRV3SGA4OXRsjP0mvrfYyS4P4yjbkg+wUDxlPBaNVBsmL9ep7VUKrBoFGiPKWE2AKsibNoJ45kQIBHT+Yvt96g= Received: from MN0PR03CA0015.namprd03.prod.outlook.com (2603:10b6:208:52f::17) by SJ2PR12MB8717.namprd12.prod.outlook.com (2603:10b6:a03:53d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.19; Sun, 12 Jul 2026 07:41:00 +0000 Received: from BL02EPF0001A102.namprd05.prod.outlook.com (2603:10b6:208:52f:cafe::65) by MN0PR03CA0015.outlook.office365.com (2603:10b6:208:52f::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.202.19 via Frontend Transport; Sun, 12 Jul 2026 07:40:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A102.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:40:59 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sun, 12 Jul 2026 02:40:51 -0500 From: Shivank Garg Date: Sun, 12 Jul 2026 07:38:47 +0000 Subject: [PATCH 1/6] mm/rmap: Add batched version of folio_try_share_anon_rmap_pte Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260712-migrate-rmap-batch-v1-1-872a734431d1@amd.com> References: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> In-Reply-To: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783842044; l=4856; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=g1KssW8+1v2HJaJEhqcOFZIerfOk9z+DUKil2dMLEUk=; b=PdS3GgyyjGtODvWWy5S3bZHiQpJzmSt5Z3pvDHLGpkbfJO6fF0ARPuaiguu4Zb7pKB5EqhIwt f3UF5gyB2T8CUazjdOfaVglvSEKjSmLV2XpYPA7F05LpojSFUyR56V0 X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A102:EE_|SJ2PR12MB8717:EE_ X-MS-Office365-Filtering-Correlation-Id: 4cb754a3-09c2-4d2c-914c-08dedfe8eabf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|23010399003|376014|36860700016|7416014|22082099003|18002099003|11063799006|13003099007|56012099006|921020; X-Microsoft-Antispam-Message-Info: MYBliWtnoQEevF9RUtEK+QXzWrlA5bUhC6zjO3xbX0GFR+qqhY9BMs6AglSy/0F0BvpGSAW8GoEGrTwFT+zskJvrlp2iL/MQwVoL6fj8A5d4MG4GIGNTuIeH6yhjCwTxKwwDlGfHwxIOlz+lh2FkoYOSHvHcSxG9mwJsDPYvw6IMieDzIWz1wkbTeosh7C9wGJYsrINtpoek5JAyLMS1Hny5VMiW4XMWsrNtsotGQTWfQQBW9FsaY/N/4iIpL4xXzRfla7WttmJhHod9KcrStypjxdnsRbWs80NSCsj4VZWfIwrHMFXE4KYFE6RobwceAYvZ4N2q3UnCx2zZr0M3vhlV6VlMPDrDrwqOPxDbaJLGnGXl4KB8jFWHOmsSJeJxRljybzO+zU01yGQtTmDapybe91pjLJEP1KgwCdJJ9SDH4px9pmxjfOJX00+Kn1GXmkKmqHvllDW17a7TwxaBWPcTq1P/D2pBiq+AAg5+I3jwEszlgoI252QAFW1kzzyOjYHdNPHhB4KYUZA/1jy6wzIOjTn4sXIP6punfVbXwH7f/LDtfIwwCrbT/VFxRVoyUG07zk9rhVbMhXFInYTb3/sqA+A6aQjOwxo3qoFi+horAFNM6/ELkIB64ckyNN1gOC5W141cSEGyTekuzVXHeNeRScRFz1IX3C2zXnXm32oeZIdyAr5pPXH7UpfRoU5CRrikYtej/8yKIqErIVti49fw41oty1YPkYLSl6v2irflNoyWoSyR3xSOGGfO0YDe X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(23010399003)(376014)(36860700016)(7416014)(22082099003)(18002099003)(11063799006)(13003099007)(56012099006)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: hjVD6tketPAgS+PzPhktg9HulDDypzqqvuZr1cPIhgYBiM9lDDf1C9n906WiDmI3JJatgtVJ+/OchlpdbXmFtscKTTrR3h4A4t4yM/mQ/zOCa1/X1GZ2+M4ZbJc+9+H4wuuSuqCouZPaDEg2T+FpZcDsfRGAmSPtNmyb59txnmfQFbDwmoYbgP8G9oZlfpA1JnxxBhgKvJq1DrdnnNbwpo6R8lGypdqEpX/yj/n6rIPm0kIoUsky6P+E5Mba9XdZf+TnyX9WoBkb/c4ND+BoBGOOXOX7tG8se449xZT9bRz56H2r8iHL+SOQqUzWvI8l6gygJA6TDFqo9eCyXhACx+UqJR8p1FZfmf1UlAQukZPnuQYdtIcnXD3jHlLBy4DnqW0e7FBVWW4gVSX4PCiaih2tR+9ha7fBl/4UGCi6Q7QLFmQ5levJKKggfXJsGfsY X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2026 07:40:59.6571 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4cb754a3-09c2-4d2c-914c-08dedfe8eabf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A102.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8717 From: Dev Jain To enable batched unmapping of anonymous folios, we need to handle the sharing of exclusive pages. Hence, a batched version of folio_try_share_anon_rmap_pte is required. Currently, the sole purpose of nr_pages in __folio_try_share_anon_rmap is to do some rmap sanity checks. Now, clear the PageAnonExclusive bit on a batch of nr_pages. Refactor the function such that the clearing of the bit can be done at one place without duplication. Note that __folio_try_share_anon_rmap can receive nr_pages =3D=3D HPAGE_PMD= _NR from the PMD path, but currently we only clear the bit on the head page. Retain this behaviour by setting nr_pages =3D 1 in case the caller is folio_try_share_anon_rmap_pmd. While at it, convert nr_pages to unsigned long to future-proof from overflow in case P4D-huge mappings etc get supported down the road. I haven't made such a change in each function receiving nr_pages in try_to_unmap_one - perhaps this can be done incrementally. Borrrowed from [PATCH v4 00/12] Optimize anonymous large folio unmapping [1= ], will drop and rebase once [1] is merged. [1] https://lore.kernel.org/linux-mm/20260526063635.61721-10-dev.jain@arm.c= om/ Signed-off-by: Dev Jain Signed-off-by: Shivank Garg --- include/linux/rmap.h | 52 +++++++++++++++++++++++++++++++++++-------------= ---- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 8dc0871e5f00..64929490a7cf 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -706,17 +706,18 @@ static inline int folio_try_dup_anon_rmap_pmd(struct = folio *folio, } =20 static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, - struct page *page, int nr_pages, enum pgtable_level level) + struct page *page, unsigned long nr_pages, enum pgtable_level level) { + /* device private folios cannot get pinned via GUP. */ + const bool pinnable =3D likely(!folio_is_device_private(folio)); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); __folio_rmap_sanity_checks(folio, page, nr_pages, level); =20 - /* device private folios cannot get pinned via GUP. */ - if (unlikely(folio_is_device_private(folio))) { - ClearPageAnonExclusive(page); - return 0; - } + /* We only clear anon-exclusive from head page of PMD folio */ + if (level =3D=3D PGTABLE_LEVEL_PMD) + nr_pages =3D 1; =20 /* * We have to make sure that when we clear PageAnonExclusive, that @@ -760,29 +761,38 @@ static __always_inline int __folio_try_share_anon_rma= p(struct folio *folio, * so we use explicit ones here. */ =20 - /* Paired with the memory barrier in try_grab_folio(). */ - if (IS_ENABLED(CONFIG_HAVE_GUP_FAST)) - smp_mb(); + if (pinnable) { + /* Paired with the memory barrier in try_grab_folio(). */ + if (IS_ENABLED(CONFIG_HAVE_GUP_FAST)) + smp_mb(); =20 - if (unlikely(folio_maybe_dma_pinned(folio))) - return -EBUSY; - ClearPageAnonExclusive(page); + if (unlikely(folio_maybe_dma_pinned(folio))) + return -EBUSY; + } + + for (;;) { + ClearPageAnonExclusive(page); + if (--nr_pages =3D=3D 0) + break; + page++; + } =20 /* * This is conceptually a smp_wmb() paired with the smp_rmb() in * gup_must_unshare(). */ - if (IS_ENABLED(CONFIG_HAVE_GUP_FAST)) + if (pinnable && IS_ENABLED(CONFIG_HAVE_GUP_FAST)) smp_mb__after_atomic(); return 0; } =20 /** - * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page - * mapped by a PTE possibly shared to prepare + * folio_try_share_anon_rmap_ptes - try marking exclusive anonymous pages + * mapped by PTEs possibly shared to prepare * for KSM or temporary unmapping * @folio: The folio to share a mapping of - * @page: The mapped exclusive page + * @page: The first mapped exclusive page of the batch in the folio + * @nr_pages: The number of pages to share in the folio (batch size) * * The caller needs to hold the page table lock and has to have the page t= able * entries cleared/invalidated. @@ -797,11 +807,19 @@ static __always_inline int __folio_try_share_anon_rma= p(struct folio *folio, * * Returns 0 if marking the mapped page possibly shared succeeded. Returns * -EBUSY otherwise. + * + * The caller needs to hold the page table lock. */ +static inline int folio_try_share_anon_rmap_ptes(struct folio *folio, + struct page *page, unsigned long nr_pages) +{ + return __folio_try_share_anon_rmap(folio, page, nr_pages, PGTABLE_LEVEL_P= TE); +} + static inline int folio_try_share_anon_rmap_pte(struct folio *folio, struct page *page) { - return __folio_try_share_anon_rmap(folio, page, 1, PGTABLE_LEVEL_PTE); + return folio_try_share_anon_rmap_ptes(folio, page, 1); } =20 /** --=20 2.43.0 From nobody Sat Jul 25 23:03:58 2026 Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azon11010009.outbound.protection.outlook.com [52.101.46.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7874371D10 for ; Sun, 12 Jul 2026 07:41:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.46.9 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842076; cv=fail; b=V/cEYzrLbuJmwILIkqSittr/zkgLrT6ijFAnNixycbKQd9fQhy1QCRHxbCv5htJrFomOqEwcB2IQ+YPFvoxOExc0rR5tJ/21LPnNJTuq7pdYYdEywhLztSi4c/3Ps9OZunICwsWfDiILGVdnCqytFtXHOm9wzOvJPUAev1z7Qro= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842076; c=relaxed/simple; bh=IqFSvR/5vhwyfDVx/GvRf77BmahPitfvnb9fPwcJzUc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=qPRQf+JLlXG0iNYSOFgQTB4WlK9s/Mly98vgk15P11QwVJviBw9OavwsHjL1LWdgLjudTUKX9Q/SyV0Qjgz693KjCYhGedmNk9/6IFiUFvFBJkCC+1kKVx1vjZwUTM38sLivmRXZpDzTIuk6fLabnAlbYTaJD7KHxst12kt43eo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=F/Y4FxX9; arc=fail smtp.client-ip=52.101.46.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="F/Y4FxX9" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gGrOK4wiIrTn7UmpNWKuEPWWYf77Sog4elqeQRSlW3A6V8fA+7EaCKS1mUvVAgtkWQ0wl9Zcw5e+JE11iu8ZgejQQMxgfGSSO8jnhPz0ymjOloOJrhSiBtIPz828IKrPCAIQEsDRIesScd2q0vZuKcLw58W3vKLvDH8ksgnPERtAnGMeUke1HckKeRhkfj+aLtUwSIIG1TjKWOuvbgaWc4bC9DWa0byNw8IveJzDAcvaq4hZon0SFNpH/JzDljULpb8KowHm1cU8f1/OfpwUivLhTHnNdnGVcr08pDotMkWprOs6FiAOS5zSbrv2DrH0sDgebobLLDVMnx4U9+BrjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E/1/PC2ZMw0rvgpLweirOTI/UNiY02ZFPp/DXWvgP/E=; b=HUVcrAN/1YfsTnrzVx/0l8rEeovwE5GACj1NAxhiQwTaTk1WAXBPRsrrs5JNjzptI4anATivWcoMdZ8WKG0PtQWqicAaXGR9fDxAVrRdxbk4IKrL6HLXf3v/C8IhSVYeVHptbDr/WOTh1MhSQIq+GPGwRq4L98ShU9Dkm44ng1ZQpuUEJrNA1o2FBqElj8oDnnm43gXPXR3RizIKtRORRiej7dqSMyGWdA0b1Ktx99VrD6mcSuX0qd+1UCLd91v2ZCo8YjGrfZeiPGQY1R3628BtPC9O9lninqT9ntbfYhuzdVVErIOnGcSroEvD9tdDochcfytqe0Ia1Iz/efjZTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E/1/PC2ZMw0rvgpLweirOTI/UNiY02ZFPp/DXWvgP/E=; b=F/Y4FxX9iYtnCgT/+oHbWCwE8rEsuF/xfxDFVKlNsc6wO4YNYFf/niqEyKJfYLUuw0eCRCifqN7WPfmvfISKRMNDDJbmJs2kMN+MwKeDW2+3qNdmZ+M3kpCLQwJwsqTkUPJfOl6wdq2084+8PgPXHHap69tL0NtlvRsUT80ughA= Received: from MN2PR05CA0065.namprd05.prod.outlook.com (2603:10b6:208:236::34) by DS5PPFBABE93B01.namprd12.prod.outlook.com (2603:10b6:f:fc00::65f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.23; Sun, 12 Jul 2026 07:41:07 +0000 Received: from BL02EPF0001A105.namprd05.prod.outlook.com (2603:10b6:208:236:cafe::83) by MN2PR05CA0065.outlook.office365.com (2603:10b6:208:236::34) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.223.9 via Frontend Transport; Sun, 12 Jul 2026 07:41:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A105.mail.protection.outlook.com (10.167.241.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:41:06 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sun, 12 Jul 2026 02:40:58 -0500 From: Shivank Garg Date: Sun, 12 Jul 2026 07:38:48 +0000 Subject: [PATCH 2/6] mm: factor out generic PTE batch detection from swap_pte_batch() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260712-migrate-rmap-batch-v1-2-872a734431d1@amd.com> References: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> In-Reply-To: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783842044; l=2080; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=IqFSvR/5vhwyfDVx/GvRf77BmahPitfvnb9fPwcJzUc=; b=Qu4R9pDnYXaCjWUgvj41jvI9UViuC+XkDfrHZ5vSEoD6lrAWzBOfNKJ0UvfIywHOmnlICdYBs JRi/btYvrFkCS5PjszyVQex+7dMEPcm2Qvw1iBixGVGTn60F+oO4ev3 X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A105:EE_|DS5PPFBABE93B01:EE_ X-MS-Office365-Filtering-Correlation-Id: dfd786f8-d981-4deb-5856-08dedfe8eefa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|376014|7416014|82310400026|23010399003|1800799024|921020|22082099003|18002099003|56012099006|11063799006; X-Microsoft-Antispam-Message-Info: jR/R+ZSdrw3XocVuFcba7InYmw1KbR2OCSMGgxBNwld2RwU93f4hM0O0IAjPIckpQls87g8/nGllsYuRjoSWmc7JVFN0cwUvusotactwd61Up6arjmx2YnQW/1m8WZfmJGXSpaPuEwldcB7+QQBqnS5JqssNy8hjeqizbUL5FclAVFYy5qvut3nM2tq/rWSWf3YN4E2LyjPWGJw9vxKCyt4s5vB5kROnayF/Q99PbmPbdX3afnhmDiT0RWyPPnXxTihCair0z9y/5ayhUK8hljiAJPEAChPidqTPNHiNPrpOvcFpXUim5Fwbu+3YFmPb6qZfgYm8zrE/tmfy1TnBPSmqg7uwBAXBJIgZvRTJF2f2/aIZ5YGzVLqyceP5WmlccaJ9uWWCgC3MIUuUeKp03PqcS2nBqHcu1Q+AktF0yEv3A0mMPEKuGXbBcTfoGS9c/vEByAdi8uH2IiVhYpeAwwRhyR+C7ND9i4DxjTp5VlFNk5IBB8euBfuWubwVVUX718YjqgNwwWRa/iw5MYkXkifx1Za/ARbOJUDgxS+Mqmo6d7Fe2qN9WqRWuQvCqDURtE1IMO9bNeJy6I6kWN7iyNzR1N4PbuoDR3VEkoChV1Y0eGjIl5GyOjNgcisT1tMsCUI+bxqTtvajIZYqiobCjzKsZSgxxpTZo0BCg3baxMwKoZmkSQNWj2qVeYA96/Nn8d51YaLb5wrN7HYVNbqNoxSSAFqSJVFJAoSEtvXspv8= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(376014)(7416014)(82310400026)(23010399003)(1800799024)(921020)(22082099003)(18002099003)(56012099006)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 6hne0cYhMbnR2XpMTn9xzmjGk8mp4EheUCU65QrQp21yJG2HtRkNx2LSN/4jL/k0rFNDcV5NdXINJGl5od6rConaBqIvJNbE2x88IR2AfIwoCIBCVmJL/BUAAcmvueLpRjxTH/TKt0Q1CQKfcMq3xBMElOqm6NCdgYJ7SGL2v8obJHjkCMp+Sfri3sPDa6ZLtEfU56TEPMIDxeH1vRMUmF5IhPfNOWLAOaBBaqH20ZRyOsG/ReufPN8Tw8hI4fyw5OO+9/Vy1oHnlNKV4fDpOsKTG+CBkpdODf2R6BzCOimV6ItxYKm/cG39Y9bbzfYEowOUWs+rlY/zgXeMNyAnpMe/nFOxBhPP+yacuWZ/3V3D9BRB4irUVbmisYLDO0pyPrqqXAlYdbQYmP9PsGuGGfc9/forIaLcRAaeU8xuqJQyCNMY7PlHC1jfEL635zTE X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2026 07:41:06.7547 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dfd786f8-d981-4deb-5856-08dedfe8eefa X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A105.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS5PPFBABE93B01 Factor out the generic PTE batch detection logic from swap_pte_batch() into softleaf_pte_batch() helper, so that it can be reused by upcoming patch that adds restore-side batching of migration entries. swap_pte_batch() now keeps only its swap-specific concerns and delegates the batch detection to softleaf_pte_batch(). No functional changes intended. Signed-off-by: Shivank Garg --- mm/internal.h | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 874be94cf257..a90721f8c687 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -505,6 +505,26 @@ static inline pte_t pte_next_swp_offset(pte_t pte) return pte_move_swp_offset(pte, 1); } =20 +static inline int softleaf_pte_batch(pte_t *start_ptep, int max_nr, pte_t = pte) +{ + pte_t expected_pte =3D pte_next_swp_offset(pte); + const pte_t *end_ptep =3D start_ptep + max_nr; + pte_t *ptep =3D start_ptep + 1; + + VM_WARN_ON(max_nr < 1); + + while (ptep < end_ptep) { + pte =3D ptep_get(ptep); + + if (!pte_same(pte, expected_pte)) + break; + expected_pte =3D pte_next_swp_offset(expected_pte); + ptep++; + } + + return ptep - start_ptep; +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. @@ -522,23 +542,9 @@ static inline pte_t pte_next_swp_offset(pte_t pte) */ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) { - pte_t expected_pte =3D pte_next_swp_offset(pte); - const pte_t *end_ptep =3D start_ptep + max_nr; - pte_t *ptep =3D start_ptep + 1; - - VM_WARN_ON(max_nr < 1); VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); =20 - while (ptep < end_ptep) { - pte =3D ptep_get(ptep); - - if (!pte_same(pte, expected_pte)) - break; - expected_pte =3D pte_next_swp_offset(expected_pte); - ptep++; - } - - return ptep - start_ptep; + return softleaf_pte_batch(start_ptep, max_nr, pte); } #endif /* CONFIG_MMU */ =20 --=20 2.43.0 From nobody Sat Jul 25 23:03:58 2026 Received: from SJ2PR03CU001.outbound.protection.outlook.com (mail-westusazon11012049.outbound.protection.outlook.com [52.101.43.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D0E9359A99 for ; Sun, 12 Jul 2026 07:41:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.43.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842081; cv=fail; b=YLcz+DUPy0N3e00gweFY5dN49m7khkn5/HD3ITpQkZIL5AyK6qT1J0YID5+8Yir1dpB11Fn8vZCj7xfaIGOdIFcumJZaf0DXcx3xQaqTxu7WMXzBQ7KNYJj17d8j6lA90kiNrPnFU0SuznZfsc98AyN9CuJFANTwmR3GqjE43dY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842081; c=relaxed/simple; bh=NiZVySPznUPmW9N2SAXqRfMvt9NyXeSzVqXhYk+QTIQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=rB6BL0E9vvkAEn4HWPIC9KPOpHBlXSKN1ZGDqavxvx1MZPAJrm3tKDMdNhQvDmsNFE15zdDktf5XMZ1T7oDiVPKdeCyi2lNI1jpqo/8LurX1e0IsdIsvNNya3uQungKjSxMmfGoUW6nBzQWqokcD+3Lo2yTg9a2UUZPXMvQfG6U= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=VecKv/LR; arc=fail smtp.client-ip=52.101.43.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="VecKv/LR" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DD68O0Ar7Js4yvEZ5c+R0/8/SP93a37JFg6WHh/S8uErRlrrkFoA2mrViLoD+c1rUQOIWUvZ4KmG/MxnuRWGkv8NGyNB1uYYwpXd8kv1jGodjeeAYiw8tlHVB2jplEbikRQ2yMDv9YrdpspI+HjB9Ogi/y555APPvw5wTcgPd4Pe+6lk2SHL6ObMDFhmxcijLPM+r9pcnt8RrovY41oljoTPL6rH/vUsGoABcxydqWZaeL69wJf+l1fShRwu77gTIpUH1NY3A5Kkb6Ypz7yhiLkcPhWycT1/Sx4bz73ZXIZw9PcZ7ZEfLXdLCrslHsAXNnrO4U4z8KgXYi/KCHvGcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VwQs3RKX7YYxLl9XMtvz/dS5L1JL+oJgeHqu+mDosi4=; b=jtmU55KPzGmgnDCS87x/Y4qf+IsxmuInzhz8MoYJo2yoFvc2rPHm0zx2c2iRrDZsNO/4PBmHuTkB3wAVquE1NcodFiVWhuO/0Uk1+vGpHpk3YRhgbvKfqp7nbOsqrALRSjfsTMWmTC6eIjYvcyp6J8qQmRg1IhzhuKYzHMWhKI1WK9PCh2wB8e8s8YcIhBfK+5j4yI00Fkp1JxSJVOieTF8+3xKPBf58msx+xnt7mme4UnI6Aso6zzp97sQMed5Y+VUI0422sa2/9rf9OM/XGlb8IMrIiyGBt0m8jStmEdqNpaJub4orIfsTisHA4C/yhII46OJEvYCKM05REiCz/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VwQs3RKX7YYxLl9XMtvz/dS5L1JL+oJgeHqu+mDosi4=; b=VecKv/LRKWWjc3Trz+Ncgd7sgigwW7IUJEdP+IbLIPez67Zr6J6Bjj4yd9UP0jCICku2LtSv4FZsj6WrF1WizBE0CGXK/0qydSRUX6bkOUwJVXIbiUKo55mf/7iXxG2VYQwnU/crpfkeQWYFifJN0jf7vGjkWoB6ghh3QCkuoAs= Received: from BLAPR05CA0044.namprd05.prod.outlook.com (2603:10b6:208:335::27) by DM4PR12MB6494.namprd12.prod.outlook.com (2603:10b6:8:ba::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.22; Sun, 12 Jul 2026 07:41:14 +0000 Received: from BL02EPF0001A103.namprd05.prod.outlook.com (2603:10b6:208:335:cafe::78) by BLAPR05CA0044.outlook.office365.com (2603:10b6:208:335::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.223.9 via Frontend Transport; Sun, 12 Jul 2026 07:41:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A103.mail.protection.outlook.com (10.167.241.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:41:13 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sun, 12 Jul 2026 02:41:06 -0500 From: Shivank Garg Date: Sun, 12 Jul 2026 07:38:49 +0000 Subject: [PATCH 3/6] mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260712-migrate-rmap-batch-v1-3-872a734431d1@amd.com> References: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> In-Reply-To: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783842044; l=7140; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=NiZVySPznUPmW9N2SAXqRfMvt9NyXeSzVqXhYk+QTIQ=; b=XWoiFd9XKKDU8A4EKlPdtlIcS0Tr7gHWBfBBTlNc4qLtGbvohSlYSZwI2MI4spNK1IEAgOJ7f I+IV48B8vhfCdHMLcMUPeLmbA5q9CivnVoSo/GY87mPekM2I8FErasc X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A103:EE_|DM4PR12MB6494:EE_ X-MS-Office365-Filtering-Correlation-Id: 160eeed6-9309-44b9-06f6-08dedfe8f335 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|23010399003|7416014|376014|1800799024|82310400026|921020|56012099006|11063799006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: kZ6MsGL32J5u+YseQPoBNghUTd2yPn9OKshjTe0geQvN/gpAdVY19zuhKOOH+KYYh/aMn8TlIZp+uwSw82Q6ZUyLSY610SY0Si6PuKgRVBWPq1qU/cnEvmkIdeINe95dVkwkMZjSWQaTxe1usNknkrPfXDR1G4MnTtr4AegZhAxA7Ca4++LEvTgRazan9lKSZfXriBhoRQfDeq7kPwsvzoiW8W0lsHYGpeM73GFM3oBmQo8LtgZfa+hAAGTXzPeqOLCpt5K2luH0k3rOm85UzJnQz3mBmZj+Tm9cQK0fzCRO5BnM5L3oxblj7RrTuJLSGoQNli6B7gcntmD0Cr+x9n7ZOFNStFU4iZuXkWpDNTHla59o7iBUJa+W+Kq1DTDmKPGF6Wqf+GldkD7vT8mk72bikKbjVIJMxvOXKVKpMgokEk+tY6su7ZeEGO4u5nJ8NMJq5OHlRhCmVQ78dDYcAiQDbmZyW5ARkfMhHRcke84uJgSFcMOPfcBIlab+VyrKtGYOgjl0HOnTXdlKXo+juGG2byWuQjuoor7DHa86wFsYkP7/lDWp7SrjB69nqQ5SKb9li2RODKObGxBkAb7TFzbqRTzfQ1TW++r6LtJe0ZwQePvlMu2hXTPS54O5yCqND2GtpIt/PgwoHjzOR/bShJrGtejtyann9A2sBfl95tdslDd+nMgiH3qLKu17hK4/Tn6jK+iJFqUFBcY4z1hhcX2OTc8agj9K4t2Y2VhMg/Z+4xcMz+d3TzjfzPMkxOq1 X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(23010399003)(7416014)(376014)(1800799024)(82310400026)(921020)(56012099006)(11063799006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: XjRiZsnMw6SnyEC3pZfBnCdVm2kmanuKmWUg+df7dRQcggHl9vBIFvE+ifnGxda+49FjxjoevISbVCMa+XtbY8GdTXhEzNJ+z8fIA2rnda8I01lWGCr7KQV1C+Jo+XcRFxP91016ELS4sJtbAPNEOp0hGcmS4z2sobaibybM7uwENbo2nyf4dkZEc/AR97bvQBhS7O7tm4TufouUDvb2oeNh7ixeYAyekCdI81VsONN6ZL/CKg/Ibn+nGK6o3kAdF1L4WeKB3V9Z7sDPdga4kDyvwTs0azLVtPWJXdCPn/R7wtdHNBbVWbya0Wh3nZ/Y5+zXS4zaaq31GsVfHuzDi7GwogiqsbUDqMfAYyAIwb6BPnYtIUZ4kFETKSVSDtBw4UzGUESfIyQMiCt7wtoHWDE8zTp8KsB0toY+rVCpXZ1xnloFwcFwUvjzKMNV3ZCy X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2026 07:41:13.8490 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 160eeed6-9309-44b9-06f6-08dedfe8f335 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A103.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6494 remove_migration_pte() interleaves the regular folio path with the hugetlb path via folio_test_hugetlb() and CONFIG_HUGETLB_PAGE special cases. Simplify this by seprating the hugetlb handling into remove_migration_pte_hugetlb(). Factor out working-PTE construction from migration entry into migration_softleaf_entry_to_pte(), used by both paths. While here, convert the VM_BUG_ON_FOLIO() to VM_WARN_ON_ONCE_FOLIO(). No functional change intended. Signed-off-by: Shivank Garg --- mm/migrate.c | 153 ++++++++++++++++++++++++++++++++++++++-----------------= ---- 1 file changed, 98 insertions(+), 55 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 222c8c15f782..8a098402b8c9 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -345,6 +345,89 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; =20 +static pte_t migration_softleaf_entry_to_pte(struct folio *folio, struct p= age *new, + softleaf_t entry, pte_t old_pte, struct vm_area_struct *vma, + rmap_t *rmap_flags) +{ + pte_t pte =3D mk_pte(new, READ_ONCE(vma->vm_page_prot)); + + if (!softleaf_is_migration_young(entry)) + pte =3D pte_mkold(pte); + if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry)) + pte =3D pte_mkdirty(pte); + if (pte_swp_soft_dirty(old_pte)) + pte =3D pte_mksoft_dirty(pte); + else + pte =3D pte_clear_soft_dirty(pte); + + if (softleaf_is_migration_write(entry)) + pte =3D pte_mkwrite(pte, vma); + else if (pte_swp_uffd(old_pte)) + pte =3D pte_mkuffd(pte); + + /* See do_swap_page(): restore PAGE_NONE for RWP */ + if (pte_swp_uffd(old_pte) && userfaultfd_rwp(vma)) + pte =3D pte_modify(pte, PAGE_NONE); + + if (folio_test_anon(folio) && !softleaf_is_migration_read(entry)) + *rmap_flags |=3D RMAP_EXCLUSIVE; + + return pte; +} + +/* + * Restore a potential migration pte to a working pte entry for hugetlb fo= lios. + */ +#ifdef CONFIG_HUGETLB_PAGE +static bool remove_migration_pte_hugetlb(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, void *arg) +{ + struct rmap_walk_arg *rmap_walk_arg =3D arg; + DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | = PVMW_MIGRATION); + struct hstate *h =3D hstate_vma(vma); + unsigned int shift =3D huge_page_shift(h); + unsigned long psize =3D huge_page_size(h); + + if (page_vma_mapped_walk(&pvmw)) { + rmap_t rmap_flags =3D RMAP_NONE; + struct page *new =3D folio_page(folio, 0); + pte_t old_pte, pte; + softleaf_t entry; + + old_pte =3D huge_ptep_get(vma->vm_mm, pvmw.address, pvmw.pte); + entry =3D softleaf_from_pte(old_pte); + + folio_get(folio); + pte =3D migration_softleaf_entry_to_pte(folio, new, entry, old_pte, + vma, &rmap_flags); + pte =3D arch_make_huge_pte(pte, shift, vma->vm_flags); + if (folio_test_anon(folio)) + hugetlb_add_anon_rmap(folio, vma, pvmw.address, + rmap_flags); + else + hugetlb_add_file_rmap(folio); + set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, psize); + + if (READ_ONCE(vma->vm_flags) & VM_LOCKED) + mlock_drain_local(); + + trace_remove_migration_pte(pvmw.address, pte_val(pte), + compound_order(new)); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, pvmw.address, pvmw.pte); + } + + return true; +} +#else +static bool remove_migration_pte_hugetlb(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, void *arg) +{ + return false; +} +#endif /* CONFIG_HUGETLB_PAGE */ + /* * Restore a potential migration pte to a working pte entry */ @@ -363,52 +446,28 @@ static bool remove_migration_pte(struct folio *folio, unsigned long idx =3D 0; =20 /* pgoff is invalid for ksm pages, but they are never large */ - if (folio_test_large(folio) && !folio_test_hugetlb(folio)) + if (folio_test_large(folio)) idx =3D linear_page_index(vma, pvmw.address) - pvmw.pgoff; new =3D folio_page(folio, idx); =20 #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES /* PMD-mapped THP migration entry */ if (!pvmw.pte) { - VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || - !folio_test_pmd_mappable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_pmd_mappable(folio), folio); remove_migration_pmd(&pvmw, new); continue; } #endif - if (folio_test_hugetlb(folio)) - old_pte =3D huge_ptep_get(vma->vm_mm, pvmw.address, - pvmw.pte); - else - old_pte =3D ptep_get(pvmw.pte); + old_pte =3D ptep_get(pvmw.pte); if (rmap_walk_arg->map_unused_to_zeropage && try_to_map_unused_to_zeropage(&pvmw, folio, old_pte, idx)) continue; =20 - folio_get(folio); - pte =3D mk_pte(new, READ_ONCE(vma->vm_page_prot)); - entry =3D softleaf_from_pte(old_pte); - if (!softleaf_is_migration_young(entry)) - pte =3D pte_mkold(pte); - if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry)) - pte =3D pte_mkdirty(pte); - if (pte_swp_soft_dirty(old_pte)) - pte =3D pte_mksoft_dirty(pte); - else - pte =3D pte_clear_soft_dirty(pte); =20 - if (softleaf_is_migration_write(entry)) - pte =3D pte_mkwrite(pte, vma); - else if (pte_swp_uffd(old_pte)) - pte =3D pte_mkuffd(pte); - - /* See do_swap_page(): restore PAGE_NONE for RWP */ - if (pte_swp_uffd(old_pte) && userfaultfd_rwp(vma)) - pte =3D pte_modify(pte, PAGE_NONE); - - if (folio_test_anon(folio) && !softleaf_is_migration_read(entry)) - rmap_flags |=3D RMAP_EXCLUSIVE; + folio_get(folio); + pte =3D migration_softleaf_entry_to_pte(folio, new, entry, old_pte, + vma, &rmap_flags); =20 if (unlikely(is_device_private_page(new))) { if (pte_write(pte)) @@ -424,30 +483,12 @@ static bool remove_migration_pte(struct folio *folio, pte =3D pte_swp_mkuffd(pte); } =20 -#ifdef CONFIG_HUGETLB_PAGE - if (folio_test_hugetlb(folio)) { - struct hstate *h =3D hstate_vma(vma); - unsigned int shift =3D huge_page_shift(h); - unsigned long psize =3D huge_page_size(h); - - pte =3D arch_make_huge_pte(pte, shift, vma->vm_flags); - if (folio_test_anon(folio)) - hugetlb_add_anon_rmap(folio, vma, pvmw.address, - rmap_flags); - else - hugetlb_add_file_rmap(folio); - set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, - psize); - } else -#endif - { - if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, new, vma, - pvmw.address, rmap_flags); - else - folio_add_file_rmap_pte(folio, new, vma); - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); - } + if (folio_test_anon(folio)) + folio_add_anon_rmap_pte(folio, new, vma, + pvmw.address, rmap_flags); + else + folio_add_file_rmap_pte(folio, new, vma); + set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); if (READ_ONCE(vma->vm_flags) & VM_LOCKED) mlock_drain_local(); =20 @@ -474,7 +515,9 @@ void remove_migration_ptes(struct folio *src, struct fo= lio *dst, }; =20 struct rmap_walk_control rwc =3D { - .rmap_one =3D remove_migration_pte, + .rmap_one =3D folio_test_hugetlb(src) ? + remove_migration_pte_hugetlb : + remove_migration_pte, .arg =3D &rmap_walk_arg, }; =20 --=20 2.43.0 From nobody Sat Jul 25 23:03:58 2026 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011052.outbound.protection.outlook.com [52.101.57.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 039552561A7 for ; Sun, 12 Jul 2026 07:41:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.52 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842089; cv=fail; b=dN3BiTgB+3MT68+rMeb8pAlN6PPZaVqhlaRPAPXB7WRVJUNKjXvsReGh4AsFJ8IV4705LZY7VBxYMvz+JG2IbAAyGYXH7bp8hQ1N+PODbvIZL3Qm8QcR6JnczwOlOgTDeyq7PLtGGLmsC64p6hauN6XCFkXGk7P7xNO5bw0k0Nk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842089; c=relaxed/simple; bh=KPzNifbI7PmPRQvQWc6sP+itQG02VGDINUXQGQWdZPQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=XVCr0aPF1GNK2J+nB1lDlq9YhkwwIMYX95xl1OZe1kWO6jmhOtxHq6+q7H0SNArZUZQUu6HY7nGFz+MKV8ImNmxihELoITM7nRy9R8pobjZt3qa0jsnjMGJtaJJoxWt+6ojp/qQabJiSU76nWzXxoRV+FDeWR0V6wPxmTjCexxI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=O6//IY6q; arc=fail smtp.client-ip=52.101.57.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="O6//IY6q" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uewvZedIh8p6mNJyNNJpc7SthXJZ6Z0aBopGvRnzPgDLXBgn+2/LxEy/ptSJFIwPkzxKuczV9i/5Pb/dOT8uDm7mFRQjqbLtgw4rCSxsov2bWRTd1m/Lr4J0dGDn9y5da3AwL/Lwd9UarLELj0uKEIXW/bVVbI9FIl+MJIzK/IroxcH6Mld7ttW0yH+/gPe2WpKtixii5UVfb05q0atBOK/vHo7a5GKuxN1zdZaOxIk6aeWRkedfRheouxi2nYr7N6vyoMDQ2J2146gmeoCZlplTA/RXTKtxaiyFaO0m0L6WHPzfYvl2GhXcPgznzvhv8etEn6WBBa4E4GGmDZGkFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rnMC8aGBoA4+MFdtra1b+Fcp5uvcOnfcFWmUVebdKcw=; b=QEzXQhJp6ubB04Uk2yeT/bRsOpizB1y4mrxJId8pVGqu/r5ccisO0SMhX0kq1q1T7rtGIDUmG6za0OkT1N4QtnGfDD2DEG4WVQ4nKbLv2lBuEB6piNr7kCnmYlHNBx1iyaHCqfCTw+Zepgje+4pzSPIpaY3sjOuxoRjzvYpVaOOT3UGLVrmMWoQLjylVCFFQzKJc+NKI7TAci6J0erswpoeBcPPlSWPzZRc3d9Mnz6HzkW0t0Ps7NlbkJDoWwx9Et+t5fsR81dA3YlIwdIh7FNa76MIAyVRPbacAjTc+WhEB9nllZ6CNbIp3PNZDUNyROLOxM2wmVE+ur2jTcyUXBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rnMC8aGBoA4+MFdtra1b+Fcp5uvcOnfcFWmUVebdKcw=; b=O6//IY6qs1wfgVlwcRiDts79vQfLPJ6skd4CzPnq0B7/3zEVxcp7w2oKE8CYBxG9ccPKasja3LruPHOHA/gUD7xbMBnOqCPo4g4SPrmcLc0oQ+B0qfx29AdS8qNSD409aDkxqoGZEpY/yjwzhAUSJt5SkslAO7czbaEmEiTLG2M= Received: from MN0PR04CA0011.namprd04.prod.outlook.com (2603:10b6:208:52d::7) by CY8PR12MB7193.namprd12.prod.outlook.com (2603:10b6:930:5b::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.22; Sun, 12 Jul 2026 07:41:22 +0000 Received: from BL02EPF0001A108.namprd05.prod.outlook.com (2603:10b6:208:52d:cafe::75) by MN0PR04CA0011.outlook.office365.com (2603:10b6:208:52d::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:41:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A108.mail.protection.outlook.com (10.167.241.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.9 via Frontend Transport; Sun, 12 Jul 2026 07:41:21 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sun, 12 Jul 2026 02:41:13 -0500 From: Shivank Garg Date: Sun, 12 Jul 2026 07:38:50 +0000 Subject: [PATCH 4/6] mm/migrate: batch the restore-side migration rmap walk Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260712-migrate-rmap-batch-v1-4-872a734431d1@amd.com> References: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> In-Reply-To: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783842044; l=4278; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=KPzNifbI7PmPRQvQWc6sP+itQG02VGDINUXQGQWdZPQ=; b=eXfk3cc3LyZQuBa6jNPVkKsajnJuqFzigyboOEqF8gJZP/Ihn8sotuuB4RsJ4NiJ0AFVLv/Ey CJ9K47gCWx3DHVNDTAJnD+fGIYEtBzWYfjzLFURsbXAO6hkoUG4lxsh X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A108:EE_|CY8PR12MB7193:EE_ X-MS-Office365-Filtering-Correlation-Id: a964e2bc-b374-4aaf-35df-08dedfe8f7f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700016|23010399003|22082099003|18002099003|56012099006|11063799006|921020; X-Microsoft-Antispam-Message-Info: oOp9+EMJSIt9hYyBnWzErkQ1IzPsQHS3p+ydwSZTpZfSIAWrfkIutyO0ONFDP85u5dhyWmP94AXuEsw1m6luMv9IhMSTVl+Wg6ZUpboVN/zk+l+TgQ4JbfRHpJ9jJ+c53siA7T7LpawWuSQLlteRzv4LhMWc3pkchnPTbIIymcwIzI9gMN8D0oi/O3xRBjYfaCNVya/jqYpxT2DVwOg18uUF5mngZVfFKSisQglk4GHZA7ykvEQ/W1ml5EOR3XDTitlfaBFRSCaOw9P+IWDXvePGr/4c5/thNPwzNJqirlK5lOdgCruFrDht+GxQVlaUUQXQKcGR5oX0v8yKn2yWLsanugYTa9tt8aHK74QyupZm6Z8llDpeIi8p3CYirL0cN+4AqHlOvGW9dTn+6L+qtgS7hr8VcKqHvjm429YIGezboE/EbHQaZYInKZuyEhDS3x6HFp0HIsZ0wVcCCRhLdkTkqZHFn2D2k/1MMEJ2avHPhYfDV2ysYAJAprbCYlZMaT3plsGoKiLmceLqy5G43xlyEsCpNMRxtcyOrKGmhnf4g3DcC1AGeVDlcemDfc9Q7dsIIemYeBLGbBLiJznjGDjLad2Gu59o7mZkZx6BC7T8LEvBJu/NFrMungXHdkD9dg6JC9nSq0LXEbT3cfAYoSMthxQaWNJsh6ubHT7AtlrjP41ilvSY8lytYO6HxdeUV+mLg1F+g2HkIZmrBcAdP6yLyFDPRgieVBq0Jj0AVoGjFDXC2IX5/COaBK8P4k7f X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700016)(23010399003)(22082099003)(18002099003)(56012099006)(11063799006)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: J8CZ9mRqfHETFMt8MwOxiBYMvgWWjFh4OcYmOVfKQ/bDsNce2n2WupoREONXauZrugy1JXh9QgmrMs2Yc7v7ETJRwN43P2W9owRxpyZOKp6dAp8J+ZdAKfP/fkGO8V4Il71ZSuVmyrHUEvzEsyhCUAsJ+g1D7Zm8D5P/s6Fkwf8bnh8w/EMzRG2aC95M968nXNAOy05X4d2wsQU+Pun9jB1/sVY278mEWGbwupxd45aPXODsCEjX2UIyEk2JpnahmWzV/XNNBB/uSP8YPs2hPs6IxEaGsIeZG+lzUidzTuoXaH97OhFmJiriVMPT1iF4ynP245olcVd9YLcRVzk566lIjj3fRnir6vGTnFIj8eahnAx1n+QJ/5m6FevA/o5m+vVV1wYF3eeVcnBr9hcy6CzzDjPtlRDvAe2SVNhOct3zpEP8LSbaCw5SwF3fZfuo X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2026 07:41:21.8419 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a964e2bc-b374-4aaf-35df-08dedfe8f7f7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A108.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7193 remove_migration_pte() restores migration entries one PTE at a time. For a PTE-mapped large folio, it repeats calls to building the PTE, add rmap, set_pte_at() and page_vma_mapped_walk() per base page (256 times for a 1M folio). Add migration_pte_batch() to detect batch of contiguous migration entries that map consecutive subpages, and have identical PTE bits. While at it, use folio_is_device_private() instead of is_device_private_page(). Signed-off-by: Shivank Garg --- mm/migrate.c | 49 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 7 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 8a098402b8c9..6d05d3f41095 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -345,6 +345,28 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; =20 +/* + * Detect a batch of contiguous migration entries: consecutive (non-presen= t) PTEs + * containing migration entries with consecutive offsets and matching pte = bits. + */ +static unsigned int migration_pte_batch(struct page_vma_mapped_walk *pvmw, + struct folio *folio, pte_t first_pte, unsigned long start_idx) +{ + struct vm_area_struct *vma =3D pvmw->vma; + unsigned long end_addr =3D pmd_addr_end(pvmw->address, vma->vm_end); + unsigned int folio_nr =3D folio_nr_pages(folio); + unsigned int max_nr; + + VM_WARN_ON(!softleaf_is_migration(softleaf_from_pte(first_pte))); + + /* Bound by VMA / PMD end and by the remaining subpages of the folio. */ + max_nr =3D min((end_addr - pvmw->address) >> PAGE_SHIFT, folio_nr - start= _idx); + if (max_nr <=3D 1) + return 1; + + return softleaf_pte_batch(pvmw->pte, max_nr, first_pte); +} + static pte_t migration_softleaf_entry_to_pte(struct folio *folio, struct p= age *new, softleaf_t entry, pte_t old_pte, struct vm_area_struct *vma, rmap_t *rmap_flags) @@ -435,6 +457,7 @@ static bool remove_migration_pte(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, void *arg) { struct rmap_walk_arg *rmap_walk_arg =3D arg; + bool is_devpriv =3D folio_is_device_private(folio); DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | = PVMW_MIGRATION); =20 while (page_vma_mapped_walk(&pvmw)) { @@ -444,6 +467,7 @@ static bool remove_migration_pte(struct folio *folio, softleaf_t entry; struct page *new; unsigned long idx =3D 0; + unsigned int nr =3D 1; =20 /* pgoff is invalid for ksm pages, but they are never large */ if (folio_test_large(folio)) @@ -465,11 +489,18 @@ static bool remove_migration_pte(struct folio *folio, =20 entry =3D softleaf_from_pte(old_pte); =20 - folio_get(folio); + /* + * Try to restore nr>1 contiguous PTEs in one shot. Falls back + * to original per-PTE path (nr=3D1) if batching is not possible. + */ + if (!rmap_walk_arg->map_unused_to_zeropage && likely(!is_devpriv)) + nr =3D migration_pte_batch(&pvmw, folio, old_pte, idx); + + folio_ref_add(folio, nr); pte =3D migration_softleaf_entry_to_pte(folio, new, entry, old_pte, vma, &rmap_flags); =20 - if (unlikely(is_device_private_page(new))) { + if (unlikely(is_devpriv)) { if (pte_write(pte)) entry =3D make_writable_device_private_entry( page_to_pfn(new)); @@ -484,11 +515,11 @@ static bool remove_migration_pte(struct folio *folio, } =20 if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, new, vma, - pvmw.address, rmap_flags); + folio_add_anon_rmap_ptes(folio, new, nr, vma, + pvmw.address, rmap_flags); else - folio_add_file_rmap_pte(folio, new, vma); - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); + folio_add_file_rmap_ptes(folio, new, nr, vma); + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); if (READ_ONCE(vma->vm_flags) & VM_LOCKED) mlock_drain_local(); =20 @@ -496,7 +527,11 @@ static bool remove_migration_pte(struct folio *folio, compound_order(new)); =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, pvmw.address, pvmw.pte); + update_mmu_cache_range(NULL, vma, pvmw.address, pvmw.pte, nr); + + /* Skip the batched PTEs */ + pvmw.pte +=3D nr - 1; + pvmw.address +=3D (nr - 1) * PAGE_SIZE; } =20 return true; --=20 2.43.0 From nobody Sat Jul 25 23:03:58 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012001.outbound.protection.outlook.com [52.101.48.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 381C82561A7 for ; Sun, 12 Jul 2026 07:41:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.1 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842100; cv=fail; b=W3LG8xq6ae6P73zhh08N9mXpa2sl0mVVbYPS41VtZYranIUYdXsBac10seafQ5Ff1pp6RquCyBsBj9Oin0+QA17AgaDf+z2TK31hkyT79DgHIIiouqOPQg15Nc1GPTpCUSoRoL1Ay3PUVet098RdPpQqLNkQHT25SFtBCTQJXOQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842100; c=relaxed/simple; bh=kA4wEi7ppwGHQEuijqPdO/498Z6da5jU8Oyz0X6nkYA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=tNyWq1ePajfe7q5bdblh+ltIPRONzKxi2xokiK6gsNE3LTPjRFG4vd6/yOPkUxxeldip9IPZwDSfUTK9akznzh1/lUZElABaxMbX1YpvGrTQfrEtYEjQqh/Ziau+cMUh4h3cWWmhbdzB1t15NqDiKJRqAKtDcw8Cy/JZmr1PdKE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QC3cKXF/; arc=fail smtp.client-ip=52.101.48.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QC3cKXF/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DG5+NymSfOAZqOIhijvwMu0eBN1WZAvciLgZda/R//sxwml/oKBHlC8Azle9dVRdAzkGOPRVZy8mrjkZ8Krk1G4F+HDdhxiQwpatl2T7bHz/Wc/MjibnthetFwMDvywFKF48/V4lFNnZRMGK/FwMENhKIRagmH0RmpyHbNKty3wZqHYGy1heNQvJu9UiBjBWsCrKLwOkWJmESeKdc6uLNH3u+OI+Ct9lLJkbey6jBvIsP7XSU4MFddotpEdLzLoBTYfSw3pqMXUAWElkzjdSQfSX0Ra/lNdLEZy/4CR8iipZtiZr+JJjnQ2TmTAIKqUchohQLnjrcl9njoLKggTdlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cDRb3q89wfKwwC3j/8iU8EEJ2BM5C0f3qSFbGzDevzI=; b=Phu/lXdfaT78VG9MpwAnXopPZaZpBhFr+6n7Etmxj6iMVEuNFLGm/mybn7NFmzf7gf3Wc/ERik5DMp8L7YAYZrdS+JtOzUjXFmDsWw6z8lc28zb1c6+RKa1AhpJD6aIrxCGpQe4YnnPlPg5/89pAHsd5ZulkRkI9f4DaPh2jjyFhY7L4plCrJsC1Xhb+TU/Ox6jNiQngVLGQGxKQp6/W+bj07QHzWdcioejLJ3+LPEqdnLe3oCdDfoD7UtlArm9h/WBBR4UNgE043g+Niql2tRImHFNlk3Trh56NLlshDOGtTzh/UwzCAJeFgKT7ecnePPpVueOp8q2DMciFZZJiTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cDRb3q89wfKwwC3j/8iU8EEJ2BM5C0f3qSFbGzDevzI=; b=QC3cKXF/NIfrN8g2Mf+V0CYMwuDFN6PW/UO+9LK3Od12p64xwJOUGAT9M3YNP0v/7tq2iFSqA+HhVMQi3XE6kkM6t6a/Pu7JAzecP0JM/klbZeb6iE+Odu8MrWnw3WkCZ5K9hWIG4pDfq7aN+A1Cp5cZszD7SBPabFgGkYehxos= Received: from MN2PR05CA0057.namprd05.prod.outlook.com (2603:10b6:208:236::26) by CH8PR12MB9742.namprd12.prod.outlook.com (2603:10b6:610:27a::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.19; Sun, 12 Jul 2026 07:41:30 +0000 Received: from BL02EPF0001A105.namprd05.prod.outlook.com (2603:10b6:208:236:cafe::98) by MN2PR05CA0057.outlook.office365.com (2603:10b6:208:236::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.223.9 via Frontend Transport; Sun, 12 Jul 2026 07:41:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A105.mail.protection.outlook.com (10.167.241.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:41:30 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sun, 12 Jul 2026 02:41:20 -0500 From: Shivank Garg Date: Sun, 12 Jul 2026 07:38:51 +0000 Subject: [PATCH 5/6] mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260712-migrate-rmap-batch-v1-5-872a734431d1@amd.com> References: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> In-Reply-To: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783842044; l=15491; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=kA4wEi7ppwGHQEuijqPdO/498Z6da5jU8Oyz0X6nkYA=; b=wRbrjbzqlIzHpuSnUEX/BHkdBXd/1TmBTOLV+WXvAfZ4v2sAJMMSDeDmqZXWa+e86rAx6FTaH ZD+wh8TX4aDCKLKdNxD+ZpNskRh8q2QAgydafmhrfjVI/Rg+isnacUS X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A105:EE_|CH8PR12MB9742:EE_ X-MS-Office365-Filtering-Correlation-Id: 23912fd4-225b-4b49-3674-08dedfe8fcef X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|376014|7416014|23010399003|1800799024|82310400026|921020|56012099006|11063799006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: ykaes4BDVlA2bRg/Ouxss2zWlINkBI/y1+8tPp+AddwNiogQNi0uZXhy3V2aPOwUNlxBZXj2NZu73eWxIY/sfhCQL/KhGS1pAMq/8NN0EswiVcCysofm5R+cuOaTxS5lLYVUfrBxecwwsZS1sHA9FgvONk6pdirqaGU6AM00+5VRoAkP9sFbFgEatJ530XZKFKxCx3QxISNSNn4k00IqWxMcAcQpX22II0RM3pWaXACP42H7UR2a5ejdNvITpXTmYjKDTWPYIn5wt7Q1PZ8TjJwfWyPbRcBVxJHI6unRFUZt+nS1EcBun+nwgYo5mlvYuL9FnSN4BrhjnmvVTfUxt0G+fgK5WHZ1cKdjoN+xtEZoyG8fSINuyjMVaiI3lJcUrIWjZvYbD2t6k8LHK9kCFpiiwYKuhR92VRvXl8srJQXiS5+bJ+G1UduoPnZyJ+stnKblgFQOJDEV4hltNO2NdEW42hN4Y0htINlapiJPafv3MyPCrTxPFyUl88IiDwXNEiPjwF1+hVbQiqMRRBBde7f6KpjlH21VEUKAF8VIV5ttyRYcvTnA79fnswu1gKQFF9195nR/IMcEXKid2cYohmmfYZ+GS8GyWjrZxIcN7DcU5tr+Jh6gYpT2zGtdxiNNcqu/EFGL8I+Q+zsTFOYeFOd3Mr1oJ+u5YyjKOOrbvz4y9Gm4gBZFo7/+rmISi/GZPdar75RRmaZ2IRmYz44YP0upWVocz58lzhAOaGpdZ23gCmI0Bojn0HvrU13CnCjK X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(376014)(7416014)(23010399003)(1800799024)(82310400026)(921020)(56012099006)(11063799006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DR+BKDQbUnmNlQuJ9S9NqwLPpYdimRT+BQcnfKopY71oLxWfxYsC0YkCyqWlfcTDP6E9iLxIsL2rIUBSN7J48cEx//3k8KrRThbPzICOoBPT6bg7detC+ou0J2uwQ9EWCQQxIwC+CiDIZ1FBP1miFokR0xBkW9euNTOHRxwjig8FcnGaOUOPmBpMKuvoSc0/dgeKROrn+GMIqwofbdQiLarxrplI7ynz1F3mtQMtHeHv4IiB30exUnCxTchGak2MIXdfTv7FrqvP7FKEUot2ycfdhTG87qc+gxQeLamjIr1FafqEofMtNiSRAtSgaLmIxWQxMc+jWe6tKXUtLIrdPcnS1B8cw+atHuw/gTAOqj4sjPbdmJXKBXCmXyUMgzO8EQJjW+Rzf5Q9dj24Cp2TmdKhpytVQnDjcTjtHZVvf6OFoYwdCK3MKnEdvoDSJy8E X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2026 07:41:30.1765 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 23912fd4-225b-4b49-3674-08dedfe8fcef X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A105.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH8PR12MB9742 try_to_migrate_one() interleaves hugetlb special handling through the regular PTE/PMD migration path with folio_test_hugetlb() checks scattered across the walk. Simplify this by moving hugetlb handling into its own rmap_one callback, try_to_migrate_hugetlb_one(). Factor the migration swap-pte construction shared by both hugetlb and regular folios path into make_migration_swp_pte(). Few minor changes: - Since hugetlb uses separate counters, update_hiwater_rss() is not needed for hugetlb path. Also, use huge_pte_dirty instead of pte_dirty - No PVMW_SYNC flag is needed for hugetlb - hugetlb folio has single mapping per VMA, so no loop is needed. - Convert VM_BUG_ON to VM_WARN_ON_ONCE and not use irrelevant WARN check. Signed-off-by: Shivank Garg --- mm/rmap.c | 329 +++++++++++++++++++++++++++++++++++++---------------------= ---- 1 file changed, 195 insertions(+), 134 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index e854679553b9..ef68adba89c4 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2405,6 +2405,187 @@ void try_to_unmap(struct folio *folio, enum ttu_fla= gs flags) rmap_walk(folio, &rwc); } =20 +/* Build the migration swap PTE that replaces pteval for a page. */ +static pte_t make_migration_swp_pte(struct page *page, pte_t pteval, + bool writable, bool anon_exclusive) +{ + swp_entry_t entry; + pte_t swp_pte; + + if (writable) + entry =3D make_writable_migration_entry(page_to_pfn(page)); + else if (anon_exclusive) + entry =3D make_readable_exclusive_migration_entry( + page_to_pfn(page)); + else + entry =3D make_readable_migration_entry(page_to_pfn(page)); + + if (likely(pte_present(pteval))) { + if (pte_young(pteval)) + entry =3D make_migration_entry_young(entry); + if (pte_dirty(pteval)) + entry =3D make_migration_entry_dirty(entry); + swp_pte =3D swp_entry_to_pte(entry); + if (pte_soft_dirty(pteval)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd(pteval)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } else { + swp_pte =3D swp_entry_to_pte(entry); + if (pte_swp_soft_dirty(pteval)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd(pteval)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } + + return swp_pte; +} + +/* + * Replace the hugetlb page table mapping for folio with migration entries. + * + * @arg: enum ttu_flags will be passed to this argument. + */ +static bool try_to_migrate_hugetlb_one(struct folio *folio, + struct vm_area_struct *vma, unsigned long address, void *arg) +{ + const unsigned long hsz =3D huge_page_size(hstate_vma(vma)); + DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + enum ttu_flags flags =3D (enum ttu_flags)(long)arg; + bool anon_exclusive, writable, ret =3D true; + bool anon =3D folio_test_anon(folio); + struct mm_struct *mm =3D vma->vm_mm; + struct mmu_notifier_range range; + unsigned long pfn; + struct page *page; + pte_t pteval; + + range.end =3D vma_address_end(&pvmw); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, + address, range.end); + /* If sharing is possible, start and end will be adjusted accordingly. */ + adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); + mmu_notifier_invalidate_range_start(&range); + + if (page_vma_mapped_walk(&pvmw)) { + address =3D pvmw.address; + pteval =3D huge_ptep_get(mm, address, pvmw.pte); + if (likely(pte_present(pteval))) + pfn =3D pte_pfn(pteval); + else + pfn =3D softleaf_to_pfn(softleaf_from_pte(pteval)); + + page =3D folio_page(folio, pfn - folio_pfn(folio)); + anon_exclusive =3D anon && PageAnonExclusive(page); + + /* + * huge_pmd_unshare may unmap an entire PMD page. There is no + * way of knowing exactly which PMDs may be cached for this mm, + * so we must flush them all. start/end were already adjusted + * above to cover this range. + */ + flush_cache_range(vma, range.start, range.end); + + /* + * To call huge_pmd_unshare, i_mmap_rwsem must be held in write + * mode. Caller needs to explicitly do this outside rmap + * routines. We also must hold hugetlb vma_lock in write mode. + * Lock order dictates acquiring vma_lock BEFORE i_mmap_rwsem. + * We can only try lock here and fail if unsuccessful. + */ + if (!anon) { + struct mmu_gather tlb; + + VM_WARN_ON_ONCE(!(flags & TTU_RMAP_LOCKED)); + if (!hugetlb_vma_trylock_write(vma)) { + ret =3D false; + goto walk_done; + } + + tlb_gather_mmu_vma(&tlb, vma); + if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) { + hugetlb_vma_unlock_write(vma); + huge_pmd_unshare_flush(&tlb, vma); + tlb_finish_mmu(&tlb); + /* + * The PMD table was unmapped, + * consequently unmapping the folio. + */ + goto walk_done; + } + hugetlb_vma_unlock_write(vma); + tlb_finish_mmu(&tlb); + } + + /* Nuke the hugetlb page table entry */ + pteval =3D huge_ptep_clear_flush(vma, address, pvmw.pte); + if (huge_pte_dirty(pteval)) + folio_mark_dirty(folio); + writable =3D pte_write(pteval); + + VM_WARN_ON_FOLIO(writable && anon && !anon_exclusive, folio); + + if (PageHWPoison(page)) { + pteval =3D swp_entry_to_pte(make_hwpoison_entry(page)); + hugetlb_count_sub(folio_nr_pages(folio), mm); + set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); + } else if (likely(pte_present(pteval)) && pte_unused(pteval) && + !userfaultfd_armed(vma)) { + /* + * The guest indicated that the page content is of no + * interest anymore. Simply discard the pte, vmscan + * will take care of the rest. + */ + dec_mm_counter(mm, mm_counter(folio)); + } else { + pte_t swp_pte; + + /* + * arch_unmap_one() is expected to be a NOP on + * architectures where we could have PFN swap PTEs, + * so we'll not check/care. + */ + if (arch_unmap_one(mm, vma, address, pteval) < 0) { + set_huge_pte_at(mm, address, pvmw.pte, + pteval, hsz); + ret =3D false; + goto walk_done; + } + + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + if (anon_exclusive && + hugetlb_try_share_anon_rmap(folio)) { + set_huge_pte_at(mm, address, pvmw.pte, + pteval, hsz); + ret =3D false; + goto walk_done; + } + + /* + * Store the pfn of the page in a special migration + * pte. do_swap_page() will wait until the migration + * pte is removed and then restart fault handling. + */ + swp_pte =3D make_migration_swp_pte(page, pteval, + writable, anon_exclusive); + set_huge_pte_at(mm, address, pvmw.pte, swp_pte, hsz); + trace_set_migration_pte(address, pte_val(swp_pte), + folio_order(folio)); + } + + hugetlb_remove_rmap(folio); + if (vma->vm_flags & VM_LOCKED) + mlock_drain_local(); + folio_put(folio); +walk_done: + page_vma_mapped_walk_done(&pvmw); + } + + mmu_notifier_invalidate_range_end(&range); + + return ret; +} + /* * @arg: enum ttu_flags will be passed to this argument. * @@ -2422,7 +2603,6 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, struct mmu_notifier_range range; enum ttu_flags flags =3D (enum ttu_flags)(long)arg; unsigned long pfn; - unsigned long hsz =3D 0; =20 /* * When racing against e.g. zap_pte_range() on another cpu, @@ -2435,26 +2615,13 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, =20 /* * For THP, we have to assume the worse case ie pmd for invalidation. - * For hugetlb, it could be much worse if we need to do pud - * invalidation in the case of pmd sharing. * * Note that the page can not be free in this function as call of - * try_to_unmap() must hold a reference on the page. + * try_to_migrate() must hold a reference on the page. */ range.end =3D vma_address_end(&pvmw); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, address, range.end); - if (folio_test_hugetlb(folio)) { - /* - * If sharing is possible, start and end will be adjusted - * accordingly. - */ - adjust_range_if_pmd_sharing_possible(vma, &range.start, - &range.end); - - /* We need the huge page size for set_huge_pte_at() */ - hsz =3D huge_page_size(hstate_vma(vma)); - } mmu_notifier_invalidate_range_start(&range); =20 while (page_vma_mapped_walk(&pvmw)) { @@ -2501,10 +2668,7 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, VM_BUG_ON_FOLIO(!pvmw.pte, folio); =20 address =3D pvmw.address; - if (folio_test_hugetlb(folio)) - pteval =3D huge_ptep_get(mm, address, pvmw.pte); - else - pteval =3D ptep_get(pvmw.pte); + pteval =3D ptep_get(pvmw.pte); if (likely(pte_present(pteval))) { pfn =3D pte_pfn(pteval); } else { @@ -2515,66 +2679,13 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, const softleaf_t entry =3D softleaf_from_pte(pteval); =20 pfn =3D softleaf_to_pfn(entry); - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); } =20 subpage =3D folio_page(folio, pfn - folio_pfn(folio)); anon_exclusive =3D folio_test_anon(folio) && PageAnonExclusive(subpage); =20 - if (folio_test_hugetlb(folio)) { - bool anon =3D folio_test_anon(folio); - - /* - * huge_pmd_unshare may unmap an entire PMD page. - * There is no way of knowing exactly which PMDs may - * be cached for this mm, so we must flush them all. - * start/end were already adjusted above to cover this - * range. - */ - flush_cache_range(vma, range.start, range.end); - - /* - * To call huge_pmd_unshare, i_mmap_rwsem must be - * held in write mode. Caller needs to explicitly - * do this outside rmap routines. - * - * We also must hold hugetlb vma_lock in write mode. - * Lock order dictates acquiring vma_lock BEFORE - * i_mmap_rwsem. We can only try lock here and - * fail if unsuccessful. - */ - if (!anon) { - struct mmu_gather tlb; - - VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); - if (!hugetlb_vma_trylock_write(vma)) { - page_vma_mapped_walk_done(&pvmw); - ret =3D false; - break; - } - - tlb_gather_mmu_vma(&tlb, vma); - if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) { - hugetlb_vma_unlock_write(vma); - huge_pmd_unshare_flush(&tlb, vma); - tlb_finish_mmu(&tlb); - /* - * The PMD table was unmapped, - * consequently unmapping the folio. - */ - page_vma_mapped_walk_done(&pvmw); - break; - } - hugetlb_vma_unlock_write(vma); - tlb_finish_mmu(&tlb); - } - /* Nuke the hugetlb page table entry */ - pteval =3D huge_ptep_clear_flush(vma, address, pvmw.pte); - if (pte_dirty(pteval)) - folio_mark_dirty(folio); - writable =3D pte_write(pteval); - } else if (likely(pte_present(pteval))) { + if (likely(pte_present(pteval))) { flush_cache_page(vma, address, pfn); /* Nuke the page table entry. */ if (should_defer_flush(mm, flags)) { @@ -2613,14 +2724,8 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, VM_WARN_ON_FOLIO(folio_is_device_private(folio), folio); =20 pteval =3D swp_entry_to_pte(make_hwpoison_entry(subpage)); - if (folio_test_hugetlb(folio)) { - hugetlb_count_sub(folio_nr_pages(folio), mm); - set_huge_pte_at(mm, address, pvmw.pte, pteval, - hsz); - } else { - dec_mm_counter(mm, mm_counter(folio)); - set_pte_at(mm, address, pvmw.pte, pteval); - } + dec_mm_counter(mm, mm_counter(folio)); + set_pte_at(mm, address, pvmw.pte, pteval); } else if (likely(pte_present(pteval)) && pte_unused(pteval) && !userfaultfd_armed(vma)) { /* @@ -2635,7 +2740,6 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, */ dec_mm_counter(mm, mm_counter(folio)); } else { - swp_entry_t entry; pte_t swp_pte; =20 /* @@ -2644,28 +2748,15 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, * so we'll not check/care. */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { - if (folio_test_hugetlb(folio)) - set_huge_pte_at(mm, address, pvmw.pte, - pteval, hsz); - else - set_pte_at(mm, address, pvmw.pte, pteval); + set_pte_at(mm, address, pvmw.pte, pteval); ret =3D false; page_vma_mapped_walk_done(&pvmw); break; } =20 /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ - if (folio_test_hugetlb(folio)) { - if (anon_exclusive && - hugetlb_try_share_anon_rmap(folio)) { - set_huge_pte_at(mm, address, pvmw.pte, - pteval, hsz); - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; - } - } else if (anon_exclusive && - folio_try_share_anon_rmap_pte(folio, subpage)) { + if (anon_exclusive && + folio_try_share_anon_rmap_pte(folio, subpage)) { set_pte_at(mm, address, pvmw.pte, pteval); ret =3D false; page_vma_mapped_walk_done(&pvmw); @@ -2677,37 +2768,9 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, * pte. do_swap_page() will wait until the migration * pte is removed and then restart fault handling. */ - if (writable) - entry =3D make_writable_migration_entry( - page_to_pfn(subpage)); - else if (anon_exclusive) - entry =3D make_readable_exclusive_migration_entry( - page_to_pfn(subpage)); - else - entry =3D make_readable_migration_entry( - page_to_pfn(subpage)); - if (likely(pte_present(pteval))) { - if (pte_young(pteval)) - entry =3D make_migration_entry_young(entry); - if (pte_dirty(pteval)) - entry =3D make_migration_entry_dirty(entry); - swp_pte =3D swp_entry_to_pte(entry); - if (pte_soft_dirty(pteval)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd(pteval)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } else { - swp_pte =3D swp_entry_to_pte(entry); - if (pte_swp_soft_dirty(pteval)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_swp_uffd(pteval)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } - if (folio_test_hugetlb(folio)) - set_huge_pte_at(mm, address, pvmw.pte, swp_pte, - hsz); - else - set_pte_at(mm, address, pvmw.pte, swp_pte); + swp_pte =3D make_migration_swp_pte(subpage, pteval, + writable, anon_exclusive); + set_pte_at(mm, address, pvmw.pte, swp_pte); trace_set_migration_pte(address, pte_val(swp_pte), folio_order(folio)); /* @@ -2716,10 +2779,7 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, */ } =20 - if (unlikely(folio_test_hugetlb(folio))) - hugetlb_remove_rmap(folio); - else - folio_remove_rmap_pte(folio, subpage, vma); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2741,7 +2801,8 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, void try_to_migrate(struct folio *folio, enum ttu_flags flags) { struct rmap_walk_control rwc =3D { - .rmap_one =3D try_to_migrate_one, + .rmap_one =3D folio_test_hugetlb(folio) ? + try_to_migrate_hugetlb_one : try_to_migrate_one, .arg =3D (void *)flags, .done =3D folio_not_mapped, .anon_lock =3D folio_lock_anon_vma_read, --=20 2.43.0 From nobody Sat Jul 25 23:03:58 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012063.outbound.protection.outlook.com [52.101.53.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C440D306764 for ; Sun, 12 Jul 2026 07:41:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842108; cv=fail; b=MDC9Gz8tCATkdB/8lLWoXexdi646PJKC7d7GW8G60FYK5x0J8GxZ4WGn3xYuDHcsYG7pgGorVkkAvW0bJ8uUebMBg5RugnjqDQRh01Ru6ctnNzxXaFLeGpw4jPtBfyVYvJwfIEpIPvtNyqxUO+MdzZ9QxCgPTbnoj/SsXJkYOLo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783842108; c=relaxed/simple; bh=DaXaWAks3ni26r5eG7Mgqp8Z7R1+V8py4omKSnb7ukA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=KzfjZJ0xWZmVJvLpxFcecBiiTT9RuOry6bXBdrR8i9oRegXbHqy6WmUie9KCYFs6X/XkcDBC8vgTA+LZO0VLlXryYKPNlZhhXK4seoN6FG+SkEq5z7LBqS4lQ2N0aujXReZZvjAIk6GW5hdOiSd5dDJcqRT4KoyHlyOZY/9qzjM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=krcXOpEK; arc=fail smtp.client-ip=52.101.53.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="krcXOpEK" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XaBSU8sFoFr5qQl+YLaI9r34/ifr+I04384vCh7NdSgXTdTTNny2Tfxs4xOCdNAitNSp3v2yeZ/VzvTHJUt3vg2cZ6p8/Js0XOC2we57zojv+T0Pz8FvZr4ezhZhFz6UqtdqiN/CXm32k039UCSVm1a1BkTOksnnUMi0q8cVP52tqUVHni4qI7KSxWG32u12W04tCXE7evQVXMokGEul2meWTQH/7IoNFumaV6JESnDr7ffDaJbBAvgudKahHqzSEQF0YaBCHlr3ELLM4jFVlX77Y7Zum57lf45lkwjeSnDm/d14bOvI07q+RwIVtzYujcda4flxtxjmaZAABV804A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=T4/By1nSX/TaUxcLN/jiwd7uO1bHTgeQaXSwpyDwmao=; b=UUbr60gh9uEzrE94ux2AflIR0XT07m9CrvmPIfrp7768IDCC+PYOT1LFLu9Vg0OkmVeNfoaujKL1WEIhwls6qbjTg0iNIaNmWKILI6OVYMtMGWVf4xQ0k2WDyaLX8dHtUXYfvVEMLUOYsZSf0LzNQNDSrX5VRzmpIVWW4OBIhbYp9eLhg1tInLbO0ezQBe6NGmdvU3g8605Ocb4tveglUCeICx63pUQeh5boNGS6kJNd0esfsXwwSeey10hh0K+4+yLffwLWFuIHdBEjTEo94wEIKsHtxuz4aSn3Q0yASqG43zVhG/81bnDeCppOWcmMOeC791+VVjS5P7fVygf/yA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T4/By1nSX/TaUxcLN/jiwd7uO1bHTgeQaXSwpyDwmao=; b=krcXOpEKDH3Du2lcJUH/7h6ew5o3D5PjeXB9YTa20mE5ezIxJ9vLBJ7KFP7vAL9i65acAqOKXCcvdk1Hhb90vUDsdtBcohijEcR5lIvshG3NiWzxK0Kpd6+3IkOHZDMyLrlSPKCzZaE0uSZVQHvq0yqKdnLmP3zXadhANO0OvWw= Received: from MN2PR18CA0003.namprd18.prod.outlook.com (2603:10b6:208:23c::8) by DS7PR12MB6118.namprd12.prod.outlook.com (2603:10b6:8:9a::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.22; Sun, 12 Jul 2026 07:41:36 +0000 Received: from BL02EPF0001A107.namprd05.prod.outlook.com (2603:10b6:208:23c:cafe::63) by MN2PR18CA0003.outlook.office365.com (2603:10b6:208:23c::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:41:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0001A107.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.202.18 via Frontend Transport; Sun, 12 Jul 2026 07:41:35 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Sun, 12 Jul 2026 02:41:28 -0500 From: Shivank Garg Date: Sun, 12 Jul 2026 07:38:52 +0000 Subject: [PATCH 6/6] mm/rmap: batch the unmap of large folios in try_to_migrate_one() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260712-migrate-rmap-batch-v1-6-872a734431d1@amd.com> References: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> In-Reply-To: <20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783842044; l=7150; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=DaXaWAks3ni26r5eG7Mgqp8Z7R1+V8py4omKSnb7ukA=; b=ptWuqXtlG/w5MOWYfGviPHSj9VRSJnXloZaQ7AmlnebTb5eZBIPSDJfKm0nD+4FpnRYKKBJS5 gp4GDZBBzEbC2iSijZXydaAPTFBKnO4dcduYaUazT44ap6j7+S7wPEW X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A107:EE_|DS7PR12MB6118:EE_ X-MS-Office365-Filtering-Correlation-Id: 9e0ec902-ce60-446f-1ac0-08dedfe90056 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|7416014|36860700016|376014|23010399003|56012099006|11063799006|5023799004|18002099003|22082099003|921020; X-Microsoft-Antispam-Message-Info: BPn8+5xcZ4GnQXoipR4qZCWdppc+TUkZcu0G8/rdnq/bqt1c1bR/UoJStqh22Lk6uj8uA1D+nbKH1Uec7XpqcCXRUR62YnQ3f/GANhUfWh999fgBijHtr8a6JrcWfwDa/g4wR8+E8jkbLc94pwKCJUPq5c5vVSeiEM1hZgprnNtOw1D9wAujfe6M9CEsZAQhiXD0RJ24iLNM01jHurk2HdexAaQacdpf5WnghpSBWiEYxD3JVOK8cqmtHOafaP5aiNGVPK8UkH8sOh98tBPcJqKJo/OHR7h0EAxlJ1GoCbDzvBYn3i6t0JvWm5CAQ4g6zoMrUl8fYHUTHWTJ+r01mRutG9vnLlPM7iN4DjToz+AbPKDLbQGQLAdKdPuOahxQ60Q8nB+jgQHWycy4ABjZposYb/aoZSw9N1eoi741C6ewRaG44CZTyDYJ2EsRNlqNqJ9HkWu66VkGpo3uQHNlGSVq1pDFga/RAEK2WDVJFwlj7ef8KHmwjNOykCeUfHbUwk9G9+OlbaKSxUpAIs2Jc7nKGqmoJ500nF1EsKKyaP7Hm7o4ARBgpY3DFapA+lXUPWqTLAy/l7isgJXeng07IhO8Y5N67VgOwdatn/X1x8DKe/RU5+Ycp5Jv0pxv7x9OVjet5TJeJgcI9q45DrYUP3EcKrXl2q7c7sb2UlLjkMTLxVJ8PVriDGcYXypsvazXeCW9Yo5xL5c8sISn2Zn4+SrJfcdWAa3BBauAeu2Ge+k+AhqIido8/KcmQGp5Baga X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(7416014)(36860700016)(376014)(23010399003)(56012099006)(11063799006)(5023799004)(18002099003)(22082099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 3gYw0DT34VqPFUwl9bANQQRC+tfJNWTOm/ZqT3EwqQbGVT5lvsGNs2wrG/FrLHtus2+hfCowlkin04gl8hTt735ahncwGh6TsDdhkL46P0uRY3o2mYaUIBdnzBOAdCnhHOdpCN/xfS9w0GT90mVHZyzRqasM6mbijmpqNk0fKhPz9MNVknKMa+BcJVq+naQYWbNLozxQacsVLS2sFjeNc0r2m5MJUnbL5XzYpLhRwaU2PNcjrYSQ+xkSgMyR04o2dV84+GvSRNM+baZ2vykJsbcGwbrk+3RKwvJ/EZQrg+BBz9cbJLM1EoyCpCSqS5D3/MtwsP4VoHcdwSnlEebQpUVxIXMvQ5tztPuvjEHqNS++Is6SZrEJbTVin1UZjKf5yF9ZP1IFXlYiAEsa5dQ0rTTLadRA9lEB2dU8X99URP1C908vPiQKE3iXHu0xhBTb X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2026 07:41:35.8768 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9e0ec902-ce60-446f-1ac0-08dedfe90056 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A107.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6118 try_to_migrate_one() converts present PTEs to migration entries one at a time. For a PTE-mapped large folio, this repeat calls to ptep clear+flush, the migration entry build and set, folio_remove_rmap_pte() and folio_put(), each re-entering page_vma_mapped_walk() once per base page (256 times for 1M folio). Mirror try_to_unmap_one() to introduce folio_migrate_pte_batch() to detect eligible batch for PTEs mapping conseuctive subpages of a large folios, and convert the whole batch in one shot using the batched helpers. A side-effect of this change is trace_set_migration_pte() will record one event per batched run instead of earlier behavior of one per base page. Signed-off-by: Shivank Garg --- mm/rmap.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++++----------= ---- 1 file changed, 85 insertions(+), 25 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index ef68adba89c4..cfac85f6f68a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2586,6 +2586,48 @@ static bool try_to_migrate_hugetlb_one(struct folio = *folio, return ret; } =20 +static inline unsigned int folio_migrate_pte_batch(struct folio *folio, + struct page_vma_mapped_walk *pvmw, pte_t pte, + struct page *subpage, bool anon_exclusive) +{ + unsigned long end_addr, addr =3D pvmw->address; + struct vm_area_struct *vma =3D pvmw->vma; + unsigned int max_nr, nr, i; + +#ifdef __HAVE_ARCH_UNMAP_ONE + /* Cannot batch unmap if arch_unmap_one() is defined. */ + return 1; +#endif + + if (!folio_test_large(folio)) + return 1; + if (folio_is_zone_device(folio) || folio_test_has_hwpoisoned(folio)) + return 1; + if (pte_unused(pte)) + return 1; + + /* We may only batch within a single VMA and a single page table. */ + end_addr =3D pmd_addr_end(addr, vma->vm_end); + max_nr =3D (end_addr - addr) >> PAGE_SHIFT; + /* + * If unmap fails, we need to restore the ptes. To avoid accidentally + * upgrading write permissions for ptes that were not originally writable, + * and to avoid losing the soft-dirty bit, use the appropriate FPB flags. + */ + nr =3D folio_pte_batch_flags(folio, vma, pvmw->pte, &pte, max_nr, + FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY); + + /* Limit possible batch count to a uniform PageAnonExclusive value */ + if (folio_test_anon(folio)) { + for (i =3D 1; i < nr; i++) + if (PageAnonExclusive(subpage + i) !=3D anon_exclusive) + break; + nr =3D i; + } + + return nr; +} + /* * @arg: enum ttu_flags will be passed to this argument. * @@ -2602,7 +2644,7 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, struct page *subpage; struct mmu_notifier_range range; enum ttu_flags flags =3D (enum ttu_flags)(long)arg; - unsigned long pfn; + unsigned long pfn, end_addr, nr_pages; =20 /* * When racing against e.g. zap_pte_range() on another cpu, @@ -2655,11 +2697,8 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || !folio_test_pmd_mappable(folio), folio); =20 - if (set_pmd_migration_entry(&pvmw, subpage)) { - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; - } + if (set_pmd_migration_entry(&pvmw, subpage)) + goto walk_abort; continue; #endif } @@ -2685,9 +2724,16 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, anon_exclusive =3D folio_test_anon(folio) && PageAnonExclusive(subpage); =20 + nr_pages =3D 1; if (likely(pte_present(pteval))) { - flush_cache_page(vma, address, pfn); - /* Nuke the page table entry. */ + nr_pages =3D folio_migrate_pte_batch(folio, &pvmw, pteval, + subpage, anon_exclusive); + + end_addr =3D address + nr_pages * PAGE_SIZE; + flush_cache_range(vma, address, end_addr); + + /* Nuke the page table entries. */ + pteval =3D get_and_clear_ptes(mm, address, pvmw.pte, nr_pages); if (should_defer_flush(mm, flags)) { /* * We clear the PTE but do not flush so potentially @@ -2697,11 +2743,9 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, * transition on a cached TLB entry is written through * and traps if the PTE is unmapped. */ - pteval =3D ptep_get_and_clear(mm, address, pvmw.pte); - - set_tlb_ubc_flush_pending(mm, pteval, address, address + PAGE_SIZE); + set_tlb_ubc_flush_pending(mm, pteval, address, end_addr); } else { - pteval =3D ptep_clear_flush(vma, address, pvmw.pte); + flush_tlb_range(vma, address, end_addr); } if (pte_dirty(pteval)) folio_mark_dirty(folio); @@ -2741,6 +2785,7 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, dec_mm_counter(mm, mm_counter(folio)); } else { pte_t swp_pte; + unsigned int i; =20 /* * arch_unmap_one() is expected to be a NOP on @@ -2748,19 +2793,15 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, * so we'll not check/care. */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { - set_pte_at(mm, address, pvmw.pte, pteval); - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; + set_ptes(mm, address, pvmw.pte, pteval, nr_pages); + goto walk_abort; } =20 - /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + /* See folio_try_share_anon_rmap_ptes(): clear PTE first. */ if (anon_exclusive && - folio_try_share_anon_rmap_pte(folio, subpage)) { - set_pte_at(mm, address, pvmw.pte, pteval); - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; + folio_try_share_anon_rmap_ptes(folio, subpage, nr_pages)) { + set_ptes(mm, address, pvmw.pte, pteval, nr_pages); + goto walk_abort; } =20 /* @@ -2770,19 +2811,38 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, */ swp_pte =3D make_migration_swp_pte(subpage, pteval, writable, anon_exclusive); - set_pte_at(mm, address, pvmw.pte, swp_pte); trace_set_migration_pte(address, pte_val(swp_pte), folio_order(folio)); + + /* Set nr_pages migration entries, advancing the PFN. */ + for (i =3D 0; i < nr_pages; i++) { + set_pte_at(mm, address + (unsigned long)i * PAGE_SIZE, + pvmw.pte + i, swp_pte); + swp_pte =3D pte_next_swp_offset(swp_pte); + } /* * No need to invalidate here it will synchronize on * against the special swap migration pte. */ } =20 - folio_remove_rmap_pte(folio, subpage, vma); + folio_remove_rmap_ptes(folio, subpage, nr_pages, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); - folio_put(folio); + folio_put_refs(folio, nr_pages); + + /* + * If we batched the entire folio, there is nothing left to + * walk; stop right here. + */ + if (nr_pages =3D=3D folio_nr_pages(folio)) + goto walk_done; + continue; +walk_abort: + ret =3D false; +walk_done: + page_vma_mapped_walk_done(&pvmw); + break; } =20 mmu_notifier_invalidate_range_end(&range); --=20 2.43.0