From nobody Tue Sep 29 02:33:20 2026 Received: from SJ2PR03CU001.outbound.protection.outlook.com (mail-westusazon11012029.outbound.protection.outlook.com [52.101.43.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68D92384233 for ; Thu, 13 Aug 2026 04:24:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.43.29 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595058; cv=fail; b=IjRAl9BFVn6Tdt1tty79NT77IP43tnXOU4dpbXlIsKjJNexGEsnveQS/98pZlqeY0V3F38VWtB2Ho/e0gmhU6pa1f1lvm1eo0wZFxDxk5weNcATgXf5UmGyLigIJZJr4Bs6XWpCSykju+TnxwXw2zohptvdB+z6zyddrXLr5Toc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595058; c=relaxed/simple; bh=Cxu9ELvKmplpElna3JI88TGVrIhnPavPEhcP1F10lYE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=kunzE7jj0sZsqLSGc7Km1V6QfJiWwN0QWDQBt0bXT25rK/TfRK+vWbTq4MpPMOXCxrvXJnruK5A4w11cJfqUMJOhU/3g01//+s+loJ4uBl4qJatpuZwzvRuE9VYQ9I1KJ3vEHe4iQIyp21bt6RTLZsn/NOg0zhOa69K5fMfrm6I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=1pQxI1Pj; arc=fail smtp.client-ip=52.101.43.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="1pQxI1Pj" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NGfxcekwkJMyh3XK+LfkpSDpa4MmsBETgK1eo0DQj6bxqvoBQdyNyTFbIeCR+6peK6DBQLak50pA/wAG0y+Nhoa6gdSRmTTJ5rtMUP+mmOsV5IVwE0T98dGJI0od6vhgVs92Ux9Z1TLAe9boNknFuAfz2JY4EHz+OCEYMOiXl2GA05lwWrBtKttMB3gZsMZ0M1YqvlvhIwAhgNKGJmQqOP5+XHyzojVbHOvxvrOwAluLMQIbN/JpGbVX7jnmwMJ8EhAtWhqVSYxv7cAJn2eitNAMlKRSnduCAZpVHdcb0nH8jt/cON6lrrIgzraEgvpjxr/CgfHwdaauficQOVBZTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cVf5yo691MlmhIFP/C9Zo6yzsQASTr3Bvaov1srUftQ=; b=e5rJUxjY9rrWpISdKZg8qzDvjgHmubZdBwz48RHWueF7qBspDURnS9CHt2j7UlSwKxjRKJOV3QSpMZzt3+xnXbsWswIlJDuMIC9GWJQbpcKPzWMxeeWUVbeJ5alIMSIRar4W9eksk6XiSVxmscZmOSsBpBrDK7MDYxBNyycR8JCJ+CIblVGMCPsAhIXLJRzao3pXASjqjb0SLzf4KdAFxP+dg3sWtMR5I9hZL0XMs5uEtqLalgzm0pPi4eEk6fLJk82U6/m8WHFNittFD8E51GrS6gJB0GWXKjSRFpjHPJKBwoRI+t7X4BG0aj5RTuduAdSKfMAswhCbwOMtntBlwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=surriel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cVf5yo691MlmhIFP/C9Zo6yzsQASTr3Bvaov1srUftQ=; b=1pQxI1Pj2lKuORNLS4K+GswW4jMj81EG29w4+tf78U5RUqWUgFI/pN0z1tb+Ka7c/zZBc9RGKI2G6pkJJtcWjWDIlx2huvdmqoqSB89ssDu0rmESL/by0E895Wh91o4GcvaE+YD3ho5Snz+24s0lW2ITV7lBl7/+sSHqnAgvXEQ= Received: from BN8PR04CA0047.namprd04.prod.outlook.com (2603:10b6:408:d4::21) by CY5PR12MB6132.namprd12.prod.outlook.com (2603:10b6:930:24::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.11; Thu, 13 Aug 2026 04:24:11 +0000 Received: from BN3PEPF0000B374.namprd21.prod.outlook.com (2603:10b6:408:d4:cafe::a8) by BN8PR04CA0047.outlook.office365.com (2603:10b6:408:d4::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.315.12 via Frontend Transport; Thu, 13 Aug 2026 04:24:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN3PEPF0000B374.mail.protection.outlook.com (10.167.243.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.0 via Frontend Transport; Thu, 13 Aug 2026 04:24:11 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 12 Aug 2026 23:24:01 -0500 From: Shivank Garg Date: Thu, 13 Aug 2026 04:23:12 +0000 Subject: [PATCH v2 1/7] mm: factor out generic PTE batch detection from swap_pte_batch() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260813-migrate-rmap-batch-v2-1-3c5424c555c7@amd.com> References: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> In-Reply-To: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786595034; l=2122; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=Cxu9ELvKmplpElna3JI88TGVrIhnPavPEhcP1F10lYE=; b=hUlCPlvLE/GULh4fk/h3JpprlQXcCH/R+38RKBmsk8G0avdiTfmqgwLZd+ESs8NqhZRmLw2Vq oFZi8XA04ksC/qUBYK9IKsSAFUuiciZeOZLnx4zWuqVZx7ofXCMYlr2 X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B374:EE_|CY5PR12MB6132:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e95d8c0-6e58-4184-c813-08def8f2b9b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|36860700016|82310400026|1800799024|7416014|376014|10067099003|921020|56012099006|11063799006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: NnLZSNR02sdKmqwSle1M/Ha+MPd0/JUmmdYloEvycHUPIk7T4hdPlM4J1sxHDMYG7d3+7zx9xIDdMvwrDido7VwpPHxD6KBICvYgk61twwrc7aRSY337aEuoDddgBI0hg2JHFdyUlm4B5TRx/PGo+Q2hhtZ++oQBcFZcPNgD3XCkE6UKAu1A0Wr67T5fXjbLOTT8t0UABQ3oK6Zmf2Bh+o0GEztGybm5DxI2II39UcRJlTklk4iMsdMH0v7JsxbGQdPVeZdajdygJmn1h/AvwpxU+KtBg4dNESkpbtu0NmJDpQ+VpsUPK3uN6vyyvjSM+AbjH//KUmz27k9eg7fL0ELgUJJadmvhRL6GefdEXAmiGky/IKSnwZFLyssOerHg12959k5g2FYeCi+PRliaPTOd1SzNh3m+lhGKpaHuEeaRJBkrL20yeCp0FeBjH6GmJTpFuzOLuXJ1y1Hw0ihhhkXvaIqsuMOgjKCwDh2JmKkJ87CN5H/qJTpceeeA0MobiYAi9AVithJchrvajou8v1Jv+2lK458slwa4eAL63DV5eGHoc21Z2AtZFrCsV0Akhm1l9aktxMXjZuPAqfrl+RglokR5aAbRMxVVLEi/QU7N+bbnRegX61gNRsCJ8uga8NsqFRsA+oDQMVf0E8thQL/aE+5NrF/MwM3CwCb4oBCLmgLgkIW8VcE7rIDt/Fdr5uTkCiip1GSSMGn/uqGcUtvXonmfzhZ/kYAKuKQ4TzU= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(23010399003)(36860700016)(82310400026)(1800799024)(7416014)(376014)(10067099003)(921020)(56012099006)(11063799006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: h7BV1CY9wTu5ZpYpwVTcdZHW2lJAbAZBEG6mHHuZTXpUo3gvZD+3a6M5FwJmoAr3/bKs8BBPPYhbIwKBKowEf+VzuZCAcvxNoPifYZmZwTt9TY+TJTBmRCU044z51Sj8xTg4iusLQW7mnIjuliF+TA1wPoNj6aYmnoHaaULIXt9Do4RQ/X5nd0/Il9MCS2HU+ENZB9Yqyb4LtgisgUKzf5sqix2hKoCtVgVpGQdTTf4gi9yJsR6Ihr9G+c1AC8kBLgvl64HKvP6H3Hj224ghMo2XtHZ3o8zVVrc9ug9VSMGnr82e18dABeemTseQlGAmwzeNuebZXw3T7Y9H5wrfg2VinLnAUUbDU9T8I8ZR/J7w2gUmFog+1UVKqnQeWT0mxaMN8kWCIbGLWc5xzRNMVM3vrkXx9Hw8wuy65am/qogpyPezJtFe+3lhzASC1svH X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2026 04:24:11.4244 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1e95d8c0-6e58-4184-c813-08def8f2b9b5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B374.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6132 Factor out the generic PTE batch detection logic from swap_pte_batch() into softleaf_pte_batch() helper, so that it can be reused by upcoming patch that adds restore-side batching of migration entries. swap_pte_batch() now keeps only its swap-specific concerns and delegates the batch detection to softleaf_pte_batch(). No functional changes intended. Signed-off-by: Shivank Garg --- mm/internal.h | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index a5d0488a54dc..1e1ab53b642e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -560,6 +560,26 @@ static inline void set_softleaf_ptes(struct mm_struct = *mm, unsigned long addr, } } =20 +static inline int softleaf_pte_batch(pte_t *start_ptep, int max_nr, pte_t = pte) +{ + pte_t expected_pte =3D pte_next_softleaf_offset(pte); + const pte_t *end_ptep =3D start_ptep + max_nr; + pte_t *ptep =3D start_ptep + 1; + + VM_WARN_ON(max_nr < 1); + + while (ptep < end_ptep) { + pte =3D ptep_get(ptep); + + if (!pte_same(pte, expected_pte)) + break; + expected_pte =3D pte_next_softleaf_offset(expected_pte); + ptep++; + } + + return ptep - start_ptep; +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. @@ -577,23 +597,9 @@ static inline void set_softleaf_ptes(struct mm_struct = *mm, unsigned long addr, */ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) { - pte_t expected_pte =3D pte_next_softleaf_offset(pte); - const pte_t *end_ptep =3D start_ptep + max_nr; - pte_t *ptep =3D start_ptep + 1; - - VM_WARN_ON(max_nr < 1); VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); =20 - while (ptep < end_ptep) { - pte =3D ptep_get(ptep); - - if (!pte_same(pte, expected_pte)) - break; - expected_pte =3D pte_next_softleaf_offset(expected_pte); - ptep++; - } - - return ptep - start_ptep; + return softleaf_pte_batch(start_ptep, max_nr, pte); } #endif /* CONFIG_MMU */ =20 --=20 2.43.0 From nobody Tue Sep 29 02:33:20 2026 Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010069.outbound.protection.outlook.com [52.101.85.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7483C384233 for ; Thu, 13 Aug 2026 04:24:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.85.69 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595063; cv=fail; b=DNtU/EBY1kv95x9+J1AviiU9mgyj6ydyg27EeHDejzN9VuFWMlb+DORqjuQ5iBkDHia36/3vEsdBdrhIE+zulG8+XfEVuM2oSwpovT0wfu0kR6ggyAcbarpM/VrBfEpuHdZRxE68aBu2xpEYcQHW2/H+5ahH1dOwCMAk/KRn7lc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595063; c=relaxed/simple; bh=gneolqKRhOLi/xmG4jUme2u1klegVcoHCUCJUJlgL98=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=PNre317gk/w7CnW/i2RO9vgVYg16AWG033ubUO8Mi3BMZg31KXaJ1rngbHaddjPzm1GeTH+xMQthzziJOFFuSB+JT0IgE+jb+AyfabVuVrl1l6qLIQXde7Oo/nABFMtqMBTsbK9TLmz8g8U4s1NLNoP9JiE3odxvbrygx+1cBvI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=zAeAi6of; arc=fail smtp.client-ip=52.101.85.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="zAeAi6of" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EXVMW5qEnmUHDWdNP71mDyTFXdduGf1y2WaU3i49/eTtN1MCWyUZaVQRl77Qq0c8yreW/z1HwEo/PxswfXLGaHOyOzuqWsdAJOk/XWH7IiRq2H+Pn+HrQMU5jUKTRhkiE9bUVLVeSEUET/JcfKhJpeZGm3op4IbFqFsLy4jfWLAovIWW1yzyEVaHsZC507M2BA9viDNZRAMfUPD86cmcDjq8sXfxUVxg0Wd5OigscmHGaifTqMQBhzyX9XkJ0da2xUae1JP2xmU9eMIIFg9C3IgaG/EdD77lmNVkzw4ioNzRBIElGq3Caae4BvZl4FX8pmivDh6du5YvCMdaT59U5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GZx0Ge6ZDreH1DBnQvrfoLUwMhi3ucN8qoNiw2CdSxg=; b=PiNGL6U45+kGxSI1PPM1uwCAwsdqgrt/ys9QtjlOs3zyqtaacaP9r357U2ZhjrPr/YyHswYcV0ROH/bZMQf3buwmRmwhZrmybiD1OLEhLkb32BIv81irVh3/VNiX0PTKxJQGhG0maiKaF3K1CoK2xuiukt2UpDoI2arX3DU558t/nHI8sGEbaCz3U+qgnAHEkka3EqQ4hoD6AWECAENxDT+sMKVfZ9XjVm+By1knYq1aNAHHEDbbKFVrUMFZs8NOZek3UVYlV1qsqGUKjSDj9tzkM1p7pqAAYeUKhwIGeaS4pMvwZcL6S/+4itgBorRkIPQj19nPKisAYfQE/RcNKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=surriel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GZx0Ge6ZDreH1DBnQvrfoLUwMhi3ucN8qoNiw2CdSxg=; b=zAeAi6ofm9bq8LabrTWNBcHZFx6fDby7dMbhNbiOVXUEQlh7rDeKjtZ2zQ9wG5yFbRgqLu/TSTFNIANwEC53wXw70dl6pmj0LZ25AiwGZQ5R+d8IlkPTxX3N1JmEMJ5eLoE8fLHUougUNDzMsr0d8Q+Mjg1mWKe6gh9EkiJpqgA= Received: from BN1PR12CA0021.namprd12.prod.outlook.com (2603:10b6:408:e1::26) by BL4PR12MB9536.namprd12.prod.outlook.com (2603:10b6:208:590::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.13; Thu, 13 Aug 2026 04:24:16 +0000 Received: from BN3PEPF0000B36D.namprd21.prod.outlook.com (2603:10b6:408:e1:cafe::81) by BN1PR12CA0021.outlook.office365.com (2603:10b6:408:e1::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.315.14 via Frontend Transport; Thu, 13 Aug 2026 04:24:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN3PEPF0000B36D.mail.protection.outlook.com (10.167.243.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.0 via Frontend Transport; Thu, 13 Aug 2026 04:24:16 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 12 Aug 2026 23:24:09 -0500 From: Shivank Garg Date: Thu, 13 Aug 2026 04:23:13 +0000 Subject: [PATCH v2 2/7] mm/migrate: factor out migration PTE construction Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260813-migrate-rmap-batch-v2-2-3c5424c555c7@amd.com> References: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> In-Reply-To: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786595034; l=2695; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=gneolqKRhOLi/xmG4jUme2u1klegVcoHCUCJUJlgL98=; b=R/y+I6Jl1+vGPpMitwmnPsCU2+R1U3wNMIW68+2Q7iSi7GEBcH9vabuuNNMpVDVORIykNqwkI JnwADKqjCn5DC6d3Ad3Q2Q9x3RxlPLGg8af+7Y7g6H059bW4ZAYsyMv X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B36D:EE_|BL4PR12MB9536:EE_ X-MS-Office365-Filtering-Correlation-Id: 360e2414-83b1-4f69-970b-08def8f2bcd3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|36860700016|23010399003|82310400026|22082099003|18002099003|56012099006|11063799006|10067099003|921020; X-Microsoft-Antispam-Message-Info: 9I/kDeKKpO+oKUL6vDJ/plY2EEvQ1wjL8ZyUvRLWXCTEyPs7NYplcAc6YenelmOAR4TXt8gADRF2LeoABSYv5b50iuoehUQtH38uXjGCmQsngaGFxUjohBDfLXWpAYEXrlovJc/UjcAz8WC/XSEqxdr57hQdYiB/I+TiTEb6rwKpR7WJIFSvBdEdk5J7pKWQI1vcssKM5+Wis7xvlF7AyLw1DAcZoZybyBBFZnJXSSXqRIHJ9zo2dAxoM/b8lCOYbElwCNicujnRE/0sGh2xdvj3uaGv5f+H0kUZxrFlKmyTYFtzy5uSVoUJ2Jd33IH6R6dMn52NnXkNuWgHMnuho8rLCMoWkkTG0nRn2NZbgqfdLoLihF9HF0hMNeZ82JTs2K0kJRYRyoMtaScRxC/BZwNzJqe8/YZ22AXaL/3pvih8+OS4V4a1Lqy9KC9k2Ti6in8H6rogGn7/9bLuwzN0geJPQExaHnLNuBWj0eTMFyZV2tspAt5AS78KtYefgLGHAsEaLJBspmLZi6yjrShFHMUqOh9jcQcjuAwmLrR0nZ3Ywa2BhnHm6UmSMt4BjKg8yFeS2M84FZqayzGzCGGcD17rphZHePHwIpAKucEYoT/3hBdBeTGeV2l+YaHmdYy/QJwL6aTTkBqHDG9omBg0usHi07C1WK36BVH6Ag6jktwvBCvnmt5qZ56wslqoPT+8J8pIYlDH9z6R0hBVnKnmjtW9aBUOt7rvdWNlCabOOwogCiVC7tVTrMOAjRF3oKfy X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(36860700016)(23010399003)(82310400026)(22082099003)(18002099003)(56012099006)(11063799006)(10067099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: imv8K8hM9M7qcVsMX+yH5EmqjMPDbAgJCcbgzwGwq5/BNJO9xcXtzHcHrgoeNvTsHhb6WTvarzlri3fpuSpa0eADNLhifkq/nVRGXtPvTTwR08V4qyc1QQAg9tP8aHeA0I3Bp3YAuZa+fK2a5Fj/U69tCjYlZXg3+Ytd8dmgqW1e255/WTtf+ZE5EVOi91SnU8MBxoS5tZ7kpG+Cw9vbGv7/P1o10EsfB3VYGMY1U3dA+mSugtUo+BPpH+OR26KcuyHzbl+Q4CIfzv9YTJ4cM80wnyWclHbGof6ALTtXOVW/tr7r/34h5eiVdsV+XgA2Mmq5xUN0u5/4cahc/5Qdz2GW1levG/sjhNSXCvRDl/WePlEmjt4SEOqu9sR6DtCU/0OAr1odo4y7FSZ8qTMomnU9zfKWkPlzZHDuG4ZoantUyXhpHXGBtEeFAifB7mfO X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2026 04:24:16.6531 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 360e2414-83b1-4f69-970b-08def8f2bcd3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B36D.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL4PR12MB9536 remove_migration_pte() constructs a working-PTE from a migration entry inline. Factor it into migration_entry_to_pte() so the logic can be shared when the hugetlb restore path is separated. No functional change intended. Signed-off-by: Shivank Garg --- mm/migrate.c | 53 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 15b45832bcfa..a3362cc9ef66 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -345,6 +345,36 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; =20 +static pte_t migration_entry_to_pte(struct folio *folio, struct page *new, + softleaf_t entry, pte_t old_pte, struct vm_area_struct *vma, + rmap_t *rmap_flags) +{ + pte_t pte =3D mk_pte(new, READ_ONCE(vma->vm_page_prot)); + + if (!softleaf_is_migration_young(entry)) + pte =3D pte_mkold(pte); + if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry)) + pte =3D pte_mkdirty(pte); + if (pte_swp_soft_dirty(old_pte)) + pte =3D pte_mksoft_dirty(pte); + else + pte =3D pte_clear_soft_dirty(pte); + + if (softleaf_is_migration_write(entry)) + pte =3D pte_mkwrite(pte, vma); + else if (pte_swp_uffd(old_pte)) + pte =3D pte_mkuffd(pte); + + /* See do_swap_page(): restore PAGE_NONE for RWP */ + if (pte_swp_uffd(old_pte) && userfaultfd_rwp(vma)) + pte =3D pte_modify(pte, PAGE_NONE); + + if (folio_test_anon(folio) && !softleaf_is_migration_read(entry)) + *rmap_flags |=3D RMAP_EXCLUSIVE; + + return pte; +} + /* * Restore a potential migration pte to a working pte entry */ @@ -387,27 +417,8 @@ static bool remove_migration_pte(struct folio *folio, =20 folio_get(folio); new =3D folio_page(folio, idx); - pte =3D mk_pte(new, READ_ONCE(vma->vm_page_prot)); - if (!softleaf_is_migration_young(entry)) - pte =3D pte_mkold(pte); - if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry)) - pte =3D pte_mkdirty(pte); - if (pte_swp_soft_dirty(old_pte)) - pte =3D pte_mksoft_dirty(pte); - else - pte =3D pte_clear_soft_dirty(pte); - - if (softleaf_is_migration_write(entry)) - pte =3D pte_mkwrite(pte, vma); - else if (pte_swp_uffd(old_pte)) - pte =3D pte_mkuffd(pte); - - /* See do_swap_page(): restore PAGE_NONE for RWP */ - if (pte_swp_uffd(old_pte) && userfaultfd_rwp(vma)) - pte =3D pte_modify(pte, PAGE_NONE); - - if (folio_test_anon(folio) && !softleaf_is_migration_read(entry)) - rmap_flags |=3D RMAP_EXCLUSIVE; + pte =3D migration_entry_to_pte(folio, new, entry, old_pte, vma, + &rmap_flags); =20 if (unlikely(is_device_private_page(new))) { if (pte_write(pte)) --=20 2.43.0 From nobody Tue Sep 29 02:33:20 2026 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010039.outbound.protection.outlook.com [52.101.56.39]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67F3C359A68 for ; Thu, 13 Aug 2026 04:24:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.39 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595074; cv=fail; b=rW9PzKV+9EUecuHwL0J/DPNgRwIYm/ChYZmbAayXSyz6R73XpSXBchUs6WFHpOMUMD6k8jsgjNQqhh1hVuqPMolJrzGPMD8wHeJXBaCwcaXtjjZEGca+zBE84upfoJYe4zYvrwasQ3KPI6IMB7YRmLtmkOqv9yX0u3XvXNr/lpw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595074; c=relaxed/simple; bh=V2Csu+W5OddYNfw1PGZXsHW/zSePcDCED8EWGpg8Wb4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=rgHosycD7Ug99r7q09G+Gy2YQOMtg5vHimKwlN9EwHgQdeu+0c5LQo0OGKL7vpCu9BM1DIrONtGPdz2I9pxP06cs4Ce5aqXg/uhIMJAw86lLKi7AkBBdDW6llAwGRmmgrBvGs9JQ+NTdGKXCGjw1ibNZU1UU+sOEfxqS0rFdp6Y= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=NR70t14g; arc=fail smtp.client-ip=52.101.56.39 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="NR70t14g" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=sJzBA3W7lnGFclbanKQ3eo6SD+TOMdxlFqxJy91MQ2bFgAj2bg/o4aWA21N8A//+iTqOcIdQTibSpQB5hcrl/uJ41oxNQvoweO5DHBmZkp6UnYUGcNN8bidouU9hTj0YsqjN9ax2g3dGnB8lR/VCTZARLFqF6vh8aphq4xkw8B8IkCfN8TAwrcY2+Qp7A6AMY7eH/mf4DbAVozF1khAZ9o6chB9GMFhWf/nqVKXtJClmhW5/DHHIl1kcmCG0tscOMtD74CRuAgd+HqaoAddKwA4AkiV5NPmYLR9yLl3Bp6dy+7vtVwgQMroDIMaT7HugFzH5feSxI4UP/6gjZbaq5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BykK82YDo1LySzOhloAkWorz+g6kLKF76otmIGbgpbs=; b=lBXKDlA15AUx90wA0d4/OUxyw7t8jByRFrRWbymOwGmlPSsHCosacaww0mf0MeqjqLXcIJgJO65s9KeCwG42ZfLiOb15fBXSwsMAnrkK/lETrKOXMw+nq1AupTVgUZ0W+jFdjo13SMjfse4P3WeWBtZY9LUJ1cxgaNtfwHzZvgVpCRB2vGMX13m8eze6FK5th1AEVvXbXkqfYCmOaZcR23AmmlSPRwOwpsY8Wwsw00X53qC5+ptFjuGDqP9Uw+PiiNg70AIZv3TTucnF4luPh7hDHJktW3QO095SGKFapfJZKvVSEBSJsIbhk3VxMqGDU9keV69ihIyDi46SWv9Bng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=surriel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BykK82YDo1LySzOhloAkWorz+g6kLKF76otmIGbgpbs=; b=NR70t14g08Lo9/L1vkIlevr11JsQjE4YmpOKSNPYot92uhJBj2g8nEnSfyGHB7FLbMDH5ibhEw8N4x5wnLTmGDd0JYEA4XiWokKE9e1MLZQRoMDPDQvVTs7hS6xI3bdzvmU3JV1yzDDNji9PKsEexSSXcTcUBvZSTTm8bK4B1+g= Received: from BN1PR12CA0016.namprd12.prod.outlook.com (2603:10b6:408:e1::21) by DM4PR12MB7527.namprd12.prod.outlook.com (2603:10b6:8:111::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.13; Thu, 13 Aug 2026 04:24:24 +0000 Received: from BN3PEPF0000B36D.namprd21.prod.outlook.com (2603:10b6:408:e1:cafe::9b) by BN1PR12CA0016.outlook.office365.com (2603:10b6:408:e1::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.315.14 via Frontend Transport; Thu, 13 Aug 2026 04:24:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN3PEPF0000B36D.mail.protection.outlook.com (10.167.243.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.0 via Frontend Transport; Thu, 13 Aug 2026 04:24:24 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 12 Aug 2026 23:24:16 -0500 From: Shivank Garg Date: Thu, 13 Aug 2026 04:23:14 +0000 Subject: [PATCH v2 3/7] mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260813-migrate-rmap-batch-v2-3-3c5424c555c7@amd.com> References: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> In-Reply-To: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786595034; l=4926; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=V2Csu+W5OddYNfw1PGZXsHW/zSePcDCED8EWGpg8Wb4=; b=ob0iC1dS6KmO+0v44bGeBcevYeRg4ooQfD1WaJVqISFV/8ubExlf4yGL1/neik+nanQ1dT1Q9 bR4VUZBw9kFCEfLx9QKTECvkzXmsJnXhtt8U2xfjJmui4ivE+U7skDS X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B36D:EE_|DM4PR12MB7527:EE_ X-MS-Office365-Filtering-Correlation-Id: 3988a3a8-2bac-4873-f10a-08def8f2c14e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|23010399003|1800799024|36860700016|11063799006|56012099006|10067099003|921020|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: fbU5U0QAA+4w2dZN7t/E1zRw7X/DtUtM0gOZKgWZdKIbdtjdhAAnEZxoQzzvhhT4hJar8pJg1w5oAYxqJpkp5tRLrNWiQRiB196sycyVN2xTl0GHCoXRclsvlrw4qidls9xNneb8qfiOEn2oTYRf6JKecokSbXEUDJ3UKVP4k2TepZj52gy+tgLKeLwA6C1E0HbW956v/xG0fJKreA5gC3Ya0yEWRyhERKIXli/DdVl6ZyQqSuSzg6g0CCfG6GcdQFTzoLhfQWjlkW6bZ9+Hw1ksJxMOB31zyGO1Nhsb6uF5eAll1zsB3q6nID974yR8kyXwcQ2xemk+ulxrj6o3kswkhEoSHql+zgcPBhLfxIRgXmIMeCZ6j7oSqNb7oGRkbdiX90rFy/XA89I68ovbRwDVbT7ToZE0xxiJJ4D7UfXXyxDg6lvJ7MX2Irbo2fYytpNAKKQHNb5exsrcbuPnXDJUXWnhzS5KzCwejIzEP3vKi6jEJzdSCzIlk746qWTduuV1k6QE+hPhNmFMne/xaupX31z3tSGAb55pLw6qmBJP36XCrEIIqcyO+fJgpnFbXnRuNpqyatSFqtOxjBRifwwXf5briWiwPZezRkWTb281g029oBdu6sS/hwlOx/kqDukT1SDYbF6dREYOBgLNznw8rXDzaldEkfEfQtuYzYKLdhO4+M5osfOQt90j0H6BfqjTGjn6dj0vr7bQ6AFvafAuxWemqsGqJAGcIE2vQIjQyufcpAK0XkWlJN0XOg6l X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(23010399003)(1800799024)(36860700016)(11063799006)(56012099006)(10067099003)(921020)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PcubKfli0gMysUIPmHrNnMJUNdUmltzW6KCvgXGCmuPElXOZlLaEh1ov96CpO3EspfWwbCnv1eTgk34DxRlP0VvKJRbIjmtJJWhGfr6EzX9++ayO5a2WPba0ojSWIWqFHBCoC5XFCdUmT405AJpghjwGIpLdx617kzQzEbX8jO30FxmL44lXcOXZwtysqubNZ1fWpDBlV2YYw9749ozVVaUz7Oh2RnxyCQb3DBCbtZts1JzsIQkrNrbQ3C5eHmBepII794d3zEI4apsA8wbd6SujzYfdZWWRsof93s9b0D4/ckqN/KpQFQLgMVuNvVtqVcSyHs6zv+JRqAJKFnUTqemSk5rps07yotrqTKWFLN89T4zTVkOsYC4ZuQXsebq1nv0mRSHEywJcR9m2Z+dL35kvXcxx9R1Eh43fWiuwXr8wgKiN8K1wkteFmU8cl76a X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2026 04:24:24.1733 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3988a3a8-2bac-4873-f10a-08def8f2c14e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B36D.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB7527 remove_migration_pte() interleaves hugetlb handling with the regular folio path. Move it into a dedicated callback selected by remove_migration_ptes(), leaving the generic callback focused on regular folios ahead of PTE batching. With hugetlb folios routed separately, simplify the PMD mapping check to a one-time warning. Signed-off-by: Shivank Garg --- mm/migrate.c | 97 +++++++++++++++++++++++++++++++++++++++-----------------= ---- 1 file changed, 63 insertions(+), 34 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index a3362cc9ef66..ee1b8a55a2a4 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -375,6 +375,57 @@ static pte_t migration_entry_to_pte(struct folio *foli= o, struct page *new, return pte; } =20 +/* + * Restore a potential migration pte to a working pte entry for hugetlb fo= lios. + */ +#ifdef CONFIG_HUGETLB_PAGE +static bool remove_migration_pte_hugetlb(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, void *arg) +{ + struct rmap_walk_arg *rmap_walk_arg =3D arg; + DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | = PVMW_MIGRATION); + struct hstate *h =3D hstate_vma(vma); + unsigned int shift =3D huge_page_shift(h); + unsigned long psize =3D huge_page_size(h); + struct page *new =3D folio_page(folio, 0); + rmap_t rmap_flags =3D RMAP_NONE; + pte_t old_pte, pte; + softleaf_t entry; + + /* There is only a single mapping in a VMA. */ + if (!page_vma_mapped_walk(&pvmw)) + return true; + + old_pte =3D huge_ptep_get(vma->vm_mm, pvmw.address, pvmw.pte); + entry =3D softleaf_from_pte(old_pte); + folio_get(folio); + pte =3D migration_entry_to_pte(folio, new, entry, old_pte, vma, &rmap_fla= gs); + pte =3D arch_make_huge_pte(pte, shift, vma->vm_flags); + if (folio_test_anon(folio)) + hugetlb_add_anon_rmap(folio, vma, pvmw.address, rmap_flags); + else + hugetlb_add_file_rmap(folio); + set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, psize); + + if (READ_ONCE(vma->vm_flags) & VM_LOCKED) + mlock_drain_local(); + + trace_remove_migration_pte(pvmw.address, pte_val(pte), compound_order(new= )); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, pvmw.address, pvmw.pte); + page_vma_mapped_walk_done(&pvmw); + + return true; +} +#else +static bool remove_migration_pte_hugetlb(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, void *arg) +{ + return false; +} +#endif /* CONFIG_HUGETLB_PAGE */ + /* * Restore a potential migration pte to a working pte entry */ @@ -395,20 +446,14 @@ static bool remove_migration_pte(struct folio *folio, #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES /* PMD-mapped THP migration entry */ if (!pvmw.pte) { - VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || - !folio_test_pmd_mappable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_pmd_mappable(folio), folio); remove_migration_pmd(&pvmw, folio); continue; } #endif - if (folio_test_hugetlb(folio)) - old_pte =3D huge_ptep_get(vma->vm_mm, pvmw.address, - pvmw.pte); - else - old_pte =3D ptep_get(pvmw.pte); - + old_pte =3D ptep_get(pvmw.pte); entry =3D softleaf_from_pte(old_pte); - if (folio_test_large(folio) && !folio_test_hugetlb(folio)) + if (folio_test_large(folio)) idx =3D softleaf_to_pfn(entry) - pvmw.pfn; =20 if (rmap_walk_arg->map_unused_to_zeropage && @@ -434,30 +479,12 @@ static bool remove_migration_pte(struct folio *folio, pte =3D pte_swp_mkuffd(pte); } =20 -#ifdef CONFIG_HUGETLB_PAGE - if (folio_test_hugetlb(folio)) { - struct hstate *h =3D hstate_vma(vma); - unsigned int shift =3D huge_page_shift(h); - unsigned long psize =3D huge_page_size(h); - - pte =3D arch_make_huge_pte(pte, shift, vma->vm_flags); - if (folio_test_anon(folio)) - hugetlb_add_anon_rmap(folio, vma, pvmw.address, - rmap_flags); - else - hugetlb_add_file_rmap(folio); - set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, - psize); - } else -#endif - { - if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, new, vma, - pvmw.address, rmap_flags); - else - folio_add_file_rmap_pte(folio, new, vma); - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); - } + if (folio_test_anon(folio)) + folio_add_anon_rmap_pte(folio, new, vma, + pvmw.address, rmap_flags); + else + folio_add_file_rmap_pte(folio, new, vma); + set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); if (READ_ONCE(vma->vm_flags) & VM_LOCKED) mlock_drain_local(); =20 @@ -484,7 +511,9 @@ void remove_migration_ptes(struct folio *src, struct fo= lio *dst, }; =20 struct rmap_walk_control rwc =3D { - .rmap_one =3D remove_migration_pte, + .rmap_one =3D folio_test_hugetlb(src) ? + remove_migration_pte_hugetlb : + remove_migration_pte, .arg =3D &rmap_walk_arg, }; =20 --=20 2.43.0 From nobody Tue Sep 29 02:33:20 2026 Received: from PH0PR06CU001.outbound.protection.outlook.com (mail-westus3azon11011028.outbound.protection.outlook.com [40.107.208.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 763FA3803EF for ; Thu, 13 Aug 2026 04:24:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.208.28 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595083; cv=fail; b=QZh7TIVeFgn+Ps8puoUsSj03cx6bc0u3CA/azxRbADZ+pW1bNWcwYmhGGmSSVTGWMT3ylSDnuY7cLM4D7lJxW10FOIsOd/jQkbAifv/TzUjIe8mBQPCzDnETsBNEXBlLUrBDFfPJZVSRKDzpq5VKw6KhCegn4zb9EViEcd1e87g= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595083; c=relaxed/simple; bh=CaibqPtud+q20PIvSXd24MCW+xkI5GLqHzjtkAhhKHg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=KiC8w+W+14WjOSpoPtsFA/eQuCAsG1soeIS+sTPzTQ7opljLZ+QKLYwTEpcAR7mjHO5PWFbSRd5lqHb9/KJtV0Zz5OzVPyDtjlx8RNhu/6n4YZKL2e849/G1xOQNRHLOlfU9v2EP9ZjfBRkt77lFffrIz3aFzjhjhQKLrSuvbvg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=mgtyu5hi; arc=fail smtp.client-ip=40.107.208.28 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="mgtyu5hi" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=F0KdEyMkBZxy6uhQ5DvR3l6Hua7EjBzU0aUj1WZOymIkbjkshEkkClxrvhoefLKFAluKnMkNLpSaEB2QpDcb6w/1x3VClhnX2eQgXGzATOuaVg8XmJtRtXdhDyhkqc4Dg5xPzXnoaPlAwicVJ3/u5TKJuiz0u5n5ohO5+nNdlvGXtHSJDNwwXQ1dXNYOB/k4H5/qlLIEiVTQAtSU+P1gN7TS5F+4RzJSHRDD6Q/DBS8b52ZUvROhgZJn7f/GuqCUvW6fQDQ0XIYk5EMKFQTvZ5ruIvI0p2AnoR6yf6yK4WvpWKlOS/PyOgjFdLuHyBZrvGl7zbU0uxO6W/gCyewKxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FHFPS0BbpCM1EliPslOE50oMu9h6cy1rk2aOZUavTtQ=; b=Vv0nuf4vYGs3clmBCOjy2tJckdxCfmDmH1xalxjjlDy9gKU+OHkuoSNg08AJZ/xA3TuL5LDVo4ZC0Hx4AMel5yerPq7sQTFHwdiO2Mt9fB7lrhra/ZRCkwS2PUb0ZENopV2xcgXparabGwj2Ep1I6JycHr6A74W1YDPbnI99CjFko2QNlrm2+n0rvt6a+fd8SsVagFOS0qH8gBRE29K4MxJvHabqNd9gRIWnINO4Slz8hgjfNg/wn01cDuX7y8zXULZ7Xj3FZxzZgMFflSq3c/8RwLTPFKH8X5D2KZ8WgV6eQTQ+3aLyLhBRf6i7BNX8XOZiWHix4cV6+Cqz2Ciq9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=surriel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FHFPS0BbpCM1EliPslOE50oMu9h6cy1rk2aOZUavTtQ=; b=mgtyu5hidZ1uDPbzfJwhQSe9Za1pr6bU1S3+sm3pOkgDs2S9E3Vhzk9gs2AIdvJl6M3hugXX6qU1lwM9GlJDaQPlxOAj8LbDJ33wNgjkKwaR/dzQstvXuPOCtJ9LXuP6nX8KcVIw5wJE1gDx69aVP3ldh0TkBXtZzHKJs/6MXUI= Received: from BN1PR12CA0016.namprd12.prod.outlook.com (2603:10b6:408:e1::21) by SJ2PR12MB8955.namprd12.prod.outlook.com (2603:10b6:a03:542::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.12; Thu, 13 Aug 2026 04:24:36 +0000 Received: from BN3PEPF0000B36D.namprd21.prod.outlook.com (2603:10b6:408:e1:cafe::52) by BN1PR12CA0016.outlook.office365.com (2603:10b6:408:e1::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.315.14 via Frontend Transport; Thu, 13 Aug 2026 04:24:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN3PEPF0000B36D.mail.protection.outlook.com (10.167.243.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.0 via Frontend Transport; Thu, 13 Aug 2026 04:24:36 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 12 Aug 2026 23:24:23 -0500 From: Shivank Garg Date: Thu, 13 Aug 2026 04:23:15 +0000 Subject: [PATCH v2 4/7] mm/migrate: batch the restore-side migration rmap walk Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260813-migrate-rmap-batch-v2-4-3c5424c555c7@amd.com> References: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> In-Reply-To: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786595034; l=4209; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=CaibqPtud+q20PIvSXd24MCW+xkI5GLqHzjtkAhhKHg=; b=HSLbcAu245mLxUSh34KgnA1wKwGyRR3aBV9cZ5M+p5haVS/zrnL9dnmwLzOkKLtaU2smzjb7f u0H2vnxKDaTDgfHkbrA3AIgxjprJ9ujPCAievc0VbYTq6mKBPU52Nby X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B36D:EE_|SJ2PR12MB8955:EE_ X-MS-Office365-Filtering-Correlation-Id: e32a0393-74c6-47df-24f2-08def8f2c880 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|23010399003|36860700016|22082099003|18002099003|56012099006|11063799006|10067099003|921020; X-Microsoft-Antispam-Message-Info: 2AfzihJeHq1y4ktrgLMe33BZXFdudprg/SrKL6z2G+vKJVWMBMdmqnZ2a3E1BuYKhUjEL2VcwV+bDpY7OyAvmu9DKymlVKV+pQkSZM6cyZS3UM5f+r+IWsc0FwfNxUrAOABMRVmcz+WJaDbARNcPpyEGrrzEoEkit6vcEQMFPKsJpbRQjf5kQtE1qs789BqqIrcVSphTzocgSKu/NMzzucEQPknoDmmhhoFrkaNuMBXRjou3odbomnkAGvC8bS1K+2Pbw4bazZqqCUnUNxoHs/NnENE9qKndF4JR1h8eDwqDyBx06JIpgh2zxpTap/d+i/7BUdauAJGUgLb/imQ4OLpX+KaoW5u/M7hToTJwEJEKB+cSJTb+qdIMOzYUkpbBod778jl+C+0NrsE8zUcwZ1S+p/SaapRn59pOEWeOWNP9nIHJLrlbfsqPFU1KAcj5XHplmKnfbC+x34ffxdLvGivCOuNoqQpFt2ua4aUOlE52SeZWosQPyTwH/3NpnpfOsdym7G4k+GeKfIWPyGNPjq+gs6suLDaDr6BLLL+mbFSfq9OhTDcBwLvOyz5fgypFKKaxl61QpJ19GuL4AEOxtkEyqpxMXs+Z0bpTouqF2fD2IgrW2Fx1Scoy9xtf/94wRZke8mYJBgB5Xc4Y/JWUCrCFpOrGYt6xS1MuzbvJvJCABitGbepUiGuhMgPkxLVYN1GYqWa4VJ3WcONxEu6xn4Jgg9Ps77MobiohUkEsKdSW+QqjuCY+/ZBodVqwnPG5 X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(23010399003)(36860700016)(22082099003)(18002099003)(56012099006)(11063799006)(10067099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Vt/q5OQhKgiG88HF4JziofN1L0Zh7YHs7xq7iW6Aq8c8p7ZVnJ95gtnE5BxXlQywJunk5vKw8XZI4AQL3R3xKoqHfPA3jmDr8day+AifFq3RZng61VAIyEzD2t/NEOqAWrEeJQdQcQWGpXk5AYh0K7imlACm6kJuSUnQLpACTv13Gusc6jshy/Aye/8MQ0Ya8PA4aCZBQIc01i3ObISaqXsQG86KbMX26jHBVevuPIrHUGxnV1VZ41DvOaI9c3R1PAgLrurc5eW1QnGxeFNYEHQHU3N5j/+o/+XTX+r1egS6Bcpf6WDLD5YfPdqw2nvwCHxiufRYUishRMbvbc6ER6hMpyd+8UNVzmtvB2XVzFgAF31+wkqe0bXLA3h04anBKQ6RC2XgiOSFyu7aKnu/DTv1SBcRaJ1QO3Fc36WC1H5AV8gZusArMRA4hivFpnDP X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2026 04:24:36.2386 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e32a0393-74c6-47df-24f2-08def8f2c880 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B36D.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8955 remove_migration_pte() restores migration entries one PTE at a time. For a PTE-mapped large folio, it repeats calls to building the PTE, add rmap, set_pte_at() and page_vma_mapped_walk() per base page (256 times for a 1M folio). Add migration_pte_batch() to detect batch of contiguous migration entries that map consecutive subpages, and have identical PTE bits. While at it, use folio_is_device_private() instead of is_device_private_page(). Signed-off-by: Shivank Garg --- mm/migrate.c | 49 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 7 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index ee1b8a55a2a4..aad555453469 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -345,6 +345,28 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; =20 +/* + * Detect a batch of contiguous migration entries: consecutive (non-presen= t) PTEs + * containing migration entries with consecutive offsets and matching pte = bits. + */ +static unsigned int migration_pte_batch(struct page_vma_mapped_walk *pvmw, + struct folio *folio, pte_t first_pte, unsigned long start_idx) +{ + struct vm_area_struct *vma =3D pvmw->vma; + unsigned long end_addr =3D pmd_addr_end(pvmw->address, vma->vm_end); + unsigned int folio_nr =3D folio_nr_pages(folio); + unsigned int max_nr; + + VM_WARN_ON(!softleaf_is_migration(softleaf_from_pte(first_pte))); + + /* Bound by VMA / PMD end and by the remaining subpages of the folio. */ + max_nr =3D min((end_addr - pvmw->address) >> PAGE_SHIFT, folio_nr - start= _idx); + if (max_nr <=3D 1) + return 1; + + return softleaf_pte_batch(pvmw->pte, max_nr, first_pte); +} + static pte_t migration_entry_to_pte(struct folio *folio, struct page *new, softleaf_t entry, pte_t old_pte, struct vm_area_struct *vma, rmap_t *rmap_flags) @@ -433,11 +455,13 @@ static bool remove_migration_pte(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, void *arg) { struct rmap_walk_arg *rmap_walk_arg =3D arg; + bool is_devpriv =3D folio_is_device_private(folio); DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | = PVMW_MIGRATION); =20 while (page_vma_mapped_walk(&pvmw)) { rmap_t rmap_flags =3D RMAP_NONE; unsigned long idx =3D 0; + unsigned int nr =3D 1; softleaf_t entry; struct page *new; pte_t old_pte; @@ -460,12 +484,19 @@ static bool remove_migration_pte(struct folio *folio, try_to_map_unused_to_zeropage(&pvmw, folio, old_pte, idx)) continue; =20 - folio_get(folio); + /* + * Try to restore nr>1 contiguous PTEs in one shot. Falls back + * to original per-PTE path (nr=3D1) if batching is not possible. + */ + if (!rmap_walk_arg->map_unused_to_zeropage && likely(!is_devpriv)) + nr =3D migration_pte_batch(&pvmw, folio, old_pte, idx); + + folio_ref_add(folio, nr); new =3D folio_page(folio, idx); pte =3D migration_entry_to_pte(folio, new, entry, old_pte, vma, &rmap_flags); =20 - if (unlikely(is_device_private_page(new))) { + if (unlikely(is_devpriv)) { if (pte_write(pte)) entry =3D make_writable_device_private_entry( page_to_pfn(new)); @@ -480,11 +511,11 @@ static bool remove_migration_pte(struct folio *folio, } =20 if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, new, vma, - pvmw.address, rmap_flags); + folio_add_anon_rmap_ptes(folio, new, nr, vma, + pvmw.address, rmap_flags); else - folio_add_file_rmap_pte(folio, new, vma); - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); + folio_add_file_rmap_ptes(folio, new, nr, vma); + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); if (READ_ONCE(vma->vm_flags) & VM_LOCKED) mlock_drain_local(); =20 @@ -492,7 +523,11 @@ static bool remove_migration_pte(struct folio *folio, compound_order(new)); =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, pvmw.address, pvmw.pte); + update_mmu_cache_range(NULL, vma, pvmw.address, pvmw.pte, nr); + + /* Skip the batched PTEs */ + pvmw.pte +=3D nr - 1; + pvmw.address +=3D (nr - 1) * PAGE_SIZE; } =20 return true; --=20 2.43.0 From nobody Tue Sep 29 02:33:20 2026 Received: from BL2PR02CU003.outbound.protection.outlook.com (mail-eastusazon11011026.outbound.protection.outlook.com [52.101.52.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64D1F384233 for ; Thu, 13 Aug 2026 04:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.52.26 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595086; cv=fail; b=Sj/mguBkQtbqY5rx6O0ftuAeeiy8EnXlYtQFAdxVup/4WJJNTrR1y9WUAwcinKNzsiCgcRmnWSjHbp1EKfOWx8tPwzmWx6nzhe+QQQYZLhNk89bOaDfLiR6sYL4vLQQQTWZ9LCXM4pdf4SikNseAhldlgnjKVeAlCzO2UdYC2Ek= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595086; c=relaxed/simple; bh=XPr8YUsucdGRr57yCFjtBHp/S9a5uQAWQYAef73VHRg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=mOz3tIZOXGYwm40TLCrZUYnPcjvdrZ5whaAQeYKzKVyVbomvxZHRzU24t+sekNCB8/gTCSVUkwOCGdLHne4aKAgoNnSmONgWLQAZBJ8Ul38DgqQViYGozbhXBVxYUfRfgrvgDFFH3FsmWYiyvJBQAow6xZFIYqi1Snupuxr+B4A= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=mnqDn7Q0; arc=fail smtp.client-ip=52.101.52.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="mnqDn7Q0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=moEOJ8v6aUgy+egCipR0DnZFrF6C8mz+yj/IFPim2Eie65lyhK23yQdQHtVMZNfPWsPpfEZRkh+3KAMzPY5x9W+GoGcwgMxTGrhMJa23NHb2hP92h1xPYUJvCjsiun0UYCa2vzNs8fgE3mNvAgLz8V68JPEXoF7Ag1P2YTxmoyjgdxATKQyqKTcbHNnAbAVmKUJ+FK3afZTlVf6zaTlZ3UIAqMrOzlRPTY0hALigcEmdjA5+YGWdLoqgk579sloC2W0Xo1QKOgnWaLsLBkJNOSxI0gZbaS74uF7rgzXj0fkobs0vKeZ56WL4+51OTZtcN4PBaGB7z/XyXTGo+ikQcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZRqRWWSb5j6yTTmBTX55vjP5pv3HsdIGpRfv7y0RJsg=; b=bVqoH3GHKg+6UFN6P1O8xlKnd58jMuqyOMF2U9usiQkboyqctO9+4weaBWWlJPofTaMFGJqLfTzbdT99jZFzPo9DOX6PewHJ6SG9HAaZz9b8MXatHy1xAlojLz2acecTgAJ7fauwfiXeyfnGKRW5qSru3NM7Fh/2zI3nbcYo7tFAMvr6NcTqlkwVpk88ZQDLvoSq6Td6u5nzJn7kQT0mBiyUylV9TY7jceatBbbXiClnkKPl2neSD/vnEqC406bJxAQbDF7Mi1wsZYUNLCBXYOWnkIPD6K9CkOIu7QHr0+UC41oSfpDzwh8hnnHPuKb43fCC/5vtsfzIfcM1/sTUuA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=surriel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZRqRWWSb5j6yTTmBTX55vjP5pv3HsdIGpRfv7y0RJsg=; b=mnqDn7Q0sylZgz05kW2C3N9JljzhEuFQQRpapsGlGkz208IRVlTVlYj83P5i5/c4ZQLQ6HiyntlP5yF6KkP9OS+e+WlkG/F2goy4hvH3tJHPm8dj665jGIdWjrrLVM0FSdbWQIo2uq36Xc/xbSUoFSKLrDeJDvYhvY+a0Mxmzcg= Received: from BN1PR12CA0024.namprd12.prod.outlook.com (2603:10b6:408:e1::29) by DS0PR12MB8070.namprd12.prod.outlook.com (2603:10b6:8:dc::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.14; Thu, 13 Aug 2026 04:24:39 +0000 Received: from BN3PEPF0000B36D.namprd21.prod.outlook.com (2603:10b6:408:e1:cafe::76) by BN1PR12CA0024.outlook.office365.com (2603:10b6:408:e1::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.315.13 via Frontend Transport; Thu, 13 Aug 2026 04:24:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN3PEPF0000B36D.mail.protection.outlook.com (10.167.243.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.0 via Frontend Transport; Thu, 13 Aug 2026 04:24:39 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 12 Aug 2026 23:24:31 -0500 From: Shivank Garg Date: Thu, 13 Aug 2026 04:23:16 +0000 Subject: [PATCH v2 5/7] mm/rmap: factor out migration PTE construction Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260813-migrate-rmap-batch-v2-5-3c5424c555c7@amd.com> References: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> In-Reply-To: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786595034; l=3397; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=XPr8YUsucdGRr57yCFjtBHp/S9a5uQAWQYAef73VHRg=; b=q81NvIpe7hWML/CjPNB/HNBJcafRJrFxDfh1/7ESLRM1OrkP01YOlvJcanhOI0SgCeFJaQ5zG wV37t8RWhQWByM9dW4Z9BKaU3ke+DXV0b8J8CI9CcdVK27+aUmRnW7Y X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B36D:EE_|DS0PR12MB8070:EE_ X-MS-Office365-Filtering-Correlation-Id: f70c63a4-5ef5-434e-5573-08def8f2ca6a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|7416014|36860700016|82310400026|1800799024|921020|56012099006|10067099003|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: rdubjEIDasCMRV/3tXRrH68mMi4Y+oJgKnPOVInXGXmJm0QajC/P/EgogpeI+GrHZDwfwpVBXzdLAvkzefcgNHRE9Y8/I0+8wTvYLq6JarYafJeo89GUsoV7vXT1QLBeDJ8vTaeGH2A+XjY3G7ONKzzQVQ1RevEPeC0QruqXIJq1Wp9iaEOeGKlst+k/b0IG/aWNlq3YWpbwdD7erUxOhYm37QOt0fk/607V5vT6HkWk9YATnXL97glzvoP1eve7k6ZCTEOsG3qFyeP51j/k23bfzhOlGltYxuPQNMqq8YB7mv9jxDC5UGEsGWZ3ugV3szDcOy2kp0GC7hvaIFVlwtcbxVaeESgCQjyYC8JX5pc+vWrkXENxD6TY8qhRBwUeUlqOO7ephrsCz9NfK86E344h9GQ/dLQdQoclSM79hJ0SDjBvbiElptfqSPV7YBOPthqdZSR5kA72rZPOI8BB4B3U3PVz9bOCiRh57yePEooBj2E84EUOM3jYpVUWSMasIqTvos8mxZs0L82dSGgULLBz5YJ0mVdZPcUm+BKuXFwC142uEcGmSKJeRYkz0Nt3Qwg7WLnLbsIYEgdn+z3xt0bPiAj+w9Z6QbsBtV6tGnWOuxx6yuwr4b3kNeA7q4rbjiQ5bCUcCW+zBvMUl6JKq1FvkUh04duWHwAqAYsKuH69RHV5SIBZ0L6ddUN6WvmubUhb82rB3h+MWVAsNzqUPL/RFfo0aUB/+xpiTZOMw6M= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(23010399003)(376014)(7416014)(36860700016)(82310400026)(1800799024)(921020)(56012099006)(10067099003)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UAFB79WREz4yPOCC8bYcuNHTIq88dnvbNA6kU4z+lvARH3BeytrQWIgGmSyHfRGi52I2KvRLNMB6/L+v9uq6b081lvCAKe1i/nVtrkm+zHboazck8A1Q1JNZPh7JV7XOQX6tvHC4nOSZnBkrlKn/qYWxmQFRf2UWJ6VE7Zn6AIjcWZS7j2X/meXuOAZ3cxzECRFRNW7vix0DN03oiU+sT/SKj66A5WoHSFiPQJDcpNA3fshbNdPMCeAF20uzTUs92mtjQsk/pYsESoXMXC7HdDa0FwsaC2dd3sqI+MSLJem7UYiHAnFqwUHc8pLo6yQQYpVOyJgtsxQSXqq+GY9V65GcEZ96JE0UGW7+MEYbG2ntfva7xYF2+REFwcXuqarQWK9In8VdEydHqmc62crLt7ZL3OVv6DWNHiltwFeCjmeo3q1EQ8YoorwVqPtpQLvj X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2026 04:24:39.4515 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f70c63a4-5ef5-434e-5573-08def8f2ca6a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B36D.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8070 try_to_migrate_one() constructs the migration PTE that replaces a present PTE or non-present swap PTE. Factor it into make_migration_pte() which can be shared when the hugetlb migration path is separated. No functional change intended. Signed-off-by: Shivank Garg --- mm/rmap.c | 65 +++++++++++++++++++++++++++++++++++++----------------------= ---- 1 file changed, 38 insertions(+), 27 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 0a4a0945dc65..8b89e32ae489 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2507,6 +2507,42 @@ void try_to_unmap(struct folio *folio, enum ttu_flag= s flags) rmap_walk(folio, &rwc); } =20 +/* Build the migration PTE that replaces pteval for a page. */ +static pte_t make_migration_pte(struct page *page, pte_t pteval, + bool writable, bool anon_exclusive) +{ + swp_entry_t entry; + pte_t swp_pte; + + if (writable) + entry =3D make_writable_migration_entry(page_to_pfn(page)); + else if (anon_exclusive) + entry =3D make_readable_exclusive_migration_entry( + page_to_pfn(page)); + else + entry =3D make_readable_migration_entry(page_to_pfn(page)); + + if (likely(pte_present(pteval))) { + if (pte_young(pteval)) + entry =3D make_migration_entry_young(entry); + if (pte_dirty(pteval)) + entry =3D make_migration_entry_dirty(entry); + swp_pte =3D swp_entry_to_pte(entry); + if (pte_soft_dirty(pteval)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_uffd(pteval)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } else { + swp_pte =3D swp_entry_to_pte(entry); + if (pte_swp_soft_dirty(pteval)) + swp_pte =3D pte_swp_mksoft_dirty(swp_pte); + if (pte_swp_uffd(pteval)) + swp_pte =3D pte_swp_mkuffd(swp_pte); + } + + return swp_pte; +} + /* * @arg: enum ttu_flags will be passed to this argument. * @@ -2737,7 +2773,6 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, */ dec_mm_counter(mm, mm_counter(folio)); } else { - swp_entry_t entry; pte_t swp_pte; =20 /* @@ -2779,32 +2814,8 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, * pte. do_swap_page() will wait until the migration * pte is removed and then restart fault handling. */ - if (writable) - entry =3D make_writable_migration_entry( - page_to_pfn(subpage)); - else if (anon_exclusive) - entry =3D make_readable_exclusive_migration_entry( - page_to_pfn(subpage)); - else - entry =3D make_readable_migration_entry( - page_to_pfn(subpage)); - if (likely(pte_present(pteval))) { - if (pte_young(pteval)) - entry =3D make_migration_entry_young(entry); - if (pte_dirty(pteval)) - entry =3D make_migration_entry_dirty(entry); - swp_pte =3D swp_entry_to_pte(entry); - if (pte_soft_dirty(pteval)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_uffd(pteval)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } else { - swp_pte =3D swp_entry_to_pte(entry); - if (pte_swp_soft_dirty(pteval)) - swp_pte =3D pte_swp_mksoft_dirty(swp_pte); - if (pte_swp_uffd(pteval)) - swp_pte =3D pte_swp_mkuffd(swp_pte); - } + swp_pte =3D make_migration_pte(subpage, pteval, + writable, anon_exclusive); if (folio_test_hugetlb(folio)) set_huge_pte_at(mm, address, pvmw.pte, swp_pte, hsz); --=20 2.43.0 From nobody Tue Sep 29 02:33:20 2026 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011011.outbound.protection.outlook.com [52.101.62.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81010345CDA for ; Thu, 13 Aug 2026 04:24:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.11 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595096; cv=fail; b=uVtvVglSmVeAnFA1VubTC9bducF9aLMTCyrbizDO4XZ6Vq8snWXeaRX34aBySj+/zsK0wg/LVxsj3L/5jHwQz7YXxxXeLhMby0kXKeDb1prk+fnWRjO65169YQG5HiOk7SX/gkXDW6vDgFCVYUF/wXlSvDV2f+R/FLKI1HxV9W0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595096; c=relaxed/simple; bh=m9fxe8wNgOIOvVrgEOQTh0iQGBrzrGsGDvtG1c3clfU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=q9MBqKB/0ohBof+fFs4iZ82w9TMYZ4vycZoSjsde2K355Epa3MP98hDUC0n0MZZrNsFclEOPW1CouI2/9BPSpNZLqlKHYhpPDi6RXzGWFz8Xeo17DYeefpciqSDgEdI9wM8Nf/gSxOiW+c/GXElg5bslBPryH416hypdQjruWWs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=K/QNjs+E; arc=fail smtp.client-ip=52.101.62.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="K/QNjs+E" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=X7tqCpyH3x0F5Mfk7FKP0m2E/1A1LkcBualBJrUXBPgjizJYkvxytl65OYEU8Ug8zEAMdsZqFycmUpaED10WEtjVLAqR0yabR6xhdoclXAQjKS4lEFv9r/CFaAMLvKammqghPFLW+ia6I3u4fLnlm/N7RDkTQdu/8vWDj29dR2ZW8423Pifu5bVsEEOCC3wvKr81YwhtL93fKmnn63BAc915Qnf62JHaIb2l6ACq/adqt2IN7ap/Qp+2O0MnystYQI0/tagZPrlM0rA4J2C9P6tUAXrLOr7Dl0aJElwhWXtzYHoKowvWcxlvk8QyBVnpBduZgGHBJ3ANBSiQcYfneA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cLhtG+vvZYDJbT9CsAbPScGVywpiDLa6N6jkmr1n5ro=; b=hC7LtduiFsoxkyj/R/snstraATyi0ORWS5BiybUhexHBMaNLC6s79X06OxRkGoAVGpc2gHm8PNqtom6B8hOo1MIYf20n8fMga7uceLGz8c+B2aMtuE8CcQwspkb/TXEtDMpms9P75E9iwrO8IVpMctjA5zzj+EijJzKyKXQykB4lWEHoU9QYR8ROlyedDs9S9H9xwreDBkLbKt8hTVVT2bNk9HBkfROLpDRS5KgyrwxiQyX6dRRXDm88xCuHH9Qu6gvxQwPcZSFX+rMdkXZCyn8uGoeZ1vX7Cngc5JtOTLgfTgf8/imj/w/Ll6UpeiSmclS/0iXCkeHJCwoK/Nh5pQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=surriel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cLhtG+vvZYDJbT9CsAbPScGVywpiDLa6N6jkmr1n5ro=; b=K/QNjs+EQf429aCH00cYTALsEPDkVnwA+dNAkDQIKiOiD44Y+T87ox7yCQ2UIkOi9b1N7umkBAlvaEn2/3HnPqO3uzH9gWbMm4Tk2lU0VG9yS2EVwziFm968/ESAhd0G0tc3IPDHw5acAlNBW3Nt0maE4TW3ZO9TWM3w9HyCRPw= Received: from BLAPR03CA0027.namprd03.prod.outlook.com (2603:10b6:208:32b::32) by DM4PR12MB5793.namprd12.prod.outlook.com (2603:10b6:8:60::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.292.25; Thu, 13 Aug 2026 04:24:46 +0000 Received: from BN3PEPF0000B36F.namprd21.prod.outlook.com (2603:10b6:208:32b:cafe::3a) by BLAPR03CA0027.outlook.office365.com (2603:10b6:208:32b::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.315.13 via Frontend Transport; Thu, 13 Aug 2026 04:24:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN3PEPF0000B36F.mail.protection.outlook.com (10.167.243.166) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.0 via Frontend Transport; Thu, 13 Aug 2026 04:24:46 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 12 Aug 2026 23:24:38 -0500 From: Shivank Garg Date: Thu, 13 Aug 2026 04:23:17 +0000 Subject: [PATCH v2 6/7] mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260813-migrate-rmap-batch-v2-6-3c5424c555c7@amd.com> References: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> In-Reply-To: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786595034; l=12600; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=m9fxe8wNgOIOvVrgEOQTh0iQGBrzrGsGDvtG1c3clfU=; b=p3GireR7fAOwvnRrM4DGa/qYHpMuykBdEo/ivlYvvzohzomDceknvYqKSUj8yKBwOz0DgoaA/ P2nI34rQEtNCHPta3YEkDQFKUWeb4PyoYVJdvdDibwtWzCWyxILZeiG X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B36F:EE_|DM4PR12MB5793:EE_ X-MS-Office365-Filtering-Correlation-Id: 5e72defc-749b-4998-35c4-08def8f2ce52 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|23010399003|36860700016|376014|1800799024|82310400026|921020|10067099003|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: T+HwzmffNXQ+ganm2tF3sjOUVSe02OW3tR1lcqU9rSLBdpXp66ysY09qfIk89LlYYRo6haWmTeCKgAdLC+jFzaAmkLNScTxoCB0hrJrXxzDHw/zj9xL/cwyAGWIvAgjXfFM9DTKGsqbOORjfJTvj7nOn4VTr4zYURbQklLKbiOxjLFlKJWngKShyKoKUugUemtSXhYui/4n5hhc3r1Zv+xWcdK9xOleu3yxK8Tfy3CUjaPULdA7T+AlKuC/huGgQVcflPqk9izeDCdOhEe3idBdOn/qeUCvJEbbpoKYqxx4bwDBBjEanIw5LqmylyZaPhsE21r7tKrIezqUnRHSweFt+eiIokLvRv4ZDsrslSMflT/cJQV+NTpqUyjxWTg9MoNdt/N8rlE3H/SEfQYLQdTQpJLeSdTE5uKUllepshEVdIo602VZMwwu1GKaJ0S/prS3Ieh3XQ0T0Y/UfNu7k2txl95ryD3Wylkk9dSnaVNgXZ8HKSVN1F2urfNehFa0rsySGX/gH23mmgMLR/nDWaPBzRuQC7Ws5MQHOGP1tRgdHBR/Fg2zeFZx7xzYCq+2UBcY1UBe0a2rLpaqTZF+IDxq/Qauj08XWeskg9UI6YdE4XwGa5Q2nT5xE7dHMGMmrF28Y3bVdHuxeVoctmRbKbAYX6Q38T85Hr//EsiNEjqRF+LlbI7fMWkEvb1b8A69zwrpmzGosLYVBHSOCvThijdj7pzCTtdMoP8UuOLRpdvqeFsOXAUMzWYAWQ+1WvI/q X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(23010399003)(36860700016)(376014)(1800799024)(82310400026)(921020)(10067099003)(56012099006)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4KZ1dK74jyOyg7+ZCk90PHmDXdGvH2vTfhX/4EqTBJ34/Fvil9mWBbL4oXOXbjBkF079bNn/0eFqJN2Ry3BmA06pi/8mDUm19w6AekkABTElL3irL/PUG/hfHwg0OAoRQKnsxUqlrLXU/n1EYXfbeswcnTzq258QGBGJZk8ec28rLSbPr6wd3v7j/apCkap68UgJy4uQm78R6l3DC0u23Jh7bBbGhz1ubfLGK2UN8NlbdEAVJouxWtyJ9cr2DzOURcEGLUewDjctDlrwKIUKoDQ+1Z+BTV8wnL4wSaW9fe/ETHVSEvmyBSAWS+mQROWKdJ7ox4i0CWXbyB9ZY1/fNeDhR+f79frav0t+9xwV4RZvTv8+kQECaCii15uIuHXI9ICgst2QBUA3Gj8BNsNj7yzrVGkGqdcsiAkWA3W0R31YMHakYZL91N1XL/77UZDw X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2026 04:24:46.0031 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5e72defc-749b-4998-35c4-08def8f2ce52 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B36F.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5793 try_to_migrate_one() interleaves hugetlb special handling through the regular PTE/PMD migration path with folio_test_hugetlb() checks scattered across the walk. Simplify this by moving hugetlb handling into its own rmap_one callback, try_to_migrate_hugetlb_one(). Few minor changes: - Since hugetlb uses separate counters, update_hiwater_rss() is not needed for hugetlb path. Also, use huge_pte_dirty instead of pte_dirty - No PVMW_SYNC flag is needed for hugetlb - hugetlb folio has single mapping per VMA, so no loop is needed. - Convert VM_BUG_ON to VM_WARN_ON_ONCE and not use irrelevant WARN check. Signed-off-by: Shivank Garg --- mm/rmap.c | 251 +++++++++++++++++++++++++++++++++++-----------------------= ---- 1 file changed, 144 insertions(+), 107 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 8b89e32ae489..35752a70f3a0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2543,6 +2543,138 @@ static pte_t make_migration_pte(struct page *page, = pte_t pteval, return swp_pte; } =20 +/* + * Replace the hugetlb page table mapping for folio with migration entries. + * + * @arg: enum ttu_flags will be passed to this argument. + */ +static bool try_to_migrate_hugetlb_one(struct folio *folio, + struct vm_area_struct *vma, unsigned long address, void *arg) +{ + const unsigned long hsz =3D huge_page_size(hstate_vma(vma)); + DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + enum ttu_flags flags =3D (enum ttu_flags)(long)arg; + bool anon_exclusive, writable, ret =3D true; + bool anon =3D folio_test_anon(folio); + struct mm_struct *mm =3D vma->vm_mm; + struct mmu_notifier_range range; + unsigned long pfn; + struct page *page; + pte_t pteval; + + range.end =3D vma_address_end(&pvmw); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, + address, range.end); + /* If sharing is possible, start and end will be adjusted accordingly. */ + adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); + mmu_notifier_invalidate_range_start(&range); + + /* There is only a single mapping in a VMA. */ + if (!page_vma_mapped_walk(&pvmw)) + goto range_end; + + address =3D pvmw.address; + pteval =3D huge_ptep_get(mm, address, pvmw.pte); + if (likely(pte_present(pteval))) + pfn =3D pte_pfn(pteval); + else + pfn =3D softleaf_to_pfn(softleaf_from_pte(pteval)); + + page =3D folio_page(folio, pfn - folio_pfn(folio)); + anon_exclusive =3D anon && PageAnonExclusive(page); + + /* + * huge_pmd_unshare may unmap an entire PMD page. There is no way of + * knowing exactly which PMDs may be cached for this mm, so we must flush + * them all. start/end were already adjusted above to cover this range. + */ + flush_cache_range(vma, range.start, range.end); + + /* + * To call huge_pmd_unshare, i_mmap_rwsem must be held in write mode. + * Caller needs to explicitly do this outside rmap routines. We also must + * hold hugetlb vma_lock in write mode. Lock order dictates acquiring + * vma_lock BEFORE i_mmap_rwsem. We can only try lock here and fail if + * unsuccessful. + */ + if (!anon) { + struct mmu_gather tlb; + + VM_WARN_ON_ONCE(!(flags & TTU_RMAP_LOCKED)); + if (!hugetlb_vma_trylock_write(vma)) { + ret =3D false; + goto walk_done; + } + + tlb_gather_mmu_vma(&tlb, vma); + if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) { + hugetlb_vma_unlock_write(vma); + huge_pmd_unshare_flush(&tlb, vma); + tlb_finish_mmu(&tlb); + /* + * The PMD table was unmapped, + * consequently unmapping the folio. + */ + goto walk_done; + } + hugetlb_vma_unlock_write(vma); + tlb_finish_mmu(&tlb); + } + + /* Nuke the hugetlb page table entry */ + pteval =3D huge_ptep_clear_flush(vma, address, pvmw.pte); + if (huge_pte_dirty(pteval)) + folio_mark_dirty(folio); + writable =3D pte_write(pteval); + + VM_WARN_ON_FOLIO(writable && anon && !anon_exclusive, folio); + + if (PageHWPoison(page)) { + pteval =3D swp_entry_to_pte(make_hwpoison_entry(page)); + hugetlb_count_sub(folio_nr_pages(folio), mm); + set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); + } else { + pte_t swp_pte; + + /* + * arch_unmap_one() is expected to be a NOP on architectures + * where we could have PFN swap PTEs, so we'll not check/care. + */ + if (arch_unmap_one(mm, vma, address, pteval) < 0) { + set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); + ret =3D false; + goto walk_done; + } + + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + if (anon_exclusive && hugetlb_try_share_anon_rmap(folio)) { + set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); + ret =3D false; + goto walk_done; + } + + /* + * Store the pfn of the page in a special migration pte. A + * hugetlb fault waits for migration to complete before retrying. + */ + swp_pte =3D make_migration_pte(page, pteval, writable, anon_exclusive); + set_huge_pte_at(mm, address, pvmw.pte, swp_pte, hsz); + trace_set_migration_pte(address, pte_val(swp_pte), + folio_order(folio)); + } + + hugetlb_remove_rmap(folio); + if (vma->vm_flags & VM_LOCKED) + mlock_drain_local(); + folio_put(folio); +walk_done: + page_vma_mapped_walk_done(&pvmw); +range_end: + mmu_notifier_invalidate_range_end(&range); + + return ret; +} + /* * @arg: enum ttu_flags will be passed to this argument. * @@ -2560,7 +2692,6 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, struct mmu_notifier_range range; enum ttu_flags flags =3D (enum ttu_flags)(long)arg; unsigned long pfn; - unsigned long hsz =3D 0; =20 /* * When racing against e.g. zap_pte_range() on another cpu, @@ -2573,26 +2704,13 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, =20 /* * For THP, we have to assume the worse case ie pmd for invalidation. - * For hugetlb, it could be much worse if we need to do pud - * invalidation in the case of pmd sharing. * * Note that the page can not be free in this function as call of - * try_to_unmap() must hold a reference on the page. + * try_to_migrate() must hold a reference on the page. */ range.end =3D vma_address_end(&pvmw); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, address, range.end); - if (folio_test_hugetlb(folio)) { - /* - * If sharing is possible, start and end will be adjusted - * accordingly. - */ - adjust_range_if_pmd_sharing_possible(vma, &range.start, - &range.end); - - /* We need the huge page size for set_huge_pte_at() */ - hsz =3D huge_page_size(hstate_vma(vma)); - } mmu_notifier_invalidate_range_start(&range); =20 while (page_vma_mapped_walk(&pvmw)) { @@ -2639,10 +2757,7 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, VM_BUG_ON_FOLIO(!pvmw.pte, folio); =20 address =3D pvmw.address; - if (folio_test_hugetlb(folio)) - pteval =3D huge_ptep_get(mm, address, pvmw.pte); - else - pteval =3D ptep_get(pvmw.pte); + pteval =3D ptep_get(pvmw.pte); if (likely(pte_present(pteval))) { pfn =3D pte_pfn(pteval); } else { @@ -2653,66 +2768,13 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, const softleaf_t entry =3D softleaf_from_pte(pteval); =20 pfn =3D softleaf_to_pfn(entry); - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); } =20 subpage =3D folio_page(folio, pfn - folio_pfn(folio)); anon_exclusive =3D folio_test_anon(folio) && PageAnonExclusive(subpage); =20 - if (folio_test_hugetlb(folio)) { - bool anon =3D folio_test_anon(folio); - - /* - * huge_pmd_unshare may unmap an entire PMD page. - * There is no way of knowing exactly which PMDs may - * be cached for this mm, so we must flush them all. - * start/end were already adjusted above to cover this - * range. - */ - flush_cache_range(vma, range.start, range.end); - - /* - * To call huge_pmd_unshare, i_mmap_rwsem must be - * held in write mode. Caller needs to explicitly - * do this outside rmap routines. - * - * We also must hold hugetlb vma_lock in write mode. - * Lock order dictates acquiring vma_lock BEFORE - * i_mmap_rwsem. We can only try lock here and - * fail if unsuccessful. - */ - if (!anon) { - struct mmu_gather tlb; - - VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); - if (!hugetlb_vma_trylock_write(vma)) { - page_vma_mapped_walk_done(&pvmw); - ret =3D false; - break; - } - - tlb_gather_mmu_vma(&tlb, vma); - if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) { - hugetlb_vma_unlock_write(vma); - huge_pmd_unshare_flush(&tlb, vma); - tlb_finish_mmu(&tlb); - /* - * The PMD table was unmapped, - * consequently unmapping the folio. - */ - page_vma_mapped_walk_done(&pvmw); - break; - } - hugetlb_vma_unlock_write(vma); - tlb_finish_mmu(&tlb); - } - /* Nuke the hugetlb page table entry */ - pteval =3D huge_ptep_clear_flush(vma, address, pvmw.pte); - if (pte_dirty(pteval)) - folio_mark_dirty(folio); - writable =3D pte_write(pteval); - } else if (likely(pte_present(pteval))) { + if (likely(pte_present(pteval))) { flush_cache_page(vma, address, pfn); /* Nuke the page table entry. */ if (should_defer_flush(mm, flags)) { @@ -2751,14 +2813,8 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, VM_WARN_ON_FOLIO(folio_is_device_private(folio), folio); =20 pteval =3D swp_entry_to_pte(make_hwpoison_entry(subpage)); - if (folio_test_hugetlb(folio)) { - hugetlb_count_sub(folio_nr_pages(folio), mm); - set_huge_pte_at(mm, address, pvmw.pte, pteval, - hsz); - } else { - dec_mm_counter(mm, mm_counter(folio)); - set_pte_at(mm, address, pvmw.pte, pteval); - } + dec_mm_counter(mm, mm_counter(folio)); + set_pte_at(mm, address, pvmw.pte, pteval); } else if (likely(pte_present(pteval)) && pte_unused(pteval) && !userfaultfd_armed(vma)) { /* @@ -2781,28 +2837,15 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, * so we'll not check/care. */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { - if (folio_test_hugetlb(folio)) - set_huge_pte_at(mm, address, pvmw.pte, - pteval, hsz); - else - set_pte_at(mm, address, pvmw.pte, pteval); + set_pte_at(mm, address, pvmw.pte, pteval); ret =3D false; page_vma_mapped_walk_done(&pvmw); break; } =20 /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ - if (folio_test_hugetlb(folio)) { - if (anon_exclusive && - hugetlb_try_share_anon_rmap(folio)) { - set_huge_pte_at(mm, address, pvmw.pte, - pteval, hsz); - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; - } - } else if (anon_exclusive && - folio_try_share_anon_rmap_pte(folio, subpage)) { + if (anon_exclusive && + folio_try_share_anon_rmap_pte(folio, subpage)) { set_pte_at(mm, address, pvmw.pte, pteval); ret =3D false; page_vma_mapped_walk_done(&pvmw); @@ -2816,11 +2859,7 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, */ swp_pte =3D make_migration_pte(subpage, pteval, writable, anon_exclusive); - if (folio_test_hugetlb(folio)) - set_huge_pte_at(mm, address, pvmw.pte, swp_pte, - hsz); - else - set_pte_at(mm, address, pvmw.pte, swp_pte); + set_pte_at(mm, address, pvmw.pte, swp_pte); trace_set_migration_pte(address, pte_val(swp_pte), folio_order(folio)); /* @@ -2829,10 +2868,7 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, */ } =20 - if (unlikely(folio_test_hugetlb(folio))) - hugetlb_remove_rmap(folio); - else - folio_remove_rmap_pte(folio, subpage, vma); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2854,7 +2890,8 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, void try_to_migrate(struct folio *folio, enum ttu_flags flags) { struct rmap_walk_control rwc =3D { - .rmap_one =3D try_to_migrate_one, + .rmap_one =3D folio_test_hugetlb(folio) ? + try_to_migrate_hugetlb_one : try_to_migrate_one, .arg =3D (void *)flags, .done =3D folio_not_mapped, .anon_lock =3D folio_lock_anon_vma_read, --=20 2.43.0 From nobody Tue Sep 29 02:33:20 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012060.outbound.protection.outlook.com [52.101.48.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC6792F3C18 for ; Thu, 13 Aug 2026 04:25:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.60 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595106; cv=fail; b=O7avBb9k0+5/zg6wjaz+IfILkkMTMqnUcjHuhlIXkHp5KcilfCBihm1zXNPDb6jiedsYBUW+ujjFFRb6v7o3stasUgLbePBnYXGr2vIBseEvXlTu+2kMj4UC9CiZhHhFVOraydgiyh0+qfVh574WYaYotHNROXCxXRQzB/pxEko= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786595106; c=relaxed/simple; bh=N04qQs6ab3QOp2plxNgK76AdZ7rwOslb9VXB2PbRwNk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-ID:References: In-Reply-To:To:CC; b=ESbMkiRVcq1NsZ/U/1RF0qSidIo8M1CByrrFowyse1D4ykazLh8FCkT4KdT7DjkEtpk0A7WDL96/hnBLp70rSBmqu0GD+ptNwtXcgN4FBjBK1pjgL5BvfKZR+9rhDDU5r0J6wFZ3HpALEwDgtgHwxQn2ZIGHoLvLMGEHgR5+M2g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=x3fd42jB; arc=fail smtp.client-ip=52.101.48.60 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="x3fd42jB" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ns+WIQmp0dLl/WQQgCyX8jeGpRqzoiI4rWTRBxLCG6vD4MxExKKukgp7LsY0v6UPZpA0M6mg2zBpjKA6g6NfjLvBOOKXY94J5i00rmtk9JtnSVfNm1bv3iwMELg4uumof11louGVtZi8LLLqBzWo0/h2UD17Ss5wTmTqQtqP9DFNUKsqhrTGogVYRYgmiSnc7Q1KVtJc4kd14HxctCIBEtRKLmELPfufFasm0FkOucFTBdzLYc2/g+nprnYgMnZBfQmjFb3o8/HBfoHAAgp8lXVkeLez41k3rGsibLmfybO9xC3+xY+uyLvszLLl2uGe2YzTfQsMdOnRw18RROzJOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xJmcLgRoaeaouSkf3yJv6TF6RMZzz0FrmGos9xFDxFc=; b=G1z5TVFWiSdqVPC2L+8xgbJlHtnUAXYacTleZQuoHfRsFoI3s12UY4VFoZr3bfg6UyYj/8ww2px0CcPXANSdKUtKuiRoK5GVlHnyFtMyOHuRCDWHCwLS665NrMNE0L8xeVFg7levicNaneXLRwtYkL9QonNgnMK73rzfMbhkOHxeYEyxST0M7Bsuw9EocyeUT/sfW4I6Djt1kQQiHtUxJMJf/x0ZnHXXbC+85tgOJbPCCPDRirkFVu7hX7V5M/rZnICe1h9SkVQDXGY5ovd8RcpSSrhcMo1tU2EBdY1n4nQ2z1XMUirdEcqsXZA4bXlGYCQV+v1g2Jc5njJWEdC3mg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=surriel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xJmcLgRoaeaouSkf3yJv6TF6RMZzz0FrmGos9xFDxFc=; b=x3fd42jBBoFApQ9qSgNFVts1TnhSLPEclP2/38dK7NQs25ObQ++rcPwWGU34hW4xJsNCJrEYAftEvl14c/NiQMZA4UZiTGzyyypRk5FTw3A0V2u0pgqdIayMPeL5UziHaop4eYeKwBDkWprunjyd02hDRt3ACn8TMga/PnogaVo= Received: from BLAPR05CA0005.namprd05.prod.outlook.com (2603:10b6:208:36e::14) by CY1PR12MB9601.namprd12.prod.outlook.com (2603:10b6:930:107::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.14; Thu, 13 Aug 2026 04:24:55 +0000 Received: from BN3PEPF0000B371.namprd21.prod.outlook.com (2603:10b6:208:36e:cafe::91) by BLAPR05CA0005.outlook.office365.com (2603:10b6:208:36e::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.339.3 via Frontend Transport; Thu, 13 Aug 2026 04:24:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN3PEPF0000B371.mail.protection.outlook.com (10.167.243.168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.0 via Frontend Transport; Thu, 13 Aug 2026 04:24:54 +0000 Received: from [127.0.1.1] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 12 Aug 2026 23:24:46 -0500 From: Shivank Garg Date: Thu, 13 Aug 2026 04:23:18 +0000 Subject: [PATCH v2 7/7] mm/rmap: batch the unmap of large folios in try_to_migrate_one() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: <20260813-migrate-rmap-batch-v2-7-3c5424c555c7@amd.com> References: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> In-Reply-To: <20260813-migrate-rmap-batch-v2-0-3c5424c555c7@amd.com> To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Rik van Riel , "Liam R. Howlett" , Vlastimil Babka , Harry Yoo , Jann Horn , Lance Yang , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , "Alistair Popple" CC: Karim Manaouil , Frank van der Linden , Kinsey Ho , Wei Xu , Bharata B Rao , David Rientjes , Dev Jain , , , Shivank Garg X-Mailer: b4 0.15-dev-47d62 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786595034; l=7623; i=shivankg@amd.com; s=20260518; h=from:subject:message-id; bh=N04qQs6ab3QOp2plxNgK76AdZ7rwOslb9VXB2PbRwNk=; b=LZGQQRKQxxrd5UfHQJW67iLHdSc9AzHrsMjwBxSj4DBrm7sUVRZomdXWG4/fFq8Ad7t5c3mfj QO6Jw4nyR60B9R/9He0xlAFU9Xhjd2luoGU9D8S/uaTvVXUJTmcUDUM X-Developer-Key: i=shivankg@amd.com; a=ed25519; pk=2l2QGTeXuGkZTtfmx0nPQU8iFZfjYmX/ymMojitevx4= X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B371:EE_|CY1PR12MB9601:EE_ X-MS-Office365-Filtering-Correlation-Id: d9cb8934-ae4a-4355-c3a4-08def8f2d336 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|7416014|36860700016|82310400026|1800799024|921020|56012099006|10067099003|11063799006|5023799004|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: slDoFa0XVeCXamLiOKfcrYlDMUSkk+6/44H2Hljoj0sEsNrdyG8GC9Q40zhGT5pVSij1vGxGom9suOY3tT3lLLfDbPiCoNSWVKlWTOly08wRjIyJlpMHsiEt/L1C0ediR9z66IghQCLzMl6GAMp6PpBfe9anwXd77Ogi2tj/lOVG+SoJJ8jxy3pH++cM5sAHjg0pKVjo+up5MTiPJT5xFx23Is27xpE2FbhopqrRF+Eep3R2HMilYAD3CdPIHhUcnY7lN+tIyo/nt92DELZq8uCz1ME2I1kkx6bEb8mR10hwcEIiry6aEHuWoiTTPNi7laTfal8rk7HCRM73JiMaLr7tKAiwRxsTgWxnUhCHeckhgV5CJlXi08oZZFNH14N7LDFJMcKyINBwWPcDhqE0VaztIcfkaicDMpdsgnGlwcms1OBe4SsgFhxHmkotOgr/A59OnitmNvjEINK0T5a9HL4HdIEgaLDXDtnJn6GbJ+c56ySi23/+uU/+uXwchKICqV2SXKp8NFeEWQ2Zr2bZ9MpoAIK+dvUrfAptyTerAv+HQUWirTKA6GdzE1UkTbenW+VbafEOpEyuwBxpOBWQQVFf/FHWMJMf0gQZ00wlRcoS+N1E7fdGQb6Tp/KEd9vVK8NN3L+5zpoB15wKX4l2UlflNFcQib8O/Qmx3/Rm+m+gCEkwmM4bpfvv+WW1IVy2a2HqOF92Kumi+Tvx/6P5dr5BBwBsljpCip2/8J97W21dTNWGT52jThalMRRul5ln X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(23010399003)(376014)(7416014)(36860700016)(82310400026)(1800799024)(921020)(56012099006)(10067099003)(11063799006)(5023799004)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8Hmd0TyVHivaYeSWowyOE5RXs+Vt/crDQk32hQG97Mlws36m/5I7CipAGBWrJyl2aa7gyGCBrM6ztDTFmpWZRJtlqKld0e4qs855DRybmNt7rUlIojUKPgx9gU7lHR/AgVCsvFdiFUxQu66lvdyHdjow4hG7fT/csGPuWpWCYS0Lzi2ongPw3e+5lNojTfAPuyOulkVwEoDIvMm8beQYALNV4UO323UEe0xtps70X2cNh5eyK5JnRhcCrJCC69msgRY0kKcN6sIUHZwqBbcTN00aCzSQM3QWVJAsq9SiojamBpuHK9KOiQoaaoihyaPF8rQlwX8buVL8plavSub/6AQUZOF85L6+oj07OBWFuZ+dUf3anzyhHKb3C/Xbh4Wim0RGWOx75TVbBtL+mDJSb2aDadWd5MReLHx9A5eozxhXqTRNHHyaG4XFAefiUQZg X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2026 04:24:54.2139 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d9cb8934-ae4a-4355-c3a4-08def8f2d336 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B371.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR12MB9601 try_to_migrate_one() converts present PTEs to migration entries one at a time. For a PTE-mapped large folio, this repeat calls to ptep clear+flush, the migration entry build and set, folio_remove_rmap_pte() and folio_put(), each re-entering page_vma_mapped_walk() once per base page (256 times for 1M folio). Mirror try_to_unmap_one() to introduce folio_migrate_pte_batch() to detect eligible batch for PTEs mapping conseuctive subpages of a large folios, and convert the whole batch in one shot using the batched helpers. A side-effect of this change is trace_set_migration_pte() will record one event per batched run instead of earlier behavior of one per base page. Signed-off-by: Shivank Garg --- mm/rmap.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++------------= ---- 1 file changed, 86 insertions(+), 29 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 35752a70f3a0..63b885c0b7ef 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2675,6 +2675,44 @@ static bool try_to_migrate_hugetlb_one(struct folio = *folio, return ret; } =20 +static inline unsigned int folio_migrate_pte_batch(struct folio *folio, + struct page_vma_mapped_walk *pvmw, pte_t pte, + struct page *subpage, bool anon_exclusive) +{ + unsigned long end_addr, addr =3D pvmw->address; + struct vm_area_struct *vma =3D pvmw->vma; + unsigned int max_nr, nr; + +#ifdef __HAVE_ARCH_UNMAP_ONE + /* Cannot batch unmap if arch_unmap_one() is defined. */ + return 1; +#endif + + if (!folio_test_large(folio)) + return 1; + if (folio_is_zone_device(folio) || folio_test_has_hwpoisoned(folio)) + return 1; + if (pte_unused(pte)) + return 1; + + /* We may only batch within a single VMA and a single page table. */ + end_addr =3D pmd_addr_end(addr, vma->vm_end); + max_nr =3D (end_addr - addr) >> PAGE_SHIFT; + /* + * If unmap fails, we need to restore the ptes. To avoid accidentally + * upgrading write permissions for ptes that were not originally writable, + * and to avoid losing the soft-dirty bit, use the appropriate FPB flags. + */ + nr =3D folio_pte_batch_flags(folio, vma, pvmw->pte, &pte, max_nr, + FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY); + + /* Limit possible batch count to a uniform PageAnonExclusive value */ + if (folio_test_anon(folio)) + nr =3D page_anon_exclusive_batch(0, nr, subpage, anon_exclusive); + + return nr; +} + /* * @arg: enum ttu_flags will be passed to this argument. * @@ -2686,12 +2724,12 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, { struct mm_struct *mm =3D vma->vm_mm; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); - bool anon_exclusive, writable, ret =3D true; + bool anon_exclusive, hwpoison, writable, ret =3D true; pte_t pteval; struct page *subpage; struct mmu_notifier_range range; enum ttu_flags flags =3D (enum ttu_flags)(long)arg; - unsigned long pfn; + unsigned long pfn, end_addr, nr_pages; =20 /* * When racing against e.g. zap_pte_range() on another cpu, @@ -2744,11 +2782,8 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || !folio_test_pmd_mappable(folio), folio); =20 - if (set_pmd_migration_entry(&pvmw, subpage)) { - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; - } + if (set_pmd_migration_entry(&pvmw, subpage)) + goto walk_abort; continue; #endif } @@ -2773,10 +2808,25 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, subpage =3D folio_page(folio, pfn - folio_pfn(folio)); anon_exclusive =3D folio_test_anon(folio) && PageAnonExclusive(subpage); + /* + * memory_failure() can set PageHWPoison concurrently without holding + * the folio lock. Snapshot the flag here to decide whether to batch + * PTEs or install hwpoison entry. + */ + hwpoison =3D PageHWPoison(subpage); =20 + nr_pages =3D 1; if (likely(pte_present(pteval))) { - flush_cache_page(vma, address, pfn); - /* Nuke the page table entry. */ + if (!hwpoison) + nr_pages =3D folio_migrate_pte_batch(folio, &pvmw, + pteval, subpage, + anon_exclusive); + + end_addr =3D address + nr_pages * PAGE_SIZE; + flush_cache_range(vma, address, end_addr); + + /* Nuke the page table entries. */ + pteval =3D get_and_clear_ptes(mm, address, pvmw.pte, nr_pages); if (should_defer_flush(mm, flags)) { /* * We clear the PTE but do not flush so potentially @@ -2786,11 +2836,9 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, * transition on a cached TLB entry is written through * and traps if the PTE is unmapped. */ - pteval =3D ptep_get_and_clear(mm, address, pvmw.pte); - - set_tlb_ubc_flush_pending(mm, pteval, address, address + PAGE_SIZE); + set_tlb_ubc_flush_pending(mm, pteval, address, end_addr); } else { - pteval =3D ptep_clear_flush(vma, address, pvmw.pte); + flush_tlb_range(vma, address, end_addr); } if (pte_dirty(pteval)) folio_mark_dirty(folio); @@ -2809,7 +2857,8 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, /* Update high watermark before we lower rss */ update_hiwater_rss(mm); =20 - if (PageHWPoison(subpage)) { + if (hwpoison) { + VM_WARN_ON_ONCE(nr_pages !=3D 1); VM_WARN_ON_FOLIO(folio_is_device_private(folio), folio); =20 pteval =3D swp_entry_to_pte(make_hwpoison_entry(subpage)); @@ -2837,19 +2886,15 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, * so we'll not check/care. */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { - set_pte_at(mm, address, pvmw.pte, pteval); - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; + set_ptes(mm, address, pvmw.pte, pteval, nr_pages); + goto walk_abort; } =20 - /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + /* See folio_try_share_anon_rmap_ptes(): clear PTE first. */ if (anon_exclusive && - folio_try_share_anon_rmap_pte(folio, subpage)) { - set_pte_at(mm, address, pvmw.pte, pteval); - ret =3D false; - page_vma_mapped_walk_done(&pvmw); - break; + folio_try_share_anon_rmap_ptes(folio, subpage, nr_pages)) { + set_ptes(mm, address, pvmw.pte, pteval, nr_pages); + goto walk_abort; } =20 /* @@ -2859,19 +2904,31 @@ static bool try_to_migrate_one(struct folio *folio,= struct vm_area_struct *vma, */ swp_pte =3D make_migration_pte(subpage, pteval, writable, anon_exclusive); - set_pte_at(mm, address, pvmw.pte, swp_pte); trace_set_migration_pte(address, pte_val(swp_pte), folio_order(folio)); + + /* Set nr_pages migration entries, advancing the PFN. */ + set_softleaf_ptes(mm, address, pvmw.pte, swp_pte, nr_pages); /* * No need to invalidate here it will synchronize on * against the special swap migration pte. */ } =20 - folio_remove_rmap_pte(folio, subpage, vma); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); - folio_put(folio); + finish_folio_unmap(vma, folio, subpage, nr_pages); + + /* + * If we batched the entire folio, there is nothing left to + * walk; stop right here. + */ + if (nr_pages =3D=3D folio_nr_pages(folio)) + goto walk_done; + continue; +walk_abort: + ret =3D false; +walk_done: + page_vma_mapped_walk_done(&pvmw); + break; } =20 mmu_notifier_invalidate_range_end(&range); --=20 2.43.0