From nobody Wed Jun 17 06:11:26 2026 Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010030.outbound.protection.outlook.com [52.101.201.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 013E73B2FE5 for ; Mon, 27 Apr 2026 14:26:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.201.30 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777299980; cv=fail; b=tnhIU7tt1x3hNifFDNqb2vt1000DC7vAd3LVmD8vDNMkpUeBBoEC66RP7sIQxSgr0yunsDs9AV7loYvdAnyUMmdm7U4O3LSoRgaJI3Duewi1CY7FazMdjqaj77LtygZc/XKH8KRYji+M1C268n2PY6t32fbK/u2/pe3NY26R/0U= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777299980; c=relaxed/simple; bh=OONPNk9VfDt77VJG5BKQfJ87kdQovq6G89Ike1vlb1M=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gn3n0+8eNVgrrNXwandppM13uO2bViRRFW/K9cuYKIkpym9UYOkP/qX4h5omRQ2dD5YihIQTIS3YygtrAuSr6G7glp+uTGhHNEhCmuqhc5dEcCKNJaJG/tgD9+y4qomHu3tQhv15XXGqkKcGUBiaV0wK8GODOVfregiHlaGt6QE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RPvxd4xK; arc=fail smtp.client-ip=52.101.201.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RPvxd4xK" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YmsWPUf8AtN5UtIVwg9OHAOrxmG28ppQJigr+6PGzk/ZtIDkypXqcEAkat2MqQAA4/RouOY2U/emWoOgOHtGvBk3lcLj+a9UB/K2uRo3v3boAgckmDSG5P3sqbNyAUortjaocvT5iTCw74txxggRzLF8bW97N8typhuf0GBNZX+27CS+svvhAgowVel0m2zeRBY2nHtWDC8U5yGcQmW69MbJGix0xN1rI2/A79HpHV+zHnckbbLAShT6bJjRXpF3uPzI0cqn6fC+2+1YWwP892zeUc9f8kIDJh0Q2hPZoX+xhkh5TNAiAf6AgFWn/krOnIHy+21EsGXVVSXzU9QDtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bJ0aMJ6q5PumEHrs3re93Z6hWL0nQNX+QntJkYKc7os=; b=sTBiacoO0RUQ4LM4QzfxzWwgCPD3uD2bcnkn/kFp3GjZoRB3FTmMyK8x4nm9Bwww5G4hOFaFjGO5OdNQZCUrsxEmbeMiCxGXeuUgwIhNyfAG3zYXdmCVZU1ru2OAeigUuLc15M6yoNIjuWz1e9xiSySTPEvUBhTQCTOObLvOjMvGEMZkMyAAllMg65WP21VdK1w5BZxBP46VamJ8LxJ74PURYrI9WQeJEL3NKxl7nTI+kAemNgNNxQFikmw4UE33dVGn5ikWVOkMA5wMUtZxjV2sy+dEwoxVcPgetSWMrkthmtOcvqWl/mxB0D1Kh9feNw6ZnTbGl0K/9VJ9K1P7NA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bJ0aMJ6q5PumEHrs3re93Z6hWL0nQNX+QntJkYKc7os=; b=RPvxd4xKDYZhtSDKbZrxWFeQNUz+yJnqFQYHHz61wpuRhW6y2QlKZi43zCXhiVMri1x1pOe0baxAMdiCDCALhp2wayjClxicBgeeOr0pX0erJlxVu1c3Gm2OFjF2E/K9hHWx3TDUOXsCjOamRAa9O/gswazedQ5CUh9hjJ1XO/Y= Received: from DS7PR03CA0233.namprd03.prod.outlook.com (2603:10b6:5:3ba::28) by SA3PR12MB7860.namprd12.prod.outlook.com (2603:10b6:806:307::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.15; Mon, 27 Apr 2026 14:26:02 +0000 Received: from CY4PEPF0000E9CD.namprd03.prod.outlook.com (2603:10b6:5:3ba:cafe::68) by DS7PR03CA0233.outlook.office365.com (2603:10b6:5:3ba::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9846.26 via Frontend Transport; Mon, 27 Apr 2026 14:26:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by CY4PEPF0000E9CD.mail.protection.outlook.com (10.167.241.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.18 via Frontend Transport; Mon, 27 Apr 2026 14:26:01 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 27 Apr 2026 09:25:55 -0500 From: Shivank Garg To: Andrew Morton , David Hildenbrand , , , CC: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H . Peter Anvin" , Ankur Arora , Bharata B Rao , "Hrushikesh Salunke" , David Rientjes , "Shivank Garg" Subject: [RFC PATCH 1/1] mm: batch page copies in folio_copy() and folio_mc_copy() Date: Mon, 27 Apr 2026 14:20:38 +0000 Message-ID: <20260427142036.111940-4-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260427142036.111940-2-shivankg@amd.com> References: <20260427142036.111940-2-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb08.amd.com (10.181.42.217) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9CD:EE_|SA3PR12MB7860:EE_ X-MS-Office365-Filtering-Correlation-Id: c6a46e46-8cb9-4f61-535c-08dea468e865 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|82310400026|36860700016|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: GAgcNw5pQJFNfkz52lBBo/UzZdYN8PtBLDCdvjcV5j0EvZ4GP8CJKnHTEXFkHNBUq/hxBX8msPFxYcRix/0CXg/lcOn5xwr0gf8e4BEkRVMz26jUlVbsiGyi5Sy0Hb3/bX5izr1ImDb4MDazF+VgQNd8FbN1dMLN2glzi4lrBV8lZSWLAMB3ee9ckjMuN4q7lC4PL88KnMyqz3rbPkf9fMVtLSp97eum/T8A+FWVRog6JwvsvuqLUq6qPulVbq/oLymiWQsYJgW/JV1DzzZcOhqwqV+NEg3Ebh7A872qZILMw2efGSPYRQgB0rUu1yLc0zcSTdpz9vxT/0VVNM7nPdsLYpkN7xCRR5W8phNQzCLubVqrKD0OqsLZdg201c20dL4UfJ4CfizaYjAp+dRJ2KWO2ZSaEC2ZJ0ptiwnI8h9JThMQFgev4qf7gck2VUM4QIogfcZMhvvAQgMA8SeTO6kg5GpngQvBx7Qfkzc1jZFLMAwfUyxpmCp0L3QMEDiWryoDyJfXYPM5SbtMJTurVUvaxR7wgf6NahWqWolvf0Ac/EG7EMEUCdCMY6kplAZ5JrysWWQ5489mP3g0zFsnnXPY0BVT3uAVRYZvWC/IhSKNANUBByhpFwm2uGw4G5jjlLh3V2ftPuFCVXrf0Ny806Vhx7gNtb7o0U1DuIqZKsQ6abV2xrfSX4KiRrCHol+a+MVu4mpXls6F5wi/JMoMy6TgsHxyTCH65zKuRMnLds4YBXOz1iE2sXqU4VkpZRqKpfdNI9xcvd5FvTkASZuu5A== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(82310400026)(36860700016)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: XyUCBnoWBHuXUN3utbQj2pm3LuFDdaUhLUgjayJFAUS+UIGzc+w+Hv+TUOjhN4UKWtcRo+qNMqsLYyZy4ics1MFtNzG4pfg1EcuwempweDPfhRzbI0auUd18sIhdMomHP5kLFiUlCaUcsZhnv8QHXV6K7AoXmUjFv2RDrCMNnomOum5gYvZWpP0wCNogTKjL9WTRYA1/KgspZ41ovpWrLkLtCno/HvH1IWZsf7iIzp+3i3d/13oCgx7pnNeY1kaJiajPOOY7kpmg6et5Q3vRtxsfghyh2cI6C+nh0S5pIxAg5g1iuybD1sLTNxPEamomydVpRhd15to1D3bgwNjR+lUCkQX7iRAfwfIexIwr+I0C62VovjXE0WH5QNNleMJTyrTjLOkC2bxmhYwVosk4jdV7UbPW76iOFtVOURUpkM+CmyUInq2b7trHJaCW+iEG X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2026 14:26:01.4811 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c6a46e46-8cb9-4f61-535c-08dea468e865 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9CD.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7860 Rewrite folio_copy() and folio_mc_copy() as thin wrappers around new batched helpers copy_highpages() and copy_mc_highpages(). The current implementations iterate copy_highpage() (or its #MC-aware variant) per 4 KB page. For a single 2 MB folio that loop runs 512 times and pays, per page: - kmap_local_page() / kunmap_local() - cond_resched() - one invocation of the architecture copy_page()/memcpy() primitive The new helpers issue a single copy_mc_to_kernel()/memcpy() over the whole contiguous range when CONFIG_HIGHMEM is off and no architecture overrides (__HAVE_ARCH_COPY_HIGHPAGE) copy_highpage(). HIGHMEM and arch overrides keep the existing per-page path. Tested on dual-socket AMD EPYC 9655 (Zen 5) with a CXL.mem node. In-kernel folio_mc_copy() microbenchmark on 2 MB folios, source evicted from cache before each iteration and measured throughput: direction baseline GB/s optimized GB/s speedup DRAM0 -> DRAM1 18.65 =C2=B1 1.37 38.03 =C2=B1 3.21 2.04x DRAM0 -> CXL 25.46 =C2=B1 2.89 39.29 =C2=B1 1.17 1.54x CXL -> DRAM0 20.61 =C2=B1 3.95 35.07 =C2=B1 0.62 1.70x End-to-end move_pages(2) throughput on anonymous 2 MB mTHP folios, 1 GB migrated per run: direction baseline GB/s optimized GB/s speedup DRAM0 -> DRAM1 7.20 =C2=B1 0.03 8.01 =C2=B1 0.02 1.11x DRAM0 -> CXL 11.12 =C2=B1 0.15 13.07 =C2=B1 0.03 1.18x DRAM1 -> DRAM0 7.21 =C2=B1 0.02 7.95 =C2=B1 0.02 1.10x CXL -> DRAM0 9.10 =C2=B1 0.05 9.49 =C2=B1 0.01 1.04x On AMD EPYC 7713 (Zen 3 / Milan, REP_GOOD without FSRM/ERMS) the folio_copy() bulk path regresses because memcpy() falls through to memcpy_orig (an unrolled movq loop), which is slower than the per-page copy_page() (microcoded rep movsq) it replaces.=20 Same 2 MB folio_copy() microbench, source evicted before each iteration: direction baseline GB/s optimized GB/s speedup DRAM0 -> DRAM1 13.03 =C2=B1 0.76 11.59 =C2=B1 0.18 0.89x DRAM1 -> DRAM0 12.85 =C2=B1 0.25 11.02 =C2=B1 0.10 0.86x Cover letter discusses introducing a copy_pages() helper to avoid this regression.=20 Signed-off-by: Shivank Garg --- include/linux/highmem.h | 58 +++++++++++++++++++++++++++++++++++++++++ mm/util.c | 25 +++--------------- 2 files changed, 62 insertions(+), 21 deletions(-) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 871d817426bc..daee3f1863d1 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -439,6 +439,23 @@ static inline void copy_highpage(struct page *to, stru= ct page *from) =20 #endif =20 +static inline void copy_highpages(struct page *to, struct page *from, + unsigned long nr_pages) +{ + unsigned long i; + +#ifndef __HAVE_ARCH_COPY_HIGHPAGE + if (!IS_ENABLED(CONFIG_HIGHMEM)) { + memcpy(page_address(to), page_address(from), nr_pages << PAGE_SHIFT); + for (i =3D 0; i < nr_pages; i++) + kmsan_copy_page_meta(to + i, from + i); + return; + } +#endif + for (i =3D 0; i < nr_pages; i++) + copy_highpage(to + i, from + i); +} + #ifdef copy_mc_to_kernel /* * If architecture supports machine check exception handling, define the @@ -484,6 +501,40 @@ static inline int copy_mc_highpage(struct page *to, st= ruct page *from) =20 return ret; } + +static inline int copy_mc_highpages(struct page *to, struct page *from, + unsigned long nr_pages) +{ + unsigned long i; + +#ifndef __HAVE_ARCH_COPY_HIGHPAGE + if (!IS_ENABLED(CONFIG_HIGHMEM)) { + unsigned long len =3D nr_pages << PAGE_SHIFT; + unsigned long ret; + + ret =3D copy_mc_to_kernel(page_address(to), + page_address(from), len); + if (!ret) { + for (i =3D 0; i < nr_pages; i++) + kmsan_copy_page_meta(to + i, from + i); + return 0; + } + /* + * copy_mc_to_kernel() returns the number bytes that were not copied, + * counted from the end. The first failing page is therefore at + * offset (len - ret) >> PAGE_SHIFT within the range. + */ + memory_failure_queue(page_to_pfn(from) + + ((len - ret) >> PAGE_SHIFT), 0); + return -EHWPOISON; + } +#endif + + for (i =3D 0; i < nr_pages; i++) + if (copy_mc_highpage(to + i, from + i)) + return -EHWPOISON; + return 0; +} #else static inline int copy_mc_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma) @@ -497,6 +548,13 @@ static inline int copy_mc_highpage(struct page *to, st= ruct page *from) copy_highpage(to, from); return 0; } + +static inline int copy_mc_highpages(struct page *to, struct page *from, + unsigned long nr_pages) +{ + copy_highpages(to, from, nr_pages); + return 0; +} #endif =20 static inline void memcpy_page(struct page *dst_page, size_t dst_off, diff --git a/mm/util.c b/mm/util.c index 3cc949a0b7ed..93f0d9daffce 100644 --- a/mm/util.c +++ b/mm/util.c @@ -749,32 +749,15 @@ EXPORT_SYMBOL(folio_mapping); */ void folio_copy(struct folio *dst, struct folio *src) { - long i =3D 0; - long nr =3D folio_nr_pages(src); - - for (;;) { - copy_highpage(folio_page(dst, i), folio_page(src, i)); - if (++i =3D=3D nr) - break; - cond_resched(); - } + copy_highpages(folio_page(dst, 0), folio_page(src, 0), + folio_nr_pages(src)); } EXPORT_SYMBOL(folio_copy); =20 int folio_mc_copy(struct folio *dst, struct folio *src) { - long nr =3D folio_nr_pages(src); - long i =3D 0; - - for (;;) { - if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i))) - return -EHWPOISON; - if (++i =3D=3D nr) - break; - cond_resched(); - } - - return 0; + return copy_mc_highpages(folio_page(dst, 0), folio_page(src, 0), + folio_nr_pages(src)); } EXPORT_SYMBOL(folio_mc_copy); =20 --=20 2.43.0