From nobody Thu Apr 2 17:16:01 2026 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012065.outbound.protection.outlook.com [40.93.195.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27ED5198A17 for ; Mon, 9 Mar 2026 12:11:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773058306; cv=fail; b=k4Zf19r/T2T/SaEi6EyDPM1L8aYMFtKS1uxZOWkcmz4v3FOwMU06TNDLDmG4fKZLfbKEoCsi5k34BVAAlPJKCq9WOvyPclysODh+w4CUmdaEReneFZj/zfiNea3SbGl612u6nMK5kXiLub+SdsidQzZJjIRkpHXcMyu6YnAH1Wc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773058306; c=relaxed/simple; bh=OcG5Kj3FIs7Ju1GwoCIzjxl09LcMFldVP70Ok2qJ0VM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ttm1C4f9vzdhshvX6VS/P7FoZFgpI0A0gC16uI4Hd79+2wwI/C1dqSxMl6xRYEqh7VgfjB8L335OtjFCU1RNvHFKovZEDcjKiiCcdUJz9JBGN8Wyh0xy282YUekZya3gZ5tnxOtL29E93Hc1jCy51Qw8OFaJ1n/XedKDicxtFa0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=zfeoRhtg; arc=fail smtp.client-ip=40.93.195.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="zfeoRhtg" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=S1GoyGaycwwWFwTMdEL6X3UlJ8MDOamluT6IpWHIQJ8IKnFD7fGqoBB5+8U25HBKWj6F6uf+LYgHhT30EQU+7KNpHQ6vYZ5DASuw99Pa8RaK+md1vIf1wQhoQy1nnga2E7Tuj/iYz33fLMpcjwqIGIOIr1uf7KIk2F2P0624bhjGbm3qu9bWwkDFPBYmg4lL/gnZZ70NMakE4X9MMlz3nzQrbB7lWssmkD4SAoiO+AqlOxu9ilpeW+YKNTJiHXyhnYfgw4Dadtf9xsf9XeAKPsSdHM8sOIyO0GvcGLu+z8q9hkjU+6vPdEhuDqet25OJILVNoK24ySOYc3XBHETzhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kOsJznQvzpPNkoFq6ElzZE5tlcE0D+t5P1bqJv1kvJM=; b=dpQR7oFpTPB6RWuxLA7azEP01KaLUX5HIpfdNtrHeFW1KTpfmPiZAhf7/D4rM6r+z/ZNDvZ4gLSJU488SPKtovSd/OG8amv1W3p8YT9OO/SeEfhqeTt5OstfqsKpXnNb4wkYvu8lED7KLfSPkSSkmnKnUrU5xz0hOOz6qLqn5JK8J4b7zgbMic3svalrdMnYd4+nYaOcN1btv958sXjnK0hE11dyQTvidiOZDp1h/Sp3UPMTRP0T/vXvyRNYFpSkRDt8hVQfw9DR70rFys0H0BzJY3csNivUXOxImFW4g95bCny9f7vS9dhpoAW4tq0o/xoxFMCYMFFCp2ekti3mSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kOsJznQvzpPNkoFq6ElzZE5tlcE0D+t5P1bqJv1kvJM=; b=zfeoRhtgtY8SVeG34SK7FkEHLwXyIYx0chsCwh85VSeqCl+jmQlMod/qfiwIacSxPIKt0nIPYGui2ARiUKWU4ZytfbPSD79dUUjPTGin/WrqG6QdKyZx/zXuluZqQxl1X/ThQNb5usZELrgeOLp5AhHP4+IDHekv841lxrv0UBA= Received: from MN0P223CA0029.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:52b::18) by EAYPR12MB999155.namprd12.prod.outlook.com (2603:10b6:303:2be::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9700.11; Mon, 9 Mar 2026 12:11:37 +0000 Received: from BL6PEPF0001AB51.namprd04.prod.outlook.com (2603:10b6:208:52b:cafe::b2) by MN0P223CA0029.outlook.office365.com (2603:10b6:208:52b::18) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9678.25 via Frontend Transport; Mon, 9 Mar 2026 12:11:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by BL6PEPF0001AB51.mail.protection.outlook.com (10.167.242.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9678.18 via Frontend Transport; Mon, 9 Mar 2026 12:11:36 +0000 Received: from satlexmb10.amd.com (10.181.42.219) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 9 Mar 2026 07:11:36 -0500 Received: from kaveri.amd.com (10.180.168.240) by satlexmb10.amd.com (10.181.42.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 9 Mar 2026 07:11:25 -0500 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Shivank Garg , Mike Day Subject: [RFC PATCH v4 4/6] mm/migrate: add copy offload registration infrastructure Date: Mon, 9 Mar 2026 12:07:29 +0000 Message-ID: <20260309120725.308854-12-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260309120725.308854-3-shivankg@amd.com> References: <20260309120725.308854-3-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb10.amd.com (10.181.42.219) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB51:EE_|EAYPR12MB999155:EE_ X-MS-Office365-Filtering-Correlation-Id: a6f34e71-09ed-426b-6691-08de7dd5033b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700016; X-Microsoft-Antispam-Message-Info: R1uQwdUPokr4PltjlEmsMxyYP/NHWwpN+ME7k7Zyy+R6AJ/fQR2Jpda6fdESb8LV2ddRGA1jIdE3YJ84xfmJOrjLuAXSe8ce+YD8IqhLm/po/shpQTL6yVgzqtqNQcg3bNAt2AINLvGPE0r5SQaAoi/+48W52bQE7zFO/4bbgc8nE2GZZPp6pX2X2vmvnoGQId9A9QSohial0Vmw01jelVZhw3NojyBzuua9x9ipDrFGsTFs/Bhc35mHQiky0GYLcNJZ1bwUE9/hIfvF0L1JjYBGqXeOFUJNde5SWNNbVUVD8lox8kHlYcuATc6uJJmZQ4FstdkcWGYwtm+UY7oxUSQKM+X+0WjNxSpxKPcLOExTbcJmIEnKfEGja07AmO9zK6hq1sQ8+bmA9bh/yxONQZbiSkOi5YYjNU/gwJE3/ikO/XsBFiHuivPLCTcnzymJDVwoFosiVnH3zJx0pEenaenwCjhmMJdTdHdywbntrLW+xR26K0X2MLBUiWQ6OaotmD0SMS/XVUheh8K0yotS13RjOW9V2RkUJtFtW0zOnPF4f8hFlpc/K4UcAIccuJQe3zaIuVoWrLW1/P4yYZVZ9thQe8PfSHcIjmIkjhs2gdS9cmt5E8lxaO5Gb+j0BRxrhY0s38rZS8tQyAbLdVVo366NJwywON6v1TcTVA+w39IRtaoFCMDZGcLJ8N5gWDWCCHnWhRFC4Zf4woSC6tI9KggXTlw+SFQyyrgD9vPXsLZualo8g1cVMqxgfL70aY5utZdYc3lq5kiT3MSs2nwYKg== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IoZNU/j4gAQ5fquyp+lUJ2b16A4D0RmN7FWF8l8JajLPL7e8OYCGKaHSPcyyhNMqzaMWmNVReczyTvH9rgOqAVt6MuiJFLUZJ9L4gZDF3qekpHzFTcmDMU4gVyVMKefsRIc8ARTjonHKWliAkcdSqGxmh/o539mfeBYwXYZPVKN+CV4nfCFeO3CO/2lFAC7q16uaPxakZft1Dgv0mlNp6QDdNete5opkUH0Sbub9WhVcz/Uen8hncVB9K9Hnvp1B9y84RWXB1Kkk/ppwwKP4FoNXvp4R6T8k26VT/Q0Z6X2xsD15dOBA+0BBpSHOsMhbgKsbYT4ME7jA9gbqlSBIMk4V7XmhwzSLCSDfhQ9qilKy+uYNgdwgZrG+02b0rLTrS8wgs0Eg1XRcznnZhUVAnKRWW4hERtmbnfWVmV85XCHLPCxr6M+/vnRj69IyD+WC X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2026 12:11:36.8487 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a6f34e71-09ed-426b-6691-08de7dd5033b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB51.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: EAYPR12MB999155 Content-Type: text/plain; charset="utf-8" Introduce CONFIG_MIGRATION_COPY_OFFLOAD, which lets offload driver (DMA, multi-threaded CPU copy, etc) take over the batch folio copy in migrate_pages_batch(). Offload driver fill in a struct migrator with their offload_copy() and should_batch() implementation and call migrate_offload_start(), which patches the migrate_offload_copy() static_call and flips the migrate_offload_enabled static branch. The migrate_offload_stop() call reverts both. Only one migrator can be active a time. A second registration returns -EBUSY, and only the active migrator can stop itself. The static_call dispatch is under SRCU so synchronize_srcu() in stop path guarantees no in-flight copy before the module reference is dropped. Co-developed-by: Mike Day Signed-off-by: Mike Day Signed-off-by: Shivank Garg --- include/linux/migrate_copy_offload.h | 34 ++++++++++ mm/Kconfig | 9 +++ mm/Makefile | 1 + mm/migrate.c | 30 ++++++++- mm/migrate_copy_offload.c | 99 ++++++++++++++++++++++++++++ 5 files changed, 171 insertions(+), 2 deletions(-) create mode 100644 include/linux/migrate_copy_offload.h create mode 100644 mm/migrate_copy_offload.c diff --git a/include/linux/migrate_copy_offload.h b/include/linux/migrate_c= opy_offload.h new file mode 100644 index 000000000000..ee112826ebdf --- /dev/null +++ b/include/linux/migrate_copy_offload.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MIGRATE_COPY_OFFLOAD_H +#define _LINUX_MIGRATE_COPY_OFFLOAD_H + +#include +#include +#include + +struct list_head; +struct module; + +#define MIGRATOR_NAME_LEN 32 + +struct migrator { + char name[MIGRATOR_NAME_LEN]; + int (*offload_copy)(struct list_head *dst_list, + struct list_head *src_list, + unsigned int folio_cnt); + bool (*should_batch)(int reason); + struct module *owner; +}; + +#ifdef CONFIG_MIGRATION_COPY_OFFLOAD +extern struct static_key_false migrate_offload_enabled; +extern struct srcu_struct migrate_offload_srcu; +bool migrate_should_batch_default(int reason); +int migrate_offload_start(struct migrator *m); +int migrate_offload_stop(struct migrator *m); +#else +static inline int migrate_offload_start(struct migrator *m) { return 0; } +static inline int migrate_offload_stop(struct migrator *m) { return 0; } +#endif /* CONFIG_MIGRATION_COPY_OFFLOAD */ + +#endif /* _LINUX_MIGRATE_COPY_OFFLOAD_H */ diff --git a/mm/Kconfig b/mm/Kconfig index ebd8ea353687..faf0cae9991b 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -648,6 +648,15 @@ config MIGRATION config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE =20 +config MIGRATION_COPY_OFFLOAD + bool "Page migration copy offload" + depends on MIGRATION + help + Adds migration copy offload infrastructure which allow + offload engines (DMA, multi-threaded CPU copy, etc.) to + register as the batch-copy provider for page migration + via migrate_offload_start()/migrate_offload_stop(). + config ARCH_ENABLE_HUGEPAGE_MIGRATION bool =20 diff --git a/mm/Makefile b/mm/Makefile index 8ad2ab08244e..db1ac8097089 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) +=3D failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) +=3D fail_page_alloc.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o +obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) +=3D migrate_copy_offload.o obj-$(CONFIG_NUMA) +=3D memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o diff --git a/mm/migrate.c b/mm/migrate.c index 69daa16f9cf3..acaaa9cc0d4f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -44,6 +44,8 @@ #include #include #include +#include +#include =20 #include =20 @@ -54,6 +56,17 @@ =20 DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled); =20 +#ifdef CONFIG_MIGRATION_COPY_OFFLOAD +DEFINE_SRCU(migrate_offload_srcu); +DEFINE_STATIC_CALL(migrate_offload_copy, folios_mc_copy); + +bool migrate_should_batch_default(int reason) +{ + return false; +} +DEFINE_STATIC_CALL(migrate_should_batch, migrate_should_batch_default); +#endif + static const struct movable_operations *offline_movable_ops; static const struct movable_operations *zsmalloc_movable_ops; =20 @@ -1820,11 +1833,18 @@ static int migrate_pages_batch(struct list_head *fr= om, LIST_HEAD(dst_batch); LIST_HEAD(src_std); LIST_HEAD(dst_std); + bool do_batch =3D false; bool nosplit =3D (reason =3D=3D MR_NUMA_MISPLACED); =20 VM_WARN_ON_ONCE(mode !=3D MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); =20 +#ifdef CONFIG_MIGRATION_COPY_OFFLOAD + /* Check if the offload driver wants to batch for this reason */ + if (static_branch_unlikely(&migrate_offload_enabled)) + do_batch =3D static_call(migrate_should_batch)(reason); +#endif + for (pass =3D 0; pass < nr_pass && retry; pass++) { retry =3D 0; thp_retry =3D 0; @@ -1967,7 +1987,7 @@ static int migrate_pages_batch(struct list_head *from, break; case 0: if (static_branch_unlikely(&migrate_offload_enabled) && - folio_supports_batch_copy(folio)) { + do_batch && folio_supports_batch_copy(folio)) { list_move_tail(&folio->lru, &src_batch); list_add_tail(&dst->lru, &dst_batch); nr_batch++; @@ -1997,11 +2017,17 @@ static int migrate_pages_batch(struct list_head *fr= om, /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); =20 +#ifdef CONFIG_MIGRATION_COPY_OFFLOAD /* Batch-copy eligible folios before the move phase */ if (!list_empty(&src_batch)) { - rc =3D folios_mc_copy(&dst_batch, &src_batch, nr_batch); + int idx =3D srcu_read_lock(&migrate_offload_srcu); + + rc =3D static_call(migrate_offload_copy)(&dst_batch, + &src_batch, nr_batch); + srcu_read_unlock(&migrate_offload_srcu, idx); batch_copied =3D (rc =3D=3D 0); } +#endif =20 retry =3D 1; for (pass =3D 0; pass < nr_pass && retry; pass++) { diff --git a/mm/migrate_copy_offload.c b/mm/migrate_copy_offload.c new file mode 100644 index 000000000000..c22068fe09a0 --- /dev/null +++ b/mm/migrate_copy_offload.c @@ -0,0 +1,99 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include + +static DEFINE_MUTEX(migrator_mutex); +static struct migrator *active_migrator; + +DECLARE_STATIC_CALL(migrate_offload_copy, folios_mc_copy); +DECLARE_STATIC_CALL(migrate_should_batch, migrate_should_batch_default); + +/** + * migrate_offload_start - register a batch-copy provider for page migrati= on. + * @m: migrator to install. + * + * Only one provider can be active at a time, returns -EBUSY if another mi= grator + * is already registered. + * + * Return: 0 on success, negative errno on failure. + */ +int migrate_offload_start(struct migrator *m) +{ + int ret =3D 0; + + if (!m || !m->offload_copy) + return -EINVAL; + + mutex_lock(&migrator_mutex); + if (active_migrator) { + ret =3D -EBUSY; + goto unlock; + } + + if (m->owner && !try_module_get(m->owner)) { + ret =3D -ENODEV; + goto unlock; + } + + static_call_update(migrate_offload_copy, m->offload_copy); + static_call_update(migrate_should_batch, + m->should_batch ? m->should_batch : migrate_should_batch_default); + active_migrator =3D m; + static_branch_enable(&migrate_offload_enabled); + +unlock: + mutex_unlock(&migrator_mutex); + + if (ret) + pr_err("migrate_offload: %s: failed to register (%d)\n", + m->name, ret); + else + pr_info("migrate_offload: enabled by %s\n", m->name); + return ret; +} +EXPORT_SYMBOL_GPL(migrate_offload_start); + +/** + * migrate_offload_stop - unregister the active batch-copy provider. + * @m: migrator to remove (must be the currently active one). + * + * Reverts static_call targets and waits for SRCU grace period so that + * no in-flight migration is still calling the driver functions before + * releasing the module. + * + * Return: 0 on success, negative errno on failure. + */ +int migrate_offload_stop(struct migrator *m) +{ + struct module *owner; + + mutex_lock(&migrator_mutex); + if (active_migrator !=3D m) { + mutex_unlock(&migrator_mutex); + return -EINVAL; + } + + /* + * Disable the static branch first so new migrate_pages_batch calls + * won't enter the batch copy path. + */ + static_branch_disable(&migrate_offload_enabled); + static_call_update(migrate_offload_copy, folios_mc_copy); + static_call_update(migrate_should_batch, migrate_should_batch_default); + owner =3D active_migrator->owner; + active_migrator =3D NULL; + mutex_unlock(&migrator_mutex); + + /* Wait for all in-flight callers to finish before module_put(). */ + synchronize_srcu(&migrate_offload_srcu); + if (owner) + module_put(owner); + + pr_info("migrate_offload: disabled by %s\n", m->name); + return 0; +} +EXPORT_SYMBOL_GPL(migrate_offload_stop); --=20 2.43.0