From nobody Thu Oct 2 02:13:30 2025 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010063.outbound.protection.outlook.com [40.93.198.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6AFE9284B25 for ; Tue, 23 Sep 2025 17:48:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649724; cv=fail; b=Lnqb2mmv7mHEfvjweY34XqorxY8jElI1hQq2l/v8aPjX6fkeMDAFhzHGQQD8//+Ke/h4lOo5+/DDvGTZcHauFKHWe4K5MGmFu330kVXZHyXp1xQDW/O7OBFYDAnBoMT3SNKKpb4gOu1++NNAnz3xguplhUtNreN9X5B6vlHLgdw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649724; c=relaxed/simple; bh=NWFASIMQpnXhbyOjQB7sqvAq6pClKFwXQwP8yXTsR0g=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=jSHS41hQT0frTZ4zQk9mQ41Fg7XlkX0dLesd7eMApM4SU1bl2QcFdvKcsLcUnRzSI8sbI56tKcPzRmpz3ueTJEgw4g7hLEiDpaTGO1cdBtVkxHT64hptMZ0jZRHVWV5oms6Kt6Xx9SeZy5tGlx7PIbumPNljEudQbHHIlmBexLc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=fY5cveic; arc=fail smtp.client-ip=40.93.198.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="fY5cveic" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=okWvd62QOAduesI5W7Is5Ly3i2hCzQkIZEldLMY3wMn7wlo60KW5/ygNoqgm7oxdfSSW14GDm7jWbg1FsHZ8y0nu6DVl0pJSEbrPChEmkXJfy2EJLcx4hbMv4Tckq9NfKiQwTDr73wahmlgCbQHAaB8YMuvRmQVH1zGfTd3nfZAoYN0psf6tAcL9v/XZF7NBGapbFxUHtkDQqZeJCANIDs/RdzjgBnqjvQrA4a8Q/D93KIhPpU9IcbmTvv2qvDwQfyCBvTLqJOCFkf/MD/xpBY4cT0X97gTuhNcce5gk5cm+Q+FM4lBljaOHfZpM4sfHW7OCYKrVC9znkxm7wHaa7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=c/XSA7C0+VZv1xAdsgXYAZVX5Wrmi6R5gU5c9V8a6tk=; b=Yz8JmmtZ7kiuAPxG2b35No9kfDn+J3QaGSqhZTGloPo+G+Q1zufbgtcMBWBSzktFOZ94TLKzX4BjxxfM4TVaCjX7LpqwYL1eSTYuQYYPOjMYPeJXkoxitWY5NXVB2I+9ExFKAnP5DryTEKXb1YJ+08lQatv2JKx+qg0FAIZuwfApFm136wpndx/WDIVzwYkId46NVet7aGhNu8/h8JkutifbDDaMfvyZ+K7s3z8TijzGcD2bUt04Oymr69Di1pKZVcSJ77IDN4YJSzeGz+UCrO46ZOuZfnoN7aaF777fIdlAfkXzZuDRo6sf1kKhN50vBnor33PPm9UnKAxCdhjUMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c/XSA7C0+VZv1xAdsgXYAZVX5Wrmi6R5gU5c9V8a6tk=; b=fY5cveicdNW0tfUsWMHYDWGWt8Ot3wKGJmEgQMiPiXYTzgCxWSKwE/55ENlLLkNgo/n8nkbzc0yez2wzZcGRyR5tkPjREPCZ8l5qIdsHyNyXwbO+cMxid7rGQQiNU3sSz9C6znBbp+oKarehOBLSn3o8ScQ/mCQeom60+i+kkiY= Received: from MN0PR05CA0005.namprd05.prod.outlook.com (2603:10b6:208:52c::20) by LV3PR12MB9168.namprd12.prod.outlook.com (2603:10b6:408:19a::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.19; Tue, 23 Sep 2025 17:48:39 +0000 Received: from MN1PEPF0000F0E2.namprd04.prod.outlook.com (2603:10b6:208:52c:cafe::50) by MN0PR05CA0005.outlook.office365.com (2603:10b6:208:52c::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Tue, 23 Sep 2025 17:48:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E2.mail.protection.outlook.com (10.167.242.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:48:39 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:48:30 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 1/9] mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move() Date: Tue, 23 Sep 2025 17:47:36 +0000 Message-ID: <20250923174752.35701-2-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E2:EE_|LV3PR12MB9168:EE_ X-MS-Office365-Filtering-Correlation-Id: cf309ae3-de53-4fd0-8253-08ddfac96dce X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?1LbizjOFSdnI1QfLxADwhmq90ITfTCdaPsWhKkxDxp4DM9wF9nw8QjdQEIjY?= =?us-ascii?Q?+0DcBk/R8US0YHkgnZsFOXAVaxOnM9gOOE5gnw5sfYSFdj+0+E3/h+HZUU+U?= =?us-ascii?Q?4zIje0y8s3DdhfQe8y4Bi/TvfBFB/azoKvriea15iOkG3glr+TP4t9tbvszK?= =?us-ascii?Q?yl+3n8I7xMc51sfAWLBmEawlcLYEKkAInPT0G4f19T3r4W4HLaGf6vqr+y+i?= =?us-ascii?Q?gsWPtbB7RxlLkJZ1lS25xXZ53Y54UYoDTE6ZrlTJF1A0rbYKcxR0Ga8QFYkR?= =?us-ascii?Q?g/pm/Dw6q9ZNJLVm8RcHhYWFDbGlcyk5Z2GavXqquomFbO1yFGtsSrjlvgxl?= =?us-ascii?Q?AJ5WefOiLrPyTWW2F79FCAEf6ieXi2M4U/S8o3NK9iE8DSTRQ/OWI4yid5A0?= =?us-ascii?Q?J75ZlJjhr0Ngj991xUAJjDmFteH+jtz06j+nzkeJjuxblA1yxPM9iY3D6mXc?= =?us-ascii?Q?u6a1jJqWamCUFYqYTVCLfZRb+Q3qd4yfJREaVCZj0uK0SM/6xQYQxvYDyWC/?= =?us-ascii?Q?8bwGG56wiFhJ0ozqquoqeyMRrM88IOkey9OQsV7fClN6Gjqlu0/VXYtQpDTp?= =?us-ascii?Q?pM0ephTW2boHrMA19x/6ijNgYDBv7gYZrAsL1D8dlqcqaacbvsB0en5blmqE?= =?us-ascii?Q?yaBrie9dIN3tJQXPFhOLq3jBomHfqZShwx8iUX1zkiiVrsr3Fas8JoIz/bGB?= =?us-ascii?Q?wPWP/jZt5Y5K4QALj1SKAALILJI0/rgYbPo0w0twIa62sanVaLYD0+12sHV6?= =?us-ascii?Q?GuB1SMB9BelXN4Ydb+enlw03hgRbcWc/idTFxybKmbODvyhX75ItraAR7amf?= =?us-ascii?Q?leMwZ9BSRwiTVEXTwZlrzCHMTPgLlooWiq047WYD94R2avXACv4NW6MKxm5V?= =?us-ascii?Q?LoL1cQz2oPZU9aG+EduIYwR3irCWaMPNVVcyMOY+V7/RNLPEJscIiKLxpDbF?= =?us-ascii?Q?W6iqbmBvTdMOyFQG86OWxsb/7CWqw+q69s+FZcxCpiswvYS9shP45CGWV4M5?= =?us-ascii?Q?YkooxLbusZ9IffZJB8BCM2szb6MPYlgOFSGsqLa/WnVR8zKbx/iOPw1K7YEw?= =?us-ascii?Q?EepAEwB9J6ZXXDNNCssP9VaoFQTAfSwlslUYYR80cxh2WmUsC2BfqD6XxOlt?= =?us-ascii?Q?a74Rd43Kt0hcX4+9+J8TAs6Xjx219k9PwddpP4G+LAEoP2M9AKr8Vz80NsDD?= =?us-ascii?Q?7QbJstUeb5chuo928mfkIVy/UHw79L9ZQa5lwU+ek7i+Cmehp3iRBfc58/kQ?= =?us-ascii?Q?bMs+iDHbpqQA/NfIRFO+LXW2kXqlq4G4u0av+WfhBy5g26YULmyt6BKN5QaS?= =?us-ascii?Q?ylcZy/FyL9s+PYzzu5Pm9G0IeYdfE2sDlyqFNXAiFKeTPGY/qW7I2J0iDWug?= =?us-ascii?Q?yT0IKheAU4yf8U+A0X8minFgj3lkLa8t9BqzQWFwUVVMCGn7ajbbgiEp/si1?= =?us-ascii?Q?DrVL0RlJ7Z5cy1fvUq96iRPhZQau7QpiMWLl6l4jAXUE9EUuyYjJTGdMA867?= =?us-ascii?Q?2O2ufwk0MYNiRjKjS7VYifZMbihcWdLtSIgR?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:48:39.3667 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cf309ae3-de53-4fd0-8253-08ddfac96dce X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E2.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9168 Content-Type: text/plain; charset="utf-8" From: Zi Yan No function change is intended. The factored out code will be reused in an upcoming batched folio move function. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- mm/migrate.c | 106 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 67 insertions(+), 39 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 9e5ef39ce73a..ad03e7257847 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1061,19 +1061,7 @@ static int fallback_migrate_folio(struct address_spa= ce *mapping, return migrate_folio(mapping, dst, src, mode); } =20 -/* - * Move a src folio to a newly allocated dst folio. - * - * The src and dst folios are locked and the src folios was unmapped from - * the page tables. - * - * On success, the src folio was replaced by the dst folio. - * - * Return value: - * < 0 - error code - * MIGRATEPAGE_SUCCESS - success - */ -static int move_to_new_folio(struct folio *dst, struct folio *src, +static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, enum migrate_mode mode) { struct address_space *mapping =3D folio_mapping(src); @@ -1098,7 +1086,12 @@ static int move_to_new_folio(struct folio *dst, stru= ct folio *src, mode); else rc =3D fallback_migrate_folio(mapping, dst, src, mode); + return rc; +} =20 +static void _move_to_new_folio_finalize(struct folio *dst, struct folio *s= rc, + int rc) +{ if (rc =3D=3D MIGRATEPAGE_SUCCESS) { /* * For pagecache folios, src->mapping must be cleared before src @@ -1110,6 +1103,29 @@ static int move_to_new_folio(struct folio *dst, stru= ct folio *src, if (likely(!folio_is_zone_device(dst))) flush_dcache_folio(dst); } +} + +/* + * Move a src folio to a newly allocated dst folio. + * + * The src and dst folios are locked and the src folios was unmapped from + * the page tables. + * + * On success, the src folio was replaced by the dst folio. + * + * Return value: + * < 0 - error code + * MIGRATEPAGE_SUCCESS - success + */ +static int move_to_new_folio(struct folio *dst, struct folio *src, + enum migrate_mode mode) +{ + int rc; + + rc =3D _move_to_new_folio_prep(dst, src, mode); + + _move_to_new_folio_finalize(dst, src, rc); + return rc; } =20 @@ -1345,32 +1361,9 @@ static int migrate_folio_unmap(new_folio_t get_new_f= olio, return rc; } =20 -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_folio_t put_new_folio, unsigned long pr= ivate, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) +static void _migrate_folio_move_finalize1(struct folio *src, struct folio = *dst, + int old_page_state) { - int rc; - int old_page_state =3D 0; - struct anon_vma *anon_vma =3D NULL; - struct list_head *prev; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - prev =3D dst->lru.prev; - list_del(&dst->lru); - - if (unlikely(page_has_movable_ops(&src->page))) { - rc =3D migrate_movable_ops_page(&dst->page, &src->page, mode); - if (rc) - goto out; - goto out_unlock_both; - } - - rc =3D move_to_new_folio(dst, src, mode); - if (rc) - goto out; - /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1386,8 +1379,12 @@ static int migrate_folio_move(free_folio_t put_new_f= olio, unsigned long private, =20 if (old_page_state & PAGE_WAS_MAPPED) remove_migration_ptes(src, dst, 0); +} =20 -out_unlock_both: +static void _migrate_folio_move_finalize2(struct folio *src, struct folio = *dst, + enum migrate_reason reason, + struct anon_vma *anon_vma) +{ folio_unlock(dst); folio_set_owner_migrate_reason(dst, reason); /* @@ -1407,6 +1404,37 @@ static int migrate_folio_move(free_folio_t put_new_f= olio, unsigned long private, put_anon_vma(anon_vma); folio_unlock(src); migrate_folio_done(src, reason); +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_folio_t put_new_folio, unsigned long pr= ivate, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + int old_page_state =3D 0; + struct anon_vma *anon_vma =3D NULL; + struct list_head *prev; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + prev =3D dst->lru.prev; + list_del(&dst->lru); + + if (unlikely(page_has_movable_ops(&src->page))) { + rc =3D migrate_movable_ops_page(&dst->page, &src->page, mode); + if (rc) + goto out; + goto out_unlock_both; + } + + rc =3D move_to_new_folio(dst, src, mode); + if (rc) + goto out; + + _migrate_folio_move_finalize1(src, dst, old_page_state); +out_unlock_both: + _migrate_folio_move_finalize2(src, dst, reason, anon_vma); =20 return rc; out: --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012049.outbound.protection.outlook.com [52.101.53.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 384E1189 for ; Tue, 23 Sep 2025 17:48:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649737; cv=fail; b=QEORNexmQb3tw/h8V/y3TGyhD4IXK/5zh1brnm83GsdGD15qVVGaoKlcWFCirTTKdZC28MnmvIWZzmrbXtRcuIQ1ZvK6FoIZ1SL9TzKDn8tD3mEuCKJaM+ziyimolqZI2ypDdWRsnakpEocH7AUQ/m7brQlyWvsltvTm0Q25DAk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649737; c=relaxed/simple; bh=q85A8m++X7UCkOmg/9rSaiyLUssj1Fj/XBNDBAms4H4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=I4ZFZ24SGlzpNFnJhw6GnCE4wu8iEcRPt/gOJ0UyCsRDUulIfQc6nbJ3x5zQvdhCkjO6tFE89onYY2irFeeeKT1XRQlOkjwTd0fZvlhcijYPU7qPE9BTI29qAfDhuraelkTaZDYDoKR91LE0ZstCjMOg+IzexHuJGIkLK24uYRY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=N7zs602E; arc=fail smtp.client-ip=52.101.53.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="N7zs602E" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ichqhSRUk/9jgrTaaO+EsSETkHxI4I0ekmQTFLO5KfLHPX0A3QszGQ59hDqnUK0utyrqJMcj9UC+0RktqgdnspCi3tkrKeZ+Rl/TS4mJzLsi3PpXgIxNU74FkZIxal9pmxcSrDiFTAKXBch9jL1+I6FoTL0fUCfNvrUcY9BRjgPj6HO6coQND/AMEAFBRHKjUglpRaGqVLSXx3mY7RngGkACRWQydTprP3h4ASKLjHYvTdBJGP0lCDUkfFbQdRDtKyDKjxgNn+e7QuunFQePKpBge7hoekg9hUmx3RA4le0e0cBPyv3gUDqNqtByOhSNGoBL0fut/wd6vitAywl/dg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=juCl6Q7IDVVaEwDIPOfsLFRmleBkegBnFU+fvOlhQWI=; b=LGkWTB6hE6bbBz5WH/wg7U7iN9zHbf01LyVV6Yj6c8u++flgNsxpddUDUxzEu5b2+Sn0hYGEXeMqKlwIzYfQNmeSXkreVnFUTFY4qTBlAM9qtBYNpurRlvGn/b5PE73oNOxILKT28mO3d3VUoWEN5n/fW+UmeVLHhCx+p1se4aQTNz5CaedmLW0XzStSnuxMpdzHhIdt/RUorNi9uz2nyzMUc3NF2fcPi+MjmDkUVqTWGuB85goA9LBlG/FdjWNd74lMfI3GHy3HZgSx1AeJURyj6r59iI7566QhrR6bB1sfr8I3NoKUi6TOQ/GuqTod8k6zHB9fnen8zbPHMYDaIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=juCl6Q7IDVVaEwDIPOfsLFRmleBkegBnFU+fvOlhQWI=; b=N7zs602EzuuGAFYsykzECl0u7WOQ3fR9RqfAKqDWKrGIct/wKmUeMQHIwLXJoVRpLaMgiQTh8q+0ugcukAtixdS308kI3VB6QH4nzzhQI0RHmi7Al1PwPbQNEKeYAKFfMJt8d4WeS6noa+VK2GtJD/pLbRpYUBVwoqA4rCCpsnY= Received: from BN0PR04CA0200.namprd04.prod.outlook.com (2603:10b6:408:e9::25) by MW4PR12MB7120.namprd12.prod.outlook.com (2603:10b6:303:222::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.19; Tue, 23 Sep 2025 17:48:49 +0000 Received: from MN1PEPF0000F0E4.namprd04.prod.outlook.com (2603:10b6:408:e9:cafe::9a) by BN0PR04CA0200.outlook.office365.com (2603:10b6:408:e9::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9137.21 via Frontend Transport; Tue, 23 Sep 2025 17:48:47 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E4.mail.protection.outlook.com (10.167.242.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:48:47 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:48:38 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 2/9] mm/migrate: revive MIGRATE_NO_COPY in migrate_mode Date: Tue, 23 Sep 2025 17:47:37 +0000 Message-ID: <20250923174752.35701-3-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E4:EE_|MW4PR12MB7120:EE_ X-MS-Office365-Filtering-Correlation-Id: f9018db0-0233-4c05-ad8d-08ddfac972bc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|82310400026|36860700013|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?kYDyYId8ciugnAHuyfe+TRfbG2RWK8Hj/BCha2RFo6vjZpRD/9NLvGo1lCJ6?= =?us-ascii?Q?PcobJtxMKRYb4o1Ng7ZiC/rXwlef+K//3+P5Z6/HjDhzow/IdV/smfOvn/tE?= =?us-ascii?Q?8xCqyll/K8j2wVDDTnACOlv51fNVJTI3/S3aePvUOIrPVG5ckENjVHVzuIr1?= =?us-ascii?Q?kgiimBYMyBpgUIUhW/vGQQAeAE1qFNCZLmpUEW280QGzkYFo0SSSDFFxf2rT?= =?us-ascii?Q?H2iWfEUVA5/Bi9t4Ycov3Q55uaiXUkGMxjTFzL9iZSigzWTwu6slYl3IpDnT?= =?us-ascii?Q?cqh50bP++fH1LfMDF2kT2R+m6tL82MWNfwlF2N4xJb673rZlEA9c13mTiUDQ?= =?us-ascii?Q?NRJQCdcnTi+1GB8a6rdI6AvmChvK35MLtaCMz7d/zh6r328g2MOreTVcvotb?= =?us-ascii?Q?9tnxEyuWPybDvREDSvshX3IcYzIUFScPYpxMyuSXz89/RPLOz/VgxT4Dlibk?= =?us-ascii?Q?QSMLWSkZBEOh8P8XB8YV4myZ/8SjNhCa8hlr9IRFKfZ68YGhD9+ecRW26dJ1?= =?us-ascii?Q?Y1n4VZ15HfLYQVw8VQZfBV5WKF+s27BanofHRUfwusJlFwsLgIfgR+R9l97g?= =?us-ascii?Q?9JFNS/IBR4m25BffNlYlF076RqcsCWFlwYtL8O/5AkBiFTRujdl5ecWnrs2L?= =?us-ascii?Q?hxjh/KVNVcs7SNLbt9GhGaFelWDL4TV9RBTgi5I76VzRiFaI5ErrQBrIZLXV?= =?us-ascii?Q?h3pQQ9S7J4mQNGjxVv7MJFhCbLNpdaW6riEDwwf0107hFRrtu+1j3ScrclDc?= =?us-ascii?Q?35w+9qaxtsye7vTIVPHNBlK6mIFMZBGKzy3mw4DbRzel/cYwi9kBdgBWZNN2?= =?us-ascii?Q?4D44g/LTUGI3359us0yooQu9lGyxzvcppwR0Nyg5x6gUYGWyzpgPIJFngwI+?= =?us-ascii?Q?OlW1LwX3I6wgs62qeG06VOK+WLaN/mg3EqGsA9w0Li/Isl0/fLWeAGpdu+9+?= =?us-ascii?Q?GuD0e09lVTlYl6FSqgLzQzhe34EaNqlb+B/G2Bj1saM3Bt5T43MXSGZr4SGb?= =?us-ascii?Q?irtgNBQ28IJbH9RmOMJ4dN0jVGyZ1gYOG7A4H6zlHluNiOTv24o71T6HrqNS?= =?us-ascii?Q?a0q08lrAIDQ3t1xuixSE2AKmbhM68v9vVI2G3WAhI1ibVo7L1S9zcplw5im2?= =?us-ascii?Q?lWqg5uvw4VqXlgASFxZkZ7hPzeSnOVVRw3jGToOUHuLtTE0Yva3+aoQviBcA?= =?us-ascii?Q?k5WcjNLohfQmwciOih+YqDLDhZGB/X00J8BXaQarRoFQy/lComgcfMDUy9ig?= =?us-ascii?Q?2SNzswggzF3JAG5b9SyBaPF9BJ2OuQcPWPuQBs3oGmhZvqFllNmWKr9mfmBj?= =?us-ascii?Q?XDm7W6B/FhLAUfJ2hACodwfFZnoPxxbTY9rg/TNTKCIzMqlBUU/OYqrqf8NZ?= =?us-ascii?Q?q65CyxnIKBnlDLTDHdvruAPqVQOszyKd0xGwuBURRQ+aoZ5NYTr9gRkXA4Yd?= =?us-ascii?Q?XL/TzJDOo/eySCsQP94+SO0qGxdd3SCgZ3vwhlVOvwiuuO79ORtF+uCCgmrS?= =?us-ascii?Q?XPKG4cOp6U61ck3eBuOli9nunLM+VWrlmc88?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(82310400026)(36860700013)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:48:47.6388 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f9018db0-0233-4c05-ad8d-08ddfac972bc X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E4.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7120 Content-Type: text/plain; charset="utf-8" From: Zi Yan It is a preparation patch. The added MIGRATE_NO_COPY will be used by the following patches to implement batched page copy functions by skipping folio copy process in __migrate_folio() and copying folios in one shot at the end. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- include/linux/migrate_mode.h | 2 ++ mm/migrate.c | 8 +++++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index 265c4328b36a..9af6c949a057 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -7,11 +7,13 @@ * on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages + * MIGRATE_NO_COPY will not copy page content */ enum migrate_mode { MIGRATE_ASYNC, MIGRATE_SYNC_LIGHT, MIGRATE_SYNC, + MIGRATE_NO_COPY, }; =20 enum migrate_reason { diff --git a/mm/migrate.c b/mm/migrate.c index ad03e7257847..3fe78ecb146a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -848,9 +848,11 @@ static int __migrate_folio(struct address_space *mappi= ng, struct folio *dst, if (folio_ref_count(src) !=3D expected_count) return -EAGAIN; =20 - rc =3D folio_mc_copy(dst, src); - if (unlikely(rc)) - return rc; + if (mode !=3D MIGRATE_NO_COPY) { + rc =3D folio_mc_copy(dst, src); + if (unlikely(rc)) + return rc; + } =20 rc =3D __folio_migrate_mapping(mapping, dst, src, expected_count); if (rc !=3D MIGRATEPAGE_SUCCESS) --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011004.outbound.protection.outlook.com [52.101.57.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6F8427FB21 for ; Tue, 23 Sep 2025 17:49:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.4 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649745; cv=fail; b=ODop8UVJBP+SgfgZzhyCyY6N8LhtZU31uIjaltXkkAhDt451y51+fKjks8142EivPLeuTPmSYzl9vUhn9mOEveeqJjmTs6HFCFnlJhR2RtAXVOb4SuVHEpw5qdnHskFXoucO7+FywaF/y4dPx5BJQq1TFrPgTR0JHe2wAkFbGGE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649745; c=relaxed/simple; bh=VHe2DsATWLXZ8BrioSn9lCajYXXjmSGD/JdEmf9df+8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qpXtZRw6w4mxknjWfMkWX7rRY6yLDC79Nr+WGmLXZTBzC1ZXUmMrWxF/2SrYmh2Lgx5QJSDErONvOnB70/Mlli2CUb51bi0XTgjGfnagSTRqhJuPLxJlOOPFpSLTaVOcXyzM0XmStPP2d28imDQufSzGiiOozc1XVPpCnzyEBg8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=YHC1oHhp; arc=fail smtp.client-ip=52.101.57.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="YHC1oHhp" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UPMZMmedeerN61QKWBvGNOI+2+igyItFxuezchOcAe8Gmr4LlUiAVTfKlMq9UPm21OavIjY1RngreEAxZxDdOz/LJnY8YIan+086GoL8JDyGkLLOK7usRTmfbj0I3a+N1Z3GESisRe6/cg8CuYMkA8mOW4Z6umQ91JDkabEMMgN5zeJgDR+nJOa+x8FaIqHfVfOAKWvhxl37DD0D5YS4pQOaTTfq4CtLSBLPCR4CWBt13BpvPp4uQFAwz6TPtviDcEP4OmH6oOUL/azei2/mWByni8SuAr+n9JJqc/m0r6HilTo605qD0ecjKO0i9KNmDSxfkA89G7Lzc0QtqpxG4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F937NWw09Hcn4DgbpPIs1W5CVMOH4s8rDaCGjvXFz1A=; b=NH1pAB8xTvvcrZwxrB7LSYeBil1QnZerNIFCJql+iKROORY0Fr5Xa+xYyX7xxBmFzem6VQvWECsr8CiOFL4rqyYU9f4wFlf+UAsaRi2ICSCyQr1dxsCdY5oVLHr2abB9OQ3pvD/+pabDNNKRZ04YelKZ3Mzdgqdk0eS+wKVbMMJBuJPEbXmn8NdVuOYCqPHiCrwP2zsMGuejlWZST3TFomoTTVCFJ2/3LZ8Zmfe1MLz7rA8NlIlaUMA/5h6gFEbyaNW54ds/9STXmR8CNr3yc45H+5ThWA1f2bdsRbso6VRfu/tNmtcj/wAqvf/yVs8nlEQixW6mHmrUrhSw2oMiVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F937NWw09Hcn4DgbpPIs1W5CVMOH4s8rDaCGjvXFz1A=; b=YHC1oHhp/svdLX9TnTbYTE8T4d6a1LPVsRhPh945zz0gjiVwQiPcLibv9J32iMif1gG/VV3Me95z4Lo+foNdah9JKer0MTXP9Q/GlTjjeKxx4meOGNIVeKO3QSPoxo7atIlkkkL5HAz/8fJ/Nk/KpXIlS9Wl8d6JK/0nsL8vYzc= Received: from BL1PR13CA0189.namprd13.prod.outlook.com (2603:10b6:208:2be::14) by SA1PR12MB8967.namprd12.prod.outlook.com (2603:10b6:806:38b::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.9; Tue, 23 Sep 2025 17:48:57 +0000 Received: from MN1PEPF0000F0E0.namprd04.prod.outlook.com (2603:10b6:208:2be:cafe::91) by BL1PR13CA0189.outlook.office365.com (2603:10b6:208:2be::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9137.20 via Frontend Transport; Tue, 23 Sep 2025 17:48:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E0.mail.protection.outlook.com (10.167.242.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:48:56 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:48:47 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 3/9] mm: Introduce folios_mc_copy() for batch copying folios Date: Tue, 23 Sep 2025 17:47:38 +0000 Message-ID: <20250923174752.35701-4-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E0:EE_|SA1PR12MB8967:EE_ X-MS-Office365-Filtering-Correlation-Id: c91e2cb6-d07d-45f4-97f1-08ddfac9784b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?BbPDQlXucdIkXrAu8tG+tfxjVJjnxbhr2+iLh1s/eb3aSC7F/EotoIQmRu+r?= =?us-ascii?Q?rM6vtVLy+AuLIsKZONDjn+bv1gtI4gHm32c/XnN7l+Ec8xwjiwtLxm61LyzH?= =?us-ascii?Q?zQuDYxHyPsI16Mpgt0WtpHc+pSRgBVm+3vKs7XE3TcbmSBeayU5XQ7h3SGBG?= =?us-ascii?Q?pEUo4G0TNKAjmj25+z/BAAA6MXCU+DrHSyMOwaAZL6b4PEZYnAZ+O5TZGiq7?= =?us-ascii?Q?/si5Lo8B+aC1ZWAK8LPEAPRwZGbQBSAlJDB4VXOkKIuPNyvZjkUpxy04Cp/u?= =?us-ascii?Q?JWHJuBZIg1eTlHbYZycuuvOkc7WvSwy7xiL+RYHu7vmJMDfI6XkiM+Lw0Dac?= =?us-ascii?Q?zxDQGa1NTGWNpGWRdeWwCa60vr/+LA7iMES29udMLJJO8X/NmfmlYjXJOcf+?= =?us-ascii?Q?Kal3UycpvCCeIwwL21/Jikphm32paPFAwHpRFWMc5A80S+Ij29GTBByMdSUT?= =?us-ascii?Q?lbC/AC/ozyTvYEOW21mEqVvFY04uKiOUQU4/mDxvczYcifNo/imP2Aa5CQHN?= =?us-ascii?Q?re04nH0x7cNydVd4SEJun6HGPNFXykLPn4DlXVfExWRrdKUibkEgcW1Nwier?= =?us-ascii?Q?ZDXbuRSvgl1a0oeHOYOt6NUvxtklfPbEJ4C3jMkMQUnZ3kyJ6ZifsIx7dfks?= =?us-ascii?Q?Akg1rr8jvjndZ677CMWc65WZ23yo3Hwo5RbjKbtpBdmkuvKw+p51j6JtZfkb?= =?us-ascii?Q?KGXJnEyOc4caevp3RVYRBNLgjuYhVN8JCo/pSmrXQ78jSADuv3grbe84ETRv?= =?us-ascii?Q?l3L8lZQYzUXesYgx6tKBZa9cRFeQQDyhhd18xWpFji+X8jA3TJgvTkXMq5zK?= =?us-ascii?Q?/k1T99XHZ8mFHVdTQi61x0+nngblJ0qFcqvbBdQhyOLkaE7Uv5C3vVBCOYMB?= =?us-ascii?Q?P2patumxBgWUEpL4McXj9T0UGcv5pnDqIgEOGPC0Ssmmk/pr8CiYCXwokY4P?= =?us-ascii?Q?pQvHzb6061dzyU2IJXsqG06ottDq3rQ0zIhQ7uhl/4I3cBxP+Y0b7tsIQkQV?= =?us-ascii?Q?3+SwZQmLiieZxeqTlsVzhjrRpZ2nj4h3oE22IjPmHo/3SaTl1DuEC8wFAbBF?= =?us-ascii?Q?kTnPuyaSQktYa4vnipIwqoC2Z69RNuPFkRmBsJdN4DMvOQ5sXTO+5BrJVC67?= =?us-ascii?Q?BIrPhSOqiyJ4e9akNMcc7CA1BAGWpWSo4qdfI2ekShxV/hoIOz3nPJzxkq65?= =?us-ascii?Q?XK42Mk0WlabyXzveI0+Hie99mbu4/145QO4pkGviq+yqpJLBB8vQZfC0UO5N?= =?us-ascii?Q?7tSXNVRV6jh9A7KRUN7KEZxoKpJoDgkyK9n5Ol6zQAzDF/bq8FQD44yTkmc2?= =?us-ascii?Q?HxMBXJtxbirtBgLq1kzUPgPe17smw4JGJTIRzsO6OHWtie1Zfbj6Z/3xwBcM?= =?us-ascii?Q?9nylGpMEK53agXzVQMLFbswFnvWVhbe67+sZsTn6MbgGMSGqTEsOU7jgwi8z?= =?us-ascii?Q?nlYYM+zt904+aUyJI4En2t03C+uqMEHJhHPMUciY5DHH5F2N1nlpneU75rCa?= =?us-ascii?Q?6UU0cRtU2bCPcnYg1gnSIUuRgMdfO93l6E0C?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:48:56.9635 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c91e2cb6-d07d-45f4-97f1-08ddfac9784b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E0.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB8967 Content-Type: text/plain; charset="utf-8" Introduce the folios_mc_copy() to copy the folio content from the list of src folios to the list of dst folios. This is preparatory patch for batch page migration offloading. Signed-off-by: Shivank Garg --- include/linux/mm.h | 2 ++ mm/util.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1ae97a0b8ec7..383702a819ac 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1187,6 +1187,8 @@ void __folio_put(struct folio *folio); void split_page(struct page *page, unsigned int order); void folio_copy(struct folio *dst, struct folio *src); int folio_mc_copy(struct folio *dst, struct folio *src); +int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list, + unsigned int __maybe_unused folios_cnt); =20 unsigned long nr_free_buffer_pages(void); =20 diff --git a/mm/util.c b/mm/util.c index f814e6a59ab1..2d7758f33fc6 100644 --- a/mm/util.c +++ b/mm/util.c @@ -748,6 +748,35 @@ int folio_mc_copy(struct folio *dst, struct folio *src) } EXPORT_SYMBOL(folio_mc_copy); =20 +/** + * folios_mc_copy - Copy the contents of list of folios. + * @dst_list: Folios to copy to. + * @src_list: Folios to copy from. + * @folios_cnt: Number of folios in each list (unused). + * + * The folio contents are copied from @src_list to @dst_list. + * Assume the caller has validated that lists are not empty and both lists + * have equal number of folios. This may sleep. + */ +int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list, + unsigned int __maybe_unused folios_cnt) +{ + struct folio *src, *dst; + int ret; + + dst =3D list_first_entry(dst_list, struct folio, lru); + list_for_each_entry(src, src_list, lru) { + cond_resched(); + ret =3D folio_mc_copy(dst, src); + if (ret) + return ret; + dst =3D list_next_entry(dst, lru); + } + + return 0; +} +EXPORT_SYMBOL(folios_mc_copy); + int sysctl_overcommit_memory __read_mostly =3D OVERCOMMIT_GUESS; static int sysctl_overcommit_ratio __read_mostly =3D 50; static unsigned long sysctl_overcommit_kbytes __read_mostly; --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012032.outbound.protection.outlook.com [52.101.48.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C717285C80 for ; Tue, 23 Sep 2025 17:49:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649760; cv=fail; b=RbQy1KRPTA6zFFQeNDd/f3PnOe9QDE8VDqdD+syS0N7h2l8YlqQBop/5hm6is/XMjtvBTWItSWH6ZPMl+kBVbHa8IVVgl5+/OlRxao1YPo/QxR8MZRL4+0st7QKXvKj9AF8mCkwyKVelSwKI93FW5Gl6o3ZWj66EbDDFCObZANc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649760; c=relaxed/simple; bh=Io5+4aTgQLEM3AHjdwZNvRcHW5Qb0uElANacJ01Yffg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Zf+vuCLlBrLvEORHKo5UdMJpEX2SmDP8wqBEqfD0wtN4iuumf7uS78Ss4SlrWzb2cU7zKRCn2p5BVCiygI7OCTCjGBxyWSkwNCjYu+yoBgbmrPWNjrjq1OLe/KT3S/5jwFV81QjVspq4GfzPSvWj1yyigU5BldHxlrZnBb5F3Iw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=bsp4l238; arc=fail smtp.client-ip=52.101.48.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="bsp4l238" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GiYmHo0esN+kPnRBIE9R23iaxhCTSoTViLHSevOoaOi+5iD6uEFGloKO3q4/Uyq4PoonzNeLQK8sjqYyGglUka0kPA4oq+djUoyrXbIUDhebg2RBJHy4USZBdWckVMHWVI3xzpzA4TnQzlkuacQjxwL6gXLgRMI8bUWHX2qW++yVvrrDBycJ1MfwZv5Z7qViRAH0NQq8pC01BYq6uU59hqeE9ZMPYWFZthPX4I7fUXv9caOjBw7B4kr8/uj5Oo8hTE264x3DXRYb++Jo+RgVEWYywayG4H4aKPqEgqEWAnoxAoE0P9i/9Ez6LEQomcaeTRv3CB1kYkJ/kzsv6N8ixg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zF2Qac4zUCI7Q24GJOdmKjFI0FxBhpcs3iOGl7cRHs0=; b=woo0ZWx+KKF40kwGEqxawzw1HN4uHIZg/QRpE8xP3+7IbNoWnUJVV5ZNT6N07qJVEr3X97rOcw9OeCrgpVp/Aagayq0OKLPyeDObMgLliUgOSg4OqvVcMCaj3Vs/DFLQlxijr/smKRQmz3kCX68O+Y1OAeqYUiJRi914h8D83zK81OviD6gnIhMSMpagm8JFgh9gKAZukvnIQibRm9ZHbJvDc0h6wwTfMQb2QHE3fvzXOoS9nbLnLDe99t2YN9R3o0Gs4FOfkJDx1lVHO9nJHXzWaRLbX5eZ9QkpOhmBdSwJ+HQB23nj04ZkHP18ta0WqGv6LSYCAnldyIJJhP0Uqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zF2Qac4zUCI7Q24GJOdmKjFI0FxBhpcs3iOGl7cRHs0=; b=bsp4l238UDubzNFtznrq/V946YiBWakYFuQo44JiW/+UJgDVtRRlnA55KwDRP+5pOdgm6T9vDx+4v1iJsurNxAaz/pIK95yPS8EHTQ7hbjS5wI5uuP/StetosGjdpTXGCAIqXuDkpQhd93BlJUaqbq7ZrJ8JferKK5VtR6S5I4I= Received: from BL1P222CA0005.NAMP222.PROD.OUTLOOK.COM (2603:10b6:208:2c7::10) by DM4PR12MB5746.namprd12.prod.outlook.com (2603:10b6:8:5d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.20; Tue, 23 Sep 2025 17:49:11 +0000 Received: from MN1PEPF0000F0E1.namprd04.prod.outlook.com (2603:10b6:208:2c7:cafe::9c) by BL1P222CA0005.outlook.office365.com (2603:10b6:208:2c7::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9137.20 via Frontend Transport; Tue, 23 Sep 2025 17:49:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E1.mail.protection.outlook.com (10.167.242.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:49:10 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:48:56 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 4/9] mm/migrate: add migrate_folios_batch_move to batch the folio move operations Date: Tue, 23 Sep 2025 17:47:39 +0000 Message-ID: <20250923174752.35701-5-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E1:EE_|DM4PR12MB5746:EE_ X-MS-Office365-Filtering-Correlation-Id: 2bea6ddb-5773-4ef4-2555-08ddfac98070 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?KaKai3uP6nuNb1PEoRBCSLpbtqY1nMD+24k4jQECK23bJTjNtJ4N6UPUILtx?= =?us-ascii?Q?f7RXBghW1JLUzsfgZg77xRbh2m06owSQ3QiKxvZbG6yaJUQmcWNAHOKdcqIa?= =?us-ascii?Q?Y1olaYAxtr0ff0dSm1NvjCLMxmuAgMT+z56v3iQlHddAC3SSORXRYpjII52F?= =?us-ascii?Q?p52sw46qUyManZv6rIo30eSC49XHDj8FfZdQ5UHR2poDgi1U9pgYQGn0IjD9?= =?us-ascii?Q?Wwy/DvwR5UM2wbXSFBOwGB16RBukCZOj7z3SxYdQK/AQww0PDGt9xkeDJH+j?= =?us-ascii?Q?9Bp+wg5EXjlMfxFrvQaNUyKHieNJXXVw8APNbORNmghbjAAy5Y4GS2m3SW3t?= =?us-ascii?Q?PBFI968xBNUcKAVPTvs/aiViYwP2KNra64p9SjPHbM/ZsUUYA19VGXzMkX1H?= =?us-ascii?Q?rQuePgojalCq/hzV/PZHLAtChEzdgHQN3kiVPdN56LSg3z2A8pDCz8yd/yL7?= =?us-ascii?Q?rVO02CaahB6siLqnwt/FwR/xPE/nDKUcfjDuxwU7AQF8akx2spQoFPODC/mS?= =?us-ascii?Q?UGu8ewUWrSz1IFVvZaXqOD+HmhMQv1o1OS8aFEsclPPJDdLoLW9trUSwczZz?= =?us-ascii?Q?EkV3fYu+XZX2Ex9pm5Ce3lRsXVjCqJXNZT6C5U7efJq3tHA56Z0U3o/H6Ut7?= =?us-ascii?Q?dQJ6a1FKjW+PcqPfzTjVnX6z0eMWnZ8Nj3nUW4PwpTcZPBIwmXDINH93e0es?= =?us-ascii?Q?D2T2GYVcwusNhoK3F2uTe3m9wEdS7s+KjSM1ipjT0F/erDB0oZAZA7L1oqqa?= =?us-ascii?Q?+11AWQ2J4tWLxEHdHI+ubu98UqxVysWVhgAUEx8R+MRFD13wOA140o+gBha5?= =?us-ascii?Q?RmHl/BbBuvCbUKoSSJwydFrxSKJMDKI9MHe2PNdiQ6yNrqyxWanhMwdi+C/g?= =?us-ascii?Q?48FHvIGRtfhs+0av7vaIK9J8y9fscTKfdIB9K1lYxoRK2j26URXYpEC5zDNU?= =?us-ascii?Q?83fYn7c8gH06Cyf7FiXpaCBRz05NASqKSuWJoVgRSW8sZkMQqrtuGfUTKH9s?= =?us-ascii?Q?l8SD8oT2ps3X03hNMcJoNCRKjFaNVxr1blukE2vXjNClBcpI3S8uPgI1qPew?= =?us-ascii?Q?wGjV0RT/jrSoabdCCOV8nIdZ1ikR3h7S/EBCb2tgqciLrF7+bfNAga9JtZcH?= =?us-ascii?Q?FcsjdgnKOz7Cb5VLcKC/9wENYqUQFz9CPqAIyK8d0Lz5Ke4snhVsj3A8QEVP?= =?us-ascii?Q?5Kdv5sXgqZbUfLzwfUXvOeu06zpfnnlpfxeK4CjD+577qkXfsj4pd6Q1MJd/?= =?us-ascii?Q?VXPr5iuHowZ4on4pfutN5sA1XHIWyr5nlcwKr6u2+iAekKiek3LCjA3YzSu0?= =?us-ascii?Q?pK2z7TO2DcnnElifHBeNVtEQRGSDk0kuVIM4ZzQP7naSzJSTo4vSdpZhZzBf?= =?us-ascii?Q?ZMhFnehlj1iDJ5K3Rq4H110Jmb6WfymX1GndBnHO4YwclhIIlWXvN7QergS+?= =?us-ascii?Q?ycXrTGIMaI4EG+WpM2/6EnQJMOYbxOyLgRhpGVsGLn352iKwKesguRGyQKsa?= =?us-ascii?Q?5xclNhf8tpwJYrQzRpzVz2FEFJex0ULGQvRu?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:49:10.6252 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2bea6ddb-5773-4ef4-2555-08ddfac98070 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E1.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5746 Content-Type: text/plain; charset="utf-8" This is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It uses MIGRATE_NO_COPY to skip folio copy during metadata copy process and performed the copies in a batch later. Currently, the folio move operation is performed individually for each folio in sequential manner: for_each_folio() { Copy folio metadata like flags and mappings Copy the folio content from src to dst Update page tables with dst folio } With this patch, we transition to a batch processing approach as shown below: for_each_folio() { Copy folio metadata like flags and mappings } Batch copy all src folios to dst for_each_folio() { Update page tables with dst folios } dst->private is used to store page states and possible anon_vma value, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process, src->private is used to store the data after metadata copy process, since src is no longer used. Co-developed-by: Zi Yan Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- mm/migrate.c | 197 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 193 insertions(+), 4 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 3fe78ecb146a..ce94e73a930d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -843,12 +843,15 @@ static int __migrate_folio(struct address_space *mapp= ing, struct folio *dst, enum migrate_mode mode) { int rc, expected_count =3D folio_expected_ref_count(src) + 1; + unsigned long dst_private =3D (unsigned long)dst->private; =20 /* Check whether src does not have extra refs before we do more work */ if (folio_ref_count(src) !=3D expected_count) return -EAGAIN; =20 - if (mode !=3D MIGRATE_NO_COPY) { + if (mode =3D=3D MIGRATE_NO_COPY) { + dst->private =3D NULL; + } else { rc =3D folio_mc_copy(dst, src); if (unlikely(rc)) return rc; @@ -862,6 +865,10 @@ static int __migrate_folio(struct address_space *mappi= ng, struct folio *dst, folio_attach_private(dst, folio_detach_private(src)); =20 folio_migrate_flags(dst, src); + + if (mode =3D=3D MIGRATE_NO_COPY) + src->private =3D (void *)dst_private; + return MIGRATEPAGE_SUCCESS; } =20 @@ -1149,7 +1156,7 @@ static void __migrate_folio_record(struct folio *dst, dst->private =3D (void *)anon_vma + old_page_state; } =20 -static void __migrate_folio_extract(struct folio *dst, +static void __migrate_folio_read(struct folio *dst, int *old_page_state, struct anon_vma **anon_vmap) { @@ -1157,6 +1164,12 @@ static void __migrate_folio_extract(struct folio *ds= t, =20 *anon_vmap =3D (struct anon_vma *)(private & ~PAGE_OLD_STATES); *old_page_state =3D private & PAGE_OLD_STATES; +} +static void __migrate_folio_extract(struct folio *dst, + int *old_page_state, + struct anon_vma **anon_vmap) +{ + __migrate_folio_read(dst, old_page_state, anon_vmap); dst->private =3D NULL; } =20 @@ -1776,6 +1789,176 @@ static void migrate_folios_move(struct list_head *s= rc_folios, } } =20 +static void migrate_folios_batch_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + int rc, nr_pages =3D 0, nr_batched_folios =3D 0; + int old_page_state =3D 0; + struct anon_vma *anon_vma =3D NULL; + int is_thp =3D 0; + LIST_HEAD(err_src); + LIST_HEAD(err_dst); + + /* + * Iterate over the list of locked src/dst folios to copy the metadata + */ + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp =3D folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages =3D folio_nr_pages(folio); + + /* + * dst->private is not cleared here. It is cleared and moved to + * src->private in __migrate_folio(). + */ + __migrate_folio_read(dst, &old_page_state, &anon_vma); + + /* + * Use MIGRATE_NO_COPY mode in migrate_folio family functions + * to copy the flags, mapping and some other ancillary information. + * This does everything except the page copy. The actual page copy + * is handled later in a batch manner. + */ + if (unlikely(page_movable_ops(&folio->page))) + rc =3D -EAGAIN; + else + rc =3D _move_to_new_folio_prep(dst, folio, MIGRATE_NO_COPY); + /* + * The rules are: + * Success: folio will be copied in batch + * -EAGAIN: move src/dst folios to tmp lists for + * non-batch retry + * Other errno: put src folio on ret_folios list, restore + * the dst folio + */ + if (rc =3D=3D -EAGAIN) { + *retry +=3D 1; + *thp_retry +=3D is_thp; + *nr_retry_pages +=3D nr_pages; + + list_move_tail(&folio->lru, &err_src); + list_move_tail(&dst->lru, &err_dst); + __migrate_folio_record(dst, old_page_state, anon_vma); + } else if (rc !=3D MIGRATEPAGE_SUCCESS) { + *nr_failed +=3D 1; + stats->nr_thp_failed +=3D is_thp; + stats->nr_failed_pages +=3D nr_pages; + + list_del(&dst->lru); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + } else { /* MIGRATEPAGE_SUCCESS */ + nr_batched_folios++; + } + + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } + + /* Exit if folio list for batch migration is empty */ + if (!nr_batched_folios) + goto out; + + /* Batch copy the folios */ + rc =3D folios_mc_copy(dst_folios, src_folios, nr_batched_folios); + + /* TODO: Is there a better way of handling the poison + * recover for batch copy, instead of falling back to serial copy? + */ + /* fallback to serial page copy if needed */ + if (rc) { + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp =3D folio_test_large(folio) && + folio_test_pmd_mappable(folio); + nr_pages =3D folio_nr_pages(folio); + rc =3D folio_mc_copy(dst, folio); + + if (rc) { + /* + * dst->private is moved to src->private in + * __migrate_folio(), so page state and anon_vma + * values can be extracted from (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, + &anon_vma); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, + private); + } + + switch (rc) { + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded +=3D nr_pages; + stats->nr_thp_succeeded +=3D is_thp; + break; + default: + *nr_failed +=3D 1; + stats->nr_thp_failed +=3D is_thp; + stats->nr_failed_pages +=3D nr_pages; + break; + } + + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } + } + + /* + * Iterate the folio lists to remove migration pte and restore them + * as working pte. Unlock the folios, add/remove them to LRU lists (if + * applicable) and release the src folios. + */ + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp =3D folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages =3D folio_nr_pages(folio); + /* + * dst->private is moved to src->private in __migrate_folio(), + * so page state and anon_vma values can be extracted from + * (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, &anon_vma); + list_del(&dst->lru); + + _move_to_new_folio_finalize(dst, folio, MIGRATEPAGE_SUCCESS); + + /* + * Below few steps are only applicable for lru pages which is + * ensured as we have removed the non-lru pages from our list. + */ + _migrate_folio_move_finalize1(folio, dst, old_page_state); + + _migrate_folio_move_finalize2(folio, dst, reason, anon_vma); + + /* Page migration successful, increase stat counter */ + stats->nr_succeeded +=3D nr_pages; + stats->nr_thp_succeeded +=3D is_thp; + + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } +out: + /* Add tmp folios back to the list to re-attempt migration. */ + list_splice(&err_src, src_folios); + list_splice(&err_dst, dst_folios); +} + static void migrate_folios_undo(struct list_head *src_folios, struct list_head *dst_folios, free_folio_t put_new_folio, unsigned long private, @@ -1986,13 +2169,19 @@ static int migrate_pages_batch(struct list_head *fr= om, /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); =20 - retry =3D 1; + retry =3D 0; + /* Batch move the unmapped folios */ + migrate_folios_batch_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); + for (pass =3D 0; pass < nr_pass && retry; pass++) { retry =3D 0; thp_retry =3D 0; nr_retry_pages =3D 0; =20 - /* Move the unmapped folios */ + /* Move the remaining unmapped folios */ migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from PH0PR06CU001.outbound.protection.outlook.com (mail-westus3azon11011021.outbound.protection.outlook.com [40.107.208.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3E82285045 for ; Tue, 23 Sep 2025 17:49:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.208.21 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649763; cv=fail; b=SKADiFMXffKd2s2K04kjwDZZHMdbLtVb9zOVDftWKb4KdwKsiVbZ/nQQ3AHQco+xquFL2SBK5SSh4fOS5aoNXeUw0u1Tqr9x/c29wSVkG5FZYB2FFGWhiqYpny2MCJ6Bv1eUIR/OGRR0oI4ntW/+SNKof7mTo53VDSZt/1M6Sjs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649763; c=relaxed/simple; bh=2aXHevH31sgse920q2SqycuRycRgfLLUoyaqa3Utfpo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OO2FVw7sH50pZLjAXUbkZPtvqTnWRACA2dcQqxlb5zkcQspobe5js7FXBvWu3/jJYapxfjr+spSQYNA63Ao5W6lqb7eIU2O6HYD5ObO5FtaYTS/XOvJ0MZefNLDOgar+CAYPyLkKrk+ixhs3374BBk1m43QgV6uFmoE+uKisoSM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=lbpy0kcV; arc=fail smtp.client-ip=40.107.208.21 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="lbpy0kcV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=keRn8YfKTPOhPWrX6ILImKgAEIl2xiL9V0QBEEz9x0lHbPcJxsNmTy3ywzZItyaRgqe8tUYRQ2+YWPD/KcG/8bRUj0hKoeKRz+7go2YIGfaEzww6/6mQWf6UmWfAb+3mQq3l7PH3JWlNvUf7vgNFxwHkt+wRnOGrjaW+F8JdQB5HVk5o40r3H1fqQrNYC/pZOimzxb0/Y2JWYjdgyh2xOluPXg/1FQFAYFLMxAiOkB0iZWjztvA1D1Knu0cSNU2a4tueT3ErDm/qWY8SWt9CZ+rrcOjYd5AUbUU73eqpCjUpxHXF4B93hGRA3vGdlqw7A+eLFBVN2AbqmdIV6trBHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MRU306ef9zUm+sdej3oeM9RD3nGLOvTgHl+98pndOfo=; b=Cg9sy3ClYXxXo/O6CnTfVWKgaSlABqGUyL4BYfcS3LbRlAMRdyUVpUP2A1POt+ZprALTP+KbqEjUHrEyhSDPW2XbzJYpyjpP07f6snWAkmLRxZYXpilQCAr3ZgErVhadOlfBUfTeKyz8+Mmf2ockQVdyO8JBdkbVLStnx/PGvcK13dXidzoOuVJvirudEnBa2bi1XTPA9VcPzhmJEjZnO7+J7NwxYfjYZm5Y3Kqx3BxoyN81jdq97ub4h62vjlu/XSEEDGLNXxESq/eYY+0CbZoR0N5CVFVR9laLyw6hvv5oIzltH35IybjKEHYu5iSOFIkNtb4qkgSWm0MsLT2KNQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MRU306ef9zUm+sdej3oeM9RD3nGLOvTgHl+98pndOfo=; b=lbpy0kcV1fIAxzsj4G4sEOV1OBwdyX+vyziqDzfDGnAq786VOXGrRV0K10CGlNEfEE8CgCBr+8NydebiX6aZFK/dgkSeVuR7Ajy/vKIvt6qxihq27dNI5GB6vhZ62spWSwOBHAofHgxBleRi8SyOYKWT2ctoaDmAx/zkCF2MwaQ= Received: from BL1P222CA0026.NAMP222.PROD.OUTLOOK.COM (2603:10b6:208:2c7::31) by LV3PR12MB9187.namprd12.prod.outlook.com (2603:10b6:408:194::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.9; Tue, 23 Sep 2025 17:49:14 +0000 Received: from MN1PEPF0000F0E1.namprd04.prod.outlook.com (2603:10b6:208:2c7:cafe::fb) by BL1P222CA0026.outlook.office365.com (2603:10b6:208:2c7::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9137.20 via Frontend Transport; Tue, 23 Sep 2025 17:49:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E1.mail.protection.outlook.com (10.167.242.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:49:13 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:49:05 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 5/9] mm: add support for copy offload for folio Migration Date: Tue, 23 Sep 2025 17:47:40 +0000 Message-ID: <20250923174752.35701-6-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E1:EE_|LV3PR12MB9187:EE_ X-MS-Office365-Filtering-Correlation-Id: 0c3d85d2-5d39-49cd-86ec-08ddfac9824d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?YcugHtTCi8P90NIBiPelIWFUAP3kOIzF15HP0+GDaH1BWP8bkG/gwzn41bNV?= =?us-ascii?Q?YHcrz9GyehL3ptbnWecLOUOAfgzFTxC5Wgcgj0e3ln+/Z5glAvreTR5bD8VU?= =?us-ascii?Q?IFNve/3/pdeSkooaYf3ZO5VOffbVgLCT12DYmy4FfBXo1srbTTjBx/N9b3BN?= =?us-ascii?Q?mngKS4jUbEMJSpwXksKYGF/FbmdH6hSQlrSNneKguNPEng4ihFWuDjtouQM2?= =?us-ascii?Q?BJIieTtTREc4U45LqyT2Sw8pyQZDEVo1QwY/h7r/FYrFmG4+HuGeOL7kK6EI?= =?us-ascii?Q?ypMZ7LJ2gc2RBvgIR3DnCaxf/iAs5OTGLl/O7lhaYSnMpeymdvFv3s9Km4vv?= =?us-ascii?Q?FwPpy1oddwMke1tI3nQN7p1tElo23y8ViGgh9XQv3xdwIIZY0WrkOMeFjQsW?= =?us-ascii?Q?hstPgHZ38+vGIBOEAVmcYFxxws3FV2hV2RBO0Rju2ldC1uIWlN/LcW8pCs60?= =?us-ascii?Q?lgbl6Sq3QROCayfILWuBoZVemY2xFCW1TyIKKnktUGKtaZpRkqpc2tYkxiP1?= =?us-ascii?Q?xqdor1pYRSOj88FcSFb0uyKzpd7n9v5dOtMgFx7A6GpnK0HAvLDloHojv5zN?= =?us-ascii?Q?VKa63Mt3RYhj7SIxQQxeTp7C+kv+Q2h4PM9s/ctSisjYq6JQF+QbCDaJMhpb?= =?us-ascii?Q?sw+wBo8vcURlMKWr4A4KT3MHnmXyMzEZYirm3SjCVJd7Dq+2uVCW9Pyiesbt?= =?us-ascii?Q?ArT1bK9DTMtoctaYMLAeZv4Y2AWwIyrXAdhFipRVtQp0F9/kQa3R/WBzt+vP?= =?us-ascii?Q?P2gO0MUKwKEq6sGnPdBYALSJuaCvd8To2QqL2n5MqcHwjMkVOiwHwutcP21X?= =?us-ascii?Q?hkDe4XyB9znX3HausYs0CV1DwmLA6DI4Xdj+aTm0OO9iFDlHSFLrgUXuPTHj?= =?us-ascii?Q?5s0EAbf7TscEUXqJtmix2f2eO07pdEHmDVy5Yt6+aSqopv059yLgvxB1sErk?= =?us-ascii?Q?+KbOdvd53TAzslsSG9yDh7dljK07qvWNswzhtVNnDHvMazixDKEnyruE3CCV?= =?us-ascii?Q?A+SQ+wxgF7RPOJ8CUBUgmjoyVNGk1R8aDSBcd7+osVVSW30mAFhXB7vGhMqA?= =?us-ascii?Q?B3QBPf9ovlVxiKdKfOPGuFk6KMO4N3x6Z95BzNWG5S5iybsgWh4AL5zpEa3S?= =?us-ascii?Q?rdwvkvQQm5fil+pYkRmooJ+jV27UVu3vK/ixhXaEpQcTJr5xO5UsuOh303Hh?= =?us-ascii?Q?eib+Mvps/FeLo76KaRD7gdisimo6VHfiJS6sBljcYW879aaSTr+Gq3GBG/Ls?= =?us-ascii?Q?KUTCp3G8RcG1tD5jvWSD5RyjnTN6hbdkUtz+9v8l4EimRhG+mG4alIJQHAeU?= =?us-ascii?Q?Mgmh206y8P9DNVBIn9PZCa/htiiynkvNd0Lemn1Fr56puhVm37hnDxqfke3o?= =?us-ascii?Q?UHYIRyYrI60WBOxnZ2jKoEumqo/MQzEXGtqi/6fhtOL5WO8aaboEYf4O4qau?= =?us-ascii?Q?T8BS2BP9X4ETIObxG110fiOHKc+9d7WLFefbn7dsF9izytEmHSS0NLJx7NQi?= =?us-ascii?Q?vueTKKEkAC4kXnvMvlgnVOBJVhGSJiVg0Cyt?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:49:13.7536 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0c3d85d2-5d39-49cd-86ec-08ddfac9824d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E1.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9187 Content-Type: text/plain; charset="utf-8" From: Mike Day Offload-Copy drivers should implement following functions to enable folio migration offloading: migrate_offc() - This function takes src and dst folios list undergoing migration. It is responsible for transfer of page content between the src and dst folios. can_migrate_offc() - It performs necessary checks if offload copying migration is supported for the give src and dst folios. Offload-Copy driver should include a mechanism to call start_offloading and stop_offloading for enabling and disabling migration offload respectively. Signed-off-by: Mike Day Co-developed-by: Shivank Garg Signed-off-by: Shivank Garg --- include/linux/migrate_offc.h | 34 +++++++++++++++++++++ mm/Kconfig | 8 +++++ mm/Makefile | 1 + mm/migrate.c | 49 +++++++++++++++++++++++++++++- mm/migrate_offc.c | 58 ++++++++++++++++++++++++++++++++++++ 5 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 include/linux/migrate_offc.h create mode 100644 mm/migrate_offc.c diff --git a/include/linux/migrate_offc.h b/include/linux/migrate_offc.h new file mode 100644 index 000000000000..e9e8a30f40f0 --- /dev/null +++ b/include/linux/migrate_offc.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _MIGRATE_OFFC_H +#define _MIGRATE_OFFC_H +#include + +#define MIGRATOR_NAME_LEN 32 +struct migrator { + char name[MIGRATOR_NAME_LEN]; + int (*migrate_offc)(struct list_head *dst_list, struct list_head *src_lis= t, + unsigned int folio_cnt); + struct rcu_head srcu_head; + struct module *owner; +}; + +extern struct migrator migrator; +extern struct mutex migrator_mut; +extern struct srcu_struct mig_srcu; + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head); +int offc_update_migrator(struct migrator *mig); +unsigned char *get_active_migrator_name(void); +int start_offloading(struct migrator *migrator); +int stop_offloading(void); +#else +static inline void srcu_mig_cb(struct rcu_head *head) { }; +static inline int offc_update_migrator(struct migrator *mig) { return 0; }; +static inline unsigned char *get_active_migrator_name(void) { return NULL;= }; +static inline void start_offloading(struct migrator *migrator) { return 0;= }; +static inline void stop_offloading(void) { return 0; }; +#endif /* CONFIG_OFFC_MIGRATION */ + +#endif /* _MIGRATE_OFFC_H */ diff --git a/mm/Kconfig b/mm/Kconfig index e443fe8cd6cf..a9cbb8d1f1f6 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -689,6 +689,14 @@ config MIGRATION config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE =20 +config OFFC_MIGRATION + bool "Migrate Pages offloading copy" + def_bool n + depends on MIGRATION + help + An interface allowing external modules or driver to offload + page copying in page migration. + config ARCH_ENABLE_HUGEPAGE_MIGRATION bool =20 diff --git a/mm/Makefile b/mm/Makefile index ef54aa615d9d..f609d3899992 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) +=3D failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) +=3D fail_page_alloc.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o +obj-$(CONFIG_OFFC_MIGRATION) +=3D migrate_offc.o obj-$(CONFIG_NUMA) +=3D memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o diff --git a/mm/migrate.c b/mm/migrate.c index ce94e73a930d..41bea48d823c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -43,6 +43,7 @@ #include #include #include +#include =20 #include =20 @@ -834,6 +835,52 @@ void folio_migrate_flags(struct folio *newfolio, struc= t folio *folio) } EXPORT_SYMBOL(folio_migrate_flags); =20 +#ifdef CONFIG_HAVE_STATIC_CALL +DEFINE_STATIC_CALL(_folios_copy, folios_mc_copy); + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head) +{ + static_call_query(_folios_copy); +} + +int offc_update_migrator(struct migrator *mig) +{ + struct module *old_owner, *new_owner; + int index; + int ret =3D 0; + + mutex_lock(&migrator_mut); + index =3D srcu_read_lock(&mig_srcu); + old_owner =3D READ_ONCE(migrator.owner); + new_owner =3D mig ? mig->owner : NULL; + + if (new_owner && !try_module_get(new_owner)) { + ret =3D -ENODEV; + goto out_unlock; + } + + strscpy(migrator.name, mig ? mig->name : "kernel", MIGRATOR_NAME_LEN); + static_call_update(_folios_copy, mig ? mig->migrate_offc : folios_mc_copy= ); + xchg(&migrator.owner, mig ? mig->owner : NULL); + if (old_owner) + module_put(old_owner); + +out_unlock: + WARN_ON(ret < 0); + srcu_read_unlock(&mig_srcu, index); + mutex_unlock(&migrator_mut); + + if (ret =3D=3D 0) { + call_srcu(&mig_srcu, &migrator.srcu_head, srcu_mig_cb); + srcu_barrier(&mig_srcu); + } + return ret; +} + +#endif /* CONFIG_OFFC_MIGRATION */ +#endif /* CONFIG_HAVE_STATIC_CALL */ + /************************************************************ * Migration functions ***********************************************************/ @@ -1870,7 +1917,7 @@ static void migrate_folios_batch_move(struct list_hea= d *src_folios, goto out; =20 /* Batch copy the folios */ - rc =3D folios_mc_copy(dst_folios, src_folios, nr_batched_folios); + rc =3D static_call(_folios_copy)(dst_folios, src_folios, nr_batched_folio= s); =20 /* TODO: Is there a better way of handling the poison * recover for batch copy, instead of falling back to serial copy? diff --git a/mm/migrate_offc.c b/mm/migrate_offc.c new file mode 100644 index 000000000000..a6530658a3f7 --- /dev/null +++ b/mm/migrate_offc.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +atomic_t dispatch_to_offc =3D ATOMIC_INIT(0); +EXPORT_SYMBOL_GPL(dispatch_to_offc); + +DEFINE_MUTEX(migrator_mut); +DEFINE_SRCU(mig_srcu); + +struct migrator migrator =3D { + .name =3D "kernel", + .migrate_offc =3D folios_mc_copy, + .srcu_head.func =3D srcu_mig_cb, + .owner =3D NULL, +}; + +int start_offloading(struct migrator *m) +{ + int offloading =3D 0; + int ret; + + pr_info("starting migration offload by %s\n", m->name); + ret =3D offc_update_migrator(m); + if (ret < 0) { + pr_err("failed to start migration offload by %s, err=3D%d\n", + m->name, ret); + return ret; + } + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 1); + return 0; +} +EXPORT_SYMBOL_GPL(start_offloading); + +int stop_offloading(void) +{ + int offloading =3D 1; + int ret; + + pr_info("stopping migration offload by %s\n", migrator.name); + ret =3D offc_update_migrator(NULL); + if (ret < 0) { + pr_err("failed to stop migration offload by %s, err=3D%d\n", + migrator.name, ret); + return ret; + } + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 0); + return 0; +} +EXPORT_SYMBOL_GPL(stop_offloading); + +unsigned char *get_active_migrator_name(void) +{ + return migrator.name; +} +EXPORT_SYMBOL_GPL(get_active_migrator_name); --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from PH0PR06CU001.outbound.protection.outlook.com (mail-westus3azon11011032.outbound.protection.outlook.com [40.107.208.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDBFC2D47FF for ; Tue, 23 Sep 2025 17:49:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.208.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649793; cv=fail; b=nbiKOZt1IkPi+yNT95m1f9LYA5HEjoDwfOWlb2kq5XewQzdom/xMkfUJrJLlB4r6QsIWbw0RgAXBDuvf+SRJ43IPEnFFnyyqw7GKNze+mQddqaj4M7X7tNsI2tHB+oDo8uTTj0zV+IwEDiOvx4VcJUJHm1lCXl6IZBQvPwINY+M= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649793; c=relaxed/simple; bh=zkumftL6RQwa1kkcfJ6pHE0nt741JDi8aRrCOKrHzNQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=uiWD18Cfh8AZv3DM33M6odIKeR0wCVGFjyO0NfowAINDjGpf3ZjTK8rABE13tkkKkPKg/pFVwa/4vuEipX27RT+6E9IEYmW1TDvA75ZDkh4+NczahxSsOQzOsHQRrCaIhzi2/JJgdrGjR03hGy0npqGhBB3NjrBZPn+f0oNxORw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=x7midnkY; arc=fail smtp.client-ip=40.107.208.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="x7midnkY" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vTSDa/8qsPoGc6qDn+zm9zqKkYMa/Hz+M+OAjqSLDSqxG1xBx6U2feTRNu4iBe09BUK1ELQg/1QtFslhz5xNZjwy7nMymdZz3v8CR7HRDtVJskUGjmX/knovJ4aSx2EEGWYW54AGRBputiHxS/9RQz/QfV9ogARsHYZ8p605Ywq69d6nMzYWxqnxD93Sc+NRwJ3ut8SeiYXYWeFpHwEeNBIht+Vn1GTSEUl3H1U+cckryCmtAH5WWQ3XPtcgIujRIih7FkvHs1Q3dBOxGa/memQkhpsM1HxtuKPkfL9e5NOEU98XK59Sd/rCUUI9U+Ng7FdMLRMoJGUfb0S3njr9Rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0wFXi6tT2mAPntQV3UQ5NM7KkuZqtFOlhy0rFkInSa4=; b=InU6Sqb+DAEevQ41h551tutzA4cwYR2IlIxrxiI+iJU/FJ/eRoq5iFfKpdGVuWGlXHCoKDcOouanpGRpxHlNLNme+Yffm9DQBuHvd5FYiCrJ+42QxI2z4bPWWjBuHguSy2usjUJoUEgFvfLHxs1Kk1D88S02jqWvspHkbuwFLx/rKL759r1620+l4jZ4rmhidesTAI5UIN2LeW5xlXY/7b6IIU8n5xm9G22Ft/XCsyCyDEJ/STFaw7NIOgcqI5rM7B6NX6NpVmosygWaVuFGJQFvIqwVwsFetmXaINcvoNV/+QjO45O5q0lKhvxXa1LqoRaE6pG28FvNHGNz+VPLiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0wFXi6tT2mAPntQV3UQ5NM7KkuZqtFOlhy0rFkInSa4=; b=x7midnkYDsK/Q6I9yIjizpY3M56ktbUyNyYpJanx8xoZQJ8dPU/GAKjc1box06Y2aHen3O+CJJ+usegAzfx4h2WhGRcaxwfsH1b1QO07VLrhZrTanddA64ko4yPDOZ7NRdRwrvceW6DZKe1Fc9tlIF0mKPLux4Fvt7R2BKsbO7Q= Received: from BL1PR13CA0151.namprd13.prod.outlook.com (2603:10b6:208:2bd::6) by BL3PR12MB6475.namprd12.prod.outlook.com (2603:10b6:208:3bb::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.22; Tue, 23 Sep 2025 17:49:33 +0000 Received: from MN1PEPF0000F0DF.namprd04.prod.outlook.com (2603:10b6:208:2bd:cafe::aa) by BL1PR13CA0151.outlook.office365.com (2603:10b6:208:2bd::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Tue, 23 Sep 2025 17:49:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0DF.mail.protection.outlook.com (10.167.242.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Tue, 23 Sep 2025 17:49:33 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:49:13 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 6/9] mtcopy: introduce multi-threaded page copy routine Date: Tue, 23 Sep 2025 17:47:41 +0000 Message-ID: <20250923174752.35701-7-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0DF:EE_|BL3PR12MB6475:EE_ X-MS-Office365-Filtering-Correlation-Id: 736972ed-754a-42e8-0513-08ddfac98e1a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?HQmF6aRZUWSXwMH7y/RR0z1RlyPdcwrT6YkJANnRZZeiIguhPyWByM6SMVSY?= =?us-ascii?Q?FRXDCc8L9ElahyllY1eeMm1sLwrTvlW71esYF+2jddqmIzuV/9SWYl+fPrCz?= =?us-ascii?Q?dhMcGc7q10RJpH7CH4UewMY+Kmc7N9FTvG5svf/9YJQ156OeOox6ZrKuUnqL?= =?us-ascii?Q?CIto6DYqjmq70Wd2xNKYyxQDSKE6JS70eGVmIGF3LOMdfPrnda8ehQWoAtnF?= =?us-ascii?Q?+0lEykx1Jf2LYHmFr0LW2oDTBNAwnvDJDhhn0jngWm+WFpm6SmcwydMDguqI?= =?us-ascii?Q?aCvCTOp6BkO4XhdbaT8F9JpagVsavNseHHl8o05Z0cBojqxjdmZX2q8ZEMwb?= =?us-ascii?Q?I/JGue2DIxwGSjDIcnN5BUGmv0Arc1NR8CnRgEBBJuoX12o2ko61eKctIneR?= =?us-ascii?Q?AUhahQljos6zZL/Yc0SFAYVBKSEaTvsJHjH691pHuaguH03Wxj882/v3SUsX?= =?us-ascii?Q?KSywmhYC/vzq7QO+15qnNit7J9PWOKZdt+1LZRxdSedhYZKGusoqUEg1HgiK?= =?us-ascii?Q?Fa78VEx1sEaoge1hjdvmmtiEgRBHpUM9SbT3xZnox3vI8LI4fVeJymJcAWoH?= =?us-ascii?Q?IZEN4PkIgjr/XzyeE1Gr7jkp088QY61brvRlW8pL6adh4NR1Kq9jJBdMoNeu?= =?us-ascii?Q?UY4c9MpcwsLCne+HTUTeT76YFD5d89JrPVo/yklOjFU4CBDrg+GH7sFT0in+?= =?us-ascii?Q?CqDWg9Hlcq2KnIM2lBCQl7ryKsI2p4yCuwfXxkUzZomaHW1FW+InM/e2mTJy?= =?us-ascii?Q?quwT4MZxnB9y2DB8uZQTU59Wwh9ca/69M2JOR0KJkcF1woxh+Wvtz0S6aLN1?= =?us-ascii?Q?LOqqNdN/kXqCDN0c053/YVkbv7NlVAkv04ouO9wL37qO9TTAS1qvwL4oUrYg?= =?us-ascii?Q?lJZbk/pdLFlqPmox2aA+pHR7nGTOwjCeosUcZnzz60r9Epw/XYZjUegLXo2U?= =?us-ascii?Q?GEG6JxSpW8HnCF8CmYqdcDN7ZUAX1tml5TEIMV57LXf/MAtmednQAa7qiPFr?= =?us-ascii?Q?0g0sJAkFozQvZFiOO7zsiJAkkYc89rXMTUZnjOuGwwgGnA1XY8Xy5cVYjf3i?= =?us-ascii?Q?YDdaiD2Q4AcjNEheKpdOT7TM6wIOlpf+z2vyPB4Jan8Mh4AbhPo5B0uwhtWp?= =?us-ascii?Q?Q27IQinAfASNMbIg7bd+/LZN/PxMF5qT9E5vVfmuGUmUKcr8GdEN7ErQyFky?= =?us-ascii?Q?et6tFErVhEK3bHmpNJnO5sh44pXu/FGXqojvlLm/X2jJIRHFK+QXXT4goDiF?= =?us-ascii?Q?Lj+ZHrVTqyJ0tVefBjf/fPPRfROb1YzE9uO/Hv1CG79G8iamJMmxrb9ic1hb?= =?us-ascii?Q?WMzOVw6rHS6UdC1t2cZPIvQwIy8FoH4GXos5/c3n2E0ckv1oziKslV78j2k8?= =?us-ascii?Q?2fZn8wsZBlEUkhnyS2XcBwUaLI803Xi+igG2dNmr0ut2VIIdPOrCqXt2vv3O?= =?us-ascii?Q?T5ZaU314STH6uFczncOBS6BUlM7NaKCMEpnxpXzlaTRgsxuNK1C7FtUxTkWS?= =?us-ascii?Q?RflglKJOcMFNdoKULzjgxNRrptIiKo7jdkBM?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:49:33.5480 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 736972ed-754a-42e8-0513-08ddfac98e1a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6475 Content-Type: text/plain; charset="utf-8" From: Zi Yan Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Enable using: echo 1 > /sys/kernel/cpu_mt/offloading echo NR_THREADS > /sys/kernel/cpu_mt/threads Disable: echo 0 > /sys/kernel/cpu_mt/offloading Signed-off-by: Zi Yan Co-developed-by: Shivank Garg Signed-off-by: Shivank Garg --- drivers/Kconfig | 2 + drivers/Makefile | 3 + drivers/migoffcopy/Kconfig | 9 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/copy_pages.c | 327 +++++++++++++++++++++++++ 6 files changed, 343 insertions(+) create mode 100644 drivers/migoffcopy/Kconfig create mode 100644 drivers/migoffcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/copy_pages.c diff --git a/drivers/Kconfig b/drivers/Kconfig index 4915a63866b0..d2cbc97a7683 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -251,4 +251,6 @@ source "drivers/hte/Kconfig" =20 source "drivers/cdx/Kconfig" =20 +source "drivers/migoffcopy/Kconfig" + endmenu diff --git a/drivers/Makefile b/drivers/Makefile index b5749cf67044..5326d88cf31c 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -42,6 +42,9 @@ obj-y +=3D clk/ # really early. obj-$(CONFIG_DMADEVICES) +=3D dma/ =20 +# Migration copy Offload +obj-$(CONFIG_OFFC_MIGRATION) +=3D migoffcopy/ + # SOC specific infrastructure drivers. obj-y +=3D soc/ obj-$(CONFIG_PM_GENERIC_DOMAINS) +=3D pmdomain/ diff --git a/drivers/migoffcopy/Kconfig b/drivers/migoffcopy/Kconfig new file mode 100644 index 000000000000..e73698af3e72 --- /dev/null +++ b/drivers/migoffcopy/Kconfig @@ -0,0 +1,9 @@ +config MTCOPY_CPU + bool "Multi-Threaded Copy with CPU" + depends on OFFC_MIGRATION + default n + help + Interface MT COPY CPU driver for batch page migration + offloading. Say Y if you want to try offloading with + MultiThreaded CPU copy APIs. + diff --git a/drivers/migoffcopy/Makefile b/drivers/migoffcopy/Makefile new file mode 100644 index 000000000000..0a3c356d67e6 --- /dev/null +++ b/drivers/migoffcopy/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) +=3D mtcopy/ diff --git a/drivers/migoffcopy/mtcopy/Makefile b/drivers/migoffcopy/mtcopy= /Makefile new file mode 100644 index 000000000000..b4d7da85eda9 --- /dev/null +++ b/drivers/migoffcopy/mtcopy/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) +=3D copy_pages.o diff --git a/drivers/migoffcopy/mtcopy/copy_pages.c b/drivers/migoffcopy/mt= copy/copy_pages.c new file mode 100644 index 000000000000..68e50de602d6 --- /dev/null +++ b/drivers/migoffcopy/mtcopy/copy_pages.c @@ -0,0 +1,327 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Parallel page copy routine. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_NUM_COPY_THREADS 64 + +unsigned int limit_mt_num =3D 4; +static int is_dispatching; + +static int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, unsigned int nr_items); + +static DEFINE_MUTEX(migratecfg_mutex); + +/* CPU Multithreaded Batch Migrator */ +struct migrator cpu_migrator =3D { + .name =3D "CPU_MT_COPY\0", + .migrate_offc =3D copy_page_lists_mt, + .owner =3D THIS_MODULE, +}; + +struct copy_item { + char *to; + char *from; + unsigned long chunk_size; +}; + +struct copy_page_info { + struct work_struct copy_page_work; + int ret; + unsigned long num_items; + struct copy_item item_list[]; +}; + +static unsigned long copy_page_routine(char *vto, char *vfrom, + unsigned long chunk_size) +{ + return copy_mc_to_kernel(vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread(struct work_struct *work) +{ + struct copy_page_info *my_work =3D (struct copy_page_info *)work; + int i; + + my_work->ret =3D 0; + for (i =3D 0; i < my_work->num_items; ++i) + my_work->ret |=3D !!copy_page_routine(my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +} + +static ssize_t mt_offloading_set(struct kobject *kobj, struct kobj_attribu= te *attr, + const char *buf, size_t count) +{ + int ccode; + int action; + + ccode =3D kstrtoint(buf, 0, &action); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + /* + * action is 0: User wants to disable MT offloading. + * action is 1: User wants to enable MT offloading. + */ + switch (action) { + case 0: + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 1) { + stop_offloading(); + is_dispatching =3D 0; + } else + pr_debug("MT migration offloading is already OFF\n"); + mutex_unlock(&migratecfg_mutex); + break; + case 1: + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 0) { + start_offloading(&cpu_migrator); + is_dispatching =3D 1; + } else + pr_debug("MT migration offloading is already ON\n"); + mutex_unlock(&migratecfg_mutex); + break; + default: + pr_debug("input should be zero or one, parsed as %d\n", action); + } + return sizeof(action); +} + +static ssize_t mt_offloading_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", is_dispatching); +} + +static ssize_t mt_threads_set(struct kobject *kobj, struct kobj_attribute = *attr, + const char *buf, size_t count) +{ + int ccode; + unsigned int threads; + + ccode =3D kstrtouint(buf, 0, &threads); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + if (threads > 0 && threads <=3D MAX_NUM_COPY_THREADS) { + mutex_lock(&migratecfg_mutex); + limit_mt_num =3D threads; + mutex_unlock(&migratecfg_mutex); + pr_debug("MT threads set to %u\n", limit_mt_num); + } else { + pr_debug("Invalid thread count. Must be between 1 and %d\n", MAX_NUM_COP= Y_THREADS); + return -EINVAL; + } + + return count; +} + +static ssize_t mt_threads_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", limit_mt_num); +} + +int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, unsigned int nr_items) +{ + struct copy_page_info *work_items[MAX_NUM_COPY_THREADS] =3D {0}; + unsigned int total_mt_num =3D limit_mt_num; + struct folio *src, *src2, *dst, *dst2; + int max_items_per_thread; + int item_idx; + int err =3D 0; + int cpu; + int i; + + if (IS_ENABLED(CONFIG_HIGHMEM)) + return -EOPNOTSUPP; + + /* Each threads get part of each page, if nr_items < totla_mt_num */ + if (nr_items < total_mt_num) + max_items_per_thread =3D nr_items; + else + max_items_per_thread =3D (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ? 1 : 0); + + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + work_items[cpu] =3D kzalloc(sizeof(struct copy_page_info) + + sizeof(struct copy_item) * + max_items_per_thread, + GFP_NOWAIT); + if (!work_items[cpu]) { + err =3D -ENOMEM; + goto free_work_items; + } + } + + if (nr_items < total_mt_num) { + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items =3D max_items_per_thread; + } + + item_idx =3D 0; + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + unsigned long chunk_size =3D PAGE_SIZE * folio_nr_pages(src) / total_mt= _num; + char *vfrom =3D page_address(&src->page); + char *vto =3D page_address(&dst->page); + + VM_WARN_ON(PAGE_SIZE * folio_nr_pages(src) % total_mt_num); + VM_WARN_ON(folio_nr_pages(dst) !=3D folio_nr_pages(src)); + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to =3D + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from =3D + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size =3D + chunk_size; + } + + item_idx++; + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) + queue_work(system_unbound_wq, + (struct work_struct *)work_items[cpu]); + } else { + int num_xfer_per_thread =3D nr_items / total_mt_num; + int per_cpu_item_idx; + + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items =3D num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu =3D 0; + per_cpu_item_idx =3D 0; + item_idx =3D 0; + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + work_items[cpu]->item_list[per_cpu_item_idx].to =3D + page_address(&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from =3D + page_address(&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size =3D + PAGE_SIZE * folio_nr_pages(src); + + VM_WARN_ON(folio_nr_pages(dst) !=3D + folio_nr_pages(src)); + + per_cpu_item_idx++; + item_idx++; + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + + if (per_cpu_item_idx =3D=3D work_items[cpu]->num_items) { + queue_work(system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx =3D 0; + cpu++; + } + } + if (item_idx !=3D nr_items) + pr_warn("%s: only %d out of %d pages are transferred\n", + __func__, item_idx - 1, nr_items); + } + + /* Wait until it finishes */ + for (i =3D 0; i < total_mt_num; ++i) { + flush_work((struct work_struct *)work_items[i]); + /* retry if any copy fails */ + if (work_items[i]->ret) + err =3D -EAGAIN; + } + +free_work_items: + for (cpu =3D 0; cpu < total_mt_num; ++cpu) + kfree(work_items[cpu]); + + return err; +} + +static struct kobject *mt_kobj_ref; +static struct kobj_attribute mt_offloading_attribute =3D __ATTR(offloading= , 0664, + mt_offloading_show, mt_offloading_set); +static struct kobj_attribute mt_threads_attribute =3D __ATTR(threads, 0664, + mt_threads_show, mt_threads_set); + +static int __init cpu_mt_module_init(void) +{ + int ret =3D 0; + + mt_kobj_ref =3D kobject_create_and_add("cpu_mt", kernel_kobj); + if (!mt_kobj_ref) + return -ENOMEM; + + ret =3D sysfs_create_file(mt_kobj_ref, &mt_offloading_attribute.attr); + if (ret) + goto out_offloading; + + ret =3D sysfs_create_file(mt_kobj_ref, &mt_threads_attribute.attr); + if (ret) + goto out_threads; + + is_dispatching =3D 0; + + return 0; + +out_threads: + sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); +out_offloading: + kobject_put(mt_kobj_ref); + return ret; +} + +static void __exit cpu_mt_module_exit(void) +{ + /* Stop the MT offloading to unload the module */ + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 1) { + stop_offloading(); + is_dispatching =3D 0; + } + mutex_unlock(&migratecfg_mutex); + + sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); + sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); + kobject_put(mt_kobj_ref); +} + +module_init(cpu_mt_module_init); +module_exit(cpu_mt_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Zi Yan"); +MODULE_DESCRIPTION("CPU_MT_COPY"); /* CPU Multithreaded Batch Migrator */ --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010065.outbound.protection.outlook.com [40.93.198.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24A232C3761 for ; Tue, 23 Sep 2025 17:49:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649788; cv=fail; b=oSyeW/lSaV9QdHkwTqzxCPE4a/XwT36jGdt/tnNtbmubfl6wPMEpTOqCZmM6SVrvq8ye95YdBIX95jW4sT+jFQ2j7PnE/h1LxbqC5WsCgdAi0HMiz2JdVXYbE79s/W/yDJZhdmOyrxdSt4BqvJJnK1Q4FIbzseBSRsADI0zblR4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649788; c=relaxed/simple; bh=w71cbq/0NHoopfP4ZVhQTLuYwbbabAeU3Wl4P0C8JLA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=n+VmzzfKfghY/YXSzN4jhGLKHvfI8xt4S9jmKUkwu1CnNc7fsQ6iF1rQPTlDqBt3ZvlMHrWt6kkve3jXjnXBGeOgMGULbzVm8peVfBj47liDPB/bq60aB0CxUWzJI5f7JmPtUMPw1csmq3+ZPJO1WfAykLt07hyKYMrhMVpqAe8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=hvIC9Hib; arc=fail smtp.client-ip=40.93.198.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="hvIC9Hib" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QwJp/YkKdxFeI2hFtz9up6W98SLrDquAyqPH39cRwxLmSSjxcU3TSI1TB4IugvpYKkfzVGjKetnBlSDj15tZ1aOPsicJ9WsNkkhqDy//WHXHijDFisGoU6Chmq6z5c1PvfdK6rcf2ULEBEaMjRf+GlOxmD6oltNGu7JYIFSC2aXoarFtlT4ITMJ0GLajUSSd2h2mUd/xopCmJwuyKPXOnH5ra9x79g14w8lkK3hpR6lm17tp84CVTa+VFzmd/gcFFwB+iAAwd4O6T9KTh8u5IybKtyueg/UQnsKY+c98dyYM0GDt7kcUR4pa1pJnDmj4jG+guhIrX0SEoz8bBAjXCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ySxMyb6AxP1VEu8p9pR2Cxm8wJ0Dj/H34gqdILOtxH0=; b=mURaNmf6otSl9hyskm0voSibCoMGsK4iRu8bIRQm4uTa6m3k8dWxso8+hQl+i1GEotNp8zJn4U3qo4LQoJlz1uS1pz9c9jd8+LTfDhjn4kO0kbAEawSv7Dv7PHkBK9ICyrkN87paOKpMOpZrZ+dfQDAUdTdue8pptaULcUGsm9SGyykEzYxGipq1DaXbY+ATVeB7P1lHUUXH1zSGHykGV3JwPYkPqYIP5gwM/SUy34xdleO5nyMd9IUelc/trHx3zmI6mK11pfZuavXBPE+y0vNJai04k2U+p9DvQE5OQ1TO0PwVz+n8ssZcMsT7c9ZakbFwPbu3lAfwyR/6oJUmOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ySxMyb6AxP1VEu8p9pR2Cxm8wJ0Dj/H34gqdILOtxH0=; b=hvIC9HibVtJf0adTAY6x9yIsS2jD73MDc5Ej7TKbCMOHDPphYNGE9vBCXTWEc1qR6xl8MFWRrcobB6kO9hsmcHduQZvss0YN/U1fBV5hq4a1Qvq+m7uv9mSC19D4eiFoUdfDawYYdArE7zSArjf84NgC4tBZGl9BbjuEb2m5ZU0= Received: from BL1PR13CA0269.namprd13.prod.outlook.com (2603:10b6:208:2ba::34) by CY8PR12MB7097.namprd12.prod.outlook.com (2603:10b6:930:51::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.14; Tue, 23 Sep 2025 17:49:40 +0000 Received: from MN1PEPF0000F0E5.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::7e) by BL1PR13CA0269.outlook.office365.com (2603:10b6:208:2ba::34) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Tue, 23 Sep 2025 17:49:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E5.mail.protection.outlook.com (10.167.242.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:49:39 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:49:22 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 7/9] dcbm: add dma core batch migrator for batch page offloading Date: Tue, 23 Sep 2025 17:47:42 +0000 Message-ID: <20250923174752.35701-8-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E5:EE_|CY8PR12MB7097:EE_ X-MS-Office365-Filtering-Correlation-Id: 085cbdc6-0f45-4926-7eeb-08ddfac991d9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?3pbclC735Jkd5U9DCIAKodNSGrv9SfbjycCeOnmd4nJQUrZY7FmaLS9XZwzP?= =?us-ascii?Q?gRX+Ol7oxRSRXWE85jtxUJdJuMKNkgZpLpJUarjpER5MGK0zFyJhIjznpTiQ?= =?us-ascii?Q?0btR4PkBhVw6kGHBOZF5LEhb5NiLy8QgR0hhQhKDyRj6KJZOGGrD9fxvq7c3?= =?us-ascii?Q?hj+GG7r36Z81lz4pc+OUc3+BKmLP9e1dGN3jSS1zfcKYczTF/ZjMApjGwDHr?= =?us-ascii?Q?XWNHbhYfq8Aj7u6+o/TbBRWhN648CY/emGECN8MkOooGrZrMsSWcw05eX1U2?= =?us-ascii?Q?XqMoP7Dqi0BFURqna8XDCUaxSajrssym+a5D0+8FiEDdNVpKdfNsFE6i33h1?= =?us-ascii?Q?WQcJOVtDHEbtJ105gSFKuainpB1ZIRPq2GxqFMLwqouxkFraKR+VaCreJ2pX?= =?us-ascii?Q?316SgBSq3J6JRsA5WTOboH2ARELBD0anVL0hl59ALk5rNoGL2FHTubynvks3?= =?us-ascii?Q?zDabZbexqKmep1XoScEINPLhsO8df2NH4tQX2zBFSAEFOlXK2kkCSGKddX3B?= =?us-ascii?Q?qYvnxt7yPLs9MPUwlrZ+7aNsBD+xkHK6HxKFpL3d16vhOOizriWOg+KaQaNH?= =?us-ascii?Q?bV7S6OW4JmfpYtrn9wJUEeJTMhxEQCSa5apDTu8txpZBIaRVhAgBCGUSGzby?= =?us-ascii?Q?LZvw0ek5J1Q1MrXvyTEFCnOPCGbT3C4lFQeFARrgZYOk41RH+kQnffS/yfLZ?= =?us-ascii?Q?0zr5Qmgyma0zaJQM+jHJ5b+wc6FtpahtjCrSUaxOkg7FK/dbt6CTyd/KErSV?= =?us-ascii?Q?ivNOEdm9xQ2n2X0RX+3+KlEaYEBJAUNcb8Os4GtM+n5UlOnGlOaKiGKpCaQb?= =?us-ascii?Q?z9JsYExeNvVgFrU5sVQ/+R0AUrQBtdDIVBgvdS8OFKMnkJtbSqET5QftFaY1?= =?us-ascii?Q?KjaF1yzR5kE/xCNcuksRYc+AjCdkgQlk8EwSjE8wgZP97FY4TNnVp+s/Rm7Q?= =?us-ascii?Q?hUtIeHhZe9B9TZv378Tsf1ecM8g7lGk1k9gafSrzKwuYKSA5n5LT0xxfs15L?= =?us-ascii?Q?xBHydxfz29dfd+M4E7q4OQT5r5hu83t2FlpZSBYdTcyxnwz96CoM/PaEsO6F?= =?us-ascii?Q?XtNQawRuVBWF1Vu6utwLicmO4V4qpG2fQDBPVNjGd2DxpS79VXKb9mYhwpIe?= =?us-ascii?Q?po6+nf0dOnOy1N+HV5dmW2I1d6rOaq80gqhbDlXMx+K6MCcSgTyGNLUixxAJ?= =?us-ascii?Q?X9+KSdFxuD1mlBM6+pw1mHPkqWlOenNBc3mw4q//SuEqIdvMttrzE5nvmzyO?= =?us-ascii?Q?J7/1T47s+CBufWaQLtsdupmNYf3etiiKKl2TKnhZp7eILi70T8TgbqCVFUeT?= =?us-ascii?Q?bws1cUQAfXWgVpMP4DfIisNTltvXONud9XxHx9M3iHxyBQ62kQ9+SRYkdFa0?= =?us-ascii?Q?B62XrDKQgV7BI2qGzgLM2mhebVJniURXHgdpO6xwntAONHGv4X3SIs7uVDkM?= =?us-ascii?Q?FytYJ6G0GHqtm2RikZ9ScObLscqURbQf/VSjpaTUux++yYclLndLn3eze5wo?= =?us-ascii?Q?oVBlR7ydYbqQ4rYiv2f5ID8i6MSD85o6AxOa?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:49:39.8360 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 085cbdc6-0f45-4926-7eeb-08ddfac991d9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7097 Content-Type: text/plain; charset="utf-8" The dcbm (DMA core batch migrator) provides a generic interface using DMAEngine for end-to-end testing of the batch page migration offload feature. Enable DCBM offload: echo 1 > /sys/kernel/dcbm/offloading echo NR_DMA_CHAN_TO_USE > /sys/kernel/dcbm/nr_dma_chan Disable DCBM offload: echo 0 > /sys/kernel/dcbm/offloading Signed-off-by: Shivank Garg --- drivers/migoffcopy/Kconfig | 8 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/dcbm/Makefile | 1 + drivers/migoffcopy/dcbm/dcbm.c | 415 +++++++++++++++++++++++++++++++ 4 files changed, 425 insertions(+) create mode 100644 drivers/migoffcopy/dcbm/Makefile create mode 100644 drivers/migoffcopy/dcbm/dcbm.c diff --git a/drivers/migoffcopy/Kconfig b/drivers/migoffcopy/Kconfig index e73698af3e72..c1b2eff7650d 100644 --- a/drivers/migoffcopy/Kconfig +++ b/drivers/migoffcopy/Kconfig @@ -6,4 +6,12 @@ config MTCOPY_CPU Interface MT COPY CPU driver for batch page migration offloading. Say Y if you want to try offloading with MultiThreaded CPU copy APIs. +config DCBM_DMA + bool "DMA Core Batch Migrator" + depends on OFFC_MIGRATION && DMA_ENGINE + default n + help + Interface DMA driver for batch page migration offloading. + Say Y if you want to try offloading with DMAEngine APIs + based driver. =20 diff --git a/drivers/migoffcopy/Makefile b/drivers/migoffcopy/Makefile index 0a3c356d67e6..dedc86ff54c1 100644 --- a/drivers/migoffcopy/Makefile +++ b/drivers/migoffcopy/Makefile @@ -1 +1,2 @@ obj-$(CONFIG_MTCOPY_CPU) +=3D mtcopy/ +obj-$(CONFIG_DCBM_DMA) +=3D dcbm/ diff --git a/drivers/migoffcopy/dcbm/Makefile b/drivers/migoffcopy/dcbm/Mak= efile new file mode 100644 index 000000000000..56ba47cce0f1 --- /dev/null +++ b/drivers/migoffcopy/dcbm/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_DCBM_DMA) +=3D dcbm.o diff --git a/drivers/migoffcopy/dcbm/dcbm.c b/drivers/migoffcopy/dcbm/dcbm.c new file mode 100644 index 000000000000..87a58c0c3b9b --- /dev/null +++ b/drivers/migoffcopy/dcbm/dcbm.c @@ -0,0 +1,415 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * + * DMA batch-offloading interface driver + * + * Copyright (C) 2024-25 Advanced Micro Devices, Inc. + */ + +#include +#include +#include +#include +#include + +#define MAX_DMA_CHANNELS 16 + +static bool offloading_enabled; +static unsigned int nr_dma_channels =3D 1; +static DEFINE_MUTEX(dcbm_mutex); + +struct dma_work { + struct dma_chan *chan; + struct completion done; + atomic_t pending; + struct sg_table *src_sgt; + struct sg_table *dst_sgt; + bool mapped; +}; + +static void dma_completion_callback(void *data) +{ + struct dma_work *work =3D data; + + if (atomic_dec_and_test(&work->pending)) + complete(&work->done); +} + +static int setup_sg_tables(struct dma_work *work, struct list_head **src_p= os, + struct list_head **dst_pos, int nr) +{ + struct scatterlist *sg_src, *sg_dst; + struct device *dev; + int i, ret; + + work->src_sgt =3D kmalloc(sizeof(*work->src_sgt), GFP_KERNEL); + if (!work->src_sgt) + return -ENOMEM; + work->dst_sgt =3D kmalloc(sizeof(*work->dst_sgt), GFP_KERNEL); + if (!work->dst_sgt) + goto err_free_src; + + ret =3D sg_alloc_table(work->src_sgt, nr, GFP_KERNEL); + if (ret) + goto err_free_dst; + ret =3D sg_alloc_table(work->dst_sgt, nr, GFP_KERNEL); + if (ret) + goto err_free_src_table; + + sg_src =3D work->src_sgt->sgl; + sg_dst =3D work->dst_sgt->sgl; + for (i =3D 0; i < nr; i++) { + struct folio *src =3D list_entry(*src_pos, struct folio, lru); + struct folio *dst =3D list_entry(*dst_pos, struct folio, lru); + + sg_set_folio(sg_src, src, folio_size(src), 0); + sg_set_folio(sg_dst, dst, folio_size(dst), 0); + + *src_pos =3D (*src_pos)->next; + *dst_pos =3D (*dst_pos)->next; + + if (i < nr - 1) { + sg_src =3D sg_next(sg_src); + sg_dst =3D sg_next(sg_dst); + } + } + + dev =3D dmaengine_get_dma_device(work->chan); + if (!dev) { + ret =3D -ENODEV; + goto err_free_dst_table; + } + ret =3D dma_map_sgtable(dev, work->src_sgt, DMA_TO_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING); + if (ret) + goto err_free_dst_table; + ret =3D dma_map_sgtable(dev, work->dst_sgt, DMA_FROM_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING); + if (ret) + goto err_unmap_src; + /* Verify mapping produced same number of entries */ + if (work->src_sgt->nents !=3D work->dst_sgt->nents) { + pr_err("Mismatched SG entries after mapping: src=3D%d dst=3D%d\n", + work->src_sgt->nents, work->dst_sgt->nents); + ret =3D -EINVAL; + goto err_unmap_dst; + } + work->mapped =3D true; + return 0; + +err_unmap_dst: + dma_unmap_sgtable(dev, work->dst_sgt, DMA_FROM_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING); +err_unmap_src: + dma_unmap_sgtable(dev, work->src_sgt, DMA_TO_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING); +err_free_dst_table: + sg_free_table(work->dst_sgt); +err_free_src_table: + sg_free_table(work->src_sgt); +err_free_dst: + kfree(work->dst_sgt); + work->dst_sgt =3D NULL; +err_free_src: + kfree(work->src_sgt); + work->src_sgt =3D NULL; + pr_err("DCBM: Failed to setup SG tables\n"); + return ret; +} + +static void cleanup_dma_work(struct dma_work *works, int actual_channels) +{ + struct device *dev; + int i; + + if (!works) + return; + + for (i =3D 0; i < actual_channels; i++) { + if (!works[i].chan) + continue; + + dev =3D dmaengine_get_dma_device(works[i].chan); + + /* Terminate any pending transfers */ + if (atomic_read(&works[i].pending) > 0) + dmaengine_terminate_sync(works[i].chan); + + if (dev && works[i].mapped) { + if (works[i].src_sgt) { + dma_unmap_sgtable(dev, works[i].src_sgt, DMA_TO_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING); + sg_free_table(works[i].src_sgt); + kfree(works[i].src_sgt); + } + if (works[i].dst_sgt) { + dma_unmap_sgtable(dev, works[i].dst_sgt, DMA_FROM_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING); + sg_free_table(works[i].dst_sgt); + kfree(works[i].dst_sgt); + } + } + dma_release_channel(works[i].chan); + } + kfree(works); +} + +static int submit_dma_transfers(struct dma_work *work) +{ + struct scatterlist *sg_src, *sg_dst; + struct dma_async_tx_descriptor *tx; + unsigned long flags =3D DMA_CTRL_ACK; + dma_cookie_t cookie; + int i; + + /* Logic: send single callback for the entire batch */ + atomic_set(&work->pending, 1); + + sg_src =3D work->src_sgt->sgl; + sg_dst =3D work->dst_sgt->sgl; + /* Iterate over DMA-mapped entries */ + for_each_sgtable_dma_sg(work->src_sgt, sg_src, i) { + /* Only interrupt on the last transfer */ + if (i =3D=3D work->src_sgt->nents - 1) + flags |=3D DMA_PREP_INTERRUPT; + + tx =3D dmaengine_prep_dma_memcpy(work->chan, + sg_dma_address(sg_dst), + sg_dma_address(sg_src), + sg_dma_len(sg_src), + flags); + if (!tx) { + atomic_set(&work->pending, 0); + return -EIO; + } + + /* Only set callback on last transfer */ + if (i =3D=3D work->src_sgt->nents - 1) { + tx->callback =3D dma_completion_callback; + tx->callback_param =3D work; + } + + cookie =3D dmaengine_submit(tx); + if (dma_submit_error(cookie)) { + atomic_set(&work->pending, 0); + return -EIO; + } + sg_dst =3D sg_next(sg_dst); + } + return 0; +} + +/** + * folios_copy_dma - Copy folios using DMA engine + * @dst_list: Destination folio list + * @src_list: Source folio list + * @nr_folios: Number of folios to copy + * + * Return: 0. Fallback to CPU copy on any error. + */ +static int folios_copy_dma(struct list_head *dst_list, + struct list_head *src_list, + unsigned int nr_folios) +{ + struct dma_work *works; + struct list_head *src_pos =3D src_list->next; + struct list_head *dst_pos =3D dst_list->next; + int i, folios_per_chan, ret =3D 0; + dma_cap_mask_t mask; + int actual_channels =3D 0; + int max_channels; + + max_channels =3D min3(nr_dma_channels, nr_folios, MAX_DMA_CHANNELS); + + works =3D kcalloc(max_channels, sizeof(*works), GFP_KERNEL); + if (!works) + goto fallback; + + dma_cap_zero(mask); + dma_cap_set(DMA_MEMCPY, mask); + + for (i =3D 0; i < max_channels; i++) { + works[actual_channels].chan =3D dma_request_chan_by_mask(&mask); + if (IS_ERR(works[actual_channels].chan)) + break; + init_completion(&works[actual_channels].done); + actual_channels++; + } + + if (actual_channels =3D=3D 0) { + kfree(works); + goto fallback; + } + + for (i =3D 0; i < actual_channels; i++) { + folios_per_chan =3D nr_folios * (i + 1) / actual_channels - + (nr_folios * i) / actual_channels; + if (folios_per_chan =3D=3D 0) + continue; + + ret =3D setup_sg_tables(&works[i], &src_pos, &dst_pos, folios_per_chan); + if (ret) + goto cleanup; + } + + for (i =3D 0; i < actual_channels; i++) { + ret =3D submit_dma_transfers(&works[i]); + if (ret) { + dev_err(dmaengine_get_dma_device(works[i].chan), + "Failed to submit transfers for channel %d\n", i); + goto cleanup; + } + } + + for (i =3D 0; i < actual_channels; i++) { + if (atomic_read(&works[i].pending) > 0) + dma_async_issue_pending(works[i].chan); + } + + for (i =3D 0; i < actual_channels; i++) { + if (atomic_read(&works[i].pending) > 0) { + if (!wait_for_completion_timeout(&works[i].done, msecs_to_jiffies(10000= ))) { + dev_err(dmaengine_get_dma_device(works[i].chan), + "DMA timeout on channel %d\n", i); + ret =3D -ETIMEDOUT; + goto cleanup; + } + } + } + +cleanup: + cleanup_dma_work(works, actual_channels); + if (ret) + goto fallback; + return 0; +fallback: + /* Fall back to CPU copy */ + pr_err("DCBM: Falling back to CPU copy\n"); + folios_mc_copy(dst_list, src_list, nr_folios); + return 0; +} + +static struct migrator dma_migrator =3D { + .name =3D "DCBM", + .migrate_offc =3D folios_copy_dma, + .owner =3D THIS_MODULE, +}; + +static ssize_t offloading_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", offloading_enabled); +} + +static ssize_t offloading_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + bool enable; + int ret; + + ret =3D kstrtobool(buf, &enable); + if (ret) + return ret; + + mutex_lock(&dcbm_mutex); + + if (enable =3D=3D offloading_enabled) { + pr_err("migration offloading is already %s\n", + enable ? "ON" : "OFF"); + goto out; + } + + if (enable) { + start_offloading(&dma_migrator); + offloading_enabled =3D true; + pr_info("migration offloading is now ON\n"); + } else { + stop_offloading(); + offloading_enabled =3D false; + pr_info("migration offloading is now OFF\n"); + } +out: + mutex_unlock(&dcbm_mutex); + return count; +} + +static ssize_t nr_dma_chan_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", nr_dma_channels); +} + +static ssize_t nr_dma_chan_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int val; + int ret; + + ret =3D kstrtouint(buf, 0, &val); + if (ret) + return ret; + + if (val < 1 || val > MAX_DMA_CHANNELS) + return -EINVAL; + + mutex_lock(&dcbm_mutex); + nr_dma_channels =3D val; + mutex_unlock(&dcbm_mutex); + + return count; +} + +static struct kobj_attribute offloading_attr =3D __ATTR_RW(offloading); +static struct kobj_attribute nr_dma_chan_attr =3D __ATTR_RW(nr_dma_chan); + +static struct attribute *dcbm_attrs[] =3D { + &offloading_attr.attr, + &nr_dma_chan_attr.attr, + NULL, +}; +ATTRIBUTE_GROUPS(dcbm); + +static struct kobject *dcbm_kobj; + +static int __init dcbm_init(void) +{ + int ret; + + dcbm_kobj =3D kobject_create_and_add("dcbm", kernel_kobj); + if (!dcbm_kobj) + return -ENOMEM; + + ret =3D sysfs_create_groups(dcbm_kobj, dcbm_groups); + if (ret) { + kobject_put(dcbm_kobj); + return ret; + } + + pr_info("DMA Core Batch Migrator initialized\n"); + return 0; +} + +static void __exit dcbm_exit(void) +{ + /* Ensure offloading is stopped before module unload */ + mutex_lock(&dcbm_mutex); + if (offloading_enabled) { + stop_offloading(); + offloading_enabled =3D false; + } + mutex_unlock(&dcbm_mutex); + + sysfs_remove_groups(dcbm_kobj, dcbm_groups); + kobject_put(dcbm_kobj); + + pr_info("DMA Core Batch Migrator unloaded\n"); +} + +module_init(dcbm_init); +module_exit(dcbm_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Shivank Garg"); +MODULE_DESCRIPTION("DMA Core Batch Migrator"); --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012066.outbound.protection.outlook.com [40.107.209.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC9532C08CE for ; Tue, 23 Sep 2025 17:49:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649791; cv=fail; b=YGTn7OJGtslfFFIp44ogqZZO7ARvB6Irs1NqiDR9Heke2yRzBBmavt8JkzlTMQAxsin5zzZY0D7+0rzyHNuWhlmvRNOazbeZPRBESY8w7g3BUYuF99j5JvF08mEYtxbfpoUnAdXhXtsEWuSkYjh9ye3CoSx5pyIQZiH8nhXG7s8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649791; c=relaxed/simple; bh=ifQrazwVHftfJwkklH2JQgt+5Z4lrGJBu1mqQii0xIM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=g5kmF2Z/0R57m9Zxal+FcWWOX6nc32M8wUK0wQKQUrtdKvWk+7OgBW1zOtK99QzcvqciuJvn74YltpdGXPdRll9hgN0AsGoE88NyCb3YrIvo0t0Bga87GPxcp5/9p7RbyA47LI51yz0SRTO0NsSCIK6/7LO/aFGKfTmUTD0gnuY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=mlSWpLYC; arc=fail smtp.client-ip=40.107.209.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="mlSWpLYC" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UuTuv+O0dCY9+hQ1qWkAG6Elze9adBWSDRNIbVaJqIFvJogBIvMmXAwpOShWO/IDUz9VkdTETriBLovD3kvaB2kj7+0XcMpFne/DyQ6CRdAXYUWIEXOJBQxQR1xQ7pWJ9BitnvSBo+QrTt1CKfyWDZ4Xd2ThKi7rlO6e8eJmgDZA3NYnhJ8MPTcqCuBvxoDFqkUm7vYDwsIIwO5iyh4k60whmSsdwbFqArwLtld6iCXR+nzzNlaQFaM7uC32rChGahV6xoq6H/vHQmqfkPWZ0P+Dl/NnffHD8Gj5oXavxKTGy0aPnoAQUKL33LT+IIUgoGbeXCW/eUEWSLnBBREW9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Zro2hd4Ble5cJHXwetB3OFEMfa0I21dYEBYdyGm1024=; b=p6qPaRJriL4Or6GIXTAERJeox2aMC/V6lNsBkCDU2B+mcb+xlmCrK0S6+8yaiGmw2TziCwTrUr5bGbNfK54YW8nOOQj/MKRP2B6dVUamk+b1lHVWIAsMjpVzK5ck0jyQIg4EtbzJW6MfLBW816Gw9p+x+FhQIjLoJc5HnVN8TUDcSCfNpXK1MFF5KX5U3oenvhemIc+xvGi4vnAbO0bgwEaAk/rrPFA3vaY79ITeBH119F2P0fiJfDMVKPbbIq/SiFkx6TO/WyhnTHKYhKUL8TXAMPh5hnHX/ixN7O+78kvtNyL27udpxk5i2TIp/gSfkW6e5c6BuSMDxFYsoVrFaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zro2hd4Ble5cJHXwetB3OFEMfa0I21dYEBYdyGm1024=; b=mlSWpLYCdutoKhwJK50QHmMdUIKl+xwBsGJOl+73GIbU34b10+kogIGn5wokiMGWJs0zlMb1A8x5dVajVrFOEcQrlo4IPkhhh8JHv0DlNaoGMiz55GnJ0ZzsXr63evZ7Ol//GuXukZGHIwnUYK9K9EnC4Ah+YTGhb/u471Q9SXQ= Received: from BL1PR13CA0246.namprd13.prod.outlook.com (2603:10b6:208:2ba::11) by SA5PPF5D41D38AD.namprd12.prod.outlook.com (2603:10b6:80f:fc04::8cb) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.18; Tue, 23 Sep 2025 17:49:46 +0000 Received: from MN1PEPF0000F0E5.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::a3) by BL1PR13CA0246.outlook.office365.com (2603:10b6:208:2ba::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9137.20 via Frontend Transport; Tue, 23 Sep 2025 17:49:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E5.mail.protection.outlook.com (10.167.242.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:49:46 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:49:31 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 8/9] adjust NR_MAX_BATCHED_MIGRATION for testing Date: Tue, 23 Sep 2025 17:47:43 +0000 Message-ID: <20250923174752.35701-9-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E5:EE_|SA5PPF5D41D38AD:EE_ X-MS-Office365-Filtering-Correlation-Id: cdba7694-7d0c-4780-18ac-08ddfac995e1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700013|82310400026|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?QsYamSeARRtEJf8M93bgYAEskzEFS/5tCh3TQu6I11uQB4hKJfzm6FaecIHd?= =?us-ascii?Q?DQtPX90U4zhrvV9r31QHgaz3n/flYiRU0LsNgnvB4RgCN4VNM+x6OgyqMfnP?= =?us-ascii?Q?bKo9zgWdFoqwTCmZ9gg/+8rJzawL5iMk/cZ4/2iIHWhrb9Fa/pGMqvFX2/xu?= =?us-ascii?Q?f5VDPo1G3l0dhExa1aFHfPpvWuDPBTlCpg3Tca4m/m1YC8cyU6gz7NqGECEz?= =?us-ascii?Q?sJXfY3hAKO+0jZ6Q3BQEaOMi25C244VJ8NJWJ+DxKZfndTJHvwBuKbt5WMkv?= =?us-ascii?Q?ETWve7QlAW/Vdxqv8KaQdz6RyrOnS7hRdEwzHcz1j9E7/a+98qSNXS+RcEnL?= =?us-ascii?Q?jPtDtDE73X/dPB+ha8/ZjDzEQo71+SNyey1tt04xJHKe3OWInPWAB5o12wHG?= =?us-ascii?Q?UTVkqf1ZlyEYQPDR2oCcMIQj7A11v45DmEdJWzgl9T0Vg0ECl+DqRqKIudVi?= =?us-ascii?Q?GnfxWXR78SnEjvQG+67wgB7ykqQi4YAxufAa7AZ4WHowtFuy15hohJPBS3/S?= =?us-ascii?Q?lmzOQNJsa41bwKHzV7BdG9CrPmZGWzGBcC3Fce1yoUYqf5W5nzMpLs6GFYnf?= =?us-ascii?Q?4lqWAn7HHVynpFRzCYJCWeDK3YfgYg/kwnDQusUGYhAMaJDoj/59atMfe6Lx?= =?us-ascii?Q?JdnMUgW1GOz4NFofRteLlYLXgdQn03Cia7I3XKGa1hOhzeL5l14rvXSz+OTA?= =?us-ascii?Q?FSHOPuo1/Ut3iLgfOBYoNN++f4+LFn5mv/CppK8nB9jhpJduBqFBFhEx5Pup?= =?us-ascii?Q?axC2mL6a+jbMQQKHlP77Vhj69lxTuBPuL4uF2VBOBq/Zcs+6ATuUDczezKsk?= =?us-ascii?Q?y1TpDWMIcyj0Dk5OrKiv/9SHvXoyw/Jj625pZw9GxzUhQn4cAi6HEYgnv5ck?= =?us-ascii?Q?Bq+Q4XG63TFNdTz9jpneFYwGVAOmWH1Of1ypoaXc+rqpHolAgMkA1QesRHvf?= =?us-ascii?Q?cj70MdJ+EWN7LhNgtNRwZKwb4s9kGhxZEKq+fKXUvlvXzdETvIcYbSeJwJrY?= =?us-ascii?Q?JuYboA51o92mICal9RFyKXWunXOWUfmAhQDXr1z+i1LLANHsodztsYGdPv4X?= =?us-ascii?Q?27pJM+a35mj5Z/vlNR17JJeLaIht4RE5eIstthFmmFI+LiPTzwu8qKpWv2f8?= =?us-ascii?Q?4cq0hOdmi8jjwRSVKcREkakOrcfPxuS/gUcS5//vXU9aRHNKGS8eMAfEn+gT?= =?us-ascii?Q?NNAKkVtuO1MQNBhPynanSKHVbmctKhsopzImuwTitKg6J/oH+DR6IjEjHQpf?= =?us-ascii?Q?Bd6YXzMYYVf3nZp3gS6xqM/OyFVUjuIG61L4WBwNrjX0bEVYBtOnvX7qrxxq?= =?us-ascii?Q?MMIWvBITkswD6wBZBGkysTkQ0hqdtfmhERg1Wl2/vsj1E/oVa/RxCx9UHuTL?= =?us-ascii?Q?VwrN9wPYHs0R9VUU4dp4H6hp8mCjcetbe8vjaDRm8PYZbHsR+vfyh9i38bOU?= =?us-ascii?Q?5RzZPbArKjw/PtEcxXVeGKdPT2JGXbGoV7gZJcXEbyZlDp1PAhOdvskd5h8K?= =?us-ascii?Q?cMSwIHLzgxSWA2weQQ4qZVSeRPIF9Rpii6ob?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(36860700013)(82310400026)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:49:46.5981 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cdba7694-7d0c-4780-18ac-08ddfac995e1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA5PPF5D41D38AD Content-Type: text/plain; charset="utf-8" From: Zi Yan change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies. These are for testing purpose only. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index 41bea48d823c..7f50813d87e4 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1668,7 +1668,7 @@ static inline int try_split_folio(struct folio *folio= , struct list_head *split_f } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR #else #define NR_MAX_BATCHED_MIGRATION 512 #endif --=20 2.43.0 From nobody Thu Oct 2 02:13:30 2025 Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010014.outbound.protection.outlook.com [52.101.85.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A7802DA76B for ; Tue, 23 Sep 2025 17:49:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.85.14 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649801; cv=fail; b=dtcKDXzpF7Scy8vR6CqlU+RWbgFOtQ8Af5Trhj5AJUNdfG3aSwQsmv4cQwvP3TdXNZu8f6v4+Iq8iog6euuqUdQKYsZYJjk/+SroHdAwRVkxOHYnmZ39sVWj6i3wnydNy/aqBelwbi7bKIqWoFrLR9rWTfS0hNScKJQmPlrf6kI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758649801; c=relaxed/simple; bh=CfPJWIDGiTuhqlEwmHvATi7RM3HG4WEVkZPum38p3dw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Vgg0czQ6sD2lp4cetSuElLqmIKm9dvfRfqyGFaNXeyYejo8waIPgc1Py+jvvIrf6lgLv06NKBC2ChMnHyDBKu2oG6GvFWNvGP7UpWsXKoJMkTLHSM0Wu0wjmbr8a/hutlfEgvBjllaYHnzV5IfrJBoxfs5sVhXzUsOvVkrKf3Ck= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=bbDl/JX5; arc=fail smtp.client-ip=52.101.85.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="bbDl/JX5" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Wu/nCjwbvsTuse4P1mDFFQ9Wue/aWVK26bI4ikH3bq7kjLHM7y0yFyJ+pxQksu9Z2isz3CAejMsMcZwa6UAdRcAXMDJFHJ9pVJoS63DeCH0ws/fODRO4qYT4nhU9Tk+LZuL8qf0fBgeIdj5d+KY+4tQYd8jdX53qDohqneOAVs/HDXsVlajk7xy7JVYLZFCLFNK9Nljxc5J79TUtv1byzeGAPHkfxqXIq5CTO4y9keXZGttq9M7irkbtGfMQqDNWqC/N+lyqEJPLJogELzbykS8qNeSFB6xRAnv4Ayx+S6wPUsxXCeE+/SzICf53U0Yj5kj8AvZgJEOwG8um1WnRgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tn8s3EdMLzlUqkrxpuPjkMHYijrpzU78gf7sS2AGTd8=; b=ft40fE2CJaIXsXjXydI0DeDnO8cx77ZyV1igFTpQWPZWu4NcmsG2NIXqG7R+WIg0TPcUN5ciUgpHBKp6vbzjpGyi245h8nWXiDksAN/Q61oUAffsBvm+ZNC3a7n8+rYKuYFLJhOJqr0gt61XohKW9/AvBXw5Rlkdp78pZ2PSs/ri6Rcb4Ub4T7fo8icyKyeL0xK0mj487R9dqgb17o6YYrnWU/kUfL5Iv61bnEsaSYFJniahRp62+xMC5GJNf/A+iOZfYnWXcVi1n2gdweEfzkuXLh8Gh10AUCiTmxdL3r8KoJNn9cL75gBivCFBYHJBGGKiYEmeEIcHZVCybOqZBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tn8s3EdMLzlUqkrxpuPjkMHYijrpzU78gf7sS2AGTd8=; b=bbDl/JX5+5cAg8wPeuM8OvXXbEpGXMIgcgS6EPhGK8ZcchoU2oWnbIvT6eKBbrI82sCORzleE88pQfg/ZwADz7ixf2zs/p2s2gxLedg2aGQCfjHN6M3BLNTxQnyUrV13CHdboV+e8vF4snXJEPd3brObET5ooYFD36bnTOTUO7g= Received: from BL1PR13CA0265.namprd13.prod.outlook.com (2603:10b6:208:2ba::30) by DS7PR12MB8291.namprd12.prod.outlook.com (2603:10b6:8:e6::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.19; Tue, 23 Sep 2025 17:49:49 +0000 Received: from MN1PEPF0000F0E5.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::6e) by BL1PR13CA0265.outlook.office365.com (2603:10b6:208:2ba::30) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9160.9 via Frontend Transport; Tue, 23 Sep 2025 17:49:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0E5.mail.protection.outlook.com (10.167.242.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9137.12 via Frontend Transport; Tue, 23 Sep 2025 17:49:49 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 23 Sep 2025 10:49:40 -0700 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC V3 9/9] mtcopy: spread threads across die for testing Date: Tue, 23 Sep 2025 17:47:44 +0000 Message-ID: <20250923174752.35701-10-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250923174752.35701-1-shivankg@amd.com> References: <20250923174752.35701-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0E5:EE_|DS7PR12MB8291:EE_ X-MS-Office365-Filtering-Correlation-Id: a0223cec-a673-4acc-b9b4-08ddfac99769 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?eRuhbrYZNX9LnUM2tzyCiKOnann/bFO9RT42qi2duvDeCCA07O9tmZfvA1OT?= =?us-ascii?Q?cswGYEYXsJYvLxfKUdSiKAy6F41l3wLJF4MP8nCGT3Rt6fQEsQAhJF09QJ6n?= =?us-ascii?Q?VWRg9gfUyoF2vGuEP8G0zx0DZ+/aV6tazyHqUJyZ3x1h20h8tRP33GeNRjeT?= =?us-ascii?Q?R6kKMcDhfB5DUuZwFhxBMH4MFroYjQxAiqX+Uov1prTQE6fxP7K1Bv0Z7hcR?= =?us-ascii?Q?ntp4EmRdNTO9eOl765ij4Lmhd0wsyxsv+zznIvH+O+Azr5Ht9IN7RuzlsKEi?= =?us-ascii?Q?iCelbRdd3nfBJrsp/JHC/B4+cDWSPxvbo1V76c2TZaF5ZcdG0N7RHDEgI9e1?= =?us-ascii?Q?eTrP1XgKcYtKwXWJo6WqXJ7QZM/jeAr/yNxsT1HlyqEOzjejXRUp87iGxJ9E?= =?us-ascii?Q?D7FsT6R4kTSzYj5hcJS1QhbgJ0N5yHkYSQ0MG8/rYp4qR5tDlnrVl2uc2Sp6?= =?us-ascii?Q?wSEkoGI2jQ60DqmIT5CnqizmD/BKHAF6SIUgn0+eKzGz1b9E4Dr1hFJYGRgs?= =?us-ascii?Q?EehTq4O9g4m+VTNMyD8e8r1hO6GRIHPNW1eM9CJT9RkqH0E/lqBzvkUG2dlw?= =?us-ascii?Q?Nge4A4n+h/8N7nIhpUjr7A1+f4ifKbpRvy1IYEusQgod+MtJ6Y1n/3BIA1Y0?= =?us-ascii?Q?6RFuR0e8OpnM/UcsffeTyJd27fWtDupLx6h85QDfdpaI5K3J4Ij7OplQjbfX?= =?us-ascii?Q?bSTVSFn1yjZGGb6oixpbwYTPFQyFFvi42KuMcxt9UMAFM0dhVEvcI19PRazR?= =?us-ascii?Q?cJGQbH9QK8zuzyWM4okfTg5zO6SBkEci4CFIlO3tB7T/qg2r767bKmAQ05oa?= =?us-ascii?Q?44M6RjZxgyX2aqpon2I03rZD6qTvdZIXid3UXqpQwwMWTiZrirk5M1l77b43?= =?us-ascii?Q?F9eGw1/80/bkijwCL5QHFkvxG4aVmaH3V9ZhC/C0pc28DAVmJivcS2BLZkcn?= =?us-ascii?Q?ev9ajaUDaXva4V1DZ+cdbiHTAl1FHNEQwXh4HweSE9RslTRAttuFGFdodsrT?= =?us-ascii?Q?ZCtnu6tWvrEY1ECuOb9cn+8o7Hakmnw7ZI/RMV/472OZn5QsPXhSPlZdO10K?= =?us-ascii?Q?VbUZapv/nbisvZCzarOfejj/1qDSKxaQ4nVoO/Of0QfMY2yIsSuePXtPZ7Mc?= =?us-ascii?Q?Ib8LHNX7hHiJrEsjJqVbgJk3Yav8QygteXV3U64zJAI0XeIgZp0Ia3jFvTe4?= =?us-ascii?Q?9Ey/tZYI/jkHyNijMzdBq/AbSgVossnqKDFwmUFBM0Vj7/L8f1lZROQJnv/I?= =?us-ascii?Q?uqrbqjt52qzoQ0+5bjGPaGrG/WYQ9qWpCpwwLpUVWSJXj1PjM8oqXk1D+z7X?= =?us-ascii?Q?L4x4YGEyTlKVRN+J7tX/6Kzcvgs6JExzHsto6l3SBAcmWrOjuA6Lj86zdbrc?= =?us-ascii?Q?bTkkireVQOyCx04cJhhsxw6khbOlNJ13D2+/GZGnIIs+iJd8K7kfJQbAUBcF?= =?us-ascii?Q?6UymzMGgyGdeMjKyBzNA2d8cTcF2YfTM9/lfB4m7OZS5CtOq22MBfv3J7d3Y?= =?us-ascii?Q?ihUvByRIsuvTSeC2qKsL45Oq9Iie7rsLn0Ij?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2025 17:49:49.1705 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a0223cec-a673-4acc-b9b4-08ddfac99769 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0E5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8291 Content-Type: text/plain; charset="utf-8" Select CPUs using sysfs For testing purpose only. Signed-off-by: Shivank Garg --- drivers/migoffcopy/mtcopy/copy_pages.c | 76 +++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 3 deletions(-) diff --git a/drivers/migoffcopy/mtcopy/copy_pages.c b/drivers/migoffcopy/mt= copy/copy_pages.c index 68e50de602d6..e605acca39d0 100644 --- a/drivers/migoffcopy/mtcopy/copy_pages.c +++ b/drivers/migoffcopy/mtcopy/copy_pages.c @@ -15,11 +15,37 @@ #include #include =20 -#define MAX_NUM_COPY_THREADS 64 +#define MAX_NUM_COPY_THREADS 32 =20 unsigned int limit_mt_num =3D 4; static int is_dispatching; =20 +static int cpuselect; + +// spread across die +static const int cpu_id_list_0[] =3D { + 0, 8, 16, 24, + 32, 40, 48, 56, + 64, 72, 80, 88, + 96, 104, 112, 120, + 128, 136, 144, 152, + 160, 168, 176, 184, + 192, 200, 208, 216, + 224, 232, 240, 248}; + +// don't spread, fill the die +static const int cpu_id_list_1[] =3D { + 0, 1, 2, 3, + 4, 5, 6, 7, + 8, 9, 10, 11, + 12, 13, 14, 15, + 16, 17, 18, 19, + 20, 21, 22, 23, + 24, 25, 26, 27, + 28, 29, 30, 31}; + +int cpu_id_list[32] =3D {0}; + static int copy_page_lists_mt(struct list_head *dst_folios, struct list_head *src_folios, unsigned int nr_items); =20 @@ -141,6 +167,39 @@ static ssize_t mt_threads_show(struct kobject *kobj, return sysfs_emit(buf, "%u\n", limit_mt_num); } =20 +static ssize_t mt_cpuselect_set(struct kobject *kobj, struct kobj_attribut= e *attr, + const char *buf, size_t count) +{ + int ccode; + unsigned int cpuconfig; + + ccode =3D kstrtouint(buf, 0, &cpuconfig); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + mutex_lock(&migratecfg_mutex); + cpuselect =3D cpuconfig; + switch (cpuselect) { + case 1: + memcpy(cpu_id_list, cpu_id_list_1, MAX_NUM_COPY_THREADS*sizeof(int)); + break; + default: + memcpy(cpu_id_list, cpu_id_list_0, MAX_NUM_COPY_THREADS*sizeof(int)); + break; + } + + mutex_unlock(&migratecfg_mutex); + + return count; +} + +static ssize_t mt_cpuselect_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", cpuselect); +} + int copy_page_lists_mt(struct list_head *dst_folios, struct list_head *src_folios, unsigned int nr_items) { @@ -208,7 +267,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, } =20 for (cpu =3D 0; cpu < total_mt_num; ++cpu) - queue_work(system_unbound_wq, + queue_work_on(cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); } else { int num_xfer_per_thread =3D nr_items / total_mt_num; @@ -245,7 +304,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, dst2 =3D list_next_entry(dst, lru); =20 if (per_cpu_item_idx =3D=3D work_items[cpu]->num_items) { - queue_work(system_unbound_wq, + queue_work_on(cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); per_cpu_item_idx =3D 0; cpu++; @@ -276,6 +335,8 @@ static struct kobj_attribute mt_offloading_attribute = =3D __ATTR(offloading, 0664, mt_offloading_show, mt_offloading_set); static struct kobj_attribute mt_threads_attribute =3D __ATTR(threads, 0664, mt_threads_show, mt_threads_set); +static struct kobj_attribute mt_cpuselect_attribute =3D __ATTR(cpuselect, = 0664, + mt_cpuselect_show, mt_cpuselect_set); =20 static int __init cpu_mt_module_init(void) { @@ -293,10 +354,18 @@ static int __init cpu_mt_module_init(void) if (ret) goto out_threads; =20 + ret =3D sysfs_create_file(mt_kobj_ref, &mt_cpuselect_attribute.attr); + if (ret) + goto out_cpuselect; + + memcpy(cpu_id_list, cpu_id_list_0, MAX_NUM_COPY_THREADS*sizeof(int)); + is_dispatching =3D 0; =20 return 0; =20 +out_cpuselect: + sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); out_threads: sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); out_offloading: @@ -314,6 +383,7 @@ static void __exit cpu_mt_module_exit(void) } mutex_unlock(&migratecfg_mutex); =20 + sysfs_remove_file(mt_kobj_ref, &mt_cpuselect_attribute.attr); sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); kobject_put(mt_kobj_ref); --=20 2.43.0