From nobody Wed Dec 17 05:28:29 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2057.outbound.protection.outlook.com [40.107.244.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89CD92135AD for ; Wed, 19 Mar 2025 19:23:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412205; cv=fail; b=ry+tCLriuH5193vQndYT+6eVCEJCkVLFOtwPFaP8w4i9MxRg3IFgh/26I4lVySevCRNFFvUTpSd/Ac5iApL3lIJMnlLkTDXb4kt0QjLNDZxj8wcbRBCYzLpb1SQCXRVYvSA+vA6CnTI7QYM1RAqbXjybHeiWyjrVd50rt6iZ19A= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412205; c=relaxed/simple; bh=09FRraZaxIQjrsfRmvAb1GVaTFpjltkNmF4E/S6+9Vo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JYSqM/polXkFZss0Gai+2Lf4sWzNTXsWGTAuysxqcTRHUzcwL/eNRcYUStpgz7WoQhncyDQhdPAXYxwcIHO+kL80b3G/wPm7YQUV7v7t1G8z2Kn6qzgAOAB2Ds3KnWcnLG2X/B3VKSmHk2VMYUCqU5m9E0VQ+1oy24XGFYLFkB8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=REd7jNs8; arc=fail smtp.client-ip=40.107.244.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="REd7jNs8" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TBJqvH37che25yt2cYzIZGRPrKVkMD1fXZuQl8GsY04kK97ha0rgH0mSuGDAl7/RllrcuHawYedsbP/sMQfZYdqywovqqNYfRdqcSq3AThpLRuCoiKC/Y3+vDchjIJ1mOkBBjNdaVd8gHDObb5M26peo+xv4q2ffX2kkMG90E0WWuNnOMCkctxIIFYiiszeoZLjSNhehJRCyF6indTtWUbmKfgPUFPJ6o3d5yGPTMgFKyJcmM8e2qN6nYTA9950O6wAkf6iSirIOcmK3+wcvvZKSfowNl+lvaEN0ZSvjvJNFK0fEMOH7GHpSrdCzkm92u8lOOo1o9Vzyx81n1eNAHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SZyoWMY8GPkC6c+QiCnwrogSvPL8DBaC1BvGNUgqyIg=; b=DZ7ssxCoLNeb3/tiZQ3uD8g/PGjEbrOQZWrelCnLeGp4CI0d43aQNazRwJ1jLsZ3qScSEon7wKS+rq7+yymV0m55QJlfEbzrl0cJj7itt9lP+tBbIZew9Iw4Erb8EszE01ZY3mDbJGKmy7Shlg4lOtMu1oK+DH5v0ZjEmR88DCuRalsowOlyL2OMirxdcIyF37HHJ8WL7Hl48T9zqRUYkNY8vdEAEUWVSAaZyfHxR8LBiNS9rbtGrMmJrcJP094TAjqTCHbbgNSWJpa2dGHybZKLCR7+IdVXcybya1bx3wGhGx0fmLad/1RCQopRv6bIoTKsijx26fO76tVfTlwyxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SZyoWMY8GPkC6c+QiCnwrogSvPL8DBaC1BvGNUgqyIg=; b=REd7jNs8gHGP2mKaPvk8TYELAzgTvhFxbCAHUnwRWxPXWfx+JDYGIa4+Bh4ugqs6e6YvIl7esWQML58gL7fVcu7vvc9P42DIhc8IQ/2q0NDNwbPii5Xk01shvIvoXzM5NWjKSDvTfSmpOSGCeFZcHOpR/O6DHGLl2PgWl4TuItg= Received: from CYXPR02CA0026.namprd02.prod.outlook.com (2603:10b6:930:cc::9) by DS0PR12MB7993.namprd12.prod.outlook.com (2603:10b6:8:14b::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:18 +0000 Received: from CY4PEPF0000EDD5.namprd03.prod.outlook.com (2603:10b6:930:cc:cafe::c) by CYXPR02CA0026.outlook.office365.com (2603:10b6:930:cc::9) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8511.26 via Frontend Transport; Wed, 19 Mar 2025 19:23:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD5.mail.protection.outlook.com (10.167.241.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:18 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:09 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 1/9] mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move() Date: Wed, 19 Mar 2025 19:22:04 +0000 Message-ID: <20250319192211.10092-2-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD5:EE_|DS0PR12MB7993:EE_ X-MS-Office365-Filtering-Correlation-Id: 0dd7a6a9-10d7-4d68-f27a-08dd671b8106 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dt0tl2cnY7vDBfpDdyeL2xwnBUyZT4Je/1hE2mS3nxOgahtC1f64Myv/XqTU?= =?us-ascii?Q?txL5ZhxVQrSdvu6i+KD2rsVI0FSrGiorB78wJocLI9i6P2r7RlhPPVOfibUu?= =?us-ascii?Q?ftRbYAkXiVQqx1XD0u5tjuR/P5SsZnDjLWVou7tPo+/Mn9luVlSDbzMcvp3F?= =?us-ascii?Q?ag2x+Lc2H3CV3WXSfDwmYHEmPEC8T/qSpQnjh638rxvBMLENyW+jEryQIJ7j?= =?us-ascii?Q?cwOawQ0BJafk9p8TpzbxH2iwEgxNSshf1sN6KC8xBU9zUmJLIbf3H8UlZOr0?= =?us-ascii?Q?E4rU2AV+xy6LFhbKTpsqQTNqv2XARZ3iQMOZ44TfXFOfICS+XYei7zW4Eras?= =?us-ascii?Q?gwNCwKZOkLH8EpZ9egV4tMQ4s3+ME8Dg2j/zDz4bmhqbu0xjaAJkyh7FVfhm?= =?us-ascii?Q?7RlZAiaZI3aFP1eUwzuaW/3PliAPS0bswbZ0pQrYaJobsn460B9kH1bdnfFX?= =?us-ascii?Q?Pfr+SfewDL0UdOwpcPtz86Imu8EtZljDsZlU6070Mwq0YY/z6hUqtB7oh764?= =?us-ascii?Q?Yc2x3FDSnYacerxHDL7XbH5bmA6b8Xdjqjzjuru8jkPGZb4uSoLfumYaGobM?= =?us-ascii?Q?xo4QbFuEse3CA5LJ1O6KB/k8sWdxiA1JXb2ZCgr0sTG6r4sYCpMhsu6O7FsJ?= =?us-ascii?Q?DEXRGoGfQL7EiaUIXX65y3J+p7JwFPGdoZZCwgfca0sLG8Wul8YBb5S5vJYv?= =?us-ascii?Q?UD28gTk6KHCQqUffQx/NhFVsT13swKG0FxHjaDdngXtdVbjyc392z7dgZoIH?= =?us-ascii?Q?09GzL6tuTzcFMliyXaPYBE5bENRT8dCkk2oI0NbaYG/qh+EFcZfwi85a1GWv?= =?us-ascii?Q?Ah4PsftL5ybVEtoH0m/Fw0/1G+jJTIB8gVSGGSih7hs1N6Zd/rpSDhNgb7jR?= =?us-ascii?Q?Fodyasd8vpYkdW11s9rnHi0K2x/QusLg//xGecTOB4qKBoXWK1kIfuQBWoxz?= =?us-ascii?Q?k1nQtuEvMv1HbX5UGmi705NLw39MNhrz7kMgzIqnkR/QJvyTc3kiL7oiEDXh?= =?us-ascii?Q?tONcITOyIg4G/Y08iyL4GOire3cw9MHvEp3RfLKyh1av3MQU1G60uQO/xj75?= =?us-ascii?Q?Rv4/W9jP8izvxZuVPRxJ1/WFwm64DIJ6dqeL5iPd05hvXzAaF6hWSUj4jkoy?= =?us-ascii?Q?+SmRtlItJ/hQZAs49hWULIcxkgUowB5cFBr2S6w7nW2NBpMJ4owGVmWGGbu/?= =?us-ascii?Q?d6iM291EGnRwvyjQSeTE6z3wsEXiI7st6/bP5QpS9AIX5M3y9XSQ35NPG6uL?= =?us-ascii?Q?YkUYjzRTIYQcU+Kw5Gkj764R6ddZKnTDzMObCHgpHS0gvKhvmHIh00xCtlB3?= =?us-ascii?Q?5owofGBAoIXoydI+WWXESkY5qg5aTmJ1cHIL3HKl0FRJhKsuousg35JwYNSA?= =?us-ascii?Q?QDBAyZistjtvZYTyH9jJGoFVEYTqmLLiNDwg6JEDBP/aOJ9bP9Mf6U2p4YO7?= =?us-ascii?Q?hX9BApM9s1DCV9elrOrhrWmKgh5AstjAnRNSO6r6x2DGjJiK/JSM3JXSG+yO?= =?us-ascii?Q?IoVxfdqY0Y17Ql8=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:18.1410 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0dd7a6a9-10d7-4d68-f27a-08dd671b8106 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD5.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7993 Content-Type: text/plain; charset="utf-8" From: Zi Yan No function change is intended. The factored out code will be reused in an upcoming batched folio move function. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- mm/migrate.c | 101 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 36 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index fb19a18892c8..ce7ddc56e878 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1014,18 +1014,7 @@ static int fallback_migrate_folio(struct address_spa= ce *mapping, return migrate_folio(mapping, dst, src, mode); } =20 -/* - * Move a page to a newly allocated page - * The page is locked and all ptes have been successfully removed. - * - * The new page will have replaced the old page if this function - * is successful. - * - * Return value: - * < 0 - error code - * MIGRATEPAGE_SUCCESS - success - */ -static int move_to_new_folio(struct folio *dst, struct folio *src, +static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, enum migrate_mode mode) { int rc =3D -EAGAIN; @@ -1072,7 +1061,13 @@ static int move_to_new_folio(struct folio *dst, stru= ct folio *src, WARN_ON_ONCE(rc =3D=3D MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); } +out: + return rc; +} =20 +static void _move_to_new_folio_finalize(struct folio *dst, struct folio *s= rc, + int rc) +{ /* * When successful, old pagecache src->mapping must be cleared before * src is freed; but stats require that PageAnon be left as PageAnon. @@ -1099,7 +1094,29 @@ static int move_to_new_folio(struct folio *dst, stru= ct folio *src, if (likely(!folio_is_zone_device(dst))) flush_dcache_folio(dst); } -out: +} + + +/* + * Move a page to a newly allocated page + * The page is locked and all ptes have been successfully removed. + * + * The new page will have replaced the old page if this function + * is successful. + * + * Return value: + * < 0 - error code + * MIGRATEPAGE_SUCCESS - success + */ +static int move_to_new_folio(struct folio *dst, struct folio *src, + enum migrate_mode mode) +{ + int rc; + + rc =3D _move_to_new_folio_prep(dst, src, mode); + + _move_to_new_folio_finalize(dst, src, rc); + return rc; } =20 @@ -1341,29 +1358,9 @@ static int migrate_folio_unmap(new_folio_t get_new_f= olio, return rc; } =20 -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_folio_t put_new_folio, unsigned long pr= ivate, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) +static void _migrate_folio_move_finalize1(struct folio *src, struct folio = *dst, + int old_page_state) { - int rc; - int old_page_state =3D 0; - struct anon_vma *anon_vma =3D NULL; - bool is_lru =3D !__folio_test_movable(src); - struct list_head *prev; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - prev =3D dst->lru.prev; - list_del(&dst->lru); - - rc =3D move_to_new_folio(dst, src, mode); - if (rc) - goto out; - - if (unlikely(!is_lru)) - goto out_unlock_both; - /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1379,8 +1376,12 @@ static int migrate_folio_move(free_folio_t put_new_f= olio, unsigned long private, =20 if (old_page_state & PAGE_WAS_MAPPED) remove_migration_ptes(src, dst, 0); +} =20 -out_unlock_both: +static void _migrate_folio_move_finalize2(struct folio *src, struct folio = *dst, + enum migrate_reason reason, + struct anon_vma *anon_vma) +{ folio_unlock(dst); set_page_owner_migrate_reason(&dst->page, reason); /* @@ -1400,6 +1401,34 @@ static int migrate_folio_move(free_folio_t put_new_f= olio, unsigned long private, put_anon_vma(anon_vma); folio_unlock(src); migrate_folio_done(src, reason); +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_folio_t put_new_folio, unsigned long pr= ivate, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + int old_page_state =3D 0; + struct anon_vma *anon_vma =3D NULL; + bool is_lru =3D !__folio_test_movable(src); + struct list_head *prev; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + prev =3D dst->lru.prev; + list_del(&dst->lru); + + rc =3D move_to_new_folio(dst, src, mode); + if (rc) + goto out; + + if (unlikely(!is_lru)) + goto out_unlock_both; + + _migrate_folio_move_finalize1(src, dst, old_page_state); +out_unlock_both: + _migrate_folio_move_finalize2(src, dst, reason, anon_vma); =20 return rc; out: --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2048.outbound.protection.outlook.com [40.107.220.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79ADB2135DD for ; Wed, 19 Mar 2025 19:23:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.48 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412213; cv=fail; b=Omx5QINgPrUUBIkCaGwt/3sA7DjlghkcEkbrHYhe/6FQ0l0P/SYkk4NqO8Cdxo3QrZmdnoLdKA8VJdVOTCehl9DZFxjV+facB6kXi1Ue4B3oH9kMYzwI7mzvnn2std42gFu8VTR3o5ZBXRa/8x41PAct8TCMFZRQs/3AWTQ57b4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412213; c=relaxed/simple; bh=sp3U8H9H6FcRtylpSfqWW+tzI68zWa3aEkPvuPUj6/c=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ugkSIqArtXJBPnCF5NtMtl47cSMNRBe6HF+DquLJ+9NVlc+r2cKSaxyCKlFq3p/hBGSQXIfL4tlQK4kwlHFHzR7izezZtiGgsG1idPbmsVj38rPvtqH2+Q2SfJLB0oZVw3N+UU46Xljl2QZgP/WBf/e9Y2mXUP0rs7nRSTI9G+s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=qeGOLX6i; arc=fail smtp.client-ip=40.107.220.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="qeGOLX6i" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qWvpPFWSghZpx18cZKyIHWG1NgS9cWN6z3l+oQn0V1C/89RZh94bnmjvCrf5hg5RJm6IwlIeypVYYChFPgEc8PG2v7SfazhJAXKaZ2+HOahyymok6QY5XgVPLZfGvP8G+L25siTzlSzM1HcXmFuu3IB+bHFqO0Eum7mrkO8/N6ICqPUUtLuyKcv6uTQGa8ZwYnMUR5Amz071WZoAE2VOxv5/Son3l/67kI/nf+tS7Q4ZCHgDSrlhBZjKLoN0lhw/b3fseW5VJkDeZSFwA6Y2eCsMfzepEwjzZRwQmAPCpXW5evqdb72eCdumXj8PfmjWnWeh4WU0NOgNYT4dnsD9VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+H8IxYYIa9GO9xurZ2fEGRkp5bA92jEu5XUKX/h7S5M=; b=FO8fdlAbAx25OO/NgOlXFEilYXqm7J8V3mF13OmFVU5ttqq7JnlZK94FLOUlEAw61J3lme/tIRqAuRL2Sb+7bWKzwMV+hO9TxhWRlc7l5bMLusF09f+Py5K+QM+niQIlCKRVY54u7gFEOXK+SROWsQS1kzmlD1XS3b9oGkMlrWsvDmGs7/eTBpg2bSB+nw7oz/5mWWaxR4bV9PFUyRSuQFOloYoiYIaCWaUZ5JHWWFzvvyIIMnkbdsZJosoi8LG8ReCvAGtSx5iqGVfgyI5jA4qwFjhwWWtSYmw+1GVj3OO8w18vqPnWq37HbubmDUjpfJVMgMzDc0WMiVXat1Ygqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+H8IxYYIa9GO9xurZ2fEGRkp5bA92jEu5XUKX/h7S5M=; b=qeGOLX6i7A1PxDVw91iXZDbU2fZeeIa06xpcZckcsHd9x2LVAiOseRt5bMrxZZ6ZvRCAuBTywUVryvJ3IDuKhICbFqSMLB0LFGr5+jwBmRzS0/XQf9xDW2ZaIIGT3CzWRPqoGkklOXnAmNXGw0IvOFhdIzWsSFK85zRbiaADGSI= Received: from DM6PR02CA0129.namprd02.prod.outlook.com (2603:10b6:5:1b4::31) by IA1PR12MB6356.namprd12.prod.outlook.com (2603:10b6:208:3e0::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:26 +0000 Received: from CY4PEPF0000EDD7.namprd03.prod.outlook.com (2603:10b6:5:1b4:cafe::30) by DM6PR02CA0129.outlook.office365.com (2603:10b6:5:1b4::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:23:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD7.mail.protection.outlook.com (10.167.241.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:26 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:17 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 2/9] mm/migrate: revive MIGRATE_NO_COPY in migrate_mode. Date: Wed, 19 Mar 2025 19:22:05 +0000 Message-ID: <20250319192211.10092-3-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD7:EE_|IA1PR12MB6356:EE_ X-MS-Office365-Filtering-Correlation-Id: 300d8667-9e74-45dd-d7aa-08dd671b85cf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?kWnrc7ERbG4FxpzsQLmD8IqYETAHyIFAWNK2mLbzvqcLB+73Jj/iorg/oK1k?= =?us-ascii?Q?zal4e4nth2qqUCbi4gyccdnMHoFpB2rjZ3OXDpvAwh4AxiLKiwJ5DjD7EwiL?= =?us-ascii?Q?6YPS7eAXpcEE7jxAZDgZyQmggTplxGVwZXCgWuQABbapLj8vXdwtjpy4LT7q?= =?us-ascii?Q?odl/aqIfnles9xIQVqVZjfXYlzcsQH7pzE0yvkinaTYNy1xrBkK/fvCgufvC?= =?us-ascii?Q?92x/w128jlji+CmnE+5I4H7uAhFMNzeNzbRPI5O07vywB6/URGaEY2U+vE6P?= =?us-ascii?Q?o4EJFfeB+Mdbd2/8JGZoK8Vsk9JavWPyrwJ9/t3kbfMLZFOE7gOe1Yq3/xEr?= =?us-ascii?Q?q7bGXYbO8ebWLav7upT4zrOmCTUluLAqXqHJpvWgEFwhK1PmMWz884eOOwfJ?= =?us-ascii?Q?9MtZh+1HM2VJ+2UFfi7nByPFpNGXDfD/VXQGaOElYtvo2xKTL9EKt64c0aDO?= =?us-ascii?Q?sNGPv0m2Wvijw0afKMV6Pm6PTmeyOy1PVDhXP5T1VEz2BMw4Cz0S+H7xk/z5?= =?us-ascii?Q?1ndKfXeb8n1FKJWi3564Zdn6CvplNuRwOz0MbiCU5rdTpdfYQ31wCKSMMGCK?= =?us-ascii?Q?z4TDunMrQQYkX6sK8J16jhYgIEfa2/Hu0+QYieJbehAJxq/oSrb4oGvpp8K2?= =?us-ascii?Q?u3Cxt4wLX2a17Ku4UxBb831Zqn/66C0DyMF6+IrwTCun4U3jVpZJNJ92+Wrk?= =?us-ascii?Q?lNqBurB/RQ3g6zomdgqBGfL98HUTckSIH14bNNrnrEsOopb8VxEYAceKQ+9g?= =?us-ascii?Q?U0OBmA7ei4ojxkEcbjrGoJ3E1vuzaUAR4GH6uYpBYAYnC15vhMTzobsxYUUz?= =?us-ascii?Q?X4kjfZMvxLCqlDLvB1TUaJoN75fZ+G6dEkNb4Kjhp6VhL4IG93sNEuTY/L0D?= =?us-ascii?Q?FA2TZ+wSkXxj0C9WoSd7VnwkiD8PxM0rczrQRL7nwlsRrFpuXzsuxwsVuxVL?= =?us-ascii?Q?+GHveudqjhAPq9xZdx9Ba6GoJhzNoVozQV74SiUYCftChW+KoN4BOnYhBEYj?= =?us-ascii?Q?MKpwQuxpP+DVEWSHRtv9SJ/9YpGeGv+j5o+oCFK4huhjJoiS15Jh6uoI6yR8?= =?us-ascii?Q?ZpDxa58BxjOs9q/F7WebEyXcJmLD4IXu8hIpOEebZq+9YbBAUoa23MCvSTDw?= =?us-ascii?Q?EfoIVARkF01IqXuJOdgQJWxoPPHq2bXYZb15U64bx1/m4A8oKX2LjbxuBr6D?= =?us-ascii?Q?jEzNJZq0WIPhFOa51T8vENGLhEcpLjDZRQ63/mpKGOkt07jN8ljFRoZCR5sd?= =?us-ascii?Q?BJLlZR0FckmnQSWjp5oKxpUkS9n283MOjf/lUw7X+NOjcCgehAxU5Hnb1Q8j?= =?us-ascii?Q?sSexFLR/qB2+AfTnV1Z2v2NwrliCZgW6aUVZdx2l+TKWLwpRuomP8KD48Wzp?= =?us-ascii?Q?qizOYzxIW7qTdlFKSIoy1Il9S+DRSAjrxzlNkdJW508SdsSHDsUYKE0RsOLW?= =?us-ascii?Q?ao/wyO3Do7X/npGiCYA40kE2UAS3Fxjw/Z/iyV9yo7Q3KxwGt+A85kd/KDJX?= =?us-ascii?Q?DUrp8fa231+y0+w=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:26.1684 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 300d8667-9e74-45dd-d7aa-08dd671b85cf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD7.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6356 Content-Type: text/plain; charset="utf-8" From: Zi Yan It is a preparation patch. The added MIGRATE_NO_COPY will be used by the following patches to implement batched page copy functions by skipping folio copy process in __migrate_folio() and copying folios in one shot at the end. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- include/linux/migrate_mode.h | 2 ++ mm/migrate.c | 8 +++++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index 265c4328b36a..9af6c949a057 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -7,11 +7,13 @@ * on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages + * MIGRATE_NO_COPY will not copy page content */ enum migrate_mode { MIGRATE_ASYNC, MIGRATE_SYNC_LIGHT, MIGRATE_SYNC, + MIGRATE_NO_COPY, }; =20 enum migrate_reason { diff --git a/mm/migrate.c b/mm/migrate.c index ce7ddc56e878..0d40ac069cea 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -757,9 +757,11 @@ static int __migrate_folio(struct address_space *mappi= ng, struct folio *dst, if (folio_ref_count(src) !=3D expected_count) return -EAGAIN; =20 - rc =3D folio_mc_copy(dst, src); - if (unlikely(rc)) - return rc; + if (mode !=3D MIGRATE_NO_COPY) { + rc =3D folio_mc_copy(dst, src); + if (unlikely(rc)) + return rc; + } =20 rc =3D __folio_migrate_mapping(mapping, dst, src, expected_count); if (rc !=3D MIGRATEPAGE_SUCCESS) --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2081.outbound.protection.outlook.com [40.107.223.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AFA3211A24 for ; Wed, 19 Mar 2025 19:23:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412222; cv=fail; b=PcelWEnIbitBPgv8FKUni5mY6tZh7roJX5LpitktYYtsCjWT7prw+VXQc1JssFiCbYNwsN7tKGtDrXegGCW/uW8rN/LrfK+jItAGBBZyt0ev87lJ5VjtJI46/Lw5O2alhIS066Zq5Z1/Lo57/ndo0+AE4UrytqC/4+NAZejd3lY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412222; c=relaxed/simple; bh=SJELEDpGcOYodGOsAJUN+TQP9INqTp83IeT9MgcvOMQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Nl4j47fKupepTALwuASLj6BV9rUySwcUA/qZYtAvQ6swZxWaH1Fs5kJTbLxBE3pE4xzJQr7bzMhJtk3oRgEN+fvyjg48DfsDx75tGVm2Fs81IkpVmgG7NEIoNYcrLwQg1UINmoMZ6Ogzo48Q3AYUI6ZM1PS7Nylwkun/z1JYg5s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=WlhflszX; arc=fail smtp.client-ip=40.107.223.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="WlhflszX" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=m4dUnyTuEVtWO5ixBvw1Mlx/mUrHEoRGfYswiiTDW5igzi0YAA7J+qhTKyW1jhapGg7+e6JhwpIf/ZAL6c59IUe7EzCvJeAMFg2yiU3Vlm6EnLyia9MyJSsMfPRi3uiUxhYApcfL7z+jEbufFBTXyiraQRMxsUobD+0XTcvbEApHMTHTAFXywexbOHnRQZAUd0uvAwnKjJvaqoUrsqyoeF/WbrYknGBSVn0olGsReOnmUWZvtppbXhMOuUJtJ+7KWhfmJ9IcnyEDLtcfTUR8iUrXA/dGWM6DoSbbohrItnu8t42sUWMMnzqd0D3StPD4M+xw5gtcnoRj2uNciyPbnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R4pTY4qimGp+eX/PIhEn6Rb2JoweUqEplJQfgDoLjGM=; b=kZQB2Dgm8XbVoZ0op+HJRnhpJALAg//kmIQrrn6AJijxM5d6iEFDzmrIzNn1WlFcBfJ3LKHl8svphY3/ZcBbdvjU6F6HPvhryJc2VK8BfywwabjBlxnOfqG8UNTkdtVIqTzODXPBzZ/09zOAY4IU00qQ/WpUdxxYM0jbXQAQ0VMFIg5JTig7mqzUBw/rzWmgwY/1/9AQvNsx8J51Nba3rlDGqclQkhgaiPGqkDRyMWbr3cfqSuL17z12KcuG3X79xk/8eHaNuqZx4AC82wziazCtTTTm6h92zFbA/z5FZwG4CSQD3hooxYVpnMB4sev3KbDeoV+fvNL1LZakKbZXlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R4pTY4qimGp+eX/PIhEn6Rb2JoweUqEplJQfgDoLjGM=; b=WlhflszXByX/8Yyxr6+7//rUPB0Vwe1e31YteQOOvwUOGg2WE37BGVqiyHYpBtOmW7w4pcn3ppCd9SNP3Zz0Yvzp37V+vfOnWkjp8h8Zj+wfUrl6zdopUUS9fJNnL3/ELD+uFF//uvPyQG4/Z6oo6OsrQf72QQMGvLt35K76Wlc= Received: from DS7PR03CA0061.namprd03.prod.outlook.com (2603:10b6:5:3bb::6) by CY5PR12MB6480.namprd12.prod.outlook.com (2603:10b6:930:33::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:34 +0000 Received: from CY4PEPF0000EDD4.namprd03.prod.outlook.com (2603:10b6:5:3bb:cafe::b7) by DS7PR03CA0061.outlook.office365.com (2603:10b6:5:3bb::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:23:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD4.mail.protection.outlook.com (10.167.241.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:34 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:25 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 3/9] mm: batch folio copying during migration Date: Wed, 19 Mar 2025 19:22:06 +0000 Message-ID: <20250319192211.10092-4-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD4:EE_|CY5PR12MB6480:EE_ X-MS-Office365-Filtering-Correlation-Id: 0eba36b8-6c5c-4688-3f01-08dd671b8a85 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?5K5op3W+FPxeOfUYOGxG0vXfoJ+uNKB5RMlWqQIu6QsHSFIfHWlmjyJKGEF8?= =?us-ascii?Q?C5hQw3aos53j5x0zVfoLuM8nLbR0CcNnEHlj7VqXqKhrhfTuPby1mBAl3y5W?= =?us-ascii?Q?VizIqWGcYuJdb9ij4utyC18JyH5gBYcnROFfNleg4wn97khtluO7m/nVKCpY?= =?us-ascii?Q?HGi/gT+fZ1Xv4Lbn9wHuW/xwfklOz58zW0cssENnvPoFkCTiyhE1RuUXDO9w?= =?us-ascii?Q?ONvBCLVLns2jB6P9GrKbxV+IJLn+V9YtA3zJnD+8SqQZ5vGPLYN2GxNHPCiz?= =?us-ascii?Q?2F+ZMADtcuSXh2tKzwXgA9eytdOVNODXlMLDshytuy3fJ26vem4AXdnAdbiL?= =?us-ascii?Q?2iQ6U2Yotll/zH5iLM3+I5CAzqkOdycmND4GHO/1mL3/uhuXenZqLH2OBhJU?= =?us-ascii?Q?g1PnkpTjH3sIlwJJOH1dY2l5f/BORbqUi9dN/fxOHJXzRWceoEC3qC7Pzxa5?= =?us-ascii?Q?K4Z2qtkUaDAqBYhdpYHLUp2rpVrhHOY4kIcEcZWpwDOaWuMBFs05HcHbXy1+?= =?us-ascii?Q?rZ837z+MFm2tIXowTKVSeScxFzRo3FxCEAeqhQM4CQJ5DNXvXjPSjPlFzNjj?= =?us-ascii?Q?vHc7Xsxb2hoYnRaUkqsxGQbGeIsPeabfz4xE2lHj3JfG53PW30Q3kmghjUam?= =?us-ascii?Q?VDKB7ARqfThSa+OnNN+pmAKwB3M+xerW8igwzAqHI8r2EBjBB3k/+/xZ0ZEU?= =?us-ascii?Q?zKLxF8PJlOhL0aOI2IWRVWtUeEDUSjvjbuJJ0+qnkEjySvdTDuxXrq57xG5l?= =?us-ascii?Q?gnz0JjjTiZxAzxwqe1nItN8FkRQVJnJSNAreOeMQtUR79OWTjEDy+N/BblST?= =?us-ascii?Q?qkQ4IhpwJdundh9xCIzM4e0JsBtX/0+emORifrZc6McCClbpl4TmgeLSysTo?= =?us-ascii?Q?FwMSr1j8tXpe2BiRQN9B5UcBMpw39g1jbht5APM0NsdtcWv3htsnDzACw8yH?= =?us-ascii?Q?joleOq4lEgnuPbL1RL4TbTUydqHx51ebDZaJmg0kGgznbs5BW0VnzVa2mA1a?= =?us-ascii?Q?vNcouFCRxgNyfNN7amv4QIB9xwU/IhFKxYj6B5QYLdm6RwDTrBlsISd0dMSH?= =?us-ascii?Q?OYLAgHv/d8iD4VTStY5qvtqzaWA+eTC4e1u/FaxzzV/9/CvbscdZLeLZTpuo?= =?us-ascii?Q?yd/tG4PK40y4nsZZz2Q16YtSo3r278MsSbrOsNXP8e2xbbSsNF/VwUPBA9Nn?= =?us-ascii?Q?z9kjPZYgS/umjdTVKWCyNCeaeb/Lb+1C9b28HfuTDuuZLalaHMg3VK+/mvA2?= =?us-ascii?Q?mmu3g5lFVGBjjCbc2Hp5sq2SF1o8HRZuNMNyZBwX6UR1azYbm5y76NPIcCd6?= =?us-ascii?Q?pyjowq+L3vUmt6ckT4b8R+ZrjN0vPW1WbKe8kWjo6/R6Bbxgz6tcQ48iVaF0?= =?us-ascii?Q?egbwLlXYc8CuqfnoK3fxWXm69SBsdbDpNoYLnyWT3tT46aPd6UIY1nd/NA+s?= =?us-ascii?Q?W8J9Nk62NmUNG/Ja8Y65DwWLBMdd5K1KIkVzqiAcoIzXEJJUTGJbNA=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:34.0682 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0eba36b8-6c5c-4688-3f01-08dd671b8a85 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6480 Content-Type: text/plain; charset="utf-8" Introduce the folios_copy() and folios_mc_copy() to copy the folio content from the list of src folios to the list of dst folios. This is preparatory patch for batch page migration offloading. Signed-off-by: Shivank Garg --- include/linux/mm.h | 4 ++++ mm/util.c | 41 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8483e09aeb2c..612cba3d3dac 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1301,7 +1301,11 @@ void __folio_put(struct folio *folio); =20 void split_page(struct page *page, unsigned int order); void folio_copy(struct folio *dst, struct folio *src); +void folios_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt); int folio_mc_copy(struct folio *dst, struct folio *src); +int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt); =20 unsigned long nr_free_buffer_pages(void); =20 diff --git a/mm/util.c b/mm/util.c index 8c965474d329..5d00d4c5b2dd 100644 --- a/mm/util.c +++ b/mm/util.c @@ -908,6 +908,47 @@ int folio_mc_copy(struct folio *dst, struct folio *src) } EXPORT_SYMBOL(folio_mc_copy); =20 +/** + * folios_copy - Copy the contents of list of folios. + * @dst_list: Folios to copy to. + * @src_list: Folios to copy from. + * + * The folio contents are copied from @src_list to @dst_list. + * Assume the caller has validated that lists are not empty and both lists + * have equal number of folios. This may sleep. + */ +void folios_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt) +{ + struct folio *src, *dst; + + dst =3D list_first_entry(dst_list, struct folio, lru); + list_for_each_entry(src, src_list, lru) { + cond_resched(); + folio_copy(dst, src); + dst =3D list_next_entry(dst, lru); + } +} + +int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt) +{ + struct folio *src, *dst; + int ret; + + dst =3D list_first_entry(dst_list, struct folio, lru); + list_for_each_entry(src, src_list, lru) { + cond_resched(); + ret =3D folio_mc_copy(dst, src); + if (ret) + return ret; + dst =3D list_next_entry(dst, lru); + } + + return 0; +} +EXPORT_SYMBOL(folios_mc_copy); + int sysctl_overcommit_memory __read_mostly =3D OVERCOMMIT_GUESS; int sysctl_overcommit_ratio __read_mostly =3D 50; unsigned long sysctl_overcommit_kbytes __read_mostly; --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2067.outbound.protection.outlook.com [40.107.223.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95B1F1E0B86 for ; Wed, 19 Mar 2025 19:23:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412230; cv=fail; b=FWbWTGgWNqk0Z7VpgjgWeUmAo4WJ3glUL3xcLZYeTglk+MN3nqXqqkfKwhJut85MI7aMSzB68XYB352GNwppSPDwZmRf5oxYQ0881TrWIzwm3wCNRDngJfIVjQZi2iJcut6pXWQtxHfN6YvCul+2sUkjO0Ah9HEHRgdahLjiFng= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412230; c=relaxed/simple; bh=pOHlxDcIQ2vTSb71xqhcVRTTZ2Up3EuOQU3sKiqXxSY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=GyePGV/d/AO66sk4Oj+UlP6uu85ojeLVL2Ncy13gAf2LKxCelNSZDYF9BgES8lJ+i61GVBBKC/EXYLaI47PsU6/iluXsPO2qWblheOkDQ6gpRh+081QZh+Mdj+3iy+vkgC+LJPOKNRbKw434UicVA2j0DD65sUHFvkM3uQ2GaQs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Lk94Pf9C; arc=fail smtp.client-ip=40.107.223.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Lk94Pf9C" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZHYBrowxLzylWRqCuSomJ3W2ojvElgZrcEw5b/AGhpwFcnoYwBIMIO7TFrEiraY2RjWRj/0V6jaQGzUjKEPhToSBqCEGxJWif7cH0ItH7HLeELYjc9TGqy9w/8QUf5i9nu+mRuTc1Hvc6xRNM0Xu92UMGec8QD00ZxvO44cqZEM8w5u7g4f3tLf9zmcOhPTr4CTG0eo4ip1Y4enXDazbVJcZol1074R7DDt9neRvW812P2jXsi495XMRRLKMy6MXAWOgC1aKDfGjXl7xDIvBP0dRRy3ZOP9xo9kY4Hs3FNdgDdsYyoNzdb/Pw06MJY2P140qxePxX8pMm97bth59Fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oi6OB1LzwfSy1hb0t8o9q2QfNMHz3zUKojtPDKuE4c8=; b=D8uok86m0PB2jftkhjK09lAdgGZqpo0SfDjgX6jEEg7Mu7rUZirrzss/i+l/w8gAfjFnwVkS/H2zUDT6MlpgRZz+yGeNaBCEhAhuvth7K3iWSVb67iEbzaoYYZ+ASKyeCVajCVVugllED0xPcHbMCyc0AWOuxchJTSLggJ3y370StbFzsHl826RDKv3OsXDIVQxS9tEEAmqQaM71aggVkYmlrJQ247mq3mhzVkjlj8yT5qRO3vawVMnmgseCH5kJ+7yvcPMbiYIDsoA8dIT6F2NzPq8CU8G8Ix3H0HW+KWXfMzPhASD81VPPy85WjmTsFhOBK1YA8EbipWcNBa3WRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oi6OB1LzwfSy1hb0t8o9q2QfNMHz3zUKojtPDKuE4c8=; b=Lk94Pf9CuwJPH0mwyc+h1TJjethlp3qZe+CXHRr0Hxq359V6dGtHZhHFGCZvfEDuuyE0JJYbJGboti+y56MmWiOruZWJq9FEWi9G9RZ9CWYdEOP/wRAHtqpkO+yP/SceC5WM/i5ypjvOkCjjZPjc58X3sRe4OmOuBDa/FB0Vx+A= Received: from DS7PR03CA0071.namprd03.prod.outlook.com (2603:10b6:5:3bb::16) by CY5PR12MB6156.namprd12.prod.outlook.com (2603:10b6:930:24::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:42 +0000 Received: from CY4PEPF0000EDD4.namprd03.prod.outlook.com (2603:10b6:5:3bb:cafe::49) by DS7PR03CA0071.outlook.office365.com (2603:10b6:5:3bb::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:23:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD4.mail.protection.outlook.com (10.167.241.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:42 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:33 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 4/9] mm/migrate: add migrate_folios_batch_move to batch the folio move operations Date: Wed, 19 Mar 2025 19:22:07 +0000 Message-ID: <20250319192211.10092-5-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD4:EE_|CY5PR12MB6156:EE_ X-MS-Office365-Filtering-Correlation-Id: be185686-a276-4bb1-03c1-08dd671b8f9b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?4yG82fjcfSYrRunChtn23amHxAVznV+Gi2cCH9bNtqym/WYKDBhGecZeIn3L?= =?us-ascii?Q?uUKGUW95Vukl99iE2rkNGvqGkhG5uRhHYTREfvqcQkK8wJfxazWxUoCV3XeH?= =?us-ascii?Q?j4MzbFTovqN+A/mSgSz7dZzFKMgTopgOw5TUC2qqzm9Im3nb6ed+W2uszBOZ?= =?us-ascii?Q?WoVivX2v6SrHJJwB8/0BfV/EXyj7WyBs+dq/N1iv1Au7BwwSym/4Nvo0crPH?= =?us-ascii?Q?R/IOpW6/D3ZFNyzM127E8QRSwo7ckJY6qA/LkIBYZevXDtM1HoFUF9lv5TIg?= =?us-ascii?Q?IKvXBwFb9Pi4vkTCDdfPVXOLDbdXqG3Ude3GkYJA6cDbe9Ef3773ttLUuzD1?= =?us-ascii?Q?zQxLuDckZuwH0l5opuG3nyugemkHg24aE0j2cGuA45HSZQFlalGk/xcSq7rI?= =?us-ascii?Q?hYaGIYjNQOggi8Lh5EMhnQMMRCLwWdbtCE+QSc5f88LbkfTWEurIJRhO2xjI?= =?us-ascii?Q?2rZXi/SqVFhLdyjmEjMNcA/Sivx+hEU4OzkQM5o8UA6oGObWJfnh4wqISmDP?= =?us-ascii?Q?5300sbg9j9ZkkygtBqIJOumyiiuUdSAGzZbq+iCU5R/j+upGwF2AjQJ6V97k?= =?us-ascii?Q?kx1jkntcZXwgfcH23xkLawI03nzx/nA8Gzn9Z72EdJ1L/FISFVRzOfe2bhkQ?= =?us-ascii?Q?zcNBzEj56bMjR5ABz73onoHVVSc+dZY8PTFd8he8EqNOkBTgy+E//ORqtjP+?= =?us-ascii?Q?rl1fmEbR9PAKTxLWJxiqA4Xjl8Qmw/UaV6bv/GnenxOJYGt0qS48+GUAOPaE?= =?us-ascii?Q?/P8TbMFSygcpCVpiJP1H52MUn38hZCs7AWtItZ91JyxgH18u1aHhDHaEF6SN?= =?us-ascii?Q?pFtQt/jsxC7pJzLRlJGrbe+zr3gRwPC6yAymkBUebXW3Vlr56PzsVB6f1MVb?= =?us-ascii?Q?vpVRR57jG6S3qae6m6TLmpRiDeAKLpDEExW+qmQhbcC5gE7U67ye0AcPDmtX?= =?us-ascii?Q?VpVMhkGd6BHl8/pSkPfFXnNdZb/kEC1ega6mY7G5Y15EYraNzQPVBlY8KF/J?= =?us-ascii?Q?GYVYLzKQyEcGD57fRf2VPhRDUPPYjChQht3ObRo7nKefCxqfhU0Y920I2omX?= =?us-ascii?Q?TgVhdhUKXtMhv/uxbvHlJHgbuGbNs55Tv0Fuel7HzzSgKQC8YrT+Vpl1V8I4?= =?us-ascii?Q?FEWJGDaAO6fNCfL5JIMk4r/dPqwMBc3holnGte4iMKbLP8nDOvSdsuOb9JZm?= =?us-ascii?Q?i2RhIBwL7TA/dM+FBZbLnRWZniKyUC1rxLnylFnx9osdXovIK0SgQ14jJOq4?= =?us-ascii?Q?1rBz8qMpgzwBtR2UafTq0MLlMwOnPkxHp9gJFAgXIWAnwucsLoxQBCfJUiY7?= =?us-ascii?Q?BpE5t4ar5dbQM4kDbVkF84SQw9R0339HOLS7DNvlinMsZBqCY09W4J3Lmmhy?= =?us-ascii?Q?Rj/yLaFbOBtM+JkmNRmgBOypymYSb8aRhthhIdlRhsHtlrLk3U0b0t/cVw5X?= =?us-ascii?Q?i0wAjldmLxuNKbBQ60KFAljXNkTc+49lZBMKbZGPXg/QyUwGU5EJt7HmVGG0?= =?us-ascii?Q?dtSib5lWoNIxpGU=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:42.5994 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: be185686-a276-4bb1-03c1-08dd671b8f9b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6156 Content-Type: text/plain; charset="utf-8" This is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It uses MIGRATE_NO_COPY to skip folio copy during metadata copy process and performed the copies in a batch later. Currently, the folio move operation is performed individually for each folio in sequential manner: for_each_folio() { Copy folio metadata like flags and mappings Copy the folio content from src to dst Update page tables with dst folio } With this patch, we transition to a batch processing approach as shown below: for_each_folio() { Copy folio metadata like flags and mappings } Batch copy all src folios to dst for_each_folio() { Update page tables with dst folios } dst->private is used to store page states and possible anon_vma value, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process, src->private is used to store the data after metadata copy process, since src is no longer used. [Zi Yan: Refactor the patch. Improved the original patch by removing the need for an extra mig_info allocation (for storing anon_vma and old page state). Instead, reuse src->private to store the data, making the implementation simpler and efficient.] Signed-off-by: Shivank Garg Signed-off-by: Zi Yan --- mm/migrate.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 198 insertions(+), 6 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 0d40ac069cea..8b6cfb60087c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -752,12 +752,15 @@ static int __migrate_folio(struct address_space *mapp= ing, struct folio *dst, enum migrate_mode mode) { int rc, expected_count =3D folio_expected_refs(mapping, src); + unsigned long dst_private =3D (unsigned long)dst->private; =20 /* Check whether src does not have extra refs before we do more work */ if (folio_ref_count(src) !=3D expected_count) return -EAGAIN; =20 - if (mode !=3D MIGRATE_NO_COPY) { + if (mode =3D=3D MIGRATE_NO_COPY) + dst->private =3D NULL; + else { rc =3D folio_mc_copy(dst, src); if (unlikely(rc)) return rc; @@ -771,6 +774,10 @@ static int __migrate_folio(struct address_space *mappi= ng, struct folio *dst, folio_attach_private(dst, folio_detach_private(src)); =20 folio_migrate_flags(dst, src); + + if (mode =3D=3D MIGRATE_NO_COPY) + src->private =3D (void *)dst_private; + return MIGRATEPAGE_SUCCESS; } =20 @@ -1044,7 +1051,7 @@ static int _move_to_new_folio_prep(struct folio *dst,= struct folio *src, mode); else rc =3D fallback_migrate_folio(mapping, dst, src, mode); - } else { + } else if (mode !=3D MIGRATE_NO_COPY) { const struct movable_operations *mops; =20 /* @@ -1062,7 +1069,8 @@ static int _move_to_new_folio_prep(struct folio *dst,= struct folio *src, rc =3D mops->migrate_page(&dst->page, &src->page, mode); WARN_ON_ONCE(rc =3D=3D MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); - } + } else + rc =3D -EAGAIN; out: return rc; } @@ -1140,7 +1148,7 @@ static void __migrate_folio_record(struct folio *dst, dst->private =3D (void *)anon_vma + old_page_state; } =20 -static void __migrate_folio_extract(struct folio *dst, +static void __migrate_folio_read(struct folio *dst, int *old_page_state, struct anon_vma **anon_vmap) { @@ -1148,6 +1156,13 @@ static void __migrate_folio_extract(struct folio *ds= t, =20 *anon_vmap =3D (struct anon_vma *)(private & ~PAGE_OLD_STATES); *old_page_state =3D private & PAGE_OLD_STATES; +} + +static void __migrate_folio_extract(struct folio *dst, + int *old_page_state, + struct anon_vma **anon_vmap) +{ + __migrate_folio_read(dst, old_page_state, anon_vmap); dst->private =3D NULL; } =20 @@ -1770,6 +1785,178 @@ static void migrate_folios_move(struct list_head *s= rc_folios, } } =20 +static void migrate_folios_batch_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + int rc, nr_pages =3D 0, nr_batched_folios =3D 0; + int old_page_state =3D 0; + struct anon_vma *anon_vma =3D NULL; + int is_thp =3D 0; + LIST_HEAD(err_src); + LIST_HEAD(err_dst); + + if (mode !=3D MIGRATE_ASYNC) { + *retry +=3D 1; + return; + } + + /* + * Iterate over the list of locked src/dst folios to copy the metadata + */ + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp =3D folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages =3D folio_nr_pages(folio); + + /* + * dst->private is not cleared here. It is cleared and moved to + * src->private in __migrate_folio(). + */ + __migrate_folio_read(dst, &old_page_state, &anon_vma); + + /* + * Use MIGRATE_NO_COPY mode in migrate_folio family functions + * to copy the flags, mapping and some other ancillary information. + * This does everything except the page copy. The actual page copy + * is handled later in a batch manner. + */ + rc =3D _move_to_new_folio_prep(dst, folio, MIGRATE_NO_COPY); + + /* + * The rules are: + * Success: folio will be copied in batch + * -EAGAIN: move src/dst folios to tmp lists for + * non-batch retry + * Other errno: put src folio on ret_folios list, restore + * the dst folio + */ + if (rc =3D=3D -EAGAIN) { + *retry +=3D 1; + *thp_retry +=3D is_thp; + *nr_retry_pages +=3D nr_pages; + + list_move_tail(&folio->lru, &err_src); + list_move_tail(&dst->lru, &err_dst); + __migrate_folio_record(dst, old_page_state, anon_vma); + } else if (rc !=3D MIGRATEPAGE_SUCCESS) { + *nr_failed +=3D 1; + stats->nr_thp_failed +=3D is_thp; + stats->nr_failed_pages +=3D nr_pages; + + list_del(&dst->lru); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + } else /* MIGRATEPAGE_SUCCESS */ + nr_batched_folios++; + + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } + + /* Exit if folio list for batch migration is empty */ + if (!nr_batched_folios) + goto out; + + /* Batch copy the folios */ + rc =3D folios_mc_copy(dst_folios, src_folios, nr_batched_folios); + + /* TODO: Is there a better way of handling the poison + * recover for batch copy, instead of falling back to serial copy? + */ + /* fallback to serial page copy if needed */ + if (rc) { + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp =3D folio_test_large(folio) && + folio_test_pmd_mappable(folio); + nr_pages =3D folio_nr_pages(folio); + rc =3D folio_mc_copy(dst, folio); + + if (rc) { + /* + * dst->private is moved to src->private in + * __migrate_folio(), so page state and anon_vma + * values can be extracted from (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, + &anon_vma); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, + private); + } + + switch (rc) { + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded +=3D nr_pages; + stats->nr_thp_succeeded +=3D is_thp; + break; + default: + *nr_failed +=3D 1; + stats->nr_thp_failed +=3D is_thp; + stats->nr_failed_pages +=3D nr_pages; + break; + } + + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } + } + + /* + * Iterate the folio lists to remove migration pte and restore them + * as working pte. Unlock the folios, add/remove them to LRU lists (if + * applicable) and release the src folios. + */ + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp =3D folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages =3D folio_nr_pages(folio); + /* + * dst->private is moved to src->private in __migrate_folio(), + * so page state and anon_vma values can be extracted from + * (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, &anon_vma); + list_del(&dst->lru); + + _move_to_new_folio_finalize(dst, folio, MIGRATEPAGE_SUCCESS); + + /* + * Below few steps are only applicable for lru pages which is + * ensured as we have removed the non-lru pages from our list. + */ + _migrate_folio_move_finalize1(folio, dst, old_page_state); + + _migrate_folio_move_finalize2(folio, dst, reason, anon_vma); + + /* Page migration successful, increase stat counter */ + stats->nr_succeeded +=3D nr_pages; + stats->nr_thp_succeeded +=3D is_thp; + + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } +out: + /* Add tmp folios back to the list to re-attempt migration. */ + list_splice(&err_src, src_folios); + list_splice(&err_dst, dst_folios); +} + static void migrate_folios_undo(struct list_head *src_folios, struct list_head *dst_folios, free_folio_t put_new_folio, unsigned long private, @@ -1980,13 +2167,18 @@ static int migrate_pages_batch(struct list_head *fr= om, /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); =20 - retry =3D 1; + retry =3D 0; + /* Batch move the unmapped folios */ + migrate_folios_batch_move(&unmap_folios, &dst_folios, put_new_folio, + private, mode, reason, ret_folios, stats, &retry, + &thp_retry, &nr_failed, &nr_retry_pages); + for (pass =3D 0; pass < nr_pass && retry; pass++) { retry =3D 0; thp_retry =3D 0; nr_retry_pages =3D 0; =20 - /* Move the unmapped folios */ + /* Move the remaining unmapped folios */ migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2065.outbound.protection.outlook.com [40.107.223.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 209E5213E78 for ; Wed, 19 Mar 2025 19:23:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412237; cv=fail; b=uxahI3e9NuiVH9Ex0KnSoy+opFhpEdsILFzce1oBFyJMOTvVngIDsqh8d/48ecbpFtIjLdG4JlFbXzdFplqaXBOcjudnvDD19ANESBG/hVZPfLDP4pvLVNUQ/uaytlA1y9Wz7MWQE/XowdoSFj0L+cUcqP4kYkQ6Ncbg7PxFqXY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412237; c=relaxed/simple; bh=stV6qHmXKpZPHjuRDVq3c4bV+y+ZMkrovx8edPOxAyo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gz2Gpk8Q8RiYN0jZjWsmsMVmwJeuN0jns0adR92xzqRCbjS3hDHz6cG10tn5B3t5lXnLIrjJTWMAPv8Ez8Xjm9Z4HeSWkqt1jPEyrxbP0ZO/eXQHFTCJLEWJcxOhyLZTvs671FVJmS6WCFNRrhS/Co/++V5/oVlZMBdywKXVQI0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QnipFg4+; arc=fail smtp.client-ip=40.107.223.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QnipFg4+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LfD8pgNR3dG18aXRfuhxSJKUITEQ1hQwT8uiTUZBZ+WrJm7pX150hn8/2h9Ja0jjdO6jru0oy1ZxcBhNyAY4D7fOnAlnAhw2prqlxJEqZa2kw6BTqL3/BMQxunZBTwM1rly3qtB8osLid5sjTvrbVMWDAO4h/yTsM5xh/ps33g+MVk/rsJevr9UAHY0p8OegqR44oXL1IbRIfFIFBO7PqMP96DO18A0YWESXFaquVM02e97GsYIJrZIgwkhaqzYr2nv7XygvYGAZ29S+lAzh1UZmPHYnlSrddLniMuaQB6rVFEFmKCI3uxdF1eyl5M7CxBmh/ZA8sz+w0TI9Do6waw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=ZOrqcAdPZ3A5grvaDQDzgx5rlI9dYEPbinShaB++dUcN/eyeiWFP2olhM8qWJhsQze//fVjMGJDQlSIuCc20woi8ELf1hg7Gy/02e04MHJE05Ce3YyXE4UTLbsA37DP/zDJFS75HDkj+jwCGnFBcplOUs50gbWBhZ024BNd0kUcoBmq88EETJzkqLetjj6ZUncYztdzxwh9GRyT4jAILE15U6rMgc6l6+9I/fryh7o95R2JH78XxLtxp5MXr0gY4IlzF/i1GXxtSJl8N9ngOcjYqueXaHK3C9bOGgOw9x2/XTpuYto3dPlOPqemOSBZnlK/EeyjR0/DkTz0gDJC2ZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=QnipFg4+uIIn811W7K3qKeZpmZZc56zTiARSfWh953l8yk/WyUoDUAYd1HEbg8kIWF28bSE9zQvOBCe8uxgLrNaLsQfxr/dPtNqrp3eU+ZbO+ywp55/JxnW/ndd1WzLo+uWRMqrtY1RYclCdA5WUuqhP0t3nSDCzb7Oq/qw2WmM= Received: from CH0PR03CA0303.namprd03.prod.outlook.com (2603:10b6:610:118::16) by MN2PR12MB4336.namprd12.prod.outlook.com (2603:10b6:208:1df::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:50 +0000 Received: from CH1PEPF0000AD7B.namprd04.prod.outlook.com (2603:10b6:610:118:cafe::9) by CH0PR03CA0303.outlook.office365.com (2603:10b6:610:118::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.35 via Frontend Transport; Wed, 19 Mar 2025 19:23:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD7B.mail.protection.outlook.com (10.167.244.58) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:50 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:42 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Mike Day Subject: [PATCH RFC V2 5/9] mm: add support for copy offload for folio Migration Date: Wed, 19 Mar 2025 19:22:08 +0000 Message-ID: <20250319192211.10092-6-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD7B:EE_|MN2PR12MB4336:EE_ X-MS-Office365-Filtering-Correlation-Id: ab83a946-6cf6-4667-1e0b-08dd671b9450 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?gcPZODinQLlEs6YEZGp33h2jgJIRZ6JR5G/qgtIPPGZtG9ntNW9s52KrynBj?= =?us-ascii?Q?1nCYzuO3q9Vv16DKoksgjhgDyUUE5ehyD29XpTKj8HRWOs5wX8kvLeclVO+y?= =?us-ascii?Q?jmPo/SJTR6IIO6hiG8D/TPdTS3Ob//vIsmeeGsvUJB6FXI9FS2im8ZoR1TRr?= =?us-ascii?Q?0UlNnNHf6R/Gr24HxdDNiHwevTZTsWCOcdfEkQ25/ESTvaE/RUiREHQa9BIy?= =?us-ascii?Q?LCecPgoqLpx7l38+tBRTIZMrxBjrcddPIVsTux22R6a5YxgfXYKYzAKPgVRq?= =?us-ascii?Q?iGF3eXf2wSy+AX/gc7tQJxr7SAYBrWP50pkAqTtgqYnUfhWx7NmUrEGdDZGy?= =?us-ascii?Q?hkc/Q1EZMqTLnxHOlP1j4Dwdv16Y0Q3//MNpXH45ChXRvQfoa4OBy8+WhWaI?= =?us-ascii?Q?hj26Imc/wVgLDdw4sfhIhVcQLyIapchYPLh5LbTlUP5Wjk1/7SVagkESUrmn?= =?us-ascii?Q?r2Wn+vvxJvXPzdtLBWFASqxcyFqo38QeeacvWMjIvwTRBTNiNAQIi4NMoxGe?= =?us-ascii?Q?5QGVS1ICjvCX4C2PriiDHdbK9s5hiIckMQNl1BaoyBdrd3zKtDKVQIvqzwDR?= =?us-ascii?Q?ypEmFYVQ2wttc6hnrRxOIXjpnwI8InWPXVAUzrEQ2AbSPZGraUwxMZgVJveu?= =?us-ascii?Q?qjuGV809TpcvhsP1WQbbIM8JbYGhuFiO8DkPmXDP+JIIKSrk37fRc0irq/lj?= =?us-ascii?Q?9oTdkpoZKStvvKxG5JTFkaBYXuvBfBjFGF9bDd6UdoV/2gglylIMOmFyCBjt?= =?us-ascii?Q?mIRtlIBG2AAd2vi1Grw+9SQTcurHvpQJUofAa45BuDKh8Iio+3aM0U7RZweq?= =?us-ascii?Q?oYOJqbFgdIShI3LOC21KUMxbsgf/lmYWfwR6Ts/PkVjH5jX0mWxoJbwqidvB?= =?us-ascii?Q?R+5F6jVfp6Dn35K6yVjytTrwltFknokTK2ijC55MT7r1gIuis1u3kGVaE2S+?= =?us-ascii?Q?/k+FAqsaLkzKNTvbuSXAhZyTxCX7KGVpt9yUriQ//UItx2rDWCj8mJutlcga?= =?us-ascii?Q?G5meT1iGkZfV7hDH+DlG+oSfW+KHTZ/YHx6Wp/ABHmOgCyefwXpVSbqZ68LA?= =?us-ascii?Q?oLUrsmpE/ZhugAg7MlB0+a7XYmoS1L9pjBl24E/BGHqdrmdxHKFgSMUo0mEm?= =?us-ascii?Q?sA+5oNOA5MMMA49LYHs8m9LaaKqADhDmfodgBAkWaQI+UleqgWahTi3GgWu6?= =?us-ascii?Q?qq5cm4WaAmU6aEz2CropiwRDdFT3Tn81Mpvj48ByX7gZfgmQYPG1U+U35acq?= =?us-ascii?Q?d1t7+pba/SpfSc4MDAGRqZTWU8fJOocwngxyL+F/26guxe/hU6NfOCeEMdU7?= =?us-ascii?Q?cxr0OKWaj+kgFQ6fuvNozxFXZUhUEw2ytWg8wwyDYt3dPsJJs94xZcvqvvdp?= =?us-ascii?Q?14d/cAccnOJp71Xyh7Q+kOIQ9JtI9KHBIcMpqWXmgijsI4F6XHMznRB9OYo7?= =?us-ascii?Q?96K5bib/gJdIn/AYuQV9XFDcwsiT8aNrk0RIOjDgG/FWrUV8ZRw5LLVe2w8Y?= =?us-ascii?Q?P9ftAhEFDaHu728=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:50.5909 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ab83a946-6cf6-4667-1e0b-08dd671b9450 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD7B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4336 Content-Type: text/plain; charset="utf-8" From: Mike Day Offload-Copy drivers should implement following functions to enable folio migration offloading: migrate_offc() - This function takes src and dst folios list undergoing migration. It is responsible for transfer of page content between the src and dst folios. can_migrate_offc() - It performs necessary checks if offload copying migration is supported for the give src and dst folios. Offload-Copy driver should include a mechanism to call start_offloading and stop_offloading for enabling and disabling migration offload respectively. [Shivank: Rename the APIs and files to generalize the original DMA-specific offload implementation to support various copy offloading mechanisms such as DMA engines, CPU multi-threading, or other hardware accelerators.] Signed-off-by: Mike Day Signed-off-by: Shivank Garg --- include/linux/migrate_offc.h | 36 +++++++++++++++++++++++++ mm/Kconfig | 8 ++++++ mm/Makefile | 1 + mm/migrate.c | 43 ++++++++++++++++++++++++++++-- mm/migrate_offc.c | 51 ++++++++++++++++++++++++++++++++++++ 5 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 include/linux/migrate_offc.h create mode 100644 mm/migrate_offc.c diff --git a/include/linux/migrate_offc.h b/include/linux/migrate_offc.h new file mode 100644 index 000000000000..908f81ebd621 --- /dev/null +++ b/include/linux/migrate_offc.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _MIGRATE_OFFC_H +#define _MIGRATE_OFFC_H +#include + +#define MIGRATOR_NAME_LEN 32 +struct migrator { + char name[MIGRATOR_NAME_LEN]; + int (*migrate_offc)(struct list_head *dst_list, struct list_head *src_lis= t, int folio_cnt); + bool (*can_migrate_offc)(struct folio *dst, struct folio *src); + struct rcu_head srcu_head; + struct module *owner; +}; + +extern struct migrator migrator; +extern struct mutex migrator_mut; +extern struct srcu_struct mig_srcu; + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head); +void offc_update_migrator(struct migrator *mig); +unsigned char *get_active_migrator_name(void); +bool can_offc_migrate(struct folio *dst, struct folio *src); +void start_offloading(struct migrator *migrator); +void stop_offloading(void); +#else +static inline void srcu_mig_cb(struct rcu_head *head) { }; +static inline void offc_update_migrator(struct migrator *mig) { }; +static inline unsigned char *get_active_migrator_name(void) { return NULL;= }; +static inline bool can_offc_migrate(struct folio *dst, struct folio *src) = {return true; }; +static inline void start_offloading(struct migrator *migrator) { }; +static inline void stop_offloading(void) { }; +#endif /* CONFIG_OFFC_MIGRATION */ + +#endif /* _MIGRATE_OFFC_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db06417..7a0693c3be4e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -722,6 +722,14 @@ config MIGRATION config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE =20 +config OFFC_MIGRATION + bool "Migrate Pages offloading copy" + def_bool n + depends on MIGRATION + help + An interface allowing external modules or driver to offload + page copying in page migration. + config ARCH_ENABLE_HUGEPAGE_MIGRATION bool =20 diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..010142414176 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -93,6 +93,7 @@ obj-$(CONFIG_FAILSLAB) +=3D failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) +=3D fail_page_alloc.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o +obj-$(CONFIG_OFFC_MIGRATION) +=3D migrate_offc.o obj-$(CONFIG_NUMA) +=3D memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o diff --git a/mm/migrate.c b/mm/migrate.c index 8b6cfb60087c..862a3d1eff60 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -44,6 +44,7 @@ #include #include #include +#include =20 #include =20 @@ -743,6 +744,37 @@ void folio_migrate_flags(struct folio *newfolio, struc= t folio *folio) } EXPORT_SYMBOL(folio_migrate_flags); =20 +DEFINE_STATIC_CALL(_folios_copy, folios_mc_copy); +DEFINE_STATIC_CALL(_can_offc_migrate, can_offc_migrate); + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head) +{ + static_call_query(_folios_copy); +} + +void offc_update_migrator(struct migrator *mig) +{ + int index; + + mutex_lock(&migrator_mut); + index =3D srcu_read_lock(&mig_srcu); + strscpy(migrator.name, mig ? mig->name : "kernel", MIGRATOR_NAME_LEN); + static_call_update(_folios_copy, mig ? mig->migrate_offc : folios_mc_copy= ); + static_call_update(_can_offc_migrate, mig ? mig->can_migrate_offc : can_o= ffc_migrate); + if (READ_ONCE(migrator.owner)) + module_put(migrator.owner); + xchg(&migrator.owner, mig ? mig->owner : NULL); + if (READ_ONCE(migrator.owner)) + try_module_get(migrator.owner); + srcu_read_unlock(&mig_srcu, index); + mutex_unlock(&migrator_mut); + call_srcu(&mig_srcu, &migrator.srcu_head, srcu_mig_cb); + srcu_barrier(&mig_srcu); +} + +#endif /* CONFIG_OFFC_MIGRATION */ + /************************************************************ * Migration functions ***********************************************************/ @@ -1028,11 +1060,15 @@ static int _move_to_new_folio_prep(struct folio *ds= t, struct folio *src, { int rc =3D -EAGAIN; bool is_lru =3D !__folio_test_movable(src); + bool can_migrate; =20 VM_BUG_ON_FOLIO(!folio_test_locked(src), src); VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst); =20 - if (likely(is_lru)) { + can_migrate =3D static_call(_can_offc_migrate)(dst, src); + if (unlikely(!can_migrate)) + rc =3D -EAGAIN; + else if (likely(is_lru)) { struct address_space *mapping =3D folio_mapping(src); =20 if (!mapping) @@ -1868,7 +1904,10 @@ static void migrate_folios_batch_move(struct list_he= ad *src_folios, goto out; =20 /* Batch copy the folios */ - rc =3D folios_mc_copy(dst_folios, src_folios, nr_batched_folios); + rc =3D static_call(_folios_copy)(dst_folios, src_folios, nr_batched_folio= s); + /* TODO: Is there a better way of handling the poison + * recover for batch copy and falling back to serial copy? + */ =20 /* TODO: Is there a better way of handling the poison * recover for batch copy, instead of falling back to serial copy? diff --git a/mm/migrate_offc.c b/mm/migrate_offc.c new file mode 100644 index 000000000000..c632928a7c27 --- /dev/null +++ b/mm/migrate_offc.c @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +atomic_t dispatch_to_offc =3D ATOMIC_INIT(0); +EXPORT_SYMBOL_GPL(dispatch_to_offc); + +DEFINE_MUTEX(migrator_mut); +DEFINE_SRCU(mig_srcu); + +struct migrator migrator =3D { + .name =3D "kernel", + .migrate_offc =3D folios_mc_copy, + .can_migrate_offc =3D can_offc_migrate, + .srcu_head.func =3D srcu_mig_cb, + .owner =3D NULL, +}; + +bool can_offc_migrate(struct folio *dst, struct folio *src) +{ + return true; +} +EXPORT_SYMBOL_GPL(can_offc_migrate); + +void start_offloading(struct migrator *m) +{ + int offloading =3D 0; + + pr_info("starting migration offload by %s\n", m->name); + offc_update_migrator(m); + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 1); +} +EXPORT_SYMBOL_GPL(start_offloading); + +void stop_offloading(void) +{ + int offloading =3D 1; + + pr_info("stopping migration offload by %s\n", migrator.name); + offc_update_migrator(NULL); + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 0); +} +EXPORT_SYMBOL_GPL(stop_offloading); + +unsigned char *get_active_migrator_name(void) +{ + return migrator.name; +} +EXPORT_SYMBOL_GPL(get_active_migrator_name); --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2071.outbound.protection.outlook.com [40.107.223.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E7032F37 for ; Wed, 19 Mar 2025 19:24:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.71 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412247; cv=fail; b=u9CiTq3/ZFIcYbUvJ2fG4GrS+m8CS/eH0m+uQz6QfOnHStqWFxsa3J+nI/2a2bKalUTdBOeWUxg3u70NY2g2QxhA8N0Gb0qI1k8NzRWPyYrtAbyqcCVv2cW+9GpN9S4uh8svDYyePHwE2Q8CknhUhPSdQc4bGYq/LqwQYjPK0BI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412247; c=relaxed/simple; bh=UNCaA9yZBYICy2B4JAN8lZACfzZkBLeh6Vt6HoJODS4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QOJI4DnUmEP49M1mdTJN+Z7IAUAN+cbbNGf3j/lw38DkpPssn0bvAm6n5PsKmJM+c0gDx1c8sJnDu1jr6O/JuV9bONOR/DQQivcMggQIgi9QWcx0h/FxCkB5du4V+FRqpZ8ZDkVAqR/EXBRzYCJN6utESMSW/28m0Qt4QT1J8FU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=q1bOdpHS; arc=fail smtp.client-ip=40.107.223.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="q1bOdpHS" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lUieM0gYppq/RbWGvc33RPxKs0d9/Gd1bcKiCxDAgTiKlaZPQELdyyclNAp0ItjQ45NFgkyN+feBUK4LEZFDJN8S5XEIOOnl5/TArtIxm46iUlEPtUj6m7rsOFoyHXShfmvSlc0iMymPZRNwTvT2Gy2Ej2jHKVN24HAqcVDInIjY3ukOgB5rddjb8JenkOEUVtxp93x68jciSr8IpfuQZpoEgNRj4U9Y9TT8JBUBKQtBRD/6U7xgi29tvJbFi2i0/6ILP94i9mvyL6k9LZxuCVxLajkDyh/fWz8H1aCRad3hCvcfMIxiEr9rh1RKnGLAR9IDLrgU8mQyuwA0NXWVVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=flOo3vLO2rJeiBptjrTWYFaJiojxSG0aBTYgf9G0Q9s=; b=XlNAhSEkmKEAm1qJSKRwbsv9LLdFPftwEJ6AWI5pFnixBALHFlzEVwDzWzFzN6k14cwXDsxJlpacYp6K7sRJhqI29311CwGtXJgh/92VWyNCJP6WTHWC8gp+drtBV+4TyqL9ukUxyABCqo4ez8FMVi+bN0h4BmXhQlYT1BbmzQuuU2MDbiLGuDaye02tg2YUqM7pHP4lEeUVlbsV6THHPFxV8QgShM5dOr59kRrjAm7nuCl8M1W1ltTy+hvgKuJYtLrI/oBDr76mnxnv7zLkQVDNdLn9I2iuAr/Bkxdxf1BXfQnjvsYvD7ocQcq/wnSSQvvsqDL21EGwi+4q/7/TQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=flOo3vLO2rJeiBptjrTWYFaJiojxSG0aBTYgf9G0Q9s=; b=q1bOdpHSzm+929plILyPP2MaFBZQkRuCDkqUtOBtd/Bpi7Vu6FEMrc+WnyCAuZY4ZKCrElbZAMUm4osSJA1ejWM/CxmmY3d+zqzI7rIReOgfbq3hBxs24gid7m3Gh5avDGmD6SGXIdfs++9xVjAVqCrM9pqyMpVLmrGMueYnUxE= Received: from CH0PR07CA0015.namprd07.prod.outlook.com (2603:10b6:610:32::20) by DS0PR12MB8563.namprd12.prod.outlook.com (2603:10b6:8:165::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:59 +0000 Received: from CH1PEPF0000AD76.namprd04.prod.outlook.com (2603:10b6:610:32:cafe::de) by CH0PR07CA0015.outlook.office365.com (2603:10b6:610:32::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:23:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD76.mail.protection.outlook.com (10.167.244.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:58 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:50 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 6/9] mm/migrate: introduce multi-threaded page copy routine Date: Wed, 19 Mar 2025 19:22:09 +0000 Message-ID: <20250319192211.10092-7-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD76:EE_|DS0PR12MB8563:EE_ X-MS-Office365-Filtering-Correlation-Id: 782e3ac6-43ea-462c-fa77-08dd671b9926 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?MzvXbJP6hhfW9pg5Pbpd3JDkG0nWx8nG9XvPpl717xPcaQv4uot6gkGyEiUV?= =?us-ascii?Q?eZvXgeIxYEkOyIf4XHlhIh3wQnpxTgkY6XtZK8rVUhH0JcXam8ljtyE3IJUd?= =?us-ascii?Q?uhyUgQFxV/mXillyndg5fUo5CWpJvuvtEPwUiGbjZjRzLe0aWzcpRrnp6Pqr?= =?us-ascii?Q?K3y1KPqndhQ4/uJovRFRaVDKDlwAlszv0ifWXjdPIClrOuPhLV8T9BNb2gF/?= =?us-ascii?Q?NpgMemk344gFgAOR2avQf/a0CfMJ2xRn7kxbC612au4spFSikE6xslp209HB?= =?us-ascii?Q?GcF1CqA9ztLUOOZ1+WwACxDyhvQzEVq/Dxhed1qLurBSah7CfrNC9y52G4yP?= =?us-ascii?Q?E+2H7LuiUWQxRNBcy5itCLxfBfftDXwlW+bkGpn74reEb6WVWAV/J6FKMz7M?= =?us-ascii?Q?cW8Xvn5mlmf6w2hzKAAGUXreKiqGWtTydGOauGzhIP4aE3ZRr3YFA2ICq5c/?= =?us-ascii?Q?MKOQQQG1ZiOvVpmFfEzYu8NMjiDLqA7QBNHZGGCvDuzBw8CBXnbgZeYBgdpY?= =?us-ascii?Q?cPdQk/One5B5yYt9JBUgKcBMS7HuCg7CsOwk0N+XZTM44tt3ENHTL9R6zyAL?= =?us-ascii?Q?qxwMEEZt/hVfqGFkE1vOT13RH3BRic/WM0370feNDrEyLXiNm2VAT9jLi4Fq?= =?us-ascii?Q?dhqmGx5gJwnuBY/o4UoWu0EzdYIeC5bSEXyBYmnm5hoEN8XaIvdzzNIsECJq?= =?us-ascii?Q?LfZ9QMsNXtMJAYoSQeGRTg/iZ1SpMheHmORJgzzOPlq4oqqvPW2nqTbwjgrc?= =?us-ascii?Q?jckZXS79SHd+m4EXSNCdHyEduJcKdNiXzf0De2ajW+SlJnrIDm7e5ScNAhO1?= =?us-ascii?Q?8M4cui4WZe0lNSPy7Kjfnkz2jl96jlcw7ek3ZPyG9VmpEzfrabUEJSGN6l2l?= =?us-ascii?Q?ExoQZHrWXemFsE8/ZuDyquK/BNV4OdANHKK2AnlODqSmmnVExA//i5xtU+5c?= =?us-ascii?Q?yMUlH7dFn6vMYHm1e3IjNN4aPcNnrnrSiWZYlybhA3xrMOH3Hz8EFfdVODPu?= =?us-ascii?Q?K2sYvPVurPVns/iAHeGSOgSfrwyll79zJqdElmbr/4KpEz1/wGrLP6L8zARS?= =?us-ascii?Q?I9gPLF9ZBRgOBTO/M42LWKn0V5imSUrA2fdMNaCT3NVlM2pirGrBUungPEVQ?= =?us-ascii?Q?2VbCSDumN43qdDUWzadvJYhlPTOvgSoR9Nj9AsogWb8yvMJIgc/KH5EVtqjJ?= =?us-ascii?Q?ALOecFf3giskAXEx2j41PoFbgwmFadzCaub0pJZhkbMq+1O+CI6rdM51/Gax?= =?us-ascii?Q?BX7wQPG/ssoqiOdycQG7EmMro9Rdfzdl3FzN5GhrIxbJU/nMjAc61CBF68IY?= =?us-ascii?Q?UlQ+PgjdamyajH5idcbCL8akjXd+2+nxJNim98/YRsYdLBrJ2eDQiU5f7yoI?= =?us-ascii?Q?ZfiWLDyjCJOVJaT0SuNgQe1PbN9v6NqWFNWqxNc1EmDiwQahnvHbDXKY1UYA?= =?us-ascii?Q?HI7V6xCiEd9BHsX4duz2Y+iv2zeD8Ny+F70sLS9sfC/0Z2o7SuWefqABm3ot?= =?us-ascii?Q?Te+5WRaZ7o1cqB8=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:58.5849 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 782e3ac6-43ea-462c-fa77-08dd671b9926 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD76.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8563 Content-Type: text/plain; charset="utf-8" From: Zi Yan Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Enable using: echo 1 > /sys/kernel/cpu_mt/offloading echo NR_THREADS > /sys/kernel/cpu_mt/threads Disable: echo 0 > /sys/kernel/cpu_mt/offloading [Shivank: Convert the original MT copy_pages implementation into a module, leveraging migrate offload infrastructure and sysfs interface.] Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- drivers/Kconfig | 2 + drivers/Makefile | 3 + drivers/migoffcopy/Kconfig | 9 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/copy_pages.c | 337 +++++++++++++++++++++++++ mm/migrate.c | 11 +- 7 files changed, 357 insertions(+), 7 deletions(-) create mode 100644 drivers/migoffcopy/Kconfig create mode 100644 drivers/migoffcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/copy_pages.c diff --git a/drivers/Kconfig b/drivers/Kconfig index 7bdad836fc62..2e20eb83cd0b 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -245,4 +245,6 @@ source "drivers/cdx/Kconfig" =20 source "drivers/dpll/Kconfig" =20 +source "drivers/migoffcopy/Kconfig" + endmenu diff --git a/drivers/Makefile b/drivers/Makefile index 45d1c3e630f7..4df928a36ea3 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -42,6 +42,9 @@ obj-y +=3D clk/ # really early. obj-$(CONFIG_DMADEVICES) +=3D dma/ =20 +# Migration copy Offload +obj-$(CONFIG_OFFC_MIGRATION) +=3D migoffcopy/ + # SOC specific infrastructure drivers. obj-y +=3D soc/ obj-$(CONFIG_PM_GENERIC_DOMAINS) +=3D pmdomain/ diff --git a/drivers/migoffcopy/Kconfig b/drivers/migoffcopy/Kconfig new file mode 100644 index 000000000000..e73698af3e72 --- /dev/null +++ b/drivers/migoffcopy/Kconfig @@ -0,0 +1,9 @@ +config MTCOPY_CPU + bool "Multi-Threaded Copy with CPU" + depends on OFFC_MIGRATION + default n + help + Interface MT COPY CPU driver for batch page migration + offloading. Say Y if you want to try offloading with + MultiThreaded CPU copy APIs. + diff --git a/drivers/migoffcopy/Makefile b/drivers/migoffcopy/Makefile new file mode 100644 index 000000000000..0a3c356d67e6 --- /dev/null +++ b/drivers/migoffcopy/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) +=3D mtcopy/ diff --git a/drivers/migoffcopy/mtcopy/Makefile b/drivers/migoffcopy/mtcopy= /Makefile new file mode 100644 index 000000000000..b4d7da85eda9 --- /dev/null +++ b/drivers/migoffcopy/mtcopy/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) +=3D copy_pages.o diff --git a/drivers/migoffcopy/mtcopy/copy_pages.c b/drivers/migoffcopy/mt= copy/copy_pages.c new file mode 100644 index 000000000000..4c9c7d90c9fd --- /dev/null +++ b/drivers/migoffcopy/mtcopy/copy_pages.c @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Parallel page copy routine. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_NUM_COPY_THREADS 64 + +unsigned int limit_mt_num =3D 4; +static int is_dispatching; + +static int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items); +static bool can_migrate_mt(struct folio *dst, struct folio *src); + +static DEFINE_MUTEX(migratecfg_mutex); + +/* CPU Multithreaded Batch Migrator */ +struct migrator cpu_migrator =3D { + .name =3D "CPU_MT_COPY\0", + .migrate_offc =3D copy_page_lists_mt, + .can_migrate_offc =3D can_migrate_mt, + .owner =3D THIS_MODULE, +}; + +struct copy_item { + char *to; + char *from; + unsigned long chunk_size; +}; + +struct copy_page_info { + struct work_struct copy_page_work; + int ret; + unsigned long num_items; + struct copy_item item_list[]; +}; + +static unsigned long copy_page_routine(char *vto, char *vfrom, + unsigned long chunk_size) +{ + return copy_mc_to_kernel(vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread(struct work_struct *work) +{ + struct copy_page_info *my_work =3D (struct copy_page_info *)work; + int i; + + my_work->ret =3D 0; + for (i =3D 0; i < my_work->num_items; ++i) + my_work->ret |=3D !!copy_page_routine(my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +} + +static ssize_t mt_offloading_set(struct kobject *kobj, struct kobj_attribu= te *attr, + const char *buf, size_t count) +{ + int ccode; + int action; + + ccode =3D kstrtoint(buf, 0, &action); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + /* + * action is 0: User wants to disable MT offloading. + * action is 1: User wants to enable MT offloading. + */ + switch (action) { + case 0: + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 1) { + stop_offloading(); + is_dispatching =3D 0; + } else + pr_debug("MT migration offloading is already OFF\n"); + mutex_unlock(&migratecfg_mutex); + break; + case 1: + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 0) { + start_offloading(&cpu_migrator); + is_dispatching =3D 1; + } else + pr_debug("MT migration offloading is already ON\n"); + mutex_unlock(&migratecfg_mutex); + break; + default: + pr_debug("input should be zero or one, parsed as %d\n", action); + } + return sizeof(action); +} + +static ssize_t mt_offloading_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", is_dispatching); +} + +static ssize_t mt_threads_set(struct kobject *kobj, struct kobj_attribute = *attr, + const char *buf, size_t count) +{ + int ccode; + unsigned int threads; + + ccode =3D kstrtouint(buf, 0, &threads); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + if (threads > 0 && threads <=3D MAX_NUM_COPY_THREADS) { + mutex_lock(&migratecfg_mutex); + limit_mt_num =3D threads; + mutex_unlock(&migratecfg_mutex); + pr_debug("MT threads set to %u\n", limit_mt_num); + } else { + pr_debug("Invalid thread count. Must be between 1 and %d\n",MAX_NUM_COPY= _THREADS); + return -EINVAL; + } + + return count; +} + +static ssize_t mt_threads_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", limit_mt_num); +} + +static bool can_migrate_mt(struct folio *dst, struct folio *src) +{ + return true; +} + +int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items) +{ + struct copy_page_info *work_items[MAX_NUM_COPY_THREADS] =3D {0}; + unsigned int total_mt_num =3D limit_mt_num; + struct folio *src, *src2, *dst, *dst2; + int max_items_per_thread; + int item_idx; + int err =3D 0; + int cpu; + int i; + + if (IS_ENABLED(CONFIG_HIGHMEM)) + return -ENOTSUPP; + + if (total_mt_num > MAX_NUM_COPY_THREADS) + total_mt_num =3D MAX_NUM_COPY_THREADS; + + /* Each threads get part of each page, if nr_items < totla_mt_num */ + if (nr_items < total_mt_num) + max_items_per_thread =3D nr_items; + else + max_items_per_thread =3D (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ? 1 : 0); + + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + work_items[cpu] =3D kzalloc(sizeof(struct copy_page_info) + + sizeof(struct copy_item) * + max_items_per_thread, + GFP_NOWAIT); + if (!work_items[cpu]) { + err =3D -ENOMEM; + goto free_work_items; + } + } + + if (nr_items < total_mt_num) { + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items =3D max_items_per_thread; + } + + item_idx =3D 0; + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + unsigned long chunk_size =3D PAGE_SIZE * folio_nr_pages(src) / total_mt= _num; + char *vfrom =3D page_address(&src->page); + char *vto =3D page_address(&dst->page); + + VM_WARN_ON(PAGE_SIZE * folio_nr_pages(src) % total_mt_num); + VM_WARN_ON(folio_nr_pages(dst) !=3D folio_nr_pages(src)); + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to =3D + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from =3D + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size =3D + chunk_size; + } + + item_idx++; + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) + queue_work(system_unbound_wq, + (struct work_struct *)work_items[cpu]); + } else { + int num_xfer_per_thread =3D nr_items / total_mt_num; + int per_cpu_item_idx; + + + for (cpu =3D 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items =3D num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu =3D 0; + per_cpu_item_idx =3D 0; + item_idx =3D 0; + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + work_items[cpu]->item_list[per_cpu_item_idx].to =3D + page_address(&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from =3D + page_address(&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size =3D + PAGE_SIZE * folio_nr_pages(src); + + VM_WARN_ON(folio_nr_pages(dst) !=3D + folio_nr_pages(src)); + + per_cpu_item_idx++; + item_idx++; + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + + if (per_cpu_item_idx =3D=3D work_items[cpu]->num_items) { + queue_work(system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx =3D 0; + cpu++; + } + } + if (item_idx !=3D nr_items) + pr_warn("%s: only %d out of %d pages are transferred\n", + __func__, item_idx - 1, nr_items); + } + + /* Wait until it finishes */ + for (i =3D 0; i < total_mt_num; ++i) { + flush_work((struct work_struct *)work_items[i]); + /* retry if any copy fails */ + if (work_items[i]->ret) + err =3D -EAGAIN; + } + +free_work_items: + for (cpu =3D 0; cpu < total_mt_num; ++cpu) + kfree(work_items[cpu]); + + return err; +} + +static struct kobject *mt_kobj_ref; +static struct kobj_attribute mt_offloading_attribute =3D __ATTR(offloading= , 0664, + mt_offloading_show, mt_offloading_set); +static struct kobj_attribute mt_threads_attribute =3D __ATTR(threads, 0664, + mt_threads_show, mt_threads_set); + +static int __init cpu_mt_module_init(void) +{ + int ret =3D 0; + + mt_kobj_ref =3D kobject_create_and_add("cpu_mt", kernel_kobj); + if (!mt_kobj_ref) + return -ENOMEM; + + ret =3D sysfs_create_file(mt_kobj_ref, &mt_offloading_attribute.attr); + if (ret) + goto out_offloading; + + ret =3D sysfs_create_file(mt_kobj_ref, &mt_threads_attribute.attr); + if (ret) + goto out_threads; + + is_dispatching =3D 0; + + return 0; + +out_threads: + sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); +out_offloading: + kobject_put(mt_kobj_ref); + return ret; +} + +static void __exit cpu_mt_module_exit(void) +{ + /* Stop the MT offloading to unload the module */ + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 1) { + stop_offloading(); + is_dispatching =3D 0; + } + mutex_unlock(&migratecfg_mutex); + + sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); + sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); + kobject_put(mt_kobj_ref); +} + +module_init(cpu_mt_module_init); +module_exit(cpu_mt_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Zi Yan"); +MODULE_DESCRIPTION("CPU_MT_COPY"); /* CPU Multithreaded Batch Migrator */ diff --git a/mm/migrate.c b/mm/migrate.c index 862a3d1eff60..e74dbc7a4758 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1831,18 +1831,13 @@ static void migrate_folios_batch_move(struct list_h= ead *src_folios, int *nr_retry_pages) { struct folio *folio, *folio2, *dst, *dst2; - int rc, nr_pages =3D 0, nr_batched_folios =3D 0; + int rc, nr_pages =3D 0, total_nr_pages =3D 0, nr_batched_folios =3D 0; int old_page_state =3D 0; struct anon_vma *anon_vma =3D NULL; int is_thp =3D 0; LIST_HEAD(err_src); LIST_HEAD(err_dst); =20 - if (mode !=3D MIGRATE_ASYNC) { - *retry +=3D 1; - return; - } - /* * Iterate over the list of locked src/dst folios to copy the metadata */ @@ -1892,8 +1887,10 @@ static void migrate_folios_batch_move(struct list_he= ad *src_folios, old_page_state & PAGE_WAS_MAPPED, anon_vma, true, ret_folios); migrate_folio_undo_dst(dst, true, put_new_folio, private); - } else /* MIGRATEPAGE_SUCCESS */ + } else { /* MIGRATEPAGE_SUCCESS */ + total_nr_pages +=3D nr_pages; nr_batched_folios++; + } =20 dst =3D dst2; dst2 =3D list_next_entry(dst, lru); --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2087.outbound.protection.outlook.com [40.107.212.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56853214A66 for ; Wed, 19 Mar 2025 19:24:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.212.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412255; cv=fail; b=l2xXHJ9kDjrm6SKj0zZV5/o/glmxfxCi1hQoIG0qp/KmmfyuK4FRYIEjgdvZV5H/2i9QFBUoioVJgj1nHmitRNRrIGXLWDMeHhrYKzEVwHZlHQhwbIIfW4PdFJ/766d1YvZ9pC0AXP3cy8Syuw1hDh+teVnJMQMUluUpers4kyk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412255; c=relaxed/simple; bh=zoBJMZnCsHBreLGRFqINdcwUQR6uqGw11E67yjy6m10=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XkZesOZ5GpgVBh3n+bqANgbE12nKMkbEY/PBpG0CF/8CEGOmoBKFNhVDP7Qjb19b4SBvDzUDDZ17NabhCtSkM8tGYF/xx17pdXWsrPxYvV7+bkMH5+MjHb4b2uyb3j6Pic5KceQAka3MwaORf/HrrYfQhc/kgxFO6DFKChWbw8I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=hca4S8Vb; arc=fail smtp.client-ip=40.107.212.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="hca4S8Vb" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TvfQ621e1doNOrcBS1py9X0CzzNjVIY/YO1wRKFxunsk0WuwNc/foKf5vWEM++fHD7cqofmaxJmDdj0BbC8OsBPmWaPhbrKZpoFfCfPeZCCDv33AgCdUGBd2SOKlY/BJPqV4zJEyT/ZfG1SfgsrIDn8eT/yQh3fFdXG70iQL69E5Z6qjXBqzfRQsGmq3xNWwNy1/cQt61LRYqAWoBIRhSRJoPKfHvl0qnn5Y4JmXfmiPzLNk2ipT3BQ7NZMWOsakVW73Hta8hJP/lXty6DdxbDtqcLVmp57f0Qk4Qv5Fbh4bXXdv0nBbxQEe3E7fbq1+XXVx90A5jR01mBvbAdvodQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LpXcxvR43WjPNZga97MpcBVDDzX5U8hqC4yBCUzSERo=; b=zG0dbuRmRnykwBuMMrJrwMX2j+IwEeCKmmmJryqh6zFpVSeqh+LdZpJsZ92+cMaByba5M+LKWIJkqIkEY7fYdPDztcJR22iLvpzajm08cdy04YNbWewNC4fAYvSLNgFuLADZTPIt6hK+9rF9NSWR0ZerpJO8i16JX6H617TuEi8SMi5ePGPcOva/HJGulAFf8ew3rE03oWGOkD6ST915ohumsuWtPyc4zE6gd3wKQOe9VS42ylbo0ykMLPpoMnPCoGjTGPfpqPwuiqQatvs5vuFoY7MIO/qQCC3tSIigDhGbpmPZn3ZiDXVmSS/tbB6/Q8oi2hLGAuEH9605hMShrQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LpXcxvR43WjPNZga97MpcBVDDzX5U8hqC4yBCUzSERo=; b=hca4S8VbCULl5rMm26fdTMYeH9s1LyBi50x0JiCk0gcNHn1UKAbHOViGmVAqcWhNOEJIxCmjHuueJ0z4XKRV3G6coc5eKz8r2/BJ8HgpWc1rgYJLcc1x8bSc5eFNav3AQE8XuNLAjRbY/RXkbcFHpnoOsqw/6SOK2qUpZjpkQDg= Received: from CH2PR14CA0036.namprd14.prod.outlook.com (2603:10b6:610:56::16) by LV2PR12MB5846.namprd12.prod.outlook.com (2603:10b6:408:175::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:24:06 +0000 Received: from CH1PEPF0000AD7A.namprd04.prod.outlook.com (2603:10b6:610:56:cafe::b3) by CH2PR14CA0036.outlook.office365.com (2603:10b6:610:56::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:24:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD7A.mail.protection.outlook.com (10.167.244.59) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:24:06 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:58 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 7/9] dcbm: add dma core batch migrator for batch page offloading Date: Wed, 19 Mar 2025 19:22:10 +0000 Message-ID: <20250319192211.10092-8-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD7A:EE_|LV2PR12MB5846:EE_ X-MS-Office365-Filtering-Correlation-Id: 2d478642-28ce-4af5-b404-08dd671b9dcf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?+xQDoEqiktFI/GctzlhTV4b01Jap78eIsK+aQeK8LsVChK7eoGmEprWSrQa2?= =?us-ascii?Q?hCVr9wAYyNSu3UkSsoqfbYby0C96ZHqFDKyCvUgb86UJoeLowlLdbaU+sDtu?= =?us-ascii?Q?X4c6nET7I7bYzzYi/qPnip1xpT9xAigd0j+tvg2SLxW9wTKlt2u1RsPpd+ES?= =?us-ascii?Q?m+5RVI2yCITfHhxkU9PmSmn1n+LGyE3KwvThHLC8acf9TX5VJmzo0c5ojmKZ?= =?us-ascii?Q?GNlzg6gX3hlPlAMg3fhlry8SiWWf8NrywCNehRuXn3fcovRpMN+xFbZHWQa5?= =?us-ascii?Q?+u/AAlSYfukQAF43tiKQakrfcL9r3fE+VLZTIVct1U0MCWZpt+DW7y+lpdnB?= =?us-ascii?Q?KqC6nXk3DBLw13h6NY3FcA2JqevLOIeoX8WnHoZRkz6pzKjAUrfG31QbZ631?= =?us-ascii?Q?LrGR8ysRgDZxj6ckTp6/KJVbpytD+n33+b9iSCPA5JpSnmO1YtDErZtj0IKl?= =?us-ascii?Q?7j/yJLs0bzBlqf8v+S1jwGKv9y22ZqhqhAetz/lBHXCl1wEmgyLrh28BieVV?= =?us-ascii?Q?O9N6bx2UBbTfiK7I+0ma+ERrj3nhS6x6w3si94oUZ25/GVncv1+fwnLU5bgd?= =?us-ascii?Q?TvPLL1kuLoQPmeFkPlGglF3QyFxoqIRZq1oKZEVRuslndLBK5ZrToe4k6fqp?= =?us-ascii?Q?kQalbgfa8sniTYZlFBOKfcQhf7IJ1pWJPIcdE1JNh4k6xz/AKPfX7iRt70e0?= =?us-ascii?Q?XRnMzlB/TjKs1iAK/Qv9KS4QQVpaB/Y0E5eiPQRvkDjk8rQauhnAbSoWHc07?= =?us-ascii?Q?UtL1w5XqX8mxSXV1KpbC1UwsYHTVZnXwjuqu154lZpe9boAl+gjYI+OC9T2r?= =?us-ascii?Q?kv2X/DNSZHosLNeczQQHnWl5+FuKmlMqChFyHv8VmPO0B1BLB/NYsHbLBgw/?= =?us-ascii?Q?s/ayXNMZDNRdp7zVzBVFHDFvQxskAfhD0cbY5AUuNEz+Vt0Fnf74bjGPjL8G?= =?us-ascii?Q?TtzU5GvDuewQkI14WJXlaQUCypQ5MghooIGppzXfkTXufhGJdaHrT5SrM1Iw?= =?us-ascii?Q?hcuP0+HN6bF9zJXGzPDFcjGKLfYJTmNkpDlGSUXU/66dyqTg2pH5+gOs9eic?= =?us-ascii?Q?KM3k6y/NslIdvRA/e+R4MG6YrjigV1FlKZloSzvLrO5HBqMKaVWB0CKrOGjB?= =?us-ascii?Q?sARg9lRWsdRF38fFdIDO/KbqUJ6rYl+kHJ2ijBz8cwLasCq2FMdtDTXCXazl?= =?us-ascii?Q?1nS3lCjtJeKNSjgCCC7TVPxUSRIhmlFpkPBHX6xDRzwCH99mqF0Dyd01gXiS?= =?us-ascii?Q?LoEoR52wgNmHHhz2lkIkZb9kb95D/ukF7bDcv21wqQMkfsDzPqz/UnxDiu1v?= =?us-ascii?Q?5ELs8DmWjj15V14AsAIAsbyH1Wdj5GUD+qQiFd24+0Ucd3qZuHX5OO2UGN72?= =?us-ascii?Q?WZkn0rVUPu8jUCCK6cJwThy6Ks+cz3kwt7HgcvzJKTuP+J2rDys9VYoKxjxI?= =?us-ascii?Q?344mgNw/4IOSQ3FDsgRZXoPj4/OulCrJIFAiQIadoIVbc5vw2znKtoHarMQA?= =?us-ascii?Q?4aMzQh4FdBflFdU=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:24:06.5266 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2d478642-28ce-4af5-b404-08dd671b9dcf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD7A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5846 Content-Type: text/plain; charset="utf-8" The dcbm (DMA core batch migrator) provides a generic interface using DMAEngine for end-to-end testing of the batch page migration offload feature. Enable DCBM offload: echo 1 > /sys/kernel/dcbm/offloading echo NR_DMA_CHAN_TO_USE > /sys/kernel/dcbm/nr_dma_chan Disable DCBM offload: echo 0 > /sys/kernel/dcbm/offloading Signed-off-by: Shivank Garg --- drivers/migoffcopy/Kconfig | 8 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/dcbm/Makefile | 1 + drivers/migoffcopy/dcbm/dcbm.c | 393 +++++++++++++++++++++++++++++++ 4 files changed, 403 insertions(+) create mode 100644 drivers/migoffcopy/dcbm/Makefile create mode 100644 drivers/migoffcopy/dcbm/dcbm.c diff --git a/drivers/migoffcopy/Kconfig b/drivers/migoffcopy/Kconfig index e73698af3e72..c1b2eff7650d 100644 --- a/drivers/migoffcopy/Kconfig +++ b/drivers/migoffcopy/Kconfig @@ -6,4 +6,12 @@ config MTCOPY_CPU Interface MT COPY CPU driver for batch page migration offloading. Say Y if you want to try offloading with MultiThreaded CPU copy APIs. +config DCBM_DMA + bool "DMA Core Batch Migrator" + depends on OFFC_MIGRATION && DMA_ENGINE + default n + help + Interface DMA driver for batch page migration offloading. + Say Y if you want to try offloading with DMAEngine APIs + based driver. =20 diff --git a/drivers/migoffcopy/Makefile b/drivers/migoffcopy/Makefile index 0a3c356d67e6..dedc86ff54c1 100644 --- a/drivers/migoffcopy/Makefile +++ b/drivers/migoffcopy/Makefile @@ -1 +1,2 @@ obj-$(CONFIG_MTCOPY_CPU) +=3D mtcopy/ +obj-$(CONFIG_DCBM_DMA) +=3D dcbm/ diff --git a/drivers/migoffcopy/dcbm/Makefile b/drivers/migoffcopy/dcbm/Mak= efile new file mode 100644 index 000000000000..56ba47cce0f1 --- /dev/null +++ b/drivers/migoffcopy/dcbm/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_DCBM_DMA) +=3D dcbm.o diff --git a/drivers/migoffcopy/dcbm/dcbm.c b/drivers/migoffcopy/dcbm/dcbm.c new file mode 100644 index 000000000000..185d8d2502fd --- /dev/null +++ b/drivers/migoffcopy/dcbm/dcbm.c @@ -0,0 +1,393 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * + * DMA batch-offlading interface driver + * + * Copyright (C) 2024 Advanced Micro Devices, Inc. + */ + +/* + * This code exemplifies how to leverage mm layer's migration offload supp= ort + * for batch page offloading using DMA Engine APIs. + * Developers can use this template to write interface for custom hardware + * accelerators with specialized capabilities for batch page migration. + * This interface driver is end-to-end working and can be used for testing= the + * patch series without special hardware given DMAEngine support is availa= ble. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_DMA_CHANNELS 16 + +static int is_dispatching; +static int nr_dma_chan; + +static int folios_copy_dma(struct list_head *dst_list, struct list_head *s= rc_list, int folios_cnt); +static int folios_copy_dma_parallel(struct list_head *dst_list, struct lis= t_head *src_list, int folios_cnt, int thread_count); +static bool can_migrate_dma(struct folio *dst, struct folio *src); + +static DEFINE_MUTEX(migratecfg_mutex); + +/* DMA Core Batch Migrator */ +struct migrator dmigrator =3D { + .name =3D "DCBM\0", + .migrate_offc =3D folios_copy_dma, + .can_migrate_offc =3D can_migrate_dma, + .owner =3D THIS_MODULE, +}; + +static ssize_t offloading_set(struct kobject *kobj, struct kobj_attribute = *attr, + const char *buf, size_t count) +{ + int ccode; + int action; + + ccode =3D kstrtoint(buf, 0, &action); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + /* + * action is 0: User wants to disable DMA offloading. + * action is 1: User wants to enable DMA offloading. + */ + switch (action) { + case 0: + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 1) { + stop_offloading(); + is_dispatching =3D 0; + } else + pr_debug("migration offloading is already OFF\n"); + mutex_unlock(&migratecfg_mutex); + break; + case 1: + mutex_lock(&migratecfg_mutex); + if (is_dispatching =3D=3D 0) { + start_offloading(&dmigrator); + is_dispatching =3D 1; + } else + pr_debug("migration offloading is already ON\n"); + mutex_unlock(&migratecfg_mutex); + break; + default: + pr_debug("input should be zero or one, parsed as %d\n", action); + } + return sizeof(action); +} + +static ssize_t offloading_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", is_dispatching); +} + +static ssize_t nr_dma_chan_set(struct kobject *kobj, struct kobj_attribute= *attr, + const char *buf, size_t count) +{ + int ccode; + int action; + + ccode =3D kstrtoint(buf, 0, &action); + if (ccode) { + pr_err("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + if (action < 1) { + pr_err("%s: invalid value, at least 1 channel\n",__func__); + return -EINVAL; + } + if (action >=3D MAX_DMA_CHANNELS) + action =3D MAX_DMA_CHANNELS; + + mutex_lock(&migratecfg_mutex); + nr_dma_chan =3D action; + mutex_unlock(&migratecfg_mutex); + + return sizeof(action); +} + +static ssize_t nr_dma_chan_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", nr_dma_chan); +} + +static bool can_migrate_dma(struct folio *dst, struct folio *src) +{ + +// printk("folio_size %d\n",folio_size(src)); + if (folio_test_hugetlb(src) || folio_test_hugetlb(dst) || + folio_has_private(src) || folio_has_private(dst) || + (folio_nr_pages(src) !=3D folio_nr_pages(dst))) { + pr_err("can NOT DMA migrate this folio %p\n",src); + return false; + } + return true; +} + +/** + * DMA channel and track its transfers + */ +struct dma_channel_work { + struct dma_chan *chan; + struct completion done; + int active_transfers; + spinlock_t lock; +}; + +/** + * Callback for DMA completion + */ +static void folios_dma_completion_callback(void *param) +{ + struct dma_channel_work *chan_work =3D param; + + spin_lock(&chan_work->lock); + chan_work->active_transfers--; + if (chan_work->active_transfers =3D=3D 0) + complete(&chan_work->done); + spin_unlock(&chan_work->lock); +} + +/** + * process dma transfer: preparation part: map, prep_memcpy + */ +static int process_folio_dma_transfer(struct dma_channel_work *chan_work, + struct folio *src, struct folio *dst) +{ + struct dma_chan *chan =3D chan_work->chan; + struct dma_device *dev =3D chan->device; + struct device *dma_dev =3D dmaengine_get_dma_device(chan); + dma_cookie_t cookie; + struct dma_async_tx_descriptor *tx; + enum dma_ctrl_flags flags =3D DMA_CTRL_ACK | DMA_PREP_INTERRUPT; + dma_addr_t srcdma_handle, dstdma_handle; + size_t data_size =3D folio_size(src); + + /* Map source and destination pages */ + srcdma_handle =3D dma_map_page(dma_dev, &src->page, 0, data_size, DMA_TO_= DEVICE); + if (dma_mapping_error(dma_dev, srcdma_handle)) { + pr_err("src mapping error\n"); + return -ENOMEM; + } + + dstdma_handle =3D dma_map_page(dma_dev, &dst->page, 0, data_size, DMA_FRO= M_DEVICE); + if (dma_mapping_error(dma_dev, dstdma_handle)) { + pr_err("dst mapping error\n"); + dma_unmap_page(dma_dev, srcdma_handle, data_size, DMA_TO_DEVICE); + return -ENOMEM; + } + + /* Prepare DMA descriptor */ + tx =3D dev->device_prep_dma_memcpy(chan, dstdma_handle, srcdma_handle, + data_size, flags); + if (unlikely(!tx)) { + pr_err("prep_dma_memcpy error\n"); + dma_unmap_page(dma_dev, dstdma_handle, data_size, DMA_FROM_DEVICE); + dma_unmap_page(dma_dev, srcdma_handle, data_size, DMA_TO_DEVICE); + return -EBUSY; + } + + /* Set up completion callback */ + tx->callback =3D folios_dma_completion_callback; + tx->callback_param =3D chan_work; + + /* Submit DMA transaction */ + spin_lock(&chan_work->lock); + chan_work->active_transfers++; + spin_unlock(&chan_work->lock); + + cookie =3D tx->tx_submit(tx); + if (dma_submit_error(cookie)) { + pr_err("dma_submit_error\n"); + spin_lock(&chan_work->lock); + chan_work->active_transfers--; + spin_unlock(&chan_work->lock); + dma_unmap_page(dma_dev, dstdma_handle, data_size, DMA_FROM_DEVICE); + dma_unmap_page(dma_dev, srcdma_handle, data_size, DMA_TO_DEVICE); + return -EINVAL; + } + + return 0; +} + +/** + * Copy folios using DMA in parallel. + * Divide into chunks, submit to DMA channels. + * if error, falls back to CPU + * Note: return 0 for all cases as error is taken care. + * TODO: Add poison recovery support. + */ +int folios_copy_dma_parallel(struct list_head *dst_list, + struct list_head *src_list, + int folios_cnt_total, int thread_count) +{ + struct dma_channel_work *chan_works; + struct dma_chan **channels; + int i, actual_channels =3D 0; + struct folio *src, *dst; + dma_cap_mask_t mask; + int channel_idx =3D 0; + int failed =3D 0; + int ret; + + /* TODO: optimise actual number of channels needed + at what point DMA set-up overheads < mig cost for N folio*/ + thread_count =3D min(thread_count, folios_cnt_total); + + /* Allocate memory for channels */ + channels =3D kmalloc_array(thread_count, sizeof(struct dma_chan *), GFP_K= ERNEL); + if (unlikely(!channels)) { + pr_err("failed to allocate memory for channels\n"); + folios_copy(dst_list, src_list, folios_cnt_total); + return 0; + } + + /* Request DMA channels */ + dma_cap_zero(mask); + dma_cap_set(DMA_MEMCPY, mask); + for (i =3D 0; i < thread_count; i++) { + channels[i] =3D dma_request_channel(mask, NULL, NULL); + if (!channels[i]) { + pr_err("could only allocate %d DMA channels\n", i); + break; + } + actual_channels++; + } + + if (unlikely(actual_channels =3D=3D 0)) { + pr_err("couldn't allocate any DMA channels, falling back to CPU copy\n"); + kfree(channels); + folios_copy(dst_list, src_list, folios_cnt_total); + return 0; + } + + /* Allocate work structures */ + chan_works =3D kmalloc_array(actual_channels, sizeof(*chan_works), GFP_KE= RNEL); + if (unlikely(!chan_works)) { + pr_err("failed to allocate memory for work structures\n"); + for (i =3D 0; i < actual_channels; i++) + dma_release_channel(channels[i]); + kfree(channels); + folios_copy(dst_list, src_list, folios_cnt_total); + return 0; + } + + /* Initialize work structures */ + for (i =3D 0; i < actual_channels; i++) { + chan_works[i].chan =3D channels[i]; + init_completion(&chan_works[i].done); + chan_works[i].active_transfers =3D 0; + spin_lock_init(&chan_works[i].lock); + } + + /* STEP 1: Submit all DMA transfers across all channels */ + dst =3D list_first_entry(dst_list, struct folio, lru); + list_for_each_entry(src, src_list, lru) { + ret =3D process_folio_dma_transfer(&chan_works[channel_idx], src, dst); + if (unlikely(ret)) { + /* Fallback to CPU */ + folio_copy(dst, src); + failed++; + } + + channel_idx =3D (channel_idx + 1) % actual_channels; + + dst =3D list_next_entry(dst, lru); + } + + /* STEP 2: Issue all pending DMA requests */ + for (i =3D 0; i < actual_channels; i++) { + dma_async_issue_pending(chan_works[i].chan); + } + + /* STEP 3: Wait for all DMA operations to complete */ + for (i =3D 0; i < actual_channels; i++) { + wait_for_completion(&chan_works[i].done); + } + + if (failed) + pr_err("processed %d fallback with CPU\n", failed); + + /* Release all resources */ + for (i =3D 0; i < actual_channels; i++) { + dma_release_channel(channels[i]); + } + + kfree(chan_works); + kfree(channels); + + return 0; +} + +/** + * Similar to folios_copy but use dma. + */ +static int folios_copy_dma(struct list_head *dst_list, + struct list_head *src_list, + int folios_cnt) +{ + return folios_copy_dma_parallel(dst_list, src_list, folios_cnt, nr_dma_ch= an); +} + +static struct kobject *kobj_ref; +static struct kobj_attribute offloading_attribute =3D __ATTR(offloading, 0= 664, + offloading_show, offloading_set); +static struct kobj_attribute nr_dma_chan_attribute =3D __ATTR(nr_dma_chan,= 0664, + nr_dma_chan_show, nr_dma_chan_set); + +static int __init dma_module_init(void) +{ + int ret =3D 0; + + kobj_ref =3D kobject_create_and_add("dcbm", kernel_kobj); + if (!kobj_ref) + return -ENOMEM; + + ret =3D sysfs_create_file(kobj_ref, &offloading_attribute.attr); + if (ret) + goto out; + + ret =3D sysfs_create_file(kobj_ref, &nr_dma_chan_attribute.attr); + if (ret) + goto out; + + is_dispatching =3D 0; + nr_dma_chan =3D 1; + + return 0; +out: + kobject_put(kobj_ref); + return ret; +} + +static void __exit dma_module_exit(void) +{ + /* Stop the DMA offloading to unload the module */ + sysfs_remove_file(kobj_ref, &offloading_attribute.attr); + sysfs_remove_file(kobj_ref, &nr_dma_chan_attribute.attr); + kobject_put(kobj_ref); +} + +module_init(dma_module_init); +module_exit(dma_module_exit); + +/* DMA Core Batch Migrator */ +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Shivank Garg"); +MODULE_DESCRIPTION("DCBM"); --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2089.outbound.protection.outlook.com [40.107.220.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50138214A7D for ; Wed, 19 Mar 2025 19:24:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.89 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412263; cv=fail; b=nc9daJkpbA0CjjfiFrH4RiKR8O/AFtmdjAxkHAlMrvvvHtrHFNxNj0LMxj+eAGHSjyE1R/dnAd6jfg22qYFhiNaGism/uADvpzv1Nnhr5bkqGu6jtLiC+Pal0Tpusns2YTz1rxykS6OLY6BGvFAKF/eSuU7Dihs5x3l6kX1mFbE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412263; c=relaxed/simple; bh=hZK7sg3yKRMdU+rwmyJOPmRRgRmnAbB2q0yF43t3LWk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rRxFQv4eNefBmDV0uCBp7jJ2rGjk/A5jFz10sFEsdrO3L+44Aam1D32DdCxJlIDySPioupyzuLJSz/ML88MC0S6cWPwtE17L5P6qyuzHQlnzYWrvmbFRN1y8msT5NjJpFEC+IPjmdNv5UKPCajB+ohY5EE4GeSAQViqJyPTWOeI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QiPbmOhV; arc=fail smtp.client-ip=40.107.220.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QiPbmOhV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KOCcuIf3vvCy8AdALaVQQ+6KxS9tLGdGp8OCnhrHzmdlYg7OfJNBlNQoEWhCif8TWC5qTVndN20QXmr1rGvG6h1oFExJsjWBOh7598GFRGoJ6Qr8RV5CuonPq8M6PS+UB5tlm0eB/onjp8el1Zho4NAW5QULVHsc1TzFfx6H1e6V7AvE3DnRrFqfs9UT8RaT8xsDip/qCgXrsR8brDsTYHcJWHzqtxhnuK1xqHRmZ8w/8tFg/aO4VJ+xb6vMMvRn+Iq5p2zgIgm6OSLcu4amN7ikuYcmQErKFI4UdStYWwEIQ10LZ7AfJ+vfK8XuN7ip2Ku4gegOxZj6Lla4/62YPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZaI3tqmugpKqCcloLD1mp5kwytZpVfANTVAtyTPxnVQ=; b=OPQppiH2T3dOBzraecc1tQmj7so62ucYz8tFcQX11ZhyRUObDxoRhmmTZq0IFlKCgw549M9UFaXnJJpfE+rSYiqQVb7x5SvySZmK4YxS13LvFRRhvaTiIoP4TwokpkUW6569MM2lsL0wtCf+w5ZoitDvluAbWq8EUBLcfxRpY4DwJQG+bNJbzdW6//tY2HDCznmXjW51kZ/8RRhVjhmr1NGXyeWstfSrwdS2L+yVpjz8bo5cv/I0fKvnSiJwyylEysdTcknq797RJj4Y+F4ssUlPPdwdHCM7U7oqr/0czLy/ZJR2D+SVlMEsATCMsxmE3tygSI5shNOz9COe/CD4DA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZaI3tqmugpKqCcloLD1mp5kwytZpVfANTVAtyTPxnVQ=; b=QiPbmOhVcZjr5zNIHlHKgkC3f2n86ldOQskpll2clBYusFDyk6a7ZsbVH/mFyBUAo7CVb9OnAmuwFsY9mNB3Gyi2qJ4CfSFm4UuEO3VTNUmTknfZ80/7Zd3aGxP3HCoPRFpC/GpUGFCDbczYYAxPJgQJKdD8smiFRgo+CNKMMDA= Received: from CH2PR10CA0013.namprd10.prod.outlook.com (2603:10b6:610:4c::23) by SN7PR12MB8601.namprd12.prod.outlook.com (2603:10b6:806:26e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:24:16 +0000 Received: from CH1PEPF0000AD78.namprd04.prod.outlook.com (2603:10b6:610:4c:cafe::8a) by CH2PR10CA0013.outlook.office365.com (2603:10b6:610:4c::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:24:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD78.mail.protection.outlook.com (10.167.244.56) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:24:14 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:24:06 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 8/9] adjust NR_MAX_BATCHED_MIGRATION for testing Date: Wed, 19 Mar 2025 19:22:11 +0000 Message-ID: <20250319192211.10092-9-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD78:EE_|SN7PR12MB8601:EE_ X-MS-Office365-Filtering-Correlation-Id: c373366d-91f9-4806-12bb-08dd671ba2a0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|1800799024|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?K1fQyQGAe8p6vVUZFiVp36ZxQyIGAjDDmr/rfLUlFlOXld4x7strnOzoIVlO?= =?us-ascii?Q?U5YQmN8ZhyOiOK/sMBc9KUKgfuxaTuij3ZGPbK4oOnWIo6qfeNHpVTqZJ0mv?= =?us-ascii?Q?+PmCa9BUcRwWX+dwEeMLCT28BZEtA8Hr9z7ZFdJXJLNyR1JivaR2xFKL/ER6?= =?us-ascii?Q?Q2VKcMyUBY7aXuy++6MpLVDmwmXNLJ+LIF2eV5Kbuh2LCGR66E/+MBUE3RgV?= =?us-ascii?Q?crxtK1zgxmFiAEDY3v2MWgtY6k3rb9KPugtSwVmzJ74uAWPDG1XQsMSJsn2v?= =?us-ascii?Q?3CSSG0MlFESDGDKj7VlJ5Bjo8Fw9K1R2VdO17BT+SrXvjYsVxUZt5MEHYn2K?= =?us-ascii?Q?S3ZN1AO048TMZKu4pyinMKwG/P9FHA7wnHEjQKxOhuQdQdUK8rxHIrDZ5P4s?= =?us-ascii?Q?eDTjVYCjuHU4bhJ2Q9woDgGpVpexKHMb1JGS1rZ45ZfAHJmRrygQrMk1Ybip?= =?us-ascii?Q?WlxenNhN+NtONkCubv2i3KSZ/WMFO0sAx3ot8q6lF5FA7XaA28mEV3zczVj6?= =?us-ascii?Q?4y8gwQMmdg4/mVoSfmJ6Zd995L/9kXpNLF3eKZwvMNqr7+HI9Q9XxP64bELB?= =?us-ascii?Q?G5ip+li1CUA4op7FItT7ECIOH+mZtAMrn+K/Sqj8JqzAgVyKbjghspptzYgi?= =?us-ascii?Q?+eKUQjQF3F8/73KjMHNtthpQrW5VYCi7ImEzYL2qiUr64haV4p4ILZym/yxY?= =?us-ascii?Q?UaftqOAMNtz/7V7YxKd5qaNSIYmLkJIwLSzXTAPD/vZdbBW2ETH6P1+r57IT?= =?us-ascii?Q?CGZAJGK8bXGcHfYw0thYmpSS4ea05Qa/kol7zg6xPtJcFRy2NFbojCyDjarb?= =?us-ascii?Q?uYklaXR7BzqIr8dEnYSwPe7QsVvAT/KaklZJvf38vS4Q8Ss/EvMv6B/UKW2N?= =?us-ascii?Q?PN2dXSlCwu86diYoZp2sT2nRqvStow8B+6GGUPTKppf+IVJg2TT9BZnAXosD?= =?us-ascii?Q?QHhc2/RVGHx0YDQyS286O4mFSUdIhJzqvxhmWEcyFh7MSpF2EUmBGRwmCmn+?= =?us-ascii?Q?4nGb3hQP5d4zC88WONbl4K08N3eHvF2cK9zWN+4PzsUMZ9trVxNb7Y7bBfm2?= =?us-ascii?Q?i1GFxXx4X1fN6XgkapGqGpFaKKuPZ6c5VHX73iUiNsQzMvHcZ9LwwS9XdUzH?= =?us-ascii?Q?rk1zrRc3pnRg5siBJTq3HLBCxJma5PFsOBqiqGa3yPvgoYz1+wvxGxrX68ev?= =?us-ascii?Q?rrdVWrk2Cy2Tno5Cqb4mh73P2qnt8y+FOy14ucxSekyEP0BMTIHQKmm/9Aze?= =?us-ascii?Q?c4RCiOkCQfRyNLbwipJZbG4tth/pgJTBvoVGMx0oGPqWuLHNZt2TDP5djPBp?= =?us-ascii?Q?VbG3codCW+V8ObOZklT33o9odKAujJg/lGprDwbErcJrQZ47w8squQvCyrAH?= =?us-ascii?Q?9bkk4ujpKVvnIj/HLngYBJr1FkHmTD4WkQtYDSMKAw6ysNH+nXebksZ/PpN1?= =?us-ascii?Q?9uL4HvEsdZtf7cjqps6Fy3tvI6k2IU4yFBLEnqYefky7A6E7AR0o42EYnFa1?= =?us-ascii?Q?51NsujRXp59Th9M=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:24:14.5562 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c373366d-91f9-4806-12bb-08dd671ba2a0 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD78.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8601 Content-Type: text/plain; charset="utf-8" From: Zi Yan change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies. These are for testing purpose only. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index e74dbc7a4758..f9eea16a975d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1653,7 +1653,7 @@ static inline int try_split_folio(struct folio *folio= , struct list_head *split_f } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR #else #define NR_MAX_BATCHED_MIGRATION 512 #endif --=20 2.34.1 From nobody Wed Dec 17 05:28:29 2025 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2062.outbound.protection.outlook.com [40.107.102.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8328A2144C8 for ; Wed, 19 Mar 2025 19:24:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412270; cv=fail; b=q1Y/tVuLjM54wVTrq18hY7ucgsY4ECFaD4JNcRuU8fX0OsMeRTymqiY6ejB7nrsthN7OSjsWlB8aXxw/bDhgI+yfZrgEMb+UQjZizdbsf8wsBMF10pOrEFodYLTkvndEFk7/ZnyJDnnYuJ/fKmFO+zor4JoBR1QXb3OWzuDeCi0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412270; c=relaxed/simple; bh=NKRZ/xWWdUU4TQOO73M07MNOWFvDUFQGTrnUqZA+lEo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XYnO6xS/IEmNroPTRhx+LF6Db5dG7rv6NYGoiN7uRAWKQHsIFWuW25sU4iGrgOCaFx8RrCjsDCFh48LYLo2oV4Hdbzs/+soi6r9lY4MjAaDGeIKGfcm+xQ2WDFkIUogExxzThdTw2N77UfkmNUU/GJAAHWXq0lM5HRwOeV04rBg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=pSL+Qafx; arc=fail smtp.client-ip=40.107.102.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="pSL+Qafx" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nQriWq3ZXlUSxGN5mClOuyTuKJs+tFeNJP1FAOXrL9IIdT8K7Ll/vgxUqNq8K+T4NtNZO7EFFvTuZNQspcFg71iw4q6mu/IYDwrr/65BLHTOVHI6dchw2UxP0QTIzqHOCgVoL654h8D0zAcSXBy2G3b5y0HtajDUEChiZY/qsUVogpf3wlevtD3xMcYD8cNXGBnYNfH5PZbWN6wnYexsa02b7HJe3tm8uotzK8uRNF/uLLxvNl5HkryL2janmh+6TlxmfkqI0YTB0N20dvQPtFTaStT18oMWC0QNYTz7+BFyuMkVkF8fIHWDMGj/nCHvdngjr2buk2OxuSX0/xdCyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=c8P+xOLNadTEZ5L2TBrKUW55HXRM6KmjsOy9cLgaKdE=; b=IPYtHdfe3hSXcZ6uJU7Ga5DgIB8zFXak0nffgG60ApzPCN/P17O4LkwjOzIBYlj5wfePX+imc0GxgGLx4oSmGWecIbKs8S5esI3tjePRQey1hR+Sl2Y6/JGrHinMbG7KJlzPiPqB1E1xbkFuiV1e+9M7eejnDICYwwkCCrynzzK65rJE3cDjmMPlfIRYO5PUFq0EGn6K+8Knl1OIzyv0/3pS0aFCzycC2yqqdyRxvNwmY5j+HUX01EQlqmmCAlPDXLKYTMQdXgnjWcH3PxhiMnUNk1+m1Kt7eeQeh2ljxji68h5XaIX0PLZOmv7fGGX+CJoA1UqVex963lbMLI3/Tw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c8P+xOLNadTEZ5L2TBrKUW55HXRM6KmjsOy9cLgaKdE=; b=pSL+QafxoE77gAYvKVjOIo28l49ybnj3kTIfqekB0W6EODp2JmbGnDdPW5pOPjTWnEfviVSh1h4psoSGTSU2NZEydTea8nbeBgHPz+nKsnCVhkzPMESunmXwt3xu9r46BKEWwxh+iX3pbbf9n17+W04F/yP+jSoNxdht4Ke3Za0= Received: from CH2PR10CA0015.namprd10.prod.outlook.com (2603:10b6:610:4c::25) by CH3PR12MB8331.namprd12.prod.outlook.com (2603:10b6:610:12f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:24:25 +0000 Received: from CH1PEPF0000AD78.namprd04.prod.outlook.com (2603:10b6:610:4c:cafe::f9) by CH2PR10CA0015.outlook.office365.com (2603:10b6:610:4c::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:24:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD78.mail.protection.outlook.com (10.167.244.56) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:24:25 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:24:14 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 9/9] mtcopy: spread threads across die for testing Date: Wed, 19 Mar 2025 19:22:12 +0000 Message-ID: <20250319192211.10092-10-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD78:EE_|CH3PR12MB8331:EE_ X-MS-Office365-Filtering-Correlation-Id: 79fb7930-f6ad-44f3-293e-08dd671ba8ee X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?V6niP53U8rM7xPccJMkBaqVBh8LIChEV0FEUf1bYsp+egNW7sxbH7/GapdCy?= =?us-ascii?Q?TmBgzqxOS210W+IYR5LkQHSp+EDHDqae/WD4MpZ+4eZOY08VlA3R1GPOOc2+?= =?us-ascii?Q?bWUmozKS2KAjflFTRvW8D532aTEH4OYysjWDd5t8429QKl5XPRZUlbPTkWWR?= =?us-ascii?Q?Wwnvzxs+GRDgrBZAmB+hp73EDEDrVhNYajX7b3FGnWYHLfLF/kW1AftFaDp8?= =?us-ascii?Q?8E7L+q7mrdkj7u4Uw/mc/aKBBQGiEPlI53CUNdV4VmX5jRttjDvR7YqztNtF?= =?us-ascii?Q?3vq48wG5n6lUXT7xO2eK/ddIhPnn8L+SxA7oqeJPHfcG7Sl3e3TzMEpLfe1x?= =?us-ascii?Q?jZE2K9rL6KCeiEYGNfI6OGI5WfjeVwgL/K8eSvUkuVbU3qHk6TERQOX0ZKda?= =?us-ascii?Q?/AmzRj5v0N63gAdf4IgaJJEBwvqeZ3IbL+yFrlfPd66oHf8b3T0Jv3KDpBdI?= =?us-ascii?Q?L+f1AtZcohLHRVAKOi0Ud0orsWIOY8XHOjKlnQHRG9E5ZUa8R06MHpF163Zo?= =?us-ascii?Q?n/LoJvJZAb2euBxVJihiOyh6sKu381Z3KMR8sQMql7AT2IPzyDqxAizoO45q?= =?us-ascii?Q?64g0avJzXu+maDHIZ+M42RZ3HlpuvyVeQ7kwsGU06wLsqp93LyoDxJgT153p?= =?us-ascii?Q?M2LjUJGBGlXk30okcNTkeRB0glo4SBkgG6nbcLYyO9hUlF3e4/SF2La+4tNm?= =?us-ascii?Q?64LWElBCBgPdbSpY+gMDyfRXwuXi5l6BkHQ577MVQUplVfX9tvt4XEkEQ8E1?= =?us-ascii?Q?6y2SX/bzdDW+eq48mZ95TioyLvfaiE+ay8KHq6fGynhjm05Odub9o7Z/gAJr?= =?us-ascii?Q?z4f0vbubfVQvyVkujrSijNQCFunj53uaR2bUiaMeB/Q2wkOiRidUtx2LKmKA?= =?us-ascii?Q?KeA87ljO7pLgpz/JKMSAJ4Urqw8DAeRdSI5eTH/kQb5d/DUeyhmOdvEhoVp5?= =?us-ascii?Q?TWNGl5cmpLHcPjRCK4VhuJfqm3Yf6URIrYiQHnAbDmLGzE7nAHSOnVEkc4Gk?= =?us-ascii?Q?bWeKgfDLZ3mSmSTXMlf17jMpzbzpKwkknsoVuX/5c4g0BFLUfQI4LPqrsznX?= =?us-ascii?Q?qWCxcJoJPSgPAMvqv0W74dLHvi36QI5Q6EZJgv77cCH1W0axJrrzQs8jtK0R?= =?us-ascii?Q?YhXZUe3ao6EKaGHJ0L9E2IF2fxY7tr+CpMWp+xNcTOf7DFbrvHBojm9aE9Zz?= =?us-ascii?Q?xEUqnbUtg06cc6xYG8bdsi3FMR1EIuSLBdHKflYUMKCZf8hv68mKk/1QV7rj?= =?us-ascii?Q?07AAGJWP1O0O9GFRjQnPfeLzbDL87yJbRUN94mCE5xGHODHwmC8NwSpnZq69?= =?us-ascii?Q?yEd7lsOE5q3wPIXPDwHLPdtz+do/UHAcWAM1B+ZBQ4M2iSUvBJm99esk6Bs1?= =?us-ascii?Q?Cg2aFWAECwexaBreZs6M+EqIZ22K6hwfGKAiR/l0661YiEKx0tnXh+NpFOE3?= =?us-ascii?Q?z6OZ8iIc0lQm6QnazT8PnLjsoGSJkK3gdPhToiTichdLD2Wyq4mIsXE899/Z?= =?us-ascii?Q?0t3rgnbAlG7uoXs=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:24:25.1655 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 79fb7930-f6ad-44f3-293e-08dd671ba8ee X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD78.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8331 Content-Type: text/plain; charset="utf-8" Select CPUs using sysfs For testing purpose only. Signed-off-by: Shivank Garg --- drivers/migoffcopy/mtcopy/copy_pages.c | 77 +++++++++++++++++++++++++- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/drivers/migoffcopy/mtcopy/copy_pages.c b/drivers/migoffcopy/mt= copy/copy_pages.c index 4c9c7d90c9fd..5178e6846890 100644 --- a/drivers/migoffcopy/mtcopy/copy_pages.c +++ b/drivers/migoffcopy/mtcopy/copy_pages.c @@ -15,11 +15,37 @@ #include #include =20 -#define MAX_NUM_COPY_THREADS 64 +#define MAX_NUM_COPY_THREADS 32 =20 unsigned int limit_mt_num =3D 4; static int is_dispatching; =20 +static int cpuselect =3D 0; + +// spread across die +static const int cpu_id_list_0[] =3D + {0, 8, 16, 24, + 32, 40, 48, 56, + 64, 72, 80, 88, + 96, 104, 112, 120, + 128, 136, 144, 152, + 160, 168, 176, 184, + 192, 200, 208, 216, + 224, 232, 240, 248}; + +// don't spread, fill the die +static const int cpu_id_list_1[] =3D + {0, 1, 2, 3, + 4, 5, 6, 7, + 8, 9, 10, 11, + 12, 13, 14, 15, + 16, 17, 18, 19, + 20, 21, 22, 23, + 24, 25, 26, 27, + 28, 29, 30, 31}; + +int cpu_id_list[32] =3D {0}; + static int copy_page_lists_mt(struct list_head *dst_folios, struct list_head *src_folios, int nr_items); static bool can_migrate_mt(struct folio *dst, struct folio *src); @@ -143,6 +169,40 @@ static ssize_t mt_threads_show(struct kobject *kobj, return sysfs_emit(buf, "%u\n", limit_mt_num); } =20 +static ssize_t mt_cpuselect_set(struct kobject *kobj, struct kobj_attribut= e *attr, + const char *buf, size_t count) +{ + int ccode; + unsigned int cpuconfig; + + ccode =3D kstrtouint(buf, 0, &cpuconfig); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + mutex_lock(&migratecfg_mutex); + cpuselect =3D cpuconfig; + switch (cpuselect) { + case 1: + memcpy(cpu_id_list, cpu_id_list_1, MAX_NUM_COPY_THREADS*sizeof(int)); + break; + default: + memcpy(cpu_id_list, cpu_id_list_0, MAX_NUM_COPY_THREADS*sizeof(int)); + break; + } + + mutex_unlock(&migratecfg_mutex); + + return count; +} + + +static ssize_t mt_cpuselect_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", cpuselect); +} + static bool can_migrate_mt(struct folio *dst, struct folio *src) { return true; @@ -218,7 +278,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, } =20 for (cpu =3D 0; cpu < total_mt_num; ++cpu) - queue_work(system_unbound_wq, + queue_work_on(cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); } else { int num_xfer_per_thread =3D nr_items / total_mt_num; @@ -255,7 +315,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, dst2 =3D list_next_entry(dst, lru); =20 if (per_cpu_item_idx =3D=3D work_items[cpu]->num_items) { - queue_work(system_unbound_wq, + queue_work_on(cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); per_cpu_item_idx =3D 0; cpu++; @@ -286,6 +346,8 @@ static struct kobj_attribute mt_offloading_attribute = =3D __ATTR(offloading, 0664, mt_offloading_show, mt_offloading_set); static struct kobj_attribute mt_threads_attribute =3D __ATTR(threads, 0664, mt_threads_show, mt_threads_set); +static struct kobj_attribute mt_cpuselect_attribute =3D __ATTR(cpuselect, = 0664, + mt_cpuselect_show, mt_cpuselect_set); =20 static int __init cpu_mt_module_init(void) { @@ -303,10 +365,18 @@ static int __init cpu_mt_module_init(void) if (ret) goto out_threads; =20 + ret =3D sysfs_create_file(mt_kobj_ref, &mt_cpuselect_attribute.attr); + if (ret) + goto out_cpuselect; + + memcpy(cpu_id_list, cpu_id_list_0, MAX_NUM_COPY_THREADS*sizeof(int)); + is_dispatching =3D 0; =20 return 0; =20 +out_cpuselect: + sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); out_threads: sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); out_offloading: @@ -324,6 +394,7 @@ static void __exit cpu_mt_module_exit(void) } mutex_unlock(&migratecfg_mutex); =20 + sysfs_remove_file(mt_kobj_ref, &mt_cpuselect_attribute.attr); sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); kobject_put(mt_kobj_ref); --=20 2.34.1