From nobody Sun Feb 8 07:17:54 2026 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2071.outbound.protection.outlook.com [40.107.93.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8723234987 for ; Wed, 21 May 2025 08:03:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.71 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747814625; cv=fail; b=okZodoMlF9RzbpnNefD0BlxmR1engpnV52lrScY7hr5Q6LxCcq+mJxwzU1r7dJkTzKi0HZj9ngYGFhWiHwms1sn1PGwzDxYIrbfparrPXkavFwrfMGzjB2ubA7hoT6l3aMbu2UbB986KNav1382W+XNPoVoP4+D2lXkNguV7Vrw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747814625; c=relaxed/simple; bh=YyHZT3d+hGf1ycysCuo2rFVHfvInsZOXL9cu01mtTKY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=uIEaS/0Xr1maHzxCVNtNhTZlxFmhfXi1FsxKOkLG3T+o15AEW6MBVYNrTnmQ6FOcU1BK/g8e0BYJpia0dnEXd+Z9QTxGQGCCLi2gnVC9NCVyIMIl2GRbjHCwStKWfkxEEfir2nfSZ4YReVwomRLp6aByo2XnI0jLSp0aN9CZAiA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=03tgen3r; arc=fail smtp.client-ip=40.107.93.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="03tgen3r" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AThd6koEcJjpmNAu3LO6JhyfwI+N1zSxfAiciSCFr1eoWbZgHorppFDAWF4tCOJHDWQeSGEa68eVJrz+zTXSB8InQCCcLMEOYMYzA6vTvBHCdKSYkmWvPH7RbXM2TPVjeX9JN+6QccfBNUVKY7FdqkR1J6GpojFT9n4Qg/uHwBgEgRQK0flVmhDgHE4bxIoHeKlmnOgH6Tt9SHDjB8ourGl31Hu/0iKtyeJnkU07w2QUw4/gkOQzFObnnjH+/KFWgAh9fDBV7IBJuMb3smiBYhkISEn5OtjsGZl0ni5Y8QvxYnDlej5GjgPPlwNuo+02msJn9Y/0tzfupRgYrn25Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2VQ0ngAkiMVh1sIFmUYAtBMFFYeBOEQbdTCivu4Eodk=; b=qxfxy9A8MHHLiRraSxtEDQlLbZUJts7tAjiLlnvr90Qbn2oQ5RUxYqGq5Vsa00nLzrWiYGvFkDC2gNhfk5u6k0bDYxDLK4g9t7TM3i0gGtn18FQFnVgRsXX7ZGeC5BxtDxixmyU4rgkF1GOo4SR+O5dCZg/PHVre5j134aa5cMeHLDdDrQuFv4tdYTOAyuHqWy96bQzgt/de9MbgTxK6kYixZ1+rQNx+RnYtlZVyeH+mJRpXRcl8LgU5kskRVH7w6t/3exQzQ+7Y84whz5UjnFD3KbswLKUrhd2OJCuLiIRQee9K0svofbWeLXtFClhbqgapG7/SpU/3/cGp6kD8hg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2VQ0ngAkiMVh1sIFmUYAtBMFFYeBOEQbdTCivu4Eodk=; b=03tgen3rzv30So5Ha4GnOXJX6uCmUXz9NoO+vif80FfMzxNUnW9BU9k5MFTQCUDUCPdjC4YoFH/ACm7SM8+2WwvuHgwph5CHAc+p9mvEANZ1GVFurERha495EZfpTS1Pmkov4DPwNmotfnZBAKWCyVSWPmqV+4P7GRlQTriAank= Received: from CH2PR14CA0008.namprd14.prod.outlook.com (2603:10b6:610:60::18) by PH7PR12MB6787.namprd12.prod.outlook.com (2603:10b6:510:1ad::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8722.33; Wed, 21 May 2025 08:03:39 +0000 Received: from DS2PEPF00003443.namprd04.prod.outlook.com (2603:10b6:610:60:cafe::46) by CH2PR14CA0008.outlook.office365.com (2603:10b6:610:60::18) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8769.19 via Frontend Transport; Wed, 21 May 2025 08:03:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS2PEPF00003443.mail.protection.outlook.com (10.167.17.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8769.18 via Frontend Transport; Wed, 21 May 2025 08:03:39 +0000 Received: from BLR-L-BHARARAO.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 21 May 2025 03:03:32 -0500 From: Bharata B Rao To: , CC: , , , , , , , , , , , , , , , , , , , , , , Bharata B Rao Subject: [RFC PATCH v0 1/2] migrate: implement migrate_misplaced_folio_batch Date: Wed, 21 May 2025 13:32:37 +0530 Message-ID: <20250521080238.209678-2-bharata@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250521080238.209678-1-bharata@amd.com> References: <20250521080238.209678-1-bharata@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003443:EE_|PH7PR12MB6787:EE_ X-MS-Office365-Filtering-Correlation-Id: 897d14db-da90-429f-39bb-08dd983dfedb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?EibvMinFQLLCvEPoS22goV8FFJ2f9dbLm8wUhBrmZDCKyyiE6BeXH/0tfgYh?= =?us-ascii?Q?by0A8eyMVCO5IIAy/q6Fc3sxHQ9rxMp9MHUcND+z/lxQubb0ATenLR/gOVHN?= =?us-ascii?Q?TNRqNfhERib1Zo5VJr+9MhR8TGIZl24NaLzXZYUdiPmvZU5QJYRG/vieK7bc?= =?us-ascii?Q?61XBOeTOKPPuc3wnuMv+W+cezMsLsaVhOZY5JhWjKsgHKtOPAV3XvAtfBvHJ?= =?us-ascii?Q?xPobAvcf/udv1QcwWcqrq12jHKpHgffM6joNDK7MB6pyPUuBT4C7F+GMN+vg?= =?us-ascii?Q?fD5Jwy4hSBOgFsuQJzLGJe9tshFheu87bg7F/6AcZOe0IGFColgBvG0N+2DU?= =?us-ascii?Q?LGmjigDcsndYjZ5otXPlNSNu1e7z+w1woBeETFrNdHtO6Q/6detoiid3IEaG?= =?us-ascii?Q?fMyfspjZgWfSogFWXSCqW3vpMTQpooPGQ94fciBoSENaqxyXRw+/IPBoTZgn?= =?us-ascii?Q?WHDs0fZioTclYoBUoklvNQ3BsSjDcgWY0lRZThcM+wp1+H+4XWeycPVeTe+8?= =?us-ascii?Q?IRCSY45HJ4ieSof3DTOPdRxrQekSrY+PrUy0L+Q6lGlDyAnMbEhQxTC0/iNy?= =?us-ascii?Q?Mkh83guOeMCbG7Aiaw5buojTnUPrmwSVfT/cxehfEqzSvtgXjnxajF0Fneug?= =?us-ascii?Q?TdJ8EfWbreCW7O161JzYiCUq47K/OhULvaHxez2pU07tIPO4SCD09w0S1wvo?= =?us-ascii?Q?NZZZDNbua2RXG8kRBdcavz/jKoJebN7Bq3MjcEmljOWYqvdK6eX1CTkTmjn1?= =?us-ascii?Q?y0IgmFf/eObg+U7OsE3tXuVnUAQuLX4GG0rbkYSPeuNtfMxjWaK4Yu37CQnN?= =?us-ascii?Q?QRSEuOg1BBapiXcc3zGdfsOS1ZK0ITa+W/cpGR0XGpt1hFWrFPL6Wbgf5+Jg?= =?us-ascii?Q?JONesJZqIYvzi21XPc73Ym/Tgu5q6g0YnSRbHZp9mkLbc+YUXpQGzyCmyaJg?= =?us-ascii?Q?pEw3c1vdLfJ0QJVmDsy++J4BMvWasMehGBFxHH+uUYdxJgB9tIaTxSmY7pLC?= =?us-ascii?Q?HbqEcUWqj3froZz/wo2bBCNYiJpS38AZ/p/zioeLRisKkHaMiC2r8fxbEoNg?= =?us-ascii?Q?DItLyoAUDbUfTD0AQfOLLmcgO67AOo0scP+xtfQrryIVoAj06tfObSecBVRv?= =?us-ascii?Q?SoL+SGE7EKUvqA5lnVrRBAAVImuhlAsIg7DSGcoY9mKz/W8y8zXayJF4wG4z?= =?us-ascii?Q?hoM2vUpL5hKqq996P42qPbX5WcNA2sf2RZ8aVdnSlhfWg4Jch4keutfmwYr1?= =?us-ascii?Q?iNBRnByMsEngoWPGwGY9rbDID49CuLopl0jDURFhCAcEd/v0X5wHqogO3NT2?= =?us-ascii?Q?cYxT/SWdBsJ1eN8I2aE/BsUMeCPKlnxyDzREgrOQzfuOQg6fr5qVMFJdSiGz?= =?us-ascii?Q?njMglwn9np98BLXseN1t+H3eye1HDnAjr+F+T1RUbZYAAurMTwlQLuppetK4?= =?us-ascii?Q?+AwvlH0GeTcCs4qzbwxaMlIMm/6wxcBaoz3uxwAHCNyOtSTZ/Cf6WFLHb28m?= =?us-ascii?Q?/49pX1VYXgpBeXLgMd+phi+Lk+wOim+4nSUh?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2025 08:03:39.1944 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 897d14db-da90-429f-39bb-08dd983dfedb X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003443.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6787 Content-Type: text/plain; charset="utf-8" From: Gregory Price A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations. Signed-off-by: Gregory Price Signed-off-by: Bharata B Rao --- include/linux/migrate.h | 6 ++++++ mm/migrate.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index aaa2114498d6..c9496adcf192 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -145,6 +145,7 @@ const struct movable_operations *page_movable_ops(struc= t page *page) int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); +int migrate_misplaced_folio_batch(struct list_head *foliolist, int node); #else static inline int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node) @@ -155,6 +156,11 @@ static inline int migrate_misplaced_folio(struct folio= *folio, int node) { return -EAGAIN; /* can't migrate now */ } +static inline int migrate_misplaced_folio_batch(struct list_head *foliolis= t, + int node) +{ + return -EAGAIN; /* can't migrate now */ +} #endif /* CONFIG_NUMA_BALANCING */ =20 #ifdef CONFIG_MIGRATION diff --git a/mm/migrate.c b/mm/migrate.c index 676d9cfc7059..32cc2eafb037 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2733,5 +2733,36 @@ int migrate_misplaced_folio(struct folio *folio, int= node) BUG_ON(!list_empty(&migratepages)); return nr_remaining ? -EAGAIN : 0; } + +/* + * Batch variant of migrate_misplaced_folio. Attempts to migrate + * a folio list to the specified destination. + * + * Caller is expected to have isolated the folios by calling + * migrate_misplaced_folio_prepare(), which will result in an + * elevated reference count on the folio. + * + * This function will un-isolate the folios, dereference them, and + * remove them from the list before returning. + */ +int migrate_misplaced_folio_batch(struct list_head *folio_list, int node) +{ + pg_data_t *pgdat =3D NODE_DATA(node); + unsigned int nr_succeeded; + int nr_remaining; + + nr_remaining =3D migrate_pages(folio_list, alloc_misplaced_dst_folio, + NULL, node, MIGRATE_ASYNC, + MR_NUMA_MISPLACED, &nr_succeeded); + if (nr_remaining) + putback_movable_pages(folio_list); + + if (nr_succeeded) { + count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded); + mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded); + } + BUG_ON(!list_empty(folio_list)); + return nr_remaining ? -EAGAIN : 0; +} #endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */ --=20 2.34.1 From nobody Sun Feb 8 07:17:54 2026 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2072.outbound.protection.outlook.com [40.107.220.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58CB320D509 for ; Wed, 21 May 2025 08:05:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.72 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747814705; cv=fail; b=twT3nYH4ghs7o1HnnxDLR3iRDjjMAZhm6TLgUyQuDc1jCB+VfFuScmNCeMUEYXTI/gHoahW4HHSTPaU4kBTo7vYUGNnLeF7WBwxMXP6/XDVAeEms1HxAJHBrfnz73tIbIdS+uK4OuEKfADnimPf9AoDZlYLHaPF0gqXKvVygQkE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747814705; c=relaxed/simple; bh=6Nv+sZcRe4cseEUcVuEgogjtncqQ6JnCgyFh76W4HkI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HA2ifKQ5JPUog0v9uuDc5Rcdf7AmZCvrxsyUsyQAkhwKwR0jd2hPRb5nETE+wM4+jazCf0N1P5/8g9CYSzEuLh3Gae3qz5zi2EjeJ9lpYAE8VTEU3nfVxKqTG/DDO4rbT/44ABhpKLAxG6AfV0iKsfhQYi9ctt+KUsBQUg57pSw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=YzfW55Dc; arc=fail smtp.client-ip=40.107.220.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="YzfW55Dc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XcxiTNX/RliENQQGGAIVo6sJ6MkHOnkT+f8ZAEQeVWALSuTOsGitix5X6Xk2liHWnOG6IOqzKF1wvr0gdAus8kNaAuRmI/Z+sY/V1bSamqTPbEiO0o5REKdzG7+M0YqZwCxkzNYr4r0vTzcKkiOhqS5VZD+0wFwkjJ9TBHyBsKjdlXALQf2Eh5vV+lc5Bxibfa/fdIvMtOELX58TrLkOKuMpEWfTDphA48BPDBAQLUGcpUIWn2wbYplmQ0a11YdnHUTYb0xPiAUsm8mP133cjN1+l8zdCF1E3jAJ8DuMoPNT0Fw14RxtscAfNY7mPMtkgJvoSJWOAqScPPZNJsZS8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PW6vB0r/b/8yFSizvGvpUEKWG6/vnpS6qXGLcnTnUkA=; b=hJa0VY7Zxdi3CVp/MUaviVYhyLrLQOjpepT8EE3UNj0MrvB97T8lmZvFHIbQD0sflN9sSdFBJ+RLPFwPICnL5lJKOCX9gBIIlgGDTtPMaLuqe/dJFs+ghUZY5oJHaN9saqmFnWvqQiuD18RYPBwkHqCBEWWve42Hau8Waf0Vq+wv28eOE/w8QcCycWp1QpeMfAtzHdP7h+G1lf79JCTEyH0mincCBSqJdFhQMZ8FfbYSDdyAjun3NxvCLxVKQ7q5VqDnddVPboK//Ljc+aQ3r4YNEbBbBjwK5GBwVWr6wU10zUfm1QHq7X+ePt4IPqRTOmvxqgPwOePSHMpeEMzRwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PW6vB0r/b/8yFSizvGvpUEKWG6/vnpS6qXGLcnTnUkA=; b=YzfW55DcUUv/CgZnPZHgKLrKVhNaWmdDo5PKj+Fr8YKyrWoQPeKirTggzaY8U8G4b4rcmNEAJQD34fmPN9/FTOC6QxH6L5gH2yFmPBcPQ2QO6qhj0xjSVWCuMf85T81jDh/RZExdD4Hqv4RqKryI+T5N8cYkzZGdJe1drTN6xks= Received: from CH0PR03CA0027.namprd03.prod.outlook.com (2603:10b6:610:b0::32) by PH0PR12MB7929.namprd12.prod.outlook.com (2603:10b6:510:284::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8746.30; Wed, 21 May 2025 08:04:59 +0000 Received: from DS2PEPF00003445.namprd04.prod.outlook.com (2603:10b6:610:b0:cafe::87) by CH0PR03CA0027.outlook.office365.com (2603:10b6:610:b0::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8769.18 via Frontend Transport; Wed, 21 May 2025 08:04:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS2PEPF00003445.mail.protection.outlook.com (10.167.17.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8769.18 via Frontend Transport; Wed, 21 May 2025 08:04:59 +0000 Received: from BLR-L-BHARARAO.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 21 May 2025 03:04:49 -0500 From: Bharata B Rao To: , CC: , , , , , , , , , , , , , , , , , , , , , , Bharata B Rao Subject: [RFC PATCH v0 2/2] mm: sched: Batch-migrate misplaced pages Date: Wed, 21 May 2025 13:32:38 +0530 Message-ID: <20250521080238.209678-3-bharata@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250521080238.209678-1-bharata@amd.com> References: <20250521080238.209678-1-bharata@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003445:EE_|PH0PR12MB7929:EE_ X-MS-Office365-Filtering-Correlation-Id: 519e9c91-95d9-4691-f178-08dd983e2e8e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?RPS3qK0Zg5J1OyWcY9kta1njzx7eOWbD41YWeZioHwBl55IUQ1syhuo8KRuz?= =?us-ascii?Q?oYISiXH4n42TxAoD4p/0w5vHmLti2a3VdtHtrSqVdF8qYwILwZnWI7uj3WL3?= =?us-ascii?Q?GMpN1Zx7gaVp/M5qZQxLMdrGnxJVDGsYmdzqC8iiQuHRTPCZLYKJWxKXQ+SY?= =?us-ascii?Q?H/Xk0ZwJ9hut1tmQ3p6HBZjaIH6sYdf5UASnLazAatKwohSscH6jbS1ewC4W?= =?us-ascii?Q?8Jgufen1Fpf1axTpcO9WUwhDH6z4pw2ylpjyvhRoc9Gr48VXVagWqooAIfMA?= =?us-ascii?Q?uKLrYHxrmLEEivD9LfiE9v75iF5IjNi7xqgYJA2FUBZba8yevFSuiBn6fgV4?= =?us-ascii?Q?Soo/mErfSMIMtN+9+/QCYypEH7v6Kd00YlR+VkDSMGFN6rVvW4H3oZ7SaS0H?= =?us-ascii?Q?2u85h6XoT7IwBDiYIowJdSIrNIuk41wpSoSg1eAtlYO3Hj4/PdvDYAo3svCJ?= =?us-ascii?Q?U0V7DHm8DAwUeqJV/P+1DovlF/sjWEJ9UU/zBP+fU1M+kVmiiEx8mNpGSn8Q?= =?us-ascii?Q?MOMBzR8euKgdlYU0AMMz3h3VpIlZwH+cjT2MrbxgIm3r+LaNTyKLak80sCOU?= =?us-ascii?Q?bmFTFXYJGWKzpflHs1504zCir2VU2BZnpZn6cYb2ctRszrcolW+gJ9/acr4m?= =?us-ascii?Q?K3ThnVVG4Uk1cEVNnOWGuqfJuGtFKHgE1oHHqAtKZbF6WNShmhlL1sP/8mCq?= =?us-ascii?Q?y93H/MniKOU1sVh1ppzm7dB5Ve0QVzOk88CFkQyNIinxJKca7DJkBqyQuJwP?= =?us-ascii?Q?szjBKRUQJQuUQWLukIBMvq2e0pv+TKR9KKPGCadD3IdGEqYlJU3gW/7aRoYG?= =?us-ascii?Q?YKZ3in5s8nI6TN7MbNTAtnOhl3A9OrTucbsZFznSSaL6J0IS9uDFCSa60tlu?= =?us-ascii?Q?0zQf9CmHvUhKzVgSRT9WZPUFOnD1bluWes7DR6Fnm+56nPli5jiHaVnkVLDv?= =?us-ascii?Q?ltRq0q0C4AoyS70zXTxzyj/SPHTcesBjmip9cP9VH5TT3Nepb93h7JmNRxCx?= =?us-ascii?Q?+PQOxW21nJWF1q5oIqyA1503rdmRn/OZs3hnS+zeJPwhDlvG2jcfbyFxvEru?= =?us-ascii?Q?UHa0WhOTmNBztSBv17KetKT23A/aqmrsoJbuUa93K3F0TyrapgcgPJ2qvgzx?= =?us-ascii?Q?l1HTUTHvKmiyjYAYAdlTYf0G9JnFORjRT1EjL6LCJtVakhqTKZPzZIc4Ga7k?= =?us-ascii?Q?h3p1+YWGJ2Z5VDyb7kLpa21TQwUJGVJ2OpVo9muMD0z679N2RoyrVOjLE7Ez?= =?us-ascii?Q?HKkbKXdQIAYsFAJZUDRdSHL2g3+qjXUzH+Cssh0UiMm8MM8KAnGO0iRGtnbf?= =?us-ascii?Q?dGC51l7HTE2tJUFz2+qQyA+0Ds4WAs03kl4eu8IWLWpUgmGOsC4p04G0nyHO?= =?us-ascii?Q?FkJX3i+ULW3/CJNZJiI8tM6rukS6B7h4eNkd2B+30T4boAekqG53qTWxEPEs?= =?us-ascii?Q?ogXOkWoi9nKF5qPbh27XQpKjZFsKbA5JVaK7GVtK5jnvEEpk7cqhP3JZX+G4?= =?us-ascii?Q?i/iPsJ/LS6eoO0NxT8+lM4dgY3lpbV5bhbOD?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 May 2025 08:04:59.2198 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 519e9c91-95d9-4691-f178-08dd983e2e8e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003445.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7929 Content-Type: text/plain; charset="utf-8" Currently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced. Instead of such singe folio migrations, batch them and migrate them at once. Identified misplaced folios are isolated and stored in a per-task list. A new task_work is queued from task tick handler to migrate them in batches. Migration is done periodically or if pending number of isolated foios exceeds a threshold. The PTEs for the isolated folios are restored to PRESENT state right after isolation. The last_cpupid field of isolated folios is used to store the target_nid to which the folios need to be migrated to. This needs changes to (at least) a couple of places where last_cpupid field is updated/reset which now should happen conditionally. The updation in folio_migrate_flags() isn't handled yet but the reset in write page fault case is handled. The failed migration count isn't fed back to the scan period update heuristics currently. Signed-off-by: Bharata B Rao --- include/linux/sched.h | 4 +++ init/init_task.c | 2 ++ kernel/sched/fair.c | 64 +++++++++++++++++++++++++++++++++++++++++++ mm/memory.c | 44 +++++++++++++++-------------- 4 files changed, 93 insertions(+), 21 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index f96ac1982893..4177ecf53633 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1360,6 +1360,8 @@ struct task_struct { u64 last_task_numa_placement; u64 last_sum_exec_runtime; struct callback_head numa_work; + struct callback_head numa_mig_work; + unsigned long numa_mig_interval; =20 /* * This pointer is only modified for current in syscall and @@ -1397,6 +1399,8 @@ struct task_struct { unsigned long numa_faults_locality[3]; =20 unsigned long numa_pages_migrated; + struct list_head migrate_list; + unsigned long migrate_count; #endif /* CONFIG_NUMA_BALANCING */ =20 #ifdef CONFIG_RSEQ diff --git a/init/init_task.c b/init/init_task.c index e557f622bd90..997af6ab67a7 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -187,6 +187,8 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = =3D { .numa_preferred_nid =3D NUMA_NO_NODE, .numa_group =3D NULL, .numa_faults =3D NULL, + .migrate_count =3D 0, + .migrate_list =3D LIST_HEAD_INIT(init_task.migrate_list), #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) .kasan_depth =3D 1, diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0fb9bf995a47..d6cbf8be76e1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -49,6 +49,7 @@ #include #include #include +#include =20 #include =20 @@ -1463,6 +1464,8 @@ unsigned int sysctl_numa_balancing_scan_delay =3D 100= 0; /* The page with hint page fault latency < threshold in ms is considered h= ot */ unsigned int sysctl_numa_balancing_hot_threshold =3D MSEC_PER_SEC; =20 +#define NUMAB_BATCH_MIGRATION_THRESHOLD 512 + struct numa_group { refcount_t refcount; =20 @@ -3297,6 +3300,46 @@ static bool vma_is_accessed(struct mm_struct *mm, st= ruct vm_area_struct *vma) =20 #define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay) =20 +/* + * TODO: Feed failed migration count back to scan period update + * mechanism. + */ +static void migrate_queued_pages(struct list_head *migrate_list) +{ + int cur_nid, nid; + struct folio *folio, *tmp; + LIST_HEAD(nid_list); + + folio =3D list_entry(migrate_list, struct folio, lru); + cur_nid =3D folio_last_cpupid(folio); + + list_for_each_entry_safe(folio, tmp, migrate_list, lru) { + nid =3D folio_xchg_last_cpupid(folio, -1); + + if (cur_nid !=3D nid) { + migrate_misplaced_folio_batch(&nid_list, cur_nid); + cur_nid =3D nid; + } + list_move(&folio->lru, &nid_list); + } + migrate_misplaced_folio_batch(&nid_list, cur_nid); +} + +static void task_migration_work(struct callback_head *work) +{ + struct task_struct *p =3D current; + + WARN_ON_ONCE(p !=3D container_of(work, struct task_struct, numa_mig_work)= ); + + work->next =3D work; + + if (list_empty(&p->migrate_list)) + return; + + migrate_queued_pages(&p->migrate_list); + p->migrate_count =3D 0; +} + /* * The expensive part of numa migration is done from task_work context. * Triggered from task_tick_numa(). @@ -3567,14 +3610,19 @@ void init_numa_balancing(unsigned long clone_flags,= struct task_struct *p) p->numa_migrate_retry =3D 0; /* Protect against double add, see task_tick_numa and task_numa_work */ p->numa_work.next =3D &p->numa_work; + p->numa_mig_work.next =3D &p->numa_mig_work; + p->numa_mig_interval =3D 0; p->numa_faults =3D NULL; p->numa_pages_migrated =3D 0; p->total_numa_faults =3D 0; RCU_INIT_POINTER(p->numa_group, NULL); p->last_task_numa_placement =3D 0; p->last_sum_exec_runtime =3D 0; + p->migrate_count =3D 0; + INIT_LIST_HEAD(&p->migrate_list); =20 init_task_work(&p->numa_work, task_numa_work); + init_task_work(&p->numa_mig_work, task_migration_work); =20 /* New address space, reset the preferred nid */ if (!(clone_flags & CLONE_VM)) { @@ -3596,6 +3644,20 @@ void init_numa_balancing(unsigned long clone_flags, = struct task_struct *p) } } =20 +static void task_check_pending_migrations(struct task_struct *curr) +{ + struct callback_head *work =3D &curr->numa_mig_work; + + if (work->next !=3D work) + return; + + if (time_after(jiffies, curr->numa_mig_interval) || + (curr->migrate_count > NUMAB_BATCH_MIGRATION_THRESHOLD)) { + curr->numa_mig_interval =3D jiffies + HZ; + task_work_add(curr, work, TWA_RESUME); + } +} + /* * Drive the periodic memory faults.. */ @@ -3610,6 +3672,8 @@ static void task_tick_numa(struct rq *rq, struct task= _struct *curr) if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next = !=3D work) return; =20 + task_check_pending_migrations(curr); + /* * Using runtime rather than walltime has the dual advantage that * we (mostly) drive the selection from busy threads and that the diff --git a/mm/memory.c b/mm/memory.c index 49199410805c..11d07004cb04 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3375,8 +3375,13 @@ static inline void wp_page_reuse(struct vm_fault *vm= f, struct folio *folio) * Clear the folio's cpupid information as the existing * information potentially belongs to a now completely * unrelated process. + * + * If the page is found to be isolated pending migration, + * then don't reset as last_cpupid will be holding the + * target_nid information. */ - folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1); + if (folio_test_lru(folio)) + folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1); } =20 flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); @@ -5766,12 +5771,13 @@ static void numa_rebuild_large_mapping(struct vm_fa= ult *vmf, struct vm_area_stru =20 static vm_fault_t do_numa_page(struct vm_fault *vmf) { + struct task_struct *task =3D current; struct vm_area_struct *vma =3D vmf->vma; struct folio *folio =3D NULL; int nid =3D NUMA_NO_NODE; bool writable =3D false, ignore_writable =3D false; bool pte_write_upgrade =3D vma_wants_manual_pte_write_upgrade(vma); - int last_cpupid; + int last_cpupid =3D (-1 & LAST_CPUPID_MASK); int target_nid; pte_t pte, old_pte; int flags =3D 0, nr_pages; @@ -5807,6 +5813,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) nid =3D folio_nid(folio); nr_pages =3D folio_nr_pages(folio); =20 + /* + * If it is a non-LRU folio, it has been already + * isolated and is in migration list. + */ + if (!folio_test_lru(folio)) + goto out_map; + target_nid =3D numa_migrate_check(folio, vmf, vmf->address, &flags, writable, &last_cpupid); if (target_nid =3D=3D NUMA_NO_NODE) @@ -5815,28 +5828,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) flags |=3D TNF_MIGRATE_FAIL; goto out_map; } - /* The folio is isolated and isolation code holds a folio reference. */ - pte_unmap_unlock(vmf->pte, vmf->ptl); writable =3D false; ignore_writable =3D true; + nid =3D target_nid; =20 - /* Migrate to the requested node */ - if (!migrate_misplaced_folio(folio, target_nid)) { - nid =3D target_nid; - flags |=3D TNF_MIGRATED; - task_numa_fault(last_cpupid, nid, nr_pages, flags); - return 0; - } - - flags |=3D TNF_MIGRATE_FAIL; - vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); - if (unlikely(!vmf->pte)) - return 0; - if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - return 0; - } + /* + * Store target_nid in last_cpupid field for the isolated + * folios. + */ + folio_xchg_last_cpupid(folio, target_nid); + list_add_tail(&folio->lru, &task->migrate_list); + task->migrate_count +=3D nr_pages; out_map: /* * Make it present again, depending on how arch implements --=20 2.34.1