From nobody Sun Feb 8 04:23:27 2026 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2059.outbound.protection.outlook.com [40.107.93.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1251AD5A for ; Sun, 1 Dec 2024 15:39:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067545; cv=fail; b=O6keAplz7TDGOshO4pDpQxLZqxXcuNQXIvQnPDwgw+To2b/uwfargVXGPUt9Pl1GxOSzPSmo7ZDPJVBU2DlYMAzl5MMMYlOHuYIVRFKbO30TArdX5uWIUi8I1+DEEwh1gHsGA8totiqqkloAZt9gQBVfupGdFJRu33lpJWizBWU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067545; c=relaxed/simple; bh=sn6DD+CwKrEhgJdEIwc+FPg5KR4Z47r5apppQmWtiBA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=CQuoJGg+7rvyDs+FmY3Pk5LUJacEOmmHCEW4WtPLm6XFlTLOyb03BXcoEUpHI8MbDOVqd/kXqBh9nxHr62mjUuOdJY4RQ1sc+qV8yYUqOd0C9FOtJ6XltKdQECjDcuZ5OFDkgK7B39eSDNAUKEt3KYR3GARatwN8+LbWQSD6z7c= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=sukvBRJl; arc=fail smtp.client-ip=40.107.93.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="sukvBRJl" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MSPRPKRshsBxfOx4VewDpH+PJjc46AwGY37+pY6k4CgmdqafQW5x25z2iJWSvjvKJHvozOgp6r9rSLoqkcankO3nqIA5gZBUzcVOTWeSjdrbgTvPxQRFl/d9EJOI5GVBONo3Hg21UOLn1JhwdjfMD2lQ/4vUM6XIAHiuk4r5zmc+52Mkg6BkZqu1F/QNNT2UE1aj+HkQ6V9F0wgTHDygmnvKoLdm6XpfR51cJNR9c+aVY/o/pkBA8ydKtesJEFeLn3bvF+3RiLe0D0cSVwn+LaxaRBSW/pY2NOApvVXGOMH3Gc6qJUUcKCSuqDc1UBD0g7RHPKmoRL3Gi9qKD89uig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/nlPY/6Qshc9zj3Hpbodb22GoEP29Mg/IgKDFMvsBNc=; b=Ld7owSr8YlAmmZQe7gtnP2q7CFA/AY15bYG+e6HbL8YR4CpHG5cvM2JrQUC04hig1jaOGJAnX0HavoG9YnnkYLrK1nkD18TKIfiUQTWbFv9VLEcYRVvmp3cI02VWhK6fnMlOnY6Ds5JSYRpGfkoB+Zns6JtjNsUWnMsLT7sPJOKedVyw2Hh7koX6MvdtSlHlLjHooR0eoF47aavaMsH0eWBUagf1gk2cdarfF7Lo+CbqwwHFyZUk8ERTBiz7WyaPDQK6qFaHyvXJqMs0Vg8RNuIZuHQBPluwSaq3dmqnrBK8U6+usryuhfNbVa2ie8kV4AMa2MKJYsvVBlPtwsNi5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/nlPY/6Qshc9zj3Hpbodb22GoEP29Mg/IgKDFMvsBNc=; b=sukvBRJlIkcMRJ+fn5riIQ4dkTu/zCBGkha0armeTp0BPWd905G4d/UVO6j0b0Y1Af4wXROe3tKA5pJf6BrG2hL4223lhOeP+XE4pyMb7zn2+jJt4FEx7WX5/MXmxfJufOTLZy3Mka4Eb4EOqqBWHzzNwWN5fHCTR6Hbpm8VxqQ= Received: from MW4PR04CA0219.namprd04.prod.outlook.com (2603:10b6:303:87::14) by IA0PR12MB7751.namprd12.prod.outlook.com (2603:10b6:208:430::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.15; Sun, 1 Dec 2024 15:38:58 +0000 Received: from CO1PEPF000044F9.namprd21.prod.outlook.com (2603:10b6:303:87:cafe::48) by MW4PR04CA0219.outlook.office365.com (2603:10b6:303:87::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.15 via Frontend Transport; Sun, 1 Dec 2024 15:38:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000044F9.mail.protection.outlook.com (10.167.241.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8251.1 via Frontend Transport; Sun, 1 Dec 2024 15:38:56 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:38:49 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 01/10] mm: Add kmmscand kernel daemon Date: Sun, 1 Dec 2024 15:38:09 +0000 Message-ID: <20241201153818.2633616-2-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044F9:EE_|IA0PR12MB7751:EE_ X-MS-Office365-Filtering-Correlation-Id: 219beb38-5627-4a3c-1cf7-08dd121e44e4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|7416014|376014|1800799024|82310400026|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?yvyWoAMG7HFH7o3xa+atHiKEqPOWqkz3Php0Yj1FzUj3UesWW4s7amGLZMKN?= =?us-ascii?Q?P9gZc07/J7XYYOFv9jy5G9L0iu21nvCYbO42G7RxvrFYq0q4sKatDZjL3fnM?= =?us-ascii?Q?hJyyRcBZMptD5h/w2XHTDy6YKRC9ezvUSwlKs8sxwTFqK4CHahb5e1yAvkxt?= =?us-ascii?Q?F03dnogHGEFFCCt6TPhmnrZk/NCCYbQkT/2y7ub2YQjZtmORANMzqmI6ecWj?= =?us-ascii?Q?eu/dYtVZJcEU0EitutumNRnb3N3+w0p+GAsP1FM1vLpM5P+ATsznUl143Pqw?= =?us-ascii?Q?BPY4uF4r2xSvpTf+xcm5wAyc9qyUrsXwPr8d19fyv1lQ41tIFnX7GTNaRi/X?= =?us-ascii?Q?LRmLz5PoYG78cei67Nnv0iEM3Jv6LGY4LoedV1CmyUsDp9yBTILNYCvAMITv?= =?us-ascii?Q?c2OptDkEwuKPQ+CoyPJ3wzORzYkbuFWvG2iLdksbbwEuNOzrKfaax2TWWLWC?= =?us-ascii?Q?n1F4pawcOzI9FgdJRdnZ/o/kQkKiisZfvDOysmWrOgpenQ78xDUN9JM4Y2i4?= =?us-ascii?Q?iFcCTHYvhEMjafpebs2YWgjCFEGsN8AaTdcuQ/s33pTF148SmgF+8v8Eg0Km?= =?us-ascii?Q?rxxnQbHgOXuIdXkGx0YD56Fj+gFUSgmmZ5UjctmBUoKc8bJFV0IqtZopGbxF?= =?us-ascii?Q?+t3xIADBzhu83V/dMzr3qYNCI36xRDQnuF5o1q6O49pMcVZLky/DWr6be7rc?= =?us-ascii?Q?WzhYjU8rqfqU8QR9cg5ynCGktgt7RrG16wgF/uZCqf14D0chB99sYbxYq2m1?= =?us-ascii?Q?1rEuG7/rz2DIQ1uSIomaL1UnRqUYUhzxnML72RIVoO83FvQ3XNmA/QCenj4u?= =?us-ascii?Q?LYBk2nmcku3vo9UQIdymdUrMfp87iOGq7XwWUy7ynyiFcmdAXUCSqPlnxgn1?= =?us-ascii?Q?n71R6Jj9Ht+eUNoAEALD7ghOMiecHEPBOh2d2/XqQzPstfbRDfOCiok7PgAS?= =?us-ascii?Q?TuIaUIPODQnRJrKvcVf+lBuiw/2PBtVgHRg7GhlOl7T42SdmEXLBnQyuBA1r?= =?us-ascii?Q?wN3V4S2hTHTbWP0Lzo6k68VDE2CJVVB6/qMtVsXzKp4miAzsk98FLC7QWXeP?= =?us-ascii?Q?gAO5phYc+DVG9cV97h9yV2sspvOvO1ZbJUEEfqMab6kCymDlSmuaiSXOqo2N?= =?us-ascii?Q?zvrC3k5o6o5WCQlFcjSXvH4IwFkM+2qOQxD3fft4O/nB2vtS2xAO6ZEkeaEb?= =?us-ascii?Q?o6f/wB5cLZS1Hah9rTjLuNKjXT/juNDXtHK/Ds/g0EMGheid0yy/W49PrXc3?= =?us-ascii?Q?OQQEygLWbMSffGxVvlV54SFDo5v6kbAesKzqmeXeypt5D+Kj36S/nn1X+cs1?= =?us-ascii?Q?6/myzDi0VRJFqhcSnm960BGC+tzSP6Ezp1zXFEpKnsuCpUCDl/kfYxNnzjyA?= =?us-ascii?Q?ZRN6DTv1kN3iQAC8NwyoeV9czeVcUCRj7gQxK40EA9ELLWdFQHWiMXaxwPA8?= =?us-ascii?Q?bp5xqNO5iB9qRhl4bMmdMHm7rrX2oMGJwXqLneQEvH2wl8y7FPVmRQ=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(7416014)(376014)(1800799024)(82310400026)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:38:56.8878 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 219beb38-5627-4a3c-1cf7-08dd121e44e4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F9.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7751 Content-Type: text/plain; charset="utf-8" Add a skeleton to support scanning and migration. Also add a config option for the same. High level design: While (1): scan the slowtier pages belonging to VMAs of a task. Add to migation list migrate scanned pages to node 0 (default) The overall code is heavily influenced by khugepaged design. Signed-off-by: Raghavendra K T --- mm/Kconfig | 7 ++ mm/Makefile | 1 + mm/kmmscand.c | 182 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 190 insertions(+) create mode 100644 mm/kmmscand.c diff --git a/mm/Kconfig b/mm/Kconfig index 84000b016808..a0b5ab6a9b67 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -740,6 +740,13 @@ config KSM until a program has madvised that an area is MADV_MERGEABLE, and root has set /sys/kernel/mm/ksm/run to 1 (if CONFIG_SYSFS is set). =20 +config KMMSCAND + bool "Enable PTE A bit scanning and Migration" + depends on NUMA_BALANCING + help + Enable PTE A bit scanning of page. CXL pages accessed are migrated to + regular NUMA node (node 0 - default). + config DEFAULT_MMAP_MIN_ADDR int "Low address space to protect from user allocation" depends on MMU diff --git a/mm/Makefile b/mm/Makefile index dba52bb0da8a..1b6b00e39d12 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -94,6 +94,7 @@ obj-$(CONFIG_FAIL_PAGE_ALLOC) +=3D fail_page_alloc.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o obj-$(CONFIG_NUMA) +=3D memory-tiers.o +obj-$(CONFIG_KMMSCAND) +=3D kmmscand.o obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) +=3D page_counter.o diff --git a/mm/kmmscand.c b/mm/kmmscand.c new file mode 100644 index 000000000000..23cf5638fe10 --- /dev/null +++ b/mm/kmmscand.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include "internal.h" + + +static struct task_struct *kmmscand_thread __read_mostly; +static DEFINE_MUTEX(kmmscand_mutex); + +/* How long to pause between two scan and migration cycle */ +static unsigned int kmmscand_scan_sleep_ms __read_mostly =3D 16; + +/* Max number of mms to scan in one scan and migration cycle */ +#define KMMSCAND_MMS_TO_SCAN (4 * 1024UL) +static unsigned long kmmscand_mms_to_scan __read_mostly =3D KMMSCAND_MMS_T= O_SCAN; + +volatile bool kmmscand_scan_enabled =3D true; +static bool need_wakeup; + +static unsigned long kmmscand_sleep_expire; + +static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); + +struct kmmscand_scan { + struct list_head mm_head; +}; + +struct kmmscand_scan kmmscand_scan =3D { + .mm_head =3D LIST_HEAD_INIT(kmmscand_scan.mm_head), +}; + +static int kmmscand_has_work(void) +{ + return !list_empty(&kmmscand_scan.mm_head); +} + +static bool kmmscand_should_wakeup(void) +{ + bool wakeup =3D kthread_should_stop() || need_wakeup || + time_after_eq(jiffies, kmmscand_sleep_expire); + if (need_wakeup) + need_wakeup =3D false; + + return wakeup; +} + +static void kmmscand_wait_work(void) +{ + if (kmmscand_has_work()) { + const unsigned long scan_sleep_jiffies =3D + msecs_to_jiffies(kmmscand_scan_sleep_ms); + + if (!scan_sleep_jiffies) + return; + + kmmscand_sleep_expire =3D jiffies + scan_sleep_jiffies; + wait_event_timeout(kmmscand_wait, + kmmscand_should_wakeup(), + scan_sleep_jiffies); + return; + } +} + +static void kmmscand_migrate_folio(void) +{ +} + +static unsigned long kmmscand_scan_mm_slot(void) +{ + /* placeholder for scanning */ + msleep(100); + return 0; +} + +static void kmmscand_do_scan(void) +{ + unsigned long iter =3D 0, mms_to_scan; + + mms_to_scan =3D READ_ONCE(kmmscand_mms_to_scan); + + while (true) { + cond_resched(); + + if (unlikely(kthread_should_stop()) || !READ_ONCE(kmmscand_scan_enabled)) + break; + + if (kmmscand_has_work()) + kmmscand_scan_mm_slot(); + + kmmscand_migrate_folio(); + iter++; + if (iter >=3D mms_to_scan) + break; + } +} + +static int kmmscand(void *none) +{ + for (;;) { + if (unlikely(kthread_should_stop())) + break; + + kmmscand_do_scan(); + + while (!READ_ONCE(kmmscand_scan_enabled)) { + cpu_relax(); + kmmscand_wait_work(); + } + + kmmscand_wait_work(); + } + return 0; +} + +static int start_kmmscand(void) +{ + int err =3D 0; + + guard(mutex)(&kmmscand_mutex); + + /* Some one already succeeded in starting daemon */ + if (kmmscand_thread) + goto end; + + kmmscand_thread =3D kthread_run(kmmscand, NULL, "kmmscand"); + if (IS_ERR(kmmscand_thread)) { + pr_err("kmmscand: kthread_run(kmmscand) failed\n"); + err =3D PTR_ERR(kmmscand_thread); + kmmscand_thread =3D NULL; + goto end; + } else { + pr_info("kmmscand: Successfully started kmmscand"); + } + + if (!list_empty(&kmmscand_scan.mm_head)) + wake_up_interruptible(&kmmscand_wait); + +end: + return err; +} + +static int stop_kmmscand(void) +{ + int err =3D 0; + + guard(mutex)(&kmmscand_mutex); + + if (kmmscand_thread) { + kthread_stop(kmmscand_thread); + kmmscand_thread =3D NULL; + } + + return err; +} + +static int __init kmmscand_init(void) +{ + int err; + + err =3D start_kmmscand(); + if (err) + goto err_kmmscand; + + return 0; + +err_kmmscand: + stop_kmmscand(); + + return err; +} +subsys_initcall(kmmscand_init); --=20 2.39.3 From nobody Sun Feb 8 04:23:27 2026 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2067.outbound.protection.outlook.com [40.107.100.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD15F1632FB; Sun, 1 Dec 2024 15:39:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067554; cv=fail; b=rNch/dAn45fQZASKPLlXHQTNatgi9JCEiMCNhP5ksWMF+8rfAJ/QvZTfxg1C6LVT72Nv0T0IAeAyMY/GvgskUsWSWjagM2WaoyVDO3N2U0bxq783eTOLtgTO066KrYD+IRMC43AZMxYVVcjjY86Ylob4zdpt86WFN3j3pwhoio8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067554; c=relaxed/simple; bh=hZy3TTWshPAxImvze0gZD3WMP0vDm2FY/LGwd0ZCsdE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XVE/UH8NIS/b0VaLfjFZv4q+OnvkpptRp5K5aVShCi2z5HNweqKoWxbzoRKpkDxFIWVAFZ6xFG2aEbeH6RQm6E228b0JNVDr9BmuCoxtufwaR6SnAuIGj3TxF6QWSzsZoY7JZxcpMdMLAt6NVjC01dCowH2tHZW0U7CFGTOmiAw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=pQ+doSH8; arc=fail smtp.client-ip=40.107.100.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="pQ+doSH8" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gb5u9F5hK/odT+SAs6kUNC1rDMGYSVBJtXm9Zwtxma3v1WuoAq4QsPfn+p+XZnteauxk3okHDM6NrVvQP3g0pEfzbu/mJLAdVGJ5lFVy0m1TIW+GNpqeuRSfeG6JKbtfcBSnLgPAnI0K7xUOg5pxArqWjvIzS/5UiS4ruQVEo3wSvmwBhFsMFKNMqtKiAEDBl4kzkU0aH2+fnvIzAk8SdkbEaIuDFzDiAKw5lKQsHsS1IUPwCc5SOtoQ3OzKjDpzw0vjEMA0aP7irmAtQGayXEJt8nNEPth+UDz2eCirBsaPNqFoxkZCpYflYByhvmF6qm+SYjVOd/xS5zIxbsO2gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=THqpOYzVxsi2M9Jxr93ZEPARXIH0jqWyq/8EdCQRhh4=; b=tgvggDPG7uUQaZBA64PfPyYFzUMI0hhuuEFZk7Fcyo04dQK4FPdo/vTuwo3GqiXlRGvXkR7aqayoROHdwGDaTseXqnBqWJONB37s5D43UcnOCklqCp+0GGc9EDEhCe7ye7h9OSHN/eH/9kdtRu4Wl6IzSLjYLFvPRmMQohexY92l93gU6ZSeuiXVgHyCrq4Y/TtBdPTJGsT+Fo7SvlYCPwjoDgHGJFmBvhPtNP4wyWc6yVzcTYXcxlBKlBZpTFJK9P8mTCkCaG++IreB33SwKKp47rKuV19sBPWtmzF9iADg7bXXrK4/UoqZYNnA6OdU0odDrW0oW69rBHYeNBOESA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=THqpOYzVxsi2M9Jxr93ZEPARXIH0jqWyq/8EdCQRhh4=; b=pQ+doSH864GJ+c/Ch66pVLvlp6uxuP4vahkZOt56OK8yzVEnUFUDIiYYSrV/nZuqJ/jNpCSDlQNHTrC18qnf7pQjRzObnjZl/xH/sBOThHf6DFMyp09p0LctaRLmBCpsbcOXzXu7tNxmhwNHgkNtHSgOS+xnWEIGCuJolONauVE= Received: from SJ0PR03CA0203.namprd03.prod.outlook.com (2603:10b6:a03:2ef::28) by DS0PR12MB9445.namprd12.prod.outlook.com (2603:10b6:8:1a1::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.15; Sun, 1 Dec 2024 15:39:04 +0000 Received: from CO1PEPF000044FB.namprd21.prod.outlook.com (2603:10b6:a03:2ef:cafe::cc) by SJ0PR03CA0203.outlook.office365.com (2603:10b6:a03:2ef::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000044FB.mail.protection.outlook.com (10.167.241.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8251.1 via Frontend Transport; Sun, 1 Dec 2024 15:39:03 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:38:56 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T , Subject: [RFC PATCH V0 02/10] mm: Maintain mm_struct list in the system Date: Sun, 1 Dec 2024 15:38:10 +0000 Message-ID: <20241201153818.2633616-3-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044FB:EE_|DS0PR12MB9445:EE_ X-MS-Office365-Filtering-Correlation-Id: 05cc5bb9-b9b7-4986-fe6b-08dd121e48b7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|36860700013|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?+xxasN5tC1YEgwOqWe7D9mzO3hoMyzCCl7+OGOoIqirHPX/+SMVbHqjhRTSf?= =?us-ascii?Q?sTuds4iwOoNBiDOaAA+1nwyaCLXuBGehRfeYhleaeM8rXJQtPYm2G6ntplVo?= =?us-ascii?Q?8A35QWIC+nuf9KYRTekb29yor9cdQQnE4g08I8UNxQ1TASQiRabUTuchECm5?= =?us-ascii?Q?SbNy8zEjFd743pUomJcFOJGQgMxQwi4zf8FR3nRf8uEJmF24brSwsMe2dPkC?= =?us-ascii?Q?JEwUIusuB3+V8Xb20fTT1N1Rp+dbH1fYO6Eh7RwD6Mwgj5EwnoZyeRd8RFv5?= =?us-ascii?Q?fhUOekLcLjTB6KMzooeD/BRa71/8nBBmBtz3TCR6/7oE0+MCGO8yh+6oXS63?= =?us-ascii?Q?9BcVEtPJSScTVGWJWTBlO6/JXOoPSQ0dz/L1SZBQ/Vuk4ndiWkFJ2x9JLKtr?= =?us-ascii?Q?UqKKRMompfQpSXX1Ji1pagoB1gwbqNxJ6BxraZ9oe4pdSgVQxBVWTYZ+SsC8?= =?us-ascii?Q?rf6MA4V638/8l04//QVFEaBzs0fbWEC6Ufn+ds005dFKvrGM76+38dSicb9j?= =?us-ascii?Q?LWWy8hFpJa0s3fiqFT/+ljlJttFRoVQg+5ew+204ViYbD/TEolWcjgvAOhz5?= =?us-ascii?Q?QecOQrYqi5L6cCRhvudR9OvklHUjpyL12V6RoMNmGMb8kN5W/bGvVAS0j+sV?= =?us-ascii?Q?+fkbr/nfJn3l24NR3yxyspleymd8+PZP4tuP95ICXcs8bv/Xtd547gnU2roR?= =?us-ascii?Q?kSRsUdxuRfB6sdwCDUg3NkRG9ti3QPA+7ZYX93NVzswdg9HUUqYjZ89noQ5j?= =?us-ascii?Q?o4j65UTly11UMbyiiIpKzWkz41vzNQ2jAK2CHmQ+ZP43EitJUzKGLObphusp?= =?us-ascii?Q?CiDwBXHSd5sBG7NHxiZd6nmXk6GbRFkyREGvR7mFzBaWUM3zAT44tNka9Eou?= =?us-ascii?Q?naYYIBJvLQMXh+pXp9w67SFq1R02gJP6d8IcXtR12/JmvjYBkSJF4ZzUKWOx?= =?us-ascii?Q?YRzyZOh7wPOSWvojLbhPtC4rehlYrIaPvhrcUfBYLyLHHpgxr+dOdmCkg3vn?= =?us-ascii?Q?Pu6GkHbM56FegXgAuqmCrRwHFil5X07ELERSMqqUgR2QB9YYeB2/VgRYc0H7?= =?us-ascii?Q?g5tLzBnhY//YIoXh997aylcAh/7pYvNrE1ElQnW2gkTTFQQD6DJwkbdhijFe?= =?us-ascii?Q?ewmPDQfrwEBQI2EOJPLVh3nun741yLQ8IlAca7XqeGn77GtJ/j8OQ4Difgm6?= =?us-ascii?Q?YFozVb1JwyRKoJyonOZPzR6QeXGLAJZvn/XtIsoysxf4MfG4eRmw0Vwv1a0v?= =?us-ascii?Q?t1OQob5QuHR3HKPRynASg8BWiosWsAuLEC6vvGozuOIWMVHsUQrGGqnf1A1F?= =?us-ascii?Q?kQr4M4l4zvvadqR1F5hHNpfl7vb++MbjOzsenlOaBN5OIxRw0FPgCVV5WMIO?= =?us-ascii?Q?4eTahIK7wsUVoM/d3R2cJ877VGr1aejbc4rqH9PF/qIs03UdgJ7EIDkPkXko?= =?us-ascii?Q?HFzRtg74gyZixCNepykqk+vf+B+HfR0TlDFKMoJVhMOiFtLw1nf2Kw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(36860700013)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:03.3252 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 05cc5bb9-b9b7-4986-fe6b-08dd121e48b7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044FB.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB9445 Content-Type: text/plain; charset="utf-8" Add a hook in the fork and exec path to link mm_struct. Reuse the mm_slot infrastructure to aid insert and lookup of mm_struct. CC: linux-fsdevel@vger.kernel.org Suggested-by: Bharata B Rao Signed-off-by: Raghavendra K T --- fs/exec.c | 4 ++ include/linux/kmmscand.h | 30 ++++++++++++++ kernel/fork.c | 4 ++ mm/kmmscand.c | 86 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 include/linux/kmmscand.h diff --git a/fs/exec.c b/fs/exec.c index 98cb7ba9983c..bd72107b2ab1 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -68,6 +68,7 @@ #include #include #include +#include =20 #include #include @@ -274,6 +275,8 @@ static int __bprm_mm_init(struct linux_binprm *bprm) if (err) goto err_ksm; =20 + kmmscand_execve(mm); + /* * Place the stack at the largest stack address the architecture * supports. Later, we'll move this to an appropriate place. We don't @@ -296,6 +299,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) return 0; err: ksm_exit(mm); + kmmscand_exit(mm); err_ksm: mmap_write_unlock(mm); err_free: diff --git a/include/linux/kmmscand.h b/include/linux/kmmscand.h new file mode 100644 index 000000000000..b120c65ee8c6 --- /dev/null +++ b/include/linux/kmmscand.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KMMSCAND_H_ +#define _LINUX_KMMSCAND_H_ + +#ifdef CONFIG_KMMSCAND +extern void __kmmscand_enter(struct mm_struct *mm); +extern void __kmmscand_exit(struct mm_struct *mm); + +static inline void kmmscand_execve(struct mm_struct *mm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *o= ldmm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_exit(struct mm_struct *mm) +{ + __kmmscand_exit(mm); +} +#else /* !CONFIG_KMMSCAND */ +static inline void __kmmscand_enter(struct mm_struct *mm) {} +static inline void __kmmscand_exit(struct mm_struct *mm) {} +static inline void kmmscand_execve(struct mm_struct *mm) {} +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *o= ldmm) {} +static inline void kmmscand_exit(struct mm_struct *mm) {} +#endif +#endif /* _LINUX_KMMSCAND_H_ */ diff --git a/kernel/fork.c b/kernel/fork.c index 1450b461d196..812b0032592e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -659,6 +660,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *= mm, mm->exec_vm =3D oldmm->exec_vm; mm->stack_vm =3D oldmm->stack_vm; =20 + kmmscand_fork(mm, oldmm); + /* Use __mt_dup() to efficiently build an identical maple tree. */ retval =3D __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL); if (unlikely(retval)) @@ -1350,6 +1353,7 @@ static inline void __mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ + kmmscand_exit(mm); exit_mmap(mm); mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 23cf5638fe10..957128d4e425 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -7,13 +7,14 @@ #include #include #include +#include #include #include #include =20 #include #include "internal.h" - +#include "mm_slot.h" =20 static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); @@ -30,10 +31,21 @@ static bool need_wakeup; =20 static unsigned long kmmscand_sleep_expire; =20 +static DEFINE_SPINLOCK(kmmscand_mm_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); =20 +#define KMMSCAND_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash, KMMSCAND_SLOT_HAS= H_BITS); + +static struct kmem_cache *kmmscand_slot_cache __read_mostly; + +struct kmmscand_mm_slot { + struct mm_slot slot; +}; + struct kmmscand_scan { struct list_head mm_head; + struct kmmscand_mm_slot *mm_slot; }; =20 struct kmmscand_scan kmmscand_scan =3D { @@ -76,6 +88,11 @@ static void kmmscand_migrate_folio(void) { } =20 +static inline int kmmscand_test_exit(struct mm_struct *mm) +{ + return atomic_read(&mm->mm_users) =3D=3D 0; +} + static unsigned long kmmscand_scan_mm_slot(void) { /* placeholder for scanning */ @@ -123,6 +140,65 @@ static int kmmscand(void *none) return 0; } =20 +static inline void kmmscand_destroy(void) +{ + kmem_cache_destroy(kmmscand_slot_cache); +} + +void __kmmscand_enter(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *kmmscand_slot; + struct mm_slot *slot; + int wakeup; + + /* __kmmscand_exit() must not run from under us */ + VM_BUG_ON_MM(kmmscand_test_exit(mm), mm); + + kmmscand_slot =3D mm_slot_alloc(kmmscand_slot_cache); + + if (!kmmscand_slot) + return; + + slot =3D &kmmscand_slot->slot; + + spin_lock(&kmmscand_mm_lock); + mm_slot_insert(kmmscand_slots_hash, mm, slot); + + wakeup =3D list_empty(&kmmscand_scan.mm_head); + list_add_tail(&slot->mm_node, &kmmscand_scan.mm_head); + spin_unlock(&kmmscand_mm_lock); + + mmgrab(mm); + if (wakeup) + wake_up_interruptible(&kmmscand_wait); +} + +void __kmmscand_exit(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *mm_slot; + struct mm_slot *slot; + int free =3D 0; + + spin_lock(&kmmscand_mm_lock); + slot =3D mm_slot_lookup(kmmscand_slots_hash, mm); + mm_slot =3D mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + if (mm_slot && kmmscand_scan.mm_slot !=3D mm_slot) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + } + + spin_unlock(&kmmscand_mm_lock); + + if (free) { + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } else if (mm_slot) { + mmap_write_lock(mm); + mmap_write_unlock(mm); + } +} + static int start_kmmscand(void) { int err =3D 0; @@ -168,6 +244,13 @@ static int __init kmmscand_init(void) { int err; =20 + kmmscand_slot_cache =3D KMEM_CACHE(kmmscand_mm_slot, 0); + + if (!kmmscand_slot_cache) { + pr_err("kmmscand: kmem_cache error"); + return -ENOMEM; + } + err =3D start_kmmscand(); if (err) goto err_kmmscand; @@ -176,6 +259,7 @@ static int __init kmmscand_init(void) =20 err_kmmscand: stop_kmmscand(); + kmmscand_destroy(); =20 return err; } --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2086.outbound.protection.outlook.com [40.107.92.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 053CF1494DD for ; Sun, 1 Dec 2024 15:39:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.86 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067557; cv=fail; b=tzPY6QOUYTom7mX0y1HfQFN2W+zrmRmsHRW52gOziSJ7uCTlehQtUDTP/yS+/FzyeRBfhXSjIDtf/YY4o1kTU5pqgGijBeEUcX9HppfAtlYADc/y+HgiXHY4e/037JDvzMuZsJ6joFYopqT6x7M/Fq+z6hOeIipwUIqYc5poWG4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067557; c=relaxed/simple; bh=FWT2RbyW5fpOd7+VBI6RDDWiseonuI1NviEWMWwWQxU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=oT/nYYDSlujlY8IeWiwYNahR1Q9uJfFs9T26rmr6FTvo4deLVMgaCLin3Z99nlAa0KHBcKRrfMItPZw22j5MCmE5IojwlHqc0yw4v5DfzMapRotpEMSPZVYiOMDKJ9F9EMzWmErZxPElBvFJlEi+Hv0F5tivKoUfl7k+9N/8J/k= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=fSCkw3cU; arc=fail smtp.client-ip=40.107.92.86 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="fSCkw3cU" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=syGeH+gKJgDqE7aysx9U6Z0bB2HqRrtiiv7quCdbiYReQQjqzriVDRkpNNiSvkNao4YkyhEPVjHPSvWs4bYioNaJx8zmX5pJ/3lcjQDkQpjiiQcFduwgNiDTzWE+zTxJ11/3W30mr9MrRzAt0lKnz6P0RtuHU2T0xiZCydFXqmRQyCA9WoZuI9OLI7JDi6qmvQ9x+ujCA8Vz6Pom7IGCyT5DBcUZSzUOB9+w57WDBEVEsLGNmD2tfF+d1joqcjdpPv9U2Ibj9Dp0kr1X4webr50mT5E2zA9bo+ECLVydtHcxq9460jIiWpEqpz+kvKiHsy2qVX0aVFs0V0jbynfxTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B3EzOrANg4iCE+eMwCm/oM95tbuBSCOb5dUX6vHV3Qk=; b=yPShB/E8V/tCWx3dq786jK41SFxdccRxJVFLGkDk2XIqf/HBS1ObOb4Q5HhadDjzeGKfjc22TOk19brqPLzKTqiffLawLwjo6xOujQogltmo0jiFtx+vA0B0yui5KpClJSXz7R4hSVNFG1z6JMuNU/Y8DSGBJfBA1S3ftikmboiM4lxhYx+Udc66B+Cc2r7fBy4wLBpwewcz27TKnPxTsz8HWQNE8JegmKo3/s2ANwiBp8loDRBpHaMSZXv1ouSb9YoRfvLbOvRbkh3pLF5kHrecJIEpJfwJS5XZiUdsHfQRXuJqBg6O4gycYRsb/VT/m2g7w0ijDd4wVQtqm/Xo6A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B3EzOrANg4iCE+eMwCm/oM95tbuBSCOb5dUX6vHV3Qk=; b=fSCkw3cUeIDnNCP3YdvcVXpStLU9kT+jaOn15dr973dKYm6J6YnN3K9fa6VFCJLb6gETXotTCkEWklBs54dT+mwz1FHxGS9faPeOtRfrh9gwbjpm3fBJiQ+nAKiZEAsCwMri5p3FJtjULP6n/v+Yo8E1y4hmu/qvYzgTvXk8CGA= Received: from SJ0PR03CA0190.namprd03.prod.outlook.com (2603:10b6:a03:2ef::15) by IA1PR12MB6412.namprd12.prod.outlook.com (2603:10b6:208:3af::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 15:39:10 +0000 Received: from CO1PEPF000044FB.namprd21.prod.outlook.com (2603:10b6:a03:2ef:cafe::75) by SJ0PR03CA0190.outlook.office365.com (2603:10b6:a03:2ef::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.15 via Frontend Transport; Sun, 1 Dec 2024 15:39:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000044FB.mail.protection.outlook.com (10.167.241.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8251.1 via Frontend Transport; Sun, 1 Dec 2024 15:39:10 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:02 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 03/10] mm: Scan the mm and create a migration list Date: Sun, 1 Dec 2024 15:38:11 +0000 Message-ID: <20241201153818.2633616-4-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044FB:EE_|IA1PR12MB6412:EE_ X-MS-Office365-Filtering-Correlation-Id: d5669c10-1f3d-411e-08b9-08dd121e4cd1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?wcg2HOEFQwkfbNvf+WHCIRz5/Qxg20G66Dn9ZbJ4zwcEqNmZV0YpLt3e7PD0?= =?us-ascii?Q?wid7dlFa8XGPAevX37fgiU8SD8eZdYReHL62U5iPgA5HVVqGfUqr/YnwW7kV?= =?us-ascii?Q?nPXfJeLb7HZFY/690TfWYNOqW2uJtjbE3OIGyqELGz0vBhILYRMfnCMDZFbV?= =?us-ascii?Q?LScv5NgA525XdEZQGo6yhU/cUibe4FEZn+kYFiW7UWmNSVggjw2/nqlrsvdl?= =?us-ascii?Q?DHdus3yNen7flSPSGnqbLM08L79f7ZqX66Oc117bOEV5NuSeFbuiSDnUQyfC?= =?us-ascii?Q?VPHxBUwE20VOTTFNOHKf3/5k54EjRXlF3bdE5306FcNweiwxJnfBM8I6NOLP?= =?us-ascii?Q?SRTkhB9UaGQXwgfpn8X37S8vrA2Jm9OVaNiSOkbFZG5DBIN7bNsbon9nLpWG?= =?us-ascii?Q?tz0WdlQVIFKeg+gSGwD+prrAUDN99ESsh7kulUm56xqNDWDGSXjm5JHCsikM?= =?us-ascii?Q?cPVSAujFSCgISUoTrwT01amvbG+Bq6mTldyBBEX1ZXgh7WkEtqZezW0py07L?= =?us-ascii?Q?l4/JY2zORCwmbe62VAwMk/QW9f7l/PoLEi3qv6ZGOaRWGGxk2W3IOB5UlqoY?= =?us-ascii?Q?ew5LrLe/qOVVksldY3Fyu9al2taD1vjva9N+2S+VWzCT9cPFv/tvXg4ArjHl?= =?us-ascii?Q?EcLYc7er8jGXy2VUYmiU2R09bVMLNneG+HVN+sTmnqXcPhew2jf2NI+q9+fM?= =?us-ascii?Q?sdXvofFgb7DNe97ODTvwDd+ugDzJOwAJEB86OgSsCrdbsWfwIeyj5mL2e541?= =?us-ascii?Q?HS7haYt6G2tyKxLPHLx0kuH5xK6Hbzko2+ncCGOONVTRJmd3WgoRJTTpDKSX?= =?us-ascii?Q?X+OQeWcaa+27PhUhUL8MaoqHMBRZ5it8RkhHMJsJPoEwQebEkGAURzJYkHQI?= =?us-ascii?Q?RYiHeKpS2VzoOC1SgGEBpkVLWnLZJo0X0H0ceqeSQ0kJB/Dqm0ciCI8ENeM7?= =?us-ascii?Q?dbk4xQAUZx1kSRVpIJWHzMfwRTPR09hVFm1iN9f4lUbcvOD8YVyuFcD05w14?= =?us-ascii?Q?70P2Fc9tkdwHEuHyCTf0SvKnLNiVN2lvxsT7eOUgmBKcCcG/fRL1WFuOPlRE?= =?us-ascii?Q?IV3GwSRvLZZB83ZQ0cUiSSK3hW6K65KTuXCnop437BIp6CsVnhHG6D9pk3xN?= =?us-ascii?Q?pcVHyDWEgRWbIGlSkcc2BVzC+VyALW6hgDvODas7bU7HBfU4YoQHOGbQWMhZ?= =?us-ascii?Q?xOT2lfk6Q62hpal7IK90KBnBVhKstka4SeXynxobJlEeRUjng87m9wm7IQ7S?= =?us-ascii?Q?IMPFJkCjYcfl/OI6ZRYigilvXNbVrBnduZBHKa1BxNZWJXV5blAN5TWykwaN?= =?us-ascii?Q?dY6CNgjxf8fDnGAP+qaq/8RtMQInIkvlIlkaPeW8J7mgEqzmj1YL+SmN6pud?= =?us-ascii?Q?18KZ0C+9dnIhYoZLOiZAczZeUaY7YM7JxeLyHEufPUvaVr6ZPD9b912edFk3?= =?us-ascii?Q?RGkrAV82EJqdvjtOQkc6gHR22tPSQS8b?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:10.1690 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d5669c10-1f3d-411e-08b9-08dd121e4cd1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044FB.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6412 Content-Type: text/plain; charset="utf-8" Since we already have list of mm_struct in the system, add a module to scan each mm that walks VMAs of each mm_struct and scan all the pages associated with that. In the scan path: Check for the recently acccessed pages (folios) belongin= g to slowtier nodes. Add all those folios to a migration list. Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 268 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 264 insertions(+), 4 deletions(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 957128d4e425..0496359d07f5 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -4,12 +4,19 @@ #include #include #include +#include +#include +#include +#include +#include +#include #include #include #include #include +#include +#include #include -#include #include =20 #include @@ -32,6 +39,7 @@ static bool need_wakeup; static unsigned long kmmscand_sleep_expire; =20 static DEFINE_SPINLOCK(kmmscand_mm_lock); +static DEFINE_SPINLOCK(kmmscand_migrate_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); =20 #define KMMSCAND_SLOT_HASH_BITS 10 @@ -41,6 +49,7 @@ static struct kmem_cache *kmmscand_slot_cache __read_most= ly; =20 struct kmmscand_mm_slot { struct mm_slot slot; + long address; }; =20 struct kmmscand_scan { @@ -52,6 +61,21 @@ struct kmmscand_scan kmmscand_scan =3D { .mm_head =3D LIST_HEAD_INIT(kmmscand_scan.mm_head), }; =20 +struct kmmscand_migrate_list { + struct list_head migrate_head; +}; + +struct kmmscand_migrate_list kmmscand_migrate_list =3D { + .migrate_head =3D LIST_HEAD_INIT(kmmscand_migrate_list.migrate_head), +}; + +struct kmmscand_migrate_info { + struct list_head migrate_node; + struct mm_struct *mm; + struct vm_area_struct *vma; + struct folio *folio; + unsigned long address; +}; static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -84,8 +108,140 @@ static void kmmscand_wait_work(void) } } =20 -static void kmmscand_migrate_folio(void) +static bool kmmscand_eligible_srcnid(int nid) { + if (!node_is_toptier(nid)) + return true; + return false; +} + +static bool folio_idle_clear_pte_refs_one(struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr, + pte_t *ptep) +{ + bool referenced =3D false; + struct mm_struct *mm =3D vma->vm_mm; + pmd_t *pmd =3D pmd_off(mm, addr); + + if (ptep) { + if (ptep_clear_young_notify(vma, addr, ptep)) + referenced =3D true; + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + if (!pmd_present(*pmd)) + WARN_ON_ONCE(1); + if (pmdp_clear_young_notify(vma, addr, pmd)) + referenced =3D true; + } else { + WARN_ON_ONCE(1); + } + + if (referenced) { + folio_clear_idle(folio); + folio_set_young(folio); + } + return true; +} + +static void page_idle_clear_pte_refs(struct page *page, pte_t *pte, struct= mm_walk *walk) +{ + bool need_lock; + struct folio *folio =3D page_folio(page); + unsigned long address; + + if (!folio_mapped(folio) || !folio_raw_mapping(folio)) + return; + + need_lock =3D !folio_test_anon(folio) || folio_test_ksm(folio); + if (need_lock && !folio_trylock(folio)) + return; + address =3D vma_address(walk->vma, page_pgoff(folio, page), compound_nr(p= age)); + VM_BUG_ON_VMA(address =3D=3D -EFAULT, vma); + folio_idle_clear_pte_refs_one(folio, walk->vma, address, pte); + + if (need_lock) + folio_unlock(folio); +} + +static int hot_vma_idle_pte_entry(pte_t *pte, + unsigned long addr, + unsigned long next, + struct mm_walk *walk) +{ + struct page *page; + struct folio *folio; + struct mm_struct *mm; + struct vm_area_struct *vma; + struct kmmscand_migrate_info *info; + struct kmmscand_migrate_list *migrate_list =3D walk->private; + int srcnid; + + pte_t pteval =3D ptep_get(pte); + + if (pte_none(pteval)) + return 1; + vma =3D walk->vma; + mm =3D vma->vm_mm; + page =3D pte_page(*pte); + + page_idle_clear_pte_refs(page, pte, walk); + + folio =3D page_folio(page); + folio_get(folio); + + if (!folio || folio_is_zone_device(folio)) { + folio_put(folio); + return 1; + } + + srcnid =3D folio_nid(folio); + + if (!folio_test_idle(folio) || folio_test_young(folio) || + mmu_notifier_test_young(mm, addr) || + folio_test_referenced(folio) || pte_young(pteval)) { + + /* Do not try to promote pages from regular nodes */ + if (!kmmscand_eligible_srcnid(srcnid)) + goto end; + + info =3D kzalloc(sizeof(struct kmmscand_migrate_info), GFP_KERNEL); + if (info && migrate_list) { + + info->mm =3D mm; + info->vma =3D vma; + info->folio =3D folio; + + spin_lock(&kmmscand_migrate_lock); + list_add_tail(&info->migrate_node, &migrate_list->migrate_head); + spin_unlock(&kmmscand_migrate_lock); + } + } +end: + folio_set_idle(folio); + folio_put(folio); + return 0; +} + +static const struct mm_walk_ops hot_vma_set_idle_ops =3D { + .pte_entry =3D hot_vma_idle_pte_entry, + .walk_lock =3D PGWALK_RDLOCK, +}; + +static void kmmscand_walk_page_vma(struct vm_area_struct *vma) +{ + if (!vma_migratable(vma) || !vma_policy_mof(vma) || + is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + return; + } + + if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) =3D=3D (VM_READ= ))) + return; + + if (!vma_is_accessible(vma)) + return; + + walk_page_vma(vma, &hot_vma_set_idle_ops, &kmmscand_migrate_list); } =20 static inline int kmmscand_test_exit(struct mm_struct *mm) @@ -93,10 +249,113 @@ static inline int kmmscand_test_exit(struct mm_struct= *mm) return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) +{ + struct mm_slot *slot =3D &mm_slot->slot; + struct mm_struct *mm =3D slot->mm; + + lockdep_assert_held(&kmmscand_mm_lock); + + if (kmmscand_test_exit(mm)) { + /* free mm_slot */ + hash_del(&slot->hash); + list_del(&slot->mm_node); + + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } +} + +static void kmmscand_migrate_folio(void) +{ +} + static unsigned long kmmscand_scan_mm_slot(void) { - /* placeholder for scanning */ - msleep(100); + bool update_mmslot_info =3D false; + + unsigned long address; + + struct mm_slot *slot; + struct mm_struct *mm; + struct vma_iterator vmi; + struct vm_area_struct *vma =3D NULL; + struct kmmscand_mm_slot *mm_slot; + + /* Retrieve mm */ + spin_lock(&kmmscand_mm_lock); + + if (kmmscand_scan.mm_slot) { + mm_slot =3D kmmscand_scan.mm_slot; + slot =3D &mm_slot->slot; + address =3D mm_slot->address; + } else { + slot =3D list_entry(kmmscand_scan.mm_head.next, + struct mm_slot, mm_node); + mm_slot =3D mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + address =3D mm_slot->address; + kmmscand_scan.mm_slot =3D mm_slot; + } + + mm =3D slot->mm; + + spin_unlock(&kmmscand_mm_lock); + + if (unlikely(!mmap_read_trylock(mm))) + goto outerloop_mmap_lock; + + if (unlikely(kmmscand_test_exit(mm))) + goto outerloop; + + + vma_iter_init(&vmi, mm, address); + + for_each_vma(vmi, vma) { + /* Count the scanned pages here to decide exit */ + kmmscand_walk_page_vma(vma); + + address =3D vma->vm_end; + } + + if (!vma) + address =3D 0; + + update_mmslot_info =3D true; + +outerloop: + /* exit_mmap will destroy ptes after this */ + mmap_read_unlock(mm); + +outerloop_mmap_lock: + spin_lock(&kmmscand_mm_lock); + VM_BUG_ON(kmmscand_scan.mm_slot !=3D mm_slot); + + if (update_mmslot_info) + mm_slot->address =3D address; + /* + * Release the current mm_slot if this mm is about to die, or + * if we scanned all vmas of this mm. + */ + if (unlikely(kmmscand_test_exit(mm)) || !vma) { + /* + * Make sure that if mm_users is reaching zero while + * kmmscand runs here, kmmscand_exit will find + * mm_slot not pointing to the exiting mm. + */ + if (slot->mm_node.next !=3D &kmmscand_scan.mm_head) { + slot =3D list_entry(slot->mm_node.next, + struct mm_slot, mm_node); + kmmscand_scan.mm_slot =3D + mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + + } else + kmmscand_scan.mm_slot =3D NULL; + + if (kmmscand_test_exit(mm)) + kmmscand_collect_mm_slot(mm_slot); + } + + spin_unlock(&kmmscand_mm_lock); return 0; } =20 @@ -159,6 +418,7 @@ void __kmmscand_enter(struct mm_struct *mm) if (!kmmscand_slot) return; =20 + kmmscand_slot->address =3D 0; slot =3D &kmmscand_slot->slot; =20 spin_lock(&kmmscand_mm_lock); --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2083.outbound.protection.outlook.com [40.107.223.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4F261D6DB4 for ; Sun, 1 Dec 2024 15:39:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.83 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067564; cv=fail; b=Gm4I5DD6zfODtFIjjJnzwZB98D1G+uhPpzVQU8sLHsWb64RGt6EJEzWB0RKyTGwGSzkPywEvS58/umcCZc+uZKyZDgDfOEUu+CX/Qj3f1nsREeT4DMziSPdPHvQfrDX73EfJL9mgdLvy01X7kXsk/OFshpmjopv57K8NOHV07b4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067564; c=relaxed/simple; bh=vapLt8f/TAUyErMgW4oujUahxOE0EHVbkwfYMlLoKt4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=d6qO/zuHe34ZfIRkGDawalKH1uwuabaeU8NBvR2qrOMuCQSoFuGLI827hTQOM1mnKgsKOrR6XhsvsHiZMkkndoywbrF0aNU1jjxqTlRkO+6aNAHmM20AvEGRb2CrI3xxI8dPIX+o+iXuDn3aceIaaDJFeCjGANsQ7mI6Lzipt7k= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=V0p5WyhP; arc=fail smtp.client-ip=40.107.223.83 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="V0p5WyhP" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Eq/wLDx6ugTfmZshqkqmyVHV9T2lCfTgmz4ibIvVB4gSJE5A10DAMqs6zhixk+gJLWZPNDbNXXDoGj+ia2TbSNcdKJgIqKwHrS5Zw9N7fB+eB24X/qC4O/Rgj0rdcDuoSqqW1JV+oXRy3W+C4+oC0DcLP3w2Bo0BL06wDbWP3gq4MGDb7ZqxKJxnuvuTSG9WB3Y+RueqQSER1hgpca3DmOYEY6xB7X5mBDepbxPx4PCy9pKdap45QJrjNrlC26W7ng9hZ2KYF81pvoNlMsKhZMrcsN8D7BnWy+hD/CUAnQrgOPgElcMhDo9xtR4jSe3B2/g084F3gjVW1jdI318j0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Se4i7oWsjWVI2dcozt7u1pG3jASFmxn3pMpuXEaD3CM=; b=ROBN0q8trI19gWNd1HweayjKHLg2EMe6BtKSR6IfYshmhvvx3tZ+3T5T1Cp4Tr2RZE+IX8X/e5miAOX5isVWgw7ove2Oi5M06tHjB9a/Y/pctfksUMSnt1DDmkSh1Gqkysi3dsowQGu8mLWTvUR/stzJqckuBMNoM9jOVFU0vwKccQsvc9ka/GEXqKRpagHdbbdbgD19OT5d9du8hK0iOj1hstODUyuQBKZq6hR1a+r0kMs8fc75RhhmQ73aMy5vgRGvyKhDWdRIgjVCPrW0358dXGHmCBHD+Jb3B0gK9RaPK3BSVqt5F37P0ne6r/tl8QIslr9whHt1gPXmXLmNUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Se4i7oWsjWVI2dcozt7u1pG3jASFmxn3pMpuXEaD3CM=; b=V0p5WyhPr85VBWbuRnbHwhW7ndb+B323uVjVWHtw+8pziiNpmyd89OX8pSYARC1pAK0BvWZ2TgPjG04KQsY8+ZFqUmQSMmozB2rQoFDsyh0dJXq8PSOuiXvwpa8HxGYN7rqq2EGGHjEgfvOyOHi86rrfFeJkd91K33aXZOCPZDw= Received: from BY3PR04CA0015.namprd04.prod.outlook.com (2603:10b6:a03:217::20) by CY8PR12MB7266.namprd12.prod.outlook.com (2603:10b6:930:56::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 15:39:17 +0000 Received: from CO1PEPF000044F7.namprd21.prod.outlook.com (2603:10b6:a03:217:cafe::7f) by BY3PR04CA0015.outlook.office365.com (2603:10b6:a03:217::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000044F7.mail.protection.outlook.com (10.167.241.197) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8251.1 via Frontend Transport; Sun, 1 Dec 2024 15:39:16 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:09 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 04/10] mm/migration: Migrate accessed folios to toptier node Date: Sun, 1 Dec 2024 15:38:12 +0000 Message-ID: <20241201153818.2633616-5-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044F7:EE_|CY8PR12MB7266:EE_ X-MS-Office365-Filtering-Correlation-Id: 0e3f6520-b0f3-4df4-344d-08dd121e50ad X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?/lyhM3aHDzOb6iHkc+rtO3hrTk0/24KGdoW3LbpckxcQLrTD3s1KO4VWcdat?= =?us-ascii?Q?D7oAHBkHxPQpWtaHjE4Je62rTpRGoxbJmqqw+Z+Qn/dEZV4R1eDq9G4RBQfh?= =?us-ascii?Q?yFPAS0dAB+/g/vmxaQnsB7sAZmMvRvvXzq9EsuzOPb+9585h64m3skR5E8nr?= =?us-ascii?Q?W2M/YCf2pYOlvGO3j3gPS4Ee3WAoJ55jHIks4TnSBPV9TnOd5PDLnvvIIixd?= =?us-ascii?Q?mZ8PN5hilKMqXSYJ2uECIG/B23MFRWGzfZexN2TsOJNAGzS4KCbudkTqQeIe?= =?us-ascii?Q?2jHlLgTAG/TnitgNtys9UfFTZ1Xy+gDFQeafXOIx3Xbrze0/jp42qEJiBem/?= =?us-ascii?Q?ddahQRAWlHivYAtatoOxjhiqUqYxxbbBeN7vjDUz/9Ocg/D5beXSyMRpetqn?= =?us-ascii?Q?qmk2qMmb8AASg7Bvlb5ZMe3B4NP3n/lmjD98ujS9UHWcO/zWr5xIX+Mw9xU2?= =?us-ascii?Q?C0WLFb7ihzz31Pnc02f/zBh2O2Ur4aHWhLfaWNGfpyayc3qAlX/zo3WSLyw8?= =?us-ascii?Q?N3o8VBpQIeEyK258TP/dr3AJRd9bJowRvdT20cEh2BWeK0L/VkybKxu1eLKi?= =?us-ascii?Q?kzRt6SxPv2UAdQAdjUzmOBrD+4How8XpbI850kWIl1jcDSRO3Gpp4ti+Xi7c?= =?us-ascii?Q?Df/RhxSXY45cZ6Vjn1hTbeJEBWZi1rovq6AsDQjePI2tlOmfkSvlfYYFU9J/?= =?us-ascii?Q?qNiINhP6W42d9cpJOdpabmuj1073C9fZ0Ea/8v6sdJ5/xZXLLyYoUbh3obd8?= =?us-ascii?Q?OPbBYaXfEcVbYVtgcK+OfkBsIJrD+UAmlGRUHUTsvhGUutNDLalxkfQTvoGV?= =?us-ascii?Q?XG1HR71jwW1NF1mfF89Ls+bh9BkN9DIwxTyVW1Pe/iaLsnFPjuciZUM/xALm?= =?us-ascii?Q?4li1SwjKuXFjfqRwTezk6Le1AL5y0n0X2gQytxSIom1nKqefP3dLtgJQKKMC?= =?us-ascii?Q?3a9Fl7kraQACqjbl+AAxbQcdeMmzjyufa8DITjdubgmAFbLi1L1z22FiRpHF?= =?us-ascii?Q?CNxQbySm6QOdh41ojo3ONuXkLgDRs5WrdoPnsdYQjGvDfbVIarg3OlvOYKAb?= =?us-ascii?Q?kXfhUuADShC3qKaEgdLtUp/GBjCcmUkbFV6vphdCbX0tpJI+qgZrglukJ7Iv?= =?us-ascii?Q?2juL6BYRHXrLEdZrriuFqbyEE0dexUzRYDT5lFZmclMxlRlpqfabZYop6vO4?= =?us-ascii?Q?K9B5SYXNacu8BffV9HAwON9r9PLt3Wy8eeFt3UC5uw0XxdDzy1toMXAD1ghH?= =?us-ascii?Q?fBCpTlLc/zU9MXD4w6bTRO+phQNu68zslnyuEmRX0/3IX7aNYVXp+aCF5aUH?= =?us-ascii?Q?aQFd/chAJJ1PjQmS5Ug/O+ze1SXAm7Xl0pbkslvFcqKt9E5geqAw45nD3f97?= =?us-ascii?Q?ezbyVKkBpeM0pr1sAHxA9Bjmws9uQalPSuAA2uOJNoVLH3Xolv9co73qLfuj?= =?us-ascii?Q?Xa5LhNcy6ZvF1FsbPdhNofu9y7z+psl8oqdJU/+BxUdZeyS63VtY3Q=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:16.6763 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0e3f6520-b0f3-4df4-344d-08dd121e50ad X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F7.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7266 Content-Type: text/plain; charset="utf-8" For each recently accessed slowtier folio in the migration list: - Isolate LRU pages - Migrate to a regular node. The rationale behind whole migration is to speedup the access to recently accessed pages. Limitation: PTE A bit scanning approach lacks information about exact destination node to migrate to. Reason: PROT_NONE hint fault based scanning is done in a process context. Here when the fault occurs, source CPU of the fault associated task is known. Time of page access is also accurate. With the lack of above information, migration is done to node 0 by default. Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 178 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 178 insertions(+) TBD: Before calling migrate_misplaced_folio, we need to hold a PTL lock. But since we are not coming from fault path, it is tricky. We need to fix this before the final patch. diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 0496359d07f5..3b4453b053f4 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -36,7 +37,15 @@ static unsigned long kmmscand_mms_to_scan __read_mostly = =3D KMMSCAND_MMS_TO_SCAN; volatile bool kmmscand_scan_enabled =3D true; static bool need_wakeup; =20 +/* mm of the migrating folio entry */ +static struct mm_struct *kmmscand_cur_migrate_mm; + +/* Migration list is manipulated underneath because of mm_exit */ +static bool kmmscand_migration_list_dirty; + static unsigned long kmmscand_sleep_expire; +#define KMMSCAND_DEFAULT_TARGET_NODE (0) +static int kmmscand_target_node =3D KMMSCAND_DEFAULT_TARGET_NODE; =20 static DEFINE_SPINLOCK(kmmscand_mm_lock); static DEFINE_SPINLOCK(kmmscand_migrate_lock); @@ -115,6 +124,107 @@ static bool kmmscand_eligible_srcnid(int nid) return false; } =20 +/* + * Do not know what info to pass in the future to make + * decision on taget node. Keep it void * now. + */ +static int kmmscand_get_target_node(void *data) +{ + return kmmscand_target_node; +} + +static int kmmscand_migrate_misplaced_folio_prepare(struct folio *folio, + struct vm_area_struct *vma, int node) +{ + if (folio_is_file_lru(folio)) { + /* + * Do not migrate file folios that are mapped in multiple + * processes with execute permissions as they are probably + * shared libraries. + * + * See folio_likely_mapped_shared() on possible imprecision + * when we cannot easily detect if a folio is shared. + */ + if (vma && (vma->vm_flags & VM_EXEC) && + folio_likely_mapped_shared(folio)) + return -EACCES; + /* + * Do not migrate dirty folios as not all filesystems can move + * dirty folios in MIGRATE_ASYNC mode which is a waste of + * cycles. + */ + if (folio_test_dirty(folio)) + return -EAGAIN; + } + + if (!folio_isolate_lru(folio)) + return -EAGAIN; + + return 0; +} + +enum kmmscand_migration_err { + KMMSCAND_NULL_MM =3D 1, + KMMSCAND_INVALID_FOLIO, + KMMSCAND_INVALID_VMA, + KMMSCAND_INELIGIBLE_SRC_NODE, + KMMSCAND_SAME_SRC_DEST_NODE, + KMMSCAND_LRU_ISOLATION_ERR, +}; + +static int kmmscand_promote_folio(struct kmmscand_migrate_info *info) +{ + unsigned long pfn; + struct page *page; + struct folio *folio; + struct vm_area_struct *vma; + int ret; + + int srcnid, destnid; + + if (info->mm =3D=3D NULL) + return KMMSCAND_NULL_MM; + + folio =3D info->folio; + + /* Check again if the folio is really valid now */ + if (folio) { + pfn =3D folio_pfn(folio); + page =3D pfn_to_online_page(pfn); + } + + if (!page || !folio || !folio_test_lru(folio) || + folio_is_zone_device(folio) || !folio_mapped(folio)) + return KMMSCAND_INVALID_FOLIO; + + vma =3D info->vma; + + /* XXX: Need to validate vma here?. vma_lookup() results in 2x regression= */ + if (!vma) + return KMMSCAND_INVALID_VMA; + + srcnid =3D folio_nid(folio); + + /* Do not try to promote pages from regular nodes */ + if (!kmmscand_eligible_srcnid(srcnid)) + return KMMSCAND_INELIGIBLE_SRC_NODE; + + destnid =3D kmmscand_get_target_node(NULL); + + if (srcnid =3D=3D destnid) + return KMMSCAND_SAME_SRC_DEST_NODE; + + folio_get(folio); + ret =3D kmmscand_migrate_misplaced_folio_prepare(folio, vma, destnid); + if (ret) { + folio_put(folio); + return KMMSCAND_LRU_ISOLATION_ERR; + } + folio_put(folio); + + return migrate_misplaced_folio(folio, vma, destnid); +} + static bool folio_idle_clear_pte_refs_one(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, @@ -266,8 +376,74 @@ static void kmmscand_collect_mm_slot(struct kmmscand_m= m_slot *mm_slot) } } =20 +static void kmmscand_cleanup_migration_list(struct mm_struct *mm) +{ + struct kmmscand_migrate_info *info, *tmp; + +start_again: + spin_lock(&kmmscand_migrate_lock); + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + + if (mm =3D=3D READ_ONCE(kmmscand_cur_migrate_mm)) { + /* A folio in this mm is being migrated. wait */ + WRITE_ONCE(kmmscand_migration_list_dirty, true); + spin_unlock(&kmmscand_migrate_lock); + goto start_again; + } + + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (info && (info->mm =3D=3D mm)) { + info->mm =3D NULL; + WRITE_ONCE(kmmscand_migration_list_dirty, true); + } + } + } + spin_unlock(&kmmscand_migrate_lock); +} + static void kmmscand_migrate_folio(void) { + int ret =3D 0; + struct kmmscand_migrate_info *info, *tmp; + + spin_lock(&kmmscand_migrate_lock); + + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (READ_ONCE(kmmscand_migration_list_dirty)) { + kmmscand_migration_list_dirty =3D false; + list_del(&info->migrate_node); + /* + * Do not try to migrate this entry because mm might have + * vanished underneath. + */ + kfree(info); + spin_unlock(&kmmscand_migrate_lock); + goto dirty_list_handled; + } + + list_del(&info->migrate_node); + /* Note down the mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, info->mm); + spin_unlock(&kmmscand_migrate_lock); + + if (info->mm) + ret =3D kmmscand_promote_folio(info); + + kfree(info); + + spin_lock(&kmmscand_migrate_lock); + /* Reset mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, NULL); + spin_unlock(&kmmscand_migrate_lock); +dirty_list_handled: + //cond_resched(); + spin_lock(&kmmscand_migrate_lock); + } + } + spin_unlock(&kmmscand_migrate_lock); } =20 static unsigned long kmmscand_scan_mm_slot(void) @@ -450,6 +626,8 @@ void __kmmscand_exit(struct mm_struct *mm) =20 spin_unlock(&kmmscand_mm_lock); =20 + kmmscand_cleanup_migration_list(mm); + if (free) { mm_slot_free(kmmscand_slot_cache, mm_slot); mmdrop(mm); --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2079.outbound.protection.outlook.com [40.107.220.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECDB31D7985 for ; Sun, 1 Dec 2024 15:39:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.79 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067573; cv=fail; b=Wxa1B6KHYsGRv0C4BE7nScXyVSrLatD3q7bubg1CnWuxr/5pYFOgzkVAmNkNplqCQ8GCqmXcWhNzsJ5Z7w9BvYB85tw0+LjgFUlNzxGwvmnjOeKove6tmMAQI/rtPn8IPcIlPIrjepPbOLUX6rVBmgtdfkzAvoNuBhFtrLGojyg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067573; c=relaxed/simple; bh=Q363yhiptxTVnDokhH8vdNq3s9Pk9lI+CS8MFqXpNo0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=U3eL7+NYECvkt1LZ3EZDsGaiTdcGyIuYXWE2s8Wjtq7cSUpAXKjAxjXuyta6iWjndqr70LCzH/P6Y05pAlmF7b0SfnBuAD4q4sRHPxC747hzpWZB+jxANWy+t90qUvbOmYGuwNxW23cXfMVoql2NfuPECAkHNguV2NvU8q1MC8I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Y8QWYVS4; arc=fail smtp.client-ip=40.107.220.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Y8QWYVS4" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=v3M2GiJ+5HnqwjHBsSj1ySGBLVfJlf6m9svftCOjuVzok9n+RfA665XN7LKH2C2QkATUW8eVrqE3erQ+VSqBE7RyzAd7b+9kJSaxTeRnhTPwd5r0EGjPOQ6MUvIvLXpGTdsQ/bUfuZoYRU5sPDqEcZO9ual73VygZNtubyhYKuOxxNKA8vfT6gw5rSNHihXhgvEEb3YjCyIIoB6kgZvZ+AH6kdlFbEykGb3fAGazhFKNAv4W5HYazsjVYk8b3HWha17hnM6HxZTsCu6zZA1w23Bks9pD6LGBXJxYh5aQEGECPuc95GJaEFgW/FRZclC11pAdnpLBI2BvZOaIh1z09A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Dwe8vbJBrdvHNicsKKFYYBYqThCiOEgC4bm7rduMFtg=; b=l46RNy1KWyPdgfrEMguBb7Cw8u2+XUzDSZ8GSUvE3eeTaX50br9FgPpxzaXdsYarIUaooi3AXoQ7hfY6j8LRdzYqAD0IRdmskd3FtMCYXKROXRik2Ec7Vvp07r/oMUzaSHQ/V+Lfb/Kn5+g1mMVLgba7rS2YnPGBqfUy66HYcFbkNwYN6LZbxll3RNF2PX4Pm0KZIfWsBitbZG1fFJxIQa7T/TVdepl+pr59+q1leIxUbNfH/ohQDNn1R2IV4Ej0/MyV399iFjg4kv1mikbskWiVI3pOln3Mx4fmBAe2ajN07bi9iFlWxw8sH/dWLRw41lE0Om3IdikaHcnO8k3OIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Dwe8vbJBrdvHNicsKKFYYBYqThCiOEgC4bm7rduMFtg=; b=Y8QWYVS4sjsQcE+1BVJbHA4/pflFsca7fkQy/cN7b9M7D9h338V7IC6R6mewtZvrF78zzsti2407sLZB0XeH3lrKxsOB+yvwCDHH+HjOLrVb0hJ4N8diAYQsvXKTYf5bCvCFd6NV/UKr7x2GrIKF4swe0b9wq40exNcKDvGH4N0= Received: from CH0PR08CA0016.namprd08.prod.outlook.com (2603:10b6:610:33::21) by BL1PR12MB5900.namprd12.prod.outlook.com (2603:10b6:208:398::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 15:39:24 +0000 Received: from CH2PEPF00000148.namprd02.prod.outlook.com (2603:10b6:610:33:cafe::10) by CH0PR08CA0016.outlook.office365.com (2603:10b6:610:33::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH2PEPF00000148.mail.protection.outlook.com (10.167.244.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 15:39:23 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:16 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 05/10] mm: Add throttling of mm scanning using scan_period Date: Sun, 1 Dec 2024 15:38:13 +0000 Message-ID: <20241201153818.2633616-6-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000148:EE_|BL1PR12MB5900:EE_ X-MS-Office365-Filtering-Correlation-Id: 5a0f1a83-c50a-4a53-3fc2-08dd121e54b4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dz0MkRLMoL2exnRuum9I7KSoT0Yy8/A4NJZW0jUkYTbBPP+KWXlOOPaUyvHE?= =?us-ascii?Q?wSWciQEQzXcCmtsoHNfaPNFDTrs3LrNOe5ame/fZLel/EhSWspJeZNx4EIV5?= =?us-ascii?Q?9Wlb6CuvbtgwsSOecym9GKSQkRUi3vufqEb8T3+E8oDoZlN0JwANIwaDfTgN?= =?us-ascii?Q?KzcM/0ZfUlPb83PVHImqJOsoczcjn2D5SpbRkIvG345bfDRkg4bxf1IHZHS5?= =?us-ascii?Q?NPqlbWoogqD8TvR1WCXdgsWEV8XHXwF3s+Enp/4PyXj81ON6Z3VCU9mTP5nA?= =?us-ascii?Q?H9zzg4e73lZ/UkHEo5Go0HI1iU4WjU3+rpfkPcag2Uv2QN+Qteaaa0qCxv3h?= =?us-ascii?Q?qn8tPgtohpGyWP2MnqcfVqW1vgEemheHDlGoeRpu+WpiU/hBjwGuiwGYYuGm?= =?us-ascii?Q?CWPk3lTlOUxqh+jElzGFmO6LDyOT4UMvQ104go6KIOw+/z+aWETIdm64f7BB?= =?us-ascii?Q?H5irrhUAHh39buDz9tnmowoM0IuyFcz34qkrbL0BiORzD62hBvRIGvtPDzkI?= =?us-ascii?Q?HnRfAAXMYpqqkPUyHeAyHshjN8d+tbC8mRm8G4pk3cHZNa7TQLoPzR3/dEmE?= =?us-ascii?Q?+W3GqKTHkF1Qi3tkeo5h9Y9us6Q+iEEd+tf5pcSIlU/lheM/C2YI+XRTTkki?= =?us-ascii?Q?StyNKZmsfyZ6vKVnnIC4jJ68qQn7VuFKjIMOUkHXXX+32JyGKWtE/TMmgi3u?= =?us-ascii?Q?uLtXxL+F0v+NNxA5ol26hFbn3JDe0bWuhcqYhv6yqrHZZuSSfKf4tD05r2CD?= =?us-ascii?Q?70RfS0fXq7gkuNNQRk8/H97PK2li0KJHFaJM2SQomlhZPLn9VYiaaIAvKlg8?= =?us-ascii?Q?4kokLb0nrZywC/QA9e2ObOGU2jHn3HupwTE+Cfgd+jyxiO+Ek5ZdAKsX9XEh?= =?us-ascii?Q?+DEH58E6DxsKUPrL+YrdOkhMI7p/N7xR6XKRjTsONVoxEQ+wrioBY8JbI59h?= =?us-ascii?Q?Y+jgsCMoY85d6m8Gz2Jgjypxp466GMMEIHReKbsIvSF7rZ8W9atM9mZeZl6d?= =?us-ascii?Q?uRE2AY8i8Z48RW4WxU0AhRVJQ1V58dIu2GX+F7GBzIH9kuVYWOgqIJFOsIED?= =?us-ascii?Q?Erpzwd5x+kNH0u1yahQO+nbrspgVqUtWvk2YYYgw4CzElFTBuqsw/eJ5ZxZM?= =?us-ascii?Q?zsOcyV03JwtMKGnqLg5u4r1gRV8Q7z/0Q0AvEVxZSTxZ5bUWBdv+m34NMmz/?= =?us-ascii?Q?Gv5b8pgzRU5vAhflFCyM3r/CTwGTi4OQdghrNvdufr1ieMp08LF6wSYVj5ki?= =?us-ascii?Q?kCS2XYIfDWz59luPszNdp1kENQvr46jOtY2ZWnIcmlhWlLcYRkw4ld8AZHpr?= =?us-ascii?Q?iQOqrT6EOO12lnxVICevqhTKmYp93edkMhJupY3S3zCq6d2sU2e4KbZ/RUrT?= =?us-ascii?Q?y5fCzlPNw2JyUIjJc20HevXO4WgXtFZeHvuvnviUwSGHRTxMKsvicWXf9L7A?= =?us-ascii?Q?VTE17WXKE4vHEcgXBI+EBacuBppNoaSowsds2XJmwFFVIL7cuGdrSQ=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:23.5115 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5a0f1a83-c50a-4a53-3fc2-08dd121e54b4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000148.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5900 Content-Type: text/plain; charset="utf-8" Before this patch, scanning of tasks' mm is done continuously and also at the same rate. Improve that by adding a throttling logic: 1) if there were useful pages found during last scan and current scan, decrease the scan_period (to increase scan rate) by TUNE_PERCENT (15%). 2) if there were no useful pages found in last scan, and there are candidate migration pages in the current scan decrease the scan_period aggressively by 2 power SCAN_CHANGE_SCALE (2^3 =3D 8 now). Vice versa is done for the reverse case. Scan period is clamped between MIN (400ms) and MAX (5sec). Signed-off-by: Raghavendra K T --- include/linux/mm_types.h | 4 ++ mm/kmmscand.c | 123 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 125 insertions(+), 2 deletions(-) Future improvements: 1. Consider the slope of useful pages found in last scan and current scan for finer tuning. 2. Use migration failure information. diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7361a8f3ab68..620b360b06fe 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -978,6 +978,10 @@ struct mm_struct { =20 /* numa_scan_seq prevents two threads remapping PTEs. */ int numa_scan_seq; +#endif +#ifdef CONFIG_KMMSCAND + /* Tracks number of pages with PTE A bit set after scanning. */ + atomic_long_t nr_accessed; #endif /* * An operation with batched TLB flushing is going on. Anything diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 3b4453b053f4..589aed604cd6 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -27,6 +28,16 @@ static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); =20 +/* + * Scan period for each mm. + * Min: 400ms default: 2sec Max: 5sec + */ +#define KMMSCAND_SCAN_PERIOD_MAX 5000U +#define KMMSCAND_SCAN_PERIOD_MIN 400U +#define KMMSCAND_SCAN_PERIOD 2000U + +static unsigned int kmmscand_mm_scan_period_ms __read_mostly =3D KMMSCAND_= SCAN_PERIOD; + /* How long to pause between two scan and migration cycle */ static unsigned int kmmscand_scan_sleep_ms __read_mostly =3D 16; =20 @@ -58,6 +69,11 @@ static struct kmem_cache *kmmscand_slot_cache __read_mos= tly; =20 struct kmmscand_mm_slot { struct mm_slot slot; + /* Unit: ms. Determines how aften mm scan should happen. */ + unsigned int scan_period; + unsigned long next_scan; + /* Tracks how many useful pages obtained for migration in the last scan */ + unsigned long scan_delta; long address; }; =20 @@ -85,6 +101,7 @@ struct kmmscand_migrate_info { struct folio *folio; unsigned long address; }; + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -324,6 +341,12 @@ static int hot_vma_idle_pte_entry(pte_t *pte, spin_lock(&kmmscand_migrate_lock); list_add_tail(&info->migrate_node, &migrate_list->migrate_head); spin_unlock(&kmmscand_migrate_lock); + + /* + * XXX: Should nr_accessed be per vma for finer control? + * XXX: We are increamenting atomic var under mmap_readlock + */ + atomic_long_inc(&mm->nr_accessed); } } end: @@ -446,11 +469,85 @@ static void kmmscand_migrate_folio(void) spin_unlock(&kmmscand_migrate_lock); } =20 +/* + * This is the normal change percentage when old and new delta remain same. + * i.e., either both positive or both zero. + */ +#define SCAN_PERIOD_TUNE_PERCENT 15 + +/* This is to change the scan_period aggressively when deltas are differen= t */ +#define SCAN_PERIOD_CHANGE_SCALE 3 +/* + * XXX: Hack to prevent unmigrated pages coming again and again while scan= ning. + * Actual fix needs to identify the type of unmigrated pages OR consider m= igration + * failures in next scan. + */ +#define KMMSCAND_IGNORE_SCAN_THR 100 + +/* + * X : Number of useful pages in the last scan. + * Y : Number of useful pages found in current scan. + * Tuning scan_period: + * Initial scan_period is 2s. + * case 1: (X =3D 0, Y =3D 0) + * Increase scan_period by SCAN_PERIOD_TUNE_PERCENT. + * case 2: (X =3D 0, Y > 0) + * Decrease scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 3: (X > 0, Y =3D 0 ) + * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 4: (X > 0, Y > 0) + * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + */ +static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm= _slot, unsigned long total) +{ + unsigned int scan_period; + unsigned long now; + unsigned long old_scan_delta; + + /* XXX: Hack to get rid of continuously failing/unmigrateable pages */ + if (total < KMMSCAND_IGNORE_SCAN_THR) + total =3D 0; + + scan_period =3D mm_slot->scan_period; + + old_scan_delta =3D mm_slot->scan_delta; + + /* + * case 1: old_scan_delta and new delta are similar, (slow) TUNE_PERCENT = used. + * case 2: old_scan_delta and new delta are different. (fast) CHANGE_SCAL= E used. + * TBD: + * 1. Further tune scan_period based on delta between last and current sc= an delta. + * 2. Optimize calculation + */ + if (!old_scan_delta && !total) { + scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && total) { + scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && !total) { + scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; + } else { + scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + } + + scan_period =3D clamp(scan_period, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCA= N_PERIOD_MAX); + + now =3D jiffies; + mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); + mm_slot->scan_period =3D scan_period; + mm_slot->scan_delta =3D total; +} + static unsigned long kmmscand_scan_mm_slot(void) { bool update_mmslot_info =3D false; =20 + unsigned int mm_slot_scan_period; + unsigned long now; + unsigned long mm_slot_next_scan; unsigned long address; + unsigned long folio_nr_access_s, folio_nr_access_e, total =3D 0; =20 struct mm_slot *slot; struct mm_struct *mm; @@ -473,6 +570,8 @@ static unsigned long kmmscand_scan_mm_slot(void) kmmscand_scan.mm_slot =3D mm_slot; } =20 + mm_slot_next_scan =3D mm_slot->next_scan; + mm_slot_scan_period =3D mm_slot->scan_period; mm =3D slot->mm; =20 spin_unlock(&kmmscand_mm_lock); @@ -483,6 +582,16 @@ static unsigned long kmmscand_scan_mm_slot(void) if (unlikely(kmmscand_test_exit(mm))) goto outerloop; =20 + now =3D jiffies; + /* + * Dont scan if : + * This is not a first scan AND + * Reaching here before designated next_scan time. + */ + if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) + goto outerloop; + + folio_nr_access_s =3D atomic_long_read(&mm->nr_accessed); =20 vma_iter_init(&vmi, mm, address); =20 @@ -492,6 +601,8 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 address =3D vma->vm_end; } + folio_nr_access_e =3D atomic_long_read(&mm->nr_accessed); + total =3D folio_nr_access_e - folio_nr_access_s; =20 if (!vma) address =3D 0; @@ -506,8 +617,12 @@ static unsigned long kmmscand_scan_mm_slot(void) spin_lock(&kmmscand_mm_lock); VM_BUG_ON(kmmscand_scan.mm_slot !=3D mm_slot); =20 - if (update_mmslot_info) + + if (update_mmslot_info) { mm_slot->address =3D address; + kmmscand_update_mmslot_info(mm_slot, total); + } + /* * Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. @@ -532,7 +647,7 @@ static unsigned long kmmscand_scan_mm_slot(void) } =20 spin_unlock(&kmmscand_mm_lock); - return 0; + return total; } =20 static void kmmscand_do_scan(void) @@ -595,6 +710,10 @@ void __kmmscand_enter(struct mm_struct *mm) return; =20 kmmscand_slot->address =3D 0; + kmmscand_slot->scan_period =3D kmmscand_mm_scan_period_ms; + kmmscand_slot->next_scan =3D 0; + kmmscand_slot->scan_delta =3D 0; + slot =3D &kmmscand_slot->slot; =20 spin_lock(&kmmscand_mm_lock); --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2073.outbound.protection.outlook.com [40.107.220.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A9E91D61AC for ; Sun, 1 Dec 2024 15:39:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.73 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067575; cv=fail; b=THdGVAzvC1iWeS22VKinmiZjsa2x4Yq7o+RMSc7PQ0t5zaRa3O3odGgGEKSjie3NoxZshEWZT7nNG+a3a36+0RkXvg4Zj+gfUOUGidfh8scTDz3Z6GELCde8+zH+PnHprdETJGxJ1FQprtaXpY6l1nQwsFqrpb728lSGXQhdz3o= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067575; c=relaxed/simple; bh=HhQi3rDUNjGoDJYn0Yk5tmdNkFAkWcYhXub5r6gsKAU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=q/tSaVrN+553t4TSIEC6Bd3lr3Fve5YfH9oj3DnDahKL7IsLfY+PIi0CxM6NjmWrceXSGAsqUCFRc4/cylfUbczylw6dJshNHL2WP+wjBR006ipTgN6mQPJOts5UaPAIgsydrOBOCXfcxmx9JA1Xf4sTPQc1NSXUGIK1ordPDkU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=FkuzqLkv; arc=fail smtp.client-ip=40.107.220.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="FkuzqLkv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qgAZtCd/K6oEdWFTayvuFfzXTd4GuFgE868qmOAu5uKBC6bSYvikBUfsvCE87fcXYR0YPM89pE9Ezsx3rnSNzw8uigmJH53xLZ/fYuEbFIZ/GybZk8iJrG8sPXEUWhcapTYYuDU6+KTmLl8ujmssGf18BCjF+V2mMfbfAICFCjARiEDkj2WxB56Wv4Ie0a0A01FEuOtWhHAYTRxuo9bwfZVHxgQFQBA/qKM1BKKxlmbgbPlZtc9bO1RWL2wZA50vlWDXKVytnamcTDmFwfB9oN1BoCIOW0lDLrZiZ7LWIa/l8FriYD5ow6mmxMuymaKEohG4p59ITVSGCJum7QyNeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bJv7WsTryDOyfwOqO3A2L59wJ0uSEjn6EgRQcX/ccfU=; b=L6Xj9vE+QOQd3XpfeG+9d/ujtNUbb58Bzqve2shGYYgHanLAFgcIPWnuh4I9RqgTN4Owbb49DD7vCSLO7iM2Zo2ko45XHl4Mvm4XjsKMhr5fEHREYXyssaplL3iVzKuforNQfZZLmdZQJL0yH+NZIBcmiBEFnTW4lxDvdHNscYBnvjQpP4cl3ub1G/SiMOGz0Xnj+kODSnLchvWxAJ6ktm0mYK54jY+SBIpIpmoZJpy7iH30JnXbp3uqMJrdfKLTe936jPVLCMvt4PXxQaz7jiOv5vTT9/ytoDTif6lP8BG1muE5wwR8+QJ4usKuKLx0rMDYsPK8JFIBFyuS2eR3Hg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bJv7WsTryDOyfwOqO3A2L59wJ0uSEjn6EgRQcX/ccfU=; b=FkuzqLkvKgZ+efC58wH+CVac4vcFL9HyTepB5JnTT+/5wDZ0VPtNkkpjAX7n2Xps6TIiX459+Sp/jUAZRQEdtWwHzFIk7Qbs4+Vjd2weZEpgEwu2yYkwnueWpcgGjVhThePOwkDTTB/6cw8B6+Mo5T0QZ6wPN096kJxhxhHkaUM= Received: from CH2PR11CA0027.namprd11.prod.outlook.com (2603:10b6:610:54::37) by LV8PR12MB9109.namprd12.prod.outlook.com (2603:10b6:408:18a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 15:39:29 +0000 Received: from CH2PEPF00000149.namprd02.prod.outlook.com (2603:10b6:610:54:cafe::99) by CH2PR11CA0027.outlook.office365.com (2603:10b6:610:54::37) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH2PEPF00000149.mail.protection.outlook.com (10.167.244.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 15:39:29 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:22 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 06/10] mm: Add throttling of mm scanning using scan_size Date: Sun, 1 Dec 2024 15:38:14 +0000 Message-ID: <20241201153818.2633616-7-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000149:EE_|LV8PR12MB9109:EE_ X-MS-Office365-Filtering-Correlation-Id: e76979ae-80a3-4302-3fb7-08dd121e5822 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?qfe4CLmuE/yJGcj7rRTQxFqxAuS7SWsVNnRs79Eg93zeovjtISjqO8t7H5sZ?= =?us-ascii?Q?fhTr/1VImRAmQWSQeaiguX02isW02L7gp3c+kP6ijdoWhkWD9Y+IJyKyC283?= =?us-ascii?Q?p68Otiirq+s+XG6ALt3HwGBX3YjvIYdIXqetDIkFW5wdrZx2fwtwM/huOYXY?= =?us-ascii?Q?2ELccibFoVez5DEmTbPq9pemcGp2MHmx8MsnplLDG1+K2oti4IVT7Z3O2PFA?= =?us-ascii?Q?SZ2HZQlDGjxLPvJsUydWzJmDXhjwIdyQ/Nss11ql/yxxuCKJZNyvnk/1gfsO?= =?us-ascii?Q?u6anmFXz99ZSOqNWKkMPgprAvVGVmeg+W6EwcduVYEymtTKEWzjAGGNa1gH8?= =?us-ascii?Q?vc6ljRpzWHZmY5MA0K7HU3tlC34WXEzxg+GbLl7LtkH5dVX3LbTy28xoKNu9?= =?us-ascii?Q?/NeKIB5cNYKTq+ext6EHTrYpJ8QBt/JJwTckh0hSou1lxWkNq2nBpT5Nc57N?= =?us-ascii?Q?U0OnjpqStpGgS8Y8sdFGFAGpptOZnYkxKUro2qHoVBqBvvcxd8cc3uUE10Pc?= =?us-ascii?Q?EoODyZ1Al+2/xPKrYvrrDbcB4ebHE4+nWeLyuhF6T6f+TdOdUG4R1DSVT16Z?= =?us-ascii?Q?T7CbOcqSPe4RvSflh/zy0PYfgHv1ySSZpFNFfEP6i5F27DO/B5ECYl4Hr896?= =?us-ascii?Q?mtPXGaPVvaWkhsa47T3idbideaudyQiiCF0QEE8UU3HJo8VD7ujLaWsC7gCp?= =?us-ascii?Q?BLlRFEershS6rOe4OBa3fm1e9eDyQr3Uo5rWdqQ9u+iZOHYXZzi/pPv/RZMB?= =?us-ascii?Q?KzbPOWBzzImH5Z+VD8disInrSIf09PN06xR+o9RGw2iuRpNbGDnp8x3PFtub?= =?us-ascii?Q?TPCVMrCtlwl5IL9y/qXIprfQ+nvgbA70tHLjB9SQM0ktoQm8my59Vk7I74cD?= =?us-ascii?Q?RS1MrLlhVUZC13cHJ06k3d9q8FDO9P1AsfsGCpokuqq3Cogbmp8ByYrFaNkp?= =?us-ascii?Q?85wzUPYT2Tgmhk9wI3RDt3j/WeHGgua4cGbZlAxnHpFubh/qkrY1HXtQwbkO?= =?us-ascii?Q?ZISKzLPaK3jHVs8+jHzYuVcjb1iWLpTLYWXP7GVMJ6kgWCxiP/LcYuI2BlTp?= =?us-ascii?Q?D+BPyLaDHbTlzBURVsWHE9hqe0jNNFlZWtgRHUDpeANkPBFlBa1bXeG7J7pr?= =?us-ascii?Q?ooeEmZLaTj+A0gcEAG552hFP+QjlS6huqZnBMlF5AGmBDbtzsvF8YC+ybP4T?= =?us-ascii?Q?B6x4hy8TBmfswAaoHD2HMgq/uj3ZQY+IYOyMmty12pMsYxw08xGV0g1eVjWA?= =?us-ascii?Q?X6etxKOl24IoWyadepRFVCeq0GeflW1f2OgsdtGWd3qOeP+eZ3rRHzK8s3eW?= =?us-ascii?Q?H/qL+00WI3vucEFf+aignMe9H/QRHEWno4hQCm6y4XZRKcls2EI/c586zTbC?= =?us-ascii?Q?KhWuBDod7DP+jEXbNClwTvAsBrwfcs1pvBojpePCFv/x/apLNncYi7eTiZJ2?= =?us-ascii?Q?E0AuX4xJpQbMQ2nND20fRJr5iMcqypqB6OuWQtcgK4eZ2YFeZZbNJg=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:29.2655 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e76979ae-80a3-4302-3fb7-08dd121e5822 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000149.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9109 Content-Type: text/plain; charset="utf-8" Before this patch, scanning is done on entire virtual address space of all the tasks. Now the scan size is shrunk or expanded based on the useful pages found in the last scan. This helps to quickly get out of unnecessary scanning thus burning lesser CPU. Drawback: If a useful chunk is at the other end of the VMA space, it will delay scanning and migration. Shrink/expand algorithm for scan_size: X : Number of useful pages in the last scan. Y : Number of useful pages found in current scan. Initial scan_size is 4GB case 1: (X =3D 0, Y =3D 0) Decrease scan_size by 2 case 2: (X =3D 0, Y > 0) Aggressively change to MAX (16GB) case 3: (X > 0, Y =3D 0 ) No change case 4: (X > 0, Y > 0) Increase scan_size by 2 Scan size is clamped between MIN (512MB) and MAX (16GB)). TBD: Tuning this based on real workload Experiment: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Abench microbenchmark, - Allocates 8GB/32GB of memory on CXL node - 64 threads created, and each thread randomly accesses pages in 4K granularity. - 512 iterations. SUT: 512 CPU, 2 node 256GB, AMD EPYC. 3 runs, command: abench -m 2 -d 1 -i 512 -s Calculates how much time is taken to complete the task, lower is better. Expectation is CXL node memory is expected to be migrated as fast as possible. Base case: 6.11-rc6 w/ numab mode =3D 2 (hot page promotion is enabled). patched case: 6.11-rc6 w/ numab mode =3D 0 (numa balancing is disabled). we expect daemon to do page promotion. Result: =3D=3D=3D=3D=3D=3D=3D=3D base patched time in sec (%stdev) time in sec (%stdev) %gain 8GB 133.66 ( 0.38 ) 113.77 ( 1.83 ) 14.88 32GB 584.77 ( 0.19 ) 542.79 ( 0.11 ) 7.17 Overhead: The below time is calculated using patch 10. Actual overhead for patched case may be even lesser. (scan + migration) time in sec Total memory base kernel patched kernel %gain 8GB 65.743 13.93 78.8114324 32GB 153.95 132.12 14.17992855 Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 3 deletions(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 589aed604cd6..2efef53f9402 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -28,6 +28,16 @@ static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); =20 +/* + * Total VMA size to cover during scan. + * Min: 512MB default: 4GB max: 16GB + */ +#define KMMSCAND_SCAN_SIZE_MIN (512 * 1024 * 1024UL) +#define KMMSCAND_SCAN_SIZE_MAX (16 * 1024 * 1024 * 1024UL) +#define KMMSCAND_SCAN_SIZE (4 * 1024 * 1024 * 1024UL) + +static unsigned long kmmscand_scan_size __read_mostly =3D KMMSCAND_SCAN_SI= ZE; + /* * Scan period for each mm. * Min: 400ms default: 2sec Max: 5sec @@ -74,6 +84,8 @@ struct kmmscand_mm_slot { unsigned long next_scan; /* Tracks how many useful pages obtained for migration in the last scan */ unsigned long scan_delta; + /* Determines how much VMA address space to be covered in the scanning */ + unsigned long scan_size; long address; }; =20 @@ -484,6 +496,7 @@ static void kmmscand_migrate_folio(void) */ #define KMMSCAND_IGNORE_SCAN_THR 100 =20 +#define SCAN_SIZE_CHANGE_SCALE 1 /* * X : Number of useful pages in the last scan. * Y : Number of useful pages found in current scan. @@ -497,11 +510,22 @@ static void kmmscand_migrate_folio(void) * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). * case 4: (X > 0, Y > 0) * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + * Tuning scan_size: + * Initial scan_size is 4GB + * case 1: (X =3D 0, Y =3D 0) + * Decrease scan_size by (1 << SCAN_SIZE_CHANGE_SCALE). + * case 2: (X =3D 0, Y > 0) + * scan_size =3D KMMSCAND_SCAN_SIZE_MAX + * case 3: (X > 0, Y =3D 0 ) + * No change + * case 4: (X > 0, Y > 0) + * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SCALE). */ static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm= _slot, unsigned long total) { unsigned int scan_period; unsigned long now; + unsigned long scan_size; unsigned long old_scan_delta; =20 /* XXX: Hack to get rid of continuously failing/unmigrateable pages */ @@ -509,6 +533,7 @@ static inline void kmmscand_update_mmslot_info(struct k= mmscand_mm_slot *mm_slot, total =3D 0; =20 scan_period =3D mm_slot->scan_period; + scan_size =3D mm_slot->scan_size; =20 old_scan_delta =3D mm_slot->scan_delta; =20 @@ -522,30 +547,38 @@ static inline void kmmscand_update_mmslot_info(struct= kmmscand_mm_slot *mm_slot, if (!old_scan_delta && !total) { scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size >> SCAN_SIZE_CHANGE_SCALE; } else if (old_scan_delta && total) { scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size << SCAN_SIZE_CHANGE_SCALE; } else if (old_scan_delta && !total) { scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; } else { scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + scan_size =3D KMMSCAND_SCAN_SIZE_MAX; } =20 scan_period =3D clamp(scan_period, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCA= N_PERIOD_MAX); + scan_size =3D clamp(scan_size, KMMSCAND_SCAN_SIZE_MIN, KMMSCAND_SCAN_SIZE= _MAX); =20 now =3D jiffies; mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); mm_slot->scan_period =3D scan_period; + mm_slot->scan_size =3D scan_size; mm_slot->scan_delta =3D total; } =20 static unsigned long kmmscand_scan_mm_slot(void) { + bool next_mm =3D false; bool update_mmslot_info =3D false; =20 unsigned int mm_slot_scan_period; unsigned long now; unsigned long mm_slot_next_scan; + unsigned long mm_slot_scan_size; + unsigned long scanned_size =3D 0; unsigned long address; unsigned long folio_nr_access_s, folio_nr_access_e, total =3D 0; =20 @@ -572,6 +605,7 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 mm_slot_next_scan =3D mm_slot->next_scan; mm_slot_scan_period =3D mm_slot->scan_period; + mm_slot_scan_size =3D mm_slot->scan_size; mm =3D slot->mm; =20 spin_unlock(&kmmscand_mm_lock); @@ -579,8 +613,10 @@ static unsigned long kmmscand_scan_mm_slot(void) if (unlikely(!mmap_read_trylock(mm))) goto outerloop_mmap_lock; =20 - if (unlikely(kmmscand_test_exit(mm))) + if (unlikely(kmmscand_test_exit(mm))) { + next_mm =3D true; goto outerloop; + } =20 now =3D jiffies; /* @@ -598,8 +634,20 @@ static unsigned long kmmscand_scan_mm_slot(void) for_each_vma(vmi, vma) { /* Count the scanned pages here to decide exit */ kmmscand_walk_page_vma(vma); - + scanned_size +=3D vma->vm_end - vma->vm_start; address =3D vma->vm_end; + + if (scanned_size >=3D mm_slot_scan_size) { + folio_nr_access_e =3D atomic_long_read(&mm->nr_accessed); + total =3D folio_nr_access_e - folio_nr_access_s; + /* If we had got accessed pages, ignore the current scan_size threshold= */ + if (total > KMMSCAND_IGNORE_SCAN_THR) { + mm_slot_scan_size =3D KMMSCAND_SCAN_SIZE_MAX; + continue; + } + next_mm =3D true; + break; + } } folio_nr_access_e =3D atomic_long_read(&mm->nr_accessed); total =3D folio_nr_access_e - folio_nr_access_s; @@ -627,7 +675,7 @@ static unsigned long kmmscand_scan_mm_slot(void) * Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. */ - if (unlikely(kmmscand_test_exit(mm)) || !vma) { + if (unlikely(kmmscand_test_exit(mm)) || !vma || next_mm) { /* * Make sure that if mm_users is reaching zero while * kmmscand runs here, kmmscand_exit will find @@ -711,6 +759,7 @@ void __kmmscand_enter(struct mm_struct *mm) =20 kmmscand_slot->address =3D 0; kmmscand_slot->scan_period =3D kmmscand_mm_scan_period_ms; + kmmscand_slot->scan_size =3D kmmscand_scan_size; kmmscand_slot->next_scan =3D 0; kmmscand_slot->scan_delta =3D 0; =20 --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2079.outbound.protection.outlook.com [40.107.92.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86489AD5A for ; Sun, 1 Dec 2024 15:39:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.79 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067585; cv=fail; b=Eqr3fwdfEbL38dAoIDvi6ju/vikjcRpLIyh8hyiwleca6FrJwBJcFGOsAae5NyvXMTtAyIJHLD3c74WAo3prOPfz3Nr+ihK4C0GE0vlQlyPTDXr1Eom6vxxYpzrZs3KkQQIXo2wEn+RI47CCNGvERmsMDvT6vcAR9GBS9cvHNXI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067585; c=relaxed/simple; bh=zTxbnYSIPRZXkCY/xG3gA0h13UEazWNchmoHqFQj33A=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OlyD0L02JYECydi0/4co2YXxBNP9JRJBzAlxts7BQlyJFf9qTZcmJ0jatNFcse62Aj2eYkWYs+0Yjr50Gh9EUnB74JxerYTqqYpN32u9P0wOX5CJTRIlWh8CEk36zkulm94SD3oQbupfRoTJ/8ubc5ZNTtncBDjipbGS06Nsdlw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=FF0fC2z8; arc=fail smtp.client-ip=40.107.92.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="FF0fC2z8" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yoD2Y+XEZmHfUC6Of8IQQ40/WNMF3+fWXmqquEfCmQYi5rhPBJrzmy34f38LONBtBL9REkosMMnPjPtVuMhL+LVsglxz+tUtVWxCn66JcHwWRbjlPU+LjHeQtiXUXFwBrjNOHVWoYnXef/edi2ZXf9HryfT8jPyK47MO+1DMG7Evg6JslU/TdexeoxQCeYEM31iKrWPZu3GH5DQ6PPiaL2moQZn9s/E6rijN9v+Di9yXMV8OooulyoGWk86Nc99Ux/ups8dNATiNz8j4OYKJY7SZa1kK8Lane1u2tXsZ0weVf9Xo5IKx9S7Kp/yI7waKylKRR/XYAVWQd1xBmm436Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=X+xaFjNavCQ6nXnpTcSddvD7CT+2Vhgf2Amvg+doJEk=; b=AXojOOOMc/mVv0JOIPf+Ft3rhk48GkDg1TcTdw2c47TRdCVcxDuR4djf2LQKXToenvl47VZ1mKXKU/sD4UR0KXu8mqzp5ItuO1FX1JT1L6O7v8yJIQv08u30D/P7guU7n6BiOQr3S16/EDYVrs2YMvZGxD277La6g5iWb08etB1W2eI0lkET0voiEW7Sb66iItjZwWFEbhoO9wKtsY5uyhGmEJKLhguJp4c3I7VOXC6Bs02PCUQQIZTVEonL8D/NP5sIiUkygPG6KH9xF1MKJ2wTQAlOqq3DXgyL3Cx/cc74Y2odaJC4Bsynob/qIMRvcGWfVbbVKryCBHP4wv+fyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X+xaFjNavCQ6nXnpTcSddvD7CT+2Vhgf2Amvg+doJEk=; b=FF0fC2z8eCpx7yojIIXtBaCkpMKP9yXM2RZm1wSH3IAWF4k2z2LtEwSs3mj4ZV2Hc8nVYsDJdM3Vqkr/bWYAJdBU1IZsQnXTl0UUQp9icZLJoocVHNkhNAZP/+FvoGj8o+wFuC48pjfCZRPYteaeX3Z1rJa0YNt3MdAseMLplrM= Received: from CH2PR07CA0065.namprd07.prod.outlook.com (2603:10b6:610:5b::39) by SA3PR12MB9108.namprd12.prod.outlook.com (2603:10b6:806:37d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 15:39:36 +0000 Received: from CH2PEPF00000146.namprd02.prod.outlook.com (2603:10b6:610:5b:cafe::d0) by CH2PR07CA0065.outlook.office365.com (2603:10b6:610:5b::39) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.15 via Frontend Transport; Sun, 1 Dec 2024 15:39:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH2PEPF00000146.mail.protection.outlook.com (10.167.244.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 15:39:35 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:28 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 07/10] sysfs: Add sysfs support to tune scanning Date: Sun, 1 Dec 2024 15:38:15 +0000 Message-ID: <20241201153818.2633616-8-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000146:EE_|SA3PR12MB9108:EE_ X-MS-Office365-Filtering-Correlation-Id: 12e304cc-4097-44b1-a8f8-08dd121e5be9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?wlHk46/nKVXwem+HsIADIuIkA93XzVkjCjnEyGR8bIfKg8h8iWPwq1Z3pHs8?= =?us-ascii?Q?qzkCFa119tVA4tlqWCFq6HbCu5j9vn2VHIKT9+8+0GPD0gbCrxgMTGieRm8g?= =?us-ascii?Q?ZkWWrqbVJhxh6uEjtiU40/GnWwOuJWv25u0b9NJaPV4+qQY4KhXb1Bmd6GaP?= =?us-ascii?Q?QM2EPrkJ5xVvHl4vlIdYauVYp1c6BgqEW6fguc/AtV7gv6tcX0ngoIjL/BxC?= =?us-ascii?Q?6ZDYaVZ5903JB3reLHmw+49SCPlnbwPRKBojHbthofumaoTb2c32EUCohKuW?= =?us-ascii?Q?oXsQz8F+RvANP7uEbXPiDT8kPpUzkpHPMGWUqi0N39cEiKUTD8TfMfGf438b?= =?us-ascii?Q?zvkmq9tNH7mYPa9T70bSgpdvvhtWy7kdPEzW9HwBfwIsC0gnfASZ/mdOVnHQ?= =?us-ascii?Q?uHehvQsIik8mVmP/jpEoCYYE//VuGUpm0nuhUjfXSmxQDtevRareViDabiRa?= =?us-ascii?Q?UagF7gSxXnPoxeCUCo+4dwViU6JPNeAr4slxh1WMGItZoDngubrY65SxQDWq?= =?us-ascii?Q?uWIiqkjRDNUi2ZW6pYlgn0y6R6gNp1HRbFjOzjXYqOTQfTHgOkzsguJ0m9ol?= =?us-ascii?Q?bzwVK+dOsXTLSbEEP19F+WcxLL8U30/CRh4Nf+1uuMw706z/nK5GIOV98TiS?= =?us-ascii?Q?4tr9JUBV3fFGDhQ1TEsbUycXVPgEIpltgqUH6oxzWHIAC5inA92UUjOCfBrD?= =?us-ascii?Q?9aevBb14aTuZJqZswg/V57GXiqAiVwGhWpKIq1kKOQBRmv0XYPwGw00V799N?= =?us-ascii?Q?NOVtrLo1kphicL3Z4PxZMA0xzawdbqsuSkkbrxoUPef5E4VpeHL1uv0Iuw6V?= =?us-ascii?Q?H/u4R650ko5ScRCklskZ367YVTqvWY8cwX+gWtoEARluP7eznRh2G5Z+8+7T?= =?us-ascii?Q?mrDE75Yo2CXqXgaMAAU6o2Qhmo1q58dc21RrPhrM3Syq7SD1tzihQLtrLHuP?= =?us-ascii?Q?khLJ05WW+qLdCAr8gLqfY2sxd+USfOPmoqjWcYRg+TwtKRgO2DVCEafCLZBT?= =?us-ascii?Q?E7LDPIhdp48dm1pcHYJYkrZuvTPiEftqRwIa3FlYA5/3fPr6JeMIIEiaj3Bh?= =?us-ascii?Q?ZGQUnPmuiHwQ/GHM/2w3RND/Y+GcVeq/puUR1xjsJTypQIuajbXcs8U/blNe?= =?us-ascii?Q?PjsSxJxSmilEyLu9jZZo0LNqnVitakbtEi4N2Y2UR0otGc36Y1Ux/r/nwsTm?= =?us-ascii?Q?b8tsYSDAyHuEyXNrgDTCwHO6MZG98ugR3B4y3jeTLZCnt2OQKslXbow9YQHT?= =?us-ascii?Q?6Ba+zJDYVZUJorIL799XJXOkTLeU6uEYgvJY1UPXWH8MAf2VaAB/nYSlUPU7?= =?us-ascii?Q?oy6rEw62bvHiR/ciKGoAE+E08j5hbHtH9QEakbwXVEojr/zlQdpelxRSoRLJ?= =?us-ascii?Q?sNJk3e5NF0aSmIs/OreMVx/tEEj0hoOF6/1YsoMD45swEHW2NA2PmS5tW/8A?= =?us-ascii?Q?By01QVGGMUgug/xK+DNWPDVp+R+LaRWT66w3o1CyeaRbIsub0mePzg=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:35.6055 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 12e304cc-4097-44b1-a8f8-08dd121e5be9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000146.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB9108 Content-Type: text/plain; charset="utf-8" Support below tunables: scan_enable: turn on or turn off mm_struct scanning scan_period: initial scan_period (default: 2sec) scan_sleep_ms: sleep time between two successive round of scanning and migration. mms_to_scan: total mm_struct to scan before taking a pause. target_node: default regular node to which migration of accessed pages is done Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 205 insertions(+) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 2efef53f9402..344a45bd2d3e 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -20,6 +20,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -114,6 +115,170 @@ struct kmmscand_migrate_info { unsigned long address; }; =20 +#ifdef CONFIG_SYSFS +static ssize_t scan_sleep_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_scan_sleep_ms); +} + +static ssize_t scan_sleep_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + kmmscand_scan_sleep_ms =3D msecs; + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} +static struct kobj_attribute scan_sleep_ms_attr =3D + __ATTR_RW(scan_sleep_ms); + +static ssize_t mm_scan_period_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_mm_scan_period_ms); +} + +/* If a value less than MIN or greater than MAX asked for store value is c= lamped */ +static ssize_t mm_scan_period_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs, stored_msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + stored_msecs =3D clamp(msecs, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCAN_PER= IOD_MAX); + + kmmscand_mm_scan_period_ms =3D stored_msecs; + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute mm_scan_period_ms_attr =3D + __ATTR_RW(mm_scan_period_ms); + +static ssize_t mms_to_scan_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%lu\n", kmmscand_mms_to_scan); +} + +static ssize_t mms_to_scan_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned long val; + int err; + + err =3D kstrtoul(buf, 10, &val); + if (err) + return -EINVAL; + + kmmscand_mms_to_scan =3D val; + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute mms_to_scan_attr =3D + __ATTR_RW(mms_to_scan); + +static ssize_t scan_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_scan_enabled ? 1 : 0); +} + +static ssize_t scan_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int val; + int err; + + err =3D kstrtouint(buf, 10, &val); + if (err || val > 1) + return -EINVAL; + + if (val) { + kmmscand_scan_enabled =3D true; + need_wakeup =3D true; + } else + kmmscand_scan_enabled =3D false; + + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute scan_enabled_attr =3D + __ATTR_RW(scan_enabled); + +static ssize_t target_node_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_target_node); +} + +static ssize_t target_node_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int err, node; + + err =3D kstrtoint(buf, 10, &node); + if (err) + return -EINVAL; + + kmmscand_sleep_expire =3D 0; + if (!node_is_toptier(node)) + return -EINVAL; + + kmmscand_target_node =3D node; + wake_up_interruptible(&kmmscand_wait); + + return count; +} +static struct kobj_attribute target_node_attr =3D + __ATTR_RW(target_node); + +static struct attribute *kmmscand_attr[] =3D { + &scan_sleep_ms_attr.attr, + &mm_scan_period_ms_attr.attr, + &mms_to_scan_attr.attr, + &scan_enabled_attr.attr, + &target_node_attr.attr, + NULL, +}; + +struct attribute_group kmmscand_attr_group =3D { + .attrs =3D kmmscand_attr, + .name =3D "kmmscand", +}; +#endif + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -738,9 +903,43 @@ static int kmmscand(void *none) return 0; } =20 +#ifdef CONFIG_SYSFS +extern struct kobject *mm_kobj; +static int __init kmmscand_init_sysfs(struct kobject **kobj) +{ + int err; + + err =3D sysfs_create_group(*kobj, &kmmscand_attr_group); + if (err) { + pr_err("failed to register kmmscand group\n"); + goto err_kmmscand_attr; + } + + return 0; + +err_kmmscand_attr: + sysfs_remove_group(*kobj, &kmmscand_attr_group); + return err; +} + +static void __init kmmscand_exit_sysfs(struct kobject *kobj) +{ + sysfs_remove_group(kobj, &kmmscand_attr_group); +} +#else +static inline int __init kmmscand_init_sysfs(struct kobject **kobj) +{ + return 0; +} +static inline void __init kmmscand_exit_sysfs(struct kobject *kobj) +{ +} +#endif + static inline void kmmscand_destroy(void) { kmem_cache_destroy(kmmscand_slot_cache); + kmmscand_exit_sysfs(mm_kobj); } =20 void __kmmscand_enter(struct mm_struct *mm) @@ -857,6 +1056,11 @@ static int __init kmmscand_init(void) return -ENOMEM; } =20 + err =3D kmmscand_init_sysfs(&mm_kobj); + + if (err) + goto err_init_sysfs; + err =3D start_kmmscand(); if (err) goto err_kmmscand; @@ -865,6 +1069,7 @@ static int __init kmmscand_init(void) =20 err_kmmscand: stop_kmmscand(); +err_init_sysfs: kmmscand_destroy(); =20 return err; --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2046.outbound.protection.outlook.com [40.107.92.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BB111D61B1 for ; Sun, 1 Dec 2024 15:39:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067590; cv=fail; b=YKXpVy6+RMGXYZbPcFqdUJINEqDRu3oXAn1kaFCxkKK8DOuuKrQAeWWZVdT8ykhnuVwHNQkCAgr/Z8+pDSKV+d1Lhx1PA1BomWIeY1w93h36f0ozr5yraOAD9fo4/Y+EGdhbBsCiLx/kvMN0LCUH2drwcJeg85RNAV05g+J+AX8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067590; c=relaxed/simple; bh=ljafux1/tL4+SrxEjKXNmAkiYg4YAj9JGGR3c0WI3/k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=M9YLpwBOgGBFcqeUrkywpEz5mDDQLFFtxcvFfSB6UjvKn0iVDmgl4TSQl1WipzSCpJCx6wTXWPk2BoXIANGM7Eka5eB8hXIDlfM/Rf5RuJjAkkrbZuGTb9a33Ag9SO6t5ebDo3NUM/YrWgd96iJQHSTwXThkuWB3Oo3gpCgQlxk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=cIAjPUyI; arc=fail smtp.client-ip=40.107.92.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="cIAjPUyI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ek6Erq1UWSgPeUAm1lqUFRpfM/UYyqIxOSyJtei0J35LB3c6mY3lq+Ke9mW7q+YafRkABqfaXZ6VdeQTCmFobrYhvH0HBeh3CSeYifnfO3kbZ3PJEsYDl3qHBiVZ3zhWfb3/wSpFK0exTOzgASt+O3yZvBCouBB1AZpka7fnsd6xORV1qhvWupqw+bc2gmyz+UJP9pSJdu8JmmY2QtxQSSRs+9H7q9Q3zDovuzg5PEpbSQ4HbzoHu0mxL0JiwlHzDuqz0qhhYRmjJTzPI/iuSgrmiMOyxDD2ubPqrXkRLtgSf0E6FAcYpN7JQFk32ICbdGw0HxNVFPBDL0Qzt3j3ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4Tyigini5Kwm8cIJJd8zOop087IwBFUJq4O47InjsQw=; b=PtgHY4V6c6pophiI3RAiKogWfPyzu2BzVElN1874p1HeQnRgfgXu7tQ70mZJjMb8clcJ11RnxMe2tEChtWvtSTVVRn7pEp4KjWonKC/pj8sQeYXmsIqQ1UO7XtP6Ax5TvXjiXFLc4KIOJI61yM4WJ+GbxWJB1upq1a0wBXExagxj0LfyQDnTlDJlxAFKuJ8JzPumuHD+DcU3JlpB3f6ExeihPSZTdvbWIxIlDoWEhupF8b6lsf6yrfzIeeYm/NOtv2xDkCVmjbEl7MlfO2IicGxfXfnZzognJCUSQ2Rgsmma8MoyVOntVrm0JCuKh1ZnWwdDtIj3qRVQLd4wLpPVrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4Tyigini5Kwm8cIJJd8zOop087IwBFUJq4O47InjsQw=; b=cIAjPUyIzJtlsCd0hyNdSC3621OoZyPmNyZhJt0uS0CFjOoWThjcXe7oNnqUgWJtXNbAxwnAIAQ/Qeha9nnNu8fPXOOfHEuqC1E7oXqwyrSHwxoWiv6BVYv4VfdGm4KrGNYZdtJkQ36E6N+1rwKaFK9RCUJOH3NQvvcIs37JcAQ= Received: from CH5PR03CA0015.namprd03.prod.outlook.com (2603:10b6:610:1f1::13) by CH2PR12MB4133.namprd12.prod.outlook.com (2603:10b6:610:7a::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.14; Sun, 1 Dec 2024 15:39:42 +0000 Received: from CH2PEPF0000014A.namprd02.prod.outlook.com (2603:10b6:610:1f1:cafe::ef) by CH5PR03CA0015.outlook.office365.com (2603:10b6:610:1f1::13) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH2PEPF0000014A.mail.protection.outlook.com (10.167.244.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 15:39:42 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:35 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 08/10] vmstat: Add vmstat counters Date: Sun, 1 Dec 2024 15:38:16 +0000 Message-ID: <20241201153818.2633616-9-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000014A:EE_|CH2PR12MB4133:EE_ X-MS-Office365-Filtering-Correlation-Id: 187baa7d-6a5e-4c9e-8595-08dd121e5fce X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|7416014|376014|1800799024|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?YqPcNQHjjTD5nXdzmTzRylvMCFUecnE40VUaQ5KzGDJF/edjGxwtWuJ3Y+M/?= =?us-ascii?Q?2q/H4W8em0GunbXy+d0wPLwxFIwCffuR8M7cROjVmi2OQrf4yMDOZY8ypyNp?= =?us-ascii?Q?+KrP/TSKrywzjFzAt9gKRmV0Pp3cCNXCWvTLuhVLmfNadjS7E5ZwiabQlrez?= =?us-ascii?Q?IkIYbMnf14014Xnlo5pKxs3PSIXrU/tADaHfrBD3PW6Qr7e1nv5gNXknWISD?= =?us-ascii?Q?/KDJrxm8hklfe+xizpPPCRkmxar5cIZWEKk6+8fZaBtxLzFb8lTYfCtDM+0N?= =?us-ascii?Q?HVYINh3TL7G/YrNyca4q1RKfu43HhtEdhb90zBEHQR8K3Es4ZQ7mPCHDbxOH?= =?us-ascii?Q?ndUpoYE0kkhByVbkzpWAOSItAjTPjodPNqUlfLsovabsnVZIZeFx70tMIhG6?= =?us-ascii?Q?4JJcC8yxdgkgLuvi1yIHkvBQR+7J/e/3NEthCmPH1Gj3B2MFJ1swN3TZPIfr?= =?us-ascii?Q?W4fQGqm3csefIlNmSDAelEB6wJpNKHhOPjwkBDAbnT4CZivJV0N9xVH6MkIG?= =?us-ascii?Q?RldCvzWpIIn+wqfWjxFNMU+c+ZuAsMQj2Qkc5+2IlzeToahPA7T5wJ/arWTd?= =?us-ascii?Q?senw8xr6O0rKEtRHRWagrvdBa+zYr8+k4GTUzWj1GiJ0MzcIhgH/Jo7jYcQ3?= =?us-ascii?Q?/Zj1CPH2IcoGMhLhhqKFx6mCR0Ao8dz+hOgDcim9Cilq/YgmNYunSzgjmIno?= =?us-ascii?Q?lIs5l/UELYUUab90/q/++NlujObydY+oFjfnFMce3gbiAIjdqJYsLS12WfY4?= =?us-ascii?Q?JEFdyvHuve62VuCbWlyMmKACj2Y29QRIah4Ql3iLeYMNXcJ3YYK0uVKXBeDB?= =?us-ascii?Q?VlITfAMykSyQ+kAS9CPzQOnaDmU9gS+5kuI0aivkqjdx0yZq/iz26q6v4jeo?= =?us-ascii?Q?jvDDAziEtqbMCWcYJhQmwavSOFmWGA2utwWS4Wh9RhYv3k8juQvHtMU9Tbpv?= =?us-ascii?Q?tQK0MYsmTpuEGhwcsGs7ZKr9ifzoa58WCN8VD+WLuJy1WdeuKRdrQHFK0Dfo?= =?us-ascii?Q?pEcmecth5tkN3cIrLwjd509w09WT3Pw5cfQIKPYavNymX6YEPYm94kaQqurf?= =?us-ascii?Q?bvFl3njcvJvfs5ON5UerTnehzQw61UwGfk5L61NWztqz4TAygXNmWPfasT2E?= =?us-ascii?Q?hpZeJ01VjkussaaTk+iZqRCLYLQDVlFjFmZqAkNeVT54pXJGK8qd9vgbTxbA?= =?us-ascii?Q?1UvB54ftJGKYtu1LATHtc2IIcW5Sm3hB4K9iUfDa1DNLWa+qsraG+TmUxXyN?= =?us-ascii?Q?9XAr0vo5aF/FuQI71EigKUIoH4asuQ1GBfX+USzYRrFlsTqEq52J0Nuy5hMg?= =?us-ascii?Q?S6nUkE4+rzXIrHn07X4mJwLZJZg174SsomKUrRQPdobpkshy244FJhvl14Ea?= =?us-ascii?Q?hU+rJWHtbNkKH49jv6UryVicezLXfAsJ8WOKMJeeJJLsDAWlFKSAq4iKklOQ?= =?us-ascii?Q?mAwkwHZTk/wOJxIh7qM4k8vMzrbs1O4VWOLm3076ZhDUgcfqG4ACEw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(7416014)(376014)(1800799024)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:42.1229 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 187baa7d-6a5e-4c9e-8595-08dd121e5fce X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000014A.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4133 Content-Type: text/plain; charset="utf-8" Add vmstat counter to track scanning, migration and type of pages.=20 Signed-off-by: Raghavendra K T --- include/linux/mm.h | 11 ++++++++ include/linux/vm_event_item.h | 10 +++++++ mm/kmmscand.c | 50 ++++++++++++++++++++++++++++++++++- mm/vmstat.c | 10 +++++++ 4 files changed, 80 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c39c4945946c..306452c11d31 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -681,6 +681,17 @@ struct vm_operations_struct { unsigned long addr); }; =20 +#ifdef CONFIG_KMMSCAND +void count_kmmscand_mm_scans(void); +void count_kmmscand_vma_scans(void); +void count_kmmscand_migadded(void); +void count_kmmscand_migrated(void); +void count_kmmscand_migrate_failed(void); +void count_kmmscand_slowtier(void); +void count_kmmscand_toptier(void); +void count_kmmscand_idlepage(void); +#endif + #ifdef CONFIG_NUMA_BALANCING static inline void vma_numab_state_init(struct vm_area_struct *vma) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f70d0958095c..b2ccd4f665aa 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -65,6 +65,16 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, #endif +#ifdef CONFIG_KMMSCAND + KMMSCAND_MM_SCANS, + KMMSCAND_VMA_SCANS, + KMMSCAND_MIGADDED, + KMMSCAND_MIGRATED, + KMMSCAND_MIGRATE_FAILED, + KMMSCAND_SLOWTIER, + KMMSCAND_TOPTIER, + KMMSCAND_IDLEPAGE, +#endif #ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, THP_MIGRATION_SUCCESS, diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 344a45bd2d3e..682c0523c0b4 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -279,6 +279,39 @@ struct attribute_group kmmscand_attr_group =3D { }; #endif =20 +void count_kmmscand_mm_scans(void) +{ + count_vm_numa_event(KMMSCAND_MM_SCANS); +} +void count_kmmscand_vma_scans(void) +{ + count_vm_numa_event(KMMSCAND_VMA_SCANS); +} +void count_kmmscand_migadded(void) +{ + count_vm_numa_event(KMMSCAND_MIGADDED); +} +void count_kmmscand_migrated(void) +{ + count_vm_numa_event(KMMSCAND_MIGRATED); +} +void count_kmmscand_migrate_failed(void) +{ + count_vm_numa_event(KMMSCAND_MIGRATE_FAILED); +} +void count_kmmscand_slowtier(void) +{ + count_vm_numa_event(KMMSCAND_SLOWTIER); +} +void count_kmmscand_toptier(void) +{ + count_vm_numa_event(KMMSCAND_TOPTIER); +} +void count_kmmscand_idlepage(void) +{ + count_vm_numa_event(KMMSCAND_IDLEPAGE); +} + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -500,6 +533,9 @@ static int hot_vma_idle_pte_entry(pte_t *pte, =20 srcnid =3D folio_nid(folio); =20 + if (node_is_toptier(srcnid)) + count_kmmscand_toptier(); + if (!folio_test_idle(folio) || folio_test_young(folio) || mmu_notifier_test_young(mm, addr) || folio_test_referenced(folio) || pte_young(pteval)) { @@ -511,6 +547,7 @@ static int hot_vma_idle_pte_entry(pte_t *pte, info =3D kzalloc(sizeof(struct kmmscand_migrate_info), GFP_KERNEL); if (info && migrate_list) { =20 + count_kmmscand_slowtier(); info->mm =3D mm; info->vma =3D vma; info->folio =3D folio; @@ -524,8 +561,10 @@ static int hot_vma_idle_pte_entry(pte_t *pte, * XXX: We are increamenting atomic var under mmap_readlock */ atomic_long_inc(&mm->nr_accessed); + count_kmmscand_migadded(); } - } + } else + count_kmmscand_idlepage(); end: folio_set_idle(folio); folio_put(folio); @@ -632,6 +671,12 @@ static void kmmscand_migrate_folio(void) if (info->mm) ret =3D kmmscand_promote_folio(info); =20 + /* TBD: encode migrated count here, currently assume folio_nr_pages */ + if (!ret) + count_kmmscand_migrated(); + else + count_kmmscand_migrate_failed(); + kfree(info); =20 spin_lock(&kmmscand_migrate_lock); @@ -799,6 +844,7 @@ static unsigned long kmmscand_scan_mm_slot(void) for_each_vma(vmi, vma) { /* Count the scanned pages here to decide exit */ kmmscand_walk_page_vma(vma); + count_kmmscand_vma_scans(); scanned_size +=3D vma->vm_end - vma->vm_start; address =3D vma->vm_end; =20 @@ -822,6 +868,8 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + count_kmmscand_mm_scans(); + outerloop: /* exit_mmap will destroy ptes after this */ mmap_read_unlock(mm); diff --git a/mm/vmstat.c b/mm/vmstat.c index 4d016314a56c..d758e7155042 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1340,6 +1340,16 @@ const char * const vmstat_text[] =3D { "numa_hint_faults_local", "numa_pages_migrated", #endif +#ifdef CONFIG_KMMSCAND + "nr_kmmscand_mm_scans", + "nr_kmmscand_vma_scans", + "nr_kmmscand_migadded", + "nr_kmmscand_migrated", + "nr_kmmscand_migrate_failed", + "nr_kmmscand_slowtier", + "nr_kmmscand_toptier", + "nr_kmmscand_idlepage", +#endif #ifdef CONFIG_MIGRATION "pgmigrate_success", "pgmigrate_fail", --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2076.outbound.protection.outlook.com [40.107.243.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF75B1D14F8; Sun, 1 Dec 2024 15:39:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067598; cv=fail; b=fPOowkckeCwyODisCB1Mk2njGFNZq4EHUCJlUSGPaJRPNJWTUwVVYfTVmucrsjtyQc+DTsdUdspaU0rHLCrba58yn2A5o8Jz4h7I5S5ZxaoSi7/lBBWAWVSlqbXt37LKQCWnFMhY+XjN6/MO1VwtemGmjAqnpSnXDlZJcHwKOt8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067598; c=relaxed/simple; bh=YwcOx3T4/8+NAeC2q7ZMYcYRWS5D9+R0Dl4TsSBTe6Y=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=WBA2Vrg2JvJViw9s7lh5LKZ3rSNxoQ2zFqenll4xPpO7ApU9TvGnSEdaPv/RYnWiYdOBnUah8fWKnUMjQpoQ470V/jffZMnSEEila6y7njLb8w67Qwb/79vY6XGL7iTj3kgib9aQsP6OWhUf9V2cIJmBGpgrrzSQjGZRM80kt+k= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=oqfYKGdo; arc=fail smtp.client-ip=40.107.243.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="oqfYKGdo" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gs3A/GjIq+t/gBe+rLFC452vxst1bSaalYMr+nLu76FAWcZcYkzPuPxNzUldfO81J7tBXm+I9G2PHd8F4EthS8TgiHDS3qyy6AhxlKD2oLap+ZWsh9JM2H5mCsCXrhIJ3/JSSwckfwb/up8PriShdmGDh4hCeCFaWA8wojPTWHIcerjhfoYlgREIDz5N0lGRYzRj3qWpQgEGb2kSpYnhG7RlnRle3BN/liHKKPaWFdAuGrjD5JTEIvMt40aSHx7DHpa7Gv5rzEyUtOWRzF7Ss1XkghJ8b/N7GR+FMDdgYN2F11Fap6Txjecf33jJ1yRzxAlyBq5kP0sXTFB6tb0gSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+rBTqNeMywA3YqEk7Nh/vZKAp1YDQTBwl5ueiIClenY=; b=FJgBb0B0HKeaxKwrumhX8e+x8nMIEC3/zlb+cffXj25sr7/IbOcD15eskBZYoCWGz8ze8e+y0mnPjCNbRg4/3qJEBaMR2+JZd6SIsLKOX4+yxzRsaeTDcEJ2WnOhsXudcR0xqnhjtHawMJCBBK1gbVcH8ojjfFC/C3fgy19jJGwcIUZ1txI/HeXMuKlW8kGX6oTEzt/+YJQs7veCblYX8KE9D/Xb3JP7HIgJiWCL7EogYPansw9eLhxXDZLEXUy8vQO1R+oki2qdI3/hj5oJkoNY8ruFgxtzrVnA+RuYXeu2qiQF3ark+y8zDsAHJJUuFd9Dd9iKfX5RNdqV7tE7YA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+rBTqNeMywA3YqEk7Nh/vZKAp1YDQTBwl5ueiIClenY=; b=oqfYKGdoZGnakZH5ik3/RuuVhWyIo8vAj6DnrGbzR3XFevAFv1TXtylOc4NuP3T6wUHidQBk6ZTmp7Qv4PSf+/XCzTVyqKkBs+s2Otc7HrHyElxFinBYt0szqE13TikYjmWM186VAdFuEV2gSIooxEiZl6eFyAqYmMBNbdvzrwk= Received: from CH2PR11CA0017.namprd11.prod.outlook.com (2603:10b6:610:54::27) by PH0PR12MB8821.namprd12.prod.outlook.com (2603:10b6:510:28d::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.16; Sun, 1 Dec 2024 15:39:49 +0000 Received: from CH2PEPF00000145.namprd02.prod.outlook.com (2603:10b6:610:54:cafe::19) by CH2PR11CA0017.outlook.office365.com (2603:10b6:610:54::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.17 via Frontend Transport; Sun, 1 Dec 2024 15:39:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH2PEPF00000145.mail.protection.outlook.com (10.167.244.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 15:39:49 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:41 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T , Steven Rostedt , "Masami Hiramatsu" , Subject: [RFC PATCH V0 09/10] trace/kmmscand: Add tracing of scanning and migration Date: Sun, 1 Dec 2024 15:38:17 +0000 Message-ID: <20241201153818.2633616-10-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000145:EE_|PH0PR12MB8821:EE_ X-MS-Office365-Filtering-Correlation-Id: c18a81a7-b413-4120-3fb7-08dd121e63f8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|1800799024|36860700013|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?+S4vgTk4HYFyPkj3jl3MLxi3ijH2yDRrLPPvtmCALRco9zr0x7ISsDI7V6uV?= =?us-ascii?Q?Mcfz1+BLMZtNE+J+xRg/P/oTU2Wmks0q995FiWOBLy/KQJcXEAEXXmfZ5GIn?= =?us-ascii?Q?YtKTJ3Lz3sxW3OM0qGtRAu8Wjyzvkjw8YFKrfrD2tu82ZD4jfC8e3Cv+bhwo?= =?us-ascii?Q?jJDgahPPVtzFYzQKg8r8XsBVf+7HPweTZC+qm4/GmCevoNGE2iWMq7oficJ0?= =?us-ascii?Q?tZX8fh6eQGU1FzAXa6WFknI65peVW5nA1byj8H8nLAWni//Iyuk9HWSWBpdC?= =?us-ascii?Q?e1LWzajAXv0NhJ/Vm3JQxGb3G4vS8MUpWKf9VUgcAkTrJXZt0CFRBPc6Bval?= =?us-ascii?Q?iYjxq7/gd5hnzObMmfK9Z54ZAc5nOQB3SKBO/B4owgVZ0w9QVMNnOzo373Lg?= =?us-ascii?Q?3OzM0Tr0Y1FoHx7w0pGuiRLTeX1Joo9II4cniBzA/OdEV7Be5332oTmyn/ak?= =?us-ascii?Q?mEM0DWH0feAW/NnArCfUCdpI2cxfFCpauk5eZI9j2b+MJZ2n4+EIiMDjdZVD?= =?us-ascii?Q?rr2OX8vPds95axd6Q5fdNudecNJCWAxTb8lbnaq4oAb3pVAOzs33YyIVYdFp?= =?us-ascii?Q?AgP4g154l2hXml7IVxsQ8iwGyGdbNnRayWyqa+SPr29nm5lzW3MjNFQXC5VS?= =?us-ascii?Q?m6Rwb3/fZblU+x9J7gdulB2tCyaxhQRgf5hCdQNGRhpYX3PfuCZ5gQUE8DIV?= =?us-ascii?Q?3S3k5HlsRwe+zHvaSmwFmk7PZi1CZDW05yXKTqU969hdMebQmM6tCH62PGJr?= =?us-ascii?Q?+BNtlazISEhadjZYoDSNVpXJFHu7sqMerkelTf4ZXtjwxlZcDSVl3+Xtnony?= =?us-ascii?Q?ozjxJ6iuWwJHirXeLjTppZTcjDxZS2y7VbaNw5rSmbzSrpAIQCQgbDxm5t5P?= =?us-ascii?Q?VTCXf6kqS+8hzEkRpI3/QHnev6+OryrDsubtdZTa0dza+5vbLfrB9H2dEfmn?= =?us-ascii?Q?U2tBfIVIKsveYedT9X80rfbWOXf3MnUpnWIT7TRtaWma1ncRiis+JwZtOODX?= =?us-ascii?Q?SoR1sSdkxPqIuX55N8Whp17jrafFsGSJDIXhz9cBwTqC/Bc0U7oau+Qr2Wg8?= =?us-ascii?Q?J44pVZvCcc+Gnvqj1jf7xnxnIFSnF4c3KWUctaOExxhgakTlBkG79FlF8LIk?= =?us-ascii?Q?9vSKtopL7aeEDvn+4ZKVJmrJmr9fLpmNChIsWSyA77zidD6jhUE16r0Vj+Wc?= =?us-ascii?Q?CNUigX7p4Mf56MMs4SX9/wmw8ffqV8GqOQgtOFTGiCOi/8V0vd3BAENIdDDo?= =?us-ascii?Q?vN4uSiuzekeMDyr+x5p5Hk/K/mb2YU5qBH9JPF19QFt+qF+18GCqdT/ReDXg?= =?us-ascii?Q?6Cev3Te36ujI3bD8uVVWX4kHXHWvvEJBbfkgFHPRjdtywBwrUTK4g4dpsQQV?= =?us-ascii?Q?hWB4OXBMLafgfr1gAEbo7IPy1EijSEEo0TB06Mc1Y96sch+0haEFBXiBrPiC?= =?us-ascii?Q?qqbMk49ROYOJbfXskLzdx9gD4r8R6Ju7Xd0u64murEVMkuVHzmeiWg=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(1800799024)(36860700013)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:49.1542 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c18a81a7-b413-4120-3fb7-08dd121e63f8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000145.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8821 Content-Type: text/plain; charset="utf-8" Add tracing support to track - start and end of scanning. - migration. CC: Steven Rostedt CC: Masami Hiramatsu CC: linux-trace-kernel@vger.kernel.org Signed-off-by: Raghavendra K T --- include/trace/events/kmem.h | 99 +++++++++++++++++++++++++++++++++++++ mm/kmmscand.c | 12 ++++- 2 files changed, 110 insertions(+), 1 deletion(-) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index b37eb0a7060f..80978d85300d 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -9,6 +9,105 @@ #include #include =20 +TRACE_EVENT(kmem_mm_enter, + + TP_PROTO(struct task_struct *t, + struct mm_struct *mm), + + TP_ARGS(t, mm), + + TP_STRUCT__entry( + __array( char, comm, TASK_COMM_LEN ) + __field( struct mm_struct *, mm ) + ), + + TP_fast_assign( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->mm =3D mm; + ), + + TP_printk("kmmscand: mm_enter comm =3D%s mm=3D%p", __entry->comm, __entry= ->mm) +); + +TRACE_EVENT(kmem_scan_mm_start, + + TP_PROTO(struct task_struct *t, + struct mm_struct *mm), + + TP_ARGS(t, mm), + + TP_STRUCT__entry( + __array( char, comm, TASK_COMM_LEN ) + __field( struct mm_struct *, mm ) + ), + + TP_fast_assign( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->mm =3D mm; + ), + + TP_printk("kmmscand: scan_mm_start comm =3D%s mm=3D%p", __entry->comm, __= entry->mm) +); + +TRACE_EVENT(kmem_scan_mm_end, + + TP_PROTO(struct task_struct *t, + struct mm_struct *mm, + unsigned long start, + unsigned long total, + unsigned long scan_size, + unsigned long scan_period), + + TP_ARGS(t, mm, start, total, scan_period, scan_size), + + TP_STRUCT__entry( + __array( char, comm, TASK_COMM_LEN ) + __field( struct mm_struct *, mm ) + __field( unsigned long, start ) + __field( unsigned long, total ) + __field( unsigned long, scan_period ) + __field( unsigned long, scan_size ) + ), + + TP_fast_assign( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->mm =3D mm; + __entry->start =3D start; + __entry->total =3D total; + __entry->scan_period =3D scan_period; + __entry->scan_size =3D scan_size; + ), + + TP_printk("kmmscand: scan_mm_end comm =3D%s mm=3D%p, start =3D %ld, total= =3D %ld, scan_period =3D %ld, scan_size =3D %ld", + __entry->comm, __entry->mm, __entry->start, + __entry->total, __entry->scan_period, __entry->scan_size) +); + +TRACE_EVENT(kmem_scan_mm_migrate, + + TP_PROTO(struct task_struct *t, + struct mm_struct *mm, + int rc), + + TP_ARGS(t, mm, rc), + + TP_STRUCT__entry( + __array( char, comm, TASK_COMM_LEN ) + __field( struct mm_struct *, mm ) + __field( int, rc ) + ), + + TP_fast_assign( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->mm =3D mm; + __entry->rc =3D rc; + ), + + TP_printk("kmmscand: scan_mm_migrate comm =3D%s mm=3D%p rc=3D%d", __entry= ->comm, + __entry->mm, __entry->rc) +); + + TRACE_EVENT(kmem_cache_alloc, =20 TP_PROTO(unsigned long call_site, diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 682c0523c0b4..70f588a210dd 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -668,8 +668,10 @@ static void kmmscand_migrate_folio(void) WRITE_ONCE(kmmscand_cur_migrate_mm, info->mm); spin_unlock(&kmmscand_migrate_lock); =20 - if (info->mm) + if (info->mm) { ret =3D kmmscand_promote_folio(info); + trace_kmem_scan_mm_migrate(info->mm->owner, info->mm, ret); + } =20 /* TBD: encode migrated count here, currently assume folio_nr_pages */ if (!ret) @@ -828,6 +830,9 @@ static unsigned long kmmscand_scan_mm_slot(void) goto outerloop; } =20 + if (mm->owner) + trace_kmem_scan_mm_start(mm->owner, mm); + now =3D jiffies; /* * Dont scan if : @@ -868,6 +873,10 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + if (mm->owner) + trace_kmem_scan_mm_end(mm->owner, mm, address, total, + mm_slot_scan_period, mm_slot_scan_size); + count_kmmscand_mm_scans(); =20 outerloop: @@ -1020,6 +1029,7 @@ void __kmmscand_enter(struct mm_struct *mm) spin_unlock(&kmmscand_mm_lock); =20 mmgrab(mm); + trace_kmem_mm_enter(mm->owner, mm); if (wakeup) wake_up_interruptible(&kmmscand_wait); } --=20 2.39.3 From nobody Sun Feb 8 04:23:28 2026 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2060.outbound.protection.outlook.com [40.107.220.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E1D51D7E47 for ; Sun, 1 Dec 2024 15:40:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.60 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067602; cv=fail; b=DfP9NltBqtUxXigZl1PXYJvWFWxnnkJzrA88KG9wiaCA6TXY/eQZPLQM7kTO4FIM9N910DK2A46CX95mRCV/tTco8pQrxoUGQT3dgJ2z2FoQh3CT/5QnfEuWOGUcdP2y24Jpnm36noZ15E04iAx9CmGh9sIo73TD8af0d0gFpzw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733067602; c=relaxed/simple; bh=vUMKifGphLk2rkjeITLewD7MDjD9cBi8oTvcEFCa13w=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=a8jk/opCWnbX9ot3j472i5WnjEX0Sohy0WOejkP9ECiEaoY5m53V5C1luTu8nLNnD04CWwgD36AVlcvyvFDkULy28JiKsJh9Rrzt/2ckcb2bjJyZ4HcLfDKkvZS0n18fMyea3TL3QliF4KFitt5wXAFqhe1raVvvB8w7pNXT2qM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=VNrTv36v; arc=fail smtp.client-ip=40.107.220.60 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="VNrTv36v" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=U7DXZ6v5yJrhxkR96+O9CrE5Fha5lhaxIUAM5WyfyrdJ+dViehcoe4Gj5GQD22PqkUiOKQ8DOZnKo1epyR+ZZbHR8dW14dvIv4DtZ37dXQkueLvTu6Y2P+2i+lMuLqcbJKtAZz/TAGDxXJ2RnYos8hgd8snJOUd5Vaj1303n+mw4zmgJafPDyjo4/o2vH6y0WnDYjLc7S3Ga7YtE/TQPU/CtgiO6LPFnfzxLkzQGYKb9G+eXtm42yiz3ofskWPjEgB2859284+8HUQvBpqeRqzs6KPu1miGqG4pume2AzS5VPJ3vl0RVKT9L+3j7NRVg6PDK9bI6gCd90X48DpQ4ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2KPvoZ18wyLsmWIQuLjOWxUMZNiR39hSPKzJVIw8JaU=; b=w4ZLdPDr8dGvTnlRcyOAppeX2RlHULMXij3OF6/in0hhwONLuLKTUlx70KB0plsSTxu0+45VSE23DZ6EEZZzBJop+4KYIQxvqoQ4Dob40edJfN73DQIJHkm7s5I1hrlP5fIBqwUTR+WkqbolcjifycKQdrSAfHgyAcoEWv0eVK7Lex3EkrEuwCm2q25g8Relzy5HDCRr022N+N8i1ZxUZpKs50sAwEG0WXFkWHwvc/ITjJ5B8RkkMf1lu5/QKt7gqoZaLXdaNcGBXvwVLGPOzUvYbUWP/R41+whY/2E+RiNB/y8nyGF5BELvMQqV+JbipW0XZuN6yhoRciS4DJzmEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2KPvoZ18wyLsmWIQuLjOWxUMZNiR39hSPKzJVIw8JaU=; b=VNrTv36vsW4GZdK/uBhiEKIq56oHWfdk5nLtgCR+dfz1CB/gGvheeE77VxDP3emRFtKsVRrWOyGpwbl6DTuLQ2baOqHfFSOXEQHcD+w0c6wklBTDgztCC5nlx4I4LMlxPDnKlZWSg97NVHrR7Ob4zs2GZdb+yqRyry3jk6OPxAw= Received: from CH0PR08CA0030.namprd08.prod.outlook.com (2603:10b6:610:33::35) by MW6PR12MB8899.namprd12.prod.outlook.com (2603:10b6:303:248::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 15:39:56 +0000 Received: from CH2PEPF00000148.namprd02.prod.outlook.com (2603:10b6:610:33:cafe::e8) by CH0PR08CA0030.outlook.office365.com (2603:10b6:610:33::35) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH2PEPF00000148.mail.protection.outlook.com (10.167.244.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 15:39:55 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:48 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 DO NOT MERGE 10/10] kmmscand: Add scanning Date: Sun, 1 Dec 2024 15:38:18 +0000 Message-ID: <20241201153818.2633616-11-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000148:EE_|MW6PR12MB8899:EE_ X-MS-Office365-Filtering-Correlation-Id: 9487d1b8-ca1e-4444-29d4-08dd121e67da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?qRHIFNS3ZA5Ti26MqSvOA3U+lhXfOwORwh7krx2bdo1WG/e0162tr8qJHlVr?= =?us-ascii?Q?WXdCBNPBk0pShVTaY3V0dbVG1Sskh6SnOjCJQRnfPUOA82PibX4enD+Bak/J?= =?us-ascii?Q?aeBcAWbI9ZUA9XabADYADutSoa9NXv7E8goT7NCxra4x8DE8Qm09WoPF1YK+?= =?us-ascii?Q?0oNQvGhkylkLoaki53aIvmmlVzcdgSitcHq4biKvBR31uDdpkZFWE7Wv1ICn?= =?us-ascii?Q?277j8zMb/Y+Rk+vLPxODYYSvPUiALyHomWRHkatempm9w2mwc0I6+baEco1+?= =?us-ascii?Q?pOMJMuxYJlKMlm9lbNP5UPGOnIJIEHpj9XHYDbWFFuTRbWC1hBwf5fxQCLEU?= =?us-ascii?Q?jkd+QrYkngSGbZJ/CQwGDo7itKB90iuXgMhjo7jvtk0P9B0ZKYuJqswXW+DT?= =?us-ascii?Q?F+EGsRhoNTlkHVIfo4rQp9Xa8it+KqAqgb1ryW9QM60InxElYjuEgDTxwt+Z?= =?us-ascii?Q?G9Ygk+4zIvVXk4WsBm/uN34P3m/N6KduI0V7mXXvTtF5qvUhANXzqJab42ZI?= =?us-ascii?Q?jvFz6VhonyK2R4oKvyh351yqpOnFduEoMoQ2czNakM7eimrNTYa343A6+O9r?= =?us-ascii?Q?NUTKHY84uswbN2qtoXYpr+Cxg/RdBjidlHTbNrV8EVXj24oGJok/QXL405zx?= =?us-ascii?Q?fH9rE1izR8Kh+7fecKIZgVc6qcIA0n/rOn5DjVsPSMphXImnjkiEn4j1ZVYT?= =?us-ascii?Q?NiKXgdneSmx1LdBKUFCyJ18RVap9T6llTg/oE5O/j7iErb5b+hLXvb0/M2os?= =?us-ascii?Q?Zh3fY0N6qBgpO6M7im6C6TPfUi1fPJc9dziP0Fva/0wCk5Cu6luGn2obBn1v?= =?us-ascii?Q?2kMqu801CNw7kbFo0fTc0mkHpyr7EWLik7VncZNIp3nI+ve1PpfOZt6keyEA?= =?us-ascii?Q?Pud+nyfTlbXbTDCurKIhsSJBtjw/tNeFhX8JK08pOaCPHHsbOPJWD0cgKT6O?= =?us-ascii?Q?bd4PPGzmNOhV5yCBl+ieNspzfihc9CkVRt5rTgGABpWBr2WG8jfRsGCSrSu1?= =?us-ascii?Q?u8sUK5DjId3HTz1xO/ngexKOhSw0x9GihXwjmDgp2xa/XaR9S7nffFv6NFHG?= =?us-ascii?Q?Fpni9JYDCu6raSwS3jN2JXCPWxD4I+mreHj3+a9Z+xrMMYSjbtihrVSmDZoo?= =?us-ascii?Q?1xSYt08sZDxHzLzfp4AzkEdi0/eanCK06TlTzHsdKJXoBLJKCchoc91LZDqY?= =?us-ascii?Q?RWRQZ0tKNR8epFIpgoCXd9z80nHnzqKhK4KhF/os5QVwHFmdtJZV7L5L0YfD?= =?us-ascii?Q?K+OGVnc0HB5azKzmcZWrdQArQgdjjgq/fZVoXa53g/JZupLwFbbPn7VWxo7s?= =?us-ascii?Q?5R+DfR//NhB7oLNjwmMFsnlpl8/Y+E1t8W2Q0SinKoSuDDZNCPy4IMVK+ubD?= =?us-ascii?Q?KOa/byqGy8XRaaJqMR1i4yzoNXNsB0j4d/kbZ6trFZVxdKMdsvqW8YEdDmNE?= =?us-ascii?Q?yYXPgpHF50Phxa32M6EItnsPFOXBhKPrsiGBFhk/aHCM4NFi+sWkXg=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:55.6366 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9487d1b8-ca1e-4444-29d4-08dd121e67da X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000148.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8899 Content-Type: text/plain; charset="utf-8" overhead caclulation support Intended to be used only for experimental purpose. Not to be merged. Signed-off-by: Raghavendra K T --- include/linux/mm.h | 3 +++ include/linux/vm_event_item.h | 4 ++++ kernel/sched/fair.c | 13 ++++++++----- mm/huge_memory.c | 1 + mm/kmmscand.c | 9 +++++++++ mm/memory.c | 12 ++++++++---- mm/vmstat.c | 4 ++++ 7 files changed, 37 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 306452c11d31..7380aab1fa62 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -600,6 +600,7 @@ struct vm_fault { * page table to avoid allocation from * atomic context. */ + unsigned long start_time; }; =20 /* @@ -690,6 +691,8 @@ void count_kmmscand_migrate_failed(void); void count_kmmscand_slowtier(void); void count_kmmscand_toptier(void); void count_kmmscand_idlepage(void); +void count_kmmscand_scan_oh(long delta); +void count_kmmscand_migration_oh(long delta); #endif =20 #ifdef CONFIG_NUMA_BALANCING diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index b2ccd4f665aa..4c7eaea01f13 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -64,6 +64,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HINT_FAULTS, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, + NUMA_TASK_WORK_OH, + NUMA_HF_MIGRATION_OH, #endif #ifdef CONFIG_KMMSCAND KMMSCAND_MM_SCANS, @@ -74,6 +76,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KMMSCAND_SLOWTIER, KMMSCAND_TOPTIER, KMMSCAND_IDLEPAGE, + KMMSCAND_SCAN_OH, + KMMSCAND_MIGRATION_OH, #endif #ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fbdca89c677f..d205be30ae6c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3299,6 +3299,7 @@ static void task_numa_work(struct callback_head *work) struct vma_iterator vmi; bool vma_pids_skipped; bool vma_pids_forced =3D false; + unsigned long old =3D jiffies; =20 SCHED_WARN_ON(p !=3D container_of(work, struct task_struct, numa_work)); =20 @@ -3312,7 +3313,7 @@ static void task_numa_work(struct callback_head *work) * work. */ if (p->flags & PF_EXITING) - return; + goto out1; =20 if (!mm->numa_next_scan) { mm->numa_next_scan =3D now + @@ -3324,7 +3325,7 @@ static void task_numa_work(struct callback_head *work) */ migrate =3D mm->numa_next_scan; if (time_before(now, migrate)) - return; + goto out1; =20 if (p->numa_scan_period =3D=3D 0) { p->numa_scan_period_max =3D task_scan_max(p); @@ -3333,7 +3334,7 @@ static void task_numa_work(struct callback_head *work) =20 next_scan =3D now + msecs_to_jiffies(p->numa_scan_period); if (!try_cmpxchg(&mm->numa_next_scan, &migrate, next_scan)) - return; + goto out1; =20 /* * Delay this task enough that another task of this mm will likely win @@ -3345,11 +3346,11 @@ static void task_numa_work(struct callback_head *wo= rk) pages <<=3D 20 - PAGE_SHIFT; /* MB in pages */ virtpages =3D pages * 8; /* Scan up to this much virtual space */ if (!pages) - return; + goto out1; =20 =20 if (!mmap_read_trylock(mm)) - return; + goto out1; =20 /* * VMAs are skipped if the current PID has not trapped a fault within @@ -3526,6 +3527,8 @@ static void task_numa_work(struct callback_head *work) u64 diff =3D p->se.sum_exec_runtime - runtime; p->node_stamp +=3D 32 * diff; } +out1: + __count_vm_events(NUMA_TASK_WORK_OH, jiffies_to_usecs(jiffies - old)); } =20 void init_numa_balancing(unsigned long clone_flags, struct task_struct *p) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ee335d96fc39..d948d1fbbffd 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1250,6 +1250,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct= vm_fault *vmf) spin_unlock(vmf->ptl); } =20 + __count_vm_events(NUMA_HF_MIGRATION_OH, jiffies_to_usecs(jiffies - vmf->s= tart_time)); return 0; unlock_release: spin_unlock(vmf->ptl); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 70f588a210dd..bd2c65f38da2 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -644,8 +644,10 @@ static void kmmscand_cleanup_migration_list(struct mm_= struct *mm) static void kmmscand_migrate_folio(void) { int ret =3D 0; + unsigned long tstart, tend; struct kmmscand_migrate_info *info, *tmp; =20 + tstart =3D jiffies; spin_lock(&kmmscand_migrate_lock); =20 if (!list_empty(&kmmscand_migrate_list.migrate_head)) { @@ -691,6 +693,8 @@ static void kmmscand_migrate_folio(void) } } spin_unlock(&kmmscand_migrate_lock); + tend =3D jiffies; + __count_vm_events(KMMSCAND_MIGRATION_OH, jiffies_to_usecs(tend - tstart)); } =20 /* @@ -788,6 +792,8 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 unsigned int mm_slot_scan_period; unsigned long now; + + unsigned long tstart, tend; unsigned long mm_slot_next_scan; unsigned long mm_slot_scan_size; unsigned long scanned_size =3D 0; @@ -800,6 +806,7 @@ static unsigned long kmmscand_scan_mm_slot(void) struct vm_area_struct *vma =3D NULL; struct kmmscand_mm_slot *mm_slot; =20 + tstart =3D jiffies; /* Retrieve mm */ spin_lock(&kmmscand_mm_lock); =20 @@ -917,6 +924,8 @@ static unsigned long kmmscand_scan_mm_slot(void) } =20 spin_unlock(&kmmscand_mm_lock); + tend =3D jiffies; + __count_vm_events(KMMSCAND_SCAN_OH, jiffies_to_usecs(tend - tstart)); return total; } =20 diff --git a/mm/memory.c b/mm/memory.c index 75c2dfd04f72..baea436124b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5590,7 +5590,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) =20 if (unlikely(!pte_same(old_pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); - return 0; + goto out; } =20 pte =3D pte_modify(old_pte, vma->vm_page_prot); @@ -5629,17 +5629,18 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) nid =3D target_nid; flags |=3D TNF_MIGRATED; task_numa_fault(last_cpupid, nid, nr_pages, flags); - return 0; + goto out; } =20 flags |=3D TNF_MIGRATE_FAIL; vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte)) - return 0; + goto out; + if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); - return 0; + goto out; } out_map: /* @@ -5656,6 +5657,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) =20 if (nid !=3D NUMA_NO_NODE) task_numa_fault(last_cpupid, nid, nr_pages, flags); +out: + __count_vm_events(NUMA_HF_MIGRATION_OH, jiffies_to_usecs(jiffies - vmf->s= tart_time)); return 0; } =20 @@ -5858,6 +5861,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_st= ruct *vma, .flags =3D flags, .pgoff =3D linear_page_index(vma, address), .gfp_mask =3D __get_fault_gfp_mask(vma), + .start_time =3D jiffies, }; struct mm_struct *mm =3D vma->vm_mm; unsigned long vm_flags =3D vma->vm_flags; diff --git a/mm/vmstat.c b/mm/vmstat.c index d758e7155042..b7fe51342970 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1339,6 +1339,8 @@ const char * const vmstat_text[] =3D { "numa_hint_faults", "numa_hint_faults_local", "numa_pages_migrated", + "numa_task_work_oh", + "numa_hf_migration_oh", #endif #ifdef CONFIG_KMMSCAND "nr_kmmscand_mm_scans", @@ -1349,6 +1351,8 @@ const char * const vmstat_text[] =3D { "nr_kmmscand_slowtier", "nr_kmmscand_toptier", "nr_kmmscand_idlepage", + "kmmscand_scan_oh", + "kmmscand_migration_oh", #endif #ifdef CONFIG_MIGRATION "pgmigrate_success", --=20 2.39.3