From nobody Sat Oct 4 15:57:17 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2052.outbound.protection.outlook.com [40.107.223.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FEA222CBD3 for ; Thu, 14 Aug 2025 15:33:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.52 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185629; cv=fail; b=q8hOMcyfWixocOwR4g3t/Gz1Vm8DkXkDr67Sdp/cX4YehNSYr12Z4sTgzcEyG4+uQ0m0I+aHg1T+ss7Z2zLBIgdQHwrrJ8sB4LErc05KkxBkPb5f/CmA5FPIDdMo8ULI/dj8E8fmnicTxFyBeg7EXn0+1z3fsnyyu3SjA5bMJCg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185629; c=relaxed/simple; bh=WFHzEkmBJPmOUQBndk9D31F/Lx/AhrKCtA1eXPm0vEs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=R8sYi2vfBOVAbRlLQHRYD8K7zhoKIKXhwkYdYfRHapZwauFzRujBPJRr57hftY8X0M8xQhDrHRu31rK95lhWz0brv+EbzanJLaWge2KpE9Y1nvc4FJe1JhsBILQ0EvxX3x3ao6OzNe4cWrTX7TQkDUeJQcoOpDfPmIL8ppL5Wsg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=NVXcTV8I; arc=fail smtp.client-ip=40.107.223.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="NVXcTV8I" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TdUU7Ck7x6dZNShMXaxIVekgHrFGGe8PAlfgjtY9hom+hUANsJSBedsvrEx2zFtixFA/V6zuvQgHHBVWttWpVLEtajYtx6Kcon269mCN+oYB8HeoCRFwHpb7of1PCo3t/j6FwZ99zNE1ZECz8DfZaUEJt1uhTloGp+fVAT799xYd1Q/QNnJBKqN07uXgZCY8+IArU+EcpAAeMehmNBYTHChZ6zFnoLnkEiIZwwblypSAU4n/jb4Yi+JJM82ZwWujXNClMS3GoKdyktNsEsh5+CWpLBLMtyGH4jbuRnOEuRoLIuPuzmh8CzjxJALCypOp63Dbv63HlMroJCSZLzz+5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=U8ZBHsg2ICU7iooyE/1e0HC8I4bKgZSlDZ7JgRXUGVc=; b=nv51vOeQR3zgDkhN88W0/2MwSjkAKBMsiyYri9/JBaQY4gS1gNUeSQ4VUKL5ZbhXCnaXLHichFijyHVr6or5Epp/qNSrWfWg17oJzb4WkVUA15Q+vnzke7hLCuTvNVTZSPLQofGmcLPAoNeqk9Bu6FytLtVkH6HJR3kUCYDaIZhz/llJvQcnx+O8+1YiNzAP3EknfGhQy9RNetlyse/xSZUtXAZVrLC79Rcc0vd8yeNrAkU9bbTiZo6U0e/M/aA+pZ7+E3sxAAnRXeKZwYcflM6wPW+7ieA+fc+qiln6vp9p+jtYfLOl8qdquwjtzvC2yRz2L1jU2EkRj5GWswcrFw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U8ZBHsg2ICU7iooyE/1e0HC8I4bKgZSlDZ7JgRXUGVc=; b=NVXcTV8I79svMtDXpPLQXb3A+MMjtT5kcOjUsP6sgXtuianhYbnKy4U5N8wkZW4+D7+63bvJJUXrc56k4tQxksXzWdnftBrdTjo8XCEtsUbrn/08D1ye+f2EItnaMC0wZXRFQoUxXlj/g8SVV83uUScOU2VAAYXm/KKr6GQMyzo= Received: from MN0PR04CA0003.namprd04.prod.outlook.com (2603:10b6:208:52d::15) by LV8PR12MB9417.namprd12.prod.outlook.com (2603:10b6:408:204::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.15; Thu, 14 Aug 2025 15:33:43 +0000 Received: from BN3PEPF0000B077.namprd04.prod.outlook.com (2603:10b6:208:52d:cafe::bf) by MN0PR04CA0003.outlook.office365.com (2603:10b6:208:52d::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:33:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B077.mail.protection.outlook.com (10.167.243.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:33:42 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:33:31 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 01/17] mm: Add kscand kthread for PTE A bit scan Date: Thu, 14 Aug 2025 15:32:51 +0000 Message-ID: <20250814153307.1553061-2-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B077:EE_|LV8PR12MB9417:EE_ X-MS-Office365-Filtering-Correlation-Id: 5fc26e38-0182-48ea-8adc-08dddb47f338 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|82310400026|36860700013|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?IaI4EpwIE6SWNJTbatCz3Wi+R1Xlop3wsKrgg0M1U5/S1ha6cNrVgdX4ievG?= =?us-ascii?Q?ZJLJfMEYAgEWbHJxPx1QPCOiN0H/WMBvq/whI1bhh1QIb4OCmld4dUzTU3tG?= =?us-ascii?Q?wWNFAlACBmoeaTHdHhoslkmBYptFmwzvvihRWd1Znkv2909Bl/334zbyeLEp?= =?us-ascii?Q?UY5TrpryA1PFlt8viKQGrmDwJuJ9dw5KansFCnY76OjowNawnN/Qqr9rBErr?= =?us-ascii?Q?O309mR1XtFVrheMprlOXHiFNYjATYSTgiXxouFdiZo7RHG+1AGvwRkuUyzYg?= =?us-ascii?Q?S1h4pZpldhwoGzQFHtD8AQruQmBsTvJltmZzeQ3cntaVqHnDjQasYkwov2Po?= =?us-ascii?Q?N2dAIDv8dt1aZqFWvpBP/gTDXazU04BMKyKx16T+/veuPN19v7zowK2PstPD?= =?us-ascii?Q?WYRYF/obZw5Eg7mN7bbwXnu/rhPteT7VeseSoiWwg4Fh2sbIYsgLKDpSbPF+?= =?us-ascii?Q?H/u7ztZVqCUMctGLLfMshF3zGWDH4XIPFjVRng3JxwLghD/s0VsL0yc1fpq4?= =?us-ascii?Q?3USosIKgaLQtit6ud4FSweUOeQivSIHzfvAWyX+BB/UglfZqQun0h93sle4P?= =?us-ascii?Q?IidrbormP1foLGa+8tQP2IVCQCp3VRBHim0aYmXddYijcjARDhj8aAn8fwSJ?= =?us-ascii?Q?hhJWNr4R+bQA+4U7ZTlJCAH/9Y877u9Mxpmakr5GNuJVX4twKD1OApuoQeLP?= =?us-ascii?Q?9Egb/cAeCUvIQLcHhIFZI+N+7rJcsaXB8ESolYL6XVYoj2/vn8w28QL9zsQB?= =?us-ascii?Q?EEbithZnjU+3aO0Ytvqrsa/Cxb83YDscN0qiW4nr6C7otMhGjkjcuWacP7PC?= =?us-ascii?Q?bgdaS6Ew45W/HfiuKDI42zlYXYobbEJAihS4udRnRwuCqwxnvrXzXU+KWtMc?= =?us-ascii?Q?YaIOOgiyrb1Nbzzye19Nq5muv2kJzbSvyT1n0mV+HIphzmKKUS5Kl8qfY02o?= =?us-ascii?Q?vKm1+bWJme6xv6c7YB/w42NTKqFnhtvdy9KFmzwcn7D322vCTvIQG1Rgh1kE?= =?us-ascii?Q?F3J/FKF/wg0j2UEstQjuEcahifNHm4sJ4EsIqz3X6emNu5nxfmPoU4FLWgXO?= =?us-ascii?Q?hbNOS3h0LOA5bnCf3leqA1da5w2i3SWvdz8ttFcXhBlg/emARgiqGmp5EpIg?= =?us-ascii?Q?Uk0GM9jYNz3x0qHj3D0gw6cz8d3wqXOT5baEUb2TBzGO0nFJ0Ptl/fqUbydj?= =?us-ascii?Q?bU0CaXGDohI73rluqAyTjeatzzXSr5cmNw8yTww/HH4LppMhE7refstdHyIy?= =?us-ascii?Q?K4faDjBaghNOx8owR9Sa7G+mUYkgLZ3pEa8ZC4PgYPu1wE8OKoxYGgBx6W/z?= =?us-ascii?Q?LJtWgfHBesSDEDVP4w8ZGwtEtphxC+o6F+PAptmoGmdPDsvviI7spOZkdkMa?= =?us-ascii?Q?l2rR5GGuh25lPSz2kkXmHuKWFAvP1yyoSpP/Whk0RRJJq4WE9r/8SCRKhbNh?= =?us-ascii?Q?XN2AU36ItT6P2EmDqnIpsA60U+WG/sYc299XpfmH2rw8l8MjDOTSy35CiZWp?= =?us-ascii?Q?BSqbxoW05eNlKOYvwrFFRkKm792Ehsm6E2hX?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(82310400026)(36860700013)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:33:42.5792 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5fc26e38-0182-48ea-8adc-08dddb47f338 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B077.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9417 Content-Type: text/plain; charset="utf-8" Also add a config option for the same. High level design: While (1): Scan the slowtier pages belonging to VMAs of a task. Add to migation list. A separate thread: Migrate scanned pages to a toptier node based on heuristics. The overall code is influenced by khugepaged design. Signed-off-by: Raghavendra K T --- mm/Kconfig | 8 +++ mm/Makefile | 1 + mm/kscand.c | 163 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 172 insertions(+) create mode 100644 mm/kscand.c diff --git a/mm/Kconfig b/mm/Kconfig index 781be3240e21..d1e5be76a96e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -750,6 +750,14 @@ config KSM until a program has madvised that an area is MADV_MERGEABLE, and root has set /sys/kernel/mm/ksm/run to 1 (if CONFIG_SYSFS is set). =20 +config KSCAND + bool "Enable PTE A bit scanning and Migration" + depends on NUMA_BALANCING + help + Enable PTE A bit scanning of page. The option creates a separate + kthread for scanning and migration. Accessed slow-tier pages are + migrated to a regular NUMA node to reduce hot page access latency. + config DEFAULT_MMAP_MIN_ADDR int "Low address space to protect from user allocation" depends on MMU diff --git a/mm/Makefile b/mm/Makefile index 1a7a11d4933d..a16ef2ff3da1 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -97,6 +97,7 @@ obj-$(CONFIG_FAIL_PAGE_ALLOC) +=3D fail_page_alloc.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o obj-$(CONFIG_NUMA) +=3D memory-tiers.o +obj-$(CONFIG_KSCAND) +=3D kscand.o obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) +=3D page_counter.o diff --git a/mm/kscand.c b/mm/kscand.c new file mode 100644 index 000000000000..f7bbbc70c86a --- /dev/null +++ b/mm/kscand.c @@ -0,0 +1,163 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include "internal.h" + +static struct task_struct *kscand_thread __read_mostly; +static DEFINE_MUTEX(kscand_mutex); + +/* How long to pause between two scan cycles */ +static unsigned int kscand_scan_sleep_ms __read_mostly =3D 20; + +/* Max number of mms to scan in one scan cycle */ +#define KSCAND_MMS_TO_SCAN (4 * 1024UL) +static unsigned long kscand_mms_to_scan __read_mostly =3D KSCAND_MMS_TO_SC= AN; + +bool kscand_scan_enabled =3D true; +static bool need_wakeup; + +static unsigned long kscand_sleep_expire; + +static DECLARE_WAIT_QUEUE_HEAD(kscand_wait); + +/* Data structure to keep track of current mm under scan */ +struct kscand_scan { + struct list_head mm_head; +}; + +struct kscand_scan kscand_scan =3D { + .mm_head =3D LIST_HEAD_INIT(kscand_scan.mm_head), +}; + +static inline int kscand_has_work(void) +{ + return !list_empty(&kscand_scan.mm_head); +} + +static inline bool kscand_should_wakeup(void) +{ + bool wakeup =3D kthread_should_stop() || need_wakeup || + time_after_eq(jiffies, kscand_sleep_expire); + + need_wakeup =3D false; + + return wakeup; +} + +static void kscand_wait_work(void) +{ + const unsigned long scan_sleep_jiffies =3D + msecs_to_jiffies(kscand_scan_sleep_ms); + + if (!scan_sleep_jiffies) + return; + + kscand_sleep_expire =3D jiffies + scan_sleep_jiffies; + + /* Allows kthread to pause scanning */ + wait_event_timeout(kscand_wait, kscand_should_wakeup(), + scan_sleep_jiffies); +} +static void kscand_do_scan(void) +{ + unsigned long iter =3D 0, mms_to_scan; + + mms_to_scan =3D READ_ONCE(kscand_mms_to_scan); + + while (true) { + if (unlikely(kthread_should_stop()) || + !READ_ONCE(kscand_scan_enabled)) + break; + + if (kscand_has_work()) + msleep(100); + + iter++; + + if (iter >=3D mms_to_scan) + break; + cond_resched(); + } +} + +static int kscand(void *none) +{ + while (true) { + if (unlikely(kthread_should_stop())) + break; + + while (!READ_ONCE(kscand_scan_enabled)) { + cpu_relax(); + kscand_wait_work(); + } + + kscand_do_scan(); + + kscand_wait_work(); + } + return 0; +} + +static int start_kscand(void) +{ + struct task_struct *kthread; + + guard(mutex)(&kscand_mutex); + + if (kscand_thread) + return 0; + + kthread =3D kthread_run(kscand, NULL, "kscand"); + if (IS_ERR(kscand_thread)) { + pr_err("kscand: kthread_run(kscand) failed\n"); + return PTR_ERR(kthread); + } + + kscand_thread =3D kthread; + pr_info("kscand: Successfully started kscand"); + + if (!list_empty(&kscand_scan.mm_head)) + wake_up_interruptible(&kscand_wait); + + return 0; +} + +static int stop_kscand(void) +{ + guard(mutex)(&kscand_mutex); + + if (kscand_thread) { + kthread_stop(kscand_thread); + kscand_thread =3D NULL; + } + + return 0; +} + +static int __init kscand_init(void) +{ + int err; + + err =3D start_kscand(); + if (err) + goto err_kscand; + + return 0; + +err_kscand: + stop_kscand(); + + return err; +} +subsys_initcall(kscand_init); --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2062.outbound.protection.outlook.com [40.107.220.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EF8322576C for ; Thu, 14 Aug 2025 15:33:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185638; cv=fail; b=cAj881cwIaUnw4Wup+dAVdPTcafeLhbvcVL+PjJq/3lq2WQDjvc0XaStSvZ4fRjDMwFgq4bOQAQsTT9kOjfNFwnnFlJWeDbj2th7yUvMSmGtn7RkgY/sDZht8PpPpZ3PQcVzPGqMqD2DJHu6sL3qcBMSCgz2/cCh2X+VNSa+g94= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185638; c=relaxed/simple; bh=A3jtbuvNl9gYzDXvktGv1Ecq9IpTj+web81EA6B+AuM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kd8o8DixEa0foV8BxINvewhHGJgI66W2Cdrewx/Jx1L+2vZ8BJVLke68dKqEHGgh9MtA2idW17kkfJFRFpBpJ6hP3GsqsMi0pBbKhA5fvbuki15fN3Dp2M1Ozo+L39izluABM+mV+Q/L6HKQpdS6pZaz+N2px7begXu/Yb6Nfjc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=FU9e0qkk; arc=fail smtp.client-ip=40.107.220.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="FU9e0qkk" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CVV+kE2JtHYXWDAPfhxihO8glzcvGvR5Ke0rAb4S+k0hpOIsZjFFmSwfdNxg3bxPfpzkwC4pT3SBUWQ/c/rwZjDSgqSO/7zKHgm/yZAPf0CfDObhCtVz0nvZvWg2atjmSbNoft0vpsItGzVynIKL4g9mprTeRUfLoHbVb/B/vdTrzNv35v1pJDXC9tOxk+NUTbw4RzdmcCthwjsiMHCSr9O/RWszxeiZZrXSWpdaYmxsI9PTp/FFXtCFWCji7PhWnul2tyPFOfGYAnjb7BXozQn14Ce4Btx7F2HpgcupNceOnfNqPRPWPezT/z53nGEp+tUHan4jXDqcLrpW0aAwgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IeDzatv8nplj1egVMBXNqacjGynW1A8Evwh8y5ayuBw=; b=FsSqbVIIUU45kMgt6EXT0OZL5hYWEqImsX7H81w+6Wk77i03MIyFPAqCpK1efAtxk9Eguay+kihZJMvjq2IRg9M4zyjVF8txnS4OfLGFrdnFCy2/YkPHYXfYOg1zPoV/M8F+YnDAGUzlRd7lnLZCKIh0inDR85o3Ac/wAZjuNGl+araumnKLqWQICvrpNvZYcJjGE1FEyphzyrm6iIxZdyNEFw4+GSyXyvQXjuFQV2RDIesPO/ROxcNhqCzD+TVoQw2GfmQ9rG/QMuyn8lxHj8h1SH7m/FKC/rmivk55hl/r46TpE0KIZLT21hpMIf/8MhqvnkH5bfPIbKhjQo4yvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IeDzatv8nplj1egVMBXNqacjGynW1A8Evwh8y5ayuBw=; b=FU9e0qkkKeUv+PIykVOz2McPHxRau/cJKEZ0KnctIAOWK5zvG+Gy6jpwjqb0Dde1iXM0Ch5QKcgCytuTItjRxyiMDKQ3nBRGqxjfwVkYY0tM0zdQU9aQNQ6fi5r0T603qTBfVOKd3VSgs8UH7XDO2MhS/i5d08yoxfc+aNrQUt4= Received: from BL6PEPF00016411.NAMP222.PROD.OUTLOOK.COM (2603:10b6:22e:400:0:1004:0:9) by DS7PR12MB8321.namprd12.prod.outlook.com (2603:10b6:8:ec::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.14; Thu, 14 Aug 2025 15:33:53 +0000 Received: from BN3PEPF0000B075.namprd04.prod.outlook.com (2a01:111:f403:c803::4) by BL6PEPF00016411.outlook.office365.com (2603:1036:903:4::a) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:33:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B075.mail.protection.outlook.com (10.167.243.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:33:53 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:33:42 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 02/17] mm: Maintain mm_struct list in the system Date: Thu, 14 Aug 2025 15:32:52 +0000 Message-ID: <20250814153307.1553061-3-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B075:EE_|DS7PR12MB8321:EE_ X-MS-Office365-Filtering-Correlation-Id: 87876250-7ca4-4c4b-d80f-08dddb47f9b0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|1800799024|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?kwDzxvcldJrH2kaFkTRZmr6d+ntqAxhZMCWS8d91DK0MsmEfpeHeAoQ6b9Wv?= =?us-ascii?Q?gcsyfz2xfhtxPt2UHPmdo//GUNgBp6anq3NUecy6aaeofcvKykR8oAaDdA85?= =?us-ascii?Q?0fHD1J2Lzn7M0iYF1k/2Ky/Is8RupQmzVB+1iaggNHoJjYcWTbfk7rRFlK8Z?= =?us-ascii?Q?Jcb6X5zEFHn2mPqf8+XpUy5OY0OhdkQsB4F9IDs1U5ef+qGpds0M00YZGO7/?= =?us-ascii?Q?42mbcX4Ab3ZGfq3MRXSsnZl/fOcWq5vhKqIuI8pJ9nGgffQ8tMBynd/ZC3wA?= =?us-ascii?Q?UG5lWR5UBocOshOaynx+I3bB78QtejW8209ukAsQI8mue/BUOvF5rtCcV0t3?= =?us-ascii?Q?3+17Ozy9e+bzYeeIBLdF9u4YSAAHo2x5T2Cd4pBzCReUY4wO6xmhQVPlVnj/?= =?us-ascii?Q?o6rAre0s0l00g9JSeSimt7ygO/oVnOk+5/P+BXKwUy41wd4IxQ3IGyQZLdnK?= =?us-ascii?Q?bCqRTnsCXzOMSbX33OatMMfZCealLs+it9ArsgFb87wUQV37lm4RnUHMDPiX?= =?us-ascii?Q?Db4KjsSlmoCNmenc00hgF26mVDtJZid2zC4xmI0+H7q54AMvCXcqbV96OXaE?= =?us-ascii?Q?JLTEkzi3IlRImi0+FzoYP8WMMai9YSo1CujeuiDVtssDKaVsUo82QXzyrpHN?= =?us-ascii?Q?3nqkrKOFOdyJHLOaRzreNGSIwMXHWJgf2QmtK4zNel4G8R2CkdLBJzcQxDgb?= =?us-ascii?Q?e7O9cRLEuyajIzS9cH8PsiEE4Ko0vknN5dibkG3W/locrV/LgnT7xw/2jP9o?= =?us-ascii?Q?3oi1Ym4fqyHjpI7nf8Iyq2RJezTl5EY3wgo+TP00AOw9dimF7WG0QDP6XvIz?= =?us-ascii?Q?AldP5kNn7xIz6+rP4QL5uJ/FzFthSF6DcC0PFwAb+HO9NcJ+oajUtQpm6r5N?= =?us-ascii?Q?Z51CQ054+3KQpOaCr/bnQo+O9DIwx+IPgpdg0bJ2+ePyQDyun8ORoW/trjgH?= =?us-ascii?Q?UKomDxDW0TkqL356CKh1etdwauTlaEDwR1mlhVvcytAq91FhvUKBDitkxahk?= =?us-ascii?Q?iohwmHBOJ2bZH3nftxW1UbiPlUmjJiT74ISywcKfm1RedlF7C3YF/fPIFTfw?= =?us-ascii?Q?rGvKu6Cf/+7kmvH493dNWPO7jCKJjb2A3TZ8Y2jitf6UMzTggnPPGTSMxXss?= =?us-ascii?Q?4u4ZqIAO/qnnbDHkEzZrn2j9Pfk4opy0ioHyOqm/Gf5Ifk/apCW8C7eMiHRE?= =?us-ascii?Q?jp6D7aTcQYok/x6jhmd14fIf1LrLpwcL9GkFExrjR2JD0Iw483Y9Axm/IyB0?= =?us-ascii?Q?N8hlPXCVEEIGh76ukfecBZF+UlccSaGh2a3J1Ufh7Ld/7FJNMspdrPV79frM?= =?us-ascii?Q?BKNXVEXuKY7etn50tqNL3fXQCqfUw3GiDj751xMV78YVOMntR5EdQDjU7Fr1?= =?us-ascii?Q?odzNdmcEV2V61GdbeN0693HTL8FZimRsjqyu1u/QnY/3bok8150QX1LrscKm?= =?us-ascii?Q?3hChAgT64qkkI7dAQphMXPDKY/Tv6jFi2nuw+sY76bb3uqP/laGbYECvHS3c?= =?us-ascii?Q?tSpGbp0UFgng68Zt5iVFZFJeU0CObVT1l0mP?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:33:53.4326 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 87876250-7ca4-4c4b-d80f-08dddb47f9b0 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B075.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8321 Content-Type: text/plain; charset="utf-8" The list is used to iterate over all the mm and do PTE A bit scanning. mm_slot infrastructure is reused to aid insert and lookup of mm_struct. CC: linux-fsdevel@vger.kernel.org Suggested-by: Bharata B Rao Signed-off-by: Raghavendra K T --- include/linux/kscand.h | 30 +++++++++++++++ kernel/fork.c | 2 + mm/internal.h | 1 + mm/kscand.c | 86 ++++++++++++++++++++++++++++++++++++++++++ mm/mmap.c | 2 + mm/vma_exec.c | 3 ++ 6 files changed, 124 insertions(+) create mode 100644 include/linux/kscand.h diff --git a/include/linux/kscand.h b/include/linux/kscand.h new file mode 100644 index 000000000000..ef9947a33ee5 --- /dev/null +++ b/include/linux/kscand.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KSCAND_H_ +#define _LINUX_KSCAND_H_ + +#ifdef CONFIG_KSCAND +extern void __kscand_enter(struct mm_struct *mm); +extern void __kscand_exit(struct mm_struct *mm); + +static inline void kscand_execve(struct mm_struct *mm) +{ + __kscand_enter(mm); +} + +static inline void kscand_fork(struct mm_struct *mm, struct mm_struct *old= mm) +{ + __kscand_enter(mm); +} + +static inline void kscand_exit(struct mm_struct *mm) +{ + __kscand_exit(mm); +} +#else /* !CONFIG_KSCAND */ +static inline void __kscand_enter(struct mm_struct *mm) {} +static inline void __kscand_exit(struct mm_struct *mm) {} +static inline void kscand_execve(struct mm_struct *mm) {} +static inline void kscand_fork(struct mm_struct *mm, struct mm_struct *old= mm) {} +static inline void kscand_exit(struct mm_struct *mm) {} +#endif +#endif /* _LINUX_KSCAND_H_ */ diff --git a/kernel/fork.c b/kernel/fork.c index 1ee8eb11f38b..a13043de91b0 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -1116,6 +1117,7 @@ static inline void __mmput(struct mm_struct *mm) =20 uprobe_clear_state(mm); exit_aio(mm); + kscand_exit(mm); ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ exit_mmap(mm); diff --git a/mm/internal.h b/mm/internal.h index 6b8ed2017743..dd86efc54885 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -8,6 +8,7 @@ #define __MM_INTERNAL_H =20 #include +#include #include #include #include diff --git a/mm/kscand.c b/mm/kscand.c index f7bbbc70c86a..d5b0d3041b0f 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -7,12 +7,14 @@ #include #include #include +#include #include #include #include =20 #include #include "internal.h" +#include "mm_slot.h" =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); @@ -29,11 +31,23 @@ static bool need_wakeup; =20 static unsigned long kscand_sleep_expire; =20 +static DEFINE_SPINLOCK(kscand_mm_lock); static DECLARE_WAIT_QUEUE_HEAD(kscand_wait); =20 +#define KSCAND_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kscand_slots_hash, KSCAND_SLOT_HASH_BI= TS); + +static struct kmem_cache *kscand_slot_cache __read_mostly; + +/* Per mm information collected to control VMA scanning */ +struct kscand_mm_slot { + struct mm_slot slot; +}; + /* Data structure to keep track of current mm under scan */ struct kscand_scan { struct list_head mm_head; + struct kscand_mm_slot *mm_slot; }; =20 struct kscand_scan kscand_scan =3D { @@ -69,6 +83,12 @@ static void kscand_wait_work(void) wait_event_timeout(kscand_wait, kscand_should_wakeup(), scan_sleep_jiffies); } + +static inline int kscand_test_exit(struct mm_struct *mm) +{ + return atomic_read(&mm->mm_users) =3D=3D 0; +} + static void kscand_do_scan(void) { unsigned long iter =3D 0, mms_to_scan; @@ -109,6 +129,65 @@ static int kscand(void *none) return 0; } =20 +static inline void kscand_destroy(void) +{ + kmem_cache_destroy(kscand_slot_cache); +} + +void __kscand_enter(struct mm_struct *mm) +{ + struct kscand_mm_slot *kscand_slot; + struct mm_slot *slot; + int wakeup; + + /* __kscand_exit() must not run from under us */ + VM_BUG_ON_MM(kscand_test_exit(mm), mm); + + kscand_slot =3D mm_slot_alloc(kscand_slot_cache); + + if (!kscand_slot) + return; + + slot =3D &kscand_slot->slot; + + spin_lock(&kscand_mm_lock); + mm_slot_insert(kscand_slots_hash, mm, slot); + + wakeup =3D list_empty(&kscand_scan.mm_head); + list_add_tail(&slot->mm_node, &kscand_scan.mm_head); + spin_unlock(&kscand_mm_lock); + + mmgrab(mm); + if (wakeup) + wake_up_interruptible(&kscand_wait); +} + +void __kscand_exit(struct mm_struct *mm) +{ + struct kscand_mm_slot *mm_slot; + struct mm_slot *slot; + int free =3D 0; + + spin_lock(&kscand_mm_lock); + slot =3D mm_slot_lookup(kscand_slots_hash, mm); + mm_slot =3D mm_slot_entry(slot, struct kscand_mm_slot, slot); + if (mm_slot && kscand_scan.mm_slot !=3D mm_slot) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + } + + spin_unlock(&kscand_mm_lock); + + if (free) { + mm_slot_free(kscand_slot_cache, mm_slot); + mmdrop(mm); + } else if (mm_slot) { + mmap_write_lock(mm); + mmap_write_unlock(mm); + } +} + static int start_kscand(void) { struct task_struct *kthread; @@ -149,6 +228,12 @@ static int __init kscand_init(void) { int err; =20 + kscand_slot_cache =3D KMEM_CACHE(kscand_mm_slot, 0); + + if (!kscand_slot_cache) { + pr_err("kscand: kmem_cache error"); + return -ENOMEM; + } err =3D start_kscand(); if (err) goto err_kscand; @@ -157,6 +242,7 @@ static int __init kscand_init(void) =20 err_kscand: stop_kscand(); + kscand_destroy(); =20 return err; } diff --git a/mm/mmap.c b/mm/mmap.c index 09c563c95112..c9ffe65866de 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -1849,6 +1850,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, s= truct mm_struct *oldmm) if (!retval) { mt_set_in_rcu(vmi.mas.tree); ksm_fork(mm, oldmm); + kscand_fork(mm, oldmm); khugepaged_fork(mm, oldmm); } else { =20 diff --git a/mm/vma_exec.c b/mm/vma_exec.c index 2dffb02ed6a2..8576b377f7ad 100644 --- a/mm/vma_exec.c +++ b/mm/vma_exec.c @@ -128,6 +128,8 @@ int create_init_stack_vma(struct mm_struct *mm, struct = vm_area_struct **vmap, if (err) goto err_ksm; =20 + kscand_execve(mm); + /* * Place the stack at the largest stack address the architecture * supports. Later, we'll move this to an appropriate place. We don't @@ -151,6 +153,7 @@ int create_init_stack_vma(struct mm_struct *mm, struct = vm_area_struct **vmap, return 0; =20 err: + kscand_exit(mm); ksm_exit(mm); err_ksm: mmap_write_unlock(mm); --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2058.outbound.protection.outlook.com [40.107.100.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9B482264B8 for ; Thu, 14 Aug 2025 15:34:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185651; cv=fail; b=Sla0HdpuT1QHsfB7GiAEdcT5/SPaY4dJJ1Taoa+m3FbDJRYSLDREa/RnUJ0HWEdoOFCUQjFe0A5tXj5iDtYdHlszttmilJvICX2yA6AoZ1IJ/9/L4R4Z7zPVzlH32s6VCMUGBGB0H9H0bvsWPYCWtSV92xAVq0E3BWfYYdmnvTw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185651; c=relaxed/simple; bh=fpIn7DYWxD6/5L88coubosr+TA00MTCPGUP29NwtBwU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=c4qSo2FFhphagSIc0EFHwR7WdMAtE8mdJKEG1qHIPnzf2dE/lR5ME7gLT4fWiPqs4w6+4FZM1tXhAkkfm9nhCcAWZUKqehLIQZxkR5K8g59M1M9tkCsZ0UOveKypxregPY7+SXwlIPwziFJ78y+YRgLOlmoiClqgOKOrgkQpmhI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=IpTGdT3y; arc=fail smtp.client-ip=40.107.100.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="IpTGdT3y" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kjODYYIxwM7NftDrilUpDZSkm0VIv3/FDGJHL1P6Saw8NbisV1Tuop/APPjMRR1hLd/Mtw9cOSfpzlS/cunayn80VCjuuIilc/prrRL6sr3pUHVeizQx384dXmsio0ZIOIeGFWpqj90f7Z/sufGlAc0mwnjqDv1kUDal+wZ2N3g5nIBQ26M1t5uTyAnFjOg2XoR5kLJRUIfdwnHt7k/G4Nw5hWfXs13UlM+V6uKv/KZDXflH0J/i9Osg6lIaBTsow+ZseWlTk1TOB9q59yHQdtPG/gqcDo8KWXayOpTZZwdraw9Bj4uw9o7U+2RiYN2T6gEuXQrUpWNIvHQReOMRTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=D7cB5zU1ByDq5a++Mtl9WuTdBPv0fgnoAzM9mfcpqIg=; b=MJhZsDmnVDLBjavhBrYXTGFPXP+7hvjHgkgv13RG6VfORnjQ6iuSWdXVUeIN71EO2CfIFLMfyVfcl6t0OIuFiHe1TS9nqYB+55NsQBxfDfz12FirMfrjhDPpxG0UtIXbVgezvOrhVd/d5gHbI7wBeRiUlKrPSFrNKHg9OH9YyS++4T3qh7LuP/11doNeLaeYkvE/Z5naCpSIJF0Nrv4PvE+OUBMfHw2vS4cDwAjYIzQrZezSbEsVq+cP9NpqvKkafCzIE1ACVcuBVsvQOWpAOKmQJSkAwII0RqS7+MWArq4k/7UmcNljBGjpfaEaXr0agaFQ+OcNzc6dJr/3blAtOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D7cB5zU1ByDq5a++Mtl9WuTdBPv0fgnoAzM9mfcpqIg=; b=IpTGdT3y99vV05TTKMHVHwIsYrhV8wbBRAo9orT938jJzauSy9M3YQcX7FI2+mL+UwRyt129wJyDymWcsEjaLjOV+Es5Ji1wieif3M8ZXRCZq8YsEiKVaLTSrJJdOhkQNQBS1t/QUsNqvyAVZYwHdkFDdbkRPyMZfi/GyaNIr90= Received: from MN0PR04CA0018.namprd04.prod.outlook.com (2603:10b6:208:52d::11) by PH0PR12MB7816.namprd12.prod.outlook.com (2603:10b6:510:28c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.13; Thu, 14 Aug 2025 15:34:06 +0000 Received: from BN3PEPF0000B077.namprd04.prod.outlook.com (2603:10b6:208:52d:cafe::d0) by MN0PR04CA0018.outlook.office365.com (2603:10b6:208:52d::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:34:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B077.mail.protection.outlook.com (10.167.243.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:34:05 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:33:52 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 03/17] mm: Scan the mm and create a migration list Date: Thu, 14 Aug 2025 15:32:53 +0000 Message-ID: <20250814153307.1553061-4-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B077:EE_|PH0PR12MB7816:EE_ X-MS-Office365-Filtering-Correlation-Id: 20d75f28-a20c-45c5-ca67-08dddb480121 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Ou/yncPx/fCmioDGh51mshp6fmT96HwmtNojBMrjL/3Fg9lht6cVfNvAvw9a?= =?us-ascii?Q?V7Us184xzfKEdSQr8HcPPkAGrw2XWSyoGgwvL0hqsNo7YSJbF+73/mXZT9VC?= =?us-ascii?Q?7vaT4tNIWGOQ9PRo83J9huOcwF/b8SwruAdfrqFL55lyZSz5R2oQ84GmprJq?= =?us-ascii?Q?sQ+rap0l/LI/FUdnKZnoVJunFDHN0tAFwJlzDd1kDERvgJBIDq6UdtrKl2YH?= =?us-ascii?Q?dUCfnoCI40BSnYwqmOM8bS2aF1xLcGMMOiYT+zFolMsNDAujhynrm/nWbc6i?= =?us-ascii?Q?aOclMmBCTf2nOn+RJk6xenczStnOx8363MGctI1ANBD2RTPNTrsio3arbTiH?= =?us-ascii?Q?KwRLWACrad7rSFxDGNURVhyEduLYLXG1vBp6/4C3sT3sGLWkeM8QgWCHJZBQ?= =?us-ascii?Q?vmW6PvYzr2Z5pJlvCYRSqYnE2vfFmZbQhAJhy2pW5FPPE7jP40F5Ob4WyNqN?= =?us-ascii?Q?YEjSh6rvmuzERU0cU2CkWN7vrUQNgcJriyoRjRS2zN07Mziu8GOm3naLMtXD?= =?us-ascii?Q?66R7S7WWpBIaYQs2StifPbsJWsxnaBEO1aTaoUQwDQaKP/kNOvjZSfNKUHZW?= =?us-ascii?Q?WURuuoMAuXTJDP3OOQYer3wRRhJca3vgcjH0q/ZSP92PKf5Mh5qCLdDjvmFz?= =?us-ascii?Q?SZtKe5viVMyDfy4rbqk253MUfDasEzLZepoV/WgArMkZRQwBxgdDNp6i8znA?= =?us-ascii?Q?pNj1iFb3LG1dyMpPrQ5CgQQ9cKZ6xSnzENqF2WkvYLgQd1QivF5+g9k9LG+i?= =?us-ascii?Q?/eAFTHKZOBrDZ1LnHYRkYcSfcfNxbUJl1FdMNy04vlAttzvwYVxFBYCKP3+8?= =?us-ascii?Q?HfxoKK5rmFARxSy0kwRFdNLvp3bOCVgQsS7Ydg8JFukKNCPlIfBkIRNkub1J?= =?us-ascii?Q?KZssPzXyGyit5gcLBzXYMZ+952t/a63SfHm+EIoGtQyBhETnpdpBJqS0J2Eh?= =?us-ascii?Q?wnkUjd1DCPlTgnHg3jt5HxindbxLZva2bhEP48JhiBzZYnb3LsCzBPN3dqsx?= =?us-ascii?Q?5+L8YeGQnQmM4I6N3qwplfhU0Ehg7lA15fw+zM+grKgzWF9tKjM/6tkj0q+Y?= =?us-ascii?Q?A1hmU6lEkawaB7tIS7yjOVk++KaMhDcjSJs4swH/FI9NAiLOt3Z5lHkBTncq?= =?us-ascii?Q?fh6yN6mZ7AZdXEmnn00aSXvkLH2Fi/wHjKYuR17FXdUEpFNg6U0S8wEeumtq?= =?us-ascii?Q?cJqYGGhD1jsZtPHh6UnIQGpvuszhGBUacGAfWdd5PEXgSItTIJ9XCL3cX9N8?= =?us-ascii?Q?5OO4W1a1nQ+oOLBxHtGND4DNGzjn2jhVlUTXNH4XAZFnoJNjUwglnq6YYW+S?= =?us-ascii?Q?zJfku1cwY++dWzS0x4aS/Ix1KKQdEO8oIhfhFxKG5Sgxay/I00NS+fZlCuD2?= =?us-ascii?Q?PPvHCJ5OVor708ak85IjH8b6XPVB55+95OW2fumPUGo5C5WV1sGzn5un82Zf?= =?us-ascii?Q?QK1bjmF63KgctVwpJ4h/FEKINgKUbzmAp2fCvLIpGd4z+wFwlhzetWa1memc?= =?us-ascii?Q?PaSObciRvly/5mDdOrPRWMDjFhNq3mrHwSk3?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:34:05.9194 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 20d75f28-a20c-45c5-ca67-08dddb480121 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B077.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7816 Content-Type: text/plain; charset="utf-8" Since we already have the list of mm_struct in the system, add a module to scan each mm that walks VMAs of each mm_struct and scan all the pages associated with that. In the scan path: Check for the recently acccessed pages (PFNs) belonging to slowtier nodes. Add all those to a list. Signed-off-by: Raghavendra K T --- mm/kscand.c | 321 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 320 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index d5b0d3041b0f..1d883d411664 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -4,10 +4,18 @@ #include #include #include +#include +#include +#include +#include +#include +#include #include #include #include #include +#include +#include #include #include #include @@ -18,6 +26,11 @@ =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); +/* + * Total VMA size to cover during scan. + */ +#define KSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) +static unsigned long kscand_scan_size __read_mostly =3D KSCAND_SCAN_SIZE; =20 /* How long to pause between two scan cycles */ static unsigned int kscand_scan_sleep_ms __read_mostly =3D 20; @@ -42,6 +55,8 @@ static struct kmem_cache *kscand_slot_cache __read_mostly; /* Per mm information collected to control VMA scanning */ struct kscand_mm_slot { struct mm_slot slot; + long address; + bool is_scanned; }; =20 /* Data structure to keep track of current mm under scan */ @@ -54,6 +69,29 @@ struct kscand_scan kscand_scan =3D { .mm_head =3D LIST_HEAD_INIT(kscand_scan.mm_head), }; =20 +/* + * Data structure passed to control scanning and also collect + * per memory node information + */ +struct kscand_scanctrl { + struct list_head scan_list; + unsigned long address; +}; + +struct kscand_scanctrl kscand_scanctrl; +/* Per folio information used for migration */ +struct kscand_migrate_info { + struct list_head migrate_node; + unsigned long pfn; + unsigned long address; +}; + +static bool kscand_eligible_srcnid(int nid) +{ + /* Only promotion case is considered */ + return !node_is_toptier(nid); +} + static inline int kscand_has_work(void) { return !list_empty(&kscand_scan.mm_head); @@ -84,11 +122,277 @@ static void kscand_wait_work(void) scan_sleep_jiffies); } =20 +static inline bool is_valid_folio(struct folio *folio) +{ + if (!folio || !folio_mapped(folio) || !folio_raw_mapping(folio)) + return false; + + if (folio_test_unevictable(folio) || folio_is_zone_device(folio) || + folio_maybe_mapped_shared(folio)) + return false; + + return true; +} + + +static bool folio_idle_clear_pte_refs_one(struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr, + pte_t *ptep) +{ + bool referenced =3D false; + struct mm_struct *mm =3D vma->vm_mm; + pmd_t *pmd =3D pmd_off(mm, addr); + + if (ptep) { + if (ptep_clear_young_notify(vma, addr, ptep)) + referenced =3D true; + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + if (!pmd_present(*pmd)) + WARN_ON_ONCE(1); + if (pmdp_clear_young_notify(vma, addr, pmd)) + referenced =3D true; + } else { + WARN_ON_ONCE(1); + } + + if (referenced) { + folio_clear_idle(folio); + folio_set_young(folio); + } + + return true; +} + +static void page_idle_clear_pte_refs(struct page *page, pte_t *pte, struct= mm_walk *walk) +{ + bool need_lock; + struct folio *folio =3D page_folio(page); + unsigned long address; + + if (!folio_mapped(folio) || !folio_raw_mapping(folio)) + return; + + need_lock =3D !folio_test_anon(folio) || folio_test_ksm(folio); + if (need_lock && !folio_trylock(folio)) + return; + address =3D vma_address(walk->vma, page_pgoff(folio, page), compound_nr(p= age)); + VM_BUG_ON_VMA(address =3D=3D -EFAULT, walk->vma); + folio_idle_clear_pte_refs_one(folio, walk->vma, address, pte); + + if (need_lock) + folio_unlock(folio); +} + +static int hot_vma_idle_pte_entry(pte_t *pte, + unsigned long addr, + unsigned long next, + struct mm_walk *walk) +{ + struct page *page; + struct folio *folio; + struct mm_struct *mm; + struct vm_area_struct *vma; + struct kscand_migrate_info *info; + struct kscand_scanctrl *scanctrl =3D walk->private; + int srcnid; + + scanctrl->address =3D addr; + pte_t pteval =3D ptep_get(pte); + + if (!pte_present(pteval)) + return 0; + + if (pte_none(pteval)) + return 0; + + vma =3D walk->vma; + mm =3D vma->vm_mm; + + page =3D pte_page(*pte); + + + folio =3D page_folio(page); + folio_get(folio); + + if (!is_valid_folio(folio)) { + folio_put(folio); + return 0; + } + folio_set_idle(folio); + page_idle_clear_pte_refs(page, pte, walk); + srcnid =3D folio_nid(folio); + + + if (!folio_test_lru(folio)) { + folio_put(folio); + return 0; + } + + if (!kscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return 0; + } + if (!folio_test_idle(folio) && + (folio_test_young(folio) || folio_test_referenced(folio))) { + + /* XXX: Leaking memory. TBD: consume info */ + + info =3D kzalloc(sizeof(struct kscand_migrate_info), GFP_NOWAIT); + if (info && scanctrl) { + info->pfn =3D folio_pfn(folio); + info->address =3D addr; + list_add_tail(&info->migrate_node, &scanctrl->scan_list); + } + } + + folio_put(folio); + return 0; +} + +static const struct mm_walk_ops hot_vma_set_idle_ops =3D { + .pte_entry =3D hot_vma_idle_pte_entry, + .walk_lock =3D PGWALK_RDLOCK, +}; + +static void kscand_walk_page_vma(struct vm_area_struct *vma, struct kscand= _scanctrl *scanctrl) +{ + if (!vma_migratable(vma) || !vma_policy_mof(vma) || + is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + return; + } + if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) =3D=3D (VM_READ= ))) + return; + + if (!vma_is_accessible(vma)) + return; + + walk_page_vma(vma, &hot_vma_set_idle_ops, scanctrl); +} + static inline int kscand_test_exit(struct mm_struct *mm) { return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +static void kscand_collect_mm_slot(struct kscand_mm_slot *mm_slot) +{ + struct mm_slot *slot =3D &mm_slot->slot; + struct mm_struct *mm =3D slot->mm; + + lockdep_assert_held(&kscand_mm_lock); + + if (kscand_test_exit(mm)) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + + mm_slot_free(kscand_slot_cache, mm_slot); + mmdrop(mm); + } +} + +static unsigned long kscand_scan_mm_slot(void) +{ + bool next_mm =3D false; + bool update_mmslot_info =3D false; + + unsigned long vma_scanned_size =3D 0; + unsigned long address; + + struct mm_slot *slot; + struct mm_struct *mm; + struct vm_area_struct *vma =3D NULL; + struct kscand_mm_slot *mm_slot; + + + spin_lock(&kscand_mm_lock); + + if (kscand_scan.mm_slot) { + mm_slot =3D kscand_scan.mm_slot; + slot =3D &mm_slot->slot; + address =3D mm_slot->address; + } else { + slot =3D list_entry(kscand_scan.mm_head.next, + struct mm_slot, mm_node); + mm_slot =3D mm_slot_entry(slot, struct kscand_mm_slot, slot); + address =3D mm_slot->address; + kscand_scan.mm_slot =3D mm_slot; + } + + mm =3D slot->mm; + mm_slot->is_scanned =3D true; + spin_unlock(&kscand_mm_lock); + + if (unlikely(!mmap_read_trylock(mm))) + goto outerloop_mmap_lock; + + if (unlikely(kscand_test_exit(mm))) { + next_mm =3D true; + goto outerloop; + } + + VMA_ITERATOR(vmi, mm, address); + + for_each_vma(vmi, vma) { + kscand_walk_page_vma(vma, &kscand_scanctrl); + vma_scanned_size +=3D vma->vm_end - vma->vm_start; + + if (vma_scanned_size >=3D kscand_scan_size) { + next_mm =3D true; + /* TBD: Add scanned folios to migration list */ + break; + } + } + + if (!vma) + address =3D 0; + else + address =3D kscand_scanctrl.address + PAGE_SIZE; + + update_mmslot_info =3D true; + + if (update_mmslot_info) + mm_slot->address =3D address; + +outerloop: + /* exit_mmap will destroy ptes after this */ + mmap_read_unlock(mm); + +outerloop_mmap_lock: + spin_lock(&kscand_mm_lock); + WARN_ON(kscand_scan.mm_slot !=3D mm_slot); + + /* + * Release the current mm_slot if this mm is about to die, or + * if we scanned all vmas of this mm. + */ + if (unlikely(kscand_test_exit(mm)) || !vma || next_mm) { + /* + * Make sure that if mm_users is reaching zero while + * kscand runs here, kscand_exit will find + * mm_slot not pointing to the exiting mm. + */ + if (slot->mm_node.next !=3D &kscand_scan.mm_head) { + slot =3D list_entry(slot->mm_node.next, + struct mm_slot, mm_node); + kscand_scan.mm_slot =3D + mm_slot_entry(slot, struct kscand_mm_slot, slot); + + } else + kscand_scan.mm_slot =3D NULL; + + if (kscand_test_exit(mm)) { + kscand_collect_mm_slot(mm_slot); + goto end; + } + } + mm_slot->is_scanned =3D false; +end: + spin_unlock(&kscand_mm_lock); + return 0; +} + static void kscand_do_scan(void) { unsigned long iter =3D 0, mms_to_scan; @@ -101,7 +405,7 @@ static void kscand_do_scan(void) break; =20 if (kscand_has_work()) - msleep(100); + kscand_scan_mm_slot(); =20 iter++; =20 @@ -148,6 +452,7 @@ void __kscand_enter(struct mm_struct *mm) if (!kscand_slot) return; =20 + kscand_slot->address =3D 0; slot =3D &kscand_slot->slot; =20 spin_lock(&kscand_mm_lock); @@ -175,6 +480,12 @@ void __kscand_exit(struct mm_struct *mm) hash_del(&slot->hash); list_del(&slot->mm_node); free =3D 1; + } else if (mm_slot && kscand_scan.mm_slot =3D=3D mm_slot && !mm_slot->is_= scanned) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + /* TBD: Set the actual next slot */ + kscand_scan.mm_slot =3D NULL; } =20 spin_unlock(&kscand_mm_lock); @@ -224,6 +535,12 @@ static int stop_kscand(void) return 0; } =20 +static inline void init_list(void) +{ + INIT_LIST_HEAD(&kscand_scanctrl.scan_list); + init_waitqueue_head(&kscand_wait); +} + static int __init kscand_init(void) { int err; @@ -234,6 +551,8 @@ static int __init kscand_init(void) pr_err("kscand: kmem_cache error"); return -ENOMEM; } + + init_list(); err =3D start_kscand(); if (err) goto err_kscand; --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2063.outbound.protection.outlook.com [40.107.220.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02E6A23C397 for ; Thu, 14 Aug 2025 15:34:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185659; cv=fail; b=iVqCschPYojrMBqljpN72MIO4H558wcP89XbuqeZYXu++o7rtPeIL4v2xvL+kIuF+hvGM2xQu5P/LDWWO/W8r+m61ge/h+RcQPoNqJ+HJniHWOisqVjjrKjoP3ngK2p7JRJHW2mw84cWhMg9lCK+kKFYUnZ0WzoFBWns/kvlphg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185659; c=relaxed/simple; bh=iAf9ugzP/ndFqsUtIJq9llrrTSdpsL0RraaHlI4sh1U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XBMMlfhRXJyFhn+osTsV2pcygsMdOC7cywvPLOSDr35rSt4ROcJtVS3TCcXMZJctQjwdg2GTLaAV5eENBFyWh/1V23g0MzshYi4ktdlPQRo7cJgiQzbnlaYD5g09R1o9uJcqSyNt2lL1dJA11VUTxi/qzJOQDjuWZvJ2f6aI4zw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=TzcMM+zj; arc=fail smtp.client-ip=40.107.220.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="TzcMM+zj" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GiesXj6Vt15HwW8viwr+36okH4RmOlARHzF/lAV7QcpIK+oykFwvdWhaAlIQAC9IyfcWqmd3lLGzf/J2YNOhbjMC1ZuPTHwN6b0CRY1ou4DMgPrUA2dkdNwSUnVtjyeW7z91+Oj7RUOFu/EgiOnopDWhDqCjAxSl8/ihxQj3402ZlBdYzMeFjqZQ9R0vaNcJYaKm3fYWp3+nrghp6ctrlgk5jRYiKlke/6VRJ52/u7AqnOd83/DBIf4l+763ilZY9Ze1AgjQo92eLxnT1ytqJZGzIMtmGVyFlRWliFZDzqJn3S/OhMk+mvV2JY19ntILRPn6br+B0bGOGy976WzllA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MqSNVJKUarTtk4qidv7ivgZ7qR/WuPYJzNjNjEwUemA=; b=D9J3jVrft9yXHV9f+NZlEc6rTo49bacxWeRD07HMEqSLeNICOMB0AB4YrM039EE98R+/55FSQspg6xTvoHrxwptmtWtlIlR4WodDWyAfMvkCmvSSoeAzhyQNB8mWeqVE2tilipMi5QiqdvZQfOY3eXW2tcOlx6uqAXkh9sKZAdeZe7tjlFNgVcNjr1yPSbpFrAA4cbpj5PoQhyQbRl1PqzhHr13hFGT37z8mQb3v9I3fHxB1dVtqAxehc35fVftI3xCR+oxWebjzut4u951vdvr8JhSOtvKiJ+wjVbyr10gQV4awhSpPgC427rsfuJrv1wF6STYtfARdn1SGCGr62g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MqSNVJKUarTtk4qidv7ivgZ7qR/WuPYJzNjNjEwUemA=; b=TzcMM+zj7gjlJuVeAXy9JYPHrebnaW9ZgV7cGOfvy2GXwshmVYdJNfE0m08oQV1fKzRY2qhFzt4c2cY27JGvKswvRu85KXtk5jhO7KUEa/7BKsXkrKXevYAsPAoCyfyTPjTqktrW2JzI8StsapER/fmgb6RDomWUgNcKShO0WOs= Received: from MN0PR04CA0021.namprd04.prod.outlook.com (2603:10b6:208:52d::32) by CH3PR12MB8481.namprd12.prod.outlook.com (2603:10b6:610:157::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.16; Thu, 14 Aug 2025 15:34:14 +0000 Received: from BN3PEPF0000B077.namprd04.prod.outlook.com (2603:10b6:208:52d:cafe::eb) by MN0PR04CA0021.outlook.office365.com (2603:10b6:208:52d::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:34:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B077.mail.protection.outlook.com (10.167.243.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:34:14 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:34:03 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 04/17] mm/kscand: Add only hot pages to migration list Date: Thu, 14 Aug 2025 15:32:54 +0000 Message-ID: <20250814153307.1553061-5-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B077:EE_|CH3PR12MB8481:EE_ X-MS-Office365-Filtering-Correlation-Id: abce2e1e-1336-42d1-0e5b-08dddb480653 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?E2Z9490f9ZjIk1Tmo9uHh3dUPiEkAPEmEpWzlAjLZbtFf0kefrowYe9+uRgO?= =?us-ascii?Q?75/+aGxffofyj861wjNRpkTOaXE2nsl3A8FO0ET3x0zhfrUezPe8SP0tZLUP?= =?us-ascii?Q?eJaGmDeKydl53Z9GdMLKufedOthNptD362eTZxgn2P0wfnKSOufVWjopYWkK?= =?us-ascii?Q?4FU67mR2AOqdefyp4r9yX10qni57E8kvtWmFPQzRbDkWgNgQOsC8qIUpQLe/?= =?us-ascii?Q?CH9/A59mzXbJSGtfb6/hU46kIyH8civCBTtrZhO5AEdcLCAg9Sf66IbpyPm9?= =?us-ascii?Q?aNYZkDr48VhALZqDigVbVB5VtOAVuThAbAKb9q+HwdAHHZxRfb9MF7Ks7a+q?= =?us-ascii?Q?4BMKN9f/e0VyWTyrV9tQ5qXI175mhKIg9PulCGZACHeD3VvQWPQ6EMjZM1Q3?= =?us-ascii?Q?L4UBEt5qxUlYwYfHvzIP1z1/tEJ0QRRugFGfhoR8nl0I/qSZk5BljXXwQhal?= =?us-ascii?Q?iaea52Qbzn/YBDxIT/CWIl/fB8ckFhznn1qSW8X7EkzKVk/pup6p1k12LJcl?= =?us-ascii?Q?ps5EPv8wHhynpIPmpcS2TCHVSEnbLxls/HEXYB2dim0nQSihJq/NNvQWA3LC?= =?us-ascii?Q?T75QBIIwXlORgrYKg3tb3C/XpgOFaMPerkVS2yKoH+mMaTFIdDxbSS8LsxYR?= =?us-ascii?Q?0t34nsml6ee0vfSPT5mxI1fBkeO0ZhzZTx6Ig6Dr0eme0OcUyhQmYuhwwx17?= =?us-ascii?Q?DO2XeCcDQvtsycFtGWOhZatUUrRx3t6Vmhb7+WW6O8QFZTcTr94JaDu4Jkj7?= =?us-ascii?Q?whkDikCiXGGp+FIZCitiuVFjucIV2tiExFVrZHspP/vPJFgnlxqUd35q3Lh3?= =?us-ascii?Q?NRswZMul8gl/2IryNvj1ORsafh1wOewwCAcwRh+7hHEJnCPIAIJ71M10hAVB?= =?us-ascii?Q?LEqHC+zqgxJPukrjyH8BaR7BXKP2m7N2Lwn5fHm2bpXkFVtJ0DU9v9RN3UQ+?= =?us-ascii?Q?KVIJl67eynCW035vMYDUG2ngF0xpJyXjRZdR75N1JHm6ZYs3EN7hI9/HTmeL?= =?us-ascii?Q?P9cJzLuT79zKJz5CIE2g1GX8NCId3wmxb9qDPlDxKa6gNIo97LHC2rg0DW70?= =?us-ascii?Q?rIpBpumLrm8Qut1mNGLYc4FtsNt3X6DLiWa/2AOalCZamLdANyV1gK/pj35H?= =?us-ascii?Q?lrf5zQBTehpPBhEKjTFu8MzeiLI2B8mdyiqzkR+Yc8uKuFvkXeFiwB/fbVUt?= =?us-ascii?Q?tPC60TPfNXUHGPqLO5snDTpfM1VRH1bY8E9VexwT5KzqZ+psKHhWNJoyXq2g?= =?us-ascii?Q?xe1Rki5uth9a8o/U9mnMcGq81I/SfHMaPxScjr1jpbBhlqBxWzweaRrCCHhn?= =?us-ascii?Q?OY6V5eMJY/07LBuVSBl3UA0durVw2HA8TvOQ0ZhE5LWftF2T0sTVCctSeQFG?= =?us-ascii?Q?ymgQFIA/aGShFx9jA51pfgDKJFE0ttc5k9MU4S5wDWca0idcIlGbo8RAmCKR?= =?us-ascii?Q?Lb8erM6t/aB27MT0/F5clVmfP+xRHbTlgA0/3EBeWdxflxx7jZvH7ueOP9af?= =?us-ascii?Q?gcIImjwaoqZtGohrzuEdNi6BakRjoqQn//Se?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:34:14.6290 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: abce2e1e-1336-42d1-0e5b-08dddb480653 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B077.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8481 Content-Type: text/plain; charset="utf-8" Previously all pages, accessed once are added. Improve it by adding those that are accessed second time. This logic is closer to current NUMAB implementation of spotting hot pages. Signed-off-by: Raghavendra K T --- mm/kscand.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index 1d883d411664..7552ce32beea 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -196,6 +196,7 @@ static int hot_vma_idle_pte_entry(pte_t *pte, struct kscand_migrate_info *info; struct kscand_scanctrl *scanctrl =3D walk->private; int srcnid; + bool prev_idle; =20 scanctrl->address =3D addr; pte_t pteval =3D ptep_get(pte); @@ -219,6 +220,7 @@ static int hot_vma_idle_pte_entry(pte_t *pte, folio_put(folio); return 0; } + prev_idle =3D folio_test_idle(folio); folio_set_idle(folio); page_idle_clear_pte_refs(page, pte, walk); srcnid =3D folio_nid(folio); @@ -233,7 +235,7 @@ static int hot_vma_idle_pte_entry(pte_t *pte, folio_put(folio); return 0; } - if (!folio_test_idle(folio) && + if (!folio_test_idle(folio) && !prev_idle && (folio_test_young(folio) || folio_test_referenced(folio))) { =20 /* XXX: Leaking memory. TBD: consume info */ --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2081.outbound.protection.outlook.com [40.107.220.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5217617AE1D for ; Thu, 14 Aug 2025 15:34:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185672; cv=fail; b=Jv/Q4iila+lf1VkSo1LWITEBTlUNFy4H15m6pwlQaOOMXrug/COC6lC2Wp8swelxd+hDbdNOv7yfVUgnJxDaPlRrh3tMixc7AY3r4m92awSNrQEOmZmXpVP+FynnzxpsNIjTBChtMi2E9FTz5lj2y6nnI27mtCW5rptNsGbz0HI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185672; c=relaxed/simple; bh=tNB6puLZUATBUCN/EA1kKkFaLOe95pDtLDXQK2yw2TY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Xuavqf3cClxx+telmp19UhHYIVAj6gSMBd1UoMXU2kQgBQOmx3CPT8XfiB4I76gmBpZ82fXvcbWTq/Ddb1NJNECNmiP8IJrDDAeV0w0DoEHshOeR5XUi7dU7KjK3TWO/dU/yAOUah2CC7JtV00tJDiV6y+mr9QELWzKC8TC3oaQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=YgA16R4/; arc=fail smtp.client-ip=40.107.220.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="YgA16R4/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=H9Ey9ore+VT8m/tiRLdr9s9RjnR+NuNvTofr4/ti2lm2Q8op4LtHg6pt47S0wbirTdJfU8XogSmOyquZm8RLoadrffkMA7BSEdc6uLEod07s7k4cIxHi5qJE3krLUI4arpM9sJY6VLtrtPvpNBn+H+lQES8hshpcb8WNIyTdTXXmaJCwadxY4tdzpIXCgyetgW6wgPWmc3AAA9YrUQ/bhH3KLVvklEmBTZjH6RR/P9WDvxidzX2YNCqMUDk1hJfOzOxSl6h2ePcqubm8+dpDcEJ2xi5jPmmIo61RUlZeIuQAK7MKoQQjA7UftTdVhz4DdGIUTjsKpq3MkzoXmhKZbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/6yUiVOmVxuU72JxpsJZhSRgH6o0kl8/xMLa+WIVrPk=; b=nBCUIfIrymmXSYygrDNuwMAKp12HR5QuDjCKUnUxDT5X9tTFA7cjRBhPMtltkdpR3cwrz/mtgT+o5U0HbPFdIe3cyGZ/2LBNJ5XYpM1N5vSjir8yYxVPTar6T0hhkm9/pURWxWzw2bYhVhocmOxSHbWjDayel3vkvLHyHijTyQmZKd5Q15eG9ndrHUMnyYJjJtIvGg4ZkWCJ0VOBv8K8pV1KdRjTRg42CSSEBP2u3+dC+ZYR3hB773+WgPgpMYPVt9HRe0juOpFfXMX/D3Svy6ywlB7ligyuwPBJJLLcepLO1isFrnp1+Bzj0HHa0MKajsvVoPm8q2aMjzCtG3ATZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/6yUiVOmVxuU72JxpsJZhSRgH6o0kl8/xMLa+WIVrPk=; b=YgA16R4/mFSD+z0Vn0U3EaQ0aJmWod5efqhDPWiK8Jjmi5Yfa48AycQX9wtV68Nga6Ji9g1MAAtlNVFVC+VnOmZUdkRjrLpFm8RgbsKMxnVY+p49cLyLDlznKgMlrhoQ28kwkqvJ1ojbOIV0BT2Sx7r8gq3Mw0qNh0Jj1JQd1xo= Received: from BN9PR03CA0475.namprd03.prod.outlook.com (2603:10b6:408:139::30) by DS0PR12MB7748.namprd12.prod.outlook.com (2603:10b6:8:130::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.15; Thu, 14 Aug 2025 15:34:27 +0000 Received: from BN3PEPF0000B072.namprd04.prod.outlook.com (2603:10b6:408:139:cafe::84) by BN9PR03CA0475.outlook.office365.com (2603:10b6:408:139::30) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.18 via Frontend Transport; Thu, 14 Aug 2025 15:34:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B072.mail.protection.outlook.com (10.167.243.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:34:26 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:34:14 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 05/17] mm: Create a separate kthread for migration Date: Thu, 14 Aug 2025 15:32:55 +0000 Message-ID: <20250814153307.1553061-6-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B072:EE_|DS0PR12MB7748:EE_ X-MS-Office365-Filtering-Correlation-Id: 0ecad30c-670c-40ac-5238-08dddb480db2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Py1TkQi0/UjwyEVgoUhf3vHIx0vjwvzOW5sVwZAmRiVxQyrawxkHrAytR5d5?= =?us-ascii?Q?KfGLd2kPDyXQKZR5DFJ2cHXmfrwaC4RrFe9EGYV82enaY6SFTpvGjko1G4nl?= =?us-ascii?Q?o/X+2rhw/9Xhgm0dibSEL5FXN/BjVR17LdGy7Huc1RJVpFviFkRlOWnHJvlA?= =?us-ascii?Q?KZ0yEiwbHJGkdO83NR1LIj9AKGQnBR6LGvJ8g7tEZ6sqoCMB/qAjqpkJv7lV?= =?us-ascii?Q?XDvGLeqBJXjbbvMwdUTT3XqALW7Egy8zyUG8TvDs9wMKYMDQ7cx3TXQ3G2Gg?= =?us-ascii?Q?6gKd4sUabzB54q59VAZ7MiQY2ecLIRvz+IKpg3fmHz1CYToLfBu48O8qIdVE?= =?us-ascii?Q?cgI+VTwhgRqG3xojfI8/M5xB/Q+d1fMo747sjTn4jsB6KBMsJ50Lc/nAza5S?= =?us-ascii?Q?WCXOrv0n0cCkwg0Nzz7ZDi8jfiWGoZke18UOiMtuZR0vZLA1/hFKztA4TNq8?= =?us-ascii?Q?1MGHRVppG0zCqWLPjGhken5ke75AZmysJasyRL/thLEMwx3lWgg1zKHHJZro?= =?us-ascii?Q?ep3EzpmFeOVr0P3w5iwCxdP+n7K7Nv89uihw6QoCl16jPxSqA63yJ7se7+CK?= =?us-ascii?Q?wjE1spFHZPcmXWjUMA2//zUhPTiOlkiE+r7DXg+xmIniy9AflkqKNfLYvbHH?= =?us-ascii?Q?vPTFn2DQlJtA+Av7dFny9fkt0uC7xC/023M5eOawJ53JkD/VTrXiiwQzTQ4W?= =?us-ascii?Q?v4pe9PLb6DhFPV29H+V1n+oDqp1/yN16Rk5cGC/F1hYu4txNKGSOoKzLTVc6?= =?us-ascii?Q?WbeQxMP+Y6CFd9rBH13vNHxtlCu8657ipn/D1ssbUIo0N81oIpPGf5BSsHPo?= =?us-ascii?Q?D3oQGzdBmXGBAzHWY9YX6KC5HA1ZcuUIWxsULM56Y+kUkFmaLjcsHlR3F0L4?= =?us-ascii?Q?vuJzNNbVW9sRwyVIyxPm6ROEVbpd6bv7rnxSFRebXejyCsPnVeebE5j1uQhl?= =?us-ascii?Q?ABLr+2NNcDdOlIub5jtOjgUOwlS5MEUin1QtTmFo54PlZxk8QY7LMqBR/6Cj?= =?us-ascii?Q?uStOqulQc+UgexNIfZK65DXeZyNt9DjkGKRwBgsCMWyELeDgnrEo6lZ/xLq4?= =?us-ascii?Q?8Rd02PCFEeGlK0ei2NYxKZbQTQK7PzhXuqUorka6/qkdCxn/kwYtL74cLE1E?= =?us-ascii?Q?hk3GOsr/LauwrugO8DQE4VntyaNph7OdBVHI622u8qXicfOcIDEh0FM41nl7?= =?us-ascii?Q?YSzbw0+EtgCBDNv1yhJOjh0Cgzr851DXR5QlwqMNkJctZ/3S1Sk9BAK2YnK2?= =?us-ascii?Q?g89nlWhHMBZfsgVkSvYWrlDQk437zsMqS8DfHgM5DL0fiMSGa9w+AS0L35Ny?= =?us-ascii?Q?nreHpTkiHx04xgeSQ9eU1whnC0vSf0ky5YDp6Qgqtx44RwyHNsBk1mmBx+BN?= =?us-ascii?Q?UcZrdq34JXBC4FHuvf7M8FkoMiGuHaiTZ3ColrF3np3QiuP3Rz/vopOzoRAI?= =?us-ascii?Q?1qeXGvaqFJ5RsUEUW6NNmsj5Sossa/nO1iEol2Y5bz8NoU0/uKdR3lyVZ/T6?= =?us-ascii?Q?e70ZensNJYXuO7gj5kTR9v1wJKxd+njdPqXb?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:34:26.9960 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0ecad30c-670c-40ac-5238-08dddb480db2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B072.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7748 Content-Type: text/plain; charset="utf-8" Having independent thread helps in: - Alleviating the need for multiple scanning threads - Aids to control batch migration (TBD) - Migration throttling (TBD) Signed-off-by: Raghavendra K T --- mm/kscand.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index 7552ce32beea..55efd0a6e5ba 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -41,6 +42,15 @@ static unsigned long kscand_mms_to_scan __read_mostly = =3D KSCAND_MMS_TO_SCAN; =20 bool kscand_scan_enabled =3D true; static bool need_wakeup; +static bool migrated_need_wakeup; + +/* How long to pause between two migration cycles */ +static unsigned int kmigrate_sleep_ms __read_mostly =3D 20; + +static struct task_struct *kmigrated_thread __read_mostly; +static DEFINE_MUTEX(kmigrated_mutex); +static DECLARE_WAIT_QUEUE_HEAD(kmigrated_wait); +static unsigned long kmigrated_sleep_expire; =20 static unsigned long kscand_sleep_expire; =20 @@ -79,6 +89,7 @@ struct kscand_scanctrl { }; =20 struct kscand_scanctrl kscand_scanctrl; + /* Per folio information used for migration */ struct kscand_migrate_info { struct list_head migrate_node; @@ -134,6 +145,19 @@ static inline bool is_valid_folio(struct folio *folio) return true; } =20 +static inline void kmigrated_wait_work(void) +{ + const unsigned long migrate_sleep_jiffies =3D + msecs_to_jiffies(kmigrate_sleep_ms); + + if (!migrate_sleep_jiffies) + return; + + kmigrated_sleep_expire =3D jiffies + migrate_sleep_jiffies; + wait_event_timeout(kmigrated_wait, + true, + migrate_sleep_jiffies); +} =20 static bool folio_idle_clear_pte_refs_one(struct folio *folio, struct vm_area_struct *vma, @@ -537,6 +561,49 @@ static int stop_kscand(void) return 0; } =20 +static int kmigrated(void *arg) +{ + while (true) { + WRITE_ONCE(migrated_need_wakeup, false); + if (unlikely(kthread_should_stop())) + break; + msleep(20); + kmigrated_wait_work(); + } + return 0; +} + +static int start_kmigrated(void) +{ + struct task_struct *kthread; + + guard(mutex)(&kmigrated_mutex); + + /* Someone already succeeded in starting daemon */ + if (kmigrated_thread) + return 0; + + kthread =3D kthread_run(kmigrated, NULL, "kmigrated"); + if (IS_ERR(kmigrated_thread)) { + pr_err("kmigrated: kthread_run(kmigrated) failed\n"); + return PTR_ERR(kthread); + } + + kmigrated_thread =3D kthread; + pr_info("kmigrated: Successfully started kmigrated"); + + wake_up_interruptible(&kmigrated_wait); + + return 0; +} + +static int stop_kmigrated(void) +{ + guard(mutex)(&kmigrated_mutex); + kthread_stop(kmigrated_thread); + return 0; +} + static inline void init_list(void) { INIT_LIST_HEAD(&kscand_scanctrl.scan_list); @@ -559,8 +626,15 @@ static int __init kscand_init(void) if (err) goto err_kscand; =20 + err =3D start_kmigrated(); + if (err) + goto err_kmigrated; + return 0; =20 +err_kmigrated: + stop_kmigrated(); + err_kscand: stop_kscand(); kscand_destroy(); --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2070.outbound.protection.outlook.com [40.107.101.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 983152BD11 for ; Thu, 14 Aug 2025 15:34:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.101.70 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185682; cv=fail; b=aa19Z0JxRVCfCZffakLaSfzsTFne176FSL3wn6O/Vls3rMfuDMu9E7/pw8vyI9ZrH32KSNp8qhyTZyEMqwOIGLj31L8ivJgJoMSychukmj/UWnL/7qhnX5GUKttvkLRUvh0Y35MxIaM/U2a5MvwFRpCIRe7g8s3MqjAXc1AgdgM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185682; c=relaxed/simple; bh=XjdqJoTVuWnAQvu9U7GhTZmPY5ZvlJ+9YL69b3I9wdc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fOr4kz26ipAqNq57nrI/NOiSY7EgloNqtc6eixQH/WoM33w65kncA9zqaYyLZOBfz9uSx2JnCMaD7Dwlp7cbQUUo6911APlXGsFb36DQ/qOdsyd55nqg0xQMvfy+sWxgxcJdVSARRttP/BnEm+FzuqFFb9oX1Qiy0na8GaPuP4s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=s5Bi/G9a; arc=fail smtp.client-ip=40.107.101.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="s5Bi/G9a" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=C6PweDl9X/qb8/frCmiBvCoCextbq0U56AcXCWYior6ZVEmfOqtz0SXnu67Mibdm5XVJgwlApabG4bw6O19Ozj5dahvTgpWMNrkum00E4XaZ1Brej2TXp1G/IiqqLSj7Xj+0hVLqfpb8jOfiPNyJWNXfLhfSma8NSdDGa1mVWeWIVF1GKuKeUs0US8b3L/E1BYJipo+rcMzNcFfpyiCJtFuYw9rHb/O8k8/c0UIEoPsZ/nejtiM1B9pYDRA24EWgaFmfDpiV3SKKB5R6WZpigd5Np3bj0Zhcsv4Z6wqOGLA8S2jlwCYyxtevd4908DlytDcXiA87CJIkg9ieMFyPFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rs8kACRyEGPKQvnLaHlhDH+kmdQbaBr+b8USTRmdZl4=; b=PnmdhObwsH5PXXya0UQQqVayI1/wj4cSZILtJtsB8/Og9+GYliPpKtcOv3HPBJnYqWjZNXTKzwrP1JA+QLlqesj1OIcfu4TjpKR7gvQIef1pa5pE3I7SBBzdVPq+JsXTIkvCsQpf99G0KNUY1lW1ilrF3Uxnap9h2NCLTw1AnH9llekcUlHAhw8gKQ3NHMMZVW9XTepZO0q7/tPMCICaK9+cDOmtaGjmqzXcjnIU+Kkzo9As/Y3UCv3OZJdPMEnynZ29UdfqWV4BdvfdrDTR7H1DcGQEfusdW13gSE4lI91jBRyea9okqSVbAp9gpOP/IVlXWPk1HhsqOx+ButGYww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rs8kACRyEGPKQvnLaHlhDH+kmdQbaBr+b8USTRmdZl4=; b=s5Bi/G9aTDLKECvSprbDIn7g4eOxIbxTE9FAx34qcs5o67uVdeZ/pFJoV6+p/d7Gcss9n76Hj1mM5OKdEfQy2GhcuHOkXpd6GaX2otEj6Q39sFG/qrI1Kdw/3GASbJCxjIEgSohJJ6qQb6BMhrsKFXVT1FPkCEDx/iwOb6DsOBM= Received: from BL6PEPF00016412.NAMP222.PROD.OUTLOOK.COM (2603:10b6:22e:400:0:1004:0:a) by DM4PR12MB6230.namprd12.prod.outlook.com (2603:10b6:8:a7::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.14; Thu, 14 Aug 2025 15:34:36 +0000 Received: from BN3PEPF0000B075.namprd04.prod.outlook.com (2a01:111:f403:c803::4) by BL6PEPF00016412.outlook.office365.com (2603:1036:903:4::a) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:34:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B075.mail.protection.outlook.com (10.167.243.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:34:35 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:34:24 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 06/17] mm/migration: migrate accessed folios to toptier node Date: Thu, 14 Aug 2025 15:32:56 +0000 Message-ID: <20250814153307.1553061-7-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B075:EE_|DM4PR12MB6230:EE_ X-MS-Office365-Filtering-Correlation-Id: be27cdc8-f3c8-4d6c-8ac7-08dddb4812d5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|7416014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?RVdPMYKyC8i7bOBnRlBS/XCYAGJez1uIFMd2NPzpoLVzoWJYQ5oDqrMkjyzj?= =?us-ascii?Q?xtZn6fkXKBogKGPbtXh91IXS783ug5d3x2HJ65j/Jr4JOU3TZ3ZiuncLKg6l?= =?us-ascii?Q?PL8o7faXhgTZvfTZlkwy58XE7xmHYOZznF0suLS+l1xdG5eSMuV2YWH5VKgv?= =?us-ascii?Q?SY9h+ygTeRtbUlRd1AKWh/NfMNODqAV14oFES63sNUoj5xTZ0um9OkQs2xX8?= =?us-ascii?Q?ytWghkGaZwhaEW40pro1G5gO/FT0+i+6n3nq2gIVirkpt+Puy7Z7Jj4qTn8S?= =?us-ascii?Q?PwTAZL3DRUtalyQO7LOYAP0m+jXjk/svOe6DmUj4FYsEvJaBIc16NWwQCsWm?= =?us-ascii?Q?VQAfe35MpPMKiJQO3/bUchVlVbrRcUMLUsW2KtPhAbyH92xb408yldYv6ios?= =?us-ascii?Q?2Q5DYqunD+58hezUn7NwjmrKg7uwEukPb2SX4Mzdt3/JVQsBMcqYo9q3qLJl?= =?us-ascii?Q?IZHjrjTLexqlXv/GQkvPxwEAw6HW5zBhx2hUSYAEwbVqSRccGmPWYq82AAN6?= =?us-ascii?Q?tOdkfhCuyDEf6MdoGxWianc9xCw2HcZfmZeMNJkJoOkYyFMwewqdSI/2a9u2?= =?us-ascii?Q?v1RHBIKDLQdNlniXpPe9pdrK5exNx1tFho3GUcsRqEvqWEKNNrsEy7+ZvI7k?= =?us-ascii?Q?GvFn7W8ehjj1v0eRDnzRqLPW9cupC9VxIZix2fEXZTJS1FKP4ImMJeiFEVCH?= =?us-ascii?Q?TFdlTK2RtpVBQL3dS2orboqf4o3s4ywVCKBnIIzN2gt8QkKFU0/KHWBXDdSl?= =?us-ascii?Q?vW3cqDfwHKcXwiSEh1AHGL1qIoFk/RS/YvwuOq9ovTIgadCkASC2e69DDdHw?= =?us-ascii?Q?d1zDFshiTWDFP+I4Ahhc3U/jeMorPe9cHkyjfJOK16AvafWlk3nXGnEei+Xn?= =?us-ascii?Q?IxLFdnHE9ttpI72FVbpXE9T/frjT8IYSiQtVx7PoTrjyJtQAuoYBK4sYMCR/?= =?us-ascii?Q?wpe1QrkNIGD53w5zVCldQsB+3YSpFYe5KPMLVSYwLeQznxHLq9SmHDjPVKvX?= =?us-ascii?Q?Y9qJt6i6ZbriFCfimyUThmVxky6zGMz2JIo33gYZ0w9Sr5YfkXomgQDFggq6?= =?us-ascii?Q?WJO80jQiU/HgBAXdoy3nBlca7qPfsVOxCLEs3IS1A1AdAY9kq7rLo1HS5QUk?= =?us-ascii?Q?zHyme3FegNlLweR5069fGjTH7jrf9ZgfyH2vn0G89DCbviD66dMTCh1S70Vu?= =?us-ascii?Q?d1099V0pl6dvX7zrxUiYMeCoA9wNtUV2v5jCEobBWnrFOzPpSDgXv+gAsGz1?= =?us-ascii?Q?pLuTQn6FQix6Vp1QATpwNVjW/FeHNVsgnbfetn2nqrFmjIa4nIZc0Nf1Fit/?= =?us-ascii?Q?ZNTTgxfULqXAnhSkOS7qTuEaXs1jFlZSRXSxZetASWFPOkSEj4I8gXPiJyjX?= =?us-ascii?Q?IJm6+LQ9W6fV/F8toSys1vC1AwWgIW8nnE21soZYEHx7mGqRlf+D+FEeVvYh?= =?us-ascii?Q?KEf6WwmXqf0Eq+jwMUg/9qEdJ1cd0qwHtsVC4tb5CVtgJS4yCxNKkPgog0iU?= =?us-ascii?Q?n3v4G8SWBWd6qj1BwcnSDxmt8OhWWzybOI6n?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(7416014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:34:35.6163 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: be27cdc8-f3c8-4d6c-8ac7-08dddb4812d5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B075.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6230 Content-Type: text/plain; charset="utf-8" A per mm migration list is added and a kernel thread iterates over each of them. For each recently accessed slowtier folio in the migration list: - Isolate LRU pages - Migrate to a regular node. The rationale behind whole migration is to speedup the access to recently accessed pages. Currently, PTE A bit scanning approach lacks the information about exact destination node to migrate to. Reason: PROT_NONE hint fault based scanning is done in a process context. Here when the fault occurs, source CPU of the fault associated the task is known. Time of page access is also accurate. With the lack of above information, migration is done to node 0 by default. Signed-off-by: Raghavendra K T --- include/linux/migrate.h | 2 + mm/kscand.c | 448 +++++++++++++++++++++++++++++++++++++++- mm/migrate.c | 2 +- 3 files changed, 440 insertions(+), 12 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index aaa2114498d6..59547f72d150 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -142,6 +142,8 @@ const struct movable_operations *page_movable_ops(struc= t page *page) } =20 #ifdef CONFIG_NUMA_BALANCING +bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); diff --git a/mm/kscand.c b/mm/kscand.c index 55efd0a6e5ba..5cd2764114df 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -52,9 +52,18 @@ static DEFINE_MUTEX(kmigrated_mutex); static DECLARE_WAIT_QUEUE_HEAD(kmigrated_wait); static unsigned long kmigrated_sleep_expire; =20 +/* mm of the migrating folio entry */ +static struct mm_struct *kmigrated_cur_mm; + +/* Migration list is manipulated underneath because of mm_exit */ +static bool kmigrated_clean_list; + static unsigned long kscand_sleep_expire; +#define KSCAND_DEFAULT_TARGET_NODE (0) +static int kscand_target_node =3D KSCAND_DEFAULT_TARGET_NODE; =20 static DEFINE_SPINLOCK(kscand_mm_lock); +static DEFINE_SPINLOCK(kscand_migrate_lock); static DECLARE_WAIT_QUEUE_HEAD(kscand_wait); =20 #define KSCAND_SLOT_HASH_BITS 10 @@ -62,6 +71,10 @@ static DEFINE_READ_MOSTLY_HASHTABLE(kscand_slots_hash, K= SCAND_SLOT_HASH_BITS); =20 static struct kmem_cache *kscand_slot_cache __read_mostly; =20 +#define KMIGRATED_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kmigrated_slots_hash, KMIGRATED_SLOT_H= ASH_BITS); +static struct kmem_cache *kmigrated_slot_cache __read_mostly; + /* Per mm information collected to control VMA scanning */ struct kscand_mm_slot { struct mm_slot slot; @@ -90,6 +103,26 @@ struct kscand_scanctrl { =20 struct kscand_scanctrl kscand_scanctrl; =20 +/* Per mm migration list */ +struct kmigrated_mm_slot { + /* Tracks mm that has non empty migration list */ + struct mm_slot mm_slot; + /* Per mm lock used to synchronize migration list */ + spinlock_t migrate_lock; + /* Head of per mm migration list */ + struct list_head migrate_head; +}; + +/* System wide list of mms that maintain migration list */ +struct kmigrated_daemon { + struct list_head mm_head; + struct kmigrated_mm_slot *mm_slot; +}; + +struct kmigrated_daemon kmigrated_daemon =3D { + .mm_head =3D LIST_HEAD_INIT(kmigrated_daemon.mm_head), +}; + /* Per folio information used for migration */ struct kscand_migrate_info { struct list_head migrate_node; @@ -108,6 +141,11 @@ static inline int kscand_has_work(void) return !list_empty(&kscand_scan.mm_head); } =20 +static inline int kmigrated_has_work(void) +{ + return !list_empty(&kmigrated_daemon.mm_head); +} + static inline bool kscand_should_wakeup(void) { bool wakeup =3D kthread_should_stop() || need_wakeup || @@ -118,6 +156,16 @@ static inline bool kscand_should_wakeup(void) return wakeup; } =20 +static inline bool kmigrated_should_wakeup(void) +{ + bool wakeup =3D kthread_should_stop() || migrated_need_wakeup || + time_after_eq(jiffies, kmigrated_sleep_expire); + + migrated_need_wakeup =3D false; + + return wakeup; +} + static void kscand_wait_work(void) { const unsigned long scan_sleep_jiffies =3D @@ -133,6 +181,85 @@ static void kscand_wait_work(void) scan_sleep_jiffies); } =20 +static void kmigrated_wait_work(void) +{ + const unsigned long migrate_sleep_jiffies =3D + msecs_to_jiffies(kmigrate_sleep_ms); + + if (!migrate_sleep_jiffies) + return; + + kmigrated_sleep_expire =3D jiffies + migrate_sleep_jiffies; + wait_event_timeout(kmigrated_wait, kmigrated_should_wakeup(), + migrate_sleep_jiffies); +} + +/* + * Do not know what info to pass in the future to make + * decision on taget node. Keep it void * now. + */ +static int kscand_get_target_node(void *data) +{ + return kscand_target_node; +} + +extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); + +/*XXX: Taken from migrate.c to avoid NUMAB mode=3D2 and NULL vma checks*/ +static int kscand_migrate_misplaced_folio_prepare(struct folio *folio, + struct vm_area_struct *vma, int node) +{ + int nr_pages =3D folio_nr_pages(folio); + pg_data_t *pgdat =3D NODE_DATA(node); + + if (folio_is_file_lru(folio)) { + /* + * Do not migrate file folios that are mapped in multiple + * processes with execute permissions as they are probably + * shared libraries. + * + * See folio_maybe_mapped_shared() on possible imprecision + * when we cannot easily detect if a folio is shared. + */ + if (vma && (vma->vm_flags & VM_EXEC) && + folio_maybe_mapped_shared(folio)) + return -EACCES; + /* + * Do not migrate dirty folios as not all filesystems can move + * dirty folios in MIGRATE_ASYNC mode which is a waste of + * cycles. + */ + if (folio_test_dirty(folio)) + return -EAGAIN; + } + + /* Avoid migrating to a node that is nearly full */ + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + for (z =3D pgdat->nr_zones - 1; z >=3D 0; z--) { + if (managed_zone(pgdat->node_zones + z)) + break; + } + + if (z < 0) + return -EAGAIN; + + wakeup_kswapd(pgdat->node_zones + z, 0, + folio_order(folio), ZONE_MOVABLE); + return -EAGAIN; + } + + if (!folio_isolate_lru(folio)) + return -EAGAIN; + + node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), + nr_pages); + + return 0; +} + static inline bool is_valid_folio(struct folio *folio) { if (!folio || !folio_mapped(folio) || !folio_raw_mapping(folio)) @@ -145,18 +272,113 @@ static inline bool is_valid_folio(struct folio *foli= o) return true; } =20 -static inline void kmigrated_wait_work(void) +enum kscand_migration_err { + KSCAND_NULL_MM =3D 1, + KSCAND_EXITING_MM, + KSCAND_INVALID_FOLIO, + KSCAND_NONLRU_FOLIO, + KSCAND_INELIGIBLE_SRC_NODE, + KSCAND_SAME_SRC_DEST_NODE, + KSCAND_PTE_NOT_PRESENT, + KSCAND_PMD_NOT_PRESENT, + KSCAND_NO_PTE_OFFSET_MAP_LOCK, + KSCAND_NOT_HOT_PAGE, + KSCAND_LRU_ISOLATION_ERR, +}; + + +static bool is_hot_page(struct folio *folio) { - const unsigned long migrate_sleep_jiffies =3D - msecs_to_jiffies(kmigrate_sleep_ms); + bool ret =3D false; =20 - if (!migrate_sleep_jiffies) - return; + if (!folio_test_idle(folio)) + ret =3D folio_test_referenced(folio) || folio_test_young(folio); =20 - kmigrated_sleep_expire =3D jiffies + migrate_sleep_jiffies; - wait_event_timeout(kmigrated_wait, - true, - migrate_sleep_jiffies); + return ret; +} + +static int kmigrated_promote_folio(struct kscand_migrate_info *info, + struct mm_struct *mm, + int destnid) +{ + unsigned long pfn; + unsigned long address; + struct page *page; + struct folio *folio =3D NULL; + int ret; + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; + pmd_t pmde; + int srcnid; + + if (mm =3D=3D NULL) + return KSCAND_NULL_MM; + + if (mm =3D=3D READ_ONCE(kmigrated_cur_mm) && + READ_ONCE(kmigrated_clean_list)) { + WARN_ON_ONCE(mm); + return KSCAND_EXITING_MM; + } + + pfn =3D info->pfn; + address =3D info->address; + page =3D pfn_to_online_page(pfn); + + if (page) + folio =3D page_folio(page); + + if (!page || PageTail(page) || !is_valid_folio(folio)) + return KSCAND_INVALID_FOLIO; + + if (!folio_test_lru(folio)) + return KSCAND_NONLRU_FOLIO; + + if (!is_hot_page(folio)) + return KSCAND_NOT_HOT_PAGE; + + folio_get(folio); + + srcnid =3D folio_nid(folio); + + /* Do not try to promote pages from regular nodes */ + if (!kscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return KSCAND_INELIGIBLE_SRC_NODE; + } + + /* Also happen when it is already migrated */ + if (srcnid =3D=3D destnid) { + folio_put(folio); + return KSCAND_SAME_SRC_DEST_NODE; + } + + address =3D info->address; + pmd =3D pmd_off(mm, address); + pmde =3D pmdp_get(pmd); + + if (!pmd_present(pmde)) { + folio_put(folio); + return KSCAND_PMD_NOT_PRESENT; + } + + pte =3D pte_offset_map_lock(mm, pmd, address, &ptl); + if (!pte) { + folio_put(folio); + WARN_ON_ONCE(!pte); + return KSCAND_NO_PTE_OFFSET_MAP_LOCK; + } + + ret =3D kscand_migrate_misplaced_folio_prepare(folio, NULL, destnid); + + folio_put(folio); + pte_unmap_unlock(pte, ptl); + + if (ret) + return KSCAND_LRU_ISOLATION_ERR; + + + return migrate_misplaced_folio(folio, destnid); } =20 static bool folio_idle_clear_pte_refs_one(struct folio *folio, @@ -302,6 +524,115 @@ static inline int kscand_test_exit(struct mm_struct *= mm) return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +struct destroy_list_work { + struct list_head migrate_head; + struct work_struct dwork; +}; + +static void kmigrated_destroy_list_fn(struct work_struct *work) +{ + struct destroy_list_work *dlw; + struct kscand_migrate_info *info, *tmp; + + dlw =3D container_of(work, struct destroy_list_work, dwork); + + if (!list_empty(&dlw->migrate_head)) { + list_for_each_entry_safe(info, tmp, &dlw->migrate_head, migrate_node) { + list_del(&info->migrate_node); + kfree(info); + } + } + + kfree(dlw); +} + +static void kmigrated_destroy_list(struct list_head *list_head) +{ + struct destroy_list_work *destroy_list_work; + + + destroy_list_work =3D kmalloc(sizeof(*destroy_list_work), GFP_KERNEL); + if (!destroy_list_work) + return; + + INIT_LIST_HEAD(&destroy_list_work->migrate_head); + list_splice_tail_init(list_head, &destroy_list_work->migrate_head); + INIT_WORK(&destroy_list_work->dwork, kmigrated_destroy_list_fn); + schedule_work(&destroy_list_work->dwork); +} + +static struct kmigrated_mm_slot *kmigrated_get_mm_slot(struct mm_struct *m= m, bool alloc) +{ + struct kmigrated_mm_slot *mm_slot =3D NULL; + struct mm_slot *slot; + + guard(spinlock)(&kscand_migrate_lock); + + slot =3D mm_slot_lookup(kmigrated_slots_hash, mm); + mm_slot =3D mm_slot_entry(slot, struct kmigrated_mm_slot, mm_slot); + + if (!mm_slot && alloc) { + mm_slot =3D mm_slot_alloc(kmigrated_slot_cache); + if (!mm_slot) { + spin_unlock(&kscand_migrate_lock); + return NULL; + } + + slot =3D &mm_slot->mm_slot; + INIT_LIST_HEAD(&mm_slot->migrate_head); + spin_lock_init(&mm_slot->migrate_lock); + mm_slot_insert(kmigrated_slots_hash, mm, slot); + list_add_tail(&slot->mm_node, &kmigrated_daemon.mm_head); + } + + return mm_slot; +} + +static void kscand_cleanup_migration_list(struct mm_struct *mm) +{ + struct kmigrated_mm_slot *mm_slot; + struct mm_slot *slot; + + mm_slot =3D kmigrated_get_mm_slot(mm, false); + + slot =3D &mm_slot->mm_slot; + + if (mm_slot && slot && slot->mm =3D=3D mm) { + spin_lock(&mm_slot->migrate_lock); + + if (!list_empty(&mm_slot->migrate_head)) { + if (mm =3D=3D READ_ONCE(kmigrated_cur_mm)) { + /* A folio in this mm is being migrated. wait */ + WRITE_ONCE(kmigrated_clean_list, true); + } + + kmigrated_destroy_list(&mm_slot->migrate_head); + spin_unlock(&mm_slot->migrate_lock); +retry: + if (!spin_trylock(&mm_slot->migrate_lock)) { + cpu_relax(); + goto retry; + } + + if (mm =3D=3D READ_ONCE(kmigrated_cur_mm)) { + spin_unlock(&mm_slot->migrate_lock); + goto retry; + } + } + /* Reset migrated mm_slot if it was pointing to us */ + if (kmigrated_daemon.mm_slot =3D=3D mm_slot) + kmigrated_daemon.mm_slot =3D NULL; + + hash_del(&slot->hash); + list_del(&slot->mm_node); + mm_slot_free(kmigrated_slot_cache, mm_slot); + + WRITE_ONCE(kmigrated_clean_list, false); + + spin_unlock(&mm_slot->migrate_lock); + } +} + static void kscand_collect_mm_slot(struct kscand_mm_slot *mm_slot) { struct mm_slot *slot =3D &mm_slot->slot; @@ -313,11 +644,77 @@ static void kscand_collect_mm_slot(struct kscand_mm_s= lot *mm_slot) hash_del(&slot->hash); list_del(&slot->mm_node); =20 + kscand_cleanup_migration_list(mm); + mm_slot_free(kscand_slot_cache, mm_slot); mmdrop(mm); } } =20 +static void kmigrated_migrate_mm(struct kmigrated_mm_slot *mm_slot) +{ + int ret =3D 0, dest =3D -1; + struct mm_slot *slot; + struct mm_struct *mm; + struct kscand_migrate_info *info, *tmp; + + spin_lock(&mm_slot->migrate_lock); + + slot =3D &mm_slot->mm_slot; + mm =3D slot->mm; + + if (!list_empty(&mm_slot->migrate_head)) { + list_for_each_entry_safe(info, tmp, &mm_slot->migrate_head, + migrate_node) { + if (READ_ONCE(kmigrated_clean_list)) + goto clean_list_handled; + + list_del(&info->migrate_node); + + spin_unlock(&mm_slot->migrate_lock); + + dest =3D kscand_get_target_node(NULL); + ret =3D kmigrated_promote_folio(info, mm, dest); + + kfree(info); + + cond_resched(); + spin_lock(&mm_slot->migrate_lock); + } + } +clean_list_handled: + /* Reset mm of folio entry we are migrating */ + WRITE_ONCE(kmigrated_cur_mm, NULL); + spin_unlock(&mm_slot->migrate_lock); +} + +static void kmigrated_migrate_folio(void) +{ + /* for each mm do migrate */ + struct kmigrated_mm_slot *kmigrated_mm_slot =3D NULL; + struct mm_slot *slot; + + if (!list_empty(&kmigrated_daemon.mm_head)) { + + scoped_guard (spinlock, &kscand_migrate_lock) { + if (kmigrated_daemon.mm_slot) { + kmigrated_mm_slot =3D kmigrated_daemon.mm_slot; + } else { + slot =3D list_entry(kmigrated_daemon.mm_head.next, + struct mm_slot, mm_node); + + kmigrated_mm_slot =3D mm_slot_entry(slot, + struct kmigrated_mm_slot, mm_slot); + kmigrated_daemon.mm_slot =3D kmigrated_mm_slot; + } + WRITE_ONCE(kmigrated_cur_mm, kmigrated_mm_slot->mm_slot.mm); + } + + if (kmigrated_mm_slot) + kmigrated_migrate_mm(kmigrated_mm_slot); + } +} + static unsigned long kscand_scan_mm_slot(void) { bool next_mm =3D false; @@ -331,6 +728,7 @@ static unsigned long kscand_scan_mm_slot(void) struct vm_area_struct *vma =3D NULL; struct kscand_mm_slot *mm_slot; =20 + struct kmigrated_mm_slot *kmigrated_mm_slot =3D NULL; =20 spin_lock(&kscand_mm_lock); =20 @@ -360,13 +758,23 @@ static unsigned long kscand_scan_mm_slot(void) =20 VMA_ITERATOR(vmi, mm, address); =20 + kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, false); + for_each_vma(vmi, vma) { kscand_walk_page_vma(vma, &kscand_scanctrl); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 if (vma_scanned_size >=3D kscand_scan_size) { next_mm =3D true; - /* TBD: Add scanned folios to migration list */ + + if (!list_empty(&kscand_scanctrl.scan_list)) { + if (!kmigrated_mm_slot) + kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, true); + spin_lock(&kmigrated_mm_slot->migrate_lock); + list_splice_tail_init(&kscand_scanctrl.scan_list, + &kmigrated_mm_slot->migrate_head); + spin_unlock(&kmigrated_mm_slot->migrate_lock); + } break; } } @@ -462,6 +870,8 @@ static int kscand(void *none) static inline void kscand_destroy(void) { kmem_cache_destroy(kscand_slot_cache); + /* XXX: move below to kmigrated thread */ + kmem_cache_destroy(kmigrated_slot_cache); } =20 void __kscand_enter(struct mm_struct *mm) @@ -497,7 +907,7 @@ void __kscand_exit(struct mm_struct *mm) { struct kscand_mm_slot *mm_slot; struct mm_slot *slot; - int free =3D 0; + int free =3D 0, serialize =3D 1; =20 spin_lock(&kscand_mm_lock); slot =3D mm_slot_lookup(kscand_slots_hash, mm); @@ -512,10 +922,15 @@ void __kscand_exit(struct mm_struct *mm) free =3D 1; /* TBD: Set the actual next slot */ kscand_scan.mm_slot =3D NULL; + } else if (mm_slot && kscand_scan.mm_slot =3D=3D mm_slot && mm_slot->is_s= canned) { + serialize =3D 0; } =20 spin_unlock(&kscand_mm_lock); =20 + if (serialize) + kscand_cleanup_migration_list(mm); + if (free) { mm_slot_free(kscand_slot_cache, mm_slot); mmdrop(mm); @@ -567,6 +982,8 @@ static int kmigrated(void *arg) WRITE_ONCE(migrated_need_wakeup, false); if (unlikely(kthread_should_stop())) break; + if (kmigrated_has_work()) + kmigrated_migrate_folio(); msleep(20); kmigrated_wait_work(); } @@ -607,7 +1024,9 @@ static int stop_kmigrated(void) static inline void init_list(void) { INIT_LIST_HEAD(&kscand_scanctrl.scan_list); + spin_lock_init(&kscand_migrate_lock); init_waitqueue_head(&kscand_wait); + init_waitqueue_head(&kmigrated_wait); } =20 static int __init kscand_init(void) @@ -621,6 +1040,13 @@ static int __init kscand_init(void) return -ENOMEM; } =20 + kmigrated_slot_cache =3D KMEM_CACHE(kmigrated_mm_slot, 0); + + if (!kmigrated_slot_cache) { + pr_err("kmigrated: kmem_cache error"); + return -ENOMEM; + } + init_list(); err =3D start_kscand(); if (err) diff --git a/mm/migrate.c b/mm/migrate.c index 2c88f3b33833..1f74dd5e6776 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2541,7 +2541,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long= , nr_pages, * Returns true if this is a safe migration target node for misplaced NUMA * pages. Currently it only checks the watermarks which is crude. */ -static bool migrate_balanced_pgdat(struct pglist_data *pgdat, +bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages) { int z; --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2083.outbound.protection.outlook.com [40.107.92.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BFC2240611 for ; Thu, 14 Aug 2025 15:34:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.83 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185697; cv=fail; b=JOCzJMyDrZ/wx1HgUTjRBIzr9qx8pI42jKd7iQGyCs1jx9GHsnrS6RU62WT8bmzukCjJXQitFAk/QrxMrpAemABtkb2nXjjtH/khsEgN6eYS7R/Ys49clZ5B95qiM/HETcAQSXKDWiNVmHp5dRJS1Zapls1GyypiEXUg+oZhTcc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185697; c=relaxed/simple; bh=Pf+8bsfqu3Y/A0K3zJtTlQYxpcH5DdxPy2ysNiSh1B8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Tlf0T+1fHopAfo/Rnh7tsTcYVx7R48Gy08lfj2od/2+AA30CFbOrXdzAH40oYIWTqZGewc5V9bC5IPgVwmkr2Q1Xt0CimutNdQc8gqnjAmjEBgaE7P5qdtx0jSCMRyto/ZpBXZHXacV5xHL9Hi/JXQYpvgdXZRJkCOHWOFM9qLc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=UYuVJwPS; arc=fail smtp.client-ip=40.107.92.83 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="UYuVJwPS" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bZFerDHx3OqYKTErY+3AnRnDnuCUkIx3tHJbwOQZlrbawLemybtvmsRcmDGeWuJMHkbdC32hKXCICBIz0JP3m23hY4GrZjdyNfxHoSZyYelo//QVNqebo7x/Lliiv8KRcmL/M0dQ3S+UI5Y0VjmeTm9JvgabLuH5ZEYiTvag1Neu/no/kyDWvkI7JacMcHzRG/ZtdCNawagpYp/CEMP2tTigAjHSzMB0v42nYRT9L41pbzFDPc8LCGa8tkmjWlUi9BRNVB24P65AxPwQjDVAvxNV2QpwKJFkEa2OPLUZvdqhLwNRToVa07zrMEZXR9myopabHzomDL7PpmDmfnzUvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2gS7utFbKjKorv2yT1Bf/w1CLnRdTjrnJQe1GOY/Q9Y=; b=KeTUvX6QubqFPzif1N2RpqSzuXF4uzTyKZFtmdiMaZ/YTwx5yVrvVHE0JbzUdyeS8MqkpJ1Ppx3Wyh9+/VLwXrH3frEc+F4uJWbuRTvt6r/rDsoSZ9WQJ5pVO2H3Ap7oYFVQEluzbrwW4mmDW1MmgZpliA98MiJ2cPyDh9z8m+Ncz519wXa9jE53p3KE5TIcW9Z3k1k/bXdRPjsfWm91MBmOXLN+ptByLWcwZ2zrdchCCFYhZR1cV1aB5wBgNXsa1zZ7hIJcNvVahboOJVcDyr96HXshoN81HULcYJDefdjZ5XjYlfoh5U5PgLF6Bwki9apZomXi3GidO52IKV/ZnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2gS7utFbKjKorv2yT1Bf/w1CLnRdTjrnJQe1GOY/Q9Y=; b=UYuVJwPS11l2EOFw17xfj/FtreMFgyzm1yWzNabIARaJO5puQKwzB39E2TWkKwNx3TSaZm3KqLSxnYukUhOHFYIXUwK+G2FBy6br7mq8ig6pTYTNY+6MB2N4zZKxDBV84nAISa8RZ/gnxuZ2fDPoHxJT07jScwfq0CjR2Deee1w= Received: from MN2PR06CA0020.namprd06.prod.outlook.com (2603:10b6:208:23d::25) by CH3PR12MB9730.namprd12.prod.outlook.com (2603:10b6:610:253::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.17; Thu, 14 Aug 2025 15:34:52 +0000 Received: from BN3PEPF0000B078.namprd04.prod.outlook.com (2603:10b6:208:23d:cafe::eb) by MN2PR06CA0020.outlook.office365.com (2603:10b6:208:23d::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:34:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B078.mail.protection.outlook.com (10.167.243.123) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:34:52 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:34:35 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 07/17] mm: Add throttling of mm scanning using scan_period Date: Thu, 14 Aug 2025 15:32:57 +0000 Message-ID: <20250814153307.1553061-8-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B078:EE_|CH3PR12MB9730:EE_ X-MS-Office365-Filtering-Correlation-Id: 7b434760-e2d1-4668-e8c0-08dddb481cc2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?fNKHlSNsqoUvXqr0q+xMbMxJfU10I0xAEVrnYSiGqcULWW5ZHpLCy5wzyVlO?= =?us-ascii?Q?HGYIlY4KK9bwIinDHmz6SDPnbtFhziYbZ0S+AR9v0lk49vPTJrYCpDz+KPE+?= =?us-ascii?Q?WSuapCvDU+9HHMCUFktx0gDDTYbZpiJwMwZd2+ErBY7BgbT6CAzeTE3Lqwi4?= =?us-ascii?Q?BTLKWHva0q1tQ9T1Gyyun/q9MQKaeGPMjVFcVVfwK0NoEEr1bLcxmmzlSNOn?= =?us-ascii?Q?2hZiH4/kucgikbwLo7sw7inhlzi9ZQo/OD5ZeY3XFfSDCNa/eWRgUK5iyFku?= =?us-ascii?Q?jd8jZDueAVDXgtI1fe9TMhyYrv8WdIJNrX82FL4zZGryKLy0Pw+tSPpPmsEI?= =?us-ascii?Q?3sksd2zwYpcw0Yaqt/50EWykses6xX5Sg4S5R1Eq7GptubDtLf7ZdIlSrj1r?= =?us-ascii?Q?aB2PZZ5yPDnC55D4RgHDj6+fMvsChyT9P0Tedhk3rKJRXSBgbq6vBhmpC90n?= =?us-ascii?Q?SERveyGPAQKUjgp5kHUr9qhxVCBk8m8zVVOZHk6p+thbsTHKMUu1YL4VhwTg?= =?us-ascii?Q?xTI+L96Ljztggyoq1aQm2nFpuY23+VUBY7hxQ5qJiDp2g/PmohRY5BTwU1qi?= =?us-ascii?Q?Xlft5lrem+SjOvgVyjvq01UrbWOueVnmABPQxD1FBOiJmFQQbm2Nga3vmLBx?= =?us-ascii?Q?MqiukQM5p01H1LvpUbQEWImO+aeY3Wqqjo0iTu1qDdFpznkCAKga1Kul2QOa?= =?us-ascii?Q?wxIOor2aVohhAciC/y8Ma1ETmHWipMGLAHgRsQw4YFfUCIPHas7RU+3HX+uO?= =?us-ascii?Q?Z+OGZ4bn5o9tJzjjkFwwAx2T2vjbPn5A2mV19dcCyfk99TehM9anzluFV040?= =?us-ascii?Q?aqGydcWd/xH4FLpa7Nug+/0+hlvDOnLzFXzX5Cfl2xlmPg8rYpNm5BywqQpj?= =?us-ascii?Q?o8ozqfKnE6MlZxYIYgUt4QjFrXG/Zx+uU/wazUu+vw3ezEw0HE5OvWj2ZYaD?= =?us-ascii?Q?F8myPbtq1YyPVqCrrhkCc/p7jbJOaPBvR3y/zOuWGXXw3T2WJxiqTMy1+APm?= =?us-ascii?Q?lHlYtLFNpv7iFMr6PUwKfPUJ9mpef6sdYAgVLq3OyiNusxXYfB55xAmmKvPv?= =?us-ascii?Q?4tbCUl54CPLXSHusKOWvpQA/eF3if8fwlYZL5//qKD0fzTMkOpDi6u4j0Plu?= =?us-ascii?Q?Rq4/4xd3ztwXnHWgflDyy01XXo4M+uPNytveW/P+JJ2ZO769PGDaY8Y1lzVP?= =?us-ascii?Q?HPiunUXxj9duFMx5DmSnp+gKVlVYHZjcry2eeMko4qJUJ9oKnkeKyWpkRtra?= =?us-ascii?Q?SQVoyWRMFM4Nl5UdCldct1JnLdXXun7Kcl+ISVFD9xO28R42y5KMbLasJkcO?= =?us-ascii?Q?U9YAkv6qcJxKPLgnYDr56mVud1gzWQzK4iPJiJGcsPxtC7yHAYHbwNOWcOA+?= =?us-ascii?Q?xu5sk/CUekLrjYlgR1FjnuyXoyn2LwytJNpuJdJnZgFaxcXvQgf1s+0LreTw?= =?us-ascii?Q?AJ+j+Ayxl5hyfiF+b6+WnuOx32r7J+8vQuiKNp7cLA9Vj47g/RUAjLv7URWo?= =?us-ascii?Q?1zx5STivK185OHptq5Dy3CdRDwK9GPYRTh29?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:34:52.2727 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7b434760-e2d1-4668-e8c0-08dddb481cc2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B078.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9730 Content-Type: text/plain; charset="utf-8" Before this patch, scanning of tasks' mm is done continuously and also at the same rate. Improve that by adding a throttling logic: 1) If there were useful pages found during last scan and current scan, decrease the scan_period (to increase scan rate) by TUNE_PERCENT (15%). 2) If there were no useful pages found in last scan, and there are candidate migration pages in the current scan decrease the scan_period aggressively by 2 power SCAN_CHANGE_SCALE (2^3 =3D 8 now). Vice versa is done for the reverse case. Scan period is clamped between MIN (600ms) and MAX (5sec). Signed-off-by: Raghavendra K T --- mm/kscand.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index 5cd2764114df..843069048c61 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -20,6 +20,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -33,6 +34,16 @@ static DEFINE_MUTEX(kscand_mutex); #define KSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) static unsigned long kscand_scan_size __read_mostly =3D KSCAND_SCAN_SIZE; =20 +/* + * Scan period for each mm. + * Min: 600ms default: 2sec Max: 5sec + */ +#define KSCAND_SCAN_PERIOD_MAX 5000U +#define KSCAND_SCAN_PERIOD_MIN 600U +#define KSCAND_SCAN_PERIOD 2000U + +static unsigned int kscand_mm_scan_period_ms __read_mostly =3D KSCAND_SCAN= _PERIOD; + /* How long to pause between two scan cycles */ static unsigned int kscand_scan_sleep_ms __read_mostly =3D 20; =20 @@ -78,6 +89,11 @@ static struct kmem_cache *kmigrated_slot_cache __read_mo= stly; /* Per mm information collected to control VMA scanning */ struct kscand_mm_slot { struct mm_slot slot; + /* Unit: ms. Determines how aften mm scan should happen. */ + unsigned int scan_period; + unsigned long next_scan; + /* Tracks how many useful pages obtained for migration in the last scan */ + unsigned long scan_delta; long address; bool is_scanned; }; @@ -715,13 +731,92 @@ static void kmigrated_migrate_folio(void) } } =20 +/* + * This is the normal change percentage when old and new delta remain same. + * i.e., either both positive or both zero. + */ +#define SCAN_PERIOD_TUNE_PERCENT 15 + +/* This is to change the scan_period aggressively when deltas are differen= t */ +#define SCAN_PERIOD_CHANGE_SCALE 3 +/* + * XXX: Hack to prevent unmigrated pages coming again and again while scan= ning. + * Actual fix needs to identify the type of unmigrated pages OR consider m= igration + * failures in next scan. + */ +#define KSCAND_IGNORE_SCAN_THR 256 + +/* Maintains stability of scan_period by decaying last time accessed pages= */ +#define SCAN_DECAY_SHIFT 4 +/* + * X : Number of useful pages in the last scan. + * Y : Number of useful pages found in current scan. + * Tuning scan_period: + * Initial scan_period is 2s. + * case 1: (X =3D 0, Y =3D 0) + * Increase scan_period by SCAN_PERIOD_TUNE_PERCENT. + * case 2: (X =3D 0, Y > 0) + * Decrease scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 3: (X > 0, Y =3D 0 ) + * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 4: (X > 0, Y > 0) + * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + */ +static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, + unsigned long total) +{ + unsigned int scan_period; + unsigned long now; + unsigned long old_scan_delta; + + scan_period =3D mm_slot->scan_period; + old_scan_delta =3D mm_slot->scan_delta; + + /* decay old value */ + total =3D (old_scan_delta >> SCAN_DECAY_SHIFT) + total; + + /* XXX: Hack to get rid of continuously failing/unmigrateable pages */ + if (total < KSCAND_IGNORE_SCAN_THR) + total =3D 0; + + /* + * case 1: old_scan_delta and new delta are similar, (slow) TUNE_PERCENT = used. + * case 2: old_scan_delta and new delta are different. (fast) CHANGE_SCAL= E used. + * TBD: + * 1. Further tune scan_period based on delta between last and current sc= an delta. + * 2. Optimize calculation + */ + if (!old_scan_delta && !total) { + scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && total) { + scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && !total) { + scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; + } else { + scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + } + + scan_period =3D clamp(scan_period, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PE= RIOD_MAX); + + now =3D jiffies; + mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); + mm_slot->scan_period =3D scan_period; + mm_slot->scan_delta =3D total; +} + static unsigned long kscand_scan_mm_slot(void) { bool next_mm =3D false; bool update_mmslot_info =3D false; =20 + unsigned int mm_slot_scan_period; + unsigned long now; + unsigned long mm_slot_next_scan; unsigned long vma_scanned_size =3D 0; unsigned long address; + unsigned long total =3D 0; =20 struct mm_slot *slot; struct mm_struct *mm; @@ -746,6 +841,8 @@ static unsigned long kscand_scan_mm_slot(void) =20 mm =3D slot->mm; mm_slot->is_scanned =3D true; + mm_slot_next_scan =3D mm_slot->next_scan; + mm_slot_scan_period =3D mm_slot->scan_period; spin_unlock(&kscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -756,6 +853,11 @@ static unsigned long kscand_scan_mm_slot(void) goto outerloop; } =20 + now =3D jiffies; + + if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) + goto outerloop; + VMA_ITERATOR(vmi, mm, address); =20 kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, false); @@ -786,8 +888,10 @@ static unsigned long kscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 - if (update_mmslot_info) + if (update_mmslot_info) { mm_slot->address =3D address; + kscand_update_mmslot_info(mm_slot, total); + } =20 outerloop: /* exit_mmap will destroy ptes after this */ @@ -889,6 +993,10 @@ void __kscand_enter(struct mm_struct *mm) return; =20 kscand_slot->address =3D 0; + kscand_slot->scan_period =3D kscand_mm_scan_period_ms; + kscand_slot->next_scan =3D 0; + kscand_slot->scan_delta =3D 0; + slot =3D &kscand_slot->slot; =20 spin_lock(&kscand_mm_lock); --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2079.outbound.protection.outlook.com [40.107.220.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE799242D66 for ; Thu, 14 Aug 2025 15:34:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.79 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185701; cv=fail; b=bNdQr0U7+nyGDOXWrXXwvv4tOUx2Swaiwgvb8mGOsSreekYsefyJ99avF8hXUbkDwZHNDYua0Quv1tLMq5pVmsje6OXp11tyQ1bJ/DMhgab0JDop5ZIpenzOR2wJNHf1yM0wVLypBtvwf1ad+4bKkjBVtTZSqi0TEMqB7tBdNuw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185701; c=relaxed/simple; bh=K60BnPefbd/XIvlmDbBMuNUeYFmJPAar1pEwSOoAFZU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MRVi687GX+SuOAuNbJAyGY6060a9maOl/i+jLtNuOrjCvOy/EXjPSsAcjckivEIOl8suCD17lw0FD/2nHaWwItsIcx4XkO1nKiqy88DfHYpBQ/QKJenmtNAMfNMYoQFOSDBxMWB4JM+L+zPJysnrFpAsWypd8QvNCWo4LWu4bjg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=YRsAZaMm; arc=fail smtp.client-ip=40.107.220.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="YRsAZaMm" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=jvM6iM5ceCuOQVC3oj7oefy6BqfmFH4zAg9bOctTrjnPidNrKzWMNShVITc4kD/R5tDZI2kCAeFQhew2o0fMz0IrQ+WNK67DpksJ2pdXP+1tuBmpmZdQQVWlrnj64crhOrKeXxdCiMaRyqjKqSDy3gLnUYF843c1/iKcM1EvA3BLGAS2LjJ6WREwIUuvoJ0bJ45XcoZNumLiioga2253CcaM3r9G2cychZQ8ap2BZjulFhS6gglgHCMFZSDlq623B80sNeys96iBpy6RpWK+3wnCy3uIgwiNx+nz/aAOMEifMTIdWAZNRmmLFWZ6Tkl+AwA+pC/spLj83DoKdXI6ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WPVAyiNS4OZw8nJS07dTRaK/Lrchm9e7Lwh000KIeVY=; b=zVIPLjSHaLLEdiflPThM90/Ii5UTJQEekvz0MotIe9w9BiQbkMxK4NJB0ZNt7wp2wrQOxHMTqPiDTY744jZrjz29TFk0tq7md4FeYTW4f8uwXruqpjSm3q5dunrfxkFfS+K4bZCyaTBHwzzMSC2v58JX26/+CMbiQXZkh+ITDUNr9sD2xcJGnjSKOe9AEBGibOlFOchI9pgTNiifTWJCllZ7w1b9ExjTcsB0AIYWryMjXbt3ubXpzkm57faW/W5grU6ToPC2L0kygYKn7I3X0iGVhgsyecjrRrR8x6/pxvC1jpU1UFPo/5oB0FjR2sOqDJTWXbSRyEQukBa6Wi2OIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WPVAyiNS4OZw8nJS07dTRaK/Lrchm9e7Lwh000KIeVY=; b=YRsAZaMmtP9uVLef7zKlohaEvrxMzJvTta6J/mQh3kmFZ/yZXA9vsfI1T8buprz+yMsIT2kwuKO0wEnvbkw55Zyv3Z96xscyV4wTgAMG9/KYArOWIsPGR97c4FDrDJnx85jeM0atHqxSBOYmzbXyyV5ftboTd/6OBT31rMDQfhc= Received: from MN2PR05CA0026.namprd05.prod.outlook.com (2603:10b6:208:c0::39) by SA0PR12MB7076.namprd12.prod.outlook.com (2603:10b6:806:2d5::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.13; Thu, 14 Aug 2025 15:34:57 +0000 Received: from BN3PEPF0000B075.namprd04.prod.outlook.com (2603:10b6:208:c0:cafe::46) by MN2PR05CA0026.outlook.office365.com (2603:10b6:208:c0::39) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9052.6 via Frontend Transport; Thu, 14 Aug 2025 15:34:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B075.mail.protection.outlook.com (10.167.243.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:34:56 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:34:46 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 08/17] mm: Add throttling of mm scanning using scan_size Date: Thu, 14 Aug 2025 15:32:58 +0000 Message-ID: <20250814153307.1553061-9-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B075:EE_|SA0PR12MB7076:EE_ X-MS-Office365-Filtering-Correlation-Id: c99a4623-ea3f-43c3-16d3-08dddb481f81 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?pJli8E3NtzuwHCX561DIowNErRI4bURBXTfcsdjZytezFdcXJSQ0cuEHOmD2?= =?us-ascii?Q?fZe4x6kQPhXGpRrfFGzViJPm1SwqCCo57tloOXRtTCoLPEOUA7GCRzUUAFkj?= =?us-ascii?Q?AmtPHZgZNE/xFQaYAy2yR0al9QqTcB94VfdRHaf58kMn2nHzIIwFqU9YgCzG?= =?us-ascii?Q?Jw0t8quDaUpUy1H2IU5tpr6JnUeHbZxkUyzP4OZ1lsPqIZwR/Vmbn6FakIsp?= =?us-ascii?Q?O9No9dlBVMkznNvehmCOE7qlRKVMwSVVvRow6T6FD6jBluqGSb602z3xREVa?= =?us-ascii?Q?HU7Nf3crg/GY2C2UsjyuW/IO0UWP1XmB4c+Xg1d4+15c/cQkvaVhjAvvRsGV?= =?us-ascii?Q?2xMVBy2pfkZmbuzkUv817UPAuwvsYYQeOKGE/16s/Jj4nMm1wEZbSjZSvUtv?= =?us-ascii?Q?pTdzGv7/xgkqZUwgqGIbhAe/96LRWA/dXNjpb+qVVGVOHbbdNkXXQAIniUlJ?= =?us-ascii?Q?KwblkJOsGm0zkTfR2tEQm0b6Aeb7DA6O64fSgmyXzo9UYsH6oaYver017kaM?= =?us-ascii?Q?n4910WMlfnYlPhHXeRrLA3Vbcom5j2rxoXZcT5bwk3dblzvJ/4djtXjNhYRC?= =?us-ascii?Q?vsTqzydQTwqh8WOtfZQ9ydwvUOTk2zAdUXK9kMaKTI/M97Sko+2s6+lNhG5x?= =?us-ascii?Q?QeYkvVGPEyIzAHHserlmVmQBHFm2R/Bazt36VMEg57YPlaH8vz/GpnOm9mxJ?= =?us-ascii?Q?BqDY12XkrcNSJgh1j2W8lbwlQ9ISO+Q++KL1JnV+Ql0IObiSgALh+ml1G8Ps?= =?us-ascii?Q?4yaKX0JPuqiwKO9ubnZjRqVxx6rgoIPe/fLk7bkCOAfhPrQMEFWAMyeikT+T?= =?us-ascii?Q?g/yAT8m5MS7qPg/o9NANb6XTq+jPd8/bPLHFrnWu6XfuvdslGKAgIMHHn+kk?= =?us-ascii?Q?44gbIZJftHwHuMPyA/mvFWuJxJxsp8nFrtetzfdGa867RdqjLA9OCxg1EbPA?= =?us-ascii?Q?77o38tpBTJunSWKQcw8vJ2gIf4uQAfl+G9zQsOXkeHfS9fK24Y0MoqODYVTu?= =?us-ascii?Q?z0S+MXLlARcdo1ah4kDxzzgCmoY0wTLWIXK4VJm+CQypuz5PaX1N+XbTVuH1?= =?us-ascii?Q?w77xerjpkKXDDtXpNaYuZy9sbn6ziC2/xDVzoqhLc/Z/UDCC4kohOFeW6LOE?= =?us-ascii?Q?OJZAnowGD+P/cKSL2DEYgc0yEDrqO9ZhGf1yAnka5d3w4gUtejnilGlCE4sU?= =?us-ascii?Q?7pzTCO5ZzynUHTmIf7vPHA4TIGxbeVZ1mOMx8pJCEZ9iFOBtnrJbEJFVXiqq?= =?us-ascii?Q?1Q9W/sGTG9uM+to+bwurWxG08FKym/LHuMgyBeP/Cf1LC+7G1kf8Fscgp39x?= =?us-ascii?Q?upkn7e9e05rnh9Jn8OtCAzXapWPdWF2ReTH3uQGDBetshpH2PFbRDgrhl4mU?= =?us-ascii?Q?DnhS6+sH5UjJwSEpUUYeI9hUvzarldYtRs4K4S77al8Z7O/oy2b9AjvB6e1h?= =?us-ascii?Q?v0m9KZKMMbxz8Hw1PG1Vm3nrgPP7uovuFP6ZqwVssd7YqRh27SupNT/NMzUp?= =?us-ascii?Q?AqynZMFH8d3NbBKM/xcZczIFtJtJHkkDo6f0?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:34:56.8733 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c99a4623-ea3f-43c3-16d3-08dddb481f81 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B075.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7076 Content-Type: text/plain; charset="utf-8" Before this patch, scanning is done on entire virtual address space of all the tasks. Now the scan size is shrunk or expanded based on the useful pages found in the last scan. This helps to quickly get out of unnecessary scanning thus burning lesser CPU. Drawback: If a useful chunk is at the other end of the VMA space, it will delay scanning and migration. Shrink/expand algorithm for scan_size: X : Number of useful pages in the last scan. Y : Number of useful pages found in current scan. Initial scan_size is 1GB case 1: (X =3D 0, Y =3D 0) Decrease scan_size by 2 case 2: (X =3D 0, Y > 0) Aggressively change to MAX (4GB) case 3: (X > 0, Y =3D 0 ) No change case 4: (X > 0, Y > 0) Increase scan_size by 2 Scan size is clamped between MIN (256MB) and MAX (4GB)). TBD: Tuning based on real workloads Signed-off-by: Raghavendra K T --- mm/kscand.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index 843069048c61..39a7fcef7de8 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -28,10 +28,15 @@ =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); + /* * Total VMA size to cover during scan. + * Min: 256MB default: 1GB max: 4GB */ +#define KSCAND_SCAN_SIZE_MIN (256 * 1024 * 1024UL) +#define KSCAND_SCAN_SIZE_MAX (4 * 1024 * 1024 * 1024UL) #define KSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) + static unsigned long kscand_scan_size __read_mostly =3D KSCAND_SCAN_SIZE; =20 /* @@ -94,6 +99,8 @@ struct kscand_mm_slot { unsigned long next_scan; /* Tracks how many useful pages obtained for migration in the last scan */ unsigned long scan_delta; + /* Determines how much VMA address space to be covered in the scanning */ + unsigned long scan_size; long address; bool is_scanned; }; @@ -746,6 +753,8 @@ static void kmigrated_migrate_folio(void) */ #define KSCAND_IGNORE_SCAN_THR 256 =20 +#define SCAN_SIZE_CHANGE_SHIFT 1 + /* Maintains stability of scan_period by decaying last time accessed pages= */ #define SCAN_DECAY_SHIFT 4 /* @@ -761,14 +770,26 @@ static void kmigrated_migrate_folio(void) * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). * case 4: (X > 0, Y > 0) * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + * Tuning scan_size: + * Initial scan_size is 4GB + * case 1: (X =3D 0, Y =3D 0) + * Decrease scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). + * case 2: (X =3D 0, Y > 0) + * scan_size =3D KSCAND_SCAN_SIZE_MAX + * case 3: (X > 0, Y =3D 0 ) + * No change + * case 4: (X > 0, Y > 0) + * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, unsigned long total) { unsigned int scan_period; unsigned long now; + unsigned long scan_size; unsigned long old_scan_delta; =20 + scan_size =3D mm_slot->scan_size; scan_period =3D mm_slot->scan_period; old_scan_delta =3D mm_slot->scan_delta; =20 @@ -789,20 +810,25 @@ static inline void kscand_update_mmslot_info(struct k= scand_mm_slot *mm_slot, if (!old_scan_delta && !total) { scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size >> SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && total) { scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size << SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && !total) { scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; } else { scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + scan_size =3D KSCAND_SCAN_SIZE_MAX; } =20 scan_period =3D clamp(scan_period, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PE= RIOD_MAX); + scan_size =3D clamp(scan_size, KSCAND_SCAN_SIZE_MIN, KSCAND_SCAN_SIZE_MAX= ); =20 now =3D jiffies; mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); mm_slot->scan_period =3D scan_period; + mm_slot->scan_size =3D scan_size; mm_slot->scan_delta =3D total; } =20 @@ -814,6 +840,7 @@ static unsigned long kscand_scan_mm_slot(void) unsigned int mm_slot_scan_period; unsigned long now; unsigned long mm_slot_next_scan; + unsigned long mm_slot_scan_size; unsigned long vma_scanned_size =3D 0; unsigned long address; unsigned long total =3D 0; @@ -843,6 +870,7 @@ static unsigned long kscand_scan_mm_slot(void) mm_slot->is_scanned =3D true; mm_slot_next_scan =3D mm_slot->next_scan; mm_slot_scan_period =3D mm_slot->scan_period; + mm_slot_scan_size =3D mm_slot->scan_size; spin_unlock(&kscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -994,6 +1022,7 @@ void __kscand_enter(struct mm_struct *mm) =20 kscand_slot->address =3D 0; kscand_slot->scan_period =3D kscand_mm_scan_period_ms; + kscand_slot->scan_size =3D kscand_scan_size; kscand_slot->next_scan =3D 0; kscand_slot->scan_delta =3D 0; =20 --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2082.outbound.protection.outlook.com [40.107.102.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10565221F11 for ; Thu, 14 Aug 2025 15:35:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.82 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185717; cv=fail; b=Ray6Cl0m7Xas1ctrjQgvavNAyTMwvEjFc622GEbS7Ha2YLIvLlVRZyrNlgCUG0QzIOMudI1/iiz4c8vhi2sRDAsWT5AP7iYWVkqBWwes7bSN+8PdqJItERIU66PRUfeOvw+flkEpQz0YEnKkN6Gu0dk74R5z2q3zNJpD0c9QIiM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185717; c=relaxed/simple; bh=JrRmM7OePuZBNHbaPopJvYEkUqfDESztrs1tUUSpe5o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TTvSX0qIdFx5rVzXg9ZuhP3lKzOccnf2md7JX4ahyUn6NUBECozFvL1mLg7WfyQbTK44fk9GVw+kKPN3Y/0id2xxmsMDdC5/BtHs6hOXIcdMNVFtN4eif1g8Gi7jV2HeGfAUM2FjXbLLbUEEvcGGyp/nV34ETfybzqXDXsQ7F7E= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=PUvuD1/6; arc=fail smtp.client-ip=40.107.102.82 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="PUvuD1/6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qqLqiiGXd+X+F4PWXy7oHVaixU021X9evXbmT0oV7tf4O77pWi6rnVFNoJmcEnsKM+7oDTyzjt5w0NmH8kFCV9/0JBmFzDyc8RdHGH6XMfxX7eYJZSUdMwqDZPtWvY3BalzWimVTmyI1+ngAn4b9ihFRQUc+CAfTikCzfYA82Fg6p/4WQQssT2IgDnFtoLYIN9qePgiZTiqU0L/Dct42zGnMbTS55GvtWUZZO8o7ximPj2eY2zaF0uGH72UnEj2qpujg/ZweDYR9nsGNoKWHqXCA9IEYr+CtbxJt1LBvCCfj8Glo471+os73wKGUxtmZIwJeFS5RmUtc0X5f+oI4Aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=woX07bXSCoSE29Y9wKE+L2cWmZShzkBFCGt3WmiIeoM=; b=EgcQ+ku1jnYow+WowCfw7LmTuuIwMEC3PHH3C4ttQwOxdXcNX+UPoJELH82WIjyMiXl1KA6rFbDhs9i6abc3qQfLrEnOpDJ2U6gEQt5eQQmEPUSi7XkfzjHtKXA0ryF4oMN/sVVTHQiUslZYTi0ZWLDvAJESjm3AkWYYW0OxjwA7viZIK3amrb76fszrF68DZ2++BPShw4Zgn2hviRRUSPPGn71//7hJJD5po/KkNcgbGnw4T1OHZ/Sg5W08kkIVY2i6/g6Y2pBkRKnDzWblRUW7vFhRcdHCz5+HxuItVeC/oSKpzDSUjlYGk0J2OBQ2xLVV0XVY81bvrsIYSBFdxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=woX07bXSCoSE29Y9wKE+L2cWmZShzkBFCGt3WmiIeoM=; b=PUvuD1/6HsdFcViD08WTKKHmf4JIxXhBQigPGPz5j5Z2OyOXq391aUDXV7fgMV6jJgCkkSVkzryx5IZZlOhkjPckUhaQK06z/NdI7LEXIsAb++Nq6/sFLs/QOp6A6SYcPQvfy7n1CgwGhJeE12tCewGefgSJUMjS91KpyO53C9k= Received: from MN0PR04CA0028.namprd04.prod.outlook.com (2603:10b6:208:52d::27) by MN0PR12MB5980.namprd12.prod.outlook.com (2603:10b6:208:37f::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.15; Thu, 14 Aug 2025 15:35:13 +0000 Received: from BN3PEPF0000B074.namprd04.prod.outlook.com (2603:10b6:208:52d:cafe::53) by MN0PR04CA0028.outlook.office365.com (2603:10b6:208:52d::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:35:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B074.mail.protection.outlook.com (10.167.243.119) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:35:13 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:34:56 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 09/17] mm: Add initial scan delay Date: Thu, 14 Aug 2025 15:32:59 +0000 Message-ID: <20250814153307.1553061-10-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B074:EE_|MN0PR12MB5980:EE_ X-MS-Office365-Filtering-Correlation-Id: e20a28e1-8ea9-475c-d4d7-08dddb48293c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?3QjItbxSWK55f9S8dJPN8kJzmutDu8gaeU6Xz9KXs8LZZMM34wtS9STeL9+W?= =?us-ascii?Q?OYL0PNYV7vQNcydspLuUb0nqjxBUWzXcx+pgJA9L2BvXFFjI61q4SPdhtDfC?= =?us-ascii?Q?MrtI3KBLNT1rtTZNc4FD578qGHhlnGjpMTIeAUC53qFu/6qZ92XpoWC2J1D7?= =?us-ascii?Q?iBRzOz1FAuxNOWrxQ7gPKBkyMCjsusEaRFVORB2zD1pOfLx9U1ptXeCh3R2P?= =?us-ascii?Q?dVlmpKb3GC4Zj8HPLEI9hcQ5sxRTUNNV4saROoJ3nJBQ2dqn8eFibYLrrwA4?= =?us-ascii?Q?p8x+6j36Ptx6XOCVD8ROZXGJvjvYgqh8KPqfwBpsZVWpjkWX36pvHkaTPgAp?= =?us-ascii?Q?HJijaOmxVwHc53J0r1EMTEhMBNlbPYeXGyTjJs1VBirKO4NG4wy4WFoI3IJW?= =?us-ascii?Q?wiNgqEFic6FuTtzgD63rKM0BttRE68RsRnJVk4hXcbZzvamtE1/vQVBX75CB?= =?us-ascii?Q?v7Mv3XAzY3vDE1DQ5OSOcZlbHQ8ytJXpHw+/tXjC5WMi6hSAIUMNrWhopMk5?= =?us-ascii?Q?CS4G0L6W40dC03xp9qm49+EyDyIq44GOVGoLyrTuWvaXrpMpjo7B5WDYIPvx?= =?us-ascii?Q?i8T1MxPhW04PS2s4VAtP1oFyIXJG3T44O6jImDHuCHFm4yrFDJWjqnqi3Ri+?= =?us-ascii?Q?kylzML5OLz7GCxPCaAysFykHeO49UBm/6dUi+7hiEcs0Ko4XjQ2R4swR9nD2?= =?us-ascii?Q?0CKGvnKy8XQIah8utA8HSmbqT83dSiceivXXCc95d83sCg/OtONnp/yDLc4M?= =?us-ascii?Q?hZD6DWBERWURUUgrAqbbczkgSVxre7ND1brrrCDb5Tt1VRNu4lpgxDcBsTV6?= =?us-ascii?Q?ozFQ/AR9zxaFW5rxwhPpybJr4J3YzjNSTt7rRtS3ywuYs4adN8S1vzeNK9J3?= =?us-ascii?Q?Td7WsK16v96H3chF3QO9cfY2iIteFNyoxhtJD7CsH6+rIG3uAqBD2VWesFqa?= =?us-ascii?Q?NGJ27Bypi2bQIcIkTB6L0G7yxyl+beKYLnJclcFpmzzVgQRtTF1Z1Dh8XHoO?= =?us-ascii?Q?VqLMlxD1C0iod9410TxFn95pw2bEddocbvBClPRHtEq48+if+iUv2d5ruW0t?= =?us-ascii?Q?4TEFhzy6Vu5nPrz6mPHqWLEfVuMGo01R+f16LJQTGPg4Wfk8X/F9A0NtH7Oy?= =?us-ascii?Q?nhmXPYPT3l7wVRoRgvJal+AZghPGjiSnSddXF+VKGaKO0KccoeFttOr4YdCf?= =?us-ascii?Q?UydJWOrV45w6yulZYrZx0W2pcRLoFCdOUw4y4JLOq5tGOGp8F7fK0FmgsB0W?= =?us-ascii?Q?1OKu53BJxBrb/zZVHe/XzD8c2YGdR/F+1irRmrZV/Cm1fOmxkCpFqJLKcD6a?= =?us-ascii?Q?UVzZtnguNs4je1g9wPrliyKE1pi3SuQxgeVdeJ9KxrjoGTfsTQjtaAL0NF4S?= =?us-ascii?Q?rgc2opJGtNiDNxdugTJipJtr/LaLDZp3+OBZn87IZWKTx55tPe+P4zDLkbWp?= =?us-ascii?Q?np3nduxCmOMVU+IMnHwhJJj6e14WIVLCGbMTogSp2Jh0Z/zTMcN4WyN25+vj?= =?us-ascii?Q?n9YvFSQygAYKao6TJqTTusOKllbpWarJaEvn?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:35:13.2043 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e20a28e1-8ea9-475c-d4d7-08dddb48293c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B074.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5980 Content-Type: text/plain; charset="utf-8" This is to prevent unnecessary scanning of short lived tasks to reduce CPU burning. Signed-off-by: Raghavendra K T --- mm/kscand.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index 39a7fcef7de8..880c3693866d 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -28,6 +28,7 @@ =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); +extern unsigned int sysctl_numa_balancing_scan_delay; =20 /* * Total VMA size to cover during scan. @@ -1010,6 +1011,7 @@ void __kscand_enter(struct mm_struct *mm) { struct kscand_mm_slot *kscand_slot; struct mm_slot *slot; + unsigned long now; int wakeup; =20 /* __kscand_exit() must not run from under us */ @@ -1020,10 +1022,12 @@ void __kscand_enter(struct mm_struct *mm) if (!kscand_slot) return; =20 + now =3D jiffies; kscand_slot->address =3D 0; kscand_slot->scan_period =3D kscand_mm_scan_period_ms; kscand_slot->scan_size =3D kscand_scan_size; - kscand_slot->next_scan =3D 0; + kscand_slot->next_scan =3D now + + 2 * msecs_to_jiffies(sysctl_numa_balancing_scan_delay); kscand_slot->scan_delta =3D 0; =20 slot =3D &kscand_slot->slot; --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2085.outbound.protection.outlook.com [40.107.92.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1CCD22D7B6 for ; Thu, 14 Aug 2025 15:35:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185727; cv=fail; b=agzQx5Eb/ELN5i9p6zmj31+SuvYyEZivIImwbRL6wMaab6Rj+9fedD7KL8a5gmPEIiWs2HtjbFg20PERTs8wn24gRPZL+teRvYUKMa1flXopU0gkG34bEwfNXTAMPKQEOIfNxhgHmC7rx/06rN8gOgAWGXvFXVK5iL2mbKjvLBE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185727; c=relaxed/simple; bh=7PRg5+l7X8eA38FPF9YhmKckcm0cs/WpCor/SULb8OI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HU1gfuRUoK2QWqF/QTAd01FuC8VI3JCLenvwcAy9H7/B3wt5Np6pZhDiHBGt4OtOAsEgCY5LueOVBnnERfZBj5f6Eu3sriSM3WRW1tLiBcma5HGVKuEmuXm/IdNoGVJBBkX52CMMRl/vLbeA7wmaiu2wcrK3XEWkYu99qGbIM8M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=1TUzs1dU; arc=fail smtp.client-ip=40.107.92.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="1TUzs1dU" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uI1iCpTMFzh6xKY7eAjLcfA15CuJwKPsCvK2hc/LWtncyw1ag+wU7nekPAiQW3uXQzHgfMhIu0enzBaXwv6FBdHSA2vt//N4Y2EqBMg94e7ti8UlgiceRt2ybXfyY50gw0eVcl0qDq2pfgEPZXABjCMQsMy63wd4lonH4kwyFfffZUneeKCRIAlFrI8i/LpMJEDveqduETUfsW6Nqn5vL/dtFvTzRYh5yCcNqOecFKtWLx6oMN2xQeYOAOMcoxFogeKbAZzxOXR4qg8n31SoINnev+lWZiV3TUsyCqPq5XBEWyKh8Jt0E3CkPEjdGnUNP6iv2TQtg+hiRn+MLvA7XA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=k5ESOs9Y3pdm6jtYpS7KpN9D0UPsottGxVqFvAze+WU=; b=LQrnhgt63k1YMd0okbRQfx0DRiKSOsGs5wMVSc2fWgI+euvKHqAvZ5HWE1D7GJgCjT+g6CE3PqJDI3K0CXGTE+0XSz57c8bWTrsgI9H6kc4puTDvPTK8YQ+s+FrvXEMRF8yN5Yr8jlLt+nSJzRgPU4aWRZeFtugcDcf/FdUHADp6pfQ91/ufQ5/6sWvauFiR5i4NQ0gmzkgRKl8bimfVOzsK/bVqjJRZKHsiKuS+MLf6Hf8nOcBOV6uGHhbDHT8Vg/7XWwy04Xx24sdv2OjMviZu3sV3aiMoJx1jW0qyg2uF6hF3r044asncN27upMYDv/ztX5R3sImaJLzvJyidTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k5ESOs9Y3pdm6jtYpS7KpN9D0UPsottGxVqFvAze+WU=; b=1TUzs1dU2Tlw9CWRd/KCNSO2zMNlkxMBPxuYeLE7QZxlGmb2MrktrfgZDNf3GD3SXSl/CRK0GLExgp4hrCcOwHIPiov3JVLOJ1/hWA62jtC2J0h3u1Zhd+cplfijQVWKbQ6dQkqcFHmpy/LPZu4CDuL1Hks4hdSBAKaGH0gpFuY= Received: from BN0PR02CA0005.namprd02.prod.outlook.com (2603:10b6:408:e4::10) by CY8PR12MB7730.namprd12.prod.outlook.com (2603:10b6:930:85::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.17; Thu, 14 Aug 2025 15:35:18 +0000 Received: from BN3PEPF0000B071.namprd04.prod.outlook.com (2603:10b6:408:e4:cafe::eb) by BN0PR02CA0005.outlook.office365.com (2603:10b6:408:e4::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.18 via Frontend Transport; Thu, 14 Aug 2025 15:35:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B071.mail.protection.outlook.com (10.167.243.116) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:35:18 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:35:07 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 10/17] mm: Add a heuristic to calculate target node Date: Thu, 14 Aug 2025 15:33:00 +0000 Message-ID: <20250814153307.1553061-11-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B071:EE_|CY8PR12MB7730:EE_ X-MS-Office365-Filtering-Correlation-Id: e18778c0-8d4f-4708-afe8-08dddb482c2c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?xgLWA6DSXgk6bASZqFROIciQMibSILWhlt4uul1VCCRvUVxaTFQsqNCVJMbJ?= =?us-ascii?Q?P/PN5BF0MYCw0OW2h6J8Axr/0QjloWo6EY1DElkDcC5YjmLkL1IvPFuPOynj?= =?us-ascii?Q?Oori7j/VCGbGPNS7k5c169LlOHPHpD6hC69loZWlDh7wXr4XdLByrUxh6v4s?= =?us-ascii?Q?FAN6+cxZxxMEim+SgIuwVgpqr+cePFtpMihuF00zWSlcPbLn4zWHzA14TsGL?= =?us-ascii?Q?aEkDglltP6ie/q+MINGScw6LvfamqV8oHo1ThypWuWFNa+X4FjMRABWV26wv?= =?us-ascii?Q?/3P6BIOUHKQZuwm3mOAsYbmwXsJ52HQRKKY5ktxIkkDIuFLxC/zTBSEyUBiB?= =?us-ascii?Q?yuoYQRrA2gm8YA4L3nU/OeO2pK881bjLfhgL6yusL23/k/Y1tdak7q126byI?= =?us-ascii?Q?T296qngF4DMvNkuKvaJ+AWBVVli4F7P3T/xANhdgnx0LN1A9CN3xia3nMD63?= =?us-ascii?Q?wXHdcPfFV/gHqBRbJMu10mm1dydUJk1IgouUXV5Gmzdm5nrBvXeijtccGp2r?= =?us-ascii?Q?6QSwBxJtMu6cnymdwEhTzofUsGnQqMsLDiiSgPuUFh971ybgcn0dEUWZwYhz?= =?us-ascii?Q?rvkkHKG6s9SsVmw1n4n7nZijENScqxPUahpX5df1LZw+2M/4PZ3qmvEWrX5L?= =?us-ascii?Q?98UtqeTJb9tnWoaNgfA30G2cXT5GEIcKylmAI6gFWb1TxJPozxTOQ+fo4ns8?= =?us-ascii?Q?3GicTiPYufxaj2Nnbi2J1rVgTIndLthg0ullEpbezijehPty2BgKdn7pqO7l?= =?us-ascii?Q?awEFRnK2PC9SQq5VpRyjMW0FITInkqht2AXAzf+QFEpBdRq8qahvUqZU4YMI?= =?us-ascii?Q?X0mekNqL0D7qbc/Zfd8tGGDDXMk/YvFurqMGpZhRcND+4pke4smi0fcarTn9?= =?us-ascii?Q?kKv8CPMxazD3XVfynM2TkakOaDakGSdGfSPb847oG60+DshSpnQZLxmU0ymR?= =?us-ascii?Q?HePrNmImxm1tc1T4bTgayfoKkm4cEhOseuHAIl4Js0zAZ14q3X02qbjU4sW4?= =?us-ascii?Q?p9+IwSHzJsDvdCaQnTyM4fHVONM82gS5B9ZXvOeLaughpomyUwWqp2Jcf68u?= =?us-ascii?Q?tCO+mXWGUm75hsCP8uaF6bVUyZDOtd58KPFm7G7GsgsBKMgyZxspGTXfEv6Q?= =?us-ascii?Q?QKTNDM7Z9jGtNuD9zP3j9pvgLYfgs6cOL/tvPoztbSVX8WsS+K/zLV0jh1+V?= =?us-ascii?Q?TlF1/tUnlACYPfygNbljXFd0OgljneEHw0z/KHUQwyIHKpjjgq0NFd/Vdp7d?= =?us-ascii?Q?jUdkn9c1zITAYvFzulf7SVaL//cHFLCD8VMw/pG8MHxOeWSrCnEB7AEN59Ek?= =?us-ascii?Q?eUev7F0osHSJi+dUQ4XVRPLpxzwFsDSifcEd5xABJ7ZceuEPl34YePdF4R3Z?= =?us-ascii?Q?GFvgAGPkZb3nZ0D/5gc9T6h8aY02NkIqSJ8EHn1gPFUWar3glJAmmQuIwTk1?= =?us-ascii?Q?3mq1n8QhhYuCrDkO95smoIVBa1s3ynLesPbe89K/Jbxout0L98mcqo9HqBCA?= =?us-ascii?Q?UnSnVBMpgmdDP6E4xaOJf8tdIUiXvfsS/y6K?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:35:18.1283 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e18778c0-8d4f-4708-afe8-08dddb482c2c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B071.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7730 Content-Type: text/plain; charset="utf-8" One of the key challenges in PTE A bit based scanning is to find right target node to promote to. Here is a simple heuristic based approach: 1. While scanning pages of any mm, also scan toptier pages that belong to that mm. 2. Accumulate the insight on the distribution of active pages on toptier nodes. 3. Walk all the top-tier nodes and pick the one with highest accesses. This method tries to consolidate application to a single node. TBD: Create a list of preferred nodes for fallback when highest access node is nearly full. Signed-off-by: Raghavendra K T --- include/linux/mm_types.h | 4 + mm/kscand.c | 198 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 192 insertions(+), 10 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d6b91e8a66d6..e3d8f11a5a04 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1148,6 +1148,10 @@ struct mm_struct { /* numa_scan_seq prevents two threads remapping PTEs. */ int numa_scan_seq; #endif +#ifdef CONFIG_KSCAND + /* Tracks promotion node. XXX: use nodemask */ + int target_node; + #endif /* * An operation with batched TLB flushing is going on. Anything * that can move process memory needs to flush the TLB when diff --git a/mm/kscand.c b/mm/kscand.c index 880c3693866d..bf975e82357d 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -104,6 +104,7 @@ struct kscand_mm_slot { unsigned long scan_size; long address; bool is_scanned; + int target_node; }; =20 /* Data structure to keep track of current mm under scan */ @@ -116,13 +117,23 @@ struct kscand_scan kscand_scan =3D { .mm_head =3D LIST_HEAD_INIT(kscand_scan.mm_head), }; =20 +/* Per memory node information used to caclulate target_node for migration= */ +struct kscand_nodeinfo { + unsigned long nr_scanned; + unsigned long nr_accessed; + int node; + bool is_toptier; +}; + /* * Data structure passed to control scanning and also collect * per memory node information */ struct kscand_scanctrl { struct list_head scan_list; + struct kscand_nodeinfo *nodeinfo[MAX_NUMNODES]; unsigned long address; + unsigned long nr_to_scan; }; =20 struct kscand_scanctrl kscand_scanctrl; @@ -218,15 +229,129 @@ static void kmigrated_wait_work(void) migrate_sleep_jiffies); } =20 -/* - * Do not know what info to pass in the future to make - * decision on taget node. Keep it void * now. - */ +static unsigned long get_slowtier_accesed(struct kscand_scanctrl *scanctrl) +{ + int node; + unsigned long accessed =3D 0; + + for_each_node_state(node, N_MEMORY) { + if (!node_is_toptier(node) && scanctrl->nodeinfo[node]) + accessed +=3D scanctrl->nodeinfo[node]->nr_accessed; + } + return accessed; +} + +static inline unsigned long get_nodeinfo_nr_accessed(struct kscand_nodeinf= o *ni) +{ + return ni->nr_accessed; +} + +static inline void set_nodeinfo_nr_accessed(struct kscand_nodeinfo *ni, un= signed long val) +{ + ni->nr_accessed =3D val; +} + +static inline unsigned long get_nodeinfo_nr_scanned(struct kscand_nodeinfo= *ni) +{ + return ni->nr_scanned; +} + +static inline void set_nodeinfo_nr_scanned(struct kscand_nodeinfo *ni, uns= igned long val) +{ + ni->nr_scanned =3D val; +} + +static inline void reset_nodeinfo_nr_scanned(struct kscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); +} + +static inline void reset_nodeinfo(struct kscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); + set_nodeinfo_nr_accessed(ni, 0); +} + +static void init_one_nodeinfo(struct kscand_nodeinfo *ni, int node) +{ + ni->nr_scanned =3D 0; + ni->nr_accessed =3D 0; + ni->node =3D node; + ni->is_toptier =3D node_is_toptier(node) ? true : false; +} + +static struct kscand_nodeinfo *alloc_one_nodeinfo(int node) +{ + struct kscand_nodeinfo *ni; + + ni =3D kzalloc(sizeof(*ni), GFP_KERNEL); + + if (!ni) + return NULL; + + init_one_nodeinfo(ni, node); + + return ni; +} + +/* TBD: Handle errors */ +static void init_scanctrl(struct kscand_scanctrl *scanctrl) +{ + struct kscand_nodeinfo *ni; + int node; + + for_each_node(node) { + ni =3D alloc_one_nodeinfo(node); + if (!ni) + WARN_ON_ONCE(ni); + scanctrl->nodeinfo[node] =3D ni; + } +} + +static void reset_scanctrl(struct kscand_scanctrl *scanctrl) +{ + int node; + + for_each_node_state(node, N_MEMORY) + reset_nodeinfo(scanctrl->nodeinfo[node]); + + /* XXX: Not rellay required? */ + scanctrl->nr_to_scan =3D kscand_scan_size; +} + +static void free_scanctrl(struct kscand_scanctrl *scanctrl) +{ + int node; + + for_each_node(node) + kfree(scanctrl->nodeinfo[node]); +} + static int kscand_get_target_node(void *data) { return kscand_target_node; } =20 +static int get_target_node(struct kscand_scanctrl *scanctrl) +{ + int node, target_node =3D NUMA_NO_NODE; + unsigned long prev =3D 0; + + for_each_node(node) { + if (node_is_toptier(node) && scanctrl->nodeinfo[node]) { + /* This creates a fallback migration node list */ + if (get_nodeinfo_nr_accessed(scanctrl->nodeinfo[node]) > prev) { + prev =3D get_nodeinfo_nr_accessed(scanctrl->nodeinfo[node]); + target_node =3D node; + } + } + } + if (target_node =3D=3D NUMA_NO_NODE) + target_node =3D kscand_get_target_node(NULL); + + return target_node; +} + extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages); =20 @@ -495,6 +620,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, page_idle_clear_pte_refs(page, pte, walk); srcnid =3D folio_nid(folio); =20 + scanctrl->nodeinfo[srcnid]->nr_scanned++; + if (scanctrl->nr_to_scan) + scanctrl->nr_to_scan--; + + if (!scanctrl->nr_to_scan) { + folio_put(folio); + return 1; + } =20 if (!folio_test_lru(folio)) { folio_put(folio); @@ -502,13 +635,17 @@ static int hot_vma_idle_pte_entry(pte_t *pte, } =20 if (!kscand_eligible_srcnid(srcnid)) { + if (folio_test_young(folio) || folio_test_referenced(folio) + || pte_young(pteval)) { + scanctrl->nodeinfo[srcnid]->nr_accessed++; + } folio_put(folio); return 0; } if (!folio_test_idle(folio) && !prev_idle && (folio_test_young(folio) || folio_test_referenced(folio))) { =20 - /* XXX: Leaking memory. TBD: consume info */ + scanctrl->nodeinfo[srcnid]->nr_accessed++; =20 info =3D kzalloc(sizeof(struct kscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { @@ -697,7 +834,13 @@ static void kmigrated_migrate_mm(struct kmigrated_mm_s= lot *mm_slot) =20 spin_unlock(&mm_slot->migrate_lock); =20 - dest =3D kscand_get_target_node(NULL); + if (!mmap_read_trylock(mm)) { + dest =3D kscand_get_target_node(NULL); + } else { + dest =3D READ_ONCE(mm->target_node); + mmap_read_unlock(mm); + } + ret =3D kmigrated_promote_folio(info, mm, dest); =20 kfree(info); @@ -783,7 +926,7 @@ static void kmigrated_migrate_folio(void) * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, - unsigned long total) + unsigned long total, int target_node) { unsigned int scan_period; unsigned long now; @@ -831,6 +974,7 @@ static inline void kscand_update_mmslot_info(struct ksc= and_mm_slot *mm_slot, mm_slot->scan_period =3D scan_period; mm_slot->scan_size =3D scan_size; mm_slot->scan_delta =3D total; + mm_slot->target_node =3D target_node; } =20 static unsigned long kscand_scan_mm_slot(void) @@ -839,6 +983,7 @@ static unsigned long kscand_scan_mm_slot(void) bool update_mmslot_info =3D false; =20 unsigned int mm_slot_scan_period; + int target_node, mm_slot_target_node, mm_target_node; unsigned long now; unsigned long mm_slot_next_scan; unsigned long mm_slot_scan_size; @@ -872,6 +1017,7 @@ static unsigned long kscand_scan_mm_slot(void) mm_slot_next_scan =3D mm_slot->next_scan; mm_slot_scan_period =3D mm_slot->scan_period; mm_slot_scan_size =3D mm_slot->scan_size; + mm_slot_target_node =3D mm_slot->target_node; spin_unlock(&kscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -882,6 +1028,9 @@ static unsigned long kscand_scan_mm_slot(void) goto outerloop; } =20 + mm_target_node =3D READ_ONCE(mm->target_node); + if (mm_target_node !=3D mm_slot_target_node) + WRITE_ONCE(mm->target_node, mm_slot_target_node); now =3D jiffies; =20 if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -889,24 +1038,41 @@ static unsigned long kscand_scan_mm_slot(void) =20 VMA_ITERATOR(vmi, mm, address); =20 + /* Either Scan 25% of scan_size or cover vma size of scan_size */ + kscand_scanctrl.nr_to_scan =3D mm_slot_scan_size >> PAGE_SHIFT; + /* Reduce actual amount of pages scanned */ + kscand_scanctrl.nr_to_scan =3D mm_slot_scan_size >> 1; + + /* XXX: skip scanning to avoid duplicates until all migrations done? */ kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, false); =20 for_each_vma(vmi, vma) { kscand_walk_page_vma(vma, &kscand_scanctrl); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 - if (vma_scanned_size >=3D kscand_scan_size) { + if (vma_scanned_size >=3D mm_slot_scan_size || + !kscand_scanctrl.nr_to_scan) { next_mm =3D true; =20 if (!list_empty(&kscand_scanctrl.scan_list)) { if (!kmigrated_mm_slot) kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, true); + /* Add scanned folios to migration list */ spin_lock(&kmigrated_mm_slot->migrate_lock); + list_splice_tail_init(&kscand_scanctrl.scan_list, &kmigrated_mm_slot->migrate_head); spin_unlock(&kmigrated_mm_slot->migrate_lock); + break; } - break; + } + if (!list_empty(&kscand_scanctrl.scan_list)) { + if (!kmigrated_mm_slot) + kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, true); + spin_lock(&kmigrated_mm_slot->migrate_lock); + list_splice_tail_init(&kscand_scanctrl.scan_list, + &kmigrated_mm_slot->migrate_head); + spin_unlock(&kmigrated_mm_slot->migrate_lock); } } =20 @@ -917,9 +1083,19 @@ static unsigned long kscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + total =3D get_slowtier_accesed(&kscand_scanctrl); + target_node =3D get_target_node(&kscand_scanctrl); + + mm_target_node =3D READ_ONCE(mm->target_node); + + /* XXX: Do we need write lock? */ + if (mm_target_node !=3D target_node) + WRITE_ONCE(mm->target_node, target_node); + reset_scanctrl(&kscand_scanctrl); + if (update_mmslot_info) { mm_slot->address =3D address; - kscand_update_mmslot_info(mm_slot, total); + kscand_update_mmslot_info(mm_slot, total, target_node); } =20 outerloop: @@ -1113,6 +1289,7 @@ static int stop_kscand(void) kthread_stop(kscand_thread); kscand_thread =3D NULL; } + free_scanctrl(&kscand_scanctrl); =20 return 0; } @@ -1168,6 +1345,7 @@ static inline void init_list(void) spin_lock_init(&kscand_migrate_lock); init_waitqueue_head(&kscand_wait); init_waitqueue_head(&kmigrated_wait); + init_scanctrl(&kscand_scanctrl); } =20 static int __init kscand_init(void) --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2081.outbound.protection.outlook.com [40.107.244.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E348025525F for ; Thu, 14 Aug 2025 15:35:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185738; cv=fail; b=VoQ0vjZ0xff+ZgfR0IBhAuZesb09EASiXKwFbqF6xUe7yaqGqXui8N3D0M8ttJpRTELoeOQRaxpecx+CDeGe68MvFlYfTz6h0vMLlTT/xQcxhDp2OjhmSPHxGmt9c+E4TTUVepDiWGj/2TbNndPXehX81DKUPVH/XSDxD17wlmo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185738; c=relaxed/simple; bh=fNddrjGGNiCPLgz+xPLVQTySO215kqA3Zr0a4JmDTdg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=VtngddM8P9ThgoFB0w6sWu3gwR6R7T3a+QbXUdWn3bRCvdrvnLNb5wsIa0fXyWXfFD8OuNYvgzbFpDBNHHksgLbxP/OaGtOvXgHDjtVtkygJAKqI6vvDO2P1lX9c1CEfcC7m9CpEMf+oLDWIAMlE/oExi/mWcNLzdeEs4ADoPyY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=wQALEeC0; arc=fail smtp.client-ip=40.107.244.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="wQALEeC0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UI1Fs44ai4ciArbbhRbU1oz3kPL6Gws5tB2Hz1hMav1LYtOf44QDm718J4QrjYC/bsUEll6GPqSCZHENlSGFHYneRv0c21v7wBxUlWihIXIPPWubcGxhk91wvJ5SRoZu3mluZFUVR+jqlIsXl5yu3Dc2ZN4CLVjC0NzIt7/cGXAsuavg2ZlftZGmWvq/y3WGDAETUMK22gt+UERijGqBSLoxvTwSMa0J6JifDO1VcW3N+Wu6VUt9rTJNPVA9gujmd286Uqo+aP5uRIw47tQo1svjMo0JBOGeLH4/qgnBDkSv7yqQR8/wAD3calcQPFsUW8gdBu2+hFWkAaOOgZb0CA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=d1tgNC8I1gFvD95n4ns5y9DfXvdC88aWsx9sa4P1GXA=; b=HaaTcEG32EySgQGowGvmn0CsNqZs9Zzp5TZ87VinV33Xa3Wj3SJWSVXUSAPad8Lmvqdaq0On2PcXwkqebu3uAa1rbFHUe43HJDTEryRFHJ5fUwubmBUK+vb4OMLYhMZ19tgHG7SCZmMVFNM7ewXsm03eZ+TOqN0qVNSrIZRR5pMRwtyjqHB13N9L0vb5hTvdesfaYAKguHF31dMkHYS5fu3qE+tcRnkXC1GZVENh9tZDuPQt4cRgg3mw7NVLR1T17s2VKvr2lZlNohtRaajnD+j1KYLBY06PIzZe37UUoWaXnF0l8KvaISm3lKSPcLeWNbXek6oDFJop0W8LoErZbw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d1tgNC8I1gFvD95n4ns5y9DfXvdC88aWsx9sa4P1GXA=; b=wQALEeC02joMU1Ker+/z7y3C+gw6G3Ly3g9JL4/JenbPY0rvMPtVfmQOyyywcNXoFijGYync5RMWGcWoiWoFwxuKhVYUcpPOKozzRXSZKtMHs7qremcEIsInJex/ZUKIDFHRyVRVgQHbM36ANgWWpBDaJqUZULc1AdnRDVdLfmw= Received: from MN2PR06CA0028.namprd06.prod.outlook.com (2603:10b6:208:23d::33) by IA1PR12MB8554.namprd12.prod.outlook.com (2603:10b6:208:450::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.17; Thu, 14 Aug 2025 15:35:32 +0000 Received: from BN3PEPF0000B078.namprd04.prod.outlook.com (2603:10b6:208:23d:cafe::85) by MN2PR06CA0028.outlook.office365.com (2603:10b6:208:23d::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.18 via Frontend Transport; Thu, 14 Aug 2025 15:35:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B078.mail.protection.outlook.com (10.167.243.123) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:35:32 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:35:18 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 11/17] mm/kscand: Implement migration failure feedback Date: Thu, 14 Aug 2025 15:33:01 +0000 Message-ID: <20250814153307.1553061-12-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B078:EE_|IA1PR12MB8554:EE_ X-MS-Office365-Filtering-Correlation-Id: 41fc2e25-db10-4c11-096d-08dddb48349f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|36860700013|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?hUslX89EqrE6EzGD1eNDlDq/roZUKF7lpU53Qp1EDZFV3agUNnp7wCASyiAm?= =?us-ascii?Q?Sngbvb0GJRqnZUT2gommcV61KvBXw7c0bJbT11G5SMnDU+MIAGfyA2ViTvOJ?= =?us-ascii?Q?SUQXqQUWsc7yBOXbfV/jAPRXnkXvg3E8j/PzY22bSsraWEIm3jmfmwKFU4au?= =?us-ascii?Q?JYxqQQqKegjM0VfV0qhtcIcmezJt34RCU/8fr87uXhrmdeu24vFq955O/ceK?= =?us-ascii?Q?1Ii5N9uzS9pf6D46swXSUlq14plFL/G016Ss9lSlhh6kOemOi0gjZ3kDQ3bY?= =?us-ascii?Q?T6a7LKFhoOxWll3erhaJd1sjwFjK2zblyZTMOd4g7YCId4cV9ndRj4gT8s26?= =?us-ascii?Q?aCkJrVfgfS8vMGeXIsFdR//mMnkVq5gi5ONVjRpJ+zMB2rCd796XGadj1fti?= =?us-ascii?Q?dHijn2uAJFMVWXHOo4x1kKVfYUa81e52A7pj/Y7AdmdQ/e1sblFq8UwP3skE?= =?us-ascii?Q?Pi5Gqj7M2JXmmFtXoo2Q6sRgcqinmhYHSDhZ72kL+UeGouk/jQEpo+jwQveP?= =?us-ascii?Q?ac6xB7DYl4Eol02yGksjQZMTXV1+7OuYHVKlZpUuaxPrTjCK8U12FCxIduMe?= =?us-ascii?Q?c4H9RIZ8FaNwBKPc3fnvUwC8GBXDrLuQWXo8108tahOVUDs7dCJAnIVb5ML/?= =?us-ascii?Q?w5cz7CP//7dJ9FNNbm465SAvV1K0m4AjKIguMNvOdRvnH5kUZAv2zJyQ36/a?= =?us-ascii?Q?mvTsRZeONoeZxTtS6pWd0mek1ZdZBawAQ0XSerHjaPD7miEMoCwX1wKodzr4?= =?us-ascii?Q?OODXk6vRW/62CchtTB3+/kuhIlFvoFoZwFDUWELmt6MuL/5w6KwzHU/DNsBV?= =?us-ascii?Q?rSdqf+L5BGhswoFq4iKUr4YV8ICUbt462o0l6W80SI5+s89PPhbEcSkyVeZU?= =?us-ascii?Q?GVQ9rhtjTC6GvxkJrpKns/9M2ya0S3HiRXwbqNELoU6y91avrzVi03w/S9MG?= =?us-ascii?Q?O07TK+Yrf4/k0x1OB36GdJUnSnQ5Q/Gy6lQT6VBcQtJoIT4I2K4w26IxYQvt?= =?us-ascii?Q?mRCvMuG6/WxrOqcl1Khfk5qT1RaBvjuOVPf+nmtWczJOqk8T1Fb6VmD5ulFC?= =?us-ascii?Q?ZBHYMUciKdGavpn6WFfcIoTXfWE3xJQDjz+jCLbJi+kLrv7zz5DJWz8nnN30?= =?us-ascii?Q?yulz/q+JjTwseOGG+cAd1kVGisXNfhvNxJRFe6ZSCCtitet34vf/K19MNyut?= =?us-ascii?Q?OORLO4AD5w+pmRUHxAGpbZbAIDwWdbs9onp9MKLIS1Xh7qZXPZiLJe+uBUEK?= =?us-ascii?Q?4uAErYFp5xTOvhQf1rSKsX0N1xjYi9U119I5ycFjlB1uMk6K7AxoJhz/kKEa?= =?us-ascii?Q?hjk7aaqZjZap/sOXTgZb/NgXLDy6tE1VxITzrD0vO/7NBT1yW2LKkYKH+QuB?= =?us-ascii?Q?rsYORHC0uVsmoW9OiSvguqEtOYxus7sXrwAbHThxxtUQd3gOddmP3kjgQdHw?= =?us-ascii?Q?AsjX0pUkCpTZz10J4fvbJqM61UDuu7tijSD3ieDYtBMPDXx7xblgyAMaxDuN?= =?us-ascii?Q?ywSWT59+VutPQotXavrvLL0OHazt6FytMLgn?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(36860700013)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:35:32.3020 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 41fc2e25-db10-4c11-096d-08dddb48349f X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B078.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8554 Content-Type: text/plain; charset="utf-8" Before this, scanning kthread continues to scan even after migration fails. To control migration, scanning is slowed down based on the failure/success ratio obtained from migration thread. Decaying failure ratio is maintained for 1024 migration window. The ratio further contributes to approximately 10% scaling of scan_period. Signed-off-by: Raghavendra K T --- mm/kscand.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index bf975e82357d..41321d373be7 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -146,6 +146,8 @@ struct kmigrated_mm_slot { spinlock_t migrate_lock; /* Head of per mm migration list */ struct list_head migrate_head; + /* Indicates weighted success, failure */ + int msuccess, mfailed, fratio; }; =20 /* System wide list of mms that maintain migration list */ @@ -812,13 +814,45 @@ static void kscand_collect_mm_slot(struct kscand_mm_s= lot *mm_slot) } } =20 +static int kmigrated_get_mstat_fratio(struct mm_struct *mm) +{ + int fratio =3D 0; + struct kmigrated_mm_slot *mm_slot =3D NULL; + struct mm_slot *slot; + + guard(spinlock)(&kscand_migrate_lock); + + slot =3D mm_slot_lookup(kmigrated_slots_hash, mm); + mm_slot =3D mm_slot_entry(slot, struct kmigrated_mm_slot, mm_slot); + + if (mm_slot) + fratio =3D mm_slot->fratio; + + return fratio; +} + +static void update_mstat_ratio(struct kmigrated_mm_slot *mm_slot, + int msuccess, int mfailed) +{ + mm_slot->msuccess =3D (mm_slot->msuccess >> 2) + msuccess; + mm_slot->mfailed =3D (mm_slot->mfailed >> 2) + mfailed; + mm_slot->fratio =3D mm_slot->mfailed * 100; + mm_slot->fratio /=3D (mm_slot->msuccess + mm_slot->mfailed); +} + +#define MSTAT_UPDATE_FREQ 1024 + static void kmigrated_migrate_mm(struct kmigrated_mm_slot *mm_slot) { + int mfailed =3D 0; + int msuccess =3D 0; + int mstat_counter; int ret =3D 0, dest =3D -1; struct mm_slot *slot; struct mm_struct *mm; struct kscand_migrate_info *info, *tmp; =20 + mstat_counter =3D MSTAT_UPDATE_FREQ; spin_lock(&mm_slot->migrate_lock); =20 slot =3D &mm_slot->mm_slot; @@ -842,11 +876,23 @@ static void kmigrated_migrate_mm(struct kmigrated_mm_= slot *mm_slot) } =20 ret =3D kmigrated_promote_folio(info, mm, dest); + mstat_counter--; + + /* TBD: encode migrated count here, currently assume folio_nr_pages */ + if (!ret) + msuccess++; + else + mfailed++; =20 kfree(info); =20 cond_resched(); spin_lock(&mm_slot->migrate_lock); + if (!mstat_counter) { + update_mstat_ratio(mm_slot, msuccess, mfailed); + msuccess =3D mfailed =3D 0; + mstat_counter =3D MSTAT_UPDATE_FREQ; + } } } clean_list_handled: @@ -882,6 +928,12 @@ static void kmigrated_migrate_folio(void) } } =20 +/* Get scan_period based on migration failure statistics */ +static int kscand_mstat_scan_period(unsigned int scan_period, int fratio) +{ + return scan_period * (1 + fratio / 10); +} + /* * This is the normal change percentage when old and new delta remain same. * i.e., either both positive or both zero. @@ -928,6 +980,7 @@ static void kmigrated_migrate_folio(void) static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, unsigned long total, int target_node) { + int fratio; unsigned int scan_period; unsigned long now; unsigned long scan_size; @@ -967,6 +1020,8 @@ static inline void kscand_update_mmslot_info(struct ks= cand_mm_slot *mm_slot, } =20 scan_period =3D clamp(scan_period, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PE= RIOD_MAX); + fratio =3D kmigrated_get_mstat_fratio((&mm_slot->slot)->mm); + scan_period =3D kscand_mstat_scan_period(scan_period, fratio); scan_size =3D clamp(scan_size, KSCAND_SCAN_SIZE_MIN, KSCAND_SCAN_SIZE_MAX= ); =20 now =3D jiffies; --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2065.outbound.protection.outlook.com [40.107.237.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 306121B4248 for ; Thu, 14 Aug 2025 15:35:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185751; cv=fail; b=bOgclgOqvtIiIPEBy031a+OJDFVRBa9LgNdHXg/KBM4ZUGuV/2BZPgGRTuziuHOgocYrzWOxug54rndWIAwutNAy1WwQvVKpjjb2x6Fvs/xKincKkODbIF7CtNJilbFGgTCGVyWh3Ba4Y4QdYMcTJfZpXBhVr2QYKkhKSAooXlI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185751; c=relaxed/simple; bh=C7d7hJ57HNZRrRBCAGn9KzPkOtc8thPSB48IloYKBy4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=jwgkUNcRFfzMYA2vBBd1VZxMcXXOb/DmbbDxCOuIfE5v0BRfmdYD98xyw1LTNFbxLVmCK6F1e897vtSEAwD3kMuifRMjR5ozijdvrRS1oZa/BU7De1Ta5Ri5wuOZc/+Q4tlBMwFhHjMDVrkbLcNMXtel69fNHn5Up2UtRb1Sztg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=pk3n6CEr; arc=fail smtp.client-ip=40.107.237.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="pk3n6CEr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZrkZCU0dvGnL6qIuk0DUxxzktuN0bNe+Aq5VZlRR1YmMlX+T65cvfFYI0m8ODar9Wh4mz6nGPtNBhQhVohNJCzzPJCEs/yEWQzcwpPljKU4yG2Th8zXnVV1dMRUlyFw8OysfEITO5TT3GlJ6ArS9mxnbfd9IROpwE9fNlY19D0ku4rAveXf35nimXtMG2uLwRys/10NbbY65IuUldviVrG5PsKZ4EydLamzPSAVAb+jezOygvkGtKUIAQSi/fSdcO1i8pH4uAjB8YhT0jkmP6mbS7RFjY630gd5MR+4I9lgXZNQ15fkQ1J4gN5PnHtYdB0t7w9sch7ucQLNWHTPvhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IPvvE3+n2lSKn62J/m3/DT43sgyaoQup32DsQYT/oXE=; b=XvlCnk4vHVVWFRzX4QpA9Ebvj5rPH1D3q+gheT2umazdTFp1ZQKxY3bP3ZHrI99cC9038NL1cPmAzY+MmLlOw06V4Yva1IuZCVPoVj8Xe77CxwDfpGLRoRTvt7iL3o8a+5dO5M5hXCojVc4/5tNbIe6I5pE2UqoPwJDC6d1IgtZzjMrjqfQfYpJknpP6MAuXNbb6ApPwn5DRWGTG3qVuL9FISneb3FKGJJMSklNDT1Iku5B3sPizXutely58KgcgC1d+DhqXxDC0sa+ARCyOFOEeD29EyVwLIz4upFp0uvBAJo5xXymQdfZcZGTDkZkuxdDDBcigVNEdl2DsJqSZBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IPvvE3+n2lSKn62J/m3/DT43sgyaoQup32DsQYT/oXE=; b=pk3n6CErenQsOMAioHNtTcCJYuzK+vjvU0ZJ8AfTiVo0+nxJfY1fCQtpyKb7XIJlPT0/IFHwPzzUu8spOsBL33xmzHgCkPoHI5/N+pJpc6rSl4GgfHXxHitwz3N34l+LA4/G7WhlDqJreFDIJadzDxajbZsJvXD+Z+cP4+HN9AA= Received: from DM5PR07CA0064.namprd07.prod.outlook.com (2603:10b6:4:ad::29) by CH3PR12MB8076.namprd12.prod.outlook.com (2603:10b6:610:127::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.16; Thu, 14 Aug 2025 15:35:43 +0000 Received: from DS1PEPF00017098.namprd05.prod.outlook.com (2603:10b6:4:ad:cafe::c4) by DM5PR07CA0064.outlook.office365.com (2603:10b6:4:ad::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.18 via Frontend Transport; Thu, 14 Aug 2025 15:35:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS1PEPF00017098.mail.protection.outlook.com (10.167.18.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:35:43 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:35:28 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 12/17] sysfs: Add sysfs support to tune scanning Date: Thu, 14 Aug 2025 15:33:02 +0000 Message-ID: <20250814153307.1553061-13-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF00017098:EE_|CH3PR12MB8076:EE_ X-MS-Office365-Filtering-Correlation-Id: 7b8e64ff-ff82-4a95-0f12-08dddb483b63 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?fqXDWItc6tuJHbNP/b9NTlYyhtgPiCdjveOcq+z/fPlE5btmWKFPVAChdAdt?= =?us-ascii?Q?PtJgB+PiY6zoiQKko2Moz75ED1jkb2jp5e+1ulF/NWpEKMzJgAtNrw0Zm57x?= =?us-ascii?Q?TqImSGpdM3gw+z5mOABF7k6vnI1DHy74sSIYxxBZRSrsZOafvraz41VdYKTH?= =?us-ascii?Q?rniSsA5lreE+cLhCqjbwFJAmqsbkrhKVe4yTFEZakehsusXa5/8K+mIbuf4C?= =?us-ascii?Q?7U7u1mU5ElP37SFDuBWNIrGlB4aMv3wDXbfpP0FVoil5SGnxSPKkQUeUgWLf?= =?us-ascii?Q?WlREH9Uh5QpG0gJm/YkOcYqFPtVcSk09u8K6erD7vPliCw7Nqb2nonjy5sc1?= =?us-ascii?Q?DKFhRY6Z/WfaMX4GPc87n/DpnFL6mKdVNASZWGR5MdjzuLARs8+5r+qmLxGl?= =?us-ascii?Q?bYLKzabtfZO1CDDyLx7vqnwyDNhical5r0RG8QlnrAyYEE6KGM1HM2WvPKZ4?= =?us-ascii?Q?kzA5y0d+dsBt+GN9SuCNIwRJ9sR2qD1lkMJ32MAthH8IRt1LNUdRQSQghqyl?= =?us-ascii?Q?i8hqgGIH81+6/qO/7wOT8M/h6NGQgnvEYoVxElv039dCBoj45gtya9cInNnv?= =?us-ascii?Q?w5xa0NphpduJ3Gd1bhLlM6rNOuhejDLOik8qqkvgjKpAEYnlbCs3PIWE5MHZ?= =?us-ascii?Q?78KLFk1VvnfrGobJNo/kGzXeTbGPi2ecoUZvS/aVSkdARClzCjFGWHBWi97M?= =?us-ascii?Q?y0SUtlgPzVYZ+hTkZPH07zycqAEw3fbd/T/Zcm5n5aPMgF8qOEi8CCclcruy?= =?us-ascii?Q?Z1nw6NYFFKqmwLjZMyDDtL/uGwmuISw0rIbZoOtYFpL5kv8fID79kTCngwhC?= =?us-ascii?Q?DvpOtjYaJ9lnCJcD/tBfG0yNfvjPp55FTCLtVSSC6Cu7SUCDJyBNIPsiFloK?= =?us-ascii?Q?OgjpeZXohNIEJ/FCPvxGBhX1NzHSuB/CIgQnNrFi1RjVEom4Zvly5OhfrOid?= =?us-ascii?Q?BvWB9jxN0w9KaQPCcF2R+rcJ4Db2ZSmPBJut5Bkt0m63oXmfCzkTW0Lq+VBH?= =?us-ascii?Q?BKh3IdVjNMp4rMN1XZUQFqMyakasmZDwKLBLGq4/yuGmGXi4/GP6V+AmisZH?= =?us-ascii?Q?ocD6Bi1c7rfoFQsGz7p+iD4Ex3l1jw/vfhSAb2DmT5aJFzVA3qc/dgIDAxAp?= =?us-ascii?Q?wdzO4Si2uAxa5OCniZ80neDkvMvpLFn8lpBDop4Zp/jiA8KHfEpWVzhXafyP?= =?us-ascii?Q?Bt4DEqGijr+Ev9skO0qi1JuX7hkYj4YOUDEBQrS/dP36komeDqV2ZF3Hyaom?= =?us-ascii?Q?Ojayla4jsfpFuyBcN1PP5uHQ+ouUBgKMA5jMaBZ33FAq8gZH8cXVdquN81ER?= =?us-ascii?Q?IXpgmnoN+WJXBa3MchauBj7RGi1Hj5eP2YKltLIjNYA5tpkPuRnp5KK0MJYH?= =?us-ascii?Q?0UJIbpBy+szHvyDJuCEkRDok5mNyAzxuShjZ7EdFxmLfIGYJFbVyJReaTcyj?= =?us-ascii?Q?RzqzBjdBAB2WeqVAVzdKv8Yso19vggwRibJKtTac/wpA/PMO6slgKPfVpFJ3?= =?us-ascii?Q?0Dc0mdY5QxvG4izEMdq8K1wXRqVk+9pSlbcv?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:35:43.6135 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7b8e64ff-ff82-4a95-0f12-08dddb483b63 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF00017098.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8076 Content-Type: text/plain; charset="utf-8" Support below tunables: scan_enable: turn on or turn off mm_struct scanning scan_period: initial scan_period (default: 2sec) scan_sleep_ms: sleep time between two successive round of scanning and migration. mms_to_scan: total mm_struct to scan before taking a pause. target_node: default regular node to which migration of accessed pages is done (this is only fall back mechnism, otherwise target_node heuristic is used). Signed-off-by: Raghavendra K T --- mm/kscand.c | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 205 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index 41321d373be7..a73606f7ca3c 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -21,6 +21,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -173,6 +174,171 @@ static bool kscand_eligible_srcnid(int nid) return !node_is_toptier(nid); } =20 +#ifdef CONFIG_SYSFS +static ssize_t scan_sleep_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_scan_sleep_ms); +} + +static ssize_t scan_sleep_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + kscand_scan_sleep_ms =3D msecs; + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute scan_sleep_ms_attr =3D + __ATTR_RW(scan_sleep_ms); + +static ssize_t mm_scan_period_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_mm_scan_period_ms); +} + +/* If a value less than MIN or greater than MAX asked for store value is c= lamped */ +static ssize_t mm_scan_period_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs, stored_msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + stored_msecs =3D clamp(msecs, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PERIOD_= MAX); + + kscand_mm_scan_period_ms =3D stored_msecs; + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute mm_scan_period_ms_attr =3D + __ATTR_RW(mm_scan_period_ms); + +static ssize_t mms_to_scan_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%lu\n", kscand_mms_to_scan); +} + +static ssize_t mms_to_scan_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned long val; + int err; + + err =3D kstrtoul(buf, 10, &val); + if (err) + return -EINVAL; + + kscand_mms_to_scan =3D val; + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute mms_to_scan_attr =3D + __ATTR_RW(mms_to_scan); + +static ssize_t scan_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_scan_enabled ? 1 : 0); +} + +static ssize_t scan_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int val; + int err; + + err =3D kstrtouint(buf, 10, &val); + if (err || val > 1) + return -EINVAL; + + if (val) { + kscand_scan_enabled =3D true; + need_wakeup =3D true; + } else + kscand_scan_enabled =3D false; + + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute scan_enabled_attr =3D + __ATTR_RW(scan_enabled); + +static ssize_t target_node_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_target_node); +} + +static ssize_t target_node_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int err, node; + + err =3D kstrtoint(buf, 10, &node); + if (err) + return -EINVAL; + + kscand_sleep_expire =3D 0; + if (!node_is_toptier(node)) + return -EINVAL; + + kscand_target_node =3D node; + wake_up_interruptible(&kscand_wait); + + return count; +} +static struct kobj_attribute target_node_attr =3D + __ATTR_RW(target_node); + +static struct attribute *kscand_attr[] =3D { + &scan_sleep_ms_attr.attr, + &mm_scan_period_ms_attr.attr, + &mms_to_scan_attr.attr, + &scan_enabled_attr.attr, + &target_node_attr.attr, + NULL, +}; + +struct attribute_group kscand_attr_group =3D { + .attrs =3D kscand_attr, + .name =3D "kscand", +}; +#endif + static inline int kscand_has_work(void) { return !list_empty(&kscand_scan.mm_head); @@ -1231,11 +1397,45 @@ static int kscand(void *none) return 0; } =20 +#ifdef CONFIG_SYSFS +extern struct kobject *mm_kobj; +static int __init kscand_init_sysfs(struct kobject **kobj) +{ + int err; + + err =3D sysfs_create_group(*kobj, &kscand_attr_group); + if (err) { + pr_err("failed to register kscand group\n"); + goto err_kscand_attr; + } + + return 0; + +err_kscand_attr: + sysfs_remove_group(*kobj, &kscand_attr_group); + return err; +} + +static void __init kscand_exit_sysfs(struct kobject *kobj) +{ + sysfs_remove_group(kobj, &kscand_attr_group); +} +#else +static inline int __init kscand_init_sysfs(struct kobject **kobj) +{ + return 0; +} +static inline void __init kscand_exit_sysfs(struct kobject *kobj) +{ +} +#endif + static inline void kscand_destroy(void) { kmem_cache_destroy(kscand_slot_cache); /* XXX: move below to kmigrated thread */ kmem_cache_destroy(kmigrated_slot_cache); + kscand_exit_sysfs(mm_kobj); } =20 void __kscand_enter(struct mm_struct *mm) @@ -1421,6 +1621,10 @@ static int __init kscand_init(void) return -ENOMEM; } =20 + err =3D kscand_init_sysfs(&mm_kobj); + if (err) + goto err_init_sysfs; + init_list(); err =3D start_kscand(); if (err) @@ -1437,6 +1641,7 @@ static int __init kscand_init(void) =20 err_kscand: stop_kscand(); +err_init_sysfs: kscand_destroy(); =20 return err; --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2074.outbound.protection.outlook.com [40.107.223.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF7E1225A4F for ; Thu, 14 Aug 2025 15:35:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185758; cv=fail; b=gGylKDSAEr6o+5pyIOCzOoC+7NDLHcu330j7Xv6xfOwTcJYZwEJxEpsFTy+2MEc62UGuJ8WJh1917/79aCCyzEOANVBfcjqLwA1OjFNg0eG/9BIb+wu5XgY6YPZGo4i08nQXQfeK+XDuwpEjZC4JNI67bY1DLKXhZPaVlBccPGE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185758; c=relaxed/simple; bh=a2VjCsMDF1M9pxkDJhsM/9s6/6Su7pJL9W2C0WlJ6/o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kp68b2bpGyLpQBnPStwAZiqI6y10pD9mUbrABaj3+ntUNzmoqEG0U6rWmpiZyajFHtXYOgOA99kDo29bu+gKHXddaf0tH9ngk8IsREpembwlU1eqiSgC4nTm6DhvXkMYCU08jdEBZna/BRPZzxoM/6fJ+sYQAgPXHwKxZG4UE0Y= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=IWs6HFFr; arc=fail smtp.client-ip=40.107.223.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="IWs6HFFr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=STBahrl88GG1Dw7Xn9NnhYJ0AO4Z9sxo5dCW9ARRtb9wejqFICJgCThV2lKIiLw/+hiH0MMSSLAvJwamRt1zLvXuPC3apNTSeQASjiXI//KBUeF6xlJeInnovlduKHBdJIvgXqMdRQ7g3LcHkzan4Im9jWaIvFMR9tE2rVHZgSw+1p80z/Fod+jlCiE2cJ1T5k3YouUOLJ6CGqL7o5/wNNUtt5DERD8yziorGxu9I4kDlDT6cuTmPyne54+ZLSOEzqG/qQcYdI5NmBfXo89cDI6FXWZXUG6E9OUoHUktapG3+4YAVe/BRRS0pRr6UVpDNCq/MUiT4hJSir7/mC97aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9hE+4BNyN+gD/VwJCW/xU/rYEKf9T8D51JEHv+tgwhY=; b=FcqWg9XBgMltvTk78PAjMsIw2RIo7rqfjZZtAxu1cO5qAuNBu21ictaFcDhdP2QZoSiL1QDCqMjNj8EX65uHRxBatsQIE3SBI1a4US8z7f5rrapeiQqdcW7lc5tQKFV9SIg2I4oRFYQIf8kviem3aGtgeGkOwZa2RVESvsNVHQ02aYtIHymPuhSDZDQv8xXS3dW5myvA6iSymvSRNprOrrhlMxBzIcIzMhwXvxcpw/MLx9snfuhidjlurDJ+zX+9ehZPVKc48Algd1WCUMqCcHfitoL9XcymoLUF7QoN+ym1Om8u1JbNp4AThqUf+klmOTMZiMlpLJ4Yc9wo2j53tw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9hE+4BNyN+gD/VwJCW/xU/rYEKf9T8D51JEHv+tgwhY=; b=IWs6HFFr4yqec5o3FS6ht58dm8o33a1nw2+AQ1c17ONX0UTVIrFONibkVV8e1dUOWDA0tSYtS5Pw3xjDV9fuKQpElhhC1hurvVz7HFXs5T+tnrbvdgv8D54d3qR3VInD33TYf0hkVmZrfjjNP1EWQBzYBnWVAUvwHCC9P21bsz0= Received: from DM5PR07CA0075.namprd07.prod.outlook.com (2603:10b6:4:ad::40) by PH8PR12MB7279.namprd12.prod.outlook.com (2603:10b6:510:221::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.15; Thu, 14 Aug 2025 15:35:53 +0000 Received: from DS1PEPF00017098.namprd05.prod.outlook.com (2603:10b6:4:ad:cafe::58) by DM5PR07CA0075.outlook.office365.com (2603:10b6:4:ad::40) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:35:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS1PEPF00017098.mail.protection.outlook.com (10.167.18.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:35:52 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:35:39 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 13/17] mm/vmstat: Add vmstat counters Date: Thu, 14 Aug 2025 15:33:03 +0000 Message-ID: <20250814153307.1553061-14-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF00017098:EE_|PH8PR12MB7279:EE_ X-MS-Office365-Filtering-Correlation-Id: ffdcd589-b38e-4b0f-640d-08dddb4840dc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?uIjbIYhhnyNi1qXMJlwYkw+KmmJxXnfu/AWeZQUjjRl/JMvFDAtLQsOyZtlv?= =?us-ascii?Q?+QyCrtDRUA+YEqJliZLKmC+fnmXErmHhfHhmoq1vp1FZivPYcQluRCtra9m7?= =?us-ascii?Q?H4qJMCJ8zcBnPu2aaBgzYnFKEBRxqmXyO5VsanaMsMtnA1RdZ330ufpZ63YD?= =?us-ascii?Q?0ICCkRBdTYWVe4dHKioB1TAxwWvIKfFntjHzTqncJmMzgy1jw3sPviEc4PRZ?= =?us-ascii?Q?qLPmSfI4ixHJhYo2Vp3tSrRULZyujWUzYXeD5tOL5zbYCJPXJgTgyJGpGdO0?= =?us-ascii?Q?z1YHi1vYQoD9JxR95oGdpOhQKTO4pHxBwZHxM6NZF/w9OhTF6Xar+eWXn79R?= =?us-ascii?Q?PjGghqBoAPtNgONA6xXf1AOTQqlzQkM2MgQevv067xb5jKOG0PleAI+Pk1ml?= =?us-ascii?Q?WwZjeqfh5sDm1RAKnYk8bHsweao54TcFo2T8zMYMQUrpT6Ot/ZsjfUKcbUZS?= =?us-ascii?Q?y5FzkH6LHOK1A7GIP2w18EyMWtdv6tLrh0D5IPIcuzpHTWjCISiKNfAu7SAM?= =?us-ascii?Q?IVLLro73e8wfILailNpQb9RAztA0GkyN93FkvC53MTMdS7Iu7sgwygIdgGlv?= =?us-ascii?Q?fi6xJn23O0xVyEysYJ4wt78hZVjDC30p4qjU9kzigt0QaGGigQQ0XNYgvNXT?= =?us-ascii?Q?oyO+a/ojpV7pR/FL8e7FTSu6Q6VlFviabPfB5EFGApTaX9ESS3bLZn0s5AnB?= =?us-ascii?Q?ji9Ao+QyTivVElQO9XMV3Mx4wSlzZnb1ERcBVEgrP1l4mv+xvhuG1AIyvOnM?= =?us-ascii?Q?ajHXmqgNdlKY5BNTerlj1crj0gKI9v0hT08UTQeOt3WkaAH8TD+B6G92TeMh?= =?us-ascii?Q?6AjusS8BBpQ3ropfBXSeIrBGHTQ6Z+4Sek/XHuJ1fg/Ndtb0xwRKiFxSX+fT?= =?us-ascii?Q?mDO9o8ojKFC3+N8IAyg6YuICOZSl0/8ZUZ/X/7WdTcMTUoM1WdQpKYzwCIIq?= =?us-ascii?Q?gYCWX3WDmQbkwgbVLsQScdtbr4ovp+TbeykKfOWzTC6Hf1T5ScFod+Rk2Nh1?= =?us-ascii?Q?6+r4TrdE+4oFKLLaCnoWtREjQdZeWwrlwosM4BjpRRWx1kD+fVMkBoHGWs/m?= =?us-ascii?Q?CDzYiMbejnSs8a+5JgPXH5MPwCEL+TUlbWOXJ2OKYAIWnNQRrGoiT3qKFwLn?= =?us-ascii?Q?Ftz8zFZHccG1rrZe1Lfbmsg03mYx1zq42fuNhplau5wJBtCQDU6OVa2uHJrd?= =?us-ascii?Q?Jr+Wo2RdlSXiD8UeRARzeRNg2LBO7a3PRCEF7bM1rmpyNN4JpRXcHUOKpcIL?= =?us-ascii?Q?KONFU6hSxjFcyGMqpNa6fQCICftzXB1L0RRhAyz2jbW2UVrnlwvkv/TcWsrB?= =?us-ascii?Q?73AQcLxblRWDYuB+dSWYvkUzQ1ECvjPB6AlcMN/+Hw2a6KMFp/dZYDUtNgJI?= =?us-ascii?Q?mN+XS5RBKNN5GRudRk6waVsAuTIUYQvGa4TvGH2YTA6DWbxOoEREEFxFltxH?= =?us-ascii?Q?S9slVoBgBFDKq9M7IJud5MUPzo2DfExcZsERd3EDCTwqBUYfjZeSPAYCt/AP?= =?us-ascii?Q?ypkqDOFBI0ixakNxC69LJC/MVsEmKeDhJddJ?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:35:52.7976 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ffdcd589-b38e-4b0f-640d-08dddb4840dc X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF00017098.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7279 Content-Type: text/plain; charset="utf-8" Add vmstat counter to track scanning, migration and type of pages. Signed-off-by: Raghavendra K T --- include/linux/mm.h | 13 ++++++++ include/linux/vm_event_item.h | 12 +++++++ mm/kscand.c | 63 +++++++++++++++++++++++++++++++++-- mm/vmstat.c | 12 +++++++ 4 files changed, 98 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fa538feaa8d9..0d579d0294bf 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -655,6 +655,19 @@ struct vm_operations_struct { unsigned long addr); }; =20 +#ifdef CONFIG_KSCAND +void count_kscand_mm_scans(void); +void count_kscand_vma_scans(void); +void count_kscand_migadded(void); +void count_kscand_migrated(void); +void count_kscand_migrate_failed(void); +void count_kscand_slowtier(void); +void count_kscand_toptier(void); +void count_kscand_idlepage(void); +void count_kscand_hotpage(void); +void count_kscand_coldpage(void); +#endif + #ifdef CONFIG_NUMA_BALANCING static inline void vma_numab_state_init(struct vm_area_struct *vma) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 9e15a088ba38..b5643be5dd94 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -67,6 +67,18 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, #endif +#ifdef CONFIG_KSCAND + KSCAND_MM_SCANS, + KSCAND_VMA_SCANS, + KSCAND_MIGADDED, + KSCAND_MIGRATED, + KSCAND_MIGRATE_FAILED, + KSCAND_SLOWTIER, + KSCAND_TOPTIER, + KSCAND_IDLEPAGE, + KSCAND_HOTPAGE, + KSCAND_COLDPAGE, +#endif #ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, THP_MIGRATION_SUCCESS, diff --git a/mm/kscand.c b/mm/kscand.c index a73606f7ca3c..e14645565ba7 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -339,6 +339,47 @@ struct attribute_group kscand_attr_group =3D { }; #endif =20 +void count_kscand_mm_scans(void) +{ + count_vm_numa_event(KSCAND_MM_SCANS); +} +void count_kscand_vma_scans(void) +{ + count_vm_numa_event(KSCAND_VMA_SCANS); +} +void count_kscand_migadded(void) +{ + count_vm_numa_event(KSCAND_MIGADDED); +} +void count_kscand_migrated(void) +{ + count_vm_numa_event(KSCAND_MIGRATED); +} +void count_kscand_migrate_failed(void) +{ + count_vm_numa_event(KSCAND_MIGRATE_FAILED); +} +void count_kscand_slowtier(void) +{ + count_vm_numa_event(KSCAND_SLOWTIER); +} +void count_kscand_toptier(void) +{ + count_vm_numa_event(KSCAND_TOPTIER); +} +void count_kscand_idlepage(void) +{ + count_vm_numa_event(KSCAND_IDLEPAGE); +} +void count_kscand_hotpage(void) +{ + count_vm_numa_event(KSCAND_HOTPAGE); +} +void count_kscand_coldpage(void) +{ + count_vm_numa_event(KSCAND_COLDPAGE); +} + static inline int kscand_has_work(void) { return !list_empty(&kscand_scan.mm_head); @@ -653,6 +694,8 @@ static int kmigrated_promote_folio(struct kscand_migrat= e_info *info, =20 if (!is_hot_page(folio)) return KSCAND_NOT_HOT_PAGE; + else + count_kscand_hotpage(); =20 folio_get(folio); =20 @@ -803,12 +846,15 @@ static int hot_vma_idle_pte_entry(pte_t *pte, } =20 if (!kscand_eligible_srcnid(srcnid)) { + count_kscand_toptier(); if (folio_test_young(folio) || folio_test_referenced(folio) || pte_young(pteval)) { scanctrl->nodeinfo[srcnid]->nr_accessed++; } folio_put(folio); return 0; + } else { + count_kscand_slowtier(); } if (!folio_test_idle(folio) && !prev_idle && (folio_test_young(folio) || folio_test_referenced(folio))) { @@ -820,7 +866,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, info->pfn =3D folio_pfn(folio); info->address =3D addr; list_add_tail(&info->migrate_node, &scanctrl->scan_list); + count_kscand_migadded(); } + folio_put(folio); + return 0; + } else { + if (prev_idle) + count_kscand_coldpage(); + count_kscand_idlepage(); } =20 folio_put(folio); @@ -1045,10 +1098,13 @@ static void kmigrated_migrate_mm(struct kmigrated_m= m_slot *mm_slot) mstat_counter--; =20 /* TBD: encode migrated count here, currently assume folio_nr_pages */ - if (!ret) + if (!ret) { + count_kscand_migrated(); msuccess++; - else + } else { + count_kscand_migrate_failed(); mfailed++; + } =20 kfree(info); =20 @@ -1269,6 +1325,7 @@ static unsigned long kscand_scan_mm_slot(void) =20 for_each_vma(vmi, vma) { kscand_walk_page_vma(vma, &kscand_scanctrl); + count_kscand_vma_scans(); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 if (vma_scanned_size >=3D mm_slot_scan_size || @@ -1304,6 +1361,8 @@ static unsigned long kscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + count_kscand_mm_scans(); + total =3D get_slowtier_accesed(&kscand_scanctrl); target_node =3D get_target_node(&kscand_scanctrl); =20 diff --git a/mm/vmstat.c b/mm/vmstat.c index a78d70ddeacd..5ba82d2ffe71 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1347,6 +1347,18 @@ const char * const vmstat_text[] =3D { "numa_hint_faults_local", "numa_pages_migrated", #endif +#ifdef CONFIG_KSCAND + "nr_kscand_mm_scans", + "nr_kscand_vma_scans", + "nr_kscand_migadded", + "nr_kscand_migrated", + "nr_kscand_migrate_failed", + "nr_kscand_slowtier", + "nr_kscand_toptier", + "nr_kscand_idlepage", + "nr_kscand_hotpage", + "nr_kscand_coldpage", +#endif #ifdef CONFIG_MIGRATION "pgmigrate_success", "pgmigrate_fail", --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2043.outbound.protection.outlook.com [40.107.93.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0EC4225A4F for ; Thu, 14 Aug 2025 15:36:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.43 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185765; cv=fail; b=YSBmiC3++1Il0jJ2rxKYibwHGtf4JC2FjtraHUhuBZAxrgWpGOAXtkRMDizEG1Mj3LLppw6vhAnC1Qv6DNOMhui43uV8xD5/h0EFCMLGj5pAKJHJ2sF09b5J6GfyZk57E8XlZ9Y6XCNWUHHrlGe3UAqocbD/5+t2nEzWTC6hlTo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185765; c=relaxed/simple; bh=v04Ch/l/Vr8PZBK4bJHFng7PrscFrW6navB/6nQLHOI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rs/8TAEMDgK05nDr60yDmCVz+RWiSgGnyUUYFECGThgX6k9lq4LVgvhyik93yHvp8NGvXGU8Ce8IuotZbLodmwHaNw4VlP4utULDSZxoQt+AVaAPPt14s1fit+BfoDND9Ejlyfcs+m9MrCWBSy6zLZVSQOlg2t/5bDNktnSxzOM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=fj7F6dtc; arc=fail smtp.client-ip=40.107.93.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="fj7F6dtc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HdjRq4vMme31BjUYUUFds10QKRUY43EJxV3LIazOrzVdt4Xd8bDqq1lwbLPj2gUg8IdegCT0cKaQkfgp8AM8UF+8T/aYLKA3M5LD4Jp/BYIp4nn9wG+bzN9KuFqrIwx559mrcq/DXywJGPUl2LFRIMoPU4CTon3tO0mM13Q4ilcfA3wZqgJQ6SYxnpzRGcdWVECSx5acEZwuXfWu2ob8DyWwwj7p/NT+J0dKygjxGuwLE/76L1hOHVWNVKh2tewwjp5yyL6yTGdzFMTvoW6BAAoy/E2evN9/a7jfj3RCE8jP+H2Dzy2IPfcV4PBOyXOg3fcanLyGy7FETBebxnjfyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZaWbpDRzrNxNxl+NAvt/vlQzbosqTSFRGmvuYGTslqs=; b=MNb8yG4lv6OJXioU4u4VQLohtoKgXc8pBGEhb8Cv0xEqjwHWelc+dRrvybZupkGEv9KeJ271xB364MRY9ostm+CsaA5TvjPhlq5wuSJrV/d3xCmYTLhB5z5IaV59OWPAD5FRsTmjOxdYbxppxbV4Y5cOTkjq57aMHp5J16c6bVWfSJ7wr9Cdjr+NetHqIuVXsXqzz83SHLxXFuT27gRE5X+fFGG1EAxOMFHp6yABig9CFV76LMs8ALVgx+tiPE7jKNWiuYeQeGFqZPrwg3UK50dK8JSTCyZJZUpe1l0LEAxThwzOlRd7MtnkrF2d0DwqdAb+4GUFkR0solHppgmKwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZaWbpDRzrNxNxl+NAvt/vlQzbosqTSFRGmvuYGTslqs=; b=fj7F6dtcMqAV2ddLzFTCWq5wYEAWwifcCjIYbW8Xgphj9GoM4dcj9EcRnPx/mDTEvFTA4YuwJL0oy8Z+/yf3rp84dPSxnkgqMXd2AxBkbxtaXzMxBCwbfFqEKsJqL04yRewECr6K3hHG6LEBfeBfkBlYrPggQUhL0FgXVA+ybcc= Received: from DM6PR11CA0039.namprd11.prod.outlook.com (2603:10b6:5:14c::16) by MN2PR12MB4269.namprd12.prod.outlook.com (2603:10b6:208:1d4::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.15; Thu, 14 Aug 2025 15:36:00 +0000 Received: from DS1PEPF00017098.namprd05.prod.outlook.com (2603:10b6:5:14c:cafe::65) by DM6PR11CA0039.outlook.office365.com (2603:10b6:5:14c::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.18 via Frontend Transport; Thu, 14 Aug 2025 15:36:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS1PEPF00017098.mail.protection.outlook.com (10.167.18.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:36:00 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:35:49 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 14/17] trace/kscand: Add tracing of scanning and migration Date: Thu, 14 Aug 2025 15:33:04 +0000 Message-ID: <20250814153307.1553061-15-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF00017098:EE_|MN2PR12MB4269:EE_ X-MS-Office365-Filtering-Correlation-Id: 9f0a5301-2d55-4991-bb69-08dddb484570 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?gKxrV5m1TAr+tL5NuToSy3M4E/550rgARs8irH/6NtPX/jOW7MY0XwwQYtyN?= =?us-ascii?Q?uSUst2/iuHxEXscVnSIPDRXoXAXhhnc2z+5Ddx1lcOyjnyyKWJ62m5ii8+k7?= =?us-ascii?Q?aSgZT8zhrr3qcRU1ac+55zNjN1nJjw3T7yYMVGNNmNTn8cEzNSMHRqoPz7VK?= =?us-ascii?Q?2FDh52QYaFdU+LA6lE0OsueKhwm6kpDSVFAFcHpfB6f1RilTbDTIpG/FK9uG?= =?us-ascii?Q?IezFWZImNIIup+yMPH/yaD4UH7ARROuwXfx+ZUw5VfU8hRRwn8lQGp8za+8E?= =?us-ascii?Q?s1QtWCAkdGzvBsKLfpXeSKB0xIgs/Dsl+Ii98UCkmni7/brXoKXNjwCMSWLY?= =?us-ascii?Q?k+giVFblE2RtSwyIN0zW9n5aPjIdjKCfAu9ssN+Wvc4x0cb+K3Nk13pBwy7+?= =?us-ascii?Q?jD6RLdRQ1qw4UL6PIyBSVDrcl95KZ/Ad1BkKlNTLMmGsS9ncWZXL3Bu2t2us?= =?us-ascii?Q?MlWjY2awtE2s/hGXcx2sfEkQo08k3lT5Le+yrbi0UNVhwfDwCCaaODHbRCfa?= =?us-ascii?Q?BeJRmjy2gqt9HUdOQ9U1vpU5rwdB8tX/167I44jsTvfyDPBy232WzUAjfNp2?= =?us-ascii?Q?5tn9QhoOPSmb6Tu8FKXogS/9auJwbfHU7038Sx7YCbUvdBc7wbs3CiiAvl1s?= =?us-ascii?Q?M0vF2TRhMm5VpWc5Imr51qz5Z01KmSBuDCULKx+5uqmYl2/N2C+u2Gdd8x26?= =?us-ascii?Q?sYppNibWJm2LgzXLCPWeOxtayIqouFQlS8m3/mJgJ4azTUe6E2Lrwn5egGhz?= =?us-ascii?Q?zXhbeaMcQbIGou0TBD5nw3tj2CpLz6Qjf2s8TXCJ7TQchtEL/G21veFqZSyB?= =?us-ascii?Q?OhN1Gkx5qnyO3/E5okeLkxfuGvo/pmiluj03HFVDePe7JDRBeVADuI6ZMlw/?= =?us-ascii?Q?E3b6rZfZc8h1sHeKQFIledDjNgMXQ/IB6x5DqE7kldPW665hevrsOzHgphaj?= =?us-ascii?Q?OQQgAC7xaWS2uuC8wEhQt2K5wMIUENJPepHBwga7yI7ohRteCHKEqDwxFHIH?= =?us-ascii?Q?VBnSA+IDls+7tBp7QRUcqCybCUm+z9Ga+zCM1R6+GfuHWCiJvYDr1p4XmIhq?= =?us-ascii?Q?E3V2fpnQ8UGH4JA2iGXdAMjyW6f8BA8HH/gxlKz7LAParrGRID/BbDhkP6gV?= =?us-ascii?Q?kCuGI14O6sKiKothSR2UMBW3j1AWpbEjnc3TgCN5Iu6sdyaYj5zErYZ5D26a?= =?us-ascii?Q?KAnVl7t3Mdb0LGOasY0lxF0NxtJ31md5Ja+oGK34T3l1NBwo+v1h311qAbSA?= =?us-ascii?Q?9xrdU24l+uiZcO5qvASHtnfQA7FO/tCdIarEEZvC3BFnnas0zYbL6iBTsNDy?= =?us-ascii?Q?k99+9obBPKqmkvC6L3nHUHW30MNTqc+nwWPv/Xoy/PtSOdBaBSzUqhH2PbxC?= =?us-ascii?Q?/fiOneNaZ7vrfOt5iFhtaxOPRb2PjLTDwNNeEngxAcVDvG6moYZ4BwMK5Ke3?= =?us-ascii?Q?hiL3W5iU2iieikUUb273LvnZ/cP/rwuOg9OqeCIyp8bfBO1wdBvbNLWNkDYO?= =?us-ascii?Q?hyMrswbsRozGViDPPpewZgWsSe68feCGQ3Or?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:36:00.4781 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9f0a5301-2d55-4991-bb69-08dddb484570 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF00017098.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4269 Content-Type: text/plain; charset="utf-8" Add tracing support to track - start and end of scanning. - migration. CC: Steven Rostedt CC: Masami Hiramatsu CC: linux-trace-kernel@vger.kernel.org Signed-off-by: Raghavendra K T --- include/trace/events/kmem.h | 99 +++++++++++++++++++++++++++++++++++++ mm/kscand.c | 9 ++++ 2 files changed, 108 insertions(+) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index f74925a6cf69..d6e544b067b9 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -9,6 +9,105 @@ #include #include =20 +#ifdef CONFIG_KSCAND +DECLARE_EVENT_CLASS(kmem_mm_class, + + TP_PROTO(struct mm_struct *mm), + + TP_ARGS(mm), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + ), + + TP_printk("mm =3D %p", __entry->mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_enter, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_exit, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_scan_mm_start, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +TRACE_EVENT(kmem_scan_mm_end, + + TP_PROTO( struct mm_struct *mm, + unsigned long start, + unsigned long total, + unsigned long scan_period, + unsigned long scan_size, + int target_node), + + TP_ARGS(mm, start, total, scan_period, scan_size, target_node), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( unsigned long, start ) + __field( unsigned long, total ) + __field( unsigned long, scan_period ) + __field( unsigned long, scan_size ) + __field( int, target_node ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + __entry->start =3D start; + __entry->total =3D total; + __entry->scan_period =3D scan_period; + __entry->scan_size =3D scan_size; + __entry->target_node =3D target_node; + ), + + TP_printk("mm=3D%p, start =3D %ld, total =3D %ld, scan_period =3D %ld, sc= an_size =3D %ld node =3D %d", + __entry->mm, __entry->start, __entry->total, __entry->scan_period, + __entry->scan_size, __entry->target_node) +); + +TRACE_EVENT(kmem_scan_mm_migrate, + + TP_PROTO(struct mm_struct *mm, + int rc, + int target_node, + int msuccess, + int mfailed), + + TP_ARGS(mm, rc, target_node, msuccess, mfailed), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( int, rc ) + __field( int, target_node ) + __field( int, msuccess ) + __field( int, mfailed ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + __entry->rc =3D rc; + __entry->target_node =3D target_node; + __entry->msuccess =3D msuccess; + __entry->mfailed =3D mfailed; + ), + + TP_printk("mm =3D %p rc =3D %d node =3D %d msuccess =3D %d mfailed =3D %d= ", + __entry->mm, __entry->rc, __entry->target_node, + __entry->msuccess, __entry->mfailed) +); +#endif + TRACE_EVENT(kmem_cache_alloc, =20 TP_PROTO(unsigned long call_site, diff --git a/mm/kscand.c b/mm/kscand.c index e14645565ba7..273306f47553 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -1105,6 +1105,7 @@ static void kmigrated_migrate_mm(struct kmigrated_mm_= slot *mm_slot) count_kscand_migrate_failed(); mfailed++; } + trace_kmem_scan_mm_migrate(mm, ret, dest, msuccess, mfailed); =20 kfree(info); =20 @@ -1308,6 +1309,9 @@ static unsigned long kscand_scan_mm_slot(void) mm_target_node =3D READ_ONCE(mm->target_node); if (mm_target_node !=3D mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node); + + trace_kmem_scan_mm_start(mm); + now =3D jiffies; =20 if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -1378,6 +1382,9 @@ static unsigned long kscand_scan_mm_slot(void) kscand_update_mmslot_info(mm_slot, total, target_node); } =20 + trace_kmem_scan_mm_end(mm, address, total, mm_slot_scan_period, + mm_slot_scan_size, target_node); + outerloop: /* exit_mmap will destroy ptes after this */ mmap_read_unlock(mm); @@ -1530,6 +1537,7 @@ void __kscand_enter(struct mm_struct *mm) spin_unlock(&kscand_mm_lock); =20 mmgrab(mm); + trace_kmem_mm_enter(mm); if (wakeup) wake_up_interruptible(&kscand_wait); } @@ -1540,6 +1548,7 @@ void __kscand_exit(struct mm_struct *mm) struct mm_slot *slot; int free =3D 0, serialize =3D 1; =20 + trace_kmem_mm_exit(mm); spin_lock(&kscand_mm_lock); slot =3D mm_slot_lookup(kscand_slots_hash, mm); mm_slot =3D mm_slot_entry(slot, struct kscand_mm_slot, slot); --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2079.outbound.protection.outlook.com [40.107.244.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6027225A4F for ; Thu, 14 Aug 2025 15:36:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.79 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185777; cv=fail; b=fvWp/Dl539nSqcJgwzxttxDZCbCUnM2cvWcWNKgaImGTtgyztAY0XzT8kh1CqQaeubc8gu3/ZJMvUiw+srrXROrpzPUopwlSKwM1bpK9YdXGcaCu4Ruv42N0UBrekXHWM9mAi8Z4Kf0F38vc7UNUFz7GjM4f92T99OlUQzzKWtI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185777; c=relaxed/simple; bh=ZrcXyqH0b8fwpD6YPmb020LxxuZBeGnP3WJBeg/Su8E=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QkKv/mfl4yyNCPWaCv29dv8wpJzUSSJ/l6HqkgTBO7XTNrcRq7QVy7oc1zKneUWLL1xf1U6polYuJVlpaeSavPdZJG5J8C1frIBmLKS0MvXicaibFh0CHqPd1bgZcJ+hToqv0bRBN3e6xG9rItLbvKn3MY5OFZ9xTqYh7qI//90= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RpCTePc+; arc=fail smtp.client-ip=40.107.244.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RpCTePc+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ibiof957zINn5uoM5E9B/vDUdCdXXjjP4V9HuOPc8dT0WZXqXZrNaqvfziZj7rtwegIqfzufTP0ohbpdBhAPxyGs4cZY0wGjzpCoh2SNLOKj4liLiW+cWHNZKbhyrz4p5icBVbzCrqfOIqtZEYNSx00x5ObzGenYTyx5hVD5JFdQWzPCBlTWz5J2MysiyKza2US+Xqj/lwt80U7G1m8jbh+s/+S9WCHbgLyWFa/R4oVcCY2hV/NN7qLLJGNNHLrn8ObwpUuLeEs5o8KsFwEzVjPzo+51m55idTnklqrUea3nhF+AYyql240IAhsTaZkA1H22EAmxtl4Rsg8cDC2isA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tH6WDjy9Gx73HB03x6/iHMIinsfhhhAxSTl2toAIzd0=; b=aqt2iCANuvtX234/Z407lYgGNrnS0ZBa+AgKGWbiMSmEYT4bGgWT+2SSKx0A1DBTUuVg8yU24nTVOaXR71xb4d73/mYANFeC50s1DFn10vaNBM1NTOcE6dmT2+dXWqAMtsoN0FGd3PzP3To8vs2+F/Z1v4zratn0rMx5XHRAtPpYTdclUtaQxNk3fiAANPMLzFMk0AMP5w6AL/NMoJJj/oDNPgTUt5MZVqLZ27dQO9/UhTKm19ki5m0Clx07gPWJdWllMBDYwAIZSlf4QxbTLD2uLdJbCvSbR8fjMZxLH0JR2UnZ4Z3H4nhVVSAwqw2jDyYlzWh5p15/G50ZTgx1Yg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tH6WDjy9Gx73HB03x6/iHMIinsfhhhAxSTl2toAIzd0=; b=RpCTePc+sCJ9y+6HCyWoZWQPDboHNowAkeihTLEJ0iYZEa+3GtM4Dw1oGabhSRGc8bjWBhrXkqva1AfjzuSNiDZr7p1qp/vt7lLJ+vgCrWf1Y19iOL22tTyP30BkhUscHj5Nyb6MGIxssbSK/ltrc6msn1mkcaRRuzLanBCNp9Q= Received: from DS7P220CA0011.NAMP220.PROD.OUTLOOK.COM (2603:10b6:8:1ca::16) by DM4PR12MB5820.namprd12.prod.outlook.com (2603:10b6:8:64::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.21; Thu, 14 Aug 2025 15:36:11 +0000 Received: from DS1PEPF00017097.namprd05.prod.outlook.com (2603:10b6:8:1ca:cafe::b) by DS7P220CA0011.outlook.office365.com (2603:10b6:8:1ca::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.18 via Frontend Transport; Thu, 14 Aug 2025 15:36:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS1PEPF00017097.mail.protection.outlook.com (10.167.18.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:36:11 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:36:00 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 15/17] prctl: Introduce new prctl to control scanning Date: Thu, 14 Aug 2025 15:33:05 +0000 Message-ID: <20250814153307.1553061-16-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF00017097:EE_|DM4PR12MB5820:EE_ X-MS-Office365-Filtering-Correlation-Id: 0a9c6325-3791-4215-d5cc-08dddb484bb8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Dn3vY73JARjYDEzuJ3z93MctRJg3ikfy7I8wI+I22KL7Ji3rOHlpzbimW2X4?= =?us-ascii?Q?39nikR0jaBiJFUf4Rt7sRb0+DqujXnbiry/0pElvkxozX+CWhlOr7TkxlSNZ?= =?us-ascii?Q?4Y++zwkuhD5YMUbAJjWvGgLoAt8fZF2184HHErNdF9koBW8VuRguOkC3O+hz?= =?us-ascii?Q?rEnQWuW3vjfbycX6ZMRr4jYvdngku4rZR8e5MBMcIO5krMC393qvTzzpiwBX?= =?us-ascii?Q?SSXwUcJzBfCs6Jck2eU5+7gNyL7UDs8iB4mzCVzk2N47Fg5VU6JU6mlRHMs+?= =?us-ascii?Q?YRXGt7ntESGrj+tdLrCnG9hIEVFDMiQ2lSdEHK/WwGz7X0xOatRJyeCc6Sak?= =?us-ascii?Q?5XomUWRKprZL6yxryWuoVM89+O2h63jH0mET6X0vEJc9h0xwBS/ktY7IdlgJ?= =?us-ascii?Q?QeGPp7mOlrDmig2DccqoQwvRwjpulJHjHtzkwhIVvDpBfci0rI1owiMB3JRU?= =?us-ascii?Q?JF6mplAimzL9ea2LBVzDpPvxbwTpvBNmn40ya3LZCjdDscgErtsAvs1xRPnI?= =?us-ascii?Q?JHKx+X76eKFCj8yMpIrd7sWhtQA8U9a+iGEPj8hi2+3tAqRFxcwxMi3B3ZZg?= =?us-ascii?Q?RFk9WaTF5JptYK3+OIQqNiMKOT5FjfPgOnx/y7h3/mbb2781uzpm59SgeafR?= =?us-ascii?Q?PIi9jy2kPSCj8rqgT1tn2Qgnxxqf0OSS9Pj2FRvySSIv1bhfSqyzbtGwgzyJ?= =?us-ascii?Q?AjSnVkn6JhbOCUpgyGO3n6Proio9fr3OM9oPQnw90pwD3dj1Ukr3vhcg2TcT?= =?us-ascii?Q?z/3cLOTHBILSMNx9ZIQQzM9qyFgpwiK0QiF3mMh4ppMDnOz8IC+HGlmWeDgc?= =?us-ascii?Q?+K9e6R2GUdHpTY2BGWrhy1kT01EZvhaUOLO5YH1TkWWIwyfM2HgOespYN4OH?= =?us-ascii?Q?NeSrzQYByUYDF2I+mop/nbQFXMHcc2rhOVXTf6CmzKyqc2PlyJngtS+1ByZe?= =?us-ascii?Q?N5inoEo7ZCEh6UeEZjexQGUTKc7/tiMs6xjHZreGnIoBHGndJj2aEc7mJIPA?= =?us-ascii?Q?AkHtS8oKVrHtGYpzn8cLYk1IsUHnQ+re0SWUgCbyweYF5U5STa493pZ3AkRP?= =?us-ascii?Q?kCBeHJXH6dWuwyDlA/3v2lay3R6JBKWG8zlA96SP5VpzLdEnQsQZMlrtFmgG?= =?us-ascii?Q?jiAhJjvOLtw603ftkv1QgAZKYBXybZORTHk2VuudSp0u6ozXpm/L6H94m7o5?= =?us-ascii?Q?wD1EHAfLOf5giIyHjzRdlY1udGgbjoNjM1HxEdlvmjM4dAlT/ztf4e0MZiKZ?= =?us-ascii?Q?tcSg2+cHhibGRREmaPDrER0oIuEkBCr49mfmi4HWPMpmU2JxEAILeQdY4q5q?= =?us-ascii?Q?rdRoUtgNsud5+tTxCSc+PGINBwETT9T7bBTZ23K5CSz0fFa03kN5EcN3sId7?= =?us-ascii?Q?BjFcAycoJp0DScuaYXXH6CuuIN4cON/cMFFAHEr7VR3SWP1cNaAJ02SP316X?= =?us-ascii?Q?8xhRScxMs0PXlO0lxWwdP4uhhplBKvkbUb6xmBH8jYpP2cMWou+jxxciQubg?= =?us-ascii?Q?dxHoTWv01Fpm1opEsKNEFthzfqfS68Pe5wb2?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:36:11.0205 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0a9c6325-3791-4215-d5cc-08dddb484bb8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF00017097.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5820 Content-Type: text/plain; charset="utf-8" A new scalar value (PTEAScanScale) to control per task PTE A bit scanning is introduced. 0 : scanning disabled 1-10 : scanning enabled. In future PTEAScanScale could be used to control aggressiveness of scanning. CC: linux-doc@vger.kernel.org CC: Jonathan Corbet CC: linux-fsdevel@vger.kernel.org Suggested-by: David Rientjes Signed-off-by: Raghavendra K T --- Documentation/filesystems/proc.rst | 2 ++ fs/proc/task_mmu.c | 4 ++++ include/linux/mm_types.h | 3 +++ include/uapi/linux/prctl.h | 7 +++++++ kernel/fork.c | 4 ++++ kernel/sys.c | 25 +++++++++++++++++++++++++ mm/kscand.c | 5 +++++ 7 files changed, 50 insertions(+) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems= /proc.rst index 5236cb52e357..0e99d1ca229a 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -205,6 +205,7 @@ read the file /proc/PID/status:: VmLib: 1412 kB VmPTE: 20 kb VmSwap: 0 kB + PTEAScanScale: 0 HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 @@ -288,6 +289,7 @@ It's slow but very precise. VmPTE size of page table entries VmSwap amount of swap used by anonymous private data (shmem swap usage is not included) + PTEAScanScale Integer representing async PTE A bit scan agr= ression HugetlbPages size of hugetlb memory portions CoreDumping process's memory is currently being dumped (killing the process may lead to a corrupted = core) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 751479eb128f..05be24e4bc4f 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -79,6 +79,10 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) " kB\nVmPTE:\t", mm_pgtables_bytes(mm) >> 10, 8); SEQ_PUT_DEC(" kB\nVmSwap:\t", swap); seq_puts(m, " kB\n"); +#ifdef CONFIG_KSCAND + seq_put_decimal_ull_width(m, "PTEAScanScale:\t", mm->pte_scan_scale, 8); + seq_puts(m, "\n"); +#endif hugetlb_report_usage(m, mm); } #undef SEQ_PUT_DEC diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e3d8f11a5a04..798e6053eebe 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1151,6 +1151,9 @@ struct mm_struct { #ifdef CONFIG_KSCAND /* Tracks promotion node. XXX: use nodemask */ int target_node; + + /* Integer representing PTE A bit scan aggression (0-10) */ + unsigned int pte_scan_scale; #endif /* * An operation with batched TLB flushing is going on. Anything diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 43dec6eed559..6b5877865e08 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -371,4 +371,11 @@ struct prctl_mm_map { # define PR_FUTEX_HASH_GET_SLOTS 2 # define PR_FUTEX_HASH_GET_IMMUTABLE 3 =20 +/* Set/get PTE A bit scan scale */ +#define PR_SET_PTE_A_SCAN_SCALE 79 +#define PR_GET_PTE_A_SCAN_SCALE 80 +# define PR_PTE_A_SCAN_SCALE_MIN 0 +# define PR_PTE_A_SCAN_SCALE_MAX 10 +# define PR_PTE_A_SCAN_SCALE_DEFAULT 8 + #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/fork.c b/kernel/fork.c index a13043de91b0..bb780215024c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -106,6 +106,7 @@ #include #include #include +#include =20 #include #include @@ -1050,6 +1051,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, futex_mm_init(mm); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !defined(CONFIG_SPLIT_PMD_PTLO= CKS) mm->pmd_huge_pte =3D NULL; +#endif +#ifdef CONFIG_KSCAND + mm->pte_scan_scale =3D PR_PTE_A_SCAN_SCALE_DEFAULT; #endif mm_init_uprobes_state(mm); hugetlb_count_init(mm); diff --git a/kernel/sys.c b/kernel/sys.c index adc0de0aa364..f6c893b22bc6 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2147,6 +2147,19 @@ static int prctl_set_auxv(struct mm_struct *mm, unsi= gned long addr, =20 return 0; } +#ifdef CONFIG_KSCAND +static int prctl_pte_scan_scale_write(unsigned int scale) +{ + scale =3D clamp(scale, PR_PTE_A_SCAN_SCALE_MIN, PR_PTE_A_SCAN_SCALE_MAX); + current->mm->pte_scan_scale =3D scale; + return 0; +} + +static unsigned int prctl_pte_scan_scale_read(void) +{ + return current->mm->pte_scan_scale; +} +#endif =20 static int prctl_set_mm(int opt, unsigned long addr, unsigned long arg4, unsigned long arg5) @@ -2824,6 +2837,18 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, a= rg2, unsigned long, arg3, case PR_FUTEX_HASH: error =3D futex_hash_prctl(arg2, arg3, arg4); break; +#ifdef CONFIG_KSCAND + case PR_SET_PTE_A_SCAN_SCALE: + if (arg3 || arg4 || arg5) + return -EINVAL; + error =3D prctl_pte_scan_scale_write((unsigned int) arg2); + break; + case PR_GET_PTE_A_SCAN_SCALE: + if (arg2 || arg3 || arg4 || arg5) + return -EINVAL; + error =3D prctl_pte_scan_scale_read(); + break; +#endif default: trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5); error =3D -EINVAL; diff --git a/mm/kscand.c b/mm/kscand.c index 273306f47553..8aef6021c6ba 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -1306,6 +1306,11 @@ static unsigned long kscand_scan_mm_slot(void) goto outerloop; } =20 + if (!mm->pte_scan_scale) { + next_mm =3D true; + goto outerloop; + } + mm_target_node =3D READ_ONCE(mm->target_node); if (mm_target_node !=3D mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node); --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2069.outbound.protection.outlook.com [40.107.244.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 403802F60C7 for ; Thu, 14 Aug 2025 15:36:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.69 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185786; cv=fail; b=Tqk3whpyRLsX08XHQ3sBTaV2ix3PUjpy2kRG/QQHAYEJQuGhOF2gJTBFIKE+cTQrn6PBJIYn2xaJusSIqr4lMx+brVXiIuj4S2AbBr+j5ZHFFI+yce4d8RSvBJAEEfgdOVfvu++wvu0T5isYt7JwHNVnBABAywJj7NRj5hBEcUk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185786; c=relaxed/simple; bh=y4V15B3iYTlTUAGf1eUhOzHHTNW5w9o/RpgT+d04N5k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Dq2DPmVlFFG8HzjiqtI9Mk8UvfkhHh1hPRY/BHZHbgjptWxlAnBjeFgFjkI2RqSZOCalImzhI7ceg/C9WA88JE0y5LfbaX+gwmLAJc+qkuKNSmulxiXlxbhFqMVFm8/o/Jdt1oOZcMHe1VAbuhjJu7m8feuKoZV497l2kkfmjqU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=dpO89jon; arc=fail smtp.client-ip=40.107.244.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="dpO89jon" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lCx7MiG7z1QWmZG4n4gjHbwqrYpoIo9FAFpnv8cSsbiGR2wOr9uZs6T14eS/inD0phMQ1t5nkk8KCR2fpcgp2gS7u/WTH5fXNfM3JkMupbWn+cG7vj+ujudTMGy5KQjMnE166IiNkfT1/+OfyCWjJoQHHVd1jbU68pYxFKYV/PTDWZw2R3pGcf8RCk1FlylSrNMBwq5za4+OFdb+dbwuKMiALUQ0vfsRGddXL57sQ4dlFfMjzGnxeOQ+ZDocdupjRXt34l+Lr0P4dsXJrCcqm6DVfCRug4dqFkGv9xlBl3Hz0H4QEdqiuDKyz8qVLu11BZn4rgm28kHPrVXbcPyrQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JidjaM9jyZU+0KQmXw5u/FNar0F7TNpXCByTADsZ7fE=; b=wIa7iuZxBQUscRC4RBBclmhdJYVI0vP0XrOTIkqtIG/cTWCPqUibqCHaKoYyIoAl5zbDEllnsMVFtyf5KIO7A+eHwvaGYiXrcApULT4rsj8mMCMrUfUDJTcp/mkwl7n0Tquh016b8HGK1BlAFw1MMWsnAZ2ssjKAIDYvDjZOOonzyKlFuyS2/mG/NWjuva5/aHRXHhSsCW1Wkq8OL4J1UBjarWnB7RSVOLKontPZLUGRtBEar2dQIT4y1Ra8k0vZQv32f28dfYzxEAprBEG3y3hqa/WOHHzTKE8C+2i479Xc/wRohP5ovFotbU/Jcz52o4TKAZ2tMGmX8UiFb+Soiw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JidjaM9jyZU+0KQmXw5u/FNar0F7TNpXCByTADsZ7fE=; b=dpO89joneZc8e/tzjujX3rgPjYN7c7YCa2od75DjpAq7rLe+GE6RWch/tVZ2/1kMd6MC+4Z210j2tSk/iITlEaiwxSlLQgWJJacmRu3YAN9Z1GsDesVC3TgJMLk4d0Ixjw2oXzYSBK0mCL/7oZpuQ9ixJKtnXGfUzvwrmjeeyug= Received: from DS7PR07CA0005.namprd07.prod.outlook.com (2603:10b6:5:3af::15) by CH3PR12MB7740.namprd12.prod.outlook.com (2603:10b6:610:145::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.16; Thu, 14 Aug 2025 15:36:22 +0000 Received: from DS1PEPF0001709C.namprd05.prod.outlook.com (2603:10b6:5:3af:cafe::3c) by DS7PR07CA0005.outlook.office365.com (2603:10b6:5:3af::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.18 via Frontend Transport; Thu, 14 Aug 2025 15:36:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS1PEPF0001709C.mail.protection.outlook.com (10.167.18.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:36:22 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:36:10 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 16/17] prctl: Fine tune scan_period with prctl scale param Date: Thu, 14 Aug 2025 15:33:06 +0000 Message-ID: <20250814153307.1553061-17-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF0001709C:EE_|CH3PR12MB7740:EE_ X-MS-Office365-Filtering-Correlation-Id: f725ea02-d89b-47c0-1b76-08dddb485277 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|7416014|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?0V3LN+pINwsONuEG0fMkpEEYC8MCG+x9nbBKeqAHeSn98sEbY1bgxrrMJFty?= =?us-ascii?Q?WT+iILxlniKvWhP5lyrs076kuHCODtD0RsFBX4wpohJMbROmowxBMZ/QF9f5?= =?us-ascii?Q?VxArbrzoJVzgmEcz8GhhGsFvef/LZGkDLpWWqINzJFj94pjc984oas862U6w?= =?us-ascii?Q?2FrzK1k3ELjeLzYV4x4DFg9tjuqEc6KSIwJSAmddao/0tRm2WfC8dwq12CtP?= =?us-ascii?Q?19x84zK9UqitqBK7ooJ2E7FKSOJQGSOEECcB/aBQSMivDN4Mltdm0Wwn3oPq?= =?us-ascii?Q?qq7SagcZ8XTEMwN9K4jRHcdy/XqTHgkoF5wL/m3x2c06tPlNiSiFfojhG8CR?= =?us-ascii?Q?D71ufEaE30O3O1RheW7PUVssNicXh9UXrtV1yBL1yo5dLVQkF+Fx/5XqMOvy?= =?us-ascii?Q?GF1f/kXi50ynXEpXru7fuHIK6cHEXle9ulmQRlmbqtoGoWuds0bn6ydzpzcX?= =?us-ascii?Q?zpNUfTRCEVrbSoLacdoVORANGzdyO8YwWtMgPagMo+60cN312LQAz0XRVcmY?= =?us-ascii?Q?DTXU1IxzzRZjTofuTgSnr0gdtdsQYZy/hWk3yKy1xyMZQn5fS1amUnxFKYct?= =?us-ascii?Q?GmFBQlPzOccnAF+lpTPuEi6KZvf5ZB3B1VRHCYKQK+KQDVskTMnYypJfYXEx?= =?us-ascii?Q?YgDsxfKtgJcFK5DMybtZhGthk5sOzhLGf4dwI+lk2sBCRpD321VTnuek88o8?= =?us-ascii?Q?0TyZgkguCKknWuz+nuIpyMj03Hsh7yqE5gvYpjInqKH5x0IJS/FvTcCgqq5t?= =?us-ascii?Q?wd+azj8pfg0j0HqNS4DqPVh67gvO9ogWBJtESC+EGCfQ9qq/lM8kNQokHtRJ?= =?us-ascii?Q?zZW4+uTSu/7gzYzDOVuIdEDfWq5VKi/Zu6Ma2K68IB8rKUzxJZ/Uf9RrPThS?= =?us-ascii?Q?r3Ey6ee6r67k1MOe+drm3YjE9XG+db7rxMnr9mKZy3aDarktWQhujlMFkx0n?= =?us-ascii?Q?hOxsJrCMvXaLQQnz2vCH7F0JWeJib3/BtVBSRBrcwW13wXQ70tLSSpgV1IzK?= =?us-ascii?Q?5iHhiQLSn/HbzStlyznthx108pF76KtUNgVBDYHxx/bkIlk1nMn0b9r5tP1w?= =?us-ascii?Q?HTAxTIgnIT56Y1xX02le4vAkdSdEvF9rCYl5DCaBhfcgH7hmAC94QEHoHu/J?= =?us-ascii?Q?W90LrcnVW2+XPsZaMZuHVN5HOcJd7ftQcgg4VGvi54m0i1mANVpTcTnXuyyI?= =?us-ascii?Q?DwlnpLsIpKiBej9VcNUpn7qpnM2+J05hL+csVeGpYUUuUnsug1FliVoqYI7j?= =?us-ascii?Q?8Y+FR6xpIIpZBCGxdQzMvVXXM48zk62Gt+R9sX4Ee1DvnP4a8KceoDUur6cb?= =?us-ascii?Q?n3V10VEbIWHNLxLm5y3TJEu4xta7Olz0IgYDwTUKF0h8mus14G23KEHLgNjz?= =?us-ascii?Q?so3dJ3u1mWtNL4ZsyFs7iCIgPmuYunK6ragYRfxocqMDJVBdFHFtjJLAFljX?= =?us-ascii?Q?i9jg9aqfNUlVbMLuKSlf+gesPNjSExfqfK/GrPtDRha969jaqY+VucN+t3of?= =?us-ascii?Q?MnBYh4ON1Nw8GDe4yk0516Xpve0nM+yLC9P5?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(7416014)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:36:22.3337 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f725ea02-d89b-47c0-1b76-08dddb485277 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF0001709C.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7740 Content-Type: text/plain; charset="utf-8" Absolute value of pte_scan_scale param further tunes scan_period by 20%. Signed-off-by: Raghavendra K T --- mm/kscand.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/mm/kscand.c b/mm/kscand.c index 8aef6021c6ba..641150755517 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -21,6 +21,7 @@ #include #include #include +#include #include =20 #include @@ -1157,6 +1158,26 @@ static int kscand_mstat_scan_period(unsigned int sca= n_period, int fratio) return scan_period * (1 + fratio / 10); } =20 +/* + * Scanning aggression is further controlled by prctl. pte_scan_scale value + * further tunes the scan period by 20%. + * 0 =3D> scanning disabled. + * 1 =3D> current min max is retained. + * 2..10 =3D> scale the scan period by 20% * scale factor. + */ +static unsigned long kscand_get_scaled_scan_period(unsigned int scan_perio= d, + unsigned int scale) +{ + int delta =3D 0; + + if (scale) { + delta =3D scan_period * (scale - 1); + delta /=3D (PR_PTE_A_SCAN_SCALE_MAX - 1); + } + + return scan_period + delta; +} + /* * This is the normal change percentage when old and new delta remain same. * i.e., either both positive or both zero. @@ -1201,7 +1222,7 @@ static int kscand_mstat_scan_period(unsigned int scan= _period, int fratio) * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, - unsigned long total, int target_node) + unsigned long total, int target_node, unsigned int scale) { int fratio; unsigned int scan_period; @@ -1243,6 +1264,7 @@ static inline void kscand_update_mmslot_info(struct k= scand_mm_slot *mm_slot, } =20 scan_period =3D clamp(scan_period, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PE= RIOD_MAX); + scan_period =3D kscand_get_scaled_scan_period(scan_period, scale); fratio =3D kmigrated_get_mstat_fratio((&mm_slot->slot)->mm); scan_period =3D kscand_mstat_scan_period(scan_period, fratio); scan_size =3D clamp(scan_size, KSCAND_SCAN_SIZE_MIN, KSCAND_SCAN_SIZE_MAX= ); @@ -1384,7 +1406,8 @@ static unsigned long kscand_scan_mm_slot(void) =20 if (update_mmslot_info) { mm_slot->address =3D address; - kscand_update_mmslot_info(mm_slot, total, target_node); + kscand_update_mmslot_info(mm_slot, total, + target_node, mm->pte_scan_scale); } =20 trace_kmem_scan_mm_end(mm, address, total, mm_slot_scan_period, --=20 2.34.1 From nobody Sat Oct 4 15:57:17 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2063.outbound.protection.outlook.com [40.107.236.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BEFD22253FB for ; Thu, 14 Aug 2025 15:36:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185800; cv=fail; b=Lpm1WEGRojKtpVfgBG7uAvIvJJqkmkn2B9qapO6tRsaDW+jkQOFzVAeuvknF/N5g0h0G6ByeQDwCrU9ReoXB4iiyRC4Bg2onyXXcrZZuIHtVQbNEjPOPb3mypRi+GHgesRL/yHsqy+m9AKStvLh4O0L4IXffsl15nub7cmRPvV4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185800; c=relaxed/simple; bh=Iz4FGAEph09umivSHnfvO5ahBpuinVOsK4petaymIwM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qDRU+jpltFIUOTjbY97bHcVJjqjULBCUTfstRk5qrk14pWzdYjWKDSYGbdrI5SkraV5u7AeOMHUuRpskifJ5YXHsjlCrDeSAUkiQiUovG675QM3yeS+FeTN9vQ70iPZjimqYnO7G5ubaTdt8ujr+akZ7A0ro609h52n/BexUsvI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=BePjrkNr; arc=fail smtp.client-ip=40.107.236.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="BePjrkNr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KW9cJXZxGlDaWDwnRsfKH9YURp21qxiqxC7qBoDNBK5wei6eT5CuMyaS9ndfeYNa1Al5zHGqx7I3ZbAwT92Ehi2EbO42+N7XvTSViJzgb/hm1F7IOblJRPgejIsXXy9AFkKnUZf84MKDlXiW6YXmkGEWp6rxQbgIdwlVAIuRYlNCn3mJsVgLLfqWD7zo/kBh68RdeBBtW5LLTNoubwUayGpWfoa8vNcBqzoThlH6kAZ4dEvzfS7UKOF0FBkECq3tjtBgun2jjj0xN7cKlF+DvCMU7Zu+8sXI5g4krhHfGP7YNU5Plf7vSLc+fy+ZpntYvjBSrcxqBU0iDpxiFZWmBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4LpyqC5iH8cjBqeEES0QKxJ0Bd/2XNjJ6HVXe4kFexg=; b=X8CPB+JV8AE1QeJkujYlAsXaQBZe/kMn/uqdkHt9HT7hXFzELBF7WbbZg1FI6Aw/RwM2RY0jTnnq9mFcJ/QhvBUWzjMksa3V3Wikf6QRZU6vjIN+WYzPf1U+939SpSELooPCKUf2Z0bGMHyQcjN9sNZ5/RUKwFwLXkxfgOYBQDgtZfFbVvYbhnK1jSZAMWV27pAcul3rdzlUee98aVGidNM5BzMA2p8E7GHfsq3CY2aN/GXEQdMriLOHd2h1zUCMT7TaFQzQe9ltquytTkHO/HyDsLcZPdR1L916Y8EdAcD2WXbmSagWMEcJ7pGIIpbJJKLEb9aib5XPSRTH3p2v4A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4LpyqC5iH8cjBqeEES0QKxJ0Bd/2XNjJ6HVXe4kFexg=; b=BePjrkNrymGMIzHMP5zuaP4sN83kyt9VCTHzuMYCPH+GbnNnCM59hEb5xt66TuLk71TOHInmYJtEHFsT/DQxFtNENucxATOgOOhuSjTdk9d9Vgy3NsFbYCbrRXs1zZElNb7HCmHd9FB2dD6LhAHQfqyLJma3bMNdcC/SmOCb/oU= Received: from DM6PR13CA0035.namprd13.prod.outlook.com (2603:10b6:5:bc::48) by LV8PR12MB9336.namprd12.prod.outlook.com (2603:10b6:408:208::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.16; Thu, 14 Aug 2025 15:36:33 +0000 Received: from DS1PEPF00017099.namprd05.prod.outlook.com (2603:10b6:5:bc:cafe::67) by DM6PR13CA0035.outlook.office365.com (2603:10b6:5:bc::48) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.15 via Frontend Transport; Thu, 14 Aug 2025 15:36:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS1PEPF00017099.mail.protection.outlook.com (10.167.18.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:36:33 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:36:21 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 17/17] mm: Create a list of fallback target nodes Date: Thu, 14 Aug 2025 15:33:07 +0000 Message-ID: <20250814153307.1553061-18-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF00017099:EE_|LV8PR12MB9336:EE_ X-MS-Office365-Filtering-Correlation-Id: 9dfae0b2-3562-4f71-f1a7-08dddb485905 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?6XAnzrlC+zYps7hyt54mj3KEkuiHz32UlByyDXWGDDNcfREmiVaAGyD7vqY4?= =?us-ascii?Q?J4oX0boUGlRvyIUOeebWNh7VFPkL+lWmbnY61ZzPbKbsgO401zXr658RLwQ7?= =?us-ascii?Q?VxRNluu7Q/c2WgBO3OsHEcPfLAWe7PgU877Y57ktx+Ddw8Elq/Y/vFeTiuff?= =?us-ascii?Q?67CjpnEgOnQcEAlLnNZ2ZsaqkHoNxt/IirdsKnb4Ic7QMLbKrIOXi+jUYtKi?= =?us-ascii?Q?mfqqpXXvCvSzeMSc4BV8KOP7/OdWDpDa+speV4ycOfCTdGyGBO3oHONgY4GI?= =?us-ascii?Q?RJCyVv5kpC1QXlgseOUIEOoNkvTaUjhttzzl1v6nsecf4tTY8A46p6wzb7nb?= =?us-ascii?Q?xedFQVfqXyaHlYdvs5KM83iv3b1mSoDlzJDnkqwOkRMDf1fZKm5pFurq439t?= =?us-ascii?Q?ZfsvwD+vWLXbqBz/VmF6RXLJrLRcZMpufcP+lKIUrHDMAPtK4LVbsog0/5LR?= =?us-ascii?Q?mZvYJUPOakUU5dgoWdbZWnAL8YgWtajejlVwOb4W3qZy6uRjHKyQ1xAl9OYI?= =?us-ascii?Q?9EK1Xd5ZWBQ293qPKaQ4P3nAOuDMjsyzbmHhLTGI7uv9ieFHLVzWRc1pc1Ho?= =?us-ascii?Q?1jPTu8y86hvT7TNpLEMgR56+1UuBEdIGeWMMCODWidOIkplLNsHgYhSi2xlH?= =?us-ascii?Q?5TTBOzUCPCdgMlDrkFV5RpcM9IoFLNQ57Lxa0idHJZ+hFZz7OeRjnsJdJWjm?= =?us-ascii?Q?xNSmU+BdfbmUYIvGjj+weeIq7huGuumKgWuopS4P6Bfwh6sgMqjw+AIc/z0M?= =?us-ascii?Q?eaJic+j9lW91OgzwrsxHBt9uQWbDuaViMAQ1mOW8UqTZaMVDCZv+WjZEs9/p?= =?us-ascii?Q?hC4wJxh2gktKrUPr1qfnKbDrlYOE7OqdP9YihoxWfxq0jQEH37uW5s+uY9+e?= =?us-ascii?Q?ke8veS87AsgoCDVS+53JofoQ8RsflC/0vjJxwKqxeO11GNgloDVY56vcsdZr?= =?us-ascii?Q?lqGTAY1YOWlFkyB7MUUANuSGnRsgDDU5bQ/2ty1gPFaUFBeiWtcqXr73C/r5?= =?us-ascii?Q?MCkP1BfcbDXdIpsBBJlLOpop5r0/oqTiuMHgAUk9Oa2A7yfwlN9tnNLh3QaT?= =?us-ascii?Q?HLcvyKUZKSjYZkl5CBQBfq5qWBXole0yw6JKKtmzEKGYfqlaG6XYihNJvrf5?= =?us-ascii?Q?Np53gTdJ2gIURdGOSEVHjVSPHYtMGy7ILZT9d+4TL+nRxJyopter2SAyk3AP?= =?us-ascii?Q?RuKQlRi7Eljzc5JYNGjp54P4fQbLVRgG6pN7mAQOtzFzvF4JXWZ+vwNEK+lY?= =?us-ascii?Q?TlIInYgMAsuKa3oC7ViLW0TD4WhiqN1v5HL1PyGUI3uXWwwuuhgrTtFy32DE?= =?us-ascii?Q?MxS5Zfo35Cyk/xKcU49QlNDsC09K3jPrLNG+7Q11S9wtmd/bcogSJiuhmHr8?= =?us-ascii?Q?2E4NuI+TRfJt8P6GHGrSVJifaNR8LSwvEf5OG8zhtLIIFuiVqwNVoMYvNi9H?= =?us-ascii?Q?qGFyqh+c1JJqgJWbxVF75hk2T5TGc4Bi4G8nytRxAlu754fr9jghVclw/ZT/?= =?us-ascii?Q?cUF3h3eJjK9PohMa/VGCwcbsOIwC3LPXyCLb?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:36:33.3359 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9dfae0b2-3562-4f71-f1a7-08dddb485905 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF00017099.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9336 Content-Type: text/plain; charset="utf-8" These fallback target nodes are used as hints for migration when current target node is near full. TBD: Implementing migration to fallback nodes Signed-off-by: Raghavendra K T --- mm/kscand.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index 641150755517..a88df9ac2eaa 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -136,6 +136,7 @@ struct kscand_scanctrl { struct kscand_nodeinfo *nodeinfo[MAX_NUMNODES]; unsigned long address; unsigned long nr_to_scan; + nodemask_t nmask; }; =20 struct kscand_scanctrl kscand_scanctrl; @@ -148,6 +149,8 @@ struct kmigrated_mm_slot { spinlock_t migrate_lock; /* Head of per mm migration list */ struct list_head migrate_head; + /* Indicates set of fallback nodes to migrate. */ + nodemask_t migration_nmask; /* Indicates weighted success, failure */ int msuccess, mfailed, fratio; }; @@ -522,6 +525,7 @@ static void reset_scanctrl(struct kscand_scanctrl *scan= ctrl) { int node; =20 + nodes_clear(scanctrl->nmask); for_each_node_state(node, N_MEMORY) reset_nodeinfo(scanctrl->nodeinfo[node]); =20 @@ -547,9 +551,11 @@ static int get_target_node(struct kscand_scanctrl *sca= nctrl) int node, target_node =3D NUMA_NO_NODE; unsigned long prev =3D 0; =20 + nodes_clear(scanctrl->nmask); for_each_node(node) { if (node_is_toptier(node) && scanctrl->nodeinfo[node]) { /* This creates a fallback migration node list */ + node_set(node, scanctrl->nmask); if (get_nodeinfo_nr_accessed(scanctrl->nodeinfo[node]) > prev) { prev =3D get_nodeinfo_nr_accessed(scanctrl->nodeinfo[node]); target_node =3D node; @@ -1396,6 +1402,9 @@ static unsigned long kscand_scan_mm_slot(void) =20 total =3D get_slowtier_accesed(&kscand_scanctrl); target_node =3D get_target_node(&kscand_scanctrl); + if (kmigrated_mm_slot) + nodes_copy(kmigrated_mm_slot->migration_nmask, + kscand_scanctrl.nmask); =20 mm_target_node =3D READ_ONCE(mm->target_node); =20 --=20 2.34.1