From nobody Wed Oct 8 22:32:51 2025 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2041.outbound.protection.outlook.com [40.107.95.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3730622CBEC for ; Tue, 24 Jun 2025 05:57:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744626; cv=fail; b=OoLnqnwXASlEpcavnLTA5XlFWxD+rEU/tnCCEvqYkhYSD89mLrPmQyTnLITPtEpjUYGNRztuylMUumaIS1ItWec/ctC4ctcc7R26GnTZwRqit+ab5VUbPNQG2qkNqKM+UkdMU/tAXnyf1PuDP67Gny8zbC6Y4/alIcaCrvkMnvg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744626; c=relaxed/simple; bh=Y9BBbN0giuULO+dtcOaFqnffqgXakuiQQ8se8ggY9+Y=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=C5PUl4KlrrWLKOyV+g+qavKl4GjUJUTCpq3SXctneDChyJDb+SgWwqcPKBdh0NqTtAo90vSJCB5ngXtXdsriz4bhCT2muUkWOF9NT80xtY+rWxLreI7Sn/I/JO6I5OQdqw8wdlCNaA0q57ThNaouCSd5rQchOngMc7p9j3Q+rzc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=3j9zdA5e; arc=fail smtp.client-ip=40.107.95.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="3j9zdA5e" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PtIRBR2/Gi5Y6FDtq+3XQEBbxzqOJ876Xa4VMKSN8PWzkYY8tXp+seXPABckTuekXMRPWpDSNX7XkzHx/TCN07QEwrYHjCEcqjbt7tCkHPbuoof9pa7i6CZI6Jm2IRbR1NDvlXjJeJVQuVTb5lqGCxrti+PhTpD9XQe5Z+VdK4egKVCzIrQZqJW5eSDpL4NHjSvCvz+3O5Z4rhC357whoXIloayhCv1henTM3Lh8ZBJAi4BgBrA2kI4tlHZzYNApRrf9RSA7pKQNu6s0t0BqacbsNq0FtoO0NdNGQTqjTAQTDWVHMgscvjDPg61v/cxgjrHt3D68qxJ6Q1tbqaR1Ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hglcXND/1hQIbXGjSGlRxrAVs0gQHvtkwFHzUUuRDYI=; b=H2LUO9RDmIvRlQ6likFOoupMalR0dH8eXZ0RnnJZd6I6Mc/0N3W/8Mem0xB+lNcyytBUbUqvkB/1I2eI9BPbF2HGzthfLGNTJm/Ds0BHe0PsBfyRG7V5JU374zQ7+2CrbkaW0KQCdRSJBl9Czee8VVAVTcoBT8vHPuhPbh1msQbe4u0rI0GU/9JLdFzLcEZu+O+sebEVRatVhU7WWjasgb3kUNU9MgTUoh7nOkWwtgmowBb/CaWGLWqsO5oc35JRtYacRQtkk8hbr0+tOmqSlul2tt92EezLS1KThWLJ5LGjDUgUTLKGjETgDXGW/Nx2FZBv4jEKeKD7yTjChDU+Uw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hglcXND/1hQIbXGjSGlRxrAVs0gQHvtkwFHzUUuRDYI=; b=3j9zdA5e4CkPTCimLti1pqQhJc2Gl/Ae1g4p41aXZtqQLTtY3eANj5+7DicnESsC3ohsJjLEYFK4n+xoMDLJiaaKdWMw70GorsJfFMR1yulIocwuRTC4V4n38+VT9RNfsZJSnSI0o+CFbsqoMI+ScpQBzo4MPLdPmkjGZfGCA7M= Received: from BN9PR03CA0169.namprd03.prod.outlook.com (2603:10b6:408:f4::24) by CY1PR12MB9604.namprd12.prod.outlook.com (2603:10b6:930:108::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.28; Tue, 24 Jun 2025 05:57:00 +0000 Received: from BN1PEPF00006002.namprd05.prod.outlook.com (2603:10b6:408:f4:cafe::fe) by BN9PR03CA0169.outlook.office365.com (2603:10b6:408:f4::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8857.30 via Frontend Transport; Tue, 24 Jun 2025 05:57:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006002.mail.protection.outlook.com (10.167.243.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:57:00 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:56:49 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 01/13] mm: Add kscand kthread for PTE A bit scan Date: Tue, 24 Jun 2025 05:56:05 +0000 Message-ID: <20250624055617.1291159-2-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006002:EE_|CY1PR12MB9604:EE_ X-MS-Office365-Filtering-Correlation-Id: d6ad5650-416b-4b1d-99ef-08ddb2e3ef9b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Byw9aLpx2wPmNIAXmc6rLjZLr1GTfNzk/9vMW9JpL60l82Us8AjjONfOotR+?= =?us-ascii?Q?oVhZ5i0arDyMhcwspKRGeWi0uo3Jha3kuwbjrt6bQNMEcBr0C0pYw+yuKyrk?= =?us-ascii?Q?FFyVyBKQyiwlB4GlvtS7gu6Cj1238vya2dncqL7ngpKfELP7B13g6wbv/ze7?= =?us-ascii?Q?yhsIlagU0jA+aPCToK7qpZwSwsswtvnTqpg8p6j8OsSwnPlM1YAuptRUFXKV?= =?us-ascii?Q?5pLpiDAtJWkGUFyerZBqJi3NjY2z5jPZacZwNHvaDfMiU4rYKAHiNoFgVuc3?= =?us-ascii?Q?yS5p9h9asvxILIlGcfp4Z0RO2ziFuou5sxvIUWsk5UXSLuqtCd/htvoRpiha?= =?us-ascii?Q?PJoTgP/owZLXANXMmJKRv3TO7qYvQzhWKa8NqevYApyT23QWwZxyvIRXs/GS?= =?us-ascii?Q?t7W2dtp9nyf16ZERWLliUR3DtNNt414rwUUiZDuvNkhxcRSB7zYqHVm7+gd3?= =?us-ascii?Q?/Ze2aqQZYynWINiiED+o+bP/L5fcsF5WPn/e1zAptqTzBJquUDEw+hJhYQEe?= =?us-ascii?Q?th4++IXvhSvLnJuNUFjvFgP7vxAeXwVZ0eOaxA3Ac9GW6iJuUlcV3hJ0MizP?= =?us-ascii?Q?TBWZRB0jIV3hCp39R2A5jqyBeXROeBfEFtLPrHgOiQSyZmZ4c2CLCic1HBq+?= =?us-ascii?Q?yNqEGDzCt7KP3ZVzwbiVKLl0O7kDkD9jU+ytlaYEFtKoWEIjw/xEHV3ocCzo?= =?us-ascii?Q?taKaBfVXS5AnSVTJlb9kcHxg1AUWAY3P8zRQ9trxDjZWdrnDS4Kvz6gpe7Wl?= =?us-ascii?Q?E0X8DYqwBAPTHEUbbms3td4SL+sHPxAJDdQ9KsL5yQHjyWiW/vo4TN29R0KX?= =?us-ascii?Q?cISUORp+smjjFkyc+lubGipRkbsG8eDyvDO9wbd/Acudw4QdVMa0lYV+NBGI?= =?us-ascii?Q?qpOOLa/Rp3r0wMFVCJLebIG4e3XdxWmQd/AYBkmsfH0ktUDKeua2inXkMg9j?= =?us-ascii?Q?geCnX45Ha/U8B51l4D/bDUHTEU4Sz3/qNS1h3JlRtVFQ6VXdfG9oDNf8iAA8?= =?us-ascii?Q?ymJdp0tNNsbm/pH/5M072EgNE5JILO8VWhiUj1+WVcc21w650S1AXji7WWh0?= =?us-ascii?Q?+gQCCiGS1/Unmf/8tzGjMke8+sSq5povuxuhzPr8xel2X1PtP6ZpXBaaiq6Q?= =?us-ascii?Q?zyaQOJMOf0SSlDtVMIw9LcL9kZyRH2e9vog7RnfqsCiKTuPm5u7YmczxUCgI?= =?us-ascii?Q?IyhGstlEp0Ziufx9AK7i1h2ZLYtB+JWvLX4a7oaOK1hh+ZHyH4vPqLJyC85+?= =?us-ascii?Q?IyW4PpfirrYyq54tZdg4G1s22QGwO1hA91WOUMyD52EhKLQL1LiiJ8ZqJhLW?= =?us-ascii?Q?AlZuxPSdSiPlt4zpQ039nNJsyxdMxzmLH121YSRAgb3TRscuT7KY6H58uA3p?= =?us-ascii?Q?jJhocTwuGz5YwNvxuJcUbG4PlgG/2xI54p5u3hWLbesT0ebYVT/EYohWdDKv?= =?us-ascii?Q?TU4hWQ8XXhMgrZaR3eq6BXbpeD2qCu/Kuj4ok7d1AnBqmWu72DqfIQDDRu4p?= =?us-ascii?Q?AUv87WhyBq9sjhoHoHBhaPfJiR8l3HPZd/Uw?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:57:00.3331 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d6ad5650-416b-4b1d-99ef-08ddb2e3ef9b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006002.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR12MB9604 Content-Type: text/plain; charset="utf-8" Also add a config option for the same. High level design: While (1): Scan the slowtier pages belonging to VMAs of a task. Add to migation list. A separate thread: Migrate scanned pages to a toptier node based on heuristics. The overall code is influenced by khugepaged design. Signed-off-by: Raghavendra K T --- mm/Kconfig | 8 +++ mm/Makefile | 1 + mm/kscand.c | 163 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 172 insertions(+) create mode 100644 mm/kscand.c diff --git a/mm/Kconfig b/mm/Kconfig index e113f713b493..062fa6bdf3fb 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -746,6 +746,14 @@ config KSM until a program has madvised that an area is MADV_MERGEABLE, and root has set /sys/kernel/mm/ksm/run to 1 (if CONFIG_SYSFS is set). =20 +config KSCAND + bool "Enable PTE A bit scanning and Migration" + depends on NUMA_BALANCING + help + Enable PTE A bit scanning of page. The option creates a separate + kthread for scanning and migration. Accessed slow-tier pages are + migrated to a regular NUMA node to reduce hot page access latency. + config DEFAULT_MMAP_MIN_ADDR int "Low address space to protect from user allocation" depends on MMU diff --git a/mm/Makefile b/mm/Makefile index e7f6bbf8ae5f..f5e02d6fb1bc 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -97,6 +97,7 @@ obj-$(CONFIG_FAIL_PAGE_ALLOC) +=3D fail_page_alloc.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o obj-$(CONFIG_NUMA) +=3D memory-tiers.o +obj-$(CONFIG_KSCAND) +=3D kscand.o obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) +=3D page_counter.o diff --git a/mm/kscand.c b/mm/kscand.c new file mode 100644 index 000000000000..f7bbbc70c86a --- /dev/null +++ b/mm/kscand.c @@ -0,0 +1,163 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include "internal.h" + +static struct task_struct *kscand_thread __read_mostly; +static DEFINE_MUTEX(kscand_mutex); + +/* How long to pause between two scan cycles */ +static unsigned int kscand_scan_sleep_ms __read_mostly =3D 20; + +/* Max number of mms to scan in one scan cycle */ +#define KSCAND_MMS_TO_SCAN (4 * 1024UL) +static unsigned long kscand_mms_to_scan __read_mostly =3D KSCAND_MMS_TO_SC= AN; + +bool kscand_scan_enabled =3D true; +static bool need_wakeup; + +static unsigned long kscand_sleep_expire; + +static DECLARE_WAIT_QUEUE_HEAD(kscand_wait); + +/* Data structure to keep track of current mm under scan */ +struct kscand_scan { + struct list_head mm_head; +}; + +struct kscand_scan kscand_scan =3D { + .mm_head =3D LIST_HEAD_INIT(kscand_scan.mm_head), +}; + +static inline int kscand_has_work(void) +{ + return !list_empty(&kscand_scan.mm_head); +} + +static inline bool kscand_should_wakeup(void) +{ + bool wakeup =3D kthread_should_stop() || need_wakeup || + time_after_eq(jiffies, kscand_sleep_expire); + + need_wakeup =3D false; + + return wakeup; +} + +static void kscand_wait_work(void) +{ + const unsigned long scan_sleep_jiffies =3D + msecs_to_jiffies(kscand_scan_sleep_ms); + + if (!scan_sleep_jiffies) + return; + + kscand_sleep_expire =3D jiffies + scan_sleep_jiffies; + + /* Allows kthread to pause scanning */ + wait_event_timeout(kscand_wait, kscand_should_wakeup(), + scan_sleep_jiffies); +} +static void kscand_do_scan(void) +{ + unsigned long iter =3D 0, mms_to_scan; + + mms_to_scan =3D READ_ONCE(kscand_mms_to_scan); + + while (true) { + if (unlikely(kthread_should_stop()) || + !READ_ONCE(kscand_scan_enabled)) + break; + + if (kscand_has_work()) + msleep(100); + + iter++; + + if (iter >=3D mms_to_scan) + break; + cond_resched(); + } +} + +static int kscand(void *none) +{ + while (true) { + if (unlikely(kthread_should_stop())) + break; + + while (!READ_ONCE(kscand_scan_enabled)) { + cpu_relax(); + kscand_wait_work(); + } + + kscand_do_scan(); + + kscand_wait_work(); + } + return 0; +} + +static int start_kscand(void) +{ + struct task_struct *kthread; + + guard(mutex)(&kscand_mutex); + + if (kscand_thread) + return 0; + + kthread =3D kthread_run(kscand, NULL, "kscand"); + if (IS_ERR(kscand_thread)) { + pr_err("kscand: kthread_run(kscand) failed\n"); + return PTR_ERR(kthread); + } + + kscand_thread =3D kthread; + pr_info("kscand: Successfully started kscand"); + + if (!list_empty(&kscand_scan.mm_head)) + wake_up_interruptible(&kscand_wait); + + return 0; +} + +static int stop_kscand(void) +{ + guard(mutex)(&kscand_mutex); + + if (kscand_thread) { + kthread_stop(kscand_thread); + kscand_thread =3D NULL; + } + + return 0; +} + +static int __init kscand_init(void) +{ + int err; + + err =3D start_kscand(); + if (err) + goto err_kscand; + + return 0; + +err_kscand: + stop_kscand(); + + return err; +} +subsys_initcall(kscand_init); --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2052.outbound.protection.outlook.com [40.107.102.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34FDE231CB0 for ; Tue, 24 Jun 2025 05:57:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.52 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744639; cv=fail; b=NFubV/VCZPWZ7rJ1TZJZRuZuMHsgwtOcVq7aYAwSGnjXJ6yfMT1FnuP22euQIMCLBU/dQs2K7XqtdljDDWhj75fAvJNHUAr1ITaonLbvt6eLCruAdhpEAjMHV6y+zSxpS2c1tZ0sQqqGCi1Q7VjcUFIfUadCqGgeJtptDplLtFQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744639; c=relaxed/simple; bh=6nTuaXCMsh4Sn6rX4Z7YbTQq6GSm0IVqrPfNUkVprV4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rb5SdiV06NR+RDlunBko6dRyS8VJqaOFUA8tdQOhc5Q17UIr7Wjxl/XMCIoYJNhvXkvh8gh1xlx1PqzjzPGtpMKqVaxY85Q8R6K4hkgfAL4fHi0etzMFyYYy9nhr2aQGQOH3zQhGlTaIunUm9GEx/tF8HTtxQScjZg4Q8CGNBDA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=dovGAxSn; arc=fail smtp.client-ip=40.107.102.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="dovGAxSn" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=P3r9NK1GzbJE3YTAQkYcfkjcIMksyzcWv1jEoh6Eg6ZmuJ1tuzCW59+7pqLi8912wMCyiyTEgEhXF3EzORUEDO5W/AkyvIlOqkkeX94XdQk6tOLz2mwBv7G7wmub/BbdLMf3qbfK2OTGqcuFUARPva/7aMFUCT6le0uPr2g1hbk7xagLwrIU0KpHpYKcQ/jg7qcPvclK+r8nVZvQAcyfdEWjh3O2r35RXLzSA2hY5QqWArOL3ZY7+rC//F8HPWHmc/T3qfrkApUpSba3x1KQS00UZyPNxMPtrzVq6WBJexvpOihu4f179pDIMYogPC4bzXtiyl9hoSIdd36cLa/Psw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LbBjlJuKSkKDjaW6EjtJEY38TaofG9xVCZkD4L5OqLo=; b=E9zxhQEVRd7/OB36Z4zC4io1Zu50qxM781SCnyGMpUHIo06v4iM4CIOWAAT90RMw2oh4i4qvb111FkyQioS153nHLcJP+/SH2fRN6ForOIDkG2xDhgC/UjX00yWwKM9l94gIONImvjOj4VQDjxQqFc2O5hzvadp95NSi+fXZ9DFCGAhu/wYxG1MxLpBjnF800No1upOVeoC5LLzTrUZ3yW8Y4CBMF9mpg+UI9Nn4IxCTruKj1mCgPjCz0n3AQ0LR9kcuzRssNr/gt6Fjw7gPCg2uKfrJqTU4/ndGYXvl0WIbr0FzQp+wSQ9tj/XwZ1uN/eKVSEoU7/ypMW1oGklc1w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LbBjlJuKSkKDjaW6EjtJEY38TaofG9xVCZkD4L5OqLo=; b=dovGAxSnLx27PHEhC6CjNsUpLfiL/cQUUN0Plzjso+4gf8Dqncz97istVVcShgLykHGu6MQqbxYDgNfWi5LZdab1gTfH/fWiUnYga6KENYNK2YAMiA3IFIiX/Hj4nt+aJhM+3iiXnrimcS8dLoXqURENwSEpH30vD7AMtDLgsOc= Received: from BL1PR13CA0206.namprd13.prod.outlook.com (2603:10b6:208:2be::31) by IA0PR12MB8840.namprd12.prod.outlook.com (2603:10b6:208:490::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8835.29; Tue, 24 Jun 2025 05:57:11 +0000 Received: from BN1PEPF00006002.namprd05.prod.outlook.com (2603:10b6:208:2be:cafe::ba) by BL1PR13CA0206.outlook.office365.com (2603:10b6:208:2be::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8880.16 via Frontend Transport; Tue, 24 Jun 2025 05:57:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006002.mail.protection.outlook.com (10.167.243.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:57:11 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:57:00 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 02/13] mm: Maintain mm_struct list in the system Date: Tue, 24 Jun 2025 05:56:06 +0000 Message-ID: <20250624055617.1291159-3-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006002:EE_|IA0PR12MB8840:EE_ X-MS-Office365-Filtering-Correlation-Id: a1f167fb-d69e-44ea-a7f8-08ddb2e3f625 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?1hnd2Oq1faSHF+l57IX4+5lxZPYT3d4pYroxGz5/oHnFIG8BL3jMsBvdBtj/?= =?us-ascii?Q?IuURy9y+41C7fQWg+iAJKnfb6M6CX3uumpWIXWAZqtIVIjpmb/HNW1so4Mok?= =?us-ascii?Q?UERa5mpxXzxe+fH2lJi0/6fLw00KK8y2r4uUA9KY9ZZEaqHdpiWSq3VS2kps?= =?us-ascii?Q?j0+RNgLrx/WXSxYpAr4Mo1MRNN3vkJ73+3hZFeggSH/jDjLETA06qi4XLlc7?= =?us-ascii?Q?dl22EthVt7nWwkUhpfqSCRYEkNYs4+Gw4jKzY+kBxMECn0KPfmVOYXb6vfhe?= =?us-ascii?Q?OxVhu6pTRk3PtGut7rou+01nu/GmWVOxaIyncPcWwwg9qmU91IKpIQvg3uzv?= =?us-ascii?Q?l9aRxkUgbdrkkRLjWXglZpNYvgYbtxqYiB6/24K+iV3sQKNwpX8zmfX9XeJ5?= =?us-ascii?Q?O+sI3OAyFSIw+9OFXH3qMlxy0UR9tLX2YlFf7TvqAnab0i95+es8nFAW7+QQ?= =?us-ascii?Q?QzLHDBxweeBzrge8Jfel//w8s0WBJL4iMS4OCfMlx85YI4+1MONw9G/Yg9zi?= =?us-ascii?Q?nAHeisNJnqO1C6RhKgLn7SgdjMVDPZd0tKzf9FoY0kxLZ+0Pg1e8UX7rf1vo?= =?us-ascii?Q?yeY0ecxJ6mrtKu2tULr+gYrr5kiDKKJIVJzcF5xC5L8ZGibl/Q/9NGaLtxBh?= =?us-ascii?Q?tfsncS8iA3qNvWExACoUn2YPhGZpRFlXoC7VGLUuBuRkkCENxVgzEDCsFS99?= =?us-ascii?Q?fneIHRxo8f+QbofEGAxmVzGT9KsITVNr5o9+SnEWFP5CE7UwTsIgY0KYZsDs?= =?us-ascii?Q?VflRpOzEOQ4tVbFyhsMq2Cv24nlkmiW99F1dSfAiMjXstV+XOk5shbe4Glyb?= =?us-ascii?Q?ipb/wcCEv9RdH0CLh5P+EKu9pN6Fx21qiXCphozkV8GGtM9m0ZlS1padtNah?= =?us-ascii?Q?rd1JW1fjLWN//8o5/JmLnd0unzgr3uB815JD0pktQXYu+7++rFbFbK15F7BM?= =?us-ascii?Q?UbhFwVu2mvzk1rZ0D8NVS2PIJAQbDFLChy2tafyx4kQ75DQAEozf1I3u1nhF?= =?us-ascii?Q?dR1FPb0XOZtqephBrXov7STCck4AhBDB6WACd540jBU27J+krLRpZRhqNQWE?= =?us-ascii?Q?8npNIefpeD/6GihfHYE1eFa/ghWhhHgb9pj3UiD5J05rsECLUPv7yEk/7yxd?= =?us-ascii?Q?4oCv0NeUaWc4uB2uZu+vT6H1Dedyp5W4w/O6ygXbs9dY0f5hH+fXNfaSw0VP?= =?us-ascii?Q?j6F8Fpnf4+xJX/k4bppgdJ5wwHjCTGjmFrkgTsqsRFUlQ13b3ev1MBfCVe0p?= =?us-ascii?Q?JjK1cWKSWF67onDlff6DpVcKzqqbmQJYZvXGlIY4zp77lWqyCSU2oSXsvCLG?= =?us-ascii?Q?iN9hoiBi1k4VrgGqrGIaTB+/ETPIKllrothiwJ0i7CXALHj/rcw9iMmAs6Yk?= =?us-ascii?Q?uBPh/iRnoRZySqHuAUZwsoUWMVZEk7OzX2v1h1CQNxvmhIN7CCBi6aIE4t42?= =?us-ascii?Q?wgaLjum8wKE3SkyP0KdrLr+DOvSDHMYQ3q/fy0g7fDoqgXpQ7Z169VO4xORu?= =?us-ascii?Q?9vrXfDYfflvy+QGfAVqz0igkoGymbfHbUWrQ?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:57:11.3046 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a1f167fb-d69e-44ea-a7f8-08ddb2e3f625 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006002.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8840 Content-Type: text/plain; charset="utf-8" The list is used to iterate over all the mm and do PTE A bit scanning. mm_slot infrastructure is reused to aid insert and lookup of mm_struct. CC: linux-fsdevel@vger.kernel.org Suggested-by: Bharata B Rao Signed-off-by: Raghavendra K T --- fs/exec.c | 4 ++ include/linux/kscand.h | 30 +++++++++++++++ kernel/fork.c | 4 ++ mm/kscand.c | 86 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 124 insertions(+) create mode 100644 include/linux/kscand.h diff --git a/fs/exec.c b/fs/exec.c index 8e4ea5f1e64c..e21c590bfdfc 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -68,6 +68,7 @@ #include #include #include +#include =20 #include #include @@ -266,6 +267,8 @@ static int __bprm_mm_init(struct linux_binprm *bprm) if (err) goto err_ksm; =20 + kscand_execve(mm); + /* * Place the stack at the largest stack address the architecture * supports. Later, we'll move this to an appropriate place. We don't @@ -288,6 +291,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) return 0; err: ksm_exit(mm); + kscand_exit(mm); err_ksm: mmap_write_unlock(mm); err_free: diff --git a/include/linux/kscand.h b/include/linux/kscand.h new file mode 100644 index 000000000000..ef9947a33ee5 --- /dev/null +++ b/include/linux/kscand.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KSCAND_H_ +#define _LINUX_KSCAND_H_ + +#ifdef CONFIG_KSCAND +extern void __kscand_enter(struct mm_struct *mm); +extern void __kscand_exit(struct mm_struct *mm); + +static inline void kscand_execve(struct mm_struct *mm) +{ + __kscand_enter(mm); +} + +static inline void kscand_fork(struct mm_struct *mm, struct mm_struct *old= mm) +{ + __kscand_enter(mm); +} + +static inline void kscand_exit(struct mm_struct *mm) +{ + __kscand_exit(mm); +} +#else /* !CONFIG_KSCAND */ +static inline void __kscand_enter(struct mm_struct *mm) {} +static inline void __kscand_exit(struct mm_struct *mm) {} +static inline void kscand_execve(struct mm_struct *mm) {} +static inline void kscand_fork(struct mm_struct *mm, struct mm_struct *old= mm) {} +static inline void kscand_exit(struct mm_struct *mm) {} +#endif +#endif /* _LINUX_KSCAND_H_ */ diff --git a/kernel/fork.c b/kernel/fork.c index 168681fc4b25..af6dd315b106 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -630,6 +631,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *= mm, mm->exec_vm =3D oldmm->exec_vm; mm->stack_vm =3D oldmm->stack_vm; =20 + kscand_fork(mm, oldmm); + /* Use __mt_dup() to efficiently build an identical maple tree. */ retval =3D __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL); if (unlikely(retval)) @@ -1377,6 +1380,7 @@ static inline void __mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ + kscand_exit(mm); exit_mmap(mm); mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); diff --git a/mm/kscand.c b/mm/kscand.c index f7bbbc70c86a..d5b0d3041b0f 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -7,12 +7,14 @@ #include #include #include +#include #include #include #include =20 #include #include "internal.h" +#include "mm_slot.h" =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); @@ -29,11 +31,23 @@ static bool need_wakeup; =20 static unsigned long kscand_sleep_expire; =20 +static DEFINE_SPINLOCK(kscand_mm_lock); static DECLARE_WAIT_QUEUE_HEAD(kscand_wait); =20 +#define KSCAND_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kscand_slots_hash, KSCAND_SLOT_HASH_BI= TS); + +static struct kmem_cache *kscand_slot_cache __read_mostly; + +/* Per mm information collected to control VMA scanning */ +struct kscand_mm_slot { + struct mm_slot slot; +}; + /* Data structure to keep track of current mm under scan */ struct kscand_scan { struct list_head mm_head; + struct kscand_mm_slot *mm_slot; }; =20 struct kscand_scan kscand_scan =3D { @@ -69,6 +83,12 @@ static void kscand_wait_work(void) wait_event_timeout(kscand_wait, kscand_should_wakeup(), scan_sleep_jiffies); } + +static inline int kscand_test_exit(struct mm_struct *mm) +{ + return atomic_read(&mm->mm_users) =3D=3D 0; +} + static void kscand_do_scan(void) { unsigned long iter =3D 0, mms_to_scan; @@ -109,6 +129,65 @@ static int kscand(void *none) return 0; } =20 +static inline void kscand_destroy(void) +{ + kmem_cache_destroy(kscand_slot_cache); +} + +void __kscand_enter(struct mm_struct *mm) +{ + struct kscand_mm_slot *kscand_slot; + struct mm_slot *slot; + int wakeup; + + /* __kscand_exit() must not run from under us */ + VM_BUG_ON_MM(kscand_test_exit(mm), mm); + + kscand_slot =3D mm_slot_alloc(kscand_slot_cache); + + if (!kscand_slot) + return; + + slot =3D &kscand_slot->slot; + + spin_lock(&kscand_mm_lock); + mm_slot_insert(kscand_slots_hash, mm, slot); + + wakeup =3D list_empty(&kscand_scan.mm_head); + list_add_tail(&slot->mm_node, &kscand_scan.mm_head); + spin_unlock(&kscand_mm_lock); + + mmgrab(mm); + if (wakeup) + wake_up_interruptible(&kscand_wait); +} + +void __kscand_exit(struct mm_struct *mm) +{ + struct kscand_mm_slot *mm_slot; + struct mm_slot *slot; + int free =3D 0; + + spin_lock(&kscand_mm_lock); + slot =3D mm_slot_lookup(kscand_slots_hash, mm); + mm_slot =3D mm_slot_entry(slot, struct kscand_mm_slot, slot); + if (mm_slot && kscand_scan.mm_slot !=3D mm_slot) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + } + + spin_unlock(&kscand_mm_lock); + + if (free) { + mm_slot_free(kscand_slot_cache, mm_slot); + mmdrop(mm); + } else if (mm_slot) { + mmap_write_lock(mm); + mmap_write_unlock(mm); + } +} + static int start_kscand(void) { struct task_struct *kthread; @@ -149,6 +228,12 @@ static int __init kscand_init(void) { int err; =20 + kscand_slot_cache =3D KMEM_CACHE(kscand_mm_slot, 0); + + if (!kscand_slot_cache) { + pr_err("kscand: kmem_cache error"); + return -ENOMEM; + } err =3D start_kscand(); if (err) goto err_kscand; @@ -157,6 +242,7 @@ static int __init kscand_init(void) =20 err_kscand: stop_kscand(); + kscand_destroy(); =20 return err; } --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2055.outbound.protection.outlook.com [40.107.220.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 771DB235041 for ; Tue, 24 Jun 2025 05:57:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.55 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744651; cv=fail; b=ro0C9xQ4bquBUNnB4rrx2gt6Oq58pvLv5wTm4llJ2qBYTAg+PrhK2A0V8XLwwudtW7Kx02OUllwY7DzAzQyM0sa/VpJNx1/1+hRHjtCpoN0qKcdrpo7NlWEigq8liENYnwiY84Auz7ONdJQrxQhWeF5bL1RY6zBhaSRP5d4LVG8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744651; c=relaxed/simple; bh=DDTvV3EH0oSfBQ4LqDU8leK/MsSOmbUc0njsmecTL9s=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gS1uI/Xa3eXPSmnqPN2VbvBQZRb/bVHh9jYiuKx++hmcT5wqwPIfkXQxFqc0WYRj7EhkjrQ+grExJfAAoPM9MjjSoImUitQ7vThFnQBFv3CREyrQhZXYrVKu+aV6KagVz/IvA6ObtjfiPyIHw2y8M87Bidu1GNHuvt5ssMsC8Wc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=lWawDI9/; arc=fail smtp.client-ip=40.107.220.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="lWawDI9/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=F+Oc7nEAaUzkoFmZXQfAiBfRv3fhC7m+2bnWfXZ/xVH04aU+lCQDqIFI1VuEfA7fbUXN7cLJt2xWyfnqtsmrIcobOHItI0aITk0y71DeCi1byig3Dk0tqeVWRl/envNwgQ55M0vSSZw2qoxJe3E6QQdbWadNaUl8kxUhAct+rgAg67ePEVxck2FvTA+xh29FFu0InhMcyAWMRMZubZhR4431+n9KpTPrOktjImzBNeFA2saSki3UuqPbJROvUMPQ2wSDmq1eJa3ygie65UJsPvQ/qRy3EHc8f34KLeR9r+OhMtzMhScFxEC1OZE48+AQNWW5RsH7kC+fomNJrd2Qjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l6STkvmOzsktuMwCeKmHnc/mXOYML/fSkkNuBZ+9J9M=; b=fA2wKAkVeBbSzqbN7NaJSGopQN7msLIQHyc3ilhG8LzhhScZQHTSnKlO/uISSK2vPbIEvabnBsV/U70s7P5AHqWtt+38anOw8ehJapKfLRLhYJXSqD+O1tIfiZlIvoC+oqot3gD4kLk6DzcouLPYdYgKJFDOmB14+w4FJBDCM4z6n8gPeeRKZnhXWuXDo3fuWSX/iKaqvf224+wKeqcvaV1e9IpnWHWGXpjrYcmpHzfPom7tr5aDv3wa9y6vL+vchIOTKP2i7SbjCEfPilAhPjfU9Sd0AhScvEVeORVHFBSU3doyaHTWqFuF/6OpaZQ/CumX8Bi9o2GfDpsoYSBkMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l6STkvmOzsktuMwCeKmHnc/mXOYML/fSkkNuBZ+9J9M=; b=lWawDI9/rOubQolwHHHKDCR5poq1mm8uebTHUppO5h6BXIQFp5iLabDm0F18+b2+2qnuMEy/mnJVcF29IsJqKIYXKn4zjpXipi3Oic9YqHaIobXE/pPHMwAyNJ24s1QuHqvjDKJqsdKF1kZe6RSa8vAO6S2YYKW0Aq4M+O21dOw= Received: from BN9PR03CA0274.namprd03.prod.outlook.com (2603:10b6:408:f5::9) by PH7PR12MB8154.namprd12.prod.outlook.com (2603:10b6:510:2b9::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8880.16; Tue, 24 Jun 2025 05:57:23 +0000 Received: from BN1PEPF00005FFE.namprd05.prod.outlook.com (2603:10b6:408:f5:cafe::b2) by BN9PR03CA0274.outlook.office365.com (2603:10b6:408:f5::9) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8857.30 via Frontend Transport; Tue, 24 Jun 2025 05:57:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00005FFE.mail.protection.outlook.com (10.167.243.230) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:57:22 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:57:11 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 03/13] mm: Scan the mm and create a migration list Date: Tue, 24 Jun 2025 05:56:07 +0000 Message-ID: <20250624055617.1291159-4-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00005FFE:EE_|PH7PR12MB8154:EE_ X-MS-Office365-Filtering-Correlation-Id: 32fe6c97-093e-418b-b181-08ddb2e3fce1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|7416014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?cuXkVDsJeDTK03J3IUvB14rPiyNpLXSXJ+WnirzEit4wCmzYQxDPZZ7HJRpj?= =?us-ascii?Q?v53PN8MoZiXrE+SAmc80ZxaV+KTg4QSQuDuTgmVF18qwFCwFT4l2DItTgxXU?= =?us-ascii?Q?CA/FC6OeTHi8KdpRXpGy5ny/1XVg4jtfOx0QJOaOR3iypSWuHlfSBJ2n6cRk?= =?us-ascii?Q?f7vS4cCAIFMj2Zc6KnD/jcRm0hWKab4A9vcw6w+nlWj8Dy1jvEvm0yqrDyTq?= =?us-ascii?Q?e5jpSwg64y+JJj/T23gn1siT34CYN58qwt3qgfHxX7AueXKZw2WykfKltKnU?= =?us-ascii?Q?x+FWKvYkYmTTBPfMv/W0sA1Jl25RleQ4OQxB4qJz9s1rZYwDn0TbzopuCBO+?= =?us-ascii?Q?vJYgGNMdjU/jFn+OrevsGiJX6tbAwmbIR3HMkUeD0o0kIiCy/UU8r/AjbYD5?= =?us-ascii?Q?2/BJJUdYeUMAjUrDAphAM4dt4Ovi2Igo3FA3a3i/eIHnm0Ev+/44WKkaR9IG?= =?us-ascii?Q?b/TkPjhtkdidpo2MGOAturKMJn9j/JJYLRvdZHPvn6nqK4xKTiaZgtioUXU6?= =?us-ascii?Q?rYbEko820QOrHPaEfeO15SpD9goDEIrpotn9X1799k1zErp9xeldtMi4w+bf?= =?us-ascii?Q?GQFXwQJOUU0KNzQUjPl4rCnQBdQSMQishGks8yktTlUOA/mWGQpHjl29dFhO?= =?us-ascii?Q?GCgFLKVSpOp1lwTCvUCsmxAayY3eUk7zKCMl9SIQ8JVZaUeq+KIEzXUMYmKy?= =?us-ascii?Q?X5k073l1CgvdxK41HVq+gyi1acOBeri9ikwS1klkR/6hmfYLSUwbnb+qwZeA?= =?us-ascii?Q?JLlKjA50HfHDuJFagVig4LAHntQTb1F7ql/nfXk7tttli4uZaM7Ka0QqXoDB?= =?us-ascii?Q?nqmHpwlTxmuPQ4zgz4OuTgr0AB1Z9DzrIZDWLcx2OZcdt5AcuhihdmTn89aU?= =?us-ascii?Q?ZdPuoIThO+xs5aEq5OztVbNLFJI8CZ31rz4cUucpGGOdab9yZPP+lhnOs5dO?= =?us-ascii?Q?yR+9RquAsgkF3S99MCD5Tb28UyjcJqhX8hrUnVxZEAhGRmzG3NLIG1vUuOog?= =?us-ascii?Q?SGZxuXu1pxvbEiVXfqp0cBOq3V34dr6NVbZXO0W6YJpxwIFLQUe9gB0Og4cs?= =?us-ascii?Q?E1qLz3Im0db+Bnfnn3b7ixLBSXV96t8NvP0yQU3ZRWhRJn4daOaCYeu8aO3E?= =?us-ascii?Q?Q8N2JySrBzT6b0dxJGdDJdikM1Ux+go8OYjUgzog/A7YFE1ED9f7L5GL8X7h?= =?us-ascii?Q?RiN1XvdTCVbaVdrgJEpRVlRZL8vsYQtgrpk7qoMhjDnQzh+81DQq7Za5fo7C?= =?us-ascii?Q?U5LjpMN9O2jD4LV0eOTQkRv9AIl2AovqEBe/9jCqLw7nP1M8pFThZk5I/Oj7?= =?us-ascii?Q?ktATp7DewuOHzc0XUhSEVUK7zLtX8Pmv5WX0sinCQxrQgwplbqfgFer8ILN4?= =?us-ascii?Q?Fo3TEa8icz2DVylqdXkKoXLz/PeVcDZUtofDe4vwITk7wfGfOZPbbvL30Czd?= =?us-ascii?Q?QG4Dz/Pr1Pi3wxdQ5OYRwgB/yVcpHzZtCKU9cBxtd6uyC08GsVjZ+dGsZ36B?= =?us-ascii?Q?5ebKFi7X80m7RXP6SgVza+lpcp/dkU68PSKa?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(7416014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:57:22.6029 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32fe6c97-093e-418b-b181-08ddb2e3fce1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00005FFE.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8154 Content-Type: text/plain; charset="utf-8" Since we already have the list of mm_struct in the system, add a module to scan each mm that walks VMAs of each mm_struct and scan all the pages associated with that. In the scan path: Check for the recently acccessed pages (folios) belonging to slowtier nodes. Add all those folios to a list. Signed-off-by: Raghavendra K T --- mm/kscand.c | 319 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 318 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index d5b0d3041b0f..0edec1b7730d 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -4,10 +4,18 @@ #include #include #include +#include +#include +#include +#include +#include +#include #include #include #include #include +#include +#include #include #include #include @@ -18,6 +26,11 @@ =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); +/* + * Total VMA size to cover during scan. + */ +#define KSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) +static unsigned long kscand_scan_size __read_mostly =3D KSCAND_SCAN_SIZE; =20 /* How long to pause between two scan cycles */ static unsigned int kscand_scan_sleep_ms __read_mostly =3D 20; @@ -42,6 +55,8 @@ static struct kmem_cache *kscand_slot_cache __read_mostly; /* Per mm information collected to control VMA scanning */ struct kscand_mm_slot { struct mm_slot slot; + long address; + bool is_scanned; }; =20 /* Data structure to keep track of current mm under scan */ @@ -54,6 +69,29 @@ struct kscand_scan kscand_scan =3D { .mm_head =3D LIST_HEAD_INIT(kscand_scan.mm_head), }; =20 +/* + * Data structure passed to control scanning and also collect + * per memory node information + */ +struct kscand_scanctrl { + struct list_head scan_list; + unsigned long address; +}; + +struct kscand_scanctrl kscand_scanctrl; +/* Per folio information used for migration */ +struct kscand_migrate_info { + struct list_head migrate_node; + struct folio *folio; + unsigned long address; +}; + +static bool kscand_eligible_srcnid(int nid) +{ + /* Only promotion case is considered */ + return !node_is_toptier(nid); +} + static inline int kscand_has_work(void) { return !list_empty(&kscand_scan.mm_head); @@ -84,11 +122,275 @@ static void kscand_wait_work(void) scan_sleep_jiffies); } =20 +static inline bool is_valid_folio(struct folio *folio) +{ + if (!folio || folio_test_unevictable(folio) || !folio_mapped(folio) || + folio_is_zone_device(folio) || folio_maybe_mapped_shared(folio)) + return false; + + return true; +} + + +static bool folio_idle_clear_pte_refs_one(struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr, + pte_t *ptep) +{ + bool referenced =3D false; + struct mm_struct *mm =3D vma->vm_mm; + pmd_t *pmd =3D pmd_off(mm, addr); + + if (ptep) { + if (ptep_clear_young_notify(vma, addr, ptep)) + referenced =3D true; + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + if (!pmd_present(*pmd)) + WARN_ON_ONCE(1); + if (pmdp_clear_young_notify(vma, addr, pmd)) + referenced =3D true; + } else { + WARN_ON_ONCE(1); + } + + if (referenced) { + folio_clear_idle(folio); + folio_set_young(folio); + } + + return true; +} + +static void page_idle_clear_pte_refs(struct page *page, pte_t *pte, struct= mm_walk *walk) +{ + bool need_lock; + struct folio *folio =3D page_folio(page); + unsigned long address; + + if (!folio_mapped(folio) || !folio_raw_mapping(folio)) + return; + + need_lock =3D !folio_test_anon(folio) || folio_test_ksm(folio); + if (need_lock && !folio_trylock(folio)) + return; + address =3D vma_address(walk->vma, page_pgoff(folio, page), compound_nr(p= age)); + VM_BUG_ON_VMA(address =3D=3D -EFAULT, walk->vma); + folio_idle_clear_pte_refs_one(folio, walk->vma, address, pte); + + if (need_lock) + folio_unlock(folio); +} + +static int hot_vma_idle_pte_entry(pte_t *pte, + unsigned long addr, + unsigned long next, + struct mm_walk *walk) +{ + struct page *page; + struct folio *folio; + struct mm_struct *mm; + struct vm_area_struct *vma; + struct kscand_migrate_info *info; + struct kscand_scanctrl *scanctrl =3D walk->private; + int srcnid; + + scanctrl->address =3D addr; + pte_t pteval =3D ptep_get(pte); + + if (!pte_present(pteval)) + return 0; + + if (pte_none(pteval)) + return 0; + + vma =3D walk->vma; + mm =3D vma->vm_mm; + + page =3D pte_page(*pte); + + page_idle_clear_pte_refs(page, pte, walk); + + folio =3D page_folio(page); + folio_get(folio); + + if (!is_valid_folio(folio)) { + folio_put(folio); + return 0; + } + srcnid =3D folio_nid(folio); + + + if (!folio_test_lru(folio)) { + folio_put(folio); + return 0; + } + + if (!folio_test_idle(folio) || folio_test_young(folio) || + mmu_notifier_test_young(mm, addr) || + folio_test_referenced(folio) || pte_young(pteval)) { + + if (!kscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return 0; + } + /* XXX: Leaking memory. TBD: consume info */ + + info =3D kzalloc(sizeof(struct kscand_migrate_info), GFP_NOWAIT); + if (info && scanctrl) { + info->address =3D addr; + info->folio =3D folio; + list_add_tail(&info->migrate_node, &scanctrl->scan_list); + } + } + + folio_set_idle(folio); + folio_put(folio); + return 0; +} + +static const struct mm_walk_ops hot_vma_set_idle_ops =3D { + .pte_entry =3D hot_vma_idle_pte_entry, + .walk_lock =3D PGWALK_RDLOCK, +}; + +static void kscand_walk_page_vma(struct vm_area_struct *vma, struct kscand= _scanctrl *scanctrl) +{ + if (!vma_migratable(vma) || !vma_policy_mof(vma) || + is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + return; + } + if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) =3D=3D (VM_READ= ))) + return; + + if (!vma_is_accessible(vma)) + return; + + walk_page_vma(vma, &hot_vma_set_idle_ops, scanctrl); +} + static inline int kscand_test_exit(struct mm_struct *mm) { return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +static void kscand_collect_mm_slot(struct kscand_mm_slot *mm_slot) +{ + struct mm_slot *slot =3D &mm_slot->slot; + struct mm_struct *mm =3D slot->mm; + + lockdep_assert_held(&kscand_mm_lock); + + if (kscand_test_exit(mm)) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + + mm_slot_free(kscand_slot_cache, mm_slot); + mmdrop(mm); + } +} + +static unsigned long kscand_scan_mm_slot(void) +{ + bool next_mm =3D false; + bool update_mmslot_info =3D false; + + unsigned long vma_scanned_size =3D 0; + unsigned long address; + + struct mm_slot *slot; + struct mm_struct *mm; + struct vm_area_struct *vma =3D NULL; + struct kscand_mm_slot *mm_slot; + + + spin_lock(&kscand_mm_lock); + + if (kscand_scan.mm_slot) { + mm_slot =3D kscand_scan.mm_slot; + slot =3D &mm_slot->slot; + address =3D mm_slot->address; + } else { + slot =3D list_entry(kscand_scan.mm_head.next, + struct mm_slot, mm_node); + mm_slot =3D mm_slot_entry(slot, struct kscand_mm_slot, slot); + address =3D mm_slot->address; + kscand_scan.mm_slot =3D mm_slot; + } + + mm =3D slot->mm; + mm_slot->is_scanned =3D true; + spin_unlock(&kscand_mm_lock); + + if (unlikely(!mmap_read_trylock(mm))) + goto outerloop_mmap_lock; + + if (unlikely(kscand_test_exit(mm))) { + next_mm =3D true; + goto outerloop; + } + + VMA_ITERATOR(vmi, mm, address); + + for_each_vma(vmi, vma) { + kscand_walk_page_vma(vma, &kscand_scanctrl); + vma_scanned_size +=3D vma->vm_end - vma->vm_start; + + if (vma_scanned_size >=3D kscand_scan_size) { + next_mm =3D true; + /* TBD: Add scanned folios to migration list */ + break; + } + } + + if (!vma) + address =3D 0; + else + address =3D kscand_scanctrl.address + PAGE_SIZE; + + update_mmslot_info =3D true; + + if (update_mmslot_info) + mm_slot->address =3D address; + +outerloop: + /* exit_mmap will destroy ptes after this */ + mmap_read_unlock(mm); + +outerloop_mmap_lock: + spin_lock(&kscand_mm_lock); + WARN_ON(kscand_scan.mm_slot !=3D mm_slot); + + /* + * Release the current mm_slot if this mm is about to die, or + * if we scanned all vmas of this mm. + */ + if (unlikely(kscand_test_exit(mm)) || !vma || next_mm) { + /* + * Make sure that if mm_users is reaching zero while + * kscand runs here, kscand_exit will find + * mm_slot not pointing to the exiting mm. + */ + if (slot->mm_node.next !=3D &kscand_scan.mm_head) { + slot =3D list_entry(slot->mm_node.next, + struct mm_slot, mm_node); + kscand_scan.mm_slot =3D + mm_slot_entry(slot, struct kscand_mm_slot, slot); + + } else + kscand_scan.mm_slot =3D NULL; + + if (kscand_test_exit(mm)) { + kscand_collect_mm_slot(mm_slot); + goto end; + } + } + mm_slot->is_scanned =3D false; +end: + spin_unlock(&kscand_mm_lock); + return 0; +} + static void kscand_do_scan(void) { unsigned long iter =3D 0, mms_to_scan; @@ -101,7 +403,7 @@ static void kscand_do_scan(void) break; =20 if (kscand_has_work()) - msleep(100); + kscand_scan_mm_slot(); =20 iter++; =20 @@ -148,6 +450,7 @@ void __kscand_enter(struct mm_struct *mm) if (!kscand_slot) return; =20 + kscand_slot->address =3D 0; slot =3D &kscand_slot->slot; =20 spin_lock(&kscand_mm_lock); @@ -175,6 +478,12 @@ void __kscand_exit(struct mm_struct *mm) hash_del(&slot->hash); list_del(&slot->mm_node); free =3D 1; + } else if (mm_slot && kscand_scan.mm_slot =3D=3D mm_slot && !mm_slot->is_= scanned) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + /* TBD: Set the actual next slot */ + kscand_scan.mm_slot =3D NULL; } =20 spin_unlock(&kscand_mm_lock); @@ -224,6 +533,12 @@ static int stop_kscand(void) return 0; } =20 +static inline void init_list(void) +{ + INIT_LIST_HEAD(&kscand_scanctrl.scan_list); + init_waitqueue_head(&kscand_wait); +} + static int __init kscand_init(void) { int err; @@ -234,6 +549,8 @@ static int __init kscand_init(void) pr_err("kscand: kmem_cache error"); return -ENOMEM; } + + init_list(); err =3D start_kscand(); if (err) goto err_kscand; --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2059.outbound.protection.outlook.com [40.107.223.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99D9223D291 for ; Tue, 24 Jun 2025 05:57:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744658; cv=fail; b=k+O1PCxOqcG1dIyNDw/80ZBOVQPNI2sgCy+tQxxYyNrAuilvTpuWu8MjK4nXvUztPvFFkZo5ijsP9qvfob7c4jpDsycivt8aVfYBnx74qzxvo24oOBFZ8poSof6ejUrr6a8xmRIJUDXrvNGD08E7pm6WBU03vszHr+1qtJsEZ34= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744658; c=relaxed/simple; bh=mlKbFyLO879L9ZGO6W2Todke4iYvjPDaXw9kiH2FtTc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rt0rdlvL2urAC0BvLqc6ilfpQtEUXO333OMs2S8DWiKQSD7vTjQLj5TK8jpMwmcCLV6y0kwt4fN4qq+C2UY7Qn9RiSIHVfavTDr3Lb2wA87vHpweZTKeM+XkNYjAYKtAuepCsXuhFhB3klETVLvhIGnzjVBms01SBf295WC14M8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=xvJyB0q6; arc=fail smtp.client-ip=40.107.223.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="xvJyB0q6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xuKhrtBrhzTdBq979O6kM3zs1fyNCi9z8w7mnUCzno+lAaM54jXUWolxaVe7NDzZ/d+5ry+swQRWZ1HFht9L5KVl+5ySICV4/1S2GLhpaGkAxiI0DYNj2ojPsKHof8wVx5LbkW1pFbSmnUx31vy+DyX3zeNC2diw/dnc3528jgQQLvr8P0z3XMBp5u4vH4KmsgEhUtSjS0JeTGa/ggVIM+SYwUqhiTsYmamRDX0WAvyqCR4mv3Hq+BG2saT5UOBe5g0KQAWczOV+FnIoFdtSX4xyYzdysi7wi5YGJYiiH2r4+ZKKL+dLe2mgY5667x4atjgxgPbpvQ8obwNIGLwB8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CdOQYLhTNCqfmU1NWburArptQfvvwJQRaJdxML9mfgk=; b=PB58cMY6lnzAop+782r1g8sUScELVrMi6cETs4wh9bRzwGkTlRQuKy+pkUMNfxwzm34jEe7iaTgRS8F3rNoLx/yESETSQ9jHVHBFxe2K6wg0SLhF1tD1rGAxdSAXsOkdG6xoeNRVFJ0JpdYdQS2dzV71slSTemaC5jASZJEVnji2etEWFA8a0yMgFVjTOAOLjs5wCCChlI5vemObOfpkrhNRoBWtuzZBNnm4kJAUveiNoOSDmofCvam4/mVAyKBwglSnqnhdfNfkucYVpYnxR9OmDcf09WsdI4cpzA0+NqrQAlbw1Z6PFf7cQMfL+2yMxVEVauSmESCbDIX+V6nUVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CdOQYLhTNCqfmU1NWburArptQfvvwJQRaJdxML9mfgk=; b=xvJyB0q6TRk+nHp/ClBL6zctGZtF4P2X3Q7lB1UlxNQONDvYSRI3jdm/vrhpU/J2bpHHecdDqzMjaQn8sy1E7I41Ajy1p2HEMqByxcBWzQUJTT8X8bzSmrAUJ8flI0AwQgkvBRtMC1SLuPGL4rINuoQJ3oflgeQu3iJlE08I05w= Received: from BL1PR13CA0314.namprd13.prod.outlook.com (2603:10b6:208:2c1::19) by MW4PR12MB7311.namprd12.prod.outlook.com (2603:10b6:303:227::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.29; Tue, 24 Jun 2025 05:57:33 +0000 Received: from BN1PEPF00006003.namprd05.prod.outlook.com (2603:10b6:208:2c1:cafe::8f) by BL1PR13CA0314.outlook.office365.com (2603:10b6:208:2c1::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8880.15 via Frontend Transport; Tue, 24 Jun 2025 05:57:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006003.mail.protection.outlook.com (10.167.243.235) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:57:33 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:57:22 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 04/13] mm: Create a separate kthread for migration Date: Tue, 24 Jun 2025 05:56:08 +0000 Message-ID: <20250624055617.1291159-5-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006003:EE_|MW4PR12MB7311:EE_ X-MS-Office365-Filtering-Correlation-Id: 2ee3e0f7-99d2-4050-1d4b-08ddb2e40376 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?D5TdYBKICLghZNF4X5Ls02v2mzMzaXij1eFPDsYzkx+bLvF3n9wix8oSD0Vh?= =?us-ascii?Q?IncSw2r+5g7X86EbIscInfr677P63VNE8lsV5nXn2KSqOSF5ZBQXtINv6giL?= =?us-ascii?Q?htZREeIaWWGXBLFhwbmp9oNvdpWrmCh6ExIl4mqHsjIfTZiYom2xj6xnFBc0?= =?us-ascii?Q?yy7xiDHGCkbTH+xvTbDI4tQP+IerW8T7vp3jO/DVRaekah3MDJUPi+n0bmhr?= =?us-ascii?Q?owSE2JuKX8LCBCCEUP1vAbcfcb3ENSx+WcKvHGeoGugOBOrWyxIiEgUgl1/e?= =?us-ascii?Q?7xrljFbagC3/p7wItJWmcULigbdPhgXjw6N4/LnygybUaBd0JPjPJSMbuz3o?= =?us-ascii?Q?16De7sUFSRWAEh0G3ROaSw8U+x8CiJfMbjQDFJfkipGOdCYzjvbHQ6c34Z+y?= =?us-ascii?Q?yP/YytF7a7Dxum9kJQ6HKOHlQbBS0JE0HWyQsMvM2AWaVMXFqbRDWtr+udDl?= =?us-ascii?Q?93ZQyHFfFXL6AUboIxfpDtAOg26sBlu2BniKZc6xLuRFawk23eDKMbJCSaB5?= =?us-ascii?Q?vpifciJ0//D7H83yNQUaI020zu4q1wsZzDiy4g89KJfGvX0zKJVr2m6Rk7rh?= =?us-ascii?Q?M/K9MlAlXZg/sxzA4yc0UUEkhbISaPWmmpTGI6in93P85wAtcIf9GYVRI/Mh?= =?us-ascii?Q?Gxqe/FItZt+0UsouFGO/VsnUxsi4VBLVYnKzmKtH0KtVGvS20PB44X1b1NwV?= =?us-ascii?Q?yoXp2TxRsnVnB+/W36lQvjsGQ9C9tjBviz7j/M7+kpkTFWOXQqw9C9Ck3NcQ?= =?us-ascii?Q?hp+m9a0C6PBfLziW5Dd+st/eu4c/G6hxDrg/WpPq7H+AoutU5ykCYCBthJy7?= =?us-ascii?Q?H6viW4vUkR3HDFgv4SKTVnEwp4MzjCbq75jTDY2h9H/NTTtfXYkKKSd+kgGb?= =?us-ascii?Q?k2qeJEAUGDaK9HJKmV7iguiherKdaiRe6/438atTh0lV95p39O25AFFy+KST?= =?us-ascii?Q?0XtyLgGl3EJZzZzRF1UYpcITFT/YQ6xsIVib25QOC1nTh9qyS5Zu/xu1ZPEh?= =?us-ascii?Q?NXHYxi+lx+CJTKRIkzctT6IYHJl/D8MyCVRAew6cc+QnqY32Tbia5yL2kZ5U?= =?us-ascii?Q?Wm2m8xBJcdu9as1+axfRN7kLtaEq6w7D8yfk2uhioqqmRVZhKEcbSTVILxS7?= =?us-ascii?Q?Z2Qg00mRIfJRIauquqF0IQdCowoq3+Wnh4YtXl/LmCTEYBrbrUZEbef0i7zE?= =?us-ascii?Q?FFNJE1ociL8ztDu9Qu+JIFovfpAvMILlpvmOaAWVNh3nBgDn791L3hBOZkVg?= =?us-ascii?Q?QdM0bAk2xlxEiihnKOX55FnP3AT+aEtLj6UZziXUCVQaGuSK3rDBCQGtCQ1J?= =?us-ascii?Q?kaUt/0noCAYAN2ZBuU0GV47ErhKf3oxcMIcFfV5eJGLUfzv0uI+GUOqDQ/Zu?= =?us-ascii?Q?NKs15+gqdT4sn25K17XS/W0T71vCMV4T/QoFUyFLh1kEhLkTOerhIYkK2+lL?= =?us-ascii?Q?gbRkHI8FRr3lZtzo6H517BoMkrD/bujj7UBT3xX/VRC2r5zViHH5NE9QdqNW?= =?us-ascii?Q?U8NHbdQvGiCCbrUR2xKDM+GITSZ7SEt7w1gs?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:57:33.6417 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2ee3e0f7-99d2-4050-1d4b-08ddb2e40376 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006003.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7311 Content-Type: text/plain; charset="utf-8" Having independent thread helps in: - Alleviating the need for multiple scanning threads - Aids to control batch migration (TBD) - Migration throttling (TBD) Signed-off-by: Raghavendra K T --- mm/kscand.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index 0edec1b7730d..e4d8452b0c50 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -41,6 +42,15 @@ static unsigned long kscand_mms_to_scan __read_mostly = =3D KSCAND_MMS_TO_SCAN; =20 bool kscand_scan_enabled =3D true; static bool need_wakeup; +static bool migrated_need_wakeup; + +/* How long to pause between two migration cycles */ +static unsigned int kmigrate_sleep_ms __read_mostly =3D 20; + +static struct task_struct *kmigrated_thread __read_mostly; +static DEFINE_MUTEX(kmigrated_mutex); +static DECLARE_WAIT_QUEUE_HEAD(kmigrated_wait); +static unsigned long kmigrated_sleep_expire; =20 static unsigned long kscand_sleep_expire; =20 @@ -79,6 +89,7 @@ struct kscand_scanctrl { }; =20 struct kscand_scanctrl kscand_scanctrl; + /* Per folio information used for migration */ struct kscand_migrate_info { struct list_head migrate_node; @@ -131,6 +142,19 @@ static inline bool is_valid_folio(struct folio *folio) return true; } =20 +static inline void kmigrated_wait_work(void) +{ + const unsigned long migrate_sleep_jiffies =3D + msecs_to_jiffies(kmigrate_sleep_ms); + + if (!migrate_sleep_jiffies) + return; + + kmigrated_sleep_expire =3D jiffies + migrate_sleep_jiffies; + wait_event_timeout(kmigrated_wait, + true, + migrate_sleep_jiffies); +} =20 static bool folio_idle_clear_pte_refs_one(struct folio *folio, struct vm_area_struct *vma, @@ -533,6 +557,49 @@ static int stop_kscand(void) return 0; } =20 +static int kmigrated(void *arg) +{ + while (true) { + WRITE_ONCE(migrated_need_wakeup, false); + if (unlikely(kthread_should_stop())) + break; + msleep(20); + kmigrated_wait_work(); + } + return 0; +} + +static int start_kmigrated(void) +{ + struct task_struct *kthread; + + guard(mutex)(&kmigrated_mutex); + + /* Someone already succeeded in starting daemon */ + if (kmigrated_thread) + return 0; + + kthread =3D kthread_run(kmigrated, NULL, "kmigrated"); + if (IS_ERR(kmigrated_thread)) { + pr_err("kmigrated: kthread_run(kmigrated) failed\n"); + return PTR_ERR(kthread); + } + + kmigrated_thread =3D kthread; + pr_info("kmigrated: Successfully started kmigrated"); + + wake_up_interruptible(&kmigrated_wait); + + return 0; +} + +static int stop_kmigrated(void) +{ + guard(mutex)(&kmigrated_mutex); + kthread_stop(kmigrated_thread); + return 0; +} + static inline void init_list(void) { INIT_LIST_HEAD(&kscand_scanctrl.scan_list); @@ -555,8 +622,15 @@ static int __init kscand_init(void) if (err) goto err_kscand; =20 + err =3D start_kmigrated(); + if (err) + goto err_kmigrated; + return 0; =20 +err_kmigrated: + stop_kmigrated(); + err_kscand: stop_kscand(); kscand_destroy(); --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2077.outbound.protection.outlook.com [40.107.244.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49CB9231CB0 for ; Tue, 24 Jun 2025 05:57:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.77 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744674; cv=fail; b=kKgffPdcyx0JbnbeVtmsRxMRAAzoJAytN5KfCHoVqLpvOrkxsGKTkgSFZnIuLkG6CBwFcSvE+lOQ+U6yxRRA0qWZPaVngI5aKDG8OetsBieCHvC3xBf4wYbQLVX/jUCDsKVxrHIMvzQVqid6LeGAXbTdeEA8DObDNP5jDepaFEE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744674; c=relaxed/simple; bh=LACgJvTdaMP5BQHMQYLHzkWc+TlbTzL8BGpGZUURSFw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KCT0O1x1qXeB0AQQlh5j1WSXRDk42RPyYAi1HJOgAl7BktX8oi0CbeoKiwdfik3pCgZMadm33X8wHJ14Zk0Ikyuom5fBLwu02u+EdV1u1WJDgT5UTdX02t1OSRcMTl8U6Iy+E5TyH2lC+HrNfIrpuhYBTPO2F+DOH7yb5+Qparo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=3Yb7fdTg; arc=fail smtp.client-ip=40.107.244.77 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="3Yb7fdTg" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KAW4h6O6BVAtnrRdK322YufSv4Tc1s2g+/pwdqu5sWiiZi0WaBllnIJFrI7nRMZ5Iwupb5k71Fyb2R40Fm1XBjloRG6vUym0/ABlZF+TlrLK4s6Bo+73yHCLxGkcYl+EtSaOKMlwyeklVcLmn6pRQD99WZcIZ6G4USXobx/v6A0l/H1Bo/d8XPDt0NuY+t8kx7OS7fNjMNnUSRPPBfHL4KyawK372J4x2AeAuA1lNPjgDwYRXkccLFrkud+jysMt9YxUqbz92aGQeXvXM1eV8NAT1oEnMm/qAugrZKHAevz6faaAYLegOpSLaiZWr6zS/HS/yAPctNyLE9S6Qw2YcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=glQsVyQHnnGhIjor8P2Px5T+5e8BzfooNwGlpwuKpOQ=; b=rGW1mrUjdBSUSye1FkfbRDJwjfrB8dLLB6EYxPlj+nGubOlYywas5MX12Pc4AwC5Gd/cqSdL6SIZiN3rQab0cYaYsuhIeMddFVtAM7SPa7HO3zz2AZbyQOlvb1c/Ei2EfQYljJMd20D3f7aQslYtHkWz3EEpTPGUBeNfou3uM3vs1oouEhrKz/DvnGhwnJZu+garbDSCMypKTCRKK8KOvvUayKcOA0+3oY/NQYF/lEGT1hdk0r32VzrkBMn7Kc1awRw7o5QqEmOL11bVQ9OR1eG1ot4iEgp8oE1VHQ+cmfZE15WcT02vyG2Ze+pkYrjss7yzcIoGbPv+c2CQUF4rYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=glQsVyQHnnGhIjor8P2Px5T+5e8BzfooNwGlpwuKpOQ=; b=3Yb7fdTgGHTR2GoFUxt6LNCkjT8H9YXGQF26xmt/17Zv09ydWbnrqn1fDAHDZJQc2D9apPmSswUrhG+PlBJ+B17+qLm9LiqZpDAbuOg4txLyvgj6nPpxFc1NiA3dusCzQk/RauMEUb57bLA+8MuqOWhZw7KFLRrv+fV2qLcoDiI= Received: from MW4PR03CA0347.namprd03.prod.outlook.com (2603:10b6:303:dc::22) by SA3PR12MB7782.namprd12.prod.outlook.com (2603:10b6:806:31c::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.28; Tue, 24 Jun 2025 05:57:46 +0000 Received: from SJ5PEPF00000209.namprd05.prod.outlook.com (2603:10b6:303:dc:cafe::57) by MW4PR03CA0347.outlook.office365.com (2603:10b6:303:dc::22) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8857.30 via Frontend Transport; Tue, 24 Jun 2025 05:57:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000209.mail.protection.outlook.com (10.167.244.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:57:45 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:57:33 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 05/13] mm/migration: Migrate accessed folios to toptier node Date: Tue, 24 Jun 2025 05:56:09 +0000 Message-ID: <20250624055617.1291159-6-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000209:EE_|SA3PR12MB7782:EE_ X-MS-Office365-Filtering-Correlation-Id: 35bc97f3-ca80-423a-ac18-08ddb2e40aa6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?V6K/j6f+3UYG9U6SzW+Zca6P8Ca2JbvHKgbhtq9+mZbsI3To4XCmCMtiDHud?= =?us-ascii?Q?4S4cq3qRmtlm9YEAQjyAOZH2ZYlu9sRF2+aJLPJ1ZOxV4j9pOhV9swzxN426?= =?us-ascii?Q?zAhTX2h5xs6ezMAmtfaU/eJ1kTJck+S/FzciYWu9pKBIgZAeOHEb0KTAZHS9?= =?us-ascii?Q?sZAsdYg6uidvus5glpHuvWJsMPLGqBPIpESxxk+Tnqe/t4gWBY5EsAq2RfbF?= =?us-ascii?Q?wXubFQHaqb9gaxLmU985AwhijvAH+gOS2CjV4pp1oCB/7bMO2LveDrKvN81P?= =?us-ascii?Q?4loSOhH/+VPJrB0/Lh7YYxYo4BIpngOkI516itSIUG/enppi1iVzsITT4Laq?= =?us-ascii?Q?lroy5PazArCvRWMtYzz60RU3QX5FgvrtQMbvqbyd6Lip200zKFEBYF8zzqPu?= =?us-ascii?Q?UxwY5GPVeV28NJZjLwFZ8edfV6gOYKveuUivOQ4VZibtPBP2BF0eXuLiFCGN?= =?us-ascii?Q?CS3QUJCdf2hYV3oHdExaT+LuvqXKLqoMN70fUfc+bh+OtcfCcpipntQtsZqR?= =?us-ascii?Q?sWBe1ajKbNdRSjZzCMOuvBmcZhwiw+wqso5jAcTugsiVqzCH3N39DoCwRAyB?= =?us-ascii?Q?dt14rMX91s/oMajjYbCLUjdTRqfGKkNL7qrZOHDFtmj5CzfEJzrbLaKws6/Z?= =?us-ascii?Q?epW16V0gdLPNMwX4bk3u/WruYDDawBTv2F4QOcsStKHkPJ7EaCX7648dIN4a?= =?us-ascii?Q?7N9fjxwQh+1hyo6/D3nPxkXUj6RXG4RzENeuI07Q+ISM05LbGjq2x6uhLkuD?= =?us-ascii?Q?sKrlMklwEhi1Uets7XtL8mFaJRks0BYvXY1bEC4oAKt48UVS3v/tN5pe639I?= =?us-ascii?Q?hZcJVOQA6BvgMyxk8I6ktswC7s118fnJj4VycO58qVfcRLl2n/DncgQk4mHa?= =?us-ascii?Q?11zvobOVG3C+oJh0NIA1kmOu/OlESASKgclthyAFCz97DHUckA9Nb7vJ58A0?= =?us-ascii?Q?WkFyIez9ZklNPeuYA32K8yWd11eUysbDlI1bLs96LSrD6aruxJ0cb+TdGYeO?= =?us-ascii?Q?EvI6SZ0LsKYDGTbfU2cZZCNxeRPe0gRQlRpMcUVvi2V3NyYukCfDeVylYHE4?= =?us-ascii?Q?zfQW+PWRn4pg4Ja6xoOCcigiUeSrbNYn5qdBaK4z4UlV2Yh7JLWMCVpTQ4tZ?= =?us-ascii?Q?VkGGk0dc0p7YJm8bg5gJ4BM3/xMDlvPX3NyJr0rk7/mdFkemI4Vurc35cZVT?= =?us-ascii?Q?OQYqkYKbkWo2SltkAlwjiKueQdwN5NHsLSX9z+34rMseLJvKzAqr8C6Qiltx?= =?us-ascii?Q?dKa06el4aOaAbr+9j/Hk3aVWFVnElTwk9giTydsA/l6SEtXfM/oBgzcXv7Bg?= =?us-ascii?Q?GPiQ8bA0sn6LzhQc6jgzV4tH1i9cDWH+qogCXCNk4i+PAvlqq4qkb6ZjHf6Z?= =?us-ascii?Q?AgHV0GczKAw10wR8/IJfi7LlC1kOTb9Cuvq+KYTlCKLdcoqH46Th71dG1Z7a?= =?us-ascii?Q?Bv7exzVcoSSn61+Nrh2xPyG4pSwyNttMm6oeQgxWzu5tIIdDHPZW9M8oXBOR?= =?us-ascii?Q?2h1I950ubp8EmRzzgv/Y3E0ZGYZqiuXZwy/f?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:57:45.6247 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 35bc97f3-ca80-423a-ac18-08ddb2e40aa6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000209.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7782 Content-Type: text/plain; charset="utf-8" A per mm migration list is added and a kernel thread iterates over each of them. For each recently accessed slowtier folio in the migration list: - Isolate LRU pages - Migrate to a regular node. The rationale behind whole migration is to speedup the access to recently accessed pages. Currently, PTE A bit scanning approach lacks information about exact destination node to migrate to. Reason: PROT_NONE hint fault based scanning is done in a process context. Here when the fault occurs, source CPU of the fault associated task is known. Time of page access is also accurate. With the lack of above information, migration is done to node 0 by default. Signed-off-by: Raghavendra K T --- include/linux/migrate.h | 2 + mm/kscand.c | 452 ++++++++++++++++++++++++++++++++++++++-- mm/migrate.c | 2 +- 3 files changed, 443 insertions(+), 13 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index aaa2114498d6..59547f72d150 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -142,6 +142,8 @@ const struct movable_operations *page_movable_ops(struc= t page *page) } =20 #ifdef CONFIG_NUMA_BALANCING +bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); diff --git a/mm/kscand.c b/mm/kscand.c index e4d8452b0c50..f79e05590cde 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -52,9 +52,18 @@ static DEFINE_MUTEX(kmigrated_mutex); static DECLARE_WAIT_QUEUE_HEAD(kmigrated_wait); static unsigned long kmigrated_sleep_expire; =20 +/* mm of the migrating folio entry */ +static struct mm_struct *kmigrated_cur_mm; + +/* Migration list is manipulated underneath because of mm_exit */ +static bool kmigrated_clean_list; + static unsigned long kscand_sleep_expire; +#define KSCAND_DEFAULT_TARGET_NODE (0) +static int kscand_target_node =3D KSCAND_DEFAULT_TARGET_NODE; =20 static DEFINE_SPINLOCK(kscand_mm_lock); +static DEFINE_SPINLOCK(kscand_migrate_lock); static DECLARE_WAIT_QUEUE_HEAD(kscand_wait); =20 #define KSCAND_SLOT_HASH_BITS 10 @@ -62,6 +71,10 @@ static DEFINE_READ_MOSTLY_HASHTABLE(kscand_slots_hash, K= SCAND_SLOT_HASH_BITS); =20 static struct kmem_cache *kscand_slot_cache __read_mostly; =20 +#define KMIGRATED_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kmigrated_slots_hash, KMIGRATED_SLOT_H= ASH_BITS); +static struct kmem_cache *kmigrated_slot_cache __read_mostly; + /* Per mm information collected to control VMA scanning */ struct kscand_mm_slot { struct mm_slot slot; @@ -90,6 +103,26 @@ struct kscand_scanctrl { =20 struct kscand_scanctrl kscand_scanctrl; =20 +/* Per mm migration list */ +struct kmigrated_mm_slot { + /* Tracks mm that has non empty migration list */ + struct mm_slot mm_slot; + /* Per mm lock used to synchronize migration list */ + spinlock_t migrate_lock; + /* Head of per mm migration list */ + struct list_head migrate_head; +}; + +/* System wide list of mms that maintain migration list */ +struct kmigrated_daemon { + struct list_head mm_head; + struct kmigrated_mm_slot *mm_slot; +}; + +struct kmigrated_daemon kmigrated_daemon =3D { + .mm_head =3D LIST_HEAD_INIT(kmigrated_daemon.mm_head), +}; + /* Per folio information used for migration */ struct kscand_migrate_info { struct list_head migrate_node; @@ -108,6 +141,11 @@ static inline int kscand_has_work(void) return !list_empty(&kscand_scan.mm_head); } =20 +static inline int kmigrated_has_work(void) +{ + return !list_empty(&kmigrated_daemon.mm_head); +} + static inline bool kscand_should_wakeup(void) { bool wakeup =3D kthread_should_stop() || need_wakeup || @@ -118,6 +156,16 @@ static inline bool kscand_should_wakeup(void) return wakeup; } =20 +static inline bool kmigrated_should_wakeup(void) +{ + bool wakeup =3D kthread_should_stop() || migrated_need_wakeup || + time_after_eq(jiffies, kmigrated_sleep_expire); + + migrated_need_wakeup =3D false; + + return wakeup; +} + static void kscand_wait_work(void) { const unsigned long scan_sleep_jiffies =3D @@ -133,6 +181,85 @@ static void kscand_wait_work(void) scan_sleep_jiffies); } =20 +static void kmigrated_wait_work(void) +{ + const unsigned long migrate_sleep_jiffies =3D + msecs_to_jiffies(kmigrate_sleep_ms); + + if (!migrate_sleep_jiffies) + return; + + kmigrated_sleep_expire =3D jiffies + migrate_sleep_jiffies; + wait_event_timeout(kmigrated_wait, kmigrated_should_wakeup(), + migrate_sleep_jiffies); +} + +/* + * Do not know what info to pass in the future to make + * decision on taget node. Keep it void * now. + */ +static int kscand_get_target_node(void *data) +{ + return kscand_target_node; +} + +extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); + +/*XXX: Taken from migrate.c to avoid NUMAB mode=3D2 and NULL vma checks*/ +static int kscand_migrate_misplaced_folio_prepare(struct folio *folio, + struct vm_area_struct *vma, int node) +{ + int nr_pages =3D folio_nr_pages(folio); + pg_data_t *pgdat =3D NODE_DATA(node); + + if (folio_is_file_lru(folio)) { + /* + * Do not migrate file folios that are mapped in multiple + * processes with execute permissions as they are probably + * shared libraries. + * + * See folio_maybe_mapped_shared() on possible imprecision + * when we cannot easily detect if a folio is shared. + */ + if (vma && (vma->vm_flags & VM_EXEC) && + folio_maybe_mapped_shared(folio)) + return -EACCES; + /* + * Do not migrate dirty folios as not all filesystems can move + * dirty folios in MIGRATE_ASYNC mode which is a waste of + * cycles. + */ + if (folio_test_dirty(folio)) + return -EAGAIN; + } + + /* Avoid migrating to a node that is nearly full */ + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + for (z =3D pgdat->nr_zones - 1; z >=3D 0; z--) { + if (managed_zone(pgdat->node_zones + z)) + break; + } + + if (z < 0) + return -EAGAIN; + + wakeup_kswapd(pgdat->node_zones + z, 0, + folio_order(folio), ZONE_MOVABLE); + return -EAGAIN; + } + + if (!folio_isolate_lru(folio)) + return -EAGAIN; + + node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), + nr_pages); + + return 0; +} + static inline bool is_valid_folio(struct folio *folio) { if (!folio || folio_test_unevictable(folio) || !folio_mapped(folio) || @@ -142,18 +269,116 @@ static inline bool is_valid_folio(struct folio *foli= o) return true; } =20 -static inline void kmigrated_wait_work(void) +enum kscand_migration_err { + KSCAND_NULL_MM =3D 1, + KSCAND_EXITING_MM, + KSCAND_INVALID_FOLIO, + KSCAND_NONLRU_FOLIO, + KSCAND_INELIGIBLE_SRC_NODE, + KSCAND_SAME_SRC_DEST_NODE, + KSCAND_PTE_NOT_PRESENT, + KSCAND_PMD_NOT_PRESENT, + KSCAND_NO_PTE_OFFSET_MAP_LOCK, + KSCAND_NOT_HOT_PAGE, + KSCAND_LRU_ISOLATION_ERR, +}; + +static bool is_hot_page(struct folio *folio) { - const unsigned long migrate_sleep_jiffies =3D - msecs_to_jiffies(kmigrate_sleep_ms); +#ifdef CONFIG_LRU_GEN + struct lruvec *lruvec; + int gen =3D folio_lru_gen(folio); + + lruvec =3D folio_lruvec(folio); + return lru_gen_is_active(lruvec, gen); +#else + return folio_test_active(folio); +#endif +} =20 - if (!migrate_sleep_jiffies) - return; +static int kmigrated_promote_folio(struct kscand_migrate_info *info, + struct mm_struct *mm, + int destnid) +{ + unsigned long pfn; + unsigned long address; + struct page *page; + struct folio *folio; + int ret; + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; + pmd_t pmde; + int srcnid; =20 - kmigrated_sleep_expire =3D jiffies + migrate_sleep_jiffies; - wait_event_timeout(kmigrated_wait, - true, - migrate_sleep_jiffies); + if (mm =3D=3D NULL) + return KSCAND_NULL_MM; + + if (mm =3D=3D READ_ONCE(kmigrated_cur_mm) && + READ_ONCE(kmigrated_clean_list)) { + WARN_ON_ONCE(mm); + return KSCAND_EXITING_MM; + } + + folio =3D info->folio; + + /* Check again if the folio is really valid now */ + if (folio) { + pfn =3D folio_pfn(folio); + page =3D pfn_to_online_page(pfn); + } + + if (!page || PageTail(page) || !is_valid_folio(folio)) + return KSCAND_INVALID_FOLIO; + + if (!folio_test_lru(folio)) + return KSCAND_NONLRU_FOLIO; + + if (!is_hot_page(folio)) + return KSCAND_NOT_HOT_PAGE; + + folio_get(folio); + + srcnid =3D folio_nid(folio); + + /* Do not try to promote pages from regular nodes */ + if (!kscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return KSCAND_INELIGIBLE_SRC_NODE; + } + + /* Also happen when it is already migrated */ + if (srcnid =3D=3D destnid) { + folio_put(folio); + return KSCAND_SAME_SRC_DEST_NODE; + } + address =3D info->address; + pmd =3D pmd_off(mm, address); + pmde =3D pmdp_get(pmd); + + if (!pmd_present(pmde)) { + folio_put(folio); + return KSCAND_PMD_NOT_PRESENT; + } + + pte =3D pte_offset_map_lock(mm, pmd, address, &ptl); + if (!pte) { + folio_put(folio); + WARN_ON_ONCE(!pte); + return KSCAND_NO_PTE_OFFSET_MAP_LOCK; + } + + ret =3D kscand_migrate_misplaced_folio_prepare(folio, NULL, destnid); + if (ret) { + folio_put(folio); + pte_unmap_unlock(pte, ptl); + return KSCAND_LRU_ISOLATION_ERR; + } + + folio_put(folio); + pte_unmap_unlock(pte, ptl); + + return migrate_misplaced_folio(folio, destnid); } =20 static bool folio_idle_clear_pte_refs_one(struct folio *folio, @@ -257,7 +482,6 @@ static int hot_vma_idle_pte_entry(pte_t *pte, folio_put(folio); return 0; } - /* XXX: Leaking memory. TBD: consume info */ =20 info =3D kzalloc(sizeof(struct kscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { @@ -298,6 +522,115 @@ static inline int kscand_test_exit(struct mm_struct *= mm) return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +struct destroy_list_work { + struct list_head migrate_head; + struct work_struct dwork; +}; + +static void kmigrated_destroy_list_fn(struct work_struct *work) +{ + struct destroy_list_work *dlw; + struct kscand_migrate_info *info, *tmp; + + dlw =3D container_of(work, struct destroy_list_work, dwork); + + if (!list_empty(&dlw->migrate_head)) { + list_for_each_entry_safe(info, tmp, &dlw->migrate_head, migrate_node) { + list_del(&info->migrate_node); + kfree(info); + } + } + + kfree(dlw); +} + +static void kmigrated_destroy_list(struct list_head *list_head) +{ + struct destroy_list_work *destroy_list_work; + + + destroy_list_work =3D kmalloc(sizeof(*destroy_list_work), GFP_KERNEL); + if (!destroy_list_work) + return; + + INIT_LIST_HEAD(&destroy_list_work->migrate_head); + list_splice_tail_init(list_head, &destroy_list_work->migrate_head); + INIT_WORK(&destroy_list_work->dwork, kmigrated_destroy_list_fn); + schedule_work(&destroy_list_work->dwork); +} + +static struct kmigrated_mm_slot *kmigrated_get_mm_slot(struct mm_struct *m= m, bool alloc) +{ + struct kmigrated_mm_slot *mm_slot =3D NULL; + struct mm_slot *slot; + + guard(spinlock)(&kscand_migrate_lock); + + slot =3D mm_slot_lookup(kmigrated_slots_hash, mm); + mm_slot =3D mm_slot_entry(slot, struct kmigrated_mm_slot, mm_slot); + + if (!mm_slot && alloc) { + mm_slot =3D mm_slot_alloc(kmigrated_slot_cache); + if (!mm_slot) { + spin_unlock(&kscand_migrate_lock); + return NULL; + } + + slot =3D &mm_slot->mm_slot; + INIT_LIST_HEAD(&mm_slot->migrate_head); + spin_lock_init(&mm_slot->migrate_lock); + mm_slot_insert(kmigrated_slots_hash, mm, slot); + list_add_tail(&slot->mm_node, &kmigrated_daemon.mm_head); + } + + return mm_slot; +} + +static void kscand_cleanup_migration_list(struct mm_struct *mm) +{ + struct kmigrated_mm_slot *mm_slot; + struct mm_slot *slot; + + mm_slot =3D kmigrated_get_mm_slot(mm, false); + + slot =3D &mm_slot->mm_slot; + + if (mm_slot && slot && slot->mm =3D=3D mm) { + spin_lock(&mm_slot->migrate_lock); + + if (!list_empty(&mm_slot->migrate_head)) { + if (mm =3D=3D READ_ONCE(kmigrated_cur_mm)) { + /* A folio in this mm is being migrated. wait */ + WRITE_ONCE(kmigrated_clean_list, true); + } + + kmigrated_destroy_list(&mm_slot->migrate_head); + spin_unlock(&mm_slot->migrate_lock); +retry: + if (!spin_trylock(&mm_slot->migrate_lock)) { + cpu_relax(); + goto retry; + } + + if (mm =3D=3D READ_ONCE(kmigrated_cur_mm)) { + spin_unlock(&mm_slot->migrate_lock); + goto retry; + } + } + /* Reset migrated mm_slot if it was pointing to us */ + if (kmigrated_daemon.mm_slot =3D=3D mm_slot) + kmigrated_daemon.mm_slot =3D NULL; + + hash_del(&slot->hash); + list_del(&slot->mm_node); + mm_slot_free(kmigrated_slot_cache, mm_slot); + + WRITE_ONCE(kmigrated_clean_list, false); + + spin_unlock(&mm_slot->migrate_lock); + } +} + static void kscand_collect_mm_slot(struct kscand_mm_slot *mm_slot) { struct mm_slot *slot =3D &mm_slot->slot; @@ -309,11 +642,77 @@ static void kscand_collect_mm_slot(struct kscand_mm_s= lot *mm_slot) hash_del(&slot->hash); list_del(&slot->mm_node); =20 + kscand_cleanup_migration_list(mm); + mm_slot_free(kscand_slot_cache, mm_slot); mmdrop(mm); } } =20 +static void kmigrated_migrate_mm(struct kmigrated_mm_slot *mm_slot) +{ + int ret =3D 0, dest =3D -1; + struct mm_slot *slot; + struct mm_struct *mm; + struct kscand_migrate_info *info, *tmp; + + spin_lock(&mm_slot->migrate_lock); + + slot =3D &mm_slot->mm_slot; + mm =3D slot->mm; + + if (!list_empty(&mm_slot->migrate_head)) { + list_for_each_entry_safe(info, tmp, &mm_slot->migrate_head, + migrate_node) { + if (READ_ONCE(kmigrated_clean_list)) + goto clean_list_handled; + + list_del(&info->migrate_node); + + spin_unlock(&mm_slot->migrate_lock); + + dest =3D kscand_get_target_node(NULL); + ret =3D kmigrated_promote_folio(info, mm, dest); + + kfree(info); + + cond_resched(); + spin_lock(&mm_slot->migrate_lock); + } + } +clean_list_handled: + /* Reset mm of folio entry we are migrating */ + WRITE_ONCE(kmigrated_cur_mm, NULL); + spin_unlock(&mm_slot->migrate_lock); +} + +static void kmigrated_migrate_folio(void) +{ + /* for each mm do migrate */ + struct kmigrated_mm_slot *kmigrated_mm_slot =3D NULL; + struct mm_slot *slot; + + if (!list_empty(&kmigrated_daemon.mm_head)) { + + scoped_guard (spinlock, &kscand_migrate_lock) { + if (kmigrated_daemon.mm_slot) { + kmigrated_mm_slot =3D kmigrated_daemon.mm_slot; + } else { + slot =3D list_entry(kmigrated_daemon.mm_head.next, + struct mm_slot, mm_node); + + kmigrated_mm_slot =3D mm_slot_entry(slot, + struct kmigrated_mm_slot, mm_slot); + kmigrated_daemon.mm_slot =3D kmigrated_mm_slot; + } + WRITE_ONCE(kmigrated_cur_mm, kmigrated_mm_slot->mm_slot.mm); + } + + if (kmigrated_mm_slot) + kmigrated_migrate_mm(kmigrated_mm_slot); + } +} + static unsigned long kscand_scan_mm_slot(void) { bool next_mm =3D false; @@ -327,6 +726,7 @@ static unsigned long kscand_scan_mm_slot(void) struct vm_area_struct *vma =3D NULL; struct kscand_mm_slot *mm_slot; =20 + struct kmigrated_mm_slot *kmigrated_mm_slot =3D NULL; =20 spin_lock(&kscand_mm_lock); =20 @@ -356,13 +756,23 @@ static unsigned long kscand_scan_mm_slot(void) =20 VMA_ITERATOR(vmi, mm, address); =20 + kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, false); + for_each_vma(vmi, vma) { kscand_walk_page_vma(vma, &kscand_scanctrl); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 if (vma_scanned_size >=3D kscand_scan_size) { next_mm =3D true; - /* TBD: Add scanned folios to migration list */ + + if (!list_empty(&kscand_scanctrl.scan_list)) { + if (!kmigrated_mm_slot) + kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, true); + spin_lock(&kmigrated_mm_slot->migrate_lock); + list_splice_tail_init(&kscand_scanctrl.scan_list, + &kmigrated_mm_slot->migrate_head); + spin_unlock(&kmigrated_mm_slot->migrate_lock); + } break; } } @@ -458,6 +868,8 @@ static int kscand(void *none) static inline void kscand_destroy(void) { kmem_cache_destroy(kscand_slot_cache); + /* XXX: move below to kmigrated thread */ + kmem_cache_destroy(kmigrated_slot_cache); } =20 void __kscand_enter(struct mm_struct *mm) @@ -493,7 +905,7 @@ void __kscand_exit(struct mm_struct *mm) { struct kscand_mm_slot *mm_slot; struct mm_slot *slot; - int free =3D 0; + int free =3D 0, serialize =3D 1; =20 spin_lock(&kscand_mm_lock); slot =3D mm_slot_lookup(kscand_slots_hash, mm); @@ -508,10 +920,15 @@ void __kscand_exit(struct mm_struct *mm) free =3D 1; /* TBD: Set the actual next slot */ kscand_scan.mm_slot =3D NULL; + } else if (mm_slot && kscand_scan.mm_slot =3D=3D mm_slot && mm_slot->is_s= canned) { + serialize =3D 0; } =20 spin_unlock(&kscand_mm_lock); =20 + if (serialize) + kscand_cleanup_migration_list(mm); + if (free) { mm_slot_free(kscand_slot_cache, mm_slot); mmdrop(mm); @@ -563,6 +980,8 @@ static int kmigrated(void *arg) WRITE_ONCE(migrated_need_wakeup, false); if (unlikely(kthread_should_stop())) break; + if (kmigrated_has_work()) + kmigrated_migrate_folio(); msleep(20); kmigrated_wait_work(); } @@ -603,7 +1022,9 @@ static int stop_kmigrated(void) static inline void init_list(void) { INIT_LIST_HEAD(&kscand_scanctrl.scan_list); + spin_lock_init(&kscand_migrate_lock); init_waitqueue_head(&kscand_wait); + init_waitqueue_head(&kmigrated_wait); } =20 static int __init kscand_init(void) @@ -617,6 +1038,13 @@ static int __init kscand_init(void) return -ENOMEM; } =20 + kmigrated_slot_cache =3D KMEM_CACHE(kscand_mm_slot, 0); + + if (!kmigrated_slot_cache) { + pr_err("kmigrated: kmem_cache error"); + return -ENOMEM; + } + init_list(); err =3D start_kscand(); if (err) diff --git a/mm/migrate.c b/mm/migrate.c index 676d9cfc7059..94448062d009 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2598,7 +2598,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long= , nr_pages, * Returns true if this is a safe migration target node for misplaced NUMA * pages. Currently it only checks the watermarks which is crude. */ -static bool migrate_balanced_pgdat(struct pglist_data *pgdat, +bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages) { int z; --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2053.outbound.protection.outlook.com [40.107.92.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0CCBB241116 for ; Tue, 24 Jun 2025 05:58:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.53 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744682; cv=fail; b=aMHA4wmvRh3lawfCB4RciWP3tOrF3c4Yg5ZPFZYGRzyzotcnDzNq9X9v362St00yglVZC5xMmwc17v1UF/F3A1546YT7j2z1UefRpbW31UU5hMOnjSnWZyF6rIshrepm+e1650QESAy/o39K0cIcg/koZJUT7vBMExg+ZjONvMk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744682; c=relaxed/simple; bh=QMxHxEisa9gd/DwGZI+p4bpmKZdWipqdRizPxFi94Js=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AHzJ02tUvgisnLdr1FNCA2hXwpyqoRjX3t+jWpoA2vh1CAAUsx6KtruQaXcZm+4mIqGZ5ImOWomHJ+OH3d3V13+X60qfuzdD3JN+P5vZiX9e/WIHk1iHS8FM6iHihleVncgP42LnKgbB7SH/QtVaHKWaWtY3TrGTO2JMX+vpQ1I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=VSuTX4k0; arc=fail smtp.client-ip=40.107.92.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="VSuTX4k0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NWKiEZ2HEUHSKW3ZVtUZAgswzlboMXFosdhc8OcE7EeaOSS29PjOqayh/J697u+DeUFoz+tTRH/O3LUu2EmugU0bgUy9ujSQj0OJAXHDq3MA0cB95AxHW4wRfxy79HPacn0NwBd0v96E7GOqyXMLUNNnVaf1WwGpV55HVHZoAWS5adHGHxN8ot2EZLQvi4ynTcnjsFtjLGfd74RazYDECk3zQSq/L627tCzOQOoaGJVXuVEZU4m5jkGTpIGAiG1uWPK2LwhdoWbkGt58WrpAagbt/M5r21PQUwPyG1gWw2HyICcajel9c79Er+byKdR2xrDkDilrICuTm8cgxog6Hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Dk35rIFkUSNo9wlPLmTXZUYQYCvIiEHxKbhpJSS5BDE=; b=W467jeNWm0fqw97mEZvtYqW/RFhuBJCaXOzD5p7FxOEpOaiDDb1yrmJBg36Eq7q+H4uZQ4a8aMnt7Lrba0UefojKWgdEJiwbv6NCRe52SMDkccw7WP3s6F8/f2Ho8gzvihV6nH4TaVhklghPjybC/G7sv5S2OB1tjBFj7sX6HYlGdZc4WDrA4FMKS5bAHTsHE78b7d6IhpWqqIk2KeF2oVQGn0lkBk6eNZqAOk8pS9gvw7+IIgu7dlVuWggbVIYQH7dIqK4Tu267r/FuAkU6bY6HrSXbWTRf1LstTpyR7ZnEEC57faE9fGIE6Ts3tULaUrpYsVwfiIK7fG/PzbhLCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Dk35rIFkUSNo9wlPLmTXZUYQYCvIiEHxKbhpJSS5BDE=; b=VSuTX4k0dcS8MDaOP2xkl9vxog7XLrAG76ZMirXkPs9gESefdeVtCNi8JYMFRFSuhvwyqXrFUNeuvW48c0/3OGUG5m04d12PsbdJ2WyJN4wUnslc+KuhBXjNdjzM7qkcbzr/+y0Y95lArs4vAHgC5sDtfM8gacNYZ+tp6NErT3Q= Received: from SJ0PR13CA0033.namprd13.prod.outlook.com (2603:10b6:a03:2c2::8) by MN2PR12MB4376.namprd12.prod.outlook.com (2603:10b6:208:26c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.30; Tue, 24 Jun 2025 05:57:56 +0000 Received: from SJ5PEPF00000206.namprd05.prod.outlook.com (2603:10b6:a03:2c2:cafe::9f) by SJ0PR13CA0033.outlook.office365.com (2603:10b6:a03:2c2::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8880.16 via Frontend Transport; Tue, 24 Jun 2025 05:57:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000206.mail.protection.outlook.com (10.167.244.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:57:56 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:57:44 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 06/13] mm: Add throttling of mm scanning using scan_period Date: Tue, 24 Jun 2025 05:56:10 +0000 Message-ID: <20250624055617.1291159-7-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000206:EE_|MN2PR12MB4376:EE_ X-MS-Office365-Filtering-Correlation-Id: 6d367f3d-0163-4ec8-6b32-08ddb2e410da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?2W9w5xTgJaqloHejEqnhSUwBdYalnd7Odtt0mrl+mTo5Q8OPAjDuw6HI+7/Q?= =?us-ascii?Q?7JUwqTZsnBqZestbQHRm93D+pvk1kJgre2Rg/BErTNyElGuYNnD/zSLIVDBM?= =?us-ascii?Q?CbanTofunY0e/XWQxluO1h7EoSRrU46yu5G/ogw8Ipqv0TXZ1/O04GyW0sjv?= =?us-ascii?Q?jJYDbmwR6MROn2eMwQen9Bsd5M29qIcW3JIXu7hgYGmnY9AWq+RgEQp6SbE7?= =?us-ascii?Q?yb7eEQs9W/5IMxAWiZcvrP+G0efSCwPhZa4TT4GoGI/U5goAQm4rwyHRbjs8?= =?us-ascii?Q?fxbtGAbmmz9FlfBNr6htYHqTkd8Rm7GFATgys7rW3hdJ/W//28KsrzdMI/8F?= =?us-ascii?Q?CFXmUGrGsbzGqFDl5nQD3uPyQLOiS8UO9a3DzcZbT7MBp/V7HvnfHC+QNkB0?= =?us-ascii?Q?jnyX3WzTjXlK4ZDfLvU83hUC4F3evqcP8Chr9T+gFZpRTbIwkrm8XISt6aW0?= =?us-ascii?Q?KCGYiV7k1nNq9lz/ITLp2O7ZCNP1uaNTB0CFWklaq89dHSVw7hdrfJI8ghH4?= =?us-ascii?Q?5QOZEqxZirVXlirnHZu7RrxQkRJm3c7WBfo2JOsmzt1yRL24/WezmyYHNRVi?= =?us-ascii?Q?a1Jw07LzFH7V8Eeb90tpuz3xqsxFX5K6FofEJBFmQ9I33pppQNs+1FVdaqBe?= =?us-ascii?Q?Uxr6uWjSuNueBLnURezVKppJJ2kifnimHCGNzL3O4hHkJtpRimnfvFdRkEKr?= =?us-ascii?Q?azHpdBiwRptR1XLcOg4ybapK66Wn6M2q0+bgr6wA7Hrfmj/EDTy086a/bYGX?= =?us-ascii?Q?ShCLyKtG1cStk6P99K/pwAModkgZr7flO7+jhdAvMEFHML5o2MWgQUuKpTks?= =?us-ascii?Q?mWTrGiPGGfRclUGbxUHU2VHtCz60jUEBNBM2v2ELIBxmHg5zIEDcxEbUDTbT?= =?us-ascii?Q?iYzEKRMDtPHjpXyjS/ICqsARAJmmtS3rnkbQN89bymktQCcOUszkPrMY6JaB?= =?us-ascii?Q?SbwGmIY8AU9WiwGfBdv018v8wuIJa/HZTDqDh5NE4wnGNlbeuGAXbpPX9qGs?= =?us-ascii?Q?0e1drJCBF1aXO37dL9SIQf6GuCu24AO6wviDw4aYshI1nN8unzVKkle82uGh?= =?us-ascii?Q?y2BYDOis6zqqoYzAG1NlwYp2ijLS1URn9i05k10uboVFhO5YTkj5EGUByzsT?= =?us-ascii?Q?EdZ30TAqcE1abhGANWt4BVeBj1VYifgAgyz30TgX1KwNh3G6bY4xC5wmKTRw?= =?us-ascii?Q?qoAlGJp1Dru+Tt2/mj+EpSaUH8lulav5InC/d3xyEh2bBIG8DmEj4rfwV0Tp?= =?us-ascii?Q?EGR6InlMNq5PFJaluTkYYh9PailWvs9E744SSGrZuUsJqDnzDYOcBcLf336V?= =?us-ascii?Q?yzALiWOQYqxSsUoA6yduEeNvNcqXpKiJrB5rvD68xVaXRhqh2j4hc2IRPnQa?= =?us-ascii?Q?d4Tl+ndFGvjQbH3d64yc+u9oObEuGd85sqHvYiGOG3UUFqQKVNHEC3xwPe+S?= =?us-ascii?Q?W6T1FuSe0bv0UJ551x8rFdM384xLkQOM6fAgz9GFxgu5iailvSIg1gaNpe2a?= =?us-ascii?Q?xP49n/TcgJifOJM5r3EH91l+CAiDvklNzKDu?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:57:56.0309 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6d367f3d-0163-4ec8-6b32-08ddb2e410da X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000206.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4376 Content-Type: text/plain; charset="utf-8" Before this patch, scanning of tasks' mm is done continuously and also at the same rate. Improve that by adding a throttling logic: 1) If there were useful pages found during last scan and current scan, decrease the scan_period (to increase scan rate) by TUNE_PERCENT (15%). 2) If there were no useful pages found in last scan, and there are candidate migration pages in the current scan decrease the scan_period aggressively by 2 power SCAN_CHANGE_SCALE (2^3 =3D 8 now). Vice versa is done for the reverse case. Scan period is clamped between MIN (600ms) and MAX (5sec). Signed-off-by: Raghavendra K T --- mm/kscand.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index f79e05590cde..fca4b7b4a81f 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -20,6 +20,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -33,6 +34,16 @@ static DEFINE_MUTEX(kscand_mutex); #define KSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) static unsigned long kscand_scan_size __read_mostly =3D KSCAND_SCAN_SIZE; =20 +/* + * Scan period for each mm. + * Min: 600ms default: 2sec Max: 5sec + */ +#define KSCAND_SCAN_PERIOD_MAX 5000U +#define KSCAND_SCAN_PERIOD_MIN 600U +#define KSCAND_SCAN_PERIOD 2000U + +static unsigned int kscand_mm_scan_period_ms __read_mostly =3D KSCAND_SCAN= _PERIOD; + /* How long to pause between two scan cycles */ static unsigned int kscand_scan_sleep_ms __read_mostly =3D 20; =20 @@ -78,6 +89,11 @@ static struct kmem_cache *kmigrated_slot_cache __read_mo= stly; /* Per mm information collected to control VMA scanning */ struct kscand_mm_slot { struct mm_slot slot; + /* Unit: ms. Determines how aften mm scan should happen. */ + unsigned int scan_period; + unsigned long next_scan; + /* Tracks how many useful pages obtained for migration in the last scan */ + unsigned long scan_delta; long address; bool is_scanned; }; @@ -713,13 +729,92 @@ static void kmigrated_migrate_folio(void) } } =20 +/* + * This is the normal change percentage when old and new delta remain same. + * i.e., either both positive or both zero. + */ +#define SCAN_PERIOD_TUNE_PERCENT 15 + +/* This is to change the scan_period aggressively when deltas are differen= t */ +#define SCAN_PERIOD_CHANGE_SCALE 3 +/* + * XXX: Hack to prevent unmigrated pages coming again and again while scan= ning. + * Actual fix needs to identify the type of unmigrated pages OR consider m= igration + * failures in next scan. + */ +#define KSCAND_IGNORE_SCAN_THR 256 + +/* Maintains stability of scan_period by decaying last time accessed pages= */ +#define SCAN_DECAY_SHIFT 4 +/* + * X : Number of useful pages in the last scan. + * Y : Number of useful pages found in current scan. + * Tuning scan_period: + * Initial scan_period is 2s. + * case 1: (X =3D 0, Y =3D 0) + * Increase scan_period by SCAN_PERIOD_TUNE_PERCENT. + * case 2: (X =3D 0, Y > 0) + * Decrease scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 3: (X > 0, Y =3D 0 ) + * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 4: (X > 0, Y > 0) + * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + */ +static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, + unsigned long total) +{ + unsigned int scan_period; + unsigned long now; + unsigned long old_scan_delta; + + scan_period =3D mm_slot->scan_period; + old_scan_delta =3D mm_slot->scan_delta; + + /* decay old value */ + total =3D (old_scan_delta >> SCAN_DECAY_SHIFT) + total; + + /* XXX: Hack to get rid of continuously failing/unmigrateable pages */ + if (total < KSCAND_IGNORE_SCAN_THR) + total =3D 0; + + /* + * case 1: old_scan_delta and new delta are similar, (slow) TUNE_PERCENT = used. + * case 2: old_scan_delta and new delta are different. (fast) CHANGE_SCAL= E used. + * TBD: + * 1. Further tune scan_period based on delta between last and current sc= an delta. + * 2. Optimize calculation + */ + if (!old_scan_delta && !total) { + scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && total) { + scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && !total) { + scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; + } else { + scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + } + + scan_period =3D clamp(scan_period, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PE= RIOD_MAX); + + now =3D jiffies; + mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); + mm_slot->scan_period =3D scan_period; + mm_slot->scan_delta =3D total; +} + static unsigned long kscand_scan_mm_slot(void) { bool next_mm =3D false; bool update_mmslot_info =3D false; =20 + unsigned int mm_slot_scan_period; + unsigned long now; + unsigned long mm_slot_next_scan; unsigned long vma_scanned_size =3D 0; unsigned long address; + unsigned long total =3D 0; =20 struct mm_slot *slot; struct mm_struct *mm; @@ -744,6 +839,8 @@ static unsigned long kscand_scan_mm_slot(void) =20 mm =3D slot->mm; mm_slot->is_scanned =3D true; + mm_slot_next_scan =3D mm_slot->next_scan; + mm_slot_scan_period =3D mm_slot->scan_period; spin_unlock(&kscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -754,6 +851,11 @@ static unsigned long kscand_scan_mm_slot(void) goto outerloop; } =20 + now =3D jiffies; + + if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) + goto outerloop; + VMA_ITERATOR(vmi, mm, address); =20 kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, false); @@ -784,8 +886,10 @@ static unsigned long kscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 - if (update_mmslot_info) + if (update_mmslot_info) { mm_slot->address =3D address; + kscand_update_mmslot_info(mm_slot, total); + } =20 outerloop: /* exit_mmap will destroy ptes after this */ @@ -887,6 +991,10 @@ void __kscand_enter(struct mm_struct *mm) return; =20 kscand_slot->address =3D 0; + kscand_slot->scan_period =3D kscand_mm_scan_period_ms; + kscand_slot->next_scan =3D 0; + kscand_slot->scan_delta =3D 0; + slot =3D &kscand_slot->slot; =20 spin_lock(&kscand_mm_lock); --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2085.outbound.protection.outlook.com [40.107.94.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2582379F2 for ; Tue, 24 Jun 2025 05:58:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744701; cv=fail; b=oh+v8fWdbz8c8ceAd9UKURTHroYyyWb35wJ5JZkiIVj7Yg2UO0QrxPZq7GDVfKUJnE1jBjXcpEBrVusirGcOiW9v4yDtBlk5MS53O+NiN1/vCzzncOJrRjpc7sybZJsGDoVaa9kdJkNEnwnyoKurUqyeSTeftbRbVZWj7ngGDcQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744701; c=relaxed/simple; bh=ym+0KEzVCBFSrM9PS86YJgk/PURdOi4Lmxcp+kTvGqw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eVL0kPcS0ItDoosozhz5dIia1PLDolMTZbXG8Rp+bqKi1e+pvnoG1E1ouISDdKYDwOo0Z8qCAV+jVVXkr2PLKA82SyEet5cvMtaj/OFU8Dq9BzjsrDar1ncnLDsZYQCEAm1sTb6GcAQQsUB6vh/j4Gn4cPiMIgNTMUXCnw5zezM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QzN4QK/B; arc=fail smtp.client-ip=40.107.94.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QzN4QK/B" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tVtIEjj2X04uqrrlA9HtJcZypqQYtBqZ8nsy9ejLWfkP8/E2fPSTszFn7LHhYOKivdcHvDozPLStesSyyJF+fXqu6WoSUUq+7mRSR5DGQzfI14OTbd/jumSvQ2KlqgOwHiP3nqtPl8ECkQcTlsf27i4vvJlIhwUtilLrnEXFtEA9SoQ5WRAVzlp2oXZNodi8keRIcMMttvFhAezd1eeo343XIAQqTSKgEh778jeSDPGnL1cCRR1JMgTDXVnKtRanTlVjtfNv66Nm8YQFNVBjoYb7P3q/Qgk5650PBTfPTc05CPAC5O/rEC+pBVX8ikANc4eJkdEI9gjuPFiBb5HBBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=npimcWnWmW3Czx8tRelspzT5xqxVtPyB2JuYUhRJ0Q4=; b=rTuF4KNP75TI4ald/CrfFXA401h5iNMa3g4Zxx0OtPNPoWdv9adqm9omTC23bT0D4pOvutedk+R5hukQP+j6qZ6947FzSMyhUeG6cm73Iv0lFVsV9v+8WjhYG8OO3woiXQNUoXtP6AxDX2fALvZZOK1S919KhdNdIAGIyPLeZEUUKtvpORSNJZ3sVlXOP9DN18AQlYtVrE6Z2kNKEByq4YPsiQiVVAtsI1bZA1vLhDj3z6+BuGW8dRNgRId1vGiztcYKAPQ3EKrEJT70sW/qk3VaXPBtpNI1luA6ZuEKAmsXMtwUlGrKtUMf0ifJYt+9C9O5LTB6OC2C0qzq/Pnhaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=npimcWnWmW3Czx8tRelspzT5xqxVtPyB2JuYUhRJ0Q4=; b=QzN4QK/B9iiuyg4qNVnznUTGN7m/mTW+3xLtrvNNs/4AO8PApIOdGjKRnjzdRFKaYr24hnbSS46SoiGXWQz1+rjokCYEaRPLVp4vH83th8OXFQKAD3joOUACKeTNLUzL0zy49g19lkg9XtGdxtL8cGDpeX6d7AWUxyRD8fttquc= Received: from MW4PR03CA0352.namprd03.prod.outlook.com (2603:10b6:303:dc::27) by DS0PR12MB8293.namprd12.prod.outlook.com (2603:10b6:8:f3::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8835.35; Tue, 24 Jun 2025 05:58:14 +0000 Received: from SJ5PEPF00000209.namprd05.prod.outlook.com (2603:10b6:303:dc:cafe::9d) by MW4PR03CA0352.outlook.office365.com (2603:10b6:303:dc::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8857.29 via Frontend Transport; Tue, 24 Jun 2025 05:58:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000209.mail.protection.outlook.com (10.167.244.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:58:14 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:57:55 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 07/13] mm: Add throttling of mm scanning using scan_size Date: Tue, 24 Jun 2025 05:56:11 +0000 Message-ID: <20250624055617.1291159-8-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000209:EE_|DS0PR12MB8293:EE_ X-MS-Office365-Filtering-Correlation-Id: 228ec93d-0a35-4ac5-e8cf-08ddb2e41ba6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?I1/aSF0walt6ErTnmuSSaplkvZOgq3P5H8qzE35mfE6uwpeeSfI2WIrMeMvn?= =?us-ascii?Q?Lrj5Bicz3cqdYVrGtOOtCYSdRN5KZls1djA/I2vIV161F0ZyRmpD68lqblkh?= =?us-ascii?Q?AwgUyaziUdqTuYy+OxBndeQaOsclBjMtrPltgBMxgecAuuxzLEFbcuEG5hyB?= =?us-ascii?Q?Lzhbd9uCdfUDY1yW8mJGWNgm9jBiQ7IszgJBTlQwNjOZla9WYbH20bYJdFaH?= =?us-ascii?Q?QulF/4+K41I3GrLpduKfjGRkUF5BBKkJ2AWurxTdYz5mIM6zeha6MUlPjvoc?= =?us-ascii?Q?hpZDn0v9BiEVUrlNLjg3JwL3Kct13rQF7lS744WDeXAXE58BjPB9zevJtMTt?= =?us-ascii?Q?735uHVtJebIvSIccnzRHCvwBvGYxtWTYUKEqxz5R2VgpHZa2YC1+t4Fgn1jv?= =?us-ascii?Q?C+3Mo05p7dLf9pLKJue/qvZGOVxqk9yfv70DXm0+C5sIv+CNCCPnGqxTQRDa?= =?us-ascii?Q?N6Mbi4LV3uN/nJJTtSeEgZi1J9YfZBwMksJDnyFWWZ6xkLUPfb/eHhV6HY75?= =?us-ascii?Q?k1LZSwsxNZ/U3hDXSJw8KKWpIBrE88hu3/u/JGm9CrZQwMW5NI5jaOxEmxPr?= =?us-ascii?Q?yfQ9Ii3n6gY461GYYtwt4ROrCSjmaigERaFZOzSUbQ+61Gb95y0HfcurFQ+Q?= =?us-ascii?Q?F7AkehA2RKSvN+X3MxFyMXoapvuHkgD+dyFY/UGC9YzVWa8BziNiAFMyZq5e?= =?us-ascii?Q?WQWC9YY8/z2qrRVuPuTmTlpNP5VKo/UCEwnxjVg159EzhFJXDJKUT5UVfU5s?= =?us-ascii?Q?7EvTm90E4LUvlD2f7RvojCrYQoBwbZamj16r2SJdAU2QQ+LvyKnG81zxhTym?= =?us-ascii?Q?fBeba7BZMKven/he0ZNZCXo0MIbfgTKYXlfZ8d1KmxhIjo7t6ozCRi9A+UMw?= =?us-ascii?Q?RPy0FDktn1EWRcqDtCttLNCf36MOgbmFkJ4iAS5NUUKPBmgmjww/pKnMESAw?= =?us-ascii?Q?5HtvD+ltkykLU5HmfmSnaJez7/e0P56Mo4wNGUmrpLI/UEThSEUXNuH56Vow?= =?us-ascii?Q?771byr4NpK7EKzqh0wFcDAb7ibqUjQpeS3x2qCWJnadigWzaobxB0tD3by57?= =?us-ascii?Q?S/dBS2uwLtj+C2kz6KCZ7FDDrWzkaY3nVLSMnO5QGr7jgTcUfC7OrdX6bgkD?= =?us-ascii?Q?uhsSkYqz9ha9TyOB+poPCYpTvACbPZskGA9jka7uUEyUz3tBnaCnPGnokPCW?= =?us-ascii?Q?aOI+BEw5eUikViy4silOumzPVOeCF3rvd0tqOKJ5LmA+q5EhRn/h3p4E+cqY?= =?us-ascii?Q?VW1HjABUmcXDcO84rMz6Tb0inR5dJD5WUEh9G8cCtFNepaIeEIkD9GmdTznj?= =?us-ascii?Q?7GbHIx98QoA3qw+TbB0L+qJmG3iDnjWye4Aj+to3gMl0ezimC1uFTW1vg5Gb?= =?us-ascii?Q?DGLZputMRkbuBN6Cx1kXvIRdO6i24u+9qAdmLDpvsVMsrBcIVBQ+S47EKkAs?= =?us-ascii?Q?JY6BIE3Azo3FNtxQvhLuZh6e/oN40z4I2xyuRPKCr+CusYpMlpMGyytF3oSd?= =?us-ascii?Q?kGtxwp6xEd9GfYDGybM9wzjggAiQJNotVEFT?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:58:14.1397 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 228ec93d-0a35-4ac5-e8cf-08ddb2e41ba6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000209.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8293 Content-Type: text/plain; charset="utf-8" Before this patch, scanning is done on entire virtual address space of all the tasks. Now the scan size is shrunk or expanded based on the useful pages found in the last scan. This helps to quickly get out of unnecessary scanning thus burning lesser CPU. Drawback: If a useful chunk is at the other end of the VMA space, it will delay scanning and migration. Shrink/expand algorithm for scan_size: X : Number of useful pages in the last scan. Y : Number of useful pages found in current scan. Initial scan_size is 1GB case 1: (X =3D 0, Y =3D 0) Decrease scan_size by 2 case 2: (X =3D 0, Y > 0) Aggressively change to MAX (4GB) case 3: (X > 0, Y =3D 0 ) No change case 4: (X > 0, Y > 0) Increase scan_size by 2 Scan size is clamped between MIN (256MB) and MAX (4GB)). TBD: Tuning based on real workloads Signed-off-by: Raghavendra K T --- mm/kscand.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index fca4b7b4a81f..26b40865d3e5 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -28,10 +28,15 @@ =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); + /* * Total VMA size to cover during scan. + * Min: 256MB default: 1GB max: 4GB */ +#define KSCAND_SCAN_SIZE_MIN (256 * 1024 * 1024UL) +#define KSCAND_SCAN_SIZE_MAX (4 * 1024 * 1024 * 1024UL) #define KSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) + static unsigned long kscand_scan_size __read_mostly =3D KSCAND_SCAN_SIZE; =20 /* @@ -94,6 +99,8 @@ struct kscand_mm_slot { unsigned long next_scan; /* Tracks how many useful pages obtained for migration in the last scan */ unsigned long scan_delta; + /* Determines how much VMA address space to be covered in the scanning */ + unsigned long scan_size; long address; bool is_scanned; }; @@ -744,6 +751,8 @@ static void kmigrated_migrate_folio(void) */ #define KSCAND_IGNORE_SCAN_THR 256 =20 +#define SCAN_SIZE_CHANGE_SHIFT 1 + /* Maintains stability of scan_period by decaying last time accessed pages= */ #define SCAN_DECAY_SHIFT 4 /* @@ -759,14 +768,26 @@ static void kmigrated_migrate_folio(void) * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). * case 4: (X > 0, Y > 0) * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + * Tuning scan_size: + * Initial scan_size is 4GB + * case 1: (X =3D 0, Y =3D 0) + * Decrease scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). + * case 2: (X =3D 0, Y > 0) + * scan_size =3D KSCAND_SCAN_SIZE_MAX + * case 3: (X > 0, Y =3D 0 ) + * No change + * case 4: (X > 0, Y > 0) + * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, unsigned long total) { unsigned int scan_period; unsigned long now; + unsigned long scan_size; unsigned long old_scan_delta; =20 + scan_size =3D mm_slot->scan_size; scan_period =3D mm_slot->scan_period; old_scan_delta =3D mm_slot->scan_delta; =20 @@ -787,20 +808,25 @@ static inline void kscand_update_mmslot_info(struct k= scand_mm_slot *mm_slot, if (!old_scan_delta && !total) { scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size >> SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && total) { scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size << SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && !total) { scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; } else { scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + scan_size =3D KSCAND_SCAN_SIZE_MAX; } =20 scan_period =3D clamp(scan_period, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PE= RIOD_MAX); + scan_size =3D clamp(scan_size, KSCAND_SCAN_SIZE_MIN, KSCAND_SCAN_SIZE_MAX= ); =20 now =3D jiffies; mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); mm_slot->scan_period =3D scan_period; + mm_slot->scan_size =3D scan_size; mm_slot->scan_delta =3D total; } =20 @@ -812,6 +838,7 @@ static unsigned long kscand_scan_mm_slot(void) unsigned int mm_slot_scan_period; unsigned long now; unsigned long mm_slot_next_scan; + unsigned long mm_slot_scan_size; unsigned long vma_scanned_size =3D 0; unsigned long address; unsigned long total =3D 0; @@ -841,6 +868,7 @@ static unsigned long kscand_scan_mm_slot(void) mm_slot->is_scanned =3D true; mm_slot_next_scan =3D mm_slot->next_scan; mm_slot_scan_period =3D mm_slot->scan_period; + mm_slot_scan_size =3D mm_slot->scan_size; spin_unlock(&kscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -992,6 +1020,7 @@ void __kscand_enter(struct mm_struct *mm) =20 kscand_slot->address =3D 0; kscand_slot->scan_period =3D kscand_mm_scan_period_ms; + kscand_slot->scan_size =3D kscand_scan_size; kscand_slot->next_scan =3D 0; kscand_slot->scan_delta =3D 0; =20 --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2086.outbound.protection.outlook.com [40.107.237.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 704ED238C3B for ; Tue, 24 Jun 2025 05:58:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.86 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744704; cv=fail; b=pPq+YTDCiYGkZensj9MaOu6aZV20YlKPF2HqQS7W93AkvJe+xYdij5YNkR0TQUTcTyc3xG0pViHPzyy6CpHwDA3QvzsssEO7w7VaTYKtSMFwdL3+dYhklDCVjRYdh8jU28eCIKPaEM5XQW3wQHPPAXKlSVGTqdfP3x15BPuPc3s= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744704; c=relaxed/simple; bh=terOXdCiQe5u4vzAX61HGKAL6eL4zuPEpMcfbqYp5Ts=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=n/U9Q5f3xfWRoWsO/xX1cRrynl8u9I/m3M1uzGN3eTRfaJYeRNP7tVqy23EShfQBoJKGZei1CzNFN09rMM9R8HqFIGTTStgER5Kv1ZAb7LHz/CK7KuNvmfG50e5MGIGv1zj4HZa3m4TCLJxlWnTB4CTKfpHCw/IINbZub3UOqVw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=WYwIE9+r; arc=fail smtp.client-ip=40.107.237.86 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="WYwIE9+r" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=E/XGZNDZ39D7NvaGQOPVvjkIsCXgnAi4w2ptUVspaoYcf9H5nv3nIfxYTiugMVNy9EwqDqONv4YeIdHagkoVt02B1JlMY4UGxnIuomec8MS3sK74V8yBjk7rlWq2LGoJslm44R6CgjFHGhK2tU8WbFrp8iJkzmpisK2ykLpOya/neKb0sJGY8awDS14ghf7Lp1sQ+Cq5OsVDNJC6s+IxlO3W11qw9qfy9+ZATfjM7AaBGd4VJtPgKPTods/JrBq7YuJyodA4qhvd1rgsdbeFNkvwZ1Tw7PEYnYPUbsx+OMGgS2Lgqlf5mg+TMLy0CavNggrqorAAcmeLjXMF2SWtDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pzvM2ZLdrxrFaLq5Cfs+BIK16r2rHvbnPluKKsKrktE=; b=n+C/OZN8FdibnVhGyAYxHxAxFo1Yu3MixPOlWwmdqBdrsF6ydfyQcYkMGkEinjR/mEvpAarI0IObZbvSkNA+h/vOiN06G4L0gRWeSw95Z3C2DpqdQF8TzCKpNfQcAjHx4rvPYWfeS+cG17uXZemiUp1DnqAngf2Oj+8gEEj+4FJBJnZB6YjhSY+Sy2Ezv7vpL4X9AQtBXy0WLiM9JjW7H8W6EG7PUFVzd/Pdi7t3irSHK313MMIxE7YnAZ3GGaRBSBPEmmHtayz+z1dK1iaeo0AhgxgNQ0UYD1o+EecbbQ/B9duMjArsUmTn2NJ3lsAPYQa4WKkVf+tZB2FGYe2nrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pzvM2ZLdrxrFaLq5Cfs+BIK16r2rHvbnPluKKsKrktE=; b=WYwIE9+ry9cBjQzw6LHWuDERdbZQGfuiiBQIn5/UkbvSElbAnki6ikE/ULX3AWq/hwpmTuomY6HqOnd5g0eZpTXmRY1HMyD57XrnROqpogofitUaoze6+QVAr1Z6oF7BBATyshiae7H19YU2avybCzlOOao+5UawfMmZaNJpXUs= Received: from BYAPR05CA0101.namprd05.prod.outlook.com (2603:10b6:a03:e0::42) by CY8PR12MB7588.namprd12.prod.outlook.com (2603:10b6:930:9b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.28; Tue, 24 Jun 2025 05:58:18 +0000 Received: from SJ5PEPF00000205.namprd05.prod.outlook.com (2603:10b6:a03:e0:cafe::6f) by BYAPR05CA0101.outlook.office365.com (2603:10b6:a03:e0::42) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8880.16 via Frontend Transport; Tue, 24 Jun 2025 05:58:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000205.mail.protection.outlook.com (10.167.244.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:58:17 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:58:06 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 08/13] mm: Add initial scan delay Date: Tue, 24 Jun 2025 05:56:12 +0000 Message-ID: <20250624055617.1291159-9-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000205:EE_|CY8PR12MB7588:EE_ X-MS-Office365-Filtering-Correlation-Id: 4cd7491e-130c-4ac2-9c0b-08ddb2e41dec X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?9TAXq5aAKfxqVjhpToz2UtATVHK12EEEnwPHELSz9RMYNKuTYJHMdezkJXIu?= =?us-ascii?Q?VXXkgEbG2AAiX9+MxmDrSgZAk2+l80Oj3MGa7MApzEVy26CNWYeud4qIjMdM?= =?us-ascii?Q?qPZcQ+Q/wRRCWYgmr9L2FpemJfdsUJSOYpCSbaDRkUMqgj40irXhsx89MwQg?= =?us-ascii?Q?MuovZYHrZpjUT3lUdy/kl+oGCi0VHBb/ioZazGURe7wit6qWtSOAOXoCoN4Z?= =?us-ascii?Q?+GEjDufPLMPU8UhKE0cDzniy/Tnxuixcj7EdmgcR74tMPu5EGgJ+BoVf79Vp?= =?us-ascii?Q?g+rsAqjWl9uRho/7fr5voSQE/2our5D530JUIFAV6NFrDzqCMn/sxoYCwYYZ?= =?us-ascii?Q?jq5BNYJma/6Ckkzxo0lHZA+hw0v5roHouBVX5mGBKEF0DF1zFQMNVHiFRR6s?= =?us-ascii?Q?zDrnyMbJJCOdh52q/QpbXlaH1/NUodO0U/mrpqt9PzOhIJ957KPL3y+vO13O?= =?us-ascii?Q?DV9CD9MPUUm0TTADWNhB0kJONGitRfW2/yyibCO13JpUk4dwO+w4sgzqHCA8?= =?us-ascii?Q?i6/bumoH8MAxKGb8aBm8X+goDjUC57aE25rCUJ+F22fRjSAtd6vR8L8azjwC?= =?us-ascii?Q?LSvWywq6MtdvUmCmKTx8odXP20c9P8ukyvkMXnK3TZjpERKYe1rz8xUxMC7R?= =?us-ascii?Q?LzwYeUF3fAnjCJfiRn524xYbwjUiiwnDQS1dlHhNnrLWUoBz+gt+F5moJAxo?= =?us-ascii?Q?ujDmLagIb8UBO/PDKaK5K9rMiTMGgS0af32bCdyUJSu2HomKYOAtCJpcpfCC?= =?us-ascii?Q?wuBhFW5bFQ4deJcJBS/szfoRSlbQqdU54wQ5+nj+GkD71ZiejL4mR5bK7LLI?= =?us-ascii?Q?ZJgHzJuGAuzTw5wzKd5PKHEgXARct1EzsFzFoyYnH4lnu1JrZRSiEdGth/g9?= =?us-ascii?Q?w26/+q3lGz8ecuAfGTCcek9Tu2sjhIC7wn8GVSPAZEPOpQyIUOI3VTF1giiS?= =?us-ascii?Q?mEUSZN6zzr6+dk3js7tD/bUzCa3zei/z2eyO8f/3QQIkBliaOsO9LwFZ8wEi?= =?us-ascii?Q?RflNwafxiCYSemBZtd7fnuMhAEpw/T9hbsr0bYwBvPGZ13I1aDSwf4JVt7fq?= =?us-ascii?Q?NWK14qIG0Zm6tfGCV1yPAWhVeMWdDgtSQPg1E0cqZnT7b4IVffxBIwiui/Rl?= =?us-ascii?Q?ICKCVU1pacagOotAQaSKROgDV2lzpT9Orxirn+slbnKvc9IBYh5lrIztaUFc?= =?us-ascii?Q?3hE3BP743WmcL/mjTb5H6WPMHRHdt+UEg054TbuzHR/BKQiwmQ/lmfqEb8Zr?= =?us-ascii?Q?r96Sgwamg8CcI2FwzIjmb6LeecpyqGPU7hX6LAAIYpwXaHqE5TMOKQM3SCUX?= =?us-ascii?Q?vIjXvpoFJ5oCaY8xjyYO0YqEusK3b+7jpyurqaw0I3xlACflXhjlkiIE94b1?= =?us-ascii?Q?8gkdVwwEdbKcCh409+9EjTgus1+JIFN7UqFfPZL2reiEykgQz9+U+fl0Cjg/?= =?us-ascii?Q?B2eYbeEV5V6cQ+Q7+OQoGjKp1b+8XCS6vWkcobnuB17xZ4yiBRYCbC6p/BEG?= =?us-ascii?Q?/LX5UyZ7E8E7F5D2GHpJ5p0YfcPT7MSpZQhh?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:58:17.9591 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4cd7491e-130c-4ac2-9c0b-08ddb2e41dec X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000205.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7588 Content-Type: text/plain; charset="utf-8" This is to prevent unnecessary scanning of short lived tasks to reduce CPU burning. Signed-off-by: Raghavendra K T --- mm/kscand.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index 26b40865d3e5..8fbe70faea4e 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -28,6 +28,7 @@ =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); +extern unsigned int sysctl_numa_balancing_scan_delay; =20 /* * Total VMA size to cover during scan. @@ -1008,6 +1009,7 @@ void __kscand_enter(struct mm_struct *mm) { struct kscand_mm_slot *kscand_slot; struct mm_slot *slot; + unsigned long now; int wakeup; =20 /* __kscand_exit() must not run from under us */ @@ -1018,10 +1020,12 @@ void __kscand_enter(struct mm_struct *mm) if (!kscand_slot) return; =20 + now =3D jiffies; kscand_slot->address =3D 0; kscand_slot->scan_period =3D kscand_mm_scan_period_ms; kscand_slot->scan_size =3D kscand_scan_size; - kscand_slot->next_scan =3D 0; + kscand_slot->next_scan =3D now + + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); kscand_slot->scan_delta =3D 0; =20 slot =3D &kscand_slot->slot; --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2040.outbound.protection.outlook.com [40.107.223.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE90723BD00 for ; Tue, 24 Jun 2025 05:58:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.40 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744718; cv=fail; b=uvwD+XATMKhZUn1VlTsEQqE+XaYaA5juQLIWZUl8stijh32ReuBCqJFwRLiuR0JTOdAAkHtwzQcP2WNLJGAv2M+9zZn7e/4C8IQDgtvyEa9JZHDj/NJ/3p4zbP3IF9Yj1L4pqkTQB/wqI8urnD44wsnZHKcxKI8+U0VLr90pFWQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744718; c=relaxed/simple; bh=p39gu98CVTJ6pC7RKtxWLBj+Hcy/UcLQAfJhkAhfU24=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PRQmXywA86FOBS+ipnrWNcOmD0EejPNaIlN4w7z4daBFtCXPVookpEOAZVbfOZc/LAbEO7P8GW1JrkE28uK3FY0O+gQj3KuXA/YGgeOPoVOplrGJyREmRpN2qj/Rmw7XiOu7N3N59ckUlRB+XMxvf6kQCSlAKJPWA1w8br/EWdE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=WI+vaT20; arc=fail smtp.client-ip=40.107.223.40 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="WI+vaT20" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qxGFC/eErxXVhu3XCdvioLclpUhY07pb+mKukeUcVFTRvLk+8gbuoBVWT5xa5VXAOpNd6wa3ce+9QXeMBuuJ2wQ7Pc76XtXG9cLEcYDuqmZwjlHjqX+rMK9hnr+dEw435bYLXafIiwKKByqHRXI4XOZjAQb8F+J1ZQ1K/SIlQCHZpQFBabZUEjU1K8eORqN/ltAScs+/wmnqFqSKPwhiye45QN5ODTQg8l9QLZdaL2gQoeqQFo25FCo2ZbbRZIPt6zMoABhfF+TYPVeP9jE53p866OBBp66tbBfJGnBW44/07zqIZTgJ5GqfIXKVC39OZzhge0RYW0weSOWB9WVVMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6W9V9RNju1GPWwk32Vv9FAlyVQ1UuiOqdNEqvGwMF/A=; b=cm6G/dy90Ho6mZGscf9rLxeUaOHyrViOM/BUnmw0Z5ZgkicvJe93ShQN/VqX148uPrT6jh3wQKRye+ynqMCC4T7cJAUYzoOyMV1Nx7yzJu9cYFvmCthCiV8ZOvH+vw2ae2dEEDX+3hwsGNxLQ4s6sLJPhR217Su7bkje/L6150N6m6tUTkl4KvhqoqppZk0wrDJC0+9DWb9qCaG+PcyWp3nIuzie1T1U1YgrMQIWWTsrArCwIg/37QiQe3PFGzueopHfV16YTVcJXyP8O4LOpYqAOS3tl5m+p5FNPOZewKyXJNTcIXqCy3tSCGWs3RdhVp/Hq+RkyxLXlkF4YSU2Nw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6W9V9RNju1GPWwk32Vv9FAlyVQ1UuiOqdNEqvGwMF/A=; b=WI+vaT20fzAbVyTwvYMBqnUgjzzm+7K3oJarSl3AWG7TAwKsuNgFIYjRrhx+l9zi4VlY5r+uSOp7RmRu62nTT73spzjiTF4vDl1yEaujLaTMILrLDcWoMZ52/gMeVnMenzNJd2Z28x8GOPPCe1lUl3qjrIXsAmyVVkhAU96X4xY= Received: from MW4PR04CA0228.namprd04.prod.outlook.com (2603:10b6:303:87::23) by SA1PR12MB8919.namprd12.prod.outlook.com (2603:10b6:806:38e::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.28; Tue, 24 Jun 2025 05:58:29 +0000 Received: from SJ5PEPF00000208.namprd05.prod.outlook.com (2603:10b6:303:87:cafe::4d) by MW4PR04CA0228.outlook.office365.com (2603:10b6:303:87::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8880.17 via Frontend Transport; Tue, 24 Jun 2025 05:58:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000208.mail.protection.outlook.com (10.167.244.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:58:28 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:58:17 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 09/13] mm: Add a heuristic to calculate target node Date: Tue, 24 Jun 2025 05:56:13 +0000 Message-ID: <20250624055617.1291159-10-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000208:EE_|SA1PR12MB8919:EE_ X-MS-Office365-Filtering-Correlation-Id: 137fca18-1447-45a3-0c7c-08ddb2e42479 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?XVcA8s9OO2RvGjEIv7UnBCCETpigH7plF3l+XeZlLC5hHWXZ5gSP+c0hymtE?= =?us-ascii?Q?s4M/dMCliGgW/3vCmYZDHv2HiZMrsIkk7WiqEQiFo4VxT7a8yZOXuUi7JgPM?= =?us-ascii?Q?RUAC5BH2QT+ugx9zuRbaC/mwIg5Nuro6DxmBL/f8tkrYwhAJ2BBQislRBpAM?= =?us-ascii?Q?T4eD6NHrU9//VsjNKGzLeIx4+Ju5efu6/qNe13ltNRwjH+bZFiFfPFUVnyKW?= =?us-ascii?Q?B/ZWQA2QBFlo1soL/wRqrv6cw9lOZsjM7hvvvQskQigMBuzkW1kBWShb1rm4?= =?us-ascii?Q?DrYMKPqAomgjy2gbz7EwcU8DeRpEbwel9zTj/OZMTj6PEsOyP5BPVB1aof53?= =?us-ascii?Q?9owQD381HNQSGs9Rjzw393gO55KQpcmrlAHNVDZkqhIuC0gp4Q7iI3DZF+gA?= =?us-ascii?Q?BhAYLAZ0RwMyopeVAXZkC9CoAwfpUhKYo7g9l5ETaV4XV/zJIIT5hY8z/NM4?= =?us-ascii?Q?VK56BP+yVqOX+jdH9TFzt5jOe6Fg5Jk2i8nKxSJKmguqXgaUlirPc3140E4b?= =?us-ascii?Q?dXBbFKLkKEtRtkFqxh9aNrpn0pRJPFsIlFY97WQt8q49Tq+cmM6N0DRWgti9?= =?us-ascii?Q?F2uqj5kVdx3qicXLOuZYLoIbSMT/h9O+OSUkitPzHHD8yVDU3cdfXpZkEj9l?= =?us-ascii?Q?TQWEB9PTw2Jo/yB2vGugrTM12x+o9kO2quzZKXTOkvqWkwiAb8N8j7+NvOVl?= =?us-ascii?Q?f5z2z6F4cPJAdtaXmP2NJxwV+Po4mbn3T+HKeHnMGAAFxi0T0vWB1JA+8c0e?= =?us-ascii?Q?0zNS2pEKUlDmzFT9ksKw26lpZD6Qn5m32iT5FC1KRhNHGpyPil+9MqhfLZO5?= =?us-ascii?Q?3kCSqfM7urdmbTKM5X5GXZrqIUNcp4kRpIPP3JMkJLEpOcogWz3YRKVS0Hww?= =?us-ascii?Q?9kwRnPRuLZl0KSYA371Coos6Hih/GxkI8b+px5zNxlqSFCrPPZ1GSe78223D?= =?us-ascii?Q?HA0lh4i/Kk9fPZpekQhiVMxT3QstKZR5iVfNKilNO4UFpPUFvwqNMffAniz7?= =?us-ascii?Q?JlI5TtVzeXsxVA/Ok/hc5Qq1e7AzfcFKGQcFMel3WR78nkISDduWzKpd+gGQ?= =?us-ascii?Q?ER7m0Tx7pUjmKg8a1g31AzG2c619Fqek9iG26v2qsYDh7G8n51a4P1zfHIBP?= =?us-ascii?Q?lIV/P8FDlRgMJM6v5+wIOdLbzaVlwfEzWSIO4pUKC7Kgwse3PT9ACzAWmHE0?= =?us-ascii?Q?PPdAFWmLK1ThpDGb2UYRpjekKjlY/AS+2mVIu7Au/k/TxcbIp6Nn+hH2U7V5?= =?us-ascii?Q?gJz9JA9towzQ5So0IEwYOdW/B7n41RwOyvRmszs5e0SqccF3sspU+j/BeiCA?= =?us-ascii?Q?Pl+D8trYA6QFtbyJkH3Odhh+W6+AjXtz1vGgcClwjfd7hhtSLSnhonfDzI4P?= =?us-ascii?Q?8FhEMu9FSwideXhvzVL/JtMiST//lccwjgvmnHMe6HfUH0xXox5fdofYMwSW?= =?us-ascii?Q?C+c2gqmiwvYqVmZkqU4pnE3ZaOh/ibIYsrzBJntnt2VgnoJZ6ZRVpxEdhfW5?= =?us-ascii?Q?uN/7ChEtLlrOU8a8TLrAsZZfW47uCNxUJmfpSxI/KKV/WmHkKwlxjyXk+jCx?= =?us-ascii?Q?5xZG2vqeeIzHNsFoALgwDrnpI5afDfYb8lNOVBNvBmOVc1CBQh+BrbCGywDN?= =?us-ascii?Q?5O/cOUPVfwcKxnefOpXtTZ4ztJG5Ue1dh5EEY0AqhBFyYnTP+pR7tzyIesKY?= =?us-ascii?Q?cDwQNSdy6LlM2GYwysu4TuJDVoZ7ZLNW/zKU+i+kkbKAEFbjICrro2rDIMLk?= =?us-ascii?Q?yFzz3AhsTALgkeQUV7eWVPhj1aXayilp7IykbgB4QjcpaOi5gYiqfPKLvpD5?= =?us-ascii?Q?KS/IVdWUYyA9Zw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:58:28.9467 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 137fca18-1447-45a3-0c7c-08ddb2e42479 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000208.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB8919 Content-Type: text/plain; charset="utf-8" One of the key challenges in PTE A bit based scanning is to find right target node to promote to. Here is a simple heuristic based approach: 1. While scanning pages of any mm, also scan toptier pages that belong to that mm. 2. Accumulate the insight on the distribution of active pages on toptier nodes. 3. Walk all the top-tier nodes and pick the one with highest accesses. This method tries to consolidate application to a single node. TBD: Create a list of preferred nodes for fallback when highest access node is nearly full. Signed-off-by: Raghavendra K T --- include/linux/mm_types.h | 4 + mm/kscand.c | 186 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 181 insertions(+), 9 deletions(-) TBD: Also maintain nodemask instead of single target node to handle failed migrations?. diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 56d07edd01f9..571be1ad12ab 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1109,6 +1109,10 @@ struct mm_struct { /* numa_scan_seq prevents two threads remapping PTEs. */ int numa_scan_seq; #endif +#ifdef CONFIG_KSCAND + /* Tracks promotion node. XXX: use nodemask */ + int target_node; + #endif /* * An operation with batched TLB flushing is going on. Anything * that can move process memory needs to flush the TLB when diff --git a/mm/kscand.c b/mm/kscand.c index 8fbe70faea4e..2996aaad65d6 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -104,6 +104,7 @@ struct kscand_mm_slot { unsigned long scan_size; long address; bool is_scanned; + int target_node; }; =20 /* Data structure to keep track of current mm under scan */ @@ -116,13 +117,23 @@ struct kscand_scan kscand_scan =3D { .mm_head =3D LIST_HEAD_INIT(kscand_scan.mm_head), }; =20 +/* Per memory node information used to caclulate target_node for migration= */ +struct kscand_nodeinfo { + unsigned long nr_scanned; + unsigned long nr_accessed; + int node; + bool is_toptier; +}; + /* * Data structure passed to control scanning and also collect * per memory node information */ struct kscand_scanctrl { struct list_head scan_list; + struct kscand_nodeinfo *nodeinfo[MAX_NUMNODES]; unsigned long address; + unsigned long nr_to_scan; }; =20 struct kscand_scanctrl kscand_scanctrl; @@ -218,15 +229,121 @@ static void kmigrated_wait_work(void) migrate_sleep_jiffies); } =20 -/* - * Do not know what info to pass in the future to make - * decision on taget node. Keep it void * now. - */ +static unsigned long get_slowtier_accesed(struct kscand_scanctrl *scanctrl) +{ + int node; + unsigned long accessed =3D 0; + + for_each_node_state(node, N_MEMORY) { + if (!node_is_toptier(node) && scanctrl->nodeinfo[node]) + accessed +=3D scanctrl->nodeinfo[node]->nr_accessed; + } + return accessed; +} + +static inline void set_nodeinfo_nr_accessed(struct kscand_nodeinfo *ni, un= signed long val) +{ + ni->nr_accessed =3D val; +} +static inline unsigned long get_nodeinfo_nr_scanned(struct kscand_nodeinfo= *ni) +{ + return ni->nr_scanned; +} + +static inline void set_nodeinfo_nr_scanned(struct kscand_nodeinfo *ni, uns= igned long val) +{ + ni->nr_scanned =3D val; +} + +static inline void reset_nodeinfo_nr_scanned(struct kscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); +} + +static inline void reset_nodeinfo(struct kscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); + set_nodeinfo_nr_accessed(ni, 0); +} + +static void init_one_nodeinfo(struct kscand_nodeinfo *ni, int node) +{ + ni->nr_scanned =3D 0; + ni->nr_accessed =3D 0; + ni->node =3D node; + ni->is_toptier =3D node_is_toptier(node) ? true : false; +} + +static struct kscand_nodeinfo *alloc_one_nodeinfo(int node) +{ + struct kscand_nodeinfo *ni; + + ni =3D kzalloc(sizeof(*ni), GFP_KERNEL); + + if (!ni) + return NULL; + + init_one_nodeinfo(ni, node); + + return ni; +} + +/* TBD: Handle errors */ +static void init_scanctrl(struct kscand_scanctrl *scanctrl) +{ + struct kscand_nodeinfo *ni; + int node; + + for_each_node(node) { + ni =3D alloc_one_nodeinfo(node); + if (!ni) + WARN_ON_ONCE(ni); + scanctrl->nodeinfo[node] =3D ni; + } +} + +static void reset_scanctrl(struct kscand_scanctrl *scanctrl) +{ + int node; + + for_each_node_state(node, N_MEMORY) + reset_nodeinfo(scanctrl->nodeinfo[node]); + + /* XXX: Not rellay required? */ + scanctrl->nr_to_scan =3D kscand_scan_size; +} + +static void free_scanctrl(struct kscand_scanctrl *scanctrl) +{ + int node; + + for_each_node(node) + kfree(scanctrl->nodeinfo[node]); +} + static int kscand_get_target_node(void *data) { return kscand_target_node; } =20 +static int get_target_node(struct kscand_scanctrl *scanctrl) +{ + int node, target_node =3D NUMA_NO_NODE; + unsigned long prev =3D 0; + + for_each_node(node) { + if (node_is_toptier(node) && scanctrl->nodeinfo[node] && + get_nodeinfo_nr_scanned(scanctrl->nodeinfo[node]) > prev) { + prev =3D get_nodeinfo_nr_scanned(scanctrl->nodeinfo[node]); + target_node =3D node; + } + } + if (target_node =3D=3D NUMA_NO_NODE) + target_node =3D kscand_get_target_node(NULL); + + return target_node; +} + extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages); =20 @@ -492,6 +609,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, } srcnid =3D folio_nid(folio); =20 + scanctrl->nodeinfo[srcnid]->nr_scanned++; + if (scanctrl->nr_to_scan) + scanctrl->nr_to_scan--; + + if (!scanctrl->nr_to_scan) { + folio_put(folio); + return 1; + } =20 if (!folio_test_lru(folio)) { folio_put(folio); @@ -502,6 +627,8 @@ static int hot_vma_idle_pte_entry(pte_t *pte, mmu_notifier_test_young(mm, addr) || folio_test_referenced(folio) || pte_young(pteval)) { =20 + scanctrl->nodeinfo[srcnid]->nr_accessed++; + if (!kscand_eligible_srcnid(srcnid)) { folio_put(folio); return 0; @@ -695,7 +822,13 @@ static void kmigrated_migrate_mm(struct kmigrated_mm_s= lot *mm_slot) =20 spin_unlock(&mm_slot->migrate_lock); =20 - dest =3D kscand_get_target_node(NULL); + if (!mmap_read_trylock(mm)) { + dest =3D kscand_get_target_node(NULL); + } else { + dest =3D READ_ONCE(mm->target_node); + mmap_read_unlock(mm); + } + ret =3D kmigrated_promote_folio(info, mm, dest); =20 kfree(info); @@ -781,7 +914,7 @@ static void kmigrated_migrate_folio(void) * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kscand_update_mmslot_info(struct kscand_mm_slot *mm_slo= t, - unsigned long total) + unsigned long total, int target_node) { unsigned int scan_period; unsigned long now; @@ -829,6 +962,7 @@ static inline void kscand_update_mmslot_info(struct ksc= and_mm_slot *mm_slot, mm_slot->scan_period =3D scan_period; mm_slot->scan_size =3D scan_size; mm_slot->scan_delta =3D total; + mm_slot->target_node =3D target_node; } =20 static unsigned long kscand_scan_mm_slot(void) @@ -837,6 +971,7 @@ static unsigned long kscand_scan_mm_slot(void) bool update_mmslot_info =3D false; =20 unsigned int mm_slot_scan_period; + int target_node, mm_slot_target_node, mm_target_node; unsigned long now; unsigned long mm_slot_next_scan; unsigned long mm_slot_scan_size; @@ -870,6 +1005,7 @@ static unsigned long kscand_scan_mm_slot(void) mm_slot_next_scan =3D mm_slot->next_scan; mm_slot_scan_period =3D mm_slot->scan_period; mm_slot_scan_size =3D mm_slot->scan_size; + mm_slot_target_node =3D mm_slot->target_node; spin_unlock(&kscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -880,6 +1016,9 @@ static unsigned long kscand_scan_mm_slot(void) goto outerloop; } =20 + mm_target_node =3D READ_ONCE(mm->target_node); + if (mm_target_node !=3D mm_slot_target_node) + WRITE_ONCE(mm->target_node, mm_slot_target_node); now =3D jiffies; =20 if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -887,24 +1026,41 @@ static unsigned long kscand_scan_mm_slot(void) =20 VMA_ITERATOR(vmi, mm, address); =20 + /* Either Scan 25% of scan_size or cover vma size of scan_size */ + kscand_scanctrl.nr_to_scan =3D mm_slot_scan_size >> PAGE_SHIFT; + /* Reduce actual amount of pages scanned */ + kscand_scanctrl.nr_to_scan =3D mm_slot_scan_size >> 1; + + /* XXX: skip scanning to avoid duplicates until all migrations done? */ kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, false); =20 for_each_vma(vmi, vma) { kscand_walk_page_vma(vma, &kscand_scanctrl); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 - if (vma_scanned_size >=3D kscand_scan_size) { + if (vma_scanned_size >=3D mm_slot_scan_size || + !kscand_scanctrl.nr_to_scan) { next_mm =3D true; =20 if (!list_empty(&kscand_scanctrl.scan_list)) { if (!kmigrated_mm_slot) kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, true); + /* Add scanned folios to migration list */ spin_lock(&kmigrated_mm_slot->migrate_lock); + list_splice_tail_init(&kscand_scanctrl.scan_list, &kmigrated_mm_slot->migrate_head); spin_unlock(&kmigrated_mm_slot->migrate_lock); + break; } - break; + } + if (!list_empty(&kscand_scanctrl.scan_list)) { + if (!kmigrated_mm_slot) + kmigrated_mm_slot =3D kmigrated_get_mm_slot(mm, true); + spin_lock(&kmigrated_mm_slot->migrate_lock); + list_splice_tail_init(&kscand_scanctrl.scan_list, + &kmigrated_mm_slot->migrate_head); + spin_unlock(&kmigrated_mm_slot->migrate_lock); } } =20 @@ -915,9 +1071,19 @@ static unsigned long kscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + total =3D get_slowtier_accesed(&kscand_scanctrl); + target_node =3D get_target_node(&kscand_scanctrl); + + mm_target_node =3D READ_ONCE(mm->target_node); + + /* XXX: Do we need write lock? */ + if (mm_target_node !=3D target_node) + WRITE_ONCE(mm->target_node, target_node); + reset_scanctrl(&kscand_scanctrl); + if (update_mmslot_info) { mm_slot->address =3D address; - kscand_update_mmslot_info(mm_slot, total); + kscand_update_mmslot_info(mm_slot, total, target_node); } =20 outerloop: @@ -1111,6 +1277,7 @@ static int stop_kscand(void) kthread_stop(kscand_thread); kscand_thread =3D NULL; } + free_scanctrl(&kscand_scanctrl); =20 return 0; } @@ -1166,6 +1333,7 @@ static inline void init_list(void) spin_lock_init(&kscand_migrate_lock); init_waitqueue_head(&kscand_wait); init_waitqueue_head(&kmigrated_wait); + init_scanctrl(&kscand_scanctrl); } =20 static int __init kscand_init(void) --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2053.outbound.protection.outlook.com [40.107.243.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5896923D2B6 for ; Tue, 24 Jun 2025 05:58:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.53 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744727; cv=fail; b=uwNr/uvpLNYCASiqZVXzdHBj1hfKLzgmkFgYOMrT/670JV9aEqETBmMbxcCHqcD94/sU+UKC2+tiskD1Dcx6JISkOyM2PXryH9n3fLGn6PHPyoCVebLeBgGDlaNKdU2+rav1QK66kgLX5KH6r5lyxjiIVFe8UH2OLq7lRi6IJOs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744727; c=relaxed/simple; bh=Q6AajCqillq0jIoDGnugMLikNrREY03STpAzfW+inwg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Y7K2zw2cu/lir3irQTCurGlKKiyNT6Ka4qOmuWoHD4wLUUUHeHIs1brXU2qu8gUiyma0ortbpUBAm6bdsPbknAPbmFF0KW52EW816lI+eirBeHdFIuDCDZ6eHe0ALK4aDYdtindIXKLuEJiYdNtiG7idhmeFoHjapbs/vYNwRpI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=favbM7BI; arc=fail smtp.client-ip=40.107.243.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="favbM7BI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lQsMJFgfLR4/+ir36Vc8tL/5nIhEB5UphOUjEqkiM3OLYrW3ZNUnpKjdp3PZcrCr4Z0PTTVy45zW/ZTynGLmomlQnidKa2HtSphyyzfh5pivgfEKI2yjassSx9eK5DnbY3rKQY2BxOyDmNfbmoq3NfrIE8CRa21sEOy4ZBCcdw6u3Nyn3Jpxqwt0SFR0CJGvstr0NoPGML2OtBm1eYrVml3ns7xdHXPa54OlCo1htZuA1rbY/KVR+mnHIkwzjsBpc7ctBfSUxkQEc6K2uisCPlvAYxDCAqzfBUKSApECkcsl4e32Bu5m0ePJCQ8JgEJOV1lpVdOckJv1dcfe7ffmzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jNZuoxVfvqVikHLgk9R6ICz6ag6HwvS891F4xfGb3wg=; b=QTEeJfNzRPpAG0xqa6fSms7/YkLHd6cROvo/D8+XeITxyWEEG/bSDyp3byvnwI41Cf/427DAMR7sHvT1fgACtiXKhr2cd+4VpHKXXFW9RIhImAEkmT319YLJeVKNVIinyk15vi3iL4zRNUd3Ksz4dAcnMplaNxP313xM2nKGDPcUJ2ilAGZ714KVF5RdyKjjXoLZtHaOWO/S2n6ABWknFYmjY/JVtY8hzWoPpFxwKfR+GVxZSQXImwuPkSofWLy+vmAVk0p9pEjPEmTdJjvXOXBtcKctkVAA6fZMHZnnndyhcnUpCsGZhEzGSQdTcOP++7GWaPoDsQ/KOTbQjTO7oA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jNZuoxVfvqVikHLgk9R6ICz6ag6HwvS891F4xfGb3wg=; b=favbM7BIDT78EHJ8C6c3btex3w6nC+66JxBS/xD2LJJuIlbK9MNhbucHgBRZV+zA0+xjNsRRUO1ED1eY6kjvkIH4hZgIBkgp9wO/GvvwjYzFrhv0ivKDuICz2qIoOzVUoxI44xQgahinPPmyt8L7hhBWAfLTWgQDYBxKUyJk6U8= Received: from MW4PR03CA0347.namprd03.prod.outlook.com (2603:10b6:303:dc::22) by PH7PR12MB6561.namprd12.prod.outlook.com (2603:10b6:510:213::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.31; Tue, 24 Jun 2025 05:58:40 +0000 Received: from SJ5PEPF00000209.namprd05.prod.outlook.com (2603:10b6:303:dc:cafe::98) by MW4PR03CA0347.outlook.office365.com (2603:10b6:303:dc::22) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8857.30 via Frontend Transport; Tue, 24 Jun 2025 05:58:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000209.mail.protection.outlook.com (10.167.244.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:58:39 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:58:28 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 10/13] sysfs: Add sysfs support to tune scanning Date: Tue, 24 Jun 2025 05:56:14 +0000 Message-ID: <20250624055617.1291159-11-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000209:EE_|PH7PR12MB6561:EE_ X-MS-Office365-Filtering-Correlation-Id: abac8816-6768-463e-281b-08ddb2e42b05 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|36860700013|1800799024|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?+7ROxtvCMLSpdqqt6YA2D+WFWNj+f925KGHuMB6io0M+wZpnY5cQDVnDauX8?= =?us-ascii?Q?VI2BSOPVav77tPVhD5IeXIDyjbdYo3rxEi0J1NJi3FKTsv1XtMx/4yNw+oNp?= =?us-ascii?Q?XFcsV+o88uBEtk8uyMYFh/xZi7u/Df1hC9cEdD/DMKtCVOeZBPA3WWSycEar?= =?us-ascii?Q?81nO5KqFmCtKa6QG2Mqwq9/WNFZ+92yo/WGS992/Rs6XGgjQrWYrcSAucQMV?= =?us-ascii?Q?pn7cp96R8ns0gmZdcxvLHWG9RBJpaBO5UVHBv4jSg+9Ge8TU9Igwp5jAh6cA?= =?us-ascii?Q?JDagkwsoQ/HYpwJPSV96DZUCG6+h6Pj3Ky8c3V4ea2dVTmKaq3lpRpChZlRT?= =?us-ascii?Q?VeucUqId/XMfh8Tm2N1vQpgiPWNeOWNPUXvwovW/zJcxESqoETnRlK+ziflR?= =?us-ascii?Q?aKUrmPLuqyMotQsB2PdK/yUTwSSn4l17di/9Ls1T8CBNbVikXSkCvm7RdWBD?= =?us-ascii?Q?07bOvMSMo7AkPX64QOPLhVBPbl6vvONqfxd/EqDjt3XN57hYbU3x+OfqsfYw?= =?us-ascii?Q?4DK5EvVmlmbtpyYi5vmc6aKwq9C33FlRbAW/CmUxwoMHjLBchxQEdSKCd5Qg?= =?us-ascii?Q?usmL1O3lbsck0wGyF6WWZtWje+oIaS2Dk3clmPjmSNwdTkk7OQ9fXGRFKNHt?= =?us-ascii?Q?Ub38AqXqb51SBgujOiiSPKlnQxSxIVEr/nTYcCMP8gbjJgYYFhp+4ZWJP3k1?= =?us-ascii?Q?jSXUv/0U9vIBAcO7BCjQC23IFZw+iTglynrgY7U5xepK9qRYiabmUDXljSJH?= =?us-ascii?Q?1X2/dGt8oQ3n5ZzDVr2waeMxicEYAkSukiFRHd31hRqmr90vOBJc6oD6Dzz1?= =?us-ascii?Q?lhH9fRloDqy9oopPmDWKcvBoN3D7ATSN47cA7XQ2RzG5AmRW6ZLcJIzlT92o?= =?us-ascii?Q?gl6AVLfoVdYvwoj0xNF8Y4sovaID4lRBEdxRM5aRZdYXWTUadKzbbD337j2e?= =?us-ascii?Q?PabbJYXtiqe01b4WNiUBziQRbv7WJ63pevPoKlVS+8grKY6QOsihJASSS9PC?= =?us-ascii?Q?+Tq8gu3jFr3Ycli72VhESwT3V2MtqLNYXX55cQ/H6mdXQVDV7XdTPa+Oz5km?= =?us-ascii?Q?unE3W+GUAAEQB9ipr1yT941YJmiufndLj1OIiWebyekvag7mijUnyiTxH3Fi?= =?us-ascii?Q?xt/1ygE9u59Txjeup3MimFHo4PrkKMSZIW+Jpbu1sWuQHb1DVb0F/9OclSb4?= =?us-ascii?Q?9We3etwnSC7EljyBvlfDXdi8xA6U21/URRzBjkCVFjOTxnDKq7LQW4fmXwKk?= =?us-ascii?Q?8sqSvSnEtt1P4++dUL0Eo3S9jZLIGPBtv+BrHZzD5t02HtJqlbeed37ZToWQ?= =?us-ascii?Q?WFEF4yc79OrF/gNv1cazp/6VPLw3RzxanHCAyvIvWCHlUlb8Idms4xTL+i+3?= =?us-ascii?Q?Fm1tdtlVaOnvEAHuuFP1c/sR/3HyS/UM3bM708/1tc4PsSBMnGBsRVQw7Lm5?= =?us-ascii?Q?q8qUZ+DsjC/dblbZyWsYoHoKI8r76whxtmaDY50WEhIx7I0KoveahT0NZ9Q5?= =?us-ascii?Q?EdzdhPpTBWIc6aneVwzYdTjhlnLsEzle34TO?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(376014)(36860700013)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:58:39.9274 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: abac8816-6768-463e-281b-08ddb2e42b05 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000209.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6561 Content-Type: text/plain; charset="utf-8" Support below tunables: scan_enable: turn on or turn off mm_struct scanning scan_period: initial scan_period (default: 2sec) scan_sleep_ms: sleep time between two successive round of scanning and migration. mms_to_scan: total mm_struct to scan before taking a pause. target_node: default regular node to which migration of accessed pages is done (this is only fall back mechnism, otherwise target_node heuristic is used). Signed-off-by: Raghavendra K T --- mm/kscand.c | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 205 insertions(+) diff --git a/mm/kscand.c b/mm/kscand.c index 2996aaad65d6..abffcb868447 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -21,6 +21,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -171,6 +172,171 @@ static bool kscand_eligible_srcnid(int nid) return !node_is_toptier(nid); } =20 +#ifdef CONFIG_SYSFS +static ssize_t scan_sleep_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_scan_sleep_ms); +} + +static ssize_t scan_sleep_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + kscand_scan_sleep_ms =3D msecs; + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute scan_sleep_ms_attr =3D + __ATTR_RW(scan_sleep_ms); + +static ssize_t mm_scan_period_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_mm_scan_period_ms); +} + +/* If a value less than MIN or greater than MAX asked for store value is c= lamped */ +static ssize_t mm_scan_period_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs, stored_msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + stored_msecs =3D clamp(msecs, KSCAND_SCAN_PERIOD_MIN, KSCAND_SCAN_PERIOD_= MAX); + + kscand_mm_scan_period_ms =3D stored_msecs; + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute mm_scan_period_ms_attr =3D + __ATTR_RW(mm_scan_period_ms); + +static ssize_t mms_to_scan_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%lu\n", kscand_mms_to_scan); +} + +static ssize_t mms_to_scan_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned long val; + int err; + + err =3D kstrtoul(buf, 10, &val); + if (err) + return -EINVAL; + + kscand_mms_to_scan =3D val; + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute mms_to_scan_attr =3D + __ATTR_RW(mms_to_scan); + +static ssize_t scan_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_scan_enabled ? 1 : 0); +} + +static ssize_t scan_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int val; + int err; + + err =3D kstrtouint(buf, 10, &val); + if (err || val > 1) + return -EINVAL; + + if (val) { + kscand_scan_enabled =3D true; + need_wakeup =3D true; + } else + kscand_scan_enabled =3D false; + + kscand_sleep_expire =3D 0; + wake_up_interruptible(&kscand_wait); + + return count; +} + +static struct kobj_attribute scan_enabled_attr =3D + __ATTR_RW(scan_enabled); + +static ssize_t target_node_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kscand_target_node); +} + +static ssize_t target_node_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int err, node; + + err =3D kstrtoint(buf, 10, &node); + if (err) + return -EINVAL; + + kscand_sleep_expire =3D 0; + if (!node_is_toptier(node)) + return -EINVAL; + + kscand_target_node =3D node; + wake_up_interruptible(&kscand_wait); + + return count; +} +static struct kobj_attribute target_node_attr =3D + __ATTR_RW(target_node); + +static struct attribute *kscand_attr[] =3D { + &scan_sleep_ms_attr.attr, + &mm_scan_period_ms_attr.attr, + &mms_to_scan_attr.attr, + &scan_enabled_attr.attr, + &target_node_attr.attr, + NULL, +}; + +struct attribute_group kscand_attr_group =3D { + .attrs =3D kscand_attr, + .name =3D "kscand", +}; +#endif + static inline int kscand_has_work(void) { return !list_empty(&kscand_scan.mm_head); @@ -1164,11 +1330,45 @@ static int kscand(void *none) return 0; } =20 +#ifdef CONFIG_SYSFS +extern struct kobject *mm_kobj; +static int __init kscand_init_sysfs(struct kobject **kobj) +{ + int err; + + err =3D sysfs_create_group(*kobj, &kscand_attr_group); + if (err) { + pr_err("failed to register kscand group\n"); + goto err_kscand_attr; + } + + return 0; + +err_kscand_attr: + sysfs_remove_group(*kobj, &kscand_attr_group); + return err; +} + +static void __init kscand_exit_sysfs(struct kobject *kobj) +{ + sysfs_remove_group(kobj, &kscand_attr_group); +} +#else +static inline int __init kscand_init_sysfs(struct kobject **kobj) +{ + return 0; +} +static inline void __init kscand_exit_sysfs(struct kobject *kobj) +{ +} +#endif + static inline void kscand_destroy(void) { kmem_cache_destroy(kscand_slot_cache); /* XXX: move below to kmigrated thread */ kmem_cache_destroy(kmigrated_slot_cache); + kscand_exit_sysfs(mm_kobj); } =20 void __kscand_enter(struct mm_struct *mm) @@ -1354,6 +1554,10 @@ static int __init kscand_init(void) return -ENOMEM; } =20 + err =3D kscand_init_sysfs(&mm_kobj); + if (err) + goto err_init_sysfs; + init_list(); err =3D start_kscand(); if (err) @@ -1370,6 +1574,7 @@ static int __init kscand_init(void) =20 err_kscand: stop_kscand(); +err_init_sysfs: kscand_destroy(); =20 return err; --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2061.outbound.protection.outlook.com [40.107.220.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AD5623909F for ; Tue, 24 Jun 2025 05:58:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.61 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744736; cv=fail; b=LW9KLx1EK9HUIYqPXycO23s3TwPfhuKt8zsQP0pVw4/WKYuSiqn0lPw6y4EkXuYD2gbsSy0Gh2Cp0EC//Sna9JCtRcdkjmtnVLr5MaduotZ/fvmOYUrYWB2UmAROKYXjUFzMCD9ERzLWsVtqc6eAWOC37ZHU3UOB9dFWQIZqtA4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744736; c=relaxed/simple; bh=mi5Ld5J0qeg55sZL7uIeJlWfD3SUESBso3PQvYyKL/I=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SnxmlTCBhYJg/93fonArK4SegX+TxATcWwrgBfGCDkrOKGwN4vtjhX8EziHMimyrgLrf+AGvmUFEw2jxTkimJE/UvXAXvZzZJ4SJbKAmF+8W4XxDhXTnu85qkGtRJfOoyHOa+lvBFBv+QyfXnxGvP0nMSXixCdIn1DhuAU39dOY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=3qEME/OO; arc=fail smtp.client-ip=40.107.220.61 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="3qEME/OO" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cXU6DrKqPinu+WjXflLJPeCIij5yrVcqXNmVtbDwHNn9khFFUtH03uRbQS0flr3n843L+8O+yWMZiI8fhyibMbeXXH/0nGWphybzWHnbXhVORfTb0UzDuwNRpgBVc5VbPH8YyHe9Rg5iH9OvPVmBcv/kS3LLiQh1pN/Gq44v+ry3F0cLVbBIerr+nRbYcuwQyhqbnAMurYpL00IIk2kaqpMLRLjCjk2HnQaeyENSrggsw529eVZubgBuf0MZrvZrlE6OBNz0qbdm4pEmf3HKvgntE9MhsxE3yBgk0Ou9radfgdnmzne2F2+/hEdDEyPjMcF86x8FrQwv5ucnCsGUEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jPBB6uzuV5RsHsgW28KKO9VKzZHyzSSqRmv8hz2TbHA=; b=rSbn27AeLH+LZ+TLrKlh3CTckZnZB6j9N+rwvtIW0xfpb7UhohIPsHmMdaL6Xsp0gMgkseuXSCuZGxc9R/haQOfJs92CwU07TLW0WEBD3v7pJndtCMxI9O+9yupTrGb1+282/07Siw09GGncZbnph7lco6tLhve6HpNsGfKGxXBYFJb7jKM0JQFUdQQrR0za1xMaSLFPYbbEi6/TR/2KlW8EH++HJ3MJuZL/cuhtmCRgQx0jaLNcciaKk3i78SIkIUwILR8iHzVziq75gx1rSSwcbNAhgUp6Jo06t7Yfk++crjIm/o2eM8I3lyMTxHJGupDwu+T7GyS9mkVpvxjhiw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jPBB6uzuV5RsHsgW28KKO9VKzZHyzSSqRmv8hz2TbHA=; b=3qEME/OOcJp74VZ5R4EdCiw9MZkLajZD/I2e82vypNqknfbncMjp5w4aX0ym3BdEvOo8vdMjZ1dG50JYDOp8bp2AQWImy7QySAKkQuSvAHa+BQ7L+qg5aPRTs05U3oI/ZEfHh0xUgO4jXoPXZZIyYVlHI/PWo0xuvd0J+EqCk+k= Received: from BYAPR05CA0101.namprd05.prod.outlook.com (2603:10b6:a03:e0::42) by CYYPR12MB9016.namprd12.prod.outlook.com (2603:10b6:930:c4::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8835.29; Tue, 24 Jun 2025 05:58:51 +0000 Received: from SJ5PEPF00000205.namprd05.prod.outlook.com (2603:10b6:a03:e0:cafe::cf) by BYAPR05CA0101.outlook.office365.com (2603:10b6:a03:e0::42) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8880.16 via Frontend Transport; Tue, 24 Jun 2025 05:58:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000205.mail.protection.outlook.com (10.167.244.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:58:50 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:58:39 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 11/13] vmstat: Add vmstat counters Date: Tue, 24 Jun 2025 05:56:15 +0000 Message-ID: <20250624055617.1291159-12-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000205:EE_|CYYPR12MB9016:EE_ X-MS-Office365-Filtering-Correlation-Id: f3c8cd30-f8a2-4a43-9ea0-08ddb2e4319c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?LPhrc0L55uraJKyXWK+Gu08xJWpDSUbya7vjUcpASaB45PbMMwV7d7Iamp7P?= =?us-ascii?Q?TgiJ3c/bvAeEtZoJsltDrRC5CAYdvrYNf65/l8WiqPFm1ZMN5PibfENdMnln?= =?us-ascii?Q?eZeyHALtO9tOSJKg9wfekSYauBAYKiAaLFxLhbAQCrdotNm7UWaP4TWeuEv1?= =?us-ascii?Q?WBy7fmBTKg8xLPFQE7fNhLJDzRwRb2V5KFga0Bq8BB29NK0jQqlnlyB4y6lD?= =?us-ascii?Q?85DokvSRi6SBtQyf4KHXyXPJvYO3tgB/JCXSpUxL0cEMdckVzJfZJyqJMZ7R?= =?us-ascii?Q?LTkMkrDQKMWkrdr0WAwM2qb4UVwEDkGYKM6CGJH0lvEyGzSyj976vLq1SREQ?= =?us-ascii?Q?rM3vEXm2y/D+sNhEGzuSRMoJnD5fCyY7JcxsITyaldJHSCY9AfMKhLmCJ3bZ?= =?us-ascii?Q?oJ140QhytjTFQ068mP2NLH/7tNmQDANEn1AKif8+NHUtVZsmfAoQNWbbZR6h?= =?us-ascii?Q?ovVjSyd7ekdCGj0sMZqrSn7kZ/AwceAuaZzuantsCNCifVJUWzvuEY6bF1xU?= =?us-ascii?Q?pW51IDowJ3H58yrnktOrEUGk1mNgvUCGdUfEBi2C2lx61wrnT93U9KyJ5Qfk?= =?us-ascii?Q?nwM5dDdwT2Vy+zoCSEK/R+nWdzubUh12U8IlHFZcLIBZHkckRwTwQrHPntg4?= =?us-ascii?Q?Rqcnkr9SVNAgXQvaBAqlORx8C1a7ehpqe7m5DbGEBpnzZQJUSIvM2k1DvO14?= =?us-ascii?Q?JNmAYJM72/kB5tXOKiKV2NSOS2fBmAYomsbiJtme1YcP1+YT9W4Ay3UX20z2?= =?us-ascii?Q?kndH+M/spNY9LqAgDxORo+mOT3XBEJsZth8PRJ+LyFVI/VHKtTWtbVrU3SN2?= =?us-ascii?Q?jDQ09NixPvw6IFW2/FaQYTOHmbcrgK74JJnrbs9Y3dHRyrqjYxrUm6Up3K0W?= =?us-ascii?Q?Ie0DaSXaw1vZpPLOgwQmk1ozYeilVxRxAP1sR7vWGwSZShTZPSt/TF2gd12w?= =?us-ascii?Q?PQocfLXfaLEpQwhjW+Jt9EcbFOuc7Djisqk2zOy6nDTd82leqxgkPfCtgC5A?= =?us-ascii?Q?THA5f+EKCy1RoICZs8kCndwukSvKmOgkt+IFqnT2votF7yn4GjMDVJYl7NGY?= =?us-ascii?Q?vU3gBixswKtmufpKRXVbSWU0ATPdsfSHNR921I2bO9Pt0uuJFEP+rvnAk9qZ?= =?us-ascii?Q?PyFyRIz6cytrpxtOJ+wm5lkxXvCGvbId0a+gctT19nrlz8u0XlJismT0e4c6?= =?us-ascii?Q?aWQStmV1dNr994P1dA7l+EpaqPEyCpZykT6UrwBLCA66qsZnTYYHwY2pvPcQ?= =?us-ascii?Q?FFE3Sv0kOXV1xVBSH784g7wQ0C41Zcxl7mfWtFa9N5SAWNzxsT53Z1P8igjr?= =?us-ascii?Q?KI0ZcUoVVnTGgWPLdvWtSNCTvZeZA0Ie51ppmJ3i1u+aC0mQDwdUQ03/8M+5?= =?us-ascii?Q?ux33OHMcBs6R0PiGsErA08zRcEzz91s4rHYqwFvdKo6RojHxLGzAaBDzvd2K?= =?us-ascii?Q?JwWXKK5xUMB6dMdRC5eRfZHoRZ0QgM2S7u5SanVPmfr8n8at3X2Y2qOSkfYN?= =?us-ascii?Q?d/Q+p69d/hgGUC0LXL50y2Ce3lMhLtW4SqhZ?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:58:50.9907 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f3c8cd30-f8a2-4a43-9ea0-08ddb2e4319c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000205.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB9016 Content-Type: text/plain; charset="utf-8" Add vmstat counter to track scanning, migration and type of pages. Signed-off-by: Raghavendra K T --- include/linux/mm.h | 11 ++++++++ include/linux/vm_event_item.h | 10 +++++++ mm/kscand.c | 51 ++++++++++++++++++++++++++++++++++- mm/vmstat.c | 10 +++++++ 4 files changed, 81 insertions(+), 1 deletion(-) This implementation will change with upcoming changes in vmstat. diff --git a/include/linux/mm.h b/include/linux/mm.h index fdda6b16263b..b67d06cbc2ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -656,6 +656,17 @@ struct vm_operations_struct { unsigned long addr); }; =20 +#ifdef CONFIG_KSCAND +void count_kscand_mm_scans(void); +void count_kscand_vma_scans(void); +void count_kscand_migadded(void); +void count_kscand_migrated(void); +void count_kscand_migrate_failed(void); +void count_kscand_slowtier(void); +void count_kscand_toptier(void); +void count_kscand_idlepage(void); +#endif + #ifdef CONFIG_NUMA_BALANCING static inline void vma_numab_state_init(struct vm_area_struct *vma) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 9e15a088ba38..8f324ad73821 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -67,6 +67,16 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, #endif +#ifdef CONFIG_KSCAND + KSCAND_MM_SCANS, + KSCAND_VMA_SCANS, + KSCAND_MIGADDED, + KSCAND_MIGRATED, + KSCAND_MIGRATE_FAILED, + KSCAND_SLOWTIER, + KSCAND_TOPTIER, + KSCAND_IDLEPAGE, +#endif #ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, THP_MIGRATION_SUCCESS, diff --git a/mm/kscand.c b/mm/kscand.c index abffcb868447..db7b2f940f36 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -337,6 +337,39 @@ struct attribute_group kscand_attr_group =3D { }; #endif =20 +void count_kscand_mm_scans(void) +{ + count_vm_numa_event(KSCAND_MM_SCANS); +} +void count_kscand_vma_scans(void) +{ + count_vm_numa_event(KSCAND_VMA_SCANS); +} +void count_kscand_migadded(void) +{ + count_vm_numa_event(KSCAND_MIGADDED); +} +void count_kscand_migrated(void) +{ + count_vm_numa_event(KSCAND_MIGRATED); +} +void count_kscand_migrate_failed(void) +{ + count_vm_numa_event(KSCAND_MIGRATE_FAILED); +} +void count_kscand_slowtier(void) +{ + count_vm_numa_event(KSCAND_SLOWTIER); +} +void count_kscand_toptier(void) +{ + count_vm_numa_event(KSCAND_TOPTIER); +} +void count_kscand_idlepage(void) +{ + count_vm_numa_event(KSCAND_IDLEPAGE); +} + static inline int kscand_has_work(void) { return !list_empty(&kscand_scan.mm_head); @@ -789,6 +822,9 @@ static int hot_vma_idle_pte_entry(pte_t *pte, return 0; } =20 + if (node_is_toptier(srcnid)) + count_kscand_toptier(); + if (!folio_test_idle(folio) || folio_test_young(folio) || mmu_notifier_test_young(mm, addr) || folio_test_referenced(folio) || pte_young(pteval)) { @@ -802,11 +838,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, =20 info =3D kzalloc(sizeof(struct kscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { + count_kscand_slowtier(); info->address =3D addr; info->folio =3D folio; list_add_tail(&info->migrate_node, &scanctrl->scan_list); + count_kscand_migadded(); } - } + } else + count_kscand_idlepage(); =20 folio_set_idle(folio); folio_put(folio); @@ -997,6 +1036,12 @@ static void kmigrated_migrate_mm(struct kmigrated_mm_= slot *mm_slot) =20 ret =3D kmigrated_promote_folio(info, mm, dest); =20 + /* TBD: encode migrated count here, currently assume folio_nr_pages */ + if (!ret) + count_kscand_migrated(); + else + count_kscand_migrate_failed(); + kfree(info); =20 cond_resched(); @@ -1202,6 +1247,7 @@ static unsigned long kscand_scan_mm_slot(void) =20 for_each_vma(vmi, vma) { kscand_walk_page_vma(vma, &kscand_scanctrl); + count_kscand_vma_scans(); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 if (vma_scanned_size >=3D mm_slot_scan_size || @@ -1237,6 +1283,8 @@ static unsigned long kscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + count_kscand_mm_scans(); + total =3D get_slowtier_accesed(&kscand_scanctrl); target_node =3D get_target_node(&kscand_scanctrl); =20 @@ -1252,6 +1300,7 @@ static unsigned long kscand_scan_mm_slot(void) kscand_update_mmslot_info(mm_slot, total, target_node); } =20 + outerloop: /* exit_mmap will destroy ptes after this */ mmap_read_unlock(mm); diff --git a/mm/vmstat.c b/mm/vmstat.c index 4c268ce39ff2..d32e88e4153d 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1348,6 +1348,16 @@ const char * const vmstat_text[] =3D { "numa_hint_faults_local", "numa_pages_migrated", #endif +#ifdef CONFIG_KSCAND + "nr_kscand_mm_scans", + "nr_kscand_vma_scans", + "nr_kscand_migadded", + "nr_kscand_migrated", + "nr_kscand_migrate_failed", + "nr_kscand_slowtier", + "nr_kscand_toptier", + "nr_kscand_idlepage", +#endif #ifdef CONFIG_MIGRATION "pgmigrate_success", "pgmigrate_fail", --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2058.outbound.protection.outlook.com [40.107.220.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 406911ACEC7 for ; Tue, 24 Jun 2025 05:59:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744752; cv=fail; b=IHXlXAtixDnnNzChCbVticXvC35BDTwwE6c5B+OhQuiBcZ1VK3wUHrUOTM4o3fQ2O1wnYzDr9g+aEqxL2L1mVj9YqOpg2i/gXYzLWavbo82yYUD7OTqL+w2FThnfAzp+b2Bpx+Ehm4/jwaSGgWawrLdYq/jgh/bPWbF8SnADzTI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744752; c=relaxed/simple; bh=9M65CGbM34QfG9K8CMW9NfGvvdd2jHcsw7LFVRwxST0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fZVY9xEYkd1rpJKxWnsqTUaX2Icr6L8JdsXyTGBmE1su/BNemtQrBW8BUahhwf3tD5g0M3FSxy9naj8UgwSLeMA4Ticx6uszX2Be6W7bbIMnEm92B6MswSEUuO0peghFOIorglfiOn04VZBcQHGYWNp7gbVHUNfW6FBtk0ZmLPk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=JLZDWksO; arc=fail smtp.client-ip=40.107.220.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="JLZDWksO" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gRJKZtIGEg1u3cVy3moNzrbGt+vLaB0nWKroEcJVBZysi59IR3jEDtKleA8LbR3MbsHQJT2n7yBUb04XpFZrEh47SJPhVixN+cAkHIk1YE2jOn4N+KX++j8pT2LiEnZXC3wSlIYZw/9AjGemejdwuo9Z6ye9Vyl3PJPq+w+/u2/lY4Hl0WvyY4be5h66E1etmcVYo+Z7aPp+rlNyQqwR5OE2cblHcgUlmpTn6hO5m6iMnqjQ7+KRJteZSWB7eFXJrjP9Lt5OfR3vHOngcFogBMfLfHd5qYu8UFzL7Ujeq63qPFbxh+APxjSx75fy9VCJluWPEWSVmEQCUyO1S+x0zQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KKaAakPxqUSAFILU7cEmHZbRTs/vYYCAD9wl7zSr8CI=; b=L2VK3QN8FigIJkpf9ZnwkEC0HsstvoFKSWMJ6+CekooBj8eGMkDpKjH3KGCGOTRDdVbd1iqGoBoQrcXHhBfv9DzHl6XEHWEDJm/bvluB1ShSIVD+bSXL9ROZvLczDZMi6ZVbjj4NGDIEaGZzr9w6n7Dnlb+AdBYZ8WhPO//OCBoaZGszHoRtIJkCFFWHCCnr9aWe4XuwlXnqEdvKr4cQxe0HAwrmLPXrJLbGmR9Q5LkWO2Rc0TkvpZMjLbGfhz7e1c+g+L4EWLUUT7SY1Iq0ab2EB4CLkCvhX2wxlwx2fv96+Nd8ikJQyv56U62kywKUb/8JWdENgDlwojx5DDVyLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KKaAakPxqUSAFILU7cEmHZbRTs/vYYCAD9wl7zSr8CI=; b=JLZDWksO4vqNptwHIQSMiE/Dfb7fZz+kWtCphwOwTPeKtQoJeUH7VBknUfeGYMG6LnNyDOsfaSrPHEvyDzCyON6K1pqkrpFUkHsKUOemv4j10nOfN5MgeLIOzyNUpQoDXNNxcyH4KA/leums9krL8gfrk9/EiQEukjDwxK6vFEw= Received: from MW4PR03CA0336.namprd03.prod.outlook.com (2603:10b6:303:dc::11) by SN7PR12MB6838.namprd12.prod.outlook.com (2603:10b6:806:266::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.28; Tue, 24 Jun 2025 05:59:06 +0000 Received: from SJ5PEPF00000209.namprd05.prod.outlook.com (2603:10b6:303:dc:cafe::84) by MW4PR03CA0336.outlook.office365.com (2603:10b6:303:dc::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8857.30 via Frontend Transport; Tue, 24 Jun 2025 05:59:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000209.mail.protection.outlook.com (10.167.244.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:59:06 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:58:50 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 12/13] trace/kscand: Add tracing of scanning and migration Date: Tue, 24 Jun 2025 05:56:16 +0000 Message-ID: <20250624055617.1291159-13-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000209:EE_|SN7PR12MB6838:EE_ X-MS-Office365-Filtering-Correlation-Id: b063c693-2074-4ae4-4a9a-08ddb2e43ad9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?RMHyX30NOlqFNBO6jjfnqZhfXP+xRs7VDzMgBCuYP74zNGPrYQl3inYcOu8b?= =?us-ascii?Q?c4qq2j06VR6RuQZGw+Yo0xV+Vg8XfiOWrUZdwsBqpBGTIEDsE+0SmHu0bs6n?= =?us-ascii?Q?SM/mqpiPUcaaVVkfuMiJEx16vA649FTh/LpF0EiGVCEeqtat6bmDxEUw+utH?= =?us-ascii?Q?87EMyvMoD7r39end/zOv70IfH6KyMKvavri0BmaagDvWKL2h9yV9fPFLPQEN?= =?us-ascii?Q?/n4YjnKJnSm8xZ3ty9HWM9Kd1pKaePkpEMqiua/sk2juMTVhhc7G46rCT3RU?= =?us-ascii?Q?sRdo7xQSMfV2+zNU6/smwM13W5Tf+TPCSlNjhNDMqkw//D6ubeEZOzSlbpLx?= =?us-ascii?Q?o8O1PBHNAKoU6Uad6L8PRdgHRwH3eq1xCFWHDTP2hxbWnBSwguMKzI4jnncR?= =?us-ascii?Q?ePGxIocxfCX0zNh4iwpxvhcT4v/5kGb6+hIIskg7Thp+JvKFT4ILTvvA/RvL?= =?us-ascii?Q?zsbCyGZ87YPvFwk+Xc/aCtG00yNT5+0J2M4gUcPRAC9lvgY2bB/ap+SgOauf?= =?us-ascii?Q?qJUA6njWZmaLr3sDlt3Y/Q5rt1vxgTvyjRG6bqvgiNav/Z8oZh4SUiq6hWKP?= =?us-ascii?Q?EWxYGQuxLJoiNCTjR5wjAZRHIX8qWvGz/ZYu+for5yOm2rEc71Zh/JMHtzh1?= =?us-ascii?Q?ZlV/okT2iX2GMHWB+27h9nrWC7hmyq+hV4q4DovLRCJdsZG7Z8xFNduELk9O?= =?us-ascii?Q?FkiMjmcL59Q9gPEZKlkzNpm9Ytp4ibUm0ZKB0Lzx7R5SAXX9otLAjuW6ARa5?= =?us-ascii?Q?KFWP/1WD0kO28VXhlP3RgZqnN2zoMJ+rhBoxWg8JVsAkd88ioixF4usfXWMj?= =?us-ascii?Q?aD3isZAmw2nh1mqbOBYYGeogoXHK/7ioX7gbH4NKw2PlaJHzmzBHD0nP45t9?= =?us-ascii?Q?IgU1RSG6UhwKiVmgC++tdG4933HPpPEYQUM1mDWqNe+eHB8RG0+8DbXea+NF?= =?us-ascii?Q?7IVaem/wd2UHECYkUPhH/SNjl2v32NIFtfBPhFyVCuHdu/x65g4R4dUyYNGn?= =?us-ascii?Q?ju4Ozr3B5WVqrH+CXnCWgEqhA8Zcv1bnRpbB1mEDogEBkx10P/suvy5owmZG?= =?us-ascii?Q?/fTFN9HGEJ2vjp19BlCuaxwmx55bb+U719lok3tyk/9rKvm602IQWtYjFwkl?= =?us-ascii?Q?PcvcycukdM34BTXEfuFhh3i7Vo1IxHgztRpPB0/1hYVtyUDRoX2v6Vegzm0Y?= =?us-ascii?Q?OU+QIlKl1okIiC+OIXsIHc/+ykq1AuMzm6RoT/b9ZVW1uuX8lfRz3K7xSh4+?= =?us-ascii?Q?YAzoEhiC1GPsrI7G3nBnzFmJhs13oKnAGaFe9IUSlOkivjwCP1JxWmOSyFvL?= =?us-ascii?Q?O/l/RJw0JC2uUG1hdGvV8dzqFedbzbuIvB71O/o0H8MEeGBKjviNHcOydIqp?= =?us-ascii?Q?HfMafrHrT6sjWBLXGMP0auiZTBYa4fOwFEJ+eWDfFX1YqOQi5XbX1gb/Up5L?= =?us-ascii?Q?9QgrvSMyDTxJqD5N/O8ow6EKeH/QcXO2KDVIlqFBQ1/+wlmnY7RWh/Uly22/?= =?us-ascii?Q?j4X01igDJ3pDbYXm3DqGxx8Bz4lLgWLZqe5A?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:59:06.4876 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b063c693-2074-4ae4-4a9a-08ddb2e43ad9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000209.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6838 Content-Type: text/plain; charset="utf-8" Add tracing support to track - start and end of scanning. - migration. CC: Steven Rostedt CC: Masami Hiramatsu CC: linux-trace-kernel@vger.kernel.org Signed-off-by: Raghavendra K T --- include/trace/events/kmem.h | 90 +++++++++++++++++++++++++++++++++++++ mm/kscand.c | 8 ++++ 2 files changed, 98 insertions(+) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index f74925a6cf69..682c4015414f 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -9,6 +9,96 @@ #include #include =20 +DECLARE_EVENT_CLASS(kmem_mm_class, + + TP_PROTO(struct mm_struct *mm), + + TP_ARGS(mm), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + ), + + TP_printk("mm =3D %p", __entry->mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_enter, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_exit, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_scan_mm_start, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +TRACE_EVENT(kmem_scan_mm_end, + + TP_PROTO( struct mm_struct *mm, + unsigned long start, + unsigned long total, + unsigned long scan_period, + unsigned long scan_size, + int target_node), + + TP_ARGS(mm, start, total, scan_period, scan_size, target_node), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( unsigned long, start ) + __field( unsigned long, total ) + __field( unsigned long, scan_period ) + __field( unsigned long, scan_size ) + __field( int, target_node ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + __entry->start =3D start; + __entry->total =3D total; + __entry->scan_period =3D scan_period; + __entry->scan_size =3D scan_size; + __entry->target_node =3D target_node; + ), + + TP_printk("mm=3D%p, start =3D %ld, total =3D %ld, scan_period =3D %ld, sc= an_size =3D %ld node =3D %d", + __entry->mm, __entry->start, __entry->total, __entry->scan_period, + __entry->scan_size, __entry->target_node) +); + +TRACE_EVENT(kmem_scan_mm_migrate, + + TP_PROTO(struct mm_struct *mm, + int rc, + int target_node), + + TP_ARGS(mm, rc, target_node), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( int, rc ) + __field( int, target_node ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + __entry->rc =3D rc; + __entry->target_node =3D target_node; + ), + + TP_printk("mm =3D %p rc =3D %d node =3D %d", + __entry->mm, __entry->rc, __entry->target_node) +); + TRACE_EVENT(kmem_cache_alloc, =20 TP_PROTO(unsigned long call_site, diff --git a/mm/kscand.c b/mm/kscand.c index db7b2f940f36..029d6d2bedc3 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -1035,6 +1035,7 @@ static void kmigrated_migrate_mm(struct kmigrated_mm_= slot *mm_slot) } =20 ret =3D kmigrated_promote_folio(info, mm, dest); + trace_kmem_scan_mm_migrate(mm, ret, dest); =20 /* TBD: encode migrated count here, currently assume folio_nr_pages */ if (!ret) @@ -1230,6 +1231,9 @@ static unsigned long kscand_scan_mm_slot(void) mm_target_node =3D READ_ONCE(mm->target_node); if (mm_target_node !=3D mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node); + + trace_kmem_scan_mm_start(mm); + now =3D jiffies; =20 if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -1300,6 +1304,8 @@ static unsigned long kscand_scan_mm_slot(void) kscand_update_mmslot_info(mm_slot, total, target_node); } =20 + trace_kmem_scan_mm_end(mm, address, total, mm_slot_scan_period, + mm_slot_scan_size, target_node); =20 outerloop: /* exit_mmap will destroy ptes after this */ @@ -1453,6 +1459,7 @@ void __kscand_enter(struct mm_struct *mm) spin_unlock(&kscand_mm_lock); =20 mmgrab(mm); + trace_kmem_mm_enter(mm); if (wakeup) wake_up_interruptible(&kscand_wait); } @@ -1463,6 +1470,7 @@ void __kscand_exit(struct mm_struct *mm) struct mm_slot *slot; int free =3D 0, serialize =3D 1; =20 + trace_kmem_mm_exit(mm); spin_lock(&kscand_mm_lock); slot =3D mm_slot_lookup(kscand_slots_hash, mm); mm_slot =3D mm_slot_entry(slot, struct kscand_mm_slot, slot); --=20 2.34.1 From nobody Wed Oct 8 22:32:51 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2054.outbound.protection.outlook.com [40.107.244.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14E812417F0 for ; Tue, 24 Jun 2025 05:59:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744758; cv=fail; b=hpLIF3eCB7xcjTk81a8rLzSB6a4wMOHnpTeS2AnkpKpBV/VG6BTnJAYN4RhZV9FWuuGjb6NCt6txvgj7zrayFeA1amaB/o01zuZ5km94Ge7+srwR8X5ZegU7LbkuPlyYrUn9xSinO5uwye86t+QnVeT95ua+OuWYtuPpi1oCx6U= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750744758; c=relaxed/simple; bh=KyR2ATLk8X3xjIYnrfHBewEjlYUcshYiDnfpoCputW0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=LxJUZKP0J4Ai4pS7qwvVqGvq5v/tQyUdN1MtxFiOh71hBLaprYDZUva/3nKu88sEfYKzGbKKpxqwXY0uNoxY19kRvLTew22ktjqIpunV0mTed1oIJXsuKbqxXsdeavPVzUTIiD3QoyedWLd70Qjw2sek4ur2mvLZ9V4Bo8NZYO4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=kVtyonLc; arc=fail smtp.client-ip=40.107.244.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="kVtyonLc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=oS1tGSk33rzvxxQSiTmFKnyq2pwb5hJEavQ0pIuACyfrxs3YT6fQHONTdzbOFvDm1eQMkUJDma71s32jQH0iU8/Y4lScJtzzf9823ewP+02USIcfItogxU9salraDtLkl5QEEtNxSpgEJwqNu0ir+QTQ0tBzg+c9s7ZYThBxTMhUhg2HqJuXDfeuGXwZTPN8ObdS3X8jp8cfOtPm56G3WW9jGRy9zE/gnPYFG2f2u2JzGo/5FSTsXzzp8UnW/OGa2cvYTrTZNK3bGhdxOiZXZMce250QAQi98rfQdSjb7iL3G3PJnIZFJDXzrV5DtIkjE4iBdY0R4kRTrWNz0HvN+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Pks/CxUpRVx4ri1CeO6Rf+GCF52kOhG3dUuEvxrIVZk=; b=FBHHVXsdlnof6ouTkWJfPeyEB7N7It18G7j6Qq1wM14yUSt5dcpem28PT3MYpfr0+FgGEo07gqjGxAEeBg+PeCrvZzMvTRCW9cBHuEvoZUs0FMdndj0Bp0mL/4k8sh92EacsjlQisLZEXsZgtaxza7ikdrH2ffgrSA1UcZBEsogFxyHicKsr84hruuRWmSfffM6d06pFk0m4l0rlBpsi7emIpgUTdcGuDHaxBeAjVhFrqZ6t7LcCmUouKxp/fLbGRnSUEJFVP8tTk9WUX0vJgDjjawcuovktSB373hi/sn4ilZ61AaUSNUZZWFAvp545a67iccB4lQ5Naf+0aevZ9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Pks/CxUpRVx4ri1CeO6Rf+GCF52kOhG3dUuEvxrIVZk=; b=kVtyonLcF+yDYjH/04zk7WgdEZIdyqGYHHPASEqy+ouG4CXhLXjgy1tHz+Y86yzlcOIT+jvvGFVGutEwoOM18u7kWP3KJUgWajtM+B8xANYPkO/LKta/dhHY7MAfC4j3gqdQCmzE++N1B5pwipY2TH/FEjdSzuEp9qw4IqWY9j0= Received: from MW4PR03CA0346.namprd03.prod.outlook.com (2603:10b6:303:dc::21) by CY8PR12MB7337.namprd12.prod.outlook.com (2603:10b6:930:53::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.28; Tue, 24 Jun 2025 05:59:13 +0000 Received: from SJ5PEPF00000209.namprd05.prod.outlook.com (2603:10b6:303:dc:cafe::36) by MW4PR03CA0346.outlook.office365.com (2603:10b6:303:dc::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8857.30 via Frontend Transport; Tue, 24 Jun 2025 05:59:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ5PEPF00000209.mail.protection.outlook.com (10.167.244.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8880.14 via Frontend Transport; Tue, 24 Jun 2025 05:59:13 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 24 Jun 2025 00:59:01 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V2 13/13] prctl: Introduce new prctl to control scanning Date: Tue, 24 Jun 2025 05:56:17 +0000 Message-ID: <20250624055617.1291159-14-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250624055617.1291159-1-raghavendra.kt@amd.com> References: <20250624055617.1291159-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF00000209:EE_|CY8PR12MB7337:EE_ X-MS-Office365-Filtering-Correlation-Id: d8ded103-750c-4ef0-8ba4-08ddb2e43ed3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?qge56PGRPLdXHMK+o7XxZdUdH7ITQtYeS9zEMMlG8IjNK8EnziMHTXXfwa/1?= =?us-ascii?Q?wL55fHYNG/mXJZcwP7nGzFgto1WMH7vWgjqZNdTbmupKNldMlZSOr/FqvDSp?= =?us-ascii?Q?XDgNgaI5lr+eFCB91WQ1+Tguz811sxfYmcPIJmbRuvqGNNiY2/D4We/UdTmF?= =?us-ascii?Q?SFZJBmZiRwjd/SqVe7gmZys0s4GRIzoyWKR5Av7uFUwAgmpM57IBgcp+A9Yk?= =?us-ascii?Q?ipoO8OPvuovOYXmGWinrpEPPatS3lkC3WuVEotETQwLgWPc4dNpUmzKkTmhY?= =?us-ascii?Q?T1pk0iwmuig/xASdeIW49v4cPT8DGn4snkbvhnyH+WOTlrNpD7qGDeIUURnU?= =?us-ascii?Q?R03hVF0slzA4jeuCjkvgCMqIWf3fJnksnphLx61O6ixkAYotnHxK9ep3YIc/?= =?us-ascii?Q?bRQy7yMMXfsHrxHHp6sALsbwXX/eI0ak9/eThyB5GMpm1FPtShwQvZ7lGxg0?= =?us-ascii?Q?kdolNWFZs3GmUygoqWENDXPRa9ph35h2LcWMp3qhU/NoIOFOTMEvxyV5YEmT?= =?us-ascii?Q?5PE+ubUz84mz+OB2RyAcxIcFNznmI7NAqPRsZqveJvqyuNGLcXBUHaoD14Y3?= =?us-ascii?Q?Lbzy1Fm64Qi6KIZOgy1CAfva4f6lO7p5bhJRUqYzg9YA+0x4f1SSWjDCNU0Y?= =?us-ascii?Q?3k5NTPUEgJ/CNStN410l0tYECMUqBppcwjsqc5iJSReuX+UWmM6D8w7Ba7Wc?= =?us-ascii?Q?HsltCjp8yS6a+iiE25f8HipVTIq5jaGH/cNKnBbgstsq7Yu/TmsTBHlhPRQ8?= =?us-ascii?Q?UnkVLvS1tC9soDLGslEizUcGTkF9/Sy21dB6yq1n5z/EmFpjnKxyixrlIHBp?= =?us-ascii?Q?b+Hlm9SHaCQhnhroXf6phUjxkENcXKIIDKBhJBEwLJLymhlpk/odOVk45Caf?= =?us-ascii?Q?jC8TRCsvqu6QlJdkxcClU23l+vV7GKDxpOXUt7h1W2aIplJKITf5dblrl10y?= =?us-ascii?Q?Vkq/Am7p7mQ4Ug5BqxjfWcVmE6/gd9u7kSjtMe2zIT9eycZyTx5vkuBeVzT8?= =?us-ascii?Q?sSUP36DzRVIu8AjXosyCR3GTz5s+QqoW2zLqcZaptn/ZTHh5rzLjJKS4QcdT?= =?us-ascii?Q?jPnPpqcCs4ZPh92zyi/gKJjNXzS2H47VhAKKJHCv8aDl5y5Wa+awLHrGo6dJ?= =?us-ascii?Q?TDP6RAxhR1xiQ69IjOH3CjXnOnSVMU/ZgCGjFAu9ovCHI4Tmc2UsAxSFutCg?= =?us-ascii?Q?wXNU9p86jybyih2Rb4gA6WA84EybkGV0X9p32G/dKxaUR3QbucDzUUYAPObQ?= =?us-ascii?Q?mzqtVP2LtXiP5Po8aQ8EOmnu/wSor0b0VM4gUnNlOuhumHwkTMxxqhd+vXJf?= =?us-ascii?Q?O7AMRk+zWJtAqT2Qz4G275t1f+fxrah21wqvXcj9UDktdrOiNB1Zehty34zx?= =?us-ascii?Q?78KUYzMrxqAFO9MAIqRx/Nsq68JHhq1Ah2608wtkoS4sS1naMg5PILV1Do77?= =?us-ascii?Q?3w7gyieS4vg5DZQYSqdrK/dG4CxZgkXwEqa6ZbCXqVkl7mK+fQBxY3OlMTPg?= =?us-ascii?Q?YoRhaPOiHsJafynrtZZdxZVRcWzduxJF/dOj?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 05:59:13.1528 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d8ded103-750c-4ef0-8ba4-08ddb2e43ed3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF00000209.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7337 Content-Type: text/plain; charset="utf-8" A new scalar value (PTEAScanScale) to control per task PTE A bit scanning is introduced. 0 : scanning disabled 1-10 : scanning enabled. In future PTEAScanScale could be used to control aggressiveness of scanning. CC: linux-doc@vger.kernel.org CC: Jonathan Corbet CC: linux-fsdevel@vger.kernel.org Suggested-by: David Rientjes Signed-off-by: Raghavendra K T --- Documentation/filesystems/proc.rst | 2 ++ fs/proc/task_mmu.c | 4 ++++ include/linux/mm_types.h | 3 +++ include/uapi/linux/prctl.h | 7 +++++++ kernel/fork.c | 4 ++++ kernel/sys.c | 25 +++++++++++++++++++++++++ mm/kscand.c | 5 +++++ 7 files changed, 50 insertions(+) Rebasing to upstream tree will change prctl number. diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems= /proc.rst index 2a17865dfe39..429409c341ac 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -205,6 +205,7 @@ read the file /proc/PID/status:: VmLib: 1412 kB VmPTE: 20 kb VmSwap: 0 kB + PTEAScanScale: 0 HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 @@ -288,6 +289,7 @@ It's slow but very precise. VmPTE size of page table entries VmSwap amount of swap used by anonymous private data (shmem swap usage is not included) + PTEAScanScale Integer representing async PTE A bit scan agr= ression HugetlbPages size of hugetlb memory portions CoreDumping process's memory is currently being dumped (killing the process may lead to a corrupted = core) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 994cde10e3f4..6a1a660d9824 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -79,6 +79,10 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) " kB\nVmPTE:\t", mm_pgtables_bytes(mm) >> 10, 8); SEQ_PUT_DEC(" kB\nVmSwap:\t", swap); seq_puts(m, " kB\n"); +#ifdef CONFIG_KSCAND + seq_put_decimal_ull_width(m, "PTEAScanScale:\t", mm->pte_scan_scale, 8); + seq_puts(m, "\n"); +#endif hugetlb_report_usage(m, mm); } #undef SEQ_PUT_DEC diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 571be1ad12ab..ffdb9207cc4f 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1112,6 +1112,9 @@ struct mm_struct { #ifdef CONFIG_KSCAND /* Tracks promotion node. XXX: use nodemask */ int target_node; + + /* Integer representing PTE A bit scan aggression (0-10) */ + unsigned int pte_scan_scale; #endif /* * An operation with batched TLB flushing is going on. Anything diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 15c18ef4eb11..2f64a80e5cdf 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -364,4 +364,11 @@ struct prctl_mm_map { # define PR_TIMER_CREATE_RESTORE_IDS_ON 1 # define PR_TIMER_CREATE_RESTORE_IDS_GET 2 =20 +/* Set/get PTE A bit scan scale */ +#define PR_SET_PTE_A_SCAN_SCALE 78 +#define PR_GET_PTE_A_SCAN_SCALE 79 +# define PR_PTE_A_SCAN_SCALE_MIN 0 +# define PR_PTE_A_SCAN_SCALE_MAX 10 +# define PR_PTE_A_SCAN_SCALE_DEFAULT 1 + #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/fork.c b/kernel/fork.c index af6dd315b106..120ee2ba7d30 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -106,6 +106,7 @@ #include #include #include +#include =20 #include #include @@ -1311,6 +1312,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, init_tlb_flush_pending(mm); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !defined(CONFIG_SPLIT_PMD_PTLO= CKS) mm->pmd_huge_pte =3D NULL; +#endif +#ifdef CONFIG_KSCAND + mm->pte_scan_scale =3D PR_PTE_A_SCAN_SCALE_DEFAULT; #endif mm_init_uprobes_state(mm); hugetlb_count_init(mm); diff --git a/kernel/sys.c b/kernel/sys.c index c434968e9f5d..aff92ff2c7dd 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2146,6 +2146,19 @@ static int prctl_set_auxv(struct mm_struct *mm, unsi= gned long addr, =20 return 0; } +#ifdef CONFIG_KSCAND +static int prctl_pte_scan_scale_write(unsigned int scale) +{ + scale =3D clamp(scale, PR_PTE_A_SCAN_SCALE_MIN, PR_PTE_A_SCAN_SCALE_MAX); + current->mm->pte_scan_scale =3D scale; + return 0; +} + +static unsigned int prctl_pte_scan_scale_read(void) +{ + return current->mm->pte_scan_scale; +} +#endif =20 static int prctl_set_mm(int opt, unsigned long addr, unsigned long arg4, unsigned long arg5) @@ -2820,6 +2833,18 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, a= rg2, unsigned long, arg3, return -EINVAL; error =3D posixtimer_create_prctl(arg2); break; +#ifdef CONFIG_KSCAND + case PR_SET_PTE_A_SCAN_SCALE: + if (arg3 || arg4 || arg5) + return -EINVAL; + error =3D prctl_pte_scan_scale_write((unsigned int) arg2); + break; + case PR_GET_PTE_A_SCAN_SCALE: + if (arg2 || arg3 || arg4 || arg5) + return -EINVAL; + error =3D prctl_pte_scan_scale_read(); + break; +#endif default: trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5); error =3D -EINVAL; diff --git a/mm/kscand.c b/mm/kscand.c index 029d6d2bedc3..2be7e71c2c8f 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -1228,6 +1228,11 @@ static unsigned long kscand_scan_mm_slot(void) goto outerloop; } =20 + if (!mm->pte_scan_scale) { + next_mm =3D true; + goto outerloop; + } + mm_target_node =3D READ_ONCE(mm->target_node); if (mm_target_node !=3D mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node); --=20 2.34.1