From nobody Wed Dec 17 05:28:28 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2047.outbound.protection.outlook.com [40.107.237.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2BBA212FBD for ; Wed, 19 Mar 2025 19:31:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.47 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412681; cv=fail; b=n7PQUxx0QS+IfvIFeSY3U18Ko8+Q2a4MvbNl+PRD6+EmqKhUwkV8SQQB7KkPqAXwlIFRUo2PERAf2sgjBvx3z/WdyfC7i1wTLyGd/8ZK85tnLaTPFADLtITKU4hnUcuhmbM7a8XIld32InFroi8EQrgCl0D3vcJUAJUEnhRE1v8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412681; c=relaxed/simple; bh=2pUCrRZWlOG6fc23vIhfvL/s5SvBC38MUtr9NJCKhQE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=A/sP+MgoNwAGHQIKf1DEJt4eVbSog85EB/UnGSOHFVAFvmb3FX0VbvPCxIkBA+fn0HE3tEVT+xyCXnK9QJ8XRIqoe8xWRURHzfa2B5HT3j5fXfLK60/BJhDjC91lu4AWybqZ2+veCU3TBPsSZ22Yx5qq5dMhcu/eoJQsorwR3XA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=NHUsEDeZ; arc=fail smtp.client-ip=40.107.237.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="NHUsEDeZ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xk1QqGh6HBCRoIKnxFcZ2n//DJ1VX8R/RLfqy9TQGGdFkDbXRw/+sKX/VlQmhz/rrPlkXFxoyWh6wBfHRYAPKWPTrmHd9wC0GebVv6helMsYqo2rOYs4ldGG6N3hsm56XSo+O9tptPQsB7LgkKAEvdcPO/jIbC4Kp5dqMDDLwiy6aDmXMH/5Axcl3tQaCaaQ2Iw54x1mfQKsNIwzCssf29hU2hr52n9wcOQOe1ZwwT9sTMMrgOG+lekLfCeIqYbB4Dwe1+qtabYpufZiCFbtH04XUQv79GhHbBWOCIH8NYVFeEYw6ILk7erpfU0wKHC/N0tA7qwB+GUOUJOyhUMtIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n1YlgkAYpPVRIE0Exq0WZ36EVvjUeFBDZwrxw1q80i8=; b=wRirfef74xUPsuzFwZM5FpQyP0iCAv48ehirJYkgPSZXPBgBKKY7eJGTvDGaAuRnITiQpTPw44BCYkVSoijeJDJA5wkuCdz2ZUrWZoOk1iXRIw7hcEfbwZJt4S9vTJOQEruYk8OTd+qhRyQXXuLhBCDKMEvmpkW84rPJmMhKCADN5rXb4oyR1DXEpILyAKwONkfrPm0EMTKNmMNh2+IWNNJjf0moWKROfNLsgAkmG+LBx5XFpLRIDn/BHopTxnSLTVxuyyhsKJhLduTaiBnXG2x+sjDda6uVFb+0c6ML+j8iZx9D+C5FLhIG3xLQ9csNTcKrbYZB4bGs8/G6Hv/jaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n1YlgkAYpPVRIE0Exq0WZ36EVvjUeFBDZwrxw1q80i8=; b=NHUsEDeZnSYaavYG0VPsefHnUOziaVjhbm8JatK9VWDk0dT9AKULm6iR8rVOuBQSRH/ZKRafcKqJ3wRP08euY47XsJ2UmLPoOfH+GsD8WGi/w580nHMQCNTW1y9LpB5JKhshK+JlMNz5AX1gNQCfZbW+ZckAKI4ekiMlzOLRYNM= Received: from SA1P222CA0187.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:3c4::14) by PH7PR12MB7258.namprd12.prod.outlook.com (2603:10b6:510:206::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:31:13 +0000 Received: from SN1PEPF00026369.namprd02.prod.outlook.com (2603:10b6:806:3c4:cafe::c5) by SA1P222CA0187.outlook.office365.com (2603:10b6:806:3c4::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026369.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:13 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:02 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 01/13] mm: Add kmmscand kernel daemon Date: Wed, 19 Mar 2025 19:30:16 +0000 Message-ID: <20250319193028.29514-2-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026369:EE_|PH7PR12MB7258:EE_ X-MS-Office365-Filtering-Correlation-Id: 8966c4a0-bc7f-44a7-2b6a-08dd671c9c2e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?uOsbzxn9Ra/71b3QdilmFLcwsYqgpUH8zMdp5Ax2j3KUkZTt1blWU399UecF?= =?us-ascii?Q?QwEHWrSbSx1RCwQ1lKT3aRgkytNVHhitIxH0J62qi3Oyk8pBqRc0364V5Yag?= =?us-ascii?Q?qGlzJu4x7qNo16eBdtWcm22bD+3AWUoXCrpv6yMvCXaSVomIFS380EWlUJhD?= =?us-ascii?Q?c7mRi7HL4i+b2+tkkIBi72i1Nphw5FqKhPAEwf8Rc+JvMpx0yBZ9Ema3en6/?= =?us-ascii?Q?OhVeDqJSwATPY39rxxNx4AZu6+i4mKRMGUX9snlUL7rMrR5bUf5OODCGrdmE?= =?us-ascii?Q?vbJ+fSHDw98bPapVLq2Y/Dx+3j4QjJMx2Pzi0WzxOwStTuT/iqlX6dQPZmSU?= =?us-ascii?Q?GH/ukniK0yNXF8jWNh2C7uWlVDzeRP7QTaA/pZheFcFhwWwA97ioTaV6OzeE?= =?us-ascii?Q?ZGDs/tgrT4JBKa2DmPK31J5dYccFy1e8i9xE81ln6ROZ5fdOPOtKEbD+q4QP?= =?us-ascii?Q?pDIo2eqnHQ4pKEThhtbv2DLi63iCg+7Ahjwzdv5t1cWGG3/u65aCyICGHzcC?= =?us-ascii?Q?eE/ulAR2cGcFRDnQsUC+hHO7tuXXgJ6+m9SdcA3ed9TtwJU9GtZZUWwTNMXe?= =?us-ascii?Q?F3lBnZhRpx755OR8nFEZbTsB6ymA+ySJzXrUf3Fq9Qi14Eu9rl8BGj2oD/HS?= =?us-ascii?Q?WZ2r40OXANzApJnmz5SXNUpUFyOp9MDmp24ROyFaSfp+/jNdoYYdY01RilVw?= =?us-ascii?Q?UokQhG/DuKdqb0QpFY/TfIo97Hx+H1jw2kGolN9VyfDQA+X5z81lobaHL4gv?= =?us-ascii?Q?3aYvsiI8xWuPGmcMbIsIWNSapSc2AwLWEQ4WM2tM+VlbZ3f2bnzuhZpUz2hN?= =?us-ascii?Q?llNCWF1tKncjMWZPDQOCrYlRpRfx4Jyiemye+4AyawaMjplZgXfEf5qrMqfN?= =?us-ascii?Q?08Y8ZXXD3vC6sgYICsJ92HcyHQuObrWtyLfL4iYWB+Q85vj3C/+rrEvpd8z8?= =?us-ascii?Q?1TM45KDg5UZo506zDgAmNFcaUQnDktniFWdKWl8haOpNG0kVPJStTU90w8c3?= =?us-ascii?Q?iFAzznfMx0I072IuGpw/5gJmiHhPBz25EZjXC1Eg4hommyAnoZHjOrgtiLTa?= =?us-ascii?Q?zkpD+cgNahu8sufAPoJrmMThPKF27UghP4Ltc0C8jdqg2ixyoULboNA92oie?= =?us-ascii?Q?vqC2/SxKuU7SbA5ZH8hv4xUb45f1ZQZVfhMwS+OQWIL2f7DAi3jpVTNeWSp9?= =?us-ascii?Q?qvsV0m4Hsceb+dqiXkUEZzA80xmKBZUaKwCb+Zav1E9aemWnUf6kOSgYNx4x?= =?us-ascii?Q?dQGohjJ0Xir7Ajva61uScEoIIrqGIs0KyzAmioFvHmLILJpLg8er3PzgtGKU?= =?us-ascii?Q?9DTsEme197NdthIoMGLFDVEvw5do4Ahp/G0IMUrwc/ZTnH3PuwfLkW1FXRUw?= =?us-ascii?Q?RqprU7lvm75q3nUpFMqaIGBBk5ZZxpnFzpi8CQtVSm/Lms7XgkBBvjyDtoGW?= =?us-ascii?Q?R1bE6yzdmcGMZuth5X5k3w8oBAsEotNcTjwxc1KY0hPc0nseeDCuJjLwiD2H?= =?us-ascii?Q?TR8z1FeUhtpnR7A=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:13.1620 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8966c4a0-bc7f-44a7-2b6a-08dd671c9c2e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026369.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7258 Content-Type: text/plain; charset="utf-8" Add a skeleton to support scanning and migration. Also add a config option for the same. High level design: While (1): scan the slowtier pages belonging to VMAs of a task. Add to migation list Separate thread: migrate scanned pages to a toptier node based on heuristics The overall code is heavily influenced by khugepaged design. Signed-off-by: Raghavendra K T --- mm/Kconfig | 8 +++ mm/Makefile | 1 + mm/kmmscand.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 185 insertions(+) create mode 100644 mm/kmmscand.c diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db06417..5a4931633e15 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -783,6 +783,14 @@ config KSM until a program has madvised that an area is MADV_MERGEABLE, and root has set /sys/kernel/mm/ksm/run to 1 (if CONFIG_SYSFS is set). =20 +config KMMSCAND + bool "Enable PTE A bit scanning and Migration" + depends on NUMA_BALANCING + help + Enable PTE A bit scanning of page. CXL pages accessed are migrated to + a regular NUMA node. The option creates a separate kthread for + scanning and migration. + config DEFAULT_MMAP_MIN_ADDR int "Low address space to protect from user allocation" depends on MMU diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..45e2f8cc8fd6 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -94,6 +94,7 @@ obj-$(CONFIG_FAIL_PAGE_ALLOC) +=3D fail_page_alloc.o obj-$(CONFIG_MEMTEST) +=3D memtest.o obj-$(CONFIG_MIGRATION) +=3D migrate.o obj-$(CONFIG_NUMA) +=3D memory-tiers.o +obj-$(CONFIG_KMMSCAND) +=3D kmmscand.o obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) +=3D page_counter.o diff --git a/mm/kmmscand.c b/mm/kmmscand.c new file mode 100644 index 000000000000..6c55250b5cfb --- /dev/null +++ b/mm/kmmscand.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include "internal.h" + + +static struct task_struct *kmmscand_thread __read_mostly; +static DEFINE_MUTEX(kmmscand_mutex); + +/* How long to pause between two scan and migration cycle */ +static unsigned int kmmscand_scan_sleep_ms __read_mostly =3D 16; + +/* Max number of mms to scan in one scan and migration cycle */ +#define KMMSCAND_MMS_TO_SCAN (4 * 1024UL) +static unsigned long kmmscand_mms_to_scan __read_mostly =3D KMMSCAND_MMS_T= O_SCAN; + +bool kmmscand_scan_enabled =3D true; +static bool need_wakeup; + +static unsigned long kmmscand_sleep_expire; + +static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); + +struct kmmscand_scan { + struct list_head mm_head; +}; + +struct kmmscand_scan kmmscand_scan =3D { + .mm_head =3D LIST_HEAD_INIT(kmmscand_scan.mm_head), +}; + +static int kmmscand_has_work(void) +{ + return !list_empty(&kmmscand_scan.mm_head); +} + +static bool kmmscand_should_wakeup(void) +{ + bool wakeup =3D kthread_should_stop() || need_wakeup || + time_after_eq(jiffies, kmmscand_sleep_expire); + if (need_wakeup) + need_wakeup =3D false; + + return wakeup; +} + +static void kmmscand_wait_work(void) +{ + const unsigned long scan_sleep_jiffies =3D + msecs_to_jiffies(kmmscand_scan_sleep_ms); + + if (!scan_sleep_jiffies) + return; + + kmmscand_sleep_expire =3D jiffies + scan_sleep_jiffies; + wait_event_timeout(kmmscand_wait, + kmmscand_should_wakeup(), + scan_sleep_jiffies); + return; +} + +static unsigned long kmmscand_scan_mm_slot(void) +{ + /* placeholder for scanning */ + msleep(100); + return 0; +} + +static void kmmscand_do_scan(void) +{ + unsigned long iter =3D 0, mms_to_scan; + + mms_to_scan =3D READ_ONCE(kmmscand_mms_to_scan); + + while (true) { + cond_resched(); + + if (unlikely(kthread_should_stop()) || + !READ_ONCE(kmmscand_scan_enabled)) + break; + + if (kmmscand_has_work()) + kmmscand_scan_mm_slot(); + + iter++; + if (iter >=3D mms_to_scan) + break; + } +} + +static int kmmscand(void *none) +{ + for (;;) { + if (unlikely(kthread_should_stop())) + break; + + kmmscand_do_scan(); + + while (!READ_ONCE(kmmscand_scan_enabled)) { + cpu_relax(); + kmmscand_wait_work(); + } + + kmmscand_wait_work(); + } + return 0; +} + +static int start_kmmscand(void) +{ + int err =3D 0; + + guard(mutex)(&kmmscand_mutex); + + /* Some one already succeeded in starting daemon */ + if (kmmscand_thread) + goto end; + + kmmscand_thread =3D kthread_run(kmmscand, NULL, "kmmscand"); + if (IS_ERR(kmmscand_thread)) { + pr_err("kmmscand: kthread_run(kmmscand) failed\n"); + err =3D PTR_ERR(kmmscand_thread); + kmmscand_thread =3D NULL; + goto end; + } else { + pr_info("kmmscand: Successfully started kmmscand"); + } + + if (!list_empty(&kmmscand_scan.mm_head)) + wake_up_interruptible(&kmmscand_wait); + +end: + return err; +} + +static int stop_kmmscand(void) +{ + int err =3D 0; + + guard(mutex)(&kmmscand_mutex); + + if (kmmscand_thread) { + kthread_stop(kmmscand_thread); + kmmscand_thread =3D NULL; + } + + return err; +} + +static int __init kmmscand_init(void) +{ + int err; + + err =3D start_kmmscand(); + if (err) + goto err_kmmscand; + + return 0; + +err_kmmscand: + stop_kmmscand(); + + return err; +} +subsys_initcall(kmmscand_init); --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2079.outbound.protection.outlook.com [40.107.236.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EBEF215162 for ; Wed, 19 Mar 2025 19:31:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.79 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412690; cv=fail; b=IwaT1d2mUUCgK6TxmbjTxJOS/SeW2+bPAyLbZllhx0afLuqoWzd87/GublY37GQBXQviWP21YIJxAETeSmDAlL4O2yIPRBny2zIMT0nHJ+KfLjR926vq0wLilat9ZnpTrZWmtTYaNZKW5R/kT60wGABx9/3Ctsc2e9Ds6gIBGT0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412690; c=relaxed/simple; bh=ksY23e3UlOZAYayDclYl6xxMuiuQfRlkLaJxhW3SZx4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Tg5du7Uod5JJdpbLmvNUpt9e71zDH9GcCTyP3h4TDIvtDzMKIAVLha1fVXVlU35GW2TfE47TRAseTxl83uLFacYD0Dm9kjTmFKdDIJYNK8p8qbEJW++PCEB9VGzkwP2oxh5BonLQC/UxU87Z/+c1gfgdeFFhc2nzI53eNOB4iCs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=rcFspfru; arc=fail smtp.client-ip=40.107.236.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="rcFspfru" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kibCpY+AWIHnpIBsHswAK7OW9k8W0vfpD4oZeaGnpRyaTaoPs1NooAsWgc/bp1oep/N5HQAWXVxmKNId6bVE8MYl53R1DZZMKHZSJ1KBQJijIQTOs90osHu4Ki+9Rj9gHjQPKta91IeDE7T7nQZNRAjdKgLByMg18MVfGTbpKUvezQ7T7cGkGR1Z20w9+cQberO9lGagvw+sp+1f53LN3bd/yaoXCq4XrvpqRlfHvTJMBpCNaNDcoH78ZwNKMc7p+rG3wy95MtZcbA8fW9MRWXWoOJp97RtfgYY6ULUYs3Rb3gysWK3HvHv6chGhCVfH2VqK1pb9ETyAbVpa13nmLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=BQxN1+zoYrzxYDKXDthRpMaUaTcmQV4i80r86u5rVouBhOGvaasLvkBGiwlM/LwHMuxCVyC6pIvykXJex3cLoO3+/BDa6P+t48PSu5TfUp/akkmkum0gByFvYszDcCwARkQ7UQmhZe7IEj/UEocRq7jIDRRYsQsmFSmWiyiflnPtgTybTR2eIRPQeuwduIA35tgd4JvYoY5gESSNeeSaiDphW5/EcifTxsUPqGsX1J2cD357Jgujuy04YlMtqWdVGjh+eJ/0lQbh/xmvuKMoz/HE45kPo4Y3ZIZMt+y2ZiRBTYKa2NJ3DhiyUaFN28iGtV/Viu3qrJk7W3S3xCZjwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=rcFspfruOE/sAd/tYVASi3Ib8RxA4n+CSPANb6WncngYZwng7fYy3Xg0hB62tpvH1wDEMiZrnOZhmLpB0fzsVVp1Zm19X0WGugptC4J8Miqe4xZhQnL495gjjpphGLEfTxm6199RsWG4herI3gEh4ptSl2gc84tItucGYC/xpYE= Received: from SA9PR10CA0015.namprd10.prod.outlook.com (2603:10b6:806:a7::20) by IA0PR12MB7675.namprd12.prod.outlook.com (2603:10b6:208:433::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:31:23 +0000 Received: from SN1PEPF0002636E.namprd02.prod.outlook.com (2603:10b6:806:a7:cafe::85) by SA9PR10CA0015.outlook.office365.com (2603:10b6:806:a7::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636E.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:22 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:12 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 02/13] mm: Maintain mm_struct list in the system Date: Wed, 19 Mar 2025 19:30:17 +0000 Message-ID: <20250319193028.29514-3-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636E:EE_|IA0PR12MB7675:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e91f9a3-0125-4018-4357-08dd671ca186 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dexsYIeqHGJqJtQI2EXSAibliOi3YzKMApOlwl2TqowRFfLgxr1MOTyuSD7P?= =?us-ascii?Q?vOFhWadrbBOa5bdaxUiHpruKEBwTJt0W0epoh1bzTuQrIoMERDwMLoWdPgYR?= =?us-ascii?Q?jrjO8E8CMLSR+WadaTCmJX6XkMXE13JehXlw5vXEdy39m6i5Mk78wwlnExHh?= =?us-ascii?Q?/8671u9iqowCP0chs7KdC2dgQo9gpwtxlBnHnfGtkqy+AUPzky1W0zK/WVPJ?= =?us-ascii?Q?6/WBijiJbV5/MVGTMf2xPJGz5nny2x9HDTkW1CvCm/dpvQhuiKUgRtZ5pXu9?= =?us-ascii?Q?gjwdK1WJy8FBv2nRh5xET80sws4Zht1wdTkl69PEi0rYzsH70/415UHc6rZK?= =?us-ascii?Q?Xk5+yEFs2H0cbJXTXRZ79r8ldgKRQtVtj0vIPh6u8Xtkx2ulBbb1okXiWiIu?= =?us-ascii?Q?f2S+YNf7INExZvbSrVEInhsuvt0SmjMlah2Y+kYTRjKZ6BPfwrXaDMDJwWmt?= =?us-ascii?Q?mMftKTAX3AF24p95kE7PV3qLKlr7bib3QXUShKRb0KM1Em2U93ekvn+5GdQH?= =?us-ascii?Q?o4FZjy4l9IzX85HHoE97IoLMIEoE/pEPHp5owgn9Fffpo2SBSdcJPej40OAZ?= =?us-ascii?Q?Up6TTW0+p2YZ5m7Xr4fMdw0wQY08Hsh2GwjCFOvvqRFh/7VdGvY3enIdVEiw?= =?us-ascii?Q?n744sPTQ99LyZAfnuAWeCg3UZso/bIJpttI6tlKceoXJwUoJitDIML62JaFc?= =?us-ascii?Q?x4rAuYS9VONsmtqK2c5GhLwg1vVLly9oqsUhOW3V3eWpNl/snk6+B1JcRJwt?= =?us-ascii?Q?vdlatmQE6xZCtETCkgtdosMx45M/3jagSSgabe9kCVWywMdR1Tj3vPfhmPVl?= =?us-ascii?Q?t3iKzCoCbTEaZnamQ4x/TlsGeFeuCA6LIZyOzZWEkJuBjXv4cCeDIleBB85v?= =?us-ascii?Q?K+lqqCG9w/eR5LtK5sWiDlNOMhTzgP9KDTtm2jcPQZ17WLqM1zVqJ2OZYEwy?= =?us-ascii?Q?WI17xeVHrh/3QjCBDO7r6+Xmim5FpTEuzafLDKX1EMqxTZyIL/fls5+8/gRu?= =?us-ascii?Q?L8zM2/TCdSJzr8Px5zx78OWJjjXeEzzdTZIxYWSHMSI7o9//y6nKxlu5xLug?= =?us-ascii?Q?GWywB2OwtgDpZD81RysEfb8B1pJHYB8JSbl+UNgmvq+XDfC9HuX+VLXp4f95?= =?us-ascii?Q?4X2gzzoKrKZ3LcO+02ZHpT97WR2usYIDfq3ssGmv3POznNnIh2mzkGL1wFCX?= =?us-ascii?Q?Mvbpcs9hBT87di6FGNiMyqaPvtIpLv6rbhrSAbSnZY8flN3PEUZVtJVNyM+y?= =?us-ascii?Q?WX8YEia30F45p9oL8zzdW2BkpdWbNOb2XC6EAO/Nci+oTEE8DgDnTfuMD30T?= =?us-ascii?Q?okg268RRAYxtzGdpSo/GsmFKCh1rj6QxSMc/CPqY9Zv7Ig4epp4VIhkOJ1VX?= =?us-ascii?Q?MzCwCk5V41htmvybGZ+8iA0nVdvNCdUJRxDoFvtVou67+k2C55Uzy1yk296U?= =?us-ascii?Q?KUUdowAO3BQH3UWkvN+vE2t+2h1xN3jFRsLJ0e75aVHBvIuqhD8NLmr8Xa/A?= =?us-ascii?Q?tcujGCjIiJXwT/A=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:22.2373 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1e91f9a3-0125-4018-4357-08dd671ca186 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7675 Content-Type: text/plain; charset="utf-8" Add a hook in the fork and exec path to link mm_struct. Reuse the mm_slot infrastructure to aid insert and lookup of mm_struct. CC: linux-fsdevel@vger.kernel.org Suggested-by: Bharata B Rao Signed-off-by: Raghavendra K T --- fs/exec.c | 4 ++ include/linux/kmmscand.h | 30 ++++++++++++++ kernel/fork.c | 4 ++ mm/kmmscand.c | 86 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 include/linux/kmmscand.h diff --git a/fs/exec.c b/fs/exec.c index 506cd411f4ac..e76285e4bc73 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -68,6 +68,7 @@ #include #include #include +#include =20 #include #include @@ -266,6 +267,8 @@ static int __bprm_mm_init(struct linux_binprm *bprm) if (err) goto err_ksm; =20 + kmmscand_execve(mm); + /* * Place the stack at the largest stack address the architecture * supports. Later, we'll move this to an appropriate place. We don't @@ -288,6 +291,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) return 0; err: ksm_exit(mm); + kmmscand_exit(mm); err_ksm: mmap_write_unlock(mm); err_free: diff --git a/include/linux/kmmscand.h b/include/linux/kmmscand.h new file mode 100644 index 000000000000..b120c65ee8c6 --- /dev/null +++ b/include/linux/kmmscand.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KMMSCAND_H_ +#define _LINUX_KMMSCAND_H_ + +#ifdef CONFIG_KMMSCAND +extern void __kmmscand_enter(struct mm_struct *mm); +extern void __kmmscand_exit(struct mm_struct *mm); + +static inline void kmmscand_execve(struct mm_struct *mm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *o= ldmm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_exit(struct mm_struct *mm) +{ + __kmmscand_exit(mm); +} +#else /* !CONFIG_KMMSCAND */ +static inline void __kmmscand_enter(struct mm_struct *mm) {} +static inline void __kmmscand_exit(struct mm_struct *mm) {} +static inline void kmmscand_execve(struct mm_struct *mm) {} +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *o= ldmm) {} +static inline void kmmscand_exit(struct mm_struct *mm) {} +#endif +#endif /* _LINUX_KMMSCAND_H_ */ diff --git a/kernel/fork.c b/kernel/fork.c index 735405a9c5f3..f61c55cf33c2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -656,6 +657,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *= mm, mm->exec_vm =3D oldmm->exec_vm; mm->stack_vm =3D oldmm->stack_vm; =20 + kmmscand_fork(mm, oldmm); + /* Use __mt_dup() to efficiently build an identical maple tree. */ retval =3D __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL); if (unlikely(retval)) @@ -1353,6 +1356,7 @@ static inline void __mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ + kmmscand_exit(mm); exit_mmap(mm); mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 6c55250b5cfb..36d0fea31dea 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -7,13 +7,14 @@ #include #include #include +#include #include #include #include =20 #include #include "internal.h" - +#include "mm_slot.h" =20 static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); @@ -30,10 +31,21 @@ static bool need_wakeup; =20 static unsigned long kmmscand_sleep_expire; =20 +static DEFINE_SPINLOCK(kmmscand_mm_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); =20 +#define KMMSCAND_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash, KMMSCAND_SLOT_HAS= H_BITS); + +static struct kmem_cache *kmmscand_slot_cache __read_mostly; + +struct kmmscand_mm_slot { + struct mm_slot slot; +}; + struct kmmscand_scan { struct list_head mm_head; + struct kmmscand_mm_slot *mm_slot; }; =20 struct kmmscand_scan kmmscand_scan =3D { @@ -70,6 +82,11 @@ static void kmmscand_wait_work(void) return; } =20 +static inline int kmmscand_test_exit(struct mm_struct *mm) +{ + return atomic_read(&mm->mm_users) =3D=3D 0; +} + static unsigned long kmmscand_scan_mm_slot(void) { /* placeholder for scanning */ @@ -117,6 +134,65 @@ static int kmmscand(void *none) return 0; } =20 +static inline void kmmscand_destroy(void) +{ + kmem_cache_destroy(kmmscand_slot_cache); +} + +void __kmmscand_enter(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *kmmscand_slot; + struct mm_slot *slot; + int wakeup; + + /* __kmmscand_exit() must not run from under us */ + VM_BUG_ON_MM(kmmscand_test_exit(mm), mm); + + kmmscand_slot =3D mm_slot_alloc(kmmscand_slot_cache); + + if (!kmmscand_slot) + return; + + slot =3D &kmmscand_slot->slot; + + spin_lock(&kmmscand_mm_lock); + mm_slot_insert(kmmscand_slots_hash, mm, slot); + + wakeup =3D list_empty(&kmmscand_scan.mm_head); + list_add_tail(&slot->mm_node, &kmmscand_scan.mm_head); + spin_unlock(&kmmscand_mm_lock); + + mmgrab(mm); + if (wakeup) + wake_up_interruptible(&kmmscand_wait); +} + +void __kmmscand_exit(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *mm_slot; + struct mm_slot *slot; + int free =3D 0; + + spin_lock(&kmmscand_mm_lock); + slot =3D mm_slot_lookup(kmmscand_slots_hash, mm); + mm_slot =3D mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + if (mm_slot && kmmscand_scan.mm_slot !=3D mm_slot) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + } + + spin_unlock(&kmmscand_mm_lock); + + if (free) { + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } else if (mm_slot) { + mmap_write_lock(mm); + mmap_write_unlock(mm); + } +} + static int start_kmmscand(void) { int err =3D 0; @@ -162,6 +238,13 @@ static int __init kmmscand_init(void) { int err; =20 + kmmscand_slot_cache =3D KMEM_CACHE(kmmscand_mm_slot, 0); + + if (!kmmscand_slot_cache) { + pr_err("kmmscand: kmem_cache error"); + return -ENOMEM; + } + err =3D start_kmmscand(); if (err) goto err_kmmscand; @@ -170,6 +253,7 @@ static int __init kmmscand_init(void) =20 err_kmmscand: stop_kmmscand(); + kmmscand_destroy(); =20 return err; } --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2058.outbound.protection.outlook.com [40.107.244.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1BED2144A5 for ; Wed, 19 Mar 2025 19:31:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412702; cv=fail; b=V/kkk23NzZfO6W0vcPwQRPjr1kphvRkk6N0fwAeuMq613MaPzOerlELX3hynNmqGskb9smrTm3OnRNJkH2RFCnb2quYWC+tbt75vsFm60BjHo9Pn42wrXictT2BMtb1cQXvjXU3yQnbKZtfBMaBrxpc99b983zOX2xqijfRslgk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412702; c=relaxed/simple; bh=7HU71naNgONWYZmMotLkEvcczG6LKMtPS0soeTGGnGM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AF5uBpHYhFoTp6E+GIozGcnPDyv3ZhVVmrcZemh0YKqhjsaOpyHaEf6u1MOJS+y/1whQtc8NFfLytTvN3FyIHTwloaeXW9p5VfLNan0aRn0XUB5BLI9tYFhpi6csPmH/tiCFcBjr2AIcXaY6dgNqLkvcLRyIyt0Ol5GSDDHFkWA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=FVBN95Of; arc=fail smtp.client-ip=40.107.244.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="FVBN95Of" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EnrfdaYEkoqgwUJbkZKpDtm81CIdd3DCcFJsc4OyQ9hXOfMYMgayiYdvcy74Eb8WBt/KHDe4KC8297sTNWliY4HV1b3jqSaxXcFumY9cMq9yBwhtgz0oM7ncutuNYcDDAAH4q7C9reAPDwPILMUIesFGkIbOLqq/I3tRXvDPhvnr48j80WIui7cTHSql6u/qx7sIvRF8yE2zn9oAxH/3Oqk3AyVkSSNJFgD9usG3EKHmg5qjFwNRy5QkJ+Ztxlv6AT/TbHTpXRPUpKHab4MWoQQ5sg5Ijkcgkaw80IsEcWBVn7TnMSpWh0WGD2BqwtBBr7FK+WoLpWxn6EGznJkWHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cx9mqxLbBvGY1Qdr7POiHWnrJdk6S1CQ2w02ch47VKs=; b=Pr4585eyMNXyJgLLKLkLf8qhGTZBqRuHBHk911iNN2VjPIWgE6tAu0FD2fBnqfweIWNcD4vhUqxL/xWf6Hu4n/ZI0bl9uxlAx+sA34/2ygC3VLaLb9PLWvsvOeV8xhMa/nQM9yjjYV5RkgcbNg+D/RzJr9wP3wSjMHPxpHZxhuUunuoDd6eyt+h0PYUBm+d8O0d/udOc7y3rWj2jvgyEDjBuC8QvMQfbZqOEc8pK3uDIQcFDFlM7FF8yIl1PCKokr9OaHl+WksRnZ0/pwGAjs1/N3nAvLMrjp5nUnhUU3XGV+8m11Iba3lLbBu48fZddey3py+OUDbqySwn7XdF+1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cx9mqxLbBvGY1Qdr7POiHWnrJdk6S1CQ2w02ch47VKs=; b=FVBN95Of5poV8druQhY+53PKt2RzSYkW9rha8O7wtd7L1HRrkT8tHel/r4zmASi/Gh4ne4OhYkEk5kYRO0+htAZMGfzpuJLS2RzJOI6yeFuvIZuN4uVAMbmNBQy0iAMkTUKkUQF99mLVGnfQsPDeptmOlnnKLsOU98J46Ivqje4= Received: from SA0PR11CA0165.namprd11.prod.outlook.com (2603:10b6:806:1bb::20) by SJ1PR12MB6338.namprd12.prod.outlook.com (2603:10b6:a03:455::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:31:33 +0000 Received: from SN1PEPF00026367.namprd02.prod.outlook.com (2603:10b6:806:1bb:cafe::e4) by SA0PR11CA0165.outlook.office365.com (2603:10b6:806:1bb::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.35 via Frontend Transport; Wed, 19 Mar 2025 19:31:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026367.mail.protection.outlook.com (10.167.241.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:32 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:22 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 03/13] mm: Scan the mm and create a migration list Date: Wed, 19 Mar 2025 19:30:18 +0000 Message-ID: <20250319193028.29514-4-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026367:EE_|SJ1PR12MB6338:EE_ X-MS-Office365-Filtering-Correlation-Id: 5feff88a-149c-4187-afa1-08dd671ca78a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?NTMEQ56E2JF6KYEc9gw2QsQreOr597HyqufvmoNP16bDmzEPj/QIC0li9G9E?= =?us-ascii?Q?2yswVh6KBA7wUU3vX016XL2X675QUZDZIeWO/JrTF+ojXQydJd2qSE/w8FQ9?= =?us-ascii?Q?OYDIXQKYhwuEC4kaz7cHtoFhE8lN4+fZxGk+5C4JmEtOX1nISqfoYIokylwh?= =?us-ascii?Q?vODH+3/aVSk5p8Q/ayzL1rtQ0QakDhhlVclV7Bq72QyoWlrsQilelkyIhoc/?= =?us-ascii?Q?inUDSwAWXyQX1hE6p5tD7R0IXjhI04bT9SJcR4I8NTssdtvvMigTRiVC8Xpv?= =?us-ascii?Q?FVnS+JnPFzahSx75PeoXgV9BQKRWeOFgFHwsX+EQ2fxro6Xr6q2in8hQdEYG?= =?us-ascii?Q?uOsTLnMF78YSJIc8SL6nQp0vpAFQjIysXB0Tc+HiUi0rxjyLCfOcydld6u1G?= =?us-ascii?Q?KDzrwC7uwac+cLV1CrIj76Cmu63j3KyxBFA5RfbouYFEj7huS2lTFLGZJYiy?= =?us-ascii?Q?9Uibkd/mmZBBIr/6TRchZLKBYk3NYP50o8SOG8n7GPMQF9bsMx3fNxJF9ux7?= =?us-ascii?Q?pbM+PVOBmLuOrd+cIeFl6q9bgAsbSh0P89xd275/p+qQ2VlKv1U4nWF2M6lI?= =?us-ascii?Q?tjBa0aefc7szodiWGcbO8z45bskN+e62169pYxnWjafkHQ273QDH+/tWj7sQ?= =?us-ascii?Q?5ymDyHY8kVqymvQbNkhEc2tb3ael+gPjzzsZzNZmbyUNyWbdP2m9gYnCbgLs?= =?us-ascii?Q?itWLkCgBZfqS2x8OmOuCV7V9x8ER/aNo8l9oqQR77BW446LEIABZvr3/LCGE?= =?us-ascii?Q?9D49jRIp0XlEJyZZrXACh03ItoDUv5pJRn83C9ECjGNXTaefpKOL4kSUUMQ/?= =?us-ascii?Q?dwbfhWc1bSD/qziStqlMSIuk3Ta2J+nhy2GjjouQySnO60B7VFL5Qp2u72og?= =?us-ascii?Q?OO8BXCA3lWYKoNgdBW4mr4ZKBK/7yE1dJL4D4aNLeGlq2IyWXVfuf9JbjPW9?= =?us-ascii?Q?B/z3luK5O7vlU5mNTxLv/SwOXItj5dvNqKRtlt9t+XkwGEyzyOJiOOZQRJEC?= =?us-ascii?Q?w4WraMt47euEU/pXRpQZPcCmvLqEajsrtryan3WMDTPjVub+Zn+25ZN1gsbn?= =?us-ascii?Q?ju+/rHU5JQnEjtbsOohW3g6vlqonOMP2fbOdQHtSpxZ69s06vM+3mDELBuJI?= =?us-ascii?Q?Yya8k1wW446bcs51fBaivAvV22/++c8PQ7Nphewdk2qhtBz3Sm4eKSAbh2i3?= =?us-ascii?Q?kjAUq9WUaEwucSiTiTfZ/a7NPs67VQFOCvIwEpiB6Trcm+7NYSfU3RJFZziQ?= =?us-ascii?Q?pC3o7v5WVK8xIsXcOCqNM+g+XAfh2uIbk78bhNEI63QCDgtK+VJ1RCIVbw0r?= =?us-ascii?Q?x2XqXx2qocWUVk/69oylzK36Nx3Jk6sSVcTBAHCrcGTm2SvYJpyVzMdgQIWS?= =?us-ascii?Q?SegQamIF6qvD6f8ty3T0eJAzEhHsrjWHkYb7xcdqAsyMb6oWfnsvxm+3gDIH?= =?us-ascii?Q?nu2FJmi3Gsb5F5ToT32W5XHgJlA9E57YZ6s9DUaXyj4wKGvF+lFWkij7Pp01?= =?us-ascii?Q?EBCc4QngPf42MTk=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:32.3482 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5feff88a-149c-4187-afa1-08dd671ca78a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026367.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6338 Content-Type: text/plain; charset="utf-8" Since we already have the list of mm_struct in the system, add a module to scan each mm that walks VMAs of each mm_struct and scan all the pages associated with that. In the scan path: Check for the recently acccessed pages (folios) belongin= g to slowtier nodes. Add all those folios to a migration list. Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 323 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 321 insertions(+), 2 deletions(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 36d0fea31dea..a76a58bf37b2 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -4,10 +4,18 @@ #include #include #include +#include +#include +#include +#include +#include +#include #include #include #include #include +#include +#include #include #include #include @@ -18,6 +26,11 @@ =20 static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); +/* + * Total VMA size to cover during scan. + */ +#define KMMSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) +static unsigned long kmmscand_scan_size __read_mostly =3D KMMSCAND_SCAN_SI= ZE; =20 /* How long to pause between two scan and migration cycle */ static unsigned int kmmscand_scan_sleep_ms __read_mostly =3D 16; @@ -39,10 +52,14 @@ static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash= , KMMSCAND_SLOT_HASH_BITS =20 static struct kmem_cache *kmmscand_slot_cache __read_mostly; =20 +/* Per mm information collected to control VMA scanning */ struct kmmscand_mm_slot { struct mm_slot slot; + long address; + bool is_scanned; }; =20 +/* Data structure to keep track of current mm under scan */ struct kmmscand_scan { struct list_head mm_head; struct kmmscand_mm_slot *mm_slot; @@ -52,6 +69,33 @@ struct kmmscand_scan kmmscand_scan =3D { .mm_head =3D LIST_HEAD_INIT(kmmscand_scan.mm_head), }; =20 +/* + * Data structure passed to control scanning and also collect + * per memory node information + */ +struct kmmscand_scanctrl { + struct list_head scan_list; + unsigned long address; +}; + +struct kmmscand_scanctrl kmmscand_scanctrl; + +/* Per folio information used for migration */ +struct kmmscand_migrate_info { + struct list_head migrate_node; + struct mm_struct *mm; + struct folio *folio; + unsigned long address; +}; + +static bool kmmscand_eligible_srcnid(int nid) +{ + if (!node_is_toptier(nid)) + return true; + + return false; +} + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -82,15 +126,277 @@ static void kmmscand_wait_work(void) return; } =20 + +static inline bool is_valid_folio(struct folio *folio) +{ + if (!folio || folio_test_unevictable(folio) || !folio_mapped(folio) || + folio_is_zone_device(folio) || folio_likely_mapped_shared(folio)) + return false; + + return true; +} + +static bool folio_idle_clear_pte_refs_one(struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr, + pte_t *ptep) +{ + bool referenced =3D false; + struct mm_struct *mm =3D vma->vm_mm; + pmd_t *pmd =3D pmd_off(mm, addr); + + if (ptep) { + if (ptep_clear_young_notify(vma, addr, ptep)) + referenced =3D true; + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + if (!pmd_present(*pmd)) + WARN_ON_ONCE(1); + if (pmdp_clear_young_notify(vma, addr, pmd)) + referenced =3D true; + } else { + WARN_ON_ONCE(1); + } + + if (referenced) { + folio_clear_idle(folio); + folio_set_young(folio); + } + + return true; +} + +static void page_idle_clear_pte_refs(struct page *page, pte_t *pte, struct= mm_walk *walk) +{ + bool need_lock; + struct folio *folio =3D page_folio(page); + unsigned long address; + + if (!folio_mapped(folio) || !folio_raw_mapping(folio)) + return; + + need_lock =3D !folio_test_anon(folio) || folio_test_ksm(folio); + if (need_lock && !folio_trylock(folio)) + return; + address =3D vma_address(walk->vma, page_pgoff(folio, page), compound_nr(p= age)); + VM_BUG_ON_VMA(address =3D=3D -EFAULT, walk->vma); + folio_idle_clear_pte_refs_one(folio, walk->vma, address, pte); + + if (need_lock) + folio_unlock(folio); +} + +static int hot_vma_idle_pte_entry(pte_t *pte, + unsigned long addr, + unsigned long next, + struct mm_walk *walk) +{ + struct page *page; + struct folio *folio; + struct mm_struct *mm; + struct vm_area_struct *vma; + struct kmmscand_migrate_info *info; + struct kmmscand_scanctrl *scanctrl =3D walk->private; + int srcnid; + + scanctrl->address =3D addr; + pte_t pteval =3D ptep_get(pte); + + if (!pte_present(pteval)) + return 0; + + if (pte_none(pteval)) + return 0; + + vma =3D walk->vma; + mm =3D vma->vm_mm; + + page =3D pte_page(*pte); + + page_idle_clear_pte_refs(page, pte, walk); + + folio =3D page_folio(page); + folio_get(folio); + + if (!is_valid_folio(folio)) { + folio_put(folio); + return 0; + } + srcnid =3D folio_nid(folio); + + + if (!folio_test_lru(folio)) { + folio_put(folio); + return 0; + } + + if (!folio_test_idle(folio) || folio_test_young(folio) || + mmu_notifier_test_young(mm, addr) || + folio_test_referenced(folio) || pte_young(pteval)) { + + /* Do not try to promote pages from regular nodes */ + if (!kmmscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return 0; + } + /* XXX: Leaking memory. TBD: consume info */ + info =3D kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); + if (info && scanctrl) { + + info->mm =3D mm; + info->address =3D addr; + info->folio =3D folio; + + /* No need of lock now */ + list_add_tail(&info->migrate_node, &scanctrl->scan_list); + } + } + + folio_set_idle(folio); + folio_put(folio); + return 0; +} + +static const struct mm_walk_ops hot_vma_set_idle_ops =3D { + .pte_entry =3D hot_vma_idle_pte_entry, + .walk_lock =3D PGWALK_RDLOCK, +}; + +static void kmmscand_walk_page_vma(struct vm_area_struct *vma, struct kmms= cand_scanctrl *scanctrl) +{ + if (!vma_migratable(vma) || !vma_policy_mof(vma) || + is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + return; + } + if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) =3D=3D (VM_READ= ))) + return; + + if (!vma_is_accessible(vma)) + return; + + walk_page_vma(vma, &hot_vma_set_idle_ops, scanctrl); +} + static inline int kmmscand_test_exit(struct mm_struct *mm) { return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) +{ + struct mm_slot *slot =3D &mm_slot->slot; + struct mm_struct *mm =3D slot->mm; + + lockdep_assert_held(&kmmscand_mm_lock); + + if (kmmscand_test_exit(mm)) { + /* free mm_slot */ + hash_del(&slot->hash); + list_del(&slot->mm_node); + + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } +} + static unsigned long kmmscand_scan_mm_slot(void) { - /* placeholder for scanning */ - msleep(100); + bool next_mm =3D false; + bool update_mmslot_info =3D false; + + unsigned long vma_scanned_size =3D 0; + unsigned long address; + + struct mm_slot *slot; + struct mm_struct *mm; + struct vm_area_struct *vma =3D NULL; + struct kmmscand_mm_slot *mm_slot; + + /* Retrieve mm */ + spin_lock(&kmmscand_mm_lock); + + if (kmmscand_scan.mm_slot) { + mm_slot =3D kmmscand_scan.mm_slot; + slot =3D &mm_slot->slot; + address =3D mm_slot->address; + } else { + slot =3D list_entry(kmmscand_scan.mm_head.next, + struct mm_slot, mm_node); + mm_slot =3D mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + address =3D mm_slot->address; + kmmscand_scan.mm_slot =3D mm_slot; + } + + mm =3D slot->mm; + mm_slot->is_scanned =3D true; + spin_unlock(&kmmscand_mm_lock); + + if (unlikely(!mmap_read_trylock(mm))) + goto outerloop_mmap_lock; + + if (unlikely(kmmscand_test_exit(mm))) { + next_mm =3D true; + goto outerloop; + } + + VMA_ITERATOR(vmi, mm, address); + + for_each_vma(vmi, vma) { + kmmscand_walk_page_vma(vma, &kmmscand_scanctrl); + vma_scanned_size +=3D vma->vm_end - vma->vm_start; + + if (vma_scanned_size >=3D kmmscand_scan_size) { + next_mm =3D true; + /* TBD: Add scanned folios to migration list */ + break; + } + } + + if (!vma) + address =3D 0; + else + address =3D kmmscand_scanctrl.address + PAGE_SIZE; + + update_mmslot_info =3D true; + + if (update_mmslot_info) + mm_slot->address =3D address; + +outerloop: + /* exit_mmap will destroy ptes after this */ + mmap_read_unlock(mm); + +outerloop_mmap_lock: + spin_lock(&kmmscand_mm_lock); + WARN_ON(kmmscand_scan.mm_slot !=3D mm_slot); + + /* + * Release the current mm_slot if this mm is about to die, or + * if we scanned all vmas of this mm. + */ + if (unlikely(kmmscand_test_exit(mm)) || !vma || next_mm) { + /* + * Make sure that if mm_users is reaching zero while + * kmmscand runs here, kmmscand_exit will find + * mm_slot not pointing to the exiting mm. + */ + if (slot->mm_node.next !=3D &kmmscand_scan.mm_head) { + slot =3D list_entry(slot->mm_node.next, + struct mm_slot, mm_node); + kmmscand_scan.mm_slot =3D + mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + + } else + kmmscand_scan.mm_slot =3D NULL; + + if (kmmscand_test_exit(mm)) { + kmmscand_collect_mm_slot(mm_slot); + goto end; + } + } + mm_slot->is_scanned =3D false; +end: + spin_unlock(&kmmscand_mm_lock); return 0; } =20 @@ -153,6 +459,7 @@ void __kmmscand_enter(struct mm_struct *mm) if (!kmmscand_slot) return; =20 + kmmscand_slot->address =3D 0; slot =3D &kmmscand_slot->slot; =20 spin_lock(&kmmscand_mm_lock); @@ -180,6 +487,12 @@ void __kmmscand_exit(struct mm_struct *mm) hash_del(&slot->hash); list_del(&slot->mm_node); free =3D 1; + } else if (mm_slot && kmmscand_scan.mm_slot =3D=3D mm_slot && !mm_slot->i= s_scanned) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + /* TBD: Set the actual next slot */ + kmmscand_scan.mm_slot =3D NULL; } =20 spin_unlock(&kmmscand_mm_lock); @@ -233,6 +546,11 @@ static int stop_kmmscand(void) =20 return err; } +static void init_list(void) +{ + INIT_LIST_HEAD(&kmmscand_scanctrl.scan_list); + init_waitqueue_head(&kmmscand_wait); +} =20 static int __init kmmscand_init(void) { @@ -245,6 +563,7 @@ static int __init kmmscand_init(void) return -ENOMEM; } =20 + init_list(); err =3D start_kmmscand(); if (err) goto err_kmmscand; --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2073.outbound.protection.outlook.com [40.107.94.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83578219319 for ; Wed, 19 Mar 2025 19:31:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.73 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412709; cv=fail; b=FCHTKp7Hh5OB/M3sxpJeTpdCcUVKsj7ELnslxtPRb19O9tlth5/FwdwAHuUKCTSuOHO0WxZTES+6q3oQqjlbS7BUroPhQJiuP5QWBfJlEcqWh8XDRFlQXnmay2AOng0ZtQm17t0d0UO3U9Gflgf6mfScMxRwxm7I8IfrdC/8lt0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412709; c=relaxed/simple; bh=CwACzvsXFGQ+4o9JzFM3SXvkdZnCjvPpRstRwwmveA8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=UzGb61GXa6u/DVpWwvcdii6y6omAd0k90qLG1EOGpo91f86VS6lG35Kac8AhXkj/uOuTZmXrLuNGqDJp+E5Zvw8DQzqsPXHnymBfs0hFJABxVYYrh7LDNw3SenfXf8VYl8Kk3O9H7OwcVeDzyhKtKcDNYdYv6lhZ6T1z8bAkr7w= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=4wesvsZf; arc=fail smtp.client-ip=40.107.94.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="4wesvsZf" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iNV/yleleGLbHG0RUop+A6UkniDrrRy4IYpeAMMS1V26DgPmancCSx1fIGiFlpLCwq0wDW4/nZ589zth+1/6zLdCGVAaiNvNVgD2Ct/bjiJBDxOITTtbluRfQguIYq+Vn6i8mEjjXnsLiJL9hopeY9/t3a+69XPmJE2CNOIGUdRk0tRD6yzDt3SXpXd3wp+/3gEq37aq7yjSCXieVoDCq9ojR1scYxpKC64OM2ZRgo3UmuKn/k3AI/g4ty/56taYMmmLfWQPmrLWkZRqQB/EvMu588XKAUfVW0IaJl79y5Mxfk0IuRcC9Q/ggLIe6XQhOAb2UQFP9nFhPFU7ewtzcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PtVePGSIcc5EsBIhdgHjgHCwmTFiPZreK0dOtW/EQes=; b=HMvq2dy2xMZCHrc+bBpn6UEUdOENKD2M6KFTMLrNOErKVW2Fwg8RksAWVO5xwCVKApbmUHX26RcDsU7aMoFgNTsVSjw6LcdmfB9DB3TE8YpyKfJisWttbENZGH+TBe5sN98z9cov79OCzunvlp5hqnuCTJoCk8EibQm+KebhmLvpuxU9Lh4RSMvk+IGIEQJPjDir7VtePflvctiAKcKLWI1WxiAnElV454tjrDhwRbkOex5jvlq6gR1Qt9WXMsHPZMI5iefcqDcHMo/Y0lY0BSDHNpk/i7/VeiSsB7fsVhX4bSFEGPalwgx/NZspa9mZXY12GQc/puyfHT6wz5nVIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PtVePGSIcc5EsBIhdgHjgHCwmTFiPZreK0dOtW/EQes=; b=4wesvsZfUuHPArobaYmr9axYEevL+wv5xx5S0HYFwRo7xevuL69iBJsp4IzaXSk4tmvBSBwlhyGzIfqWxwtTD37Kyuy2fSvQMap3CJWl/6cJAu11V3j/EnAhnm1q4dnxmgWep6iUEiSb2j8cLT2XlVTqsPoLuPEgbA4ol7bwZls= Received: from SA9PR10CA0006.namprd10.prod.outlook.com (2603:10b6:806:a7::11) by SA0PR12MB7091.namprd12.prod.outlook.com (2603:10b6:806:2d5::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:31:43 +0000 Received: from SN1PEPF0002636E.namprd02.prod.outlook.com (2603:10b6:806:a7:cafe::25) by SA9PR10CA0006.outlook.office365.com (2603:10b6:806:a7::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636E.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:42 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:32 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 04/13] mm: Create a separate kernel thread for migration Date: Wed, 19 Mar 2025 19:30:19 +0000 Message-ID: <20250319193028.29514-5-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636E:EE_|SA0PR12MB7091:EE_ X-MS-Office365-Filtering-Correlation-Id: 03dff38e-7350-4ae2-5724-08dd671caddd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?sF/pEHEPvPGRjshqRhWXUqdKI7+oW+GK9tWDSUXoqK4GeHHdDo3fs41nG5pG?= =?us-ascii?Q?ha3aua8nSmYi6OPLyk9yV965JdaVIpfe7wImESSkxK/eZFr9HQ89FJP3cNtd?= =?us-ascii?Q?XktD0GpVFcX4qcLfwgDTI6LR/UK6PE1SOJxf2qAU1DgsBnHrnK/r4Gt2u/B+?= =?us-ascii?Q?uM0NqPF9l9BUadoqM0IGSQACUjX9OP7MbtgW5+zlMFOXp5jviFl0KY7i7Mzx?= =?us-ascii?Q?mDmVh+LSr29nEaDOMsMb900FDIBYO4RQMcqyrYVJl/AQnZcasqxjsrDf0oQu?= =?us-ascii?Q?e1xO3B8xwGt+GHq5vowy8fSlYPtYi3o44N5HjZptHTkqu9sDwzTwtJfpXyck?= =?us-ascii?Q?i2rXJC17KRmJkf16HLABgtb/Jw6MghAcdIjfuGHK29HPOVVzaXWXtI1/8OAN?= =?us-ascii?Q?uTv/BY5Ra6xEYZ7HEtvgik0qfSvwPlalush4CkkLfVPVkZQSO38ElxEcOzuW?= =?us-ascii?Q?Smb2sBLsTRHu+39WZj89gs+J9mwnMWUUg0uWUNEDIXq//3zF1x67pojvQUY0?= =?us-ascii?Q?xmcU2AHZs3t3f4ZAzvPyVNsG+66PUJ2Jn4JGa5QhScbdhUG3e07LtWMj29Nb?= =?us-ascii?Q?pH5IWTMX7M8qVvAHb/BtzUE+0DA4w8DhmUGiyQ8qFMcxYzZKBDTjE+BfSMxk?= =?us-ascii?Q?qXZDRkVaEZg294uANsfECkfZzEEu/Ndbq1lh9rXprjSvBQPnFZfOrfR3fvj3?= =?us-ascii?Q?vdLQP7650+xPUGyojGpxJ5LkiPDagCvjkp8L+l3ywbnvs/NOixLPgFQN0I/l?= =?us-ascii?Q?E/OcLzvKvfjihv/3F/N1ewMZq3Bx2MSjZR8vHxuvd1esqVlC6bnI9bpuuQci?= =?us-ascii?Q?4PmRzm2AFTEbZ/JuyyXQKcnYWU2uyWtgbSTcD9mtQkbALM4pOot/mQ6W/fYp?= =?us-ascii?Q?9f89ARQnfGgwy4J87vHyPFdJH5Dqs3ZY9TDdGmcEj6KXeGad16RxDE10artQ?= =?us-ascii?Q?eNZ5e565XFTgTU1WvEgKWsdquSIIq56PsUbKDsR/jVN0GrVkrx0nEP8ao4c6?= =?us-ascii?Q?j9+W5rgAmBIkIB2Rp3PaXFqmdhWrwNtgGc/Cmv9Mn4i5ySGsWj2tA+vrBRO/?= =?us-ascii?Q?Tk+xU/BTpc/CCmzzhpBYIvsgzVTsrtlF2lCFP5eMnKVAUHTRifePZQv007rB?= =?us-ascii?Q?YMugAu9cBbM8JoXbxvT2d1z9eOtlgJPVsxlQDcbZs7fufZt8nwoupQ9/cQ+O?= =?us-ascii?Q?61VyEaGRnRJ7eU9MfwbfXPn9Q2kUYF+pPjrXMq7OYzNMl6H7LO2uflNkTNOB?= =?us-ascii?Q?GpLh7o9wp9jv65GYqWOJePcAPCEShZA6hI2cyil3N/WiE5fFhFl6xatsKBhv?= =?us-ascii?Q?y1EuEsXKVNk0simDenz8ttaNS9MGDJ1gjM/J0lwu74vsHivRLTDpOq2F1lkV?= =?us-ascii?Q?czJflnwf9o8Tm2VG7ePNOsIYdpKJq2SyLPo1P16ou0Jw01cdZW+UztqLdAUA?= =?us-ascii?Q?UxnwuWmIuZ0ztW0d2Oaq/lXx5RZSD1HcvrWr+1TGsyiUSXu7M3QQYUrI8q76?= =?us-ascii?Q?xDDWdnW289XYLVU=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:42.9406 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 03dff38e-7350-4ae2-5724-08dd671caddd X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7091 Content-Type: text/plain; charset="utf-8" Having independent thread helps in: - Alleviating the need for multiple scanning threads - Aids to control batch migration (TBD) - Migration throttling (TBD) Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 157 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 154 insertions(+), 3 deletions(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index a76a58bf37b2..6e96cfab5b85 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -41,10 +42,26 @@ static unsigned long kmmscand_mms_to_scan __read_mostly= =3D KMMSCAND_MMS_TO_SCAN; =20 bool kmmscand_scan_enabled =3D true; static bool need_wakeup; +static bool migrated_need_wakeup; + +/* How long to pause between two migration cycles */ +static unsigned int kmmmigrate_sleep_ms __read_mostly =3D 20; + +static struct task_struct *kmmmigrated_thread __read_mostly; +static DEFINE_MUTEX(kmmmigrated_mutex); +static DECLARE_WAIT_QUEUE_HEAD(kmmmigrated_wait); +static unsigned long kmmmigrated_sleep_expire; + +/* mm of the migrating folio entry */ +static struct mm_struct *kmmscand_cur_migrate_mm; + +/* Migration list is manipulated underneath because of mm_exit */ +static bool kmmscand_migration_list_dirty; =20 static unsigned long kmmscand_sleep_expire; =20 static DEFINE_SPINLOCK(kmmscand_mm_lock); +static DEFINE_SPINLOCK(kmmscand_migrate_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); =20 #define KMMSCAND_SLOT_HASH_BITS 10 @@ -80,6 +97,14 @@ struct kmmscand_scanctrl { =20 struct kmmscand_scanctrl kmmscand_scanctrl; =20 +struct kmmscand_migrate_list { + struct list_head migrate_head; +}; + +struct kmmscand_migrate_list kmmscand_migrate_list =3D { + .migrate_head =3D LIST_HEAD_INIT(kmmscand_migrate_list.migrate_head), +}; + /* Per folio information used for migration */ struct kmmscand_migrate_info { struct list_head migrate_node; @@ -101,6 +126,13 @@ static int kmmscand_has_work(void) return !list_empty(&kmmscand_scan.mm_head); } =20 +static int kmmmigrated_has_work(void) +{ + if (!list_empty(&kmmscand_migrate_list.migrate_head)) + return true; + return false; +} + static bool kmmscand_should_wakeup(void) { bool wakeup =3D kthread_should_stop() || need_wakeup || @@ -111,6 +143,16 @@ static bool kmmscand_should_wakeup(void) return wakeup; } =20 +static bool kmmmigrated_should_wakeup(void) +{ + bool wakeup =3D kthread_should_stop() || migrated_need_wakeup || + time_after_eq(jiffies, kmmmigrated_sleep_expire); + if (migrated_need_wakeup) + migrated_need_wakeup =3D false; + + return wakeup; +} + static void kmmscand_wait_work(void) { const unsigned long scan_sleep_jiffies =3D @@ -126,6 +168,19 @@ static void kmmscand_wait_work(void) return; } =20 +static void kmmmigrated_wait_work(void) +{ + const unsigned long migrate_sleep_jiffies =3D + msecs_to_jiffies(kmmmigrate_sleep_ms); + + if (!migrate_sleep_jiffies) + return; + + kmmmigrated_sleep_expire =3D jiffies + migrate_sleep_jiffies; + wait_event_timeout(kmmmigrated_wait, + kmmmigrated_should_wakeup(), + migrate_sleep_jiffies); +} =20 static inline bool is_valid_folio(struct folio *folio) { @@ -238,7 +293,6 @@ static int hot_vma_idle_pte_entry(pte_t *pte, folio_put(folio); return 0; } - /* XXX: Leaking memory. TBD: consume info */ info =3D kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { =20 @@ -282,6 +336,28 @@ static inline int kmmscand_test_exit(struct mm_struct = *mm) return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +static void kmmscand_cleanup_migration_list(struct mm_struct *mm) +{ + struct kmmscand_migrate_info *info, *tmp; + + spin_lock(&kmmscand_migrate_lock); + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + if (mm =3D=3D READ_ONCE(kmmscand_cur_migrate_mm)) { + /* A folio in this mm is being migrated. wait */ + WRITE_ONCE(kmmscand_migration_list_dirty, true); + } + + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (info && (info->mm =3D=3D mm)) { + info->mm =3D NULL; + WRITE_ONCE(kmmscand_migration_list_dirty, true); + } + } + } + spin_unlock(&kmmscand_migrate_lock); +} + static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) { struct mm_slot *slot =3D &mm_slot->slot; @@ -294,11 +370,17 @@ static void kmmscand_collect_mm_slot(struct kmmscand_= mm_slot *mm_slot) hash_del(&slot->hash); list_del(&slot->mm_node); =20 + kmmscand_cleanup_migration_list(mm); + mm_slot_free(kmmscand_slot_cache, mm_slot); mmdrop(mm); } } =20 +static void kmmscand_migrate_folio(void) +{ +} + static unsigned long kmmscand_scan_mm_slot(void) { bool next_mm =3D false; @@ -347,9 +429,17 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 if (vma_scanned_size >=3D kmmscand_scan_size) { next_mm =3D true; - /* TBD: Add scanned folios to migration list */ + /* Add scanned folios to migration list */ + spin_lock(&kmmscand_migrate_lock); + list_splice_tail_init(&kmmscand_scanctrl.scan_list, + &kmmscand_migrate_list.migrate_head); + spin_unlock(&kmmscand_migrate_lock); break; } + spin_lock(&kmmscand_migrate_lock); + list_splice_tail_init(&kmmscand_scanctrl.scan_list, + &kmmscand_migrate_list.migrate_head); + spin_unlock(&kmmscand_migrate_lock); } =20 if (!vma) @@ -478,7 +568,7 @@ void __kmmscand_exit(struct mm_struct *mm) { struct kmmscand_mm_slot *mm_slot; struct mm_slot *slot; - int free =3D 0; + int free =3D 0, serialize =3D 1; =20 spin_lock(&kmmscand_mm_lock); slot =3D mm_slot_lookup(kmmscand_slots_hash, mm); @@ -493,10 +583,15 @@ void __kmmscand_exit(struct mm_struct *mm) free =3D 1; /* TBD: Set the actual next slot */ kmmscand_scan.mm_slot =3D NULL; + } else if (mm_slot && kmmscand_scan.mm_slot =3D=3D mm_slot && mm_slot->is= _scanned) { + serialize =3D 0; } =20 spin_unlock(&kmmscand_mm_lock); =20 + if (serialize) + kmmscand_cleanup_migration_list(mm); + if (free) { mm_slot_free(kmmscand_slot_cache, mm_slot); mmdrop(mm); @@ -546,10 +641,59 @@ static int stop_kmmscand(void) =20 return err; } +static int kmmmigrated(void *arg) +{ + for (;;) { + WRITE_ONCE(migrated_need_wakeup, false); + if (unlikely(kthread_should_stop())) + break; + if (kmmmigrated_has_work()) + kmmscand_migrate_folio(); + msleep(20); + kmmmigrated_wait_work(); + } + return 0; +} + +static int start_kmmmigrated(void) +{ + int err =3D 0; + + guard(mutex)(&kmmmigrated_mutex); + + /* Someone already succeeded in starting daemon */ + if (kmmmigrated_thread) + goto end; + + kmmmigrated_thread =3D kthread_run(kmmmigrated, NULL, "kmmmigrated"); + if (IS_ERR(kmmmigrated_thread)) { + pr_err("kmmmigrated: kthread_run(kmmmigrated) failed\n"); + err =3D PTR_ERR(kmmmigrated_thread); + kmmmigrated_thread =3D NULL; + goto end; + } else { + pr_info("kmmmigrated: Successfully started kmmmigrated"); + } + + wake_up_interruptible(&kmmmigrated_wait); +end: + return err; +} + +static int stop_kmmmigrated(void) +{ + guard(mutex)(&kmmmigrated_mutex); + kthread_stop(kmmmigrated_thread); + return 0; +} + static void init_list(void) { + INIT_LIST_HEAD(&kmmscand_migrate_list.migrate_head); INIT_LIST_HEAD(&kmmscand_scanctrl.scan_list); + spin_lock_init(&kmmscand_migrate_lock); init_waitqueue_head(&kmmscand_wait); + init_waitqueue_head(&kmmmigrated_wait); } =20 static int __init kmmscand_init(void) @@ -568,8 +712,15 @@ static int __init kmmscand_init(void) if (err) goto err_kmmscand; =20 + err =3D start_kmmmigrated(); + if (err) + goto err_kmmmigrated; + return 0; =20 +err_kmmmigrated: + stop_kmmmigrated(); + err_kmmscand: stop_kmmscand(); kmmscand_destroy(); --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2046.outbound.protection.outlook.com [40.107.93.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4484F2144A6 for ; Wed, 19 Mar 2025 19:32:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412723; cv=fail; b=j3ou4Yc0K3H3EF/7lkmmAJ6AQ7isSMXrYNmsJVBIHICmEtqF/Nt+tRawAst0PUjIirnHCt4OcQRZP7huZEPe+on8orzXqUxGt11RJNiScRl3F8g9rxAX0TpYvEdD78gRXvF0Lv5CNEhwXiCLRIY7g13BVRhxKMqB7uXJFbVFl0I= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412723; c=relaxed/simple; bh=0AfJvqsXjDyDoCMdLjEMOrEjW8bwNMPcCAxf81HbnUo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=q2ZNNyUF6EQ2WTkoARsST+MUqNDdx57U+OiQwBQs40sRwJN47J58U4PGqC4urlkPWE8cCIXrBzAfrh6gOiE/cNECDH/CFuRj/5Jn9B22vae/SZNqkF30XN7SW0LxM7EmZT7+IA8kA93/SwSj3M72p+xofNAbF98UyF1jqaFpNt0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=GN6uGhX0; arc=fail smtp.client-ip=40.107.93.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="GN6uGhX0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ppJDLvpTagOkHNCEnoyM3sxIhBOiWvGelk244CtIPpPkQfYgXNqAMyMORKnNsIfTKwN/85jkQce5NO0VL2dKM/T4naJeRCVs74MVcrZRpCVujBxpbUsr+cHfcke5FtJHedZM8sNliJ9e9eGZko8LJK/cQbTmiL15VFkGqo6mqGEzdGr1ooKe6+a5N3P02abp70iagIyNqoFdx6XLmJggBe4hDrCf9li7xn90YAAuD7Ui14QuqYVjCLJdk60BfImO6p3kuOLNZKqoU5SgXUq8ZI5NlvKCRZQ2IlqEDKjnMV3x5C8f6t+4vqSsgR4DmFBeHXIJ49lyL0ipJTwGt6obBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hsLJE/h2f3iBqJCSh/p9SqyDaJnSWoeY2MYMRp9SgIE=; b=dMtZIz2+VzWo0AZ/R1zgTPHh3Nne99eI4Pp137HEAi3I54/CSMU/0m0Sv0HD4xqSRXFk28/4GEiF48tYeyF90O0zJwO8jj0QZwW5fvZ2Nwh30dJFXzWDCU5cSzloWAwvloEzOlYCOogdTUB59Lti1tJ8TnI7dNMtEaAxoQrG2CcMUcqQzLYi41Q84k64vMQRlPwzWilcFuQaFP29giGBupRKum0tA3/QhkpfEVJ1sFnBOBBI1Se48w2tf4t+F/ZvvDkHkcRgaUfxEC2my8bvPZ5Q8rvVebhVo/+EoaLlN9+1Bk8879jelRjy13KnkivD11yqnm/g3VmEadZd3f9aqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hsLJE/h2f3iBqJCSh/p9SqyDaJnSWoeY2MYMRp9SgIE=; b=GN6uGhX0nDlSZjalEwMOu0CfPljZOg/pns9KC6kD1YebA7RJS87FAL3RTa0VPRR8mONbABnMl50az+FyYaw8rowVxjz+sPo4Ujc1iHtJjHHxKND9W1lXbm9d3IwWFVZOEdX4oL0/lZDojuD4FaWl4eQa39QYZc3Klkst42kFhKI= Received: from SN6PR05CA0020.namprd05.prod.outlook.com (2603:10b6:805:de::33) by SJ1PR12MB6290.namprd12.prod.outlook.com (2603:10b6:a03:457::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:31:54 +0000 Received: from SN1PEPF0002636C.namprd02.prod.outlook.com (2603:10b6:805:de:cafe::e3) by SN6PR05CA0020.outlook.office365.com (2603:10b6:805:de::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636C.mail.protection.outlook.com (10.167.241.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:53 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:42 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 05/13] mm/migration: Migrate accessed folios to toptier node Date: Wed, 19 Mar 2025 19:30:20 +0000 Message-ID: <20250319193028.29514-6-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636C:EE_|SJ1PR12MB6290:EE_ X-MS-Office365-Filtering-Correlation-Id: 7f498ec8-74af-44c1-3a40-08dd671cb460 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|36860700013|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?9SwE2bFnse8n/39XTWVI1UMO2afQRlhIF8aCdF5uxb/WfCFINloTJRFd2dpO?= =?us-ascii?Q?N/Pnm32NEMNWd0XwaAEdpe2ReO5rgUb2eNhJ8zt7W3w03P7eAOYza+Pf1adp?= =?us-ascii?Q?ui2Kp5BtHYXpvi0tIYP5SIJLFv5p2CGp3dri6bWmrLecHARzhXFc0ayCICsF?= =?us-ascii?Q?RkHWSYjsELwo8m6J5JoM2sFDoLfJ3RevXdA1NCq57uIYV271IoWa9UL77fD1?= =?us-ascii?Q?w+rJij7FG6fx+jHFoyOKejtF80SBC4ZIgHKU6lqIsi2RL4x6+MyyJ34Ogsrl?= =?us-ascii?Q?TUv6ZyGSyF0b74b0OoJyh5ta46VR0tx8uee1i6pSf/wbozoPIQnO96hObqAH?= =?us-ascii?Q?p5aFebuS0l5PrOdnKPW4zyDW1EZ12yks0CAiVS1VT5Fn79zu4axuKQHisG2o?= =?us-ascii?Q?0jChdLhzK3T44RMTepeP2iiJdzz2DqH27/yZsGtbSX/pLnUc7zbgwN9mNTf1?= =?us-ascii?Q?gds0gnehGeFZkYiEBAC1RbOZrfWHjRkX4odPYJpHSlzYRYtwZxMYkZ+Ln/mL?= =?us-ascii?Q?rKyP1RbiBHM/uJma8Y1b239WRAwzs+eDs+/fNiTiDMrQZG8fHNciezaLvRaP?= =?us-ascii?Q?YUGvXxup3bApBgyK39zX3XYg/PVQS2ok8j9qkgaAg/UM/kvjLmTh6AVMgmlp?= =?us-ascii?Q?azHo7MQ0sBzGbV2sZbCGusaJwPCHJoXDZui0HIOGLNxE5i+eFntn2ijMP1JC?= =?us-ascii?Q?p+d0mJ46Grjzwr4AhhbbBpnJtUjlgycHTSjfTqn8dslsTFunuk6SfkAoHMfg?= =?us-ascii?Q?5Z0XKlcQmS8Zy8dV2+2BHkZj1USdulusZpGZm5tECrJMtTsTCIEKOKhcliw4?= =?us-ascii?Q?610N70Yfxmsz5ZaPmGwPX+RmycQqATZsYrycJzqFWel31bxPgLLihF8SWr/3?= =?us-ascii?Q?OWKxiAjF41pbQkXYZvs7QnOICs6ZVvJhIhKPrvxSXBGUTnwjxBV56D5gB8Yo?= =?us-ascii?Q?Lc/yDwdro0yl46zxKnooyrOstJTvVok863/OKVbOIrHRi/B9zHO0WsZhUlXP?= =?us-ascii?Q?Dz6OxM1T77ht0FWMO3I0kKpKgHvVQ/7LxfZHfDSx+mO36iNnptkeY8WM1gYS?= =?us-ascii?Q?GTwkms5G/hN+YwuK0iFoIcrkLbELnQtxeuLgqd9g+emPzJ8vvYMyPOgPpIwV?= =?us-ascii?Q?cFmGFybBT6GuE2RWh1pVNz1BpO69d3lCWMrrK61T2Lo4bqn3A1JxKed8Obtv?= =?us-ascii?Q?dcDY1QVyXNxJVaTgjX9UL/Eg1Ryg2Ncf5JVd5VPx9A3PCrISJBrtbNRuO/uw?= =?us-ascii?Q?Ba7l2q34mpyC+P5jdh9qCeX+r6PqqkPzoDtBSwz2dC5NPhsJXOfbC5VmRSgN?= =?us-ascii?Q?4se0UYagG8kjwWtPUWbLcGdR+zYxa9SXbwTISPGfs8Buc4UfK1b5G6oz9+Ad?= =?us-ascii?Q?n4yp2+C6ejm5EhL/Hyt6PEen1WYH6ZIAs3D5/9dR3aF+NiA+krFbum/mO0oG?= =?us-ascii?Q?LwZsHtsvj/jRLk3s8TVmwLDxj43aeoG825qGP+xzzpP9Spab8pgy8w=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(36860700013)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:53.8803 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7f498ec8-74af-44c1-3a40-08dd671cb460 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636C.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6290 Content-Type: text/plain; charset="utf-8" For each recently accessed slowtier folio in the migration list: - Isolate LRU pages - Migrate to a regular node. The rationale behind whole migration is to speedup the access to recently accessed pages. Currently, PTE A bit scanning approach lacks information about exact destination node to migrate to. Reason: PROT_NONE hint fault based scanning is done in a process context. Here when the fault occurs, source CPU of the fault associated task is known. Time of page access is also accurate. With the lack of above information, migration is done to node 0 by default. Signed-off-by: Raghavendra K T --- include/linux/migrate.h | 2 + mm/kmmscand.c | 209 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 2 +- 3 files changed, 212 insertions(+), 1 deletion(-) PS: Later in the patch a simple heuristic is used to find the target node diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 29919faea2f1..22abae80cbb7 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -142,6 +142,8 @@ const struct movable_operations *page_movable_ops(struc= t page *page) } =20 #ifdef CONFIG_NUMA_BALANCING +bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 6e96cfab5b85..feca775d0191 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -59,6 +59,8 @@ static struct mm_struct *kmmscand_cur_migrate_mm; static bool kmmscand_migration_list_dirty; =20 static unsigned long kmmscand_sleep_expire; +#define KMMSCAND_DEFAULT_TARGET_NODE (0) +static int kmmscand_target_node =3D KMMSCAND_DEFAULT_TARGET_NODE; =20 static DEFINE_SPINLOCK(kmmscand_mm_lock); static DEFINE_SPINLOCK(kmmscand_migrate_lock); @@ -182,6 +184,76 @@ static void kmmmigrated_wait_work(void) migrate_sleep_jiffies); } =20 +/* + * Do not know what info to pass in the future to make + * decision on taget node. Keep it void * now. + */ +static int kmmscand_get_target_node(void *data) +{ + return kmmscand_target_node; +} + +extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); + +/*XXX: Taken from migrate.c to avoid NUMAB mode=3D2 and NULL vma checks*/ +static int kmmscand_migrate_misplaced_folio_prepare(struct folio *folio, + struct vm_area_struct *vma, int node) +{ + int nr_pages =3D folio_nr_pages(folio); + pg_data_t *pgdat =3D NODE_DATA(node); + + if (folio_is_file_lru(folio)) { + /* + * Do not migrate file folios that are mapped in multiple + * processes with execute permissions as they are probably + * shared libraries. + * + * See folio_likely_mapped_shared() on possible imprecision + * when we cannot easily detect if a folio is shared. + */ + if (vma && (vma->vm_flags & VM_EXEC) && + folio_likely_mapped_shared(folio)) + return -EACCES; + /* + * Do not migrate dirty folios as not all filesystems can move + * dirty folios in MIGRATE_ASYNC mode which is a waste of + * cycles. + */ + if (folio_test_dirty(folio)) + return -EAGAIN; + } + + /* Avoid migrating to a node that is nearly full */ + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + for (z =3D pgdat->nr_zones - 1; z >=3D 0; z--) { + if (managed_zone(pgdat->node_zones + z)) + break; + } + + /* + * If there are no managed zones, it should not proceed + * further. + */ + if (z < 0) + return -EAGAIN; + + wakeup_kswapd(pgdat->node_zones + z, 0, + folio_order(folio), ZONE_MOVABLE); + return -EAGAIN; + } + + if (!folio_isolate_lru(folio)) + return -EAGAIN; + + node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), + nr_pages); + + return 0; +} + static inline bool is_valid_folio(struct folio *folio) { if (!folio || folio_test_unevictable(folio) || !folio_mapped(folio) || @@ -191,6 +263,101 @@ static inline bool is_valid_folio(struct folio *folio) return true; } =20 +enum kmmscand_migration_err { + KMMSCAND_NULL_MM =3D 1, + KMMSCAND_EXITING_MM, + KMMSCAND_INVALID_FOLIO, + KMMSCAND_NONLRU_FOLIO, + KMMSCAND_INELIGIBLE_SRC_NODE, + KMMSCAND_SAME_SRC_DEST_NODE, + KMMSCAND_PTE_NOT_PRESENT, + KMMSCAND_PMD_NOT_PRESENT, + KMMSCAND_NO_PTE_OFFSET_MAP_LOCK, + KMMSCAND_LRU_ISOLATION_ERR, +}; + +static int kmmscand_promote_folio(struct kmmscand_migrate_info *info, int = destnid) +{ + unsigned long pfn; + unsigned long address; + struct page *page; + struct folio *folio; + int ret; + struct mm_struct *mm; + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; + pmd_t pmde; + int srcnid; + + if (info->mm =3D=3D NULL) + return KMMSCAND_NULL_MM; + + if (info->mm =3D=3D READ_ONCE(kmmscand_cur_migrate_mm) && + READ_ONCE(kmmscand_migration_list_dirty)) { + WARN_ON_ONCE(mm); + return KMMSCAND_EXITING_MM; + } + + mm =3D info->mm; + folio =3D info->folio; + + /* Check again if the folio is really valid now */ + if (folio) { + pfn =3D folio_pfn(folio); + page =3D pfn_to_online_page(pfn); + } + + if (!page || PageTail(page) || !is_valid_folio(folio)) + return KMMSCAND_INVALID_FOLIO; + + if (!folio_test_lru(folio)) + return KMMSCAND_NONLRU_FOLIO; + + folio_get(folio); + + srcnid =3D folio_nid(folio); + + /* Do not try to promote pages from regular nodes */ + if (!kmmscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return KMMSCAND_INELIGIBLE_SRC_NODE; + } + + /* Also happen when it is already migrated */ + if (srcnid =3D=3D destnid) { + folio_put(folio); + return KMMSCAND_SAME_SRC_DEST_NODE; + } + address =3D info->address; + pmd =3D pmd_off(mm, address); + pmde =3D pmdp_get(pmd); + + if (!pmd_present(pmde)) { + folio_put(folio); + return KMMSCAND_PMD_NOT_PRESENT; + } + + pte =3D pte_offset_map_lock(mm, pmd, address, &ptl); + if (!pte) { + folio_put(folio); + WARN_ON_ONCE(!pte); + return KMMSCAND_NO_PTE_OFFSET_MAP_LOCK; + } + + ret =3D kmmscand_migrate_misplaced_folio_prepare(folio, NULL, destnid); + if (ret) { + folio_put(folio); + pte_unmap_unlock(pte, ptl); + return KMMSCAND_LRU_ISOLATION_ERR; + } + + folio_put(folio); + pte_unmap_unlock(pte, ptl); + + return migrate_misplaced_folio(folio, destnid); +} + static bool folio_idle_clear_pte_refs_one(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, @@ -379,6 +546,48 @@ static void kmmscand_collect_mm_slot(struct kmmscand_m= m_slot *mm_slot) =20 static void kmmscand_migrate_folio(void) { + int ret =3D 0, dest =3D -1; + struct kmmscand_migrate_info *info, *tmp; + + spin_lock(&kmmscand_migrate_lock); + + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (READ_ONCE(kmmscand_migration_list_dirty)) { + kmmscand_migration_list_dirty =3D false; + list_del(&info->migrate_node); + /* + * Do not try to migrate this entry because mm might have + * vanished underneath. + */ + kfree(info); + spin_unlock(&kmmscand_migrate_lock); + goto dirty_list_handled; + } + + list_del(&info->migrate_node); + /* Note down the mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, info->mm); + spin_unlock(&kmmscand_migrate_lock); + + if (info->mm) { + dest =3D kmmscand_get_target_node(NULL); + ret =3D kmmscand_promote_folio(info, dest); + } + + kfree(info); + + spin_lock(&kmmscand_migrate_lock); + /* Reset mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, NULL); + spin_unlock(&kmmscand_migrate_lock); +dirty_list_handled: + cond_resched(); + spin_lock(&kmmscand_migrate_lock); + } + } + spin_unlock(&kmmscand_migrate_lock); } =20 static unsigned long kmmscand_scan_mm_slot(void) diff --git a/mm/migrate.c b/mm/migrate.c index fb19a18892c8..a073eb6c5009 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2598,7 +2598,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long= , nr_pages, * Returns true if this is a safe migration target node for misplaced NUMA * pages. Currently it only checks the watermarks which is crude. */ -static bool migrate_balanced_pgdat(struct pglist_data *pgdat, +bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages) { int z; --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2056.outbound.protection.outlook.com [40.107.236.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E2AD214A97 for ; Wed, 19 Mar 2025 19:32:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.56 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412729; cv=fail; b=ZnAS5WF17ATbozhhZa1u1QSxMhHhMh1qkekmwVSCYwN9QuNYBExX5n1tiVVrm6JGPo+1xhHBySfQsR2drLjVHbJnocG5uQNnRyEoD9azRl7I7Dc+UNuWr1amlewGFwsuC4AnztgvtTXNjkN8stErWNw65o/dpN/z16xhEGzWd50= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412729; c=relaxed/simple; bh=6AGBjKenBHAhwaYNC+MD2M8hWUUxdKb9TBugHzpmkqs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=O2GOKak6qu0LOq91e0KvB4CL2LV8bJcSW6KZdQeO4AqJnlYFZ/uQ1MXcRkn+YlFNGBj/r8L6Ag/LHikoiC1enGlID9y+iueq2iUnJkwA5lSJXH8mbChW6AETQW3+RPL69z9S5m2n5RM6Ln6ML+PD/cEwFhrCSxOj0MYndKTVEc8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=sa7knc7K; arc=fail smtp.client-ip=40.107.236.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="sa7knc7K" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ec0JQ5oTyYwgXEw+UGkjBJk8LeoTOfpWlqoggoTT9BJdYZyUleAU5GOsjqlnq+q29BGAW7be9IsxfcwsvAzT30rbrasSungoUvFusfzZB3OwQdjJX11cBGuvRUafJ1r19O93eOcflrWqpWNOCcAQSGvrAbz1zrY+qBZ3KQ19hvlTrE1lBMTHBly7ADA+gu/9NRfO7BJkgKCGP/o0W05ymuPPIyKxJers+GynWtNr+iKGKiTpGEedgu/W865YET31Pz+kQkjn+obxkbDBXkIfpsPXjthRD0HsV/LL33CpI8B+0vqIgIYjju/Te2Jud3C49SpBOvlsXoyaF1CGIqtRMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4wGhb7LYtxUiwXUG0HCYGUwtr/i49pZpHiAdkz0j43g=; b=LKqQSltTEfcAvDtNKc2gmntGYa/Z8RWPKgrEkNhw8ye0SLOjFW7IIORd4n3dPF2xHpfukvH+5OlR5l4DtxUTzISiVSZkarMQ1n8MY3i8Qlsw7uyZKsULUTEXY89DpmcIeEerWJfS/H4C8u8WK7Q2tSADVwij47zoAxhbHcof6talKy+PYbxjgiX4GRmuXebWhkFy4fCfDcQ9OIel1wOwStA1k1wg/AO3lGeNPIcIvAK8UPnTSZ/cBryvJ9parg4rSQNZJALxbnS8dANxFBrz3UGqUJuFWIVfIi/qR6IMjDTBDcYgVltWAbAWK5fZ+9+XKnfL4URteT6JgKX52bOBLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4wGhb7LYtxUiwXUG0HCYGUwtr/i49pZpHiAdkz0j43g=; b=sa7knc7KDjngonBEJXlqBh6UWjTsT87vDOGx9x/GK9LEtwCW3XN4QoGNKjmW062H0cbNtA2WLNFHx0l5CpE783xKbngoVJOIx2H6/KtP5lZDhJWtL+qJGLYO3sB7qUPEEOeIRmqOmPdPXW6wXq/zxBlXYQhMmAJw5Mj9Ex+UEBs= Received: from SA9PR13CA0118.namprd13.prod.outlook.com (2603:10b6:806:24::33) by SN7PR12MB7370.namprd12.prod.outlook.com (2603:10b6:806:299::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:32:02 +0000 Received: from SN1PEPF0002636D.namprd02.prod.outlook.com (2603:10b6:806:24:cafe::21) by SA9PR13CA0118.outlook.office365.com (2603:10b6:806:24::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:32:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636D.mail.protection.outlook.com (10.167.241.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:01 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:52 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 06/13] mm: Add throttling of mm scanning using scan_period Date: Wed, 19 Mar 2025 19:30:21 +0000 Message-ID: <20250319193028.29514-7-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636D:EE_|SN7PR12MB7370:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d38bf76-7eaa-4fd3-927a-08dd671cb923 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?EZMw0dXOIE1FJAwUi3ajEr3mFpLDxIn4VgVVCm5R4kyu/DrBTEZyjMekcC2b?= =?us-ascii?Q?hQAEauWrNTLX2108tSbLMDDAaMD/AcjFCB2hmRdnctwE9UeyAsaUgOHxq8OG?= =?us-ascii?Q?8NAEkJ2UlSbmU3lVFKzD0hv9WqBDX4KRg+VA5LfYiVV3HL5saMlDUHW6+Ir8?= =?us-ascii?Q?/boPiGN6NI92qriPAEavNWrYnvZrb6rlSXw86cABtkmxaNKcdugbvcGtuah8?= =?us-ascii?Q?GNKz54tyUNvtOcRNk9ha5WgcTc9f1A0Zy2U6Z+CQq+Sm98V5Ks+CQmyRQ9BG?= =?us-ascii?Q?ZFWUEAiiYSit2oYBJEbQMqH+O3RV0XZVE/pW+B9pPTD9nF15wNodahVpbDAF?= =?us-ascii?Q?xEDFG4Be29ir0NJ9VYHgPcS4BaXCxEOWgVAS2zw5wVTsVk2tq6pjALwd7Qpn?= =?us-ascii?Q?f8NQVQ41kITpMm5Rd0jx4bG87lo1QToTvR56b82DOxFFhNDCe9WrCbN8qH2D?= =?us-ascii?Q?9CuTifXTglOr2/cPalvYswuybkevKY4EoL2U22YWgnmsRxOVj4fSXxPtBDzZ?= =?us-ascii?Q?aGg5M3GAEimkSwRQCNmui6UlvyFrZ1yU+fGbdAPQl8Grh0iJtwzoE4xKcxmQ?= =?us-ascii?Q?0IhMXlMhdC0RMBLLVuiANTvd3i/JncY/8dfS+lwjKgcc0N5aKur5fh3czrxr?= =?us-ascii?Q?aHN7zE3G+KVFoeI5QH3UltTDd1Zuha1bnziDN/3UAWabbX2fkf/W7/MGRVMf?= =?us-ascii?Q?8z/RxPA0s9Faas5+fYyFzY8xhUhsC5mLHg+fZIUIVlpYSyzlrExj3v5SYQqO?= =?us-ascii?Q?ktpruj7RXclvFfmzmifmiko1mnONk4DSJSOOYtu46jOlYP8ZtXL5Mh60R7dQ?= =?us-ascii?Q?yqhDLjP9YbUvLzalXv1VjTwuSeuxpGfe62Q+b5pIN7zsLOjosUwEui65ThZa?= =?us-ascii?Q?xcXvQY9AZohdePwr71eJL1O1iWOTDYVUb6p5y2Y8i47CcFmq/bUcimXfX9db?= =?us-ascii?Q?14aoHo1ByqeSBX32NTh9rTqFO5x8anVp00+heLvh2nSHBP2F/JlBzQc0d9f3?= =?us-ascii?Q?cN8tgfxGC37nMIUXrwegakiGRU6ljaBkmBAr5sAyYP5UvxWDQIKLKfauHgyo?= =?us-ascii?Q?ZXrw6L6rw7W2PygQrQ6fG4ovFqhsnKHtZ3uVXljIfoL+q8/P0SSBZ+cfKM3r?= =?us-ascii?Q?IUja3uJtpXoMCzkL3mE67a6blvFu3JwttdMBJs6taw5ki8cVgsdtQMESUVcB?= =?us-ascii?Q?82OPvr2VrnbVfwag5y4KqvJ2pxgHkfzjxdPeWWu+AfaII19kkTO/K4LwCgxZ?= =?us-ascii?Q?gvfD7GYwvAyNF7s23Z/wDzqxRkO+DXRijRiGjikOcE/QgnXifpDHJivBhp2p?= =?us-ascii?Q?1TLIq4I2tWXAgoxReMC6C0KqfytzWxMKawRYTll/hcN44oavhwSr3osQXv37?= =?us-ascii?Q?mEyI81ZNmq2fQYZuj53zJF7Hnb54d7BSnnwrXTBwTGXZmBEH2r4GiD1Vc++l?= =?us-ascii?Q?LTu0MHHFao4aH58AqrBtkZy4IJiPke5NdGGXNTdNiuh2Z2LplMjk+GzjM4Gg?= =?us-ascii?Q?srAFvy8KElPsfms=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:01.8714 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9d38bf76-7eaa-4fd3-927a-08dd671cb923 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7370 Content-Type: text/plain; charset="utf-8" Before this patch, scanning of tasks' mm is done continuously and also at the same rate. Improve that by adding a throttling logic: 1) If there were useful pages found during last scan and current scan, decrease the scan_period (to increase scan rate) by TUNE_PERCENT (15%). 2) If there were no useful pages found in last scan, and there are candidate migration pages in the current scan decrease the scan_period aggressively by 2 power SCAN_CHANGE_SCALE (2^3 =3D 8 now). Vice versa is done for the reverse case. Scan period is clamped between MIN (500ms) and MAX (5sec). Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 1 deletion(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index feca775d0191..cd2215f2e00e 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -20,6 +20,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -33,6 +34,16 @@ static DEFINE_MUTEX(kmmscand_mutex); #define KMMSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) static unsigned long kmmscand_scan_size __read_mostly =3D KMMSCAND_SCAN_SI= ZE; =20 +/* + * Scan period for each mm. + * Min: 500ms default: 2sec Max: 5sec + */ +#define KMMSCAND_SCAN_PERIOD_MAX 5000U +#define KMMSCAND_SCAN_PERIOD_MIN 500U +#define KMMSCAND_SCAN_PERIOD 2000U + +static unsigned int kmmscand_mm_scan_period_ms __read_mostly =3D KMMSCAND_= SCAN_PERIOD; + /* How long to pause between two scan and migration cycle */ static unsigned int kmmscand_scan_sleep_ms __read_mostly =3D 16; =20 @@ -74,6 +85,11 @@ static struct kmem_cache *kmmscand_slot_cache __read_mos= tly; /* Per mm information collected to control VMA scanning */ struct kmmscand_mm_slot { struct mm_slot slot; + /* Unit: ms. Determines how aften mm scan should happen. */ + unsigned int scan_period; + unsigned long next_scan; + /* Tracks how many useful pages obtained for migration in the last scan */ + unsigned long scan_delta; long address; bool is_scanned; }; @@ -590,13 +606,92 @@ static void kmmscand_migrate_folio(void) spin_unlock(&kmmscand_migrate_lock); } =20 +/* + * This is the normal change percentage when old and new delta remain same. + * i.e., either both positive or both zero. + */ +#define SCAN_PERIOD_TUNE_PERCENT 15 + +/* This is to change the scan_period aggressively when deltas are differen= t */ +#define SCAN_PERIOD_CHANGE_SCALE 3 +/* + * XXX: Hack to prevent unmigrated pages coming again and again while scan= ning. + * Actual fix needs to identify the type of unmigrated pages OR consider m= igration + * failures in next scan. + */ +#define KMMSCAND_IGNORE_SCAN_THR 256 + +/* Maintains stability of scan_period by decaying last time accessed pages= */ +#define SCAN_DECAY_SHIFT 4 +/* + * X : Number of useful pages in the last scan. + * Y : Number of useful pages found in current scan. + * Tuning scan_period: + * Initial scan_period is 2s. + * case 1: (X =3D 0, Y =3D 0) + * Increase scan_period by SCAN_PERIOD_TUNE_PERCENT. + * case 2: (X =3D 0, Y > 0) + * Decrease scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 3: (X > 0, Y =3D 0 ) + * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 4: (X > 0, Y > 0) + * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + */ +static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm= _slot, + unsigned long total) +{ + unsigned int scan_period; + unsigned long now; + unsigned long old_scan_delta; + + scan_period =3D mm_slot->scan_period; + old_scan_delta =3D mm_slot->scan_delta; + + /* decay old value */ + total =3D (old_scan_delta >> SCAN_DECAY_SHIFT) + total; + + /* XXX: Hack to get rid of continuously failing/unmigrateable pages */ + if (total < KMMSCAND_IGNORE_SCAN_THR) + total =3D 0; + + /* + * case 1: old_scan_delta and new delta are similar, (slow) TUNE_PERCENT = used. + * case 2: old_scan_delta and new delta are different. (fast) CHANGE_SCAL= E used. + * TBD: + * 1. Further tune scan_period based on delta between last and current sc= an delta. + * 2. Optimize calculation + */ + if (!old_scan_delta && !total) { + scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && total) { + scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /=3D 100; + } else if (old_scan_delta && !total) { + scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; + } else { + scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + } + + scan_period =3D clamp(scan_period, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCA= N_PERIOD_MAX); + + now =3D jiffies; + mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); + mm_slot->scan_period =3D scan_period; + mm_slot->scan_delta =3D total; +} + static unsigned long kmmscand_scan_mm_slot(void) { bool next_mm =3D false; bool update_mmslot_info =3D false; =20 + unsigned int mm_slot_scan_period; + unsigned long now; + unsigned long mm_slot_next_scan; unsigned long vma_scanned_size =3D 0; unsigned long address; + unsigned long total =3D 0; =20 struct mm_slot *slot; struct mm_struct *mm; @@ -620,6 +715,8 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 mm =3D slot->mm; mm_slot->is_scanned =3D true; + mm_slot_next_scan =3D mm_slot->next_scan; + mm_slot_scan_period =3D mm_slot->scan_period; spin_unlock(&kmmscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -630,6 +727,11 @@ static unsigned long kmmscand_scan_mm_slot(void) goto outerloop; } =20 + now =3D jiffies; + + if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) + goto outerloop; + VMA_ITERATOR(vmi, mm, address); =20 for_each_vma(vmi, vma) { @@ -658,8 +760,10 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 - if (update_mmslot_info) + if (update_mmslot_info) { mm_slot->address =3D address; + kmmscand_update_mmslot_info(mm_slot, total); + } =20 outerloop: /* exit_mmap will destroy ptes after this */ @@ -759,6 +863,10 @@ void __kmmscand_enter(struct mm_struct *mm) return; =20 kmmscand_slot->address =3D 0; + kmmscand_slot->scan_period =3D kmmscand_mm_scan_period_ms; + kmmscand_slot->next_scan =3D 0; + kmmscand_slot->scan_delta =3D 0; + slot =3D &kmmscand_slot->slot; =20 spin_lock(&kmmscand_mm_lock); --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2081.outbound.protection.outlook.com [40.107.92.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DC392144A4 for ; Wed, 19 Mar 2025 19:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412742; cv=fail; b=K917jnjLop0Q+dbQEA/iVQyA2zqH77GkYMq0cactAiiHpk9/Gkbi+arHgkVEuz0+X166SyzKefe7fQ7Micy8/3aM6KlI+awwAR8MRU3S3sKw3+X0lrhv9BfiAKLoe85tiLYvDcR5mupDAmYeXaveD/dPOLe5lE5zWy/Z+HfPA9k= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412742; c=relaxed/simple; bh=2dBo6tFePsR70JNi4xjbwtoN7lJPyl8dJNHJ9qFCbPk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HbZAMYWY6zYIf+/6E4ve6PIkHfI2rmz/zrqSB6w967jrUmKeTmw/R2iuEL/aF6GFmJ49lb45zUMEAxUJAZdxUVOcdIrkWsMaPYaw/6H34A4U61/8eEN85bByDcxi81fanjm3rErYQkEsMo0tJ8qAmvjHgz4J5udWX0+2q6VnKHc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=HjF3FpjC; arc=fail smtp.client-ip=40.107.92.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="HjF3FpjC" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=jaeMGZIzpn/7NaOLFXmIqLlnj30KRtggkgoqEGWkbp7KD6g8h8hqfMUTn1rdbgmkOTr56/O2HJ5gqIOhZdwM3g5nElHPIHBKvoB+pES9zRpyro0864q3ORf4fQ+Aymf8sAV365D11B8P+whJR57nGQhWOSYTJDeo04SsQrKyDlK2oFI1E6wPwdBYvebwb5BG/L6MwL79yt2JlQEwOBPvdTlpLnQm1vRiX+VxRW/EAq9SEHXVkLIAQKktofk95BHczXTT0ZC7RkzMFTy8BNIFPfAdGiHY0muaDqzozGpKbHlZOZMwf1Ucg9MLbn0DlezuQZ8zXOGxM8DV7kN3PH8djA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=y8Xj39u2uR4MqPXlurtQNALFnpUi3B6OTCfu53Izm5A=; b=VhPuzF8/nMphf4cUbXtwTEdhhkYcZrWifpoXINhkMmW2I32nNZMK+P1h0lvPI4Ba2QGuJv5Q+eO0soT2/kfiNLi0y5tbXgSuNUmo85RTdAdty67QEBoG/WqcR8+E9hz42ZMC28gPPAi+wQGTxCGjPVjUmtxAwkD8ezGXTOB3t4ZE3cyal0Bl5ZzjNB2ijl3zGuJ4OaT0GasO+KQqx+e8EXi/r9v1sXAJDwc5IOi6ndUasBda/SITr+kz4NskDyGgZEjuhPPIa5MLWHWJcWK5ZsbNQOcJRw7wNLohbSUTMw8SHgoh0hP8eZeo5bF83kMNWdNGpfCNDJ/OSvSdbBjHIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y8Xj39u2uR4MqPXlurtQNALFnpUi3B6OTCfu53Izm5A=; b=HjF3FpjCdG2sJ45ngskjt6Apo3L+DGDIei8LJ2LHRqQXtiPNsFBVvJv3vma38Wiv8EvOI0pbP1GuxxnfwOkMDhRGQMvrBlxb1rD8ukRTTzC1LRp46xJuATaeioq/02JcItv8b+McwhguuP442Jv5YJn8QBq0cmeJuRuzelLXoCY= Received: from SA1P222CA0080.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:35e::17) by SJ0PR12MB6805.namprd12.prod.outlook.com (2603:10b6:a03:44f::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:12 +0000 Received: from SN1PEPF00026368.namprd02.prod.outlook.com (2603:10b6:806:35e:cafe::9f) by SA1P222CA0080.outlook.office365.com (2603:10b6:806:35e::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.25 via Frontend Transport; Wed, 19 Mar 2025 19:32:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026368.mail.protection.outlook.com (10.167.241.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:11 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:02 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 07/13] mm: Add throttling of mm scanning using scan_size Date: Wed, 19 Mar 2025 19:30:22 +0000 Message-ID: <20250319193028.29514-8-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026368:EE_|SJ0PR12MB6805:EE_ X-MS-Office365-Filtering-Correlation-Id: 32265f0b-0c30-4522-e16b-08dd671cbf18 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Em3rmXGsOsYvfggcI43NOGCql+BeQdIqHmAARBsb3q/umZaXNEy0RE5QY7Jz?= =?us-ascii?Q?isqVLi487UNMc+lTORKpjGQRuZfM1+lyfEPkgMQmI73N31nc3+gsb9X/k9dc?= =?us-ascii?Q?tLYN5/eqy11WaQtHBOuHBo6YzbUs6nqtXf7bh2ICiTmfONwAvzrXFol9oc4q?= =?us-ascii?Q?tEitkKAx1O9vZptgNuI2qW502hyt9yY7A8NwKAN8Q5fYPqjfXRxPzI44zU+s?= =?us-ascii?Q?7YAXhuk3E9RMSEpZzMwPXiZzfCyXcXQmr/apMS0HYrK0uLoM7TeZ5hGGmFBa?= =?us-ascii?Q?72grW7GNtJ1up5gxCzuTTRAjp4jMC7hT/ZeIDuoHnR8DbpIu6VFs5HuY+oSJ?= =?us-ascii?Q?1SmZIz32S0cfQt1TJgXPt/l8p/Hm4FzLh/PZEGaNH95+mOfZqHHXbh1QZcku?= =?us-ascii?Q?enm5zu7EKPY0RiLJUNMISt08dw/JY6GUVapW2aj9VEDZZnvWLdWpZO2Ek22m?= =?us-ascii?Q?XbK0hgvvtOzOxYTL8S05qrGEaKOPFbELn1OE2ZAc/831P94O8ngNRzmuYXPD?= =?us-ascii?Q?Kg4h2+9cPwFXGce7SaIbum8CYKEc9p4oJv90KqLxdbrrB+hNdmPXqMCB2dPD?= =?us-ascii?Q?JezkjAJn5y/Q393QMFNzOk9TGO9lqnpJi9fg5XXdb9cryXkoaxbMnrIVGTuo?= =?us-ascii?Q?49wibWqSXLw2Pj5b2Kwg8D0MIiYXCh+9prJXqgdziEv2PXSbRSJMmpR+mdfW?= =?us-ascii?Q?uXF2vkA2gomM4HbeSVWwzZ69e07cS8Q03G6lhOptV0Rt84NluS4lFcMu4Fb+?= =?us-ascii?Q?cwQiqBXkHcLMu5FBelhainN7fPvdbLzasDFKgMRD77Bt58PWHBDCv07ZpYHL?= =?us-ascii?Q?ULUOwLIEOQiYP8xpgFb/BwVsYQAx9Y6eLBtNPX9sn1+EI+PkB1IH4cnVAZBJ?= =?us-ascii?Q?/80ut0QuHGJOTGuIiZgGiUh+9KfzfT0e8fCkRFiewMKORQ8rtCNOTWLbaeSV?= =?us-ascii?Q?XVoTJ7+56SUQP0y1QOn5U5/z468ZB+SKKsBQOE6dp6lmWl5fJqgttrVrXhmC?= =?us-ascii?Q?bTD6O4ElfaGbQfjVq/RvbKp+9xzwdo/qTIGCAUx9FcQOHC8pDnI7pb9Lzfl+?= =?us-ascii?Q?eBkK6Fg7iRcPVzn9iFXTZNX3SZES0xXFG8Tc9RAQdcTfodP1/SRbc3cPQaTY?= =?us-ascii?Q?C2wrDj0DkBefy+553CkxN48Nax2+TmaB1k1Ghf4qjpdw4lBX4reuqiae3wqI?= =?us-ascii?Q?oqVHOJ+wS2wDXUe3TdNixqku443wrvt7bMUdqx94U7aMBghCXIoDXDrm2MKL?= =?us-ascii?Q?Mx7v+dyCA2hlktSQ/HxQFGYLqhefXZkT5/YYWoLRZ99o9wTeHpbX+RYvnFkQ?= =?us-ascii?Q?8KMBmarpWw0CQg+B93crsnfNbNsOguOnSVBiKguBGibjI3ezrqiy4v734OKA?= =?us-ascii?Q?HZ02jhUH6aOy3XXMTxd0XPkAvtJ0VysVtuPoVKDxeumnLM2uWMR2c4ASUy2w?= =?us-ascii?Q?snxChhq1zdJoQPX58wytkRHTOORvxwlBECJ/m4y6/a0XXl5bgZwfRCD6lKnQ?= =?us-ascii?Q?nrw8SoHVl71quE0=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:11.8677 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32265f0b-0c30-4522-e16b-08dd671cbf18 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026368.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6805 Content-Type: text/plain; charset="utf-8" Before this patch, scanning is done on entire virtual address space of all the tasks. Now the scan size is shrunk or expanded based on the useful pages found in the last scan. This helps to quickly get out of unnecessary scanning thus burning lesser CPU. Drawback: If a useful chunk is at the other end of the VMA space, it will delay scanning and migration. Shrink/expand algorithm for scan_size: X : Number of useful pages in the last scan. Y : Number of useful pages found in current scan. Initial scan_size is 1GB case 1: (X =3D 0, Y =3D 0) Decrease scan_size by 2 case 2: (X =3D 0, Y > 0) Aggressively change to MAX (4GB) case 3: (X > 0, Y =3D 0 ) No change case 4: (X > 0, Y > 0) Increase scan_size by 2 Scan size is clamped between MIN (256MB) and MAX (4GB)). TBD: Tuning this based on real workload Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index cd2215f2e00e..a19b1f31271d 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -28,10 +28,15 @@ =20 static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); + /* * Total VMA size to cover during scan. + * Min: 256MB default: 1GB max: 4GB */ +#define KMMSCAND_SCAN_SIZE_MIN (256 * 1024 * 1024UL) +#define KMMSCAND_SCAN_SIZE_MAX (4 * 1024 * 1024 * 1024UL) #define KMMSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) + static unsigned long kmmscand_scan_size __read_mostly =3D KMMSCAND_SCAN_SI= ZE; =20 /* @@ -90,6 +95,8 @@ struct kmmscand_mm_slot { unsigned long next_scan; /* Tracks how many useful pages obtained for migration in the last scan */ unsigned long scan_delta; + /* Determines how much VMA address space to be covered in the scanning */ + unsigned long scan_size; long address; bool is_scanned; }; @@ -621,6 +628,8 @@ static void kmmscand_migrate_folio(void) */ #define KMMSCAND_IGNORE_SCAN_THR 256 =20 +#define SCAN_SIZE_CHANGE_SHIFT 1 + /* Maintains stability of scan_period by decaying last time accessed pages= */ #define SCAN_DECAY_SHIFT 4 /* @@ -636,14 +645,26 @@ static void kmmscand_migrate_folio(void) * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). * case 4: (X > 0, Y > 0) * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + * Tuning scan_size: + * Initial scan_size is 4GB + * case 1: (X =3D 0, Y =3D 0) + * Decrease scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). + * case 2: (X =3D 0, Y > 0) + * scan_size =3D KMMSCAND_SCAN_SIZE_MAX + * case 3: (X > 0, Y =3D 0 ) + * No change + * case 4: (X > 0, Y > 0) + * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm= _slot, unsigned long total) { unsigned int scan_period; unsigned long now; + unsigned long scan_size; unsigned long old_scan_delta; =20 + scan_size =3D mm_slot->scan_size; scan_period =3D mm_slot->scan_period; old_scan_delta =3D mm_slot->scan_delta; =20 @@ -664,20 +685,25 @@ static inline void kmmscand_update_mmslot_info(struct= kmmscand_mm_slot *mm_slot, if (!old_scan_delta && !total) { scan_period =3D (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size >> SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && total) { scan_period =3D (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /=3D 100; + scan_size =3D scan_size << SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && !total) { scan_period =3D scan_period << SCAN_PERIOD_CHANGE_SCALE; } else { scan_period =3D scan_period >> SCAN_PERIOD_CHANGE_SCALE; + scan_size =3D KMMSCAND_SCAN_SIZE_MAX; } =20 scan_period =3D clamp(scan_period, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCA= N_PERIOD_MAX); + scan_size =3D clamp(scan_size, KMMSCAND_SCAN_SIZE_MIN, KMMSCAND_SCAN_SIZE= _MAX); =20 now =3D jiffies; mm_slot->next_scan =3D now + msecs_to_jiffies(scan_period); mm_slot->scan_period =3D scan_period; + mm_slot->scan_size =3D scan_size; mm_slot->scan_delta =3D total; } =20 @@ -689,6 +715,7 @@ static unsigned long kmmscand_scan_mm_slot(void) unsigned int mm_slot_scan_period; unsigned long now; unsigned long mm_slot_next_scan; + unsigned long mm_slot_scan_size; unsigned long vma_scanned_size =3D 0; unsigned long address; unsigned long total =3D 0; @@ -717,6 +744,7 @@ static unsigned long kmmscand_scan_mm_slot(void) mm_slot->is_scanned =3D true; mm_slot_next_scan =3D mm_slot->next_scan; mm_slot_scan_period =3D mm_slot->scan_period; + mm_slot_scan_size =3D mm_slot->scan_size; spin_unlock(&kmmscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -864,6 +892,7 @@ void __kmmscand_enter(struct mm_struct *mm) =20 kmmscand_slot->address =3D 0; kmmscand_slot->scan_period =3D kmmscand_mm_scan_period_ms; + kmmscand_slot->scan_size =3D kmmscand_scan_size; kmmscand_slot->next_scan =3D 0; kmmscand_slot->scan_delta =3D 0; =20 --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2086.outbound.protection.outlook.com [40.107.236.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B362C215162 for ; Wed, 19 Mar 2025 19:32:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.86 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412756; cv=fail; b=i1AiFB5n+OCIatKS0Aa8Wr/9NWoLUGwtPcERUyOP3Uh1J61tEaXwNXZ4nzAwZTAWEDXbGX9S4rundeIbhTxWjGRypoGqfoSCm116MZw2KZp3p+UQiM42fWmQO5tqNy8OWzsSa5ZZct/XnM/c1zhG32qsRaUb+UjSSTTjbYNEd9Q= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412756; c=relaxed/simple; bh=oaZ/AiMA6+vJ6S5JZbHNXsodD/q3TOiQ9vQ19iktFks=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hMTf6sp4yfXJ8TnXvotY8dIrDAYpk2J2r9a2C1lM+Wq/F0WCpJcGvou8kMje6HLZdZshU/UPsFkLbB5GmADHHC/t5D/YKzCGeVV0jqsAaIc9PB7GqZt9iOnvVZ8r1JPyDNgZML19fzhuR19T6AYe1p3o63vb69SmTlVGXoFQgN0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=kvsoUwai; arc=fail smtp.client-ip=40.107.236.86 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="kvsoUwai" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zTMHDZR2EEMCdZs3UWwMQndK8OkpnKimRntopJahrXuFZ9kbYPkfgmfLikaZcxBUyN6e4iW+IhYhLafnE/mjxIirvtZ2ebDW2dcWmbgHqgl0ybuayEuh5/3ZHdfjGI7JrnfcryNmCQCthPVTN9BR/rcwLXgirUtXMjXLQyANy56rWKgKD40wlBMP3ggQBlj2DRrs0FjQUpUIRzgzrB7S5BtDC48DtgGlAF9VzuypLmlfutLwjVr5BhM5xwNi90arGI6ptu85+IXNkLutWiQ0kLDTgItgicQKXGWF1aFZ2u9g+Y6j3FERNzDm3HajwtTXDYZWzZkXoKLSU0XpsJ3KAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OWt1GZDA3EZY+FP4C7YhcSYsQNWOvQYIMshl1kgSG1I=; b=o62dgaE19l4LOptxjkvP+XjZ0lG30h1xwGc/4g7n+T+d3EdrQbfUCayQFA9ERB8aj3MVaKDjrg8ucnPrFgWEEimVDsTFtGaN/MdXZkNZpBDDaYsg7/dymSWI1zZ42MkLxD/izsykI6vMllfbi2x+T3V5+PKni+8Lcwtt4jsxqqTs3uL8TARYV1TCwo/nL6g6qMPqlppg4YnjkTQhs3z2bjpuOf95i8y8VmGXc4EBXqTVnFDJ1cyu1d8B4pKApHvZIk6crDHMLPfJbyPKhmxQ4b6CgZJOx6t/x75QZESBuI49IgbLM1OXgqdj8Eb0blok2ipKOX7fy55NUHblKmhxOA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OWt1GZDA3EZY+FP4C7YhcSYsQNWOvQYIMshl1kgSG1I=; b=kvsoUwaiYrAJ1PZq9XnhDKRKfvY20q2UzvMZzRGqeZQEJQ5KQnLCsDueb00ml0POvaK8hTRuL4F6JQ50aD8UjuqBIgdq9YvPuPeDrwRUTvHG8xILDFV+ZuTR38FOi4Hd8EbPyDHUPv3gCL38pLtatQ1xjLe1tR9LtLoTH2IfOx8= Received: from SN7PR04CA0005.namprd04.prod.outlook.com (2603:10b6:806:f2::10) by MW6PR12MB8835.namprd12.prod.outlook.com (2603:10b6:303:240::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:22 +0000 Received: from SN1PEPF0002636B.namprd02.prod.outlook.com (2603:10b6:806:f2:cafe::6e) by SN7PR04CA0005.outlook.office365.com (2603:10b6:806:f2::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:32:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636B.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:21 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:11 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 08/13] mm: Add initial scan delay Date: Wed, 19 Mar 2025 19:30:23 +0000 Message-ID: <20250319193028.29514-9-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636B:EE_|MW6PR12MB8835:EE_ X-MS-Office365-Filtering-Correlation-Id: 4eebad5a-3e68-4ae3-2f75-08dd671cc510 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Ok7TasDbDnnHhl32YkpffKvFAVstQVfvDxyTfIRD1RdHJZ3ti1gvSNPBEDPW?= =?us-ascii?Q?sN1fVWvcigqPcOWPmIPYELNylgi63+TXbAairLoptFGJYgOUa4XVqwOX1nUy?= =?us-ascii?Q?qPqImScE61rlLT+7VB/H/ulIEJynayR/aSstXP7O6LZOXGJy2y8fm7LvxAFt?= =?us-ascii?Q?H7lY0Auu5UxztfRNm9N3rT3AYlUzjDpPkyk14fGVlniQWmMxZxIlyDFjaG6x?= =?us-ascii?Q?3EAf4/lWeXNEEgHsTP8dqiG5Htj1/Ard9fX/ZQHkvozKp27KWH2nLgi/V6rd?= =?us-ascii?Q?c66JTjkeczlpiw9+gydWmA6WC+vU6Mm/SBVWNFXFqmjGl2dFBOVojp7H2xuV?= =?us-ascii?Q?dHUo3J+OQUwmEncpvOnv1J0sg6clnw+n58NX9h6q0ppf6OVqbXZWu0NZzCl3?= =?us-ascii?Q?/z5/rB7joDrWbiq3jXB0hpz4K9rH3oCWL6BZda0XgWYyvkKWete2FkQZysyN?= =?us-ascii?Q?4cYdz5rE61KgZ7rP50nACBMlmK/R1p3z3utvQJWOGoE82/Sz+gGawovWWy31?= =?us-ascii?Q?K0UEYO6pVCpGd3c490g9AGvO56e/FrrM2anDjAEJIqqNowmaDL8c4452hY15?= =?us-ascii?Q?cyQciR2Cwi1UfF9oC7RDRxgu8dP1JwA3cWPaGPFQKl19GMDLdbMS9p0Xoagc?= =?us-ascii?Q?eSJ9LMbd32DEjU3f+U9soU9/xFgZoi3L/R78vkBqrG/fRziW6XmzI/Wbp6DK?= =?us-ascii?Q?MGqw/+A7V+y9bgcXCY4zulxjDnvi8FPMZwutksC4CUBRCtsUl1jy0/dZRKSr?= =?us-ascii?Q?5Y5V92qiA56aM/BlDwdowGuQd8ekUUIHWNzGsIk4MLGyeE1McoCd1sUc1Rmz?= =?us-ascii?Q?4Ecq2ImplJbRk1cSsjP9/q8LcyKbHJ99SEpCbManKluizDJml3C43Nt+BUvX?= =?us-ascii?Q?U2BYEYsrSQebKWsEayWoXBKpdOUtN+6VBPUciZmxqd+F75FvaJoJXNKe7Od5?= =?us-ascii?Q?YpDEoZ6s4YgetfA987E7mArZK66CGweH6OyQQ8eX4AZoZq84Ufc157EMyw12?= =?us-ascii?Q?q9xnSegHZfu6vO0BQNjy+x3C67yp2WFo1/5XET8A9bEdTvx89n5uMV/DX2rP?= =?us-ascii?Q?VZcqcAVoDB2gRjyYynA0PILW1vhrjRzu/26j7OMFR/RL48g3SP2vG9VfF+7M?= =?us-ascii?Q?8zm+WZgVsxYpDgez9GvUG30N87GHrLaMC+yQRVry7BSuD+Ra6B+w0ELdp+2k?= =?us-ascii?Q?ToVHdH2wOqv9no9OM3uWrarSN6frojAmsos5n4bWyI+JIE6B/IrXvQqlxO/8?= =?us-ascii?Q?spPD+ViNX6YioHyqYp+nThxBOP7ovoe4wP1LKIt6ojWRhvKWs8YGy3vg/sK7?= =?us-ascii?Q?c2+lfNwRL6h+nkulDGY/qGFtxELrUry20xslvDvble6qZAT3Vo62JHkFm6vl?= =?us-ascii?Q?OZycWbUkyGg0sTg51aoK4ojqbF/iJY4S8sarn13Zs2XA6/jSauW7MR0v4pvw?= =?us-ascii?Q?BUlI+x4mdRGfTWHgokwNqSs8VmGPR+bo62g1FB1RaDVM7qbCmxug7lvvCfAc?= =?us-ascii?Q?zNsTARlCwJhWBGE=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:21.8770 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4eebad5a-3e68-4ae3-2f75-08dd671cc510 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8835 Content-Type: text/plain; charset="utf-8" This is to prevent unnecessary scanning of short lived tasks spending in scanning. Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index a19b1f31271d..84140b9e8ce2 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -28,6 +28,7 @@ =20 static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); +extern unsigned int sysctl_numa_balancing_scan_delay; =20 /* * Total VMA size to cover during scan. @@ -880,6 +881,7 @@ void __kmmscand_enter(struct mm_struct *mm) { struct kmmscand_mm_slot *kmmscand_slot; struct mm_slot *slot; + unsigned long now; int wakeup; =20 /* __kmmscand_exit() must not run from under us */ @@ -890,10 +892,12 @@ void __kmmscand_enter(struct mm_struct *mm) if (!kmmscand_slot) return; =20 + now =3D jiffies; kmmscand_slot->address =3D 0; kmmscand_slot->scan_period =3D kmmscand_mm_scan_period_ms; kmmscand_slot->scan_size =3D kmmscand_scan_size; - kmmscand_slot->next_scan =3D 0; + kmmscand_slot->next_scan =3D now + + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); kmmscand_slot->scan_delta =3D 0; =20 slot =3D &kmmscand_slot->slot; --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2076.outbound.protection.outlook.com [40.107.236.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8045218AA3 for ; Wed, 19 Mar 2025 19:32:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412761; cv=fail; b=rzxbDi3gn1IUlYUR4zqmK688ySzJwWsEoXO24ucw9nYLD/P3LH2326bmcpvCo3fvgUJELfHYO0ERrvusRIRDWRVmeaRNsknHydFxIgapO2vHz8HEhsuXDIO98FXbJXZd5LDLasQMY++VTA8fLKVhPcxjzdZg0cGN1/UWZ2Zz7wM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412761; c=relaxed/simple; bh=Ea7kqqzHF5klzC94BE/SaIJNEU6+sSe9LZg3e6WgOCM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fbyoYZznWket74QLiiEFe7K4xmlUg7B0uyntYx5L8tcI8hr3xiqs0dLXrX2A0OsyhsKejUNcMkJb1KwcD4qHmyp6ECvyj0njQmS2Jzm0QlxAJyuO74iWhk5SLa6AaWeLaatwlwJxjOzLobhx6Wa70lV28DP9ZY4Hqa3Fai8geOU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=L2lXQwqg; arc=fail smtp.client-ip=40.107.236.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="L2lXQwqg" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MqGRPycymWQayi5YBWyQAECzfD0Igb5HuJ1v8IUah1ESpXcTRjPE/1a6KfmWVGaXbpfEcr6DaRor8rRB9eNoaO3X/cYF1lQXZ/4GYLPqmLlf2tTQys5S1YysfYvxO769C9iA7fpKjqGEyZiGbwGtXVbZ3Mzi/12uOYBjYZGyQ3+u6rCUZm/y2K+rEuH4PBIDpioYawKic8xPVTTi5uZvsiU6yN0a1MMqnqv++a2cI51N5feHP7Fyca9W/dbbGDYhVeY9CYixxJxW38dlbq11FLxU7zRoYUaW2Ord3Zl0M/apxxasCgAwn1d798JHCNBRO8HkGe61490+lTzVbvspzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3JobkGtP3Dd44LrrvsSjv2D3CsygQXYXyfCaumfB9Hk=; b=lfisZdkpKVMdNpvdLTaxf4Z70bEXrcXO7ygHjYpIOiYPlHGB5syIa5rB5IPdn9ItR8Ca2hORVmKDtoF6M0nTUVd4Kh8uHf3E135cB+Li5HLYmCgGXDjZAJ0Bcs6treIpRCvsc9AzvjhS2TfsCU5gbsTRtSu476/ka0MN5f48nM3Xg7PPdySze+KO7rR4/IXP7zeFTKuPHp6yABBZ8BERaqzPbK9el4vslcAKKAN8uUA1EfMxqmSACtQ7554FgA4GA0WZrhM4VbyhorobjIlylBVQg8E7+N1gMP4TG19rfH99OtSP/tM0l+JmKecUIDUMutMzieVBYA+7IZ4T+1csxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3JobkGtP3Dd44LrrvsSjv2D3CsygQXYXyfCaumfB9Hk=; b=L2lXQwqgg3ziT7XgoNHqKtaFbmXyDj68hmkdNeNou/eETuOQJf0V74A7tEFeVFgrnLsnbnJ/6nWv8nrt4Ia9B2LI3I5wcER8omV+r3ybIe5zDheECHdN1Zs/Mq9NU3Yz2W4fs4LSKveNLY6G1YyYwJ4zrK5g6WLm0MVY0cvj8wM= Received: from SA9PR13CA0096.namprd13.prod.outlook.com (2603:10b6:806:24::11) by MN2PR12MB4110.namprd12.prod.outlook.com (2603:10b6:208:1dd::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:32 +0000 Received: from SN1PEPF0002636D.namprd02.prod.outlook.com (2603:10b6:806:24:cafe::be) by SA9PR13CA0096.outlook.office365.com (2603:10b6:806:24::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:32:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636D.mail.protection.outlook.com (10.167.241.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:31 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:21 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 09/13] mm: Add heuristic to calculate target node Date: Wed, 19 Mar 2025 19:30:24 +0000 Message-ID: <20250319193028.29514-10-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636D:EE_|MN2PR12MB4110:EE_ X-MS-Office365-Filtering-Correlation-Id: 89c7117f-eac1-4ed3-7784-08dd671ccae1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?epnti3YhX8xkc7q4GWx1wq3kfO81RLoY3glgj+4tJV44spY3HwAKppiQO7/K?= =?us-ascii?Q?YM4+dk+jZFH/PCtXjS+byH884CxZqpGzwFpck+TfRfycWqA7MOZypkWVVbT9?= =?us-ascii?Q?mCC7dyldfY76kBFsuMH8B767I1+6PFjieR8O7zwa4PmrBnLkjPJUcakzWFtc?= =?us-ascii?Q?lBorCyknsiPqN0KwsCIpattrMcLmLXbcWtwAby6gNWetKHT3OugXD4kICgZG?= =?us-ascii?Q?47COWjBnSezE7evblWVscKkpC29e4TpOH7W9pu8RFMDbwytF0ygm/O//zc0m?= =?us-ascii?Q?VK7iNtttWKXZWUKz/0+7BvzRlRo7/0IcShIpnw0hNCm5I7G241CYjyrGMWl2?= =?us-ascii?Q?fQ5Y2eTUYEZKdQMWaAFzhe1+Wgo9SVY7vTGGFfpXCZB8kpEMVreLv6WjRx5K?= =?us-ascii?Q?/HD/JuIOy6/1bK+IH4Y5covYUmwkVBfeiZVoRypSxRyfn1kr/soQ+Uhpgogl?= =?us-ascii?Q?rstWV7KRfMjBVTVnJXgLHw49ObNqeWN3AE6J4IJUzVJFpemgVvfidyMZ1wA0?= =?us-ascii?Q?FF5UPO2SM/0xoTlJp6KTh/fTNqqQ5foGx+SPFi+Cs93S6kCs3nfU47fZIcu4?= =?us-ascii?Q?4K7LOU1y7o+6p76G5MVZJdOBmEMLgSUmWnYYBUZ4/Qz2Wa6AfMG3nbMgNJjH?= =?us-ascii?Q?AKOEKiDh6tJoqx1S0EqFJ6gEe+rd8JcnI9fxHbvU0lDGE/qSmtze370PLpNI?= =?us-ascii?Q?qjzrJXudETSVEeB+jHchxI2JmgIzMjGC4gmTqF9Oq2TyCd8wX3UFmcZE/jbr?= =?us-ascii?Q?S90Z3JBT3TyN8hu5D+zc4ihSOLX56L8IkVfcgTdDNOWbo9K/qisH3E3g61ZO?= =?us-ascii?Q?29bcSTJlCTMr1iEm1JjTuX3TGD6AUfbDDw3aqPCem1GgtJjyY7U5nNqaacMb?= =?us-ascii?Q?4VSIwS9WMfbtQYI1DT8UaxBOwgAbMT1Qs2P3qE4Rs8QUg7+/VC0y/uaJdlH4?= =?us-ascii?Q?EOg6YOqHowsW4h6hq+Ia5ojcZewUewjwbk9m6URCEkhgdBYeZoKAHkwLsZIs?= =?us-ascii?Q?VuEIIj6NI1wR8iB3iCLLDC5vPVIhgPsywHTPsFxFrRd9znhPAqwRqdOvlu/r?= =?us-ascii?Q?rJ7I2FmCq0LV7nYemNWFBbZRdw3P1mnfwg7fRdrCDiHwAkXnAXTe/R/Q34M8?= =?us-ascii?Q?GXkiWyvEgOnlAiCZkOEX4BYs24sDHpNtKgIgm0VeelvcxnCZ/Tq3HgVsO7ko?= =?us-ascii?Q?NIENn9xs7/Is1p8Vke6ec7YtSw2nKWKkFgVS6fdozwDGSayM7DYvy7JZp8ik?= =?us-ascii?Q?gM7xicWvE5dW/sLeJR1wX8YBv5/cDdSxzWxf5Xh6HMfvQdY6BgGp3SfsrAwr?= =?us-ascii?Q?KCUX4eNiQnDKsnImyu7hiZQb9p5NH2GVCi9d8E1wXuADrnOXneHTa7/Y/JCh?= =?us-ascii?Q?4dR7CMkJmpdqFm7mMmUHgZtYwPGs5/Lg5NuSlEb3h1eFvG20FI5cOw5frGj/?= =?us-ascii?Q?cRbVqwq5MzOIo1SIlnK+nSV7UgMMhXzwQfMJeUFinMbo8G6TD9/JX2A0vijt?= =?us-ascii?Q?Bt9XdIekMLBNeok=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:31.5747 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 89c7117f-eac1-4ed3-7784-08dd671ccae1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4110 Content-Type: text/plain; charset="utf-8" One of the key challenges in PTE A bit based scanning is to find right target node to promote to. Here is a simple heuristic based approach: While scanning pages of any mm we also scan toptier pages that belong to that mm. We get an insight on the distribution of pages that potentially belonging to particular toptier node and also its recent access. Current logic walks all the toptier node, and picks the one with highest accesses. Signed-off-by: Raghavendra K T --- PS: There are many potential idea possible here. 1. we can do a quick sort on toptier nodes scan and access info and maintain the list of preferred nodes/fallback nodes in case of current target_node is getting filled up 2. We can also keep the history of access/scan information from last scan used its decayed value to get a stable view etc etc. include/linux/mm_types.h | 4 + mm/kmmscand.c | 174 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 174 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0234f14f2aa6..eeaedc7473b1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1015,6 +1015,10 @@ struct mm_struct { /* numa_scan_seq prevents two threads remapping PTEs. */ int numa_scan_seq; #endif +#ifdef CONFIG_KMMSCAND + /* Tracks promotion node. XXX: use nodemask */ + int target_node; + #endif /* * An operation with batched TLB flushing is going on. Anything * that can move process memory needs to flush the TLB when diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 84140b9e8ce2..c2924b2e8a6d 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -88,6 +88,14 @@ static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash,= KMMSCAND_SLOT_HASH_BITS =20 static struct kmem_cache *kmmscand_slot_cache __read_mostly; =20 +/* Per memory node information used to caclulate target_node for migration= */ +struct kmmscand_nodeinfo { + unsigned long nr_scanned; + unsigned long nr_accessed; + int node; + bool is_toptier; +}; + /* Per mm information collected to control VMA scanning */ struct kmmscand_mm_slot { struct mm_slot slot; @@ -100,6 +108,7 @@ struct kmmscand_mm_slot { unsigned long scan_size; long address; bool is_scanned; + int target_node; }; =20 /* Data structure to keep track of current mm under scan */ @@ -118,7 +127,9 @@ struct kmmscand_scan kmmscand_scan =3D { */ struct kmmscand_scanctrl { struct list_head scan_list; + struct kmmscand_nodeinfo *nodeinfo[MAX_NUMNODES]; unsigned long address; + unsigned long nr_to_scan; }; =20 struct kmmscand_scanctrl kmmscand_scanctrl; @@ -208,6 +219,98 @@ static void kmmmigrated_wait_work(void) migrate_sleep_jiffies); } =20 +static unsigned long get_slowtier_accesed(struct kmmscand_scanctrl *scanct= rl) +{ + int node; + unsigned long accessed =3D 0; + + for_each_node_state(node, N_MEMORY) { + if (!node_is_toptier(node) && scanctrl->nodeinfo[node]) + accessed +=3D scanctrl->nodeinfo[node]->nr_accessed; + } + return accessed; +} + +static inline void set_nodeinfo_nr_accessed(struct kmmscand_nodeinfo *ni, = unsigned long val) +{ + ni->nr_accessed =3D val; +} +static inline unsigned long get_nodeinfo_nr_scanned(struct kmmscand_nodein= fo *ni) +{ + return ni->nr_scanned; +} + +static inline void set_nodeinfo_nr_scanned(struct kmmscand_nodeinfo *ni, u= nsigned long val) +{ + ni->nr_scanned =3D val; +} + +static inline void reset_nodeinfo_nr_scanned(struct kmmscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); +} + +static inline void reset_nodeinfo(struct kmmscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); + set_nodeinfo_nr_accessed(ni, 0); +} + +static void init_one_nodeinfo(struct kmmscand_nodeinfo *ni, int node) +{ + ni->nr_scanned =3D 0; + ni->nr_accessed =3D 0; + ni->node =3D node; + ni->is_toptier =3D node_is_toptier(node) ? true : false; +} + +static struct kmmscand_nodeinfo *alloc_one_nodeinfo(int node) +{ + struct kmmscand_nodeinfo *ni; + + ni =3D kzalloc(sizeof(*ni), GFP_KERNEL); + + if (!ni) + return NULL; + + init_one_nodeinfo(ni, node); + + return ni; +} + +/* TBD: Handle errors */ +static void init_scanctrl(struct kmmscand_scanctrl *scanctrl) +{ + struct kmmscand_nodeinfo *ni; + int node; + + for_each_node(node) { + ni =3D alloc_one_nodeinfo(node); + if (!ni) + WARN_ON_ONCE(ni); + scanctrl->nodeinfo[node] =3D ni; + } +} + +static void reset_scanctrl(struct kmmscand_scanctrl *scanctrl) +{ + int node; + + for_each_node_state(node, N_MEMORY) + reset_nodeinfo(scanctrl->nodeinfo[node]); + + /* XXX: Not rellay required? */ + scanctrl->nr_to_scan =3D kmmscand_scan_size; +} + +static void free_scanctrl(struct kmmscand_scanctrl *scanctrl) +{ + int node; + + for_each_node(node) + kfree(scanctrl->nodeinfo[node]); +} + /* * Do not know what info to pass in the future to make * decision on taget node. Keep it void * now. @@ -217,6 +320,24 @@ static int kmmscand_get_target_node(void *data) return kmmscand_target_node; } =20 +static int get_target_node(struct kmmscand_scanctrl *scanctrl) +{ + int node, target_node =3D NUMA_NO_NODE; + unsigned long prev =3D 0; + + for_each_node(node) { + if (node_is_toptier(node) && scanctrl->nodeinfo[node] && + get_nodeinfo_nr_scanned(scanctrl->nodeinfo[node]) > prev) { + prev =3D get_nodeinfo_nr_scanned(scanctrl->nodeinfo[node]); + target_node =3D node; + } + } + if (target_node =3D=3D NUMA_NO_NODE) + target_node =3D kmmscand_get_target_node(NULL); + + return target_node; +} + extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages); =20 @@ -469,6 +590,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, } srcnid =3D folio_nid(folio); =20 + scanctrl->nodeinfo[srcnid]->nr_scanned++; + if (scanctrl->nr_to_scan) + scanctrl->nr_to_scan--; + + if (!scanctrl->nr_to_scan) { + folio_put(folio); + return 1; + } =20 if (!folio_test_lru(folio)) { folio_put(folio); @@ -479,11 +608,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, mmu_notifier_test_young(mm, addr) || folio_test_referenced(folio) || pte_young(pteval)) { =20 + scanctrl->nodeinfo[srcnid]->nr_accessed++; + /* Do not try to promote pages from regular nodes */ if (!kmmscand_eligible_srcnid(srcnid)) { folio_put(folio); return 0; } + info =3D kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { =20 @@ -571,6 +703,7 @@ static void kmmscand_collect_mm_slot(struct kmmscand_mm= _slot *mm_slot) static void kmmscand_migrate_folio(void) { int ret =3D 0, dest =3D -1; + struct mm_struct *oldmm =3D NULL; struct kmmscand_migrate_info *info, *tmp; =20 spin_lock(&kmmscand_migrate_lock); @@ -596,7 +729,16 @@ static void kmmscand_migrate_folio(void) spin_unlock(&kmmscand_migrate_lock); =20 if (info->mm) { - dest =3D kmmscand_get_target_node(NULL); + if (oldmm !=3D info->mm) { + if (!mmap_read_trylock(info->mm)) { + dest =3D kmmscand_get_target_node(NULL); + } else { + dest =3D READ_ONCE(info->mm->target_node); + mmap_read_unlock(info->mm); + } + oldmm =3D info->mm; + } + ret =3D kmmscand_promote_folio(info, dest); } =20 @@ -658,7 +800,7 @@ static void kmmscand_migrate_folio(void) * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm= _slot, - unsigned long total) + unsigned long total, int target_node) { unsigned int scan_period; unsigned long now; @@ -706,6 +848,7 @@ static inline void kmmscand_update_mmslot_info(struct k= mmscand_mm_slot *mm_slot, mm_slot->scan_period =3D scan_period; mm_slot->scan_size =3D scan_size; mm_slot->scan_delta =3D total; + mm_slot->target_node =3D target_node; } =20 static unsigned long kmmscand_scan_mm_slot(void) @@ -714,6 +857,7 @@ static unsigned long kmmscand_scan_mm_slot(void) bool update_mmslot_info =3D false; =20 unsigned int mm_slot_scan_period; + int target_node, mm_slot_target_node, mm_target_node; unsigned long now; unsigned long mm_slot_next_scan; unsigned long mm_slot_scan_size; @@ -746,6 +890,7 @@ static unsigned long kmmscand_scan_mm_slot(void) mm_slot_next_scan =3D mm_slot->next_scan; mm_slot_scan_period =3D mm_slot->scan_period; mm_slot_scan_size =3D mm_slot->scan_size; + mm_slot_target_node =3D mm_slot->target_node; spin_unlock(&kmmscand_mm_lock); =20 if (unlikely(!mmap_read_trylock(mm))) @@ -756,6 +901,9 @@ static unsigned long kmmscand_scan_mm_slot(void) goto outerloop; } =20 + mm_target_node =3D READ_ONCE(mm->target_node); + if (mm_target_node !=3D mm_slot_target_node) + WRITE_ONCE(mm->target_node, mm_slot_target_node); now =3D jiffies; =20 if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -763,11 +911,17 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 VMA_ITERATOR(vmi, mm, address); =20 + /* Either Scan 25% of scan_size or cover vma size of scan_size */ + kmmscand_scanctrl.nr_to_scan =3D mm_slot_scan_size >> PAGE_SHIFT; + /* Reduce actual amount of pages scanned */ + kmmscand_scanctrl.nr_to_scan =3D mm_slot_scan_size >> 1; + for_each_vma(vmi, vma) { kmmscand_walk_page_vma(vma, &kmmscand_scanctrl); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 - if (vma_scanned_size >=3D kmmscand_scan_size) { + if (vma_scanned_size >=3D mm_slot_scan_size || + !kmmscand_scanctrl.nr_to_scan) { next_mm =3D true; /* Add scanned folios to migration list */ spin_lock(&kmmscand_migrate_lock); @@ -789,9 +943,19 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + total =3D get_slowtier_accesed(&kmmscand_scanctrl); + target_node =3D get_target_node(&kmmscand_scanctrl); + + mm_target_node =3D READ_ONCE(mm->target_node); + + /* XXX: Do we need write lock? */ + if (mm_target_node !=3D target_node) + WRITE_ONCE(mm->target_node, target_node); + reset_scanctrl(&kmmscand_scanctrl); + if (update_mmslot_info) { mm_slot->address =3D address; - kmmscand_update_mmslot_info(mm_slot, total); + kmmscand_update_mmslot_info(mm_slot, total, target_node); } =20 outerloop: @@ -988,6 +1152,7 @@ static int stop_kmmscand(void) kthread_stop(kmmscand_thread); kmmscand_thread =3D NULL; } + free_scanctrl(&kmmscand_scanctrl); =20 return err; } @@ -1044,6 +1209,7 @@ static void init_list(void) spin_lock_init(&kmmscand_migrate_lock); init_waitqueue_head(&kmmscand_wait); init_waitqueue_head(&kmmmigrated_wait); + init_scanctrl(&kmmscand_scanctrl); } =20 static int __init kmmscand_init(void) --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2073.outbound.protection.outlook.com [40.107.236.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86E3C21B18C for ; Wed, 19 Mar 2025 19:32:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.73 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412769; cv=fail; b=T4a6sN2T2GWuA/bmn0ZqA0PCk3vRcPLLi7eUGdhhb6Tl3zg2AWIdU8AXumdqW2MbROzaW8pqw3TbRkCRznjbHVpmrmUDAh3HnqMKqoco+K7GZx1hshwDxBA3RA562NSkN7LHXfxqRTz9jMBgAsZm+wqkrZaNoBSG8SAMJATxkJk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412769; c=relaxed/simple; bh=DJO24KfRJWq4+xtlrzWp+ZYa9gQiKx2/IT3dOwYhdRI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OnD6vvniYUb1Vq3Mkadni0c1Y3OyqNDgeiEFIXnuc4Y1O7CricwWhH9cybNs3CJRcMXO2LB4DTXUPpGykOhEluNlvzdp82LEyjPkUoRpMD8YqeHbc9tyfZoQ+aRtGwM2ZK5OnhPrCVEBMdaWaV+AMh+Gs91lwgXmCp6yX/U7BQ8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=228ZUA6S; arc=fail smtp.client-ip=40.107.236.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="228ZUA6S" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tYjFmh+W2io2rjBHEVxDI9GaaUKZOfDf5MVJ6g/AcjxXkmJ6xKEV05al1ShoP8XYOuHv6t1atVPULsXy50t0RL7ZWpwZmyAi+XrlH+xkd4yVZXOHYF9RzdjMyqTjfG4GrN1ra4Df36Nyhi0k/TYVQAL8GxC46j97IB0NELdxpIMD9JHD7VWYn2jGJEsnuIjcb2BSJ01AzfLWlDBNoDGnM+VgDXUnG4M5AxB7/nKb2ODYBEEpQR9CB7uc3aWTWDw7OJoQjBH3lXR34llP1krMm2kfxq/c3Wm+vPyBWjeS5s7+mW5m7SkqCdPMvNdKtHWPcbKWGmylKaXOe1nJ2pZQLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yMVd8QbmmNVnERYroypm9kZTSC/0smb5CRgcSN7QW8I=; b=gnNADV4GF1XTZncC167hF97yk+H3xBi8RipuLK3gJIReuRJ2+BmiewoFT09r92uAJB+VTR2rWArbOJJ/kFnt/SZhsv3P290XZ3nvpQ+igHYznKvL/eeQQDXWN6p6TIrJfgYzVaKHKM2C/0hscytaTuRs9D4TyhlW3XdE2Suw9qXnxZNR1zH3ZKRx5lnHaXzErFG9twqeDGcYdI4jI+Y5R/7fiyziCdC/4kJNW+eXamfqUKSJtehlJd3Zc2v3/mF0t73yzGdNNrId7AbEGOH8n1iWzQlqFEZLo7bcrE54SHrSwO01Zk7wcO2dv+iFEZ114jHKgigSf7i9mHKNQBhTNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yMVd8QbmmNVnERYroypm9kZTSC/0smb5CRgcSN7QW8I=; b=228ZUA6SiY0sEEmsrKpVoGHg1E07YX7wiZa+7UpI2obR3Buce0snpVjCd3v84qJ6VtklMC/BT8jokyFhM7uV/nvv5hgM97XdFciEa3cxSpXSf22L3MIaQ6miIDmidmgzoAC/W8GP2uxgFTmNLW2JqTnNoEtVi4kynjR3uCuAbIE= Received: from SA9PR10CA0001.namprd10.prod.outlook.com (2603:10b6:806:a7::6) by LV8PR12MB9229.namprd12.prod.outlook.com (2603:10b6:408:191::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:32:42 +0000 Received: from SN1PEPF0002636E.namprd02.prod.outlook.com (2603:10b6:806:a7:cafe::96) by SA9PR10CA0001.outlook.office365.com (2603:10b6:806:a7::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:32:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636E.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:41 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:31 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 10/13] sysfs: Add sysfs support to tune scanning Date: Wed, 19 Mar 2025 19:30:25 +0000 Message-ID: <20250319193028.29514-11-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636E:EE_|LV8PR12MB9229:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c35ec17-c9d3-4ec8-20f0-08dd671cd0ab X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|82310400026|1800799024|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?d9gCgkgz0Dqnk0NRtvaDMsDCHBlj6IEcm7cax/yWMTCEsoCu86Vq787p/SyY?= =?us-ascii?Q?H9CAFSe5TGdhbkVGkygkaDjqmRgVUq6r0DxQtR3wRziWS9tLzBwn5IMJtch6?= =?us-ascii?Q?AL2gWrRkFBY+isXKvoIS1uLLzHU3lz53KAs1ShGMCiiHnr6g+Z2JL3dLm0Hm?= =?us-ascii?Q?RNrHqcaH94C1nnizTPb+icT8kXE/xg1VQjWiJpDQBHTnF8zEag2kJaw4zfN5?= =?us-ascii?Q?nUsTpmziatkA/+a/F5STen8AmqSSMtJuH5Nbku0oipS5ojGcc6F+jeHXd+tZ?= =?us-ascii?Q?y323i0zPBIzGBMz7Ojaf+ViCa7OV4Y9pN4CJG3JWk0SyLB1bqDdV8XhTm1Ws?= =?us-ascii?Q?G6C7wIGoJlvYIcyIhLPjZZB8erl/g3P888/gi4nprRnJRmVpzqpaNsBB+Dbn?= =?us-ascii?Q?axPjAyOoTcr3s7fjmlXO0eHZHRFqRzVCw4qle4MVVaDKEnY3S68ZhFs9irmp?= =?us-ascii?Q?0Uhq6btt+pccqSpqPTaFuNHppEIBKdHckgHZdLPimGUO2sfdGunX5IgZVRqV?= =?us-ascii?Q?L0XlPqcdGwkXV2tKbvL+MEkdWscgPXfRlEX4IH1n+xV/xpY5m26UAsyUnn8p?= =?us-ascii?Q?z187+pmSTKb5T6uCcbcHes2Ly4hzhxva4vwbfbP3gMo1rEKaMVz1L2dRFjZu?= =?us-ascii?Q?qX/x6a+RVCqvmQlQK1gIjckjkt5w5XjGIEFJSSioLA75ucFTGKgpwwgpCCuO?= =?us-ascii?Q?MFhGUjATw6vpUPgsfoKTVz6ea7xR6v4eBGb7njXiKX4wYlMlU+Gk/COBRUfO?= =?us-ascii?Q?MEvk/faY4gC2MHPzfXzuAT6RTrQQzNxAQ+vYyoV45CezyEtYRtwhfckgOr/Q?= =?us-ascii?Q?9pAypYJOYdYZdiW6wRVDDUrh/B6F9a4BpUso+sTwWpFXU1RQFQg7SOJD6htq?= =?us-ascii?Q?xEfcmwllM3DqaWsTiRfwArNnUNV00VuD5Szzzwlkoo9lIRv2FCAtxgiz76Pa?= =?us-ascii?Q?bMmvUxt8Ed6cX1eDX6c9Ic9L0B0QAbo7z8cRvtpykXlPYIGif4cOZZjgiXuu?= =?us-ascii?Q?d4y07eTxcjHbkz7+4soKAyiQjQXYjbjZUvJdx/zqotc1RtHIYVDDjnAc5vH/?= =?us-ascii?Q?Cau842QSHVcnDhEvRJD+OexsjKv10DpU0bDBzA9miK4tKFlbZu2YFWHVPu+R?= =?us-ascii?Q?UpgOLySBOdpVBlk5BCZIeCVaDwv8Rsmzt3dSy5HSgDlHXBzj6GLoa8ln9hZp?= =?us-ascii?Q?dlKNX3eVvj18dnfIDbc+eIj7HIeXujRpJaaXd8iEs+FxibM+XWTMe7ZQkIZj?= =?us-ascii?Q?FvNt0Kn18eougjJH0gkMh6ebIkDf97wssyhibTdcBJ8w75s1wlqp93PorKTo?= =?us-ascii?Q?yZQKqtpPlMSgv1A7pxYXx9XD8nybc96J+MkdkJtOSHxYnR4FfzBuFUMLgseu?= =?us-ascii?Q?fochu8clfuqOBtMHNqPRAGJbkEZDgT4ELy6BcYa5Rfgq2VHvIqdvNxIKenmZ?= =?us-ascii?Q?FratRa/lJyx8mUqYi16mnwe48ExXM5Qi64FVy3GquY+rMcHg6/urkA=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(82310400026)(1800799024)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:41.3162 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c35ec17-c9d3-4ec8-20f0-08dd671cd0ab X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9229 Content-Type: text/plain; charset="utf-8" Support below tunables: scan_enable: turn on or turn off mm_struct scanning scan_period: initial scan_period (default: 2sec) scan_sleep_ms: sleep time between two successive round of scanning and migration. mms_to_scan: total mm_struct to scan before taking a pause. target_node: default regular node to which migration of accessed pages is done (this is only fall back mechnism, otherwise target_node heuristic is used). Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 206 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 206 insertions(+) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index c2924b2e8a6d..618594d7c148 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -21,6 +21,7 @@ #include #include #include +#include =20 #include #include "internal.h" @@ -158,6 +159,170 @@ static bool kmmscand_eligible_srcnid(int nid) return false; } =20 +#ifdef CONFIG_SYSFS +static ssize_t scan_sleep_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_scan_sleep_ms); +} + +static ssize_t scan_sleep_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + kmmscand_scan_sleep_ms =3D msecs; + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} +static struct kobj_attribute scan_sleep_ms_attr =3D + __ATTR_RW(scan_sleep_ms); + +static ssize_t mm_scan_period_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_mm_scan_period_ms); +} + +/* If a value less than MIN or greater than MAX asked for store value is c= lamped */ +static ssize_t mm_scan_period_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs, stored_msecs; + int err; + + err =3D kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + stored_msecs =3D clamp(msecs, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCAN_PER= IOD_MAX); + + kmmscand_mm_scan_period_ms =3D stored_msecs; + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute mm_scan_period_ms_attr =3D + __ATTR_RW(mm_scan_period_ms); + +static ssize_t mms_to_scan_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%lu\n", kmmscand_mms_to_scan); +} + +static ssize_t mms_to_scan_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned long val; + int err; + + err =3D kstrtoul(buf, 10, &val); + if (err) + return -EINVAL; + + kmmscand_mms_to_scan =3D val; + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute mms_to_scan_attr =3D + __ATTR_RW(mms_to_scan); + +static ssize_t scan_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_scan_enabled ? 1 : 0); +} + +static ssize_t scan_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int val; + int err; + + err =3D kstrtouint(buf, 10, &val); + if (err || val > 1) + return -EINVAL; + + if (val) { + kmmscand_scan_enabled =3D true; + need_wakeup =3D true; + } else + kmmscand_scan_enabled =3D false; + + kmmscand_sleep_expire =3D 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute scan_enabled_attr =3D + __ATTR_RW(scan_enabled); + +static ssize_t target_node_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_target_node); +} + +static ssize_t target_node_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int err, node; + + err =3D kstrtoint(buf, 10, &node); + if (err) + return -EINVAL; + + kmmscand_sleep_expire =3D 0; + if (!node_is_toptier(node)) + return -EINVAL; + + kmmscand_target_node =3D node; + wake_up_interruptible(&kmmscand_wait); + + return count; +} +static struct kobj_attribute target_node_attr =3D + __ATTR_RW(target_node); + +static struct attribute *kmmscand_attr[] =3D { + &scan_sleep_ms_attr.attr, + &mm_scan_period_ms_attr.attr, + &mms_to_scan_attr.attr, + &scan_enabled_attr.attr, + &target_node_attr.attr, + NULL, +}; + +struct attribute_group kmmscand_attr_group =3D { + .attrs =3D kmmscand_attr, + .name =3D "kmmscand", +}; +#endif + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -1036,9 +1201,43 @@ static int kmmscand(void *none) return 0; } =20 +#ifdef CONFIG_SYSFS +extern struct kobject *mm_kobj; +static int __init kmmscand_init_sysfs(struct kobject **kobj) +{ + int err; + + err =3D sysfs_create_group(*kobj, &kmmscand_attr_group); + if (err) { + pr_err("failed to register kmmscand group\n"); + goto err_kmmscand_attr; + } + + return 0; + +err_kmmscand_attr: + sysfs_remove_group(*kobj, &kmmscand_attr_group); + return err; +} + +static void __init kmmscand_exit_sysfs(struct kobject *kobj) +{ + sysfs_remove_group(kobj, &kmmscand_attr_group); +} +#else +static inline int __init kmmscand_init_sysfs(struct kobject **kobj) +{ + return 0; +} +static inline void __init kmmscand_exit_sysfs(struct kobject *kobj) +{ +} +#endif + static inline void kmmscand_destroy(void) { kmem_cache_destroy(kmmscand_slot_cache); + kmmscand_exit_sysfs(mm_kobj); } =20 void __kmmscand_enter(struct mm_struct *mm) @@ -1223,7 +1422,13 @@ static int __init kmmscand_init(void) return -ENOMEM; } =20 + err =3D kmmscand_init_sysfs(&mm_kobj); + + if (err) + goto err_init_sysfs; + init_list(); + err =3D start_kmmscand(); if (err) goto err_kmmscand; @@ -1239,6 +1444,7 @@ static int __init kmmscand_init(void) =20 err_kmmscand: stop_kmmscand(); +err_init_sysfs: kmmscand_destroy(); =20 return err; --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2066.outbound.protection.outlook.com [40.107.236.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0123921481E for ; Wed, 19 Mar 2025 19:32:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412778; cv=fail; b=oz4J7nh+pdgHKNP1Ll8jPNHAscj5H3PX6mbrREeWSK4KhkujAMCFlQATrcTyN9m5YDr3qN2th2oT6AOu7DSNbp2i6gDnZMq2023iMDdHOgVrDoJfosejOL1Y4NDp+uwTfbNuicYWo4OjZU/5SEUrw1wMLZzsNJy2+exoJiCVcKU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412778; c=relaxed/simple; bh=vqhVZEqfzvz64eoLRfrl0/5zZCtF0jAWVjQKYxO58sc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=X8G/9eGlY1O26GmwuNuqJwMbHV0z0jpbrMYCC5BZKImFklpC+m8vm85a7P9aVjO/wzUHTULFVoRg28kkWXcvL656f6Z+FGQ1D7g9cw6JenpI1MnRWuETqoMdKSLelV8joOXoibjvRfrCYpAOMcnqpcSruhbHJrOdrDC1hideB6w= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=ZiIXnefF; arc=fail smtp.client-ip=40.107.236.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="ZiIXnefF" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tcrRp8ibA1L8lZ/lJnU4UdcQBLzd/+9sMG+ccYx7+EaPlqpbtKCJICX46n2lR04M9nnZ3vf7CrY9mPh/gunz+BCmqJliF5PGmivbUDU2XTd1v6giuRWaTQalSKjMA4CLeg+lpOJKradrjhQTfnhPT9aHJzbsyL2Zb/bZn031Zh4LaRC+SnUk9dNxl/iH0yq5kfrFfFwERNcPFvojEgFbs5n+1HYf2Pwugm8226fGLOYHXTUEgJ/T5H4UxA2yUW/8klPVgmd8nplT4/P+QZLOih2tOBSSUu+lPxfFdz8/Eozu+TTzHMK3FZPeqYDlH2gUQd34f6YY/advwepsO1cceg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sMsrH2gwsvfzOvnhHaikDdLZ0RBu7hhS898jqAkzLyg=; b=NrJGMPqUjR8GANHmbt7fv8A8LFVqXVUbhOO34OKSPX41wf7xZcpjQms2qOl+gH8VCgsV8rduWI3ApVygX2qO03vDqg+kx+x3jW5ot+ezAi+PiqUYE5mj+LAgyHlVacqk0VunZMEtXew+FXm6dHIVeVOalXLstbR0xKZYvjgV2xTRy2RE4WNrrMmmnI7HUWLEu6sLZOOiJYjpiaWT429ROKLH6rNlFmzSguJgsZBDGL4hIUsapsFxk3BVVxuesec7QHYgtSeiqc0QfjNxVbT5YvHCC0/5pT5z2aPKtSQzrAfOmkdWjy1sQCLb361WPx+0Zecyr6IJRsN6x/cx66lWnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sMsrH2gwsvfzOvnhHaikDdLZ0RBu7hhS898jqAkzLyg=; b=ZiIXnefF1S/ioqWWZlOnbYguX0sEyJgPjFz3aZWjvwM7Wy5HE//UoVmGR2sKYnyvWuLAs4YtmIbXnADJT4A5rhexLE9gbmmWEu3jwnyJ3UaBcnLdWztiSMnJQvPoZHrcxLnnCuqSFq4+tdmLoAUMfO18sAeFpNLUuLEg00VpePs= Received: from SN7PR04CA0024.namprd04.prod.outlook.com (2603:10b6:806:f2::29) by CH2PR12MB4102.namprd12.prod.outlook.com (2603:10b6:610:a9::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:51 +0000 Received: from SN1PEPF0002636B.namprd02.prod.outlook.com (2603:10b6:806:f2:cafe::2f) by SN7PR04CA0024.outlook.office365.com (2603:10b6:806:f2::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:32:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636B.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:51 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:41 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 11/13] vmstat: Add vmstat counters Date: Wed, 19 Mar 2025 19:30:26 +0000 Message-ID: <20250319193028.29514-12-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636B:EE_|CH2PR12MB4102:EE_ X-MS-Office365-Filtering-Correlation-Id: 187ab32e-35b4-463c-0133-08dd671cd69c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?388euDnwXyJ8MZTQe0dJ4kEIxumWHeEgVXY4HIQTOOofsXvWc6We2QmB0Vvy?= =?us-ascii?Q?H07J2lUgKahX1szjHK4TErKl3Mx6UfEeprEdcCJLkIqcu0QHMBjo2Hbqd0fq?= =?us-ascii?Q?xULwriLDR38n2JU5sJSacVQhW2It7nrzRgFDS++UCnPxMmEisDXRfnxYkAMw?= =?us-ascii?Q?FGl77Nx30wj1Ls767ZMYhLnRY9ElzxGjnkEJIn0CR/tYQB8LHKSlOl0A6FHT?= =?us-ascii?Q?TibU1qRtoPAS49sMqHPNAjk8+Uo97Yjd0jpU8a5zSfyBRAVixs5g0k53DI8U?= =?us-ascii?Q?e3OcD/AtwEZKLNXtPk4MtTDcp6W+yysj4z3AQ4jUGP03J/C9cR1C40dc6+4b?= =?us-ascii?Q?1kv8sXrnDQR66josTGcSLJumFuVN9tmm/Dbg7YTTWv3afP1gMGmPXonMXDBm?= =?us-ascii?Q?wosibZRuCRKpQ3qxU07bUV8a/ikJsTcwczVENdr5NzZRZbjpWxSkhWeAyzHz?= =?us-ascii?Q?/qVawpa9QXhSK9eZ2Mot9sVXI1iP2Ha9jXlBkAsyRGZEsGT4MrgIK0MtMA4+?= =?us-ascii?Q?inktlHgCmvJbv2QkVr+6h6Pd7IxnAviuK2KAJD6odTP7JTs35ouU0RGKUh4Z?= =?us-ascii?Q?lhoOmkS19GR4iAPP4ZXDej8bvdZ7ik92XrbboibiDyG1oR7XM4cbpRYnc04E?= =?us-ascii?Q?wAvXyEAbHZYq4e6hOD/MAbMKKSPtqIoPz+2b3KhPRfNjg+2fM3/clWltBUw4?= =?us-ascii?Q?FsrHgR+/2PqSiKp/DRmcFo0O2wYKHPtbobAZwQhzrBSfZySlcANOI1nPeYLK?= =?us-ascii?Q?SwpbpRYpCzITbIhXUybeAXMjtWcKdjlDJ+ExlGD9qyauhrtTu+LFZ0Aq1wLf?= =?us-ascii?Q?/2Rz3RPKgOgyMUiC7rnwgFXnklQLZisd5DgPRgjBSckq4YQrcOSSACW/G9cP?= =?us-ascii?Q?pLnYEdHZL4wyZ5W2BfKYYfMWsUp5ij9cYDb4T8nC0GN9PXFyqumYkDBi6KjU?= =?us-ascii?Q?atNmWChBShsFyFRmcVXfg83P0GrIGLhH/Lf2H5qOPIbxXey0LpwWxMSLdnms?= =?us-ascii?Q?40tBGOabFRNrbLJM8OkZiXfO6q2WjCCrwj0U2VW+Fr/oaxm7XB1qJT575ybg?= =?us-ascii?Q?csZ9KaL5YtU35yeFi3a9nisgsl3pFIjijSvGcx/IeLFup5ez3lsHE02V/ZGB?= =?us-ascii?Q?54pmqIF/II1xr3TOPX7akjlwsUdglLFUPqwYFTI9TPBBZDod8J2PTeBG/Z2A?= =?us-ascii?Q?TvdPh7ktUyAqdq2jrvmowdcOotOCP6synpx/rciBBg5o3AT3g3zMmVWntniM?= =?us-ascii?Q?GpQa214qQ5PGYPzpuXJPxWlAirBYTZNZMdXH52h4+gOHDzEq3L+kCEpGhs9/?= =?us-ascii?Q?Yko6Ac2zK6Ik5TU8Gs65HOAzhbrPsRazz13Ud2dKbwyTpcWtax+DCqsABQ9k?= =?us-ascii?Q?KKpgGj0ztja9TqisYXMolo1ytl6KlJtvIO2Ydk9sgowvq+X8zZ+UvfuwI8eJ?= =?us-ascii?Q?pVMhHu2Zs9cXdQZtMO9yJNp4WhRe1deO2Te+id6IOnLo4GZxAidDLNTNOZBM?= =?us-ascii?Q?bNmjE3s/ySHpwOE=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:51.3146 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 187ab32e-35b4-463c-0133-08dd671cd69c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4102 Content-Type: text/plain; charset="utf-8" Add vmstat counter to track scanning, migration and type of pages. Signed-off-by: Raghavendra K T --- include/linux/mm.h | 11 ++++++++ include/linux/vm_event_item.h | 10 +++++++ mm/kmmscand.c | 52 ++++++++++++++++++++++++++++++++++- mm/vmstat.c | 10 +++++++ 4 files changed, 82 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b1068ddcbb7..e40a38c28a63 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -682,6 +682,17 @@ struct vm_operations_struct { unsigned long addr); }; =20 +#ifdef CONFIG_KMMSCAND +void count_kmmscand_mm_scans(void); +void count_kmmscand_vma_scans(void); +void count_kmmscand_migadded(void); +void count_kmmscand_migrated(void); +void count_kmmscand_migrate_failed(void); +void count_kmmscand_slowtier(void); +void count_kmmscand_toptier(void); +void count_kmmscand_idlepage(void); +#endif + #ifdef CONFIG_NUMA_BALANCING static inline void vma_numab_state_init(struct vm_area_struct *vma) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f70d0958095c..b2ccd4f665aa 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -65,6 +65,16 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, #endif +#ifdef CONFIG_KMMSCAND + KMMSCAND_MM_SCANS, + KMMSCAND_VMA_SCANS, + KMMSCAND_MIGADDED, + KMMSCAND_MIGRATED, + KMMSCAND_MIGRATE_FAILED, + KMMSCAND_SLOWTIER, + KMMSCAND_TOPTIER, + KMMSCAND_IDLEPAGE, +#endif #ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, THP_MIGRATION_SUCCESS, diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 618594d7c148..c88b30e0fc7d 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -323,6 +323,39 @@ struct attribute_group kmmscand_attr_group =3D { }; #endif =20 +void count_kmmscand_mm_scans(void) +{ + count_vm_numa_event(KMMSCAND_MM_SCANS); +} +void count_kmmscand_vma_scans(void) +{ + count_vm_numa_event(KMMSCAND_VMA_SCANS); +} +void count_kmmscand_migadded(void) +{ + count_vm_numa_event(KMMSCAND_MIGADDED); +} +void count_kmmscand_migrated(void) +{ + count_vm_numa_event(KMMSCAND_MIGRATED); +} +void count_kmmscand_migrate_failed(void) +{ + count_vm_numa_event(KMMSCAND_MIGRATE_FAILED); +} +void count_kmmscand_slowtier(void) +{ + count_vm_numa_event(KMMSCAND_SLOWTIER); +} +void count_kmmscand_toptier(void) +{ + count_vm_numa_event(KMMSCAND_TOPTIER); +} +void count_kmmscand_idlepage(void) +{ + count_vm_numa_event(KMMSCAND_IDLEPAGE); +} + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -769,6 +802,9 @@ static int hot_vma_idle_pte_entry(pte_t *pte, return 0; } =20 + if (node_is_toptier(srcnid)) + count_kmmscand_toptier(); + if (!folio_test_idle(folio) || folio_test_young(folio) || mmu_notifier_test_young(mm, addr) || folio_test_referenced(folio) || pte_young(pteval)) { @@ -784,14 +820,18 @@ static int hot_vma_idle_pte_entry(pte_t *pte, info =3D kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { =20 + count_kmmscand_slowtier(); info->mm =3D mm; info->address =3D addr; info->folio =3D folio; =20 /* No need of lock now */ list_add_tail(&info->migrate_node, &scanctrl->scan_list); + + count_kmmscand_migadded(); } - } + } else + count_kmmscand_idlepage(); =20 folio_set_idle(folio); folio_put(folio); @@ -907,6 +947,12 @@ static void kmmscand_migrate_folio(void) ret =3D kmmscand_promote_folio(info, dest); } =20 + /* TBD: encode migrated count here, currently assume folio_nr_pages */ + if (!ret) + count_kmmscand_migrated(); + else + count_kmmscand_migrate_failed(); + kfree(info); =20 spin_lock(&kmmscand_migrate_lock); @@ -1083,6 +1129,7 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 for_each_vma(vmi, vma) { kmmscand_walk_page_vma(vma, &kmmscand_scanctrl); + count_kmmscand_vma_scans(); vma_scanned_size +=3D vma->vm_end - vma->vm_start; =20 if (vma_scanned_size >=3D mm_slot_scan_size || @@ -1108,6 +1155,8 @@ static unsigned long kmmscand_scan_mm_slot(void) =20 update_mmslot_info =3D true; =20 + count_kmmscand_mm_scans(); + total =3D get_slowtier_accesed(&kmmscand_scanctrl); target_node =3D get_target_node(&kmmscand_scanctrl); =20 @@ -1123,6 +1172,7 @@ static unsigned long kmmscand_scan_mm_slot(void) kmmscand_update_mmslot_info(mm_slot, total, target_node); } =20 + outerloop: /* exit_mmap will destroy ptes after this */ mmap_read_unlock(mm); diff --git a/mm/vmstat.c b/mm/vmstat.c index 16bfe1c694dd..3a6fa834ebe0 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1340,6 +1340,16 @@ const char * const vmstat_text[] =3D { "numa_hint_faults_local", "numa_pages_migrated", #endif +#ifdef CONFIG_KMMSCAND + "nr_kmmscand_mm_scans", + "nr_kmmscand_vma_scans", + "nr_kmmscand_migadded", + "nr_kmmscand_migrated", + "nr_kmmscand_migrate_failed", + "nr_kmmscand_slowtier", + "nr_kmmscand_toptier", + "nr_kmmscand_idlepage", +#endif #ifdef CONFIG_MIGRATION "pgmigrate_success", "pgmigrate_fail", --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2045.outbound.protection.outlook.com [40.107.223.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86E0D2135D8 for ; Wed, 19 Mar 2025 19:33:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.45 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412789; cv=fail; b=sPibLcUXUPxL61vIojoP5iPgyZy90aGVt0NEh/Y0P11gTIzBCqHRDmRj/+a20F3XVMQyy9pShh+bZOtwzfgV9HOnU+FFKPdQULzn6WwEXAGsq0lNA7Ke/Wi5IjJkBWLC5z3DEAXmdcFfl5+hZwFsBPB6VQEgP4EgXYGofic5I1Y= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412789; c=relaxed/simple; bh=48hgjTBd08olsfQeQ9o0gpLsz9RucJiCP+WrMnIXhto=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=G2dfu1cPiwjlF5fiTJcke/193BplzWUhwdXCaspWIsOi4B7v8R6FTon0M756GOVzErJ2AVoaADREEspwfWXcxUMEKzofb2aPe4G/2ZCELveRBsOX7+CTmaMKpdi2ikZ6Cxn+7iBYLB3Y5C7PHImpUfMguZA+LDjLqakh8vtIFQM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=EmXCUdck; arc=fail smtp.client-ip=40.107.223.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="EmXCUdck" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HbThxnNT3sDyTD+HFT8KeE/eohImjEKvRngChsKKkTiOtbkfhnCsOUhyfoHG5flfIeMJ9GJ3kev9wIp2MV6li87aQ5WdPwiH0APTpVN/bqjvJkIWF/NPhVZPEQiAD8aG6YJhNL+lAtd+DMnCMPFiv6E8oWpMOAhDSy4t27U4viaovusMxhQQl8pw/pnx6RHO5svpnvaMpS4PyCAtbBJHPflllzol/aFDi8rp9ZUX4ocZYFRV1Pkr9YKmKf3Gp8wIYuJnPjxijP9njQjpyprmS+HViIjhC8YOaBFVm6XTgM/CGpargSihJgE/t11rxRDWmeU7LQoBfM1UA52Mw8PkYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oYUB9Y5oL/vnLd9PbKivHKRUxW1ZVjk0QHYKJpkjL8Y=; b=d+Z5ZmzGtq2Zq6ArdCC2/XgRAbBUugeKcm4QVHW0muQxgrVac4WzVvzUtRGkTzglIVsGE2RTinA9Ft4acsyrT2U913MpBMM84r31RcAwUdT5MR8B5c3/z/KuIdUP1S995RQl/ezcFxxLr0hpsNm4D1ZPjWEbrz8Yxi/+x3NcLbc1Qf1iyLk0uMtnJGxxgfN6zZltc8Pnm+9t85fB9ctXDeCKQC1LfTc/HfS87YFPrSLMIKxF3ESIloze/eG3cyN+ePWscAcKpEF6e+Z1Y9gNljiSfXrwub/uFVtIG/sQ0gSv82yIVf40m38dm41ePpbmxyIpUofedKPezq1VW/etYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oYUB9Y5oL/vnLd9PbKivHKRUxW1ZVjk0QHYKJpkjL8Y=; b=EmXCUdckkV9nMB7LnWlYx3I9uMajNBcX/RNm0oTUmK9BnLf5uKbp/MEyk8vd9vjpGvB82yNLWFDmJSQVhOM90DuocRFGE7kPrAd69iDlH4/a1AUL/EGgg09NMvmBEP5ssxUOf4VgzMASJgdj+mGVe39yRDaVP0nonGqdAOy7VoQ= Received: from SA0PR11CA0169.namprd11.prod.outlook.com (2603:10b6:806:1bb::24) by PH7PR12MB5925.namprd12.prod.outlook.com (2603:10b6:510:1d8::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:33:02 +0000 Received: from SN1PEPF00026367.namprd02.prod.outlook.com (2603:10b6:806:1bb:cafe::1e) by SA0PR11CA0169.outlook.office365.com (2603:10b6:806:1bb::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:33:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026367.mail.protection.outlook.com (10.167.241.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:33:01 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:51 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 12/13] trace/kmmscand: Add tracing of scanning and migration Date: Wed, 19 Mar 2025 19:30:27 +0000 Message-ID: <20250319193028.29514-13-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026367:EE_|PH7PR12MB5925:EE_ X-MS-Office365-Filtering-Correlation-Id: 868d5814-7e1f-409a-7c98-08dd671cdc6e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?NzZ3jZA9+096bwOrh/i5UbiNtTMFuIT766qHEUdYOmOvR4TiAetx8Mlu3EKd?= =?us-ascii?Q?UhdDMc0RVMbHvtNrv08eGYJQF+24uL/n3D1hdgahLYysmOYw7IoCv95z4hbt?= =?us-ascii?Q?7KvmpwQr+4pBZrOHs8qtVa2pLjXX6hzWzaNZK1t3drwEXFHKdc3SV0UoXSej?= =?us-ascii?Q?kgC90qBVm+npYY3tqGK4ANsMkcX5TuffSPGqyhjezZiXN78YZdCREpya+0uZ?= =?us-ascii?Q?+OfWmuv9BBdy3JLRviNZMXnVBfikXvsjUCV2srPGA0TUNsQ6u/HUFTGbMZGz?= =?us-ascii?Q?30lek8Oprc4ktjb++9Oao6IJMTlLvuzNE1qTIDb1Jk92jNE7imGv4vEAAbOa?= =?us-ascii?Q?s5zU+0jaDbkA0vbwq/UJgKcZcnx7X59G+L+fdoViDJZgPOFrFpaY3lYXF9BG?= =?us-ascii?Q?LUKZRv65xxrv6X62ijXKoHyWai5TQwpEGhSRrZy5T+vIHlbnIwi5LaHBlTuH?= =?us-ascii?Q?VRS/8KrYrCjHkXdwM487lnpcXeZhHPsJ4lYybw5O/KbX6nTK8U5fQl0sGbF1?= =?us-ascii?Q?lu9qKupdlwaEgfAqAdZTWibhzykoriPJEpNvxAT1h/vBB/Y0xqFLUiqtUBIB?= =?us-ascii?Q?awE+8QwR3s4hd+uH4W3G/+mUYgmZOU/zc/5hkaT5aSGSeXOitNsETPdm/8KD?= =?us-ascii?Q?eMs2SSGbjiffR1nmiqZ3ii9hU8SKdwf+mu4mpIQMk+Xgjycgd6G1KY1MCCTs?= =?us-ascii?Q?Ft3OKqy7od50zlw6pVzk3W3VUBgpaITpLs3o04RmLqSPFyOQ69xrTP0us6AW?= =?us-ascii?Q?WJr4eA8bnCoy0TkreVPL82SLRXW0niyTlBWTp4+/9vdYXHGhep3R7fKKYsrd?= =?us-ascii?Q?0DYf6HIghTIBHoXuooEDQHLH5O9Po5GaeaS1zXr2MznQEEXKJboP2pDoI539?= =?us-ascii?Q?HIRK27uwzxybBP6wfGnSyRyxx3vUQsb6g83r8s3ctUwJxBAME7J3BjXUISIm?= =?us-ascii?Q?ZrJINnIFQV2q4hBCHG0D8uL6JWo9H8SI3QrMvudne70xX12dGFcmvtpU6IeJ?= =?us-ascii?Q?TaYixhoIRjQ6nT0ew1yO2d5D8W0mSX1AqljvpSZWTUssPcW5HazaNbYPtLWo?= =?us-ascii?Q?t4F4swUMtJCcoliFtdONyae3TcJE4SutHclnJDUGyki7JY+3+Krm7v2f2sXM?= =?us-ascii?Q?6bg6RWNnXErgneRcCjipBOpU5jmyEQo2axjybSrb//M0ZZRmvuYmm2NeEqKv?= =?us-ascii?Q?OWmriHj81LODuV/GrYb5UFYZz3UKvRuU4WRKb9HJfq285t2lwR5gAEXyV2ES?= =?us-ascii?Q?BnGEmD0G3fCfjy6vDgY7KTqGHmfYFD4Yof80SGPX3wOMx+kDVtLRoBbk5XeM?= =?us-ascii?Q?ncVqH1noXshHKJ2v1DxabaT8Zjd1NYD336V3ToFTsP9uZa4JGpAEy/U0E15N?= =?us-ascii?Q?0NK9easAccvRAs+QB8QjKCrxsyOoRfoloEyvbo3uKR5fNvBA6m+t44o46KrU?= =?us-ascii?Q?EdycAqcoXEUzBbqLadsWw343ar8f8fcSyiZ75ixBCOo5sP/bOPhH2lPvR+cN?= =?us-ascii?Q?OQedkDWCFTTHUnQ=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:33:01.0827 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 868d5814-7e1f-409a-7c98-08dd671cdc6e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026367.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5925 Content-Type: text/plain; charset="utf-8" Add tracing support to track - start and end of scanning. - migration. CC: Steven Rostedt CC: Masami Hiramatsu CC: linux-trace-kernel@vger.kernel.org Signed-off-by: Raghavendra K T --- Changes done based Steves feedback: 1) Using EVENT class for similar traces 2) Dropping task_comm 3) remove unnecessary module name in print include/trace/events/kmem.h | 90 +++++++++++++++++++++++++++++++++++++ mm/kmmscand.c | 8 ++++ 2 files changed, 98 insertions(+) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index b37eb0a7060f..cef527ef9d79 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -9,6 +9,96 @@ #include #include =20 +DECLARE_EVENT_CLASS(kmem_mm_class, + + TP_PROTO(struct mm_struct *mm), + + TP_ARGS(mm), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + ), + + TP_printk("mm =3D %p", __entry->mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_enter, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_exit, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_scan_mm_start, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +TRACE_EVENT(kmem_scan_mm_end, + + TP_PROTO( struct mm_struct *mm, + unsigned long start, + unsigned long total, + unsigned long scan_period, + unsigned long scan_size, + int target_node), + + TP_ARGS(mm, start, total, scan_period, scan_size, target_node), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( unsigned long, start ) + __field( unsigned long, total ) + __field( unsigned long, scan_period ) + __field( unsigned long, scan_size ) + __field( int, target_node ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + __entry->start =3D start; + __entry->total =3D total; + __entry->scan_period =3D scan_period; + __entry->scan_size =3D scan_size; + __entry->target_node =3D target_node; + ), + + TP_printk("mm=3D%p, start =3D %ld, total =3D %ld, scan_period =3D %ld, sc= an_size =3D %ld node =3D %d", + __entry->mm, __entry->start, __entry->total, __entry->scan_period, + __entry->scan_size, __entry->target_node) +); + +TRACE_EVENT(kmem_scan_mm_migrate, + + TP_PROTO(struct mm_struct *mm, + int rc, + int target_node), + + TP_ARGS(mm, rc, target_node), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( int, rc ) + __field( int, target_node ) + ), + + TP_fast_assign( + __entry->mm =3D mm; + __entry->rc =3D rc; + __entry->target_node =3D target_node; + ), + + TP_printk("mm =3D %p rc =3D %d node =3D %d", + __entry->mm, __entry->rc, __entry->target_node) +); + TRACE_EVENT(kmem_cache_alloc, =20 TP_PROTO(unsigned long call_site, diff --git a/mm/kmmscand.c b/mm/kmmscand.c index c88b30e0fc7d..38d7825c0d62 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -945,6 +945,7 @@ static void kmmscand_migrate_folio(void) } =20 ret =3D kmmscand_promote_folio(info, dest); + trace_kmem_scan_mm_migrate(info->mm, ret, dest); } =20 /* TBD: encode migrated count here, currently assume folio_nr_pages */ @@ -1115,6 +1116,9 @@ static unsigned long kmmscand_scan_mm_slot(void) mm_target_node =3D READ_ONCE(mm->target_node); if (mm_target_node !=3D mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node); + + trace_kmem_scan_mm_start(mm); + now =3D jiffies; =20 if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -1172,6 +1176,8 @@ static unsigned long kmmscand_scan_mm_slot(void) kmmscand_update_mmslot_info(mm_slot, total, target_node); } =20 + trace_kmem_scan_mm_end(mm, address, total, mm_slot_scan_period, + mm_slot_scan_size, target_node); =20 outerloop: /* exit_mmap will destroy ptes after this */ @@ -1323,6 +1329,7 @@ void __kmmscand_enter(struct mm_struct *mm) spin_unlock(&kmmscand_mm_lock); =20 mmgrab(mm); + trace_kmem_mm_enter(mm); if (wakeup) wake_up_interruptible(&kmmscand_wait); } @@ -1333,6 +1340,7 @@ void __kmmscand_exit(struct mm_struct *mm) struct mm_slot *slot; int free =3D 0, serialize =3D 1; =20 + trace_kmem_mm_exit(mm); spin_lock(&kmmscand_mm_lock); slot =3D mm_slot_lookup(kmmscand_slots_hash, mm); mm_slot =3D mm_slot_entry(slot, struct kmmscand_mm_slot, slot); --=20 2.34.1 From nobody Wed Dec 17 05:28:28 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2086.outbound.protection.outlook.com [40.107.244.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02759214A9E for ; Wed, 19 Mar 2025 19:33:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.86 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412797; cv=fail; b=q9sSEHsUIBWVeK9/M274Rek1lHL8iAF5n2hL/nCRFP+ZtzAzTHOxdSMlu/sAtB1pT6CS6+V+fP04bUG+DG71MW4ABn48FV/UvzgwUfesQ7/YatDXlTh151fWDNHclbkh5oB9C8ObhwY/mmrdeoZn0Wo98Cj9RbkDvpYteIjv+zo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742412797; c=relaxed/simple; bh=iLZh5xxDHXbPPPGG5TtdGuIpB+Px9YVtKEdylvQm7Z0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=M5TkjHIvcBul5/qLcUNWZqEoHo3rPUEQet6dEUICXQX1VOM3NFp6/nCSPDijItq8VofYAOZFky6UaLHHsLELH1fdqDcgy4DKMTX3B90y8Bqh63Qrf3T2SgcyAuN8YunTqskpFnrGYYXGAE+Q+e0p41FpAul6LDxV2apksjCjybE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QFBJeEhH; arc=fail smtp.client-ip=40.107.244.86 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QFBJeEhH" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TN3lCGAF5u8M5d6j/wPcIe6UjgTbf7gbVirJ3yFuPlpTvr37+duBAwsAhOy0A00oSigtCjG1PdT7VyH/0cBYRSaEdKjk8exwQpCBJJFxKKeWdUduIuNE4t2wJUZhb70C6tOtXUk+kx/8OSmWQt5rMfXZv6Xseo9pK6VVeNeSg0318QnNRPi4wxxw32jDBGtNvPNrde1gF+yPau0kQlLd+p01s3P3X9wuQ27yhtvtvnQZ2VbzSZ1fl+/7wxEdKQnFWkZT/7Bhwgdw3tcARlKlP9Jlb5HitiZ0IVejIw41dQrzl2by3f/BFnSQUzoajBkJISKDKbqHGX8aIrBqI49m/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=krPBi94jIB/Baadfzed9TBHF0/sXB2vjDdvQbhvxMPQ=; b=jngXRrtDe8q9SGFkMpeyt0zkbPP7kzd0tMEMlyUunPcdyBJZcBrk8xMNXXMLLAF7CSEl2OJWviz5iVXIwik+7zJcZbC1v9HjZuwEYtU5QK0NzIo11OIUuWovEfajYqQR8upVtjFcwvDU8grTMEmF4SfOlWYln/IBNtEvgUF7bcvqFeklPNCUOVlN3nBvqnmH1XCjKRVXkRRvFycq3T1hj8I5B9b4SF+nmLorzEDKLHzi9FvvnFZnPryyrm8Un2YCd3CLDFsKayFSqOLQk7dMtHHuXYDf2vHbx+AwtOv2d3rTzEQ97+WTcYI5aOPT5IS7w8hp0Ggzbc6o2UAgs66FYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=krPBi94jIB/Baadfzed9TBHF0/sXB2vjDdvQbhvxMPQ=; b=QFBJeEhHLhhXvWKPNvlPnlh6M71JbF76bfBmp1k45jZzMn2x5SOqHdkTxr39C3poXbTuOqE0gUqePuaJg3yAXsDLJS/1F+8dZdjeQz7RP99+ztdjA3SJwWqcJKrVJ5hNOgnn9pUg7K8AydC0W4027/KbfQzZTI4HoyF9fv0+rKw= Received: from SN7PR04CA0018.namprd04.prod.outlook.com (2603:10b6:806:f2::23) by SA0PR12MB4383.namprd12.prod.outlook.com (2603:10b6:806:94::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:33:11 +0000 Received: from SN1PEPF0002636B.namprd02.prod.outlook.com (2603:10b6:806:f2:cafe::22) by SN7PR04CA0018.outlook.office365.com (2603:10b6:806:f2::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:33:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636B.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:33:10 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:33:01 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 13/13] prctl: Introduce new prctl to control scanning Date: Wed, 19 Mar 2025 19:30:28 +0000 Message-ID: <20250319193028.29514-14-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636B:EE_|SA0PR12MB4383:EE_ X-MS-Office365-Filtering-Correlation-Id: 9bbe0152-2683-452a-ef4b-08dd671ce250 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?SX9G7uYor+8ZG/8teWgp8/APefddEfHP2qYFZRz/ZQ/7AlQvE2pt2uXS4aD/?= =?us-ascii?Q?4+F7a26mAcztlNXxfyKd6/UTY1YSJsNRyZ+HMZRBDB2YiyBGVH8VXo6KUSvq?= =?us-ascii?Q?2YPowAeK5jn9QG4jAz752YBYz3Oyrbysl5EeYOQY0f4AltB3mVNDeUGvDRaw?= =?us-ascii?Q?aVOAmSu81naqE4/ThJSw0wTOrEcO2PGRZ91SODg6Rrjo0zViom2te/F7us51?= =?us-ascii?Q?+tCJSdWlBNMhDOw+zKm9MvaAtgtDp/KAwAwiG2jZI0jLphXvb4ptdvuQIdwz?= =?us-ascii?Q?3cD++R9VLlXWpvbcFt+Bvfz4/N6Bbmkv4ji7/n7+xrfz3RTkjVng4eyN2Amc?= =?us-ascii?Q?vdXgGacy89uOOX5PPZ4w0L7ubROVKVVZl2k/U/7Zr8OjCBlnh5inyoFI/Di0?= =?us-ascii?Q?nwI7c0tzwH3zI0DAFocWJwPY0wzFieSAkrODMWFfp6DhRMS5OI8YsfC+smPM?= =?us-ascii?Q?HgmRhtU4/xomLtAYBmqxX976AGM/y9qj7uYW16sAbC7zby/zHIiK0RluzhA0?= =?us-ascii?Q?p7QLT/7wIPVWn4K50VXw2xopP3noPnyFdhKpxY9bPtx/odUKCJJ4Mp3oCWJO?= =?us-ascii?Q?XhZA0aIjkXaaWRDKB5lUoKqtin8i3F4IpJsrFlqDRCSP6Gv8h1EJRVNCghB1?= =?us-ascii?Q?4IStbbolPHn9ebuLDI+hp56c/ifrXRfgvHCQApPmq2bDYSfpih5VGm3qrnl1?= =?us-ascii?Q?h+X3M4F3olD9UWv3zCDLOJksydmo7THiKy+rnkQegI27+WABQDbtb5/zo3Ey?= =?us-ascii?Q?1eyj53C+ImlcURKJtDTQDNQ5V6OlpE0Y7WNNNVPbcTRwJHM47+uxL8TmygMq?= =?us-ascii?Q?Ll2eQLaxpD5/dtNxAqfIM9ElPw/79Dn+NCN02BTDDrAto+RRvbXxfi8D8EVg?= =?us-ascii?Q?crfPyPBlymkxChwrgjJgxTwHAm2TdleOMuWsbh2rrkQQhTWmVoBHNSu7Z0fR?= =?us-ascii?Q?RKmo0CzKdmNTjKbUPf7R3Dp7L3npAZzFTfcKMI69H48FztAcDt66J/sT6You?= =?us-ascii?Q?Doln5ZjGpKBffw7sEubPMuRqXOSefTBL4F+kFsfzUCFGPPYrV5L1q1gT0yiA?= =?us-ascii?Q?yAw++cMd9p58DENZ+rVxC0Gpjp96QT1ZOD1tg1LIrPMgah/hv6f/JntKSXBl?= =?us-ascii?Q?55TnMGRXVGoi8xgO90cH8DK9W73KqwaClA8hoYLS0TcmJhK946VWpJeCjGl9?= =?us-ascii?Q?AxExVl7yZiyqydy0diCCjiDoxJkvw8rjONsMg6Di9KlLH68T8EzuCCXB3w2V?= =?us-ascii?Q?62D+X8fnxUKCniSM1FztJRdk8wBtwTSz1Xb9onIGXv/ri/Wblz6GsGGGVg8j?= =?us-ascii?Q?zwb9zK2b14q2KvA+ntU0K6paSnxoFTAoXn2ze1EpCpXB2cOheo/vqYZ8nb5f?= =?us-ascii?Q?KTuG6cgqnutf+wX93cMrsDzG8FUvm8dEfmz8rY6ZScxhKZqSJ8vs8E3H/Cj/?= =?us-ascii?Q?C63+qQDsgiGFZC439GsXkZchi/gsn3A6OQwAwtA9FoSD3qn+qV1sXw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:33:10.9241 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9bbe0152-2683-452a-ef4b-08dd671ce250 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4383 Content-Type: text/plain; charset="utf-8" A new scalar value (PTEAScanScale) to control per task PTE A bit scanning = is introduced. 0 : scanning disabled 1-10 : scanning enabled. In future PTEAScanScale could be used to control aggressiveness of scanning. CC: linux-doc@vger.kernel.org=20 CC: Jonathan Corbet CC: linux-fsdevel@vger.kernel.org Suggested-by: David Rientjes Signed-off-by: Raghavendra K T --- Documentation/filesystems/proc.rst | 2 ++ fs/proc/task_mmu.c | 4 ++++ include/linux/kmmscand.h | 1 + include/linux/mm_types.h | 3 +++ include/uapi/linux/prctl.h | 7 +++++++ kernel/fork.c | 4 ++++ kernel/sys.c | 25 +++++++++++++++++++++++++ mm/kmmscand.c | 5 +++++ 8 files changed, 51 insertions(+) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems= /proc.rst index 09f0aed5a08b..78633cab3f1a 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -195,6 +195,7 @@ read the file /proc/PID/status:: VmLib: 1412 kB VmPTE: 20 kb VmSwap: 0 kB + PTEAScanScale: 0 HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 @@ -278,6 +279,7 @@ It's slow but very precise. VmPTE size of page table entries VmSwap amount of swap used by anonymous private data (shmem swap usage is not included) + PTEAScanScale Integer representing async PTE A bit scan agr= ression HugetlbPages size of hugetlb memory portions CoreDumping process's memory is currently being dumped (killing the process may lead to a corrupted = core) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index f02cd362309a..55620a5178fb 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -79,6 +79,10 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) " kB\nVmPTE:\t", mm_pgtables_bytes(mm) >> 10, 8); SEQ_PUT_DEC(" kB\nVmSwap:\t", swap); seq_puts(m, " kB\n"); +#ifdef CONFIG_KMMSCAND + seq_put_decimal_ull_width(m, "PTEAScanScale:\t", mm->pte_scan_scale, 8); + seq_puts(m, "\n"); +#endif hugetlb_report_usage(m, mm); } #undef SEQ_PUT_DEC diff --git a/include/linux/kmmscand.h b/include/linux/kmmscand.h index b120c65ee8c6..7021f7d979a6 100644 --- a/include/linux/kmmscand.h +++ b/include/linux/kmmscand.h @@ -13,6 +13,7 @@ static inline void kmmscand_execve(struct mm_struct *mm) =20 static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *o= ldmm) { + mm->pte_scan_scale =3D oldmm->pte_scan_scale; __kmmscand_enter(mm); } =20 diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index eeaedc7473b1..12184e8ebc58 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1018,6 +1018,9 @@ struct mm_struct { #ifdef CONFIG_KMMSCAND /* Tracks promotion node. XXX: use nodemask */ int target_node; + + /* Integer representing PTE A bit scan aggression (0-10) */ + unsigned int pte_scan_scale; #endif /* * An operation with batched TLB flushing is going on. Anything diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 5c6080680cb2..18face11440a 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -353,4 +353,11 @@ struct prctl_mm_map { */ #define PR_LOCK_SHADOW_STACK_STATUS 76 =20 +/* Set/get PTE A bit scan scale */ +#define PR_SET_PTE_A_SCAN_SCALE 77 +#define PR_GET_PTE_A_SCAN_SCALE 78 +# define PR_PTE_A_SCAN_SCALE_MIN 0 +# define PR_PTE_A_SCAN_SCALE_MAX 10 +# define PR_PTE_A_SCAN_SCALE_DEFAULT 1 + #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/fork.c b/kernel/fork.c index f61c55cf33c2..bfbbacb8ec36 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -106,6 +106,7 @@ #include #include #include +#include =20 #include #include @@ -1292,6 +1293,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, init_tlb_flush_pending(mm); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !defined(CONFIG_SPLIT_PMD_PTLO= CKS) mm->pmd_huge_pte =3D NULL; +#endif +#ifdef CONFIG_KMMSCAND + mm->pte_scan_scale =3D PR_PTE_A_SCAN_SCALE_DEFAULT; #endif mm_init_uprobes_state(mm); hugetlb_count_init(mm); diff --git a/kernel/sys.c b/kernel/sys.c index cb366ff8703a..0518480d8f78 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2142,6 +2142,19 @@ static int prctl_set_auxv(struct mm_struct *mm, unsi= gned long addr, =20 return 0; } +#ifdef CONFIG_KMMSCAND +static int prctl_pte_scan_scale_write(unsigned int scale) +{ + scale =3D clamp(scale, PR_PTE_A_SCAN_SCALE_MIN, PR_PTE_A_SCAN_SCALE_MAX); + current->mm->pte_scan_scale =3D scale; + return 0; +} + +static unsigned int prctl_pte_scan_scale_read(void) +{ + return current->mm->pte_scan_scale; +} +#endif =20 static int prctl_set_mm(int opt, unsigned long addr, unsigned long arg4, unsigned long arg5) @@ -2811,6 +2824,18 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, a= rg2, unsigned long, arg3, return -EINVAL; error =3D arch_lock_shadow_stack_status(me, arg2); break; +#ifdef CONFIG_KMMSCAND + case PR_SET_PTE_A_SCAN_SCALE: + if (arg3 || arg4 || arg5) + return -EINVAL; + error =3D prctl_pte_scan_scale_write((unsigned int) arg2); + break; + case PR_GET_PTE_A_SCAN_SCALE: + if (arg2 || arg3 || arg4 || arg5) + return -EINVAL; + error =3D prctl_pte_scan_scale_read(); + break; +#endif default: trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5); error =3D -EINVAL; diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 38d7825c0d62..68ef2141c349 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -1113,6 +1113,11 @@ static unsigned long kmmscand_scan_mm_slot(void) goto outerloop; } =20 + if (!mm->pte_scan_scale) { + next_mm =3D true; + goto outerloop; + } + mm_target_node =3D READ_ONCE(mm->target_node); if (mm_target_node !=3D mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node); --=20 2.34.1