From nobody Sat Oct 4 17:30:22 2025 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2058.outbound.protection.outlook.com [40.107.100.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9B482264B8 for ; Thu, 14 Aug 2025 15:34:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185651; cv=fail; b=Sla0HdpuT1QHsfB7GiAEdcT5/SPaY4dJJ1Taoa+m3FbDJRYSLDREa/RnUJ0HWEdoOFCUQjFe0A5tXj5iDtYdHlszttmilJvICX2yA6AoZ1IJ/9/L4R4Z7zPVzlH32s6VCMUGBGB0H9H0bvsWPYCWtSV92xAVq0E3BWfYYdmnvTw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755185651; c=relaxed/simple; bh=fpIn7DYWxD6/5L88coubosr+TA00MTCPGUP29NwtBwU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=c4qSo2FFhphagSIc0EFHwR7WdMAtE8mdJKEG1qHIPnzf2dE/lR5ME7gLT4fWiPqs4w6+4FZM1tXhAkkfm9nhCcAWZUKqehLIQZxkR5K8g59M1M9tkCsZ0UOveKypxregPY7+SXwlIPwziFJ78y+YRgLOlmoiClqgOKOrgkQpmhI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=IpTGdT3y; arc=fail smtp.client-ip=40.107.100.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="IpTGdT3y" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kjODYYIxwM7NftDrilUpDZSkm0VIv3/FDGJHL1P6Saw8NbisV1Tuop/APPjMRR1hLd/Mtw9cOSfpzlS/cunayn80VCjuuIilc/prrRL6sr3pUHVeizQx384dXmsio0ZIOIeGFWpqj90f7Z/sufGlAc0mwnjqDv1kUDal+wZ2N3g5nIBQ26M1t5uTyAnFjOg2XoR5kLJRUIfdwnHt7k/G4Nw5hWfXs13UlM+V6uKv/KZDXflH0J/i9Osg6lIaBTsow+ZseWlTk1TOB9q59yHQdtPG/gqcDo8KWXayOpTZZwdraw9Bj4uw9o7U+2RiYN2T6gEuXQrUpWNIvHQReOMRTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=D7cB5zU1ByDq5a++Mtl9WuTdBPv0fgnoAzM9mfcpqIg=; b=MJhZsDmnVDLBjavhBrYXTGFPXP+7hvjHgkgv13RG6VfORnjQ6iuSWdXVUeIN71EO2CfIFLMfyVfcl6t0OIuFiHe1TS9nqYB+55NsQBxfDfz12FirMfrjhDPpxG0UtIXbVgezvOrhVd/d5gHbI7wBeRiUlKrPSFrNKHg9OH9YyS++4T3qh7LuP/11doNeLaeYkvE/Z5naCpSIJF0Nrv4PvE+OUBMfHw2vS4cDwAjYIzQrZezSbEsVq+cP9NpqvKkafCzIE1ACVcuBVsvQOWpAOKmQJSkAwII0RqS7+MWArq4k/7UmcNljBGjpfaEaXr0agaFQ+OcNzc6dJr/3blAtOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D7cB5zU1ByDq5a++Mtl9WuTdBPv0fgnoAzM9mfcpqIg=; b=IpTGdT3y99vV05TTKMHVHwIsYrhV8wbBRAo9orT938jJzauSy9M3YQcX7FI2+mL+UwRyt129wJyDymWcsEjaLjOV+Es5Ji1wieif3M8ZXRCZq8YsEiKVaLTSrJJdOhkQNQBS1t/QUsNqvyAVZYwHdkFDdbkRPyMZfi/GyaNIr90= Received: from MN0PR04CA0018.namprd04.prod.outlook.com (2603:10b6:208:52d::11) by PH0PR12MB7816.namprd12.prod.outlook.com (2603:10b6:510:28c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9031.13; Thu, 14 Aug 2025 15:34:06 +0000 Received: from BN3PEPF0000B077.namprd04.prod.outlook.com (2603:10b6:208:52d:cafe::d0) by MN0PR04CA0018.outlook.office365.com (2603:10b6:208:52d::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9031.17 via Frontend Transport; Thu, 14 Aug 2025 15:34:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B077.mail.protection.outlook.com (10.167.243.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9031.11 via Frontend Transport; Thu, 14 Aug 2025 15:34:05 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 14 Aug 2025 10:33:52 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V3 03/17] mm: Scan the mm and create a migration list Date: Thu, 14 Aug 2025 15:32:53 +0000 Message-ID: <20250814153307.1553061-4-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250814153307.1553061-1-raghavendra.kt@amd.com> References: <20250814153307.1553061-1-raghavendra.kt@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B077:EE_|PH0PR12MB7816:EE_ X-MS-Office365-Filtering-Correlation-Id: 20d75f28-a20c-45c5-ca67-08dddb480121 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Ou/yncPx/fCmioDGh51mshp6fmT96HwmtNojBMrjL/3Fg9lht6cVfNvAvw9a?= =?us-ascii?Q?V7Us184xzfKEdSQr8HcPPkAGrw2XWSyoGgwvL0hqsNo7YSJbF+73/mXZT9VC?= =?us-ascii?Q?7vaT4tNIWGOQ9PRo83J9huOcwF/b8SwruAdfrqFL55lyZSz5R2oQ84GmprJq?= =?us-ascii?Q?sQ+rap0l/LI/FUdnKZnoVJunFDHN0tAFwJlzDd1kDERvgJBIDq6UdtrKl2YH?= =?us-ascii?Q?dUCfnoCI40BSnYwqmOM8bS2aF1xLcGMMOiYT+zFolMsNDAujhynrm/nWbc6i?= =?us-ascii?Q?aOclMmBCTf2nOn+RJk6xenczStnOx8363MGctI1ANBD2RTPNTrsio3arbTiH?= =?us-ascii?Q?KwRLWACrad7rSFxDGNURVhyEduLYLXG1vBp6/4C3sT3sGLWkeM8QgWCHJZBQ?= =?us-ascii?Q?vmW6PvYzr2Z5pJlvCYRSqYnE2vfFmZbQhAJhy2pW5FPPE7jP40F5Ob4WyNqN?= =?us-ascii?Q?YEjSh6rvmuzERU0cU2CkWN7vrUQNgcJriyoRjRS2zN07Mziu8GOm3naLMtXD?= =?us-ascii?Q?66R7S7WWpBIaYQs2StifPbsJWsxnaBEO1aTaoUQwDQaKP/kNOvjZSfNKUHZW?= =?us-ascii?Q?WURuuoMAuXTJDP3OOQYer3wRRhJca3vgcjH0q/ZSP92PKf5Mh5qCLdDjvmFz?= =?us-ascii?Q?SZtKe5viVMyDfy4rbqk253MUfDasEzLZepoV/WgArMkZRQwBxgdDNp6i8znA?= =?us-ascii?Q?pNj1iFb3LG1dyMpPrQ5CgQQ9cKZ6xSnzENqF2WkvYLgQd1QivF5+g9k9LG+i?= =?us-ascii?Q?/eAFTHKZOBrDZ1LnHYRkYcSfcfNxbUJl1FdMNy04vlAttzvwYVxFBYCKP3+8?= =?us-ascii?Q?HfxoKK5rmFARxSy0kwRFdNLvp3bOCVgQsS7Ydg8JFukKNCPlIfBkIRNkub1J?= =?us-ascii?Q?KZssPzXyGyit5gcLBzXYMZ+952t/a63SfHm+EIoGtQyBhETnpdpBJqS0J2Eh?= =?us-ascii?Q?wnkUjd1DCPlTgnHg3jt5HxindbxLZva2bhEP48JhiBzZYnb3LsCzBPN3dqsx?= =?us-ascii?Q?5+L8YeGQnQmM4I6N3qwplfhU0Ehg7lA15fw+zM+grKgzWF9tKjM/6tkj0q+Y?= =?us-ascii?Q?A1hmU6lEkawaB7tIS7yjOVk++KaMhDcjSJs4swH/FI9NAiLOt3Z5lHkBTncq?= =?us-ascii?Q?fh6yN6mZ7AZdXEmnn00aSXvkLH2Fi/wHjKYuR17FXdUEpFNg6U0S8wEeumtq?= =?us-ascii?Q?cJqYGGhD1jsZtPHh6UnIQGpvuszhGBUacGAfWdd5PEXgSItTIJ9XCL3cX9N8?= =?us-ascii?Q?5OO4W1a1nQ+oOLBxHtGND4DNGzjn2jhVlUTXNH4XAZFnoJNjUwglnq6YYW+S?= =?us-ascii?Q?zJfku1cwY++dWzS0x4aS/Ix1KKQdEO8oIhfhFxKG5Sgxay/I00NS+fZlCuD2?= =?us-ascii?Q?PPvHCJ5OVor708ak85IjH8b6XPVB55+95OW2fumPUGo5C5WV1sGzn5un82Zf?= =?us-ascii?Q?QK1bjmF63KgctVwpJ4h/FEKINgKUbzmAp2fCvLIpGd4z+wFwlhzetWa1memc?= =?us-ascii?Q?PaSObciRvly/5mDdOrPRWMDjFhNq3mrHwSk3?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2025 15:34:05.9194 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 20d75f28-a20c-45c5-ca67-08dddb480121 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B077.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7816 Content-Type: text/plain; charset="utf-8" Since we already have the list of mm_struct in the system, add a module to scan each mm that walks VMAs of each mm_struct and scan all the pages associated with that. In the scan path: Check for the recently acccessed pages (PFNs) belonging to slowtier nodes. Add all those to a list. Signed-off-by: Raghavendra K T --- mm/kscand.c | 321 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 320 insertions(+), 1 deletion(-) diff --git a/mm/kscand.c b/mm/kscand.c index d5b0d3041b0f..1d883d411664 100644 --- a/mm/kscand.c +++ b/mm/kscand.c @@ -4,10 +4,18 @@ #include #include #include +#include +#include +#include +#include +#include +#include #include #include #include #include +#include +#include #include #include #include @@ -18,6 +26,11 @@ =20 static struct task_struct *kscand_thread __read_mostly; static DEFINE_MUTEX(kscand_mutex); +/* + * Total VMA size to cover during scan. + */ +#define KSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) +static unsigned long kscand_scan_size __read_mostly =3D KSCAND_SCAN_SIZE; =20 /* How long to pause between two scan cycles */ static unsigned int kscand_scan_sleep_ms __read_mostly =3D 20; @@ -42,6 +55,8 @@ static struct kmem_cache *kscand_slot_cache __read_mostly; /* Per mm information collected to control VMA scanning */ struct kscand_mm_slot { struct mm_slot slot; + long address; + bool is_scanned; }; =20 /* Data structure to keep track of current mm under scan */ @@ -54,6 +69,29 @@ struct kscand_scan kscand_scan =3D { .mm_head =3D LIST_HEAD_INIT(kscand_scan.mm_head), }; =20 +/* + * Data structure passed to control scanning and also collect + * per memory node information + */ +struct kscand_scanctrl { + struct list_head scan_list; + unsigned long address; +}; + +struct kscand_scanctrl kscand_scanctrl; +/* Per folio information used for migration */ +struct kscand_migrate_info { + struct list_head migrate_node; + unsigned long pfn; + unsigned long address; +}; + +static bool kscand_eligible_srcnid(int nid) +{ + /* Only promotion case is considered */ + return !node_is_toptier(nid); +} + static inline int kscand_has_work(void) { return !list_empty(&kscand_scan.mm_head); @@ -84,11 +122,277 @@ static void kscand_wait_work(void) scan_sleep_jiffies); } =20 +static inline bool is_valid_folio(struct folio *folio) +{ + if (!folio || !folio_mapped(folio) || !folio_raw_mapping(folio)) + return false; + + if (folio_test_unevictable(folio) || folio_is_zone_device(folio) || + folio_maybe_mapped_shared(folio)) + return false; + + return true; +} + + +static bool folio_idle_clear_pte_refs_one(struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr, + pte_t *ptep) +{ + bool referenced =3D false; + struct mm_struct *mm =3D vma->vm_mm; + pmd_t *pmd =3D pmd_off(mm, addr); + + if (ptep) { + if (ptep_clear_young_notify(vma, addr, ptep)) + referenced =3D true; + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + if (!pmd_present(*pmd)) + WARN_ON_ONCE(1); + if (pmdp_clear_young_notify(vma, addr, pmd)) + referenced =3D true; + } else { + WARN_ON_ONCE(1); + } + + if (referenced) { + folio_clear_idle(folio); + folio_set_young(folio); + } + + return true; +} + +static void page_idle_clear_pte_refs(struct page *page, pte_t *pte, struct= mm_walk *walk) +{ + bool need_lock; + struct folio *folio =3D page_folio(page); + unsigned long address; + + if (!folio_mapped(folio) || !folio_raw_mapping(folio)) + return; + + need_lock =3D !folio_test_anon(folio) || folio_test_ksm(folio); + if (need_lock && !folio_trylock(folio)) + return; + address =3D vma_address(walk->vma, page_pgoff(folio, page), compound_nr(p= age)); + VM_BUG_ON_VMA(address =3D=3D -EFAULT, walk->vma); + folio_idle_clear_pte_refs_one(folio, walk->vma, address, pte); + + if (need_lock) + folio_unlock(folio); +} + +static int hot_vma_idle_pte_entry(pte_t *pte, + unsigned long addr, + unsigned long next, + struct mm_walk *walk) +{ + struct page *page; + struct folio *folio; + struct mm_struct *mm; + struct vm_area_struct *vma; + struct kscand_migrate_info *info; + struct kscand_scanctrl *scanctrl =3D walk->private; + int srcnid; + + scanctrl->address =3D addr; + pte_t pteval =3D ptep_get(pte); + + if (!pte_present(pteval)) + return 0; + + if (pte_none(pteval)) + return 0; + + vma =3D walk->vma; + mm =3D vma->vm_mm; + + page =3D pte_page(*pte); + + + folio =3D page_folio(page); + folio_get(folio); + + if (!is_valid_folio(folio)) { + folio_put(folio); + return 0; + } + folio_set_idle(folio); + page_idle_clear_pte_refs(page, pte, walk); + srcnid =3D folio_nid(folio); + + + if (!folio_test_lru(folio)) { + folio_put(folio); + return 0; + } + + if (!kscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return 0; + } + if (!folio_test_idle(folio) && + (folio_test_young(folio) || folio_test_referenced(folio))) { + + /* XXX: Leaking memory. TBD: consume info */ + + info =3D kzalloc(sizeof(struct kscand_migrate_info), GFP_NOWAIT); + if (info && scanctrl) { + info->pfn =3D folio_pfn(folio); + info->address =3D addr; + list_add_tail(&info->migrate_node, &scanctrl->scan_list); + } + } + + folio_put(folio); + return 0; +} + +static const struct mm_walk_ops hot_vma_set_idle_ops =3D { + .pte_entry =3D hot_vma_idle_pte_entry, + .walk_lock =3D PGWALK_RDLOCK, +}; + +static void kscand_walk_page_vma(struct vm_area_struct *vma, struct kscand= _scanctrl *scanctrl) +{ + if (!vma_migratable(vma) || !vma_policy_mof(vma) || + is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + return; + } + if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) =3D=3D (VM_READ= ))) + return; + + if (!vma_is_accessible(vma)) + return; + + walk_page_vma(vma, &hot_vma_set_idle_ops, scanctrl); +} + static inline int kscand_test_exit(struct mm_struct *mm) { return atomic_read(&mm->mm_users) =3D=3D 0; } =20 +static void kscand_collect_mm_slot(struct kscand_mm_slot *mm_slot) +{ + struct mm_slot *slot =3D &mm_slot->slot; + struct mm_struct *mm =3D slot->mm; + + lockdep_assert_held(&kscand_mm_lock); + + if (kscand_test_exit(mm)) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + + mm_slot_free(kscand_slot_cache, mm_slot); + mmdrop(mm); + } +} + +static unsigned long kscand_scan_mm_slot(void) +{ + bool next_mm =3D false; + bool update_mmslot_info =3D false; + + unsigned long vma_scanned_size =3D 0; + unsigned long address; + + struct mm_slot *slot; + struct mm_struct *mm; + struct vm_area_struct *vma =3D NULL; + struct kscand_mm_slot *mm_slot; + + + spin_lock(&kscand_mm_lock); + + if (kscand_scan.mm_slot) { + mm_slot =3D kscand_scan.mm_slot; + slot =3D &mm_slot->slot; + address =3D mm_slot->address; + } else { + slot =3D list_entry(kscand_scan.mm_head.next, + struct mm_slot, mm_node); + mm_slot =3D mm_slot_entry(slot, struct kscand_mm_slot, slot); + address =3D mm_slot->address; + kscand_scan.mm_slot =3D mm_slot; + } + + mm =3D slot->mm; + mm_slot->is_scanned =3D true; + spin_unlock(&kscand_mm_lock); + + if (unlikely(!mmap_read_trylock(mm))) + goto outerloop_mmap_lock; + + if (unlikely(kscand_test_exit(mm))) { + next_mm =3D true; + goto outerloop; + } + + VMA_ITERATOR(vmi, mm, address); + + for_each_vma(vmi, vma) { + kscand_walk_page_vma(vma, &kscand_scanctrl); + vma_scanned_size +=3D vma->vm_end - vma->vm_start; + + if (vma_scanned_size >=3D kscand_scan_size) { + next_mm =3D true; + /* TBD: Add scanned folios to migration list */ + break; + } + } + + if (!vma) + address =3D 0; + else + address =3D kscand_scanctrl.address + PAGE_SIZE; + + update_mmslot_info =3D true; + + if (update_mmslot_info) + mm_slot->address =3D address; + +outerloop: + /* exit_mmap will destroy ptes after this */ + mmap_read_unlock(mm); + +outerloop_mmap_lock: + spin_lock(&kscand_mm_lock); + WARN_ON(kscand_scan.mm_slot !=3D mm_slot); + + /* + * Release the current mm_slot if this mm is about to die, or + * if we scanned all vmas of this mm. + */ + if (unlikely(kscand_test_exit(mm)) || !vma || next_mm) { + /* + * Make sure that if mm_users is reaching zero while + * kscand runs here, kscand_exit will find + * mm_slot not pointing to the exiting mm. + */ + if (slot->mm_node.next !=3D &kscand_scan.mm_head) { + slot =3D list_entry(slot->mm_node.next, + struct mm_slot, mm_node); + kscand_scan.mm_slot =3D + mm_slot_entry(slot, struct kscand_mm_slot, slot); + + } else + kscand_scan.mm_slot =3D NULL; + + if (kscand_test_exit(mm)) { + kscand_collect_mm_slot(mm_slot); + goto end; + } + } + mm_slot->is_scanned =3D false; +end: + spin_unlock(&kscand_mm_lock); + return 0; +} + static void kscand_do_scan(void) { unsigned long iter =3D 0, mms_to_scan; @@ -101,7 +405,7 @@ static void kscand_do_scan(void) break; =20 if (kscand_has_work()) - msleep(100); + kscand_scan_mm_slot(); =20 iter++; =20 @@ -148,6 +452,7 @@ void __kscand_enter(struct mm_struct *mm) if (!kscand_slot) return; =20 + kscand_slot->address =3D 0; slot =3D &kscand_slot->slot; =20 spin_lock(&kscand_mm_lock); @@ -175,6 +480,12 @@ void __kscand_exit(struct mm_struct *mm) hash_del(&slot->hash); list_del(&slot->mm_node); free =3D 1; + } else if (mm_slot && kscand_scan.mm_slot =3D=3D mm_slot && !mm_slot->is_= scanned) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free =3D 1; + /* TBD: Set the actual next slot */ + kscand_scan.mm_slot =3D NULL; } =20 spin_unlock(&kscand_mm_lock); @@ -224,6 +535,12 @@ static int stop_kscand(void) return 0; } =20 +static inline void init_list(void) +{ + INIT_LIST_HEAD(&kscand_scanctrl.scan_list); + init_waitqueue_head(&kscand_wait); +} + static int __init kscand_init(void) { int err; @@ -234,6 +551,8 @@ static int __init kscand_init(void) pr_err("kscand: kmem_cache error"); return -ENOMEM; } + + init_list(); err =3D start_kscand(); if (err) goto err_kscand; --=20 2.34.1