From nobody Sat Sep 26 21:14:08 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 302802DECC2 for ; Sun, 30 Aug 2026 05:14:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066852; cv=none; b=qWu8hUBT544EeME1McQ+unYbpoDiB645wacRIwjALqmGFAa1Zua/QR54QSEH/9X92qon4ZcGWRmaph3UPiEXfFDN7R3j8y+zoYDkJdDa1CouEAquIRfnLWgA/ZfBHkrs3I6wA3EpdEKl3NPsl1AUxRp5thmPYfs+WXjzi8DOjB0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066852; c=relaxed/simple; bh=Q33HR5Se8BzslnbssUes1Zl68yaouorY+HmA1jQ/xyY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o7EgfdxFK8r5JzJzqnrEb/R6JwHltduMCaKSQFbCD1wS8GBiUa6wVk+1UlMX+sWyVUP4fb0UQsx3ki/LCpMqV7ztvhn3iNxHAuEKmuhWA+D9Vy0nXRNEATdw/TXnbiKPcP9IzNjfc6/r4cksQUxYVN9g5ZsRt67S1iTzZghGYqQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai; spf=pass smtp.mailfrom=crusoe.ai; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b=hFpXAfMQ; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b="hFpXAfMQ" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-ca766c1c9ccso1571742a12.0 for ; Sat, 29 Aug 2026 22:14:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crusoe.ai; s=google; t=1788066850; x=1788671650; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lJRx0Jd1M2+awD7NVhe4CKIIkdBtWXqqpTIhZLTtZn4=; b=hFpXAfMQWDZA3upTLdrIx6yI3ZqL/SG8I4Wvk+26v/EPlHgFxCI1G2Fcn6R/WhF+SL mkzqjWjLuO0NrGcNouHdwJxt2ZfaKHoxXG6W9RKUjmXHF3YOHpazfG5jBdIwUvwMjHa2 DwpZR4o+48hIsGRrrcIKcvyKr354JEknO/u+I2olm9rFUmvJCZl0bOCMBh2adQtZsMBY eJKgiIpRU+47o3hf6xNqeTLJplO6CYAlAJEd8gMiVdwr67Mu3x4CYRwZNEBpKVPqYfTh 3+Vu6c02zzi2vqdC5BgP6V8MVd5hKbUXxRzV5lKcgikrdVudNjD0ER7DGaqCYVeV9qO+ iReg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788066850; x=1788671650; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=lJRx0Jd1M2+awD7NVhe4CKIIkdBtWXqqpTIhZLTtZn4=; b=V1jVjsD2ukNAWBsE0x9QLbzr9ZeSRpbMJbzP2sA/D4Ctx5yLaamCflyV1PZ4NRt3mZ QP1hJiBXuiv/QgeoBuqEjD36SkEDlHEqn+zHsX2zKxjyOoGNMljN1RfP8R6NW8kR4b2+ mAqiP4zqrXTUAJR/5vNze4911XSuv871DFl3Ez12uFyp7Q6NcJ7n3fdwdZE/tLgnb6qD jvH+jwu/2TBNwCjV5Rvg81uhrl73OZZHI5Rex2Cu/uOV+8a8yVSmQXDFmjnDxvbyJ7nv 78wOsFyf+M8Sz5KkOeFqXxr/pXSOeFOXg1Z/XPxBDfcw0AHDuLxkQO64I9vks5QkccFI yIow== X-Forwarded-Encrypted: i=1; AHgh+RrZUsR88X0ZoRLL0X3mIz/GeAEg9cUfYhSvBh7iei+9ttLKnLD1XwXBFOjk2LIWl9DxlF6t2VpvhPLvxK8=@vger.kernel.org X-Gm-Message-State: AFuF++lHhwV+q1LhOB+ANF36v4cclSax2djitZuMKQ74R4V3ZClBrD2k 4NC+QM4sJGhA9A26cwOAfE+VNPzVPiUtf1qzE517h3uKfdEm2JwKwPOxuLnbTlze1zw8Lqtgg+o V0s9IBajLdA== X-Gm-Gg: AR+sD13sTvqKxwv3p7zXaGjhzhp5ckBzfVTUJbHrZcOPaLVnhnn5FYRRXCW7CQYsNJ7 Sj3aSkkSLWOHTqPQmybVVK3ilb71jEb7IHip7y4NtbzyZKJKDSt596lhBZikXIFTvDa6PTbjnLd CRTCPbiDKTSvRsev2qiDiKQvabozX2xQLSjr/5EwYpjGLoUlzWRRKtlSvnOjRM8jmgJ9/TnxquB 8v9aDksNAR38CE9l84eAGHWxOKtgmWf94MDaKVlhpq9sTti9sceqTIuaYcMBWl0JBCZ0SI3l/8c ScFJYjrvJsXg9H3jZ/aU/ltiVFnmuvIuR9BEDDJvdZQrMora8T0GYZRO9EG4+MBcgmiL8VXFi2V ETcppK17LVPNZEyf+M3yMJ/CK0N21IlJQ/Y4/RIZXRUhafv9MWMOCNReNRhkMIgc18h8m2c0ISx JZqlPb7xqtdGsUE8z2Hg+jlmTj0Dxeh8AbgcTRgvsmFLFQHgH0RowUSrXOe/TXdu3ntSplJxrWo SY3vMK0aFw37TsHC9IfDfLVtHpt/etg88o= X-Received: by 2002:a05:6a21:328b:b0:3c3:7a0a:18a6 with SMTP id adf61e73a8af0-3d2688b71f4mr29680395637.18.1788066850343; Sat, 29 Aug 2026 22:14:10 -0700 (PDT) Received: from MBP-Krishna-Iyer.civet-hops.ts.net ([2601:645:c68a:b830:d101:1d:112c:715e]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286fa37fa5sm22595810eec.29.2026.08.29.22.14.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 29 Aug 2026 22:14:10 -0700 (PDT) From: Krishna Iyer To: SeongJae Park Cc: Andrew Morton , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Krishna Iyer Subject: [PATCH 1/6] mm/damon: move damon_hugetlb_mkold() from vaddr to ops-common Date: Sat, 29 Aug 2026 22:14:02 -0700 Message-ID: <20260830051407.50008-2-kiyer@crusoe.ai> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260830051407.50008-1-kiyer@crusoe.ai> References: <20260830051407.50008-1-kiyer@crusoe.ai> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" damon_hugetlb_mkold() clears the accessed bit of a hugetlb-mapping huge PTE and propagates the aging to secondary MMUs via mmu_notifier_clear_young(), spanning the whole huge page size. It currently lives in vaddr.c, and is thus usable only by the virtual address space monitoring operations set. The physical address space monitoring operations set will need the same logic, to support access monitoring of hugetlb-backed memory. Move the function to ops-common as-is, with no behavioral change. A follow-up change will use it from the folio-granular rmap walkers. Assisted-by: Claude:claude-fable-5 Signed-off-by: Krishna Iyer Reviewed-by: SJ Park --- mm/damon/ops-common.c | 30 ++++++++++++++++++++++++++++++ mm/damon/ops-common.h | 9 +++++++++ mm/damon/vaddr.c | 27 --------------------------- 3 files changed, 39 insertions(+), 27 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index fbda70d8ea4d..f5fe92b825bb 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -3,6 +3,7 @@ * Common Code for Data Access Monitoring */ =20 +#include #include #include #include @@ -98,6 +99,35 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct = *vma, unsigned long addr #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ } =20 +#ifdef CONFIG_HUGETLB_PAGE +void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long addr) +{ + bool referenced =3D false; + pte_t entry =3D huge_ptep_get(mm, addr, pte); + struct folio *folio =3D pfn_folio(pte_pfn(entry)); + unsigned long psize =3D huge_page_size(hstate_vma(vma)); + + folio_get(folio); + + if (pte_young(entry)) { + referenced =3D true; + entry =3D pte_mkold(entry); + set_huge_pte_at(mm, addr, pte, entry, psize); + } + + if (mmu_notifier_clear_young(mm, addr, + addr + huge_page_size(hstate_vma(vma)))) + referenced =3D true; + + if (referenced) + folio_set_young(folio); + + folio_set_idle(folio); + folio_put(folio); +} +#endif /* CONFIG_HUGETLB_PAGE */ + #define DAMON_MAX_SUBSCORE (100) #define DAMON_MAX_AGE_IN_LOG (32) =20 diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index 38d295488fa1..f7811c9c7a02 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -9,6 +9,15 @@ struct folio *damon_get_folio(unsigned long pfn); =20 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr); void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr); +#ifdef CONFIG_HUGETLB_PAGE +void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long addr); +#else +static inline void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long addr) +{ +} +#endif /* CONFIG_HUGETLB_PAGE */ void damon_folio_mkold(struct folio *folio); bool damon_folio_young(struct folio *folio); =20 diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 0648400b2d65..15379d984689 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -293,33 +293,6 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned = long addr, } =20 #ifdef CONFIG_HUGETLB_PAGE -static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long addr) -{ - bool referenced =3D false; - pte_t entry =3D huge_ptep_get(mm, addr, pte); - struct folio *folio =3D pfn_folio(pte_pfn(entry)); - unsigned long psize =3D huge_page_size(hstate_vma(vma)); - - folio_get(folio); - - if (pte_young(entry)) { - referenced =3D true; - entry =3D pte_mkold(entry); - set_huge_pte_at(mm, addr, pte, entry, psize); - } - - if (mmu_notifier_clear_young(mm, addr, - addr + huge_page_size(hstate_vma(vma)))) - referenced =3D true; - - if (referenced) - folio_set_young(folio); - - folio_set_idle(folio); - folio_put(folio); -} - static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long end, struct mm_walk *walk) --=20 2.54.0 From nobody Sat Sep 26 21:14:08 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D0882DFA3A for ; Sun, 30 Aug 2026 05:14:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066853; cv=none; b=kYc1OpEs/Wo4tmOI4eST3+NVV4gVmYTltMoHWn6XiQShpiI3DlIovcA0Zw4zrvX2Eaj6qjbraXh7Ep4MrCMN+1DXhQGBc+HNv98VTS184f+8LsFJmUDG07u9QuRJPYpTmpOaeFoC61uz3enrS8wN0xY8taERTUvMqvf1OsbU3R8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066853; c=relaxed/simple; bh=HkCP7VGJZpo8tqbPFVRl/GGgUC8qiBQ05t6gcGsjZ+Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UvcAlzCt8fx4yH0Gcn7XldpQYUe52Yn/gllwWS2Z/jcheDSxq0CGoiGPdrP3vscyf6rNCafNr1O8JzAZsOThIev4LObOyQBF2/8pIntyt/SPb7y0NHsb7u3c0P8bDVQshlnEYO0JKjGggkto/6P4WIGW21dyPi6N0GFDR0H/Vfo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai; spf=pass smtp.mailfrom=crusoe.ai; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b=iScyj66Q; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b="iScyj66Q" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-cc1c3d79b9bso2055307a12.3 for ; Sat, 29 Aug 2026 22:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crusoe.ai; s=google; t=1788066851; x=1788671651; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+Qq90Jf7KadtS5HpHzOCYry3UrGmes12BSmNzbFVazM=; b=iScyj66QET0xUCMvPK83i8rJ3H4sZEfq7RHfA7W1Mdi6nzXjChdeeOE/xUeipZhsVD Qk0kWqII5Tozx2TXVyJmb48mXFd1p6elHe84WdZDRmXxpcxYJAdhL6fhXuvX8r7pUQY8 /uH/S4kD1cATvKAo3O5P+NSaVnAG9UZWwDcCt4idt43dd9I6ZBVHXcyu3DHtasnUbL/Q UGp7hXH+cpl/nnu5AF0c+IlMj+UzypwUZx4LaOr5Omo6GFXNfn4Hxb3OFw2K7j4tY5fl YLeBgtnqC38CpCXrpGR0UlAOOcenWQlR6lOyvSPYqPdpHVn1SzDL5HnhWmdsXMTB4Res T2ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788066851; x=1788671651; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=+Qq90Jf7KadtS5HpHzOCYry3UrGmes12BSmNzbFVazM=; b=OlVGNfrAzgWdCuHhfbszcJMDUU5VMmZzt+XenOU39Eky+8jlhUHjwFNzePwmz69IZa eEECtin5sqy8HWsK3X42vTzCdut0/oSFPk3QtpTrfOWcXtkDAz0dV73FK+RaYVzWTRKk gWyeg/WMexGqsUVEjToWYSKzgKU+tIOxdaySh+fdI1OvwdjbrWQugg3dGNeQ/hFnQ9OI X34cXkqyrtC6a4EpGyd/zPS6S4DSOljMl4MUlOytCpLnYnDp0Qpu6wj5DxtCU2c+eBMt H3e6JcF39qQT6046eqHABEg9X8BaXGfsuw3o7wInX62vPSTaZLP1AMvPPjykHbERQSwB eCfw== X-Forwarded-Encrypted: i=1; AHgh+Rpg8QZDTmDzOuEXL/+XLCwGf70KNOLzrEI5tebO3m5sUMQKzuTC76f4VluzdBhFbuorlgRMIAs37vfN2wM=@vger.kernel.org X-Gm-Message-State: AFuF++kxiHN5ShNLiaEY1Ve0lp69J7BfR0HmrQR8Uvy5xRFZ3FBRp/+j TIA0TmgM1p1OJaGpfOB62EAbZWwBqNIm8CawdMrjXMqovvEMf06O7rjqrH3wV2HBC+g= X-Gm-Gg: AR+sD13P2ILz8Tu9GqW4QG6fkuqXqItA4Saro2SlDLMuvLtOHXi3GyfvE1j9mod+IXt dTOI8OMkMO34VsLPhwviVqN5llm7ajJpB/o2d3mAMPl5swES1C1/ejWOdyVqEQTnBYvAXHWxRJE NPAHrRnQR6t5rHXE5l5RaLnIPXqF5hgjNeTvOq1TMhtIxJKt0FxHmUWmRY0dZaeaf98PWhJoSr+ ih8oT6hK3HvXNPqjUo+puwwWy2kAHzfM09fQxSPyKyK1FJk00q5pq/jSFi4hbYuNB2OlTmOgRSo R06lFbyo+L5IVaS1ZPkXEQBieESCqHS4s5HzJObO0H/WJR+I5BIAVDrV6fGNHb2s5yP08MtzIK6 JUPdsw50N1UESRCp3mV+ej8XdbKpf8oyIvT8Z9gedmMNByMo70IMFFTeVQGvAy1KLdHLjJ8cytO hjfRJ4U4pIfsYMZ9GDiT9v4cyT8dorlPQaoeYtTt3Bt4x4H/3ykEcpjpT8x1Ep2L+TAek1mULqV M2rx1VZPdKmHhW3bN0vUZeKk5xS7elS3IKl X-Received: by 2002:a05:6a21:998a:b0:3c4:2b8b:e88f with SMTP id adf61e73a8af0-3d26646a960mr24213242637.5.1788066851383; Sat, 29 Aug 2026 22:14:11 -0700 (PDT) Received: from MBP-Krishna-Iyer.civet-hops.ts.net ([2601:645:c68a:b830:d101:1d:112c:715e]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286fa37fa5sm22595810eec.29.2026.08.29.22.14.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 29 Aug 2026 22:14:11 -0700 (PDT) From: Krishna Iyer To: SeongJae Park Cc: Andrew Morton , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Krishna Iyer Subject: [PATCH 2/6] mm/damon/ops-common: handle hugetlb folios in folio mkold/young rmap walkers Date: Sat, 29 Aug 2026 22:14:03 -0700 Message-ID: <20260830051407.50008-3-kiyer@crusoe.ai> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260830051407.50008-1-kiyer@crusoe.ai> References: <20260830051407.50008-1-kiyer@crusoe.ai> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" damon_folio_mkold_one() and damon_folio_young_one() assume the folios they walk are mapped by normal PTEs or THP PMDs. When the folio is a hugetlb folio, page_vma_mapped_walk() returns the huge PTE in pvmw.pte with its page table lock held, but the walkers treat it as a normal PTE: they read and age it with PAGE_SIZE-granularity helpers, which is wrong for huge PTEs (up to PUD level), and notify secondary MMUs for only PAGE_SIZE of the mapping. Add hugetlb branches to both walkers. The mkold walker reuses damon_hugetlb_mkold(), which the virtual address space operations set has been using for hugetlb aging: it clears the young bit of the huge PTE via set_huge_pte_at() and calls mmu_notifier_clear_young() spanning the whole huge page size. The young walker gets an equivalent new helper, damon_hugetlb_young(), which reads the huge PTE with huge_ptep_get() and consults the page idle flag and mmu_notifier_test_young() like the existing PTE branch. Locking mirrors what page_vma_mapped_walk() provides: the huge PTE's page table lock is held inside the walk, and for shared hugetlb mappings (the only ones subject to huge PMD sharing), rmap_walk_file() already holds i_mmap_rwsem, satisfying hugetlb_walk()'s locking requirements. This is currently dead code: both rmap walkers are only reachable through damon_get_folio(), which rejects hugetlb folios since they are not on the LRU lists. A following commit will let the physical address space monitoring primitives opt in to hugetlb folios. Assisted-by: Claude:claude-fable-5 Signed-off-by: Krishna Iyer --- mm/damon/ops-common.c | 63 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 52 insertions(+), 11 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index f5fe92b825bb..62004206ca31 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -193,10 +193,20 @@ static bool damon_folio_mkold_one(struct folio *folio, =20 while (page_vma_mapped_walk(&pvmw)) { addr =3D pvmw.address; - if (pvmw.pte) - damon_ptep_mkold(pvmw.pte, vma, addr); - else + if (pvmw.pte) { + /* + * For hugetlb folios, page_vma_mapped_walk() sets + * pvmw.pte to the huge PTE with its page table lock + * held. + */ + if (folio_test_hugetlb(folio)) + damon_hugetlb_mkold(pvmw.pte, vma->vm_mm, vma, + addr); + else + damon_ptep_mkold(pvmw.pte, vma, addr); + } else { damon_pmdp_mkold(pvmw.pmd, vma, addr); + } } return true; } @@ -221,6 +231,24 @@ void damon_folio_mkold(struct folio *folio) =20 } =20 +#ifdef CONFIG_HUGETLB_PAGE +static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma, + unsigned long addr, struct folio *folio) +{ + pte_t entry =3D huge_ptep_get(vma->vm_mm, addr, pte); + + return (pte_present(entry) && pte_young(entry)) || + !folio_test_idle(folio) || + mmu_notifier_test_young(vma->vm_mm, addr); +} +#else +static bool damon_hugetlb_young(pte_t *pte, struct vm_area_struct *vma, + unsigned long addr, struct folio *folio) +{ + return false; +} +#endif /* CONFIG_HUGETLB_PAGE */ + static bool damon_folio_young_one(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, void *arg) { @@ -232,16 +260,29 @@ static bool damon_folio_young_one(struct folio *folio, while (page_vma_mapped_walk(&pvmw)) { addr =3D pvmw.address; if (pvmw.pte) { - pte =3D ptep_get(pvmw.pte); - /* - * PFN swap PTEs, such as device-exclusive ones, that - * actually map pages are "old" from a CPU perspective. - * The MMU notifier takes care of any device aspects. + * For hugetlb folios, page_vma_mapped_walk() sets + * pvmw.pte to the huge PTE with its page table lock + * held. */ - *accessed =3D (pte_present(pte) && pte_young(pte)) || - !folio_test_idle(folio) || - mmu_notifier_test_young(vma->vm_mm, addr); + if (folio_test_hugetlb(folio)) { + *accessed =3D damon_hugetlb_young(pvmw.pte, vma, + addr, folio); + } else { + pte =3D ptep_get(pvmw.pte); + + /* + * PFN swap PTEs, such as device-exclusive + * ones, that actually map pages are "old" + * from a CPU perspective. The MMU notifier + * takes care of any device aspects. + */ + *accessed =3D (pte_present(pte) && + pte_young(pte)) || + !folio_test_idle(folio) || + mmu_notifier_test_young(vma->vm_mm, + addr); + } } else { #ifdef CONFIG_TRANSPARENT_HUGEPAGE pmd_t pmd =3D pmdp_get(pvmw.pmd); --=20 2.54.0 From nobody Sat Sep 26 21:14:08 2026 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 074A72DEA95 for ; Sun, 30 Aug 2026 05:14:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066854; cv=none; b=AuYC/l29gc5ks+s5+JIX4JUnTHucFAX97k4P7rgIZNK/JicvzhdvfVTYNbYknFR7NC5atm8pb3UZmZS39ljX0z49wPhhp4oHsWsBkvm1Ckmtau7pgf6QN8WuAU9x8jVp3dnW8iiinqF4/Tf2jkCXc4dVEU+Z4nxo2xJtwWOV3r0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066854; c=relaxed/simple; bh=qDar28JySQZOFBLHqw4DU1n4wiE0RevwYt64tfL76/M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qPT20RN9wNMDCSR8slbT00virl+plyO2LnbEaEFIwBKjF4OqdOex47y7Bs66fAAqwxEILS1XjNKR0WE99Dsp7tqvIbNbsRJblxvwqlJtzSUH/I1usWbios2vpVZyknZYHkDeXuKVFw0K/jW3g5tULyJec6j7hb4OOLAtv3XVEyI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai; spf=pass smtp.mailfrom=crusoe.ai; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b=cLSdzcQ0; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b="cLSdzcQ0" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2ce7d2adef4so36228035ad.3 for ; Sat, 29 Aug 2026 22:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crusoe.ai; s=google; t=1788066852; x=1788671652; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Ezj01Z6T9okgOiQa4EOGNSsW3Mil1bOoLksIRT0LM8A=; b=cLSdzcQ0B669YvQHpR9RGQTQtmoUbFAgfvsUNpKI6/51FHgXS/zA9OqeF7StFly33k L4jtT2i2YpQ1Okk8Th4rxZ2cXmk9KPl7Edad3mq8uDVhUoVcAwk9gB6CWHYL1jQr92LG ndQQOmtwFwGwZ8nWUZUaW+c/iEHXfuSyy1oac9afe6QtI83fdj9estWIDS3YczMpjwi8 EBz7OZu4aotked7NMtF+6PnQGdDpz1kFZtpD8LJX614YYl9dfLZM89aXjQwIKDvex7wn sGjW1URJB7QJWxE97dN5iBw6K1kYkNwUmgFzp8Q/HJjXAUE0CTiAX3Ax4yaWSGmzZKiy wfTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788066852; x=1788671652; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Ezj01Z6T9okgOiQa4EOGNSsW3Mil1bOoLksIRT0LM8A=; b=T3Smbitp7Ial0QTl2s15mfc8cU0cRbP+nakZlQ8j+2YkgF8CVDnidL7NSkvayO0ADH APzqxgE5rPxcUzX2RQ2IVW950S8xIvX2EX4JlRqUEFoshDsp8cQxkI1gm0iegndCo129 94qCRaltzlrVXwoJds+AEY7/K8/4vDIldL2J04bKEy/jVlSOvvgCTvr6cqc+mVr4a9wB nH2fAzrme8Z0CGuVAlJqy7kDjuSPOLiQod+OYsiuUtITgy0lDE0ojsW3+fRHV96kcag2 pH0mMDy7ghSGiLUQDEV63cfh7hJ39b0O4pdZdjVE8lVVi1VgESq3uolfzX7nqgJBw0R/ skTw== X-Forwarded-Encrypted: i=1; AKwUvBxA7lk5VvJ/Sfd4V5npqRgf3K5W6oRwM5LTHPn7BmDTMN0JD0HKe775WQDWj+d2fv3gS+k3AB/iSlr139o=@vger.kernel.org X-Gm-Message-State: AFuF++m6kzilCim9oZ/cKzgjSFaip/Ml+ei6xlb5nIa7YntQUaAvSLZf YcISany3iRICiMu5/ezQhtHMm/WZDP0gSQSnf1Efv7T1MX7vcLQor5A9NvaB5DRD4oA= X-Gm-Gg: AR+sD13oNrqjmKLo3h2z++OkvitFlD5qdXr909UIOliuBGxElZBDc0FIZ/lAl9WdKDA PzZ4izfH0PWR6j9HGkmkxe7lM5VDb/kvhrfGUcxKPEp2Mm4b7ppsiALh65zEeE2sl5AiXN0k/rD GdW3/QbgqZrf1wBD35tbf7ShIJPCxIjYU/c/nHb6I9TJkyXY6dZX1Ch6X6tSNObFwZMXr0Gu9QK OrXDI4sAmn4aLB7N/9qAbRw1V1Zz/Vj6qjX9XlHvPNJ0eCJt0IiQOpZKdmSb7WSovw5WyV8Rg8D VrVgziJpNll4M0WwvJOKFyK2QnQrpT2Y8d7yNjzyY9kw+EqXmOBLEG4nr21gIUBIMJR6fnRTwCF yy3wBx2hgIGDllfakmOmWRfKt4+6P+dx3da0/PswZbcYSTfjUzqUZUfyJ39Ru3/vSep69iNlsjL 999NLjaKnaYEPFtoQ9eposYbteU/q7iH4PmgGEgL4w7GnU4Zaxztq7kc30s/rsoasqs195oQM1V 1la0D6NJcR7OmRT6ZqrBcdCVEV3ljnPhrPb X-Received: by 2002:a17:903:2342:b0:2d7:414e:f2f with SMTP id d9443c01a7336-2d74def676fmr265059445ad.14.1788066852465; Sat, 29 Aug 2026 22:14:12 -0700 (PDT) Received: from MBP-Krishna-Iyer.civet-hops.ts.net ([2601:645:c68a:b830:d101:1d:112c:715e]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286fa37fa5sm22595810eec.29.2026.08.29.22.14.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 29 Aug 2026 22:14:12 -0700 (PDT) From: Krishna Iyer To: SeongJae Park Cc: Andrew Morton , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Krishna Iyer Subject: [PATCH 3/6] mm/damon/paddr: support hugetlb folios in access monitoring Date: Sat, 29 Aug 2026 22:14:04 -0700 Message-ID: <20260830051407.50008-4-kiyer@crusoe.ai> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260830051407.50008-1-kiyer@crusoe.ai> References: <20260830051407.50008-1-kiyer@crusoe.ai> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" DAMON's physical address space monitoring is blind to hugetlb-backed memory. Every access check starts at damon_get_folio(), which rejects folios that are not on the LRU lists. Hugetlb folios are managed outside of the LRU by design, so every sampling attempt on hugetlb-backed memory silently fails and the pages are reported as never accessed. This is a significant blind spot on virtualization hosts. Cloud hypervisor hosts commonly back guest memory with 1 GiB hugetlbfs pages, covering the vast majority of the machine's memory. On such hosts, modules like DAMON_STAT observe only the host-side remainder (kernel, page cache, daemons) and report all guest working sets as permanently idle, defeating the purpose of host-level access monitoring. In testing on a 1 TiB host, an hour of 4-thread random access over 842 GiB inside a guest was statistically indistinguishable from an idle host, while a 40x smaller host-side workload produced a quantitatively correct response. Add damon_get_folio_incl_hugetlb(), which additionally accepts hugetlb folios, and use it in the two paddr access monitoring primitives, damon_pa_mkold() and damon_pa_young(). With the previous commit teaching the folio-granular rmap walkers to age huge PTEs and to call the mmu notifiers spanning the whole huge page, this makes guest accesses visible through secondary MMU (e.g. KVM/EPT) young bits. Free hugetlb pool folios have a zero refcount, so folio_try_get() naturally keeps rejecting them. The DAMOS action appliers (damon_pa_pageout(), damon_pa_mark_accessed_or_deactivate(), damon_pa_migrate(), damon_pa_stat()) keep using damon_get_folio(): reclaim, LRU manipulation and migration cannot act on hugetlb folios, so their behavior is unchanged. Note that the access check granularity for hugetlb-backed memory is the huge page size: one touched byte reports the whole (up to 1 GiB) page as accessed. Also, DAMON now consumes secondary MMU young bits that KVM's own aging uses; at DAMON's sampling rate (one page per region per sampling interval) the interference is negligible. Assisted-by: Claude:claude-fable-5 Signed-off-by: Krishna Iyer --- mm/damon/ops-common.c | 31 +++++++++++++++++++++++++++---- mm/damon/ops-common.h | 1 + mm/damon/paddr.c | 4 ++-- 3 files changed, 30 insertions(+), 6 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 62004206ca31..ece101d34684 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -15,14 +15,20 @@ #include "../internal.h" #include "ops-common.h" =20 +static bool damon_folio_observable(struct folio *folio, bool incl_hugetlb) +{ + return folio_test_lru(folio) || + (incl_hugetlb && folio_test_hugetlb(folio)); +} + /* - * Get an online page for a pfn if it's in the LRU list. Otherwise, retur= ns - * NULL. + * Get an online page for a pfn if it's in the LRU list, or a hugetlb foli= o if + * @incl_hugetlb is set. Otherwise, returns NULL. * * The body of this function is stolen from the 'page_idle_get_folio()'. = We * steal rather than reuse it because the code is quite simple. */ -struct folio *damon_get_folio(unsigned long pfn) +static struct folio *__damon_get_folio(unsigned long pfn, bool incl_hugetl= b) { struct page *page =3D pfn_to_online_page(pfn); struct folio *folio; @@ -33,13 +39,30 @@ struct folio *damon_get_folio(unsigned long pfn) folio =3D page_folio(page); if (!folio_try_get(folio)) return NULL; - if (unlikely(page_folio(page) !=3D folio) || !folio_test_lru(folio)) { + if (unlikely(page_folio(page) !=3D folio) || + !damon_folio_observable(folio, incl_hugetlb)) { folio_put(folio); folio =3D NULL; } return folio; } =20 +struct folio *damon_get_folio(unsigned long pfn) +{ + return __damon_get_folio(pfn, false); +} + +/* + * Same to damon_get_folio(), but also accepts hugetlb folios, which are + * managed outside of the LRU lists. Aimed to be used by access monitoring + * primitives. DAMOS actions that assume LRU-managed folios should keep u= sing + * damon_get_folio(). + */ +struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn) +{ + return __damon_get_folio(pfn, true); +} + void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr) { pte_t pteval =3D ptep_get(pte); diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index f7811c9c7a02..68d7de49c87a 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -6,6 +6,7 @@ #include =20 struct folio *damon_get_folio(unsigned long pfn); +struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn); =20 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr); void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr); diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 5c6c3a597fd0..09d418b2874b 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -37,7 +37,7 @@ static unsigned long damon_pa_core_addr( =20 static void damon_pa_mkold(phys_addr_t paddr) { - struct folio *folio =3D damon_get_folio(PHYS_PFN(paddr)); + struct folio *folio =3D damon_get_folio_incl_hugetlb(PHYS_PFN(paddr)); =20 if (!folio) return; @@ -67,7 +67,7 @@ static void damon_pa_prepare_access_checks(struct damon_c= tx *ctx) =20 static bool damon_pa_young(phys_addr_t paddr) { - struct folio *folio =3D damon_get_folio(PHYS_PFN(paddr)); + struct folio *folio =3D damon_get_folio_incl_hugetlb(PHYS_PFN(paddr)); bool accessed; =20 if (!folio) --=20 2.54.0 From nobody Sat Sep 26 21:14:08 2026 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 244C32EBBB7 for ; Sun, 30 Aug 2026 05:14:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066857; cv=none; b=elgYH2UElsyhEBLGHmo9HdBQIHf8/OyIylOGfzOb6I7xydmall+PApz7GFYG8h+/oeYLJiYbZhxryrvNLrJgQKbB2XEp4b405AaHQglqPRxX12VkxL3XBgvhlukVeH6wnHiTXyw8qNZhxjThlkMkCJhDbx2F1sc/2my5CkzOiJk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066857; c=relaxed/simple; bh=iV3sIWrGxvWungLl3IqCNLrET/Em/Bu90IrLBSW4Yxk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qGyzekMq9VWYXbozBihn9vjf3AMG2QD72bueL/ok3565eJf1n5B8i92VDLqoHAPfjdIvxrS/0mKDKYtzB6EsTsZOnXUPzFIHkbDVAeryVbuQ8R6a6JsKrQnBOy50dCebd5VFF6HPAvjUF5FR2DBZsZ3eP/av99oh7MXfAFrnpR0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai; spf=pass smtp.mailfrom=crusoe.ai; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b=U18EWI+g; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b="U18EWI+g" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-855d2bfae95so2535920b3a.1 for ; Sat, 29 Aug 2026 22:14:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crusoe.ai; s=google; t=1788066853; x=1788671653; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=1gXFO776y6OOEgXqFyZQVNGCJghM1qCvRqbZsaMk0vw=; b=U18EWI+gSzFCn8/umrsma+543e5b/Mnro+Vt2VLZTX6qC42XOSOOEss/rKxU6IHlyM l5AguLk9SFQcLINwmqB2Sc3UpE1C+hSsnNfR6w1S0iueJ4YltSCIhDxRxLwenS+3iTE5 dkrExZ0yYrDtBfTxoC5FtnXq5BA+U/ZgsrNNGxF2AK+lOZki0ZM5KhuJVYCLJMQBrlmI 1Pwin+2I3HEGyqAaPLJlzeGbhbyto63+OdjWvjDoV6ZJmyYCfd2MbTJCc3pNVCfipTdv PFCgFjxbtVmRXLjdYO7iUD8PIorAjxZ1Az0gLCcB4wnQROriH9iCrrcGH1+yGrtyYzgG SXUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788066853; x=1788671653; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=1gXFO776y6OOEgXqFyZQVNGCJghM1qCvRqbZsaMk0vw=; b=XuYIPe5XfTjsh9zrzETih9f/ld5UnPfq7O4Q64+015cljruilUYIbEqc19j63rK3Eq o5Zs+OQzxRtQqHjNSyWM4McOPRprvQgOmloJGOdC139ZSK2seW2oPmiYba/DSF7N32K1 LTubi5FnAeUTdbSPExd8yPOJWp1g/uTMxTcWXH1BsslKcEAkFSRECMP1KEYhRnZ+MeQr BqicnHZ7W0U6O41j0gz8NxNOhcxc74cVFJcJSj1zuKJZ0/RDiXUt2WZHproMZTETu/B3 pmBjwn7SBIaf6VH3AjIqUGrdEVMrmrp6GqciaJinzx+354lEn3OvfUd8b+FU+FGctDCw e8WQ== X-Forwarded-Encrypted: i=1; AHgh+RrXlEOjgK6Cf2ofNN5IMJqRa4Z0yHU2QHacJ9QPN+qfqAPsbmiFcFGH4gyr3AT7z3ekKg5/NcACa9rvwMw=@vger.kernel.org X-Gm-Message-State: AFuF++mzGs3D6BxIWZWpNPdHpaBDaiRoh8sVFSxVpsrae946gJKU4/lf ucN2gIDpznxCW4GH+d3NzD2YT3dtoZCxX2qwBBMsHoRVywObyR2u5M3I5+m7CbmxUv1QGXxpGIl 3i9FM7c4= X-Gm-Gg: AR+sD10W/aOSQ/hQG1MtEAhAazmIjYglKiWLDi5DEGf21+AdDYiJPosFL+GKGNfZ+3K ccJ4mbWMqmqO/PEA2NuD8t1Y+VuXdklGuVF3205uPzjrF9N1S5pRBpP+KbHtucT3pKrUSH6IegZ nymX1+ZWHZj+i00QOwtpsQ80G2W1lAbEmJakhra0zvxpRxSTcnexNNUam3ti5T37uL+Ymu3vksb 2pFW/MFH6BHycc8aOscHkKLB+E0HDzc4+iYF6cqapOR/JDXYp8atQsV2ALBbwI7iUUi1wqOyTGm 1Roi+SYysf6OoiEjzVM6rCv4ioa7G2VpQYLQRakBr0qtmS/CfcK637x+K05nOK6b+7HV1uEvzCr B+AOmHvf+37/cvwL8HDfZik8sAvjo7fKpGj3pdQ9loBu/oPiTV+qIpuQbs90Mz1ggnZgFLmI6b6 lq0PVvWKL+by4uKjTAJss3SE7rHuHDOcAHmdF/+YEisSqRUMDLrEEorJYx6a+fazshYYiYmFQ30 aFoktUAHOodY2LmMJE8pH5Ij6GZ14SIQmU= X-Received: by 2002:a05:6a21:7e84:b0:3d3:3dca:f0a3 with SMTP id adf61e73a8af0-3d5d787403dmr293487637.8.1788066853425; Sat, 29 Aug 2026 22:14:13 -0700 (PDT) Received: from MBP-Krishna-Iyer.civet-hops.ts.net ([2601:645:c68a:b830:d101:1d:112c:715e]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286fa37fa5sm22595810eec.29.2026.08.29.22.14.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 29 Aug 2026 22:14:13 -0700 (PDT) From: Krishna Iyer To: SeongJae Park Cc: Andrew Morton , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Krishna Iyer Subject: [PATCH 4/6] mm/damon: support flush-assisted access bit clearing for monitoring Date: Sat, 29 Aug 2026 22:14:05 -0700 Message-ID: <20260830051407.50008-5-kiyer@crusoe.ai> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260830051407.50008-1-kiyer@crusoe.ai> References: <20260830051407.50008-1-kiyer@crusoe.ai> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" DAMON's access sampling clears the accessed bit of the sampled page and re-reads it one sampling interval later. The clearing uses the non-flushing primitives, deliberately trading accuracy for low overhead: cached TLB translations survive the clearing, and accesses through such cached translations do not perform page table walks, hence do not set the accessed bit again. The bit therefore re-arms at the TLB-refill rate rather than the actual access rate. For most setups this is a good trade-off, since TLB pressure re-walks hot pages frequently anyway. It breaks down when translations are long-lived, most prominently on virtualization hosts backing guest memory with 1 GiB hugetlb pages: covering hundreds of GiB takes only hundreds of TLB entries, translations essentially never get evicted, and secondary MMU (e.g. KVM/EPT) accessed bits cleared via mmu_notifier_clear_young() are rarely set again. Measurements on a 1 TiB host showed the reported access fraction pinned at the TLB-refill rate: a guest workload continuously accessing 842 GiB was indistinguishable from an idle guest. Add an 'aging_flush' monitoring context option, default off. When set, the sampling primitives pair the accessed bit clearing with mmu_notifier_clear_flush_young(), so secondary MMUs invalidate their cached translations and the next guest access must re-walk and re-set the accessed bit, making the sampled young state reflect the actual access rate. The flush covers secondary MMU TLBs only, uniformly across the PTE, PMD and hugetlb paths. Primary MMU TLB young-flushing is deliberately not added: x86 already implements ptep_clear_flush_young() without a flush on the grounds that primary-TLB accessed-bit staleness is not worth the flush cost, and the observability target here is secondary MMU (guest) access state. The cost is secondary TLB invalidations during the sampling prepare pass. KVM coalesces flush requests issued in a burst (pending-request deduping), so a prepare pass over all regions results in roughly one guest TLB flush per pass, and hosts whose guests are backed by huge pages have small TLB working sets that are cheap to re-fill -- exactly the hosts that need this option. Measured on a 1 TiB host with a 176-vCPU guest: ~240 flush calls/sec at auto-tuned intervals, bounding worst-case aggregate overhead below 0.1%. DAMOS filter paths keep the non-flushing behavior unconditionally. Assisted-by: Claude:claude-fable-5 Signed-off-by: Krishna Iyer --- include/linux/damon.h | 2 ++ mm/damon/core.c | 2 ++ mm/damon/ops-common.c | 40 ++++++++++++++++++++++++++++------------ mm/damon/ops-common.h | 12 +++++++----- mm/damon/paddr.c | 7 ++++--- mm/damon/vaddr.c | 19 +++++++++++-------- 6 files changed, 54 insertions(+), 28 deletions(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 0c8b7ddef9ab..7fc438677dd8 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -848,6 +848,7 @@ struct damon_attrs { * including damon_call() and damos_walk(). * * @addr_unit: Scale factor for core to ops address conversion. + * @aging_flush: Flush TLBs when clearing access bits for sampling. * @min_region_sz: Minimum region size. * @pause: Pause kdamond main loop. */ @@ -899,6 +900,7 @@ struct damon_ctx { =20 /* public: */ unsigned long addr_unit; + bool aging_flush; unsigned long min_region_sz; bool pause; =20 diff --git a/mm/damon/core.c b/mm/damon/core.c index 644daf5a1656..46d6ec3dbffa 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -859,6 +859,7 @@ struct damon_ctx *damon_new_ctx(void) INIT_LIST_HEAD(&ctx->probes); =20 ctx->addr_unit =3D 1; + ctx->aging_flush =3D false; ctx->min_region_sz =3D DAMON_MIN_REGION_SZ; =20 INIT_LIST_HEAD(&ctx->adaptive_targets); @@ -1800,6 +1801,7 @@ static int __damon_commit_ctx(struct damon_ctx *dst, = struct damon_ctx *src) if (err) return err; dst->addr_unit =3D src->addr_unit; + dst->aging_flush =3D src->aging_flush; dst->min_region_sz =3D src->min_region_sz; =20 dst->maybe_corrupted =3D false; diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index ece101d34684..5c2cd13b9b05 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -63,7 +63,8 @@ struct folio *damon_get_folio_incl_hugetlb(unsigned long = pfn) return __damon_get_folio(pfn, true); } =20 -void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr) +void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr, + bool flush) { pte_t pteval =3D ptep_get(pte); struct folio *folio; @@ -86,7 +87,12 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct = *vma, unsigned long addr */ if (likely(pte_present(pteval))) young |=3D ptep_test_and_clear_young(vma, addr, pte); - young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + if (flush) + young |=3D mmu_notifier_clear_flush_young(vma->vm_mm, addr, + addr + PAGE_SIZE); + else + young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, + addr + PAGE_SIZE); if (young) folio_set_young(folio); =20 @@ -94,7 +100,8 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct = *vma, unsigned long addr folio_put(folio); } =20 -void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr) +void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr, + bool flush) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE pmd_t pmdval =3D pmdp_get(pmd); @@ -113,7 +120,12 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struc= t *vma, unsigned long addr =20 if (likely(pmd_present(pmdval))) young |=3D pmdp_test_and_clear_young(vma, addr, pmd); - young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, addr + HPAGE_PMD_SI= ZE); + if (flush) + young |=3D mmu_notifier_clear_flush_young(vma->vm_mm, addr, + addr + HPAGE_PMD_SIZE); + else + young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, + addr + HPAGE_PMD_SIZE); if (young) folio_set_young(folio); =20 @@ -124,7 +136,7 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct= *vma, unsigned long addr =20 #ifdef CONFIG_HUGETLB_PAGE void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long addr) + struct vm_area_struct *vma, unsigned long addr, bool flush) { bool referenced =3D false; pte_t entry =3D huge_ptep_get(mm, addr, pte); @@ -139,8 +151,10 @@ void damon_hugetlb_mkold(pte_t *pte, struct mm_struct = *mm, set_huge_pte_at(mm, addr, pte, entry, psize); } =20 - if (mmu_notifier_clear_young(mm, addr, - addr + huge_page_size(hstate_vma(vma)))) + if (flush ? mmu_notifier_clear_flush_young(mm, addr, + addr + huge_page_size(hstate_vma(vma))) : + mmu_notifier_clear_young(mm, addr, + addr + huge_page_size(hstate_vma(vma)))) referenced =3D true; =20 if (referenced) @@ -212,6 +226,7 @@ int damon_cold_score(struct damon_ctx *c, struct damon_= region *r, static bool damon_folio_mkold_one(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, void *arg) { + bool flush =3D *(bool *)arg; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, addr, 0); =20 while (page_vma_mapped_walk(&pvmw)) { @@ -224,19 +239,20 @@ static bool damon_folio_mkold_one(struct folio *folio, */ if (folio_test_hugetlb(folio)) damon_hugetlb_mkold(pvmw.pte, vma->vm_mm, vma, - addr); + addr, flush); else - damon_ptep_mkold(pvmw.pte, vma, addr); + damon_ptep_mkold(pvmw.pte, vma, addr, flush); } else { - damon_pmdp_mkold(pvmw.pmd, vma, addr); + damon_pmdp_mkold(pvmw.pmd, vma, addr, flush); } } return true; } =20 -void damon_folio_mkold(struct folio *folio) +void damon_folio_mkold(struct folio *folio, bool flush) { struct rmap_walk_control rwc =3D { + .arg =3D &flush, .rmap_one =3D damon_folio_mkold_one, .anon_lock =3D folio_lock_anon_vma_read, }; @@ -377,7 +393,7 @@ bool damos_folio_filter_match(struct damos_filter *filt= er, struct folio *folio) case DAMOS_FILTER_TYPE_YOUNG: matched =3D damon_folio_young(folio); if (matched) - damon_folio_mkold(folio); + damon_folio_mkold(folio, false); break; case DAMOS_FILTER_TYPE_HUGEPAGE_SIZE: folio_sz =3D folio_size(folio); diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index 68d7de49c87a..df7ef025bc1a 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -8,18 +8,20 @@ struct folio *damon_get_folio(unsigned long pfn); struct folio *damon_get_folio_incl_hugetlb(unsigned long pfn); =20 -void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr); -void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr); +void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr, + bool flush); +void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr, + bool flush); #ifdef CONFIG_HUGETLB_PAGE void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long addr); + struct vm_area_struct *vma, unsigned long addr, bool flush); #else static inline void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long addr) + struct vm_area_struct *vma, unsigned long addr, bool flush) { } #endif /* CONFIG_HUGETLB_PAGE */ -void damon_folio_mkold(struct folio *folio); +void damon_folio_mkold(struct folio *folio, bool flush); bool damon_folio_young(struct folio *folio); =20 int damon_cold_score(struct damon_ctx *c, struct damon_region *r, diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 09d418b2874b..f2aaa82325d1 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -35,14 +35,14 @@ static unsigned long damon_pa_core_addr( return pa / addr_unit; } =20 -static void damon_pa_mkold(phys_addr_t paddr) +static void damon_pa_mkold(phys_addr_t paddr, bool flush) { struct folio *folio =3D damon_get_folio_incl_hugetlb(PHYS_PFN(paddr)); =20 if (!folio) return; =20 - damon_folio_mkold(folio); + damon_folio_mkold(folio, flush); folio_put(folio); } =20 @@ -51,7 +51,8 @@ static void __damon_pa_prepare_access_check(struct damon_= region *r, { r->sampling_addr =3D damon_rand(ctx, r->ar.start, r->ar.end); =20 - damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, ctx->addr_unit)); + damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, ctx->addr_unit), + ctx->aging_flush); } =20 static void damon_pa_prepare_access_checks(struct damon_ctx *ctx) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 15379d984689..7c11022bb664 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -276,7 +276,8 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned l= ong addr, pmd_t pmde =3D pmdp_get(pmd); =20 if (pmd_present(pmde)) - damon_pmdp_mkold(pmd, walk->vma, addr); + damon_pmdp_mkold(pmd, walk->vma, addr, + *(bool *)walk->private); spin_unlock(ptl); return 0; } @@ -286,7 +287,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned l= ong addr, return 0; if (!pte_present(ptep_get(pte))) goto out; - damon_ptep_mkold(pte, walk->vma, addr); + damon_ptep_mkold(pte, walk->vma, addr, *(bool *)walk->private); out: pte_unmap_unlock(pte, ptl); return 0; @@ -306,7 +307,8 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsign= ed long hmask, if (!pte_present(entry)) goto out; =20 - damon_hugetlb_mkold(pte, walk->mm, walk->vma, addr); + damon_hugetlb_mkold(pte, walk->mm, walk->vma, addr, + *(bool *)walk->private); =20 out: spin_unlock(ptl); @@ -316,14 +318,15 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsi= gned long hmask, #define damon_mkold_hugetlb_entry NULL #endif /* CONFIG_HUGETLB_PAGE */ =20 -static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) +static void damon_va_mkold(struct mm_struct *mm, unsigned long addr, + bool flush) { struct mm_walk_ops damon_mkold_ops =3D { .pmd_entry =3D damon_mkold_pmd_entry, .hugetlb_entry =3D damon_mkold_hugetlb_entry, }; =20 - damon_va_walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL); + damon_va_walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, &flush); } =20 /* @@ -336,7 +339,7 @@ static void __damon_va_prepare_access_check(struct mm_s= truct *mm, { r->sampling_addr =3D damon_rand(ctx, r->ar.start, r->ar.end); =20 - damon_va_mkold(mm, r->sampling_addr); + damon_va_mkold(mm, r->sampling_addr, ctx->aging_flush); } =20 static void damon_va_prepare_access_checks(struct damon_ctx *ctx) @@ -508,9 +511,9 @@ static bool damos_va_filter_young_match(struct damos_fi= lter *filter, mmu_notifier_test_young(vma->vm_mm, addr); =20 if (young && ptep) - damon_ptep_mkold(ptep, vma, addr); + damon_ptep_mkold(ptep, vma, addr, false); else if (young && pmdp) - damon_pmdp_mkold(pmdp, vma, addr); + damon_pmdp_mkold(pmdp, vma, addr, false); =20 return young =3D=3D filter->matching; } --=20 2.54.0 From nobody Sat Sep 26 21:14:08 2026 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4E502EEE8A for ; Sun, 30 Aug 2026 05:14:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066857; cv=none; b=RiZt3Rhn5oUr20UeSnlfYd2moeM1J7E4+3tPjI+3WM5i6cUTjyTi4XvbGjBbW80zViAiuViO9GnmSKxYab4ALS3y8maaJEaETJYDlaK5kkCULDLW962KOgRjvPdcnCljdxhWa1ue2CJ5dIv25gESYrZE6tLY03B/KyPmhVZoAFk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066857; c=relaxed/simple; bh=1IoYOQdQXFvwKzLHxgB+uq3xpg0ikIIhqsw4J/nkMyc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aIvJ262O+RtsXK+5sOQHeRTu3CWSOwNdG3d4Uv3uKxcburBLkL+xTtQgJqhzTxp/9Wm2vzqUCCu3/zO3Vstx+SUHcNAqKRxWGbwA814GlKjx+tNW49ISjBP0jhhIIiJcPXFxilWYGu7yGZ24Yzi+Cn9OMnNCmwmtPSpmNjuLq+g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai; spf=pass smtp.mailfrom=crusoe.ai; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b=D3rvEC+o; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b="D3rvEC+o" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-39647aa9d52so2338860a91.0 for ; Sat, 29 Aug 2026 22:14:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crusoe.ai; s=google; t=1788066854; x=1788671654; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=nDQFaZkSZClgPI6vQ9uvEW8golFAoyljS7GedCtCSvM=; b=D3rvEC+olE1+Wu1sy1lP2oKd2kkWDYpDpNSmcjmtWVGXxAhyI4V0oXi4KZtPluRIf4 1CEpl21ZXgE6LRNjqew+KrLfvpXn8Q9i5AebP6w+/ruTU+zHPOuhjGFHHbHldXarvtiC ATGXmr+k+jKWIZmYBuVxnt4B4SFBa/4WkdZMsMX+wal2dA0DHObIliq5eKd4pyJ8nYc5 akZ7HcdWNMzBIE6l9yBKVj2iyDrvXx04RYrrDUVtNh7sTRRa2WhRwEE6A5PFjrQQnvtC Z2AaOoEv3LezPoceSEPjvDnxhuri0s6TcM9KPse8I8+0BuGMKvNIvdHc/JukhVHr0NpH ovYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788066854; x=1788671654; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=nDQFaZkSZClgPI6vQ9uvEW8golFAoyljS7GedCtCSvM=; b=aRSrQmRXCLItlOswHbWF2GDo01cLEXjpjkKxibLnig4ZdB7Apm56Z5ZyYNi6C06atc K3byss2q7gLmwRbo/FCHsE1pf9smonFGYsqE+/sicZcux3iNwh6et2eLfwZ5hDkAOTnY tJLPS2JFXQBnX5RXpUb1FRIRGg+hGywn89qeF1m9mr365dOyaHluUGDHBXT/nZFjIFuZ JhnfgsE9YQrbfHREMXAdZN8UhWVJD2LkNbSYfyKPn8f58Nqgsvq8r1Hxo71OjPR7CnZY DxLgl9u37xuOsYxzQ21nZCyJEkyc+TfYZZcf9lHbxt2R0RrTdkxK98oHCe+w5yoSYtDK f6yA== X-Forwarded-Encrypted: i=1; AKwUvBxxpb3pbENr6/K0Eg4JKkxQfmQLBar9Gn/l4bnLgmZu8RBFSoKBjUAGnzaTF2jE7hZlLF+o7eQ7E5zaiK0=@vger.kernel.org X-Gm-Message-State: AFuF++lnUxYxer9XP7u8zR/NfYQRIK5BtnwosNR953E0kYqbr6rmN86V kH0mL/g0hqSA40a0YorSpdUGLwcoluVUDCaeueq3PYpuEiiurGenw0guC6TLeXcrX4k= X-Gm-Gg: AYBFou0mJdXFG0OOFquDcDi80gxlvWq1GZGjfQnJVfO5FRmCMDaYYIJ9f/YPBwgJZQn UjBlavocgmnCBmROxV3kpLK5mnMX32233SLZeHPVMPXr+YJ9OWMFQx4o2LnEG+85rfiq0QhTs3E t8Jt7IsocV3n8Wt6OKQqgx/yYT6qx4wLx5hd6Bnk0Lc88fZ21xoN23pF1HHFCq8/XbfEj5wL2jq 88kX90D9dmRYpq7ZH17BNVzszR4IPNovEvGw6wBVxGPV7rQ1mxPdC6afMc3e58RNHtXSSjjX1cK UQTzG40S3baZKP/tm2PepH7ZW3ah6x0gHPDT/5OwKpHXSlqgQw44SkvJGM3cEMP+lusd2Uy5phH 8mdXMMOgQOgwOOWdki/oc/Jvi83Ml670pUfY3jspVUYrpU9aE/SyCL+oRfUoAKCCr4qec+Lvk37 nYih+CdGguwrIZSvq/r7pzynYud6jHkEnHDT3JNZ3/FmQvtNYEI294D77JJqCjx0RpQutbXCAOo 3GDI0DHhZryiuSC8mSfYmAsioGyMwQn0vs= X-Received: by 2002:a17:90b:53ce:b0:398:c0a2:8f8e with SMTP id 98e67ed59e1d1-398da8b6e92mr295447a91.26.1788066854435; Sat, 29 Aug 2026 22:14:14 -0700 (PDT) Received: from MBP-Krishna-Iyer.civet-hops.ts.net ([2601:645:c68a:b830:d101:1d:112c:715e]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286fa37fa5sm22595810eec.29.2026.08.29.22.14.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 29 Aug 2026 22:14:14 -0700 (PDT) From: Krishna Iyer To: SeongJae Park Cc: Andrew Morton , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Krishna Iyer Subject: [PATCH 5/6] mm/damon/sysfs: support aging_flush Date: Sat, 29 Aug 2026 22:14:06 -0700 Message-ID: <20260830051407.50008-6-kiyer@crusoe.ai> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260830051407.50008-1-kiyer@crusoe.ai> References: <20260830051407.50008-1-kiyer@crusoe.ai> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Expose the aging_flush monitoring context option via a DAMON sysfs context file of the same name, next to addr_unit. Accepts boolean strings (kstrtobool); defaults to N (current behavior). Assisted-by: Claude:claude-fable-5 Signed-off-by: Krishna Iyer --- mm/damon/sysfs.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/mm/damon/sysfs.c b/mm/damon/sysfs.c index e3858ffab4b2..c8bc9cfbe4cf 100644 --- a/mm/damon/sysfs.c +++ b/mm/damon/sysfs.c @@ -1403,6 +1403,7 @@ struct damon_sysfs_context { struct kobject kobj; enum damon_ops_id ops_id; unsigned long addr_unit; + bool aging_flush; struct damon_sysfs_attrs *attrs; struct damon_sysfs_targets *targets; struct damon_sysfs_schemes *schemes; @@ -1618,6 +1619,29 @@ static ssize_t pause_store(struct kobject *kobj, str= uct kobj_attribute *attr, return count; } =20 +static ssize_t aging_flush_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct damon_sysfs_context *context =3D container_of(kobj, + struct damon_sysfs_context, kobj); + + return sysfs_emit(buf, "%c\n", context->aging_flush ? 'Y' : 'N'); +} + +static ssize_t aging_flush_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buf, size_t count) +{ + struct damon_sysfs_context *context =3D container_of(kobj, + struct damon_sysfs_context, kobj); + bool input_aging_flush; + int err =3D kstrtobool(buf, &input_aging_flush); + + if (err) + return err; + + context->aging_flush =3D input_aging_flush; + return count; +} =20 static void damon_sysfs_context_release(struct kobject *kobj) { @@ -1636,11 +1660,15 @@ static struct kobj_attribute damon_sysfs_context_ad= dr_unit_attr =3D static struct kobj_attribute damon_sysfs_context_pause_attr =3D __ATTR_RW_MODE(pause, 0600); =20 +static struct kobj_attribute damon_sysfs_context_aging_flush_attr =3D + __ATTR_RW_MODE(aging_flush, 0600); + static struct attribute *damon_sysfs_context_attrs[] =3D { &damon_sysfs_context_avail_operations_attr.attr, &damon_sysfs_context_operations_attr.attr, &damon_sysfs_context_addr_unit_attr.attr, &damon_sysfs_context_pause_attr.attr, + &damon_sysfs_context_aging_flush_attr.attr, NULL, }; ATTRIBUTE_GROUPS(damon_sysfs_context); @@ -2102,6 +2130,7 @@ static int damon_sysfs_apply_inputs(struct damon_ctx = *ctx, if (err) return err; ctx->addr_unit =3D READ_ONCE(sys_ctx->addr_unit); + ctx->aging_flush =3D READ_ONCE(sys_ctx->aging_flush); /* addr_unit is respected by only DAMON_OPS_PADDR */ if (ops_id =3D=3D DAMON_OPS_PADDR) ctx->min_region_sz =3D max( --=20 2.54.0 From nobody Sat Sep 26 21:14:08 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C0FB2EEE96 for ; Sun, 30 Aug 2026 05:14:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066857; cv=none; b=tNyu/5KqBJYl9MuG+vEjl3i6wfzZpfoNDKZKGAv4j5Wrf49iPPqM0KtcObxxoqxW6lpy1qU3bDq+GYcvzfqfWEmlGqVLya25MXmHhEe8QOjUy4gTCTaoy8mxMaRQTDVRNQOmwcbFqLkLOnCphom2oQoYpV6eXK5CGtK67Zm3Sqo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788066857; c=relaxed/simple; bh=r0ftD7HCIBo2WTmrRLgxokYhFEMjbae70/mAgw1rVsQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uQtZ5AxU7z0QfEwO9QiH7XTBkvFz0Ri7sOE3LIJZztAlzzRMJ9z+jvvVWosLQCCKXRt+wF11MgrNcHcBs48dyCWZSBUltGtMhVpMbxPthXa8X9YvcdokcgGvJV33AuZSRm7ZgG+g6vKT32wZ9+MFrLZandqnwhvtUuHpZKZYiHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai; spf=pass smtp.mailfrom=crusoe.ai; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b=IKU5gWHY; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crusoe.ai Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=crusoe.ai header.i=@crusoe.ai header.b="IKU5gWHY" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2d6f55a7089so23377145ad.3 for ; Sat, 29 Aug 2026 22:14:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crusoe.ai; s=google; t=1788066855; x=1788671655; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4psP7AFDz+Kluf5IGgOzZSSfstXE1tXapxgbnPUwWEA=; b=IKU5gWHYrbIXn/IYvCIEbY/rBJO8rez+0YNiIlUaszyAUICiJuMiqtKCfD+HP89Go9 ekj40NW+alqcrW2qOhknnoakvIDorZAUksm76XogQa3fPKI2xtPxCrz5Vf8/Ylg9ZkKp t64v0h2KfMuEMDhgTk720tmyZ+/LsCt+FsSUHPB7WDxRuLMCb6ShystOJECGhyrccm15 dPeNuZ8rCuLuzjAD9SgyrKr9J9c/uWeJd+KZweZ56HhHRskRia5xwZVjxQt2fsIjFmD3 4t6PdGZ6rYfJ2bpOKdPY4MB6sHDn+Mj7ZmAY0LUrkosWKLl5BDCWQoFi5ybm7c1+DtvC mH5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788066855; x=1788671655; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=4psP7AFDz+Kluf5IGgOzZSSfstXE1tXapxgbnPUwWEA=; b=n00oFz7xTBf4LVnrVQrdvEDSo9/YCaOZoQhVq4Y6PDVJbSMNlo23RqBbSTGTQyCGaF KhTgjlhL8Svaomx2zlXyYCYwSjg77VAnsT/o2L1OGndWe2E/0idHqGtOatQPvLWAeAbu FboeMko6h0nRw+zpgzUo+PyggmihXirq/VHrtt8fIE/QBuxobf+y0yys9E0LqTWgZb7l pZEAdRQs5QK7tKqcjRFAT0PHtqvc0OJlpJyRTI5j1r623NgWtzpwPDIChoHiBSlr9fys dKc7xa02TP2Zl9xdVBdvOIRpfIM4on2TSXchsooZZ4EgGpFRsvezTvsf1/xduo6IbvNN XTDg== X-Forwarded-Encrypted: i=1; AKwUvBzMLGoH4UhxOXqWUt2XeYNcAOO+H3SmZQ3YQcfWXpAo3j5pCcbwTpsZrF/FG/YOLJP6wcwE+yfpC8xzNVk=@vger.kernel.org X-Gm-Message-State: AFuF++kRwqIvmTd/DQPyorLJdBURBAhahCTNpj32XU7tLgc18K+SvvSW H9rGlX9CdGEYCOvPP8NyFFS9/YN3qu75vu/5LRcujdM7uO1eZSt3MJiVHawO5Z38pMU= X-Gm-Gg: AYBFou1WbO8cX+ZGW/dg2+YSZ2I3EvWyUXPlbrEw4bevdYw8G1+jGQZcet63t9HiHxR b3dYdMZsGcbNiby+nBfEwY5W/Q5JxGFpcxzsgGxlCMmCvKdHOn8u2udC6DuvmHbjznBI4ah6WAB 7wI9qVQUfS+5RwKzS407sUqcjX3dWepA9ay8SgTFNHbJoXiUkxLQiRkcvOnd+F77ySfimhoQUdS dHioCf6tn8/FxfvgOHuZo2EPCWeD6wHg/hzdhADXLzukiIUsVAsxRzgzhEmmodHf0yjDtigOwi8 9ZHComQO+Ix26NhmuTDMv5EKnhW7bId2lGOCJl8IXdmsttL/x+MJ9udD956ARCOM/KPIflW4dMe RrTx7Qjdghxwlo/oNLtMwHWMoxKpZX4en+r4xPdghPMUGChwsD1NUXCebDl8Ec02tElpb51N6fF 0nynx5YWeQzKW6v78GVPK+yQRG+1r8A6ng0Yv3/wQsR+FeuS3LCbBQLyj105xbtZSSyAnvID52W tMnt7/vcGxPKGsZP36E32vKITW+8MgKCqha X-Received: by 2002:a17:90b:2e4f:b0:398:9bd5:490e with SMTP id 98e67ed59e1d1-3989bd54c43mr16185087a91.21.1788066855436; Sat, 29 Aug 2026 22:14:15 -0700 (PDT) Received: from MBP-Krishna-Iyer.civet-hops.ts.net ([2601:645:c68a:b830:d101:1d:112c:715e]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3286fa37fa5sm22595810eec.29.2026.08.29.22.14.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sat, 29 Aug 2026 22:14:15 -0700 (PDT) From: Krishna Iyer To: SeongJae Park Cc: Andrew Morton , damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Krishna Iyer Subject: [PATCH 6/6] mm/damon/stat: support aging_flush Date: Sat, 29 Aug 2026 22:14:07 -0700 Message-ID: <20260830051407.50008-7-kiyer@crusoe.ai> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260830051407.50008-1-kiyer@crusoe.ai> References: <20260830051407.50008-1-kiyer@crusoe.ai> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let DAMON_STAT users opt in to flush-assisted access bit clearing via a module parameter of the same name as the monitoring context option. The parameter is applied when the monitoring context is built, i.e., at enable time. Defaults to off (current behavior). Intended for virtualization hosts backing guest memory with hugetlb pages, where non-flushing aging pins the observed access rate to the TLB-refill rate (see the aging_flush core patch for details). Such deployments can set damon_stat.aging_flush=3D1 on the kernel command line. Assisted-by: Claude:claude-fable-5 Signed-off-by: Krishna Iyer --- mm/damon/stat.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/damon/stat.c b/mm/damon/stat.c index b05b68f73e10..e093383ba406 100644 --- a/mm/damon/stat.c +++ b/mm/damon/stat.c @@ -47,6 +47,11 @@ module_param(aggr_interval_us, ulong, 0400); MODULE_PARM_DESC(aggr_interval_us, "Current tuned aggregation interval in microseconds"); =20 +static bool aging_flush __read_mostly; +module_param(aging_flush, bool, 0600); +MODULE_PARM_DESC(aging_flush, + "Flush TLBs when clearing access bits (applied at enable time)"); + static struct damon_ctx *damon_stat_context; =20 static unsigned long damon_stat_last_refresh_jiffies; @@ -178,6 +183,7 @@ static struct damon_ctx *damon_stat_build_ctx(void) =20 if (damon_select_ops(ctx, DAMON_OPS_PADDR)) goto free_out; + ctx->aging_flush =3D aging_flush; =20 target =3D damon_new_target(); if (!target) --=20 2.54.0