From nobody Mon Sep 28 05:51:03 2026 Received: from SHSQR01.spreadtrum.com (mx1.unisoc.com [222.66.158.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1820732C94A for ; Wed, 26 Aug 2026 08:11:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=222.66.158.135 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787731887; cv=none; b=uTjWdeQxC75tW202zDY3JXNbr0yuzgwBoFYgrKx3xjNzHwpbHy7koEn2QfaJaBfEBCtyyIb10FBZ/W3I+KE+6j9TopEUAbzoSD2v+6KTjBR8N4oPRr57wFN3PCQ2QrG2nnxUzGHnYTMcJy3nC2fzAnHQrHvvwzqGOsW4pgSNeBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787731887; c=relaxed/simple; bh=+rtlylyTfs5NoT1g460Udz9STPXM0Z2O0EAF8uP7Xqk=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type; b=RzUCxUhv8fou7n7fAyZ+uxlXROuPgBDUSbBj2do86/NwJmc7lWkEsJPnwQ/sep/SjecX/ATE2aJ1YdkJoYW0uRYtdU1Kop8Odas5CZtr4ziPygZKz0PrWQ5GlZaVpR32UUuCXpHGteBBS2ZBopIyG7e8mYVCCUkiG8kHRQfE7mA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=unisoc.com; spf=pass smtp.mailfrom=unisoc.com; dkim=pass (2048-bit key) header.d=unisoc.com header.i=@unisoc.com header.b=EC5u6CFv; arc=none smtp.client-ip=222.66.158.135 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=unisoc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=unisoc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=unisoc.com header.i=@unisoc.com header.b="EC5u6CFv" Received: from SHSQR01.spreadtrum.com (localhost [127.0.0.2] (may be forged)) by SHSQR01.spreadtrum.com with ESMTP id 67Q7q8Jx008753 for ; Wed, 26 Aug 2026 15:52:08 +0800 (+08) (envelope-from zhaoyang.huang@unisoc.com) Received: from dlp.unisoc.com ([10.29.3.86]) by SHSQR01.spreadtrum.com with ESMTPS id 67Q7piCL006362 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 26 Aug 2026 15:51:44 +0800 (+08) (envelope-from zhaoyang.huang@unisoc.com) Received: from SHDLP.spreadtrum.com (BJMBX01.spreadtrum.com [10.0.64.7]) by dlp.unisoc.com (SkyGuard) with ESMTPS id 4hVGzP6wZjz2K1r9x; Wed, 26 Aug 2026 15:50:41 +0800 (CST) Received: from bj03382pcu03.spreadtrum.com (10.0.73.40) by BJMBX01.spreadtrum.com (10.0.64.7) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Wed, 26 Aug 2026 15:51:41 +0800 From: "zhaoyang.huang" To: Andrew Morton , David Hildenbrand , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , , , Zhaoyang Huang , Subject: [PATCH] mm: workingset: judge cross-memcg refaults by generation age Date: Wed, 26 Aug 2026 15:51:33 +0800 Message-ID: <20260826075133.751049-1-zhaoyang.huang@unisoc.com> X-Mailer: git-send-email 2.50.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SHCAS03.spreadtrum.com (10.0.1.207) To BJMBX01.spreadtrum.com (10.0.64.7) X-MAIL: SHSQR01.spreadtrum.com 67Q7piCL006362 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=unisoc.com; s=default; t=1787730717; bh=1ljfxh0QQvke3/JuhIoWXsbYDuhyNNb2x/9KR07BP2M=; h=From:To:Subject:Date; b=EC5u6CFvKVqzykWYyhm1uLick9XrvOidGA+taumg/amu1lxTKY4075bAY/oFlXMxR xd8vEU/rA2vhK3/iolI7WDGlw6f+wanBP2RqtDqookzwFm+IL0CjevzmRpReQlItLZ cRaYr7fDWnbJAXbcKi5GxzToUpEVPq4p87mBkW6QNF8ugI5M718SPf07F+oveUqrnF +r1caHAJjRbi0lbRU9bBpCbDa2iKgvjk6jxjyeq6QZLMep8EIGz/BeYlSYW5YrIm9e r1CXqiPkZf+P1nNiySx7PGMSo9Otue/7td6UFVInF4Ip7LjIw2x8Fq10jz0FEshMSc p06Emn4bgyqsQ== Content-Type: text/plain; charset="utf-8" From: Zhaoyang Huang Unexpected anon folio scan rate is observed in a per-pid topology memcgv2's hierarchy even with swappiness=3D150, which is believed as the concequences of belowing rule which discarded all feedback for the destination cgroup: no WORKINGSET_REFAULT accounting, no activation hint, and no tier or workingset restoration. Shared mappings faulted back into another memcg could not signal recent usefulness to MGLRU in that cgroup, weakening thrashing detection and protection for cross-cgroup file cache and folios that moved between memcgs. if (lruvec !=3D folio_lruvec(folio)) goto unlock; The failure mode is structural: min_seq and max_seq are per-lruvec generation counters with independent sequences, so their numeric gap is meaningless across lruvecs. Store a truncated eviction timestamp (jiffies) in the shadow token next to min_seq, reserving at least 16 bits for min_seq so the same-lruvec sequence test does not wrap at the shorter LRU_GEN_WIDTH interval. Same-lruvec refaults keep the existing sequence-based recency check. For cross-lruvec refaults, compare elapsed time since eviction with the age of the middle generation in the destination lruvec's MAX_NR_GENS window. Refaults inside that window are treated as recent and workingset feedback is attributed to the destination lruvec. This restores refault-driven reclaim hints for shared cache without comparing unrelated generation numbers, while leaving the common same-lruvec path unchanged. With this commit, we can observe the anon scan rate raises under same enviroment. Co-authored-by: Cursor Signed-off-by: Zhaoyang Huang --- mm/workingset.c | 65 ++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 59 insertions(+), 6 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index 07e6836d0502..e55e5a75acb9 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -199,6 +199,28 @@ */ static unsigned int bucket_order[ANON_AND_FILE] __read_mostly; =20 +#ifdef CONFIG_LRU_GEN +/* + * LRU_GEN shadow token layout within the eviction field (low to high): + * LRU_REFS_WIDTH bits: refs + * EVICT_SEQ_WIDTH bits: min_seq + * EVICT_TS_WIDTH bits: eviction time in jiffies + * + * Size the layout for the tighter anon mask so the token fits both file a= nd + * anon shadows. Keep at least 16 bits for min_seq so the same-lruvec rece= ncy + * check does not wrap at the much shorter LRU_GEN_WIDTH interval. + */ +#define EVICTION_BITS (BITS_PER_LONG - EVICTION_SHIFT_ANON) +#define EVICT_PAYLOAD_BITS (EVICTION_BITS - LRU_REFS_WIDTH) +#define EVICT_TS_WIDTH (EVICT_PAYLOAD_BITS > 40 ? 24 : \ + EVICT_PAYLOAD_BITS > 16 ? \ + EVICT_PAYLOAD_BITS - 16 : 0) +#define EVICT_SEQ_WIDTH (EVICT_PAYLOAD_BITS - EVICT_TS_WIDTH) +#define EVICT_TS_PGOFF (LRU_REFS_WIDTH + EVICT_SEQ_WIDTH) +#define EVICT_SEQ_MASK ((1UL << EVICT_SEQ_WIDTH) - 1) +#define EVICT_TS_MASK ((1UL << EVICT_TS_WIDTH) - 1) +#endif + static void *pack_shadow(int memcgid, pg_data_t *pgdat, unsigned long evic= tion, bool workingset, bool file) { @@ -250,13 +272,19 @@ static void *lru_gen_eviction(struct folio *folio) =20 BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH > BITS_PER_LONG - max(EVICTION_SHIFT, EVICTION_SHIFT_ANON)); + BUILD_BUG_ON(EVICT_TS_WIDTH <=3D 0); + BUILD_BUG_ON(EVICT_SEQ_WIDTH < 16); + BUILD_BUG_ON(LRU_REFS_WIDTH + EVICT_SEQ_WIDTH + + EVICT_TS_WIDTH !=3D EVICTION_BITS); =20 rcu_read_lock(); memcg =3D folio_memcg(folio); lruvec =3D mem_cgroup_lruvec(memcg, pgdat); lrugen =3D &lruvec->lrugen; min_seq =3D READ_ONCE(lrugen->min_seq[type]); - token =3D (min_seq << LRU_REFS_WIDTH) | max(refs - 1, 0); + token =3D ((min_seq & EVICT_SEQ_MASK) << LRU_REFS_WIDTH) | + max(refs - 1, 0); + token |=3D (jiffies & EVICT_TS_MASK) << EVICT_TS_PGOFF; =20 hist =3D lru_hist_from_seq(min_seq); atomic_long_add(delta, &lrugen->evicted[hist][type][tier]); @@ -275,18 +303,21 @@ static bool lru_gen_test_recent(void *shadow, struct = lruvec **lruvec, { int memcg_id; unsigned long max_seq; + unsigned long token_seq; struct mem_cgroup *memcg; struct pglist_data *pgdat; =20 + (void)file; + unpack_shadow(shadow, &memcg_id, &pgdat, token, workingset); =20 memcg =3D mem_cgroup_from_private_id(memcg_id); *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); =20 - max_seq =3D READ_ONCE((*lruvec)->lrugen.max_seq); - max_seq &=3D (file ? EVICTION_MASK : EVICTION_MASK_ANON) >> LRU_REFS_WIDT= H; + max_seq =3D READ_ONCE((*lruvec)->lrugen.max_seq) & EVICT_SEQ_MASK; + token_seq =3D (*token >> LRU_REFS_WIDTH) & EVICT_SEQ_MASK; =20 - return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS; + return abs_diff(max_seq, token_seq) < MAX_NR_GENS; } =20 static void lru_gen_refault(struct folio *folio, void *shadow) @@ -295,7 +326,10 @@ static void lru_gen_refault(struct folio *folio, void = *shadow) int hist, tier, refs; bool workingset; unsigned long token; + unsigned long evict_ts, refault_distance; + unsigned long max_seq, middle_seq, birth, window_age; struct lruvec *lruvec; + struct lruvec *dst_lruvec; struct lru_gen_folio *lrugen; int type =3D folio_is_file_lru(folio); int delta =3D folio_nr_pages(folio); @@ -303,8 +337,27 @@ static void lru_gen_refault(struct folio *folio, void = *shadow) rcu_read_lock(); =20 recent =3D lru_gen_test_recent(shadow, &lruvec, &token, &workingset, type= ); - if (lruvec !=3D folio_lruvec(folio)) - goto unlock; + dst_lruvec =3D folio_lruvec(folio); + + if (lruvec !=3D dst_lruvec) { + /* + * min_seq values from different lruvecs are not comparable. + * Judge a cross-memcg refault by whether it happened within + * the age of the middle generation in the destination + * lruvec's MAX_NR_GENS window. + */ + lruvec =3D dst_lruvec; + lrugen =3D &lruvec->lrugen; + max_seq =3D READ_ONCE(lrugen->max_seq); + middle_seq =3D max_seq - MAX_NR_GENS / 2; + birth =3D READ_ONCE(lrugen->timestamps[lru_gen_from_seq(middle_seq)]); + window_age =3D jiffies - birth; + + evict_ts =3D (token >> EVICT_TS_PGOFF) & EVICT_TS_MASK; + refault_distance =3D ((jiffies & EVICT_TS_MASK) - evict_ts) & + EVICT_TS_MASK; + recent =3D refault_distance <=3D window_age; + } =20 mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + type, delta); =20 --=20 2.50.1