From nobody Fri Sep 25 22:18:52 2026 Received: from mta1.migadu.com (out-72.mta1.migadu.com [95.215.58.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 388AB3559E1 for ; Tue, 8 Sep 2026 03:41:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838920; cv=none; b=m2jMWthKB0Z75UIIsO/hJHV99HNAG/UlNMrzWArwHJJ/Rq44wL67eomJERjHS1QpmJclZ1h1y5AZswRuyClPlHfxQ1KvqaerJ5Q60ybJi24seu3E4+z//TKtxgpL6qKMlnF4saiTS8ec3pBK5bldnEe8MZ1pp+Skm6oyDSOGDgw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838920; c=relaxed/simple; bh=48OIeErQYrhkpbuOIqiS9L91aRSY0G25gCPRWoMJaKs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Os2a1gx+wIjXmJq5o82LCIRF/AvuNAItwnOtK357XlKOL7JMOjY2hOZL736mPp7iv81hM8ZkCb2eDOabJ1NWAqHq+qNpE7CaZHYh9WSjJjHl+xEkI62oiwRWIUh90lIgaigxgkorTqzZa12DhbNdQor3GKWFCQ9U31RU7Gnk5Xw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=NLe3FZ7+; arc=none smtp.client-ip=95.215.58.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="NLe3FZ7+" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=48OIeErQYrhkpbuOIqiS9L91aRSY0G25gCPRWoMJaKs=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788838895; v=1; x=1789443695; b=NLe3FZ7+kyHAfxxejboKegFHJddLVkrGkMVpODeH+9bijIBT0h7NKm3az5F08um5Zyg23BBV WkDlK+1iJVk9+fFrAoTNhUaZmrTgq/f0GFysMLVdZAKaTkBPzFb92JV8+OokKf91e9/EpTN9j4M 8vK0VsoG+ESuw8C4vr4vlm2Y= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id bc731f78d792f31d; Tue, 08 Sep 2026 03:41:34 +0000 X-Mizu-Trace-ID: bc731f78d792f31d X-Migadu-Flow: FLOW_OUT From: Hui Zhu To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Qi Zheng , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hui Zhu , stable@vger.kernel.org Subject: [PATCH v4 1/4] mm: memcg: redirect stats updates of dying memcgs for all hierarchies Date: Tue, 8 Sep 2026 11:41:11 +0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hui Zhu get_non_dying_memcg_start() redirects the stat updates of a dying memcg to its closest non-dying ancestor, but only on cgroup v1; on cgroup v2 the stats keep being accounted to the dying memcg itself. A later patch in this series restores lruvec_page_state_local() in count_shadow_nodes() to fix the broken workingset shadow node budget under MGLRU. count_shadow_nodes() is the only reader of those non-hierarchical state_locals on cgroup v2: when a memcg is offlined, its pages are reparented to the ancestor but their stat updates keep being accounted to the dying memcg, so count_shadow_nodes() computes a wrong shadow node budget and workingset thrashing protection is lost. This is user visible as premature reclaim of hot page cache and degraded performance under memory pressure. Apply the redirection to all hierarchies to fix this. Offlining is rare, so the added cost on the stat update fast path is limited to an rcu_read_lock() and a css_is_dying() check; the upward walk happens only while a memcg is dying. Fixes: 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the numb= er of lru pages") Cc: stable@vger.kernel.org Signed-off-by: Hui Zhu Acked-by: Shakeel Butt --- mm/memcontrol.c | 30 +++++------------------------- 1 file changed, 5 insertions(+), 25 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1271d390b617e..ace9fe2b46084 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -845,20 +845,14 @@ static long memcg_state_val_in_pages(int idx, long va= l) return val < 0 ? -res : res; } =20 -#ifdef CONFIG_MEMCG_V1 /* - * Used in mod_memcg_state() and mod_memcg_lruvec_state() to avoid race wi= th - * reparenting of non-hierarchical state_locals. + * Used in mod_memcg_state() and mod_memcg_lruvec_state() to avoid race + * with reparenting of non-hierarchical state_locals. Offlining a + * memcg is rare, so do the redirection for all cgroup hierarchies. */ -static inline struct mem_cgroup *get_non_dying_memcg_start(struct mem_cgro= up *memcg, - bool *rcu_locked) +static inline struct mem_cgroup * +get_non_dying_memcg_start(struct mem_cgroup *memcg, bool *rcu_locked) { - /* Rebinding can cause this value to be changed at runtime */ - if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) { - *rcu_locked =3D false; - return memcg; - } - rcu_read_lock(); *rcu_locked =3D true; =20 @@ -870,22 +864,8 @@ static inline struct mem_cgroup *get_non_dying_memcg_s= tart(struct mem_cgroup *me =20 static inline void get_non_dying_memcg_end(bool rcu_locked) { - if (!rcu_locked) - return; - rcu_read_unlock(); } -#else -static inline struct mem_cgroup *get_non_dying_memcg_start(struct mem_cgro= up *memcg, - bool *rcu_locked) -{ - return memcg; -} - -static inline void get_non_dying_memcg_end(bool rcu_locked) -{ -} -#endif =20 static void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, long val) --=20 2.43.0 From nobody Fri Sep 25 22:18:52 2026 Received: from mta0.migadu.com (out-57.mta0.migadu.com [91.218.175.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8F9E34E741 for ; Tue, 8 Sep 2026 03:41:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.57 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838920; cv=none; b=T2oHM/SXKMGVMHxghSVKX5R20La4HXG9eR2oFZ0c51VU2r2QfBhQX6T0g2fhMQA7HV0mOOtA31SPmqpVcmSYhMttOOqVVaPiwDjSoXJ0i2xgHyG+IcV0gYtiXZR5Ag6/tjDbzQgupJe5r76XCXZWs+RYmUq46romadXnfz+VeQo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838920; c=relaxed/simple; bh=lG7+VxQE9zUCiUgD2bZJA+1KtHaOwgs94O/ucBNwUz8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XDfqn9LeTU5nOzei6+bb6crXnGwnXgWRzX8qhH3BMAcb7hLUqUmvhpHRbIscKtsJhGhgHkQEnWCYiG5CJR+UMuoOFmvblkTIVBuRNo5sgBpMdwq/AHENwxWgCSXs5RqwGUkwnd1FzOtmzN7tG0h4SnJ2lBrqcI9obWxVEFPM1QA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=PAEnLWfu; arc=none smtp.client-ip=91.218.175.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="PAEnLWfu" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=lG7+VxQE9zUCiUgD2bZJA+1KtHaOwgs94O/ucBNwUz8=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788838900; v=1; x=1789443700; b=PAEnLWfuOG0+HCa7sS2d8nGMQfXu+5fj5SWvYq7fhSADlz4g+Wj6A0YnulKz3+Zp6uKyr0ez qsZZ4QFSJrVxl5btS1pna884qWw0ctfW75u5A4kOv6kKg/7fW5Tb8KGU4lRYmSUvQXwu4uOFFST GgOr1T18xfNOp+9ZA83GGogk= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id c4918b52a91f44bb; Tue, 08 Sep 2026 03:41:39 +0000 X-Mizu-Trace-ID: c4918b52a91f44bb X-Migadu-Flow: FLOW_OUT From: Hui Zhu To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Qi Zheng , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hui Zhu , stable@vger.kernel.org Subject: [PATCH v4 2/4] mm: workingset: use lruvec_page_state_local() to count lru pages Date: Tue, 8 Sep 2026 11:41:12 +0800 Message-ID: <2ed42f96aca124856ea30f774afb55cbe6d8ba58.1788837625.git.zhuhui@kylinos.cn> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hui Zhu Commit 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the number of lru pages") switched count_shadow_nodes() to lruvec_lru_size(). With CONFIG_MEMCG enabled, lruvec_lru_size() reads mz->lru_zone_size, which only the classic LRU paths maintain. MGLRU accounts its pages through __update_lru_size(), which skips that array, so with MGLRU on the four evictable LRU lists are always seen as empty. The shadow node budget (pages >> 3) then collapses to slab plus unevictable pages, and the workingset shadow shrinker reclaims eviction tokens almost as fast as they are created, losing thrashing protection. lruvec_page_state_local() reads lruvec_stats->state_local instead, which both classic LRU and MGLRU maintain. Switch back to it. The reparenting race this re-exposes on cgroup v2 is closed by the preceding patch that redirects dying-memcg stat updates for all hierarchies. Fixes: 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the numb= er of lru pages") Cc: stable@vger.kernel.org Signed-off-by: Hui Zhu Acked-by: Shakeel Butt --- mm/workingset.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index 7ac2b88c80ae5..8412f4840ae35 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -688,10 +688,9 @@ static unsigned long count_shadow_nodes(struct shrinke= r *shrinker, =20 mem_cgroup_flush_stats_ratelimited(sc->memcg); lruvec =3D mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); - for (pages =3D 0, i =3D 0; i < NR_LRU_LISTS; i++) - pages +=3D lruvec_lru_size(lruvec, i, MAX_NR_ZONES - 1); - + pages +=3D lruvec_page_state_local(lruvec, + NR_LRU_BASE + i); pages +=3D lruvec_page_state_local( lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT; pages +=3D lruvec_page_state_local( --=20 2.43.0 From nobody Fri Sep 25 22:18:52 2026 Received: from mta1.migadu.com (out-80.mta1.migadu.com [95.215.58.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 819F1355F53 for ; Tue, 8 Sep 2026 03:41:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.80 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838926; cv=none; b=NK3Rh5VWtjJO12n9fTggNYkinc/e6yKQm72YTXT3HH0wyapFyDg1fuqji0Fpn/5FXMoF45BGNb2XjAvhWc8GPODVWHvoxcVVN3fpU8MU+nc1MY71XhlWwPlldAcK0FI8ZpbV+Wb6B+r0QY9EcWAUTm1MAFlmkB1+kZsvqkegK0g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838926; c=relaxed/simple; bh=fjfBBGmsJJ56y0JVyIG84X0Pp//85lfIIw3IuGHXEuw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lFF8dA1hRtKvlqmiLzU7EzbgfNXHjj0l2NjCgUZZ6t3ej9tawBSI5kQgKxEHHNyWRQfskyU1eNxT/XJGcqgmMPmA8gjb7oQSBHNUuhqe4h/C+1YdtwR9mIV86OTESKkjxbRZfjd7gB1gwkehobMpS8YPka9dAh869HW5NTWZ5uI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=TV/1aYmT; arc=none smtp.client-ip=95.215.58.80 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="TV/1aYmT" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=fjfBBGmsJJ56y0JVyIG84X0Pp//85lfIIw3IuGHXEuw=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788838905; v=1; x=1789443705; b=TV/1aYmTB6zEuD+B0PHKtv1u1K1lnMpW6hxK9r8KoQP47BTsSpXlj2wqw9OdbBdeosF/HZdE 2Cu8FFePzLLpExSt6G6D82Epdu6UzLk6PfbIemxvKwvJ3A/RlCcmJamleL8jPjJsAIlZrEFg+BT Ww8fDQfzKzNL4msglA59OX3Y= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id 20f6f116f302dedc; Tue, 08 Sep 2026 03:41:45 +0000 X-Mizu-Trace-ID: 20f6f116f302dedc X-Migadu-Flow: FLOW_OUT From: Hui Zhu To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Qi Zheng , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hui Zhu , stable@vger.kernel.org Subject: [PATCH v4 3/4] mm: memcg: skip the RCU lock when the memcg is not dying Date: Tue, 8 Sep 2026 11:41:13 +0800 Message-ID: <9ffdbdfc96312e3e13cb8f056bfe26649492d949.1788837625.git.zhuhui@kylinos.cn> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hui Zhu get_non_dying_memcg_start() takes rcu_read_lock() on every stat update, but the lock only protects the upward walk to a non-dying ancestor, which happens solely while a memcg is being offlined. The dying check itself reads the CSS_DYING flag of a memcg the caller already holds a reference to, so it is safe without the lock. Check memcg_is_dying() first and return immediately when the memcg is alive, taking the RCU lock only on the rare dying path. On an anon fault/charge churn workload in a memcg this recovers the ~0.6% overhead added by the previous patch (4368077 vs 4343159 pages/s before, back to ~4377000 pages/s after). Fixes: 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the numb= er of lru pages") Cc: stable@vger.kernel.org Signed-off-by: Hui Zhu Acked-by: Shakeel Butt --- mm/memcontrol.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ace9fe2b46084..9995f3d2aae1b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -853,6 +853,17 @@ static long memcg_state_val_in_pages(int idx, long val) static inline struct mem_cgroup * get_non_dying_memcg_start(struct mem_cgroup *memcg, bool *rcu_locked) { + /* + * Fast path: the caller holds a reference to @memcg, so reading + * its CSS_DYING flag without the RCU lock is safe. The RCU lock + * is only needed to walk up to a non-dying ancestor, which + * happens only while a memcg is actually being offlined. + */ + if (!memcg_is_dying(memcg)) { + *rcu_locked =3D false; + return memcg; + } + rcu_read_lock(); *rcu_locked =3D true; =20 @@ -864,6 +875,9 @@ get_non_dying_memcg_start(struct mem_cgroup *memcg, boo= l *rcu_locked) =20 static inline void get_non_dying_memcg_end(bool rcu_locked) { + if (!rcu_locked) + return; + rcu_read_unlock(); } =20 --=20 2.43.0 From nobody Fri Sep 25 22:18:52 2026 Received: from mta1.migadu.com (out-87.mta1.migadu.com [95.215.58.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 150F1328B5E for ; Tue, 8 Sep 2026 03:42:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.87 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838932; cv=none; b=Un3qQfRqlFA+KoNFc2kma8J2Vi+WduxAQ8c7j6BQR1EySNIxHLhxLHbhGo+75KNMBJuHyS6TbbWu+iQpIGdE1kVffO0EdIurQMlsPWKdvxWwgaL7iQmp1zh0lJOqVgDZv0MiGK7CyN3WSJfde7bJHIPxCMc4cOMabeZCnHTqEKk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788838932; c=relaxed/simple; bh=MlIZcCXSxTICdrlqvuuIMch3l/hSvENFUEtEORDa6Hc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BXuS3FiQTR2ERsfqx1Dq9I0tlFWcEj85W00WJiuWL6htJ0PChO/m+lEdUvkhedd73OR87JN4XPbLe4KK30uFsPhpXUMWKQWHZ5H8LVjXg3pstl+IZlnb4ICT/R6zJ5pjEG9FIzhIeXcrUarBVYGsmnmQadk8fKuks4jwAquEGVg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=MwBqBnL3; arc=none smtp.client-ip=95.215.58.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="MwBqBnL3" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=MlIZcCXSxTICdrlqvuuIMch3l/hSvENFUEtEORDa6Hc=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788838914; v=1; x=1789443714; b=MwBqBnL3ytwpnPgdbMGRuZcfvxrd0kp14PW/enIyEnE0bK5M8xEx/+P3mVSugRjiNfOy38s8 /6DxJyQem8tlDIrbJWpX2dW94lry9A6VDFf1TfHysQ6TuB8JU6dsaxxU8CgXdGu7tF7JanEscpY oT/1B+2y38nwtcmYaBU8Q/hs= X-Envelope-To: linux-kernel@vger.kernel.org Received: by mta12.migadu.com with ESMTPS id 0e9898f317fc1207; Tue, 08 Sep 2026 03:41:53 +0000 X-Mizu-Trace-ID: 0e9898f317fc1207 X-Migadu-Flow: FLOW_OUT From: Hui Zhu To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Qi Zheng , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hui Zhu , stable@vger.kernel.org Subject: [PATCH v4 4/4] mm: memcg: reparent non-hierarchical lruvec stats on cgroup v2 Date: Tue, 8 Sep 2026 11:41:14 +0800 Message-ID: <4a7a64eed2b145ad535fedaea3624f8310c29d5b.1788837625.git.zhuhui@kylinos.cn> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hui Zhu On cgroup v2, reparent_state_local() returns early and never moves the dying memcg's non-hierarchical state_local base counts to its parent. Meanwhile memcg_reparent_objcgs() rewrites objcg->memcg to the parent, so when the reparented folios are freed later, the negative deltas land on the parent's lruvec. The parent therefore receives the uncharges without ever having received the matching charges, and its state_local (NR_LRU_BASE + lru, MEMCG_SOCK, NR_SLAB_RECLAIMABLE_B, NR_SLAB_UNRECLAIMABLE_B) permanently underflows. Since lruvec_page_state_local() clamps negative values to zero, the underflow masks the parent's own legitimate pages. count_shadow_nodes() is the only reader of these non-hierarchical state_locals on cgroup v2, so the underflow directly distorts the workingset shadow node budget. Fix this by reparenting the lruvec state_locals on cgroup v2 as well, mirroring what cgroup v1 already does. Only the lruvec stats consumed by count_shadow_nodes() are moved; the memcg-level stats are left alone because on v2 they are exposed through the rstat hierarchical tree and are not read from state_local. Fixes: 8285917d6f38 ("mm: memcontrol: prepare for reparenting non-hierarchi= cal stats") Cc: stable@vger.kernel.org Signed-off-by: Hui Zhu --- mm/memcontrol-v1.h | 5 +++-- mm/memcontrol.c | 42 ++++++++++++++++++++++++++++-------------- 2 files changed, 31 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol-v1.h b/mm/memcontrol-v1.h index 1e394269c613d..0578b7076764d 100644 --- a/mm/memcontrol-v1.h +++ b/mm/memcontrol-v1.h @@ -25,6 +25,9 @@ int memory_stat_show(struct seq_file *m, void *v); struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *mem= cg, unsigned int n); =20 +void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, + struct mem_cgroup *parent, int idx); + /* Cgroup v1-specific declarations */ #ifdef CONFIG_MEMCG_V1 =20 @@ -73,8 +76,6 @@ void reparent_memcg1_lruvec_state_local(struct mem_cgroup= *memcg, struct mem_cgr =20 void reparent_memcg_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent, int idx); -void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, - struct mem_cgroup *parent, int idx); =20 void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages); static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9995f3d2aae1b..f13030f75fa54 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -232,14 +232,29 @@ static inline struct obj_cgroup *__memcg_reparent_obj= cgs(struct mem_cgroup *memc return objcg; } =20 -#ifdef CONFIG_MEMCG_V1 static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force); =20 -static inline void reparent_state_local(struct mem_cgroup *memcg, struct m= em_cgroup *parent) +/* + * Reparent the non-hierarchical lruvec stats that count_shadow_nodes() re= ads + * to approximate the shadow node budget. They are not exposed to userspa= ce + * on cgroup v2, but they must follow the reparented folios; otherwise the + * ancestor would only receive the negative deltas when the folios are fre= ed + * without ever having received the positive base, and its local stats wou= ld + * permanently underflow. + */ +static void reparent_v2_lruvec_state_local(struct mem_cgroup *memcg, struc= t mem_cgroup *parent) { - if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return; + int i; + + for (i =3D 0; i < NR_LRU_LISTS; i++) + reparent_memcg_lruvec_state_local(memcg, parent, NR_LRU_BASE + i); + + reparent_memcg_lruvec_state_local(memcg, parent, NR_SLAB_RECLAIMABLE_B); + reparent_memcg_lruvec_state_local(memcg, parent, NR_SLAB_UNRECLAIMABLE_B); +} =20 +static inline void reparent_state_local(struct mem_cgroup *memcg, struct m= em_cgroup *parent) +{ /* * Reparent stats exposed non-hierarchically. Flush @memcg's stats first * to read its stats accurately , and conservatively flush @parent's @@ -248,17 +263,18 @@ static inline void reparent_state_local(struct mem_cg= roup *memcg, struct mem_cgr */ __mem_cgroup_flush_stats(memcg, true); =20 - /* The following counts are all non-hierarchical and need to be reparente= d. */ - reparent_memcg1_state_local(memcg, parent); - reparent_memcg1_lruvec_state_local(memcg, parent); + if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) { + reparent_v2_lruvec_state_local(memcg, parent); + } else { +#ifdef CONFIG_MEMCG_V1 + /* The following counts are all non-hierarchical and need to be reparent= ed. */ + reparent_memcg1_state_local(memcg, parent); + reparent_memcg1_lruvec_state_local(memcg, parent); +#endif + } =20 __mem_cgroup_flush_stats(parent, true); } -#else -static inline void reparent_state_local(struct mem_cgroup *memcg, struct m= em_cgroup *parent) -{ -} -#endif =20 static inline void reparent_locks(struct mem_cgroup *memcg, struct mem_cgr= oup *parent, int nid) { @@ -570,7 +586,6 @@ unsigned long lruvec_page_state_local(struct lruvec *lr= uvec, return x; } =20 -#ifdef CONFIG_MEMCG_V1 static void __mod_memcg_lruvec_state(struct mem_cgroup_per_node *pn, enum node_stat_item idx, long val); =20 @@ -592,7 +607,6 @@ void reparent_memcg_lruvec_state_local(struct mem_cgrou= p *memcg, __mod_memcg_lruvec_state(parent_pn, idx, value); } } -#endif =20 /* Subset of vm_event_item to report for memcg event stats */ static const unsigned int memcg_vm_event_stat[] =3D { --=20 2.43.0