From nobody Sat Sep 26 19:33:47 2026 Received: from mta0.migadu.com (out-164.mta0.migadu.com [91.218.175.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71E1F3D3339 for ; Mon, 31 Aug 2026 09:46:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.164 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788169603; cv=none; b=CIsJjPvCW4bOICOI8fcrdHdON6maP+AUxDZhwX0S/ifUDSZ1Z6S1Gl3BOi8RoPBmuvGP4hFpgYmUXQqa33sRICgJDUgHbiL2FxH2q13mGfxgVUM3XKIG7Z1kh4pkzRLcHmygepQQSavMtczOipysEDUracYVryD14YXOaB5UMVc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788169603; c=relaxed/simple; bh=gLq6dTRueazhtZMb/V4BL5tx3YydnlrYd+x5AgErLV0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AMwEtZsVRtkcF9gfHI2LkSnOViw0mvHNNu6bvJ2wRwXneE7tA9tcCwLcy7gmwVfcbAzy8oKDYwbW2GDDO9/BpuYYlbufT0Tzu8QQjUJqH+i1yxOm5NcO+Vdeh1nWDz3P0gQ8XV8FQ6nRtl8JCETMo09hYO+G+phaz+V9BEOA6FI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=NW4vXXiz; arc=none smtp.client-ip=91.218.175.164 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="NW4vXXiz" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=gLq6dTRueazhtZMb/V4BL5tx3YydnlrYd+x5AgErLV0=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788169599; v=1; x=1788774399; b=NW4vXXizt6meWPAXDM+eYSu6b9F/w+WIaLtKLgX5XWAk0osIeecEezgLzw/39BdRZZZFZTNx bQHuWAFtFc2OCiE21LyYQwgvDA1lYW6dzWsA9RsdyLCqK3XK6YjG1GPgfmVr0Eee4vvNNdzBydQ QvAZ5/RU8nvxso7X8LPOgBtQ= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 151bee8d5fdfa34c; Mon, 31 Aug 2026 09:46:39 +0000 X-Mizu-Trace-ID: 151bee8d5fdfa34c X-Migadu-Flow: FLOW_OUT From: "Hui Zhu" To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Qi Zheng , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hui Zhu , stable@vger.kernel.org Subject: [PATCH v2 1/3] mm: workingset: use lruvec_page_state_local() to count lru pages Date: Mon, 31 Aug 2026 17:46:09 +0800 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hui Zhu Commit 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the number of lru pages") switched count_shadow_nodes() to lruvec_lru_size(). With CONFIG_MEMCG enabled, lruvec_lru_size() reads mz->lru_zone_size, which only the classic LRU paths maintain. MGLRU accounts its pages through __update_lru_size(), which skips that array, so with MGLRU on the four evictable LRU lists are always seen as empty. The shadow node budget (pages >> 3) then collapses to slab plus unevictable pages, and the workingset shadow shrinker reclaims eviction tokens almost as fast as they are created, losing thrashing protection. lruvec_page_state_local() reads lruvec_stats->state_local instead, which both classic LRU and MGLRU maintain. Switch back to it. The reparenting race this re-exposes on cgroup v2 is closed by the follow-up patch that redirects dying-memcg stat updates for all hierarchies. Fixes: 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the numb= er of lru pages") Cc: stable@vger.kernel.org Signed-off-by: Hui Zhu Acked-by: Shakeel Butt --- mm/workingset.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index f351798e723a..85a4e14e95d5 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -693,10 +693,9 @@ static unsigned long count_shadow_nodes(struct shrinke= r *shrinker, =20 mem_cgroup_flush_stats_ratelimited(sc->memcg); lruvec =3D mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); - for (pages =3D 0, i =3D 0; i < NR_LRU_LISTS; i++) - pages +=3D lruvec_lru_size(lruvec, i, MAX_NR_ZONES - 1); - + pages +=3D lruvec_page_state_local(lruvec, + NR_LRU_BASE + i); pages +=3D lruvec_page_state_local( lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT; pages +=3D lruvec_page_state_local( --=20 2.53.0 From nobody Sat Sep 26 19:33:47 2026 Received: from mta1.migadu.com (out-224.mta1.migadu.com [95.215.58.224]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCAF93E0091 for ; Mon, 31 Aug 2026 09:46:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.224 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788169606; cv=none; b=g9eQBXDCH4Kmw0NVPuGg27CTU6U+xP4ylY/3Sx4NWj5EIgvLE7r5nG0yN6LsKcZ5/eAtqJDVBhzqc2284+ZE+mayD8AhfhDv36o+kzPPAcFct/6h9RnnV0uLBj7cMyGFPtkMQ/dkik6iGZvU39pNz8RTI3RqVxTW5QAwMLXu6xk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788169606; c=relaxed/simple; bh=3A3/xJ8l9zjo+u20t95UB+aIofHnDSAgrY2cv9uuTZo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f8usKtE5WTlj2T4eNxHyiklxvpdY7bvKWossWOMEt3ATJH+Vo7Kh1Ftbjrrajry3s+cWknNI/rW+qv2aoNMBlw2UwgtjeNCifjxNySIKk6+OL/zJSUGu+pcySEiQcr1RI8sIgqQIvskzblcXRPHi5b3q9XmEbJi8/b6PbAgtrhE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=hei3gk4X; arc=none smtp.client-ip=95.215.58.224 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="hei3gk4X" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=3A3/xJ8l9zjo+u20t95UB+aIofHnDSAgrY2cv9uuTZo=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788169602; v=1; x=1788774402; b=hei3gk4XP3S3A5xIwixknjToXXhEqrRLO/tjzC7hWzHByD9c60pl+aLxrjD+a4t6YUrlNIaG Lx9laineMOcSIgipSNkrXapT+L3IAn47z/Nzo/RpDeqCtV3cEpHr80CT6sGSO63lYWeYr+RSbuX TxVOy+whAiStNqnGwwWDyL+g= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 7fd054419e97544c; Mon, 31 Aug 2026 09:46:42 +0000 X-Mizu-Trace-ID: 7fd054419e97544c X-Migadu-Flow: FLOW_OUT From: "Hui Zhu" To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Qi Zheng , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hui Zhu Subject: [PATCH v2 2/3] mm: memcg: redirect stats updates of dying memcgs for all hierarchies Date: Mon, 31 Aug 2026 17:46:10 +0800 Message-ID: <0214c0228695db474b86d3b4778b681a6c47a6e2.1788169145.git.zhuhui@kylinos.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hui Zhu get_non_dying_memcg_start() redirects the stat updates of a dying memcg to its closest non-dying ancestor, but only on cgroup v1; on cgroup v2 the stats keep being accounted to the dying memcg itself. The previous patch restored lruvec_page_state_local() in count_shadow_nodes(), which reads those state_locals on cgroup v2 too, so apply the redirection to all hierarchies. Offlining is rare, so the added cost on the stat update fast path is limited to an rcu_read_lock() and a css_is_dying() check; the upward walk happens only while a memcg is dying. Signed-off-by: Hui Zhu Acked-by: Shakeel Butt --- mm/memcontrol.c | 30 +++++------------------------- 1 file changed, 5 insertions(+), 25 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8319ad8c5c23..b3d1ac3fe0aa 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -805,20 +805,14 @@ static long memcg_state_val_in_pages(int idx, long va= l) return val < 0 ? -res : res; } =20 -#ifdef CONFIG_MEMCG_V1 /* - * Used in mod_memcg_state() and mod_memcg_lruvec_state() to avoid race wi= th - * reparenting of non-hierarchical state_locals. + * Used in mod_memcg_state() and mod_memcg_lruvec_state() to avoid race + * with reparenting of non-hierarchical state_locals. Offlining a + * memcg is rare, so do the redirection for all cgroup hierarchies. */ -static inline struct mem_cgroup *get_non_dying_memcg_start(struct mem_cgro= up *memcg, - bool *rcu_locked) +static inline struct mem_cgroup * +get_non_dying_memcg_start(struct mem_cgroup *memcg, bool *rcu_locked) { - /* Rebinding can cause this value to be changed at runtime */ - if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) { - *rcu_locked =3D false; - return memcg; - } - rcu_read_lock(); *rcu_locked =3D true; =20 @@ -830,22 +824,8 @@ static inline struct mem_cgroup *get_non_dying_memcg_s= tart(struct mem_cgroup *me =20 static inline void get_non_dying_memcg_end(bool rcu_locked) { - if (!rcu_locked) - return; - rcu_read_unlock(); } -#else -static inline struct mem_cgroup *get_non_dying_memcg_start(struct mem_cgro= up *memcg, - bool *rcu_locked) -{ - return memcg; -} - -static inline void get_non_dying_memcg_end(bool rcu_locked) -{ -} -#endif =20 static void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, long val) --=20 2.53.0 From nobody Sat Sep 26 19:33:47 2026 Received: from mta1.migadu.com (out-229.mta1.migadu.com [95.215.58.229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 073C33E0C47 for ; Mon, 31 Aug 2026 09:46:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.229 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788169610; cv=none; b=DBaTPHTDLY3+4clgLi4QVRCgtoQhS9h1QylDRMTHTWzStfsVo6J39+U4yeb31IggdFUYXfiS8imPOjizY5sDyXEglnHbp18t2wz/veNget9Bbq9t2KMcBl5iokj3lQRC644+Pd0sKjsL2KGHOo47MAH6j4vXmFVwk9WM3GIdjS8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788169610; c=relaxed/simple; bh=6jhXtJO17DBB7qI18ikslI33Zv8zYlXSXdj+9tIORRU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tCbO3lOG6VWP1R2SNWWHUqjZENW2rOuzpOqo4Qj8+yCA0oqPOTWmP6B1MkwsyN0p8gyHDDLEmN6woLRetgo1uja5DoGSAuB/pwOkvCkp9LvQxzQYIZDig/Soqzjd1TZXf8oFgq+W/u3LOoXAn/wAjTYTnP5A202cXShJClPSD8I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=R9okeS85; arc=none smtp.client-ip=95.215.58.229 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="R9okeS85" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=6jhXtJO17DBB7qI18ikslI33Zv8zYlXSXdj+9tIORRU=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788169606; v=1; x=1788774406; b=R9okeS85NqmAhuuAI7UUSUA+gI3T5d+ERJZYJiOBuYL1RLp2wbwHaDrtH9fIzOcPPCJEwwUC Bq3X5qSfk2XzEJTjavH521qet+xaV7O9xvZirr1R7J/vof1BjLxK1T2bH8n5vD3/qhR/d7n1/jm 6ogAeuxtWZVh6jibxEKal1WQ= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 8bffe5621f7da27d; Mon, 31 Aug 2026 09:46:45 +0000 X-Mizu-Trace-ID: 8bffe5621f7da27d X-Migadu-Flow: FLOW_OUT From: "Hui Zhu" To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , David Hildenbrand , Qi Zheng , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hui Zhu Subject: [PATCH v2 3/3] mm: memcg: skip the RCU lock when the memcg is not dying Date: Mon, 31 Aug 2026 17:46:11 +0800 Message-ID: <3fc10f32131097bb57efaec72d049d635b227cf1.1788169145.git.zhuhui@kylinos.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hui Zhu get_non_dying_memcg_start() takes rcu_read_lock() on every stat update, but the lock only protects the upward walk to a non-dying ancestor, which happens solely while a memcg is being offlined. The dying check itself reads the CSS_DYING flag of a memcg the caller already holds a reference to, so it is safe without the lock. Check memcg_is_dying() first and return immediately when the memcg is alive, taking the RCU lock only on the rare dying path. On an anon fault/charge churn workload in a memcg this recovers the ~0.6% overhead added by the previous patch (4368077 vs 4343159 pages/s before, back to ~4377000 pages/s after). Signed-off-by: Hui Zhu Acked-by: Shakeel Butt --- mm/memcontrol.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b3d1ac3fe0aa..f454d02746e9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -813,6 +813,17 @@ static long memcg_state_val_in_pages(int idx, long val) static inline struct mem_cgroup * get_non_dying_memcg_start(struct mem_cgroup *memcg, bool *rcu_locked) { + /* + * Fast path: the caller holds a reference to @memcg, so reading + * its CSS_DYING flag without the RCU lock is safe. The RCU lock + * is only needed to walk up to a non-dying ancestor, which + * happens only while a memcg is actually being offlined. + */ + if (!memcg_is_dying(memcg)) { + *rcu_locked =3D false; + return memcg; + } + rcu_read_lock(); *rcu_locked =3D true; =20 @@ -824,6 +835,9 @@ get_non_dying_memcg_start(struct mem_cgroup *memcg, boo= l *rcu_locked) =20 static inline void get_non_dying_memcg_end(bool rcu_locked) { + if (!rcu_locked) + return; + rcu_read_unlock(); } =20 --=20 2.53.0