cgroup: rstat: relax NMI guard after switch to try_cmpxchg

[PATCH] cgroup: rstat: relax NMI guard after switch to try_cmpxchg

Posted by Cunlong Li 4 days, 20 hours ago

Commit 36df6e3dbd7e ("cgroup: make css_rstat_updated nmi safe") used
this_cpu_cmpxchg() for the lockless insertion, and therefore required
both ARCH_HAVE_NMI_SAFE_CMPXCHG and ARCH_HAS_NMI_SAFE_THIS_CPU_OPS in
the NMI guard: on archs without the latter, this_cpu_cmpxchg() falls
back to "local_irq_save() + plain cmpxchg", and local_irq_save()
cannot mask NMIs.

Commit 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in
css_rstat_updated") later replaced this_cpu_cmpxchg() with plain
try_cmpxchg() to fix cross-CPU lockless-list corruption, but left the
NMI guard untouched.  After that switch, css_rstat_updated() no longer
performs any this_cpu_*() RMW operations and only relies on the arch
having NMI-safe cmpxchg, so ARCH_HAS_NMI_SAFE_THIS_CPU_OPS is no
longer required in the guard.

Relax the guard accordingly so that archs which have HAVE_NMI and
ARCH_HAVE_NMI_SAFE_CMPXCHG but not ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
(e.g. sparc, powerpc on PPC64/BOOK3S) can benefit from the existing
CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC path.  Without this, the css
is never queued in NMI on those archs, and the atomics staged by
account_{slab,kmem}_nmi_safe() are not drained by flush_nmi_stats().

Fixes: 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in css_rstat_updated")
Signed-off-by: Cunlong Li <shenxiaogll@gmail.com>
---
 kernel/cgroup/rstat.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c
index 150e5871e66f..fa46611098a5 100644
--- a/kernel/cgroup/rstat.c
+++ b/kernel/cgroup/rstat.c
@@ -83,11 +83,10 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
 	lockdep_assert_preemption_disabled();
 
 	/*
-	 * For archs withnot nmi safe cmpxchg or percpu ops support, ignore
-	 * the requests from nmi context.
+	 * The lockless insertion below relies on NMI-safe cmpxchg;
+	 * bail out in NMI on archs that don't provide it.
 	 */
-	if ((!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) ||
-	     !IS_ENABLED(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS)) && in_nmi())
+	if (!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) && in_nmi())
 		return;
 
 	rstatc = css_rstat_cpu(css, cpu);

---
base-commit: 27fa82620cbaa89a7fc11ac3057701d598813e87
change-id: 20260520-nmi-0493a1569716

Best regards,
-- 
Cunlong Li <shenxiaogll@gmail.com>

Re: [PATCH] cgroup: rstat: relax NMI guard after switch to try_cmpxchg

Posted by Shakeel Butt 4 days, 1 hour ago

On Wed, May 20, 2026 at 11:30:54AM +0800, Cunlong Li wrote:
> Commit 36df6e3dbd7e ("cgroup: make css_rstat_updated nmi safe") used
> this_cpu_cmpxchg() for the lockless insertion, and therefore required
> both ARCH_HAVE_NMI_SAFE_CMPXCHG and ARCH_HAS_NMI_SAFE_THIS_CPU_OPS in
> the NMI guard: on archs without the latter, this_cpu_cmpxchg() falls
> back to "local_irq_save() + plain cmpxchg", and local_irq_save()
> cannot mask NMIs.
> 
> Commit 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in
> css_rstat_updated") later replaced this_cpu_cmpxchg() with plain
> try_cmpxchg() to fix cross-CPU lockless-list corruption, but left the
> NMI guard untouched.  After that switch, css_rstat_updated() no longer
> performs any this_cpu_*() RMW operations and only relies on the arch
> having NMI-safe cmpxchg, so ARCH_HAS_NMI_SAFE_THIS_CPU_OPS is no
> longer required in the guard.
> 
> Relax the guard accordingly so that archs which have HAVE_NMI and
> ARCH_HAVE_NMI_SAFE_CMPXCHG but not ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
> (e.g. sparc, powerpc on PPC64/BOOK3S) can benefit from the existing
> CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC path.  Without this, the css
> is never queued in NMI on those archs, and the atomics staged by
> account_{slab,kmem}_nmi_safe() are not drained by flush_nmi_stats().
> 
> Fixes: 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in css_rstat_updated")
> Signed-off-by: Cunlong Li <shenxiaogll@gmail.com>

Looks fine but how did you find this? AI?

Acked-by: Shakeel Butt <shakeel.butt@linux.dev>

Re: [PATCH] cgroup: rstat: relax NMI guard after switch to try_cmpxchg

Posted by Cunlong Li 3 days, 21 hours ago

On Wed, May 20, 2026 at 03:41:02PM -0700, Shakeel Butt wrote:
> On Wed, May 20, 2026 at 11:30:54AM +0800, Cunlong Li wrote:
> > Commit 36df6e3dbd7e ("cgroup: make css_rstat_updated nmi safe") used
> > this_cpu_cmpxchg() for the lockless insertion, and therefore required
> > both ARCH_HAVE_NMI_SAFE_CMPXCHG and ARCH_HAS_NMI_SAFE_THIS_CPU_OPS in
> > the NMI guard: on archs without the latter, this_cpu_cmpxchg() falls
> > back to "local_irq_save() + plain cmpxchg", and local_irq_save()
> > cannot mask NMIs.
> > 
> > Commit 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in
> > css_rstat_updated") later replaced this_cpu_cmpxchg() with plain
> > try_cmpxchg() to fix cross-CPU lockless-list corruption, but left the
> > NMI guard untouched.  After that switch, css_rstat_updated() no longer
> > performs any this_cpu_*() RMW operations and only relies on the arch
> > having NMI-safe cmpxchg, so ARCH_HAS_NMI_SAFE_THIS_CPU_OPS is no
> > longer required in the guard.
> > 
> > Relax the guard accordingly so that archs which have HAVE_NMI and
> > ARCH_HAVE_NMI_SAFE_CMPXCHG but not ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
> > (e.g. sparc, powerpc on PPC64/BOOK3S) can benefit from the existing
> > CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC path.  Without this, the css
> > is never queued in NMI on those archs, and the atomics staged by
> > account_{slab,kmem}_nmi_safe() are not drained by flush_nmi_stats().
> > 
> > Fixes: 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in css_rstat_updated")
> > Signed-off-by: Cunlong Li <shenxiaogll@gmail.com>
> 
> Looks fine but how did you find this? AI?
> 
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> 

Yes, AI-assisted.

I'm new to kernel development and was studying the memcg code.
When I came across the guard in css_rstat_updated():

	if ((!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) ||
	     !IS_ENABLED(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS)) && in_nmi())
		return;

I asked Opus what those two CONFIGs mean and why the function
returns when in_nmi(). It suggested ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
may no longer be required after the switch from this_cpu_cmpxchg()
to try_cmpxchg(). I then went through the related commit history
and confirmed the analysis.

Thanks for the ack!

Re: [PATCH] cgroup: rstat: relax NMI guard after switch to try_cmpxchg

Posted by Tejun Heo 4 days, 4 hours ago

Hello,

On Wed, May 20, 2026 at 11:30:54AM +0800, Cunlong Li wrote:
> [PATCH] cgroup: rstat: relax NMI guard after switch to try_cmpxchg

Applied to cgroup/for-7.1-fixes.

Thanks.

--
tejun