From nobody Sat Oct 11 00:21:51 2025 Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4C9622E406 for ; Wed, 11 Jun 2025 22:16:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680167; cv=none; b=QWqb0oq8rgvzeLxQITWJdREZdGnUvsYfk0l/PQFipIekN3YPQ2d7yvm/8NzW2HyMzkdjZ9xXK27VVus68NL/4VoRcjkMDp1zEJXOSeQFFf1u5TgpQ5xdNC+ZO/j/9dQXh8vJMyKyc6sRTJQDUESy/ifPfv1LVRK2EVSgYPAwknQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680167; c=relaxed/simple; bh=6g6iPNuguanRU8zfDKFIKWAIyMllrkBDSKchTQLvrFo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jmEMl/rSdltmyu+l7ieJdgDmQBevtuOD2pHT27GLkEky3tXiisvtPrtnz9DrE1DT3xUmckeFElSh3juFefyNvrwJiRtITqVdi497pTkGSUkFbsc4pqY9wbu/4TsGuJR+si/zGLZYE98Cf1qzq/6Y7mEW8bPlZpe/phMthcDsjpg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=lmvnBckx; arc=none smtp.client-ip=91.218.175.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="lmvnBckx" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1749680160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vjwAclPLPNB0cOL9u2mwbYWPRL/XnTNpPecdd20u+W8=; b=lmvnBckx5QukiFLgXOXw6Ddt0Ee9rJzer/x35eKhcwsjY3akw+jLLRISXc5Go7csrKE2hJ Gv2I3QBXukt/xjHNHnTa2/tAbrQMv5cQVeENziG39GtwDnLti7Wdgb2hvQ0n7dWUxxznN/ iNIh7XjN2aO+x7TaKCMu/LqerSxxep8= From: Shakeel Butt To: Tejun Heo , Andrew Morton Cc: JP Kobryn , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 1/4] cgroup: support to enable nmi-safe css_rstat_updated Date: Wed, 11 Jun 2025 15:15:29 -0700 Message-ID: <20250611221532.2513772-2-shakeel.butt@linux.dev> In-Reply-To: <20250611221532.2513772-1-shakeel.butt@linux.dev> References: <20250611221532.2513772-1-shakeel.butt@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Add necessary infrastructure to enable the nmi-safe execution of css_rstat_updated(). Currently css_rstat_updated() takes a per-cpu per-css raw spinlock to add the given css in the per-cpu per-css update tree. However the kernel can not spin in nmi context, so we need to remove the spinning on the raw spinlock in css_rstat_updated(). To support lockless css_rstat_updated(), let's add necessary data structures in the css and ss structures. Signed-off-by: Shakeel Butt Tested-by: JP Kobryn --- include/linux/cgroup-defs.h | 4 ++++ kernel/cgroup/rstat.c | 23 +++++++++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index e61687d5e496..45860fe5dd0c 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -384,6 +384,9 @@ struct css_rstat_cpu { */ struct cgroup_subsys_state *updated_children; struct cgroup_subsys_state *updated_next; /* NULL if not on the list */ + + struct llist_node lnode; /* lockless list for update */ + struct cgroup_subsys_state *owner; /* back pointer */ }; =20 /* @@ -822,6 +825,7 @@ struct cgroup_subsys { =20 spinlock_t rstat_ss_lock; raw_spinlock_t __percpu *rstat_ss_cpu_lock; + struct llist_head __percpu *lhead; /* lockless update list head */ }; =20 extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index cbeaa499a96a..a5608ae2be27 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -11,6 +11,7 @@ =20 static DEFINE_SPINLOCK(rstat_base_lock); static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); +static DEFINE_PER_CPU(struct llist_head, rstat_backlog_list); =20 static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); =20 @@ -45,6 +46,13 @@ static spinlock_t *ss_rstat_lock(struct cgroup_subsys *s= s) return &rstat_base_lock; } =20 +static inline struct llist_head *ss_lhead_cpu(struct cgroup_subsys *ss, in= t cpu) +{ + if (ss) + return per_cpu_ptr(ss->lhead, cpu); + return per_cpu_ptr(&rstat_backlog_list, cpu); +} + static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) { if (ss) { @@ -468,7 +476,8 @@ int css_rstat_init(struct cgroup_subsys_state *css) for_each_possible_cpu(cpu) { struct css_rstat_cpu *rstatc =3D css_rstat_cpu(css, cpu); =20 - rstatc->updated_children =3D css; + rstatc->owner =3D rstatc->updated_children =3D css; + init_llist_node(&rstatc->lnode); =20 if (is_self) { struct cgroup_rstat_base_cpu *rstatbc; @@ -532,9 +541,19 @@ int __init ss_rstat_init(struct cgroup_subsys *ss) return -ENOMEM; } =20 + if (ss) { + ss->lhead =3D alloc_percpu(struct llist_head); + if (!ss->lhead) { + free_percpu(ss->rstat_ss_cpu_lock); + return -ENOMEM; + } + } + spin_lock_init(ss_rstat_lock(ss)); - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { raw_spin_lock_init(ss_rstat_cpu_lock(ss, cpu)); + init_llist_head(ss_lhead_cpu(ss, cpu)); + } =20 return 0; } --=20 2.47.1 From nobody Sat Oct 11 00:21:51 2025 Received: from out-170.mta1.migadu.com (out-170.mta1.migadu.com [95.215.58.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5B1624C68D for ; Wed, 11 Jun 2025 22:16:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680173; cv=none; b=rD2zFo5N/och7spIouARMiTmrLR+heGkztBb4SEWSNmWQcjfvM+J8x/dJHZ9+pr+RyB9jWgQ9WD8n+8+RfJDQolf3QUkiIp47RHApf5N4AvpICYB1PTDdrtHI/ZaZ5b15dW4Z8UynaIc3xYhWIEH3CPrxo2R3Yl9b/AHTs9r9pU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680173; c=relaxed/simple; bh=RfPByibffsHxwyyKyWbhjhMeyyMJgchWlrqNxxph5bo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eygSz9BhV1CURFvgW9DFAM2ulho4ijx9f8Hs0KqTZf4NpjstbhoaccsCyLVmaTVE3UWb1igkNUBJNrXUjPKzPtZ+XP3PnxVkUE6rfPVTAG01MIE9DMXN8tZyFCCeAQrPmL4PHH2wUiEBUSCE9l8Bs2uLZSAPitB+Wy5FBZt7ZHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=M8xikJAU; arc=none smtp.client-ip=95.215.58.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="M8xikJAU" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1749680168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uFFIt1y9IJBu59K3m1CLaEDnhewuqL+3YiQDRhpGo1g=; b=M8xikJAUUp+5J+F6OXGY0j/RY1qIO7AN56Jrj0EoxB1JNdEk94Lk3WEPxO+EEQox5EItzf F/9mmpwEXB4Xucz2AYFR/V85LFzRwMYlMwM9IfQ/X2f0py38EnvGCHyh/qF6pgZgi5cLIj aOJVeh4ieT280EVlpAUexdDUv1wLz9k= From: Shakeel Butt To: Tejun Heo , Andrew Morton Cc: JP Kobryn , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 2/4] cgroup: make css_rstat_updated nmi safe Date: Wed, 11 Jun 2025 15:15:30 -0700 Message-ID: <20250611221532.2513772-3-shakeel.butt@linux.dev> In-Reply-To: <20250611221532.2513772-1-shakeel.butt@linux.dev> References: <20250611221532.2513772-1-shakeel.butt@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" To make css_rstat_updated() able to safely run in nmi context, let's move the rstat update tree creation at the flush side and use per-cpu lockless lists in struct cgroup_subsys to track the css whose stats are updated on that cpu. The struct cgroup_subsys_state now has per-cpu lnode which needs to be inserted into the corresponding per-cpu lhead of struct cgroup_subsys. Since we want the insertion to be nmi safe, there can be multiple inserters on the same cpu for the same lnode. Here multiple inserters are from stacked contexts like softirq, hardirq and nmi. The current llist does not provide function to protect against the scenario where multiple inserters can use the same lnode. So, using llist_node() out of the box is not safe for this scenario. However we can protect against multiple inserters using the same lnode by using the fact llist node points to itself when not on the llist and atomically reset it and select the winner as the single inserter. Signed-off-by: Shakeel Butt Tested-by: JP Kobryn --- Changes since v1: - More clear code comment as suggested by Tejun. kernel/cgroup/rstat.c | 65 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 53 insertions(+), 12 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a5608ae2be27..a7550961dd12 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -138,13 +138,16 @@ void _css_rstat_cpu_unlock(struct cgroup_subsys_state= *css, int cpu, * @css: target cgroup subsystem state * @cpu: cpu on which rstat_cpu was updated * - * @css's rstat_cpu on @cpu was updated. Put it on the parent's matching - * rstat_cpu->updated_children list. See the comment on top of - * css_rstat_cpu definition for details. + * Atomically inserts the css in the ss's llist for the given cpu. This is + * reentrant safe i.e. safe against softirq, hardirq and nmi. The ss's lli= st + * will be processed at the flush time to create the update tree. */ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cp= u) { - unsigned long flags; + struct llist_head *lhead; + struct css_rstat_cpu *rstatc; + struct css_rstat_cpu __percpu *rstatc_pcpu; + struct llist_node *self; =20 /* * Since bpf programs can call this function, prevent access to @@ -153,19 +156,44 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subs= ys_state *css, int cpu) if (!css_uses_rstat(css)) return; =20 + lockdep_assert_preemption_disabled(); + + /* + * For archs withnot nmi safe cmpxchg or percpu ops support, ignore + * the requests from nmi context. + */ + if ((!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) || + !IS_ENABLED(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS)) && in_nmi()) + return; + + rstatc =3D css_rstat_cpu(css, cpu); + /* If already on list return. */ + if (llist_on_list(&rstatc->lnode)) + return; + /* - * Speculative already-on-list test. This may race leading to - * temporary inaccuracies, which is fine. + * This function can be renentered by irqs and nmis for the same cgroup + * and may try to insert the same per-cpu lnode into the llist. Note + * that llist_add() does not protect against such scenarios. * - * Because @parent's updated_children is terminated with @parent - * instead of NULL, we can tell whether @css is on the list by - * testing the next pointer for NULL. + * To protect against such stacked contexts of irqs/nmis, we use the + * fact that lnode points to itself when not on a list and then use + * this_cpu_cmpxchg() to atomically set to NULL to select the winner + * which will call llist_add(). The losers can assume the insertion is + * successful and the winner will eventually add the per-cpu lnode to + * the llist. */ - if (data_race(css_rstat_cpu(css, cpu)->updated_next)) + self =3D &rstatc->lnode; + rstatc_pcpu =3D css->rstat_cpu; + if (this_cpu_cmpxchg(rstatc_pcpu->lnode.next, self, NULL) !=3D self) return; =20 - flags =3D _css_rstat_cpu_lock(css, cpu, true); + lhead =3D ss_lhead_cpu(css->ss, cpu); + llist_add(&rstatc->lnode, lhead); +} =20 +static void __css_process_update_tree(struct cgroup_subsys_state *css, int= cpu) +{ /* put @css and all ancestors on the corresponding updated lists */ while (true) { struct css_rstat_cpu *rstatc =3D css_rstat_cpu(css, cpu); @@ -191,8 +219,19 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsy= s_state *css, int cpu) =20 css =3D parent; } +} + +static void css_process_update_tree(struct cgroup_subsys *ss, int cpu) +{ + struct llist_head *lhead =3D ss_lhead_cpu(ss, cpu); + struct llist_node *lnode; + + while ((lnode =3D llist_del_first_init(lhead))) { + struct css_rstat_cpu *rstatc; =20 - _css_rstat_cpu_unlock(css, cpu, flags, true); + rstatc =3D container_of(lnode, struct css_rstat_cpu, lnode); + __css_process_update_tree(rstatc->owner, cpu); + } } =20 /** @@ -300,6 +339,8 @@ static struct cgroup_subsys_state *css_rstat_updated_li= st( =20 flags =3D _css_rstat_cpu_lock(root, cpu, false); =20 + css_process_update_tree(root->ss, cpu); + /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) goto unlock_ret; --=20 2.47.1 From nobody Sat Oct 11 00:21:51 2025 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C58324DD17 for ; Wed, 11 Jun 2025 22:16:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680178; cv=none; b=MshdL4/3PBjq6EPssCIB/2bkchh2LBPQPaQr5kIc+7G/vjyfvirioRLcpZBHAWEMMBCWPXhSNbXmeWCqsvK2Z2ef5brF8g5ummoqwjwVdvHFmxcgsOQTmYOfB/5y6MzkGcydM/IVeJgFnRTSKY1izU4e8LCQxEtgDPUxo0fTTCs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680178; c=relaxed/simple; bh=D68pq3sMPpHvWgI65rGf3z86KZKo1vQhqxt3HJb4DCI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FPYWRBviecwfbgrYNAV+FZc6wAYfQudOjtBeK67RpmEG65qGK59qbxYoSPGsnXq9SSlxF7nfAblyAgufU4Byu7Wtxwt8GhfuitxVmtKkvDlhz5BRXt1+Wye7ttM/I9M5Q6PulORSxPnr+t5/4f6cVjEwARrwB/MiH16mdV1ZmdY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=OV/VvR3x; arc=none smtp.client-ip=95.215.58.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="OV/VvR3x" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1749680173; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nkbZifWNsj/jvVHvX2J02HCzx/mgIhtufDCBJPadwMA=; b=OV/VvR3xcNT6rWYSnEPGYAXvZ6T08E2DVqQ91TNN8F7Nla1meRLbeLt8SoZA8cAFGXL8AL hHsbj18GcI7in7vU6DKq6ooauHFQ5lXLVx4EZU3TvIjrivt3QVjMpPRcHVcx/qxyVXGXCv MambegDnX8BmL6SlDCIqf6Pa5GKUNR4= From: Shakeel Butt To: Tejun Heo , Andrew Morton Cc: JP Kobryn , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 3/4] cgroup: remove per-cpu per-subsystem locks Date: Wed, 11 Jun 2025 15:15:31 -0700 Message-ID: <20250611221532.2513772-4-shakeel.butt@linux.dev> In-Reply-To: <20250611221532.2513772-1-shakeel.butt@linux.dev> References: <20250611221532.2513772-1-shakeel.butt@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" The rstat update side used to insert the cgroup whose stats are updated in the update tree and the read side flush the update tree to get the latest uptodate stats. The per-cpu per-subsystem locks were used to synchronize the update and flush side. However now the update side does not access update tree but uses per-cpu lockless lists. So there is no need for locks to synchronize update and flush side. Let's remove them. Suggested-by: JP Kobryn Signed-off-by: Shakeel Butt Tested-by: JP Kobryn --- include/linux/cgroup-defs.h | 7 --- include/trace/events/cgroup.h | 47 --------------- kernel/cgroup/rstat.c | 107 ++-------------------------------- 3 files changed, 4 insertions(+), 157 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 45860fe5dd0c..bca3562e3df4 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -375,12 +375,6 @@ struct css_rstat_cpu { * Child cgroups with stat updates on this cpu since the last read * are linked on the parent's ->updated_children through * ->updated_next. updated_children is terminated by its container css. - * - * In addition to being more compact, singly-linked list pointing to - * the css makes it unnecessary for each per-cpu struct to point back - * to the associated css. - * - * Protected by per-cpu css->ss->rstat_ss_cpu_lock. */ struct cgroup_subsys_state *updated_children; struct cgroup_subsys_state *updated_next; /* NULL if not on the list */ @@ -824,7 +818,6 @@ struct cgroup_subsys { unsigned int depends_on; =20 spinlock_t rstat_ss_lock; - raw_spinlock_t __percpu *rstat_ss_cpu_lock; struct llist_head __percpu *lhead; /* lockless update list head */ }; =20 diff --git a/include/trace/events/cgroup.h b/include/trace/events/cgroup.h index 7d332387be6c..ba9229af9a34 100644 --- a/include/trace/events/cgroup.h +++ b/include/trace/events/cgroup.h @@ -257,53 +257,6 @@ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_unlock, TP_ARGS(cgrp, cpu, contended) ); =20 -/* - * Related to per CPU locks: - * global rstat_base_cpu_lock for base stats - * cgroup_subsys::rstat_ss_cpu_lock for subsystem stats - */ -DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_lock_contended, - - TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), - - TP_ARGS(cgrp, cpu, contended) -); - -DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_lock_contended_fastpath, - - TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), - - TP_ARGS(cgrp, cpu, contended) -); - -DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_locked, - - TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), - - TP_ARGS(cgrp, cpu, contended) -); - -DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_locked_fastpath, - - TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), - - TP_ARGS(cgrp, cpu, contended) -); - -DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_unlock, - - TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), - - TP_ARGS(cgrp, cpu, contended) -); - -DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_unlock_fastpath, - - TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), - - TP_ARGS(cgrp, cpu, contended) -); - #endif /* _TRACE_CGROUP_H */ =20 /* This part must be outside protection */ diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a7550961dd12..c8a48cf83878 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -10,7 +10,6 @@ #include =20 static DEFINE_SPINLOCK(rstat_base_lock); -static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); static DEFINE_PER_CPU(struct llist_head, rstat_backlog_list); =20 static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); @@ -53,86 +52,6 @@ static inline struct llist_head *ss_lhead_cpu(struct cgr= oup_subsys *ss, int cpu) return per_cpu_ptr(&rstat_backlog_list, cpu); } =20 -static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) -{ - if (ss) { - /* - * Depending on config, the subsystem per-cpu lock type may be an - * empty struct. In enviromnents where this is the case, allocation - * of this field is not performed in ss_rstat_init(). Avoid a - * cpu-based offset relative to NULL by returning early. When the - * lock type is zero in size, the corresponding lock functions are - * no-ops so passing them NULL is acceptable. - */ - if (sizeof(*ss->rstat_ss_cpu_lock) =3D=3D 0) - return NULL; - - return per_cpu_ptr(ss->rstat_ss_cpu_lock, cpu); - } - - return per_cpu_ptr(&rstat_base_cpu_lock, cpu); -} - -/* - * Helper functions for rstat per CPU locks. - * - * This makes it easier to diagnose locking issues and contention in - * production environments. The parameter @fast_path determine the - * tracepoints being added, allowing us to diagnose "flush" related - * operations without handling high-frequency fast-path "update" events. - */ -static __always_inline -unsigned long _css_rstat_cpu_lock(struct cgroup_subsys_state *css, int cpu, - const bool fast_path) -{ - struct cgroup *cgrp =3D css->cgroup; - raw_spinlock_t *cpu_lock; - unsigned long flags; - bool contended; - - /* - * The _irqsave() is needed because the locks used for flushing are - * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring this lock - * with the _irq() suffix only disables interrupts on a non-PREEMPT_RT - * kernel. The raw_spinlock_t below disables interrupts on both - * configurations. The _irqsave() ensures that interrupts are always - * disabled and later restored. - */ - cpu_lock =3D ss_rstat_cpu_lock(css->ss, cpu); - contended =3D !raw_spin_trylock_irqsave(cpu_lock, flags); - if (contended) { - if (fast_path) - trace_cgroup_rstat_cpu_lock_contended_fastpath(cgrp, cpu, contended); - else - trace_cgroup_rstat_cpu_lock_contended(cgrp, cpu, contended); - - raw_spin_lock_irqsave(cpu_lock, flags); - } - - if (fast_path) - trace_cgroup_rstat_cpu_locked_fastpath(cgrp, cpu, contended); - else - trace_cgroup_rstat_cpu_locked(cgrp, cpu, contended); - - return flags; -} - -static __always_inline -void _css_rstat_cpu_unlock(struct cgroup_subsys_state *css, int cpu, - unsigned long flags, const bool fast_path) -{ - struct cgroup *cgrp =3D css->cgroup; - raw_spinlock_t *cpu_lock; - - if (fast_path) - trace_cgroup_rstat_cpu_unlock_fastpath(cgrp, cpu, false); - else - trace_cgroup_rstat_cpu_unlock(cgrp, cpu, false); - - cpu_lock =3D ss_rstat_cpu_lock(css->ss, cpu); - raw_spin_unlock_irqrestore(cpu_lock, flags); -} - /** * css_rstat_updated - keep track of updated rstat_cpu * @css: target cgroup subsystem state @@ -335,15 +254,12 @@ static struct cgroup_subsys_state *css_rstat_updated_= list( { struct css_rstat_cpu *rstatc =3D css_rstat_cpu(root, cpu); struct cgroup_subsys_state *head =3D NULL, *parent, *child; - unsigned long flags; - - flags =3D _css_rstat_cpu_lock(root, cpu, false); =20 css_process_update_tree(root->ss, cpu); =20 /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) - goto unlock_ret; + return NULL; =20 /* * Unlink @root from its parent. As the updated_children list is @@ -375,8 +291,7 @@ static struct cgroup_subsys_state *css_rstat_updated_li= st( rstatc->updated_children =3D root; if (child !=3D root) head =3D css_rstat_push_children(head, child, cpu); -unlock_ret: - _css_rstat_cpu_unlock(root, cpu, flags, false); + return head; } =20 @@ -572,29 +487,15 @@ int __init ss_rstat_init(struct cgroup_subsys *ss) { int cpu; =20 - /* - * Depending on config, the subsystem per-cpu lock type may be an empty - * struct. Avoid allocating a size of zero in this case. - */ - if (ss && sizeof(*ss->rstat_ss_cpu_lock)) { - ss->rstat_ss_cpu_lock =3D alloc_percpu(raw_spinlock_t); - if (!ss->rstat_ss_cpu_lock) - return -ENOMEM; - } - if (ss) { ss->lhead =3D alloc_percpu(struct llist_head); - if (!ss->lhead) { - free_percpu(ss->rstat_ss_cpu_lock); + if (!ss->lhead) return -ENOMEM; - } } =20 spin_lock_init(ss_rstat_lock(ss)); - for_each_possible_cpu(cpu) { - raw_spin_lock_init(ss_rstat_cpu_lock(ss, cpu)); + for_each_possible_cpu(cpu) init_llist_head(ss_lhead_cpu(ss, cpu)); - } =20 return 0; } --=20 2.47.1 From nobody Sat Oct 11 00:21:51 2025 Received: from out-189.mta1.migadu.com (out-189.mta1.migadu.com [95.215.58.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79F6E251792 for ; Wed, 11 Jun 2025 22:16:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.189 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680184; cv=none; b=nm3tVINYC/+WLtlpQ4pKSs8ajQ6z4x7RrqgjzNchIlE7h8VQ+4SeINuVcDgjw4WjvZxLhSCrabCkUKMCNr1CQxRs3GnTrisffcOxkAaoyBuyoucnBfXqDgqTj9juMFDXNJDtr3n66JUpOj5IkYH2BeuC0xQTf4DjZtOyHGZnQ2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749680184; c=relaxed/simple; bh=oXikNS+kgN+2w3qPcZgZY0QITtEf/BerEbAt8eoZsE0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CA5xksxetJrzH886OFP59GZwX5AG79M063OGcZbK331wIawpD0jBALLmx+hbHCUVca6Vqg38x24X6XKF7yS0S3OZJPq0t/evO3JDgnjhSySg3QmTNVzTzpWYupuHnsUesEFq6wHPP3iyaLKGalKEwBhjm+SdjyYfh4TEuBXEPF8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=lSQxI91H; arc=none smtp.client-ip=95.215.58.189 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="lSQxI91H" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1749680180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ffjH9W3JevRTfPRrE/1Jq8fYf5w4/j5rXWB6dZVk6cA=; b=lSQxI91H5F+GlORNGReJtTn3xZcEGrrYKuBeb06a7HDXF5w6tidwKZRbwdRJGnI89e63/1 BY2mTqISo55dehhlwbfxDRY528tYIJ9xWj1x4rhM5/wHJr/3Zjvk97PkNmeT0/v/u3WBtd aKleVI3TXZiaCC0I4+dc0EulIJrxb4k= From: Shakeel Butt To: Tejun Heo , Andrew Morton Cc: JP Kobryn , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 4/4] memcg: cgroup: call css_rstat_updated irrespective of in_nmi() Date: Wed, 11 Jun 2025 15:15:32 -0700 Message-ID: <20250611221532.2513772-5-shakeel.butt@linux.dev> In-Reply-To: <20250611221532.2513772-1-shakeel.butt@linux.dev> References: <20250611221532.2513772-1-shakeel.butt@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" css_rstat_updated() is nmi safe, so there is no need to avoid it in in_nmi(), so remove the check. Signed-off-by: Shakeel Butt Tested-by: JP Kobryn --- mm/memcontrol.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 902da8a9c643..d122bfe33e98 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -573,9 +573,7 @@ static inline void memcg_rstat_updated(struct mem_cgrou= p *memcg, int val, if (!val) return; =20 - /* TODO: add to cgroup update tree once it is nmi-safe. */ - if (!in_nmi()) - css_rstat_updated(&memcg->css, cpu); + css_rstat_updated(&memcg->css, cpu); statc_pcpu =3D memcg->vmstats_percpu; for (; statc_pcpu; statc_pcpu =3D statc->parent_pcpu) { statc =3D this_cpu_ptr(statc_pcpu); @@ -2530,7 +2528,8 @@ static inline void account_slab_nmi_safe(struct mem_c= group *memcg, } else { struct mem_cgroup_per_node *pn =3D memcg->nodeinfo[pgdat->node_id]; =20 - /* TODO: add to cgroup update tree once it is nmi-safe. */ + /* preemption is disabled in_nmi(). */ + css_rstat_updated(&memcg->css, smp_processor_id()); if (idx =3D=3D NR_SLAB_RECLAIMABLE_B) atomic_add(nr, &pn->slab_reclaimable); else @@ -2753,7 +2752,8 @@ static inline void account_kmem_nmi_safe(struct mem_c= group *memcg, int val) if (likely(!in_nmi())) { mod_memcg_state(memcg, MEMCG_KMEM, val); } else { - /* TODO: add to cgroup update tree once it is nmi-safe. */ + /* preemption is disabled in_nmi(). */ + css_rstat_updated(&memcg->css, smp_processor_id()); atomic_add(val, &memcg->kmem_stat); } } --=20 2.47.1