From nobody Sat Apr 18 09:27:38 2026 Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAD193A784C for ; Sat, 28 Feb 2026 14:20:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772288445; cv=none; b=TXx/gANMJYnBdvOJIpXb/g8pe3eXjOPbgBqYbdOlg+SvgYipM0QE8UAhkV01YTKDwDsu3ihSvskvD2stWzVD0QD2MqRpcd6jRyQRSDBeznNcx3yJyewL5gfalhHhnKpneWKkiGOThYvCcpqn5tPT3EVZfP+HKnfxS2GvcNbv1d8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772288445; c=relaxed/simple; bh=7f5+r7ycTP2wUCbYBrRzzA6BhtcZIHqy7pWqX2qOIF4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KPiUQPkrRAyyIvw58Jlte4lUFSc/H16erVwFJtbiw+kt8w/Xcg+qhbykMgQ3WwyBqIwQ5t0cGbigYxLbWP1jbmaVLlUmjGKA5pptY5KYQMN16Nib2XSVk9AqSsZdRbZ1vIV4XYk+3gwUtApGSX1NKoz5APVqpHn5SU+aWJj9zoI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=kdNYSrcW; arc=none smtp.client-ip=95.215.58.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="kdNYSrcW" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1772288441; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oaHUxVaobjs+nvwAKL/h30L1BqLPqcHDSgP0kHDwdc8=; b=kdNYSrcWgsufXjWGjib/VZy2Bqp1osTgP3+loYpCfqscJwNdtD8HlbRAcQzfLKQE3q16mx PE7HXXkmhnUgu4h9+jVLjjRtKRB67xXlWuOuk1KmufEOndo3fpYmiPIvPrwImoBx5Bg9CM SUGNO9XzYlFARce+XpR8oDn2vTDJVjI= From: Shakeel Butt To: Tejun Heo Cc: Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Roman Gushchin , Kuniyuki Iwashima , Daniel Sedlak , Meta kernel team , linux-mm@kvack.org, netdev@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 1/3] cgroup: reduce cgroup_file_kn_lock hold time in cgroup_file_notify() Date: Sat, 28 Feb 2026 06:20:16 -0800 Message-ID: <20260228142018.3178529-2-shakeel.butt@linux.dev> In-Reply-To: <20260228142018.3178529-1-shakeel.butt@linux.dev> References: <20260228142018.3178529-1-shakeel.butt@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" cgroup_file_notify() calls kernfs_notify() while holding the global cgroup_file_kn_lock. kernfs_notify() does non-trivial work including wake_up_interruptible() and acquisition of a second global spinlock (kernfs_notify_lock), inflating the hold time. Take a kernfs_get() reference under the lock and call kernfs_notify() after dropping it, following the pattern from cgroup_file_show(). Signed-off-by: Shakeel Butt Reported-by: Jakub Kicinski --- kernel/cgroup/cgroup.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index be1d71dda317..33282c7d71e4 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -4687,6 +4687,7 @@ int cgroup_add_legacy_cftypes(struct cgroup_subsys *s= s, struct cftype *cfts) void cgroup_file_notify(struct cgroup_file *cfile) { unsigned long flags; + struct kernfs_node *kn =3D NULL; =20 spin_lock_irqsave(&cgroup_file_kn_lock, flags); if (cfile->kn) { @@ -4696,11 +4697,17 @@ void cgroup_file_notify(struct cgroup_file *cfile) if (time_in_range(jiffies, last, next)) { timer_reduce(&cfile->notify_timer, next); } else { - kernfs_notify(cfile->kn); + kn =3D cfile->kn; + kernfs_get(kn); cfile->notified_at =3D jiffies; } } spin_unlock_irqrestore(&cgroup_file_kn_lock, flags); + + if (kn) { + kernfs_notify(kn); + kernfs_put(kn); + } } EXPORT_SYMBOL_GPL(cgroup_file_notify); =20 --=20 2.47.3 From nobody Sat Apr 18 09:27:38 2026 Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F2B340FDA9 for ; Sat, 28 Feb 2026 14:20:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.183 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772288448; cv=none; b=BfDs5aiXj+wzllE1WBM2e0vNf3jFcE8iqjmybwP9eKdy/FEpcp9oZmO9ZxI1dMGcV6wFtgbKduxdAFbNVbZYGPp8NzVu0phkEX0iCJw0bUIdvLjjWBkpObvRwUUI2h8Ez+4f6Wy4Yl14/yvsa79mtP2yXwNHEyTdikUdqZroE1M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772288448; c=relaxed/simple; bh=nBmlPo5O/CjJtH+LnUDf9peXd0tMjIrIR3KIGzfZdA4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ut051UmlOC4qaWopQ8QHtQuhe+GHSSCcpiGY4H1skIOKQmgi+YgGSguFr0A962oDiLq8kc25B5nTe5/A8emFPPh4I3cHQxpkSfdItuZX4NkPqGuDRxwszfqr8ld3CTGnf6qwFjBnZEd4t4bqUnhLZ17FiNRZJMo29p1Wa6EntEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=CoTmr6uO; arc=none smtp.client-ip=91.218.175.183 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="CoTmr6uO" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1772288445; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mBR7DSPv0SQTUni7LYAlRsXlj4uQvQDqYNhgjcl//5E=; b=CoTmr6uOZUtm/aSGvKdQHWKa+316V6kLKQ3lL9j/1AgwB07qO6ga1qSdBot1NMHwMCvzqI Nc8JlGKFJYpmR3QLBk0JvXCLav1konHLC2VKHKBodvXDu1asnQl0bmvbQawP4tOvijwRFz 5XerhCSv6PraS2o22S9wCKt/q2MOwtE= From: Shakeel Butt To: Tejun Heo Cc: Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Roman Gushchin , Kuniyuki Iwashima , Daniel Sedlak , Meta kernel team , linux-mm@kvack.org, netdev@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 2/3] cgroup: add lockless fast-path checks to cgroup_file_notify() Date: Sat, 28 Feb 2026 06:20:17 -0800 Message-ID: <20260228142018.3178529-3-shakeel.butt@linux.dev> In-Reply-To: <20260228142018.3178529-1-shakeel.butt@linux.dev> References: <20260228142018.3178529-1-shakeel.butt@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Add two lockless checks before acquiring the lock: 1. READ_ONCE(cfile->kn) NULL check to skip torn-down files. 2. READ_ONCE(cfile->notified_at) check to skip when within the rate-limit window (~10ms). Both checks have safe error directions -- a stale read can only cause unnecessary lock acquisition, never a missed notification. Annotate all write sites with WRITE_ONCE() to pair with the lockless readers. The trade-off is that trailing timer_reduce() calls during bursts are skipped, so the deferred notification that delivers the final state may be lost. This is acceptable for the primary callers like __memcg_memory_event() where events keep arriving. Signed-off-by: Shakeel Butt Reported-by: Jakub Kicinski --- kernel/cgroup/cgroup.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 33282c7d71e4..5473ebd0f6c1 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -1749,7 +1749,7 @@ static void cgroup_rm_file(struct cgroup *cgrp, const= struct cftype *cft) struct cgroup_file *cfile =3D (void *)css + cft->file_offset; =20 spin_lock_irq(&cgroup_file_kn_lock); - cfile->kn =3D NULL; + WRITE_ONCE(cfile->kn, NULL); spin_unlock_irq(&cgroup_file_kn_lock); =20 timer_delete_sync(&cfile->notify_timer); @@ -4430,7 +4430,7 @@ static int cgroup_add_file(struct cgroup_subsys_state= *css, struct cgroup *cgrp, timer_setup(&cfile->notify_timer, cgroup_file_notify_timer, 0); =20 spin_lock_irq(&cgroup_file_kn_lock); - cfile->kn =3D kn; + WRITE_ONCE(cfile->kn, kn); spin_unlock_irq(&cgroup_file_kn_lock); } =20 @@ -4686,20 +4686,27 @@ int cgroup_add_legacy_cftypes(struct cgroup_subsys = *ss, struct cftype *cfts) */ void cgroup_file_notify(struct cgroup_file *cfile) { - unsigned long flags; + unsigned long flags, last, next; struct kernfs_node *kn =3D NULL; =20 + if (!READ_ONCE(cfile->kn)) + return; + + last =3D READ_ONCE(cfile->notified_at); + if (time_before_eq(jiffies, last + CGROUP_FILE_NOTIFY_MIN_INTV)) + return; + spin_lock_irqsave(&cgroup_file_kn_lock, flags); if (cfile->kn) { - unsigned long last =3D cfile->notified_at; - unsigned long next =3D last + CGROUP_FILE_NOTIFY_MIN_INTV; + last =3D cfile->notified_at; + next =3D last + CGROUP_FILE_NOTIFY_MIN_INTV; =20 - if (time_in_range(jiffies, last, next)) { + if (time_before_eq(jiffies, next)) { timer_reduce(&cfile->notify_timer, next); } else { kn =3D cfile->kn; kernfs_get(kn); - cfile->notified_at =3D jiffies; + WRITE_ONCE(cfile->notified_at, jiffies); } } spin_unlock_irqrestore(&cgroup_file_kn_lock, flags); --=20 2.47.3 From nobody Sat Apr 18 09:27:38 2026 Received: from out-174.mta0.migadu.com (out-174.mta0.migadu.com [91.218.175.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 202EE3939AA for ; Sat, 28 Feb 2026 14:20:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772288451; cv=none; b=e83PHA3ZhIBvUAXUShXxvoXqvWVdImqCczO5wc4RopvCsN8SPCErOSgJ/pTd5xbmO5DQMz1tXyoJghuftKtBLZbzxyzbuQP75Po6kMAjlAPTs5L5sodWzBDDY2BwIB77QnktUUF6xTvWhjMbzDdeXnyCiY/crI7HInzwbnKfYTI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772288451; c=relaxed/simple; bh=Nt/BUPJSvd9WEDeocpmWi6aZzL7kRdJmd8JiUYGr7OA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kiY+o4qVOdlg5nlRX/C6k33OWsigIcmn/dn0hNuUJpDl3mCAbzcOdyUKdBb3mUAW6aZVvfM3b/AgqyIof/nzwIXHmUupuVGeDanIQHSfN1YPD7jEPSHjfECJayMkcwNBoTPyeYXbPFEYH31Ggk2mM7/Jvik7ks9IOguOHlqwpgE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=JcI/7Imt; arc=none smtp.client-ip=91.218.175.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="JcI/7Imt" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1772288448; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cngsu4QC/s/LsygBwa0X1EgXCc0toiqGLgWGzEjWZ30=; b=JcI/7ImtJYPNViw4hRAStRxiuwGD/K2BPN2nAe3Qmvd6t5RC1XpXCbi6w+EM62Bo875+l4 5tDwOAflDGo6MkCtOuMzbje/vhPK9w/h+fiw6v7rup3BxR94zCBkD1jBGwdz45EHLE8ITp x150gwG5DGtZHS+kNuAC8Jjzi8/vgj4= From: Shakeel Butt To: Tejun Heo Cc: Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Roman Gushchin , Kuniyuki Iwashima , Daniel Sedlak , Meta kernel team , linux-mm@kvack.org, netdev@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 3/3] cgroup: replace global cgroup_file_kn_lock with per-cgroup_file lock Date: Sat, 28 Feb 2026 06:20:18 -0800 Message-ID: <20260228142018.3178529-4-shakeel.butt@linux.dev> In-Reply-To: <20260228142018.3178529-1-shakeel.butt@linux.dev> References: <20260228142018.3178529-1-shakeel.butt@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Replace the global cgroup_file_kn_lock with a per-cgroup_file spinlock to eliminate cross-cgroup contention as it is not really protecting data shared between different cgroups. The lock is initialized in cgroup_add_file() alongside timer_setup(). No lock acquisition is needed during initialization since the cgroup directory is being populated under cgroup_mutex and no concurrent accessors exist at that point. Signed-off-by: Shakeel Butt Reported-by: Jakub Kicinski --- include/linux/cgroup-defs.h | 1 + kernel/cgroup/cgroup.c | 24 ++++++++---------------- 2 files changed, 9 insertions(+), 16 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index bb92f5c169ca..ba26b5d05ce3 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -167,6 +167,7 @@ struct cgroup_file { struct kernfs_node *kn; unsigned long notified_at; struct timer_list notify_timer; + spinlock_t lock; }; =20 /* diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 5473ebd0f6c1..b502acad3c5c 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -107,12 +107,6 @@ static bool cgroup_debug __read_mostly; */ static DEFINE_SPINLOCK(cgroup_idr_lock); =20 -/* - * Protects cgroup_file->kn for !self csses. It synchronizes notifications - * against file removal/re-creation across css hiding. - */ -static DEFINE_SPINLOCK(cgroup_file_kn_lock); - DEFINE_PERCPU_RWSEM(cgroup_threadgroup_rwsem); =20 #define cgroup_assert_mutex_or_rcu_locked() \ @@ -1748,9 +1742,9 @@ static void cgroup_rm_file(struct cgroup *cgrp, const= struct cftype *cft) struct cgroup_subsys_state *css =3D cgroup_css(cgrp, cft->ss); struct cgroup_file *cfile =3D (void *)css + cft->file_offset; =20 - spin_lock_irq(&cgroup_file_kn_lock); + spin_lock_irq(&cfile->lock); WRITE_ONCE(cfile->kn, NULL); - spin_unlock_irq(&cgroup_file_kn_lock); + spin_unlock_irq(&cfile->lock); =20 timer_delete_sync(&cfile->notify_timer); } @@ -4428,10 +4422,8 @@ static int cgroup_add_file(struct cgroup_subsys_stat= e *css, struct cgroup *cgrp, struct cgroup_file *cfile =3D (void *)css + cft->file_offset; =20 timer_setup(&cfile->notify_timer, cgroup_file_notify_timer, 0); - - spin_lock_irq(&cgroup_file_kn_lock); - WRITE_ONCE(cfile->kn, kn); - spin_unlock_irq(&cgroup_file_kn_lock); + spin_lock_init(&cfile->lock); + cfile->kn =3D kn; } =20 return 0; @@ -4696,7 +4688,7 @@ void cgroup_file_notify(struct cgroup_file *cfile) if (time_before_eq(jiffies, last + CGROUP_FILE_NOTIFY_MIN_INTV)) return; =20 - spin_lock_irqsave(&cgroup_file_kn_lock, flags); + spin_lock_irqsave(&cfile->lock, flags); if (cfile->kn) { last =3D cfile->notified_at; next =3D last + CGROUP_FILE_NOTIFY_MIN_INTV; @@ -4709,7 +4701,7 @@ void cgroup_file_notify(struct cgroup_file *cfile) WRITE_ONCE(cfile->notified_at, jiffies); } } - spin_unlock_irqrestore(&cgroup_file_kn_lock, flags); + spin_unlock_irqrestore(&cfile->lock, flags); =20 if (kn) { kernfs_notify(kn); @@ -4727,10 +4719,10 @@ void cgroup_file_show(struct cgroup_file *cfile, bo= ol show) { struct kernfs_node *kn; =20 - spin_lock_irq(&cgroup_file_kn_lock); + spin_lock_irq(&cfile->lock); kn =3D cfile->kn; kernfs_get(kn); - spin_unlock_irq(&cgroup_file_kn_lock); + spin_unlock_irq(&cfile->lock); =20 if (kn) kernfs_show(kn, show); --=20 2.47.3