From nobody Wed Oct 1 22:32:39 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E842303C9F; Wed, 1 Oct 2025 14:48:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759330120; cv=none; b=M/JWSkNhrqb+TWjaKWZNiWfjm6+RtykTwOLmtu4P8QlnsiJLigkuDQmd1r6JtwsrFP0TG1QqVll58cDg9hbWkJBNfHc2xlFiGBf+D6+7FyyJjtZm+SmCbqhmi8A15QPCGff5tiBaBGXW3xsAKGMIRzBj8r0Nb6JBA0XEt7LHDrg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759330120; c=relaxed/simple; bh=KsyAiYK2BbMqGVso6WNz3IqAqctiHCv1RlmZwiUylvk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Mp+T3/7lkZdiBMItKzKloMpErY+HtZlhnkK8+rS3hv8z2eo+P0iTpsT5IEymICh1zMIU1kIOnp1pUAhDs2zbljmt8LM7uki1/+ySK1PhY2rrqDhOh7+lGF4BiipgdPfIBJOTsze0CF4LnxbuXMuTIiTretHQD3M99vjNRQKhpAY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IOLocjYY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IOLocjYY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7BB2DC4CEF1; Wed, 1 Oct 2025 14:48:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1759330119; bh=KsyAiYK2BbMqGVso6WNz3IqAqctiHCv1RlmZwiUylvk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IOLocjYYkXFAZXMqryXV0ZEbA/RIlrBIXShFF8/TaQvjQdSePGfozymi1/eawhMy2 P2iH8sz6rt8WFLF5WXu+a2GXKbRuyFeavaHob7MB+B5W/FgkJUrnbdQVHKoEgzBXfb /iCwHdFh7KAZsgae0okgLf6uXOweHfq8LUIgxCsVQaRK+CECLYg0PdXVr2Xk9kqpVy JcFKTX5BkrFVkzNO+ZeNoNT3RqPl9H7uxOVfCH1q3wYgE2LaGuSiO7/xwzFlv958TE FSnMAVGKwgFB2JflQMo7o1m4TaCMU+3xRDm2VamMncgDf1rjcEgKJSvy1m6F8OxV7w z4DtVuH/GBzVQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7D305CE1418; Wed, 1 Oct 2025 07:48:34 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 11/21] srcu: Create a DEFINE_SRCU_FAST() Date: Wed, 1 Oct 2025 07:48:22 -0700 Message-Id: <20251001144832.631770-11-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7fa58961-2dce-4e08-8174-1d1cc592210f@paulmck-laptop> References: <7fa58961-2dce-4e08-8174-1d1cc592210f@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST() macros that are similar to DEFINE_SRCU() and DEFINE_STATIC_SRCU(), but which create srcu_struct structures that are usable only by readers initiated by srcu_read_lock_fast() and friends. This commit does make DEFINE_SRCU_FAST() available to modules, in which case the per-CPU srcu_data structures are not created at compile time, but rather at module-load time. This means that the >srcu_reader_flavor field of the srcu_data structure is not available. Therefore, this commit instead creates an ->srcu_reader_flavor field in the srcu_struct structure, adds arguments to the DEFINE_SRCU()-related macros to initialize this new field, and extends the checks in the __srcu_check_read_flavor() function to include this new field. This commit also allows dynamically allocated srcu_struct structure to be marked for SRCU-fast readers. It does so by defining a new init_srcu_struct_fast() function that marks the specified srcu_struct structure for use by srcu_read_lock_fast() and friends. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/notifier.h | 2 +- include/linux/srcu.h | 16 ++++++++++++++-- include/linux/srcutiny.h | 13 ++++++++++--- include/linux/srcutree.h | 34 +++++++++++++++++++++++----------- kernel/rcu/srcutree.c | 36 ++++++++++++++++++++++++++++++++++-- 5 files changed, 82 insertions(+), 19 deletions(-) diff --git a/include/linux/notifier.h b/include/linux/notifier.h index b42e6473496882..01b6c9d9956f90 100644 --- a/include/linux/notifier.h +++ b/include/linux/notifier.h @@ -109,7 +109,7 @@ extern void srcu_init_notifier_head(struct srcu_notifie= r_head *nh); .mutex =3D __MUTEX_INITIALIZER(name.mutex), \ .head =3D NULL, \ .srcuu =3D __SRCU_USAGE_INIT(name.srcuu), \ - .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu), \ + .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu, 0), \ } =20 #define ATOMIC_NOTIFIER_HEAD(name) \ diff --git a/include/linux/srcu.h b/include/linux/srcu.h index ada65b58bc4c5f..26de47820c58cd 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -25,8 +25,10 @@ struct srcu_struct; =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key); +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key); +#ifndef CONFIG_TINY_SRCU +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define init_srcu_struct(ssp) \ ({ \ @@ -35,10 +37,20 @@ int __init_srcu_struct(struct srcu_struct *ssp, const c= har *name, __init_srcu_struct((ssp), #ssp, &__srcu_key); \ }) =20 +#define init_srcu_struct_fast(ssp) \ +({ \ + static struct lock_class_key __srcu_key; \ + \ + __init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \ +}) + #define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map =3D { .name =3D #srcu_name= }, #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 int init_srcu_struct(struct srcu_struct *ssp); +#ifndef CONFIG_TINY_SRCU +int init_srcu_struct_fast(struct srcu_struct *ssp); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define __SRCU_DEP_MAP_INIT(srcu_name) #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 3bfbd44cb1b37d..92e6ab53398fc0 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -31,7 +31,7 @@ struct srcu_struct { =20 void srcu_drive_gp(struct work_struct *wp); =20 -#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \ +#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored, ____ignored) \ { \ .srcu_wq =3D __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \ .srcu_cb_tail =3D &name.srcu_cb_head, \ @@ -44,13 +44,20 @@ void srcu_drive_gp(struct work_struct *wp); * Tree SRCU, which needs some per-CPU data. */ #define DEFINE_SRCU(name) \ - struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) + struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, name) #define DEFINE_STATIC_SRCU(name) \ - static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) +#define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) +#define DEFINE_STATIC_SRCU_FAST(name) \ + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) =20 // Dummy structure for srcu_notifier_head. struct srcu_usage { }; #define __SRCU_USAGE_INIT(name) { } +#define __init_srcu_struct_fast __init_srcu_struct +#ifndef CONFIG_DEBUG_LOCK_ALLOC +#define init_srcu_struct_fast init_srcu_struct +#endif // #ifndef CONFIG_DEBUG_LOCK_ALLOC =20 void synchronize_srcu(struct srcu_struct *ssp); =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 93ad18acd6d02d..05400f70baa40a 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -104,6 +104,7 @@ struct srcu_usage { struct srcu_struct { struct srcu_ctr __percpu *srcu_ctrp; struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ + u8 srcu_reader_flavor; struct lockdep_map dep_map; struct srcu_usage *srcu_sup; /* Update-side data. */ }; @@ -162,20 +163,21 @@ struct srcu_struct { .work =3D __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ } =20 -#define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ +#define __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ .srcu_sup =3D &usage_name, \ + .srcu_reader_flavor =3D fast, \ __SRCU_DEP_MAP_INIT(name) =20 -#define __SRCU_STRUCT_INIT_MODULE(name, usage_name) \ +#define __SRCU_STRUCT_INIT_MODULE(name, usage_name, fast) \ { \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 -#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name) \ +#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name, fast) \ { \ .sda =3D &pcpu_name, \ .srcu_ctrp =3D &pcpu_name.srcu_ctrs[0], \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 /* @@ -196,23 +198,33 @@ struct srcu_struct { * init_srcu_struct(&my_srcu); * * See include/linux/percpu-defs.h for the rules on per-CPU variables. + * + * DEFINE_SRCU_FAST() creates an srcu_struct and associated structures + * whose readers must be of the SRCU-fast variety. DEFINE_SRCU_FAST() + * cannot be used within modules due to there being no place to + * put the desired ->srcu_reader_flavor value. If in-module use of + * DEFINE_SRCU_FAST() becomes necessary, the srcu_struct structure will + * need to grow in order to store this value. */ #ifdef MODULE -# define __DEFINE_SRCU(name, is_static) \ +# define __DEFINE_SRCU(name, fast, is_static) \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ - is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage); \ + is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage, \ + fast); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") =3D &name #else -# define __DEFINE_SRCU(name, is_static) \ +# define __DEFINE_SRCU(name, fast, is_static) \ static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ is_static struct srcu_struct name =3D \ - __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) + __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data, fast) #endif -#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) -#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) +#define DEFINE_SRCU(name) __DEFINE_SRCU(name, 0, /* not static */) +#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static) +#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) +#define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST, static) =20 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 38b440b0b0c80b..9869a13b876342 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -286,16 +286,29 @@ static int init_srcu_struct_fields(struct srcu_struct= *ssp, bool is_static) =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key) +static int +__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struc= t lock_class_key *key) { /* Don't re-initialize a lock while it is held. */ debug_check_no_locks_freed((void *)ssp, sizeof(*ssp)); lockdep_init_map(&ssp->dep_map, name, key, 0); return init_srcu_struct_fields(ssp, false); } + +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key) +{ + ssp->srcu_reader_flavor =3D 0; + return __init_srcu_struct_common(ssp, name, key); +} EXPORT_SYMBOL_GPL(__init_srcu_struct); =20 +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return __init_srcu_struct_common(ssp, name, key); +} +EXPORT_SYMBOL_GPL(__init_srcu_struct_fast); + #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /** @@ -308,10 +321,26 @@ EXPORT_SYMBOL_GPL(__init_srcu_struct); */ int init_srcu_struct(struct srcu_struct *ssp) { + ssp->srcu_reader_flavor =3D 0; return init_srcu_struct_fields(ssp, false); } EXPORT_SYMBOL_GPL(init_srcu_struct); =20 +/** + * init_srcu_struct_fast - initialize a fast-reader sleep-RCU structure + * @ssp: structure to initialize. + * + * Must invoke this on a given srcu_struct before passing that srcu_struct + * to any other function. Each srcu_struct represents a separate domain + * of SRCU protection. + */ +int init_srcu_struct_fast(struct srcu_struct *ssp) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return init_srcu_struct_fields(ssp, false); +} +EXPORT_SYMBOL_GPL(init_srcu_struct_fast); + #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* @@ -734,6 +763,9 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, = int read_flavor) =20 sdp =3D raw_cpu_ptr(ssp->sda); old_read_flavor =3D READ_ONCE(sdp->srcu_reader_flavor); + WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor !=3D ssp->srcu_reader= _flavor); + WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && + old_read_flavor !=3D ssp->srcu_reader_flavor); if (!old_read_flavor) { old_read_flavor =3D cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); if (!old_read_flavor) --=20 2.40.1