From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C194BC433EF for ; Wed, 22 Jun 2022 22:51:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376285AbiFVWv6 (ORCPT ); Wed, 22 Jun 2022 18:51:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345695AbiFVWvp (ORCPT ); Wed, 22 Jun 2022 18:51:45 -0400 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD6914091D for ; Wed, 22 Jun 2022 15:51:42 -0700 (PDT) Received: by mail-qv1-xf33.google.com with SMTP id 59so4714712qvb.3 for ; Wed, 22 Jun 2022 15:51:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dKuAST7WhYd+0VtTZTrKu1Ea8AH7dgcHLvVcnqvIxho=; b=Y2wdkXmbf7rO9gEc5BnotLszY/jIFC/OaOPYIbCWkurvj20tv4MykWguNWJZJqrnhN 6EQDvAEPEfhTOyuc7V5ko8qBy3H8mRtR6GHR4Ul+EWg/8vE7tZ0Fgg55wWRx6Mq9+vVE kziMz6rB5qYs1149MZ3v+5gpHyFrx/b4oCeMc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dKuAST7WhYd+0VtTZTrKu1Ea8AH7dgcHLvVcnqvIxho=; b=6L+eR1MVO6w7PWPdiIQeAVCXu3UWO5DJEglSqmEB4ud4n6Uau70tp55+qpNrjl5Tj0 hDy+Drp5Eayl9IW+Iw/Ir8qf5fDNUWY0J1goZWOo+Fwv6v0C19fpQd8VBVqilxxMJdSy HxjC64J3dMXhADjn95rp/apyWhlq5ewKSZ9H8k3vTx0eSqEsFlQYQ+dYGAnazEtPF/T8 H6xMe2au6/qTlCKQjQ8qGLG8OP58rlRDJvJ8HaruhLACEQnnQK79CpoDH8gyM0Sel+3l rZdgOzJCNFRyEpSsDQmrBvXv/amaKEQe2VRG9JkH32ty2Y0NdOmgQt1OkkIRKNYZmm6D 6vfQ== X-Gm-Message-State: AJIora8o+mgEMsvOM0pCJYKs46mk+7BikhcRWcPuKWXpmzoQ7M8DgI3B qikqY/i9XwyVzZjCkbHb/LHTXg== X-Google-Smtp-Source: AGRyM1t/Hqq3A/9IMpyiuPb4DeaFA6/usCi6I3f8b62OSiyiVflFbN0/xx2rNMcT+eAC773+jaEmKw== X-Received: by 2002:ad4:5def:0:b0:46e:6974:e2e with SMTP id jn15-20020ad45def000000b0046e69740e2emr29933209qvb.129.1655938301789; Wed, 22 Jun 2022 15:51:41 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:41 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 1/8] rcu: Introduce call_rcu_lazy() API implementation Date: Wed, 22 Jun 2022 22:50:55 +0000 Message-Id: <20220622225102.2112026-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Implement timer-based RCU lazy callback batching. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure will flush it in a future patch. To handle several corner cases automagically (such as rcu_barrier() and hotplug), we re-use bypass lists to handle lazy CBs. The bypass list length has the lazy CB length included in it. A separate lazy CB length counter is also introduced to keep track of the number of lazy CBs. Suggested-by: Paul McKenney Signed-off-by: Joel Fernandes (Google) Reported-by: kernel test robot --- include/linux/rcu_segcblist.h | 1 + include/linux/rcupdate.h | 6 ++ kernel/rcu/Kconfig | 8 +++ kernel/rcu/rcu_segcblist.c | 19 ++++++ kernel/rcu/rcu_segcblist.h | 14 ++++ kernel/rcu/tree.c | 24 +++++-- kernel/rcu/tree.h | 10 +-- kernel/rcu/tree_nocb.h | 125 +++++++++++++++++++++++++--------- 8 files changed, 164 insertions(+), 43 deletions(-) diff --git a/include/linux/rcu_segcblist.h b/include/linux/rcu_segcblist.h index 659d13a7ddaa..9a992707917b 100644 --- a/include/linux/rcu_segcblist.h +++ b/include/linux/rcu_segcblist.h @@ -22,6 +22,7 @@ struct rcu_cblist { struct rcu_head *head; struct rcu_head **tail; long len; + long lazy_len; }; =20 #define RCU_CBLIST_INITIALIZER(n) { .head =3D NULL, .tail =3D &n.head } diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 1a32036c918c..9191a3d88087 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -82,6 +82,12 @@ static inline int rcu_preempt_depth(void) =20 #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ =20 +#ifdef CONFIG_RCU_LAZY +void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func); +#else +#define call_rcu_lazy(head, func) call_rcu(head, func) +#endif + /* Internal to kernel */ void rcu_init(void); extern int rcu_scheduler_active; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 27aab870ae4c..0bffa992fdc4 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -293,4 +293,12 @@ config TASKS_TRACE_RCU_READ_MB Say N here if you hate read-side memory barriers. Take the default if you are unsure. =20 +config RCU_LAZY + bool "RCU callback lazy invocation functionality" + depends on RCU_NOCB_CPU + default n + help + To save power, batch RCU callbacks and flush after delay, memory + pressure or callback list growing too big. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c index c54ea2b6a36b..627a3218a372 100644 --- a/kernel/rcu/rcu_segcblist.c +++ b/kernel/rcu/rcu_segcblist.c @@ -20,6 +20,7 @@ void rcu_cblist_init(struct rcu_cblist *rclp) rclp->head =3D NULL; rclp->tail =3D &rclp->head; rclp->len =3D 0; + rclp->lazy_len =3D 0; } =20 /* @@ -32,6 +33,15 @@ void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct = rcu_head *rhp) WRITE_ONCE(rclp->len, rclp->len + 1); } =20 +/* + * Enqueue an rcu_head structure onto the specified callback list. + */ +void rcu_cblist_enqueue_lazy(struct rcu_cblist *rclp, struct rcu_head *rhp) +{ + rcu_cblist_enqueue(rclp, rhp); + WRITE_ONCE(rclp->lazy_len, rclp->lazy_len + 1); +} + /* * Flush the second rcu_cblist structure onto the first one, obliterating * any contents of the first. If rhp is non-NULL, enqueue it as the sole @@ -60,6 +70,15 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, } } =20 +void rcu_cblist_flush_enqueue_lazy(struct rcu_cblist *drclp, + struct rcu_cblist *srclp, + struct rcu_head *rhp) +{ + rcu_cblist_flush_enqueue(drclp, srclp, rhp); + if (rhp) + WRITE_ONCE(srclp->lazy_len, 1); +} + /* * Dequeue the oldest rcu_head structure from the specified callback * list. diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index 431cee212467..c3d7de65b689 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -15,14 +15,28 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *= rclp) return READ_ONCE(rclp->len); } =20 +/* Return number of callbacks in the specified callback list. */ +static inline long rcu_cblist_n_lazy_cbs(struct rcu_cblist *rclp) +{ +#ifdef CONFIG_RCU_LAZY + return READ_ONCE(rclp->lazy_len); +#else + return 0; +#endif +} + /* Return number of callbacks in segmented callback list by summing seglen= . */ long rcu_segcblist_n_segment_cbs(struct rcu_segcblist *rsclp); =20 void rcu_cblist_init(struct rcu_cblist *rclp); void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct rcu_head *rhp); +void rcu_cblist_enqueue_lazy(struct rcu_cblist *rclp, struct rcu_head *rhp= ); void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, struct rcu_cblist *srclp, struct rcu_head *rhp); +void rcu_cblist_flush_enqueue_lazy(struct rcu_cblist *drclp, + struct rcu_cblist *srclp, + struct rcu_head *rhp); struct rcu_head *rcu_cblist_dequeue(struct rcu_cblist *rclp); =20 /* diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index c25ba442044a..d2e3d6e176d2 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3098,7 +3098,8 @@ static void check_cb_ovld(struct rcu_data *rdp) * Implementation of these memory-ordering guarantees is described here: * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. */ -void call_rcu(struct rcu_head *head, rcu_callback_t func) +static void +__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) { static atomic_t doublefrees; unsigned long flags; @@ -3139,7 +3140,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t f= unc) } =20 check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); @@ -3161,8 +3162,21 @@ void call_rcu(struct rcu_head *head, rcu_callback_t = func) local_irq_restore(flags); } } -EXPORT_SYMBOL_GPL(call_rcu); =20 +#ifdef CONFIG_RCU_LAZY +void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, true); +} +EXPORT_SYMBOL_GPL(call_rcu_lazy); +#endif + +void call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, false); + +} +EXPORT_SYMBOL_GPL(call_rcu); =20 /* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (HZ / 50) @@ -4056,7 +4070,7 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func =3D rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -4476,7 +4490,7 @@ void rcutree_migrate_callbacks(int cpu) my_rdp =3D this_cpu_ptr(&rcu_data); my_rnp =3D my_rdp->mynode; rcu_nocb_lock(my_rdp); /* irqs already disabled. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies, false)); raw_spin_lock_rcu_node(my_rnp); /* irqs already disabled. */ /* Leverage recent GPs and set GP for new callbacks. */ needwake =3D rcu_advance_cbs(my_rnp, rdp) || diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 2ccf5845957d..fec4fad6654b 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -267,8 +267,9 @@ struct rcu_data { /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 #define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE 2 -#define RCU_NOCB_WAKE_FORCE 3 +#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE 3 +#define RCU_NOCB_WAKE_FORCE 4 =20 #define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500)) /* For jiffies_till_first_fqs and */ @@ -436,9 +437,10 @@ static struct swait_queue_head *rcu_nocb_gp_get(struct= rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *r= hp, - unsigned long j); + unsigned long j, bool lazy); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags); + bool *was_alldone, unsigned long flags, + bool lazy); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index e369efe94fda..b9244f22e102 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -256,6 +256,8 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool for= ce) return __wake_nocb_gp(rdp_gp, rdp, force, flags); } =20 +#define LAZY_FLUSH_JIFFIES (10 * HZ) + /* * Arrange to wake the GP kthread for this NOCB group at some future * time when it is safe to do so. @@ -272,7 +274,10 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, i= nt waketype, * Bypass wakeup overrides previous deferments. In case * of callback storm, no need to wake up too early. */ - if (waketype =3D=3D RCU_NOCB_WAKE_BYPASS) { + if (waketype =3D=3D RCU_NOCB_WAKE_LAZY) { + mod_timer(&rdp_gp->nocb_timer, jiffies + LAZY_FLUSH_JIFFIES); + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + } else if (waketype =3D=3D RCU_NOCB_WAKE_BYPASS) { mod_timer(&rdp_gp->nocb_timer, jiffies + 2); WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); } else { @@ -296,7 +301,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, in= t waketype, * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head= *rhp, - unsigned long j) + unsigned long j, bool lazy) { struct rcu_cblist rcl; =20 @@ -310,7 +315,13 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *= rdp, struct rcu_head *rhp, /* Note: ->cblist.len already accounts for ->nocb_bypass contents. */ if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + + trace_printk("call_rcu_lazy callbacks =3D %ld\n", READ_ONCE(rdp->nocb_byp= ass.lazy_len)); + /* The lazy CBs are being flushed, but a new one might be enqueued. */ + if (lazy) + rcu_cblist_flush_enqueue_lazy(&rcl, &rdp->nocb_bypass, rhp); + else + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); @@ -326,13 +337,13 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data = *rdp, struct rcu_head *rhp, * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *r= hp, - unsigned long j) + unsigned long j, bool lazy) { if (!rcu_rdp_is_offloaded(rdp)) return true; rcu_lockdep_assert_cblist_protected(rdp); rcu_nocb_bypass_lock(rdp); - return rcu_nocb_do_flush_bypass(rdp, rhp, j); + return rcu_nocb_do_flush_bypass(rdp, rhp, j, lazy); } =20 /* @@ -345,7 +356,7 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *= rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; - WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j)); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } =20 /* @@ -367,12 +378,14 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data= *rdp, unsigned long j) * there is only one CPU in operation. */ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { unsigned long c; unsigned long cur_gp_seq; unsigned long j =3D jiffies; long ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + long n_lazy_cbs =3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); =20 lockdep_assert_irqs_disabled(); =20 @@ -414,30 +427,37 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp,= struct rcu_head *rhp, } WRITE_ONCE(rdp->nocb_nobypass_count, c); =20 - // If there hasn't yet been all that many ->cblist enqueues - // this jiffy, tell the caller to enqueue onto ->cblist. But flush - // ->nocb_bypass first. - if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy) { + // If caller passed a non-lazy CB and there hasn't yet been all that + // many ->cblist enqueues this jiffy, tell the caller to enqueue it + // onto ->cblist. But flush ->nocb_bypass first. Also do so, if total + // number of CBs (lazy + non-lazy) grows too much. + // + // Note that if the bypass list has lazy CBs, and the main list is + // empty, and rhp happens to be non-lazy, then we end up flushing all + // the lazy CBs to the main list as well. That's the right thing to do, + // since we are kick-starting RCU GP processing anyway for the non-lazy + // one, we can just reuse that GP for the already queued-up lazy ones. + if ((rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy && !lazy) || + (lazy && n_lazy_cbs >=3D qhimark)) { rcu_nocb_lock(rdp); *was_alldone =3D !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("FirstQ")); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j)); + lazy ? TPS("FirstLazyQ") : TPS("FirstQ")); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false)); WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } =20 // If ->nocb_bypass has been used too long or is too full, // flush ->nocb_bypass to ->cblist. - if ((ncbs && j !=3D READ_ONCE(rdp->nocb_bypass_first)) || - ncbs >=3D qhimark) { + if ((ncbs && j !=3D READ_ONCE(rdp->nocb_bypass_first)) || ncbs >=3D qhima= rk) { rcu_nocb_lock(rdp); - if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { + if (!rcu_nocb_flush_bypass(rdp, rhp, j, true)) { *was_alldone =3D !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("FirstQ")); + lazy ? TPS("FirstLazyQ") : TPS("FirstQ")); WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } @@ -455,12 +475,20 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp,= struct rcu_head *rhp, rcu_nocb_wait_contended(rdp); rcu_nocb_bypass_lock(rdp); ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + n_lazy_cbs =3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ - rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + if (lazy) + rcu_cblist_enqueue_lazy(&rdp->nocb_bypass, rhp); + else + rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); if (!ncbs) { WRITE_ONCE(rdp->nocb_bypass_first, j); - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, + lazy ? TPS("FirstLazyBQ") : TPS("FirstBQ")); + } else if (!n_lazy_cbs && lazy) { + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstLazyBQ")); } + rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ if (ncbs) { @@ -493,7 +521,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, = bool was_alldone, { unsigned long cur_gp_seq; unsigned long j; - long len; + long len, lazy_len, bypass_len; struct task_struct *t; =20 // If we are being polled or there is no kthread, just leave. @@ -506,9 +534,16 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp,= bool was_alldone, } // Need to actually to a wakeup. len =3D rcu_segcblist_n_cbs(&rdp->cblist); + bypass_len =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_len =3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); if (was_alldone) { rdp->qlen_last_fqs_check =3D len; - if (!irqs_disabled_flags(flags)) { + // Only lazy CBs in bypass list + if (lazy_len && bypass_len =3D=3D lazy_len) { + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazy")); + } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); @@ -599,8 +634,8 @@ static inline bool nocb_gp_update_state_deoffloading(st= ruct rcu_data *rdp, */ static void nocb_gp_wait(struct rcu_data *my_rdp) { - bool bypass =3D false; - long bypass_ncbs; + bool bypass =3D false, lazy =3D false; + long bypass_ncbs, lazy_ncbs; int __maybe_unused cpu =3D my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; @@ -648,12 +683,21 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) continue; } bypass_ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); - if (bypass_ncbs && + lazy_ncbs =3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + if (lazy_ncbs && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + LAZY_FLUSH_JIFFIE= S) || + bypass_ncbs > qhimark)) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs =3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + } else if (bypass_ncbs && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || bypass_ncbs > 2 * qhimark)) { // Bypass full or old, so flush it. (void)rcu_nocb_try_flush_bypass(rdp, j); bypass_ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs =3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { rcu_nocb_unlock_irqrestore(rdp, flags); if (needwake_state) @@ -662,8 +706,11 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) } if (bypass_ncbs) { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("Bypass")); - bypass =3D true; + bypass_ncbs =3D=3D lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + if (bypass_ncbs =3D=3D lazy_ncbs) + lazy =3D true; + else + bypass =3D true; } rnp =3D rdp->mynode; =20 @@ -713,12 +760,21 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) my_rdp->nocb_gp_gp =3D needwait_gp; my_rdp->nocb_gp_seq =3D needwait_gp ? wait_gp_seq : 0; =20 - if (bypass && !rcu_nocb_poll) { - // At least one child with non-empty ->nocb_bypass, so set - // timer in order to avoid stranding its callbacks. - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); + // At least one child with non-empty ->nocb_bypass, so set + // timer in order to avoid stranding its callbacks. + if (!rcu_nocb_poll) { + // If bypass list only has lazy CBs. Add a deferred + // lazy wake up. + if (lazy && !bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazyIsDeferred")); + // Otherwise add a deferred bypass wake up. + } else if (bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, + TPS("WakeBypassIsDeferred")); + } } + if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ if (gotcbs) @@ -999,7 +1055,7 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); /* * Start with invoking rcu_core() early. This way if the current thread * happens to preempt an ongoing call to rcu_core() in the middle, @@ -1500,13 +1556,14 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) } =20 static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *r= hp, - unsigned long j) + unsigned long j, bool lazy) { return true; } =20 static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { return false; } --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 689BCCCA479 for ; Wed, 22 Jun 2022 22:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359700AbiFVWvr (ORCPT ); Wed, 22 Jun 2022 18:51:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235515AbiFVWvm (ORCPT ); Wed, 22 Jun 2022 18:51:42 -0400 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6717040A3E for ; Wed, 22 Jun 2022 15:51:41 -0700 (PDT) Received: by mail-qv1-xf33.google.com with SMTP id cs6so22935352qvb.6 for ; Wed, 22 Jun 2022 15:51:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K8zLqAuQGGInMEO/tC5oFW9jqhiZEM0UzbeIkjAgT/E=; b=u6ATeb3gtaca1zArGlTV20bvGnJYCnyGi0lrVTh4JYP4rTbIjKfKdeM9RDdwdI9RAi ujrJtVDrjaCtmGg5lfrv26gOYUGzGdvprVDIpOwUgQm35vlAlgg6z/uQvruLfpgAqFnf LBh4FHH8YPXW9tWhspYhEKaA5r2GbBI4MSxtM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K8zLqAuQGGInMEO/tC5oFW9jqhiZEM0UzbeIkjAgT/E=; b=7wwznK2J8JpW32YcTuzA5tkJud5ivcNl9OSgmtfu1zJAHhBbaBRsa68lCsFk30d5ox 7Q8d8t1OxV7jKLkZL/rbQDlyuCLqH4iDUN07SYqwMgMs8GxPyfoc1INCwOvR1qPBBv5L UQdBXyXcnz07KD13+sSCBirNCHOOxzlQv0OBKfBH8injJawKKLYApQi458DaHpcLNzWO VAhUSu/Hpzp8e6zbO0mXb/KDS1pOHuq8QZ9YOyT4u+T3QKcksJn7NZhkBpBAt3ZJW5Qa VsGxR0oUJATQ8PBXWdPjExoQ2Rwq4c0El7jhnq8r2dGqrBUAE3WqzHXs06+e81nFQDr5 t0Yw== X-Gm-Message-State: AJIora8CKGmK+3RIpdy9RzlDgaKOda3+npkTOTGWL+fE6WwTIULCnEiI cabHiYMG92UlgaYGdybPIhnJEA== X-Google-Smtp-Source: AGRyM1vNLmCVtta0shMDBQ0BoHzenJeOoo6ATvfQjsRBrP8Iu05+Inum8C5HKWzZ9QzXs2v3TExN7g== X-Received: by 2002:a05:6214:2501:b0:470:3f6d:c075 with SMTP id gf1-20020a056214250100b004703f6dc075mr16486903qvb.95.1655938300584; Wed, 22 Jun 2022 15:51:40 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:39 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, Marco Elver Subject: [PATCH v2 1/1] context_tracking: Use arch_atomic_read() in __ct_state for KASAN Date: Wed, 22 Jun 2022 22:50:54 +0000 Message-Id: <20220622225102.2112026-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: "Paul E. McKenney" Context tracking's __ct_state() function can be invoked from noinstr state where RCU is not watching. This means that its use of atomic_read() causes KASAN to invoke the non-noinstr __kasan_check_read() function from the noinstr function __ct_state(). This is problematic because someone tracing the __kasan_check_read() function could get a nasty surprise because of RCU not watching. This commit therefore replaces the __ct_state() function's use of atomic_read() with arch_atomic_read(), which KASAN does not attempt to add instrumention to. Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker Cc: Marco Elver Reviewed-by: Marco Elver --- include/linux/context_tracking_state.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index 0aecc07fb4f5..81c51e5f0314 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -49,7 +49,7 @@ DECLARE_PER_CPU(struct context_tracking, context_tracking= ); =20 static __always_inline int __ct_state(void) { - return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK; + return arch_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE= _MASK; } #endif =20 --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B156CC43334 for ; Wed, 22 Jun 2022 22:51:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359812AbiFVWv4 (ORCPT ); Wed, 22 Jun 2022 18:51:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345645AbiFVWvp (ORCPT ); Wed, 22 Jun 2022 18:51:45 -0400 Received: from mail-qv1-xf2f.google.com (mail-qv1-xf2f.google.com [IPv6:2607:f8b0:4864:20::f2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDBB740A3E for ; Wed, 22 Jun 2022 15:51:43 -0700 (PDT) Received: by mail-qv1-xf2f.google.com with SMTP id y14so22113329qvs.10 for ; Wed, 22 Jun 2022 15:51:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tZ2VmzE2lHGW4TDS5ibElbKctouxZ1GAoFxv9DWSNFQ=; b=uAvtO1wTymA1OKGzMjhdAg4Xrl4FVIKHWotoJraXcn07le/b0XJ7l01X76a8WK4Q90 kmUbwQlNMXoi/6t3IXkCKFj3BN/t49X5NpYdWKYhWTCqUvfHy+wkYe934MhNMPgDiTFa hN6xqJX72Ktoan5doV3c3TtrMl4Bo7T3X2iuo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tZ2VmzE2lHGW4TDS5ibElbKctouxZ1GAoFxv9DWSNFQ=; b=BSXZ3NpuKo4reGTLj93kvrgXuUBwAjG/cyUd3VbUqQyezACXSXu6VjIGQmUNNxLi+W Udu1JtINVlI7mVjM/lkNMGyELGHMQi6AowGZg6KpMS4zCo2sRRkn/VTqhHwN57Kgbb+S /Ap9okn7vrMDXtm33rnHcB5LLCyfo1a22MslsXQdJGCxaq0XwetRbGOz1lg9mS7IhKhA KW/lf/d3H/UHNrw1ICX0LFRjF8vNoyerOBOCOyH8k4Tv7407PYj17fmGEyHGb8udnAk+ NMNDXBSvWFbyAAP1+nBGIZlemsbjB1euY+IpqoHfafcFstX2p/OvF8closmDtnHJ6s43 wKRg== X-Gm-Message-State: AJIora/N6QQmnt6BoPuknDoBEGYExGm0/reNni37bXfn33eqmQfE9/CI PJWs3F9BDHk8yiq643ujOKo+yQ== X-Google-Smtp-Source: AGRyM1tPJC4HgfQNhb89caYKVnJktcVYAzMTrPx9I8PteeedjGdtVYdEKZzoNlG7gLoLRzY4NHCDLQ== X-Received: by 2002:a05:6214:f08:b0:470:519a:168a with SMTP id gw8-20020a0562140f0800b00470519a168amr9962716qvb.29.1655938303002; Wed, 22 Jun 2022 15:51:43 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:42 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, Joel Fernandes Subject: [PATCH v2 2/8] rcu: shrinker for lazy rcu Date: Wed, 22 Jun 2022 22:50:56 +0000 Message-Id: <20220622225102.2112026-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcu_segcblist.h | 14 +++++++++-- kernel/rcu/tree_nocb.h | 48 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index c3d7de65b689..cf71425dbb5e 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -15,16 +15,26 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *= rclp) return READ_ONCE(rclp->len); } =20 +#ifdef CONFIG_RCU_LAZY /* Return number of callbacks in the specified callback list. */ static inline long rcu_cblist_n_lazy_cbs(struct rcu_cblist *rclp) { -#ifdef CONFIG_RCU_LAZY return READ_ONCE(rclp->lazy_len); +} + +static inline void rcu_cblist_reset_lazy_len(struct rcu_cblist *rclp) +{ + WRITE_ONCE(rclp->lazy_len, 0); +} #else +static inline long rcu_cblist_n_lazy_cbs(struct rcu_cblist *rclp) +{ return 0; -#endif } =20 +static inline void rcu_cblist_reset_lazy_len(struct rcu_cblist *rclp) {} +#endif + /* Return number of callbacks in segmented callback list by summing seglen= . */ long rcu_segcblist_n_segment_cbs(struct rcu_segcblist *rsclp); =20 diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index b9244f22e102..2f5da12811a5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1207,6 +1207,51 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); =20 +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { + int cpu; + unsigned long count =3D 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); + count +=3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) { + int cpu; + unsigned long flags; + unsigned long count =3D 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); + int _count =3D rcu_cblist_n_lazy_cbs(&rdp->nocb_bypass); + if (_count =3D=3D 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + rcu_cblist_reset_lazy_len(&rdp->nocb_bypass); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -=3D _count; + count +=3D _count; + if (sc->nr_to_scan <=3D 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker =3D { + .count_objects =3D lazy_rcu_shrink_count, + .scan_objects =3D lazy_rcu_shrink_scan, + .batch =3D 0, + .seeks =3D DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1244,6 +1289,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; =20 + if (register_shrinker(&lazy_rcu_shrinker)) + pr_err("Failed to register lazy_rcu shrinker!\n"); + #if defined(CONFIG_NO_HZ_FULL) if (tick_nohz_full_running) cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask); --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC83CC43334 for ; Wed, 22 Jun 2022 22:52:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376420AbiFVWwA (ORCPT ); Wed, 22 Jun 2022 18:52:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1358156AbiFVWvq (ORCPT ); Wed, 22 Jun 2022 18:51:46 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C205141302 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id n15so23395363qvh.12 for ; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H+kSqBl3RB9I7OY9hm0XkeWhykcK3oDF7t29sGiTzsw=; b=UXqJkdf/KUdA/7F69IfFzfqaGez+SJKEsYw7nuoWGBu0X1hTbSLO5mv5SvEHwxyffi v6O/+XZ6kAy6ihIhq5dQgHhxeYv1dabpAGdmOWyyEl2imSX0RJXhmRwGEI5Yg9J0bB3R MPHCpMk4uVaFRmEAqQ1tWKl/a8e4oKq/zkQgE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H+kSqBl3RB9I7OY9hm0XkeWhykcK3oDF7t29sGiTzsw=; b=D+g6STB0ZEVS039oJ2z+QwNmIftYd+f6wjdcI+IfbJzYW6Yzf4+plDok3bhwxdWQwW Edpkq3eHx0bFU68mkUfleBT8BeT6pgXLlDjIIJg/qf81P5mPd13O2AExRvL5d/UhyO6a 5hRwmbeglhkM1YODgAv6KqokGmDaxvTSoGycqQxky3mGi+OM4xufPxrKZHQgBhL59wEI eNhnFKLYYaHkr5HmHJAOlyHp4SG57WAn7vO/pY3RIU4sEaiH2VYKts3c3P43ltC9Js8S bqUmeVuHg9VguzUxm/3VBVEJ8L0bux8QENBEqOls97b94SGQEYc7YB1ArWGMH3lJbFZp ri5g== X-Gm-Message-State: AJIora/LC2CkCOtLdVmbGzc9gIqIBsVa2lqpqMi1jIdWwcoj9th7jRBw s8xLbPjWOrll1/djaRq41k9I4g== X-Google-Smtp-Source: AGRyM1uuVGVPfaJ0Ie+qNySjbSbMPeGHjQQx1Ig5cgUKgMzyi+kfvrEg7TFmRbp9baN9SneC9y1kag== X-Received: by 2002:ac8:594d:0:b0:305:1a14:de6c with SMTP id 13-20020ac8594d000000b003051a14de6cmr5135737qtz.677.1655938303882; Wed, 22 Jun 2022 15:51:43 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:43 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 3/8] fs: Move call_rcu() to call_rcu_lazy() in some paths Date: Wed, 22 Jun 2022 22:50:57 +0000 Message-Id: <20220622225102.2112026-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. When testing, we found that these paths were invoked often when the system is not doing anything (screen is ON but otherwise idle). Signed-off-by: Joel Fernandes (Google) --- fs/dcache.c | 4 ++-- fs/eventpoll.c | 2 +- fs/file_table.c | 2 +- fs/inode.c | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 93f4f5ee07bf..7f51bac390c8 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -366,7 +366,7 @@ static void dentry_free(struct dentry *dentry) if (unlikely(dname_external(dentry))) { struct external_name *p =3D external_name(dentry); if (likely(atomic_dec_and_test(&p->u.count))) { - call_rcu(&dentry->d_u.d_rcu, __d_free_external); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free_external); return; } } @@ -374,7 +374,7 @@ static void dentry_free(struct dentry *dentry) if (dentry->d_flags & DCACHE_NORCU) __d_free(&dentry->d_u.d_rcu); else - call_rcu(&dentry->d_u.d_rcu, __d_free); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free); } =20 /* diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 971f98af48ff..57b3f781760c 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -729,7 +729,7 @@ static int ep_remove(struct eventpoll *ep, struct epite= m *epi) * ep->mtx. The rcu read side, reverse_path_check_proc(), does not make * use of the rbn field. */ - call_rcu(&epi->rcu, epi_rcu_free); + call_rcu_lazy(&epi->rcu, epi_rcu_free); =20 percpu_counter_dec(&ep->user->epoll_watches); =20 diff --git a/fs/file_table.c b/fs/file_table.c index 5424e3a8df5f..417f57e9cb30 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -56,7 +56,7 @@ static inline void file_free(struct file *f) security_file_free(f); if (!(f->f_mode & FMODE_NOACCOUNT)) percpu_counter_dec(&nr_files); - call_rcu(&f->f_u.fu_rcuhead, file_free_rcu); + call_rcu_lazy(&f->f_u.fu_rcuhead, file_free_rcu); } =20 /* diff --git a/fs/inode.c b/fs/inode.c index bd4da9c5207e..38fe040ddbd6 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -312,7 +312,7 @@ static void destroy_inode(struct inode *inode) return; } inode->free_inode =3D ops->free_inode; - call_rcu(&inode->i_rcu, i_callback); + call_rcu_lazy(&inode->i_rcu, i_callback); } =20 /** --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54CC9C43334 for ; Wed, 22 Jun 2022 22:52:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376536AbiFVWwF (ORCPT ); Wed, 22 Jun 2022 18:52:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1358284AbiFVWvq (ORCPT ); Wed, 22 Jun 2022 18:51:46 -0400 Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAC274162A for ; Wed, 22 Jun 2022 15:51:45 -0700 (PDT) Received: by mail-qv1-xf34.google.com with SMTP id i17so20760325qvo.13 for ; Wed, 22 Jun 2022 15:51:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=czRWmqXwJRiSEH25SM5GYb5Yl+/vPCw6VIvQPTxRUu8=; b=TffKA6dV8lgb0owObCDYZAoszUr3Wlohvc9Oe7lMyM8i3Tp+lw7j5I5jYbuBeUeBHr hw6rj2Ic8XvDgkEB7GWWhmAB28oH0ZodN41wjG+/n3WraZU8356yTc+2MMq9mW58RFg1 SJPUS+ANKIuHUETM3nyiMJpm3QA/edyOhnM0M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=czRWmqXwJRiSEH25SM5GYb5Yl+/vPCw6VIvQPTxRUu8=; b=YFCJU0uXSCLQf+5HBRwBvaRJEFMWc0FDSMWafmryKoFAUSCK7Byf6rqr7fYUy0SFz9 EuUI7fGknQEel9mIcsUDWCf7a4/Pvsnl4aK/av1sJL6dOueUwnajRD1wP2ovaDK8eMEC PNRjgLFbA8r3wnBW/5M08xXfjUL/f1wqTAQ03gzCXB7eW5WwhdAW+smuMfNoK4fBlgtf xBvmjV/wYaAnElcTl4LfggtKSIZiWQ7OvivroHNSOqvhPwwD3NRREyxG01NI3vF9aCQZ 0ymsBZ1KF4plVrsnTdtGmhRZXIGbynXhLPexfRbo2PbtOuIdX5LIbQ0TzbJ8IgopDmRD +tog== X-Gm-Message-State: AJIora8HlLTLBiocpLWhPk7LII2ojzCqxBfIMEscQrXXmQalpG6Mdmke ae0lKIt4nIOC2R2MgqHv+B+RAw== X-Google-Smtp-Source: AGRyM1uo/5L9en8Zb8UJ/9q6iw0BqvqvJQYPuudXxkgc81Cjbz58Q0qLgY+GAxCRwsEEJphkgx2Ofg== X-Received: by 2002:a05:6214:ac7:b0:470:6dd0:c3da with SMTP id g7-20020a0562140ac700b004706dd0c3damr3578574qvi.43.1655938304958; Wed, 22 Jun 2022 15:51:44 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:44 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 4/8] rcu/nocb: Add option to force all call_rcu() to lazy Date: Wed, 22 Jun 2022 22:50:58 +0000 Message-Id: <20220622225102.2112026-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This will be used in the rcu scale test, to ensure that fly-by call_rcu()s = do no cause call_rcu_lazy() CBs to be flushed to the rdp ->cblist. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcu.h | 2 ++ kernel/rcu/tree.c | 11 ++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 4916077119f3..71c0f45e70c3 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -472,6 +472,7 @@ void do_trace_rcu_torture_read(const char *rcutorturena= me, unsigned long c_old, unsigned long c); void rcu_gp_set_torture_wait(int duration); +void rcu_force_call_rcu_to_lazy(bool force); #else static inline void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq) @@ -490,6 +491,7 @@ void do_trace_rcu_torture_read(const char *rcutorturena= me, do { } while (0) #endif static inline void rcu_gp_set_torture_wait(int duration) { } +static inline void rcu_force_call_rcu_to_lazy(bool force) { } #endif =20 #if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) || IS_MODULE(CONFIG_RCU_TORTURE_TE= ST) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d2e3d6e176d2..711679d10cbb 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3171,9 +3171,18 @@ void call_rcu_lazy(struct rcu_head *head, rcu_callba= ck_t func) EXPORT_SYMBOL_GPL(call_rcu_lazy); #endif =20 +static bool force_call_rcu_to_lazy; + +void rcu_force_call_rcu_to_lazy(bool force) +{ + if (IS_ENABLED(CONFIG_RCU_SCALE_TEST)) + WRITE_ONCE(force_call_rcu_to_lazy, force); +} +EXPORT_SYMBOL_GPL(rcu_force_call_rcu_to_lazy); + void call_rcu(struct rcu_head *head, rcu_callback_t func) { - return __call_rcu_common(head, func, false); + return __call_rcu_common(head, func, force_call_rcu_to_lazy); =20 } EXPORT_SYMBOL_GPL(call_rcu); --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDC31C43334 for ; Wed, 22 Jun 2022 22:52:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376482AbiFVWwC (ORCPT ); Wed, 22 Jun 2022 18:52:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1358800AbiFVWvr (ORCPT ); Wed, 22 Jun 2022 18:51:47 -0400 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 490084199E for ; Wed, 22 Jun 2022 15:51:46 -0700 (PDT) Received: by mail-qv1-xf33.google.com with SMTP id cs6so22935352qvb.6 for ; Wed, 22 Jun 2022 15:51:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Vd4a4Sb5uy4qMpkEMdY0l97I8Oa96Mx9jouj7Tqv17A=; b=m94cURFdCK6y/WgXeNVeJSsjpwtrlSDB3PxHYiJdPWnOgrL7pP1TMYNYz1W7b+MV+D o4mjJJIuJZdtP6EZIKXzc4DK8WuAV8kIBrLtnHGBvt0N/j4sCit/6477vompbS+f68Rl jrHVbbjKj6qyxBdoOzjlkNClgq4YCQxnGirjQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Vd4a4Sb5uy4qMpkEMdY0l97I8Oa96Mx9jouj7Tqv17A=; b=PKhypLLQDk1HnxebfLhea+VW9veoR5wdbH1QcMbAq9yGFHrEBza4BfOGhPtamRaJ+q OxqMaTKNcBVs7/CP08NEUI+vU5oIFCoYtZOvAaLYradMEDjDwcuF78lzNoF7yELSt9Tl 8IWpiLNrWbGVsSjvDZveoRk8JARgAZA0/wXydNWIvE/IkwgW/15EU/6+XHWQBCYU05jX hb8gwFY7Pj32Ue6FSjBFlklT6ljPItU2Vw08PhLEExCG6sAP3xSKIAETg0ay6KLdk32u u5YrNMthze0nu9VxorNUZtTuo9hlNvpGY64ZGMyc1QzszsmWqHmYFusCZs8uyRYiLE9a Frcw== X-Gm-Message-State: AJIora9g2bxYv/NWHm6pSs8/nJbnLDQoKjhrgcvWplWzO0bxogU/7/Ob A49SxDCC/nP41VFdJki34GDPBg== X-Google-Smtp-Source: AGRyM1sVrXdFvKoX00v5/yoAGUyGmQGrFinrp1P16vG+Qt7RLn229XFOM58LGmehXc+yFXMyMumP1g== X-Received: by 2002:a05:622a:148d:b0:305:1a61:1003 with SMTP id t13-20020a05622a148d00b003051a611003mr5335099qtx.269.1655938305958; Wed, 22 Jun 2022 15:51:45 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:45 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 5/8] rcu/nocb: Wake up gp thread when flushing Date: Wed, 22 Jun 2022 22:50:59 +0000 Message-Id: <20220622225102.2112026-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We notice that rcu_barrier() can take a really long time. It appears that this can happen when all CBs are lazy and the timer does not fire yet. So after flushing, nothing wakes up GP thread. This patch forces GP thread to wake when bypass flushing happens, this fixes the rcu_barrier() delays with lazy CBs. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 2f5da12811a5..b481f1ea57c0 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -325,6 +325,8 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *r= dp, struct rcu_head *rhp, rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); + + wake_nocb_gp(rdp, true); return true; } =20 --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BDE7C433EF for ; Wed, 22 Jun 2022 22:52:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234293AbiFVWwJ (ORCPT ); Wed, 22 Jun 2022 18:52:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235515AbiFVWvs (ORCPT ); Wed, 22 Jun 2022 18:51:48 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCE9240A3E for ; Wed, 22 Jun 2022 15:51:47 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id n197so13716696qke.1 for ; Wed, 22 Jun 2022 15:51:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VLpRj7ISmpLc7bYzOd0iAFFn8wSs/6G7/zuKg/d2BV4=; b=uUovH7zphZj5j44vTCjc6fQKvn5vsTqcZUI+oVj1KSCucV7WrcEJQ8xUffNEA3EpXw GDZIrTbjd/nV4pr6O7V+SKKTEYqB8ykmpL9P4Z5A7FiTLFLHAH9e+1C715gu+U1RgZmp p9M3xiILqZ6xnVI8MP1Vs70BvMK0jUzTgAYIs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VLpRj7ISmpLc7bYzOd0iAFFn8wSs/6G7/zuKg/d2BV4=; b=aQoHLfViZY6ZNlZV9A1caPmqaB5nNB7I7UAsVzP/+lOyD1QRfL8yaT+gFcSY6Y9K3Q UiNNc1oH6TA9HU2bL0PwFt0s/L3by4dfWmASOpCwRU9bXOOn8apKi07umeruGpxYYiQj x+jRlcCTgLqdbFtFr8DAzEOCZhgqLwJpg9Mwb/+/FPh3S3gEX9SwefSDswKeZjhZ9u5s 9F1u/0kCQzHmcwYqFzz0O2CMboAO3Edgq8VYsfk7DhZnakzogVc0eoHjR3NJ02bLtmc3 7vp3Ldk/bSElWirllL3zlvyjq9EIo6P/cD0XmmkpU16crZdDo6Rw4nkyHXp6mbNxAIc9 Xksw== X-Gm-Message-State: AJIora/o/5pgCrlo6nNb3NyUcUWeN+q2Wxac7FV6tKwxvl8z6kTQsA3S bmYuallLLEdI4VcNh/0McLRZCA== X-Google-Smtp-Source: AGRyM1sPh4hJnzcz0VNfhHC4UeQg+s3UcijJzqY655+yVI85xyoOc6cAptRFlkJHV19NUU17LL9gZg== X-Received: by 2002:a37:c86:0:b0:6ae:ded5:2002 with SMTP id 128-20020a370c86000000b006aeded52002mr3616328qkm.594.1655938306664; Wed, 22 Jun 2022 15:51:46 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:46 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 6/8] rcuscale: Add test for using call_rcu_lazy() to emulate kfree_rcu() Date: Wed, 22 Jun 2022 22:51:00 +0000 Message-Id: <20220622225102.2112026-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Reuse the kfree_rcu() test in order to be able to compare the memory reclai= ming properties of call_rcu_lazy() with kfree_rcu(). With this test, we find similar memory footprint and time call_rcu_lazy() free'ing takes compared to kfree_rcu(). Also we confirm that call_rcu_lazy() can survive OOM during extremely frequent calls. If we really push it, i.e. boot system with low memory and compare kfree_rcu() with call_rcu_lazy(), I find that call_rcu_lazy() is more resilient and is much harder to produce OOM as compared to kfree_rcu(). Signed-off-by: Joel Fernandes (Google) Reported-by: kernel test robot --- kernel/rcu/rcu.h | 6 ++++ kernel/rcu/rcuscale.c | 64 +++++++++++++++++++++++++++++++++++++++++- kernel/rcu/tree_nocb.h | 17 ++++++++++- 3 files changed, 85 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 71c0f45e70c3..436faf80a66b 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -473,6 +473,12 @@ void do_trace_rcu_torture_read(const char *rcutorturen= ame, unsigned long c); void rcu_gp_set_torture_wait(int duration); void rcu_force_call_rcu_to_lazy(bool force); + +#if IS_ENABLED(CONFIG_RCU_SCALE_TEST) +unsigned long rcu_scale_get_jiffies_till_flush(void); +void rcu_scale_set_jiffies_till_flush(unsigned long j); +#endif + #else static inline void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 277a5bfb37d4..58ee5c2cb37b 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -95,6 +95,7 @@ torture_param(int, verbose, 1, "Enable verbose debugging = printk()s"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to d= isable"); torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?= "); torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate.= "); +torture_param(int, kfree_rcu_by_lazy, 0, "Use call_rcu_lazy() to emulate k= free_rcu()?"); =20 static char *scale_type =3D "rcu"; module_param(scale_type, charp, 0444); @@ -658,6 +659,13 @@ struct kfree_obj { struct rcu_head rh; }; =20 +/* Used if doing RCU-kfree'ing via call_rcu_lazy(). */ +void kfree_rcu_lazy(struct rcu_head *rh) +{ + struct kfree_obj *obj =3D container_of(rh, struct kfree_obj, rh); + kfree(obj); +} + static int kfree_scale_thread(void *arg) { @@ -695,6 +703,11 @@ kfree_scale_thread(void *arg) if (!alloc_ptr) return -ENOMEM; =20 + if (kfree_rcu_by_lazy) { + call_rcu_lazy(&(alloc_ptr->rh), kfree_rcu_lazy); + continue; + } + // By default kfree_rcu_test_single and kfree_rcu_test_double are // initialized to false. If both have the same value (false or true) // both are randomly tested, otherwise only the one with value true @@ -737,6 +750,9 @@ kfree_scale_cleanup(void) { int i; =20 + if (kfree_rcu_by_lazy) + rcu_force_call_rcu_to_lazy(false); + if (torture_cleanup_begin()) return; =20 @@ -766,11 +782,55 @@ kfree_scale_shutdown(void *arg) return -EINVAL; } =20 +// Used if doing RCU-kfree'ing via call_rcu_lazy(). +unsigned long jiffies_at_lazy_cb; +struct rcu_head lazy_test1_rh; +int rcu_lazy_test1_cb_called; +void call_rcu_lazy_test1(struct rcu_head *rh) +{ + jiffies_at_lazy_cb =3D jiffies; + WRITE_ONCE(rcu_lazy_test1_cb_called, 1); +} + static int __init kfree_scale_init(void) { long i; int firsterr =3D 0; + unsigned long orig_jif, jif_start; + + // Force all call_rcu() to call_rcu_lazy() so that non-lazy CBs + // do not remove laziness of the lazy ones (since the test tries + // to stress call_rcu_lazy() for OOM). + // + // Also, do a quick self-test to ensure laziness is as much as + // expected. + if (kfree_rcu_by_lazy) { + /* do a test to check the timeout. */ + orig_jif =3D rcu_scale_get_jiffies_till_flush(); + + rcu_force_call_rcu_to_lazy(true); + rcu_scale_set_jiffies_till_flush(2 * HZ); + rcu_barrier(); + + jif_start =3D jiffies; + jiffies_at_lazy_cb =3D 0; + call_rcu_lazy(&lazy_test1_rh, call_rcu_lazy_test1); + + smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL =3D=3D 1); + + rcu_scale_set_jiffies_till_flush(orig_jif); + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { + pr_alert("Lazy CBs are not being lazy as expected!\n"); + return -1; + } + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) { + pr_alert("Lazy CBs are being too lazy!\n"); + return -1; + } + } =20 kfree_nrealthreads =3D compute_real(kfree_nthreads); /* Start up the kthreads. */ @@ -783,7 +843,9 @@ kfree_scale_init(void) schedule_timeout_uninterruptible(1); } =20 - pr_alert("kfree object size=3D%zu\n", kfree_mult * sizeof(struct kfree_ob= j)); + pr_alert("kfree object size=3D%zu, kfree_rcu_by_lazy=3D%d\n", + kfree_mult * sizeof(struct kfree_obj), + kfree_rcu_by_lazy); =20 kfree_reader_tasks =3D kcalloc(kfree_nrealthreads, sizeof(kfree_reader_ta= sks[0]), GFP_KERNEL); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index b481f1ea57c0..255f2945b0fc 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -257,6 +257,21 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool fo= rce) } =20 #define LAZY_FLUSH_JIFFIES (10 * HZ) +unsigned long jiffies_till_flush =3D LAZY_FLUSH_JIFFIES; + +#ifdef CONFIG_RCU_SCALE_TEST +void rcu_scale_set_jiffies_till_flush(unsigned long jif) +{ + jiffies_till_flush =3D jif; +} +EXPORT_SYMBOL(rcu_scale_set_jiffies_till_flush); + +unsigned long rcu_scale_get_jiffies_till_flush(void) +{ + return jiffies_till_flush; +} +EXPORT_SYMBOL(rcu_scale_get_jiffies_till_flush); +#endif =20 /* * Arrange to wake the GP kthread for this NOCB group at some future @@ -275,7 +290,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, in= t waketype, * of callback storm, no need to wake up too early. */ if (waketype =3D=3D RCU_NOCB_WAKE_LAZY) { - mod_timer(&rdp_gp->nocb_timer, jiffies + LAZY_FLUSH_JIFFIES); + mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); } else if (waketype =3D=3D RCU_NOCB_WAKE_BYPASS) { mod_timer(&rdp_gp->nocb_timer, jiffies + 2); --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B055C43334 for ; Wed, 22 Jun 2022 22:52:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376657AbiFVWwO (ORCPT ); Wed, 22 Jun 2022 18:52:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359771AbiFVWvt (ORCPT ); Wed, 22 Jun 2022 18:51:49 -0400 Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 450FE41303 for ; Wed, 22 Jun 2022 15:51:48 -0700 (PDT) Received: by mail-qv1-xf2a.google.com with SMTP id o43so27420978qvo.4 for ; Wed, 22 Jun 2022 15:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oulAy7NlmqgAX/AG2BIWch0UpN6lWOBq/Xx+LqGxyiU=; b=Zd4Acj6vFejlAuM4LGySk0mn5EpwW09kqb0tPrWdZuUSIL1zdVj6CpnngjAG5CU35U 9yra0A4qcuU3QelVy9Thq3pL7e37068nnzNVRkCJG9aRq+zBR8VCYjCfHgtlXa90Ue4H IGj41tdwPLrWondJBlm13KXtRnLZJprFieFjI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oulAy7NlmqgAX/AG2BIWch0UpN6lWOBq/Xx+LqGxyiU=; b=dV1EhxEze450Ji2XM1P0ciQ7PP+9pflsDkGpUnFAHXHer7PC2KLsV/5jMNgVw3j9vc uKzU3sWD0m9cUrfu4gA1b4EjjbKH4Bkdnxqna2MCFriEVHAYQNcPb+IQ3+7WhtfKeprq lrdjT+nKwyq5fiHvB49YE4U6cMZjc5cJIJcA9dwyI+Wauem8A1jolkRuBdKiRaHat0J1 +4duOkAM9FTA59SEyjby3pP1r+1iVZrEzej7GmqGmtCQaA8gN90FwYQdi2hqXCL+7fAy t79TCsjq5u766vNxlVurivMhPqI0zWxfkC26hMkWhCyq2QKwxBAqacG3y/JC06mWsPxL u/hw== X-Gm-Message-State: AJIora8n79w+44bc67w6DrhfxSXF3/vcQROVWsLXx8DoGo+49l0CO/Rm m0QxeyINGCWEBuSAWHf2yskSXA== X-Google-Smtp-Source: AGRyM1uNf6I5+fQ6KcX/9MOZDknEktSDE/Z0l3YRPxDg86JuWUz6QknaienDhYHazulLUj/lBR6DPA== X-Received: by 2002:a0c:9066:0:b0:470:2b7a:2078 with SMTP id o93-20020a0c9066000000b004702b7a2078mr22020533qvo.51.1655938307947; Wed, 22 Jun 2022 15:51:47 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:47 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 7/8] rcu/nocb: Rewrite deferred wake up logic to be more clean Date: Wed, 22 Jun 2022 22:51:01 +0000 Message-Id: <20220622225102.2112026-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There are 2 things this function does: 1. modify the gp wake timer. 2. save the value of the strongest requested wake up so far. The strongest is "wake force" and the weakest is "lazy". The existing logic already does the following: 1. if the existing deferred wake is stronger than the requested one (requested in waketype), modify the gp timer to be more in the future. For example, if the existing one is WAKE and the new waketype requested is BYPASS, then the timer is made to expire later than earlier. 2. even though the timer is modified in #1, a weaker waketype does not end up changing rdp->nocb_gp_defer to be weaker. In other words, ->nocb_gp_defer records the strongest waketype requested so far, even though the timer may or may not be the soonest expiry possible. For simplicity, we write this logic using switch statements and consolidate some of the timer modification operations. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 255f2945b0fc..67b0bd5d233a 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -282,6 +282,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, in= t waketype, { unsigned long flags; struct rcu_data *rdp_gp =3D rdp->nocb_gp_rdp; + unsigned long mod_jif =3D 0; =20 raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); =20 @@ -289,19 +290,31 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, = int waketype, * Bypass wakeup overrides previous deferments. In case * of callback storm, no need to wake up too early. */ - if (waketype =3D=3D RCU_NOCB_WAKE_LAZY) { - mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); - } else if (waketype =3D=3D RCU_NOCB_WAKE_BYPASS) { - mod_timer(&rdp_gp->nocb_timer, jiffies + 2); - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); - } else { - if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE) - mod_timer(&rdp_gp->nocb_timer, jiffies + 1); - if (rdp_gp->nocb_defer_wakeup < waketype) - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + switch (waketype) { + case RCU_NOCB_WAKE_LAZY: + mod_jif =3D jiffies_till_flush; + break; + + case RCU_NOCB_WAKE_BYPASS: + mod_jif =3D 2; + break; + + case RCU_NOCB_WAKE: + case RCU_NOCB_WAKE_FORCE: + // If the type of deferred wake is "stronger" + // than it was before, make it wake up the soonest. + if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE) + mod_jif =3D 1; + break; } =20 + if (mod_jif) + mod_timer(&rdp_gp->nocb_timer, jiffies + mod_jif); + + // If new type of wake up is strong than before, promote. + if (rdp_gp->nocb_defer_wakeup < waketype) + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags); =20 trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, reason); --=20 2.37.0.rc0.104.g0611611a94-goog From nobody Mon Apr 20 02:46:43 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94ABCC43334 for ; Wed, 22 Jun 2022 22:52:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376662AbiFVWwX (ORCPT ); Wed, 22 Jun 2022 18:52:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359829AbiFVWvu (ORCPT ); Wed, 22 Jun 2022 18:51:50 -0400 Received: from mail-qv1-xf35.google.com (mail-qv1-xf35.google.com [IPv6:2607:f8b0:4864:20::f35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95C8040918 for ; Wed, 22 Jun 2022 15:51:49 -0700 (PDT) Received: by mail-qv1-xf35.google.com with SMTP id 89so27465367qvc.0 for ; Wed, 22 Jun 2022 15:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FIRVjLUngtzvaOGGJhPX9zs36OvO+dWcUJ0YzPStXJo=; b=RP4Xw5PjLXl1/ttonHyWA6UJqEVlPBaI/VbWuKT2GdpriQ5RbqwHgHOBGOdfOTxTyq TDGa++K8gS/R+55d58vj0W+rk6ukDyJmrWWx/adgrlWS5p6iXgZaCXklmrAQ3/o06bKv sjXJp95JkuuCiMNbBqArpXD5cBjAQs44IfTfg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FIRVjLUngtzvaOGGJhPX9zs36OvO+dWcUJ0YzPStXJo=; b=XLf+XM5wlKNYBqlGrVhHAZ14E8PnykYXYtZdUgKYCivWTCbWTtr9X3OoTsmEMDZSXw 6gT4eY5oGQGmMV3/Ii+Q0VQJ4WaQCIK1WkdwX5D7s4bAvyOQk//MAdSgIos/pDcaZzVr D3e/oJlMlnnUF87EBIbLFAkXybru1Vn2bTll+b6/PK7XCeKsvQpRTb3IZkb06CJrQERE MXvdzH6EPyTxWrwaolNQBvVlWENIGBy+FM4XVX652/e6UKlLDW2B/fPgMFebLUgcsApb Pusb7xyGcXDX+T5bgTnkL+nbl2ge9tV8KAkptb+Zoc4BoDmMVIfGY1b6mKVOwE0Xe+ip Hqog== X-Gm-Message-State: AJIora+YxRDxLgVE0bmTcS0Az+dZehRjUiICWIIOBB6gw34oBhfo7l81 qceFk1F0I1P+UqXxwq89kbZ/uA== X-Google-Smtp-Source: AGRyM1vaA5Ao94/E0rX5gloX6owI1QnEQU1eoreWEcsbtCYin0ct2kXn34bL0xAor+gWulrg/0CnCA== X-Received: by 2002:a05:6214:76b:b0:46e:2704:7c42 with SMTP id f11-20020a056214076b00b0046e27047c42mr30212669qvz.90.1655938308765; Wed, 22 Jun 2022 15:51:48 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id k66-20020a37a145000000b006ab91fd03fasm1794022qke.19.2022.06.22.15.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jun 2022 15:51:48 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, "Joel Fernandes (Google)" Subject: [PATCH v2 8/8] rcu/kfree: Fix kfree_rcu_shrink_count() return value Date: Wed, 22 Jun 2022 22:51:02 +0000 Message-Id: <20220622225102.2112026-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.0.rc0.104.g0611611a94-goog In-Reply-To: <20220622225102.2112026-1-joel@joelfernandes.org> References: <20220622225102.2112026-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" As per the comments in include/linux/shrinker.h, .count_objects callback should return the number of freeable items, but if there are no objects to free, SHRINK_EMPTY should be returned. The only time 0 is returned should be when we are unable to determine the number of objects, or the cache should be skipped for another reason. Signed-off-by: Joel Fernandes (Google) Reviewed-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 711679d10cbb..935788e8d2d7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3722,7 +3722,7 @@ kfree_rcu_shrink_count(struct shrinker *shrink, struc= t shrink_control *sc) atomic_set(&krcp->backoff_page_cache_fill, 1); } =20 - return count; + return count =3D=3D 0 ? SHRINK_EMPTY : count; } =20 static unsigned long --=20 2.37.0.rc0.104.g0611611a94-goog