From nobody Sun Feb 8 14:53:24 2026 Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E60C225F7A5 for ; Wed, 14 Jan 2026 18:34:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768415663; cv=none; b=fWtcBfQCdKEmS/SR+ZUsQ/GD6fwzPlkiDDTeGMCwXjz+K4wc5DmfHTWSI8uWysvz9EMTT4cX7nMR+U9UKjkzQLDB8tErpWMDa6fI/0OYN/VK1tpxo7mHpE1liKmv4N5qvCwj9JqwMAV1bKuywYK62bCkO4nj2B2M6sPZKxZtkkM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768415663; c=relaxed/simple; bh=KZ6jA9OgWKzFSS9WndGq9wCP7OCFnoFubnkro8EsDTg=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=pN3y2ell2yX2QYQjLiBVJrk9G+dqqckJbJVP9UB5moeQIsDhyQFSEvTDeJM0PxT2O9T0fZZQp3qAk8dHwjxlRVPDetlLBbV1YFEbIDz9J9rI2+DmSvVqrdP65sVcf3MZJZuWS0eOqXi97m6hVByIvGo5uIBtc1mCdBF+IVneqcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Zgyp4/8p; arc=none smtp.client-ip=209.85.167.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Zgyp4/8p" Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-59b6df3d6b4so179251e87.0 for ; Wed, 14 Jan 2026 10:34:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1768415658; x=1769020458; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=CUPAgkqqJq6WozfacWaQbbhzVEXuv7nVNXJfVf9PrLw=; b=Zgyp4/8pRu835dOm9qPJBBCWGvOXvmJEpBonkBYboLFOafdiVVvOkDpJxgClgss4is YkwpYNC5nEvyc/erF9dD9upOxN403dMI+ots66OTaJDVsJ+Nk3B42Pf/jvYU49g69wKe OASIQbyNMdbrKWrznZXY2BY69X+XuHpZMby/JzgEXWvm8jpDF2CZiyIhS5WER+sI9ggx YhiappHyES8snkut7VteFjedLGG8XlCte77Xr+ztG7yGUZFZFJ22Zvt78mLE6Bn8RDI8 tloTAL7IaJLYFpilawhffOCK2V0TLNoWJyB4POhgoVXIpjDbKbzOiA6o7LYqpZQja3XA Fvxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768415658; x=1769020458; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=CUPAgkqqJq6WozfacWaQbbhzVEXuv7nVNXJfVf9PrLw=; b=Yi8d50OGUgDORVOSzKX2XoZqxHOTd9LlDYmWIWiiTWYcwg1I5PX1FHKeYCsBIMp1Ag px0PsS+S3WuT9N1bG/Jf6mfeORk8hqon0lSuXIDXTDt4dqS1HgtYZ0lBT3c4CI+7B6hq tYQLFn18xr4INQ5QsNCe6zf/oXD/b2s82vX3S3xbP0uVqbY2HFMxG3fJw1/UXsZnm9uZ P3Kkf6Dw/IijgXs8idLydagXX76eC4JEbbSaitW4lKwepfHTCpHCA4IuR8MtcbH5p6ou sdNq2ZEfOb+cDVueskdPYT/jmqJg4M2poIXYFCvZlqwITI0rSIr48mqLHpsgO7blP3pU A0Qw== X-Forwarded-Encrypted: i=1; AJvYcCXPtEdYAf9VuTE2n7a7MegaaHY9G7blN9sP/hw9HjXcQefFdbc1/Ur8FULHzfPa9H2z3ToG/5gj2kJf3mg=@vger.kernel.org X-Gm-Message-State: AOJu0YzDFkO+DrIizXubF7yww9iEfA6IILGV5Oqu6++11UVlNzG7LeTA xCoNEVXu7FNReaf/+HltjlY3FTqVkXJXSLgOhbDjJdMlMF0w3zce9wh0339i0PLN X-Gm-Gg: AY/fxX4COri5HyYcwiGh86xuXnN2WR5TGQbVp2qUigJHo3JYVOtioYKqZzCLYljeFbw urbYcvabo+V0Ig2ptEzjyZXKuVR6gZG26y7090AaNtd1+2PLgymLRrjPhRz72rqE0zfQwJ7crr4 yssOABLdXqqg2ButTU7BaAbDYZKe8r2c+DcuC5nSDjGmR8isSSjnIzXLVQxZk0n7BVThRQeyNd1 RXs5lwBqAm0O+22rJkCH6MQg7ngfQ4QDJJ3aDSv6iVpkGLZJt5iQ1jCPQ4ZLC9pkRZXQ6trYvoA X/NF5Avt7ipxpc3+aBAy1f3lZH1vah3n+LzHSklCJr/u9bjIrcEOnIlrw6Q27bcCFeRleA08Awl 2Hj51/LbZWSDu/vG2DpgW2O++cLUV2eCdOkactPl8MRl+trta1kWuszeQbE+hXJFytsllEjkeLl +k+Yo= X-Received: by 2002:a05:6512:ad6:b0:598:eed6:4975 with SMTP id 2adb3069b0e04-59ba1752d33mr999594e87.48.1768415657414; Wed, 14 Jan 2026 10:34:17 -0800 (PST) Received: from localhost.localdomain ([2001:9b1:d5a0:a500::24b]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-59ba1045bbdsm917730e87.78.2026.01.14.10.34.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jan 2026 10:34:16 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: "Paul E . McKenney" , Joel Fernandes , Vishal Chourasia , Shrikanth Hegde Cc: Neeraj upadhyay , RCU , LKML , Frederic Weisbecker , Uladzislau Rezki Subject: [PATCH] rcu: Latch normal synchronize_rcu() path on flood Date: Wed, 14 Jan 2026 19:34:15 +0100 Message-ID: <20260114183415.286489-1-urezki@gmail.com> X-Mailer: git-send-email 2.47.3 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, rcu_normal_wake_from_gp is only enabled by default on small systems(<=3D 16 CPUs) or when a user explicitly set it enabled. This patch introduces an adaptive latching mechanism: * Tracks the number of in-flight synchronize_rcu() requests using a new atomic_t counter(rcu_sr_normal_count); * If the count exceeds RCU_SR_NORMAL_LATCH_THR(64), it sets the rcu_sr_normal_latched, reverting new requests onto the scaled wait_rcu_gp() path; * The latch is cleared only when the pending requests are fully drained(nr =3D=3D 0); * Enables rcu_normal_wake_from_gp by default for all systems, relying on this dynamic throttling instead of static CPU limits. Suggested-by: Joel Fernandes Signed-off-by: Uladzislau Rezki (Sony) Tested-by: Samir M --- kernel/rcu/tree.c | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 293bbd9ac3f4..c42d480d6e0b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1631,17 +1631,21 @@ static void rcu_sr_put_wait_head(struct llist_node = *node) atomic_set_release(&sr_wn->inuse, 0); } =20 -/* Enable rcu_normal_wake_from_gp automatically on small systems. */ -#define WAKE_FROM_GP_CPU_THRESHOLD 16 - -static int rcu_normal_wake_from_gp =3D -1; +static int rcu_normal_wake_from_gp =3D 1; module_param(rcu_normal_wake_from_gp, int, 0644); static struct workqueue_struct *sync_wq; =20 +#define RCU_SR_NORMAL_LATCH_THR 64 + +/* Number of in-flight synchronize_rcu() calls queued on srs_next. */ +static atomic_long_t rcu_sr_normal_count; +static atomic_t rcu_sr_normal_latched; + static void rcu_sr_normal_complete(struct llist_node *node) { struct rcu_synchronize *rs =3D container_of( (struct rcu_head *) node, struct rcu_synchronize, head); + long nr; =20 WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && !poll_state_synchronize_rcu_full(&rs->oldstate), @@ -1649,6 +1653,15 @@ static void rcu_sr_normal_complete(struct llist_node= *node) =20 /* Finally. */ complete(&rs->completion); + nr =3D atomic_long_dec_return(&rcu_sr_normal_count); + WARN_ON_ONCE(nr < 0); + + /* + * Unlatch: switch back to normal path when fully + * drained and if it has been latched. + */ + if (nr =3D=3D 0) + (void)atomic_cmpxchg(&rcu_sr_normal_latched, 1, 0); } =20 static void rcu_sr_normal_gp_cleanup_work(struct work_struct *work) @@ -1794,7 +1807,14 @@ static bool rcu_sr_normal_gp_init(void) =20 static void rcu_sr_normal_add_req(struct rcu_synchronize *rs) { + long nr; + llist_add((struct llist_node *) &rs->head, &rcu_state.srs_next); + nr =3D atomic_long_inc_return(&rcu_sr_normal_count); + + /* Latch: only when flooded and if unlatched. */ + if (nr >=3D RCU_SR_NORMAL_LATCH_THR) + (void)atomic_cmpxchg(&rcu_sr_normal_latched, 0, 1); } =20 /* @@ -3268,7 +3288,8 @@ static void synchronize_rcu_normal(void) =20 trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request")); =20 - if (READ_ONCE(rcu_normal_wake_from_gp) < 1) { + if (READ_ONCE(rcu_normal_wake_from_gp) < 1 || + atomic_read(&rcu_sr_normal_latched)) { wait_rcu_gp(call_rcu_hurry); goto trace_complete_out; } @@ -4892,12 +4913,6 @@ void __init rcu_init(void) sync_wq =3D alloc_workqueue("sync_wq", WQ_MEM_RECLAIM | WQ_UNBOUND, 0); WARN_ON(!sync_wq); =20 - /* Respect if explicitly disabled via a boot parameter. */ - if (rcu_normal_wake_from_gp < 0) { - if (num_possible_cpus() <=3D WAKE_FROM_GP_CPU_THRESHOLD) - rcu_normal_wake_from_gp =3D 1; - } - /* Fill in default value for rcutree.qovld boot parameter. */ /* -After- the rcu_node ->lock fields are initialized! */ if (qovld < 0) --=20 2.47.3