From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A6E9ECAAD2 for ; Thu, 1 Sep 2022 22:18:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233686AbiIAWSH (ORCPT ); Thu, 1 Sep 2022 18:18:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234299AbiIAWSA (ORCPT ); Thu, 1 Sep 2022 18:18:00 -0400 Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D44A58091 for ; Thu, 1 Sep 2022 15:17:59 -0700 (PDT) Received: by mail-qv1-xf32.google.com with SMTP id l5so143142qvs.13 for ; Thu, 01 Sep 2022 15:17:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=+2CQoWOzORwmFm7W8hobFnBssU2v8aOfIo1J22tLn5A=; b=gYFVtvH6IT88YOW+VuemDQdOw9jN1vrqUhqD4F59DGDWIburm49TpkXGVQY87YJ84L qH6wa/UO5erX3ioCJY2/tLyTnBhtQ1+KSn2ciZl3SpPmczkva7MHX/hbiZhzntNqv0Q4 KArjDPxW26NuxuN0qO1Zj5kVlXjgrE+8nGLpo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=+2CQoWOzORwmFm7W8hobFnBssU2v8aOfIo1J22tLn5A=; b=siDCCNGXXe00suQrnc4aSn0ZAQAyOaR7STGYwzhmn0/bq+PxiL3NJv0FJqm/wtXYe7 JyCVaTaACoX19TxXXGD8Pe3ivwWLTvOJ6vzAh7TQKyRUWNO3+s8WZ/azQJX7IvxY6ffK iBvKygp2hw2UTVZpeZeUf/Q9D+b9j4S87IAtSrcw/WrucSNCahyrPIpr/kBkqt52re73 7LLwj5GmSZ5qktF+2Ibw5i/RmlHNUorR2NnZN5ZfJdVAxar4WA5RIAkYtxIdHV38ibQn sW0PKa6kVZBuC7DJSIXaDx0nudHR7WkI/xc2HS5KdrBw1v/xZzrEyYwsO0B8XCuHAF1M TTGw== X-Gm-Message-State: ACgBeo1zKQ+LPw0AIRwKkeS3RC7AnCDDWnA61JoskztuQ3D9xSfWxjYA HRmNGgTwO/aev2LUOvUUKD/7xi8m3NeD6Q== X-Google-Smtp-Source: AA6agR4kxLtEYufGPqZXnLYzR+tmjfowzJgSuRNILICeEUEdChLKGM9hmPATH/Xdqmcnc+IW5AQBNw== X-Received: by 2002:a05:6214:2a4b:b0:499:eb9:6ed3 with SMTP id jf11-20020a0562142a4b00b004990eb96ed3mr14878236qvb.66.1662070678188; Thu, 01 Sep 2022 15:17:58 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:57 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, Vlastimil Babka Subject: [PATCH v5 01/18] mm/slub: perform free consistency checks before call_rcu Date: Thu, 1 Sep 2022 22:17:03 +0000 Message-Id: <20220901221720.1105021-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Vlastimil Babka For SLAB_TYPESAFE_BY_RCU caches we use call_rcu to perform empty slab freeing. The rcu callback rcu_free_slab() calls __free_slab() that currently includes checking the slab consistency for caches with SLAB_CONSISTENCY_CHECKS flags. This check needs the slab->objects field to be intact. Because in the next patch we want to allow rcu_head in struct slab to become larger in debug configurations and thus potentially overwrite more fields through a union than slab_list, we want to limit the fields used in rcu_free_slab(). Thus move the consistency checks to free_slab() before call_rcu(). This can be done safely even for SLAB_TYPESAFE_BY_RCU caches where accesses to the objects can still occur after freeing them. As a result, only the slab->slab_cache field has to be physically separate from rcu_head for the freeing callback to work. We also save some cycles in the rcu callback for caches with consistency checks enabled. Signed-off-by: Vlastimil Babka --- mm/slub.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 862dbd9af4f5..d86be1b0d09f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2036,14 +2036,6 @@ static void __free_slab(struct kmem_cache *s, struct= slab *slab) int order =3D folio_order(folio); int pages =3D 1 << order; =20 - if (kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) { - void *p; - - slab_pad_check(s, slab); - for_each_object(p, s, slab_address(slab), slab->objects) - check_object(s, slab, p, SLUB_RED_INACTIVE); - } - __slab_clear_pfmemalloc(slab); __folio_clear_slab(folio); folio->mapping =3D NULL; @@ -2062,9 +2054,17 @@ static void rcu_free_slab(struct rcu_head *h) =20 static void free_slab(struct kmem_cache *s, struct slab *slab) { - if (unlikely(s->flags & SLAB_TYPESAFE_BY_RCU)) { + if (kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) { + void *p; + + slab_pad_check(s, slab); + for_each_object(p, s, slab_address(slab), slab->objects) + check_object(s, slab, p, SLUB_RED_INACTIVE); + } + + if (unlikely(s->flags & SLAB_TYPESAFE_BY_RCU)) call_rcu(&slab->rcu_head, rcu_free_slab); - } else + else __free_slab(s, slab); } =20 --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 687ECECAAD3 for ; Thu, 1 Sep 2022 22:18:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234706AbiIAWSK (ORCPT ); Thu, 1 Sep 2022 18:18:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234344AbiIAWSA (ORCPT ); Thu, 1 Sep 2022 18:18:00 -0400 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3AF95809E for ; Thu, 1 Sep 2022 15:17:59 -0700 (PDT) Received: by mail-qt1-x833.google.com with SMTP id j17so205318qtp.12 for ; Thu, 01 Sep 2022 15:17:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=ocxsdgSxr9WyJwVb16iinpfYqPxMSVghGuyFQwZXQfs=; b=ti/1WG1u8VpXWufM64ni12DTreKGCzniiXPz9JlY5rrgucCra9WVKDdRPJHpbwSpxj iG++ZS/nv4W4EuHP+8BlbKh7cvUFrg0n06h/UoIiKDHin6Qnra6rJCmg3+AA9tjOu7oI ZELEHVazqBBYH+5PbuvWb3Wxw/KtdCPykBIsE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=ocxsdgSxr9WyJwVb16iinpfYqPxMSVghGuyFQwZXQfs=; b=nXGVQGljQ83KZIHaCZ4I85MTpji7DO1HIbGpBdYgVJLh7DZcZMbE+uSY+BZryN+2Nu y0dMwME2u2MwZxryTBi/YPAPtn9SB9QzZ6+Wf9sJ56hb+Zj+orHICgsIM0kJa72G5fKa di272phBRzYke+j61PyER7tFy3stOFx/vwwqnRcVxmuxvje7xFFdV+ypnNhNitdW0OV6 0//7UkSKs1zPoelE9XHtuxhoU7AN5i2Ys+WnMd9+4VAEPb8qgP2L1Zu015Y0L4g2No0l WZ33C0kPnrLPW/4nZsWFzlfK7oCMCuxZxpX5kWLNEWZdNEPweVdaikNYxit3bNQOd6Ax eMjg== X-Gm-Message-State: ACgBeo0QUI7VHcUhbxuQt+IoQC0pzRz80CCtuy6bkLAiF86Qp3qPrbxv ZZzFtF3OBEdURO+/aD7jmpJk2g== X-Google-Smtp-Source: AA6agR6eIoBqsq90vcCegOqFmOL0xyor84GJfMSmDM+50XbL31b6MpOWJKpxG/NZU2dOPdvIGdA/1Q== X-Received: by 2002:ac8:5f53:0:b0:343:a8d0:c81b with SMTP id y19-20020ac85f53000000b00343a8d0c81bmr25798028qta.489.1662070678847; Thu, 01 Sep 2022 15:17:58 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:58 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, Vlastimil Babka , Joel Fernandes Subject: [PATCH v5 02/18] mm/sl[au]b: rearrange struct slab fields to allow larger rcu_head Date: Thu, 1 Sep 2022 22:17:04 +0000 Message-Id: <20220901221720.1105021-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Vlastimil Babka Joel reports [1] that increasing the rcu_head size for debugging purposes used to work before struct slab was split from struct page, but now runs into the various SLAB_MATCH() sanity checks of the layout. This is because the rcu_head in struct page is in union with large sub-structures and has space to grow without exceeding their size, while in struct slab (for SLAB and SLUB) it's in union only with a list_head. On closer inspection (and after the previous patch) we can put all fields except slab_cache to a union with rcu_head, as slab_cache is sufficient for the rcu freeing callbacks to work and the rest can be overwritten by rcu_head without causing issues. This is only somewhat complicated by the need to keep SLUB's freelist+counters aligned for cmpxchg_double. As a result the fields need to be reordered so that slab_cache is first (after page flags) and the union with rcu_head follows. For consistency, do that for SLAB as well, although not necessary there. As a result, the rcu_head field in struct page and struct slab is no longer at the same offset, but that doesn't matter as there is no casting that would rely on that in the slab freeing callbacks, so we can just drop the respective SLAB_MATCH() check. Also we need to update the SLAB_MATCH() for compound_head to reflect the new ordering. While at it, also add a static_assert to check the alignment needed for cmpxchg_double so mistakes are found sooner than a runtime GPF. [1] https://lore.kernel.org/all/85afd876-d8bb-0804-b2c5-48ed3055e702@joelfe= rnandes.org/ Reported-by: Joel Fernandes Signed-off-by: Vlastimil Babka Acked-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slab.h | 54 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 4ec82bec15ec..2c248864ea91 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -11,37 +11,43 @@ struct slab { =20 #if defined(CONFIG_SLAB) =20 + struct kmem_cache *slab_cache; union { - struct list_head slab_list; + struct { + struct list_head slab_list; + void *freelist; /* array of free object indexes */ + void *s_mem; /* first object */ + }; struct rcu_head rcu_head; }; - struct kmem_cache *slab_cache; - void *freelist; /* array of free object indexes */ - void *s_mem; /* first object */ unsigned int active; =20 #elif defined(CONFIG_SLUB) =20 - union { - struct list_head slab_list; - struct rcu_head rcu_head; -#ifdef CONFIG_SLUB_CPU_PARTIAL - struct { - struct slab *next; - int slabs; /* Nr of slabs left */ - }; -#endif - }; struct kmem_cache *slab_cache; - /* Double-word boundary */ - void *freelist; /* first free object */ union { - unsigned long counters; struct { - unsigned inuse:16; - unsigned objects:15; - unsigned frozen:1; + union { + struct list_head slab_list; +#ifdef CONFIG_SLUB_CPU_PARTIAL + struct { + struct slab *next; + int slabs; /* Nr of slabs left */ + }; +#endif + }; + /* Double-word boundary */ + void *freelist; /* first free object */ + union { + unsigned long counters; + struct { + unsigned inuse:16; + unsigned objects:15; + unsigned frozen:1; + }; + }; }; + struct rcu_head rcu_head; }; unsigned int __unused; =20 @@ -66,9 +72,10 @@ struct slab { #define SLAB_MATCH(pg, sl) \ static_assert(offsetof(struct page, pg) =3D=3D offsetof(struct slab, sl)) SLAB_MATCH(flags, __page_flags); -SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */ #ifndef CONFIG_SLOB -SLAB_MATCH(rcu_head, rcu_head); +SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ +#else +SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */ #endif SLAB_MATCH(_refcount, __page_refcount); #ifdef CONFIG_MEMCG @@ -76,6 +83,9 @@ SLAB_MATCH(memcg_data, memcg_data); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <=3D sizeof(struct page)); +#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && defined(CONFIG_SLUB) +static_assert(IS_ALIGNED(offsetof(struct slab, freelist), 16)); +#endif =20 /** * folio_slab - Converts from folio to slab. --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A6AEECAAD3 for ; Thu, 1 Sep 2022 22:18:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234747AbiIAWSQ (ORCPT ); Thu, 1 Sep 2022 18:18:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234381AbiIAWSB (ORCPT ); Thu, 1 Sep 2022 18:18:01 -0400 Received: from mail-qv1-xf2c.google.com (mail-qv1-xf2c.google.com [IPv6:2607:f8b0:4864:20::f2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59AF8543C2 for ; Thu, 1 Sep 2022 15:18:00 -0700 (PDT) Received: by mail-qv1-xf2c.google.com with SMTP id q8so157347qvr.9 for ; Thu, 01 Sep 2022 15:18:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=wacV+wNWevSZZedckm8hHMeO8v0HhmKZhpXkr12CWiU=; b=e7rfmVttSIAE2dX9U/UrdKI2/QNY3ZP3fKeyJGley56wIpwe2qS6saz3KZesypfmqe hM1riUNN/4tHj4mrt7IXt6dpL2LLv2du/nCXvZZI8MdoBiSD0/AnUB8528he0Bh8pZQV vrMkt1S957E51Pelj+JfGSpSAlfQVDY8utxxc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=wacV+wNWevSZZedckm8hHMeO8v0HhmKZhpXkr12CWiU=; b=QHryVwDuDQby4iBBA1Jg0/KHBxd/gRq++7UpJVZaN3xibzy0LJPBId0md/p//8kPbi ZMIR7ebkh1IsmshbVVawQetj/xvm45BJr/B927W/hYWyufWzohjpkmJ3eQp6b9Yn0JE4 cmWlj0RL8NJEYDTaE8yIn2aGJjTqqZp4+L+H4CozKI2kRxIxZ5NjUXDfdSTZWzfKjkc9 AYBZuW4LIhNyDg/aLf/x+A3N2vnOkW1iPnOZAL46t9S2nnbnz98q47POu58lB42bNeeC fM3IzB3AQnm8tz8T1X6XpVdI+NLPxIPRqx8bPeysw3C8paChf0I7spGBiHhDy1MMVg6E l+Zg== X-Gm-Message-State: ACgBeo2olqCcaizUI+k8VYeDCnoVgPefncMtwRG+9IILON1pu9bVpMkv lRidM825PRA9jJ9yIy6LOp+vww== X-Google-Smtp-Source: AA6agR7ebVdicXTu0lYFUslN4hqrp3yrx/UXs4hw278sz5joSWluao70f6a70ft5FQ57ZlDUz+4CoA== X-Received: by 2002:a05:6214:27c1:b0:473:85c9:3eeb with SMTP id ge1-20020a05621427c100b0047385c93eebmr26877303qvb.53.1662070679524; Thu, 01 Sep 2022 15:17:59 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:59 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 03/18] rcu/tree: Use READ_ONCE() for lockless read of rnp->qsmask Date: Thu, 1 Sep 2022 22:17:05 +0000 Message-Id: <20220901221720.1105021-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The rnp->qsmask is locklessly accessed from rcutree_dying_cpu(). This may help avoid load/store tearing due to concurrent access, KCSAN issues, and preserve sanity of people reading the mask in tracing. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0ca21ac0f064..5ec97e3f7468 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2106,7 +2106,7 @@ int rcutree_dying_cpu(unsigned int cpu) if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) return 0; =20 - blkd =3D !!(rnp->qsmask & rdp->grpmask); + blkd =3D !!(READ_ONCE(rnp->qsmask) & rdp->grpmask); trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq), blkd ? TPS("cpuofl-bgp") : TPS("cpuofl")); return 0; --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB5F8ECAAD3 for ; Thu, 1 Sep 2022 22:18:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233374AbiIAWSd (ORCPT ); Thu, 1 Sep 2022 18:18:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234469AbiIAWSC (ORCPT ); Thu, 1 Sep 2022 18:18:02 -0400 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4074F58091 for ; Thu, 1 Sep 2022 15:18:01 -0700 (PDT) Received: by mail-qt1-x82a.google.com with SMTP id l5so238841qtv.4 for ; Thu, 01 Sep 2022 15:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=8tBDdZaeoT3b5CBzqvW/ZB2aZTVcUlmvdmlRm1sAyM4=; b=QBkSAea3WwRWmhiRWRuvpOfEw+FTTLufqMSEdc1gAtyAq+xBunox+Nuh8C6IUqQDo1 +LxFhVcyiQIx4jvB2azfPR7AmGcN5zZvubRtp5adhPS46cT5+VACoX35Em9rNj9ukK+M L11Wz8mOl+0mNcbPFv1bMoPDD7RhIAz59Gntw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8tBDdZaeoT3b5CBzqvW/ZB2aZTVcUlmvdmlRm1sAyM4=; b=tibIpqqbxmT2x6BuQQK1w54RI93xQjWD/EZSBoX8EmbEOVeRyVMzZkBMQSNbBrZeg5 dGhdZTPk+TkUs/5tlw4iK33GxQXGFegz4DSiGoI9bisVixAZt12K95i07qiynJmG4L/I 2BX1IIbPwmy6YpnLPCMH68iHAJrEWrbR9dZeWhycD9EYWabK5ua6knEHSP58QbFiyIf+ 8AJkf2TeKEMou9jhCpZi4ZTflvLGhDy2PmjgHv6lpGBUZD9S1s9Yvnw4wxmGp/RiRiRV y34tPepGSiyKafsRByvk6SzI2Hut8wrDG7UxILxARb7UTkT0Cj061UGznewUp1DzOmXi /xrg== X-Gm-Message-State: ACgBeo2fNhQ2262CYFWGr9CNO6+7iPNXh9x4B0fGOc6WgUXwiz3Ex3Cd bhmx5PZ0M70XD79uhgOmzNELI+M9Ma1oFw== X-Google-Smtp-Source: AA6agR4RLjnlk0HhdTMZyE6uJDeBeCmArIKBraVqa1sV/k/4xV1xpnGzfyzL2EqEG8NqbDqWrq33bg== X-Received: by 2002:a05:622a:649:b0:343:7b74:5cd1 with SMTP id a9-20020a05622a064900b003437b745cd1mr26095535qtb.407.1662070680221; Thu, 01 Sep 2022 15:18:00 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.17.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:17:59 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 04/18] rcu: Fix late wakeup when flush of bypass cblist happens Date: Thu, 1 Sep 2022 22:17:06 +0000 Message-Id: <20220901221720.1105021-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush happens. Otherwise, the lazy-timer will go off much later and the now-non-lazy cblist CBs can get stranded for the duration of the timer. This is a good thing to do anyway (regardless of this series), since it makes the behavior consistent with behavior of other code paths where queue= ing something into the ->cblist makes the GP kthread in a non-sleeping state quickly. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 0a5f0ef41484..31068dd31315 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -447,7 +447,13 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, = struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time =3D j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the ->cblist. + // However, the bypass timer might still be running. Wakeup the + // GP thread by calling a helper with was_all_done set so that + // wake up happens (needed if main CB list was empty before). + __call_rcu_nocb_wake(rdp, true, flags) + return true; // Callback already enqueued. } =20 --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62DFBECAAD3 for ; Thu, 1 Sep 2022 22:18:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234846AbiIAWS1 (ORCPT ); Thu, 1 Sep 2022 18:18:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234435AbiIAWSC (ORCPT ); Thu, 1 Sep 2022 18:18:02 -0400 Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F7E55809E for ; Thu, 1 Sep 2022 15:18:01 -0700 (PDT) Received: by mail-qv1-xf32.google.com with SMTP id x14so168373qvr.6 for ; Thu, 01 Sep 2022 15:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=JOSQJQVjQiQE0lsAHsymw5P/1/LCdxuxYv3QrWVdzDXjflavEeAYjmd5/uZ37cCbmF zhBGTGNbXYEWseXrqhdIgH5eL3LYuesO89moXDEXUlnu/SLgEN2N3XUtV/AIkJ3Zw4BO xRsqlByOAsBzFkWDPtCz2rpRDgUAwAOEujFyU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=etB/HfOTcWmHsF5CbpNyxWl1M85+w49iAJX0vrPF/XCsKofwNx81Cl/inpydEofLNp Ka3k7jea7SAnY6PYir91r+in9Zi2A6EnjAveQkkYt7NvQvCDT8nY6erx20ZgpJS7D0Ps oFQ0LDlZK5dnHly4qkgor5gabUrnS8GEjta7UWKiqLd5MpJKtSp9DZIPJ6i4OVSBVZ03 rxwCGycx4NgwW3tJnXHrWJh12aHj/QPJTe8/IhwI8dwGVrMf8pMbzNgDaDPNzST6/vz0 qN9/pMG6p9Be1AM2fctMV/GxKldiPGX1f/yWkfqWT3SrmftTmM4OsNkzJGxU8M8ZwhzZ QpJA== X-Gm-Message-State: ACgBeo3yBjuGnrYkJ00sTrFTyfj/GeX90qOl3uEBaQ6b/146pMQiPqSJ ypQaivMxnjV4d/UTvePT/184WQ== X-Google-Smtp-Source: AA6agR5R0KAg3JQMxVIXb/rgAzL7TYmscgkTXfoxMN9Q5ya69Oe6nssTMAHS9mjy5O7k72Bf0BpSEg== X-Received: by 2002:a05:6214:29cc:b0:477:38c2:105e with SMTP id gh12-20020a05621429cc00b0047738c2105emr26673145qvb.93.1662070680808; Thu, 01 Sep 2022 15:18:00 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:00 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 05/18] rcu: Move trace_rcu_callback() before bypassing Date: Thu, 1 Sep 2022 22:17:07 +0000 Message-Id: <20220901221720.1105021-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If any CB is queued into the bypass list, then trace_rcu_callback() does not show it. This makes it not clear when a callback was actually queued, as you only end up getting a trace_rcu_invoke_callback() trace. Fix it by moving trace_rcu_callback() before trace_rcu_nocb_try_bypass(). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5ec97e3f7468..9fe581be8696 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2809,10 +2809,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t = func) } =20 check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) - return; // Enqueued onto ->nocb_bypass, so just leave. - // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. - rcu_segcblist_enqueue(&rdp->cblist, head); + if (__is_kvfree_rcu_offset((unsigned long)func)) trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, @@ -2821,6 +2818,11 @@ void call_rcu(struct rcu_head *head, rcu_callback_t = func) trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); =20 + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + return; // Enqueued onto ->nocb_bypass, so just leave. + // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. + rcu_segcblist_enqueue(&rdp->cblist, head); + trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); =20 /* Go handle any RCU core processing required. */ --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 494FFECAAD3 for ; Thu, 1 Sep 2022 22:18:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234938AbiIAWSt (ORCPT ); Thu, 1 Sep 2022 18:18:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233034AbiIAWSF (ORCPT ); Thu, 1 Sep 2022 18:18:05 -0400 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C12E958091 for ; Thu, 1 Sep 2022 15:18:02 -0700 (PDT) Received: by mail-qv1-xf33.google.com with SMTP id x14so168394qvr.6 for ; Thu, 01 Sep 2022 15:18:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=PromVDxaTL5WRddWriuWKU7Rbkxnb5+n/4/JObcP9Ec=; b=q2+8jSlhJdEfVIzEptXiAisnxq1C3aFrZ6nAKdtNOLjISF2DyKf8rkqBCnHXLKiIPn ZqNhjLktTPc+ksu7IOUXfo4uKhHokkVZPpP35Bpt5qEnMGtzr/Px+VpXfgqxhV7ehd7Q a0KiUP7IlfukMeOmvA7oazBvmh8t9k2K8ZHpo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=PromVDxaTL5WRddWriuWKU7Rbkxnb5+n/4/JObcP9Ec=; b=VDNCoVwrPSsoegDMwUKS0DreqpuY7HIVu7XjMpK7kJ0GdDOravvJmyuKSlsJF8joA3 HM+QqVHHbFsjcq67z9KMwAOYVDLSOYav6CTbuPLxU8gYzLORYtBN8fYYJGMgFDETqnBK 3ieuJ9vPejYhzTqPnR9ZXWnafDZ4B9uXKgiGDMT7BDt+xlKJEjWf6YE6tXs1UPkjiC02 wOAh0HKPZq16hVdDIt80XnuStRY6xbwWhglK6NvnFfzz2VBfNmZaVZx1n2YsfFtSLNDl 3Y1WNJpCBjP8aZhyCmF88AvmBJ6vuq1dnyfiPXCBfgV85TzgKe8GFUl7zJ4HN3ypJqnb 6CSw== X-Gm-Message-State: ACgBeo0FkmdwEe/lEGm/vAsFT1MzZzcDcxZYCVHD8rzIRWZOfMMNmie+ R0Lp+bCubqm94+3Ce7BqVxNxhg== X-Google-Smtp-Source: AA6agR7O1opNtswPRY5hmUZ4DpbRs5kCINsqiSaFc91LD+aZxx/8YeroeDAhJbG76N+5kQjZfVpHDw== X-Received: by 2002:a05:6214:2aa2:b0:477:1882:3dc with SMTP id js2-20020a0562142aa200b00477188203dcmr26038206qvb.11.1662070681727; Thu, 01 Sep 2022 15:18:01 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:01 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 06/18] rcu: Introduce call_rcu_lazy() API implementation Date: Thu, 1 Sep 2022 22:17:08 +0000 Message-Id: <20220901221720.1105021-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Implement timer-based RCU lazy callback batching. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure will flush it in a future patch. To handle several corner cases automagically (such as rcu_barrier() and hotplug), we re-use bypass lists to handle lazy CBs. The bypass list length has the lazy CB length included in it. A separate lazy CB length counter is also introduced to keep track of the number of lazy CBs. Suggested-by: Paul McKenney Signed-off-by: Joel Fernandes (Google) Reported-by: Frederic Weisbecker --- include/linux/rcupdate.h | 6 ++ kernel/rcu/Kconfig | 8 ++ kernel/rcu/rcu.h | 11 +++ kernel/rcu/rcu_segcblist.c | 2 +- kernel/rcu/tree.c | 130 +++++++++++++++--------- kernel/rcu/tree.h | 13 ++- kernel/rcu/tree_nocb.h | 198 ++++++++++++++++++++++++++++++------- 7 files changed, 280 insertions(+), 88 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 08605ce7379d..82e8a07e0856 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -108,6 +108,12 @@ static inline int rcu_preempt_depth(void) =20 #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ =20 +#ifdef CONFIG_RCU_LAZY +void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func); +#else +#define call_rcu_lazy(head, func) call_rcu(head, func) +#endif + /* Internal to kernel */ void rcu_init(void); extern int rcu_scheduler_active; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index d471d22a5e21..3128d01427cb 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -311,4 +311,12 @@ config TASKS_TRACE_RCU_READ_MB Say N here if you hate read-side memory barriers. Take the default if you are unsure. =20 +config RCU_LAZY + bool "RCU callback lazy invocation functionality" + depends on RCU_NOCB_CPU + default n + help + To save power, batch RCU callbacks and flush after delay, memory + pressure or callback list growing too big. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index be5979da07f5..94675f14efe8 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -474,6 +474,14 @@ enum rcutorture_type { INVALID_RCU_FLAVOR }; =20 +#if defined(CONFIG_RCU_LAZY) +unsigned long rcu_lazy_get_jiffies_till_flush(void); +void rcu_lazy_set_jiffies_till_flush(unsigned long j); +#else +static inline unsigned long rcu_lazy_get_jiffies_till_flush(void) { return= 0; } +static inline void rcu_lazy_set_jiffies_till_flush(unsigned long j) { } +#endif + #if defined(CONFIG_TREE_RCU) void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq); @@ -483,6 +491,8 @@ void do_trace_rcu_torture_read(const char *rcutorturena= me, unsigned long c_old, unsigned long c); void rcu_gp_set_torture_wait(int duration); +void rcu_force_call_rcu_to_lazy(bool force); + #else static inline void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq) @@ -501,6 +511,7 @@ void do_trace_rcu_torture_read(const char *rcutorturena= me, do { } while (0) #endif static inline void rcu_gp_set_torture_wait(int duration) { } +static inline void rcu_force_call_rcu_to_lazy(bool force) { } #endif =20 #if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) || IS_MODULE(CONFIG_RCU_TORTURE_TE= ST) diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c index c54ea2b6a36b..55b50e592986 100644 --- a/kernel/rcu/rcu_segcblist.c +++ b/kernel/rcu/rcu_segcblist.c @@ -38,7 +38,7 @@ void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct r= cu_head *rhp) * element of the second rcu_cblist structure, but ensuring that the second * rcu_cblist structure, if initially non-empty, always appears non-empty * throughout the process. If rdp is NULL, the second rcu_cblist structure - * is instead initialized to empty. + * is instead initialized to empty. Also account for lazy_len for lazy CBs. */ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, struct rcu_cblist *srclp, diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 9fe581be8696..aaced29a0a71 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2728,47 +2728,8 @@ static void check_cb_ovld(struct rcu_data *rdp) raw_spin_unlock_rcu_node(rnp); } =20 -/** - * call_rcu() - Queue an RCU callback for invocation after a grace period. - * @head: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a full grace - * period elapses, in other words after all pre-existing RCU read-side - * critical sections have completed. However, the callback function - * might well execute concurrently with RCU read-side critical sections - * that started after call_rcu() was invoked. - * - * RCU read-side critical sections are delimited by rcu_read_lock() - * and rcu_read_unlock(), and may be nested. In addition, but only in - * v5.0 and later, regions of code across which interrupts, preemption, - * or softirqs have been disabled also serve as RCU read-side critical - * sections. This includes hardware interrupt handlers, softirq handlers, - * and NMI handlers. - * - * Note that all CPUs must agree that the grace period extended beyond - * all pre-existing RCU read-side critical section. On systems with more - * than one CPU, this means that when "func()" is invoked, each CPU is - * guaranteed to have executed a full memory barrier since the end of its - * last RCU read-side critical section whose beginning preceded the call - * to call_rcu(). It also means that each CPU executing an RCU read-side - * critical section that continues beyond the start of "func()" must have - * executed a memory barrier after the call_rcu() but before the beginning - * of that RCU read-side critical section. Note that these guarantees - * include CPUs that are offline, idle, or executing in user mode, as - * well as CPUs that are executing in the kernel. - * - * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the - * resulting RCU callback function "func()", then both CPU A and CPU B are - * guaranteed to execute a full memory barrier during the time interval - * between the call to call_rcu() and the invocation of "func()" -- even - * if CPU A and CPU B are the same CPU (but again only if the system has - * more than one CPU). - * - * Implementation of these memory-ordering guarantees is described here: - * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. - */ -void call_rcu(struct rcu_head *head, rcu_callback_t func) +static void +__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) { static atomic_t doublefrees; unsigned long flags; @@ -2818,7 +2779,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t f= unc) trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); =20 - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); @@ -2833,8 +2794,86 @@ void call_rcu(struct rcu_head *head, rcu_callback_t = func) local_irq_restore(flags); } } -EXPORT_SYMBOL_GPL(call_rcu); =20 +#ifdef CONFIG_RCU_LAZY +/** + * call_rcu_lazy() - Lazily queue RCU callback for invocation after grace = period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. + * + * Use this API instead of call_rcu() if you don't mind the callback being + * invoked after very long periods of time on systems without memory press= ure + * and on systems which are lightly loaded or mostly idle. + * + * Other than the extra delay in callbacks being invoked, this function is + * identical to, and reuses call_rcu()'s logic. Refer to call_rcu() for mo= re + * details about memory ordering and other functionality. + */ +void call_rcu_lazy(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, true); +} +EXPORT_SYMBOL_GPL(call_rcu_lazy); +#endif + +static bool force_call_rcu_to_lazy; + +void rcu_force_call_rcu_to_lazy(bool force) +{ + if (IS_ENABLED(CONFIG_RCU_SCALE_TEST)) + WRITE_ONCE(force_call_rcu_to_lazy, force); +} +EXPORT_SYMBOL_GPL(rcu_force_call_rcu_to_lazy); + +/** + * call_rcu() - Queue an RCU callback for invocation after a grace period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. However, the callback function + * might well execute concurrently with RCU read-side critical sections + * that started after call_rcu() was invoked. + * + * RCU read-side critical sections are delimited by rcu_read_lock() + * and rcu_read_unlock(), and may be nested. In addition, but only in + * v5.0 and later, regions of code across which interrupts, preemption, + * or softirqs have been disabled also serve as RCU read-side critical + * sections. This includes hardware interrupt handlers, softirq handlers, + * and NMI handlers. + * + * Note that all CPUs must agree that the grace period extended beyond + * all pre-existing RCU read-side critical section. On systems with more + * than one CPU, this means that when "func()" is invoked, each CPU is + * guaranteed to have executed a full memory barrier since the end of its + * last RCU read-side critical section whose beginning preceded the call + * to call_rcu(). It also means that each CPU executing an RCU read-side + * critical section that continues beyond the start of "func()" must have + * executed a memory barrier after the call_rcu() but before the beginning + * of that RCU read-side critical section. Note that these guarantees + * include CPUs that are offline, idle, or executing in user mode, as + * well as CPUs that are executing in the kernel. + * + * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the + * resulting RCU callback function "func()", then both CPU A and CPU B are + * guaranteed to execute a full memory barrier during the time interval + * between the call to call_rcu() and the invocation of "func()" -- even + * if CPU A and CPU B are the same CPU (but again only if the system has + * more than one CPU). + * + * Implementation of these memory-ordering guarantees is described here: + * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. + */ +void call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, force_call_rcu_to_lazy); +} +EXPORT_SYMBOL_GPL(call_rcu); =20 /* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (5 * HZ) @@ -3904,7 +3943,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func =3D rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, + /* wake gp thread */ true)); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -4325,7 +4365,7 @@ void rcutree_migrate_callbacks(int cpu) my_rdp =3D this_cpu_ptr(&rcu_data); my_rnp =3D my_rdp->mynode; rcu_nocb_lock(my_rdp); /* irqs already disabled. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies, false, false)); raw_spin_lock_rcu_node(my_rnp); /* irqs already disabled. */ /* Leverage recent GPs and set GP for new callbacks. */ needwake =3D rcu_advance_cbs(my_rnp, rdp) || diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c..946d819b23fc 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -263,14 +263,18 @@ struct rcu_data { unsigned long last_fqs_resched; /* Time of last rcu_resched(). */ unsigned long last_sched_clock; /* Jiffies of last rcu_sched_clock_irq().= */ =20 +#ifdef CONFIG_RCU_LAZY + long lazy_len; /* Length of buffered lazy callbacks. */ +#endif int cpu; }; =20 /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 #define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE 2 -#define RCU_NOCB_WAKE_FORCE 3 +#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE 3 +#define RCU_NOCB_WAKE_FORCE 4 =20 #define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500)) /* For jiffies_till_first_fqs and */ @@ -440,9 +444,10 @@ static struct swait_queue_head *rcu_nocb_gp_get(struct= rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *r= hp, - unsigned long j); + unsigned long j, bool lazy, bool wakegp); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags); + bool *was_alldone, unsigned long flags, + bool lazy); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 31068dd31315..7e97a7b6e046 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -256,6 +256,31 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool fo= rce) return __wake_nocb_gp(rdp_gp, rdp, force, flags); } =20 +/* + * LAZY_FLUSH_JIFFIES decides the maximum amount of time that + * can elapse before lazy callbacks are flushed. Lazy callbacks + * could be flushed much earlier for a number of other reasons + * however, LAZY_FLUSH_JIFFIES will ensure no lazy callbacks are + * left unsubmitted to RCU after those many jiffies. + */ +#define LAZY_FLUSH_JIFFIES (10 * HZ) +unsigned long jiffies_till_flush =3D LAZY_FLUSH_JIFFIES; + +#ifdef CONFIG_RCU_LAZY +// To be called only from test code. +void rcu_lazy_set_jiffies_till_flush(unsigned long jif) +{ + jiffies_till_flush =3D jif; +} +EXPORT_SYMBOL(rcu_lazy_set_jiffies_till_flush); + +unsigned long rcu_lazy_get_jiffies_till_flush(void) +{ + return jiffies_till_flush; +} +EXPORT_SYMBOL(rcu_lazy_get_jiffies_till_flush); +#endif + /* * Arrange to wake the GP kthread for this NOCB group at some future * time when it is safe to do so. @@ -265,23 +290,39 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, = int waketype, { unsigned long flags; struct rcu_data *rdp_gp =3D rdp->nocb_gp_rdp; + unsigned long mod_jif =3D 0; =20 raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); =20 /* - * Bypass wakeup overrides previous deferments. In case - * of callback storm, no need to wake up too early. + * Bypass and lazy wakeup overrides previous deferments. In case of + * callback storm, no need to wake up too early. */ - if (waketype =3D=3D RCU_NOCB_WAKE_BYPASS) { - mod_timer(&rdp_gp->nocb_timer, jiffies + 2); - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); - } else { + switch (waketype) { + case RCU_NOCB_WAKE_LAZY: + if (rdp->nocb_defer_wakeup !=3D RCU_NOCB_WAKE_LAZY) + mod_jif =3D jiffies_till_flush; + break; + + case RCU_NOCB_WAKE_BYPASS: + mod_jif =3D 2; + break; + + case RCU_NOCB_WAKE: + case RCU_NOCB_WAKE_FORCE: + // For these, make it wake up the soonest if we + // were in a bypass or lazy sleep before. if (rdp_gp->nocb_defer_wakeup < RCU_NOCB_WAKE) - mod_timer(&rdp_gp->nocb_timer, jiffies + 1); - if (rdp_gp->nocb_defer_wakeup < waketype) - WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + mod_jif =3D 1; + break; } =20 + if (mod_jif) + mod_timer(&rdp_gp->nocb_timer, jiffies + mod_jif); + + if (rdp_gp->nocb_defer_wakeup < waketype) + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags); =20 trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, reason); @@ -293,10 +334,13 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, = int waketype, * proves to be initially empty, just return false because the no-CB GP * kthread may need to be awakened in this case. * + * Return true if there was something to be flushed and it succeeded, othe= rwise + * false. + * * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head= *rhp, - unsigned long j) + unsigned long j, bool lazy) { struct rcu_cblist rcl; =20 @@ -310,7 +354,18 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *= rdp, struct rcu_head *rhp, /* Note: ->cblist.len already accounts for ->nocb_bypass contents. */ if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + + /* + * If the new CB requested was a lazy one, queue it onto the main + * ->cblist so we can take advantage of a sooner grade period. + */ + if (lazy && rhp) { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); + rcu_cblist_enqueue(&rcl, rhp); + } else { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + } + rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); @@ -326,13 +381,20 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data = *rdp, struct rcu_head *rhp, * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *r= hp, - unsigned long j) + unsigned long j, bool lazy, bool wake_gp) { + bool ret; + if (!rcu_rdp_is_offloaded(rdp)) return true; rcu_lockdep_assert_cblist_protected(rdp); rcu_nocb_bypass_lock(rdp); - return rcu_nocb_do_flush_bypass(rdp, rhp, j); + ret =3D rcu_nocb_do_flush_bypass(rdp, rhp, j, lazy); + + if (wake_gp) + wake_nocb_gp(rdp, true); + + return ret; } =20 /* @@ -345,7 +407,7 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *= rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; - WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j)); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } =20 /* @@ -367,12 +429,14 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data= *rdp, unsigned long j) * there is only one CPU in operation. */ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { unsigned long c; unsigned long cur_gp_seq; unsigned long j =3D jiffies; long ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + bool bypass_is_lazy =3D (ncbs =3D=3D READ_ONCE(rdp->lazy_len)); =20 lockdep_assert_irqs_disabled(); =20 @@ -417,23 +481,29 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp,= struct rcu_head *rhp, // If there hasn't yet been all that many ->cblist enqueues // this jiffy, tell the caller to enqueue onto ->cblist. But flush // ->nocb_bypass first. - if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy) { + // Lazy CBs throttle this back and do immediate bypass queuing. + if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy && !lazy) { rcu_nocb_lock(rdp); *was_alldone =3D !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j)); + + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false, false)); + WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } =20 // If ->nocb_bypass has been used too long or is too full, // flush ->nocb_bypass to ->cblist. - if ((ncbs && j !=3D READ_ONCE(rdp->nocb_bypass_first)) || + if ((ncbs && !bypass_is_lazy && j !=3D READ_ONCE(rdp->nocb_bypass_first))= || + (ncbs && bypass_is_lazy && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush)))= || ncbs >=3D qhimark) { rcu_nocb_lock(rdp); - if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { + + if (!rcu_nocb_flush_bypass(rdp, rhp, j, lazy, false)) { *was_alldone =3D !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, @@ -460,16 +530,29 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp,= struct rcu_head *rhp, // We need to use the bypass. rcu_nocb_wait_contended(rdp); rcu_nocb_bypass_lock(rdp); + ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + + if (IS_ENABLED(CONFIG_RCU_LAZY) && lazy) + WRITE_ONCE(rdp->lazy_len, rdp->lazy_len + 1); + if (!ncbs) { WRITE_ONCE(rdp->nocb_bypass_first, j); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); } + rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ - if (ncbs) { + + // We had CBs in the bypass list before. There is nothing else to do if: + // There were only non-lazy CBs before, in this case, the bypass timer + // or GP-thread will handle the CBs including any new lazy ones. + // Or, the new CB is lazy and the old bypass-CBs were also lazy. In this + // case the old lazy timer would have been setup. When that expires, + // the new lazy one will be handled. + if (ncbs && (!bypass_is_lazy || lazy)) { local_irq_restore(flags); } else { // No-CBs GP kthread might be indefinitely asleep, if so, wake. @@ -478,6 +561,10 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, = struct rcu_head *rhp, trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQwake")); __call_rcu_nocb_wake(rdp, true, flags); + } else if (bypass_is_lazy && !lazy) { + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, + TPS("FirstBQwakeLazy2Non")); + __call_rcu_nocb_wake(rdp, true, flags); } else { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQnoWake")); @@ -499,7 +586,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, = bool was_alldone, { unsigned long cur_gp_seq; unsigned long j; - long len; + long len, lazy_len, bypass_len; struct task_struct *t; =20 // If we are being polled or there is no kthread, just leave. @@ -512,9 +599,16 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp,= bool was_alldone, } // Need to actually to a wakeup. len =3D rcu_segcblist_n_cbs(&rdp->cblist); + bypass_len =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_len =3D READ_ONCE(rdp->lazy_len); if (was_alldone) { rdp->qlen_last_fqs_check =3D len; - if (!irqs_disabled_flags(flags)) { + // Only lazy CBs in bypass list + if (lazy_len && bypass_len =3D=3D lazy_len) { + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazy")); + } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); @@ -604,8 +698,8 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int = cpu) */ static void nocb_gp_wait(struct rcu_data *my_rdp) { - bool bypass =3D false; - long bypass_ncbs; + bool bypass =3D false, lazy =3D false; + long bypass_ncbs, lazy_ncbs; int __maybe_unused cpu =3D my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; @@ -640,24 +734,41 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { + bool flush_bypass =3D false; + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); lockdep_assert_held(&rdp->nocb_lock); bypass_ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); - if (bypass_ncbs && + lazy_ncbs =3D READ_ONCE(rdp->lazy_len); + + if (bypass_ncbs && (lazy_ncbs =3D=3D bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flus= h) || + bypass_ncbs > 2 * qhimark)) { + flush_bypass =3D true; + } else if (bypass_ncbs && (lazy_ncbs !=3D bypass_ncbs) && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || bypass_ncbs > 2 * qhimark)) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + flush_bypass =3D true; } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { rcu_nocb_unlock_irqrestore(rdp, flags); continue; /* No callbacks here, try next. */ } + + if (flush_bypass) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs =3D rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs =3D READ_ONCE(rdp->lazy_len); + } + if (bypass_ncbs) { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("Bypass")); - bypass =3D true; + bypass_ncbs =3D=3D lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + if (bypass_ncbs =3D=3D lazy_ncbs) + lazy =3D true; + else + bypass =3D true; } rnp =3D rdp->mynode; =20 @@ -705,12 +816,21 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) my_rdp->nocb_gp_gp =3D needwait_gp; my_rdp->nocb_gp_seq =3D needwait_gp ? wait_gp_seq : 0; =20 - if (bypass && !rcu_nocb_poll) { - // At least one child with non-empty ->nocb_bypass, so set - // timer in order to avoid stranding its callbacks. - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); + // At least one child with non-empty ->nocb_bypass, so set + // timer in order to avoid stranding its callbacks. + if (!rcu_nocb_poll) { + // If bypass list only has lazy CBs. Add a deferred + // lazy wake up. + if (lazy && !bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazyIsDeferred")); + // Otherwise add a deferred bypass wake up. + } else if (bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, + TPS("WakeBypassIsDeferred")); + } } + if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ if (gotcbs) @@ -1036,7 +1156,7 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, false)); /* * Start with invoking rcu_core() early. This way if the current thread * happens to preempt an ongoing call to rcu_core() in the middle, @@ -1290,6 +1410,7 @@ static void __init rcu_boot_init_nocb_percpu_data(str= uct rcu_data *rdp) raw_spin_lock_init(&rdp->nocb_gp_lock); timer_setup(&rdp->nocb_timer, do_nocb_deferred_wakeup_timer, 0); rcu_cblist_init(&rdp->nocb_bypass); + WRITE_ONCE(rdp->lazy_len, 0); mutex_init(&rdp->nocb_gp_kthread_mutex); } =20 @@ -1571,13 +1692,14 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) } =20 static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *r= hp, - unsigned long j) + unsigned long j, bool lazy, bool wakegp) { return true; } =20 static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { return false; } --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9924ECAAD3 for ; Thu, 1 Sep 2022 22:18:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234916AbiIAWSl (ORCPT ); Thu, 1 Sep 2022 18:18:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234587AbiIAWSD (ORCPT ); Thu, 1 Sep 2022 18:18:03 -0400 Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB987580B6 for ; Thu, 1 Sep 2022 15:18:02 -0700 (PDT) Received: by mail-qv1-xf31.google.com with SMTP id y17so171048qvr.5 for ; Thu, 01 Sep 2022 15:18:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=5Y5kbAl2ZL0hvyMP3+b8/GM5tIuNKa2oHKBHOOJke14=; b=DLk0B/+Kjl2sEDnaJ5HNoOkF5QFsAHl3d/WO0kQdWt7MwUX6E9cr00jSheEW6r0ohh eKqnQ2mBJFjJ3kK0CphbuerOCVjht107mlZlGLAxmmWEJQT3dq2Zm0nFUcX0KPUZ68Qt UbZiWOQ9CWe4p662dEMWk9EB4AQrJh0Qr8MdY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=5Y5kbAl2ZL0hvyMP3+b8/GM5tIuNKa2oHKBHOOJke14=; b=vsdCds4KjKjgaxmU56IoN113fLi8jlVQnywXEG3e/fpeb1rSIabf/n3S3SNcSKFVE0 e5FpdSZaPpB7gaqigU2pnIVnlyBJnAfTm74aF7g1dk9Wb3WENwGuekMV/DoZ/+ikg2xM Zp0Yv6ClLlmJhL/R18xeEAugCmoQUaodgviizowNrySOEzdi28NrLnPIf/KFh+bRmjLG VoH0AVHT7FZG4dVuDCDmmG55+WgtOfviBd8y0V6NULKbj4XjoqwwV9QsTIkk0fjnE+19 e9aG5wEsnl5PCCx88HQ2sj0KuyQKgstB7pf9ZowHblD4Dhb9T87U2oZQ3eCp4pJcGHk4 2KIw== X-Gm-Message-State: ACgBeo05EFOFiG/riAM3MtXsHq5YXYfc24GGv4yGlTEb9EZGAt++cAwn zWBzsJK0wnC60kmnuohrQFFeQg== X-Google-Smtp-Source: AA6agR5SVRH/RM5xAYfxYhk8E1h0qVAjY9FUK/2Tu/dAn793HN5Tons0Nj46w1jA+Idc8q2jHDCtow== X-Received: by 2002:ad4:5bc2:0:b0:499:2c2c:1b0b with SMTP id t2-20020ad45bc2000000b004992c2c1b0bmr4677172qvt.115.1662070682335; Thu, 01 Sep 2022 15:18:02 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:01 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, Joel Fernandes Subject: [PATCH v5 07/18] rcu: shrinker for lazy rcu Date: Thu, 1 Sep 2022 22:17:09 +0000 Message-Id: <20220901221720.1105021-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 7e97a7b6e046..560ba87911c5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1333,6 +1333,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); =20 +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count =3D 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); + + count +=3D READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count =3D 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); + int _count =3D READ_ONCE(rdp->lazy_len); + + if (_count =3D=3D 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -=3D _count; + count +=3D _count; + if (sc->nr_to_scan <=3D 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker =3D { + .count_objects =3D lazy_rcu_shrink_count, + .scan_objects =3D lazy_rcu_shrink_scan, + .batch =3D 0, + .seeks =3D DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1367,6 +1416,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; =20 + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + #if defined(CONFIG_NO_HZ_FULL) if (tick_nohz_full_running) cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask); --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70648ECAAD2 for ; Thu, 1 Sep 2022 22:18:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234982AbiIAWSy (ORCPT ); Thu, 1 Sep 2022 18:18:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234574AbiIAWSF (ORCPT ); Thu, 1 Sep 2022 18:18:05 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04E665809E for ; Thu, 1 Sep 2022 15:18:04 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id cb8so280900qtb.0 for ; Thu, 01 Sep 2022 15:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=sbbotK/X6hyBCxf4X8eHnMZLe8+ikwZ/eJHvPr5PB5Y=; b=kPA5Qz5YkoMPW7SkgX+l0Ko5jkDfQyr3bsOZBOyAN8nMIw0Iw9j1I6H4sdDZvxhdxW 7wsvjTl8ccTmCT2hpTRxUIQlxHTAEhDAge3obd9j18OBY9X0NXJOi3EOnRakNGRSgOXu 1ZyJ9+IgAB/0g4Y3K5cwMXnAb9TFR59cfhq4s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=sbbotK/X6hyBCxf4X8eHnMZLe8+ikwZ/eJHvPr5PB5Y=; b=fVUbaOyGT1Dzoc1g+U7qYNs35r1V6+kEtkfIEzqAh8m/kEYrkndGBX0VFFKSE0OjRW CzV6FbV1ZMZFHRihIe0m+tRJ/RKXtq07182E6VwSCsCbIvkjw2CsxXBf6crOBXT2Cdmv bUpkB+TqLTKfFhItMrqOc57KJ7a22usJVTVoE09G7AiGz/nFCUZDw0d5IeKCKBURd6XT d3Pbqe8yCjMY52pUVya7Bguq3kv2LErgH++HitXXYIt9PPxM+aTge8L/ZFSiXg67r6II 4zv9rCL+Ab1ka09S3VEfz9tWwRI6Q3fELU25ckjSVRCniyiEcC6dkiRTtOcQ/vCJvBAG 3HLA== X-Gm-Message-State: ACgBeo3eia4Sr05qb6SXJQHuxUeKP4iGE3xneBfWV8PE2BIg4pUvl5Ut Jp2J+xKsrzYJUaKZpLOKrnjtSQ== X-Google-Smtp-Source: AA6agR6KszQ19UJa+Vd0R4VFnWWwwBVRqP48H2wjT6CRlwWeihtQuS7Q4n4Mfo+Tvf1h2BWMZDzvKA== X-Received: by 2002:a05:622a:143:b0:344:95bf:8f02 with SMTP id v3-20020a05622a014300b0034495bf8f02mr24705082qtw.202.1662070683012; Thu, 01 Sep 2022 15:18:03 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:02 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 08/18] rcu: Add per-CB tracing for queuing, flush and invocation. Date: Thu, 1 Sep 2022 22:17:10 +0000 Message-Id: <20220901221720.1105021-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is very useful to debug whether call_rcu_lazy() is working correctly. In the future, any BPF tools could also be modified to read the new information. The additional tracing is enabled by a new CONFIG_RCU_TRACE_CB and is kept disabled by default. Signed-off-by: Joel Fernandes (Google) Reported-by: kernel test robot --- include/linux/types.h | 44 ++++++++++++++++++++++++ include/trace/events/rcu.h | 69 +++++++++++++++++++++++++++++++++++--- kernel/rcu/Kconfig | 11 ++++++ kernel/rcu/rcu_segcblist.c | 21 ++++++++++++ kernel/rcu/rcu_segcblist.h | 8 +++++ kernel/rcu/tree.c | 46 +++++++++++++++++++++++-- kernel/rcu/tree_nocb.h | 26 +++++++++++++- 7 files changed, 217 insertions(+), 8 deletions(-) diff --git a/include/linux/types.h b/include/linux/types.h index ea8cf60a8a79..47501fdfd3c1 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -198,10 +198,51 @@ struct ustat { char f_fpack[6]; }; =20 +#ifdef CONFIG_RCU_TRACE_CB +/* + * Debug information that a caller can store within a callback_head. + * Its expected to provide at least 12 bytes before BUILD_BUG starts + * complaining. + */ +enum cb_debug_flags { + CB_DEBUG_KFREE, + CB_DEBUG_LAZY, + CB_DEBUG_BYPASS, + + // A new non-lazy CB showed up and we decided to not use + // the bypass list for it. So we flushed the old ones. + CB_DEBUG_NON_LAZY_FLUSHED, + + // We decided to use the bypass list but had to flush + // the old bypass CBs because they got too old or too big. + CB_DEBUG_BYPASS_FLUSHED, + CB_DEBUG_BYPASS_LAZY_FLUSHED, + + // The GP thread flushed the bypass CBs if they got old or big. + CB_DEBUG_GPTHREAD_FLUSHED, + + // De-offload from NOCB mode. + CB_DEBUG_DEOFFLOAD_FLUSHED, + + // rcu_barrier() flushes lazy/bypass CBs for CB exec ordering. + CB_DEBUG_BARRIER_FLUSHED +}; + +struct cb_debug_info { + // 16-bit jiffie deltas can provide about 60 seconds of resolution + // at HZ=3D1000 before wrapping. That's enough for debug. + u16 cb_queue_jiff; + u16 first_bp_jiff; + u16 cb_flush_jiff; + enum cb_debug_flags flags:16; +}; +#endif + /** * struct callback_head - callback structure for use with RCU and task_work * @next: next update requests in a list * @func: actual update function to call after the grace period. + * @di: debug information that can be stored. * * The struct is aligned to size of pointer. On most architectures it happ= ens * naturally due ABI requirements, but some architectures (like CRIS) have @@ -220,6 +261,9 @@ struct ustat { struct callback_head { struct callback_head *next; void (*func)(struct callback_head *head); +#ifdef CONFIG_RCU_TRACE_CB + struct cb_debug_info di; +#endif } __attribute__((aligned(sizeof(void *)))); #define rcu_head callback_head =20 diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 90b2fb0292cb..dd6baec6745c 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -630,24 +630,85 @@ TRACE_EVENT_RCU(rcu_batch_start, */ TRACE_EVENT_RCU(rcu_invoke_callback, =20 - TP_PROTO(const char *rcuname, struct rcu_head *rhp), + TP_PROTO(const char *rcuname, struct rcu_head *rhp +#ifdef CONFIG_RCU_TRACE_CB + , unsigned long jiffies_first +#endif + ), =20 - TP_ARGS(rcuname, rhp), + TP_ARGS(rcuname, rhp +#ifdef CONFIG_RCU_TRACE_CB + , jiffies_first +#endif + ), =20 TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp) __field(void *, func) +#ifdef CONFIG_RCU_TRACE_CB + __field(u16, cb_debug_flags) + __field(int, cb_queue_exec_latency) + __array(char, bypass_flush_reason, 32) + __field(int, cb_bypass_slack) + __field(int, cb_queue_flush_latency) +#endif ), =20 TP_fast_assign( __entry->rcuname =3D rcuname; __entry->rhp =3D rhp; __entry->func =3D rhp->func; +#ifdef CONFIG_RCU_TRACE_CB + __entry->cb_debug_flags =3D rhp->di.flags; + __entry->cb_queue_exec_latency =3D + (jiffies - jiffies_first) - rhp->di.cb_queue_jiff, + + /* The following 3 fields are valid only for bypass/lazy CBs. */ + __entry->cb_bypass_slack =3D + (rhp->di.flags & BIT(CB_DEBUG_BYPASS)) ? + rhp->di.cb_queue_jiff - rhp->di.first_bp_jiff : 0, + __entry->cb_queue_flush_latency =3D + (rhp->di.flags & BIT(CB_DEBUG_BYPASS)) ? + rhp->di.cb_flush_jiff - rhp->di.cb_queue_jiff : 0; + + + if (__entry->cb_debug_flags & BIT(CB_DEBUG_NON_LAZY_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "non-lazy"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_BYPASS_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "bypass"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_BYPASS_LAZY_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "bypass-lazy"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_GPTHREAD_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "gpthread"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_BARRIER_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "rcu_barrier"); + else if (__entry->cb_debug_flags & BIT(CB_DEBUG_DEOFFLOAD_FLUSHED)) + strcpy(__entry->bypass_flush_reason, "deoffload"); +#endif ), =20 - TP_printk("%s rhp=3D%p func=3D%ps", - __entry->rcuname, __entry->rhp, __entry->func) + TP_printk("%s rhp=3D%p func=3D%ps" +#ifdef CONFIG_RCU_TRACE_CB + " lazy=3D%d " + "bypass=3D%d " + "flush_reason=3D%s " + "queue_exec_latency=3D%d " + "bypass_slack=3D%d " + "queue_flush_latency=3D%d" +#endif + , + __entry->rcuname, __entry->rhp, __entry->func +#ifdef CONFIG_RCU_TRACE_CB + , + !!(__entry->cb_debug_flags & BIT(CB_DEBUG_LAZY)), + !!(__entry->cb_debug_flags & BIT(CB_DEBUG_BYPASS)), + __entry->bypass_flush_reason, + __entry->cb_queue_exec_latency, + __entry->cb_bypass_slack, + __entry->cb_queue_flush_latency +#endif + ) ); =20 /* diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 3128d01427cb..cd6ff63517f4 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -319,4 +319,15 @@ config RCU_LAZY To save power, batch RCU callbacks and flush after delay, memory pressure or callback list growing too big. =20 +config RCU_TRACE_CB + bool "Enable additional callback tracing" + depends on RCU_LAZY + default n + help + Enable additional callback tracing for RCU. Currently only + tracing for lazy/bypass callbacks is done. In the future, it could be + extended to non-lazy callbacks as well. The trace point contains + detailed information about callback flushing reason and execution + time. This information can be retrieved by BPF tools via tracepoints. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c index 55b50e592986..8f0b024f7bc5 100644 --- a/kernel/rcu/rcu_segcblist.c +++ b/kernel/rcu/rcu_segcblist.c @@ -32,6 +32,27 @@ void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct = rcu_head *rhp) WRITE_ONCE(rclp->len, rclp->len + 1); } =20 +/* + * Set a debug flag on all CBs in the unsegmented cblist @rclp. + * + * The callback causing the flush. Should be NULL if its a timer that does= it. + */ +#ifdef CONFIG_RCU_TRACE_CB +void rcu_cblist_set_flush(struct rcu_cblist *rcl, + enum cb_debug_flags flags, + unsigned long flush_jiff) +{ + if (!rcl || !rcl->head) + return; + + for (struct rcu_head *head =3D rcl->head; + head && head !=3D *(rcl->tail); head =3D head->next) { + head->di.flags |=3D flags; + head->di.cb_flush_jiff =3D flush_jiff; + } +} +#endif + /* * Flush the second rcu_cblist structure onto the first one, obliterating * any contents of the first. If rhp is non-NULL, enqueue it as the sole diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index 431cee212467..ac1f34024b84 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -53,6 +53,14 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcbl= ist *rsclp) #endif } =20 +#ifdef CONFIG_RCU_TRACE_CB +void rcu_cblist_set_flush(struct rcu_cblist *rcl, + enum cb_debug_flags flags, + unsigned long flush_jiff); +#else +#define rcu_cblist_set_flush(...) +#endif + static inline void rcu_segcblist_set_flags(struct rcu_segcblist *rsclp, int flags) { diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index aaced29a0a71..8111d9f37621 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2179,6 +2179,12 @@ int rcutree_dead_cpu(unsigned int cpu) return 0; } =20 +static unsigned long cb_debug_jiffies_first; + +#if defined(CONFIG_RCU_TRACE_CB) && defined(CONFIG_RCU_LAZY) +extern unsigned long jiffies_till_flush; +#endif + /* * Invoke any RCU callbacks that have made it to the end of their grace * period. Throttle as specified by rdp->blimit. @@ -2241,7 +2247,12 @@ static void rcu_do_batch(struct rcu_data *rdp) debug_rcu_head_unqueue(rhp); =20 rcu_lock_acquire(&rcu_callback_map); - trace_rcu_invoke_callback(rcu_state.name, rhp); + + trace_rcu_invoke_callback(rcu_state.name, rhp +#ifdef CONFIG_RCU_TRACE_CB + , READ_ONCE(cb_debug_jiffies_first) +#endif + ); =20 f =3D rhp->func; WRITE_ONCE(rhp->func, (rcu_callback_t)0L); @@ -2736,6 +2747,17 @@ __call_rcu_common(struct rcu_head *head, rcu_callbac= k_t func, bool lazy) struct rcu_data *rdp; bool was_alldone; =20 + /* + * For CB debugging, record the starting reference to jiffies while + * making sure it does not wrap around. The starting reference is + * used to calculate 16-bit jiffie deltas. To be safe, reset the + * reference once the delta exceeds 15-bits worth. + */ + if (IS_ENABLED(CONFIG_RCU_TRACE_CB) && + (!READ_ONCE(cb_debug_jiffies_first) || + (jiffies - READ_ONCE(cb_debug_jiffies_first) > (1 << 15)))) + WRITE_ONCE(cb_debug_jiffies_first, jiffies); + /* Misaligned rcu_head! */ WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1)); =20 @@ -2771,14 +2793,29 @@ __call_rcu_common(struct rcu_head *head, rcu_callba= ck_t func, bool lazy) =20 check_cb_ovld(rdp); =20 - if (__is_kvfree_rcu_offset((unsigned long)func)) +#ifdef CONFIG_RCU_TRACE_CB + head->di.flags =3D 0; + WARN_ON_ONCE(jiffies - READ_ONCE(cb_debug_jiffies_first) > 65535); + head->di.cb_queue_jiff =3D (u16)(jiffies - READ_ONCE(cb_debug_jiffies_fir= st)); +#endif + + if (__is_kvfree_rcu_offset((unsigned long)func)) { trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, rcu_segcblist_n_cbs(&rdp->cblist)); - else +#ifdef CONFIG_RCU_TRACE_CB + head->di.flags |=3D BIT(CB_DEBUG_KFREE); +#endif + } else { trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); =20 +#ifdef CONFIG_RCU_TRACE_CB + if (lazy) + head->di.flags |=3D BIT(CB_DEBUG_LAZY); +#endif + } + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. @@ -3943,6 +3980,9 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func =3D rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); + + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_BARRIER_FLUSHED), + (jiffies - READ_ONCE(cb_debug_jiffies_first))); WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, /* wake gp thread */ true)); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 560ba87911c5..ee5924ba2f3b 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -355,6 +355,11 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *= rdp, struct rcu_head *rhp, if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ =20 + /* The lazy CBs are being flushed, but a new one might be enqueued. */ +#ifdef CONFIG_RCU_TRACE_CB + WARN_ON_ONCE(rhp && lazy !=3D !!(rhp->di.flags & BIT(CB_DEBUG_LAZY))); +#endif + /* * If the new CB requested was a lazy one, queue it onto the main * ->cblist so we can take advantage of a sooner grade period. @@ -407,6 +412,10 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data = *rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; + + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_GPTHREAD_FLUSHED), + (j - READ_ONCE(cb_debug_jiffies_first))); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } =20 @@ -489,8 +498,10 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, = struct rcu_head *rhp, trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); =20 - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false, false)); + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_NON_LAZY_FLUSHED), + (j - READ_ONCE(cb_debug_jiffies_first))); =20 + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false, false)); WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } @@ -503,6 +514,10 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, = struct rcu_head *rhp, ncbs >=3D qhimark) { rcu_nocb_lock(rdp); =20 + rcu_cblist_set_flush(&rdp->nocb_bypass, + lazy ? BIT(CB_DEBUG_BYPASS_LAZY_FLUSHED) : BIT(CB_DEBUG_BYPASS_FLUSHED= ), + (j - READ_ONCE(cb_debug_jiffies_first))); + if (!rcu_nocb_flush_bypass(rdp, rhp, j, lazy, false)) { *was_alldone =3D !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) @@ -543,6 +558,11 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, = struct rcu_head *rhp, trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); } =20 +#ifdef CONFIG_RCU_TRACE_CB + rhp->di.flags |=3D BIT(CB_DEBUG_BYPASS); + rhp->di.first_bp_jiff =3D READ_ONCE(rdp->nocb_bypass_first) - READ_ONCE(c= b_debug_jiffies_first); +#endif + rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ =20 @@ -1156,6 +1176,10 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ + + rcu_cblist_set_flush(&rdp->nocb_bypass, BIT(CB_DEBUG_DEOFFLOAD_FLUSHED), + (jiffies - READ_ONCE(cb_debug_jiffies_first))); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false, false)); /* * Start with invoking rcu_core() early. This way if the current thread --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9E6AECAAD2 for ; Thu, 1 Sep 2022 22:19:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233729AbiIAWTC (ORCPT ); Thu, 1 Sep 2022 18:19:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234624AbiIAWSF (ORCPT ); Thu, 1 Sep 2022 18:18:05 -0400 Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BE70580B6 for ; Thu, 1 Sep 2022 15:18:04 -0700 (PDT) Received: by mail-qv1-xf31.google.com with SMTP id kh8so185970qvb.1 for ; Thu, 01 Sep 2022 15:18:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=9dE4Rk8OIOb+ihEgM/v0EEwGQjrbZnrnEwGtQG+Tokc=; b=hduLdeo23eUYXwfIi5kOwRTpmsQWKz5WMzdJryBENxzTtKqWyj8jKS1pkZfs7K8ix/ 433izYbBsAKpcvkSRytC/Vy95tY1M4jnMjqRDxPDnm8amlDKZui5CBFRq87fL6LuqJ/2 5PT2xT3jsfon0zpijEzJBydZ5JMeqjs5NKQVw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=9dE4Rk8OIOb+ihEgM/v0EEwGQjrbZnrnEwGtQG+Tokc=; b=YMCRYajNcmRT5ezkJCeBY/4DlKXAhdpWPIPs98tUMaDAavVt0a3nS1407Vn4i/aVyb soOunjtcw6mfLlkdJAzWnbYDSE+TXsfdClS10KqiQsUoflDHHl7COxiD1XcNR12F+741 3l3CDFHpIZPqhIefIN3TEJ+m7KLdpmckKnshcG2Vx45Fewnsyi3CDZKpU7tvnTmCA8S8 iIoXmHnnL7TWfgWuMSmy/1zohvPXBNolX9omXXLWv3GA+NNaIILNY2dF1VN4WWxt0jps q5d1Sn7+eVuaZ8C64VuQ8fYQcUCaW9gzOOXNcKfq3BvCXu6rJdekJNDvrcyPwdrKmejX nuLA== X-Gm-Message-State: ACgBeo3bZ8l/hx27/+3xAKwYKBj82yV7j+YC2UzRQICyn1eSj1VV4ess yIvKLTo+E3+jTSE/N/BNeFYQcA== X-Google-Smtp-Source: AA6agR5JaoGB677mb+F1Gn0KMgIKjVq4iVwwN+1P2BSJcZKXS+rgh0glw3Qtr1+KQ96rCexOh2tcOA== X-Received: by 2002:a05:6214:b61:b0:498:ff24:9b59 with SMTP id ey1-20020a0562140b6100b00498ff249b59mr21011911qvb.98.1662070683846; Thu, 01 Sep 2022 15:18:03 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:03 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 09/18] rcuscale: Add laziness and kfree tests Date: Thu, 1 Sep 2022 22:17:11 +0000 Message-Id: <20220901221720.1105021-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We aad 2 tests to rcuscale, first one is a startup test to check whether we are not too lazy or too hard working. Two, emulate kfree_rcu() itself to use call_rcu_lazy() and check memory pressure. In my testing, call_rcu_lazy() does well to keep memory pressure under control, similar to kfree_rcu(). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcuscale.c | 74 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3ef02d4a8108..6808f2e14385 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -95,6 +95,7 @@ torture_param(int, verbose, 1, "Enable verbose debugging = printk()s"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to d= isable"); torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?= "); torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate.= "); +torture_param(int, kfree_rcu_by_lazy, 0, "Use call_rcu_lazy() to emulate k= free_rcu()?"); =20 static char *scale_type =3D "rcu"; module_param(scale_type, charp, 0444); @@ -659,6 +660,14 @@ struct kfree_obj { struct rcu_head rh; }; =20 +/* Used if doing RCU-kfree'ing via call_rcu_lazy(). */ +static void kfree_rcu_lazy(struct rcu_head *rh) +{ + struct kfree_obj *obj =3D container_of(rh, struct kfree_obj, rh); + + kfree(obj); +} + static int kfree_scale_thread(void *arg) { @@ -696,6 +705,11 @@ kfree_scale_thread(void *arg) if (!alloc_ptr) return -ENOMEM; =20 + if (kfree_rcu_by_lazy) { + call_rcu_lazy(&(alloc_ptr->rh), kfree_rcu_lazy); + continue; + } + // By default kfree_rcu_test_single and kfree_rcu_test_double are // initialized to false. If both have the same value (false or true) // both are randomly tested, otherwise only the one with value true @@ -738,6 +752,9 @@ kfree_scale_cleanup(void) { int i; =20 + if (kfree_rcu_by_lazy) + rcu_force_call_rcu_to_lazy(false); + if (torture_cleanup_begin()) return; =20 @@ -767,11 +784,64 @@ kfree_scale_shutdown(void *arg) return -EINVAL; } =20 +// Used if doing RCU-kfree'ing via call_rcu_lazy(). +static unsigned long jiffies_at_lazy_cb; +static struct rcu_head lazy_test1_rh; +static int rcu_lazy_test1_cb_called; +static void call_rcu_lazy_test1(struct rcu_head *rh) +{ + jiffies_at_lazy_cb =3D jiffies; + WRITE_ONCE(rcu_lazy_test1_cb_called, 1); +} + static int __init kfree_scale_init(void) { long i; int firsterr =3D 0; + unsigned long orig_jif, jif_start; + + // If lazy-rcu based kfree'ing is requested, then for kernels that + // support it, force all call_rcu() to call_rcu_lazy() so that non-lazy + // CBs do not remove laziness of the lazy ones (since the test tries to + // stress call_rcu_lazy() for OOM). + // + // Also, do a quick self-test to ensure laziness is as much as + // expected. + if (kfree_rcu_by_lazy && !IS_ENABLED(CONFIG_RCU_LAZY)) { + pr_alert("CONFIG_RCU_LAZY is disabled, falling back to kfree_rcu() " + "for delayed RCU kfree'ing\n"); + kfree_rcu_by_lazy =3D 0; + } + + if (kfree_rcu_by_lazy) { + /* do a test to check the timeout. */ + orig_jif =3D rcu_lazy_get_jiffies_till_flush(); + + rcu_force_call_rcu_to_lazy(true); + rcu_lazy_set_jiffies_till_flush(2 * HZ); + rcu_barrier(); + + jif_start =3D jiffies; + jiffies_at_lazy_cb =3D 0; + call_rcu_lazy(&lazy_test1_rh, call_rcu_lazy_test1); + + smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL =3D=3D 1); + + rcu_lazy_set_jiffies_till_flush(orig_jif); + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { + pr_alert("ERROR: Lazy CBs are not being lazy as expected!\n"); + WARN_ON_ONCE(1); + return -1; + } + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) { + pr_alert("ERROR: Lazy CBs are being too lazy!\n"); + WARN_ON_ONCE(1); + return -1; + } + } =20 kfree_nrealthreads =3D compute_real(kfree_nthreads); /* Start up the kthreads. */ @@ -784,7 +854,9 @@ kfree_scale_init(void) schedule_timeout_uninterruptible(1); } =20 - pr_alert("kfree object size=3D%zu\n", kfree_mult * sizeof(struct kfree_ob= j)); + pr_alert("kfree object size=3D%zu, kfree_rcu_by_lazy=3D%d\n", + kfree_mult * sizeof(struct kfree_obj), + kfree_rcu_by_lazy); =20 kfree_reader_tasks =3D kcalloc(kfree_nrealthreads, sizeof(kfree_reader_ta= sks[0]), GFP_KERNEL); --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 199DAECAAD2 for ; Thu, 1 Sep 2022 22:19:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232699AbiIAWTJ (ORCPT ); Thu, 1 Sep 2022 18:19:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234705AbiIAWSK (ORCPT ); Thu, 1 Sep 2022 18:18:10 -0400 Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DDB65A174 for ; Thu, 1 Sep 2022 15:18:05 -0700 (PDT) Received: by mail-qt1-x82f.google.com with SMTP id r6so230259qtx.6 for ; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=iF0VQ620OmNNEeBQv3gOJrSK8RXzJIsiopJiGn2gXgE=; b=SBhFYlQ3f3i6jV7HBq4Orbuwu/hinW5fDS4ifU7J74KEwQlKYPt/auQAncrY9fVdcu 5aTjF9w2HhCnuHyPQCCO0vD25oTat84cFEXTuBIKNKGkVKKSvEreggR7kMkSFMbjjldR i8wuBUjYYWqJSbR7o2Nre3jfWKBbsb2Lamrto= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=iF0VQ620OmNNEeBQv3gOJrSK8RXzJIsiopJiGn2gXgE=; b=L6Y5CulT6qhnwJ0xTDsEQ7p+F3U+c3kJ5KC0r9Usll8SviSfCZRhboAMonPKIM6mPG Bl84NODmKS+0e4fG70wGK6DwkkRqlrREk2CFRDhau1De9xbAHJ2MxlmapHOWOY6Wwo9b rbxt3vZ3sJW2+BP5F4q2DAr/zvOHgEd8cx95ES/q+WtqANTTyQtzpwmFgk9et//LvVMz Adz1fXdAuOXGkLWFo2taHwi9vmW/HtqakmYy9DymnAhLTA5zXqLfVTX/nTf4CtTN+1s6 lxf1wQaIzSLePRr8orTS4MLeeM8fQJBXU1t1YLT0X9sOLVFO0utNax3Tz5fBdIMcTpI4 W8Zg== X-Gm-Message-State: ACgBeo1TqV758uk+SPs1/GKcNe4Z415VyVvDl9UzGj516osWg2QT/kzn TMXTelftgIvTizpW1ntcDYZC3w== X-Google-Smtp-Source: AA6agR7H18+EdoJRj/8Zp5aQpkjWP9gIyH82b2JWKmLuxIWI5VEu84ZwMh5yQZdQ8rViSDXh8dHaDg== X-Received: by 2002:a05:622a:614:b0:343:487:45d1 with SMTP id z20-20020a05622a061400b00343048745d1mr25242495qta.443.1662070684496; Thu, 01 Sep 2022 15:18:04 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:04 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 10/18] rcutorture: Add test code for call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:12 +0000 Message-Id: <20220901221720.1105021-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We add a new RCU type to test call_rcu_lazy(). This allows us to just override the '.call' callback. To compensate for the laziness, we force the laziness to a small number of jiffies. The idea of this test is to stress the new code paths for stability and ensure it at least is providing behavior in parity with, or similar to, call_rcu(). The actual check for amount of laziness is in another test (rcuscale). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/rcu.h | 1 + kernel/rcu/rcutorture.c | 60 ++++++++++++++++++- kernel/rcu/tree.c | 1 + .../selftests/rcutorture/configs/rcu/CFLIST | 1 + .../selftests/rcutorture/configs/rcu/TREE11 | 18 ++++++ .../rcutorture/configs/rcu/TREE11.boot | 8 +++ 6 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TREE11 create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TREE11.b= oot diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 94675f14efe8..a6efb831673f 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -471,6 +471,7 @@ enum rcutorture_type { RCU_TASKS_TRACING_FLAVOR, RCU_TRIVIAL_FLAVOR, SRCU_FLAVOR, + RCU_LAZY_FLAVOR, INVALID_RCU_FLAVOR }; =20 diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 9ad5301385a4..b4114f29ab61 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -927,6 +927,64 @@ static struct rcu_torture_ops tasks_rude_ops =3D { =20 #endif // #else #ifdef CONFIG_TASKS_RUDE_RCU =20 +#ifdef CONFIG_RCU_LAZY + +/* + * Definitions for lazy RCU torture testing. + */ +static unsigned long orig_jiffies_till_flush; + +static void rcu_sync_torture_init_lazy(void) +{ + rcu_sync_torture_init(); + + orig_jiffies_till_flush =3D rcu_lazy_get_jiffies_till_flush(); + rcu_lazy_set_jiffies_till_flush(50); +} + +static void rcu_lazy_cleanup(void) +{ + rcu_lazy_set_jiffies_till_flush(orig_jiffies_till_flush); +} + +static struct rcu_torture_ops rcu_lazy_ops =3D { + .ttype =3D RCU_LAZY_FLAVOR, + .init =3D rcu_sync_torture_init_lazy, + .cleanup =3D rcu_lazy_cleanup, + .readlock =3D rcu_torture_read_lock, + .read_delay =3D rcu_read_delay, + .readunlock =3D rcu_torture_read_unlock, + .readlock_held =3D torture_readlock_not_held, + .get_gp_seq =3D rcu_get_gp_seq, + .gp_diff =3D rcu_seq_diff, + .deferred_free =3D rcu_torture_deferred_free, + .sync =3D synchronize_rcu, + .exp_sync =3D synchronize_rcu_expedited, + .get_gp_state =3D get_state_synchronize_rcu, + .start_gp_poll =3D start_poll_synchronize_rcu, + .poll_gp_state =3D poll_state_synchronize_rcu, + .cond_sync =3D cond_synchronize_rcu, + .call =3D call_rcu_lazy, + .cb_barrier =3D rcu_barrier, + .fqs =3D rcu_force_quiescent_state, + .stats =3D NULL, + .gp_kthread_dbg =3D show_rcu_gp_kthreads, + .check_boost_failed =3D rcu_check_boost_fail, + .stall_dur =3D rcu_jiffies_till_stall_check, + .irq_capable =3D 1, + .can_boost =3D IS_ENABLED(CONFIG_RCU_BOOST), + .extendables =3D RCUTORTURE_MAX_EXTEND, + .name =3D "rcu_lazy" +}; + +#define LAZY_OPS &rcu_lazy_ops, + +#else // #ifdef CONFIG_RCU_LAZY + +#define LAZY_OPS + +#endif // #else #ifdef CONFIG_RCU_LAZY + =20 #ifdef CONFIG_TASKS_TRACE_RCU =20 @@ -3466,7 +3524,7 @@ rcu_torture_init(void) unsigned long gp_seq =3D 0; static struct rcu_torture_ops *torture_ops[] =3D { &rcu_ops, &rcu_busted_ops, &srcu_ops, &srcud_ops, &busted_srcud_ops, - TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS + TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS LAZY_OPS &trivial_ops, }; =20 diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8111d9f37621..08bafe13c3ad 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -537,6 +537,7 @@ void rcutorture_get_gp_data(enum rcutorture_type test_t= ype, int *flags, { switch (test_type) { case RCU_FLAVOR: + case RCU_LAZY_FLAVOR: *flags =3D READ_ONCE(rcu_state.gp_flags); *gp_seq =3D rcu_seq_current(&rcu_state.gp_seq); break; diff --git a/tools/testing/selftests/rcutorture/configs/rcu/CFLIST b/tools/= testing/selftests/rcutorture/configs/rcu/CFLIST index 98b6175e5aa0..609c3370616f 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/CFLIST +++ b/tools/testing/selftests/rcutorture/configs/rcu/CFLIST @@ -5,6 +5,7 @@ TREE04 TREE05 TREE07 TREE09 +TREE11 SRCU-N SRCU-P SRCU-T diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE11 b/tools/= testing/selftests/rcutorture/configs/rcu/TREE11 new file mode 100644 index 000000000000..436013f3e015 --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE11 @@ -0,0 +1,18 @@ +CONFIG_SMP=3Dy +CONFIG_PREEMPT_NONE=3Dn +CONFIG_PREEMPT_VOLUNTARY=3Dn +CONFIG_PREEMPT=3Dy +#CHECK#CONFIG_PREEMPT_RCU=3Dy +CONFIG_HZ_PERIODIC=3Dn +CONFIG_NO_HZ_IDLE=3Dy +CONFIG_NO_HZ_FULL=3Dn +CONFIG_RCU_TRACE=3Dy +CONFIG_HOTPLUG_CPU=3Dy +CONFIG_MAXSMP=3Dy +CONFIG_CPUMASK_OFFSTACK=3Dy +CONFIG_RCU_NOCB_CPU=3Dy +CONFIG_DEBUG_LOCK_ALLOC=3Dn +CONFIG_RCU_BOOST=3Dn +CONFIG_DEBUG_OBJECTS_RCU_HEAD=3Dn +CONFIG_RCU_EXPERT=3Dy +CONFIG_RCU_LAZY=3Dy diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot b/t= ools/testing/selftests/rcutorture/configs/rcu/TREE11.boot new file mode 100644 index 000000000000..9b6f720d4ccd --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot @@ -0,0 +1,8 @@ +maxcpus=3D8 nr_cpus=3D43 +rcutree.gp_preinit_delay=3D3 +rcutree.gp_init_delay=3D3 +rcutree.gp_cleanup_delay=3D3 +rcu_nocbs=3D0-7 +rcutorture.torture_type=3Drcu_lazy +rcutorture.nocbs_nthreads=3D8 +rcutorture.fwd_progress=3D0 --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 434DCECAAD3 for ; Thu, 1 Sep 2022 22:19:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235225AbiIAWTc (ORCPT ); Thu, 1 Sep 2022 18:19:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234784AbiIAWST (ORCPT ); Thu, 1 Sep 2022 18:18:19 -0400 Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07B0B5A8BE for ; Thu, 1 Sep 2022 15:18:06 -0700 (PDT) Received: by mail-qt1-x82f.google.com with SMTP id c20so221190qtw.8 for ; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=GmZL9QRJG3JGAUAVfGfSTA34gmVVbLMug2meh+hLHJM=; b=dWo8eLEyEw5jcTSOIhe0J42iN+xFaer3fDsss15lmAIIREY+MawFShJoU5lQXs152W 5bBsVEo8OeqyIN1jsIXSjKgdCCZW0EZC9yD75V/Uy2UCsKQA1A2x2/RnRq2779uXc7NO fuaowWwCJUf5JBRU4sL3dTWtVU2pKz7JGf3F4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=GmZL9QRJG3JGAUAVfGfSTA34gmVVbLMug2meh+hLHJM=; b=5cd18HgW2qVVt4ENrpZtH6OU04mw3Ox4fz4FehfRL3HeOy/Gbj91ETvdsmbKTOcYDF QHBK0brzAnWc11Qr6uoTqlXOsKxNX8swIazEdMDae3bg9wT/u4wfNuPgEJD+V1UVrjrT EG/pH9dWLq2X9n+x9JKtJccv7ifkHY3tfbFkcdDw1sQvv3W8iMn3sy+QhofvxZv5Y4tg B+3MDs27yfO5L/hzHEHRSr/s0+8vg/RKGBtoriuK2hg8BMExN2EcbDYhOgJn4vW24X8n rIVRUJzjpHj3CD9tF4E6PSl/QUAd4ZibWFqPDaC8ydQv4jPaBCA6Php9jFP2Zj3ayEj1 ilDA== X-Gm-Message-State: ACgBeo0F1sRDGf6XtIKYVr4ysyju6PE89fSsazmAokwDmWVCKVzp2k5x gl0V67nVYmCrstrgRyHfYSQldA== X-Google-Smtp-Source: AA6agR5d0+TO4TcTHxSbXssriWeXw2oFTaGpGLqzcaqhAOpXB66/FUakNqneNIoAQAkF8kDVHuirhQ== X-Received: by 2002:a05:622a:3d0:b0:343:58db:d5c7 with SMTP id k16-20020a05622a03d000b0034358dbd5c7mr25537687qtx.21.1662070685069; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:04 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 11/18] fs: Move call_rcu() to call_rcu_lazy() in some paths Date: Thu, 1 Sep 2022 22:17:13 +0000 Message-Id: <20220901221720.1105021-12-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. When testing, we found that these paths were invoked often when the system is not doing anything (screen is ON but otherwise idle). Signed-off-by: Joel Fernandes (Google) --- fs/dcache.c | 4 ++-- fs/eventpoll.c | 2 +- fs/file_table.c | 2 +- fs/inode.c | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index c5dc32a59c76..28a159a7e460 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -366,7 +366,7 @@ static void dentry_free(struct dentry *dentry) if (unlikely(dname_external(dentry))) { struct external_name *p =3D external_name(dentry); if (likely(atomic_dec_and_test(&p->u.count))) { - call_rcu(&dentry->d_u.d_rcu, __d_free_external); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free_external); return; } } @@ -374,7 +374,7 @@ static void dentry_free(struct dentry *dentry) if (dentry->d_flags & DCACHE_NORCU) __d_free(&dentry->d_u.d_rcu); else - call_rcu(&dentry->d_u.d_rcu, __d_free); + call_rcu_lazy(&dentry->d_u.d_rcu, __d_free); } =20 /* diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 8b56b94e2f56..530e62538d3d 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -728,7 +728,7 @@ static int ep_remove(struct eventpoll *ep, struct epite= m *epi) * ep->mtx. The rcu read side, reverse_path_check_proc(), does not make * use of the rbn field. */ - call_rcu(&epi->rcu, epi_rcu_free); + call_rcu_lazy(&epi->rcu, epi_rcu_free); =20 percpu_counter_dec(&ep->user->epoll_watches); =20 diff --git a/fs/file_table.c b/fs/file_table.c index 99c6796c9f28..36b3de669771 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -56,7 +56,7 @@ static inline void file_free(struct file *f) security_file_free(f); if (!(f->f_mode & FMODE_NOACCOUNT)) percpu_counter_dec(&nr_files); - call_rcu(&f->f_rcuhead, file_free_rcu); + call_rcu_lazy(&f->f_rcuhead, file_free_rcu); } =20 /* diff --git a/fs/inode.c b/fs/inode.c index 6462276dfdf0..7d1573cf3a97 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -312,7 +312,7 @@ static void destroy_inode(struct inode *inode) return; } inode->free_inode =3D ops->free_inode; - call_rcu(&inode->i_rcu, i_callback); + call_rcu_lazy(&inode->i_rcu, i_callback); } =20 /** --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B665ECAAD3 for ; Thu, 1 Sep 2022 22:19:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234560AbiIAWTQ (ORCPT ); Thu, 1 Sep 2022 18:19:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234707AbiIAWSK (ORCPT ); Thu, 1 Sep 2022 18:18:10 -0400 Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDC8E5AC74 for ; Thu, 1 Sep 2022 15:18:06 -0700 (PDT) Received: by mail-qt1-x82b.google.com with SMTP id h21so243254qta.3 for ; Thu, 01 Sep 2022 15:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=TrK5dOUuJ472Va/MQxlZM4Ul27mMRNSxArIaJwpPfZg=; b=TUcC0u4QM5Ny9KLvPnrxQITVTokuBxEATE/WsQfbGYFGD9PpxIg9MJCiMdUCQf7Laq X6rikyqdDpvNk6vlDW0YVq5/0S8OJYELbrNlakftwrW0WN0AsYlxa096qY1FJJBELlxI jL2uVxkUi+pIbxFoV4OaGYaWTNiex9ahiveZ0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=TrK5dOUuJ472Va/MQxlZM4Ul27mMRNSxArIaJwpPfZg=; b=B99T5Fl/DOrjsYWPiMhiItZeY72mgk5+OsEVmEF2WqUDntPIim4pt9T2BVUsSBmlma iFKCagbFNObWAgWqmi9tw4lzhRRL4Ca5h9AOJag1ciDlfxnHweZi1naNO7jnTfO/19dL 3crFzGZURie3+muTKmYXuks2rEUb65mXK4SCQNSmrOyVG4b/H1FOZsQU/cIbWCIUDgbP 1otpRcAqfQMFe8g2bR0K6I/h1U97mSc5QEuVulFQaaLquKibY3Ob46aDgV+QRmsyhTyE SaT8R+YeZnkugerfR2y/E+m7niviq23nKlycYNsaj4JbRY5UKC7gBSDTBf9FdRt09eSz M7Ug== X-Gm-Message-State: ACgBeo3D3UP5MVbG+cqlT0/xt4dotyhG2t1RKYlqI8CJxNoTKrtFgurT mlY50zpOYJtz9X5qrL9hQngARQ== X-Google-Smtp-Source: AA6agR4uiuD53PR2JnbYvQYDsN+ofsO7SW63BGOkp2t0qORapVEs6g6ATsIcI2VvrBk+/fuLb48KeQ== X-Received: by 2002:a05:622a:53:b0:344:6f46:9b16 with SMTP id y19-20020a05622a005300b003446f469b16mr26319627qtw.664.1662070685785; Thu, 01 Sep 2022 15:18:05 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:05 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 12/18] cred: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:14 +0000 Message-Id: <20220901221720.1105021-13-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- kernel/cred.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/cred.c b/kernel/cred.c index e10c15f51c1f..c7cb2e3ac73a 100644 --- a/kernel/cred.c +++ b/kernel/cred.c @@ -150,7 +150,7 @@ void __put_cred(struct cred *cred) if (cred->non_rcu) put_cred_rcu(&cred->rcu); else - call_rcu(&cred->rcu, put_cred_rcu); + call_rcu_lazy(&cred->rcu, put_cred_rcu); } EXPORT_SYMBOL(__put_cred); =20 --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 745F7ECAAD2 for ; Thu, 1 Sep 2022 22:19:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235060AbiIAWTg (ORCPT ); Thu, 1 Sep 2022 18:19:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234728AbiIAWSL (ORCPT ); Thu, 1 Sep 2022 18:18:11 -0400 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77F395C9D3 for ; Thu, 1 Sep 2022 15:18:07 -0700 (PDT) Received: by mail-qt1-x832.google.com with SMTP id x5so219442qtv.9 for ; Thu, 01 Sep 2022 15:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=5xn60rDjdB3Rf7Wir8y+6b2hKXDgCuU2pqQZqimv/ck=; b=Os4hTQX+7UasiETS+y9NSvg4ja5hDOYYPQYGOfJ5sv7/j1Yu4A2w2gXnSlW0jOFCDp bLTIIkeTRy+1cbbBwSjwl3rHEw4Ocq3dzKdcSC+TunfEtEl+yuu5I44XNzQiXUAY3Ewk QEeYCOdMfWSgWfckJw059MAJSVHel1agDUyiA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=5xn60rDjdB3Rf7Wir8y+6b2hKXDgCuU2pqQZqimv/ck=; b=g+vn5cWP5fzsVmqzSQlqg3qIkJ6gOuLuBOGQtYWFBLSGluxbFyzAlJhW5F6jhoUH7R TvXTu66rvuH6yjMv8Lc21OIA2i5vxOK+DLIepD8/fkltm2LGzqncIWkSRD++iKtDmS6g szxT4UMXt42oF5PjmLA7ZLa2c5s13zWNuIizzMv0nPCVNhAWyz8NE+L7xDcqdqPO4YEa AJUFyWQ0rm84CMTR8Qd6KhxLCGijpVdEbyxu+ATDJVeBAiX1ALCxdXVRhvUjCbaqJW75 X7dWS3xxpDl7OfSBlJkUT9y8CfSYht5+jNak0/MnQfxrrzCD6dv+orseA1V1TYWyvc4W RQOw== X-Gm-Message-State: ACgBeo1tGvsUoujeZ8vuu2CtjxhvEoX7q22PJ50yO1dzf9aHxRiYetc8 4MVqdBjtPJBmlw4cg0S5vKHgTw== X-Google-Smtp-Source: AA6agR4diTGOE7swnbaEjFmZk8h1Ql35CL92QOFXkcefETWvplUeM3qM08T8V0Y69YU08rm48PY+vQ== X-Received: by 2002:ac8:4e45:0:b0:343:5faf:3af6 with SMTP id e5-20020ac84e45000000b003435faf3af6mr26987399qtw.340.1662070686395; Thu, 01 Sep 2022 15:18:06 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:06 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 13/18] security: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:15 +0000 Message-Id: <20220901221720.1105021-14-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- security/security.c | 2 +- security/selinux/avc.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/security/security.c b/security/security.c index 14d30fec8a00..b51b4bdb567d 100644 --- a/security/security.c +++ b/security/security.c @@ -1053,7 +1053,7 @@ void security_inode_free(struct inode *inode) * The inode will be freed after the RCU grace period too. */ if (inode->i_security) - call_rcu((struct rcu_head *)inode->i_security, + call_rcu_lazy((struct rcu_head *)inode->i_security, inode_free_by_rcu); } =20 diff --git a/security/selinux/avc.c b/security/selinux/avc.c index 9a43af0ebd7d..381f046d820f 100644 --- a/security/selinux/avc.c +++ b/security/selinux/avc.c @@ -442,7 +442,7 @@ static void avc_node_free(struct rcu_head *rhead) static void avc_node_delete(struct selinux_avc *avc, struct avc_node *node) { hlist_del_rcu(&node->list); - call_rcu(&node->rhead, avc_node_free); + call_rcu_lazy(&node->rhead, avc_node_free); atomic_dec(&avc->avc_cache.active_nodes); } =20 @@ -458,7 +458,7 @@ static void avc_node_replace(struct selinux_avc *avc, struct avc_node *new, struct avc_node *old) { hlist_replace_rcu(&old->list, &new->list); - call_rcu(&old->rhead, avc_node_free); + call_rcu_lazy(&old->rhead, avc_node_free); atomic_dec(&avc->avc_cache.active_nodes); } =20 --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C8ACC54EE9 for ; Thu, 1 Sep 2022 22:19:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235182AbiIAWTV (ORCPT ); Thu, 1 Sep 2022 18:19:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234773AbiIAWST (ORCPT ); Thu, 1 Sep 2022 18:18:19 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D1F55A14D for ; Thu, 1 Sep 2022 15:18:08 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id w28so224184qtc.7 for ; Thu, 01 Sep 2022 15:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Uetgjnmco7XMjJ0pPvFMMYQeK2oqyNBCbg9zI/9WK6U=; b=Qlh767M6LCeX0fcTUExKI0vDqoTRBfz2hg5EurE23Vs4l8vVL/qjU47CREwknrDU0m rLGyL/hdoXaQ3k8Vo5xqmJuRolH5BcV8urpgipRpHre9ncZoZRuwvvGiH/z84Xkf9PKx w32OhzRkeqYAp5tKbH/fi2wbMWp9nPLv/U/O4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Uetgjnmco7XMjJ0pPvFMMYQeK2oqyNBCbg9zI/9WK6U=; b=nq3aUTRsqln4odhSFxpqdPRDu+HuWBYF42wnXC79LbJuEtstcIh4THFhXAAoJ/VDE1 1I+BnXMhiAjy3HU0gALHS1/gnlZYiWSo3ZHZDpfQuvGzYwhD8Ie96lkrSJRdneNPeJzP kIWh8yRjDS0o+TZUesWJ0JeUNYhS0U/GERnutG3S7o5UJiaFO396pS/rGfPm4G18fSC3 Od/xC+2Nwhs7M2ftmex++6BbFxAJiJFg4e7FCfV3oDZzbJy6Fc6MGbUUHgzdHMPsLK0+ nhNE6bNWOXIVpIohayroCibqDTUzNa0faW8SSmYDyymYfKg56SdZplXULf05FXY5VxVG xZ8g== X-Gm-Message-State: ACgBeo3txMgK7Fj5Zyhjp78AIb5sPvGKkBMYS5OJuRmQmiFtzRgzlvZB GAFw6xIqN56gi1q8+GLioTnMdw== X-Google-Smtp-Source: AA6agR7NX0LPVmvfR8AYnY7HzxGSlvcrW6s4HEA32zP4sXGb6qTrqmmBag2GXgdyln10oQkqNMt8zw== X-Received: by 2002:ac8:5a96:0:b0:344:b3be:6931 with SMTP id c22-20020ac85a96000000b00344b3be6931mr25930091qtc.509.1662070687197; Thu, 01 Sep 2022 15:18:07 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:06 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 14/18] net/core: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:16 +0000 Message-Id: <20220901221720.1105021-15-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- net/core/dst.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dst.c b/net/core/dst.c index bc9c9be4e080..babf49e413e1 100644 --- a/net/core/dst.c +++ b/net/core/dst.c @@ -174,7 +174,7 @@ void dst_release(struct dst_entry *dst) net_warn_ratelimited("%s: dst:%p refcnt:%d\n", __func__, dst, newrefcnt); if (!newrefcnt) - call_rcu(&dst->rcu_head, dst_destroy_rcu); + call_rcu_lazy(&dst->rcu_head, dst_destroy_rcu); } } EXPORT_SYMBOL(dst_release); --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98DB4ECAAD2 for ; Thu, 1 Sep 2022 22:19:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235191AbiIAWTZ (ORCPT ); Thu, 1 Sep 2022 18:19:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234779AbiIAWST (ORCPT ); Thu, 1 Sep 2022 18:18:19 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 723995809E for ; Thu, 1 Sep 2022 15:18:08 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id cb8so281045qtb.0 for ; Thu, 01 Sep 2022 15:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=3vWNR1lOZ3nlNpU4UUmxfmJYajX79MfO1ObVeBbomlg=; b=W9V/oqA+OZMtsddIG83NBCyyox5x44+ormS/AjYTPNmA6tq1LLUSu1mTv373zuQAmL stXAk/bY0nw2tv3MorrBWKQVz0OdcObvezxEjnW0fZHXmStt0csE5tLilyFNZXZtzQ7e hZ35XQl1bl0u/iHRILvxndSITlzXxbNnrQw8k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=3vWNR1lOZ3nlNpU4UUmxfmJYajX79MfO1ObVeBbomlg=; b=YSwBhQGXsFr3c7HzdnDCs3XvM9v18yOP9pVDjSaewIhoroEHKWjRd6tSCHTUVhfF9M WYCwNGPA4Qfulx5tY2jvztWREoWyQyGo3pyk1MsAYX27YO0dhu0VSi7kebZLf+WTFVD/ fHz9J/VpNkNDxYiR1aYGP62Y2BRYYMz6T3VqlYFAwZ6IiVI6kBPcf4xooYCnANVUJL8M r5aHTlq3iAEUtUVW5LY7eX5uGSL8j4fg0a3pZ8WENoK1RgD/tXCex1ThtZelNYYOE6IV b+zZKukM7Kg0E9rabWvq419W6f0q13vfPop8unNhoIZOGFRo5x2SAH2U4jtJjaWH91MP oDpA== X-Gm-Message-State: ACgBeo3poCL+QVpdhqj/dLPaGtkgU1Qf32e1JcNf1pDoyh0FqYaoNDEC HXAAMc/BNoCcqvXh5nIvfljFyA== X-Google-Smtp-Source: AA6agR7Md6xYqwWArMXYWwgIpGr8i6bhrZ317pWwHfXCZUjbF4/68mUbJbpxMEPOI/zS/oZi2CAc+g== X-Received: by 2002:ac8:5786:0:b0:343:3051:170d with SMTP id v6-20020ac85786000000b003433051170dmr25646460qta.429.1662070688111; Thu, 01 Sep 2022 15:18:08 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:07 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 15/18] kernel: Move various core kernel usages to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:17 +0000 Message-Id: <20220901221720.1105021-16-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Signed-off-by: Joel Fernandes (Google) --- kernel/exit.c | 2 +- kernel/pid.c | 2 +- kernel/time/posix-timers.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/exit.c b/kernel/exit.c index 84021b24f79e..b2a96356980a 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -180,7 +180,7 @@ static void delayed_put_task_struct(struct rcu_head *rh= p) void put_task_struct_rcu_user(struct task_struct *task) { if (refcount_dec_and_test(&task->rcu_users)) - call_rcu(&task->rcu, delayed_put_task_struct); + call_rcu_lazy(&task->rcu, delayed_put_task_struct); } =20 void release_task(struct task_struct *p) diff --git a/kernel/pid.c b/kernel/pid.c index 2fc0a16ec77b..5a5144519d70 100644 --- a/kernel/pid.c +++ b/kernel/pid.c @@ -153,7 +153,7 @@ void free_pid(struct pid *pid) } spin_unlock_irqrestore(&pidmap_lock, flags); =20 - call_rcu(&pid->rcu, delayed_put_pid); + call_rcu_lazy(&pid->rcu, delayed_put_pid); } =20 struct pid *alloc_pid(struct pid_namespace *ns, pid_t *set_tid, diff --git a/kernel/time/posix-timers.c b/kernel/time/posix-timers.c index 5dead89308b7..660daa427b41 100644 --- a/kernel/time/posix-timers.c +++ b/kernel/time/posix-timers.c @@ -485,7 +485,7 @@ static void release_posix_timer(struct k_itimer *tmr, i= nt it_id_set) } put_pid(tmr->it_pid); sigqueue_free(tmr->sigq); - call_rcu(&tmr->rcu, k_itimer_rcu_free); + call_rcu_lazy(&tmr->rcu, k_itimer_rcu_free); } =20 static int common_timer_create(struct k_itimer *new_timer) --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD28FECAAD3 for ; Thu, 1 Sep 2022 22:19:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235199AbiIAWT2 (ORCPT ); Thu, 1 Sep 2022 18:19:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234857AbiIAWS3 (ORCPT ); Thu, 1 Sep 2022 18:18:29 -0400 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 091E75C35B for ; Thu, 1 Sep 2022 15:18:10 -0700 (PDT) Received: by mail-qt1-x833.google.com with SMTP id x5so219538qtv.9 for ; Thu, 01 Sep 2022 15:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=dpVq45gOBHghNC476XxsHo12GSN2n+TlCs0w1yjI1G8=; b=LHvRA8QK5VFw/jrPAKHEmEEsl0L+oZGGdwvCDSEY1J7Zf3XW17vgFCaaiJ5p+DinkX JywVDBQDKWVZmOUD+PWJONyE7y3Pj72l2fcvyocgTGJqG/OsejHHXd4qGxJCj3m5m77v d3hMh3KrXjEdiRpUQMsxM9gI5WAwTmvMZcliE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=dpVq45gOBHghNC476XxsHo12GSN2n+TlCs0w1yjI1G8=; b=epLbq9savlHc+5AnexDp/L4NpU/3z5pl1BS2U+RkNXy2Zm16ouTC9WAxcs3MvTXL0C mvNT1QsrTR/j5oFEFERpTkkd+7abXITIdxmhXVFym124Bm+Qbt4SzhtMPCy0FHPX5ska avRW1q1lWoVLbtoMcZ+IARbPG4qhiYD4UuUyejdu6/8bZRiaSvA1BLqsw6pxqWpHXhu/ bB5PrJkubxGhWYbcSsTx6zFy33PPYtDuwG3Snwn+SV0PNi3niDpNb9CcCdeni41ngmJ5 WJYdUSO/7qfi5Zh7ArRUnpCPFwweGWcrCz38QN3Bg2R9aTwXs6xRzMBRmnmyWkR648UW bD/w== X-Gm-Message-State: ACgBeo0pjEWxMltqhhhqQtbyEWGMUX/fl1CA6BCkoiK3b7RQ33RfKJbR HYCZkLPMdmdVPhcNlXj4ZnyP2A== X-Google-Smtp-Source: AA6agR5fwM2qBQofpQTaAhwXy9MXIukc637mu6NfIRAgZbUg8vF1uoX4k76JfSjiT21Y9vviNS6TMg== X-Received: by 2002:ac8:4e4a:0:b0:343:7e05:d2a8 with SMTP id e10-20020ac84e4a000000b003437e05d2a8mr26201508qtw.67.1662070689060; Thu, 01 Sep 2022 15:18:09 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:08 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 16/18] lib: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:18 +0000 Message-Id: <20220901221720.1105021-17-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Move radix-tree and xarray to call_rcu_lazy(). This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- lib/radix-tree.c | 2 +- lib/xarray.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 3c78e1e8b2ad..49c821b16acc 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -305,7 +305,7 @@ void radix_tree_node_rcu_free(struct rcu_head *head) static inline void radix_tree_node_free(struct radix_tree_node *node) { - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu_lazy(&node->rcu_head, radix_tree_node_rcu_free); } =20 /* diff --git a/lib/xarray.c b/lib/xarray.c index ea9ce1f0b386..230abc8045fe 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -257,7 +257,7 @@ static void xa_node_free(struct xa_node *node) { XA_NODE_BUG_ON(node, !list_empty(&node->private_list)); node->array =3D XA_RCU_FREE; - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu_lazy(&node->rcu_head, radix_tree_node_rcu_free); } =20 /* --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27ACBECAAD3 for ; Thu, 1 Sep 2022 22:19:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235250AbiIAWTl (ORCPT ); Thu, 1 Sep 2022 18:19:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234915AbiIAWSl (ORCPT ); Thu, 1 Sep 2022 18:18:41 -0400 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39B1D647E6 for ; Thu, 1 Sep 2022 15:18:10 -0700 (PDT) Received: by mail-qt1-x833.google.com with SMTP id j17so205626qtp.12 for ; Thu, 01 Sep 2022 15:18:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=uvMol3uyneGva4YyiwYlT7IjMY9wNfnMaY4BpajVnC0=; b=vB0TPhC+ETNKaorpvy0zXtEQMbIqrE0wkU/h5SP9iHJyWzeG/Kp/jpHxx6dt746Npa QOHvSTUY4nY6DTxMILIli4C/sMT/xiS5q23jK7GsK9NDcZWBJDyMJp2HL8fP0VFkqEZF VvTw7as9Jv9mHmLJUawLZKjNlgVaU2bZ1YoPg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=uvMol3uyneGva4YyiwYlT7IjMY9wNfnMaY4BpajVnC0=; b=iG2wZMNrnhD7EMdR7MZEro8OvjVkmNYmNaQOcaCOQUI8KLIXmoTdOunFjpzbmsbKpK 3BoepUc5n91lG+IGn4lie3ccOteBtbna4LRIQWf/Qxm492+29aeUTxcthv8X7CX7LKL2 pa5DkPNQV/XFjylqZUFyG3NZSqAoHi7Wbh3P9Nx2kFHdRKY70IiqDNWrJizwy4VVBf++ lVy8ZdJwY2+NCXb1VjpLJhj/UyYl9riHkgZwaTxdijhzYo1Qu8nywoTVelTtV5+h/Otj H1eJYWkU6f8YgHGyCyStQiUUSmW8+XQ8/UFZbT8F1Z/d9HKXufOWGyZQKl7X5nziK2G6 n+MA== X-Gm-Message-State: ACgBeo08PKGRAlRBoPuTN2X5AmcrWjpSwzYvaHUFQX+v8075JSHcn+IW OuutqkdD8roeEEm3fNlMBYAIaA== X-Google-Smtp-Source: AA6agR7IUlccHMDPHEeKk4mHGnWwUpV/eTK+kjnIV8R3LGYG/QjIQq6jZqPANDerA0E/GIkXT0vkdA== X-Received: by 2002:ac8:58c8:0:b0:344:87c2:c495 with SMTP id u8-20020ac858c8000000b0034487c2c495mr25705752qta.631.1662070689867; Thu, 01 Sep 2022 15:18:09 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:09 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 17/18] i915: Move call_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:19 +0000 Message-Id: <20220901221720.1105021-18-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- drivers/gpu/drm/i915/gem/i915_gem_object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i= 915/gem/i915_gem_object.c index ccec4055fde3..21d6f66fb394 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c @@ -343,7 +343,7 @@ static void __i915_gem_free_objects(struct drm_i915_pri= vate *i915, __i915_gem_free_object(obj); =20 /* But keep the pointer alive for RCU-protected lookups */ - call_rcu(&obj->rcu, __i915_gem_free_object_rcu); + call_rcu_lazy(&obj->rcu, __i915_gem_free_object_rcu); cond_resched(); } } --=20 2.37.2.789.g6183377224-goog From nobody Mon Apr 6 21:33:53 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E259ECAAD3 for ; Thu, 1 Sep 2022 22:19:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235255AbiIAWTr (ORCPT ); Thu, 1 Sep 2022 18:19:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234947AbiIAWSm (ORCPT ); Thu, 1 Sep 2022 18:18:42 -0400 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5C786B8F2 for ; Thu, 1 Sep 2022 15:18:11 -0700 (PDT) Received: by mail-qt1-x82e.google.com with SMTP id w28so224295qtc.7 for ; Thu, 01 Sep 2022 15:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=8gmIjQiZKF8ycgdoC5ApOjU5UIrSl/T2FqJa/TgPO1I=; b=Gi604wSkIg+z78h4KmndskftB7f3p4DdvyE1ILism9RQ5yjlyt75A9qW/StOXfb+fU pDy5O0BpNJsnx5TS3sl3liH6GLjNaUxC14UjaYlRGEFOLzqWV+fbldv/OdhgigmrP6sa icbHfTEFZStTZC5sYgXRiVe8Q6pqgB7yD7VlM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8gmIjQiZKF8ycgdoC5ApOjU5UIrSl/T2FqJa/TgPO1I=; b=HzFBGvM4KYE8hTVtaXlQFlalw+xq3m2rOxgVIhc95fNc47i5b0qeOUXfXEjdjFVFZE nsrB+TBjLOGB2lyf42R8SjOiKh0ROjPEU/ymVRCc+EDhfy15xI4ZGGqfk2tXxWr71ToU wcaaTBkWVyEz7xvkiGZr8QogVDe5LrJ+uZtnshCg6sW9OssCwg9aojkWDGHv4Wiq7P2Q /ntMVGisSqwycZWcNabppft8OrnC6uKWZYYYYjDcL/uoKneEG+MqxTdi+2YBQsJUl1U6 /XF6bgNUh729VGJ8qAHbi1VUMQkqo38TpWGe8w4Q3HtWqP4eiLaQ69n7w0Ap32TTNor7 jl6g== X-Gm-Message-State: ACgBeo1yeO/G882omDGq8GL0np0AY5nkfkTVVP7Bjfw7P87+n51YuJkY HVpm2T6/iWjcHkMiWKcM9acOjw5kBeIyOA== X-Google-Smtp-Source: AA6agR7ds55vxuronbAkhPvG+w3tnnySL6akfA2/bMY5WcFbqrnJ09MHDN2EqtY3lODJWuD4wHe3wA== X-Received: by 2002:a05:622a:198b:b0:344:7de5:c516 with SMTP id u11-20020a05622a198b00b003447de5c516mr25834335qtc.7.1662070690604; Thu, 01 Sep 2022 15:18:10 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id s16-20020ac85290000000b0034305a91aaesm11060794qtn.83.2022.09.01.15.18.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 15:18:10 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com, urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org, paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org, boqun.feng@gmail.com, "Joel Fernandes (Google)" Subject: [PATCH v5 18/18] fork: Move thread_stack_free_rcu() to call_rcu_lazy() Date: Thu, 1 Sep 2022 22:17:20 +0000 Message-Id: <20220901221720.1105021-19-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220901221720.1105021-1-joel@joelfernandes.org> References: <20220901221720.1105021-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is required to prevent callbacks triggering RCU machinery too quickly and too often, which adds more power to the system. Signed-off-by: Joel Fernandes (Google) --- kernel/fork.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 90c85b17bf69..08a2298b1f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -226,7 +226,7 @@ static void thread_stack_delayed_free(struct task_struc= t *tsk) struct vm_stack *vm_stack =3D tsk->stack; =20 vm_stack->stack_vm_area =3D tsk->stack_vm_area; - call_rcu(&vm_stack->rcu, thread_stack_free_rcu); + call_rcu_lazy(&vm_stack->rcu, thread_stack_free_rcu); } =20 static int free_vm_stack_cache(unsigned int cpu) @@ -353,7 +353,7 @@ static void thread_stack_delayed_free(struct task_struc= t *tsk) { struct rcu_head *rh =3D tsk->stack; =20 - call_rcu(rh, thread_stack_free_rcu); + call_rcu_lazy(rh, thread_stack_free_rcu); } =20 static int alloc_thread_stack_node(struct task_struct *tsk, int node) @@ -388,7 +388,7 @@ static void thread_stack_delayed_free(struct task_struc= t *tsk) { struct rcu_head *rh =3D tsk->stack; =20 - call_rcu(rh, thread_stack_free_rcu); + call_rcu_lazy(rh, thread_stack_free_rcu); } =20 static int alloc_thread_stack_node(struct task_struct *tsk, int node) --=20 2.37.2.789.g6183377224-goog