From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46C951D90DF; Wed, 25 Jun 2025 03:11:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821090; cv=none; b=tgtbVTZVUu0X7UzCuyZWQIhScrtOGn5W62z+csnIXw7ywSc+NQXh6E2jjQeQOPJZMTsBJ7eayJAJRIkBeK+Mw+dSh/dNd7CzLq8oqRLpg3qCRUx9ZjRoVqTmejEpK3Q/Y+JAiRVi6OK9IcGr/aARm78S+OsAt+jUornuIPgsBRY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821090; c=relaxed/simple; bh=TpNMyZrlm4tnbU8qIbQP9OAezqCkZ8eRw46ditrX03g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cc84KsUrCKWTEIX1R6DV6oYpVOacXdkZ59/kYM1FViZTGyZzigbJD3WbRPVYFQyInaWoJg0xVnVx9eAwB23E8AuuGC+8MICmRZCeDYeySe4Y7UCY/rL6R0kVvH/y/YCAqDLj/+x5/Gny7JU2dm/pgf15IdjESpPA42VT3pT76V8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=j9ItMkGz; arc=none smtp.client-ip=209.85.219.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="j9ItMkGz" Received: by mail-qv1-f43.google.com with SMTP id 6a1803df08f44-6fafdd322d3so68603096d6.3; Tue, 24 Jun 2025 20:11:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821087; x=1751425887; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=boywPtnWXFrorYFTYWBJNeUbr+NvAx7NTSh4JFed8T4=; b=j9ItMkGz0FixI+Qa7A44usw4bQ/lLJM2tmoKM0qTAt2z2/mLtqWfMAF2B3kzY8PzRp Mb8EjXF/HUU+7nYBxfnvyoqT3S0Xhi/nvLBoDpNbSaiR6ThecK7LOiMuEj1TEY9IBGy/ ac8Hdi5L7DI8m6qxwLylzsrUApGCsj9uXcMBOHuYHxdlGbzfVLHcD5qKuSqq6MR84Znz Xx/uDimfaxgubDfpeM1B4ahtriAfAeavW5dxSiZGqxRcS+Q9EvksiSCiDDRsI/S2VWz4 ZLEsOLDrQXxM5V17QuseZRya93ma8E8tlVh73AZ+TRhoAAcFTYYCjTukQqvPJxGj/erl A/eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821087; x=1751425887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=boywPtnWXFrorYFTYWBJNeUbr+NvAx7NTSh4JFed8T4=; b=tOQ6gM+QshLUq5u2IHHpcrwDDwnq/XHVtt6D1hgDdclEzzbjSidS+mV+s9A5CX8u3H doXcOFKHCEPps+Ysg/1T0ZqBsB7NfxlBvL/GCg01Mc/7RRJb1ReVJRofa6lDBZWLTDlh Dkk6KOV80hEjfyOVS/+TuElLKDG8zB/MKPkyTc9uLdk7Rad/ZDRk5NqbJX5q0pe0WF3s KZXXBHX1ImScJczlzCyPbaxgjhQvWXkFUEqMwwJnIynVHRYnYNkWjMsemeFF89L5/7PV oeKUrqmsVjvMjoFl3bnzI/xH2v0iK0jYLTNELdEQPjKBTjj2X5nF/RqxA3lMKpHOyjeQ pIFw== X-Forwarded-Encrypted: i=1; AJvYcCWZebJT/5CE3yD8JRJ1h+ZYcF+xQXXrGCWaSA4QIs8z8RkvdBCu7LaNj4fDNV9SaM3ijQ3A@vger.kernel.org, AJvYcCWrFgEizx9g0EFWSS2R2m06cSl5pW6cXrsdYjSddS7d94oV99u4Rfz9nJLiJPs7lnwaceHGUMM=@vger.kernel.org X-Gm-Message-State: AOJu0YwNhrH8ZrDqkk+tll1rvMwtlmw/c+pDdkINRJF29PIDXQoq87CV u5Vw0MralNuIK/wlKKh4PUA90QRlPfInS7eZJIjnCT6O2KUmHsXHGzeh X-Gm-Gg: ASbGncsUKfFlVRwsD5nEohsJJ0NS3hSgLCP6vh4OvUERVXAKsDr4p52ctKZx+sHipPR H+KhvgxVB0z8h3Wn8lp3oPmYtndYGfv75PdZsjxcyeKrXDoFPSbq7RRcta12EYJSFpltHED9S4g drrbLUZ/FCAd7LeC8I/vtm5a9nYwEfqykolWdnoFL5tdkuQ4GQmlpks24o242mT+vxHg86TGz6R mmPDkroWumxq/YTUuL4PPIOx6HrjxeVnMBmxllMmQ0V3asOnoxdlOCa+K28nfRPA91cwrtMH9Mq Vopat+OMYUpk14m5428N2h2LFNo9RyxfQJnVjh4n6c+zemDj+KBHhMM2ICVZkartHSrEmqiNUlF CeFZuKpfu1ttRl/76bv8pRNj4/uGLrZw6OwPy+ls3rqyFCrrUum6f X-Google-Smtp-Source: AGHT+IGSm71fAe04SBpyuIDvnyyRGSPKtLQRNbYp3JCDXDkvCoRWay4YKL+d8dRstEpJV4yfKMo0Cg== X-Received: by 2002:ad4:5f85:0:b0:6fa:c81a:621b with SMTP id 6a1803df08f44-6fd5ef42043mr21228666d6.14.1750821087055; Tue, 24 Jun 2025 20:11:27 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6fd1c2c544bsm43838116d6.30.2025.06.24.20.11.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:26 -0700 (PDT) Received: from phl-compute-09.internal (phl-compute-09.phl.internal [10.202.2.49]) by mailfauth.phl.internal (Postfix) with ESMTP id 17AD4F40066; Tue, 24 Jun 2025 23:11:26 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-09.internal (MEProxy); Tue, 24 Jun 2025 23:11:26 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeihecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgffhffevhffhvdfgjefgkedvlefgkeegveeuheelhfeivdegffejgfetuefgheei necuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghr shhonhgrlhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvg hngheppehgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohep vdeipdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehlihhnuhigqdhkvghrnhgvlh esvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehrtghusehvghgvrhdrkhgv rhhnvghlrdhorhhgpdhrtghpthhtoheplhhkmhhmsehlihhsthhsrdhlihhnuhigrdguvg hvpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgoheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopegsohhquhhnrdhfvghnghesghhmrghilhdrtghomhdp rhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgtphhtthhopegurg hvvgesshhtghholhgrsghsrdhnvght X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:24 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 1/8] Introduce simple hazard pointers Date: Tue, 24 Jun 2025 20:10:54 -0700 Message-Id: <20250625031101.12555-2-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" As its name suggests, simple hazard pointers (shazptr) is a simplification of hazard pointers [1]: it has only one hazard pointer slot per-CPU and is targeted for simple use cases where the read-side already has preemption disabled. It's a trade-off between full features of a normal hazard pointer implementation (multiple slots, dynamic slot allocation, etc.) and the simple use scenario. Since there's only one slot per-CPU, so shazptr read-side critical section nesting is a problem that needs to be resolved, because at very least, interrupts and NMI can introduce nested shazptr read-side critical sections. A SHAZPTR_WILDCARD is introduced to resolve this: SHAZPTR_WILDCARD is a special address value that blocks *all* shazptr waiters. In an interrupt-causing shazptr read-side critical section nesting case (i.e. an interrupt happens while the per-CPU hazard pointer slot being used and tries to acquire a hazard pointer itself), the inner critical section will switch the value of the hazard pointer slot into SHAZPTR_WILDCARD, and let the outer critical section eventually zero the slot. The SHAZPTR_WILDCARD still provide the correct protection because it blocks all the waiters. It's true that once the wildcard mechanism is activated, shazptr mechanism may be downgrade to something similar to RCU (and probably with a worse implementation), which generally has longer wait time and larger memory footprint compared to a typical hazard pointer implementation. However, that can only happen with a lot of users using hazard pointers, and then it's reasonable to introduce the fully-featured hazard pointer implementation [2] and switch users to it. Note that shazptr_protect() may be added later, the current potential usage doesn't require it, and a shazptr_acquire(), which installs the protected value to hazard pointer slot and proves the smp_mb(), is enough for now. [1]: M. M. Michael, "Hazard pointers: safe memory reclamation for lock-free objects," in IEEE Transactions on Parallel and Distributed Systems, vol. 15, no. 6, pp. 491-504, June 2004 Link: https://lore.kernel.org/lkml/20240917143402.930114-1-boqun.feng@gmail= .com/ [2] Signed-off-by: Boqun Feng --- include/linux/shazptr.h | 73 ++++++++++++++++++++++++++++++++++++++++ kernel/locking/Makefile | 2 +- kernel/locking/shazptr.c | 29 ++++++++++++++++ 3 files changed, 103 insertions(+), 1 deletion(-) create mode 100644 include/linux/shazptr.h create mode 100644 kernel/locking/shazptr.c diff --git a/include/linux/shazptr.h b/include/linux/shazptr.h new file mode 100644 index 000000000000..287cd04b4be9 --- /dev/null +++ b/include/linux/shazptr.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Simple hazard pointers + * + * Copyright (c) 2025, Microsoft Corporation. + * + * Author: Boqun Feng + * + * A simple variant of hazard pointers, the users must ensure the preempti= on + * is already disabled when calling a shazptr_acquire() to protect an addr= ess. + * If one shazptr_acquire() is called after another shazptr_acquire() has = been + * called without the corresponding shazptr_clear() has been called, the l= ater + * shazptr_acquire() must be cleared first. + * + * The most suitable usage is when only one address need to be protected i= n a + * preemption disabled critical section. + */ + +#ifndef _LINUX_SHAZPTR_H +#define _LINUX_SHAZPTR_H + +#include +#include + +/* Make ULONG_MAX the wildcard value */ +#define SHAZPTR_WILDCARD ((void *)(ULONG_MAX)) + +DECLARE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); + +/* Represent a held hazard pointer slot */ +struct shazptr_guard { + void **slot; + bool use_wildcard; +}; + +/* + * Acquire a hazptr slot and begin the hazard pointer critical section. + * + * Must be called with preemption disabled, and preemption must remain dis= abled + * until shazptr_clear(). + */ +static inline struct shazptr_guard shazptr_acquire(void *ptr) +{ + struct shazptr_guard guard =3D { + /* Preemption is disabled. */ + .slot =3D this_cpu_ptr(&shazptr_slots), + .use_wildcard =3D false, + }; + + if (likely(!READ_ONCE(*guard.slot))) { + WRITE_ONCE(*guard.slot, ptr); + } else { + guard.use_wildcard =3D true; + WRITE_ONCE(*guard.slot, SHAZPTR_WILDCARD); + } + + smp_mb(); /* Synchronize with smp_mb() at synchronize_shazptr(). */ + + return guard; +} + +static inline void shazptr_clear(struct shazptr_guard guard) +{ + /* Only clear the slot when the outermost guard is released */ + if (likely(!guard.use_wildcard)) + smp_store_release(guard.slot, NULL); /* Pair with ACQUIRE at synchronize= _shazptr() */ +} + +void synchronize_shazptr(void *ptr); + +DEFINE_CLASS(shazptr, struct shazptr_guard, shazptr_clear(_T), + shazptr_acquire(ptr), void *ptr); +#endif diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile index a114949eeed5..1517076c98ec 100644 --- a/kernel/locking/Makefile +++ b/kernel/locking/Makefile @@ -3,7 +3,7 @@ # and is generally not a function of system call inputs. KCOV_INSTRUMENT :=3D n =20 -obj-y +=3D mutex.o semaphore.o rwsem.o percpu-rwsem.o +obj-y +=3D mutex.o semaphore.o rwsem.o percpu-rwsem.o shazptr.o =20 # Avoid recursion lockdep -> sanitizer -> ... -> lockdep & improve perform= ance. KASAN_SANITIZE_lockdep.o :=3D n diff --git a/kernel/locking/shazptr.c b/kernel/locking/shazptr.c new file mode 100644 index 000000000000..991fd1a05cfd --- /dev/null +++ b/kernel/locking/shazptr.c @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Simple hazard pointers + * + * Copyright (c) 2025, Microsoft Corporation. + * + * Author: Boqun Feng + */ + +#include +#include +#include + +DEFINE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); +EXPORT_PER_CPU_SYMBOL_GPL(shazptr_slots); + +void synchronize_shazptr(void *ptr) +{ + int cpu; + + smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + /* Pair with smp_store_release() in shazptr_clear(). */ + smp_cond_load_acquire(slot, + VAL !=3D ptr && VAL !=3D SHAZPTR_WILDCARD); + } +} +EXPORT_SYMBOL_GPL(synchronize_shazptr); --=20 2.39.5 (Apple Git-154) From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACCF91DDA32; Wed, 25 Jun 2025 03:11:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821091; cv=none; b=T2b3BUtWc8c/ozN73RYJy48fM0u1BIfdfEvJsbHE2t30ALk00rP72tvNkUY0VnwnmxOr7OFZgHoMEdxWCvO1mIrojZXZNQOWcFdiM5ahag3nV/w4p6hcvq/ZXYs0Syz8gUjMjMZLwITV1EXH5saFqzDbVS2MjIBMJ+gSpktwJP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821091; c=relaxed/simple; bh=duTW7lZrJNtK7UEEDJSzyS45AgiLLDWdc/06eYlkxEM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t/AKuW9Ow2r/SZJ99vGkkrS8l5w4V8t+rKYrLYd0a1T8iXSHghifOH0IlSKbbFcRwx+6hQXz35jkmOAbqkQf3Rusw/gCp6dQgmAn/rJdXkkNSTk436I+cWWqCohMhoUgT+u8Vt7INAnjrKbB95NcWyJeMy0HqL7UeC/WTSAtUY4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iiz6nPsY; arc=none smtp.client-ip=209.85.222.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iiz6nPsY" Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-7d3da67de87so111649585a.3; Tue, 24 Jun 2025 20:11:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821088; x=1751425888; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=DK7y8fCnybk/HJabsb4vjDUHN1LvZsSF478PQROiY+k=; b=iiz6nPsY416dkI1IlRC7/DO+ri8nSrRnZ++f3cJCTt9m3spQqmH1e2XA5io/KtlDE9 uTia9EFtoWdSjW6iPcbo+QlNJrt0QmK94OvwDC5FKpQ/AdgtJH9Tk3+kMGrDaswz+D05 aMuu5zzTZP2zM55CPJVLAdsm1fSE3/JQ2adyocbuQrKdGR3p5l25mA7oa6iz0vdNMFmX vf24Wnd1SvnxUcHlv8PKwC1oV+5SAv51m3ah1PfcXbj3eltpDKw2aDIn184ccs3x47O0 lpFn4Fp5CvMEnzitUQTYr4OgsAGSiphcHEoX2l+7TNxVQNSM59bHjXOmnzUtVD0o6dTw 6BDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821088; x=1751425888; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=DK7y8fCnybk/HJabsb4vjDUHN1LvZsSF478PQROiY+k=; b=DypFQuz/li/3QlhkXVyCnXafVIPTXGPX0lruW7acisTjnYZ/kfYQzI8cD/YLpjrWIn GC/rWhkwC3eep98bc2WpL4dxOtfefl2s8I0t8sTQhSZra0mRZHK+fNnp9yuofHQRvsNK vq7d+LvyoJjDS/w+8JzuxHKFMhuPzWBOpx2roKMwS7qk/LnjCTqS1YwPagWQO8WRJAOx yCKozP+sInxDZOqYQz8QxsT0nbPr9t2zMIxeDdJdtWnM6BvdR0RaOyZMq/FN1ao3licd CsD7DvM3yP8KwZXEqdx/5m2fi+uc2wa2lBWnG5VEcaiklN8nODidD23GaZZ7I2GRz4MC Pzyw== X-Forwarded-Encrypted: i=1; AJvYcCUp7SHJ+/ROPjC/3D3jNz1d3A7tro/KOGlcXBIBcVLWUzbNiKcu5jZv2WoiePDnbFFw5PNt@vger.kernel.org, AJvYcCVVQkD89XU4Z1OXJMAae824616FNp2S8BF/AyYMRU35a6FyRKZ299/fpTAq1/ZZJgPsLE7BTy0=@vger.kernel.org X-Gm-Message-State: AOJu0YxdRun+afHrXamGOWIEgC/T9zNlQbSrBQVAB71mN1veylpY1sVu djoGuiytMbn+DtgHy3oVucG0OiVUa93qcDFzAtklt2F/OJg+UQswkg85 X-Gm-Gg: ASbGnctkA0biCAtIwVOhs2/fSMwcd9chsukYzSUvq6UMcojHz13a/eziLMDdYxppC7p OwgsSznKEafx+3Hwqbb9r3vFX54djIE+yJysU0ml4X1yyim9JiLH3jf/K7QadAw7SIQ/XZ6t/UC O2+Xf+uL8r0rkOnJS58ZvXQrzlbWz7233ZfFWeSACwt7/imRWLRms9GRghVhst85l+1vwuZSCbI Wfa+qkNJFC+ov7v/xOWnlMwXwHggXD/DM2pwS5Cq5w0SfnyJaCbXcpgyngGOGUN/B0hEO8xAFm0 ayVG169Bp+a/hf7U0Z5FW22vtFpdjbmwEiKP/jiqCNPUE104n9afqRskYtGINy97zLkZlQK8MhO KobfvUZHRRGUkI3WCxEAcrAE7pijEpnKKVpOSrt+knVddpvkHL4Ms X-Google-Smtp-Source: AGHT+IExxiStgtde1OhSWnRVxkmvUnW7k/+Lfu1ZBv7pox0OQJqTmpZsld1V4TSwetgR20fZq5Y+Bw== X-Received: by 2002:a05:620a:1987:b0:7d2:fc7:9572 with SMTP id af79cd13be357-7d42971d8acmr177374985a.57.1750821088463; Tue, 24 Jun 2025 20:11:28 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7d3f99a6724sm562079485a.41.2025.06.24.20.11.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:28 -0700 (PDT) Received: from phl-compute-05.internal (phl-compute-05.phl.internal [10.202.2.45]) by mailfauth.phl.internal (Postfix) with ESMTP id 7CAC4F40066; Tue, 24 Jun 2025 23:11:27 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Tue, 24 Jun 2025 23:11:27 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeihecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeggefg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopedviedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlkhhm mheslhhishhtshdrlhhinhhugidruggvvhdprhgtphhtthhopehpvghtvghriiesihhnfh hrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehkvghrnhgvlhdrohhrghdp rhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepsghoqhhunh drfhgvnhhgsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhonhhgmhgrnhesrhgvughh rghtrdgtohhmpdhrtghpthhtohepuggrvhgvsehsthhgohhlrggsshdrnhgvth X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:26 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 2/8] shazptr: Add refscale test Date: Tue, 24 Jun 2025 20:10:55 -0700 Message-Id: <20250625031101.12555-3-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the refscale test for shazptr to measure the reader side performance. Signed-off-by: Boqun Feng --- kernel/rcu/refscale.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index f11a7c2af778..154520e4ee4c 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -890,6 +891,43 @@ static const struct ref_scale_ops typesafe_seqlock_ops= =3D { .name =3D "typesafe_seqlock" }; =20 +static void ref_shazptr_read_section(const int nloops) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { guard(shazptr)(ref_shazptr_read_section); } + preempt_enable(); + } +} + +static void ref_shazptr_delay_section(const int nloops, const int udl, con= st int ndl) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { + guard(shazptr)(ref_shazptr_delay_section); + un_delay(udl, ndl); + } + preempt_enable(); + } +} + +static bool ref_shazptr_init(void) +{ + return true; +} + +static const struct ref_scale_ops shazptr_ops =3D { + .init =3D ref_shazptr_init, + .readsection =3D ref_shazptr_read_section, + .delaysection =3D ref_shazptr_delay_section, + .name =3D "shazptr" +}; + static void rcu_scale_one_reader(void) { if (readdelay <=3D 0) @@ -1197,6 +1235,7 @@ ref_scale_init(void) &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &sched_clock_ops, &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops, + &shazptr_ops, }; =20 if (!torture_init_begin(scale_type, verbose)) --=20 2.39.5 (Apple Git-154) From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E65351DE4DC; Wed, 25 Jun 2025 03:11:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821093; cv=none; b=IUWGbLh2DqUxDyNMusExQ75SYfp0zwj1C+fnWyeZLZ2HhklNmMvCE5oy8rnCmCOJeyRVeJ2Khfyd8Y95bQghmTHjSsTliu78Uh4EqqYfaVUoF386BjZwd/7g69IhOau12lPo7DESwKANrf/jYAYa8pIURCC9RIlUTowwHcO2tLE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821093; c=relaxed/simple; bh=TZOtCUxplLpVZwlulyF2AdR7IutlsKFrx5aA44BpK6Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NnnO8k0gxFQk+WdqHiCzjTvnrdUjzL/p434y5R856PBLYwsC6g7jq5HqbiqSUjXJqT6iUamwdleI7ouWtTszbD/wD+Le3JDgYHrJYCHBxQDzPlcocGgD5NSdjuPWXzV3BKVQzQ39kPNkj+ipve5oEvwBL9VJadz/BHXuRe3a5RI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QIJS0OfZ; arc=none smtp.client-ip=209.85.222.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QIJS0OfZ" Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-7d3e5df4785so408181685a.0; Tue, 24 Jun 2025 20:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821090; x=1751425890; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=qhQXR4q/1FKoBNWknRJmtYgN/63wc+pb7z14qfrzMOI=; b=QIJS0OfZeKYhj3IlYZFBGiCOB4LBouPxvd102+FS1UqnVFuWop6xkvtBRECGtR+FNz E02GrMKJeFyclFseoZSwoyPbSjcFIx+SUciunYBwdeInLuJsFcaNGMOuGoxQ6VGknS5E f31l8RTkpNKr7uwTBw66nNma6oJzb5rzTfLf/B9uTZsf7w6XkE6wfhqV+lJu/QF/ilXp qhmIggKdCJmHG3T2UqwCC7JjPupu+9uh/iuJcm/YUPAUTAah4IqcWkUTR+e6IT61pbMm jUbK2qpU9+imo6EUHYNKUZY02gsp+w+wP5rMyxnG8FTu6h0fBlfdmO1YIlywWTZCIUo8 aXzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821090; x=1751425890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=qhQXR4q/1FKoBNWknRJmtYgN/63wc+pb7z14qfrzMOI=; b=lgK5mKe5AvjuxSHoz8v4U718/6OjouaYTpMSrBHeXFOUJB4ogA1x5V0cxeb/r8edbR yvr/JraBWWb0waUjHIAAZxBieK7nS/+KnV8aEb94wqFLO5Ik7HdlMd1GpTisJ5nkSJVN CLzDvZQNdG4N7QvbPfS2K69OuT7gv7KKil4rT50GvJeGfGLL4suLtOE/CPyq0gObFYdI iT+JZ8KbUVRsC/QSu0UC57LZf65FBbRAVL1ZgTo9usxWwtOYEaQEj0bhUSEwJ9VGFu1B 7M9xjHRWkyTuU9z0Cahp+mV0acLmlfV/Vk6i+6Fc516k4qIYP/IqmxDJO5aTdeppPgwG kS5A== X-Forwarded-Encrypted: i=1; AJvYcCUQIxhpUTp0EdkFnMSGEHucxRoBfrTugh9IJu1G4RVpGZbAqgpWmRCgX6zs05VuHSK6JH10GOM=@vger.kernel.org, AJvYcCWX7JUfd01GENW12pROpfpxLWcFi0PnBDNvJFh6Km3gZ0OqgJ9EDbx4T2m2c1Ms5kmDrjQf@vger.kernel.org X-Gm-Message-State: AOJu0Yxl3X9pZd8pYtPMQg29nwELI/5wyqrinZkvv2CsKzb/btl4Ghog Z0g6+obYO63ICAvC3CTlwr4fXIevyUHHgFQTR11VlMXQA6UAvLAQOix7 X-Gm-Gg: ASbGnctNyF80pokmm3vGO4Agr/vJ1IclFZtsH2qx5GGmts9N3W3aXVcEsXnuzEju00x P5BMsxGCmwyO43gX3phD1RNez8FhyWVbvP4O7TTr9hEU7/zWWSfKJHue24yUQUJITZS8nwe7qwS hpNZ8O3/fTHg6LdzmPl9zOXi7peYaRbfrnN7cWah01ViAZYr44Qs7Sg5w6aaDoO+rElJ/HfrO/k zvnYtHs7tY9uRKkXSFqAwcs+DzUEfw0N6le5pGfFeJUblvxO+odBLUFOetd4FhHEpeDVjuenC90 7mAuyd/8k5Rfc0KhnaG+++VQ+0oOwAnA7k50nrJNLcyMz8IWcWmDwvmFDrwUj4cjAmTAxaScopQ xSdg4JC7x8pb8Kgh6ZVcO0iYL6wGiO520xjdM8trVKpeTdi6rXyRd X-Google-Smtp-Source: AGHT+IEcTWAIOOFSCQca6pLktm0oe2Iu3mQqxK8tubUlzRS0uHFLS6o5cXHt4DbBg4x47R2Z4s8MFw== X-Received: by 2002:a05:620a:6a14:b0:7d4:29ea:a9a7 with SMTP id af79cd13be357-7d429eaaaf9mr113733385a.49.1750821089756; Tue, 24 Jun 2025 20:11:29 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7d3f999a5a8sm566908585a.18.2025.06.24.20.11.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:29 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id C0A4CF40066; Tue, 24 Jun 2025 23:11:28 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Tue, 24 Jun 2025 23:11:28 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeiiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeggefg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopedviedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlkhhm mheslhhishhtshdrlhhinhhugidruggvvhdprhgtphhtthhopehpvghtvghriiesihhnfh hrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehkvghrnhgvlhdrohhrghdp rhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepsghoqhhunh drfhgvnhhgsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhonhhgmhgrnhesrhgvughh rghtrdgtohhmpdhrtghpthhtohepuggrvhgvsehsthhgohhlrggsshdrnhgvth X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:28 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 3/8] shazptr: Add refscale test for wildcard Date: Tue, 24 Jun 2025 20:10:56 -0700 Message-Id: <20250625031101.12555-4-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the refscale test for shazptr, which starts another shazptr critical section inside an existing one to measure the reader side performance when wildcard logic is triggered. Signed-off-by: Boqun Feng --- kernel/rcu/refscale.c | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index 154520e4ee4c..fdbb4a2c91fe 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -928,6 +928,44 @@ static const struct ref_scale_ops shazptr_ops =3D { .name =3D "shazptr" }; =20 +static void ref_shazptr_wc_read_section(const int nloops) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { + guard(shazptr)(ref_shazptr_read_section); + /* Trigger wildcard logic */ + guard(shazptr)(ref_shazptr_wc_read_section); + } + preempt_enable(); + } +} + +static void ref_shazptr_wc_delay_section(const int nloops, const int udl, = const int ndl) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { + guard(shazptr)(ref_shazptr_delay_section); + /* Trigger wildcard logic */ + guard(shazptr)(ref_shazptr_wc_delay_section); + un_delay(udl, ndl); + } + preempt_enable(); + } +} + +static const struct ref_scale_ops shazptr_wildcard_ops =3D { + .init =3D ref_shazptr_init, + .readsection =3D ref_shazptr_wc_read_section, + .delaysection =3D ref_shazptr_wc_delay_section, + .name =3D "shazptr_wildcard" +}; + static void rcu_scale_one_reader(void) { if (readdelay <=3D 0) @@ -1235,7 +1273,7 @@ ref_scale_init(void) &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &sched_clock_ops, &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops, - &shazptr_ops, + &shazptr_ops, &shazptr_wildcard_ops, }; =20 if (!torture_init_begin(scale_type, verbose)) --=20 2.39.5 (Apple Git-154) From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43FF71DF982; Wed, 25 Jun 2025 03:11:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821094; cv=none; b=Ysu7uZRbkYHH1w/FV+BAZZrsEy/SID0hkfufVzbFn+Sedj4pkGWZRiQTcG24ZbAIlwA9PU3s55755/lFmVQG9EHOjDXhHt8BMdGkpJpUPI2XuuacI4YpDRdqakI9aiLn5KGIB7SXuEnVuGQa5m2siMZs0Atr1NkgFKvCpj3RsxU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821094; c=relaxed/simple; bh=sxyTkmClYKCx2KAlJR1DYraQKkP+bsD2bxeiep/d0Nw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q/qyxvDIfV84gV0PC3P15SyfHW8jv8uYxem0kCpcCdgEjHCwFrNjy0X3SjaF/ZVivUazbHx6+JtjJm86ZSfcpwTp/o0hlc9iZuMWTtpRszpSXOlVAZkpg+6VWPqwiv+riwWDtMK9/8V7KjOWBscZfJHV1w47M8dhOvH9xqbjthk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QEv4+NhE; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QEv4+NhE" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7d21cecc11fso182447585a.3; Tue, 24 Jun 2025 20:11:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821091; x=1751425891; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=w/jmQRtJolvJ8MRtLxIpfpnxQl5ZuerDrjfqxOk6x7w=; b=QEv4+NhEtZDTxMxx1ReFFiMhECxwzAT3DsNchtjJQ4LEblXhUKuIm1rdnZRVVkodz/ UNJm3Ai93ZR+Q1ted9guYUqDgir6FteXzO0jiUZdXkg1/nRaxIzVIhvuTVou8QPyoYt8 2fHJAaNl9tNjA5RY/YSB2N2oELupHzhOKIqS1pN7y+Uf2TztKV8cnqnKj9kq8bRmSf5p KfMJ/ZEg0ORFbJ2a7ZE2Pyz9e3hjOtLbu4Zbi+5vAr/jtikNQQOJf0q11NHtKpxezdqO RUfR+8aK0afTN4ELU2X98QI590KHRqgrxgYBXOE6pHjMn/rMiZnGuV5wToCUJq2gaGFG N3iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821091; x=1751425891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=w/jmQRtJolvJ8MRtLxIpfpnxQl5ZuerDrjfqxOk6x7w=; b=awiCBRZ46Vog2XshN2zCYQ1Bxp4PdnE45kdmFg8VpIfqLtWix/J4StoSKvnI6eZ+CL Hh27vPyJAykEfSpsuiOW1l32gIqPfSbWMf0E6gdcMh2TtbzXCeRDBmP+xDD1Th+b+wNg 0AMlOkMzHmj0VNe3asOcQZpg3BwsF3p0sglZ1FACoo76W7Fg990bAUjUYYrxW7rqRANn VI4yPOPCWwQrIFkwz5Bf4ymSP3ItevNiqETrNbqe/+Y3mKoLTxKgljHmDFMMsdSzjqSt tdHKCt/sBcGLtqzp0n8AxH2KQaECQeGXl+v/GjKrBMHYRwwYfH4wMX9u09PE7V3C2w9c 7UzA== X-Forwarded-Encrypted: i=1; AJvYcCU+W5qfH6+7SYL/SWnjLsMeWKVPNjp9ieYiC1FHjbU6r9kDpXHsFuY1YhpfP4VRv2laF71EBYM=@vger.kernel.org, AJvYcCUMtUKIb2nPYuCnj8CQ3rn8EcUnuEwyzg6+pp30e+h0p3uJlII34yc1J5ZmpaCvfmljRTHC@vger.kernel.org X-Gm-Message-State: AOJu0Yz2TEfq3Jbdfywo4CEdwQbFQ0n9hpN9ya8sIO+KNvZK0w2wU0wU UtoKxr4wz/xGTSmXKS0HPgPFeGrYv92kyAv6CRMVmMCPkq0MIb865Q6y X-Gm-Gg: ASbGncu3GalY0fmQ/B7pTZC2EY/kT0RSdCQo4dvMWwEcoW7fBsuqeyfgvrMIW8biNHY nKztbTDv7qP0u/0BZwViIzPWXAKypMq0gpK9nxnERWC/WHWJpvd8i3LsBnkMbxGEfYdnRAA7HSg ZtRZx5mgkit3DQafkug82NFaV//53BvsxD2Eq+1aHk4Vm3K4QlOmk1hePmEQt/7E7JmYdCLTBPz inyuyyuLFKBvWOA2avEvw2hOqQw+y4DvY1hcPc9fGJZ4Yn5ccqoyQzIpclBxvy1RGQS0DosUZ8O Jcj8xisAYuPW6YspBn41X3QObVjWtq/c8N8akWF5tnG/51oZDMm8D3VzvW8E6cJ7MdK8KgXoWYI oPDLsoBEZDIFiZ2J7aRas18uxR+D+aPiOTAOHRMQXFn0JTXnHNGUV X-Google-Smtp-Source: AGHT+IFVRwKccPAflt2V1XXD1Q8zsrzo7w5VtHeTyKZceopLjwAcJzQGavaQnodNpvUONHM1DDrC7Q== X-Received: by 2002:a05:620a:2916:b0:7cf:159:9aea with SMTP id af79cd13be357-7d4296ca04bmr190876685a.2.1750821091010; Tue, 24 Jun 2025 20:11:31 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6fd167ff184sm52004366d6.43.2025.06.24.20.11.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:30 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id 0FAF7F40066; Tue, 24 Jun 2025 23:11:30 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Tue, 24 Jun 2025 23:11:30 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeiiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeggefg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopedviedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlkhhm mheslhhishhtshdrlhhinhhugidruggvvhdprhgtphhtthhopehpvghtvghriiesihhnfh hrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehkvghrnhgvlhdrohhrghdp rhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepsghoqhhunh drfhgvnhhgsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhonhhgmhgrnhesrhgvughh rghtrdgtohhmpdhrtghpthhtohepuggrvhgvsehsthhgohhlrggsshdrnhgvth X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:29 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 4/8] shazptr: Avoid synchronize_shaptr() busy waiting Date: Tue, 24 Jun 2025 20:10:57 -0700 Message-Id: <20250625031101.12555-5-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For a general purpose hazard pointers implemenation, always busy waiting is not an option. It may benefit some special workload, but overall it hurts the system performance when more and more users begin to call synchronize_shazptr(). Therefore avoid busy waiting for hazard pointer slots changes by using a scan kthread, and each synchronize_shazptr() queues themselves if a quick scan shows they are blocked by some slots. A simple optimization is done inside the scan: each synchronize_shazptr() tracks which CPUs (or CPU groups if nr_cpu_ids > BITS_PER_LONG) are blocking it and the scan function updates this information for each synchronize_shazptr() (via shazptr_wait) individually. In this way, synchronize_shazptr() doesn't need to wait until a scan result showing all slots are not blocking (as long as the scan has observed each slot has changed into non-block state once). Signed-off-by: Boqun Feng --- kernel/locking/shazptr.c | 277 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 276 insertions(+), 1 deletion(-) diff --git a/kernel/locking/shazptr.c b/kernel/locking/shazptr.c index 991fd1a05cfd..a8559cb559f8 100644 --- a/kernel/locking/shazptr.c +++ b/kernel/locking/shazptr.c @@ -7,18 +7,243 @@ * Author: Boqun Feng */ =20 +#define pr_fmt(fmt) "shazptr: " fmt + #include #include +#include +#include +#include +#include #include +#include +#include =20 DEFINE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); EXPORT_PER_CPU_SYMBOL_GPL(shazptr_slots); =20 -void synchronize_shazptr(void *ptr) +/* Wait structure for synchronize_shazptr(). */ +struct shazptr_wait { + struct list_head list; + /* Which groups of CPUs are blocking. */ + unsigned long blocking_grp_mask; + void *ptr; + struct completion done; +}; + +/* Snapshot for hazptr slot. */ +struct shazptr_snapshot { + unsigned long ptr; + unsigned long grp_mask; +}; + +static inline int +shazptr_snapshot_cmp(const void *a, const void *b) +{ + const struct shazptr_snapshot *snap_a =3D (struct shazptr_snapshot *)a; + const struct shazptr_snapshot *snap_b =3D (struct shazptr_snapshot *)b; + + if (snap_a->ptr > snap_b->ptr) + return 1; + else if (snap_a->ptr < snap_b->ptr) + return -1; + else + return 0; +} + +/* *In-place* merge @n together based on ->ptr and accumulate the >grp_mas= k. */ +static int shazptr_snapshot_merge(struct shazptr_snapshot *snaps, int n) +{ + int new, i; + + /* Sort first. */ + sort(snaps, n, sizeof(*snaps), shazptr_snapshot_cmp, NULL); + + new =3D 0; + + /* Skip NULLs. */ + for (i =3D 0; i < n; i++) { + if (snaps[i].ptr) + break; + } + + while (i < n) { + /* Start with a new address. */ + snaps[new] =3D snaps[i]; + + for (; i < n; i++) { + /* Merge if the next one has the same address. */ + if (snaps[new].ptr =3D=3D snaps[i].ptr) { + snaps[new].grp_mask |=3D snaps[i].grp_mask; + } else + break; + } + + /* + * Either the end has been reached or need to start with a new + * record. + */ + new++; + } + + return new; +} + +/* + * Calculate which group is still blocking @ptr, this assumes the @snaps is + * already merged. + */ +static unsigned long +shazptr_snapshot_blocking_grp_mask(struct shazptr_snapshot *snaps, + int n, void *ptr) +{ + unsigned long mask =3D 0; + + if (!n) + return mask; + else if (snaps[n-1].ptr =3D=3D (unsigned long)SHAZPTR_WILDCARD) { + /* + * Take SHAZPTR_WILDCARD slots, which is ULONG_MAX, into + * consideration if any. + */ + mask =3D snaps[n-1].grp_mask; + } + + /* TODO: binary search if n is big. */ + for (int i =3D 0; i < n; i++) { + if (snaps[i].ptr =3D=3D (unsigned long)ptr) { + mask |=3D snaps[i].grp_mask; + break; + } + } + + return mask; +} + +/* Scan structure for synchronize_shazptr(). */ +struct shazptr_scan { + /* The scan kthread */ + struct task_struct *thread; + + /* Wait queue for the scan kthread */ + struct swait_queue_head wq; + + /* Whether the scan kthread has been scheduled to scan */ + bool scheduled; + + /* The lock protecting ->queued and ->scheduled */ + struct mutex lock; + + /* List of queued synchronize_shazptr() request. */ + struct list_head queued; + + int cpu_grp_size; + + /* List of scanning synchronize_shazptr() request. */ + struct list_head scanning; + + /* Buffer used for hazptr slot scan, nr_cpu_ids slots*/ + struct shazptr_snapshot* snaps; +}; + +static struct shazptr_scan shazptr_scan; + +static void shazptr_do_scan(struct shazptr_scan *scan) +{ + int cpu; + int snaps_len; + struct shazptr_wait *curr, *next; + + scoped_guard(mutex, &scan->lock) { + /* Move from ->queued to ->scanning. */ + list_splice_tail_init(&scan->queued, &scan->scanning); + } + + memset(scan->snaps, nr_cpu_ids, sizeof(struct shazptr_snapshot)); + + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + void *val; + + /* Pair with smp_store_release() in shazptr_clear(). */ + val =3D smp_load_acquire(slot); + + scan->snaps[cpu].ptr =3D (unsigned long)val; + scan->snaps[cpu].grp_mask =3D 1UL << (cpu / scan->cpu_grp_size); + } + + snaps_len =3D shazptr_snapshot_merge(scan->snaps, nr_cpu_ids); + + /* Only one thread can access ->scanning, so can be lockless. */ + list_for_each_entry_safe(curr, next, &scan->scanning, list) { + /* Accumulate the shazptr slot scan result. */ + curr->blocking_grp_mask &=3D + shazptr_snapshot_blocking_grp_mask(scan->snaps, + snaps_len, + curr->ptr); + + if (curr->blocking_grp_mask =3D=3D 0) { + /* All shots are observed as not blocking once. */ + list_del(&curr->list); + complete(&curr->done); + } + } +} + +static int __noreturn shazptr_scan_kthread(void *unused) +{ + for (;;) { + swait_event_idle_exclusive(shazptr_scan.wq, + READ_ONCE(shazptr_scan.scheduled)); + + shazptr_do_scan(&shazptr_scan); + + scoped_guard(mutex, &shazptr_scan.lock) { + if (list_empty(&shazptr_scan.queued) && + list_empty(&shazptr_scan.scanning)) + shazptr_scan.scheduled =3D false; + } + } +} + +static int __init shazptr_scan_init(void) +{ + struct shazptr_scan *scan =3D &shazptr_scan; + struct task_struct *t; + + init_swait_queue_head(&scan->wq); + mutex_init(&scan->lock); + INIT_LIST_HEAD(&scan->queued); + INIT_LIST_HEAD(&scan->scanning); + scan->scheduled =3D false; + + /* Group CPUs into at most BITS_PER_LONG groups. */ + scan->cpu_grp_size =3D DIV_ROUND_UP(nr_cpu_ids, BITS_PER_LONG); + + scan->snaps =3D kcalloc(nr_cpu_ids, sizeof(scan->snaps[0]), GFP_KERNEL); + + if (scan->snaps) { + t =3D kthread_run(shazptr_scan_kthread, NULL, "shazptr_scan"); + if (!IS_ERR(t)) { + smp_store_release(&scan->thread, t); + /* Kthread creation succeeds */ + return 0; + } else { + kfree(scan->snaps); + } + } + + pr_info("Failed to create the scan thread, only busy waits\n"); + return 0; +} +core_initcall(shazptr_scan_init); + +static void synchronize_shazptr_busywait(void *ptr) { int cpu; =20 smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ + for_each_possible_cpu(cpu) { void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); /* Pair with smp_store_release() in shazptr_clear(). */ @@ -26,4 +251,54 @@ void synchronize_shazptr(void *ptr) VAL !=3D ptr && VAL !=3D SHAZPTR_WILDCARD); } } + +static void synchronize_shazptr_normal(void *ptr) +{ + int cpu; + unsigned long blocking_grp_mask =3D 0; + + smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ + + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + void *val; + + /* Pair with smp_store_release() in shazptr_clear(). */ + val =3D smp_load_acquire(slot); + + if (val =3D=3D ptr || val =3D=3D SHAZPTR_WILDCARD) + blocking_grp_mask |=3D 1UL << (cpu / shazptr_scan.cpu_grp_size); + } + + /* Found blocking slots, prepare to wait. */ + if (blocking_grp_mask) { + struct shazptr_scan *scan =3D &shazptr_scan; + struct shazptr_wait wait =3D { + .blocking_grp_mask =3D blocking_grp_mask, + }; + + INIT_LIST_HEAD(&wait.list); + init_completion(&wait.done); + + scoped_guard(mutex, &scan->lock) { + list_add_tail(&wait.list, &scan->queued); + + if (!scan->scheduled) { + WRITE_ONCE(scan->scheduled, true); + swake_up_one(&shazptr_scan.wq); + } + } + + wait_for_completion(&wait.done); + } +} + +void synchronize_shazptr(void *ptr) +{ + /* Busy waiting if the scan kthread has not been created. */ + if (!smp_load_acquire(&shazptr_scan.thread)) + synchronize_shazptr_busywait(ptr); + else + synchronize_shazptr_normal(ptr); +} EXPORT_SYMBOL_GPL(synchronize_shazptr); --=20 2.39.5 (Apple Git-154) From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70BE71E32BE; Wed, 25 Jun 2025 03:11:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821095; cv=none; b=KvXPxAs7Sj1bjkBBpMhGL4XrShTRKohxa/Vn6Mr+bl6FHbznhNMzjEws7OxR5KuS1K8LBuM1TFDO+S2VdAuuWlkMax+RgGmZG4aiT1UPUfkCOBFhcXXR2INhf4MHiJVoJlFjd4SI6JJGyT69CE1JUPNBcNSdlFmRvrQuChBqo+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821095; c=relaxed/simple; bh=JPcvKh/jQAHEETOG+k6olIvrMgI+nTXU1PIG0qFgfik=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VCo5wcqXFiK5uyuVhrajnabFUq/ApHSZ4i6r+Tc2KpVOdHgsF7bx3medRV0JV3gOqRygY3F4rAbrILDas/urf067Xh8a2KnF2VEEmt2JCAa1M7qIlXnnvD78RVboB2cA5whdn+lt5NhDpwjtbCKx50kDEtbdT1HvR7VhA32TPeA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HpSRA0v1; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HpSRA0v1" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-4a58d95ea53so5962521cf.0; Tue, 24 Jun 2025 20:11:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821092; x=1751425892; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=CPJydrkhnERanEDJeh8iKFAtkTK1x51r4pmOIL1PfXo=; b=HpSRA0v19F9mUJnOXdepSt3nr9KiSJPHqh+KMuMeiu6vyC/ou3HAjGW6ag6nw+d8VB BwgtjTwyibi9Dxe4ircFPB3Xdt+GjmAmYHAJeV18DESVSeCJc9orxxpWMbtHOAmrPiHV 4EPFvVtETjDdMiW/xsxMO6UqTxY9E3bqeL484w3QN5IEXG3wG3QirWuwiwN7LehRsl3i l4utsRcduPSQT7mCuqE3cX89hS0DnvHTl5xb9qVF/HmpJeBdr1KHbhwQhIN5UHP7TL+3 O3R3kbrIbL4DS5HbQEwNdxa2hi8TTUKdbW3e0ctAsRatbeReNqRKrPycwJwfXx4F3xS8 WNVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821092; x=1751425892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=CPJydrkhnERanEDJeh8iKFAtkTK1x51r4pmOIL1PfXo=; b=u2PZzqla7FtZ8DbKju3KikB7YK0jc0pPLix+lXsARS+QmNvglEP1lf1lbHkU84Jxwu lJDUAIpzPumUmDEidGlaEdGyDn13D69OqlF4pm1AvOyz5V+hs4Ji6+tMSuhBVo0UApRW UCj9oYeVL2TkDmONwrLuk3R5MKouiclFJROPy8EGoSUmPr2gdQznc5DnTGgKW5FCXiAz oyrCRQ9BfUQYMJZBHimty19krJqjKkz0JmAMCqtNRqNFNRzkWhQXOdIkIAgEFe4/97ZD T2FTLUXrFhoHmxm1lCIolXkmqRpFKI1mo9NSPD80cjBGE24lUsCME7xX66BQ89vH7Pen XfMw== X-Forwarded-Encrypted: i=1; AJvYcCUsDCa0Dp5d1zHItyVOW6duPOYB3NGoofukZTPPJBS5ZPkHLkxaXGJYCaS/kaWf4NazjI3E@vger.kernel.org, AJvYcCXU6BKp9DiKvmmmMQ5/ZBpn0WALTSSjxhVVGdojd/D0ZkNBxGP6sUZm8zMe5NxcijsKNuG4ZpI=@vger.kernel.org X-Gm-Message-State: AOJu0YwQCFmmiNk62cozgS6xd/SxO5GXhmlvfCEUt4wTE2roq3Jf8Nxo EmEDZmjfxckpEKt5ky+8ae33c7Fm7x0Bfc6Vzl0wswSdJykGmMjQUzC6 X-Gm-Gg: ASbGncuOOhC1xuzcENGnQl+30ehszkZ3bJzy8CmGG+J2TC19IiGK0ogm9f/FmRPR/KZ P7RUmL59gq5OGJ4PYtPn3L24kyMwsVl1ca+PdhbsL3HS7xNU1KCYO/yh68f08xkxKL53fV/fpbe grCm/oDOzgRKIpoFKDYpfaHKb3CuX6DCK+3XKrcdG6eqLoRKKgFIq8uTT8aeNppkOuSEYw5ZCAl VxtNaOv74EblBADdR/uf1Ck4iJTg14HwrK1jlwWmfQLCyp59taYPEabh7Kl+8O4vCnfjZ27G4tp UD20nQE4rDkRIFvGDAaNZp/ic71xF1Uz4dhk3y8wbe5lSYOG9yVaB0NaGJd0NvIRFbLGQmJxcfG 3jRG0vAons0y+yjILHYJozZix9OMVutOm1gtk9wISmDls/MDzDPh0 X-Google-Smtp-Source: AGHT+IGW0TiZzG9hPxpBHcORi1BL38UTs0PtOJHlHm1exg6B9dlOGYVh+aZvZ1C6ZdIypaz5HFXCZA== X-Received: by 2002:a05:622a:198c:b0:4a7:896:f2d3 with SMTP id d75a77b69052e-4a7c112d0a4mr23737041cf.0.1750821092305; Tue, 24 Jun 2025 20:11:32 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4a79de8d57bsm30445411cf.72.2025.06.24.20.11.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:31 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id 480CEF40066; Tue, 24 Jun 2025 23:11:31 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Tue, 24 Jun 2025 23:11:31 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeiiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeggefg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopedviedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlkhhm mheslhhishhtshdrlhhinhhugidruggvvhdprhgtphhtthhopehpvghtvghriiesihhnfh hrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehkvghrnhgvlhdrohhrghdp rhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepsghoqhhunh drfhgvnhhgsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhonhhgmhgrnhesrhgvughh rghtrdgtohhmpdhrtghpthhtohepuggrvhgvsehsthhgohhlrggsshdrnhgvth X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:30 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 5/8] shazptr: Allow skip self scan in synchronize_shaptr() Date: Tue, 24 Jun 2025 20:10:58 -0700 Message-Id: <20250625031101.12555-6-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a module parameter for shazptr to allow skip the self scan in synchronize_shaptr(). This can force every synchronize_shaptr() to use shazptr scan kthread, and help testing the shazptr scan kthread. Another reason users may want to set this paramter is to reduce the self scan CPU cost in synchronize_shaptr(). Signed-off-by: Boqun Feng --- kernel/locking/shazptr.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/kernel/locking/shazptr.c b/kernel/locking/shazptr.c index a8559cb559f8..b3f7e8390eb2 100644 --- a/kernel/locking/shazptr.c +++ b/kernel/locking/shazptr.c @@ -14,11 +14,17 @@ #include #include #include +#include #include #include #include #include =20 +#ifdef MODULE_PARAM_PREFIX +#undef MODULE_PARAM_PREFIX +#endif +#define MODULE_PARAM_PREFIX "shazptr." + DEFINE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); EXPORT_PER_CPU_SYMBOL_GPL(shazptr_slots); =20 @@ -252,6 +258,10 @@ static void synchronize_shazptr_busywait(void *ptr) } } =20 +/* Disabled by default. */ +static int skip_synchronize_self_scan; +module_param(skip_synchronize_self_scan, int, 0644); + static void synchronize_shazptr_normal(void *ptr) { int cpu; @@ -259,15 +269,19 @@ static void synchronize_shazptr_normal(void *ptr) =20 smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ =20 - for_each_possible_cpu(cpu) { - void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); - void *val; + if (unlikely(skip_synchronize_self_scan)) { + blocking_grp_mask =3D ~0UL; + } else { + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + void *val; =20 - /* Pair with smp_store_release() in shazptr_clear(). */ - val =3D smp_load_acquire(slot); + /* Pair with smp_store_release() in shazptr_clear(). */ + val =3D smp_load_acquire(slot); =20 - if (val =3D=3D ptr || val =3D=3D SHAZPTR_WILDCARD) - blocking_grp_mask |=3D 1UL << (cpu / shazptr_scan.cpu_grp_size); + if (val =3D=3D ptr || val =3D=3D SHAZPTR_WILDCARD) + blocking_grp_mask |=3D 1UL << (cpu / shazptr_scan.cpu_grp_size); + } } =20 /* Found blocking slots, prepare to wait. */ --=20 2.39.5 (Apple Git-154) From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDE7F1E990E; Wed, 25 Jun 2025 03:11:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821096; cv=none; b=T8B9K0MowSbPDG3zrJ9osM+RQvcFdrs5ykwkmAfX2IUHrZ95mfKER5GeA9kS371SwGOohp4NgYEWEHfnYi1qtD6ymns99Bnhva5DR9ePqgI5C62r6KwXEVSkhAPO6FHORCY168VJYLfjHinhm4ul6YbuJSvhDt0ZjzA20rctFYc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821096; c=relaxed/simple; bh=asr5uTUioL/0h5I0aStm17Mh+pOtbR0ofOQLjCqfd+E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t6CjiCKRPkizY4sEuoweK/zprZWPVpBeRr0KiL9iTWijCxj5BCA2yjUPtlee7HrWVzyhJoamzTeh5Qsu7j6x2+bIjffM2R+bD5diXOrESAJWzVUoSGrLgfKuHKWzCeDqJyls3BOXwMX1ODTLeDGkJ32mFBL0K0ul/JFe8m1NJhc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gSVfGMb8; arc=none smtp.client-ip=209.85.160.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gSVfGMb8" Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-4a44b0ed780so79452271cf.3; Tue, 24 Jun 2025 20:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821094; x=1751425894; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=678FLBRyV/EGtYHAvszDngJJOFd0kRmHfL4yh1GGT0U=; b=gSVfGMb8Dz+etAOKYW6FEyv6s5YVXoFJVGWONpWVrBlw2lUghmo+98rs+pfRcw3Odz uA0968tmTFOO+bunC/7kSlWPXgLEID310gjGQnf9VG4L0Fq+CNGQT9CEkKA/UZqtf8mz FAo6bLjWVvHXrUliqPn377eKuc/sEev5HJTCD86HnAcsblrVEr7OAbtBHF//i7iDmLYh g5uKjJkjfa+CWtFr+h3acE/46+XaBKSf1QLymt0HzANkRb7Pc9vLcdiJiG3+IqnGjr4C F5EDWWI8fRIjbL1bObL01nIW3vXj3CkcmfqJHM4MX7gP8vlmbLwbbCLSMFxWEIpUoFZS xNrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821094; x=1751425894; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=678FLBRyV/EGtYHAvszDngJJOFd0kRmHfL4yh1GGT0U=; b=UdwImOPhEFg95tOcMPNMXAoPen//tSk8mPzdYvIRa/hU5M0RD5qyuyCPjN+dY5T+F1 QftDRG52rIM4dVds90lgFzJZIz/fNWLFLv9PDMtlifp4sXZ/5/F31+IhhHRrSAcdJyQ0 m/vB3rq3ADxhUim9BC5lqAlRu2/QU1qVXP38QgarFhROFekzE/H3zpQqbilGYDhwKr7n Gjlp6Dzbrnwr40oj5voSh1hDlCk7tdjM/WxvGs4+/wXDRDPxMffQAxBYvtXCcTV0dNqn K3IwaEz2Vputt6+T+5PLtiAzw2lkAXTKhRwAHJWsRTCANIXPcjxQgqn7LCwh1yRW775d A7Ng== X-Forwarded-Encrypted: i=1; AJvYcCVmZOziImLPMkSBxUL7x7fW6xVZgQrhXRwswLAcHrpS7+T0yTb0Q9HQi/ouVlNDJyoGUcqK@vger.kernel.org, AJvYcCXfMONElzGggzJCeQL5RFeQa/Y2Vb814wFgb/t52yn8Y5YdaKYhsu1bV6N4F80ZSxOK5IFkjIU=@vger.kernel.org X-Gm-Message-State: AOJu0YzYGiTLzio5lnMMAopzDJ4CsJGKQhoZO3H4yO+DEU9hFbrSyDvp kovpOaoUvO5go9+K6wXpem9o0aBJmfAWFyi11MHc20/lL5lTlCQ286oY X-Gm-Gg: ASbGncuyjy2FF57b5qmTbfMgsexLo2FBfv7oGAjk1pEaPKxUXxlzx5fPZwthL/ngHEK qavdW0IN9VzsV1M9oT9ddfk7UHs5HolHOp9njDV1WmuHrvQ98Hxg/fBYx51+/QcqcrmR5iojklH SyT7aUPGsF7+T2P4Y+lcODjd++o4X562vu4sDoUfoegZD1jpZCj1h4E/DtyJmoamNjYSeAYJldn n3S/HaftQo+4rZyk+k2M5cmofB1/TVC3Gcg4kfgipoCtoKtXFMWSBGVcgVKHo4AtSsiy68fMDnd zIezoh8Tk4A7ykuEeDqb6QSDl0BUu/tlW75TctzkULaf+HtGs7szGaPFFHllxr7CDsiQaxsWbgs kNcNTBMZNwbnO0mYlk3V0eIW0IrNz/rWLrHWgc1wmItf7iu5DAXnZ X-Google-Smtp-Source: AGHT+IHLBlxfOfHRIysWYSzviK3Jt4IAeQ/77Q8AVFjgBBxKd1sBVPA9U6rgMUaVbnw8S3yFqLBbuQ== X-Received: by 2002:a05:622a:40b:b0:4a4:31a0:68d1 with SMTP id d75a77b69052e-4a7c06a01cfmr29966041cf.21.1750821093620; Tue, 24 Jun 2025 20:11:33 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4a779e910c8sm55291021cf.71.2025.06.24.20.11.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:33 -0700 (PDT) Received: from phl-compute-05.internal (phl-compute-05.phl.internal [10.202.2.45]) by mailfauth.phl.internal (Postfix) with ESMTP id 8C62AF40066; Tue, 24 Jun 2025 23:11:32 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Tue, 24 Jun 2025 23:11:32 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeihecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeggefg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopedviedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlkhhm mheslhhishhtshdrlhhinhhugidruggvvhdprhgtphhtthhopehpvghtvghriiesihhnfh hrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehkvghrnhgvlhdrohhrghdp rhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepsghoqhhunh drfhgvnhhgsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhonhhgmhgrnhesrhgvughh rghtrdgtohhmpdhrtghpthhtohepuggrvhgvsehsthhgohhlrggsshdrnhgvth X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:32 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 6/8] rcuscale: Allow rcu_scale_ops::get_gp_seq to be NULL Date: Tue, 24 Jun 2025 20:10:59 -0700 Message-Id: <20250625031101.12555-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For synchronization mechanisms similar to RCU, there could be no "grace period" concept (e.g. hazard pointers), therefore allow rcu_scale_ops::get_gp_seq to be a NULL pointer for these cases, and simply treat started and finished grace period as 0. Signed-off-by: Boqun Feng --- kernel/rcu/rcuscale.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index b521d0455992..45413a73d61e 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -568,8 +568,10 @@ rcu_scale_writer(void *arg) if (gp_exp) { b_rcu_gp_test_started =3D cur_ops->exp_completed() / 2; - } else { + } else if (cur_ops->get_gp_seq) { b_rcu_gp_test_started =3D cur_ops->get_gp_seq(); + } else { + b_rcu_gp_test_started =3D 0; } } =20 @@ -625,9 +627,11 @@ rcu_scale_writer(void *arg) if (gp_exp) { b_rcu_gp_test_finished =3D cur_ops->exp_completed() / 2; - } else { + } else if (cur_ops->get_gp_seq) { b_rcu_gp_test_finished =3D cur_ops->get_gp_seq(); + } else { + b_rcu_gp_test_finished =3D 0; } if (shutdown) { smp_mb(); /* Assign before wake. */ --=20 2.39.5 (Apple Git-154) From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C0FF1EF39E; Wed, 25 Jun 2025 03:11:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821097; cv=none; b=ef6EutfwFy3Oo7MOcyXuAx2BfMGg9BhyjWBm1bufrtr/gdki6i+OcryST0Mh2PllDfEHy4fYd77ki+EBSDc02KdeDF1qc/7aXxH8OaAT+9fhxR1N7a9NJWyW0q85LfwgHivTujEtW4lYudmuqgw12DvcmbMj5VTjIs4cXYtEadw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821097; c=relaxed/simple; bh=2Zl3n64ENVCvkp3q5SHTf+XFtSOXGOOsCzI/DQkaM/Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MP4oPwlsza8iF9u07EofbHbXP5Kri59NxTFcxYSFCTwCvHVi+erxHwl7Wn3KEHjQ03y76/tv1K3APd9NfMigHXfDObDjxmEqvIF4qpTwZdWCdktwDIFoM6kTUttLHnAUjbTG89qrTJ6Aw7nbk2TXQpgeF7ld4FcC80xArivnqk8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=h7tyJscd; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="h7tyJscd" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7d098f7bd77so57718985a.0; Tue, 24 Jun 2025 20:11:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821095; x=1751425895; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=/1uL4SnLWRugsWnQE9sdQcVDzQKPklHv+4Xueonr/8Q=; b=h7tyJscd3NJKMhZAy75lakEvZZY78q/8021OvjMvoufBv4W44TrPrNDSNCnkYFJyv5 DQ/G4KJBxXPIJyUqRQqvpqRzyAE1G7V7iukGj2cZZODGBW2y4JpRyT+2KI0CNsKPnJVn krfaoBWdbOszlVpFTIBirC2Rg1/8QfZexruWhyXs/cGwDqWDx/fvTcHAPiviYL4F5XS/ uwE2ZAYvyaGUkWYjuyl3RKz3R4Vlsix47TTUsEgiXFsqdoCNapy80b0GwglXbFO75Ngd nolWmQwLcNaj+ArwwLKHucluo+VJAoD1o/m770rMZX3w8+V0xeSoeC2aGiutyPQQ4n6H TMJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821095; x=1751425895; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=/1uL4SnLWRugsWnQE9sdQcVDzQKPklHv+4Xueonr/8Q=; b=wRmz425VP87x8njxmUq7XTdPH5UyEQQ+a8+D8pa11qtorgYLooUGQ7fJRhHugi/ra2 M3qjpp1yrMRFE5rHAhwEPu0fQ/T5Y19w2eBJn/11NwNjvO152sg4HYkefyvc+izeDxim mJGZGt9U87ikfrV9uB0zKJ/B1rCygQjglhwaa5qFlGRHJnvUyi2VZfa7ucQp0oV4dj+S jzWQ+5WyWt45thQwFVR2ejRQNhnW/vrUPvzd57ewgO23HpDUuuEuQm5mJTdnX9OHdoWq o9XxGWqWruQyauELIwXxv37nWdCJFsSOeiWj++kQvKcXMQEXQxm32fng+krTb7bXN9qG ly5w== X-Forwarded-Encrypted: i=1; AJvYcCXUzO6BLHtc7o2eXyKxsjZy5NmlnYv6FERxsef6KgfsRMkuTZOuUjGU5LT5AW53NUgtWY3h@vger.kernel.org, AJvYcCXnlYe2kVj0US+UfcV4qmIKx0aYJ8LeBBa4AjHVPnGX72d0ObTIAHHagCiobml+EmcOZw4Q5hQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yy4w0AwspkhWdQR8arV+rJN+TRj5MZOCc0XpQw4pa7Ekh+8/aNT sj7VD3LYMsEKd6z26o+Y08QomlqJRYuLwU2T0WFRaqn+uL3egAXKqaB+ X-Gm-Gg: ASbGncvtjTg0q8ef3CSAwuJAB+RNW+WHTUpMEIHBVATT2jNflBIWWYLwmWny7NNeNKY jn3/NE/WlZb6k5tGRg+liPHogqfs14jzNxIuimgmZz1IrxleX089eQeWSFxkXU5FDt1X40Ppjrz 7aOhnC2Qra+SED2YQk83DeoVbtcz3DNr9C4NLopzgKik4WztgXvmmuWB9V2MqtIiCT9Iq0KOptf /FvLACLI2fU+qNmx7aMXInraHL2UZZwkPS9TSWa29RFtp+65VoHHgMGYUZB0G4oR5ML/lllXE8E 8BewxU2CpflzoC/iCChfCMsu4NzOq24BDm7veatr9o7M2W5ZvtvQJIUMCE0T6W9irNKArAjWT+3 lBPgLqpsLF90oQHd3dbdaH5UOpTjoJTRbEPQttu4XAx5z4+SDGgh0 X-Google-Smtp-Source: AGHT+IEd9r5s1fnXpoG01dbVdt03H4HfKlf+nXWdSHDs3RHqDgvG/mKgmy92mv9tB0zyvfPMzHQ/Ug== X-Received: by 2002:a05:620a:1a02:b0:7ce:fc0b:d39d with SMTP id af79cd13be357-7d41ebf3aa2mr831583985a.6.1750821094938; Tue, 24 Jun 2025 20:11:34 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7d3f99faaacsm564338085a.87.2025.06.24.20.11.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:34 -0700 (PDT) Received: from phl-compute-12.internal (phl-compute-12.phl.internal [10.202.2.52]) by mailfauth.phl.internal (Postfix) with ESMTP id C51FAF40066; Tue, 24 Jun 2025 23:11:33 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Tue, 24 Jun 2025 23:11:33 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeihecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgeeljeeitdehvdehgefgjeevfeejjeekgfevffeiueejhfeuiefggeeuheeggefg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopedviedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtoheprhgtuhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehlkhhm mheslhhishhtshdrlhhinhhugidruggvvhdprhgtphhtthhopehpvghtvghriiesihhnfh hrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehkvghrnhgvlhdrohhrghdp rhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepsghoqhhunh drfhgvnhhgsehgmhgrihhlrdgtohhmpdhrtghpthhtoheplhhonhhgmhgrnhesrhgvughh rghtrdgtohhmpdhrtghpthhtohepuggrvhgvsehsthhgohhlrggsshdrnhgvth X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:33 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 7/8] rcuscale: Add tests for simple hazard pointers Date: Tue, 24 Jun 2025 20:11:00 -0700 Message-Id: <20250625031101.12555-8-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add two rcu_scale_ops to include tests from simple hazard pointers (shazptr). One is with evenly distributed readers, and the other is with all WILDCARD readers. This could show the best and worst case scenarios for the synchronization time of simple hazard pointers. Signed-off-by: Boqun Feng --- kernel/rcu/rcuscale.c | 52 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 45413a73d61e..357431bf802b 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -429,6 +430,54 @@ static struct rcu_scale_ops tasks_tracing_ops =3D { =20 #endif // #else // #ifdef CONFIG_TASKS_TRACE_RCU =20 +static int shazptr_scale_read_lock(void) +{ + long cpu =3D raw_smp_processor_id(); + + /* Use cpu + 1 as the key */ + guard(shazptr)((void *)(cpu + 1)); + + return 0; +} + +static int shazptr_scale_wc_read_lock(void) +{ + guard(shazptr)(SHAZPTR_WILDCARD); + + return 0; +} + + +static void shazptr_scale_read_unlock(int idx) +{ + /* Do nothing, it's OK since readers are doing back-to-back lock+unlock*/ +} + +static void shazptr_scale_sync(void) +{ + long cpu =3D raw_smp_processor_id(); + + synchronize_shazptr((void *)(cpu + 1)); +} + +static struct rcu_scale_ops shazptr_ops =3D { + .ptype =3D RCU_FLAVOR, + .readlock =3D shazptr_scale_read_lock, + .readunlock =3D shazptr_scale_read_unlock, + .sync =3D shazptr_scale_sync, + .exp_sync =3D shazptr_scale_sync, + .name =3D "shazptr" +}; + +static struct rcu_scale_ops shazptr_wc_ops =3D { + .ptype =3D RCU_FLAVOR, + .readlock =3D shazptr_scale_wc_read_lock, + .readunlock =3D shazptr_scale_read_unlock, + .sync =3D shazptr_scale_sync, + .exp_sync =3D shazptr_scale_sync, + .name =3D "shazptr_wildcard" +}; + static unsigned long rcuscale_seq_diff(unsigned long new, unsigned long ol= d) { if (!cur_ops->gp_diff) @@ -1090,7 +1139,8 @@ rcu_scale_init(void) long i; long j; static struct rcu_scale_ops *scale_ops[] =3D { - &rcu_ops, &srcu_ops, &srcud_ops, TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_= OPS + &rcu_ops, &srcu_ops, &srcud_ops, &shazptr_ops, &shazptr_wc_ops, + TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS }; =20 if (!torture_init_begin(scale_type, verbose)) --=20 2.39.5 (Apple Git-154) From nobody Wed Oct 8 19:23:12 2025 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FC611F4611; Wed, 25 Jun 2025 03:11:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821099; cv=none; b=hSTfpYQWaJqneDR9kaIeEaIJG7BaeVNfLUT3uFRmh6SxYkmXIJqT0wo32NYkZ5+lls/KefcuSpYhljOH6pI1h2+CEdM71Tz4wTHd+UA5NTgXQ8DTgdMyYSTWbJxI6ovNzUMVn4El4/EirdGcdYW12UuXchNNvFw8mEJ1ll9alNk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750821099; c=relaxed/simple; bh=A73pes+JuDp33aVXpLNlEeOGjBrNNA3VpTS+s9bm4yg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KoSclaRNcZtO7p60HM5+dcakTu6mvBJA6oAHOO7E8Ju4Cq6NgtAk+0SLe9UTl/h3MZcsbeBRPhOqpaT5ofv51+oBdbIpQU9PHrhrFmdLODfZXLmDnpsalb6BE9YLr3lq76G/QhW5G0jaBJev/cEKb6BlGfNhQpgTckSdYrnjcnE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=A2pVApyt; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A2pVApyt" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-4a42cb03673so13756811cf.3; Tue, 24 Jun 2025 20:11:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821096; x=1751425896; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=zuqVIBjGXQL3Pb0oRS7JXPWUTOWoDxLBBUTqhc/Fk5g=; b=A2pVApyt9GS4pQ5YYfwGwcCEoaKVm+tuAik77YJUHLRIVotpF+CeaVRgVmgWrwXiv5 DJdZkBJAtQ8ngvfps6uV6SFv0Ot/M0OpvQTpAl/s3rkWc9FbdxOSX+CTZMn/PhXZmnRb m5wr3Dws2PY7ZgulJ1unaLgoEMbh8cT9AUwBwl3M3dxGQmSuPbsl6XztKlYbwNRBwhuX 9zCx9D9IgrdOM0V0Sda6QmFwBlqQJAkJ+qZ23mM27aVeXlMqG7Jt/jbQGBBZkq7BtcwG 7DmBmriO9GpcLXYQWU+kAYBOVblcos4igIiOwEaGX6MqcBNM3CyLKsn1QM8i/2OFsYN4 aQcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821096; x=1751425896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zuqVIBjGXQL3Pb0oRS7JXPWUTOWoDxLBBUTqhc/Fk5g=; b=bub1QYRWEwvI7/UniPwzYikvpjq6nJoCObBj/aRZbt7vgMRRZpiH2KpqVCYpW6T3PK tOnQsH/0PmPHegEUOMMMWi/XNxipnp2AxX3Ji/AIxtDQEoEoBYsRK+IBYL3W3Pte2GAk LVf2zLHIocAyfemEeDkDRBVyw3bhvL7ULlmWAHKhgHoDxwHjblwYrpmjSNHYGpQRKNoN cSP3t/o7Rkkho9HOVk+lhoXshxiyIZj3Lmc+GOx8BdY5D1334tWixveNxLd6JrT/8HKg lndjk6qUa/x2dT3CWhkVPeRkbRGYKHQHCigInM2REGsQt0JY8UoFhQ8QHGTxFpMgcg6D ZuiA== X-Forwarded-Encrypted: i=1; AJvYcCWeNg8wI1Fr0U46FhKrnBLRz6PDidaQVsrmNfgpQa+psPyWKsYu9og/1iWxi/KzpL5ySB6N7rA=@vger.kernel.org, AJvYcCWkIzu9JsnUNRv63CRSdVeIbHmjapx0S6Pra6PWxJQgp8WDMkvbAr8ImmoAlHx7caRzZyfW@vger.kernel.org X-Gm-Message-State: AOJu0Yy7HEJYYGfZOqEf3qd04K5jLpLFJQ10ftYl9wGGRj0e9cpnC4qx EyMZVAirI4GXcLIBfaylK+ZDaRlpo9Mo3bmMhkCC7T0DtQGssK/O9yWf X-Gm-Gg: ASbGncvV2hDcjdJjIr8P3yH8YXk+l1RIAU8A9h3hPIWmOfLT+elEQwZlu8t+DyyWPav nveTY+B5Z6EZ6Y7zURtVHx0ULBNhgNNkh4l7EIoGx5eqlM4KL7o94vDygKItnQImTM7P/jpx93p 18C3dYAnFVGqej4Uy2H/6KH1LUwZbfzmuR3t+CSSbwiKJxxzG93q1G9mxRdCLyMdEEiZ/lgA2q2 5cGG2d96EP+po6JkiCG5ldCgjSfVKz3D2uwgjAgtzOocXBPhbzOXffh0zAlC++m+fTR0K/rbEZo HTK+Ga5ZEEKUhsm1e3X+kFAYEL9VyDHKojWu/UziAKABSSIZ8tQ8a1vD9aVX/tlKE0zA4ODIKX+ LGWeugi8JegRECNxkHEfBRIgBKt1DlNsW6aoxy7TRckoeheHyQKOK X-Google-Smtp-Source: AGHT+IEbJPBHSvbUjLDI/ho9eHxTAS4JXGBSJ1tuasc06odARRqHpanKjW7XTONtNu3DCCjVSio2Vg== X-Received: by 2002:a05:620a:bc9:b0:7d3:f3a5:71d7 with SMTP id af79cd13be357-7d42973f34emr215814785a.40.1750821096186; Tue, 24 Jun 2025 20:11:36 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7d3f999a566sm570896785a.8.2025.06.24.20.11.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 20:11:35 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id 0826DF40067; Tue, 24 Jun 2025 23:11:35 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Tue, 24 Jun 2025 23:11:35 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtddvgddvudeiiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepgffhffevhffhvdfgjefgkedvlefgkeegveeuheelhfeivdegffejgfetuefgheei necuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghr shhonhgrlhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvg hngheppehgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohep vdeipdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehlihhnuhigqdhkvghrnhgvlh esvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehrtghusehvghgvrhdrkhgv rhhnvghlrdhorhhgpdhrtghpthhtoheplhhkmhhmsehlihhsthhsrdhlihhnuhigrdguvg hvpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgoheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopegsohhquhhnrdhfvghnghesghhmrghilhdrtghomhdp rhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgtphhtthhopegurg hvvgesshhtghholhgrsghsrdhnvght X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Jun 2025 23:11:34 -0400 (EDT) From: Boqun Feng To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org, lkmm@lists.linux.dev Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Breno Leitao , aeh@meta.com, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren Subject: [PATCH 8/8] locking/lockdep: Use shazptr to protect the key hashlist Date: Tue, 24 Jun 2025 20:11:01 -0700 Message-Id: <20250625031101.12555-9-boqun.feng@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250625031101.12555-1-boqun.feng@gmail.com> References: <20250625031101.12555-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Erik Lundgren and Breno Leitao reported [1] a case where lockdep_unregister_key() can be called from time critical code pathes where rntl_lock() may be held. And the synchronize_rcu() in it can slow down operations such as using tc to replace a qdisc in a network device. In fact the synchronize_rcu() in lockdep_unregister_key() is to wait for all is_dynamic_key() callers to finish so that removing a key from the key hashlist, and we can use shazptr to protect the hashlist as well. Compared to the proposed solution which replaces synchronize_rcu() with synchronize_rcu_expedited(), using shazptr here can achieve the same/better synchronization time without the need to send IPI. Hence use shazptr here. Reported-by: Erik Lundgren Reported-by: Breno Leitao Link: https://lore.kernel.org/all/20250321-lockdep-v1-1-78b732d195fb@debian= .org/ [1] Signed-off-by: Boqun Feng Tested-by: Breno Leitao --- kernel/locking/lockdep.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index dd2bbf73718b..5c205dd425f8 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -58,6 +58,7 @@ #include #include #include +#include =20 #include =20 @@ -1267,14 +1268,18 @@ static bool is_dynamic_key(const struct lock_class_= key *key) =20 hash_head =3D keyhashentry(key); =20 - rcu_read_lock(); + /* Need preemption disable for using shazptr. */ + guard(preempt)(); + + /* Protect the list search with shazptr. */ + guard(shazptr)(hash_head); + hlist_for_each_entry_rcu(k, hash_head, hash_entry) { if (k =3D=3D key) { found =3D true; break; } } - rcu_read_unlock(); =20 return found; } @@ -6620,7 +6625,7 @@ void lockdep_unregister_key(struct lock_class_key *ke= y) call_rcu(&delayed_free.rcu_head, free_zapped_rcu); =20 /* Wait until is_dynamic_key() has finished accessing k->hash_entry. */ - synchronize_rcu(); + synchronize_shazptr(keyhashentry(key)); } EXPORT_SYMBOL_GPL(lockdep_unregister_key); =20 --=20 2.39.5 (Apple Git-154)