From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4198B199EBB; Mon, 14 Apr 2025 06:01:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610485; cv=none; b=hiAlSO51PDBDzdLPN/I08jL7q5MO5LCouOHDHXJU5ySxaFgItyZh54Lx1cERt+QqsVjOOTzudg0JNE5DjgOoqdxMnFqI+9vKy3ychGiQxpl7iE7pnM/DmBbSSjb81A6yIr0w64Gd4tj3LVRH+TvSaHsAa/+rR4n+/bgmX8tG5uE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610485; c=relaxed/simple; bh=MEZ7giHmtw9fWMENPevjSoJlsHffoTXUsiIwAwrxeW0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KM/navw2/70Qe23LGGy/PxhnWtbq8t59tzjuEUsGBVhV4Dq4s5iDD8w5a326KRW+bYutoecFfOW2U9GZbF81jBpOW64fTfGvdOmluM1GxGgTrTTFbDJrc9ONaItPAsqsIhQp8mWyeB6cDTJApblaaCbQUgrjKV2NL2nTqWXAKt8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hTJiMdCC; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hTJiMdCC" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7c5e39d1e0eso395084885a.1; Sun, 13 Apr 2025 23:01:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610481; x=1745215281; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=Vz0LbkXlPV4anrqa0D131BJNHrlwrd4cBtjVcyihCP4=; b=hTJiMdCCiceA5iMq315ops3QGFLCZXRJmFWWnCu9pp+DRtQyUfSbrB1cwO6StQiGW8 hfqrkctBNY74I9FabAVJ0+mtrsgB63XkTTU7JHwYYG3hqKZaRhqlO9W4vxAy29r+ysIU S6/a3rYXgw7XHFI4EJv+EBnX75kaYDJ4zzT50NvaUpXRLv0krBiobJdusXPzECxIF3Wz /oEQTE6bFruW/psZ9MbMm9QuAr5Yhjy0O21M+1DNpxGTolpuGt3xTlhwj2emrGjaBUoa BlQzKq/SMVs4EWub5j1wGT630I/xxa1jrn2iTldfx3jw8w0OcfS3gWs60bm0kc2wvP91 ks/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610481; x=1745215281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Vz0LbkXlPV4anrqa0D131BJNHrlwrd4cBtjVcyihCP4=; b=dovlGT2C5jq0yPck7pAC/FtKjnjMVW5JwhzzTEZYiS/SvX+DMpGpGrKd/lbKzr4+wt iKxVLM9HP8jTf9Z0BKIuD0CIZlIno4lK3NvD4tw/3xeGTpfVgr40qPz54ra/kFbERN4o cNIImcMNOxK4gy+jQnycC2fxkj4sEot5b5DIfeKL9gDwghPf+Oi732SoDq0qg70naYJN /dg9ZiHjaCKxGlAQWeEt07i1q+E6c+5/Z5ZxZ6NskTAcwKewc4yUHP793p3KeDy9EIeV G2Tv0b6m1NOLwfH2TUxkOh1MJSaYYbJENhyJ+zSkD65ri0yD0M9Jcfcsi7szMioVwzvN sVuQ== X-Forwarded-Encrypted: i=1; AJvYcCULf5FHUy2eQOiEeHUSMs68tbj7ENInfbuOeQp+EYmpgqQihPUgL+b1fVj114ZhlMlFD7v0@vger.kernel.org, AJvYcCUnhR7/ZR/115vrysA+0QVTril1QRJVF276i8qaCXbgrXZQOD8cm84PiopsK9411CLK8tP6Upe2@vger.kernel.org, AJvYcCXG7EIF0XdO3ivJ8He9ACFipjVatQCJnLlEuHulQlttfF8s3X3pvI8KhkarIFizf0C96YTOIAbljfWRT68=@vger.kernel.org X-Gm-Message-State: AOJu0YxQa/r0xNvc9ysji5RwYrVBDNLj4wo7/NVSkyrRKO+QT3fb9vS5 tRPXztBdKOZ5v5O0z+frSx37H+V2XaXK4GmlPtTJVmLynLglDLNp X-Gm-Gg: ASbGncvoGT2K6CpubYf8vTbZkKer6paRgPws4C/ZlhXdRykOSUC5fmA54EDLXxfgLP6 2h15oCBkNg5ADIwtnikPiqXLr0wTbv3cadB61YK+QCCCGbuGY8Kgjq4WWFkyFv4Dp6RyPQ+6V3t kihODi4u4+S4dEweC2Xqz1yf6t+B5nO3IyoGkn1KZMPagErDF+HdP5ICgXSKPkuCk7nWk4ZQWHH 1NGdfmG/nEAppTuAZjyDfuSmK50cieZBaM2cRKRCXvphXCmzQOs/nILvY1W/iGGfURo7+iM2cdH rrGpI/oMP6zK/v+kEtz31J5He+YEAJ14pxj3ZdzaHsYgjg4CLZMKbD3TXLQibvFosmJOAc19XYW 2wHxTwwW4XLXWIzKir+TGLc2f2rRZc2nQfmCF5o6WZQ== X-Google-Smtp-Source: AGHT+IFejSegVaNUERFZO2ig/sOvpu3Gz+JaRNwPXpESuceMZMl1hKDnV7fJekIv/VtMTAXs5iP55A== X-Received: by 2002:a05:620a:4708:b0:7c5:54d7:d744 with SMTP id af79cd13be357-7c7af13289dmr1821167585a.23.1744610480626; Sun, 13 Apr 2025 23:01:20 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c7a8969e66sm686730585a.52.2025.04.13.23.01.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:20 -0700 (PDT) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id 8A8E91200043; Mon, 14 Apr 2025 02:01:19 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Mon, 14 Apr 2025 02:01:19 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpefghfffvefhhfdvgfejgfekvdelgfekgeev ueehlefhiedvgeffjefgteeugfehieenucffohhmrghinhepkhgvrhhnvghlrdhorhhgne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqhhu nhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqdduje ejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvgdr nhgrmhgvpdhnsggprhgtphhtthhopeduledpmhhouggvpehsmhhtphhouhhtpdhrtghpth htoheplhgvihhtrghoseguvggsihgrnhdrohhrghdprhgtphhtthhopehpvghtvghriies ihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehrvgguhhgrthdrtg homhdprhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhho nhhgmhgrnhesrhgvughhrghtrdgtohhmpdhrtghpthhtoheprggvhhesmhgvthgrrdgtoh hmpdhrtghpthhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdho rhhgpdhrtghpthhtohepnhgvthguvghvsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtohepvgguuhhmrgiivghtsehgohhoghhlvgdrtghomh X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:18 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 1/8] Introduce simple hazard pointers Date: Sun, 13 Apr 2025 23:00:48 -0700 Message-ID: <20250414060055.341516-2-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" As its name suggests, simple hazard pointers (shazptr) is a simplification of hazard pointers [1]: it has only one hazard pointer slot per-CPU and is targeted for simple use cases where the read-side already has preemption disabled. It's a trade-off between full features of a normal hazard pointer implementation (multiple slots, dynamic slot allocation, etc.) and the simple use scenario. Since there's only one slot per-CPU, so shazptr read-side critical section nesting is a problem that needs to be resolved, because at very least, interrupts and NMI can introduce nested shazptr read-side critical sections. A SHAZPTR_WILDCARD is introduced to resolve this: SHAZPTR_WILDCARD is a special address value that blocks *all* shazptr waiters. In an interrupt-causing shazptr read-side critical section nesting case (i.e. an interrupt happens while the per-CPU hazard pointer slot being used and tries to acquire a hazard pointer itself), the inner critical section will switch the value of the hazard pointer slot into SHAZPTR_WILDCARD, and let the outer critical section eventually zero the slot. The SHAZPTR_WILDCARD still provide the correct protection because it blocks all the waiters. It's true that once the wildcard mechanism is activated, shazptr mechanism may be downgrade to something similar to RCU (and probably with a worse implementation), which generally has longer wait time and larger memory footprint compared to a typical hazard pointer implementation. However, that can only happen with a lot of users using hazard pointers, and then it's reasonable to introduce the fully-featured hazard pointer implementation [2] and switch users to it. Note that shazptr_protect() may be added later, the current potential usage doesn't require it, and a shazptr_acquire(), which installs the protected value to hazard pointer slot and proves the smp_mb(), is enough for now. [1]: M. M. Michael, "Hazard pointers: safe memory reclamation for lock-free objects," in IEEE Transactions on Parallel and Distributed Systems, vol. 15, no. 6, pp. 491-504, June 2004 Link: https://lore.kernel.org/lkml/20240917143402.930114-1-boqun.feng@gmail= .com/ [2] Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- include/linux/shazptr.h | 73 ++++++++++++++++++++++++++++++++++++++++ kernel/locking/Makefile | 2 +- kernel/locking/shazptr.c | 29 ++++++++++++++++ 3 files changed, 103 insertions(+), 1 deletion(-) create mode 100644 include/linux/shazptr.h create mode 100644 kernel/locking/shazptr.c diff --git a/include/linux/shazptr.h b/include/linux/shazptr.h new file mode 100644 index 000000000000..287cd04b4be9 --- /dev/null +++ b/include/linux/shazptr.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Simple hazard pointers + * + * Copyright (c) 2025, Microsoft Corporation. + * + * Author: Boqun Feng + * + * A simple variant of hazard pointers, the users must ensure the preempti= on + * is already disabled when calling a shazptr_acquire() to protect an addr= ess. + * If one shazptr_acquire() is called after another shazptr_acquire() has = been + * called without the corresponding shazptr_clear() has been called, the l= ater + * shazptr_acquire() must be cleared first. + * + * The most suitable usage is when only one address need to be protected i= n a + * preemption disabled critical section. + */ + +#ifndef _LINUX_SHAZPTR_H +#define _LINUX_SHAZPTR_H + +#include +#include + +/* Make ULONG_MAX the wildcard value */ +#define SHAZPTR_WILDCARD ((void *)(ULONG_MAX)) + +DECLARE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); + +/* Represent a held hazard pointer slot */ +struct shazptr_guard { + void **slot; + bool use_wildcard; +}; + +/* + * Acquire a hazptr slot and begin the hazard pointer critical section. + * + * Must be called with preemption disabled, and preemption must remain dis= abled + * until shazptr_clear(). + */ +static inline struct shazptr_guard shazptr_acquire(void *ptr) +{ + struct shazptr_guard guard =3D { + /* Preemption is disabled. */ + .slot =3D this_cpu_ptr(&shazptr_slots), + .use_wildcard =3D false, + }; + + if (likely(!READ_ONCE(*guard.slot))) { + WRITE_ONCE(*guard.slot, ptr); + } else { + guard.use_wildcard =3D true; + WRITE_ONCE(*guard.slot, SHAZPTR_WILDCARD); + } + + smp_mb(); /* Synchronize with smp_mb() at synchronize_shazptr(). */ + + return guard; +} + +static inline void shazptr_clear(struct shazptr_guard guard) +{ + /* Only clear the slot when the outermost guard is released */ + if (likely(!guard.use_wildcard)) + smp_store_release(guard.slot, NULL); /* Pair with ACQUIRE at synchronize= _shazptr() */ +} + +void synchronize_shazptr(void *ptr); + +DEFINE_CLASS(shazptr, struct shazptr_guard, shazptr_clear(_T), + shazptr_acquire(ptr), void *ptr); +#endif diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile index a114949eeed5..1517076c98ec 100644 --- a/kernel/locking/Makefile +++ b/kernel/locking/Makefile @@ -3,7 +3,7 @@ # and is generally not a function of system call inputs. KCOV_INSTRUMENT :=3D n =20 -obj-y +=3D mutex.o semaphore.o rwsem.o percpu-rwsem.o +obj-y +=3D mutex.o semaphore.o rwsem.o percpu-rwsem.o shazptr.o =20 # Avoid recursion lockdep -> sanitizer -> ... -> lockdep & improve perform= ance. KASAN_SANITIZE_lockdep.o :=3D n diff --git a/kernel/locking/shazptr.c b/kernel/locking/shazptr.c new file mode 100644 index 000000000000..991fd1a05cfd --- /dev/null +++ b/kernel/locking/shazptr.c @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Simple hazard pointers + * + * Copyright (c) 2025, Microsoft Corporation. + * + * Author: Boqun Feng + */ + +#include +#include +#include + +DEFINE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); +EXPORT_PER_CPU_SYMBOL_GPL(shazptr_slots); + +void synchronize_shazptr(void *ptr) +{ + int cpu; + + smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + /* Pair with smp_store_release() in shazptr_clear(). */ + smp_cond_load_acquire(slot, + VAL !=3D ptr && VAL !=3D SHAZPTR_WILDCARD); + } +} +EXPORT_SYMBOL_GPL(synchronize_shazptr); --=20 2.47.1 From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B1D91A2564; Mon, 14 Apr 2025 06:01:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610487; cv=none; b=XIcI+ORQb18KYy5k9ACM3tje/5a8y1bm44HqQxrUyIe6y15b5alNrod6xH9mq8yk3Igl3KtmnUx1f4/1GvLHdpXOQAHgBKW8fKgZw18B7zf9nFL9LjtlbQEsewn5bATRLfoIcq7DMCg9fwb/30DgC4zLV7M4X4HfJ5V9vkNvods= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610487; c=relaxed/simple; bh=0a0sKTLnZCu8ERlafxFR8eDC2liFbwaV8kBs40QJ5AE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uspe4jhIueAtxjiFvEWbnqcx9D0e/91E17qF+nG3ENm0AjzzdARzYVLODHZDg+JUqcZN76ItoUunBxkapS8KM+JcRAHFM5bhC/89XCKEVW4t6klW5CioEYj2THWEXZGdWfsVa0MSQvN2HssbVOF/VRgyzWpOqm2lCCo7hxPtFQA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nmp04mx0; arc=none smtp.client-ip=209.85.219.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nmp04mx0" Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-6e904f53151so35091236d6.3; Sun, 13 Apr 2025 23:01:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610484; x=1745215284; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=crSoWmS+wlxuDSJzcFBATB5TfRUeFAx3Edno5+zB6/o=; b=nmp04mx06rTQiN+A7tKaCjhvmrSokinyu3LHHlvmc7529SNIMw+HSTISh/Pt8D2htI Pd/DGiM7r8Cu6GZwSmKCe1rCDVr9ZWytGG3r30Yo79rE4rRcqFYdrABBLQsgs6c5hmw+ Up09p1GpYbptUnGdSE7sMcuNXhRBBF0TVAhwOH/h2dYWATXTubRGvcHlt6enmCOjCBEO SnBuCwHD02aDOIFVj7DJvLFrhjKQk2oYap62rEtHgJQeK+em8SdqHR5y0iMPeqvkZCe8 YIUTWVhsdUk4CabGHjGOHqN6drx1SZPsZXh7QhNYUMDKOXuYqkxcnPF4gv6gWajkH2Q3 WEpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610484; x=1745215284; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=crSoWmS+wlxuDSJzcFBATB5TfRUeFAx3Edno5+zB6/o=; b=HnczNQaP8YCIOAbhhoeB72YvrCXmvjj9PDRtI/Ag7eWmDYZvBE8YGTkMGk98H6rJzV 7fY954NimpIblCmL4cg/G4btKFonf9ruhbDDafc7u+bqVxZGX365zp8g7dsk8dnjD++n qbOTrHEWUMV7wjG40eOwlDne1IAyR133JY0YVlxx6QlfHKu3qT8QiamjRjkT2T75x9cn 4i8PWk2TMnKkqFHttLYRWTTmW5x20R06gEdOlNsfJZoJBZplRyitjCjjl/PYunR0DdfA ZiTIk3mEh7HfW5hxX35iQ4eXqsLe4D42PaAgZ4JMzoVcHRAcxGWkK8Egp23EQE54FpMu HFZw== X-Forwarded-Encrypted: i=1; AJvYcCUHJafggFpaEU2CfStkxVO1hOj7ErcE1amZ+jSG0xCOoiZ97J9N+IrtRBH/uKhzBCcdHVXCLWgZ8HFCZFY=@vger.kernel.org, AJvYcCWnXBGzej159CUMBpvjZOjwst/Ehh+xv5MwiS/d1TzGQfFDnKTqIyb5lhij7yth/geuB97Qagdz@vger.kernel.org, AJvYcCXmTc7Z0qBrP6gHKp42e8PBFfXnGJTv4aUjDGAH8eMbRltth7mHwRYLI+7qwRx9HbcOolnH@vger.kernel.org X-Gm-Message-State: AOJu0Yx9oPbEGZ8Q8QT3FvzxV2/LCnB+4WdaJOcExTf1ywszMNgiI0hK fEDggQMZGVbJPaYV2cMjQryf+RE9TpeWXeqZUsa6eE1b0I//1EXUMyEFLg== X-Gm-Gg: ASbGnctIPRSteLY16pJmd08h2FZCWKlMvlcv3jMHgMadwFT9sMwPcapwp9BR76BhQ5k 4ikvyCA37r8dT4CE5U+kzEIqrEYIlPDAAxKR7J086UiXv2L0/X89ZPPserLtsGMsfyj7x6WFujJ 7azTddogBDqhYjwbQG/t/p9h/pnCOxVtHbrpTRwvs9NiYP7nq/GLY3Vu2a9yQb0127zrGeXC5Iq A64rd4Zs/a2PtqeGvibRDfEj/3nYVV5E9aiqaskGpJSe/cttrNIfJVAkvxCXJR8qJ9JEeIGSyyD LgjNTVApSjVNsC1it4z194F31HFajGZUridcRqFQxgw5cWifJz/yz9cvEBhsDBK9hPdmy4cnFn8 GlaeyBqsUp5K6w696oxNWscYJ9vptFDo= X-Google-Smtp-Source: AGHT+IGUm83pSj0m13ld8odbNA7X/XFuelxmJULW6GNlofNBV4oRp5k19QpyqoObZltGPO3/o2xurA== X-Received: by 2002:a05:6214:5089:b0:6e8:86d3:be78 with SMTP id 6a1803df08f44-6f23f14665dmr155990676d6.37.1744610484032; Sun, 13 Apr 2025 23:01:24 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c7a896b01fsm684471485a.65.2025.04.13.23.01.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:23 -0700 (PDT) Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfauth.phl.internal (Postfix) with ESMTP id 046FB1200043; Mon, 14 Apr 2025 02:01:21 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Mon, 14 Apr 2025 02:01:21 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeejucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpeegleejiedthedvheeggfejveefjeejkefg veffieeujefhueeigfegueehgeeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgr lhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppe hgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohepudelpdhm ohguvgepshhmthhpohhuthdprhgtphhtthhopehlvghithgrohesuggvsghirghnrdhorh hgpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgohesrhgvughhrghtrdgtohhmpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgt phhtthhopegrvghhsehmvghtrgdrtghomhdprhgtphhtthhopehlihhnuhigqdhkvghrnh gvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehnvghtuggvvhesvhhg vghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopegvughumhgriigvthesghhoohhglh gvrdgtohhm X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:20 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 2/8] shazptr: Add refscale test Date: Sun, 13 Apr 2025 23:00:49 -0700 Message-ID: <20250414060055.341516-3-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the refscale test for shazptr to measure the reader side performance. Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- kernel/rcu/refscale.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index f11a7c2af778..154520e4ee4c 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -890,6 +891,43 @@ static const struct ref_scale_ops typesafe_seqlock_ops= =3D { .name =3D "typesafe_seqlock" }; =20 +static void ref_shazptr_read_section(const int nloops) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { guard(shazptr)(ref_shazptr_read_section); } + preempt_enable(); + } +} + +static void ref_shazptr_delay_section(const int nloops, const int udl, con= st int ndl) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { + guard(shazptr)(ref_shazptr_delay_section); + un_delay(udl, ndl); + } + preempt_enable(); + } +} + +static bool ref_shazptr_init(void) +{ + return true; +} + +static const struct ref_scale_ops shazptr_ops =3D { + .init =3D ref_shazptr_init, + .readsection =3D ref_shazptr_read_section, + .delaysection =3D ref_shazptr_delay_section, + .name =3D "shazptr" +}; + static void rcu_scale_one_reader(void) { if (readdelay <=3D 0) @@ -1197,6 +1235,7 @@ ref_scale_init(void) &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &sched_clock_ops, &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops, + &shazptr_ops, }; =20 if (!torture_init_begin(scale_type, verbose)) --=20 2.47.1 From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02EE71A23B1; Mon, 14 Apr 2025 06:01:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610486; cv=none; b=qYIVp/BMDPz5ISY1yBfZllPA5XG1b/v2VDOD9asUCjCTeLqUFyC0EkIHFuEUZ4I6oNYF1YfSGtljwg4AgYQFbf6058s77EZAJAF3N7JcEDo1tM1tI0hx/b5K3gPwsgq6r8s4sGMIYHQ2InFc+p6+KSjIVZ9nLK6aPI0EX5lmWlY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610486; c=relaxed/simple; bh=46AnLWGOvwy4R4xqUGhMY1GRT3BPqIPloV90uxpkFIs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cHc+lcRSFdXLNOf0p85oesQsm4d3TjzHb8i0F/RryLCzc5ssYsfZQQ/vCZPBRm8jiP+3LJ6/4c4uOjkL312HYzHgHUa+Lw87y/yOnoG/Sp9dAXh4oD3I2dMz+NAnGauHzp0JzzlqW2tcEZXxIiIuUWhmaIKU0gWWLhONQQnZqDc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=h3N7LvDU; arc=none smtp.client-ip=209.85.219.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="h3N7LvDU" Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-6f0ad744811so23548276d6.1; Sun, 13 Apr 2025 23:01:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610484; x=1745215284; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=hLSzZQgiNEWo3FbcTs2LNPwbUJUrF+q+NANxOYOv6MA=; b=h3N7LvDUFY+dJ4T3xJbkR8Pl7u5+JA1j8ZhaixW/mvcbe42qwhqvsuFHXpB6Kr6ztG vNuJXjZAlaAT1mVl5444W1qMAn5K8NJV941UH7fvsg79wc1DkTygEooRER9bWMu3jKrq U9yG2Uz5F3vYjeC29kpcXbiE3K95zMQZVf3B4uUVVWxI3N2//oYU4BTH6HDoctAE6CWa H8mp1me3vW0gcP/aGvDvHOSlQIpulkluNkroNyGAvSkl39yuyl843Y8PvkRxEMehEbKp DFYGCpE/rhw0xKasPmaD0l41rYbKY/Ne/vmn8+KQj/wjfLImhaeOlaFErd2vS6R3MWNJ bpQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610484; x=1745215284; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=hLSzZQgiNEWo3FbcTs2LNPwbUJUrF+q+NANxOYOv6MA=; b=aQa1Gdc+BzQ00COmLrS9CB/0mhua3Zh2y25s88tEwn51377+RkXsPQoQgSSdy6RrPE q7Qe+XSW0iD4rzN+maX1X2eeWcR/m+Q1/jh4mFd7ACDTAyzWKIAS3Zd4u/OIYMiSOgCS 3bV+n7L+msNf2Ozol4enfWHs8hPAZoukhdy30FLLBx9F7OLPN4c6ObP4H8ekzit+++hw NM8zBWa1dru+hbZH+eXsvBt5YEn9B99M5Ux/iTkcuoZoEwA7mzFkbjYBL+El5T1eS3xb hQduXqCy+k1eS2sXBUIdQUlPXRGDOaOHLhBpp+GGKgTWhYm3kmmWpLJSjYMobq5kWDYE s55g== X-Forwarded-Encrypted: i=1; AJvYcCVHhYV7TkBpXFzSLShOYXkdv6IBYXu32EMqxpvQaYm3mzP6Vg8f3fb7p2FwJ/l2dbTu4qqm@vger.kernel.org, AJvYcCWTMpaXjMWd2uIyOZi8UtqYfevs7TtspuDi2KoKHHbe6cjkdNtPHJQ10F16Yj+78NZFA96MPO6v@vger.kernel.org, AJvYcCXNjFLS2i7r96QbaB6+KEUYftYcCEU8LeMwegIoD28nRDJzUsNBjXvc305h92JnsK0wmJL8hLkM4ZvJ1YA=@vger.kernel.org X-Gm-Message-State: AOJu0YweWRUs8Ie4xywaYcxmoPZr0/Vk5s1tNnRnIp1rK747TIcHuJRq WvTM5FkDMl1IS7EF0thkjsCRwaUW5LB65SOrriiW/PDOQ0Z8T2tv X-Gm-Gg: ASbGncsNhld3BGThyACVXB+Kbt2z+N59oRRE4d3O+MfqxjnVmPTmLXtNvXAUChJ6vgN nyB3TpgUZWNUZ4t8sA+3Njx+x+kvXPsi7/CgdVd4ojJkxMIm0sIGTkDc0XYguNJaiYBcTTxJ0ts 4ZEHnnEeW8Hb7+iU8ayAEpco2QIVyVvS4haZjyVUvtDnMNjZbGP6cl+QFWpjyz3IewnLyea8Myd 3Vp8RzrMO4VECr6hLXh4X8eRbE1XNhUlXEIn3HGuhpU2GW8kbvHRRSu3vM5IXIj0AeV9fT4rRwP 5GR4GkOq8IHwPt6SYL3Is/VfqY7b+RmWUaFALPAKj+IO6cBqDsZ5drf7+l/bfiwJiyuJkpZf+DE 6k4bkZQvJnhLG2RfyiWZgzd05rSbpdH1p97M/14XtHw== X-Google-Smtp-Source: AGHT+IHxjTo84xD1QJtcQfo6b2PbGTrIcy6W1myY8lywI+/f1fLgFOTQuFvktp2h5zmiI7EZ4JRoJQ== X-Received: by 2002:a05:6214:d83:b0:6e8:f3af:ed44 with SMTP id 6a1803df08f44-6f230ccfa37mr147657136d6.12.1744610483624; Sun, 13 Apr 2025 23:01:23 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6f0de95f7bdsm79022596d6.23.2025.04.13.23.01.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:23 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id 720101200068; Mon, 14 Apr 2025 02:01:22 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Mon, 14 Apr 2025 02:01:22 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpeegleejiedthedvheeggfejveefjeejkefg veffieeujefhueeigfegueehgeeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgr lhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppe hgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohepudelpdhm ohguvgepshhmthhpohhuthdprhgtphhtthhopehlvghithgrohesuggvsghirghnrdhorh hgpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgohesrhgvughhrghtrdgtohhmpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgt phhtthhopegrvghhsehmvghtrgdrtghomhdprhgtphhtthhopehlihhnuhigqdhkvghrnh gvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehnvghtuggvvhesvhhg vghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopegvughumhgriigvthesghhoohhglh gvrdgtohhm X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:21 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 3/8] shazptr: Add refscale test for wildcard Date: Sun, 13 Apr 2025 23:00:50 -0700 Message-ID: <20250414060055.341516-4-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the refscale test for shazptr, which starts another shazptr critical section inside an existing one to measure the reader side performance when wildcard logic is triggered. Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- kernel/rcu/refscale.c | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index 154520e4ee4c..fdbb4a2c91fe 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -928,6 +928,44 @@ static const struct ref_scale_ops shazptr_ops =3D { .name =3D "shazptr" }; =20 +static void ref_shazptr_wc_read_section(const int nloops) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { + guard(shazptr)(ref_shazptr_read_section); + /* Trigger wildcard logic */ + guard(shazptr)(ref_shazptr_wc_read_section); + } + preempt_enable(); + } +} + +static void ref_shazptr_wc_delay_section(const int nloops, const int udl, = const int ndl) +{ + int i; + + for (i =3D nloops; i >=3D 0; i--) { + preempt_disable(); + { + guard(shazptr)(ref_shazptr_delay_section); + /* Trigger wildcard logic */ + guard(shazptr)(ref_shazptr_wc_delay_section); + un_delay(udl, ndl); + } + preempt_enable(); + } +} + +static const struct ref_scale_ops shazptr_wildcard_ops =3D { + .init =3D ref_shazptr_init, + .readsection =3D ref_shazptr_wc_read_section, + .delaysection =3D ref_shazptr_wc_delay_section, + .name =3D "shazptr_wildcard" +}; + static void rcu_scale_one_reader(void) { if (readdelay <=3D 0) @@ -1235,7 +1273,7 @@ ref_scale_init(void) &refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops, &acqrel_ops, &sched_clock_ops, &clock_ops, &jiffies_ops, &typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops, - &shazptr_ops, + &shazptr_ops, &shazptr_wildcard_ops, }; =20 if (!torture_init_begin(scale_type, verbose)) --=20 2.47.1 From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A5CF1AAA2F; Mon, 14 Apr 2025 06:01:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610488; cv=none; b=pDIFqYKnewgB/hfZyvg0uVvSqk22Ym6RxZ0G244D1WpE5E88/g56DyLm17VzGA2Z/p2CLMyM9cwsIz4qB5kWHzrURiezYw6ThHa2IGw/9X9afixbitsrw/eDJxpS1FJqovPzVSCH6oCWRB9RWIWKMPKx7jnUfe/vxH5uTEx90j0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610488; c=relaxed/simple; bh=vULrQ0jpP4keJ2UcJjqnghubGJoOGWiOvBA/vJ0Lm3Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sywc5Qyi53VhWZDVviBBykrDrr5PmpwYqXzfeU1LnQd0qRmr+RVcAy1LJ9QkNvVFno8gwx2VG1obNQPKpHa5vlwfhJxRRoK33DjgNy3QW5kLzztvbx3fE88DZHbZdlAg631RJsNMQ0yFSb7p2xc47pTfqMxRbIhkL+0g03H+9wM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cScjNYd4; arc=none smtp.client-ip=209.85.160.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cScjNYd4" Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-476ae781d21so39777631cf.3; Sun, 13 Apr 2025 23:01:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610485; x=1745215285; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=feihuqhdShvh+PYbO8b6wItOkRthqGBFbDYSVZiL8z8=; b=cScjNYd4kySVJ1MhWNC8JRplznFXTIvj64gy+4oAKSNHizkztNPWDvBy7vnWgNxxpR 6nK/FHbc3cTOqktJ4gHYc4NiLlXjj9j7xWJ8LUZ7NxRAPoOrNhX3MjkOZt2hM9uLjE1P HLNS87XRLVTe4OCd1KbprB/TTVdHe5/JNrhRSg/ctBQqWmtf9eGlqCkigUF2C9y3pqGn r5O+PSgTt9ivo7ly893PtbgTd5yxO5EB1oI7drtzz/wTLi3PePXhniTubyl+E1fi4s7U GJmfwBCP++Y9IgRviPoDs6DZ++t6wvEavVmCWrwOebil68bTt8ZW2pboSQJ+AWxJg3vR RZcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610485; x=1745215285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=feihuqhdShvh+PYbO8b6wItOkRthqGBFbDYSVZiL8z8=; b=EmDixZhporFLp2QMCfWep7D5XLxIwPGOhnsofrAf8fCclxtm+6Ql9+/vlG0nmgfiww tQHc8XfP+t/FAORDV7yndCwHI2WU0qZo9wFdI9XbJtB9r8fChnIjdLFdHq0sYQ+bGpPP ZlmEmWaYqkH/JsYQ7R5Aa6qSiMwcAr/FQ4NHfO+Np/29dozf6X+sYdF3cteUQiRIxHIt /6U06ojKbIaHRW/P2VtXH2A6PwNvJXN62PZar89vubV0DLxMW4Jk4Fwag37pojRBDCT8 QE8Yo9runFWlx3Ht4i8b2qQmmTmz/fkUHKz1KgngqarnOPlgM+8v4Eyh0ilz5keyxpkM 8ACg== X-Forwarded-Encrypted: i=1; AJvYcCVJzWoO0r4ZuW4nzdCe7MVDFgsXlP30p9oq7LtD/XlbmfvXDsxS9C1BNFt10wXtd6Ayp0VK@vger.kernel.org, AJvYcCWIg4doOSscxYqULooipYWYYrvVQvCz4n4lcRUw1D7n+fZ9yZvYeBMJhvZYHXvXm4bqv8xMD/ma@vger.kernel.org, AJvYcCXcxjdGayxgFoxTURsdrs8k/0TK7ZPTa2qzHwtrZaEkaCiEu2tP/8P7kVP2V67CmqMstKyAO2ydgHwXphQ=@vger.kernel.org X-Gm-Message-State: AOJu0YzQ9JdO+6xdgseH7YADXD+IeiTcdNSj1+3SXyTHeGtHe/qfjtOX 3HyP7PrLJVj49w8vhu17XYXpBEYzQzv76rwQxnwIJVY6fH1J+bKP X-Gm-Gg: ASbGncuWvHoR/ZccdaVngpzPJUglKTU4gAMV2ftTk+EmWK0/M4CReX16qz2T4UqK9ZU 4wcV1N3I0nTs3T6p9NzARGP7UrjIMNsZ1tB9euJbCHrtUpgHWjF4ChfrG1wUuudRKgCYEknjgdI cyWw4vifqAB1n1p1gRKB0AMYA6O5zSV8IusLizMyXSkTPNtZuSnsfOdXmXhDYpx3/w/lOsiyHpS Oloxt8KZ1XAOpMAyyxnwMjKrs2y9yj6FUNjBhQMX4jFVIv5t1biSq+X/BVbZWcsNfPFPQz9nT/Y i01m4kdw0ETxadoYQv7vf7FI5bJKRAMV57WzffDUanDD/02C/GyNZZ701L27MSBwq4BVxbO1zFy comCo8IDdlqSVyh8KXA32wlOIjZplGO4= X-Google-Smtp-Source: AGHT+IGEJkYLHwxzmAyw65s263Q5wULXY2qJ8+qcdHRRZ40qmh84yNATRrITlB51bM300IVI1v4ibg== X-Received: by 2002:ac8:5f95:0:b0:476:5fd5:4de6 with SMTP id d75a77b69052e-479775b92dcmr170988601cf.40.1744610485009; Sun, 13 Apr 2025 23:01:25 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4796eb0b853sm69296191cf.14.2025.04.13.23.01.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:24 -0700 (PDT) Received: from phl-compute-05.internal (phl-compute-05.phl.internal [10.202.2.45]) by mailfauth.phl.internal (Postfix) with ESMTP id 0C74E1200068; Mon, 14 Apr 2025 02:01:24 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Mon, 14 Apr 2025 02:01:24 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpeegleejiedthedvheeggfejveefjeejkefg veffieeujefhueeigfegueehgeeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgr lhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppe hgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohepudelpdhm ohguvgepshhmthhpohhuthdprhgtphhtthhopehlvghithgrohesuggvsghirghnrdhorh hgpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgohesrhgvughhrghtrdgtohhmpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgt phhtthhopegrvghhsehmvghtrgdrtghomhdprhgtphhtthhopehlihhnuhigqdhkvghrnh gvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehnvghtuggvvhesvhhg vghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopegvughumhgriigvthesghhoohhglh gvrdgtohhm X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:23 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 4/8] shazptr: Avoid synchronize_shaptr() busy waiting Date: Sun, 13 Apr 2025 23:00:51 -0700 Message-ID: <20250414060055.341516-5-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For a general purpose hazard pointers implemenation, always busy waiting is not an option. It may benefit some special workload, but overall it hurts the system performance when more and more users begin to call synchronize_shazptr(). Therefore avoid busy waiting for hazard pointer slots changes by using a scan kthread, and each synchronize_shazptr() queues themselves if a quick scan shows they are blocked by some slots. A simple optimization is done inside the scan: each synchronize_shazptr() tracks which CPUs (or CPU groups if nr_cpu_ids > BITS_PER_LONG) are blocking it and the scan function updates this information for each synchronize_shazptr() (via shazptr_wait) individually. In this way, synchronize_shazptr() doesn't need to wait until a scan result showing all slots are not blocking (as long as the scan has observed each slot has changed into non-block state once). Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- kernel/locking/shazptr.c | 277 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 276 insertions(+), 1 deletion(-) diff --git a/kernel/locking/shazptr.c b/kernel/locking/shazptr.c index 991fd1a05cfd..a8559cb559f8 100644 --- a/kernel/locking/shazptr.c +++ b/kernel/locking/shazptr.c @@ -7,18 +7,243 @@ * Author: Boqun Feng */ =20 +#define pr_fmt(fmt) "shazptr: " fmt + #include #include +#include +#include +#include +#include #include +#include +#include =20 DEFINE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); EXPORT_PER_CPU_SYMBOL_GPL(shazptr_slots); =20 -void synchronize_shazptr(void *ptr) +/* Wait structure for synchronize_shazptr(). */ +struct shazptr_wait { + struct list_head list; + /* Which groups of CPUs are blocking. */ + unsigned long blocking_grp_mask; + void *ptr; + struct completion done; +}; + +/* Snapshot for hazptr slot. */ +struct shazptr_snapshot { + unsigned long ptr; + unsigned long grp_mask; +}; + +static inline int +shazptr_snapshot_cmp(const void *a, const void *b) +{ + const struct shazptr_snapshot *snap_a =3D (struct shazptr_snapshot *)a; + const struct shazptr_snapshot *snap_b =3D (struct shazptr_snapshot *)b; + + if (snap_a->ptr > snap_b->ptr) + return 1; + else if (snap_a->ptr < snap_b->ptr) + return -1; + else + return 0; +} + +/* *In-place* merge @n together based on ->ptr and accumulate the >grp_mas= k. */ +static int shazptr_snapshot_merge(struct shazptr_snapshot *snaps, int n) +{ + int new, i; + + /* Sort first. */ + sort(snaps, n, sizeof(*snaps), shazptr_snapshot_cmp, NULL); + + new =3D 0; + + /* Skip NULLs. */ + for (i =3D 0; i < n; i++) { + if (snaps[i].ptr) + break; + } + + while (i < n) { + /* Start with a new address. */ + snaps[new] =3D snaps[i]; + + for (; i < n; i++) { + /* Merge if the next one has the same address. */ + if (snaps[new].ptr =3D=3D snaps[i].ptr) { + snaps[new].grp_mask |=3D snaps[i].grp_mask; + } else + break; + } + + /* + * Either the end has been reached or need to start with a new + * record. + */ + new++; + } + + return new; +} + +/* + * Calculate which group is still blocking @ptr, this assumes the @snaps is + * already merged. + */ +static unsigned long +shazptr_snapshot_blocking_grp_mask(struct shazptr_snapshot *snaps, + int n, void *ptr) +{ + unsigned long mask =3D 0; + + if (!n) + return mask; + else if (snaps[n-1].ptr =3D=3D (unsigned long)SHAZPTR_WILDCARD) { + /* + * Take SHAZPTR_WILDCARD slots, which is ULONG_MAX, into + * consideration if any. + */ + mask =3D snaps[n-1].grp_mask; + } + + /* TODO: binary search if n is big. */ + for (int i =3D 0; i < n; i++) { + if (snaps[i].ptr =3D=3D (unsigned long)ptr) { + mask |=3D snaps[i].grp_mask; + break; + } + } + + return mask; +} + +/* Scan structure for synchronize_shazptr(). */ +struct shazptr_scan { + /* The scan kthread */ + struct task_struct *thread; + + /* Wait queue for the scan kthread */ + struct swait_queue_head wq; + + /* Whether the scan kthread has been scheduled to scan */ + bool scheduled; + + /* The lock protecting ->queued and ->scheduled */ + struct mutex lock; + + /* List of queued synchronize_shazptr() request. */ + struct list_head queued; + + int cpu_grp_size; + + /* List of scanning synchronize_shazptr() request. */ + struct list_head scanning; + + /* Buffer used for hazptr slot scan, nr_cpu_ids slots*/ + struct shazptr_snapshot* snaps; +}; + +static struct shazptr_scan shazptr_scan; + +static void shazptr_do_scan(struct shazptr_scan *scan) +{ + int cpu; + int snaps_len; + struct shazptr_wait *curr, *next; + + scoped_guard(mutex, &scan->lock) { + /* Move from ->queued to ->scanning. */ + list_splice_tail_init(&scan->queued, &scan->scanning); + } + + memset(scan->snaps, nr_cpu_ids, sizeof(struct shazptr_snapshot)); + + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + void *val; + + /* Pair with smp_store_release() in shazptr_clear(). */ + val =3D smp_load_acquire(slot); + + scan->snaps[cpu].ptr =3D (unsigned long)val; + scan->snaps[cpu].grp_mask =3D 1UL << (cpu / scan->cpu_grp_size); + } + + snaps_len =3D shazptr_snapshot_merge(scan->snaps, nr_cpu_ids); + + /* Only one thread can access ->scanning, so can be lockless. */ + list_for_each_entry_safe(curr, next, &scan->scanning, list) { + /* Accumulate the shazptr slot scan result. */ + curr->blocking_grp_mask &=3D + shazptr_snapshot_blocking_grp_mask(scan->snaps, + snaps_len, + curr->ptr); + + if (curr->blocking_grp_mask =3D=3D 0) { + /* All shots are observed as not blocking once. */ + list_del(&curr->list); + complete(&curr->done); + } + } +} + +static int __noreturn shazptr_scan_kthread(void *unused) +{ + for (;;) { + swait_event_idle_exclusive(shazptr_scan.wq, + READ_ONCE(shazptr_scan.scheduled)); + + shazptr_do_scan(&shazptr_scan); + + scoped_guard(mutex, &shazptr_scan.lock) { + if (list_empty(&shazptr_scan.queued) && + list_empty(&shazptr_scan.scanning)) + shazptr_scan.scheduled =3D false; + } + } +} + +static int __init shazptr_scan_init(void) +{ + struct shazptr_scan *scan =3D &shazptr_scan; + struct task_struct *t; + + init_swait_queue_head(&scan->wq); + mutex_init(&scan->lock); + INIT_LIST_HEAD(&scan->queued); + INIT_LIST_HEAD(&scan->scanning); + scan->scheduled =3D false; + + /* Group CPUs into at most BITS_PER_LONG groups. */ + scan->cpu_grp_size =3D DIV_ROUND_UP(nr_cpu_ids, BITS_PER_LONG); + + scan->snaps =3D kcalloc(nr_cpu_ids, sizeof(scan->snaps[0]), GFP_KERNEL); + + if (scan->snaps) { + t =3D kthread_run(shazptr_scan_kthread, NULL, "shazptr_scan"); + if (!IS_ERR(t)) { + smp_store_release(&scan->thread, t); + /* Kthread creation succeeds */ + return 0; + } else { + kfree(scan->snaps); + } + } + + pr_info("Failed to create the scan thread, only busy waits\n"); + return 0; +} +core_initcall(shazptr_scan_init); + +static void synchronize_shazptr_busywait(void *ptr) { int cpu; =20 smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ + for_each_possible_cpu(cpu) { void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); /* Pair with smp_store_release() in shazptr_clear(). */ @@ -26,4 +251,54 @@ void synchronize_shazptr(void *ptr) VAL !=3D ptr && VAL !=3D SHAZPTR_WILDCARD); } } + +static void synchronize_shazptr_normal(void *ptr) +{ + int cpu; + unsigned long blocking_grp_mask =3D 0; + + smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ + + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + void *val; + + /* Pair with smp_store_release() in shazptr_clear(). */ + val =3D smp_load_acquire(slot); + + if (val =3D=3D ptr || val =3D=3D SHAZPTR_WILDCARD) + blocking_grp_mask |=3D 1UL << (cpu / shazptr_scan.cpu_grp_size); + } + + /* Found blocking slots, prepare to wait. */ + if (blocking_grp_mask) { + struct shazptr_scan *scan =3D &shazptr_scan; + struct shazptr_wait wait =3D { + .blocking_grp_mask =3D blocking_grp_mask, + }; + + INIT_LIST_HEAD(&wait.list); + init_completion(&wait.done); + + scoped_guard(mutex, &scan->lock) { + list_add_tail(&wait.list, &scan->queued); + + if (!scan->scheduled) { + WRITE_ONCE(scan->scheduled, true); + swake_up_one(&shazptr_scan.wq); + } + } + + wait_for_completion(&wait.done); + } +} + +void synchronize_shazptr(void *ptr) +{ + /* Busy waiting if the scan kthread has not been created. */ + if (!smp_load_acquire(&shazptr_scan.thread)) + synchronize_shazptr_busywait(ptr); + else + synchronize_shazptr_normal(ptr); +} EXPORT_SYMBOL_GPL(synchronize_shazptr); --=20 2.47.1 From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB1671B0435; Mon, 14 Apr 2025 06:01:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610489; cv=none; b=tc6XHN/8Y2lQmp75m2nLWT3KLCIV2Mw/Te94aBrg6YKtbgY0cQR9yFcDC4iJ76QZXSpJgIWA1Qg7pv9zMlvmlJ5Tx0+eDLZNxWiMP9koCW9wgs1vzxGimZ1MmUkNev3xgzHa5YxxeDBCRvXzZj0ALsRKG/97yGolpeyV4S1BhXg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610489; c=relaxed/simple; bh=HpxyqsdMGeeK9YhHPSA4BDxEq1DuqcU6xX/njThfNuA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cJ1II+DUqgHYqNEQ4tJelpHJYf+UqqmLk/ohvhhzEt+V2Yv7U37vlKQpVjHtvUhnAtPCkVk+dgKeBrUYpyw1hUirEvOhYJcpXqGVlfIVDHUW5h7KQAnIdIaKXDpIn2nHDNISxzSIejaTJMoc/69xZhLM6cCgvJbcrPtiKAos3Bw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VNcoStDc; arc=none smtp.client-ip=209.85.160.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VNcoStDc" Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-476ac73c76fso40470441cf.0; Sun, 13 Apr 2025 23:01:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610486; x=1745215286; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=3mqBPYEpj3oDYcW+ecgfymE5jpTfEb8iQzjKpUR+M/M=; b=VNcoStDclQZuf2M11vA9a3gFCUslE8UE1N2vuih1nKbVIqpYE8Dlnvna5waqxB/3Cf 7HTWUrCh09S84vMgHaJ3rUJUiz6VlnGTq+DLs2FvVKQAtAWVkVJg0hPjL5T5KR8YOIdO P2p6HYXEDT1DRNvE1mxbBjlFWWp2cl7TeuD4A0RsHBguzws3ECsrri2po3iZj2c9OAtO R8RIexw+XDAhldqBFLLFqfs7I2GtRjj5gOcRxzDa5llCAfE3oXhaMoKAZ0QVp06vDOSl G6UVT+4n56pxqzETD+OUSuSJQcivXYcN3iiq4cjgSoV6nTBvDpkvFbWq8V6trFL1pWFN hURA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610486; x=1745215286; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=3mqBPYEpj3oDYcW+ecgfymE5jpTfEb8iQzjKpUR+M/M=; b=bsdog8c2jkyaT8BZfi9U7DR8flUVX/j58Gx4iH2D/cqCrhknOTvcJuxn2QHDMGbzwz qYFR1YDvXqdptjRMV0X3mBn0WHqGOk/58OztP4lntBgJ+05gMgHvkI0opNIt2vyOqk17 B2BNE1YMhTrhJ4j8DQsG7t9YQTDJgm+pEo61idgGLLMqzxTwZoDFwfn2YLdDqpXzh8fp E08slNIxdVcb35g+QZGWQvOtvDjtH87s5o2j3EsRC+75Nu55Mef4FVAlYVTwWDS0pTIc 4ZkbwZ9D9frkrK4+Juh4ZHv7DVE/2QBKBvQDxekOd+jta7FWny9uv0eUGQwjYWvjezIU E1hQ== X-Forwarded-Encrypted: i=1; AJvYcCUlkDk24fY6O8ncHGAZml7GdbYC2MTPtU040tJ1iLXmup/oA2LaKAcnMTB35v41uH2ptrQO0RoB0lIyVHA=@vger.kernel.org, AJvYcCVOpPooVTy+9JjFVPIPLqZYUvHJv+v51swpGDCsDgMtYDsIZ/Sa2Uar9DBanbV3XS5L06Du@vger.kernel.org, AJvYcCWmVOjl66Ggrw5K7VZc1mQH64t8zmCcnc27BzDwjMMIfBq6DWxbFUANwEFh8wSp17rnmutiemi4@vger.kernel.org X-Gm-Message-State: AOJu0Yx4eOVEg8dgVrgGLtTD6/wfn4IqwBf8SaV5Ma/jtYHbnGKXj0vd rfi5LpW1vYkt+w/gispiWyeDwmW2pLC0lDiBUjZdWdsiKiLSw3zf X-Gm-Gg: ASbGncvq7RpDdG7eBIVUBdYRcoZWtstfDBmzBg6NWe3INLp1uUjd0JIeJAvCCWkDZyD ahbAbwB20h1H8az83GWJ/cLfMIANi80RbmrlwyxvTlWMuJWzL2NvtyqE7Z4cmwnYqjIPhY0QnGT Y5c8Qa06B3gs2eHg/a2z54Vqxfo6h1/OkYJzUBp9fZNTOa/iDq4R0T1aX/+ids8TsdXA8sGuj09 u7tXd169VkXHj6S1m/4l5ZMESWZ5rVKOBoFGJMMhuIPl017iQttxobpRy1tAVO79GpnOOIv4t1t +z1o1KaWgKylMec6qvqCgoQZInmFNN67r0RshwZzEO7lvqHLAU4OEv8NqoHYqGqxk2lMz8Zua5O NW8hvCeqS/6bb9uBTQJrJEaEGUAdvGZo= X-Google-Smtp-Source: AGHT+IFgJw9RjZ7bPzTHoqthvvHfyn2CHESDBCpwdMKpPEwzMJFJSQWnogh732fCX+skQ5ehIli/Bw== X-Received: by 2002:a05:622a:190b:b0:477:8a01:afc7 with SMTP id d75a77b69052e-479773fa138mr136252621cf.0.1744610486486; Sun, 13 Apr 2025 23:01:26 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4796edc20aesm69144241cf.78.2025.04.13.23.01.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:26 -0700 (PDT) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id 6A59C1200043; Mon, 14 Apr 2025 02:01:25 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Mon, 14 Apr 2025 02:01:25 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpeegleejiedthedvheeggfejveefjeejkefg veffieeujefhueeigfegueehgeeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgr lhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppe hgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohepudelpdhm ohguvgepshhmthhpohhuthdprhgtphhtthhopehlvghithgrohesuggvsghirghnrdhorh hgpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgohesrhgvughhrghtrdgtohhmpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgt phhtthhopegrvghhsehmvghtrgdrtghomhdprhgtphhtthhopehlihhnuhigqdhkvghrnh gvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehnvghtuggvvhesvhhg vghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopegvughumhgriigvthesghhoohhglh gvrdgtohhm X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:24 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 5/8] shazptr: Allow skip self scan in synchronize_shaptr() Date: Sun, 13 Apr 2025 23:00:52 -0700 Message-ID: <20250414060055.341516-6-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a module parameter for shazptr to allow skip the self scan in synchronize_shaptr(). This can force every synchronize_shaptr() to use shazptr scan kthread, and help testing the shazptr scan kthread. Another reason users may want to set this paramter is to reduce the self scan CPU cost in synchronize_shaptr(). Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- kernel/locking/shazptr.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/kernel/locking/shazptr.c b/kernel/locking/shazptr.c index a8559cb559f8..b3f7e8390eb2 100644 --- a/kernel/locking/shazptr.c +++ b/kernel/locking/shazptr.c @@ -14,11 +14,17 @@ #include #include #include +#include #include #include #include #include =20 +#ifdef MODULE_PARAM_PREFIX +#undef MODULE_PARAM_PREFIX +#endif +#define MODULE_PARAM_PREFIX "shazptr." + DEFINE_PER_CPU_SHARED_ALIGNED(void *, shazptr_slots); EXPORT_PER_CPU_SYMBOL_GPL(shazptr_slots); =20 @@ -252,6 +258,10 @@ static void synchronize_shazptr_busywait(void *ptr) } } =20 +/* Disabled by default. */ +static int skip_synchronize_self_scan; +module_param(skip_synchronize_self_scan, int, 0644); + static void synchronize_shazptr_normal(void *ptr) { int cpu; @@ -259,15 +269,19 @@ static void synchronize_shazptr_normal(void *ptr) =20 smp_mb(); /* Synchronize with the smp_mb() in shazptr_acquire(). */ =20 - for_each_possible_cpu(cpu) { - void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); - void *val; + if (unlikely(skip_synchronize_self_scan)) { + blocking_grp_mask =3D ~0UL; + } else { + for_each_possible_cpu(cpu) { + void **slot =3D per_cpu_ptr(&shazptr_slots, cpu); + void *val; =20 - /* Pair with smp_store_release() in shazptr_clear(). */ - val =3D smp_load_acquire(slot); + /* Pair with smp_store_release() in shazptr_clear(). */ + val =3D smp_load_acquire(slot); =20 - if (val =3D=3D ptr || val =3D=3D SHAZPTR_WILDCARD) - blocking_grp_mask |=3D 1UL << (cpu / shazptr_scan.cpu_grp_size); + if (val =3D=3D ptr || val =3D=3D SHAZPTR_WILDCARD) + blocking_grp_mask |=3D 1UL << (cpu / shazptr_scan.cpu_grp_size); + } } =20 /* Found blocking slots, prepare to wait. */ --=20 2.47.1 From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CE0F1BEF7D; Mon, 14 Apr 2025 06:01:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610490; cv=none; b=f+TePK/USjqLReOSXgZJTMAdborS5TRzUZ9BoBZSzM0dlgK++TKyWe4Kq+RKXl57HK5PiEa39S+c9fSn07gpetplwNIpqSnAlG2jh1H3vfOtQLln9TtmaxFZ+NBLs1Oi8Tp5cfCGs1wbvXTwWncL6uUMQcsJiYikkkvk5JVryiQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610490; c=relaxed/simple; bh=cPAIxLpCJd/jsKJtkLMVLNpBa5sHhRlJb5p/IMXI0Fc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ocr7jIGf9/i7btwFVLmoTXwYg89B+zUR2ndBROWDBzalwOnIVLD/18CmzgoRSCsJDN8ZIaaX98+B5oFKNjo+Bz5Ey2WkWC1Zd+MBKxaZyBdeNqjfqFBernXJztmT0kjrhIr9RyEuPcJmwZJHLxH9kedlkIngK67U+UtAQjrKprg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iNFJBilc; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iNFJBilc" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-7c5ba363f1aso539441785a.0; Sun, 13 Apr 2025 23:01:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610488; x=1745215288; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=G1QmxPKlOY2AJ0bwfuxV24rh+8soVup+peq0V7qXYkg=; b=iNFJBilczHPum+QZwwP7VKiWrR3hoPdPmwJmuFlnXz3VQUCgWewPWekg5sswW3yUnv P37gN7yqcaj/pEfG6inKsey4Ky23uqiY8jTvhkG11Z4zHtJ6ETp/YgC1MPDmJc7U1dN2 9p5yFa0KOWXnEaoKOkUy+qifk9afUytJU3trdIMFhZAj/p+v/x7JYVzR6pLUAkcokKLO Tpvn+1Tc6hX38KRLABfOO8KDnkmRW/blapfn6V0qJzN+wisJ2JgnOICGkR+urHrcoC6f E5D0dUMgJNNbs0fklcwVJlIkvj4yR6ZbWI7Y6ukj0mEUixZx/iCaHVNx6TqMiDtB7Ich 35GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610488; x=1745215288; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=G1QmxPKlOY2AJ0bwfuxV24rh+8soVup+peq0V7qXYkg=; b=ViBlomGRhc7m2Bb93UE0W0bD7BuhZAIJdsMq75LiPPHkvbRywBmI1YoAALxppbbzn2 idiDhozesVU92hDVrafX1U0kHFZ8UesevoK2Vk0N0EhoR5FhwliVzIosbhuKlpEUF6yk mePT/Ro5F2Fa3Z9wNvptfkV0vadR3AGTjSvy6O1S/dlOr279GnkgRddiW+TgK/QOtUd4 MR9HvBfLele7gLxSiymScWRrl4VcYiZLiYNumnkGnVbmfxGFB10BIIoLfgRGSf1GXuNS J1TsnSXaTe9RTysT+ffphr8HIe1LPtxb68JgX8GpL0YZVJpS894UDSKIfMOdDDzveh9l GOWw== X-Forwarded-Encrypted: i=1; AJvYcCWHlSjsabVzeot+TpMOEHlC81IwYxJLPrAMVFZ2TCUE/pJm4r+nAtecqzzNcwyXf05Z77Mo@vger.kernel.org, AJvYcCXP945JKQ8dRqxiy3QJfFVIHxptbFpsPmwXwFjQHqLKYwybeNP4jdF0dpM/CjzcpQJvu//BSNJH5dygYQM=@vger.kernel.org, AJvYcCXgxbujlxgdOkS3L3HdDXsOjqGT/rSZ1oRlCwTHFiFPMpuX0rkRvkUB2XIzU0++lLA9ZI3kojHV@vger.kernel.org X-Gm-Message-State: AOJu0YyzTqEXXmaBb57JqXCduuW024d1z3lnPxG/UtK5xChkEpZk7gAy XreMQ9Hz0y0eQRQ403e5Xmlw2bSdgODYX/Se/X9fGC1Nlk33BK09 X-Gm-Gg: ASbGncvK2SU2NMqhxoQQfNAzmPHEXsNaYTGJZZ+oAupzrORI6MME/EHgCAGtMevp3gK 6RgmQ4Xv9NBY4tEhQpPWCKb1dSK6cLR9tpds17JumjzfP4uYTkYDBu96SfKgx2rwN4fSZQFzOPc Zuj0CSq0rp3tCV355As3pLzTi9oFKYqAYL9kuD+4jxGo2GnWLck38Tt1HrU+yMwZproHymmab4r 1XIAiEygBGqz2Owov/nNZDluElAGYV4lLXAPzFSwSJOzaIRbB4Zu6SPKd8CMwTw25aef1GcMQza 2MltI1fQUlfQg0psZh63QqM51tPrEEUEMegNFCdRMGs6+Z5hIL8O03CcKKw9frkicP2K8iemdQT uK9c0MGUm4J2T+gXcTswbLruVHc1klGc= X-Google-Smtp-Source: AGHT+IEkDKGgjvm3RVX2cBPaGJLH15qauFWnGGanryIKVwCwoPAa7fmI4pjLTlYeFgZ79MAusuX89A== X-Received: by 2002:a05:620a:280e:b0:7c5:6dc7:7e7c with SMTP id af79cd13be357-7c7af12a073mr1532923285a.26.1744610487682; Sun, 13 Apr 2025 23:01:27 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6f0de95f7a7sm78160046d6.13.2025.04.13.23.01.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:27 -0700 (PDT) Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfauth.phl.internal (Postfix) with ESMTP id D3F181200066; Mon, 14 Apr 2025 02:01:26 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Mon, 14 Apr 2025 02:01:26 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpeegleejiedthedvheeggfejveefjeejkefg veffieeujefhueeigfegueehgeeggfenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgr lhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppe hgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohepudelpdhm ohguvgepshhmthhpohhuthdprhgtphhtthhopehlvghithgrohesuggvsghirghnrdhorh hgpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgohesrhgvughhrghtrdgtohhmpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgt phhtthhopegrvghhsehmvghtrgdrtghomhdprhgtphhtthhopehlihhnuhigqdhkvghrnh gvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehnvghtuggvvhesvhhg vghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopegvughumhgriigvthesghhoohhglh gvrdgtohhm X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:26 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 6/8] rcuscale: Allow rcu_scale_ops::get_gp_seq to be NULL Date: Sun, 13 Apr 2025 23:00:53 -0700 Message-ID: <20250414060055.341516-7-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For synchronization mechanisms similar to RCU, there could be no "grace period" concept (e.g. hazard pointers), therefore allow rcu_scale_ops::get_gp_seq to be a NULL pointer for these cases, and simply treat started and finished grace period as 0. Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 0f3059b1b80d..d9bff4b1928b 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -568,8 +568,10 @@ rcu_scale_writer(void *arg) if (gp_exp) { b_rcu_gp_test_started =3D cur_ops->exp_completed() / 2; - } else { + } else if (cur_ops->get_gp_seq) { b_rcu_gp_test_started =3D cur_ops->get_gp_seq(); + } else { + b_rcu_gp_test_started =3D 0; } } =20 @@ -625,9 +627,11 @@ rcu_scale_writer(void *arg) if (gp_exp) { b_rcu_gp_test_finished =3D cur_ops->exp_completed() / 2; - } else { + } else if (cur_ops->get_gp_seq) { b_rcu_gp_test_finished =3D cur_ops->get_gp_seq(); + } else { + b_rcu_gp_test_finished =3D 0; } if (shutdown) { smp_mb(); /* Assign before wake. */ --=20 2.47.1 From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6AAB61A4F2F; Mon, 14 Apr 2025 06:01:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610492; cv=none; b=KZqrBfevVmz7XGqxWTos5w2ddeqnizcubJySzqbOYqjQ0VF0Lv0rkQidhAEG68YL+6WtCy8wz6D9Kk9Beq+vEuruoaTUCkqVA2UczK7dZsGSO/tFuamTIPKrnWvS8Qw/81jJ18ZmbOcAAEI5m6sMusBmcDXAVSSlPTwdM8hC6ko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610492; c=relaxed/simple; bh=o4mB1YbCjqOm0ivTnYwzfnJZ2vRxG5qWVplnqdlWr2I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mwf2ZbRLK/aeYRtPzssXdEEta1rJWYBGoTfc4j202teH+D0QEsfVzd9HAHxYh+jzF5Cx3mVISkjQ/9/I390KwFZ1InEtJ4kI+40q8M06gVXPq6drgBNlXAKWf5nOZkkncTgf5Dak+WA9pfBp3q6VtXy5RIf+VumWhxz5fvZaYmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RyMIcpFR; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RyMIcpFR" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7c59e7039eeso580951885a.2; Sun, 13 Apr 2025 23:01:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610489; x=1745215289; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=0yyPz4XrJINYrh8fPH3G34ZmNN9tSR8UbhBvw2fpH+U=; b=RyMIcpFRRxLpbj5SzaIEqQ6Kxa49pZIVQZKIiY/H/oSzObbCLb8LqeuOKVB+gCgaHp ga+7s7MwVMfL+tWnQNIlNCYxrsLZXr87rXAjyz7M9ruK2ws3SHFRGWTwVJn9uGr2Vrzr h57w5hur/PWHxZ3Nukokz+XApyJ2pAxsK64iGr+1jrPFzDfiEwi62l68qOD8P41Q8CWN E9dMgg4qX/CYLLGxm6nSrDYyeDA8yPmr8eYOWYvMte0COlMBR50bLE7ObOyXNMnVlwX7 uYw4ylUdaMoO5l2N7mTkOddKudsSt4YIO8EdJIXbckoLmSWSs8ZBBWDQ/VfZ5Ldc/wM0 ijJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610489; x=1745215289; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=0yyPz4XrJINYrh8fPH3G34ZmNN9tSR8UbhBvw2fpH+U=; b=M2r6wwPJ+gxrPIRyBDvFCLI3Vh80MK7zfwIh69nZNYqCBBoS7LB+mmgXOUsNqe99Sn 0KMCm6BeMt4RAPdjBNvwbiFZD4E0fQ2SpUP1cLBUhfpZ8GfBCv8SDW/G4btqKqiKlhjZ LfsZFooBJbhoBolLYt8vN0Pi/hZ0N8+hu1WoWWpqa4mINy7rVSJfs2A9vCsBCHkXmUxv oXlpz6bGkP5WLqOSgw5izAcS1u9sxPzXR3FLOHvlh6FtxoNFagbpaCFr3ulTCRuhaqYs mSDcuSEU+gTmTGc3suWBIdVkeUdn7WW0KGXl7u6GZbpcPtbkiUF9d3v7tCv9B7d/QlZe 5GLA== X-Forwarded-Encrypted: i=1; AJvYcCUWIqCyPo37+sWtMrHHb3b37IzuNtL7W1zXo18oGgxqG/Bdz9D/zprhIqBI18SWDaJeLf1/@vger.kernel.org, AJvYcCVpGKD7vr/U+PR/qwnVnY7qLxskvQvNWWRTtWoKVoARkDHTnWJiIoxbAiH+P45LbSYb+gH2ezue@vger.kernel.org, AJvYcCVwBl0OJ2MljJJaEJY31R9RTLTXtQ7/OwpKsv0O8k/rEispiuQm55Gm8INWJVgkHRs0p8IOsI0zjJU8kGg=@vger.kernel.org X-Gm-Message-State: AOJu0YynsAXrBhMYHkHkYEJwJttH5/g6zOUcqhgtf53BRi8X8UdVDRwv P0oz5o8rmMzGS4ddaxxmtKcjmwjwprwiE0RR4VSJywOhij17P90A X-Gm-Gg: ASbGnct7npZmZEvutGJOkqm43vXQd8doZLemm0co0mpIY6u5cL2SMP1ljWiwWWH88+b GGk7RcFUIg0FhXYqJvTMYVA2HTCH8eweg6kIzUDaXrq4ahEFMCzG/HKfDPUyvITKfQtrhD+5g39 zNYYbOTLfTr1zSvbA0dm8rk1Dqylrm0ijGFPhJ5YKimicutWAwqC1B04FKT0WXT7dwjd28Fb8WO PXQ/qxVvVeMvrSLB1hWeXefxgKLie0djLfzSCsG6IZ1RxZlNepseYMLHKSa19e4SqdZ1MiNRnQH dkseiLaY9L/uqKLiYdeKciHlvgtQIRXYQz9vF7m2Vk3iFd12txU6d5IP6YBy+okBi+sI71cYddJ JDuwx8DzetAZrKpVBW1xF2XG0p/aFzwQ= X-Google-Smtp-Source: AGHT+IGdwbtcc/LhSM+g7SxvZuQaQmJqgZUb+rWK7xVQAi5pEHJFx90WVfW5h8ejkETd2bD7aLB+Fg== X-Received: by 2002:a05:620a:4001:b0:7c7:a5f5:5616 with SMTP id af79cd13be357-7c7af1d8653mr1402849885a.42.1744610489258; Sun, 13 Apr 2025 23:01:29 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6f0de98096bsm77957846d6.59.2025.04.13.23.01.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:28 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id 616EC1200066; Mon, 14 Apr 2025 02:01:28 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Mon, 14 Apr 2025 02:01:28 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpeegleejiedthedvheeggfejveefjeejkefg veffieeujefhueeigfegueehgeeggfenucevlhhushhtvghrufhiiigvpedunecurfgrrh grmhepmhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgr lhhithihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppe hgmhgrihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohepudelpdhm ohguvgepshhmthhpohhuthdprhgtphhtthhopehlvghithgrohesuggvsghirghnrdhorh hgpdhrtghpthhtohepphgvthgvrhiisehinhhfrhgruggvrggurdhorhhgpdhrtghpthht ohepmhhinhhgohesrhgvughhrghtrdgtohhmpdhrtghpthhtohepfihilhhlsehkvghrnh gvlhdrohhrghdprhgtphhtthhopehlohhnghhmrghnsehrvgguhhgrthdrtghomhdprhgt phhtthhopegrvghhsehmvghtrgdrtghomhdprhgtphhtthhopehlihhnuhigqdhkvghrnh gvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehnvghtuggvvhesvhhg vghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopegvughumhgriigvthesghhoohhglh gvrdgtohhm X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:27 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 7/8] rcuscale: Add tests for simple hazard pointers Date: Sun, 13 Apr 2025 23:00:54 -0700 Message-ID: <20250414060055.341516-8-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add two rcu_scale_ops to include tests from simple hazard pointers (shazptr). One is with evenly distributed readers, and the other is with all WILDCARD readers. This could show the best and worst case scenarios for the synchronization time of simple hazard pointers. Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 52 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index d9bff4b1928b..cab42bcc1d26 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -429,6 +430,54 @@ static struct rcu_scale_ops tasks_tracing_ops =3D { =20 #endif // #else // #ifdef CONFIG_TASKS_TRACE_RCU =20 +static int shazptr_scale_read_lock(void) +{ + long cpu =3D raw_smp_processor_id(); + + /* Use cpu + 1 as the key */ + guard(shazptr)((void *)(cpu + 1)); + + return 0; +} + +static int shazptr_scale_wc_read_lock(void) +{ + guard(shazptr)(SHAZPTR_WILDCARD); + + return 0; +} + + +static void shazptr_scale_read_unlock(int idx) +{ + /* Do nothing, it's OK since readers are doing back-to-back lock+unlock*/ +} + +static void shazptr_scale_sync(void) +{ + long cpu =3D raw_smp_processor_id(); + + synchronize_shazptr((void *)(cpu + 1)); +} + +static struct rcu_scale_ops shazptr_ops =3D { + .ptype =3D RCU_FLAVOR, + .readlock =3D shazptr_scale_read_lock, + .readunlock =3D shazptr_scale_read_unlock, + .sync =3D shazptr_scale_sync, + .exp_sync =3D shazptr_scale_sync, + .name =3D "shazptr" +}; + +static struct rcu_scale_ops shazptr_wc_ops =3D { + .ptype =3D RCU_FLAVOR, + .readlock =3D shazptr_scale_wc_read_lock, + .readunlock =3D shazptr_scale_read_unlock, + .sync =3D shazptr_scale_sync, + .exp_sync =3D shazptr_scale_sync, + .name =3D "shazptr_wildcard" +}; + static unsigned long rcuscale_seq_diff(unsigned long new, unsigned long ol= d) { if (!cur_ops->gp_diff) @@ -1090,7 +1139,8 @@ rcu_scale_init(void) long i; long j; static struct rcu_scale_ops *scale_ops[] =3D { - &rcu_ops, &srcu_ops, &srcud_ops, TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_= OPS + &rcu_ops, &srcu_ops, &srcud_ops, &shazptr_ops, &shazptr_wc_ops, + TASKS_OPS TASKS_RUDE_OPS TASKS_TRACING_OPS }; =20 if (!torture_init_begin(scale_type, verbose)) --=20 2.47.1 From nobody Fri Dec 19 12:50:28 2025 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 498F31D63EE; Mon, 14 Apr 2025 06:01:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610493; cv=none; b=tPo6fSwLVIWwUMvNTQvGNmgq4Gm89jRN8lfTreHHpE/yxb847BiyD2MwBJyGIVk25/626QD0c758pEF4UU/QBScYmiSYa6jcvBLVLmy2DoOKTvOKO0mDAU5NZseaH890XAjPsSEuwzxYN5nJ/GBST1PIt8GBu9iFx6oPHbbfB3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610493; c=relaxed/simple; bh=gSFvINbMx+mpurJjbuzYdJO9KVleuEcbzdDZgdWmmk8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=llP4Ffw1UzYoVRfXdq11ZA/yjJo7DCcUWFPJchui2J/5t7Hax9LBuLYEUMMDXaaR5K0dOZKy3aVMHShnhBFBMkhwy/fVwzOsRP/lPqa2+Hl+/FZbPhFBY/4JI8s2MZaNCxmjSXFFJJmkuHYQaXbJBjw8uQOY6aLtzOMnA62nYCg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eKQXjyFX; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eKQXjyFX" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-4774d68c670so56759131cf.0; Sun, 13 Apr 2025 23:01:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744610491; x=1745215291; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:from:to:cc:subject :date:message-id:reply-to; bh=8yEZCCjAIKX8Bcif/S/pGprjlBDXpto+eqqOQvtfBHY=; b=eKQXjyFXrSCPvnyv1YTcT3o5a9SCJyKmbdV994s5qz0eLxjG/1EJbPrdxeqWWIVdU0 F7mnGFazccSlEJ6M9GxxPtfJMaqaD9TPfVqiTu2gs576kDHm2kms/zVoJGeInFicXhA3 8dRj4vtzfGhKlxHbS8qfzzF/wWzz/VUL17fTxvIWbyuVsftNzVji2AWpFn8VVhqXKgPi b5K/2GZonQO7WbKQqvBTvZi7f3FE4A1MQfp35i5+VOkSsjNOyc4KIL7x8aBbK6+35m7E dII/VjUG9jzD08qh3m8J1+mISJvBCBN/s9UbOK7qMcW91AGE8BbcSvy7ebtnF9abFcfT ww+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744610491; x=1745215291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=8yEZCCjAIKX8Bcif/S/pGprjlBDXpto+eqqOQvtfBHY=; b=mx3v1YcYt/1Hgop4AowfkPd2+VrKiL8RguJ9/8SY05bt8XZrEc6Yo75lK2/xGRNoEF IICVVLFVY0/065RHEW+PKgnTNUW3jGjSxt3IzmtKTn/MOfiSnR7gj5zzUWfRtDmtPTMO oW1WL1Y5rm3G/e/B1AKN0pC+NVM7d5IlSFajsDx7ENxoxv3/3C0tHJuJ5qwMNi8e2hcz gSdrGmV277Hg2NPCRbqFlmGFlrWbx2rFF1u/pZd/OQPa8KltlknDiuCv3RjlNjL47zSm qVePhNCeOaIAXmhLAf1gFh8SJue0efcbsqzAP4OuAFsOsJlNg6O/YPHcDFIQkmLhxbTl eqhA== X-Forwarded-Encrypted: i=1; AJvYcCUW2hktAFmUcxcPKAXP8Y94vMdyTH855bz3eIEoB2ARJrfJmGN0dTf3i5F9oiebzNOtmcDm@vger.kernel.org, AJvYcCUzyfHmAupOw+s33NcleUnubsG8VVLg4Yr1rMEVzP4kkkCmOzHukl/yrlSICfzHjqDFUicHjLOV@vger.kernel.org, AJvYcCW6rb/313aloITG0NDsRUUugyrJD7PP78uDoAUsVIGU4fMp38lDdUIZQIGiZIyuvXcdyWlnTxPzP7hVdx0=@vger.kernel.org X-Gm-Message-State: AOJu0YyDoYdItgEKpq7qqfjxvjJnYNYSiTZ5MGua6HHuGfYMZVvdlkNr QnWA/9cBUq52ojyZVbW3sHerVC8PsumVc0BnKQttj+73/A5gg2oZ X-Gm-Gg: ASbGncvOULVFHUAHzBZRjE1xipAZp9cRf2Ff3Z/UURmcViLwA1Hm8eWpkply+qnkonX 2QaQ1YCGn9e1hhwlD/bz+A1cSP7LG5q6CdDMyNqiH0dhy1gG5mp1spWvhuOOI/4K7wXS5TNKNuq r7qstTGYefA9BsyQhKAIt0yQqoaPiId0KBokrB31Upec7EpVLrcGyHHY45mdk/sHkq3HiruI9oc UJgEJ7Lm/8KPvjus0KTLkyPw5POIoVxrwhjpGrcJxqBvCJlUu1f9LK/IY/1rVMEANDmcHXViO8N wZasUpnfsN6XgKUZF9K7UHzBkyW/YCGEiChxTJHpcw+swHgvBo++1aZFGEk+dWlvCHi+lCEJaFJ 6wFQpIDn82VbsnT/Jk6sF86CKM1KAwUI= X-Google-Smtp-Source: AGHT+IHDWExJgXvc7ntxhN+BtiblsGHzc2LmReFgtphjkcdAbqKB5qry0e5AfEx4TNIw1Qs3SvIh3g== X-Received: by 2002:a05:622a:1449:b0:476:87f6:3ce4 with SMTP id d75a77b69052e-479775d3200mr175981031cf.39.1744610490778; Sun, 13 Apr 2025 23:01:30 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4796eb15d12sm69685261cf.27.2025.04.13.23.01.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Apr 2025 23:01:30 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id D4E601200043; Mon, 14 Apr 2025 02:01:29 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Mon, 14 Apr 2025 02:01:29 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvudeljeekucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredt tdenucfhrhhomhepuehoqhhunhcuhfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrih hlrdgtohhmqeenucggtffrrghtthgvrhhnpefghfffvefhhfdvgfejgfekvdelgfekgeev ueehlefhiedvgeffjefgteeugfehieenucffohhmrghinhepkhgvrhhnvghlrdhorhhgne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqhhu nhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqdduje ejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvgdr nhgrmhgvpdhnsggprhgtphhtthhopeduledpmhhouggvpehsmhhtphhouhhtpdhrtghpth htoheplhgvihhtrghoseguvggsihgrnhdrohhrghdprhgtphhtthhopehpvghtvghriies ihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehmihhnghhosehrvgguhhgrthdrtg homhdprhgtphhtthhopeifihhllheskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhho nhhgmhgrnhesrhgvughhrghtrdgtohhmpdhrtghpthhtoheprggvhhesmhgvthgrrdgtoh hmpdhrtghpthhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdho rhhgpdhrtghpthhtohepnhgvthguvghvsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtg hpthhtohepvgguuhhmrgiivghtsehgohhoghhlvgdrtghomh X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 14 Apr 2025 02:01:29 -0400 (EDT) From: Boqun Feng To: Breno Leitao , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long Cc: aeh@meta.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, edumazet@google.com, jhs@mojatatu.com, kernel-team@meta.com, Erik Lundgren , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Uladzislau Rezki , rcu@vger.kernel.org, Boqun Feng Subject: [RFC PATCH 8/8] locking/lockdep: Use shazptr to protect the key hashlist Date: Sun, 13 Apr 2025 23:00:55 -0700 Message-ID: <20250414060055.341516-9-boqun.feng@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250414060055.341516-1-boqun.feng@gmail.com> References: <20250414060055.341516-1-boqun.feng@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Erik Lundgren and Breno Leitao reported [1] a case where lockdep_unregister_key() can be called from time critical code pathes where rntl_lock() may be held. And the synchronize_rcu() in it can slow down operations such as using tc to replace a qdisc in a network device. In fact the synchronize_rcu() in lockdep_unregister_key() is to wait for all is_dynamic_key() callers to finish so that removing a key from the key hashlist, and we can use shazptr to protect the hashlist as well. Compared to the proposed solution which replaces synchronize_rcu() with synchronize_rcu_expedited(), using shazptr here can achieve the same/better synchronization time without the need to send IPI. Hence use shazptr here. Reported-by: Erik Lundgren Reported-by: Breno Leitao Link: https://lore.kernel.org/lkml/20250321-lockdep-v1-1-78b732d195fb@debia= n.org/ Signed-off-by: Boqun Feng Reviewed-by: Paul E. McKenney --- kernel/locking/lockdep.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 58d78a33ac65..c5781d2dc8c6 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -58,6 +58,7 @@ #include #include #include +#include =20 #include =20 @@ -1265,14 +1266,18 @@ static bool is_dynamic_key(const struct lock_class_= key *key) =20 hash_head =3D keyhashentry(key); =20 - rcu_read_lock(); + /* Need preemption disable for using shazptr. */ + guard(preempt)(); + + /* Protect the list search with shazptr. */ + guard(shazptr)(hash_head); + hlist_for_each_entry_rcu(k, hash_head, hash_entry) { if (k =3D=3D key) { found =3D true; break; } } - rcu_read_unlock(); =20 return found; } @@ -6614,7 +6619,7 @@ void lockdep_unregister_key(struct lock_class_key *ke= y) call_rcu(&delayed_free.rcu_head, free_zapped_rcu); =20 /* Wait until is_dynamic_key() has finished accessing k->hash_entry. */ - synchronize_rcu(); + synchronize_shazptr(keyhashentry(key)); } EXPORT_SYMBOL_GPL(lockdep_unregister_key); =20 --=20 2.47.1