From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53A703CB2C7; Fri, 5 Jun 2026 05:41:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638098; cv=none; b=NLsPlFGHIUDkLXgfgyBEGx/RY2rRqpOBPtRO9gdQr79HD2TSlpWUcRwC4PBsVH4gtQL47g8ophrpm9rspM99sJvKID0JEeO5mmFslrTrPhK4XqzFH9sVCVCuxyQcqMmiK1UACHXYMSdBaF1O2ZDkTEK0kXDfGsUzcf3ToiLx434= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638098; c=relaxed/simple; bh=hv8O/bheqVgn2baSbMciHGKRLF94QTuSTMuFCgVRSts=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nTsih/Ycw/Sw/R3VwD1n1ptdLJGXB7YhdbryFdQxLsYuDWdUQSuMevp5jbNUR4sTTkPnaG+oVii5SaaYNccwLziO1Om8ywgFSNiTLlkS1dlLU2n38LD18uRBYF7F8MyGyO2naATirZDU8uC2/X5qW0WJUNsvsm/QZbvvYeHlwz4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=F7Wo3U8m; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="F7Wo3U8m" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5799E1F0089A; Fri, 5 Jun 2026 05:41:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638097; bh=72tLoFmAzAI2R/LaKi/6YHpBql7AKiP21W7mrcDmmM8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=F7Wo3U8mbQbAb2VgYKv8NfIMTkg2sVlyyqwyeJMOcRnP+u8Ml3sOQRZH+KfC61G/4 WdQGxzAapDOEmCBgXNJd4bm6SN7ZDF49KF4Zx+nEKykIuIdQxS26WbwMZbfXI5acAb C6l7IW4SVnXslcPCXHipKS8EoOXqBogEgfteDRX79wsbYvv50Llt5p6CZi7wb+ZJX0 yK+yuQXqAuOGKGRSic5hIDMNb94UBIkfmI8PMthfSQaHjzbMHZoKPC/uQ7BrqPW15v gQxJsCgQxCwxNqxcpg5EsiEF0rckoStAWkxQWzg1HcBRbOu5EJ2rGNcjqC7JoplU0H 3PL3gN0vlff1w== Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfauth.phl.internal (Postfix) with ESMTP id 9F3FDF40077; Fri, 5 Jun 2026 01:41:34 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Fri, 05 Jun 2026 01:41:34 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGNdErcnsY9nz6zZJNqzBGcFpZ3oCsZyVSAsXPEfKD6fiAd7MM8RW3ErpHOKO8vqP y7FRxpXUBU5yb8Y73fmewycTgcYRdiPbEiXJpQaKxk1+aC6Vj3AAV2l/TRkhTd+1u+mIGl dQfEAWycnSLZO+qO217j/GTm+GQxoHWyRwvdph+wtnOrnU/D7PvfaSdl50PQ+1o9QW7RYj snzJFBhgLjSJ/jkZdib8Dc9ob5AHsrvEPqt2vef8Hmx2uDf0JqmoZZ+HexXrtwk5r0bry3 HVu0i63lg3nGwPLJ2r8Np94WJ6jG6+OWNN7TpxDaYMxO59AtKcBppdJ6IoJJH6Zm9jQmI9 U6emNrDYfEZR39DkwfoH5zt8f1YwHkfFANtNrvoAJ+dQH683V2Zc1PEA6Xfy/fajTfBKsD 92l7UsiWE1X5zsUDrj2NKzMx7UIkcvaiiHXGB2ozO0l8nG93rYi8LHQ1qbXrJSHLFkuTBU rY/mHdNJlpmXihaP5e0zxLrQ+nh490MaJ+wMxD1vJNso6UCG/elc68EmRimsFXaP8Hsm6g 3Xq0Cks0oNLsHDN148PLantNxbD9r12jBpgRmASx1ZYLAw4I52PMMNa0Qo/kewkN31ZVPe D3Wjzgpzgspdnw2A0l6YnOJrChVUBNktW4lQrBYrgqMqd1TLBPp/xuzpo0WA X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:33 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 01/13] preempt: Track NMI nesting to separate per-CPU counter Date: Thu, 4 Jun 2026 22:41:16 -0700 Message-ID: <20260605054128.5925-2-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Joel Fernandes Move NMI nesting tracking from the preempt_count bits to a separate per-CPU counter (nmi_nesting). This is to free up the NMI bits in the preempt_count, allowing those bits to be repurposed for other uses. Reduce NMI_BITS from 4 to 1, using it only to detect if we're in an NMI. The per-CPU counter currently caps nesting at 15. [boqun: Solve Steven Rostedt's comment on the BUG_ON() condition] [boqun: Use preempt_count_set() in __nmi_exit() to avoid underflow] Suggested-by: Boqun Feng Signed-off-by: Joel Fernandes Signed-off-by: Lyude Paul Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260121223933.1568682-3-lyude@redhat.com --- include/linux/hardirq.h | 17 +++++++++++++---- include/linux/preempt.h | 9 +++++++-- kernel/softirq.c | 2 ++ tools/testing/selftests/bpf/bpf_experimental.h | 2 +- 4 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index d57cab4d4c06..8d4895531a45 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -10,6 +10,8 @@ #include #include =20 +DECLARE_PER_CPU(unsigned int, nmi_nesting); + extern void synchronize_irq(unsigned int irq); extern bool synchronize_hardirq(unsigned int irq); =20 @@ -102,14 +104,17 @@ void irq_exit_rcu(void); */ =20 /* - * nmi_enter() can nest up to 15 times; see NMI_BITS. + * nmi_enter() can nest - nesting is tracked in a per-CPU counter. */ #define __nmi_enter() \ do { \ lockdep_off(); \ arch_nmi_enter(); \ - BUG_ON(in_nmi() =3D=3D NMI_MASK); \ - __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); \ + /* Maximum NMI nesting is 15. */ \ + BUG_ON(__this_cpu_read(nmi_nesting) >=3D 15); \ + __this_cpu_inc(nmi_nesting); \ + __preempt_count_add(HARDIRQ_OFFSET); \ + preempt_count_set(preempt_count() | NMI_MASK); \ } while (0) =20 #define nmi_enter() \ @@ -124,8 +129,12 @@ void irq_exit_rcu(void); =20 #define __nmi_exit() \ do { \ + unsigned int nesting; \ BUG_ON(!in_nmi()); \ - __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ + __preempt_count_sub(HARDIRQ_OFFSET); \ + nesting =3D __this_cpu_dec_return(nmi_nesting); \ + if (!nesting) \ + preempt_count_set(preempt_count() & ~NMI_MASK); \ arch_nmi_exit(); \ lockdep_on(); \ } while (0) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index d964f965c8ff..586f96688325 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -17,6 +17,8 @@ * * - bits 0-7 are the preemption count (max preemption depth: 256) * - bits 8-15 are the softirq count (max # of softirqs: 256) + * - bits 16-19 are the hardirq count (max # of hardirqs: 16) + * - bit 20 is the NMI flag (no nesting count, tracked separately) * * The hardirq count could in theory be the same as the number of * interrupts in the system, but we run all interrupt handlers with @@ -24,16 +26,19 @@ * there are a few palaeontologic drivers which reenable interrupts in * the handler, so we need more than one bit here. * + * NMI nesting depth is tracked in a separate per-CPU variable + * (nmi_nesting) to save bits in preempt_count. + * * PREEMPT_MASK: 0x000000ff * SOFTIRQ_MASK: 0x0000ff00 * HARDIRQ_MASK: 0x000f0000 - * NMI_MASK: 0x00f00000 + * NMI_MASK: 0x00100000 * PREEMPT_NEED_RESCHED: 0x80000000 */ #define PREEMPT_BITS 8 #define SOFTIRQ_BITS 8 #define HARDIRQ_BITS 4 -#define NMI_BITS 4 +#define NMI_BITS 1 =20 #define PREEMPT_SHIFT 0 #define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) diff --git a/kernel/softirq.c b/kernel/softirq.c index 4425d8dce44b..10af5ed859e7 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); #endif =20 +DEFINE_PER_CPU(unsigned int, nmi_nesting); + /* * SOFTIRQ_OFFSET usage: * diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing= /selftests/bpf/bpf_experimental.h index 2234bd6bc9d3..2d4256ff471f 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -449,7 +449,7 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgroup,= const char *name__str, #define PREEMPT_BITS 8 #define SOFTIRQ_BITS 8 #define HARDIRQ_BITS 4 -#define NMI_BITS 4 +#define NMI_BITS 1 =20 #define PREEMPT_SHIFT 0 #define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00CCE3FE645; Fri, 5 Jun 2026 05:41:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638102; cv=none; b=gf7vNME3EmpjhVF7xuKH2w7EUgMGSpDtXtxno1QroJvZd1QAOjBI/vYv9H0KMSNLPVOwKlu9zmhjTebkhVosbN7i6w3UdCHJXqIz/yh7mgQDvuGKCv+Fuj5KyUJa+1Io7mUlTOeGUpgFb8+2CYToDO/T4x2ffD6iEf2CZ3eZmWY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638102; c=relaxed/simple; bh=VsHssmG4iLDQg0XELCmXp+HbJFOZQrjGWZwsM0sxDDg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hIZo/NzhmK0oClwlPSGYIlVmzUjYdRgvRrDDUyUa1/ac+MRDTbdeq3VNQPdW5iPAthF71PwhS+bgopuzwRW6+z4y0uexnZZIhoZGhvBZmgss04gqpqEtfSnzh0R93+c8MJYxfOiC7WyQR+td/1gZXhqR8K0cBctNUriKc1Vq4vQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=N0AvAI0j; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="N0AvAI0j" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D14CA1F0089D; Fri, 5 Jun 2026 05:41:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638098; bh=/DGq2VclRBHsAUIaHXCPJUpfdhEM7VNlnnsEfM8ae3g=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=N0AvAI0je5HFbIxwVACgj35f4RdZer0MXZ/UIHzfG1jAyoAvLihR0O9/m+zYkJYyo zlpKXNDLb6LaXMR3apTHwQh7DTu/Qc3jW+NI29dqJ86Vx5Xx3g2T6nHhw5vYs9DRUh bAaoZ87s/ynM2A0QXo1YAWbQg+wjLlJhF7VXil5+bpQ9LK6qFhlcxgRKSuX8blATZC cOVbxddynzeCA9mZjV49ACv2cmzwC2DJLc+YbTWFurPby5/iWnCd73B+cUCORDAcUE /uiHka4A9jjgh7SmcKQn+QyGkW0WQ/QYX8GmGsDaA/E2JFT57xcokBw2QVnpSQSUq7 zT0/If1WTvy7Q== Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfauth.phl.internal (Postfix) with ESMTP id 27096F40078; Fri, 5 Jun 2026 01:41:36 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Fri, 05 Jun 2026 01:41:36 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFhOfDTFTb8fH37X+ASfp2vsHy1Fqb2Lywq6X3h5fMlxk7wW33nxe7aSTekseFO11 xBiB7Md3Dfg950zHftACRl4BCrZIGkOLORwuU667qK/t4b7uPTnvvrgSwCuAred6lniidQ x6+o43dstQnrbqqi/v6Nl3jrEdQj/OnTFIkpkf7j4tfd81Pm1F4hDhUzK5vMKXR0zokP4K KZGWpXQjfkPb9q0v8LW9bszX8CgnIWJLXLSKFiZe/Ul+6TINXLg1BnxiItQp/bB4zXvV+6 Qcp88FhWc5Q0N1TQTHx/VoF2pZ7nT3Uyapa4Doti8umM92UFgjuSWC48vH+lnu7P/MqXtO 1x7/NWjqOpmZJG2oWYOrLpGPqVyo+okymTgf44j8sFEdWMojKiseEGVjpZUelIIeOS9o8E HdWyH8Xufbs1TTRzWZRr1qJr/SNNelAtATCOTjox8ShD9mKRQ0u6c+b8rbSLel/LZID4n5 nKR7RaEb4K4FBscijWIxu+5c6cSlGa00JCIjG8oTYmaxKXfsUjn4MkrdC75vOYqwMvtM4/ RtBWb6Crse49u4woUwI7sMvScGA4raR5d6KZ1LjCgAWQE725NkNeNtG/3PBOa2bR4KmLAv ir2FrfQ+geGHbWvA1TRaFPHsUQ3H8ZhEYlm4yLkAWTs8rm1vumFN2owWXa9Q X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:35 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org, Boqun Feng Subject: [PATCH v3 02/13] preempt: Introduce HARDIRQ_DISABLE_BITS Date: Thu, 4 Jun 2026 22:41:17 -0700 Message-ID: <20260605054128.5925-3-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Boqun Feng In order to support preempt_disable()-like interrupt disabling, that is, using part of preempt_count() to track interrupt disabling nested level, change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE count. Signed-off-by: Lyude Paul Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260121223933.1568682-2-lyude@redhat.com --- include/linux/preempt.h | 16 +++++++++++----- tools/testing/selftests/bpf/bpf_experimental.h | 5 ++++- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 586f96688325..e2d3079d3f5f 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -17,8 +17,9 @@ * * - bits 0-7 are the preemption count (max preemption depth: 256) * - bits 8-15 are the softirq count (max # of softirqs: 256) - * - bits 16-19 are the hardirq count (max # of hardirqs: 16) - * - bit 20 is the NMI flag (no nesting count, tracked separately) + * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 2= 56) + * - bits 24-27 are the hardirq count (max # of hardirqs: 16) + * - bit 28 is the NMI flag (no nesting count, tracked separately) * * The hardirq count could in theory be the same as the number of * interrupts in the system, but we run all interrupt handlers with @@ -31,29 +32,34 @@ * * PREEMPT_MASK: 0x000000ff * SOFTIRQ_MASK: 0x0000ff00 - * HARDIRQ_MASK: 0x000f0000 - * NMI_MASK: 0x00100000 + * HARDIRQ_DISABLE_MASK: 0x00ff0000 + * HARDIRQ_MASK: 0x0f000000 + * NMI_MASK: 0x10000000 * PREEMPT_NEED_RESCHED: 0x80000000 */ #define PREEMPT_BITS 8 #define SOFTIRQ_BITS 8 +#define HARDIRQ_DISABLE_BITS 8 #define HARDIRQ_BITS 4 #define NMI_BITS 1 =20 #define PREEMPT_SHIFT 0 #define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) -#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_DISABLE_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_SHIFT (HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS) #define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS) =20 #define __IRQ_MASK(x) ((1UL << (x))-1) =20 #define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT) #define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT) +#define HARDIRQ_DISABLE_MASK (__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_= DISABLE_SHIFT) #define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT) #define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT) =20 #define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT) #define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT) +#define HARDIRQ_DISABLE_OFFSET (1UL << HARDIRQ_DISABLE_SHIFT) #define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT) #define NMI_OFFSET (1UL << NMI_SHIFT) =20 diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing= /selftests/bpf/bpf_experimental.h index 2d4256ff471f..a811b080db02 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -448,17 +448,20 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgrou= p, const char *name__str, =20 #define PREEMPT_BITS 8 #define SOFTIRQ_BITS 8 +#define HARDIRQ_DISABLE_BITS 8 #define HARDIRQ_BITS 4 #define NMI_BITS 1 =20 #define PREEMPT_SHIFT 0 #define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) -#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_DISABLE_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_SHIFT (HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS) #define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS) =20 #define __IRQ_MASK(x) ((1UL << (x))-1) =20 #define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT) +#define HARDIRQ_DISABLE_MASK (__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_= DISABLE_SHIFT) #define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT) #define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT) =20 --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B57F3BB10D; Fri, 5 Jun 2026 05:41:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638102; cv=none; b=lT8wm3SqeET6YkJ9tuXETIXsK40RFpAYG42D7psbEhu+I7OS52Z8wcAD/0532zoF/lxFl+sCrBm/k/vGakmi5T+FZFq13rLBaH1/kIX3qsmwjmr0aKEACSPHJMnZDFwh3FQeZTxuwk0oodf/bPJ+rACtASUc+gfVH4H2zNfFtlI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638102; c=relaxed/simple; bh=OYMeV7jWag8jjr/SGJIJdoM4RRdbcgCBusx85a29JNQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GFXv91MMNSubPxG3cbDW8+qPfZGjb929u16/81852QNCsseDPyvILWO35v5Fs2DIrAOZ3i0z5s2v8IPoom22Djw/H5TlRykUuWWf6AAyBE3DqZuvt+vzHxXQRoFKSfAPWyG5eHgbLKmHJqHNLbEJEoNY+CR/f9iFG/wPm0+19q0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aEVqsp4i; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aEVqsp4i" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5F1001F00898; Fri, 5 Jun 2026 05:41:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638099; bh=q22StVZMYHsonhMzLNuiRfEYi7dvpll9rGA0FOoOgUM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=aEVqsp4iYNJUhy8+//nV8kLacdG87jLd+Z9dGYCa0+TXk9qx8pSEvHMD8RGDS6pFi jgpvpl91arhoFUPeP5+BTLtC/YYlTB1AeOkMmDMr+AtfYzIKc4XpdP4plCI12NGT1u dxHJsVNmLhLHkqeOTCDnrvEbm0T1K+1ApWMxAmBY0AjVyqHYhtpVpof8EPvhmSWtM3 yrLWO675IkFl5rR/dS7WrYzlPFTwLcAy8sGsm9kZYvcFv3s1SSU1dp5IVQ7lTf7vl6 SqE+dl1QIhiVkM14c95c/0GOGGZW5Y9EvgMyrzTCUG7vpkZF0l17PIfnHt5yw9Mjqk KJGnDe8n2Ykig== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id AA5CAF40077; Fri, 5 Jun 2026 01:41:37 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Fri, 05 Jun 2026 01:41:37 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFhOfDTFTb8fH37X+ASfp2vsHy1Fqb2Lywq6X3h5fMlxk7wW33nxe7aSTekseFO11 xBiB7Md3Dfg950zHftACRl4BCrZIGkOLORwuU667qK/t4b7uPTnvvrgSwCuAred6lniidQ x6+o43dstQnrbqqi/v6Nl3jrEdQj/OnTFIkpkf7j4tfd81Pm1F4hDhUzK5vMKXR0zokP4K KZGWpXQjfkPb9q0v8LW9bszX8CgnIWJLXLSKFiZe/Ul+6TINXLg1BnxiItQp/bB4zXvV+6 Qcp88FhWc5Q0N1TQTHx/VoF2pZ7nT3Uyapa4Doti8umM92UFgjuSWC48vH+lnu7P/MqXQy 5fmb7JXidSEtZ94DTq5AOqHkZS7B2jcW9RZpYNSTVsMl9756g7TyoyUAFQFdU+5zUF86JT O36jgGKgvkCiI3g/Alv7jhIIamDTa/2iF9zV4B9PuQLxCfp9mSan8it3Ky4vGO0AzNa51w vgVxfBvfzy2ytIVHsdR/uT05Yd6f2+ctUGK6dq5LzLXUloZnu6mrSspEjLUK874UUfCORI FJccIc8RovUYWSz3K+Ftmeex7VWfnVHNcot6oL84ZpNZHdYPSiXSvNW+O2od2TihDsP6xV St0xihM/B25nP3OR4yRm05xPXFUOB1ge+YKPKimm6RhFij7O/o/GIeT+R93g X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:36 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org, Boqun Feng Subject: [PATCH v3 03/13] preempt: Introduce __preempt_count_{sub, add}_return() Date: Thu, 4 Jun 2026 22:41:18 -0700 Message-ID: <20260605054128.5925-4-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Boqun Feng In order to use preempt_count() to tracking the interrupt disable nesting level, __preempt_count_{add,sub}_return() are introduced, as their name suggest, these primitives return the new value of the preempt_count() after changing it. The following example shows the usage of it in local_interrupt_disable(): // increase the HARDIRQ_DISABLE bit new_count =3D __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET); // if it's the first-time increment, then disable the interrupt // at hardware level. if (new_count & HARDIRQ_DISABLE_MASK =3D=3D HARDIRQ_DISABLE_OFFSET) { local_irq_save(flags); raw_cpu_write(local_interrupt_disable_state.flags, flags); } Having these primitives will avoid a read of preempt_count() after changing preempt_count() on certain architectures. Acked-by: Heiko Carstens # s390 Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260121223933.1568682-4-lyude@redhat.com --- arch/arm64/include/asm/preempt.h | 18 ++++++++++++++++++ arch/s390/include/asm/preempt.h | 10 ++++++++++ arch/x86/include/asm/preempt.h | 10 ++++++++++ include/asm-generic/preempt.h | 14 ++++++++++++++ 4 files changed, 52 insertions(+) diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/pree= mpt.h index 932ea4b62042..0dd8221d1bef 100644 --- a/arch/arm64/include/asm/preempt.h +++ b/arch/arm64/include/asm/preempt.h @@ -55,6 +55,24 @@ static inline void __preempt_count_sub(int val) WRITE_ONCE(current_thread_info()->preempt.count, pc); } =20 +static inline int __preempt_count_add_return(int val) +{ + u32 pc =3D READ_ONCE(current_thread_info()->preempt.count); + pc +=3D val; + WRITE_ONCE(current_thread_info()->preempt.count, pc); + + return pc; +} + +static inline int __preempt_count_sub_return(int val) +{ + u32 pc =3D READ_ONCE(current_thread_info()->preempt.count); + pc -=3D val; + WRITE_ONCE(current_thread_info()->preempt.count, pc); + + return pc; +} + static inline bool __preempt_count_dec_and_test(void) { struct thread_info *ti =3D current_thread_info(); diff --git a/arch/s390/include/asm/preempt.h b/arch/s390/include/asm/preemp= t.h index 6e5821bb047e..0a25d4648b4c 100644 --- a/arch/s390/include/asm/preempt.h +++ b/arch/s390/include/asm/preempt.h @@ -139,6 +139,16 @@ static __always_inline bool should_resched(int preempt= _offset) return unlikely(READ_ONCE(get_lowcore()->preempt_count) =3D=3D preempt_of= fset); } =20 +static __always_inline int __preempt_count_add_return(int val) +{ + return val + __atomic_add(val, &get_lowcore()->preempt_count); +} + +static __always_inline int __preempt_count_sub_return(int val) +{ + return __preempt_count_add_return(-val); +} + #define init_task_preempt_count(p) do { } while (0) /* Deferred to CPU bringup time */ #define init_idle_preempt_count(p, cpu) do { } while (0) diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h index 578441db09f0..1220656f3370 100644 --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -85,6 +85,16 @@ static __always_inline void __preempt_count_sub(int val) raw_cpu_add_4(__preempt_count, -val); } =20 +static __always_inline int __preempt_count_add_return(int val) +{ + return raw_cpu_add_return_4(__preempt_count, val); +} + +static __always_inline int __preempt_count_sub_return(int val) +{ + return raw_cpu_add_return_4(__preempt_count, -val); +} + /* * Because we keep PREEMPT_NEED_RESCHED set when we do _not_ need to resch= edule * a decrement which hits zero means we have no preempt_count and should diff --git a/include/asm-generic/preempt.h b/include/asm-generic/preempt.h index 51f8f3881523..c8683c046615 100644 --- a/include/asm-generic/preempt.h +++ b/include/asm-generic/preempt.h @@ -59,6 +59,20 @@ static __always_inline void __preempt_count_sub(int val) *preempt_count_ptr() -=3D val; } =20 +static __always_inline int __preempt_count_add_return(int val) +{ + *preempt_count_ptr() +=3D val; + + return *preempt_count_ptr(); +} + +static __always_inline int __preempt_count_sub_return(int val) +{ + *preempt_count_ptr() -=3D val; + + return *preempt_count_ptr(); +} + static __always_inline bool __preempt_count_dec_and_test(void) { /* --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B2513FD971; Fri, 5 Jun 2026 05:41:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638103; cv=none; b=st75rvZDHssPQ/959K5BH7yzUqfA8CiVmU3cUBAffovveH+9q6PI7Ax+JJ1EDYlWC9gbCBhaRHsszBcHGyD9Go+xOc9yy8hRyzcrDVIXMzUwUnNP/3RtG49/jqss7g37siP5RsHkHce0aAXue79wpKLxKTSkSeH6XsgtHNAe6w4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638103; c=relaxed/simple; bh=MbtvuBNH3GysMzOJRSBr0Nwx1iy6lfFNP2fljYwUfY0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tRYSKqH2N/KDSLd1ZZgjjbQsCXQWXgM7zPiHfhA99vRDci7jKGVU/PzXQ3r7xg6AIJ0Fi2xwGuRKQm0019Y3Wgsfzv8+VaffcLWlEgw9xwq5KL6YPk3IAcLn61NRtMucl5CfC0gebgLW+4tDyyACCdIF4Uh//u3PX3mBjJLa2Ak= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WJLeiJLp; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WJLeiJLp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2F4D1F008A2; Fri, 5 Jun 2026 05:41:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638101; bh=3bY5pS2rVgQa9nC8tbPnRn1j/SHdGkv637KqxwEbuVU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=WJLeiJLpGTULrGaD3KJ5ZO/qhAD/WTXhS88AlVTFLOs0aEjG5xtuCJqqos8RhJbat mhChDxDbwYUJxeI0ODSa/E0r5b1XWCOKqOdT25UJhv6TGQix4wMpvjFST/D5kTuJFk 27FwOTqTMEKtwJN2N6W9wm5pWcOYnYuG7OU6u/Y/VwZaGJ047neU3hDZOzt+uJ3rsl G9u+LiEKJRa6WeYT4eVdlb5WPAX6/uTHy7tzVCKVlsdS0xFwBLqATE+4bnJVyYww1Z 3y0kwJU7MisSREttB7QzSQ6+O2nrwCPqi4p6ikyNj17aW5wdwTKN61KaHzmEPIHFLT vKcGr7z7vAMsg== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id 26FE3F40075; Fri, 5 Jun 2026 01:41:39 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Fri, 05 Jun 2026 01:41:39 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFCwYGExB8+n21rrRlgZz2BZEPfw3mIE0bRNe76RMQAdH8f2KDaPLT52qZo7gxyWp 0wXQACdPWdspv4jN5UYFCpihH+vrJtJ5nM33JJDnQyPnqbA42Pv6gENFpmiA7vDUbZDCAC N+ZmE7MklGAiGBNJA045y18Z9khhPwbxKZPuahNIwIz69+nOH0m/THs5eb4pu9rOFYQQI9 b1h3voWmc5QUasjwQpRDRo+W3jNrB4TeCrmSTmM73SZRB0LYstjQaA4JitE82TBWNXKOy5 NeQugQn5+UA17tSJ+MYIgsz9Sw+H8LaFdylvI7Cq2OhiaEqC4YLyUJJOJm9vOjE+2O1Njz jp4sreZDsZ269nsz2mo/kcC7coE3T6hddbvivsAbKivqCFPV3/0vs6kZgVMc2e9TJrhRt4 fHYKqMx8M1PZxSYs7AntkRGfCKC7aBsXRQ6kbOk1KsC10Ei1hZZuLRE8BTnXhSkdEHM54H lhpM6NGWlljg7EPOyVAxg2wG6/DdZVDRtaWTY3aH6bYu8JoW+/jY1i4UdVw5IgTsxxQHR1 ltgW9dvVp0pCbLg/ftDSu7Dc+bAfuiwlzk6SJccnFUDu09D5+4zqk8qhwu9T3M94LlcZXi jn3q9K0ZJyPphv2GjNN8d5hWgag33nJE1mZKYhWRYKxAOyy1OdD4jaybX7SA X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:38 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 04/13] openrisc: Include in smp.h Date: Thu, 4 Jun 2026 22:41:19 -0700 Message-ID: <20260605054128.5925-5-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lyude Paul While OpenRISC currently doesn't fail to build upstream, it appears that include in the right headers is enough to break that - primarily because OpenRISC's asm/smp.h header doesn't actually provide any definition for struct cpumask. Which means the only reason we aren't failing to build kernel is because we've been lucky enough that every spot including asm/smp.h already has definitions for struct cpumask pulled in. This became evident when trying to work on a patch series for adding ref-counted interrupt enable/disables to the kernel, where introducing a new interrupt_rc.h header suddenly introduced a build error on OpenRISC: In file included from include/linux/interrupt_rc.h:17, from include/linux/spinlock.h:60, from include/linux/mmzone.h:8, from include/linux/gfp.h:7, from include/linux/mm.h:7, from arch/openrisc/include/asm/pgalloc.h:20, from arch/openrisc/include/asm/io.h:18, from include/linux/io.h:12, from drivers/irqchip/irq-ompic.c:61: arch/openrisc/include/asm/smp.h:21:59: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration 21 | extern void arch_send_call_function_ipi_mask(const struct cpum= ask *mask); | ^~~~= ~~~ arch/openrisc/include/asm/smp.h:23:54: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration 23 | extern void set_smp_cross_call(void (*)(const struct cpumask *= , unsigned int)); | ^~~~~~~ drivers/irqchip/irq-ompic.c: In function 'ompic_of_init': >> drivers/irqchip/irq-ompic.c:191:28: error: passing argument 1 of 'set_smp_cross_call' from incompatible pointer type [-Werror=3Dincompatible-pointer-types] 191 | set_smp_cross_call(ompic_raise_softirq); | ^~~~~~~~~~~~~~~~~~~ | | | void (*)(const struct cpumask *, un= signed int) arch/openrisc/include/asm/smp.h:23:32: note: expected 'void (*)(const struct cpumask *, unsigned int)' but argument is of type 'void (*)(const struct cpumask *, unsigned int)' 23 | extern void set_smp_cross_call(void (*)(const struct cpumask *= , unsigned int)); To fix this, let's take an example from the smp.h headers of other architectures (x86, hexagon, arm64, probably more): just include linux/cpumask.h at the top. Signed-off-by: Lyude Paul Acked-by: Stafford Horne Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260121223933.1568682-5-lyude@redhat.com --- arch/openrisc/include/asm/smp.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/openrisc/include/asm/smp.h b/arch/openrisc/include/asm/sm= p.h index 007296f160ef..84653aaffa96 100644 --- a/arch/openrisc/include/asm/smp.h +++ b/arch/openrisc/include/asm/smp.h @@ -9,6 +9,8 @@ #ifndef __ASM_OPENRISC_SMP_H #define __ASM_OPENRISC_SMP_H =20 +#include + #include #include =20 --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4CAA3FE355; Fri, 5 Jun 2026 05:41:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638106; cv=none; b=WhIYuyppD/sBw32wUrc3H/kix3wqOkNdRSEIRNBw950FydVaDhSO7lDzfu4la6DmYottnT1SKFW40WhS+Bar4W6+VuAxy1eeoFSDhTn6cAjcA+/nGzmeGRy8zVFAN37gmsoF7X7eTUBAdGXPYIPkMgEpJtEY3x7e5eSEsYTMMlQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638106; c=relaxed/simple; bh=mOT67SRge2DYGQ3V8PH8u80wEcnMPeiwjCuNb2Q9Cg4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gr8DR3zZzGL7S7VHbvrKy2+DhBEc26otUFtUiEIF29Phk6b/5URY5L2faSdkQchJR0OtR9BibkzndAnEc3qf3qVqb4YJVmawLiOCDwn2xVangHQTkQpLb2oNl5ms/fzylfCP6yVkYR0d+SdFhMQ0iAbRxEjq7dXMjs1NkZ76yMg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=J7LedXB+; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="J7LedXB+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8DEE61F0089A; Fri, 5 Jun 2026 05:41:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638103; bh=i20Jbk8jR938mnl3MQP4hTOg893J4gee3kCeY2sRyw8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=J7LedXB+DsB1SHAd9h1Rz2Tf7+pN4DMTHYyTgsN9wKktZlSGcuEo6Dlc/pfnmkf5+ rDiKilYo62oSkQaLOi3yrXvq+IRH/fVp4BBrvxvOiZ3J197z//CHL0rOZNRzW1LDAb IlCJJN0pQoPLBuP69NH+8zF8OssT0hmtp8BV+gDyyszLTeAT0thmwCsQqz+KOjmlVV +bk1Hrxkc0CrdZUCdUFWtgQpOSTSh2MAbMaJVqPh9VhwroRCtRQruVZJW8bNLQL+YM /aa0kUqdGf1+c3QZS7lmsU7qYVpPztbuPbGtZbcZjilb6Ky5/U7GiaHkI932MD+vQG WGopPEc1u3Fng== Received: from phl-compute-12.internal (phl-compute-12.internal [10.202.2.52]) by mailfauth.phl.internal (Postfix) with ESMTP id C9A23F40078; Fri, 5 Jun 2026 01:41:40 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-12.internal (MEProxy); Fri, 05 Jun 2026 01:41:40 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTF+6OUUl/WCeAyQA+P/KqXroNkAuAtPUHNl1vALYE5KKUMlpAJi8c1hSqvkDOBmSu og8IoH4iECmUOPDYoHvgTGwLFu6kpNwoB75p+MLMl8MT6vaA1WsL6CU8agmpbETpm4E1cB bLMkCxTyHwEdm3oOfwtuXZvOzHVWP6/DVgkNaX8k7bGDiWnLoWwQX5656s96warllOs5ok lO4tbNuKg5fkFkZ+l9SEBg3RmtnD7tA2FRJKy8heCeUo0hK2TErE8oXhHlnNx8f5tN9bsv 2tJXSev/YKJTwprsCkjkEBicRFFbL5UvjUrjGWcaHl1TCldB0K/McQaby8W6BdJptNtsaV wfprYAqf+JgzoRy0wHvrjhVnSofg6dd4F0Ds4CZAVY6wicbeBBBNsK0eUtiCmQLfDKO1R/ HDXTJZWaNphTvPAsZoB3+1Ddr0EVF1XpM+ssGIYkczRiiOCykV44EjneZ78/8BMthNsrKh CDEJtpN00wm8DzJPv0IPYwHbOwY/IG8RvMtHedDQiIwupkBfthaVXuffFxUQ2cECuhETGN WDHnvtu8McePxyjUK/16rLRf85Os5+K7wMp/m9GDT5+FOGfMi0nOpXNtq4QiPqDbkjz3Oi obQc3WAwd+94QbFLT/XUt94+gh679yIgNov5A+X5St3OTIyYmJ2p+EJjclBQ X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:39 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org, Boqun Feng Subject: [PATCH v3 05/13] irq & spin_lock: Add counted interrupt disabling/enabling Date: Thu, 4 Jun 2026 22:41:20 -0700 Message-ID: <20260605054128.5925-6-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Boqun Feng Currently the nested interrupt disabling and enabling is present by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example: spin_lock_irqsave(l1, flag1); spin_lock_irqsave(l2, flag2); spin_unlock_irqrestore(l1, flags1); // accesses to interrupt-disable protect data will cause races. This is even easier to triggered with guard facilities: unsigned long flag2; scoped_guard(spin_lock_irqsave, l1) { spin_lock_irqsave(l2, flag2); } // l2 locked but interrupts are enabled. spin_unlock_irqrestore(l2, flag2); (Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design) And because this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design. To resolve this, introduce a new set of interrupt disabling APIs: * local_interrupt_disable(); * local_interrupt_enable(); They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call save the interrupt state into a percpu variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a percpu counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section. Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follow: spin_lock_irq_disable(l1); spin_lock_irq_disable(l2); spin_unlock_irq_enable(l1); // Interrupts are still disabled. spin_unlock_irq_enable(l2); doesn't have the issue that interrupts are accidentally enabled. This also makes the wrapper of interrupt-disabling locks on Rust easier to design. Signed-off-by: Lyude Paul Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260121223933.1568682-6-lyude@redhat.com --- include/linux/interrupt_rc.h | 67 ++++++++++++++++++++++++++++++++ include/linux/preempt.h | 4 ++ include/linux/spinlock.h | 23 +++++++++++ include/linux/spinlock_api_smp.h | 41 +++++++++++++++++++ include/linux/spinlock_api_up.h | 16 ++++++++ include/linux/spinlock_rt.h | 18 +++++++++ kernel/locking/spinlock.c | 29 ++++++++++++++ kernel/softirq.c | 14 ++++++- 8 files changed, 211 insertions(+), 1 deletion(-) create mode 100644 include/linux/interrupt_rc.h diff --git a/include/linux/interrupt_rc.h b/include/linux/interrupt_rc.h new file mode 100644 index 000000000000..dd4444c61330 --- /dev/null +++ b/include/linux/interrupt_rc.h @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * include/linux/interrupt_rc.h - refcounted local processor interrupt + * management. + * + * Since the implementation of this API currently depends on + * local_irq_save()/local_irq_restore(), we split this into it's own heade= r to + * make it easier to include without hitting circular header dependencies. + */ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H + +#include +#include +#ifdef CONFIG_SMP +#include +#endif + +/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}(= ) */ +struct interrupt_disable_state { + unsigned long flags; +}; + +DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_st= ate); + +static inline void local_interrupt_disable(void) +{ + unsigned long flags; + int new_count; + + WARN_ON_ONCE(in_nmi()); + + new_count =3D hardirq_disable_enter(); + + /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + + if ((new_count & HARDIRQ_DISABLE_MASK) =3D=3D HARDIRQ_DISABLE_OFFSET) { + local_irq_save(flags); + raw_cpu_write(local_interrupt_disable_state.flags, flags); + } +} + +static inline void local_interrupt_enable(void) +{ + int new_count; + + new_count =3D hardirq_disable_exit(); + + if ((new_count & HARDIRQ_DISABLE_MASK) =3D=3D 0) { + unsigned long flags; + + flags =3D raw_cpu_read(local_interrupt_disable_state.flags); + local_irq_restore(flags); + /* + * TODO: re-read preempt count can be avoided, but it needs + * should_resched() taking another parameter as the current + * preempt count + */ +#ifdef CONFIG_PREEMPTION + if (should_resched(0)) + __preempt_schedule(); +#endif + } +} + +#endif /* !__LINUX_INTERRUPT_RC_H */ diff --git a/include/linux/preempt.h b/include/linux/preempt.h index e2d3079d3f5f..33fc4c814a9f 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context= _level(void) #define in_softirq() (softirq_count()) #define in_interrupt() (irq_count()) =20 +#define hardirq_disable_count() ((preempt_count() & HARDIRQ_DISABLE_MASK) = >> HARDIRQ_DISABLE_SHIFT) +#define hardirq_disable_enter() __preempt_count_add_return(HARDIRQ_DISABLE= _OFFSET) +#define hardirq_disable_exit() __preempt_count_sub_return(HARDIRQ_DISABLE_= OFFSET) + /* * The preempt_count offset after preempt_disable(); */ diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index 241277cd34cf..3d405cc4c121 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -57,6 +57,7 @@ #include #include #include +#include #include #include #include @@ -273,9 +274,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *= lock) __releases(lock) #endif =20 #define raw_spin_lock_irq(lock) _raw_spin_lock_irq(lock) +#define raw_spin_lock_irq_disable(lock) _raw_spin_lock_irq_disable(lock) #define raw_spin_lock_bh(lock) _raw_spin_lock_bh(lock) #define raw_spin_unlock(lock) _raw_spin_unlock(lock) #define raw_spin_unlock_irq(lock) _raw_spin_unlock_irq(lock) +#define raw_spin_unlock_irq_enable(lock) _raw_spin_unlock_irq_enable(lock) =20 #define raw_spin_unlock_irqrestore(lock, flags) \ do { \ @@ -290,6 +293,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *l= ock) __releases(lock) =20 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lo= ck, &(flags)) =20 +#define raw_spin_trylock_irq_disable(lock) _raw_spin_trylock_irq_disable(l= ock) + #ifndef CONFIG_PREEMPT_RT /* Include rwlock functions for !RT */ #include @@ -372,6 +377,12 @@ static __always_inline void spin_lock_irq(spinlock_t *= lock) raw_spin_lock_irq(&lock->rlock); } =20 +static __always_inline void spin_lock_irq_disable(spinlock_t *lock) + __acquires(lock) __no_context_analysis +{ + raw_spin_lock_irq_disable(&lock->rlock); +} + #define spin_lock_irqsave(lock, flags) \ do { \ raw_spin_lock_irqsave(spinlock_check(lock), flags); \ @@ -402,6 +413,12 @@ static __always_inline void spin_unlock_irq(spinlock_t= *lock) raw_spin_unlock_irq(&lock->rlock); } =20 +static __always_inline void spin_unlock_irq_enable(spinlock_t *lock) + __releases(lock) __no_context_analysis +{ + raw_spin_unlock_irq_enable(&lock->rlock); +} + static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsig= ned long flags) __releases(lock) __no_context_analysis { @@ -427,6 +444,12 @@ static __always_inline bool _spin_trylock_irqsave(spin= lock_t *lock, unsigned lon } #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(fl= ags)) =20 +static __always_inline int spin_trylock_irq_disable(spinlock_t *lock) + __cond_acquires(true, lock) __no_context_analysis +{ + return raw_spin_trylock_irq_disable(&lock->rlock); +} + /** * spin_is_locked() - Check whether a spinlock is locked. * @lock: Pointer to the spinlock. diff --git a/include/linux/spinlock_api_smp.h b/include/linux/spinlock_api_= smp.h index bda5e7a390cd..07a94ba1d760 100644 --- a/include/linux/spinlock_api_smp.h +++ b/include/linux/spinlock_api_smp.h @@ -28,6 +28,8 @@ _raw_spin_lock_nest_lock(raw_spinlock_t *lock, struct loc= kdep_map *map) void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock) __acquires(lock); void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock) __acquires(lock); +void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock) + __acquires(lock); =20 unsigned long __lockfunc _raw_spin_lock_irqsave(raw_spinlock_t *lock) __acquires(lock); @@ -39,6 +41,7 @@ int __lockfunc _raw_spin_trylock_bh(raw_spinlock_t *lock)= __cond_acquires(true, void __lockfunc _raw_spin_unlock(raw_spinlock_t *lock) __releases(lock); void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock) __releases(lock); void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock) __releases(lock= ); +void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock) __releas= es(lock); void __lockfunc _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags) __releases(lock); @@ -55,6 +58,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsign= ed long flags) #define _raw_spin_lock_irq(lock) __raw_spin_lock_irq(lock) #endif =20 +/* Use the same config as spin_lock_irq() temporarily. */ +#ifdef CONFIG_INLINE_SPIN_LOCK_IRQ +#define _raw_spin_lock_irq_disable(lock) __raw_spin_lock_irq_disable(lock) +#endif + #ifdef CONFIG_INLINE_SPIN_LOCK_IRQSAVE #define _raw_spin_lock_irqsave(lock) __raw_spin_lock_irqsave(lock) #endif @@ -79,6 +87,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsign= ed long flags) #define _raw_spin_unlock_irq(lock) __raw_spin_unlock_irq(lock) #endif =20 +/* Use the same config as spin_unlock_irq() temporarily. */ +#ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQ +#define _raw_spin_unlock_irq_enable(lock) __raw_spin_unlock_irq_enable(loc= k) +#endif + #ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE #define _raw_spin_unlock_irqrestore(lock, flags) __raw_spin_unlock_irqrest= ore(lock, flags) #endif @@ -105,6 +118,18 @@ static __always_inline bool _raw_spin_trylock_irq(raw_= spinlock_t *lock) return false; } =20 +static __always_inline bool _raw_spin_trylock_irq_disable(raw_spinlock_t *= lock) + __cond_acquires(true, lock) +{ + local_interrupt_disable(); + if (do_raw_spin_trylock(lock)) { + spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); + return true; + } + local_interrupt_enable(); + return false; +} + static __always_inline bool _raw_spin_trylock_irqsave(raw_spinlock_t *lock= , unsigned long *flags) __cond_acquires(true, lock) { @@ -143,6 +168,14 @@ static inline void __raw_spin_lock_irq(raw_spinlock_t = *lock) LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock); } =20 +static inline void __raw_spin_lock_irq_disable(raw_spinlock_t *lock) + __acquires(lock) __no_context_analysis +{ + local_interrupt_disable(); + spin_acquire(&lock->dep_map, 0, 0, _RET_IP_); + LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock); +} + static inline void __raw_spin_lock_bh(raw_spinlock_t *lock) __acquires(lock) __no_context_analysis { @@ -188,6 +221,14 @@ static inline void __raw_spin_unlock_irq(raw_spinlock_= t *lock) preempt_enable(); } =20 +static inline void __raw_spin_unlock_irq_enable(raw_spinlock_t *lock) + __releases(lock) +{ + spin_release(&lock->dep_map, _RET_IP_); + do_raw_spin_unlock(lock); + local_interrupt_enable(); +} + static inline void __raw_spin_unlock_bh(raw_spinlock_t *lock) __releases(lock) { diff --git a/include/linux/spinlock_api_up.h b/include/linux/spinlock_api_u= p.h index a9d5c7c66e03..e4de8bb26a15 100644 --- a/include/linux/spinlock_api_up.h +++ b/include/linux/spinlock_api_up.h @@ -42,6 +42,9 @@ #define __LOCK_IRQSAVE(lock, flags, ...) \ do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0) =20 +#define __LOCK_IRQ_DISABLE(lock, ...) \ + do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0) + #define ___UNLOCK_(lock) \ do { __release(lock); (void)(lock); } while (0) =20 @@ -61,6 +64,10 @@ #define __UNLOCK_IRQRESTORE(lock, flags, ...) \ do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0) =20 +#define __UNLOCK_IRQ_ENABLE(lock, ...) \ + do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0) + + #define _raw_spin_lock(lock) __LOCK(lock) #define _raw_spin_lock_nested(lock, subclass) __LOCK(lock) #define _raw_read_lock(lock) __LOCK(lock, shared) @@ -70,6 +77,7 @@ #define _raw_read_lock_bh(lock) __LOCK_BH(lock, shared) #define _raw_write_lock_bh(lock) __LOCK_BH(lock) #define _raw_spin_lock_irq(lock) __LOCK_IRQ(lock) +#define _raw_spin_lock_irq_disable(lock) __LOCK_IRQ_DISABLE(lock) #define _raw_read_lock_irq(lock) __LOCK_IRQ(lock, shared) #define _raw_write_lock_irq(lock) __LOCK_IRQ(lock) #define _raw_spin_lock_irqsave(lock, flags) __LOCK_IRQSAVE(lock, flags) @@ -97,6 +105,13 @@ static __always_inline int _raw_spin_trylock_irq(raw_sp= inlock_t *lock) return 1; } =20 +static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *l= ock) + __cond_acquires(true, lock) +{ + __LOCK_IRQ_DISABLE(lock); + return 1; +} + static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock,= unsigned long *flags) __cond_acquires(true, lock) { @@ -132,6 +147,7 @@ static __always_inline int _raw_write_trylock_irqsave(r= wlock_t *lock, unsigned l #define _raw_write_unlock_bh(lock) __UNLOCK_BH(lock) #define _raw_read_unlock_bh(lock) __UNLOCK_BH(lock, shared) #define _raw_spin_unlock_irq(lock) __UNLOCK_IRQ(lock) +#define _raw_spin_unlock_irq_enable(lock) __UNLOCK_IRQ_ENABLE(lock) #define _raw_read_unlock_irq(lock) __UNLOCK_IRQ(lock, shared) #define _raw_write_unlock_irq(lock) __UNLOCK_IRQ(lock) #define _raw_spin_unlock_irqrestore(lock, flags) \ diff --git a/include/linux/spinlock_rt.h b/include/linux/spinlock_rt.h index 373618a4243c..560d06384e0c 100644 --- a/include/linux/spinlock_rt.h +++ b/include/linux/spinlock_rt.h @@ -96,6 +96,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lo= ck) rt_spin_lock(lock); } =20 +static __always_inline void spin_lock_irq_disable(spinlock_t *lock) + __acquires(lock) +{ + rt_spin_lock(lock); +} + #define spin_lock_irqsave(lock, flags) \ do { \ typecheck(unsigned long, flags); \ @@ -122,6 +128,12 @@ static __always_inline void spin_unlock_irq(spinlock_t= *lock) rt_spin_unlock(lock); } =20 +static __always_inline void spin_unlock_irq_enable(spinlock_t *lock) + __releases(lock) +{ + rt_spin_unlock(lock); +} + static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) __releases(lock) @@ -131,6 +143,12 @@ static __always_inline void spin_unlock_irqrestore(spi= nlock_t *lock, =20 #define spin_trylock(lock) rt_spin_trylock(lock) =20 +static __always_inline int spin_trylock_irq_disable(spinlock_t *lock) + __cond_acquires(true, lock) +{ + return rt_spin_trylock(lock); +} + #define spin_trylock_bh(lock) rt_spin_trylock_bh(lock) =20 #define spin_trylock_irq(lock) rt_spin_trylock(lock) diff --git a/kernel/locking/spinlock.c b/kernel/locking/spinlock.c index b42d293da38b..764641f6ec57 100644 --- a/kernel/locking/spinlock.c +++ b/kernel/locking/spinlock.c @@ -129,6 +129,19 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##= _t *lock) \ */ BUILD_LOCK_OPS(spin, raw_spinlock, __acquires); =20 +/* No rwlock_t variants for now, so just build this function by hand */ +static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock) +{ + for (;;) { + local_interrupt_disable(); + if (likely(do_raw_spin_trylock(lock))) + break; + local_interrupt_enable(); + + arch_spin_relax(&lock->raw_lock); + } +} + #ifndef CONFIG_PREEMPT_RT BUILD_LOCK_OPS(read, rwlock, __acquires_shared); BUILD_LOCK_OPS(write, rwlock, __acquires); @@ -176,6 +189,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlo= ck_t *lock) EXPORT_SYMBOL(_raw_spin_lock_irq); #endif =20 +#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ +noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock) +{ + __raw_spin_lock_irq_disable(lock); +} +EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable); +#endif + #ifndef CONFIG_INLINE_SPIN_LOCK_BH noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock) { @@ -208,6 +229,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spin= lock_t *lock) EXPORT_SYMBOL(_raw_spin_unlock_irq); #endif =20 +#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ +noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock) +{ + __raw_spin_unlock_irq_enable(lock); +} +EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable); +#endif + #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock) { diff --git a/kernel/softirq.c b/kernel/softirq.c index 10af5ed859e7..d1ab1799794c 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -88,6 +88,9 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); #endif =20 +DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_sta= te); +EXPORT_PER_CPU_SYMBOL_GPL(local_interrupt_disable_state); + DEFINE_PER_CPU(unsigned int, nmi_nesting); =20 /* @@ -728,7 +731,16 @@ static inline void __irq_exit_rcu(void) #endif account_hardirq_exit(current); preempt_count_sub(HARDIRQ_OFFSET); - if (!in_interrupt() && local_softirq_pending()) { + /* + * Interrupts may happen between hardirq_disable_enter() and + * local_irq_save() in local_interrupt_disable(), if irq_exit() invokes + * softirq here, we may have a softirq handler calling + * local_interrupt_disable() but it won't disable the irq because + * hardirq disabling count is already 1, hence we need to prevent + * invoking softirq when a local_interrupt_disable() is ongoing. + */ + if (!in_interrupt() && !hardirq_disable_count() && + local_softirq_pending()) { /* * If we left hrtimers unarmed, make sure to arm them now, * before enabling interrupts to run SoftIRQ. --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1551401496; Fri, 5 Jun 2026 05:41:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638107; cv=none; b=sV2YhSDi5Y60SVCIKKa6JCdVqtObMyIBhsyLtOpJniUMgWwKictClG2Xs+Wepvy3/RTPSmOB8daxRCuGH3twa3VTT9deGezuCmbKg4GV1hyKpMA/qJ+gDNoPlbsKfG1e4C/YnGPrbWGfk7AUjDswbecI2hF9k53JHEY1/CuGjk8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638107; c=relaxed/simple; bh=VxbsBjHdRvXTR0KV6QwpQV6PvkcezsCcgEhIoL++T/Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DK5hycB9VymRJ/h2XNydWGIxgsqmAfQLyecJDy1p7MrTlupf8QYthT92VTTMDjR82/0lOYdMzkILpKt9gLY8isP8WYmeUetkXc2w6F6D4oto/O1yoYvPEpZzJ7wREZeP86U/rt661OfW9fNTzJpTXXJ8ygHni2cbBul6LIKVU+0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ECZ+E5n4; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ECZ+E5n4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4DD0E1F008A2; Fri, 5 Jun 2026 05:41:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638105; bh=rQENKqT9pyTIBLJdn4Wt5SoRtkTc7Vr1VCzymuIIqXI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ECZ+E5n4zdozdVYTiJeQE4/UsZ0YhS+89IgOEI3BqR6ERvz39HALWNz2j7ovf5QqO ndYbYl37JF6tzBVviSUgVQwc+4Kx/SVOtEj3sKlaYfYrJVCXK0VwCMCjbQhJfoVu3G HAMj1ASM9i1ZEkqnc2W5P8ogFI9yufaD5SvRHo/w5IT7gzTrCYA/4n2oI6QB12QHXr nMQUe6z+4cQGiNPCddSvWP7gXERADmm5gQ3JVc2l3DE7iRdsNZlvJ4UXKgLNgpKYp1 NqFv7Yj/STFmHfsr/dAQ9bCfYoIG6nZ4gFqokFhSeF+0AUj2gb3AxkT3nM8+HbkTZG UETldyG/L74RQ== Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfauth.phl.internal (Postfix) with ESMTP id 641FEF40072; Fri, 5 Jun 2026 01:41:42 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Fri, 05 Jun 2026 01:41:42 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFCwYGExB8+n21rrRlgZz2BZEPfw3mIE0bRNe76RMQAdH8f2KDaPLT52qZo7gxyWp 0wXQACdPWdspv4jN5UYFCpihH+vrJtJ5nM33JJDnQyPnqbA42Pv6gENFpmiA7vDUbZDCAC N+ZmE7MklGAiGBNJA045y18Z9khhPwbxKZPuahNIwIz69+nOH0m/THs5eb4pu9rOFYQQI9 b1h3voWmc5QUasjwQpRDRo+W3jNrB4TeCrmSTmM73SZRB0LYstjQaA4JitE82TBWNXKOy5 NeQugQn5+UA17tSJ+MYIgsz9Sw+H8LaFdylvI7Cq2OhiaEqC4YLyUJJOJm9vOjE+2O1NIb ifGBvOfA1bRvdD1rzC6dalPswwrstFlyf7Pap/tukclD4RSnOep05M0BoiV6vs7Bgu2EkN p72ke8BUte3XkipOs/+CKWn/TsXg6p4p138xCLDsao5vi8Ksw7pMG7tzPlOYBuOqdg6z8c MidLD+dCzBqmfkLMsHdOHLfs1Pt53AU/mth+xQxJ6Wnq5W1rjhMKkcb5wW6tQdd0P7matz OCENDIVnfvIMJ8TC4oMgg+jtzeGeX2yW4wFkT7CuUw7nTNzjxH9wZghvZaZUbOwF7xfC59 Af1ZSoVdz9AYOcDTSgfAw9IS9LKJ+IE4V38tIidtjIBck5rDsCz/vk+vM9iQ X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:41 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 06/13] irq: Add KUnit test for refcounted interrupt enable/disable Date: Thu, 4 Jun 2026 22:41:21 -0700 Message-ID: <20260605054128.5925-7-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lyude Paul While making changes to the refcounted interrupt patch series, at some point on my local branch I broke something and ended up writing some kunit tests for testing refcounted interrupts as a result. So, let's include these tests now that we have refcounted interrupts. Signed-off-by: Lyude Paul Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260121223933.1568682-7-lyude@redhat.com --- kernel/irq/Makefile | 1 + kernel/irq/refcount_interrupt_test.c | 109 +++++++++++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100644 kernel/irq/refcount_interrupt_test.c diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile index 86a2e5ae08f9..44c4d6fc502a 100644 --- a/kernel/irq/Makefile +++ b/kernel/irq/Makefile @@ -16,3 +16,4 @@ obj-$(CONFIG_SMP) +=3D affinity.o obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) +=3D debugfs.o obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) +=3D matrix.o obj-$(CONFIG_IRQ_KUNIT_TEST) +=3D irq_test.o +obj-$(CONFIG_KUNIT) +=3D refcount_interrupt_test.o diff --git a/kernel/irq/refcount_interrupt_test.c b/kernel/irq/refcount_int= errupt_test.c new file mode 100644 index 000000000000..b4f224595f26 --- /dev/null +++ b/kernel/irq/refcount_interrupt_test.c @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KUnit test for refcounted interrupt enable/disables. + */ + +#include +#include + +#define TEST_IRQ_ON() KUNIT_EXPECT_FALSE(test, irqs_disabled()) +#define TEST_IRQ_OFF() KUNIT_EXPECT_TRUE(test, irqs_disabled()) + +/* =3D=3D=3D=3D=3D Test cases =3D=3D=3D=3D=3D */ +static void test_single_irq_change(struct kunit *test) +{ + local_interrupt_disable(); + TEST_IRQ_OFF(); + local_interrupt_enable(); +} + +static void test_nested_irq_change(struct kunit *test) +{ + local_interrupt_disable(); + TEST_IRQ_OFF(); + local_interrupt_disable(); + TEST_IRQ_OFF(); + local_interrupt_disable(); + TEST_IRQ_OFF(); + + local_interrupt_enable(); + TEST_IRQ_OFF(); + local_interrupt_enable(); + TEST_IRQ_OFF(); + local_interrupt_enable(); + TEST_IRQ_ON(); +} + +static void test_multiple_irq_change(struct kunit *test) +{ + local_interrupt_disable(); + TEST_IRQ_OFF(); + local_interrupt_disable(); + TEST_IRQ_OFF(); + + local_interrupt_enable(); + TEST_IRQ_OFF(); + local_interrupt_enable(); + TEST_IRQ_ON(); + + local_interrupt_disable(); + TEST_IRQ_OFF(); + local_interrupt_enable(); + TEST_IRQ_ON(); +} + +static void test_irq_save(struct kunit *test) +{ + unsigned long flags; + + local_irq_save(flags); + TEST_IRQ_OFF(); + local_interrupt_disable(); + TEST_IRQ_OFF(); + local_interrupt_enable(); + TEST_IRQ_OFF(); + local_irq_restore(flags); + TEST_IRQ_ON(); + + local_interrupt_disable(); + TEST_IRQ_OFF(); + local_irq_save(flags); + TEST_IRQ_OFF(); + local_irq_restore(flags); + TEST_IRQ_OFF(); + local_interrupt_enable(); + TEST_IRQ_ON(); +} + +static struct kunit_case test_cases[] =3D { + KUNIT_CASE(test_single_irq_change), + KUNIT_CASE(test_nested_irq_change), + KUNIT_CASE(test_multiple_irq_change), + KUNIT_CASE(test_irq_save), + {}, +}; + +/* (init and exit are the same */ +static int test_init(struct kunit *test) +{ + TEST_IRQ_ON(); + + return 0; +} + +static void test_exit(struct kunit *test) +{ + TEST_IRQ_ON(); +} + +static struct kunit_suite refcount_interrupt_test_suite =3D { + .name =3D "refcount_interrupt", + .test_cases =3D test_cases, + .init =3D test_init, + .exit =3D test_exit, +}; + +kunit_test_suite(refcount_interrupt_test_suite); +MODULE_AUTHOR("Lyude Paul "); +MODULE_DESCRIPTION("Refcounted interrupt unit test suite"); +MODULE_LICENSE("GPL"); --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 334254014A1; Fri, 5 Jun 2026 05:41:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638108; cv=none; b=LVCUUIqs4IVQ9wx5d6UFFlww4gEJhOp+grXhdTmdoE00xBDqm7sVow5rhvjI2EBqStoSVx8XXCszHadovnPFvG49zlzoPLfX969/NhI5Mm/aeSZL2dE8E5kBbE7kzNLVqlM49Cr80GGpefY9xePysqFkU9pztkYneM8jgehdNVo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638108; c=relaxed/simple; bh=CBE6V82/9cSnE5zKiDaINuabdT3G7eBPLHLX8ZZAYo4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QSjKjeV9bfMyg2PellGSv7z1itmecQzsRkTsDz/BCG1SJDTCtBsJfxg0DK5N2MpqMebKtnx2PQ6BSVIkR5SYcXmsbK5SmVpA/EmHkROpvRd16KsKb8SEazcqXvHdsbTvMNZg0hbG66gKIljY9I7pidEUw6BJDB01HaAEqV1T3Vs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YnDGd/nT; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YnDGd/nT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B29F31F00899; Fri, 5 Jun 2026 05:41:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638107; bh=3RwpUO4CBjZA9J030nD/wwixercDouC6GsvjDNlY070=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=YnDGd/nTdfwFkeLUrGAWfWEDOgtxAk2VkimxNu4cdz50kwDsVHIX3Rp8+SAN1xteJ I1ZKiauKMNznaA9/TvTcQCOSrnP0YOUUS+71uZjEX+XiJzvme/5iS7rmwaj0TFFIVl 7gYI8f16/w/r1DM56dtzIeLhORae8a0Hn3enVqpNtJSj+fNjk+LMDzjGK3zGNR50PG tWytHlQNCif7JwyrAsBKrXrGeRd1YZl/jG9D02MnMH7UDfbHuaK7UPtZB58ulwJJsZ rD/F0tE9aE82dyB9HQn0fphKuiW6B0bKog6pzbxqlpzwVDzQaQ/9KWjNoj0jBawHx/ KcnnVwcsGhkNA== Received: from phl-compute-10.internal (phl-compute-10.internal [10.202.2.50]) by mailfauth.phl.internal (Postfix) with ESMTP id 0791DF40068; Fri, 5 Jun 2026 01:41:44 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-10.internal (MEProxy); Fri, 05 Jun 2026 01:41:44 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFhOfDTFTb8fH37X+ASfp2vsHy1Fqb2Lywq6X3h5fMlxk7wW33nxe7aSTekseFO11 xBiB7Md3Dfg950zHftACRl4BCrZIGkOLORwuU667qK/t4b7uPTnvvrgSwCuAred6lniidQ x6+o43dstQnrbqqi/v6Nl3jrEdQj/OnTFIkpkf7j4tfd81Pm1F4hDhUzK5vMKXR0zokP4K KZGWpXQjfkPb9q0v8LW9bszX8CgnIWJLXLSKFiZe/Ul+6TINXLg1BnxiItQp/bB4zXvV+6 Qcp88FhWc5Q0N1TQTHx/VoF2pZ7nT3Uyapa4Doti8umM92UFgjuSWC48vH+lnu7P/MqXlF GeOVtGQCkz/77hw7utJcM8Nnu748HG644DL3wNUgTwYWNRbIZnxx2oDFfkUYK7pXuhbe+s dKyyp+45sTnSjZpM49Rg9gkpuNINa5ytP+OgAKtc8MjL1tnGTrmEpiizdjBTsXrNWbyux7 btZSz6tbrb7IkG4Knv3M1tLoxodRBPle8REkapVui30uzARmYC+ezi4lf9ZqW11/pk6n/d aZofGXE7LFwGU4avCC6fZigXlFUdQi8WVZW6cHlczGiNm69BpTx4jYafd/kKNIeoGA1AH9 xDsb4kw/sx05m+lLlDfRnF4BZ5/M7ZwZZL3qnsdXb8+tUNDopP1WeJGQMnMQ X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:43 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org, Boqun Feng Subject: [PATCH v3 07/13] locking: Switch to _irq_{disable,enable}() variants in cleanup guards Date: Thu, 4 Jun 2026 22:41:22 -0700 Message-ID: <20260605054128.5925-8-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Boqun Feng The semantics of various irq disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives. Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260121223933.1568682-17-lyude@redhat.com --- include/linux/spinlock.h | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index 3d405cc4c121..799a8f7d2741 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acq= uires(_T), __releases(*(raw #define class_raw_spinlock_nested_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(= raw_spinlock_nested, _T) =20 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t, - raw_spin_lock_irq(_T->lock), - raw_spin_unlock_irq(_T->lock)) + raw_spin_lock_irq_disable(_T->lock), + raw_spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(= raw_spinlock_t **)_T)) #define class_raw_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw= _spinlock_irq, _T) =20 -DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->= lock)) +DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disa= ble(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __release= s(*(raw_spinlock_t **)_T)) #define class_raw_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS= (raw_spinlock_irq_try, _T) =20 @@ -592,14 +592,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, __acq= uires(_T), __releases(*(raw #define class_raw_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(= raw_spinlock_bh_try, _T) =20 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t, - raw_spin_lock_irqsave(_T->lock, _T->flags), - raw_spin_unlock_irqrestore(_T->lock, _T->flags), - unsigned long flags) + raw_spin_lock_irq_disable(_T->lock), + raw_spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, __acquires(_T), __release= s(*(raw_spinlock_t **)_T)) #define class_raw_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS= (raw_spinlock_irqsave, _T) =20 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try, - raw_spin_trylock_irqsave(_T->lock, _T->flags)) + raw_spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, __acquires(_T), __rel= eases(*(raw_spinlock_t **)_T)) #define class_raw_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_A= TTRS(raw_spinlock_irqsave_try, _T) =20 @@ -618,13 +617,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_= T), __releases(*(spinlock_t #define class_spinlock_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinloc= k_try, _T) =20 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t, - spin_lock_irq(_T->lock), - spin_unlock_irq(_T->lock)) + spin_lock_irq_disable(_T->lock), + spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spin= lock_t **)_T)) #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinloc= k_irq, _T) =20 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try, - spin_trylock_irq(_T->lock)) + spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(= spinlock_t **)_T)) #define class_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spi= nlock_irq_try, _T) =20 @@ -640,14 +639,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_bh_try, __acquire= s(_T), __releases(*(spinloc #define class_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spin= lock_bh_try, _T) =20 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t, - spin_lock_irqsave(_T->lock, _T->flags), - spin_unlock_irqrestore(_T->lock, _T->flags), - unsigned long flags) + spin_lock_irq_disable(_T->lock), + spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave, __acquires(_T), __releases(*(= spinlock_t **)_T)) #define class_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spi= nlock_irqsave, _T) =20 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try, - spin_trylock_irqsave(_T->lock, _T->flags)) + spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, __acquires(_T), __release= s(*(spinlock_t **)_T)) #define class_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS= (spinlock_irqsave_try, _T) =20 --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DCF1401A0D; Fri, 5 Jun 2026 05:41:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638109; cv=none; b=ZXkSIyxzY8cEkrpYAGNQ8f7rOEEQgLUDR0pHhDgsw2miGxaOcrGB3SQMbKFmXNMUgNJ+zvNAcR0Hu2g57Us24GRrn8SsMF1E5RhxSIyzgVEXplqRtacM4SHr4w/w9g+oGPS/WD8QfRjbf6WwFNbbmdTIHm4hXpuqOrZrVteBONY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638109; c=relaxed/simple; bh=OCIQ1JNYaYn1HO1wd9Wum3zvGhBgUH5HslNCfI5WKfk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kkVvUUf8gblidwrekLGt+SgwDqRX4fJjJ110yk6SRWjkV/sbs3c0Yr3DfdML4fRwNdPucAiWjE2weupMfxDoppJ76yDzWb7kiVyyenQM1Zi9qoHNe78jV75zkcn1qDVj5RhHg452BG3h9GLNIign4Qsu7ubZALAvrfjrRalgZ8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RShrFXJC; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RShrFXJC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 577001F0089B; Fri, 5 Jun 2026 05:41:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638108; bh=na6ChhIW2x+wIN9W0OHq/rAhwLKFaVTMYx/jBn9ncTo=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=RShrFXJCQcqwe0yCV7e2WjfLKFGrueu70yGHB59yRWOyCSwQzAtnrFSBdjYVBwyNu jcL8zIRElJyagC9JOOSTS0BrBXu8bA8aY5Q9wKkA990JVg9197ltIP+zAyw+ADyNlp /Wn8/+hQgaj3+fX3jxJGg+0Nb2UoNTmOWcwDzoXuVTPAj4EHSl4luv54xMxeqfgnxH 7tmQJkiPX0pDBx/KFQeY3gVq0LboIc/94tMT+WPmCPfSV4PhEA0aNkZCZCaM+gdERi S0U4N91eEjpEC7VSfZC63+dUxdS2S38SofG9Cm5tA8t1c/0S02P3t5AGk2R4o1lg8+ J74yyg5TaRV3g== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.phl.internal (Postfix) with ESMTP id A0AAAF4006D; Fri, 5 Jun 2026 01:41:45 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Fri, 05 Jun 2026 01:41:45 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFG0WJa+84B5ZLakFCJusLrHx60u/M2e4tgb8vvSWGQWHXMFZeKktHow9VrFaQhwL 60vkPLe/GHUXrF9uQjn9EtWhYCiYebO343SXKkauObGNd0gaHEDMGyAUdMEWI3xvV5+MXM Y5XIGIH+RCj/YpjPsc3XHQUzqyjdf8JDGbXE7uRDsVlh2adsQWoVoEG+11Li6tmpxBjo+n GbGHXxcMsG5BC+v/PWXz65OFiJmnJTn18gjrdwmPsAQA0I/XEj3HJhBv36WGSvCgORqSG4 pkDfFOinWXk05Px2n3ZYSDFusuNrnah7Cqk2jZsViR99U7a3ccatK3SGLf/zDxH9xSS8M3 BYqcTNPoSFWTUAYnzRQsPMjoijWzTQhJOBECvxoAHstZsQERmbxNGFxrYJ8nuf451xz6Gi JZckgK1Vr5EFN2I5rL2HrR+uvyRf0Y+ZKW5Aw/kjsrHcSbsPMzgMKvgil5ZUhIbAr+LJ5o eosapYegMdAxAxC/a1+cCt7IJcekuaTmfVEbYwAJwRPjaQcP6RF4fHYxtzwOSbipcxtilW zSZYZFVdtyChk/sU7KPXknN2o+XmbO6+PO+/G8uhRcp2pRsqwCpE9WqpYEAFBbyOIc0hBF e6ovp43WLtoCFTStA2qhLzt36hynsitQNYUu5oVB7X6mw6jqChkvdvQitnRw X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:44 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 08/13] sched: Remove the unused preempt_offset parameter of __cant_sleep() Date: Thu, 4 Jun 2026 22:41:23 -0700 Message-ID: <20260605054128.5925-9-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The preempt_offset is always 0 in all the callsites of __cant_sleep(), hence remove it. It also allows us to clear the code a bit by stopping using a "preempt_count() > .." comparison. Signed-off-by: Boqun Feng --- include/linux/kernel.h | 4 ++-- kernel/sched/core.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/kernel.h b/include/linux/kernel.h index e5570a16cbb1..24414c79e59a 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -72,7 +72,7 @@ extern int dynamic_might_resched(void); #ifdef CONFIG_DEBUG_ATOMIC_SLEEP extern void __might_resched(const char *file, int line, unsigned int offse= ts); extern void __might_sleep(const char *file, int line); -extern void __cant_sleep(const char *file, int line, int preempt_offset); +extern void __cant_sleep(const char *file, int line); extern void __cant_migrate(const char *file, int line); =20 /** @@ -95,7 +95,7 @@ extern void __cant_migrate(const char *file, int line); * this macro will print a stack trace if it is executed with preemption e= nabled */ # define cant_sleep() \ - do { __cant_sleep(__FILE__, __LINE__, 0); } while (0) + do { __cant_sleep(__FILE__, __LINE__); } while (0) # define sched_annotate_sleep() (current->task_state_change =3D 0) =20 /** diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b8871449d3c6..75dba7cc09bd 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9165,7 +9165,7 @@ void __might_resched(const char *file, int line, unsi= gned int offsets) } EXPORT_SYMBOL(__might_resched); =20 -void __cant_sleep(const char *file, int line, int preempt_offset) +void __cant_sleep(const char *file, int line) { static unsigned long prev_jiffy; =20 @@ -9175,7 +9175,7 @@ void __cant_sleep(const char *file, int line, int pre= empt_offset) if (!IS_ENABLED(CONFIG_PREEMPT_COUNT)) return; =20 - if (preempt_count() > preempt_offset) + if (preempt_count()) return; =20 if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy) --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B866740586A; Fri, 5 Jun 2026 05:41:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638113; cv=none; b=URH8KyQTrQuMaxRZLHg+YXExCJiR1k6I0mYacCBAFB+nT2CKDcLPZASUmB5EJNTccntSdL3jh/kWgZ2GU0vrlTLnYDyndK0eB9jh9oK1/IlWQhosnL1ZQOfrn9e2Jlz/6ttJgq8L2fwZ3awcfJLIgPoykwxLqykNhMang8SfJJQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638113; c=relaxed/simple; bh=EEF7M1M56zwwhTGXRepJ6Df3YmQZblYNwV9p/gbKAlY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H0iaSxusWlzCnDVFl56XSAZmA1CIL021MiqHYuAH96xBPocnVXauTQgt2G7tdtdyegWa1li+cdwAceBAD8uZaH5uOATkLYhwykiaRysD2actu/qHsWTC+FquKxc4An1G08/IzBL8g8cBakngPp1/TDp1emEcCHQbqRVrvDqweRo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bZoiokMs; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bZoiokMs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2D561F008A1; Fri, 5 Jun 2026 05:41:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638110; bh=pFJVb0XMzYiDkWRfHTccDmCKdRjygd0kE9XeKrNBbFM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=bZoiokMslLIjN2vzT4TIFP69TYMaOYz/Rs8dBvcDVfh8znI5PyQZZ/MI6h3qmF8+J b8FKP5o9Y5Kg/pfhZicPqaKRm0YEhlLb4BD1sbQoJ5K2gC2cU0VzVIDPO8b+U58q5q FA8v9c06PZwWgVmOt8V3eDYVr3o5b+hTQ76ys3oo2xVbh+ItLecG0caDz97MCKImn5 aWP5IjVIw+HArzbUh8huLosDMiMAchoYiheeZ0nQI8sPAZsiMSij+Kw6ccKBJJnEVR r+r8gSfJl4Q+t6jJrcUyMG8SwUAG3vrB+lzxKaGcs3tIy9tn7JnJInhrAQ1pm8GeuL f9ui1biHAMgQw== Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfauth.phl.internal (Postfix) with ESMTP id 0C7FEF40068; Fri, 5 Jun 2026 01:41:48 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Fri, 05 Jun 2026 01:41:48 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFG0WJa+84B5ZLakFCJusLrHx60u/M2e4tgb8vvSWGQWHXMFZeKktHow9VrFaQhwL 60vkPLe/GHUXrF9uQjn9EtWhYCiYebO343SXKkauObGNd0gaHEDMGyAUdMEWI3xvV5+MXM Y5XIGIH+RCj/YpjPsc3XHQUzqyjdf8JDGbXE7uRDsVlh2adsQWoVoEG+11Li6tmpxBjo+n GbGHXxcMsG5BC+v/PWXz65OFiJmnJTn18gjrdwmPsAQA0I/XEj3HJhBv36WGSvCgORqSG4 pkDfFOinWXk05Px2n3ZYSDFusuNrnah7Cqk2jZsViR99U7a3ccatK3SGLf/zDxH9xSS8Bx NJDPeypZR+UT9k/SqltVTKWiw30nGSTGNtoJTZhFEw535tbPvHx+fxLRs5Z4ou1e4WVvr5 fjYq3d0sE8tdFC8joSEqZ+Uo7JwQns/hPHB/VwsR4PvGZtpea1JqcCyFMlPJgyyx8EE4pv jvz4p8DgyUVrtSDfrlYz1plh7KfMMU0jKMO8SRKIOnV1u69YyXD/2yFnyoWy2B1n65QqoC B93ANNq/Xa14drwvS5S7v3OdLu8oSVLylHkLZ+VvgmcmCDl4meFSxlGDWOyi5EBtb/cBbE fuJpgrv2lvgm+etFqRsdjN9nNR9q729Z/AtVZ6SmDmULleejLzV6BAMpyAjQ X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:46 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 09/13] sched: Avoid signed comparison of preempt_count() in __cant_migrate() Date: Thu, 4 Jun 2026 22:41:24 -0700 Message-ID: <20260605054128.5925-10-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently preempt_count() is always a non-negative int on all archs (PREEMPT_NEED_RESCHED archs will mask out the MSB when return preempt_count()), hence the checking in __cant_migrate() is in fact just checking whether preempt_count() is 0 or not. In a future change, we are going to use all the 32 bits of preempt_count(), which would make negative int values possible from preempt_count(). Therefore convert the "> 0" comparison into a zero checking to prepare for the future change. No functional changes are intended. Signed-off-by: Boqun Feng --- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 75dba7cc09bd..636e6a15f104 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9207,7 +9207,7 @@ void __cant_migrate(const char *file, int line) if (!IS_ENABLED(CONFIG_PREEMPT_COUNT)) return; =20 - if (preempt_count() > 0) + if (preempt_count()) return; =20 if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy) --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6465F3F54C7; Fri, 5 Jun 2026 05:41:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638120; cv=none; b=VkiwNSw63BEUa57QsCODsdyAbyxXv2kbNs/jp1PJCRT8f53b6FWiHz2vrtlXO+QON/YYWnVXIyfwm5hKNZ9DoZE0c8BQjlPC2+Z/oGkyJoUTkrz383cNbL4ZTKO9nnuOrY2ngBMisJU/xwy7yu4Ds5WvuURbX2ki2r2Hc5Yyt4I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638120; c=relaxed/simple; bh=GQcB2irY+5fvbG7jypfJORHvkL0py2l9L68BR/+gH5k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n2C2pvH+6dzr0TLFcpE5QtOnbOA11nhai1DHXtUPni6xIxkSYTdWTd96f5PXfnW+86FgPMZ0Q4eTvJxUlEpW0eeEWx7dtbpiWdtC4k8GEQjnPV744Q5Ghcxlsj9mQyeksn7QsmrlnZg6MVYPVWfl7ScGZlcRnVHim0mbRckhsR4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ikd0bBdk; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ikd0bBdk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C3331F00898; Fri, 5 Jun 2026 05:41:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638113; bh=0mPxrbBivbpzQrGwnMXtIl1/fg5S1RI4hpeS7qRM9AI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Ikd0bBdk69NP91MfJhlMrCO2IH6G4KXohWTL1J6xdWnrn19Or8EGUQ+jeqvGJUsL1 w7KGao2dpPh/F98lxIMoCl+XDPAcxP+jYzxdiC1Fu8nhuoyJZfnGKt0MzbkNe52WNA iNG8CvvHi/9tSZLsgYtAkzqQK4lrYCtyw4kY89A0GugwcbL0xIc8EQYCbOknbrcPoj 0Q9ehDGpIX2uhp0A8MAd6tyn7Stc1LfwDYBROo1LA5GEDpxy1NRI82yOxJ9KMyLm+K Yl1kFbTrNQjiW1UcMsb5voAaLuE6WVTrkvtYpX74ADTPYoGKes0K8MNSkDQ8Mo9mgG hmi17udHUqx7A== Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfauth.phl.internal (Postfix) with ESMTP id 536B5F4006D; Fri, 5 Jun 2026 01:41:50 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Fri, 05 Jun 2026 01:41:50 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFG0WJa+84B5ZLakFCJusLrHx60u/M2e4tgb8vvSWGQWHXMFZeKktHow9VrFaQhwL 60vkPLe/GHUXrF9uQjn9EtWhYCiYebO343SXKkauObGNd0gaHEDMGyAUdMEWI3xvV5+MXM Y5XIGIH+RCj/YpjPsc3XHQUzqyjdf8JDGbXE7uRDsVlh2adsQWoVoEG+11Li6tmpxBjo+n GbGHXxcMsG5BC+v/PWXz65OFiJmnJTn18gjrdwmPsAQA0I/XEj3HJhBv36WGSvCgORqSG4 pkDfFOinWXk05Px2n3ZYSDFusuNrnah7Cqk2jZsViR99U7a3ccatK3SGLf/zDxH9xSS8eP evo1C1tdvdsW22rguSMiyGhb91vOPz1JMGy/dAC8+b57XEttl8KUCjqIV63p0YMoxNW6pk NJRjB/fcA+k20epvbFfev+gbwksz8T6PzNVKqm0FCkxFAJGT6V7UWUiVT4yMenf2km2OBL LIDGjit+Stvx8RgQ/+f1YlsO1xj4o5d3Dm9gw95alvhK0w35SrVU+y9Atd1IUrylyOkx57 RLeY4uqNNMkPi8PfhqmecwQo3Fbck1lHuoF3ee1JyAGbeX7v5G1uf/lv+0ODXAJXYI0Z8C ByMgPqWEV3THt+bVqfddzh0sofM9ysG1wdnUAvOqu5AziUZTeSQ8i9v235dw X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:48 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 10/13] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS Date: Thu, 4 Jun 2026 22:41:25 -0700 Message-ID: <20260605054128.5925-11-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With the changes that enable preempt count to tracking irq disabling nesting, we don't have enough bits in 32bit preempt count implementation, as a result we move NMI nesting bits out of the 32bit preempt count. However on the architectures that can support 64bit preempt count implementation, we can keep the NMI nesting bits in the 32bit preempt count and avoid maintaining NMI nesting bits out of the same cache line. Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow architectures to select this. Note that under this kconfig, preempt count is maintained in a 64bit word however preempt_count() still remains as an int because all the effective bits still fit in (previously we mask out NEED_RESCHED bit in preempt_count()). This should make no functional changes for existing preempt_count() users. Enable this for x86_64 along with the introduction of the Kconfig. [boqun: Undo the __preempt_count_{add,sub}() optimization in 32bit preempt count since it may introduce {over,under}flow] Originally-by: Peter Zijlstra Signed-off-by: Boqun Feng --- arch/x86/Kconfig | 1 + arch/x86/include/asm/preempt.h | 55 +++++++++++++++++++++++----------- arch/x86/kernel/cpu/common.c | 2 +- include/linux/hardirq.h | 48 +++++++++++++++++++++-------- include/linux/preempt.h | 20 +++++++------ kernel/Kconfig.preempt | 4 +++ kernel/sched/core.c | 12 ++++++-- kernel/softirq.c | 6 ++++ lib/locking-selftest.c | 2 +- 9 files changed, 107 insertions(+), 43 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index fdaef60b46d6..9a35c3a08757 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -329,6 +329,7 @@ config X86 select USER_STACKTRACE_SUPPORT select HAVE_ARCH_KCSAN if X86_64 select PROC_PID_ARCH_STATUS if PROC_FS + select HAS_SEPARATE_PREEMPT_RESCHED_BITS if X86_64 select HAVE_ARCH_NODE_DEV_GROUP if X86_SGX select FUNCTION_ALIGNMENT_16B if X86_64 || X86_ALIGNMENT_16 select FUNCTION_ALIGNMENT_4B diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h index 1220656f3370..12353eeebc52 100644 --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -7,10 +7,20 @@ =20 #include =20 -DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count); +DECLARE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count); =20 -/* We use the MSB mostly because its available */ -#define PREEMPT_NEED_RESCHED 0x80000000 +/* + * We use the MSB for PREEMPT_NEED_RESCHED mostly because it is available. + */ +#define PREEMPT_NEED_RESCHED (~(((unsigned long)-1L) >> 1)) + +#ifdef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS +#define __pc_dec "decq" +#define __pc_op(op, ...) raw_cpu_##op##_8(__VA_ARGS__) +#else +#define __pc_dec "decl" +#define __pc_op(op, ...) raw_cpu_##op##_4(__VA_ARGS__) +#endif =20 /* * We use the PREEMPT_NEED_RESCHED bit as an inverted NEED_RESCHED such @@ -24,18 +34,26 @@ DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count); */ static __always_inline int preempt_count(void) { - return raw_cpu_read_4(__preempt_count) & ~PREEMPT_NEED_RESCHED; + return __pc_op(read, __preempt_count) & ~PREEMPT_NEED_RESCHED; } =20 -static __always_inline void preempt_count_set(int pc) +/* + * unsigned long preempt count parameter works for both 32bit and 64bit ca= ses: + * + * - For 32bit, "int" (the return of preempt_count()) and "unsigned long" = have + * the same size. + * - For 64bit, the effective bits of a preempt count sits in 32bit, and we + * reserve the NEED_RESCHED bit from the old count. + */ +static __always_inline void preempt_count_set(unsigned long pc) { - int old, new; + unsigned long old, new; =20 - old =3D raw_cpu_read_4(__preempt_count); + old =3D __pc_op(read, __preempt_count); do { new =3D (old & PREEMPT_NEED_RESCHED) | (pc & ~PREEMPT_NEED_RESCHED); - } while (!raw_cpu_try_cmpxchg_4(__preempt_count, &old, new)); + } while (!__pc_op(try_cmpxchg, __preempt_count, &old, new)); } =20 /* @@ -58,17 +76,17 @@ static __always_inline void preempt_count_set(int pc) =20 static __always_inline void set_preempt_need_resched(void) { - raw_cpu_and_4(__preempt_count, ~PREEMPT_NEED_RESCHED); + __pc_op(and, __preempt_count, ~PREEMPT_NEED_RESCHED); } =20 static __always_inline void clear_preempt_need_resched(void) { - raw_cpu_or_4(__preempt_count, PREEMPT_NEED_RESCHED); + __pc_op(or, __preempt_count, PREEMPT_NEED_RESCHED); } =20 static __always_inline bool test_preempt_need_resched(void) { - return !(raw_cpu_read_4(__preempt_count) & PREEMPT_NEED_RESCHED); + return !(__pc_op(read, __preempt_count) & PREEMPT_NEED_RESCHED); } =20 /* @@ -77,22 +95,22 @@ static __always_inline bool test_preempt_need_resched(v= oid) =20 static __always_inline void __preempt_count_add(int val) { - raw_cpu_add_4(__preempt_count, val); + __pc_op(add, __preempt_count, val); } =20 static __always_inline void __preempt_count_sub(int val) { - raw_cpu_add_4(__preempt_count, -val); + __pc_op(add, __preempt_count, -val); } =20 static __always_inline int __preempt_count_add_return(int val) { - return raw_cpu_add_return_4(__preempt_count, val); + return __pc_op(add_return, __preempt_count, val); } =20 static __always_inline int __preempt_count_sub_return(int val) { - return raw_cpu_add_return_4(__preempt_count, -val); + return __pc_op(add_return, __preempt_count, -val); } =20 /* @@ -102,7 +120,7 @@ static __always_inline int __preempt_count_sub_return(i= nt val) */ static __always_inline bool __preempt_count_dec_and_test(void) { - return GEN_UNARY_RMWcc("decl", __my_cpu_var(__preempt_count), e, + return GEN_UNARY_RMWcc(__pc_dec, __my_cpu_var(__preempt_count), e, __percpu_arg([var])); } =20 @@ -111,7 +129,7 @@ static __always_inline bool __preempt_count_dec_and_tes= t(void) */ static __always_inline bool should_resched(int preempt_offset) { - return unlikely(raw_cpu_read_4(__preempt_count) =3D=3D preempt_offset); + return unlikely(__pc_op(read, __preempt_count) =3D=3D preempt_offset); } =20 #ifdef CONFIG_PREEMPTION @@ -158,4 +176,7 @@ do { \ =20 #endif /* PREEMPTION */ =20 +#undef __pc_op +#undef __pc_dec + #endif /* __ASM_PREEMPT_H */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index a4268c47f2bc..182772b6ad6d 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2240,7 +2240,7 @@ DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, curren= t_task) =3D &init_task; EXPORT_PER_CPU_SYMBOL(current_task); EXPORT_PER_CPU_SYMBOL(const_current_task); =20 -DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; +DEFINE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count) =3D INIT_PREEMPT_= COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) =3D TOP_= OF_INIT_STACK; diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index 8d4895531a45..b5dc907274db 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -10,8 +10,6 @@ #include #include =20 -DECLARE_PER_CPU(unsigned int, nmi_nesting); - extern void synchronize_irq(unsigned int irq); extern bool synchronize_hardirq(unsigned int irq); =20 @@ -94,6 +92,38 @@ void irq_exit_rcu(void); #define arch_nmi_exit() do { } while (0) #endif =20 +#ifdef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS +static __always_inline void __preempt_count_nmi_enter(void) +{ + __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); +} + +static __always_inline void __preempt_count_nmi_exit(void) +{ + __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); +} +#else +DECLARE_PER_CPU(unsigned int, nmi_nesting); + +#define __preempt_count_nmi_enter() \ + do { \ + __preempt_count_add(HARDIRQ_OFFSET); \ + /* Maximum NMI nesting is 15. */ \ + BUG_ON(__this_cpu_read(nmi_nesting) >=3D 15); \ + __this_cpu_inc(nmi_nesting); \ + preempt_count_set(preempt_count() | NMI_MASK); \ + } while (0) + +#define __preempt_count_nmi_exit() \ + do { \ + __preempt_count_sub(HARDIRQ_OFFSET); \ + if (!__this_cpu_dec_return(nmi_nesting)) \ + preempt_count_set(preempt_count() & ~NMI_MASK); \ + } while (0) + +#endif + + /* * NMI vs Tracing * -------------- @@ -110,18 +140,14 @@ void irq_exit_rcu(void); do { \ lockdep_off(); \ arch_nmi_enter(); \ - /* Maximum NMI nesting is 15. */ \ - BUG_ON(__this_cpu_read(nmi_nesting) >=3D 15); \ - __this_cpu_inc(nmi_nesting); \ - __preempt_count_add(HARDIRQ_OFFSET); \ - preempt_count_set(preempt_count() | NMI_MASK); \ + __preempt_count_nmi_enter(); \ } while (0) =20 #define nmi_enter() \ do { \ __nmi_enter(); \ lockdep_hardirq_enter(); \ - ct_nmi_enter(); \ + ct_nmi_enter(); \ instrumentation_begin(); \ ftrace_nmi_enter(); \ instrumentation_end(); \ @@ -129,12 +155,8 @@ void irq_exit_rcu(void); =20 #define __nmi_exit() \ do { \ - unsigned int nesting; \ BUG_ON(!in_nmi()); \ - __preempt_count_sub(HARDIRQ_OFFSET); \ - nesting =3D __this_cpu_dec_return(nmi_nesting); \ - if (!nesting) \ - preempt_count_set(preempt_count() & ~NMI_MASK); \ + __preempt_count_nmi_exit(); \ arch_nmi_exit(); \ lockdep_on(); \ } while (0) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 33fc4c814a9f..87d5367f986c 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -30,18 +30,20 @@ * NMI nesting depth is tracked in a separate per-CPU variable * (nmi_nesting) to save bits in preempt_count. * - * PREEMPT_MASK: 0x000000ff - * SOFTIRQ_MASK: 0x0000ff00 - * HARDIRQ_DISABLE_MASK: 0x00ff0000 - * HARDIRQ_MASK: 0x0f000000 - * NMI_MASK: 0x10000000 - * PREEMPT_NEED_RESCHED: 0x80000000 + * 32bit HAS_SEPARATE_PREEMPT_RESCHED_BITS + * + * PREEMPT_MASK: 0x000000ff 0x00000000000000ff + * SOFTIRQ_MASK: 0x0000ff00 0x000000000000ff00 + * HARDIRQ_DISABLE_MASK: 0x00ff0000 0x0000000000ff0000 + * HARDIRQ_MASK: 0x0f000000 0x000000000f000000 + * NMI_MASK: 0x10000000 0x00000000f0000000 + * PREEMPT_NEED_RESCHED: 0x80000000 0x8000000000000000 */ #define PREEMPT_BITS 8 #define SOFTIRQ_BITS 8 #define HARDIRQ_DISABLE_BITS 8 #define HARDIRQ_BITS 4 -#define NMI_BITS 1 +#define NMI_BITS (1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BIT= S)) =20 #define PREEMPT_SHIFT 0 #define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) @@ -116,8 +118,8 @@ static __always_inline unsigned char interrupt_context_= level(void) * preempt_count() is commonly implemented with READ_ONCE(). */ =20 -#define nmi_count() (preempt_count() & NMI_MASK) -#define hardirq_count() (preempt_count() & HARDIRQ_MASK) +#define nmi_count() (preempt_count() & NMI_MASK) +#define hardirq_count() (preempt_count() & HARDIRQ_MASK) #ifdef CONFIG_PREEMPT_RT # define softirq_count() (current->softirq_disable_cnt & SOFTIRQ_MASK) # define irq_count() ((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | sof= tirq_count()) diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt index 88c594c6d7fc..35f546a042b1 100644 --- a/kernel/Kconfig.preempt +++ b/kernel/Kconfig.preempt @@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK config PREEMPT_COUNT bool =20 +config HAS_SEPARATE_PREEMPT_RESCHED_BITS + bool + depends on PREEMPT_COUNT && 64BIT + config PREEMPTION bool select PREEMPT_COUNT diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 636e6a15f104..f4c944878516 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5847,8 +5847,13 @@ void preempt_count_add(int val) #ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? + * + * Cannot detect underflow based on the current preempt_count() value + * if using HAS_SEPARATE_PREEMPT_RESCHED_BITS because preempt count takes= all 32 + * bits. */ - if (DEBUG_LOCKS_WARN_ON((preempt_count() < 0))) + if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) && + DEBUG_LOCKS_WARN_ON((preempt_count() < 0))) return; #endif __preempt_count_add(val); @@ -5880,7 +5885,10 @@ void preempt_count_sub(int val) /* * Underflow? */ - if (DEBUG_LOCKS_WARN_ON(val > preempt_count())) + unsigned int uval =3D val; + unsigned int pc =3D preempt_count(); + + if (DEBUG_LOCKS_WARN_ON(pc - uval > pc)) return; /* * Is the spinlock portion underflowing? diff --git a/kernel/softirq.c b/kernel/softirq.c index d1ab1799794c..491136a313db 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -91,7 +91,13 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_sta= te); EXPORT_PER_CPU_SYMBOL_GPL(local_interrupt_disable_state); =20 +#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS +/* + * Any 32bit architecture that still cares about performance should + * probably ensure this is near preempt_count. + */ DEFINE_PER_CPU(unsigned int, nmi_nesting); +#endif =20 /* * SOFTIRQ_OFFSET usage: diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c index bfafe1204c7b..c3d976c801bb 100644 --- a/lib/locking-selftest.c +++ b/lib/locking-selftest.c @@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures; =20 static void dotest(void (*testcase_fn)(void), int expected, int lockclass_= mask) { - int saved_preempt_count =3D preempt_count(); + long saved_preempt_count =3D preempt_count(); #ifdef CONFIG_PREEMPT_RT int saved_mgd_count =3D current->migration_disabled; int saved_rcu_count =3D current->rcu_read_lock_nesting; --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D4433FF1C5; Fri, 5 Jun 2026 05:41:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638120; cv=none; b=SmfiPex/qHjP2DVAq2x4568mMtAO6bqgJhUThE5PGyVwAjOdLDrX1AYKQFUfIs0hemJRqSCNO22JLFYR7a04Nsa+SVCO7UmC1AkQXRBBgPVsaDJzDRuYID4H++ehiKROip39xCjj+hK6usB5XdQlGI9ifee/TGVgLFKjQ4BZyDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638120; c=relaxed/simple; bh=1Z0r1dd+5UaBxiJIV7yyJHPW2au7yP4Ki/behDAnWqk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d1CnL8xzC5MJM3c5nOUsbD3rGiMeJ0d3QSwk1gnH4kee9jUClXUcQwiH1gFi2eCkZjLqjI3AqZH75OVsJlh50NR8iSD5T+mvTz8IwoRzykdG2Uj7x76SD8aO5w+u1onMucOAh1jWc/hz0ifvbz9zESvrkmHK6vyEasm7+RjioEA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gi/kWcPB; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gi/kWcPB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8BE0D1F0089F; Fri, 5 Jun 2026 05:41:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638115; bh=5fQe3JBbW1ecnzj9lOjL0hmTD9Wg7GMvup213RdzMTc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=gi/kWcPBW3EaGbMHej9lPJNdHzD0aaT8t6m94Sjb60a27aJe7SiCne06Jsn+qxN96 37Y/n1XbOxEYpKdk49Ahtlzy2PwZ0iIpLwdYfYd8psvWHzyR8CEgchyzJPtRZvHS4r tIrYbyabjXurnV9xC63mXw5P0tsIRx+JcoMvwdpsep7j/w9zRYfFMO4AuLsmq2cgvv nBkwod7gMrEG8isIGul3Ofp7t9ajodlfFo/by06YuEXHLen9VPQtepXJiCal9ygT/G 8DlzVulejJ3igmv2NIcyt0Gtxy1ms1LGR2LOgAXAHlh3kiDSpks4jlsEamObzqj9xq E7FpkXeg/8QMg== Received: from phl-compute-08.internal (phl-compute-08.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id C9C7CF40068; Fri, 5 Jun 2026 01:41:52 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Fri, 05 Jun 2026 01:41:52 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFG0WJa+84B5ZLakFCJusLrHx60u/M2e4tgb8vvSWGQWHXMFZeKktHow9VrFaQhwL 60vkPLe/GHUXrF9uQjn9EtWhYCiYebO343SXKkauObGNd0gaHEDMGyAUdMEWI3xvV5+MXM Y5XIGIH+RCj/YpjPsc3XHQUzqyjdf8JDGbXE7uRDsVlh2adsQWoVoEG+11Li6tmpxBjo+n GbGHXxcMsG5BC+v/PWXz65OFiJmnJTn18gjrdwmPsAQA0I/XEj3HJhBv36WGSvCgORqSG4 pkDfFOinWXk05Px2n3ZYSDFusuNrnah7Cqk2jZsViR99U7a3ccatK3SGLf/zDxH9xSS8rM AOeG5ePFZZfA76kOVqoCfj7btvpGgV7BkafEpMDLqRqYQteE3BP+Zkc3tGEg6HKAx5ebSG +mfyKbfml1bRlYZdW9QkLLsmbzmqf4KpTRUTEWVFfODasljVxHQMv5gX9K8gy1vxzFXTaU cZCZZhuI5rcfutz0yazglWB3rJwiDthh8IdApnGVTsXOrx3v5UsJw77FXSlh9Kv42p15JE WawlWP7bZ34qpayMprYpdDmWSqEzyCJKbCu+zWYZNgKoMIsVhnPsVU7hmpSe6Ah/EF1PrW mwcS+qw7SgGtRJaM1gozqo5XbhQOYZn73l24WeQ/UR8UYb5mg5SORHHlOdLw X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:51 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 11/13] arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS Date: Thu, 4 Jun 2026 22:41:26 -0700 Message-ID: <20260605054128.5925-12-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ARM64 already uses 64bit preempt count and the need reschedule bit is maintained in a separate 32bit than the preempt count. Therefore preempt count has enough bits to represent 16 level of NMI nesting, hence enable it for ARM64. This saves a per-CPU variable and additional instructions in the NMI path. Signed-off-by: Boqun Feng --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fe60738e5943..8178cb857115 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -248,6 +248,7 @@ config ARM64 select PCI_SYSCALL if PCI select POWER_RESET select POWER_SUPPLY + select HAS_SEPARATE_PREEMPT_RESCHED_BITS select SPARSE_IRQ select SWIOTLB select SYSCTL_EXCEPTION_TRACE --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 153D640149F; Fri, 5 Jun 2026 05:41:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638119; cv=none; b=lbQqJH7Etqtc4oLjB66gfY5S2m7tM2mqrIgH3UmwcOOttGNHyzVLuVW14Y6oi5h1L6xpHovqZ4OWL4Ik7PrhWBM771RxY/SCtokIXm+QOw+QOqh3aud3ma3etv53FvYOnTJmtFyHkW6JPl/InDmtk6NLU7/577es6R0zH/HgqoM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638119; c=relaxed/simple; bh=dLOHKW7WAr1j2ZaVPZw1AOGUtt3G5GCtMKdlXWtC1+g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HMhcHg2Pn+5o4jxfHN+f7aK8dvoTGxRROi9rPtDh8X7N0A1Cx8cRbOMMAK4uQjhiN5mFek36GjzU50qxlfEXLfM87GjideRAGLXdzjmTL0ay0bqEvjFUOJ3AOHWGaoRlJcnqE0mhUqs/txtl4Vl+ngjcqiC/ohNaUdQ4cNhwr6g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SRZfQ1Em; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SRZfQ1Em" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29B241F008A1; Fri, 5 Jun 2026 05:41:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638117; bh=cte+3SORcsAa7eW9vZLxBcB/RUJa01gm+E+K6ToVCog=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=SRZfQ1EmhHh/fcOTKw4OmaKBk72iM6URnBfbl4OIYW3EsxD79F26PyzDv5DakTIKk 0gazh2CEDaAOaWaWCJ8JVd8Bn+go1fER4lmBlZ+v/utn/XT/SDXBfu151UbTFnzlNg vA4/kcwzm+IfMRgOGGBdjBjyEIIKL8COuTwzJoxH/0FDY8d3DgMWLwgzId6XQ90ily 0TVqxVBKmxgrojgAAoFF7Hk5PozoFp5EvVKSUjGHmd2hng9z6mUZpt/DaDD03HnvUD oyrH246/GqVTxxGMypjesyTV/StUgAjJy25GqmhsQbjsUyJBwS8Z8R6devg1dGrdUd Ed+uDFgESmGQw== Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfauth.phl.internal (Postfix) with ESMTP id 7094EF4006D; Fri, 5 Jun 2026 01:41:54 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Fri, 05 Jun 2026 01:41:54 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGNdErcnsY9nz6zZJNqzBGcFpZ3oCsZyVSAsXPEfKD6fiAd7MM8RW3ErpHOKO8vqP y7FRxpXUBU5yb8Y73fmewycTgcYRdiPbEiXJpQaKxk1+aC6Vj3AAV2l/TRkhTd+1u+mIGl dQfEAWycnSLZO+qO217j/GTm+GQxoHWyRwvdph+wtnOrnU/D7PvfaSdl50PQ+1o9QW7RYj snzJFBhgLjSJ/jkZdib8Dc9ob5AHsrvEPqt2vef8Hmx2uDf0JqmoZZ+HexXrtwk5r0bry3 HVu0i63lg3nGwPLJ2r8Np94WJ6jG6+OWNN7TpxDaYMxO59AtKcBppdJ6IoJJH6Zm9jQmfy d/M23rmFX/UEDMXUP5h4monMRjH3DBFbRBl3dq3TGCS9QG3xV5SjlbUSvMywVg8PIR6IFK KXMTZHNWG50pa++o/LTrjqU+tS5L11u4ow7nYXPXlPIFaek05dqOtUwW/zxRhcsbtpEZSR ISus0WT/tkwAGiTp3tz0+/6ZUWTvLgozaBgYkLZTwlbtRl/bPzm1sk3zU61ZBdP4XfrXtm nk3D+iIL4bflhzo6zjYipul55NCqZP5sTBozuwFgumkeLXVXTq/GWN7Czth//d3jja9eh1 h/wjOwNzkRTyhq3JTkj4jSZ4w0vgmr38G80/zXvpjAFcVTP7qJZ/MeUZNobA X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:53 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 12/13] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS Date: Thu, 4 Jun 2026 22:41:27 -0700 Message-ID: <20260605054128.5925-13-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Heiko Carstens Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly. Signed-off-by: Heiko Carstens Signed-off-by: Boqun Feng Link: https://patch.msgid.link/20260509181249.16281C67-hca@linux.ibm.com --- arch/s390/Kconfig | 1 + arch/s390/include/asm/lowcore.h | 13 +++++++---- arch/s390/include/asm/preempt.h | 41 +++++++++++++++------------------ 3 files changed, 29 insertions(+), 26 deletions(-) diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index ecbcbb781e40..cbbca82f8443 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -276,6 +276,7 @@ config S390 select PCI_MSI if PCI select PCI_MSI_ARCH_FALLBACKS if PCI_MSI select PCI_QUIRKS if PCI + select HAS_SEPARATE_PREEMPT_RESCHED_BITS select SPARSE_IRQ select SWIOTLB select SYSCTL_EXCEPTION_TRACE diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcor= e.h index 50ffe75adeb4..0974ab278169 100644 --- a/arch/s390/include/asm/lowcore.h +++ b/arch/s390/include/asm/lowcore.h @@ -160,10 +160,15 @@ struct lowcore { /* SMP info area */ __u32 cpu_nr; /* 0x03a0 */ __u32 softirq_pending; /* 0x03a4 */ - __s32 preempt_count; /* 0x03a8 */ - __u32 spinlock_lockval; /* 0x03ac */ - __u32 spinlock_index; /* 0x03b0 */ - __u8 pad_0x03b4[0x03b8-0x03b4]; /* 0x03b4 */ + union { + struct { + __u32 need_resched; /* 0x03a8 */ + __u32 count; /* 0x03ac */ + } preempt; + __u64 preempt_count; /* 0x03a8 */ + }; + __u32 spinlock_lockval; /* 0x03b0 */ + __u32 spinlock_index; /* 0x03b4 */ __u64 percpu_offset; /* 0x03b8 */ __u8 pad_0x03c0[0x0400-0x03c0]; /* 0x03c0 */ =20 diff --git a/arch/s390/include/asm/preempt.h b/arch/s390/include/asm/preemp= t.h index 0a25d4648b4c..1d5e4d7e9e1b 100644 --- a/arch/s390/include/asm/preempt.h +++ b/arch/s390/include/asm/preempt.h @@ -8,11 +8,8 @@ #include #include =20 -/* - * Use MSB so it is possible to read preempt_count with LLGT which - * reads the least significant 31 bits with a single instruction. - */ -#define PREEMPT_NEED_RESCHED 0x80000000 +/* Use MSB for PREEMPT_NEED_RESCHED mostly because it is available. */ +#define PREEMPT_NEED_RESCHED 0x8000000000000000UL =20 /* * We use the PREEMPT_NEED_RESCHED bit as an inverted NEED_RESCHED such @@ -26,25 +23,25 @@ */ static __always_inline int preempt_count(void) { - unsigned long lc_preempt, count; + unsigned long lc_preempt; + int count; =20 - BUILD_BUG_ON(sizeof_field(struct lowcore, preempt_count) !=3D sizeof(int)= ); - lc_preempt =3D offsetof(struct lowcore, preempt_count); + lc_preempt =3D offsetof(struct lowcore, preempt.count); /* READ_ONCE(get_lowcore()->preempt_count) & ~PREEMPT_NEED_RESCHED */ asm_inline( - ALTERNATIVE("llgt %[count],%[offzero](%%r0)\n", - "llgt %[count],%[offalt](%%r0)\n", + ALTERNATIVE("ly %[count],%[offzero](%%r0)\n", + "ly %[count],%[offalt](%%r0)\n", ALT_FEATURE(MFEATURE_LOWCORE)) : [count] "=3Dd" (count) : [offzero] "i" (lc_preempt), [offalt] "i" (lc_preempt + LOWCORE_ALT_ADDRESS), - "m" (((struct lowcore *)0)->preempt_count)); + "m" (((struct lowcore *)0)->preempt.count)); return count; } =20 -static __always_inline void preempt_count_set(int pc) +static __always_inline void preempt_count_set(unsigned long pc) { - int old, new; + unsigned long old, new; =20 old =3D READ_ONCE(get_lowcore()->preempt_count); do { @@ -63,12 +60,12 @@ static __always_inline void preempt_count_set(int pc) =20 static __always_inline void set_preempt_need_resched(void) { - __atomic_and(~PREEMPT_NEED_RESCHED, &get_lowcore()->preempt_count); + __atomic64_and(~PREEMPT_NEED_RESCHED, (long *)&get_lowcore()->preempt_cou= nt); } =20 static __always_inline void clear_preempt_need_resched(void) { - __atomic_or(PREEMPT_NEED_RESCHED, &get_lowcore()->preempt_count); + __atomic64_or(PREEMPT_NEED_RESCHED, (long *)&get_lowcore()->preempt_count= ); } =20 static __always_inline bool test_preempt_need_resched(void) @@ -88,8 +85,8 @@ static __always_inline void __preempt_count_add(int val) =20 lc_preempt =3D offsetof(struct lowcore, preempt_count); asm_inline( - ALTERNATIVE("asi %[offzero](%%r0),%[val]\n", - "asi %[offalt](%%r0),%[val]\n", + ALTERNATIVE("agsi %[offzero](%%r0),%[val]\n", + "agsi %[offalt](%%r0),%[val]\n", ALT_FEATURE(MFEATURE_LOWCORE)) : "+m" (((struct lowcore *)0)->preempt_count) : [offzero] "i" (lc_preempt), [val] "i" (val), @@ -98,7 +95,7 @@ static __always_inline void __preempt_count_add(int val) return; } } - __atomic_add(val, &get_lowcore()->preempt_count); + __atomic64_add(val, (long *)&get_lowcore()->preempt_count); } =20 static __always_inline void __preempt_count_sub(int val) @@ -119,15 +116,15 @@ static __always_inline bool __preempt_count_dec_and_t= est(void) =20 lc_preempt =3D offsetof(struct lowcore, preempt_count); asm_inline( - ALTERNATIVE("alsi %[offzero](%%r0),%[val]\n", - "alsi %[offalt](%%r0),%[val]\n", + ALTERNATIVE("algsi %[offzero](%%r0),%[val]\n", + "algsi %[offalt](%%r0),%[val]\n", ALT_FEATURE(MFEATURE_LOWCORE)) : "=3D@cc" (cc), "+m" (((struct lowcore *)0)->preempt_count) : [offzero] "i" (lc_preempt), [val] "i" (-1), [offalt] "i" (lc_preempt + LOWCORE_ALT_ADDRESS)); return (cc =3D=3D 0) || (cc =3D=3D 2); #else - return __atomic_add_const_and_test(-1, &get_lowcore()->preempt_count); + return __atomic64_add_const_and_test(-1, (long *)&get_lowcore()->preempt_= count); #endif } =20 @@ -141,7 +138,7 @@ static __always_inline bool should_resched(int preempt_= offset) =20 static __always_inline int __preempt_count_add_return(int val) { - return val + __atomic_add(val, &get_lowcore()->preempt_count); + return val + __atomic64_add(val, (long *)&get_lowcore()->preempt_count); } =20 static __always_inline int __preempt_count_sub_return(int val) --=20 2.51.0 From nobody Mon Jun 8 07:22:05 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7201E3FE35D; Fri, 5 Jun 2026 05:41:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638120; cv=none; b=gnsrG90ZH5toBD+E7wmtY4uHgYPADcgaj+dtCWfLYMxCzBU7YM5YXV8Kgtlv9EA8uEmFdaVVUlvX//LDr5VE4SKH9+lt55fy+dmDYalB7lbNZdYTsl8JxN51W7Ifd0319ZY3dWBkgkklNAgHWE7AIZIBjOhoZwwuBAlin8VV0p8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780638120; c=relaxed/simple; bh=ub2dNJq/xnlK3TL0MDbTdj+ww2iW+ojYYwWRDfK7gu0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dG2N+vBV54rwYJZF/eJKF9OW846n6AxI7L2U8+dAMped+NVk891BCEFJrer6YWHlw1bIQVbBmwpHcnRI39+hHn/TVyv+/iHRKvnt/nPK7IgISjwc+3O1KGws8pT6ufaWFfEgDn+mT74Omz0W1voayuo9EqrlAFkOJ+7SlhPVqoE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bt1Gv9AW; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bt1Gv9AW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 96BA51F0089C; Fri, 5 Jun 2026 05:41:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780638118; bh=tXrxowM9l6DqFYeAjn5Tac1f8S3PU9cy43ibbb4VImQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=bt1Gv9AW2oKS0LY5l6jXhoKt+z+vO3BpaR1T6SbnU+f0deSZ1ABjn9VvZBuuJI/Cj HJkpmzNSK1/QRjbKldJrwhKNOI3XCUlCM4XxVKbeXIia/Z9WCEa10/TH3SRx1S3/ji M3/YFEI2Et/Ost9/EUjQjNPOjR/1RFvoToXJBCMe/UvBXmucwpkv6udMQCtVJTfPbL qIIQ0e+RlihB6+GWfIIWHdAxl8mp85f4D7Li8TAfS5xGp/lm6fGH9xAWTlLvUB33JI guYe2w4tTcB3TNLQRwwQuiccm//hNTjzk3tcXsdy3RTAQ7oMgfOR2lLu90bbAsoDVY NIlGOpngZ1iAQ== Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfauth.phl.internal (Postfix) with ESMTP id E0BD1F40068; Fri, 5 Jun 2026 01:41:55 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Fri, 05 Jun 2026 01:41:55 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFG0WJa+84B5ZLakFCJusLrHx60u/M2e4tgb8vvSWGQWHXMFZeKktHow9VrFaQhwL 60vkPLe/GHUXrF9uQjn9EtWhYCiYebO343SXKkauObGNd0gaHEDMGyAUdMEWI3xvV5+MXM Y5XIGIH+RCj/YpjPsc3XHQUzqyjdf8JDGbXE7uRDsVlh2adsQWoVoEG+11Li6tmpxBjo+n GbGHXxcMsG5BC+v/PWXz65OFiJmnJTn18gjrdwmPsAQA0I/XEj3HJhBv36WGSvCgORqSG4 pkDfFOinWXk05Px2n3ZYSDFusuNrnah7Cqk2jZsViR99U7a3ccatK3SGLf/zDxH9xSS8qX d7jFqhlw8739C7P+Vw4YAKRdjftKM5qwXrp10RhjCbtlXYkXUtg/lihoMG8MroPCLFkMdh yBe76zLchQusN2z3yRtFCMERRJAag5Jqf4Cx3wSwUn+pSj2N/WkzFAkCBVqPuzAnHg+6En f9+adxIwdJJiICLDMxNucNU0d9RCsLswbbtCnlG10BOkgHUhO0qe2BLEUccUrAE9JIiFSH ScggFvzBTtTSM3RlvjRzv4BQE2D+Kfb6Vn940IvcY/mrtjs4rqDKUt2ulWpgZ0B1nIqMbW RJO95aqzPjRtgDkK0WoypiOUWop4aiyV2dWzyenVoeC5GH1Kw4m/qn4gPcaw X-ME-Proxy: Feedback-ID: i8dbe485b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 5 Jun 2026 01:41:55 -0400 (EDT) From: Boqun Feng To: Peter Zijlstra Cc: Catalin Marinas , Will Deacon , Jonas Bonn , Stefan Kristiansson , Stafford Horne , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Boqun Feng , Waiman Long , Andrew Morton , Andrii Nakryiko , Eduard Zingerman , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Kumar Kartikeya Dwivedi , Song Liu , Yonghong Song , Jiri Olsa , Shuah Khan , Miguel Ojeda , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Jinjie Ruan , Lyude Paul , Thomas Huth , Sohil Mehta , Pawan Gupta , Sean Christopherson , Nikunj A Dadhania , "Xin Li (Intel)" , Joel Fernandes , Andy Shevchenko , Randy Dunlap , Yury Norov , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v3 13/13] irq: Optimize reschedule check in local_interrupt_enable() Date: Thu, 4 Jun 2026 22:41:28 -0700 Message-ID: <20260605054128.5925-14-boqun@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260605054128.5925-1-boqun@kernel.org> References: <20260605054128.5925-1-boqun@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In local_interrupt_enable(), we could avoid re-reading preempt count because of should_resched() by using the result from hardirq_disable_exit(), however this means __preempt_count_add_return() and __preempt_count_sub_return() need to return all the preempt count bits (including the PREEMPT_NEED_RESCHED bit), since the only user of __preempt_count_{add,sub}_return() is hardirq_disable_{enter,exit}(), hence make them return "unsigned long" to optimize this. Signed-off-by: Boqun Feng --- arch/arm64/include/asm/preempt.h | 12 ++++++------ arch/s390/include/asm/preempt.h | 4 ++-- arch/x86/include/asm/preempt.h | 4 ++-- include/asm-generic/preempt.h | 4 ++-- include/linux/interrupt_rc.h | 31 ++++++++++++++++++++----------- 5 files changed, 32 insertions(+), 23 deletions(-) diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/pree= mpt.h index 0dd8221d1bef..e9f597d87413 100644 --- a/arch/arm64/include/asm/preempt.h +++ b/arch/arm64/include/asm/preempt.h @@ -55,20 +55,20 @@ static inline void __preempt_count_sub(int val) WRITE_ONCE(current_thread_info()->preempt.count, pc); } =20 -static inline int __preempt_count_add_return(int val) +static inline unsigned long __preempt_count_add_return(int val) { - u32 pc =3D READ_ONCE(current_thread_info()->preempt.count); + u64 pc =3D READ_ONCE(current_thread_info()->preempt_count); pc +=3D val; - WRITE_ONCE(current_thread_info()->preempt.count, pc); + WRITE_ONCE(current_thread_info()->preempt_count, pc); =20 return pc; } =20 -static inline int __preempt_count_sub_return(int val) +static inline unsigned long __preempt_count_sub_return(int val) { - u32 pc =3D READ_ONCE(current_thread_info()->preempt.count); + u64 pc =3D READ_ONCE(current_thread_info()->preempt_count); pc -=3D val; - WRITE_ONCE(current_thread_info()->preempt.count, pc); + WRITE_ONCE(current_thread_info()->preempt_count, pc); =20 return pc; } diff --git a/arch/s390/include/asm/preempt.h b/arch/s390/include/asm/preemp= t.h index 1d5e4d7e9e1b..d0021b979a5d 100644 --- a/arch/s390/include/asm/preempt.h +++ b/arch/s390/include/asm/preempt.h @@ -136,12 +136,12 @@ static __always_inline bool should_resched(int preemp= t_offset) return unlikely(READ_ONCE(get_lowcore()->preempt_count) =3D=3D preempt_of= fset); } =20 -static __always_inline int __preempt_count_add_return(int val) +static __always_inline unsigned long __preempt_count_add_return(int val) { return val + __atomic64_add(val, (long *)&get_lowcore()->preempt_count); } =20 -static __always_inline int __preempt_count_sub_return(int val) +static __always_inline unsigned long __preempt_count_sub_return(int val) { return __preempt_count_add_return(-val); } diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h index 12353eeebc52..fc1a2799990a 100644 --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -103,12 +103,12 @@ static __always_inline void __preempt_count_sub(int v= al) __pc_op(add, __preempt_count, -val); } =20 -static __always_inline int __preempt_count_add_return(int val) +static __always_inline unsigned long __preempt_count_add_return(int val) { return __pc_op(add_return, __preempt_count, val); } =20 -static __always_inline int __preempt_count_sub_return(int val) +static __always_inline unsigned long __preempt_count_sub_return(int val) { return __pc_op(add_return, __preempt_count, -val); } diff --git a/include/asm-generic/preempt.h b/include/asm-generic/preempt.h index c8683c046615..7629e23102d1 100644 --- a/include/asm-generic/preempt.h +++ b/include/asm-generic/preempt.h @@ -59,14 +59,14 @@ static __always_inline void __preempt_count_sub(int val) *preempt_count_ptr() -=3D val; } =20 -static __always_inline int __preempt_count_add_return(int val) +static __always_inline unsigned long __preempt_count_add_return(int val) { *preempt_count_ptr() +=3D val; =20 return *preempt_count_ptr(); } =20 -static __always_inline int __preempt_count_sub_return(int val) +static __always_inline unsigned long __preempt_count_sub_return(int val) { *preempt_count_ptr() -=3D val; =20 diff --git a/include/linux/interrupt_rc.h b/include/linux/interrupt_rc.h index dd4444c61330..c044dc395452 100644 --- a/include/linux/interrupt_rc.h +++ b/include/linux/interrupt_rc.h @@ -27,7 +27,7 @@ DECLARE_PER_CPU(struct interrupt_disable_state, local_int= errupt_disable_state); static inline void local_interrupt_disable(void) { unsigned long flags; - int new_count; + unsigned long new_count; =20 WARN_ON_ONCE(in_nmi()); =20 @@ -41,9 +41,25 @@ static inline void local_interrupt_disable(void) } } =20 +#ifdef CONFIG_PREEMPTION +static inline void local_interrupt_enable_reched(unsigned long pc) +{ + if (pc) + return; + /* No PREEMPT_NEED_RESCHED bit? Check tif_need_resched() */ +#ifndef PREEMPT_NEED_RESCHED + if (!tif_need_resched()) + return; +#endif + __preempt_schedule(); +} +#else +static inline void local_interrupt_enable_reched(unsigned long pc) {} +#endif + static inline void local_interrupt_enable(void) { - int new_count; + unsigned long new_count; =20 new_count =3D hardirq_disable_exit(); =20 @@ -52,15 +68,8 @@ static inline void local_interrupt_enable(void) =20 flags =3D raw_cpu_read(local_interrupt_disable_state.flags); local_irq_restore(flags); - /* - * TODO: re-read preempt count can be avoided, but it needs - * should_resched() taking another parameter as the current - * preempt count - */ -#ifdef CONFIG_PREEMPTION - if (should_resched(0)) - __preempt_schedule(); -#endif + + local_interrupt_enable_reched(new_count); } } =20 --=20 2.51.0