From nobody Sat Feb 7 08:02:38 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 493283A89C7 for ; Wed, 21 Jan 2026 22:40:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769035209; cv=none; b=UuB/X1KxYcF9lqFR24jrczC4Js5PaRAZCz5Zq2LOGO22NRpU+6r8uTSefF7gK+UA+tyNZ7e96k2GuQY3BuRG9Thbz1meHpgdxUh9u7w/FDy8UnHeoafl+ugS4Es5mJKvmpux2dxdaLnPutQdFrKMxK4BooQmOZ5+x1dFhVSCkoQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769035209; c=relaxed/simple; bh=Hv8poHE2O98fYOp3Y68wbGNXSFxcm3ueSfXe3mNmI4U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SagP2yFi6t5joa13hedpRw/Kc1KCTqSkQNRaVH/sufZ5B7RsmLLYoGOdUqWG/YuGkUfCNgBfCj4xuHxuznMf/u05MiHDXceJmTX7YTixjmKKDtLhJvudwJPhxrwNWFIU21fh4gT9uh1WvoFXlrvGL9wVoDqSK1Uu6mLix4da3WU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Wr/Soe/v; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Wr/Soe/v" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1769035205; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t/2vrXZGCGDvtwhDve6dSbvbtmIASyHWERC1j3nx7E8=; b=Wr/Soe/vfgvfnbcv9cT/Hh8MHm9zuwHky7bmQtTY+WR0MsE9WgRZjvkYPCPT08lv5NMivi 1SkVgoZl07IFjtrMNGtM1iuiyIjgCeLTG0iPdPehFeaAnlGu5tT4bNLA8Rmr6ASJ3OMrnZ EkvvL1ICBir/5Mm4Xfd7S69JpGBSFow= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-283-2u-boV-bMjWxWwGPJ0Wlkw-1; Wed, 21 Jan 2026 17:40:02 -0500 X-MC-Unique: 2u-boV-bMjWxWwGPJ0Wlkw-1 X-Mimecast-MFC-AGG-ID: 2u-boV-bMjWxWwGPJ0Wlkw_1769035199 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4A99C180057E; Wed, 21 Jan 2026 22:39:59 +0000 (UTC) Received: from GoldenWind.redhat.com (unknown [10.22.89.232]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 730711955F43; Wed, 21 Jan 2026 22:39:56 +0000 (UTC) From: Lyude Paul To: rust-for-linux@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner Cc: Boqun Feng , Daniel Almeida , Miguel Ojeda , Alex Gaynor , Gary Guo , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Andrew Morton , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Joel Fernandes Subject: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter Date: Wed, 21 Jan 2026 17:39:05 -0500 Message-ID: <20260121223933.1568682-3-lyude@redhat.com> In-Reply-To: <20260121223933.1568682-1-lyude@redhat.com> References: <20260121223933.1568682-1-lyude@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Content-Type: text/plain; charset="utf-8" From: Joel Fernandes Move NMI nesting tracking from the preempt_count bits to a separate per-CPU counter (nmi_nesting). This is to free up the NMI bits in the preempt_count, allowing those bits to be repurposed for other uses. This also has the ben= efit of tracking more than 16-levels deep if there is ever a need. Reduce multiple bits in preempt_count for NMI tracking. Reduce NMI_BITS from 3 to 1, using it only to detect if we're in an NMI. Suggested-by: Boqun Feng Signed-off-by: Joel Fernandes Signed-off-by: Lyude Paul --- include/linux/hardirq.h | 16 ++++++++++++---- include/linux/preempt.h | 13 +++++++++---- kernel/softirq.c | 2 ++ 3 files changed, 23 insertions(+), 8 deletions(-) diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index d57cab4d4c06f..cc06bda52c3e5 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -10,6 +10,8 @@ #include #include =20 +DECLARE_PER_CPU(unsigned int, nmi_nesting); + extern void synchronize_irq(unsigned int irq); extern bool synchronize_hardirq(unsigned int irq); =20 @@ -102,14 +104,16 @@ void irq_exit_rcu(void); */ =20 /* - * nmi_enter() can nest up to 15 times; see NMI_BITS. + * nmi_enter() can nest - nesting is tracked in a per-CPU counter. */ #define __nmi_enter() \ do { \ lockdep_off(); \ arch_nmi_enter(); \ - BUG_ON(in_nmi() =3D=3D NMI_MASK); \ - __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); \ + BUG_ON(__this_cpu_read(nmi_nesting) =3D=3D UINT_MAX); \ + __this_cpu_inc(nmi_nesting); \ + __preempt_count_add(HARDIRQ_OFFSET); \ + preempt_count_set(preempt_count() | NMI_MASK); \ } while (0) =20 #define nmi_enter() \ @@ -124,8 +128,12 @@ void irq_exit_rcu(void); =20 #define __nmi_exit() \ do { \ + unsigned int nesting; \ BUG_ON(!in_nmi()); \ - __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ + __preempt_count_sub(HARDIRQ_OFFSET); \ + nesting =3D __this_cpu_dec_return(nmi_nesting); \ + if (!nesting) \ + __preempt_count_sub(NMI_OFFSET); \ arch_nmi_exit(); \ lockdep_on(); \ } while (0) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index f07e7f37f3ca5..e2d3079d3f5f1 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -18,6 +18,8 @@ * - bits 0-7 are the preemption count (max preemption depth: 256) * - bits 8-15 are the softirq count (max # of softirqs: 256) * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 2= 56) + * - bits 24-27 are the hardirq count (max # of hardirqs: 16) + * - bit 28 is the NMI flag (no nesting count, tracked separately) * * The hardirq count could in theory be the same as the number of * interrupts in the system, but we run all interrupt handlers with @@ -25,18 +27,21 @@ * there are a few palaeontologic drivers which reenable interrupts in * the handler, so we need more than one bit here. * + * NMI nesting depth is tracked in a separate per-CPU variable + * (nmi_nesting) to save bits in preempt_count. + * * PREEMPT_MASK: 0x000000ff * SOFTIRQ_MASK: 0x0000ff00 * HARDIRQ_DISABLE_MASK: 0x00ff0000 - * HARDIRQ_MASK: 0x07000000 - * NMI_MASK: 0x38000000 + * HARDIRQ_MASK: 0x0f000000 + * NMI_MASK: 0x10000000 * PREEMPT_NEED_RESCHED: 0x80000000 */ #define PREEMPT_BITS 8 #define SOFTIRQ_BITS 8 #define HARDIRQ_DISABLE_BITS 8 -#define HARDIRQ_BITS 3 -#define NMI_BITS 3 +#define HARDIRQ_BITS 4 +#define NMI_BITS 1 =20 #define PREEMPT_SHIFT 0 #define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) diff --git a/kernel/softirq.c b/kernel/softirq.c index 77198911b8dd4..af47ea23aba3b 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); #endif =20 +DEFINE_PER_CPU(unsigned int, nmi_nesting); + /* * SOFTIRQ_OFFSET usage: * --=20 2.52.0