From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 595B83B6BF3 for ; Tue, 7 Apr 2026 13:17:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567844; cv=none; b=f7zY11aJZxJPs5aYK9qrLO8QOgmM4lppXhk5ZRQGWcR8ZewjaR5EyA7WVt0vE96b8f0Xjd3tq125LCJjzl9YIfDP0pl4UNNXGSuiJBRLiTQc/i30OP7uiZdbH3GtyQxxAmdOW2iIpe0U9zRMkzCrVmMTCkF/KQ5XsAWnwFoUlLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567844; c=relaxed/simple; bh=8P9f8BbJ2TRd1Q/adLEhuGEvf+j1QF04uAmpliD5V8U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gf03/9tMPEQhpSEIbukgAmKqiRmSjBc2DWPDt5eojQL5l2TLHSTJe4a0Yd26N2QzOFHCY1vfGcDArN7z9GUhVS9VwKHCfg5xt1CjbakqeP9uH8x1pdoba6sN50nfVLi1wiuilEprodM0dfPj5Pg8tlkIGw8nHdYT8KtHMycQG3w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=asCl5hyw; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="asCl5hyw" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 00E8F16F2; Tue, 7 Apr 2026 06:17:16 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C882F3F7D8; Tue, 7 Apr 2026 06:17:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567841; bh=8P9f8BbJ2TRd1Q/adLEhuGEvf+j1QF04uAmpliD5V8U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=asCl5hywD4UTawl26PJNL8OISldIxFq9PKTK+1bH5O+42KMAHPQRmGq9PmlOSTGT3 l5NLMY76ixkydF2jxP9ZLe6a0KkH2PhzKXET29m0hudTtY1U6qWAZM+V+HbWeS2uzC KAAmoGa5uhOFBY7SHg8S1+8R32XR7E3kg4x7sPMU= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Andy Lutomirski , Peter Zijlstra , Thomas Gleixner Cc: ada.coupriediaz@arm.com, catalin.marinas@arm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org Subject: [PATCH 01/10] entry: Fix stale comment for irqentry_enter() Date: Tue, 7 Apr 2026 14:16:41 +0100 Message-Id: <20260407131650.3813777-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The kerneldoc comment for irqentry_enter() refers to idtentry_exit(), which is an accidental holdover from the x86 entry code that the generic irqentry code was based on. Correct this to refer to irqentry_exit(). Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- include/linux/irq-entry-common.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index d26d1b1bcbfb9..3cf4d21168ba1 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -394,7 +394,7 @@ typedef struct irqentry_state { * establish the proper context for NOHZ_FULL. Otherwise scheduling on exit * would not be possible. * - * Returns: An opaque object that must be passed to idtentry_exit() + * Returns: An opaque object that must be passed to irqentry_exit() */ irqentry_state_t noinstr irqentry_enter(struct pt_regs *regs); =20 --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 555DB393DEB for ; Tue, 7 Apr 2026 13:17:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567851; cv=none; b=P1MZKoqshTYc4l7VRtjxJwiqhT4ofDs8UMWN7RgFDxXGtfNAQNlAozM4eafFIeXQdM07RbnO0dXgALGKf/4HuFs639VAw+Jza2D1RWoPk8iCcNNBAw2vMLBCwkAYT1Bnb8YiLL0WhzSeK+zyvHdXVpYiUMs/u4hDahJBjgEkkBw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567851; c=relaxed/simple; bh=PnpEGCqDI+7mLHxmW6SNuY7AJrvNh7KFQZ2iYdYu5IY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ma+Oo8R3jN117RsW2IBudO80cFG3HcHqzRw7V6RhJrCM/5lNhhKYVywvS9HLBcgwwt4ORrRJDTql9acCzWQCqW6o/dvvEyjXDZ80FWQjT6Bk8UFQr2RdnhOF1H+tkT+Uf0nlfL0MKLc2ghzm/3AXF7LKmpsehg322wW9aH0bEBY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=goIwi3pN; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="goIwi3pN" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EB7C416F2; Tue, 7 Apr 2026 06:17:23 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C8ACF3F7D8; Tue, 7 Apr 2026 06:17:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567849; bh=PnpEGCqDI+7mLHxmW6SNuY7AJrvNh7KFQZ2iYdYu5IY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=goIwi3pNbLHft1v0o6PDcVBwKRIElAkF/C2v+8yrw7o/8jllrfZHizN6sqfFhCav5 6CKmK0/PJr5biaroT/Rz1K3DdzCFJ4d7p3+6qMCpawcGZ5py0FpOJcXoBV9SGO05ls stXyhRh4t2GbbffxdusJ8XhjZLojzHgURuG2HaAk= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Andy Lutomirski , Peter Zijlstra , Thomas Gleixner Cc: ada.coupriediaz@arm.com, catalin.marinas@arm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org Subject: [PATCH 02/10] entry: Remove local_irq_{enable,disable}_exit_to_user() Date: Tue, 7 Apr 2026 14:16:42 +0100 Message-Id: <20260407131650.3813777-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The local_irq_enable_exit_to_user() and local_irq_disable_exit_to_user() functions are never overridden by architecture code, and are always equivalent to local_irq_enable() and local_irq_disable(). These functions were added on the assumption that arm64 would override them to manage 'DAIF' exception masking, as described by Thomas Gleixner in these threads: https://lore.kernel.org/all/20190919150809.340471236@linutronix.de/ https://lore.kernel.org/all/alpine.DEB.2.21.1910240119090.1852@nanos.tec.= linutronix.de/ In practice arm64 did not need to override either. Prior to moving to the generic irqentry code, arm64's management of DAIF was reworked in commit: 97d935faacde ("arm64: Unmask Debug + SError in do_notify_resume()") Since that commit, arm64 only masks interrupts during the 'prepare' step when returning to user mode, and masks other DAIF exceptions later. Within arm64_exit_to_user_mode(), the arm64 entry code is as follows: local_irq_disable(); exit_to_user_mode_prepare_legacy(regs); local_daif_mask(); mte_check_tfsr_exit(); exit_to_user_mode(); Remove the unnecessary local_irq_enable_exit_to_user() and local_irq_disable_exit_to_user() functions. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- include/linux/entry-common.h | 2 +- include/linux/irq-entry-common.h | 31 ------------------------------- kernel/entry/common.c | 4 ++-- 3 files changed, 3 insertions(+), 34 deletions(-) diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index f83ca0abf2cdb..dbaa153100f44 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -321,7 +321,7 @@ static __always_inline void syscall_exit_to_user_mode(s= truct pt_regs *regs) { instrumentation_begin(); syscall_exit_to_user_mode_work(regs); - local_irq_disable_exit_to_user(); + local_irq_disable(); syscall_exit_to_user_mode_prepare(regs); instrumentation_end(); exit_to_user_mode(); diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 3cf4d21168ba1..93b4b551f7ae4 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -100,37 +100,6 @@ static __always_inline void enter_from_user_mode(struc= t pt_regs *regs) instrumentation_end(); } =20 -/** - * local_irq_enable_exit_to_user - Exit to user variant of local_irq_enabl= e() - * @ti_work: Cached TIF flags gathered with interrupts disabled - * - * Defaults to local_irq_enable(). Can be supplied by architecture specific - * code. - */ -static inline void local_irq_enable_exit_to_user(unsigned long ti_work); - -#ifndef local_irq_enable_exit_to_user -static __always_inline void local_irq_enable_exit_to_user(unsigned long ti= _work) -{ - local_irq_enable(); -} -#endif - -/** - * local_irq_disable_exit_to_user - Exit to user variant of local_irq_disa= ble() - * - * Defaults to local_irq_disable(). Can be supplied by architecture specif= ic - * code. - */ -static inline void local_irq_disable_exit_to_user(void); - -#ifndef local_irq_disable_exit_to_user -static __always_inline void local_irq_disable_exit_to_user(void) -{ - local_irq_disable(); -} -#endif - /** * arch_exit_to_user_mode_work - Architecture specific TIF work for exit * to user mode. diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 9ef63e4147913..b5e05d87ba391 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -47,7 +47,7 @@ static __always_inline unsigned long __exit_to_user_mode_= loop(struct pt_regs *re */ while (ti_work & EXIT_TO_USER_MODE_WORK_LOOP) { =20 - local_irq_enable_exit_to_user(ti_work); + local_irq_enable(); =20 if (ti_work & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY)) { if (!rseq_grant_slice_extension(ti_work & TIF_SLICE_EXT_DENY)) @@ -74,7 +74,7 @@ static __always_inline unsigned long __exit_to_user_mode_= loop(struct pt_regs *re * might have changed while interrupts and preemption was * enabled above. */ - local_irq_disable_exit_to_user(); + local_irq_disable(); =20 /* Check if any of the above work has queued a deferred wakeup */ tick_nohz_user_enter_prepare(); --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C7A7725CC79 for ; Tue, 7 Apr 2026 13:17:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567859; cv=none; b=sipDmRQB4bSFCUB7tB10kRF0GWUzdM07MseEo6slkgNOp6sMO9ii2obcRKWf17/xwuV1IUe/Jc79mHh17UwD3oPn6YGLGmXiXHrV/Ibci3nx3GOh88Z/Ivef07Vnc/91Vj6Cn05Vp73Y6DjIk4O8sZvSUmiXzuKOGCGmP9djOLM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567859; c=relaxed/simple; bh=nL/6sqoqt3SzzXYp2uRKEzMSNyRXDO4GvLQ6+fh1NXs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IomzCH+iua7F9Kk16cSo/b4932zF9iRU4PXZvycMAfjoWueqe5ABhCb/cuQMAxadJNob5KgJwnFwNmlHh9DuzxSSa5OUnnVxtVse9Qtc3VH5o33Btep0nWbISiX2UY7TuYlHz0TJmcOLEG5t/IHyhVbToUJeEpMSORNp15cHj3s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=f+ZY3Za1; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="f+ZY3Za1" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7490016F2; Tue, 7 Apr 2026 06:17:31 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 558443F7D8; Tue, 7 Apr 2026 06:17:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567857; bh=nL/6sqoqt3SzzXYp2uRKEzMSNyRXDO4GvLQ6+fh1NXs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f+ZY3Za17PzlohERV7xh8kVTf7+6EzO3xlpw2XHLlc9c5YTUBHPex11JMbub22PvS y6mA4hK/QVjHe47Vh6hZD/9bIn1oIeuPRxUVQwIkZUI3RhSPD0Al5OI70lN/Y9DYye qEdBaaJdWlKcHpiq5sn5F85z1N4Ce8Z8xp9vVOGk= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Andy Lutomirski , Peter Zijlstra , Thomas Gleixner Cc: ada.coupriediaz@arm.com, catalin.marinas@arm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org Subject: [PATCH 03/10] entry: Move irqentry_enter() prototype later Date: Tue, 7 Apr 2026 14:16:43 +0100 Message-Id: <20260407131650.3813777-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Subsequent patches will rework the irqentry_*() functions. The end result (and the intermediate diffs) will be much clearer if the prototype for the irqentry_enter() function is moved later, immediately before the prototype of the irqentry_exit() function. Move the prototype later. This is purely a move; there should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- include/linux/irq-entry-common.h | 44 ++++++++++++++++---------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 93b4b551f7ae4..d1e8591a59195 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -334,6 +334,28 @@ typedef struct irqentry_state { } irqentry_state_t; #endif =20 +/** + * irqentry_exit_cond_resched - Conditionally reschedule on return from in= terrupt + * + * Conditional reschedule with additional sanity checks. + */ +void raw_irqentry_exit_cond_resched(void); + +#ifdef CONFIG_PREEMPT_DYNAMIC +#if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) +#define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_= resched +#define irqentry_exit_cond_resched_dynamic_disabled NULL +DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_res= ched); +#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resche= d)() +#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) +DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); +void dynamic_irqentry_exit_cond_resched(void); +#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched() +#endif +#else /* CONFIG_PREEMPT_DYNAMIC */ +#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched() +#endif /* CONFIG_PREEMPT_DYNAMIC */ + /** * irqentry_enter - Handle state tracking on ordinary interrupt entries * @regs: Pointer to pt_regs of interrupted context @@ -367,28 +389,6 @@ typedef struct irqentry_state { */ irqentry_state_t noinstr irqentry_enter(struct pt_regs *regs); =20 -/** - * irqentry_exit_cond_resched - Conditionally reschedule on return from in= terrupt - * - * Conditional reschedule with additional sanity checks. - */ -void raw_irqentry_exit_cond_resched(void); - -#ifdef CONFIG_PREEMPT_DYNAMIC -#if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) -#define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_= resched -#define irqentry_exit_cond_resched_dynamic_disabled NULL -DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_res= ched); -#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resche= d)() -#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) -DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); -void dynamic_irqentry_exit_cond_resched(void); -#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched() -#endif -#else /* CONFIG_PREEMPT_DYNAMIC */ -#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched() -#endif /* CONFIG_PREEMPT_DYNAMIC */ - /** * irqentry_exit - Handle return from exception that used irqentry_enter() * @regs: Pointer to pt_regs (exception entry regs) --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D43F7399359 for ; Tue, 7 Apr 2026 13:17:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567864; cv=none; b=cfgfmKQfc6T0YSbSG/hB6pAS6g6yyA21IJgM0vgc4o8YGGkhhfQqYifnCH2+752s4Gptl0wGAJYPs6WAnt0dVeK6xTfY7d2x0xDvGNUQ+IKBTBcQVLWe66fRDm6ItOanxKhPGdHwP1Hl79+Fi0xMVnAmaoCL9leE6NnI2k5Fjpw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567864; c=relaxed/simple; bh=A2D6lxL6Il0+BUIp51/50fVPIMF1mAhdZSwxbep0s0A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=s7VHzO4MGeFIzoXwX9MZsOtUGA1kTCX2cfJIFgVxig9N4TP1VT0Nxk01aJaJi0HPV1kaaTHJyDKSX/txrs9nKz1g4wcwonApmmoAt44iUaCpZAZEYk9Y93fswG53sPFykKST7pi10lygdIQ8Zf3Natz4ea7BWf8Rp17CEZedNUQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=eVwhDsGq; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="eVwhDsGq" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 842BC16F2; Tue, 7 Apr 2026 06:17:36 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 62E0A3F7D8; Tue, 7 Apr 2026 06:17:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567862; bh=A2D6lxL6Il0+BUIp51/50fVPIMF1mAhdZSwxbep0s0A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eVwhDsGqS69W8pyM5O2Irxf3e+RSC1VmIe6Qpvxlpls5xl3IutTi9qO84rCJbqmB9 TYf59sOhZX5hYSLbiPCI22HTxv/gGabXgva3U5rC6981gflphFbwp+pqAmNSPwfuUd sPEtQFvUq/V8wqJXLNdRevCJ4w26ssEX3MKKccxU= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Andy Lutomirski , Peter Zijlstra , Thomas Gleixner Cc: ada.coupriediaz@arm.com, catalin.marinas@arm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org Subject: [PATCH 04/10] entry: Split kernel mode logic from irqentry_{enter,exit}() Date: Tue, 7 Apr 2026 14:16:44 +0100 Message-Id: <20260407131650.3813777-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generic irqentry code has entry/exit functions specifically for exceptions taken from user mode, but doesn't have entry/exit functions specifically for exceptions taken from kernel mode. It would be helpful to have separate entry/exit functions specifically for exceptions taken from kernel mode. This would make the structure of the entry code more consistent, and would make it easier for architectures to manage logic specific to exceptions taken from kernel mode. Move the logic specific to kernel mode out of irqentry_enter() and irqentry_exit() into new irqentry_enter_from_kernel_mode() and irqentry_exit_to_kernel_mode() functions. These are marked __always_inline and placed in irq-entry-common.h, as with irqentry_enter_from_user_mode() and irqentry_exit_to_user_mode(), so that they can be inlined into architecture-specific wrappers. The existing out-of-line irqentry_enter() and irqentry_exit() functions retained as callers of the new functions. The lockdep assertion from irqentry_exit() is moved into irqentry_exit_to_user_mode() and irqentry_exit_to_kernel_mode(). This was previously missing from irqentry_exit_to_user_mode() when called directly, and any new lockdep assertion failure relating from this change is a latent bug. Aside from the lockdep change noted above, there should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- include/linux/irq-entry-common.h | 103 +++++++++++++++++++++++++++++++ kernel/entry/common.c | 103 +++---------------------------- 2 files changed, 111 insertions(+), 95 deletions(-) Thomas/Peter/Andy, as mentioned on IRC, I haven't created kerneldoc comments for these new functions because the existing comments don't seem all that consistent (e.g. for user mode vs kernel mode), and I suspect we want to rewrite them all in one go for wider consistency. I'm happy to respin this, or to follow-up with that as per your preference. Mark. diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index d1e8591a59195..2206150e526d8 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -304,6 +304,8 @@ static __always_inline void irqentry_enter_from_user_mo= de(struct pt_regs *regs) */ static __always_inline void irqentry_exit_to_user_mode(struct pt_regs *reg= s) { + lockdep_assert_irqs_disabled(); + instrumentation_begin(); irqentry_exit_to_user_mode_prepare(regs); instrumentation_end(); @@ -356,6 +358,107 @@ void dynamic_irqentry_exit_cond_resched(void); #define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched() #endif /* CONFIG_PREEMPT_DYNAMIC */ =20 +static __always_inline irqentry_state_t irqentry_enter_from_kernel_mode(st= ruct pt_regs *regs) +{ + irqentry_state_t ret =3D { + .exit_rcu =3D false, + }; + + /* + * If this entry hit the idle task invoke ct_irq_enter() whether + * RCU is watching or not. + * + * Interrupts can nest when the first interrupt invokes softirq + * processing on return which enables interrupts. + * + * Scheduler ticks in the idle task can mark quiescent state and + * terminate a grace period, if and only if the timer interrupt is + * not nested into another interrupt. + * + * Checking for rcu_is_watching() here would prevent the nesting + * interrupt to invoke ct_irq_enter(). If that nested interrupt is + * the tick then rcu_flavor_sched_clock_irq() would wrongfully + * assume that it is the first interrupt and eventually claim + * quiescent state and end grace periods prematurely. + * + * Unconditionally invoke ct_irq_enter() so RCU state stays + * consistent. + * + * TINY_RCU does not support EQS, so let the compiler eliminate + * this part when enabled. + */ + if (!IS_ENABLED(CONFIG_TINY_RCU) && + (is_idle_task(current) || arch_in_rcu_eqs())) { + /* + * If RCU is not watching then the same careful + * sequence vs. lockdep and tracing is required + * as in irqentry_enter_from_user_mode(). + */ + lockdep_hardirqs_off(CALLER_ADDR0); + ct_irq_enter(); + instrumentation_begin(); + kmsan_unpoison_entry_regs(regs); + trace_hardirqs_off_finish(); + instrumentation_end(); + + ret.exit_rcu =3D true; + return ret; + } + + /* + * If RCU is watching then RCU only wants to check whether it needs + * to restart the tick in NOHZ mode. rcu_irq_enter_check_tick() + * already contains a warning when RCU is not watching, so no point + * in having another one here. + */ + lockdep_hardirqs_off(CALLER_ADDR0); + instrumentation_begin(); + kmsan_unpoison_entry_regs(regs); + rcu_irq_enter_check_tick(); + trace_hardirqs_off_finish(); + instrumentation_end(); + + return ret; +} + +static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *r= egs, irqentry_state_t state) +{ + lockdep_assert_irqs_disabled(); + + if (!regs_irqs_disabled(regs)) { + /* + * If RCU was not watching on entry this needs to be done + * carefully and needs the same ordering of lockdep/tracing + * and RCU as the return to user mode path. + */ + if (state.exit_rcu) { + instrumentation_begin(); + /* Tell the tracer that IRET will enable interrupts */ + trace_hardirqs_on_prepare(); + lockdep_hardirqs_on_prepare(); + instrumentation_end(); + ct_irq_exit(); + lockdep_hardirqs_on(CALLER_ADDR0); + return; + } + + instrumentation_begin(); + if (IS_ENABLED(CONFIG_PREEMPTION)) + irqentry_exit_cond_resched(); + + /* Covers both tracing and lockdep */ + trace_hardirqs_on(); + instrumentation_end(); + } else { + /* + * IRQ flags state is correct already. Just tell RCU if it + * was not watching on entry. + */ + if (state.exit_rcu) + ct_irq_exit(); + } +} + /** * irqentry_enter - Handle state tracking on ordinary interrupt entries * @regs: Pointer to pt_regs of interrupted context diff --git a/kernel/entry/common.c b/kernel/entry/common.c index b5e05d87ba391..1034be02eae84 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -105,70 +105,16 @@ __always_inline unsigned long exit_to_user_mode_loop(= struct pt_regs *regs, =20 noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) { - irqentry_state_t ret =3D { - .exit_rcu =3D false, - }; - if (user_mode(regs)) { - irqentry_enter_from_user_mode(regs); - return ret; - } + irqentry_state_t ret =3D { + .exit_rcu =3D false, + }; =20 - /* - * If this entry hit the idle task invoke ct_irq_enter() whether - * RCU is watching or not. - * - * Interrupts can nest when the first interrupt invokes softirq - * processing on return which enables interrupts. - * - * Scheduler ticks in the idle task can mark quiescent state and - * terminate a grace period, if and only if the timer interrupt is - * not nested into another interrupt. - * - * Checking for rcu_is_watching() here would prevent the nesting - * interrupt to invoke ct_irq_enter(). If that nested interrupt is - * the tick then rcu_flavor_sched_clock_irq() would wrongfully - * assume that it is the first interrupt and eventually claim - * quiescent state and end grace periods prematurely. - * - * Unconditionally invoke ct_irq_enter() so RCU state stays - * consistent. - * - * TINY_RCU does not support EQS, so let the compiler eliminate - * this part when enabled. - */ - if (!IS_ENABLED(CONFIG_TINY_RCU) && - (is_idle_task(current) || arch_in_rcu_eqs())) { - /* - * If RCU is not watching then the same careful - * sequence vs. lockdep and tracing is required - * as in irqentry_enter_from_user_mode(). - */ - lockdep_hardirqs_off(CALLER_ADDR0); - ct_irq_enter(); - instrumentation_begin(); - kmsan_unpoison_entry_regs(regs); - trace_hardirqs_off_finish(); - instrumentation_end(); - - ret.exit_rcu =3D true; + irqentry_enter_from_user_mode(regs); return ret; } =20 - /* - * If RCU is watching then RCU only wants to check whether it needs - * to restart the tick in NOHZ mode. rcu_irq_enter_check_tick() - * already contains a warning when RCU is not watching, so no point - * in having another one here. - */ - lockdep_hardirqs_off(CALLER_ADDR0); - instrumentation_begin(); - kmsan_unpoison_entry_regs(regs); - rcu_irq_enter_check_tick(); - trace_hardirqs_off_finish(); - instrumentation_end(); - - return ret; + return irqentry_enter_from_kernel_mode(regs); } =20 /** @@ -212,43 +158,10 @@ void dynamic_irqentry_exit_cond_resched(void) =20 noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) { - lockdep_assert_irqs_disabled(); - - /* Check whether this returns to user mode */ - if (user_mode(regs)) { + if (user_mode(regs)) irqentry_exit_to_user_mode(regs); - } else if (!regs_irqs_disabled(regs)) { - /* - * If RCU was not watching on entry this needs to be done - * carefully and needs the same ordering of lockdep/tracing - * and RCU as the return to user mode path. - */ - if (state.exit_rcu) { - instrumentation_begin(); - /* Tell the tracer that IRET will enable interrupts */ - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(); - instrumentation_end(); - ct_irq_exit(); - lockdep_hardirqs_on(CALLER_ADDR0); - return; - } - - instrumentation_begin(); - if (IS_ENABLED(CONFIG_PREEMPTION)) - irqentry_exit_cond_resched(); - - /* Covers both tracing and lockdep */ - trace_hardirqs_on(); - instrumentation_end(); - } else { - /* - * IRQ flags state is correct already. Just tell RCU if it - * was not watching on entry. - */ - if (state.exit_rcu) - ct_irq_exit(); - } + else + irqentry_exit_to_kernel_mode(regs, state); } =20 irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs) --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7A8A53B6BF3 for ; Tue, 7 Apr 2026 13:17:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567878; cv=none; b=PTi/880UFYYfSnYbCqneFzIqrAmRCXdKvTqgyIIYAw3V5k6ML1mK+MyjP4/9OdAVaTrMPiA/LHKv5FLQpRgFiPJJIud2b3iQFVEzxPeYBZr+rxmN83sSdssTWKr9O9uicCcAQ8wrQr98chtF5KAnHzN6VMPYaJbV6290dVHO7v8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567878; c=relaxed/simple; bh=tUnltGjcHlvxDk6vVHLGfWgevDbKvhXeMPsvKLPUGi0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LOH2mdpnKH9ym3UppNuw6H13eFI+4/LferjzN7g9i9XSteexww0zziOO+d54tcoqrpr+ujKwEPhYp2MpbE/nIDJPE02zdVhnDsjWZi1OqKfAj8vRPtvRYiGHoKYWuVH2iH9Rn9tlEPo1KrAExWounvnNK7Aa3NMyLzg2J0P0Wj0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=i1GCfrQy; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="i1GCfrQy" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0EACC1BB2; Tue, 7 Apr 2026 06:17:50 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DFFA93F7D8; Tue, 7 Apr 2026 06:17:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567875; bh=tUnltGjcHlvxDk6vVHLGfWgevDbKvhXeMPsvKLPUGi0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i1GCfrQypMEjFouN11jFhy570D5SApBrIpY3oo1ghHbwpi+3FQik0WHbrIoeg4PIy oEy/KcYEqwQtdxl6bbDdhdZ7wsLZvdU4F1CPmrkySts9U6HeKxlfprLXKBbwmurcy+ QYODOJQ13glCj+K16sN6NsPbo3CZqejEPGl8FIig= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Andy Lutomirski , Peter Zijlstra , Thomas Gleixner Cc: ada.coupriediaz@arm.com, catalin.marinas@arm.com, linux-kernel@vger.kernel.org, mark.rutland@arm.com, ruanjinjie@huawei.com, vladimir.murzin@arm.com, will@kernel.org Subject: [PATCH 05/10] entry: Split preemption from irqentry_exit_to_kernel_mode() Date: Tue, 7 Apr 2026 14:16:45 +0100 Message-Id: <20260407131650.3813777-6-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some architecture-specific work needs to be performed between the state management for exception entry/exit and the "real" work to handle the exception. For example, arm64 needs to manipulate a number of exception masking bits, with different exceptions requiring different masking. Generally this can all be hidden in the architecture code, but for arm64 the current structure of irqentry_exit_to_kernel_mode() makes this particularly difficult to handle in a way that is correct, maintainable, and efficient. The gory details are described in the thread surrounding: https://lore.kernel.org/lkml/acPAzdtjK5w-rNqC@J2N7QTR9R3/ The summary is: * Currently, irqentry_exit_to_kernel_mode() handles both involuntary preemption AND state management necessary for exception return. * When scheduling (including involuntary preemption), arm64 needs to have all arm64-specific exceptions unmasked, though regular interrupts must be masked. * Prior to the state management for exception return, arm64 needs to mask a number of arm64-specific exceptions, and perform some work with these exceptions masked (with RCU watching, etc). While in theory it is possible to handle this with a new arch_*() hook called somewhere under irqentry_exit_to_kernel_mode(), this is fragile and complicated, and doesn't match the flow used for exception return to user mode, which has a separate 'prepare' step (where preemption can occur) prior to the state management. To solve this, refactor irqentry_exit_to_kernel_mode() to match the style of {irqentry,syscall}_exit_to_user_mode(), moving preemption logic into a new irqentry_exit_to_kernel_mode_preempt() function, and moving state management in a new irqentry_exit_to_kernel_mode_after_preempt() function. The existing irqentry_exit_to_kernel_mode() is left as a caller of both of these, avoiding the need to modify existing callers. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- include/linux/irq-entry-common.h | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) Thomas/Peter/Andy, as mentioned on IRC, I haven't created kerneldoc comments for these new functions because the existing comments don't seem all that consistent (e.g. for user mode vs kernel mode), and I suspect we want to rewrite them all in one go for wider consistency. I'm happy to respin this, or to follow-up with that as per your preference. Mark. diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 2206150e526d8..24830baa539c6 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -421,10 +421,18 @@ static __always_inline irqentry_state_t irqentry_ente= r_from_kernel_mode(struct p return ret; } =20 -static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *r= egs, irqentry_state_t state) +static inline void irqentry_exit_to_kernel_mode_preempt(struct pt_regs *re= gs, irqentry_state_t state) { - lockdep_assert_irqs_disabled(); + if (regs_irqs_disabled(regs) || state.exit_rcu) + return; + + if (IS_ENABLED(CONFIG_PREEMPTION)) + irqentry_exit_cond_resched(); +} =20 +static __always_inline void +irqentry_exit_to_kernel_mode_after_preempt(struct pt_regs *regs, irqentry_= state_t state) +{ if (!regs_irqs_disabled(regs)) { /* * If RCU was not watching on entry this needs to be done @@ -443,9 +451,6 @@ static __always_inline void irqentry_exit_to_kernel_mod= e(struct pt_regs *regs, i } =20 instrumentation_begin(); - if (IS_ENABLED(CONFIG_PREEMPTION)) - irqentry_exit_cond_resched(); - /* Covers both tracing and lockdep */ trace_hardirqs_on(); instrumentation_end(); @@ -459,6 +464,17 @@ static __always_inline void irqentry_exit_to_kernel_mo= de(struct pt_regs *regs, i } } =20 +static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *r= egs, irqentry_state_t state) +{ + lockdep_assert_irqs_disabled(); + + instrumentation_begin(); + irqentry_exit_to_kernel_mode_preempt(regs, state); + instrumentation_end(); + + irqentry_exit_to_kernel_mode_after_preempt(regs, state); +} + /** * irqentry_enter - Handle state tracking on ordinary interrupt entries * @regs: Pointer to pt_regs of interrupted context --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 775323B6374 for ; Tue, 7 Apr 2026 13:18:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567899; cv=none; b=lEl7X/K9O3n+XDjtnb494fRXr3f2x6+6zlkPu+VCieLG9LbAzIwUnJaMesgaYGJDT9t0Nppsyu0C4urmjRwzMtwzxBFw0Ku/Lw6A8AhTmVLLA52IJnKzwKHcQ+ZESdSuBwRXYvYu9JH5Cmwmul+pzhPnvpmYLMcvo4AMXUBpMuI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567899; c=relaxed/simple; bh=ZuXrXFNPUNGWF9OTidkPCfi5CGz2rFMvnd5pwEhF87U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jYxBF8abXUx1ACSdQR4+Y7dq43LIqA9Xzqd+S7nXOckEhyDJYpu6Bcn1/7lDmOB8MGeq4EC0D9lnJMm4IKkt018Qq1xBdrjBUo1wSIei5SJGWvTS6iYGFa+Yk4iIn45LhatVWwV0yEGSNVX8ZTyP1jIu19VGAXOms1bbYfwasOk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=H+dvome6; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="H+dvome6" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 186411BB2; Tue, 7 Apr 2026 06:18:12 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E9B053F7D8; Tue, 7 Apr 2026 06:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567897; bh=ZuXrXFNPUNGWF9OTidkPCfi5CGz2rFMvnd5pwEhF87U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H+dvome6yqpWMyHqY5q4pRDb/hQF+E9tkIEsDP0Sd8/vXrXubScsuI8sAQjwVmlVu 490VQjLTasSXJBfw5nba6wFFp5yafMYxCvnGcmTq8434hgnGytaYZ6TU/dvIucxvfH +RbPscCRYsScpYATQ22jyqRox4UAuxXX6jKgdy6Y= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon Cc: ada.coupriediaz@arm.com, linux-kernel@vger.kernel.org, luto@kernel.org, mark.rutland@arm.com, peterz@infradead.org, ruanjinjie@huawei.com, tglx@kernel.org, vladimir.murzin@arm.com Subject: [PATCH 06/10] arm64: entry: Don't preempt with SError or Debug masked Date: Tue, 7 Apr 2026 14:16:46 +0100 Message-Id: <20260407131650.3813777-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On arm64, involuntary kernel preemption has been subtly broken since the move to the generic irqentry code. When preemption occurs, the new task may run with SError and Debug exceptions masked unexpectedly, leading to a loss of RAS events, breakpoints, watchpoints, and single-step exceptions. Prior to moving to the generic irqentry code, involuntary preemption of kernel mode would only occur when returning from regular interrupts, in a state where interrupts were masked and all other arm64-specific exceptions (SError, Debug, and pseudo-NMI) were unmasked. This is the only state in which it is valid to switch tasks. As part of moving to the generic irqentry code, the involuntary preemption logic was moved such that involuntary preemption could occur when returning from any (non-NMI) exception. As most exception handlers mask all arm64-specific exceptions before this point, preemption could occur in a state where arm64-specific exceptions were masked. This is not a valid state to switch tasks, and resulted in the loss of exceptions described above. As a temporary bodge, avoid the loss of exceptions by avoiding involuntary preemption when SError and/or Debug exceptions are masked. Practically speaking this means that involuntary preemption will only occur when returning from regular interrupts, as was the case before moving to the generic irqentry code. Fixes: 99eb057ccd67 ("arm64: entry: Move arm64_preempt_schedule_irq() into = __exit_to_kernel_mode()") Reported-by: Ada Couprie Diaz Reported-by: Vladimir Murzin Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- arch/arm64/include/asm/entry-common.h | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm= /entry-common.h index cab8cd78f6938..20f0a7c7bde15 100644 --- a/arch/arm64/include/asm/entry-common.h +++ b/arch/arm64/include/asm/entry-common.h @@ -29,14 +29,19 @@ static __always_inline void arch_exit_to_user_mode_work= (struct pt_regs *regs, =20 static inline bool arch_irqentry_exit_need_resched(void) { - /* - * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC - * priority masking is used the GIC irqchip driver will clear DAIF.IF - * using gic_arch_enable_irqs() for normal IRQs. If anything is set in - * DAIF we must have handled an NMI, so skip preemption. - */ - if (system_uses_irq_prio_masking() && read_sysreg(daif)) - return false; + if (system_uses_irq_prio_masking()) { + /* + * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC + * priority masking is used the GIC irqchip driver will clear DAIF.IF + * using gic_arch_enable_irqs() for normal IRQs. If anything is set in + * DAIF we must have handled an NMI, so skip preemption. + */ + if (read_sysreg(daif)) + return false; + } else { + if (read_sysreg(daif) & (PSR_D_BIT | PSR_A_BIT)) + return false; + } =20 /* * Preempting a task from an IRQ means we leave copies of PSTATE --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CE1DC3B7746 for ; Tue, 7 Apr 2026 13:18:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567920; cv=none; b=EufHbtMwVwy11OcBk2ZzO3IIP4RXQW2fD+ErEM+iCpNXJoTpTIK9XCDF7BxLemXahCuyBcXW0ZFk0b/Q2u09IZn9NxrUOZh2wDqzNzux9n9JPgbXKTB+Vsic3nKk+BMffBBZ9ZVQ83joetBgvc1Ehkz19JX1+PwL8O5fRB1smPg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567920; c=relaxed/simple; bh=460FgsMVIrnBcOu9FijuyxZpOYyi6/RG8mHA95P9+Io=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SEnybusn2+xAhL+eEBg4np6UJCDQ8PcSlO1rj8wDncQ+1PV59r/2SjGDYXpYShYMhTh7Sl8ACSKYlAuyFW7TvV0elzddo9zyMK8mjokiA4I8WuJsesPrwP/ivdgIfR2mmlHNhU8S8rs3+9YS8xI+99/6ia9sTcBbTE6Qnf6vXgE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=gA7xhDQq; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="gA7xhDQq" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7CCEC1BB2; Tue, 7 Apr 2026 06:18:31 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5C2093F7D8; Tue, 7 Apr 2026 06:18:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567917; bh=460FgsMVIrnBcOu9FijuyxZpOYyi6/RG8mHA95P9+Io=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gA7xhDQqj9L8+kOdQ/cZfGURpSwMsEo4+Mcz5SwFjo0kDIGGcHFNJKLyIHpPBR7wW DyRM4Wu7cliyr6q9WKab9xiAkOm20AxOmLryXsoO5AwpZKkGCgLHjYRfGMZDzVZZk3 yKfzKwKvtOWSu2TwFhzYKB64uvWDKU3RE7taYTCk= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon Cc: ada.coupriediaz@arm.com, linux-kernel@vger.kernel.org, luto@kernel.org, mark.rutland@arm.com, peterz@infradead.org, ruanjinjie@huawei.com, tglx@kernel.org, vladimir.murzin@arm.com Subject: [PATCH 07/10] arm64: entry: Consistently prefix arm64-specific wrappers Date: Tue, 7 Apr 2026 14:16:47 +0100 Message-Id: <20260407131650.3813777-8-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For historical reasons, arm64's entry code has arm64-specific functions named enter_from_kernel_mode() and exit_to_kernel_mode(), which are wrappers for similarly-named functions from the generic irqentry code. Other arm64-specific wrappers have an 'arm64_' prefix to clearly distinguish them from their generic counterparts, e.g. arm64_enter_from_user_mode() and arm64_exit_to_user_mode(). For consistency and clarity, add an 'arm64_' prefix to these functions. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- arch/arm64/kernel/entry-common.c | 38 ++++++++++++++++---------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-com= mon.c index 3625797e9ee8f..3d01cdacdc7a2 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -35,7 +35,7 @@ * Before this function is called it is not safe to call regular kernel co= de, * instrumentable code, or any code which may trigger an exception. */ -static noinstr irqentry_state_t enter_from_kernel_mode(struct pt_regs *reg= s) +static noinstr irqentry_state_t arm64_enter_from_kernel_mode(struct pt_reg= s *regs) { irqentry_state_t state; =20 @@ -51,8 +51,8 @@ static noinstr irqentry_state_t enter_from_kernel_mode(st= ruct pt_regs *regs) * After this function returns it is not safe to call regular kernel code, * instrumentable code, or any code which may trigger an exception. */ -static void noinstr exit_to_kernel_mode(struct pt_regs *regs, - irqentry_state_t state) +static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs, + irqentry_state_t state) { mte_check_tfsr_exit(); irqentry_exit(regs, state); @@ -298,11 +298,11 @@ static void noinstr el1_abort(struct pt_regs *regs, u= nsigned long esr) unsigned long far =3D read_sysreg(far_el1); irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_mem_abort(far, esr, regs); local_daif_mask(); - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } =20 static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr) @@ -310,55 +310,55 @@ static void noinstr el1_pc(struct pt_regs *regs, unsi= gned long esr) unsigned long far =3D read_sysreg(far_el1); irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_sp_pc_abort(far, esr, regs); local_daif_mask(); - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } =20 static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr) { irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_undef(regs, esr); local_daif_mask(); - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } =20 static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr) { irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_bti(regs, esr); local_daif_mask(); - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } =20 static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr) { irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_gcs(regs, esr); local_daif_mask(); - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } =20 static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr) { irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_mops(regs, esr); local_daif_mask(); - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } =20 static void noinstr el1_breakpt(struct pt_regs *regs, unsigned long esr) @@ -420,11 +420,11 @@ static void noinstr el1_fpac(struct pt_regs *regs, un= signed long esr) { irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_fpac(regs, esr); local_daif_mask(); - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } =20 asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) @@ -491,13 +491,13 @@ static __always_inline void __el1_irq(struct pt_regs = *regs, { irqentry_state_t state; =20 - state =3D enter_from_kernel_mode(regs); + state =3D arm64_enter_from_kernel_mode(regs); =20 irq_enter_rcu(); do_interrupt_handler(regs, handler); irq_exit_rcu(); =20 - exit_to_kernel_mode(regs, state); + arm64_exit_to_kernel_mode(regs, state); } static void noinstr el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 978353B6368 for ; Tue, 7 Apr 2026 13:18:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567928; cv=none; b=KuJ7B7KMcc5X2QNPN+W85gQGkO1q1EDiV4BQwF/Mx46BNE4q8XyXyVDorZvuaOF89wAdwcDI1u4sea+oaSA/Frc33SKV23SliAKRswqbOymXoimX4D0TxVz/8CgubaD/6GQADC6+xxycwSjcycmQepnf5tUTtZNJgU0MYOLhvgA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567928; c=relaxed/simple; bh=i+c6ra3/8cZfXlo5XIHBX2j6T0++N9EkiOZ/PzP/xNw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hjnHOG4GQWoznibUZei8rv98Li1UJyA8XrfyY5nRxr8uxeZhwxz2Pfig3QI6vkU5xcz2PVtcHHgV+pRwxtSXEabJZXA8jb7ZqtzwVjG5zxa1x+0OTX3QrQEl1DQU4HDnBJDui+SKrrcQN8JgrsG15p9QQwpzcsHwqJ0K4zLJkiU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=OshVwYxT; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="OshVwYxT" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4572A1BB2; Tue, 7 Apr 2026 06:18:40 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2299C3F7D8; Tue, 7 Apr 2026 06:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567926; bh=i+c6ra3/8cZfXlo5XIHBX2j6T0++N9EkiOZ/PzP/xNw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OshVwYxTFGTFGSawa5bRFmJFGOmv6oo9ggoXahGU2HPlPekNW42uksSRVLBHgshN8 qhZpaqGD1kCOGOzH4tumL/B5WziD4rsw2/JXT82fF7Q8r43PJjDMf9hYWNsdIEA2kC lj69fBtqTtoymQnq/o18Itvj/tC4do9loaIRqu0I= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon Cc: ada.coupriediaz@arm.com, linux-kernel@vger.kernel.org, luto@kernel.org, mark.rutland@arm.com, peterz@infradead.org, ruanjinjie@huawei.com, tglx@kernel.org, vladimir.murzin@arm.com Subject: [PATCH 08/10] arm64: entry: Use irqentry_{enter_from,exit_to}_kernel_mode() Date: Tue, 7 Apr 2026 14:16:48 +0100 Message-Id: <20260407131650.3813777-9-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generic irqentry code now provides irqentry_enter_from_kernel_mode() and irqentry_exit_to_kernel_mode(), which can be used when an exception is known to be taken from kernel mode. These can be inlined into architecture-specific entry code, and avoid redundant work to test whether the exception was taken from user mode. Use these in arm64_enter_from_kernel_mode() and arm64_exit_to_kernel_mode(), which are only used for exceptions known to be taken from kernel mode. This will remove a small amount of redundant work, and will permit further changes to arm64_exit_to_kernel_mode() in subsequent patches. There should be no funcitonal change as a result of this patch. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- arch/arm64/kernel/entry-common.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-com= mon.c index 3d01cdacdc7a2..16a65987a6a9b 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -39,7 +39,7 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_m= ode(struct pt_regs *reg { irqentry_state_t state; =20 - state =3D irqentry_enter(regs); + state =3D irqentry_enter_from_kernel_mode(regs); mte_check_tfsr_entry(); mte_disable_tco_entry(current); =20 @@ -55,7 +55,7 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_r= egs *regs, irqentry_state_t state) { mte_check_tfsr_exit(); - irqentry_exit(regs, state); + irqentry_exit_to_kernel_mode(regs, state); } =20 /* --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 622533B5825 for ; Tue, 7 Apr 2026 13:18:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567937; cv=none; b=ilkei/A9BxP5GzhxezhmnkIMzMstacPGJXw4tHLVvm4NRCwXTvbEVQjFKe+rPw/uIpcTm18Ps+Q3+cmfxMjAU/GrqX4kKUYHJf0LTmgOpD/MSVN/rw6kBVTOqXa+1j8WPLw0BZxtg+RJL+Q3cOZjor/tAIpX145euerZyfPF8YM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567937; c=relaxed/simple; bh=2GspNv828Xl2KX6qp8neNpclH69k3zwgwYawgy5AC/k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pjsBIWqZXDAeegMGLcraIKu6Q5I7v1BwJwORoA0s3JZLjkFihhYDLXHIJVRfUfoOw5ZnrUxnkx4kDe9raleFr+BcThZ+ICZcG5jrvfidDnmzMp7qKV+cxI8DPjxTBsFaiYNZT3962lmkqj2H6EoDamc2S4DkwwSpD+DWrXPwYbA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=bmZlTDyE; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="bmZlTDyE" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 142061BB2; Tue, 7 Apr 2026 06:18:49 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E87773F7D8; Tue, 7 Apr 2026 06:18:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567934; bh=2GspNv828Xl2KX6qp8neNpclH69k3zwgwYawgy5AC/k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bmZlTDyERLi4Xu/Ur7JYtQNwtgWONgcg1HR2cHS0Pv2toiJ4i+q3TU3LC67YbRsMG dZLCOI+C42wZbrNhYG0PF6j2+nApBR1+ZhEGuovcHqLYkNwoWkmG1O0Pvifqv3Giyt Dg9vh+AW4sFwXkbtKHlXOkI+/c1av6MBiF0sicf8= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon Cc: ada.coupriediaz@arm.com, linux-kernel@vger.kernel.org, luto@kernel.org, mark.rutland@arm.com, peterz@infradead.org, ruanjinjie@huawei.com, tglx@kernel.org, vladimir.murzin@arm.com Subject: [PATCH 09/10] arm64: entry: Use split preemption logic Date: Tue, 7 Apr 2026 14:16:49 +0100 Message-Id: <20260407131650.3813777-10-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generic irqentry code now provides irqentry_exit_to_kernel_mode_preempt() and irqentry_exit_to_kernel_mode_after_preempt(), which can be used where architectures have different state requirements for involuntary preemption and exception return, as is the case on arm64. Use the new functions on arm64, aligning our exit to kernel mode logic with the style of our exit to user mode logic. This removes the need for the recently-added bodge in arch_irqentry_exit_need_resched(), and allows preemption to occur when returning from any exception taken from kernel mode, which is nicer for RT. In an ideal world, we'd remove arch_irqentry_exit_need_resched(), and fold the conditionality directly into the architecture-specific entry code. That way all the logic necessary to avoid preempting from a pseudo-NMI could be constrained specifically to the EL1 IRQ/FIQ paths, avoiding redundant work for other exceptions, and making the flow a bit clearer. At present it looks like that would require a larger refactoring (e.g. for the PREEMPT_DYNAMIC logic), and so I've left that as-is for now. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) Reviewed-by: Jinjie Ruan --- arch/arm64/include/asm/entry-common.h | 21 ++++++++------------- arch/arm64/kernel/entry-common.c | 12 ++++-------- 2 files changed, 12 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm= /entry-common.h index 20f0a7c7bde15..cab8cd78f6938 100644 --- a/arch/arm64/include/asm/entry-common.h +++ b/arch/arm64/include/asm/entry-common.h @@ -29,19 +29,14 @@ static __always_inline void arch_exit_to_user_mode_work= (struct pt_regs *regs, =20 static inline bool arch_irqentry_exit_need_resched(void) { - if (system_uses_irq_prio_masking()) { - /* - * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC - * priority masking is used the GIC irqchip driver will clear DAIF.IF - * using gic_arch_enable_irqs() for normal IRQs. If anything is set in - * DAIF we must have handled an NMI, so skip preemption. - */ - if (read_sysreg(daif)) - return false; - } else { - if (read_sysreg(daif) & (PSR_D_BIT | PSR_A_BIT)) - return false; - } + /* + * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC + * priority masking is used the GIC irqchip driver will clear DAIF.IF + * using gic_arch_enable_irqs() for normal IRQs. If anything is set in + * DAIF we must have handled an NMI, so skip preemption. + */ + if (system_uses_irq_prio_masking() && read_sysreg(daif)) + return false; =20 /* * Preempting a task from an IRQ means we leave copies of PSTATE diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-com= mon.c index 16a65987a6a9b..f42ce7b5c67f3 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -54,8 +54,11 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_= mode(struct pt_regs *reg static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs, irqentry_state_t state) { + local_irq_disable(); + irqentry_exit_to_kernel_mode_preempt(regs, state); + local_daif_mask(); mte_check_tfsr_exit(); - irqentry_exit_to_kernel_mode(regs, state); + irqentry_exit_to_kernel_mode_after_preempt(regs, state); } =20 /* @@ -301,7 +304,6 @@ static void noinstr el1_abort(struct pt_regs *regs, uns= igned long esr) state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_mem_abort(far, esr, regs); - local_daif_mask(); arm64_exit_to_kernel_mode(regs, state); } =20 @@ -313,7 +315,6 @@ static void noinstr el1_pc(struct pt_regs *regs, unsign= ed long esr) state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_sp_pc_abort(far, esr, regs); - local_daif_mask(); arm64_exit_to_kernel_mode(regs, state); } =20 @@ -324,7 +325,6 @@ static void noinstr el1_undef(struct pt_regs *regs, uns= igned long esr) state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_undef(regs, esr); - local_daif_mask(); arm64_exit_to_kernel_mode(regs, state); } =20 @@ -335,7 +335,6 @@ static void noinstr el1_bti(struct pt_regs *regs, unsig= ned long esr) state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_bti(regs, esr); - local_daif_mask(); arm64_exit_to_kernel_mode(regs, state); } =20 @@ -346,7 +345,6 @@ static void noinstr el1_gcs(struct pt_regs *regs, unsig= ned long esr) state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_gcs(regs, esr); - local_daif_mask(); arm64_exit_to_kernel_mode(regs, state); } =20 @@ -357,7 +355,6 @@ static void noinstr el1_mops(struct pt_regs *regs, unsi= gned long esr) state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_mops(regs, esr); - local_daif_mask(); arm64_exit_to_kernel_mode(regs, state); } =20 @@ -423,7 +420,6 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsi= gned long esr) state =3D arm64_enter_from_kernel_mode(regs); local_daif_inherit(regs); do_el1_fpac(regs, esr); - local_daif_mask(); arm64_exit_to_kernel_mode(regs, state); } =20 --=20 2.30.2 From nobody Sun Jun 21 04:20:35 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1C90D3382E8 for ; Tue, 7 Apr 2026 13:19:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567943; cv=none; b=m+M53+Unym5Zh0oyzepKpjS92skumxgpbER5eA8GDx09MTrStVGRqrnsm03/QPWU4SWyyLreKU0KqsAjLxPLysX59SBzqKgEnQZKlMYwz+zIGwnNORyKn/Bq0HGnM9QNOj1oz9Eg/HkqpvTqGHyD01bUamrmBhG7KVLoxesliSQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775567943; c=relaxed/simple; bh=IyTqocNhQbVO47j3rlqKeekBesuxoAkVgsNZyMigNLA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nXaDHMuIMcUNGIOCBXQ4uJJJ7ZvwyanCuOi0plrs0Q+C36z6IgPylFZanwzTbnMpjqAJbTEaAABTB92W54C+Uuf9mK6g1OJCEmA2RBaNfZducF2Q4bGTsXD6elctMh17BUbiX4FjDgQT57bjVFemXyM90m+ix4CwvHqtHW6rv10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=pCEi4gQ3; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="pCEi4gQ3" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AD79D1BB2; Tue, 7 Apr 2026 06:18:54 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 88BBA3F7D8; Tue, 7 Apr 2026 06:18:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1775567940; bh=IyTqocNhQbVO47j3rlqKeekBesuxoAkVgsNZyMigNLA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pCEi4gQ3nYGEvolTIcNKqrTlxR2q65E9GRSNuHbNRH4Gqj6fzxfvq5RbbOrOolA2q Et9UYSGZ+rdg8FKv5nHIZQJOnl8w6513BlXuO6BQoIqrNJ+Rsqo9sswWFDc81f5uhk gu6PLCqgsB6sXUvYTAuzCLof2iEEBdKgTmcF/coU= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon Cc: ada.coupriediaz@arm.com, linux-kernel@vger.kernel.org, luto@kernel.org, mark.rutland@arm.com, peterz@infradead.org, ruanjinjie@huawei.com, tglx@kernel.org, vladimir.murzin@arm.com Subject: [PATCH 10/10] arm64: Check DAIF (and PMR) at task-switch time Date: Tue, 7 Apr 2026 14:16:50 +0100 Message-Id: <20260407131650.3813777-11-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260407131650.3813777-1-mark.rutland@arm.com> References: <20260407131650.3813777-1-mark.rutland@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When __switch_to() switches from a 'prev' task to a 'next' task, various pieces of CPU state are expected to have specific values, such that these do not need to be saved/restored. If any of these hold an unexpected value when switching away from the prev task, they could lead to surprising behaviour in the context of the next task, and it would be difficult to determine where they were configured to their unexpected value. Add some checks for DAIF and PMR at task-switch time so that we can detect such issues. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Catalin Marinas Cc: Jinjie Ruan Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vladimir Murzin Cc: Will Deacon Acked-by: Peter Zijlstra (Intel) --- arch/arm64/kernel/process.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 489554931231e..ba9038434d2fb 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -699,6 +699,29 @@ void update_sctlr_el1(u64 sctlr) isb(); } =20 +static inline void debug_switch_state(void) +{ + if (system_uses_irq_prio_masking()) { + unsigned long daif_expected =3D 0; + unsigned long daif_actual =3D read_sysreg(daif); + unsigned long pmr_expected =3D GIC_PRIO_IRQOFF; + unsigned long pmr_actual =3D read_sysreg_s(SYS_ICC_PMR_EL1); + + WARN_ONCE(daif_actual !=3D daif_expected || + pmr_actual !=3D pmr_expected, + "Unexpected DAIF + PMR: 0x%lx + 0x%lx (expected 0x%lx + 0x%lx)\n", + daif_actual, pmr_actual, + daif_expected, pmr_expected); + } else { + unsigned long daif_expected =3D DAIF_PROCCTX_NOIRQ; + unsigned long daif_actual =3D read_sysreg(daif); + + WARN_ONCE(daif_actual !=3D daif_expected, + "Unexpected DAIF value: 0x%lx (expected 0x%lx)\n", + daif_actual, daif_expected); + } +} + /* * Thread switching. */ @@ -708,6 +731,8 @@ struct task_struct *__switch_to(struct task_struct *pre= v, { struct task_struct *last; =20 + debug_switch_state(); + fpsimd_thread_switch(next); tls_thread_switch(next); hw_breakpoint_thread_switch(next); --=20 2.30.2