From nobody Mon Jun 22 13:22:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A0F6C433EF for ; Wed, 23 Mar 2022 15:33:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245199AbiCWPej (ORCPT ); Wed, 23 Mar 2022 11:34:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245187AbiCWPef (ORCPT ); Wed, 23 Mar 2022 11:34:35 -0400 Received: from out0.migadu.com (out0.migadu.com [94.23.1.103]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BD30330 for ; Wed, 23 Mar 2022 08:33:03 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1648049582; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Mmc3nZQPUIWzZbIw9R7zdg/6fIrR3qfP7vluF20CZjk=; b=UTJfa80A9sjfbSyh7XvFmbslezbBssH3pgpUdBaLBgs3IL2A3I6NA415UycL5yEGhbwHb6 vpUhIr+s+DTQPNB7OYobvxHrvMaOkJFmdVvDpJbnJ21kD5Gc+xjflW3QFSbiifKEwW887J jc+p/iTWkpBgyYYx+H4uOeaKNyAMx8U= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Catalin Marinas , Will Deacon , Andrew Morton Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Mark Rutland , Vincenzo Frascino , Sami Tolvanen , Peter Collingbourne , Evgenii Stepanov , Florian Mayer , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 1/4] stacktrace: add interface based on shadow call stack Date: Wed, 23 Mar 2022 16:32:52 +0100 Message-Id: <21e3e20ea58e242e3c82c19abbfe65b579e0e4b8.1648049113.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Andrey Konovalov Add a new interface stack_trace_save_shadow() for collecting stack traces by copying frames from the Shadow Call Stack. Collecting stack traces this way is significantly faster: boot time of a defconfig build with KASAN enabled gets descreased by ~30%. The few patches following this one add an implementation of stack_trace_save_shadow() for arm64. The implementation of the added interface is not meant to use stack_trace_consume_fn to avoid making a function call for each collected frame to further improve performance. Signed-off-by: Andrey Konovalov --- arch/Kconfig | 6 ++++++ include/linux/stacktrace.h | 15 +++++++++++++++ kernel/stacktrace.c | 21 +++++++++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index e12a4268c01d..207c1679c53a 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1041,6 +1041,12 @@ config HAVE_RELIABLE_STACKTRACE arch_stack_walk_reliable() function which only returns a stack trace if it can guarantee the trace is reliable. =20 +config HAVE_SHADOW_STACKTRACE + bool + help + If this is set, the architecture provides the arch_stack_walk_shadow() + function, which collects the stack trace from the shadow call stack. + config HAVE_ARCH_HASH bool default n diff --git a/include/linux/stacktrace.h b/include/linux/stacktrace.h index 97455880ac41..b74d1e42e157 100644 --- a/include/linux/stacktrace.h +++ b/include/linux/stacktrace.h @@ -60,6 +60,9 @@ int arch_stack_walk_reliable(stack_trace_consume_fn consu= me_entry, void *cookie, =20 void arch_stack_walk_user(stack_trace_consume_fn consume_entry, void *cook= ie, const struct pt_regs *regs); + +int arch_stack_walk_shadow(unsigned long *store, unsigned int size, + unsigned int skipnr); #endif /* CONFIG_ARCH_STACKWALK */ =20 #ifdef CONFIG_STACKTRACE @@ -108,4 +111,16 @@ static inline int stack_trace_save_tsk_reliable(struct= task_struct *tsk, } #endif =20 +#if defined(CONFIG_STACKTRACE) && defined(CONFIG_HAVE_SHADOW_STACKTRACE) +int stack_trace_save_shadow(unsigned long *store, unsigned int size, + unsigned int skipnr); +#else +static inline int stack_trace_save_shadow(unsigned long *store, + unsigned int size, + unsigned int skipnr) +{ + return -ENOSYS; +} +#endif + #endif /* __LINUX_STACKTRACE_H */ diff --git a/kernel/stacktrace.c b/kernel/stacktrace.c index 9ed5ce989415..fe305861fd55 100644 --- a/kernel/stacktrace.c +++ b/kernel/stacktrace.c @@ -237,6 +237,27 @@ unsigned int stack_trace_save_user(unsigned long *stor= e, unsigned int size) } #endif =20 +#ifdef CONFIG_HAVE_SHADOW_STACKTRACE +/** + * stack_trace_save_shadow - Save a stack trace based on shadow call stack + * @store: Pointer to the storage array + * @size: Size of the storage array + * @skipnr: Number of entries to skip at the start of the stack trace + * + * Return: Number of trace entries stored. + */ +int stack_trace_save_shadow(unsigned long *store, unsigned int size, + unsigned int skipnr) +{ + /* + * Do not use stack_trace_consume_fn to avoid making a function + * call for each collected frame to improve performance. + * Skip + 1 frame to skip stack_trace_save_shadow. + */ + return arch_stack_walk_shadow(store, size, skipnr + 1); +} +#endif + #else /* CONFIG_ARCH_STACKWALK */ =20 /* --=20 2.25.1 From nobody Mon Jun 22 13:22:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8D3FC433EF for ; Wed, 23 Mar 2022 15:33:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245253AbiCWPfM (ORCPT ); Wed, 23 Mar 2022 11:35:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245196AbiCWPeg (ORCPT ); Wed, 23 Mar 2022 11:34:36 -0400 Received: from out0.migadu.com (out0.migadu.com [94.23.1.103]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA83EC1F for ; Wed, 23 Mar 2022 08:33:04 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1648049583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T6yJKZfG38tzu4tVtlVcnWPw3LrS2FpvC9JjJySIrEg=; b=EQv2T55iO0XNuE9Zocz+eSnxdpOE37a0eKTUoOWjQBf0G/rtWzaOW94txvknnNt6fjH8rH TMhoiE+1d2dOiYsiXfYUwVsvDk9DQN2SMx0EnangRmRQWooE8wLmODk2yHTk7sEwRvtlAj gVqG0GwWr/hvjjulhh0g+wg3PbIeo/o= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Catalin Marinas , Will Deacon , Andrew Morton Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Mark Rutland , Vincenzo Frascino , Sami Tolvanen , Peter Collingbourne , Evgenii Stepanov , Florian Mayer , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 2/4] arm64, scs: save scs_sp values per-cpu when switching stacks Date: Wed, 23 Mar 2022 16:32:53 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Andrey Konovalov When an interrupt happens, the current Shadow Call Stack (SCS) pointer is switched to a per-interrupt one stored in a per-CPU variable. The old pointer is then saved on the normal stack and restored when the interrupt is handled. To collect the current stack trace based on SCS when the interrupt is being handled, we need to know the SCS pointers that belonged to the task and potentially other interrupts that were interrupted. Instead of trying to retrieve the SCS pointers from the stack, change interrupt handlers (for hard IRQ, Normal and Critical SDEI) to save the previous SCS pointer in a per-CPU variable. Note that interrupts stack. A task can be interrupted by a hard IRQ, which then can interrupted by a normal SDEI, etc. This is handled by using a separate per-CPU variable for each interrupt type. Also reset the saved SCS pointer when exiting the interrupt. This allows checking whether we should include any interrupt frames when collecting the stack trace. While we could use in_hardirq(), there seems to be no easy way to check whether we are in an SDEI handler. Directly checking the per-CPU variables for being non-zero is more resilient. Also expose both the the added saved SCS variables and the existing SCS base variables in arch/arm64/include/asm/scs.h so that the stack trace collection impementation can use them. Signed-off-by: Andrey Konovalov Reported-by: kernel test robot --- arch/arm64/include/asm/assembler.h | 12 ++++++++++++ arch/arm64/include/asm/scs.h | 13 ++++++++++++- arch/arm64/kernel/entry.S | 28 ++++++++++++++++++++++++---- arch/arm64/kernel/irq.c | 4 +--- arch/arm64/kernel/sdei.c | 5 ++--- 5 files changed, 51 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/as= sembler.h index 8c5a61aeaf8e..ca018e981d13 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -270,6 +270,18 @@ alternative_endif ldr \dst, [\dst, \tmp] .endm =20 + /* + * @src: Register whose value gets stored in sym + * @sym: The name of the per-cpu variable + * @tmp0: Scratch register + * @tmp1: Another scratch register + */ + .macro str_this_cpu src, sym, tmp0, tmp1 + adr_l \tmp0, \sym + get_this_cpu_offset \tmp1 + str \src, [\tmp0, \tmp1] + .endm + /* * vma_vm_mm - get mm pointer from vma pointer (vma->vm_mm) */ diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 8297bccf0784..2bb2b32f787b 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -24,6 +24,17 @@ .endm #endif /* CONFIG_SHADOW_CALL_STACK */ =20 -#endif /* __ASSEMBLY __ */ +#else /* __ASSEMBLY__ */ + +#include + +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_saved_ptr); +DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_ptr); +DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_saved_ptr); +DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_ptr); +DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_saved_ptr= ); + +#endif /* __ASSEMBLY__ */ =20 #endif /* _ASM_SCS_H */ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ede028dee81b..1c62fecda172 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -880,7 +880,8 @@ NOKPROBE(ret_from_fork) */ SYM_FUNC_START(call_on_irq_stack) #ifdef CONFIG_SHADOW_CALL_STACK - stp scs_sp, xzr, [sp, #-16]! + /* Save the current SCS pointer and load the per-IRQ one. */ + str_this_cpu scs_sp, irq_shadow_call_stack_saved_ptr, x15, x17 ldr_this_cpu scs_sp, irq_shadow_call_stack_ptr, x17 #endif /* Create a frame record to save our LR and SP (implicit in FP) */ @@ -902,7 +903,9 @@ SYM_FUNC_START(call_on_irq_stack) mov sp, x29 ldp x29, x30, [sp], #16 #ifdef CONFIG_SHADOW_CALL_STACK - ldp scs_sp, xzr, [sp], #16 + /* Restore saved SCS pointer and reset the saved value. */ + ldr_this_cpu scs_sp, irq_shadow_call_stack_saved_ptr, x17 + str_this_cpu xzr, irq_shadow_call_stack_saved_ptr, x15, x17 #endif ret SYM_FUNC_END(call_on_irq_stack) @@ -1024,11 +1027,16 @@ SYM_CODE_START(__sdei_asm_handler) #endif =20 #ifdef CONFIG_SHADOW_CALL_STACK - /* Use a separate shadow call stack for normal and critical events */ + /* + * Use a separate shadow call stack for normal and critical events. + * Save the current SCS pointer and load the per-SDEI one. + */ cbnz w4, 3f + str_this_cpu src=3Dscs_sp, sym=3Dsdei_shadow_call_stack_normal_saved_ptr,= tmp0=3Dx5, tmp1=3Dx6 ldr_this_cpu dst=3Dscs_sp, sym=3Dsdei_shadow_call_stack_normal_ptr, tmp= =3Dx6 b 4f -3: ldr_this_cpu dst=3Dscs_sp, sym=3Dsdei_shadow_call_stack_critical_ptr, t= mp=3Dx6 +3: str_this_cpu src=3Dscs_sp, sym=3Dsdei_shadow_call_stack_critical_saved_= ptr, tmp0=3Dx5, tmp1=3Dx6 + ldr_this_cpu dst=3Dscs_sp, sym=3Dsdei_shadow_call_stack_critical_ptr, tmp= =3Dx6 4: #endif =20 @@ -1062,6 +1070,18 @@ SYM_CODE_START(__sdei_asm_handler) ldp lr, x1, [x4, #SDEI_EVENT_INTREGS + S_LR] mov sp, x1 =20 +#ifdef CONFIG_SHADOW_CALL_STACK + /* Restore saved SCS pointer and reset the saved value. */ + ldrb w5, [x4, #SDEI_EVENT_PRIORITY] + cbnz w5, 5f + ldr_this_cpu dst=3Dscs_sp, sym=3Dsdei_shadow_call_stack_normal_saved_ptr,= tmp=3Dx6 + str_this_cpu src=3Dxzr, sym=3Dsdei_shadow_call_stack_normal_saved_ptr, tm= p0=3Dx5, tmp1=3Dx6 + b 6f +5: ldr_this_cpu dst=3Dscs_sp, sym=3Dsdei_shadow_call_stack_critical_saved_= ptr, tmp=3Dx6 + str_this_cpu src=3Dxzr, sym=3Dsdei_shadow_call_stack_critical_saved_ptr, = tmp0=3Dx5, tmp1=3Dx6 +6: +#endif + mov x1, x0 // address to complete_and_resume /* x0 =3D (x0 <=3D SDEI_EV_FAILED) ? * EVENT_COMPLETE:EVENT_COMPLETE_AND_RESUME diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index bda49430c9ea..4199f900714a 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -28,11 +28,9 @@ DEFINE_PER_CPU(struct nmi_ctx, nmi_contexts); =20 DEFINE_PER_CPU(unsigned long *, irq_stack_ptr); =20 - -DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); - #ifdef CONFIG_SHADOW_CALL_STACK DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_saved_ptr); #endif =20 static void init_irq_scs(void) diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c index d20620a1c51a..269adcb9e854 100644 --- a/arch/arm64/kernel/sdei.c +++ b/arch/arm64/kernel/sdei.c @@ -39,12 +39,11 @@ DEFINE_PER_CPU(unsigned long *, sdei_stack_normal_ptr); DEFINE_PER_CPU(unsigned long *, sdei_stack_critical_ptr); #endif =20 -DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_ptr); -DECLARE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_ptr); - #ifdef CONFIG_SHADOW_CALL_STACK DEFINE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_ptr); +DEFINE_PER_CPU(unsigned long *, sdei_shadow_call_stack_normal_saved_ptr); DEFINE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_ptr); +DEFINE_PER_CPU(unsigned long *, sdei_shadow_call_stack_critical_saved_ptr); #endif =20 static void _free_sdei_stack(unsigned long * __percpu *ptr, int cpu) --=20 2.25.1 From nobody Mon Jun 22 13:22:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEBE6C433EF for ; Wed, 23 Mar 2022 15:33:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245270AbiCWPfR (ORCPT ); Wed, 23 Mar 2022 11:35:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245195AbiCWPej (ORCPT ); Wed, 23 Mar 2022 11:34:39 -0400 Received: from out0.migadu.com (out0.migadu.com [IPv6:2001:41d0:2:267::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79A2AD8B for ; Wed, 23 Mar 2022 08:33:06 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1648049584; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MP8VVStpf2u5pFq9LJYVTfW5maMxRrDNwrS4C/iOhFE=; b=CHABMq/eZMH3CKmnepGiem7lAUyvtGITme8Qll4LQCROHN+V6C06OVGLwZuteRZ+PbizzX B/d0tsbkAj03Q2KNrb7PC4vE3rYeOl8qckRDCHswffvIEUY+QTm6icfg6nxgM6fpgtv52V PXKw6cD+WU2EotoizAcwfEiGoH3Y8cA= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Catalin Marinas , Will Deacon , Andrew Morton Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Mark Rutland , Vincenzo Frascino , Sami Tolvanen , Peter Collingbourne , Evgenii Stepanov , Florian Mayer , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 3/4] arm64: implement stack_trace_save_shadow Date: Wed, 23 Mar 2022 16:32:54 +0100 Message-Id: <0bb72ea8fa88ef9ae3508c23d993952a0ae6f0f9.1648049113.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Andrey Konovalov Implement the stack_trace_save_shadow() interface that collects stack traces based on the Shadow Call Stack (SCS) for arm64. The implementation walks through available SCS pointers (the per-task one and the per-interrupt-type ones) and copies the frames. Note that the frame of the interrupted function is not included into the stack trace, as it is not yet saved on the SCS when an interrupt happens. Signed-off-by: Andrey Konovalov Reported-by: kernel test robot --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/stacktrace.c | 83 ++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a659e238f196..d89cecf6c923 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -201,6 +201,7 @@ config ARM64 select MMU_GATHER_RCU_TABLE_FREE select HAVE_RSEQ select HAVE_RUST + select HAVE_SHADOW_STACKTRACE select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index e4103e085681..89daa710d91b 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -12,9 +12,11 @@ #include #include #include +#include =20 #include #include +#include #include #include =20 @@ -210,3 +212,84 @@ noinline notrace void arch_stack_walk(stack_trace_cons= ume_fn consume_entry, =20 walk_stackframe(task, &frame, consume_entry, cookie); } + +static const struct { + unsigned long ** __percpu saved; + unsigned long ** __percpu base; +} scs_parts[] =3D { +#ifdef CONFIG_ARM_SDE_INTERFACE + { + .saved =3D &sdei_shadow_call_stack_critical_saved_ptr, + .base =3D &sdei_shadow_call_stack_critical_ptr, + }, + { + .saved =3D &sdei_shadow_call_stack_normal_saved_ptr, + .base =3D &sdei_shadow_call_stack_normal_ptr, + }, +#endif /* CONFIG_ARM_SDE_INTERFACE */ + { + .saved =3D &irq_shadow_call_stack_saved_ptr, + .base =3D &irq_shadow_call_stack_ptr, + }, +}; + +static inline bool walk_shadow_stack_part( + unsigned long *scs_top, unsigned long *scs_base, + unsigned long *store, unsigned int size, + unsigned int *skipnr, unsigned int *len) +{ + unsigned long *frame; + + for (frame =3D scs_top; frame >=3D scs_base; frame--) { + if (*skipnr > 0) { + (*skipnr)--; + continue; + } + /* + * Do not leak PTR_AUTH tags in stack traces. + * Use READ_ONCE_NOCHECK as SCS is poisoned with Generic KASAN. + */ + store[(*len)++] =3D + ptrauth_strip_insn_pac(READ_ONCE_NOCHECK(*frame)); + if (*len >=3D size) + return true; + } + + return false; +} + +noinline notrace int arch_stack_walk_shadow(unsigned long *store, + unsigned int size, + unsigned int skipnr) +{ + unsigned long *scs_top, *scs_base, *scs_next; + unsigned int len =3D 0, part; + + preempt_disable(); + + /* Get the SCS pointer. */ + asm volatile("mov %0, x18" : "=3D&r" (scs_top)); + + /* The top SCS slot is empty. */ + scs_top -=3D 1; + + /* Handle SDEI and hardirq frames. */ + for (part =3D 0; part < ARRAY_SIZE(scs_parts); part++) { + scs_next =3D *this_cpu_ptr(scs_parts[part].saved); + if (scs_next) { + scs_base =3D *this_cpu_ptr(scs_parts[part].base); + if (walk_shadow_stack_part(scs_top, scs_base, store, + size, &skipnr, &len)) + goto out; + scs_top =3D scs_next; + } + } + + /* Handle task and softirq frames. */ + scs_base =3D task_scs(current); + walk_shadow_stack_part(scs_top, scs_base, store, size, &skipnr, &len); + +out: + preempt_enable(); + return len; +} --=20 2.25.1 From nobody Mon Jun 22 13:22:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4D88C433F5 for ; Wed, 23 Mar 2022 15:33:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245262AbiCWPfO (ORCPT ); Wed, 23 Mar 2022 11:35:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245192AbiCWPej (ORCPT ); Wed, 23 Mar 2022 11:34:39 -0400 Received: from out0.migadu.com (out0.migadu.com [IPv6:2001:41d0:2:267::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79AC3D8D for ; Wed, 23 Mar 2022 08:33:06 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1648049585; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BKf6jB0gbxzfxKMTwtGAqqdjGZr2rSaHYmUmfoDqc2M=; b=BHH7IXDotxEtbDywjIVk+wOZWOrLCY8o1Y0wKr7beWzVUIIg6NcRjZo4Ng098t1O+hRQBY Hj+XGflhkg2q6NrW+V2es5S1tFSC2M9D+KPkQyagcNNNo5WUxx/gSWh9/nZ8KEiacAZ/0J wefcvgBqi5K7ERAx+/NxwhoJ1KWavK4= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Catalin Marinas , Will Deacon , Andrew Morton Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Mark Rutland , Vincenzo Frascino , Sami Tolvanen , Peter Collingbourne , Evgenii Stepanov , Florian Mayer , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 4/4] kasan: use stack_trace_save_shadow Date: Wed, 23 Mar 2022 16:32:55 +0100 Message-Id: <7027b9b6b0cae2921ff65739582ae499bf61470c.1648049113.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Andrey Konovalov Now that stack_trace_save_shadow() is implemented by arm64, use it whenever CONFIG_HAVE_SHADOW_STACKTRACE is enabled. This improves the boot time of a defconfig build by ~30% for all KASAN modes. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index d9079ec11f31..8d9d35c6562b 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -33,10 +33,13 @@ depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc) { unsigned long entries[KASAN_STACK_DEPTH]; - unsigned int nr_entries; + unsigned int size; =20 - nr_entries =3D stack_trace_save(entries, ARRAY_SIZE(entries), 0); - return __stack_depot_save(entries, nr_entries, flags, can_alloc); + if (IS_ENABLED(CONFIG_HAVE_SHADOW_STACKTRACE)) + size =3D stack_trace_save_shadow(entries, ARRAY_SIZE(entries), 0); + else + size =3D stack_trace_save(entries, ARRAY_SIZE(entries), 0); + return __stack_depot_save(entries, size, flags, can_alloc); } =20 void kasan_set_track(struct kasan_track *track, gfp_t flags) --=20 2.25.1