From nobody Wed Dec 17 23:54:18 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0EAEC04A94 for ; Tue, 15 Aug 2023 20:35:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238085AbjHOUfI (ORCPT ); Tue, 15 Aug 2023 16:35:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238042AbjHOUev (ORCPT ); Tue, 15 Aug 2023 16:34:51 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C113DF7 for ; Tue, 15 Aug 2023 13:34:49 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-583d1d0de65so75735517b3.0 for ; Tue, 15 Aug 2023 13:34:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131689; x=1692736489; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2g9kzHqhSNJwgBBDx7OBACW2GSQNvlkcjq1gGBNV2Bo=; b=5fhSxlGCoP9eFSdFtRHiL4qRSdSwqPd7gzbVSPTiAU38g2wOHlOIpVYbOfieO8z/+X IEpl6nfntGQWgA924+5Z5nTBocRQ7FPksFYyL4EZCQja00MCxhy36o2za0FYibqRE21z fhQVudLJ+ihYNMXZnwK0rM1AI6CbD6GzVHb8KRBMJvZZVXLvVusxB3MJII05LCR7ksRN PYnsC/MjI2Tcyl2WcvR8r3fx2Vi+tJ1PCApbwyEgQcH+auQ7/GUCjqEaAjfE2WA22rZc CubL4Er4FcAbsDiTz/MAVkn6/HhJ3YMRAc+duqqYw/4Ml47tNzlvl2g4SmXazUrX9UGJ H+Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131689; x=1692736489; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2g9kzHqhSNJwgBBDx7OBACW2GSQNvlkcjq1gGBNV2Bo=; b=JrrAXSwkxAVHA7yyjE6v13UCUn3WS6S96dXTCgYwwZ5YjyZbBh/m7orlhr9jwvTzJn lXZIRakqwfNauZdwTwC501zc+UCYEQfh1BH0QyFN9odoF4RQ6gCt+WUZfNkNd5FVCw1w 9oc9lLLtJEQKsAmye4Kqf4cUWaW8k67zoHbQrGmoma4oQUN27lWyJWQOh26N/qC5ct+K /j2bqrt6364NjzLBBS5Z8AgpFFZD1CJSpYgUai142V1TZPS3qQZbE9BRzrLgXinPHvL0 DVrNoF1jN9G6Dhi6pt7i9ErJd4IFEHKq/hcaBQ+P/lQiGMgCutZSiEVkr6LkmFoevKZu awIA== X-Gm-Message-State: AOJu0Yx2knYs73Wn48eFR04e1k+p+NipXenjBB0YOoOoVWHdpjO7+3Tn MnH0NuWKAqjXjR7C/quWS9gQCLjJzM4JDAFzJpk= X-Google-Smtp-Source: AGHT+IGivQ4nxBqRW7/5l/sAoV4/vI2cYX6H54kXeR8lMldcoVTYpg5DO9jfP7O9CrkY98boxli26xltNMvid9T8YFI= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:af1f:0:b0:56d:647:664b with SMTP id n31-20020a81af1f000000b0056d0647664bmr196328ywh.5.1692131689073; Tue, 15 Aug 2023 13:34:49 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:44 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=10263; i=samitolvanen@google.com; h=from:subject; bh=ZSehk0HlS5lFssHilJirKHoqNUMKSxbPAfpprDSxzWc=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiY/LmC6Mrof2Ek0FECg4Lyrb602OprdqPv Z36fbNfcwKJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7mndC/0bMg+3a2jsWzVZKgcRI3wlt+nRTUv0WtQprozA0ki2lLlThz02RnGre6VdcJGWDs2NOTt ES5Mvq6ZCaP40k/y1vg293dS4YTr9axfndkManB9r3jzd+46U+nbQPpqvBUCeDJm2kuPNft/q9I SuubRVgtai/4nbcZXB7QrsYWW3Lek10TYFv+6sCU7NLij5hPrKN/NvDqJie0mVeZzvQO8hd/he0 C/vRSL6xNpExXTskKfQamDInoer/hiOiLI5OhAZSi+5Ov+W2uO+6zP5dUGTJ0OkOXj4Yd06IKcB hh16MhU5PVBJUPfQUL3OC8hCPaqJN5p6mPquadLTO0YZD3P6CTCjenY1Of0PBz4Zgo01fOIb+c6 U1XeBHSUBI4dbMW7VzXngTAkYd09OV+JNLJtsg3Rz0zTtsYRCr7oVmq8WQbPB8sBujzWJ36CKPF AOY17PAK2ebKmBDAss2umHOFq2t+7DF3bxncYTA1g/IKXLt66QJ2pZQ/1Ws0Mywoykmxs= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-9-samitolvanen@google.com> Subject: [PATCH v2 1/6] riscv: VMAP_STACK overflow detection thread-safe From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Jisheng Zhang , Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Deepak Gupta commit 31da94c25aea ("riscv: add VMAP_STACK overflow detection") added support for CONFIG_VMAP_STACK. If overflow is detected, CPU switches to `shadow_stack` temporarily before switching finally to per-cpu `overflow_stack`. If two CPUs/harts are racing and end up in over flowing kernel stack, one or both will end up corrupting each other state because `shadow_stack` is not per-cpu. This patch optimizes per-cpu overflow stack switch by directly picking per-cpu `overflow_stack` and gets rid of `shadow_stack`. Following are the changes in this patch - Defines an asm macro to obtain per-cpu symbols in destination register. - In entry.S, when overflow is detected, per-cpu overflow stack is located using per-cpu asm macro. Computing per-cpu symbol requires a temporary register. x31 is saved away into CSR_SCRATCH (CSR_SCRATCH is anyways zero since we're in kernel). Please see Links for additional relevant disccussion and alternative solution. Tested by `echo EXHAUST_STACK > /sys/kernel/debug/provoke-crash/DIRECT` Kernel crash log below Insufficient stack space to handle exception!/debug/provoke-crash/DIRECT Task stack: [0xff20000010a98000..0xff20000010a9c000] Overflow stack: [0xff600001f7d98370..0xff600001f7d99370] CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) epc : __memset+0x60/0xfc ra : recursive_loop+0x48/0xc6 [lkdtm] epc : ffffffff808de0e4 ra : ffffffff0163a752 sp : ff20000010a97e80 gp : ffffffff815c0330 tp : ff600000820ea280 t0 : ff20000010a97e88 t1 : 000000000000002e t2 : 3233206874706564 s0 : ff20000010a982b0 s1 : 0000000000000012 a0 : ff20000010a97e88 a1 : 0000000000000000 a2 : 0000000000000400 a3 : ff20000010a98288 a4 : 0000000000000000 a5 : 0000000000000000 a6 : fffffffffffe43f0 a7 : 00007fffffffffff s2 : ff20000010a97e88 s3 : ffffffff01644680 s4 : ff20000010a9be90 s5 : ff600000842ba6c0 s6 : 00aaaaaac29e42b0 s7 : 00fffffff0aa3684 s8 : 00aaaaaac2978040 s9 : 0000000000000065 s10: 00ffffff8a7cad10 s11: 00ffffff8a76a4e0 t3 : ffffffff815dbaf4 t4 : ffffffff815dbaf4 t5 : ffffffff815dbab8 t6 : ff20000010a9bb48 status: 0000000200000120 badaddr: ff20000010a97e88 cause: 000000000000000f Kernel panic - not syncing: Kernel stack overflow CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) Call Trace: [] dump_backtrace+0x30/0x38 [] show_stack+0x40/0x4c [] dump_stack_lvl+0x44/0x5c [] dump_stack+0x18/0x20 [] panic+0x126/0x2fe [] walk_stackframe+0x0/0xf0 [] recursive_loop+0x48/0xc6 [lkdtm] SMP: stopping secondary CPUs ---[ end Kernel panic - not syncing: Kernel stack overflow ]--- Cc: Guo Ren Cc: Jisheng Zhang Link: https://lore.kernel.org/linux-riscv/Y347B0x4VUNOd6V7@xhacker/T/#t Link: https://lore.kernel.org/lkml/20221124094845.1907443-1-debug@rivosinc.= com/ Signed-off-by: Deepak Gupta Co-developed-by: Sami Tolvanen Signed-off-by: Sami Tolvanen Acked-by: Guo Ren Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm.h | 22 +++++++++ arch/riscv/include/asm/thread_info.h | 3 -- arch/riscv/kernel/asm-offsets.c | 1 + arch/riscv/kernel/entry.S | 70 ++++------------------------ arch/riscv/kernel/traps.c | 36 +------------- 5 files changed, 34 insertions(+), 98 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 114bbadaef41..bfb4c26f113c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -82,6 +82,28 @@ .endr .endm =20 +#ifdef CONFIG_SMP +#ifdef CONFIG_32BIT +#define PER_CPU_OFFSET_SHIFT 2 +#else +#define PER_CPU_OFFSET_SHIFT 3 +#endif + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT + la \dst, __per_cpu_offset + add \dst, \dst, \tmp + REG_L \tmp, 0(\dst) + la \dst, \sym + add \dst, \dst, \tmp +.endm +#else /* CONFIG_SMP */ +.macro asm_per_cpu dst sym tmp + la \dst, \sym +.endm +#endif /* CONFIG_SMP */ + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/= thread_info.h index 1833beb00489..d18ce0113ca1 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -34,9 +34,6 @@ =20 #ifndef __ASSEMBLY__ =20 -extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)]; -extern unsigned long spin_shadow_stack; - #include #include =20 diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index d6a75aac1d27..9f535d5de33f 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,7 @@ void asm_offsets(void) OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); =20 + OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3d11aa3af105 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -10,9 +10,11 @@ #include #include #include +#include #include #include #include +#include =20 SYM_CODE_START(handle_exception) /* @@ -170,67 +172,15 @@ SYM_CODE_END(ret_from_exception) =20 #ifdef CONFIG_VMAP_STACK SYM_CODE_START_LOCAL(handle_kernel_stack_overflow) - /* - * Takes the psuedo-spinlock for the shadow stack, in case multiple - * harts are concurrently overflowing their kernel stacks. We could - * store any value here, but since we're overflowing the kernel stack - * already we only have SP to use as a scratch register. So we just - * swap in the address of the spinlock, as that's definately non-zero. - * - * Pairs with a store_release in handle_bad_stack(). - */ -1: la sp, spin_shadow_stack - REG_AMOSWAP_AQ sp, sp, (sp) - bnez sp, 1b - - la sp, shadow_stack - addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE - - //save caller register to shadow stack - addi sp, sp, -(PT_SIZE_ON_STACK) - REG_S x1, PT_RA(sp) - REG_S x5, PT_T0(sp) - REG_S x6, PT_T1(sp) - REG_S x7, PT_T2(sp) - REG_S x10, PT_A0(sp) - REG_S x11, PT_A1(sp) - REG_S x12, PT_A2(sp) - REG_S x13, PT_A3(sp) - REG_S x14, PT_A4(sp) - REG_S x15, PT_A5(sp) - REG_S x16, PT_A6(sp) - REG_S x17, PT_A7(sp) - REG_S x28, PT_T3(sp) - REG_S x29, PT_T4(sp) - REG_S x30, PT_T5(sp) - REG_S x31, PT_T6(sp) - - la ra, restore_caller_reg - tail get_overflow_stack - -restore_caller_reg: - //save per-cpu overflow stack - REG_S a0, -8(sp) - //restore caller register from shadow_stack - REG_L x1, PT_RA(sp) - REG_L x5, PT_T0(sp) - REG_L x6, PT_T1(sp) - REG_L x7, PT_T2(sp) - REG_L x10, PT_A0(sp) - REG_L x11, PT_A1(sp) - REG_L x12, PT_A2(sp) - REG_L x13, PT_A3(sp) - REG_L x14, PT_A4(sp) - REG_L x15, PT_A5(sp) - REG_L x16, PT_A6(sp) - REG_L x17, PT_A7(sp) - REG_L x28, PT_T3(sp) - REG_L x29, PT_T4(sp) - REG_L x30, PT_T5(sp) - REG_L x31, PT_T6(sp) + /* we reach here from kernel context, sscratch must be 0 */ + csrrw x31, CSR_SCRATCH, x31 + asm_per_cpu sp, overflow_stack, x31 + li x31, OVERFLOW_STACK_SIZE + add sp, sp, x31 + /* zero out x31 again and restore x31 */ + xor x31, x31, x31 + csrrw x31, CSR_SCRATCH, x31 =20 - //load per-cpu overflow stack - REG_L sp, -8(sp) addi sp, sp, -(PT_SIZE_ON_STACK) =20 //save context to overflow stack diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index f910dfccbf5d..deb2144d9143 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -397,48 +397,14 @@ int is_valid_bugaddr(unsigned long pc) #endif /* CONFIG_GENERIC_BUG */ =20 #ifdef CONFIG_VMAP_STACK -/* - * Extra stack space that allows us to provide panic messages when the ker= nel - * has overflowed its stack. - */ -static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16); -/* - * A temporary stack for use by handle_kernel_stack_overflow. This is use= d so - * we can call into C code to get the per-hart overflow stack. Usage of t= his - * stack must be protected by spin_shadow_stack. - */ -long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE/sizeof(long)] __aligned(16); - -/* - * A pseudo spinlock to protect the shadow stack from being used by multip= le - * harts concurrently. This isn't a real spinlock because the lock side m= ust - * be taken without a valid stack and only a single register, it's only ta= ken - * while in the process of panicing anyway so the performance and error - * checking a proper spinlock gives us doesn't matter. - */ -unsigned long spin_shadow_stack; - -asmlinkage unsigned long get_overflow_stack(void) -{ - return (unsigned long)this_cpu_ptr(overflow_stack) + - OVERFLOW_STACK_SIZE; -} =20 asmlinkage void handle_bad_stack(struct pt_regs *regs) { unsigned long tsk_stk =3D (unsigned long)current->stack; unsigned long ovf_stk =3D (unsigned long)this_cpu_ptr(overflow_stack); =20 - /* - * We're done with the shadow stack by this point, as we're on the - * overflow stack. Tell any other concurrent overflowing harts that - * they can proceed with panicing by releasing the pseudo-spinlock. - * - * This pairs with an amoswap.aq in handle_kernel_stack_overflow. - */ - smp_store_release(&spin_shadow_stack, 0); - console_verbose(); =20 pr_emerg("Insufficient stack space to handle exception!\n"); --=20 2.41.0.694.ge786442a9b-goog From nobody Wed Dec 17 23:54:18 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFE22C41513 for ; Tue, 15 Aug 2023 20:35:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238115AbjHOUfK (ORCPT ); Tue, 15 Aug 2023 16:35:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238051AbjHOUew (ORCPT ); Tue, 15 Aug 2023 16:34:52 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8657D1BF7 for ; Tue, 15 Aug 2023 13:34:51 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-26b67f0a773so2086631a91.1 for ; Tue, 15 Aug 2023 13:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131691; x=1692736491; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WrfRqvXS/YQAQD3rvUURRhi4O4V3buFLgz+cT4E6CXY=; b=0b0AfFag2OqtlzKfyBXLWRGSF8gW1eMeleZQdhIEnHmK7L+mnBqsuJuqNSwV6rtOEU u0s8fw9OKaQxUqiJRpnU0Hj7p2gKX/LzNZrGl2doSZ/b3K7fqQBZZ0gtHwr3NNrZmsmB bElC4ZVwGPAE2nH9h5njd1+8dir9Iv7EJaT4Lvaplu8can7LpTu0q410aYDm4oygCKG6 CfmWyM/5WXnsrFI9sYk8nJqgbKJWDH+e0Ftx6cUw7Ad5/bw0dCu4KZjUmFeQGgW27LQW b1+64f0E00cHsDKGcITMxivXieyFjcRQHcbigwhvU5bczQjxjY4/w6pRvIEHpu+0KRhz IQEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131691; x=1692736491; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WrfRqvXS/YQAQD3rvUURRhi4O4V3buFLgz+cT4E6CXY=; b=Yv/Qg0/SIqCTPe7isSDEtHWY2kdA8JGqpdc5LSKL8vda6AN55j12yft/vHr7HuNROq SHIb0KW1oYlDa5CZjne/wQPMzoH8pbn3TuWcFv7HDE8L+yHFIFwy7RcwF9V6cg4FCLy3 o5s/Otzw0jLW3UcsDAyWQrA18hDgW0RUF7XCIifRV6t7VYOyIxIORce/0kPQ7lKcqosJ ciOID6MncZibfqGBtkFhZ4qkKZb8MrkSDFq71dYg4a5boMIrK0pL1W1anuFn8F7AblQq NhZJPhqb/IzxIqA0GyRMTpaErbdbUvXcpMTs5vE43JK9PdhY3SKn0WLILX59lzY87B52 YI6Q== X-Gm-Message-State: AOJu0YyBGyGY6anBY3rNGETt6idVIyMV++VARHC55EgjiDGfCLtdJExk nTohIGa/yeJLyVdw1lpXV6VMbGt6yvPNKbej0H4= X-Google-Smtp-Source: AGHT+IEchNZaBEINpLlSn/GPNkBO+VrwWfHXdxqZqQVAkBDyQGibTsdMvRWQoxISxiHRWmaP5XqKzzpcu2EXo/lIxzI= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:90a:d502:b0:26b:5fe2:5892 with SMTP id t2-20020a17090ad50200b0026b5fe25892mr1185559pju.6.1692131690966; Tue, 15 Aug 2023 13:34:50 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:45 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=5593; i=samitolvanen@google.com; h=from:subject; bh=fOkgWkASfAqgpBfAvGFiQ8Ap5BTE+sMjcVhFVqoevU4=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiD9deMgIJ3fhkaSc5TjCFw9aOJgO3lojDl IJKQL6WcO2JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7uETC/oDWTmoCmprkVOzJT880FOolG/86ApcG+2Vp5pFm9jTSXw0uhsv0QJYCKBTupTIjofrNuk yID9Hm9vbntt8SHaS+jwKoheRWZDOAdh0tJuWru56EB0QyTPLLuI6f2NTAFPPDF7/N0MEfFY9Q2 73sbckQRqQxt7tkF7oxFDkS6SMsogOzT8IZ2NuTszPpFCpFOybsEIX+sds97QqG22w0LjKn120J ABr9wCeGmcRNk7+5igryjV3nOw6/uMjWyZwGKIyzWrC3FFDqbHm5AuDcIgkhUd0DK5YK13HFgls qtnla0SK6MtOkM/Zg37Tt4SMgUlX9a75WKGDlVjX1j6KQSvU3TGRjp1Lyas587+RFyuvHXVjdX1 qYlEOUKf8qQjIcqiNnCqllRAsCjzds/YrdVFgoHxrkzeXt+aq89oaE2tLG8gh3OC9y2pQ04Bpz8 vLLi1qj7Wm/3TWsrEydTvIulNO8xWEYLDag4+LI+L8CYYpBAwjrgoVvB0cCcF1VGQ0U5Y= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-10-samitolvanen@google.com> Subject: [PATCH v2 2/6] riscv: Deduplicate IRQ stack switching From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redudant inline assembly. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/entry.S | 32 ++++++++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 32 ++++++++---------------------- arch/riscv/kernel/traps.c | 29 ++++----------------------- 5 files changed, 52 insertions(+), 49 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index bfb4c26f113c..8e446be2d57c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -104,6 +104,11 @@ .endm #endif /* CONFIG_SMP */ =20 +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/irq_stack.h b/arch/riscv/include/asm/ir= q_stack.h index e4042d297580..6441ded3b0cf 100644 --- a/arch/riscv/include/asm/irq_stack.h +++ b/arch/riscv/include/asm/irq_stack.h @@ -12,6 +12,9 @@ =20 DECLARE_PER_CPU(ulong *, irq_stack_ptr); =20 +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); + #ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMA= P'd diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 3d11aa3af105..39875f5e08a6 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -218,6 +218,38 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) =20 +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -RISCV_SZPTR + REG_S ra, (sp) + addi sp, sp, -RISCV_SZPTR + REG_S s0, (sp) + addi s0, sp, 2*RISCV_SZPTR + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -2*RISCV_SZPTR + REG_L s0, (sp) + addi sp, sp, RISCV_SZPTR + REG_L ra, (sp) + addi sp, sp, RISCV_SZPTR + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored. diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index d0577cc6a081..95dafdcbd135 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -61,32 +61,16 @@ static void init_irq_stacks(void) #endif /* CONFIG_VMAP_STACK */ =20 #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq(struct pt_regs *regs) +{ + __do_softirq(); +} + void do_softirq_own_stack(void) { -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp =3D per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "call __do_softirq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + if (on_thread_stack()) + call_on_irq_stack(NULL, ___do_softirq); + else __do_softirq(); } #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index deb2144d9143..83319b6816da 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -350,31 +350,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *= regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) { irqentry_state_t state =3D irqentry_enter(regs); -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp =3D per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + + if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack()) + call_on_irq_stack(regs, handle_riscv_irq); + else handle_riscv_irq(regs); =20 irqentry_exit(regs, state); --=20 2.41.0.694.ge786442a9b-goog From nobody Wed Dec 17 23:54:18 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0AAA1C001E0 for ; Tue, 15 Aug 2023 20:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238147AbjHOUfL (ORCPT ); Tue, 15 Aug 2023 16:35:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238052AbjHOUey (ORCPT ); Tue, 15 Aug 2023 16:34:54 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12174F7 for ; Tue, 15 Aug 2023 13:34:53 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-26943ac3fa2so3126567a91.3 for ; Tue, 15 Aug 2023 13:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131692; x=1692736492; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BfDLHnMbXQceNrVQLGjVUevqJ2xwYxenmhNK/tAF/sk=; b=DveYx2ypgHekd9E3zy9CT1ilJONYIrAnPqHRj9ZtXg944Rf0TWVVNG5jaFobMnoKx0 Qp6jSJbrWh3E2/h9IXPRd2IM3jaJArvh4oKLp7/khzJJ/4YoiwEv/DKqr19Ru89mBaYv 1gt7hiExW5s9lLZMvCZRWPyrP/GDG1LqMTngfLcIolNUd8lCABgY4ON75DRwtNUOCk8B JY5g55OHAze9zUBVPQ+rocjrZ/v9jCkE6obEf+/vP5i9YGYK/d6QF7IAgDOSXS1kHXVD FeNp6SoviL3M8abvX30FCaMuBBgTkMnTrZyfZofD7Pl87xvQwDtW7fNUw3gGmTWAH0AY dFmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131692; x=1692736492; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BfDLHnMbXQceNrVQLGjVUevqJ2xwYxenmhNK/tAF/sk=; b=aMfUpnTHb1+bprpIr3JiJbmyfz3QYXAvNWgnQ3qQK62kdZrWVIb2bVGzqP0sxOlZJ9 oN+W+Uonuee5RQqbSQMFB8/s33IkUANttF8IrNofQILXRRp1NXrPGH4gKyuIVhY8eOZY wPGbaupu5T9BrfJOTyQpqD00UYyc6l8dbjQhLl1ez9Y328Tumi7K2EsyXXJAmTqJ6Vps WXz06Qtwlt3feJkFPB8m4rL8fj4CWk7RCGkwlkv/f5IEaJCJwZpE2ZuCiHh4ZpSNC9Bb 6XubJg6gzsKFsUXV7zLqVUBRqrI4bpXk1m9aa5GFM3oBlWBhEbjIW4Pvjw51zoRDblOK YDeg== X-Gm-Message-State: AOJu0Yw7WBw/e0OwhDsXIh5NntaI78iyb+d/RPcjoZAtlFhKWhDYQPE7 +IDkmDIzROJLcr7si6O70csJ1IXqWKbcWQXa5kE= X-Google-Smtp-Source: AGHT+IG4n+M3PSO3DhS/2Scpx+tUwpXnKVdLJNTSN6YSIUZ+vrp2chjur8+avcRYM+CJmcVopciJDOywc23KOj4HoFo= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:90b:4393:b0:263:3437:a0b0 with SMTP id in19-20020a17090b439300b002633437a0b0mr3063684pjb.3.1692131692589; Tue, 15 Aug 2023 13:34:52 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:46 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3217; i=samitolvanen@google.com; h=from:subject; bh=5ixoDk+W6BxwBv4tIu2eVSrOr56ebYlcn47cfqFXXgA=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiHUFljIaux+9IC+0XLbF6reJW5Y5V3fbd/ /MD1Qt4OqyJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7kdQDACYUsshok1LuWlv5stJjrwH99NJQrdDSX0WI0YLs+JulAYvsnIPJAatZgneQWhTPWBynMO rJ3Ix+mbOgesN6Los5e0Z5iR+60FzUznS4csogCnjAI1+8EEVryA7ygmOnVm4/wnNpslAOjORp7 byQ+dXEu0N175Wnuc3vqbRnwlcaSIgJR+3bGVEuk8ezzcuTN42WVem7P7gBBgGWi+MzsEtJk9IA +oCPRG0zWkpcBkqHS77+B58Wg9Z/YAaYEeyeqUKq1AMU2MIUP1ul8YRN6CX93320mx9vMOO6tff vPHHw66ck3QzRW5YPmT1EYusoHAeXpCxw6RBC7i2r5Wyqw/QVNOqtKvrkJuzNmMRFJnl9zlGZbs C0IN9uG9VSOuYRky9JDPhBidXrnk7bvsMGzrNuj703Iz9WVkn+LfAUI+Nn/LiYYHYOqhR9w+PRl ERHb8HUO9XJqfcJPtffC4okUlO3NCdKwN6FcN6izZWDFGeuHyBokuvxqemECQWRpFEqkM= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-11-samitolvanen@google.com> Subject: [PATCH v2 3/6] riscv: Move global pointer loading to a macro From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In Clang 17, -fsanitize=3Dshadow-call-stack uses the newly declared platform register gp for storing shadow call stack pointers. As this is obviously incompatible with gp relaxation, in preparation for CONFIG_SHADOW_CALL_STACK support, move global pointer loading to a single macro, which we can cleanly disable when SCS is used instead. Link: https://reviews.llvm.org/rGaa1d2693c256 Link: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e= 6eeb51f0cb7b8819e50da6d2444d769 Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm.h | 8 ++++++++ arch/riscv/kernel/entry.S | 6 ++---- arch/riscv/kernel/head.S | 15 +++------------ arch/riscv/kernel/suspend_entry.S | 5 +---- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 8e446be2d57c..f34dd1a526a1 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,14 @@ REG_L \dst, 0(\dst) .endm =20 +/* load __global_pointer to gp */ +.macro load_global_pointer +.option push +.option norelax + la gp, __global_pointer$ +.option pop +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 39875f5e08a6..2b4248c6b0a9 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -75,10 +75,8 @@ _save_context: csrw CSR_SCRATCH, x0 =20 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer + move a0, sp /* pt_regs */ la ra, ret_from_exception =20 diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 11c3b94c4534..79b5a863c782 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -110,10 +110,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a0 =20 /* Reload the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer =20 /* * Switch to kernel page tables. A full fence is necessary in order to @@ -134,10 +131,7 @@ secondary_start_sbi: csrw CSR_IP, zero =20 /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer =20 /* * Disable FPU & VECTOR to detect illegal usage of @@ -228,10 +222,7 @@ pmp_done: #endif /* CONFIG_RISCV_M_MODE */ =20 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer =20 /* * Disable FPU & VECTOR to detect illegal usage of diff --git a/arch/riscv/kernel/suspend_entry.S b/arch/riscv/kernel/suspend_= entry.S index 12b52afe09a4..556a4b166d8c 100644 --- a/arch/riscv/kernel/suspend_entry.S +++ b/arch/riscv/kernel/suspend_entry.S @@ -60,10 +60,7 @@ END(__cpu_suspend_enter) =20 ENTRY(__cpu_resume_enter) /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer =20 #ifdef CONFIG_MMU /* Save A0 and A1 */ --=20 2.41.0.694.ge786442a9b-goog From nobody Wed Dec 17 23:54:18 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B2CBC04E69 for ; Tue, 15 Aug 2023 20:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238052AbjHOUfN (ORCPT ); Tue, 15 Aug 2023 16:35:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238060AbjHOUe4 (ORCPT ); Tue, 15 Aug 2023 16:34:56 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D109FF7 for ; Tue, 15 Aug 2023 13:34:54 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-586bd766310so59362797b3.0 for ; Tue, 15 Aug 2023 13:34:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131694; x=1692736494; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=N3ibQG9kPTeaqq1uk119WhHtvGpC05ARPoe95r4WB3Q=; b=ii3NiBwjW2S4FrV7yjM64htaZ/K+xMTAe8C3FjqVBGt9E6qnK9LrJmBelNkdwUPq0S /u8xXLY/b2D0v9G+ODd2yoH2w0MBi7+xjG488iodmtTpyqSxuJWj17X4fZWhEmWIX+fR wjELHT7fOZ01fKMWI8/otfmtTHXgHB8Fq+wgPvPcWxYFNX4dnVbEX57BoPyZuUGCDZWR 1O51jehCVbCgc9APBR35NAd6nxRB0SailOkfeWnWCnEHaQWkyZhKxC/zwlFuL/DZOHhC N3u254/fONTgoNDkrUS6Iwkbg/W8KeV4athdOLQ3LmKxCTJeJrUnNXykkj6KSDMpHum5 ICCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131694; x=1692736494; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=N3ibQG9kPTeaqq1uk119WhHtvGpC05ARPoe95r4WB3Q=; b=FbKQ6CVbhVqKNZzToHBzZDwbxMePT0T+rhS0MEdaWX6PWq8kCAS94soZK4cH9Ypdk4 BrZBV/ErxAS695OsC1+TTSWLSVcN5aPgBjfDaKgne4O0kPZe7ow2Rcdd/7fMkrzMlZjl v2ESBvHM9djiDghK/HgKBVBTAsbcq78Is/LDmlg913iGFl1+4q0FT5xNArDK+/cCO5G4 5mB9P5hs7EimK9tEFsKqMhNiRKFz9wK4gSSkkPyWKujtidFArkC7H5Gvw0NVCqFcmF+w PSns8a996JIpfQSvDZinXFTe0jQs/SOW5pbjHM3hRoIXWr52L7nPQ1b6yZG+HuYghiiZ 9rcQ== X-Gm-Message-State: AOJu0YyrewIfyjA6gABu4myFl6qpx4Co3EzHZvA+o1xnCAUQym3yiAtd K9flrf+pddeeSYxo8blK+h6WgKpM55v/FPqHsfU= X-Google-Smtp-Source: AGHT+IFOBtAfOTGqgJAn+DcvcJUfpiueeWWzkqYoVx3UNQD79ntr/Mc1j91DYQQN2fSTiPkbFkjokwMeqbqG2PDHuNw= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:b60d:0:b0:58c:6ddd:d27c with SMTP id u13-20020a81b60d000000b0058c6dddd27cmr5856ywh.6.1692131694123; Tue, 15 Aug 2023 13:34:54 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:47 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=9645; i=samitolvanen@google.com; h=from:subject; bh=e0JAEbcNV8QaCejgu3nDKo0gvZQXOTr0vBP/0yqZP/Q=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiTMC3KO4Q7y58NU00sySutTm4Un6tjcRZM NE+DfEzpYyJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7omFDACbtMcHmBN0cplT0k67u1z6qmmA2zIU82etEYjowZNjt2dLorEXMhCUCsSGBB+WOfE8wdK T1Ic5tQpvOODjDM1gAr0DzGeI4zHBtdL35iFmGDHe/K4JAihBOPM9ukDaPBywSyWMp7PwOT3R3E dpTpGqVyZCAz+ggnRAItZlj9hr1xlBPwi0Lgga40mOUDi0A83mhYPFar290fiu3bHBFHSkWWkEZ bE/cB9jlBzaDLK9fE7B2um3TvkudWYaCpimIbKNJhDGBrBr3Zk98tW4+jHe4VIu7SUc3i5JJcc7 jqK/+9Tl8CLtFl1lHwAqYHCz9esWAPVyWQtAQtyZd+bd8ZK+JUa5QpEso9V+8vl+/ntd8bl+5Te faYUB+jUrc5PH2ROCiBNrQYem3x8ezSXhtS6j5xHs0YoGWCQWNZFA2NqtOeZNDejk/2TFYbLSPl y7/WihDsjDhucR7Td+gZSUJ0WSB288e8icuCoB2C7SfPlj7zn6m7ktq/naHzMqTDS4c2c= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-12-samitolvanen@google.com> Subject: [PATCH v2 4/6] riscv: Implement Shadow Call Stack From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement CONFIG_SHADOW_CALL_STACK for RISC-V. When enabled, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning, which makes it harder to corrupt the return address through a stack overflow, for example. The active shadow call stack pointer is stored in the gp register, which makes SCS incompatible with gp relaxation. Use --no-relax-gp to ensure gp relaxation is disabled and disable global pointer loading. Add SCS pointers to struct thread_info, implement SCS initialization, and task switching Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/Kconfig | 6 ++++ arch/riscv/Makefile | 4 +++ arch/riscv/include/asm/asm.h | 6 ++++ arch/riscv/include/asm/scs.h | 47 ++++++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 13 ++++++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/entry.S | 11 +++++++ arch/riscv/kernel/head.S | 4 +++ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 +++ 10 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/scs.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 4c07b9189c86..8fe31ec59da4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -46,6 +46,7 @@ config RISCV select ARCH_SUPPORTS_HUGETLBFS if MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU select ARCH_SUPPORTS_PER_VMA_LOCK if MMU + select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU @@ -169,6 +170,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=3D8) =20 +config HAVE_SHADOW_CALL_STACK + def_bool $(cc-option,-fsanitize=3Dshadow-call-stack) + # https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6e= eb51f0cb7b8819e50da6d2444d769 + depends on $(ld-option,--no-relax-gp) + config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT default 8 diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6ec6d52a4180..e518a74640fb 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -55,6 +55,10 @@ endif endif endif =20 +ifeq ($(CONFIG_SHADOW_CALL_STACK),y) + KBUILD_LDFLAGS +=3D --no-relax-gp +endif + # ISA string setting riscv-march-$(CONFIG_ARCH_RV32I) :=3D rv32ima riscv-march-$(CONFIG_ARCH_RV64I) :=3D rv64ima diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index f34dd1a526a1..b0487b39e674 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,11 @@ REG_L \dst, 0(\dst) .endm =20 +#ifdef CONFIG_SHADOW_CALL_STACK +/* gp is used as the shadow call stack pointer instead */ +.macro load_global_pointer +.endm +#else /* load __global_pointer to gp */ .macro load_global_pointer .option push @@ -116,6 +121,7 @@ la gp, __global_pointer$ .option pop .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ =20 /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h new file mode 100644 index 000000000000..94726ea773e3 --- /dev/null +++ b/arch/riscv/include/asm/scs.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifdef __ASSEMBLY__ +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* Load init_shadow_call_stack to gp. */ +.macro scs_load_init_stack + la gp, init_shadow_call_stack + XIP_FIXUP_OFFSET gp +.endm + +/* Load task_scs_sp(current) to gp. */ +.macro scs_load_current + REG_L gp, TASK_TI_SCS_SP(tp) +.endm + +/* Load task_scs_sp(current) to gp, but only if tp has changed. */ +.macro scs_load_current_if_task_changed prev + beq \prev, tp, _skip_scs + scs_load_current +_skip_scs: +.endm + +/* Save gp to task_scs_sp(current). */ +.macro scs_save_current + REG_S gp, TASK_TI_SCS_SP(tp) +.endm + +#else /* CONFIG_SHADOW_CALL_STACK */ + +.macro scs_load_init_stack +.endm +.macro scs_load_current +.endm +.macro scs_load_current_if_task_changed prev +.endm +.macro scs_save_current +.endm + +#endif /* CONFIG_SHADOW_CALL_STACK */ +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/= thread_info.h index d18ce0113ca1..574779900bfb 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -57,8 +57,20 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ +#ifdef CONFIG_SHADOW_CALL_STACK + void *scs_base; + void *scs_sp; +#endif }; =20 +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base =3D init_shadow_call_stack, \ + .scs_sp =3D init_shadow_call_stack, +#else +#define INIT_SCS +#endif + /* * macros/functions for gaining access to the thread information structure * @@ -68,6 +80,7 @@ struct thread_info { { \ .flags =3D 0, \ .preempt_count =3D INIT_PREEMPT_COUNT, \ + INIT_SCS \ } =20 void arch_release_task_struct(struct task_struct *tsk); diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index 9f535d5de33f..177cef43a2ee 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -38,6 +38,9 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); +#ifdef CONFIG_SHADOW_CALL_STACK + OFFSET(TASK_TI_SCS_SP, task_struct, thread_info.scs_sp); +#endif =20 OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 2b4248c6b0a9..ad34507d3c96 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -9,6 +9,7 @@ =20 #include #include +#include #include #include #include @@ -77,6 +78,9 @@ _save_context: /* Load the global pointer */ load_global_pointer =20 + /* Load the kernel shadow call stack pointer if coming from userspace */ + scs_load_current_if_task_changed s5 + move a0, sp /* pt_regs */ la ra, ret_from_exception =20 @@ -123,6 +127,9 @@ SYM_CODE_START_NOALIGN(ret_from_exception) addi s0, sp, PT_SIZE_ON_STACK REG_S s0, TASK_TI_KERNEL_SP(tp) =20 + /* Save the kernel shadow call stack pointer */ + scs_save_current + /* * Save TP into the scratch register , so we can find the kernel data * structures again. @@ -277,6 +284,8 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + /* Save the kernel shadow call stack pointer */ + scs_save_current /* Restore context from next->thread */ REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) @@ -294,6 +303,8 @@ SYM_FUNC_START(__switch_to) REG_L s11, TASK_THREAD_S11_RA(a4) /* The offset of thread_info in task_struct is zero. */ move tp, a1 + /* Switch to the next shadow call stack */ + scs_load_current ret SYM_FUNC_END(__switch_to) =20 diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 79b5a863c782..c3d0ee77483b 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "efi-header.S" =20 @@ -153,6 +154,7 @@ secondary_start_sbi: XIP_FIXUP_OFFSET a3 add a3, a3, a1 REG_L sp, (a3) + scs_load_current =20 .Lsecondary_start_common: =20 @@ -293,6 +295,7 @@ clear_bss_done: la sp, init_thread_union + THREAD_SIZE XIP_FIXUP_OFFSET sp addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_BUILTIN_DTB la a0, __dtb_start XIP_FIXUP_OFFSET a0 @@ -311,6 +314,7 @@ clear_bss_done: la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack =20 #ifdef CONFIG_KASAN call kasan_early_init diff --git a/arch/riscv/kernel/vdso/Makefile b/arch/riscv/kernel/vdso/Makef= ile index 6b1dba11bf6d..48c362c0cb3d 100644 --- a/arch/riscv/kernel/vdso/Makefile +++ b/arch/riscv/kernel/vdso/Makefile @@ -36,7 +36,7 @@ CPPFLAGS_vdso.lds +=3D -DHAS_VGETTIMEOFDAY endif =20 # Disable -pg to prevent insert call site -CFLAGS_REMOVE_vgettimeofday.o =3D $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_vgettimeofday.o =3D $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) =20 # Disable profiling and instrumentation for VDSO code GCOV_PROFILE :=3D n diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index dc20e166983e..d5d60c040560 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -77,6 +77,10 @@ ifdef CONFIG_STACKPROTECTOR_STRONG PURGATORY_CFLAGS_REMOVE +=3D -fstack-protector-strong endif =20 +ifdef CONFIG_SHADOW_CALL_STACK +PURGATORY_CFLAGS_REMOVE +=3D $(CC_FLAGS_SCS) +endif + CFLAGS_REMOVE_purgatory.o +=3D $(PURGATORY_CFLAGS_REMOVE) CFLAGS_purgatory.o +=3D $(PURGATORY_CFLAGS) =20 --=20 2.41.0.694.ge786442a9b-goog From nobody Wed Dec 17 23:54:18 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D733C04FE0 for ; Tue, 15 Aug 2023 20:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238234AbjHOUfP (ORCPT ); Tue, 15 Aug 2023 16:35:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238067AbjHOUe5 (ORCPT ); Tue, 15 Aug 2023 16:34:57 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93B48F7 for ; Tue, 15 Aug 2023 13:34:56 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-583c49018c6so75049997b3.0 for ; Tue, 15 Aug 2023 13:34:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131696; x=1692736496; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uSOlMNCKHmfn5xfjrLh25HLPAdSumgQ6fSlbmoDJ2Yc=; b=qox904O6890VuPVeOFie+1gdmVBYEK7DiCslIx75yuPhU25HodT/JX1fwIBsNjAqoW iQp2mkqgx00drzHTCY7LI8aABLkp1VnAeuVSPs0VEuZYZ47KInVdm7L62A4CvkC5bp0l C5/HLIBI+E08srWstaA2cDxRL3JtGccv9yCImwYaUrvr3deqtmN7h8aeIdx6lyEucs/m pw4vrlK5aHRM1FnlmLa+RzgujHxZ4WrJcngnnxCUiIxM+P8tDmNNr9Vt8Ab6fbkG20n1 yF5suVbZp+f/JhUCo5ExAiU6zsPSve/ui3KtPmU9Ua6b1Fdva7K+Na15IB35x1Wnlhgr y1UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131696; x=1692736496; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uSOlMNCKHmfn5xfjrLh25HLPAdSumgQ6fSlbmoDJ2Yc=; b=C4b7YSdjLpWGlQBE3HJPnAQZ/2eW1Ytk6OhNQ74omGnKwcOG14rpc/a+OVFAIXR+ED voDL23Iiu/IgFib+Hx8hptC5mooLoDabtufF6JiKJi5JEMPwDAr2mEZ8sT4RCauqBtWu unkkkpu1JRRkH4nul09vlFGYtH70V+OQnHo4SdetvYcCUX/9ycR1Bi2EAztd+1Z+YEh6 gGljsPVl6au4KHyIp5dBnG/4oheVb16eXNFiI6crWaxhJHxB9iE8XWVQVge63X5Q52GC uN00AAj9OVIq2tHvtNuNzHgJkyHOwRIxfCCSjKs4S81XbnvdDGLQZhaOngrHrQ70leaf 6EgA== X-Gm-Message-State: AOJu0Yxo4hMxW6EK0QixXr6GPXstytijuIX8/DdCD5T1Zw4Fke4kygPX vjOnQKVoSGG8eeLtuQ2GpFk/PgOAr28CSwIiKaY= X-Google-Smtp-Source: AGHT+IFIG6x56rr09NEIucFsFTb2xdeaHALY+wrc10+vyz3Ri9z+DqUmjXG78S/oVc7ErGyWTEp4JYnxEZ20xlRZUd0= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:ad12:0:b0:589:a935:b13 with SMTP id l18-20020a81ad12000000b00589a9350b13mr197844ywh.5.1692131695824; Tue, 15 Aug 2023 13:34:55 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:48 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3124; i=samitolvanen@google.com; h=from:subject; bh=+CT5zR4YANM8SQ9/yXUv6987s9+QQOw/OXe2qcTQ3NU=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiWKdkEjcK+oM60PCK6k6ESVE8u+T4K+xL3 kx9YUX2gkmJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7kXbDACVmNAL8YYcd9TlWO6FP17kmKyhCvdTHqqrxGafGKiyJDXD7dTQipBujrxw7uevlZIkOiM 2krCuz5YPDlaC9LMHF4iLkWccLPwZY1NMQV8VwQyWPIHG97WO4fo4aZ/kbZf/FVoXcEBkZzmXD8 8OIDeYf8fCjDiqITUNFS8qFFrppbk0LerRQp9AYy9rIXE9LIuxkoEJ0I892xLtE2jEJLbAj13UM vTaq0hnLWd9E5rxKWuGMqUheN3DI1xIYFHjZX6lYODTUzRVyukKdeCXAyG0XNeyaVi2AdmvgGo5 eMPQz8InG+kdQSqH3PSbaR/H788IhSheQImtpgsJhycp41G5XbIAVuq5ndC1pik0iTdXM/sQoR9 /Tr6kIR9wN8YC3VWXEKaLCPfVOvzGEta9WYy7NAxOQ1tAjE9PjNaLg75QDxE9Ji92wRuwPgCcqh 1avsH5hj5GulKKx9q9ZdyeIoLP/3xzhAq5SgONJcm7ldjpCnTpGVtm6muw6YiAJ0EUnNY= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-13-samitolvanen@google.com> Subject: [PATCH v2 5/6] riscv: Use separate IRQ shadow call stacks From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 94726ea773e3..0e45db78b24b 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm =20 +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -34,6 +39,8 @@ =20 .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index ad34507d3c96..c86b76584d2d 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -238,12 +238,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, (sp) addi s0, sp, 2*RISCV_SZPTR =20 + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 =20 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -2*RISCV_SZPTR REG_L s0, (sp) diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 95dafdcbd135..7bfea97ee7e7 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); #ifdef CONFIG_IRQ_STACKS #include =20 +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs(void) +{ + int cpu; + + if (!scs_is_enabled()) + return; + + for_each_possible_cpu(cpu) + per_cpu(irq_shadow_call_stack_ptr, cpu) =3D + scs_alloc(cpu_to_node(cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr); =20 #ifdef CONFIG_VMAP_STACK @@ -76,6 +95,7 @@ void do_softirq_own_stack(void) #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ =20 #else +static void init_irq_scs(void) {} static void init_irq_stacks(void) {} #endif /* CONFIG_IRQ_STACKS */ =20 @@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) =20 void __init init_IRQ(void) { + init_irq_scs(); init_irq_stacks(); irqchip_init(); if (!handle_arch_irq) --=20 2.41.0.694.ge786442a9b-goog From nobody Wed Dec 17 23:54:18 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D4A0C04FDF for ; Tue, 15 Aug 2023 20:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238279AbjHOUfQ (ORCPT ); Tue, 15 Aug 2023 16:35:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238074AbjHOUe6 (ORCPT ); Tue, 15 Aug 2023 16:34:58 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5410F7 for ; Tue, 15 Aug 2023 13:34:57 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id d2e1a72fcca58-686b879f630so6261875b3a.3 for ; Tue, 15 Aug 2023 13:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131697; x=1692736497; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0Ykk2Dcm0sp1r3wFF3O+L+QDwz/a2RXVAzUE310xTCE=; b=c51n2/tJShODyeeUWChM6SNmOtBH4TwAB27gHlz0U2MW1KhJAFsJAI7HodLDVzEs2L h5EacJeK3yQ7xI1bDM7pS/8+NTTnu/pBkkf0mHnOLxusk130BgOMc3UgF/ILE4pTCv3b FoK/mdVbDTdzIm+PesccCvkjIraV3wLcVysrd2PMSHMHQsQbcViexWNnSAbHE6yrDtgT Mp+VyMCTaEctp9R1ryTin05vgD7DzxczuWzmHabxbBnipkxhOH1Z8NXKF+B+bk3ZWN2C pzEoTOjVu6w2TWjI1tCV+5ZHtdXlLljOrV5rvKK0dr/ZGKr7YBnabTma9ayXOLsdZ9cs ecvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131697; x=1692736497; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0Ykk2Dcm0sp1r3wFF3O+L+QDwz/a2RXVAzUE310xTCE=; b=BlmY6QCqYTOVoTzRIUih1vAU+GtmuIDD3WRbT9Xf/TE5VZF5RyI5KKlQiJzRa4fQVJ sds44cSIxSWyL7mIuD9bGrmqCC/b+MuM4Bo3+jQjKl+dhD1XGrK+48C3t3Xo5ATfEOvW NuBSgeC0Ny5AWK0StTMUSKYpw8GhRltx/yXNW5X8QmC2i3c6isCHS2sCaPSmmaKV1jUJ 7IGbaGvtrLxAlNaiKhWXVVSiVGJQ4ybL6l9rSQyCv/laD0k3MY6NRP6Q2naCFVTGCb9y H5oSeLEk5TCjv8TitlPikfOy2lqxTvtNWlZHCrJMeHwLhD4d9qyMNJP+BO8kK3srmE40 auWw== X-Gm-Message-State: AOJu0YxblDjB3CEYKl11Pdo/7tSC7u9/qcOfocaLWlHYX6FTjDQMco41 7zWK+dfFHSf3Iub+Ui6lVoyuCps4JbYLrlh3BbE= X-Google-Smtp-Source: AGHT+IFSPEKlQSmP4El45E8vBc0Epq9faMbNSeZpAP4jMdZu51DXHIvs320ofYVLaVzWyPCFTx03x5OLJ9JW1V1TgBQ= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a05:6a00:1a87:b0:674:1663:1270 with SMTP id e7-20020a056a001a8700b0067416631270mr6012423pfv.4.1692131697326; Tue, 15 Aug 2023 13:34:57 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:49 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=1987; i=samitolvanen@google.com; h=from:subject; bh=ki2ulz3S+Qkr8AIV7YWzX5sRx6qIOMgf7DcUTP9c/kA=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+Fi+17Bq1AKxHm6SJ6quZHGmn+eubZLGDQWv bi76aScYZaJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7uDxC/4yeVJIZQGthB7VhCrB1mQ9aPlWa4zl/qPRC5GlncPWQV/MwLuo1ExhOVxQY1M+pu82F+m 8PqQG33dyzALoBf1c1t0RJgk8hXrSg+on+5gsx856SahKsjV3+GpMyptJgjPEwnNHk+6wKMMK8A N/pPZs3mhzf+AhJROOGKLcZi1Ux1G5qizF4mL1TsJXLT2ky85P+u9iE+XnlUUOVj0o6+zdRc6E4 T3hgbyVAvct21OxraGNq6gZ4hFfI1gD0Vzy9TsNhYvkKUv6h9A3mjyabLDgnWYIneyLL5Aw8hLM 778p+uRcOJ4tKxAH+wveS7E7YVuTsPIR0MtZTeW4TlXWx284O7O/LdzF0u0artPihMYgdyyeXkT fbWExG0nGe0BC399mEX2b1gOK/3yeNpSy5DzlUGN+aTa7vNStcHxmPR4yr6Qlfw7uDrg4g305Ra AkJHJ/qsdN+6PCM8BiYrPK7icqJ3WmLfwFC54pjnKI93e+BJiBx3TlG5aczelxjKucC60= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-14-samitolvanen@google.com> Subject: [PATCH v2 6/6] lkdtm: Fix CFI_BACKWARD on RISC-V From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On RISC-V, the return address is before the current frame pointer, unlike on most other architectures. Use the correct offset on RISC-V to fix the CFI_BACKWARD test. Signed-off-by: Sami Tolvanen Acked-by: Kees Cook Tested-by: Nathan Chancellor --- drivers/misc/lkdtm/cfi.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/misc/lkdtm/cfi.c b/drivers/misc/lkdtm/cfi.c index fc28714ae3a6..6a33889d0902 100644 --- a/drivers/misc/lkdtm/cfi.c +++ b/drivers/misc/lkdtm/cfi.c @@ -68,12 +68,20 @@ static void lkdtm_CFI_FORWARD_PROTO(void) #define no_pac_addr(addr) \ ((__force __typeof__(addr))((uintptr_t)(addr) | PAGE_OFFSET)) =20 +#ifdef CONFIG_RISCV +/* https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-= cc.adoc#frame-pointer-convention */ +#define FRAME_RA_OFFSET (-1) +#else +#define FRAME_RA_OFFSET 1 +#endif + /* The ultimate ROP gadget. */ static noinline __no_ret_protection void set_return_addr_unchecked(unsigned long *expected, unsigned long *add= r) { /* Use of volatile is to make sure final write isn't seen as a dead store= . */ - unsigned long * volatile *ret_addr =3D (unsigned long **)__builtin_frame_= address(0) + 1; + unsigned long * volatile *ret_addr =3D + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; =20 /* Make sure we've found the right place on the stack before writing it. = */ if (no_pac_addr(*ret_addr) =3D=3D expected) @@ -88,7 +96,8 @@ static noinline void set_return_addr(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store= . */ - unsigned long * volatile *ret_addr =3D (unsigned long **)__builtin_frame_= address(0) + 1; + unsigned long * volatile *ret_addr =3D + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; =20 /* Make sure we've found the right place on the stack before writing it. = */ if (no_pac_addr(*ret_addr) =3D=3D expected) --=20 2.41.0.694.ge786442a9b-goog