From nobody Fri Dec 19 12:48:28 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D017CC83F12 for ; Mon, 28 Aug 2023 19:59:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233551AbjH1T7R (ORCPT ); Mon, 28 Aug 2023 15:59:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49110 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233538AbjH1T6o (ORCPT ); Mon, 28 Aug 2023 15:58:44 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9792C189 for ; Mon, 28 Aug 2023 12:58:40 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-56c556b5aceso3537834a12.1 for ; Mon, 28 Aug 2023 12:58:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252720; x=1693857520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t72XOnwbzECF8GhVlL1pH0HFKftu91Ma7cUPYGPtetA=; b=WZrYBY3ho+HCCDP1FNKRdj5ezMO/wMySjNeFY9G+uOwejtvXlVrwbR8kmj6vddDtUc aqiPJNVaI2KWobFvj85tMLZkTndypS0s4b0dIqIcvCPoBeKWUJjd3sVYgpoScGFlaEw5 BCe/6mRhWNwoLP03aMvdOMvl55wDb+Wwkgpa5Jj/lPc+q2LXL+lIQ7pnEKtdiBr5tN1T dNWa1ELTP6r1biv9I1bMPFdmjmxGW+btzWAfC2cIncymnVG50+n09MBdmqLdyBP2vaD1 H3gM/2eLsiEq3Q+J9YZynhM/wfvf2NO4c70IB0Hy694bou6t1Czg63ZUxC/SLqYahrbi sSKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252720; x=1693857520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t72XOnwbzECF8GhVlL1pH0HFKftu91Ma7cUPYGPtetA=; b=Hmd7WESAt47QCY8oT1LYbTSm2JU+F21X7IsFjMa6Vv275gL0UGayCGb75tG4MJM8JX 94xSntnJtOiM8+lmycqy82/XeSZ0lVyk2n520qUc4OuCrDaKlde8m/J62s924nuXkQzq dWMfJU97vIvjG7nVCME677oLiMoROIlW9pm5EqioP5vcDOl7hEc+6AxSPe0BD2AxaaCw fE3RSXEPL38S0cnnLL416ZzqlBgsT5YTqZnvfVnU3Wplax21NMRJPuzdXQANLoXD6niT FVuxFYZY+QQDnvHbUzjnwRp8CbSGumWDK2xIy5qWikTBIM7fBEOFIcX8wGdJVf+uxjNM UthA== X-Gm-Message-State: AOJu0YxSVUIYsKlHyWtyA1jq46J0x3JcEe3JaJRicU6mB7gp98fP0rSi 0fnp+2ON+G5SFO86MbeWoMzVD7CDYX39+waGGeE= X-Google-Smtp-Source: AGHT+IHbDMp1KPlbpSM031jXdTCRFJuh5xUwCvFAfRqKNHsphJHlbIQMX0YF0Z1FsX6MDgNL+BriZU4ZvulJC66z3Pg= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a63:3d04:0:b0:56b:cd71:6094 with SMTP id k4-20020a633d04000000b0056bcd716094mr4746709pga.1.1693252720142; Mon, 28 Aug 2023 12:58:40 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:35 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=10965; i=samitolvanen@google.com; h=from:subject; bh=UIBcAVgt1wQaHtPmKryqprOkmbFFoOTbmDsyS0eCtqk=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqvloXMV34+z11KS13LdGv1YQAUQNx6Wwp/ OrO+gdAcW6JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7q7bDACIdTbaLnmxOTBA4Mwg2Plne99OzA7EIUAwlRyp+BnE3ns/L8CLKVxiApCq8ThbhFVIrL0 14cAPD8SNMGK8h3G22DU3e/h9+eqjpsaBGIw3/qpl3ybi+qER/66VFv8ZbtyCgadix+wAFfDcBA KTn/Uh2gf3LTrXlDaFZG/cXJ+4QAsTUYMPOgpdNCaMef47uQUYsBSNB9G7kWGOacXWT9OKiH242 QR+q/F/h7zUSjjVs2VM90Mttf2051CUksVkSnbSSRKQJCDbs2w/mx1ylDyFNuTfsbP3jz75mtCw 8bybeR3fvLo741sE+s8DPIzmIh3Lox+7kGHuHZjvFET5H2DYN0BLia7quLSBd1Ji6jDEnxte7At E3D2WyCqfo56vU8UGzvdts2d613IrbSnSSanx3eFeayMlslmMfGECfXz8XkFCbGs5YAGw+5RtiJ Xp8LmpvhY1sa1l0W6YWZvq2KxnhAmtmKzb2fP6cnGCpqEUoZzYQJK1YPXMZP/8UOBemrs= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-9-samitolvanen@google.com> Subject: [PATCH v3 1/6] riscv: VMAP_STACK overflow detection thread-safe From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Jisheng Zhang , Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Deepak Gupta commit 31da94c25aea ("riscv: add VMAP_STACK overflow detection") added support for CONFIG_VMAP_STACK. If overflow is detected, CPU switches to `shadow_stack` temporarily before switching finally to per-cpu `overflow_stack`. If two CPUs/harts are racing and end up in over flowing kernel stack, one or both will end up corrupting each other state because `shadow_stack` is not per-cpu. This patch optimizes per-cpu overflow stack switch by directly picking per-cpu `overflow_stack` and gets rid of `shadow_stack`. Following are the changes in this patch - Defines an asm macro to obtain per-cpu symbols in destination register. - In entry.S, when overflow is detected, per-cpu overflow stack is located using per-cpu asm macro. Computing per-cpu symbol requires a temporary register. x31 is saved away into CSR_SCRATCH (CSR_SCRATCH is anyways zero since we're in kernel). Please see Links for additional relevant disccussion and alternative solution. Tested by `echo EXHAUST_STACK > /sys/kernel/debug/provoke-crash/DIRECT` Kernel crash log below Insufficient stack space to handle exception!/debug/provoke-crash/DIRECT Task stack: [0xff20000010a98000..0xff20000010a9c000] Overflow stack: [0xff600001f7d98370..0xff600001f7d99370] CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) epc : __memset+0x60/0xfc ra : recursive_loop+0x48/0xc6 [lkdtm] epc : ffffffff808de0e4 ra : ffffffff0163a752 sp : ff20000010a97e80 gp : ffffffff815c0330 tp : ff600000820ea280 t0 : ff20000010a97e88 t1 : 000000000000002e t2 : 3233206874706564 s0 : ff20000010a982b0 s1 : 0000000000000012 a0 : ff20000010a97e88 a1 : 0000000000000000 a2 : 0000000000000400 a3 : ff20000010a98288 a4 : 0000000000000000 a5 : 0000000000000000 a6 : fffffffffffe43f0 a7 : 00007fffffffffff s2 : ff20000010a97e88 s3 : ffffffff01644680 s4 : ff20000010a9be90 s5 : ff600000842ba6c0 s6 : 00aaaaaac29e42b0 s7 : 00fffffff0aa3684 s8 : 00aaaaaac2978040 s9 : 0000000000000065 s10: 00ffffff8a7cad10 s11: 00ffffff8a76a4e0 t3 : ffffffff815dbaf4 t4 : ffffffff815dbaf4 t5 : ffffffff815dbab8 t6 : ff20000010a9bb48 status: 0000000200000120 badaddr: ff20000010a97e88 cause: 000000000000000f Kernel panic - not syncing: Kernel stack overflow CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) Call Trace: [] dump_backtrace+0x30/0x38 [] show_stack+0x40/0x4c [] dump_stack_lvl+0x44/0x5c [] dump_stack+0x18/0x20 [] panic+0x126/0x2fe [] walk_stackframe+0x0/0xf0 [] recursive_loop+0x48/0xc6 [lkdtm] SMP: stopping secondary CPUs ---[ end Kernel panic - not syncing: Kernel stack overflow ]--- Cc: Guo Ren Cc: Jisheng Zhang Link: https://lore.kernel.org/linux-riscv/Y347B0x4VUNOd6V7@xhacker/T/#t Link: https://lore.kernel.org/lkml/20221124094845.1907443-1-debug@rivosinc.= com/ Signed-off-by: Deepak Gupta Co-developed-by: Sami Tolvanen Signed-off-by: Sami Tolvanen Acked-by: Guo Ren Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm-prototypes.h | 1 - arch/riscv/include/asm/asm.h | 22 ++++++++ arch/riscv/include/asm/thread_info.h | 3 -- arch/riscv/kernel/asm-offsets.c | 1 + arch/riscv/kernel/entry.S | 70 ++++--------------------- arch/riscv/kernel/traps.c | 36 +------------ 6 files changed, 34 insertions(+), 99 deletions(-) diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/a= sm/asm-prototypes.h index 61ba8ed43d8f..36b955c762ba 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -25,7 +25,6 @@ DECLARE_DO_ERROR_INFO(do_trap_ecall_s); DECLARE_DO_ERROR_INFO(do_trap_ecall_m); DECLARE_DO_ERROR_INFO(do_trap_break); =20 -asmlinkage unsigned long get_overflow_stack(void); asmlinkage void handle_bad_stack(struct pt_regs *regs); asmlinkage void do_page_fault(struct pt_regs *regs); asmlinkage void do_irq(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 114bbadaef41..bfb4c26f113c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -82,6 +82,28 @@ .endr .endm =20 +#ifdef CONFIG_SMP +#ifdef CONFIG_32BIT +#define PER_CPU_OFFSET_SHIFT 2 +#else +#define PER_CPU_OFFSET_SHIFT 3 +#endif + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT + la \dst, __per_cpu_offset + add \dst, \dst, \tmp + REG_L \tmp, 0(\dst) + la \dst, \sym + add \dst, \dst, \tmp +.endm +#else /* CONFIG_SMP */ +.macro asm_per_cpu dst sym tmp + la \dst, \sym +.endm +#endif /* CONFIG_SMP */ + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/= thread_info.h index 1833beb00489..d18ce0113ca1 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -34,9 +34,6 @@ =20 #ifndef __ASSEMBLY__ =20 -extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)]; -extern unsigned long spin_shadow_stack; - #include #include =20 diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index d6a75aac1d27..9f535d5de33f 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,7 @@ void asm_offsets(void) OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); =20 + OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3d11aa3af105 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -10,9 +10,11 @@ #include #include #include +#include #include #include #include +#include =20 SYM_CODE_START(handle_exception) /* @@ -170,67 +172,15 @@ SYM_CODE_END(ret_from_exception) =20 #ifdef CONFIG_VMAP_STACK SYM_CODE_START_LOCAL(handle_kernel_stack_overflow) - /* - * Takes the psuedo-spinlock for the shadow stack, in case multiple - * harts are concurrently overflowing their kernel stacks. We could - * store any value here, but since we're overflowing the kernel stack - * already we only have SP to use as a scratch register. So we just - * swap in the address of the spinlock, as that's definately non-zero. - * - * Pairs with a store_release in handle_bad_stack(). - */ -1: la sp, spin_shadow_stack - REG_AMOSWAP_AQ sp, sp, (sp) - bnez sp, 1b - - la sp, shadow_stack - addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE - - //save caller register to shadow stack - addi sp, sp, -(PT_SIZE_ON_STACK) - REG_S x1, PT_RA(sp) - REG_S x5, PT_T0(sp) - REG_S x6, PT_T1(sp) - REG_S x7, PT_T2(sp) - REG_S x10, PT_A0(sp) - REG_S x11, PT_A1(sp) - REG_S x12, PT_A2(sp) - REG_S x13, PT_A3(sp) - REG_S x14, PT_A4(sp) - REG_S x15, PT_A5(sp) - REG_S x16, PT_A6(sp) - REG_S x17, PT_A7(sp) - REG_S x28, PT_T3(sp) - REG_S x29, PT_T4(sp) - REG_S x30, PT_T5(sp) - REG_S x31, PT_T6(sp) - - la ra, restore_caller_reg - tail get_overflow_stack - -restore_caller_reg: - //save per-cpu overflow stack - REG_S a0, -8(sp) - //restore caller register from shadow_stack - REG_L x1, PT_RA(sp) - REG_L x5, PT_T0(sp) - REG_L x6, PT_T1(sp) - REG_L x7, PT_T2(sp) - REG_L x10, PT_A0(sp) - REG_L x11, PT_A1(sp) - REG_L x12, PT_A2(sp) - REG_L x13, PT_A3(sp) - REG_L x14, PT_A4(sp) - REG_L x15, PT_A5(sp) - REG_L x16, PT_A6(sp) - REG_L x17, PT_A7(sp) - REG_L x28, PT_T3(sp) - REG_L x29, PT_T4(sp) - REG_L x30, PT_T5(sp) - REG_L x31, PT_T6(sp) + /* we reach here from kernel context, sscratch must be 0 */ + csrrw x31, CSR_SCRATCH, x31 + asm_per_cpu sp, overflow_stack, x31 + li x31, OVERFLOW_STACK_SIZE + add sp, sp, x31 + /* zero out x31 again and restore x31 */ + xor x31, x31, x31 + csrrw x31, CSR_SCRATCH, x31 =20 - //load per-cpu overflow stack - REG_L sp, -8(sp) addi sp, sp, -(PT_SIZE_ON_STACK) =20 //save context to overflow stack diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index f798c853bede..a05905d88802 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -400,48 +400,14 @@ int is_valid_bugaddr(unsigned long pc) #endif /* CONFIG_GENERIC_BUG */ =20 #ifdef CONFIG_VMAP_STACK -/* - * Extra stack space that allows us to provide panic messages when the ker= nel - * has overflowed its stack. - */ -static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16); -/* - * A temporary stack for use by handle_kernel_stack_overflow. This is use= d so - * we can call into C code to get the per-hart overflow stack. Usage of t= his - * stack must be protected by spin_shadow_stack. - */ -long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE/sizeof(long)] __aligned(16); - -/* - * A pseudo spinlock to protect the shadow stack from being used by multip= le - * harts concurrently. This isn't a real spinlock because the lock side m= ust - * be taken without a valid stack and only a single register, it's only ta= ken - * while in the process of panicing anyway so the performance and error - * checking a proper spinlock gives us doesn't matter. - */ -unsigned long spin_shadow_stack; - -asmlinkage unsigned long get_overflow_stack(void) -{ - return (unsigned long)this_cpu_ptr(overflow_stack) + - OVERFLOW_STACK_SIZE; -} =20 asmlinkage void handle_bad_stack(struct pt_regs *regs) { unsigned long tsk_stk =3D (unsigned long)current->stack; unsigned long ovf_stk =3D (unsigned long)this_cpu_ptr(overflow_stack); =20 - /* - * We're done with the shadow stack by this point, as we're on the - * overflow stack. Tell any other concurrent overflowing harts that - * they can proceed with panicing by releasing the pseudo-spinlock. - * - * This pairs with an amoswap.aq in handle_kernel_stack_overflow. - */ - smp_store_release(&spin_shadow_stack, 0); - console_verbose(); =20 pr_emerg("Insufficient stack space to handle exception!\n"); --=20 2.42.0.rc2.253.gd59a3bf2b4-goog From nobody Fri Dec 19 12:48:28 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F442C83F14 for ; Mon, 28 Aug 2023 19:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233559AbjH1T7V (ORCPT ); Mon, 28 Aug 2023 15:59:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233539AbjH1T6p (ORCPT ); Mon, 28 Aug 2023 15:58:45 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0AEB18B for ; Mon, 28 Aug 2023 12:58:42 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d795e8483f8so5283661276.1 for ; Mon, 28 Aug 2023 12:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252722; x=1693857522; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SyPtEI03Xg1fBX3TZi3WfjXMOPTYu0dO+LXPL6FV3JY=; b=aeycJH5+6uxe8GlNC+YyRdXkjill87qRi7L6FkS96p4g2q1s4N94P2OZ5QDES1S1TC MfwdeCDtem0I3jQ+onVBjqfV8svuERSpLpxq6jjZ+Ufji5mcFbBp+g6ChKwsHhVNFNna LQonssHhBVrDczuTEO5l5yf74aTZFJ60kJfW3XcL7yn1/dPDDGFUudDGC9NGcXlPudY4 BuAw3Yfd3OAcCpnpMMIFLgvMXpik/3/GT/wQbEzdRIYCNBGI57sXpX998s0bzbLp+XDr +aCFpNwXtn4mlGFVj/IbkR53Q4xdpfP+oJruefxqPB++Dgcy9SSWoWGeTGKWxGTCKkQQ XlFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252722; x=1693857522; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SyPtEI03Xg1fBX3TZi3WfjXMOPTYu0dO+LXPL6FV3JY=; b=aWY8AIB3Leh4wrpq0xXfMQrpB4+23VM7FmS7KELWxTuvB53por7UqeuoigJQXLA5mW vSNqZ7mfC83m8WxIUkguBon95e9kFXwPLqWLj9VNHRMiIUqCGRASEQxCAH+6k5wyUXtr LYEonfKhRCZoVJTNZg5PIzy0ptZzndCZKXNcLwVi+ZS6ZAqfogi8RBg9zQ1aJ5nbyGNo Owzcnf9eYkkEKcX2AVg/R4OmhH3g1iHPcebXtlKxRtwXXIOEblsyzKXzrQEQiv/Lxhyh ZXqiEQdNb2M6zDVcD37vjWvrg4VcRUO5AO8GP1y+qHpWOgbxhHquCMSpCQTVzOo5XPku 2OSg== X-Gm-Message-State: AOJu0YwpwyNRP5nisvOy5Kkuba/oudWNkKyLIZB/yiG/MNR5x8ScBGmi wyzfxTzRifzKxEoIOOC6tbfBiQvOVEx8UOGgUMI= X-Google-Smtp-Source: AGHT+IFJVA5tn0i/teRMdLPB4BUylObsezBXemw+f2wLcIS31CUsFOoZ7JWpR/ILow1bMm25R+eFhC5JiSZJ6WzUrIw= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a05:6902:100d:b0:d47:3d35:1604 with SMTP id w13-20020a056902100d00b00d473d351604mr24925ybt.2.1693252722088; Mon, 28 Aug 2023 12:58:42 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:36 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=6654; i=samitolvanen@google.com; h=from:subject; bh=wnBUbn/4zTqpJwVvv6oK/jRIJiyG33tTx6IcCOYR/uQ=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqgcThGLqOU6/Nh1FKa2ioMm4jOSxmZxX+a sZvOfWhvQCJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7qN/DACO5167AKEHcx/87vqquJtHBN9KGNAuOkAYwEvKikfyASynXP4pukx0I5Ac1cOyZbdSKL8 Ce0qro1KPSj+B7uJLugyfen0UVTnlQTtNm24YRU2AnuqLCWm7zflBibZwWzhTqJsGZzImyJVLkB Yr3GszptMkh3O1gX6Tv2tmDjb+qkBlUMMRdltW45f/g9ibmbfcG5SO/vWQqcc3SMWqsc6m7JYwk keXj7naHVlBrPYSo+984uuAqdsHZ9q0aEVwqYIJOJIWhP5XgrEZZe3AJIp8OF8ZaefGmW5LiQ0I ixCukGxRFhOHpj9R9wdn6BFhLwcyzYzLwTqRW+Y7ko261wZrwLuApeBe44ru+05jqQ5SVj+y501 vjpXz+IlnWoGzvDP7e5QFY1rNNe2X4QdPODLjs6C/p2TfoPH/k92JNa6S7ITaa1o90sUmL9p4bO NCj0TY91rJ1SesDvIqMSsrklIMv4du5khyVUbh1YZyymDHBkAXomC5VVAKeZHqxyNk6oQ= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-10-samitolvanen@google.com> Subject: [PATCH v3 2/6] riscv: Deduplicate IRQ stack switching From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redudant inline assembly. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor Reviewed-by: Guo Ren --- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 30 +++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 35 +++++++----------------------- arch/riscv/kernel/traps.c | 32 ++++----------------------- 6 files changed, 55 insertions(+), 55 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index bfb4c26f113c..8e446be2d57c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -104,6 +104,11 @@ .endm #endif /* CONFIG_SMP */ =20 +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/irq_stack.h b/arch/riscv/include/asm/ir= q_stack.h index e4042d297580..6441ded3b0cf 100644 --- a/arch/riscv/include/asm/irq_stack.h +++ b/arch/riscv/include/asm/irq_stack.h @@ -12,6 +12,9 @@ =20 DECLARE_PER_CPU(ulong *, irq_stack_ptr); =20 +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); + #ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMA= P'd diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index 9f535d5de33f..0af8860f9d68 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -14,6 +14,7 @@ #include #include #include +#include #include =20 void asm_offsets(void); @@ -480,4 +481,8 @@ void asm_offsets(void) OFFSET(KERNEL_MAP_VIRT_ADDR, kernel_mapping, virt_addr); OFFSET(SBI_HART_BOOT_TASK_PTR_OFFSET, sbi_hart_boot_data, task_ptr); OFFSET(SBI_HART_BOOT_STACK_PTR_OFFSET, sbi_hart_boot_data, stack_ptr); + + DEFINE(STACKFRAME_SIZE_ON_STACK, ALIGN(sizeof(struct stackframe), STACK_A= LIGN)); + OFFSET(STACKFRAME_FP, stackframe, fp); + OFFSET(STACKFRAME_RA, stackframe, ra); } diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 3d11aa3af105..a306562636e4 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -218,6 +218,36 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) =20 +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -STACKFRAME_SIZE_ON_STACK + REG_S ra, STACKFRAME_RA(sp) + REG_S s0, STACKFRAME_FP(sp) + addi s0, sp, STACKFRAME_SIZE_ON_STACK + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -STACKFRAME_SIZE_ON_STACK + REG_L ra, STACKFRAME_RA(sp) + REG_L s0, STACKFRAME_FP(sp) + addi sp, sp, STACKFRAME_SIZE_ON_STACK + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored. diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index a8efa053c4a5..95dafdcbd135 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -61,35 +61,16 @@ static void init_irq_stacks(void) #endif /* CONFIG_VMAP_STACK */ =20 #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq(struct pt_regs *regs) +{ + __do_softirq(); +} + void do_softirq_own_stack(void) { -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp =3D per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "call __do_softirq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", -#ifndef CONFIG_FRAME_POINTER - "s0", -#endif - "memory"); - } else -#endif + if (on_thread_stack()) + call_on_irq_stack(NULL, ___do_softirq); + else __do_softirq(); } #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index a05905d88802..1fe6b475cdfb 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -350,34 +350,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *= regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) { irqentry_state_t state =3D irqentry_enter(regs); -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp =3D per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", -#ifndef CONFIG_FRAME_POINTER - "s0", -#endif - "memory"); - } else -#endif + + if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack()) + call_on_irq_stack(regs, handle_riscv_irq); + else handle_riscv_irq(regs); =20 irqentry_exit(regs, state); --=20 2.42.0.rc2.253.gd59a3bf2b4-goog From nobody Fri Dec 19 12:48:28 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41EE6C83F15 for ; Mon, 28 Aug 2023 19:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233567AbjH1T7Y (ORCPT ); Mon, 28 Aug 2023 15:59:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233540AbjH1T6q (ORCPT ); Mon, 28 Aug 2023 15:58:46 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91DE1189 for ; Mon, 28 Aug 2023 12:58:44 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d74a012e6a6so4432254276.2 for ; Mon, 28 Aug 2023 12:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252724; x=1693857524; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GN6X1nlK+F0kjkEC50FG9os31FAuUCJYmCM49+1obnI=; b=q81rXqLshiF0sln/xS3h/1Wt74xT2Rdg2j7FkroNKWNJx+s5YHbY+oxX80kCoOEdXg Z7tbUBEmnha1x55o337Yu6Lp+PK9LnYIpjNDnXpf7Vvob1fBYdUFHdVNRYzgfh16aSLv eFNKGP5rRTkOAMSM3mrjAPNqs5knGLIBbNRZAC6qGU+CNi/lzLXAiQhO62xa0d/tNvJ+ HZQCnftScs5HC5H/QHtUIo0uBGDaDIJXYW2D2SPFvrSpGGn81MfWM1OEFhJRudgn2+le WexQrz+RIoMq4d/FZyHuGt5KMF8ZH4GobjeJX5y+t0Ma6T4e2S48xP0NJhR5CYYZChPH OKqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252724; x=1693857524; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GN6X1nlK+F0kjkEC50FG9os31FAuUCJYmCM49+1obnI=; b=GEyoWvVVKpe4fVkd5Y7FcmKYnxaOtIH8wdoTXyG/jv2Xd1KrgzEUE+nKWalR/iD1dJ W5574ZI+82dE4zwi63/3iybRDQQTakPd1GcO/xYCUu+p0KHgfuEFW8+5HnpCDqZ8JTJU QQwecIwGFzOWTB/8dKZJ5ZtI4Wdzp4hL87VZetRtEtjHe8HfXczOcWrLvJ2Mfe+0F0Qp 6h05ZcUb/MHU7nPAEjWdB5LeTvuTI1e4vQwzCYGHY2LnBcosQ/g9LoaMMIsppzpmid49 0c/JKYWo6G+afVlFIyBo4hahU3vD86L2bT5ho884eVHBuEJnGq18fy303HcbYT75Emli oSIg== X-Gm-Message-State: AOJu0YzhnvWgasedoyqBOHHsBQdyL+es3z61JRGSxBzBDFYdYvgCQzYF i4BCbiW6JeavF90x1CpP/yF729b+0lcwpheSi70= X-Google-Smtp-Source: AGHT+IH/uHwJj6+qOeylfRyEPqpB+nnMtItRxR7gf+ULc0gWYu3yZPJVS4kkyblR9fUvFWo2HvJU3EuGasR5rTd2B1w= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:9011:0:b0:d5d:b03c:49b2 with SMTP id s17-20020a259011000000b00d5db03c49b2mr820198ybl.11.1693252723904; Mon, 28 Aug 2023 12:58:43 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:37 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3271; i=samitolvanen@google.com; h=from:subject; bh=Chgku5okNNw1TPF+7h/tJxx5VnEIks6hZ2ypKsZTKYg=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqRbtkifAND8x/XfLzfQWcnICH3txr02qPt 4eiKf2aYjSJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7sqPC/4yfD8F4LJhON/0j9LDQmdASW7omk5GP9QTz9u9V2kMfasvvS5/J6shUYxruhqnrbLQBUp oZG+GjK1FZXYmm2NhWg4FLwoGp73evhYqS3wI+Ft3xabSmr5Na3lU51SZXDGqV8uyQ7euwUibuM 1IB+PWku6qCOxTHa+PmjyOlXV/KauzFtIzwmz+0x5d45qcRehgOD29B8x93CCG4ocGo16RTP8LI F/iGt9KvESUFJHI8k0VjTovZNvf6YxjaSUpyeHxe3VVQSccGWV/T3h55eOORae0j0Myri46bWi6 G6OIOjCjM5wvPYvwQTVyYFnbv5wTS0DJqhUWZt3ZEVviB/dFtXNBfxwtPWVuNItQMaZQ/8iaxqH ZZt1G2FWe/DvsQ5lHk95L6xpiDHTU8W8m68q4rh/0qju4f8K6jxJcN6sSbo2s98UyF1YVfErhSD UX5opXBgLT/rq9utajC5hslrYEgFPNP1pUHjVWRg8zD8f17xYD7/57e0gz+NISpgLceIs= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-11-samitolvanen@google.com> Subject: [PATCH v3 3/6] riscv: Move global pointer loading to a macro From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In Clang 17, -fsanitize=3Dshadow-call-stack uses the newly declared platform register gp for storing shadow call stack pointers. As this is obviously incompatible with gp relaxation, in preparation for CONFIG_SHADOW_CALL_STACK support, move global pointer loading to a single macro, which we can cleanly disable when SCS is used instead. Link: https://reviews.llvm.org/rGaa1d2693c256 Link: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e= 6eeb51f0cb7b8819e50da6d2444d769 Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm.h | 8 ++++++++ arch/riscv/kernel/entry.S | 6 ++---- arch/riscv/kernel/head.S | 15 +++------------ arch/riscv/kernel/suspend_entry.S | 5 +---- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 8e446be2d57c..f34dd1a526a1 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,14 @@ REG_L \dst, 0(\dst) .endm =20 +/* load __global_pointer to gp */ +.macro load_global_pointer +.option push +.option norelax + la gp, __global_pointer$ +.option pop +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index a306562636e4..6215dcf2e83b 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -75,10 +75,8 @@ _save_context: csrw CSR_SCRATCH, x0 =20 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer + move a0, sp /* pt_regs */ la ra, ret_from_exception =20 diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 11c3b94c4534..79b5a863c782 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -110,10 +110,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a0 =20 /* Reload the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer =20 /* * Switch to kernel page tables. A full fence is necessary in order to @@ -134,10 +131,7 @@ secondary_start_sbi: csrw CSR_IP, zero =20 /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer =20 /* * Disable FPU & VECTOR to detect illegal usage of @@ -228,10 +222,7 @@ pmp_done: #endif /* CONFIG_RISCV_M_MODE */ =20 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer =20 /* * Disable FPU & VECTOR to detect illegal usage of diff --git a/arch/riscv/kernel/suspend_entry.S b/arch/riscv/kernel/suspend_= entry.S index 12b52afe09a4..556a4b166d8c 100644 --- a/arch/riscv/kernel/suspend_entry.S +++ b/arch/riscv/kernel/suspend_entry.S @@ -60,10 +60,7 @@ END(__cpu_suspend_enter) =20 ENTRY(__cpu_resume_enter) /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer =20 #ifdef CONFIG_MMU /* Save A0 and A1 */ --=20 2.42.0.rc2.253.gd59a3bf2b4-goog From nobody Fri Dec 19 12:48:28 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 559A0C83F18 for ; Mon, 28 Aug 2023 19:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233572AbjH1T70 (ORCPT ); Mon, 28 Aug 2023 15:59:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233542AbjH1T6t (ORCPT ); Mon, 28 Aug 2023 15:58:49 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F503189 for ; Mon, 28 Aug 2023 12:58:46 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d7493fcd829so4440101276.3 for ; Mon, 28 Aug 2023 12:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252725; x=1693857525; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aHEpjDWpZKheS5AgI4FhMURxdLRiA7f+7eI6PSwoShs=; b=oU7u4yIwCqEfIEncvirifpc8jGp8C2PonLKbfLyys1pYIlGszZDoAooaF/rA0xipsa tGEdMGobqAe00UCFtvQUmb7m/LoLfLDhGkakB7CJCf2UzXywKzO4OGDJx36TWR6p1UpO gH+CFvy2T/d4aEliQy8VqP+pcWw0DQUKphP1cBN2b9Cr6KoXrSCWy2xSHa4O70TgLTtV kcouhP3xTN66BUOTos30E+RKqDSHDRJfp+8zYKZh20qXio3GEQ7TFh7/q3ws6NvVxZiO 9DkaSSJqea9knPB9Fi+KfetI4RjF02r8Q43oiZstz6SsA+TRNeaIoRtwg6FkO64uo7DC 3gOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252725; x=1693857525; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aHEpjDWpZKheS5AgI4FhMURxdLRiA7f+7eI6PSwoShs=; b=HcS1PnfFN11lFEUGQKEnbbrMfdEL0ybW62OFNNjJ61pVAFw+n1qERFK04ukRUSqisH INh9UlKTDRvlLHQyZH/eATWyddmAGC90xaJyQBA3e3t9+/panaYChk8mJ5gqVncVLsWi 638iZ4DeZUDzYYzLCpHlEsscpRSm5DipwXIeCYEYrOqri0j/KarfP2UYvNQ57ZFovBvU wdzBlhHTtL1k7B+ok2KXQupeq7jXt289uHwcN2fVhCuj5r9D0WQgTHKfrpkJlTjmv6Mx Iu6un+NmcJBSk89YcfcyVaZHzf4GNzQEHfgF1qG5ekGghujcgKBZVNWi1JY0I6/bZotH hJHA== X-Gm-Message-State: AOJu0Yx+X1cL8eZ4DlMIU8YGh9MQdVK/pH5xzFfC/tcgS1fq/T5d3gqy XqsOdkVTQgOqYm0fFmiUxi9inQt4a6sSBOXawkI= X-Google-Smtp-Source: AGHT+IFPe1xW/cZcqMnEmhJXa9gkio6zfEjjBVIssP4qY8EDsEHxCzdXB9iRA0bQANGRB+YXwQYzP941dcW/ssXB124= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:d750:0:b0:d7a:bfcf:2d7 with SMTP id o77-20020a25d750000000b00d7abfcf02d7mr386698ybg.6.1693252725580; Mon, 28 Aug 2023 12:58:45 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:38 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=9699; i=samitolvanen@google.com; h=from:subject; bh=yzq9X6lovJwCj7M3MxLTb+ZbxWOKAC+68l1vG/LS5Go=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqnP8GY39pSf11gSk0fB0bvEpXRF2ZEN/E+ G3d+76nn0OJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7gsfC/9z0ZHF2b9t4rZr3ySGuRBhgmJMEDj1qu+NhecypGTqBV+8Cf+bL2247doUWGkApt+72LI Pie3DKwp/m/VGmPDKi5q/CHYiJLP8680OGP1+bTrgYosZW8hyJ9IhiKBiO1KBRy47o/T50jlhIc VLTG0/pv3Y+jeXP0qgdkSrHZlTv7QZPhn289pObdT/2Ec0VHRVw53yDsfnirmd1Z2H+N5PkP/6f 1xreDmv6SydopVQvfabiJYbA2ISjihl6rFRDqU2ArU8zwBiuubsGlLXzPpeoKDiqvBWN86lYmXY X2DoWBrpXBLJwxNGk+gdnntxZXM5+hHaCh1D1a5jW2C9OajMf4lqHP9lG/XUDynL1Gmxxrid5JG xgY811eAOdQva10xkT5N8Zv/OzuURC3GqI2P2Dhr0c3G4WTomuJ13WFUOPg5QDhJsU3a5B9DZv/ f1ZMyp4Vr2IT2tV5madananCzjiSG8k+hxqtfu1tvhCKQ4SOJW10fDi80IvVzGfHZqXy8= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-12-samitolvanen@google.com> Subject: [PATCH v3 4/6] riscv: Implement Shadow Call Stack From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement CONFIG_SHADOW_CALL_STACK for RISC-V. When enabled, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning, which makes it harder to corrupt the return address through a stack overflow, for example. The active shadow call stack pointer is stored in the gp register, which makes SCS incompatible with gp relaxation. Use --no-relax-gp to ensure gp relaxation is disabled and disable global pointer loading. Add SCS pointers to struct thread_info, implement SCS initialization, and task switching Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/Kconfig | 6 ++++ arch/riscv/Makefile | 4 +++ arch/riscv/include/asm/asm.h | 6 ++++ arch/riscv/include/asm/scs.h | 47 ++++++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 13 ++++++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/entry.S | 11 +++++++ arch/riscv/kernel/head.S | 4 +++ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 +++ 10 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/scs.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bea7b73e895d..93122634f9c2 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -46,6 +46,7 @@ config RISCV select ARCH_SUPPORTS_HUGETLBFS if MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU select ARCH_SUPPORTS_PER_VMA_LOCK if MMU + select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU @@ -169,6 +170,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=3D8) =20 +config HAVE_SHADOW_CALL_STACK + def_bool $(cc-option,-fsanitize=3Dshadow-call-stack) + # https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6e= eb51f0cb7b8819e50da6d2444d769 + depends on $(ld-option,--no-relax-gp) + config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT default 8 diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6ec6d52a4180..e518a74640fb 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -55,6 +55,10 @@ endif endif endif =20 +ifeq ($(CONFIG_SHADOW_CALL_STACK),y) + KBUILD_LDFLAGS +=3D --no-relax-gp +endif + # ISA string setting riscv-march-$(CONFIG_ARCH_RV32I) :=3D rv32ima riscv-march-$(CONFIG_ARCH_RV64I) :=3D rv64ima diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index f34dd1a526a1..b0487b39e674 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,11 @@ REG_L \dst, 0(\dst) .endm =20 +#ifdef CONFIG_SHADOW_CALL_STACK +/* gp is used as the shadow call stack pointer instead */ +.macro load_global_pointer +.endm +#else /* load __global_pointer to gp */ .macro load_global_pointer .option push @@ -116,6 +121,7 @@ la gp, __global_pointer$ .option pop .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ =20 /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h new file mode 100644 index 000000000000..94726ea773e3 --- /dev/null +++ b/arch/riscv/include/asm/scs.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifdef __ASSEMBLY__ +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* Load init_shadow_call_stack to gp. */ +.macro scs_load_init_stack + la gp, init_shadow_call_stack + XIP_FIXUP_OFFSET gp +.endm + +/* Load task_scs_sp(current) to gp. */ +.macro scs_load_current + REG_L gp, TASK_TI_SCS_SP(tp) +.endm + +/* Load task_scs_sp(current) to gp, but only if tp has changed. */ +.macro scs_load_current_if_task_changed prev + beq \prev, tp, _skip_scs + scs_load_current +_skip_scs: +.endm + +/* Save gp to task_scs_sp(current). */ +.macro scs_save_current + REG_S gp, TASK_TI_SCS_SP(tp) +.endm + +#else /* CONFIG_SHADOW_CALL_STACK */ + +.macro scs_load_init_stack +.endm +.macro scs_load_current +.endm +.macro scs_load_current_if_task_changed prev +.endm +.macro scs_save_current +.endm + +#endif /* CONFIG_SHADOW_CALL_STACK */ +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/= thread_info.h index d18ce0113ca1..574779900bfb 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -57,8 +57,20 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ +#ifdef CONFIG_SHADOW_CALL_STACK + void *scs_base; + void *scs_sp; +#endif }; =20 +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base =3D init_shadow_call_stack, \ + .scs_sp =3D init_shadow_call_stack, +#else +#define INIT_SCS +#endif + /* * macros/functions for gaining access to the thread information structure * @@ -68,6 +80,7 @@ struct thread_info { { \ .flags =3D 0, \ .preempt_count =3D INIT_PREEMPT_COUNT, \ + INIT_SCS \ } =20 void arch_release_task_struct(struct task_struct *tsk); diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index 0af8860f9d68..a03129f40c46 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,9 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); +#ifdef CONFIG_SHADOW_CALL_STACK + OFFSET(TASK_TI_SCS_SP, task_struct, thread_info.scs_sp); +#endif =20 OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 6215dcf2e83b..52793193a763 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -9,6 +9,7 @@ =20 #include #include +#include #include #include #include @@ -77,6 +78,9 @@ _save_context: /* Load the global pointer */ load_global_pointer =20 + /* Load the kernel shadow call stack pointer if coming from userspace */ + scs_load_current_if_task_changed s5 + move a0, sp /* pt_regs */ la ra, ret_from_exception =20 @@ -123,6 +127,9 @@ SYM_CODE_START_NOALIGN(ret_from_exception) addi s0, sp, PT_SIZE_ON_STACK REG_S s0, TASK_TI_KERNEL_SP(tp) =20 + /* Save the kernel shadow call stack pointer */ + scs_save_current + /* * Save TP into the scratch register , so we can find the kernel data * structures again. @@ -275,6 +282,8 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + /* Save the kernel shadow call stack pointer */ + scs_save_current /* Restore context from next->thread */ REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) @@ -292,6 +301,8 @@ SYM_FUNC_START(__switch_to) REG_L s11, TASK_THREAD_S11_RA(a4) /* The offset of thread_info in task_struct is zero. */ move tp, a1 + /* Switch to the next shadow call stack */ + scs_load_current ret SYM_FUNC_END(__switch_to) =20 diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 79b5a863c782..c3d0ee77483b 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "efi-header.S" =20 @@ -153,6 +154,7 @@ secondary_start_sbi: XIP_FIXUP_OFFSET a3 add a3, a3, a1 REG_L sp, (a3) + scs_load_current =20 .Lsecondary_start_common: =20 @@ -293,6 +295,7 @@ clear_bss_done: la sp, init_thread_union + THREAD_SIZE XIP_FIXUP_OFFSET sp addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_BUILTIN_DTB la a0, __dtb_start XIP_FIXUP_OFFSET a0 @@ -311,6 +314,7 @@ clear_bss_done: la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack =20 #ifdef CONFIG_KASAN call kasan_early_init diff --git a/arch/riscv/kernel/vdso/Makefile b/arch/riscv/kernel/vdso/Makef= ile index 6b1dba11bf6d..48c362c0cb3d 100644 --- a/arch/riscv/kernel/vdso/Makefile +++ b/arch/riscv/kernel/vdso/Makefile @@ -36,7 +36,7 @@ CPPFLAGS_vdso.lds +=3D -DHAS_VGETTIMEOFDAY endif =20 # Disable -pg to prevent insert call site -CFLAGS_REMOVE_vgettimeofday.o =3D $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_vgettimeofday.o =3D $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) =20 # Disable profiling and instrumentation for VDSO code GCOV_PROFILE :=3D n diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index dc20e166983e..d5d60c040560 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -77,6 +77,10 @@ ifdef CONFIG_STACKPROTECTOR_STRONG PURGATORY_CFLAGS_REMOVE +=3D -fstack-protector-strong endif =20 +ifdef CONFIG_SHADOW_CALL_STACK +PURGATORY_CFLAGS_REMOVE +=3D $(CC_FLAGS_SCS) +endif + CFLAGS_REMOVE_purgatory.o +=3D $(PURGATORY_CFLAGS_REMOVE) CFLAGS_purgatory.o +=3D $(PURGATORY_CFLAGS) =20 --=20 2.42.0.rc2.253.gd59a3bf2b4-goog From nobody Fri Dec 19 12:48:28 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66517C83F17 for ; Mon, 28 Aug 2023 19:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233580AbjH1T73 (ORCPT ); Mon, 28 Aug 2023 15:59:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233543AbjH1T6u (ORCPT ); Mon, 28 Aug 2023 15:58:50 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07ED8188 for ; Mon, 28 Aug 2023 12:58:48 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5956735bdc3so2946917b3.0 for ; Mon, 28 Aug 2023 12:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252727; x=1693857527; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=03jC+OBVw6NtqXGsEGvclVCdQhu3EQx6jlJc9yvphdo=; b=E+bfdH1lDaLKrKcjWlt3v7mLrI0mXX3/qYWVcii5PVVLZRAYiy1nsib+qeG6H8MOuT 9OhEHYpNgvWxBky8trYBSEV2JubnwBjeu7di8zHvRoZf0j/YO3KJeryt/leOQaOk8U0+ E0YLLPMtCwqk+7XZu6/A3/70OtYGnMDc9dICr0zRd4xccIbb8Fl0DxJGU04K1ySPvtRA rOrJmojR9TLloOAjWkHIpTDXARYBxhHwOkJ5h1zSqKkVSIGksfJHQ6WU6QjJ2FOyH2nJ Z25Phbti7WwdNvRX5aYN5oYcnIcIxNODEPr4eLAqGQxyHCx64dEv2WvC0HWZsxOdspdf wAyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252727; x=1693857527; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=03jC+OBVw6NtqXGsEGvclVCdQhu3EQx6jlJc9yvphdo=; b=h27BdhRzwZu3oHaLG3n2/K+Oy67LH4B3RuanjfXwnvNlkVNXQTnwCBHjrHScV6YnWU PwtLRviWeN0uGmIp9EGpZV5qXpuviJrGYCfrq0147PUEp0R9kx3qVrKHHUl2ccASs2th jGOLZxuc89//WLSNjcURHnZVJlgYZBRaac3LP4c82OjfQJsZ1Ot3daHTV2T+S8Wqa2p1 MLAePBkSXiqN/dWDK3kzL2urvssWH1h8JXHD+s3UIez68ZIWeNYUddPI2k5CM1RxgyoW vsVOkedVfVkfTsml3S4GdmIEjk1Ank7XpvsTBHf0krLkL7mPQrEcbtRo3+apwPCMMcTi 9cuQ== X-Gm-Message-State: AOJu0Yw9qFtIDyRdwSYdbw7Ow7NcGI16NhJOD5ufCtNZVALCZvnIQd0a ntjaLOgrnZXgG8UNftVOmxt91cBLgmFSsu+ilBE= X-Google-Smtp-Source: AGHT+IGX3a8SAyFuYllCKEJp1p8BAuBf4bX7iN13I4toDObzdsg0UQOHsOqXYzlSHWKRqNRQ4bgZw4/tHZOh9SXDs4o= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:a904:0:b0:576:8cb6:62a9 with SMTP id g4-20020a81a904000000b005768cb662a9mr947313ywh.6.1693252727342; Mon, 28 Aug 2023 12:58:47 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:39 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3226; i=samitolvanen@google.com; h=from:subject; bh=/mGBqVAJStUatVnFnltMn+I7N64iI2QEtl2iB/CZBSg=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqQdVKNNuAiMW7udPNqMcNpKCbHaCzUHSyh KrgH39THuGJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7guJC/9ByIZj6HC9xt8M48iEuc3eleOXi7LLtLxGuUV5BC0aCnuet8/CP1zkrGWtv2JZ39aif62 X2Lq1gXUXz4nIi7rwmDEz0XjtrBeZtsI5xghAkZCCubYNPWatl0y+0UTL0qa+VuYO+MUp7CwJ4q 19Any1fNwFxVAXnOTh+0UAcjst/pmBxL9W1CG4VnyA9NHMX3EXKgrQw1SpC0g1+mUut9n6bcrJr W0ppsOU/Jqpc/8Xg4TgExx30gJgt1XpXcZSyNaDZk4jXED/GJ0cOupaFz/3bhVzR1J6/y9z3y70 b9ZHkq5uFdJ6LPJDcdccev+aWmRxrkaw1vZ/4ksObnxApxpaGmFk+kc09Q4+Ab83M50knpeUZkB 2lPPYgLx0SeP3+XQdiJywRmtKnipviQqot/8K0XYRWNukxdtd30QBKhYtlFI7f5uDYw2bpedKeO 5hPzmPQaQUqbOlLYrH/VXQIVhJ5BZDRq9B+djftJejiGDFObOFuxsPk6QpMbtOq6LFIVA= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-13-samitolvanen@google.com> Subject: [PATCH v3 5/6] riscv: Use separate IRQ shadow call stacks From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 94726ea773e3..0e45db78b24b 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm =20 +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -34,6 +39,8 @@ =20 .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 52793193a763..3a0db310325a 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -237,12 +237,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, STACKFRAME_FP(sp) addi s0, sp, STACKFRAME_SIZE_ON_STACK =20 + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 =20 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -STACKFRAME_SIZE_ON_STACK REG_L ra, STACKFRAME_RA(sp) diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 95dafdcbd135..7bfea97ee7e7 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); #ifdef CONFIG_IRQ_STACKS #include =20 +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs(void) +{ + int cpu; + + if (!scs_is_enabled()) + return; + + for_each_possible_cpu(cpu) + per_cpu(irq_shadow_call_stack_ptr, cpu) =3D + scs_alloc(cpu_to_node(cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr); =20 #ifdef CONFIG_VMAP_STACK @@ -76,6 +95,7 @@ void do_softirq_own_stack(void) #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ =20 #else +static void init_irq_scs(void) {} static void init_irq_stacks(void) {} #endif /* CONFIG_IRQ_STACKS */ =20 @@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) =20 void __init init_IRQ(void) { + init_irq_scs(); init_irq_stacks(); irqchip_init(); if (!handle_arch_irq) --=20 2.42.0.rc2.253.gd59a3bf2b4-goog From nobody Fri Dec 19 12:48:28 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ED6EC83F1A for ; Mon, 28 Aug 2023 19:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233612AbjH1T7a (ORCPT ); Mon, 28 Aug 2023 15:59:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233544AbjH1T6w (ORCPT ); Mon, 28 Aug 2023 15:58:52 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEA8F188 for ; Mon, 28 Aug 2023 12:58:49 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-56fe210b7edso2222289a12.0 for ; Mon, 28 Aug 2023 12:58:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252729; x=1693857529; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BqxXsJhK/PDnu5QKzOuhVQVuGBQ7YV59awRNcw8M9KA=; b=5vbcngL8ZVHRFE2eljBR/WQoqnwHuLVs7upb4C37eJWQbZVjr56f/ur647yJ4AUyqU odlSa8/RjIkDV07G0UBaBb4auy8DU4amPuju8M5IQqLJYH54J4qila2NhRd+E1O4RyAM uVYOKvozn6NT8UEW8VWKOZJe/DKWNoXAIn9cnQYVR4AleyEg1/b63QvqUxI0dcSLbnMx XLuUylSfyiMonef/eNWAAMq+tnS4EWsKMBu8wIhTW3R0YLJ6tJf56CDFzDgaITSCuq+s BLuYHuv7RfEgVzf9k8LBhuxRvd2HXWmtf2Vk/OTijRK93Crc/qqwJGPUvgiTw7PzvnTZ ll/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252729; x=1693857529; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BqxXsJhK/PDnu5QKzOuhVQVuGBQ7YV59awRNcw8M9KA=; b=TF4mSj2Y4xjw+L8/RPckBSebxa9rWcYXh7RF+PNvfLCWAdD3ZpQ03CIM2KGPVVEl5v xaEjYX31LADPesxNxRSpXF1DnXlYCJaWTlsPfNB3qhzKHBbvYYSif8SxlHCeueliCXQd 38AvA/zQYH+38xp3GRcSSFa/Ul2KdoFhTQdxvaTWlV/LUeqPcNWWxmBj156MwTaNsBKz XFru4oHH7qHzcxNGoXNIWxBrpi2j7OaDZYtboB4dVhJiiaqUMNiKRo2VhAKvNsHHKIxO wB2ROiWeIFFVa0wgRT5fHJ28gUU/Ys/Ss4zdnSPkA9ik1ul0DGsMG1QR7SZ0hg8TmVky pXnw== X-Gm-Message-State: AOJu0YwZFW40w7Bd1lh5I6Cq95veQwD4Yvgt4G+gHs0UZXd6qTAoJiqX J8NmAUzqBTy2qDNKLJbKGP7Lu6DhegrJVOjrRJ8= X-Google-Smtp-Source: AGHT+IFGsaaO0XSM/Y67KiglbiGwn9oCum6xrnQEFf9Pggoc6RCuQ0rJ2Xb5U12wKvaEKKMsCRfVVSyEDOKWowHVSW0= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a63:9351:0:b0:569:52d4:fb68 with SMTP id w17-20020a639351000000b0056952d4fb68mr2368998pgm.5.1693252729217; Mon, 28 Aug 2023 12:58:49 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:40 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=2086; i=samitolvanen@google.com; h=from:subject; bh=P2l56HkzVV4iJDbhsS59ZlfJgm96l/buqsb6+trKTfA=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqKl9k+yTdNALVx1ZMSiIiYkPp3ifD0Ueug LDeD+59dp2JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7ombC/44I2N+oNE0mhFWTI0lDlgL+4/JMQhfMKSufRM3U1T5OqabvtPwFHQhOXRz1TiwEcDr7i8 5yQiZuqOXgg3iZ4h+9F4cRzw28ArehJIEGz+O7FO70FgOBNXOSHaWGcyv7VuHdprWhST+C1v9Zk ZgslARn9fBw5HJwmwr6bN8M9im4r9jKtVsPr55L3xnUWbk+knu12/aZpE/GGE0bbf3QK+N4vt3A 0nNh0iiFlV4YayOAPFabrTDShrkvvWlE4rzF+fG11hd/h8APu37zoH57B2AywUWNj14p83mXT6k u9y2x57j40yu0iK/b1JflZvDK7CJ0s8ajtQhMOXhjDmApgyeWZVLMLAZTpKv4gs/oO2xodXiw4R ksl1GmDHjdIlhyeMTWWtCCcg/DIiAd6f1B740jP4kkXUDiP9Sy3P/G0MHt1K+RGieGHpaB5B13/ UvPppmx2PeR8h40O/fHIATSyyElr9SQ8Dg84t2Fd6nUgiqKsWp+OrKSdL0qF6hmaWxQ0o= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-14-samitolvanen@google.com> Subject: [PATCH v3 6/6] lkdtm: Fix CFI_BACKWARD on RISC-V From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On RISC-V, the return address is before the current frame pointer, unlike on most other architectures. Use the correct offset on RISC-V to fix the CFI_BACKWARD test. Signed-off-by: Sami Tolvanen Acked-by: Kees Cook Tested-by: Nathan Chancellor --- drivers/misc/lkdtm/cfi.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/misc/lkdtm/cfi.c b/drivers/misc/lkdtm/cfi.c index fc28714ae3a6..6a33889d0902 100644 --- a/drivers/misc/lkdtm/cfi.c +++ b/drivers/misc/lkdtm/cfi.c @@ -68,12 +68,20 @@ static void lkdtm_CFI_FORWARD_PROTO(void) #define no_pac_addr(addr) \ ((__force __typeof__(addr))((uintptr_t)(addr) | PAGE_OFFSET)) =20 +#ifdef CONFIG_RISCV +/* https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-= cc.adoc#frame-pointer-convention */ +#define FRAME_RA_OFFSET (-1) +#else +#define FRAME_RA_OFFSET 1 +#endif + /* The ultimate ROP gadget. */ static noinline __no_ret_protection void set_return_addr_unchecked(unsigned long *expected, unsigned long *add= r) { /* Use of volatile is to make sure final write isn't seen as a dead store= . */ - unsigned long * volatile *ret_addr =3D (unsigned long **)__builtin_frame_= address(0) + 1; + unsigned long * volatile *ret_addr =3D + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; =20 /* Make sure we've found the right place on the stack before writing it. = */ if (no_pac_addr(*ret_addr) =3D=3D expected) @@ -88,7 +96,8 @@ static noinline void set_return_addr(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store= . */ - unsigned long * volatile *ret_addr =3D (unsigned long **)__builtin_frame_= address(0) + 1; + unsigned long * volatile *ret_addr =3D + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; =20 /* Make sure we've found the right place on the stack before writing it. = */ if (no_pac_addr(*ret_addr) =3D=3D expected) --=20 2.42.0.rc2.253.gd59a3bf2b4-goog