From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1CAFC38145 for ; Fri, 2 Sep 2022 14:28:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235903AbiIBO2y (ORCPT ); Fri, 2 Sep 2022 10:28:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235494AbiIBO1Z (ORCPT ); Fri, 2 Sep 2022 10:27:25 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 193D8145C70 for ; Fri, 2 Sep 2022 06:54:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=7BSAeKLrm7gxwtIhgZeIusnS5RkuLU2DoTkgdYtznEo=; b=sN5bJJTrDllOzx0dNSRWQNV/dJ Qo2dgkQd586esW/wXN/Bf7mHrv5xzgo5Z+8JtqK8/CZ9pnt0bYL1nZdWA9K44XLVu24M6o26QGOGZ Wl/lApotklkM033Q1HkvprM9bJ7aKeovRuPXpgLQsoX71zGkF4jVQEo2r9x9AVank2G41uQe0O//1 vUlKcewrjWkviEgqaEF+NSoAq2l5G2yqNjMahffWrKVKocgSidFeGJf1ZbFE7gZ3jnqhbhgLqsxAp eEAdQReNsqgIYD9kIDzQ9y26yMyTZsuXJHjNCE7pWz/WMcUzd22OZXpfVB+o7BSKbV4aiGIu1jvll eCovXtxA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-0074Su-I0; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 5231C3002A3; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1732725E26380; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130946.457567054@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:26 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 01/59] x86/paravirt: Ensure proper alignment References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The entries in the .parainstr sections are 8 byte aligned and the corresponding C struct makes the array offset 16 bytes. Though the pushed entries are only using 12 bytes. .parainstr_end is therefore 4 bytes short. That works by chance because it's only used in a loop: for (p =3D start; p < end; p++) But this falls flat when calculating the number of elements: n =3D end - start That's obviously off by one. Ensure that the gap is filled and the last entry is occupying 16 bytes. Cc: Juergen Gross Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Tested-by: Mel Gorman --- arch/x86/include/asm/paravirt.h | 1 + arch/x86/include/asm/paravirt_types.h | 1 + 2 files changed, 2 insertions(+) --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -743,6 +743,7 @@ extern void default_banner(void); word 771b; \ .byte ptype; \ .byte 772b-771b; \ + _ASM_ALIGN; \ .popsection =20 =20 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -294,6 +294,7 @@ extern struct paravirt_patch_template pv " .byte " type "\n" \ " .byte 772b-771b\n" \ " .short " clobber "\n" \ + _ASM_ALIGN "\n" \ ".popsection\n" =20 /* Generate patchable code, with the default asm parameters. */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C84FECAAD5 for ; Fri, 2 Sep 2022 14:28:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238165AbiIBO2s (ORCPT ); Fri, 2 Sep 2022 10:28:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236937AbiIBO1P (ORCPT ); Fri, 2 Sep 2022 10:27:15 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D94910F0A2 for ; Fri, 2 Sep 2022 06:54:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ziGAKr7QhrrTtLR+TFwgCK+wh0GbBV7swcz66BqGyr4=; b=revA3Ff1Gh4CEThzDwESmXJwPG BXoLoEXD5pMPZeSDZ/eZKGRAoppNWNSXTxd7oXTeTCs6R3T8h75SuDkJK4y/3OJM5uEDlFcSRAWWH Hi6q3wSIR85GiLHVCLPdceF74va4waOWxELRuj9EkBOmM/50v/qV4118Q6Taio4bVZ5F0bAqc67IW goJmN8H8UitHrvigpZZ26dFDX1O8/WhT+IuGPJW4w8NSyyUFNbAlS4PngcuTBsuNwS4KmznE42Jc2 /br5mZJJSNN0AetiMAcnWAwmbkUp8ZmCZOcB3FSsl2PnoXKXZvzwOUqGLJuk58BDWVw8sy+xEuVKO a4RNSN0Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-0074Ss-HM; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 56501300431; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1AD3720257205; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130946.566050315@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:27 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 02/59] x86/cpu: Remove segment load from switch_to_new_gdt() References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner On 32bit FS and on 64bit GS segments are already set up correctly, but load_percpu_segment() still sets [FG]S after switching from the early GDT to the direct GDT. For 32bit the segment load has no side effects, but on 64bit it causes GSBASE to become 0, which means that any per CPU access before GSBASE is set to the new value is going to fault. That's the reason why the whole file containing this code has stackprotector removed. But that's a pointless exercise for both 32 and 64 bit as the relevant segment selector is already correct. Loading the new GDT does not change that. Remove the segment loads and add comments. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/processor.h | 1=20 arch/x86/kernel/cpu/common.c | 47 +++++++++++++++++++++++++---------= ----- 2 files changed, 31 insertions(+), 17 deletions(-) --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -673,7 +673,6 @@ extern struct desc_ptr early_gdt_descr; extern void switch_to_new_gdt(int); extern void load_direct_gdt(int); extern void load_fixmap_gdt(int); -extern void load_percpu_segment(int); extern void cpu_init(void); extern void cpu_init_secondary(void); extern void cpu_init_exception_handling(void); --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -701,16 +701,6 @@ static const char *table_lookup_model(st __u32 cpu_caps_cleared[NCAPINTS + NBUGINTS] __aligned(sizeof(unsigned long= )); __u32 cpu_caps_set[NCAPINTS + NBUGINTS] __aligned(sizeof(unsigned long)); =20 -void load_percpu_segment(int cpu) -{ -#ifdef CONFIG_X86_32 - loadsegment(fs, __KERNEL_PERCPU); -#else - __loadsegment_simple(gs, 0); - wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu)); -#endif -} - #ifdef CONFIG_X86_32 /* The 32-bit entry code needs to find cpu_entry_area. */ DEFINE_PER_CPU(struct cpu_entry_area *, cpu_entry_area); @@ -738,16 +728,41 @@ void load_fixmap_gdt(int cpu) } EXPORT_SYMBOL_GPL(load_fixmap_gdt); =20 -/* - * Current gdt points %fs at the "master" per-cpu area: after this, - * it's on the real one. +/** + * switch_to_new_gdt - Switch form early GDT to the direct one + * @cpu: The CPU number for which this is invoked + * + * Invoked during early boot to switch from early GDT and early per CPU + * (%fs on 32bit, GS_BASE on 64bit) to the direct GDT and the runtime per + * CPU area. */ void switch_to_new_gdt(int cpu) { - /* Load the original GDT */ load_direct_gdt(cpu); - /* Reload the per-cpu base */ - load_percpu_segment(cpu); + +#ifdef CONFIG_X86_64 + /* + * No need to load %gs. It is already correct. + * + * Writing %gs on 64bit would zero GSBASE which would make any per + * CPU operation up to the point of the wrmsrl() fault. + * + * Set GSBASE to the new offset. Until the wrmsrl() happens the + * early mapping is still valid. That means the GSBASE update will + * lose any prior per CPU data which was not copied over in + * setup_per_cpu_areas(). + */ + wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu)); +#else + /* + * %fs is already set to __KERNEL_PERCPU, but after switching GDT + * it is required to load FS again so that the 'hidden' part is + * updated from the new GDT. Up to this point the early per CPU + * translation is active. Any content of the early per CPU data + * which was not copied over in setup_per_cpu_areas() is lost. + */ + loadsegment(fs, __KERNEL_PERCPU); +#endif } =20 static const struct cpu_dev *cpu_devs[X86_VENDOR_NUM] =3D {}; From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB8EFC38145 for ; Fri, 2 Sep 2022 14:29:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236388AbiIBO3v (ORCPT ); Fri, 2 Sep 2022 10:29:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236959AbiIBO1d (ORCPT ); Fri, 2 Sep 2022 10:27:33 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47F911581AC for ; Fri, 2 Sep 2022 06:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=O5PdF8HIrWFo7pNjLhP8ffWoE9Ddd7vvjM+ehcZizzE=; b=FjhkDCOGmXUNe2wXedgrU3T1nQ ZPcYHDUEqSmTbFe+xPafLNNZ2aI9l1MAZR8PNDc8AZu8p3JNoWQ7keihSIIBUJW3iMfDF10IEo9Uo lrA9TxFKAAe41CocmSYE9yc8aul9bt5PkZaQDAp5fPdg7a0n76YMjgIieNSuUDz8Brxj57E2K9x0J e93Hrz01rYmycbMRU+S0+YDFABA9ZVvX3XIKBZ9riAYOUJb7NqfTDj7Up2KdEc9kNYo37DwJRurJ4 7koISDoX5uIwbQDl6e6T8R6INU+ZsiKcttYQmSXiNYzvXm/CMG3rvWmlkPdiw76hdULtbyNvsokwB 0afw55RA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-0074St-Hp; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 546513002C7; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1E02229A333FB; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130946.669965052@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:28 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 03/59] x86/cpu: Get rid of redundant switch_to_new_gdt() invocations References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The only place where switch_to_new_gdt() is required is early boot to switch from the early GDT to the direct GDT. Any other invocation is completely redundant because it does not change anything. Secondary CPUs come out of the ASM code with GDT and GSBASE correctly set up. The same is true for XEN_PV. Remove all the voodoo invocations which are left overs from the ancient past, rename the function to switch_gdt_and_percpu_base() and mark it init. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- V2: Rename it (Linus) --- arch/x86/include/asm/processor.h | 2 +- arch/x86/kernel/cpu/common.c | 17 ++++++----------- arch/x86/kernel/setup_percpu.c | 2 +- arch/x86/kernel/smpboot.c | 6 +++++- arch/x86/xen/enlighten_pv.c | 2 +- 5 files changed, 14 insertions(+), 15 deletions(-) --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -670,7 +670,7 @@ extern int sysenter_setup(void); /* Defined in head.S */ extern struct desc_ptr early_gdt_descr; =20 -extern void switch_to_new_gdt(int); +extern void switch_gdt_and_percpu_base(int); extern void load_direct_gdt(int); extern void load_fixmap_gdt(int); extern void cpu_init(void); --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -729,14 +729,15 @@ void load_fixmap_gdt(int cpu) EXPORT_SYMBOL_GPL(load_fixmap_gdt); =20 /** - * switch_to_new_gdt - Switch form early GDT to the direct one + * switch_gdt_and_percpu_base - Switch to direct GDT and runtime per CPU b= ase * @cpu: The CPU number for which this is invoked * - * Invoked during early boot to switch from early GDT and early per CPU - * (%fs on 32bit, GS_BASE on 64bit) to the direct GDT and the runtime per - * CPU area. + * Invoked during early boot to switch from early GDT and early per CPU to + * the direct GDT and the runtime per CPU area. On 32-bit the percpu base + * switch is implicit by loading the direct GDT. On 64bit this requires + * to update GSBASE. */ -void switch_to_new_gdt(int cpu) +void __init switch_gdt_and_percpu_base(int cpu) { load_direct_gdt(cpu); =20 @@ -2251,12 +2252,6 @@ void cpu_init(void) boot_cpu_has(X86_FEATURE_TSC) || boot_cpu_has(X86_FEATURE_DE)) cr4_clear_bits(X86_CR4_VME|X86_CR4_PVI|X86_CR4_TSD|X86_CR4_DE); =20 - /* - * Initialize the per-CPU GDT with the boot GDT, - * and set up the GDT descriptor: - */ - switch_to_new_gdt(cpu); - if (IS_ENABLED(CONFIG_X86_64)) { loadsegment(fs, 0); memset(cur->thread.tls_array, 0, GDT_ENTRY_TLS_ENTRIES * 8); --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -211,7 +211,7 @@ void __init setup_per_cpu_areas(void) * area. Reload any changed state for the boot CPU. */ if (!cpu) - switch_to_new_gdt(cpu); + switch_gdt_and_percpu_base(cpu); } =20 /* indicate the early static arrays will soon be gone */ --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1453,7 +1453,11 @@ void arch_thaw_secondary_cpus_end(void) void __init native_smp_prepare_boot_cpu(void) { int me =3D smp_processor_id(); - switch_to_new_gdt(me); + + /* SMP handles this from setup_per_cpu_areas() */ + if (!IS_ENABLED(CONFIG_SMP)) + switch_gdt_and_percpu_base(me); + /* already set me in cpu_online_mask in boot_cpu_init() */ cpumask_set_cpu(me, cpu_callout_mask); cpu_set_state_online(me); --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -1167,7 +1167,7 @@ static void __init xen_setup_gdt(int cpu pv_ops.cpu.write_gdt_entry =3D xen_write_gdt_entry_boot; pv_ops.cpu.load_gdt =3D xen_load_gdt_boot; =20 - switch_to_new_gdt(cpu); + switch_gdt_and_percpu_base(cpu); =20 pv_ops.cpu.write_gdt_entry =3D xen_write_gdt_entry; pv_ops.cpu.load_gdt =3D xen_load_gdt; From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E40A1C38145 for ; Fri, 2 Sep 2022 14:28:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237602AbiIBO2k (ORCPT ); Fri, 2 Sep 2022 10:28:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237104AbiIBO1N (ORCPT ); Fri, 2 Sep 2022 10:27:13 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 181B7168A08 for ; Fri, 2 Sep 2022 06:54:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=exdW9cTzIEZMDSgoNhyqMCUVrHrKRGK4pmqnX15phYU=; b=ft6Iz6l/UXQcWYc4if2tCQx73L dD50HHIY3H43UeRnB/6DA2rgloUopDrUxumWOtTZLslSYQSgzcaqAKUVzzT+fMk/uRaGgKKHPvtWl RdFHe9Kp0rulxW4jUw1SZlbmtVnOR8w6yMh0FWKaZYGLgqXlnCq3gMKnyr1QJlSgaO+zurBU4EqKs C93sepP37xJAjkT5bgmegHvEb/TmUXWmWM9pL7Ccd5i0JAQgRk/E6VwtHqPBQlz5bGR8E2IC9Mx3Q P4JkTkEmBh+GygLCp89UGoVVo45Rulb8jjCC3XvLIS54KkYfwVYPeoopczgwQTpCoe4NJ2vAcrcl1 LAdGU1Uw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77K-008g7n-ND; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 59EFB3005DB; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2563D25D60EC2; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130946.775455579@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:29 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 04/59] x86/cpu: Re-enable stackprotector References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Commit 5416c2663517 ("x86: make sure load_percpu_segment has no stackprotector") disabled the stackprotector for cpu/common.c because of load_percpu_segment(). Back then the boot stack canary was initialized very early in start_kernel(). Switching the per CPU area by loading the GDT caused the stackprotector to fail with paravirt enabled kernels as the GSBASE was not updated yet. In hindsight a wrong change because it would have been sufficient to ensure that the canary is the same in both per CPU areas. Commit d55535232c3d ("random: move rand_initialize() earlier") moved the stack canary initialization to a later point in the init sequence. As a consequence the per CPU stack canary is 0 when switching the per CPU areas, so there is no requirement anymore to exclude this file. Add a comment to load_percpu_segment(). Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/cpu/Makefile | 3 --- arch/x86/kernel/cpu/common.c | 3 +++ 2 files changed, 3 insertions(+), 3 deletions(-) --- a/arch/x86/kernel/cpu/Makefile +++ b/arch/x86/kernel/cpu/Makefile @@ -16,9 +16,6 @@ KCOV_INSTRUMENT_perf_event.o :=3D n # As above, instrumenting secondary CPU boot code causes boot hangs. KCSAN_SANITIZE_common.o :=3D n =20 -# Make sure load_percpu_segment has no stackprotector -CFLAGS_common.o :=3D -fno-stack-protector - obj-y :=3D cacheinfo.o scattered.o topology.o obj-y +=3D common.o obj-y +=3D rdrand.o --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -752,6 +752,9 @@ void __init switch_gdt_and_percpu_base(i * early mapping is still valid. That means the GSBASE update will * lose any prior per CPU data which was not copied over in * setup_per_cpu_areas(). + * + * This works even with stackprotector enabled because the + * per CPU stack canary is 0 in both per CPU areas. */ wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu)); #else From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14E1CECAAD5 for ; Fri, 2 Sep 2022 14:28:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236373AbiIBO21 (ORCPT ); Fri, 2 Sep 2022 10:28:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236207AbiIBO1K (ORCPT ); Fri, 2 Sep 2022 10:27:10 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DCF6145C66 for ; Fri, 2 Sep 2022 06:54:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=esbaxv8bwXI/iX43XRRYBSC6TFpUTFx/YoIUh5Tifmk=; b=JvXyLbdcPIj/Me9w0zPrp4fQn9 M+yyGbe7gcpYehSxRrACWc2Kv56Uhf9DhFQcdzPHgpuC5UAuAW4DqoZOTpVoF4th60OrENjxCFS8Y 9+UevSI/Y3yIe3shPNM187AbNmvsTVXBdCc1LelQSU8LAu4I1khi6/6+vBL40CFtvfFKola272Tw1 74xkESyxYUjrtSvoawSPGIZ9Q9B/cL68/c5aW0PDqr9n+Q7C42Uhe9W48m1xzz6f/k/rJNBpb5yFi rejPz+1wXwXw9iDWkjvxj2KPpmvasMVrX/lCciXRLJ3fP83QHOfkUzPvQphRzYRkht5DO7jqp3ZV1 tebyILuw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-0074T6-QL; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 00EDE300813; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2E8F4299EA069; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130946.878962398@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:30 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 05/59] x86/modules: Set VM_FLUSH_RESET_PERMS in module_alloc() References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Instead of resetting permissions all over the place when freeing module memory tell the vmalloc code to do so. Avoids the exercise for the next upcoming user. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/ftrace.c | 2 -- arch/x86/kernel/kprobes/core.c | 1 - arch/x86/kernel/module.c | 9 +++++---- 3 files changed, 5 insertions(+), 7 deletions(-) --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -413,8 +413,6 @@ create_trampoline(struct ftrace_ops *ops /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |=3D FTRACE_OPS_FL_ALLOC_TRAMP; =20 - set_vm_flush_reset_perms(trampoline); - if (likely(system_state !=3D SYSTEM_BOOTING)) set_memory_ro((unsigned long)trampoline, npages); set_memory_x((unsigned long)trampoline, npages); --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -416,7 +416,6 @@ void *alloc_insn_page(void) if (!page) return NULL; =20 - set_vm_flush_reset_perms(page); /* * First make the page read-only, and only then make it executable to * prevent it from being W+X in between. --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -74,10 +74,11 @@ void *module_alloc(unsigned long size) return NULL; =20 p =3D __vmalloc_node_range(size, MODULE_ALIGN, - MODULES_VADDR + get_module_load_offset(), - MODULES_END, gfp_mask, - PAGE_KERNEL, VM_DEFER_KMEMLEAK, NUMA_NO_NODE, - __builtin_return_address(0)); + MODULES_VADDR + get_module_load_offset(), + MODULES_END, gfp_mask, PAGE_KERNEL, + VM_FLUSH_RESET_PERMS | VM_DEFER_KMEMLEAK, + NUMA_NO_NODE, __builtin_return_address(0)); + if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { vfree(p); return NULL; From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC9E4C54EE9 for ; Fri, 2 Sep 2022 14:31:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236095AbiIBObK (ORCPT ); Fri, 2 Sep 2022 10:31:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51238 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236221AbiIBO1j (ORCPT ); Fri, 2 Sep 2022 10:27:39 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0643D15819F for ; Fri, 2 Sep 2022 06:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=6xIMyXngTvaZTcz5zEWS3IkhPHWVQF/6vMw/4Le/NHI=; b=MMh1eRW99NY5VqplEY/cHAsRye VSh7wYa7CFCKlnpCEhcCOcRnEZVv9/YvpP/3n7VwhNI6MxcGHLsPcVRd1Hp7ujmFYY43yI4dtvqWV mitJNvcTjqkWl8SoZYb79DcHojY5qEeylRaUsTYMWrjK1qpLg2yDHeFnqM9kgrYioGERltC5d31Ry NScBaYkGJL+K6TG6f5DHezo57G45itZTSc8Fwr0BXX5Xxx91eWKuYUYw4gHtgB8yJHxaH5yhtAUy2 17UX1J+lSD3d+d9Ilgqj8ImiWnQe4MU51eb54t2QatlYQaCCSijc1SeyjWrBDWnCovBpcL0JlDVEK J2B9DCYg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77K-008g7q-V0; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 00E953006C4; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 36130236E3766; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130946.981640225@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:31 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 06/59] x86/vdso: Ensure all kernel code is seen by objtool References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner extable.c is kernel code and not part of the VDSO Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/entry/vdso/Makefile | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -30,11 +30,12 @@ vobjs32-y +=3D vdso32/vclock_gettime.o vobjs-$(CONFIG_X86_SGX) +=3D vsgx.o =20 # files to link into kernel -obj-y +=3D vma.o extable.o -KASAN_SANITIZE_vma.o :=3D y -UBSAN_SANITIZE_vma.o :=3D y -KCSAN_SANITIZE_vma.o :=3D y -OBJECT_FILES_NON_STANDARD_vma.o :=3D n +obj-y +=3D vma.o extable.o +KASAN_SANITIZE_vma.o :=3D y +UBSAN_SANITIZE_vma.o :=3D y +KCSAN_SANITIZE_vma.o :=3D y +OBJECT_FILES_NON_STANDARD_vma.o :=3D n +OBJECT_FILES_NON_STANDARD_extable.o :=3D n =20 # vDSO images to build vdso_img-$(VDSO64-y) +=3D 64 From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E8D5C38145 for ; Fri, 2 Sep 2022 14:28:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236162AbiIBO2V (ORCPT ); Fri, 2 Sep 2022 10:28:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235976AbiIBO1K (ORCPT ); Fri, 2 Sep 2022 10:27:10 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0842F168A25 for ; Fri, 2 Sep 2022 06:54:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=yIoG35cVddrSUYas1ovPM/6azqMQrScMR1ghVWokYpY=; b=KV6fDSwW14+spIe+PGUs5xODJs 0xBc83Rr/tlrxyR6iTx8pQ62NfCAv36/KS1GB8tl8bTbmY/g8EA8dMDgLFd17m+xqPnpazbEboag6 6w5fCRUuSb4NdX5fdHbR2cHzOZojBPCsslx9mtJ25ad8d885zKxz28tpJoLp1ysRrbBy7vG5Uz01K XrjtygRHhLldm7mLBU8OIpDCOW4t6ra0VBZR1nSXcOwS3tGlKe1Vck2uZ9YNmpGnvvb512WQ5mNPX JOBiEh3epN5q14LbE8jWArCmDACWlR1vR5He9fGYIzBa+d0m8vRe+zW1yyKQqF2n3aVU+0YHtAyF8 T/eQ1dEA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-0074T2-6c; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 00E7430067B; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3AE2529C4662E; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.086463061@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:32 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 07/59] x86: Sanitize linker script References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The section ordering in the text section is more than suboptimal: ALIGN_ENTRY_TEXT_BEGIN ENTRY_TEXT ALIGN_ENTRY_TEXT_END SOFTIRQENTRY_TEXT STATIC_CALL_TEXT INDIRECT_THUNK_TEXT ENTRY_TEXT is in a seperate PMD so it can be mapped into the cpu entry area when KPTI is enabled. That means the sections after it are also in a seperate PMD. That's wasteful especially as the indirect thunk text is a hotpath on retpoline enabled systems and the static call text is fairly hot on 32bit. Move the entry text section last so that the other sections share a PMD with the text before it. This is obviously just best effort and not guaranteed when the previous text is just at a PMD boundary. The text section placement needs an overhaul in general. There is e.g. no point to have debugfs, sysfs, cpuhotplug and other rarely used functions next to hot path text. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/vmlinux.lds.S | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -132,18 +132,19 @@ SECTIONS CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT - ALIGN_ENTRY_TEXT_BEGIN - ENTRY_TEXT - ALIGN_ENTRY_TEXT_END SOFTIRQENTRY_TEXT - STATIC_CALL_TEXT - *(.gnu.warning) - #ifdef CONFIG_RETPOLINE __indirect_thunk_start =3D .; *(.text.__x86.*) __indirect_thunk_end =3D .; #endif + STATIC_CALL_TEXT + + ALIGN_ENTRY_TEXT_BEGIN + ENTRY_TEXT + ALIGN_ENTRY_TEXT_END + *(.gnu.warning) + } :text =3D0xcccc =20 /* End of text section, which should occupy whole number of pages */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32A0DECAAD5 for ; Fri, 2 Sep 2022 14:30:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236158AbiIBOaY (ORCPT ); Fri, 2 Sep 2022 10:30:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236098AbiIBO1g (ORCPT ); Fri, 2 Sep 2022 10:27:36 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B975168A26 for ; Fri, 2 Sep 2022 06:54:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=YUhA2B9hWTAdbvtvLfAUCL48qsFF8kF9euAYgBwVN7c=; b=F1sdS6J0B07JaNau9gs7zg9nAa JFqYigGC2sVpUTdJG0Ry7PoQlkHW5sSmWOwYlDjzxi0d0ZM6ucwdxaOGxu5ED0CyjXB1eFdseUDqN jwYnJJAj/nI/+x1Nn/1VIPrh3UMEkX7GGDXBG01mgtwTMBVmqY6RCMX/1y2nfHXvPnIECNItRa0ai pYFOHwxuBnSyFVrowORsKnxV/Pfj5qX8j1TjDhEXp7dTB7qw+DNU+KCzRAEgHisdvG+FcsW8y+KXb yEr7fHD9l8gWsNkHiG0DwPEbdj52ujSMHBOrxSeGgjq57exmwfsJS4w1PQL9B1AH467E0chEIDvyn ZEur5Odg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-0074T1-6d; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 00E8430069C; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3F8F32B077D23; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.190618587@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:33 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 08/59] x86/build: Ensure proper function alignment References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The Intel Architectures Optimization Reference Manual explains that functions should be aligned at 16 bytes because for a lot of (Intel) uarchs the I-fetch width is 16 bytes. The AMD Software Optimization Guide (for recent chips) mentions a 32 byte I-fetch window but a 16 byte decode window. Follow this advice and align functions to 16 bytes to optimize instruction delivery to decode and reduce front-end bottlenecks. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/Kconfig.cpu | 6 ++++++ arch/x86/Makefile | 4 ++++ arch/x86/include/asm/linkage.h | 7 ++++--- include/asm-generic/vmlinux.lds.h | 7 ++++++- 4 files changed, 20 insertions(+), 4 deletions(-) --- a/arch/x86/Kconfig.cpu +++ b/arch/x86/Kconfig.cpu @@ -517,3 +517,9 @@ config CPU_SUP_VORTEX_32 makes the kernel a tiny bit smaller. =20 If unsure, say N. + +# Defined here so it is defined for UM too +config FUNCTION_ALIGNMENT + int + default 16 if X86_64 || X86_ALIGNMENT_16 + default 8 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -84,6 +84,10 @@ else KBUILD_CFLAGS +=3D $(call cc-option,-fcf-protection=3Dnone) endif =20 +ifneq ($(CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B),y) +KBUILD_CFLAGS +=3D -falign-functions=3D$(CONFIG_FUNCTION_ALIGNMENT) +endif + ifeq ($(CONFIG_X86_32),y) BITS :=3D 32 UTS_MACHINE :=3D i386 --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -14,9 +14,10 @@ =20 #ifdef __ASSEMBLY__ =20 -#if defined(CONFIG_X86_64) || defined(CONFIG_X86_ALIGNMENT_16) -#define __ALIGN .p2align 4, 0x90 -#define __ALIGN_STR __stringify(__ALIGN) +#if CONFIG_FUNCTION_ALIGNMENT =3D=3D 16 +#define __ALIGN .p2align 4, 0x90 +#define __ALIGN_STR __stringify(__ALIGN) +#define FUNCTION_ALIGNMENT 16 #endif =20 #if defined(CONFIG_RETHUNK) && !defined(__DISABLE_EXPORTS) && !defined(BUI= LD_VDSO) --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -82,7 +82,12 @@ #endif =20 /* Align . to a 8 byte boundary equals to maximum function alignment. */ -#define ALIGN_FUNCTION() . =3D ALIGN(8) +#ifndef CONFIG_FUNCTION_ALIGNMENT +#define __FUNCTION_ALIGNMENT 8 +#else +#define __FUNCTION_ALIGNMENT CONFIG_FUNCTION_ALIGNMENT +#endif +#define ALIGN_FUNCTION() . =3D ALIGN(__FUNCTION_ALIGNMENT) =20 /* * LD_DEAD_CODE_DATA_ELIMINATION option enables -fdata-sections, which From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69B63C38145 for ; Fri, 2 Sep 2022 14:33:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236209AbiIBOc4 (ORCPT ); Fri, 2 Sep 2022 10:32:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238084AbiIBO2s (ORCPT ); Fri, 2 Sep 2022 10:28:48 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB85321E0E for ; Fri, 2 Sep 2022 06:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=iyMt99aEuDsmq01H4vnoTwNVsujF1AlFYZo6vxM6wm8=; b=LOWPcYwarw+3cR9biYHzY8+3n9 kDt4l2BqObOIEE42ww5YWvoAHWXnVNiHZcyH8tX4otV6HVDEDUNy80sNYb3YhMAAJH5wJnxuaMmyO Ya98irsbhLBT/Q1eEk/1hH3Z50RVLK/H9qn0fvAf+cQSMpx/QPWWJeNkx+f6oNhBehouVd1LE56/4 ru+bSagREjMxpOwgT/nlrLmRd4Y1TCd4GaFeny9yPBThpMtBW7SLFlCgF9ighH1i+KRyGi4HnBary 7Wf1FWBqpCKKwT3KNF8a7S0OKLD7pLOc99kfPGcqxULEEb3wBchmpebiZufEuTpiZOi4eIjZTLvBF C0E4qUyQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77K-008g7o-Ua; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 00EC5300779; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 432242B1A6BC9; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.295146217@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:34 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 09/59] x86/asm: Ensure proper function alignment References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner With the compiler now aligning functions to 16 bytes, make sure the assmbler does the same. Change the SYM_FUNC_START*() variants to have matching alignment. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/linkage.h | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -12,13 +12,20 @@ #define asmlinkage CPP_ASMLINKAGE __attribute__((regparm(0))) #endif /* CONFIG_X86_32 */ =20 -#ifdef __ASSEMBLY__ +#if CONFIG_FUNCTION_ALIGNMENT =3D=3D 8 +#define __ALIGN .p2align 3, 0x90; +#elif CONFIG_FUNCTION_ALIGNMENT =3D=3D 16 +#define __ALIGN .p2align 4, 0x90; +#else +# error Unsupported function alignment +#endif =20 -#if CONFIG_FUNCTION_ALIGNMENT =3D=3D 16 -#define __ALIGN .p2align 4, 0x90 #define __ALIGN_STR __stringify(__ALIGN) -#define FUNCTION_ALIGNMENT 16 -#endif +#define ASM_FUNC_ALIGN __ALIGN_STR +#define __FUNC_ALIGN __ALIGN +#define SYM_F_ALIGN __FUNC_ALIGN + +#ifdef __ASSEMBLY__ =20 #if defined(CONFIG_RETHUNK) && !defined(__DISABLE_EXPORTS) && !defined(BUI= LD_VDSO) #define RET jmp __x86_return_thunk @@ -46,7 +53,7 @@ =20 /* SYM_FUNC_START -- use for global functions */ #define SYM_FUNC_START(name) \ - SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) \ + SYM_START(name, SYM_L_GLOBAL, SYM_F_ALIGN) \ ENDBR =20 /* SYM_FUNC_START_NOALIGN -- use for global functions, w/o alignment */ @@ -56,7 +63,7 @@ =20 /* SYM_FUNC_START_LOCAL -- use for local functions */ #define SYM_FUNC_START_LOCAL(name) \ - SYM_START(name, SYM_L_LOCAL, SYM_A_ALIGN) \ + SYM_START(name, SYM_L_LOCAL, SYM_F_ALIGN) \ ENDBR =20 /* SYM_FUNC_START_LOCAL_NOALIGN -- use for local functions, w/o alignment = */ @@ -66,7 +73,7 @@ =20 /* SYM_FUNC_START_WEAK -- use for weak functions */ #define SYM_FUNC_START_WEAK(name) \ - SYM_START(name, SYM_L_WEAK, SYM_A_ALIGN) \ + SYM_START(name, SYM_L_WEAK, SYM_F_ALIGN) \ ENDBR =20 /* SYM_FUNC_START_WEAK_NOALIGN -- use for weak functions, w/o alignment */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13B72C38145 for ; Fri, 2 Sep 2022 14:27:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236423AbiIBO1n (ORCPT ); Fri, 2 Sep 2022 10:27:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237078AbiIBO1I (ORCPT ); Fri, 2 Sep 2022 10:27:08 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07D23168A24 for ; Fri, 2 Sep 2022 06:54:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=9QYIh8jQSl50RNoTtByBwdectJ68tUz/IPDpIU6AK7o=; b=fh5xwzjEJoiHoArDYodXaKOPzB 3sHrNANwxw02c12yHuEGSNj0GuQBLf47DWS/kZbi+z3i2j9K1omyFocuQAj+KW1qH410BvVUsAUpW NrF/Ju77aCcU42b7I6srN5zP7HVFGhFun1Mu6Hyk1erRokiLpUEPsty5b9GCpoNkDuFXWix75Mjqf Fvrax1CbdLynF7mZp5FeHzuqmQRImmrMWYLAXTYj14d2oYLBNmUkZ8+afY+lPA/c/eACDeRjRrKJn mGyyj6gryGI5tA0qs2WSOJGQ7FETL5+Ff8qiDet2Id0UWky530Nspuxgbo2JI9SyAhE8UF1J2KR9N UdiBA+rQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g7y-JF; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 084F330088D; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 492222B1A6BD1; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.396644686@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:35 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 10/59] x86/error_inject: Align function properly References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Ensure inline asm functions are consistently aligned with compiler generated and SYM_FUNC_START*() functions. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/lib/error-inject.c | 1 + 1 file changed, 1 insertion(+) --- a/arch/x86/lib/error-inject.c +++ b/arch/x86/lib/error-inject.c @@ -11,6 +11,7 @@ asm( ".text\n" ".type just_return_func, @function\n" ".globl just_return_func\n" + ASM_FUNC_ALIGN "just_return_func:\n" ANNOTATE_NOENDBR ASM_RET From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2574DECAAD5 for ; Fri, 2 Sep 2022 14:30:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236310AbiIBOaz (ORCPT ); Fri, 2 Sep 2022 10:30:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237335AbiIBO1i (ORCPT ); Fri, 2 Sep 2022 10:27:38 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 661661581A5 for ; Fri, 2 Sep 2022 06:54:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=HFxAeBcqZzx1YzuFlftNJ033Kmjw/OvruEBmGEN0HcI=; b=Oa2G8BWpsj8uyUvV/CiV/15FjQ zBV1SPgVZMnFaVxUQAk2DiFZ9ufbK1BwWqBqPlrWUycI2Yqb9YKqzp7IFt6XwL1Y3OoXTvxu+qKyw IfNRfNkoV6aUlNObp0nWvVmhqRfHjWOSyAfP4w8GQD48MOTmIC6G4I4m1W/e9k8X8M9pyw+5qx1SC iXz8vQ8Gr7yDtHvO/tX24vPqmP3FtnY/hCmzwibMcEiK5h/HQa1GQKsFB6YLlzaZQpiIizC1yyY9S ISkdlV7cwegy1viTXnndnRbeieBDIz0+AszkSG1fGr2J1AkkBEgnJ4D9uOscrJE2Ya3lVqJpfE6k5 Zl3hCOHA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g7z-JF; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 0ABC6301482; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4D3E32B1A6BE6; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.502476538@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:36 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 11/59] x86/paravirt: Properly align PV functions References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Ensure inline asm functions are consistently aligned with compiler generated and SYM_FUNC_START*() functions. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/paravirt.h | 1 + arch/x86/include/asm/qspinlock_paravirt.h | 2 +- arch/x86/kernel/kvm.c | 1 + arch/x86/kernel/paravirt.c | 2 ++ 4 files changed, 5 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -665,6 +665,7 @@ bool __raw_callee_save___native_vcpu_is_ asm(".pushsection " section ", \"ax\";" \ ".globl " PV_THUNK_NAME(func) ";" \ ".type " PV_THUNK_NAME(func) ", @function;" \ + ASM_FUNC_ALIGN \ PV_THUNK_NAME(func) ":" \ ASM_ENDBR \ FRAME_BEGIN \ --- a/arch/x86/include/asm/qspinlock_paravirt.h +++ b/arch/x86/include/asm/qspinlock_paravirt.h @@ -39,7 +39,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_sp asm (".pushsection .text;" ".globl " PV_UNLOCK ";" ".type " PV_UNLOCK ", @function;" - ".align 4,0x90;" + ASM_FUNC_ALIGN PV_UNLOCK ": " ASM_ENDBR FRAME_BEGIN --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -802,6 +802,7 @@ asm( ".pushsection .text;" ".global __raw_callee_save___kvm_vcpu_is_preempted;" ".type __raw_callee_save___kvm_vcpu_is_preempted, @function;" +ASM_FUNC_ALIGN "__raw_callee_save___kvm_vcpu_is_preempted:" ASM_ENDBR "movq __per_cpu_offset(,%rdi,8), %rax;" --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -40,6 +40,7 @@ extern void _paravirt_nop(void); asm (".pushsection .entry.text, \"ax\"\n" ".global _paravirt_nop\n" + ASM_FUNC_ALIGN "_paravirt_nop:\n\t" ASM_ENDBR ASM_RET @@ -50,6 +51,7 @@ asm (".pushsection .entry.text, \"ax\"\n /* stub always returning 0. */ asm (".pushsection .entry.text, \"ax\"\n" ".global paravirt_ret0\n" + ASM_FUNC_ALIGN "paravirt_ret0:\n\t" ASM_ENDBR "xor %" _ASM_AX ", %" _ASM_AX ";\n\t" From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13AFAC38145 for ; Fri, 2 Sep 2022 14:27:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235644AbiIBO1r (ORCPT ); Fri, 2 Sep 2022 10:27:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236107AbiIBO1J (ORCPT ); Fri, 2 Sep 2022 10:27:09 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4E1211519F for ; Fri, 2 Sep 2022 06:54:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=dU2Gcv7fJ6+XDB4dHrX1+Ks1xNnLuKHyEB6BsKn/hys=; b=gDP+G+ubAw6CYtRqJTETmY3XRV f4DtdBEIggptmpCsrGzAtbzP0Exj5E/I5hnv+MaQJ8OXuU+jn9ijiWTyHm+6/EskhjEHcZmJ1qfKi gCt1cZfQrlusMHP9Fd9P6t4EIjdqZ4WkQ/VDRzBaLWmS4s6z7PrPdF4hTpFpoPAHDJ84j2//IPPbN wA4M3hmzBQWKstFLP8n5CqQRHcuzL+t0VbiG/H08UPUcuCUd6hw7whsKEoaRYDnvomVVuIZGwkHzb s1B8v/wYelQEg8ZRuLjjJFjs+E1rhUxE5mMDjuDVgwy0Stz8XTJUXl/V7ovbi72NJYviA9OsnS+lQ 15ozaF2Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-0074T7-RK; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 0EA083015B5; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 52E122B1E1E5B; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.606609517@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:37 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 12/59] x86/entry: Align SYM_CODE_START() variants References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Explicitly align a bunch of commonly called SYM_CODE_START() symbols. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/entry/entry_64.S | 12 ++++++++---- arch/x86/entry/thunk_64.S | 4 ++-- 2 files changed, 10 insertions(+), 6 deletions(-) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -284,7 +284,8 @@ SYM_FUNC_END(__switch_to_asm) * r12: kernel thread arg */ .pushsection .text, "ax" -SYM_CODE_START(ret_from_fork) + __FUNC_ALIGN +SYM_CODE_START_NOALIGN(ret_from_fork) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // copy_thread movq %rax, %rdi @@ -828,7 +829,8 @@ EXPORT_SYMBOL(asm_load_gs_index) * * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs) */ -SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) + __FUNC_ALIGN +SYM_CODE_START_LOCAL_NOALIGN(exc_xen_hypervisor_callback) =20 /* * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will @@ -856,7 +858,8 @@ SYM_CODE_END(exc_xen_hypervisor_callback * We distinguish between categories by comparing each saved segment regis= ter * with its current contents: any discrepancy means we in category 1. */ -SYM_CODE_START(xen_failsafe_callback) + __FUNC_ALIGN +SYM_CODE_START_NOALIGN(xen_failsafe_callback) UNWIND_HINT_EMPTY ENDBR movl %ds, %ecx @@ -1516,7 +1519,8 @@ SYM_CODE_END(ignore_sysret) #endif =20 .pushsection .text, "ax" -SYM_CODE_START(rewind_stack_and_make_dead) + __FUNC_ALIGN +SYM_CODE_START_NOALIGN(rewind_stack_and_make_dead) UNWIND_HINT_FUNC /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp --- a/arch/x86/entry/thunk_64.S +++ b/arch/x86/entry/thunk_64.S @@ -11,7 +11,7 @@ =20 /* rdi: arg1 ... normal C conventions. rax is saved/restored. */ .macro THUNK name, func -SYM_FUNC_START_NOALIGN(\name) +SYM_FUNC_START(\name) pushq %rbp movq %rsp, %rbp =20 @@ -36,7 +36,7 @@ SYM_FUNC_END(\name) EXPORT_SYMBOL(preempt_schedule_thunk) EXPORT_SYMBOL(preempt_schedule_notrace_thunk) =20 -SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore) +SYM_CODE_START_LOCAL(__thunk_restore) popq %r11 popq %r10 popq %r9 From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C26BC38145 for ; Fri, 2 Sep 2022 14:29:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236086AbiIBO32 (ORCPT ); Fri, 2 Sep 2022 10:29:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236779AbiIBO1b (ORCPT ); Fri, 2 Sep 2022 10:27:31 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18C6E141A1C for ; Fri, 2 Sep 2022 06:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=BJHqXamgAJqTcT2FFMKxQyrqmVcujF7M2+0CHXkEhUI=; b=H5hU9sfBcD7OUyu4swO7EK/OTZ MmGmNZef0mMHGjyQq8bgUPvguOphKna5I1Hw1FlXrFgP+EA3wQXJKyrUCNhyLfCdgW8oS+QXIqd0x DhvuaRO98BdPJYnC0Ln68CNn8w39QW4ES0XmD5+hhh1XJTLoGtPvTVnsAEWUEbWL5HJY7WdQrHnL8 HYh4m7LkOW6z/VoylyEZiNBPPHmYCbjLoRt3IsoOUlHBj6NI+8Rw//OZKvj9HsI8Z1Mmp7FRQTOHm mTeMYNpOoiOzUCMAlur2oAWSN5u0kfts3ENZIYvxLLRqxAjFKh6k1g0Rgr4xOxVAwBWD0l48CBAYY NoHy5w7Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-0074T9-TV; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 128AC301BCC; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 57F1F2B077D29; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.709764677@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:38 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 13/59] crypto: x86/camellia: Remove redundant alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Also, with having pushed the function alignment to 16 bytes, this custom alignment is completely superfluous. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 2 -- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 4 ---- 2 files changed, 6 deletions(-) --- a/arch/x86/crypto/camellia-aesni-avx-asm_64.S +++ b/arch/x86/crypto/camellia-aesni-avx-asm_64.S @@ -712,7 +712,6 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1 =20 .text =20 -.align 8 SYM_FUNC_START_LOCAL(__camellia_enc_blk16) /* input: * %rdi: ctx, CTX @@ -799,7 +798,6 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk1 jmp .Lenc_done; SYM_FUNC_END(__camellia_enc_blk16) =20 -.align 8 SYM_FUNC_START_LOCAL(__camellia_dec_blk16) /* input: * %rdi: ctx, CTX --- a/arch/x86/crypto/camellia-aesni-avx2-asm_64.S +++ b/arch/x86/crypto/camellia-aesni-avx2-asm_64.S @@ -221,7 +221,6 @@ * Size optimization... with inlined roundsm32 binary would be over 5 times * larger and would only marginally faster. */ -.align 8 SYM_FUNC_START_LOCAL(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y= 6_y7_cd) roundsm32(%ymm0, %ymm1, %ymm2, %ymm3, %ymm4, %ymm5, %ymm6, %ymm7, %ymm8, %ymm9, %ymm10, %ymm11, %ymm12, %ymm13, %ymm14, %ymm15, @@ -229,7 +228,6 @@ SYM_FUNC_START_LOCAL(roundsm32_x0_x1_x2_ RET; SYM_FUNC_END(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) =20 -.align 8 SYM_FUNC_START_LOCAL(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y= 2_y3_ab) roundsm32(%ymm4, %ymm5, %ymm6, %ymm7, %ymm0, %ymm1, %ymm2, %ymm3, %ymm12, %ymm13, %ymm14, %ymm15, %ymm8, %ymm9, %ymm10, %ymm11, @@ -748,7 +746,6 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1 =20 .text =20 -.align 8 SYM_FUNC_START_LOCAL(__camellia_enc_blk32) /* input: * %rdi: ctx, CTX @@ -835,7 +832,6 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk3 jmp .Lenc_done; SYM_FUNC_END(__camellia_enc_blk32) =20 -.align 8 SYM_FUNC_START_LOCAL(__camellia_dec_blk32) /* input: * %rdi: ctx, CTX From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13E6AECAAD5 for ; Fri, 2 Sep 2022 14:32:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236337AbiIBOcT (ORCPT ); Fri, 2 Sep 2022 10:32:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237807AbiIBO2p (ORCPT ); Fri, 2 Sep 2022 10:28:45 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F0D9168A3A for ; Fri, 2 Sep 2022 06:54:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LJYRuy5MUBpvdFzRrxNRJ46Veaw2HYHPfKqWAAmsnTo=; b=CiXmaLxLaQ6GadKViPGJF64Mc9 40DevdvwZBgEZlbI6kZNsqA+NQccGQkrKmHLZ8++UYo+BQdLTX772ybOl/bWDtkZafEQxNizLZR3a cQyw3HWc4tI7n/qtIcxg54yr66xlkV05XOVJMGodblKOodZGlgTUeAFcjciqpY7D1rnQW6mgp3hyb XIUpfLL1IZzYWFx9n4k1XAFYRsiPpxdRB7iBrDVfMRTd6a3wV3I9xwps6d8HK327r1byIaqThA4m7 OHwcihArYdoi+V3UM8dI/tQXB/61VF2ilnbQ+4DBXATRCvQ3h7EUrgWdH3fYjzzYxn8ULF9sJsKcR jIBxoptQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g86-PY; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 16553301C3A; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 5B2052B3C513B; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.813598767@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:39 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 14/59] crypto: x86/cast5: Remove redundant alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Also, with having pushed the function alignment to 16 bytes, this custom alignment is completely superfluous. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 2 -- 1 file changed, 2 deletions(-) --- a/arch/x86/crypto/cast5-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/cast5-avx-x86_64-asm_64.S @@ -208,7 +208,6 @@ =20 .text =20 -.align 16 SYM_FUNC_START_LOCAL(__cast5_enc_blk16) /* input: * %rdi: ctx @@ -282,7 +281,6 @@ SYM_FUNC_START_LOCAL(__cast5_enc_blk16) RET; SYM_FUNC_END(__cast5_enc_blk16) =20 -.align 16 SYM_FUNC_START_LOCAL(__cast5_dec_blk16) /* input: * %rdi: ctx From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C71EBECAAD5 for ; Fri, 2 Sep 2022 14:28:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237138AbiIBO2b (ORCPT ); Fri, 2 Sep 2022 10:28:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236159AbiIBO1L (ORCPT ); Fri, 2 Sep 2022 10:27:11 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AC6F145C55 for ; Fri, 2 Sep 2022 06:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=i3Uz19PgVRfZIUacrE/ZrBQkKie5hxegnp8/5GnSRe0=; b=CvLdc8+8GZmSFm4gLwiOwmrHci xBUm8kMJl8ycTLFvH3Bik6Eko1xvJbJyQjV/dprYAuVLqGK6p+ZOI79jK6IAQx0cDxfum1Fx8WEC6 3Swr9pVmuSLrh5FRAg0r1fUWX5uA6vtplHyz4J7KjtmLoHzj5UioY9Q7Qov1oajhnQDYH8HbATqw4 0YbDwzjX6czeg0XOXrMm0V5NZsz+uBfL70wOnZAOtV3d5KDtEU5mj3o5rtjE0XO9C5O9i5sfTnuU3 4OyEey/C9Gmmtj+qd7SWBQr+dbIhBQMZcIFFr1Y8vZbcKhjF7gkOdkRKr+ck867ejnnIsEXA+1iyt zo237iOg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-0074TA-UL; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 144F4301C36; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 609552B572DF7; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130947.914005103@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:40 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 15/59] crypto: x86/crct10dif-pcl: Remove redundant alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Also, with having pushed the function alignment to 16 bytes, this custom alignment is completely superfluous. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/crct10dif-pcl-asm_64.S | 1 - 1 file changed, 1 deletion(-) --- a/arch/x86/crypto/crct10dif-pcl-asm_64.S +++ b/arch/x86/crypto/crct10dif-pcl-asm_64.S @@ -94,7 +94,6 @@ # # Assumes len >=3D 16. # -.align 16 SYM_FUNC_START(crc_t10dif_pcl) =20 movdqa .Lbswap_mask(%rip), BSWAP_MASK From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 121FDC38145 for ; Fri, 2 Sep 2022 14:28:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236955AbiIBO2Q (ORCPT ); Fri, 2 Sep 2022 10:28:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236132AbiIBO1J (ORCPT ); Fri, 2 Sep 2022 10:27:09 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18B56141A12 for ; Fri, 2 Sep 2022 06:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=kiUBY9L9H1enuPzKP5b8g1QT9pTcpJ/p9g5AdHuOGVs=; b=CGSvGIdS7BzFlGPC66zM67Zsuy kJNdSIHdK9ZLcOaYZTpT6E3z703HT21C8Dat37616OGCJG3cZtmwPUDloeuiEmIB569JS1SHyEIHi MYz8jua+e4cZMkrg4hapyzRfhvuOxD9J22EOsXlvO4yWnsNp3SvAqklmFN3x0BoVSl6NYTE+/X94c xTPSwpQsrnjlIvkv3wh2JRO1BMkJN+HjuHu0g718UhTmlKIngZWrPkvVgd2JH5iiaXoLoYLYA8tgv JxTNyEh6RSQ5OEFBnfnBTuPTQY5ZWLuDJNNQdO3R9SsMhCa/qSguFEroBEU+l1Z8DINn+je5bPSPz bS/9PKfQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g83-Mx; Fri, 02 Sep 2022 13:53:55 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1446E301BF2; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 65F492B1A6BC7; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.022127024@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:41 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 16/59] crypto: x86/serpent: Remove redundant alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Also, with having pushed the function alignment to 16 bytes, this custom alignment is completely superfluous. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 2 -- arch/x86/crypto/serpent-avx2-asm_64.S | 2 -- 2 files changed, 4 deletions(-) --- a/arch/x86/crypto/serpent-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/serpent-avx-x86_64-asm_64.S @@ -550,7 +550,6 @@ #define write_blocks(x0, x1, x2, x3, t0, t1, t2) \ transpose_4x4(x0, x1, x2, x3, t0, t1, t2) =20 -.align 8 SYM_FUNC_START_LOCAL(__serpent_enc_blk8_avx) /* input: * %rdi: ctx, CTX @@ -604,7 +603,6 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk8_ RET; SYM_FUNC_END(__serpent_enc_blk8_avx) =20 -.align 8 SYM_FUNC_START_LOCAL(__serpent_dec_blk8_avx) /* input: * %rdi: ctx, CTX --- a/arch/x86/crypto/serpent-avx2-asm_64.S +++ b/arch/x86/crypto/serpent-avx2-asm_64.S @@ -550,7 +550,6 @@ #define write_blocks(x0, x1, x2, x3, t0, t1, t2) \ transpose_4x4(x0, x1, x2, x3, t0, t1, t2) =20 -.align 8 SYM_FUNC_START_LOCAL(__serpent_enc_blk16) /* input: * %rdi: ctx, CTX @@ -604,7 +603,6 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk16 RET; SYM_FUNC_END(__serpent_enc_blk16) =20 -.align 8 SYM_FUNC_START_LOCAL(__serpent_dec_blk16) /* input: * %rdi: ctx, CTX From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16ADAECAAD5 for ; Fri, 2 Sep 2022 14:27:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237224AbiIBO1g (ORCPT ); Fri, 2 Sep 2022 10:27:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236702AbiIBO1B (ORCPT ); Fri, 2 Sep 2022 10:27:01 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCA9FE926C for ; Fri, 2 Sep 2022 06:54:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=81/ujGXHG3X7Sjz3zTn5ireIwtU9AJNdDga1fZrvIMw=; b=Ek2vrkgbZxClCajtuiRSa01Fk5 8aDMIUsCLNjafzuzDQUSaGUWGRfyTX6qTJ+eEXVqyrPlC5N58skYODUOCJnhGZsmVXXJEeq/vqX4a aGRSh0rCTuV0Fci6qBOQ90nv4X4MfT/wb+rM4IXGdLTsjiZ9fSsDKqwFVQfBTYiK4bULIb6d11lmY uTQ9aEyfigNZBo+it+WR/3HufoHeBNFv+3ZurYP6YbXrrFFA28k2whCktOwEjZ6gpNIY/dan3t2w7 SrgZ2pD4KOsMwoJdxmNqOg3g0oTqXl0vlFBSDfGCSCJAlCUuCgSg527ac1kdaAUke4gQb0vE8uX8u rYpFiv/w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g84-Pb; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1440D301BEC; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 6C7B72B363A1E; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.123654749@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:42 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 17/59] crypto: x86/sha1: Remove custom alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/sha1_ni_asm.S | 1 - 1 file changed, 1 deletion(-) --- a/arch/x86/crypto/sha1_ni_asm.S +++ b/arch/x86/crypto/sha1_ni_asm.S @@ -92,7 +92,6 @@ * numBlocks: Number of blocks to process */ .text -.align 32 SYM_FUNC_START(sha1_ni_transform) push %rbp mov %rsp, %rbp From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9849FC38145 for ; Fri, 2 Sep 2022 14:31:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236240AbiIBOb3 (ORCPT ); Fri, 2 Sep 2022 10:31:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236107AbiIBO1t (ORCPT ); Fri, 2 Sep 2022 10:27:49 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4816A1581AD for ; Fri, 2 Sep 2022 06:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=i/KYE4ASCs+l+r/1pSA03LLegKU4fUGJ4neS2scNk+A=; b=jHvC2jsPvvpquUQaNH+gCvsIn8 Ed7ctOYaiVnoc5RJTn++PQv0x6MP05+uGAlSGBSXRaWqq/+CCzB/3Y0xeROW1QrEe4iAE1sAD2Lc7 7UXc3LTAQVzin3qflzaiNhvF35Oc+LpMdGQEL+98FQRSB72PyMQP8TuOZfOJV+Noy0f4Vf3JYiAOj 4WP8h4aFPQfV0qGTjoaEKWuxES+BCZTXxl+dhG/ZEYGqch/XW5PaId+BBhFz1ZcyEzN+BHa8Hp5o6 uyCWRttIrF0hq8cwnWO1tPQpoU3O/HLLgAkyA8LqL4nRdACs7iULE0dw8mKBMNTJPwqAfL8bClBh1 DwREJWbQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g81-MI; Fri, 02 Sep 2022 13:54:03 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 14483301C35; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 749582B8AE594; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.228934694@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:43 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 18/59] crypto: x86/sha256: Remove custom alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/sha256-avx-asm.S | 1 - arch/x86/crypto/sha256-avx2-asm.S | 1 - arch/x86/crypto/sha256-ssse3-asm.S | 1 - arch/x86/crypto/sha256_ni_asm.S | 1 - 4 files changed, 4 deletions(-) --- a/arch/x86/crypto/sha256-avx-asm.S +++ b/arch/x86/crypto/sha256-avx-asm.S @@ -347,7 +347,6 @@ a =3D TMP_ ######################################################################## .text SYM_FUNC_START(sha256_transform_avx) -.align 32 pushq %rbx pushq %r12 pushq %r13 --- a/arch/x86/crypto/sha256-avx2-asm.S +++ b/arch/x86/crypto/sha256-avx2-asm.S @@ -524,7 +524,6 @@ STACK_SIZE =3D _CTX + _CTX_SIZE ######################################################################## .text SYM_FUNC_START(sha256_transform_rorx) -.align 32 pushq %rbx pushq %r12 pushq %r13 --- a/arch/x86/crypto/sha256-ssse3-asm.S +++ b/arch/x86/crypto/sha256-ssse3-asm.S @@ -356,7 +356,6 @@ a =3D TMP_ ######################################################################## .text SYM_FUNC_START(sha256_transform_ssse3) -.align 32 pushq %rbx pushq %r12 pushq %r13 --- a/arch/x86/crypto/sha256_ni_asm.S +++ b/arch/x86/crypto/sha256_ni_asm.S @@ -96,7 +96,6 @@ */ =20 .text -.align 32 SYM_FUNC_START(sha256_ni_transform) =20 shl $6, NUM_BLKS /* convert to bytes */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 539BAC38145 for ; Fri, 2 Sep 2022 14:32:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235893AbiIBOcY (ORCPT ); Fri, 2 Sep 2022 10:32:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237890AbiIBO2q (ORCPT ); Fri, 2 Sep 2022 10:28:46 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B42DD13F66 for ; Fri, 2 Sep 2022 06:54:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=I2N57Nlq+w/EFdJFmxq8wOkT8KRRYqRZgwOlt/Ls3NU=; b=QjRVDgaoHYpUC28SmZJRPYSCLy SMBrXOg7vu9rL9QEBkuG8oq2cVrnE9hFLzi1NM3jHRuZDolKvjrMef8MzZhyL2//Kk3tSMjAUvfOH WJhE39IUWDgQ3ZsdCxW/A9ioev4BQmNXwUXAfqC1cPoKl8EDH+BnvsHmVYruCpjb3cBMOcgOwPoja 93Fgb1jWv7M3kL6rdET+wBs1hsf+rrVCMS6td1x1RaCqT4BG4uKu3IE/wH944kBm8OBmaCXKpn/3A Lr0GDKz6Km82kS+yH/SOWFNLkHVHADVITYXr4HyRq2rGk4Qj7TLLrqxm3vP4Co+WXD3MJWRSaPEub b+MiHXUQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g88-RI; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1B1C6301C3D; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 7C8FA2B8EF7E9; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.334367209@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:44 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 19/59] crypto: x86/sm[34]: Remove redundant alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Also, with having pushed the function alignment to 16 bytes, this custom alignment is completely superfluous. ( this code couldn't seem to make up it's mind about what alignment it actually wanted, randomly mixing 8 and 16 bytes ) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/sm3-avx-asm_64.S | 1 - arch/x86/crypto/sm4-aesni-avx-asm_64.S | 7 ------- arch/x86/crypto/sm4-aesni-avx2-asm_64.S | 6 ------ 3 files changed, 14 deletions(-) --- a/arch/x86/crypto/sm3-avx-asm_64.S +++ b/arch/x86/crypto/sm3-avx-asm_64.S @@ -327,7 +327,6 @@ * void sm3_transform_avx(struct sm3_state *state, * const u8 *data, int nblocks); */ -.align 16 SYM_FUNC_START(sm3_transform_avx) /* input: * %rdi: ctx, CTX --- a/arch/x86/crypto/sm4-aesni-avx-asm_64.S +++ b/arch/x86/crypto/sm4-aesni-avx-asm_64.S @@ -139,13 +139,11 @@ =20 =20 .text -.align 16 =20 /* * void sm4_aesni_avx_crypt4(const u32 *rk, u8 *dst, * const u8 *src, int nblocks) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx_crypt4) /* input: * %rdi: round key array, CTX @@ -249,7 +247,6 @@ SYM_FUNC_START(sm4_aesni_avx_crypt4) RET; SYM_FUNC_END(sm4_aesni_avx_crypt4) =20 -.align 8 SYM_FUNC_START_LOCAL(__sm4_crypt_blk8) /* input: * %rdi: round key array, CTX @@ -363,7 +360,6 @@ SYM_FUNC_END(__sm4_crypt_blk8) * void sm4_aesni_avx_crypt8(const u32 *rk, u8 *dst, * const u8 *src, int nblocks) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx_crypt8) /* input: * %rdi: round key array, CTX @@ -419,7 +415,6 @@ SYM_FUNC_END(sm4_aesni_avx_crypt8) * void sm4_aesni_avx_ctr_enc_blk8(const u32 *rk, u8 *dst, * const u8 *src, u8 *iv) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx_ctr_enc_blk8) /* input: * %rdi: round key array, CTX @@ -494,7 +489,6 @@ SYM_FUNC_END(sm4_aesni_avx_ctr_enc_blk8) * void sm4_aesni_avx_cbc_dec_blk8(const u32 *rk, u8 *dst, * const u8 *src, u8 *iv) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx_cbc_dec_blk8) /* input: * %rdi: round key array, CTX @@ -544,7 +538,6 @@ SYM_FUNC_END(sm4_aesni_avx_cbc_dec_blk8) * void sm4_aesni_avx_cfb_dec_blk8(const u32 *rk, u8 *dst, * const u8 *src, u8 *iv) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx_cfb_dec_blk8) /* input: * %rdi: round key array, CTX --- a/arch/x86/crypto/sm4-aesni-avx2-asm_64.S +++ b/arch/x86/crypto/sm4-aesni-avx2-asm_64.S @@ -153,9 +153,6 @@ .long 0xdeadbeef, 0xdeadbeef, 0xdeadbeef =20 .text -.align 16 - -.align 8 SYM_FUNC_START_LOCAL(__sm4_crypt_blk16) /* input: * %rdi: round key array, CTX @@ -281,7 +278,6 @@ SYM_FUNC_END(__sm4_crypt_blk16) * void sm4_aesni_avx2_ctr_enc_blk16(const u32 *rk, u8 *dst, * const u8 *src, u8 *iv) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx2_ctr_enc_blk16) /* input: * %rdi: round key array, CTX @@ -394,7 +390,6 @@ SYM_FUNC_END(sm4_aesni_avx2_ctr_enc_blk1 * void sm4_aesni_avx2_cbc_dec_blk16(const u32 *rk, u8 *dst, * const u8 *src, u8 *iv) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx2_cbc_dec_blk16) /* input: * %rdi: round key array, CTX @@ -448,7 +443,6 @@ SYM_FUNC_END(sm4_aesni_avx2_cbc_dec_blk1 * void sm4_aesni_avx2_cfb_dec_blk16(const u32 *rk, u8 *dst, * const u8 *src, u8 *iv) */ -.align 8 SYM_FUNC_START(sm4_aesni_avx2_cfb_dec_blk16) /* input: * %rdi: round key array, CTX From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38EB4ECAAD5 for ; Fri, 2 Sep 2022 14:32:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236556AbiIBOct (ORCPT ); Fri, 2 Sep 2022 10:32:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238081AbiIBO2s (ORCPT ); Fri, 2 Sep 2022 10:28:48 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2FAB645A for ; Fri, 2 Sep 2022 06:54:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=NCzQoa/lN1S6axs7r7HZ/1PQF1CVTuK/sTcNriIHpb0=; b=ljrTOqkCgoayvuxcjnVV7eiFw6 M6eXznFxeOdC5Y4TbPjCRbMqpaL9s0y0cUA2kV8GHhLjNTMFRn1SqmxcMo5Zq6h9/QCYa++nkIHrM XOjFC83+X88mpEH5FdGk9GiwNOF1CVkciKNM0E0dh9aiaH/Sr8Vh1GzE6cav1vXg8pYuUGK4oDFNz GIPztkClBz3KY0zu02crVmNMrVpeOanaU8MmgDVRqArA8CRfNobKMyhsUkqvDkcEvicTMdzUlR7cT bARtgIW2fGjto14HZ0EtI48Lju9DQgpc0AYlDqW5HuTMwA2xR3/fpNSHOCtXAm+JrbL0o0jbIKg8a Lbjn3QtA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TB-2z; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1D06A301C5C; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 848772B8EF7EA; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.436411782@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:45 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 20/59] crypto: twofish: Remove redundant alignments References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Also, with having pushed the function alignment to 16 bytes, this custom alignment is completely superfluous. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 2 -- 1 file changed, 2 deletions(-) --- a/arch/x86/crypto/twofish-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/twofish-avx-x86_64-asm_64.S @@ -228,7 +228,6 @@ vpxor x2, wkey, x2; \ vpxor x3, wkey, x3; =20 -.align 8 SYM_FUNC_START_LOCAL(__twofish_enc_blk8) /* input: * %rdi: ctx, CTX @@ -270,7 +269,6 @@ SYM_FUNC_START_LOCAL(__twofish_enc_blk8) RET; SYM_FUNC_END(__twofish_enc_blk8) =20 -.align 8 SYM_FUNC_START_LOCAL(__twofish_dec_blk8) /* input: * %rdi: ctx, CTX From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A073CECAAD5 for ; Fri, 2 Sep 2022 14:29:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236774AbiIBO3H (ORCPT ); Fri, 2 Sep 2022 10:29:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51442 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237142AbiIBO12 (ORCPT ); Fri, 2 Sep 2022 10:27:28 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A4FF141A39 for ; Fri, 2 Sep 2022 06:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=l2/obFJyKasfRTQBGbLPrEIMupVJLFma++FBtdTQaqY=; b=K3LsArsiWo63cd+vIFwD1WOOSr +QBQ9MYoBfLvmgxJ1+hBka7/PZjaCMU2FKsCtdGKXQN/Jr4N+mnu8Crq34JrE7xCgd49bxphfYRwu BA4q3f/pl8c0oGcli6K8UCmsqqi9p3Lio+DZSLBDDaX/7AZM4txPIxjk8cRSyZ99Bi4o+gYG1Vfr4 Gzf+mi2FsCS06FV165UcOjeRz/9ndGFQI3UZrRD3x1aQC/cmREjAco/12+8IiLOiQ6pnJckM0J61x n+WDlpm9fQzKysJPYcnl+gz1KsX02cuzqgT0mb/GwcR/+95fYTHbPba7Buxnz3oP+oqCk+/MxDMYq FJRMzfRA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77L-008g89-Rj; Fri, 02 Sep 2022 13:53:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1E227301C77; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8DB712B8EF7EB; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.540944209@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:46 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 21/59] crypto: x86/poly1305: Remove custom function alignment References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner SYM_FUNC_START*() and friends already imply alignment, remove custom alignment hacks to make code consistent. This prepares for future function call ABI changes. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 1 - 1 file changed, 1 deletion(-) --- a/arch/x86/crypto/poly1305-x86_64-cryptogams.pl +++ b/arch/x86/crypto/poly1305-x86_64-cryptogams.pl @@ -108,7 +108,6 @@ if (!$kernel) { sub declare_function() { my ($name, $align, $nargs) =3D @_; if($kernel) { - $code .=3D ".align $align\n"; $code .=3D "SYM_FUNC_START($name)\n"; $code .=3D ".L$name:\n"; } else { From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92DC1ECAAD5 for ; Fri, 2 Sep 2022 14:27:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237368AbiIBO1j (ORCPT ); Fri, 2 Sep 2022 10:27:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237095AbiIBO1I (ORCPT ); Fri, 2 Sep 2022 10:27:08 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A867168A21 for ; Fri, 2 Sep 2022 06:54:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=GP9RQ5R7xsKdkjMYemE/9mRCoebu4tK80lw9bb9cJCs=; b=BJqMF9zindO1dcWsqVkXWSn4Um lFUGB50o6xAswL6hhKGURpmNqnDX898DJ1rqgM0YJ1HoNbo107P9If5zusATBgenXpwMzaA0NcZgm kbVwwIfO/OoyD01U70BUC22jDdWN1LXggcRJBojM/N0uHgjbkc7sb0cAOfW60w84etfL2725izOrv XL47fpcSI2gki2fD/H5MKKKRwCOHdqWzSALN30F0Rl1AyKd+cKbePzUK07ZxPxtbgtCjfSf98FrnP hLE10AMVQL9u1/jaYH1RHzYCe2hm+ibRFfHx87vPKplpkTeyXuYWcF+KG4RjfxfY4hPoQpuaeCFXE pKAabX/Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TE-EG; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 21146301C78; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 973BE2B8EF7EC; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.643735860@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:47 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 22/59] x86: Put hot per CPU variables into a struct References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The layout of per-cpu variables is at the mercy of the compiler. This can lead to random performance fluctuations from build to build. Create a structure to hold some of the hottest per-cpu variables, starting with current_task. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/current.h | 19 ++++++++++++++++--- arch/x86/kernel/cpu/common.c | 14 +++++--------- arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- 5 files changed, 24 insertions(+), 15 deletions(-) --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -3,16 +3,29 @@ #define _ASM_X86_CURRENT_H =20 #include -#include =20 #ifndef __ASSEMBLY__ + +#include +#include + struct task_struct; =20 -DECLARE_PER_CPU(struct task_struct *, current_task); +struct pcpu_hot { + union { + struct { + struct task_struct *current_task; + }; + u8 pad[64]; + }; +}; +static_assert(sizeof(struct pcpu_hot) =3D=3D 64); + +DECLARE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot); =20 static __always_inline struct task_struct *get_current(void) { - return this_cpu_read_stable(current_task); + return this_cpu_read_stable(pcpu_hot.current_task); } =20 #define current get_current() --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2000,18 +2000,16 @@ static __init int setup_clearcpuid(char } __setup("clearcpuid=3D", setup_clearcpuid); =20 +DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { + .current_task =3D &init_task, +}; +EXPORT_PER_CPU_SYMBOL(pcpu_hot); + #ifdef CONFIG_X86_64 DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __aligned(PAGE_SIZE) __visible; EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_data); =20 -/* - * The following percpu variables are hot. Align current_task to - * cacheline size such that they fall in the same cacheline. - */ -DEFINE_PER_CPU(struct task_struct *, current_task) ____cacheline_aligned = =3D - &init_task; -EXPORT_PER_CPU_SYMBOL(current_task); =20 DEFINE_PER_CPU(void *, hardirq_stack_ptr); DEFINE_PER_CPU(bool, hardirq_stack_inuse); @@ -2071,8 +2069,6 @@ void syscall_init(void) =20 #else /* CONFIG_X86_64 */ =20 -DEFINE_PER_CPU(struct task_struct *, current_task) =3D &init_task; -EXPORT_PER_CPU_SYMBOL(current_task); DEFINE_PER_CPU(int, __preempt_count) =3D INIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -207,7 +207,7 @@ EXPORT_SYMBOL_GPL(start_thread); if (prev->gs | next->gs) loadsegment(gs, next->gs); =20 - this_cpu_write(current_task, next_p); + raw_cpu_write(pcpu_hot.current_task, next_p); =20 switch_fpu_finish(); =20 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -616,7 +616,7 @@ void compat_start_thread(struct pt_regs /* * Switch the PDA and FPU contexts. */ - this_cpu_write(current_task, next_p); + raw_cpu_write(pcpu_hot.current_task, next_p); this_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(); --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1046,7 +1046,7 @@ int common_cpu_up(unsigned int cpu, stru /* Just in case we booted with a single CPU. */ alternatives_enable_smp(); =20 - per_cpu(current_task, cpu) =3D idle; + per_cpu(pcpu_hot.current_task, cpu) =3D idle; cpu_init_stack_canary(cpu, idle); =20 /* Initialize the interrupt stack(s) */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3904ECAAD5 for ; Fri, 2 Sep 2022 14:29:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236195AbiIBO3h (ORCPT ); Fri, 2 Sep 2022 10:29:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235728AbiIBO1b (ORCPT ); Fri, 2 Sep 2022 10:27:31 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F28115819B for ; Fri, 2 Sep 2022 06:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=xQuJtl6Oe7h8UkTWqXBwC3SMQ2iiCp9oKrM8vI/KEDI=; b=hpQY45xhJ745Sp+Wv01ZRBCnJH 72SrTb9YfMus0tEIH5wTjZwYGpXXOGG5kVV6AFYrH9YRGjUKP6rZI1VrZ4So4ihylSNd7kaP3nVnK JQRaw53ydRQlNFFCI2Bi7C7/JkP3Wl7/NolVm/wk0nsFSR+hma8qgIM5kXT33eR/GvmGQIEM4m3PQ s1iRPtm4fSI5UhtRJtg1XPbaoGadnPWyCmAbyH2/aCgeIo7lmdYsh1Hh/s3Z9gncewu7YwV0J2i/B 6dOp92Nez1Ab0p//4McD8+9AS1198GOl/wo10v0EjkrZjXzsyfszCZ3u/R0ZB8xeJYo/We1p71/zq /dLYdF9g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TC-3G; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 22DC6301CA0; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id A06FE2B8EF7ED; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.748413205@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:48 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 23/59] x86/percpu: Move preempt_count next to current_task References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Add preempt_count to pcpu_hot, since it is once of the most used per-cpu variables. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/current.h | 1 + arch/x86/include/asm/preempt.h | 27 ++++++++++++++------------- arch/x86/kernel/cpu/common.c | 8 +------- 3 files changed, 16 insertions(+), 20 deletions(-) --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -15,6 +15,7 @@ struct pcpu_hot { union { struct { struct task_struct *current_task; + int preempt_count; }; u8 pad[64]; }; --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -4,11 +4,11 @@ =20 #include #include +#include + #include #include =20 -DECLARE_PER_CPU(int, __preempt_count); - /* We use the MSB mostly because its available */ #define PREEMPT_NEED_RESCHED 0x80000000 =20 @@ -24,7 +24,7 @@ DECLARE_PER_CPU(int, __preempt_count); */ static __always_inline int preempt_count(void) { - return raw_cpu_read_4(__preempt_count) & ~PREEMPT_NEED_RESCHED; + return raw_cpu_read_4(pcpu_hot.preempt_count) & ~PREEMPT_NEED_RESCHED; } =20 static __always_inline void preempt_count_set(int pc) @@ -32,10 +32,10 @@ static __always_inline void preempt_coun int old, new; =20 do { - old =3D raw_cpu_read_4(__preempt_count); + old =3D raw_cpu_read_4(pcpu_hot.preempt_count); new =3D (old & PREEMPT_NEED_RESCHED) | (pc & ~PREEMPT_NEED_RESCHED); - } while (raw_cpu_cmpxchg_4(__preempt_count, old, new) !=3D old); + } while (raw_cpu_cmpxchg_4(pcpu_hot.preempt_count, old, new) !=3D old); } =20 /* @@ -44,7 +44,7 @@ static __always_inline void preempt_coun #define init_task_preempt_count(p) do { } while (0) =20 #define init_idle_preempt_count(p, cpu) do { \ - per_cpu(__preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ + per_cpu(pcpu_hot.preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ } while (0) =20 /* @@ -58,17 +58,17 @@ static __always_inline void preempt_coun =20 static __always_inline void set_preempt_need_resched(void) { - raw_cpu_and_4(__preempt_count, ~PREEMPT_NEED_RESCHED); + raw_cpu_and_4(pcpu_hot.preempt_count, ~PREEMPT_NEED_RESCHED); } =20 static __always_inline void clear_preempt_need_resched(void) { - raw_cpu_or_4(__preempt_count, PREEMPT_NEED_RESCHED); + raw_cpu_or_4(pcpu_hot.preempt_count, PREEMPT_NEED_RESCHED); } =20 static __always_inline bool test_preempt_need_resched(void) { - return !(raw_cpu_read_4(__preempt_count) & PREEMPT_NEED_RESCHED); + return !(raw_cpu_read_4(pcpu_hot.preempt_count) & PREEMPT_NEED_RESCHED); } =20 /* @@ -77,12 +77,12 @@ static __always_inline bool test_preempt =20 static __always_inline void __preempt_count_add(int val) { - raw_cpu_add_4(__preempt_count, val); + raw_cpu_add_4(pcpu_hot.preempt_count, val); } =20 static __always_inline void __preempt_count_sub(int val) { - raw_cpu_add_4(__preempt_count, -val); + raw_cpu_add_4(pcpu_hot.preempt_count, -val); } =20 /* @@ -92,7 +92,8 @@ static __always_inline void __preempt_co */ static __always_inline bool __preempt_count_dec_and_test(void) { - return GEN_UNARY_RMWcc("decl", __preempt_count, e, __percpu_arg([var])); + return GEN_UNARY_RMWcc("decl", pcpu_hot.preempt_count, e, + __percpu_arg([var])); } =20 /* @@ -100,7 +101,7 @@ static __always_inline bool __preempt_co */ static __always_inline bool should_resched(int preempt_offset) { - return unlikely(raw_cpu_read_4(__preempt_count) =3D=3D preempt_offset); + return unlikely(raw_cpu_read_4(pcpu_hot.preempt_count) =3D=3D preempt_off= set); } =20 #ifdef CONFIG_PREEMPTION --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2002,6 +2002,7 @@ static __init int setup_clearcpuid(char =20 DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, + .preempt_count =3D INIT_PREEMPT_COUNT, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); =20 @@ -2010,13 +2011,9 @@ DEFINE_PER_CPU_FIRST(struct fixed_percpu fixed_percpu_data) __aligned(PAGE_SIZE) __visible; EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_data); =20 - DEFINE_PER_CPU(void *, hardirq_stack_ptr); DEFINE_PER_CPU(bool, hardirq_stack_inuse); =20 -DEFINE_PER_CPU(int, __preempt_count) =3D INIT_PREEMPT_COUNT; -EXPORT_PER_CPU_SYMBOL(__preempt_count); - DEFINE_PER_CPU(unsigned long, cpu_current_top_of_stack) =3D TOP_OF_INIT_ST= ACK; =20 static void wrmsrl_cstar(unsigned long val) @@ -2069,9 +2066,6 @@ void syscall_init(void) =20 #else /* CONFIG_X86_64 */ =20 -DEFINE_PER_CPU(int, __preempt_count) =3D INIT_PREEMPT_COUNT; -EXPORT_PER_CPU_SYMBOL(__preempt_count); - /* * On x86_32, vm86 modifies tss.sp0, so sp0 isn't a reliable way to find * the top of the kernel stack. Use an extra percpu variable to track the From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98421ECAAD5 for ; Fri, 2 Sep 2022 14:31:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232701AbiIBObc (ORCPT ); Fri, 2 Sep 2022 10:31:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237034AbiIBO1e (ORCPT ); Fri, 2 Sep 2022 10:27:34 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CE6F158196 for ; Fri, 2 Sep 2022 06:54:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=NPIEgVCIlOh8dD4yk3OVetSQQerJmiLfbsdkTOcH3dg=; b=ICzV2MsxV/qfCUsTkmrBYEPHtv a6tR+U9KZPQOQHkHOq30t8Whom4buPfGLdCKoZwilmqZSiwx+98RLBosTT25xs2w3FqCAA25eAOPo C0dA0KB1pd5nsH2uuYDQ/IGpW5Hcw6jo/+u3tI5exboz2i0y6FO9HaIeaWnyzSGKkU8psrol+qp05 biNxIsY29rjmXRBvFo0S0kHWovToVwNxtxvjFe+m7eB++9Q/6/5vKFH0xqBUpULrLtHtG4GxggIJF PK5qAJZLx8LEbdQwiEWdOXw+xGysnHnkUBrHWh6zNyZ/aWiktOVbjXA+ukpQyL8oU6KdYWvHn2Oyv QJr03x3A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TF-ED; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 24D2C301CE0; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id AA1CB2B8EF7EE; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.851118488@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:49 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 24/59] x86/percpu: Move cpu_number next to current_task References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Also add cpu_number to the pcpu_hot structure, it is often referenced and this cacheline is there. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/current.h | 1 + arch/x86/include/asm/smp.h | 12 +++++------- arch/x86/kernel/setup_percpu.c | 5 +---- 3 files changed, 7 insertions(+), 11 deletions(-) --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,6 +16,7 @@ struct pcpu_hot { struct { struct task_struct *current_task; int preempt_count; + int cpu_number; }; u8 pad[64]; }; --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -3,10 +3,10 @@ #define _ASM_X86_SMP_H #ifndef __ASSEMBLY__ #include -#include =20 -#include #include +#include +#include =20 extern int smp_num_siblings; extern unsigned int num_processors; @@ -19,7 +19,6 @@ DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_ DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_l2c_shared_map); DECLARE_PER_CPU_READ_MOSTLY(u16, cpu_llc_id); DECLARE_PER_CPU_READ_MOSTLY(u16, cpu_l2c_id); -DECLARE_PER_CPU_READ_MOSTLY(int, cpu_number); =20 static inline struct cpumask *cpu_llc_shared_mask(int cpu) { @@ -160,11 +159,10 @@ asmlinkage __visible void smp_reboot_int =20 /* * This function is needed by all SMP systems. It must _always_ be valid - * from the initial startup. We map APIC_BASE very early in page_setup(), - * so this is correct in the x86 case. + * from the initial startup. */ -#define raw_smp_processor_id() this_cpu_read(cpu_number) -#define __smp_processor_id() __this_cpu_read(cpu_number) +#define raw_smp_processor_id() this_cpu_read(pcpu_hot.cpu_number) +#define __smp_processor_id() __this_cpu_read(pcpu_hot.cpu_number) =20 #ifdef CONFIG_X86_32 extern int safe_smp_processor_id(void); --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -23,9 +23,6 @@ #include #include =20 -DEFINE_PER_CPU_READ_MOSTLY(int, cpu_number); -EXPORT_PER_CPU_SYMBOL(cpu_number); - #ifdef CONFIG_X86_64 #define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load) #else @@ -172,7 +169,7 @@ void __init setup_per_cpu_areas(void) for_each_possible_cpu(cpu) { per_cpu_offset(cpu) =3D delta + pcpu_unit_offsets[cpu]; per_cpu(this_cpu_off, cpu) =3D per_cpu_offset(cpu); - per_cpu(cpu_number, cpu) =3D cpu; + per_cpu(pcpu_hot.cpu_number, cpu) =3D cpu; setup_percpu_segment(cpu); /* * Copy data used in early init routines from the From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54B72ECAAD5 for ; Fri, 2 Sep 2022 14:29:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237799AbiIBO2p (ORCPT ); Fri, 2 Sep 2022 10:28:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236675AbiIBO1N (ORCPT ); Fri, 2 Sep 2022 10:27:13 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2062EE6BA for ; Fri, 2 Sep 2022 06:54:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=hulrMKM+G54XvOkRK8niCZtARXKO3DviugUwXwClgsU=; b=SLQ+/x0CpgeWTIXTF6IqFFbfqp ubAGGAYvbmSzXE+ajDGZNRCE45i5ooIeWBWn7X6tOitjJ3msEWc/fojkHLxiuzlyrPSrsyiF9+l/f YDMe+rIOT2WsjaMmUX3S6dZdsPwGBxNeKTwfCC1wcCtf7kHaQLK72+1g5298qY2zC6zIm3ClL9tBo Lx6X1T+5J3AG0MkQtcIrfkoG76jQIO/OUYAQ2hMsOs6iODlZkXNLpHZx4jtQAMQgRXNU68t/44D2L 0l6qVQ1H3hJiCisXMm20q86cx3FH45UnSjUnfQody7ApMDkDMe0Db5OpzUgB3CFyq6wT14H959+Y5 dIEYFUbw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8G-FK; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 28394301D73; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B46092B8EF7EF; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130948.954682077@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:50 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 25/59] x86/percpu: Move current_top_of_stack next to current_task References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Extend the struct pcpu_hot cacheline with current_top_of_stack; another very frequently used value. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/entry/entry_32.S | 4 ++-- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 6 +++--- arch/x86/include/asm/current.h | 1 + arch/x86/include/asm/processor.h | 4 +--- arch/x86/kernel/asm-offsets.c | 2 ++ arch/x86/kernel/cpu/common.c | 12 +----------- arch/x86/kernel/process_32.c | 4 ++-- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/traps.c | 4 ++-- 11 files changed, 19 insertions(+), 28 deletions(-) --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -1181,7 +1181,7 @@ SYM_CODE_START(asm_exc_nmi) * is using the thread stack right now, so it's safe for us to use it. */ movl %esp, %ebx - movl PER_CPU_VAR(cpu_current_top_of_stack), %esp + movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esp call exc_nmi movl %ebx, %esp =20 @@ -1243,7 +1243,7 @@ SYM_CODE_START(rewind_stack_and_make_dea /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movl PER_CPU_VAR(cpu_current_top_of_stack), %esi + movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esi leal -TOP_OF_KERNEL_STACK_PADDING-PTREGS_SIZE(%esi), %esp =20 call make_task_dead --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -92,7 +92,7 @@ SYM_CODE_START(entry_SYSCALL_64) /* tss.sp2 is scratch space. */ movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -1209,7 +1209,7 @@ SYM_CODE_START(asm_exc_nmi) FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdx movq %rsp, %rdx - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp UNWIND_HINT_IRET_REGS base=3D%rdx offset=3D8 pushq 5*8(%rdx) /* pt_regs->ss */ pushq 4*8(%rdx) /* pt_regs->rsp */ @@ -1525,7 +1525,7 @@ SYM_CODE_START_NOALIGN(rewind_stack_and_ /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movq PER_CPU_VAR(cpu_current_top_of_stack), %rax + movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rax leaq -PTREGS_SIZE(%rax), %rsp UNWIND_HINT_REGS =20 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -58,7 +58,7 @@ SYM_CODE_START(entry_SYSENTER_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax popq %rax =20 - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp =20 /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ @@ -191,7 +191,7 @@ SYM_CODE_START(entry_SYSCALL_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp =20 /* Switch to the kernel stack */ - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_compat_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -332,7 +332,7 @@ SYM_CODE_START(entry_INT80_compat) ALTERNATIVE "", "jmp .Lint80_keep_stack", X86_FEATURE_XENPV =20 movq %rsp, %rax - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp + movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp =20 pushq 5*8(%rax) /* regs->ss */ pushq 4*8(%rax) /* regs->rsp */ --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -17,6 +17,7 @@ struct pcpu_hot { struct task_struct *current_task; int preempt_count; int cpu_number; + unsigned long top_of_stack; }; u8 pad[64]; }; --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -426,8 +426,6 @@ struct irq_stack { char stack[IRQ_STACK_SIZE]; } __aligned(IRQ_STACK_SIZE); =20 -DECLARE_PER_CPU(unsigned long, cpu_current_top_of_stack); - #ifdef CONFIG_X86_64 struct fixed_percpu_data { /* @@ -566,7 +564,7 @@ static __always_inline unsigned long cur * and around vm86 mode and sp0 on x86_64 is special because of the * entry trampoline. */ - return this_cpu_read_stable(cpu_current_top_of_stack); + return this_cpu_read_stable(pcpu_hot.top_of_stack); } =20 static __always_inline bool on_thread_stack(void) --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -109,6 +109,8 @@ static void __used common(void) OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); =20 + OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); + if (IS_ENABLED(CONFIG_KVM_INTEL)) { BLANK(); OFFSET(VMX_spec_ctrl, vcpu_vmx, spec_ctrl); --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2003,6 +2003,7 @@ static __init int setup_clearcpuid(char DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, .preempt_count =3D INIT_PREEMPT_COUNT, + .top_of_stack =3D TOP_OF_INIT_STACK, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); =20 @@ -2014,8 +2015,6 @@ EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_d DEFINE_PER_CPU(void *, hardirq_stack_ptr); DEFINE_PER_CPU(bool, hardirq_stack_inuse); =20 -DEFINE_PER_CPU(unsigned long, cpu_current_top_of_stack) =3D TOP_OF_INIT_ST= ACK; - static void wrmsrl_cstar(unsigned long val) { /* @@ -2066,15 +2065,6 @@ void syscall_init(void) =20 #else /* CONFIG_X86_64 */ =20 -/* - * On x86_32, vm86 modifies tss.sp0, so sp0 isn't a reliable way to find - * the top of the kernel stack. Use an extra percpu variable to track the - * top of the kernel stack directly. - */ -DEFINE_PER_CPU(unsigned long, cpu_current_top_of_stack) =3D - (unsigned long)&init_thread_union + THREAD_SIZE; -EXPORT_PER_CPU_SYMBOL(cpu_current_top_of_stack); - #ifdef CONFIG_STACKPROTECTOR DEFINE_PER_CPU(unsigned long, __stack_chk_guard); EXPORT_PER_CPU_SYMBOL(__stack_chk_guard); --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -191,13 +191,13 @@ EXPORT_SYMBOL_GPL(start_thread); arch_end_context_switch(next_p); =20 /* - * Reload esp0 and cpu_current_top_of_stack. This changes + * Reload esp0 and pcpu_hot.top_of_stack. This changes * current_thread_info(). Refresh the SYSENTER configuration in * case prev or next is vm86. */ update_task_stack(next_p); refresh_sysenter_cs(next); - this_cpu_write(cpu_current_top_of_stack, + this_cpu_write(pcpu_hot.top_of_stack, (unsigned long)task_stack_page(next_p) + THREAD_SIZE); =20 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -617,7 +617,7 @@ void compat_start_thread(struct pt_regs * Switch the PDA and FPU contexts. */ raw_cpu_write(pcpu_hot.current_task, next_p); - this_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); + raw_cpu_write(pcpu_hot.top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(); =20 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1056,7 +1056,7 @@ int common_cpu_up(unsigned int cpu, stru =20 #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ - per_cpu(cpu_current_top_of_stack, cpu) =3D task_top_of_stack(idle); + per_cpu(pcpu_hot.top_of_stack, cpu) =3D task_top_of_stack(idle); #else initial_gs =3D per_cpu_offset(cpu); #endif --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -849,7 +849,7 @@ DEFINE_IDTENTRY_RAW(exc_int3) */ asmlinkage __visible noinstr struct pt_regs *sync_regs(struct pt_regs *ere= gs) { - struct pt_regs *regs =3D (struct pt_regs *)this_cpu_read(cpu_current_top_= of_stack) - 1; + struct pt_regs *regs =3D (struct pt_regs *)this_cpu_read(pcpu_hot.top_of_= stack) - 1; if (regs !=3D eregs) *regs =3D *eregs; return regs; @@ -867,7 +867,7 @@ asmlinkage __visible noinstr struct pt_r * trust it and switch to the current kernel stack */ if (ip_within_syscall_gap(regs)) { - sp =3D this_cpu_read(cpu_current_top_of_stack); + sp =3D this_cpu_read(pcpu_hot.top_of_stack); goto sync; } From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 816DAC38145 for ; Fri, 2 Sep 2022 14:28:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237481AbiIBO2g (ORCPT ); Fri, 2 Sep 2022 10:28:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236200AbiIBO1M (ORCPT ); Fri, 2 Sep 2022 10:27:12 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 697BF141A36 for ; Fri, 2 Sep 2022 06:54:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=PFfW8V4/jVOWAwq+y9cqQUX13nMDKmYOZG3YIHdvifE=; b=Ttg6g5Xhu9XRQ3qFnHxWqKZC3T YPDo/4B/2G996i3BKIZSXY64Fy+VTc1Hs0JqsaRqjzDSqiXxbbFb8Gax+ujuqlvkunKeFAbcbhRYx adYq4+gVOEbwvmA+jGvezQ7vC0wTE6qnHi+USA3mlQmDUWtSgxd1ZzRO0LoI4Je31naIWCs5Pw54+ PA0LyO9wWuHXi3x9qYaBSbO2c5eRxSCs21wlBWuv39op2aogHjgoU2yOv0CQzkavExRNMfwvDabZe aM2eHkacnFlYiPONOWk6S92jaQuPNHxAvXV/UCdlWmLMT4ttiHYhdxr/3wKE8/lOYWXvB4d77/rhB /0RCUM1Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8D-DD; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 27D11301D53; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id BC2BB2B8EF7F0; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.060211705@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:51 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 26/59] x86/percpu: Move irq_stack variables next to current_task References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Further extend struct pcpu_hot with the hard and soft irq stack pointers. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/current.h | 6 ++++++ arch/x86/include/asm/irq_stack.h | 12 ++++++------ arch/x86/include/asm/processor.h | 4 ---- arch/x86/kernel/cpu/common.c | 3 --- arch/x86/kernel/dumpstack_32.c | 4 ++-- arch/x86/kernel/dumpstack_64.c | 2 +- arch/x86/kernel/irq_32.c | 13 +++++-------- arch/x86/kernel/irq_64.c | 6 +++--- arch/x86/kernel/process_64.c | 2 +- 9 files changed, 24 insertions(+), 28 deletions(-) --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -18,6 +18,12 @@ struct pcpu_hot { int preempt_count; int cpu_number; unsigned long top_of_stack; + void *hardirq_stack_ptr; +#ifdef CONFIG_X86_64 + bool hardirq_stack_inuse; +#else + void *softirq_stack_ptr; +#endif }; u8 pad[64]; }; --- a/arch/x86/include/asm/irq_stack.h +++ b/arch/x86/include/asm/irq_stack.h @@ -116,7 +116,7 @@ ASM_CALL_ARG2 =20 #define call_on_irqstack(func, asm_call, argconstr...) \ - call_on_stack(__this_cpu_read(hardirq_stack_ptr), \ + call_on_stack(__this_cpu_read(pcpu_hot.hardirq_stack_ptr), \ func, asm_call, argconstr) =20 /* Macros to assert type correctness for run_*_on_irqstack macros */ @@ -135,7 +135,7 @@ * User mode entry and interrupt on the irq stack do not \ * switch stacks. If from user mode the task stack is empty. \ */ \ - if (user_mode(regs) || __this_cpu_read(hardirq_stack_inuse)) { \ + if (user_mode(regs) || __this_cpu_read(pcpu_hot.hardirq_stack_inuse)) { \ irq_enter_rcu(); \ func(c_args); \ irq_exit_rcu(); \ @@ -146,9 +146,9 @@ * places. Invoke the stack switch macro with the call \ * sequence which matches the above direct invocation. \ */ \ - __this_cpu_write(hardirq_stack_inuse, true); \ + __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ call_on_irqstack(func, asm_call, constr); \ - __this_cpu_write(hardirq_stack_inuse, false); \ + __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ } \ } =20 @@ -212,9 +212,9 @@ */ #define do_softirq_own_stack() \ { \ - __this_cpu_write(hardirq_stack_inuse, true); \ + __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ call_on_irqstack(__do_softirq, ASM_CALL_ARG0); \ - __this_cpu_write(hardirq_stack_inuse, false); \ + __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ } =20 #endif --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -448,8 +448,6 @@ static inline unsigned long cpu_kernelmo return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu); } =20 -DECLARE_PER_CPU(void *, hardirq_stack_ptr); -DECLARE_PER_CPU(bool, hardirq_stack_inuse); extern asmlinkage void ignore_sysret(void); =20 /* Save actual FS/GS selectors and bases to current->thread */ @@ -458,8 +456,6 @@ void current_save_fsgs(void); #ifdef CONFIG_STACKPROTECTOR DECLARE_PER_CPU(unsigned long, __stack_chk_guard); #endif -DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr); -DECLARE_PER_CPU(struct irq_stack *, softirq_stack_ptr); #endif /* !X86_64 */ =20 struct perf_event; --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2012,9 +2012,6 @@ DEFINE_PER_CPU_FIRST(struct fixed_percpu fixed_percpu_data) __aligned(PAGE_SIZE) __visible; EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_data); =20 -DEFINE_PER_CPU(void *, hardirq_stack_ptr); -DEFINE_PER_CPU(bool, hardirq_stack_inuse); - static void wrmsrl_cstar(unsigned long val) { /* --- a/arch/x86/kernel/dumpstack_32.c +++ b/arch/x86/kernel/dumpstack_32.c @@ -37,7 +37,7 @@ const char *stack_type_name(enum stack_t =20 static bool in_hardirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr= ); + unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_= stack_ptr); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* @@ -62,7 +62,7 @@ static bool in_hardirq_stack(unsigned lo =20 static bool in_softirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(softirq_stack_ptr= ); + unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.softirq_= stack_ptr); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* --- a/arch/x86/kernel/dumpstack_64.c +++ b/arch/x86/kernel/dumpstack_64.c @@ -134,7 +134,7 @@ static __always_inline bool in_exception =20 static __always_inline bool in_irq_stack(unsigned long *stack, struct stac= k_info *info) { - unsigned long *end =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr); + unsigned long *end =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_st= ack_ptr); unsigned long *begin; =20 /* --- a/arch/x86/kernel/irq_32.c +++ b/arch/x86/kernel/irq_32.c @@ -52,9 +52,6 @@ static inline int check_stack_overflow(v static inline void print_stack_overflow(void) { } #endif =20 -DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr); -DEFINE_PER_CPU(struct irq_stack *, softirq_stack_ptr); - static void call_on_stack(void *func, void *stack) { asm volatile("xchgl %%ebx,%%esp \n" @@ -77,7 +74,7 @@ static inline int execute_on_irq_stack(i u32 *isp, *prev_esp, arg1; =20 curstk =3D (struct irq_stack *) current_stack(); - irqstk =3D __this_cpu_read(hardirq_stack_ptr); + irqstk =3D __this_cpu_read(pcpu_hot.hardirq_stack_ptr); =20 /* * this is where we switch to the IRQ stack. However, if we are @@ -115,7 +112,7 @@ int irq_init_percpu_irqstack(unsigned in int node =3D cpu_to_node(cpu); struct page *ph, *ps; =20 - if (per_cpu(hardirq_stack_ptr, cpu)) + if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) return 0; =20 ph =3D alloc_pages_node(node, THREADINFO_GFP, THREAD_SIZE_ORDER); @@ -127,8 +124,8 @@ int irq_init_percpu_irqstack(unsigned in return -ENOMEM; } =20 - per_cpu(hardirq_stack_ptr, cpu) =3D page_address(ph); - per_cpu(softirq_stack_ptr, cpu) =3D page_address(ps); + per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D page_address(ph); + per_cpu(pcpu_hot.softirq_stack_ptr, cpu) =3D page_address(ps); return 0; } =20 @@ -138,7 +135,7 @@ void do_softirq_own_stack(void) struct irq_stack *irqstk; u32 *isp, *prev_esp; =20 - irqstk =3D __this_cpu_read(softirq_stack_ptr); + irqstk =3D __this_cpu_read(pcpu_hot.softirq_stack_ptr); =20 /* build the stack frame on the softirq stack */ isp =3D (u32 *) ((char *)irqstk + sizeof(*irqstk)); --- a/arch/x86/kernel/irq_64.c +++ b/arch/x86/kernel/irq_64.c @@ -50,7 +50,7 @@ static int map_irq_stack(unsigned int cp return -ENOMEM; =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #else @@ -63,14 +63,14 @@ static int map_irq_stack(unsigned int cp void *va =3D per_cpu_ptr(&irq_stack_backing_store, cpu); =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #endif =20 int irq_init_percpu_irqstack(unsigned int cpu) { - if (per_cpu(hardirq_stack_ptr, cpu)) + if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) return 0; return map_irq_stack(cpu); } --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -562,7 +562,7 @@ void compat_start_thread(struct pt_regs int cpu =3D smp_processor_id(); =20 WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) && - this_cpu_read(hardirq_stack_inuse)); + this_cpu_read(pcpu_hot.hardirq_stack_inuse)); =20 if (!test_thread_flag(TIF_NEED_FPU_LOAD)) switch_fpu_prepare(prev_fpu, cpu); From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0EB9C38145 for ; Fri, 2 Sep 2022 14:28:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236045AbiIBO26 (ORCPT ); Fri, 2 Sep 2022 10:28:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236470AbiIBO11 (ORCPT ); Fri, 2 Sep 2022 10:27:27 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 198DF145C40 for ; Fri, 2 Sep 2022 06:54:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=3pHERFsmuK61/aX42Un1sPpA4i778m0NUaKcblu/jsE=; b=MiQrGmnSSmUN+m3AY8XVrIIMxA Ij4T3rSxtjKMDZBHu+f20DFMXl61hxKfBVmxDXHQ1yqV0hvL6YL6tM9eHnGYvW4S564sPN5gD0K5+ HGfmoRt7l9LXJx8RN+LpCVbQbAfDOCKoT68U2FP7lpcJfzANHAjiHhRO46G4IXIM9qqC1R8jyNV5b 799JT+RF1RFpU+ZU6gCUc0Dj2jgrhsqzMZASKaipMxT+Zufr8oeI3oVBtyYsjldqBZ+rcdZTU5icv lcbC/ePpgr2rY7I8DXLE0ozcUhWTYwrTOr8JM6xrU3CJvSdApR5wpFS1UuBsjW0yOY7tDYIG2n2+U uNySeT+w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TK-KB; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 26CE7301D1E; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C4A3F2B8EF7F1; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.166343269@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:52 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 27/59] x86/softirq: Move softirq pending next to current task References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Another hot variable which is strict per CPU and benefits from being in the same cache line. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/current.h | 1 + arch/x86/include/asm/hardirq.h | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -19,6 +19,7 @@ struct pcpu_hot { int cpu_number; unsigned long top_of_stack; void *hardirq_stack_ptr; + u16 softirq_pending; #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -3,9 +3,9 @@ #define _ASM_X86_HARDIRQ_H =20 #include +#include =20 typedef struct { - u16 __softirq_pending; #if IS_ENABLED(CONFIG_KVM_INTEL) u8 kvm_cpu_l1tf_flush_l1d; #endif @@ -60,6 +60,7 @@ extern u64 arch_irq_stat_cpu(unsigned in extern u64 arch_irq_stat(void); #define arch_irq_stat arch_irq_stat =20 +#define local_softirq_pending_ref pcpu_hot.softirq_pending =20 #if IS_ENABLED(CONFIG_KVM_INTEL) static inline void kvm_set_cpu_l1tf_flush_l1d(void) From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EF27C38145 for ; Fri, 2 Sep 2022 14:34:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236728AbiIBOdc (ORCPT ); Fri, 2 Sep 2022 10:33:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236977AbiIBOas (ORCPT ); Fri, 2 Sep 2022 10:30:48 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3288714D3F for ; Fri, 2 Sep 2022 06:54:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=VtHSfWHBCM/dkewK26y+PcMV0oq6+DiYrWTtkB689iQ=; b=Ycygkh2jN7m1q17HxTbE+bSJD5 d0nMCb3gQG4Kb408oFfp3kF6NGqMcXaxIcF7E4S7cCmHsKzost2zxuq0AH/KyNfDAiCneJi+lo2Ul 9L9U6/qs0SPPQMsv0/HoACJF3CXUPEh/c1MeWPzCI3ztxPNUuorOk4YuHI1KP4/SauR14o4aRxKX/ Ge8NTyhbMvjCI0Vp1Ovvx8VmkgKWCwyrOp1hzmh6D1vyUBxpibD41WaM2XtesOqveimS+LNpAfBOU nQuq2w6W6Rv57Bs4wMedkujSY9+3PVVeSSPeTCFCEVBlOvEJIauA54/dMgEbnW1b2aPmP5RNzHUHy IFnW7D8A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TJ-JZ; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 26CD9301D17; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id CD4CB2B8EF7F2; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.268856815@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:53 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 28/59] objtool: Allow !PC relative relocations References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Objtool doesn't currently much like per-cpu usage in alternatives: arch/x86/entry/entry_64.o: warning: objtool: .altinstr_replacement+0xf: uns= upported relocation in alternatives section f: 65 c7 04 25 00 00 00 00 00 00 00 80 movl $0x80000000,%gs:0x0 = 13: R_X86_64_32S __x86_call_depth Since the R_X86_64_32S relocation is location invariant (it's computation doesn't include P - the address of the location itself), it can be trivially allowed. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- tools/objtool/arch/x86/decode.c | 42 +++++++++++++++++++++++++++++= ----- tools/objtool/check.c | 6 +--- tools/objtool/include/objtool/arch.h | 6 ++-- tools/objtool/include/objtool/check.h | 17 +++++++------ 4 files changed, 51 insertions(+), 20 deletions(-) --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -73,6 +73,30 @@ unsigned long arch_jump_destination(stru return insn->offset + insn->len + insn->immediate; } =20 +bool arch_pc_relative_reloc(struct reloc *reloc) +{ + /* + * All relocation types where P (the address of the target) + * is included in the computation. + */ + switch (reloc->type) { + case R_X86_64_PC8: + case R_X86_64_PC16: + case R_X86_64_PC32: + case R_X86_64_PC64: + + case R_X86_64_PLT32: + case R_X86_64_GOTPC32: + case R_X86_64_GOTPCREL: + return true; + + default: + break; + } + + return false; +} + #define ADD_OP(op) \ if (!(op =3D calloc(1, sizeof(*op)))) \ return -1; \ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1622,7 +1620,7 @@ static int handle_group_alt(struct objto * accordingly. */ alt_reloc =3D insn_reloc(file, insn); - if (alt_reloc && + if (alt_reloc && arch_pc_relative_reloc(alt_reloc) && !arch_support_alt_relocation(special_alt, insn, alt_reloc)) { =20 WARN_FUNC("unsupported relocation in alternatives section", --- a/tools/objtool/include/objtool/arch.h +++ b/tools/objtool/include/objtool/arch.h @@ -93,4 +91,6 @@ bool arch_is_rethunk(struct symbol *sym) =20 int arch_rewrite_retpolines(struct objtool_file *file); =20 +bool arch_pc_relative_reloc(struct reloc *reloc); + #endif /* _ARCH_H */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E980EC38145 for ; Fri, 2 Sep 2022 14:30:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236676AbiIBOaS (ORCPT ); Fri, 2 Sep 2022 10:30:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235936AbiIBO1f (ORCPT ); Fri, 2 Sep 2022 10:27:35 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17EA8168A07 for ; Fri, 2 Sep 2022 06:54:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=KetPcUlF1b8CoRduWYRVJ1Z0OAFAjWB3tdLqyvUvF+o=; b=Q+LN1KJ5tTML0ppwIjo9awpJFF YTRMCTltV1xg0UBdjueEEpetKChLxj719aEN149U5fjpOJ0u6K0iRj9SHZrDIq+N9Sc/mnJfs8A2K LG+UGD/IUcsVZ6boMmQxYRy+wzm76RZm0UlZJYqCz+eRWcXtXJVLe2MGuUHSDyDUwA9F2oEkHSNLH D18MJfCo8IFZ4Njbxz8h6+7UEzQHr/qNDQh+KiZEKeUl0QZ503ZyRuA7g8wxTGmHwjBb1FOOwBSIM 2RByOfoO5Jp9VcsU4cylaaNE66DPHj06TezT/rTjFyo5Oli01hfPBrDCxKYSuUaQSBfyDYflPwWzB xivfqgxg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8C-BV; Fri, 02 Sep 2022 13:53:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 26CC4301D06; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D5C952B8EF7F3; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.370354037@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:54 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 29/59] objtool: Track init section References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra For future usage of .init.text exclusion track the init section in the instruction decoder and use the result in retpoline validation. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- tools/objtool/check.c | 17 ++++++++++------- tools/objtool/include/objtool/elf.h | 2 +- 2 files changed, 11 insertions(+), 8 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -380,6 +380,15 @@ static int decode_instructions(struct ob !strncmp(sec->name, ".text.__x86.", 12)) sec->noinstr =3D true; =20 + /* + * .init.text code is ran before userspace and thus doesn't + * strictly need retpolines, except for modules which are + * loaded late, they very much do need retpoline in their + * .init.text + */ + if (!strcmp(sec->name, ".init.text") && !opts.module) + sec->init =3D true; + for (offset =3D 0; offset < sec->sh.sh_size; offset +=3D insn->len) { insn =3D malloc(sizeof(*insn)); if (!insn) { @@ -3720,13 +3729,7 @@ static int validate_retpoline(struct obj if (insn->retpoline_safe) continue; =20 - /* - * .init.text code is ran before userspace and thus doesn't - * strictly need retpolines, except for modules which are - * loaded late, they very much do need retpoline in their - * .init.text - */ - if (!strcmp(insn->sec->name, ".init.text") && !opts.module) + if (insn->sec->init) continue; =20 if (insn->type =3D=3D INSN_RETURN) { --- a/tools/objtool/include/objtool/elf.h +++ b/tools/objtool/include/objtool/elf.h @@ -38,7 +38,7 @@ struct section { Elf_Data *data; char *name; int idx; - bool changed, text, rodata, noinstr; + bool changed, text, rodata, noinstr, init; }; =20 struct symbol { From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB5CBC38145 for ; Fri, 2 Sep 2022 14:32:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235127AbiIBOcH (ORCPT ); Fri, 2 Sep 2022 10:32:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236769AbiIBO2h (ORCPT ); Fri, 2 Sep 2022 10:28:37 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 188AC12A90 for ; Fri, 2 Sep 2022 06:54:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=cbOcmEbLj8D337GsZMcKWuXa0J6c/XUPtmKz8eI6AKk=; b=SPWRrj8JBuaPvEBBOzPGyJJHzP bIWxJq4UMb5Xz0Cd/fYW6xefBzhvVkH2bsHeDV23PsgdXC41IshWbQeW3KhB7NjRmPDaaCZ9ZyjwF VSVaLVPJF+5tv4CepoI+hztTyCos/R2gsgkarQcHUni10RGhCpVlkrAwxtrZEQaHs/qPAElvNRwef wekAsDqlWXDzfajbNzwLmsssXo9rMj6m6KQiUNBHrTELDehG6QoQB8nrrM5wK0tA4QDGoJ8UB3byX E5N9YbQ7jykfiFokjihHpJlgxUwWe3I+dGHIqeWCAaX4DTugot06iLqPEGo6ymivCHxH7JJDv3g6s t/7GEp8A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8E-E8; Fri, 02 Sep 2022 13:54:08 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 28DB1301D8A; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id DE80D2B8EF7F4; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.474319273@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:55 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 30/59] objtool: Add .call_sites section References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra In preparation for call depth tracking provide a section which collects all direct calls. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/vmlinux.lds.S | 7 ++++ tools/objtool/check.c | 51 +++++++++++++++++++++++++++= +++++ tools/objtool/include/objtool/objtool.h | 1=20 tools/objtool/objtool.c | 1=20 4 files changed, 60 insertions(+) --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -291,6 +291,13 @@ SECTIONS *(.return_sites) __return_sites_end =3D .; } + + . =3D ALIGN(8); + .call_sites : AT(ADDR(.call_sites) - LOAD_OFFSET) { + __call_sites =3D .; + *(.call_sites) + __call_sites_end =3D .; + } #endif =20 #ifdef CONFIG_X86_KERNEL_IBT --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -898,6 +898,49 @@ static int create_mcount_loc_sections(st return 0; } =20 +static int create_direct_call_sections(struct objtool_file *file) +{ + struct instruction *insn; + struct section *sec; + unsigned int *loc; + int idx; + + sec =3D find_section_by_name(file->elf, ".call_sites"); + if (sec) { + INIT_LIST_HEAD(&file->call_list); + WARN("file already has .call_sites section, skipping"); + return 0; + } + + if (list_empty(&file->call_list)) + return 0; + + idx =3D 0; + list_for_each_entry(insn, &file->call_list, call_node) + idx++; + + sec =3D elf_create_section(file->elf, ".call_sites", 0, sizeof(unsigned i= nt), idx); + if (!sec) + return -1; + + idx =3D 0; + list_for_each_entry(insn, &file->call_list, call_node) { + + loc =3D (unsigned int *)sec->data->d_buf + idx; + memset(loc, 0, sizeof(unsigned int)); + + if (elf_add_reloc_to_insn(file->elf, sec, + idx * sizeof(unsigned int), + R_X86_64_PC32, + insn->sec, insn->offset)) + return -1; + + idx++; + } + + return 0; +} + /* * Warnings shouldn't be reported for ignored functions. */ @@ -1252,6 +1295,9 @@ static void annotate_call_site(struct ob return; } =20 + if (insn->type =3D=3D INSN_CALL && !insn->sec->init) + list_add_tail(&insn->call_node, &file->call_list); + if (!sibling && dead_end_function(file, sym)) insn->dead_end =3D true; } @@ -4274,6 +4320,11 @@ int check(struct objtool_file *file) if (ret < 0) goto out; warnings +=3D ret; + + ret =3D create_direct_call_sections(file); + if (ret < 0) + goto out; + warnings +=3D ret; } =20 if (opts.mcount) { --- a/tools/objtool/include/objtool/objtool.h +++ b/tools/objtool/include/objtool/objtool.h @@ -28,6 +28,7 @@ struct objtool_file { struct list_head static_call_list; struct list_head mcount_loc_list; struct list_head endbr_list; + struct list_head call_list; bool ignore_unreachables, hints, rodata; =20 unsigned int nr_endbr; --- a/tools/objtool/objtool.c +++ b/tools/objtool/objtool.c @@ -106,6 +106,7 @@ struct objtool_file *objtool_open_read(c INIT_LIST_HEAD(&file.static_call_list); INIT_LIST_HEAD(&file.mcount_loc_list); INIT_LIST_HEAD(&file.endbr_list); + INIT_LIST_HEAD(&file.call_list); file.ignore_unreachables =3D opts.no_unreachable; file.hints =3D false; From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B3E7ECAAD5 for ; Fri, 2 Sep 2022 14:33:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236472AbiIBOdI (ORCPT ); Fri, 2 Sep 2022 10:33:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236524AbiIBO3E (ORCPT ); Fri, 2 Sep 2022 10:29:04 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A11FC317 for ; Fri, 2 Sep 2022 06:54:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ZlEOD9v5avy/olHFrLm+o6b9KaDGaNLliYExb9rswpk=; b=EMfm5CDrtFkDFSIExK9du3Jafl bMrBWq4qp9d9Ilzt1Rqk3UpAOfCGeh39tTN+VeboGd3m6aAWJEhuDEKVn/jCYGRqpx3DvCzUACoXB FjHn1VgpkwNkgOAvTxfmwwdyY6hZ2cG9R8Y9yf2Cwv+0jdvu1l2+wYKmPS7sX8sNn2y1MX11gbJp8 Tu8OJKPlIfjdWtUV63l34IzM7Um0WAfAM2A8G1FinYrtG5x5k+mFuUjudZcocue4FIfXzmLtBqTn/ zahLWnKrNeltNNWQbEuySciUxfDsCDd9mF6A5kLxRSCIdYSeqYh4DwZcPNpT2AOOc6WtXd0q8jfTI DWljinmQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TO-NU; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2A2CF301FC4; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id E7B6A2B8EF7F5; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.578815518@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:56 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 31/59] objtool: Add --hacks=skylake References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Make the call/func sections selectable via the --hacks option. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- scripts/Makefile.lib | 3 ++- tools/objtool/builtin-check.c | 7 ++++++- tools/objtool/check.c | 10 ++++++---- tools/objtool/include/objtool/builtin.h | 1 + 4 files changed, 15 insertions(+), 6 deletions(-) --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -231,7 +231,8 @@ objtool :=3D $(objtree)/tools/objtool/objt =20 objtool_args =3D \ $(if $(CONFIG_HAVE_JUMP_LABEL_HACK), --hacks=3Djump_label) \ - $(if $(CONFIG_HAVE_NOINSTR_HACK), --hacks=3Dnoinstr) \ + $(if $(CONFIG_HAVE_NOINSTR_HACK), --hacks=3Dnoinstr) \ + $(if $(CONFIG_CALL_DEPTH_TRACKING), --hacks=3Dskylake) \ $(if $(CONFIG_X86_KERNEL_IBT), --ibt) \ $(if $(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL), --mcount) \ $(if $(CONFIG_UNWINDER_ORC), --orc) \ --- a/tools/objtool/builtin-check.c +++ b/tools/objtool/builtin-check.c @@ -57,12 +57,17 @@ static int parse_hacks(const struct opti found =3D true; } =20 + if (!str || strstr(str, "skylake")) { + opts.hack_skylake =3D true; + found =3D true; + } + return found ? 0 : -1; } =20 const struct option check_options[] =3D { OPT_GROUP("Actions:"), - OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr", "patc= h toolchain bugs/limitations", parse_hacks), + OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake= ", "patch toolchain bugs/limitations", parse_hacks), OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"), OPT_BOOLEAN('m', "mcount", &opts.mcount, "annotate mcount/fentry calls fo= r ftrace"), OPT_BOOLEAN('n', "noinstr", &opts.noinstr, "validate noinstr rules"), --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -4321,10 +4321,12 @@ int check(struct objtool_file *file) goto out; warnings +=3D ret; =20 - ret =3D create_direct_call_sections(file); - if (ret < 0) - goto out; - warnings +=3D ret; + if (opts.hack_skylake) { + ret =3D create_direct_call_sections(file); + if (ret < 0) + goto out; + warnings +=3D ret; + } } =20 if (opts.mcount) { --- a/tools/objtool/include/objtool/builtin.h +++ b/tools/objtool/include/objtool/builtin.h @@ -14,6 +14,7 @@ struct opts { bool dump_orc; bool hack_jump_label; bool hack_noinstr; + bool hack_skylake; bool ibt; bool mcount; bool noinstr; From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12933C38145 for ; Fri, 2 Sep 2022 14:33:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236374AbiIBOdL (ORCPT ); Fri, 2 Sep 2022 10:33:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236920AbiIBO3J (ORCPT ); Fri, 2 Sep 2022 10:29:09 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48EFF2648 for ; Fri, 2 Sep 2022 06:54:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=zQlKGQHBFgjVvjoX8AWmJKxWpsURbcGwMojDrLJkF48=; b=ALJrRxb1oOAv3oR6l6jjujyd61 RAneOzfB60U91aF/P5PkjNPeCQCUT62upDitSH8L65BG9joWG0VhBm9GC812c05U9QUB6Hm56IOsA abeowS4cmR8B1nb7ZHIwwZdebJoGMhCU9TkBwspMjrv70IDKVqXJQMIYJo1a9KYJlDxKoOWlDsFGB LqjGkN9FrH/K4TTEJjcP+8wbtd1MtGqYKfpkAqGYeETk/CNvcizLoCp1Uq4OjLKANE2Wr2uJs5Mu1 xF0w1x1Y/1RBi2P4x9SbrBcGimAfwqgXWb6xElAQuE/A/7JbDtPJbb9GT3EM8KQdh+bDep7ple0oC c5a2yGCA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TN-MK; Fri, 02 Sep 2022 13:53:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2BC90302188; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id ED9EA2B8EF7F6; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.683665538@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:57 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 32/59] objtool: Allow STT_NOTYPE -> STT_FUNC+0 tail-calls References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Allow STT_NOTYPE to tail-call into STT_FUNC, per definition STT_NOTYPE is not a sub-function of the STT_FUNC. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- tools/objtool/check.c | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1370,6 +1370,16 @@ static void add_return_call(struct objto =20 static bool same_function(struct instruction *insn1, struct instruction *i= nsn2) { + if (!insn1->func && !insn2->func) + return true; + + /* Allow STT_NOTYPE -> STT_FUNC+0 tail-calls */ + if (!insn1->func && insn1->func !=3D insn2->func) + return false; + + if (!insn2->func) + return true; + return insn1->func->pfunc =3D=3D insn2->func->pfunc; } =20 @@ -1487,18 +1497,19 @@ static int add_jump_destinations(struct strstr(jump_dest->func->name, ".cold")) { insn->func->cfunc =3D jump_dest->func; jump_dest->func->pfunc =3D insn->func; - - } else if (!same_function(insn, jump_dest) && - is_first_func_insn(file, jump_dest)) { - /* - * Internal sibling call without reloc or with - * STT_SECTION reloc. - */ - add_call_dest(file, insn, jump_dest->func, true); - continue; } } =20 + if (!same_function(insn, jump_dest) && + is_first_func_insn(file, jump_dest)) { + /* + * Internal sibling call without reloc or with + * STT_SECTION reloc. + */ + add_call_dest(file, insn, jump_dest->func, true); + continue; + } + insn->jump_dest =3D jump_dest; } From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFC7CC38145 for ; Fri, 2 Sep 2022 14:31:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236545AbiIBObF (ORCPT ); Fri, 2 Sep 2022 10:31:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237362AbiIBO1j (ORCPT ); Fri, 2 Sep 2022 10:27:39 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C234315819D for ; Fri, 2 Sep 2022 06:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=mx58zQy98Wj+VFeO7kNG/+7nWSkg+eR2wmG7+5WoSrk=; b=NaNh3R/iwlKYiAWSqd7sSxJQkV UyvtNlhsdu8uUq+Z5CRjn88AROZ4+nBTZG4LJrKteU2OkXnZ3cqdb4A0D+OseLZSr31X2597PhrOR 0WFi8bfK+o8cNpVX2PPin1WtDWyNBSxovkHW05bjqCVsDRmifhnQoehDPV2dLMWUd1CzTLmsDWb46 F36FCkbJ03VbxFjRdofEkZ9PvRdAYTVvzFt0byhZ5K5bDyhtDqdu7gzTAWnMPg3z2NXCUUbvfuPgY 74OWWyB2Zy0FKFR8vOojteSYGNsmbu4in3biSK/2zn1tYQ7AtM/0LuygpkVRIq6Y3QmMBb9sDD5S2 E7qd/g2g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8H-FS; Fri, 02 Sep 2022 13:54:03 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2D3E13021FB; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id F33D92B8AE5A4; Fri, 2 Sep 2022 15:53:52 +0200 (CEST) Message-ID: <20220902130949.789826745@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:58 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 33/59] objtool: Fix find_{symbol,func}_containing() References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra The current find_{symbol,func}_containing() functions are broken in the face of overlapping symbols, exactly the case that is needed for a new ibt/endbr supression. Import interval_tree_generic.h into the tools tree and convert the symbol tree to an interval tree to support proper range stabs. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- tools/include/linux/interval_tree_generic.h | 187 +++++++++++++++++++++++= +++++ tools/objtool/elf.c | 93 +++++-------- tools/objtool/include/objtool/elf.h | 3=20 3 files changed, 229 insertions(+), 54 deletions(-) --- /dev/null +++ b/tools/include/linux/interval_tree_generic.h @@ -0,0 +1,187 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + Interval Trees + (C) 2012 Michel Lespinasse + + + include/linux/interval_tree_generic.h +*/ + +#include + +/* + * Template for implementing interval trees + * + * ITSTRUCT: struct type of the interval tree nodes + * ITRB: name of struct rb_node field within ITSTRUCT + * ITTYPE: type of the interval endpoints + * ITSUBTREE: name of ITTYPE field within ITSTRUCT holding last-in-subtree + * ITSTART(n): start endpoint of ITSTRUCT node n + * ITLAST(n): last endpoint of ITSTRUCT node n + * ITSTATIC: 'static' or empty + * ITPREFIX: prefix to use for the inline tree definitions + * + * Note - before using this, please consider if generic version + * (interval_tree.h) would work for you... + */ + +#define INTERVAL_TREE_DEFINE(ITSTRUCT, ITRB, ITTYPE, ITSUBTREE, \ + ITSTART, ITLAST, ITSTATIC, ITPREFIX) \ + \ +/* Callbacks for augmented rbtree insert and remove */ \ + \ +RB_DECLARE_CALLBACKS_MAX(static, ITPREFIX ## _augment, \ + ITSTRUCT, ITRB, ITTYPE, ITSUBTREE, ITLAST) \ + \ +/* Insert / remove interval nodes from the tree */ \ + \ +ITSTATIC void ITPREFIX ## _insert(ITSTRUCT *node, \ + struct rb_root_cached *root) \ +{ \ + struct rb_node **link =3D &root->rb_root.rb_node, *rb_parent =3D NULL; = \ + ITTYPE start =3D ITSTART(node), last =3D ITLAST(node); \ + ITSTRUCT *parent; \ + bool leftmost =3D true; \ + \ + while (*link) { \ + rb_parent =3D *link; \ + parent =3D rb_entry(rb_parent, ITSTRUCT, ITRB); \ + if (parent->ITSUBTREE < last) \ + parent->ITSUBTREE =3D last; \ + if (start < ITSTART(parent)) \ + link =3D &parent->ITRB.rb_left; \ + else { \ + link =3D &parent->ITRB.rb_right; \ + leftmost =3D false; \ + } \ + } \ + \ + node->ITSUBTREE =3D last; \ + rb_link_node(&node->ITRB, rb_parent, link); \ + rb_insert_augmented_cached(&node->ITRB, root, \ + leftmost, &ITPREFIX ## _augment); \ +} \ + \ +ITSTATIC void ITPREFIX ## _remove(ITSTRUCT *node, \ + struct rb_root_cached *root) \ +{ \ + rb_erase_augmented_cached(&node->ITRB, root, &ITPREFIX ## _augment); \ +} \ + \ +/* \ + * Iterate over intervals intersecting [start;last] \ + * \ + * Note that a node's interval intersects [start;last] iff: \ + * Cond1: ITSTART(node) <=3D last \ + * and \ + * Cond2: start <=3D ITLAST(node) \ + */ \ + \ +static ITSTRUCT * \ +ITPREFIX ## _subtree_search(ITSTRUCT *node, ITTYPE start, ITTYPE last) = \ +{ \ + while (true) { \ + /* \ + * Loop invariant: start <=3D node->ITSUBTREE \ + * (Cond2 is satisfied by one of the subtree nodes) \ + */ \ + if (node->ITRB.rb_left) { \ + ITSTRUCT *left =3D rb_entry(node->ITRB.rb_left, \ + ITSTRUCT, ITRB); \ + if (start <=3D left->ITSUBTREE) { \ + /* \ + * Some nodes in left subtree satisfy Cond2. \ + * Iterate to find the leftmost such node N. \ + * If it also satisfies Cond1, that's the \ + * match we are looking for. Otherwise, there \ + * is no matching interval as nodes to the \ + * right of N can't satisfy Cond1 either. \ + */ \ + node =3D left; \ + continue; \ + } \ + } \ + if (ITSTART(node) <=3D last) { /* Cond1 */ \ + if (start <=3D ITLAST(node)) /* Cond2 */ \ + return node; /* node is leftmost match */ \ + if (node->ITRB.rb_right) { \ + node =3D rb_entry(node->ITRB.rb_right, \ + ITSTRUCT, ITRB); \ + if (start <=3D node->ITSUBTREE) \ + continue; \ + } \ + } \ + return NULL; /* No match */ \ + } \ +} \ + \ +ITSTATIC ITSTRUCT * \ +ITPREFIX ## _iter_first(struct rb_root_cached *root, \ + ITTYPE start, ITTYPE last) \ +{ \ + ITSTRUCT *node, *leftmost; \ + \ + if (!root->rb_root.rb_node) \ + return NULL; \ + \ + /* \ + * Fastpath range intersection/overlap between A: [a0, a1] and \ + * B: [b0, b1] is given by: \ + * \ + * a0 <=3D b1 && b0 <=3D a1 \ + * \ + * ... where A holds the lock range and B holds the smallest \ + * 'start' and largest 'last' in the tree. For the later, we \ + * rely on the root node, which by augmented interval tree \ + * property, holds the largest value in its last-in-subtree. \ + * This allows mitigating some of the tree walk overhead for \ + * for non-intersecting ranges, maintained and consulted in O(1). \ + */ \ + node =3D rb_entry(root->rb_root.rb_node, ITSTRUCT, ITRB); \ + if (node->ITSUBTREE < start) \ + return NULL; \ + \ + leftmost =3D rb_entry(root->rb_leftmost, ITSTRUCT, ITRB); \ + if (ITSTART(leftmost) > last) \ + return NULL; \ + \ + return ITPREFIX ## _subtree_search(node, start, last); \ +} \ + \ +ITSTATIC ITSTRUCT * \ +ITPREFIX ## _iter_next(ITSTRUCT *node, ITTYPE start, ITTYPE last) \ +{ \ + struct rb_node *rb =3D node->ITRB.rb_right, *prev; \ + \ + while (true) { \ + /* \ + * Loop invariants: \ + * Cond1: ITSTART(node) <=3D last \ + * rb =3D=3D node->ITRB.rb_right \ + * \ + * First, search right subtree if suitable \ + */ \ + if (rb) { \ + ITSTRUCT *right =3D rb_entry(rb, ITSTRUCT, ITRB); \ + if (start <=3D right->ITSUBTREE) \ + return ITPREFIX ## _subtree_search(right, \ + start, last); \ + } \ + \ + /* Move up the tree until we come from a node's left child */ \ + do { \ + rb =3D rb_parent(&node->ITRB); \ + if (!rb) \ + return NULL; \ + prev =3D &node->ITRB; \ + node =3D rb_entry(rb, ITSTRUCT, ITRB); \ + rb =3D node->ITRB.rb_right; \ + } while (prev =3D=3D rb); \ + \ + /* Check if the node intersects [start;last] */ \ + if (last < ITSTART(node)) /* !Cond1 */ \ + return NULL; \ + else if (start <=3D ITLAST(node)) /* Cond2 */ \ + return node; \ + } \ +} --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -16,6 +16,7 @@ #include #include #include +#include #include =20 #include @@ -50,38 +51,22 @@ static inline u32 str_hash(const char *s __elf_table(name); \ }) =20 -static bool symbol_to_offset(struct rb_node *a, const struct rb_node *b) +static inline unsigned long __sym_start(struct symbol *s) { - struct symbol *sa =3D rb_entry(a, struct symbol, node); - struct symbol *sb =3D rb_entry(b, struct symbol, node); - - if (sa->offset < sb->offset) - return true; - if (sa->offset > sb->offset) - return false; - - if (sa->len < sb->len) - return true; - if (sa->len > sb->len) - return false; - - sa->alias =3D sb; - - return false; + return s->offset; } =20 -static int symbol_by_offset(const void *key, const struct rb_node *node) +static inline unsigned long __sym_last(struct symbol *s) { - const struct symbol *s =3D rb_entry(node, struct symbol, node); - const unsigned long *o =3D key; + return s->offset + s->len - 1; +} =20 - if (*o < s->offset) - return -1; - if (*o >=3D s->offset + s->len) - return 1; +INTERVAL_TREE_DEFINE(struct symbol, node, unsigned long, __subtree_last, + __sym_start, __sym_last, static, __sym) =20 - return 0; -} +#define __sym_for_each(_iter, _tree, _start, _end) \ + for (_iter =3D __sym_iter_first((_tree), (_start), (_end)); \ + _iter; _iter =3D __sym_iter_next(_iter, (_start), (_end))) =20 struct symbol_hole { unsigned long key; @@ -147,13 +132,12 @@ static struct symbol *find_symbol_by_ind =20 struct symbol *find_symbol_by_offset(struct section *sec, unsigned long of= fset) { - struct rb_node *node; - - rb_for_each(node, &offset, &sec->symbol_tree, symbol_by_offset) { - struct symbol *s =3D rb_entry(node, struct symbol, node); + struct rb_root_cached *tree =3D (struct rb_root_cached *)&sec->symbol_tre= e; + struct symbol *iter; =20 - if (s->offset =3D=3D offset && s->type !=3D STT_SECTION) - return s; + __sym_for_each(iter, tree, offset, offset) { + if (iter->offset =3D=3D offset && iter->type !=3D STT_SECTION) + return iter; } =20 return NULL; @@ -161,13 +145,12 @@ struct symbol *find_symbol_by_offset(str =20 struct symbol *find_func_by_offset(struct section *sec, unsigned long offs= et) { - struct rb_node *node; + struct rb_root_cached *tree =3D (struct rb_root_cached *)&sec->symbol_tre= e; + struct symbol *iter; =20 - rb_for_each(node, &offset, &sec->symbol_tree, symbol_by_offset) { - struct symbol *s =3D rb_entry(node, struct symbol, node); - - if (s->offset =3D=3D offset && s->type =3D=3D STT_FUNC) - return s; + __sym_for_each(iter, tree, offset, offset) { + if (iter->offset =3D=3D offset && iter->type =3D=3D STT_FUNC) + return iter; } =20 return NULL; @@ -175,13 +158,12 @@ struct symbol *find_func_by_offset(struc =20 struct symbol *find_symbol_containing(const struct section *sec, unsigned = long offset) { - struct rb_node *node; - - rb_for_each(node, &offset, &sec->symbol_tree, symbol_by_offset) { - struct symbol *s =3D rb_entry(node, struct symbol, node); + struct rb_root_cached *tree =3D (struct rb_root_cached *)&sec->symbol_tre= e; + struct symbol *iter; =20 - if (s->type !=3D STT_SECTION) - return s; + __sym_for_each(iter, tree, offset, offset) { + if (iter->type !=3D STT_SECTION) + return iter; } =20 return NULL; @@ -202,7 +184,7 @@ int find_symbol_hole_containing(const st /* * Find the rightmost symbol for which @offset is after it. */ - n =3D rb_find(&hole, &sec->symbol_tree, symbol_hole_by_offset); + n =3D rb_find(&hole, &sec->symbol_tree.rb_root, symbol_hole_by_offset); =20 /* found a symbol that contains @offset */ if (n) @@ -224,13 +206,12 @@ int find_symbol_hole_containing(const st =20 struct symbol *find_func_containing(struct section *sec, unsigned long off= set) { - struct rb_node *node; - - rb_for_each(node, &offset, &sec->symbol_tree, symbol_by_offset) { - struct symbol *s =3D rb_entry(node, struct symbol, node); + struct rb_root_cached *tree =3D (struct rb_root_cached *)&sec->symbol_tre= e; + struct symbol *iter; =20 - if (s->type =3D=3D STT_FUNC) - return s; + __sym_for_each(iter, tree, offset, offset) { + if (iter->type =3D=3D STT_FUNC) + return iter; } =20 return NULL; @@ -373,6 +354,7 @@ static void elf_add_symbol(struct elf *e { struct list_head *entry; struct rb_node *pnode; + struct symbol *iter; =20 INIT_LIST_HEAD(&sym->pv_target); sym->alias =3D sym; @@ -386,7 +368,12 @@ static void elf_add_symbol(struct elf *e sym->offset =3D sym->sym.st_value; sym->len =3D sym->sym.st_size; =20 - rb_add(&sym->node, &sym->sec->symbol_tree, symbol_to_offset); + __sym_for_each(iter, &sym->sec->symbol_tree, sym->offset, sym->offset) { + if (iter->offset =3D=3D sym->offset && iter->type =3D=3D sym->type) + iter->alias =3D sym; + } + + __sym_insert(sym, &sym->sec->symbol_tree); pnode =3D rb_prev(&sym->node); if (pnode) entry =3D &rb_entry(pnode, struct symbol, node)->list; @@ -401,7 +388,7 @@ static void elf_add_symbol(struct elf *e * can exist within a function, confusing the sorting. */ if (!sym->len) - rb_erase(&sym->node, &sym->sec->symbol_tree); + __sym_remove(sym, &sym->sec->symbol_tree); } =20 static int read_symbols(struct elf *elf) --- a/tools/objtool/include/objtool/elf.h +++ b/tools/objtool/include/objtool/elf.h @@ -30,7 +30,7 @@ struct section { struct hlist_node hash; struct hlist_node name_hash; GElf_Shdr sh; - struct rb_root symbol_tree; + struct rb_root_cached symbol_tree; struct list_head symbol_list; struct list_head reloc_list; struct section *base, *reloc; @@ -53,6 +53,7 @@ struct symbol { unsigned char bind, type; unsigned long offset; unsigned int len; + unsigned long __subtree_last; struct symbol *pfunc, *cfunc, *alias; u8 uaccess_safe : 1; u8 static_call_tramp : 1; From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36138ECAAD5 for ; Fri, 2 Sep 2022 14:33:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236706AbiIBOdV (ORCPT ); Fri, 2 Sep 2022 10:33:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236366AbiIBO3a (ORCPT ); Fri, 2 Sep 2022 10:29:30 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 485BB10B7 for ; Fri, 2 Sep 2022 06:54:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=9OeUQ8WKvjmDCQa64vw6vBkwEukGFdObNeGXaylenLQ=; b=McjR//5MoPFNSEkoQkl0Uhh4wU TT57WIwusLJBg2g3/TZ3ToxGWMcoQ3bkV50EUoTjEXQO4XQMjZopdCyNncRieMMB6sC885BfEntSB uPxD+bNYNeQWcHgUHvUlbd234CQnCdNBonoTnMrnH+qrHVCwFEavPg2vydzc8D/uHn6DVc6fts8ys w68n/h6yvneGAuO5zdgDnDIxNshWnkizHMZIJGPAf4qT32W2csJID8N+5ijWR33zm6ZIwQ48KaWQk LmM1ddXc+XaH6dEsHuCY1jmPNFg2ocvgP5a/Y8xXiAB1fkhEwqkGs1PGsS5oktyLkcbVQZKMw29/H yXTfPSSA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8J-Su; Fri, 02 Sep 2022 13:54:09 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2FC0E30220F; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 0492B2B8EF7F7; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130949.893888696@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:06:59 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 34/59] objtool: Allow symbol range comparisons for IBT/ENDBR References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra A semi common pattern is where code checks if a code address is within a specific range. All text addresses require either ENDBR or ANNOTATE_ENDBR, however the ANNOTATE_NOENDBR past the range is unnatural. Instead, suppress this warning when this is exactly at the end of a symbol that itself starts with either ENDBR/ANNOTATE_ENDBR. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/entry/entry_64_compat.S | 1 - tools/objtool/check.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -128,7 +128,6 @@ SYM_INNER_LABEL(entry_SYSENTER_compat_af popfq jmp .Lsysenter_flags_fixed SYM_INNER_LABEL(__end_entry_SYSENTER_compat, SYM_L_GLOBAL) - ANNOTATE_NOENDBR // is_sysenter_singlestep SYM_CODE_END(entry_SYSENTER_compat) =20 /* --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -4016,6 +4016,24 @@ static void mark_endbr_used(struct instr list_del_init(&insn->call_node); } =20 +static bool noendbr_range(struct objtool_file *file, struct instruction *i= nsn) +{ + struct symbol *sym =3D find_symbol_containing(insn->sec, insn->offset-1); + struct instruction *first; + + if (!sym) + return false; + + first =3D find_insn(file, sym->sec, sym->offset); + if (!first) + return false; + + if (first->type !=3D INSN_ENDBR && !first->noendbr) + return false; + + return insn->offset =3D=3D sym->offset + sym->len; +} + static int validate_ibt_insn(struct objtool_file *file, struct instruction= *insn) { struct instruction *dest; @@ -4088,9 +4106,19 @@ static int validate_ibt_insn(struct objt continue; } =20 + /* + * Accept anything ANNOTATE_NOENDBR. + */ if (dest->noendbr) continue; =20 + /* + * Accept if this is the instruction after a symbol + * that is (no)endbr -- typical code-range usage. + */ + if (noendbr_range(file, dest)) + continue; + WARN_FUNC("relocation to !ENDBR: %s", insn->sec, insn->offset, offstr(dest->sec, dest->offset)); From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA166ECAAD5 for ; Fri, 2 Sep 2022 14:29:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237092AbiIBO3W (ORCPT ); Fri, 2 Sep 2022 10:29:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236674AbiIBO1a (ORCPT ); Fri, 2 Sep 2022 10:27:30 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EDFA5FACE for ; Fri, 2 Sep 2022 06:54:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LufghqVQE4/oVdylEJUs4Nk7E1aIBOIRXlB89gM3l3o=; b=B9YERCypWa3gNK9KRNrc0Nq8qk G23GqGCK6TlIekmRNbydDhsXiCnPqm8equTiVdwXFDf3Nl6wau1RACEMOIkm6DyfE2j22fql5jaws emA/5Tj6pczWKaF9ZBw6VFfhM5+3S7YE8ewXm1PWv0F2YPvePXH2OWa+n5DZdQGlkAItzWt05c6TD FrIRjaAc2Lk1uUQj5dzA0j9RTL/mhOSiIrQitgUXHeukbv7TFjjYn5LvNRXi7qwNfNU75pIvvo984 JsKZZFSjfYr5hMM+BEWr4Zksj7L2/BPfXIB9NeY5Q3XorHPCHFZjd0372KH9rSoZdrgVHEhJrcs0L ovuZp4YQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-0074TP-Nf; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2FAE6302203; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 0BF3F2B8EF7F9; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130949.998407562@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:00 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 35/59] x86/entry: Make sync_regs() invocation a tail call References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra No point in having a call there. Spare the call/ret overhead. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/entry/entry_64.S | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1062,11 +1062,8 @@ SYM_CODE_START_LOCAL(error_entry) UNTRAIN_RET =20 leaq 8(%rsp), %rdi /* arg0 =3D pt_regs pointer */ -.Lerror_entry_from_usermode_after_swapgs: - /* Put us onto the real thread stack. */ - call sync_regs - RET + jmp sync_regs =20 /* * There are two places in the kernel that can potentially fault with @@ -1124,7 +1121,7 @@ SYM_CODE_START_LOCAL(error_entry) leaq 8(%rsp), %rdi /* arg0 =3D pt_regs pointer */ call fixup_bad_iret mov %rax, %rdi - jmp .Lerror_entry_from_usermode_after_swapgs + jmp sync_regs SYM_CODE_END(error_entry) =20 SYM_CODE_START_LOCAL(error_return) From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CC9BECAAD5 for ; Fri, 2 Sep 2022 14:30:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235835AbiIBOaA (ORCPT ); Fri, 2 Sep 2022 10:30:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236425AbiIBO1e (ORCPT ); Fri, 2 Sep 2022 10:27:34 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BC11158193 for ; Fri, 2 Sep 2022 06:54:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=PRFezF+A6ZMXdXkPHYzuMdhPUJWoNiZcQ9+a8pisS0c=; b=Bq53KEbQq8MsTK3OPz2qEIJ0fG bDB/hSgHLUSwToEmKrJ/xQxg4jO5KCbHKjME+WgEbGHSR3vVEbjVZujSaj/NGbEf5MqRSc97SXIKG an55pvmqAqpnQ9NZAyw0JSBoUhlaWA6SAvBkIGqqoJcDf2LWv3DAsa1xGg1Y9H6nsZaG1+L5VcDWs eYgivt53bktCDhHT8t2vzB4Ozyhhf9IugTYNe+p58mPA9DZ9EoD4zGJa158knik9XHcz8ZLhdtIZb k8DrRcakv2cyAhAVfqFShqfood7Z2mQS6+TTt8xNfzGkQx0SZYKVz5PjYgbjjJFyPMr2JAV7m49N/ SKXSp62A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-0074UC-Df; Fri, 02 Sep 2022 13:53:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3319230241E; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1399E2B8EF7FA; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.102327457@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:01 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 36/59] ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra (Intel) x86 will shortly start using -fpatchable-function-entry for purposes other than ftrace, make sure the __patchable_function_entry section isn't merged in the mcount_loc section. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- include/asm-generic/vmlinux.lds.h | 11 ++++++++++- kernel/trace/Kconfig | 6 ++++++ tools/objtool/check.c | 3 ++- 3 files changed, 18 insertions(+), 2 deletions(-) --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -159,6 +159,14 @@ #define MEM_DISCARD(sec) *(.mem##sec) #endif =20 +#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE +#define KEEP_PATCHABLE KEEP(*(__patchable_function_entries)) +#define PATCHABLE_DISCARDS +#else +#define KEEP_PATCHABLE +#define PATCHABLE_DISCARDS *(__patchable_function_entries) +#endif + #ifdef CONFIG_FTRACE_MCOUNT_RECORD /* * The ftrace call sites are logged to a section whose name depends on the @@ -177,7 +185,7 @@ #define MCOUNT_REC() . =3D ALIGN(8); \ __start_mcount_loc =3D .; \ KEEP(*(__mcount_loc)) \ - KEEP(*(__patchable_function_entries)) \ + KEEP_PATCHABLE \ __stop_mcount_loc =3D .; \ ftrace_stub_graph =3D ftrace_stub; \ ftrace_ops_list_func =3D arch_ftrace_ops_list_func; @@ -1029,6 +1037,7 @@ =20 #define COMMON_DISCARDS \ SANITIZER_DISCARDS \ + PATCHABLE_DISCARDS \ *(.discard) \ *(.discard.*) \ *(.modinfo) \ --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -51,6 +51,12 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS This allows for use of regs_get_kernel_argument() and kernel_stack_pointer(). =20 +config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE + bool + help + If the architecture generates __patchable_function_entries sections + but does not want them included in the ftrace locations. + config HAVE_FTRACE_MCOUNT_RECORD bool help --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -4196,7 +4196,8 @@ static int validate_ibt(struct objtool_f !strcmp(sec->name, "__bug_table") || !strcmp(sec->name, "__ex_table") || !strcmp(sec->name, "__jump_table") || - !strcmp(sec->name, "__mcount_loc")) + !strcmp(sec->name, "__mcount_loc") || + strstr(sec->name, "__patchable_function_entries")) continue; =20 list_for_each_entry(reloc, &sec->reloc->reloc_list, list) From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2DE3C38145 for ; Fri, 2 Sep 2022 14:33:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236636AbiIBOdE (ORCPT ); Fri, 2 Sep 2022 10:33:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238114AbiIBO2s (ORCPT ); Fri, 2 Sep 2022 10:28:48 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3295115712 for ; Fri, 2 Sep 2022 06:54:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=KqaDozNK5h/Os0p9USJOHPsjw+JZHJxcCJnx+VmNqrU=; b=Jy3Utsp5uuyNaCGPnPEy/dqGuA PV0cwwIU05WOMxlcE6wY99LUbZtN5FAnx85xzdMUFQA8NQJWpTxj6WGFaMMYleA+sM0Sg2gjuNzvL CiSFytTxlMQTnOAnlSjoMECb5BStw8uXLLkOuWMm+6tAJReD2yOxhFt9fiULdy+NuUL/6FKKgd7Ys KeqSaDjhyW+i+p+nP5yxayS2necTt3nDyKWGb2yNjNERHdGNcd/A80OMVd6+vD2L2HDHWMA1Y54sP JBtIE/dDpNtQIJWJRD84veIHKwgeVtGWBCU2iwh3QTOxBNY6CHSPQcVEA+t+Z4geY9WyxNMqwyeY1 yNSAZK0A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8M-WC; Fri, 02 Sep 2022 13:53:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 34DD9302911; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1E0DE2B8EF7FB; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.205726504@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:02 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 37/59] x86/putuser: Provide room for padding References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/lib/putuser.S | 62 ++++++++++++++++++++++++++++++++++++++------= ----- 1 file changed, 49 insertions(+), 13 deletions(-) --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -47,8 +47,6 @@ SYM_FUNC_START(__put_user_1) LOAD_TASK_SIZE_MINUS_N(0) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user -SYM_INNER_LABEL(__put_user_nocheck_1, SYM_L_GLOBAL) - ENDBR ASM_STAC 1: movb %al,(%_ASM_CX) xor %ecx,%ecx @@ -56,54 +54,87 @@ SYM_INNER_LABEL(__put_user_nocheck_1, SY RET SYM_FUNC_END(__put_user_1) EXPORT_SYMBOL(__put_user_1) + +SYM_FUNC_START(__put_user_nocheck_1) + ENDBR + ASM_STAC +2: movb %al,(%_ASM_CX) + xor %ecx,%ecx + ASM_CLAC + RET +SYM_FUNC_END(__put_user_nocheck_1) EXPORT_SYMBOL(__put_user_nocheck_1) =20 SYM_FUNC_START(__put_user_2) LOAD_TASK_SIZE_MINUS_N(1) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user -SYM_INNER_LABEL(__put_user_nocheck_2, SYM_L_GLOBAL) - ENDBR ASM_STAC -2: movw %ax,(%_ASM_CX) +3: movw %ax,(%_ASM_CX) xor %ecx,%ecx ASM_CLAC RET SYM_FUNC_END(__put_user_2) EXPORT_SYMBOL(__put_user_2) + +SYM_FUNC_START(__put_user_nocheck_2) + ENDBR + ASM_STAC +4: movw %ax,(%_ASM_CX) + xor %ecx,%ecx + ASM_CLAC + RET +SYM_FUNC_END(__put_user_nocheck_2) EXPORT_SYMBOL(__put_user_nocheck_2) =20 SYM_FUNC_START(__put_user_4) LOAD_TASK_SIZE_MINUS_N(3) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user -SYM_INNER_LABEL(__put_user_nocheck_4, SYM_L_GLOBAL) - ENDBR ASM_STAC -3: movl %eax,(%_ASM_CX) +5: movl %eax,(%_ASM_CX) xor %ecx,%ecx ASM_CLAC RET SYM_FUNC_END(__put_user_4) EXPORT_SYMBOL(__put_user_4) + +SYM_FUNC_START(__put_user_nocheck_4) + ENDBR + ASM_STAC +6: movl %eax,(%_ASM_CX) + xor %ecx,%ecx + ASM_CLAC + RET +SYM_FUNC_END(__put_user_nocheck_4) EXPORT_SYMBOL(__put_user_nocheck_4) =20 SYM_FUNC_START(__put_user_8) LOAD_TASK_SIZE_MINUS_N(7) cmp %_ASM_BX,%_ASM_CX jae .Lbad_put_user -SYM_INNER_LABEL(__put_user_nocheck_8, SYM_L_GLOBAL) - ENDBR ASM_STAC -4: mov %_ASM_AX,(%_ASM_CX) +7: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 -5: movl %edx,4(%_ASM_CX) +8: movl %edx,4(%_ASM_CX) #endif xor %ecx,%ecx ASM_CLAC RET SYM_FUNC_END(__put_user_8) EXPORT_SYMBOL(__put_user_8) + +SYM_FUNC_START(__put_user_nocheck_8) + ENDBR + ASM_STAC +9: mov %_ASM_AX,(%_ASM_CX) +#ifdef CONFIG_X86_32 +10: movl %edx,4(%_ASM_CX) +#endif + xor %ecx,%ecx + ASM_CLAC + RET +SYM_FUNC_END(__put_user_nocheck_8) EXPORT_SYMBOL(__put_user_nocheck_8) =20 SYM_CODE_START_LOCAL(.Lbad_put_user_clac) @@ -117,6 +148,11 @@ SYM_CODE_END(.Lbad_put_user_clac) _ASM_EXTABLE_UA(2b, .Lbad_put_user_clac) _ASM_EXTABLE_UA(3b, .Lbad_put_user_clac) _ASM_EXTABLE_UA(4b, .Lbad_put_user_clac) -#ifdef CONFIG_X86_32 _ASM_EXTABLE_UA(5b, .Lbad_put_user_clac) + _ASM_EXTABLE_UA(6b, .Lbad_put_user_clac) + _ASM_EXTABLE_UA(7b, .Lbad_put_user_clac) + _ASM_EXTABLE_UA(9b, .Lbad_put_user_clac) +#ifdef CONFIG_X86_32 + _ASM_EXTABLE_UA(8b, .Lbad_put_user_clac) + _ASM_EXTABLE_UA(10b, .Lbad_put_user_clac) #endif From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B0CDECAAD5 for ; Fri, 2 Sep 2022 14:31:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236313AbiIBObg (ORCPT ); Fri, 2 Sep 2022 10:31:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236085AbiIBO1g (ORCPT ); Fri, 2 Sep 2022 10:27:36 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6ECCE158199 for ; Fri, 2 Sep 2022 06:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=/Fb9zUxhEEbcmp5Brb1qrwIPV90iUAqtYDCEw6WSBXI=; b=hKiXy0u9hoSBZF8nc2DmAek7xT TEFVtr6iI17ckJ00gdUCFKOkzxud2RS2pcRnzx+7yRivbNAjWM9jEaIjPkzI2sYFPxL4RyBYyBgVU dHM8aIxF/6VZ6DDtvX1rsU4hdUf6X+XN9TLDCay5I+u9LOIDZcoMakow3Q4CzDCy6EftPPysqztP4 aP6P5RmQ7VfzKVLHpab6CV6lJOkt5i8BV718IltoWTNNLqKNHPYkZ97oDVVzPJ/m0D5157rfAE+Nn S5Zok8o+Y5aNEW0O3IWboq7C53rBPnQek/Of8ePpXDZqaGz/S91hQ89ukinIHcdOlRQaFFFi3P+Kn P5aYbL3w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-008g8V-6c; Fri, 02 Sep 2022 13:53:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 35D4F302A1D; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 280222B8EF7FC; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.308238294@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:03 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 38/59] x86/Kconfig: Add CONFIG_CALL_THUNKS References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner In preparation for mitigating the Intel SKL RSB underflow issue in software, add a new configuration symbol which allows to build the required call thunk infrastructure conditionally. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/Kconfig | 7 +++++++ 1 file changed, 7 insertions(+) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2432,6 +2432,13 @@ config CC_HAS_SLS config CC_HAS_RETURN_THUNK def_bool $(cc-option,-mfunction-return=3Dthunk-extern) =20 +config HAVE_CALL_THUNKS + def_bool y + depends on RETHUNK && OBJTOOL + +config CALL_THUNKS + def_bool n + menuconfig SPECULATION_MITIGATIONS bool "Mitigations for speculative execution vulnerabilities" default y From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DD98C38145 for ; Fri, 2 Sep 2022 14:32:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236147AbiIBOcj (ORCPT ); Fri, 2 Sep 2022 10:32:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237952AbiIBO2r (ORCPT ); Fri, 2 Sep 2022 10:28:47 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 326DC140CE for ; Fri, 2 Sep 2022 06:54:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=yAGW7R8vTxeOqWUEPXbZtse6DzAQxppiABwB+quxCsY=; b=iTzoVo4CmQFy1IW4MYB9A0TLxW JbgR2Ls5vDF6W2uy1JxTWkUH5JH1HIUt03/RvuCwL+pJicEv1Y9a2bufrnA4CQ0eYxtbLOQj49oeQ IDxmnBVLqVS97joi41/KPfHRJxRUMdi3dS8zAdv302eB3SF7zIhUhWph2HcBEtvy7XSirX1r+kT/r sugQn6Bw4J+oGLgDJqvvy7juSCNLrqbqsU4TNTcH7r85gtLx4SF7vLxIQm0qjpNrusYBd00WUUq2V sRrG1OtXSDQSyhHt24Mb16s8i9e4WOctsW4fsUx0rK6kvyLMhX7hunc55dHfMU6QoFYKMuGaQT3Wz M36ooAFg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77M-008g8K-TR; Fri, 02 Sep 2022 13:54:02 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 346E330276B; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2F9872B8EF7FD; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.408585331@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:04 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 39/59] x86/Kconfig: Introduce function padding References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Now that all functions are 16 byte aligned, add 16 bytes of NOP padding in front of each function. This prepares things for software call stack tracking and kCFI/FineIBT. This significantly increases kernel .text size, around 5.1% on a x86_64-defconfig-ish build. However, per the random access argument used for alignment, these 16 extra bytes are code that wouldn't be used. Performance measurements back this up by showing no significant performance regressions. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- Makefile | 3 +++ arch/x86/Kconfig | 12 +++++++++++- arch/x86/Makefile | 5 +++++ arch/x86/entry/vdso/Makefile | 3 ++- arch/x86/include/asm/linkage.h | 28 ++++++++++++++++++++++++---- 5 files changed, 45 insertions(+), 6 deletions(-) --- a/Makefile +++ b/Makefile @@ -920,6 +920,9 @@ KBUILD_AFLAGS +=3D -fno-lto export CC_FLAGS_LTO endif =20 +PADDING_CFLAGS :=3D -fpatchable-function-entry=3D$(CONFIG_FUNCTION_PADDING= _BYTES),$(CONFIG_FUNCTION_PADDING_BYTES) +export PADDING_CFLAGS + ifdef CONFIG_CFI_CLANG CC_FLAGS_CFI :=3D -fsanitize=3Dcfi \ -fsanitize-cfi-cross-dso \ --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2427,9 +2427,19 @@ config CC_HAS_SLS config CC_HAS_RETURN_THUNK def_bool $(cc-option,-mfunction-return=3Dthunk-extern) =20 +config CC_HAS_ENTRY_PADDING + def_bool $(cc-option,-fpatchable-function-entry=3D16,16) + +config FUNCTION_PADDING_BYTES + int + default 32 if CALL_THUNKS_DEBUG && !CFI_CLANG + default 27 if CALL_THUNKS_DEBUG && CFI_CLANG + default 16 if !CFI_CLANG + default 11 + config HAVE_CALL_THUNKS def_bool y - depends on RETHUNK && OBJTOOL + depends on CC_HAS_ENTRY_PADDING && RETHUNK && OBJTOOL =20 config CALL_THUNKS def_bool n --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -202,6 +202,11 @@ ifdef CONFIG_SLS KBUILD_CFLAGS +=3D -mharden-sls=3Dall endif =20 +PADDING_CFLAGS :=3D -fpatchable-function-entry=3D$(CONFIG_FUNCTION_PADDING= _BYTES),$(CONFIG_FUNCTION_PADDING_BYTES) +ifdef CONFIG_CALL_THUNKS + KBUILD_CFLAGS +=3D $(PADDING_CFLAGS) +endif + KBUILD_LDFLAGS +=3D -m elf_$(UTS_MACHINE) =20 ifdef CONFIG_LTO_CLANG --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -92,7 +92,7 @@ ifneq ($(RETPOLINE_VDSO_CFLAGS),) endif endif =20 -$(vobjs): KBUILD_CFLAGS :=3D $(filter-out $(CC_FLAGS_LTO) $(RANDSTRUCT_CFL= AGS) $(GCC_PLUGINS_CFLAGS) $(RETPOLINE_CFLAGS),$(KBUILD_CFLAGS)) $(CFL) +$(vobjs): KBUILD_CFLAGS :=3D $(filter-out $(PADDING_CFLAGS) $(CC_FLAGS_LTO= ) $(RANDSTRUCT_CFLAGS) $(GCC_PLUGINS_CFLAGS) $(RETPOLINE_CFLAGS),$(KBUILD_C= FLAGS)) $(CFL) $(vobjs): KBUILD_AFLAGS +=3D -DBUILD_VDSO =20 # @@ -154,6 +154,7 @@ KBUILD_CFLAGS_32 :=3D $(filter-out $(RANDS KBUILD_CFLAGS_32 :=3D $(filter-out $(GCC_PLUGINS_CFLAGS),$(KBUILD_CFLAGS_3= 2)) KBUILD_CFLAGS_32 :=3D $(filter-out $(RETPOLINE_CFLAGS),$(KBUILD_CFLAGS_32)) KBUILD_CFLAGS_32 :=3D $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS_32)) +KBUILD_CFLAGS_32 :=3D $(filter-out $(PADDING_CFLAGS),$(KBUILD_CFLAGS_32)) KBUILD_CFLAGS_32 +=3D -m32 -msoft-float -mregparm=3D0 -fpic KBUILD_CFLAGS_32 +=3D -fno-stack-protector KBUILD_CFLAGS_32 +=3D $(call cc-option, -foptimize-sibling-calls) --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -13,17 +13,37 @@ #endif /* CONFIG_X86_32 */ =20 #if CONFIG_FUNCTION_ALIGNMENT =3D=3D 8 -#define __ALIGN .p2align 3, 0x90; +# define __ALIGN .p2align 3, 0x90; #elif CONFIG_FUNCTION_ALIGNMENT =3D=3D 16 -#define __ALIGN .p2align 4, 0x90; +# define __ALIGN .p2align 4, 0x90; +#elif CONFIG_FUNCTION_ALIGNMENT =3D=3D 32 +# define __ALIGN .p2align 5, 0x90 #else # error Unsupported function alignment #endif =20 #define __ALIGN_STR __stringify(__ALIGN) -#define ASM_FUNC_ALIGN __ALIGN_STR -#define __FUNC_ALIGN __ALIGN + +#ifdef CONFIG_CFI_CLANG +#define __FUNCTION_PADDING (CONFIG_FUNCTION_ALIGNMENT - 5) +#else +#define __FUNCTION_PADDING CONFIG_FUNCTION_ALIGNMENT +#endif + +#if defined(CONFIG_CALL_THUNKS) && !defined(__DISABLE_EXPORTS) && !defined= (BUILD_VDSO) +#define FUNCTION_PADDING .skip __FUNCTION_PADDING, 0x90; +#else +#define FUNCTION_PADDING +#endif + +#if (CONFIG_FUNCTION_ALIGNMENT > 8) && !defined(__DISABLE_EXPORTS) && !def= ined(BULID_VDSO) +# define __FUNC_ALIGN __ALIGN; FUNCTION_PADDING +#else +# define __FUNC_ALIGN __ALIGN +#endif + #define SYM_F_ALIGN __FUNC_ALIGN +#define ASM_FUNC_ALIGN __stringify(__FUNC_ALIGN) =20 #ifdef __ASSEMBLY__ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80B90C38145 for ; Fri, 2 Sep 2022 14:29:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235697AbiIBO3c (ORCPT ); Fri, 2 Sep 2022 10:29:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236799AbiIBO1b (ORCPT ); Fri, 2 Sep 2022 10:27:31 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6950D168A0C for ; Fri, 2 Sep 2022 06:54:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=klBRHEqDjEufBDz1HodoVZDCOLPcY0MzIEhSN35NGf8=; b=jauH8ValwyKlVKBNReLFguAuBv afxlG/tU7/ttnV6OE2ewYvzBbkHqQNHNeNuI4R3Act8qMFbbhDFtY2cTIQ7EP5qIb5VlvlmhLTWkU Gw3f339kLAJlqhrQ+5eGnFAqY4yj8sEWgFnMp0rFHTEwKEHTDuNU0qfq8Di3IPg7OYwTVPSKJddWt nM9XXnOoKE/4DBTxiP7kthkmKREayKds8YTtTrkQgZ6jEcKWPJ9xAIywtl2IC9KNQgqb55Vs9lKks oMmDlmxa/ShS9lgUmE/bhZxsQAuVhvGOIv2bpEuLFjEY6CNEnFyJ9703wmJVgnan0j7NDDuL2U3Rr bnyRQ9jA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77N-008g8N-1S; Fri, 02 Sep 2022 13:53:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 36B57302A31; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 371632B8EF7FE; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.514617734@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:05 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 40/59] x86/retbleed: Add X86_FEATURE_CALL_DEPTH References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Intel SKL CPUs fall back to other predictors when the RSB underflows. The only microcode mitigation is IBRS which is insanely expensive. It comes with performance drops of up to 30% depending on the workload. A way less expensive, but nevertheless horrible mitigation is to track the call depth in software and overeagerly fill the RSB when returns underflow the software counter. Provide a configuration symbol and a CPU misfeature bit. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/Kconfig | 19 +++++++++++++++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 9 ++++++++- 3 files changed, 28 insertions(+), 1 deletion(-) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2501,6 +2501,25 @@ config CPU_UNRET_ENTRY help Compile the kernel with support for the retbleed=3Dunret mitigation. =20 +config CALL_DEPTH_TRACKING + bool "Mitigate RSB underflow with call depth tracking" + depends on CPU_SUP_INTEL && HAVE_CALL_THUNKS + select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE + select CALL_THUNKS + default y + help + Compile the kernel with call depth tracking to mitigate the Intel + SKL Return-Speculation-Buffer (RSB) underflow issue. The + mitigation is off by default and needs to be enabled on the + kernel command line via the retbleed=3Dstuff option. For + non-affected systems the overhead of this option is marginal as + the call depth tracking is using run-time generated call thunks + in a compiler generated padding area and call patching. This + increases text size by ~5%. For non affected systems this space + is unused. On affected SKL systems this results in a significant + performance gain over the IBRS mitigation. + + config CPU_IBPB_ENTRY bool "Enable IBPB on kernel entry" depends on CPU_SUP_AMD && X86_64 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -304,6 +304,7 @@ #define X86_FEATURE_UNRET (11*32+15) /* "" AMD BTB untrain return */ #define X86_FEATURE_USE_IBPB_FW (11*32+16) /* "" Use IBPB during runtime = firmware calls */ #define X86_FEATURE_RSB_VMEXIT_LITE (11*32+17) /* "" Fill RSB on VM exit w= hen EIBRS is enabled */ +#define X86_FEATURE_CALL_DEPTH (11*32+18) /* "" Call depth tracking for R= SB stuffing */ =20 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -69,6 +69,12 @@ # define DISABLE_UNRET (1 << (X86_FEATURE_UNRET & 31)) #endif =20 +#ifdef CONFIG_CALL_DEPTH_TRACKING +# define DISABLE_CALL_DEPTH_TRACKING 0 +#else +# define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -101,7 +107,8 @@ #define DISABLED_MASK8 (DISABLE_TDX_GUEST) #define DISABLED_MASK9 (DISABLE_SGX) #define DISABLED_MASK10 0 -#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET) +#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ + DISABLE_CALL_DEPTH_TRACKING) #define DISABLED_MASK12 0 #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3118C38145 for ; Fri, 2 Sep 2022 14:32:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236255AbiIBOc3 (ORCPT ); Fri, 2 Sep 2022 10:32:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237927AbiIBO2q (ORCPT ); Fri, 2 Sep 2022 10:28:46 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB8BF220F3 for ; Fri, 2 Sep 2022 06:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=0/PtPljRLva+i3OXSAsXYK2WvcvC4WWFyxPXVnV3iEE=; b=pnyIvBO9gt2Cr7Gu3Ph5LKtXQr gB0a/lpgcjFvHHcHu0180MITGJ40/OKa2Y81oTS3UxhmtSxVciqbdemjiKdJkb42VapXhxi2BkxqO aNYhIF2kX1HpzWrHJdHJ1byeFfhE1KZPCZpM6FbPjNbkSjiaP0UgrewZ1LLjyLwKXcUS5zF+ZWC03 iiLLx978NrsWqR9RIJXEtz2r7ZsWjRHUISEufrAKaPNvdipf/wEE0ZLDXPlWzv89fjLEmgNG++Ils uE5y+SG+pOYQ0V9fsOxuy4+eS1XEPjYQxm+dpfZXYZeQ+Z7I6ZNJyeSpNWCZ32DSQIF+iQ1TOB8NE Lto8eggw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-008g8d-4m; Fri, 02 Sep 2022 13:54:02 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3C92F302D43; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3E8D72B8EF7FF; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.618320501@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:06 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 41/59] x86/alternatives: Provide text_poke_copy_locked() References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The upcoming call thunk patching must hold text_mutex and needs access to text_poke_copy(), which takes text_mutex. Provide a _locked postfixed variant to expose the inner workings. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/text-patching.h | 1=20 arch/x86/kernel/alternative.c | 37 ++++++++++++++++++++----------= ----- 2 files changed, 23 insertions(+), 15 deletions(-) --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -45,6 +45,7 @@ extern void *text_poke(void *addr, const extern void text_poke_sync(void); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern void *text_poke_copy(void *addr, const void *opcode, size_t len); +extern void *text_poke_copy_locked(void *addr, const void *opcode, size_t = len, bool core_ok); extern void *text_poke_set(void *addr, int c, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void text_poke_bp(void *addr, const void *opcode, size_t len, const= void *emulate); --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1227,27 +1227,15 @@ void *text_poke_kgdb(void *addr, const v return __text_poke(text_poke_memcpy, addr, opcode, len); } =20 -/** - * text_poke_copy - Copy instructions into (an unused part of) RX memory - * @addr: address to modify - * @opcode: source of the copy - * @len: length to copy, could be more than 2x PAGE_SIZE - * - * Not safe against concurrent execution; useful for JITs to dump - * new code blocks into unused regions of RX memory. Can be used in - * conjunction with synchronize_rcu_tasks() to wait for existing - * execution to quiesce after having made sure no existing functions - * pointers are live. - */ -void *text_poke_copy(void *addr, const void *opcode, size_t len) +void *text_poke_copy_locked(void *addr, const void *opcode, size_t len, + bool core_ok) { unsigned long start =3D (unsigned long)addr; size_t patched =3D 0; =20 - if (WARN_ON_ONCE(core_kernel_text(start))) + if (WARN_ON_ONCE(!core_ok && core_kernel_text(start))) return NULL; =20 - mutex_lock(&text_mutex); while (patched < len) { unsigned long ptr =3D start + patched; size_t s; @@ -1257,6 +1245,25 @@ void *text_poke_copy(void *addr, const v __text_poke(text_poke_memcpy, (void *)ptr, opcode + patched, s); patched +=3D s; } + return addr; +} + +/** + * text_poke_copy - Copy instructions into (an unused part of) RX memory + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy, could be more than 2x PAGE_SIZE + * + * Not safe against concurrent execution; useful for JITs to dump + * new code blocks into unused regions of RX memory. Can be used in + * conjunction with synchronize_rcu_tasks() to wait for existing + * execution to quiesce after having made sure no existing functions + * pointers are live. + */ +void *text_poke_copy(void *addr, const void *opcode, size_t len) +{ + mutex_lock(&text_mutex); + addr =3D text_poke_copy_locked(addr, opcode, len, false); mutex_unlock(&text_mutex); return addr; } From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66F34C38145 for ; Fri, 2 Sep 2022 14:30:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236814AbiIBOaq (ORCPT ); Fri, 2 Sep 2022 10:30:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237328AbiIBO1i (ORCPT ); Fri, 2 Sep 2022 10:27:38 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FEB51581A0 for ; Fri, 2 Sep 2022 06:54:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=SRnNkt6hDAocphBeeJEDz/2BjTiGPSIHa9g4nx269kU=; b=W9YsMVXO3ZSwz6cwrIMHRc5R+H pAPx70Q9qxyZ4fHSzemN4QHmvkXMfcQL6Pd4mRJnWXEb08hmQzO4PFwU69vObFim3DrYxmYO5vD2s stA7RoiwHu2GeCsAzgQPTvI/Xy+WDViFH13I0vwsnolDFYCQsYLfSrOXxigfTMMLU4haPFxzf4puZ s3soTbw7D/RaFGQgaB3zcLWoqrXYBUnJD47l0B1u2AgHU/V07gmXXHLnW2TJ1ZP+kK7syF7J7pdW8 DgaBgOZYOLYvEukw11aUvwv4E6h3dh7w98TnzwwK6+qf6Ls2fXnHMj0fblhbshti9zZF72nRLxzRP Vie5G/ng==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-008g8c-2B; Fri, 02 Sep 2022 13:54:10 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 39EB2302ADB; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 45F262B8EFB4D; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.722045248@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:07 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 42/59] x86/entry: Make some entry symbols global References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner paranoid_entry(), error_entry() and xen_error_entry() have to be exempted from call accounting by thunk patching because they are before UNTRAIN_RET. Expose them so they are available in the alternative code. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/entry/entry_64.S | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -327,7 +327,8 @@ SYM_CODE_END(ret_from_fork) #endif .endm =20 -SYM_CODE_START_LOCAL(xen_error_entry) +SYM_CODE_START(xen_error_entry) + ANNOTATE_NOENDBR UNWIND_HINT_FUNC PUSH_AND_CLEAR_REGS save_ret=3D1 ENCODE_FRAME_POINTER 8 @@ -906,7 +907,8 @@ SYM_CODE_END(xen_failsafe_callback) * R14 - old CR3 * R15 - old SPEC_CTRL */ -SYM_CODE_START_LOCAL(paranoid_entry) +SYM_CODE_START(paranoid_entry) + ANNOTATE_NOENDBR UNWIND_HINT_FUNC PUSH_AND_CLEAR_REGS save_ret=3D1 ENCODE_FRAME_POINTER 8 @@ -1041,7 +1043,8 @@ SYM_CODE_END(paranoid_exit) /* * Switch GS and CR3 if needed. */ -SYM_CODE_START_LOCAL(error_entry) +SYM_CODE_START(error_entry) + ANNOTATE_NOENDBR UNWIND_HINT_FUNC =20 PUSH_AND_CLEAR_REGS save_ret=3D1 From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F3ECECAAD5 for ; Fri, 2 Sep 2022 14:29:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236382AbiIBO3C (ORCPT ); Fri, 2 Sep 2022 10:29:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236462AbiIBO11 (ORCPT ); Fri, 2 Sep 2022 10:27:27 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB37313F0A8 for ; Fri, 2 Sep 2022 06:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=l7GhXBSYIn8Kq/LEBL5LqWkTVx56PKiEeOdm0uGjDBU=; b=DYSB8cS0PAYxsV9meJmvbwfRMI sZsFWjenDxeGk6NtFpDE+12Fuxvfm/sgJ5zXTki3CkxEYIVMTYHULRMhYfHdlysn62fyxUhwriIrp 0+AAUHmM1rwubUq+5G2ibbUtniZBUCmq1BLazjKGRyQaP1hRsanM3F5JV+ZFhXlM8Tb5kcMHTrLbH v6SUNo+vKdPjxLexeNSEyezr3TzuYKZD8Zv1iwHcaxVsG9z70dmNSfU30X0IRk7aWaInSG0TV1AHg BG+FcCnq0LBs1Rx1N3v+ouxJdy7sWgw7s05GNae34KJCEl3kE1F3zCNcXu7l9x3ViLsYzqkNo9AmG fQKGWHjw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77P-0074V0-Mz; Fri, 02 Sep 2022 13:53:59 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3C950302D57; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4D6462B8EFB52; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.826439931@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:08 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 43/59] x86/paravirt: Make struct paravirt_call_site unconditionally available References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner For the upcoming call thunk patching it's less ifdeffery when the data structure is unconditionally available. The code can then be trivially fenced off with IS_ENABLED(). Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juergen Gross Tested-by: Mel Gorman --- arch/x86/include/asm/paravirt.h | 4 ++-- arch/x86/include/asm/paravirt_types.h | 20 ++++++++++++-------- 2 files changed, 14 insertions(+), 10 deletions(-) --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -4,13 +4,13 @@ /* Various instructions on x86 need to be replaced for * para-virtualization: those hooks are defined here. */ =20 +#include + #ifdef CONFIG_PARAVIRT #include #include #include =20 -#include - #ifndef __ASSEMBLY__ #include #include --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -2,6 +2,17 @@ #ifndef _ASM_X86_PARAVIRT_TYPES_H #define _ASM_X86_PARAVIRT_TYPES_H =20 +#ifndef __ASSEMBLY__ +/* These all sit in the .parainstructions section to tell us what to patch= . */ +struct paravirt_patch_site { + u8 *instr; /* original instructions */ + u8 type; /* type of this instruction */ + u8 len; /* length of original instruction */ +}; +#endif + +#ifdef CONFIG_PARAVIRT + /* Bitmask of what can be clobbered: usually at least eax. */ #define CLBR_EAX (1 << 0) #define CLBR_ECX (1 << 1) @@ -584,16 +595,9 @@ unsigned long paravirt_ret0(void); =20 #define paravirt_nop ((void *)_paravirt_nop) =20 -/* These all sit in the .parainstructions section to tell us what to patch= . */ -struct paravirt_patch_site { - u8 *instr; /* original instructions */ - u8 type; /* type of this instruction */ - u8 len; /* length of original instruction */ -}; - extern struct paravirt_patch_site __parainstructions[], __parainstructions_end[]; =20 #endif /* __ASSEMBLY__ */ - +#endif /* CONFIG_PARAVIRT */ #endif /* _ASM_X86_PARAVIRT_TYPES_H */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6333CC38145 for ; Fri, 2 Sep 2022 14:29:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236971AbiIBO3M (ORCPT ); Fri, 2 Sep 2022 10:29:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236760AbiIBO13 (ORCPT ); Fri, 2 Sep 2022 10:27:29 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2315141A34 for ; Fri, 2 Sep 2022 06:54:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LERcEC5SjlLoVKziVusi6bsy/opegGuxqISxMS9EI5Y=; b=iQ8u5uqbA+FvlfMZyv5Hf+fc8C c9Ibger3fcm660nqCZUNez7AjwYAETtjqG9n3Ts87fKHGdSolD/sNwXI3n556tGiVL5sWgwwYXzM+ Mt/HprOMDQw/5dw1vI9PbfOVZxadx0LTQygrndhWpNXM6kn0Q3AduCmFIOXIwCizQo1htmBrZ8EIs g1g3CnKJAJP5jyf1fR2Ikid+54PQPEljNIU9oarE7QGi2KHvreeHCiwIUeDawRpOImrvrnUuTcL3f QSjW1yOWabj0LAaHR+VtzOj+XxX8ci7jOdeX5I1ityIKrPYitDRfkhR/L+rWoSn6wX015Sc41jmvW UQsi7EgA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-0074UD-ED; Fri, 02 Sep 2022 13:53:59 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3C133302D39; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 561DB2B8EF7F8; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130950.928899774@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:09 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 44/59] x86/callthunks: Add call patching for call depth tracking References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Mitigating the Intel SKL RSB underflow issue in software requires to track the call depth. That is every CALL and every RET need to be intercepted and additional code injected. The existing retbleed mitigations already include means of redirecting RET to __x86_return_thunk; this can be re-purposed and RET can be redirected to another function doing RET accounting. CALL accounting will use the function padding introduced in prior patches. For each CALL instruction, the destination symbol's padding is rewritten to do the accounting and the CALL instruction is adjusted to call into the padding. This ensures only affected CPUs pay the overhead of this accounting. Unaffected CPUs will leave the padding unused and have their 'JMP __x86_return_thunk' replaced with an actual 'RET' instruction. Objtool has been modified to supply a .call_sites section that lists all the 'CALL' instructions. Additionally the paravirt instruction sites are iterated since they will have been patched from an indirect call to direct calls (or direct instructions in which case it'll be ignored). Module handling and the actual thunk code for SKL will be added in subsequent steps. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/Kconfig | 11 + arch/x86/include/asm/alternative.h | 12 + arch/x86/kernel/Makefile | 2=20 arch/x86/kernel/alternative.c | 6=20 arch/x86/kernel/callthunks.c | 251 ++++++++++++++++++++++++++++++= +++++ arch/x86/kernel/head_64.S | 1=20 arch/x86/kernel/relocate_kernel_64.S | 5=20 arch/x86/kernel/vmlinux.lds.S | 8 - 8 files changed, 286 insertions(+), 10 deletions(-) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2519,6 +2519,17 @@ config CALL_DEPTH_TRACKING is unused. On affected SKL systems this results in a significant performance gain over the IBRS mitigation. =20 +config CALL_THUNKS_DEBUG + bool "Enable call thunks and call depth tracking debugging" + depends on CALL_DEPTH_TRACKING + default n + help + Enable call/ret counters for imbalance detection and build in + a noisy dmesg about callthunks generation and call patching for + trouble shooting. The debug prints need to be enabled on the + kernel command line with 'debug-callthunks'. + Only enable this, when you are debugging call thunks as this + creates a noticable runtime overhead. If unsure say N. =20 config CPU_IBPB_ENTRY bool "Enable IBPB on kernel entry" --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -80,6 +80,18 @@ extern void apply_returns(s32 *start, s3 extern void apply_ibt_endbr(s32 *start, s32 *end); =20 struct module; +struct paravirt_patch_site; + +struct callthunk_sites { + s32 *call_start, *call_end; + struct paravirt_patch_site *pv_start, *pv_end; +}; + +#ifdef CONFIG_CALL_THUNKS +extern void callthunks_patch_builtin_calls(void); +#else +static __always_inline void callthunks_patch_builtin_calls(void) {} +#endif =20 #ifdef CONFIG_SMP extern void alternatives_smp_module_add(struct module *mod, char *name, --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_GUESS) +=3D unwind_ =20 obj-$(CONFIG_AMD_MEM_ENCRYPT) +=3D sev.o =20 +obj-$(CONFIG_CALL_THUNKS) +=3D callthunks.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -938,6 +938,12 @@ void __init alternative_instructions(voi */ apply_alternatives(__alt_instructions, __alt_instructions_end); =20 + /* + * Now all calls are established. Apply the call thunks if + * required. + */ + callthunks_patch_builtin_calls(); + apply_ibt_endbr(__ibt_endbr_seal, __ibt_endbr_seal_end); =20 #ifdef CONFIG_SMP --- /dev/null +++ b/arch/x86/kernel/callthunks.c @@ -0,0 +1,251 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#define pr_fmt(fmt) "callthunks: " fmt + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int __initdata_or_module debug_callthunks; + +#define prdbg(fmt, args...) \ +do { \ + if (debug_callthunks) \ + printk(KERN_DEBUG pr_fmt(fmt), ##args); \ +} while(0) + +static int __init debug_thunks(char *str) +{ + debug_callthunks =3D 1; + return 1; +} +__setup("debug-callthunks", debug_thunks); + +extern s32 __call_sites[], __call_sites_end[]; + +struct thunk_desc { + void *template; + unsigned int template_size; +}; + +struct core_text { + unsigned long base; + unsigned long end; + const char *name; +}; + +static bool thunks_initialized __ro_after_init; + +static const struct core_text builtin_coretext =3D { + .base =3D (unsigned long)_text, + .end =3D (unsigned long)_etext, + .name =3D "builtin", +}; + +static struct thunk_desc callthunk_desc __ro_after_init; + +extern void error_entry(void); +extern void xen_error_entry(void); +extern void paranoid_entry(void); + +static inline bool within_coretext(const struct core_text *ct, void *addr) +{ + unsigned long p =3D (unsigned long)addr; + + return ct->base <=3D p && p < ct->end; +} + +static inline bool within_module_coretext(void *addr) +{ + bool ret =3D false; + +#ifdef CONFIG_MODULES + struct module *mod; + + preempt_disable(); + mod =3D __module_address((unsigned long)addr); + if (mod && within_module_core((unsigned long)addr, mod)) + ret =3D true; + preempt_enable(); +#endif + return ret; +} + +static bool is_coretext(const struct core_text *ct, void *addr) +{ + if (ct && within_coretext(ct, addr)) + return true; + if (within_coretext(&builtin_coretext, addr)) + return true; + return within_module_coretext(addr); +} + +static __init_or_module bool skip_addr(void *dest) +{ + if (dest =3D=3D error_entry) + return true; + if (dest =3D=3D paranoid_entry) + return true; + if (dest =3D=3D xen_error_entry) + return true; + /* Does FILL_RSB... */ + if (dest =3D=3D __switch_to_asm) + return true; + /* Accounts directly */ + if (dest =3D=3D ret_from_fork) + return true; +#ifdef CONFIG_HOTPLUG_CPU + if (dest =3D=3D start_cpu0) + return true; +#endif +#ifdef CONFIG_FUNCTION_TRACER + if (dest =3D=3D __fentry__) + return true; +#endif +#ifdef CONFIG_KEXEC_CORE + if (dest >=3D (void *)relocate_kernel && + dest < (void*)relocate_kernel + KEXEC_CONTROL_CODE_MAX_SIZE) + return true; +#endif +#ifdef CONFIG_XEN + if (dest >=3D (void *)hypercall_page && + dest < (void*)hypercall_page + PAGE_SIZE) + return true; +#endif + return false; +} + +static __init_or_module void *call_get_dest(void *addr) +{ + struct insn insn; + void *dest; + int ret; + + ret =3D insn_decode_kernel(&insn, addr); + if (ret) + return ERR_PTR(ret); + + /* Patched out call? */ + if (insn.opcode.bytes[0] !=3D CALL_INSN_OPCODE) + return NULL; + + dest =3D addr + insn.length + insn.immediate.value; + if (skip_addr(dest)) + return NULL; + return dest; +} + +static const u8 nops[] =3D { + 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, + 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, + 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, + 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, +}; + +static __init_or_module void *patch_dest(void *dest, bool direct) +{ + unsigned int tsize =3D callthunk_desc.template_size; + u8 *pad =3D dest - tsize; + + /* Already patched? */ + if (!bcmp(pad, callthunk_desc.template, tsize)) + return pad; + + /* Ensure there are nops */ + if (bcmp(pad, nops, tsize)) { + pr_warn_once("Invalid padding area for %pS\n", dest); + return NULL; + } + + if (direct) + memcpy(pad, callthunk_desc.template, tsize); + else + text_poke_copy_locked(pad, callthunk_desc.template, tsize, true); + return pad; +} + +static __init_or_module void patch_call(void *addr, const struct core_text= *ct) +{ + void *pad, *dest; + u8 bytes[8]; + + if (!within_coretext(ct, addr)) + return; + + dest =3D call_get_dest(addr); + if (!dest || WARN_ON_ONCE(IS_ERR(dest))) + return; + + if (!is_coretext(ct, dest)) + return; + + pad =3D patch_dest(dest, within_coretext(ct, dest)); + if (!pad) + return; + + prdbg("Patch call at: %pS %px to %pS %px -> %px \n", addr, addr, + dest, dest, pad); + __text_gen_insn(bytes, CALL_INSN_OPCODE, addr, pad, CALL_INSN_SIZE); + text_poke_early(addr, bytes, CALL_INSN_SIZE); +} + +static __init_or_module void +patch_call_sites(s32 *start, s32 *end, const struct core_text *ct) +{ + s32 *s; + + for (s =3D start; s < end; s++) + patch_call((void *)s + *s, ct); +} + +static __init_or_module void +patch_paravirt_call_sites(struct paravirt_patch_site *start, + struct paravirt_patch_site *end, + const struct core_text *ct) +{ + struct paravirt_patch_site *p; + + for (p =3D start; p < end; p++) + patch_call(p->instr, ct); +} + +static __init_or_module void +callthunks_setup(struct callthunk_sites *cs, const struct core_text *ct) +{ + prdbg("Patching call sites %s\n", ct->name); + patch_call_sites(cs->call_start, cs->call_end, ct); + patch_paravirt_call_sites(cs->pv_start, cs->pv_end, ct); + prdbg("Patching call sites done%s\n", ct->name); +} + +void __init callthunks_patch_builtin_calls(void) +{ + struct callthunk_sites cs =3D { + .call_start =3D __call_sites, + .call_end =3D __call_sites_end, + .pv_start =3D __parainstructions, + .pv_end =3D __parainstructions_end + }; + + if (!cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) + return; + + pr_info("Setting up call depth tracking\n"); + mutex_lock(&text_mutex); + callthunks_setup(&cs, &builtin_coretext); + thunks_initialized =3D true; + mutex_unlock(&text_mutex); +} --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -370,6 +370,7 @@ SYM_CODE_END(secondary_startup_64) * start_secondary() via .Ljump_to_C_code. */ SYM_CODE_START(start_cpu0) + ANNOTATE_NOENDBR UNWIND_HINT_EMPTY movq initial_stack(%rip), %rsp jmp .Ljump_to_C_code --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -41,6 +41,7 @@ .text .align PAGE_SIZE .code64 +SYM_CODE_START_NOALIGN(relocate_range) SYM_CODE_START_NOALIGN(relocate_kernel) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR @@ -312,5 +313,5 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages) int3 SYM_CODE_END(swap_pages) =20 - .globl kexec_control_code_size -.set kexec_control_code_size, . - relocate_kernel + .skip KEXEC_CONTROL_CODE_MAX_SIZE - (. - relocate_kernel), 0xcc +SYM_CODE_END(relocate_range); --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -501,11 +501,3 @@ INIT_PER_CPU(irq_stack_backing_store); #endif =20 #endif /* CONFIG_X86_64 */ - -#ifdef CONFIG_KEXEC_CORE -#include - -. =3D ASSERT(kexec_control_code_size <=3D KEXEC_CONTROL_CODE_MAX_SIZE, - "kexec control code size is too big"); -#endif - From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18FBDC38145 for ; Fri, 2 Sep 2022 14:31:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235498AbiIBObl (ORCPT ); Fri, 2 Sep 2022 10:31:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236922AbiIBO1u (ORCPT ); Fri, 2 Sep 2022 10:27:50 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47EB71581AB for ; Fri, 2 Sep 2022 06:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=PsjEG4knouG+JCTeWWUmbwqU2xySdGfCmOMQNnMDy2Q=; b=Qp0mGFPB5LdN/pvzWYro8eZiFO XbgukPSiOl0q748pBt3ft2uufZuUlM1flnG0aeMXGUW7ytO5p3fzrGUthTKCFoffSLynfCCdge9Ic BV79/DDm+L7JuJ2BRknNrkveODlgdGCOvxcpQwiGwG3oUG8kLtx9nyswqJolT+VVNJhKchML++ZpZ Ne2Xgteamj/DhtICsa+h8tD4BdCaMLXMxrokswI0ui9Gf6fPT3BLXSODHFAxnIXF4N6w8GtSSWfWq V5KetR9+Ze/YgG3A3B3qEvonpxxz5ZzgnNK1i9u2NXo+SL8bhG8lABpAWkyAJW/IWu+NW7VWuZws3 EAjLMQTQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77P-0074UO-8H; Fri, 02 Sep 2022 13:53:59 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3C9C7302D59; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 5DA2C2B8EFB54; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.029167377@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:10 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 45/59] x86/modules: Add call patching References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner As for the builtins create call thunks and patch the call sites to call the thunk on Intel SKL CPUs for retbleed mitigation. Note, that module init functions are ignored for sake of simplicity because loading modules is not something which is done in high frequent loops and the attacker has not really a handle on when this happens in order to launch a matching attack. The depth tracking will still work for calls into the builtins and because the call is not accounted it will underflow faster and overstuff, but that's mitigated by the saturating counter and the side effect is only temporary. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/alternative.h | 5 +++++ arch/x86/kernel/callthunks.c | 19 +++++++++++++++++++ arch/x86/kernel/module.c | 20 +++++++++++++++++++- 3 files changed, 43 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -89,8 +89,13 @@ struct callthunk_sites { =20 #ifdef CONFIG_CALL_THUNKS extern void callthunks_patch_builtin_calls(void); +extern void callthunks_patch_module_calls(struct callthunk_sites *sites, + struct module *mod); #else static __always_inline void callthunks_patch_builtin_calls(void) {} +static __always_inline void +callthunks_patch_module_calls(struct callthunk_sites *sites, + struct module *mod) {} #endif =20 #ifdef CONFIG_SMP --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -249,3 +249,22 @@ void __init callthunks_patch_builtin_cal thunks_initialized =3D true; mutex_unlock(&text_mutex); } + +#ifdef CONFIG_MODULES +void noinline callthunks_patch_module_calls(struct callthunk_sites *cs, + struct module *mod) +{ + struct core_text ct =3D { + .base =3D (unsigned long)mod->core_layout.base, + .end =3D (unsigned long)mod->core_layout.base + mod->core_layout.size, + .name =3D mod->name, + }; + + if (!thunks_initialized) + return; + + mutex_lock(&text_mutex); + callthunks_setup(cs, &ct); + mutex_unlock(&text_mutex); +} +#endif /* CONFIG_MODULES */ --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -254,7 +254,8 @@ int module_finalize(const Elf_Ehdr *hdr, { const Elf_Shdr *s, *text =3D NULL, *alt =3D NULL, *locks =3D NULL, *para =3D NULL, *orc =3D NULL, *orc_ip =3D NULL, - *retpolines =3D NULL, *returns =3D NULL, *ibt_endbr =3D NULL; + *retpolines =3D NULL, *returns =3D NULL, *ibt_endbr =3D NULL, + *calls =3D NULL; char *secstrings =3D (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; =20 for (s =3D sechdrs; s < sechdrs + hdr->e_shnum; s++) { @@ -274,6 +275,8 @@ int module_finalize(const Elf_Ehdr *hdr, retpolines =3D s; if (!strcmp(".return_sites", secstrings + s->sh_name)) returns =3D s; + if (!strcmp(".call_sites", secstrings + s->sh_name)) + calls =3D s; if (!strcmp(".ibt_endbr_seal", secstrings + s->sh_name)) ibt_endbr =3D s; } @@ -299,6 +302,21 @@ int module_finalize(const Elf_Ehdr *hdr, void *aseg =3D (void *)alt->sh_addr; apply_alternatives(aseg, aseg + alt->sh_size); } + if (calls || para) { + struct callthunk_sites cs =3D {}; + + if (calls) { + cs.call_start =3D (void *)calls->sh_addr; + cs.call_end =3D (void *)calls->sh_addr + calls->sh_size; + } + + if (para) { + cs.pv_start =3D (void *)para->sh_addr; + cs.pv_end =3D (void *)para->sh_addr + para->sh_size; + } + + callthunks_patch_module_calls(&cs, me); + } if (ibt_endbr) { void *iseg =3D (void *)ibt_endbr->sh_addr; apply_ibt_endbr(iseg, iseg + ibt_endbr->sh_size); From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5098FC38145 for ; Fri, 2 Sep 2022 14:30:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236553AbiIBOaK (ORCPT ); Fri, 2 Sep 2022 10:30:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236073AbiIBO1g (ORCPT ); Fri, 2 Sep 2022 10:27:36 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F0E915819A for ; Fri, 2 Sep 2022 06:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=r5OPPEtUvAvLyUEDLHTlEmYZN5yu2Cagrk4lN594PZw=; b=CNKtqKEmTRaxaCbcPO/1CnSG5I 5QjZRVGU8i3S5h81rqix3d3g7X5g9GYEeztNMoO2AmNHzFxL+PtEJy+8M+7Hrd5VJQaqegkL4uL7f zVtjbVs4mKr9d+gWNodwDZLGceZZ9KU7zCzbO6gswvNIJBGVbrE7wrDwI8uLHE973x/19NeloiCW3 coPEeGHcT2NlP24lx2sXDb1GafHl7lUflXmTBQvWZOllHeo0udvM546M2Kfy7XaFvdhSDMh7N6j+6 zKWqBUC3Wkn1Ypv548p99sQPx6MS6r0D6QHJ5a2Gk8SdH2NRmEO1+C7CwKozktq1bdfjfhwxaWQX4 Z2sEebLg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-0074UF-EX; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3CF49302D72; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 650602B8EFB56; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.135184445@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:11 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 46/59] x86/returnthunk: Allow different return thunks References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra In preparation for call depth tracking on Intel SKL CPUs, make it possible to patch in a SKL specific return thunk. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/nospec-branch.h | 6 ++++++ arch/x86/kernel/alternative.c | 19 ++++++++++++++----- arch/x86/kernel/ftrace.c | 2 +- arch/x86/kernel/static_call.c | 2 +- arch/x86/net/bpf_jit_comp.c | 2 +- 5 files changed, 23 insertions(+), 8 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -198,6 +198,12 @@ extern void __x86_return_thunk(void); extern void zen_untrain_ret(void); extern void entry_ibpb(void); =20 +#ifdef CONFIG_CALL_THUNKS +extern void (*x86_return_thunk)(void); +#else +#define x86_return_thunk (&__x86_return_thunk) +#endif + #ifdef CONFIG_RETPOLINE =20 #define GEN(reg) \ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -509,6 +509,11 @@ void __init_or_module noinline apply_ret } =20 #ifdef CONFIG_RETHUNK + +#ifdef CONFIG_CALL_THUNKS +void (*x86_return_thunk)(void) __ro_after_init =3D &__x86_return_thunk; +#endif + /* * Rewrite the compiler generated return thunk tail-calls. * @@ -524,14 +529,18 @@ static int patch_return(void *addr, stru { int i =3D 0; =20 - if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) - return -1; - - bytes[i++] =3D RET_INSN_OPCODE; + if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { + if (x86_return_thunk =3D=3D __x86_return_thunk) + return -1; + + i =3D JMP32_INSN_SIZE; + __text_gen_insn(bytes, JMP32_INSN_OPCODE, addr, x86_return_thunk, i); + } else { + bytes[i++] =3D RET_INSN_OPCODE; + } =20 for (; i < insn->length;) bytes[i++] =3D INT3_INSN_OPCODE; - return i; } =20 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -359,7 +359,7 @@ create_trampoline(struct ftrace_ops *ops =20 ip =3D trampoline + size; if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) - __text_gen_insn(ip, JMP32_INSN_OPCODE, ip, &__x86_return_thunk, JMP32_IN= SN_SIZE); + __text_gen_insn(ip, JMP32_INSN_OPCODE, ip, x86_return_thunk, JMP32_INSN_= SIZE); else memcpy(ip, retq, sizeof(retq)); =20 --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -52,7 +52,7 @@ static void __ref __static_call_transfor =20 case RET: if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) - code =3D text_gen_insn(JMP32_INSN_OPCODE, insn, &__x86_return_thunk); + code =3D text_gen_insn(JMP32_INSN_OPCODE, insn, x86_return_thunk); else code =3D &retinsn; break; --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -430,7 +430,7 @@ static void emit_return(u8 **pprog, u8 * u8 *prog =3D *pprog; =20 if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { - emit_jump(&prog, &__x86_return_thunk, ip); + emit_jump(&prog, x86_return_thunk, ip); } else { EMIT1(0xC3); /* ret */ if (IS_ENABLED(CONFIG_SLS)) From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28686C38145 for ; Fri, 2 Sep 2022 14:31:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236201AbiIBObV (ORCPT ); Fri, 2 Sep 2022 10:31:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236479AbiIBO1s (ORCPT ); Fri, 2 Sep 2022 10:27:48 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2BFE13C9DD for ; Fri, 2 Sep 2022 06:54:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=rYTe1/x3ip/EZzdOhJZl34EC0tPUaA+4E/+HaK9TnVE=; b=RCc1fHO8+bPCfSH0XqLyEcFGK/ 65wsDgOn0rj4GFO4KjoTdnn1xoJH9KPNJGPnAx9Q/QxlKbkjKJw+xGcFpmAJa9K62uMWqWBtzr+MW 4NTB7AZseLvu9pG9GPz1aFWb7CTVWgt2TuLK5qAiKmTX5smiwLufKjxhdLvUyO2i3FI7Mu63Oc/i4 r9DrYQB/SVaKWYFEdCnzCaWztD3Q3wXTAGUglYL7UjydC6nUZMjOZicixLsj4Al1atuHH53Noh1IO 5/SsP2uDado20jS0YFsgeYCvidrVWGBGj7FYtnm6ArBK0NWnIXl25bHmNLc1odXoaMf0Yts6ljhYl yHGyqP7g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-008g8b-0W; Fri, 02 Sep 2022 13:54:11 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3CF34302D63; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 6C5D72B8EFB57; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.234407437@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:12 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 47/59] x86/asm: Provide ALTERNATIVE_3 References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Fairly straight forward adaptation/extention of ALTERNATIVE_2. Required for call depth tracking. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/alternative.h | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -364,6 +364,7 @@ static inline int alternatives_text_rese #define old_len 141b-140b #define new_len1 144f-143f #define new_len2 145f-144f +#define new_len3 146f-145f =20 /* * gas compatible max based on the idea from: @@ -371,7 +372,8 @@ static inline int alternatives_text_rese * * The additional "-" is needed because gas uses a "true" value of -1. */ -#define alt_max_short(a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) +#define alt_max_2(a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) +#define alt_max_3(a, b, c) (alt_max_2(alt_max_2(a, b), c)) =20 =20 /* @@ -383,8 +385,8 @@ static inline int alternatives_text_rese 140: \oldinstr 141: - .skip -((alt_max_short(new_len1, new_len2) - (old_len)) > 0) * \ - (alt_max_short(new_len1, new_len2) - (old_len)),0x90 + .skip -((alt_max_2(new_len1, new_len2) - (old_len)) > 0) * \ + (alt_max_2(new_len1, new_len2) - (old_len)),0x90 142: =20 .pushsection .altinstructions,"a" @@ -401,6 +403,31 @@ static inline int alternatives_text_rese .popsection .endm =20 +.macro ALTERNATIVE_3 oldinstr, newinstr1, feature1, newinstr2, feature2, n= ewinstr3, feature3 +140: + \oldinstr +141: + .skip -((alt_max_3(new_len1, new_len2, new_len3) - (old_len)) > 0) * \ + (alt_max_3(new_len1, new_len2, new_len3) - (old_len)),0x90 +142: + + .pushsection .altinstructions,"a" + altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f + altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f + altinstruction_entry 140b,145f,\feature3,142b-140b,146f-145f + .popsection + + .pushsection .altinstr_replacement,"ax" +143: + \newinstr1 +144: + \newinstr2 +145: + \newinstr3 +146: + .popsection +.endm + /* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY(oldinstr, feature, newinstr_yes, newinstr_no) \ ALTERNATIVE_2 oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2977C38145 for ; Fri, 2 Sep 2022 14:32:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236311AbiIBOb7 (ORCPT ); Fri, 2 Sep 2022 10:31:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236370AbiIBO2X (ORCPT ); Fri, 2 Sep 2022 10:28:23 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB09D21253 for ; Fri, 2 Sep 2022 06:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=X/IUBCmCb5A2KwbDyGB9KbfI8YqN9o2Z0aQEZCxibuA=; b=jrw6PCH0I5uBl7UVpaf716JRxL tsQ5J2L24X29M3hhfN5xSI1bP7+htqgeCn9X9w4vCk0fCyH03MSYxoRUbh6ik3aspFuRp6iAkr92y xSMVKQ+f38M+sUtyVUT+RurLpxsNDjc1J2xtO8p4v577SPScqj2Ws3H0H413CDkimfBtL5rQTBui1 ZG2wpdhz94RRIUsoIuj6mSxdADFy0Z9oFoFjzaN6TFu1J06rf+/q6WrXXQPK9pZ6ytmebza+KUGC4 /Go5lpC/aDn4XzVDs4uo4xx/AMasvclfQgz1oRofT4JdOHl0R81MBEtwcHEJ+KEqP0pwJ2oBjd6XW uM5G8JCw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-0074UI-Fj; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3DD3B302D92; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 743862B8EFB58; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.336829336@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:13 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 48/59] x86/retbleed: Add SKL return thunk References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner To address the Intel SKL RSB underflow issue in software it's required to do call depth tracking. Provide a return thunk for call depth tracking on Intel SKL CPUs. The tracking does not use a counter. It uses uses arithmetic shift right on call entry and logical shift left on return. The depth tracking variable is initialized to 0x8000.... when the call depth is zero. The arithmetic shift right sign extends the MSB and saturates after the 12th call. The shift count is 5 so the tracking covers 12 nested calls. On return the variable is shifted left logically so it becomes zero again. CALL RET 0: 0x8000000000000000 0x0000000000000000 1: 0xfc00000000000000 0xf000000000000000 ... 11: 0xfffffffffffffff8 0xfffffffffffffc00 12: 0xffffffffffffffff 0xffffffffffffffe0 After a return buffer fill the depth is credited 12 calls before the next stuffing has to take place. There is a inaccuracy for situations like this: 10 calls 5 returns 3 calls 4 returns 3 calls .... The shift count might cause this to be off by one in either direction, but there is still a cushion vs. the RSB depth. The algorithm does not claim to be perfect, but it should obfuscate the problem enough to make exploitation extremly difficult. The theory behind this is: RSB is a stack with depth 16 which is filled on every call. On the return path speculation "pops" entries to speculate down the call chain. Once the speculative RSB is empty it switches to other predictors, e.g. the Branch History Buffer, which can be mistrained by user space and misguide the speculation path to a gadget. Call depth tracking is designed to break this speculation path by stuffing speculation trap calls into the RSB which are never getting a corresponding return executed. This stalls the prediction path until it gets resteered, The assumption is that stuffing at the 12th return is sufficient to break the speculation before it hits the underflow and the fallback to the other predictors. Testing confirms that it works. Johannes, one of the retbleed researchers. tried to attack this approach but failed. There is obviously no scientific proof that this will withstand future research progress, but all we can do right now is to speculate about it. The SAR/SHL usage was suggested by Andi Kleen. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/entry/entry_64.S | 10 +- arch/x86/include/asm/current.h | 3=20 arch/x86/include/asm/nospec-branch.h | 118 ++++++++++++++++++++++++++++++= +++-- arch/x86/kernel/asm-offsets.c | 3=20 arch/x86/kvm/svm/vmenter.S | 1=20 arch/x86/lib/retpoline.S | 31 +++++++++ 6 files changed, 156 insertions(+), 10 deletions(-) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -288,6 +288,7 @@ SYM_FUNC_END(__switch_to_asm) SYM_CODE_START_NOALIGN(ret_from_fork) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // copy_thread + CALL_DEPTH_ACCOUNT movq %rax, %rdi call schedule_tail /* rdi: 'prev' task parameter */ =20 @@ -332,7 +333,7 @@ SYM_CODE_START(xen_error_entry) UNWIND_HINT_FUNC PUSH_AND_CLEAR_REGS save_ret=3D1 ENCODE_FRAME_POINTER 8 - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL RET SYM_CODE_END(xen_error_entry) =20 @@ -977,7 +978,7 @@ SYM_CODE_START(paranoid_entry) * CR3 above, keep the old value in a callee saved register. */ IBRS_ENTER save_reg=3D%r15 - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL =20 RET SYM_CODE_END(paranoid_entry) @@ -1062,7 +1063,7 @@ SYM_CODE_START(error_entry) /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax IBRS_ENTER - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL =20 leaq 8(%rsp), %rdi /* arg0 =3D pt_regs pointer */ /* Put us onto the real thread stack. */ @@ -1097,6 +1098,7 @@ SYM_CODE_START(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + CALL_DEPTH_ACCOUNT leaq 8(%rsp), %rax /* return pt_regs pointer */ ANNOTATE_UNRET_END RET @@ -1115,7 +1117,7 @@ SYM_CODE_START(error_entry) FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax IBRS_ENTER - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL =20 /* * Pretend that the exception came from user mode: set up pt_regs --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -17,6 +17,9 @@ struct pcpu_hot { struct task_struct *current_task; int preempt_count; int cpu_number; +#ifdef CONFIG_CALL_DEPTH_TRACKING + u64 call_depth; +#endif unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -12,8 +12,80 @@ #include #include #include +#include =20 -#define RETPOLINE_THUNK_SIZE 32 +/* + * Call depth tracking for Intel SKL CPUs to address the RSB underflow + * issue in software. + * + * The tracking does not use a counter. It uses uses arithmetic shift + * right on call entry and logical shift left on return. + * + * The depth tracking variable is initialized to 0x8000.... when the call + * depth is zero. The arithmetic shift right sign extends the MSB and + * saturates after the 12th call. The shift count is 5 for both directions + * so the tracking covers 12 nested calls. + * + * Call + * 0: 0x8000000000000000 0x0000000000000000 + * 1: 0xfc00000000000000 0xf000000000000000 + * ... + * 11: 0xfffffffffffffff8 0xfffffffffffffc00 + * 12: 0xffffffffffffffff 0xffffffffffffffe0 + * + * After a return buffer fill the depth is credited 12 calls before the + * next stuffing has to take place. + * + * There is a inaccuracy for situations like this: + * + * 10 calls + * 5 returns + * 3 calls + * 4 returns + * 3 calls + * .... + * + * The shift count might cause this to be off by one in either direction, + * but there is still a cushion vs. the RSB depth. The algorithm does not + * claim to be perfect and it can be speculated around by the CPU, but it + * is considered that it obfuscates the problem enough to make exploitation + * extremly difficult. + */ +#define RET_DEPTH_SHIFT 5 +#define RSB_RET_STUFF_LOOPS 16 +#define RET_DEPTH_INIT 0x8000000000000000ULL +#define RET_DEPTH_INIT_FROM_CALL 0xfc00000000000000ULL +#define RET_DEPTH_CREDIT 0xffffffffffffffffULL + +#ifdef CONFIG_CALL_DEPTH_TRACKING +#define CREDIT_CALL_DEPTH \ + movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); + +#define ASM_CREDIT_CALL_DEPTH \ + movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); + +#define RESET_CALL_DEPTH \ + mov $0x80, %rax; \ + shl $56, %rax; \ + movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); + +#define RESET_CALL_DEPTH_FROM_CALL \ + mov $0xfc, %rax; \ + shl $56, %rax; \ + movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); + +#define INCREMENT_CALL_DEPTH \ + sarq $5, %gs:pcpu_hot + X86_call_depth; + +#define ASM_INCREMENT_CALL_DEPTH \ + sarq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth); + +#else +#define CREDIT_CALL_DEPTH +#define RESET_CALL_DEPTH +#define INCREMENT_CALL_DEPTH +#define RESET_CALL_DEPTH_FROM_CALL +#endif =20 /* * Fill the CPU return stack buffer. @@ -32,6 +104,7 @@ * from C via asm(".include ") but let's not go there. */ =20 +#define RETPOLINE_THUNK_SIZE 32 #define RSB_CLEAR_LOOPS 32 /* To forcibly overwrite all entries */ =20 /* @@ -60,7 +133,8 @@ dec reg; \ jnz 771b; \ /* barrier for jnz misprediction */ \ - lfence; + lfence; \ + ASM_CREDIT_CALL_DEPTH #else /* * i386 doesn't unconditionally have LFENCE, as such it can't @@ -185,11 +259,32 @@ * where we have a stack but before any RET instruction. */ .macro UNTRAIN_RET -#if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) +#if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ + defined(CONFIG_X86_FEATURE_CALL_DEPTH) ANNOTATE_UNRET_END - ALTERNATIVE_2 "", \ - CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ - "call entry_ibpb", X86_FEATURE_ENTRY_IBPB + ALTERNATIVE_3 "", \ + CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ + "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ + __stringify(RESET_CALL_DEPTH), X86_FEATURE_CALL_DEPTH +#endif +.endm + +.macro UNTRAIN_RET_FROM_CALL +#if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ + defined(CONFIG_X86_FEATURE_CALL_DEPTH) + ANNOTATE_UNRET_END + ALTERNATIVE_3 "", \ + CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ + "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ + __stringify(RESET_CALL_DEPTH_FROM_CALL), X86_FEATURE_CALL_DEPTH +#endif +.endm + + +.macro CALL_DEPTH_ACCOUNT +#ifdef CONFIG_CALL_DEPTH_TRACKING + ALTERNATIVE "", \ + __stringify(ASM_INCREMENT_CALL_DEPTH), X86_FEATURE_CALL_DEPTH #endif .endm =20 @@ -214,6 +309,17 @@ extern void (*x86_return_thunk)(void); #define x86_return_thunk (&__x86_return_thunk) #endif =20 +#ifdef CONFIG_CALL_DEPTH_TRACKING +extern void __x86_return_skl(void); + +static inline void x86_set_skl_return_thunk(void) +{ + x86_return_thunk =3D &__x86_return_skl; +} +#else +static inline void x86_set_skl_return_thunk(void) {} +#endif + #ifdef CONFIG_RETPOLINE =20 #define GEN(reg) \ --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -110,6 +110,9 @@ static void __used common(void) OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); =20 OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); +#ifdef CONFIG_CALL_DEPTH_TRACKING + OFFSET(X86_call_depth, pcpu_hot, call_depth); +#endif =20 if (IS_ENABLED(CONFIG_KVM_INTEL)) { BLANK(); --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include #include +#include #include #include #include --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -5,9 +5,11 @@ #include #include #include +#include #include #include #include +#include #include =20 .section .text.__x86.indirect_thunk @@ -140,3 +142,32 @@ SYM_FUNC_END(zen_untrain_ret) EXPORT_SYMBOL(__x86_return_thunk) =20 #endif /* CONFIG_RETHUNK */ + +#ifdef CONFIG_CALL_DEPTH_TRACKING + + .align 64 +SYM_FUNC_START(__x86_return_skl) + ANNOTATE_NOENDBR + /* Keep the hotpath in a 16byte I-fetch */ + shlq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth) + jz 1f + ANNOTATE_UNRET_SAFE + ret + int3 +1: + .rept 16 + ANNOTATE_INTRA_FUNCTION_CALL + call 2f + int3 +2: + .endr + add $(8*16), %rsp + + CREDIT_CALL_DEPTH + + ANNOTATE_UNRET_SAFE + ret + int3 +SYM_FUNC_END(__x86_return_skl) + +#endif /* CONFIG_CALL_DEPTH_TRACKING */ From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A3CEECAAD5 for ; Fri, 2 Sep 2022 14:32:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236220AbiIBOcN (ORCPT ); Fri, 2 Sep 2022 10:32:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237779AbiIBO2o (ORCPT ); Fri, 2 Sep 2022 10:28:44 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 054EBEA156 for ; Fri, 2 Sep 2022 06:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=kMUuDcFUgc9UaxyV3UbtUWkODc8RvW8xK23tMclc5FI=; b=laeXU9GZSyDuEsTQrxT4bU8vdF 8Pde8yYiX75aOVnSJKhrMMW0L/w4FJ0xdPRXK+TQdy4P5X75/AlSrsuaFfJChzEAJP2fyUNIVVSMX rCoXOtrwtk/SVOoUMHVbT+6RSAB6YwR/yE6GaB1soZ4Zn2xguy0u3zB4JKDiKIYbOrAgddAX4ZZ9X 0UfLM7tZyLHSJIQryrUAE90yV/+KqH6D1KxAFU1QrEnl32TzCZ4Jq1uPSaXxpu1Rkrrsz7jAENJqw exHUYDoQrtD+WdrjArABk2Skb3Zr4+w7gNvxqyFVvHmhfkUBaEvjQx6a97iqNsUkIk1sOpcfk2yux dB9SWh3Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-0074UH-Fl; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3E000302D9D; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 7D8652B8EFB55; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.440125229@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:14 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 49/59] x86/retpoline: Add SKL retthunk retpolines References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Ensure that retpolines do the proper call accounting so that the return accounting works correctly. Specifically; retpolines are used to replace both 'jmp *%reg' and 'call *%reg', however these two cases do not have the same accounting requirements. Therefore split things up and provide two different retpoline arrays for SKL. The 'jmp *%reg' case needs no accounting, the __x86_indirect_jump_thunk_array[] covers this. The retpoline is changed to not use the return thunk; it's a simple call;ret construct. [ strictly speaking it should do: andq $(~0x1f), PER_CPU_VAR(__x86_call_depth) but we can argue this can be covered by the fuzz we already have in the accounting depth (12) vs the RSB depth (16) ] The 'call *%reg' case does need accounting, the __x86_indirect_call_thunk_array[] covers this. Again, this retpoline avoids the use of the return-thunk, in this case to avoid double accounting. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/nospec-branch.h | 12 +++++ arch/x86/kernel/alternative.c | 59 +++++++++++++++++++++++++++-- arch/x86/lib/retpoline.S | 71 ++++++++++++++++++++++++++++++= +---- arch/x86/net/bpf_jit_comp.c | 5 +- 4 files changed, 135 insertions(+), 12 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -288,6 +288,8 @@ =20 typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE]; extern retpoline_thunk_t __x86_indirect_thunk_array[]; +extern retpoline_thunk_t __x86_indirect_call_thunk_array[]; +extern retpoline_thunk_t __x86_indirect_jump_thunk_array[]; =20 extern void __x86_return_thunk(void); extern void zen_untrain_ret(void); @@ -317,6 +319,16 @@ static inline void x86_set_skl_return_th #include #undef GEN =20 +#define GEN(reg) \ + extern retpoline_thunk_t __x86_indirect_call_thunk_ ## reg; +#include +#undef GEN + +#define GEN(reg) \ + extern retpoline_thunk_t __x86_indirect_jump_thunk_ ## reg; +#include +#undef GEN + #ifdef CONFIG_X86_64 =20 /* --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -377,6 +377,56 @@ static int emit_indirect(int op, int reg return i; } =20 +static inline bool is_jcc32(struct insn *insn) +{ + /* Jcc.d32 second opcode byte is in the range: 0x80-0x8f */ + return insn->opcode.bytes[0] =3D=3D 0x0f && (insn->opcode.bytes[1] & 0xf0= ) =3D=3D 0x80; +} + +static int emit_call_track_retpoline(void *addr, struct insn *insn, int re= g, u8 *bytes) +{ + u8 op =3D insn->opcode.bytes[0]; + int i =3D 0; + + /* + * Clang does 'weird' Jcc __x86_indirect_thunk_r11 conditional + * tail-calls. Deal with them. + */ + if (is_jcc32(insn)) { + bytes[i++] =3D op; + op =3D insn->opcode.bytes[1]; + goto clang_jcc; + } + + if (insn->length =3D=3D 6) + bytes[i++] =3D 0x2e; /* CS-prefix */ + + switch (op) { + case CALL_INSN_OPCODE: + __text_gen_insn(bytes+i, op, addr+i, + __x86_indirect_call_thunk_array[reg], + CALL_INSN_SIZE); + i +=3D CALL_INSN_SIZE; + break; + + case JMP32_INSN_OPCODE: +clang_jcc: + __text_gen_insn(bytes+i, op, addr+i, + __x86_indirect_jump_thunk_array[reg], + JMP32_INSN_SIZE); + i +=3D JMP32_INSN_SIZE; + break; + + default: + WARN("%pS %px %*ph\n", addr, addr, 6, addr); + return -1; + } + + WARN_ON_ONCE(i !=3D insn->length); + + return i; +} + /* * Rewrite the compiler generated retpoline thunk calls. * @@ -409,8 +459,12 @@ static int patch_retpoline(void *addr, s BUG_ON(reg =3D=3D 4); =20 if (cpu_feature_enabled(X86_FEATURE_RETPOLINE) && - !cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) + !cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { + if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) + return emit_call_track_retpoline(addr, insn, reg, bytes); + return -1; + } =20 op =3D insn->opcode.bytes[0]; =20 @@ -427,8 +481,7 @@ static int patch_retpoline(void *addr, s * [ NOP ] * 1: */ - /* Jcc.d32 second opcode byte is in the range: 0x80-0x8f */ - if (op =3D=3D 0x0f && (insn->opcode.bytes[1] & 0xf0) =3D=3D 0x80) { + if (is_jcc32(insn)) { cc =3D insn->opcode.bytes[1] & 0xf; cc ^=3D 1; /* invert condition */ =20 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -14,17 +14,18 @@ =20 .section .text.__x86.indirect_thunk =20 -.macro RETPOLINE reg + +.macro POLINE reg ANNOTATE_INTRA_FUNCTION_CALL call .Ldo_rop_\@ -.Lspec_trap_\@: - UNWIND_HINT_EMPTY - pause - lfence - jmp .Lspec_trap_\@ + int3 .Ldo_rop_\@: mov %\reg, (%_ASM_SP) UNWIND_HINT_FUNC +.endm + +.macro RETPOLINE reg + POLINE \reg RET .endm =20 @@ -54,7 +55,6 @@ SYM_INNER_LABEL(__x86_indirect_thunk_\re */ =20 #define __EXPORT_THUNK(sym) _ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym) -#define EXPORT_THUNK(reg) __EXPORT_THUNK(__x86_indirect_thunk_ ## reg) =20 .align RETPOLINE_THUNK_SIZE SYM_CODE_START(__x86_indirect_thunk_array) @@ -66,10 +66,65 @@ SYM_CODE_START(__x86_indirect_thunk_arra .align RETPOLINE_THUNK_SIZE SYM_CODE_END(__x86_indirect_thunk_array) =20 -#define GEN(reg) EXPORT_THUNK(reg) +#define GEN(reg) __EXPORT_THUNK(__x86_indirect_thunk_ ## reg) +#include +#undef GEN + +#ifdef CONFIG_CALL_DEPTH_TRACKING +.macro CALL_THUNK reg + .align RETPOLINE_THUNK_SIZE + +SYM_INNER_LABEL(__x86_indirect_call_thunk_\reg, SYM_L_GLOBAL) + UNWIND_HINT_EMPTY + ANNOTATE_NOENDBR + + CALL_DEPTH_ACCOUNT + POLINE \reg + ANNOTATE_UNRET_SAFE + ret + int3 +.endm + + .align RETPOLINE_THUNK_SIZE +SYM_CODE_START(__x86_indirect_call_thunk_array) + +#define GEN(reg) CALL_THUNK reg +#include +#undef GEN + + .align RETPOLINE_THUNK_SIZE +SYM_CODE_END(__x86_indirect_call_thunk_array) + +#define GEN(reg) __EXPORT_THUNK(__x86_indirect_call_thunk_ ## reg) #include #undef GEN =20 +.macro JUMP_THUNK reg + .align RETPOLINE_THUNK_SIZE + +SYM_INNER_LABEL(__x86_indirect_jump_thunk_\reg, SYM_L_GLOBAL) + UNWIND_HINT_EMPTY + ANNOTATE_NOENDBR + POLINE \reg + ANNOTATE_UNRET_SAFE + ret + int3 +.endm + + .align RETPOLINE_THUNK_SIZE +SYM_CODE_START(__x86_indirect_jump_thunk_array) + +#define GEN(reg) JUMP_THUNK reg +#include +#undef GEN + + .align RETPOLINE_THUNK_SIZE +SYM_CODE_END(__x86_indirect_jump_thunk_array) + +#define GEN(reg) __EXPORT_THUNK(__x86_indirect_jump_thunk_ ## reg) +#include +#undef GEN +#endif /* * This function name is magical and is used by -mfunction-return=3Dthunk-= extern * for the compiler to generate JMPs to it. --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -417,7 +417,10 @@ static void emit_indirect_jump(u8 **ppro EMIT2(0xFF, 0xE0 + reg); } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) { OPTIMIZER_HIDE_VAR(reg); - emit_jump(&prog, &__x86_indirect_thunk_array[reg], ip); + if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) + emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg], ip); + else + emit_jump(&prog, &__x86_indirect_thunk_array[reg], ip); } else { EMIT2(0xFF, 0xE0 + reg); } From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3F16ECAAD5 for ; Fri, 2 Sep 2022 14:29:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236210AbiIBO3p (ORCPT ); Fri, 2 Sep 2022 10:29:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236805AbiIBO1c (ORCPT ); Fri, 2 Sep 2022 10:27:32 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4F32168A3B for ; Fri, 2 Sep 2022 06:54:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=tmkbhGubwHT7U0oeakCgE2xWz6Q0qYpLFWybSMdPCks=; b=QF5DWXiNDuHWPE4zS5CJJsIWAA FbPDAeX4b0xcpmmizBpTeskPrSGd85F21nMgg/J8i38hJiUEnma9kNBHjxAGati1yergjdyT1O4w7 +B/mJ2JyN295QFTqm5/aI4nB05F6oxN0rwULvVN5Grs0d3L3ue7RSNU492zAC4zowQ6WUBvkcokbT Ew1GtA1flEWlZBwRVbAmPNabxm3rYgFAq/tzf5JXI3ahODkU64QK62TGhhZto9Xj4awVYBr2+x0Q/ 0X4//FGDU0gyS4xMvtJnWsNjN1VCErYADeCj6cAefyVK98DfXQcBpTLWLSiIwWz36gJGr+WlRx+bd bP4gMijg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-0074UE-EK; Fri, 02 Sep 2022 13:53:59 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3DD17302D76; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8885A2B8EFB5C; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.545005716@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:15 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 50/59] x86/retbleed: Add SKL call thunk References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Add the actual SKL call thunk for call depth accounting. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/callthunks.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -7,6 +7,7 @@ #include =20 #include +#include #include #include #include @@ -55,7 +56,21 @@ static const struct core_text builtin_co .name =3D "builtin", }; =20 -static struct thunk_desc callthunk_desc __ro_after_init; +asm ( + ".pushsection .rodata \n" + ".global skl_call_thunk_template \n" + "skl_call_thunk_template: \n" + __stringify(INCREMENT_CALL_DEPTH)" \n" + ".global skl_call_thunk_tail \n" + "skl_call_thunk_tail: \n" + ".popsection \n" +); + +extern u8 skl_call_thunk_template[]; +extern u8 skl_call_thunk_tail[]; + +#define SKL_TMPL_SIZE \ + ((unsigned int)(skl_call_thunk_tail - skl_call_thunk_template)) =20 extern void error_entry(void); extern void xen_error_entry(void); @@ -157,11 +172,11 @@ static const u8 nops[] =3D { =20 static __init_or_module void *patch_dest(void *dest, bool direct) { - unsigned int tsize =3D callthunk_desc.template_size; + unsigned int tsize =3D SKL_TMPL_SIZE; u8 *pad =3D dest - tsize; =20 /* Already patched? */ - if (!bcmp(pad, callthunk_desc.template, tsize)) + if (!bcmp(pad, skl_call_thunk_template, tsize)) return pad; =20 /* Ensure there are nops */ @@ -171,9 +186,9 @@ static __init_or_module void *patch_dest } =20 if (direct) - memcpy(pad, callthunk_desc.template, tsize); + memcpy(pad, skl_call_thunk_template, tsize); else - text_poke_copy_locked(pad, callthunk_desc.template, tsize, true); + text_poke_copy_locked(pad, skl_call_thunk_template, tsize, true); return pad; } From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C551ECAAD5 for ; Fri, 2 Sep 2022 14:28:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237261AbiIBO16 (ORCPT ); Fri, 2 Sep 2022 10:27:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237103AbiIBO1J (ORCPT ); Fri, 2 Sep 2022 10:27:09 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F26281670C8 for ; Fri, 2 Sep 2022 06:54:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=+OpChG5m5Za8mkIkCncS6zKt3a7LKRkg2fN4td+8QnM=; b=n0F0DTHOi4LHVz/REwo9QJ8fpu w0o5fQZdO5sNVVYfSbbMeuX+/XV+iHi+AZEznKjX/ieWPCOfVDiA1UBJB6VA6pReBdohbodXgKBEf wO0yhA5CgARqdNDQL1Gs/keD8UPQ9VxNDU5BArm4qOWquMdSFBZeo2H1KPisPR8SrVtXXTN/4d2MV u0jAl4cJAfchrwpfHMgxOhwvkZjfNH1ndvZibtGLz0tbblA9xcAPwuHcYnTCTQ15cg9U6r3noLIL8 T7bbsPBP9cXKdJ/zL1MjPppdQ3DFNsOS894XRzl0T9kILiOKx3xfEkAh3jBJqBnIAG3eVfRALHSNz 5WoHn8vQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-0074UJ-H3; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3FC54302D9E; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 92AA92B8EFB5D; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.648986316@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:16 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 51/59] x86/calldepth: Add ret/call counting for debug References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Add a debuigfs mechanism to validate the accounting, e.g. vs. call/ret balance and to gather statistics about the stuffing to call ratio. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/nospec-branch.h | 36 +++++++++++++++++++++-- arch/x86/kernel/callthunks.c | 53 ++++++++++++++++++++++++++++++= +++++ arch/x86/lib/retpoline.S | 7 +++- 3 files changed, 91 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -57,6 +57,22 @@ #define RET_DEPTH_INIT_FROM_CALL 0xfc00000000000000ULL #define RET_DEPTH_CREDIT 0xffffffffffffffffULL =20 +#ifdef CONFIG_CALL_THUNKS_DEBUG +# define CALL_THUNKS_DEBUG_INC_CALLS \ + incq %gs:__x86_call_count; +# define CALL_THUNKS_DEBUG_INC_RETS \ + incq %gs:__x86_ret_count; +# define CALL_THUNKS_DEBUG_INC_STUFFS \ + incq %gs:__x86_stuffs_count; +# define CALL_THUNKS_DEBUG_INC_CTXSW \ + incq %gs:__x86_ctxsw_count; +#else +# define CALL_THUNKS_DEBUG_INC_CALLS +# define CALL_THUNKS_DEBUG_INC_RETS +# define CALL_THUNKS_DEBUG_INC_STUFFS +# define CALL_THUNKS_DEBUG_INC_CTXSW +#endif + #ifdef CONFIG_CALL_DEPTH_TRACKING #define CREDIT_CALL_DEPTH \ movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); @@ -72,18 +88,23 @@ #define RESET_CALL_DEPTH_FROM_CALL \ mov $0xfc, %rax; \ shl $56, %rax; \ - movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); + movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + CALL_THUNKS_DEBUG_INC_CALLS =20 #define INCREMENT_CALL_DEPTH \ - sarq $5, %gs:pcpu_hot + X86_call_depth; + sarq $5, %gs:pcpu_hot + X86_call_depth; \ + CALL_THUNKS_DEBUG_INC_CALLS =20 #define ASM_INCREMENT_CALL_DEPTH \ - sarq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth); + sarq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + CALL_THUNKS_DEBUG_INC_CALLS =20 #else #define CREDIT_CALL_DEPTH +#define ASM_CREDIT_CALL_DEPTH #define RESET_CALL_DEPTH #define INCREMENT_CALL_DEPTH +#define ASM_INCREMENT_CALL_DEPTH #define RESET_CALL_DEPTH_FROM_CALL #endif =20 @@ -134,7 +155,8 @@ jnz 771b; \ /* barrier for jnz misprediction */ \ lfence; \ - ASM_CREDIT_CALL_DEPTH + ASM_CREDIT_CALL_DEPTH \ + CALL_THUNKS_DEBUG_INC_CTXSW #else /* * i386 doesn't unconditionally have LFENCE, as such it can't @@ -319,6 +341,12 @@ static inline void x86_set_skl_return_th { x86_return_thunk =3D &__x86_return_skl; } +#ifdef CONFIG_CALL_THUNKS_DEBUG +DECLARE_PER_CPU(u64, __x86_call_count); +DECLARE_PER_CPU(u64, __x86_ret_count); +DECLARE_PER_CPU(u64, __x86_stuffs_count); +DECLARE_PER_CPU(u64, __x86_ctxsw_count); +#endif #else static inline void x86_set_skl_return_thunk(void) {} #endif --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -2,6 +2,7 @@ =20 #define pr_fmt(fmt) "callthunks: " fmt =20 +#include #include #include #include @@ -35,6 +36,15 @@ static int __init debug_thunks(char *str } __setup("debug-callthunks", debug_thunks); =20 +#ifdef CONFIG_CALL_THUNKS_DEBUG +DEFINE_PER_CPU(u64, __x86_call_count); +DEFINE_PER_CPU(u64, __x86_ret_count); +DEFINE_PER_CPU(u64, __x86_stuffs_count); +DEFINE_PER_CPU(u64, __x86_ctxsw_count); +EXPORT_SYMBOL_GPL(__x86_ctxsw_count); +EXPORT_SYMBOL_GPL(__x86_call_count); +#endif + extern s32 __call_sites[], __call_sites_end[]; =20 struct thunk_desc { @@ -283,3 +293,46 @@ void noinline callthunks_patch_module_ca mutex_unlock(&text_mutex); } #endif /* CONFIG_MODULES */ + +#if defined(CONFIG_CALL_THUNKS_DEBUG) && defined(CONFIG_DEBUG_FS) +static int callthunks_debug_show(struct seq_file *m, void *p) +{ + unsigned long cpu =3D (unsigned long)m->private; + + seq_printf(m, "C: %16llu R: %16llu S: %16llu X: %16llu\n,", + per_cpu(__x86_call_count, cpu), + per_cpu(__x86_ret_count, cpu), + per_cpu(__x86_stuffs_count, cpu), + per_cpu(__x86_ctxsw_count, cpu)); + return 0; +} + +static int callthunks_debug_open(struct inode *inode, struct file *file) +{ + return single_open(file, callthunks_debug_show, inode->i_private); +} + +static const struct file_operations dfs_ops =3D { + .open =3D callthunks_debug_open, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static int __init callthunks_debugfs_init(void) +{ + struct dentry *dir; + unsigned long cpu; + + dir =3D debugfs_create_dir("callthunks", NULL); + for_each_possible_cpu(cpu) { + void *arg =3D (void *)cpu; + char name [10]; + + sprintf(name, "cpu%lu", cpu); + debugfs_create_file(name, 0644, dir, arg, &dfs_ops); + } + return 0; +} +__initcall(callthunks_debugfs_init); +#endif --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -203,13 +203,18 @@ EXPORT_SYMBOL(__x86_return_thunk) .align 64 SYM_FUNC_START(__x86_return_skl) ANNOTATE_NOENDBR - /* Keep the hotpath in a 16byte I-fetch */ + /* + * Keep the hotpath in a 16byte I-fetch for the non-debug + * case. + */ + CALL_THUNKS_DEBUG_INC_RETS shlq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth) jz 1f ANNOTATE_UNRET_SAFE ret int3 1: + CALL_THUNKS_DEBUG_INC_STUFFS .rept 16 ANNOTATE_INTRA_FUNCTION_CALL call 2f From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02FFDC38145 for ; Fri, 2 Sep 2022 14:31:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230151AbiIBObz (ORCPT ); Fri, 2 Sep 2022 10:31:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236233AbiIBO2X (ORCPT ); Fri, 2 Sep 2022 10:28:23 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B26421581B0 for ; Fri, 2 Sep 2022 06:54:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=As36aOynrXCL6dT8N1wy2coxxgtaR4tVqpdPjpxHvf0=; b=kjTSLkd/P+fdyZjg399uKw+EGo CSWOkWq5DKnZWYHurFbojMrZv9hZj5KLLCU5y2YPn6kshTXLME4QRISfxYKO3u3Os15zHTzMFNJ1s SddqLTxjWgiaGda3p/tngBwPkZ0pr5d2J1QXQcb37Gkx7wuTcT0JITg+xvsKTciHrMJyo0hiA99CA fY3M1o8LNnyzbXxIf2c+pCufN9Kogi0KQ0mcESLbzSulhAMzjhE2LsMW1rsdxf1uYAIWkNhdtS0AM vtlHiSH+JXzYAugGvmBEq9SWDf+Xl7FfyWV5xGoIpWUJ33dHsUd4NP0Qa4ggo9Hu4uqLCsXPjlgtY VAlhCj4Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-008g8f-NU; Fri, 02 Sep 2022 13:54:02 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 469E4302DFA; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 9B0972B8EFB5E; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.751096802@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:17 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 52/59] static_call: Add call depth tracking support References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra When indirect calls are switched to direct calls then it has to be ensured that the call target is not the function, but the call thunk when call depth tracking is enabled. But static calls are available before call thunks have been set up. Ensure a second run through the static call patching code after call thunks have been created. When call thunks are not enabled this has no side effects. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/alternative.h | 5 +++++ arch/x86/kernel/callthunks.c | 18 ++++++++++++++++++ arch/x86/kernel/static_call.c | 1 + include/linux/static_call.h | 2 ++ kernel/static_call_inline.c | 23 ++++++++++++++++++----- 5 files changed, 44 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -91,11 +91,16 @@ struct callthunk_sites { extern void callthunks_patch_builtin_calls(void); extern void callthunks_patch_module_calls(struct callthunk_sites *sites, struct module *mod); +extern void *callthunks_translate_call_dest(void *dest); #else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline void callthunks_patch_module_calls(struct callthunk_sites *sites, struct module *mod) {} +static __always_inline void *callthunks_translate_call_dest(void *dest) +{ + return dest; +} #endif =20 #ifdef CONFIG_SMP --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -6,6 +6,7 @@ #include #include #include +#include =20 #include #include @@ -270,10 +271,27 @@ void __init callthunks_patch_builtin_cal pr_info("Setting up call depth tracking\n"); mutex_lock(&text_mutex); callthunks_setup(&cs, &builtin_coretext); + static_call_force_reinit(); thunks_initialized =3D true; mutex_unlock(&text_mutex); } =20 +void *callthunks_translate_call_dest(void *dest) +{ + void *target; + + lockdep_assert_held(&text_mutex); + + if (!thunks_initialized || skip_addr(dest)) + return dest; + + if (!is_coretext(NULL, dest)) + return dest; + + target =3D patch_dest(dest, false); + return target ? : dest; +} + #ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs, struct module *mod) --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -34,6 +34,7 @@ static void __ref __static_call_transfor =20 switch (type) { case CALL: + func =3D callthunks_translate_call_dest(func); code =3D text_gen_insn(CALL_INSN_OPCODE, insn, func); if (func =3D=3D &__static_call_return0) { emulate =3D code; --- a/include/linux/static_call.h +++ b/include/linux/static_call.h @@ -162,6 +162,8 @@ extern void arch_static_call_transform(v =20 extern int __init static_call_init(void); =20 +extern void static_call_force_reinit(void); + struct static_call_mod { struct static_call_mod *next; struct module *mod; /* for vmlinux, mod =3D=3D NULL */ --- a/kernel/static_call_inline.c +++ b/kernel/static_call_inline.c @@ -15,7 +15,18 @@ extern struct static_call_site __start_s extern struct static_call_tramp_key __start_static_call_tramp_key[], __stop_static_call_tramp_key[]; =20 -static bool static_call_initialized; +static int static_call_initialized; + +/* + * Must be called before early_initcall() to be effective. + */ +void static_call_force_reinit(void) +{ + if (WARN_ON_ONCE(!static_call_initialized)) + return; + + static_call_initialized++; +} =20 /* mutex to protect key modules/sites */ static DEFINE_MUTEX(static_call_mutex); @@ -475,7 +486,8 @@ int __init static_call_init(void) { int ret; =20 - if (static_call_initialized) + /* See static_call_force_reinit(). */ + if (static_call_initialized =3D=3D 1) return 0; =20 cpus_read_lock(); @@ -490,11 +502,12 @@ int __init static_call_init(void) BUG(); } =20 - static_call_initialized =3D true; - #ifdef CONFIG_MODULES - register_module_notifier(&static_call_module_nb); + if (!static_call_initialized) + register_module_notifier(&static_call_module_nb); #endif + + static_call_initialized =3D 1; return 0; } early_initcall(static_call_init); From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D975CC38145 for ; Fri, 2 Sep 2022 14:33:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236717AbiIBOd0 (ORCPT ); Fri, 2 Sep 2022 10:33:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236217AbiIBOaX (ORCPT ); Fri, 2 Sep 2022 10:30:23 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AABDD20F4B for ; Fri, 2 Sep 2022 06:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Mk4gyPud12Euk/9XIHAPXis5fefrr9FnQg5Sg88vnL0=; b=X7Th3/a7wdt17ojxWrABzi5Ae8 oaJGENC29iMmbnDD3sh8vciSWHTDv2S3CaixkPjaGmC8OFjWx8NmpOqDVD0tbLjtH12mbRFODkxng PQo1KpOw1+NUHQnIopLOq87lu91khcWU0khwjLFicn4aOgFEtgBlxPbOp4uBsrXdnYLS39sfc+8dS OtT7nzeilsw5T/nJ2Ip42YCzEUjMrYtHimjs0zmUx0pgSsVeMP/JHZQpxjuwxSa9uMNQ1ZyGwiYVK lsebNDTOjDbP7PMGbrg1grISQc38zNKmCNsaCIvRRlR0HUz85NIXbBtvAaGEMXJEmS8wyE3fE6Qe5 nwaLhoXQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77P-008g8l-Jq; Fri, 02 Sep 2022 13:54:03 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 451CC302DA9; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id A654D2B8EFB60; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.853460809@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:18 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 53/59] kallsyms: Take callthunks into account References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Since the pre-symbol function padding is an integral part of the symbol make kallsyms report it as part of the symbol by reporting it as sym-x instead of prev_sym+y. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- kernel/kallsyms.c | 45 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 5 deletions(-) --- a/kernel/kallsyms.c +++ b/kernel/kallsyms.c @@ -292,6 +292,12 @@ static unsigned long get_symbol_pos(unsi return low; } =20 +#ifdef CONFIG_FUNCTION_PADDING_BYTES +#define PADDING_BYTES CONFIG_FUNCTION_PADDING_BYTES +#else +#define PADDING_BYTES 0 +#endif + /* * Lookup an address but don't bother to find any names. */ @@ -299,13 +305,25 @@ int kallsyms_lookup_size_offset(unsigned unsigned long *offset) { char namebuf[KSYM_NAME_LEN]; + int ret; + + addr +=3D PADDING_BYTES; =20 if (is_ksym_addr(addr)) { get_symbol_pos(addr, symbolsize, offset); - return 1; + ret =3D 1; + goto found; } - return !!module_address_lookup(addr, symbolsize, offset, NULL, NULL, name= buf) || - !!__bpf_address_lookup(addr, symbolsize, offset, namebuf); + + ret =3D !!module_address_lookup(addr, symbolsize, offset, NULL, NULL, nam= ebuf); + if (!ret) { + ret =3D !!__bpf_address_lookup(addr, symbolsize, + offset, namebuf); + } +found: + if (ret && offset) + *offset -=3D PADDING_BYTES; + return ret; } =20 static const char *kallsyms_lookup_buildid(unsigned long addr, @@ -318,6 +336,8 @@ static const char *kallsyms_lookup_build namebuf[KSYM_NAME_LEN - 1] =3D 0; namebuf[0] =3D 0; =20 + addr +=3D PADDING_BYTES; + if (is_ksym_addr(addr)) { unsigned long pos; =20 @@ -347,6 +367,8 @@ static const char *kallsyms_lookup_build =20 found: cleanup_symbol_name(namebuf); + if (ret && offset) + *offset -=3D PADDING_BYTES; return ret; } =20 @@ -373,6 +395,8 @@ int lookup_symbol_name(unsigned long add symname[0] =3D '\0'; symname[KSYM_NAME_LEN - 1] =3D '\0'; =20 + addr +=3D PADDING_BYTES; + if (is_ksym_addr(addr)) { unsigned long pos; =20 @@ -400,6 +424,8 @@ int lookup_symbol_attrs(unsigned long ad name[0] =3D '\0'; name[KSYM_NAME_LEN - 1] =3D '\0'; =20 + addr +=3D PADDING_BYTES; + if (is_ksym_addr(addr)) { unsigned long pos; =20 @@ -416,6 +442,8 @@ int lookup_symbol_attrs(unsigned long ad return res; =20 found: + if (offset) + *offset -=3D PADDING_BYTES; cleanup_symbol_name(name); return 0; } @@ -441,8 +469,15 @@ static int __sprint_symbol(char *buffer, len =3D strlen(buffer); offset -=3D symbol_offset; =20 - if (add_offset) - len +=3D sprintf(buffer + len, "+%#lx/%#lx", offset, size); + if (add_offset) { + char s =3D '+'; + + if ((long)offset < 0) { + s =3D '-'; + offset =3D 0UL - offset; + } + len +=3D sprintf(buffer + len, "%c%#lx/%#lx", s, offset, size); + } =20 if (modname) { len +=3D sprintf(buffer + len, " [%s", modname); From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3339FC38145 for ; Fri, 2 Sep 2022 14:32:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236074AbiIBOcp (ORCPT ); Fri, 2 Sep 2022 10:32:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237943AbiIBO2r (ORCPT ); Fri, 2 Sep 2022 10:28:47 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32601140CD for ; Fri, 2 Sep 2022 06:54:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=WG6G46Ua3WkaooMqyaY9G1V2PL1FK78RAajcxW19yWA=; b=Gv0FHzx9E3AunEp04b4luzhZS7 EVld5MobGXBeGMwpBn6R096leV+mMQ1IY5CXZuSN8Jf83TmSYR1D5ckkv0MfLguf+gC4r1to0aoLp cKYqBF0kOF7hA7isc1wLsFjU+t3HorPPih6yI4hFIj2Lt20XXjvG+SQ8AZxazudJJH3m4OYcI++wK vNBin//nusv8mRmOTv0+k8i2EBdnLQ19SBiDpoIk0T5tLylv3hqeVPBgvuoUEic1mbogkagdK7194 Q+b1Ldg0PJyX4IOrv3dZg5VniK5brdoJIIhi2y9oiUppRBpAr7F/iwqukzd2gpMUczH7qKm3dmc8D y/F5EHwg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77O-008g8e-9U; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4346C302DA6; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B08522B8EFB61; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.959811186@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:19 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 54/59] x86/orc: Make it callthunk aware References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Callthunks addresses on the stack would confuse the ORC unwinder. Handle them correctly and tell ORC to proceed further down the stack. Cc: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/alternative.h | 5 +++++ arch/x86/kernel/callthunks.c | 13 +++++++++++++ arch/x86/kernel/unwind_orc.c | 21 ++++++++++++++++++++- 3 files changed, 38 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -92,6 +92,7 @@ extern void callthunks_patch_builtin_cal extern void callthunks_patch_module_calls(struct callthunk_sites *sites, struct module *mod); extern void *callthunks_translate_call_dest(void *dest); +extern bool is_callthunk(void *addr); #else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline void @@ -101,6 +102,10 @@ static __always_inline void *callthunks_ { return dest; } +static __always_inline bool is_callthunk(void *addr) +{ + return false; +} #endif =20 #ifdef CONFIG_SMP --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -292,6 +292,19 @@ void *callthunks_translate_call_dest(voi return target ? : dest; } =20 +bool is_callthunk(void *addr) +{ + unsigned int tmpl_size =3D SKL_TMPL_SIZE; + void *tmpl =3D skl_call_thunk_template; + unsigned long dest; + + dest =3D roundup((unsigned long)addr, CONFIG_FUNCTION_ALIGNMENT); + if (!thunks_initialized || skip_addr((void *)dest)) + return false; + + return !bcmp((void *)(dest - tmpl_size), tmpl, tmpl_size); +} + #ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs, struct module *mod) --- a/arch/x86/kernel/unwind_orc.c +++ b/arch/x86/kernel/unwind_orc.c @@ -131,6 +131,21 @@ static struct orc_entry null_orc_entry =3D .type =3D UNWIND_HINT_TYPE_CALL }; =20 +#ifdef CONFIG_CALL_THUNKS +static struct orc_entry *orc_callthunk_find(unsigned long ip) +{ + if (!is_callthunk((void *)ip)) + return NULL; + + return &null_orc_entry; +} +#else +static struct orc_entry *orc_callthunk_find(unsigned long ip) +{ + return NULL; +} +#endif + /* Fake frame pointer entry -- used as a fallback for generated code */ static struct orc_entry orc_fp_entry =3D { .type =3D UNWIND_HINT_TYPE_CALL, @@ -184,7 +199,11 @@ static struct orc_entry *orc_find(unsign if (orc) return orc; =20 - return orc_ftrace_find(ip); + orc =3D orc_ftrace_find(ip); + if (orc) + return orc; + + return orc_callthunk_find(ip); } =20 #ifdef CONFIG_MODULES From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BF31C38145 for ; Fri, 2 Sep 2022 14:30:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236428AbiIBOaG (ORCPT ); Fri, 2 Sep 2022 10:30:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237179AbiIBO1g (ORCPT ); Fri, 2 Sep 2022 10:27:36 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70BCF158194 for ; Fri, 2 Sep 2022 06:54:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=k9PVCzDY3XTDOFcLeQOk0i9MK45d5cwVZ0RbnNAMMfM=; b=BRt1q9V6wEhV7Zahz88lFfTjMb Wmk/NXFdSNpgnongcIU1Rjx51Z4bYYjuoSNJ5BUX5LFIRXSiM/BUaiJC2xfN55dytUH1U+TO1ySXJ j6tiLCvAcNbHghy8qDtGYoVSWhhDdQqSIojxcp5xnQEJXgnfwYTg48IWrGvs8yu55QgawT5+AakuD bXK1Isdd/223Y8G5BAvuVg5gAup4yAHayRzc4fHb7hHxPhH6wlFpHkPTwgrpY/YY0kFYyS2kqwBv2 n29pg6QkZDL9YD7/yWnHBAH3nyW8taNtWJPCf0b1oqITvAKOpxGczCqemnCbg1R0ZjllxeTcGA9UO 4cNfSKJg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77R-0074Ve-KV; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 472A9302E09; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B9EE62B8EFB62; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130952.063700784@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:20 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 55/59] x86/bpf: Emit call depth accounting if required References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Ensure that calls in BPF jitted programs are emitting call depth accounting when enabled to keep the call/return balanced. The return thunk jump is already injected due to the earlier retbleed mitigations. Cc: Alexei Starovoitov Cc: Daniel Borkmann Signed-off-by: Thomas Gleixner Tested-by: Mel Gorman --- arch/x86/include/asm/alternative.h | 6 ++++++ arch/x86/kernel/callthunks.c | 19 +++++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 32 +++++++++++++++++++++++--------- 3 files changed, 48 insertions(+), 9 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -93,6 +93,7 @@ extern void callthunks_patch_module_call struct module *mod); extern void *callthunks_translate_call_dest(void *dest); extern bool is_callthunk(void *addr); +extern int x86_call_depth_emit_accounting(u8 **pprog, void *func); #else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline void @@ -106,6 +107,11 @@ static __always_inline bool is_callthunk { return false; } +static __always_inline int x86_call_depth_emit_accounting(u8 **pprog, + void *func) +{ + return 0; +} #endif =20 #ifdef CONFIG_SMP --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -305,6 +305,25 @@ bool is_callthunk(void *addr) return !bcmp((void *)(dest - tmpl_size), tmpl, tmpl_size); } =20 +#ifdef CONFIG_BPF_JIT +int x86_call_depth_emit_accounting(u8 **pprog, void *func) +{ + unsigned int tmpl_size =3D SKL_TMPL_SIZE; + void *tmpl =3D skl_call_thunk_template; + + if (!thunks_initialized) + return 0; + + /* Is function call target a thunk? */ + if (is_callthunk(func)) + return 0; + + memcpy(*pprog, tmpl, tmpl_size); + *pprog +=3D tmpl_size; + return tmpl_size; +} +#endif + #ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs, struct module *mod) --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -340,6 +340,13 @@ static int emit_call(u8 **pprog, void *f return emit_patch(pprog, func, ip, 0xE8); } =20 +static int emit_rsb_call(u8 **pprog, void *func, void *ip) +{ + OPTIMIZER_HIDE_VAR(func); + x86_call_depth_emit_accounting(pprog, func); + return emit_patch(pprog, func, ip, 0xE8); +} + static int emit_jump(u8 **pprog, void *func, void *ip) { return emit_patch(pprog, func, ip, 0xE9); @@ -1434,19 +1441,26 @@ st: if (is_imm8(insn->off)) break; =20 /* call */ - case BPF_JMP | BPF_CALL: + case BPF_JMP | BPF_CALL: { + int offs; + func =3D (u8 *) __bpf_call_base + imm32; if (tail_call_reachable) { /* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */ EMIT3_off32(0x48, 0x8B, 0x85, -round_up(bpf_prog->aux->stack_depth, 8) - 8); - if (!imm32 || emit_call(&prog, func, image + addrs[i - 1] + 7)) + if (!imm32) return -EINVAL; + offs =3D 7 + x86_call_depth_emit_accounting(&prog, func); } else { - if (!imm32 || emit_call(&prog, func, image + addrs[i - 1])) + if (!imm32) return -EINVAL; + offs =3D x86_call_depth_emit_accounting(&prog, func); } + if (emit_call(&prog, func, image + addrs[i - 1] + offs)) + return -EINVAL; break; + } =20 case BPF_JMP | BPF_TAIL_CALL: if (imm32) @@ -1823,7 +1837,7 @@ static int invoke_bpf_prog(const struct /* arg2: lea rsi, [rbp - ctx_cookie_off] */ EMIT4(0x48, 0x8D, 0x75, -run_ctx_off); =20 - if (emit_call(&prog, enter, prog)) + if (emit_rsb_call(&prog, enter, prog)) return -EINVAL; /* remember prog start time returned by __bpf_prog_enter */ emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0); @@ -1844,7 +1858,7 @@ static int invoke_bpf_prog(const struct (long) p->insnsi >> 32, (u32) (long) p->insnsi); /* call JITed bpf program or interpreter */ - if (emit_call(&prog, p->bpf_func, prog)) + if (emit_rsb_call(&prog, p->bpf_func, prog)) return -EINVAL; =20 /* @@ -1868,7 +1882,7 @@ static int invoke_bpf_prog(const struct emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); /* arg3: lea rdx, [rbp - run_ctx_off] */ EMIT4(0x48, 0x8D, 0x55, -run_ctx_off); - if (emit_call(&prog, exit, prog)) + if (emit_rsb_call(&prog, exit, prog)) return -EINVAL; =20 *pprog =3D prog; @@ -2109,7 +2123,7 @@ int arch_prepare_bpf_trampoline(struct b if (flags & BPF_TRAMP_F_CALL_ORIG) { /* arg1: mov rdi, im */ emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im); - if (emit_call(&prog, __bpf_tramp_enter, prog)) { + if (emit_rsb_call(&prog, __bpf_tramp_enter, prog)) { ret =3D -EINVAL; goto cleanup; } @@ -2141,7 +2155,7 @@ int arch_prepare_bpf_trampoline(struct b EMIT2(0xff, 0xd0); /* call *rax */ } else { /* call original function */ - if (emit_call(&prog, orig_call, prog)) { + if (emit_rsb_call(&prog, orig_call, prog)) { ret =3D -EINVAL; goto cleanup; } @@ -2185,7 +2199,7 @@ int arch_prepare_bpf_trampoline(struct b im->ip_epilogue =3D prog; /* arg1: mov rdi, im */ emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im); - if (emit_call(&prog, __bpf_tramp_exit, prog)) { + if (emit_rsb_call(&prog, __bpf_tramp_exit, prog)) { ret =3D -EINVAL; goto cleanup; } From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 491E6ECAAD5 for ; Fri, 2 Sep 2022 14:31:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236376AbiIBObB (ORCPT ); Fri, 2 Sep 2022 10:31:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235901AbiIBO1i (ORCPT ); Fri, 2 Sep 2022 10:27:38 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DB0E1581A9 for ; Fri, 2 Sep 2022 06:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=cGtjwbCjC/FUhMHf/1kXmsZcQFKDFVc++WobQ3K3nBI=; b=APDtxBO7wjqNEwPgO0C+AlCsSR jNRI5vZv7TMLB2lH79K0ZYm5ieuJgvBdwUs0dJNACeD4UsCsiv3rH7alQFYOjZ9ih65DWU/5QZpI9 qhputzlsedjplmisujZXysSyEddNEdB9Ce+XDFOiv+AXL333sCeC0cimvOvnbxYDV4W+o2yx2KKhs fTntF6UwkiWLhmaLU/frMjCH1EDPsZUucb2vCOS5iWszKxy0UHL1W3lLyN0VpTzoruJ2sX6Wx4Pe+ MGeLkcHktE6C34DJXrYAbF9+sYJQn2wqHQ09dulFVQ5ZQkSLmPmeZ+hLmpKE7pYvuAooUWTLT6/cv qZbus9nA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77P-008g8r-M2; Fri, 02 Sep 2022 13:54:03 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 490E8302E11; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C38952B8EFB63; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130952.167578304@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:21 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 56/59] x86/ftrace: Remove ftrace_epilogue() References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Remove the weird jumps to RET and simply use RET. This then promotes ftrace_stub() to a real function; which becomes important for kcfi. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/ftrace_64.S | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -172,20 +172,14 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBA */ SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR - - jmp ftrace_epilogue + RET SYM_FUNC_END(ftrace_caller); STACK_FRAME_NON_STANDARD_FP(ftrace_caller) =20 -SYM_FUNC_START(ftrace_epilogue) -/* - * This is weak to keep gas from relaxing the jumps. - */ -SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK) +SYM_FUNC_START(ftrace_stub) UNWIND_HINT_FUNC - ENDBR RET -SYM_FUNC_END(ftrace_epilogue) +SYM_FUNC_END(ftrace_stub) =20 SYM_FUNC_START(ftrace_regs_caller) /* Save the current flags before any operations that can change them */ @@ -262,14 +256,11 @@ SYM_INNER_LABEL(ftrace_regs_caller_jmp, popfq =20 /* - * As this jmp to ftrace_epilogue can be a short jump - * it must not be copied into the trampoline. - * The trampoline will add the code to jump - * to the return. + * The trampoline will add the return. */ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR - jmp ftrace_epilogue + RET =20 /* Swap the flags with orig_rax */ 1: movq MCOUNT_REG_SIZE(%rsp), %rdi @@ -280,7 +271,7 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, /* Restore flags */ popfq UNWIND_HINT_FUNC - jmp ftrace_epilogue + RET =20 SYM_FUNC_END(ftrace_regs_caller) STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller) From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6855EC38145 for ; Fri, 2 Sep 2022 14:31:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236013AbiIBObP (ORCPT ); Fri, 2 Sep 2022 10:31:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236612AbiIBO1k (ORCPT ); Fri, 2 Sep 2022 10:27:40 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FF351581A1 for ; Fri, 2 Sep 2022 06:54:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=XjJyb073rhc42xRL7dMw1nuC9q0N0CTNdvFDeYOzjH4=; b=Al7+F+8giCc+wzNvsIE7TAcHm+ N3VfIsBwm/q1stVott7n0MUB0ToZ8uFv6envqWIbZdiD3conDrv8oSzRIGx06XMiWyD08rEoRLZhr 94n+G80urfoLVIizijUp5nNI2MHNhNhN7RrEAV80wO996Rbt6zqr8LyjwyeyHowg5pjkq2QjuctFe IqYwH2BDAtatU1u9R2L7zJqQvDAXogJsQtFR3BKvR9CSZwlaxxUkvQkQZTOg/ZFEuICnLprMEjXTF y8fzL2y/k2bHTrV3rc+TOXTz9vhfiRcaO7Or26SdOqHaxmPbmbtguL0jTwv8uWyc+/qwwo9wctAGy rabrNfRA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77P-008g8i-6k; Fri, 02 Sep 2022 13:54:03 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 49155302E13; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id CCEB92B8EFB64; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130952.269277054@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:22 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 57/59] x86/ftrace: Rebalance RSB References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra (Intel) ftrace_regs_caller() uses a PUSH;RET pattern to tail-call into a direct-call function, this unbalances the RSB, fix that. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/ftrace_64.S | 11 +++++++++++ 1 file changed, 11 insertions(+) --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -271,6 +271,17 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, /* Restore flags */ popfq UNWIND_HINT_FUNC + + /* + * The above left an extra return value on the stack; effectively + * doing a tail-call without using a register. This PUSH;RET + * pattern unbalances the RSB, inject a pointless CALL to rebalance. + */ + ANNOTATE_INTRA_FUNCTION_CALL + CALL .Ldo_rebalance + int3 +.Ldo_rebalance: + add $8, %rsp RET =20 SYM_FUNC_END(ftrace_regs_caller) From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B86EECAAD5 for ; Fri, 2 Sep 2022 14:30:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236761AbiIBOai (ORCPT ); Fri, 2 Sep 2022 10:30:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237270AbiIBO1i (ORCPT ); Fri, 2 Sep 2022 10:27:38 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2A4013C9F1 for ; Fri, 2 Sep 2022 06:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=nkpy4e1HtSQn0yBC/Joemu5I/+dxImw25H8Njycu7GE=; b=gr5bIfOceN5RFMT0JskhEX8l9A j2BCiiRa/wWT+4OiusNtzYwdWDDM3HTtvzcHBYyZYGIneK2pySdi7yOK5KmbvsnUJpTulnQ2f8TZC tWYVjFkJtGRAZ1ztjMBHr3Z/lHdtmiWkhWW5r/0Y+WCs5DY0P/TvZ1nlZPCeX6etVCEKn00dBuJIj OO3F2ZxiM6btpoclenQBUXoPGW1G3k1+SLSnW+TNgp4UlDBOiTYlge0WuPR8ysSTvfZNvUA3nqm54 iJVp6MiS8ulpbKYtJ7/yAgcJFw9z021qr+Vwq6f6NK/MK+mYs0ohK+fcX+8H7VQPqH0+oEy+vAm/h NY95A2/g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77Q-0074V7-Ty; Fri, 02 Sep 2022 13:54:01 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 490DA302E0D; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D66032B8EFB66; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130952.375380675@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:23 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 58/59] x86/ftrace: Make it call depth tracking aware References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Since ftrace has trampolines, don't use thunks for the __fentry__ site but instead require that every function called from there includes accounting. This very much includes all the direct-call functions. Additionally, ftrace uses ROP tricks in two places: - return_to_handler(), and - ftrace_regs_caller() when pt_regs->orig_ax is set by a direct-call. return_to_handler() already uses a retpoline to replace an indirect-jump to defeat IBT, since this is a jump-type retpoline, make sure there is no accounting done and ALTERNATIVE the RET into a ret. ftrace_regs_caller() does much the same and gets the same treatment. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/include/asm/nospec-branch.h | 9 +++++++++ arch/x86/kernel/callthunks.c | 2 +- arch/x86/kernel/ftrace.c | 16 ++++++++++++---- arch/x86/kernel/ftrace_64.S | 22 ++++++++++++++++++++-- arch/x86/net/bpf_jit_comp.c | 6 ++++++ kernel/trace/trace_selftest.c | 5 ++++- samples/ftrace/ftrace-direct-modify.c | 3 +++ samples/ftrace/ftrace-direct-multi-modify.c | 3 +++ samples/ftrace/ftrace-direct-multi.c | 2 ++ samples/ftrace/ftrace-direct-too.c | 2 ++ samples/ftrace/ftrace-direct.c | 2 ++ 11 files changed, 64 insertions(+), 8 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -331,6 +331,12 @@ static inline void x86_set_skl_return_th { x86_return_thunk =3D &__x86_return_skl; } + +#define CALL_DEPTH_ACCOUNT \ + ALTERNATIVE("", \ + __stringify(INCREMENT_CALL_DEPTH), \ + X86_FEATURE_CALL_DEPTH) + #ifdef CONFIG_CALL_THUNKS_DEBUG DECLARE_PER_CPU(u64, __x86_call_count); DECLARE_PER_CPU(u64, __x86_ret_count); @@ -339,6 +345,9 @@ DECLARE_PER_CPU(u64, __x86_ctxsw_count); #endif #else static inline void x86_set_skl_return_thunk(void) {} + +#define CALL_DEPTH_ACCOUNT "" + #endif =20 #ifdef CONFIG_RETPOLINE --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -315,7 +315,7 @@ int x86_call_depth_emit_accounting(u8 ** return 0; =20 /* Is function call target a thunk? */ - if (is_callthunk(func)) + if (func && is_callthunk(func)) return 0; =20 memcpy(*pprog, tmpl, tmpl_size); --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -69,6 +69,10 @@ static const char *ftrace_nop_replace(vo =20 static const char *ftrace_call_replace(unsigned long ip, unsigned long add= r) { + /* + * No need to translate into a callthunk. The trampoline does + * the depth accounting itself. + */ return text_gen_insn(CALL_INSN_OPCODE, (void *)ip, (void *)addr); } =20 @@ -317,7 +321,7 @@ create_trampoline(struct ftrace_ops *ops unsigned long size; unsigned long *ptr; void *trampoline; - void *ip; + void *ip, *dest; /* 48 8b 15 is movq (%rip), %rdx */ unsigned const char op_ref[] =3D { 0x48, 0x8b, 0x15 }; unsigned const char retq[] =3D { RET_INSN_OPCODE, INT3_INSN_OPCODE }; @@ -404,10 +408,14 @@ create_trampoline(struct ftrace_ops *ops /* put in the call to the function */ mutex_lock(&text_mutex); call_offset -=3D start_offset; + /* + * No need to translate into a callthunk. The trampoline does + * the depth accounting before the call already. + */ + dest =3D ftrace_ops_get_func(ops); memcpy(trampoline + call_offset, - text_gen_insn(CALL_INSN_OPCODE, - trampoline + call_offset, - ftrace_ops_get_func(ops)), CALL_INSN_SIZE); + text_gen_insn(CALL_INSN_OPCODE, trampoline + call_offset, dest), + CALL_INSN_SIZE); mutex_unlock(&text_mutex); =20 /* ALLOC_TRAMP flags lets us know we created it */ --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -4,6 +4,7 @@ */ =20 #include +#include #include #include #include @@ -132,6 +133,7 @@ #ifdef CONFIG_DYNAMIC_FTRACE =20 SYM_FUNC_START(__fentry__) + CALL_DEPTH_ACCOUNT RET SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) @@ -140,6 +142,8 @@ SYM_FUNC_START(ftrace_caller) /* save_mcount_regs fills in first two parameters */ save_mcount_regs =20 + CALL_DEPTH_ACCOUNT + /* Stack - skipping return address of ftrace_caller */ leaq MCOUNT_REG_SIZE+8(%rsp), %rcx movq %rcx, RSP(%rsp) @@ -155,6 +159,9 @@ SYM_INNER_LABEL(ftrace_caller_op_ptr, SY /* Only ops with REGS flag set should have CS register set */ movq $0, CS(%rsp) =20 + /* Account for the function call below */ + CALL_DEPTH_ACCOUNT + SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub @@ -189,6 +196,8 @@ SYM_FUNC_START(ftrace_regs_caller) save_mcount_regs 8 /* save_mcount_regs fills in first two parameters */ =20 + CALL_DEPTH_ACCOUNT + SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */ @@ -219,6 +228,9 @@ SYM_INNER_LABEL(ftrace_regs_caller_op_pt /* regs go into 4th parameter */ leaq (%rsp), %rcx =20 + /* Account for the function call below */ + CALL_DEPTH_ACCOUNT + SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub @@ -282,7 +294,9 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, int3 .Ldo_rebalance: add $8, %rsp - RET + ALTERNATIVE __stringify(RET), \ + __stringify(ANNOTATE_UNRET_SAFE; ret; int3), \ + X86_FEATURE_CALL_DEPTH =20 SYM_FUNC_END(ftrace_regs_caller) STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller) @@ -291,6 +305,8 @@ STACK_FRAME_NON_STANDARD_FP(ftrace_regs_ #else /* ! CONFIG_DYNAMIC_FTRACE */ =20 SYM_FUNC_START(__fentry__) + CALL_DEPTH_ACCOUNT + cmpq $ftrace_stub, ftrace_trace_function jnz trace =20 @@ -347,6 +363,8 @@ SYM_CODE_START(return_to_handler) int3 .Ldo_rop: mov %rdi, (%rsp) - RET + ALTERNATIVE __stringify(RET), \ + __stringify(ANNOTATE_UNRET_SAFE; ret; int3), \ + X86_FEATURE_CALL_DEPTH SYM_CODE_END(return_to_handler) #endif --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -2095,6 +2096,11 @@ int arch_prepare_bpf_trampoline(struct b prog =3D image; =20 EMIT_ENDBR(); + /* + * This is the direct-call trampoline, as such it needs accounting + * for the __fentry__ call. + */ + x86_call_depth_emit_accounting(&prog, NULL); EMIT1(0x55); /* push rbp */ EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ EMIT4(0x48, 0x83, 0xEC, stack_size); /* sub rsp, stack_size */ --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -785,7 +785,10 @@ static struct fgraph_ops fgraph_ops __in }; =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS -noinline __noclone static void trace_direct_tramp(void) { } +noinline __noclone static void trace_direct_tramp(void) +{ + asm(CALL_DEPTH_ACCOUNT); +} #endif =20 /* --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -3,6 +3,7 @@ #include #include #include +#include =20 extern void my_direct_func1(void); extern void my_direct_func2(void); @@ -34,6 +35,7 @@ asm ( ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" + CALL_DEPTH_ACCOUNT " call my_direct_func1\n" " leave\n" " .size my_tramp1, .-my_tramp1\n" @@ -45,6 +47,7 @@ asm ( ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" + CALL_DEPTH_ACCOUNT " call my_direct_func2\n" " leave\n" ASM_RET --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -3,6 +3,7 @@ #include #include #include +#include =20 extern void my_direct_func1(unsigned long ip); extern void my_direct_func2(unsigned long ip); @@ -32,6 +33,7 @@ asm ( ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" + CALL_DEPTH_ACCOUNT " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func1\n" @@ -46,6 +48,7 @@ asm ( ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" + CALL_DEPTH_ACCOUNT " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func2\n" --- a/samples/ftrace/ftrace-direct-multi.c +++ b/samples/ftrace/ftrace-direct-multi.c @@ -5,6 +5,7 @@ #include #include #include +#include =20 extern void my_direct_func(unsigned long ip); =20 @@ -27,6 +28,7 @@ asm ( ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" + CALL_DEPTH_ACCOUNT " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func\n" --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -4,6 +4,7 @@ #include /* for handle_mm_fault() */ #include #include +#include =20 extern void my_direct_func(struct vm_area_struct *vma, unsigned long address, unsigned int flags); @@ -29,6 +30,7 @@ asm ( ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" + CALL_DEPTH_ACCOUNT " pushq %rdi\n" " pushq %rsi\n" " pushq %rdx\n" --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -4,6 +4,7 @@ #include /* for wake_up_process() */ #include #include +#include =20 extern void my_direct_func(struct task_struct *p); =20 @@ -26,6 +27,7 @@ asm ( ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" + CALL_DEPTH_ACCOUNT " pushq %rdi\n" " call my_direct_func\n" " popq %rdi\n" From nobody Mon Apr 6 20:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF6FEC38145 for ; Fri, 2 Sep 2022 14:31:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234101AbiIBObt (ORCPT ); Fri, 2 Sep 2022 10:31:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236087AbiIBO2K (ORCPT ); Fri, 2 Sep 2022 10:28:10 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D82D1581A7 for ; Fri, 2 Sep 2022 06:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=GdUExkZCCimaetm5TM7oIFdVp9RGBizt5Yv4RZaDQ/c=; b=HCop2yMRhmWyUiLz3VPIMxOlW2 6JILmVaX9kfOQXqxiYcEB2bXhi/LOkbY7uIqoSpWRrR0UsrxdAWRuRz53VMtOcIEvLmM4afs7CPIW V26u8+C7dmrHAYUCgOiZ1qT7HRWhA8L0bU6jxMzQ0bR7PyRmK/6t/uare5PRjG9gsplC6HRXNifhE XHM7XE/gsC6lmk/UUQhlw6hjjw1DnbriW30+I+mnBU3sAlLhYVnyHdwyiEmuyJdarTMJKBCQYK3mm mmMlKQxwYGx3Q9XsRah77dNz03kU9+2etkRkjCEwCN9qi3mSDtL4/tWVgjUVk+qoeiJCABuvkMuh0 feRMSgzQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77Q-008g9I-Vc; Fri, 02 Sep 2022 13:54:03 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4A457302E1D; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id DFCD42B8EFB67; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130952.478910269@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:24 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 59/59] x86/retbleed: Add call depth tracking mitigation References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The fully secure mitigation for RSB underflow on Intel SKL CPUs is IBRS, which inflicts up to 30% penalty for pathological syscall heavy work loads. Software based call depth tracking and RSB refill is not perfect, but reduces the attack surface massively. The penalty for the pathological case is about 8% which is still annoying but definitely more palatable than IBRS. Add a retbleed=3Dstuff command line option to enable the call depth tracking and software refill of the RSB. This gives admins a choice. IBeeRS are safe and cause headaches, call depth tracking is considered to be s(t)ufficiently safe. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Mel Gorman --- arch/x86/kernel/cpu/bugs.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -787,6 +787,7 @@ enum retbleed_mitigation { RETBLEED_MITIGATION_IBPB, RETBLEED_MITIGATION_IBRS, RETBLEED_MITIGATION_EIBRS, + RETBLEED_MITIGATION_STUFF, }; =20 enum retbleed_mitigation_cmd { @@ -794,6 +795,7 @@ enum retbleed_mitigation_cmd { RETBLEED_CMD_AUTO, RETBLEED_CMD_UNRET, RETBLEED_CMD_IBPB, + RETBLEED_CMD_STUFF, }; =20 static const char * const retbleed_strings[] =3D { @@ -802,6 +804,7 @@ static const char * const retbleed_strin [RETBLEED_MITIGATION_IBPB] =3D "Mitigation: IBPB", [RETBLEED_MITIGATION_IBRS] =3D "Mitigation: IBRS", [RETBLEED_MITIGATION_EIBRS] =3D "Mitigation: Enhanced IBRS", + [RETBLEED_MITIGATION_STUFF] =3D "Mitigation: Stuffing", }; =20 static enum retbleed_mitigation retbleed_mitigation __ro_after_init =3D @@ -831,6 +834,8 @@ static int __init retbleed_parse_cmdline retbleed_cmd =3D RETBLEED_CMD_UNRET; } else if (!strcmp(str, "ibpb")) { retbleed_cmd =3D RETBLEED_CMD_IBPB; + } else if (!strcmp(str, "stuff")) { + retbleed_cmd =3D RETBLEED_CMD_STUFF; } else if (!strcmp(str, "nosmt")) { retbleed_nosmt =3D true; } else { @@ -879,6 +884,21 @@ static void __init retbleed_select_mitig } break; =20 + case RETBLEED_CMD_STUFF: + if (IS_ENABLED(CONFIG_CALL_DEPTH_TRACKING) && + spectre_v2_enabled =3D=3D SPECTRE_V2_RETPOLINE) { + retbleed_mitigation =3D RETBLEED_MITIGATION_STUFF; + + } else { + if (IS_ENABLED(CONFIG_CALL_DEPTH_TRACKING)) + pr_err("WARNING: retbleed=3Dstuff depends on spectre_v2=3Dretpoline\n"= ); + else + pr_err("WARNING: kernel not compiled with CALL_DEPTH_TRACKING.\n"); + + goto do_cmd_auto; + } + break; + do_cmd_auto: case RETBLEED_CMD_AUTO: default: @@ -916,6 +936,12 @@ static void __init retbleed_select_mitig mitigate_smt =3D true; break; =20 + case RETBLEED_MITIGATION_STUFF: + setup_force_cpu_cap(X86_FEATURE_RETHUNK); + setup_force_cpu_cap(X86_FEATURE_CALL_DEPTH); + x86_set_skl_return_thunk(); + break; + default: break; } @@ -926,7 +952,7 @@ static void __init retbleed_select_mitig =20 /* * Let IBRS trump all on Intel without affecting the effects of the - * retbleed=3D cmdline option. + * retbleed=3D cmdline option except for call depth based stuffing */ if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_INTEL) { switch (spectre_v2_enabled) { @@ -939,7 +965,8 @@ static void __init retbleed_select_mitig retbleed_mitigation =3D RETBLEED_MITIGATION_EIBRS; break; default: - pr_err(RETBLEED_INTEL_MSG); + if (retbleed_mitigation !=3D RETBLEED_MITIGATION_STUFF) + pr_err(RETBLEED_INTEL_MSG); } } =20 @@ -1413,6 +1440,7 @@ static void __init spectre_v2_select_mit if (IS_ENABLED(CONFIG_CPU_IBRS_ENTRY) && boot_cpu_has_bug(X86_BUG_RETBLEED) && retbleed_cmd !=3D RETBLEED_CMD_OFF && + retbleed_cmd !=3D RETBLEED_CMD_STUFF && boot_cpu_has(X86_FEATURE_IBRS) && boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_INTEL) { mode =3D SPECTRE_V2_IBRS;