From nobody Mon Apr 6 21:53:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18FBDC38145 for ; Fri, 2 Sep 2022 14:31:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235498AbiIBObl (ORCPT ); Fri, 2 Sep 2022 10:31:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236922AbiIBO1u (ORCPT ); Fri, 2 Sep 2022 10:27:50 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47EB71581AB for ; Fri, 2 Sep 2022 06:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=PsjEG4knouG+JCTeWWUmbwqU2xySdGfCmOMQNnMDy2Q=; b=Qp0mGFPB5LdN/pvzWYro8eZiFO XbgukPSiOl0q748pBt3ft2uufZuUlM1flnG0aeMXGUW7ytO5p3fzrGUthTKCFoffSLynfCCdge9Ic BV79/DDm+L7JuJ2BRknNrkveODlgdGCOvxcpQwiGwG3oUG8kLtx9nyswqJolT+VVNJhKchML++ZpZ Ne2Xgteamj/DhtICsa+h8tD4BdCaMLXMxrokswI0ui9Gf6fPT3BLXSODHFAxnIXF4N6w8GtSSWfWq V5KetR9+Ze/YgG3A3B3qEvonpxxz5ZzgnNK1i9u2NXo+SL8bhG8lABpAWkyAJW/IWu+NW7VWuZws3 EAjLMQTQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU77P-0074UO-8H; Fri, 02 Sep 2022 13:53:59 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3C9C7302D59; Fri, 2 Sep 2022 15:53:54 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 5DA2C2B8EFB54; Fri, 2 Sep 2022 15:53:53 +0200 (CEST) Message-ID: <20220902130951.029167377@infradead.org> User-Agent: quilt/0.66 Date: Fri, 02 Sep 2022 15:07:10 +0200 From: Peter Zijlstra To: Thomas Gleixner Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, x86@kernel.org, Linus Torvalds , Tim Chen , Josh Poimboeuf , Andrew Cooper , Pawan Gupta , Johannes Wikner , Alyssa Milburn , Jann Horn , "H.J. Lu" , Joao Moreira , Joseph Nuzman , Steven Rostedt , Juergen Gross , Masami Hiramatsu , Alexei Starovoitov , Daniel Borkmann , K Prateek Nayak , Eric Dumazet Subject: [PATCH v2 45/59] x86/modules: Add call patching References: <20220902130625.217071627@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner As for the builtins create call thunks and patch the call sites to call the thunk on Intel SKL CPUs for retbleed mitigation. Note, that module init functions are ignored for sake of simplicity because loading modules is not something which is done in high frequent loops and the attacker has not really a handle on when this happens in order to launch a matching attack. The depth tracking will still work for calls into the builtins and because the call is not accounted it will underflow faster and overstuff, but that's mitigated by the saturating counter and the side effect is only temporary. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/alternative.h | 5 +++++ arch/x86/kernel/callthunks.c | 19 +++++++++++++++++++ arch/x86/kernel/module.c | 20 +++++++++++++++++++- 3 files changed, 43 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -89,8 +89,13 @@ struct callthunk_sites { =20 #ifdef CONFIG_CALL_THUNKS extern void callthunks_patch_builtin_calls(void); +extern void callthunks_patch_module_calls(struct callthunk_sites *sites, + struct module *mod); #else static __always_inline void callthunks_patch_builtin_calls(void) {} +static __always_inline void +callthunks_patch_module_calls(struct callthunk_sites *sites, + struct module *mod) {} #endif =20 #ifdef CONFIG_SMP --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -249,3 +249,22 @@ void __init callthunks_patch_builtin_cal thunks_initialized =3D true; mutex_unlock(&text_mutex); } + +#ifdef CONFIG_MODULES +void noinline callthunks_patch_module_calls(struct callthunk_sites *cs, + struct module *mod) +{ + struct core_text ct =3D { + .base =3D (unsigned long)mod->core_layout.base, + .end =3D (unsigned long)mod->core_layout.base + mod->core_layout.size, + .name =3D mod->name, + }; + + if (!thunks_initialized) + return; + + mutex_lock(&text_mutex); + callthunks_setup(cs, &ct); + mutex_unlock(&text_mutex); +} +#endif /* CONFIG_MODULES */ --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -254,7 +254,8 @@ int module_finalize(const Elf_Ehdr *hdr, { const Elf_Shdr *s, *text =3D NULL, *alt =3D NULL, *locks =3D NULL, *para =3D NULL, *orc =3D NULL, *orc_ip =3D NULL, - *retpolines =3D NULL, *returns =3D NULL, *ibt_endbr =3D NULL; + *retpolines =3D NULL, *returns =3D NULL, *ibt_endbr =3D NULL, + *calls =3D NULL; char *secstrings =3D (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; =20 for (s =3D sechdrs; s < sechdrs + hdr->e_shnum; s++) { @@ -274,6 +275,8 @@ int module_finalize(const Elf_Ehdr *hdr, retpolines =3D s; if (!strcmp(".return_sites", secstrings + s->sh_name)) returns =3D s; + if (!strcmp(".call_sites", secstrings + s->sh_name)) + calls =3D s; if (!strcmp(".ibt_endbr_seal", secstrings + s->sh_name)) ibt_endbr =3D s; } @@ -299,6 +302,21 @@ int module_finalize(const Elf_Ehdr *hdr, void *aseg =3D (void *)alt->sh_addr; apply_alternatives(aseg, aseg + alt->sh_size); } + if (calls || para) { + struct callthunk_sites cs =3D {}; + + if (calls) { + cs.call_start =3D (void *)calls->sh_addr; + cs.call_end =3D (void *)calls->sh_addr + calls->sh_size; + } + + if (para) { + cs.pv_start =3D (void *)para->sh_addr; + cs.pv_end =3D (void *)para->sh_addr + para->sh_size; + } + + callthunks_patch_module_calls(&cs, me); + } if (ibt_endbr) { void *iseg =3D (void *)ibt_endbr->sh_addr; apply_ibt_endbr(iseg, iseg + ibt_endbr->sh_size);