From nobody Wed Dec 17 07:28:36 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 319F2C4167B for ; Wed, 29 Nov 2023 13:34:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234455AbjK2Ne2 (ORCPT ); Wed, 29 Nov 2023 08:34:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234443AbjK2NeN (ORCPT ); Wed, 29 Nov 2023 08:34:13 -0500 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 258CA1BD3 for ; Wed, 29 Nov 2023 05:34:14 -0800 (PST) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9B9062199D; Wed, 29 Nov 2023 13:34:13 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 466A41376F; Wed, 29 Nov 2023 13:34:13 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id /kvTD9U9Z2WjHwAAn2gu4w (envelope-from ); Wed, 29 Nov 2023 13:34:13 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org, virtualization@lists.linux.dev Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Ajay Kaher , Alexey Makhalov , VMware PV-Drivers Reviewers , Peter Zijlstra Subject: [PATCH v5 4/5] x86/paravirt: switch mixed paravirt/alternative calls to alternative_2 Date: Wed, 29 Nov 2023 14:33:31 +0100 Message-Id: <20231129133332.31043-5-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20231129133332.31043-1-jgross@suse.com> References: <20231129133332.31043-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspamd1 Authentication-Results: smtp-out1.suse.de; none X-Rspamd-Queue-Id: 9B9062199D X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[] Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Instead of stacking alternative and paravirt patching, use the new ALT_FLAG_CALL flag to switch those mixed calls to pure alternative handling. This eliminates the need to be careful regarding the sequence of alternative and paravirt patching. Signed-off-by: Juergen Gross Acked-by: Peter Zijlstra (Intel) --- V5: - remove no longer needed extern declarations from alternative.c (Boris Petkov) - add comment about ALTERNATIVE[_2]() macro usage (Boris Petkov) - rebase to tip/master (Boris Petkov) --- arch/x86/include/asm/alternative.h | 5 ++-- arch/x86/include/asm/paravirt.h | 9 ++++-- arch/x86/include/asm/paravirt_types.h | 40 +++++++++++++++------------ arch/x86/kernel/alternative.c | 1 - arch/x86/kernel/callthunks.c | 17 ++++++------ arch/x86/kernel/module.c | 20 ++++---------- 6 files changed, 44 insertions(+), 48 deletions(-) diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alte= rnative.h index 472334eed6f3..fcd20c6dc7f9 100644 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -89,6 +89,8 @@ struct alt_instr { u8 replacementlen; /* length of new instruction */ } __packed; =20 +extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; + /* * Debug flag that can be tested to see whether alternative * instructions were patched in already: @@ -104,11 +106,10 @@ extern void apply_fineibt(s32 *start_retpoline, s32 *= end_retpoine, s32 *start_cfi, s32 *end_cfi); =20 struct module; -struct paravirt_patch_site; =20 struct callthunk_sites { s32 *call_start, *call_end; - struct paravirt_patch_site *pv_start, *pv_end; + struct alt_instr *alt_start, *alt_end; }; =20 #ifdef CONFIG_CALL_THUNKS diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravir= t.h index f18bfa7f3070..88bb5c34a171 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -738,20 +738,23 @@ void native_pv_lock_init(void) __init; =20 #ifdef CONFIG_X86_64 #ifdef CONFIG_PARAVIRT_XXL +#ifdef CONFIG_DEBUG_ENTRY =20 #define PARA_PATCH(off) ((off) / 8) #define PARA_SITE(ptype, ops) _PVSITE(ptype, ops) #define PARA_INDIRECT(addr) *addr(%rip) =20 -#ifdef CONFIG_DEBUG_ENTRY .macro PARA_IRQ_save_fl PARA_SITE(PARA_PATCH(PV_IRQ_save_fl), ANNOTATE_RETPOLINE_SAFE; call PARA_INDIRECT(pv_ops+PV_IRQ_save_fl);) + ANNOTATE_RETPOLINE_SAFE; + call PARA_INDIRECT(pv_ops+PV_IRQ_save_fl); .endm =20 -#define SAVE_FLAGS ALTERNATIVE "PARA_IRQ_save_fl;", "pushf; pop %rax;", \ - ALT_NOT_XEN +#define SAVE_FLAGS ALTERNATIVE_2 "PARA_IRQ_save_fl;", \ + ALT_CALL_INSTR, ALT_CALL_ALWAYS, \ + "pushf; pop %rax;", ALT_NOT_XEN #endif #endif /* CONFIG_PARAVIRT_XXL */ #endif /* CONFIG_X86_64 */ diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/p= aravirt_types.h index 166e9618158f..9cad536fc08d 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -276,15 +276,11 @@ extern struct paravirt_patch_template pv_ops; #define NATIVE_LABEL(a,x,b) "\n\t.globl " a #x "_" #b "\n" a #x "_" #b ":\= n\t" =20 unsigned int paravirt_patch(u8 type, void *insn_buff, unsigned long addr, = unsigned int len); +#define paravirt_ptr(op) [paravirt_opptr] "m" (pv_ops.op) =20 int paravirt_disable_iospace(void); =20 -/* - * This generates an indirect call based on the operation type number. - * The type number, computed in PARAVIRT_PATCH, is derived from the - * offset into the paravirt_patch_template structure, and can therefore be - * freely converted back into a structure offset. - */ +/* This generates an indirect call based on the operation type number. */ #define PARAVIRT_CALL \ ANNOTATE_RETPOLINE_SAFE \ "call *%[paravirt_opptr];" @@ -317,12 +313,6 @@ int paravirt_disable_iospace(void); * However, x86_64 also has to clobber all caller saved registers, which * unfortunately, are quite a bit (r8 - r11) * - * The call instruction itself is marked by placing its start address - * and size into the .parainstructions section, so that - * apply_paravirt() in arch/i386/kernel/alternative.c can do the - * appropriate patching under the control of the backend pv_init_ops - * implementation. - * * Unfortunately there's no way to get gcc to generate the args setup * for the call, and then allow the call itself to be generated by an * inline asm. Because of this, we must do the complete arg setup and @@ -421,14 +411,27 @@ int paravirt_disable_iospace(void); __mask & __eax; \ }) =20 - +/* + * Use alternative patching for paravirt calls: + * - For replacing an indirect call with a direct one, use the "normal" + * ALTERNATIVE() macro with the indirect call as the initial code sequen= ce, + * which will be replaced with the related direct call by using the + * ALT_FLAG_DIRECT_CALL special case and the "always on" feature. + * - In case the replacement is either a direct call or a short code seque= nce + * depending on a feature bit, the ALTERNATIVE_2() macro is being used. + * The indirect call is the initial code sequence again, while the speci= al + * code sequence is selected with the specified feature bit. In case the + * feature is not active, the direct call is used as above via the + * ALT_FLAG_DIRECT_CALL special case and the "always on" feature. + */ #define ____PVOP_CALL(ret, op, call_clbr, extra_clbr, ...) \ ({ \ PVOP_CALL_ARGS; \ PVOP_TEST_NULL(op); \ - asm volatile(paravirt_alt(PARAVIRT_CALL) \ + asm volatile(ALTERNATIVE(PARAVIRT_CALL, ALT_CALL_INSTR, \ + ALT_CALL_ALWAYS) \ : call_clbr, ASM_CALL_CONSTRAINT \ - : paravirt_type(op), \ + : paravirt_ptr(op), \ ##__VA_ARGS__ \ : "memory", "cc" extra_clbr); \ ret; \ @@ -439,10 +442,11 @@ int paravirt_disable_iospace(void); ({ \ PVOP_CALL_ARGS; \ PVOP_TEST_NULL(op); \ - asm volatile(ALTERNATIVE(paravirt_alt(PARAVIRT_CALL), \ - alt, cond) \ + asm volatile(ALTERNATIVE_2(PARAVIRT_CALL, \ + ALT_CALL_INSTR, ALT_CALL_ALWAYS, \ + alt, cond) \ : call_clbr, ASM_CALL_CONSTRAINT \ - : paravirt_type(op), \ + : paravirt_ptr(op), \ ##__VA_ARGS__ \ : "memory", "cc" extra_clbr); \ ret; \ diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index e7e53b9e801b..1f605079f982 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -160,7 +160,6 @@ extern s32 __retpoline_sites[], __retpoline_sites_end[]; extern s32 __return_sites[], __return_sites_end[]; extern s32 __cfi_sites[], __cfi_sites_end[]; extern s32 __ibt_endbr_seal[], __ibt_endbr_seal_end[]; -extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; void text_poke_early(void *addr, const void *opcode, size_t len); =20 diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index 57e5c2e75c2a..64ad2ddea121 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -233,14 +233,13 @@ patch_call_sites(s32 *start, s32 *end, const struct c= ore_text *ct) } =20 static __init_or_module void -patch_paravirt_call_sites(struct paravirt_patch_site *start, - struct paravirt_patch_site *end, - const struct core_text *ct) +patch_alt_call_sites(struct alt_instr *start, struct alt_instr *end, + const struct core_text *ct) { - struct paravirt_patch_site *p; + struct alt_instr *a; =20 - for (p =3D start; p < end; p++) - patch_call((void *)&p->instr_offset + p->instr_offset, ct); + for (a =3D start; a < end; a++) + patch_call((void *)&a->instr_offset + a->instr_offset, ct); } =20 static __init_or_module void @@ -248,7 +247,7 @@ callthunks_setup(struct callthunk_sites *cs, const stru= ct core_text *ct) { prdbg("Patching call sites %s\n", ct->name); patch_call_sites(cs->call_start, cs->call_end, ct); - patch_paravirt_call_sites(cs->pv_start, cs->pv_end, ct); + patch_alt_call_sites(cs->alt_start, cs->alt_end, ct); prdbg("Patching call sites done%s\n", ct->name); } =20 @@ -257,8 +256,8 @@ void __init callthunks_patch_builtin_calls(void) struct callthunk_sites cs =3D { .call_start =3D __call_sites, .call_end =3D __call_sites_end, - .pv_start =3D __parainstructions, - .pv_end =3D __parainstructions_end + .alt_start =3D __alt_instructions, + .alt_end =3D __alt_instructions_end }; =20 if (!cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index 5f71a0cf4399..e18914c0e38a 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -276,7 +276,7 @@ int module_finalize(const Elf_Ehdr *hdr, struct module *me) { const Elf_Shdr *s, *alt =3D NULL, *locks =3D NULL, - *para =3D NULL, *orc =3D NULL, *orc_ip =3D NULL, + *orc =3D NULL, *orc_ip =3D NULL, *retpolines =3D NULL, *returns =3D NULL, *ibt_endbr =3D NULL, *calls =3D NULL, *cfi =3D NULL; char *secstrings =3D (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; @@ -286,8 +286,6 @@ int module_finalize(const Elf_Ehdr *hdr, alt =3D s; if (!strcmp(".smp_locks", secstrings + s->sh_name)) locks =3D s; - if (!strcmp(".parainstructions", secstrings + s->sh_name)) - para =3D s; if (!strcmp(".orc_unwind", secstrings + s->sh_name)) orc =3D s; if (!strcmp(".orc_unwind_ip", secstrings + s->sh_name)) @@ -304,14 +302,6 @@ int module_finalize(const Elf_Ehdr *hdr, ibt_endbr =3D s; } =20 - /* - * See alternative_instructions() for the ordering rules between the - * various patching types. - */ - if (para) { - void *pseg =3D (void *)para->sh_addr; - apply_paravirt(pseg, pseg + para->sh_size); - } if (retpolines || cfi) { void *rseg =3D NULL, *cseg =3D NULL; unsigned int rsize =3D 0, csize =3D 0; @@ -341,7 +331,7 @@ int module_finalize(const Elf_Ehdr *hdr, void *aseg =3D (void *)alt->sh_addr; apply_alternatives(aseg, aseg + alt->sh_size); } - if (calls || para) { + if (calls || alt) { struct callthunk_sites cs =3D {}; =20 if (calls) { @@ -349,9 +339,9 @@ int module_finalize(const Elf_Ehdr *hdr, cs.call_end =3D (void *)calls->sh_addr + calls->sh_size; } =20 - if (para) { - cs.pv_start =3D (void *)para->sh_addr; - cs.pv_end =3D (void *)para->sh_addr + para->sh_size; + if (alt) { + cs.alt_start =3D (void *)alt->sh_addr; + cs.alt_end =3D (void *)alt->sh_addr + alt->sh_size; } =20 callthunks_patch_module_calls(&cs, me); --=20 2.35.3