From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C728837C0F3 for ; Tue, 28 Apr 2026 11:05:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374317; cv=none; b=LWC/GlBQ00ZmghwwTy5NxeijdZkdIRShW/OSBPMDl7HIOGkpFbe3lIL7nitEgFpvl7ZepaD3qptQpcGpvDXX6A6chL4yKIO7DNkV5ZUbECckH20fK8yPWewibQ8M2v8sSTniQsuu23p1jcBVXPyrqKqW/r6UwjhXUzj2QkbF26c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374317; c=relaxed/simple; bh=auRBh63JZxS/Md8dCtYnvllxjrwNFKnUaHvxyC+s8jM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fmaNK84PqPTiBuuKVmzSr3L3mo+jGFv7ifbyyyq1MAXDp2HRxO+4BfBvCi9bVxDpRt7G1YdIhL+O8vUqX75cCUUUFamkgwFRWprvlE3bvQAU7o9TH75QwsfSrBOb3mobv2Q3fh9hJbP+oiRKo1ktz/OCNJS3b8ijRR/7vZdBO4M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VZEYTLKo; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VZEYTLKo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TgGPSi7arVWiRpDPJ8V+u2Ms4i69wEtWAtSbWjbaVFY=; b=VZEYTLKorww9dTuFLzxOiXA/vZS1JuAFcq0J0s+GAsAp/U5sFVvBpURqzh3eEcRNl+/rFt EDk/1/bX2ikJD7loVV7BJldAm+hwASTn+5KiraTF/2BKFzdbSytYHsQkw3PpdkWSuCBk9D NK1QTM7IcYipf+HRCf9yZyN4nsIZJNw= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-595-C8S1g5W8PDiOypEVkdBcRw-1; Tue, 28 Apr 2026 07:05:11 -0400 X-MC-Unique: C8S1g5W8PDiOypEVkdBcRw-1 X-Mimecast-MFC-AGG-ID: C8S1g5W8PDiOypEVkdBcRw_1777374310 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 330F8195609F; Tue, 28 Apr 2026 11:05:10 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 311F1180047F; Tue, 28 Apr 2026 11:05:09 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 1/8] KVM: VMX: remove regs argument of __vmx_vcpu_run Date: Tue, 28 Apr 2026 07:05:00 -0400 Message-ID: <20260428110507.11248-2-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" Registers are reachable through vcpu_vmx, no need to pass a separate pointer to the regs[] array. No functional change intended. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/kvm-asm-offsets.c | 1 + arch/x86/kvm/vmx/vmenter.S | 58 +++++++++++++++------------------- arch/x86/kvm/vmx/vmx.c | 3 +- arch/x86/kvm/vmx/vmx.h | 3 +- 4 files changed, 28 insertions(+), 37 deletions(-) diff --git a/arch/x86/kvm/kvm-asm-offsets.c b/arch/x86/kvm/kvm-asm-offsets.c index 24a710d37323..36ac61724dd7 100644 --- a/arch/x86/kvm/kvm-asm-offsets.c +++ b/arch/x86/kvm/kvm-asm-offsets.c @@ -24,6 +24,7 @@ static void __used common(void) =20 if (IS_ENABLED(CONFIG_KVM_INTEL)) { BLANK(); + OFFSET(VMX_vcpu_arch_regs, vcpu_vmx, vcpu.arch.regs); OFFSET(VMX_spec_ctrl, vcpu_vmx, spec_ctrl); } } diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 8a481dae9cae..0a09288a8d29 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -11,24 +11,24 @@ =20 #define WORD_SIZE (BITS_PER_LONG / 8) =20 -#define VCPU_RAX __VCPU_REGS_RAX * WORD_SIZE -#define VCPU_RCX __VCPU_REGS_RCX * WORD_SIZE -#define VCPU_RDX __VCPU_REGS_RDX * WORD_SIZE -#define VCPU_RBX __VCPU_REGS_RBX * WORD_SIZE +#define VCPU_RAX (VMX_vcpu_arch_regs + __VCPU_REGS_RAX * WORD_SIZE) +#define VCPU_RCX (VMX_vcpu_arch_regs + __VCPU_REGS_RCX * WORD_SIZE) +#define VCPU_RDX (VMX_vcpu_arch_regs + __VCPU_REGS_RDX * WORD_SIZE) +#define VCPU_RBX (VMX_vcpu_arch_regs + __VCPU_REGS_RBX * WORD_SIZE) /* Intentionally omit RSP as it's context switched by hardware */ -#define VCPU_RBP __VCPU_REGS_RBP * WORD_SIZE -#define VCPU_RSI __VCPU_REGS_RSI * WORD_SIZE -#define VCPU_RDI __VCPU_REGS_RDI * WORD_SIZE +#define VCPU_RBP (VMX_vcpu_arch_regs + __VCPU_REGS_RBP * WORD_SIZE) +#define VCPU_RSI (VMX_vcpu_arch_regs + __VCPU_REGS_RSI * WORD_SIZE) +#define VCPU_RDI (VMX_vcpu_arch_regs + __VCPU_REGS_RDI * WORD_SIZE) =20 #ifdef CONFIG_X86_64 -#define VCPU_R8 __VCPU_REGS_R8 * WORD_SIZE -#define VCPU_R9 __VCPU_REGS_R9 * WORD_SIZE -#define VCPU_R10 __VCPU_REGS_R10 * WORD_SIZE -#define VCPU_R11 __VCPU_REGS_R11 * WORD_SIZE -#define VCPU_R12 __VCPU_REGS_R12 * WORD_SIZE -#define VCPU_R13 __VCPU_REGS_R13 * WORD_SIZE -#define VCPU_R14 __VCPU_REGS_R14 * WORD_SIZE -#define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE +#define VCPU_R8 (VMX_vcpu_arch_regs + __VCPU_REGS_R8 * WORD_SIZE) +#define VCPU_R9 (VMX_vcpu_arch_regs + __VCPU_REGS_R9 * WORD_SIZE) +#define VCPU_R10 (VMX_vcpu_arch_regs + __VCPU_REGS_R10 * WORD_SIZE) +#define VCPU_R11 (VMX_vcpu_arch_regs + __VCPU_REGS_R11 * WORD_SIZE) +#define VCPU_R12 (VMX_vcpu_arch_regs + __VCPU_REGS_R12 * WORD_SIZE) +#define VCPU_R13 (VMX_vcpu_arch_regs + __VCPU_REGS_R13 * WORD_SIZE) +#define VCPU_R14 (VMX_vcpu_arch_regs + __VCPU_REGS_R14 * WORD_SIZE) +#define VCPU_R15 (VMX_vcpu_arch_regs + __VCPU_REGS_R15 * WORD_SIZE) #endif =20 .macro VMX_DO_EVENT_IRQOFF call_insn call_target @@ -68,7 +68,6 @@ /** * __vmx_vcpu_run - Run a vCPU via a transition to VMX guest mode * @vmx: struct vcpu_vmx * - * @regs: unsigned long * (to guest registers) * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH * VMX_RUN_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl * VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO @@ -94,12 +93,6 @@ SYM_FUNC_START(__vmx_vcpu_run) push %_ASM_ARG1 =20 /* Save @flags (used for VMLAUNCH vs. VMRESUME and mitigations). */ - push %_ASM_ARG3 - - /* - * Save @regs, _ASM_ARG2 may be modified by vmx_update_host_rsp() and - * @regs is needed after VM-Exit to save the guest's register values. - */ push %_ASM_ARG2 =20 lea (%_ASM_SP), %_ASM_ARG2 @@ -107,6 +100,9 @@ SYM_FUNC_START(__vmx_vcpu_run) =20 ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL =20 + /* Reload @vmx, _ASM_ARG1 may be modified by vmx_update_host_rsp(). */ + mov WORD_SIZE(%_ASM_SP), %_ASM_DI + /* * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the * host's, write the MSR. @@ -115,7 +111,6 @@ SYM_FUNC_START(__vmx_vcpu_run) * there must not be any returns or indirect branches between this code * and vmentry. */ - mov 2*WORD_SIZE(%_ASM_SP), %_ASM_DI #ifdef CONFIG_X86_64 mov VMX_spec_ctrl(%rdi), %rdx cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx @@ -142,8 +137,8 @@ SYM_FUNC_START(__vmx_vcpu_run) * an LFENCE to stop speculation from skipping the wrmsr. */ =20 - /* Load @regs to RAX. */ - mov (%_ASM_SP), %_ASM_AX + /* Load @vmx to RAX. */ + mov WORD_SIZE(%_ASM_SP), %_ASM_AX =20 /* Load guest registers. Don't clobber flags. */ mov VCPU_RCX(%_ASM_AX), %_ASM_CX @@ -162,7 +157,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov VCPU_R14(%_ASM_AX), %r14 mov VCPU_R15(%_ASM_AX), %r15 #endif - /* Load guest RAX. This kills the @regs pointer! */ + /* Load guest RAX. This kills the @vmx pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX =20 /* @@ -172,7 +167,7 @@ SYM_FUNC_START(__vmx_vcpu_run) * do VERW. Else, do nothing (no mitigations needed/enabled). */ ALTERNATIVE_2 "", \ - __stringify(testl $VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO, WORD_SIZE(%= _ASM_SP); \ + __stringify(testl $VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO, (%_ASM_SP);= \ jz .Lskip_mmio_verw; \ VERW; \ .Lskip_mmio_verw:), \ @@ -180,7 +175,7 @@ SYM_FUNC_START(__vmx_vcpu_run) __stringify(VERW), X86_FEATURE_CLEAR_CPU_BUF_VM =20 /* Check @flags to see if VMLAUNCH or VMRESUME is needed. */ - testl $VMX_RUN_VMRESUME, WORD_SIZE(%_ASM_SP) + testl $VMX_RUN_VMRESUME, (%_ASM_SP) jz .Lvmlaunch =20 /* @@ -215,8 +210,8 @@ SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL) /* Temporarily save guest's RAX. */ push %_ASM_AX =20 - /* Reload @regs to RAX. */ - mov WORD_SIZE(%_ASM_SP), %_ASM_AX + /* Reload @vmx to RAX. */ + mov 2*WORD_SIZE(%_ASM_SP), %_ASM_AX =20 /* Save all guest registers, including RAX from the stack */ pop VCPU_RAX(%_ASM_AX) @@ -241,9 +236,6 @@ SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL) xor %ebx, %ebx =20 .Lclear_regs: - /* Discard @regs. The register is irrelevant, it just can't be RBX. */ - pop %_ASM_AX - /* * Clear all general purpose registers except RSP and RBX to prevent * speculative use of the guest's values, even those that are reloaded diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a29896a9ef14..7dd4aff29026 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7468,8 +7468,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vc= pu *vcpu, if (vcpu->arch.cr2 !=3D native_read_cr2()) native_write_cr2(vcpu->arch.cr2); =20 - vmx->fail =3D __vmx_vcpu_run(vmx, (unsigned long *)&vcpu->arch.regs, - flags); + vmx->fail =3D __vmx_vcpu_run(vmx, flags); =20 vcpu->arch.cr2 =3D native_read_cr2(); vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index db84e8001da5..53793d530a69 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -370,8 +370,7 @@ void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu); void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp); void vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, unsigned int flags); unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx); -bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, - unsigned int flags); +bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned int flags); void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu); =20 void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type, b= ool set); --=20 2.52.0 From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D0D03D6CD8 for ; Tue, 28 Apr 2026 11:05:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374318; cv=none; b=KVcdDuAK9QSj381aiQMzmYgEsWkJIeGN8eU8OQAoyhuLREGzO2bDhVA3a842jWALQw9rTEfKqfDls0uw1ZcewMWS3k2k7s6AVytTXrTKrBfqCdl6HyoINrTUTGjuf+yZiVl90lF+Lb8QWWH8yQQPM0hWthsoxNWNMnqvm705PmQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374318; c=relaxed/simple; bh=BPV3vT9eA04WNPHZTQqGt2+snx1/jYDEWCMTWyW1EFs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=cUnRyyzIyKSNrfmKzH0xmJLVMMYeXGu9NxA6LVjmt9pbUyImd6sMNjnP8R/Q3wE2189r6X85qcoHzHHptYsLfzOb3IkA1HxYA2B3w1RqqryxSQAccWeZk1bGy23QRra8i/qPR+0UYXYq/I07kPLwtC1h95jCTBadM1Bf5uheabs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=L46YMkI+; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="L46YMkI+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VNmP5dFTDai7i3tVbIrBGBZA/r4qew1n0ivtF6u/XX0=; b=L46YMkI+p6drYIpJiHMZ7SeLJZnuGkbShO2bsdbKY4ivjeicE6WH0AgvyrARpDbZKQY1b3 8EsN5c9yz1G3BNMMpvMtYtipheSIpZahUAh1wUJYm62fSKV8DOqsoFPxCzjoce5NvSSN4e 6NPA1KyVptTUdzk8Ev9DTpV5cERSCGU= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-31-cCMpog1NOU2qp_JIWWVEPw-1; Tue, 28 Apr 2026 07:05:12 -0400 X-MC-Unique: cCMpog1NOU2qp_JIWWVEPw-1 X-Mimecast-MFC-AGG-ID: cCMpog1NOU2qp_JIWWVEPw_1777374311 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0C76518002C0; Tue, 28 Apr 2026 11:05:11 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 46A78180047F; Tue, 28 Apr 2026 11:05:10 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 2/8] KVM: VMX: more cleanups to __vmx_vcpu_run Date: Tue, 28 Apr 2026 07:05:01 -0400 Message-ID: <20260428110507.11248-3-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" Slightly improve register allocation, loading vmx only once before vmlaunch/vmresume. This also makes the code slightly more similar to the one for AMD processors, in that both keep the pointer to struct vcpu_vmx or vcpu_svm in %rdi. The code for restoring the guest value of SPEC_CTRL is also the same for Intel and AMD. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/vmx/vmenter.S | 85 ++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 44 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 0a09288a8d29..efac31cedfde 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -98,11 +98,11 @@ SYM_FUNC_START(__vmx_vcpu_run) lea (%_ASM_SP), %_ASM_ARG2 call vmx_update_host_rsp =20 - ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL - /* Reload @vmx, _ASM_ARG1 may be modified by vmx_update_host_rsp(). */ mov WORD_SIZE(%_ASM_SP), %_ASM_DI =20 + ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL + /* * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the * host's, write the MSR. @@ -122,9 +122,9 @@ SYM_FUNC_START(__vmx_vcpu_run) mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx xor %eax, %ecx mov VMX_spec_ctrl + 4(%edi), %edx - mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edi - xor %edx, %edi - or %edi, %ecx + mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi + xor %edx, %esi + or %esi, %ecx je .Lspec_ctrl_done #endif mov $MSR_IA32_SPEC_CTRL, %ecx @@ -137,28 +137,25 @@ SYM_FUNC_START(__vmx_vcpu_run) * an LFENCE to stop speculation from skipping the wrmsr. */ =20 - /* Load @vmx to RAX. */ - mov WORD_SIZE(%_ASM_SP), %_ASM_AX - /* Load guest registers. Don't clobber flags. */ - mov VCPU_RCX(%_ASM_AX), %_ASM_CX - mov VCPU_RDX(%_ASM_AX), %_ASM_DX - mov VCPU_RBX(%_ASM_AX), %_ASM_BX - mov VCPU_RBP(%_ASM_AX), %_ASM_BP - mov VCPU_RSI(%_ASM_AX), %_ASM_SI - mov VCPU_RDI(%_ASM_AX), %_ASM_DI + mov VCPU_RAX(%_ASM_DI), %_ASM_AX + mov VCPU_RCX(%_ASM_DI), %_ASM_CX + mov VCPU_RDX(%_ASM_DI), %_ASM_DX + mov VCPU_RBX(%_ASM_DI), %_ASM_BX + mov VCPU_RBP(%_ASM_DI), %_ASM_BP + mov VCPU_RSI(%_ASM_DI), %_ASM_SI #ifdef CONFIG_X86_64 - mov VCPU_R8 (%_ASM_AX), %r8 - mov VCPU_R9 (%_ASM_AX), %r9 - mov VCPU_R10(%_ASM_AX), %r10 - mov VCPU_R11(%_ASM_AX), %r11 - mov VCPU_R12(%_ASM_AX), %r12 - mov VCPU_R13(%_ASM_AX), %r13 - mov VCPU_R14(%_ASM_AX), %r14 - mov VCPU_R15(%_ASM_AX), %r15 + mov VCPU_R8 (%_ASM_DI), %r8 + mov VCPU_R9 (%_ASM_DI), %r9 + mov VCPU_R10(%_ASM_DI), %r10 + mov VCPU_R11(%_ASM_DI), %r11 + mov VCPU_R12(%_ASM_DI), %r12 + mov VCPU_R13(%_ASM_DI), %r13 + mov VCPU_R14(%_ASM_DI), %r14 + mov VCPU_R15(%_ASM_DI), %r15 #endif - /* Load guest RAX. This kills the @vmx pointer! */ - mov VCPU_RAX(%_ASM_AX), %_ASM_AX + /* Load guest RDI. This kills the @vmx pointer! */ + mov VCPU_RDI(%_ASM_DI), %_ASM_DI =20 /* * Note, ALTERNATIVE_2 works in reverse order. If CLEAR_CPU_BUF_VM is @@ -207,29 +204,29 @@ SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL) UNWIND_HINT_RESTORE ENDBR =20 - /* Temporarily save guest's RAX. */ - push %_ASM_AX + /* Temporarily save guest's RDI. */ + push %_ASM_DI =20 - /* Reload @vmx to RAX. */ - mov 2*WORD_SIZE(%_ASM_SP), %_ASM_AX + /* Reload @vmx to RDI. */ + mov 2*WORD_SIZE(%_ASM_SP), %_ASM_DI =20 - /* Save all guest registers, including RAX from the stack */ - pop VCPU_RAX(%_ASM_AX) - mov %_ASM_CX, VCPU_RCX(%_ASM_AX) - mov %_ASM_DX, VCPU_RDX(%_ASM_AX) - mov %_ASM_BX, VCPU_RBX(%_ASM_AX) - mov %_ASM_BP, VCPU_RBP(%_ASM_AX) - mov %_ASM_SI, VCPU_RSI(%_ASM_AX) - mov %_ASM_DI, VCPU_RDI(%_ASM_AX) + /* Save all guest registers, including RDI from the stack */ + mov %_ASM_AX, VCPU_RAX(%_ASM_DI) + mov %_ASM_CX, VCPU_RCX(%_ASM_DI) + mov %_ASM_DX, VCPU_RDX(%_ASM_DI) + mov %_ASM_BX, VCPU_RBX(%_ASM_DI) + mov %_ASM_BP, VCPU_RBP(%_ASM_DI) + mov %_ASM_SI, VCPU_RSI(%_ASM_DI) + pop VCPU_RDI(%_ASM_DI) #ifdef CONFIG_X86_64 - mov %r8, VCPU_R8 (%_ASM_AX) - mov %r9, VCPU_R9 (%_ASM_AX) - mov %r10, VCPU_R10(%_ASM_AX) - mov %r11, VCPU_R11(%_ASM_AX) - mov %r12, VCPU_R12(%_ASM_AX) - mov %r13, VCPU_R13(%_ASM_AX) - mov %r14, VCPU_R14(%_ASM_AX) - mov %r15, VCPU_R15(%_ASM_AX) + mov %r8, VCPU_R8 (%_ASM_DI) + mov %r9, VCPU_R9 (%_ASM_DI) + mov %r10, VCPU_R10(%_ASM_DI) + mov %r11, VCPU_R11(%_ASM_DI) + mov %r12, VCPU_R12(%_ASM_DI) + mov %r13, VCPU_R13(%_ASM_DI) + mov %r14, VCPU_R14(%_ASM_DI) + mov %r15, VCPU_R15(%_ASM_DI) #endif =20 /* Clear return value to indicate VM-Exit (as opposed to VM-Fail). */ --=20 2.52.0 From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA75F3F20E6 for ; Tue, 28 Apr 2026 11:05:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374322; cv=none; b=hps3goWURNjH03p+1rcDHGOKK2ZQmMHwL4UgBNHznK69JnAuZtEwsH9FSbmGoPr0ftK5KOolQ8jpXDx3k/mOyX//fL/O36tCHcHdFBkTXTNSEHhVsboHTxPZmg0hsIE3cgQWzwbCGrDrKRd5yhtIOJfy9s/nj1pTZjpOL+k5be8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374322; c=relaxed/simple; bh=TaYpCTETe11C3Yfg/FrIsNKEV43YWUGQn1eYB47uKmg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Oxgrtnn84Ss1ARhbrh63nTva4aAWypybCqGvNhan0eCQF6IU56qjey/zU5ou4WYPfgS+NqOA0d1WHHa96UCHyuguP9aiuy1Ed2k5Uqvn71bENCMbD4igXfOEFkoZqg9EdNtC+HJ6HE7mHIcrwfoEynnXaqGm7FDDT3Aa4cD1Iww= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=CG99P2oT; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="CG99P2oT" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PbFMocHNKjLLQkLK9ZyuZA3bwif9NnNEuCbZf7qi6Yo=; b=CG99P2oTIdNk8AA5VfKMAv7UGFkEzh8oOLP4eQkqw5Am7KKMb/TToDhnUnWw9WPUPA5/KI ZEEzIZFDz/oecehlCQnPhXfjV2MbZOMCLsOD1t6OjxqDrWOa2KWLIpurXhb6wdSHEnKgK9 WheW6IMgaW2UeFbMRYMoSIwmoCM0Un4= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-346-g8tAIWebMLOc3_7d60GCdw-1; Tue, 28 Apr 2026 07:05:13 -0400 X-MC-Unique: g8tAIWebMLOc3_7d60GCdw-1 X-Mimecast-MFC-AGG-ID: g8tAIWebMLOc3_7d60GCdw_1777374312 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E70411956056; Tue, 28 Apr 2026 11:05:11 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 345F1180047F; Tue, 28 Apr 2026 11:05:11 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 3/8] KVM: SVM: prepare for making SPEC_CTRL switch common with VMX Date: Tue, 28 Apr 2026 07:05:02 -0400 Message-ID: <20260428110507.11248-4-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" Remove the local labels and restrict RESTORE_*_BODY to just the restore code itself. Since the alternatives are different between VMX and SVM, having labels in the per-vendor file and jumps in another would be too confusing. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/svm/vmenter.S | 54 +++++++++++++++++++++----------------- 1 file changed, 30 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S index d47c5c93c991..aa655b519fe0 100644 --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -39,10 +39,8 @@ ALTERNATIVE_2 "", \ "jmp 800f", X86_FEATURE_MSR_SPEC_CTRL, \ "", X86_FEATURE_V_SPEC_CTRL -801: .endm -.macro RESTORE_GUEST_SPEC_CTRL_BODY -800: +.macro RESTORE_GUEST_SPEC_CTRL_BODY guest_spec_ctrl:req, label:req /* * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the * host's, write the MSR. This is kept out-of-line so that the common @@ -53,24 +51,23 @@ * and vmentry. */ #ifdef CONFIG_X86_64 - mov SVM_spec_ctrl(%rdi), %rdx + mov \guest_spec_ctrl, %rdx cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx - je 801b + je \label movl %edx, %eax shr $32, %rdx #else - mov SVM_spec_ctrl(%edi), %eax + mov \guest_spec_ctrl, %eax mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx xor %eax, %ecx - mov SVM_spec_ctrl + 4(%edi), %edx + mov 4 + \guest_spec_ctrl, %edx mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi xor %edx, %esi or %esi, %ecx - je 801b + je \label #endif mov $MSR_IA32_SPEC_CTRL, %ecx wrmsr - jmp 801b .endm =20 .macro RESTORE_HOST_SPEC_CTRL @@ -78,10 +75,8 @@ ALTERNATIVE_2 "", \ "jmp 900f", X86_FEATURE_MSR_SPEC_CTRL, \ "", X86_FEATURE_V_SPEC_CTRL -901: .endm -.macro RESTORE_HOST_SPEC_CTRL_BODY spec_ctrl_intercepted:req -900: +.macro RESTORE_HOST_SPEC_CTRL_BODY guest_spec_ctrl:req, spec_ctrl_intercep= ted:req, label:req /* Same for after vmexit. */ mov $MSR_IA32_SPEC_CTRL, %ecx =20 @@ -92,28 +87,27 @@ cmpb $0, \spec_ctrl_intercepted jnz 998f rdmsr - movl %eax, SVM_spec_ctrl(%_ASM_DI) - movl %edx, SVM_spec_ctrl + 4(%_ASM_DI) + movl %eax, \guest_spec_ctrl + movl %edx, 4 + \guest_spec_ctrl 998: /* Now restore the host value of the MSR if different from the guest's. = */ #ifdef CONFIG_X86_64 mov PER_CPU_VAR(x86_spec_ctrl_current), %rdx - cmp SVM_spec_ctrl(%rdi), %rdx - je 901b + cmp \guest_spec_ctrl, %rdx + je \label movl %edx, %eax shr $32, %rdx #else mov PER_CPU_VAR(x86_spec_ctrl_current), %eax - mov SVM_spec_ctrl(%edi), %esi + mov \guest_spec_ctrl, %esi xor %eax, %esi mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edx - mov SVM_spec_ctrl + 4(%edi), %edi + mov 4 + \guest_spec_ctrl, %edi xor %edx, %edi or %edi, %esi - je 901b + je \label #endif wrmsr - jmp 901b .endm =20 #define SVM_CLEAR_CPU_BUFFERS \ @@ -162,6 +156,7 @@ SYM_FUNC_START(__svm_vcpu_run) =20 /* Clobbers RAX, RCX, RDX (and ESI on 32-bit), consumes RDI (@svm). */ RESTORE_GUEST_SPEC_CTRL +801: =20 /* * Use a single vmcb (vmcb01 because it's always valid) for @@ -242,6 +237,7 @@ SYM_FUNC_START(__svm_vcpu_run) * and RSP (pointer to @spec_ctrl_intercepted). */ RESTORE_HOST_SPEC_CTRL +901: =20 /* * Mitigate RETBleed for AMD/Hygon Zen uarch. RET should be @@ -295,8 +291,12 @@ SYM_FUNC_START(__svm_vcpu_run) pop %_ASM_BP RET =20 - RESTORE_GUEST_SPEC_CTRL_BODY - RESTORE_HOST_SPEC_CTRL_BODY (%_ASM_SP) +800: + RESTORE_GUEST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), 801b + jmp 801b +900: + RESTORE_HOST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), (%_ASM_SP), 901b + jmp 901b =20 10: cmpb $0, _ASM_RIP(virt_rebooting) jne 2b @@ -362,6 +362,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) =20 /* Clobbers RAX, RCX, and RDX (@hostsa), consumes RDI (@svm). */ RESTORE_GUEST_SPEC_CTRL +801: =20 /* Get svm->current_vmcb->pa into RAX. */ mov SVM_current_vmcb(%rdi), %rax @@ -378,6 +379,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) =20 /* Clobbers RAX, RCX, RDX, consumes RDI (@svm) and RSI (@spec_ctrl_interc= epted). */ RESTORE_HOST_SPEC_CTRL +901: =20 /* * Mitigate RETBleed for AMD/Hygon Zen uarch. RET should be @@ -391,8 +393,12 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) FRAME_END RET =20 - RESTORE_GUEST_SPEC_CTRL_BODY - RESTORE_HOST_SPEC_CTRL_BODY %sil +800: + RESTORE_GUEST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), 801b + jmp 801b +900: + RESTORE_HOST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), %sil, 901b + jmp 901b =20 3: cmpb $0, virt_rebooting(%rip) jne 2b --=20 2.52.0 From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 749542BEC52 for ; Tue, 28 Apr 2026 11:05:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374320; cv=none; b=P1wDgrNfmN254EUytzl+d6C5YiY6FepcNwYNVbWemRpxLt7V7fCjVkJwlSppa2W+ecWIOI2GnzDEPUNQIK5NPqEmNN/STzHYl2uoCHdk8fGa8AOZEDqaP9XIEfzKQGdiTeD99kSHaKikxrGVa7jwaTmGsbgfWIb6uO76rMsbgz4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374320; c=relaxed/simple; bh=L/bGiwcB2+tzqGnM1xrPKnFxS7fmHor44rVb5u+BC1o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ktgHrYL6vfBlAg3+HC4pBZtJ31+QlsKEfP4sVsQNkBXk3vSIHePUNYY8ryIIBgami+WF2iSd6FSPym1ySOXR9TWMlnYEd3WXyOj0YxBQVYfKBexBDePqBwoLjU1T0NX6GjSGaeW33y8+VINGPQhcMJaKtmUDKYrr5o8l53pCFxI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ifSBNWCB; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ifSBNWCB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fXJK+NILD++ngcNyFEJ+TF8xLG0zwpODMztUPty1LQU=; b=ifSBNWCBEg7FnWQ8AA/p9u3WVR7d3GaUPUqosdSnuVvXP5H/WV2JK5qnW/aWdVR5hM6rz5 WoRkfLK+1qvCNZmIvbJ+fuYoJNRnJJPMhWmLRWaeONInjrUA9mgCYMP5iGjwaE8P/OfB6q Jp5uZKH2Nsf10UbtURvBGqqJ+NmuYWE= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-661-CmA1RToIMaeLazl9YswofQ-1; Tue, 28 Apr 2026 07:05:14 -0400 X-MC-Unique: CmA1RToIMaeLazl9YswofQ-1 X-Mimecast-MFC-AGG-ID: CmA1RToIMaeLazl9YswofQ_1777374312 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CAAF119560AF; Tue, 28 Apr 2026 11:05:12 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 18FE0180047F; Tue, 28 Apr 2026 11:05:12 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 4/8] KVM: SVM: pass struct vcpu_svm to msr_write_intercepted Date: Tue, 28 Apr 2026 07:05:03 -0400 Message-ID: <20260428110507.11248-5-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" Remove an unnecessary difference between Intel and AMD. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/svm/svm.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index e7fdd7a9c280..0a2b85db8977 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -661,7 +661,7 @@ static void clr_dr_intercepts(struct vcpu_svm *svm) svm_mark_intercepts_dirty(svm); } =20 -static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) +static bool msr_write_intercepted(struct vcpu_svm *svm, u32 msr) { /* * For non-nested case: @@ -672,8 +672,7 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu= , u32 msr) * If the L02 MSR bitmap does not intercept the MSR, then we need to * save it. */ - void *msrpm =3D is_guest_mode(vcpu) ? to_svm(vcpu)->nested.msrpm : - to_svm(vcpu)->msrpm; + void *msrpm =3D is_guest_mode(&svm->vcpu) ? svm->nested.msrpm : svm->msrp= m; =20 return svm_test_msr_bitmap_write(msrpm, msr); } @@ -2764,7 +2763,7 @@ static bool sev_es_prevent_msr_access(struct kvm_vcpu= *vcpu, { return is_sev_es_guest(vcpu) && vcpu->arch.guest_state_protected && msr_info->index !=3D MSR_IA32_XSS && - !msr_write_intercepted(vcpu, msr_info->index); + !msr_write_intercepted(to_svm(vcpu), msr_info->index); } =20 static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) @@ -4414,7 +4413,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) { bool force_immediate_exit =3D run_flags & KVM_RUN_FORCE_IMMEDIATE_EXIT; struct vcpu_svm *svm =3D to_svm(vcpu); - bool spec_ctrl_intercepted =3D msr_write_intercepted(vcpu, MSR_IA32_SPEC_= CTRL); + bool spec_ctrl_intercepted =3D msr_write_intercepted(svm, MSR_IA32_SPEC_C= TRL); =20 trace_kvm_entry(vcpu, force_immediate_exit); =20 @@ -4556,7 +4555,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) =20 vcpu->arch.regs_avail &=3D ~SVM_REGS_LAZY_LOAD_SET; =20 - if (!msr_write_intercepted(vcpu, MSR_AMD64_PERF_CNTR_GLOBAL_CTL)) + if (!msr_write_intercepted(svm, MSR_AMD64_PERF_CNTR_GLOBAL_CTL)) rdmsrq(MSR_AMD64_PERF_CNTR_GLOBAL_CTL, vcpu_to_pmu(vcpu)->global_ctrl); =20 trace_kvm_exit(vcpu, KVM_ISA_SVM); --=20 2.52.0 From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D33B93F1662 for ; Tue, 28 Apr 2026 11:05:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374323; cv=none; b=pLAzFwnbR4YQoYjTTKo/VUAR8eQXswBWDVyKLvx0+EQsP7L4CKp3D773btdFZ4UmFQQ56wO4NWSNFrv12ozNHpkvzjE0y6p6vrWwCg0c0z+cDRqXqdFOP50vR03AGL9EY6UpONTBsIhwXPOmfygGIJZCdLhVO1lyAfJcOo1q1lc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374323; c=relaxed/simple; bh=iVYOcrFUI3Ov04/KKzzh8vuTxbSULs67wvkiQCfQMa0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=B7iXggrxo7CFaVkU6IuJhfFiE22kEfWM3agJ9KSQUm5OfCsKxIOXNav7X7AG7nQqlEOU9Q1kmOipN5zsZaT1dlmDtg2p9hOamoQo8vJSMib569XmJ/1/VNYva6WLDwBFNjXCW8c2BZ1VpFkMyOIMHbdP5rslX+uplecqw7loGQ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=T4a6TjOB; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="T4a6TjOB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kV3hPtV9u0e1Pa05EZEENQXGbfsjV0w0K2AzjGvgXRg=; b=T4a6TjOBjXpqHceqQgJahqIk/5ChZiyNWIeHFQsmGob/7ESfGs7zcMeFqTKl/MuCxHw5Ob ZHZxuFJuJCiRMl+lfwZ3spT7xGhyr+vLGPGnvZFuYj73URJrMQ9fhB8A+FK6yGC/c7o+tF 7NIquolmKdlHLqyjPcPkLID7FAdlovs= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-160-pdqXbKf6MA2QQyamMsPYGg-1; Tue, 28 Apr 2026 07:05:15 -0400 X-MC-Unique: pdqXbKf6MA2QQyamMsPYGg-1 X-Mimecast-MFC-AGG-ID: pdqXbKf6MA2QQyamMsPYGg_1777374314 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D4294195605A; Tue, 28 Apr 2026 11:05:13 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F1D13180047F; Tue, 28 Apr 2026 11:05:12 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 5/8] KVM: SVM: adopt the same VMX_RUN_* flags as VMX Date: Tue, 28 Apr 2026 07:05:04 -0400 Message-ID: <20260428110507.11248-6-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" Rename VMX_RUN_* to KVM_ENTER_* (to not confuse with KVM_RUN_* that already exists) and replace SVM's spec_ctrl_intercepted with that same bitmask. Note that KVM_ENTER_SAVE_SPEC_CTRL has the opposite polarity compared to spec_ctrl_intercepted. That is, the guest value of the MSR must be saved if it writes are *not* intercepted. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/svm/svm.c | 14 +++++++++----- arch/x86/kvm/svm/svm.h | 4 ++-- arch/x86/kvm/svm/vmenter.S | 23 ++++++++++++----------- arch/x86/kvm/vmenter.h | 9 +++++++++ arch/x86/kvm/vmx/run_flags.h | 9 --------- arch/x86/kvm/vmx/vmenter.S | 12 ++++++------ arch/x86/kvm/vmx/vmx.c | 13 +++++++------ arch/x86/kvm/vmx/vmx.h | 3 +-- 8 files changed, 46 insertions(+), 41 deletions(-) create mode 100644 arch/x86/kvm/vmenter.h delete mode 100644 arch/x86/kvm/vmx/run_flags.h diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 0a2b85db8977..353e464d6d94 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -50,6 +50,7 @@ =20 #include "trace.h" =20 +#include "vmenter.h" #include "svm.h" #include "svm_ops.h" =20 @@ -4377,7 +4378,7 @@ static fastpath_t svm_exit_handlers_fastpath(struct k= vm_vcpu *vcpu) return EXIT_FASTPATH_NONE; } =20 -static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_c= trl_intercepted) +static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, unsigned en= ter_flags) { struct svm_cpu_data *sd =3D per_cpu_ptr(&svm_data, vcpu->cpu); struct vcpu_svm *svm =3D to_svm(vcpu); @@ -4399,10 +4400,10 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_= vcpu *vcpu, bool spec_ctrl_in amd_clear_divider(); =20 if (is_sev_es_guest(vcpu)) - __svm_sev_es_vcpu_run(svm, spec_ctrl_intercepted, + __svm_sev_es_vcpu_run(svm, enter_flags, sev_es_host_save_area(sd)); else - __svm_vcpu_run(svm, spec_ctrl_intercepted); + __svm_vcpu_run(svm, enter_flags); =20 raw_local_irq_disable(); =20 @@ -4413,7 +4414,10 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm= _vcpu *vcpu, u64 run_flags) { bool force_immediate_exit =3D run_flags & KVM_RUN_FORCE_IMMEDIATE_EXIT; struct vcpu_svm *svm =3D to_svm(vcpu); - bool spec_ctrl_intercepted =3D msr_write_intercepted(svm, MSR_IA32_SPEC_C= TRL); + unsigned enter_flags =3D 0; + + if (!msr_write_intercepted(svm, MSR_IA32_SPEC_CTRL)) + enter_flags |=3D KVM_ENTER_SAVE_SPEC_CTRL; =20 trace_kvm_entry(vcpu, force_immediate_exit); =20 @@ -4496,7 +4500,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL)) x86_spec_ctrl_set_guest(svm->virt_spec_ctrl); =20 - svm_vcpu_enter_exit(vcpu, spec_ctrl_intercepted); + svm_vcpu_enter_exit(vcpu, enter_flags); =20 if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL)) x86_spec_ctrl_restore_host(svm->virt_spec_ctrl); diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index a10668d17a16..01f30f0b4eb1 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -1007,9 +1007,9 @@ static inline void sev_free_decrypted_vmsa(struct kvm= _vcpu *vcpu, struct vmcb_sa =20 /* vmenter.S */ =20 -void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepte= d, +void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, unsigned int flags, struct sev_es_save_area *hostsa); -void __svm_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepted); +void __svm_vcpu_run(struct vcpu_svm *svm, unsigned int flags); =20 #define DEFINE_KVM_GHCB_ACCESSORS(field) \ static __always_inline u64 kvm_ghcb_get_##field(struct vcpu_svm *svm) \ diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S index aa655b519fe0..2bc7ecb338a9 100644 --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -7,6 +7,7 @@ #include #include #include "kvm-asm-offsets.h" +#include "vmenter.h" =20 #define WORD_SIZE (BITS_PER_LONG / 8) =20 @@ -76,7 +77,7 @@ "jmp 900f", X86_FEATURE_MSR_SPEC_CTRL, \ "", X86_FEATURE_V_SPEC_CTRL .endm -.macro RESTORE_HOST_SPEC_CTRL_BODY guest_spec_ctrl:req, spec_ctrl_intercep= ted:req, label:req +.macro RESTORE_HOST_SPEC_CTRL_BODY guest_spec_ctrl:req, enter_flags:req, l= abel:req /* Same for after vmexit. */ mov $MSR_IA32_SPEC_CTRL, %ecx =20 @@ -84,8 +85,8 @@ * Load the value that the guest had written into MSR_IA32_SPEC_CTRL, * if it was not intercepted during guest execution. */ - cmpb $0, \spec_ctrl_intercepted - jnz 998f + testl $KVM_ENTER_SAVE_SPEC_CTRL, \enter_flags + jz 998f rdmsr movl %eax, \guest_spec_ctrl movl %edx, 4 + \guest_spec_ctrl @@ -115,8 +116,8 @@ =20 /** * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode - * @svm: struct vcpu_svm * - * @spec_ctrl_intercepted: bool + * @svm: struct vcpu_svm * + * @enter_flags: u32 */ SYM_FUNC_START(__svm_vcpu_run) push %_ASM_BP @@ -274,7 +275,7 @@ SYM_FUNC_START(__svm_vcpu_run) xor %r15d, %r15d #endif =20 - /* "Pop" @spec_ctrl_intercepted. */ + /* "Pop" @enter_flags. */ pop %_ASM_BX =20 pop %_ASM_BX @@ -335,8 +336,8 @@ SYM_FUNC_END(__svm_vcpu_run) =20 /** * __svm_sev_es_vcpu_run - Run a SEV-ES vCPU via a transition to SVM guest= mode - * @svm: struct vcpu_svm * - * @spec_ctrl_intercepted: bool + * @svm: struct vcpu_svm * + * @enter_flags: u32 */ SYM_FUNC_START(__svm_sev_es_vcpu_run) FRAME_BEGIN @@ -355,7 +356,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) =20 /* * Save volatile registers that hold arguments that are needed after - * #VMEXIT (RDI=3D@svm and RSI=3D@spec_ctrl_intercepted). + * #VMEXIT (RDI=3D@svm and RSI=3D@enter_flags). */ mov %rdi, SEV_ES_RDI (%rdx) mov %rsi, SEV_ES_RSI (%rdx) @@ -377,7 +378,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) /* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */ FILL_RETURN_BUFFER %rax, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT =20 - /* Clobbers RAX, RCX, RDX, consumes RDI (@svm) and RSI (@spec_ctrl_interc= epted). */ + /* Clobbers RAX, RCX, RDX, consumes RDI (@svm) and RSI (@enter_flags). */ RESTORE_HOST_SPEC_CTRL 901: =20 @@ -397,7 +398,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) RESTORE_GUEST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), 801b jmp 801b 900: - RESTORE_HOST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), %sil, 901b + RESTORE_HOST_SPEC_CTRL_BODY SVM_spec_ctrl(%_ASM_DI), %esi, 901b jmp 901b =20 3: cmpb $0, virt_rebooting(%rip) diff --git a/arch/x86/kvm/vmenter.h b/arch/x86/kvm/vmenter.h new file mode 100644 index 000000000000..29cdae650069 --- /dev/null +++ b/arch/x86/kvm/vmenter.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMENTER_H +#define __KVM_X86_VMENTER_H + +#define KVM_ENTER_VMRESUME BIT(0) +#define KVM_ENTER_SAVE_SPEC_CTRL BIT(1) +#define KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(2) + +#endif /* __KVM_X86_VMENTER_H */ diff --git a/arch/x86/kvm/vmx/run_flags.h b/arch/x86/kvm/vmx/run_flags.h deleted file mode 100644 index 6a87a12135fb..000000000000 --- a/arch/x86/kvm/vmx/run_flags.h +++ /dev/null @@ -1,9 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __KVM_X86_VMX_RUN_FLAGS_H -#define __KVM_X86_VMX_RUN_FLAGS_H - -#define VMX_RUN_VMRESUME BIT(0) -#define VMX_RUN_SAVE_SPEC_CTRL BIT(1) -#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(2) - -#endif /* __KVM_X86_VMX_RUN_FLAGS_H */ diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index efac31cedfde..38915ca16b05 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -7,7 +7,7 @@ #include #include #include "kvm-asm-offsets.h" -#include "run_flags.h" +#include "vmenter.h" =20 #define WORD_SIZE (BITS_PER_LONG / 8) =20 @@ -68,9 +68,9 @@ /** * __vmx_vcpu_run - Run a vCPU via a transition to VMX guest mode * @vmx: struct vcpu_vmx * - * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH - * VMX_RUN_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl - * VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO + * @flags: KVM_ENTER_VMRESUME: use VMRESUME instead of VMLAUNCH + * KVM_ENTER_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl + * KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO * * Returns: * 0 on VM-Exit, 1 on VM-Fail @@ -164,7 +164,7 @@ SYM_FUNC_START(__vmx_vcpu_run) * do VERW. Else, do nothing (no mitigations needed/enabled). */ ALTERNATIVE_2 "", \ - __stringify(testl $VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO, (%_ASM_SP);= \ + __stringify(testl $KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO, (%_ASM_SP= ); \ jz .Lskip_mmio_verw; \ VERW; \ .Lskip_mmio_verw:), \ @@ -172,7 +172,7 @@ SYM_FUNC_START(__vmx_vcpu_run) __stringify(VERW), X86_FEATURE_CLEAR_CPU_BUF_VM =20 /* Check @flags to see if VMLAUNCH or VMRESUME is needed. */ - testl $VMX_RUN_VMRESUME, (%_ASM_SP) + testl $KVM_ENTER_VMRESUME, (%_ASM_SP) jz .Lvmlaunch =20 /* diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 7dd4aff29026..a8039b0f9392 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -73,6 +73,7 @@ #include "x86_ops.h" #include "smm.h" #include "vmx_onhyperv.h" +#include "vmenter.h" #include "posted_intr.h" =20 #include "mmu/spte.h" @@ -964,12 +965,12 @@ static bool msr_write_intercepted(struct vcpu_vmx *vm= x, u32 msr) return vmx_test_msr_bitmap_write(vmx->loaded_vmcs->msr_bitmap, msr); } =20 -unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx) +unsigned int __vmx_vcpu_enter_flags(struct vcpu_vmx *vmx) { unsigned int flags =3D 0; =20 if (vmx->loaded_vmcs->launched) - flags |=3D VMX_RUN_VMRESUME; + flags |=3D KVM_ENTER_VMRESUME; =20 /* * If writes to the SPEC_CTRL MSR aren't intercepted, the guest is free @@ -977,11 +978,11 @@ unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vm= x) * it after vmexit and store it in vmx->spec_ctrl. */ if (!msr_write_intercepted(vmx, MSR_IA32_SPEC_CTRL)) - flags |=3D VMX_RUN_SAVE_SPEC_CTRL; + flags |=3D KVM_ENTER_SAVE_SPEC_CTRL; =20 if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) && kvm_vcpu_can_access_host_mmio(&vmx->vcpu)) - flags |=3D VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO; + flags |=3D KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO; =20 return flags; } @@ -7395,7 +7396,7 @@ void noinstr vmx_spec_ctrl_restore_host(struct vcpu_v= mx *vmx, if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL)) return; =20 - if (flags & VMX_RUN_SAVE_SPEC_CTRL) + if (flags & KVM_ENTER_SAVE_SPEC_CTRL) vmx->spec_ctrl =3D native_rdmsrq(MSR_IA32_SPEC_CTRL); =20 /* @@ -7586,7 +7587,7 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 ru= n_flags) kvm_wait_lapic_expire(vcpu); =20 /* The actual VMENTER/EXIT is in the .noinstr.text section. */ - vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_run_flags(vmx)); + vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_enter_flags(vmx)); =20 /* All fields are clean at this point */ if (kvm_is_using_evmcs()) { diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 53793d530a69..be6a1dc2f69f 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -15,7 +15,6 @@ #include "vmcs.h" #include "vmx_ops.h" #include "../cpuid.h" -#include "run_flags.h" #include "../mmu.h" #include "common.h" =20 @@ -369,7 +368,7 @@ struct vmx_uret_msr *vmx_find_uret_msr(struct vcpu_vmx = *vmx, u32 msr); void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu); void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp); void vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, unsigned int flags); -unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx); +unsigned int __vmx_vcpu_enter_flags(struct vcpu_vmx *vmx); bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned int flags); void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu); =20 --=20 2.52.0 From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 020993F1643 for ; Tue, 28 Apr 2026 11:05:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374321; cv=none; b=HesM4cOCYKQzecc4O3zKiowzRbG+AZEi72Bm7sQYRgNxBHdMQGfN9vHuseFLMxjpTYLyUGBUEUZH5eux+EwAVPZayhGn8lI12pAp45zSS6q0LS0eB5osOxV4PzNtr0s9ceq9UioMfK/EZgH6YIgZKZazdhXyYr+ojn1R0WeJimw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374321; c=relaxed/simple; bh=7Z/QcoB0quyknVpamQI7ee/ND7q601A3Yw2aUf9nN1U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=CmXg0SIJm7PTPt5ZGX+02kWFJODv2Tm0Gi254W2T0y/aO0C49f1tNUImO/lRi9EapLQkd92XqGhVC6TGNp76ApThb+RNJtFn1NVC6qb3OdpbwjLDeoi+b54g4fLDpyl+y/V/iK2YLwkHVTTa4EpIzbTaknj0oMcGCkOV4IARjNo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=I8uSxZiG; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="I8uSxZiG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Av4xmP7akAKu3weP+YCvPYBBTkWPaFe7RdTW3G3Kwfc=; b=I8uSxZiGJTvAlI6vqfftegnq/78+QcuOMFLZdAWue6F15icZY//4zlkR2KTKxRY0XdnrZX u5hpztDlO4N80niBSTpgC/XG/DyZfSFdxFU9A8fDvPXp9s5F0KB0uzv2kmA1xzZu01SP/2 ydrUuFjrTLXYKu/v/1Gq9fSoKyNW2cw= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-179-TzCKab-LN8iDOgacHZ40Wg-1; Tue, 28 Apr 2026 07:05:16 -0400 X-MC-Unique: TzCKab-LN8iDOgacHZ40Wg-1 X-Mimecast-MFC-AGG-ID: TzCKab-LN8iDOgacHZ40Wg_1777374314 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C1B3E18005B6; Tue, 28 Apr 2026 11:05:14 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0642A180047F; Tue, 28 Apr 2026 11:05:13 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 6/8] KVM: SVM: extract RESTORE_*_SPEC_CTRL_BODY out of svm/vmenter.S Date: Tue, 28 Apr 2026 07:05:05 -0400 Message-ID: <20260428110507.11248-7-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" Pure code movement, in preparation for using these macros for VMX as well. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/svm/vmenter.S | 62 ----------------------------------- arch/x86/kvm/vmenter.h | 66 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 62 deletions(-) diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S index 2bc7ecb338a9..f523d9e49839 100644 --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -41,35 +41,6 @@ "jmp 800f", X86_FEATURE_MSR_SPEC_CTRL, \ "", X86_FEATURE_V_SPEC_CTRL .endm -.macro RESTORE_GUEST_SPEC_CTRL_BODY guest_spec_ctrl:req, label:req - /* - * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the - * host's, write the MSR. This is kept out-of-line so that the common - * case does not have to jump. - * - * IMPORTANT: To avoid RSB underflow attacks and any other nastiness, - * there must not be any returns or indirect branches between this code - * and vmentry. - */ -#ifdef CONFIG_X86_64 - mov \guest_spec_ctrl, %rdx - cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx - je \label - movl %edx, %eax - shr $32, %rdx -#else - mov \guest_spec_ctrl, %eax - mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx - xor %eax, %ecx - mov 4 + \guest_spec_ctrl, %edx - mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi - xor %edx, %esi - or %esi, %ecx - je \label -#endif - mov $MSR_IA32_SPEC_CTRL, %ecx - wrmsr -.endm =20 .macro RESTORE_HOST_SPEC_CTRL /* No need to do anything if SPEC_CTRL is unset or V_SPEC_CTRL is set */ @@ -77,39 +48,6 @@ "jmp 900f", X86_FEATURE_MSR_SPEC_CTRL, \ "", X86_FEATURE_V_SPEC_CTRL .endm -.macro RESTORE_HOST_SPEC_CTRL_BODY guest_spec_ctrl:req, enter_flags:req, l= abel:req - /* Same for after vmexit. */ - mov $MSR_IA32_SPEC_CTRL, %ecx - - /* - * Load the value that the guest had written into MSR_IA32_SPEC_CTRL, - * if it was not intercepted during guest execution. - */ - testl $KVM_ENTER_SAVE_SPEC_CTRL, \enter_flags - jz 998f - rdmsr - movl %eax, \guest_spec_ctrl - movl %edx, 4 + \guest_spec_ctrl -998: - /* Now restore the host value of the MSR if different from the guest's. = */ -#ifdef CONFIG_X86_64 - mov PER_CPU_VAR(x86_spec_ctrl_current), %rdx - cmp \guest_spec_ctrl, %rdx - je \label - movl %edx, %eax - shr $32, %rdx -#else - mov PER_CPU_VAR(x86_spec_ctrl_current), %eax - mov \guest_spec_ctrl, %esi - xor %eax, %esi - mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edx - mov 4 + \guest_spec_ctrl, %edi - xor %edx, %edi - or %edi, %esi - je \label -#endif - wrmsr -.endm =20 #define SVM_CLEAR_CPU_BUFFERS \ ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF_VM diff --git a/arch/x86/kvm/vmenter.h b/arch/x86/kvm/vmenter.h index 29cdae650069..73f3adc301d9 100644 --- a/arch/x86/kvm/vmenter.h +++ b/arch/x86/kvm/vmenter.h @@ -6,4 +6,70 @@ #define KVM_ENTER_SAVE_SPEC_CTRL BIT(1) #define KVM_ENTER_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(2) =20 +#ifdef __ASSEMBLER__ +.macro RESTORE_GUEST_SPEC_CTRL_BODY guest_spec_ctrl:req, label:req + /* + * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the + * host's, write the MSR. This is kept out-of-line so that the common + * case does not have to jump. + * + * IMPORTANT: To avoid RSB underflow attacks and any other nastiness, + * there must not be any returns or indirect branches between this code + * and vmentry. + */ +#ifdef CONFIG_X86_64 + mov \guest_spec_ctrl, %rdx + cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx + je \label + movl %edx, %eax + shr $32, %rdx +#else + mov \guest_spec_ctrl, %eax + mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx + xor %eax, %ecx + mov 4 + \guest_spec_ctrl, %edx + mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi + xor %edx, %esi + or %esi, %ecx + je \label +#endif + mov $MSR_IA32_SPEC_CTRL, %ecx + wrmsr +.endm + +.macro RESTORE_HOST_SPEC_CTRL_BODY guest_spec_ctrl:req, enter_flags:req, l= abel:req + /* Same for after vmexit. */ + mov $MSR_IA32_SPEC_CTRL, %ecx + + /* + * Load the value that the guest had written into MSR_IA32_SPEC_CTRL, + * if it was not intercepted during guest execution. + */ + testl $KVM_ENTER_SAVE_SPEC_CTRL, \enter_flags + jz 998f + rdmsr + movl %eax, \guest_spec_ctrl + movl %edx, 4 + \guest_spec_ctrl +998: + /* Now restore the host value of the MSR if different from the guest's. = */ +#ifdef CONFIG_X86_64 + mov PER_CPU_VAR(x86_spec_ctrl_current), %rdx + cmp \guest_spec_ctrl, %rdx + je \label + movl %edx, %eax + shr $32, %rdx +#else + mov PER_CPU_VAR(x86_spec_ctrl_current), %eax + mov \guest_spec_ctrl, %esi + xor %eax, %esi + mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edx + mov 4 + \guest_spec_ctrl, %edi + xor %edx, %edi + or %edi, %esi + je \label +#endif + wrmsr +.endm + +#endif /* __ASSEMBLER__ */ #endif /* __KVM_X86_VMENTER_H */ --=20 2.52.0 From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7AC33F20E2 for ; Tue, 28 Apr 2026 11:05:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374321; cv=none; b=r23N6W9MIUlFZsGcYcesz7/w6BOQKob8tSD3Bbo0HaQpGEb9NxD6qPWVfuPEhEur5eVt2fi0cV5TgfSvTBCKGUur2Ujt9sXBqNjGdHy9l93lK1twyvUCL4QIIGS0ueItCK8fCGiapwpLpyS85SyYqdPVT+GXGQgyMR4HxwQ57NQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374321; c=relaxed/simple; bh=wOxgwsGcM6yALGqi2DIbw5uLad7lQI31kkgbWrcCcsw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=idLxcGSGlvkoOJtv+bL////4xiotAGbFZHGtZT3N50nrxfH+Z1LvnPZ4YLkGb2QqBQQ/95Sji1vfptJagirDMK4LbOKJuCGQrBtTOj5e0Mnc08XNDup+wX9wkfnzG8VCuPUkxvSsCzPYzt/oBSCKiqzH7qHWuQRZAPCyEnOSslE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=RMj8vXL0; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RMj8vXL0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374319; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=na9NIgRBc1v+C2rQ8Vz2B7eHy+7VwDVJw14Y1qRv1Sk=; b=RMj8vXL00sEJ2JY5I9CtSuNZA3AtcE52D+LhDhTcgtjmtBgl49q6IC4EPZ+IO4Mz8UqMZ7 uyS03uiG6jv/7QXucfkMH/jgx/vtsW9CnOLE/xJhfZcfkKTbKzKgBKXzVTO1IGZoIC4UqW wCmhleVFBhD83HfhUcHJJJYmc+JwNf0= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-508-r9qpgNORMhOaPkaoN41jHw-1; Tue, 28 Apr 2026 07:05:17 -0400 X-MC-Unique: r9qpgNORMhOaPkaoN41jHw-1 X-Mimecast-MFC-AGG-ID: r9qpgNORMhOaPkaoN41jHw_1777374315 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A483618005A8; Tue, 28 Apr 2026 11:05:15 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E74DC1800370; Tue, 28 Apr 2026 11:05:14 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 7/8] KVM: VMX: switch to RESTORE_GUEST_SPEC_CTRL_BODY Date: Tue, 28 Apr 2026 07:05:06 -0400 Message-ID: <20260428110507.11248-8-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" This has exactly the same expansion, so there is no change. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/vmx/vmenter.S | 31 ++++--------------------------- 1 file changed, 4 insertions(+), 27 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 38915ca16b05..7dd9bac0e454 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -101,35 +101,12 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Reload @vmx, _ASM_ARG1 may be modified by vmx_update_host_rsp(). */ mov WORD_SIZE(%_ASM_SP), %_ASM_DI =20 - ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL - /* - * SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the - * host's, write the MSR. - * - * IMPORTANT: To avoid RSB underflow attacks and any other nastiness, - * there must not be any returns or indirect branches between this code - * and vmentry. + * Unlike AMD there's no V_SPEC_CTRL here, so do not leave the body + * out of line. Clobbers RAX, RCX, RDX, RSI. */ -#ifdef CONFIG_X86_64 - mov VMX_spec_ctrl(%rdi), %rdx - cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx - je .Lspec_ctrl_done - movl %edx, %eax - shr $32, %rdx -#else - mov VMX_spec_ctrl(%edi), %eax - mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx - xor %eax, %ecx - mov VMX_spec_ctrl + 4(%edi), %edx - mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %esi - xor %edx, %esi - or %esi, %ecx - je .Lspec_ctrl_done -#endif - mov $MSR_IA32_SPEC_CTRL, %ecx - wrmsr - + ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL + RESTORE_GUEST_SPEC_CTRL_BODY VMX_spec_ctrl(%_ASM_DI), .Lspec_ctrl_done .Lspec_ctrl_done: =20 /* --=20 2.52.0 From nobody Mon Jun 15 07:33:54 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8EBE3EB7F0 for ; Tue, 28 Apr 2026 11:05:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374327; cv=none; b=mIFpaVc6l2NNRLoVNkd213y3fa+jkKc+2Ua17dKpmotBgLCxakA+lm2iW8ciHjMoKc1Oc06CdakKUp4UQ2JppZiw+lnAqSfGqExZuLw2wnu1GV1+NJ99y4f8IzW0oN61gWbiDNdc2rspEDtWOXTGfBrrGRQ1fFvkbYyjUvpMbSM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777374327; c=relaxed/simple; bh=cD9DIH7+ELj6GiyYjRm70PO91o/6JXizcPfqfiteNT4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hg9ZKKBOxbeuM69amlV2+QNltvrGByR6RAQnscfVcmf8+KABpQFAQ1cxb35o+KMBhopp225CHCdiFrQnE/jbeNQEk1TcP6eiaA63hJK/4qT8q6vztFEv55mLcze9jpyOfRraFOL78+N+tEBex2CifU8Hgb6ybo0ZWIdtOkCa/TQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ExtCmkcK; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ExtCmkcK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777374323; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HGRofkCiZMBgwi9NrIcS60E9J22RIyMII4UU2DpGcKc=; b=ExtCmkcK7fosay0OLs9kOWztTEue9SHkrQ3UVcFMgBIsBQWKmIU0Ep8acPHlbPIZFG5BOE L/dd8jFTVVFe9m1AlpwueaQZXIHqcP/ByC61jQq5WM7vghCaVPdvrW2pQWZ9CoH63jkg9A Qci84a8eouVjrUk9iQ/37GpuTBdufUk= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-349-YHHsoIsmPxGQjrM6mP2Jxw-1; Tue, 28 Apr 2026 07:05:18 -0400 X-MC-Unique: YHHsoIsmPxGQjrM6mP2Jxw-1 X-Mimecast-MFC-AGG-ID: YHHsoIsmPxGQjrM6mP2Jxw_1777374316 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8B38818001F7; Tue, 28 Apr 2026 11:05:16 +0000 (UTC) Received: from virtlab1023.lab.eng.rdu2.redhat.lab.eng.rdu2.redhat.com (virtlab1023.lab.eng.rdu2.redhat.com [10.8.1.187]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CC465180047F; Tue, 28 Apr 2026 11:05:15 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Binbin Wu , Uros Bizjak , "Chang S. Bae" , Sean Christopherson Subject: [PATCH 8/8] KVM: VMX: replace vmx_spec_ctrl_restore_host with RESTORE_HOST_SPEC_CTRL_BODY Date: Tue, 28 Apr 2026 07:05:07 -0400 Message-ID: <20260428110507.11248-9-pbonzini@redhat.com> In-Reply-To: <20260428110507.11248-1-pbonzini@redhat.com> References: <20260428110507.11248-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Content-Type: text/plain; charset="utf-8" Reuse the same assembly as SVM, just with alternatives instead of cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS). Note that Intel does need an LFENCE with eIBRS, unlike AMD's AutoIBRS. However, it is not needed for X86_FEATURE_KERNEL_IBRS because there are no conditional branches between FILL_RETURN_BUFFER and ret. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/vmenter.h | 9 +++++++-- arch/x86/kvm/vmx/vmenter.S | 28 ++++++++++++++++++++++------ arch/x86/kvm/vmx/vmx.c | 25 ------------------------- arch/x86/kvm/vmx/vmx.h | 1 - 4 files changed, 29 insertions(+), 34 deletions(-) diff --git a/arch/x86/kvm/vmenter.h b/arch/x86/kvm/vmenter.h index 73f3adc301d9..ba3f71449c62 100644 --- a/arch/x86/kvm/vmenter.h +++ b/arch/x86/kvm/vmenter.h @@ -55,7 +55,12 @@ #ifdef CONFIG_X86_64 mov PER_CPU_VAR(x86_spec_ctrl_current), %rdx cmp \guest_spec_ctrl, %rdx - je \label + /* + * For legacy IBRS, the IBRS bit always needs to be written after + * transitioning from a less privileged predictor mode, regardless of + * whether the guest/host values differ. + */ + ALTERNATIVE __stringify(je \label), "", X86_FEATURE_KERNEL_IBRS movl %edx, %eax shr $32, %rdx #else @@ -66,7 +71,7 @@ mov 4 + \guest_spec_ctrl, %edi xor %edx, %edi or %edi, %esi - je \label + ALTERNATIVE __stringify(je \label), "", X86_FEATURE_KERNEL_IBRS #endif wrmsr .endm diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 7dd9bac0e454..0c95dc5bc01d 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -105,9 +105,9 @@ SYM_FUNC_START(__vmx_vcpu_run) * Unlike AMD there's no V_SPEC_CTRL here, so do not leave the body * out of line. Clobbers RAX, RCX, RDX, RSI. */ - ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL - RESTORE_GUEST_SPEC_CTRL_BODY VMX_spec_ctrl(%_ASM_DI), .Lspec_ctrl_done -.Lspec_ctrl_done: + ALTERNATIVE "jmp .Lspec_ctrl_guest_done", "", X86_FEATURE_MSR_SPEC_CTRL + RESTORE_GUEST_SPEC_CTRL_BODY VMX_spec_ctrl(%_ASM_DI), .Lspec_ctrl_guest_d= one +.Lspec_ctrl_guest_done: =20 /* * Since vmentry is serializing on affected CPUs, there's no need for @@ -252,16 +252,32 @@ SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL) FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT,\ X86_FEATURE_RSB_VMEXIT_LITE =20 - pop %_ASM_ARG2 /* @flags */ - pop %_ASM_ARG1 /* @vmx */ + /* Clobbers RAX, RCX, RDX, RSI. */ + ALTERNATIVE "jmp .Lspec_ctrl_host_done", "", X86_FEATURE_MSR_SPEC_CTRL + mov WORD_SIZE(%_ASM_SP), %_ASM_DI + RESTORE_HOST_SPEC_CTRL_BODY VMX_spec_ctrl(%_ASM_DI), (%_ASM_SP), .Lspec_c= trl_host_done +.Lspec_ctrl_host_done: =20 - call vmx_spec_ctrl_restore_host + /* + * Halt speculation past a conditional wrmsr. Intel's eIBRS + * guarantees that the guest cannot control the RSB "once IBRS is + * set", but in the eIBRS case speculative execution past the 'je' + * can go all the way to the RET below while MSR_IA32_SPEC_CTRL + * still holds the guest value. + */ + ALTERNATIVE_2 "", "lfence", X86_FEATURE_MSR_SPEC_CTRL, \ + "", X86_FEATURE_KERNEL_IBRS =20 CLEAR_BRANCH_HISTORY_VMEXIT =20 /* Put return value in AX */ mov %_ASM_BX, %_ASM_AX =20 + /* Pop our saved arguments from the stack */ + pop %_ASM_BX + pop %_ASM_BX + + /* ... and then the callee-save registers */ pop %_ASM_BX #ifdef CONFIG_X86_64 pop %r12 diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a8039b0f9392..b033f611fa04 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7388,31 +7388,6 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vm= x, unsigned long host_rsp) } } =20 -void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, - unsigned int flags) -{ - u64 hostval =3D this_cpu_read(x86_spec_ctrl_current); - - if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL)) - return; - - if (flags & KVM_ENTER_SAVE_SPEC_CTRL) - vmx->spec_ctrl =3D native_rdmsrq(MSR_IA32_SPEC_CTRL); - - /* - * If the guest/host SPEC_CTRL values differ, restore the host value. - * - * For legacy IBRS, the IBRS bit always needs to be written after - * transitioning from a less privileged predictor mode, regardless of - * whether the guest/host values differ. - */ - if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) || - vmx->spec_ctrl !=3D hostval) - native_wrmsrq(MSR_IA32_SPEC_CTRL, hostval); - - barrier_nospec(); -} - static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu, bool force_immediate_exit) { diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index be6a1dc2f69f..f62007a5c2a7 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -367,7 +367,6 @@ void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu); struct vmx_uret_msr *vmx_find_uret_msr(struct vcpu_vmx *vmx, u32 msr); void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu); void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp); -void vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, unsigned int flags); unsigned int __vmx_vcpu_enter_flags(struct vcpu_vmx *vmx); bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned int flags); void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu); --=20 2.52.0