From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE3032EDD50 for ; Thu, 13 Nov 2025 23:37:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077072; cv=none; b=FA1/Y8KvAana3kazG3geh42VU8ZDIo3IEiA45KKKChgOotk8atNQedllf1X1n1IavMHD81aFWSo/HcM6ovG7GYpPio3NU1qB9zosORYZ0ndMTip5MDkaRzZMOhLdeKfibO5WPWb0d3mF+R/21nKu10ycRBVUXpwAEviax/JLQNI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077072; c=relaxed/simple; bh=emHSFxfSFp3NpgCgRJXcWu5Fv4DN2e5EIL4ys00ENJo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CbnTmFa4ewtEpuIUJNPYdJuu4pwkJGy/UnueAyx3bQl2rZo1/af92o+shWXNBbZoVPVrVJrt+Tvy2S74t/ighhgzXKzD4S5Gr2jHslEUgXF3aookz4nPPq9rbuH/13uRm5dLWo9PJsMJQ5HuYJCgF0Dkau4Gt3LyvJjQa/fEwjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ob92NJyF; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ob92NJyF" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-b55735710f0so2285467a12.2 for ; Thu, 13 Nov 2025 15:37:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077070; x=1763681870; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Sh+7hNyP4R/KweVRs59CPko6PNfW+Zn6L2KP2TqLiWM=; b=Ob92NJyFIQ/rHy875D5TQQnhrWrWxYGLNYKVjjD7JruiPUtjGEYHCVyfKrfmvNIx9x 9TUUV5b09y0HUna0P+I2SBrUHt79wOwC9nor+DjS/iN5rNvTbt5YTR3e2402KkKltInb dtsN4yq+EPNf72gbOOVhzIVsoF09yyVVvU2lT8haNnB186F+WG/DWDLmKdPJJVTUKhj5 Dw/FRD16n6+2OZdX1FgC/6E5NPwA5Ys1HHnqWYdGXMSJ/e/KNQ7hKaBfwyPKoEDELyDg 5IMf1tNXm9oCem5XabDQijDAYgi7njYllmxM7csKr5+huUHhJg1W9LKqlzPCINSVhU5W WG4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077070; x=1763681870; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Sh+7hNyP4R/KweVRs59CPko6PNfW+Zn6L2KP2TqLiWM=; b=SjJd7WZJUKqXfuIM8GbFy0BHBfsuvYC5fo8+L+/bn3s0dR0YZ77A4nyBadNqi8+7K7 3Lpv7OTPCO4eTYAUwMvkqva40KxzfrvQhfGNn8tbLekcn/2yzmmafS8CML7wO6ZRYb9v Wakh9XLoe54EX5KdLXpoBs4/kfpBQKNTUYfZXlLuazDYRdhjHd0wYz/D50coj0CykA9a CQMfmwwyFoeRnwUvDoB5ZoEy3OjC9BuIwufq7khNxmbeZ1ZrZxwG4DA1YP0FWNtLJNez nwAFRQabe4J3sMAQ9UoY+PBc5lNAuhla6+XArWWQJG077vyGPHf8T/tIcS/BfevE//SK LSRA== X-Forwarded-Encrypted: i=1; AJvYcCU29J1WQ1kSoINDyswk040sLwhU2bHq4/rolZwQyTbOCFH9nWA2Ph9bLCGvyMTpb954oCfeCsC772i9kKM=@vger.kernel.org X-Gm-Message-State: AOJu0YwB6gARAf521nAN89H3qp2V32iiIoMs9DwOKFPcMHD6bHSkOa6s 5/HerTjJPqSfCk3eWpIQNe2mcaKr/BOU89mPz167+87QkdGBH8BzLJPAYKAtbqY8MekACBUFQbw 4ANSVcg== X-Google-Smtp-Source: AGHT+IGzLeYY2NjluaD76UX5FC2xrHwdrPlvbPSe57TwgtfK0eclvj5UDY23Mz7ZaJE6y8GEb4pxALgqZUw= X-Received: from pgbcz2.prod.google.com ([2002:a05:6a02:2302:b0:bc3:ac3:8769]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:6da1:b0:34e:a1b2:a35f with SMTP id adf61e73a8af0-35ba2692da8mr1659168637.53.1763077070183; Thu, 13 Nov 2025 15:37:50 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:38 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-2-seanjc@google.com> Subject: [PATCH v5 1/9] KVM: VMX: Use on-stack copy of @flags in __vmx_vcpu_run() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When testing for VMLAUNCH vs. VMRESUME, use the copy of @flags from the stack instead of first moving it to EBX, and then propagating VMX_RUN_VMRESUME to RFLAGS.CF (because RBX is clobbered with the guest value prior to the conditional branch to VMLAUNCH). Stashing information in RFLAGS is gross, especially with the writer and reader being bifurcated by yet more gnarly assembly code. Opportunistically drop the SHIFT macros as they existed purely to allow the VM-Enter flow to use Bit Test. Suggested-by: Borislav Petkov Signed-off-by: Sean Christopherson Acked-by: Borislav Petkov (AMD) Reviewed-by: Brendan Jackman --- arch/x86/kvm/vmx/run_flags.h | 10 +++------- arch/x86/kvm/vmx/vmenter.S | 13 ++++--------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/vmx/run_flags.h b/arch/x86/kvm/vmx/run_flags.h index 2f20fb170def..6a87a12135fb 100644 --- a/arch/x86/kvm/vmx/run_flags.h +++ b/arch/x86/kvm/vmx/run_flags.h @@ -2,12 +2,8 @@ #ifndef __KVM_X86_VMX_RUN_FLAGS_H #define __KVM_X86_VMX_RUN_FLAGS_H =20 -#define VMX_RUN_VMRESUME_SHIFT 0 -#define VMX_RUN_SAVE_SPEC_CTRL_SHIFT 1 -#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO_SHIFT 2 - -#define VMX_RUN_VMRESUME BIT(VMX_RUN_VMRESUME_SHIFT) -#define VMX_RUN_SAVE_SPEC_CTRL BIT(VMX_RUN_SAVE_SPEC_CTRL_SHIFT) -#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(VMX_RUN_CLEAR_CPU_BUFFERS_F= OR_MMIO_SHIFT) +#define VMX_RUN_VMRESUME BIT(0) +#define VMX_RUN_SAVE_SPEC_CTRL BIT(1) +#define VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO BIT(2) =20 #endif /* __KVM_X86_VMX_RUN_FLAGS_H */ diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 574159a84ee9..93cf2ca7919a 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -92,7 +92,7 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Save @vmx for SPEC_CTRL handling */ push %_ASM_ARG1 =20 - /* Save @flags for SPEC_CTRL handling */ + /* Save @flags (used for VMLAUNCH vs. VMRESUME and mitigations). */ push %_ASM_ARG3 =20 /* @@ -101,9 +101,6 @@ SYM_FUNC_START(__vmx_vcpu_run) */ push %_ASM_ARG2 =20 - /* Copy @flags to EBX, _ASM_ARG3 is volatile. */ - mov %_ASM_ARG3L, %ebx - lea (%_ASM_SP), %_ASM_ARG2 call vmx_update_host_rsp =20 @@ -147,9 +144,6 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load @regs to RAX. */ mov (%_ASM_SP), %_ASM_AX =20 - /* Check if vmlaunch or vmresume is needed */ - bt $VMX_RUN_VMRESUME_SHIFT, %ebx - /* Load guest registers. Don't clobber flags. */ mov VCPU_RCX(%_ASM_AX), %_ASM_CX mov VCPU_RDX(%_ASM_AX), %_ASM_DX @@ -173,8 +167,9 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Clobbers EFLAGS.ZF */ CLEAR_CPU_BUFFERS =20 - /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ - jnc .Lvmlaunch + /* Check @flags to see if vmlaunch or vmresume is needed. */ + testl $VMX_RUN_VMRESUME, WORD_SIZE(%_ASM_SP) + jz .Lvmlaunch =20 /* * After a successful VMRESUME/VMLAUNCH, control flow "magically" --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAC6B32ABC6 for ; Thu, 13 Nov 2025 23:37:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077074; cv=none; b=ngFtGtHxVMBRhzG29OtyvFB68hErc4w6z0SmcIhGm4A5tRtffmFEJE87doqQrH7bvxyHD7bOuGXHZ4VCWgqmPA886t8Nc4qjseEftnbee/xXVaaZhKvNB2JvMsLqZRhHQO0SEeRJXGqzR7ECkOB8FQ+uIjgt7YG+Bt3H/x3QSxU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077074; c=relaxed/simple; bh=0rvBOQb7/KjgLAdCRV+FubR5NXzYQ6DUt4LwEzP/hw8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KF5GKPGd69WnkgLc3TFXkS4FQdddpQrTKTYuxHM/yD2V+rDucdipEhT2CobxKgu6WUxctglInP3nWmj+eGv5CIhQ2lG9oh5D0tJ5DgUpUapJawglUGVmL1D0D2sOSDVonXGYronefR3YJyWwvzhlGgAoTjxRXKrXWdTSnDzubE0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ut3ODSdc; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ut3ODSdc" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3438b1220bcso1936937a91.2 for ; Thu, 13 Nov 2025 15:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077072; x=1763681872; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=agrcyZXVeAoDNpOX3Y9kFCrU6uZpFNbMSLv5IvOZ2fY=; b=ut3ODSdcaarnt44PB8XxmhcCTFyM5KGtHpvLPKHMpGi82/PpdH3oQ4SLBemHXXN/pq ZMHK+HWs/SCoszR6NCVRaCvPZbDfmd88qDaf0AGOfbfeNxHG2vAjanfZ6QC2+nEX/IP2 jIBu00zfqhSAltS3NFOA8hjg5xwK3XHJzzfp5m+h4IYGmp4LCHUslJaQrUtwFbxv36zT KFTMAJCQY3OPXFzawTT+UEXK4Lh6JiFvfGJFEbCUVi5moUZh+IjrD5PkJNrR2VFbQLPe SvFZRD7vOr/UxbzzmqP2ag0e/QoC/WCMXvTInOyRx3gp4LbHTGSVfzg1FX6/0SLbJ+9e d0nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077072; x=1763681872; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=agrcyZXVeAoDNpOX3Y9kFCrU6uZpFNbMSLv5IvOZ2fY=; b=i7w1o2h0y1ZswKP+klU2p7sy9j31IQGiNCZiTVgxLYs5R5RFOheP8JNeYLBhxl81ue wFcsBOen3suv0xqRKmJnohsfcH3DXO9+yTOv4tk3M3hxYZDOHfylls6ub/+vCCbVxHmm E/9NCi3xQAL0PLzvCYs9ooNO+xmsXKt6G1lA/eAjExWsJgsQXhYAmL20bYOku3DRDrPo UWc3IXIWlZVzAP/IhzXX3kzbyi6MmiMVK8/iBdYyFUZ9f0RnS8pt06v1Kll6v/GdwPak SXZzXNWgiFm9k8jrw6XlCiU1o2i0ua2SNM67d/IBUyUEFmiKxfUZkdOQQMENjDtejqmy wZDA== X-Forwarded-Encrypted: i=1; AJvYcCX/aRxmf0H0nmqJHid+00Tvj8JAVTFk1xWJtovs+I+cO13GTcK67DDekxjOlhukPNx5pS1V8xHhhmaTNiw=@vger.kernel.org X-Gm-Message-State: AOJu0YxdK7CAEaA4fhhVePVb3VUtKKbcj/kUVIsogUpwsk7hZ3yBwybD RsndqAsC3eY8Bn8VeLBv98V4m755xsnMCThc3s3E4iaQptN7qAH2qCqGMJ69DoSkS+Us2cHcOa1 BiHzK3w== X-Google-Smtp-Source: AGHT+IGsIphPIOP+Qi59yhXJyxWnKWUiVNSnwW5PiF9bG5U2jX7YId16YUivlnVIwRZejZpt2zGOCzdWVj8= X-Received: from pjxg3.prod.google.com ([2002:a17:90a:dac3:b0:33b:52d6:e13e]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2781:b0:341:134:a962 with SMTP id 98e67ed59e1d1-343fa638d82mr908199a91.28.1763077071969; Thu, 13 Nov 2025 15:37:51 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:39 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-3-seanjc@google.com> Subject: [PATCH v5 2/9] x86/bugs: Use VM_CLEAR_CPU_BUFFERS in VMX as well From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Pawan Gupta TSA mitigation: d8010d4ba43e ("x86/bugs: Add a Transient Scheduler Attacks mitigation") introduced VM_CLEAR_CPU_BUFFERS for guests on AMD CPUs. Currently on Intel CLEAR_CPU_BUFFERS is being used for guests which has a much broader scope (kernel->user also). Make mitigations on Intel consistent with TSA. This would help handling the guest-only mitigations better in future. Signed-off-by: Pawan Gupta [sean: make CLEAR_CPU_BUF_VM mutually exclusive with the MMIO mitigation] Acked-by: Borislav Petkov (AMD) Signed-off-by: Sean Christopherson Reviewed-by: Brendan Jackman --- arch/x86/kernel/cpu/bugs.c | 13 +++++++++---- arch/x86/kvm/vmx/vmenter.S | 2 +- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index d7fa03bf51b4..c3a26532a209 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -194,7 +194,7 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); =20 /* * Controls CPU Fill buffer clear before VMenter. This is a subset of - * X86_FEATURE_CLEAR_CPU_BUF, and should only be enabled when KVM-only + * X86_FEATURE_CLEAR_CPU_BUF_VM, and should only be enabled when KVM-only * mitigation is required. */ DEFINE_STATIC_KEY_FALSE(cpu_buf_vm_clear); @@ -489,8 +489,8 @@ static enum rfds_mitigations rfds_mitigation __ro_after= _init =3D IS_ENABLED(CONFIG_MITIGATION_RFDS) ? RFDS_MITIGATION_AUTO : RFDS_MITIGATI= ON_OFF; =20 /* - * Set if any of MDS/TAA/MMIO/RFDS are going to enable VERW clearing - * through X86_FEATURE_CLEAR_CPU_BUF on kernel and guest entry. + * Set if any of MDS/TAA/MMIO/RFDS are going to enable VERW clearing on ex= it to + * userspace *and* on entry to KVM guests. */ static bool verw_clear_cpu_buf_mitigation_selected __ro_after_init; =20 @@ -536,6 +536,7 @@ static void __init mds_apply_mitigation(void) if (mds_mitigation =3D=3D MDS_MITIGATION_FULL || mds_mitigation =3D=3D MDS_MITIGATION_VMWERV) { setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || smt_mitigations =3D=3D SMT_MITIGATIONS_ON)) cpu_smt_disable(false); @@ -647,6 +648,7 @@ static void __init taa_apply_mitigation(void) * present on host, enable the mitigation for UCODE_NEEDED as well. */ setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM); =20 if (taa_nosmt || smt_mitigations =3D=3D SMT_MITIGATIONS_ON) cpu_smt_disable(false); @@ -748,6 +750,7 @@ static void __init mmio_apply_mitigation(void) */ if (verw_clear_cpu_buf_mitigation_selected) { setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM); static_branch_disable(&cpu_buf_vm_clear); } else { static_branch_enable(&cpu_buf_vm_clear); @@ -839,8 +842,10 @@ static void __init rfds_update_mitigation(void) =20 static void __init rfds_apply_mitigation(void) { - if (rfds_mitigation =3D=3D RFDS_MITIGATION_VERW) + if (rfds_mitigation =3D=3D RFDS_MITIGATION_VERW) { setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM); + } } =20 static __init int rfds_parse_cmdline(char *str) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 93cf2ca7919a..7e7bb9b7162f 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -165,7 +165,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov VCPU_RAX(%_ASM_AX), %_ASM_AX =20 /* Clobbers EFLAGS.ZF */ - CLEAR_CPU_BUFFERS + VM_CLEAR_CPU_BUFFERS =20 /* Check @flags to see if vmlaunch or vmresume is needed. */ testl $VMX_RUN_VMRESUME, WORD_SIZE(%_ASM_SP) --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7647D32D0EE for ; Thu, 13 Nov 2025 23:37:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077076; cv=none; b=Zn8KckO3sNEQqHUHxUU7K7sZRHsDECmAgP11IXmovSZCBkZf0oMPsczO6MpuWa8nyfeyZ4hilQQTONEXACQxrC/3xgxo/Krr4UoNr6+nqSGbc69J8fjSLeUlgnbFqhLp/qUPRu5gahVOBwW8jxRsy15ndy1J4tcSax+fvsGKfkE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077076; c=relaxed/simple; bh=9/ZEG9qS0v5yHmDM5mNncrn+pJi6DDL3CwwPd50ORBk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dQYhLp/k8HDgctyv+MpeuQ1nqtaqmySur4WZTOhTuuAn/nKyCbS1gWGpB5hkOwln4158M7DYUB28kNmJKXo1qLKaGd30dfNECsdP9eQArsuvZTWDtn3S0CRtRxU729jTjuPgGu4Bq3dKlsTAhaW7U9nZZGcr+kn/oF4X+deEDpg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CM8oshXT; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CM8oshXT" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-297fbfb4e53so26638925ad.1 for ; Thu, 13 Nov 2025 15:37:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077074; x=1763681874; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=jX3Feop83q2z4Q+hCWNf+/7P2WvpoKPC/BxcXVensJI=; b=CM8oshXTrd4XHk/baxaU0GMqOxQlqdrluN59AhGsW1YbkYpMx6pa0n2F8RK7UtYr3m /EWY8ba/0bv76djwtHkonpKPqJHDcf6dTRMgkcvqj0tX6VnnTFV2/paAHY+xBf95WmoX 2Xu75IlTklSzYmYrucrJ2flo67fJLIZf5Qf0DxTg8oE64XTYBWwlPLcSV+2H+szuRdkH qxQo9dOimE8WXzI30bLZTzMlHMmDAVwLxYqzb3SOKgL4uWkC339LC6S72Xg/whzWGvVW MA7fEIZa6va2Ugy5zyGEtRl8k2s+K+R7M9Cp1fIFLV/VhC6C1CZj7sPHvHNsO/XfqIOX Cs8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077074; x=1763681874; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jX3Feop83q2z4Q+hCWNf+/7P2WvpoKPC/BxcXVensJI=; b=svG+ynSUxsTjsbIVo01J26vJtVnBisFsRe7gCerWNVZ9rJZgnzA9GcbH1T/zvZMPHE j0KH+JcIyxFH33j9f+vb7Pf27BpeFw4DELGAPX6WjeiReL/9oyJ5M/2E/kMRD3g65VmO YgRoFgFer2vwIeeN+EJZdDwOQesAYDk57LxK0dCGhQwxsRZobXm/Jc75CAAB0eX28OA0 TL9XkGA6ZIT2qOYeYOdreATtNfcLfTHXhY9ATUglqi6ZFeIF/gY4cBhdI43ySAzk2x1d D5tamHE364tUP8sApNlZX8t0/qZPAEd6Oixc1uW6mE+zvntdRt+1o7H6DWT6qk/woIcL Rgsw== X-Forwarded-Encrypted: i=1; AJvYcCXpf6Apaa1hC2fBF/9sVYFl3x1wFaL5xoyQ6oIyvbL+BxtM3R1WZRNYmuvHx/exl3vYoGNqtbayaJe0X/U=@vger.kernel.org X-Gm-Message-State: AOJu0YxP1dT6JKhDO6Ttp2dM1kZ22EvLf+IenzDUccvZAqnNMusCresf +LKJ45J5LM5UprJofDz80dJmQNa142xTwckDyUgICXwqjpWuON45IQrZ1yIMRyCgxscnKfCVJ6w nWjAZ0g== X-Google-Smtp-Source: AGHT+IHJOzZDPILzi8nste2JY0ljSD09D7AMJE4IfZRr7nMUO63lLcqgpVvpjyngQFZ2bq/QcuhX1dI5i0Q= X-Received: from plrq24.prod.google.com ([2002:a17:902:b118:b0:298:a8:6ab2]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:b0d:b0:295:70cc:8ec4 with SMTP id d9443c01a7336-2986a74b360mr8908115ad.51.1763077073730; Thu, 13 Nov 2025 15:37:53 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:40 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-4-seanjc@google.com> Subject: [PATCH v5 3/9] x86/bugs: Decouple ALTERNATIVE usage from VERW macro definition From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Decouple the use of ALTERNATIVE from the encoding of VERW to clear CPU buffers so that KVM can use ALTERNATIVE_2 to handle "always clear buffers" and "clear if guest can access host MMIO" in a single statement. No functional change intended. Reviewed-by: Brendan Jackman Reviewed-by: Pawan Gupta Signed-off-by: Sean Christopherson --- arch/x86/include/asm/nospec-branch.h | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index 08ed5a2e46a5..8b4885a1b2ef 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -308,24 +308,24 @@ * CFLAGS.ZF. * Note: Only the memory operand variant of VERW clears the CPU buffers. */ -.macro __CLEAR_CPU_BUFFERS feature #ifdef CONFIG_X86_64 - ALTERNATIVE "", "verw x86_verw_sel(%rip)", \feature +#define VERW verw x86_verw_sel(%rip) #else - /* - * In 32bit mode, the memory operand must be a %cs reference. The data - * segments may not be usable (vm86 mode), and the stack segment may not - * be flat (ESPFIX32). - */ - ALTERNATIVE "", "verw %cs:x86_verw_sel", \feature +/* + * In 32bit mode, the memory operand must be a %cs reference. The data seg= ments + * may not be usable (vm86 mode), and the stack segment may not be flat (E= SPFIX32). + */ +#define VERW verw %cs:x86_verw_sel #endif -.endm =20 +#define __CLEAR_CPU_BUFFERS __stringify(VERW) + +/* If necessary, emit VERW on exit-to-userspace to clear CPU buffers. */ #define CLEAR_CPU_BUFFERS \ - __CLEAR_CPU_BUFFERS X86_FEATURE_CLEAR_CPU_BUF + ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF =20 #define VM_CLEAR_CPU_BUFFERS \ - __CLEAR_CPU_BUFFERS X86_FEATURE_CLEAR_CPU_BUF_VM + ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF_VM =20 #ifdef CONFIG_X86_64 .macro CLEAR_BRANCH_HISTORY --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3981932E690 for ; Thu, 13 Nov 2025 23:37:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077077; cv=none; b=mhvln2DhVJ1haCqA657dByDQDs+lZOZm9ubX32+ugolMOc43+lsK3EdRcIWU01hIodlm4LLcRtTqaP2BsI/KGI29KJMWBWB77uL3EUICoVnzdTgCVzHWgp5xE4QVooDWxur/SORxPP8RpKKC0kFcYHTA1UEqWL/sbRU3qWWRaQs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077077; c=relaxed/simple; bh=fW3qCCORzgxNu27cA27QxxVTE3InArhlK3jK46oMq9o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Dhp5PWW/6YQao4hMHO34n+Znhnnugirb2IMKOZZ1WhdxOLZtBGeDNnMr7UHnUZeTRT21VXarKLvLxIYgRGmw/n51VOv0x3LPl0yLvtjAXVp8+tnJn1ICB69UnLMFWkG80n2IQjC0gKq4vzIpxnNjpe14zqxVWMrpgyRyqpZKLSg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Cs4xnlus; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Cs4xnlus" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3437863d0easo1422246a91.0 for ; Thu, 13 Nov 2025 15:37:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077075; x=1763681875; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=0jw2KVOdakkapVYKbvuvd68x1KugquyK3xRcPayLfkA=; b=Cs4xnlusvg5QypoYiOAbOkW9RDEO/5wMwUsbbWvjV+WOVfV79Laom5dY5hhlcHSVFC pNrm/Gb7f5iAaWWT8aVfhdGyvMez+w/x+OwDBAdB3s89MRF8zxgPSaglGSOAneAF+GbA Ttb36DxnUx8BhMjlb5v7H7J3bRqImzeJbTEFwhNc8ug7pS4qOjKDF87dLAnXYMOMeO/R vOKhDGuW2rx0TadmlmFzt/ZcME6PF+vvyZuRSWL2NPadkwglyRGZdpmIjLGllMhMg+Bd 0YEHJpNeEkmnIa+Ok1dPSGpXguYS5ziHwGZtxG8RtI/OkArg4ZOZHwZ5le7iKNdDyQz+ v2dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077075; x=1763681875; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0jw2KVOdakkapVYKbvuvd68x1KugquyK3xRcPayLfkA=; b=qrwes66e8Ty/4+6SZ6YUlvfZe+Gj2ODJTFDfoXbbit0/ZlkxVZxC2Rb+/8EuNvg4w5 djMi1GwK0L57B0GKpTIoD3E9A8/cfwlhIvC4ArRzGpAUhrsijW7uV5/z8C3RxzDciWMM Q+idQr344xM5e6ZKatmpUhqmcLzLJ1+0MjzZw3+E5YXWhFEAphSUoZupjdl/YSmjUjgU t+rfnNkUDJ/A/9u1My6TmTqTIZvL/stMOlxLdn8Ur9Jo3iFJ2UtCUZXnmQ1rIr8zZ4JE qmEELRpkz55qgMymSwT/527Gn02y7c5Zw9OlIo3FZbQx66X/SlSYv4LdahGT/ioVBWkW SX0w== X-Forwarded-Encrypted: i=1; AJvYcCWUl0ft+uR971ak/DlFAUza+Opq+5TZOeJ7/Gfsi2OZHavaDoOrJ1C6dFPqjIdIJKvWRhUUrMcj/JSMs4g=@vger.kernel.org X-Gm-Message-State: AOJu0YwbCHKIev2giMe8BYtXEV+9EOLI3IuB7yxzBYStDHxKvXcThUwA 5q8JZ0nuPsiugrsxw/fUIA+K4ggtSV49UmmHYbPApmLqp+FibxykkP0eeQTjkLonLWzH8o2EjeW 9f3TrbQ== X-Google-Smtp-Source: AGHT+IFPQfuRNRc3ddrbv5kK3XNrpJhedl4e0uPKVuATUET7WelUE8oHyLa/YRZ77aZ7uz0c4GLtuWkKbmo= X-Received: from pjbdb12.prod.google.com ([2002:a17:90a:d64c:b0:33b:dbe2:7682]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3bd0:b0:341:8b2b:43c with SMTP id 98e67ed59e1d1-343fa63784fmr1024435a91.18.1763077075560; Thu, 13 Nov 2025 15:37:55 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:41 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-5-seanjc@google.com> Subject: [PATCH v5 4/9] x86/bugs: Use an x86 feature to track the MMIO Stale Data mitigation From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert the MMIO Stale Data mitigation tracking from a static branch into an x86 feature flag so that it can be used via ALTERNATIVE_2 in KVM. No functional change intended. Reviewed-by: Pawan Gupta Reviewed-by: Brendan Jackman Signed-off-by: Sean Christopherson --- arch/x86/include/asm/cpufeatures.h | 5 +++++ arch/x86/include/asm/nospec-branch.h | 2 -- arch/x86/kernel/cpu/bugs.c | 11 +---------- arch/x86/kvm/mmu/spte.c | 2 +- arch/x86/kvm/vmx/vmx.c | 4 ++-- 5 files changed, 9 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpuf= eatures.h index 7129eb44adad..646d2a77a2e2 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -501,6 +501,11 @@ #define X86_FEATURE_ABMC (21*32+15) /* Assignable Bandwidth Monitoring Co= unters */ #define X86_FEATURE_MSR_IMM (21*32+16) /* MSR immediate form instructions= */ #define X86_FEATURE_X2AVIC_EXT (21*32+17) /* AMD SVM x2AVIC support for 4= k vCPUs */ +#define X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO (21*32+18) /* + * Clear CPU buffers before VM-Enter if the vCPU + * can access host MMIO (ignored for all intents + * and purposes if CLEAR_CPU_BUF_VM is set). + */ =20 /* * BUG word(s) diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index 8b4885a1b2ef..b1b1811216c5 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -580,8 +580,6 @@ DECLARE_STATIC_KEY_FALSE(cpu_buf_idle_clear); =20 DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); =20 -DECLARE_STATIC_KEY_FALSE(cpu_buf_vm_clear); - extern u16 x86_verw_sel; =20 #include diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index c3a26532a209..70bb48ab46d7 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -192,14 +192,6 @@ EXPORT_SYMBOL_GPL(cpu_buf_idle_clear); */ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); =20 -/* - * Controls CPU Fill buffer clear before VMenter. This is a subset of - * X86_FEATURE_CLEAR_CPU_BUF_VM, and should only be enabled when KVM-only - * mitigation is required. - */ -DEFINE_STATIC_KEY_FALSE(cpu_buf_vm_clear); -EXPORT_SYMBOL_GPL(cpu_buf_vm_clear); - #undef pr_fmt #define pr_fmt(fmt) "mitigations: " fmt =20 @@ -751,9 +743,8 @@ static void __init mmio_apply_mitigation(void) if (verw_clear_cpu_buf_mitigation_selected) { setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM); - static_branch_disable(&cpu_buf_vm_clear); } else { - static_branch_enable(&cpu_buf_vm_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO); } =20 /* diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 37647afde7d3..85a0473809b0 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -292,7 +292,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_pa= ge *sp, mark_page_dirty_in_slot(vcpu->kvm, slot, gfn); } =20 - if (static_branch_unlikely(&cpu_buf_vm_clear) && + if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) && !kvm_vcpu_can_access_host_mmio(vcpu) && kvm_is_mmio_pfn(pfn, &is_host_mmio)) kvm_track_host_mmio_mapping(vcpu); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 6f374c815ce2..18d9e0eacd0f 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -903,7 +903,7 @@ unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx) if (!msr_write_intercepted(vmx, MSR_IA32_SPEC_CTRL)) flags |=3D VMX_RUN_SAVE_SPEC_CTRL; =20 - if (static_branch_unlikely(&cpu_buf_vm_clear) && + if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) && kvm_vcpu_can_access_host_mmio(&vmx->vcpu)) flags |=3D VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO; =20 @@ -7352,7 +7352,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vc= pu *vcpu, */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&cpu_buf_vm_clear) && + else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) && (flags & VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO)) x86_clear_cpu_buffers(); =20 --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A983932E73A for ; Thu, 13 Nov 2025 23:37:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077079; cv=none; b=urVGLNFZm/jk1MAuV9QJXzpV9GMOUAYFe4DTt1S3IRtKsvAFaV8gvM0SR6FFlk4pAIjuWqqUQS3DaYIv3PhqBNK7rTV27AZFkky4mbaQfhtTpwFGDqhqW+4aX7ldojVeNwU/KtRe88EaJa3yRNOBuD/9L1Sl+31FvKGa9hDq89k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077079; c=relaxed/simple; bh=d4Ss4Bie//Rr0NehfKTBbGKRdKpFqMe2NJQy4QCkjd4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WtHp3XzEGgVK+RHivLyTFGsRQko58s/n1rS+NXgZrNtsZvbSv6rSWTitn7CYk3htUCq+C+5FaZqy0ejxAe/dq4B2C91drv296q9rdjBr8XngoUX7+Pcm8hTUfCUKqb1a6ZIxz7ew7+ni4iWLGsx2TFU8NY4XcDt3foi3yXW3dSk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=S39ukbzX; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="S39ukbzX" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-343fb64cea6so360757a91.3 for ; Thu, 13 Nov 2025 15:37:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077077; x=1763681877; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Y0gZwwuvfs6rvtvGFUKGkBP9OUyn1AHhc63Drntnn+4=; b=S39ukbzXq+OBsLg8PFyZyDrf9L3fi/ka/aRz7LJwa17O76kIwaYjHAGf3ZBwiV2cuk LbTa4BloJagxmo1Eni9VsenKpbJJ8tStZN5eheOg/Qpi/SM2h1AVxyTkxY2Pv8Xv4tSq gNaKhrqqWO+PAjkoZVB//NuizK9v4mGKXhZcVIZVxFFnJ6nbBc14hJYXbUvgjH6ZNfgX YPongXQvAQCgIIMPE+HHBA38DhAgdiM+nOSZrFuuZycNLIh2HTOrvydXG1TtTw5bJdYC 52nwU4g7fBA8T/+FOK4320L10xHrSs6FT9a8HUMPhZlXBjnq1IzkEW6CMOhP0yebT6Ke GK1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077077; x=1763681877; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Y0gZwwuvfs6rvtvGFUKGkBP9OUyn1AHhc63Drntnn+4=; b=HlFcdOSUT6fGUHN86r1lFvvyjb7Pou2+sWpHjm7lN/w/v/7cRcIKYBgcyanaknjiE9 ib1FaqftkgpO8hYOSXuIuxwskT5srDdvsPFOZ8UBfxEH8h2giQKYOksRg3c4hIgh4hGj muSOnyG+RTd3omZeXAZynKgD8Sh7+zKmur0hE+PvvLLZ17EUsRudTdWjiZY4NVw4m6t/ cIcT9nX9QAjug07dyiRl8/5CNExlr9hhNy1liE1ZveejJ24KP6tMXF0Y/CxURxFKvP7u T5Ofvj6nUlscdGT06L2rdIkE1zt/SurdV7DCsauSLFCzurLELrXGdipWMeCeL/1w3d3d 6/lw== X-Forwarded-Encrypted: i=1; AJvYcCUzsbogIdhvue8f4pO+nMPuN4uuUJOwh3yRsbH+T4iHy2rUjJwM6hJ+5j9dAkcvku9ReNVcCDZoK/RSf6s=@vger.kernel.org X-Gm-Message-State: AOJu0Yy/M7sO4GNW4z80jLy4p2sKbk8kdvlNz5H1zAQVBmgWh0xEqprH ZD7ZcmmhyReYqDIDWL6fibacWwJYq5op9CNdqWGPGtTwsSas+fqAvvIKKHpdbVm6Rqf1umqDUSv hh17bXQ== X-Google-Smtp-Source: AGHT+IF/MsWtitpYBRrB7SlBYDAk36Vj1bzh4ME0GTQ/U++QGLuF+9AXN185HkhWGohHoPbQawzS+yjMke4= X-Received: from pjk5.prod.google.com ([2002:a17:90b:5585:b0:343:6849:31ae]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3dcb:b0:32e:528c:60ee with SMTP id 98e67ed59e1d1-343fa62bf59mr1148602a91.24.1763077077060; Thu, 13 Nov 2025 15:37:57 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:42 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-6-seanjc@google.com> Subject: [PATCH v5 5/9] KVM: VMX: Handle MMIO Stale Data in VM-Enter assembly via ALTERNATIVES_2 From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rework the handling of the MMIO Stale Data mitigation to clear CPU buffers immediately prior to VM-Enter, i.e. in the same location that KVM emits a VERW for unconditional (at runtime) clearing. Co-locating the code and using a single ALTERNATIVES_2 makes it more obvious how VMX mitigates the various vulnerabilities. Deliberately order the alternatives as: 0. Do nothing 1. Clear if vCPU can access MMIO 2. Clear always since the last alternative wins in ALTERNATIVES_2(), i.e. so that KVM will honor the strictest mitigation (always clear CPU buffers) if multiple mitigations are selected. E.g. even if the kernel chooses to mitigate MMIO Stale Data via X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO, another mitigation may enable X86_FEATURE_CLEAR_CPU_BUF_VM, and that other thing needs to win. Note, decoupling the MMIO mitigation from the L1TF mitigation also fixes a mostly-benign flaw where KVM wouldn't do any clearing/flushing if the L1TF mitigation is configured to conditionally flush the L1D, and the MMIO mitigation but not any other "clear CPU buffers" mitigation is enabled. For that specific scenario, KVM would skip clearing CPU buffers for the MMIO mitigation even though the kernel requested a clear on every VM-Enter. Note #2, the flaw goes back to the introduction of the MDS mitigation. The MDS mitigation was inadvertently fixed by commit 43fb862de8f6 ("KVM/VMX: Move VERW closer to VMentry for MDS mitigation"), but previous kernels that flush CPU buffers in vmx_vcpu_enter_exit() are affected (though it's unlikely the flaw is meaningfully exploitable even older kernels). Fixes: 650b68a0622f ("x86/kvm/vmx: Add MDS protection when L1D Flush is not= active") Suggested-by: Pawan Gupta Reviewed-by: Pawan Gupta Signed-off-by: Sean Christopherson Reviewed-by: Brendan Jackman --- arch/x86/kvm/vmx/vmenter.S | 16 ++++++++++++++-- arch/x86/kvm/vmx/vmx.c | 13 ------------- 2 files changed, 14 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 7e7bb9b7162f..92a85d95b3e4 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -71,6 +71,7 @@ * @regs: unsigned long * (to guest registers) * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH * VMX_RUN_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl + * VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO * * Returns: * 0 on VM-Exit, 1 on VM-Fail @@ -164,8 +165,19 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX =20 - /* Clobbers EFLAGS.ZF */ - VM_CLEAR_CPU_BUFFERS + /* + * Note, ALTERNATIVE_2 works in reverse order. If CLEAR_CPU_BUF_VM is + * enabled, do VERW unconditionally. If CPU_BUF_VM_MMIO is enabled, + * check @flags to see if the vCPU has access to host MMIO, and if so, + * do VERW. Else, do nothing (no mitigations needed/enabled). + */ + ALTERNATIVE_2 "", \ + __stringify(testl $VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO, WORD_SIZE(%= _ASM_SP); \ + jz .Lskip_mmio_verw; \ + VERW; \ + .Lskip_mmio_verw:), \ + X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO, \ + __stringify(VERW), X86_FEATURE_CLEAR_CPU_BUF_VM =20 /* Check @flags to see if vmlaunch or vmresume is needed. */ testl $VMX_RUN_VMRESUME, WORD_SIZE(%_ASM_SP) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 18d9e0eacd0f..e378bd4d875c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7340,21 +7340,8 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_v= cpu *vcpu, =20 guest_state_enter_irqoff(); =20 - /* - * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW - * mitigation for MDS is done late in VMentry and is still - * executed in spite of L1D Flush. This is because an extra VERW - * should not matter much after the big hammer L1D Flush. - * - * cpu_buf_vm_clear is used when system is not vulnerable to MDS/TAA, - * and is affected by MMIO Stale Data. In such cases mitigation in only - * needed against an MMIO capable guest. - */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO) && - (flags & VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO)) - x86_clear_cpu_buffers(); =20 vmx_disable_fb_clear(vmx); =20 --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 795EB329E77 for ; Thu, 13 Nov 2025 23:37:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077082; cv=none; b=B5O1VES7hBcLDEjziqtPvX4SunWZBp5pa29PYEmMO79e2i/kU2Gyw76XvpAtcb+o3ju/AXATL3fmPbijRYnKEf6Y31kzSkfRQvVzEgFts+H4F1GSkRsRR2itFBStkDNESQCdDdhIPXQI2mX4JCvRJSOblt35Dweni96kAPR6Org= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077082; c=relaxed/simple; bh=0snOzrDQ95f27zuaBEn47BWb8Pruz/6zOIKlPuSWTV4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HAeQZPkso0ovpTG/iSfznRJhnufb0anb0JMN9tFv0AP+ANQSDZF7lIIGwKdRW+hS67R1ELWzl8c7Kra02wFC6QHId5PJe/txP4tSv7ZnnUkx+qsYG8y31E+/wWt2bEOQOcc6FQo+P8WqyelxnMNiTlodVUpcXDQ6MuYzVLFU0j8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3RSY+WCH; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3RSY+WCH" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-7aac981b333so1278532b3a.2 for ; Thu, 13 Nov 2025 15:37:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077079; x=1763681879; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Wv5BfAollS99XFhIeOpKfkgpNs04gEhUuvVsDJUAfOk=; b=3RSY+WCH9/WGLFURTJaEXlSc22WizYLQ+UxZEugQWcgtdrM5AA9hkxj6N6LZSwMaHK Vpoecu0wavw/nmsfqI3i+mOEH5mPEryRmxi7fER8mgiEqOmWPyyYzCD7kLTwQ39z0cNt RzHBq67XuBUyN6ZV8cvmlCmOuUEK7RCH+SmmdHY72Aqlg0nlZIPxcVOEy/HmDv7qVlmV ShGWy5wzSG7kUYQpr0cnVukoLE6ApU66VfxGomozsHvrAxelWtGtWwUQaIcxWUjXG7Dc E+0lfgSpZJuZKYISg/VN6mkEQkDK3PMbN4qlVLbe3XSeIJxwyQKOWtwN/UiRk5C9BKrs AEew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077079; x=1763681879; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Wv5BfAollS99XFhIeOpKfkgpNs04gEhUuvVsDJUAfOk=; b=IR9ui9fp8ve6+bBQgsolRWgPxnj+oFjIrCpKwqPbzt7RAwQgUAKQZzb97ZHTP2OPE8 /3YjKy3wYlUWqWVPgcNomHKGuDb7DZbwLYKi+qzyeOJqHNtxFqWVtS+5yePElP/NUezM iJDE51xH2dvopRYv2oXSf9NDlhdop44XCBA62lotiVLvMg2SJ0tI4licLmfK0IBCWKt9 JPwnWmFTRdmKGQbiAAfqqAz1MQX3SNb6UbzXTzDDBvIUGck1lIQNcb0Npjic6GFFyN7Z jYUMfL7/p2+F+pgg0ULTS/SNzDQSqXLOlCPEyP1wjyxzyKkQkDtIytsfASp7ZlRFIVh/ ZE5g== X-Forwarded-Encrypted: i=1; AJvYcCXP326hJAeIVWNJ3dncTPCCpwIIiwLrzNwhA8M49D8iOxtbFwAOcVJ0+4AwgtrUQ0c5dXLUY+Igyt8UcU0=@vger.kernel.org X-Gm-Message-State: AOJu0YyurvIXtvLpPS0c6ulqVDTfIFR/nlcQWkIxCLGCLygeU1N+IJ9h Hvu/JSefbfA1yIafT6W6Fh6Vix8Ia6Hk8M7nKe1XNhqGvUhIQtV7lpzDcOn7j9t2l0Cy0aG/HR1 IJtYpug== X-Google-Smtp-Source: AGHT+IEbtSUSfVPD1OTBTXG3h1YJ+Su4Df2wRMJHldoJY/HJ5rz+ppkYTp4GcI5dxuBKeGbAnXDLEg3pOLQ= X-Received: from pgbfe16.prod.google.com ([2002:a05:6a02:2890:b0:bac:a20:5f0a]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:12d2:b0:348:9a99:6253 with SMTP id adf61e73a8af0-35ba2796ff5mr1438155637.57.1763077078702; Thu, 13 Nov 2025 15:37:58 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:43 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-7-seanjc@google.com> Subject: [PATCH v5 6/9] x86/bugs: KVM: Move VM_CLEAR_CPU_BUFFERS into SVM as SVM_CLEAR_CPU_BUFFERS From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that VMX encodes its own sequence for clearing CPU buffers, move VM_CLEAR_CPU_BUFFERS into SVM to minimize the chances of KVM botching a mitigation in the future, e.g. using VM_CLEAR_CPU_BUFFERS instead of checking multiple mitigation flags. No functional change intended. Reviewed-by: Brendan Jackman Acked-by: Borislav Petkov (AMD) Signed-off-by: Sean Christopherson --- arch/x86/include/asm/nospec-branch.h | 3 --- arch/x86/kvm/svm/vmenter.S | 6 ++++-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index b1b1811216c5..3a5ac39877a0 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -324,9 +324,6 @@ #define CLEAR_CPU_BUFFERS \ ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF =20 -#define VM_CLEAR_CPU_BUFFERS \ - ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF_VM - #ifdef CONFIG_X86_64 .macro CLEAR_BRANCH_HISTORY ALTERNATIVE "", "call clear_bhb_loop", X86_FEATURE_CLEAR_BHB_LOOP diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S index 98bfa2e00d88..3392bcadfb89 100644 --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -116,6 +116,8 @@ jmp 901b .endm =20 +#define SVM_CLEAR_CPU_BUFFERS \ + ALTERNATIVE "", __CLEAR_CPU_BUFFERS, X86_FEATURE_CLEAR_CPU_BUF_VM =20 /** * __svm_vcpu_run - Run a vCPU via a transition to SVM guest mode @@ -194,7 +196,7 @@ SYM_FUNC_START(__svm_vcpu_run) mov VCPU_RDI(%_ASM_DI), %_ASM_DI =20 /* Clobbers EFLAGS.ZF */ - VM_CLEAR_CPU_BUFFERS + SVM_CLEAR_CPU_BUFFERS =20 /* Enter guest mode */ 3: vmrun %_ASM_AX @@ -366,7 +368,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) mov KVM_VMCB_pa(%rax), %rax =20 /* Clobbers EFLAGS.ZF */ - VM_CLEAR_CPU_BUFFERS + SVM_CLEAR_CPU_BUFFERS =20 /* Enter guest mode */ 1: vmrun %rax --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4551D32A3E7 for ; Thu, 13 Nov 2025 23:38:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077083; cv=none; b=sf21kBvQEJJL3kr1mmYWUOx9tdzcY7D7XsMu+lErQfNdS+duDuEudEho7yudbU1NqKtvtjrR2O6cjBD2xVckHQCsUv2QDPeI4asQ8vAFPQqVSvE3NuQeAgmDkcEptf62WU1Rp93mJv4DG04IrLMxQV2u2X3oPl0UnSlfZssVNdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077083; c=relaxed/simple; bh=3bPudvKflH3kj1sCX5lS7gg9SOxerbLcCuk66r56odo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jdZig6yF4B6azsEETLHBe1g6XPWMHPyv4YvdLDdhdOcTuVUNwKLFykZ1AtblJVJPzfsXzntmDStKvDpMWaMCAZjy8H2o9qWd6QxLs83Nbz0DTR/A2+UicNuAA/4HVIAF2zBHFNjSzyzlf0kHTUBnGq965fmQQypqO2eYF3YxZR0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xLzvM7C5; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xLzvM7C5" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-341616a6fb7so1786725a91.0 for ; Thu, 13 Nov 2025 15:38:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077080; x=1763681880; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=y3rOfN+A2/lHkqkN+NclIFJgOCxTEoAd8AT0UopLffg=; b=xLzvM7C5p8Bfa70YfOosGWO2YqEo2TO25m9BYoIRc1UtY5SlyoGh55DZYg890Lkz0T qOI6SDHA1maewe292Hxof3VlUgg4KnkOChMXzchTn6lV8zuJ1ynlI6jZ32+IMJ8OrD5Q lhseLgD9vYRRlqSzO0KKmglPbVNpTDdZLHx2ssMpowtWFmK4kUNfZfjCRqchB8vedP3H GCohk1w2vlE36Zm4hOnc0l4X9UJEngA/dHLLd+0/et+aHLn5wgKGZXsm7o9Xf+0oG1IB Aj/cB1qUjjIvjzEn1doyz0W0heHKSJufTen+nRlbq32fbAWP/TzMgp5WUMoUbIreqSno IJjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077080; x=1763681880; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=y3rOfN+A2/lHkqkN+NclIFJgOCxTEoAd8AT0UopLffg=; b=IczgO0kfeps2qR/DxGB/rWfph4VOUWpTM1CZQJd+9nb6dGVmlKMGmMSVFYC94JU7WB W48YmG+QeL8nKPghL7Jyba1XtFbs13veCQARGYr4iZXZFGA+Tw67G3zxkpr/KxQpTkzF yHw6E51Ubev4gxYb5WkdtswqUrmhgCHyiz3knpPQpdO4mORNO5npno7Np9FfWHFaCdq7 hciIUBu61G1zU2ElZ6N5UCzl/Ty8ud+UiMSaCjCZMi4QDuXy969lRm0gQawu2UbDHi7m j+aF/L2Y5kvuUCiRcpjk2NORuxlEgPd8mA8F/WUwMwJJkaQmiUdN2NEa4nUgGTygFfiI QWeA== X-Forwarded-Encrypted: i=1; AJvYcCVZ3n9kUcN41kK3AubouM8b6pElSHA/LVr+WR9ID0r8VrpD+iEBlI9cjBwHXzwEFD5knr11qdh1DboZTZU=@vger.kernel.org X-Gm-Message-State: AOJu0Yw52xLvOnRsQ2LnkvUdzpo9IPYLmRoGFVcWjis8bakBbjqYqwcG 4XYLFbdWVjEkudAKhKc9nkF63iF6rZ9IXVIipHjOr9yhd8XbSBelpyY0F95JKdFIQp3UpvZLKHI b123yWw== X-Google-Smtp-Source: AGHT+IHo5gWtbqSMS4jNrTwaSxmrhW8XqrtYH0o1LR1MhEk949gHw6pHVjzl/xb5E5V+ut3YVq9EmercEqI= X-Received: from pjbha6.prod.google.com ([2002:a17:90a:f3c6:b0:33b:dccb:b328]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2fc3:b0:340:fb6a:cb52 with SMTP id 98e67ed59e1d1-343fa639820mr979903a91.25.1763077080518; Thu, 13 Nov 2025 15:38:00 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:44 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-8-seanjc@google.com> Subject: [PATCH v5 7/9] KVM: VMX: Bundle all L1 data cache flush mitigation code together From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move vmx_l1d_flush(), vmx_cleanup_l1d_flush(), and the vmentry_l1d_flush param code up in vmx.c so that all of the L1 data cache flushing code is bundled together. This will allow conditioning the mitigation code on CONFIG_CPU_MITIGATIONS=3Dy with minimal #ifdefs. No functional change intended. Reviewed-by: Brendan Jackman Reviewed-by: Pawan Gupta Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 174 ++++++++++++++++++++--------------------- 1 file changed, 87 insertions(+), 87 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index e378bd4d875c..b39e083671bc 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -302,6 +302,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_stat= e l1tf) return 0; } =20 +static void vmx_cleanup_l1d_flush(void) +{ + if (vmx_l1d_flush_pages) { + free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER); + vmx_l1d_flush_pages =3D NULL; + } + /* Restore state so sysfs ignores VMX */ + l1tf_vmx_mitigation =3D VMENTER_L1D_FLUSH_AUTO; +} + static int vmentry_l1d_flush_parse(const char *s) { unsigned int i; @@ -352,6 +362,83 @@ static int vmentry_l1d_flush_get(char *s, const struct= kernel_param *kp) return sysfs_emit(s, "%s\n", vmentry_l1d_param[l1tf_vmx_mitigation].optio= n); } =20 +/* + * Software based L1D cache flush which is used when microcode providing + * the cache control MSR is not loaded. + * + * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to + * flush it is required to read in 64 KiB because the replacement algorithm + * is not exactly LRU. This could be sized at runtime via topology + * information but as all relevant affected CPUs have 32KiB L1D cache size + * there is no point in doing so. + */ +static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) +{ + int size =3D PAGE_SIZE << L1D_CACHE_ORDER; + + /* + * This code is only executed when the flush mode is 'cond' or + * 'always' + */ + if (static_branch_likely(&vmx_l1d_flush_cond)) { + bool flush_l1d; + + /* + * Clear the per-vcpu flush bit, it gets set again if the vCPU + * is reloaded, i.e. if the vCPU is scheduled out or if KVM + * exits to userspace, or if KVM reaches one of the unsafe + * VMEXIT handlers, e.g. if KVM calls into the emulator. + */ + flush_l1d =3D vcpu->arch.l1tf_flush_l1d; + vcpu->arch.l1tf_flush_l1d =3D false; + + /* + * Clear the per-cpu flush bit, it gets set again from + * the interrupt handlers. + */ + flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); + kvm_clear_cpu_l1tf_flush_l1d(); + + if (!flush_l1d) + return; + } + + vcpu->stat.l1d_flush++; + + if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { + native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); + return; + } + + asm volatile( + /* First ensure the pages are in the TLB */ + "xorl %%eax, %%eax\n" + ".Lpopulate_tlb:\n\t" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $4096, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lpopulate_tlb\n\t" + "xorl %%eax, %%eax\n\t" + "cpuid\n\t" + /* Now fill the cache */ + "xorl %%eax, %%eax\n" + ".Lfill_cache:\n" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $64, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lfill_cache\n\t" + "lfence\n" + :: [flush_pages] "r" (vmx_l1d_flush_pages), + [size] "r" (size) + : "eax", "ebx", "ecx", "edx"); +} + +static const struct kernel_param_ops vmentry_l1d_flush_ops =3D { + .set =3D vmentry_l1d_flush_set, + .get =3D vmentry_l1d_flush_get, +}; +module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644); + static __always_inline void vmx_disable_fb_clear(struct vcpu_vmx *vmx) { u64 msr; @@ -404,12 +491,6 @@ static void vmx_update_fb_clear_dis(struct kvm_vcpu *v= cpu, struct vcpu_vmx *vmx) vmx->disable_fb_clear =3D false; } =20 -static const struct kernel_param_ops vmentry_l1d_flush_ops =3D { - .set =3D vmentry_l1d_flush_set, - .get =3D vmentry_l1d_flush_get, -}; -module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644); - static u32 vmx_segment_access_rights(struct kvm_segment *var); =20 void vmx_vmexit(void); @@ -6673,77 +6754,6 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t exit_fastpath) return ret; } =20 -/* - * Software based L1D cache flush which is used when microcode providing - * the cache control MSR is not loaded. - * - * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to - * flush it is required to read in 64 KiB because the replacement algorithm - * is not exactly LRU. This could be sized at runtime via topology - * information but as all relevant affected CPUs have 32KiB L1D cache size - * there is no point in doing so. - */ -static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) -{ - int size =3D PAGE_SIZE << L1D_CACHE_ORDER; - - /* - * This code is only executed when the flush mode is 'cond' or - * 'always' - */ - if (static_branch_likely(&vmx_l1d_flush_cond)) { - bool flush_l1d; - - /* - * Clear the per-vcpu flush bit, it gets set again if the vCPU - * is reloaded, i.e. if the vCPU is scheduled out or if KVM - * exits to userspace, or if KVM reaches one of the unsafe - * VMEXIT handlers, e.g. if KVM calls into the emulator. - */ - flush_l1d =3D vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d =3D false; - - /* - * Clear the per-cpu flush bit, it gets set again from - * the interrupt handlers. - */ - flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); - kvm_clear_cpu_l1tf_flush_l1d(); - - if (!flush_l1d) - return; - } - - vcpu->stat.l1d_flush++; - - if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { - native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); - return; - } - - asm volatile( - /* First ensure the pages are in the TLB */ - "xorl %%eax, %%eax\n" - ".Lpopulate_tlb:\n\t" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $4096, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lpopulate_tlb\n\t" - "xorl %%eax, %%eax\n\t" - "cpuid\n\t" - /* Now fill the cache */ - "xorl %%eax, %%eax\n" - ".Lfill_cache:\n" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $64, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lfill_cache\n\t" - "lfence\n" - :: [flush_pages] "r" (vmx_l1d_flush_pages), - [size] "r" (size) - : "eax", "ebx", "ecx", "edx"); -} - void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); @@ -8671,16 +8681,6 @@ __init int vmx_hardware_setup(void) return r; } =20 -static void vmx_cleanup_l1d_flush(void) -{ - if (vmx_l1d_flush_pages) { - free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER); - vmx_l1d_flush_pages =3D NULL; - } - /* Restore state so sysfs ignores VMX */ - l1tf_vmx_mitigation =3D VMENTER_L1D_FLUSH_AUTO; -} - void vmx_exit(void) { allow_smaller_maxphyaddr =3D false; --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6D5933122A for ; Thu, 13 Nov 2025 23:38:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077084; cv=none; b=NVjHnSNqTynw7Jq033o7FGlqUlw/e/ZTyK2rR6xoBMkqxxQARUvxVeqvpF6XLkTz0bFkp9YAdL449gWY/wOklAsQeP4E1ovci7y1ZZ57abJhfE78kgAY0PBwcIgeqT2a98Nx6He/8TgGJZsmfBby8RDuQj1IEWylbUlM7mpH0Xc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077084; c=relaxed/simple; bh=wtycuEY2azLWihklz9eK2sJ+qOCuV2/E3a/PpTiKFK0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=AvMDvPKe3WgaXMKE6eKB36qnTFJmybr+Ch0Yt6cA2WMkPPsrdFsg8xTcmrcOSk0HtqxZpScf7Pf/zw0n+SDteMozAuXOfnPvvSwk2fOFjTvWuwM9hAgxIzpc9tdASnrqEeBTWbjf6+Oimhfailj/4PnLc/Vgmx9Mvu6j09/uXnU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cQVN2nz7; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cQVN2nz7" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-7a998ab7f87so2273848b3a.3 for ; Thu, 13 Nov 2025 15:38:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077082; x=1763681882; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=qQ9BLihijeePl6z+R4l/kmqIlHBheOPmGQ6Y3DB6Ymg=; b=cQVN2nz7S1maOalLN4zPYFyQa11qddOWTK0E3vSaLWiDmyr5JZS5f/1RsIih1VNZSc WbdVFp+2/DxnRGLbH5SZ/jx16KQgTEv76c4sih9CrWxLbsUHUzWPW3UMYeRbI5slo3KX aVM5hwt9pFXo0AYL37dLbUs24WEN6eheF1L5Em+08nVSO4eJIChy6QQVJQGGorjxIWxo TAAB8zyLs//0darEYn98b0ldcuCAyxyWHzzmoacQXFdIn0bD4JEZcc083Dp5/wIOrVdi 2iPQU7yu4YzXzwamef96CMT5XvnOsf+5QeekNNACzC+DKv9cU3YVSUZaTH9cfvXZlZja Gbyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077082; x=1763681882; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qQ9BLihijeePl6z+R4l/kmqIlHBheOPmGQ6Y3DB6Ymg=; b=wlAIsnswAZ7nUmLw754I3BrCAohLozz/q5P38N4NUK1IceHcmFgFbP15HmQpzzzwsI qdlb7yKmdJLoDv81+pOIKF8+ApIh7Yjvgmrsv71aXoyPsW4jQH6pKcT04zJmhTjCd1J8 jdxkS301gBgnm3mFHLI5FxtyB+i6bRNMUKqtv/jU/pknJN0mnpscYcDkSjngY4SjlyS9 cbV2ENBX4/1koCKvm9rdjmTSN+fRXMNED4520G6U9jgSsjC4v9oeK1Fdr+wfmgUJzGB1 /T39sR5NkI2mAAqxFVFFNFL9XKefQQuhVGfjJcQ+GjowNLqhx2QRKz+UzWrm49+LkkVZ 4A7g== X-Forwarded-Encrypted: i=1; AJvYcCXEc/6G4FVCkPkwlYQT1qpG4chHKI4x6V7qmsNC0J1XHdtXEwDIaN2tG6WGSaUDgtytvDwnTm0mCBS2tJU=@vger.kernel.org X-Gm-Message-State: AOJu0Yyk6CrmfclFwxEC/PmjKHY3nrbWewo1KH5yXTNyfc3ngpuYmRX/ bjYXlpaU/qE5ni/oFfwIsDSk8jjJzW0Jgr9aK6JQj8HAsVJ1U121oIHZesUk/kVJgjmANyU16nJ HM+ectA== X-Google-Smtp-Source: AGHT+IFp21U8vXtIEaH8PM9ybXHA3sulW7mjYFjo/+rs+93pjrmeK1XzGOkdPLFjeJExEkEYzWZo81FNomQ= X-Received: from pfm8.prod.google.com ([2002:a05:6a00:728:b0:76b:f0d4:ac71]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:a28:b0:7ab:4fce:fa1c with SMTP id d2e1a72fcca58-7ba39ce323emr1304351b3a.1.1763077082235; Thu, 13 Nov 2025 15:38:02 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:45 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-9-seanjc@google.com> Subject: [PATCH v5 8/9] KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Disable support for flushing the L1 data cache to mitigate L1TF if CPU mitigations are disabled for the entire kernel. KVM's mitigation of L1TF is in no way special enough to justify ignoring CONFIG_CPU_MITIGATIONS=3Dn. Deliberately use CPU_MITIGATIONS instead of the more precise MITIGATION_L1TF, as MITIGATION_L1TF only controls the default behavior, i.e. CONFIG_MITIGATION_L1TF=3Dn doesn't completely disable L1TF mitigations in the kernel. Keep the vmentry_l1d_flush module param to avoid breaking existing setups, and leverage the .set path to alert the user to the fact that vmentry_l1d_flush will be ignored. Don't bother validating the incoming value; if an admin misconfigures vmentry_l1d_flush, the fact that the bad configuration won't be detected when running with CONFIG_CPU_MITIGATIONS=3Dn is likely the least of their worries. Reviewed-by: Brendan Jackman Signed-off-by: Sean Christopherson --- arch/x86/include/asm/hardirq.h | 4 +-- arch/x86/kvm/vmx/vmx.c | 56 ++++++++++++++++++++++++++-------- 2 files changed, 46 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index f00c09ffe6a9..6b6d472baa0b 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -5,7 +5,7 @@ #include =20 typedef struct { -#if IS_ENABLED(CONFIG_KVM_INTEL) +#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL) u8 kvm_cpu_l1tf_flush_l1d; #endif unsigned int __nmi_count; /* arch dependent */ @@ -68,7 +68,7 @@ extern u64 arch_irq_stat(void); DECLARE_PER_CPU_CACHE_HOT(u16, __softirq_pending); #define local_softirq_pending_ref __softirq_pending =20 -#if IS_ENABLED(CONFIG_KVM_INTEL) +#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL) /* * This function is called from noinstr interrupt contexts * and must be inlined to not get instrumentation. diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index b39e083671bc..ae6b102b1570 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -203,6 +203,7 @@ module_param(pt_mode, int, S_IRUGO); =20 struct x86_pmu_lbr __ro_after_init vmx_lbr_caps; =20 +#ifdef CONFIG_CPU_MITIGATIONS static DEFINE_STATIC_KEY_FALSE(vmx_l1d_should_flush); static DEFINE_STATIC_KEY_FALSE(vmx_l1d_flush_cond); static DEFINE_MUTEX(vmx_l1d_flush_mutex); @@ -225,7 +226,7 @@ static const struct { #define L1D_CACHE_ORDER 4 static void *vmx_l1d_flush_pages; =20 -static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) +static int __vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) { struct page *page; unsigned int i; @@ -302,6 +303,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_stat= e l1tf) return 0; } =20 +static int vmx_setup_l1d_flush(void) +{ + /* + * Hand the parameter mitigation value in which was stored in the pre + * module init parser. If no parameter was given, it will contain + * 'auto' which will be turned into the default 'cond' mitigation mode. + */ + return __vmx_setup_l1d_flush(vmentry_l1d_flush_param); +} + static void vmx_cleanup_l1d_flush(void) { if (vmx_l1d_flush_pages) { @@ -349,7 +360,7 @@ static int vmentry_l1d_flush_set(const char *s, const s= truct kernel_param *kp) } =20 mutex_lock(&vmx_l1d_flush_mutex); - ret =3D vmx_setup_l1d_flush(l1tf); + ret =3D __vmx_setup_l1d_flush(l1tf); mutex_unlock(&vmx_l1d_flush_mutex); return ret; } @@ -376,6 +387,9 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) { int size =3D PAGE_SIZE << L1D_CACHE_ORDER; =20 + if (!static_branch_unlikely(&vmx_l1d_should_flush)) + return; + /* * This code is only executed when the flush mode is 'cond' or * 'always' @@ -433,6 +447,31 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcp= u) : "eax", "ebx", "ecx", "edx"); } =20 +#else /* CONFIG_CPU_MITIGATIONS*/ +static int vmx_setup_l1d_flush(void) +{ + l1tf_vmx_mitigation =3D VMENTER_L1D_FLUSH_NEVER; + return 0; +} +static void vmx_cleanup_l1d_flush(void) +{ + l1tf_vmx_mitigation =3D VMENTER_L1D_FLUSH_AUTO; +} +static __always_inline void vmx_l1d_flush(struct kvm_vcpu *vcpu) +{ + +} +static int vmentry_l1d_flush_set(const char *s, const struct kernel_param = *kp) +{ + pr_warn_once("Kernel compiled without mitigations, ignoring vmentry_l1d_f= lush\n"); + return 0; +} +static int vmentry_l1d_flush_get(char *s, const struct kernel_param *kp) +{ + return sysfs_emit(s, "never\n"); +} +#endif + static const struct kernel_param_ops vmentry_l1d_flush_ops =3D { .set =3D vmentry_l1d_flush_set, .get =3D vmentry_l1d_flush_get, @@ -7350,8 +7389,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vc= pu *vcpu, =20 guest_state_enter_irqoff(); =20 - if (static_branch_unlikely(&vmx_l1d_should_flush)) - vmx_l1d_flush(vcpu); + vmx_l1d_flush(vcpu); =20 vmx_disable_fb_clear(vmx); =20 @@ -8716,14 +8754,8 @@ int __init vmx_init(void) if (r) return r; =20 - /* - * Must be called after common x86 init so enable_ept is properly set - * up. Hand the parameter mitigation value in which was stored in - * the pre module init parser. If no parameter was given, it will - * contain 'auto' which will be turned into the default 'cond' - * mitigation mode. - */ - r =3D vmx_setup_l1d_flush(vmentry_l1d_flush_param); + /* Must be called after common x86 init so enable_ept is setup. */ + r =3D vmx_setup_l1d_flush(); if (r) goto err_l1d_flush; =20 --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:38 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 835FC3328F3 for ; Thu, 13 Nov 2025 23:38:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077086; cv=none; b=QFEq8uoX0lNhv6L7wgUTZRXZADTIpAmecO+8vUsDAIG0nq8XfPDFIeuztj2ZOTqhZG62zy3zw5sjJYda8P/cKEAyLq1Qu6bWnKY8iHeaume1a03Tb8tpvvKSTxXGaH1kOTw/XhiKSpBnsbZ3ra9ppqDAmdcUlXYP67lLmRLnpgI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077086; c=relaxed/simple; bh=B/Y8IL0u/Yf38qiKxxW8J1SjwRx9A+pqtBMgpmG/tKg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RULOSGOmT32br/SQh3jlSCtGMnx+9zrLBK2M1Lj1p88ut9GyuriI2NA/xt+DJatOlbrXd/XzP/5kj304uoDBtAim1ekpI15BYBcXR0Ct1vJNM4FiSQc4aeMoPU7KbMi2dldc9JOOnsjhsSJbnzpYeuxH7gzoWmqKIUAGJHAkI88= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=T6utw3ea; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="T6utw3ea" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-7a43210187cso1282248b3a.3 for ; Thu, 13 Nov 2025 15:38:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077084; x=1763681884; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=KaHHeo0MEFYI+GLNe9ECI/Oiz1Zlq+iHNt9JMoNWhqk=; b=T6utw3eaZqI/ohqwyyXLLXM+vWu51dH1JDFdULg7up0h61ln1vmLEzE8dEtblifaKb VCTYfF8asBoOJgN41wQmE2SN+Ew9kMu35d3OgY+k69QLrQ1Y/hp59q9jqwyKQcUTZLBs fYRO3JUn6EQtLTK9bg6vmxzpHM37gGBKAwo4+o1hPivz0XTI/YDQ5zk3SApmBehkG63S 30abG8rm2jvuYHdInQ/pApPXmNS1NuhDyv2oFAh4wXe14wkZ0+dTMSNwEb+YA4BIcsO8 EymLTyA/UV9cnAL43+LXdzV0iv4OgI4VePV+0rpWCoOYPxnmkdMpFuGMfugTKt3vuzkF qLrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077084; x=1763681884; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KaHHeo0MEFYI+GLNe9ECI/Oiz1Zlq+iHNt9JMoNWhqk=; b=GLbs+xqAPskFXFe8MqF9+44m7wZaxNW46LJ/s5aSD9MEGbKWf31uof2ZnvtV7gkqkz w4Xh9nBvDnl/9tL9lZw1CEsoz6pCOMOlaZotPySiriRVoVHJlSAtfgGto9fsNQnmQMiY O33dhlO7/JuDUYwNEaRZ8NP49+MAQL+ApIZbWroBKw1t0a1jeCNjColh3CkaRrf7B9Uy BOqQYstx/6klEzRHJdYb+/y8qRahTAgPiK7K+9ly635cGWuPnEFs8oLMpqJEFu0hcR/X jtNYYeksSfNQcOdFJeChAN6rGBbOp8PLkjmK63DE8RSKE3Z4lFG6vbsWxbzHqj6zk6Tz Ce0Q== X-Forwarded-Encrypted: i=1; AJvYcCXR6NYQzWs7eW4fVDruzpXr1Gxug7U6nPRCmNFDamGyIYUxRvjFXIgHzSK8Vb3LkDziZhgGA19QFx2Q8/Q=@vger.kernel.org X-Gm-Message-State: AOJu0YyUrcoEFMLl2aWjbfbV1xksOCnGKt60Br3p3adiFVLIznobdh2g nEiPBc8IEzOHtwVY1OVsfPJ7gvV4F7k7G+BLdbIX8KBUJUTJEsrcqGT5VcIfX6Eli3sI82wDqAV zqoeL5w== X-Google-Smtp-Source: AGHT+IFA8/ID3yDfg8RK0jOw1iuj/Ls2AuCdSB9aT9rjOost0RLXhPoKOAPIKzNF+CYaofznnWFFajEs85s= X-Received: from pgbee13.prod.google.com ([2002:a05:6a02:458d:b0:bac:6acd:818b]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:3382:b0:304:313a:4bcd with SMTP id adf61e73a8af0-35ba19a0b46mr1460781637.30.1763077083720; Thu, 13 Nov 2025 15:38:03 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:46 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-10-seanjc@google.com> Subject: [PATCH v5 9/9] KVM: x86: Unify L1TF flushing under per-CPU variable From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Brendan Jackman Currently the tracking of the need to flush L1D for L1TF is tracked by two bits: one per-CPU and one per-vCPU. The per-vCPU bit is always set when the vCPU shows up on a core, so there is no interesting state that's truly per-vCPU. Indeed, this is a requirement, since L1D is a part of the physical CPU. So simplify this by combining the two bits. The vCPU bit was being written from preemption-enabled regions. To play nice with those cases, wrap all calls from KVM and use a raw write so that request a flush with preemption enabled doesn't trigger what would effectively be DEBUG_PREEMPT false positives. Preemption doesn't need to be disabled, as kvm_arch_vcpu_load() will mark the new CPU as needing a flush if the vCPU task is migrated, or if userspace runs the vCPU on a different task. Signed-off-by: Brendan Jackman [sean: put raw write in KVM instead of in a hardirq.h variant] Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 20 +++++--------------- arch/x86/kvm/x86.c | 6 +++--- arch/x86/kvm/x86.h | 14 ++++++++++++++ 6 files changed, 24 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 1a8ea5dc6699..9f9839bbce13 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1055,9 +1055,6 @@ struct kvm_vcpu_arch { /* be preempted when it's in kernel-mode(cpl=3D0) */ bool preempted_in_kernel; =20 - /* Flush the L1 Data cache for L1TF mitigation on VMENTER */ - bool l1tf_flush_l1d; - /* Host CPU on which VM-entry was most recently attempted */ int last_vmentry_cpu; =20 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 3cfabfbdd843..02c450686b4a 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4859,7 +4859,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 = error_code, */ BUILD_BUG_ON(lower_32_bits(PFERR_SYNTHETIC_MASK)); =20 - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_request_l1tf_flush_l1d(); if (!flags) { trace_kvm_page_fault(vcpu, fault_address, error_code); =20 diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 564f5af5ae86..40777278eabb 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3828,7 +3828,7 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool= launch) goto vmentry_failed; =20 /* Hide L1D cache contents from the nested guest. */ - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_request_l1tf_flush_l1d(); =20 /* * Must happen outside of nested_vmx_enter_non_root_mode() as it will diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index ae6b102b1570..df4cfcc6591a 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -395,26 +395,16 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vc= pu) * 'always' */ if (static_branch_likely(&vmx_l1d_flush_cond)) { - bool flush_l1d; - /* - * Clear the per-vcpu flush bit, it gets set again if the vCPU + * Clear the per-cpu flush bit, it gets set again if the vCPU * is reloaded, i.e. if the vCPU is scheduled out or if KVM * exits to userspace, or if KVM reaches one of the unsafe - * VMEXIT handlers, e.g. if KVM calls into the emulator. + * VMEXIT handlers, e.g. if KVM calls into the emulator, + * or from the interrupt handlers. */ - flush_l1d =3D vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d =3D false; - - /* - * Clear the per-cpu flush bit, it gets set again from - * the interrupt handlers. - */ - flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); + if (!kvm_get_cpu_l1tf_flush_l1d()) + return; kvm_clear_cpu_l1tf_flush_l1d(); - - if (!flush_l1d) - return; } =20 vcpu->stat.l1d_flush++; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 9c2e28028c2b..ae774519b701 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5209,7 +5209,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cp= u) { struct kvm_pmu *pmu =3D vcpu_to_pmu(vcpu); =20 - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_request_l1tf_flush_l1d(); =20 if (vcpu->scheduled_out && pmu->version && pmu->event_count) { pmu->need_cleanup =3D true; @@ -8032,7 +8032,7 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu= , gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception) { /* kvm_write_guest_virt_system can pull in tons of pages. */ - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_request_l1tf_flush_l1d(); =20 return kvm_write_guest_virt_helper(addr, val, bytes, vcpu, PFERR_WRITE_MASK, exception); @@ -9440,7 +9440,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gp= a_t cr2_or_gpa, return handle_emulation_failure(vcpu, emulation_type); } =20 - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_request_l1tf_flush_l1d(); =20 if (!(emulation_type & EMULTYPE_NO_DECODE)) { kvm_clear_exception_queue(vcpu); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 24c754b0db2e..fdab0ad49098 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -420,6 +420,20 @@ static inline bool kvm_check_has_quirk(struct kvm *kvm= , u64 quirk) return !(kvm->arch.disabled_quirks & quirk); } =20 +static __always_inline void kvm_request_l1tf_flush_l1d(void) +{ +#if IS_ENABLED(CONFIG_CPU_MITIGATIONS) && IS_ENABLED(CONFIG_KVM_INTEL) + /* + * Use a raw write to set the per-CPU flag, as KVM will ensure a flush + * even if preemption is currently enabled.. If the current vCPU task + * is migrated to a different CPU (or userspace runs the vCPU on a + * different task) before the next VM-Entry, then kvm_arch_vcpu_load() + * will request a flush on the new CPU. + */ + raw_cpu_write(irq_stat.kvm_cpu_l1tf_flush_l1d, 1); +#endif +} + void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc= _eip); =20 u64 get_kvmclock_ns(struct kvm *kvm); --=20 2.52.0.rc1.455.g30608eb744-goog