From nobody Mon Feb 9 15:26:19 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4551D32A3E7 for ; Thu, 13 Nov 2025 23:38:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077083; cv=none; b=sf21kBvQEJJL3kr1mmYWUOx9tdzcY7D7XsMu+lErQfNdS+duDuEudEho7yudbU1NqKtvtjrR2O6cjBD2xVckHQCsUv2QDPeI4asQ8vAFPQqVSvE3NuQeAgmDkcEptf62WU1Rp93mJv4DG04IrLMxQV2u2X3oPl0UnSlfZssVNdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763077083; c=relaxed/simple; bh=3bPudvKflH3kj1sCX5lS7gg9SOxerbLcCuk66r56odo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jdZig6yF4B6azsEETLHBe1g6XPWMHPyv4YvdLDdhdOcTuVUNwKLFykZ1AtblJVJPzfsXzntmDStKvDpMWaMCAZjy8H2o9qWd6QxLs83Nbz0DTR/A2+UicNuAA/4HVIAF2zBHFNjSzyzlf0kHTUBnGq965fmQQypqO2eYF3YxZR0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xLzvM7C5; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xLzvM7C5" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-341616a6fb7so1786725a91.0 for ; Thu, 13 Nov 2025 15:38:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763077080; x=1763681880; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=y3rOfN+A2/lHkqkN+NclIFJgOCxTEoAd8AT0UopLffg=; b=xLzvM7C5p8Bfa70YfOosGWO2YqEo2TO25m9BYoIRc1UtY5SlyoGh55DZYg890Lkz0T qOI6SDHA1maewe292Hxof3VlUgg4KnkOChMXzchTn6lV8zuJ1ynlI6jZ32+IMJ8OrD5Q lhseLgD9vYRRlqSzO0KKmglPbVNpTDdZLHx2ssMpowtWFmK4kUNfZfjCRqchB8vedP3H GCohk1w2vlE36Zm4hOnc0l4X9UJEngA/dHLLd+0/et+aHLn5wgKGZXsm7o9Xf+0oG1IB Aj/cB1qUjjIvjzEn1doyz0W0heHKSJufTen+nRlbq32fbAWP/TzMgp5WUMoUbIreqSno IJjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763077080; x=1763681880; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=y3rOfN+A2/lHkqkN+NclIFJgOCxTEoAd8AT0UopLffg=; b=IczgO0kfeps2qR/DxGB/rWfph4VOUWpTM1CZQJd+9nb6dGVmlKMGmMSVFYC94JU7WB W48YmG+QeL8nKPghL7Jyba1XtFbs13veCQARGYr4iZXZFGA+Tw67G3zxkpr/KxQpTkzF yHw6E51Ubev4gxYb5WkdtswqUrmhgCHyiz3knpPQpdO4mORNO5npno7Np9FfWHFaCdq7 hciIUBu61G1zU2ElZ6N5UCzl/Ty8ud+UiMSaCjCZMi4QDuXy969lRm0gQawu2UbDHi7m j+aF/L2Y5kvuUCiRcpjk2NORuxlEgPd8mA8F/WUwMwJJkaQmiUdN2NEa4nUgGTygFfiI QWeA== X-Forwarded-Encrypted: i=1; AJvYcCVZ3n9kUcN41kK3AubouM8b6pElSHA/LVr+WR9ID0r8VrpD+iEBlI9cjBwHXzwEFD5knr11qdh1DboZTZU=@vger.kernel.org X-Gm-Message-State: AOJu0Yw52xLvOnRsQ2LnkvUdzpo9IPYLmRoGFVcWjis8bakBbjqYqwcG 4XYLFbdWVjEkudAKhKc9nkF63iF6rZ9IXVIipHjOr9yhd8XbSBelpyY0F95JKdFIQp3UpvZLKHI b123yWw== X-Google-Smtp-Source: AGHT+IHo5gWtbqSMS4jNrTwaSxmrhW8XqrtYH0o1LR1MhEk949gHw6pHVjzl/xb5E5V+ut3YVq9EmercEqI= X-Received: from pjbha6.prod.google.com ([2002:a17:90a:f3c6:b0:33b:dccb:b328]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2fc3:b0:340:fb6a:cb52 with SMTP id 98e67ed59e1d1-343fa639820mr979903a91.25.1763077080518; Thu, 13 Nov 2025 15:38:00 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 13 Nov 2025 15:37:44 -0800 In-Reply-To: <20251113233746.1703361-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251113233746.1703361-1-seanjc@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251113233746.1703361-8-seanjc@google.com> Subject: [PATCH v5 7/9] KVM: VMX: Bundle all L1 data cache flush mitigation code together From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Pawan Gupta , Brendan Jackman Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move vmx_l1d_flush(), vmx_cleanup_l1d_flush(), and the vmentry_l1d_flush param code up in vmx.c so that all of the L1 data cache flushing code is bundled together. This will allow conditioning the mitigation code on CONFIG_CPU_MITIGATIONS=3Dy with minimal #ifdefs. No functional change intended. Reviewed-by: Brendan Jackman Reviewed-by: Pawan Gupta Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmx.c | 174 ++++++++++++++++++++--------------------- 1 file changed, 87 insertions(+), 87 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index e378bd4d875c..b39e083671bc 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -302,6 +302,16 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_stat= e l1tf) return 0; } =20 +static void vmx_cleanup_l1d_flush(void) +{ + if (vmx_l1d_flush_pages) { + free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER); + vmx_l1d_flush_pages =3D NULL; + } + /* Restore state so sysfs ignores VMX */ + l1tf_vmx_mitigation =3D VMENTER_L1D_FLUSH_AUTO; +} + static int vmentry_l1d_flush_parse(const char *s) { unsigned int i; @@ -352,6 +362,83 @@ static int vmentry_l1d_flush_get(char *s, const struct= kernel_param *kp) return sysfs_emit(s, "%s\n", vmentry_l1d_param[l1tf_vmx_mitigation].optio= n); } =20 +/* + * Software based L1D cache flush which is used when microcode providing + * the cache control MSR is not loaded. + * + * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to + * flush it is required to read in 64 KiB because the replacement algorithm + * is not exactly LRU. This could be sized at runtime via topology + * information but as all relevant affected CPUs have 32KiB L1D cache size + * there is no point in doing so. + */ +static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) +{ + int size =3D PAGE_SIZE << L1D_CACHE_ORDER; + + /* + * This code is only executed when the flush mode is 'cond' or + * 'always' + */ + if (static_branch_likely(&vmx_l1d_flush_cond)) { + bool flush_l1d; + + /* + * Clear the per-vcpu flush bit, it gets set again if the vCPU + * is reloaded, i.e. if the vCPU is scheduled out or if KVM + * exits to userspace, or if KVM reaches one of the unsafe + * VMEXIT handlers, e.g. if KVM calls into the emulator. + */ + flush_l1d =3D vcpu->arch.l1tf_flush_l1d; + vcpu->arch.l1tf_flush_l1d =3D false; + + /* + * Clear the per-cpu flush bit, it gets set again from + * the interrupt handlers. + */ + flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); + kvm_clear_cpu_l1tf_flush_l1d(); + + if (!flush_l1d) + return; + } + + vcpu->stat.l1d_flush++; + + if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { + native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); + return; + } + + asm volatile( + /* First ensure the pages are in the TLB */ + "xorl %%eax, %%eax\n" + ".Lpopulate_tlb:\n\t" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $4096, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lpopulate_tlb\n\t" + "xorl %%eax, %%eax\n\t" + "cpuid\n\t" + /* Now fill the cache */ + "xorl %%eax, %%eax\n" + ".Lfill_cache:\n" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $64, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lfill_cache\n\t" + "lfence\n" + :: [flush_pages] "r" (vmx_l1d_flush_pages), + [size] "r" (size) + : "eax", "ebx", "ecx", "edx"); +} + +static const struct kernel_param_ops vmentry_l1d_flush_ops =3D { + .set =3D vmentry_l1d_flush_set, + .get =3D vmentry_l1d_flush_get, +}; +module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644); + static __always_inline void vmx_disable_fb_clear(struct vcpu_vmx *vmx) { u64 msr; @@ -404,12 +491,6 @@ static void vmx_update_fb_clear_dis(struct kvm_vcpu *v= cpu, struct vcpu_vmx *vmx) vmx->disable_fb_clear =3D false; } =20 -static const struct kernel_param_ops vmentry_l1d_flush_ops =3D { - .set =3D vmentry_l1d_flush_set, - .get =3D vmentry_l1d_flush_get, -}; -module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644); - static u32 vmx_segment_access_rights(struct kvm_segment *var); =20 void vmx_vmexit(void); @@ -6673,77 +6754,6 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t exit_fastpath) return ret; } =20 -/* - * Software based L1D cache flush which is used when microcode providing - * the cache control MSR is not loaded. - * - * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to - * flush it is required to read in 64 KiB because the replacement algorithm - * is not exactly LRU. This could be sized at runtime via topology - * information but as all relevant affected CPUs have 32KiB L1D cache size - * there is no point in doing so. - */ -static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) -{ - int size =3D PAGE_SIZE << L1D_CACHE_ORDER; - - /* - * This code is only executed when the flush mode is 'cond' or - * 'always' - */ - if (static_branch_likely(&vmx_l1d_flush_cond)) { - bool flush_l1d; - - /* - * Clear the per-vcpu flush bit, it gets set again if the vCPU - * is reloaded, i.e. if the vCPU is scheduled out or if KVM - * exits to userspace, or if KVM reaches one of the unsafe - * VMEXIT handlers, e.g. if KVM calls into the emulator. - */ - flush_l1d =3D vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d =3D false; - - /* - * Clear the per-cpu flush bit, it gets set again from - * the interrupt handlers. - */ - flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); - kvm_clear_cpu_l1tf_flush_l1d(); - - if (!flush_l1d) - return; - } - - vcpu->stat.l1d_flush++; - - if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { - native_wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); - return; - } - - asm volatile( - /* First ensure the pages are in the TLB */ - "xorl %%eax, %%eax\n" - ".Lpopulate_tlb:\n\t" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $4096, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lpopulate_tlb\n\t" - "xorl %%eax, %%eax\n\t" - "cpuid\n\t" - /* Now fill the cache */ - "xorl %%eax, %%eax\n" - ".Lfill_cache:\n" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $64, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lfill_cache\n\t" - "lfence\n" - :: [flush_pages] "r" (vmx_l1d_flush_pages), - [size] "r" (size) - : "eax", "ebx", "ecx", "edx"); -} - void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); @@ -8671,16 +8681,6 @@ __init int vmx_hardware_setup(void) return r; } =20 -static void vmx_cleanup_l1d_flush(void) -{ - if (vmx_l1d_flush_pages) { - free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER); - vmx_l1d_flush_pages =3D NULL; - } - /* Restore state so sysfs ignores VMX */ - l1tf_vmx_mitigation =3D VMENTER_L1D_FLUSH_AUTO; -} - void vmx_exit(void) { allow_smaller_maxphyaddr =3D false; --=20 2.52.0.rc1.455.g30608eb744-goog