From nobody Fri Dec 19 15:32:06 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F85C30F7E2 for ; Mon, 13 Oct 2025 15:21:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760368866; cv=none; b=p9khzKUmUluAtXu1u/feAZLFS1lImnKGoL2Yn7CYZmP/5Zqm5tDirSxLb7/NoAvoQwXY/pFEwajFszlYpAJbCIi2vete/OxU3Lm10CZCjW4aeI28PZyWp24awk3DXMxZBWKJyz/x/x0/QjogVCNxdne5s2w3xury2VhIKqz9iI0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760368866; c=relaxed/simple; bh=uVIIcfGUtnM0AFmoNWciBQ3Oy6FcGT5tcohKaCO7Fdk=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=gr3xIXewARGC5FNQft5JwY6wITmYD7VtaJJvdcDYkilOFZDrj3JEVRn6PDBhRGRsGWZcBs1mPfMo9k0rIv4k+zs/tviyzezQ8zt8u2avokWuCxDv7oauQvIFzv87i2bCXegUurUwlm43MhGOypsemlRic3zJgNtJ7bl336WiERo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HTo7fZEw; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HTo7fZEw" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e3a049abaso32523615e9.0 for ; Mon, 13 Oct 2025 08:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1760368862; x=1760973662; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=YIYoPCNyBFGg9vKc8jF45CxIFXipDwwSdrNJ736RRBY=; b=HTo7fZEwmN/AchWolAniP+Y4PdwpMVrwbOAXne7mzNG4myDYJ5qbQ+rJ+V6PTgj/cC lqduToxSwIztdVYxE5eNzqwhYLKEhizNJLcl5XjKazvgKlm8WrAyJD7LamHnmz8/Leq0 7cXMHVYqMlF7ly6QEH5DjrboKM8xp1zdy4Ojl2qGl/YbeAYhKYiOBMZzXbhUw2T2opIW KZk8Usj+lTKDMiIb9rVeQXiakJlohU+I8aWeC+IxR1MWe8TQYs0+blaGlQq/HFcq7Em0 B46d0IFTGtC6l6mEWRniwIljSkUUIba3qrm2zU0pFB5UpdHyHW/yyy/8fCA7PPeC15XI k7wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1760368862; x=1760973662; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YIYoPCNyBFGg9vKc8jF45CxIFXipDwwSdrNJ736RRBY=; b=N1i7RA/ou5vdIQVJCQUS7uSTjaFMX5mjcN+zRKODOcFS+DlcjAtbc/3DIUgjrEPZZH sHUBJgbXv2KsVGWNO8zpo7yRRvgKHDU4iuhpy48Nn8v8ePjhcF3CKYwZPOBVIMKLINIi mmqHGMrYeXI3QF+nPM7slp6esFbV+dEVrp+AFLemwrhE1VjOGW0VFJNI+1ERo1+wWSpS 0zPAQV/I9uxfR50mbf5+Dpsm0k8nh28bLHNcI4GR1phqjI6rIRTcVE1mXKgjpSqqPT48 HRza5wQWZ34U28fMFMDyrgykCYlShUTah53HACMduSGqdxvJwgZ+FlxpZwcINq6JMw6r 6tcg== X-Gm-Message-State: AOJu0YyE/YQEJw7htXcyQoVT41qp1VZ5bFZKoWxBN883fSjFwajC2o4R v/uOYSRkKdgbr3kAwM76+pD5KDHbcaNvc8R8wkI5tHleNlyPwQNRPirOJaabxEXwjrlB2DoAwwm F6m7HTDp9lpfiTQ== X-Google-Smtp-Source: AGHT+IFVI/ebIiE03xp+IXTAIvzD7VP5Hg6xiJr5X8tY19ifba+QDyv+tPutqETwPbUzyh9TyIvbuut1FWkifw== X-Received: from wmcq4.prod.google.com ([2002:a05:600c:c104:b0:46f:aa50:d6fc]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3b1b:b0:46f:b327:31c4 with SMTP id 5b1f17b1804b1-46fb3273563mr98314565e9.14.1760368862615; Mon, 13 Oct 2025 08:21:02 -0700 (PDT) Date: Mon, 13 Oct 2025 15:20:52 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-B4-Tracking: v=1; b=H4sIANMY7WgC/6tWKk4tykwtVrJSqFYqSi3LLM7MzwNyDHUUlJIzE vPSU3UzU4B8JSMDI1NDA0Nj3SQT3RzDkjTdgtSi5IJSXXNLY0MLw7REUwsLEyWgpoKi1LTMCrC B0bG1tQCGPpaCYAAAAA== X-Change-Id: 20251013-b4-l1tf-percpu-793181fa5884 X-Mailer: b4 0.14.2 Message-ID: <20251013-b4-l1tf-percpu-v1-1-d65c5366ea1a@google.com> Subject: [PATCH] KVM: x86: Unify L1TF flushing under per-CPU variable From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Brendan Jackman Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Currently the tracking of the need to flush L1D for L1TF is tracked by two bits: one per-CPU and one per-vCPU. The per-vCPU bit is always set when the vCPU shows up on a core, so there is no interesting state that's truly per-vCPU. Indeed, this is a requirement, since L1D is a part of the physical CPU. So simplify this by combining the two bits. Since this requires a DECLARE_PER_CPU() which belongs in kvm_host.h, also move the remaining helper definitions there to live next to the declaration. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/hardirq.h | 26 -------------------------- arch/x86/include/asm/idtentry.h | 1 + arch/x86/include/asm/kvm_host.h | 21 ++++++++++++++++++--- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 17 +++-------------- arch/x86/kvm/x86.c | 12 +++++++++--- 7 files changed, 33 insertions(+), 48 deletions(-) diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index f00c09ffe6a95f07342bb0c6cea3769d71eecfa9..29d8fa43d4404add3b191821e42= c3526b0f2c950 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -5,9 +5,6 @@ #include =20 typedef struct { -#if IS_ENABLED(CONFIG_KVM_INTEL) - u8 kvm_cpu_l1tf_flush_l1d; -#endif unsigned int __nmi_count; /* arch dependent */ #ifdef CONFIG_X86_LOCAL_APIC unsigned int apic_timer_irqs; /* arch dependent */ @@ -68,27 +65,4 @@ extern u64 arch_irq_stat(void); DECLARE_PER_CPU_CACHE_HOT(u16, __softirq_pending); #define local_softirq_pending_ref __softirq_pending =20 -#if IS_ENABLED(CONFIG_KVM_INTEL) -/* - * This function is called from noinstr interrupt contexts - * and must be inlined to not get instrumentation. - */ -static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) -{ - __this_cpu_write(irq_stat.kvm_cpu_l1tf_flush_l1d, 1); -} - -static __always_inline void kvm_clear_cpu_l1tf_flush_l1d(void) -{ - __this_cpu_write(irq_stat.kvm_cpu_l1tf_flush_l1d, 0); -} - -static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void) -{ - return __this_cpu_read(irq_stat.kvm_cpu_l1tf_flush_l1d); -} -#else /* !IS_ENABLED(CONFIG_KVM_INTEL) */ -static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { } -#endif /* IS_ENABLED(CONFIG_KVM_INTEL) */ - #endif /* _ASM_X86_HARDIRQ_H */ diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentr= y.h index a4ec27c6798875900cbdbba927918e70b900f63b..67fb1adadbb8ac2bd083ba6245d= e2e7d58b5b398 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -12,6 +12,7 @@ #include =20 #include +#include =20 typedef void (*idtentry_t)(struct pt_regs *regs); =20 diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 48598d017d6f3f07263a2ffffe670be2658eb9cb..d93c2b9dbfbc9824cce65256f60= 6f32e41c93167 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1055,9 +1055,6 @@ struct kvm_vcpu_arch { /* be preempted when it's in kernel-mode(cpl=3D0) */ bool preempted_in_kernel; =20 - /* Flush the L1 Data cache for L1TF mitigation on VMENTER */ - bool l1tf_flush_l1d; - /* Host CPU on which VM-entry was most recently attempted */ int last_vmentry_cpu; =20 @@ -2476,4 +2473,22 @@ static inline bool kvm_arch_has_irq_bypass(void) return enable_device_posted_irqs; } =20 +#if IS_ENABLED(CONFIG_KVM_INTEL) + +DECLARE_PER_CPU(bool, l1tf_flush_l1d); + +/* + * This function is called from noinstr interrupt contexts + * and must be inlined to not get instrumentation. + */ +static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) +{ + __this_cpu_write(l1tf_flush_l1d, true); +} + +#else /* !IS_ENABLED(CONFIG_KVM_INTEL) */ +static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { } +#endif /* IS_ENABLED(CONFIG_KVM_INTEL) */ + + #endif /* _ASM_X86_KVM_HOST_H */ diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 667d66cf76d5e52c22f9517914307244ae868eea..8c0dce401a42d977756ca82d249= bb33c858b9c9f 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4859,7 +4859,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 = error_code, */ BUILD_BUG_ON(lower_32_bits(PFERR_SYNTHETIC_MASK)); =20 - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_set_cpu_l1tf_flush_l1d(); if (!flags) { trace_kvm_page_fault(vcpu, fault_address, error_code); =20 diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 76271962cb7083b475de6d7d24bf9cb918050650..5035cfdc4e55365bfabf08c704b= 9bff5c06267a1 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3880,7 +3880,7 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool= launch) goto vmentry_failed; =20 /* Hide L1D cache contents from the nested guest. */ - vmx->vcpu.arch.l1tf_flush_l1d =3D true; + kvm_set_cpu_l1tf_flush_l1d(); =20 /* * Must happen outside of nested_vmx_enter_non_root_mode() as it will diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 546272a5d34da301710df1d89414f41fc9b24a1f..f982f6721dc3e0dfe046881c727= 32326e16fcfb3 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6673,25 +6673,14 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *= vcpu) * 'always' */ if (static_branch_likely(&vmx_l1d_flush_cond)) { - bool flush_l1d; - /* * Clear the per-vcpu flush bit, it gets set again if the vCPU * is reloaded, i.e. if the vCPU is scheduled out or if KVM * exits to userspace, or if KVM reaches one of the unsafe - * VMEXIT handlers, e.g. if KVM calls into the emulator. + * VMEXIT handlers, e.g. if KVM calls into the emulator, or from the + * interrupt handlers. */ - flush_l1d =3D vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d =3D false; - - /* - * Clear the per-cpu flush bit, it gets set again from - * the interrupt handlers. - */ - flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); - kvm_clear_cpu_l1tf_flush_l1d(); - - if (!flush_l1d) + if (!this_cpu_xchg(l1tf_flush_l1d, false)) return; } =20 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 4b8138bd48572fd161eda73d2dbdc1dcd0bcbcac..766d61516602e0f4975930224fc= 57b5a511281e5 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -171,6 +171,12 @@ bool __read_mostly enable_vmware_backdoor =3D false; module_param(enable_vmware_backdoor, bool, 0444); EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_vmware_backdoor); =20 +#if IS_ENABLED(CONFIG_KVM_INTEL) +/* Flush the L1 Data cache for L1TF mitigation on VMENTER */ +DEFINE_PER_CPU(bool, l1tf_flush_l1d); +EXPORT_SYMBOL_FOR_KVM_INTERNAL(l1tf_flush_l1d); +#endif + /* * Flags to manipulate forced emulation behavior (any non-zero value will * enable forced emulation). @@ -5190,7 +5196,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cp= u) { struct kvm_pmu *pmu =3D vcpu_to_pmu(vcpu); =20 - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_set_cpu_l1tf_flush_l1d(); =20 if (vcpu->scheduled_out && pmu->version && pmu->event_count) { pmu->need_cleanup =3D true; @@ -8000,7 +8006,7 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu= , gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception) { /* kvm_write_guest_virt_system can pull in tons of pages. */ - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_set_cpu_l1tf_flush_l1d(); =20 return kvm_write_guest_virt_helper(addr, val, bytes, vcpu, PFERR_WRITE_MASK, exception); @@ -9396,7 +9402,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gp= a_t cr2_or_gpa, return handle_emulation_failure(vcpu, emulation_type); } =20 - vcpu->arch.l1tf_flush_l1d =3D true; + kvm_set_cpu_l1tf_flush_l1d(); =20 if (!(emulation_type & EMULTYPE_NO_DECODE)) { kvm_clear_exception_queue(vcpu); --- base-commit: 6b36119b94d0b2bb8cea9d512017efafd461d6ac change-id: 20251013-b4-l1tf-percpu-793181fa5884 Best regards, --=20 Brendan Jackman