Clean up KVM's handling of L1TF and MMIO Stale data, as the code has bit
rotted a bit and is harder than it should be to understand, and has a few
warts.
TL;DR:
- Unify L1TF flushing under per-CPU variable
- Bury L1TF L1D flushing under CONFIG_CPU_MITIGATIONS=y
- Move MMIO Stale Data into asm, and do VERW at most once per VM-Enter
To allow VMX to use ALTERNATIVE_2 to select slightly different flows for doing
VERW, tweak the low lever macros in nospec-branch.h to define the instruction
sequence, and then wrap it with __stringify() as needed.
As before, the non-VMX code is lightly tested (but there's far less chance
for breakage there). For the VMX code, I verified the KVM side of things by
hacking the code to force/clear various mitigations, and using ud2 to confirm
the right path got selected.
v5:
- Collect reviews and acks.
- Add/improve comments for various macros and flows. [Everyone]
- s/CLEAR_CPU_BUFFERS_SEQ/VERW [Pawan, Boris]
- Use the on-stack copy of @flags instead of stashing information in
RFLAGS' arithmetic flags. [Boris]
- Fix typos (hopefully). [Boris]
v4:
- https://lore.kernel.org/all/20251031003040.3491385-1-seanjc@google.com
- Drop the patch to fallback to handling the MMIO mitigation if
vmx_l1d_flush() doesn't flush, and instead use Pawan's approach of
decoupling the two entirely.
- Replace the static branch with X86_FEATURE_CLEAR_CPU_BUF_MMIO so that
it can be referenced in ALTERNATIVE macros.
- Decouple X86_FEATURE_CLEAR_CPU_BUF_VM from X86_FEATURE_CLEAR_CPU_BUF_MMIO
(though they still interact and can both be set)
v3:
- https://lore.kernel.org/all/20251016200417.97003-1-seanjc@google.com
- [Pawan's series] https://lore.kernel.org/all/20251029-verw-vm-v1-0-babf9b961519@linux.intel.com
- Put the "raw" variant in KVM, dress it up with KVM's "request" terminology,
and add a comment explaining why _KVM_ knows its usage doesn't need to
disable virtualization.
- Add the prep patches.
v2:
- https://lore.kernel.org/all/20251015-b4-l1tf-percpu-v2-1-6d7a8d3d40e9@google.com
- Moved the bit back to irq_stat
- Fixed DEBUG_PREEMPT issues by adding a _raw variant
v1: https://lore.kernel.org/r/20251013-b4-l1tf-percpu-v1-1-d65c5366ea1a@google.com
Brendan Jackman (1):
KVM: x86: Unify L1TF flushing under per-CPU variable
Pawan Gupta (1):
x86/bugs: Use VM_CLEAR_CPU_BUFFERS in VMX as well
Sean Christopherson (7):
KVM: VMX: Use on-stack copy of @flags in __vmx_vcpu_run()
x86/bugs: Decouple ALTERNATIVE usage from VERW macro definition
x86/bugs: Use an x86 feature to track the MMIO Stale Data mitigation
KVM: VMX: Handle MMIO Stale Data in VM-Enter assembly via
ALTERNATIVES_2
x86/bugs: KVM: Move VM_CLEAR_CPU_BUFFERS into SVM as
SVM_CLEAR_CPU_BUFFERS
KVM: VMX: Bundle all L1 data cache flush mitigation code together
KVM: VMX: Disable L1TF L1 data cache flush if CONFIG_CPU_MITIGATIONS=n
arch/x86/include/asm/cpufeatures.h | 5 +
arch/x86/include/asm/hardirq.h | 4 +-
arch/x86/include/asm/kvm_host.h | 3 -
arch/x86/include/asm/nospec-branch.h | 25 ++-
arch/x86/kernel/cpu/bugs.c | 22 +--
arch/x86/kvm/mmu/mmu.c | 2 +-
arch/x86/kvm/mmu/spte.c | 2 +-
arch/x86/kvm/svm/vmenter.S | 6 +-
arch/x86/kvm/vmx/nested.c | 2 +-
arch/x86/kvm/vmx/run_flags.h | 10 +-
arch/x86/kvm/vmx/vmenter.S | 29 ++--
arch/x86/kvm/vmx/vmx.c | 235 ++++++++++++++-------------
arch/x86/kvm/x86.c | 6 +-
arch/x86/kvm/x86.h | 14 ++
14 files changed, 193 insertions(+), 172 deletions(-)
base-commit: 16ec4fb4ac95d878b879192d280db2baeec43272
--
2.52.0.rc1.455.g30608eb744-goog