From nobody Wed Dec 17 10:45:09 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70A9017E44B for ; Fri, 12 Jul 2024 17:02:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720803731; cv=none; b=iQiR1Ey3Lf8qJf7JadsRV018rFoWmtAEef0V//MmDu/GdSr793xiJYDtwYY1HHbBOYabDLiEbs6gdE7E9hGpqFvaZsatZiN2hqvlRXx0ybB12aX5LYX76+SicF1kyeDVC6NbUixnZ9lp6C+iP+MWD+pu+u62RFo/zPuYrPf3SPA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720803731; c=relaxed/simple; bh=OmLuPXVN7trRIGuXekfaSbCyKoP5mCssZ2C3O0VIpDk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MvoQ0KhcJp52ZatKpdjQpzUPfsQxOtgLlUmzuUTHL/63Q/ItJtP6qHQNFOQTfEDgZNaDi6cPMwC1GLcxbkdA61ARTNjdzp75pROivm7mkz9RFVEeoLpSOxpFYMUMeewPb3ZBG1mQmvN72GQZOW9My0gUdI7vh++AXQucue690xU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=imE3CPHt; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="imE3CPHt" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-650fccfd373so35135037b3.3 for ; Fri, 12 Jul 2024 10:02:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803728; x=1721408528; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Zn579dgno7SZboGdojpY5JJU8xi+US4yr/l6Mjf8vz0=; b=imE3CPHt6jlzRoW0vuKAQGp9OZFSpMi25xzF3PEKMFcU1hZhDQWNKP1qvyePWygjd/ ufbf20V5a2sfpeaEi1gvQIPrb56a4zltC13T1R62xC8d6EpG1UhDblf2tojno2E54gzm xD3I+y4D+nRBv4ZhmxCmz+Dfg1yzStVjoUT4W/vbEzJVPhtZe8tCaH/vIIabL8O+uXu0 /zUvhLbc2ajV1oAh02quZOiCDIJg28/Lf4YpL7A0t9xHxSdLONIDvTpSWXsLGb9tcbIc k8fOfsh9Ae3hU6njntbwNjcoctdkhM5iSUeZhUZapmzoGk+wfnhsQ2s4TuatTuDZMlZm /Ewg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803728; x=1721408528; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Zn579dgno7SZboGdojpY5JJU8xi+US4yr/l6Mjf8vz0=; b=mJZuntGSAQXzKazBP7bjkgiJ4Jtefh94YIRcz7lGhIttgNtanF5OVCkA9RhIX9P8Tp 4OSMgLjKkzVVWlD5CNLJxcIKDI0EXp66dCF7WJL319nBfnfwAspufil2Brfnervx2aJv pubnnUGYi5+wEx688JSh0j5SOA9/Hu9XFynUiBWRJaXN9Xgypzy0G9WU2fotbrl+28TH INhi0ZaO+joDrbRPfd5YAzdveOtTve3i6s81x2IJlpV29P+bJ8FexINQfaY3DsSM42n6 fiyOHtOJWGDmwbwV3mouU+j/+5xkUheOqu6fM4jyd49k88V8F0GVPEaqUV17xPDMV1eL 7HJw== X-Forwarded-Encrypted: i=1; AJvYcCU5C1kxIc7YJXgwGbDFqF1k/ynVg33W4v6bwhoX7GlulK6PzsWX7Hq5RYmImvJzlhYIHaVCAyh/fM1IF/fwm942rKYBFedr9TeLzLT4 X-Gm-Message-State: AOJu0Yyh2Rcqv81Zi7tpYmmMND/CDpvFay+GIF2FsVM/QlI6EEgg5oMI DWyY3tfh89mqq2BcttcS+/oSDWVoZ0QF6nT+cvn08DQHXSVaFj+ZYpnmMPKG5Jq+wldfmxtsaQ1 H9sioEvg+Xg== X-Google-Smtp-Source: AGHT+IGmf1CJRMIpY+h+xba+0XgEdJs1Sqn1txDmOypcZe8mfXwCEjETMiBlkTEaYdYTy3241JdukP3H8LKm6w== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:690c:46c8:b0:62c:de05:5a78 with SMTP id 00721157ae682-658f01fd061mr625407b3.6.1720803728252; Fri, 12 Jul 2024 10:02:08 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:44 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-26-144b319a40d8@google.com> Subject: [PATCH 26/26] KVM: x86: asi: Add some mitigations on address space transitions From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Here we start actually turning ASI into a real exploit mitigation. On all CPUs we attempt to obliterate any indirect branch predictor training before mapping in any secrets. We can also flush side channels on the inverse transition. So, in this iteration we flush L1D, but only on CPUs affected by L1TF. The rationale for this is: L1TF seems to have been a relative outlier in terms of its impact, and the mitigation is obviously rather devastating. On the other hand, Spectre-type attacks are continuously being found, and it's quite reasonable to assume that existing systems are vulnerable to variations that are not currently mitigated by bespoke techniques like Safe RET. This is clearly an incomplete policy, for example it probably makes sense to perform MDS mitigations in post_asi_enter, and there is clearly a wide range of alternative postures with regard to per-platform vs blanket mitigation configurations. This also ought to be integrated more intelligently with bugs.c - this will probably require a fair bit of discussion so it might warrant a patchset all to itself. For now though, this ouhgt to provide an example of the kind of thing we might do with ASI. The changes to the inline asm for L1D flushes are to avoid duplicate jump labels breaking the build in the case that vmx_l1d_flush() gets inlined at multiple locations (as it seems to do in my builds). Signed-off-by: Brendan Jackman --- arch/x86/include/asm/kvm_host.h | 2 + arch/x86/include/asm/nospec-branch.h | 2 + arch/x86/kvm/vmx/vmx.c | 88 ++++++++++++++++++++++++--------= ---- arch/x86/kvm/x86.c | 33 +++++++++++++- arch/x86/lib/retpoline.S | 7 +++ 5 files changed, 101 insertions(+), 31 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 6c3326cb8273c..8b7226dd2e027 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1840,6 +1840,8 @@ struct kvm_x86_init_ops { =20 struct kvm_x86_ops *runtime_ops; struct kvm_pmu_ops *pmu_ops; + + void (*post_asi_enter)(void); }; =20 struct kvm_arch_async_pf { diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index ff5f1ecc7d1e6..9502bdafc1edd 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -605,6 +605,8 @@ static __always_inline void mds_idle_clear_cpu_buffers(= void) mds_clear_cpu_buffers(); } =20 +extern void fill_return_buffer(void); + #endif /* __ASSEMBLY__ */ =20 #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 1105d666a8ade..6efcbddf6ce27 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6629,37 +6629,18 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu, f= astpath_t exit_fastpath) * is not exactly LRU. This could be sized at runtime via topology * information but as all relevant affected CPUs have 32KiB L1D cache size * there is no point in doing so. + * + * Must be reentrant, for use by vmx_post_asi_enter. */ -static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) +static inline_or_noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) { int size =3D PAGE_SIZE << L1D_CACHE_ORDER; =20 /* - * This code is only executed when the flush mode is 'cond' or - * 'always' + * In theory we lose some of these increments to reentrancy under ASI. + * We just tolerate imprecise stats rather than deal with synchronizing. + * Anyway in practice on 64 bit it's gonna be a single instruction. */ - if (static_branch_likely(&vmx_l1d_flush_cond)) { - bool flush_l1d; - - /* - * Clear the per-vcpu flush bit, it gets set again - * either from vcpu_run() or from one of the unsafe - * VMEXIT handlers. - */ - flush_l1d =3D vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d =3D false; - - /* - * Clear the per-cpu flush bit, it gets set again from - * the interrupt handlers. - */ - flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); - kvm_clear_cpu_l1tf_flush_l1d(); - - if (!flush_l1d) - return; - } - vcpu->stat.l1d_flush++; =20 if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { @@ -6670,26 +6651,57 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *= vcpu) asm volatile( /* First ensure the pages are in the TLB */ "xorl %%eax, %%eax\n" - ".Lpopulate_tlb:\n\t" + ".Lpopulate_tlb_%=3D:\n\t" "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" "addl $4096, %%eax\n\t" "cmpl %%eax, %[size]\n\t" - "jne .Lpopulate_tlb\n\t" + "jne .Lpopulate_tlb_%=3D\n\t" "xorl %%eax, %%eax\n\t" "cpuid\n\t" /* Now fill the cache */ "xorl %%eax, %%eax\n" - ".Lfill_cache:\n" + ".Lfill_cache_%=3D:\n" "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" "addl $64, %%eax\n\t" "cmpl %%eax, %[size]\n\t" - "jne .Lfill_cache\n\t" + "jne .Lfill_cache_%=3D\n\t" "lfence\n" :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); } =20 +static noinstr void vmx_maybe_l1d_flush(struct kvm_vcpu *vcpu) +{ + /* + * This code is only executed when the flush mode is 'cond' or + * 'always' + */ + if (static_branch_likely(&vmx_l1d_flush_cond)) { + bool flush_l1d; + + /* + * Clear the per-vcpu flush bit, it gets set again + * either from vcpu_run() or from one of the unsafe + * VMEXIT handlers. + */ + flush_l1d =3D vcpu->arch.l1tf_flush_l1d; + vcpu->arch.l1tf_flush_l1d =3D false; + + /* + * Clear the per-cpu flush bit, it gets set again from + * the interrupt handlers. + */ + flush_l1d |=3D kvm_get_cpu_l1tf_flush_l1d(); + kvm_clear_cpu_l1tf_flush_l1d(); + + if (!flush_l1d) + return; + } + + vmx_l1d_flush(vcpu); +} + static void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int i= rr) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); @@ -7284,7 +7296,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vc= pu *vcpu, * This is only after asi_enter() for performance reasons. */ if (static_branch_unlikely(&vmx_l1d_should_flush)) - vmx_l1d_flush(vcpu); + vmx_maybe_l1d_flush(vcpu); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) mds_clear_cpu_buffers(); @@ -8321,6 +8333,14 @@ gva_t vmx_get_untagged_addr(struct kvm_vcpu *vcpu, g= va_t gva, unsigned int flags return (sign_extend64(gva, lam_bit) & ~BIT_ULL(63)) | (gva & BIT_ULL(63)); } =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +static noinstr void vmx_post_asi_enter(void) +{ + if (boot_cpu_has_bug(X86_BUG_L1TF)) + vmx_l1d_flush(kvm_get_running_vcpu()); +} +#endif + static struct kvm_x86_ops vmx_x86_ops __initdata =3D { .name =3D KBUILD_MODNAME, =20 @@ -8727,6 +8747,14 @@ static struct kvm_x86_init_ops vmx_init_ops __initda= ta =3D { =20 .runtime_ops =3D &vmx_x86_ops, .pmu_ops =3D &intel_pmu_ops, + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + /* + * Only Intel CPUs currently do anything in post-enter, so this is a + * vendor hook for now. + */ + .post_asi_enter =3D vmx_post_asi_enter, +#endif }; =20 static void vmx_cleanup_l1d_flush(void) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b9947e88d4ac6..b5e4df2aa1636 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9695,6 +9695,36 @@ static void kvm_x86_check_cpu_compat(void *ret) *(int *)ret =3D kvm_x86_check_processor_compatibility(); } =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +static noinstr void pre_asi_exit(void) +{ + /* + * Flush out prediction trainings by the guest before we go to access + * secrets. + */ + + /* Clear normal indirect branch predictions, if we haven't */ + if (cpu_feature_enabled(X86_FEATURE_IBPB) && + !cpu_feature_enabled(X86_FEATURE_IBPB_ON_VMEXIT)) + __wrmsr(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, 0); + + /* Flush the RAS/RSB if we haven't already. */ + if (!IS_ENABLED(CONFIG_RETPOLINE) || + !cpu_feature_enabled(X86_FEATURE_RSB_VMEXIT)) + fill_return_buffer(); +} + +struct asi_hooks asi_hooks =3D { + .pre_asi_exit =3D pre_asi_exit, + /* post_asi_enter populated later. */ +}; + +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +struct asi_hooks asi_hooks =3D {}; +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + + int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) { u64 host_pat; @@ -9753,7 +9783,8 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) if (r) goto out_free_percpu; =20 - r =3D asi_register_class("KVM", NULL); + asi_hooks.post_asi_enter =3D ops->post_asi_enter; + r =3D asi_register_class("KVM", &asi_hooks); if (r < 0) goto out_mmu_exit; kvm_asi_index =3D r; diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S index 391059b2c6fbc..db5b8ee01efeb 100644 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -396,3 +396,10 @@ SYM_CODE_END(__x86_return_thunk) EXPORT_SYMBOL(__x86_return_thunk) =20 #endif /* CONFIG_MITIGATION_RETHUNK */ + +.pushsection .noinstr.text, "ax" +SYM_CODE_START(fill_return_buffer) + __FILL_RETURN_BUFFER(%_ASM_AX,RSB_CLEAR_LOOPS) + RET +SYM_CODE_END(fill_return_buffer) +.popsection --=20 2.45.2.993.g49e7a77208-goog