From nobody Sun Nov 24 23:30:12 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF46613342F; Thu, 31 Oct 2024 15:39:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730389183; cv=none; b=pIA6/M8N9awsOEJHbz2vHZR2Oxv7DQl+2NvyI43w75q8vON3jJdk/VOfGmnNKJyUD3D3MOMvEQuoMNbYl1RKN+mTJ3/BEvO7iOQWoUPCPGsqmFmc5hWQTEvygCmiuxdqE2YMsGlB372qk4LJetGmIY1fXDk8mrv7OUp0pfeSpDw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730389183; c=relaxed/simple; bh=snmI6YMjx7sCmaq0eZOPPYPW0C2ycDeH1djt9cKAXl8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CstiTHT3byhv27PVF+Zxmo02ChoPXfKEQx/k9qKRCdKB78Jsw1+OCK/ZmghbrbxXqh0zQajG8EgFDSw7GEUgfKuXc5wnN0bPnRy7Bss/JPaHsBcBbWYplYo71mpi/nC8LD1IMjlWEngrXwy920MiL1End5nB9HMTTuzHfsmVERw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lI0P/rzz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lI0P/rzz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BAD40C4DDE6; Thu, 31 Oct 2024 15:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730389182; bh=snmI6YMjx7sCmaq0eZOPPYPW0C2ycDeH1djt9cKAXl8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lI0P/rzzRRDwX+HPXX43pikeNKkg84CFu3WGUqmbqeq9h5luU802vslH0vG4hjwyN u6x9RRdIza3fkpYJM7L+ISCA5enSRLHihBfGahAoqyvXqzuGXy3KFD9AyG4sXOhGIZ lcosX9TQTqieAsNpxz+NRKuPn9RcSFn05U5Jk+4bH9d5Y9d3wxzhyIv8aBEiX07GME ScOmWOxM2AC4O3GRFTGer6cTFzuwU/Np+eGohXDQyCeH+Vgnb23RC98HVpMj9ds5xR KVfUSVtkM1Rx+m8RjCjjHeo2tfeh8HrkWahqIyP5jsM7Swxb5qn93kyq0SplHv417W AfVe8oea5I2Iw== From: Amit Shah To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, x86@kernel.org, linux-doc@vger.kernel.org Cc: amit.shah@amd.com, thomas.lendacky@amd.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, jpoimboe@kernel.org, pawan.kumar.gupta@linux.intel.com, corbet@lwn.net, mingo@redhat.com, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, daniel.sneddon@linux.intel.com, kai.huang@intel.com, sandipan.das@amd.com, boris.ostrovsky@oracle.com, Babu.Moger@amd.com, david.kaplan@amd.com Subject: [PATCH 1/2] x86: cpu/bugs: add support for AMD ERAPS feature Date: Thu, 31 Oct 2024 16:39:24 +0100 Message-ID: <20241031153925.36216-2-amit@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241031153925.36216-1-amit@kernel.org> References: <20241031153925.36216-1-amit@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Amit Shah Remove explicit RET stuffing / filling on VMEXITs and context switches on AMD CPUs with the ERAPS feature (Turin+). With the Enhanced Return Address Prediction Security feature, any hardware TLB flush results in flushing of the RSB (aka RAP in AMD spec). This guarantees an RSB flush across context switches. The feature also explicitly tags host and guest addresses - eliminating the need for explicit flushing of the RSB on VMEXIT. The BTC_NO feature in AMD CPUs ensures RET predictions do not speculate from outside the RSB. Together, the BTC_NO and ERAPS features ensure no flushing or stuffing of the RSB is necessary anymore. Feature documented in AMD PPR 57238. Signed-off-by: Amit Shah --- Documentation/admin-guide/hw-vuln/spectre.rst | 5 +-- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/nospec-branch.h | 11 ++++++ arch/x86/kernel/cpu/bugs.c | 36 +++++++++++++------ 4 files changed, 40 insertions(+), 13 deletions(-) diff --git a/Documentation/admin-guide/hw-vuln/spectre.rst b/Documentation/= admin-guide/hw-vuln/spectre.rst index 132e0bc6007e..647c10c0307a 100644 --- a/Documentation/admin-guide/hw-vuln/spectre.rst +++ b/Documentation/admin-guide/hw-vuln/spectre.rst @@ -417,9 +417,10 @@ The possible values in this file are: =20 - Return stack buffer (RSB) protection status: =20 - =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 'RSB filling' Protection of RSB on context switch enabled - =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D + 'ERAPS' Hardware RSB flush on context switches + guest/host tags + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 - EIBRS Post-barrier Return Stack Buffer (PBRSB) protection status: =20 diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpuf= eatures.h index 913fd3a7bac6..665032b12871 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -458,6 +458,7 @@ #define X86_FEATURE_AUTOIBRS (20*32+ 8) /* Automatic IBRS */ #define X86_FEATURE_NO_SMM_CTL_MSR (20*32+ 9) /* SMM_CTL MSR is not presen= t */ =20 +#define X86_FEATURE_ERAPS (20*32+24) /* Enhanced RAP / RSB / RAS Security= */ #define X86_FEATURE_SBPB (20*32+27) /* Selective Branch Prediction Barrie= r */ #define X86_FEATURE_IBPB_BRTYPE (20*32+28) /* MSR_PRED_CMD[IBPB] flushes = all branch type predictions */ #define X86_FEATURE_SRSO_NO (20*32+29) /* CPU is not affected by SRSO */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index 96b410b1d4e8..24d0fe5d5a8b 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -117,6 +117,17 @@ * We define a CPP macro such that it can be used from both .S files and * inline assembly. It's possible to do a .macro and then include that * from C via asm(".include ") but let's not go there. + * + * AMD CPUs with the ERAPS feature may have a larger default RSB. These C= PUs + * use the default number of entries on a host, and can optionally (based = on + * hypervisor setup) use 32 (old) or the new default in a guest. The numb= er + * of default entries is reflected in CPUID 8000_0021:EBX[23:16]. + * + * With the ERAPS feature, RSB filling is not necessary anymore: the RSB is + * auto-cleared on a TLB flush (i.e. a context switch). Adapting the valu= e of + * RSB_CLEAR_LOOPS below for ERAPS would change it to a runtime variable + * instead of the current compile-time constant, so leave it as-is, as this + * works for both older CPUs, as well as newer ones with ERAPS. */ =20 #define RETPOLINE_THUNK_SIZE 32 diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 47a01d4028f6..83b34a522dd7 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1828,9 +1828,6 @@ static void __init spectre_v2_select_mitigation(void) * speculated return targets may come from the branch predictor, * which could have a user-poisoned BTB or BHB entry. * - * AMD has it even worse: *all* returns are speculated from the BTB, - * regardless of the state of the RSB. - * * When IBRS or eIBRS is enabled, the "user -> kernel" attack * scenario is mitigated by the IBRS branch prediction isolation * properties, so the RSB buffer filling wouldn't be necessary to @@ -1838,6 +1835,15 @@ static void __init spectre_v2_select_mitigation(void) * * The "user -> user" attack scenario is mitigated by RSB filling. * + * AMD CPUs without the BTC_NO bit may speculate return targets + * from the BTB. CPUs with BTC_NO do not speculate return targets + * from the BTB, even on RSB underflow. + * + * The ERAPS CPU feature (which implies the presence of BTC_NO) + * adds an RSB flush each time a TLB flush happens (i.e., on every + * context switch). So, RSB filling is not necessary for this + * attack type with ERAPS present. + * * 2) Poisoned RSB entry * * If the 'next' in-kernel return stack is shorter than 'prev', @@ -1848,17 +1854,24 @@ static void __init spectre_v2_select_mitigation(voi= d) * eIBRS. * * The "user -> user" scenario, also known as SpectreBHB, requires - * RSB clearing. + * RSB clearing on processors without ERAPS. * * So to mitigate all cases, unconditionally fill RSB on context - * switches. - * - * FIXME: Is this pointless for retbleed-affected AMD? + * switches when ERAPS is not present. */ - setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW); - pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context switc= h\n"); + if (!boot_cpu_has(X86_FEATURE_ERAPS)) { + setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW); + pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context swit= ch\n"); =20 - spectre_v2_determine_rsb_fill_type_at_vmexit(mode); + /* + * For guest -> host (or vice versa) RSB poisoning scenarios, + * determine the mitigation mode here. With ERAPS, RSB + * entries are tagged as host or guest - ensuring that neither + * the host nor the guest have to clear or fill RSB entries to + * avoid poisoning, skip RSB filling at VMEXIT in that case. + */ + spectre_v2_determine_rsb_fill_type_at_vmexit(mode); + } =20 /* * Retpoline protects the kernel, but doesn't protect firmware. IBRS @@ -2871,7 +2884,7 @@ static ssize_t spectre_v2_show_state(char *buf) spectre_v2_enabled =3D=3D SPECTRE_V2_EIBRS_LFENCE) return sysfs_emit(buf, "Vulnerable: eIBRS+LFENCE with unprivileged eBPF = and SMT\n"); =20 - return sysfs_emit(buf, "%s%s%s%s%s%s%s%s\n", + return sysfs_emit(buf, "%s%s%s%s%s%s%s%s%s\n", spectre_v2_strings[spectre_v2_enabled], ibpb_state(), boot_cpu_has(X86_FEATURE_USE_IBRS_FW) ? "; IBRS_FW" : "", @@ -2879,6 +2892,7 @@ static ssize_t spectre_v2_show_state(char *buf) boot_cpu_has(X86_FEATURE_RSB_CTXSW) ? "; RSB filling" : "", pbrsb_eibrs_state(), spectre_bhi_state(), + boot_cpu_has(X86_FEATURE_ERAPS) ? "; ERAPS hardware RSB flush" : "", /* this should always be at the end */ spectre_v2_module_string()); } --=20 2.47.0 From nobody Sun Nov 24 23:30:12 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C860613342F; Thu, 31 Oct 2024 15:39:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730389188; cv=none; b=Q2yEpoM/FsrBKj/h6JzMPXy8ur0LB4WiVdvZb0AswQQjKT6yR33fPOUpLkQT5izCtTm4LRCUPX4cmsb7CjuJ7+1t51ICl8Dy65wdi0R5P7B2JYJU7xHDrprpkBNe/CO4ZXMEQ7eyTDiLbfx6TTUu+TVd4R6VVJFNRkXa0OEWFsM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730389188; c=relaxed/simple; bh=e71oB15xt8MmnWusRVcbVCxjQ6isEJHcrB9uA5JO/lc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YzG24kNMjH+nbgwtpMPhyQthX2CZQHRdf/OoLdowa6RWAbxKcHb4THxAVm4Rs1KwCxxrxgUqbt8Xf6fOmU6Lq2AhbPVsw8GJaE7wg/9dDd2Epw7REwVu3sHnYZ6buRjAc5whX2IlYZHrK0bsmzOsUyIshX92WAYSPv8Sd8P1jTU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ULJr67lf; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ULJr67lf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D37EC4DDEF; Thu, 31 Oct 2024 15:39:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730389188; bh=e71oB15xt8MmnWusRVcbVCxjQ6isEJHcrB9uA5JO/lc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ULJr67lf+4KQuW7r9rwL1KDzGbwqfAOwmRBvRkKoF5RTN0gheHbDyKT2Pu/35w7b4 5W6JIA5KGaiO/y/ZgVNQ1wZpH7ahk4rym+A9YNHjQinVWzt/1HlI6NI/DXdWoEzAC4 7nHahXbe1dWWH7yxUNX7YQDiPGi1aQC1Mg1C5xYB1liy7Ad1MCUMnlhbUIiP+1QTy5 L66yUwMWP8SkJ8VRO3NnnzWbzyTyI5rLpnzJ8lAEm4jLLmp2DVl1q982ezeajQGymB 0PXinBfOeZlEDSlcZ6G/MIYtUpuv6vsOQ/ji6PhM7TSIVD07W4QpR5NN87X5HZKKYS TdnJpk0OBhJJw== From: Amit Shah To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, x86@kernel.org, linux-doc@vger.kernel.org Cc: amit.shah@amd.com, thomas.lendacky@amd.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, jpoimboe@kernel.org, pawan.kumar.gupta@linux.intel.com, corbet@lwn.net, mingo@redhat.com, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, daniel.sneddon@linux.intel.com, kai.huang@intel.com, sandipan.das@amd.com, boris.ostrovsky@oracle.com, Babu.Moger@amd.com, david.kaplan@amd.com Subject: [PATCH 2/2] x86: kvm: svm: add support for ERAPS and FLUSH_RAP_ON_VMRUN Date: Thu, 31 Oct 2024 16:39:25 +0100 Message-ID: <20241031153925.36216-3-amit@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241031153925.36216-1-amit@kernel.org> References: <20241031153925.36216-1-amit@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Amit Shah AMD CPUs with the ERAPS feature (Turin+) have a larger RSB (aka RAP). While the new default RSB size is used on the host without any software modification necessary, the RSB usage for guests is limited to the older value (32 entries) for backwards compatibility. With this patch, KVM enables guest mode to also use the default number of entries by setting the new ALLOW_LARGER_RAP bit in the VMCB. The two cases for backward compatibility that need special handling are nested guests, and guests using shadow paging (or when NPT is disabled): For nested guests: the ERAPS feature adds host/guest tagging to entries in the RSB, but does not distinguish between ASIDs. On a nested exit, the L0 hypervisor instructs the microcode (via another new VMCB bit, FLUSH_RAP_ON_VMRUN) to flush the RSB on the next VMRUN to prevent RSB poisoning attacks from an L2 guest to an L1 guest. With that in place, this feature can be exposed to guests. For shadow paging guests: do not expose this feature to guests; only expose if nested paging is enabled, to ensure context switches within guests trigger TLB flushes on the CPU -- thereby ensuring guest context switches flush guest RSB entries. For shadow paging, the CPU's CR3 is not used for guest processes, and hence cannot benefit from this feature. Signed-off-by: Amit Shah --- arch/x86/include/asm/svm.h | 6 +++++- arch/x86/kvm/cpuid.c | 15 ++++++++++++- arch/x86/kvm/svm/svm.c | 44 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/svm/svm.h | 15 +++++++++++++ 4 files changed, 78 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h index 2b59b9951c90..f8584a63c859 100644 --- a/arch/x86/include/asm/svm.h +++ b/arch/x86/include/asm/svm.h @@ -129,7 +129,8 @@ struct __attribute__ ((__packed__)) vmcb_control_area { u64 tsc_offset; u32 asid; u8 tlb_ctl; - u8 reserved_2[3]; + u8 erap_ctl; + u8 reserved_2[2]; u32 int_ctl; u32 int_vector; u32 int_state; @@ -175,6 +176,9 @@ struct __attribute__ ((__packed__)) vmcb_control_area { #define TLB_CONTROL_FLUSH_ASID 3 #define TLB_CONTROL_FLUSH_ASID_LOCAL 7 =20 +#define ERAP_CONTROL_ALLOW_LARGER_RAP 0 +#define ERAP_CONTROL_FLUSH_RAP 1 + #define V_TPR_MASK 0x0f =20 #define V_IRQ_SHIFT 8 diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 41786b834b16..2c2a60964a2e 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -797,6 +797,8 @@ void kvm_set_cpu_caps(void) F(WRMSR_XX_BASE_NS) ); =20 + if (tdp_enabled) + kvm_cpu_cap_check_and_set(X86_FEATURE_ERAPS); kvm_cpu_cap_check_and_set(X86_FEATURE_SBPB); kvm_cpu_cap_check_and_set(X86_FEATURE_IBPB_BRTYPE); kvm_cpu_cap_check_and_set(X86_FEATURE_SRSO_NO); @@ -1357,8 +1359,19 @@ static inline int __do_cpuid_func(struct kvm_cpuid_a= rray *array, u32 function) entry->eax =3D entry->ebx =3D entry->ecx =3D entry->edx =3D 0; break; case 0x80000021: - entry->ebx =3D entry->ecx =3D entry->edx =3D 0; + unsigned int ebx_mask =3D 0; + + entry->ecx =3D entry->edx =3D 0; cpuid_entry_override(entry, CPUID_8000_0021_EAX); + + /* + * Bits 23:16 in EBX indicate the size of the RSB. + * Expose the value in the hardware to the guest. + */ + if (kvm_cpu_cap_has(X86_FEATURE_ERAPS)) + ebx_mask |=3D GENMASK(23, 16); + + entry->ebx &=3D ebx_mask; break; /* AMD Extended Performance Monitoring and Debug */ case 0x80000022: { diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 9df3e1e5ae81..ecd290ff38f8 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1360,6 +1360,28 @@ static void init_vmcb(struct kvm_vcpu *vcpu) if (boot_cpu_has(X86_FEATURE_V_SPEC_CTRL)) set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SPEC_CTRL, 1, 1); =20 + /* + * If the hardware has a larger RSB, use it in the guest context as + * well. + * + * When running nested guests: the hardware tags host and guest RSB + * entries, but the entries are ASID agnostic. Differentiating L1 and + * L2 guests isn't possible in hardware. To prevent L2->L1 RSB + * poisoning attacks in this case, the L0 hypervisor must set + * FLUSH_RAP_ON_VMRUN in the L1's VMCB on a nested #VMEXIT to ensure + * the next VMRUN flushes the RSB. + * + * For shadow paging / NPT disabled case: the CPU's CR3 does not + * contain the CR3 of the running guest process, and hence intra-guest + * context switches will not cause a hardware TLB flush, which in turn + * does not result in a guest RSB flush that the ERAPS feature + * provides. Do not expose ERAPS or the larger RSB to the guest in + * this case, so the guest continues implementing software mitigations + * as well as only sees 32 entries for the RSB. + */ + if (boot_cpu_has(X86_FEATURE_ERAPS) && npt_enabled) + vmcb_set_larger_rap(svm->vmcb); + if (kvm_vcpu_apicv_active(vcpu)) avic_init_vmcb(svm, vmcb); =20 @@ -3393,6 +3415,7 @@ static void dump_vmcb(struct kvm_vcpu *vcpu) pr_err("%-20s%016llx\n", "tsc_offset:", control->tsc_offset); pr_err("%-20s%d\n", "asid:", control->asid); pr_err("%-20s%d\n", "tlb_ctl:", control->tlb_ctl); + pr_err("%-20s%d\n", "erap_ctl:", control->erap_ctl); pr_err("%-20s%08x\n", "int_ctl:", control->int_ctl); pr_err("%-20s%08x\n", "int_vector:", control->int_vector); pr_err("%-20s%08x\n", "int_state:", control->int_state); @@ -3559,6 +3582,27 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fa= stpath_t exit_fastpath) =20 trace_kvm_nested_vmexit(vcpu, KVM_ISA_SVM); =20 + if (boot_cpu_has(X86_FEATURE_ERAPS) + && vmcb_is_larger_rap(svm->vmcb01.ptr)) { + /* + * XXX a few further optimizations can be made: + * + * 1. In pre_svm_run() we can reset this bit when a hw + * TLB flush has happened - any context switch on a + * CPU (which causes a TLB flush) auto-flushes the RSB + * - eg when this vCPU is scheduled on a different + * pCPU. + * + * 2. This is also not needed in the case where the + * vCPU is being scheduled on the same pCPU, but there + * was a context switch between the #VMEXIT and VMRUN. + * + * 3. If the guest returns to L2 again after this + * #VMEXIT, there's no need to flush the RSB. + */ + vmcb_set_flush_rap(svm->vmcb01.ptr); + } + vmexit =3D nested_svm_exit_special(svm); =20 if (vmexit =3D=3D NESTED_EXIT_CONTINUE) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 43fa6a16eb19..8a7877f46dc5 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -500,6 +500,21 @@ static inline bool svm_is_intercept(struct vcpu_svm *s= vm, int bit) return vmcb_is_intercept(&svm->vmcb->control, bit); } =20 +static inline void vmcb_set_flush_rap(struct vmcb *vmcb) +{ + __set_bit(ERAP_CONTROL_FLUSH_RAP, (unsigned long *)&vmcb->control.erap_ct= l); +} + +static inline void vmcb_set_larger_rap(struct vmcb *vmcb) +{ + __set_bit(ERAP_CONTROL_ALLOW_LARGER_RAP, (unsigned long *)&vmcb->control.= erap_ctl); +} + +static inline bool vmcb_is_larger_rap(struct vmcb *vmcb) +{ + return test_bit(ERAP_CONTROL_ALLOW_LARGER_RAP, (unsigned long *)&vmcb->co= ntrol.erap_ctl); +} + static inline bool nested_vgif_enabled(struct vcpu_svm *svm) { return guest_can_use(&svm->vcpu, X86_FEATURE_VGIF) && --=20 2.47.0