From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B838D531; Fri, 14 Mar 2025 00:35:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912535; cv=none; b=O6PSWNSOFB1L6CDCbdqVDE47DHYCbSN71bB/JiEPIjRxoURKadKqtMrbWeCx01shTjORzYcVcl3HKqbBfHWRL+0nGL/lG1sinRKn1gCRUk1b8DcbYTgVk6rYrtWfWT4CS5Js0Gfp2C8W9oljUOA2pIsDb8RChBdcQ2CaCNQdBSY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912535; c=relaxed/simple; bh=K/8Nrb3lLZOisRi5mleFnZ95iWg89OwMyWVRd0rs/Ng=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mMo5tHQamkO2atkHGjXT4PbK4JYlE1CJ13yRna3hoVO0UMLGr2iPgkkZbiltUKEgJLT+sYa0Cuoz5Uc/+wyCDQdxNWkdaKByzMd7s0wIyP8HtEpr8prBr0y1cVkZkhllGIq9WYdtZaIBLa+0J5FdIe7DrFqhVmeLq1iPG8Hwe9A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jMyICnZB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jMyICnZB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9854FC4CEE9; Fri, 14 Mar 2025 00:35:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912535; bh=K/8Nrb3lLZOisRi5mleFnZ95iWg89OwMyWVRd0rs/Ng=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=jMyICnZB6oCalk9uUfD3LsliP2MhQ3mdSE/haq3oDCVLgmr/aHkX3BRjIRQhC4W2o HlauAas2/CEvo3Jqm6wh1YxG6zY0/QnlAoWE6IvFlc7DXf3Enjol2LpOUc+yeRhZ+d cI94jsIKDKwzDkmkOFVly1prALFZarGGgQFEUupY39XYhfxBU5bclmi8D47WQ637qX gwnvViUv5LmPEQabgetfekDX93oU0gVdnl2G2j6lfyHHylqOyrIU2uSpwXrlSYm9cH 8IrlFAjEv76xgHnfM8h25DR4v+JVuPWXFP7TbhJiEEdrBGUw0+YUHqatJannZcmOr/ Ttys35BVKiClA== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:13 +0000 Subject: [PATCH 6.12 1/8] KVM: arm64: Calculate cptr_el2 traps on activating traps Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-1-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Fuad Tabba , James Clark X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=6732; i=broonie@kernel.org; h=from:subject:message-id; bh=hVLtg776pULslAUMCPL12ZcOO9mkq7DScfSM0RuRPfU=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nJl4E85uwb4toQVjJBOcaFn6SFFPemhCWR1XoR V8TF9RaJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5yQAKCRAk1otyXVSH0I3+B/ 9yO4wNvr7IlC0/OltDYc4YfnDbqvEVNLmuGCy6EJAtLxcQbALqMv4cjND/4CAwqlGuOZBnU4akcdpM ScelihNS+EZv5444YqL368qObJtHuXonI2vDCy9bMsG1cRb9eZWJwCu2mQMqgIPlnsqY0zbuy2UpFw sg94tLwOeb7PLgPi9qcJDmmezSkAcGn9zk34m9h24UfFR5U9KIyRajcRlD141l+m3lIMIBNhITbrbM X9fnnSDMrW110uI2ASxXtfPC+O+ywv3sIUwvruAZQeQUAYIat7d/ZOgLu0+oImGzJLdjsHrAKCB3sz zQ8gM6eojWbj4MMw/4W0GQ49t9TJGh X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Fuad Tabba [ Upstream commit 2fd5b4b0e7b440602455b79977bfa64dea101e6c ] Similar to VHE, calculate the value of cptr_el2 from scratch on activate traps. This removes the need to store cptr_el2 in every vcpu structure. Moreover, some traps, such as whether the guest owns the fp registers, need to be set on every vcpu run. Reported-by: James Clark Fixes: 5294afdbf45a ("KVM: arm64: Exclude FP ownership from kvm_vcpu_arch") Signed-off-by: Fuad Tabba Link: https://lore.kernel.org/r/20241216105057.579031-13-tabba@google.com Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/arm.c | 1 - arch/arm64/kvm/hyp/nvhe/pkvm.c | 29 ---------------------- arch/arm64/kvm/hyp/nvhe/switch.c | 51 ++++++++++++++++++++++++-----------= ---- 4 files changed, 32 insertions(+), 50 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 1bf70fa1045dcd6704bdcd233767cdb7522b316f..d148cf578cb84e7dec4d1add2af= a60a3c7a1e041 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -697,7 +697,6 @@ struct kvm_vcpu_arch { u64 hcr_el2; u64 hcrx_el2; u64 mdcr_el2; - u64 cptr_el2; =20 /* Exception Information */ struct kvm_vcpu_fault_info fault; diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 3cf65daa75a51f71fa7168016f8c3d81dc326cca..e6f0443210a8b7a65f616b25b2e= 6f74a05683ed6 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1577,7 +1577,6 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_v= cpu *vcpu, } =20 vcpu_reset_hcr(vcpu); - vcpu->arch.cptr_el2 =3D kvm_get_reset_cptr_el2(vcpu); =20 /* * Handle the "start in power-off" case. diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index 077d4098548d2c87abdd3931285d87798d63adb3..7c464340bcd078df58b74164707= f263a6bdb0272 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -28,8 +28,6 @@ static void pvm_init_traps_aa64pfr0(struct kvm_vcpu *vcpu) const u64 feature_ids =3D pvm_read_id_reg(vcpu, SYS_ID_AA64PFR0_EL1); u64 hcr_set =3D HCR_RW; u64 hcr_clear =3D 0; - u64 cptr_set =3D 0; - u64 cptr_clear =3D 0; =20 /* Protected KVM does not support AArch32 guests. */ BUILD_BUG_ON(FIELD_GET(ARM64_FEATURE_MASK(ID_AA64PFR0_EL1_EL0), @@ -59,21 +57,10 @@ static void pvm_init_traps_aa64pfr0(struct kvm_vcpu *vc= pu) /* Trap AMU */ if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64PFR0_EL1_AMU), feature_ids)) { hcr_clear |=3D HCR_AMVOFFEN; - cptr_set |=3D CPTR_EL2_TAM; - } - - /* Trap SVE */ - if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64PFR0_EL1_SVE), feature_ids)) { - if (has_hvhe()) - cptr_clear |=3D CPACR_ELx_ZEN; - else - cptr_set |=3D CPTR_EL2_TZ; } =20 vcpu->arch.hcr_el2 |=3D hcr_set; vcpu->arch.hcr_el2 &=3D ~hcr_clear; - vcpu->arch.cptr_el2 |=3D cptr_set; - vcpu->arch.cptr_el2 &=3D ~cptr_clear; } =20 /* @@ -103,7 +90,6 @@ static void pvm_init_traps_aa64dfr0(struct kvm_vcpu *vcp= u) const u64 feature_ids =3D pvm_read_id_reg(vcpu, SYS_ID_AA64DFR0_EL1); u64 mdcr_set =3D 0; u64 mdcr_clear =3D 0; - u64 cptr_set =3D 0; =20 /* Trap/constrain PMU */ if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_PMUVer), feature_ids)) { @@ -130,21 +116,12 @@ static void pvm_init_traps_aa64dfr0(struct kvm_vcpu *= vcpu) if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_TraceFilt), feature_ids= )) mdcr_set |=3D MDCR_EL2_TTRF; =20 - /* Trap Trace */ - if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_TraceVer), feature_ids)= ) { - if (has_hvhe()) - cptr_set |=3D CPACR_EL1_TTA; - else - cptr_set |=3D CPTR_EL2_TTA; - } - /* Trap External Trace */ if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_ExtTrcBuff), feature_id= s)) mdcr_clear |=3D MDCR_EL2_E2TB_MASK << MDCR_EL2_E2TB_SHIFT; =20 vcpu->arch.mdcr_el2 |=3D mdcr_set; vcpu->arch.mdcr_el2 &=3D ~mdcr_clear; - vcpu->arch.cptr_el2 |=3D cptr_set; } =20 /* @@ -195,10 +172,6 @@ static void pvm_init_trap_regs(struct kvm_vcpu *vcpu) /* Clear res0 and set res1 bits to trap potential new features. */ vcpu->arch.hcr_el2 &=3D ~(HCR_RES0); vcpu->arch.mdcr_el2 &=3D ~(MDCR_EL2_RES0); - if (!has_hvhe()) { - vcpu->arch.cptr_el2 |=3D CPTR_NVHE_EL2_RES1; - vcpu->arch.cptr_el2 &=3D ~(CPTR_NVHE_EL2_RES0); - } } =20 /* @@ -579,8 +552,6 @@ int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_v= cpu *host_vcpu, return ret; } =20 - hyp_vcpu->vcpu.arch.cptr_el2 =3D kvm_get_reset_cptr_el2(&hyp_vcpu->vcpu); - return 0; } =20 diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/swi= tch.c index cc69106734ca732ba9276ac1eaf84be3e7381648..81d933a71310fd1132b2450cd08= 108e071a2cf78 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -36,33 +36,46 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); =20 extern void kvm_nvhe_prepare_backtrace(unsigned long fp, unsigned long pc); =20 -static void __activate_traps(struct kvm_vcpu *vcpu) +static void __activate_cptr_traps(struct kvm_vcpu *vcpu) { - u64 val; + u64 val =3D CPTR_EL2_TAM; /* Same bit irrespective of E2H */ =20 - ___activate_traps(vcpu, vcpu->arch.hcr_el2); - __activate_traps_common(vcpu); + if (has_hvhe()) { + val |=3D CPACR_ELx_TTA; =20 - val =3D vcpu->arch.cptr_el2; - val |=3D CPTR_EL2_TAM; /* Same bit irrespective of E2H */ - val |=3D has_hvhe() ? CPACR_EL1_TTA : CPTR_EL2_TTA; - if (cpus_have_final_cap(ARM64_SME)) { - if (has_hvhe()) - val &=3D ~CPACR_ELx_SMEN; - else - val |=3D CPTR_EL2_TSM; - } + if (guest_owns_fp_regs()) { + val |=3D CPACR_ELx_FPEN; + if (vcpu_has_sve(vcpu)) + val |=3D CPACR_ELx_ZEN; + } + } else { + val |=3D CPTR_EL2_TTA | CPTR_NVHE_EL2_RES1; =20 - if (!guest_owns_fp_regs()) { - if (has_hvhe()) - val &=3D ~(CPACR_ELx_FPEN | CPACR_ELx_ZEN); - else - val |=3D CPTR_EL2_TFP | CPTR_EL2_TZ; + /* + * Always trap SME since it's not supported in KVM. + * TSM is RES1 if SME isn't implemented. + */ + val |=3D CPTR_EL2_TSM; =20 - __activate_traps_fpsimd32(vcpu); + if (!vcpu_has_sve(vcpu) || !guest_owns_fp_regs()) + val |=3D CPTR_EL2_TZ; + + if (!guest_owns_fp_regs()) + val |=3D CPTR_EL2_TFP; } =20 + if (!guest_owns_fp_regs()) + __activate_traps_fpsimd32(vcpu); + kvm_write_cptr_el2(val); +} + +static void __activate_traps(struct kvm_vcpu *vcpu) +{ + ___activate_traps(vcpu, vcpu->arch.hcr_el2); + __activate_traps_common(vcpu); + __activate_cptr_traps(vcpu); + write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el2); =20 if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { --=20 2.39.5 From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD70E17D2; Fri, 14 Mar 2025 00:35:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912542; cv=none; b=PybxXGUWIcat2V1MPGgm6ih6cJ4YDJIOueeCxJepqYOUGKT/B8xAq+6EIya7F68gSyd1O1aG3ycpDzUGqQXEL/ho0VSS0xJN2ySn9kiNNWecv+X0u+PA4oimQxRCnCU6b5p8UXgHw/aax/h1Y/YTCxA7NPvK106t5ixvMc65CwQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912542; c=relaxed/simple; bh=Zdxybs9qTB+chInoy1tkbAbwDdjJASdda8TGhEjk+WQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mDflmEBeZq/n39nGaq3veC2jmMk+7F5P3AqRBZtvQ5Jps9euoUUo2jnpN0hScU+rVqa9YYT2uWlQ9jszLsswQVugV4shDYGyoH8M69BfUrAm0M7987KCoBTtTkB57Fo32VWV9s6Kr8hpZGHJiKFwFp8GFYCPseCX3Ay8yjQAtgo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HfHYAMA9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HfHYAMA9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id ACC2BC4CEDD; Fri, 14 Mar 2025 00:35:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912539; bh=Zdxybs9qTB+chInoy1tkbAbwDdjJASdda8TGhEjk+WQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=HfHYAMA9z4QCgeIqQUdyw1WOaV2SPAByZ3DmJZKs9YTPqx90FbGrMo63HQ6L6Vsiy qHBIB6vZr4XMj8Cemp2YSqcWxqPlqR4ofJGF/lStzrtWlDYunWs9lyb31VMeadRl+m sgY1zbkc3I1g7RDSeznBeJSYxwpDfBcz2JrqxTRqMCPteP4wlWGMZBsMaGOR9ZeyJM CLsxi3E1C1UvRksP1rZmBvnwNArD/GEn49Vf9h8vhOxmMx41gWBU8eT9YXZ5xwerqb rl9Hxb2ArZFZ2Yc6gKfea6Tfs9VHLh8OIK9Xf63wIdgrQNgGLcyojA3WuhYUHhdIYz ppqUZmzCFIfVg== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:14 +0000 Subject: [PATCH 6.12 2/8] KVM: arm64: Unconditionally save+flush host FPSIMD/SVE/SME state Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-2-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Eric Auger , Wilco Dijkstra , Eric Auger , Florian Weimer , Fuad Tabba , Jeremy Linton , Paolo Bonzini X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=6408; i=broonie@kernel.org; h=from:subject:message-id; bh=niJ7ePIXQKBW2n2ES1Y53jjOi/0EaRCqCKlSRtTk3s4=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nKkUnTwHfjPM3fgCC8ucNuJh9+nwD15AJn3y44 wBzzOOOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5ygAKCRAk1otyXVSH0LffCA CF/e+MkdkYSCkf/8T0++/2pq7U6fS0Ig21o1btqdQVbL6pKtrY1yoj4R6J33jNzKI7xrP/TKtGGhDo W7j8vjmv16TKf9zZRANT3zuTFucPQTjJz3eOaD8lNSgrc5ELEARcGBaUH0OQC7Xjug1J2aOsleoDoq haiNG8M30Cdi2KYqgWXeNaSPimK+pL6u+DjUZOmJEqP5SxVduCF2dpzoA7SwkxFHx4EffCf7juX7nk J5XfK84ZkktLwem3Cm8Xjn592Sy6Bjpex6du2LnITFojIy0oktUBuNWCyqe82xaZ90+cLVlss0jHdH 07pzbKcIT4hDDL95r8ln1BEjO/X2xI X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Mark Rutland [ Upstream commit fbc7e61195e23f744814e78524b73b59faa54ab4 ] There are several problems with the way hyp code lazily saves the host's FPSIMD/SVE state, including: * Host SVE being discarded unexpectedly due to inconsistent configuration of TIF_SVE and CPACR_ELx.ZEN. This has been seen to result in QEMU crashes where SVE is used by memmove(), as reported by Eric Auger: https://issues.redhat.com/browse/RHEL-68997 * Host SVE state is discarded *after* modification by ptrace, which was an unintentional ptrace ABI change introduced with lazy discarding of SVE st= ate. * The host FPMR value can be discarded when running a non-protected VM, where FPMR support is not exposed to a VM, and that VM uses FPSIMD/SVE. In these cases the hyp code does not save the host's FPMR before unbinding the host's FPSIMD/SVE/SME state, leaving a stale value in memory. Avoid these by eagerly saving and "flushing" the host's FPSIMD/SVE/SME state when loading a vCPU such that KVM does not need to save any of the host's FPSIMD/SVE/SME state. For clarity, fpsimd_kvm_prepare() is removed and the necessary call to fpsimd_save_and_flush_cpu_state() is placed in kvm_arch_vcpu_load_fp(). As 'fpsimd_state' and 'fpmr_ptr' should not be used, they are set to NULL; all uses of these will be removed in subsequent patches. Historical problems go back at least as far as v5.17, e.g. erroneous assumptions about TIF_SVE being clear in commit: 8383741ab2e773a9 ("KVM: arm64: Get rid of host SVE tracking/saving") ... and so this eager save+flush probably needs to be backported to ALL stable trees. Fixes: 93ae6b01bafee8fa ("KVM: arm64: Discard any SVE state when entering K= VM guests") Fixes: 8c845e2731041f0f ("arm64/sve: Leave SVE enabled on syscall if we don= 't context switch") Fixes: ef3be86021c3bdf3 ("KVM: arm64: Add save/restore support for FPMR") Reported-by: Eric Auger Reported-by: Wilco Dijkstra Reviewed-by: Mark Brown Tested-by: Mark Brown Tested-by: Eric Auger Acked-by: Will Deacon Cc: Catalin Marinas Cc: Florian Weimer Cc: Fuad Tabba Cc: Jeremy Linton Cc: Marc Zyngier Cc: Oliver Upton Cc: Paolo Bonzini Signed-off-by: Mark Rutland Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-2-mark.rutland@arm.c= om Signed-off-by: Marc Zyngier [ Mark: Handle vcpu/host flag conflict ] Signed-off-by: Mark Rutland Signed-off-by: Mark Brown --- arch/arm64/kernel/fpsimd.c | 25 ------------------------- arch/arm64/kvm/fpsimd.c | 35 ++++++++++------------------------- 2 files changed, 10 insertions(+), 50 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 6d21971ae5594f32947480cfa168db400a69a283..f38d22dac140f1b2a8de3f2f2ba= 1e5da22d2d1c8 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1694,31 +1694,6 @@ void fpsimd_signal_preserve_current_state(void) sve_to_fpsimd(current); } =20 -/* - * Called by KVM when entering the guest. - */ -void fpsimd_kvm_prepare(void) -{ - if (!system_supports_sve()) - return; - - /* - * KVM does not save host SVE state since we can only enter - * the guest from a syscall so the ABI means that only the - * non-saved SVE state needs to be saved. If we have left - * SVE enabled for performance reasons then update the task - * state to be FPSIMD only. - */ - get_cpu_fpsimd_context(); - - if (test_and_clear_thread_flag(TIF_SVE)) { - sve_to_fpsimd(current); - current->thread.fp_type =3D FP_STATE_FPSIMD; - } - - put_cpu_fpsimd_context(); -} - /* * Associate current's FPSIMD context with this cpu * The caller must have ownership of the cpu FPSIMD context before calling diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index ea5484ce1f3ba3121b6938bda15f7a8057d49051..efb54ed60fe1d1d8a904b10a4a4= bd3c820d9dac5 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -54,16 +54,18 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) if (!system_supports_fpsimd()) return; =20 - fpsimd_kvm_prepare(); - /* - * We will check TIF_FOREIGN_FPSTATE just before entering the - * guest in kvm_arch_vcpu_ctxflush_fp() and override this to - * FP_STATE_FREE if the flag set. + * Ensure that any host FPSIMD/SVE/SME state is saved and unbound such + * that the host kernel is responsible for restoring this state upon + * return to userspace, and the hyp code doesn't need to save anything. + * + * When the host may use SME, fpsimd_save_and_flush_cpu_state() ensures + * that PSTATE.{SM,ZA} =3D=3D {0,0}. */ - *host_data_ptr(fp_owner) =3D FP_STATE_HOST_OWNED; - *host_data_ptr(fpsimd_state) =3D kern_hyp_va(¤t->thread.uw.fpsimd_s= tate); - *host_data_ptr(fpmr_ptr) =3D kern_hyp_va(¤t->thread.uw.fpmr); + fpsimd_save_and_flush_cpu_state(); + *host_data_ptr(fp_owner) =3D FP_STATE_FREE; + *host_data_ptr(fpsimd_state) =3D NULL; + *host_data_ptr(fpmr_ptr) =3D NULL; =20 vcpu_clear_flag(vcpu, HOST_SVE_ENABLED); if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) @@ -73,23 +75,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) vcpu_clear_flag(vcpu, HOST_SME_ENABLED); if (read_sysreg(cpacr_el1) & CPACR_EL1_SMEN_EL0EN) vcpu_set_flag(vcpu, HOST_SME_ENABLED); - - /* - * If PSTATE.SM is enabled then save any pending FP - * state and disable PSTATE.SM. If we leave PSTATE.SM - * enabled and the guest does not enable SME via - * CPACR_EL1.SMEN then operations that should be valid - * may generate SME traps from EL1 to EL1 which we - * can't intercept and which would confuse the guest. - * - * Do the same for PSTATE.ZA in the case where there - * is state in the registers which has not already - * been saved, this is very unlikely to happen. - */ - if (read_sysreg_s(SYS_SVCR) & (SVCR_SM_MASK | SVCR_ZA_MASK)) { - *host_data_ptr(fp_owner) =3D FP_STATE_FREE; - fpsimd_save_and_flush_cpu_state(); - } } =20 /* --=20 2.39.5 From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1DE379F5; Fri, 14 Mar 2025 00:35:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912542; cv=none; b=pizwA91q+5I6E68nk2mtFD9e10fIDCvOHExhFRImv0OVjn6EgrQ9YXKJ/TqeJbuONL3BuVdX0cb6q6mRFFNdY8hBQ/vosfzGtHP3yDTNjpYA6iWOksz8cEt9ea64FHaZ20TRJm8ObZL1JYMOkk2WFNb9+p25tl8M7VsE/ig7OW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912542; c=relaxed/simple; bh=sdmZYjSKuQYlgeObZgUxNcOjWmmjERzES6RrhaWvdbs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=nACppJnlPF+ZaK2SKOb53yoBd8umLshc9Dyj1nhfcL+Baeik5shwj+sF62EvhsaYgGPhC8Y/JRnzaPO6o4Ehr8BnhAORh0Xx8k1CRm/5AEJlK9f21EUMBxEHaivcEknrLTDl8savPb6rKoSdNlkfucc8vGP9sVj4pKAuDB2u6DU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j7M3eJXh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j7M3eJXh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D16F7C4CEEE; Fri, 14 Mar 2025 00:35:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912542; bh=sdmZYjSKuQYlgeObZgUxNcOjWmmjERzES6RrhaWvdbs=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=j7M3eJXhVEi6uVZTB9xmZqHMymkc0GPeayaNiqtANpkI1FJ7HyR9+giATBk7sF3gE FBmdlG7LPzuNkqILq/IRuS1qLLTHSeHmKcv+SJEwSMg5raOdv5P5EtV9SZf49d1KP7 HzYYeoD2QOnv+4vy+bWNx/3O2c7t7zi8tDCkwXxjG0jC1yxWApC9daSCUZWEz79hh0 /wbnGDMVfg2ez/HqPRCu5zKYOv87qMhxuXEW0TaX8p4uwy2QbN3OtIZjqh+yYcViui /5y9wGzQRjqhb2XLJ0BQsPyE9UhyFih1Tzeha76AYrzUqhT3NvzttSLk5/6NstQZkG 7RvsKa6mPe3lw== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:15 +0000 Subject: [PATCH 6.12 3/8] KVM: arm64: Remove host FPSIMD saving for non-protected KVM Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-3-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=8015; i=broonie@kernel.org; h=from:subject:message-id; bh=B7wPFF9YCtOLnvOO1e5I1AsTzXQtKnhMVVgl4qZXzEw=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nLrDwOKvSDQTZPZf+b5TQyfduhDYMSzj5tQ+lO 8DZtD6+JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5ywAKCRAk1otyXVSH0MczB/ 9CI6049p0PAAl+rol27UWJXPS6g0vl1CiKPGmsFUuy3j4ZYzOLrWJll8+O54QUP/lcHklqxORGfsj0 PcWQbfogCoXW/3m6vCekyGDz0GOXJtOFDU6q2IJ1ALm6Ig6k+7M8xGvrigv8bkUKcO9qeIxtqkmJwi lnRraNw0w7vdhV6qGqXC8aBxjunl/VlhGWmaqUGRWsuwxJAOaTMEUu3py/ymc35GQp57Nc7m2xaiq3 L6aa7EeXmFV8CChd48KVSdIaU14pzZ4IefnxnFm9myH/68PasQpbd6j6c4z8E7KmMm0cQqTOMSHaeS kb0QLnaFZs3eSiL7UrfXCvfRtR25uP X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Mark Rutland Now that the host eagerly saves its own FPSIMD/SVE/SME state, non-protected KVM never needs to save the host FPSIMD/SVE/SME state, and the code to do this is never used. Protected KVM still needs to save/restore the host FPSIMD/SVE state to avoid leaking guest state to the host (and to avoid revealing to the host whether the guest used FPSIMD/SVE/SME), and that code needs to be retained. Remove the unused code and data structures. To avoid the need for a stub copy of kvm_hyp_save_fpsimd_host() in the VHE hyp code, the nVHE/hVHE version is moved into the shared switch header, where it is only invoked when KVM is in protected mode. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-3-mark.rutland@arm.c= om Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 20 +++++--------------- arch/arm64/kvm/arm.c | 8 -------- arch/arm64/kvm/fpsimd.c | 2 -- arch/arm64/kvm/hyp/include/hyp/switch.h | 25 +++++++++++++++++++++++-- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 2 +- arch/arm64/kvm/hyp/nvhe/switch.c | 28 ---------------------------- arch/arm64/kvm/hyp/vhe/switch.c | 8 -------- 7 files changed, 29 insertions(+), 64 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index d148cf578cb84e7dec4d1add2afa60a3c7a1e041..d8802490b25cba65369f03d9462= 7a2624f14b072 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -602,23 +602,13 @@ struct kvm_host_data { struct kvm_cpu_context host_ctxt; =20 /* - * All pointers in this union are hyp VA. + * Hyp VA. * sve_state is only used in pKVM and if system_supports_sve(). */ - union { - struct user_fpsimd_state *fpsimd_state; - struct cpu_sve_state *sve_state; - }; - - union { - /* HYP VA pointer to the host storage for FPMR */ - u64 *fpmr_ptr; - /* - * Used by pKVM only, as it needs to provide storage - * for the host - */ - u64 fpmr; - }; + struct cpu_sve_state *sve_state; + + /* Used by pKVM only. */ + u64 fpmr; =20 /* Ownership of the FP regs */ enum { diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index e6f0443210a8b7a65f616b25b2e6f74a05683ed6..634d3f62481827a3bf3aba6bf78= cafed71b5bd32 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -2476,14 +2476,6 @@ static void finalize_init_hyp_mode(void) per_cpu_ptr_nvhe_sym(kvm_host_data, cpu)->sve_state =3D kern_hyp_va(sve_state); } - } else { - for_each_possible_cpu(cpu) { - struct user_fpsimd_state *fpsimd_state; - - fpsimd_state =3D &per_cpu_ptr_nvhe_sym(kvm_host_data, cpu)->host_ctxt.f= p_regs; - per_cpu_ptr_nvhe_sym(kvm_host_data, cpu)->fpsimd_state =3D - kern_hyp_va(fpsimd_state); - } } } =20 diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index efb54ed60fe1d1d8a904b10a4a4bd3c820d9dac5..2ee6bde85235581d6bc9cba7e57= 8c55875b5d5a1 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -64,8 +64,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) */ fpsimd_save_and_flush_cpu_state(); *host_data_ptr(fp_owner) =3D FP_STATE_FREE; - *host_data_ptr(fpsimd_state) =3D NULL; - *host_data_ptr(fpmr_ptr) =3D NULL; =20 vcpu_clear_flag(vcpu, HOST_SVE_ENABLED); if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/i= nclude/hyp/switch.h index 5310fe1da6165bcdedfb5ce61bce353e4c9dd58b..a7f6a653f33718d1a25e232608e= 63ea287f2a672 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -344,7 +344,28 @@ static inline void __hyp_sve_save_host(void) true); } =20 -static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu); +static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu) +{ + /* + * Non-protected kvm relies on the host restoring its sve state. + * Protected kvm restores the host's sve state as not to reveal that + * fpsimd was used by a guest nor leak upper sve bits. + */ + if (system_supports_sve()) { + __hyp_sve_save_host(); + + /* Re-enable SVE traps if not supported for the guest vcpu. */ + if (!vcpu_has_sve(vcpu)) + cpacr_clear_set(CPACR_ELx_ZEN, 0); + + } else { + __fpsimd_save_state(host_data_ptr(host_ctxt.fp_regs)); + } + + if (kvm_has_fpmr(kern_hyp_va(vcpu->kvm))) + *host_data_ptr(fpmr) =3D read_sysreg_s(SYS_FPMR); +} + =20 /* * We trap the first access to the FP/SIMD to save the host context and @@ -394,7 +415,7 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu= , u64 *exit_code) isb(); =20 /* Write out the host state if it's in the registers */ - if (host_owns_fp_regs()) + if (is_protected_kvm_enabled() && host_owns_fp_regs()) kvm_hyp_save_fpsimd_host(vcpu); =20 /* Restore the guest state */ diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/h= yp-main.c index fefc89209f9e41c95478f6770881eb314a38b4c2..4e757a77322c9efc59cdff50174= 5f7c80d452358 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -83,7 +83,7 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) if (system_supports_sve()) __hyp_sve_restore_host(); else - __fpsimd_restore_state(*host_data_ptr(fpsimd_state)); + __fpsimd_restore_state(host_data_ptr(host_ctxt.fp_regs)); =20 if (has_fpmr) write_sysreg_s(*host_data_ptr(fpmr), SYS_FPMR); diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/swi= tch.c index 81d933a71310fd1132b2450cd08108e071a2cf78..3ce16f90fe6af7be21bc7b84a9d= 8b3905b8b08a7 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -193,34 +193,6 @@ static bool kvm_handle_pvm_sys64(struct kvm_vcpu *vcpu= , u64 *exit_code) kvm_handle_pvm_sysreg(vcpu, exit_code)); } =20 -static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu) -{ - /* - * Non-protected kvm relies on the host restoring its sve state. - * Protected kvm restores the host's sve state as not to reveal that - * fpsimd was used by a guest nor leak upper sve bits. - */ - if (unlikely(is_protected_kvm_enabled() && system_supports_sve())) { - __hyp_sve_save_host(); - - /* Re-enable SVE traps if not supported for the guest vcpu. */ - if (!vcpu_has_sve(vcpu)) - cpacr_clear_set(CPACR_ELx_ZEN, 0); - - } else { - __fpsimd_save_state(*host_data_ptr(fpsimd_state)); - } - - if (kvm_has_fpmr(kern_hyp_va(vcpu->kvm))) { - u64 val =3D read_sysreg_s(SYS_FPMR); - - if (unlikely(is_protected_kvm_enabled())) - *host_data_ptr(fpmr) =3D val; - else - **host_data_ptr(fpmr_ptr) =3D val; - } -} - static const exit_handler_fn hyp_exit_handlers[] =3D { [0 ... ESR_ELx_EC_MAX] =3D NULL, [ESR_ELx_EC_CP15_32] =3D kvm_hyp_handle_cp15_32, diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switc= h.c index 80581b1c399595fd64d0ccada498edac322480a6..e7ca0424107adec2371ae4553eb= ab9857c60b6d9 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -309,14 +309,6 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu,= u64 *exit_code) return true; } =20 -static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu) -{ - __fpsimd_save_state(*host_data_ptr(fpsimd_state)); - - if (kvm_has_fpmr(vcpu->kvm)) - **host_data_ptr(fpmr_ptr) =3D read_sysreg_s(SYS_FPMR); -} - static bool kvm_hyp_handle_tlbi_el2(struct kvm_vcpu *vcpu, u64 *exit_code) { int ret =3D -EINVAL; --=20 2.39.5 From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 267F53597E; Fri, 14 Mar 2025 00:35:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912546; cv=none; b=C9bVSQNK2T192DHL6EJl+37ASjOmOz7zcJrLxoGrHfM29WYxTqWqY4YAHLHViYxeKTHVE40aGLEiL8qrN97mJ+k/Odf9izy+f3Pi4HLGzorPHsfe4ZkBO1eoL+gi8tNYCdFR2dOUNeDyymt4Bw33BRMWZoYq7bZpgoC7tNaxRzY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912546; c=relaxed/simple; bh=4+mYHr6XrwTcEyqTdBUleqRnti7WbSWNsKz630OxK1Q=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=sgGCxt+2bQ/pOQ4gQ71OHhiTFMUy+WMlJbUHCQQhhKjf7W9MiAqK6Aq4rJ2fIJK8N6SuHNrl+LjfmdYXpzhcoAd1ErIDVJVFOGwM6nfnpjIZrzndOhfbPDn8r5xhII/MzXR4tKh/T8hPw6chsBvNHg1Qs57mkZRX9A7914Ytss0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rrxKdFZq; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rrxKdFZq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E90CBC4CEDD; Fri, 14 Mar 2025 00:35:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912545; bh=4+mYHr6XrwTcEyqTdBUleqRnti7WbSWNsKz630OxK1Q=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=rrxKdFZqKvTcyDncxV/TtSYVugzseXFWXAmEVnPDNOQDdLavoVRVv0dEfvOoXLPq/ 3+bnSwyB8ElyB5l/V4yYSpX6bRUsO5fDJv//ER6pYHQO4QnzbtomXc/l86EUKIh3Em o++PCkqxgMF37Zj2u1SOWXhQzz0Q0MszZy9/4LdobltK/J//j+zuECkUCmmJCbYztV YIXmy/WZZJt+rDPt2p3JYJtNmOiGlJbpgR95HxGFPsATlHAE4CZJL3UUA5MCUAr505 9/PkIn2Pz4FdFcy5Hii/U5Mdc0jh59qdo6lxSbL1o/ynmDZ4hIV/8mrpg5i6MUO3kN Q5gi6A/SzZxFQ== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:16 +0000 Subject: [PATCH 6.12 4/8] KVM: arm64: Remove VHE host restore of CPACR_EL1.ZEN Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-4-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=3661; i=broonie@kernel.org; h=from:subject:message-id; bh=IG25WLsxwjh6t3HKhjHMAeq/c25F13vYeIg5ZhI8+KE=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nMwHYWPXekYPZtInw5WqLEUbkeLchxCD9rolOf FekNG1aJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5zAAKCRAk1otyXVSH0JH8B/ 4uvhzEEwUrhCw1hZhx73ltY0sESRwMaZFpCuIxaMnt05Y47SbkbUKLjMP/4AqzQIF7YNgqq7mw4c7M rVl4fKS/IGw8Vk+N3bXuyBJlell9CbONJK+KX7Smr+8UBE9a2S9gbFm3sasyig77wMfcnn31nuaRWQ vMtlPvMOvUDfYW8BPAQR/o/EqV2/RDXSMFZLfP/OpODVYAwqCywQ9Q452OoQ40UYXSew8Fpa7dgd2r yHKVWovPDSmk4C4qLEChLTD37/R1cy/2E/FZhvm8jasA4amUrBMPDcpl0U6petXt/tKbi3P8HA42io DtvMqIO5sBHLfUSzuIZRAtVk4FJ4bX X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Mark Rutland [ Upstream commit 459f059be702056d91537b99a129994aa6ccdd35 ] When KVM is in VHE mode, the host kernel tries to save and restore the configuration of CPACR_EL1.ZEN (i.e. CPTR_EL2.ZEN when HCR_EL2.E2H=3D1) across kvm_arch_vcpu_load_fp() and kvm_arch_vcpu_put_fp(), since the configuration may be clobbered by hyp when running a vCPU. This logic is currently redundant. The VHE hyp code unconditionally configures CPTR_EL2.ZEN to 0b01 when returning to the host, permitting host kernel usage of SVE. Now that the host eagerly saves and unbinds its own FPSIMD/SVE/SME state, there's no need to save/restore the state of the EL0 SVE trap. The kernel can safely save/restore state without trapping, as described above, and will restore userspace state (including trap controls) before returning to userspace. Remove the redundant logic. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-4-mark.rutland@arm.c= om Signed-off-by: Marc Zyngier [Rework for refactoring of where the flags are stored -- broonie] Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 2 -- arch/arm64/kvm/fpsimd.c | 16 ---------------- 2 files changed, 18 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index d8802490b25cba65369f03d94627a2624f14b072..1a126fa657fcde3433f37f3aaf4= 64a5b6a5f095d 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -891,8 +891,6 @@ struct kvm_vcpu_arch { /* Save TRBE context if active */ #define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6)) =20 -/* SVE enabled for host EL0 */ -#define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0)) /* SME enabled for EL0 */ #define HOST_SME_ENABLED __vcpu_single_flag(sflags, BIT(1)) /* Physical CPU not in supported_cpus */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 2ee6bde85235581d6bc9cba7e578c55875b5d5a1..4127abfd319c2c683d2281efa52= a6abe5fac67ee 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -65,10 +65,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) fpsimd_save_and_flush_cpu_state(); *host_data_ptr(fp_owner) =3D FP_STATE_FREE; =20 - vcpu_clear_flag(vcpu, HOST_SVE_ENABLED); - if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) - vcpu_set_flag(vcpu, HOST_SVE_ENABLED); - if (system_supports_sme()) { vcpu_clear_flag(vcpu, HOST_SME_ENABLED); if (read_sysreg(cpacr_el1) & CPACR_EL1_SMEN_EL0EN) @@ -202,18 +198,6 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) * when needed. */ fpsimd_save_and_flush_cpu_state(); - } else if (has_vhe() && system_supports_sve()) { - /* - * The FPSIMD/SVE state in the CPU has not been touched, and we - * have SVE (and VHE): CPACR_EL1 (alias CPTR_EL2) has been - * reset by kvm_reset_cptr_el2() in the Hyp code, disabling SVE - * for EL0. To avoid spurious traps, restore the trap state - * seen by kvm_arch_vcpu_load_fp(): - */ - if (vcpu_get_flag(vcpu, HOST_SVE_ENABLED)) - sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_ZEN_EL0EN); - else - sysreg_clear_set(CPACR_EL1, CPACR_EL1_ZEN_EL0EN, 0); } =20 local_irq_restore(flags); --=20 2.39.5 From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 345765103F; Fri, 14 Mar 2025 00:35:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912549; cv=none; b=WfDI3qpufDxyadUsH+OmtllX/ynqcMjTH2LA4kMsOtyc1qK7CCbZAmmFRtjtXg2yhnFtUwHn9XWPYUY7f7F9HYgDBxG6qu2EAoCkOOqCKOPLN3Mcw4okhsqvLuAVm/z7sDjVe5PSYf9Nskxghg5oGjg2MFtBquO2ohlZ/s7vCW0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912549; c=relaxed/simple; bh=HIGP6VBVWear5iflnuFs+CJgbgmTu67qH3gNvcXNftM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=FwKlq1k+YUrAjF2UawUD36MqL4Ui0OFhhqutShz6jIXOxXk6Vo5w0kPvyLAgef2hnXutCrtp9X/fuGf7pUPNnKiZeAPKsk9Hn/U+4a+6oOr0gOCiUnMZgFgsdm4v0/X+zxgKFKM7tF3DplUvfqX7yl/dvvN8AQfhKZtHCd/gVo0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AGjYGP2Y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AGjYGP2Y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 073F9C4CEEA; Fri, 14 Mar 2025 00:35:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912548; bh=HIGP6VBVWear5iflnuFs+CJgbgmTu67qH3gNvcXNftM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=AGjYGP2YJS1aamsClgwJVp8lf/kvGD7uAoY25yBI1REbemyE74WLcQq+g06M468KS aUnhTHifE1olwyulIXOfBegwsLtZU3L+PD+pKyAwyf2vmFxif5nqvz5i3pCeUR9361 vA3QrC0POnnxP8GMCB7ZibojebO9BYXEeD6zPS8ONfPixf1JLhC0OrmDBggVSzbJJf YxUU9Zw2ARStBU/VEzBgs/NxsItnxNFSSQAViGOr/Bd+73QS9eNS51zzODz8yS35Bn WOz/59OHaZRjIR4iZcrtLw2URol/L2smOLmeCBst5gk5bgvV4GVQLtddWLcqnQFPYP MsZ2DhXlTQKRw== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:17 +0000 Subject: [PATCH 6.12 5/8] KVM: arm64: Remove VHE host restore of CPACR_EL1.SMEN Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-5-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=4631; i=broonie@kernel.org; h=from:subject:message-id; bh=m4KZU2ZKwzrjocVTNsB04VCPKKuL2LaKvOlEuHx81YI=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nMSFXYqxcla2yOd8Q6on5+uQX/712+j4Vs6YuY SKd3+2qJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5zAAKCRAk1otyXVSH0NwpB/ 9R+P4tBBfHyjajsFRWu7iejmODR6/43awUkFAy7YdjeXS6OC7DNo1d0jkkT+kNF+Ep0vF1ThBl1aXm kWW2QxCv7ixy2vRkXOFjUkoZBzyK28avR1VQ0o2+CL5E/PV4dqncMKypK7Bc22NYh5c4npcXhLgiS+ FZjjcjQoGIq+I9r+dSQPA0LLOtCtB/Wqq9uUY5RT76bydSYgs5aRDsTU7uK0QXBcRQsW8QbDTRBJjU PdSw4sVNx6DU0sOGN7/xtoN9vOT7S8TUZS1zDI6+sOy+qEGCBBJ5XgI9cLevDs1ltX5awZO05gRQhu 2KA92X3cjMs8+lCWAguTV7tORpNOCx X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Mark Rutland [ Upstream commit 407a99c4654e8ea65393f412c421a55cac539f5b ] When KVM is in VHE mode, the host kernel tries to save and restore the configuration of CPACR_EL1.SMEN (i.e. CPTR_EL2.SMEN when HCR_EL2.E2H=3D1) across kvm_arch_vcpu_load_fp() and kvm_arch_vcpu_put_fp(), since the configuration may be clobbered by hyp when running a vCPU. This logic has historically been broken, and is currently redundant. This logic was originally introduced in commit: 861262ab86270206 ("KVM: arm64: Handle SME host state when running guests") At the time, the VHE hyp code would reset CPTR_EL2.SMEN to 0b00 when returning to the host, trapping host access to SME state. Unfortunately, this was unsafe as the host could take a softirq before calling kvm_arch_vcpu_put_fp(), and if a softirq handler were to use kernel mode NEON the resulting attempt to save the live FPSIMD/SVE/SME state would result in a fatal trap. That issue was limited to VHE mode. For nVHE/hVHE modes, KVM always saved/restored the host kernel's CPACR_EL1 value, and configured CPTR_EL2.TSM to 0b0, ensuring that host usage of SME would not be trapped. The issue above was incidentally fixed by commit: 375110ab51dec5dc ("KVM: arm64: Fix resetting SME trap values on reset for= (h)VHE") That commit changed the VHE hyp code to configure CPTR_EL2.SMEN to 0b01 when returning to the host, permitting host kernel usage of SME, avoiding the issue described above. At the time, this was not identified as a fix for commit 861262ab86270206. Now that the host eagerly saves and unbinds its own FPSIMD/SVE/SME state, there's no need to save/restore the state of the EL0 SME trap. The kernel can safely save/restore state without trapping, as described above, and will restore userspace state (including trap controls) before returning to userspace. Remove the redundant logic. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-5-mark.rutland@arm.c= om Signed-off-by: Marc Zyngier [Update for rework of flags storage -- broonie] Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 2 -- arch/arm64/kvm/fpsimd.c | 21 --------------------- 2 files changed, 23 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 1a126fa657fcde3433f37f3aaf464a5b6a5f095d..122a1e12582c0584bce86c5416e= f7d150f90fe95 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -891,8 +891,6 @@ struct kvm_vcpu_arch { /* Save TRBE context if active */ #define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6)) =20 -/* SME enabled for EL0 */ -#define HOST_SME_ENABLED __vcpu_single_flag(sflags, BIT(1)) /* Physical CPU not in supported_cpus */ #define ON_UNSUPPORTED_CPU __vcpu_single_flag(sflags, BIT(2)) /* WFIT instruction trapped */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 4127abfd319c2c683d2281efa52a6abe5fac67ee..f64724197958e0d8a4ec17deb1f= 9826ce3625eb7 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -65,12 +65,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) fpsimd_save_and_flush_cpu_state(); *host_data_ptr(fp_owner) =3D FP_STATE_FREE; =20 - if (system_supports_sme()) { - vcpu_clear_flag(vcpu, HOST_SME_ENABLED); - if (read_sysreg(cpacr_el1) & CPACR_EL1_SMEN_EL0EN) - vcpu_set_flag(vcpu, HOST_SME_ENABLED); - } - /* * If normal guests gain SME support, maintain this behavior for pKVM * guests, which don't support SME. @@ -141,21 +135,6 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) =20 local_irq_save(flags); =20 - /* - * If we have VHE then the Hyp code will reset CPACR_EL1 to - * the default value and we need to reenable SME. - */ - if (has_vhe() && system_supports_sme()) { - /* Also restore EL0 state seen on entry */ - if (vcpu_get_flag(vcpu, HOST_SME_ENABLED)) - sysreg_clear_set(CPACR_EL1, 0, CPACR_ELx_SMEN); - else - sysreg_clear_set(CPACR_EL1, - CPACR_EL1_SMEN_EL0EN, - CPACR_EL1_SMEN_EL1EN); - isb(); - } - if (guest_owns_fp_regs()) { if (vcpu_has_sve(vcpu)) { u64 zcr =3D read_sysreg_el1(SYS_ZCR); --=20 2.39.5 From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 457F311CAF; Fri, 14 Mar 2025 00:35:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912553; cv=none; b=HVmACDcP+q0/wmPq0NyBIwZUidXn9eGYyq8AmFOndfM0aih6Yy4z1kAfBXR3zAD601I0O+SBCzU6j5gYCQWAOpF9TmYAQpIIWUp0fC7GEb/I6N5jpLJTsKwKUdcAr1o4amGcWzOdXf543iAyVIIAjWq1R/8jyeNeoyCM1Q6Xvc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912553; c=relaxed/simple; bh=LyPrPybrI4jSmvJtjiQg8E4IzuKEZoZgME8fZSHDqU0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=g6vz1V2fYuPPXg6FM4j/EsjEvqU4wl/Yob0RfbNHlPW57pAbTg1UvkiX2yKYQvYIRNjZuqYSHE1Qu9cu9S+g+kt5D2zF/clRDr7LTvblcMq2N6IMWIHbR4DXlAie2fg2691GcOhrvz6X09j7ETgwBcHdPRQWy65jZGwMy7klwPQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SfRBqLXm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SfRBqLXm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1F38EC4CEEF; Fri, 14 Mar 2025 00:35:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912551; bh=LyPrPybrI4jSmvJtjiQg8E4IzuKEZoZgME8fZSHDqU0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=SfRBqLXmd+ayJOmsPMOC2DwrQn/1rwxsh1dw7TqLQRugMPhmc1Z76zsb9btoKOaGV P62k67oeCDe5CkJLQGvCPu6q1aDYp76+kWuZ54at35Yucx9PGVrvXSLUWHo7M1Q4nQ AJI+2GECvQdEMVoElVHfxOnM30H5raC0Domak3nCF0whxikLvjPvxHw0GzTFajpE7v lXqNVBrPeeAUKOm8JXn3TuvKQf7mMPlQbfAKNYxUVwlDZRUtqGEiDTd7yw9N7DgzO7 I7V1Zcc7Envtkn7YeRhrF0DeJHOZ+c1mWWQmuVUHMnzaqXzgDSZwgUCt31VGA18TI6 +2gnZElOszgNw== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:18 +0000 Subject: [PATCH 6.12 6/8] KVM: arm64: Refactor exit handlers Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-6-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=7922; i=broonie@kernel.org; h=from:subject:message-id; bh=U+HkEfycBrVjaQM2Wq7Y3xZJglZuSCZHRDWKQggF5Cs=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nNKcnPRoqwkcK59RQjlwrI1G4QQ59PBKmiLflj r7s9noOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5zQAKCRAk1otyXVSH0N2eB/ wMoYmK+9hfcECLPb6uGHwKBeFKIQeHJ6fMcUoBtclyLyn8v88RkrZxZfJGWLy1qoTE23OJnkhMLQfv WC23yf2Gx+O9wjzHN7d1XAUuKEPBm6Rf1zwTFD8jnzVwqk+/On/OrjYAwW4yx5QgsjkneCauhwEySa 7jd8eFpGRFzjGQWWFwz66fn9cbYiCfqtl9mklRM21NvrBylItKSB9+LHPfcQfkgGmqEJFW8fWVWBNo 4vM3k8a5Ty1/sSIkXH/pPsfc+r4RoQjK0K5B5mzmJKaa5rwtXY0fySorxwSUMjtq0eakhZtrtj1/Ns K9e+/WhJh6LPPPsNZv/3DTSupRjoLM X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Mark Rutland [ Upstream commit 9b66195063c5a145843547b1d692bd189be85287 ] The hyp exit handling logic is largely shared between VHE and nVHE/hVHE, with common logic in arch/arm64/kvm/hyp/include/hyp/switch.h. The code in the header depends on function definitions provided by arch/arm64/kvm/hyp/vhe/switch.c and arch/arm64/kvm/hyp/nvhe/switch.c when they include the header. This is an unusual header dependency, and prevents the use of arch/arm64/kvm/hyp/include/hyp/switch.h in other files as this would result in compiler warnings regarding missing definitions, e.g. | In file included from arch/arm64/kvm/hyp/nvhe/hyp-main.c:8: | ./arch/arm64/kvm/hyp/include/hyp/switch.h:733:31: warning: 'kvm_get_exit_= handler_array' used but never defined | 733 | static const exit_handler_fn *kvm_get_exit_handler_array(struct k= vm_vcpu *vcpu); | | ^~~~~~~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:735:13: warning: 'early_exit_fi= lter' used but never defined | 735 | static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_co= de); | | ^~~~~~~~~~~~~~~~~ Refactor the logic such that the header doesn't depend on anything from the C files. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-7-mark.rutland@arm.c= om Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/kvm/hyp/include/hyp/switch.h | 30 ++++++------------------------ arch/arm64/kvm/hyp/nvhe/switch.c | 28 ++++++++++++++++------------ arch/arm64/kvm/hyp/vhe/switch.c | 9 ++++----- 3 files changed, 26 insertions(+), 41 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/i= nclude/hyp/switch.h index a7f6a653f33718d1a25e232608e63ea287f2a672..e14aba19847f2c66c202a869b51= 73f25a9a7f66e 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -635,23 +635,16 @@ static bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *= vcpu, u64 *exit_code) =20 typedef bool (*exit_handler_fn)(struct kvm_vcpu *, u64 *); =20 -static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *= vcpu); - -static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code); - /* * Allow the hypervisor to handle the exit with an exit handler if it has = one. * * Returns true if the hypervisor handled the exit, and control should go = back * to the guest, or false if it hasn't. */ -static inline bool kvm_hyp_handle_exit(struct kvm_vcpu *vcpu, u64 *exit_co= de) +static inline bool kvm_hyp_handle_exit(struct kvm_vcpu *vcpu, u64 *exit_co= de, + const exit_handler_fn *handlers) { - const exit_handler_fn *handlers =3D kvm_get_exit_handler_array(vcpu); - exit_handler_fn fn; - - fn =3D handlers[kvm_vcpu_trap_get_class(vcpu)]; - + exit_handler_fn fn =3D handlers[kvm_vcpu_trap_get_class(vcpu)]; if (fn) return fn(vcpu, exit_code); =20 @@ -681,20 +674,9 @@ static inline void synchronize_vcpu_pstate(struct kvm_= vcpu *vcpu, u64 *exit_code * the guest, false when we should restore the host state and return to the * main run loop. */ -static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool __fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_cod= e, + const exit_handler_fn *handlers) { - /* - * Save PSTATE early so that we can evaluate the vcpu mode - * early on. - */ - synchronize_vcpu_pstate(vcpu, exit_code); - - /* - * Check whether we want to repaint the state one way or - * another. - */ - early_exit_filter(vcpu, exit_code); - if (ARM_EXCEPTION_CODE(*exit_code) !=3D ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 =3D read_sysreg_el2(SYS_ESR); =20 @@ -724,7 +706,7 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vc= pu, u64 *exit_code) goto exit; =20 /* Check if there's an exit handler and allow it to handle the exit. */ - if (kvm_hyp_handle_exit(vcpu, exit_code)) + if (kvm_hyp_handle_exit(vcpu, exit_code, handlers)) goto guest; exit: /* Return to the host kernel and handle the exit */ diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/swi= tch.c index 3ce16f90fe6af7be21bc7b84a9d8b3905b8b08a7..ee74006c47bc44ca1d9bdf1ce7d= 4d8a41cf8e494 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -224,19 +224,21 @@ static const exit_handler_fn *kvm_get_exit_handler_ar= ray(struct kvm_vcpu *vcpu) return hyp_exit_handlers; } =20 -/* - * Some guests (e.g., protected VMs) are not be allowed to run in AArch32. - * The ARMv8 architecture does not give the hypervisor a mechanism to prev= ent a - * guest from dropping to AArch32 EL0 if implemented by the CPU. If the - * hypervisor spots a guest in such a state ensure it is handled, and don't - * trust the host to spot or fix it. The check below is based on the one = in - * kvm_arch_vcpu_ioctl_run(). - * - * Returns false if the guest ran in AArch32 when it shouldn't have, and - * thus should exit to the host, or true if a the guest run loop can conti= nue. - */ -static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) { + const exit_handler_fn *handlers =3D kvm_get_exit_handler_array(vcpu); + + synchronize_vcpu_pstate(vcpu, exit_code); + + /* + * Some guests (e.g., protected VMs) are not be allowed to run in + * AArch32. The ARMv8 architecture does not give the hypervisor a + * mechanism to prevent a guest from dropping to AArch32 EL0 if + * implemented by the CPU. If the hypervisor spots a guest in such a + * state ensure it is handled, and don't trust the host to spot or fix + * it. The check below is based on the one in + * kvm_arch_vcpu_ioctl_run(). + */ if (unlikely(vcpu_is_protected(vcpu) && vcpu_mode_is_32bit(vcpu))) { /* * As we have caught the guest red-handed, decide that it isn't @@ -249,6 +251,8 @@ static void early_exit_filter(struct kvm_vcpu *vcpu, u6= 4 *exit_code) *exit_code &=3D BIT(ARM_EXIT_WITH_SERROR_BIT); *exit_code |=3D ARM_EXCEPTION_IL; } + + return __fixup_guest_exit(vcpu, exit_code, handlers); } =20 /* Switch to the guest for legacy non-VHE systems */ diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switc= h.c index e7ca0424107adec2371ae4553ebab9857c60b6d9..46c1f5caf007331cdbbc806a184= e9b4721042fc0 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -423,13 +423,10 @@ static const exit_handler_fn hyp_exit_handlers[] =3D { [ESR_ELx_EC_MOPS] =3D kvm_hyp_handle_mops, }; =20 -static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *= vcpu) +static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) { - return hyp_exit_handlers; -} + synchronize_vcpu_pstate(vcpu, exit_code); =20 -static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) -{ /* * If we were in HYP context on entry, adjust the PSTATE view * so that the usual helpers work correctly. @@ -449,6 +446,8 @@ static void early_exit_filter(struct kvm_vcpu *vcpu, u6= 4 *exit_code) *vcpu_cpsr(vcpu) &=3D ~(PSR_MODE_MASK | PSR_MODE32_BIT); *vcpu_cpsr(vcpu) |=3D mode; } + + return __fixup_guest_exit(vcpu, exit_code, hyp_exit_handlers); } =20 /* Switch to the guest for VHE systems running in EL2 */ --=20 2.39.5 From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63EF976C61; Fri, 14 Mar 2025 00:35:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912556; cv=none; b=dz76TrscTaS35tgFWhrF2374gplIkY2osI7z7yPO+u4uu89ZISKM4jAPcvQD6shjJL4AWvJpDUZ4IeX5dP7zVigJWPBmjUESUAu3LrJZdwDJ5PMo5mdIpo2j8V4cvu0EGVtlBOGg0yqrE9LKffD5PSTPyqP3R8TyZuJ2SAtiVZ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912556; c=relaxed/simple; bh=VjQZVv44ASrO3ZXhREDR/fEJL+Pwl0VPy4knzDhlfCg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YscawvpkA4KCcvr5Kr4+6fqg3COLwsYgq8Cyf5DmcKeZDqmRyMeKJS7SvEnrFlrRR8BL2ylHTTHCxcL3UPv4BCQL09uC23gA4JpeqDINsGAvgNOZrCDV/gdJSIUvExZYo1TUUFCQhZ9UgzQ2QaVSoFafsZXeTCqpMIe8K+tZ4FE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mPtN4LCE; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mPtN4LCE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3534EC4CEE9; Fri, 14 Mar 2025 00:35:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912554; bh=VjQZVv44ASrO3ZXhREDR/fEJL+Pwl0VPy4knzDhlfCg=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=mPtN4LCE7RjGcN00qwhurTkRedKOHuBq9LPx+1mDrKBdNXgFurXwLpkJyRc6X0KYO r67VynmPp7Il5hOY42dU18QaI2gSdZk/82daZCK9TKcPKsfv5mLTpP6Dc+3M3nFfMD iMrv8Ubtagwly6QsKOHEeN/+r4hYUks17BM6y81G4PaH+Duwo6/xX1iyzt3fZPPSFH 0Rs2ZcpDh/hZHWi6GA/p3KejRWEY0RHnly3h111+BUf2QLKPBvSyobmlg3gqrGzw00 exwAgh7zDU/4BRjF7LxgoYxXwwHe+YO/0r8TtT9RDRHtgOvL2ZgI017lGIfKFfdBhP Y2dUy8Q/Cu0BA== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:19 +0000 Subject: [PATCH 6.12 7/8] KVM: arm64: Mark some header functions as inline Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-7-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=5610; i=broonie@kernel.org; h=from:subject:message-id; bh=EGF48m3PjYZWKTuTx1rrPzLs988zk10oZwRvlkxkZxY=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nOhb6KZjuO8bjSTFyl3sVIPNYSQ1gEYpqzTcqe zvXKdoaJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5zgAKCRAk1otyXVSH0IESB/ wJRy4SKaMaILHIk54GVo5HG58eIn73yXdmuSKnMosnqKeJXjphugzXKB5bX9jj52kYEx5qyhhd6s3i 7fJUEp3VcqjC2ZPvY5Caj1ASSctnNCMfX1Fp6FXtKu0HTCrjFat489AU1jgCUOjABixDtjTM5wSZVU +f+CMTjohRwSds95Xy6PGI//Mq3Omb0vT8+W7LybJa5laCWOf1ZIhPuumUGGJDrZpmKEDrvWbpQMw2 n9SlvptulO8KmRdVxYpbExOCMpnhec/E3Bk1GMXMA8+1ei/YLEP2RRMOFi0n9krW5Swd9W89rLTjmu 7M99SLVgqruOWug0RlEZAPDRIId0te X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Mark Rutland [ Upstream commit f9dd00de1e53a47763dfad601635d18542c3836d ] The shared hyp switch header has a number of static functions which might not be used by all files that include the header, and when unused they will provoke compiler warnings, e.g. | In file included from arch/arm64/kvm/hyp/nvhe/hyp-main.c:8: | ./arch/arm64/kvm/hyp/include/hyp/switch.h:703:13: warning: 'kvm_hyp_handl= e_dabt_low' defined but not used [-Wunused-function] | 703 | static bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *vcpu, u64 *e= xit_code) | | ^~~~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:682:13: warning: 'kvm_hyp_handl= e_cp15_32' defined but not used [-Wunused-function] | 682 | static bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *ex= it_code) | | ^~~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:662:13: warning: 'kvm_hyp_handl= e_sysreg' defined but not used [-Wunused-function] | 662 | static bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exi= t_code) | | ^~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:458:13: warning: 'kvm_hyp_handl= e_fpsimd' defined but not used [-Wunused-function] | 458 | static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exi= t_code) | | ^~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:329:13: warning: 'kvm_hyp_handl= e_mops' defined but not used [-Wunused-function] | 329 | static bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_= code) | | ^~~~~~~~~~~~~~~~~~~ Mark these functions as 'inline' to suppress this warning. This shouldn't result in any functional change. At the same time, avoid the use of __alias() in the header and alias kvm_hyp_handle_iabt_low() and kvm_hyp_handle_watchpt_low() to kvm_hyp_handle_memory_fault() using CPP, matching the style in the rest of the kernel. For consistency, kvm_hyp_handle_memory_fault() is also marked as 'inline'. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-8-mark.rutland@arm.c= om Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/kvm/hyp/include/hyp/switch.h | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/i= nclude/hyp/switch.h index e14aba19847f2c66c202a869b5173f25a9a7f66e..c1ab31429a0e5fab97cd06c3b7b= 6e378170bd99d 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -295,7 +295,7 @@ static inline bool __populate_fault_info(struct kvm_vcp= u *vcpu) return __get_fault_info(vcpu->arch.fault.esr_el2, &vcpu->arch.fault); } =20 -static bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_co= de) { *vcpu_pc(vcpu) =3D read_sysreg_el2(SYS_ELR); arm64_mops_reset_regs(vcpu_gp_regs(vcpu), vcpu->arch.fault.esr_el2); @@ -373,7 +373,7 @@ static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *v= cpu) * If FP/SIMD is not implemented, handle the trap and inject an undefined * instruction exception to the guest. Similarly for trapped SVE accesses. */ -static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_= code) { bool sve_guest; u8 esr_ec; @@ -564,7 +564,7 @@ static bool handle_ampere1_tcr(struct kvm_vcpu *vcpu) return true; } =20 -static bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_= code) { if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) && handle_tx2_tvm(vcpu)) @@ -584,7 +584,7 @@ static bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu= , u64 *exit_code) return false; } =20 -static bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit= _code) { if (static_branch_unlikely(&vgic_v3_cpuif_trap) && __vgic_v3_perform_cpuif_access(vcpu) =3D=3D 1) @@ -593,19 +593,18 @@ static bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *v= cpu, u64 *exit_code) return false; } =20 -static bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu, u64 *exit_c= ode) +static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu, + u64 *exit_code) { if (!__populate_fault_info(vcpu)) return true; =20 return false; } -static bool kvm_hyp_handle_iabt_low(struct kvm_vcpu *vcpu, u64 *exit_code) - __alias(kvm_hyp_handle_memory_fault); -static bool kvm_hyp_handle_watchpt_low(struct kvm_vcpu *vcpu, u64 *exit_co= de) - __alias(kvm_hyp_handle_memory_fault); +#define kvm_hyp_handle_iabt_low kvm_hyp_handle_memory_fault +#define kvm_hyp_handle_watchpt_low kvm_hyp_handle_memory_fault =20 -static bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *vcpu, u64 *exi= t_code) { if (kvm_hyp_handle_memory_fault(vcpu, exit_code)) return true; --=20 2.39.5 From nobody Wed Dec 17 22:42:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70FAE12CDBE; Fri, 14 Mar 2025 00:35:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912558; cv=none; b=drLKddDh/uIkFz1fCcZ3z/yQVdTR0A4oyv6V6TfEbyvPl12DuWLIXRYgxtiSIoVjQTeFgue6RPO55N3daVJak9znVIo/LPjonPRS7iYlLa6sk8HEsL1zThWQuAeN9MaNFxZoPROEVgcbaMEYzp1C+EHumtDGOIDP27/ejIxUK2Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741912558; c=relaxed/simple; bh=KGmYoSEyFtE1X43cKq+IuBu34k82TT9LGPOwj81uHE8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=RQ7MIcSZHfYQC4LEV/wxvuNrylfCPAwwAmagt5+2HLUkITUz4qZOoD3y9cWDj7GSsu9QRs6PmGK/w5MAq7i1nG6mDkLReqQH2M2Tif3YmQlyxdjc/DbWbIMr148GtInfMpLpV+zlHds0S8pe7H2VjYTGR2nf44JEGrC4OPcdH8U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AHJ9opFv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AHJ9opFv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 495C7C4CEEA; Fri, 14 Mar 2025 00:35:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741912558; bh=KGmYoSEyFtE1X43cKq+IuBu34k82TT9LGPOwj81uHE8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=AHJ9opFv7nhkDasotWRA2jZXxoprNLoicMiTcc2Uz+1OolqNpJqn1Uum7SukM7MKJ Dr8CjV5XONOzzHpLozaEkJBIGZ+RMo/QJJucqPGYru+2MlAL2cssbLiddSwgIdMgG3 ZFgIv6NJLdZj7LlZYQN1yaxMqxPPJkcu/pmov19HFgJjCqyfanJtddS9ecP3kE+joc +7mufonNyDJvUao6sTdMNcFahD/J7OpQApfO+9c2ovELGU8JNd9PDWNNuK6h1xpie6 Ezlz1fYGGFRkeMnNyCOGjwuxYZpE6um2S+n2vuiXkimNf/EmT4KViYKV8fZFOQ3Qxs FX5zkhNxRt0yg== From: Mark Brown Date: Fri, 14 Mar 2025 00:35:20 +0000 Subject: [PATCH 6.12 8/8] KVM: arm64: Eagerly switch ZCR_EL{1,2} Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250314-stable-sve-6-12-v1-8-ddc16609d9ba@kernel.org> References: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> In-Reply-To: <20250314-stable-sve-6-12-v1-0-ddc16609d9ba@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , Joey Gouly , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=11659; i=broonie@kernel.org; h=from:subject:message-id; bh=uqer+80vIuANd2HcpkczmfLuKTrflD2d8sZerHUW0x4=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn03nP3geRwmIah19hoCVfdp21f5jafk+8T/RV09aR odQSbGqJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9N5zwAKCRAk1otyXVSH0E9TB/ wLRV0dlIUgeFkvrkis23DQ+lXgR+trytlad0wZY5Q2iZB19QDA2QngK9viCCRjo5jkgfNxJXyn7heH P3l9fIWeCzaz3JyfPew6cCB2nNY4TSiAiv4BKP1Pk01YMkSjCnkKjZ4uUSlFwFVvJ/rxS2oFRreYQ3 4GcYrRQJ3nCmHfXvFYff1FtZAVZplUBnJqXvBAwBiM0XnfGUA/G6ZcQXI3DWWS6kt4S7QbMe6lgzBk P6Hy3zJ5kp0Wnf7N3K3HdC/Aicz5KagtiMA1DvtZ7+davqJOdMxIfPKUGfGYC+y3gU+Hx6pJuOWXL5 7lBQVZ59ckpXSWRiQ4MkZzyNC8YFzC X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB From: Mark Rutland [ Upstream commit 59419f10045bc955d2229819c7cf7a8b0b9c5b59 ] In non-protected KVM modes, while the guest FPSIMD/SVE/SME state is live on= the CPU, the host's active SVE VL may differ from the guest's maximum SVE VL: * For VHE hosts, when a VM uses NV, ZCR_EL2 contains a value constrained by the guest hypervisor, which may be less than or equal to that guest's maximum VL. Note: in this case the value of ZCR_EL1 is immaterial due to E2H. * For nVHE/hVHE hosts, ZCR_EL1 contains a value written by the guest, which may be less than or greater than the guest's maximum VL. Note: in this case hyp code traps host SVE usage and lazily restores ZCR_EL2 to the host's maximum VL, which may be greater than the guest's maximum VL. This can be the case between exiting a guest and kvm_arch_vcpu_put_fp(). If a softirq is taken during this period and the softirq handler tries to use kernel-mode NEON, then the kernel will fail to save the guest's FPSIMD/SVE state, and will pend a SIGKILL for the current thread. This happens because kvm_arch_vcpu_ctxsync_fp() binds the guest's live FPSIMD/SVE state with the guest's maximum SVE VL, and fpsimd_save_user_state() verifies that the live SVE VL is as expected before attempting to save the register state: | if (WARN_ON(sve_get_vl() !=3D vl)) { | force_signal_inject(SIGKILL, SI_KERNEL, 0, 0); | return; | } Fix this and make this a bit easier to reason about by always eagerly switching ZCR_EL{1,2} at hyp during guest<->host transitions. With this happening, there's no need to trap host SVE usage, and the nVHE/nVHE __deactivate_cptr_traps() logic can be simplified to enable host access to all present FPSIMD/SVE/SME features. In protected nVHE/hVHE modes, the host's state is always saved/restored by hyp, and the guest's state is saved prior to exit to the host, so from the host's PoV the guest never has live FPSIMD/SVE/SME state, and the host's ZCR_EL1 is never clobbered by hyp. Fixes: 8c8010d69c132273 ("KVM: arm64: Save/restore SVE state for nVHE") Fixes: 2e3cf82063a00ea0 ("KVM: arm64: nv: Ensure correct VL is loaded befor= e saving SVE state") Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Cc: Will Deacon Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-9-mark.rutland@arm.c= om Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/kvm/fpsimd.c | 30 ----------------- arch/arm64/kvm/hyp/entry.S | 5 +++ arch/arm64/kvm/hyp/include/hyp/switch.h | 59 +++++++++++++++++++++++++++++= ++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 13 ++++---- arch/arm64/kvm/hyp/nvhe/switch.c | 33 +++++++++++++++--- arch/arm64/kvm/hyp/vhe/switch.c | 4 +++ 6 files changed, 103 insertions(+), 41 deletions(-) diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index f64724197958e0d8a4ec17deb1f9826ce3625eb7..3cbb999419af7bb31ce9cec2baa= fcad00491610a 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -136,36 +136,6 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) local_irq_save(flags); =20 if (guest_owns_fp_regs()) { - if (vcpu_has_sve(vcpu)) { - u64 zcr =3D read_sysreg_el1(SYS_ZCR); - - /* - * If the vCPU is in the hyp context then ZCR_EL1 is - * loaded with its vEL2 counterpart. - */ - __vcpu_sys_reg(vcpu, vcpu_sve_zcr_elx(vcpu)) =3D zcr; - - /* - * Restore the VL that was saved when bound to the CPU, - * which is the maximum VL for the guest. Because the - * layout of the data when saving the sve state depends - * on the VL, we need to use a consistent (i.e., the - * maximum) VL. - * Note that this means that at guest exit ZCR_EL1 is - * not necessarily the same as on guest entry. - * - * ZCR_EL2 holds the guest hypervisor's VL when running - * a nested guest, which could be smaller than the - * max for the vCPU. Similar to above, we first need to - * switch to a VL consistent with the layout of the - * vCPU's SVE state. KVM support for NV implies VHE, so - * using the ZCR_EL1 alias is safe. - */ - if (!has_vhe() || (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu))) - sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, - SYS_ZCR_EL1); - } - /* * Flush (save and invalidate) the fpsimd/sve state so that if * the host tries to use fpsimd/sve, it's not using stale data diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index 4433a234aa9ba242f43b943d22011b5ddacd8af7..9f4e8d68ab505cf4a7aa8673643= d9b47ca1bc7cb 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -44,6 +44,11 @@ alternative_if ARM64_HAS_RAS_EXTN alternative_else_nop_endif mrs x1, isr_el1 cbz x1, 1f + + // Ensure that __guest_enter() always provides a context + // synchronization event so that callers don't need ISBs for anything + // that would usually be synchonized by the ERET. + isb mov x0, #ARM_EXCEPTION_IRQ ret =20 diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/i= nclude/hyp/switch.h index c1ab31429a0e5fab97cd06c3b7b6e378170bd99d..cc9cb63959463a7ddb35d868b17= e485b1223d507 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -344,6 +344,65 @@ static inline void __hyp_sve_save_host(void) true); } =20 +static inline void fpsimd_lazy_switch_to_guest(struct kvm_vcpu *vcpu) +{ + u64 zcr_el1, zcr_el2; + + if (!guest_owns_fp_regs()) + return; + + if (vcpu_has_sve(vcpu)) { + /* A guest hypervisor may restrict the effective max VL. */ + if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) + zcr_el2 =3D __vcpu_sys_reg(vcpu, ZCR_EL2); + else + zcr_el2 =3D vcpu_sve_max_vq(vcpu) - 1; + + write_sysreg_el2(zcr_el2, SYS_ZCR); + + zcr_el1 =3D __vcpu_sys_reg(vcpu, vcpu_sve_zcr_elx(vcpu)); + write_sysreg_el1(zcr_el1, SYS_ZCR); + } +} + +static inline void fpsimd_lazy_switch_to_host(struct kvm_vcpu *vcpu) +{ + u64 zcr_el1, zcr_el2; + + if (!guest_owns_fp_regs()) + return; + + /* + * When the guest owns the FP regs, we know that guest+hyp traps for + * any FPSIMD/SVE/SME features exposed to the guest have been disabled + * by either fpsimd_lazy_switch_to_guest() or kvm_hyp_handle_fpsimd() + * prior to __guest_entry(). As __guest_entry() guarantees a context + * synchronization event, we don't need an ISB here to avoid taking + * traps for anything that was exposed to the guest. + */ + if (vcpu_has_sve(vcpu)) { + zcr_el1 =3D read_sysreg_el1(SYS_ZCR); + __vcpu_sys_reg(vcpu, vcpu_sve_zcr_elx(vcpu)) =3D zcr_el1; + + /* + * The guest's state is always saved using the guest's max VL. + * Ensure that the host has the guest's max VL active such that + * the host can save the guest's state lazily, but don't + * artificially restrict the host to the guest's max VL. + */ + if (has_vhe()) { + zcr_el2 =3D vcpu_sve_max_vq(vcpu) - 1; + write_sysreg_el2(zcr_el2, SYS_ZCR); + } else { + zcr_el2 =3D sve_vq_from_vl(kvm_host_sve_max_vl) - 1; + write_sysreg_el2(zcr_el2, SYS_ZCR); + + zcr_el1 =3D vcpu_sve_max_vq(vcpu) - 1; + write_sysreg_el1(zcr_el1, SYS_ZCR); + } + } +} + static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu) { /* diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/h= yp-main.c index 4e757a77322c9efc59cdff501745f7c80d452358..1c8e2ad32e8c396fc4b11d5fec2= e86728f2829d9 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -5,6 +5,7 @@ */ =20 #include +#include =20 #include #include @@ -176,8 +177,12 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_conte= xt *host_ctxt) sync_hyp_vcpu(hyp_vcpu); pkvm_put_hyp_vcpu(hyp_vcpu); } else { + struct kvm_vcpu *vcpu =3D kern_hyp_va(host_vcpu); + /* The host is fully trusted, run its vCPU directly. */ - ret =3D __kvm_vcpu_run(host_vcpu); + fpsimd_lazy_switch_to_guest(vcpu); + ret =3D __kvm_vcpu_run(vcpu); + fpsimd_lazy_switch_to_host(vcpu); } =20 out: @@ -486,12 +491,6 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) case ESR_ELx_EC_SMC64: handle_host_smc(host_ctxt); break; - case ESR_ELx_EC_SVE: - cpacr_clear_set(0, CPACR_ELx_ZEN); - isb(); - sve_cond_update_zcr_vq(sve_vq_from_vl(kvm_host_sve_max_vl) - 1, - SYS_ZCR_EL2); - break; case ESR_ELx_EC_IABT_LOW: case ESR_ELx_EC_DABT_LOW: handle_host_mem_abort(host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/swi= tch.c index ee74006c47bc44ca1d9bdf1ce7d4d8a41cf8e494..a1245fa838319544f3770a05a58= eeed5233f0324 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -40,6 +40,9 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) { u64 val =3D CPTR_EL2_TAM; /* Same bit irrespective of E2H */ =20 + if (!guest_owns_fp_regs()) + __activate_traps_fpsimd32(vcpu); + if (has_hvhe()) { val |=3D CPACR_ELx_TTA; =20 @@ -48,6 +51,8 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) if (vcpu_has_sve(vcpu)) val |=3D CPACR_ELx_ZEN; } + + write_sysreg(val, cpacr_el1); } else { val |=3D CPTR_EL2_TTA | CPTR_NVHE_EL2_RES1; =20 @@ -62,12 +67,32 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) =20 if (!guest_owns_fp_regs()) val |=3D CPTR_EL2_TFP; + + write_sysreg(val, cptr_el2); } +} =20 - if (!guest_owns_fp_regs()) - __activate_traps_fpsimd32(vcpu); +static void __deactivate_cptr_traps(struct kvm_vcpu *vcpu) +{ + if (has_hvhe()) { + u64 val =3D CPACR_ELx_FPEN; + + if (cpus_have_final_cap(ARM64_SVE)) + val |=3D CPACR_ELx_ZEN; + if (cpus_have_final_cap(ARM64_SME)) + val |=3D CPACR_ELx_SMEN; + + write_sysreg(val, cpacr_el1); + } else { + u64 val =3D CPTR_NVHE_EL2_RES1; + + if (!cpus_have_final_cap(ARM64_SVE)) + val |=3D CPTR_EL2_TZ; + if (!cpus_have_final_cap(ARM64_SME)) + val |=3D CPTR_EL2_TSM; =20 - kvm_write_cptr_el2(val); + write_sysreg(val, cptr_el2); + } } =20 static void __activate_traps(struct kvm_vcpu *vcpu) @@ -120,7 +145,7 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) =20 write_sysreg(this_cpu_ptr(&kvm_init_params)->hcr_el2, hcr_el2); =20 - kvm_reset_cptr_el2(vcpu); + __deactivate_cptr_traps(vcpu); write_sysreg(__kvm_hyp_host_vector, vbar_el2); } =20 diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switc= h.c index 46c1f5caf007331cdbbc806a184e9b4721042fc0..496abfd3646b9858e95e06a79ed= ec11eee3a5893 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -462,6 +462,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) =20 sysreg_save_host_state_vhe(host_ctxt); =20 + fpsimd_lazy_switch_to_guest(vcpu); + /* * Note that ARM erratum 1165522 requires us to configure both stage 1 * and stage 2 translation for the guest context before we clear @@ -486,6 +488,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) =20 __deactivate_traps(vcpu); =20 + fpsimd_lazy_switch_to_host(vcpu); + sysreg_restore_host_state_vhe(host_ctxt); =20 if (guest_owns_fp_regs()) --=20 2.39.5