From nobody Mon May 25 05:12:53 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E7BD480324 for ; Mon, 18 May 2026 16:16:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779120990; cv=none; b=G+COAbCM/Eqf4CqeMmC0sGwTEJMzlHLpQd2jIWoDUWkq+agrBmBQtZ0C/LWABDWPIa9CTYvbOCUwffmsA0PrkxhmZKVRBq0ZxhH3p8Q8vzsUZ0nZPP2JO5oZO1LlrCL43Ul/ROvy8Dxzp4HMQNEayK/B/CpMp/YtNILyOgNiIvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779120990; c=relaxed/simple; bh=SssOoUW54wc8vOthl2ehr2kPULsp8+Z3pVcIEOe0GWE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=kus2drUYczH2/EsgguaoDZRdXxT8+iH+UczJ73BnOXCA5Rnj1bjZteTM11uN9CUcQErvMaHPgRHZRZWds4G8jWDjAAzYRwVFv6O45pUlP1jTLOxSj1f/YHmdEp8sL2tzsqnlXtz5fzG8VZZm1SRfsAlSOBnCyRJIosNMOsYiO2c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=um+hIIR7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="um+hIIR7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6DAA0C2BCB8; Mon, 18 May 2026 16:16:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1779120985; bh=SssOoUW54wc8vOthl2ehr2kPULsp8+Z3pVcIEOe0GWE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=um+hIIR7DFjNbVeuaHD0C7DYm+U32EIf1DRzM2nsocTtgZbPRtvPH9/9ZbR6R1HiF zr8quv7XBJS4p1Xn+aJmr0pG/9TC7VwCD/lXx1OozYULIg6RjZHNe9TmoU/gY/3d5/ wHP9fG13rFtnu4u/Fodmr77CV6QAYntYn1s98k4Q+fAw/4Av5Kr5AUonFLIXPJKBG7 Dhy5oP03wbGNz/7lAcJumzrvg10lETWh0BaemtBGsS/rxAzwepAN+v46vavcIG91T+ QWSsmB09/Oyd5+YvaHYUM7zZQ9/9XjlPfU5oFeKFFmSAXd1jtecstUyiOQ/6BMX6Up VctvQRAu9BUfw== From: Mark Brown Date: Mon, 18 May 2026 17:11:46 +0100 Subject: [PATCH v9 1/2] arm64/fpsimd: Suppress SVE access traps when loading FPSIMD state Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260518-arm64-sve-trap-mitigation-v9-1-9cc8fcd002fd@kernel.org> References: <20260518-arm64-sve-trap-mitigation-v9-0-9cc8fcd002fd@kernel.org> In-Reply-To: <20260518-arm64-sve-trap-mitigation-v9-0-9cc8fcd002fd@kernel.org> To: Catalin Marinas , Will Deacon Cc: Mark Rutland , Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Mark Brown X-Mailer: b4 0.16-dev X-Developer-Signature: v=1; a=openpgp-sha256; l=8045; i=broonie@kernel.org; h=from:subject:message-id; bh=SssOoUW54wc8vOthl2ehr2kPULsp8+Z3pVcIEOe0GWE=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBqCztUc02Z4mOQEAQOf423LTN44N+9nSbRmqXOw V5JBDU56COJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCags7VAAKCRAk1otyXVSH 0KW+B/42Eih/9EOJANH7Msj+WpECBlDEVNETo5ZATX9m1nyDNUwaDuYmc5i7QfGnJlSS2hQLeN5 0SN7db4UronONntlB0NElL7i1HikWDmcN3XBdlHAK8/PkL7yiCUZSgVh7K8y9Jl6WLnDQXVN/hr cY9g7hdpTnd/ZXewP03CTxmUvPkQsXVlHP0betSFRZGBgAJ63FoGe13nvPHo1JTbOhPqNHecIPK kCWFhQF+uVGEVTTB275M5PrXc927oXfaK5iIodOt2cCi6ZbT9vh9/MxWaZGotjDDyJlurrvL3Kp ow9JY4ZUeKDmqvIN/7zoi9X6wdoHAhUr6tR2gGwNw6trpetp X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB When we are in a syscall we take the opportunity to discard the SVE state, saving only the FPSIMD subset of the register state. If we have to reload the floating point state from memory then we reenable SVE access traps, stopping tracking SVE until the task uses SVE again at which point it will take another SVE access trap. This means that for a task which is actively using SVE and also doing many blocking system calls will have the additional overhead of SVE access traps. The use of SVE for applications like memcpy() means that frequent SVE usage is common with modern distributions, even with tasks that do not obviously use floating point. I did some instrumentation which counted the number of SVE access traps and the number of times we loaded FPSIMD only register state for each task. Testing with Debian Bookworm this showed that during boot the overwhelming majority of tasks triggered another SVE access trap more than 50% of the time after loading FPSIMD only state with a substantial number near 100%, though some programs had a very small number of SVE accesses most likely from startup. There were few tasks in the range 5-45%, most tasks either used SVE frequently or used it only a tiny proportion of times. As expected older distributions which do not have the SVE performance work available showed no SVE usage in general applications. This indicates that there should be some benefit from reducing the number of SVE access traps for blocking system calls like we did for non blocking system calls in commit 8c845e273104 ("arm64/sve: Leave SVE enabled on syscall if we don't context switch"). Let's do this with a timeout, when we take a SVE access trap record a jiffies after which we'll reeanble SVE traps and then check this whenever we load a FPSIMD only floating point state from memory. If the time has passed then we reenable traps, otherwise we leave traps disabled and flush the non-shared register state like we would on trap. The timeout is currently set to a second, I pulled this number out of thin air so there is doubtless some room for tuning. This means that for a task which is actively using SVE the number of SVE access traps will be equivalent or reduced but applications which use SVE only very infrequently will avoid the overheads associated with tracking SVE state after a second. The extra cost from additional tracking of SVE state only occurs when a task is preempted so short running tasks should be minimally affected. As would be expected fp-pidbench shows minimal change from this patch, it does not block and on a quiet system is unlikely to see it's state reloaded from memory. There should be no functional change resulting from this, it is purely a performance optimisation. Signed-off-by: Mark Brown --- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/entry-fpsimd.S | 15 +++++++++++++ arch/arm64/kernel/fpsimd.c | 46 +++++++++++++++++++++++++++++++++-= ---- 4 files changed, 58 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsim= d.h index d9d00b45ab11..b3ba8803b1d3 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -144,6 +144,7 @@ static inline void *thread_zt_state(struct thread_struc= t *thread) extern void sve_save_state(void *state, u32 *pfpsr, int save_ffr); extern void sve_load_state(void const *state, u32 const *pfpsr, int restore_ffr); +extern void sve_flush_p(bool flush_ffr); extern void sve_flush_live(bool flush_ffr, unsigned long vq_minus_1); extern unsigned int sve_get_vl(void); extern void sve_set_vq(unsigned long vq_minus_1); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/pr= ocessor.h index e30c4c8e3a7a..a174864eca5f 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -166,6 +166,7 @@ struct thread_struct { unsigned int fpsimd_cpu; void *sve_state; /* SVE registers, if any */ void *sme_state; /* ZA and ZT state, if any */ + unsigned long sve_timeout; /* jiffies to drop TIF_SVE */ unsigned int vl[ARM64_VEC_MAX]; /* vector length */ unsigned int vl_onexec[ARM64_VEC_MAX]; /* vl after next exec */ unsigned long fault_address; /* fault info */ diff --git a/arch/arm64/kernel/entry-fpsimd.S b/arch/arm64/kernel/entry-fps= imd.S index 6325db1a2179..617dd70cafd7 100644 --- a/arch/arm64/kernel/entry-fpsimd.S +++ b/arch/arm64/kernel/entry-fpsimd.S @@ -85,6 +85,21 @@ SYM_FUNC_START(sve_flush_live) 2: ret SYM_FUNC_END(sve_flush_live) =20 +/* + * Zero the predicate registers + * + * VQ must already be configured by caller, any further updates of VQ + * will need to ensure that the register state remains valid. + * + * x0 =3D include FFR? + */ +SYM_FUNC_START(sve_flush_p) + sve_flush_p + tbz x0, #0, 1f + sve_flush_ffr +1: ret +SYM_FUNC_END(sve_flush_p) + #endif /* CONFIG_ARM64_SVE */ =20 #ifdef CONFIG_ARM64_SME diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 60a45d600b46..9c3def8ddea7 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -362,6 +362,7 @@ static void task_fpsimd_load(void) { bool restore_sve_regs =3D false; bool restore_ffr; + unsigned long sve_vq_minus_one; =20 WARN_ON(!system_supports_fpsimd()); WARN_ON(preemptible()); @@ -370,16 +371,11 @@ static void task_fpsimd_load(void) if (system_supports_sve() || system_supports_sme()) { switch (current->thread.fp_type) { case FP_STATE_FPSIMD: - /* Stop tracking SVE for this task until next use. */ - clear_thread_flag(TIF_SVE); break; case FP_STATE_SVE: if (!thread_sm_enabled(¤t->thread)) WARN_ON_ONCE(!test_and_set_thread_flag(TIF_SVE)); =20 - if (test_thread_flag(TIF_SVE)) - sve_set_vq(sve_vq_from_vl(task_get_sve_vl(current)) - 1); - restore_sve_regs =3D true; restore_ffr =3D true; break; @@ -398,6 +394,15 @@ static void task_fpsimd_load(void) } } =20 + /* + * If SVE has been enabled we may keep it enabled even if + * loading only FPSIMD state, so always set the VL. + */ + if (system_supports_sve() && test_thread_flag(TIF_SVE)) { + sve_vq_minus_one =3D sve_vq_from_vl(task_get_sve_vl(current)) - 1; + sve_set_vq(sve_vq_minus_one); + } + /* Restore SME, override SVE register configuration if needed */ if (system_supports_sme()) { unsigned long sme_vl =3D task_get_sme_vl(current); @@ -427,6 +432,30 @@ static void task_fpsimd_load(void) } else { WARN_ON_ONCE(current->thread.fp_type !=3D FP_STATE_FPSIMD); fpsimd_load_state(¤t->thread.uw.fpsimd_state); + + /* + * If the task had been using SVE we keep it enabled + * when loading FPSIMD only state for a period to + * minimise overhead for tasks actively using SVE, + * disabling it periodicaly to ensure that tasks that + * use SVE intermittently do eventually avoid the + * overhead of carrying SVE state. The timeout is + * initialised when we take a SVE trap in do_sve_acc(). + */ + if (system_supports_sve() && test_thread_flag(TIF_SVE)) { + if (time_after(jiffies, current->thread.sve_timeout)) { + clear_thread_flag(TIF_SVE); + sve_user_disable(); + } else { + /* + * Loading V will have flushed the + * rest of the Z register, SVE is + * enabled at EL1 and VL was set + * above. + */ + sve_flush_p(true); + } + } } } =20 @@ -1345,6 +1374,13 @@ void do_sve_acc(unsigned long esr, struct pt_regs *r= egs) =20 get_cpu_fpsimd_context(); =20 + /* + * We will keep SVE enabled when loading FPSIMD only state for + * the next second to minimise traps when userspace is + * actively using SVE. + */ + current->thread.sve_timeout =3D jiffies + HZ; + if (test_and_set_thread_flag(TIF_SVE)) WARN_ON(1); /* SVE access shouldn't have trapped */ =20 --=20 2.47.3 From nobody Mon May 25 05:12:53 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1403E2D7DDD for ; Mon, 18 May 2026 16:16:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779120988; cv=none; b=aRcYTr4DtMuEMCzifCKYgokYR596KIFknWP5ZuEDz/PxRrJCvP7G+MNnAjALF7l9uc2KA8+T7Ujv9ux8N7QyvkytfmtL3290pn0HchSz4WVaSkVQ70jHtaNVgSm0gVU/j6jmZwFmoMLm2vA9HjaYXP873ZLajkBl8JRwX0rnXG8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779120988; c=relaxed/simple; bh=7GKFSYrY0L2am6cu4ZjHbOPqVJRK3ddPdovx7/54Msw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=anXXJYclqCvxNWS/Z5Uo8hu0g1QHTLPDx/aqf2B61rbw6ubQ1FduyRd9nbwYx8Q9GMSkWxojIFgXe03T9jYrAwmx5VFgIQ8eQtIXtGsjmf1FQOobirjeIezgYMKt+snHPcG87jkeIt5BheaHniub1M1CGRo/2xkN1hyazcmImA4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gCsPIF0r; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gCsPIF0r" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 22768C2BCB7; Mon, 18 May 2026 16:16:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1779120987; bh=7GKFSYrY0L2am6cu4ZjHbOPqVJRK3ddPdovx7/54Msw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=gCsPIF0rt8d90RybYIb3G0O82EKweISeB3WpzwsgKagzHbO1L21wjoTqNJD0zRWfn zz5TItA8+id2IwTrv9AyUrdJF6fbKU6UIuywwobhysVBN5juKPl7d9mD4HUmkAi8TZ lFQ1WgdYXbjG6HkRm3e1BX19s/2UikU5sSeyJeALKMQ2l6ZBkkqwV8rAgKbUhlkgum XvgM55w3c5RsMqCuisOuaRIj35ypeGMoVbWk1R9tVIApPNe1ASe35dusZzUC41cPMW HmGHckjWeg8tM67qiExtY2QsnWV8GqSOkEPEXH/tWcWRtC2Gb9aPfiULfdwPmZfiIx g0HA2IQ/aesMQ== From: Mark Brown Date: Mon, 18 May 2026 17:11:47 +0100 Subject: [PATCH v9 2/2] arm64/sve: Disable TIF_SVE on syscall once per second Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260518-arm64-sve-trap-mitigation-v9-2-9cc8fcd002fd@kernel.org> References: <20260518-arm64-sve-trap-mitigation-v9-0-9cc8fcd002fd@kernel.org> In-Reply-To: <20260518-arm64-sve-trap-mitigation-v9-0-9cc8fcd002fd@kernel.org> To: Catalin Marinas , Will Deacon Cc: Mark Rutland , Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Mark Brown X-Mailer: b4 0.16-dev X-Developer-Signature: v=1; a=openpgp-sha256; l=2697; i=broonie@kernel.org; h=from:subject:message-id; bh=7GKFSYrY0L2am6cu4ZjHbOPqVJRK3ddPdovx7/54Msw=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBqCztVf/8aqA8T7M7Ki+eyuqBCf0N3vet/co4xs pcLg01eH02JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCags7VQAKCRAk1otyXVSH 0KXYB/sGD8XpYQtPjxVqrX/DuIIwuKEoLweuGK2SF9rot5OYfnOSlCKmkgfSQHeWOzw+jTFwsCl 1eSb6PAT1NUaDt0QM+SyJv0TksjlzyW86JASH2OvEHMibjeKbKyVJ8KQZwYECIvt8uNHdhI+ICU b3wZucYU+MXynwBe6dtNaDKJu6c8977+3khydWTWgxS0G32g6du8jv9/Lv1b29afVbm2s3sZ5Dw AuNRGZyYq08jbYVNVbxy9SDMMvIwQ6njvvnlpNPznlK73npLWbLacmo4AWBFHEzwDcuwzX86r/8 SoP4OBYCOJAq8Jdsl75bnrsMWXCA54sA0su2fgKGNusSbWtx X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Our syscall ABI requires that when performing a syscall the portions of the Z registers not shared with the V registers, the P and FFR registers are reset to 0. Since we have no way of monitoring EL0 SVE usage this is implemented by changing the in register values on every syscall for tasks which have SVE enabled, for systems with 128 bit SVE vector lengths this has been benchmarked as a 6% overhead. We currently support disabling SVE for userspace tasks when loading the floating point state from memory during a syscall, allowing tasks that use SVE infrequently to avoid this overhead, but this may not help CPU bound tasks if they are not fortunate enough to block or be scheduled during a syscall. This is done whenever the state is loaded from a second after the last time the task generate a SVE access trap. Extend this mechanism to also apply during syscall entry, disabling SVE instead of flushing the live registers when we perform a syscall a second after the last time a SVE access trap was taken. This adds an additional memory access and branch for tasks using SVE and means that CPU bound tasks actively using SVE will take extra SVE access traps (at most one per second) but will allows CPU bound tasks that infrequently use SVE to avoid the overhead of flushing the registers on syscall. On a system with 128 bit SVE vectors fp-pidbench shows a roughly 4.5% improvement compared to baseline after having used SVE, for a roughly 0.4% overhead when SVE is used between each syscall. Obviously this is very much a microbenchmark. This is purely a performance optimisation, there should be no functional change. Signed-off-by: Mark Brown --- arch/arm64/kernel/entry-common.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-com= mon.c index cb54335465f6..e02e2c1013e3 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -240,8 +240,18 @@ static inline void fpsimd_syscall_enter(void) if (test_thread_flag(TIF_SVE)) { unsigned int sve_vq_minus_one; =20 - sve_vq_minus_one =3D sve_vq_from_vl(task_get_sve_vl(current)) - 1; - sve_flush_live(true, sve_vq_minus_one); + /* + * Ensure that tasks that don't block in a syscall + * also get a chance to drop TIF_SVE. + */ + if (unlikely(time_after(jiffies, + current->thread.sve_timeout))) { + clear_thread_flag(TIF_SVE); + sve_user_disable(); + } else { + sve_vq_minus_one =3D sve_vq_from_vl(task_get_sve_vl(current)) - 1; + sve_flush_live(true, sve_vq_minus_one); + } } =20 /* --=20 2.47.3