From nobody Sun Oct 5 00:10:00 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AD393296BCD for ; Mon, 11 Aug 2025 16:36:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930205; cv=none; b=G3Sdvz1mlcY8XmKEfvfgLXxcRctmTbsP6U9zfdDr44+Wr99hx/Kjiz6wQyRVubqyValYCVlkin2AofOa4zVSpB3ziwQJ1c6DxeI27W8zWWG0qE8KJ/95g+lctG6eS+tQdNJI+UZ3jhqX5KbW58pc/yv/E8xDpUtBGBmROt80roA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930205; c=relaxed/simple; bh=UX/K3IFZ5TmE/FSNCFO+7jjwwKr2XJe8v7ZpKdV8cn0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=I8bH1z6HP02wgECwhpJ9ANtwwJMQgtHgY1Fw0MYs71sHkNhFkyifSr7y5Pa02tqAYupu4jXA8y8eMsUQ+CjNsQYRF8AhIoYp9mU88OfmvpPsYt1tI3++7ezUkKkAAeO2XGLIjfbWHCygttZZkkdNDwVtcnE61UtviccnLZwbCjg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0ACF02680; Mon, 11 Aug 2025 09:36:35 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C1BCD3F738; Mon, 11 Aug 2025 09:36:40 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, maz@kernel.org, oliver.upton@linux.dev, shameerali.kolothum.thodi@huawei.com, joey.gouly@arm.com, james.morse@arm.com, ardb@kernel.org, scott@os.amperecomputing.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, mark.rutland@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v6 1/5] arm64: cpufeature: add FEAT_LSUI Date: Mon, 11 Aug 2025 17:36:31 +0100 Message-Id: <20250811163635.1562145-2-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250811163635.1562145-1-yeoreum.yun@arm.com> References: <20250811163635.1562145-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since Armv9.6, FEAT_LSUI supplies load/store instructions for privileged level to access user memory without clearing PSTATE.PAN bit. Add LSUI feature so that the unprevilieged load/store instructions could be used when kernel accesses user memory without clearing PSTATE.PAN = bit. Signed-off-by: Yeoreum Yun --- arch/arm64/kernel/cpufeature.c | 8 ++++++++ arch/arm64/tools/cpucaps | 1 + 2 files changed, 9 insertions(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9ad065f15f1d..fd8ec291adab 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -278,6 +278,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar2[] = =3D { =20 static const struct arm64_ftr_bits ftr_id_aa64isar3[] =3D { ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_E= L1_FPRCVT_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_EL= 1_LSUI_SHIFT, 4, ID_AA64ISAR3_EL1_LSUI_NI), ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_E= L1_FAMINMAX_SHIFT, 4, 0), ARM64_FTR_END, }; @@ -3131,6 +3132,13 @@ static const struct arm64_cpu_capabilities arm64_fea= tures[] =3D { .matches =3D has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64PFR2_EL1, GCIE, IMP) }, + { + .desc =3D "Unprivileged Load Store Instructions (LSUI)", + .capability =3D ARM64_HAS_LSUI, + .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, + .matches =3D has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP) + }, {}, }; =20 diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index ef0b7946f5a4..73f8e5211cd2 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -44,6 +44,7 @@ HAS_HCX HAS_LDAPR HAS_LPA2 HAS_LSE_ATOMICS +HAS_LSUI HAS_MOPS HAS_NESTED_VIRT HAS_BBML2_NOABORT --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Sun Oct 5 00:10:00 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 66AD729BD97 for ; Mon, 11 Aug 2025 16:36:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930207; cv=none; b=sWNyls6n61qClnufZYShT7zXBlGCsCsC60CgGHxBqOA5/BD+XxSU8CNJ2HY2NzGxXOHJ+3bJixwRTaQio0DoGxW98G7B0IrCgP5gL7i8efzxIkaMrGdSUuGuQLQR+fhqSApciwr00B7UhJxLn821YP2dWWd6VpkCYLc3St+Lb/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930207; c=relaxed/simple; bh=c/AlTkiLUhEVwzILpmNJrARDzCbA4lU0/yhG+Mjco4o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eZpo8PLNxt44MvEgEdcNlRGELoKpztO39vHf/iOzChojucCmFcIYWdkzQbOjiMkSYmR2crKOSzBujWnmNDg3ZbAtFiRgpgf4uEDQpAglxG07caSzWlZ4tRsmsARQJMzznqiJjaHv2CEqZxYCVKWcxhqGAAZjaqhRGpm7e9dfJR0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BA6FE2696; Mon, 11 Aug 2025 09:36:37 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 72DEB3F738; Mon, 11 Aug 2025 09:36:43 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, maz@kernel.org, oliver.upton@linux.dev, shameerali.kolothum.thodi@huawei.com, joey.gouly@arm.com, james.morse@arm.com, ardb@kernel.org, scott@os.amperecomputing.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, mark.rutland@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v6 2/5] KVM: arm64: expose FEAT_LSUI to guest Date: Mon, 11 Aug 2025 17:36:32 +0100 Message-Id: <20250811163635.1562145-3-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250811163635.1562145-1-yeoreum.yun@arm.com> References: <20250811163635.1562145-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" expose FEAT_LSUI to guest. Signed-off-by: Yeoreum Yun Acked-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 82ffb3b3b3cf..fb6c154aa37d 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1642,7 +1642,8 @@ static u64 __kvm_read_sanitised_id_reg(const struct k= vm_vcpu *vcpu, val &=3D ~ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_WFxT); break; case SYS_ID_AA64ISAR3_EL1: - val &=3D ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_FAMINMAX; + val &=3D ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_FAMINMAX | + ID_AA64ISAR3_EL1_LSUI; break; case SYS_ID_AA64MMFR2_EL1: val &=3D ~ID_AA64MMFR2_EL1_CCIDX_MASK; @@ -2991,7 +2992,7 @@ static const struct sys_reg_desc sys_reg_descs[] =3D { ID_AA64ISAR2_EL1_APA3 | ID_AA64ISAR2_EL1_GPA3)), ID_WRITABLE(ID_AA64ISAR3_EL1, (ID_AA64ISAR3_EL1_FPRCVT | - ID_AA64ISAR3_EL1_FAMINMAX)), + ID_AA64ISAR3_EL1_FAMINMAX | ID_AA64ISAR3_EL1_LSUI)), ID_UNALLOCATED(6,4), ID_UNALLOCATED(6,5), ID_UNALLOCATED(6,6), --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Sun Oct 5 00:10:00 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3B0BD29BDB6 for ; Mon, 11 Aug 2025 16:36:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930210; cv=none; b=LfLT2OE69PcfQpywyHPUIiNBwY6t8WbyalmHMjnXrTgQFOI42sr+eczaMrhunWkojfgFRB4OtLCAp1YLOJE2JlmKrcp1bMhsQkT16/Ae8lZXsCMAMAyUBTX2en+5ABrx2cmCLJdH6IE6VJfEHZuGmA+Jji2ZVkF1iBUweGC+itM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930210; c=relaxed/simple; bh=m5hJ5wCLuUXn1rIayTKhn+IlkMV3RyCtKcHb43Xii2o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ISNZ+UyeCQRs/5QvDMkor3MKRWa5oYlAppdvFh2vpDS88Jz/eee5lnWKrYAvvlajP7erOzCoS/SaBExspxzQYykHUuL5Z/uY7qW0eKLVvHn3gCo6UiH2fR5piedVbq+bv+p9M+l9dcd6yRHkHkqFN4euMsN1FQs4b8zGlEzEj1M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 751FC26A4; Mon, 11 Aug 2025 09:36:40 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2CFC13F738; Mon, 11 Aug 2025 09:36:46 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, maz@kernel.org, oliver.upton@linux.dev, shameerali.kolothum.thodi@huawei.com, joey.gouly@arm.com, james.morse@arm.com, ardb@kernel.org, scott@os.amperecomputing.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, mark.rutland@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v6 3/5] arm64: Kconfig: add LSUI Kconfig Date: Mon, 11 Aug 2025 17:36:33 +0100 Message-Id: <20250811163635.1562145-4-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250811163635.1562145-1-yeoreum.yun@arm.com> References: <20250811163635.1562145-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since Armv9.6, FEAT_LSUI supplies the load/store instructions for previleged level to access to access user memory without clearing PSTATE.PAN bit. It's enough to add CONFIG_AS_HAS_LSUI only because the code for LSUI uses individual `.arch_extension` entries. Signed-off-by: Yeoreum Yun --- arch/arm64/Kconfig | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index e9bbfacc35a6..c474de3dce02 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2239,6 +2239,11 @@ config ARM64_GCS =20 endmenu # "v9.4 architectural features" =20 +config AS_HAS_LSUI + def_bool $(as-instr,.arch_extension lsui) + help + Supported by LLVM 20 and later, not yet supported by GNU AS. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Sun Oct 5 00:10:00 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 99B9D29E111 for ; Mon, 11 Aug 2025 16:36:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930213; cv=none; b=MkosO4dFSUbQridSF87hXjh6d17M+MiP0u0AxGJXjyX5ekYVOSoCPnLKi+7xFsqaSQ96JvqI5/g4ZmHNN6m3+cvqH60CAkakYoWWkvuc3lwI/Cn5XakpF4grMNdhydx3nPfAYY3h7cllka0ENvLiU5sl+EMe9aNxg0GihFwG/UM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930213; c=relaxed/simple; bh=cWgygJDXMt6JwpKU8NgFPFGmx/+DwZkHXRta+fxpjwo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FbFUOAwntpEkEn+ZOAKsDNer09BfSnywUx9LLXJ5bhgW7P79nKgrOSgpx6qZqNoJc1mzjYAJ0pdMkinOAIOljVkWSWt1va7FK61+MTaWjDAAeYH/7isUQv9XPzRmciCsU5HyZvD5OPiTihuG/958SNTd2iddZJ4+mJZAipvlo4Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 262F3113E; Mon, 11 Aug 2025 09:36:43 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DC11D3F738; Mon, 11 Aug 2025 09:36:48 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, maz@kernel.org, oliver.upton@linux.dev, shameerali.kolothum.thodi@huawei.com, joey.gouly@arm.com, james.morse@arm.com, ardb@kernel.org, scott@os.amperecomputing.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, mark.rutland@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v6 4/5] arm64: futex: refactor futex atomic operation Date: Mon, 11 Aug 2025 17:36:34 +0100 Message-Id: <20250811163635.1562145-5-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250811163635.1562145-1-yeoreum.yun@arm.com> References: <20250811163635.1562145-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor futex atomic operations using ll/sc method with clearing PSTATE.PAN to prepare to apply FEAT_LSUI on them. Signed-off-by: Yeoreum Yun --- arch/arm64/include/asm/futex.h | 183 ++++++++++++++++++++++----------- 1 file changed, 124 insertions(+), 59 deletions(-) diff --git a/arch/arm64/include/asm/futex.h b/arch/arm64/include/asm/futex.h index bc06691d2062..fdec4f3f2b15 100644 --- a/arch/arm64/include/asm/futex.h +++ b/arch/arm64/include/asm/futex.h @@ -7,73 +7,164 @@ =20 #include #include +#include =20 #include =20 -#define FUTEX_MAX_LOOPS 128 /* What's the largest number you can think of?= */ +#define LLSC_MAX_LOOPS 128 /* What's the largest number you can think of? = */ =20 -#define __futex_atomic_op(insn, ret, oldval, uaddr, tmp, oparg) \ -do { \ - unsigned int loops =3D FUTEX_MAX_LOOPS; \ +#define LLSC_FUTEX_ATOMIC_OP(op, asm_op) \ +static __always_inline int \ +__llsc_futex_atomic_##op(int oparg, u32 __user *uaddr, int *oval) \ +{ \ + unsigned int loops =3D LLSC_MAX_LOOPS; \ + int ret, val, tmp; \ \ uaccess_enable_privileged(); \ - asm volatile( \ -" prfm pstl1strm, %2\n" \ -"1: ldxr %w1, %2\n" \ - insn "\n" \ -"2: stlxr %w0, %w3, %2\n" \ -" cbz %w0, 3f\n" \ -" sub %w4, %w4, %w0\n" \ -" cbnz %w4, 1b\n" \ -" mov %w0, %w6\n" \ -"3:\n" \ -" dmb ish\n" \ + asm volatile("// __llsc_futex_atomic_" #op "\n" \ + " prfm pstl1strm, %2\n" \ + "1: ldxr %w1, %2\n" \ + " " #asm_op " %w3, %w1, %w5\n" \ + "2: stlxr %w0, %w3, %2\n" \ + " cbz %w0, 3f\n" \ + " sub %w4, %w4, %w0\n" \ + " cbnz %w4, 1b\n" \ + " mov %w0, %w6\n" \ + "3:\n" \ + " dmb ish\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w0) \ - : "=3D&r" (ret), "=3D&r" (oldval), "+Q" (*uaddr), "=3D&r" (tmp), \ + : "=3D&r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), \ "+r" (loops) \ : "r" (oparg), "Ir" (-EAGAIN) \ : "memory"); \ uaccess_disable_privileged(); \ -} while (0) + \ + if (!ret) \ + *oval =3D val; \ + \ + return ret; \ +} + +LLSC_FUTEX_ATOMIC_OP(add, add) +LLSC_FUTEX_ATOMIC_OP(or, orr) +LLSC_FUTEX_ATOMIC_OP(and, and) +LLSC_FUTEX_ATOMIC_OP(eor, eor) + +static __always_inline int +__llsc_futex_atomic_set(int oparg, u32 __user *uaddr, int *oval) +{ + unsigned int loops =3D LLSC_MAX_LOOPS; + int ret, val; + + uaccess_enable_privileged(); + asm volatile("//__llsc_futex_xchg\n" + " prfm pstl1strm, %2\n" + "1: ldxr %w1, %2\n" + "2: stlxr %w0, %w4, %2\n" + " cbz %w3, 3f\n" + " sub %w3, %w3, %w0\n" + " cbnz %w3, 1b\n" + " mov %w0, %w5\n" + "3:\n" + " dmb ish\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w0) + : "=3D&r" (ret), "=3D&r" (val), "+Q" (*uaddr), "+r" (loops) + : "r" (oparg), "Ir" (-EAGAIN) + : "memory"); + uaccess_disable_privileged(); + + if (!ret) + *oval =3D val; + + return ret; +} + +static __always_inline int +__llsc_futex_cmpxchg(u32 __user *uaddr, u32 oldval, u32 newval, u32 *oval) +{ + int ret =3D 0; + unsigned int loops =3D LLSC_MAX_LOOPS; + u32 val, tmp; + + uaccess_enable_privileged(); + asm volatile("//__llsc_futex_cmpxchg\n" + " prfm pstl1strm, %2\n" + "1: ldxr %w1, %2\n" + " eor %w3, %w1, %w5\n" + " cbnz %w3, 4f\n" + "2: stlxr %w3, %w6, %2\n" + " cbz %w3, 3f\n" + " sub %w4, %w4, %w3\n" + " cbnz %w4, 1b\n" + " mov %w0, %w7\n" + "3:\n" + " dmb ish\n" + "4:\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 4b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 4b, %w0) + : "+r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), "+r" (loops) + : "r" (oldval), "r" (newval), "Ir" (-EAGAIN) + : "memory"); + uaccess_disable_privileged(); + + if (!ret) + *oval =3D val; + + return ret; +} + +#define FUTEX_ATOMIC_OP(op) \ +static __always_inline int \ +__futex_atomic_##op(int oparg, u32 __user *uaddr, int *oval) \ +{ \ + return __llsc_futex_atomic_##op(oparg, uaddr, oval); \ +} + +FUTEX_ATOMIC_OP(add) +FUTEX_ATOMIC_OP(or) +FUTEX_ATOMIC_OP(and) +FUTEX_ATOMIC_OP(eor) +FUTEX_ATOMIC_OP(set) + +static __always_inline int +__futex_cmpxchg(u32 __user *uaddr, u32 oldval, u32 newval, u32 *oval) +{ + return __llsc_futex_cmpxchg(uaddr, oldval, newval, oval); +} =20 static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *_uad= dr) { - int oldval =3D 0, ret, tmp; - u32 __user *uaddr =3D __uaccess_mask_ptr(_uaddr); + int ret; + u32 __user *uaddr; =20 if (!access_ok(_uaddr, sizeof(u32))) return -EFAULT; =20 + uaddr =3D __uaccess_mask_ptr(_uaddr); + switch (op) { case FUTEX_OP_SET: - __futex_atomic_op("mov %w3, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_set(oparg, uaddr, oval); break; case FUTEX_OP_ADD: - __futex_atomic_op("add %w3, %w1, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_add(oparg, uaddr, oval); break; case FUTEX_OP_OR: - __futex_atomic_op("orr %w3, %w1, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_or(oparg, uaddr, oval); break; case FUTEX_OP_ANDN: - __futex_atomic_op("and %w3, %w1, %w5", - ret, oldval, uaddr, tmp, ~oparg); + ret =3D __futex_atomic_and(~oparg, uaddr, oval); break; case FUTEX_OP_XOR: - __futex_atomic_op("eor %w3, %w1, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_eor(oparg, uaddr, oval); break; default: ret =3D -ENOSYS; } =20 - if (!ret) - *oval =3D oldval; - return ret; } =20 @@ -81,40 +172,14 @@ static inline int futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *_uaddr, u32 oldval, u32 newval) { - int ret =3D 0; - unsigned int loops =3D FUTEX_MAX_LOOPS; - u32 val, tmp; u32 __user *uaddr; =20 if (!access_ok(_uaddr, sizeof(u32))) return -EFAULT; =20 uaddr =3D __uaccess_mask_ptr(_uaddr); - uaccess_enable_privileged(); - asm volatile("// futex_atomic_cmpxchg_inatomic\n" -" prfm pstl1strm, %2\n" -"1: ldxr %w1, %2\n" -" sub %w3, %w1, %w5\n" -" cbnz %w3, 4f\n" -"2: stlxr %w3, %w6, %2\n" -" cbz %w3, 3f\n" -" sub %w4, %w4, %w3\n" -" cbnz %w4, 1b\n" -" mov %w0, %w7\n" -"3:\n" -" dmb ish\n" -"4:\n" - _ASM_EXTABLE_UACCESS_ERR(1b, 4b, %w0) - _ASM_EXTABLE_UACCESS_ERR(2b, 4b, %w0) - : "+r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), "+r" (loops) - : "r" (oldval), "r" (newval), "Ir" (-EAGAIN) - : "memory"); - uaccess_disable_privileged(); =20 - if (!ret) - *uval =3D val; - - return ret; + return __futex_cmpxchg(uaddr, oldval, newval, uval); } =20 #endif /* __ASM_FUTEX_H */ --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Sun Oct 5 00:10:00 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 42C9B2C21D4 for ; Mon, 11 Aug 2025 16:36:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930215; cv=none; b=SXsQs8XVZ4l/YKNF/BoD91WfchBaFKVAgK3JNSOkmHS77r6AVcM3GO/RG8OCT0Dx7V22WyEpSK5HHu4J2c+1dSXcW4gmRxwQc25L+dhbIB8bGDnggZmKZ/TOjvEZWS0kZ7aHj1kATZJL9sQKVo/hgI5s19oIl5wm6UlufYnQF3A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754930215; c=relaxed/simple; bh=R/uCGAFuKyk/yhRsJFbZFTF2BDn+0ZJdwuNVRlQb7rU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dAQceneb1+qTeaQCAySSPTVKdJ9VEjk6QVoyE/eMIKBXpdtLwVBcMhh29wHlk4arbkpp5hQmjvUPqmT0mz83n6QDcuT3onGOBIVx6woNiPNqIYfxQC7GQH2qInacZ3ML9LV9KSTUrh9YG91E7FRZE58DN3fUZApCiNd/ZTG1r1I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CA2282680; Mon, 11 Aug 2025 09:36:45 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8D7AF3F738; Mon, 11 Aug 2025 09:36:51 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, maz@kernel.org, oliver.upton@linux.dev, shameerali.kolothum.thodi@huawei.com, joey.gouly@arm.com, james.morse@arm.com, ardb@kernel.org, scott@os.amperecomputing.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, mark.rutland@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v6 5/5] arm64: futex: support futex with FEAT_LSUI Date: Mon, 11 Aug 2025 17:36:35 +0100 Message-Id: <20250811163635.1562145-6-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250811163635.1562145-1-yeoreum.yun@arm.com> References: <20250811163635.1562145-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current futex atomic operations are implemented with ll/sc instructions and clearing PSTATE.PAN. Since Armv9.6, FEAT_LSUI supplies not only load/store instructions but also atomic operation for user memory access in kernel it doesn't need to clear PSTATE.PAN bit anymore. With theses instructions some of futex atomic operations don't need to be implmented with ldxr/stlxr pair instead can be implmented with one atomic operation supplied by FEAT_LSUI. However, some of futex atomic operations still need to use ll/sc way via ldtxr/stltxr supplied by FEAT_LSUI since there is no correspondant atomic instruction or doesn't support word size operation. (i.e) eor, cas{mb}t But It's good to work without clearing PSTATE.PAN bit. Signed-off-by: Yeoreum Yun --- arch/arm64/include/asm/futex.h | 142 ++++++++++++++++++++++++++++++++- 1 file changed, 141 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/futex.h b/arch/arm64/include/asm/futex.h index fdec4f3f2b15..38fc98f4af46 100644 --- a/arch/arm64/include/asm/futex.h +++ b/arch/arm64/include/asm/futex.h @@ -9,6 +9,8 @@ #include #include =20 +#include +#include #include =20 #define LLSC_MAX_LOOPS 128 /* What's the largest number you can think of? = */ @@ -115,11 +117,149 @@ __llsc_futex_cmpxchg(u32 __user *uaddr, u32 oldval, = u32 newval, u32 *oval) return ret; } =20 +#ifdef CONFIG_AS_HAS_LSUI + +#define __LSUI_PREAMBLE ".arch_extension lsui\n" + +#define LSUI_FUTEX_ATOMIC_OP(op, asm_op, mb) \ +static __always_inline int \ +__lsui_futex_atomic_##op(int oparg, u32 __user *uaddr, int *oval) \ +{ \ + int ret =3D 0; \ + int val; \ + \ + mte_enable_tco(); \ + uaccess_ttbr0_enable(); \ + \ + asm volatile("// __lsui_futex_atomic_" #op "\n" \ + __LSUI_PREAMBLE \ + "1: " #asm_op #mb " %w3, %w2, %1\n" \ + "2:\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %w0) \ + : "+r" (ret), "+Q" (*uaddr), "=3Dr" (val) \ + : "r" (oparg) \ + : "memory"); \ + \ + mte_disable_tco(); \ + uaccess_ttbr0_disable(); \ + \ + if (!ret) \ + *oval =3D val; \ + \ + return ret; \ +} + +LSUI_FUTEX_ATOMIC_OP(add, ldtadd, al) +LSUI_FUTEX_ATOMIC_OP(or, ldtset, al) +LSUI_FUTEX_ATOMIC_OP(andnot, ldtclr, al) +LSUI_FUTEX_ATOMIC_OP(set, swpt, al) + +static __always_inline int +__lsui_futex_atomic_and(int oparg, u32 __user *uaddr, int *oval) +{ + return __lsui_futex_atomic_andnot(~oparg, uaddr, oval); +} + +static __always_inline int +__lsui_futex_atomic_eor(int oparg, u32 __user *uaddr, int *oval) +{ + unsigned int loops =3D LLSC_MAX_LOOPS; + int ret, val, tmp; + + mte_enable_tco(); + uaccess_ttbr0_enable(); + + /* + * there are no ldteor/stteor instructions... + */ + asm volatile("// __lsui_futex_atomic_eor\n" + __LSUI_PREAMBLE + " prfm pstl1strm, %2\n" + "1: ldtxr %w1, %2\n" + " eor %w3, %w1, %w5\n" + "2: stltxr %w0, %w3, %2\n" + " cbz %w0, 3f\n" + " sub %w4, %w4, %w0\n" + " cbnz %w4, 1b\n" + " mov %w0, %w6\n" + "3:\n" + " dmb ish\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w0) + : "=3D&r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), + "+r" (loops) + : "r" (oparg), "Ir" (-EAGAIN) + : "memory"); + + mte_disable_tco(); + uaccess_ttbr0_disable(); + + if (!ret) + *oval =3D val; + + return ret; +} + +static __always_inline int +__lsui_futex_cmpxchg(u32 __user *uaddr, u32 oldval, u32 newval, u32 *oval) +{ + int ret =3D 0; + unsigned int loops =3D LLSC_MAX_LOOPS; + u32 val, tmp; + + mte_enable_tco(); + uaccess_ttbr0_enable(); + + /* + * cas{al}t doesn't support word size... + */ + asm volatile("//__lsui_futex_cmpxchg\n" + __LSUI_PREAMBLE + " prfm pstl1strm, %2\n" + "1: ldtxr %w1, %2\n" + " eor %w3, %w1, %w5\n" + " cbnz %w3, 4f\n" + "2: stltxr %w3, %w6, %2\n" + " cbz %w3, 3f\n" + " sub %w4, %w4, %w3\n" + " cbnz %w4, 1b\n" + " mov %w0, %w7\n" + "3:\n" + " dmb ish\n" + "4:\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 4b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 4b, %w0) + : "+r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), "+r" (loops) + : "r" (oldval), "r" (newval), "Ir" (-EAGAIN) + : "memory"); + + mte_disable_tco(); + uaccess_ttbr0_disable(); + + if (!ret) + *oval =3D oldval; + + return ret; +} + +#define __lsui_llsc_body(op, ...) \ +({ \ + alternative_has_cap_likely(ARM64_HAS_LSUI) ? \ + __lsui_##op(__VA_ARGS__) : __llsc_##op(__VA_ARGS__); \ +}) + +#else /* CONFIG_AS_HAS_LSUI */ + +#define __lsui_llsc_body(op, ...) __llsc_##op(__VA_ARGS__) + +#endif /* CONFIG_AS_HAS_LSUI */ + + #define FUTEX_ATOMIC_OP(op) \ static __always_inline int \ __futex_atomic_##op(int oparg, u32 __user *uaddr, int *oval) \ { \ - return __llsc_futex_atomic_##op(oparg, uaddr, oval); \ + return __lsui_llsc_body(futex_atomic_##op, oparg, uaddr, oval); \ } =20 FUTEX_ATOMIC_OP(add) --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}