From nobody Mon Oct 6 13:37:45 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BAC761F4C8C for ; Mon, 21 Jul 2025 08:36:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753086997; cv=none; b=B/VEJW9RCrUohsNf/AZJW2RdqjLWkSg1lDHZmXTkIoW40cC+gt7YvxWYYs8+9dYff4JlCUZv7fme741pyQsGpT5opLZcBExppLHv0tPe1YWBIY1o+dh7TNSVlUz4Y+Vkr/5F1STPtg3B6+9FPnX+A+3bBgT0ff2IRuEaz41c3XA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753086997; c=relaxed/simple; bh=18O32Qquo+7xdJm6b6kD3Cbr6E5LknxchlbNYBSK7Dg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ULvH/7OTRUpmU6XiBIoN1ZGbDACV9W7Xqt3T3yFb53mLq4mJ9bewOI2R1omFegBC8MxzMgmxAr4xMkFjlPMeBGCyteEl1Ht6qOrkB7i9ltmGrEMWBFUGKwjNZ27MySGWYQFmU9AE3ETt8CB7dqcvaM1NL153iXk0MrvbnNeeLs0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2DEB51595; Mon, 21 Jul 2025 01:36:28 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3C7703F66E; Mon, 21 Jul 2025 01:36:32 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, oliver.upton@linux.dev, ardb@kernel.org, frederic@kernel.org, james.morse@arm.com, joey.gouly@arm.com, scott@os.amperecomputing.com, maz@kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v4 1/7] arm64: cpufeature: add FEAT_LSUI Date: Mon, 21 Jul 2025 09:36:12 +0100 Message-Id: <20250721083618.2743569-2-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250721083618.2743569-1-yeoreum.yun@arm.com> References: <20250721083618.2743569-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since Armv9.6, FEAT_LSUI supplies load/store instructions for privileged level to access user memory without clearing PSTATE.PAN bit. Add LSUI feature so that the unprevilieged load/store instructions could be used when kernel accesses user memory without clearing PSTATE.PAN = bit. Signed-off-by: Yeoreum Yun --- arch/arm64/kernel/cpufeature.c | 8 ++++++++ arch/arm64/tools/cpucaps | 1 + 2 files changed, 9 insertions(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index e151585c6cca..eaf958a0d8bc 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -278,6 +278,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar2[] = =3D { =20 static const struct arm64_ftr_bits ftr_id_aa64isar3[] =3D { ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_E= L1_FPRCVT_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_EL= 1_LSUI_SHIFT, 4, ID_AA64ISAR3_EL1_LSUI_NI), ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_E= L1_FAMINMAX_SHIFT, 4, 0), ARM64_FTR_END, }; @@ -3061,6 +3062,13 @@ static const struct arm64_cpu_capabilities arm64_fea= tures[] =3D { .matches =3D has_pmuv3, }, #endif + { + .desc =3D "Unprivileged Load Store Instructions (LSUI)", + .capability =3D ARM64_HAS_LSUI, + .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, + .matches =3D has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP) + }, {}, }; =20 diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 10effd4cff6b..31f2cd655666 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -43,6 +43,7 @@ HAS_HCX HAS_LDAPR HAS_LPA2 HAS_LSE_ATOMICS +HAS_LSUI HAS_MOPS HAS_NESTED_VIRT HAS_PAN --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Mon Oct 6 13:37:45 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EFCD6288523 for ; Mon, 21 Jul 2025 08:36:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753086999; cv=none; b=oNpFEy73S4KD/SWTTkpg8PurNlV7Y7X8DwRvQpXzX+lxg+DcB98qbylVtjnCIPpMbJlWXmfBwBokXTMhyJIpAXrG80ZLFuZEE8F9yn7DoqujIQ6Kp3gg3a2uCCF0RpZzlKzVesWhXsqMy20Mm7+Hb0y2eh6/sLRAPcCu9cPXbe4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753086999; c=relaxed/simple; bh=pjTUdVTOG9MPrzlV/yAcMLcqZdhbHR7qiM6fBTK1V/c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DV4CuQpvWt7VFsHpxyAFYUSLSNBomPHK4iVtj5RhzwdlCktK6ozc0k5Tzds0ImcdhZcODKnH1E9KpHray1P6NLAh0q2m13svkIjdIIKp8quEW/INBIoMtWittyHBTuWvsMP7kpR3ojFx6r3XnjdysDvmVxVwFSgPqNxmmjoA9HQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 617A130B1; Mon, 21 Jul 2025 01:36:30 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 64F9B3F66E; Mon, 21 Jul 2025 01:36:34 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, oliver.upton@linux.dev, ardb@kernel.org, frederic@kernel.org, james.morse@arm.com, joey.gouly@arm.com, scott@os.amperecomputing.com, maz@kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v4 2/7] KVM/arm64: expose FEAT_LSUI to guest Date: Mon, 21 Jul 2025 09:36:13 +0100 Message-Id: <20250721083618.2743569-3-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250721083618.2743569-1-yeoreum.yun@arm.com> References: <20250721083618.2743569-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" expose FEAT_LSUI to guest. Signed-off-by: Yeoreum Yun Acked-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index c20bd6f21e60..cfdf99e92cda 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1636,7 +1636,8 @@ static u64 __kvm_read_sanitised_id_reg(const struct k= vm_vcpu *vcpu, val &=3D ~ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_WFxT); break; case SYS_ID_AA64ISAR3_EL1: - val &=3D ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_FAMINMAX; + val &=3D ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_FAMINMAX | + ID_AA64ISAR3_EL1_LSUI; break; case SYS_ID_AA64MMFR2_EL1: val &=3D ~ID_AA64MMFR2_EL1_CCIDX_MASK; @@ -2921,7 +2922,7 @@ static const struct sys_reg_desc sys_reg_descs[] =3D { ID_AA64ISAR2_EL1_APA3 | ID_AA64ISAR2_EL1_GPA3)), ID_WRITABLE(ID_AA64ISAR3_EL1, (ID_AA64ISAR3_EL1_FPRCVT | - ID_AA64ISAR3_EL1_FAMINMAX)), + ID_AA64ISAR3_EL1_FAMINMAX | ID_AA64ISAR3_EL1_LSUI)), ID_UNALLOCATED(6,4), ID_UNALLOCATED(6,5), ID_UNALLOCATED(6,6), --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Mon Oct 6 13:37:45 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 245D2288CA3 for ; Mon, 21 Jul 2025 08:36:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087001; cv=none; b=IRq4s0UX4lrvj7wTPqfRGLF4Rf1Msu/5uzAwa7RyOTqXpKuNYPdbAd1EPTX+D7SONTU3kLS4SRRhS/eGpqKw0GcnUyEcPYKeG6KNCUBYtrbLNd8nQSz3M+QMLYkUC2p+pHq7lR4P5UtGhpFnQ/Y52+9zWRhRWMueTT0UMpWrLx0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087001; c=relaxed/simple; bh=FMjhLsDLNRXCPI7UG2W/acu+b+xaiVyshGkAVxY+Lwo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=apYlESWhgoi8sm6lD9ZWiF6oK4YzLTy9DfoWtMdksip/OL3lOM8mfJUkqMTBg+MORlq40hV+n9x6leur59qPzCl44sdRxxgKx7fkIOVJk1qnbCrXvWmPr/1aPxR6R2SSqmkEqaGhki3uXYhxSKRlk/NWty6sfrIuQsMYFtiyKlg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 901063105; Mon, 21 Jul 2025 01:36:32 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 959C63F66E; Mon, 21 Jul 2025 01:36:36 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, oliver.upton@linux.dev, ardb@kernel.org, frederic@kernel.org, james.morse@arm.com, joey.gouly@arm.com, scott@os.amperecomputing.com, maz@kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v4 3/7] arm64/Kconfig: add LSUI Kconfig Date: Mon, 21 Jul 2025 09:36:14 +0100 Message-Id: <20250721083618.2743569-4-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250721083618.2743569-1-yeoreum.yun@arm.com> References: <20250721083618.2743569-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since Armv9.6, FEAT_LSUI supplies the load/store instructions for previleged level to access to access user memory without clearing PSTATE.PAN bit. It's enough to add CONFIG_AS_HAS_LSUI only because the code for LUSI uses indiviual `.arch_extension` entries. Signed-off-by: Yeoreum Yun --- arch/arm64/Kconfig | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 393d71124f5d..c0beb44ed5b8 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2238,6 +2238,15 @@ config ARM64_GCS =20 endmenu # "v9.4 architectural features" =20 +config AS_HAS_LSUI + def_bool $(as-instr,.arch_extension lsui) + help + Unprivileged Load Store is an extension to introduce unprivileged + variants of load and store instructions so that clearing PSTATE.PAN + is never required in privileged mode. + This feature is available with clang version 20 and later and not yet + supported by gcc. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Mon Oct 6 13:37:45 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4EF8E289812 for ; Mon, 21 Jul 2025 08:36:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087002; cv=none; b=fAxeymucm71Wy54A/TawzxEuCv2wpZgeTe4rOrNA3fgYLTNMli0r+CLoO3GhFLRoTLfAmCokMKKJf9aXbLRVTRTN0xYMqW4ndADfiB7FXxqRxIk5nPFrm1AE7fgANt9rWqjq6H/0mi6zpVkmbhNYa6ZCCL0zc19IdLNIGG+u85I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087002; c=relaxed/simple; bh=Xty6o4O/6jE/zrmvBD8pJ9Znos7innQaLeWnUhWql2A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JUVXHVui4CyQgSaCiWFxDTdKCZWJn3mLFF3Q6+W30s00Ec+ZJ9HvPA4ccI7oA2WDd/lifyhxmdeb8t239Rfg508usgw3aW9NUQQIGFd6imyPh5smGj1bBBmVcI/u9YVNak3yjepEjdJ9hVyJPkWuWSYqdEPs54nvLSVoYKC9N/4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BDDD7153B; Mon, 21 Jul 2025 01:36:34 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C881D3F66E; Mon, 21 Jul 2025 01:36:38 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, oliver.upton@linux.dev, ardb@kernel.org, frederic@kernel.org, james.morse@arm.com, joey.gouly@arm.com, scott@os.amperecomputing.com, maz@kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v4 4/7] arm64/futex: move futex atomic logic with clearing PAN bit Date: Mon, 21 Jul 2025 09:36:15 +0100 Message-Id: <20250721083618.2743569-5-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250721083618.2743569-1-yeoreum.yun@arm.com> References: <20250721083618.2743569-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move current futex atomic logics which uses ll/sc method with cleraing PSTATE.PAN to separate file (futex_ll_sc_u.h) so that former method will be used only when FEAT_LSUI isn't supported. Signed-off-by: Yeoreum Yun --- arch/arm64/include/asm/futex_ll_sc_u.h | 115 +++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 arch/arm64/include/asm/futex_ll_sc_u.h diff --git a/arch/arm64/include/asm/futex_ll_sc_u.h b/arch/arm64/include/as= m/futex_ll_sc_u.h new file mode 100644 index 000000000000..6702ba66f1b2 --- /dev/null +++ b/arch/arm64/include/asm/futex_ll_sc_u.h @@ -0,0 +1,115 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2025 Arm Ltd. + */ +#ifndef __ASM_FUTEX_LL_SC_U_H +#define __ASM_FUTEX_LL_SC_U_H + +#include +#include + +#define FUTEX_ATOMIC_OP(op, asm_op) \ +static __always_inline int \ +__ll_sc_u_futex_atomic_##op(int oparg, u32 __user *uaddr, int *oval) \ +{ \ + unsigned int loops =3D LL_SC_MAX_LOOPS; \ + int ret, val, tmp; \ + \ + uaccess_enable_privileged(); \ + asm volatile("// __ll_sc_u_futex_atomic_" #op "\n" \ + " prfm pstl1strm, %2\n" \ + "1: ldxr %w1, %2\n" \ + " " #asm_op " %w3, %w1, %w5\n" \ + "2: stlxr %w0, %w3, %2\n" \ + " cbz %w0, 3f\n" \ + " sub %w4, %w4, %w0\n" \ + " cbnz %w4, 1b\n" \ + " mov %w0, %w6\n" \ + "3:\n" \ + " dmb ish\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w0) \ + : "=3D&r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), \ + "+r" (loops) \ + : "r" (oparg), "Ir" (-EAGAIN) \ + : "memory"); \ + uaccess_disable_privileged(); \ + \ + if (!ret) \ + *oval =3D val; \ + \ + return ret; \ +} + +FUTEX_ATOMIC_OP(add, add) +FUTEX_ATOMIC_OP(or, orr) +FUTEX_ATOMIC_OP(and, and) +FUTEX_ATOMIC_OP(eor, eor) + +#undef FUTEX_ATOMIC_OP + +static __always_inline int +__ll_sc_u_futex_atomic_set(int oparg, u32 __user *uaddr, int *oval) +{ + unsigned int loops =3D LL_SC_MAX_LOOPS; + int ret, val; + + uaccess_enable_privileged(); + asm volatile("//__ll_sc_u_futex_xchg\n" + " prfm pstl1strm, %2\n" + "1: ldxr %w1, %2\n" + "2: stlxr %w0, %w4, %2\n" + " cbz %w3, 3f\n" + " sub %w3, %w3, %w0\n" + " cbnz %w3, 1b\n" + " mov %w0, %w5\n" + "3:\n" + " dmb ish\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w0) + : "=3D&r" (ret), "=3D&r" (val), "+Q" (*uaddr), "+r" (loops) + : "r" (oparg), "Ir" (-EAGAIN) + : "memory"); + uaccess_disable_privileged(); + + if (!ret) + *oval =3D val; + + return ret; +} + +static __always_inline int +__ll_sc_u_futex_cmpxchg(u32 __user *uaddr, u32 oldval, u32 newval, u32 *ov= al) +{ + int ret =3D 0; + unsigned int loops =3D LL_SC_MAX_LOOPS; + u32 val, tmp; + + uaccess_enable_privileged(); + asm volatile("//__ll_sc_u_futex_cmpxchg\n" + " prfm pstl1strm, %2\n" + "1: ldxr %w1, %2\n" + " eor %w3, %w1, %w5\n" + " cbnz %w3, 4f\n" + "2: stlxr %w3, %w6, %2\n" + " cbz %w3, 3f\n" + " sub %w4, %w4, %w3\n" + " cbnz %w4, 1b\n" + " mov %w0, %w7\n" + "3:\n" + " dmb ish\n" + "4:\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 4b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 4b, %w0) + : "+r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), "+r" (loops) + : "r" (oldval), "r" (newval), "Ir" (-EAGAIN) + : "memory"); + uaccess_disable_privileged(); + + if (!ret) + *oval =3D val; + + return ret; +} + +#endif /* __ASM_FUTEX_LL_SC_U_H */ --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Mon Oct 6 13:37:45 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 79C992874FA for ; Mon, 21 Jul 2025 08:36:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087005; cv=none; b=M1GIvD/TnDHZQPhAgynRl+OmGJ5MLZrT+pM/MwsC60EHwxakQjeNqtPnn5Y8FYaMbznuFSYFFr3qUzJwO0z64KDIPar/24eOc8XGpvAhWrhKGOOxEqUgwBpR/EEkhAKUqb+6gIsAkVajGVPtEdTyckMV72s7+1i75N/fRQ6qjhg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087005; c=relaxed/simple; bh=DPGuwCDJOfOiHrOtdabFRKVGir64HkLXq5DYDJdM2RM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BPxfsl1ByJ5NSv71nGRxaEh4Zm0AE+vLi6hXS0PlI3v1/DFjhRuzL2KyryfEaWnCDP5IqLKuw6lUj2IzFRJfoH+bh0eXf5BHkUwG+JRPOVXVWc35Hs1va8W2G5+I4T8Dp/JtaiNqBqDrKiO35Hgw3c/UHCjFhC3plVbkrX4O7rU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E7B6E1595; Mon, 21 Jul 2025 01:36:36 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id F2A6F3F66E; Mon, 21 Jul 2025 01:36:40 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, oliver.upton@linux.dev, ardb@kernel.org, frederic@kernel.org, james.morse@arm.com, joey.gouly@arm.com, scott@os.amperecomputing.com, maz@kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v4 5/7] arm64/futex: add futex atomic operation with FEAT_LSUI Date: Mon, 21 Jul 2025 09:36:16 +0100 Message-Id: <20250721083618.2743569-6-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250721083618.2743569-1-yeoreum.yun@arm.com> References: <20250721083618.2743569-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current futex atomic operations are implemented with ll/sc instructions and clearing PSTATE.PAN. Since Armv9.6, FEAT_LSUI supplies not only load/store instructions but also atomic operation for user memory access in kernel it doesn't need to clear PSTATE.PAN bit anymore. With theses instructions some of futex atomic operations don't need to be implmented with ldxr/stlxr pair instead can be implmented with one atomic operation supplied by FEAT_LSUI. However, some of futex atomic operations still need to use ll/sc way via ldtxr/stltxr supplied by FEAT_LSUI since there is no correspondant atomic instruction or doesn't support word size operation (i.e) eor, cas{mb}t But It's good to work without clearing PSTATE.PAN bit. Signed-off-by: Yeoreum Yun --- arch/arm64/include/asm/futex_lsui.h | 132 ++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 arch/arm64/include/asm/futex_lsui.h diff --git a/arch/arm64/include/asm/futex_lsui.h b/arch/arm64/include/asm/f= utex_lsui.h new file mode 100644 index 000000000000..0dc7dca91cdb --- /dev/null +++ b/arch/arm64/include/asm/futex_lsui.h @@ -0,0 +1,132 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2025 Arm Ltd. + */ + +#ifndef __ASM_FUTEX_LSUI_H +#define __ASM_FUTEX_LSUI_H + +#include +#include + +#define FUTEX_ATOMIC_OP(op, asm_op, mb) \ +static __always_inline int \ +__lsui_futex_atomic_##op(int oparg, u32 __user *uaddr, int *oval) \ +{ \ + int ret =3D 0; \ + int val; \ + \ + mte_enable_tco(); \ + uaccess_ttbr0_enable(); \ + \ + asm volatile("// __lsui_futex_atomic_" #op "\n" \ + __LSUI_PREAMBLE \ + "1: " #asm_op #mb " %w3, %w2, %1\n" \ + "2:\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %w0) \ + : "+r" (ret), "+Q" (*uaddr), "=3Dr" (val) \ + : "r" (oparg) \ + : "memory"); \ + \ + mte_disable_tco(); \ + uaccess_ttbr0_disable(); \ + \ + if (!ret) \ + *oval =3D val; \ + \ + return ret; \ +} + +FUTEX_ATOMIC_OP(add, ldtadd, al) +FUTEX_ATOMIC_OP(or, ldtset, al) +FUTEX_ATOMIC_OP(andnot, ldtclr, al) +FUTEX_ATOMIC_OP(set, swpt, al) + +#undef FUTEX_ATOMIC_OP + +static __always_inline int +__lsui_futex_atomic_and(int oparg, u32 __user *uaddr, int *oval) +{ + return __lsui_futex_atomic_andnot(~oparg, uaddr, oval); +} + +static __always_inline int +__lsui_futex_atomic_eor(int oparg, u32 __user *uaddr, int *oval) +{ + unsigned int loops =3D LL_SC_MAX_LOOPS; + int ret, val, tmp; + + mte_enable_tco(); + uaccess_ttbr0_enable(); + + asm volatile("// __lsui_futex_atomic_eor\n" + __LSUI_PREAMBLE + " prfm pstl1strm, %2\n" + "1: ldtxr %w1, %2\n" + " eor %w3, %w1, %w5\n" + "2: stltxr %w0, %w3, %2\n" + " cbz %w0, 3f\n" + " sub %w4, %w4, %w0\n" + " cbnz %w4, 1b\n" + " mov %w0, %w6\n" + "3:\n" + " dmb ish\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w0) + : "=3D&r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), + "+r" (loops) + : "r" (oparg), "Ir" (-EAGAIN) + : "memory"); + + mte_disable_tco(); + uaccess_ttbr0_disable(); + + if (!ret) + *oval =3D val; + + return ret; +} + +static __always_inline int +__lsui_futex_cmpxchg(u32 __user *uaddr, u32 oldval, u32 newval, u32 *oval) +{ + int ret =3D 0; + unsigned int loops =3D LL_SC_MAX_LOOPS; + u32 val, tmp; + + mte_enable_tco(); + uaccess_ttbr0_enable(); + + /* + * cas{al}t doesn't support word size... + */ + asm volatile("//__lsui_futex_cmpxchg\n" + __LSUI_PREAMBLE + " prfm pstl1strm, %2\n" + "1: ldtxr %w1, %2\n" + " eor %w3, %w1, %w5\n" + " cbnz %w3, 4f\n" + "2: stltxr %w3, %w6, %2\n" + " cbz %w3, 3f\n" + " sub %w4, %w4, %w3\n" + " cbnz %w4, 1b\n" + " mov %w0, %w7\n" + "3:\n" + " dmb ish\n" + "4:\n" + _ASM_EXTABLE_UACCESS_ERR(1b, 4b, %w0) + _ASM_EXTABLE_UACCESS_ERR(2b, 4b, %w0) + : "+r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), "+r" (loops) + : "r" (oldval), "r" (newval), "Ir" (-EAGAIN) + : "memory"); + + mte_disable_tco(); + uaccess_ttbr0_disable(); + + if (!ret) + *oval =3D oldval; + + return ret; +} + +#endif /* __ASM_FUTEX_LSUI_H */ --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Mon Oct 6 13:37:45 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5FDEF28A1C0 for ; Mon, 21 Jul 2025 08:36:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087006; cv=none; b=TzZfI0pR9+RxL6PaT30gcPM36v27Mi066SXuYZt2JBN6Laci/zxHt58VypSCtmUACDzAZMvhwZLd78vqmlyEHnVeK1EPRLqBfhkuePt6pcVyl+iV4ncCUFXvcGoSi0TD1up2Sxf5+6qhBawaWurcf1DhyUXCLOLbqGy066z/FCw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087006; c=relaxed/simple; bh=1Pq3fWCbMme7can5L1VXxkKY/ixp6vJKLVRm1fVwo3g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ArzTOHtlxPrZ4nB3zRzALBaWwQx+IA/JCkQVKUeGDJAKDA9HWxFy7NcDx8zzzwsnXhfoRbBAkiVyg5eBt6j4ID2GYrKCmDBRvtygWysQ1icXHwf0LCtj5bWsE44TuVKm8OsfE+RcHMraoPgg+1YjXnIb34NlXh++G6oMG5SWUDk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1F56E153B; Mon, 21 Jul 2025 01:36:39 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2758F3F66E; Mon, 21 Jul 2025 01:36:43 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, oliver.upton@linux.dev, ardb@kernel.org, frederic@kernel.org, james.morse@arm.com, joey.gouly@arm.com, scott@os.amperecomputing.com, maz@kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v4 6/7] arm64/asm: introduce lsui.h Date: Mon, 21 Jul 2025 09:36:17 +0100 Message-Id: <20250721083618.2743569-7-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250721083618.2743569-1-yeoreum.yun@arm.com> References: <20250721083618.2743569-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch introduces lsui.h header file for applying runtime patch to use load/store unprevileged instructions when cpu supports FEAT_LSUI otherwise uses method implemented via ll/sc way with clearing PSTATE.PAN bit Signed-off-by: Yeoreum Yun --- arch/arm64/include/asm/lsui.h | 37 +++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 arch/arm64/include/asm/lsui.h diff --git a/arch/arm64/include/asm/lsui.h b/arch/arm64/include/asm/lsui.h new file mode 100644 index 000000000000..39bf232f3eb7 --- /dev/null +++ b/arch/arm64/include/asm/lsui.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2025 Arm Ltd. + */ +#ifndef __ASM_LSUI_H +#define __ASM_LSUI_H + +#define LL_SC_MAX_LOOPS 128 /* What's the largest number you can think of?= */ + +#include + +#ifdef CONFIG_AS_HAS_LSUI + +#define __LSUI_PREAMBLE ".arch_extension lsui\n" + +#include +#include +#include +#include +#include +#include + +#include + +#define __lsui_ll_sc_u_body(op, ...) \ +({ \ + alternative_has_cap_likely(ARM64_HAS_LSUI) ? \ + __lsui_##op(__VA_ARGS__) : \ + __ll_sc_u_##op(__VA_ARGS__); \ +}) + +#else /* CONFIG_AS_HAS_LSUI */ + +#define __lsui_ll_sc_u_body(op, ...) __ll_sc_u_##op(__VA_ARGS__) + +#endif /* CONFIG_AS_HAS_LSUI */ +#endif /* __ASM_LSUI_H */ --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7} From nobody Mon Oct 6 13:37:45 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CD30328A409 for ; Mon, 21 Jul 2025 08:36:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087009; cv=none; b=EuFThT1b7usEBftuoQJ9TYUpn63+40kSr3OIuJvxXkMssY3VLqkMF5zx83vydjob9EIVmHAZjPOsOjuGyEkjs6bP+PnoaPzZi9nFcSB+lcvkjVh41L4XLBzHys9MAIhI8t8rHOmRWR5CscdN/KiNWb+0PLL1lg14DHqki3q/p0I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753087009; c=relaxed/simple; bh=CCsMW2Y4vgWxPuHdWABQb40IsOOZLbjksJf8mcF8rtI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=klk2oyYEfdEi1tGatnKnUOoTAopRWnr49uNyvry1IWCWO6PiBZ0zYKv6niinszX/ODVwUEgG7FWLszRJz0TMG7ZB8SXbla86lKEWO4n552YF8F6+RjtZYZuZ6bIAfD4z0fQMsMI3gurfQ+cFfiEMzI6QuxjHGm46KYlPKg9D37c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4B49D153B; Mon, 21 Jul 2025 01:36:41 -0700 (PDT) Received: from e129823.cambridge.arm.com (e129823.arm.com [10.1.197.6]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 544283F66E; Mon, 21 Jul 2025 01:36:45 -0700 (PDT) From: Yeoreum Yun To: catalin.marinas@arm.com, will@kernel.org, broonie@kernel.org, oliver.upton@linux.dev, ardb@kernel.org, frederic@kernel.org, james.morse@arm.com, joey.gouly@arm.com, scott@os.amperecomputing.com, maz@kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Yeoreum Yun Subject: [PATCH v4 7/7] arm64/futex: support futex with FEAT_LSUI Date: Mon, 21 Jul 2025 09:36:18 +0100 Message-Id: <20250721083618.2743569-8-yeoreum.yun@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250721083618.2743569-1-yeoreum.yun@arm.com> References: <20250721083618.2743569-1-yeoreum.yun@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since Armv9.6, FEAT_LSUI supplies load/store unprevileged instructions for kernel to access user memory without clearing PSTATE.PAN. This patch makes futex use futex_atomic operations implemented with these instruction when cpu supports FEAT_LSUI otherwise they work with ldxr/stlxr with clearing PSTATE.PAN bit. Signed-off-by: Yeoreum Yun --- arch/arm64/include/asm/futex.h | 99 +++++++++++----------------------- 1 file changed, 31 insertions(+), 68 deletions(-) diff --git a/arch/arm64/include/asm/futex.h b/arch/arm64/include/asm/futex.h index bc06691d2062..ed4586776655 100644 --- a/arch/arm64/include/asm/futex.h +++ b/arch/arm64/include/asm/futex.h @@ -9,71 +9,60 @@ #include =20 #include +#include =20 -#define FUTEX_MAX_LOOPS 128 /* What's the largest number you can think of?= */ - -#define __futex_atomic_op(insn, ret, oldval, uaddr, tmp, oparg) \ -do { \ - unsigned int loops =3D FUTEX_MAX_LOOPS; \ - \ - uaccess_enable_privileged(); \ - asm volatile( \ -" prfm pstl1strm, %2\n" \ -"1: ldxr %w1, %2\n" \ - insn "\n" \ -"2: stlxr %w0, %w3, %2\n" \ -" cbz %w0, 3f\n" \ -" sub %w4, %w4, %w0\n" \ -" cbnz %w4, 1b\n" \ -" mov %w0, %w6\n" \ -"3:\n" \ -" dmb ish\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %w0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %w0) \ - : "=3D&r" (ret), "=3D&r" (oldval), "+Q" (*uaddr), "=3D&r" (tmp), \ - "+r" (loops) \ - : "r" (oparg), "Ir" (-EAGAIN) \ - : "memory"); \ - uaccess_disable_privileged(); \ -} while (0) +#define FUTEX_ATOMIC_OP(op) \ +static __always_inline int \ +__futex_atomic_##op(int oparg, u32 __user *uaddr, int *oval) \ +{ \ + return __lsui_ll_sc_u_body(futex_atomic_##op, oparg, uaddr, oval); \ +} + +FUTEX_ATOMIC_OP(add) +FUTEX_ATOMIC_OP(or) +FUTEX_ATOMIC_OP(and) +FUTEX_ATOMIC_OP(eor) +FUTEX_ATOMIC_OP(set) + +#undef FUTEX_ATOMIC_OP + +static __always_inline int +__futex_cmpxchg(u32 __user *uaddr, u32 oldval, u32 newval, u32 *oval) +{ + return __lsui_ll_sc_u_body(futex_cmpxchg, uaddr, oldval, newval, oval); +} =20 static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *_uad= dr) { - int oldval =3D 0, ret, tmp; - u32 __user *uaddr =3D __uaccess_mask_ptr(_uaddr); + int ret; + u32 __user *uaddr; =20 if (!access_ok(_uaddr, sizeof(u32))) return -EFAULT; =20 + uaddr =3D __uaccess_mask_ptr(_uaddr); + switch (op) { case FUTEX_OP_SET: - __futex_atomic_op("mov %w3, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_set(oparg, uaddr, oval); break; case FUTEX_OP_ADD: - __futex_atomic_op("add %w3, %w1, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_add(oparg, uaddr, oval); break; case FUTEX_OP_OR: - __futex_atomic_op("orr %w3, %w1, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_or(oparg, uaddr, oval); break; case FUTEX_OP_ANDN: - __futex_atomic_op("and %w3, %w1, %w5", - ret, oldval, uaddr, tmp, ~oparg); + ret =3D __futex_atomic_and(~oparg, uaddr, oval); break; case FUTEX_OP_XOR: - __futex_atomic_op("eor %w3, %w1, %w5", - ret, oldval, uaddr, tmp, oparg); + ret =3D __futex_atomic_eor(oparg, uaddr, oval); break; default: ret =3D -ENOSYS; } =20 - if (!ret) - *oval =3D oldval; - return ret; } =20 @@ -81,40 +70,14 @@ static inline int futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *_uaddr, u32 oldval, u32 newval) { - int ret =3D 0; - unsigned int loops =3D FUTEX_MAX_LOOPS; - u32 val, tmp; u32 __user *uaddr; =20 if (!access_ok(_uaddr, sizeof(u32))) return -EFAULT; =20 uaddr =3D __uaccess_mask_ptr(_uaddr); - uaccess_enable_privileged(); - asm volatile("// futex_atomic_cmpxchg_inatomic\n" -" prfm pstl1strm, %2\n" -"1: ldxr %w1, %2\n" -" sub %w3, %w1, %w5\n" -" cbnz %w3, 4f\n" -"2: stlxr %w3, %w6, %2\n" -" cbz %w3, 3f\n" -" sub %w4, %w4, %w3\n" -" cbnz %w4, 1b\n" -" mov %w0, %w7\n" -"3:\n" -" dmb ish\n" -"4:\n" - _ASM_EXTABLE_UACCESS_ERR(1b, 4b, %w0) - _ASM_EXTABLE_UACCESS_ERR(2b, 4b, %w0) - : "+r" (ret), "=3D&r" (val), "+Q" (*uaddr), "=3D&r" (tmp), "+r" (loops) - : "r" (oldval), "r" (newval), "Ir" (-EAGAIN) - : "memory"); - uaccess_disable_privileged(); - - if (!ret) - *uval =3D val; =20 - return ret; + return __futex_cmpxchg(uaddr, oldval, newval, uval); } =20 #endif /* __ASM_FUTEX_H */ --=20 LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}