From nobody Sun Dec 14 13:45:41 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C11D41D6DB9 for ; Wed, 11 Dec 2024 15:46:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932011; cv=none; b=d27gVFfeqF0RGHofgM85tm8e+3tfhDqa/h3vBqt47ljr2RukWiDg5Q/+PZbJ9BBDjaQvmBhreZwDOlvRfleKyABh63wQ/E07SN4IQET4NhJhLDw92U+0ynJTw+zSBJhNFJkMs9FHuwrsFsqZfHqCSXAugJR9CSocIDWqUxChlLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932011; c=relaxed/simple; bh=M12C/rqNOn4PmgzITi2hwHA4DckgrKjjVVNmtRic7Ns=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QWnr/TTMPTgXNB5QO6VtnIxMJW30J+sOtm/gZ5hoqbP3Tc3Abfw9dGGD7bdVqrKdQNJjpKViG+PVgiToHRu0FGagMLaDxHurhE5DkOfhuU39aOBPEIssAevpn+dWbRz/3BNHYCXuhKK3aKCCGO/58OJsCWad/Z2b+Ck3gVz0AZY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1C87EFEC; Wed, 11 Dec 2024 07:47:17 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 78DAA3F5A1; Wed, 11 Dec 2024 07:46:47 -0800 (PST) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 1/5] arm64: Add TLB Conflict Abort Exception handler to KVM Date: Wed, 11 Dec 2024 15:45:02 +0000 Message-ID: <20241211154611.40395-2-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Currently, KVM does not handle the case of a stage 2 TLB conflict abort exception. The Arm ARM specifies that the worst-case handling of such an exception requires a `tlbi vmalls12e1`. Perform such an invalidation when this exception is encountered. Signed-off-by: Miko=C5=82aj Lenczewski --- arch/arm64/include/asm/esr.h | 8 ++++++++ arch/arm64/kvm/mmu.c | 6 ++++++ 2 files changed, 14 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index d1b1a33f9a8b..8a66f81ca291 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -121,6 +121,7 @@ #define ESR_ELx_FSC_SEA_TTW(n) (0x14 + (n)) #define ESR_ELx_FSC_SECC (0x18) #define ESR_ELx_FSC_SECC_TTW(n) (0x1c + (n)) +#define ESR_ELx_FSC_TLBABT (0x30) =20 /* Status codes for individual page table levels */ #define ESR_ELx_FSC_ACCESS_L(n) (ESR_ELx_FSC_ACCESS + (n)) @@ -464,6 +465,13 @@ static inline bool esr_fsc_is_access_flag_fault(unsign= ed long esr) (esr =3D=3D ESR_ELx_FSC_ACCESS_L(0)); } =20 +static inline bool esr_fsc_is_tlb_conflict_abort(unsigned long esr) +{ + esr =3D esr & ESR_ELx_FSC; + + return esr =3D=3D ESR_ELx_FSC_TLBABT; +} + /* Indicate whether ESR.EC=3D=3D0x1A is for an ERETAx instruction */ static inline bool esr_iss_is_eretax(unsigned long esr) { diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index c9d46ad57e52..c8c6f5a97a1b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1756,6 +1756,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) ipa =3D fault_ipa =3D kvm_vcpu_get_fault_ipa(vcpu); is_iabt =3D kvm_vcpu_trap_is_iabt(vcpu); =20 + if (esr_fsc_is_tlb_conflict_abort(esr)) { + // does a `tlbi vmalls12e1is` + __kvm_tlb_flush_vmid(&vcpu->kvm->arch.mmu); + return 1; + } + if (esr_fsc_is_translation_fault(esr)) { /* Beyond sanitised PARange (which is the IPA limit) */ if (fault_ipa >=3D BIT_ULL(get_kvm_ipa_limit())) { --=20 2.45.2 From nobody Sun Dec 14 13:45:41 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B6C561DA633 for ; Wed, 11 Dec 2024 15:46:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932013; cv=none; b=hiZG3uoXWEUK8R5AHDj5gsFvWyA2hZ3KZUjTwmPHycyWKwgW7vp9mmY2ADwGX4qpNUz8txFsPSYPT9PtFgryBKAu2qApIzqgopoze/R2WUfl6jnu1NbT8b4lcoMum49kwNhTV/DmOuHOdrF8kNu3reQ0o0RPZIYByeT4WD5EUC8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932013; c=relaxed/simple; bh=TYmsKaI2XKy6nFl7ZdhjfrfqKCt+R2KKhTjaP3gr3BE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Tfh6ggHVWHf0YJOYqcVgv2D7kdVRKxNlP2rUkIFaxMNYEkOyYdFl9OPl/ANc0FprUH34u0vx52Xtysmrp1ty3gpaUpKnQYrtzgnkxdl2bw9SLl3j4xZwUQ2Zb+OZeoKHbVc8zbaZyLTk4ncFinuUQiIZtxUXpORaXZgmYoemizY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E49AC1007; Wed, 11 Dec 2024 07:47:18 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4C4993F5A1; Wed, 11 Dec 2024 07:46:49 -0800 (PST) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 2/5] arm64: Add BBM Level 2 cpu feature Date: Wed, 11 Dec 2024 15:45:03 +0000 Message-ID: <20241211154611.40395-3-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for. In supporting BBM level 2, we open ourselves up to potential TLB Conflict Abort Exceptions during expected execution, instead of only in exceptional circumstances. In the case of an abort, it is implementation defined at what stage the abort is generated, and the minimal set of required invalidations is also implementation defined. The maximal set of invalidations is to do a `tlbi vmalle1` or `tlbi vmalls12e1`, depending on the stage. Such aborts should not occur on Arm hardware, and were not seen in benchmarked systems, so unless performance concerns arise, implementing the abort handlers with the worst-case invalidations seems like an alright hack. Signed-off-by: Miko=C5=82aj Lenczewski --- arch/arm64/include/asm/cpufeature.h | 14 ++++++++++++++ arch/arm64/kernel/cpufeature.c | 7 +++++++ arch/arm64/mm/fault.c | 27 ++++++++++++++++++++++++++- arch/arm64/tools/cpucaps | 1 + 4 files changed, 48 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/c= pufeature.h index 8b4e5a3cd24c..a9f2ac335392 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -866,6 +866,20 @@ static __always_inline bool system_supports_mpam_hcr(v= oid) return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } =20 +static inline bool system_supports_bbml2(void) +{ + /* currently, BBM is only relied on by code touching the userspace page + * tables, and as such we are guaranteed that caps have been finalised. + * + * if later we want to use BBM for kernel mappings, particularly early + * in the kernel, this may return 0 even if BBML2 is actually supported, + * which means unnecessary break-before-make sequences, but is still + * correct + */ + + return alternative_has_cap_unlikely(ARM64_HAS_BBML2); +} + int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); =20 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 6ce71f444ed8..7cc94bd5da24 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2917,6 +2917,13 @@ static const struct arm64_cpu_capabilities arm64_fea= tures[] =3D { .matches =3D has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, EVT, IMP) }, + { + .desc =3D "BBM Level 2 Support", + .capability =3D ARM64_HAS_BBML2, + .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, + .matches =3D has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) + }, { .desc =3D "52-bit Virtual Addressing for KVM (LPA2)", .capability =3D ARM64_HAS_LPA2, diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index ef63651099a9..dc119358cbc1 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -844,6 +844,31 @@ static int do_tag_check_fault(unsigned long far, unsig= ned long esr, return 0; } =20 +static int do_conflict_abort(unsigned long far, unsigned long esr, + struct pt_regs *regs) +{ + if (!system_supports_bbml2()) + return do_bad(far, esr, regs); + + /* if we receive a TLB conflict abort, we know that there are multiple + * TLB entries that translate the same address range. the minimum set + * of invalidations to clear these entries is implementation defined. + * the maximum set is defined as either tlbi(vmalls12e1) or tlbi(alle1). + * + * if el2 is enabled and stage 2 translation enabled, this may be + * raised as a stage 2 abort. if el2 is enabled but stage 2 translation + * disabled, or if el2 is disabled, it will be raised as a stage 1 + * abort. + * + * local_flush_tlb_all() does a tlbi(vmalle1), which is enough to + * handle a stage 1 abort. + */ + + local_flush_tlb_all(); + + return 0; +} + static const struct fault_info fault_info[] =3D { { do_bad, SIGKILL, SI_KERNEL, "ttbr address size fault" }, { do_bad, SIGKILL, SI_KERNEL, "level 1 address size fault" }, @@ -893,7 +918,7 @@ static const struct fault_info fault_info[] =3D { { do_bad, SIGKILL, SI_KERNEL, "unknown 45" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 46" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 47" }, - { do_bad, SIGKILL, SI_KERNEL, "TLB conflict abort" }, + { do_conflict_abort, SIGKILL, SI_KERNEL, "TLB conflict abort" }, { do_bad, SIGKILL, SI_KERNEL, "Unsupported atomic hardware update fault"= }, { do_bad, SIGKILL, SI_KERNEL, "unknown 50" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 51" }, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index eb17f59e543c..4ee0fbb7765b 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -26,6 +26,7 @@ HAS_ECV HAS_ECV_CNTPOFF HAS_EPAN HAS_EVT +HAS_BBML2 HAS_FPMR HAS_FGT HAS_FPSIMD --=20 2.45.2 From nobody Sun Dec 14 13:45:41 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2A40C1DE2DC for ; Wed, 11 Dec 2024 15:46:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932015; cv=none; b=EzgfsksgmkrnnAFP2ZZ9lv/vwNIEahWqn5y1a/n30Gg5vFCBa7jGRts7ATuHErJePAM5FgxKj5+Y0VM4D2SsITfdub5TSJZgNLM7h0890IbXNxszugIGbo8BXsHxenxY7Cm1/vdrD1owOvwOaMcxKjE8pfD8t2HmKZQ4FN1kfA8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932015; c=relaxed/simple; bh=pJTOe++UHDMbAPZ8TRmIriRZCC5kCu2W3Sd8nwe84u0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=m7btJgwgVLFTZE1be6qQmkV+dtC6A9VeQjEBBjef0sLk05h9Dov8wIEk2C+xNDIFgGGxvV05vixnUJbqKNqPYNd17/D8bX+wQPLT8MkYGrhLsQ3hOZq01kedZAz5L/YOcjdhkoTvNQWCJsIVKfmi41pioF65Z8v56H6mKD1r0rs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D329F1713; Wed, 11 Dec 2024 07:47:20 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 20DFE3F5A1; Wed, 11 Dec 2024 07:46:51 -0800 (PST) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 3/5] arm64: Add errata and workarounds for systems with broken BBML2 Date: Wed, 11 Dec 2024 15:45:04 +0000 Message-ID: <20241211154611.40395-4-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable There are systems which claim support for BBML2, but whose implementation of this support is broken. Add a Kconfig erratum for each of these systems, and a cpufeature workaround that forces the supported BBM level on these systems to 0. Signed-off-by: Miko=C5=82aj Lenczewski --- Documentation/arch/arm64/silicon-errata.rst | 32 ++++ arch/arm64/Kconfig | 164 ++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 32 +++- 3 files changed, 227 insertions(+), 1 deletion(-) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/ar= ch/arm64/silicon-errata.rst index b42fea07c5ce..4b4c1dd9b671 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -126,16 +126,26 @@ stable kernels. +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A76 | #3324349 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-A76 | #3696297 | ARM64_ERRATUM_36962= 97 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A77 | #1491015 | N/A = | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A77 | #1508412 | ARM64_ERRATUM_15084= 12 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A77 | #3324348 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-A77 | #3696294 | ARM64_ERRATUM_36962= 94 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A78 | #3324344 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-A78 | #3696287 | ARM64_ERRATUM_36962= 87 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A78C | #3324346,3324347| ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-A78C | #3696291 | ARM64_ERRATUM_36962= 91 | ++----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-A78C | #3696292 | ARM64_ERRATUM_36962= 92 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A710 | #2119858 | ARM64_ERRATUM_21198= 58 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A710 | #2054223 | ARM64_ERRATUM_20542= 23 | @@ -144,6 +154,8 @@ stable kernels. +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A710 | #3324338 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-A710 | #3696244 | ARM64_ERRATUM_36962= 44 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A715 | #2645198 | ARM64_ERRATUM_26451= 98 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-A715 | #3456084 | ARM64_ERRATUM_31943= 86 | @@ -156,6 +168,8 @@ stable kernels. +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-X1 | #3324344 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-X1 | #3696287 | ARM64_ERRATUM_36962= 87 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-X1C | #3324346 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-X2 | #2119858 | ARM64_ERRATUM_21198= 58 | @@ -164,10 +178,18 @@ stable kernels. +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-X2 | #3324338 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-X2 | #3696244 | ARM64_ERRATUM_36962= 44 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-X3 | #3324335 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-X3 | #3696239 | ARM64_ERRATUM_36962= 39 | ++----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-X4 | #3043263 | ARM64_ERRATUM_30432= 63 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-X4 | #3194386 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Cortex-X925 | #3056274 | ARM64_ERRATUM_30562= 74 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Cortex-X925 | #3324334 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-N1 | #1188873,1418040| ARM64_ERRATUM_14180= 40 | @@ -180,6 +202,8 @@ stable kernels. +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-N1 | #3324349 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Neoverse-N1 | #3696297 | ARM64_ERRATUM_36962= 97 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-N2 | #2139208 | ARM64_ERRATUM_21392= 08 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-N2 | #2067961 | ARM64_ERRATUM_20679= 61 | @@ -188,14 +212,22 @@ stable kernels. +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-N2 | #3324339 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Neoverse-N2 | #3696250 | ARM64_ERRATUM_36962= 50 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-N3 | #3456111 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-V1 | #1619801 | N/A = | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-V1 | #3324341 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Neoverse-V1 | #3696285 | ARM64_ERRATUM_36962= 85 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-V2 | #3324336 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Neoverse-V2 | #3696242 | ARM64_ERRATUM_36962= 42 | ++----------------+-----------------+-----------------+--------------------= ---------+ +| ARM | Neoverse-V3 | #3053180 | ARM64_ERRATUM_30531= 80 | ++----------------+-----------------+-----------------+--------------------= ---------+ | ARM | Neoverse-V3 | #3312417 | ARM64_ERRATUM_31943= 86 | +----------------+-----------------+-----------------+--------------------= ---------+ | ARM | MMU-500 | #841119,826419 | N/A = | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 100570a048c5..9ef8418e8410 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1127,6 +1127,170 @@ config ARM64_ERRATUM_3194386 =20 If unsure, say Y. =20 +config ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + bool + +config ARM64_ERRATUM_3696250 + bool "Neoverse-N2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-N2 cores (r0p0, r0p1, r0p2, r0p3) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696244 + bool "Cortex-A710/Cortex-X2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A710 and Cortex-X2 cores (r0p0, r1p0, r2p0, r2p1) + declare break-before-make level 2 support, but changing the block + size without utilising a break-before-make sequence, or + mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696297 + bool "Cortex-A76/Neoverse-N1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + This option adds a workaround for ARM Cortex-A76/Neoverse-N1 erratum + 3696297. + + Affected Cortex-A76 and Neoverse-N1 cores (r0p0, r1p0, r2p0, r3p0, + r3p1, r4p0, r4p1) declare break-before-make level 2 support, but + changing the block size without utilising a break-before-make sequence, + or mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696294 + bool "Cortex-A77: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + This option adds a workaround for ARM Cortex-A77 erratum 3696294. + + Affected Cortex-A77 cores (r0p0, r1p0, r1p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696239 + bool "Cortex-X3: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X3 cores (r0p0, r1p0, r1p1, r1p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696242 + bool "Neoverse-V2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V2 cores (r0p0, r0p1, r0p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696285 + bool "Neoverse-V1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V1 cores (r0p0, r1p0, r1p1, r1p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696287 + bool "Cortex-A78/Cortex-X1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78 and Cortex-X1 cores (r0p0, r1p0, r1p1, r1p2) + declare break-before-make level 2 support, but changing the block + size without utilising a break-before-make sequence, or + mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696291 + bool "Cortex-A78C: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78C cores (r0p0, r0p1, r0p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696292 + bool "Cortex-A78C: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78C cores (r0p1, r0p2) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3056274 + bool "Cortex-X925: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X925 cores (r0p0, r0p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3043263 + bool "Cortex-X4: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X4 cores (r0p0, r0p1, r0p2) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3053180 + bool "Neoverse-V3: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V3 cores (r0p0, r0p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + config CAVIUM_ERRATUM_22375 bool "Cavium erratum 22375, 24313" default y diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7cc94bd5da24..e6c05b330e0f 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2167,6 +2167,36 @@ static bool hvhe_possible(const struct arm64_cpu_cap= abilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } =20 +static bool has_bbml2(const struct arm64_cpu_capabilities *entry, + int scope) +{ + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT)) { + static const struct midr_range broken_bbml2_list[] =3D { + MIDR_ALL_VERSIONS(MIDR_CORTEX_A76), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A77), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78C), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A710), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X1), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X2), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X3), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X4), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X925), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V3), + {} + }; + + if (is_midr_in_range_list(read_cpuid_id(), broken_bbml2_list)) + return false; + } + + return has_cpuid_feature(entry, scope); +} + #ifdef CONFIG_ARM64_PAN static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused) { @@ -2921,7 +2951,7 @@ static const struct arm64_cpu_capabilities arm64_feat= ures[] =3D { .desc =3D "BBM Level 2 Support", .capability =3D ARM64_HAS_BBML2, .type =3D ARM64_CPUCAP_SYSTEM_FEATURE, - .matches =3D has_cpuid_feature, + .matches =3D has_bbml2, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) }, { --=20 2.45.2 From nobody Sun Dec 14 13:45:41 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0AE191DE4D4 for ; Wed, 11 Dec 2024 15:46:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932016; cv=none; b=QHttJ2Ldef9MB4TuOKrADXNHYeX3mCom9VKb5+03f1xP5P1noPWP2v7G7NW53Jjwui6E7OebibGxtCpuxmK5FZ7jEX82xTYI2UXlS8Mf0JF0XOH3ixLd4/eNBxNYKC3tlfvyBI1ggAsl1QotdRzJhOguXJ5MWuj9mZ0NBsS4jv4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932016; c=relaxed/simple; bh=Hgpm0rcPetVs44CBUg381VyIeoJc1wVTprblymycJVU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DIQaAbFLdo+OpFpdImhVdJ9KpNJlz9V7cnAqqTHiB0fNYHcRkVYt+ugF5vdRYwY+3qgI67kB1td9vGU30aHKlh5VhY0EGYNJ+ZfTWorisAD42fTtQE0WOmpmWDLsXP2g4iDlBILKzBsfdmQ1nHq1eYG1QsJC6wQJzrlZZN5LBWc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A6D481758; Wed, 11 Dec 2024 07:47:22 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0E76B3F5A1; Wed, 11 Dec 2024 07:46:52 -0800 (PST) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 4/5] arm64/mm: Delay tlbi in contpte_convert() under BBML2 Date: Wed, 11 Dec 2024 15:45:05 +0000 Message-ID: <20241211154611.40395-5-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable When converting a region via contpte_convert() to use mTHP, we have two different goals. We have to mark each entry as contiguous, and we would like to smear the dirty and young (access) bits across all entries in the contiguous block. Currently, we do this by first accumulating the dirty and young bits in the block, using an atomic __ptep_get_and_clear() and the relevant pte_{dirty,young}() calls, performing a tlbi, and finally smearing the correct bits across the block using __set_ptes(). This approach works fine for BBM level 0, but with support for BBM level 2 we are allowed to reorder the tlbi to after setting the pagetable entries. This reordering means that other threads will not see an invalid pagetable entry, instead operating on stale data, until we have performed our smearing and issued the invalidation. Avoiding this invalid entry reduces faults in other threads, and thus improves performance marginally (more so when there are more threads). Signed-off-by: Miko=C5=82aj Lenczewski --- arch/arm64/mm/contpte.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 55107d27d3f8..fc927be800ee 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -68,9 +68,13 @@ static void contpte_convert(struct mm_struct *mm, unsign= ed long addr, pte =3D pte_mkyoung(pte); } =20 - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); + if (!system_supports_bbml2()) + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); =20 __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); + + if (system_supports_bbml2()) + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); } =20 void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, --=20 2.45.2 From nobody Sun Dec 14 13:45:41 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 998991DE893 for ; Wed, 11 Dec 2024 15:46:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932018; cv=none; b=Y5OvRwREJ+q1557jeQGkBIn0x4Zvj8DuFcmsry8Rn0Am9sN2uYKiujgLUR2sDAUXkUcud1JSN0XnepjSKqs3f4yVBZGkZhL7dh4iWHikOxEZ7Co9rGsK/RIzHNdtTYTyu0HxzcNCuFNmfdhSXSaYswrsNKmKsnhnpJzxDTd1uqE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733932018; c=relaxed/simple; bh=WNhpzgsdAKtSd1yhrheXVoze9Qh5uwyI4MdEQXOJRBE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=up4Jt/p205rQ3sWgT+pck2hAqjg0yCZzoFnTJqpxdK992IfW45fDWNb8ousIbkJpFOkqQAwb+VvEGvz06Pf9IcpyaU6TgY/CT/okSBNBLRGGXUlGdbZFnilVl+6zgDr5x+9ufnK8PlU7SS4IRUO1GG7QMsJ3Rc2c9nQipkHZFuY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 79F5DFEC; Wed, 11 Dec 2024 07:47:24 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D5FD93F5A1; Wed, 11 Dec 2024 07:46:54 -0800 (PST) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 5/5] arm64/mm: Elide tlbi in contpte_convert() under BBML2 Date: Wed, 11 Dec 2024 15:45:06 +0000 Message-ID: <20241211154611.40395-6-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable If we support BBM level 2, we can potentially avoid an intermediate TLB invalidation, as hardware is capable of managing the TLB itself in this situation. Hardware will either silently clear out the offending entry, or will take a TLB Conflict Abort Exception. Note that such aborts should not occur on Arm hardware and indeed were not seen on any of the benchmarked systems. Eliding the invalidation results in a 12% improvement on a microbenchmark which targeted the worst case of contpte_convert(), which represents an 80% reduction in the overhead of contpte_convert(). Note also that this patch is pending review to ensure that it is architecturally valid, and we are working with Arm architects to validate this patch. Signed-off-by: Miko=C5=82aj Lenczewski --- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index fc927be800ee..009690770415 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigne= d long addr, __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); =20 __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); - - if (system_supports_bbml2()) - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); } =20 void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, --=20 2.45.2