From nobody Wed Oct 8 18:33:14 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CE53625D552 for ; Wed, 25 Jun 2025 11:34:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851293; cv=none; b=ltWw3RiXuVldtt9YDMw4jghjc86GsVzFpvCYXYKBeB9QmQJWCvqx1JcHT9Ota2GozqhVNHn/dvWT+YQRN+pjJs/uzhyHJeh3i2fPg2zNKgQq9AM98Pn1gZXcLAPRyh0pf/7dn8UUOxdm5VLSCy+YDfb7IzuusqPc5lH1QTsT4aI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851293; c=relaxed/simple; bh=RbP84HcCEnwX9DUuFw/tD8N6Pg3hmMFQLfZUlLdOBs0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=j7S0KO+Yp4TVTvtvcgEB0Pr9mNP8Io5L36TMOEvlhpbWmiKqw4yCsPB230NDpjmhXNCRpDeun/vp5n1TWMpDyjEsNNlEz/qlwoWYM9LC4YXSYIcR8Ov2OsGW4sjgt4FWE2SnZCpUFgkJGAs3AGFE5OzCt6Na0mId3iX8i+mz6bs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D4D9C204C; Wed, 25 Jun 2025 04:34:32 -0700 (PDT) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.2.80.43]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 134DA3F58B; Wed, 25 Jun 2025 04:34:46 -0700 (PDT) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: ryan.roberts@arm.com, yang@os.amperecomputing.com, catalin.marinas@arm.com, will@kernel.org, jean-philippe@linaro.org, robin.murphy@arm.com, joro@8bytes.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, james.morse@arm.com, broonie@kernel.org, ardb@kernel.org, baohua@kernel.org, suzuki.poulose@arm.com, david@redhat.com, jgg@ziepe.ca, nicolinc@nvidia.com, jsnitsel@redhat.com, mshavit@google.com, kevin.tian@intel.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= , Suzuki K Poulose Subject: [PATCH v8 1/4] arm64: cpufeature: Introduce MATCH_ALL_EARLY_CPUS capability type Date: Wed, 25 Jun 2025 11:34:32 +0000 Message-ID: <20250625113435.26849-2-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250625113435.26849-1-miko.lenczewski@arm.com> References: <20250625113435.26849-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Catalin Marinas For system-wide capabilities, the kernel has the SCOPE_SYSTEM type. Such capabilities are checked once the SMP boot has completed using the sanitised ID registers. However, there is a need for a new capability type similar in scope to the system one but with checking performed locally on each CPU during boot (e.g. based on MIDR_EL1 which is not a sanitised register). Introduce ARM64_CPUCAP_MATCH_ALL_EARLY_CPUS which, together with ARM64_CPUCAP_SCOPE_LOCAL_CPU, ensures that such capability is enabled only if all early CPUs have it. For ease of use, define ARM64_CPUCAP_EARLY_LOCAL_CPU_FEATURE which combines SCOPE_LOCAL_CPU, PERMITTED_FOR_LATE_CPUS and MATCH_ALL_EARLY_CPUS. Signed-off-by: Catalin Marinas Signed-off-by: Miko=C5=82aj Lenczewski Reviewed-by: Suzuki K Poulose --- arch/arm64/include/asm/cpufeature.h | 23 +++++++++++ arch/arm64/kernel/cpufeature.c | 60 +++++++++++++++++++++++------ 2 files changed, 72 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/c= pufeature.h index c4326f1cb917..155ebd040c55 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -275,6 +275,14 @@ extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; #define ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU ((u16)BIT(5)) /* Panic when a conflict is detected */ #define ARM64_CPUCAP_PANIC_ON_CONFLICT ((u16)BIT(6)) +/* + * When paired with SCOPE_LOCAL_CPU, all early CPUs must satisfy the + * condition. This is different from SCOPE_SYSTEM where the check is perfo= rmed + * only once at the end of the SMP boot on the sanitised ID registers. + * SCOPE_SYSTEM is not suitable for cases where the capability depends on + * properties local to a CPU like MIDR_EL1. + */ +#define ARM64_CPUCAP_MATCH_ALL_EARLY_CPUS ((u16)BIT(7)) =20 /* * CPU errata workarounds that need to be enabled at boot time if one or @@ -304,6 +312,16 @@ extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; (ARM64_CPUCAP_SCOPE_LOCAL_CPU | \ ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU | \ ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU) +/* + * CPU feature detected at boot time and present on all early CPUs. Late C= PUs + * are permitted to have the feature even if it hasn't been enabled, altho= ugh + * the feature will not be used by Linux in this case. If all early CPUs h= ave + * the feature, then every late CPU must have it. + */ +#define ARM64_CPUCAP_EARLY_LOCAL_CPU_FEATURE \ + (ARM64_CPUCAP_SCOPE_LOCAL_CPU | \ + ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU | \ + ARM64_CPUCAP_MATCH_ALL_EARLY_CPUS) =20 /* * CPU feature detected at boot time, on one or more CPUs. A late CPU @@ -391,6 +409,11 @@ static inline int cpucap_default_scope(const struct ar= m64_cpu_capabilities *cap) return cap->type & ARM64_CPUCAP_SCOPE_MASK; } =20 +static inline bool cpucap_match_all_early_cpus(const struct arm64_cpu_capa= bilities *cap) +{ + return cap->type & ARM64_CPUCAP_MATCH_ALL_EARLY_CPUS; +} + /* * Generic helper for handling capabilities with multiple (match,enable) p= airs * of call backs, sharing the same capability bit. diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index b34044e20128..f9c947166322 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -3370,18 +3370,49 @@ static void update_cpu_capabilities(u16 scope_mask) =20 scope_mask &=3D ARM64_CPUCAP_SCOPE_MASK; for (i =3D 0; i < ARM64_NCAPS; i++) { + bool match_all =3D false; + bool caps_set =3D false; + bool boot_cpu =3D false; + caps =3D cpucap_ptrs[i]; - if (!caps || !(caps->type & scope_mask) || - cpus_have_cap(caps->capability) || - !caps->matches(caps, cpucap_default_scope(caps))) + if (!caps || !(caps->type & scope_mask)) continue; =20 - if (caps->desc && !caps->cpus) + match_all =3D cpucap_match_all_early_cpus(caps); + caps_set =3D cpus_have_cap(caps->capability); + boot_cpu =3D scope_mask & SCOPE_BOOT_CPU; + + /* + * Unless it's a match-all CPUs feature, avoid probing if + * already detected. + */ + if (!match_all && caps_set) + continue; + + /* + * A match-all CPUs capability is only set when probing the + * boot CPU. It may be cleared subsequently if not detected on + * secondary ones. + */ + if (match_all && !caps_set && !boot_cpu) + continue; + + if (!caps->matches(caps, cpucap_default_scope(caps))) { + if (match_all) + __clear_bit(caps->capability, system_cpucaps); + continue; + } + + /* + * Match-all CPUs capabilities are logged later when the + * system capabilities are finalised. + */ + if (!match_all && caps->desc && !caps->cpus) pr_info("detected: %s\n", caps->desc); =20 __set_bit(caps->capability, system_cpucaps); =20 - if ((scope_mask & SCOPE_BOOT_CPU) && (caps->type & SCOPE_BOOT_CPU)) + if (boot_cpu && (caps->type & SCOPE_BOOT_CPU)) set_bit(caps->capability, boot_cpucaps); } } @@ -3782,17 +3813,24 @@ static void __init setup_system_capabilities(void) enable_cpu_capabilities(SCOPE_ALL & ~SCOPE_BOOT_CPU); apply_alternatives_all(); =20 - /* - * Log any cpucaps with a cpumask as these aren't logged by - * update_cpu_capabilities(). - */ for (int i =3D 0; i < ARM64_NCAPS; i++) { const struct arm64_cpu_capabilities *caps =3D cpucap_ptrs[i]; =20 - if (caps && caps->cpus && caps->desc && - cpumask_any(caps->cpus) < nr_cpu_ids) + if (!caps || !caps->desc) + continue; + + /* + * Log any cpucaps with a cpumask as these aren't logged by + * update_cpu_capabilities(). + */ + if (caps->cpus && cpumask_any(caps->cpus) < nr_cpu_ids) pr_info("detected: %s on CPU%*pbl\n", caps->desc, cpumask_pr_args(caps->cpus)); + + /* Log match-all CPUs capabilities */ + if (cpucap_match_all_early_cpus(caps) && + cpus_have_cap(caps->capability)) + pr_info("detected: %s\n", caps->desc); } =20 /* --=20 2.49.0 From nobody Wed Oct 8 18:33:14 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C4B6025E458 for ; Wed, 25 Jun 2025 11:34:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851297; cv=none; b=sMRqrM4MaUdJXzKhWcOfIqp3WGf64phoxZy391mWGRrCrhF5V136l8niM/uQmPfKdTe9RebqsLd+/s+b2VE5iY3txID15SvSKBUV4IXc9ov0gCsLfccrSqTt/wvv2mpI/4nRW/xClNnvJdnC6+LXpzG8Nl8NjFQAeRFM1iZPEZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851297; c=relaxed/simple; bh=34RXRTCBCz54ZXIryn1Qbk9MUJlyFYgKwnPZunDPXoo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nXKPlS7S72Nu15CHEzHZuUSWm9V2oomZWhTrLJ6Xyc4scQnzi/SHJZaTBylTzp4KMgdQIByZwvuWpfAnE0OYnoqXmFTmjZnvalnIFpT1WMoVaK6OCE3cn2FZt3x2tmvgcmhLNFwdA93e6bHOmtFFrty1gG2RQymvn7dRcczP0Ok= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1FAD62050; Wed, 25 Jun 2025 04:34:37 -0700 (PDT) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.2.80.43]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 75D6C3F58B; Wed, 25 Jun 2025 04:34:51 -0700 (PDT) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: ryan.roberts@arm.com, yang@os.amperecomputing.com, catalin.marinas@arm.com, will@kernel.org, jean-philippe@linaro.org, robin.murphy@arm.com, joro@8bytes.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, james.morse@arm.com, broonie@kernel.org, ardb@kernel.org, baohua@kernel.org, suzuki.poulose@arm.com, david@redhat.com, jgg@ziepe.ca, nicolinc@nvidia.com, jsnitsel@redhat.com, mshavit@google.com, kevin.tian@intel.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= Subject: [PATCH v8 2/4] arm64: Add BBM Level 2 cpu feature Date: Wed, 25 Jun 2025 11:34:33 +0000 Message-ID: <20250625113435.26849-3-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250625113435.26849-1-miko.lenczewski@arm.com> References: <20250625113435.26849-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for. To support BBML2 in as wide a range of contexts as we can, we want not only the architectural guarantees that BBML2 makes, but additionally want BBML2 to not create TLB conflict aborts. Not causing aborts avoids us having to prove that no recursive faults can be induced in any path that uses BBML2, allowing its use for arbitrary kernel mappings. This feature builds on the previous ARM64_CPUCAP_EARLY_LOCAL_CPU_FEATURE, as all early cpus must support BBML2 for us to enable it (and any later cpus must also support it to be onlined). Not onlining late cpus that do not support BBML2 is unavoidable, as we might currently be using BBML2 semantics for kernel memory regions. This could cause faults in the late cpus, and would be difficult to unwind, so let us avoid the case altogether. Signed-off-by: Miko=C5=82aj Lenczewski Reviewed-by: Catalin Marinas Reviewed-by: Suzuki K Poulose Reviewed-by: Ryan Roberts --- arch/arm64/include/asm/cpufeature.h | 5 ++++ arch/arm64/kernel/cpufeature.c | 38 +++++++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 3 files changed, 44 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/c= pufeature.h index 155ebd040c55..bf13d676aae2 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -871,6 +871,11 @@ static inline bool system_supports_pmuv3(void) return cpus_have_final_cap(ARM64_HAS_PMUV3); } =20 +static inline bool system_supports_bbml2_noabort(void) +{ + return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT); +} + int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); =20 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index f9c947166322..7a3c404288d5 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2213,6 +2213,38 @@ static bool hvhe_possible(const struct arm64_cpu_cap= abilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } =20 +static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, i= nt scope) +{ + /* + * We want to allow usage of BBML2 in as wide a range of kernel contexts + * as possible. This list is therefore an allow-list of known-good + * implementations that both support BBML2 and additionally, fulfill the + * extra constraint of never generating TLB conflict aborts when using + * the relaxed BBML2 semantics (such aborts make use of BBML2 in certain + * kernel contexts difficult to prove safe against recursive aborts). + * + * Note that implementations can only be considered "known-good" if their + * implementors attest to the fact that the implementation never raises + * TLB conflict aborts for BBML2 mapping granularity changes. + */ + static const struct midr_range supports_bbml2_noabort_list[] =3D { + MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf), + MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), + {} + }; + + /* Does our cpu guarantee to never raise TLB conflict aborts? */ + if (!is_midr_in_range_list(supports_bbml2_noabort_list)) + return false; + + /* + * We currently ignore the ID_AA64MMFR2_EL1 register, and only care + * about whether the MIDR check passes. + */ + + return true; +} + #ifdef CONFIG_ARM64_PAN static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused) { @@ -2980,6 +3012,12 @@ static const struct arm64_cpu_capabilities arm64_fea= tures[] =3D { .matches =3D has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, EVT, IMP) }, + { + .desc =3D "BBM Level 2 without conflict abort", + .capability =3D ARM64_HAS_BBML2_NOABORT, + .type =3D ARM64_CPUCAP_EARLY_LOCAL_CPU_FEATURE, + .matches =3D has_bbml2_noabort, + }, { .desc =3D "52-bit Virtual Addressing for KVM (LPA2)", .capability =3D ARM64_HAS_LPA2, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 10effd4cff6b..2bd2bfaeddcd 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -45,6 +45,7 @@ HAS_LPA2 HAS_LSE_ATOMICS HAS_MOPS HAS_NESTED_VIRT +HAS_BBML2_NOABORT HAS_PAN HAS_PMUV3 HAS_S1PIE --=20 2.49.0 From nobody Wed Oct 8 18:33:14 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3071425EF82 for ; Wed, 25 Jun 2025 11:35:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851301; cv=none; b=EQy3kALdmZYu8YYCm/I43ZRUoujhtSvMdlellxlbwgipM+bvUHeOcUoIoKQb0F+Olpw7E4+aLB8lr3G+r1ThJ7rNpzPymTpY9kH/ealWnYo4ao8mXozpjhSHNKjr1sDMnov7jCsOHTs/RWWOMb2Q23exoBgkL4GqjjfWVTAJ1Ag= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851301; c=relaxed/simple; bh=7g2ZgDyqdK+/bHCMw8fbIJLN1rbyvElwZKb0WnbvDbg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gH2YJ4AYLpX2LsanFRDl8Kb5zHGFNeyGI29S3MajMVpcCzDnnicyr3CHljPThUlUmhftXvs8vwPrgjvv2lLjBZAKe3hGwFpJaywLFcxhNDJGDoLRL5xnNJKESm/IvEsVEQMiwe+kg9lpii9I3OS7K1IPwgLhS4Up214n0HKONL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5F8D120E3; Wed, 25 Jun 2025 04:34:41 -0700 (PDT) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.2.80.43]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B490D3F58B; Wed, 25 Jun 2025 04:34:55 -0700 (PDT) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: ryan.roberts@arm.com, yang@os.amperecomputing.com, catalin.marinas@arm.com, will@kernel.org, jean-philippe@linaro.org, robin.murphy@arm.com, joro@8bytes.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, james.morse@arm.com, broonie@kernel.org, ardb@kernel.org, baohua@kernel.org, suzuki.poulose@arm.com, david@redhat.com, jgg@ziepe.ca, nicolinc@nvidia.com, jsnitsel@redhat.com, mshavit@google.com, kevin.tian@intel.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= Subject: [PATCH v8 3/4] iommu/arm: Add BBM Level 2 smmu feature Date: Wed, 25 Jun 2025 11:34:34 +0000 Message-ID: <20250625113435.26849-4-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250625113435.26849-1-miko.lenczewski@arm.com> References: <20250625113435.26849-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed. Add the feature and testing for it under arm_smmu_sva_supported(). Signed-off-by: Miko=C5=82aj Lenczewski Reviewed-by: Catalin Marinas Reviewed-by: Robin Murphy Reviewed-by: Ryan Roberts --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++ 3 files changed, 8 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 0601dece0a0d..59a480974d80 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -220,6 +220,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smm= u) feat_mask |=3D ARM_SMMU_FEAT_VAX; } =20 + if (system_supports_bbml2_noabort()) + feat_mask |=3D ARM_SMMU_FEAT_BBML2; + if ((smmu->features & feat_mask) !=3D feat_mask) return false; =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 10cc6dc26b7b..39e933086f8f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4457,6 +4457,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_d= evice *smmu) if (FIELD_GET(IDR3_FWB, reg)) smmu->features |=3D ARM_SMMU_FEAT_S2FWB; =20 + if (FIELD_GET(IDR3_BBM, reg) =3D=3D 2) + smmu->features |=3D ARM_SMMU_FEAT_BBML2; + /* IDR5 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR5); =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index ea41d790463e..a33bf520ba97 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -60,6 +60,7 @@ struct arm_smmu_device; #define ARM_SMMU_IDR3 0xc #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) +#define IDR3_BBM GENMASK(12, 11) =20 #define ARM_SMMU_IDR5 0x14 #define IDR5_STALL_MAX GENMASK(31, 16) @@ -755,6 +756,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HA (1 << 21) #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24) u32 features; =20 #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0) --=20 2.49.0 From nobody Wed Oct 8 18:33:14 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8364525EFB7 for ; Wed, 25 Jun 2025 11:35:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851306; cv=none; b=AnGbOPfilPCxAumgmJL8OoXdNFM4lOIa8qteXkj2sff06pnr0JnOxMe2KQ2Fkj9etFILPxf5F90qKn23aMHZ5i9vmDUamgBXz7qT1JWxruC8tPmTBfqwCt+sD8LZWgaFvBtzPauH7nXSFt845YKzP/hDg+F8qkm3dk9NR5J/nSU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750851306; c=relaxed/simple; bh=zCmgJboP3hdyYZyXtLosDTfcpUEmzJVJ2CBg0JURuVA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=CZZmzdiWn3TWcIq6qNSMbxsTYoEB0wvVp9eN/qzr7FQqjTgj2DVGrDk+0gw2X5z06mK2H7nxW4tXVsb+Eb83+BZWi2cwmizuNZs6y/JLZSRrx/zBlM0/Ub3LVDnh01evAFraE3nZ4J+jGJoUegllk38JttigWG+VKMFr94iq/PE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9DB692103; Wed, 25 Jun 2025 04:34:45 -0700 (PDT) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.2.80.43]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id F3F5C3F58B; Wed, 25 Jun 2025 04:34:59 -0700 (PDT) From: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= To: ryan.roberts@arm.com, yang@os.amperecomputing.com, catalin.marinas@arm.com, will@kernel.org, jean-philippe@linaro.org, robin.murphy@arm.com, joro@8bytes.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, james.morse@arm.com, broonie@kernel.org, ardb@kernel.org, baohua@kernel.org, suzuki.poulose@arm.com, david@redhat.com, jgg@ziepe.ca, nicolinc@nvidia.com, jsnitsel@redhat.com, mshavit@google.com, kevin.tian@intel.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev Cc: =?UTF-8?q?Miko=C5=82aj=20Lenczewski?= Subject: [PATCH v8 4/4] arm64/mm: Elide tlbi in contpte_convert() under BBML2 Date: Wed, 25 Jun 2025 11:34:35 +0000 Message-ID: <20250625113435.26849-5-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250625113435.26849-1-miko.lenczewski@arm.com> References: <20250625113435.26849-1-miko.lenczewski@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable When converting a region via contpte_convert() to use mTHP, we have two different goals. We have to mark each entry as contiguous, and we would like to smear the dirty and young (access) bits across all entries in the contiguous block. Currently, we do this by first accumulating the dirty and young bits in the block, using an atomic __ptep_get_and_clear() and the relevant pte_{dirty,young}() calls, performing a tlbi, and finally smearing the correct bits across the block using __set_ptes(). This approach works fine for BBM level 0, but with support for BBM level 2 we are allowed to reorder the tlbi to after setting the pagetable entries. We expect the time cost of a tlbi to be much greater than the cost of clearing and resetting the PTEs. As such, this reordering of the tlbi outside the window where our PTEs are invalid greatly reduces the duration the PTE are visibly invalid for other threads. This reduces the likelyhood of a concurrent page walk finding an invalid PTE, reducing the likelyhood of a fault in other threads, and improving performance (more so when there are more threads). Because we support via allowlist only bbml2 implementations that never raise conflict aborts and instead invalidate the tlb entries automatically in hardware, we can avoid the final flush altogether. However, avoiding the intermediate tlbi+dsb must be carefully considered to ensure that we remain both correct and performant. We document our reasoning and the expected interactions further in the contpte_convert() source. To do so we rely on the aarch64 spec (DDI 0487L.a D8.7.1.1) requirements RNGLXZ and RJQQTC to provide guarantees that the elision is correct. Signed-off-by: Miko=C5=82aj Lenczewski Reviewed-by: Catalin Marinas Reviewed-by: Ryan Roberts Reviewed-by: Suzuki K Poulose --- arch/arm64/mm/contpte.c | 139 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 138 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index bcac4f55f9c1..203357061d0a 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -68,7 +68,144 @@ static void contpte_convert(struct mm_struct *mm, unsig= ned long addr, pte =3D pte_mkyoung(pte); } =20 - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); + /* + * On eliding the __tlb_flush_range() under BBML2+noabort: + * + * NOTE: Instead of using N=3D16 as the contiguous block length, we use + * N=3D4 for clarity. + * + * NOTE: 'n' and 'c' are used to denote the "contiguous bit" being + * unset and set, respectively. + * + * We worry about two cases where contiguous bit is used: + * - When folding N smaller non-contiguous ptes as 1 contiguous block. + * - When unfolding a contiguous block into N smaller non-contiguous pte= s. + * + * Currently, the BBML0 folding case looks as follows: + * + * 0) Initial page-table layout: + * + * +----+----+----+----+ + * |RO,n|RO,n|RO,n|RW,n| <--- last page being set as RO + * +----+----+----+----+ + * + * 1) Aggregate AF + dirty flags using __ptep_get_and_clear(): + * + * +----+----+----+----+ + * | 0 | 0 | 0 | 0 | + * +----+----+----+----+ + * + * 2) __flush_tlb_range(): + * + * |____ tlbi + dsb ____| + * + * 3) __set_ptes() to repaint contiguous block: + * + * +----+----+----+----+ + * |RO,c|RO,c|RO,c|RO,c| + * +----+----+----+----+ + * + * 4) The kernel will eventually __flush_tlb() for changed page: + * + * |____| <--- tlbi + dsb + * + * As expected, the intermediate tlbi+dsb ensures that other PEs + * only ever see an invalid (0) entry, or the new contiguous TLB entry. + * The final tlbi+dsb will always throw away the newly installed + * contiguous TLB entry, which is a micro-optimisation opportunity, + * but does not affect correctness. + * + * In the BBML2 case, the change is avoiding the intermediate tlbi+dsb. + * This means a few things, but notably other PEs will still "see" any + * stale cached TLB entries. This could lead to a "contiguous bit + * misprogramming" issue until the final tlbi+dsb of the changed page, + * which would clear out both the stale (RW,n) entry and the new (RO,c) + * contiguous entry installed in its place. + * + * What this is saying, is the following: + * + * +----+----+----+----+ + * |RO,n|RO,n|RO,n|RW,n| <--- old page tables, all non-contiguous + * +----+----+----+----+ + * + * +----+----+----+----+ + * |RO,c|RO,c|RO,c|RO,c| <--- new page tables, all contiguous + * +----+----+----+----+ + * /\ + * || + * + * If both the old single (RW,n) and new contiguous (RO,c) TLB entries + * are present, and a write is made to this address, do we fault or + * is the write permitted (via amalgamation)? + * + * The relevant Arm ARM DDI 0487L.a requirements are RNGLXZ and RJQQTC, + * and together state that when BBML1 or BBML2 are implemented, either + * a TLB conflict abort is raised (which we expressly forbid), or will + * "produce an OA, access permissions, and memory attributes that are + * consistent with any of the programmed translation table values". + * + * That is to say, will either raise a TLB conflict, or produce one of + * the cached TLB entries, but never amalgamate. + * + * Thus, as the page tables are only considered "consistent" after + * the final tlbi+dsb (which evicts both the single stale (RW,n) TLB + * entry as well as the new contiguous (RO,c) TLB entry), omitting the + * initial tlbi+dsb is correct. + * + * It is also important to note that at the end of the BBML2 folding + * case, we are still left with potentially all N TLB entries still + * cached (the N-1 non-contiguous ptes, and the single contiguous + * block). However, over time, natural TLB pressure will cause the + * non-contiguous pte TLB entries to be flushed, leaving only the + * contiguous block TLB entry. This means that omitting the tlbi+dsb is + * not only correct, but also keeps our eventual performance benefits. + * + * For the unfolding case, BBML0 looks as follows: + * + * 0) Initial page-table layout: + * + * +----+----+----+----+ + * |RW,c|RW,c|RW,c|RW,c| <--- last page being set as RO + * +----+----+----+----+ + * + * 1) Aggregate AF + dirty flags using __ptep_get_and_clear(): + * + * +----+----+----+----+ + * | 0 | 0 | 0 | 0 | + * +----+----+----+----+ + * + * 2) __flush_tlb_range(): + * + * |____ tlbi + dsb ____| + * + * 3) __set_ptes() to repaint as non-contiguous: + * + * +----+----+----+----+ + * |RW,n|RW,n|RW,n|RW,n| + * +----+----+----+----+ + * + * 4) Update changed page permissions: + * + * +----+----+----+----+ + * |RW,n|RW,n|RW,n|RO,n| <--- last page permissions set + * +----+----+----+----+ + * + * 5) The kernel will eventually __flush_tlb() for changed page: + * + * |____| <--- tlbi + dsb + * + * For BBML2, we again remove the intermediate tlbi+dsb. Here, there + * are no issues, as the final tlbi+dsb covering the changed page is + * guaranteed to remove the original large contiguous (RW,c) TLB entry, + * as well as the intermediate (RW,n) TLB entry; the next access will + * install the new (RO,n) TLB entry and the page tables are only + * considered "consistent" after the final tlbi+dsb, so software must + * be prepared for this inconsistency prior to finishing the mm dance + * regardless. + */ + + if (!system_supports_bbml2_noabort()) + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); =20 __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); } --=20 2.49.0