From nobody Fri Dec 19 15:19:36 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 86509254AF5 for ; Tue, 22 Apr 2025 08:18:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745309946; cv=none; b=E4f0wkvaOhVN7RH6fUgFhIOa6RtCGMM3MjswWtv9yqKGkQH8KAH+ZnwZG7FM/hjJvjAd/kJURLgHPC2uEUdGRgQ/8rxtvhZ4rC3IgGKJXp8+c6o50+EcvxF0lmnjnCnU+jEXPAvxbgqcF/7aocpbLsQlf0IXRR+zM7jNjjkEb4c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745309946; c=relaxed/simple; bh=ImP6K2FPR86WHw4a4yR2PdzbZYjhfQRZjB0MdBnLTaA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LmeiAskfeEgwIZwaz/0A3SvV9npfKrIUeaA/1s61PQdecuvhETxVwFy472ykmiZEwBvZh0FSG8GGjBZm2QcghOnx//G+JgmrHvNnSp7+PkQQTAdlhnBXshy+XDgox01gAU9ZkYA1Dnvm8sj+wyTVc5dduvRuFcnjTSIQ+/ekbXQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 99A3B152B; Tue, 22 Apr 2025 01:18:49 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E685E3F66E; Tue, 22 Apr 2025 01:18:51 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Anshuman Khandual , Alexandre Ghiti , Kevin Brodsky Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 06/11] arm64/mm: Hoist barriers out of set_ptes_anysz() loop Date: Tue, 22 Apr 2025 09:18:14 +0100 Message-ID: <20250422081822.1836315-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250422081822.1836315-1-ryan.roberts@arm.com> References: <20250422081822.1836315-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" set_ptes_anysz() previously called __set_pte() for each PTE in the range, which would conditionally issue a DSB and ISB to make the new PTE value immediately visible to the table walker if the new PTE was valid and for kernel space. We can do better than this; let's hoist those barriers out of the loop so that they are only issued once at the end of the loop. We then reduce the cost by the number of PTEs in the range. Reviewed-by: Catalin Marinas Reviewed-by: Anshuman Khandual Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index d80aa9ba0a16..39c331743b69 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -320,13 +320,11 @@ static inline void __set_pte_nosync(pte_t *ptep, pte_= t pte) WRITE_ONCE(*ptep, pte); } =20 -static inline void __set_pte(pte_t *ptep, pte_t pte) +static inline void __set_pte_complete(pte_t pte) { - __set_pte_nosync(ptep, pte); - /* * Only if the new pte is valid and kernel, otherwise TLB maintenance - * or update_mmu_cache() have the necessary barriers. + * has the necessary barriers. */ if (pte_valid_not_user(pte)) { dsb(ishst); @@ -334,6 +332,12 @@ static inline void __set_pte(pte_t *ptep, pte_t pte) } } =20 +static inline void __set_pte(pte_t *ptep, pte_t pte) +{ + __set_pte_nosync(ptep, pte); + __set_pte_complete(pte); +} + static inline pte_t __ptep_get(pte_t *ptep) { return READ_ONCE(*ptep); @@ -658,12 +662,14 @@ static inline void __set_ptes_anysz(struct mm_struct = *mm, pte_t *ptep, =20 for (;;) { __check_safe_pte_update(mm, ptep, pte); - __set_pte(ptep, pte); + __set_pte_nosync(ptep, pte); if (--nr =3D=3D 0) break; ptep++; pte =3D pte_advance_pfn(pte, stride); } + + __set_pte_complete(pte); } =20 static inline void __set_ptes(struct mm_struct *mm, --=20 2.43.0