From nobody Sat Sep 26 08:38:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6C44533CEBD for ; Thu, 3 Sep 2026 06:07:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415633; cv=none; b=Ruf4Ojcml0jVEm9r0x45cs6y/ydqvPnasnORDQxdWc1rCIKBV77lUsHB5RD0qoPUcH5kOrwMRhWdbGaEhSqlXGNmZRUn1qG5K6V12JfX1qSOPF09d6u4IzQQlbyK9op1ARF3QY3SaWQMAnjSRB0BCKPH4rbJrCZ4/zywQYtsM0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415633; c=relaxed/simple; bh=e0K8xOtqBeGx4lAEQyBohm7XHFu4Den9c9wf7atdv8U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZQ/n82633wC4fl5W/AofKicaXNjxrP7wgvsFoqTIiUbPB9MG+Y13dFY2Xy8SX7T6YwhMCS5CalRJk3QwTJ40FCC+SEMFKDytCzXMe0bgAmF9p3Oc3q7XYuJIddeiw2994La45Ad5TcGh2PmlyBH0meWlhGRBlnRfVIxCIAr8Q3Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=siJNcOZN; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="siJNcOZN" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D3DEF1682; Wed, 2 Sep 2026 23:07:01 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D9D7F3F673; Wed, 2 Sep 2026 23:07:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788415625; bh=e0K8xOtqBeGx4lAEQyBohm7XHFu4Den9c9wf7atdv8U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=siJNcOZNMhQYFwn/xphmLnRr8Xbq+prunkGCK+9yZW/Crwt4osUkCLna/m6t8JqlT MZrksgATbTjBPRQS62ugKWFHMZCimj73MFiOYoQG5w/yydGGP8nYg45FMkjY5BN4FR N2NG3EtK8EMRjqTgjBKsniUGFlNxhAyvwFg53Fao= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mark Rtland , linx-arm-kernel@lists.infradead.org, linx-kernel@vger.kernel.org, kasan-dev@googlegrops.com Subject: [PATCH 1/4] arm64/mm: Convert READ_ONCE() as pmdp_get() while accessing PMD Date: Thu, 3 Sep 2026 11:36:54 +0530 Message-ID: <20260903060657.2717457-2-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260903060657.2717457-1-anshuman.khandual@arm.com> References: <20260903060657.2717457-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert all READ_ONCE() based PMD accesses as pmdp_get() instead which will support both D64 and D128 translation regime going forward. That is because READ_ONCE() would need 128 bit single copy atomic guarantees, while reading 128 bit page table entries which is currently not supported on arm64. Build fails for READ_ONCE() while accessing beyond 64 bits. Load Pair/Store Pair (ldp/stp) are only single copy atomic if FEAT_LSE1 is supported (which is required when FEAT_D128 is supported). Currently 128 bit pgtables is a compile time decision - so we could have chosen to extend READ_ONCE()/WRITE_ONCE() to allow 128 bit for this configuration. But then it's a general purpose API and we were concerned that other users might eventually creep in that expect 128 and then fail to compile in the other configs. But worse, we are considering eventually making D128 a boot time option, at which point we'd have to make READ_ONCE() always allow 128 bit at compile time but then it might silently tear at runtime. So our preference is to standardize on these existing helpers, which we can override in arm64 to give the 128 bit single copy guarantee when required. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rtland Cc: linx-arm-kernel@lists.infradead.org Cc: linx-kernel@vger.kernel.org Cc: kasan-dev@googlegrops.com Signed-off-by: Anshuman Khandual Reviewed-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 3 ++- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 22 +++++++++++----------- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 2 +- 8 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 61fbfab3e06e..51f55498d43d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -805,7 +805,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) } =20 /* Find an entry in the third-level page table. */ -#define pte_offset_phys(dir,addr) (pmd_page_paddr(READ_ONCE(*(dir))) + pte= _index(addr) * sizeof(pte_t)) +#define pte_offset_phys(dir, addr) (pmd_page_paddr(pmdp_get(dir)) + \ + pte_index(addr) * sizeof(pte_t)) =20 #define pte_set_fixmap(addr) ((pte_t *)set_fixmap_offset(FIX_PTE, addr)) #define pte_set_fixmap_offset(pmd, addr) pte_set_fixmap(pte_offset_phys(pm= d, addr)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index c75bab3c2f4b..cb25cb130e83 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -188,7 +188,7 @@ static void show_pte(unsigned long addr) break; =20 pmdp =3D pmd_offset(pudp, addr); - pmd =3D READ_ONCE(*pmdp); + pmd =3D pmdp_get(pmdp); ptval_to_str(pmd_str, pmd_val(pmd)); pr_cont(", pmd=3D%s", pmd_str); if (pmd_none(pmd) || pmd_bad(pmd)) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index f66a0016dd02..3cdac8021d4f 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr) =20 static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr) { - pmd_t pmd =3D READ_ONCE(*pmdp); + pmd_t pmd =3D pmdp_get(pmdp); pte_t *ptep; =20 if (pmd_none(pmd)) { diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 8e799c1fe0aa..cdaa4500faf9 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, addr &=3D CONT_PMD_MASK; =20 pmdp =3D pmd_offset(pudp, addr); - pmd =3D READ_ONCE(*pmdp); + pmd =3D pmdp_get(pmdp); if (!(sz =3D=3D PMD_SIZE || sz =3D=3D CONT_PMD_SIZE) && pmd_none(pmd)) return NULL; diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 45fbdce684c8..7ca833c5de5e 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node) static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int= node, bool early) { - if (pmd_none(READ_ONCE(*pmdp))) { + if (pmd_none(pmdp_get(pmdp))) { phys_addr_t pte_phys =3D early ? __pa_symbol(kasan_early_shadow_pte) : kasan_alloc_zeroed_page(node); @@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsi= gned long addr, do { next =3D pmd_addr_end(addr, end); kasan_pte_populate(pmdp, addr, next, node, early); - } while (pmdp++, addr =3D next, addr !=3D end && pmd_none(READ_ONCE(*pmdp= ))); + } while (pmdp++, addr =3D next, addr !=3D end && pmd_none(pmdp_get(pmdp))= ); } =20 static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b5de4095d650..32daaa547d09 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -200,7 +200,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned lo= ng addr, int flags) { unsigned long next; - pmd_t pmd =3D READ_ONCE(*pmdp); + pmd_t pmd =3D pmdp_get(pmdp); pte_t *ptep; =20 BUG_ON(pmd_leaf(pmd)); @@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, un= signed long end, unsigned long next; =20 do { - pmd_t old_pmd =3D READ_ONCE(*pmdp); + pmd_t old_pmd =3D pmdp_get(pmdp); =20 next =3D pmd_addr_end(addr, end); =20 @@ -272,7 +272,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, un= signed long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), - READ_ONCE(pmd_val(*pmdp)))); + pmd_val(pmdp_get(pmdp)))); } else { int ret; =20 @@ -282,7 +282,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, un= signed long end, return ret; =20 VM_WARN_ON_ONCE(pmd_val(old_pmd) !=3D 0 && - pmd_val(old_pmd) !=3D READ_ONCE(pmd_val(*pmdp))); + pmd_val(old_pmd) !=3D pmd_val(pmdp_get(pmdp))); } phys +=3D next - addr; } while (pmdp++, addr =3D next, addr !=3D end); @@ -293,7 +293,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, un= signed long end, static bool pmd_range_has_valid_noncont(pmd_t *pmdp) { for (int i =3D 0; i < CONT_PMDS; i++) { - pte_t pte =3D pmd_pte(READ_ONCE(pmdp[i])); + pte_t pte =3D pmd_pte(pmdp_get(pmdp + i)); =20 if (pte_valid(pte) && !pte_cont(pte)) return true; @@ -1553,7 +1553,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsi= gned long addr, do { next =3D pmd_addr_end(addr, end); pmdp =3D pmd_offset(pudp, addr); - pmd =3D READ_ONCE(*pmdp); + pmd =3D pmdp_get(pmdp); if (pmd_none(pmd)) continue; =20 @@ -1708,7 +1708,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigne= d long addr, do { next =3D pmd_addr_end(addr, end); pmdp =3D pmd_offset(pudp, addr); - pmd =3D READ_ONCE(*pmdp); + pmd =3D pmdp_get(pmdp); if (pmd_none(pmd)) continue; =20 @@ -1729,7 +1729,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigne= d long addr, */ pmdp =3D pmd_offset(pudp, 0UL); for (i =3D 0; i < PTRS_PER_PMD; i++) { - if (!pmd_none(READ_ONCE(pmdp[i]))) + if (!pmd_none(pmdp_get(pmdp + i))) return; } =20 @@ -1881,7 +1881,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgpro= t_t prot) pmd_t new_pmd =3D pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot)); =20 /* Only allow permission changes for now */ - if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)), + if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)), pmd_val(new_pmd))) return 0; =20 @@ -1906,7 +1906,7 @@ int pud_clear_huge(pud_t *pudp) =20 int pmd_clear_huge(pmd_t *pmdp) { - if (!pmd_leaf(READ_ONCE(*pmdp))) + if (!pmd_leaf(pmdp_get(pmdp))) return 0; pmd_clear(pmdp); return 1; @@ -1917,7 +1917,7 @@ int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) pte_t *table; pmd_t pmd; =20 - pmd =3D READ_ONCE(*pmdp); + pmd =3D pmdp_get(pmdp); =20 if (!pmd_table(pmd)) { VM_WARN_ON(1); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index bbe98ac9ad8c..0ca07bd5ded9 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -414,7 +414,7 @@ bool kernel_page_present(struct page *page) return pud_valid(pud); =20 pmdp =3D pmd_offset(pudp, addr); - pmd =3D READ_ONCE(*pmdp); + pmd =3D pmdp_get(pmdp); if (pmd_none(pmd)) return false; if (pmd_leaf(pmd)) diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index cca9706a875c..b27b2d2c20c3 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -74,7 +74,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *d= st_pudp, =20 src_pmdp =3D pmd_offset(src_pudp, start); do { - pmd_t pmd =3D READ_ONCE(*src_pmdp); + pmd_t pmd =3D pmdp_get(src_pmdp); =20 next =3D pmd_addr_end(addr, end); if (pmd_none(pmd)) --=20 2.43.0 From nobody Sat Sep 26 08:38:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4B165391849 for ; Thu, 3 Sep 2026 06:07:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415632; cv=none; b=NRUXEaszu7V/sUvZ8ypas9qnSgw2FTjtG1aiaIVUf/URMqACyeoiL0o8V+Bd4vZ33otSe6Sun/EZOtQpms36PzsnqCiCL0DtTcr0CCfcelMpRExc80THLdnbcD00j+9Ux50ABmRR+l6mDTaHczujsPjbjpDdbsdj+Gwe6MbA0ss= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415632; c=relaxed/simple; bh=4452ysLjNmLryvgaNw39khfzX+Pth08JgAik4oKU6Jg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OAU2tNqmoPss/atA1WYPRhlVF6SHlnr3zuhfbQRVbZx4QbDGA8J6N+KG7CXepW5h0Dn3ebD7QeTLV2lSBdc1gXfKJjjSg+OcmwLaLikTf18wzoHaBjUtjxcp2Os8RprxOisRILaZ+xV8yxpaI0hRcactglq7bJnU/r4x04jFgg0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=hTEpmSGP; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="hTEpmSGP" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 91C351596; Wed, 2 Sep 2026 23:07:05 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 980D93F673; Wed, 2 Sep 2026 23:07:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788415629; bh=4452ysLjNmLryvgaNw39khfzX+Pth08JgAik4oKU6Jg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hTEpmSGPBTm5yHveeFi1B29Fw18kY2VSxJq0aQIMmPVsEaWdGB0Y9YTIkUIMd8nSZ M52vF1br0oobty+ZsHukgKnpLL2QVFtAgpfL3Qu9Dl/Fpd3KInMSGljQWHz/KTNOqY AJXCczCwhkHRv4K+5sKNZ1Sl7nqKjeoIBi5cWgv8= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: [PATCH 2/4] arm64/mm: Convert READ_ONCE() as pudp_get() while accessing PUD Date: Thu, 3 Sep 2026 11:36:55 +0530 Message-ID: <20260903060657.2717457-3-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260903060657.2717457-1-anshuman.khandual@arm.com> References: <20260903060657.2717457-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert all READ_ONCE() based PUD accesses as pudp_get() instead which will support both D64 and D128 translation regime going forward. That is because READ_ONCE() would need 128 bit single copy atomic guarantees, while reading 128 bit page table entries which is currently not supported on arm64. Build fails for READ_ONCE() while accessing beyond 64 bits. Load Pair/Store Pair (ldp/stp) are only single copy atomic if FEAT_LSE128 is supported (which is required when FEAT_D128 is supported). Currently 128 bit pgtables is a compile time decision - so we could have chosen to extend READ_ONCE()/WRITE_ONCE() to allow 128 bit for this configuration. But then it's a general purpose API and we were concerned that other users might eventually creep in that expect 128 and then fail to compile in the other configs. But worse, we are considering eventually making D128 a boot time option, at which point we'd have to make READ_ONCE() always allow 128 bit at compile time but then it might silently tear at runtime. So our preference is to standardize on these existing helpers, which we can override in arm64 to give the 128 bit single copy guarantee when required. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgtable.h | 3 ++- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 4 ++-- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 20 ++++++++++---------- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 8 files changed, 21 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 51f55498d43d..83087b70f192 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -867,7 +867,8 @@ static inline pmd_t *pud_pgtable(pud_t pud) } =20 /* Find an entry in the second-level page table. */ -#define pmd_offset_phys(dir, addr) (pud_page_paddr(READ_ONCE(*(dir))) + pm= d_index(addr) * sizeof(pmd_t)) +#define pmd_offset_phys(dir, addr) (pud_page_paddr(pudp_get(dir)) + \ + pmd_index(addr) * sizeof(pmd_t)) =20 #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) #define pmd_set_fixmap_offset(pud, addr) pmd_set_fixmap(pmd_offset_phys(pu= d, addr)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index cb25cb130e83..a8f07143512b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -181,7 +181,7 @@ static void show_pte(unsigned long addr) break; =20 pudp =3D pud_offset(p4dp, addr); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get(pudp); ptval_to_str(pud_str, pud_val(pud)); pr_cont(", pud=3D%s", pud_str); if (pud_none(pud) || pud_bad(pud)) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 3cdac8021d4f..d9a870836faf 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -56,7 +56,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, uns= igned long addr, unsigned long end) { unsigned long next; - pud_t pud =3D READ_ONCE(*pudp); + pud_t pud =3D pudp_get(pudp); pmd_t *pmdp; =20 if (pud_none(pud)) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index cdaa4500faf9..c9ad5e75b073 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -262,7 +262,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_a= rea_struct *vma, WARN_ON(addr & (sz - 1)); ptep =3D pte_alloc_huge(mm, pmdp, addr); } else if (sz =3D=3D PMD_SIZE) { - if (want_pmd_share(vma, addr) && pud_none(READ_ONCE(*pudp))) + if (want_pmd_share(vma, addr) && pud_none(pudp_get(pudp))) ptep =3D huge_pmd_share(mm, vma, addr, pudp); else ptep =3D (pte_t *)pmd_alloc(mm, pudp, addr); @@ -292,7 +292,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return NULL; =20 pudp =3D pud_offset(p4dp, addr); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get(pudp); if (sz !=3D PUD_SIZE && pud_none(pud)) return NULL; /* hugepage or swap? */ diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 7ca833c5de5e..aad29bcc7622 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -76,7 +76,7 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsign= ed long addr, int node, static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int= node, bool early) { - if (pud_none(READ_ONCE(*pudp))) { + if (pud_none(pudp_get(pudp))) { phys_addr_t pmd_phys =3D early ? __pa_symbol(kasan_early_shadow_pmd) : kasan_alloc_zeroed_page(node); @@ -150,7 +150,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsi= gned long addr, do { next =3D pud_addr_end(addr, end); kasan_pmd_populate(pudp, addr, next, node, early); - } while (pudp++, addr =3D next, addr !=3D end && pud_none(READ_ONCE(*pudp= ))); + } while (pudp++, addr =3D next, addr !=3D end && pud_none(pudp_get(pudp))= ); } =20 static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 32daaa547d09..58a50809f91a 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -309,7 +309,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned lo= ng addr, { int ret; unsigned long next; - pud_t pud =3D READ_ONCE(*pudp); + pud_t pud =3D pudp_get(pudp); pmd_t *pmdp; =20 /* @@ -390,7 +390,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long ad= dr, unsigned long end, } =20 do { - pud_t old_pud =3D READ_ONCE(*pudp); + pud_t old_pud =3D pudp_get(pudp); =20 next =3D pud_addr_end(addr, end); =20 @@ -408,7 +408,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long ad= dr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), - READ_ONCE(pud_val(*pudp)))); + pud_val(pudp_get(pudp)))); } else { ret =3D alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); @@ -416,7 +416,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long ad= dr, unsigned long end, goto out; =20 VM_WARN_ON_ONCE(pud_val(old_pud) !=3D 0 && - pud_val(old_pud) !=3D READ_ONCE(pud_val(*pudp))); + pud_val(old_pud) !=3D pud_val(pudp_get(pudp))); } phys +=3D next - addr; } while (pudp++, addr =3D next, addr !=3D end); @@ -1591,7 +1591,7 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsi= gned long addr, do { next =3D pud_addr_end(addr, end); pudp =3D pud_offset(p4dp, addr); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get(pudp); if (pud_none(pud)) continue; =20 @@ -1748,7 +1748,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigne= d long addr, do { next =3D pud_addr_end(addr, end); pudp =3D pud_offset(p4dp, addr); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get(pudp); if (pud_none(pud)) continue; =20 @@ -1769,7 +1769,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigne= d long addr, */ pudp =3D pud_offset(p4dp, 0UL); for (i =3D 0; i < PTRS_PER_PUD; i++) { - if (!pud_none(READ_ONCE(pudp[i]))) + if (!pud_none(pudp_get(pudp + i))) return; } =20 @@ -1867,7 +1867,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgpro= t_t prot) pud_t new_pud =3D pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); =20 /* Only allow permission changes for now */ - if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)), + if (!pgattr_change_is_safe(pud_val(pudp_get(pudp)), pud_val(new_pud))) return 0; =20 @@ -1898,7 +1898,7 @@ void p4d_clear_huge(p4d_t *p4dp) =20 int pud_clear_huge(pud_t *pudp) { - if (!pud_leaf(READ_ONCE(*pudp))) + if (!pud_leaf(pudp_get(pudp))) return 0; pud_clear(pudp); return 1; @@ -1938,7 +1938,7 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) pud_t pud; unsigned long next, end; =20 - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get(pudp); =20 if (!pud_table(pud)) { VM_WARN_ON(1); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 0ca07bd5ded9..07b2fa4de57f 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -407,7 +407,7 @@ bool kernel_page_present(struct page *page) return false; =20 pudp =3D pud_offset(p4dp, addr); - pud =3D READ_ONCE(*pudp); + pud =3D pudp_get(pudp); if (pud_none(pud)) return false; if (pud_leaf(pud)) diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index b27b2d2c20c3..d119119455f1 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -64,7 +64,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *d= st_pudp, unsigned long next; unsigned long addr =3D start; =20 - if (pud_none(READ_ONCE(*dst_pudp))) { + if (pud_none(pudp_get(dst_pudp))) { dst_pmdp =3D trans_alloc(info); if (!dst_pmdp) return -ENOMEM; @@ -109,7 +109,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t = *dst_p4dp, =20 src_pudp =3D pud_offset(src_p4dp, start); do { - pud_t pud =3D READ_ONCE(*src_pudp); + pud_t pud =3D pudp_get(src_pudp); =20 next =3D pud_addr_end(addr, end); if (pud_none(pud)) --=20 2.43.0 From nobody Sat Sep 26 08:38:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CD5773C553A for ; Thu, 3 Sep 2026 06:07:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415636; cv=none; b=mgfNcLNLt2dYj058ak2ddTEEvHxNPqjpUh5uvDe58gJcV/LfHFR3QvjmiToI/mcWiwK0DFyi7NMPP+3CdkcDPoIgHz5+2tbxsMKO1jO74GW1II/sHGj5iKFqAtK3DDFkRv+/ISmfX8tKG2/MevAEOUP2odgUMTBeaQRwxJ/owqQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415636; c=relaxed/simple; bh=WITeshA5Gjlb1wS3L6JWLqI3qWIYQX4SJkJiHNHdcIk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ICzQIfXpTvZIbsiMUWTLpOezsmuhyKjiwt/fWbQZYI/EbxQMtkUEjFFsXuZ4Ib0loFvS5X6BDInLh306nDFNG3ituBcf2S8jfwGu13BzVCIyDrIcJBG403wecIZiLcmRy86/qbEgk058VOvDuS6D9KPutbD3G/and08qn7MXtP4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=GgIhfidM; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="GgIhfidM" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 51A8E1596; Wed, 2 Sep 2026 23:07:09 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 55DF73F673; Wed, 2 Sep 2026 23:07:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788415633; bh=WITeshA5Gjlb1wS3L6JWLqI3qWIYQX4SJkJiHNHdcIk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GgIhfidMPwzwaSmCGD93QImBME/pihHBuoakFAk7mGRoLZvGyC/sr0tmulQJTP7Gi uT6BdpU0oG4pxMfnzPOeq5e2umZBqSk9NoJYIApP7wY7TOFw0eT0YUfplGvAXrnrgP qA+XtUid40qsh6PkWLuE0fRqAvD2Ch2eLgTGjWM0= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: [PATCH 3/4] arm64/mm: Convert READ_ONCE() as p4dp_get() while accessing P4D Date: Thu, 3 Sep 2026 11:36:56 +0530 Message-ID: <20260903060657.2717457-4-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260903060657.2717457-1-anshuman.khandual@arm.com> References: <20260903060657.2717457-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert all READ_ONCE() based P4D accesses as p4dp_get() instead which will support both D64 and D128 translation regime going forward. That is because READ_ONCE() would need 128 bit single copy atomic guarantees, while reading 128 bit page table entries which is currently not supported on arm64. Build fails for READ_ONCE() while accessing beyond 64 bits. Load Pair/Store Pair (ldp/stp) are only single copy atomic if FEAT_LSE128 is supported (which is required when FEAT_D128 is supported). Currently 128 bit pgtables is a compile time decision - so we could have chosen to extend READ_ONCE()/WRITE_ONCE() to allow 128 bit for this configuration. But then it's a general purpose API and we were concerned that other users might eventually creep in that expect 128 and then fail to compile in the other configs. But worse, we are considering eventually making D128 a boot time option, at which point we'd have to make READ_ONCE() always allow 128 bit at compile time but then it might silently tear at runtime. So our preference is to standardize on these existing helpers, which we can override in arm64 to give the 128 bit single copy guarantee when required. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Signed-off-by: Anshuman Khandual --- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 12 ++++++------ arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 7 files changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index a8f07143512b..95d044cf04d4 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -174,7 +174,7 @@ static void show_pte(unsigned long addr) break; =20 p4dp =3D p4d_offset(pgdp, addr); - p4d =3D READ_ONCE(*p4dp); + p4d =3D p4dp_get(p4dp); ptval_to_str(p4d_str, p4d_val(p4d)); pr_cont(", p4d=3D%s", p4d_str); if (p4d_none(p4d) || p4d_bad(p4d)) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index d9a870836faf..2a0fe0b0f771 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -74,7 +74,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, uns= igned long addr, static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end) { - p4d_t p4d =3D READ_ONCE(*p4dp); + p4d_t p4d =3D p4dp_get(p4dp); pud_t *pudp; =20 if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) && diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index c9ad5e75b073..5771bf49e1fe 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -288,7 +288,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return NULL; =20 p4dp =3D p4d_offset(pgdp, addr); - if (!p4d_present(READ_ONCE(*p4dp))) + if (!p4d_present(p4dp_get(p4dp))) return NULL; =20 pudp =3D pud_offset(p4dp, addr); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index aad29bcc7622..4dd8c1186471 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -89,7 +89,7 @@ static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsign= ed long addr, int node, static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int= node, bool early) { - if (p4d_none(READ_ONCE(*p4dp))) { + if (p4d_none(p4dp_get(p4dp))) { phys_addr_t pud_phys =3D early ? __pa_symbol(kasan_early_shadow_pud) : kasan_alloc_zeroed_page(node); @@ -162,7 +162,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsi= gned long addr, do { next =3D p4d_addr_end(addr, end); kasan_pud_populate(p4dp, addr, next, node, early); - } while (p4dp++, addr =3D next, addr !=3D end && p4d_none(READ_ONCE(*p4dp= ))); + } while (p4dp++, addr =3D next, addr !=3D end && p4d_none(p4dp_get(p4dp))= ); } =20 static void __init kasan_pgd_populate(unsigned long addr, unsigned long en= d, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 58a50809f91a..6f38e3618d26 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -367,7 +367,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long ad= dr, unsigned long end, { int ret =3D 0; unsigned long next; - p4d_t p4d =3D READ_ONCE(*p4dp); + p4d_t p4d =3D p4dp_get(p4dp); pud_t *pudp; =20 if (p4d_none(p4d)) { @@ -457,7 +457,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long ad= dr, unsigned long end, } =20 do { - p4d_t old_p4d =3D READ_ONCE(*p4dp); + p4d_t old_p4d =3D p4dp_get(p4dp); =20 next =3D p4d_addr_end(addr, end); =20 @@ -467,7 +467,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long ad= dr, unsigned long end, goto out; =20 VM_WARN_ON_ONCE(p4d_val(old_p4d) !=3D 0 && - p4d_val(old_p4d) !=3D READ_ONCE(p4d_val(*p4dp))); + p4d_val(old_p4d) !=3D (p4d_val(p4dp_get(p4dp)))); =20 phys +=3D next - addr; } while (p4dp++, addr =3D next, addr !=3D end); @@ -1622,7 +1622,7 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsi= gned long addr, do { next =3D p4d_addr_end(addr, end); p4dp =3D p4d_offset(pgdp, addr); - p4d =3D READ_ONCE(*p4dp); + p4d =3D p4dp_get(p4dp); if (p4d_none(p4d)) continue; =20 @@ -1788,7 +1788,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigne= d long addr, do { next =3D p4d_addr_end(addr, end); p4dp =3D p4d_offset(pgdp, addr); - p4d =3D READ_ONCE(*p4dp); + p4d =3D p4dp_get(p4dp); if (p4d_none(p4d)) continue; =20 @@ -1809,7 +1809,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigne= d long addr, */ p4dp =3D p4d_offset(pgdp, 0UL); for (i =3D 0; i < PTRS_PER_P4D; i++) { - if (!p4d_none(READ_ONCE(p4dp[i]))) + if (!p4d_none(p4dp_get(p4dp + i))) return; } =20 diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 07b2fa4de57f..826856ef64a8 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -403,7 +403,7 @@ bool kernel_page_present(struct page *page) return false; =20 p4dp =3D p4d_offset(pgdp, addr); - if (p4d_none(READ_ONCE(*p4dp))) + if (p4d_none(p4dp_get(p4dp))) return false; =20 pudp =3D pud_offset(p4dp, addr); diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index d119119455f1..7afe2beca4ba 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -99,7 +99,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *d= st_p4dp, unsigned long next; unsigned long addr =3D start; =20 - if (p4d_none(READ_ONCE(*dst_p4dp))) { + if (p4d_none(p4dp_get(dst_p4dp))) { dst_pudp =3D trans_alloc(info); if (!dst_pudp) return -ENOMEM; @@ -145,7 +145,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t = *dst_pgdp, src_p4dp =3D p4d_offset(src_pgdp, start); do { next =3D p4d_addr_end(addr, end); - if (p4d_none(READ_ONCE(*src_p4dp))) + if (p4d_none(p4dp_get(src_p4dp))) continue; if (copy_pud(info, dst_p4dp, src_p4dp, addr, next)) return -ENOMEM; --=20 2.43.0 From nobody Sat Sep 26 08:38:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8021F54763 for ; Thu, 3 Sep 2026 06:07:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415641; cv=none; b=EFL0979O1EHaboyml0unoDSjUGhRlJYmM2qwCWdhI6oOoXZXo57bCvhWt3mSvmMz9oSmO0YRKHDYc/RjCB71KgBAeXiuQDtdzhaTkzO19aezBqaVXuGCUwxN+a4KwZqpBnyYlGKZqUN9UXtNGagMR7bEWFJE83RT5LrSXRtdBGM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788415641; c=relaxed/simple; bh=RRCvk8AHwrIHgucZgMeo3d4bx61Rqtuocq+rKtw+8mg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HnIHCFFzlTyHF572LygzilNQQu+uDhf5XDLhgKxwXrjbGVqKunpw1MdV2+W0V4w8R4fY7h8Av/ktcrmFncq2gZuZLlgeG0oPQTCePn6+uGCHf1wPkH0J5CB48oLRgfMBcemidFXQO002urEvLDU70dYbyR949HUzA9LByJu+vds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=POYwD2pH; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="POYwD2pH" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0A4761596; Wed, 2 Sep 2026 23:07:13 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1104C3F673; Wed, 2 Sep 2026 23:07:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788415636; bh=RRCvk8AHwrIHgucZgMeo3d4bx61Rqtuocq+rKtw+8mg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=POYwD2pHXvaRovzg3w2novbaLV6mxbra4fzmXVQndnqcPZhve86LX/IlapM6s5yH8 EUcecmvxZqpITQkrexEGxv8S6W5XVZ6/y/5y+qnbwg1Cd3xmEx9G8WtOJqiQSZpU/X 6a/ICrlsqCvE41Mxqx5Ee06q6dG2rcaveCQaMW/o= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: [PATCH 4/4] arm64/mm: Convert READ_ONCE() as pgdp_get() while accessing PGD Date: Thu, 3 Sep 2026 11:36:57 +0530 Message-ID: <20260903060657.2717457-5-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260903060657.2717457-1-anshuman.khandual@arm.com> References: <20260903060657.2717457-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert all READ_ONCE() based PGD accesses as pgdp_get() instead which will support both D64 and D128 translation regime going forward. That is because READ_ONCE() would need 128 bit single copy atomic guarantees, while reading 128 bit page table entries which is currently not supported on arm64. Build fails for READ_ONCE() while accessing beyond 64 bits. Load Pair/Store Pair (ldp/stp) are only single copy atomic if FEAT_LSE128 is supported (which is required when FEAT_D128 is supported). Currently 128 bit pgtables is a compile time decision - so we could have chosen to extend READ_ONCE()/WRITE_ONCE() to allow 128 bit for this configuration. But then it's a general purpose API and we were concerned that other users might eventually creep in that expect 128 and then fail to compile in the other configs. But worse, we are considering eventually making D128 a boot time option, at which point we'd have to make READ_ONCE() always allow 128 bit at compile time but then it might silently tear at runtime. So our preference is to standardize on these existing helpers, which we can override in arm64 to give the 128 bit single copy guarantee when required. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Signed-off-by: Anshuman Khandual --- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/kasan_init.c | 6 +++--- arch/arm64/mm/mmu.c | 6 +++--- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 6 files changed, 11 insertions(+), 11 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 95d044cf04d4..1de76ced0403 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -160,7 +160,7 @@ static void show_pte(unsigned long addr) mm =3D=3D &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K, vabits_actual, mm_to_pgd_phys(mm)); pgdp =3D pgd_offset(mm, addr); - pgd =3D READ_ONCE(*pgdp); + pgd =3D pgdp_get(pgdp); ptval_to_str(pgd_str, pgd_val(pgd)); pr_alert("[%016lx] pgd=3D%s", addr, pgd_str); =20 diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 5771bf49e1fe..e05e5b41ae50 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -284,7 +284,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, pmd_t *pmdp, pmd; =20 pgdp =3D pgd_offset(mm, addr); - if (!pgd_present(READ_ONCE(*pgdp))) + if (!pgd_present(pgdp_get(pgdp))) return NULL; =20 p4dp =3D p4d_offset(pgdp, addr); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 4dd8c1186471..f61f5bc09467 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -102,7 +102,7 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsi= gned long addr, int node, static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int= node, bool early) { - if (pgd_none(READ_ONCE(*pgdp))) { + if (pgd_none(pgdp_get(pgdp))) { phys_addr_t p4d_phys =3D early ? __pa_symbol(kasan_early_shadow_p4d) : kasan_alloc_zeroed_page(node); @@ -256,7 +256,7 @@ static int __init root_level_idx(u64 addr) static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pu= d) { int idx =3D root_level_idx(addr); - pgd_t pgd =3D READ_ONCE(swapper_pg_dir[idx]); + pgd_t pgd =3D pgdp_get(swapper_pg_dir + idx); pud_t *pudp =3D (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); =20 memcpy(pud, pudp, PAGE_SIZE); @@ -280,7 +280,7 @@ static int __init next_level_idx(u64 addr) */ static void __init clear_next_level(int pgd_idx, int start, int end) { - pgd_t pgd =3D READ_ONCE(swapper_pg_dir[pgd_idx]); + pgd_t pgd =3D pgdp_get(swapper_pg_dir + pgd_idx); pud_t *pudp =3D (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); =20 memset(&pudp[start], 0, (end - start) * sizeof(pud_t)); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6f38e3618d26..46ff9a92e73a 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -434,7 +434,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long ad= dr, unsigned long end, { int ret; unsigned long next; - pgd_t pgd =3D READ_ONCE(*pgdp); + pgd_t pgd =3D pgdp_get(pgdp); p4d_t *p4dp; =20 if (pgd_none(pgd)) { @@ -1649,7 +1649,7 @@ static void unmap_hotplug_range(unsigned long addr, u= nsigned long end, do { next =3D pgd_addr_end(addr, end); pgdp =3D pgd_offset_k(addr); - pgd =3D READ_ONCE(*pgdp); + pgd =3D pgdp_get(pgdp); if (pgd_none(pgd)) continue; =20 @@ -1827,7 +1827,7 @@ static void free_empty_tables(unsigned long addr, uns= igned long end, do { next =3D pgd_addr_end(addr, end); pgdp =3D pgd_offset_k(addr); - pgd =3D READ_ONCE(*pgdp); + pgd =3D pgdp_get(pgdp); if (pgd_none(pgd)) continue; =20 diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 826856ef64a8..dfa1356daac2 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -399,7 +399,7 @@ bool kernel_page_present(struct page *page) unsigned long addr =3D (unsigned long)page_address(page); =20 pgdp =3D pgd_offset_k(addr); - if (pgd_none(READ_ONCE(*pgdp))) + if (pgd_none(pgdp_get(pgdp))) return false; =20 p4dp =3D p4d_offset(pgdp, addr); diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 7afe2beca4ba..06470d690f9f 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -134,7 +134,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t = *dst_pgdp, unsigned long next; unsigned long addr =3D start; =20 - if (pgd_none(READ_ONCE(*dst_pgdp))) { + if (pgd_none(pgdp_get(dst_pgdp))) { dst_p4dp =3D trans_alloc(info); if (!dst_p4dp) return -ENOMEM; @@ -164,7 +164,7 @@ static int copy_page_tables(struct trans_pgd_info *info= , pgd_t *dst_pgdp, dst_pgdp =3D pgd_offset_pgd(dst_pgdp, start); do { next =3D pgd_addr_end(addr, end); - if (pgd_none(READ_ONCE(*src_pgdp))) + if (pgd_none(pgdp_get(src_pgdp))) continue; if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next)) return -ENOMEM; --=20 2.43.0