From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6A1013F104F for ; Mon, 7 Sep 2026 03:50:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753046; cv=none; b=NZ0JOcLXn/TbBnCxVzu42OCFRIT7kKnviSNFuJ4atNIv+uB7QugMBfQqtQa9m3lymz7DoxwoQVO8+Vfg5iEjy42/Lzm/WNSQjEiuKwqdKwLd+ZEUUBp7DhjhN4cvZC8KxVoO4H+rzH34nCdaW0KYKoHYUkc5OCVkWJVxdXd938w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753046; c=relaxed/simple; bh=3yn6gSsOXRoyaBz1CARO0jyrfcLMBCzmt7csYw/Fkvk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QP+xN8urV70j90nu43w47WJvU0WMZb3gkOMq09d1cefPqqVRgoW8Oq6UQT7n1WSjgIQglTh7rM1XCxXzbte3mbZuQZuukjeVDowz3VGL6vpIuJORBfC9Eeqbsq1PhMw5GIGEsOt1D3rre0RY/it+h49YuPB/uJygedAmhptjyz4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=LW2jt5b0; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="LW2jt5b0" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 945811516; Sun, 6 Sep 2026 20:50:34 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 89B773F7B4; Sun, 6 Sep 2026 20:50:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753038; bh=3yn6gSsOXRoyaBz1CARO0jyrfcLMBCzmt7csYw/Fkvk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LW2jt5b0BLThiRONOynNfF1FBuy3s6vy0IP2y1G8Ho9rA3Rexh4127PHoDgS9FR98 ZDfgStQaqGpdqFi3z28eNV4z53rIyzJ5VwrpgP/WMoKM3iCdUA0Gjq3CKdnnc0WvyA DOD/Cgke+1nc0Oi/K3jQrqX5CVCODcbUyP+HNuEs= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lorenzo Stoakes Subject: [PATCH V2 01/14] mm: Add read-write accessors for vm_page_prot Date: Mon, 7 Sep 2026 09:20:16 +0530 Message-ID: <20260907035030.2839978-2-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently vma->vm_page_prot is safely read from and written to, without any locks with READ_ONCE() and WRITE_ONCE(). But with introduction of D128 page tables on arm64 platform, vm_page_prot grows to 128 bits which can't safely be handled with READ_ONCE() and WRITE_ONCE(). Add read and write accessors for vm_page_prot like pgprot_[read|write]() which any platform can override when required, although still defaulting as READ_ONCE() and WRITE_ONCE(), thus preserving the functionality for others. Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Mike Rapoport Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Acked-by: Mike Rapoport (Microsoft) Signed-off-by: Anshuman Khandual --- include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 4 ++-- mm/memory.c | 2 +- mm/migrate.c | 2 +- mm/mmap.c | 2 +- 5 files changed, 19 insertions(+), 5 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e3c8ab96941c..0df57a5c19ef 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -524,6 +524,20 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) } #endif =20 +#ifndef pgprot_read +static inline pgprot_t pgprot_read(pgprot_t *prot) +{ + return READ_ONCE(*prot); +} +#endif + +#ifndef pgprot_write +static inline void pgprot_write(pgprot_t *prot, pgprot_t val) +{ + WRITE_ONCE(*prot, val); +} +#endif + #ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline bool ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ced400f72d43..33753bffead4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3391,7 +3391,7 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, } else { pte_t entry; =20 - entry =3D mk_pte(page, READ_ONCE(vma->vm_page_prot)); + entry =3D mk_pte(page, pgprot_read(&vma->vm_page_prot)); if (write) entry =3D pte_mkwrite(entry, vma); if (!young) @@ -5103,7 +5103,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk= *pvmw, struct folio *folio =20 entry =3D softleaf_from_pmd(*pvmw->pmd); folio_get(folio); - pmde =3D folio_mk_pmd(folio, READ_ONCE(vma->vm_page_prot)); + pmde =3D folio_mk_pmd(folio, pgprot_read(&vma->vm_page_prot)); =20 if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde =3D pmd_mksoft_dirty(pmde); diff --git a/mm/memory.c b/mm/memory.c index aa6b2fc770fc..fcea6098efb2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -939,7 +939,7 @@ static void restore_exclusive_pte(struct vm_area_struct= *vma, =20 VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); =20 - pte =3D pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot))); + pte =3D pte_mkold(mk_pte(page, pgprot_read(&vma->vm_page_prot))); if (pte_swp_soft_dirty(orig_pte)) pte =3D pte_mksoft_dirty(pte); =20 diff --git a/mm/migrate.c b/mm/migrate.c index 15b45832bcfa..579152cce502 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -387,7 +387,7 @@ static bool remove_migration_pte(struct folio *folio, =20 folio_get(folio); new =3D folio_page(folio, idx); - pte =3D mk_pte(new, READ_ONCE(vma->vm_page_prot)); + pte =3D mk_pte(new, pgprot_read(&vma->vm_page_prot)); if (!softleaf_is_migration_young(entry)) pte =3D pte_mkold(pte); if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry)) diff --git a/mm/mmap.c b/mm/mmap.c index 4bf26b0f1e6e..729bc244b317 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -89,7 +89,7 @@ void vma_set_page_prot(struct vm_area_struct *vma) vm_page_prot =3D vma_pgprot_modify(vm_page_prot, vma_flags); } /* remove_protection_ptes reads vma->vm_page_prot without mmap_lock */ - WRITE_ONCE(vma->vm_page_prot, vm_page_prot); + pgprot_write(&vma->vm_page_prot, vm_page_prot); } =20 /* --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 912E13F0765 for ; Mon, 7 Sep 2026 03:50:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753045; cv=none; b=IeEkAgsmbA0MISkJTFUspUN1UUoYV3YSjraqFqBEh6ljUMszfok1R3+TL19qmSePhBH7S3wXdx45IKb9fqLfw7muED5Ogb5HhS6tpmYGcwtcIwpj0r9UbS4HUM3BqEoz2+wUFCq3gzjFCMg64wg4cU2wRYePTOnHmD+1Jh6OqRk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753045; c=relaxed/simple; bh=u0NC425vYyHxLP7hQ6qnyzz+8iyaNNOtn3y2ra/8hTA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M3Dk2rPzEIE2LBdAhwb/4qlB1zCuzE4mEvu1bYdCPMBRcfkgt8aMd2bRK22ncR+2+zebwKzorn8mea9g/JU0TbHata8isIFIdTYuqgM5gBC7r4BayNW98oDOE6y89ghqhuhH/nzpjLrYkRb/Yt+mLTJmb/brY3BWSl02MkzPhS8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=Hsxe98q8; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="Hsxe98q8" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 64D0D1BA8; Sun, 6 Sep 2026 20:50:38 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5CCC43F7B4; Sun, 6 Sep 2026 20:50:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753042; bh=u0NC425vYyHxLP7hQ6qnyzz+8iyaNNOtn3y2ra/8hTA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Hsxe98q84/3K+kWyhkJqja1wgsnGC3YeYfKTHgB+lv+0kg19pwmcx4WqvGbsmzzic 7c+COfU5ZnvixdveYZQPHnggpyxcqQm5iVzOhATDAyQ75LDL7OrBLrYkwvkpo89Pje x638FHhR/9el/45Wd5MBVq7S0rfhZ5JXSoFkoHw4= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 02/14] arm64/mm: Route all pgtable reads via ptval_get() Date: Mon, 7 Sep 2026 09:20:17 +0530 Message-ID: <20260907035030.2839978-3-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define arm64 platform specific implementations for new pxdp_get() helpers. These resolve into READ_ONCE(), thus ensuring required single copy atomic semantics for the page table entry reads. In future this infrastructure can be used for D128 to maintain single copy atomicity semantics with inline asm blocks. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual Reviewed-by: David Hildenbrand (Arm) --- arch/arm64/include/asm/pgtable.h | 38 +++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 83087b70f192..f764132ca34e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -84,6 +84,32 @@ static inline void arch_leave_lazy_mmu_mode(void) arch_flush_lazy_mmu_mode(); } =20 +#define ptval_get(x) READ_ONCE(x) + +#define pmdp_get pmdp_get +static inline pmd_t pmdp_get(pmd_t *pmdp) +{ + return ptval_get(*pmdp); +} + +#define pudp_get pudp_get +static inline pud_t pudp_get(pud_t *pudp) +{ + return ptval_get(*pudp); +} + +#define p4dp_get p4dp_get +static inline p4d_t p4dp_get(p4d_t *p4dp) +{ + return ptval_get(*p4dp); +} + +#define pgdp_get pgdp_get +static inline pgd_t pgdp_get(pgd_t *pgdp) +{ + return ptval_get(*pgdp); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE =20 @@ -381,7 +407,7 @@ static inline void __set_pte(pte_t *ptep, pte_t pte) =20 static inline pte_t __ptep_get(pte_t *ptep) { - return READ_ONCE(*ptep); + return ptval_get(*ptep); } =20 extern void __sync_icache_dcache(pte_t pteval); @@ -958,7 +984,7 @@ static inline phys_addr_t pud_offset_phys(p4d_t *p4dp, = unsigned long addr) { VM_WARN_ON_ONCE(!pgtable_l4_enabled()); =20 - return p4d_page_paddr(READ_ONCE(*p4dp)) + pud_index(addr) * sizeof(pud_t); + return p4d_page_paddr(p4dp_get(p4dp)) + pud_index(addr) * sizeof(pud_t); } =20 static inline @@ -972,7 +998,7 @@ pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsi= gned long addr) =20 static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) { - return pud_offset_lockless(p4dp, READ_ONCE(*p4dp), addr); + return pud_offset_lockless(p4dp, p4dp_get(p4dp), addr); } #define pud_offset pud_offset =20 @@ -1078,7 +1104,7 @@ static inline phys_addr_t p4d_offset_phys(pgd_t *pgdp= , unsigned long addr) { VM_WARN_ON_ONCE(!pgtable_l5_enabled()); =20 - return pgd_page_paddr(READ_ONCE(*pgdp)) + p4d_index(addr) * sizeof(p4d_t); + return pgd_page_paddr(pgdp_get(pgdp)) + p4d_index(addr) * sizeof(p4d_t); } =20 static inline @@ -1092,7 +1118,7 @@ p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, un= signed long addr) =20 static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) { - return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); + return p4d_offset_lockless(pgdp, pgdp_get(pgdp), addr); } =20 static inline p4d_t *p4d_set_fixmap(unsigned long addr) @@ -1287,7 +1313,7 @@ static inline bool pmdp_test_and_clear_young(struct v= m_area_struct *vma, unsigned long address, pmd_t *pmdp) { /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */ - VM_WARN_ON(pmd_table(READ_ONCE(*pmdp)) && !system_supports_haft()); + VM_WARN_ON(pmd_table(pmdp_get(pmdp)) && !system_supports_haft()); return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG= */ --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A18703ED3BA for ; Mon, 7 Sep 2026 03:50:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753051; cv=none; b=qoZhu8+pTJlTc+QpPQCGNVoEUl90sphZfl5HJfGzZRSxxb/gRGLTC8k1DzjiEbJ3JGz6DMGd6Z87+wMCNUnwfKZ3X1/BYAGkSvXCjL6h7XZFdPfDSXh53Jp+cquXenwFYVsIx1CBfcztEjTWbDnmPeMwB4qHVerySHr6iAzgVIE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753051; c=relaxed/simple; bh=mxuT7trp1ApTgSVX4oHgcsRIN1m6b9BgozIHOrKR1EA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JCxNCx6xFUYPRXVCzYu4mGyU/ozGCJfewOPAsMu89X/hX63r2+W5kfwsXJF4imZO+QzpaX2LVWAsUk9B6D/1B5hTZu0Ozj869obJmVOclliyFAtbNbO7M9/5bFgYcLeYqxO/RjSttXv10jtsbyeYuqpErvWfd9n3pYptKCKHt14= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=FY/dl/yz; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="FY/dl/yz" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 397EC1477; Sun, 6 Sep 2026 20:50:42 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2FA7E3F7B4; Sun, 6 Sep 2026 20:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753045; bh=mxuT7trp1ApTgSVX4oHgcsRIN1m6b9BgozIHOrKR1EA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FY/dl/yzk9KpGi6X+UbeUc4pMhX7cBqWiiGAe7E7rqPNhPI2JJmNjvO4oJOoANJwi e4/vDeO1cADKeq983cahV1Vwor9bXY8kPPJOPael7133LRm23AJ2le9iagrrKEuS4P J7Rt3c7sXHcfdFyA34V4X+bqRHvRLaq+iDaMtY14= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 03/14] arm64/mm: Route all pgtable writes via ptval_set() Date: Mon, 7 Sep 2026 09:20:18 +0530 Message-ID: <20260907035030.2839978-4-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently ptval_set() is defined as WRITE_ONCE() but this will change for D128 pgtable builds, for which WRITE_ONCE() is not sufficient for single copy atomicity. In future this infrastructure can be used for D128 to maintain single copy atomicity semantics with inline asm blocks. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual Reviewed-by: David Hildenbrand (Arm) --- arch/arm64/include/asm/pgtable.h | 11 ++++++----- arch/arm64/mm/mmu.c | 4 ++-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index f764132ca34e..692669ea2d29 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -85,6 +85,7 @@ static inline void arch_leave_lazy_mmu_mode(void) } =20 #define ptval_get(x) READ_ONCE(x) +#define ptval_set(x, val) WRITE_ONCE(x, val) =20 #define pmdp_get pmdp_get static inline pmd_t pmdp_get(pmd_t *pmdp) @@ -386,7 +387,7 @@ static inline pte_t pte_clear_uffd(pte_t pte) =20 static inline void __set_pte_nosync(pte_t *ptep, pte_t pte) { - WRITE_ONCE(*ptep, pte); + ptval_set(*ptep, pte); } =20 static inline void __set_pte_complete(pte_t pte) @@ -809,7 +810,7 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) } #endif /* __PAGETABLE_PMD_FOLDED */ =20 - WRITE_ONCE(*pmdp, pmd); + ptval_set(*pmdp, pmd); =20 if (pmd_valid(pmd)) queue_pte_barriers(); @@ -871,7 +872,7 @@ static inline void set_pud(pud_t *pudp, pud_t pud) return; } =20 - WRITE_ONCE(*pudp, pud); + ptval_set(*pudp, pud); =20 if (pud_valid(pud)) queue_pte_barriers(); @@ -950,7 +951,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) return; } =20 - WRITE_ONCE(*p4dp, p4d); + ptval_set(*p4dp, p4d); queue_pte_barriers(); } =20 @@ -1075,7 +1076,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) return; } =20 - WRITE_ONCE(*pgdp, pgd); + ptval_set(*pgdp, pgd); queue_pte_barriers(); } =20 diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 46ff9a92e73a..44af53dc9ba8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -75,7 +75,7 @@ void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) * writable in the kernel mapping. */ if (rodata_is_rw) { - WRITE_ONCE(*pgdp, pgd); + ptval_set(*pgdp, pgd); dsb(ishst); isb(); return; @@ -83,7 +83,7 @@ void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) =20 spin_lock(&swapper_pgdir_lock); fixmap_pgdp =3D pgd_set_fixmap(__pa_symbol(pgdp)); - WRITE_ONCE(*fixmap_pgdp, pgd); + ptval_set(*fixmap_pgdp, pgd); /* * We need dsb(ishst) here to ensure the page-table-walker sees * our new entry before set_p?d() returns. The fixmap's --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 762713EFFDB for ; Mon, 7 Sep 2026 03:50:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753052; cv=none; b=b6OFkTwhfzm+MPv6O+a3BvjG48qeZX5vIgtu6qJJMDPLIifKQFg9WA/J5gRFUbBG9ZpYtA9ADvA0wwWQOEuXlDKzuGdxFowCN02qwD4OWjaOFOpQ93PkS7zGajZAWvJpBOPA1naZJRwOl4mE4pg4O8HaeFGCEfOfASyYIVzDnGw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753052; c=relaxed/simple; bh=datmweSr9Wz8l6EajpJTnuP4rYWsCtPHvC4W2LbaUaA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Gg9vQKsF4umDM8oQHwYzOKXaCiInlVEg8L+pj6Adw68D8LXvTXXn9/RYu/Ftf0EkKLcLinqMUj0DiBfjL/w+YL5rFKnEZBRcTQ1EDH+IZIgy4Vy0oeeSEPQ0yNPqczdL7jy6vH/d3I9/Q1UcmdeZ8imjvjPD10wl+sVnkzJi88k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=rYQQ5NTw; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="rYQQ5NTw" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0B5CB1516; Sun, 6 Sep 2026 20:50:46 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 01ABA3F7B4; Sun, 6 Sep 2026 20:50:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753049; bh=datmweSr9Wz8l6EajpJTnuP4rYWsCtPHvC4W2LbaUaA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rYQQ5NTw++2T/tMCvMfnjCBpqab+/MaKV87YWGvK+K+0ywMBjSFsvu2yZ/L8K+tPL 1buRRth44mADdhDZXsthTzNm91mt0BP1+rChGY2IHHTaLtQo/F4AZzRMIFpaMhPeNj GwSgJ9LCfd+pEY6KoBVJvR4Y2q1BdnGGtQiZTnOA= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 04/14] arm64/mm: Route all pgtable atomics to central helpers Date: Mon, 7 Sep 2026 09:20:19 +0530 Message-ID: <20260907035030.2839978-5-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Route all cmpxchg() operations performed on various page table entries to a new ptval_cmpxchg_relaxed() helper. Similarly route all xchg() operations performed on page table entries to a new ptval_xchg_relaxed() helper. Currently these helpers just forward to the same APIs that were previously called direct, but in future we will change the routing for D128 which is too long to use the standard APIs. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual Reviewed-by: David Hildenbrand (Arm) --- arch/arm64/include/asm/pgtable.h | 26 +++++++++++++++++++------- arch/arm64/mm/fault.c | 2 +- 2 files changed, 20 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 692669ea2d29..66bf5bbe484e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -87,6 +87,17 @@ static inline void arch_leave_lazy_mmu_mode(void) #define ptval_get(x) READ_ONCE(x) #define ptval_set(x, val) WRITE_ONCE(x, val) =20 +static inline ptval_t ptval_cmpxchg_relaxed(ptval_t *ptep, ptval_t old, + ptval_t new) +{ + return cmpxchg_relaxed(ptep, old, new); +} + +static inline ptval_t ptval_xchg_relaxed(ptval_t *ptep, ptval_t new) +{ + return xchg_relaxed(ptep, new); +} + #define pmdp_get pmdp_get static inline pmd_t pmdp_get(pmd_t *pmdp) { @@ -1281,8 +1292,8 @@ static inline bool __ptep_test_and_clear_young(struct= vm_area_struct *vma, do { old_pte =3D pte; pte =3D pte_mkold(pte); - pte_val(pte) =3D cmpxchg_relaxed(&pte_val(*ptep), - pte_val(old_pte), pte_val(pte)); + pte_val(pte) =3D ptval_cmpxchg_relaxed(&pte_val(*ptep), + pte_val(old_pte), pte_val(pte)); } while (pte_val(pte) !=3D pte_val(old_pte)); =20 return pte_young(pte); @@ -1324,7 +1335,7 @@ static inline pte_t __ptep_get_and_clear_anysz(struct= mm_struct *mm, pte_t *ptep, unsigned long pgsize) { - pte_t pte =3D __pte(xchg_relaxed(&pte_val(*ptep), 0)); + pte_t pte =3D __pte(ptval_xchg_relaxed(&pte_val(*ptep), 0)); =20 switch (pgsize) { case PAGE_SIZE: @@ -1400,7 +1411,7 @@ static inline void ___ptep_set_wrprotect(struct mm_st= ruct *mm, do { old_pte =3D pte; pte =3D pte_wrprotect(pte); - pte_val(pte) =3D cmpxchg_relaxed(&pte_val(*ptep), + pte_val(pte) =3D ptval_cmpxchg_relaxed(&pte_val(*ptep), pte_val(old_pte), pte_val(pte)); } while (pte_val(pte) !=3D pte_val(old_pte)); } @@ -1438,7 +1449,7 @@ static inline void __clear_young_dirty_pte(struct vm_= area_struct *vma, if (flags & CYDP_CLEAR_DIRTY) pte =3D pte_mkclean(pte); =20 - pte_val(pte) =3D cmpxchg_relaxed(&pte_val(*ptep), + pte_val(pte) =3D ptval_cmpxchg_relaxed(&pte_val(*ptep), pte_val(old_pte), pte_val(pte)); } while (pte_val(pte) !=3D pte_val(old_pte)); } @@ -1477,7 +1488,7 @@ static inline pmd_t pmdp_establish(struct vm_area_str= uct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) { page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); - return __pmd(xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd))); + return __pmd(ptval_xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd))); } #endif =20 @@ -1804,7 +1815,8 @@ static inline bool ptep_try_set(pte_t *ptep, pte_t ne= w_pte) { pteval_t old =3D 0; =20 - if (!try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte))) + old =3D ptval_cmpxchg_relaxed(&pte_val(*ptep), old, pte_val(new_pte)); + if (old !=3D 0) return false; =20 /* diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 1de76ced0403..6db93371d649 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -244,7 +244,7 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct= *vma, pteval ^=3D PTE_RDONLY; pteval |=3D pte_val(entry); pteval ^=3D PTE_RDONLY; - pteval =3D cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval); + pteval =3D ptval_cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval); } while (pteval !=3D old_pteval); =20 /* --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6E61B3E9F93 for ; Mon, 7 Sep 2026 03:50:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753058; cv=none; b=HihgL1Cmm953lWxzRzu7iXVHyOajBNdvVnDymfJKnfO9p8NWn6B0FO62xZfraZPMnvnsUYlmmk5nossJViavaVvHlAir8b7xQ+NQEjOTWUaORZs6xZv53ZE4pKwsjLofD9HrnWjMqcvVNvrCQeg9p0AlZg/GlJubzJHvPdzzWy8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753058; c=relaxed/simple; bh=/rjGzAt748dsaT36LBnwBIEZW4gfjkEspNbbmVlfxKY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=imMWYJzUjziH8GmvG/iRpH0qoatYBXKMO0ynKI045LAhv4ZtxQNsZNzljGsq7qZpspXDY47Hs48zoLkAqbbvIWYKnUtCxs8FDDzS2+94vMNcSJIMIfEPnJlrSAMynyu95wmqF5hTSNcpJ3rtgv6fteay0TmSuCDc0xsdNWy63pM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=sMy811HE; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="sMy811HE" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D5DE61477; Sun, 6 Sep 2026 20:50:49 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CB6B53F7B4; Sun, 6 Sep 2026 20:50:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753053; bh=/rjGzAt748dsaT36LBnwBIEZW4gfjkEspNbbmVlfxKY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sMy811HEo7xsqsm1p6JcAzLBb1J9PIdm3/h3H35Mn7WqMv0uYuDIVHpZ3IQqhy+bA sWeT+NJFvEID6NFJBlSkLv1xwq/uGZiqo3RpydBbivEGHiwYUfOgUUsd4NohfsU0vo apZdfIn31OFzbtwwAWTrajkO9WPqqLFAmPjA/4BU= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 05/14] arm64/mm: Override read-write accessors for vm_page_prot Date: Mon, 7 Sep 2026 09:20:20 +0530 Message-ID: <20260907035030.2839978-6-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Override pgprot_[read|write]() accessors using ptval_[get|set]() providing required single copy atomic operation for vma->vm_page_prot. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual Reviewed-by: David Hildenbrand (Arm) --- arch/arm64/include/asm/pgtable.h | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 66bf5bbe484e..2b457db19be0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -122,6 +122,18 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) return ptval_get(*pgdp); } =20 +#define pgprot_read pgprot_read +static inline pgprot_t pgprot_read(pgprot_t *prot) +{ + return ptval_get(*prot); +} + +#define pgprot_write pgprot_write +static inline void pgprot_write(pgprot_t *prot, pgprot_t val) +{ + ptval_set(*prot, val); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE =20 --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CB4FD233941 for ; Mon, 7 Sep 2026 03:50:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753061; cv=none; b=s1MQypnYtnq5B4XcE2tc9GQCorQQBbt0flq6oC9BXECrBZLHcizQZ4PN5D1wS6GDNB2WiJ7Cn4dYhpMUmpoYvD4r8vX/bTxrjfcUSb9f8tBIoO8icPPrWf3h42ZQxjaswqgnLkBG1WtBIpSM3xOYP8xd3GNm0ImLI3C998RiVNc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753061; c=relaxed/simple; bh=qFNA6gQd3R3/N1F2NlRNUpaGJ73aKIkK2svQU86ktVQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dYZiOHrYbCMTFDFvJzf6annT5OQF1TvqAave7WnUe2HUa+QpjPfsWpiMn0dvQp7glRf8z1djqX3Yzx5bAFGwhLML2/Q0MFTgdi8/Xb7Cv+fztPdADTkq6ROwiJ5k8+YqqUBW4qnrBc/ROWzlYMxTgAL7dzwjQR0yfFhGHFeoXN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=gaU3xvQi; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="gaU3xvQi" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AA7631AED; Sun, 6 Sep 2026 20:50:53 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A13643F7B4; Sun, 6 Sep 2026 20:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753057; bh=qFNA6gQd3R3/N1F2NlRNUpaGJ73aKIkK2svQU86ktVQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gaU3xvQiDu0CvOWTMdlZQxYVQh/R/CHFyGgjq84fexhBOg1L9rxB+4w8ipcOCDa1Z E2Cs2eyqqBmiDulIGwNs832s30G8mh9bsgmEeCfvO6JqFbe7FWVnXIuS+NfF1Ekr8J yuvc0+cRZZKX6wwRSyT/+18yuMQL5X7/LgDl8Vkg= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 06/14] arm64/mm: Enable fixmap with 5 level page table Date: Mon, 7 Sep 2026 09:20:21 +0530 Message-ID: <20260907035030.2839978-7-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" FEAT_D128 halves PTRS_PER_PXX thus shrinking the VA range coverage for each page table level. Hence in order to preserve all existing VA range configurations, some geometry now need to become 5-level. Since fixmap is used to build and manipulate page tables early on during boot the mapping must also gain that additional level which was not required earlier. Enable fixmap with 5 level page table when required. This creates table entries at the PGD level. Add a fallback stub for pgd_page_paddr() when (PGTBALE_LEVELS <=3D 4) which helps in intercepting any unintended usage. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgtable.h | 1 + arch/arm64/mm/fixmap.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 2b457db19be0..7a5d6a14bb52 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1178,6 +1178,7 @@ static inline p4d_t *p4d_offset_kimg(pgd_t *pgdp, u64= addr) #else =20 static inline bool pgtable_l5_enabled(void) { return false; } +#define pgd_page_paddr(pgd) ({ BUILD_BUG(); 0; }) =20 #define p4d_index(addr) (((addr) >> P4D_SHIFT) & (PTRS_PER_P4D - 1)) =20 diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 2a0fe0b0f771..5512521fef8f 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -34,6 +34,7 @@ static_assert(NR_BM_PMD_TABLES =3D=3D 1); static pte_t bm_pte[NR_BM_PTE_TABLES][PTRS_PER_PTE] __bss_pgtbl; static pmd_t bm_pmd[PTRS_PER_PMD] __bss_pgtbl __maybe_unused; static pud_t bm_pud[PTRS_PER_PUD] __bss_pgtbl __maybe_unused; +static p4d_t bm_p4d[PTRS_PER_P4D] __bss_pgtbl __maybe_unused; =20 static inline pte_t *fixmap_pte(unsigned long addr) { @@ -95,6 +96,19 @@ static void __init early_fixmap_init_pud(p4d_t *p4dp, un= signed long addr, early_fixmap_init_pmd(pudp, addr, end); } =20 +static void __init early_fixmap_init_p4d(pgd_t *pgdp, unsigned long addr, + unsigned long end) +{ + pgd_t pgd =3D pgdp_get(pgdp); + p4d_t *p4dp; + + if (pgd_none(pgd)) + __pgd_populate(pgdp, __pa_symbol(bm_p4d), + PGD_TYPE_TABLE | PGD_TABLE_AF); + p4dp =3D p4d_offset_kimg(pgdp, addr); + early_fixmap_init_pud(p4dp, addr, end); +} + /* * The p*d_populate functions call virt_to_phys implicitly so they can't b= e used * directly on kernel symbols (bm_p*d). This function is called too early = to use @@ -107,9 +121,7 @@ void __init early_fixmap_init(void) unsigned long end =3D FIXADDR_TOP; =20 pgd_t *pgdp =3D pgd_offset_k(addr); - p4d_t *p4dp =3D p4d_offset_kimg(pgdp, addr); - - early_fixmap_init_pud(p4dp, addr, end); + early_fixmap_init_p4d(pgdp, addr, end); } =20 /* --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C61733F1057 for ; Mon, 7 Sep 2026 03:51:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753063; cv=none; b=Xje/+GEOs/CarTpre6fabPt/HJdLdmpLEfH127B+6Oz8dAh1qyTT9hl7XyoTpnkA0oRL4e/zZ7Hqk9osVTCMMCUCe3BPIJknVdiCdpZHLwApPmKSZVCjrkfCIAIDx8XuqO/Kx1PJKSVmQsOkOsXOuSgNZ2oFZsi6BQK2w5Unkf0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753063; c=relaxed/simple; bh=6wa5MsZMcn+Z3nxdy5kbv2HZNFuqAlAuRxAc1QrQTkI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QSJx5km/+1NR4WaEr1sPCt8uDKKhPmJfmj7divz6gkgOJ/+HbGq4S5NZX7O9oEiIomo4EB+ugn64L7vWzHQbtp1z/5xXmekpUx+2Z3MFG+/vOzVMAP2TEGq3bfXQ/HYltbYqNR7H8nolLs/4i8CNA9rKeOLNFH0jk2RIWZhCubs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=fl8VqLvm; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="fl8VqLvm" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7FD881477; Sun, 6 Sep 2026 20:50:57 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7793E3F7B4; Sun, 6 Sep 2026 20:51:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753061; bh=6wa5MsZMcn+Z3nxdy5kbv2HZNFuqAlAuRxAc1QrQTkI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fl8VqLvmTQZwKkcHorxP+8qDbRaaOF5rpdgiICuln2/D8a52ywJZ2eGm7xYrWI17b lPolpJsYANXrx5502C20P7O6olCTgsRw8X+lqFvgMPBR7Kq/CtPR4KZhbgShkCAeNg 832RVk5IAZkPgsN2eikujbI3fztFJ587Xr01fJzE= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 07/14] arm64/mm: Enable pgtable geometry for FEAT_D128 Date: Mon, 7 Sep 2026 09:20:22 +0530 Message-ID: <20260907035030.2839978-8-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add build time support for FEAT_D128 page tables with a new Kconfig option i.e CONFIG_ARM64_D128. When selected, PTE types become 128 bits wide and PTE bits are mapped to their new locations. Besides the basic page table geometry is also updated since each table page now holds half the number of entries (aka PTRS_PER_PXX) as it did previously. Similarly update the geometry relevant macros ARM64_CONT_[PTE|PMD]_SHIFT, PGTABLE_LEVELS, and PTDESC_ORDER. Although for now config ARM64_D128 remains disabled while all the required dependencies are getting created in subsequent patches. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- arch/arm64/Kconfig | 37 ++++++++++++++++++++++---- arch/arm64/include/asm/pgtable-hwdef.h | 4 +++ arch/arm64/include/asm/pgtable.h | 2 ++ 3 files changed, 38 insertions(+), 5 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b5a51b0ef944..306e6c803471 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -284,14 +284,18 @@ config MMU =20 config ARM64_CONT_PTE_SHIFT int - default 5 if PAGE_SIZE_64KB - default 7 if PAGE_SIZE_16KB + default 4 if PAGE_SIZE_64KB && ARM64_D128 + default 5 if PAGE_SIZE_64KB && !ARM64_D128 + default 6 if PAGE_SIZE_16KB && ARM64_D128 + default 7 if PAGE_SIZE_16KB && !ARM64_D128 default 4 =20 config ARM64_CONT_PMD_SHIFT int - default 5 if PAGE_SIZE_64KB - default 5 if PAGE_SIZE_16KB + default 6 if PAGE_SIZE_64KB && ARM64_D128 + default 5 if PAGE_SIZE_64KB && !ARM64_D128 + default 4 if PAGE_SIZE_16KB && ARM64_D128 + default 5 if PAGE_SIZE_16KB && !ARM64_D128 default 4 =20 config ARCH_MMAP_RND_BITS_MIN @@ -362,6 +366,16 @@ config FIX_EARLYCON_MEM =20 config PGTABLE_LEVELS int + default 4 if ARM64_D128 && ARM64_4K_PAGES && ARM64_VA_BITS_39 + default 5 if ARM64_D128 && ARM64_4K_PAGES && ARM64_VA_BITS_48 + default 5 if ARM64_D128 && ARM64_4K_PAGES && ARM64_VA_BITS_52 + default 3 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_36 + default 4 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_47 + default 4 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_48 + default 4 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_52 + default 3 if ARM64_D128 && ARM64_64K_PAGES && ARM64_VA_BITS_42 + default 3 if ARM64_D128 && ARM64_64K_PAGES && ARM64_VA_BITS_48 + default 3 if ARM64_D128 && ARM64_64K_PAGES && ARM64_VA_BITS_52 default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36 default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42 default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) @@ -1594,7 +1608,7 @@ config ARM64_PA_BITS =20 config ARM64_LPA2 def_bool y - depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES + depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES && !ARM64_D128 =20 choice prompt "Endianness" @@ -2289,6 +2303,19 @@ config ARM64_HAFT =20 endmenu # "ARMv8.9 architectural features" =20 +menu "ARMv9.3 architectural features" + +config ARM64_D128 + bool + default n + help + ARMv9.3 introduces FEAT_D128, which provides a 128 bit page + table format, along with related instructions. + + If unsure, say Y. + +endmenu # "ARMv9.3 architectural features" + menu "ARMv9.4 architectural features" =20 config ARM64_GCS diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/as= m/pgtable-hwdef.h index 72f31800c703..1ee1a6d9db6d 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -7,7 +7,11 @@ =20 #include =20 +#ifdef CONFIG_ARM64_D128 +#define PTDESC_ORDER 4 +#else #define PTDESC_ORDER 3 +#endif =20 /* Number of VA bits resolved by a single translation table level */ #define PTDESC_TABLE_SHIFT (PAGE_SHIFT - PTDESC_ORDER) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 7a5d6a14bb52..4b534be3f288 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1075,6 +1075,8 @@ static inline bool pgtable_l4_enabled(void) { return = false; } =20 static __always_inline bool pgtable_l5_enabled(void) { + if (IS_ENABLED(CONFIG_ARM64_D128)) + return true; if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) return vabits_actual =3D=3D VA_BITS; return alternative_has_cap_unlikely(ARM64_HAS_VA52); --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A825E3F106B for ; Mon, 7 Sep 2026 03:51:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753068; cv=none; b=S6An5F8LGzAd4CSaQym4qDb14OKkKWBvX88jFlCKewtzcTPdpwRSz0Uq+ru+XQUDCI2wI4eavmhbu+ZJXsy0zaJ7It9dzOw80ZGmgPmb7eVJ4OnFusYJcmNc++JNk9ma+9YBlDSDH7DQyOEbGc5vIOWBjdoCtMSVchalUE4uIZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753068; c=relaxed/simple; bh=+tgSDgH6S7J+Zg6W+yYzI/2ILAKRsH2eO7aDyQDfgW4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d1uQMQd+VbIpN8E75/MUy991Wrb2upH3OdNnI/vs7C2ewyIoH9JUOE11PgkSzlnIXmJvUWiQyvYqh5nBwSWDPFrXqmyuhgv5jdAHcHhM/Wo02DZsFDLeuMykuXOygiKl1nq8XoeLbDpsk5uSb1GuyuS036cgL4x68i/k0v64gE0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=gqhIIhSV; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="gqhIIhSV" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 54CED1477; Sun, 6 Sep 2026 20:51:01 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4D7F83F7B4; Sun, 6 Sep 2026 20:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753065; bh=+tgSDgH6S7J+Zg6W+yYzI/2ILAKRsH2eO7aDyQDfgW4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gqhIIhSVw4zQJJQkhhusDab2g/XO6q3GBktFLRdavIFx9RKYlLKdI2sZ3MKJ/eEkF HKjtOAfI324zpGwNyHTXwyr5oR2BTGFEvqYTFzpfLSDZ2EocS4q6Zpt5tNGlFUxJV1 7Jh9ACpfVGd3Kylj8zsPjCzJUf/kIa9RJrkPthuY= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 08/14] arm64/mm: Enable pgtable descriptor for FEAT_D128 Date: Mon, 7 Sep 2026 09:20:23 +0530 Message-ID: <20260907035030.2839978-9-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable pgtable descriptor fields and related accessors for FEAT_D128. First expand ptval_t as 128 bits, add all D128 specific pgtable descriptor fields and update accessors like ptval_get/set() and ptval_xchg_relaxed(). Also do ensure that ARM64_PA_BITS_52 specific handlings are not applicable for D128 which does not follow PA bits folding mechanisms as in FEAT_[LPA|LPA2]. Although all the above changes gets enabled only with CONFIG_ARM64_D128. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/assembler.h | 4 +- arch/arm64/include/asm/pgtable-hwdef.h | 133 +++++++++++++++++++++++++ arch/arm64/include/asm/pgtable-prot.h | 20 +++- arch/arm64/include/asm/pgtable-types.h | 4 + arch/arm64/include/asm/pgtable.h | 52 +++++++++- arch/arm64/mm/proc.S | 2 +- 6 files changed, 208 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/as= sembler.h index 0b58b550e8dc..1402f90f0030 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -623,7 +623,7 @@ alternative_else_nop_endif * ttbr: returns the TTBR value */ .macro phys_to_ttbr, ttbr, phys -#ifdef CONFIG_ARM64_PA_BITS_52 +#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128) orr \ttbr, \phys, \phys, lsr #46 and \ttbr, \ttbr, #TTBR_BADDR_MASK_52 #else @@ -632,7 +632,7 @@ alternative_else_nop_endif .endm =20 .macro phys_to_pte, pte, phys -#ifdef CONFIG_ARM64_PA_BITS_52 +#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128) orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/as= m/pgtable-hwdef.h index 1ee1a6d9db6d..16fb74c47b74 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -101,6 +101,137 @@ #define CONT_PMD_SIZE (CONT_PMDS * PMD_SIZE) #define CONT_PMD_MASK (~(CONT_PMD_SIZE - 1)) =20 +#ifdef CONFIG_ARM64_D128 + +/* + * Hardware page table definitions. + * + * Level -1 descriptor (PGD). + */ +#define PGD_SKL_SHIFT 109 +#define PGD_SKL_MASK GENMASK_U128(110, 109) +#define PGD_SKL_TABLE (_AT(pgdval_t, 0) << PGD_SKL_SHIFT) + +#define PGD_TYPE_TABLE _AT(pgdval_t, (PTE_VALID | PGD_SKL_TABLE)) +#define PGD_TYPE_MASK _AT(pgdval_t, (PTE_VALID | PGD_SKL_MASK)) +#define PGD_TABLE_AF (_AT(pgdval_t, 1) << 10) /* Ignored if no FEAT_HAFT = */ +#define PGD_TABLE_PXN _AT(pgdval_t, 0) /* Not supported for D128 */ +#define PGD_TABLE_UXN _AT(pgdval_t, 0) /* Not supported for D128 */ + +/* + * Level 0 descriptor (P4D). + */ +#define P4D_SKL_SHIFT 109 +#define P4D_SKL_MASK GENMASK_U128(110, 109) +#define P4D_SKL_TABLE (_AT(p4dval_t, 0) << P4D_SKL_SHIFT) +#define P4D_SKL_SECT (_AT(p4dval_t, 3) << P4D_SKL_SHIFT) + +#define P4D_TYPE_TABLE _AT(p4dval_t, (PTE_VALID | P4D_SKL_TABLE)) +#define P4D_TYPE_MASK _AT(p4dval_t, (PTE_VALID | P4D_SKL_MASK)) +#define P4D_TYPE_SECT _AT(p4dval_t, (PTE_VALID | P4D_SKL_SECT)) +#define P4D_SECT_RDONLY (_AT(p4dval_t, 1) << 7) /* nDirty */ +#define P4D_TABLE_AF (_AT(p4dval_t, 1) << 10) /* Ignored if no FEAT_HAFT = */ +#define P4D_TABLE_PXN _AT(p4dval_t, 0) /* Not supported for D128 */ +#define P4D_TABLE_UXN _AT(p4dval_t, 0) /* Not supported for D128 */ + +/* + * Level 1 descriptor (PUD). + */ +#define PUD_SKL_SHIFT 109 +#define PUD_SKL_MASK GENMASK_U128(110, 109) +#define PUD_SKL_TABLE (_AT(pudval_t, 0) << PUD_SKL_SHIFT) +#define PUD_SKL_SECT (_AT(pudval_t, 2) << PUD_SKL_SHIFT) + +#define PUD_TYPE_TABLE _AT(pudval_t, (PTE_VALID | PUD_SKL_TABLE)) +#define PUD_TYPE_MASK _AT(pudval_t, (PTE_VALID | PUD_SKL_MASK)) +#define PUD_TYPE_SECT _AT(pudval_t, (PTE_VALID | PUD_SKL_SECT)) +#define PUD_SECT_RDONLY (_AT(pudval_t, 1) << 7) /* nDirty */ +#define PUD_TABLE_AF (_AT(pudval_t, 1) << 10) /* Ignored if no FEAT_HAFT = */ +#define PUD_TABLE_PXN _AT(pudval_t, 0) /* Not supported for D128 */ +#define PUD_TABLE_UXN _AT(pudval_t, 0) /* Not supported for D128 */ + +/* + * Level 2 descriptor (PMD). + */ +#define PMD_SKL_SHIFT 109 +#define PMD_SKL_MASK GENMASK_U128(110, 109) +#define PMD_SKL_TABLE (_AT(pmdval_t, 0) << PMD_SKL_SHIFT) +#define PMD_SKL_SECT (_AT(pmdval_t, 1) << PMD_SKL_SHIFT) + +#define PMD_TYPE_MASK _AT(pmdval_t, (PTE_VALID | PMD_SKL_MASK)) +#define PMD_TYPE_TABLE _AT(pmdval_t, (PTE_VALID | PMD_SKL_TABLE)) +#define PMD_TYPE_SECT _AT(pmdval_t, (PTE_VALID | PMD_SKL_SECT)) +#define PMD_TABLE_AF (_AT(pmdval_t, 1) << 10) /* Ignored if no FEAT_HAFT = */ +#define PMD_TABLE_PXN _AT(pmdval_t, 0) /* Not supported for D128 */ +#define PMD_TABLE_UXN _AT(pmdval_t, 0) /* Not supported for D128 */ + +/* + * Section + */ +#define PMD_SECT_USER (_AT(pmdval_t, 1) << 115) /* PIIndex[0] */ +#define PMD_SECT_RDONLY (_AT(pmdval_t, 1) << 7) /* nDirty */ +#define PMD_SECT_S (_AT(pmdval_t, 3) << 8) +#define PMD_SECT_AF (_AT(pmdval_t, 1) << 10) +#define PMD_SECT_NG (_AT(pmdval_t, 1) << 11) +#define PMD_SECT_CONT (_AT(pmdval_t, 1) << 111) +#define PMD_SECT_PXN (_AT(pmdval_t, 1) << 117) /* PIIndex[2] */ +#define PMD_SECT_UXN (_AT(pmdval_t, 1) << 118) /* PIIndex[3] */ + +/* + * AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registe= rs). + */ +#define PMD_ATTRINDX(t) (_AT(pmdval_t, (t)) << 2) +#define PMD_ATTRINDX_MASK (_AT(pmdval_t, 7) << 2) + +/* + * Level 3 descriptor (PTE). + */ +#define PTE_SKL_SHIFT 109 +#define PTE_SKL_MASK GENMASK_U128(110, 109) +#define PTE_SKL_SECT (_AT(pteval_t, 0) << PTE_SKL_SHIFT) + +#define PTE_VALID (_AT(pteval_t, 1) << 0) +#define PTE_TYPE_MASK _AT(pteval_t, (PTE_VALID | PTE_SKL_MASK)) +#define PTE_TYPE_PAGE _AT(pteval_t, (PTE_VALID | PTE_SKL_SECT)) +#define PTE_USER (_AT(pteval_t, 1) << 115) /* PIIndex[0] */ +#define PTE_RDONLY (_AT(pteval_t, 1) << 7) /* nDirty */ +#define PTE_SHARED (_AT(pteval_t, 3) << 8) /* SH[1:0], inner shareable */ +#define PTE_AF (_AT(pteval_t, 1) << 10) /* Access Flag */ +#define PTE_NG (_AT(pteval_t, 1) << 11) /* nG */ +#define PTE_GP (_AT(pteval_t, 1) << 113) /* BTI guarded */ +#define PTE_DBM (_AT(pteval_t, 1) << 116) /* PIIndex[1] */ +#define PTE_CONT (_AT(pteval_t, 1) << 111) /* Contiguous range */ +#define PTE_PXN (_AT(pteval_t, 1) << 117) /* PIIndex[2] */ +#define PTE_UXN (_AT(pteval_t, 1) << 118) /* PIIndex[3] */ +#define PTE_SWBITS_MASK _AT(pteval_t, GENMASK_U128(100, 91)) + +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (55 - PAGE_SHIFT)) - 1) << PA= GE_SHIFT) + +/* + * AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registe= rs). + */ +#define PTE_ATTRINDX(t) (_AT(pteval_t, (t)) << 2) +#define PTE_ATTRINDX_MASK (_AT(pteval_t, 7) << 2) + +/* + * PIIndex[3:0] encoding (Permission Indirection Extension) + */ +#define PTE_PI_MASK GENMASK_U128(118, 115) +#define PTE_PI_SHIFT 115 + +/* + * POIndex[3:0] encoding (Permission Overlay Extension) + */ +#define PTE_PO_IDX_0 (_AT(pteval_t, 1) << 121) +#define PTE_PO_IDX_1 (_AT(pteval_t, 1) << 122) +#define PTE_PO_IDX_2 (_AT(pteval_t, 1) << 123) +#define PTE_PO_IDX_3 (_AT(pteval_t, 1) << 124) + +#define PTE_PO_IDX_MASK GENMASK_U128(124, 121) +#define PTE_PO_IDX_SHIFT 121 + +#else /* !CONFIG_ARM64_D128 */ + /* * Hardware page table definitions. * @@ -215,7 +346,9 @@ #define PTE_PO_IDX_2 (_AT(pteval_t, 1) << 62) =20 #define PTE_PO_IDX_MASK GENMASK_ULL(62, 60) +#define PTE_PO_IDX_SHIFT 60 =20 +#endif /* CONFIG_ARM64_D128 */ =20 /* * Memory Attribute override for Stage-2 (MemAttr[3:0]) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm= /pgtable-prot.h index 09d7c00cf405..f7df22f222b2 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -13,10 +13,15 @@ /* * Software defined PTE bits definition. */ -#define PTE_WRITE (PTE_DBM) /* same as DBM (51) */ +#define PTE_WRITE (PTE_DBM) /* same as DBM (51 / 116) */ #define PTE_SWP_EXCLUSIVE (_AT(pteval_t, 1) << 2) /* only for swp ptes */ +#ifdef CONFIG_ARM64_D128 +#define PTE_DIRTY (_AT(pteval_t, 1) << 91) +#define PTE_SPECIAL (_AT(pteval_t, 1) << 92) +#else #define PTE_DIRTY (_AT(pteval_t, 1) << 55) #define PTE_SPECIAL (_AT(pteval_t, 1) << 56) +#endif =20 /* * PTE_PRESENT_INVALID=3D1 & PTE_VALID=3D0 indicates that the pte's fields= should be @@ -28,7 +33,11 @@ #define PTE_PRESENT_VALID_KERNEL (PTE_VALID | PTE_MAYBE_NG) =20 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP -#define PTE_UFFD (_AT(pteval_t, 1) << 58) /* userfaultfd tracking */ +#ifdef CONFIG_ARM64_D128 +#define PTE_UFFD (_AT(pteval_t, 1) << 94) /* userfaultd tracking */ +#else +#define PTE_UFFD (_AT(pteval_t, 1) << 58) /* userfaultd tracking */ +#endif /* CONFIG_ARM64_D128 */ #define PTE_SWP_UFFD (_AT(pteval_t, 1) << 3) /* only for swp ptes */ #else #define PTE_UFFD (_AT(pteval_t, 0)) @@ -131,11 +140,18 @@ static inline bool __pure lpa2_is_enabled(void) =20 #endif /* __ASSEMBLER__ */ =20 +#ifdef CONFIG_ARM64_D128 +#define pte_pi_index(pte) (((pte) & PTE_PI_MASK) >> PTE_PI_SHIFT) +#define pte_po_index(pte) ((pte_val(pte) & PTE_PO_IDX_MASK) >> PTE_PO_IDX_= SHIFT) +#else #define pte_pi_index(pte) ( \ ((pte & BIT(PTE_PI_IDX_3)) >> (PTE_PI_IDX_3 - 3)) | \ ((pte & BIT(PTE_PI_IDX_2)) >> (PTE_PI_IDX_2 - 2)) | \ ((pte & BIT(PTE_PI_IDX_1)) >> (PTE_PI_IDX_1 - 1)) | \ ((pte & BIT(PTE_PI_IDX_0)) >> (PTE_PI_IDX_0 - 0))) +#define pte_po_index(pte) FIELD_GET(PTE_PO_IDX_MASK, pte_val(pte)) +#endif + =20 /* * Page types used via Permission Indirection Extension (PIE). PIE uses diff --git a/arch/arm64/include/asm/pgtable-types.h b/arch/arm64/include/as= m/pgtable-types.h index 2f2f5527930f..83160f8a19fd 100644 --- a/arch/arm64/include/asm/pgtable-types.h +++ b/arch/arm64/include/asm/pgtable-types.h @@ -17,7 +17,11 @@ * Generic page table descriptor format from which * all level specific descriptors can be derived. */ +#ifdef CONFIG_ARM64_D128 +typedef u128 ptval_t; +#else typedef u64 ptval_t; +#endif =20 typedef ptval_t pteval_t; typedef ptval_t pmdval_t; diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 4b534be3f288..516681b6db37 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -84,18 +84,62 @@ static inline void arch_leave_lazy_mmu_mode(void) arch_flush_lazy_mmu_mode(); } =20 +#ifdef CONFIG_ARM64_D128 +#define ptval_get(x) \ +({ \ + typeof(&(x)) __x =3D &(x); \ + union __u128_halves __v; \ + \ + asm volatile ("ldp %[lo], %[hi], %[v]\n" \ + : [lo] "=3Dr"(__v.low), \ + [hi] "=3Dr"(__v.high) \ + : [v] "Q"(*__x) \ + ); \ + *(typeof(__x))(&__v.full); \ +}) + +#define ptval_set(x, val) \ +({ \ + typeof(&(x)) __x =3D &(x); \ + union __u128_halves __v =3D { .full =3D *(u128*)(&(val)) }; \ + \ + asm volatile ("stp %[lo], %[hi], %[v]\n" \ + : [v] "=3DQ"(*__x) \ + : [lo] "r"(__v.low), \ + [hi] "r"(__v.high) \ + ); \ +}) +#else #define ptval_get(x) READ_ONCE(x) #define ptval_set(x, val) WRITE_ONCE(x, val) +#endif =20 static inline ptval_t ptval_cmpxchg_relaxed(ptval_t *ptep, ptval_t old, ptval_t new) { +#ifdef CONFIG_ARM64_D128 + return cmpxchg128_relaxed(ptep, old, new); +#else return cmpxchg_relaxed(ptep, old, new); +#endif } =20 static inline ptval_t ptval_xchg_relaxed(ptval_t *ptep, ptval_t new) { +#ifdef CONFIG_ARM64_D128 + union __u128_halves r =3D { .full =3D new }; + + asm volatile( + ".arch_extension lse128\n" + "swpp %[lo], %[hi], %[v]\n" + : [lo] "+r" (r.low), + [hi] "+r" (r.high), + [v] "+Q" (*ptep) + :); + return r.full; +#else return xchg_relaxed(ptep, new); +#endif } =20 #define pmdp_get pmdp_get @@ -157,7 +201,7 @@ static inline void pgprot_write(pgprot_t *prot, pgprot_= t val) __flush_tlb_range(vma, address, address + PMD_SIZE, PMD_SIZE, 2, \ TLBF_NOBROADCAST | TLBF_NONOTIFY | TLBF_NOWALKCACHE) =20 -#ifdef CONFIG_ARM64_PA_BITS_52 +#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128) static inline phys_addr_t __pte_to_phys(pte_t pte) { pte_val(pte) &=3D ~PTE_MAYBE_SHARED; @@ -272,7 +316,7 @@ static inline bool por_el0_allows_pkey(u8 pkey, bool wr= ite, bool execute) (((pte_val(pte) & (PTE_VALID | PTE_USER)) =3D=3D (PTE_VALID | PTE_USER)) = && (!(write) || pte_write(pte))) #define pte_access_permitted(pte, write) \ (pte_access_permitted_no_overlay(pte, write) && \ - por_el0_allows_pkey(FIELD_GET(PTE_PO_IDX_MASK, pte_val(pte)), write, fals= e)) + por_el0_allows_pkey(pte_po_index(pte), write, false)) #define pmd_access_permitted(pmd, write) \ (pte_access_permitted(pmd_pte(pmd), (write))) #define pud_access_permitted(pud, write) \ @@ -1582,11 +1626,15 @@ static inline void update_mmu_cache_range(struct vm= _fault *vmf, update_mmu_cache_range(NULL, vma, addr, ptep, 1) #define update_mmu_cache_pmd(vma, address, pmd) do { } while (0) =20 +#ifdef CONFIG_ARM64_D128 +#define phys_to_ttbr(addr) (addr) +#else #ifdef CONFIG_ARM64_PA_BITS_52 #define phys_to_ttbr(addr) (((addr) | ((addr) >> 46)) & TTBR_BADDR_MASK_52) #else #define phys_to_ttbr(addr) (addr) #endif +#endif =20 /* * On arm64 without hardware Access Flag, copying from user will fail beca= use diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 12aacc74f764..4e8889961b15 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -222,7 +222,7 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_= replace_ttbr1) =20 .macro pte_to_phys, phys, pte and \phys, \pte, #PTE_ADDR_LOW -#ifdef CONFIG_ARM64_PA_BITS_52 +#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128) and \pte, \pte, #PTE_ADDR_HIGH orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT #endif --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C69E73F1046 for ; Mon, 7 Sep 2026 03:51:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753071; cv=none; b=fdLGi2rJg5cfHtqmUryx3ME0eJ2W9tzHWhrjlbMYnzUEWTdeb1QPyyPukgWQEukj20LyNYO+kNyGafjRN3sJXeaEBGOtyizxr+0M5ZlI3b8TrJeUEoMLjdHTdeldPl8/k3s7bp7R44sZqZjWeE8cwXCwkuTkll9qepTxxDqaE98= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753071; c=relaxed/simple; bh=F6JIj5gZR+GM7/gf47zeVXEmc1PSYBWbUbC7eU2p0MM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FKoS6o77Rr8PXHV/r8nd3KfTXYsPCA1CtoFQFOcy5rJ1WOOeWaED6ZmsHSi3hMtfy2hLGS3pFmWtaeaBeL9awVIrw62X3Y+XKkmwqs6StaWjlJSdL2YEzwQCa1b96CdS+gPDrimiU5eCMMR5kKBzFt3sO1UPlAiBJHjLteKz3hI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=jIx0Z0Tr; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="jIx0Z0Tr" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2B0A61477; Sun, 6 Sep 2026 20:51:05 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2245D3F7B4; Sun, 6 Sep 2026 20:51:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753068; bh=F6JIj5gZR+GM7/gf47zeVXEmc1PSYBWbUbC7eU2p0MM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jIx0Z0Trzn7fRx4Bb+jxB0bvIXRfkF/yODXZit+fDkD4wfxfzbJ6ESMOs/yxlG+tD ocyhHabSN7XO3ZConckCDV5pFpYLcTCw7+ABBT+rpWgikPPLAVIuLZDnftBv0DDQYr mM9nkrmbIRKmyQCUL7Ghe59JwOKma2OrowJnFZ4g= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 09/14] arm64/mm: Enable 128 bit translation Date: Mon, 7 Sep 2026 09:20:24 +0530 Message-ID: <20260907035030.2839978-10-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All required changes to page table geometry and descriptor are now in place for D128 translation to be enabled. Since FEAT_D128 exclusively supports the permission indirection style for page table entry permission management, given kernel compiled for FEAT_D128 requires both FEAT_S1PIE and FEAT_D128. If these architecture features are not present during boot, the kernel panics just like it does when there is a granule size mismatch. So added a new stuck CPU debug identifier for such scenarios called CPU_STUCK_REASON_NO_D128. TTBR0/1_EL1 and PAR_EL1 registers become 128 bit wide when D128 is enabled, thus requiring MSRR/MRRS instructions for their updates. Because PA_BITS is still capped at 52 bits, MRS/MSR instructions are currently sufficient for the register accesses that basically operate on the lower 64 bits. Although entire 128 bits for these registers get cleared during boot via MSRR. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- arch/arm64/Makefile | 4 ++++ arch/arm64/include/asm/el2_setup.h | 9 +++++++++ arch/arm64/include/asm/smp.h | 1 + arch/arm64/kernel/head.S | 12 ++++++++++++ arch/arm64/mm/proc.S | 23 +++++++++++++++++++++++ 5 files changed, 49 insertions(+) diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 6b005c8fef70..12a6e62520e9 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -54,6 +54,10 @@ endif KBUILD_CFLAGS +=3D $(call cc-option,-mabi=3Dlp64) KBUILD_AFLAGS +=3D $(call cc-option,-mabi=3Dlp64) =20 +ifeq ($(CONFIG_ARM64_D128),y) +KBUILD_AFLAGS +=3D -march=3Darmv9.3-a+d128 +endif + # Avoid generating .eh_frame* sections. ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS +=3D -fno-asynchronous-unwind-tables -fno-unwind-tables diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el= 2_setup.h index aa8ec9df8024..331e94381dd0 100644 --- a/arch/arm64/include/asm/el2_setup.h +++ b/arch/arm64/include/asm/el2_setup.h @@ -78,6 +78,15 @@ cbz x0, .Lskip_hcrx_\@ mov_q x0, (HCRX_EL2_MSCEn | HCRX_EL2_TCR2En | HCRX_EL2_EnFPM) =20 +#ifdef CONFIG_ARM64_D128 + mrs_s x1, SYS_ID_AA64MMFR3_EL1 + ubfx x1, x1, #ID_AA64MMFR3_EL1_D128_SHIFT, #4 + cbz x1, .Lskip_d128_\@ + + orr x0, x0, HCRX_EL2_D128En // Disable MRRS/MSRR traps +.Lskip_d128_\@: +#endif + /* Enable GCS if supported */ mrs_s x1, SYS_ID_AA64PFR1_EL1 ubfx x1, x1, #ID_AA64PFR1_EL1_GCS_SHIFT, #4 diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index 10ea4f543069..1dd675d2b84d 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -22,6 +22,7 @@ =20 #define CPU_STUCK_REASON_52_BIT_VA (UL(1) << CPU_STUCK_REASON_SHIFT) #define CPU_STUCK_REASON_NO_GRAN (UL(2) << CPU_STUCK_REASON_SHIFT) +#define CPU_STUCK_REASON_NO_D128 (UL(3) << CPU_STUCK_REASON_SHIFT) =20 #ifndef __ASSEMBLER__ =20 diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 87a822e5c4ca..4ad8047963ad 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -505,6 +505,18 @@ SYM_FUNC_START_LOCAL(__no_granule_support) b 1b SYM_FUNC_END(__no_granule_support) =20 +#ifdef CONFIG_ARM64_D128 +SYM_FUNC_START(__no_d128_support) + /* Indicate that this CPU can't boot and is stuck in the kernel */ + update_early_cpu_boot_status \ + CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_NO_D128, x1, x2 +1: + wfe + wfi + b 1b +SYM_FUNC_END(__no_d128_support) +#endif + SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, __pi_init_idmap_pg_dir diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 4e8889961b15..faf16ffb0528 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -548,7 +548,30 @@ alternative_else_nop_endif =20 mrs_s x1, SYS_ID_AA64MMFR3_EL1 ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4 +#ifdef CONFIG_ARM64_D128 + cbnz x1, .Lcheck_d128 + bl __no_d128_support +.Lcheck_d128: + mrs_s x1, SYS_ID_AA64MMFR3_EL1 + ubfx x1, x1, #ID_AA64MMFR3_EL1_D128_SHIFT, #4 + cbnz x1, .Linit_d128 + bl __no_d128_support +.Linit_d128: + /* + * Although the lower 64 bits in TTBRx_EL1 registers are now + * being used it is prudent to clear out the entire 128 bits + * just in case the kernel receives non-zero value in higher + * 64 bits from the EL3 which might corrupt the page tables. + */ + mov x4, xzr + mov x5, xzr + + msrr ttbr0_el1, x4, x5 + msrr ttbr1_el1, x4, x5 + orr tcr2, tcr2, #TCR2_EL1_D128 +#else cbz x1, .Lskip_indirection +#endif =20 mov_q x0, PIE_E0_ASM msr REG_PIRE0_EL1, x0 --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3C4753F107B for ; Mon, 7 Sep 2026 03:51:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753075; cv=none; b=OoQbrDsQawOTBemNGm8SlwFord2/oOTDd4DCo4a/tc34Kt3CRPblvjroMEgJketoC86OX5YRaZV/ScGyOR020xyiBaDFtDKFU+eMFN11Q96VTwwkk4EzuDDELem1M/Zgm+DD+T5C3OkFL0VrELlGa2ZW5kqM0qPQrjjkE1vUFeY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753075; c=relaxed/simple; bh=e60cWCqWvU5usBlVAuNQW11xubuWlhpocNAxI1B8G28=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f+y8UOFx7VavK/hfKVaZ2zG7Giv/t7whNW2LuLvizrFlWutS5BkVyRkJ3qNxczvY+hmIEZ99h0XIVFFpwJeecMunwXrIXqPw3AhLd6Yl54tKKvJaoXJ6i8+7ZZbA3s+D23scsiYEPzhF4tZfeaVrZ7remunmhygfWAKRlQys1Cc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=GoO/h9TL; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="GoO/h9TL" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0018B1516; Sun, 6 Sep 2026 20:51:09 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EBE743F7B4; Sun, 6 Sep 2026 20:51:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753072; bh=e60cWCqWvU5usBlVAuNQW11xubuWlhpocNAxI1B8G28=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GoO/h9TLo5WxQU44DjPignv1t1V1GPmlyG79AXj0rqAGEsV1VdWickcqE4TMSYxKb e0iIuExCyFmFHPnmczA8v3nVJeUJ1HA9VAXFwEGmE7WaYEZXk/vp0Prb41nDO6V5jP z3hOjL0pkiO6NZlaJ4aoiMB+agf8MRlGxZ7t6X4k= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 10/14] arm64/mm: Add an abstraction level for tlbi_op Date: Mon, 7 Sep 2026 09:20:25 +0530 Message-ID: <20260907035030.2839978-11-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linu Cherian With FEAT_D128, a new instruction aka TLBIP is being introduced for the TLB -by-range operations which has an argument size of 128 bit. Add an abstraction level to void (*tlbi_op)(u64 arg) helpers to support the D128 variations when applicable. No functional changes are introduced with this patch. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Signed-off-by: Linu Cherian Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/tlbflush.h | 70 ++++++++++++++++--------------- 1 file changed, 37 insertions(+), 33 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlb= flush.h index 14a78ac0f800..88ae4e443ed6 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -130,49 +130,53 @@ static inline void sme_dvmsync_batch(void) =20 #define TLBI_TTL_UNKNOWN INT_MAX =20 -typedef void (*tlbi_op)(u64 arg); +typedef u64 tlbi_arg_t; +#define __tlbi_wrapper(op, arg) __tlbi(op, arg) +#define __tlbi_user_wrapper(op, arg) __tlbi_user(op, arg) =20 -static __always_inline void vae1is(u64 arg) +typedef void (*tlbi_op)(tlbi_arg_t arg); + +static __always_inline void vae1is(tlbi_arg_t arg) { - __tlbi(vae1is, arg); - __tlbi_user(vae1is, arg); + __tlbi_wrapper(vae1is, arg); + __tlbi_user_wrapper(vae1is, arg); } =20 -static __always_inline void vae2is(u64 arg) +static __always_inline void vae2is(tlbi_arg_t arg) { - __tlbi(vae2is, arg); + __tlbi_wrapper(vae2is, arg); } =20 -static __always_inline void vale1(u64 arg) +static __always_inline void vale1(tlbi_arg_t arg) { - __tlbi(vale1, arg); - __tlbi_user(vale1, arg); + __tlbi_wrapper(vale1, arg); + __tlbi_user_wrapper(vale1, arg); } =20 -static __always_inline void vale1is(u64 arg) +static __always_inline void vale1is(tlbi_arg_t arg) { - __tlbi(vale1is, arg); - __tlbi_user(vale1is, arg); + __tlbi_wrapper(vale1is, arg); + __tlbi_user_wrapper(vale1is, arg); } =20 -static __always_inline void vale2is(u64 arg) +static __always_inline void vale2is(tlbi_arg_t arg) { - __tlbi(vale2is, arg); + __tlbi_wrapper(vale2is, arg); } =20 -static __always_inline void vaale1is(u64 arg) +static __always_inline void vaale1is(tlbi_arg_t arg) { - __tlbi(vaale1is, arg); + __tlbi_wrapper(vaale1is, arg); } =20 -static __always_inline void ipas2e1(u64 arg) +static __always_inline void ipas2e1(tlbi_arg_t arg) { - __tlbi(ipas2e1, arg); + __tlbi_wrapper(ipas2e1, arg); } =20 -static __always_inline void ipas2e1is(u64 arg) +static __always_inline void ipas2e1is(tlbi_arg_t arg) { - __tlbi(ipas2e1is, arg); + __tlbi_wrapper(ipas2e1is, arg); } =20 static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 le= vel, @@ -443,32 +447,32 @@ static inline void arch_tlbbatch_flush(struct arch_tl= bflush_unmap_batch *batch) * operations can only span an even number of pages. We save this for l= ast to * ensure 64KB start alignment is maintained for the LPA2 case. */ -static __always_inline void rvae1is(u64 arg) +static __always_inline void rvae1is(tlbi_arg_t arg) { - __tlbi(rvae1is, arg); - __tlbi_user(rvae1is, arg); + __tlbi_wrapper(rvae1is, arg); + __tlbi_user_wrapper(rvae1is, arg); } =20 -static __always_inline void rvale1(u64 arg) +static __always_inline void rvale1(tlbi_arg_t arg) { - __tlbi(rvale1, arg); - __tlbi_user(rvale1, arg); + __tlbi_wrapper(rvale1, arg); + __tlbi_user_wrapper(rvale1, arg); } =20 -static __always_inline void rvale1is(u64 arg) +static __always_inline void rvale1is(tlbi_arg_t arg) { - __tlbi(rvale1is, arg); - __tlbi_user(rvale1is, arg); + __tlbi_wrapper(rvale1is, arg); + __tlbi_user_wrapper(rvale1is, arg); } =20 -static __always_inline void rvaale1is(u64 arg) +static __always_inline void rvaale1is(tlbi_arg_t arg) { - __tlbi(rvaale1is, arg); + __tlbi_wrapper(rvaale1is, arg); } =20 -static __always_inline void ripas2e1is(u64 arg) +static __always_inline void ripas2e1is(tlbi_arg_t arg) { - __tlbi(ripas2e1is, arg); + __tlbi_wrapper(ripas2e1is, arg); } =20 static __always_inline void __tlbi_range(tlbi_op op, u64 addr, --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 657393F1057 for ; Mon, 7 Sep 2026 03:51:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753079; cv=none; b=AoRBsHKomAELw/iADV4Jxvv/luh9Xha8i7nMRNJo0xSMmffJC+6PxD7MnJxgLdIdSxP0uWFul4yOno1sTeIIPzbs8WU0zMvZp3BqMfzZ4VaGAiPixJLy2C+dBCr2jZPlxycSXxi4sjWSZlQ3bwIe9BM15jwRtH+y1lns/gcJf60= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753079; c=relaxed/simple; bh=LhsC0NhjKIdOi9rcgt72ehmDXHWyJcPL2hu83ddAkXM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sgnwtspDz7ocDp8kBkJfawZ/AlevRzx6uaw3VrwG5JDYepFyBxu8N+sONadhhME5ug85vfzMfBEEo9d10YjLMUcFxAniK/WB5CphhT7f8FCaxHm0oTmOiecdZDmOTreAAkdDYuZF8tgNn1R9BNHjtXeEGwP+npD/iZh03+XUWH8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=GDH1iVaD; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="GDH1iVaD" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CD1AF1AED; Sun, 6 Sep 2026 20:51:12 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C28233FBF5; Sun, 6 Sep 2026 20:51:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753076; bh=LhsC0NhjKIdOi9rcgt72ehmDXHWyJcPL2hu83ddAkXM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GDH1iVaDTX9uj2NicaktrTaa+TO516GXCUn6g6G08VZgnHdHZUl1DRy1eGFL6TTxv 5QfnhHYRlF7ub/R7VFyqpPlgoON0Xu9Hhf6wHW0ROHaudZTmb5DVQB+JS16u+qgWDk bTw9+mPbTyJtpWzYfQuf/x3Ux/JhZoSnBhnxP2t0= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 11/14] arm64/mm: Enable TLBIP instruction based TLB flush Date: Mon, 7 Sep 2026 09:20:26 +0530 Message-ID: <20260907035030.2839978-12-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linu Cherian Add TLBIP instruction support for TLB flush operation updating all relevant macros as necessary. This enables back pgtable level hint and address range support during TLB flush operation for D128 translation. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Signed-off-by: Linu Cherian Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/tlbflush.h | 68 +++++++++++++++++++++++++++---- 1 file changed, 59 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlb= flush.h index 88ae4e443ed6..f31a555760f0 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -41,6 +41,25 @@ =20 #define __tlbi(op, ...) __TLBI_N(op, ##__VA_ARGS__, 1, 0) =20 +#ifdef CONFIG_ARM64_D128 +#define __tlbip(op, arg) do { \ + asm (ARM64_ASM_PREAMBLE \ + ".arch_extension d128\n\t" \ + "tlbip " #op ", %0, %H0\n" \ + : : "r" (arg.full)); \ +} while (0) + +#define __tlbip_user(op, arg) do { \ + if (arm64_kernel_unmapped_at_el0()) { \ + arg.low |=3D USER_ASID_FLAG; \ + __tlbip(op, (arg)); \ + } \ +} while (0) + +#endif + +#define TLBI_ASID_MASK GENMASK_ULL(63, 48) + #define __tlbi_user(op, arg) do { \ if (arm64_kernel_unmapped_at_el0()) \ __tlbi(op, (arg) | USER_ASID_FLAG); \ @@ -130,9 +149,15 @@ static inline void sme_dvmsync_batch(void) =20 #define TLBI_TTL_UNKNOWN INT_MAX =20 +#ifdef CONFIG_ARM64_D128 +typedef union __u128_halves tlbi_arg_t; +#define __tlbi_wrapper(op, arg) __tlbip(op, arg) +#define __tlbi_user_wrapper(op, arg) __tlbip_user(op, arg) +#else typedef u64 tlbi_arg_t; #define __tlbi_wrapper(op, arg) __tlbi(op, arg) #define __tlbi_user_wrapper(op, arg) __tlbi_user(op, arg) +#endif =20 typedef void (*tlbi_op)(tlbi_arg_t arg); =20 @@ -179,17 +204,28 @@ static __always_inline void ipas2e1is(tlbi_arg_t arg) __tlbi_wrapper(ipas2e1is, arg); } =20 -static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 le= vel, - u16 asid) +static __always_inline void __tlbi_update_level(u32 level, u64 *arg) { - u64 arg =3D __TLBI_VADDR(addr, asid); - if (alternative_has_cap_unlikely(ARM64_HAS_ARMv8_4_TTL) && level <=3D 3) { u64 ttl =3D level | (get_trans_granule() << 2); =20 - FIELD_MODIFY(TLBI_TTL_MASK, &arg, ttl); + FIELD_MODIFY(TLBI_TTL_MASK, arg, ttl); } +} + +static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 le= vel, u16 asid) +{ +#ifdef CONFIG_ARM64_D128 + union __u128_halves arg; + + arg.low =3D FIELD_PREP(TLBI_ASID_MASK, asid); + __tlbi_update_level(level, &arg.low); + arg.high =3D addr >> 12; +#else + u64 arg =3D __TLBI_VADDR(addr, asid); =20 + __tlbi_update_level(level, &arg); +#endif op(arg); } =20 @@ -475,19 +511,33 @@ static __always_inline void ripas2e1is(tlbi_arg_t arg) __tlbi_wrapper(ripas2e1is, arg); } =20 -static __always_inline void __tlbi_range(tlbi_op op, u64 addr, - u16 asid, int scale, int num, - u32 level, bool lpa2) +static __always_inline u64 __tlbi_range_args_encode_comm(u16 asid, int sca= le, int num, u32 level) { u64 arg =3D 0; =20 - arg |=3D FIELD_PREP(TLBIR_BADDR_MASK, addr >> (lpa2 ? 16 : PAGE_SHIFT)); arg |=3D FIELD_PREP(TLBIR_TTL_MASK, level > 3 ? 0 : level); arg |=3D FIELD_PREP(TLBIR_NUM_MASK, num); arg |=3D FIELD_PREP(TLBIR_SCALE_MASK, scale); arg |=3D FIELD_PREP(TLBIR_TG_MASK, get_trans_granule()); arg |=3D FIELD_PREP(TLBIR_ASID_MASK, asid); =20 + return arg; +} + +static __always_inline void __tlbi_range(tlbi_op op, u64 addr, + u16 asid, int scale, int num, + u32 level, bool lpa2) +{ +#ifdef CONFIG_ARM64_D128 + union __u128_halves arg; + + arg.low =3D __tlbi_range_args_encode_comm(asid, scale, num, level); + arg.high =3D addr >> 12; +#else + u64 arg =3D __tlbi_range_args_encode_comm(asid, scale, num, level); + + arg |=3D FIELD_PREP(TLBIR_BADDR_MASK, addr >> (lpa2 ? 16 : PAGE_SHIFT)); +#endif op(arg); } =20 --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DCE273F1064 for ; Mon, 7 Sep 2026 03:51:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753082; cv=none; b=BJ3n769rqqpBdIAaFSxkHsbhZH3utQWboK9sUjOMS2NnlA1JMXzZ7aEeE/GGr01KeL4VAO0rLRN6+rFGu6ENXtT/DYAcKTodmW3OPZ2o3+jKEoLizCVJzDNSmpdA99OXBO7hwAQ9tsgwlyaDvuUfUcnXVDLTJ2slv5Rpii2Lz4s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753082; c=relaxed/simple; bh=Qb+7E8NXipxvoAEHwL2Au3g/XDt1GmXueAfBnPSqsJs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tsfHBiXjnZlk7F191/wIfTTlmx6E2rhkZjuScEiw/BBZ9r33Nrq/4yyYU3TL8JATkX4+hTnPN/LSw34+uZcBpkhc2XJ6AgIw5FjlYkDI5tuuWGMNo4ihWVXsKejZj6etiWN76ZGcpI/5Eikj9hTfZiQCfHXtd9wcPK3d+XefqCg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=lKEnAII+; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="lKEnAII+" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A16E41BA8; Sun, 6 Sep 2026 20:51:16 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9999D3F7B4; Sun, 6 Sep 2026 20:51:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753080; bh=Qb+7E8NXipxvoAEHwL2Au3g/XDt1GmXueAfBnPSqsJs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lKEnAII+5BcNq0/1T2NrvYL3H4zsr6jbJe17vY4vHbGtC5Ka7nzO99iXt8eHS5Maf DUh3npYqcQW/MG/QV9gj8aGR+OWVu2/7LIT0nnrpnjXk7rBdDRVbBdyr+CQYOu3SFr xyxdDNO5/m6NQtaItwuz7F+PRbFHlpCj5q5LgVLM= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , kasan-dev@googlegroups.com Subject: [PATCH V2 12/14] arm64/mm: Relax KASAN shadow alignment for FEAT_D128 Date: Mon, 7 Sep 2026 09:20:27 +0530 Message-ID: <20260907035030.2839978-13-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" D128 reduces the number of descriptors in each translation table, causing some VA size and page granule combinations that use two page table levels with D64 to now require three levels. For three-level page tables, the PUD level is folded while the PMD level remains a real translation level. Hence, PUD_SIZE represents a root-level span. With D128, some configurations have a KASAN shadow region smaller than this span, making it impossible for both KASAN shadow boundaries to satisfy the existing PUD_SIZE alignment requirement. The KASAN initialization code on arm64 already supports shadow boundaries which are not root-level aligned, by installing or cloning the next-level table as required. Similarly for three level D128 page tables, relax SHADOW_ALIGN to PMD_SIZE, which corresponds to the first non-folded level below the root level. Keep the existing alignment requirements unchanged for all other configurations. Cc: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: kasan-dev@googlegroups.com Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- arch/arm64/mm/kasan_init.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index f61f5bc09467..e12abea6ad15 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -178,7 +178,9 @@ static void __init kasan_pgd_populate(unsigned long add= r, unsigned long end, } while (pgdp++, addr =3D next, addr !=3D end); } =20 -#if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4 +#if defined(CONFIG_ARM64_D128) && CONFIG_PGTABLE_LEVELS =3D=3D 3 +#define SHADOW_ALIGN PMD_SIZE +#elif defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4 #define SHADOW_ALIGN P4D_SIZE #else #define SHADOW_ALIGN PUD_SIZE --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B9C1E3F1054 for ; Mon, 7 Sep 2026 03:51:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753086; cv=none; b=ciFnQDUinrr4Yq6dK3DSDCneNZg98OV03dUE2pSumDxP5MKhWGXkxFPm+B2lzujq6sKAAjx5TldUw0XXeI7EueLEUneQyfmbauxLQTseSaonjBD84jalhs0KVUDZ0eCiOP54y3m0NwoLnJlzWfpXzicQwHVMq/fNBj8lOYGt82g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753086; c=relaxed/simple; bh=mwPzeOmFdvSni4XKS2FMhb0rbVSwZIBkk8UWH/i+Cdk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nWnZ2AI3iKHCYASqNltHtOzWrlsYmWH7itoANL/AserUFXlfnr8rOgpC8zynfAdIFsm/Bg1lJ3vTRQWgjWA90UZ9a+ydjYlM0twG2TPrHCDzo8JhfTvfh8fXaJe0H4R0rKqrV9dgECK8RjkJusrdR/2c3H/tLzEdsqXuY0uLZPQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=cf4fVwxI; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="cf4fVwxI" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7612D1BC0; Sun, 6 Sep 2026 20:51:20 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6B7F53F7B4; Sun, 6 Sep 2026 20:51:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753084; bh=mwPzeOmFdvSni4XKS2FMhb0rbVSwZIBkk8UWH/i+Cdk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cf4fVwxI/mmmgyaiTUIT/ejH9+XvsNL/MOGeyLViRVFbSZVWf1T80E/Cg3+gFm2UJ cAIt+R9hrBwm55yxbTxLbtLImTnzRHfdm1A+s2hyROu4uh4fh9RI+VdEgdms7jNlMP AabfhMhqjw06n1wosd7mIREPGtAeez160qFYBaVM= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 13/14] arm64/mm: Make ARM64_D128 selectable Date: Mon, 7 Sep 2026 09:20:28 +0530 Message-ID: <20260907035030.2839978-14-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All required changes are already in place including the TLBIP instruction support for enabling D128 translation. Just make this new ARM64_D128 user selectable, after ensuring necessary tools chain support required for the new instructions being used for D128. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- Changes in V2: - Dropped !KASAN dependency from ARM64_D128 arch/arm64/Kconfig | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 306e6c803471..483eb6d75c7a 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -276,6 +276,10 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=3D2) =20 +config CC_SUPPORTS_LSE128 + def_bool CC_IS_GCC + depends on $(cc-option, -march=3Darmv8.1-a+lse128) + config 64BIT def_bool y =20 @@ -2305,8 +2309,17 @@ endmenu # "ARMv8.9 architectural features" =20 menu "ARMv9.3 architectural features" =20 +config AS_HAS_ARMV9_3 + def_bool $(cc-option,-Wa$(comma)-march=3Darmv9.3-a) + config ARM64_D128 - bool + bool "Enable support for 128 bit page table (FEAT_D128)" + depends on ARCH_SUPPORTS_INT128 + depends on CC_SUPPORTS_LSE128 + depends on AS_HAS_ARMV9_3 + depends on EXPERT + depends on !VIRTUALIZATION + depends on !UNMAP_KERNEL_AT_EL0 default n help ARMv9.3 introduces FEAT_D128, which provides a 128 bit page --=20 2.43.0 From nobody Sat Sep 26 00:30:48 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 89E573F1071 for ; Mon, 7 Sep 2026 03:51:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753090; cv=none; b=jdHGLaFDQWhuW2JCfX9bGFldpjLAZyu2iBlaQQ6Ofyo8IjX3tOfgL8vK0jAv3CCtGvaQSEwhBsY4CafRKbjg8vQU6zBPxc1H1284SecTI0CNDrln/ao9AKwxKoVVSnIhQ6zKifuig/xnGIa97hdcaZptCvg7w0hXQH61n2JVsb8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788753090; c=relaxed/simple; bh=DRbiHknfTdWf8cVd0fa3aHJlkJRIF0q+njt+T0kU7e4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JrUMICG125H4ohZQKsAx7H3VGgqZrSJoD005a+hIlbL+/fUyyyt0RzvKUg2XQQk7xvPtjTjXwMc8c/ycZ2C3fEvz5wVVKNUiaMAVNql8g6zBN92anEtpv2P2qF2oQ9ABqceKBcCfaP46DGMUkjhELbBNCVO6pWSbx8cO/gVBmmc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=m0tE0bqS; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="m0tE0bqS" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4A0B91AED; Sun, 6 Sep 2026 20:51:24 -0700 (PDT) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 419D03F7B4; Sun, 6 Sep 2026 20:51:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788753088; bh=DRbiHknfTdWf8cVd0fa3aHJlkJRIF0q+njt+T0kU7e4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m0tE0bqS8kuL6E4xKkXdBndaV9dmrJWnYtayuHMG326QpcoLZLpPAb0/DA1T8G6jb DXnz+appGHYQ202Zmp2YaMc9Y2T4MHhlynyW2ZLMvNFmRT+PlxlR+n/SRfOqFkxe12 bGEapHLRVCvTrQK4XwEzjd8xDQXlHnUjs2nsVrK0= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH V2 14/14] arm64/configs: Add d128.config Date: Mon, 7 Sep 2026 09:20:29 +0530 Message-ID: <20260907035030.2839978-15-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260907035030.2839978-1-anshuman.khandual@arm.com> References: <20260907035030.2839978-1-anshuman.khandual@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Provide a slimline configuration intended to be booted with ARM64_D128. Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- arch/arm64/configs/d128.config | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 arch/arm64/configs/d128.config diff --git a/arch/arm64/configs/d128.config b/arch/arm64/configs/d128.config new file mode 100644 index 000000000000..e0a2d7bd8f20 --- /dev/null +++ b/arch/arm64/configs/d128.config @@ -0,0 +1,6 @@ +# ARM64_D128-specific configuration + +# CONFIG_UNMAP_KERNEL_AT_EL0 is not set +# CONFIG_VIRTUALIZATION is not set +CONFIG_EXPERT=3Dy +CONFIG_ARM64_D128=3Dy --=20 2.43.0