From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 419B038DEC; Mon, 17 Mar 2025 14:21:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221298; cv=none; b=e8O97qUxlRhLmxiTFEvZEtRLN5ZSnK0k+dz8AHbju2yWuviULCfzxj8TIwAoJjDWat6cObul8z6v+V+oiPQ853bxeUiinKMU47SoO4v3xJttwm6K0GM8QxQU2h6/RFyZtHlx+j32l5WGBD9TG0e602zTX10q4ktTmCehzPhcPLI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221298; c=relaxed/simple; bh=yd5d93R3iXz1J8VOreo7Uw53nUDqKGz1OPnXp22G2Z4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jN9yazYTop+47cmtTvmEseGaXkOXr4rjZ8By/KQr95X51NAztzhj0SU+GtMouavCFWYWxUH6ui85o9vjgHVrwjZ2rnRpJvuJ2vrvZW4nzsWRWBFitWEJdJL2wATo+5unwxSFIphNzYncWDsBCPtY4I2yGl1UQtNxpMPTfcGVWJs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 42C2F1688; Mon, 17 Mar 2025 07:21:44 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EFD3D3F63F; Mon, 17 Mar 2025 07:21:30 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 01/11] mm: Pass mm down to pagetable_{pte,pmd}_ctor Date: Mon, 17 Mar 2025 14:16:50 +0000 Message-ID: <20250317141700.3701581-2-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for calling constructors for all kernel page tables while eliding unnecessary ptlock initialisation, let's pass down the associated mm to the PTE/PMD level ctors. (These are the two levels where ptlocks are used.) In most cases the mm is already around at the point of calling the ctor so we simply pass it down. This is however not the case for special page table allocators: * arch/arm/mm/mmu.c * arch/arm64/mm/mmu.c * arch/riscv/mm/init.c In those cases, the page tables being allocated are either for standard kernel memory (init_mm) or special page directories, which may not be associated to any mm. For now let's pass NULL as mm; this will be refined where possible in future patches. No functional change in this patch. Signed-off-by: Kevin Brodsky --- arch/arm/mm/mmu.c | 2 +- arch/arm64/mm/mmu.c | 4 ++-- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 2 +- arch/m68k/include/asm/motorola_pgalloc.h | 10 +++++----- arch/m68k/mm/motorola.c | 6 +++--- arch/mips/include/asm/pgalloc.h | 2 +- arch/parisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 2 +- arch/riscv/mm/init.c | 4 ++-- arch/s390/include/asm/pgalloc.h | 2 +- arch/s390/mm/pgalloc.c | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/x86/mm/pgtable.c | 2 +- include/asm-generic/pgalloc.h | 4 ++-- include/linux/mm.h | 6 ++++-- 18 files changed, 30 insertions(+), 28 deletions(-) diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c index f02f872ea8a9..edb7f56b7c91 100644 --- a/arch/arm/mm/mmu.c +++ b/arch/arm/mm/mmu.c @@ -735,7 +735,7 @@ static void *__init late_alloc(unsigned long sz) void *ptdesc =3D pagetable_alloc(GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM, get_order(sz)); =20 - if (!ptdesc || !pagetable_pte_ctor(ptdesc)) + if (!ptdesc || !pagetable_pte_ctor(NULL, ptdesc)) BUG(); return ptdesc_to_virt(ptdesc); } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1dfe1a8efdbe..437d4977bcf5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -494,9 +494,9 @@ static phys_addr_t pgd_pgtable_alloc(int shift) * folded, and if so pagetable_pte_ctor() becomes nop. */ if (shift =3D=3D PAGE_SHIFT) - BUG_ON(!pagetable_pte_ctor(ptdesc)); + BUG_ON(!pagetable_pte_ctor(NULL, ptdesc)); else if (shift =3D=3D PMD_SHIFT) - BUG_ON(!pagetable_pmd_ctor(ptdesc)); + BUG_ON(!pagetable_pmd_ctor(NULL, ptdesc)); =20 return pa; } diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/= asm/pgalloc.h index 7211dff8c969..57b7dbcee259 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -72,7 +72,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, = unsigned long address) if (!ptdesc) return NULL; =20 - if (!pagetable_pmd_ctor(ptdesc)) { + if (!pagetable_pmd_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mc= f_pgalloc.h index 4c648b51e7fd..465a71101b7d 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -48,7 +48,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *m= m) =20 if (!ptdesc) return NULL; - if (!pagetable_pte_ctor(ptdesc)) { + if (!pagetable_pte_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/m68k/include/asm/motorola_pgalloc.h b/arch/m68k/include/a= sm/motorola_pgalloc.h index 5abe7da8ac5a..1091fb0affbe 100644 --- a/arch/m68k/include/asm/motorola_pgalloc.h +++ b/arch/m68k/include/asm/motorola_pgalloc.h @@ -15,7 +15,7 @@ enum m68k_table_types { }; =20 extern void init_pointer_table(void *table, int type); -extern void *get_pointer_table(int type); +extern void *get_pointer_table(struct mm_struct *mm, int type); extern int free_pointer_table(void *table, int type); =20 /* @@ -26,7 +26,7 @@ extern int free_pointer_table(void *table, int type); =20 static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { - return get_pointer_table(TABLE_PTE); + return get_pointer_table(mm, TABLE_PTE); } =20 static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) @@ -36,7 +36,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, = pte_t *pte) =20 static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { - return get_pointer_table(TABLE_PTE); + return get_pointer_table(mm, TABLE_PTE); } =20 static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) @@ -53,7 +53,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= pgtable_t pgtable, =20 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long add= ress) { - return get_pointer_table(TABLE_PMD); + return get_pointer_table(mm, TABLE_PMD); } =20 static inline int pmd_free(struct mm_struct *mm, pmd_t *pmd) @@ -75,7 +75,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *= pgd) =20 static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - return get_pointer_table(TABLE_PGD); + return get_pointer_table(mm, TABLE_PGD); } =20 =20 diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index 73651e093c4d..6ab3ef39ba7a 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -139,7 +139,7 @@ void __init init_pointer_table(void *table, int type) return; } =20 -void *get_pointer_table(int type) +void *get_pointer_table(struct mm_struct *mm, int type) { ptable_desc *dp =3D ptable_list[type].next; unsigned int mask =3D list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp= ); @@ -164,10 +164,10 @@ void *get_pointer_table(int type) * m68k doesn't have SPLIT_PTE_PTLOCKS for not having * SMP. */ - pagetable_pte_ctor(virt_to_ptdesc(page)); + pagetable_pte_ctor(mm, virt_to_ptdesc(page)); break; case TABLE_PMD: - pagetable_pmd_ctor(virt_to_ptdesc(page)); + pagetable_pmd_ctor(mm, virt_to_ptdesc(page)); break; case TABLE_PGD: pagetable_pgd_ctor(virt_to_ptdesc(page)); diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index 26c7a6ede983..415237af4029 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -65,7 +65,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, = unsigned long address) if (!ptdesc) return NULL; =20 - if (!pagetable_pmd_ctor(ptdesc)) { + if (!pagetable_pmd_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pg= alloc.h index 2ca74a56415c..3b84ee93edaa 100644 --- a/arch/parisc/include/asm/pgalloc.h +++ b/arch/parisc/include/asm/pgalloc.h @@ -39,7 +39,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, = unsigned long address) ptdesc =3D pagetable_alloc(gfp, PMD_TABLE_ORDER); if (!ptdesc) return NULL; - if (!pagetable_pmd_ctor(ptdesc)) { + if (!pagetable_pmd_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/= pgtable.c index ce64abea9e3e..83afc4e72088 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -423,7 +423,7 @@ static pmd_t *__alloc_for_pmdcache(struct mm_struct *mm) ptdesc =3D pagetable_alloc(gfp, 0); if (!ptdesc) return NULL; - if (!pagetable_pmd_ctor(ptdesc)) { + if (!pagetable_pmd_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index 713268ccb1a0..387e9b1fe12c 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -61,7 +61,7 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, = int kernel) ptdesc =3D pagetable_alloc(PGALLOC_GFP | __GFP_ACCOUNT, 0); if (!ptdesc) return NULL; - if (!pagetable_pte_ctor(ptdesc)) { + if (!pagetable_pte_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 15b2eda4c364..703c3648cfa9 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -409,7 +409,7 @@ static phys_addr_t __meminit alloc_pte_late(uintptr_t v= a) { struct ptdesc *ptdesc =3D pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0); =20 - BUG_ON(!ptdesc || !pagetable_pte_ctor(ptdesc)); + BUG_ON(!ptdesc || !pagetable_pte_ctor(NULL, ptdesc)); return __pa((pte_t *)ptdesc_address(ptdesc)); } =20 @@ -489,7 +489,7 @@ static phys_addr_t __meminit alloc_pmd_late(uintptr_t v= a) { struct ptdesc *ptdesc =3D pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0); =20 - BUG_ON(!ptdesc || !pagetable_pmd_ctor(ptdesc)); + BUG_ON(!ptdesc || !pagetable_pmd_ctor(NULL, ptdesc)); return __pa((pmd_t *)ptdesc_address(ptdesc)); } =20 diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index b19b6ed2ab53..cd65b93d87e9 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -98,7 +98,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, = unsigned long vmaddr) if (!table) return NULL; crst_table_init(table, _SEGMENT_ENTRY_EMPTY); - if (!pagetable_pmd_ctor(virt_to_ptdesc(table))) { + if (!pagetable_pmd_ctor(mm, virt_to_ptdesc(table))) { crst_table_free(mm, table); return NULL; } diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 30387a6e98ff..35b0ab6fab24 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -170,7 +170,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) ptdesc =3D pagetable_alloc(GFP_KERNEL, 0); if (!ptdesc) return NULL; - if (!pagetable_pte_ctor(ptdesc)) { + if (!pagetable_pte_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 05882bca5b73..cd60a0a8ca0e 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2899,7 +2899,7 @@ pgtable_t pte_alloc_one(struct mm_struct *mm) =20 if (!ptdesc) return NULL; - if (!pagetable_pte_ctor(ptdesc)) { + if (!pagetable_pte_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index dd32711022f5..f8fb4911d360 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -350,7 +350,7 @@ pgtable_t pte_alloc_one(struct mm_struct *mm) page =3D pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_inc_return(page) =3D=3D 2 && - !pagetable_pte_ctor(page_ptdesc(page))) { + !pagetable_pte_ctor(mm, page_ptdesc(page))) { page_ref_dec(page); ptep =3D NULL; } diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 1fef5ad32d5a..2d20f021c50b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -249,7 +249,7 @@ static int preallocate_pmds(struct mm_struct *mm, pmd_t= *pmds[], int count) =20 if (!ptdesc) failed =3D true; - if (ptdesc && !pagetable_pmd_ctor(ptdesc)) { + if (ptdesc && !pagetable_pmd_ctor(mm, ptdesc)) { pagetable_free(ptdesc); ptdesc =3D NULL; failed =3D true; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 892ece4558a2..e164ca66f0f6 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -70,7 +70,7 @@ static inline pgtable_t __pte_alloc_one_noprof(struct mm_= struct *mm, gfp_t gfp) ptdesc =3D pagetable_alloc_noprof(gfp, 0); if (!ptdesc) return NULL; - if (!pagetable_pte_ctor(ptdesc)) { + if (!pagetable_pte_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } @@ -137,7 +137,7 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_str= uct *mm, unsigned long ad ptdesc =3D pagetable_alloc_noprof(gfp, 0); if (!ptdesc) return NULL; - if (!pagetable_pmd_ctor(ptdesc)) { + if (!pagetable_pmd_ctor(mm, ptdesc)) { pagetable_free(ptdesc); return NULL; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 8483e09aeb2c..d92c16f6cfa2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3015,7 +3015,8 @@ static inline void pagetable_dtor_free(struct ptdesc = *ptdesc) pagetable_free(ptdesc); } =20 -static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) +static inline bool pagetable_pte_ctor(struct mm_struct *mm, + struct ptdesc *ptdesc) { if (!ptlock_init(ptdesc)) return false; @@ -3121,7 +3122,8 @@ static inline spinlock_t *pmd_lock(struct mm_struct *= mm, pmd_t *pmd) return ptl; } =20 -static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) +static inline bool pagetable_pmd_ctor(struct mm_struct *mm, + struct ptdesc *ptdesc) { if (!pmd_ptlock_init(ptdesc)) return false; --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 97465241CA3; Mon, 17 Mar 2025 14:21:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221303; cv=none; b=rV7vUCqPWe+4LDZ9n/K4YihPoX5zLbHM1ymfrBhdaGN4ZMX3YMCAldMlny++cOeS6Co2AgMIFPpOu/ETEVouZDAhs5qJWklArrh4cZsWAK1hX3ymJzAPXkTAluFsdJ+xUGqd5MU1KmZjv40mr3iTcoW813cRmCtJgd5GFYnjZSM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221303; c=relaxed/simple; bh=AIOmX96DReFPlzhtOAbuvr12YuLFspc4aJyh2yGNzRM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QaO0dtrIZiv/z6SGxDv+UILqtk+rHv/4Ln+ksLREFl3lldLzhqWAf4ZitlzvbpyWoOpuoz3TanSYdFExbcawjsTo/r6QLDZos9goayszkc+yCNcIc3u5BdOEIBgUGQWAmyTUb7WMPmTtnqUOqfbhS0gRp87Ay8OksvmuiWLQTxo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CF25D22FA; Mon, 17 Mar 2025 07:21:48 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A78843F63F; Mon, 17 Mar 2025 07:21:35 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 02/11] mm: Call ctor/dtor for kernel PTEs Date: Mon, 17 Mar 2025 14:16:51 +0000 Message-ID: <20250317141700.3701581-3-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since [1], constructors/destructors are expected to be called for all page table pages, at all levels and for both user and kernel pgtables. There is however one glaring exception: kernel PTEs are managed via separate helpers (pte_alloc_kernel/pte_free_kernel), which do not call the [cd]tor, at least not in the generic implementation. The most obvious reason for this anomaly is that init_mm is special-cased not to use split page table locks. As a result calling ptlock_init() for PTEs associated with init_mm would be wasteful, potentially resulting in dynamic memory allocation. However, pgtable [cd]tors perform other actions - currently related to accounting/statistics, and potentially more functionally significant in the future. Now that pagetable_pte_ctor() is passed the associated mm, we can make it skip the call to ptlock_init() for init_mm; this allows us to call the ctor from pte_alloc_one_kernel() too. This is matched by a call to the pgtable destructor in pte_free_kernel(); no special-casing is needed on that path, as ptlock_free() is already called unconditionally. (ptlock_free() is a no-op unless a ptlock was allocated for the given PTP.) This patch ensures that all architectures that rely on call the [cd]tor for kernel PTEs. pte_free_kernel() cannot be overridden so changing the generic implementation is sufficient. pte_alloc_one_kernel() can be overridden using __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL, and a few architectures implement it by calling the page allocator directly. We amend those so that they call the generic __pte_alloc_one_kernel() instead, if possible, ensuring that the ctor is called. A few architectures do not use ; those will be taken care of separately. [1] https://lore.kernel.org/linux-mm/20250103184415.2744423-1-kevin.brodsky= @arm.com/ Signed-off-by: Kevin Brodsky --- arch/csky/include/asm/pgalloc.h | 2 +- arch/microblaze/mm/pgtable.c | 2 +- arch/openrisc/mm/ioremap.c | 2 +- include/asm-generic/pgalloc.h | 7 ++++++- include/linux/mm.h | 2 +- 5 files changed, 10 insertions(+), 5 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index bf8400c28b5a..288dca0d160a 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) pte_t *pte; unsigned long i; =20 - pte =3D (pte_t *) __get_free_page(GFP_KERNEL); + pte =3D __pte_alloc_one_kernel(mm); if (!pte) return NULL; =20 diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 9f73265aad4e..e96dd1b7aba4 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -245,7 +245,7 @@ unsigned long iopa(unsigned long addr) __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { if (mem_init_done) - return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + return __pte_alloc_one_kernel(mm); else return memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE, MEMBLOCK_LOW_LIMIT, diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index 8e63e86251ca..3b352f97fecb 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -36,7 +36,7 @@ pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; =20 if (likely(mem_init_done)) { - pte =3D (pte_t *)get_zeroed_page(GFP_KERNEL); + pte =3D __pte_alloc_one_kernel(mm); } else { pte =3D memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index e164ca66f0f6..3c8ec3bfea44 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -23,6 +23,11 @@ static inline pte_t *__pte_alloc_one_kernel_noprof(struc= t mm_struct *mm) =20 if (!ptdesc) return NULL; + if (!pagetable_pte_ctor(mm, ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } + return ptdesc_address(ptdesc); } #define __pte_alloc_one_kernel(...) alloc_hooks(__pte_alloc_one_kernel_nop= rof(__VA_ARGS__)) @@ -48,7 +53,7 @@ static inline pte_t *pte_alloc_one_kernel_noprof(struct m= m_struct *mm) */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { - pagetable_free(virt_to_ptdesc(pte)); + pagetable_dtor_free(virt_to_ptdesc(pte)); } =20 /** diff --git a/include/linux/mm.h b/include/linux/mm.h index d92c16f6cfa2..ee31ffd7ead2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3018,7 +3018,7 @@ static inline void pagetable_dtor_free(struct ptdesc = *ptdesc) static inline bool pagetable_pte_ctor(struct mm_struct *mm, struct ptdesc *ptdesc) { - if (!ptlock_init(ptdesc)) + if (mm !=3D &init_mm && !ptlock_init(ptdesc)) return false; __pagetable_ctor(ptdesc); return true; --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4C950241CA3; Mon, 17 Mar 2025 14:21:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221306; cv=none; b=PtcM5dj1cDsr6RsE4RdzDqNkJDb8WHdgk3amPVR2SEPYaOoYn5mLVVPbHn0rfa+UsqWVMk+UO9QY1WNf45g/ruV8ylCYpgG0CX/W/QZL3/QGqxcjT59FSkCMqXCz2GFPz28XO3SWHZzJiPaL4Q6HGh08pwgMSqvdABcROuKOyP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221306; c=relaxed/simple; bh=St2Kv65BwofpU4q8HERwkv8VvzQChXdvekG2xPYCROA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ij8xNi8U2Fq3QJkn6K2SAHvq0xJkp06GJav/hDXhY7or6ja557sauW6vKwQsCk/geM7fQYxAgPAyXxqOSqDcI49lvYYGkdT6IxAIpSoYesdLTjg8UrlWBVToL2y3Kd+y8fzfn4WvZA5a/TRF249HKHxBbNzu201fXNhUNPlManA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 67A932573; Mon, 17 Mar 2025 07:21:53 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3FEA43F63F; Mon, 17 Mar 2025 07:21:40 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 03/11] m68k: mm: Call ctor/dtor for kernel PTEs Date: Mon, 17 Mar 2025 14:16:52 +0000 Message-ID: <20250317141700.3701581-4-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generic implementation of pte_{alloc_one,free}_kernel now calls the [cd]tor. Align the m68k/ColdFire implementation of those functions by calling the [cd]tor explicitly. Signed-off-by: Kevin Brodsky --- arch/m68k/include/asm/mcf_pgalloc.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mc= f_pgalloc.h index 465a71101b7d..fc5454d37da3 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -7,7 +7,7 @@ =20 static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { - pagetable_free(virt_to_ptdesc(pte)); + pagetable_dtor_free(virt_to_ptdesc(pte)); } =20 extern const char bad_pmd_string[]; @@ -19,6 +19,10 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_stru= ct *mm) =20 if (!ptdesc) return NULL; + if (!pagetable_pte_ctor(mm, ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } =20 return ptdesc_address(ptdesc); } --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 79BD72417C9; Mon, 17 Mar 2025 14:21:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221311; cv=none; b=YpJ8QS+hYv+B2IGo3cn3Ou/gMhiSnB1c2qEkzQndSeUTOVSZIK67Mhph3Ln22JfjYUHyZpfNe23VaCVNFFvIQdBKHmPJFwD4nrdPJIVvMYCIeA2wtqxAGz5InoGivVFA/4qoea5rYgkY9ZFyuXQQ2AsTnexKGtx3SPB59an4OG0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221311; c=relaxed/simple; bh=sJY3k14WBl1j2uqNZ+rNzVrFO/O8YxCuCcrCnNtLeO4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NqFeSOGav7yi8PgJgDv8gLDFrAAzURhWCNg96wSmmxvRQs4DWxq9mkOLr9/NRI0sv+OH3gmYwOMEN7uh3v/o4tgN5lY8ocehjKElObNVqFUJwrxa5vlaV2w1x9sF0uSTHNFlqYdigObfDA+PjzYf8bMQcX8Ic923SXhpMyy1sdY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 005532576; Mon, 17 Mar 2025 07:21:58 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CC9AA3F63F; Mon, 17 Mar 2025 07:21:44 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 04/11] powerpc: mm: Call ctor/dtor for kernel PTEs Date: Mon, 17 Mar 2025 14:16:53 +0000 Message-ID: <20250317141700.3701581-5-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generic implementation of pte_{alloc_one,free}_kernel now calls the [cd]tor, without initialising the ptlock needlessly as pagetable_pte_ctor() skips it for init_mm. On powerpc, all functions related to PTE allocation are implemented by common helpers, which are passed a boolean to differentiate user from kernel pgtables. This patch aligns the powerpc implementation with the generic one by calling pagetable_pte_[cd]tor() unconditionally in those helpers. Signed-off-by: Kevin Brodsky --- arch/powerpc/mm/pgtable-frag.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index 387e9b1fe12c..77e55eac16e4 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -56,19 +56,17 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm= , int kernel) { void *ret =3D NULL; struct ptdesc *ptdesc; + gfp_t gfp =3D PGALLOC_GFP; =20 - if (!kernel) { - ptdesc =3D pagetable_alloc(PGALLOC_GFP | __GFP_ACCOUNT, 0); - if (!ptdesc) - return NULL; - if (!pagetable_pte_ctor(mm, ptdesc)) { - pagetable_free(ptdesc); - return NULL; - } - } else { - ptdesc =3D pagetable_alloc(PGALLOC_GFP, 0); - if (!ptdesc) - return NULL; + if (!kernel) + gfp |=3D __GFP_ACCOUNT; + + ptdesc =3D pagetable_alloc(gfp, 0); + if (!ptdesc) + return NULL; + if (!pagetable_pte_ctor(mm, ptdesc)) { + pagetable_free(ptdesc); + return NULL; } =20 atomic_set(&ptdesc->pt_frag_refcount, 1); @@ -124,12 +122,10 @@ void pte_fragment_free(unsigned long *table, int kern= el) =20 BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <=3D 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - if (kernel) - pagetable_free(ptdesc); - else if (folio_test_clear_active(ptdesc_folio(ptdesc))) - call_rcu(&ptdesc->pt_rcu_head, pte_free_now); - else + if (kernel || !folio_test_clear_active(ptdesc_folio(ptdesc))) pte_free_now(&ptdesc->pt_rcu_head); + else + call_rcu(&ptdesc->pt_rcu_head, pte_free_now); } } =20 --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 15600245021; Mon, 17 Mar 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221315; cv=none; b=BMaf7ct1dYlAmoT79DbA2hq50jtRNwb/O9fEy0R7A8h7VypSbVNELnVaCQGcfK0FiC+VjjqQXPB8NTYo4e6S+LrKlZO2Wg+PQS8gzOSab4uAhQoyTiTbuQtaiiMGeCQwe53UmW+hcutrHphEGkpmjj+OdbpW3a2rFljPOCbP1/U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221315; c=relaxed/simple; bh=7v603hTsRoDmyjkhBUMllCP1y2jdjoBY3yl7hipr9T0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f8bEgTqT0XEQs3B0G+mDIA0G69h1CRI9Pcxbb4mLI5r1sytmy1gyukjTfvoQ/rj4y/qXxOBAtIgzguMHF/tNNW+WpfO/ncauPqWuu2dHwwjdNdAh+n1eF5uv6G/qtTZIFiNVyoDmkyOj+b+wroZXJCdez34giKnAqydEhPeErDA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8C87A25DC; Mon, 17 Mar 2025 07:22:02 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 64BEE3F63F; Mon, 17 Mar 2025 07:21:49 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 05/11] sparc64: mm: Call ctor/dtor for kernel PTEs Date: Mon, 17 Mar 2025 14:16:54 +0000 Message-ID: <20250317141700.3701581-6-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generic implementation of pte_{alloc_one,free}_kernel now calls the [cd]tor, without initialising the ptlock needlessly as pagetable_pte_ctor() skips it for init_mm. Align sparc64 with the generic implementation by ensuring pagetable_pte_[cd]tor() are called for kernel PTEs. As a result the kernel and user alloc/free functions have the same implementation, and since pgtable_t is defined as pte_t *, we can have both call a common helper. Signed-off-by: Kevin Brodsky --- arch/sparc/mm/init_64.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index cd60a0a8ca0e..0a1b2b1adad4 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2882,18 +2882,7 @@ void __flush_tlb_all(void) : : "r" (pstate)); } =20 -pte_t *pte_alloc_one_kernel(struct mm_struct *mm) -{ - struct page *page =3D alloc_page(GFP_KERNEL | __GFP_ZERO); - pte_t *pte =3D NULL; - - if (page) - pte =3D (pte_t *) page_address(page); - - return pte; -} - -pgtable_t pte_alloc_one(struct mm_struct *mm) +static pte_t *__pte_alloc_one(struct mm_struct *mm) { struct ptdesc *ptdesc =3D pagetable_alloc(GFP_KERNEL | __GFP_ZERO, 0); =20 @@ -2906,9 +2895,14 @@ pgtable_t pte_alloc_one(struct mm_struct *mm) return ptdesc_address(ptdesc); } =20 -void pte_free_kernel(struct mm_struct *mm, pte_t *pte) +pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { - free_page((unsigned long)pte); + return __pte_alloc_one(mm); +} + +pgtable_t pte_alloc_one(struct mm_struct *mm) +{ + return __pte_alloc_one(mm); } =20 static void __pte_free(pgtable_t pte) @@ -2919,6 +2913,11 @@ static void __pte_free(pgtable_t pte) pagetable_free(ptdesc); } =20 +void pte_free_kernel(struct mm_struct *mm, pte_t *pte) +{ + __pte_free(pte); +} + void pte_free(struct mm_struct *mm, pgtable_t pte) { __pte_free(pte); --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E896D23FC6B; Mon, 17 Mar 2025 14:21:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221320; cv=none; b=MHlYq51KElW4OMCrdYNa2FbFazrUbLFgwsT3v6U7vbRTaNeaLzs2/H2LvWoVl1w6FIGs3qMel7r7NVODzY2lA6F20RFDW3re8damSaenetdHcJB6tTjE1hboBcnoehAErCP9kWg51910xisSWcgIOeOkNGCM/CJG0y7P+7hEw5o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221320; c=relaxed/simple; bh=w65cv4CUNKlw6sVPxiOLyDXr602rMyWtTDqQ4n3B7mY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HeF7oeXNVaw/pbiRYso4K279cdh/Nd6HNraxTS//Y6BIbz5miEbC/K3SeJTI3U2noiU3gnAWe4N3cPZFoKFNW+PbyZ9YAGEbh+fAgAB4S3m34y+Q6Sn/fghg5wNjOLZ90mPGZ5lY6j7lbxvs534SPDxxyVYwKwD/Lb2DxfLsjLs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 24CC326BE; Mon, 17 Mar 2025 07:22:07 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id F102B3F63F; Mon, 17 Mar 2025 07:21:53 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 06/11] mm: Skip ptlock_init() for kernel PMDs Date: Mon, 17 Mar 2025 14:16:55 +0000 Message-ID: <20250317141700.3701581-7-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Split page table locks are not used for pgtables associated to init_mm, at any level. pte_alloc_kernel() does not call ptlock_init() as a result. There is however no separate alloc/free functions for kernel PMDs, and pmd_ptlock_init() is called unconditionally. When ALLOC_SPLIT_PTLOCKS is true (e.g. 32-bit architectures or if CONFIG_PREEMPT_RT is selected), this results in unnecessary dynamic memory allocation every time a kernel PMD is allocated. Now that pagetable_pmd_ctor() is passed the associated mm, we can easily remove this overhead by skipping pmd_ptlock_init() if the pgtable is associated to init_mm. No special-casing is needed on the dtor path, as ptlock_free() is already called unconditionally for all levels. (ptlock_free() is a no-op unless a ptlock was allocated for the given PTP.) Signed-off-by: Kevin Brodsky --- include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ee31ffd7ead2..4759da9cd633 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3125,7 +3125,7 @@ static inline spinlock_t *pmd_lock(struct mm_struct *= mm, pmd_t *pmd) static inline bool pagetable_pmd_ctor(struct mm_struct *mm, struct ptdesc *ptdesc) { - if (!pmd_ptlock_init(ptdesc)) + if (mm !=3D &init_mm && !pmd_ptlock_init(ptdesc)) return false; ptdesc_pmd_pts_init(ptdesc); __pagetable_ctor(ptdesc); --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7C609241665; Mon, 17 Mar 2025 14:22:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221325; cv=none; b=cNSGBUQ15mjIVqHLQMQFG7gVGghNPIv4n1aFgcP1eMWa7JplSh2YtH6TcSRfdBGVQk/4maCVwk/wYUenizGfR4gu6owW8vwRGfy2bHBwmdUN+1eAth/4Kf86Ci+Yf0k+CdcFsus18gaRaTCR/kXjU3TOXlJwxRxwMqE26Cc9E7E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221325; c=relaxed/simple; bh=BdKCWrq9fn+Orfoe6hrE90xpPOR0L96jBckyjS0Jm4w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GlrbovjgpY0rbd6IoOPK4i+TjVy+3PxCO1OznJK06B2SLNdwYrCg8py8PHCe8QHQy/psH4glY5TZaLrW2otCy7tvbCM846kgGNPUFhdXCwM9jvAUiBG3mNJHaAG97xtxFwmq3UsmpWDFRIM2CSOSrQWgC39zJzLSKTxvGjhwCIE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B17952936; Mon, 17 Mar 2025 07:22:11 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8A11D3F63F; Mon, 17 Mar 2025 07:21:58 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 07/11] arm64: mm: Use enum to identify pgtable level instead of *_SHIFT Date: Mon, 17 Mar 2025 14:16:56 +0000 Message-ID: <20250317141700.3701581-8-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit 90292aca9854 ("arm64: mm: use appropriate ctors for page tables") introduced pgtable ctor calls in pgd_pgtable_alloc(). To identify the pgtable level and call the appropriate ctor, the *_SHIFT value associated with the pgtable level is used. However, those values do not unambiguously identify a level, because if a given level is folded, the *_SHIFT value will be equal to that of the upper level (e.g. PMD_SHIFT =3D=3D PUD_SHIFT if PMD is folded). As things stand, there is probably not much damaged done by calling the ctor for a different level, and ARCH_ENABLE_SPLIT_PMD_PTLOCK is only selected if PMD isn't folded (so we don't needlessly initialise pmd_ptlock). Still, this is pretty confusing, and it would get even more confusing when adding ctor calls for the remaining levels. Let's simplify all this by using an enum to identify the pgtable level instead; this way folding becomes irrelevant. This is inspired by one of the m68k pgtable allocators (arch/m68k/include/asm/motorola_pgalloc.h). Signed-off-by: Kevin Brodsky --- arch/arm64/mm/mmu.c | 54 +++++++++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 437d4977bcf5..a7292ce9d7b8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -46,6 +46,13 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ =20 +enum pgtable_type { + TABLE_PTE, + TABLE_PMD, + TABLE_PUD, + TABLE_P4D, +}; + u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); =20 @@ -107,7 +114,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsign= ed long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); =20 -static phys_addr_t __init early_pgtable_alloc(int shift) +static phys_addr_t __init early_pgtable_alloc(enum pgtable_type pgtable_ty= pe) { phys_addr_t phys; =20 @@ -192,7 +199,7 @@ static void init_pte(pte_t *ptep, unsigned long addr, u= nsigned long end, static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + phys_addr_t (*pgtable_alloc)(enum pgtable_type), int flags) { unsigned long next; @@ -207,7 +214,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned l= ong addr, if (flags & NO_EXEC_MAPPINGS) pmdval |=3D PMD_TABLE_PXN; BUG_ON(!pgtable_alloc); - pte_phys =3D pgtable_alloc(PAGE_SHIFT); + pte_phys =3D pgtable_alloc(TABLE_PTE); ptep =3D pte_set_fixmap(pte_phys); init_clear_pgtable(ptep); ptep +=3D pte_index(addr); @@ -243,7 +250,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned l= ong addr, =20 static void init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) + phys_addr_t (*pgtable_alloc)(enum pgtable_type), int flags) { unsigned long next; =20 @@ -277,7 +284,8 @@ static void init_pmd(pmd_t *pmdp, unsigned long addr, u= nsigned long end, static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) + phys_addr_t (*pgtable_alloc)(enum pgtable_type), + int flags) { unsigned long next; pud_t pud =3D READ_ONCE(*pudp); @@ -294,7 +302,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned l= ong addr, if (flags & NO_EXEC_MAPPINGS) pudval |=3D PUD_TABLE_PXN; BUG_ON(!pgtable_alloc); - pmd_phys =3D pgtable_alloc(PMD_SHIFT); + pmd_phys =3D pgtable_alloc(TABLE_PMD); pmdp =3D pmd_set_fixmap(pmd_phys); init_clear_pgtable(pmdp); pmdp +=3D pmd_index(addr); @@ -325,7 +333,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned l= ong addr, =20 static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long = end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + phys_addr_t (*pgtable_alloc)(enum pgtable_type), int flags) { unsigned long next; @@ -339,7 +347,7 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long a= ddr, unsigned long end, if (flags & NO_EXEC_MAPPINGS) p4dval |=3D P4D_TABLE_PXN; BUG_ON(!pgtable_alloc); - pud_phys =3D pgtable_alloc(PUD_SHIFT); + pud_phys =3D pgtable_alloc(TABLE_PUD); pudp =3D pud_set_fixmap(pud_phys); init_clear_pgtable(pudp); pudp +=3D pud_index(addr); @@ -383,7 +391,7 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long a= ddr, unsigned long end, =20 static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long = end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + phys_addr_t (*pgtable_alloc)(enum pgtable_type), int flags) { unsigned long next; @@ -397,7 +405,7 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long a= ddr, unsigned long end, if (flags & NO_EXEC_MAPPINGS) pgdval |=3D PGD_TABLE_PXN; BUG_ON(!pgtable_alloc); - p4d_phys =3D pgtable_alloc(P4D_SHIFT); + p4d_phys =3D pgtable_alloc(TABLE_P4D); p4dp =3D p4d_set_fixmap(p4d_phys); init_clear_pgtable(p4dp); p4dp +=3D p4d_index(addr); @@ -427,7 +435,7 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long a= ddr, unsigned long end, static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + phys_addr_t (*pgtable_alloc)(enum pgtable_type), int flags) { unsigned long addr, end, next; @@ -455,7 +463,7 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, p= hys_addr_t phys, static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + phys_addr_t (*pgtable_alloc)(enum pgtable_type), int flags) { mutex_lock(&fixmap_lock); @@ -468,10 +476,11 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_a= ddr_t phys, extern __alias(__create_pgd_mapping_locked) void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long= virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags); + phys_addr_t (*pgtable_alloc)(enum pgtable_type), + int flags); #endif =20 -static phys_addr_t __pgd_pgtable_alloc(int shift) +static phys_addr_t __pgd_pgtable_alloc(enum pgtable_type pgtable_type) { /* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ void *ptr =3D (void *)__get_free_page(GFP_PGTABLE_KERNEL & ~__GFP_ZERO); @@ -480,23 +489,26 @@ static phys_addr_t __pgd_pgtable_alloc(int shift) return __pa(ptr); } =20 -static phys_addr_t pgd_pgtable_alloc(int shift) +static phys_addr_t pgd_pgtable_alloc(enum pgtable_type pgtable_type) { - phys_addr_t pa =3D __pgd_pgtable_alloc(shift); + phys_addr_t pa =3D __pgd_pgtable_alloc(pgtable_type); struct ptdesc *ptdesc =3D page_ptdesc(phys_to_page(pa)); =20 /* * Call proper page table ctor in case later we need to * call core mm functions like apply_to_page_range() on * this pre-allocated page table. - * - * We don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK if pmd is - * folded, and if so pagetable_pte_ctor() becomes nop. */ - if (shift =3D=3D PAGE_SHIFT) + switch (pgtable_type) { + case TABLE_PTE: BUG_ON(!pagetable_pte_ctor(NULL, ptdesc)); - else if (shift =3D=3D PMD_SHIFT) + break; + case TABLE_PMD: BUG_ON(!pagetable_pmd_ctor(NULL, ptdesc)); + break; + default: + break; + } =20 return pa; } --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 034B92459FF; Mon, 17 Mar 2025 14:22:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221329; cv=none; b=fI8nwFMx3E/Rpnl57ckHJWEqJ/6aq97XQGajuLDuZ9Ct1tGFcfRDy3mQysDYzl4VFht3Cl+GjlEIZW3SkoR/5hroTNEAGv89bFGrbPfxtpp4D6dojsrQEGYqRojbh43ix5vdoW/yzxECtrUEzY/XAG5nPBkltVqc/usctLD1N78= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221329; c=relaxed/simple; bh=f3yEWEMyHLEB+8WPuVE2Bn5lqEVcMg4KQriOh7IhUVs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=g4eKA4ODbiRP3nPycaxSSusEkOoDv/LbwY2SC0bkW+qrAynTEj/lp7ZFbI6YwrdE31wDXb3ajU36YuWx+ZihTMRFpiBEP+WOJRWe5gXl4GF5MQCszPYa7fhYW9KnQ5Ep0xx2tWeArqeq10sBp0dxLadFllqdWOrBoKHDV2kMd8o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 49B6813D5; Mon, 17 Mar 2025 07:22:16 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2274D3F63F; Mon, 17 Mar 2025 07:22:03 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 08/11] arm64: mm: Always call PTE/PMD ctor in __create_pgd_mapping() Date: Mon, 17 Mar 2025 14:16:57 +0000 Message-ID: <20250317141700.3701581-9-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" TL;DR: always call the PTE/PMD ctor, passing the appropriate mm to skip ptlock_init() if unneeded. __create_pgd_mapping() is used for creating different kinds of mappings, and may allocate page table pages if passed an allocator callback. There are currently three such cases: 1. create_pgd_mapping(), which is used to create the EFI mapping 2. arch_add_memory() 3. map_entry_trampoline() 1. uses pgd_pgtable_alloc() as allocator callback, which calls the PTE/PMD ctor, while 2. and 3. use __pgd_pgtable_alloc(), which does not. The rationale is most likely that pgtables associated with init_mm do not make use of split page table locks, and it is therefore unnecessary to initialise them by calling the ctor. 2. operates on swapper_pg_dir so the allocated pgtables are clearly associated with init_mm, this is arguably the case for 3. too (the trampoline mapping is never modified so ptlocks are anyway irrelevant). 1. corresponds to efi_mm so ptlocks need to be initialised in that case. We are now moving towards calling the ctor for all page tables, even those associated with init_mm. pagetable_{pte,pmd}_ctor() have become aware of the associated mm so that the ptlock initialisation can be skipped for init_mm. This patch therefore amends the allocator callbacks so that the PTE/PMD ctor are always called, with an appropriate mm pointer to avoid unnecessary ptlock overhead. Modifying the prototype of the allocator callbacks to take the mm and propagating that pointer all the way down would be pretty invasive. Instead: * __pgd_pgtable_alloc() (cases 2. and 3. above) is replaced with pgd_pgtable_alloc_init_mm(), resulting in the ctors being called with &init_mm. This is the main functional change in this patch; the ptlock still isn't initialised, but other ctor actions (e.g. accounting-related) are now carried out for those allocated pgtables. * pgd_pgtable_alloc() (case 1. above) is replaced with pgd_pgtable_alloc_special_mm(), resulting in the ctors being called with NULL as mm. No functional change here; NULL essentially means "not init_mm", and the ptlock is still initialised. __pgd_pgtable_alloc() is now the common implementation of those two helpers. While at it we switch it to using pagetable_alloc() like standard pgtable allocator functions and remove the comment regarding ctor calls (ctors are now always expected to be called). Signed-off-by: Kevin Brodsky --- arch/arm64/mm/mmu.c | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a7292ce9d7b8..accb0a33c59f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -480,31 +480,22 @@ void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_= t phys, unsigned long virt, int flags); #endif =20 -static phys_addr_t __pgd_pgtable_alloc(enum pgtable_type pgtable_type) +static phys_addr_t __pgd_pgtable_alloc(struct mm_struct *mm, + enum pgtable_type pgtable_type) { /* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ - void *ptr =3D (void *)__get_free_page(GFP_PGTABLE_KERNEL & ~__GFP_ZERO); + struct ptdesc *ptdesc =3D pagetable_alloc(GFP_PGTABLE_KERNEL & ~__GFP_ZER= O, 0); + phys_addr_t pa; =20 - BUG_ON(!ptr); - return __pa(ptr); -} - -static phys_addr_t pgd_pgtable_alloc(enum pgtable_type pgtable_type) -{ - phys_addr_t pa =3D __pgd_pgtable_alloc(pgtable_type); - struct ptdesc *ptdesc =3D page_ptdesc(phys_to_page(pa)); + BUG_ON(!ptdesc); + pa =3D page_to_phys(ptdesc_page(ptdesc)); =20 - /* - * Call proper page table ctor in case later we need to - * call core mm functions like apply_to_page_range() on - * this pre-allocated page table. - */ switch (pgtable_type) { case TABLE_PTE: - BUG_ON(!pagetable_pte_ctor(NULL, ptdesc)); + BUG_ON(!pagetable_pte_ctor(mm, ptdesc)); break; case TABLE_PMD: - BUG_ON(!pagetable_pmd_ctor(NULL, ptdesc)); + BUG_ON(!pagetable_pmd_ctor(mm, ptdesc)); break; default: break; @@ -513,6 +504,16 @@ static phys_addr_t pgd_pgtable_alloc(enum pgtable_type= pgtable_type) return pa; } =20 +static phys_addr_t pgd_pgtable_alloc_init_mm(enum pgtable_type pgtable_typ= e) +{ + return __pgd_pgtable_alloc(&init_mm, pgtable_type); +} + +static phys_addr_t pgd_pgtable_alloc_special_mm(enum pgtable_type pgtable_= type) +{ + return __pgd_pgtable_alloc(NULL, pgtable_type); +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -542,7 +543,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, ph= ys_addr_t phys, flags =3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; =20 __create_pgd_mapping(mm->pgd, phys, virt, size, prot, - pgd_pgtable_alloc, flags); + pgd_pgtable_alloc_special_mm, flags); } =20 static void update_mapping_prot(phys_addr_t phys, unsigned long virt, @@ -756,7 +757,7 @@ static int __init map_entry_trampoline(void) memset(tramp_pg_dir, 0, PGD_SIZE); __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, entry_tramp_text_size(), prot, - __pgd_pgtable_alloc, NO_BLOCK_MAPPINGS); + pgd_pgtable_alloc_init_mm, NO_BLOCK_MAPPINGS); =20 /* Map both the text and data into the kernel page table */ for (i =3D 0; i < DIV_ROUND_UP(entry_tramp_text_size(), PAGE_SIZE); i++) @@ -1362,7 +1363,7 @@ int arch_add_memory(int nid, u64 start, u64 size, flags |=3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; =20 __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), - size, params->pgprot, __pgd_pgtable_alloc, + size, params->pgprot, pgd_pgtable_alloc_init_mm, flags); =20 memblock_clear_nomap(start, size); --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5CA29241CBC; Mon, 17 Mar 2025 14:22:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221333; cv=none; b=Yfi7brKBcgGM8hbKe1/YH6PzrYnzJOppVRlxjq8OeuH13QN7rJ6/kEo6GuT2qhyqgO1dgHvMHE72b5Qv7Iu6oh/c66FgIQTVAFvazZOGgAJOv44aMh5elnOpPh2BhPGtqNr7xFz+9umWuiOivyMJdwqwh3DurrrMd/Am0kAyEPU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221333; c=relaxed/simple; bh=C4hoqDZiPATdMujBNXtZvLwaTPVITEWhIw9fQ2RjxFA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ELe5gEY3ieMaNSU+AqIEy06YCu/Z/IkiTfBmiC3bXPt5HkohphXgQaXMs4alphNBarhlW5yLpEHjHxlmCzQznfQ4ZHwfutwIKBMUQ7gpxAMxEjmQjGKft5QylIPKTZu/d8Gf2OAROkAuBDBN0VNFoHDbAHBbLvy9SCshVgFmWjA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D64111688; Mon, 17 Mar 2025 07:22:20 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AEC523F63F; Mon, 17 Mar 2025 07:22:07 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 09/11] riscv: mm: Clarify ctor mm argument in alloc_{pte,pmd}_late Date: Mon, 17 Mar 2025 14:16:58 +0000 Message-ID: <20250317141700.3701581-10-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" pagetable_{pte,pmd}_ctor(mm, ptdesc) skip the ptlock initialisation if mm is &init_mm. To avoid unnecessary overhead, it is therefore preferable to pass the actual mm associated to the PTE/PMD. Unfortunately, this proves challenging for alloc_{pte,pmd}_late() as the associated mm is not available at the point where they are called - in fact not even top-level functions like create_pgd_mapping() are passed the mm. As a result they both call the ctor with NULL as mm; this is safe but potentially wasteful. This is not a new situation, but let's add a couple of comments to clarify it. Signed-off-by: Kevin Brodsky --- arch/riscv/mm/init.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 703c3648cfa9..fb18940113f2 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -409,6 +409,11 @@ static phys_addr_t __meminit alloc_pte_late(uintptr_t = va) { struct ptdesc *ptdesc =3D pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0); =20 + /* + * We do not know which mm the PTE page is associated to at this point. + * Passing NULL to the ctor is the safe option, though it may result + * in unnecessary work (e.g. initialising the ptlock for init_mm). + */ BUG_ON(!ptdesc || !pagetable_pte_ctor(NULL, ptdesc)); return __pa((pte_t *)ptdesc_address(ptdesc)); } @@ -489,6 +494,7 @@ static phys_addr_t __meminit alloc_pmd_late(uintptr_t v= a) { struct ptdesc *ptdesc =3D pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0); =20 + /* See comment in alloc_pte_late() regarding NULL passed the ctor */ BUG_ON(!ptdesc || !pagetable_pmd_ctor(NULL, ptdesc)); return __pa((pmd_t *)ptdesc_address(ptdesc)); } --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 38271242936; Mon, 17 Mar 2025 14:22:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221338; cv=none; b=m5H/XzWZNi9suHc7CrIp1c2mQX3qKa5pzdde/9hD7VObAEZEHScXurOShYBxnkQFxKc5Fq4rojlGyrPqezXagHzXdn6PlDIZQX5jNhlAOeallBJuapgVcqFo65Mt62HuikZwc9cH34c4GzHRXH9J5P6UjzJFT/0lHpMU9SYkKdg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221338; c=relaxed/simple; bh=RNsmwgKd54pXI04RhtjMDwCNzHijd0GVztdeN7KSOtM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AkxjXgetYQlB00SgrQ40jFBuue0avB+hhUjvFHhdTMNiPaQPm2IEEnTnFAxxkMXdkPgjBBZPZ8Jg2WGVbShe42XkA8p6hidN+z8zmPT04r0vAtn6BKENwKjhif6b1QtSehSHT+Ah/FiylYpIW30P6NrXpl6G/ySP50GuevVPnbQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6E78C22FA; Mon, 17 Mar 2025 07:22:25 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 46C2C3F63F; Mon, 17 Mar 2025 07:22:12 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 10/11] arm64: mm: Call PUD/P4D ctor in __create_pgd_mapping() Date: Mon, 17 Mar 2025 14:16:59 +0000 Message-ID: <20250317141700.3701581-11-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Constructors for PUD/P4D-level pgtables were recently introduced. They should be called for all pgtables; make sure they are called for special kernel mappings created by __create_pgd_mapping() too. Signed-off-by: Kevin Brodsky --- arch/arm64/mm/mmu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index accb0a33c59f..10bf39654e77 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -497,7 +497,11 @@ static phys_addr_t __pgd_pgtable_alloc(struct mm_struc= t *mm, case TABLE_PMD: BUG_ON(!pagetable_pmd_ctor(mm, ptdesc)); break; - default: + case TABLE_PUD: + pagetable_pud_ctor(ptdesc); + break; + case TABLE_P4D: + pagetable_p4d_ctor(ptdesc); break; } =20 --=20 2.47.0 From nobody Wed Dec 17 13:56:01 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C54D424A056; Mon, 17 Mar 2025 14:22:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221343; cv=none; b=FHT62ByKWwXk+42sxXlFcX2TYJyD8P3y3ydQT2nXZKlr8Xw46aeJduJwjFfPFelnCR/n8uIePMOCTnVCvZ2Kh2SnD90VywSUpV+9cRcZ0kc1LyrdqAS3TTL9a4B7xu3UVMyEIclvV0kqMQ6CPU6rSRYxt9fQAU7IIuDG7fNcKWg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742221343; c=relaxed/simple; bh=rQYXNBvmvmZDvgD5WxDZTChFqhlSHr1Rda9SRbyWklA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cohh2WE2ZXLDz2Cdqk/9kwP34tVqXWEMC9vMAhDzvs31h46yJzmYbwL+QaWvTGVS4sXbOfVhTsyTUPE0P/sT9Lqr07SfBxUeCQ4asI0X3HpXP4TIZQujjF1StOZL8ODP+j2ln3hLLoW/VOgLXGGFHWaad44GXJHXEPhpCvZigZ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 06A762573; Mon, 17 Mar 2025 07:22:30 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D35C43F63F; Mon, 17 Mar 2025 07:22:16 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org Subject: [PATCH 11/11] riscv: mm: Call PUD/P4D ctor in special kernel pgtable alloc Date: Mon, 17 Mar 2025 14:17:00 +0000 Message-ID: <20250317141700.3701581-12-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250317141700.3701581-1-kevin.brodsky@arm.com> References: <20250317141700.3701581-1-kevin.brodsky@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Constructors for PUD/P4D-level pgtables were recently introduced. They should be called for all pgtables; make sure they are called for special kernel mappings created by create_pgd_mapping() too. While at it also switch to using pagetable_alloc() like in alloc_{pte,pmd}_late(). Signed-off-by: Kevin Brodsky --- arch/riscv/mm/init.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fb18940113f2..dc2715f3fd00 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -557,11 +557,11 @@ static phys_addr_t __init alloc_pud_fixmap(uintptr_t = va) =20 static phys_addr_t __meminit alloc_pud_late(uintptr_t va) { - unsigned long vaddr; + struct ptdesc *ptdesc =3D pagetable_alloc(GFP_KERNEL, 0); =20 - vaddr =3D __get_free_page(GFP_KERNEL); - BUG_ON(!vaddr); - return __pa(vaddr); + BUG_ON(!ptdesc); + pagetable_pud_ctor(ptdesc); + return __pa((pud_t *)ptdesc_address(ptdesc)); } =20 static p4d_t *__init get_p4d_virt_early(phys_addr_t pa) @@ -595,11 +595,11 @@ static phys_addr_t __init alloc_p4d_fixmap(uintptr_t = va) =20 static phys_addr_t __meminit alloc_p4d_late(uintptr_t va) { - unsigned long vaddr; + struct ptdesc *ptdesc =3D pagetable_alloc(GFP_KERNEL, 0); =20 - vaddr =3D __get_free_page(GFP_KERNEL); - BUG_ON(!vaddr); - return __pa(vaddr); + BUG_ON(!ptdesc); + pagetable_p4d_ctor(ptdesc); + return __pa((p4d_t *)ptdesc_address(ptdesc)); } =20 static void __meminit create_pud_mapping(pud_t *pudp, uintptr_t va, phys_a= ddr_t pa, phys_addr_t sz, --=20 2.47.0