From nobody Tue Apr 28 06:30:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FB21CCA47D for ; Sat, 4 Jun 2022 01:32:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244110AbiFDBcp (ORCPT ); Fri, 3 Jun 2022 21:32:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229967AbiFDBcn (ORCPT ); Fri, 3 Jun 2022 21:32:43 -0400 Received: from out30-43.freemail.mail.aliyun.com (out30-43.freemail.mail.aliyun.com [115.124.30.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F136B13CD4 for ; Fri, 3 Jun 2022 18:32:41 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=alimailimapcm10staff010182156082;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=4;SR=0;TI=SMTPD_---0VFH9Sln_1654306358; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VFH9Sln_1654306358) by smtp.aliyun-inc.com(127.0.0.1); Sat, 04 Jun 2022 09:32:39 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 1/3] mm: Factor out the pagetable pages account into new helper function Date: Sat, 4 Jun 2022 09:32:29 +0800 Message-Id: <809544e284d35caf72aa66bf7214a1ef006cad40.1654271618.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Factor out the pagetable pages account into new helper functions to avoid duplicated code. Meanwhile these helper functions also will be used to account pagetable pages which do not need split pagetale lock. Signed-off-by: Baolin Wang --- include/linux/mm.h | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8a96197..1ce3366 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2358,20 +2358,30 @@ static inline void pgtable_init(void) pgtable_cache_init(); } =20 +static inline void pgtable_set_and_inc(struct page *page) +{ + __SetPageTable(page); + inc_lruvec_page_state(page, NR_PAGETABLE); +} + +static inline void pgtable_clear_and_dec(struct page *page) +{ + __ClearPageTable(page); + dec_lruvec_page_state(page, NR_PAGETABLE); +} + static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + pgtable_set_and_inc(page); return true; } =20 static inline void pgtable_pte_page_dtor(struct page *page) { ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + pgtable_clear_and_dec(page); } =20 #define pte_offset_map_lock(mm, pmd, address, ptlp) \ @@ -2457,16 +2467,14 @@ static inline bool pgtable_pmd_page_ctor(struct pag= e *page) { if (!pmd_ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + pgtable_set_and_inc(page); return true; } =20 static inline void pgtable_pmd_page_dtor(struct page *page) { pmd_ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + pgtable_clear_and_dec(page); } =20 /* --=20 1.8.3.1 From nobody Tue Apr 28 06:30:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39B1AC433EF for ; Sat, 4 Jun 2022 01:32:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348262AbiFDBc6 (ORCPT ); Fri, 3 Jun 2022 21:32:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50484 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343715AbiFDBcq (ORCPT ); Fri, 3 Jun 2022 21:32:46 -0400 Received: from out199-2.us.a.mail.aliyun.com (out199-2.us.a.mail.aliyun.com [47.90.199.2]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2405D13CD4 for ; Fri, 3 Jun 2022 18:32:43 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=4;SR=0;TI=SMTPD_---0VFHJaat_1654306359; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VFHJaat_1654306359) by smtp.aliyun-inc.com(127.0.0.1); Sat, 04 Jun 2022 09:32:40 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 2/3] mm: Add PUD level pagetable account Date: Sat, 4 Jun 2022 09:32:30 +0800 Message-Id: <17fda3fff85235026c28824ef99bf9700d9e5a8f.1654271618.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Now the PUD level ptes are always protected by mm->page_table_lock, which means no split pagetable lock needed. So the generic PUD level pagetable pages allocation will not call pgtable_pte_page_ctor/dtor(), that means we will miss to account PUD level pagetable pages. Adding pagetable account by calling pgtable_set_and_inc() or pgtable_clear_and_dec() when allocating or freeing PUD level pagetable pages to help to get an accurate pagetable accounting. Moreover this patch will also mark the PUD level pagetable with PG_table flag, which will help to do sanity validation in unpoison_memory() and get more accurate pagetable accounting by /proc/kpageflags interface. Meanwhile converting the architectures with using generic PUD pagatable allocation to add corresponding pgtable_set_and_inc() or pgtable_clear_and_= dec() to account PUD level pagetable. Signed-off-by: Baolin Wang --- arch/arm64/include/asm/tlb.h | 5 ++++- arch/mips/include/asm/pgalloc.h | 11 ++++++++--- arch/s390/include/asm/tlb.h | 1 + arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 8 +++++++- 5 files changed, 23 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index c995d1f..47e0623 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -94,7 +94,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb= , pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pudp)); + struct page *page =3D virt_to_page(pudp); + + pgtable_clear_and_dec(page); + tlb_remove_table(tlb, page); } #endif =20 diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index 867e9c3..0950f5f 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -89,11 +89,16 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm= , unsigned long address) =20 static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long add= ress) { + struct page *pg; pud_t *pud; =20 - pud =3D (pud_t *) __get_free_pages(GFP_KERNEL, PUD_ORDER); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + pg =3D alloc_pages(GFP_KERNEL & ~__GFP_HIGHMEM, PUD_ORDER); + if (!pg) + return NULL; + + pgtable_set_and_inc(pg); + pud =3D (pud_t *)page_address(pg); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); return pud; } =20 diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index fe6407f..45f9541 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -125,6 +125,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb,= pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pgtable_clear_and_dec(virt_to_page(pud)); tlb->mm->context.flush_mm =3D 1; tlb->freed_tables =3D 1; tlb->cleared_p4ds =3D 1; diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a932d77..6cccf52 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -76,6 +76,9 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { + struct page *page =3D virt_to_page(pud); + + pgtable_clear_and_dec(page); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 977bea1..78ab9f6 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -149,11 +149,16 @@ static inline void pmd_free(struct mm_struct *mm, pmd= _t *pmd) =20 static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long a= ddr) { + struct page *page; gfp_t gfp =3D GFP_PGTABLE_USER; =20 if (mm =3D=3D &init_mm) gfp =3D GFP_PGTABLE_KERNEL; - return (pud_t *)get_zeroed_page(gfp); + page =3D alloc_pages((gfp | __GFP_ZERO) & ~__GFP_HIGHMEM, 0); + if (!page) + return NULL; + pgtable_set_and_inc(page); + return (pud_t *)page_address(page); } =20 #ifndef __HAVE_ARCH_PUD_ALLOC_ONE @@ -175,6 +180,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm= , unsigned long addr) static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); + pgtable_clear_and_dec(virt_to_page(pud)); free_page((unsigned long)pud); } =20 --=20 1.8.3.1 From nobody Tue Apr 28 06:30:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E642C433EF for ; Sat, 4 Jun 2022 01:32:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349998AbiFDBcy (ORCPT ); Fri, 3 Jun 2022 21:32:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245331AbiFDBcq (ORCPT ); Fri, 3 Jun 2022 21:32:46 -0400 Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D399612D1B for ; Fri, 3 Jun 2022 18:32:43 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=alimailimapcm10staff010182156082;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=4;SR=0;TI=SMTPD_---0VFHJab9_1654306360; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VFHJab9_1654306360) by smtp.aliyun-inc.com(127.0.0.1); Sat, 04 Jun 2022 09:32:41 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 3/3] mm: Add kernel PTE level pagetable pages account Date: Sat, 4 Jun 2022 09:32:31 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Now the kernel PTE level ptes are always protected by mm->page_table_lock instead of split pagetable lock, so the kernel PTE level pagetable pages are not accounted. To get an accurate pagetable accounting, calling new helpers pgtable_set_and_inc()/pgtable_clear_and_dec() when allocating or freeing a kernel PTE level pagetable page. Meanwhile converting architectures to use corresponding generic PTE pagetab= le allocation and freeing functions. Signed-off-by: Baolin Wang --- arch/csky/include/asm/pgalloc.h | 2 +- arch/microblaze/mm/pgtable.c | 2 +- arch/openrisc/mm/ioremap.c | 2 +- include/asm-generic/pgalloc.h | 10 +++++++++- 4 files changed, 12 insertions(+), 4 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index bbbd069..2443226 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) pte_t *pte; unsigned long i; =20 - pte =3D (pte_t *) __get_free_page(GFP_KERNEL); + pte =3D __pte_alloc_one_kernel(mm); if (!pte) return NULL; =20 diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 9f73265..e96dd1b 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -245,7 +245,7 @@ unsigned long iopa(unsigned long addr) __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { if (mem_init_done) - return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + return __pte_alloc_one_kernel(mm); else return memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE, MEMBLOCK_LOW_LIMIT, diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index daae13a..3453acc 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -118,7 +118,7 @@ pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; =20 if (likely(mem_init_done)) { - pte =3D (pte_t *)get_zeroed_page(GFP_KERNEL); + pte =3D __pte_alloc_one_kernel(mm); } else { pte =3D memblock_alloc(PAGE_SIZE, PAGE_SIZE); if (!pte) diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 78ab9f6..f5345b2 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -18,7 +18,14 @@ */ static inline pte_t *__pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(GFP_PGTABLE_KERNEL); + struct page *page; + gfp_t gfp =3D GFP_PGTABLE_KERNEL; + + page =3D alloc_pages(gfp & ~__GFP_HIGHMEM, 0); + if (!page) + return NULL; + pgtable_set_and_inc(page); + return (pte_t *)page_address(page); } =20 #ifndef __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL @@ -41,6 +48,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struc= t *mm) */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { + pgtable_clear_and_dec(virt_to_page(pte)); free_page((unsigned long)pte); } =20 --=20 1.8.3.1