From nobody Mon May 11 05:35:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85E9BC4332F for ; Wed, 13 Apr 2022 10:06:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234340AbiDMKJO (ORCPT ); Wed, 13 Apr 2022 06:09:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229792AbiDMKJM (ORCPT ); Wed, 13 Apr 2022 06:09:12 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 10BA2574A6; Wed, 13 Apr 2022 03:06:52 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CFF7E13D5; Wed, 13 Apr 2022 03:06:51 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.39.141]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DD6003F73B; Wed, 13 Apr 2022 03:06:46 -0700 (PDT) From: Anshuman Khandual To: inux-mm@kvack.org Cc: Anshuman Khandual , Andrew Morton , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Arnd Bergmann , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH] tlb/hugetlb: Add framework to handle PGDIR_SIZE HugeTLB pages Date: Wed, 13 Apr 2022 15:37:14 +0530 Message-Id: <20220413100714.509888-1-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Change tlb_remove_huge_tlb_entry() to accommodate larger PGDIR_SIZE HugeTLB pages via adding a new helper tlb_flush_pgd_range(). While here also update struct mmu_gather as required, that is add a new member cleared_pgds. Cc: Andrew Morton Cc: Will Deacon Cc: "Aneesh Kumar K.V" Cc: Nick Piggin Cc: Peter Zijlstra Cc: Arnd Bergmann Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Anshuman Khandual --- This applies on v5.18-rc2, some earlier context could be found here https://lore.kernel.org/all/20220406112124.GD2731@worktop.programming.kicks= -ass.net/ include/asm-generic/tlb.h | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index eee6f7763a39..6eaf0080ef2d 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -282,6 +282,7 @@ struct mmu_gather { unsigned int cleared_pmds : 1; unsigned int cleared_puds : 1; unsigned int cleared_p4ds : 1; + unsigned int cleared_pgds : 1; =20 /* * tracks VM_EXEC | VM_HUGETLB in tlb_start_vma @@ -325,6 +326,7 @@ static inline void __tlb_reset_range(struct mmu_gather = *tlb) tlb->cleared_pmds =3D 0; tlb->cleared_puds =3D 0; tlb->cleared_p4ds =3D 0; + tlb->cleared_pgds =3D 0; /* * Do not reset mmu_gather::vma_* fields here, we do not * call into tlb_start_vma() again to set them if there is an @@ -420,7 +422,7 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gat= her *tlb) * these bits. */ if (!(tlb->freed_tables || tlb->cleared_ptes || tlb->cleared_pmds || - tlb->cleared_puds || tlb->cleared_p4ds)) + tlb->cleared_puds || tlb->cleared_p4ds || tlb->cleared_pgds)) return; =20 tlb_flush(tlb); @@ -472,6 +474,8 @@ static inline unsigned long tlb_get_unmap_shift(struct = mmu_gather *tlb) return PUD_SHIFT; if (tlb->cleared_p4ds) return P4D_SHIFT; + if (tlb->cleared_pgds) + return PGDIR_SHIFT; =20 return PAGE_SHIFT; } @@ -545,6 +549,14 @@ static inline void tlb_flush_p4d_range(struct mmu_gath= er *tlb, tlb->cleared_p4ds =3D 1; } =20 +static inline void tlb_flush_pgd_range(struct mmu_gather *tlb, + unsigned long address, unsigned long size) +{ + __tlb_adjust_range(tlb, address, size); + tlb->cleared_pgds =3D 1; +} + + #ifndef __tlb_remove_tlb_entry #define __tlb_remove_tlb_entry(tlb, ptep, address) do { } while (0) #endif @@ -565,7 +577,9 @@ static inline void tlb_flush_p4d_range(struct mmu_gathe= r *tlb, #define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ do { \ unsigned long _sz =3D huge_page_size(h); \ - if (_sz >=3D P4D_SIZE) \ + if (_sz >=3D PGDIR_SIZE) \ + tlb_flush_pgd_range(tlb, address, _sz); \ + else if (_sz >=3D P4D_SIZE) \ tlb_flush_p4d_range(tlb, address, _sz); \ else if (_sz >=3D PUD_SIZE) \ tlb_flush_pud_range(tlb, address, _sz); \ --=20 2.20.1