From nobody Fri Dec 19 15:19:31 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 50CB12550DA for ; Tue, 22 Apr 2025 08:18:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745309934; cv=none; b=iS+/ljGADzSnKYXmOAIiVmP95rB5bThB9hTME0swNzvKG1NZBzdp0heruWD/cXW+fVX7LSCYdNGtuS3aBRD3o0HjaChQhLlKgLAJJ7/HnyZsHykHA5t1wmCPD3SuoBgF38GgzOp4PQY28cNEN/hltXPa9IqVE+nMmhVHSsUhWF8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745309934; c=relaxed/simple; bh=9tnuHllDz0UN3u08MTXde6SQv886649kDGOS9XSyYH4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XIvjJTgUpdQGErqpJT6r2MU6XYOSy7593xrrLiOS+b6TVZCvWi6OLY4hQwKaKxyy7+3UHaf83A2HfUn+yqJgAVRkYgQqJl8CfnXWIamjf1tGBiVQDl1g07ac6QOKU5eyhVxYhBsKyFZfi6RATQjrDKLOUaCAJ1Hr71w8+SHyt7c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 73133176A; Tue, 22 Apr 2025 01:18:47 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D7F853F66E; Tue, 22 Apr 2025 01:18:49 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Anshuman Khandual , Alexandre Ghiti , Kevin Brodsky Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 05/11] arm64: hugetlb: Use __set_ptes_anysz() and __ptep_get_and_clear_anysz() Date: Tue, 22 Apr 2025 09:18:13 +0100 Message-ID: <20250422081822.1836315-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250422081822.1836315-1-ryan.roberts@arm.com> References: <20250422081822.1836315-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor the huge_pte helpers to use the new common __set_ptes_anysz() and __ptep_get_and_clear_anysz() APIs. This provides 2 benefits; First, when page_table_check=3Don, hugetlb is now properly/fully checked. Previously only the first page of a hugetlb folio was checked. Second, instead of having to call __set_ptes(nr=3D1) for each pte in a loop, the whole contiguous batch can now be set in one go, which enables some efficiencies and cleans up the code. One detail to note is that huge_ptep_clear_flush() was previously calling ptep_clear_flush() for a non-contiguous pte (i.e. a pud or pmd block mapping). This has a couple of disadvantages; first ptep_clear_flush() calls ptep_get_and_clear() which transparently handles contpte. Given we only call for non-contiguous ptes, it would be safe, but a waste of effort. It's preferable to go straight to the layer below. However, more problematic is that ptep_get_and_clear() is for PAGE_SIZE entries so it calls page_table_check_pte_clear() and would not clear the whole hugetlb folio. So let's stop special-casing the non-cont case and just rely on get_clear_contig_flush() to do the right thing for non-cont entries. Reviewed-by: Catalin Marinas Signed-off-by: Ryan Roberts Reviewed-by: Anshuman Khandual --- arch/arm64/mm/hugetlbpage.c | 53 +++++++------------------------------ 1 file changed, 10 insertions(+), 43 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 087fc43381c6..d34703846ef4 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -159,12 +159,11 @@ static pte_t get_clear_contig(struct mm_struct *mm, pte_t pte, tmp_pte; bool present; =20 - pte =3D __ptep_get_and_clear(mm, addr, ptep); + pte =3D __ptep_get_and_clear_anysz(mm, ptep, pgsize); present =3D pte_present(pte); while (--ncontig) { ptep++; - addr +=3D pgsize; - tmp_pte =3D __ptep_get_and_clear(mm, addr, ptep); + tmp_pte =3D __ptep_get_and_clear_anysz(mm, ptep, pgsize); if (present) { if (pte_dirty(tmp_pte)) pte =3D pte_mkdirty(pte); @@ -208,7 +207,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr =3D addr; =20 for (i =3D 0; i < ncontig; i++, addr +=3D pgsize, ptep++) - __ptep_get_and_clear(mm, addr, ptep); + __ptep_get_and_clear_anysz(mm, ptep, pgsize); =20 __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, true); } @@ -219,32 +218,20 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned l= ong addr, size_t pgsize; int i; int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; =20 ncontig =3D num_contig_ptes(sz, &pgsize); =20 if (!pte_present(pte)) { for (i =3D 0; i < ncontig; i++, ptep++, addr +=3D pgsize) - __set_ptes(mm, addr, ptep, pte, 1); + __set_ptes_anysz(mm, ptep, pte, 1, pgsize); return; } =20 - if (!pte_cont(pte)) { - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - pfn =3D pte_pfn(pte); - dpfn =3D pgsize >> PAGE_SHIFT; - hugeprot =3D pte_pgprot(pte); - /* Only need to "break" if transitioning valid -> valid. */ - if (pte_valid(__ptep_get(ptep))) + if (pte_cont(pte) && pte_valid(__ptep_get(ptep))) clear_flush(mm, addr, ptep, pgsize, ncontig); =20 - for (i =3D 0; i < ncontig; i++, ptep++, addr +=3D pgsize, pfn +=3D dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + __set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); } =20 pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -434,11 +421,9 @@ int huge_ptep_set_access_flags(struct vm_area_struct *= vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty) { - int ncontig, i; + int ncontig; size_t pgsize =3D 0; - unsigned long pfn =3D pte_pfn(pte), dpfn; struct mm_struct *mm =3D vma->vm_mm; - pgprot_t hugeprot; pte_t orig_pte; =20 VM_WARN_ON(!pte_present(pte)); @@ -447,7 +432,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *v= ma, return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); =20 ncontig =3D num_contig_ptes(huge_page_size(hstate_vma(vma)), &pgsize); - dpfn =3D pgsize >> PAGE_SHIFT; =20 if (!__cont_access_flags_changed(ptep, pte, ncontig)) return 0; @@ -462,19 +446,14 @@ int huge_ptep_set_access_flags(struct vm_area_struct = *vma, if (pte_young(orig_pte)) pte =3D pte_mkyoung(pte); =20 - hugeprot =3D pte_pgprot(pte); - for (i =3D 0; i < ncontig; i++, ptep++, addr +=3D pgsize, pfn +=3D dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); - + __set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); return 1; } =20 void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - unsigned long pfn, dpfn; - pgprot_t hugeprot; - int ncontig, i; + int ncontig; size_t pgsize; pte_t pte; =20 @@ -487,16 +466,11 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, } =20 ncontig =3D find_num_contig(mm, addr, ptep, &pgsize); - dpfn =3D pgsize >> PAGE_SHIFT; =20 pte =3D get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); pte =3D pte_wrprotect(pte); =20 - hugeprot =3D pte_pgprot(pte); - pfn =3D pte_pfn(pte); - - for (i =3D 0; i < ncontig; i++, ptep++, addr +=3D pgsize, pfn +=3D dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + __set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); } =20 pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, @@ -505,13 +479,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, struct mm_struct *mm =3D vma->vm_mm; size_t pgsize; int ncontig; - pte_t pte; - - pte =3D __ptep_get(ptep); - VM_WARN_ON(!pte_present(pte)); - - if (!pte_cont(pte)) - return ptep_clear_flush(vma, addr, ptep); =20 ncontig =3D num_contig_ptes(huge_page_size(hstate_vma(vma)), &pgsize); return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); --=20 2.43.0