From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAC403D4101 for ; Thu, 19 Mar 2026 13:00:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925222; cv=none; b=N6/E4AA6cL/h2Y0JZqNOQiLr26z50iyP00iqFvePpCLpr2OwLBmYVqjUw+aTWl1LWGa2uWP13WscPmGNXlSM8N/2ayUd5aLeb8IxS5fKeWWZeXEJNfQfeMKA+vi8gjbLIbhhF8MDv6WohaYT3gg83NERk0AkK4ueQNPPBZFNZ2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925222; c=relaxed/simple; bh=Aa0MO4lda9XCau+hln7tuomgpTWeyeT58RJMegaGG+U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pmIiVD8/pyxKWTDpdIpdhPJMV6Mm6RinZpiARIVE5B67wGcO0wr6X8IVktgzfK6duwz/Gy4eRJRpXolR7TVQ4x16v140+G7nY1f79kZLmqurl4F0WpeqMX+INhgwv9hseSr7pUHShKf7YaD9ktbnZgokxWMW92I/l7sGZ9dmGoM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AVywcXQ4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AVywcXQ4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23D5AC19424; Thu, 19 Mar 2026 13:00:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925222; bh=Aa0MO4lda9XCau+hln7tuomgpTWeyeT58RJMegaGG+U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AVywcXQ4xyaMmk35dU9vt3sVUC8+3+RXnmasPLE7vAPaUiApGuweTyBcOqmS+aQj5 mSBtldqBENq7G4WQrYatZPICYJHIfKWzuuv5Y//N6NHpMhMUj3iXtyM6z8UTtC0FTb fY5e4QOlVZV0lnVKL25CLldjvdQY0ldGfiBsQ5Cni7I0XgQY2Eodl5nK7gUWk+aYaG cV5+bjBc/0sDjyd7/6VzgRO29ry0alPh9iHQmSWP2NaInvZLZ8NyfJFQnm4GAKwXb4 PY6oQGgVKoutrmdOO+K9QsIzWOqtSiq5DCrBd9PZ7fpwtOnTFEBELjKGDsCIXfN1dp ApHvEkXJSEt8g== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/9] mm/huge_memory: simplify vma_is_specal_huge() Date: Thu, 19 Mar 2026 13:00:07 +0000 Message-ID: <613669b1b2082d34f5632907003ae3874eff2ed9.1773924928.git.ljs@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This function is confused - it overloads the term 'special' yet again, checks for DAX but in many cases the code explicitly excludes DAX before invoking the predicate. It also unnecessarily checks for vma->vm_file - this has to be present for a driver to have set VMA_MIXEDMAP_BIT or VMA_PFNMAP_BIT. In fact, a far simpler form of this is to reverse the DAX predicate and return false if DAX is set. This makes sense from the point of view of 'special' as in vm_normal_page(), as DAX actually does potentially have retrievable folios. Also there's no need to have this in mm.h so move it to huge_memory.c. Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) --- include/linux/huge_mm.h | 4 ++-- include/linux/mm.h | 16 ---------------- mm/huge_memory.c | 30 +++++++++++++++++++++++------- 3 files changed, 25 insertions(+), 25 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index bd7f0e1d8094..61fda1672b29 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -83,7 +83,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_SPECIAL \ +#define THP_ORDERS_ALL_SPECIAL_DAX \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -92,7 +92,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAU= LT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL_DAX | THP_ORDERS_ALL_FILE_D= EFAULT) =20 enum tva_type { TVA_SMAPS, /* Exposing "THPeligible:" in smaps. */ diff --git a/include/linux/mm.h b/include/linux/mm.h index 6f0a3edb24e1..50d68b092204 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5077,22 +5077,6 @@ long copy_folio_from_user(struct folio *dst_folio, const void __user *usr_src, bool allow_pagefault); =20 -/** - * vma_is_special_huge - Are transhuge page-table entries considered speci= al? - * @vma: Pointer to the struct vm_area_struct to consider - * - * Whether transhuge page-table entries are considered "special" following - * the definition in vm_normal_page(). - * - * Return: true if transhuge page-table entries should be considered speci= al, - * false otherwise. - */ -static inline bool vma_is_special_huge(const struct vm_area_struct *vma) -{ - return vma_is_dax(vma) || (vma->vm_file && - (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))); -} - #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ =20 #if MAX_NUMNODES > 1 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3fc02913b63e..f76edfa91e96 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -100,6 +100,14 @@ static inline bool file_thp_enabled(struct vm_area_str= uct *vma) return !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode); } =20 +/* If returns true, we are unable to access the VMA's folios. */ +static bool vma_is_special_huge(struct vm_area_struct *vma) +{ + if (vma_is_dax(vma)) + return false; + return vma_test_any(vma, VMA_PFNMAP_BIT, VMA_MIXEDMAP_BIT); +} + unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, vm_flags_t vm_flags, enum tva_type type, @@ -113,8 +121,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area= _struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders =3D THP_ORDERS_ALL_ANON; - else if (vma_is_special_huge(vma)) - supported_orders =3D THP_ORDERS_ALL_SPECIAL; + else if (vma_is_dax(vma) || vma_is_special_huge(vma)) + supported_orders =3D THP_ORDERS_ALL_SPECIAL_DAX; else supported_orders =3D THP_ORDERS_ALL_FILE_DEFAULT; =20 @@ -2431,7 +2439,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_ar= ea_struct *vma, tlb->fullmm); arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); - if (!vma_is_dax(vma) && vma_is_special_huge(vma)) { + if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); @@ -2933,7 +2941,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_ar= ea_struct *vma, orig_pud =3D pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); - if (!vma_is_dax(vma) && vma_is_special_huge(vma)) { + if (vma_is_special_huge(vma)) { spin_unlock(ptl); /* No zero page support yet */ } else { @@ -3084,7 +3092,7 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, */ if (arch_needs_pgtable_deposit()) zap_deposited_table(mm, pmd); - if (!vma_is_dax(vma) && vma_is_special_huge(vma)) + if (vma_is_special_huge(vma)) return; if (unlikely(pmd_is_migration_entry(old_pmd))) { const softleaf_t old_entry =3D softleaf_from_pmd(old_pmd); @@ -4645,8 +4653,16 @@ static void split_huge_pages_all(void) =20 static inline bool vma_not_suitable_for_thp_split(struct vm_area_struct *v= ma) { - return vma_is_special_huge(vma) || (vma->vm_flags & VM_IO) || - is_vm_hugetlb_page(vma); + if (vma_is_dax(vma)) + return true; + if (vma_is_special_huge(vma)) + return true; + if (vma_test(vma, VMA_IO_BIT)) + return true; + if (is_vm_hugetlb_page(vma)) + return true; + + return false; } =20 static int split_huge_pages_pid(int pid, unsigned long vaddr_start, --=20 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B48333D4134 for ; Thu, 19 Mar 2026 13:00:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925225; cv=none; b=npcrGRbO3hI4qA5j+U06DqftVWN53S8/uP2YbNNYSaAXH73adHSyfORKuMkQg85pqWkwPj09G//jNJRRcQxXwLEfjaxH730av6x6NDLQ+zf6zwg8BjJfqxUArEJ6ysYOTJj74z3XBTj3Sd2VTzr68ZJiAGRVHtYTN5S/U4QRMF4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925225; c=relaxed/simple; bh=4y+/KaN7o4uNUZ28b17IguSawucIT3PNfW8u1Hj6ja4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XhEVA+dtEfGx8dEY98ByMlWy3PyTxHnQA+Mfq9PnYFtsduaFIOLq6Q2jSYaGqVKi7XNeqwreH74f+qKTCLnrFVVmPIJqOQP1eVsFrcHXXEljM+CqocIm4rnLZDyI/m3d0uUE+ut43LV0hqypSg+xQTe75Wcb+ZScxDVYdZgCX2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H6ekloSc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H6ekloSc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E231BC19424; Thu, 19 Mar 2026 13:00:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925225; bh=4y+/KaN7o4uNUZ28b17IguSawucIT3PNfW8u1Hj6ja4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H6ekloScPfjc7+S4h2C/Kfoe9ISQzCEiRPG+d3JF+FigoXlec6Fd81nEe8bjTT/rJ 1as6udnZgmK9AqAEYiBm7LjG+KbLDyaHxQVYPO10SGCDBhbsT3l6VGDwzz/P5LxFbT z+TzYOBq+QXc/tV/9vpw4hpjuIeSha/bnlsRzx3plJ5p8eXdQqA1KIyEaQv5OwREuk OQXPUkO411kSJJgJ7FFy6AYN6L0AU+KvqLX/TUkJomPje6/q/ab8gDsH/gJD3KY/8j kP+YZzOYTX3CL1O99luF5/DYAwEY9Cyfzj8kQe1gbLH07tECWfD0aTZTSTak0BQVMq PhWCdNo7IUg8g== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/9] mm/huge: avoid big else branch in zap_huge_pmd() Date: Thu, 19 Mar 2026 13:00:08 +0000 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We don't need to have an extra level of indentation, we can simply exit early in the first two branches. Reviewed-by: Baolin Wang Acked-by: Qi Zheng Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) --- mm/huge_memory.c | 87 +++++++++++++++++++++++++----------------------- 1 file changed, 45 insertions(+), 42 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f76edfa91e96..4ebe1f19341e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2421,8 +2421,10 @@ static inline void zap_deposited_table(struct mm_str= uct *mm, pmd_t *pmd) int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr) { - pmd_t orig_pmd; + struct folio *folio =3D NULL; + int flush_needed =3D 1; spinlock_t *ptl; + pmd_t orig_pmd; =20 tlb_change_page_size(tlb, HPAGE_PMD_SIZE); =20 @@ -2443,59 +2445,60 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_= area_struct *vma, if (arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); - } else if (is_huge_zero_pmd(orig_pmd)) { + return 1; + } + if (is_huge_zero_pmd(orig_pmd)) { if (!vma_is_dax(vma) || arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); - } else { - struct folio *folio =3D NULL; - int flush_needed =3D 1; + return 1; + } =20 - if (pmd_present(orig_pmd)) { - struct page *page =3D pmd_page(orig_pmd); + if (pmd_present(orig_pmd)) { + struct page *page =3D pmd_page(orig_pmd); =20 - folio =3D page_folio(page); - folio_remove_rmap_pmd(folio, page, vma); - WARN_ON_ONCE(folio_mapcount(folio) < 0); - VM_BUG_ON_PAGE(!PageHead(page), page); - } else if (pmd_is_valid_softleaf(orig_pmd)) { - const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); + folio =3D page_folio(page); + folio_remove_rmap_pmd(folio, page, vma); + WARN_ON_ONCE(folio_mapcount(folio) < 0); + VM_BUG_ON_PAGE(!PageHead(page), page); + } else if (pmd_is_valid_softleaf(orig_pmd)) { + const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); =20 - folio =3D softleaf_to_folio(entry); - flush_needed =3D 0; + folio =3D softleaf_to_folio(entry); + flush_needed =3D 0; =20 - if (!thp_migration_supported()) - WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); - } + if (!thp_migration_supported()) + WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); + } =20 - if (folio_test_anon(folio)) { + if (folio_test_anon(folio)) { + zap_deposited_table(tlb->mm, pmd); + add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR); + } else { + if (arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); - add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR); - } else { - if (arch_needs_pgtable_deposit()) - zap_deposited_table(tlb->mm, pmd); - add_mm_counter(tlb->mm, mm_counter_file(folio), - -HPAGE_PMD_NR); - - /* - * Use flush_needed to indicate whether the PMD entry - * is present, instead of checking pmd_present() again. - */ - if (flush_needed && pmd_young(orig_pmd) && - likely(vma_has_recency(vma))) - folio_mark_accessed(folio); - } + add_mm_counter(tlb->mm, mm_counter_file(folio), + -HPAGE_PMD_NR); =20 - if (folio_is_device_private(folio)) { - folio_remove_rmap_pmd(folio, &folio->page, vma); - WARN_ON_ONCE(folio_mapcount(folio) < 0); - folio_put(folio); - } + /* + * Use flush_needed to indicate whether the PMD entry + * is present, instead of checking pmd_present() again. + */ + if (flush_needed && pmd_young(orig_pmd) && + likely(vma_has_recency(vma))) + folio_mark_accessed(folio); + } =20 - spin_unlock(ptl); - if (flush_needed) - tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); + if (folio_is_device_private(folio)) { + folio_remove_rmap_pmd(folio, &folio->page, vma); + WARN_ON_ONCE(folio_mapcount(folio) < 0); + folio_put(folio); } + + spin_unlock(ptl); + if (flush_needed) + tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); + return 1; } =20 --=20 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65DB83D47AF for ; Thu, 19 Mar 2026 13:00:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925228; cv=none; b=X33/iCAzEYCHsYhqU2bfAMmGL1LYoqgKdk3aZq0NYNT/bZSG/BxRVQbrVolsCIoiVPmQbvgMF63jXWfQWytXCVgiqDctSQgszKiQuDJHGRDRtlmpEzDWP0IrAWs3AT7gOfaxBdWr8Ex2GqamopwFZTU/eUiAFsVo084qCGWjjHo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925228; c=relaxed/simple; bh=t/SrKTG4kMRLxaTGO20BsEclVU9dVXYVk3URzvbrQs4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hg7i8jxMt3sRBboSOEV1AwkNSKs3AX9CFjxnzrY76KH/S2xF+6aAuwIFTpCmARdBtcoU1C0aqwRNDkyNI0xCDH7UkZ/abbYPmvnBvFqnn6Zhav8reRdAj8QjPF2GbKjfMmQA0jr2RTl5l3VjGdZEuAR6j+vE2uU2g2scFQ+2+XA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WSXkbGTR; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WSXkbGTR" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E6DDC2BCB0; Thu, 19 Mar 2026 13:00:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925228; bh=t/SrKTG4kMRLxaTGO20BsEclVU9dVXYVk3URzvbrQs4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WSXkbGTR1C0mZT5R2EFaMEy7BXk4JQvJ2+6WDcLJY2aTirOp/RjlafU1yXGnyePDq td0/sLLjeF2fetOYnqUIoxG5pISvY3vcWs0ccAqAwbHTe70Vrs06ePfXbxEITsGOHD 5j+jPg9rdDdWYuUvGn2X9w3TFHh7JhDrg0WFq8Rz84TnWuQrHr+bPEgMlz7navbVvO lSFgH7WFj7Rkm3h6yJ6u2k5ifWk9/F+UB8n2+Ozyabso9SN5VMIacHXChZdk7Eg9Zs iVgcVRRkjuSYdF8WxoCaIgt5O+tWu60T3NcV9XkaO+FdD/0C+9sqpvStLlXQBJN60D irU2ZeSW5w7RQ== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/9] mm/huge_memory: have zap_huge_pmd return a boolean, add kdoc Date: Thu, 19 Mar 2026 13:00:09 +0000 Message-ID: <8bb854291383ff4713427e6d903d6d79435eb153.1773924928.git.ljs@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There's no need to use the ancient approach of returning an integer here, just return a boolean. Also update flush_needed to be a boolean, similarly. Also add a kdoc comment describing the function. Reviewed-by: Baolin Wang Acked-by: Qi Zheng Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) --- include/linux/huge_mm.h | 4 ++-- mm/huge_memory.c | 23 ++++++++++++++++------- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 61fda1672b29..2949e5acff35 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -27,8 +27,8 @@ static inline void huge_pud_set_accessed(struct vm_fault = *vmf, pud_t orig_pud) vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf); bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *= vma, pmd_t *pmd, unsigned long addr, unsigned long next); -int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t= *pmd, - unsigned long addr); +bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_= t *pmd, + unsigned long addr); int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t= *pud, unsigned long addr); bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4ebe1f19341e..bba1ba1f6b67 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2418,11 +2418,20 @@ static inline void zap_deposited_table(struct mm_st= ruct *mm, pmd_t *pmd) mm_dec_nr_ptes(mm); } =20 -int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, +/** + * zap_huge_pmd - Zap a huge THP which is of PMD size. + * @tlb: The MMU gather TLB state associated with the operation. + * @vma: The VMA containing the range to zap. + * @pmd: A pointer to the leaf PMD entry. + * @addr: The virtual address for the range to zap. + * + * Returns: %true on success, %false otherwise. + */ +bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr) { struct folio *folio =3D NULL; - int flush_needed =3D 1; + bool flush_needed =3D true; spinlock_t *ptl; pmd_t orig_pmd; =20 @@ -2430,7 +2439,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_ar= ea_struct *vma, =20 ptl =3D __pmd_trans_huge_lock(pmd, vma); if (!ptl) - return 0; + return false; /* * For architectures like ppc64 we look at deposited pgtable * when calling pmdp_huge_get_and_clear. So do the @@ -2445,13 +2454,13 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_= area_struct *vma, if (arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); - return 1; + return true; } if (is_huge_zero_pmd(orig_pmd)) { if (!vma_is_dax(vma) || arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); - return 1; + return true; } =20 if (pmd_present(orig_pmd)) { @@ -2465,7 +2474,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_ar= ea_struct *vma, const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); =20 folio =3D softleaf_to_folio(entry); - flush_needed =3D 0; + flush_needed =3D false; =20 if (!thp_migration_supported()) WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); @@ -2499,7 +2508,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_ar= ea_struct *vma, if (flush_needed) tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); =20 - return 1; + return true; } =20 #ifndef pmd_move_must_withdraw --=20 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 049073D47BC for ; Thu, 19 Mar 2026 13:00:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925231; cv=none; b=K1SRmNQMiBQxuFe1tae5oGNVmgLn7QXh7VNxYHuVKKYBkpv4T4YKtfCvVH11ffzK+hu8X4kLNNrPqcHEUmyVQ9FDqsbWhGrymPRdtFKLDh7pzhLlzdnQskX4I48jBOZz3B9mU5PGN5qHhXg3YZkFQhZw7tMwbzqYo0+qwPYYfdk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925231; c=relaxed/simple; bh=v8lQVzkXwOo8XYwemfH0T2OEyL/5prtqwnB27T50syc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GrI6kDwtUU++r5F+2a2MgJaiC83BqsgvlBqx+4iIXZTv9NXN9iwWANRFQdIamH74iCqrWEW057Cix9a8J1H9vuFx32QHZNeMGjB9YN2hLdk/+P020x7a8XQUeuhchw1uI8asaX4SwwX0grelrSVQBnRAEQ+RO+aKZlknwO/M9cU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mHFF5QM8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mHFF5QM8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74475C2BCB2; Thu, 19 Mar 2026 13:00:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925230; bh=v8lQVzkXwOo8XYwemfH0T2OEyL/5prtqwnB27T50syc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mHFF5QM8KYtYSTUl9vwuuKfFVITJcxtqGlTi62Obtb/ntJBkSFgkeXrXy4eKXorq1 56X7Hl7ESr5wFAmWaIpIImWovr77yYH/BCSq3fPGdjv9z2NOz5WNxAPOEU10ONZpWD gUHWg6DH+pv0SuY77oNzZpUZVrSa35vLxP/v4Nai+6+b8rmugbDSU8hF8EACK/URpV D99vWxvXBuAc1POpvcy+fc+HAwuiyuTOsY2zPA8RrPuVVtqF2Ah9HhY66Uzm89Giqd r3tv4ODzOkUBDkILY1T7s2Iy7ln6VjRK9QkpfDOWl4t6aEFB3t6SUQTIcQ1DQWtnu+ Gfvk8A25K/lGQ== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/9] mm/huge_memory: handle buggy PMD entry in zap_huge_pmd() Date: Thu, 19 Mar 2026 13:00:10 +0000 Message-ID: <1ec18767b106a1ee1ed6e33e332c3ad5531dc636.1773924928.git.ljs@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A recent bug I analysed [0] managed to, through a bug in the userfaultfd implementation, reach an invalid point in the zap_huge_pmd() code where the PMD was none of: - A non-DAX, PFN or mixed map. - The huge zero folio - A present PMD entry - A softleaf entry The code at this point calls folio_test_anon() on a known-NULL folio. Having logic like this explicitly NULL dereference in the code is hard to understand, and makes debugging potentially more difficult. Add an else branch to handle this case and WARN(). [0]:https://lore.kernel.org/all/6b3d7ad7-49e1-407a-903d-3103704160d8@lucife= r.local/ Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) Reviewed-by: Baolin Wang --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bba1ba1f6b67..a2f87315195d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2478,6 +2478,10 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_= area_struct *vma, if (!thp_migration_supported()) WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); + } else { + WARN_ON_ONCE(true); + spin_unlock(ptl); + return true; } if (folio_test_anon(folio)) { -- 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 325D03D4100 for ; Thu, 19 Mar 2026 13:00:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925234; cv=none; b=pLavQZgyGaa6TaZhNoKxA3p5kFjMQG9U4eiCm6yweVPpC3lYenIVWTdimjRUumM8y8g4UuQCldC+Qtvy79g3eiz36PbY1gc/zUZ3ZsKM9qY3ZVBfec2L1xeHnwibKi0dAz3vtaOIyUFmO4HsTeinY0Ylzbpaefu34+grTrez4nY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925234; c=relaxed/simple; bh=tlTUWhNPK6pHczCJVqf9Z+TZVety3K7Ng+XiEbd0tdQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sJCOPlnDNizmgg720s7jrTTvp1mcsiCJFPWjUJ/UC07k8qroQlS8NGeV5kQhfQk/IYgIcU0oQKnQ3PiqA7RdHENNJnadsmbDNYvrQWHkpdJLuX8zTvkJFi3NWWXeHNtmF1HNiGo02fZxGuy3daxa1mvBssm52wVLqvrB14HNsnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rqco5QSX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rqco5QSX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 56090C2BC9E; Thu, 19 Mar 2026 13:00:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925233; bh=tlTUWhNPK6pHczCJVqf9Z+TZVety3K7Ng+XiEbd0tdQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rqco5QSXRL2rOThabB6xWRg0bHs54Q2yKy822BvcWtOYk5p4KLkjo1KTTDm84e937 GTu9uV+rBEpSv/aBmKfG4ryN+7uqtiNeq/xNEGy6UHBAMvna1UIANatgjZ3MCYaTFE bWYDdUvIXCzJVGFkZdzzPFNxlgDRoCENORV5XCKE9vP+4n8vze4EpyzsJa9/Qb+mZ+ LMiCmGVeS6XBuucxH3x2FUFs89GOJlavSVhvSz6fWl2QuYUg6+zPrvI2NsftVgEyUP 9ikCm5FdcENujRkOeVQAMlUWUCcZq2xYPXkOLoW5PQIafIOzhROkLm9Kb1wvD3Ky70 0+mVSS7sT3Klw== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/9] mm/huge_memory: add a common exit path to zap_huge_pmd() Date: Thu, 19 Mar 2026 13:00:11 +0000 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Other than when we acquire the PTL, we always need to unlock the PTL, and optionally need to flush on exit. The code is currently very duplicated in this respect, so default flush_needed to false, set it true in the case in which it's required, then share the same logic for all exit paths. This also makes flush_needed make more sense as a function-scope value (we don't need to flush for the PFN map/mixed map, zero huge, error cases for instance). Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) Reviewed-by: Baolin Wang --- mm/huge_memory.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a2f87315195d..c84b30461cc5 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2431,7 +2431,7 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_a= rea_struct *vma, pmd_t *pmd, unsigned long addr) { struct folio *folio =3D NULL; - bool flush_needed =3D true; + bool flush_needed =3D false; spinlock_t *ptl; pmd_t orig_pmd; =20 @@ -2453,19 +2453,18 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); - spin_unlock(ptl); - return true; + goto out; } if (is_huge_zero_pmd(orig_pmd)) { if (!vma_is_dax(vma) || arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); - spin_unlock(ptl); - return true; + goto out; } =20 if (pmd_present(orig_pmd)) { struct page *page =3D pmd_page(orig_pmd); =20 + flush_needed =3D true; folio =3D page_folio(page); folio_remove_rmap_pmd(folio, page, vma); WARN_ON_ONCE(folio_mapcount(folio) < 0); @@ -2474,14 +2473,12 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); =20 folio =3D softleaf_to_folio(entry); - flush_needed =3D false; =20 if (!thp_migration_supported()) WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); } else { WARN_ON_ONCE(true); - spin_unlock(ptl); - return true; + goto out; } =20 if (folio_test_anon(folio)) { @@ -2508,10 +2505,10 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, folio_put(folio); } =20 +out: spin_unlock(ptl); if (flush_needed) tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); - return true; } =20 --=20 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E652E3D4100 for ; Thu, 19 Mar 2026 13:00:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925237; cv=none; b=TbWnINfv+d7nYyBkYlgKj8dMP7/A1SHMUO3x9C6nMi7KHTE6UrjwJj9hjj+1T7fkZxGeQvMoYuJPhU0kNNGKpfNLK5h2god4v4bqfr8OhJx3LbyMwOp/9bEevYHJ/ZlUJ1tp2MwGWcLizf7jMyNSYK1rFhvzt605oXylVYQXwHI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925237; c=relaxed/simple; bh=kspbMybLh+6i/a3mVmNNauCT4x2DdLe5LLqLRnb3Wp4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e2jtiqIfRi4iPbVZrhLrRtKW4u0ZdaQJjzmTN0XrrcHe7uAf8M/S11hUdvlu41OvI/5BK8V1EMspGfpkUDPcFsvAcmSZbTiVJtTe021rmemIKqa54cTPwUY91qaSDG4ckXsfuYUpttwRguJrZtK8kvpLlyeNDOgmvpMYPjEDN6c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H9Vr0NmT; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H9Vr0NmT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E847C2BCB1; Thu, 19 Mar 2026 13:00:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925236; bh=kspbMybLh+6i/a3mVmNNauCT4x2DdLe5LLqLRnb3Wp4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H9Vr0NmTzWFMhiAdsOMnHQ4dQ0Vgay553D0KhLhBvx3FWZt+EvoVQ7ghnu3Jiz7sG 6PC2ST2bgoPmgPo0lSUWJwDucyW3ohWywV+qU750kSjpA27k0OemOj6VdE6l/oiqw2 w4ka2IJ39V1QPp3vg+toosfQoKGw12V1oRozuTdZGM8AERILwgIwYxDO38n7c5C0Bv r3+YW1EB59c8Fk3dXZJTpxBxBRBRrHcYfrogXP7SMq5T+R3h3/l6pr8yfBphmjlfj2 MMz8dj26355XtxZ3rRyV2exYN80/h9pW3feKhvCWEHKavP0oU4PZrgpSR3e8/PYTGh tyIo3aHVF7Dmg== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 6/9] mm/huge_memory: remove unnecessary VM_BUG_ON_PAGE() Date: Thu, 19 Mar 2026 13:00:12 +0000 Message-ID: <3fcbddfbfd0408364fad4548be83455a59d17f79.1773924928.git.ljs@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This has been around since the beginnings of the THP implementation. I think we can safely assume that, if we have a THP folio, it will have a head page. Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) Reviewed-by: Baolin Wang --- mm/huge_memory.c | 1 - 1 file changed, 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c84b30461cc5..499c31bf8f83 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2468,7 +2468,6 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_a= rea_struct *vma, folio =3D page_folio(page); folio_remove_rmap_pmd(folio, page, vma); WARN_ON_ONCE(folio_mapcount(folio) < 0); - VM_BUG_ON_PAGE(!PageHead(page), page); } else if (pmd_is_valid_softleaf(orig_pmd)) { const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); =20 --=20 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22ED93D3D09 for ; Thu, 19 Mar 2026 13:00:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925240; cv=none; b=qxqdzTehjFJPUyLJ+JqUr+X9IBkKn5FcgzV2zd8OnLRX7sBmAgYGZVD8DqPdHGR5otjvYIH1ZD/Hgn2Gd8jPvbw68V8RWvpBNLuAvc37K2BjknkWmXjyK/HFoDs0xwnYMIpciENSbonW0yt1mxS1T1oN+Ee3JuWLhPsBzcDEcU4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925240; c=relaxed/simple; bh=o+lpylSbdML8A401ScmIug87esbYv315lcenpqyFup4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Bvu+d5gRESFNOZoCsrcgq+nnx3d+838cJ7ZCGStdRKvFKAD70SyY+jBlhmCNdAuo8k531AXeHZmXWxWRiVI4WfD5OUiF6RYpog7wx8h3n9VtIgbdPA7d8fMECOb4WGsYFXN3UfGotFBPbNpZILClZFOWDd1sKCXXIUPHOChPr7o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Dxiad9S3; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Dxiad9S3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 99CF5C2BCAF; Thu, 19 Mar 2026 13:00:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925240; bh=o+lpylSbdML8A401ScmIug87esbYv315lcenpqyFup4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Dxiad9S3H0CJ/OOlBfydIz2ItScNVD7AEOxS3SlYNwW60Uk/Z+vxaOyBwqCkUD2n0 GnnEjEwxc65D7gnXBahnHYzCkUdopJ3LqJSQPawb0m0fn65GS/dNeOuCEt/1HmBmPN eS8gjkoANr/PtXnPb5I3bdwKXXr3plFPhtJ+IrtlX8LkJDxPp05uBpRqrlyXUGzO// 5z/nbjuZfWoPvaiVW4oIDndmEvIKCDCb1aG5dbLgufq24cDCgT9hD4s733AwYKsE5C sGTxypHdf7hEN72j4kvD8Am5VZG+CVOsm1vOVF7hlyrQQgIXU4MwQCzhmzF7l3xImm JAf/psSuUgplw== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 7/9] mm/huge_memory: deduplicate zap deposited table call Date: Thu, 19 Mar 2026 13:00:13 +0000 Message-ID: <29fa4129cd560681c67433b4c344b2c9a027d1b4.1773924928.git.ljs@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rather than having separate logic for each case determining whether to zap the deposited table, simply track this via a boolean. We check separately if the architecture requires it. Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) --- mm/huge_memory.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 499c31bf8f83..c4e00c645e58 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2431,6 +2431,7 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_a= rea_struct *vma, pmd_t *pmd, unsigned long addr) { struct folio *folio =3D NULL; + bool needs_deposit =3D false; bool flush_needed =3D false; spinlock_t *ptl; pmd_t orig_pmd; @@ -2450,23 +2451,18 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, tlb->fullmm); arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); - if (vma_is_special_huge(vma)) { - if (arch_needs_pgtable_deposit()) - zap_deposited_table(tlb->mm, pmd); + if (vma_is_special_huge(vma)) goto out; - } if (is_huge_zero_pmd(orig_pmd)) { - if (!vma_is_dax(vma) || arch_needs_pgtable_deposit()) - zap_deposited_table(tlb->mm, pmd); + needs_deposit =3D !vma_is_dax(vma); goto out; } =20 if (pmd_present(orig_pmd)) { - struct page *page =3D pmd_page(orig_pmd); + folio =3D pmd_folio(orig_pmd); =20 flush_needed =3D true; - folio =3D page_folio(page); - folio_remove_rmap_pmd(folio, page, vma); + folio_remove_rmap_pmd(folio, &folio->page, vma); WARN_ON_ONCE(folio_mapcount(folio) < 0); } else if (pmd_is_valid_softleaf(orig_pmd)) { const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); @@ -2481,11 +2477,9 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_= area_struct *vma, } =20 if (folio_test_anon(folio)) { - zap_deposited_table(tlb->mm, pmd); + needs_deposit =3D true; add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR); } else { - if (arch_needs_pgtable_deposit()) - zap_deposited_table(tlb->mm, pmd); add_mm_counter(tlb->mm, mm_counter_file(folio), -HPAGE_PMD_NR); =20 @@ -2505,6 +2499,9 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_a= rea_struct *vma, } =20 out: + if (arch_needs_pgtable_deposit() || needs_deposit) + zap_deposited_table(tlb->mm, pmd); + spin_unlock(ptl); if (flush_needed) tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); --=20 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A7C23D47BA for ; Thu, 19 Mar 2026 13:00:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925243; cv=none; b=PO6skK7ZE/GscicPksXUQwZMWPc7V7WvKRH5CxfxIHIghfwaslbxpWHjWIgAPeVJS8g4YgSI1Sxn+4MHtBjgNOobfo2onuvLI5SjSEgG2ugTxDSInVoyOyIBqLh19cARsA3sucZ9jjEv5NHcBI7O8gAq268GtOaj8+O0ExCBKpM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925243; c=relaxed/simple; bh=jwAcjIvj94lIaFqSDU5rGJUwe2UvHWTByimGr9pyvys=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vc8Yzb6RZi/97w4GwOJDiKWakuGPZKVdZplZO/yVH/ZOrlz5dwdLe3k76QEKuek69hgW8N1KTRE4xOYP9GpVxxnZrhiADrdwVBwWHGJfhjg+j1uaLniNh964Zf6V42pgF+f6/Pc+bB5h/cq31ePk5YGiHjiQGlDqMQ3EeKSOosM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nkialUqi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nkialUqi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E223C2BCAF; Thu, 19 Mar 2026 13:00:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925242; bh=jwAcjIvj94lIaFqSDU5rGJUwe2UvHWTByimGr9pyvys=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nkialUqirVob79VJcXVODpVto2tm0ZpXaRVx2iJtHiGVZ9MOVVMaxvmeFqBDxo7a5 FvQmMrYASCI/q9JwLy2KsJH+UDaCTyu2OO/wy5a0Ouy9ZPD0F5+GF9LybkX3ifvCjf k5swhZPcrmvw9Q7rdVgFgdVjv0pZUMnX96G96lj9+hd3xUZjictASE6ENJbrMJ/mwC qy5/5wGs5oVqpLx/U1YUiige/3UllKiHQRvOGACxLyT1bfLhzKKWAwkiScuqJWmFzd G2t1N0wO17Paz8QplizaFaZox+jE2IXxkKsCzgEPW/ZbBlrsmkUzh7rjsEOE7N6vMZ gcVUBIOFrTwJA== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 8/9] mm/huge_memory: deduplicate zap_huge_pmd() further by tracking state Date: Thu, 19 Mar 2026 13:00:14 +0000 Message-ID: <440f68edcb597c28918d89b0be279d498561c89a.1773924928.git.ljs@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The flush_needed boolean is really tracking whether a PMD entry is present, so use it that way directly and rename it to is_present. Deduplicate the folio_remove_rmap_pmd() and folio map count warning between present and device private by tracking where we need to remove the rmap. We can also remove the comment about using flush_needed to track whether a PMD entry is present as it's now irrelevant. Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) --- mm/huge_memory.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c4e00c645e58..22715027e56c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2430,9 +2430,10 @@ static inline void zap_deposited_table(struct mm_str= uct *mm, pmd_t *pmd) bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr) { + bool needs_remove_rmap =3D false; struct folio *folio =3D NULL; bool needs_deposit =3D false; - bool flush_needed =3D false; + bool is_present =3D false; spinlock_t *ptl; pmd_t orig_pmd; =20 @@ -2449,6 +2450,7 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_a= rea_struct *vma, */ orig_pmd =3D pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) @@ -2458,17 +2460,15 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, goto out; } =20 - if (pmd_present(orig_pmd)) { + is_present =3D pmd_present(orig_pmd); + if (is_present) { folio =3D pmd_folio(orig_pmd); - - flush_needed =3D true; - folio_remove_rmap_pmd(folio, &folio->page, vma); - WARN_ON_ONCE(folio_mapcount(folio) < 0); + needs_remove_rmap =3D true; } else if (pmd_is_valid_softleaf(orig_pmd)) { const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); =20 folio =3D softleaf_to_folio(entry); - + needs_remove_rmap =3D folio_is_device_private(folio); if (!thp_migration_supported()) WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); } else { @@ -2483,27 +2483,25 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, add_mm_counter(tlb->mm, mm_counter_file(folio), -HPAGE_PMD_NR); =20 - /* - * Use flush_needed to indicate whether the PMD entry - * is present, instead of checking pmd_present() again. - */ - if (flush_needed && pmd_young(orig_pmd) && + if (is_present && pmd_young(orig_pmd) && likely(vma_has_recency(vma))) folio_mark_accessed(folio); } =20 - if (folio_is_device_private(folio)) { + if (needs_remove_rmap) { folio_remove_rmap_pmd(folio, &folio->page, vma); WARN_ON_ONCE(folio_mapcount(folio) < 0); - folio_put(folio); } =20 out: if (arch_needs_pgtable_deposit() || needs_deposit) zap_deposited_table(tlb->mm, pmd); =20 + if (needs_remove_rmap && !is_present) + folio_put(folio); + spin_unlock(ptl); - if (flush_needed) + if (is_present) tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); return true; } --=20 2.53.0 From nobody Mon Apr 6 11:53:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 430C03D47C1 for ; Thu, 19 Mar 2026 13:00:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925246; cv=none; b=UyWGGaB8qIg+tfZbFZfOTl31ZyhLG3xBYwnMhEk/T8PCWPaW5j6Q8JwSX3vosXSPg75B6uzmxkUD0GMMwkL1IFNNP+p8A6QLujUmRun6u/58Nm4SWueegkoB1ld6ajBznrCiFN+s9QwbngI5PebweySOhgxam9ctiNtWIB8sEXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773925246; c=relaxed/simple; bh=9GL71TGtr4Xl48AxLVp+Sw8hVkXKWEDVrPJDeJyAQmU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=acwT0SwAmJ3HuY1EZibpnfl32wNuf2ZKk7AspKUtI8SMhFIV9C4ycsy30lfHdZ4jeRAxzBuzs9QeZD+sazuNdY47kj5K40VzSrLG7thUcmAGBA1ctcQEdEWLWeeLKmiGzUTa2Ln0Sc70bWfhttgev1p2TvyiC29moNTV0Fjduh4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OA9gxtDY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OA9gxtDY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F9DAC2BC9E; Thu, 19 Mar 2026 13:00:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773925245; bh=9GL71TGtr4Xl48AxLVp+Sw8hVkXKWEDVrPJDeJyAQmU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OA9gxtDYCARiW/xFj3Sq62/bn+4nR4RBKTIioCsZQ28gTNNA9AcNWYSFoBaWEpGas LddWivmQh6y897zbvCavLhQY8U+wF9VqufAyVfsIP/KcCpP43VuFcu27fwppyYMCRq ZmpwL6pd5CoX+a6a/TfzSb9tBw6ND/qHm2ufVxR+5H6hAtMLSvB7BKQSb7tGtPRsYz UHA9O3fRmq76zqwEex9NpH2fefSW0+bYKjCNzvxYbanyFd4ankQcsFD4fPts6tIUW2 uJMQWIrTHVw33/FrL+K6yWQWUyu+e10K9vnX0fXZwvXoBwZz97LTIsVJvBrfYJMLQS bM5OYt36Az/9A== From: "Lorenzo Stoakes (Oracle)" To: Andrew Morton Cc: David Hildenbrand , Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 9/9] mm/huge_memory: have zap_huge_pmd() use vm_normal_folio_pmd() Date: Thu, 19 Mar 2026 13:00:15 +0000 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rather than having special handling for the 'special huge' and huge zero pages, have vm_normal_folio_pmd() figure this out for us, and use the presence of a folio to determine whether to exit early. We can therefore delete code, move the 'present or not' logic up and make things clearer this way. Suggested-by: David Hildenbrand (ARM) Signed-off-by: Lorenzo Stoakes (Oracle) Reviewed-by tags despite the effort people (especially David and Lorenzo) --- mm/huge_memory.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 22715027e56c..5ffe3334c80d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2431,9 +2431,9 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_a= rea_struct *vma, pmd_t *pmd, unsigned long addr) { bool needs_remove_rmap =3D false; - struct folio *folio =3D NULL; bool needs_deposit =3D false; bool is_present =3D false; + struct folio *folio; spinlock_t *ptl; pmd_t orig_pmd; =20 @@ -2453,28 +2453,26 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, =20 arch_check_zapped_pmd(vma, orig_pmd); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); - if (vma_is_special_huge(vma)) - goto out; - if (is_huge_zero_pmd(orig_pmd)) { - needs_deposit =3D !vma_is_dax(vma); - goto out; - } =20 - is_present =3D pmd_present(orig_pmd); - if (is_present) { - folio =3D pmd_folio(orig_pmd); - needs_remove_rmap =3D true; + if (pmd_present(orig_pmd)) { + folio =3D vm_normal_folio_pmd(vma, addr, orig_pmd); + if (folio) { + needs_remove_rmap =3D true; + is_present =3D true; + } else if (is_huge_zero_pmd(orig_pmd)) { + needs_deposit =3D !vma_is_dax(vma); + } } else if (pmd_is_valid_softleaf(orig_pmd)) { - const softleaf_t entry =3D softleaf_from_pmd(orig_pmd); - - folio =3D softleaf_to_folio(entry); + folio =3D softleaf_to_folio(softleaf_from_pmd(orig_pmd)); needs_remove_rmap =3D folio_is_device_private(folio); if (!thp_migration_supported()) WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); } else { WARN_ON_ONCE(true); - goto out; + folio =3D NULL; } + if (!folio) + goto out; =20 if (folio_test_anon(folio)) { needs_deposit =3D true; --=20 2.53.0