From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-100.ptr.blmpb.com (sg-1-100.ptr.blmpb.com [118.26.132.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FE6D313264 for ; Wed, 7 Jan 2026 11:34:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785685; cv=none; b=gcDsyV3qFjD22SbXyt6KY/YDC+aoQ0JXz63DOd+SbaDaPYPmlUwxCHhcaIqQTxKisXE2JgNyEDSj8leNLf9xvMGVvDbMJSh4Nd1aEvsiPtMeWx55SIH9vRtvM+QWiyNP6/55blQHxEbsZULuUPEq+wOUPlbRYf5JO1nQdfHaFE8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785685; c=relaxed/simple; bh=8bLwkHiRc0F0XoO4sNj8YqBvGAd27HJCuE936Q+4K5o=; h=In-Reply-To:Message-Id:Mime-Version:References:From:Subject:To:Cc: Date:Content-Type; b=OqBcclGu/aOLRD/Icjjiy8CdlY3Y/rlUVZMqhsvlHvar7YqE4lfqSdnt5ggYRzVDVMbNxYNDDmEK7RkryqiQH3oipXTU3FpGtRa6CfOsKWRYIY8CSGDoTLCWlEkAlkvFmX7GqL7grdIUVKGkoKqhUSFtp/8Sds2KutiN0mreZO0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QhuMMV+B; arc=none smtp.client-ip=118.26.132.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QhuMMV+B" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785562; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=GCbQrsy9EGhaI41sv7J59xf0PzXpHKfWp8oxq4i5ozI=; b=QhuMMV+B4Vw3PRjwNlR73heSQvK2VCFGs4yhGQRppwtijM3cPzyQGI4Vh1UpnQz4r8mTY0 FxCtdfkCz3BSq4QIcZa5FLTBgh966n/Rc5F3+cAN1FCSStL/sx1ybBjlDVttweuzo3JJdo ILg1mHMlSEzv+V2JC7//bZojfaXmMiTVRlRuyKdO5ZTUz2QGveji1ojPh6/Eo2HdjkqLmE uxLPSm6e0vynRWXGjPfyRcPgruXIi5U01cx6TXIRKdTNv5TIcbzC1FVwEWMH475vTd3a6X SsJQd8ybEay5c9MZd6WEeC6nuKmaO/QyL/UQpCqJH5vkZSDg1VMKtlaX9WMT9w== In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> Message-Id: <20260107113130.37231-2-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260107113130.37231-1-lizhe.67@bytedance.com> From: "Li Zhe" Subject: [PATCH v2 1/8] mm/hugetlb: add pre-zeroed framework Content-Transfer-Encoding: quoted-printable X-Original-From: Li Zhe X-Mailer: git-send-email 2.45.2 To: , , , , Cc: , , Date: Wed, 7 Jan 2026 19:31:23 +0800 X-Lms-Return-Path: Content-Type: text/plain; charset="utf-8" This patch establishes a pre-zeroing framework by introducing two new hugetlb page flags and extends the code at every point where these flags may later be required. The roles of the two flags are as follows. (1) HPG_zeroed =E2=80=93 indicates that the huge folio has already been zeroed (2) HPG_zeroing =E2=80=93 marks that the huge folio is currently being zero= ed No functional change, as nothing sets the flags yet. Co-developed-by: Frank van der Linden Signed-off-by: Frank van der Linden Signed-off-by: Li Zhe --- fs/hugetlbfs/inode.c | 3 +- include/linux/hugetlb.h | 26 ++++++++++ mm/hugetlb.c | 111 +++++++++++++++++++++++++++++++++++++--- 3 files changed, 131 insertions(+), 9 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 3b4c152c5c73..be6b32ab3ca8 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -828,8 +828,7 @@ static long hugetlbfs_fallocate(struct file *file, int = mode, loff_t offset, error =3D PTR_ERR(folio); goto out; } - folio_zero_user(folio, addr); - __folio_mark_uptodate(folio); + hugetlb_zero_folio(folio, addr); error =3D hugetlb_add_to_page_cache(folio, mapping, index); if (unlikely(error)) { restore_reserve_on_error(h, &pseudo_vma, addr, folio); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 019a1c5281e4..2daf4422a17d 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -584,6 +584,17 @@ hugetlb_get_unmapped_area(struct file *file, unsigned = long addr, * HPG_vmemmap_optimized - Set when the vmemmap pages of the page are free= d. * HPG_raw_hwp_unreliable - Set when the hugetlb page has a hwpoison sub-p= age * that is not tracked by raw_hwp_page list. + * HPG_zeroed - page was pre-zeroed. + * Synchronization: hugetlb_lock held when set by pre-zero thread. + * Only valid to read outside hugetlb_lock once the page is off + * the freelist, and HPG_zeroing is clear. Always cleared when a + * page is put (back) on the freelist. + * HPG_zeroing - page is being zeroed by the pre-zero thread. + * Synchronization: set and cleared by the pre-zero thread with + * hugetlb_lock held. Access by others is read-only. Once the page + * is off the freelist, this can only change from set -> clear, + * which the new page owner must wait for. Always cleared + * when a page is put (back) on the freelist. */ enum hugetlb_page_flags { HPG_restore_reserve =3D 0, @@ -593,6 +604,8 @@ enum hugetlb_page_flags { HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, HPG_cma, + HPG_zeroed, + HPG_zeroing, __NR_HPAGEFLAGS, }; =20 @@ -653,6 +666,8 @@ HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) HPAGEFLAG(Cma, cma) +HPAGEFLAG(Zeroed, zeroed) +HPAGEFLAG(Zeroing, zeroing) =20 #ifdef CONFIG_HUGETLB_PAGE =20 @@ -678,6 +693,12 @@ struct hstate { unsigned int nr_huge_pages_node[MAX_NUMNODES]; unsigned int free_huge_pages_node[MAX_NUMNODES]; unsigned int surplus_huge_pages_node[MAX_NUMNODES]; + + unsigned int free_huge_pages_zero_node[MAX_NUMNODES]; + + /* Queue to wait for a hugetlb folio that is being prezeroed */ + wait_queue_head_t dqzero_wait[MAX_NUMNODES]; + char name[HSTATE_NAME_LEN]; }; =20 @@ -711,6 +732,7 @@ int hugetlb_add_to_page_cache(struct folio *folio, stru= ct address_space *mapping pgoff_t idx); void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, unsigned long address, struct folio *folio); +void hugetlb_zero_folio(struct folio *folio, unsigned long address); =20 /* arch callback */ int __init __alloc_bootmem_huge_page(struct hstate *h, int nid); @@ -1303,6 +1325,10 @@ static inline bool hugetlb_bootmem_allocated(void) { return false; } + +static inline void hugetlb_zero_folio(struct folio *folio, unsigned long a= ddress) +{ +} #endif /* CONFIG_HUGETLB_PAGE */ =20 static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 51273baec9e5..001fc0ed4c48 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -93,6 +93,8 @@ static int hugetlb_param_index __initdata; static __init int hugetlb_add_param(char *s, int (*setup)(char *val)); static __init void hugetlb_parse_params(void); =20 +static void hpage_wait_zeroing(struct hstate *h, struct folio *folio); + #define hugetlb_early_param(str, func) \ static __init int func##args(char *s) \ { \ @@ -1292,21 +1294,33 @@ void clear_vma_resv_huge_pages(struct vm_area_struc= t *vma) hugetlb_dup_vma_private(vma); } =20 +/* + * Clear flags for either a fresh page or one that is being + * added to the free list. + */ +static inline void prep_clear_zeroed(struct folio *folio) +{ + folio_clear_hugetlb_zeroed(folio); + folio_clear_hugetlb_zeroing(folio); +} + static void enqueue_hugetlb_folio(struct hstate *h, struct folio *folio) { int nid =3D folio_nid(folio); =20 lockdep_assert_held(&hugetlb_lock); VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); + VM_WARN_ON_FOLIO(folio_test_hugetlb_zeroing(folio), folio); =20 list_move(&folio->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; + prep_clear_zeroed(folio); folio_set_hugetlb_freed(folio); } =20 -static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h, - int nid) +static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h, in= t nid, + gfp_t gfp_mask) { struct folio *folio; bool pin =3D !!(current->flags & PF_MEMALLOC_PIN); @@ -1316,6 +1330,16 @@ static struct folio *dequeue_hugetlb_folio_node_exac= t(struct hstate *h, if (pin && !folio_is_longterm_pinnable(folio)) continue; =20 + /* + * This shouldn't happen, as hugetlb pages are never allocated + * with GFP_ATOMIC. But be paranoid and check for it, as + * a zero_busy page might cause a sleep later in + * hpage_wait_zeroing(). + */ + if (WARN_ON_ONCE(folio_test_hugetlb_zeroing(folio) && + !gfpflags_allow_blocking(gfp_mask))) + continue; + if (folio_test_hwpoison(folio)) continue; =20 @@ -1327,6 +1351,10 @@ static struct folio *dequeue_hugetlb_folio_node_exac= t(struct hstate *h, folio_clear_hugetlb_freed(folio); h->free_huge_pages--; h->free_huge_pages_node[nid]--; + if (folio_test_hugetlb_zeroed(folio) || + folio_test_hugetlb_zeroing(folio)) + h->free_huge_pages_zero_node[nid]--; + return folio; } =20 @@ -1363,7 +1391,7 @@ static struct folio *dequeue_hugetlb_folio_nodemask(s= truct hstate *h, gfp_t gfp_ continue; node =3D zone_to_nid(zone); =20 - folio =3D dequeue_hugetlb_folio_node_exact(h, node); + folio =3D dequeue_hugetlb_folio_node_exact(h, node, gfp_mask); if (folio) return folio; } @@ -1490,7 +1518,16 @@ void remove_hugetlb_folio(struct hstate *h, struct f= olio *folio, folio_clear_hugetlb_freed(folio); h->free_huge_pages--; h->free_huge_pages_node[nid]--; + folio_clear_hugetlb_freed(folio); } + /* + * Adjust the zero page counters now. Note that + * if a page is currently being zeroed, that + * will be waited for in update_and_free_page() + */ + if (folio_test_hugetlb_zeroed(folio) || + folio_test_hugetlb_zeroing(folio)) + h->free_huge_pages_zero_node[nid]--; if (adjust_surplus) { h->surplus_huge_pages--; h->surplus_huge_pages_node[nid]--; @@ -1543,6 +1580,8 @@ static void __update_and_free_hugetlb_folio(struct hs= tate *h, { bool clear_flag =3D folio_test_hugetlb_vmemmap_optimized(folio); =20 + VM_WARN_ON_FOLIO(folio_test_hugetlb_zeroing(folio), folio); + if (hstate_is_gigantic_no_runtime(h)) return; =20 @@ -1627,6 +1666,7 @@ static void free_hpage_workfn(struct work_struct *wor= k) */ h =3D size_to_hstate(folio_size(folio)); =20 + hpage_wait_zeroing(h, folio); __update_and_free_hugetlb_folio(h, folio); =20 cond_resched(); @@ -1643,7 +1683,8 @@ static inline void flush_free_hpage_work(struct hstat= e *h) static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *= folio, bool atomic) { - if (!folio_test_hugetlb_vmemmap_optimized(folio) || !atomic) { + if ((!folio_test_hugetlb_zeroing(folio) && + !folio_test_hugetlb_vmemmap_optimized(folio)) || !atomic) { __update_and_free_hugetlb_folio(h, folio); return; } @@ -1840,6 +1881,13 @@ static void account_new_hugetlb_folio(struct hstate = *h, struct folio *folio) h->nr_huge_pages_node[folio_nid(folio)]++; } =20 +static void prep_new_hugetlb_folio(struct folio *folio) +{ + lockdep_assert_held(&hugetlb_lock); + folio_clear_hugetlb_freed(folio); + prep_clear_zeroed(folio); +} + void init_new_hugetlb_folio(struct folio *folio) { __folio_set_hugetlb(folio); @@ -1964,6 +2012,7 @@ void prep_and_add_allocated_folios(struct hstate *h, /* Add all new pool pages to free lists in one lock cycle */ spin_lock_irqsave(&hugetlb_lock, flags); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + prep_new_hugetlb_folio(folio); account_new_hugetlb_folio(h, folio); enqueue_hugetlb_folio(h, folio); } @@ -2171,6 +2220,7 @@ static struct folio *alloc_surplus_hugetlb_folio(stru= ct hstate *h, return NULL; =20 spin_lock_irq(&hugetlb_lock); + prep_new_hugetlb_folio(folio); /* * nr_huge_pages needs to be adjusted within the same lock cycle * as surplus_pages, otherwise it might confuse @@ -2214,6 +2264,7 @@ static struct folio *alloc_migrate_hugetlb_folio(stru= ct hstate *h, gfp_t gfp_mas return NULL; =20 spin_lock_irq(&hugetlb_lock); + prep_new_hugetlb_folio(folio); account_new_hugetlb_folio(h, folio); spin_unlock_irq(&hugetlb_lock); =20 @@ -2289,6 +2340,13 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hs= tate *h, int preferred_nid, preferred_nid, nmask); if (folio) { spin_unlock_irq(&hugetlb_lock); + /* + * The contents of this page will be completely + * overwritten immediately, as its a migration + * target, so no clearing is needed. Do wait in + * case pre-zero thread was working on it, though. + */ + hpage_wait_zeroing(h, folio); return folio; } } @@ -2779,6 +2837,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct fo= lio *old_folio, */ remove_hugetlb_folio(h, old_folio, false); =20 + prep_new_hugetlb_folio(new_folio); /* * Ref count on new_folio is already zero as it was dropped * earlier. It can be directly added to the pool free list. @@ -2999,6 +3058,8 @@ struct folio *alloc_hugetlb_folio(struct vm_area_stru= ct *vma, =20 spin_unlock_irq(&hugetlb_lock); =20 + hpage_wait_zeroing(h, folio); + hugetlb_set_folio_subpool(folio, spool); =20 if (map_chg !=3D MAP_CHG_ENFORCED) { @@ -3257,6 +3318,7 @@ static void __init prep_and_add_bootmem_folios(struct= hstate *h, hugetlb_bootmem_init_migratetype(folio, h); /* Subdivide locks to achieve better parallel performance */ spin_lock_irqsave(&hugetlb_lock, flags); + prep_new_hugetlb_folio(folio); account_new_hugetlb_folio(h, folio); enqueue_hugetlb_folio(h, folio); spin_unlock_irqrestore(&hugetlb_lock, flags); @@ -4190,6 +4252,40 @@ bool __init __attribute((weak)) arch_hugetlb_valid_s= ize(unsigned long size) return size =3D=3D HPAGE_SIZE; } =20 +/* + * Zero a hugetlb page. + * + * The caller has already made sure that the page is not + * being actively zeroed out in the background. + * + * If it wasn't zeroed out, do it ourselves. + */ +void hugetlb_zero_folio(struct folio *folio, unsigned long address) +{ + if (!folio_test_hugetlb_zeroed(folio)) + folio_zero_user(folio, address); + + __folio_mark_uptodate(folio); +} + +/* + * Once a page has been taken off the freelist, the new page owner + * must wait for the pre-zero thread to finish if it happens + * to be working on this page (which should be rare). + */ +static void hpage_wait_zeroing(struct hstate *h, struct folio *folio) +{ + if (!folio_test_hugetlb_zeroing(folio)) + return; + + guard(spinlock)(&hugetlb_lock); + + wait_event_cmd(h->dqzero_wait[folio_nid(folio)], + !folio_test_hugetlb_zeroing(folio), + spin_unlock_irq(&hugetlb_lock), + spin_lock_irq(&hugetlb_lock)); +} + void __init hugetlb_add_hstate(unsigned int order) { struct hstate *h; @@ -4205,8 +4301,10 @@ void __init hugetlb_add_hstate(unsigned int order) __mutex_init(&h->resize_lock, "resize mutex", &h->resize_key); h->order =3D order; h->mask =3D ~(huge_page_size(h) - 1); - for (i =3D 0; i < MAX_NUMNODES; ++i) + for (i =3D 0; i < MAX_NUMNODES; ++i) { INIT_LIST_HEAD(&h->hugepage_freelists[i]); + init_waitqueue_head(&h->dqzero_wait[i]); + } INIT_LIST_HEAD(&h->hugepage_activelist); snprintf(h->name, HSTATE_NAME_LEN, "hugepages-%lukB", huge_page_size(h)/SZ_1K); @@ -5804,8 +5902,7 @@ static vm_fault_t hugetlb_no_page(struct address_spac= e *mapping, ret =3D 0; goto out; } - folio_zero_user(folio, vmf->real_address); - __folio_mark_uptodate(folio); + hugetlb_zero_folio(folio, vmf->address); new_folio =3D true; =20 if (vma->vm_flags & VM_MAYSHARE) { --=20 2.20.1 From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-100.ptr.blmpb.com (sg-1-100.ptr.blmpb.com [118.26.132.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 670291D5160 for ; Wed, 7 Jan 2026 11:33:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785600; cv=none; b=k/EIjBjRpa/TYcgQn2Gryu2ojHoBsr0uXMJmdW0CjctkH3YnAF4Ww7tCe2ZgMD9t1TgrIJ30/zF+PBAD33CBBb0f6UuXiMImn+OZn7rYECqMbxsSdX5+vzEkJpLWqD2AH09chACnKgjZL/GmibcqcnIQRxGhsYCkmI9HxCufjFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785600; c=relaxed/simple; bh=YiOkTG+9jjbhwfXqK9o5ZlgnbVymwObS/c8XdfVwzvM=; h=Mime-Version:From:Date:Cc:Subject:References:In-Reply-To: Content-Type:To:Message-Id; b=KZ796A3emgLUeWIvBu/1DweM3PQlbtv9Ux9wkfeFXf2HZSxc86n5uTZGPhioP/VK9d7AxB9FujrgmcVP9Ff9OK4+WCBUAlBOTdFpfM0MombVWx4PXEC28a4U9oZz0q0/ro7UzwiqVTyGW3HOW+6pEBRrwKsyBdjcCMHzMd/yljU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PmpX/nDy; arc=none smtp.client-ip=118.26.132.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PmpX/nDy" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785585; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=N8AGG1ZCnIMLj3cPqWeKg86cRrsdjc3RFAx/nQ6Si9A=; b=PmpX/nDyt63mqnGxwjy7PsO2mBIGjTYevkQ1FCWBXEKGb9y6RkKuZCUfB17+ZF+ZypGrc5 dWBw5IJbWKQ/gPBQfrIalNDWcd9K+Er2KhQjZcVHXAgf8EwNjf9sH0tujkyCYLrWFhJjI4 GbyQWAOwwDeEmX4MjLiSESuWvOox2QXNRjcYNG3VzxpjPE7yjxrQUKxsme7ceH1YovTQNH Donjot/ayfX+Cx8PAG3tiGhAveuHE85Kensog6wJHA5KEH6io2M0bdej0AS0q7py3wc8B1 FZNpT2ta4MBV/4KNNGCaWJb+26QckTObnNPV6rzIIol7pYmjG5xNffuKSnwZSA== Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable From: "Li Zhe" Date: Wed, 7 Jan 2026 19:31:24 +0800 X-Mailer: git-send-email 2.45.2 X-Original-From: Li Zhe X-Lms-Return-Path: Cc: , , Subject: [PATCH v2 2/8] mm/hugetlb: convert to prep_account_new_hugetlb_folio() References: <20260107113130.37231-1-lizhe.67@bytedance.com> In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> To: , , , , Message-Id: <20260107113130.37231-3-lizhe.67@bytedance.com> Content-Type: text/plain; charset="utf-8" After a huge folio is instantiated, it is always initialized through the successive calls to prep_new_hugetlb_folio() and account_new_hugetlb_folio(). To eliminate the risk that future changes update one routine but overlook the other, the two functions have been consolidated into a single entry point prep_account_new_hugetlb_folio(). Signed-off-by: Li Zhe --- mm/hugetlb.c | 29 ++++++++++------------------- 1 file changed, 10 insertions(+), 19 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 001fc0ed4c48..a7e582abe9f9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1874,18 +1874,14 @@ void free_huge_folio(struct folio *folio) /* * Must be called with the hugetlb lock held */ -static void account_new_hugetlb_folio(struct hstate *h, struct folio *foli= o) -{ - lockdep_assert_held(&hugetlb_lock); - h->nr_huge_pages++; - h->nr_huge_pages_node[folio_nid(folio)]++; -} - -static void prep_new_hugetlb_folio(struct folio *folio) +static void prep_account_new_hugetlb_folio(struct hstate *h, + struct folio *folio) { lockdep_assert_held(&hugetlb_lock); folio_clear_hugetlb_freed(folio); prep_clear_zeroed(folio); + h->nr_huge_pages++; + h->nr_huge_pages_node[folio_nid(folio)]++; } =20 void init_new_hugetlb_folio(struct folio *folio) @@ -2012,8 +2008,7 @@ void prep_and_add_allocated_folios(struct hstate *h, /* Add all new pool pages to free lists in one lock cycle */ spin_lock_irqsave(&hugetlb_lock, flags); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { - prep_new_hugetlb_folio(folio); - account_new_hugetlb_folio(h, folio); + prep_account_new_hugetlb_folio(h, folio); enqueue_hugetlb_folio(h, folio); } spin_unlock_irqrestore(&hugetlb_lock, flags); @@ -2220,13 +2215,12 @@ static struct folio *alloc_surplus_hugetlb_folio(st= ruct hstate *h, return NULL; =20 spin_lock_irq(&hugetlb_lock); - prep_new_hugetlb_folio(folio); /* * nr_huge_pages needs to be adjusted within the same lock cycle * as surplus_pages, otherwise it might confuse * persistent_huge_pages() momentarily. */ - account_new_hugetlb_folio(h, folio); + prep_account_new_hugetlb_folio(h, folio); =20 /* * We could have raced with the pool size change. @@ -2264,8 +2258,7 @@ static struct folio *alloc_migrate_hugetlb_folio(stru= ct hstate *h, gfp_t gfp_mas return NULL; =20 spin_lock_irq(&hugetlb_lock); - prep_new_hugetlb_folio(folio); - account_new_hugetlb_folio(h, folio); + prep_account_new_hugetlb_folio(h, folio); spin_unlock_irq(&hugetlb_lock); =20 /* fresh huge pages are frozen */ @@ -2831,18 +2824,17 @@ static int alloc_and_dissolve_hugetlb_folio(struct = folio *old_folio, /* * Ok, old_folio is still a genuine free hugepage. Remove it from * the freelist and decrease the counters. These will be - * incremented again when calling account_new_hugetlb_folio() + * incremented again when calling prep_account_new_hugetlb_folio() * and enqueue_hugetlb_folio() for new_folio. The counters will * remain stable since this happens under the lock. */ remove_hugetlb_folio(h, old_folio, false); =20 - prep_new_hugetlb_folio(new_folio); /* * Ref count on new_folio is already zero as it was dropped * earlier. It can be directly added to the pool free list. */ - account_new_hugetlb_folio(h, new_folio); + prep_account_new_hugetlb_folio(h, new_folio); enqueue_hugetlb_folio(h, new_folio); =20 /* @@ -3318,8 +3310,7 @@ static void __init prep_and_add_bootmem_folios(struct= hstate *h, hugetlb_bootmem_init_migratetype(folio, h); /* Subdivide locks to achieve better parallel performance */ spin_lock_irqsave(&hugetlb_lock, flags); - prep_new_hugetlb_folio(folio); - account_new_hugetlb_folio(h, folio); + prep_account_new_hugetlb_folio(h, folio); enqueue_hugetlb_folio(h, folio); spin_unlock_irqrestore(&hugetlb_lock, flags); } --=20 2.20.1 From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-104.ptr.blmpb.com (sg-1-104.ptr.blmpb.com [118.26.132.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 780191D5160 for ; Wed, 7 Jan 2026 11:33:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.104 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785607; cv=none; b=kiWppmSGiSFLhab2COK7YDYe9ylfV01cGc+4QfhpA1iQQwBpXDPacFGQdDO9qB6zMa2A69xMUPKtX6kTP3Zt/oR1eLRekljLQH/+dasw4UZ8aJwTBH6CV03m7eYEVyq80PeNcY7jAnIO8pokZRZiSJgCGIGJm77ITs3IFzx1IJI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785607; c=relaxed/simple; bh=5BTIRFt+v2sD6CKQzyBbRcpIhStMGt2S6XD+WlkNoks=; h=To:Message-Id:References:Subject:Mime-Version:Cc:Date:In-Reply-To: Content-Type:From; b=hnegSt2w6PaNU5ZSS4bdwJTOH+hyZiUMnZWmD61X+dfp7T1ygNDMxVy1SIEHRjtbrw7uEOsMd7gAwdgdmYl0Nu7OOsAfiSXzoAjwhGRcGarM3ANELz12Dfs+VNfKjuLTrpzgSH+p7mTJTFrxYxhxkJer60D9+Z9mVhCkcaL2E6Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=nQMFR+/u; arc=none smtp.client-ip=118.26.132.104 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="nQMFR+/u" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785599; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=9nSEirhy+0vN+k5rbIUt6cf9EJAt01I3/f0PFptLibw=; b=nQMFR+/upHHs3GBwUfVERg4tUQHXza9RNnduhAYjzlqRAKHSCZnprZdVKQyMuKC8yFEEC2 hMY0vlh4koLqsh4nR//suZ9RlNWUqBJutdIkspBkSmdfRswn8nADgyWnjQ0GmBYsXRzFrc wSEsgjKkGFBfVHPgVTr02Kja8sK3RPpDn7mtc7aeM4AKxwotWKxHGIBrg+XgvrqswaivMn GzGw6tbl3XGABwytu2bamFrpJMGHzBPlqUO2A7fHBU+12uPcqdPatFIOkwBxQfZyf6ST80 /HYItB1N3K63x51lcx9S+yaKuRswk3NN7INxWKEgiv6l5LY13l/ey7Vu3jntSg== To: , , , , Message-Id: <20260107113130.37231-4-lizhe.67@bytedance.com> References: <20260107113130.37231-1-lizhe.67@bytedance.com> Content-Transfer-Encoding: quoted-printable Subject: [PATCH v2 3/8] mm/hugetlb: move the huge folio to the end of the list during enqueue Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.45.2 X-Original-From: Li Zhe X-Lms-Return-Path: Cc: , , Date: Wed, 7 Jan 2026 19:31:25 +0800 In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> From: "Li Zhe" Content-Type: text/plain; charset="utf-8" For the huge-folio free list, unzeroed huge folios are now inserted at the tail; a follow-on patch will place pre-zeroed ones at the head, so that allocations can obtain a pre-zeroed huge folio with minimal search. Also, placing newly zeroed pages at the head of the queue so they're chosen first in the next allocation helps keep the cache hot. Signed-off-by: Li Zhe --- mm/hugetlb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a7e582abe9f9..42d327152da9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1312,7 +1312,7 @@ static void enqueue_hugetlb_folio(struct hstate *h, s= truct folio *folio) VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); VM_WARN_ON_FOLIO(folio_test_hugetlb_zeroing(folio), folio); =20 - list_move(&folio->lru, &h->hugepage_freelists[nid]); + list_move_tail(&folio->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; prep_clear_zeroed(folio); --=20 2.20.1 From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-102.ptr.blmpb.com (sg-1-102.ptr.blmpb.com [118.26.132.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08E341D5160 for ; Wed, 7 Jan 2026 11:33:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.102 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785629; cv=none; b=Ri/L/X3B0LW8Ii782d62rAqhmeBLu5CcbwyocOgDm5syBcimJNYQlAE5AIxQx1amQDYooHm1TYIN+n7jrxoCn1pmyXRTsvu7O4rf70UIk/93h6Rfzo+njC8UqA32M4U6Fielga3hTBjLtXNoZ9fM0AyuRs0OP54YJA+1tV/eZ7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785629; c=relaxed/simple; bh=SSlFTiKnKXeMeThh/GoC+SsAb4Q03y952y1p7BuhA3o=; h=To:Cc:From:In-Reply-To:Date:References:Subject:Mime-Version: Content-Type:Message-Id; b=kbTMgvieaULWF6bIAS+V46JsGXiOVde0Kr7DhjzN97FO1zZkKLj2SEu/fueeU9vHNn+e9mxyaNoimTUDfZIiOzlu/CwadOjhMjfP4XkR9s/S1Z89KehLTIopJcEewv6eY3WFvq8hxIaFeofkB0jXox4ZPiQGcmx+azm7OJ4N30E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LSj7EPj1; arc=none smtp.client-ip=118.26.132.102 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LSj7EPj1" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785614; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=48A1SUtf1ORNjWppsSz9G1ylK0g/Qvc7EmDbh0wqBLU=; b=LSj7EPj1dCLZ+sfneb0GVu0gZQpr3IIyd8YOREsORjmHlDBHSow0RVF/PVEcAsodoQ7sz1 rrstyoyxomnnHcQpVrOEuw/FbT5yZGqkNeopGqWloXyBoF6c960KDIYKUm5bi1qhdoQxso N0Lg2HW+q3PRfM+CLpvo0jPFuj7v9eEgAmNFGWfdD8fF/TTj5rpjBbUG921TxamHzLT594 JQsBTo9dlj2Cf8rTN5wpbQRKY61a9ghgoNsWfn4NfX6JB4zsoW3u89aC6Y8eVV3Ionslra dglBCasLULfGWaYEBJnZ+JFV6y/emc/68pbd3ukBN2t58UCGx705VQXOXTR3pw== To: , , , , Cc: , , From: "Li Zhe" In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> Date: Wed, 7 Jan 2026 19:31:26 +0800 X-Mailer: git-send-email 2.45.2 References: <20260107113130.37231-1-lizhe.67@bytedance.com> Subject: [PATCH v2 4/8] mm/hugetlb: introduce per-node sysfs interface "zeroable_hugepages" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Message-Id: <20260107113130.37231-5-lizhe.67@bytedance.com> Content-Transfer-Encoding: quoted-printable X-Original-From: Li Zhe X-Lms-Return-Path: Content-Type: text/plain; charset="utf-8" Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 250 milliseconds for a 1G page on a Skylake machine). This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue. However, there are some use cases where a large number of hugetlb pages are touched when an application starts (such as a VM backed by these pages), rendering the launch noticeably slow. On an Skylake platform running v6.19-rc2, faulting in 64 =C3=97 1 GB huge pages takes about 16 seconds, roughly 250 ms per page. Even with Ankur's optimizations[2], the time drops only to ~13 seconds, ~200 ms per page, still a noticeable delay. To accelerate the above scenario, this patch exports a per-node, read-write "zeroable_hugepages" sysfs interface for every hugepage size. This interface reports how many hugepages on that node can currently be pre-zeroed and allows user space to request that any integer number in the range [0, max] be zeroed in a single operation. Exporting this interface offers the following advantages: (1) User space gains full control over when zeroing is triggered, enabling it to minimize the impact on both CPU and cache utilization. (2) Applications can spawn as many zeroing processes as they need, enabling concurrent background zeroing. (3) By binding the process to specific CPUs, users can confine zeroing threads to cores that do not run latency-critical tasks, eliminating interference. (4) A zeroing process can be interrupted at any time through standard signal mechanisms, allowing immediate cancellation. (5) The CPU consumption incurred by zeroing can be throttled and contained with cgroups, ensuring that the cost is not borne system-wide. Tested on the same Skylake platform as above, when the 64 GiB of memory was pre-zeroed in advance by the pre-zeroing mechanism, the faulting latency test completed in negligible time. [1]: https://lore.kernel.org/linux-mm/202412030519.W14yll4e-lkp@intel.com/T= /#t Co-developed-by: Frank van der Linden Signed-off-by: Frank van der Linden Signed-off-by: Li Zhe --- mm/hugetlb_sysfs.c | 124 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c index 79ece91406bf..68a7372d3378 100644 --- a/mm/hugetlb_sysfs.c +++ b/mm/hugetlb_sysfs.c @@ -352,6 +352,129 @@ struct node_hstate { }; static struct node_hstate node_hstates[MAX_NUMNODES]; =20 +static ssize_t zeroable_hugepages_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct hstate *h; + unsigned long free_huge_pages_zero; + int nid; + + h =3D kobj_to_hstate(kobj, &nid); + if (WARN_ON(nid =3D=3D NUMA_NO_NODE)) + return -EPERM; + + free_huge_pages_zero =3D h->free_huge_pages_node[nid] - + h->free_huge_pages_zero_node[nid]; + + return sprintf(buf, "%lu\n", free_huge_pages_zero); +} + +static inline bool zero_should_abort(struct hstate *h, int nid) +{ + return (h->free_huge_pages_zero_node[nid] =3D=3D + h->free_huge_pages_node[nid]) || + list_empty(&h->hugepage_freelists[nid]); +} + +static void zero_free_hugepages_nid(struct hstate *h, + int nid, unsigned int nr_zero) +{ + struct list_head *freelist =3D &h->hugepage_freelists[nid]; + unsigned int nr_zeroed =3D 0; + struct folio *folio; + + if (zero_should_abort(h, nid)) + return; + + spin_lock_irq(&hugetlb_lock); + + while (nr_zeroed < nr_zero) { + + if (zero_should_abort(h, nid) || fatal_signal_pending(current)) + break; + + freelist =3D freelist->prev; + folio =3D list_entry(freelist, struct folio, lru); + + if (folio_test_hugetlb_zeroed(folio)) + break; + + if (folio_test_hugetlb_zeroing(folio)) { + if (unlikely(freelist->prev =3D=3D + &h->hugepage_freelists[nid])) + break; + continue; + } + + folio_set_hugetlb_zeroing(folio); + + /* + * Incrementing this here is a bit of a fib, since + * the page hasn't been cleared yet (it will be done + * immediately after dropping the lock below). But + * it keeps the count consistent with the overall + * free count in case the page gets taken off the + * freelist while we're working on it. + */ + h->free_huge_pages_zero_node[nid]++; + spin_unlock_irq(&hugetlb_lock); + + /* + * HWPoison pages may show up on the freelist. + * Don't try to zero it out, but do set the flag + * and counts, so that we don't consider it again. + */ + if (!folio_test_hwpoison(folio)) + folio_zero_user(folio, 0); + + cond_resched(); + + spin_lock_irq(&hugetlb_lock); + folio_set_hugetlb_zeroed(folio); + folio_clear_hugetlb_zeroing(folio); + + /* + * If the page is still on the free list, move + * it to the head. + */ + if (folio_test_hugetlb_freed(folio)) + list_move(&folio->lru, &h->hugepage_freelists[nid]); + + /* + * If someone was waiting for the zero to + * finish, wake them up. + */ + if (waitqueue_active(&h->dqzero_wait[nid])) + wake_up(&h->dqzero_wait[nid]); + nr_zeroed++; + freelist =3D &h->hugepage_freelists[nid]; + } + spin_unlock_irq(&hugetlb_lock); +} + +static ssize_t zeroable_hugepages_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buf, size_t len) +{ + unsigned int nr_zero; + struct hstate *h; + int err; + int nid; + + if (!strcmp(buf, "max") || !strcmp(buf, "max\n")) { + nr_zero =3D UINT_MAX; + } else { + err =3D kstrtouint(buf, 10, &nr_zero); + if (err) + return err; + } + h =3D kobj_to_hstate(kobj, &nid); + + zero_free_hugepages_nid(h, nid, nr_zero); + + return len; +} +HSTATE_ATTR(zeroable_hugepages); + /* * A subset of global hstate attributes for node devices */ @@ -359,6 +482,7 @@ static struct attribute *per_node_hstate_attrs[] =3D { &nr_hugepages_attr.attr, &free_hugepages_attr.attr, &surplus_hugepages_attr.attr, + &zeroable_hugepages_attr.attr, NULL, }; =20 --=20 2.20.1 From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-104.ptr.blmpb.com (sg-1-104.ptr.blmpb.com [118.26.132.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1300217705 for ; Wed, 7 Jan 2026 11:33:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.104 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785630; cv=none; b=W6oeLdoJCwUY3cRV4ofwfJiJ9HEX0k30exWrmMRkwr8DHgXYRnQNgsDr9nrUPVltR839UD1Se9LKhusURXVseaWU/s15fdvlihklNuuaal2JMOew4TynMmYBx7hfTHZ0F/8Bx0seGztQPo2eAwYl/grd9Pw1b8eO8kbiKTSNxOU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785630; c=relaxed/simple; bh=6E2l99huuofYVdeeRyTPSErdtM6qWzOycD7weuOPC/c=; h=Content-Type:To:Cc:Date:In-Reply-To:Message-Id:Mime-Version:From: Subject:References; b=jCPMJM2hxfuUPVH3ywWB5VBHaHDruJeQ2t2YnVjkRTE4y9yU0zOJjdoRecpVNI94Tvuyq0ximIxDjN3zHMlj4oghZIrEtdrIJYdDcAW+YE/FLYw4V0mUBgYegY3WY+7IVWiITBvl4ER2OQ/uaV5SGgJi39lq6vcgGQ/yO0Kex+Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZvYfk9rw; arc=none smtp.client-ip=118.26.132.104 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZvYfk9rw" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785624; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=cdN0DkMz/24rvpJgnVOmkfsKqwcFTBjxdUIoAY0F9to=; b=ZvYfk9rw748S528ePTp5+ANPl8/4majtRwWTrxEDtF32l5vH2D+EfeogWjyslfS31jH4OP yg4hZoQYP6GqH+Yz5hsLJPT3kT9gHidk1TzXeo+jD3kHFURneLCxLgEIFKa9xEuaeZ9DB2 OnxnYzxIgyuu5rD6DggpAwU3zzsdbqvcwJE8ionSwOvtyM+jxXUW6zUBUwnevRH3LaArDB y9F58i5/dEyAuXQnblGlR9ny+4L6xRnH+8YUzya3OZ35k3T572jT3vUHzngCWb+YaqRGCe d0YDTvILEQ6XH6Ut7J8Yw+WCCRJG8BRijyRwN+gHLunlS5cA9X2VZU2LpH1AiA== To: , , , , Cc: , , Date: Wed, 7 Jan 2026 19:31:27 +0800 X-Lms-Return-Path: In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 Message-Id: <20260107113130.37231-6-lizhe.67@bytedance.com> X-Original-From: Li Zhe Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 From: "Li Zhe" Subject: [PATCH v2 5/8] mm/hugetlb: simplify function hugetlb_sysfs_add_hstate() References: <20260107113130.37231-1-lizhe.67@bytedance.com> Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The third parameter of hugetlb_sysfs_add_hstate() is currently an array of struct kobject *, yet the function only ever uses a single element. This patch narrows the argument to a pointer to that specific member, eliminating the unused array. Signed-off-by: Li Zhe --- mm/hugetlb_sysfs.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c index 68a7372d3378..3f25a24e27cf 100644 --- a/mm/hugetlb_sysfs.c +++ b/mm/hugetlb_sysfs.c @@ -304,31 +304,30 @@ static const struct attribute_group hstate_demote_att= r_group =3D { }; =20 static int hugetlb_sysfs_add_hstate(struct hstate *h, struct kobject *pare= nt, - struct kobject **hstate_kobjs, + struct kobject **hstate_kobj, const struct attribute_group *hstate_attr_group) { int retval; - int hi =3D hstate_index(h); =20 - hstate_kobjs[hi] =3D kobject_create_and_add(h->name, parent); - if (!hstate_kobjs[hi]) + *hstate_kobj =3D kobject_create_and_add(h->name, parent); + if (!*hstate_kobj) return -ENOMEM; =20 - retval =3D sysfs_create_group(hstate_kobjs[hi], hstate_attr_group); + retval =3D sysfs_create_group(*hstate_kobj, hstate_attr_group); if (retval) { - kobject_put(hstate_kobjs[hi]); - hstate_kobjs[hi] =3D NULL; + kobject_put(*hstate_kobj); + *hstate_kobj =3D NULL; return retval; } =20 if (h->demote_order) { - retval =3D sysfs_create_group(hstate_kobjs[hi], + retval =3D sysfs_create_group(*hstate_kobj, &hstate_demote_attr_group); if (retval) { pr_warn("HugeTLB unable to create demote interfaces for %s\n", h->name); - sysfs_remove_group(hstate_kobjs[hi], hstate_attr_group); - kobject_put(hstate_kobjs[hi]); - hstate_kobjs[hi] =3D NULL; + sysfs_remove_group(*hstate_kobj, hstate_attr_group); + kobject_put(*hstate_kobj); + *hstate_kobj =3D NULL; return retval; } } @@ -566,8 +565,8 @@ void hugetlb_register_node(struct node *node) =20 for_each_hstate(h) { err =3D hugetlb_sysfs_add_hstate(h, nhs->hugepages_kobj, - nhs->hstate_kobjs, - &per_node_hstate_attr_group); + &nhs->hstate_kobjs[hstate_index(h)], + &per_node_hstate_attr_group); if (err) { pr_err("HugeTLB: Unable to add hstate %s for node %d\n", h->name, node->dev.id); @@ -614,7 +613,7 @@ void __init hugetlb_sysfs_init(void) =20 for_each_hstate(h) { err =3D hugetlb_sysfs_add_hstate(h, hugepages_kobj, - hstate_kobjs, &hstate_attr_group); + &hstate_kobjs[hstate_index(h)], &hstate_attr_group); if (err) pr_err("HugeTLB: Unable to add hstate %s\n", h->name); } --=20 2.20.1 From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-100.ptr.blmpb.com (sg-1-100.ptr.blmpb.com [118.26.132.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5064A54723 for ; Wed, 7 Jan 2026 11:33:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785640; cv=none; b=rnA/VhhY6OkOVZyuzaAkXkehpCkgp05LuQeQARz86TVnN1cFz3KmW0CVIxHpu3IOIxSwzVKYBgEoGSNmIbhGRaItw2JwZjXJIksFKyECTZso3NEFvXFyepD1CIkmCi5dq3kD6faHJvDwpl8duj+v63WeLSbqYAzSP+SDx13F/hk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785640; c=relaxed/simple; bh=2DNV6OIRO0yNVj81jfDIqsXe19Agb673z0F3uqObgKg=; h=Subject:Date:In-Reply-To:Mime-Version:References:To:Cc:From: Message-Id:Content-Type; b=r6GufcSSZOCAoTYoH8dBmuyav6qtbhUU2eVNhtX+4/WWIsDKPS903jSv2K1CY1GIfZ6GPVpHhIRUqL5Ya4hnT2tgXY1Zid55+8ouLzWhZyBik31bYP24cJ7OZ24j+QLjo6Od18IhDSmxnG8omwAvsUW/dFGyx5VBQ2t01BFEr7k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MEWQooMw; arc=none smtp.client-ip=118.26.132.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MEWQooMw" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785634; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=vqgHPcDteXeYtkcevndtOM5gbTYMZ0McWPYXT6sbL+I=; b=MEWQooMwwf387pl7+RcGoJSDMD39jWPto0yhnWCCSFVjoDtsKPQ4G5hMoSIImK2jLeVHI4 EAevE6s3DtUImwb2xvXnZD2xmYjsfz9ApykwzFtODPNCzMYX/QH0aKGKDbWxQZj1d921ql V2+XLpBPBWbHrgI1K9TFLLabhVfEE93EFBi6DgT+fFRGdCNyiBgTJDbDO92YFkcUEwh9A8 fAZvdx1uQ5FjJVS499gRpTe9bc019rf3wousP6eylUmrAB0fyMCOp4hXmROmIAq5XK9kIx 0u/vvkvIf6QZnDiGehDMowsTkr+nkgILj6hhCfju/jIenLYzuixD1OPbOVNQNA== Subject: [PATCH v2 6/8] mm/hugetlb: relocate the per-hstate struct kobject pointer Date: Wed, 7 Jan 2026 19:31:28 +0800 X-Mailer: git-send-email 2.45.2 In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> Content-Transfer-Encoding: quoted-printable Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260107113130.37231-1-lizhe.67@bytedance.com> X-Lms-Return-Path: To: , , , , Cc: , , From: "Li Zhe" Message-Id: <20260107113130.37231-7-lizhe.67@bytedance.com> X-Original-From: Li Zhe Content-Type: text/plain; charset="utf-8" Relocate the per-hstate struct kobject pointer from struct node_hstate into a standalone structure. This change prepares for a future patch that adds epoll support to the =E2=80=9Czeroable_hugepages=E2=80=9D interface. When a huge folio is freed = we must emit an event, yet the freeing context may be atomic; therefore the notification will be delegated to a workqueue. Extracting the struct kobject pointer allows the workqueue callback to obtain it effortlessly. Signed-off-by: Li Zhe --- mm/hugetlb_sysfs.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c index 3f25a24e27cf..03b774b1191a 100644 --- a/mm/hugetlb_sysfs.c +++ b/mm/hugetlb_sysfs.c @@ -338,6 +338,10 @@ static int hugetlb_sysfs_add_hstate(struct hstate *h, = struct kobject *parent, #ifdef CONFIG_NUMA static bool hugetlb_sysfs_initialized __ro_after_init; =20 +struct node_hstate_item { + struct kobject *hstate_kobj; +}; + /* * node_hstate/s - associate per node hstate attributes, via their kobject= s, * with node devices in node_devices[] using a parallel array. The array @@ -347,7 +351,7 @@ static bool hugetlb_sysfs_initialized __ro_after_init; */ struct node_hstate { struct kobject *hugepages_kobj; - struct kobject *hstate_kobjs[HUGE_MAX_HSTATE]; + struct node_hstate_item items[HUGE_MAX_HSTATE]; }; static struct node_hstate node_hstates[MAX_NUMNODES]; =20 @@ -501,7 +505,7 @@ static struct hstate *kobj_to_node_hstate(struct kobjec= t *kobj, int *nidp) struct node_hstate *nhs =3D &node_hstates[nid]; int i; for (i =3D 0; i < HUGE_MAX_HSTATE; i++) - if (nhs->hstate_kobjs[i] =3D=3D kobj) { + if (nhs->items[i].hstate_kobj =3D=3D kobj) { if (nidp) *nidp =3D nid; return &hstates[i]; @@ -526,7 +530,7 @@ void hugetlb_unregister_node(struct node *node) =20 for_each_hstate(h) { int idx =3D hstate_index(h); - struct kobject *hstate_kobj =3D nhs->hstate_kobjs[idx]; + struct kobject *hstate_kobj =3D nhs->items[idx].hstate_kobj; =20 if (!hstate_kobj) continue; @@ -534,7 +538,7 @@ void hugetlb_unregister_node(struct node *node) sysfs_remove_group(hstate_kobj, &hstate_demote_attr_group); sysfs_remove_group(hstate_kobj, &per_node_hstate_attr_group); kobject_put(hstate_kobj); - nhs->hstate_kobjs[idx] =3D NULL; + nhs->items[idx].hstate_kobj =3D NULL; } =20 kobject_put(nhs->hugepages_kobj); @@ -565,7 +569,7 @@ void hugetlb_register_node(struct node *node) =20 for_each_hstate(h) { err =3D hugetlb_sysfs_add_hstate(h, nhs->hugepages_kobj, - &nhs->hstate_kobjs[hstate_index(h)], + &nhs->items[hstate_index(h)].hstate_kobj, &per_node_hstate_attr_group); if (err) { pr_err("HugeTLB: Unable to add hstate %s for node %d\n", --=20 2.20.1 From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-102.ptr.blmpb.com (sg-1-102.ptr.blmpb.com [118.26.132.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAC952FDC54 for ; Wed, 7 Jan 2026 11:34:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.102 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785653; cv=none; b=TUH9GJPfn4MDrRRIDZ3UkUke4izW+CBacOCvCdxJMdNDRDy0x2tDRkDu3dLV2dB5zVQo+ZBMaAiCJdo2zJyx2M2CEhN/ykjZXOMH6ACzW+JAYyfWB2TYp0cvIJSOx/GpPMBxmxBTHV4T1mgqSwbHDYEKF1Yo5ZGcqb7sYH+EvT4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785653; c=relaxed/simple; bh=JYng2GjIP1Zr6KVFfmNP13EyIvqMCS/y6rmW7E7ZVEU=; h=Subject:Date:From:Message-Id:References:To:Content-Type:Cc: Mime-Version:In-Reply-To; b=HlsJKGCFD0e5U5sfJgx5nQynA+mf5FkIqunYEB7PCOnVZy1xXLMvDWmgNWACA1Cx84xRhWkPDk+sQsn5mTkSHYOga1qyU2WhTrDhq2FWG7lp0WccaWLxzw/R1z8SNGYiOnMINSNhnu4EJYqAXHCey8QmQYdOfYu33+McIb3Mcgg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=oCBbv+Ay; arc=none smtp.client-ip=118.26.132.102 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="oCBbv+Ay" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785645; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=xEwE/5mv3+6N5jXpp615j2AhMiOJolhztasIKQwJLWI=; b=oCBbv+AyrRruOaEpQ2tgQ3K8UOLiFLP+n2T2M0rYpgczIr40X5rUEeRaXclcgbAhKoJe7f cn7Jz1/UEV8uajkxD3Nd3KfIsmfz1vepRiuphMi1H19H+Q2k+qRFNgOzB2j4pXJjtxVnHm PfsuY6p9QciFEjhAAV13HhisSbvxBexEC5zcAJ0oPkeOui0/o87XGFV5Nlf4N5vJN60e21 tl8rAiyuURDwJHZHK3vHh50bqrV24ycornSyzrTK2mlRrBeKsYjVGiijwSy+oAw991pS8u 9rpNaAwyGo+/YX4Fi6ty+26WgUx3m71/eHT3WhVOK0R23j93Ht+a6Fh3e3Hy4A== Subject: [PATCH v2 7/8] mm/hugetlb: add epoll support for interface "zeroable_hugepages" Date: Wed, 7 Jan 2026 19:31:29 +0800 X-Mailer: git-send-email 2.45.2 X-Original-From: Li Zhe From: "Li Zhe" Message-Id: <20260107113130.37231-8-lizhe.67@bytedance.com> References: <20260107113130.37231-1-lizhe.67@bytedance.com> Content-Transfer-Encoding: quoted-printable To: , , , , Cc: , , Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Lms-Return-Path: In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> Content-Type: text/plain; charset="utf-8" Add epoll support for interface "zeroable_hugepages". When no huge folios are available for pre-zeroing, user space can block on the zeroable_hugepages file with epoll, and it will be woken as soon as one or more huge folios become eligible for pre-zeroing. Signed-off-by: Li Zhe --- mm/hugetlb.c | 13 +++++++++++++ mm/hugetlb_internal.h | 6 ++++++ mm/hugetlb_sysfs.c | 22 +++++++++++++++++++++- 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 42d327152da9..314734e434e2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1868,6 +1868,7 @@ void free_huge_folio(struct folio *folio) arch_clear_hugetlb_flags(folio); enqueue_hugetlb_folio(h, folio); spin_unlock_irqrestore(&hugetlb_lock, flags); + do_zero_free_notify(h, folio_nid(folio)); } } =20 @@ -1999,8 +2000,10 @@ static struct folio *alloc_fresh_hugetlb_folio(struc= t hstate *h, void prep_and_add_allocated_folios(struct hstate *h, struct list_head *folio_list) { + nodemask_t allocated_mask =3D NODE_MASK_NONE; unsigned long flags; struct folio *folio, *tmp_f; + int nid; =20 /* Send list for bulk vmemmap optimization processing */ hugetlb_vmemmap_optimize_folios(h, folio_list); @@ -2010,8 +2013,12 @@ void prep_and_add_allocated_folios(struct hstate *h, list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { prep_account_new_hugetlb_folio(h, folio); enqueue_hugetlb_folio(h, folio); + node_set(folio_nid(folio), allocated_mask); } spin_unlock_irqrestore(&hugetlb_lock, flags); + + for_each_node_mask(nid, allocated_mask) + do_zero_free_notify(h, nid); } =20 /* @@ -2383,6 +2390,8 @@ static int gather_surplus_pages(struct hstate *h, lon= g delta) long needed, allocated; bool alloc_ok =3D true; nodemask_t *mbind_nodemask, alloc_nodemask; + nodemask_t allocated_mask =3D NODE_MASK_NONE; + int nid; =20 mbind_nodemask =3D policy_mbind_nodemask(htlb_alloc_mask(h)); if (mbind_nodemask) @@ -2455,9 +2464,12 @@ static int gather_surplus_pages(struct hstate *h, lo= ng delta) break; /* Add the page to the hugetlb allocator */ enqueue_hugetlb_folio(h, folio); + node_set(folio_nid(folio), allocated_mask); } free: spin_unlock_irq(&hugetlb_lock); + for_each_node_mask(nid, allocated_mask) + do_zero_free_notify(h, nid); =20 /* * Free unnecessary surplus pages to the buddy allocator. @@ -2841,6 +2853,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct fo= lio *old_folio, * Folio has been replaced, we can safely free the old one. */ spin_unlock_irq(&hugetlb_lock); + do_zero_free_notify(h, folio_nid(new_folio)); update_and_free_hugetlb_folio(h, old_folio, false); } =20 diff --git a/mm/hugetlb_internal.h b/mm/hugetlb_internal.h index 1d2f870deccf..9c60661283c7 100644 --- a/mm/hugetlb_internal.h +++ b/mm/hugetlb_internal.h @@ -106,6 +106,12 @@ extern ssize_t __nr_hugepages_store_common(bool obey_m= empolicy, struct hstate *h, int nid, unsigned long count, size_t len); =20 +#ifdef CONFIG_NUMA +extern void do_zero_free_notify(struct hstate *h, int nid); +#else +static inline void do_zero_free_notify(struct hstate *h, int nid) {} +#endif + extern void hugetlb_sysfs_init(void) __init; =20 #ifdef CONFIG_SYSCTL diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c index 03b774b1191a..77e7214a380e 100644 --- a/mm/hugetlb_sysfs.c +++ b/mm/hugetlb_sysfs.c @@ -340,6 +340,7 @@ static bool hugetlb_sysfs_initialized __ro_after_init; =20 struct node_hstate_item { struct kobject *hstate_kobj; + struct work_struct notify_work; }; =20 /* @@ -355,6 +356,21 @@ struct node_hstate { }; static struct node_hstate node_hstates[MAX_NUMNODES]; =20 +static void pre_zero_notify_fun(struct work_struct *work) +{ + struct node_hstate_item *item =3D + container_of(work, struct node_hstate_item, notify_work); + + sysfs_notify(item->hstate_kobj, NULL, "zeroable_hugepages"); +} + +void do_zero_free_notify(struct hstate *h, int nid) +{ + struct node_hstate *nhs =3D &node_hstates[nid]; + + schedule_work(&nhs->items[hstate_index(h)].notify_work); +} + static ssize_t zeroable_hugepages_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -568,8 +584,11 @@ void hugetlb_register_node(struct node *node) return; =20 for_each_hstate(h) { + int index =3D hstate_index(h); + struct node_hstate_item *item =3D &nhs->items[index]; + err =3D hugetlb_sysfs_add_hstate(h, nhs->hugepages_kobj, - &nhs->items[hstate_index(h)].hstate_kobj, + &item->hstate_kobj, &per_node_hstate_attr_group); if (err) { pr_err("HugeTLB: Unable to add hstate %s for node %d\n", @@ -577,6 +596,7 @@ void hugetlb_register_node(struct node *node) hugetlb_unregister_node(node); break; } + INIT_WORK(&item->notify_work, pre_zero_notify_fun); } } =20 --=20 2.20.1 From nobody Sun Feb 8 15:08:17 2026 Received: from sg-1-104.ptr.blmpb.com (sg-1-104.ptr.blmpb.com [118.26.132.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5354431619A for ; Wed, 7 Jan 2026 11:34:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.104 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785664; cv=none; b=M8tTHcyWd8WVV0irvSPKNIhTonoXpAH8OOB5rOl9blvyUSXxhGc89WTiJKTBy1RQs3jlePZ9+hBKTWFRzDrM5ZVEqq/nuilvZtV9I6Ft4p/KhEnaF8HumUZeqUQwSVoYxcNB8/r6T5GhNQrdPxCChqbUBa9TKuhOoa7cDCXwgeU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767785664; c=relaxed/simple; bh=40yvhytXGaLDqg3XSFneNPh9JRhq1EhG2iHDd344www=; h=Cc:Date:Mime-Version:To:Message-Id:Content-Type:From:Subject: In-Reply-To:References; b=B7ZLhsm6rpxWGOPEiFD9y7dh59B7zNWxr+qbt0Ho+hKRCRlnHeC1EAkakttMqoR5pckn4akShOeBxF6u/u8Qdz8fOCNAW5TZf7wjAziHzbggDszm/0Y/HExVQCfhRBVNxuv12fwh0CXK644lNTFlSK1HGuJ9SZIAWAUnwJzjCm8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dSNIvTeq; arc=none smtp.client-ip=118.26.132.104 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dSNIvTeq" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1767785656; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=pQGsuo+iA51rKHReFsoIiKz498AYBUnjfg0gVNCR4mw=; b=dSNIvTeqojGMFDN8IC8DMsdhS9EEmH2X0jq3KlLwWu6X6qMZZ5Sd4aRqcwxPZrjulQDdSs DaHfm2fMpvJahZhqz2+xtXHwNkO+7LbrtKEtgNulKDIiFJtnjw6/pa+PsC0OJKR0xSuIiy upumWCdoFpLJky9kx1TskoDENnpHIQ9sLyNynyBvqXF8lQcmK7a4afuVnjiGDHsEX8IXsi RAETEey2pViXlddvwMKUL/30oQD0sP4xss0SygzEs7HMRhSsXUIJSlStbBcY9q6vHR2lUv ZYhC1PU90pPgd092DDUNaqKPfdse44n2QfEx7whshrVzq0dmsrbIVFouX4vOng== Cc: , , Date: Wed, 7 Jan 2026 19:31:30 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.45.2 X-Original-From: Li Zhe To: , , , , Message-Id: <20260107113130.37231-9-lizhe.67@bytedance.com> X-Lms-Return-Path: Content-Transfer-Encoding: quoted-printable From: "Li Zhe" Subject: [PATCH v2 8/8] mm/hugetlb: limit event generation frequency of function do_zero_free_notify() In-Reply-To: <20260107113130.37231-1-lizhe.67@bytedance.com> References: <20260107113130.37231-1-lizhe.67@bytedance.com> Content-Type: text/plain; charset="utf-8" Throttling notifications reduces the number of scheduling notify_work making the mechanism far more efficient when huge numbers of huge folios are freed in rapid succession. Signed-off-by: Li Zhe --- mm/hugetlb_sysfs.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c index 77e7214a380e..1e3a6983457a 100644 --- a/mm/hugetlb_sysfs.c +++ b/mm/hugetlb_sysfs.c @@ -341,6 +341,8 @@ static bool hugetlb_sysfs_initialized __ro_after_init; struct node_hstate_item { struct kobject *hstate_kobj; struct work_struct notify_work; + unsigned long notified_at; + spinlock_t notify_lock; }; =20 /* @@ -364,11 +366,30 @@ static void pre_zero_notify_fun(struct work_struct *w= ork) sysfs_notify(item->hstate_kobj, NULL, "zeroable_hugepages"); } =20 +static void __do_zero_free_notify(struct node_hstate_item *item) +{ + unsigned long last; + unsigned long next; + +#define PRE_ZERO_NOTIFY_MIN_INTV DIV_ROUND_UP(HZ, 100) + spin_lock(&item->notify_lock); + last =3D item->notified_at; + next =3D last + PRE_ZERO_NOTIFY_MIN_INTV; + if (time_in_range(jiffies, last, next)) { + spin_unlock(&item->notify_lock); + return; + } + item->notified_at =3D jiffies; + spin_unlock(&item->notify_lock); + + schedule_work(&item->notify_work); +} + void do_zero_free_notify(struct hstate *h, int nid) { struct node_hstate *nhs =3D &node_hstates[nid]; =20 - schedule_work(&nhs->items[hstate_index(h)].notify_work); + __do_zero_free_notify(&nhs->items[hstate_index(h)]); } =20 static ssize_t zeroable_hugepages_show(struct kobject *kobj, @@ -597,6 +618,8 @@ void hugetlb_register_node(struct node *node) break; } INIT_WORK(&item->notify_work, pre_zero_notify_fun); + item->notified_at =3D jiffies; + spin_lock_init(&item->notify_lock); } } =20 --=20 2.20.1