From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DB583C584F for ; Thu, 10 Sep 2026 06:33:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022007; cv=none; b=I0Y/gAEnE9zlDEC3Ymqf5jQ7Xea6mXO5hSdlZpMjo2Tf1npcrYvHe8A0FcZ0D2XyxMiOWT7yXyRYVO2zMhbL1woPlkUPy+luKyHZGg0n/LA+jRf3SUwW5rH/576P5WyF7XZlhT98j3gyc11jjaQYOD4zuzOpmLr+qWuiy4edHF0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022007; c=relaxed/simple; bh=ScYqkdYU6Y38D/zkR4/+pk20mrHP1/TIZAAKXDcoung=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bU2vdGs19+sl4r8Ys7yZGdSNmxZVNTmQseq0jNwjUljR4H9FZZ1DoApHMq/MpPgJxGgIzbhK1WbHPGrReGhfRb2vEfyGzPjh3/8zFoSA/wDN7Bf7aY/Cer0A7U4PHJAtF109c9Yknj1aUDbGp0ZOF/iAFt3nPVs1bKqH+dwTXLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=O+OJZuiV; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="O+OJZuiV" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-38ea87caafeso5921109a91.3 for ; Wed, 09 Sep 2026 23:33:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022005; x=1789626805; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6DfREAR7YFflJsvomdeYa+yVOrP6yNKuV6EJKciTVrI=; b=O+OJZuiVdTXkzxrUROTxnJb0D02OiwbLGJ934DSEc/+LV+9wGqVjSCMDYQdxja4uIL 32Z7oswtrLzJxT9gtxRb2zk94MDg3uwK2NuHd4EN+ef9u2mEWW1trQCthJ5yf1FwUEmh gSNvnU4bTveQRqnlLSzpp8kQmAS18KbPnqDwjf/G1+BBZJ/4bv3ZgX7pLBrube8Nkd2/ zbn3O9Prjctrp4fklZ/ba5+2nw+1zzXZCzavO5ofOfQ9rSYxwM8D6PxFTG18yJ3fzcZo AGpCdlVnpEXrsbyNgRLPcYz6mQBDr2oP8CmCzXXkD1FqZQdNO9PP6h5wu6r5j/0F7MUl 1qWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022005; x=1789626805; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6DfREAR7YFflJsvomdeYa+yVOrP6yNKuV6EJKciTVrI=; b=q92kQ/AkEXBwd4sZF8o9lBlCrFiQ8t150LUHyF9makyWMmbULD3IUYsimq+azxZbeh 6dj+XnPscKpMRDiKKzN5o8U4WWMEFlDzdPhfGAwCZq0qzk5+CsJIcQRMNwsaed38qEe0 KLXYhKb1ocmhJwewVpjVKe+Cv74Xz8UIM/Wm9/Nzj364RtSnFS2QfJIKDByNxhdypXf7 gY/y3ulIOzAmSFwGC72W+Q2A2nG7/1wjRFSum5RVo9/gVOmZN7Wlotk49AiH0QFkQSxs UyWGcPY9FKGqvbr+9Vx8OFB8hJyZVT7INUKd3O/cMAkSxHwuPH7uLtkNwO4lvSShV1hj B7Ow== X-Forwarded-Encrypted: i=1; AKwUvByh9VwtRoalD5Pgm4CYUGuJe2W5Hd2vj0yiQmfzuC89FKpsomKBYkSAWDszOz1XkBhRM37DRAHgaDL/OLc=@vger.kernel.org X-Gm-Message-State: AFuF++lJEw8rw+OmQZ3mFlX2a+u8ZeGss54AhbtYv7WGDr8OtuxCrpr9 7rCk/p0yy0e3mkfNxr2WuaaYw3hSS3s+wNGFozrACJ2AWmpUViUhFTgS76jNDnP+KWI= X-Gm-Gg: AYBFou0NWCZ2q19Jvq4XMC3F8V/tQy8p1OQnQRuMDXNLWnTdoYOocfwtlBka5yNgH+x 6ntEyCVncsXRQ0AihnHhnPKPdXlz1CBlFctrGeVTkSq2ukaIlokORwybEqHkrRHAynC3F8XAvxb 9GjskqmNOB1CKTjCevc9HGy2V6eLXy3H8to/ybKyU/tcTg602xSExKCUoQ4b51DAqpO369MSRjM x7fZ2Lb+vgAu5LLeTVuWNbveMGDFUANWRQdi57F8QYNTUhaKYE8vmhjnDZjud7+qHf+EfkRWXuQ YUtAIAy7vhJgIrLQK/JeuA5wVjCeYYls4hm9UZbULyKAqqbs/mP2l4o1PkYv3U3H9R8swDGxyhZ aVpekcuxWWQiriyCm5C7DqwRV5CT+ebTsilQr1dQJTlr/ugIomE7kId16gAoOk1lBRxehInhUJU 0btnpBlfxg0t4JkFlPh9YjZLbHjCUpqRE/X7TVxhwxOh59PFmg66Hqst3ALhsuQlbJkkUD5JZXQ rJrCFjJcIXcfVMuZid9XdvxmzL0H6j+pDU= X-Received: by 2002:a17:90b:578c:b0:398:e6b6:acc2 with SMTP id 98e67ed59e1d1-39b261e7649mr63026015a91.12.1789022004648; Wed, 09 Sep 2026 23:33:24 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:24 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 01/17] mm/sparse: relax struct mem_section size constraints Date: Thu, 10 Sep 2026 14:32:40 +0800 Message-ID: <20260910063256.64386-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct mem_section is currently forced to a power-of-2 size so the section-to-root lookup can use a mask instead of a modulo. That requirement makes future extensions harder than necessary: adding a small field can require configuration-dependent padding or layout checks just to preserve the lookup scheme. Keep the lookup correct for any struct mem_section size by using a plain modulo instead. Do not leave the layout entirely unconstrained, though. Keep struct mem_section double-word aligned so modest size changes, such as adding another word-sized field on 64-bit systems, still keep a compact and efficient layout. If future fields grow the structure beyond that sweet spot, the lookup remains correct; only the exact layout efficiency changes. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) --- v6: - Collect Acked-by from David Hildenbrand v2: - Align struct mem_section to power-of-2 where possible, but support non-power-of-2 lookup (suggested by David Laight) - Collect Acked-by from Mike Rapoport --- include/linux/mmzone.h | 11 +++-------- mm/sparse.c | 2 -- scripts/gdb/linux/mm.py | 6 ++---- 3 files changed, 5 insertions(+), 14 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff5416..0a2428714108 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2027,13 +2027,9 @@ struct mem_section { * section. (see page_ext.h about this.) */ struct page_ext *page_ext; - unsigned long pad; #endif - /* - * WARNING: mem_section must be a power-of-2 in size for the - * calculation and use of SECTION_ROOT_MASK to make sense. - */ -}; +/* Sacrifice minor padding space for efficient lookup. */ +} __aligned(2 * sizeof(unsigned long)); =20 #ifdef CONFIG_SPARSEMEM_EXTREME #define SECTIONS_PER_ROOT (PAGE_SIZE / sizeof (struct mem_section)) @@ -2043,7 +2039,6 @@ struct mem_section { =20 #define SECTION_NR_TO_ROOT(sec) ((sec) / SECTIONS_PER_ROOT) #define NR_SECTION_ROOTS DIV_ROUND_UP(NR_MEM_SECTIONS, SECTIONS_PER_ROOT) -#define SECTION_ROOT_MASK (SECTIONS_PER_ROOT - 1) =20 #ifdef CONFIG_SPARSEMEM_EXTREME extern struct mem_section **mem_section; @@ -2067,7 +2062,7 @@ static inline struct mem_section *__nr_to_section(uns= igned long nr) if (!mem_section || !mem_section[root]) return NULL; #endif - return &mem_section[root][nr & SECTION_ROOT_MASK]; + return &mem_section[root][nr % SECTIONS_PER_ROOT]; } =20 /* diff --git a/mm/sparse.c b/mm/sparse.c index 7c15406e77f5..c84b4c7b8c70 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -322,8 +322,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count =3D 1; int nid_begin; =20 - /* see include/linux/mmzone.h 'struct mem_section' definition */ - BUILD_BUG_ON(!is_power_of_2(sizeof(struct mem_section))); memblocks_present(); =20 if (compound_info_has_mask()) { diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index 28d33624c38b..193a88d763ab 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -70,7 +70,6 @@ class x86_page_ops(): self.SECTIONS_PER_ROOT =3D 1 =20 self.NR_SECTION_ROOTS =3D DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.= SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK =3D self.SECTIONS_PER_ROOT - 1 =20 try: self.SECTION_HAS_MEM_MAP =3D 1 << int(gdb.parse_and_eval('SECT= ION_HAS_MEM_MAP_BIT')) @@ -100,7 +99,7 @@ class x86_page_ops(): def __nr_to_section(self, nr): root =3D self.SECTION_NR_TO_ROOT(nr) mem_section =3D gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] =20 def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT @@ -249,7 +248,6 @@ class aarch64_page_ops(): self.SECTIONS_PER_ROOT =3D 1 =20 self.NR_SECTION_ROOTS =3D DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.= SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK =3D self.SECTIONS_PER_ROOT - 1 self.SUBSECTION_SHIFT =3D 21 self.SEBSECTION_SIZE =3D 1 << self.SUBSECTION_SHIFT self.PFN_SUBSECTION_SHIFT =3D self.SUBSECTION_SHIFT - self.PAGE_SH= IFT @@ -304,7 +302,7 @@ class aarch64_page_ops(): def __nr_to_section(self, nr): root =3D self.SECTION_NR_TO_ROOT(nr) mem_section =3D gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] =20 def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A05D93CB542 for ; Thu, 10 Sep 2026 06:33:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022011; cv=none; b=oROmXhVBfrfy9hbaJlDNdAeU6wFBTMXSKV4Qg9O/1W/hES1q3XZOQs6MpKTV4YF4iF8ev/JQMS7QYHV4vRhI2FRTcqxI/+pqeFRLuVYFig8TPoXpBaVBlrXqgeQCqaYAJHn5PfB3j0SzFOf7Sz0NbLDM+jv6LqLzFqUJQPRNV6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022011; c=relaxed/simple; bh=S/aRLy4OXzdfmMXkVg4XDhu6KiAzVdSOJzjWprATbEs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ove5MuFaPT02zQ2uj0wcmjA0MKQCYwQImwokTCgO6pPECbo6lAEveOIwBld3O6A1EseWbeM90TUTR02+EmEZV6QQy+GGK14sApkYp0VBUA7sv8KVz6RCtY7nNuEarm6d5krQdGS6jNtXUB5jtaTiFZ7KZ27uRxcVjj0FmRjG4a4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Ji1J6Zov; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Ji1J6Zov" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2cfbbdfa60bso60148225ad.3 for ; Wed, 09 Sep 2026 23:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022009; x=1789626809; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wagISQ9d+CnFEjK/rivKR4J4aGNpnRPBrTqRitnWimk=; b=Ji1J6ZovJ9SRN4ul04hAWdPsW7nmxONlZBjmiCMNhZjJ3BBDaCFsXFh5tvDqF1paa3 ZrNVP0iqiQT4w5J8/kOhIISBqJ0mCcMGV1V1hSBg1wwA/V7zGkkfP0jM/tXnGzzVjy/0 /qa+EjQIAQbRt4T/LD6C4nDX275D54uRmFiJOQqIr3RGf+zybIvusgkrQ8aC5PFo5cC+ 1mGjNQ8ot8W6X7oZIbdaiwqVnzFxbkytBWfZEZhSew/womP7uZYke4lhGTIs4ST113xv q6NjOee1vmLE6GB0cD4WN1v468ttyouAcKHl3QJVTX4elBt4wmJ3iE5O4GfR47n3vLkD 8sfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022009; x=1789626809; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wagISQ9d+CnFEjK/rivKR4J4aGNpnRPBrTqRitnWimk=; b=RReeUvVsjhowZI98qE+cfA047T4pDwAOZHUfb9hohyFUM16TlgF8EmPZB3FviSzJmU YWW2CezibjvdkGlbSnJ6lk5VhDPnyjWUMw46BFB6vY7yU0ye2mkdlzQnX49wi23MXrJw l8nWqvRNu+VWZB29q1J/QGmHJAMCe9pkjy+f77grVXu3fKox7s+MZo2Pib8Pg3GBATm9 kZrXOWnhnaxIn1asXRQ4wy3irBnMHtl12KuFaIXCvnSwAnUDxFgW50wYKHE0vxjv3KDs IiTfDA50EH5T/4622/CaGe4IstQP1T+E/Pq8PHBizZx5J6CT6AWmV4Eda1S1/PNppwJb nEvA== X-Forwarded-Encrypted: i=1; AKwUvBz4RGUM+OYj1i3z5ZiUhBur6tqrtjnZ/TDzkeBb3ST+pD73VAe1jB3FWP8W3UvGRnPna60LILFtnTjZhTA=@vger.kernel.org X-Gm-Message-State: AFuF++l7p8swOUly4Shn6sPOGtOLex0oL5iHoO1sjougTxquOuhgP1tf t3YnNOVHnfU0DTDhIFoYj2yKTCItfiv4C919OWkuMh5DLbC1Nfch0JzIT/3bceussoA= X-Gm-Gg: AYBFou0Uderax63j8C+MB63AR9t9uIuJAt5AUyG9Ciuq6Ur519ztNdiZcuM94+DyGpA ndMWhXpjLpOzv9fdyutZeLcZSuerQnt9wDr3E4yS9v70AGMzTdAeA76uDFcZrXHfh2b0AfVrNpx w6pW6DWcNb0qGYiLWqGoHR2/HKalMqU0+P1yanfyWh7MMH3OSBDaKM1V+hKIQU+dX8a7c9vmpwO Bonys6p8HBKfHh+Pvp6kXi2o4cXVNLP52TQklbsPdmBSkjCT2TmbAKrElSvGf+Lq3rem8BXA1to 6X4jMEK5yWu4dKtgkD++Sp1DuP8DIWnldEIK6cG59xm8RGCkknyitX+alMBkU9d9EcsXzXe3ZW0 OYCt72KGcOEhXo/XyKPKAr6lsJfkkaUU9P8nEQ5WRrb3gIN81cE5JCF1OtpHXI3Dfva4T4X2Jz3 zn9byATzscFqaijZiU+UFyJ6wrx1BXIXOyNP8Ht+Z5HwN/kooGV8d7KlGjDJ+1nZM1n676/9OMh x/6pLXI8agywK5pkxaWHqI7uOX2W1rRAALZ X-Received: by 2002:a17:90b:264e:b0:398:9bd1:3215 with SMTP id 98e67ed59e1d1-39b262e28cdmr63381387a91.22.1789022008789; Wed, 09 Sep 2026 23:33:28 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:28 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 02/17] mm/sparse-vmemmap: rename HVO order macros Date: Thu, 10 Sep 2026 14:32:41 +0800 Message-ID: <20260910063256.64386-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The macros VMEMMAP_TAIL_MIN_ORDER and NR_VMEMMAP_TAILS describe the order range where HVO can be applied, but their names tie that range to the tail-page cache implementation. Rename them with a VMEMMAP_OPTIMIZATION prefix and use the new names in the HVO paths. This makes the code describe the optimization requirements rather than the tail-page cache implementation detail. No functional change intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Collect Acked-by from Qi Zheng v3: - Collect Acked-by from Mike Rapoport v2: - Rename the macros with a VMEMMAP_OPTIMIZATION prefix (suggested by Mike Rapoport) - Drop intermediate optimized-folio size macros (suggested by Mike Rapoport) --- include/linux/mmzone.h | 17 +++++++++-------- mm/hugetlb.c | 4 ++-- mm/hugetlb_vmemmap.c | 2 +- mm/sparse-vmemmap.c | 4 ++-- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 0a2428714108..5fb9b37819d5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -107,13 +107,14 @@ is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 -/* - * vmemmap optimization (like HVO) is only possible for page orders that f= ill - * two or more pages with struct pages. - */ -#define VMEMMAP_TAIL_MIN_ORDER (ilog2(2 * PAGE_SIZE / sizeof(struct page))) -#define __NR_VMEMMAP_TAILS (MAX_FOLIO_ORDER - VMEMMAP_TAIL_MIN_ORDER + 1) -#define NR_VMEMMAP_TAILS (__NR_VMEMMAP_TAILS > 0 ? __NR_VMEMMAP_TAILS : 0) +/* The number of struct pages covered by the retained vmemmap pages with H= VO enabled. */ +#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (PAGE_SIZE / sizeof(struct pa= ge)) +#define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STR= UCT_PAGES) + 1) + +#define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ + (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1) +#define VMEMMAP_OPTIMIZATION_NR_ORDERS \ + (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS = : 0) =20 enum migratetype { MIGRATE_UNMOVABLE, @@ -1158,7 +1159,7 @@ struct zone { atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct page *vmemmap_tails[NR_VMEMMAP_TAILS]; + struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 46b6703bac4e..003cfac98f14 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3368,7 +3368,7 @@ void __init hugetlb_bootmem_struct_page_init(void) struct zone *zone; =20 for_each_zone(zone) { - for (int i =3D 0; i < NR_VMEMMAP_TAILS; i++) { + for (int i =3D 0; i < VMEMMAP_OPTIMIZATION_NR_ORDERS; i++) { struct page *tail, *p; unsigned int order; =20 @@ -3376,7 +3376,7 @@ void __init hugetlb_bootmem_struct_page_init(void) if (!tail) continue; =20 - order =3D i + VMEMMAP_TAIL_MIN_ORDER; + order =3D i + VMEMMAP_OPTIMIZATION_MIN_ORDER; p =3D page_to_virt(tail); /* * prep_and_add_bootmem_folios() can access pageblock diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 917db0984143..ae8fdaa42118 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -494,7 +494,7 @@ static bool vmemmap_should_optimize_folio(const struct = hstate *h, struct folio * =20 static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) { - const unsigned int idx =3D order - VMEMMAP_TAIL_MIN_ORDER; + const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; struct page *tail, *p; int node =3D zone_to_nid(zone); =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 5a2469fb1838..aa6a4a2fae98 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -329,12 +329,12 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * unsigned int idx; int node =3D zone_to_nid(zone); =20 - if (WARN_ON_ONCE(order < VMEMMAP_TAIL_MIN_ORDER)) + if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) return NULL; if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) return NULL; =20 - idx =3D order - VMEMMAP_TAIL_MIN_ORDER; + idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; tail =3D zone->vmemmap_tails[idx]; if (tail) return tail; --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CF243C4B91 for ; Thu, 10 Sep 2026 06:33:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022015; cv=none; b=cGStVY1ApTnZv2YvY28eF0e3Sqjv+PQlFdFHkHnoa0hrKlo99tK3gmegG/E3M9EFqECg/zuVdBGCB6gp5hxNAKftVmqeXxMZD/MF/oTvKU7LHfKxdJAzEwE07Z+4Ckz96vsA6tu2+W1OUR/+Z3SF+CNMGZEcphhGFdwS22pQj1Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022015; c=relaxed/simple; bh=5jEVhUSLED19JV+A/ye0WOHwBGdcbFAykQgbHFBITD0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lwaSnzoUyyaPb+/lziSHAiRvcotOSW0LI1SGwg0o7AKSUvA346qarKysqsqIRCCdSrNh/bZx56naXiEgk9HZk6wmGKhdbexkOnFMCeZ/zMpROfL9KIzCK+QC+rOAqTj2GVHW5msOxMLM8LJy1/UU+oJ3uPesC2YtN/b12J4YRkE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PitUz7m9; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PitUz7m9" Received: by mail-pj2-f12.google.com with SMTP id d9443c01a7336-2d8fb334ddcso10047985ad.0 for ; Wed, 09 Sep 2026 23:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022013; x=1789626813; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tUBCzuDDmijmZL9894tpqO2PeM3/ckJvguvNiHP8cxU=; b=PitUz7m9DDqDvHB56OkcXvm5b5OkDA2Y0RqUCQww1ghuPI6M+qELGHc1VVkJrV1z3Q o8lBXZ8aBMtOxfH3h4Tg67Cz4mXcMxxNzUWU4vN6tUX+iTmTJZ/9Ax33vm3PqvJlmIW2 79ZmS/HzZcJPLAqhomfclFpIzY0N/gJGbnwK/pBEZN07QUA/Ps6nfWenYVkB2R+6xkIb smNA6bbRrjJI+6I+gDe7Lt7CG89iScRYY3rJ2nAdkvJ0SKKhYtwKO+6Gt9WUVX3CJlWW bvVGpqmAYLJIfgfKhaFpdsWAoBceSwj52g8BgEQ1pt9XPGoz2JtvQzls1poiDA60tT3Q DC5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022013; x=1789626813; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=tUBCzuDDmijmZL9894tpqO2PeM3/ckJvguvNiHP8cxU=; b=dn9/nWxINZQDclapI2z5Q4FjEWSh8NNyxF/hBoGlZhj6AbV5gSCdyDImyleMlYadxc 4YZ+r6ELqbd/6RS1cgMxlp/eGIr0M+NOoMZCtS2q69tD2SQBlDfm1pG8X1/3j5QPvXEO Bp7SZfED/IFGWygV84Vce69e/03Q8DXytBqpz91154ejDpqc6/jqDTo1Y8ywzvUDfdLl aC1b97kZnaxNNPox3HuGCffY5e7Qx5VGMra2PKnfFEHt1vz3OFRm5xGKK2tvH2HQubp0 P+XX6vHLgr01y53Y63CxlQo84p94nhXXTyjxrfJIP2Dqy3kghk5HHITn3/Zxi5KndZBM PvJQ== X-Forwarded-Encrypted: i=1; AKwUvBx5hCo1Vh9OPYTHHbsBPEShC+C8g9yCpOyRFE/Oq7vNMr8L6DCPRHD7+ERt52t/dUXpSTxksUE99YI9Em4=@vger.kernel.org X-Gm-Message-State: AFuF++llXup0umQOmmp2AeOGK8kCihSHllo1EQThxxtTSgn0uMkhfMHE 8jMsqzopQ8Ytj9hcTdBIGwJ8wJrlMTP8+Z1hFDRYrOI+XcXZ3gYYjwxmtGsjpoT1tB4= X-Gm-Gg: AYBFou2LKB+JEy0eITpkQQGT2T9BbKo/WtdMTmi15s9CA0qlPRGr34ZGmkjZMpobIUB IdOlLpsjs0kLiWZrkwbBTrC5bu7XzdcpycLn6wqUDOATTGN7Mx44+w1GEwFTXOfjmuovnMvoySX ArGVBiHcKXrE6c2HzTvvPD1W48t3p70cLhRizX2dpAJsVqAam+Uz/D3w2t4R99l480bYGYfevv1 R2td6nseuSH55lRLJLfxyMJbjT1cxePR4CRgNB1EwksfA02ojXfUBWD1uLL58oqThdNIbUhsvcX 6qVo+ZAmDbARDo7N22Ipa7s9xhsxDbCd6htRLDWZgtuWat43iCK8XNtxLnQ7UktuGOLNNUoFFOc 8ImfbU45m5WREFeWEirXUhLZ/7prAz53DLXzvSOvgI4vCcaUmTTJTKQCpm8SHGuqy6uLQSs8ec1 pIVedUJbIcgddAiWDap0/5Z0Tldz7yYmqih/gAA+Nj2powhRfBz2v3ld5RbsMFeexKzFBUrItXh bYmJSV9HOEEH3p9sprEOOH6cvDEuP7ZXUc= X-Received: by 2002:a17:90b:1e0d:b0:37f:ed7e:7e42 with SMTP id 98e67ed59e1d1-39d70ae12b0mr7306314a91.14.1789022012813; Wed, 09 Sep 2026 23:33:32 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:32 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 03/17] mm/mm_init: skip initializing shared vmemmap tail pages Date: Thu, 10 Sep 2026 14:32:42 +0800 Message-ID: <20260910063256.64386-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memmap_init_range() initializes every struct page in the target range. For compound pages with vmemmap optimization, the tail struct pages are backed by a shared vmemmap page. Initializing those tail struct pages would overwrite the shared vmemmap page contents, requiring users such as HugeTLB to restore the metadata afterwards. Track the compound page order for HVO-backed sections and use that metadata to detect struct pages that fall into the shared tail vmemmap range. Skip those shared tail pages in memmap_init_range(), then initialize pageblock migratetypes for the processed range with a helper after the per-page initialization loop. Keep direct mem_section access inside sparse helpers. Expose pfn_to_section_compound_order() for callers that only need the order associated with a PFN. This lets memmap_init_range() skip shared tail vmemmap pages without exposing __pfn_to_section() to !SPARSEMEM builds. This is a preparatory change for consolidating handling across users of vmemmap optimization, and it also avoids redundant initialization of shared tail vmemmap pages during early boot. That early-boot benefit appears only once HugeTLB is switched to this common handling, since HugeTLB is the early-boot user that creates those shared tail vmemmap pages. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Rename the section order field and helpers for compound page order (suggested by David Hildenbrand) - Collect Reviewed-by from Mike Rapoport - Collect Acked-by from Qi Zheng v5: - Add comments for pageblock migratetype helper callers and shared tail vmemmap skipping (suggested by Mike Rapoport) v4: - Rename pfn_vmemmap_optimizable() to vmemmap_optimizable_pfn() for consistency with vmemmap_optimizable_order() v3: - Replace the !SPARSEMEM __pfn_to_section() stub with pfn_to_section_order() (suggested by Mike Rapoport) v2: - Fold section order tracking into the first user instead of keeping a standalone API-only patch (suggested by Mike Rapoport) - Rename page_vmemmap_optimizable() to pfn_vmemmap_optimizable() and pass a PFN directly (suggested by Mike Rapoport) - Initialize pageblock migratetypes from a helper after the per-page loop (suggested by Mike Rapoport) - Use a 1G PFN chunk for cond_resched() in the pageblock helper (suggested by Mike Rapoport) - Guard section_order() with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP so it returns 0 when HVO is disabled and lets the compiler optimize the code as much as possible (suggested by Mike Rapoport) - Explain why the !SPARSEMEM __pfn_to_section() stub belongs here (suggested by Mike Rapoport) --- include/linux/mmzone.h | 8 ++++++++ mm/mm_init.c | 40 +++++++++++++++++++++++----------------- mm/sparse.h | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 17 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5fb9b37819d5..738bf3d27256 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2022,6 +2022,14 @@ struct mem_section { unsigned long section_mem_map; =20 struct mem_section_usage *usage; +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + /* + * Normally, sections hold regular (order-0) pages. However, for + * sections with HVO enabled, this tracks the compound page order + * to enable deduplication of redundant vmemmap pages. + */ + unsigned int compound_page_order; +#endif #ifdef CONFIG_PAGE_EXTENSION /* * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use diff --git a/mm/mm_init.c b/mm/mm_init.c index 1533aebafb68..6655fe696e0f 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -677,21 +678,19 @@ static inline void fixup_hashdist(void) static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ =20 -#if defined(CONFIG_ZONE_DEVICE) || defined(CONFIG_DEFERRED_STRUCT_PAGE_INI= T) static __meminit void pageblock_migratetype_init_range(unsigned long pfn, - unsigned long nr_pages, int migratetype, bool atomic) + unsigned long nr_pages, int migratetype, bool isolate, bool atomic) { const unsigned long end =3D pfn + nr_pages; =20 for (pfn =3D pageblock_align(pfn); pfn < end; pfn +=3D pageblock_nr_pages= ) { enum migratetype mt =3D kho_scratch_migratetype(pfn, migratetype); =20 - init_pageblock_migratetype(pfn_to_page(pfn), mt, false); - if (!atomic && IS_ALIGNED(pfn, PAGES_PER_SECTION)) + init_pageblock_migratetype(pfn_to_page(pfn), mt, isolate); + if (!atomic && IS_ALIGNED(pfn, PFN_DOWN(SZ_1G))) cond_resched(); } } -#endif =20 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) @@ -886,6 +885,17 @@ void __meminit memmap_init_range(unsigned long size, i= nt nid, unsigned long zone } } =20 + /* + * Vmemmap-optimizable PFNs are backed by shared tail struct pages, + * which have already been initialized during vmemmap population. + */ + if (vmemmap_optimizable_pfn(pfn)) { + const unsigned int order =3D pfn_to_section_compound_order(pfn); + + pfn =3D min(ALIGN(pfn, 1UL << order), end_pfn); + continue; + } + page =3D pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); if (context =3D=3D MEMINIT_HOTPLUG) { @@ -897,19 +907,13 @@ void __meminit memmap_init_range(unsigned long size, = int nid, unsigned long zone __SetPageOffline(page); } =20 - /* - * Usually, we want to mark the pageblock MIGRATE_MOVABLE, - * such that unmovable allocations won't be scattered all - * over the place during system boot. - */ - if (pageblock_aligned(pfn)) { - enum migratetype mt =3D kho_scratch_migratetype(pfn, migratetype); - - init_pageblock_migratetype(page, mt, isolate_pageblock); + if (pageblock_aligned(pfn)) cond_resched(); - } pfn++; } + + pageblock_migratetype_init_range(start_pfn, pfn - start_pfn, migratetype, + isolate_pageblock, /* atomic */ false); } =20 static void __init memmap_init_zone_range(struct zone *zone, @@ -1112,7 +1116,8 @@ void __ref memmap_init_zone_device(struct zone *zone, compound_nr_pages(pfn, altmap, pgmap)); } =20 - pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, fa= lse); + pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, + /* isolate */ false, /* atomic */ false); =20 pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); @@ -1921,7 +1926,8 @@ static void __init deferred_free_pages(unsigned long = pfn, if (!nr_pages) return; =20 - pageblock_migratetype_init_range(pfn, nr_pages, MIGRATE_MOVABLE, true); + pageblock_migratetype_init_range(pfn, nr_pages, MIGRATE_MOVABLE, + /* isolate */ false, /* atomic */ true); =20 page =3D pfn_to_page(pfn); =20 diff --git a/mm/sparse.h b/mm/sparse.h index 3b744667a7e6..da79c83adeae 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -10,6 +10,39 @@ =20 #include =20 +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static inline unsigned int section_compound_order(const struct mem_section= *section) +{ + return section->compound_page_order; +} + +static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) +{ + return section_compound_order(__pfn_to_section(pfn)); +} +#else +static inline unsigned int section_compound_order(const struct mem_section= *section) +{ + return 0; +} + +static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) +{ + return 0; +} +#endif + +static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +{ + const unsigned int order =3D pfn_to_section_compound_order(pfn); + const unsigned long nr_pages =3D 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + /* * mm/sparse.c */ --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FE083B4E9F for ; Thu, 10 Sep 2026 06:33:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022021; cv=none; b=BRQH5/yvR0tDN3j9c4J8nh8auNMXkv0CNsEr28WmHoBLpzfiRZduV2ZpxRU1S0/uOHRRPhjzoG6FfBvcfZIXnukjHH1258zf7pj6flqYfGORzAeEqNMiKRv2gr/yx232R6OmbxpHdK3FDARW+jy32rk48fKcgTMfxNhN1A7Q4zg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022021; c=relaxed/simple; bh=r3eTsuegrmSjsVbobf7YQYE9D/MuS2nV0nnJ9e4pThQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=crWa96tEWeeps7l3rn8T6nynYfmtazcawlV3uAqXAKDVSNDSCVJmVtr+JrVDAhkjhZk6HwaE2DQBURNtNb8Bri/RHYgFdtXKqCS3Jx7xSM5OI/30hiFOT8rQdDWrgx4LF8uMgLadINETyfVZW6XwC1qTBhc7wcHeZ+W7vegXIq8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gRPilKV/; arc=none smtp.client-ip=209.85.216.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gRPilKV/" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-39675172593so5694540a91.2 for ; Wed, 09 Sep 2026 23:33:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022017; x=1789626817; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EnePsJhsQF7eMOx8C/uEqh/ex5Od6EnmeZ+DbGqwr5U=; b=gRPilKV/EsVz6/f9c6vUvzF/hTBGHurk8rUwMiUSNKRcixmulnqQ/CrT1eA3rNOwh6 8FvR49Z8f0tgLpvmsJQkYRxP/pUxv6HQo9WzPe62x3hlAmZPDtChQ5Uu0AoIir7eTAB1 Fm9S4AJzPbC9NbbXmj6zQXu71K6U9HUVuOdrTfHUhTA2DFyf+yhj+/HeBVU1zOU8K4LY HC5OG2GcYufOxwz0f4gY/9FYWYp+SS8wTriv6+KKQQYw03+Bge7UgAeM0ByGs/iotfkU G6grG46LLraRzDP0jZ4U57Rpx4e7hFB6IzosnXakj77HeCO/Q/T5rIlzBXzG0l6WRjNk 7Hqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022017; x=1789626817; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=EnePsJhsQF7eMOx8C/uEqh/ex5Od6EnmeZ+DbGqwr5U=; b=V35XCpu5J05WYD5g2/SHzfu4PqgoTRn7eZt9OX3CleFXG4546agHF3pHH1SM1zvapD CPDI4b9nUAxY2kC7GtFVHEh8zfXvaulLnFi17aULRQj41O+7Ea2e37dn4GNp3s9Voaf0 OYRRPx30qDD1fnI1BjIjtcJGM+8jMrf5LlF2RG/xgvWuuyg1HfDn9tqny3SseVyte+jY ZtAvNrRA8EFXOMKYu/MPFvZNYSoshVjI0fYRvHZrq5f0i9MJ/ECz4iZ6QJ4ouCXMjA0j URs+1xhViRDsYuIGx72QHpdLe/BHclu5J00yxB2dut0GDmHuXhtrbTBmeALBUb+WdN9m PriQ== X-Forwarded-Encrypted: i=1; AKwUvBz+dbZAFe+DGCc3Xx8MQpmF2+J8pqP++9B2Yofczq/Vvi6pU+10VBIvDoI4AK1eHGe7PeE0uvhIomSPQZA=@vger.kernel.org X-Gm-Message-State: AFuF++lDeloRo2RAWo2ulN1Ed3jLP+ZreomNiAMXlJCDizYLaH0H5TZJ hNn+eLqp+/3gYqT4nFUPRCovJS6AvCHJlvn2St5Okkk898y+IBft1xl9UiK3uRxhz1g= X-Gm-Gg: AYBFou0DKwJZiTzQTpM/+Vg3iwXfFg6s5z8LqFve7P2rqz7Tb1QScdJGAZ+plyIWd4r ykl1077NuBt5/UYsWFJqrpT6JvGNrelFQFmMrbc/ApO1YtECmow7z2GI3TIOEgkrjLP3GrXL+Mu 3UZRui9BfeDyWi9NeIM06mbc/qX2+cKMS/2Sv7emfF0f6uZugs+xgkxncz+DsqfkzjWvnEk/P/S 1K13+biVLUv/n8yxsL5fyPWcPB8bfSFQDJt/CEM46IUfUixMoR3CAJu3L9Yw9q49eFGr5WGupA/ 4dhGXsTVZ7U+LqVHEK8SLnZ3SomMjKC6HxFwpwr7TdczOmjSVGegCyzIeS1T9pXIp62f9sfHVaQ PmX/tk8ATlzbtRy9qqXD9/uAz8S7DmrSvkObdUfjnEra2aJN41UnoSSXaWDfk3krELJTbzXarY1 XVCFcsog6L8iQjrrwLi/6GIYfa/beF1jGmcli2+Rv7S0emsKs5/CH2XH7JuS1CHg1Hb1oHan4mQ YP2DHt7rdTq9VYDk2Dyq/M6ETsiyPrI9jA= X-Received: by 2002:a17:90b:50cc:b0:396:65dd:4093 with SMTP id 98e67ed59e1d1-39b26202ebamr61814255a91.14.1789022016692; Wed, 09 Sep 2026 23:33:36 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:36 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 04/17] mm/sparse-vmemmap: initialize shared tail vmemmap pages on allocation Date: Thu, 10 Sep 2026 14:32:43 +0800 Message-ID: <20260910063256.64386-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The shared tail vmemmap page allocated in vmemmap_get_tail() used to be left uninitialized, because memmap_init_range() would later overwrite it. That forced users such as HugeTLB to defer the initialization to their own setup paths. Now that memmap_init_range() skips shared tail vmemmap pages, initialize them immediately in vmemmap_get_tail() with init_compound_tail() instead. This adds initialization at the point where the shared tail page is allocated. The existing HugeTLB initialization remains necessary until the compound page order is set and memmap_init_range() starts skipping shared tails. It is removed when HugeTLB switches to the section-based mechanism. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Clarify the timing of shared tail page initialization removal (suggested by David Hildenbrand) - Collect Acked-by from Mike Rapoport and Qi Zheng --- mm/sparse-vmemmap.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aa6a4a2fae98..107215cf8488 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -338,19 +338,11 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * tail =3D zone->vmemmap_tails[idx]; if (tail) return tail; - - /* - * Only allocate the page, but do not initialize it. - * - * Any initialization done here will be overwritten by memmap_init(). - * - * hugetlb_bootmem_struct_page_init() will take care of initialization - * after memmap_init(). - */ - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); if (!p) return NULL; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail(p + i, NULL, order, zone); =20 tail =3D virt_to_page(p); zone->vmemmap_tails[idx] =3D tail; --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B82373CB2F8 for ; Thu, 10 Sep 2026 06:33:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022028; cv=none; b=G4Bbwd8N0hpL6ZzMLU2crS8WfV9XkDn9ERpzbfkAB+MuC6qA367yiZCywFSZLqvGrlqkqdNKP6aeWD+S1gr45GZ4i2uiB4h6TQtJIlTIjB/JJS6on1/WnNbmxxw/Uu6ccHJANMPYvntwIdtalzgq5MQuLq69dSWgK8A0214g3wM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022028; c=relaxed/simple; bh=tl7UM0nvXQVy6DcRjTkdTEbH8T0nV+ihXtWeAogjpiE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=twnDLNAvBUBHf1nvg1478CLLNqvp9ofHm08fTf6k8nTlyZfyatipX4pd2A8Ch5w/zZ7EZf7JTBz9MiDcnwEduUASztOMQ/j9/d0dm2i+Z8P3mmWXRhKPTBb/Gr+K/x0pNSwUwAGt9E8SB12QrxH2q9VvfenZcrlwVgxaETkwPPs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=H30SjaII; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="H30SjaII" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-3969e82ff8fso7079181a91.0 for ; Wed, 09 Sep 2026 23:33:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022021; x=1789626821; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wA3VEQ0Gv26Nz0ZpUy3A1OXLwMHz5Ri5QNZ6qYOOTyY=; b=H30SjaIIEAtaK/+LFCjoRr3F4K92FSPdR+hZFRSAuDFTIO+fjYTlRSUL427rdb2DA3 LRi9f6Z9B1mZfyHVy9YLlD69mwLBZKrGD1lfoSJItmG9sZ9AD7RI+l2myMGzhmA2C1cj KvrTi+o75Mrrdm4N/XbqIQB9BRDjHaZfaCTzaRFCy/1WX604jAlc5inzMTfK0SdKhllT RwdQtkYfs2XzBIBiyVPOr0fzu5B7dkCl1fYJ64kHfiqNE5ONPRRinRG8FzzSDOeV/zPQ E1cf/91uN5ZyFF8v6oEcDwfsXTGe0fG2xez4nR7aPyIdJcG0qxoyn95SYjkJva3t+2zB fGbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022021; x=1789626821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wA3VEQ0Gv26Nz0ZpUy3A1OXLwMHz5Ri5QNZ6qYOOTyY=; b=HR4xwKbRa9X9hZJ5VhNiRBUHiwh/3nw6ewfedyprgzSmydUyp7NjaQ8Fl3EJswvbfx obFHv7LcPeGFH6EKH734AZ4BQOFnkTw97zvfENXXc88YH+GRnw5J+/hAI0JSkjGdW6Hr t3VYn9lHPvJMnZ9/1kz+JF8mNIPjgTtpaPJ2ysW3XSy12jgQJ6GRubWPGB7PxH5k5O7E vMMgJGd/7zjVnH7fJnlaafNLCIxjc5ITbTZl2G7RSBXe/iDmFaNkH0oKLZ/jOEFk/FaI IUgoejctNiueOHVbMGGjlJOsEaOTeQUqT/ev/jg2iJG4S/duudma9mTFbUoKbnby8qMl 47gA== X-Forwarded-Encrypted: i=1; AKwUvByjBmCy5SUq4BmCHzAjJ7TCHdBHT2IV4hPpcbCLzV0KhVcjBqgN0H7RkoZGYTFgbZ6AYk+Zr5xKT2qYsEk=@vger.kernel.org X-Gm-Message-State: AFuF++lsS78vZY30qEk3rkfgNS2uzMoTSbDM3/FE6gamxYC8N3Efc/Ov PJZK92mAC8jjAvUvqL4X8rLnSPHWdb2MsqiwigtDno3Fg1LYDmVc2gGaKO0MgK6TBJg= X-Gm-Gg: AYBFou3qlIIjDpcKN2HbpVAL/QIin4GBPDdkazkQBTzQcHooAToqVZYkWedl5pX/ZMB Zw0JbooPoyTDZ1FgF4UsJJKRoCJWJNILuuRn0/4epjvF+CRTz7BtYYJpewC5nMNQ+S+QjFKCdiQ jAu1ShlV/1ja1My/2/qIxKt0lRM/cNQLuA2fn2D2nSm+IVlTWLd1xKfaEWE0Pm1sn9KNG30gmdV zAGylvEnjVEV+aU291VWMytQHA77p8Vt9ccA4I/9ncITIRE6Jh5GvjvAQ3q6AsjfI28c9YredLu gfaw78Tm1ZhyVRCRM/2zIjjaqILAqLp6mTQo878VG7yipYGj9zIA1VNHsHkEmzoYcuAO2qMPjjQ Q+0ejmg/qRZFyLlbGTz0wG8g23q5FL2kuH93+0RzsZFs1N/PriZg2/00WFjOlTLOZYsLy+rSQ90 5FjSQji/qOxOyCpOQIa5cmxLEIjTe0gUwJaE8I4PhLlJyjp+U693hp0076ZlrDi4j9bAz2yVmGI sLVN25mEWD+03wmDrC36H+ke3aecPjgnHk= X-Received: by 2002:a17:90b:5844:b0:38f:5869:387b with SMTP id 98e67ed59e1d1-39b26101c20mr60325860a91.9.1789022020648; Wed, 09 Sep 2026 23:33:40 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:40 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 05/17] mm/sparse-vmemmap: support section-based vmemmap accounting Date: Thu, 10 Sep 2026 14:32:44 +0800 Message-ID: <20260910063256.64386-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_nr_vmemmap_pages() can account ordinary sections and DAX sections, but section-based vmemmap optimization stores the compound page order in struct mem_section and retains a different number of vmemmap pages. Teach section_nr_vmemmap_pages() to recognize section-based optimized sections and calculate their vmemmap page count from the stored compound page order and the HVO retained page count. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Use the compound page order helper for vmemmap accounting (suggested by David Hildenbrand) - Collect Acked-by from Qi Zheng v5: - Collect Acked-by from Mike Rapoport v3: - Add vmemmap_optimizable_order() for order-based optimization checks v2: - Remove an unnecessary vmemmap_can_optimize() call to simplify the code (suggested by Mike Rapoport). - Rewrite the commit message for better understanding. --- include/linux/mmzone.h | 6 ++++-- mm/sparse-vmemmap.c | 10 ++++++---- mm/sparse.h | 16 ++++++++++++++++ 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 738bf3d27256..2c4f63e379da 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -107,8 +107,10 @@ is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 -/* The number of struct pages covered by the retained vmemmap pages with H= VO enabled. */ -#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (PAGE_SIZE / sizeof(struct pa= ge)) +/* The number of retained vmemmap pages with HVO enabled. */ +#define VMEMMAP_OPTIMIZATION_PAGES 1 +#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES \ + (VMEMMAP_OPTIMIZATION_PAGES * PAGE_SIZE / sizeof(struct page)) #define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STR= UCT_PAGES) + 1) =20 #define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 107215cf8488..dea7179fbc90 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -649,24 +649,26 @@ void offline_mem_sections(unsigned long start_pfn, un= signed long end_pfn) static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - const unsigned int order =3D pgmap ? pgmap->vmemmap_shift : 0; + const struct mem_section *ms =3D __pfn_to_section(pfn); + const int order =3D pgmap ? pgmap->vmemmap_shift : section_compound_order= (ms); + const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); =20 - if (!vmemmap_can_optimize(altmap, pgmap)) + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); =20 if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return VMEMMAP_RESERVE_NR * nr_pages / pages_per_compound; + return vmemmap_pages * nr_pages / pages_per_compound; } =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); =20 if (IS_ALIGNED(pfn, pages_per_compound)) - return VMEMMAP_RESERVE_NR; + return vmemmap_pages; =20 return 0; } diff --git a/mm/sparse.h b/mm/sparse.h index da79c83adeae..563fc1f4d717 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -43,6 +43,17 @@ static inline bool vmemmap_optimizable_pfn(unsigned long= pfn) return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; } =20 +static inline bool vmemmap_optimizable_order(unsigned int order) +{ + if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + return false; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; +} + /* * mm/sparse.c */ @@ -86,6 +97,11 @@ static inline size_t mem_section_usage_size(void) return struct_size_t(struct mem_section_usage, pageblock_flags, BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS)); } + +static inline bool section_vmemmap_optimizable(const struct mem_section *m= s) +{ + return vmemmap_optimizable_order(section_compound_order(ms)); +} #else static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pz2-f12.google.com (mail-pz2-f12.google.com [74.125.228.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13FF93CAA2F for ; Thu, 10 Sep 2026 06:33:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.228.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022032; cv=none; b=rsI4iryQM6CoPmxM+YJBnQ2tYEag4q25kXJEYFWcoGSqb2MDuaVS1L8KOLMgbwr1JDvuefIGJF0nbULBT3Mml85eL2EzHSdFSU+RQvpDohzJwZKIqkWESs7lWRHmz7LgpaTUsgXVrDGDHsWDl9WaOYWQx/8b9M6gzcHOhhiEUJQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022032; c=relaxed/simple; bh=rdbrKcho81gLv+stcRUCjIzOrzflQStZZrMq5dxtYFM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rLBeB4tGn6dxgl/GFp2O4Y6EXJcZgUkcQTQ7xmnu2sj2gcF4VHOR2AiDv0umZ3mCoQXsZUXNpjUDtFIkIMBQrVJ+KjHYosRdITkG1h2e9T4IhX7UpAX4jBDiUvO+MjbezPVZhn0fcPjfPMeKfcqzxkGAPN4PghG9GzrkT74HPbc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=NZGf/nD7; arc=none smtp.client-ip=74.125.228.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="NZGf/nD7" Received: by mail-pz2-f12.google.com with SMTP id 41be03b00d2f7-cc1cebad4acso1545763a12.3 for ; Wed, 09 Sep 2026 23:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022025; x=1789626825; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EO4Rt3WEu+fPYQITg4bt0hJ4ppm+DADkMZhoEd2TtEM=; b=NZGf/nD7l7TNBMOTwkOTztkIys3FxbdVUA8ynP/aOMN78FEG5pK/vW0cAzOW9alKUK ea4+RFNnWVGwCtKdh6Jd4iV0lr+5RkeNGQTE+C6MHuh1W+UZ/YCdozgzHnWZPLOMRJsR 2eaEEJNised+/2+SaH5ImJbeagkUahJKcbuJUffLGAqrDB/BWd4HX/sZfOuHKhaQ9MJN d6b7ghNQQiJShLsr9EvmMcLu5D9VBHn+5Mcplsh05VADa7oaErg2riY1wOcm3w/SsM0I Zwk7luUDU1y3r+6rAfiw0f4tJcujA9CxHa0Ubn68chYggd/rsJTYePonmvLREGCEeV1q 6axw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022025; x=1789626825; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=EO4Rt3WEu+fPYQITg4bt0hJ4ppm+DADkMZhoEd2TtEM=; b=ccth+NMVKltnHXIGLc3+g3W3VGWCHTcUvicTzo17eRU4CiKBb+v9ymy3bbeuTjD2KB 1Ru2QaDSwszvvz4H2rnLlir6WXkGG9rSpcErZFTbFLL1ZWXnZk5CuOSfPR+0JUAwGZRC pUfM8V7AzEwoy/e8dosfmgewsWcf7BTK7fiitBaVVNr3IMa97lWIzqMrhYAUkI6JzQ21 O7Tg769F/T8ynyCietiEeuan5v2ciTXZ0LEARq6PDJhXYTd9nfOv3a7TLA2iNCIs2oQK yy6YBpx73TuI61l1MlW/XTWtdRCqJ9rynW6rti9d3egzbHbD4oA4VyZdTg9g0Km0RSan 2AVg== X-Forwarded-Encrypted: i=1; AKwUvBz+BO0zqKKIn/CVW+1/Wsm68Lm9CewhU5WgSR/LjbqXRIOx7NxDJkE2fsB3mijROBjd07kU6IyvD28EQsA=@vger.kernel.org X-Gm-Message-State: AFuF++maz2Pw4Cth+6AL1ng2w9cqWz8nfQae9wqNHPtduxXB/+4/3TeE JJNS96t7tuUFx4jNuZ5dBn7uoLXfqNQt1eoPZ+Isi+wXRF6GaYCwGrbzfV6GOQ2g5NE= X-Gm-Gg: AYBFou2k615HhH9h1w7qaLxFUr4RqukM7drn6vGEeEr7t4UGEe0gx1WPCdFU+wh/W3W CPaMjxinQ7o17LGLBsTk0zc6ATJqeKV2h9fV2n7AQRH3YTJCWRfbvVIiiYCjoMoydettV5QjYGP pywSPzJlzcRthG6DWeJHZXSGOD1FOL7HYgFeJfhuVZOFcZjNdky3Yhcp3L1CaMYG4gG7LDh1tc1 xb97/1konPAOIjl2mxxu6c1pn/fR/TtqVnI9LFEN/TUBek0Sllz2YeLd6glZ4rdzkUfGREnN7yE 8vU9scm4ip5abfUco2vQX9iW7yDnPnncXNnil9MesZhyhFfE+YCK8y5mVe4MBV3UsTeLB87tN6s gh+wfnQLC/QdDYVCRZj3Ax0uQZTnjqJz2OccP+xa5XA/iaNgQ0RIl35DW/ywR1nuklFAyuKvPot 9sl5cULPDmCIC8cXBjmtiupBSqU4RTLYnD5rLO4smoPITAt7t06yUw2Jq3Xcz5ZVEBB2UYncbyu nNaEcPMuvXxj7QIcqTVTXWebcAIM4458nY= X-Received: by 2002:a17:90b:270b:b0:398:c9be:cca8 with SMTP id 98e67ed59e1d1-39bac13b38dmr17069002a91.2.1789022024575; Wed, 09 Sep 2026 23:33:44 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:44 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 06/17] mm/mm_init: factor out pfn_to_zone() Date: Thu, 10 Sep 2026 14:32:45 +0800 Message-ID: <20260910063256.64386-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" pfn_to_zone() in hugetlb_vmemmap.c duplicates the zone lookup logic in __init_deferred_page(). Move it to mm_init.c, declare it in mm/mm_init.h, and reuse it from __init_deferred_page() and HugeTLB early vmemmap initialization instead of open-coding the zone walk there. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Collect Acked-by from Qi Zheng v3: - Fix the commit message to name mm/mm_init.h instead of mm/internal.h - Collect Acked-by from Mike Rapoport v2: - Move this preparatory patch before the sparse-vmemmap optimization changes (suggested by Mike Rapoport) --- mm/hugetlb_vmemmap.c | 17 ++--------------- mm/mm_init.c | 28 ++++++++++++++++++---------- mm/mm_init.h | 1 + 3 files changed, 21 insertions(+), 25 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index ae8fdaa42118..c48fcea076a5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -19,6 +19,7 @@ #include #include "hugetlb_vmemmap.h" #include "internal.h" +#include "mm_init.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -744,20 +745,6 @@ static bool vmemmap_should_optimize_bootmem_page(struc= t huge_bootmem_page *m) return true; } =20 -static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) -{ - struct zone *zone; - enum zone_type zone_type; - - for (zone_type =3D 0; zone_type < MAX_NR_ZONES; zone_type++) { - zone =3D &NODE_DATA(nid)->node_zones[zone_type]; - if (zone_spans_pfn(zone, pfn)) - return zone; - } - - return NULL; -} - /* * Initialize memmap section for a gigantic page, HVO-style. */ @@ -787,7 +774,7 @@ void __init hugetlb_vmemmap_init_early(int nid) map =3D pfn_to_page(pfn); start =3D (unsigned long)map; end =3D start + hugetlb_vmemmap_size(m->hstate); - zone =3D pfn_to_zone(nid, pfn); + zone =3D pfn_to_zone(pfn, nid); =20 if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) diff --git a/mm/mm_init.c b/mm/mm_init.c index 6655fe696e0f..ee3bd7921768 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -692,6 +692,20 @@ static __meminit void pageblock_migratetype_init_range= (unsigned long pfn, } } =20 +struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid) +{ + pg_data_t *pgdat =3D NODE_DATA(nid); + + for (enum zone_type zone_type =3D 0; zone_type < MAX_NR_ZONES; zone_type+= +) { + struct zone *zone =3D &pgdat->node_zones[zone_type]; + + if (zone_spans_pfn(zone, pfn)) + return zone; + } + + return NULL; +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) { @@ -750,20 +764,14 @@ defer_init(int nid, unsigned long pfn, unsigned long = end_pfn) =20 static void __meminit __init_deferred_page(unsigned long pfn, int nid) { - pg_data_t *pgdat =3D NODE_DATA(nid); - int zid; + struct zone *zone; =20 if (early_page_initialised(pfn, nid)) return; =20 - for (zid =3D 0; zid < MAX_NR_ZONES; zid++) { - struct zone *zone =3D &pgdat->node_zones[zid]; - - if (zone_spans_pfn(zone, pfn)) - break; - } - __init_single_page(pfn_to_page(pfn), pfn, zid, nid); - + zone =3D pfn_to_zone(pfn, nid); + __init_single_page(pfn_to_page(pfn), pfn, + zone ? zone_idx(zone) : MAX_NR_ZONES, nid); if (pageblock_aligned(pfn)) { enum migratetype mt =3D kho_scratch_migratetype(pfn, MIGRATE_MOVABLE); diff --git a/mm/mm_init.h b/mm/mm_init.h index 39f75df9be1c..c9fc35e7e9f1 100644 --- a/mm/mm_init.h +++ b/mm/mm_init.h @@ -39,6 +39,7 @@ void memmap_init_range(unsigned long size, int nid, unsig= ned long zone, enum meminit_context context, struct vmem_altmap *altmap, int migratetype, bool isolate_pageblock); +struct zone *pfn_to_zone(unsigned long pfn, int nid); =20 #if defined CONFIG_COMPACTION || defined CONFIG_CMA /* Free whole pageblock and set its migration type to MIGRATE_CMA. */ --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BD493C9456 for ; Thu, 10 Sep 2026 06:33:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022039; cv=none; b=IqWjlVR9uAurlGguZL5mBalf8nhI2w51AzHXsw+Bi2pJwYU1tpX2l2IX1MqtQL6dgWEzR5PnlZoCRqrxUuYll73ivfxU+2dGhJJ3osDjFGgL+zzPnV3yT9CElbcqOaxhIEqk7IWuZ2oyaH+Op5PiaMjFKIPUg9H89oMTVa5dOtU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022039; c=relaxed/simple; bh=2Mk0vaX73XX03B9QYgUcsK912ze5m7mL2XVK9F8MRGQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TlSPbC80hB9s99CvrJQSuzDF6Han3jG7odZRG2luz5tuDDyXLcABk2jSzS0bo0qBu2y8nEbcTHxF2AsQTjTj67s10r64Ti6y9F53RKTSwu5+zY2PcdOjeVlg4NoucsI3ETzm9jNqZFE6UD2042L7FyC7Di5Isvpvj9nRbTK3Wvw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LRmJ8lXm; arc=none smtp.client-ip=209.85.216.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LRmJ8lXm" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-3964e480f76so7891422a91.1 for ; Wed, 09 Sep 2026 23:33:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022029; x=1789626829; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=hiIQxlaJeEFoBo/L9xN4cjkO8AZZ+a3pO3zM7+AvDT4=; b=LRmJ8lXmiyeGM/JJT7vmq9Bs91+BuExyTii2z2UVBwIykm0VRiGEXBM2RoEU7Rwjgy 0Mm7f7TT/qpBWFuXxl2erF4u0t1fOV99Bc8KrogNZiDfjXm/f5VfsjneH413671xUp2z 0tixA+eHQnWbIrFnSUi8NkMnilaNNvTxctAF5iFJoI4UJq+RWMUihFj5gj2e+nq1uWrG u/P3cCJaEz6SGPAvjwHKMq8ZOqjKiSaGL/5pF1MoLXxVc4VRPn51zDjna0M7tXFEFJoz 0WlCbTHRgpWtcN1taEKwEDo0rk3OlsWe4rNj9m+F3bva4CwyNNSfsXKGOETZwPvvKQ6v nn5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022029; x=1789626829; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=hiIQxlaJeEFoBo/L9xN4cjkO8AZZ+a3pO3zM7+AvDT4=; b=KEUxBECbSLv9Ada0CYRi+Q1IpkVcEEpoVA4buvokrRXiowLii7CuHVFqEhAQC8CRMi 9/r+DJFWIkENi5RyDyXxx5+70qXEhtz0lXGe5G/LiwoER3tku1Qd2+mYJS+vEOUwYoaL qBRIGVA4gGbVEzKccgvo7ANP2yZNSJsWi+699JD1b/oh+QG2zX8RyHQcweBBRia0DBWk G5Wev0ajAqKpBt6kl+35IZUKjIVYQlEVbvIbRPKl8RBS2mBKwmhue/dbRcSMhe++SrV8 GsawMJA/jXXT/OsGm0Jy1XxMrTOuLO2Hs6cXNLLiV6X2QVd4LLjIMkSsuZIGF+RIZsEQ sx9w== X-Forwarded-Encrypted: i=1; AKwUvBy7K6uqFXNRInyR3PNwbXzTOEbARD6Kqrx8/6ExM1TQmuliqCh4Sa1wisCP2EoD7iMmA3d1OEkNey8WFKg=@vger.kernel.org X-Gm-Message-State: AFuF++lK74XF/ma8WSM7S5+dWAlsf5MLTYeDzdegkkl6Ziaynkn018Mf TFeT3/xR0bo+uaO9MZWC+z+IBZUayTvxkC4dMFeTR2xkfPZUFTMYKklqPCV3j6euru4= X-Gm-Gg: AYBFou0nJuU2c1H6NjoQjtv06Ek+kCURIcfwRyQC8BQzbvRul0ILeurpMGF8Ix1Ahm5 bFzXkwkXgTnhrcKDSfIp8myXEu6QdxUEnOZCBNtVcCKVob7AaDp0QPEY6DHxu0nqJXkTvAEKpu6 QgdYu2pgXiXEiX5yJm5pwtqgAh00HRYcLnyCvQIoYsCJPQ6r91lNIcB6sakOZUsHsaq435iCj4Z zwQMnSzjRD+Ws06y6XQjEp6tuh3AWZh+sSglE6vXrC20ecbNTKFWbl3PA2tEZ8tCNr0nbmfUSIW ee+raaUQHWIJ1tlESaUAzD21ivi6NmZ93Tlzo1Vk0p9H7I12bfjYCaRP4Ytc1q0sOnalTk9dLh+ eGVHZ8avCPv3oW0m/fuPoIWyWgYZZqaf4zXDyW2wl/huixscJDtTXV8kqojYVXInaN1OQFNH4e1 B9wMTmXcjnZw2rF758oFdNIRBw8Db/386rux0ySAGI6D+UMh4x42Igvlbg0MKvCiAfZhp4insLW ZmVyFzQMLcXFFeN9ogRhxELZFYUSztL4NNiDOu3evwpyg== X-Received: by 2002:a17:90a:c106:b0:398:e1f4:bda1 with SMTP id 98e67ed59e1d1-39b2624cf3fmr61513978a91.21.1789022028572; Wed, 09 Sep 2026 23:33:48 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:48 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 07/17] mm/sparse-vmemmap: move helpers ahead of future callers Date: Thu, 10 Sep 2026 14:32:46 +0800 Message-ID: <20260910063256.64386-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Prepare for section-based vmemmap optimization by moving helpers that follow-up changes will use. vmemmap_get_tail() will be called from the PTE population path. section_nr_vmemmap_pages() will be made visible outside the memory hotplug code and called from sparse_init_nid(). Move vmemmap_alloc_block_zero() together with vmemmap_get_tail(), since the tail helper depends on it. Move section_nr_vmemmap_pages() earlier into its own CONFIG_MEMORY_HOTPLUG block. That lets the later patch change its visibility and callers without also moving the function body. No functional change is intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Keep the moved accounting helper on compound page order naming (suggested by David Hildenbrand) - Collect Acked-by from Qi Zheng v5: - Move section_nr_vmemmap_pages() earlier under its own CONFIG_MEMORY_HOTPL= UG block to avoid code movement in the later visibility change - Reword the subject and commit message for the helper movement - Collect Acked-by from Mike Rapoport v2: - Add this new patch to move vmemmap_get_tail() before PTE population (suggested by Mike Rapoport) --- mm/sparse-vmemmap.c | 134 +++++++++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 65 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index dea7179fbc90..e2a2a5eab4dc 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -148,6 +148,75 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 +#ifdef CONFIG_MEMORY_HOTPLUG +static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + const struct mem_section *ms =3D __pfn_to_section(pfn); + const int order =3D pgmap ? pgmap->vmemmap_shift : section_compound_order= (ms); + const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; + const unsigned long pages_per_compound =3D 1UL << order; + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); + VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); + + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); + + if (order < PFN_SECTION_SHIFT) { + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); + return vmemmap_pages * nr_pages / pages_per_compound; + } + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); + + if (IS_ALIGNED(pfn, pages_per_compound)) + return vmemmap_pages; + + return 0; +} +#endif + +static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) +{ + void *p =3D vmemmap_alloc_block(size, node); + + if (!p) + return NULL; + memset(p, 0, size); + + return p; +} + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) +{ + struct page *p, *tail; + unsigned int idx; + int node =3D zone_to_nid(zone); + + if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) + return NULL; + if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + return NULL; + + idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; + tail =3D zone->vmemmap_tails[idx]; + if (tail) + return tail; + p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); + if (!p) + return NULL; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail(p + i, NULL, order, zone); + + tail =3D virt_to_page(p); + zone->vmemmap_tails[idx] =3D tail; + + return tail; +} +#endif + static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long ad= dr, int node, struct vmem_altmap *altmap, unsigned long ptpfn, unsigned long flags) @@ -181,17 +250,6 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *p= md, unsigned long addr, in return pte; } =20 -static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) -{ - void *p =3D vmemmap_alloc_block(size, node); - - if (!p) - return NULL; - memset(p, 0, size); - - return p; -} - static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long ad= dr, int node) { pmd_t *pmd =3D pmd_offset(pud, addr); @@ -323,33 +381,6 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigne= d long end, } =20 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) -{ - struct page *p, *tail; - unsigned int idx; - int node =3D zone_to_nid(zone); - - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) - return NULL; - - idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail =3D zone->vmemmap_tails[idx]; - if (tail) - return tail; - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) - return NULL; - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - tail =3D virt_to_page(p); - zone->vmemmap_tails[idx] =3D tail; - - return tail; -} - int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, unsigned int order, struct zone *zone, unsigned long headsize) @@ -646,33 +677,6 @@ void offline_mem_sections(unsigned long start_pfn, uns= igned long end_pfn) } } =20 -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) -{ - const struct mem_section *ms =3D __pfn_to_section(pfn); - const int order =3D pgmap ? pgmap->vmemmap_shift : section_compound_order= (ms); - const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; - const unsigned long pages_per_compound =3D 1UL << order; - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); - VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) - return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); - - if (order < PFN_SECTION_SHIFT) { - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; - } - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); - - if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; - - return 0; -} - static struct page * __meminit populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE6BC34CCE for ; Thu, 10 Sep 2026 06:33:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022038; cv=none; b=Btpo0dxaB/MlYzVmxKN7BgzOfjPqMrRhcxE+MyGjKidg1e50dMXZel1+98s7vFspjP/qxLtcASxwhnulUT84OE2sftzASxCIQi6hTCUNNT4bVNX6USH7E7CRiIFR31+XEMA7hGAz7z1hecDYedCLjiRgkQV7ywQtpZ+ZJd3smbk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022038; c=relaxed/simple; bh=hEq7qIpFnYAm54s5gTdB9wGUTTRdN/+pUlavhzSVW9g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=thvsdxCOVygraQ2NlCH6AiUKnJOjwB83vxuwzQ1BNmU6U31Vh1HZ0Z6fRv1yxcJhqfqsuOZL8E5e9s2qjOFl8yTBHOLTQS36xsDBH5MQMw1lEVvM4hXVHRldrTL4RnTTyUazG4BQ1m7t3VVhhxPYtbkP4iiOpiF5/x3jwC/p/Q4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=HVARQKpb; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="HVARQKpb" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-398b3d66515so6891450a91.0 for ; Wed, 09 Sep 2026 23:33:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022033; x=1789626833; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fScr4mRCKk6xdGZOiQP1hnWEEEG9rl2EUQeAyvKGaqc=; b=HVARQKpbdQyWkZ+3jcOLb9I10gx58XGPKg2VhZXYGaAyUTisFvxvMoa07Ilzu2oV0i As9ffA1Qz9TNoZDvJZu7skjLLuN92lrc4dqDcT3VxSQJyjtO8CWYN5xfVaCjIOptNM5p WMz2Hg9fTDkKZMapVbOzW4AOICbzPvyXBsXUv0F5ZX4Dbuv9IWbWq7sHBsGevjhXSC0y tIxOPX3bc/X1ZXOWT//5dY5E5hzFr1u3X5/VNHTDqJVV6U0sbqyLjoGzTWG8iDCv9gVe YbQiE8Nfs2mTcyxNh7joisNSeul3bXoBY2VKsgzxWF5eRFOiSo1+NXVI+RpaLWyVehQw tKMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022033; x=1789626833; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=fScr4mRCKk6xdGZOiQP1hnWEEEG9rl2EUQeAyvKGaqc=; b=I7KGbgJSR41sTtXSy3AowGf5eZc7ZkA5IhNsjnFB3inZCET3Dlkp5PfsHoHqDh8wpm mfCGcI3PN401Etejk/DCwEYW/qYsu/dF4d+8isw/xsJ/VnYcJ98/nCWiAEDQIi/x+8v0 jbIOMpVMoM0dS0bgAACLXlRijMnNnFaEwjB9Soh9y6igYnshryT+Vxku6qhfD216B5Zh 2vaPQthL8YNa02BxticfnXz7l3LeMIUDdr2C2WW0O+g3YGxhV+7FKe+jsdRsQTrHN73N qOueSjmGk0/DZeAltr3YKEJu/bUhHZJZ5NT6O8sW/urRXtZg5duzdlbKvTCf/p6ay8h4 k4KA== X-Forwarded-Encrypted: i=1; AKwUvBxCfnqNHTv6Vv0hxdHuwK+b2ZDzIHj3x7CHJt7438ahl6t2y5EGHVPxrBfqVMWz3QWcXpwElB9xNM48kIE=@vger.kernel.org X-Gm-Message-State: AFuF++nQEIsl1lxNxFsXShZObPyF1lSjuTs0OEHD+mtsXbEctJUH3Ys8 48gpa2AqkkqseRN8uOJsMQzn+V7lF4mLM6tP/BDdmk76pFarLhmyqiW69NpPnXg9L/4= X-Gm-Gg: AYBFou1mOZ1JPkBOmAS3usyM67lFQ7gf88JZxjecHC/otAkx3YJ6AoiRtBDJ7aa8iUG 1RuGPPt4oHo/so2l9JTNSF3WDY7TQd5XQx/I3h++hhOBUjX1X4Ba+pdV18zNMyNhu0wmXhCJ/fA epOnuNBkxTSRORerIR52d3Bl7C7LNSm+7YbumN/r69B4Elm+5fGzajUJSkeku29/VR7xRgdjuqi 0TWdXMGt+ZyyodlDvgR3XAhpo+ZRPWOiX7TYMaeR8aUIFs9Z3rjEf381QhJRVy7zu5uAEmWfDnt LDvG3ZuFByszegVVvp2gBX11+Z0//5g1Lx/fA2ulpWB3rGrr3Qxn3uhNLkMZL3617EGTP+ymUfV 6/4b7tj4On1onT0c4miCysfKW/AnZC8SxfpaHFWiAfmAocPqK7tiSiw4KCrOvEAwYqLhG/dPt3X rcUgkABH8dBwLa7/foYDIKv56r4mdmp0pLIWnbd2jEgHM8QqgTBKK/YK/H4sZT3g89uWDm4Z71w 8MBXcvZxsawIUBQunrHePnaV7gUX3HYwU7ZAAjAe1vL4g== X-Received: by 2002:a17:90b:50cc:b0:396:65dd:4093 with SMTP id 98e67ed59e1d1-39b26202ebamr61815629a91.14.1789022032532; Wed, 09 Sep 2026 23:33:52 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:52 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 08/17] mm/sparse-vmemmap: support section-based vmemmap optimization Date: Thu, 10 Sep 2026 14:32:47 +0800 Message-ID: <20260910063256.64386-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach sparse-vmemmap population code to use the compound page order when deciding whether a vmemmap page can be optimized. With this information, the common sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages directly instead of relying on HugeTLB-specific handling. This centralizes vmemmap optimization logic in the sparse-vmemmap code, based on section metadata, and prepares for sharing the same mechanism across different users of vmemmap optimization, including HugeTLB and DAX. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Use the compound page order helper in the population path (suggested by David Hildenbrand) - Collect Acked-by from Mike Rapoport and Qi Zheng v5: - Add vmemmap_alloc_pte() to keep PTE allocation handling clearer (suggested by Mike Rapoport) - Move preparatory section_nr_vmemmap_pages() changes to the previous patch (suggested by Mike Rapoport) v4: - Move section_nr_vmemmap_pages() outside CONFIG_MEMORY_HOTPLUG to fix CONFIG_MEMORY_HOTPLUG=3Dn builds (reported by kernel test robot) v2: - Keep vmemmap accounting and population logic in sparse-vmemmap.c (suggested by Mike Rapoport) - Move vmemmap_get_tail() before its first use instead of adding only a forward declaration in the previous patch (suggested by Mike Rapoport) - Simplify the PMD path handling for HVO-covered sections --- mm/sparse-vmemmap.c | 46 +++++++++++++++++++++++++++++++++++++-------- mm/sparse.c | 4 ++-- mm/sparse.h | 7 +++++++ 3 files changed, 47 insertions(+), 10 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e2a2a5eab4dc..faebf344cdac 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -148,8 +148,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 -#ifdef CONFIG_MEMORY_HOTPLUG -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms =3D __pfn_to_section(pfn); @@ -175,7 +174,6 @@ static int __meminit section_nr_vmemmap_pages(unsigned = long pfn, unsigned long n =20 return 0; } -#endif =20 static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) { @@ -215,19 +213,44 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * =20 return tail; } +#else +static inline struct page *vmemmap_get_tail(unsigned int order, struct zon= e *zone) +{ + return NULL; +} #endif =20 +static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, + struct vmem_altmap *altmap) +{ + struct zone *zone; + struct page *page; + const unsigned int order =3D pfn_to_section_compound_order(pfn); + + if (!vmemmap_optimizable_pfn(pfn)) + return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + + zone =3D pfn_to_zone(pfn, node); + page =3D vmemmap_get_tail(order, zone); + if (!page) + return NULL; + + return page_address(page); +} + static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long ad= dr, int node, struct vmem_altmap *altmap, unsigned long ptpfn, unsigned long flags) { pte_t *pte =3D pte_offset_kernel(pmd, addr); + unsigned long pfn =3D page_to_pfn((struct page *)addr); + if (pte_none(ptep_get(pte))) { pte_t entry; - void *p; =20 if (ptpfn =3D=3D (unsigned long)-1) { - p =3D vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + void *p =3D vmemmap_alloc_pte(pfn, node, altmap); + if (!p) return NULL; ptpfn =3D PHYS_PFN(__pa(p)); @@ -246,7 +269,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in } entry =3D pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); - } + } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) + return NULL; return pte; } =20 @@ -435,6 +459,9 @@ int __meminit vmemmap_populate_hugepages(unsigned long = start, unsigned long end, pmd_t *pmd; =20 for (addr =3D start; addr < end; addr =3D next) { + unsigned long pfn =3D page_to_pfn((struct page *)addr); + const struct mem_section *ms =3D __pfn_to_section(pfn); + next =3D pmd_addr_end(addr, end); =20 pgd =3D vmemmap_pgd_populate(addr, node); @@ -450,7 +477,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long = start, unsigned long end, return -ENOMEM; =20 pmd =3D pmd_offset(pud, addr); - if (pmd_none(pmdp_get(pmd))) { + if (pmd_none(pmdp_get(pmd)) && !section_vmemmap_optimizable(ms)) { void *p; =20 p =3D vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); @@ -468,8 +495,11 @@ int __meminit vmemmap_populate_hugepages(unsigned long= start, unsigned long end, */ return -ENOMEM; } - } else if (vmemmap_check_pmd(pmd, node, addr, next)) + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + if (WARN_ON_ONCE(section_vmemmap_optimizable(ms))) + return -EOPNOTSUPP; continue; + } if (vmemmap_populate_basepages(addr, next, node, altmap)) return -ENOMEM; } diff --git a/mm/sparse.c b/mm/sparse.c index c84b4c7b8c70..e6cb67ca9c8d 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -305,8 +305,8 @@ static void __init sparse_init_nid(int nid, unsigned lo= ng pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct pa= ge), - PAGE_SIZE)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); sparse_init_early_section(nid, map, pnum, 0); } } diff --git a/mm/sparse.h b/mm/sparse.h index 563fc1f4d717..74dd79c51e75 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -111,8 +111,15 @@ static inline void sparse_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else static inline void sparse_init_subsection_map(void) {} +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); +} #endif /* CONFIG_SPARSEMEM_VMEMMAP */ =20 #endif /* __MM_SPARSE_H */ --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D49BE3C5DB6 for ; Thu, 10 Sep 2026 06:34:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022046; cv=none; b=FAFC9IA38TVkjcrShXJIbLwdovliF4d1MLI90oZ2LGGh/cD6WLY8/PhZ+Xyl0UHJXibYiPIqIb8JQrKM6AlsEs01r3u25Fm+SImWIByEaxOWoklRMmDWAIfiYwezi9SifKolDSjjAfLHMdT667+Lo1nJfkxD2j/BwWBCaeeQyro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022046; c=relaxed/simple; bh=GRsV9hB2Anw7rd7yPDrYvEG4Zy5T32i/Dx5yOBTdbQg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=grwX0vhU38TrxakkBDh8GXrkc0FMKshbkzOQHmQQv637w7JiFjdOc8ngMRJyzA+rBg9qqHByQuhVBHrNIjg34R7T6eooaNaZ7qUYz/O0s818Sdl5Joozu87LGcQEyhqnY27wId4v5dZrf5KmQqba1MhN3i5QXLc+9AOq0Qlk/CQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=SMt+PNkV; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="SMt+PNkV" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-3856d6fbcb3so5658057a91.2 for ; Wed, 09 Sep 2026 23:33:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022037; x=1789626837; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BsqJXB+7FTTz0HgjrZ7vjc10S/x3pwFsj1u1bd3Xfng=; b=SMt+PNkVL6dehtXWCQnEghv3vwihiTw37Pl2zMBgG9ClHE0tU6fMS9Wr9K+7XkfLeo q/r727qnvLrQF7lOsX9p532ON7Z5x6fDZgrjeNwWyySqLQlx/K3j+Bw0t90iaUlb07vk 1x4lblovCod5p3XYzy398wn0jgreiov96EQ4D6zb80oh+7Vmm+viAEiyPY9yl1iU0jgl ablodC7RnSk6X/M3YQj1iooNOeCYnLBVnWY6O9lV7AMWkyUXtW1VD4IepfnCm0I43WUx hXKBZOh0cvsAvvOfF8xIhiguISIJeYDqz22Rt3bAz2KfLYHpFOJmUaaNDgECMRqqs1gd Q9aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022037; x=1789626837; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BsqJXB+7FTTz0HgjrZ7vjc10S/x3pwFsj1u1bd3Xfng=; b=KPqkIfPdoYsOFPg4Wz1rJm52bAbl2jg1XVTv7TA7+R0odrzNmMdl4uuVwa7NcCPlFd 0N7p9/T7LzJnP09SmmcZ/jMoRGhZ3bhJ6CkafXIdxWi3BstsAnT2qsqJRC/+rTJI5Nuq dPkRgwFu/ZmpgQ/qpyLiG+JvCibHFsC0oaYEhNzRxgAAWHDAxLoShFeEfwhW3K+76x01 QNs3yZxZ2mDhUcqWyA5Fv5AVI+jeRGoIB19dzepcuW9PATlMfF0HvxtM8cm20r4nvOp6 dGEvfCvT4GUGYpyw3l+OtUCn2bUU90gBeYGISWWpuROKF+f+9ZQ/wLoX6h+8Lf759VmP Amhg== X-Forwarded-Encrypted: i=1; AKwUvBwTgI43GoEd5RHp72ZRMvGlfypGVHN2fy3DQEMzPm/MTPtd2hBfREQq+6zRD1RNrsbQNI4alLby8evOQ2s=@vger.kernel.org X-Gm-Message-State: AFuF++n40/RYyLBB54RXAB3Pn430pcuvnRW7D4IjwdFa1KNqnhnaWlGk Xv9SAA/x2Kynje6fiw60Vm1HMbm2RutCtDsj7hu6oMcWHHsjxxD8zMQ2XcXvmT8ycnk= X-Gm-Gg: AYBFou2PNARW641L6jIwTm0CsyWUEfEJu7vDF4wjUwsbmnxHxZiJddnIUQkl4j4zZ5k 8pFDD3ccGB/2gxxcZ7UKcWbfyYTr6NAa96vlzcr3jZg6CMIfCBE3tumOYTnus69P/3vlE6Iib0A 3Xf7OjNLfVadBYZEpQx8O+W68BdYdCmlJKokbcAM3aUsH/fvu+2pGa8V8qXyolRM8gXDv8elp8b t1LRcxadR9zbyrfO3qPCRS8swu5d2/2pyw8klfIgtPpJz9I0mTsDN8T+j49KsgDl025kyjuFTzz qVcWvOb9XGLCOxWZL+/jiNX2n2TZu6FMcJXdLS8RvccBZ08QT0I7TlmDD0OW7jTBBHE9LN1BbsP rIXzAYpstgUBVqaCPm/uKpDvkje5pkAnbrrnntE/uSstK7WxMVaMj+k8E2FmQ9tE2kDoEKyjHLn ZhVKrvJMdb895L808XUgCSFY7+BCc0oXBX2lh4ZPojcBw2ES22T/4pKB0bCfdwTqFgCMEFLpf5g vBK5ghfvqS7Z7O9uPBspDUr8Y0cTODhw8A= X-Received: by 2002:a17:90a:616:b0:39d:8222:6436 with SMTP id 98e67ed59e1d1-39d82226893mr1803358a91.7.1789022036378; Wed, 09 Sep 2026 23:33:56 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:33:55 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 09/17] mm/sparse: initialize memory sections earlier Date: Thu, 10 Sep 2026 14:32:48 +0800 Message-ID: <20260910063256.64386-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Upcoming HugeTLB bootmem changes need sparsemem section metadata before the HugeTLB bootmem allocation path runs. The memory sections are initialized from sparse_init(), which is called too late for that setup. Move the code that initializes sparsemem section metadata for memblock ranges into mm_core_init_early(), before free_area_init() and the HugeTLB bootmem setup. Rename the helper to sparse_sections_init() so the new caller describes the sparsemem-specific initialization step. This is a preparatory change. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Drop the inaccurate sparse_sections_init() comment (suggested by David Hildenbrand) - Collect Acked-by from Qi Zheng v5: - Collect Reviewed-by from Mike Rapoport v2: - Rename the helper to sparse_sections_init() to describe the section metadata initialization (suggested by Mike Rapoport) - Fix the !SPARSEMEM stub name so SPARSEMEM=3Dn builds compile (reported by Sashiko) --- mm/mm_init.c | 1 + mm/sparse.c | 9 +-------- mm/sparse.h | 2 ++ 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index ee3bd7921768..304c88da5cce 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2642,6 +2642,7 @@ void __init mm_core_init_early(void) { kho_memory_init_early(); =20 + sparse_sections_init(); free_area_init(); =20 hugetlb_cma_reserve(); diff --git a/mm/sparse.c b/mm/sparse.c index e6cb67ca9c8d..428d81838ece 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -191,12 +191,7 @@ static void __init memory_present(int nid, unsigned lo= ng start, unsigned long en } } =20 -/* - * Mark all memblocks as present using memory_present(). - * This is a convenience function that is useful to mark all of the systems - * memory as present during initialization. - */ -static void __init memblocks_present(void) +void __init sparse_sections_init(void) { unsigned long start, end; int i, nid; @@ -322,8 +317,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count =3D 1; int nid_begin; =20 - memblocks_present(); - if (compound_info_has_mask()) { VM_WARN_ON_ONCE(!IS_ALIGNED((unsigned long) pfn_to_page(0), MAX_FOLIO_VMEMMAP_ALIGN)); diff --git a/mm/sparse.h b/mm/sparse.h index 74dd79c51e75..e998347867f4 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -59,6 +59,7 @@ static inline bool vmemmap_optimizable_order(unsigned int= order) */ #ifdef CONFIG_SPARSEMEM void sparse_init(void); +void sparse_sections_init(void); int sparse_index_init(unsigned long section_nr, int nid); =20 static inline void sparse_init_one_section(struct mem_section *ms, @@ -104,6 +105,7 @@ static inline bool section_vmemmap_optimizable(const st= ruct mem_section *ms) } #else static inline void sparse_init(void) {} +static inline void sparse_sections_init(void) {} #endif /* CONFIG_SPARSEMEM */ =20 /* --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CD8C3C9ED9 for ; Thu, 10 Sep 2026 06:34:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022048; cv=none; b=hfNJclz8cmrYjlkSWACdXGf6GBUIMDSfUrldSCiq/NLf/2O/xpomxMNtElGpQj/XeBnCW8wx9Jehnl1d1HTQT9I/gQrcG+WbK2YJSYvuytXQN1c8XBXhltEsTd+9Tf5G8AtqKL7JM4v7T2Sg8c32HQQp9aayaGk1kHGsucHClLM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022048; c=relaxed/simple; bh=Jc68frRz3TYMxe6K16Aer3C6ZYdyY8LvrpUzosBF3U8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z71Ijec9e0/jixYz1LUnTkg0KpVlIbOy4UwBbjtZvHaBvgojc9TP7o1deYe9gEM/LDykueJaczl0aerClsQZ1FIbnJC3OIUu6YyPhs+X7YD1UdvCtVzpG65F/vIshPNJCypHgpqnNO2bYpdfAtrBYy6Wq+ZwftS8JtI61RLhk+Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dCz/ljGx; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dCz/ljGx" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-398b3c37877so6295787a91.0 for ; Wed, 09 Sep 2026 23:34:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022041; x=1789626841; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bZQQoff6/2tRsBf1VWP2lVrpNpayqt4CNshAPxaCdlQ=; b=dCz/ljGxlrSICkr8af7RU0mYfpkUV7t1e8G19XBtPxYrLKcaG3+Mv7jAqrsn1AG+Sl uD3nXZTY38FwLTU61cHEY/bJT/lPegfGdq9CAVj42Q+urL1dnZ3YB9kulEFMeOY0DkOM 8qKQs56bfXVRbHGwhptIWgOO3OOymahd4X7bGaUeTwh5G1h1Wj2Lhf4BkNLL2fJ4IEzt ffGixID8IdbjVOaRd2sYoyuyt2+xPD54+welAVwFxbISeeuNknSgGOflkiiwohzxIoJ5 28QQt6u3NkD+7hY3b1yry+Py0Fe1+lE+/aQHHkgScOUcWRy1Sx+phD2/wzsqZLo8x95u PZ5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022041; x=1789626841; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=bZQQoff6/2tRsBf1VWP2lVrpNpayqt4CNshAPxaCdlQ=; b=hTtV8KdVL8Zd9xnAco3FT7GZ2/cXBi4akU6PEgPhUGfuRR1qodDqKWoVdt92vUTIo1 TAKykPMsYexdtWxLrMd9Dz1olz8gBPDvbH9C60cwJMrnOqBqkKBIhdydcjK/bPmzPy8I 2q3jgsP5FaZsXELA10xgJyQVo8ypUpFy0S4f5ARbwVlQz680uimJezWcMbJ4MCuATviu E2GSt3EKaSzUoT+9tHAN+oj3DSSxx7HnfC8sJ4Jz9XABVRfkMU+V6ou9RxEEFo7fNz4o 4FGKKPz82rc5hSsr5SyFxI4vx58+SZPOXew/7NZl1zUzxQsGtChspi5SgBmXitASL09X WY/Q== X-Forwarded-Encrypted: i=1; AKwUvBxvp3OO7L4Da6wEqIN8qABy1nv1G5VMrMLsd8rVFSJkG33ulQxQfz5hi+lEkiReGMjmHD8DxFuDfhZdyYc=@vger.kernel.org X-Gm-Message-State: AFuF++mmCH22CzDmSL/1SMFidTNhWs+gB1+UWpB56/lr/C++o02uVG+Q kriHo80d3cu1TJ7j9BSTlWXhhophdYXTtf2vNaXijhLtGg8iILHhFV7LJSZkTNBkbgIxbvcFmyQ QtR48 X-Gm-Gg: AYBFou156SjVBhW1bm3nfqRqDp5f634N7df/zmL/lbCFXNQmo+a2UDpqFiB5cYAkS3F txT7h8HrdeG6VdoMAnXTQM47G61VGsZmGQP2px0rfZCo+wWcEHoQTiEliEzX4izvapYMGaFqGhI /MtvvfWvflLkAIZtdXUt0AO+hr+9REBtQzmLkXMq2nookPGBMtlS8P3pOpwjpmkPGBSrpGZjL0k UI40DIbDXqmMQIA/7YxllvWY9SxL8r385xD/rMO1Jqf+ROxKgUVO+6am4y9iQeqO2OQsARzxP5U W6uhuTmhOEx6kv/Ylaxh+1Y4tyQcyl7CIRG0Zz012zCZj3JycHOVM2FrWU+TLFso+CZpsUX0bEX U581p7aPMQ2FDUqDWGf/jFN1HyeLsO4x1NxBWQCZcbyk0kBx/tmo4X7uxM0TnToxUqVLMoy9HdK GQh5FQD6R1wXk/GuM9rW0zd+2qF9LcVEt7GIWudSeT9u10y795p9Lln3U2ltIiFSiWP+p95zbZX TC+qrSzoTOWudiUw/LuEKTbR0LF4vo2FSkUpFHR8W21Fw== X-Received: by 2002:a17:90b:5844:b0:398:dc16:3c29 with SMTP id 98e67ed59e1d1-39b26101530mr61206968a91.7.1789022040618; Wed, 09 Sep 2026 23:34:00 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.33.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:00 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 10/17] mm/hugetlb: switch HugeTLB to section-based vmemmap optimization Date: Thu, 10 Sep 2026 14:32:49 +0800 Message-ID: <20260910063256.64386-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB bootmem vmemmap optimization still carries its own early setup path, including pre-populating optimized mappings before the generic sparse-vmemmap code runs. Now that the section-based vmemmap optimization can derive HugeTLB vmemmap deduplication from section metadata, HugeTLB only needs to mark the bootmem huge page range with the appropriate order. The generic sparse-vmemmap population path can then allocate and map the shared tail vmemmap pages without any HugeTLB-specific early population code. Do that by recording the compound page order when a bootmem huge page is allocated and dropping the dedicated pre-HVO helpers and related special-casing. This removes duplicate early setup logic and switches HugeTLB to the section-based vmemmap optimization path. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Rename section order setters for compound page order (suggested by David Hildenbrand) - Collect Acked-by from Qi Zheng v3: - Use the order-based helper for the bootmem vmemmap-optimized check v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 1 - include/linux/mm.h | 3 -- mm/hugetlb.c | 31 +++----------- mm/hugetlb_vmemmap.c | 91 ++++------------------------------------- mm/hugetlb_vmemmap.h | 14 +++---- mm/sparse-vmemmap.c | 31 -------------- mm/sparse.h | 30 ++++++++++++++ 7 files changed, 47 insertions(+), 154 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 16c4c4caa126..fe28f98e1b22 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -171,7 +171,6 @@ struct address_space *hugetlb_folio_mapping_lock_write(= struct folio *folio); =20 extern int movable_gigantic_pages __read_mostly; extern int sysctl_hugetlb_shm_group __read_mostly; -extern struct list_head huge_boot_pages[MAX_NUMNODES]; =20 void hugetlb_bootmem_struct_page_init(void); void hugetlb_bootmem_alloc(void); diff --git a/include/linux/mm.h b/include/linux/mm.h index dd09c438fa23..441bd39eab73 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5159,9 +5159,6 @@ int vmemmap_populate_hugepages(unsigned long start, u= nsigned long end, int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); -int vmemmap_populate_hvo(unsigned long start, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize); void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int nod= e, unsigned long headsize); void vmemmap_populate_print_last(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 003cfac98f14..ee5567fcdbe9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -52,6 +52,7 @@ #include "hugetlb_cma.h" #include "hugetlb_internal.h" #include "mm_init.h" +#include "sparse.h" #include =20 int hugetlb_max_hstate __read_mostly; @@ -59,7 +60,7 @@ unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; =20 __initdata nodemask_t hugetlb_bootmem_nodes; -__initdata struct list_head huge_boot_pages[MAX_NUMNODES]; +static struct list_head huge_boot_pages[MAX_NUMNODES] __initdata; =20 /* * Due to ordering constraints across the init code for various @@ -3161,6 +3162,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) } else { list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |=3D HUGE_BOOTMEM_ZONES_VALID; + hugetlb_vmemmap_optimize_bootmem_page(m); /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB @@ -3321,6 +3323,8 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) * this folio. */ folio_set_hugetlb_vmemmap_optimized(folio); + section_set_compound_order_range(folio_pfn(folio), + folio_nr_pages(folio), 0); =20 if (hugetlb_bootmem_page_earlycma(m)) folio_set_hugetlb_cma(folio); @@ -3364,31 +3368,6 @@ void __init hugetlb_bootmem_struct_page_init(void) .max_threads =3D num_node_state(N_MEMORY), .numa_aware =3D true, }; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct zone *zone; - - for_each_zone(zone) { - for (int i =3D 0; i < VMEMMAP_OPTIMIZATION_NR_ORDERS; i++) { - struct page *tail, *p; - unsigned int order; - - tail =3D zone->vmemmap_tails[i]; - if (!tail) - continue; - - order =3D i + VMEMMAP_OPTIMIZATION_MIN_ORDER; - p =3D page_to_virt(tail); - /* - * prep_and_add_bootmem_folios() can access pageblock - * flags on bootmem HugeTLB pages, so initialize the - * shared tail struct pages here before bootmem folios - * start using them. - */ - for (int j =3D 0; j < PAGE_SIZE / sizeof(struct page); j++) - init_compound_tail(p + j, NULL, order, zone); - } - } -#endif =20 padata_do_multithreaded(&job); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c48fcea076a5..d1b031dcd177 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -18,8 +18,7 @@ =20 #include #include "hugetlb_vmemmap.h" -#include "internal.h" -#include "mm_init.h" +#include "sparse.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -706,95 +705,19 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct h= state *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - -/* Return true of a bootmem allocated HugeTLB page should be pre-HVO-ed */ -static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page = *m) +void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page= *m) { - unsigned long section_size, psize, pmd_vmemmap_size; - phys_addr_t paddr; - - if (!READ_ONCE(vmemmap_optimize_enabled)) - return false; - - if (!hugetlb_vmemmap_optimizable(m->hstate)) - return false; - - psize =3D huge_page_size(m->hstate); - paddr =3D virt_to_phys(m); - - /* - * Pre-HVO only works if the bootmem huge page - * is aligned to the section size. - */ - section_size =3D (1UL << PA_SECTION_SHIFT); - if (!IS_ALIGNED(paddr, section_size) || - !IS_ALIGNED(psize, section_size)) - return false; - - /* - * The pre-HVO code does not deal with splitting PMDS, - * so the bootmem page must be aligned to the number - * of base pages that can be mapped with one vmemmap PMD. - */ - pmd_vmemmap_size =3D (PMD_SIZE / (sizeof(struct page))) << PAGE_SHIFT; - if (!IS_ALIGNED(paddr, pmd_vmemmap_size) || - !IS_ALIGNED(psize, pmd_vmemmap_size)) - return false; - - return true; -} - -/* - * Initialize memmap section for a gigantic page, HVO-style. - */ -void __init hugetlb_vmemmap_init_early(int nid) -{ - unsigned long psize, paddr, section_size; - unsigned long ns, i, pnum, pfn, nr_pages; - unsigned long start, end; - struct huge_bootmem_page *m =3D NULL; - void *map; + struct hstate *h =3D m->hstate; + unsigned long pfn =3D PHYS_PFN(__pa(m)); =20 if (!READ_ONCE(vmemmap_optimize_enabled)) return; =20 - section_size =3D (1UL << PA_SECTION_SHIFT); - - list_for_each_entry(m, &huge_boot_pages[nid], list) { - struct zone *zone; - - if (!vmemmap_should_optimize_bootmem_page(m)) - continue; - - nr_pages =3D pages_per_huge_page(m->hstate); - psize =3D nr_pages << PAGE_SHIFT; - paddr =3D virt_to_phys(m); - pfn =3D PHYS_PFN(paddr); - map =3D pfn_to_page(pfn); - start =3D (unsigned long)map; - end =3D start + hugetlb_vmemmap_size(m->hstate); - zone =3D pfn_to_zone(pfn, nid); - - if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), - zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) - panic("Failed to allocate memmap for HugeTLB page\n"); - memmap_boot_pages_add(DIV_ROUND_UP(HUGETLB_VMEMMAP_RESERVE_SIZE, PAGE_SI= ZE)); - - pnum =3D pfn_to_section_nr(pfn); - ns =3D psize / section_size; - - for (i =3D 0; i < ns; i++) { - sparse_init_early_section(nid, map, pnum, - SECTION_IS_VMEMMAP_PREINIT); - map +=3D section_map_size(); - pnum++; - } - + section_set_compound_order_range(pfn, pages_per_huge_page(h), + huge_page_order(h)); + if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn))) m->flags |=3D HUGE_BOOTMEM_HVO; - } } -#endif =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 7ac49c52457d..20eb03df542a 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,8 +9,7 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include -#include -#include +#include "internal.h" =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -27,10 +26,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate = *h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *= folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list= _head *folio_list); -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -void hugetlb_vmemmap_init_early(int nid); -#endif - +void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); =20 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -76,13 +72,13 @@ static inline void hugetlb_vmemmap_optimize_bootmem_fol= ios(struct hstate *h, { } =20 -static inline void hugetlb_vmemmap_init_early(int nid) +static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct h= state *h) { + return 0; } =20 -static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct h= state *h) +static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootm= em_page *m) { - return 0; } #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index faebf344cdac..8c2b9bed236d 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -32,8 +32,6 @@ #include #include =20 -#include "hugetlb_vmemmap.h" - /* * Flags for vmemmap_populate_range and friends. */ @@ -404,34 +402,6 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigne= d long end, } } =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize) -{ - unsigned long maddr; - struct page *tail; - pte_t *pte; - int node =3D zone_to_nid(zone); - - tail =3D vmemmap_get_tail(order, zone); - if (!tail) - return -ENOMEM; - - for (maddr =3D addr; maddr < addr + headsize; maddr +=3D PAGE_SIZE) { - pte =3D vmemmap_populate_address(maddr, node, NULL, -1, 0); - if (!pte) - return -ENOMEM; - } - - /* - * Reuse the last page struct page mapped above for the rest. - */ - return vmemmap_populate_range(maddr, end, node, NULL, - page_to_pfn(tail), 0); -} -#endif - void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, unsigned long addr, unsigned long next) { @@ -634,7 +604,6 @@ struct page * __meminit __populate_section_memmap(unsig= ned long pfn, */ void __init sparse_vmemmap_init_nid_early(int nid) { - hugetlb_vmemmap_init_early(nid); } #endif =20 diff --git a/mm/sparse.h b/mm/sparse.h index e998347867f4..d3a71ef4fad0 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -16,6 +16,26 @@ static inline unsigned int section_compound_order(const = struct mem_section *sect return section->compound_page_order; } =20 +static inline void section_set_compound_order(struct mem_section *section, + unsigned int order) +{ + VM_WARN_ON(section_compound_order(section) && order && + section_compound_order(section) !=3D order); + section->compound_page_order =3D order; +} + +static inline void section_set_compound_order_range(unsigned long pfn, + unsigned long nr_pages, unsigned int order) +{ + unsigned long section_nr =3D pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_compound_order(__nr_to_section(section_nr + i), order); +} + static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) { return section_compound_order(__pfn_to_section(pfn)); @@ -26,6 +46,16 @@ static inline unsigned int section_compound_order(const = struct mem_section *sect return 0; } =20 +static inline void section_set_compound_order(struct mem_section *section, + unsigned int order) +{ +} + +static inline void section_set_compound_order_range(unsigned long pfn, + unsigned long nr_pages, unsigned int order) +{ +} + static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) { return 0; --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A81D62F9D98 for ; Thu, 10 Sep 2026 06:34:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022050; cv=none; b=KYzv/IJopFLL9JeZoRx5z/ED6cmMJoYa1lNPeql2xKgq2hcIsTHYxN2LTyAYVTPc7o5nj+xPf43WYW4XGE0HLCVBiJHt3HqgOCdfj/GDhh7d8U5hEwxH2I0meY6kIOiKG11uDIBP7bIg6V1V491K4213rJLKoFFqveJLgwcYXlM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022050; c=relaxed/simple; bh=U/k/rld+9Eg8VGuMNjqIp0CcDPcMAkDReLdXY1Cw5fw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UO4lfQXBEi/VLeIxW6t4VPr8Gc04jIECNIW/3ACXtp9QHPptz+G1gh+mtfXznTYXLXBrJDI2RChBx/twpwzp10NJXfuiLSiosfjqyuLI8WuQYuiKZNQjUb/XM91F6ulIVTDRRaef52DVfa6E3Qz6va6ICtKf8/CvBy8OOCpP50U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IYHO6jxT; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IYHO6jxT" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2cc891373e0so70418435ad.2 for ; Wed, 09 Sep 2026 23:34:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022046; x=1789626846; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=b0F4KwhTuI2kCSSXKNIULS4SoWuNBUuvow/5FgUy1rc=; b=IYHO6jxTpaI9/C8MZO/n/+N9DUq78JIyGZksBA+xqSwNGf4OLHJiGpoFmToPTsl4D1 1t/R+Y33LeOxw5894WUZbxW3FjQvscIe4Nl8QPpOEer8sSSscHg3/LoYDvE6/A2rZ3MJ WrVsWQ+3AKsoLA0yNQw1s6xGrCo63hnaBunI9rphkK9VPcYhFt8GQgG+yLTm97oJgxbL EXD295zYqVjArsSHAWlXXheT86lVZvD4eYqBkwrObyuBFIKmrIcjKmdJDUS29IoxC0ac vMffPjuiKeEom0jwZUvs2xpIiDhWqM1yrtJj4AyfO1/EgzuSxo+ecbPG3hJ/NyHna15b +vZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022046; x=1789626846; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=b0F4KwhTuI2kCSSXKNIULS4SoWuNBUuvow/5FgUy1rc=; b=R7SbUJ6Gjpz7kraXQWUCnkakMA60tKy/2nBtb94iB2/gghNMsBJ9+9kOgCbcZSTFQK rLmF/8KVdhqtdBbZmIF6GcxOCplhEp7Hagn4BUs1/e+Z3T/rfdSspkbcfYJdrfAFleGf MhvT0kKZ+cD61/f18ibNjk28U2RpBSHjS2fEobi+x65G65qYLSIIKClINOtxGYMevlVl zO5+D3DQ3zozErzgfG80YJ+a8vO9yJGVHXOAL66N97ZJLFD1gHme4hKOQYnNtnfuFf8X el9MHwbQj4lthrUmhMbicxGSuYJOKqpgK0gAA/Bko4k6mYtri4zbi0v8FNDDBiLCRmKu 3NTg== X-Forwarded-Encrypted: i=1; AKwUvBzvLVAnRi2rEBNLqgQf+ebWOG3t0SwOsyAobGCVk5DMAzvsZY0whQnK36ezkMHX609psW28PSh3g/9KoU4=@vger.kernel.org X-Gm-Message-State: AFuF++mMznBlC0RBpNyuNigrW6VMtyWcu9mxlDfCirXd5dNwrFV9T2EX nm97oJR+wg4XiZnmMAJMDK4uojmNbWjr7e6+00kedKRk6WriMpeOASnjc5obUw7BHr0= X-Gm-Gg: AYBFou3vzPC1ZqItTtzVNay4EPkYt7tQv9I7bMOsJVrSazsxh3bpB1MUa+3Wv/Qt9m6 LvL78rFx6v1IfPcM8JPlWEe7jFPeKBxefI1Ub1GPHA4ZNoviyK1c8RDXNDkckyKZileQujKvuZ0 4sn5PO0pIdcnEOX0+ikPHLMWoa22L8SDBUNZtSpk+JHsisPvgerKSQNCfG8tNL5+4sYbpcVBvbg +UcGHHa/xtnU2bSOztlxQrYry43Bf1IIu6imvfOcUxxB2cdmPgtq3+llxpnrbLKIZ1IkUuBh4Ri ZUTl+lxAdd+jXpWabOntZfk0tVNvB25JJ0VF5QzLUTqAi0+RQxbe8mmS0yDtTMpmzI0N0X6eUOF AAbM8ezeNz0PozkDs/9IcRZVhH8Moi5CWBi0W+MXzU70H/arCnTWaWbEbr+6yi+iwFiki1Q+hdL FwRaej/QYwTu30v/uYKL1/gjDar/s0iXDSGIoP+jwffUc7s3CI86Jo/BS8BMOVGIxNpPNSCeqpc wUAzB1scm7fX1MpnEmItw1UI1IM7ihkfvw= X-Received: by 2002:a17:90b:3c45:b0:398:9beb:5c19 with SMTP id 98e67ed59e1d1-39b26232bb1mr58422244a91.20.1789022045807; Wed, 09 Sep 2026 23:34:05 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.34.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:04 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 11/17] mm/sparse-vmemmap: remove SPARSEMEM_VMEMMAP_PREINIT support Date: Thu, 10 Sep 2026 14:32:50 +0800 Message-ID: <20260910063256.64386-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" SPARSEMEM_VMEMMAP_PREINIT existed only to support HugeTLB's early vmemmap optimization setup. Now that HugeTLB bootmem vmemmap optimization uses the common section-based sparse-vmemmap path, the sparse initialization code no longer needs a separate pre-initialization mechanism for vmemmap population. Remove the related Kconfig symbols, section flag, and empty early hook, so present sections always go through the normal sparse setup path. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng Acked-by: David Hildenbrand (Arm) --- v6: - Collect Acked-by from Qi Zheng and David Hildenbrand v2: - Collect Acked-by from Mike Rapoport --- arch/x86/Kconfig | 1 - fs/Kconfig | 1 - include/linux/mmzone.h | 25 ------------------------- mm/Kconfig | 5 ----- mm/sparse-vmemmap.c | 13 ------------- mm/sparse.c | 23 ++++++++--------------- 6 files changed, 8 insertions(+), 60 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 15fd9ec5ecac..7aa74bcc72f9 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -150,7 +150,6 @@ config X86 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 - select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select ARCH_WANT_IRQS_OFF_ACTIVATE_MM diff --git a/fs/Kconfig b/fs/Kconfig index e05917adcd60..d1c210c6508f 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,7 +278,6 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP - select SPARSEMEM_VMEMMAP_PREINIT if ARCH_WANT_HUGETLB_VMEMMAP_PREINIT =20 config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 2c4f63e379da..f2d39a888eac 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2095,9 +2095,6 @@ enum { SECTION_IS_EARLY_BIT, #ifdef CONFIG_ZONE_DEVICE SECTION_TAINT_ZONE_DEVICE_BIT, -#endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - SECTION_IS_VMEMMAP_PREINIT_BIT, #endif SECTION_MAP_LAST_BIT, }; @@ -2109,9 +2106,6 @@ enum { #ifdef CONFIG_ZONE_DEVICE #define SECTION_TAINT_ZONE_DEVICE BIT(SECTION_TAINT_ZONE_DEVICE_BIT) #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -#define SECTION_IS_VMEMMAP_PREINIT BIT(SECTION_IS_VMEMMAP_PREINIT_BIT) -#endif #define SECTION_MAP_MASK (~(BIT(SECTION_MAP_LAST_BIT) - 1)) #define SECTION_NID_SHIFT SECTION_MAP_LAST_BIT =20 @@ -2166,24 +2160,6 @@ static inline int online_device_section(const struct= mem_section *section) } #endif =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -static inline int preinited_vmemmap_section(const struct mem_section *sect= ion) -{ - return (section && - (section->section_mem_map & SECTION_IS_VMEMMAP_PREINIT)); -} - -void sparse_vmemmap_init_nid_early(int nid); -#else -static inline int preinited_vmemmap_section(const struct mem_section *sect= ion) -{ - return 0; -} -static inline void sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static inline int online_section_nr(unsigned long nr) { return online_section(__nr_to_section(nr)); @@ -2385,7 +2361,6 @@ static inline unsigned long next_present_section_nr(u= nsigned long section_nr) #endif =20 #else -#define sparse_vmemmap_init_nid_early(_nid) do {} while (0) #define pfn_in_present_section pfn_valid #endif /* CONFIG_SPARSEMEM */ =20 diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..2c385f8b2944 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -461,8 +461,6 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. =20 -config SPARSEMEM_VMEMMAP_PREINIT - bool # # Select this config option from the architecture Kconfig, if it is prefer= red # to enable the feature of HugeTLB/dev_dax vmemmap optimization. @@ -473,9 +471,6 @@ config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool =20 -config ARCH_WANT_HUGETLB_VMEMMAP_PREINIT - bool - config HAVE_MEMBLOCK_PHYS_MAP bool =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 8c2b9bed236d..f22d815d7af0 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -594,19 +594,6 @@ struct page * __meminit __populate_section_memmap(unsi= gned long pfn, return pfn_to_page(pfn); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -/* - * This is called just before initializing sections for a NUMA node. - * Any special initialization that needs to be done before the - * generic initialization can be done from here. Sections that - * are initialized in hooks called from here will be skipped by - * the generic initialization. - */ -void __init sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static void subsection_mask_set(unsigned long *map, unsigned long pfn, unsigned long nr_pages) { diff --git a/mm/sparse.c b/mm/sparse.c index 428d81838ece..f4f393a033a4 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -283,27 +283,20 @@ static void __init sparse_init_nid(int nid, unsigned = long pnum_begin, if (sparse_usage_init(nid, map_count)) panic("Failed to allocate usemap for node %d\n", nid); =20 - sparse_vmemmap_init_nid_early(nid); - for_each_present_section_nr(pnum_begin, pnum) { - struct mem_section *ms; unsigned long pfn =3D section_nr_to_pfn(pnum); + struct page *map; =20 if (pnum >=3D pnum_end) break; =20 - ms =3D __nr_to_section(pnum); - if (!preinited_vmemmap_section(ms)) { - struct page *map; - - map =3D __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) - panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); - } + map =3D __populate_section_memmap(pfn, PAGES_PER_SECTION, + nid, NULL, NULL); + if (!map) + panic("Failed to allocate memmap for section %lu\n", pnum); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); + sparse_init_early_section(nid, map, pnum, 0); } sparse_usage_fini(); } --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C369F3D331D for ; Thu, 10 Sep 2026 06:34:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022053; cv=none; b=Yo+ALkfYTM4hOfZzfM98E2IGcLC37RrTU6d+/pW65wezeMcbkWbWhmH+EF44qc647w64I46hHflC3fQZJL0MkV5xe2L4QiU5GMPZrUS5sB5rxbBB+019WLvrfeSaJ03NhBxiD8dZxdyl7jVa+8azq/TXuTBwAdLaGX2BhtxIhxI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022053; c=relaxed/simple; bh=Y9vZ+gOhWiHRFF/Y8/hSuUDgngwlFLAbCpMDhndoP8g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PO34BtHNU3z6LCGN+Nkgu/xcUZ3tg4A0XyStQbaYanrXcpPTMmZwVlz+KE+Wjh1VGROQ9IPURcVbHlrgAxHCJ7kaBrzdb27leMeRGyZJpspkMuOcqkRxLOe7NQQbw7KEWYKyDh4sNjl6RBlBArxzo6coEpCwvtI/iHVMbD/yrAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=h3w3/fBF; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="h3w3/fBF" Received: by mail-pj2-f12.google.com with SMTP id 98e67ed59e1d1-39b31b4281eso1074372a91.2 for ; Wed, 09 Sep 2026 23:34:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022050; x=1789626850; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fcVbDCPba2ZtcQZxfLWU2CmbcXM9gR7gcXJIOZ63SMc=; b=h3w3/fBFJnDeAYNrd63odEzxtKG0+DcCB3USnQz5q4CPA5pnZxFS/XJDcom+Htqjhj YvMP/4zvU693l9qp2WxuXYbS0xwHmXWC5oN2gkdXbYo58hXBa3oPL1Wq/162BPIjnaQF 7KPtGvX2iflmZFJzpEWr00tItXWsQ2zEmfPZjf3fInhJYq/1KyKy7vNW7KIYzROwJ9qE tgZbe2i1CdTt1a+/n8FQ0bxqoOBBRpBZjptp2qwV6LV9JXn8IiL6nyDJhfyJ/RFFdhJu 2uNJLVbWK0oyKdiTUv+UbeSqDAn5v8PRMEZDq8xcER+S4gDFN5XCsoFtk1uV+mw+yGFm 1Gvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022050; x=1789626850; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=fcVbDCPba2ZtcQZxfLWU2CmbcXM9gR7gcXJIOZ63SMc=; b=J3gQLEufeKD8bck8oJ94W5r5uktmu/osHrnIMnSLFAJZxQGOI6sa7yxKxbsSGNeooX s+7ReI+OLP4PQcwMj1K3TQ33oV125EY2WjbQ3VUGtSb7av6HaIqXSRXYNdwtfye0Wp6X 4yUYoT1t8U1LeT+Wcf8506ryoo//VuFXnZQAzi18cUKO6dQQGSeceWxUT0gbrG88yU+1 cMnSjmRcqfUyYwTF5rZGqszXVsHkCPjI7b4g+o1/U1a8N7duVc6mH2sxWKJWiXLKH095 8m51RMkV56t+KoyeF6Up3JZDBtVjN9Lr9daPLLzdgDX7DXZKSDF7xJ4E4xsvGCVTDVkF YnwA== X-Forwarded-Encrypted: i=1; AKwUvByhOlHb8gn8mCkrMU8+l+67ytxOLOfVoYEm5xdqBIgmxuFVGClrm5Fv5zEi3Pteu/4Y8B2Sf+/5KUMsE8Q=@vger.kernel.org X-Gm-Message-State: AFuF++l9h15G9RgeON6HV3EJ9a0KPH/AwB2OcurqM8LfvsAwsBQOsVht TsWs5+5HZMl3F5QXzlwABOmSuWqbVr4I1CasP4yXZP2/IxoSZ5DNLVZvocn6T+RVppM= X-Gm-Gg: AYBFou25MCML9mF1TPHKJWWD+bcH4f+YIVbaKPxoP5s1eIevg0RL2nF0csynK0cukbr jHBaop2CfHiMWwi8bhKpdGYWUMixtDCgFZ8FnIT5T6bBVWyourujrOh7xTFeqHqg+tZZLCoBdPX HTHBIOp2mo1pcrcyeiRtcyWxb1asqXs9Y7nLTG+HOgV+/qMoeiGkhYBBuwnrp9wHfB0WSDvIsik pfzcd5L5sk54HMQud7goOMHOgvMyiUte9GqlbCvzlhscBSfK1GL+6EQDT8uRwVrYWOomCfWR+xG kGPb53D7dzK2U0sFK4D9RV4PyxWerEBbPO7FKty2ihjjfuTNV+Qa2Omjz/A8qg9UCd8cK3Zc6eR OA2kGylb12YYGzOrJGeCc3+/CF+YNtANMuGW+79242Q5/C0hujE7PPJY6A+f/rgQja44lbzjcXw rT8nZJh+5AsPf3bPmvA7jNMTmVXPDHPtbWwhRQbcKawXFDhaPhV4G4V0PscHDNPZSWEaV2niUG6 SOiVPpcgu6wP/THZQOvy96iUVopZF5bzfA= X-Received: by 2002:a17:90b:4d82:b0:398:bacb:1137 with SMTP id 98e67ed59e1d1-39d70b1c7demr7779886a91.19.1789022049796; Wed, 09 Sep 2026 23:34:09 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.34.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:09 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 12/17] mm/sparse: inline usemap allocation into sparse_init_nid() Date: Thu, 10 Sep 2026 14:32:51 +0800 Message-ID: <20260910063256.64386-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After removing SPARSEMEM_VMEMMAP_PREINIT, sparse_init_nid() no longer needs the transient sparse_usagebuf state and its helper wrappers. Allocate the usemap buffer directly in sparse_init_nid(), pass it to sparse_init_one_section(), and drop sparse_usage_init(), sparse_usage_fini(), and sparse_init_early_section(). Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) --- v6: - Collect Acked-by from David Hildenbrand v2: - Collect Acked-by from Mike Rapoport --- include/linux/mmzone.h | 3 --- mm/sparse.c | 46 +++++++----------------------------------- 2 files changed, 7 insertions(+), 42 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index f2d39a888eac..6acc14b169bb 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2223,9 +2223,6 @@ static inline bool pfn_section_first_valid(struct mem= _section *ms, unsigned long } #endif =20 -void sparse_init_early_section(int nid, struct page *map, unsigned long pn= um, - unsigned long flags); - #ifndef CONFIG_HAVE_ARCH_PFN_VALID /** * pfn_valid - check if there is a valid memory map entry for a PFN diff --git a/mm/sparse.c b/mm/sparse.c index f4f393a033a4..d19c173b40fe 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -234,42 +234,6 @@ void __weak __meminit vmemmap_populate_print_last(void) { } =20 -static void *sparse_usagebuf __initdata; -static void *sparse_usagebuf_end __initdata; - -/* - * Helper function that is used for generic section initialization, and - * can also be used by any hooks added above. - */ -void __init sparse_init_early_section(int nid, struct page *map, - unsigned long pnum, unsigned long flags) -{ - BUG_ON(!sparse_usagebuf || sparse_usagebuf >=3D sparse_usagebuf_end); - sparse_init_one_section(__nr_to_section(pnum), pnum, map, - sparse_usagebuf, SECTION_IS_EARLY | flags); - sparse_usagebuf =3D (void *)sparse_usagebuf + mem_section_usage_size(); -} - -static int __init sparse_usage_init(int nid, unsigned long map_count) -{ - unsigned long size; - - size =3D mem_section_usage_size() * map_count; - sparse_usagebuf =3D memblock_alloc_node(size, SMP_CACHE_BYTES, nid); - if (!sparse_usagebuf) { - sparse_usagebuf_end =3D NULL; - return -ENOMEM; - } - - sparse_usagebuf_end =3D sparse_usagebuf + size; - return 0; -} - -static void __init sparse_usage_fini(void) -{ - sparse_usagebuf =3D sparse_usagebuf_end =3D NULL; -} - /* * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_= end) * And number of present sections in this node is map_count. @@ -279,8 +243,11 @@ static void __init sparse_init_nid(int nid, unsigned l= ong pnum_begin, unsigned long map_count) { unsigned long pnum; + struct mem_section_usage *usage; =20 - if (sparse_usage_init(nid, map_count)) + usage =3D memblock_alloc_node(map_count * mem_section_usage_size(), + SMP_CACHE_BYTES, nid); + if (!usage) panic("Failed to allocate usemap for node %d\n", nid); =20 for_each_present_section_nr(pnum_begin, pnum) { @@ -296,9 +263,10 @@ static void __init sparse_init_nid(int nid, unsigned l= ong pnum_begin, panic("Failed to allocate memmap for section %lu\n", pnum); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); + sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, + SECTION_IS_EARLY); + usage =3D (void *)usage + mem_section_usage_size(); } - sparse_usage_fini(); } =20 /* --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A71F73D904C for ; Thu, 10 Sep 2026 06:34:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022057; cv=none; b=NgdvY6KFwidxwUN6LjwXjdmFGq+U4lGDiH/vYlYMHXKNW9ViAI4wKIWFMEM4ZBldHa1ZDVyvgLiZfBRk62ETInb4D+Ra+I7CToq5vCOCjim6R99a+iwBdZW8qb9T+kgNyZV7DzxNTCj+j9ko7dZGwy5JTA1qGAMhM2gu7r7bk7k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022057; c=relaxed/simple; bh=J3/NKoHHwAFLAgCGzlMxl8zo8az9ychwjCKQWWmP+TM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BGimSl2UxctkJ9+UBYoURGoyXEkZZkd7bEtbbLWGWJSSfbacxWUFp2lWRU83K8RgayIfu6cHJ2tgWae0Qke5myh1gz3w6S9uBt1hyxjifLE2Pl9FaS/gl+FHgED0XRVyWJkWcz+rB02BD6mMDXT5wBEWvonSVPrY2Kg/zogHBIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QEdQgoqZ; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QEdQgoqZ" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2d942c7cc2fso57509845ad.3 for ; Wed, 09 Sep 2026 23:34:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022055; x=1789626855; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ytFcTktewPTS6VkWuTaJ4fUtmqPUIIlqGvc8eyNaYzM=; b=QEdQgoqZ+1Y9s0PbzZoUjjB4IuW72ME5NFyR9UZlz7Mu9+H7KKm0SUDxkgZnu3v3/r jXwRS0E78rW0K3vF/LKhff7u3pgVcbYWBl1BP3Tm/q/UeyNnvrBz3FR0F2+qsUbIvQvG SkkqPhrcX1qLLSz2+vfOTmtyAxKPl0cfIjftdH30MwjsTCYuRiXsti+fwInrzuigkYDi aTORQu1RZWYG1pjAqsJFazs0yxAo4klvCT5Ktcx0JeNMSrRlw65dlG2OID9LWLuMv30F P41CUaByDBbP+AJGke1GbZJAK2ZfdjNsmtDLGyRto6HDOdN5S2zEzoA9ZyQyX75zLrjS 8Jqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022055; x=1789626855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ytFcTktewPTS6VkWuTaJ4fUtmqPUIIlqGvc8eyNaYzM=; b=VDHEIie4KwwyXG5WmeyFqdwkNn2uXENo1lYNyYCJVIbovGJ6XJBjnF42+mNinuprkW 2jTe/YFhNYtekgNi1MGU+V56VIAkZfpXJW9JxyRhjXzf1QWfSE0oTXbdo7Y4EBFk3+jE D9c96pdoXP8FHQs4RDsaGHX496xpVp3XS9liwDH4armVO2gWR541sRnGYj4NGmfU1Ygk f2Rlwf6UJKGYL4xO2aMsspEyIa3kJ2Vpyy/4KCQ9fb2epVAAzx1YCEBBA029x8OpACjy oWUPpLay79zSkoT59+PjeVFaL4FmvY6Bz56I1WrhmwaGpZZRr36sXHz9xSAnKJK7ODYB ITPw== X-Forwarded-Encrypted: i=1; AKwUvByOKg0bZPz6kdq4kK3I/Ntm0DFXspuHJKipkAsEWPv7JdATuumFtdcKLEWSJvawszh9v1jJFfscyReM3N4=@vger.kernel.org X-Gm-Message-State: AFuF++nLMRINwcmOW2e6Av7t6VFinEzjhR61ZuPU9ZTW2+HfszAJTqV6 fPeGo36b4WU8mVzJxheLv0zDbGH5ZthqYbZ+r39MESXKvEEgW173LOAiKBvvY8CmfOU= X-Gm-Gg: AYBFou3qmoeg55t/oCcNUbSnMCwv2/sXOKghuVOdcYIFuA82grJXErDhPCi+wbe612E fJn0Tsd3CYYLHc4+hX0ppST1aaoarb2k6rHtyivCWqsbXuvB/d4mcqhGvsfYz4/NdB5HiLcO3Vp C/I7dJqHGGTZjCHSRDDqSlxkksw82dpjG//UIQ+zucG0K9TSN7dQY7x+sr8uQNeyFY57aqADMfx MMC/g/BwuejDZq5dxH4Bqg7nONWZeg6X7sIZSyDZW2JL1W8RlpFM5S/zmtgmrbBaIHk+qRlzeSc s7PQ+LGXUaeXJNCGWIhaW1zhAfGmbgOf9QN4BiVLgDVrigXVddfOz6cq2A6EucrB9K4oaSPjZTU yrPOetPSt5z69tFh5pcx7GC/vct8VrJbupqckyuCXZDrcSKbAN4Tl08Xn0NhWlaMbyGzYfrDjqR iY7QSegiOedoSd62qRga//IsKsEIzp2G+lwrF7YboyUvhlVYyMqOaKytE/W2ivC6H4gvnCxncw5 Q5Mje2tsY2amDtZm/DtCqAUtZj3wqOJ+SNqd4GFB6cp/g== X-Received: by 2002:a17:90b:2f46:b0:398:9c39:520f with SMTP id 98e67ed59e1d1-39b261e763dmr61270657a91.15.1789022054775; Wed, 09 Sep 2026 23:34:14 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.34.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:13 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 13/17] mm/sparse: remove section_map_size() Date: Thu, 10 Sep 2026 14:32:52 +0800 Message-ID: <20260910063256.64386-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_map_size() no longer provides any shared logic. After the sparse-vmemmap changes, its only remaining user is the !CONFIG_SPARSEMEM_VMEMMAP path in __populate_section_memmap(), which can compute the size inline with PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION). Remove section_map_size() and inline the remaining calculation. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) --- v6: - Make the local vmemmap allocation size const (suggested by David Hildenbrand) - Collect Acked-by from David Hildenbrand v2: - Collect Acked-by from Mike Rapoport --- include/linux/mm.h | 1 - mm/sparse.c | 15 ++------------- 2 files changed, 2 insertions(+), 14 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 441bd39eab73..b19711b6dbc6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5140,7 +5140,6 @@ static inline void print_vma_addr(char *prefix, unsig= ned long rip) } #endif =20 -unsigned long section_map_size(void); struct page * __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); diff --git a/mm/sparse.c b/mm/sparse.c index d19c173b40fe..cc28bb41fdb1 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -208,23 +208,12 @@ void __init sparse_sections_init(void) memory_present(nid, start, end); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP -unsigned long __init section_map_size(void) -{ - return ALIGN(sizeof(struct page) * PAGES_PER_SECTION, PMD_SIZE); -} - -#else -unsigned long __init section_map_size(void) -{ - return PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION); -} - +#ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - unsigned long size =3D section_map_size(); + const unsigned long size =3D PAGE_ALIGN(sizeof(struct page) * PAGES_PER_S= ECTION); =20 return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), nid, false); } --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 598693D9527 for ; Thu, 10 Sep 2026 06:34:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022061; cv=none; b=Stf3n3jBz5vjZ82RPcrMXuSNfTUjZFTdh00/itA06NJDp33CjA71ZK4iitm7DC1i9QVna0C8XUnPR3w15Zv8wJ3jpxt87318AVzSxXnwrLuu7HFfLUd+u+KDwHQHL42q35eI3G0hphUO98jzqiReVEAFYu98lonJAYOM8q7aj3o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022061; c=relaxed/simple; bh=EluCtYfIeVaNsl2YnpAaUuO+yO4gKROLtqulYwHTUaI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nou69qJgY3GKnvRQ1TDeI7+QPwCM6PLklzu8x1raBgGl++zZMhBhAf138DV7vKqy5zX5WS0gLgqy9NXfazlVOQt3qR8WyW6qj4lPd5Ssib6N4/FzkYQ72CyTboL/p6kkC2FtqDuYvwRlWg35iICqpK5s3xUe4fQU+ZVI+4vFTqg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=UsICULqe; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="UsICULqe" Received: by mail-pj2-f12.google.com with SMTP id 98e67ed59e1d1-396ccafb750so1848200a91.1 for ; Wed, 09 Sep 2026 23:34:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022059; x=1789626859; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=mPTybuwUCG6mUc8qxpUjNdpd18RIivFekac/tDH8nBI=; b=UsICULqe7Q7onUEggBFcIxFgMDnYD3hvqOZMHHXgxoLG5kQCqz8/oi4E6uVrVdjbmI 2rtGNbs5DRQmpXwYp1nyWvwrM9TR4fPJbQtHzl7yVmWX1vtJF7Ot4cyvDUNZDeRffwpr 9xLU/78MUJSYHq0iXj6xEk37zyDaOBXJK+ETv+lmuhx9Ed3xVbjZXR3S9KWRPeL8iAPm ifdHJD/dBZO3w80XU9xnMkpkEAlzYMvgPQXJFvs875uzsIk2WvbPtH3U4to9CF6+ywMk i7Eu2UNpoQ8sCLwqoU3V3mocn/bgy9lwL9nECAEGtOv6f7pvYjthjwXuq1yHgTDWE2gz ce9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022059; x=1789626859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=mPTybuwUCG6mUc8qxpUjNdpd18RIivFekac/tDH8nBI=; b=eAzBWKZuN9dZMfmbfkpSqZQyfoMlV+8FD55tm050v7LF2vt146BMy3MDSNW86BIKwD yU2aigBybU7yKzQSNhdfZyGuqXHkDgIOzxuAljMsyX8iREF+kN4awzWbkMSWIHbc2f0c KJ0hw5+aV9OCF7ijkHxEmaviW87fAvHdLTy41EGGhNixGhrOygPc9m/IjCHQP8VpJcNE M35+L/1Vx3IG8MarJjacwkpOCewcTT0wy4Hj/Dxj9cnoZCgMYVzP1H/Aj4RCk6TRHSdn ZkGGNiP+d6C1HKqqAgsLQ7iimBxlfq/bFsllNvR+sfmi/HXiCGG4WknezvM/IXxJLxKN mICg== X-Forwarded-Encrypted: i=1; AKwUvByNnuAPh6Mzh02croAc0ngqI90Z8z5Rgk+va943OIiDYFj1bhN7stFGtCTzI6YZwgA6rQrKsI6yCS/ln6g=@vger.kernel.org X-Gm-Message-State: AFuF++mT3Fsy7JupYKzX7K2SMajshxVqEDGv53zQg3n2/Ow5OtQoS3mT sOb+gUjWs/icnhTU5n0r2fNs0uyOSO7bkz9BsO/PpiOSIZ1p9+YjmaKI0d+FFsV4vFAhu2i4TIe tD+8i X-Gm-Gg: AYBFou3UsV+0+lmOsLK8Hgk9sI5pRq5a9T/1qkbzWtXS7A00q2sbflebnUyCLbpANph Czem75utasHi+Ig8LSH3lb8cK+GQBwjt7o9OqQIQiW6oimmfIJw/J+cLMzYKnBBUDWvZC0bLCMp ZImmEPy4x9lvUMx39LYIODgAJ28kJboOJ/VR7iqavm/tTLgcw4vtn1PBCnboIOoxB9PMGT5awNc W49HdPuod7VeQhqqm9/CmPKxdCcouadXAFvl1jqswdG3/rhVGvtZ6wnZnc8LdNkeMq/tE2NGT8j 7o9ArqExAicAJg42miHnsV3VZmGUV+qb/3I/38Rn+5QALPVCyKffSAPSpPVIcpbvP9PvbTzrsjh DzLthfBrw55hQtCACJNqazcmRKxCUa41JZ6y89Cwr1LIQVXEc0l/FYEU9N+jfA1pKOv2n8jHTFw OVg9+9dF+6uHBKRp6VU1sBiy9dXYfoZrRCYDEQPYroVKBMBffjLYuH4W5MckH3e9UDqdeKlFh43 HBpc8cID10o535MetR+Yen2Q+qKufmxZto3LAmkf6snew== X-Received: by 2002:a17:90b:1d45:b0:38f:2168:b9cb with SMTP id 98e67ed59e1d1-39bac1d89b4mr15210275a91.9.1789022058554; Wed, 09 Sep 2026 23:34:18 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.34.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:18 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 14/17] mm/hugetlb: remove HUGE_BOOTMEM_HVO Date: Thu, 10 Sep 2026 14:32:53 +0800 Message-ID: <20260910063256.64386-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The HUGE_BOOTMEM_HVO flag tracked whether a bootmem huge page had already gone through the old early vmemmap optimization path. Now that HugeTLB uses section-based vmemmap optimization, that state is already reflected in the compound page order stored in section metadata. Remove HUGE_BOOTMEM_HVO and its helper, and use the section state directly when deciding whether to mark a folio as vmemmap-optimized. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Use the compound page order helper names (suggested by David Hildenbrand) - Collect Acked-by from Qi Zheng v3: - Use the order-based helper for the bootmem vmemmap-optimized check v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 5 ++--- mm/hugetlb.c | 16 +++------------- mm/hugetlb_vmemmap.c | 2 -- 3 files changed, 5 insertions(+), 18 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index fe28f98e1b22..3559041a5a57 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -675,9 +675,8 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; =20 -#define HUGE_BOOTMEM_HVO 0x0001 -#define HUGE_BOOTMEM_ZONES_VALID 0x0002 -#define HUGE_BOOTMEM_CMA 0x0004 +#define HUGE_BOOTMEM_ZONES_VALID BIT(0) +#define HUGE_BOOTMEM_CMA BIT(1) =20 int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ee5567fcdbe9..fbc2ecafef9e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3220,11 +3220,6 @@ static void __init hugetlb_folio_init_vmemmap(struct= folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } =20 -static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) -{ - return m->flags & HUGE_BOOTMEM_HVO; -} - static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page = *m) { return m->flags & HUGE_BOOTMEM_CMA; @@ -3299,6 +3294,7 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) { struct page *page =3D virt_to_page(m); struct folio *folio =3D (void *)page; + const unsigned long pfn =3D folio_pfn(folio); =20 h =3D m->hstate; /* @@ -3316,15 +3312,9 @@ static void __init gather_bootmem_prealloc_node(unsi= gned long nid) HUGETLB_VMEMMAP_RESERVE_PAGES); init_new_hugetlb_folio(folio); =20 - if (hugetlb_bootmem_page_prehvo(m)) - /* - * If pre-HVO was done, just set the - * flag, the HVO code will then skip - * this folio. - */ + if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn))) folio_set_hugetlb_vmemmap_optimized(folio); - section_set_compound_order_range(folio_pfn(folio), - folio_nr_pages(folio), 0); + section_set_compound_order_range(pfn, folio_nr_pages(folio), 0); =20 if (hugetlb_bootmem_page_earlycma(m)) folio_set_hugetlb_cma(folio); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index d1b031dcd177..fba0c5a7d46f 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -715,8 +715,6 @@ void __init hugetlb_vmemmap_optimize_bootmem_page(struc= t huge_bootmem_page *m) =20 section_set_compound_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); - if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn))) - m->flags |=3D HUGE_BOOTMEM_HVO; } =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96CA33C3C14 for ; Thu, 10 Sep 2026 06:34:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022065; cv=none; b=XKUxcmQBDBfhf9tf5qHKCZqj+aiZtZ3oV8q8cvEP4LySesQOAknyRp5Ro6dnh+HbZSdqzMo2Dw19zE8Se8P56ExpmCXNyb//XP0XPmcJdQM38vZ5curEaTUoT3pbuXUZdcqsMxC702vRKbYqPZWyI0bfcboKkdLK5evrOW3MgKU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022065; c=relaxed/simple; bh=2Z6fwz9ROGJGr1Glv8JiwUCw0S2bDYQJkl8yYB6iumM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iBKfnAkSvurLn98bMQLxQqLzFQh2iuUNBJ+eyDGKn7Nh5/4+QlE3IKo5q1ysbxIaTbAKh62X9KwRVZ1HPOPIc89D66F76WHm1UsyqP0ODBag9J9ZsHVqcMjddLpotRKOcWrWz7HN8ePav9jXLx0PBw/DOX4z0JeBuzYb13CzUiY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=D+Bduy8D; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="D+Bduy8D" Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-cc1a4c62804so4851056a12.3 for ; Wed, 09 Sep 2026 23:34:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022063; x=1789626863; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iXG9/IxdCoEJBswfQ8yvwYsfNZpsrq28j+tV3JTmESo=; b=D+Bduy8DbU8YeSzjrIrtMcOOqfk2bkwJYQ9wIrobZdTea0L8ArmJFmALeFl02fY6XN 5GxaWfNILYQbbZSqRfBbnWeQmvTVTXT4rfTOyvxqUmyQbz4sD7zQuwtyTiBAAvJwk0tM +qlUfdsFIMzUNPjNtfGQYMSAj77TzBdTL8bIFwN4EFqOnpjkaAG1plzyAObqBCUDsgvJ rlgLtmnHSOPiR7FwV//F4tjF73gM1mwqo8rygn57nD29+ctToRXEy8WS/o/zlJF5b4PO TMuyJipoRsE2ucaIsvq0ngoq77xYKKidveY2tXgWEnkQw4QXodAtvlinvkerlRgFVVxF QNBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022063; x=1789626863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iXG9/IxdCoEJBswfQ8yvwYsfNZpsrq28j+tV3JTmESo=; b=Z98x4EWPQFwabIoeazE99wlrymrj79BhRfsV/CTEanitbtEpO2jzPxbSe3wfa1vivn Nm/fDHVAEr5X0n1PJs83OV2H9e+qnmHdxKIOsr5c5uy+j/9xiOY3vXRvhK26/MeMneH5 A/kC6LD6Eh2aRwfLZVRqSo0l4F+Z6anm2uYSArSUOvyVGcqbntMZfF4V/HWByP/0ihU8 w3mu+DfUGwLNfIisRcG6bi05xrcdhPhTfAQGN0dkp2NCtH4y8trS/D7MFMnfolxa95D7 xLUs7KxG+rGe1hBdEmPsPkeBnRI+dP2NHMeh1aOYAqA+DTgx7e9QSysLYRtnM/Oj2KJS G+Nw== X-Forwarded-Encrypted: i=1; AKwUvByPsfaMtKlGZbKxMbzcpjOxpMhSTS+ILdkGaK5BzbKIqLvETeIrLGHINTeV0O+dadu3hY/RhGLNx4rMNns=@vger.kernel.org X-Gm-Message-State: AFuF++ku9vbn5X5pFXeGUoYKovDx/GCJxppzaTdAT+GbCCQt/01Fyd1O JbSLyKhB4dW5XCXYBWMbfpFAB0NRVUkZP5+zeLsZs4j59Tl0jUvw1bHG4VGlUNe8hJ8= X-Gm-Gg: AYBFou2NEl6i4WBivZ/aApdQjcoJacDI2KqLSxKvAInKtriQVoNxxKjWecYdSjmLnmb 72cgMyM69TU07eEjkNrbf9YxiZs20UpqFjPYes93jX3RC23R9BLBS1S7ks14rjxxIfiR4SmzlrL D7q5MZNj9PFuXqxqJPSJYtAXZt/ToG0aJIlKnuZkBHzHKNEPtkj9sEkzg/H5QqpNO2qxBu9aXra jKVkdRQ1UwhqRNGF+KOeGVU3u9jDFdZNPiphdVZaZOLNu8VMblmk+v346YmA7ecA+hK88lO/oRs kjKNdyes0rNzMRaFtn86xvvHlTmFrPiHGWtKbPsua0rPbvUnznfdD100wp4DCyDXCSYNH4ENJ7I zxTHz5rxec0DRiphir74mxL7/oQgyUqTTHXVXMa0nnuh4HS0VwHyWG4L/0//BRjtBklY/XzPDGR vbs6IhA48v2HVaRr+0IA1hIsQxxF4Gf3G6p2uRkKDKa+u3yaizIULPn7kSitEV18z55+zWlLdSU 0eX5bYtIBbUhvwjh2BF1P/E2YvLNyN8H18= X-Received: by 2002:a17:90b:3b8d:b0:398:b426:ca4b with SMTP id 98e67ed59e1d1-39b2624f10amr58077953a91.22.1789022062790; Wed, 09 Sep 2026 23:34:22 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.34.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:22 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 15/17] mm/hugetlb: remove HUGE_BOOTMEM_CMA Date: Thu, 10 Sep 2026 14:32:54 +0800 Message-ID: <20260910063256.64386-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Track early CMA hugetlb pages from the hstate instead of storing a redundant bootmem flag. This removes the unused helper and keeps the bootmem metadata minimal. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Collect Acked-by from Qi Zheng v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 1 - mm/hugetlb.c | 14 ++++---------- 2 files changed, 4 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3559041a5a57..255a258f11d1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -676,7 +676,6 @@ struct hstate { }; =20 #define HUGE_BOOTMEM_ZONES_VALID BIT(0) -#define HUGE_BOOTMEM_CMA BIT(1) =20 int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fbc2ecafef9e..f9b609b84509 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3144,7 +3144,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) */ INIT_LIST_HEAD(&m->list); m->hstate =3D h; - m->flags =3D hugetlb_early_cma(h) ? HUGE_BOOTMEM_CMA : 0; + m->flags =3D 0; =20 /* CMA pages: zone-crossing is validated in hugetlb_cma_reserve(). */ if (!hugetlb_early_cma(h) && @@ -3220,11 +3220,6 @@ static void __init hugetlb_folio_init_vmemmap(struct= folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } =20 -static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page = *m) -{ - return m->flags & HUGE_BOOTMEM_CMA; -} - /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABL= E) @@ -3316,9 +3311,6 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) folio_set_hugetlb_vmemmap_optimized(folio); section_set_compound_order_range(pfn, folio_nr_pages(folio), 0); =20 - if (hugetlb_bootmem_page_earlycma(m)) - folio_set_hugetlb_cma(folio); - list_add(&folio->lru, &folio_list); =20 /* @@ -3329,7 +3321,9 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) * For CMA pages, this is done in init_cma_pageblock * (via hugetlb_bootmem_init_migratetype), so skip it here. */ - if (!folio_test_hugetlb_cma(folio)) + if (hugetlb_early_cma(h)) + folio_set_hugetlb_cma(folio); + else adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC6EF3CF69F for ; Thu, 10 Sep 2026 06:34:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022070; cv=none; b=PFFhebrnzGmQSUc3LG4tkKbmDXb60BB8UhZL/rhMg9RDkn2VW/TB/84wWaFNAZV9yih+Qza4KjlTG5Djl+7KzGWS8RSIQ5NBS41bP2hdlZ61uUiB710V0XoBQE+G0olWCp+8alVD0GITQVSYGN9fXysjRBHnSlP9k3xW4DFhRHM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022070; c=relaxed/simple; bh=2G4gUus/H6CO32P9O6vDcNapNZOu15UMTc6GJdqSTEc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XBr9NxGQ65OFV7bvAcosBvvDC3YkJCvTv4O35sRd1QnauxLdodwN4m7/9FoJR8+tQBfdCK2+51hBlYqNUFCj6Ozf9Z5qbBIdVFYHxtEen+VZ76V/TrIzAyM5j6djUJ5us3T2i04R8ubrr7JrbPemYhGrtO4BEqcOZv8R2E2UI7A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=d7VeFncQ; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="d7VeFncQ" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-39675172593so5695052a91.2 for ; Wed, 09 Sep 2026 23:34:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022067; x=1789626867; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/jQmh4S0MKyAUesk/qTiiCNB9x5yb6aCEsrSfgP9kbM=; b=d7VeFncQqyIXg1QlClfSNZhBl9t9SjXAZd9uK/dGa3lxEbyD4NgTSQWHtkRp+FSerP UnTCfRivjGYibTLUBPY5CqDa/p0ELUwW/ssBHp2sfMSqOzlbPeBaXbt0RQj1Nlm/g57m DbkbzyK7GYqSAKgT02CjU/3wH0fJDl4iC/h2Jexew0T0B+IO3PjIkonq/CkhOD8TYt1z bz6FRcRFPxuyDTdMhMyAOmK9JKLZF/CSWpqIXRLsI97RXxRF6aAFVCj9mbM+I9Ptqbmu py0idrUIu3fQFlas4ideAijqQo10zPdpDjSdyS1x+12F99TplawuCeDVnzw0HMy5nCDq DTOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022067; x=1789626867; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/jQmh4S0MKyAUesk/qTiiCNB9x5yb6aCEsrSfgP9kbM=; b=OzcqHmSVLfCL2S+NVnSStA3esz8ZjXKV2KTqjSpOfor1nxm83gyGj8mjFWKbUT3DmM PhS+zGVB+BddFZPO7H048j1WsoqDPWc/tb1Qo0sCUVsKGKPL5keps4GFSZsdSaD5DBLa Xr0A4eEZGMQLSBo4TjuWKk3O1W1LEWFLLq/yAXI8cRz6AQQ+ig6Kt5rlWkd/3Jx0eYM0 34nAlkBPB7aF9Ba3O1Egnlb6kJy+cFjMfYokHZrWTciKnBLALdeei9GMJBnwuluj+CKm C8o6F10uXfQFhDRy/1TZruLPv8Wq0Dyz8qtywkHXMWvdJR3mT5Wmf3+VvfWzUCZ19QrV PJBQ== X-Forwarded-Encrypted: i=1; AKwUvBwfzUS/vg8jITuI5BTOGqG7Q9r53YxM5wo6Bt/CHr6TV/u3BSgsPw8WXarOrglMCZP6J5sFiaF6KczJk1M=@vger.kernel.org X-Gm-Message-State: AFuF++kiq4fA59swfTJonNSOELFWL7OVW+hI01jWpWelqiLZi9tk6BWL RZRWloN0/98d7qw6jMuJtLI3pH4W5WWPeWIjrfyVpeZZ1EPOelm/M9uYg8GkztSAMps= X-Gm-Gg: AYBFou01BVkwzmQgx+SDcK5D/Z+h5BcArVrDDD4d3d3tRoho1Bvkge6aiTCCOjbhgFA vJotOxgNabJKghQvp+f3sp1I3cg5lkwlWjVQT96Oa3WhiseOBEdTZahdHyGSyDOA1Z0At+uvugJ LCdtGm4lPUw9veF7oZayw2hjnacUFA7D2olQJ6d9gfiQDwuDt6a/RzO9xBqeg4LWaNDwG3/ToLu Hrcm0nsQRCKGjuNNHm4QAM7/FRnsrdQF11yGHW+68rdQr7vHzOxw2VJgHieevtDB71czmDAPbew EYLL2quGzoM1jk8IF2Qc4qz8v+LlN496YihbPaxn0w1pHCQr3hzuUTn2j6oMgPPv6FTurmSRyX4 kB/4VUdsCvrdot7WdV5nh0qsda1mh4NINygF+zarHWQ7n6Tu3Oq6IrGB/+1/9BcOPyujRuE9PFO Z3mb//+aD4iWEGcFpuGVAz2tDfqMtvvw2E+0W70qpQCm7Qn1FnEs3AEuc8fYFm/Jp/Wmguld0p6 HBmRPOTHOfTPO/PcxH3lMvGu2Jm/IuSW+O0FRNPZQucwA== X-Received: by 2002:a17:90b:538c:b0:38e:b400:a860 with SMTP id 98e67ed59e1d1-39b26204a3emr62256716a91.13.1789022066635; Wed, 09 Sep 2026 23:34:26 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.34.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:26 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 16/17] mm/hugetlb: localize struct huge_bootmem_page Date: Thu, 10 Sep 2026 14:32:55 +0800 Message-ID: <20260910063256.64386-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct huge_bootmem_page is only used by hugetlb boot-time allocation code, but its definition currently lives in mm/internal.h because hugetlb_vmemmap_optimize_bootmem_page() takes it as an argument. This exposes a hugetlb-specific internal type more broadly than needed. Change hugetlb_vmemmap_optimize_bootmem_page() to take the information it actually needs. With that interface, mm/hugetlb_vmemmap.h no longer needs to include mm/internal.h, and struct huge_bootmem_page can move into mm/hugetlb.c. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Use the compound page order range setter (suggested by David Hildenbrand) - Collect Acked-by from Qi Zheng v2: - Pass PFN and order to hugetlb_vmemmap_optimize_bootmem_page() instead of struct hstate and bootmem metadata to simplify the code further - Collect Acked-by from Mike Rapoport --- mm/hugetlb.c | 8 +++++++- mm/hugetlb_vmemmap.c | 9 +++------ mm/hugetlb_vmemmap.h | 5 ++--- mm/internal.h | 7 ------- 4 files changed, 12 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f9b609b84509..c8fd3a6ce812 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -55,6 +55,12 @@ #include "sparse.h" #include =20 +struct huge_bootmem_page { + struct list_head list; + struct hstate *hstate; + unsigned long flags; +}; + int hugetlb_max_hstate __read_mostly; unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; @@ -3162,7 +3168,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) } else { list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |=3D HUGE_BOOTMEM_ZONES_VALID; - hugetlb_vmemmap_optimize_bootmem_page(m); + hugetlb_vmemmap_optimize_bootmem_page(pfn, huge_page_order(h)); /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index fba0c5a7d46f..f977d0a7e002 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -19,6 +19,7 @@ #include #include "hugetlb_vmemmap.h" #include "sparse.h" +#include "internal.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -705,16 +706,12 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct h= state *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } =20 -void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page= *m) +void __init hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsig= ned int order) { - struct hstate *h =3D m->hstate; - unsigned long pfn =3D PHYS_PFN(__pa(m)); - if (!READ_ONCE(vmemmap_optimize_enabled)) return; =20 - section_set_compound_order_range(pfn, pages_per_huge_page(h), - huge_page_order(h)); + section_set_compound_order_range(pfn, 1UL << order, order); } =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 20eb03df542a..464192e32dec 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,7 +9,6 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include -#include "internal.h" =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -26,7 +25,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *= h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *= folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list= _head *folio_list); -void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); +void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsigned int= order); =20 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -77,7 +76,7 @@ static inline unsigned int hugetlb_vmemmap_optimizable_si= ze(const struct hstate return 0; } =20 -static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootm= em_page *m) +static inline void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn= , unsigned int order) { } #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ diff --git a/mm/internal.h b/mm/internal.h index d7673a2b5f6b..18d147487d45 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -23,13 +23,6 @@ #include "vma.h" =20 struct folio_batch; -struct hstate; - -struct huge_bootmem_page { - struct list_head list; - struct hstate *hstate; - unsigned long flags; -}; =20 /* mm/workingset.c */ bool workingset_test_recent(void *shadow, bool file, bool *workingset, --=20 2.54.0 From nobody Fri Sep 25 17:45:55 2026 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 243CB3CB2F8 for ; Thu, 10 Sep 2026 06:34:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022076; cv=none; b=lA+ZN3PBVXYM6fr69wUWRdn1f6Rgk2rfJojoCU7TCnHuL4tjzkGKqGLwCP/OIsmRf+7jVn8WZcHcdkp9+JKB1c/fd8Gf0Tf31D7fyMlm8CjaDtazcv3xwZbH3G/ELSbm6pC07N6goOJgAzqRwpUsa5IUHaCK/BQ48pzcmUu1Grg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789022076; c=relaxed/simple; bh=b4FdR30FydbrmTOLVCvDeBJIEmRP4nTqeD5QzuoTGEk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=berQHikbmg1tnYuyoaC+n3KiHUMRUGbuXd2eITQS9QcAAIZDVfCiVpRCq8uXrcSwkZF5hMoW6UMr+HxPqT2QuHTMdSELquX553HXdPORcjHeQWvRh1T8BhXXVGiVuO9LbUyM3aoffRhVSRgpXSElCuSvuMncrQnL2sk9b6IuTIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZGdGB1K6; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZGdGB1K6" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-38ea87caafeso5921971a91.3 for ; Wed, 09 Sep 2026 23:34:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789022070; x=1789626870; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=l0BGuwwnpIgT7eCl3rvqq0NlL+Ip0lSFu+Xm8xqE1I4=; b=ZGdGB1K6iIu5HwdRqCGhmjVzPSp/EZXL2nlP7rvB+1Kk8vIjtG026y6zMQX/TVGdeo WRgiro7HDr43pu9qosAOX/upsN3i5Ij20E9owkZfgptqD3UU7L/OITHi3aoQBRLGeJ4O As+5SbXP5Aoezy5Mj0uSJz8ac3Mn4tLHfaRceI9DhD+Zs6XYl/AwP2FzMBNFTMsWLy6y F5u4VIQUKVcEK6yEB3wUVrg1w/lTk5xy3MHFTb03mHjsPrpYAoq6LXvA2JhTkCFVfCkz u3sGZ4SXJgFV6Kz6ErZIlju3bbrA8IMziTewAsafPcWCZDzbcWK8S1xa67yo4NwsUft6 yxOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789022070; x=1789626870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=l0BGuwwnpIgT7eCl3rvqq0NlL+Ip0lSFu+Xm8xqE1I4=; b=b/Qieao78b8o+KEJyzHytYoKVADmpwj3E8MiN5ZIBNBlEoYKYhgn6rPNeLWb2swTGR SZHuNxYL/eMMl0thRJY4CELEGBKDf/p/CeawN/6+bhcZIFdAAcf2NMBiHS+bZ6yoZNZS 4oBacDiv3L3tlcRfOP6oAbq1lzOQ3JOXEtxExpomxQkZTy6TwzgLetlwHRlZWW8aohpW QIcELUlZqqBM0UdMeTGVJ8RV0Tkjb8/YpCvqjCrtVM73dgyYUrTd6LA/R8fW4H8+p2gk jRxfvqUOeLvYR/ArfsmU1Qtzhosza+xnUDdWEQVo1ArlPeorKNh6fE91GCkHD5e1hgxv cFLQ== X-Forwarded-Encrypted: i=1; AKwUvByuCq9SuNc+vE64Ywrr8oLJvTqJH2nAgoKCxLsUpIN3T/hvg/pQJISZyHXVe06xbBX2SUn8xWo7SQqkM9E=@vger.kernel.org X-Gm-Message-State: AFuF++kL+RFnoTvsN5xnAqnKD8LnLSHd8oTTGqu2a7jJCO8NSTXH1lVH uvpGmcOEHozenjTsqJ6CfJ/fZNg20gdDTmQcQQ3wTHXur1xkySniUMgmK1E9oOsZwWc= X-Gm-Gg: AYBFou36bqizFlKqfIjoCk7yrWwkjoQ1BMnhe/F2n8GChP465UGPFyHbSKyBUI7sg/z xBqibITWq5Im7YhjkKKwGD6UyA/YRTrBFcqddAHDj4vcyJ0tTY2eJOBfWephz1ITXHUQWSwz883 eYSEUohdzlwxiWUmFhxeArrKMXZ/S9he281svZhiWPxr5baxT641XT9hvKRm49Gi2NkDa98g3hU dC1igeDDwvJqztCQEgfTdx7IWUFFnydDH/fP2Z/jlowKrzdNZf6P5GjG8rFM1xxUQ/JC6JTuPCr 3Jv3nmu08sn3Cx5F4NqHWxTCkAJksGRwAjYXGud7DvdVScOvRZBcj1NqjqXHznyOWTt+KpQsPIM RfkCAuNXPCprUxmoyetVu+JeAHaEHCJgCcTnUhmg9rV+zxPvzHoLP1eQpikYYIm4HiuSJODI5+3 +dWumKZoBsiHrCOvOFNI2lA+1b85byhG7Mj/qwh2MZmkdvEmk7Vs8WkZ0PtzQnSqcKXI8Kz0eE7 PcViRSQ5PYXuRUIGC81r6ZsCoDDn8kONhE= X-Received: by 2002:a17:90b:54c3:b0:398:d6e8:f84e with SMTP id 98e67ed59e1d1-39b2612ecebmr61630953a91.9.1789022070455; Wed, 09 Sep 2026 23:34:30 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d84fdbae4sm1213094a91.14.2026.09.09.23.34.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 09 Sep 2026 23:34:30 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v6 17/17] mm/hugetlb: localize HUGE_BOOTMEM_ZONES_VALID Date: Thu, 10 Sep 2026 14:32:56 +0800 Message-ID: <20260910063256.64386-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260910063256.64386-1-songmuchun@bytedance.com> References: <20260910063256.64386-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HUGE_BOOTMEM_ZONES_VALID is only used by the huge_bootmem_page flag handling in mm/hugetlb.c. Keep the definition next to that private data structure instead of exposing it through the public hugetlb header. No functional change is intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v6: - Collect Acked-by from Qi Zheng v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 2 -- mm/hugetlb.c | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 255a258f11d1..900c95e346b2 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -675,8 +675,6 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; =20 -#define HUGE_BOOTMEM_ZONES_VALID BIT(0) - int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); void wait_for_freed_hugetlb_folios(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c8fd3a6ce812..894efb6b1ebc 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -55,6 +55,8 @@ #include "sparse.h" #include =20 +#define HUGE_BOOTMEM_ZONES_VALID BIT(0) + struct huge_bootmem_page { struct list_head list; struct hstate *hstate; --=20 2.54.0