From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C40B33E0758 for ; Tue, 25 Aug 2026 08:49:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647748; cv=none; b=pwgRHMG8PEB0lfXF5TLkyNWik1jOk9/jxv2dAjkiYEAgGw8hWYhRgkkfbKg1GWl+vXlHG73Hd32jZk/+ZSRKXIP7X77nqiGDAIunIeOGcnFOVOCwV6WSFX1+Mu4ZJZ+SDL9xcFWI6Vt/64CG8vi5LoYrMcAFcmtCRUHzFIlQBiQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647748; c=relaxed/simple; bh=fb22Klqi5RrupQKeoji8R+9SJ4vTRSBLUMnS/YArHeY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OV2gwdbLj/uu8C7NwrMhsUEo/BRSmwYt0XOI2ojRj5eTjPVnJcpL4SBV2W5jaIjx1bkpg7KRlI2xeP6TiKdMpQmw/8i7aQtflQFGJhsKcEtMuluYcFs+Ix0igSlT+ONtl1YWXiZ3ikGDFHAdAJvIF6eEoKIzKyztkWE609yTELU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=T5lzqA9o; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="T5lzqA9o" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-3964e76d0f4so595898a91.3 for ; Tue, 25 Aug 2026 01:49:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647744; x=1788252544; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=w63tlQxIXv6hFZNtC3XunmfoLQDKL59/Pnvl33aJ5Uo=; b=T5lzqA9owpMPDhw2MNtGHamWJyaAqNETOXkCSxL4Ixp44hFjGJXQKNmhmOjQhyAknf OXlz2QooXIx2/oSRKulEOs4m2DFj+uDv8YWzU2sC3FXr2yXFtFVnjsLsBJO4ilTJcF91 rTfLc6+pcMRwKaK19Ov7crht7AF+EDoMyYfHezml9eAyVruWXlZ2RgzMSeTz72JYqs/r cSZcbcC1wIcZ8b57gX10yos+mcQZFh+XdVhGwCQ4JX+7WuVBxQpbCjbWr0oY5ga2vF2b WvdjHMdiOB2XvUdxTVHuLtfvbeAggNrtVNcvQLKC2/R8bzFBu0NbUbPOydQXPqyDM+of jBGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647744; x=1788252544; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=w63tlQxIXv6hFZNtC3XunmfoLQDKL59/Pnvl33aJ5Uo=; b=G+4mRfPHR7qjLSoCW3CXoDVUjuH8eM4ReZ3fW9cU55z7G6TuQoJ3e6LbMRO09vbFVh 2LYb5j5nvfNqy/viiJw7sOE7wIFrNbONfpb0eETalW0qfdVcCerkQWwNqC3ALpIjB1hy PveaWMIX04oDWn1wRkdMEQA1cJwcNvTaASc5rA/GY9SqyfG/6K7vF8nG87Hye5pxkTkw kqOMGZloJucCLss5jzc9+S8gTs5FjZnQ/ZZ512yerp1Xb0QLPdsrkLYvgwup+xHVTfBM 5HJwGhJydlBny76NvvPNJg+vY8ccJSMPLg+SMuHCqPMHS91FIYSysklVVYyHOvGwuxSv hGQA== X-Forwarded-Encrypted: i=1; AHgh+Rrty+IYsl6b7mGnfFxm8THbjIFybHyUVasKx7lzZi+lYbJ2zf643wGd0gX1NsbCQACUGMgmV0Ui5AkKM+0=@vger.kernel.org X-Gm-Message-State: AFuF++ku6I6CkDQ2nz+IYoyPwlP96W/cA3hNbtPfbWoXyKU5crOgB6Ca YKKrskJwaNTmPUMgMxm0oYyD36FMebBVwatVABFgKS2uI6diDkQ2MEYaCJ0ps+XNhu8= X-Gm-Gg: AR+sD10YE+e3Iyq9FNkadIpIz8Rtg8Ge+luZqrCM5o/hrDxUAoZRPiiqcP66nJzQmSE GEsOK2O/XdlTqUZtWwRkMA20yFHpwsehg6MPUq0BaImq1TK3cgOw1oYn3rQ5yoebpFudF1Q7Wn/ q+15y7LwWbHtqIUPPxus2UnjOuC1zBrAbeivya045jxeYfTSk5fsnkg+dVqgqfA5Y0DClcPY5sR DADAIsp4v+lCBCqnQwXn6M7e5FD/Mg8CwzDFvmolcgqM5llsSqXIfWHja9wHpq+IJcKusYUK8t3 VF+8fIB8npGkPAItqqhkvD5cNYnmIpZQuvyd8Xmn9E0+fv8SeWD67uN1Dd9oC9lSh6qaAV8qrVV etxRYOU0Wc6u2/QgHFgsmVyy121BGzO0rAReN2vAGObEIvJifMMjXhHfi1vfGvA4eywiJzFma6r G7MiUN4IFNeRwefrfOLSiVTJiVl4fE+Nnngr5H5PToybqOHrun5sBYIQNMWZd8ipK4RnXa5+OVT Lx0ipZX/+alqvdV3Z/ZRl7sFwTVhDE= X-Received: by 2002:a17:90b:560d:b0:38e:9045:bac0 with SMTP id 98e67ed59e1d1-395deea6acbmr49754716a91.5.1787647744397; Tue, 25 Aug 2026 01:49:04 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:04 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 01/17] mm/sparse: relax struct mem_section size constraints Date: Tue, 25 Aug 2026 16:45:52 +0800 Message-ID: <20260825084608.47437-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct mem_section is currently forced to a power-of-2 size so the section-to-root lookup can use a mask instead of a modulo. That requirement makes future extensions harder than necessary: adding a small field can require configuration-dependent padding or layout checks just to preserve the lookup scheme. Keep the lookup correct for any struct mem_section size by using a plain modulo instead. Do not leave the layout entirely unconstrained, though. Keep struct mem_section double-word aligned so modest size changes, such as adding another word-sized field on 64-bit systems, still keep a compact and efficient layout. If future fields grow the structure beyond that sweet spot, the lookup remains correct; only the exact layout efficiency changes. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) --- v2: - Align struct mem_section to power-of-2 where possible, but support non-power-of-2 lookup (suggested by David Laight) - Collect Acked-by from Mike Rapoport --- include/linux/mmzone.h | 11 +++-------- mm/sparse.c | 2 -- scripts/gdb/linux/mm.py | 6 ++---- 3 files changed, 5 insertions(+), 14 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff5416..0a2428714108 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2027,13 +2027,9 @@ struct mem_section { * section. (see page_ext.h about this.) */ struct page_ext *page_ext; - unsigned long pad; #endif - /* - * WARNING: mem_section must be a power-of-2 in size for the - * calculation and use of SECTION_ROOT_MASK to make sense. - */ -}; +/* Sacrifice minor padding space for efficient lookup. */ +} __aligned(2 * sizeof(unsigned long)); =20 #ifdef CONFIG_SPARSEMEM_EXTREME #define SECTIONS_PER_ROOT (PAGE_SIZE / sizeof (struct mem_section)) @@ -2043,7 +2039,6 @@ struct mem_section { =20 #define SECTION_NR_TO_ROOT(sec) ((sec) / SECTIONS_PER_ROOT) #define NR_SECTION_ROOTS DIV_ROUND_UP(NR_MEM_SECTIONS, SECTIONS_PER_ROOT) -#define SECTION_ROOT_MASK (SECTIONS_PER_ROOT - 1) =20 #ifdef CONFIG_SPARSEMEM_EXTREME extern struct mem_section **mem_section; @@ -2067,7 +2062,7 @@ static inline struct mem_section *__nr_to_section(uns= igned long nr) if (!mem_section || !mem_section[root]) return NULL; #endif - return &mem_section[root][nr & SECTION_ROOT_MASK]; + return &mem_section[root][nr % SECTIONS_PER_ROOT]; } =20 /* diff --git a/mm/sparse.c b/mm/sparse.c index 7c15406e77f5..c84b4c7b8c70 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -322,8 +322,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count =3D 1; int nid_begin; =20 - /* see include/linux/mmzone.h 'struct mem_section' definition */ - BUILD_BUG_ON(!is_power_of_2(sizeof(struct mem_section))); memblocks_present(); =20 if (compound_info_has_mask()) { diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index dffadccbb01d..da4e8e9655a6 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -70,7 +70,6 @@ class x86_page_ops(): self.SECTIONS_PER_ROOT =3D 1 =20 self.NR_SECTION_ROOTS =3D DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.= SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK =3D self.SECTIONS_PER_ROOT - 1 =20 try: self.SECTION_HAS_MEM_MAP =3D 1 << int(gdb.parse_and_eval('SECT= ION_HAS_MEM_MAP_BIT')) @@ -100,7 +99,7 @@ class x86_page_ops(): def __nr_to_section(self, nr): root =3D self.SECTION_NR_TO_ROOT(nr) mem_section =3D gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] =20 def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT @@ -249,7 +248,6 @@ class aarch64_page_ops(): self.SECTIONS_PER_ROOT =3D 1 =20 self.NR_SECTION_ROOTS =3D DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.= SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK =3D self.SECTIONS_PER_ROOT - 1 self.SUBSECTION_SHIFT =3D 21 self.SEBSECTION_SIZE =3D 1 << self.SUBSECTION_SHIFT self.PFN_SUBSECTION_SHIFT =3D self.SUBSECTION_SHIFT - self.PAGE_SH= IFT @@ -304,7 +302,7 @@ class aarch64_page_ops(): def __nr_to_section(self, nr): root =3D self.SECTION_NR_TO_ROOT(nr) mem_section =3D gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] =20 def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AC523E95A8 for ; Tue, 25 Aug 2026 08:49:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647751; cv=none; b=nMrsG9UFsGwRlmgc2xeMhro8Of9DaA0ylJzylB3s4/CLIxgqbnbWIc9MshoF8E4flylPP6Dp/n9kajDbQlzqJf+Ib6eVakAoubPwVmHrA/OHkilaZdiVEQC6memUpv3yk9uwS6p0q/ZkqSAkAqc5EI5MTLFEGQ28d54jm+NtWj0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647751; c=relaxed/simple; bh=L5oM5ZS7ALnPuDRywbzT4C1Pe4FtsU9etB5iEw97NF0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZX1koA5bP9npeiXxftRAQCs8OJLGrLe3V6oedNRH8X0V/LWGGfobkgYqq7RG0ZNp0RNUE+1GE9Rvp9CH6aDuh1UQPuFZoz2VXBTbAveg0DU5aU+bDQpzz/vIJ9ztKGhvJM+RgPgH2ak80flIkIilhMzLJN2WwVMnfDvwAMzaQHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hD+hrkyz; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hD+hrkyz" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-cbee846deecso5101091a12.1 for ; Tue, 25 Aug 2026 01:49:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647749; x=1788252549; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SREjF1UBBXv2gv7dXR/21RVjzi8BP52PTZ7zJ8g1H34=; b=hD+hrkyzx55hG6KLbONqXTgUJ+peLNCucfhD3e2Du4WP+wJWLCYwn0f6UIiJbB5uu0 DIgk2MSGRetlivvqMCkNZ0G8hDmwbDb1E9aWAbAJCN5BrGLDqb1nDM+5CgzewSEAmtO2 Sjq8BwZTilETS+qx+NvkQF9QfmhhjPGCW4U5pAvE8SfuoZcgKWoeQK0K25ZD3ni75nFP 1jxAYoA8kAdXwbiWXUUngQT9P8BTRF5nK00d1liULxKg210WHYYpWp8UtkgCK+IV1bcJ GoZ6Fzber+3EopZGy3ta2/d2krHngrAiqRXWuWeTDG6IoTGWxXUqy/wW/P9g+HoErBTb Ve1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647749; x=1788252549; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=SREjF1UBBXv2gv7dXR/21RVjzi8BP52PTZ7zJ8g1H34=; b=hrmU1D+3s1l4nlhF7JGvthGot4IGlunl/2U1EpPoeJrczU829rb5DIZs4fSzqN+p2M 7W6sieASYA6VSrd9sk+dK+8UhI+imfLdC6uvI2lQKgSYEiDp7mu3jjutrFDfe4/AE7Q7 pDruebk+bthHfaK3x21QCoxolkHGER+23DDaqb7/dRwrjhIYjQWm0QShumR93MkvuUy4 SYK8uPTVDZrn6bEvAdFb4IAmDp8pmGBf6zbWkAfuWlXDfkRi/Q5FZLljWaGei7KF9Zpd uv9wlaXR8NyOjTG5qRFWL52lXY6ypxsI27qs6kPUzaVuBWUkDIaE7EdcKX180kn8U4A4 fxPQ== X-Forwarded-Encrypted: i=1; AHgh+RoIaheUqw+d8TvK5gcqUhIJFQ5HJcwRPZFgaheKTHWdcok9c+aGBWP980omZMYwz4zsaxCt+FIeW8X+U4A=@vger.kernel.org X-Gm-Message-State: AFuF++n2Pyk/Uyi20H10FF+bxPdaQvgIiPdLxlWlxckx2XbT+wYkfZFM CLa2vsd3IkFyFq5d17cTYrxwUtpa0Gi3qrDgrHVadAqBv3N/Rq1dVlVR16xple7hRFc= X-Gm-Gg: AR+sD11zvqvejbA5DiL/PM5p6UdcoHRGC8di+vtRWwrSRj+eL1MebpWMxh9UWHvv5tR eU5olgrRDRwRb36f5u3DeBosx46DZpkb+NDnt6vcdCKJ4XIerCKoTJ88IBwyMgEyDqgwGGA+563 Qega6JxxPHvEPFXMAlcDNaczb5aG1TeiJmmaa+2S/pcXJ807Q9zRzrxurwW+FdpuGr7/8qPfkSn OpE1p7ybCUJDxhUp3YSk8B6nIzLpEu4VwIlkszy/zHwAJDviGVIMxcm0X7vpE/Wme9Lw97j4Oq1 K13rkfVc2VGAZly4X6toPcQCGCTMOUEqXEPG4sRPmbstiDoUZk4ssqEBFgbNCwX3gYqWBoaP6IV lvyS7aco0XEyXRA+K7dhD8Lz+9MW5h3UcMfMwcdB/H/N+8tzBe3WSUd1FPBUD+Wcapi3wDzs2IH 5LHxoPVd+RJyW4QHBoPIslhywNKiwUlaMyZj1oeYrHGo8ltjtU2wnOXaOIiA079svL2YhwM/x9H tLBQhyRICBsmbOP4/O8V8FSqQ== X-Received: by 2002:a17:90b:56c8:b0:38a:c3f:3b87 with SMTP id 98e67ed59e1d1-3964652108bmr10799150a91.12.1787647748524; Tue, 25 Aug 2026 01:49:08 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:08 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 02/17] mm/sparse-vmemmap: rename HVO order macros Date: Tue, 25 Aug 2026 16:45:53 +0800 Message-ID: <20260825084608.47437-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The macros VMEMMAP_TAIL_MIN_ORDER and NR_VMEMMAP_TAILS describe the order range where HVO can be applied, but their names tie that range to the tail-page cache implementation. Rename them with a VMEMMAP_OPTIMIZATION prefix and use the new names in the HVO paths. This makes the code describe the optimization requirements rather than the tail-page cache implementation detail. No functional change intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v3: - Collect Acked-by from Mike Rapoport v2: - Rename the macros with a VMEMMAP_OPTIMIZATION prefix (suggested by Mike Rapoport) - Drop intermediate optimized-folio size macros (suggested by Mike Rapoport) --- include/linux/mmzone.h | 17 +++++++++-------- mm/hugetlb.c | 4 ++-- mm/hugetlb_vmemmap.c | 2 +- mm/sparse-vmemmap.c | 4 ++-- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 0a2428714108..5fb9b37819d5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -107,13 +107,14 @@ is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 -/* - * vmemmap optimization (like HVO) is only possible for page orders that f= ill - * two or more pages with struct pages. - */ -#define VMEMMAP_TAIL_MIN_ORDER (ilog2(2 * PAGE_SIZE / sizeof(struct page))) -#define __NR_VMEMMAP_TAILS (MAX_FOLIO_ORDER - VMEMMAP_TAIL_MIN_ORDER + 1) -#define NR_VMEMMAP_TAILS (__NR_VMEMMAP_TAILS > 0 ? __NR_VMEMMAP_TAILS : 0) +/* The number of struct pages covered by the retained vmemmap pages with H= VO enabled. */ +#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (PAGE_SIZE / sizeof(struct pa= ge)) +#define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STR= UCT_PAGES) + 1) + +#define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ + (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1) +#define VMEMMAP_OPTIMIZATION_NR_ORDERS \ + (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS = : 0) =20 enum migratetype { MIGRATE_UNMOVABLE, @@ -1158,7 +1159,7 @@ struct zone { atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct page *vmemmap_tails[NR_VMEMMAP_TAILS]; + struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4f6f58bf3db6..04e6c4244cd6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3346,7 +3346,7 @@ void __init hugetlb_bootmem_struct_page_init(void) struct zone *zone; =20 for_each_zone(zone) { - for (int i =3D 0; i < NR_VMEMMAP_TAILS; i++) { + for (int i =3D 0; i < VMEMMAP_OPTIMIZATION_NR_ORDERS; i++) { struct page *tail, *p; unsigned int order; =20 @@ -3354,7 +3354,7 @@ void __init hugetlb_bootmem_struct_page_init(void) if (!tail) continue; =20 - order =3D i + VMEMMAP_TAIL_MIN_ORDER; + order =3D i + VMEMMAP_OPTIMIZATION_MIN_ORDER; p =3D page_to_virt(tail); /* * prep_and_add_bootmem_folios() can access pageblock diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 917db0984143..ae8fdaa42118 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -494,7 +494,7 @@ static bool vmemmap_should_optimize_folio(const struct = hstate *h, struct folio * =20 static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) { - const unsigned int idx =3D order - VMEMMAP_TAIL_MIN_ORDER; + const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; struct page *tail, *p; int node =3D zone_to_nid(zone); =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 5a2469fb1838..aa6a4a2fae98 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -329,12 +329,12 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * unsigned int idx; int node =3D zone_to_nid(zone); =20 - if (WARN_ON_ONCE(order < VMEMMAP_TAIL_MIN_ORDER)) + if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) return NULL; if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) return NULL; =20 - idx =3D order - VMEMMAP_TAIL_MIN_ORDER; + idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; tail =3D zone->vmemmap_tails[idx]; if (tail) return tail; --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C5DE3D3D14 for ; Tue, 25 Aug 2026 08:49:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647755; cv=none; b=cANhCkNk6OWFSDWny1+omTt6yINdIgb1679c/1pcvUOaHt75/KkAqDYcySsSP7T3upqS/54NMS0YtwltKTRV/rqEpAVLYzSXBPLQzj2301Z2H7qpfF9yuBd5bWExpsUsbrdY+WPRtExfKmzdJEJ7NhiKeszLjC6YU3wlrAgg5wM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647755; c=relaxed/simple; bh=dsXqZpDQTHch2aStcnnLLnUNGYKwjL8GsF0ZaNqiIFA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i5MFrhC1EchyrbPMXrXUhQsWd/LiicvhtotaK1iNBakgHgzBo+R7OBaYbObP+DFxDz6YJlukMNEAQVyyXxEY16nJiWGT9vNLs8Of4jhFdzXm0k1xndURixdRi5Geh66gBNWHKz+StbM+Y7U8+Id1SJIRMHck+nFfbNLRP4wJO+c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=d72NLm/J; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="d72NLm/J" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-395cf2535acso696533a91.1 for ; Tue, 25 Aug 2026 01:49:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647753; x=1788252553; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XZrs8X2fjGywiu1ctnjB1m4+W6a3lmWBg96SgWKBRZo=; b=d72NLm/JN0POBJXfdBSmtgZ2pYZwnUT++lfwR9q/IHIwSddA1+6/yR14gz3Rg9pQbC SmT8VKnRyQCYc2EjGCUkJ5Mc8JFEgpmomxkf/tmgUdn4/y8MeUQH54LFKLzIQWfIYHiV 61rgJ7Lpwol8vYe+7RaR7SExeb7ueM/bIt+pNcjo+DkhtLKqrmA6og/Fk/YL2DIbiEOz BrjYQ+WR2uxo7vcf4YmVRvvxkTCdCXe7LNrCjda+VoBFDefz0OIXWAxmrEl+H+/iuNF9 IKJbCYWGemPj3FQHB40U+hjMsvQlOS4Jmf/w3YnMKcIOOMnRgOL94Beyd4r2sdQy7IO4 47tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647753; x=1788252553; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=XZrs8X2fjGywiu1ctnjB1m4+W6a3lmWBg96SgWKBRZo=; b=ek/TOqc+f6XEDr16mJPg1+XGwA6pi6F90uapTlIet44jOM7VNmK895HRrOkHxy0DQj xPPWIcrQYOy+Z0T0Ziy+NHCOHaLLH/M12mVIk0iTnmOC+GSKmRg9sNIW4fCaH0/3Y7Yc OQsDIqazd9TL7QAkOuMhZAx0vxZm6KDFfaljAOtNwjnkwGrUSmhZ+Gl49KO1g1RxVr+U MPppth2MBfMhIrZoEoit+FIh28XPtmdHskdsG04xiXY10QBX85nya73PsQPiCbA3CkZo knSFkHI6LxXUwRQ3+itCeg7OkxBbd9zXVH6CJuzy+3Fz2fNSbvhlsbMdnkUm6uOgzkc/ yRUA== X-Forwarded-Encrypted: i=1; AHgh+Rrk1NChBF2gdtRNqQqMuClbGi94f0+hmybV9CJfrlfJEZM1tUiEg4xX2ZjNqGu6vGvO+aUlW85WlOQ+b8o=@vger.kernel.org X-Gm-Message-State: AFuF++neN0YDARd3+b6o3/j/12fUo159uFBSYmVpLuIo6L3i1y9BaJ/M V07qwPr3q7PQJotd0EnLaSzaj7/UpxFhxqZ44gv0sHqiZqLPabyi4dioRt5mNZVI5LU= X-Gm-Gg: AR+sD12FHJKbqY6i7P30JHt2u+Y2D7sSuNpyoI1oWU1LnjOWK/1+mOmUHWLTIuWIiMH sjNgRh+qGG0dl5Pvmb1ZcLfFd6+xT0NbzLwRSbQEaaqDIKqJEmImn+CyOwsnem921/Oov0XPeaJ uub7q08wtCLXLEEUriJeb94gxtHthg0k9m0afNFLu4yHGpk+jQpasZlSdSyOdJMS/8vT4j1F/zR O1T61vz1AqI5zQi7cNgI9oRRNYxAioQVLBW2giiWcQO2e5OhcOqwdphb6zK7Qd+JOYBfzSaggrX 4Y0tX1q1W5VWzCM7PvdNMfL97z01zFyCpuq0wLqBkdlK4RXP8BZzLKVu/7dy8CXvUkg5zk52EYl w2eXYt9feSxyagpnm5PprFoDIXSZy0i4Qw9Mf+YQ+5fgTD14TOzVm3b4UXPy+MxLWnCbM1mP92u 3QYjVc9G/cdAAC9wN7RdL1ugIXyribhbHCIDbQdwkNUe5M6rxFv5ZOl97DYUOXBuGB8ESz42hS3 CuGk5i1BguB1QjTuqEVSjhs/A== X-Received: by 2002:a17:90b:2b48:b0:38e:7297:a92e with SMTP id 98e67ed59e1d1-39645b5b2e7mr5586208a91.9.1787647752583; Tue, 25 Aug 2026 01:49:12 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.08 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:12 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 03/17] mm/mm_init: skip initializing shared vmemmap tail pages Date: Tue, 25 Aug 2026 16:45:54 +0800 Message-ID: <20260825084608.47437-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memmap_init_range() initializes every struct page in the target range. For compound pages with vmemmap optimization, the tail struct pages are backed by a shared vmemmap page. Initializing those tail struct pages would overwrite the shared vmemmap page contents, requiring users such as HugeTLB to restore the metadata afterwards. Track the compound order for HVO-backed sections and use that metadata to detect struct pages that fall into the shared tail vmemmap range. Skip those shared tail pages in memmap_init_range(), then initialize pageblock migratetypes for the processed range with a helper after the per-page initialization loop. Keep direct mem_section access inside sparse helpers by exposing pfn_to_section_order() to users that only need the order associated with a PFN. This lets memmap_init_range() skip shared tail vmemmap pages without exposing __pfn_to_section() to !SPARSEMEM builds. This is a preparatory change for consolidating handling across users of vmemmap optimization, and it also avoids redundant initialization of shared tail vmemmap pages during early boot. That early-boot benefit appears only once HugeTLB is switched to this common handling, since HugeTLB is the early-boot user that creates those shared tail vmemmap pages. Signed-off-by: Muchun Song Acked-by: Qi Zheng Reviewed-by: Mike Rapoport (Microsoft) --- v5: - Add comments for pageblock migratetype helper callers and shared tail vmemmap skipping (suggested by Mike Rapoport) v4: - Rename pfn_vmemmap_optimizable() to vmemmap_optimizable_pfn() for consistency with vmemmap_optimizable_order() v3: - Replace the !SPARSEMEM __pfn_to_section() stub with pfn_to_section_order() (suggested by Mike Rapoport) v2: - Fold section order tracking into the first user instead of keeping a standalone API-only patch (suggested by Mike Rapoport) - Rename page_vmemmap_optimizable() to pfn_vmemmap_optimizable() and pass a PFN directly (suggested by Mike Rapoport) - Initialize pageblock migratetypes from a helper after the per-page loop (suggested by Mike Rapoport) - Use a 1G PFN chunk for cond_resched() in the pageblock helper (suggested by Mike Rapoport) - Guard section_order() with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP so it returns 0 when HVO is disabled and lets the compiler optimize the code as much as possible (suggested by Mike Rapoport) - Explain why the !SPARSEMEM __pfn_to_section() stub belongs here (suggested by Mike Rapoport) --- include/linux/mmzone.h | 8 ++++++++ mm/mm_init.c | 40 +++++++++++++++++++++++----------------- mm/sparse.h | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 17 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5fb9b37819d5..df31cac12311 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2022,6 +2022,14 @@ struct mem_section { unsigned long section_mem_map; =20 struct mem_section_usage *usage; +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + /* + * Normally, sections hold regular (order-0) pages. However, for + * sections with HVO enabled, this tracks the compound page order + * to enable deduplication of redundant vmemmap pages. + */ + unsigned int order; +#endif #ifdef CONFIG_PAGE_EXTENSION /* * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use diff --git a/mm/mm_init.c b/mm/mm_init.c index 1533aebafb68..f801fd486085 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -677,21 +678,19 @@ static inline void fixup_hashdist(void) static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ =20 -#if defined(CONFIG_ZONE_DEVICE) || defined(CONFIG_DEFERRED_STRUCT_PAGE_INI= T) static __meminit void pageblock_migratetype_init_range(unsigned long pfn, - unsigned long nr_pages, int migratetype, bool atomic) + unsigned long nr_pages, int migratetype, bool isolate, bool atomic) { const unsigned long end =3D pfn + nr_pages; =20 for (pfn =3D pageblock_align(pfn); pfn < end; pfn +=3D pageblock_nr_pages= ) { enum migratetype mt =3D kho_scratch_migratetype(pfn, migratetype); =20 - init_pageblock_migratetype(pfn_to_page(pfn), mt, false); - if (!atomic && IS_ALIGNED(pfn, PAGES_PER_SECTION)) + init_pageblock_migratetype(pfn_to_page(pfn), mt, isolate); + if (!atomic && IS_ALIGNED(pfn, PFN_DOWN(SZ_1G))) cond_resched(); } } -#endif =20 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) @@ -886,6 +885,17 @@ void __meminit memmap_init_range(unsigned long size, i= nt nid, unsigned long zone } } =20 + /* + * Vmemmap-optimizable PFNs are backed by shared tail struct pages, + * which have already been initialized during vmemmap population. + */ + if (vmemmap_optimizable_pfn(pfn)) { + unsigned int order =3D pfn_to_section_order(pfn); + + pfn =3D min(ALIGN(pfn, 1UL << order), end_pfn); + continue; + } + page =3D pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); if (context =3D=3D MEMINIT_HOTPLUG) { @@ -897,19 +907,13 @@ void __meminit memmap_init_range(unsigned long size, = int nid, unsigned long zone __SetPageOffline(page); } =20 - /* - * Usually, we want to mark the pageblock MIGRATE_MOVABLE, - * such that unmovable allocations won't be scattered all - * over the place during system boot. - */ - if (pageblock_aligned(pfn)) { - enum migratetype mt =3D kho_scratch_migratetype(pfn, migratetype); - - init_pageblock_migratetype(page, mt, isolate_pageblock); + if (pageblock_aligned(pfn)) cond_resched(); - } pfn++; } + + pageblock_migratetype_init_range(start_pfn, pfn - start_pfn, migratetype, + isolate_pageblock, /* atomic */ false); } =20 static void __init memmap_init_zone_range(struct zone *zone, @@ -1112,7 +1116,8 @@ void __ref memmap_init_zone_device(struct zone *zone, compound_nr_pages(pfn, altmap, pgmap)); } =20 - pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, fa= lse); + pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, + /* isolate */ false, /* atomic */ false); =20 pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); @@ -1921,7 +1926,8 @@ static void __init deferred_free_pages(unsigned long = pfn, if (!nr_pages) return; =20 - pageblock_migratetype_init_range(pfn, nr_pages, MIGRATE_MOVABLE, true); + pageblock_migratetype_init_range(pfn, nr_pages, MIGRATE_MOVABLE, + /* isolate */ false, /* atomic */ true); =20 page =3D pfn_to_page(pfn); =20 diff --git a/mm/sparse.h b/mm/sparse.h index 3b744667a7e6..1fcda8a1c270 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -10,6 +10,39 @@ =20 #include =20 +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static inline unsigned int section_order(const struct mem_section *section) +{ + return section->order; +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return section_order(__pfn_to_section(pfn)); +} +#else +static inline unsigned int section_order(const struct mem_section *section) +{ + return 0; +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return 0; +} +#endif + +static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +{ + const unsigned int order =3D pfn_to_section_order(pfn); + const unsigned long nr_pages =3D 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + /* * mm/sparse.c */ --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E5C530C168 for ; Tue, 25 Aug 2026 08:49:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647758; cv=none; b=nuKtlviPx7IVsy4Q25ODAhHZUWMNoZhnCgtzKKE4traP1nTtidTyxk7aChNboiDnVRDd/9Af00d8/FWisZNskZPXr8RFwlyUG3fTWaXjyLTualweILQ5RDqcSC/ODbDQhI2vKf73mZ3oB92ms9VOY19x42ory73c/M5jDYnrjFw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647758; c=relaxed/simple; bh=owQ+Elu7FhHy0RmBaGNrJ1F4/GpIAVuOEqAZSW69+JA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HYfjC9RoDbAHE+XIPsJ55SoY2NCVl63rrpwr/SXtonlwnhOr+BKESmzSOYPgT0vMOkuyFSF74YBwQ5d+kGigDftCxOpYzVWM/wAgwScQ9vSieRbW/takJDtk16iANhfXh1/yL3HnBICn+0GruJJsOD5ROPDKnOn57twB18L8OEM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BnCK1pjT; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BnCK1pjT" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-38dfe7eb825so4045282a91.0 for ; Tue, 25 Aug 2026 01:49:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647757; x=1788252557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bOzdkIPPNoupexe8+AVOrQT3DBRmXPJ6Kybkle+RLi4=; b=BnCK1pjT+TGP/bwtqD1YSa5c6SRPXHwzoQFvECvTwIcw4ms/VeEsiRUHuSAbQgvlve VC1a9CpQaY4mAFUHvT56Jq2JiC1T/KHdcnHwP1YbicWvoZs0hsezzEiZv3Wgnmc1IjdD v7nGg1j1uNkgETIeijTvKdCTepmfz3U23CB7mTADSfE6mBXhteDqyg54S5w+v1fa7MqY KRsyObf2vBoTbCztzBuCNR6bgPyfP30ybWB64nTGD/pCN9NaxJqhq5QZyiWOPZl3+eMN JNylUlRPIGIbU35mDp9aFVp5mG/nPFAoZ6wjAK5pQVcbNHiRjTHavQwwqyc8AcY+/evD XXBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647757; x=1788252557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=bOzdkIPPNoupexe8+AVOrQT3DBRmXPJ6Kybkle+RLi4=; b=OPbCAz4Cf09N73TJfnCd7BFJCmbBSAyZ3yhPozvCjalvaSq8RIKzbuaFbI1nO6hBB1 baJYWKuDShZUBkqIG5KMugBYF4OG5QkZRG30mY/fBL9tNx5TcWtVlkuRdFGnWHWrVqtk 3rOSeRge8ddeKoHAqU0zjKOEnvn4EU2QMFnQ2LQbqQqchnY9sntORZdKOyOS7uzrHnrV F075lhEgJoPzaJrb4aUeHq6+meXlMw8sFrYtj52Q2tSF+IXVzR6wTVsEqSqMOvDiDWbL WsxzXlu8JGNREBPn0WrzWOpr2eZhHKQxYPYCDehZuBjXqxrAzRKeqVjetcioZ9Ose1bD YFew== X-Forwarded-Encrypted: i=1; AHgh+RptWxmJm5vbzus2Q+/+Fmgwn74x2hvEt0Wy5e0E6su66amNuTYv6yXCqrrMfYu8SOR7/UK2DGWVL2DW/oo=@vger.kernel.org X-Gm-Message-State: AFuF++lIeAcnaNrn993KxJsCWVtNTeEA0rcH5A1gJvqRYmFANY3C9f5h TJpLBZJHrco6Htu5cmf+1GczjbVVg5MFZi15FaeEjf6u2FZvueILVKBdZz5Za2/Uv/o= X-Gm-Gg: AR+sD11zpYt962p9R0l8580jGVHQbHW/g3Nw5tvumkpOtd9ltYIoGd16c2dMn8Kk+Sf ExAFwtq4RDEJbNCEaf+E49M0pj6W6lOjWSb4+A6azms585ltxKsC6BwsrZy4aeFjmZwcY0j3Y/I u756xJGNUf5sJ8XG8Ek3pucrx+LtL7oQ93AZvJsjc5WjZhr76NH0xYSsgAf3hzM/N5XF8wvULAi 7GEpFYUo0/7pnTG2hb6vSstIvKOvHphXZRAyRvTNENkF6gTfKomkWb/0d+FC1BCgi9TvmhfXhGW csDnFROxPaQNcyU8sBcwX9RH8ezERzdORex7CQOL11QYnvDnpKaQqF0NNckQdSMrTzEXbErYKsV TmMNDAVBvC4bRf0et+CzewpRaq2q1kDbFXM7erCF7RMAvxF1VQYudy3r+FclbdKIkgDw6qmY0sb zGdfpbmOgDU808udDxTdxhsOsIq/Dsfeu5l/K2CLtqMbDfMq+FxLotnuD3pQsqULbHPpvjVfdXg AlI9zxb/yG/N9uL5KE6elQIXQ== X-Received: by 2002:a17:90b:4b0a:b0:38d:ddc2:7ccb with SMTP id 98e67ed59e1d1-396462ff21fmr8688371a91.1.1787647756678; Tue, 25 Aug 2026 01:49:16 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:16 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 04/17] mm/sparse-vmemmap: initialize shared tail vmemmap pages on allocation Date: Tue, 25 Aug 2026 16:45:55 +0800 Message-ID: <20260825084608.47437-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The shared tail vmemmap page allocated in vmemmap_get_tail() used to be left uninitialized, because memmap_init_range() would later overwrite it. That forced users such as HugeTLB to defer the initialization to their own setup paths. Now that memmap_init_range() skips shared tail vmemmap pages, initialize them immediately in vmemmap_get_tail() with init_compound_tail() instead. This moves the initialization to the point where the shared tail page is allocated and avoids relying on deferred handling in individual users. The remaining deferred initialization in HugeTLB will be removed once it switches to the section-based vmemmap optimization mechanism. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- mm/sparse-vmemmap.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aa6a4a2fae98..107215cf8488 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -338,19 +338,11 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * tail =3D zone->vmemmap_tails[idx]; if (tail) return tail; - - /* - * Only allocate the page, but do not initialize it. - * - * Any initialization done here will be overwritten by memmap_init(). - * - * hugetlb_bootmem_struct_page_init() will take care of initialization - * after memmap_init(). - */ - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); if (!p) return NULL; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail(p + i, NULL, order, zone); =20 tail =3D virt_to_page(p); zone->vmemmap_tails[idx] =3D tail; --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C66073E0080 for ; Tue, 25 Aug 2026 08:49:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647763; cv=none; b=IMqh0wFEDDGP3E+V8Fy49BVSYNBFqTFCbcc5IRbh++bU+JtdQo+sEPgDiVV7/0C4i22WS5fZYwlAyHuShNW7bgZBLGD9Bf+r3g9DQFq0gMKc5QbfPQ7Thwxlknx8u30DqO0Z6iP9FOHNo98ni/ZAXgfxm1pnDnEQFuKBHP14je0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647763; c=relaxed/simple; bh=n2o9Yholv73pEPg534AwQaSurQvN2Cw+XMEGBrEt/fw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=clkVF6gjtfoT0NK+V3pP07SzhdwUg8gl1torOzW1VJ29DqtFqgHWLAxEdQCRP4iFr1LABtAUP2rmpG2lweCwMCP0ggazvpvBvfcuju2Zle7Td8nOVZvHdGYMbZWM5BmyBGqBUVWRLozMFvW75IJwRBOlCUMH/Mh/BNV6ctxmT5M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QOGRoTI8; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QOGRoTI8" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-39266382df6so3772522a91.3 for ; Tue, 25 Aug 2026 01:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647761; x=1788252561; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=NhkaTEU/tSM/+MOCHKQTY4aSgCg16Ul5HZ94OnnT8FE=; b=QOGRoTI83PFHtekIEm/5xkc6QcAkPQysrpwVU49Q1VhPjINDjCgsN8hA78BmWPApR5 34WLyB8qD7jypEZHfXyqZsB8UXfYvGqM2sMB25xZJ+QkRS5jZVyKKOFlhJHXPZ0DohCu mJ72d3MreKdyFPLiKci44ZmikVM6JZUoB/QjC5CrKrfM9b2R70GCBzH6NtiBew1T1d8v ZylMB3Kly6d8c8dItVlYo8OAvUTiIoioHe0i/5nU6fO+jJk8/sHKWrxNpohLWUsEpide 8wYRcccVSXiT0vkUMpsYJKPxBwNn91N/Czl1+CpqZALO9Q7zxb6KlVntwX+biEbVXqlf QoPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647761; x=1788252561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=NhkaTEU/tSM/+MOCHKQTY4aSgCg16Ul5HZ94OnnT8FE=; b=sfqU4F2j2BNpn1agHjIS31/Nt+BEinU43JZ8UdhQb+XR0bPffTzN700hQHzdlavOK0 rc4kcvue9Rex5v6GrogoXcXD60wnAadu4J8CjP0GSNKNbyr5qPBejh7B81ADzAu5PhgJ obxERcouEct4vK67o8zJPkLTUniQXDg5PzfRQwPq0+k4at5Ux8YQ2yMxKXEgrYht9pYX ijuwD4YIW9TE9vZsrL2rni/WlqV38GmceglVtUSBlo133+UbaPDm6Yvr+l4zobXw4dSL QrAX09EGZAU1MUoO7GgqwOEytfWGXgij8RFuwVM2u6JFUMiFfMmnsrCQFdgCBPCrrbmT gYcw== X-Forwarded-Encrypted: i=1; AHgh+RpXyowDnFwkudaatC2NzMFxEmqyO88MwwQN+IyCufXnapWSoBKJNBj7SPJO32hY6XcwU+RuccQNxMcHayU=@vger.kernel.org X-Gm-Message-State: AFuF++kq8H2S9QEg/QDfrv23Xg/mRvh6EXr03DNke38VWqw5lvWeP2dk EJfJScjAtdIrhYRaoxjBqE7Lnkwh7UTPAvYyAal0g8P73/XacHnVfpzUaSCJoDBZMWs= X-Gm-Gg: AR+sD13nTUZ3BS4I04dryqVEU4jENUoLGw+J/btfu5Mrmz4dkcAxlMBOGw22OwCsiG6 jTOgo7CCJ5GOesxt+sYVHlTqRUmS9duJix/+DuIgXwXBR1jxc8hZ//WbsDGzpFABi4OvC+fa996 RlZzC3P1JC6pR4KnvLJZ9VuShIAvv5lOC5Pd3281K4WNXejjy3QIs/I8QIiYN2Qt4Sfo65qtLq4 53P+ZOQ4fOFtp2i1QpM2xj+B4rPiN8+MQ1F8pnqzs16oIhhvM/yW0G5AixHKRgBRaDB5thgI6kp wucEAcGh7MfczvlsXpR6c6sl5P21xdzQXfiqdQuksgt1yzy9hA8MCcwzo+mzX/2rExvULt4bm9+ Z4k/JAXvCCe0yMB5VPjah3rir5Hkc/o2u2l9kWeHaUHSyihtlLZlDwHW2TT+KzrSdzu+iA/0P52 SM7gCLOpDUvTslP4zRwKEZbGDG3GvNA51kPZekjD8JyEANYVlgUED3ueevnBo9ZJcdaKJwI6X64 t9E45jP2clKJXwzH8wBmoaQy/hwyq2c1XWB X-Received: by 2002:a17:90b:1e0d:b0:395:4e0d:bc60 with SMTP id 98e67ed59e1d1-395c354d7ebmr51570267a91.9.1787647760819; Tue, 25 Aug 2026 01:49:20 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:20 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 05/17] mm/sparse-vmemmap: support section-based vmemmap accounting Date: Tue, 25 Aug 2026 16:45:56 +0800 Message-ID: <20260825084608.47437-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_nr_vmemmap_pages() can account ordinary sections and DAX sections, but section-based vmemmap optimization keeps its compound order in struct mem_section and retains a different number of vmemmap pages. Teach section_nr_vmemmap_pages() to recognize section-based optimized sections and calculate their vmemmap page count from the section order and the HVO retained page count. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v5: - Collect Acked-by from Mike Rapoport v3: - Add vmemmap_optimizable_order() for order-based optimization checks v2: - Remove an unnecessary vmemmap_can_optimize() call to simplify the code (suggested by Mike Rapoport). - Rewrite the commit message for better understanding. --- include/linux/mmzone.h | 6 ++++-- mm/sparse-vmemmap.c | 10 ++++++---- mm/sparse.h | 16 ++++++++++++++++ 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index df31cac12311..177455d98064 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -107,8 +107,10 @@ is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 -/* The number of struct pages covered by the retained vmemmap pages with H= VO enabled. */ -#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (PAGE_SIZE / sizeof(struct pa= ge)) +/* The number of retained vmemmap pages with HVO enabled. */ +#define VMEMMAP_OPTIMIZATION_PAGES 1 +#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES \ + (VMEMMAP_OPTIMIZATION_PAGES * PAGE_SIZE / sizeof(struct page)) #define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STR= UCT_PAGES) + 1) =20 #define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 107215cf8488..b7abc5494bb9 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -649,24 +649,26 @@ void offline_mem_sections(unsigned long start_pfn, un= signed long end_pfn) static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - const unsigned int order =3D pgmap ? pgmap->vmemmap_shift : 0; + const struct mem_section *ms =3D __pfn_to_section(pfn); + const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); + const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); =20 - if (!vmemmap_can_optimize(altmap, pgmap)) + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); =20 if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return VMEMMAP_RESERVE_NR * nr_pages / pages_per_compound; + return vmemmap_pages * nr_pages / pages_per_compound; } =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); =20 if (IS_ALIGNED(pfn, pages_per_compound)) - return VMEMMAP_RESERVE_NR; + return vmemmap_pages; =20 return 0; } diff --git a/mm/sparse.h b/mm/sparse.h index 1fcda8a1c270..02ed0f34eac8 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -43,6 +43,17 @@ static inline bool vmemmap_optimizable_pfn(unsigned long= pfn) return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; } =20 +static inline bool vmemmap_optimizable_order(unsigned int order) +{ + if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + return false; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; +} + /* * mm/sparse.c */ @@ -86,6 +97,11 @@ static inline size_t mem_section_usage_size(void) return struct_size_t(struct mem_section_usage, pageblock_flags, BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS)); } + +static inline bool section_vmemmap_optimizable(const struct mem_section *m= s) +{ + return vmemmap_optimizable_order(section_order(ms)); +} #else static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D8613E6DD0 for ; Tue, 25 Aug 2026 08:49:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647767; cv=none; b=t6XSTVBWIoxJA543KrBrzFVLpALiHQkQco2c32NcE+3ik6RRHnKp5DqrLdKne3H9nx5OJQCpz3O57rkosnlgJLqERlG6fL4R6CiMrDNMtIcebvHa9BKEQ8bngLNXIbZyEuscGailZrGzuDbd72yKzgsC7aP5ojA+s8Q50446t5k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647767; c=relaxed/simple; bh=lRAIutfPJeAA4GuMAEZyF7UojBHeeZ0tJR0F1g55wQ4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b5hiWwyyWxszeZDL3dH8RnkXwSZC6WxZQIenrJKn0AlylcA6tE7pmxAIgPxzNOiieGLOfUJDSe2MSlNcCazSklysXwme31Hx4Pex3Z70hc9d5Jkh8ChLYmLQyPEIepRnP7D6kQYKFkwnOiPneEITRIwpVDv+TAPBA30d+xTMZik= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=h+dp73qQ; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="h+dp73qQ" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-ca88130e09aso2726759a12.3 for ; Tue, 25 Aug 2026 01:49:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647765; x=1788252565; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dh+RdUPU8gwH1ekKZyjiyjNcRLEH/6mbqsdeji2hvCw=; b=h+dp73qQ2ShgNjMT6xgTMhDNXxw7kUImCdFwGRku0RC6YO2PsYOD/s+b8vlf4LkEfH eA9B9dMvaOHnlYreFQ5O5FaeEK+DwWcaU9sxkHZ/dUm0mOfnLOCtQeWoQfgVcMagrYDZ SlGORXKHFtZUnAzgvlP3cc8okkXsTQ65BotIZNpFfIJ/YMWu9tKv/9UBUVjCtwsVoS+d W3DcuhryfvUDDGrW9Z+Q06Bc3qK2XQYq8taSHA/QlQFV5yFswZXd0RzW8mmv+d375fMY DyxxYwPCgYlixO5DtyOCkqsROHwnfn9mx8AGCet9KdMKcToxHO9BHvqwCzCCLCGDyCoI 2W4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647765; x=1788252565; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=dh+RdUPU8gwH1ekKZyjiyjNcRLEH/6mbqsdeji2hvCw=; b=OVNlrifIir5d/qTZDjjWljOlKwqolSvu3uqwjdZRjEyO1BPDZ65V2Rqke9MzipitHz yFntqYdu/az7lVkDZZj7imi2b8dDXfY2PNPoZ++Bzu3Xn4axVzcRDitSqoZGl13ISpH0 kliYOSOHnqhoMxi7Z5mpyv8Pr/h+z4MWYdmMhxIfLBz+zd3tkzdfk/6ICqsmqYbaZqvc patDGkLrVJRbsSW1Y3siB/IfE2g3l3NIpimwVrHN1qTZgQTVRvRZdv+oCZndix5eqlu5 VX1YjzoXIoR2ReLm8s1wEpsvWO+q85F9MRGo/7xHmgRjvV0aw7iMw1IyTRXrl9vc9doS pQng== X-Forwarded-Encrypted: i=1; AHgh+RqI/pXf3y/+R/cRVvGoKamInzQUQMc8R+Xsn4hKv4H1+Up9wHNd3Kl2ukIkoS9WwXuFucyjals1QU6ZL6A=@vger.kernel.org X-Gm-Message-State: AFuF++mBe9PCKUDjiMY8UovZ+N5LR7x9Tmv4t4erRn9HX6iycrh/e0of LF4zBWH7Zc/yMJ8LmtE0+u71wFKyXQofY8s8n/dkmHueTIHWo8S5GWBuD3yBfF1wTsI= X-Gm-Gg: AR+sD1215xBafKoraGWKYvChwl5W1kzoK4kLrJFjs5rwXonSPSC60197N6yEHaE3Hix 8c9jvCzEwr35lNTe8Sm1e6Z67wleTLh/vpvtM/EV+O221zr7XI+99IE7qO6ev/8p/51Z9nazIy2 SL8bL0cs/UgD9kS63GIwJboQvTpImHmPOhlD2iy2aHYtlky1dAfava3ys0JdpmJb2V8WZ4KS3Z9 WbJSPuksPV4+r8nPcXOBWRCEJ0h1b0xvY+iUY29DHM941HY517PdoDfTBUzj5WHyJ3yMZDUMAi0 nIVEjWU8kgqUj2HwlHoqOE1F3c5i//PaqiYQZlWbPkLXeJEqryQKh/zdym7ZGZ2wJVaPycZAjcv G6ITpFOpI8tDVNOQPeaYBY0kPDlibQhXhN4V+VNibYIrPkw6Wg0Y1BGXQM6Zs6YNmk8VUw3BrI9 f0IDGTh/pfzBe++rLn+lEl4KVhOqtGEGdXglGS3F+sVIrenw2FCATIB+l4wcuBmr86bHBVDECeL FCK/c39VRLo0dwVcSLvx6WdHA== X-Received: by 2002:a17:90a:e7c5:b0:36d:9e0b:3801 with SMTP id 98e67ed59e1d1-395c34f47cbmr55313227a91.8.1787647764791; Tue, 25 Aug 2026 01:49:24 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:24 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 06/17] mm/mm_init: factor out pfn_to_zone() Date: Tue, 25 Aug 2026 16:45:57 +0800 Message-ID: <20260825084608.47437-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" pfn_to_zone() in hugetlb_vmemmap.c duplicates the zone lookup logic in __init_deferred_page(). Move it to mm_init.c, declare it in mm/mm_init.h, and reuse it from __init_deferred_page() and HugeTLB early vmemmap initialization instead of open-coding the zone walk there. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v3: - Fix the commit message to name mm/mm_init.h instead of mm/internal.h - Collect Acked-by from Mike Rapoport v2: - Move this preparatory patch before the sparse-vmemmap optimization changes (suggested by Mike Rapoport) --- mm/hugetlb_vmemmap.c | 17 ++--------------- mm/mm_init.c | 28 ++++++++++++++++++---------- mm/mm_init.h | 1 + 3 files changed, 21 insertions(+), 25 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index ae8fdaa42118..c48fcea076a5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -19,6 +19,7 @@ #include #include "hugetlb_vmemmap.h" #include "internal.h" +#include "mm_init.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -744,20 +745,6 @@ static bool vmemmap_should_optimize_bootmem_page(struc= t huge_bootmem_page *m) return true; } =20 -static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) -{ - struct zone *zone; - enum zone_type zone_type; - - for (zone_type =3D 0; zone_type < MAX_NR_ZONES; zone_type++) { - zone =3D &NODE_DATA(nid)->node_zones[zone_type]; - if (zone_spans_pfn(zone, pfn)) - return zone; - } - - return NULL; -} - /* * Initialize memmap section for a gigantic page, HVO-style. */ @@ -787,7 +774,7 @@ void __init hugetlb_vmemmap_init_early(int nid) map =3D pfn_to_page(pfn); start =3D (unsigned long)map; end =3D start + hugetlb_vmemmap_size(m->hstate); - zone =3D pfn_to_zone(nid, pfn); + zone =3D pfn_to_zone(pfn, nid); =20 if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) diff --git a/mm/mm_init.c b/mm/mm_init.c index f801fd486085..0317fb781e77 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -692,6 +692,20 @@ static __meminit void pageblock_migratetype_init_range= (unsigned long pfn, } } =20 +struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid) +{ + pg_data_t *pgdat =3D NODE_DATA(nid); + + for (enum zone_type zone_type =3D 0; zone_type < MAX_NR_ZONES; zone_type+= +) { + struct zone *zone =3D &pgdat->node_zones[zone_type]; + + if (zone_spans_pfn(zone, pfn)) + return zone; + } + + return NULL; +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) { @@ -750,20 +764,14 @@ defer_init(int nid, unsigned long pfn, unsigned long = end_pfn) =20 static void __meminit __init_deferred_page(unsigned long pfn, int nid) { - pg_data_t *pgdat =3D NODE_DATA(nid); - int zid; + struct zone *zone; =20 if (early_page_initialised(pfn, nid)) return; =20 - for (zid =3D 0; zid < MAX_NR_ZONES; zid++) { - struct zone *zone =3D &pgdat->node_zones[zid]; - - if (zone_spans_pfn(zone, pfn)) - break; - } - __init_single_page(pfn_to_page(pfn), pfn, zid, nid); - + zone =3D pfn_to_zone(pfn, nid); + __init_single_page(pfn_to_page(pfn), pfn, + zone ? zone_idx(zone) : MAX_NR_ZONES, nid); if (pageblock_aligned(pfn)) { enum migratetype mt =3D kho_scratch_migratetype(pfn, MIGRATE_MOVABLE); diff --git a/mm/mm_init.h b/mm/mm_init.h index 39f75df9be1c..c9fc35e7e9f1 100644 --- a/mm/mm_init.h +++ b/mm/mm_init.h @@ -39,6 +39,7 @@ void memmap_init_range(unsigned long size, int nid, unsig= ned long zone, enum meminit_context context, struct vmem_altmap *altmap, int migratetype, bool isolate_pageblock); +struct zone *pfn_to_zone(unsigned long pfn, int nid); =20 #if defined CONFIG_COMPACTION || defined CONFIG_CMA /* Free whole pageblock and set its migration type to MIGRATE_CMA. */ --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5B64380FF3 for ; Tue, 25 Aug 2026 08:49:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647771; cv=none; b=qitoP/H3FSLJ60GqBSyqNbUUQyN30c5PQANCr4361tckTtWAqyWnsbLPj3/rZHDe0UlNd+IHyz5ePl2Ysl+XGlGZVNWc/dwJVQhqa/IjB63MjeGuUjeHuSAwVdkgujNSfWSA0JEcMCGA90dE+5wU3JUoASE8wrcZ33v8ao3VLC8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647771; c=relaxed/simple; bh=Iyxq8Ikh8WeBgPx7cbPiyj4yf2E4P7tEx7zbUC9PLEw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ipRsPdMLcM1hZPb2irrFBoRwtwkUKBm+oh6rarV1w2N98KTLS+oCAPnDsyneOXrGj+wtTGsFBuueuvWe657k99JGVxaVF49hwykQktMLxXFk1OBC4BMjjQM0YiP4o8SOCw3nP9MUjHegmU+qzGXifR4tg+LFTGmSrKGD+Mw5jVw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=TeHCzDpp; arc=none smtp.client-ip=209.85.215.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="TeHCzDpp" Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-cbee3777e1cso2778644a12.3 for ; Tue, 25 Aug 2026 01:49:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647769; x=1788252569; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SRpH2SWiMFBBCX+kPTKuY0N/Kvw39O9UHNj6BUBL9rc=; b=TeHCzDppTEOyoOP+nm2LhOxYRCKp3nkgrOcAG1Ion9IC9SUNgFArA445uFgqGnTY+T jF2mNfLACODh3sbv0D6JxqFMex9LNHa2qgNC+KKPa3qJ7z6AXzeFIz2+oTDC10XKaDeD 0al7sIwX2clzIp3b5EtvHSBEUBQtRYeY50qg17KHOsPnbBAb02m13fi/vuxqkQT77Owx UUxXlrXnv8UyHKbRxpC1q5RkEY8NQMDVVJNkY6W8S0Vw3E+SS05KF0H6ivgW3MRXbPxM dKfDmhRp8ztGLr0a0hxHvESw1wKjBLIWDZnS43Li4vJ95R6gAHyNiWbtdYXDJFKZWK6n K29g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647769; x=1788252569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=SRpH2SWiMFBBCX+kPTKuY0N/Kvw39O9UHNj6BUBL9rc=; b=nVxb34raehKXcBD/mun30fcMrRhoJa9YPNXOIpbXd996M/xvWmbzZF95nSE2jzKHVq BzfRHuXwS9G6MVmjiywtnh2abGgUxg9QKR9uYUieiVX+75lF4B+BeIha80oyffK0CasK 4k0xLUqNIl61asqkHp1MGxuKOReHyoDJ9vc4eXNlM24zECi8r/EgD9wQtmwFo2DcVbW4 CLHiRfj9nA1KI8rF1BwesxzMoAmeCNf3NoHerCqkLKzlXZD4+KQD3UbDV6yKrN4ND0Ki 9DtbutP05oeiyMz31TzR2CrwgSpKggBP8z65QGdupCgm1P8vKLHqk0DeHwBmuyRnYjNl oYtQ== X-Forwarded-Encrypted: i=1; AHgh+Rqrca80FLw8EqnJ7mxosbAU4liR4m7THbRVcZ0EZTQbGuVcCwRsTgHTbmCLJ2wbzYe7owLAT5Mwdehu3wk=@vger.kernel.org X-Gm-Message-State: AFuF++nypli+W5GLHCYhpvz5D6qV6jkmk+Fe+mA1O0m6USrd9X7mStVu BFdJrCSM7QsX3XiQ6ruIJoQ/+avxhfWL1RrTqLbKjtXKF6rT+xgp5QHrUFwHHUZ6zy9p7z75pxG JEOVCuAk= X-Gm-Gg: AR+sD10FUpJnv9ZCdypwnTHTlNYq72Li1L0Zw7av8SUE949/yQIdapi9UuvuaTGKYYu Mwr63HzMAmZWUeXmUKNz9ES4AuLF3KfkBB9kqW+rp7eS79mTi0lfRqrqQjPm+Z8YiOEk0OAZfGg bXLATW6t0VeddaQQGCIv6A7/hPiWVIBYrANV70cqKgfZ3YTBZt0s1TW0uCecqA7xwLr2mGvJt6P XqW7Fuc5BVPe9pjmTHMx/NiCxQKuoc/jOV4o8+q/wt67YHhvtmE/0wHvt2JOPKqYLOKEwA3Ybmn GuqZ6dvHqF592ZM2MUgNc9VIh2lDObvT8BZxxtilhi5Zy9dOCBxZkpiRprESR7a/t8b07OGA2n7 5KPH+xlJtLFIMvCWp8w3/0CHzIfV0DbEQSvQmih6jNvlKLIGa7KDdBv3NaSnJXWaDk1IsSb0F2v L6/6RG/E72EE0S5SIZVKkmaJclvAJRhbdvAvmfdXNYQuv3dSifBz/ojcdbHpIoQ5h9TTmQ/MqO0 BAZ5TBwInPCLNqz0kPqgyiG+Q== X-Received: by 2002:a17:90b:53c7:b0:38e:5c6:4db9 with SMTP id 98e67ed59e1d1-396464a15f2mr10522868a91.11.1787647768873; Tue, 25 Aug 2026 01:49:28 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:28 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 07/17] mm/sparse-vmemmap: move helpers ahead of future callers Date: Tue, 25 Aug 2026 16:45:58 +0800 Message-ID: <20260825084608.47437-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Prepare for section-based vmemmap optimization by moving helpers that follow-up changes will use. vmemmap_get_tail() will be called from the PTE population path. section_nr_vmemmap_pages() will be made visible outside the memory hotplug code and called from sparse_init_nid(). Move vmemmap_alloc_block_zero() together with vmemmap_get_tail(), since the tail helper depends on it. Move section_nr_vmemmap_pages() earlier into its own CONFIG_MEMORY_HOTPLUG block. That lets the later patch change its visibility and callers without also moving the function body. No functional change is intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v5: - Move section_nr_vmemmap_pages() earlier under its own CONFIG_MEMORY_HOTPL= UG block to avoid code movement in the later visibility change - Reword the subject and commit message for the helper movement - Collect Acked-by from Mike Rapoport v2: - Add this new patch to move vmemmap_get_tail() before PTE population (suggested by Mike Rapoport) --- mm/sparse-vmemmap.c | 134 +++++++++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 65 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b7abc5494bb9..ea3cacec0a79 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -148,6 +148,75 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 +#ifdef CONFIG_MEMORY_HOTPLUG +static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + const struct mem_section *ms =3D __pfn_to_section(pfn); + const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); + const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; + const unsigned long pages_per_compound =3D 1UL << order; + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); + VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); + + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); + + if (order < PFN_SECTION_SHIFT) { + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); + return vmemmap_pages * nr_pages / pages_per_compound; + } + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); + + if (IS_ALIGNED(pfn, pages_per_compound)) + return vmemmap_pages; + + return 0; +} +#endif + +static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) +{ + void *p =3D vmemmap_alloc_block(size, node); + + if (!p) + return NULL; + memset(p, 0, size); + + return p; +} + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) +{ + struct page *p, *tail; + unsigned int idx; + int node =3D zone_to_nid(zone); + + if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) + return NULL; + if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + return NULL; + + idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; + tail =3D zone->vmemmap_tails[idx]; + if (tail) + return tail; + p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); + if (!p) + return NULL; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail(p + i, NULL, order, zone); + + tail =3D virt_to_page(p); + zone->vmemmap_tails[idx] =3D tail; + + return tail; +} +#endif + static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long ad= dr, int node, struct vmem_altmap *altmap, unsigned long ptpfn, unsigned long flags) @@ -181,17 +250,6 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *p= md, unsigned long addr, in return pte; } =20 -static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) -{ - void *p =3D vmemmap_alloc_block(size, node); - - if (!p) - return NULL; - memset(p, 0, size); - - return p; -} - static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long ad= dr, int node) { pmd_t *pmd =3D pmd_offset(pud, addr); @@ -323,33 +381,6 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigne= d long end, } =20 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) -{ - struct page *p, *tail; - unsigned int idx; - int node =3D zone_to_nid(zone); - - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) - return NULL; - - idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail =3D zone->vmemmap_tails[idx]; - if (tail) - return tail; - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) - return NULL; - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - tail =3D virt_to_page(p); - zone->vmemmap_tails[idx] =3D tail; - - return tail; -} - int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, unsigned int order, struct zone *zone, unsigned long headsize) @@ -646,33 +677,6 @@ void offline_mem_sections(unsigned long start_pfn, uns= igned long end_pfn) } } =20 -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) -{ - const struct mem_section *ms =3D __pfn_to_section(pfn); - const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); - const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; - const unsigned long pages_per_compound =3D 1UL << order; - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); - VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) - return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); - - if (order < PFN_SECTION_SHIFT) { - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; - } - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); - - if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; - - return 0; -} - static struct page * __meminit populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D74E3876CD for ; Tue, 25 Aug 2026 08:49:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647775; cv=none; b=TCqrJ/MEdVxujg7hGV35iqP+2P/ozu573S3AoHhVmpLLuZ7AmF8nBVXLiwzJQFHlxjToOodiPKtqDhAHeGEsxywm8Xj1xaZZFSg0DLhl7BxbMLB83U6lkp1Zh5ACUnaV8AqcCBsvQNGGFuSpDyXPivjG+V8RaCsbk4r0NeirpzU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647775; c=relaxed/simple; bh=dLHlMfaiHdj3XmI9tbMYv6OR0zj0dBKppqdrAfS2FM8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j9xt3Dzs4Ip/AsB6gSAkWWmUuTVrNh/5NZVRuHE1NK+UxCg9mYOYYEAXk1mvA2Exn0CU/Ek4aKuDBwSio0DJ0UpYsNQFM+UJfggKE3sQQxvy41+DtlZclEJ9XYnAoNeKQaOgbglBgpwTKAMD/e13mrs+h0v7A7Zrd2l0PAF8BjA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=i17LGigc; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="i17LGigc" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-38e58034d05so4013681a91.2 for ; Tue, 25 Aug 2026 01:49:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647773; x=1788252573; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=JHukc19Ma/AZahsqFHAmhIDbE4H1sWhtblWFu+eIVFE=; b=i17LGigcKx2JqooETyFb4RdqVfynnLrc1XqnC3ligtfuOqfgm/xTXUUBdPEJhjOol0 SxmKxySBFossC58gn27j6Dbd3WG0p+V/In9AZ931Wt7i6M0mzOXdamFnq95f9NUeJcep iHquXgGHAPvP8wkwRb3XRu9Md7eTWUVBnqMXpkqkwbnPxYx2SdmlqeGI6yXZUSrppciD CZOMUqwhmM+RyADangPhSwGb3iIuTni6+dfrjD4jxXSFO7tQNyV/Tda/Q84dngxdMsvB ObT6EfNC0Sg8a1UCMliWK4a7JGxQEV7aFaXJf3WQnsV/3jxQLDTAWujK1kHFSjZRRovC 6wmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647773; x=1788252573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=JHukc19Ma/AZahsqFHAmhIDbE4H1sWhtblWFu+eIVFE=; b=PR+8Xr1hEA9C67h5sGlJEIjsAJwb3Nqr7PSK3TQ4200pWnb+39luiHwrsoOsElb/GE 2z1yfptIOL6FACL/5XKnDNnIx7fyHRwMk6msHZH7E3Bo5EOqJvwW2PkKs3Yd8IlHnVoF vGyPbf/j0mUPlbjMCUwWKDdyrhHsegVHWD37voz6SO0Igq9pQbcXajJc1HmxXQvgJJj3 bbKkMFLcnx76pXj1yWlA6WWpn20X91cng2iO8LIvYQHWXScM3n9XMgPT2MYBeQ77GVDn 10zfBwQ0/jCiHoeQddGsoahbGajfuzPRVRAeQqliJppeh1nqUTlYJnQZhBV/DYG+oXGN v5cQ== X-Forwarded-Encrypted: i=1; AHgh+RoZLqnVXTTOAAphMRHg5iA/PFgO/26l8ZUUdApxq0Jss6UTJdvhqw57Kn0gTL9lQ/PJnP6FOCXy5UQrzkM=@vger.kernel.org X-Gm-Message-State: AFuF++mmewH2MIpdTelHClFAlXy+tDMbmu8s+nZSjUN0Q2vPYwkWQR76 8LWBjQv+im3Euxobqy91QGW4ljT5js567mUI50ls/kYUQwEO6JZIlk5Aw3Fbf5S8+so= X-Gm-Gg: AR+sD131S25WiNlIj6+yA4aXh7ZGzFh4lZsMS4R2A/LzpIzN9BfhlFzGFxXhKncBHUQ j+akiFeuQTKYrf7C9ex+AWKJOtmKXwnCUgandgp5BTuGoDQTJzdENZvJOv5Vr1e/MOcZFh2irzm zyZDTaQmyVhLVyzC2q6m/GD4kwMNXvd7sAM/7D3Z6UuvVUB/RDn50tPD59yrFbKgQp04U2yQ3R0 OpVL82afVpLTLnzJ0a6Yj/gAuEHMDJCiiYYcx8UNZlAkPJL6q4XQ+7PUEgu2hvE1XHxLyP8lZJa 9mPlw/WKsoG/5c0rVE1bkhrP6m/0xjn3QCFOE7kAbu6tURB+sJYD1/czIOlfWx8VRM/9MfmVsY/ qFLKVc4hW5+Y0tZIyQMIBqU8Lq2jA18HOvFbirFCYB58LGUMCWkGK6rLmkPdN3P5SN33SfT5Yhj 4LH58a82aD6NgzNzrjivCp6MgGiT8xjJuK2+iKAOqhCzW7/OOmd1eNmQC8JMg3I/0TnTPksRZVK 8uFxje4t0Dfq7wKWRBXLGSbug== X-Received: by 2002:a17:90b:558b:b0:37f:fdc8:71b4 with SMTP id 98e67ed59e1d1-395c33371a9mr62072248a91.2.1787647773077; Tue, 25 Aug 2026 01:49:33 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:32 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 08/17] mm/sparse-vmemmap: support section-based vmemmap optimization Date: Tue, 25 Aug 2026 16:45:59 +0800 Message-ID: <20260825084608.47437-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach sparse-vmemmap population code to use the compound page order when deciding whether a vmemmap page can be optimized. With this information, the common sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages directly instead of relying on HugeTLB-specific handling. This centralizes vmemmap optimization logic in the sparse-vmemmap code, based on section metadata, and prepares for sharing the same mechanism across different users of vmemmap optimization, including HugeTLB and DAX. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v5: - Add vmemmap_alloc_pte() to keep PTE allocation handling clearer (suggested by Mike Rapoport) - Move preparatory section_nr_vmemmap_pages() changes to the previous patch (suggested by Mike Rapoport) v4: - Move section_nr_vmemmap_pages() outside CONFIG_MEMORY_HOTPLUG to fix CONFIG_MEMORY_HOTPLUG=3Dn builds (reported by kernel test robot) v2: - Keep vmemmap accounting and population logic in sparse-vmemmap.c (suggested by Mike Rapoport) - Move vmemmap_get_tail() before its first use instead of adding only a forward declaration in the previous patch (suggested by Mike Rapoport) - Simplify the PMD path handling for HVO-covered sections --- mm/sparse-vmemmap.c | 46 +++++++++++++++++++++++++++++++++++++-------- mm/sparse.c | 4 ++-- mm/sparse.h | 7 +++++++ 3 files changed, 47 insertions(+), 10 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index ea3cacec0a79..8d39196f6d93 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -148,8 +148,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 -#ifdef CONFIG_MEMORY_HOTPLUG -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms =3D __pfn_to_section(pfn); @@ -175,7 +174,6 @@ static int __meminit section_nr_vmemmap_pages(unsigned = long pfn, unsigned long n =20 return 0; } -#endif =20 static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) { @@ -215,19 +213,44 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * =20 return tail; } +#else +static inline struct page *vmemmap_get_tail(unsigned int order, struct zon= e *zone) +{ + return NULL; +} #endif =20 +static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, + struct vmem_altmap *altmap) +{ + struct zone *zone; + struct page *page; + const unsigned int order =3D pfn_to_section_order(pfn); + + if (!vmemmap_optimizable_pfn(pfn)) + return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + + zone =3D pfn_to_zone(pfn, node); + page =3D vmemmap_get_tail(order, zone); + if (!page) + return NULL; + + return page_address(page); +} + static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long ad= dr, int node, struct vmem_altmap *altmap, unsigned long ptpfn, unsigned long flags) { pte_t *pte =3D pte_offset_kernel(pmd, addr); + unsigned long pfn =3D page_to_pfn((struct page *)addr); + if (pte_none(ptep_get(pte))) { pte_t entry; - void *p; =20 if (ptpfn =3D=3D (unsigned long)-1) { - p =3D vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + void *p =3D vmemmap_alloc_pte(pfn, node, altmap); + if (!p) return NULL; ptpfn =3D PHYS_PFN(__pa(p)); @@ -246,7 +269,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in } entry =3D pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); - } + } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) + return NULL; return pte; } =20 @@ -435,6 +459,9 @@ int __meminit vmemmap_populate_hugepages(unsigned long = start, unsigned long end, pmd_t *pmd; =20 for (addr =3D start; addr < end; addr =3D next) { + unsigned long pfn =3D page_to_pfn((struct page *)addr); + const struct mem_section *ms =3D __pfn_to_section(pfn); + next =3D pmd_addr_end(addr, end); =20 pgd =3D vmemmap_pgd_populate(addr, node); @@ -450,7 +477,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long = start, unsigned long end, return -ENOMEM; =20 pmd =3D pmd_offset(pud, addr); - if (pmd_none(pmdp_get(pmd))) { + if (pmd_none(pmdp_get(pmd)) && !section_vmemmap_optimizable(ms)) { void *p; =20 p =3D vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); @@ -468,8 +495,11 @@ int __meminit vmemmap_populate_hugepages(unsigned long= start, unsigned long end, */ return -ENOMEM; } - } else if (vmemmap_check_pmd(pmd, node, addr, next)) + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + if (WARN_ON_ONCE(section_vmemmap_optimizable(ms))) + return -EOPNOTSUPP; continue; + } if (vmemmap_populate_basepages(addr, next, node, altmap)) return -ENOMEM; } diff --git a/mm/sparse.c b/mm/sparse.c index c84b4c7b8c70..e6cb67ca9c8d 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -305,8 +305,8 @@ static void __init sparse_init_nid(int nid, unsigned lo= ng pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct pa= ge), - PAGE_SIZE)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); sparse_init_early_section(nid, map, pnum, 0); } } diff --git a/mm/sparse.h b/mm/sparse.h index 02ed0f34eac8..e427d72e9c2d 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -111,8 +111,15 @@ static inline void sparse_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else static inline void sparse_init_subsection_map(void) {} +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); +} #endif /* CONFIG_SPARSEMEM_VMEMMAP */ =20 #endif /* __MM_SPARSE_H */ --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAD0E3DB337 for ; Tue, 25 Aug 2026 08:49:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647779; cv=none; b=iA7Pi581iIi9ttQXWdutDHu3558R7lZ74drRLH08BOoYE2+Bs+7/G1EqW/UKyv7FSz8IqtSLmhVERqpx7xxYL+VTefpBAHJDpa/laUDuiIE0c56fexzGRROC1jcsn/G48ccOc4kr7UsMv1Lxq5GWnMcfxD/w3BSj6siQbKfopz8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647779; c=relaxed/simple; bh=F80hkuMMAs3U5OLObS//LlfPV7G6rtcP+b9Xn2vU3u4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H5qslQLpJGKQJZvtLtx4/Kt3uhsM9CMuVXu0/iS+3Ow7cgahg02SCM/iPlAOR1YBM5246gy33whngVRzfSyQrYoC6J9CFL6q0IFSDjlrBwpEDJkj/YPiEFOiZZ8I+8d8rhR4ScPa7nuBHQFXv+7Z+mFyllMNjPk5XfBQal1Zgtc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hmSv6AYU; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hmSv6AYU" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-3856d6fbcb3so4055147a91.2 for ; Tue, 25 Aug 2026 01:49:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647777; x=1788252577; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vfVcATb8qZsBfkQbzRHSDHUI/BdsmgHAfHU8Cx/Oa8M=; b=hmSv6AYUXXuhiP0OrqbJlsE3elutetBpOSg2S9OcNtdoLgf/q3ObuV2+Q3a2F8eGoH 26Wjaa7mEj5FuzGPiB4ffr06NQsD+fVXc3lIUFAfaCZVqj1aezlZ4/xO54F23lkHrlb9 Ge+8H6lDBDaMBIvfTvYcL6tCttBS13usSnVTFHFv9S76K8iFnl4Hipq+rxMiGuJecZPQ JDDAQL/OJ+p/md10syAvO4oS/gbmjkQvGLPvR1UAtqaD1B+glQAn6LolYv5wPtmvhDP3 VMH6dAkRah5axa3ErY82hFJGmx7a2RKmCWWGufzLZgvJvhUkQhRgiRRXYJJv6txaWVno FCvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647777; x=1788252577; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=vfVcATb8qZsBfkQbzRHSDHUI/BdsmgHAfHU8Cx/Oa8M=; b=bkojz9F0D4SqdgZsXYR72pVyOPqqcrQ8XNTS/ZpwJfP1nKgdXs28J+Umn/kYF7dwXE Q0zIQEjFKFH4ANYIvm7/vrWFJDstgpVHPlcgo1BAulQjRPGPWADZfvQoWuRnfgRLvgRD wMcSL0hariCuWOaGo217dwprxlUgWJbwnrLPY6hcluW2IQ6xyst6Hc1H6uf2vG30QJpJ MwPcVtoRtFBQrrMsYchbYXuD9Qx8dcgSS5YeDBsyhrH+PkJoa5p8y1Imzyrllzo+oX9l 5D8RPUAD5dBswvyfmzDYlQHuSB/MvVMoCWKcUwerhuV6bkwou3ZqhFeEwZD/qH7FZNiM z7Vw== X-Forwarded-Encrypted: i=1; AHgh+RrLbNpoYVI3jYhbOCN/tOMXmLciGhDJ9X0s+MdErp5P3nphdo+ob0Ndz2ycq4v9VFuLbrpTu7+foWJd5WU=@vger.kernel.org X-Gm-Message-State: AFuF++l3MHPgI3O7wKJhqaxK+mWKebGgQ+egWwsftc9i7vZw4Md6zPaK hlXIq0eWhkj6SlaXHjaYer7Y8viDd+D62mVwuLQ58/ng4mTxxw2pDZPcBZs80h3ksks= X-Gm-Gg: AR+sD13nHy3E55RAs0jv3xw9LVlK0NCchhNin0uEMwIHK5rk9Yl3NRcnY5gT6Hg1QRe y0MZhdskPZqgK4+twS1yTpnY/pYziP2+xh/1SaeYBXceOpopWJMlBz7gunX1fv/KFlh1cf9XRaA yS/F8UXOfOyz+0wy9K/6m+5cjmhoowcVx15R600Oz+m2tfNzk7WY+12Tza1YwmhzcB7dHGwqj15 qYRxVE+PppakAI8dIDsI4wNMIJOLQhgkmnr+QBVlGQXFlTrgdOpeSdzaR49pbzOSyTOfgo4Pxwo frYUN0uaJN0sdaj76ywGyF7IdKWGZnxpRkQfTl6ogU+v9MIvBVK98TmFNSntnwcb5JrGwbQHHR9 midC51rmHPB07edavwezuE8dqZCUp3B26pdBHqQbsJWHJiCAjgW5tOdjhwxCsnTPfWKiFFtWHyl nhaVp5S6XLgs75qlclaKVOcg00sQ0DIsQ1KnvXxE6LiiUZKYWH0dYJIgKGdEzcRygjpCdEKWi+s MZbIGmfPTpefJiHpBUrjk36Jg== X-Received: by 2002:a17:90b:524c:b0:366:10f1:3d91 with SMTP id 98e67ed59e1d1-396462b6015mr10760110a91.1.1787647777141; Tue, 25 Aug 2026 01:49:37 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:36 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 09/17] mm/sparse: initialize memory sections earlier Date: Tue, 25 Aug 2026 16:46:00 +0800 Message-ID: <20260825084608.47437-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Upcoming HugeTLB bootmem changes need sparsemem section metadata before the HugeTLB bootmem allocation path runs. The memory sections are initialized from sparse_init(), which is called too late for that setup. Move the code that initializes sparsemem section metadata for memblock ranges into mm_core_init_early(), before free_area_init() and the HugeTLB bootmem setup. Rename the helper to sparse_sections_init() so the new caller describes the sparsemem-specific initialization step. This is a preparatory change. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v5: - Collect Reviewed-by from Mike Rapoport v2: - Rename the helper to sparse_sections_init() to describe the section metadata initialization (suggested by Mike Rapoport) - Fix the !SPARSEMEM stub name so SPARSEMEM=3Dn builds compile (reported by Sashiko) --- mm/mm_init.c | 1 + mm/sparse.c | 10 ++-------- mm/sparse.h | 2 ++ 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 0317fb781e77..e2a16d833635 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2642,6 +2642,7 @@ void __init mm_core_init_early(void) { kho_memory_init_early(); =20 + sparse_sections_init(); free_area_init(); =20 hugetlb_cma_reserve(); diff --git a/mm/sparse.c b/mm/sparse.c index e6cb67ca9c8d..439802e6a6ad 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -191,12 +191,8 @@ static void __init memory_present(int nid, unsigned lo= ng start, unsigned long en } } =20 -/* - * Mark all memblocks as present using memory_present(). - * This is a convenience function that is useful to mark all of the systems - * memory as present during initialization. - */ -static void __init memblocks_present(void) +/* Initialize memory section metadata for all system memory. */ +void __init sparse_sections_init(void) { unsigned long start, end; int i, nid; @@ -322,8 +318,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count =3D 1; int nid_begin; =20 - memblocks_present(); - if (compound_info_has_mask()) { VM_WARN_ON_ONCE(!IS_ALIGNED((unsigned long) pfn_to_page(0), MAX_FOLIO_VMEMMAP_ALIGN)); diff --git a/mm/sparse.h b/mm/sparse.h index e427d72e9c2d..e4617f9c8876 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -59,6 +59,7 @@ static inline bool vmemmap_optimizable_order(unsigned int= order) */ #ifdef CONFIG_SPARSEMEM void sparse_init(void); +void sparse_sections_init(void); int sparse_index_init(unsigned long section_nr, int nid); =20 static inline void sparse_init_one_section(struct mem_section *ms, @@ -104,6 +105,7 @@ static inline bool section_vmemmap_optimizable(const st= ruct mem_section *ms) } #else static inline void sparse_init(void) {} +static inline void sparse_sections_init(void) {} #endif /* CONFIG_SPARSEMEM */ =20 /* --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 141D33D3D14 for ; Tue, 25 Aug 2026 08:49:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647783; cv=none; b=Yq9Q6md82c4nZR3VgdtBx3eSoKs8XT7ROPBXdsG2bqO3NXDR0Mlvj9FLY2wpqsixmDt4755A91FlSxskY3vZO2oXbEcLso31MEd4W40frVx3XVju40BdVUzKR5GnPw3CHbM0gPUUpY2UzINJbqn055A93PPkiw77dCRQAHsmdPs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647783; c=relaxed/simple; bh=gK7d9R7oXek0AC1OSHnfglk4S3+IovecWCPe8drPbY8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H4bkk//rQnC8mowpRGwd2s84eKh441gT46n98fQsWq53Y2aT402wGQEoxaE6rLi+AvlahfBAj0qBjJm2ToYy/88ztlok814Ig/2qm2VIc8ww8349JgAkCxzlQTleGt/gq5INdi9Ei4fcGdGg5tpaXXK0O2zebnkQnJxVSGElJWM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=TKClg/4R; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="TKClg/4R" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-ca12086c06eso3448902a12.0 for ; Tue, 25 Aug 2026 01:49:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647781; x=1788252581; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BwgAWVwCNaVWIKBy7+zcDip0pYJw7K4FkYIqMzHRNkk=; b=TKClg/4RKFDLMRC9jVdZrhuvntmVYn14h9jKYfeUE0lzQRZ3zh/sfL/NoaX2IerDy/ ITSDO8/1cbPHMDvraixvonw4ylmraVqvk8RYnaq12ReR+ImWa1JgxojjVYEvtUR203o9 Y9iDg6OhpKr3X6APkJAmUzGiD1siJ3IdSaLFqL0aDnc8NBjOIgoc3B88MRX8i56HzraM +HRDUyHNY7ShdEy0kYsWC9ed1wSmyvC4aWoID1+tegkP74WdY3Wq59n+niPPfA5BqjBy nBEYl8bvhofJXn22Xn7sL9GtxfTJR+dn6Z6sVO0XxvnLIW6DqH6yoDZcddvDt+JbNgYe rOFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647781; x=1788252581; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BwgAWVwCNaVWIKBy7+zcDip0pYJw7K4FkYIqMzHRNkk=; b=P4X256Vhu3xqpLqhWjAOH6sc1tSg4sJIFcKafdQ3NIKq5IltLUOW3kHvn7Md6xcyEj taLSrxsmSA9DBXNCq5MRcZhkYmpHKsxniS8hMRP83mDjBoo5qXZxjLQHGeYexIHjw7x8 OtWs067FnpB1txVZIxuwXUMVII5W4n/dTWGC+7Yye29xoFUdZq0pB7QfEAmcN3886vK2 3UI7SU3Yy8hPwsMl3hfNyvNPyNC6pb0H3UprOQ5H+JhHzLXlUOZ/3t6SfxmJwYu8sgHI CceZn3Hn0m5cAwpsx6ykarZMbTH19cQtlB5N7YKLDfotfLQJgHd6+U5XiRSY2oBLaSW9 8JuQ== X-Forwarded-Encrypted: i=1; AHgh+Rqp9ND0hSo6iXrkNrc0CF1AwrYOgA4ln6Y4CPK2f1cGz55T79aF4tuGBbQuheL7UFFroSZo1R5U7ila//U=@vger.kernel.org X-Gm-Message-State: AFuF++keAEXkZuUDgv0HLrA9DUwJ7FA/GyzNtzGgPf4Ad08qeGdjMm/P Gw9Tu7Qo5eUUnyKs/26nJFuPdxe+EefXDak8bxhW+rAaWdhtsct1KXgvkSMxnDftMCQ= X-Gm-Gg: AR+sD11jQSZmZW3QqYUjJSMYe9+vcD93QchrT+dkaJFcCmzuTyAp96zlCgyh9Hpbd4Y /0RJbYR7GcrJbZdG20qFun4FBOKpanzSTsEkOnC9reJZS/bkMhf8FjoNCN8MZV8Wwr/nMDyBBc4 Tcfr+SBSvAFitxkMKSTX/P1+r/zUFB84DGkZY81d3M2Yuc9SXJlBDInvomsbV/c0qQpog5TMWG8 CSloV+6kvst2wh2zfBBqSYlnQLLmNmmJFG0If3QKHq86FHj4VHVgxNwIzxryV4xd924YvKIymWg u01xV2Nzo62waREwWWhCH44O29FXQG2MQ6stZYns1GRwIJAkEt9V85idmkyZByf8e0uWVusOhyF lsMZFZXacBZcboEQwwsURQ5StKC80D+VeJaXc6958cLiUERjVfh4BUOw+fYmTiwuC/Qd8zBk1oa KAmaxEbqRBIL8AWNghpazNmcu4rwK4mhrenMm/fXXU0uMwA/HyFyp81NRDn8JyKDwZyO/RtunHi dE7cn2SmZ8A8Nn1q5Eix5PvkfRBv1+eJQr1 X-Received: by 2002:a17:90b:4d0e:b0:38e:659b:f366 with SMTP id 98e67ed59e1d1-395decb87edmr35646834a91.0.1787647781257; Tue, 25 Aug 2026 01:49:41 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:40 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 10/17] mm/hugetlb: switch HugeTLB to section-based vmemmap optimization Date: Tue, 25 Aug 2026 16:46:01 +0800 Message-ID: <20260825084608.47437-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB bootmem vmemmap optimization still carries its own early setup path, including pre-populating optimized mappings before the generic sparse-vmemmap code runs. Now that the section-based vmemmap optimization can derive HugeTLB vmemmap deduplication from section metadata, HugeTLB only needs to mark the bootmem huge page range with the appropriate order. The generic sparse-vmemmap population path can then allocate and map the shared tail vmemmap pages without any HugeTLB-specific early population code. Do that by setting the section order when a bootmem huge page is allocated and dropping the dedicated pre-HVO helpers and related special-casing. This removes duplicate early setup logic and switches HugeTLB to the section-based vmemmap optimization path. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v3: - Use the order-based helper for the bootmem vmemmap-optimized check v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 1 - include/linux/mm.h | 3 -- mm/hugetlb.c | 30 ++------------ mm/hugetlb_vmemmap.c | 90 +++-------------------------------------- mm/hugetlb_vmemmap.h | 14 +++---- mm/sparse-vmemmap.c | 31 -------------- mm/sparse.h | 27 +++++++++++++ 7 files changed, 42 insertions(+), 154 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 16c4c4caa126..fe28f98e1b22 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -171,7 +171,6 @@ struct address_space *hugetlb_folio_mapping_lock_write(= struct folio *folio); =20 extern int movable_gigantic_pages __read_mostly; extern int sysctl_hugetlb_shm_group __read_mostly; -extern struct list_head huge_boot_pages[MAX_NUMNODES]; =20 void hugetlb_bootmem_struct_page_init(void); void hugetlb_bootmem_alloc(void); diff --git a/include/linux/mm.h b/include/linux/mm.h index dd09c438fa23..441bd39eab73 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5159,9 +5159,6 @@ int vmemmap_populate_hugepages(unsigned long start, u= nsigned long end, int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); -int vmemmap_populate_hvo(unsigned long start, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize); void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int nod= e, unsigned long headsize); void vmemmap_populate_print_last(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 04e6c4244cd6..fbb0c83bea79 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -52,6 +52,7 @@ #include "hugetlb_cma.h" #include "hugetlb_internal.h" #include "mm_init.h" +#include "sparse.h" #include =20 int hugetlb_max_hstate __read_mostly; @@ -59,7 +60,7 @@ unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; =20 __initdata nodemask_t hugetlb_bootmem_nodes; -__initdata struct list_head huge_boot_pages[MAX_NUMNODES]; +static struct list_head huge_boot_pages[MAX_NUMNODES] __initdata; =20 /* * Due to ordering constraints across the init code for various @@ -3139,6 +3140,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) } else { list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |=3D HUGE_BOOTMEM_ZONES_VALID; + hugetlb_vmemmap_optimize_bootmem_page(m); /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB @@ -3299,6 +3301,7 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) * this folio. */ folio_set_hugetlb_vmemmap_optimized(folio); + section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); =20 if (hugetlb_bootmem_page_earlycma(m)) folio_set_hugetlb_cma(folio); @@ -3342,31 +3345,6 @@ void __init hugetlb_bootmem_struct_page_init(void) .max_threads =3D num_node_state(N_MEMORY), .numa_aware =3D true, }; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct zone *zone; - - for_each_zone(zone) { - for (int i =3D 0; i < VMEMMAP_OPTIMIZATION_NR_ORDERS; i++) { - struct page *tail, *p; - unsigned int order; - - tail =3D zone->vmemmap_tails[i]; - if (!tail) - continue; - - order =3D i + VMEMMAP_OPTIMIZATION_MIN_ORDER; - p =3D page_to_virt(tail); - /* - * prep_and_add_bootmem_folios() can access pageblock - * flags on bootmem HugeTLB pages, so initialize the - * shared tail struct pages here before bootmem folios - * start using them. - */ - for (int j =3D 0; j < PAGE_SIZE / sizeof(struct page); j++) - init_compound_tail(p + j, NULL, order, zone); - } - } -#endif =20 padata_do_multithreaded(&job); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c48fcea076a5..7293706b532f 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -18,8 +18,7 @@ =20 #include #include "hugetlb_vmemmap.h" -#include "internal.h" -#include "mm_init.h" +#include "sparse.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -706,95 +705,18 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct h= state *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - -/* Return true of a bootmem allocated HugeTLB page should be pre-HVO-ed */ -static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page = *m) +void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page= *m) { - unsigned long section_size, psize, pmd_vmemmap_size; - phys_addr_t paddr; - - if (!READ_ONCE(vmemmap_optimize_enabled)) - return false; - - if (!hugetlb_vmemmap_optimizable(m->hstate)) - return false; - - psize =3D huge_page_size(m->hstate); - paddr =3D virt_to_phys(m); - - /* - * Pre-HVO only works if the bootmem huge page - * is aligned to the section size. - */ - section_size =3D (1UL << PA_SECTION_SHIFT); - if (!IS_ALIGNED(paddr, section_size) || - !IS_ALIGNED(psize, section_size)) - return false; - - /* - * The pre-HVO code does not deal with splitting PMDS, - * so the bootmem page must be aligned to the number - * of base pages that can be mapped with one vmemmap PMD. - */ - pmd_vmemmap_size =3D (PMD_SIZE / (sizeof(struct page))) << PAGE_SHIFT; - if (!IS_ALIGNED(paddr, pmd_vmemmap_size) || - !IS_ALIGNED(psize, pmd_vmemmap_size)) - return false; - - return true; -} - -/* - * Initialize memmap section for a gigantic page, HVO-style. - */ -void __init hugetlb_vmemmap_init_early(int nid) -{ - unsigned long psize, paddr, section_size; - unsigned long ns, i, pnum, pfn, nr_pages; - unsigned long start, end; - struct huge_bootmem_page *m =3D NULL; - void *map; + struct hstate *h =3D m->hstate; + unsigned long pfn =3D PHYS_PFN(__pa(m)); =20 if (!READ_ONCE(vmemmap_optimize_enabled)) return; =20 - section_size =3D (1UL << PA_SECTION_SHIFT); - - list_for_each_entry(m, &huge_boot_pages[nid], list) { - struct zone *zone; - - if (!vmemmap_should_optimize_bootmem_page(m)) - continue; - - nr_pages =3D pages_per_huge_page(m->hstate); - psize =3D nr_pages << PAGE_SHIFT; - paddr =3D virt_to_phys(m); - pfn =3D PHYS_PFN(paddr); - map =3D pfn_to_page(pfn); - start =3D (unsigned long)map; - end =3D start + hugetlb_vmemmap_size(m->hstate); - zone =3D pfn_to_zone(pfn, nid); - - if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), - zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) - panic("Failed to allocate memmap for HugeTLB page\n"); - memmap_boot_pages_add(DIV_ROUND_UP(HUGETLB_VMEMMAP_RESERVE_SIZE, PAGE_SI= ZE)); - - pnum =3D pfn_to_section_nr(pfn); - ns =3D psize / section_size; - - for (i =3D 0; i < ns; i++) { - sparse_init_early_section(nid, map, pnum, - SECTION_IS_VMEMMAP_PREINIT); - map +=3D section_map_size(); - pnum++; - } - + section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); + if (vmemmap_optimizable_order(pfn_to_section_order(pfn))) m->flags |=3D HUGE_BOOTMEM_HVO; - } } -#endif =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 7ac49c52457d..20eb03df542a 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,8 +9,7 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include -#include -#include +#include "internal.h" =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -27,10 +26,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate = *h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *= folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list= _head *folio_list); -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -void hugetlb_vmemmap_init_early(int nid); -#endif - +void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); =20 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -76,13 +72,13 @@ static inline void hugetlb_vmemmap_optimize_bootmem_fol= ios(struct hstate *h, { } =20 -static inline void hugetlb_vmemmap_init_early(int nid) +static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct h= state *h) { + return 0; } =20 -static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct h= state *h) +static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootm= em_page *m) { - return 0; } #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 8d39196f6d93..e48758805a8f 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -32,8 +32,6 @@ #include #include =20 -#include "hugetlb_vmemmap.h" - /* * Flags for vmemmap_populate_range and friends. */ @@ -404,34 +402,6 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigne= d long end, } } =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize) -{ - unsigned long maddr; - struct page *tail; - pte_t *pte; - int node =3D zone_to_nid(zone); - - tail =3D vmemmap_get_tail(order, zone); - if (!tail) - return -ENOMEM; - - for (maddr =3D addr; maddr < addr + headsize; maddr +=3D PAGE_SIZE) { - pte =3D vmemmap_populate_address(maddr, node, NULL, -1, 0); - if (!pte) - return -ENOMEM; - } - - /* - * Reuse the last page struct page mapped above for the rest. - */ - return vmemmap_populate_range(maddr, end, node, NULL, - page_to_pfn(tail), 0); -} -#endif - void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, unsigned long addr, unsigned long next) { @@ -634,7 +604,6 @@ struct page * __meminit __populate_section_memmap(unsig= ned long pfn, */ void __init sparse_vmemmap_init_nid_early(int nid) { - hugetlb_vmemmap_init_early(nid); } #endif =20 diff --git a/mm/sparse.h b/mm/sparse.h index e4617f9c8876..049272aba84e 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -16,6 +16,24 @@ static inline unsigned int section_order(const struct me= m_section *section) return section->order; } =20 +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ + VM_WARN_ON(section_order(section) && order && section_order(section) !=3D= order); + section->order =3D order; +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ + unsigned long section_nr =3D pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_order(__nr_to_section(section_nr + i), order); +} + static inline unsigned int pfn_to_section_order(unsigned long pfn) { return section_order(__pfn_to_section(pfn)); @@ -26,6 +44,15 @@ static inline unsigned int section_order(const struct me= m_section *section) return 0; } =20 +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ +} + static inline unsigned int pfn_to_section_order(unsigned long pfn) { return 0; --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25EAE3E6DCF for ; Tue, 25 Aug 2026 08:49:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647787; cv=none; b=ksym5g2b72OOr9doJr8Sf/hwWEk47M2OJaWzdILT4Ob9Y280ElEvUjKVJMM3LhGdQHkC/DXuYE10NGq16WGKlKCgkEsaTB1YKL0vGCMmleJG7QZik97U1vJ3agy8A2HC1lNI/bCLjHZdkMSfYNHu43aZxtVTrMZGIlYrdH3a3QM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647787; c=relaxed/simple; bh=CjapgYc+sDYtRdML+fwgzhcnrHxHJhv/yAMtqPWNivE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ttX9QCTorYUerT5kO8qzZJ7K59ccpjaC34P/w13pGaY3Uyj68KAWeQfcn/3SuB8mPuKsyaz5kWHI13uTdcpeTU3dxe+hKSG7JqZAfwwqz8KI2LWA4on2UC7J0NmhEC2YwFPcOkLaJTTLKQibxd2PoZiv5OzSt3Lx5hMz+NtC3vg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Vm2fBjDs; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Vm2fBjDs" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-38e041ea211so3831645a91.0 for ; Tue, 25 Aug 2026 01:49:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647785; x=1788252585; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Q5ifo7rguHpO8B0NJYkiw/UW3AMLKwMeqysag7KMPVw=; b=Vm2fBjDsAMUvpvEc7tetjAOS7e9vLNneD2rzp4KhyP1gewT3mL2CjlOMQ7WpgmvYKR EL9kus3elp8hsPwj4AG0V8RjjGlcfBkF4xexXWAFL9e02WF3jZqVoKiGn9zpTBe2wR9M mRsoxs17ndxKEKSq3eFIhCq4k3yg55WfnqUfIaI3LX87AWEXxxZJZ+bUGFXTv4bcfavO qKBhy8Ue3cdRL7wkmkzSyH5VMhO8wZhLTcY5ob+aTX79gr3ifnHwF1Ba/FifQfXm+qHg teYbaPISJHgLVge7s+W2D7z/FCF63uCxb3GtWs2NO+UXE7sRg0vNVzy/WFOlsvmWiHBo PMIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647785; x=1788252585; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Q5ifo7rguHpO8B0NJYkiw/UW3AMLKwMeqysag7KMPVw=; b=ErtBj+Ukrveq7pHET6+LlK+JQscE21h63lq7VXIwzj+xzWYUxP8y7VAkgTDO7AiNQK pzgI0IuU1DfbbDgXFTjxbut0Og5kLOWysh132lM/0iG+IwrQxo058vYe+fOdHDlwkOU+ U7GkbbYmQtuGVtkU1hSSy2C7Llky3F029tq5825UNaCLU4fWRcnaqAdH2SLCOj65C9bW IK8AAyJK1ioQRURYqqRkTFwp9s3OUTPrOuDWe0j7AI1Tvj5a2epvVSFUiddQdqhIGJ/0 XnIULXW/o0QeiiT+sPkvOMoN8w6U/pLjaR64Sd0aJBuFjS3TYcwRb/3pMsAKpvSQ2RKV 67GQ== X-Forwarded-Encrypted: i=1; AHgh+RouB5vIhOUqt0zCLzmt3RQrxkObH6BYtxJt1kLmIxdNFrn77rJPM0MWZzWX+VVdNIjkZ3UwYnLu8HdyOzo=@vger.kernel.org X-Gm-Message-State: AFuF++mpnM6+y6JqBrz/GZBFUuqS/ovIy8udM/hFMVk0dmOq8Sh4fSTs idJnnkzOAgFgqf+9sCUf/b9w55i8OrLFDFSqarM7nVSylujsgUW52wHscES/Ft387xc= X-Gm-Gg: AR+sD137mbTyq70ZuTwPx+OeC4vwGDk/WkFJTk/aC/AAJrVgdnjC2JgU8OypWiZvwNN oAON5Hxgd/lFzgr4+M0rVe0o6CRfT/x0PpPYHhjTUS3fuMCQ6X91mJ4jLRLLE7FPj+pGCd+CIy1 KpgahPxFOGu5FmVu7h05VLa7zgZautmHeMuiIkZysdogTcd/F5Rq/xkXCBobbgVslXcH4Z/b1Ke N4Cd3Fr7LrqGArHELRuZOgsoK8GSMt8+odXfj0Mgeprj7k1nEEBfE2ENL65KFjky5+EQZq/FGgX hb0jR3OOqiSbtmlLv0S3iUY7B0ZYxwCQbhicRtL4LwN/vXHZdSOr91I24jU91YeSSyx/CKevowa Kt+SsKgAW4lFwFL3bhELLed2cC/ei/qVqQQ50+bI2JkWIwEvdKLopLZ3vr6s2u2BhHUAu5GGQfo w3gC608pRs3vdfHi2pYuvi9+xs2ixP5qV+qxloLlSfIHBUP8RYVpvHOgY2sqrMO+durYlrq1ZDs Ke79tB1QafelteA+m4EBbRBgw== X-Received: by 2002:a17:90a:e7c5:b0:36d:9e0b:3801 with SMTP id 98e67ed59e1d1-395c34f47cbmr55316271a91.8.1787647785178; Tue, 25 Aug 2026 01:49:45 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:44 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 11/17] mm/sparse-vmemmap: remove SPARSEMEM_VMEMMAP_PREINIT support Date: Tue, 25 Aug 2026 16:46:02 +0800 Message-ID: <20260825084608.47437-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" SPARSEMEM_VMEMMAP_PREINIT existed only to support HugeTLB's early vmemmap optimization setup. Now that HugeTLB bootmem vmemmap optimization uses the common section-based sparse-vmemmap path, the sparse initialization code no longer needs a separate pre-initialization mechanism for vmemmap population. Remove the related Kconfig symbols, section flag, and empty early hook, so present sections always go through the normal sparse setup path. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Acked-by: Qi Zheng --- v2: - Collect Acked-by from Mike Rapoport --- arch/x86/Kconfig | 1 - fs/Kconfig | 1 - include/linux/mmzone.h | 25 ------------------------- mm/Kconfig | 5 ----- mm/sparse-vmemmap.c | 13 ------------- mm/sparse.c | 23 ++++++++--------------- 6 files changed, 8 insertions(+), 60 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 15fd9ec5ecac..7aa74bcc72f9 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -150,7 +150,6 @@ config X86 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 - select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select ARCH_WANT_IRQS_OFF_ACTIVATE_MM diff --git a/fs/Kconfig b/fs/Kconfig index e05917adcd60..d1c210c6508f 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,7 +278,6 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP - select SPARSEMEM_VMEMMAP_PREINIT if ARCH_WANT_HUGETLB_VMEMMAP_PREINIT =20 config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 177455d98064..8bcb522645ba 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2095,9 +2095,6 @@ enum { SECTION_IS_EARLY_BIT, #ifdef CONFIG_ZONE_DEVICE SECTION_TAINT_ZONE_DEVICE_BIT, -#endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - SECTION_IS_VMEMMAP_PREINIT_BIT, #endif SECTION_MAP_LAST_BIT, }; @@ -2109,9 +2106,6 @@ enum { #ifdef CONFIG_ZONE_DEVICE #define SECTION_TAINT_ZONE_DEVICE BIT(SECTION_TAINT_ZONE_DEVICE_BIT) #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -#define SECTION_IS_VMEMMAP_PREINIT BIT(SECTION_IS_VMEMMAP_PREINIT_BIT) -#endif #define SECTION_MAP_MASK (~(BIT(SECTION_MAP_LAST_BIT) - 1)) #define SECTION_NID_SHIFT SECTION_MAP_LAST_BIT =20 @@ -2166,24 +2160,6 @@ static inline int online_device_section(const struct= mem_section *section) } #endif =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -static inline int preinited_vmemmap_section(const struct mem_section *sect= ion) -{ - return (section && - (section->section_mem_map & SECTION_IS_VMEMMAP_PREINIT)); -} - -void sparse_vmemmap_init_nid_early(int nid); -#else -static inline int preinited_vmemmap_section(const struct mem_section *sect= ion) -{ - return 0; -} -static inline void sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static inline int online_section_nr(unsigned long nr) { return online_section(__nr_to_section(nr)); @@ -2385,7 +2361,6 @@ static inline unsigned long next_present_section_nr(u= nsigned long section_nr) #endif =20 #else -#define sparse_vmemmap_init_nid_early(_nid) do {} while (0) #define pfn_in_present_section pfn_valid #endif /* CONFIG_SPARSEMEM */ =20 diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..2c385f8b2944 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -461,8 +461,6 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. =20 -config SPARSEMEM_VMEMMAP_PREINIT - bool # # Select this config option from the architecture Kconfig, if it is prefer= red # to enable the feature of HugeTLB/dev_dax vmemmap optimization. @@ -473,9 +471,6 @@ config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool =20 -config ARCH_WANT_HUGETLB_VMEMMAP_PREINIT - bool - config HAVE_MEMBLOCK_PHYS_MAP bool =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e48758805a8f..e62e6aa07f12 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -594,19 +594,6 @@ struct page * __meminit __populate_section_memmap(unsi= gned long pfn, return pfn_to_page(pfn); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -/* - * This is called just before initializing sections for a NUMA node. - * Any special initialization that needs to be done before the - * generic initialization can be done from here. Sections that - * are initialized in hooks called from here will be skipped by - * the generic initialization. - */ -void __init sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static void subsection_mask_set(unsigned long *map, unsigned long pfn, unsigned long nr_pages) { diff --git a/mm/sparse.c b/mm/sparse.c index 439802e6a6ad..948839621f83 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -284,27 +284,20 @@ static void __init sparse_init_nid(int nid, unsigned = long pnum_begin, if (sparse_usage_init(nid, map_count)) panic("Failed to allocate usemap for node %d\n", nid); =20 - sparse_vmemmap_init_nid_early(nid); - for_each_present_section_nr(pnum_begin, pnum) { - struct mem_section *ms; unsigned long pfn =3D section_nr_to_pfn(pnum); + struct page *map; =20 if (pnum >=3D pnum_end) break; =20 - ms =3D __nr_to_section(pnum); - if (!preinited_vmemmap_section(ms)) { - struct page *map; - - map =3D __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) - panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); - } + map =3D __populate_section_memmap(pfn, PAGES_PER_SECTION, + nid, NULL, NULL); + if (!map) + panic("Failed to allocate memmap for section %lu\n", pnum); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); + sparse_init_early_section(nid, map, pnum, 0); } sparse_usage_fini(); } --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12D853E9C08 for ; Tue, 25 Aug 2026 08:49:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647791; cv=none; b=JLN+zECpBQ6cWwJ4pU0JhD9tpYAce+BohyJxgSFraoGAO9xVfR85s2s18qVe+UIvLlJmcnTmyttWC6XrpZh3/+DZJ568APk4CHA4bbwGgP1PFnM3h4oyhEiEu63gHxhJuFr1lmKJ3qI0zfRDqiMM7Y9DMgYtrS5oAkYC2fj30K4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647791; c=relaxed/simple; bh=WvHs/YYIPDCuTcJlnFuFsuo6HzyLFJYgVm8kdN3/Vs4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GBfcQDlDkOibqx+M8aMOdi40aUB0aQ4CdcetFjpbWjWZzF2u2529DOBzqrXyjuZrrYyefv5+pHcgsF/kOV2qDURd+hvWcERdMonhT8NF3OYqMGmORTFNxvHlbvg2LonoRHtdsRGOHh6QTBxzLlnVDeN8bE8n8I+qP8QY2A7481U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Dv92m6Ba; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Dv92m6Ba" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2cace91f112so51713695ad.0 for ; Tue, 25 Aug 2026 01:49:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647789; x=1788252589; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tfN6zKvJyOETRiPLuhLazGAgAbhX73SzFIauwVYcbkM=; b=Dv92m6BaDSLmJ8HYiunbgFbO78W6+bhXLc9EnuYo+HWcZ+pg/nVR6leaE46Q84Z1vm TkAkZhAzJj5YT53aQm9+XR342PQkQ7/KD3dgoMwvTearBktkFWaTN6SlZugghLsFRPhy +C3o/QFwdXgHlnL7eUAhenPsK5/V+da0Qkqa0LZRDE9/bI/Q6lu0tE7vkbvJ7++EJ6BC 5v2sFTG1rw9kyCEM+9GUTOqbDGDgWvy9tMJSkQsSQ2eH4FYfalz6qRg3aqh+AsTJEdkO OaOJ80ahuRGVehabIX7dIEl0LeqJjv1GXcTqRuTQvxQdlwiKBZORT3aOHkkOaozl1MGB JuwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647789; x=1788252589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=tfN6zKvJyOETRiPLuhLazGAgAbhX73SzFIauwVYcbkM=; b=ZZW/nYF2CEIQ6obGWWFaW9dcG4HtPBgZr8++hhIaMkP7ZB7MepxUAtuVuv5j1IjO8z 7kPvbDrlS3URYGckxDazfxnhbrnsQBkBFaIaNoV76gjRvGYmVKlSBxj8L1XX56bd4afc qtOVXtGvChPLO7fwewrjf1vjOQT0X2L6LRLAe905KZcnLSMf/OoNSVmZPKWKJI0HJsn7 Ag0YPitNVq/L+5QAI8yHoeM0to/SA2SafvIhjywrPzfCk/pPgySWODMPgtoJvHhRcYhS jx9actCO8pzwZtvqUy/nnFgyT8MxBDSWlAXK6UKOeGJS4/dwGmcTOBoxVIdu1m0ausWC IKJg== X-Forwarded-Encrypted: i=1; AHgh+RqtnuQq8My1zn6wuMUB2urnrbXsJlnFafEdZo5v8p/EJ3EPTK+0g6s1gY46eWWFXAbwP1JNUe9rFiNNKDw=@vger.kernel.org X-Gm-Message-State: AFuF++kpKaMyJIT5lOmgpFUhsewrTYWnMkfp327/VqGvRmbbVzKyGP1K r4gksb1VeOQFIfX4uFzKmJ6oGM9W+CkSyVzJQbDXTYR/TnWXF2i3MAWnPaHooimvBjk= X-Gm-Gg: AR+sD10npEN0O9R18XTBDaIvG7+K4qM8jn5F3/eFBQPcK7LYlTYKXWIBsHV3aL75Xyl 2EADwhNMVsiwvMaDh0eV/L+vA7yfPrLFiqGvgxkcyI8M1VN80QDyMpZPcYhtI5B4emFrK8Na3hB KpXrmfurWykBt+iOACp857e7S4JNuPk1KNw0mKui8p3uAO9QkqqlVv7WA1vfg5gbrVJCOAtgRTN uPoCZTcr+YOFrfd127U57FE9w6b8YSvBnaQFxUFMk3qmSn6a841fXdsDI+s0mZUZmpx1w/2ph5Y m8KXLZWm0ZkW97EILvtmp1/MpqNqD+AHJHIynD/Xt//wcJSqT64JRr1K/O/srdTAdNsacZd8lcY AjB0pnFAhwhHwgdxkDoscJMwxzmivImIXG6JinpENQMkk1v9mwXuhzfYqGD6ywpVZlB7kRd1Xf2 kGhVyAGMBL90lfSkiUmA40miCDq950bpwrGgRCNU/Kk0W/6H+ktrRc1EX7dTNINu7gdGF9dcPqX MyYT241/3eoXwK6F/kFJHpY/Mo6joIiGcFVyw== X-Received: by 2002:a17:90b:4ac6:b0:38f:18f9:785 with SMTP id 98e67ed59e1d1-395df13ced5mr45406324a91.8.1787647789251; Tue, 25 Aug 2026 01:49:49 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.45 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:48 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 12/17] mm/sparse: inline usemap allocation into sparse_init_nid() Date: Tue, 25 Aug 2026 16:46:03 +0800 Message-ID: <20260825084608.47437-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After removing SPARSEMEM_VMEMMAP_PREINIT, sparse_init_nid() no longer needs the transient sparse_usagebuf state and its helper wrappers. Allocate the usemap buffer directly in sparse_init_nid(), pass it to sparse_init_one_section(), and drop sparse_usage_init(), sparse_usage_fini(), and sparse_init_early_section(). Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/mmzone.h | 3 --- mm/sparse.c | 46 +++++++----------------------------------- 2 files changed, 7 insertions(+), 42 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8bcb522645ba..67c84a8a7258 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2223,9 +2223,6 @@ static inline bool pfn_section_first_valid(struct mem= _section *ms, unsigned long } #endif =20 -void sparse_init_early_section(int nid, struct page *map, unsigned long pn= um, - unsigned long flags); - #ifndef CONFIG_HAVE_ARCH_PFN_VALID /** * pfn_valid - check if there is a valid memory map entry for a PFN diff --git a/mm/sparse.c b/mm/sparse.c index 948839621f83..7096d8e13802 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -235,42 +235,6 @@ void __weak __meminit vmemmap_populate_print_last(void) { } =20 -static void *sparse_usagebuf __initdata; -static void *sparse_usagebuf_end __initdata; - -/* - * Helper function that is used for generic section initialization, and - * can also be used by any hooks added above. - */ -void __init sparse_init_early_section(int nid, struct page *map, - unsigned long pnum, unsigned long flags) -{ - BUG_ON(!sparse_usagebuf || sparse_usagebuf >=3D sparse_usagebuf_end); - sparse_init_one_section(__nr_to_section(pnum), pnum, map, - sparse_usagebuf, SECTION_IS_EARLY | flags); - sparse_usagebuf =3D (void *)sparse_usagebuf + mem_section_usage_size(); -} - -static int __init sparse_usage_init(int nid, unsigned long map_count) -{ - unsigned long size; - - size =3D mem_section_usage_size() * map_count; - sparse_usagebuf =3D memblock_alloc_node(size, SMP_CACHE_BYTES, nid); - if (!sparse_usagebuf) { - sparse_usagebuf_end =3D NULL; - return -ENOMEM; - } - - sparse_usagebuf_end =3D sparse_usagebuf + size; - return 0; -} - -static void __init sparse_usage_fini(void) -{ - sparse_usagebuf =3D sparse_usagebuf_end =3D NULL; -} - /* * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_= end) * And number of present sections in this node is map_count. @@ -280,8 +244,11 @@ static void __init sparse_init_nid(int nid, unsigned l= ong pnum_begin, unsigned long map_count) { unsigned long pnum; + struct mem_section_usage *usage; =20 - if (sparse_usage_init(nid, map_count)) + usage =3D memblock_alloc_node(map_count * mem_section_usage_size(), + SMP_CACHE_BYTES, nid); + if (!usage) panic("Failed to allocate usemap for node %d\n", nid); =20 for_each_present_section_nr(pnum_begin, pnum) { @@ -297,9 +264,10 @@ static void __init sparse_init_nid(int nid, unsigned l= ong pnum_begin, panic("Failed to allocate memmap for section %lu\n", pnum); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); + sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, + SECTION_IS_EARLY); + usage =3D (void *)usage + mem_section_usage_size(); } - sparse_usage_fini(); } =20 /* --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 405C33E6399 for ; Tue, 25 Aug 2026 08:49:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647796; cv=none; b=pfz3YERxcvnMJQvwd8dNOn7icDRN73gCBfBvvAjKQNsFKTPFfqPz8Re9wtmAYGDxClnZTHOR5Wfne3sUVPMCtmU8zZWTxYD8Wxn/J3baSB+Qgp/zqHBmJn6j3qmJzxzrkg/hdZSo4hPZ3njI0tILj/QLu6OOSXJKJbgHfp3XHIQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647796; c=relaxed/simple; bh=82Hkau/ZoHaXOb7RuWMyGt+EYlEY6zasNpjbWKTviLA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cb55Tx29xYTNFcB2QjLoWWXtBSAeZRaKlVQiDkCewaQX0Pe5Z75fME8nSpQnoMzVVovEp4lLIjIPq/JkZLMRanphuxv3W9F9o7HXsLJbeuXe5yj09BOo3ZEZppoSEg3/eD3ZtsjnI5r6UvM6D4gRD82qubVmdc7njj0aJuJr6do= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=SPnXtpLa; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="SPnXtpLa" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-3964e480f76so743546a91.1 for ; Tue, 25 Aug 2026 01:49:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647793; x=1788252593; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=3aBi/OVal72WiVaYLeIQyM3DGjjDwsK1rIANekiC/6s=; b=SPnXtpLarfqdnVmy2BnBFB8M6dD/Ev8k1TSt0+iNwfpGQjhWYJ6Tx4PWHa24FUQtMC DWFTX14UHNpBeMCFjGQk+cvdAEsWxyBCAr0gibN36W8YFwxvP268X8Enu5Frsux+///n BnEz9n3lhC0+ePUBl+XcjcxiN8G+rDmdmdZ6L6C9phcw3bLCv0oxjJVDWUJqKr2g+/4T ywniCi7EL0uElibD0DXLgYO7PensydqdBLKVrMeqh+pl8+oemGKCx+7zu7pVHua5fADz 75UlDD3OWth6L9/yd0mYrFB3IUOfdPsO1IQJZLZ6lPPVIh29xioiUP/ZuZI9sY38jfvS qRZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647793; x=1788252593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=3aBi/OVal72WiVaYLeIQyM3DGjjDwsK1rIANekiC/6s=; b=OD1M2AIoO0abFXcdcUN/HYTc077cwSZG1HKwVZkI/LLGmckVSdhQQlUATb/KVTBw6J 1WrNQC1lpQwZDUVFUx0DUZOmUlQ1tNnuFdB6UV+0nd29WN0CM0Ncv2RnjUDp6mUxz/Hf a/YC+dKKPwnnag0oXFZKOENpoL29d+Hu+BEqiLzzcqOy0lD2gM6WuqrHKyUtehhteEzq I3GEJ5WGPef/9+0/hgqS18qUw+8sXNO5YzHieWflganZpnHIn0Q544NOJNPjGYM+jCjW 751G7jsrNkisgFVbnhWfz52ZYFTdK/eeia4vImcSh9cL/R/fZyOqiHylc5UJ8p7uZcaZ WMag== X-Forwarded-Encrypted: i=1; AHgh+Ro5e+aN8bTFPp2icLW8Duj9weQaILGddqtfPTRRBFJY0uxL9HnqcXgTN11Og3+IjRq1U5Oarb59JcZXR+I=@vger.kernel.org X-Gm-Message-State: AFuF++noOfu+FEqZLVV5+WtMYcNTIapo2WsuxAe9ChW6qyDHdXHexGUW q9o0iKjIBbKbUuu78IYmn0AvNFjTKKsIA2Wu/Sd7WKFrzG3F2/dSFetzNccPSzCeD4U= X-Gm-Gg: AR+sD10Lhgg8Bnyo9WwB/Y3MoaOrQCE+ft/spwFaUFbt3WMeVfrCt4WyAdnNl1pMKvp oql95sY5gH457e2zhOlKklGUzQYkltfZWA/oRydlU8FxXWxpEJVWopW2iMbmQH02rAkHp4vrWwW uNfz6EyxlSw7v7CqETgLtjPueGegZcLmnrWuc8FwHvsapAg/ZtoRgDennHYNpid20hjQA66cpHn LxzzwfvCT0AJ1vYKt2eeIdvBhui/zqDZiKYY1INmEebX6HIQhCoaQEwH39r+A5zYUV+CBwODo4U r/0zuyvWlGRUMPgkon8oKDVAhgurMbJ+SfJ9MG501ISXQQTyWi98np/ltKutyHJTS51+jSOWyDb x6MSyjTJZ9HZ7hCmbWWJEhX8p7EtMxUPXW/UpuUOU7bYwffaoaAAcfQhyAZt0kOZGqwau5TdPZM 7sCtD11b8NDn7N2G8W/GdbPu97DfffsAHVJh7N3Ku4gUf5/TQgSs02J/T6LL+rARxtfdkENCF7Z of62vtZRhFsFNpkel1J0a4UIg== X-Received: by 2002:a17:90b:56c8:b0:38a:c3f:3b87 with SMTP id 98e67ed59e1d1-3964652108bmr10807861a91.12.1787647793182; Tue, 25 Aug 2026 01:49:53 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.49 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:52 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 13/17] mm/sparse: remove section_map_size() Date: Tue, 25 Aug 2026 16:46:04 +0800 Message-ID: <20260825084608.47437-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_map_size() no longer provides any shared logic. After the sparse-vmemmap changes, its only remaining user is the !CONFIG_SPARSEMEM_VMEMMAP path in __populate_section_memmap(), which can compute the size inline with PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION). Remove section_map_size() and inline the remaining calculation. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/mm.h | 1 - mm/sparse.c | 15 ++------------- 2 files changed, 2 insertions(+), 14 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 441bd39eab73..b19711b6dbc6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5140,7 +5140,6 @@ static inline void print_vma_addr(char *prefix, unsig= ned long rip) } #endif =20 -unsigned long section_map_size(void); struct page * __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); diff --git a/mm/sparse.c b/mm/sparse.c index 7096d8e13802..9349ed6326c0 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -209,23 +209,12 @@ void __init sparse_sections_init(void) memory_present(nid, start, end); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP -unsigned long __init section_map_size(void) -{ - return ALIGN(sizeof(struct page) * PAGES_PER_SECTION, PMD_SIZE); -} - -#else -unsigned long __init section_map_size(void) -{ - return PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION); -} - +#ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - unsigned long size =3D section_map_size(); + unsigned long size =3D PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION= ); =20 return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), nid, false); } --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2DF43E9C08 for ; Tue, 25 Aug 2026 08:49:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647801; cv=none; b=GD9EnZkQ5CDTeZPUgp67y5fBCmh9+Jqx+NSVfyGuDUX1/eQaNEuaT0VKRa/aXYlBNM1qDNPJQsdcKzwFAUPnnShSX8NANrQ2+6mQxRRIFw/8YDJjSeSiZrJKO1lHJX1A2W53P480weh3N3fICJZGxkyijTTAqZOc/2kOVy9Nm70= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647801; c=relaxed/simple; bh=hmip+mqhcGPK0pbUh95SSnF3l3OieR6J0iGMLcyQxfY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FCxTHlkauTL4lX26n+IOECDECCzEQdDr87rcDZtlTNHvMEHK/h2+MlWwsMpq3eyb/Dert18nD7sHc798eFjOdOZ/dku8EKDkUGbNRRD8nkXIDhaUFD/6Lm0MbtNfKU+v7tnceBsB3P6QicS+29Tn4XmLEMaqie9YebTG4SLul5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=WfqG0aS3; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="WfqG0aS3" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-3964e76d0f4so596550a91.3 for ; Tue, 25 Aug 2026 01:49:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647799; x=1788252599; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wogH2M0/MVAbv/lLrCje4/37+/7jKlgSRWIrrsIx4/8=; b=WfqG0aS32Qsh2MIjFim7IaXxZdDhm+Z7hr6aBq6/dlPhj+3QXGWr2l9E0jLUO7kf6/ b5qni2JRqDxeTiYt8hU/tSH7EqSniP5WQT5JulPiYJlSu+GpjvGHyhsgmZkI/h8rEjnw QPBQMRvpRyp4qDLipcF2bsFwIm71nuMfxSoEYvLcIeLp/GehIOosDI/9Usko4FLAPMnf 7zgyAmJ1Z0vjDlP9sgrVj0WUvTI3/jV0oWCHqJbQ9dO/ko4k5B/Wm0wz6OrkalMzbnuC tJSq+Tye8fJwZg3VZXIjJyyZA0Acp9/NCzj2DmPOH5cB33JQWqt0pvxIe22as3rA/rZo Se1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647799; x=1788252599; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wogH2M0/MVAbv/lLrCje4/37+/7jKlgSRWIrrsIx4/8=; b=Bw/v3x7dO+hvE3BN/iAkqiK6cahXXwKj2MQoqRouQha/j8OBBr9OZhyBZDn28At52D FKjlAjqkf8uqL0hvcBEXi/7MRy4Wt/y+XZwtoYWRfBkpGGzl2J4HqfPjrF/38JSvkEfd LjBAmm4KMmCTWoed3Xuuh+AzPnUKldpGnX1GMwiXaWim+IdF/t8imOhIHoPcAk/mV+MW 35T8l75MTqRUCQYrqLmyEzv891WaNNzaFcAy8zc50iL80p/EKpSe7UxDDwO/yzkoM3yJ uEz0kwQEFHreei2WIntfB6/djTMzWLs9TjgpWPqnHSlPgtrVeRIxWJwoFw7mpKo/HlWR Zx/Q== X-Forwarded-Encrypted: i=1; AHgh+RrVRi6oH4IHYXT/oSAaKrmwbpVcDojBPL7hl2CbvpcR2klESFeq+jFMU5/5tdBxJd9ogPDEa/KRiQ8VZw8=@vger.kernel.org X-Gm-Message-State: AFuF++mA0SPlJOYlCXLEnqh6lrVZh1aH45/0KXEbaomqp77Hq2GrFGcM ljCcc1R6sWENnV2np5f3nOS9XoERjLIyMrsCNmKjW4IJ83AfePRbKH5Isv18GnzCduk= X-Gm-Gg: AR+sD13Vt3H9TGAiFZMnDGumfJIS4MrYre/kR4nIkmLhD6rwyBqvICRyn1u3S/bvPOk jQ4fJMhlxHUf27walbtbMrV5DLS49uGzSJ0gj7/Jsvy3VtDd/b5ZKo6bwb7SKVFURIxLNuSBf+N EWpBvboNMcHyrVo1Bx6BsBSNkJS5iCVzJq+Xkrl3et6RoXGPn88TgJuyHxf5CA0v26EQZmaJGm5 i8GLfdDr9NI2Gi2d+E/AwCuSsNrkxptfPtZGp8JhGE8plNJUBYaPhcFkzTrtduOmmWBZNK2ujn7 GKrgfb7flfIpwVk66WDIMVR8Ulj67LNHX1vUOQp+XBv51OgYjN6mAGM5fTur0DUJhge0xEH80kd iOlOQQ7LC0x27v5XoyE1npbE899Ja/EWiK2gBYwQe3xOs04E4JXPSuwmCBsWsXzUCAnEWG4V2Hz tGTj31QWPQxKS+1rQiDJtdAMvgFiImlHik/j37JgYqza3wXSR1X1t5o+c9nWOVyGe9U3/e/0a7M Tjdo7jmRp/EQGw9fO3hh3I7bvIclwozpmk3 X-Received: by 2002:a17:90b:538f:b0:38e:b400:a860 with SMTP id 98e67ed59e1d1-395df29e128mr48228598a91.13.1787647798322; Tue, 25 Aug 2026 01:49:58 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.53 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:49:56 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 14/17] mm/hugetlb: remove HUGE_BOOTMEM_HVO Date: Tue, 25 Aug 2026 16:46:05 +0800 Message-ID: <20260825084608.47437-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The HUGE_BOOTMEM_HVO flag tracked whether a bootmem huge page had already gone through the old early vmemmap optimization path. Now that HugeTLB uses section-based vmemmap optimization, that state is already reflected in the section order. Remove HUGE_BOOTMEM_HVO and its helper, and use the section state directly when deciding whether to mark a folio as vmemmap-optimized. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v3: - Use the order-based helper for the bootmem vmemmap-optimized check v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 5 ++--- mm/hugetlb.c | 12 +----------- mm/hugetlb_vmemmap.c | 2 -- 3 files changed, 3 insertions(+), 16 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index fe28f98e1b22..3559041a5a57 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -675,9 +675,8 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; =20 -#define HUGE_BOOTMEM_HVO 0x0001 -#define HUGE_BOOTMEM_ZONES_VALID 0x0002 -#define HUGE_BOOTMEM_CMA 0x0004 +#define HUGE_BOOTMEM_ZONES_VALID BIT(0) +#define HUGE_BOOTMEM_CMA BIT(1) =20 int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fbb0c83bea79..4bb5a25849b1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3198,11 +3198,6 @@ static void __init hugetlb_folio_init_vmemmap(struct= folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } =20 -static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) -{ - return m->flags & HUGE_BOOTMEM_HVO; -} - static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page = *m) { return m->flags & HUGE_BOOTMEM_CMA; @@ -3294,12 +3289,7 @@ static void __init gather_bootmem_prealloc_node(unsi= gned long nid) HUGETLB_VMEMMAP_RESERVE_PAGES); init_new_hugetlb_folio(folio); =20 - if (hugetlb_bootmem_page_prehvo(m)) - /* - * If pre-HVO was done, just set the - * flag, the HVO code will then skip - * this folio. - */ + if (vmemmap_optimizable_order(pfn_to_section_order(folio_pfn(folio)))) folio_set_hugetlb_vmemmap_optimized(folio); section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); =20 diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 7293706b532f..a25adc474351 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -714,8 +714,6 @@ void __init hugetlb_vmemmap_optimize_bootmem_page(struc= t huge_bootmem_page *m) return; =20 section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); - if (vmemmap_optimizable_order(pfn_to_section_order(pfn))) - m->flags |=3D HUGE_BOOTMEM_HVO; } =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F23E30C168 for ; Tue, 25 Aug 2026 08:50:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647804; cv=none; b=kP9/zdVodGpHBTzmikjxCan2H66qmO6O9/kQtaFyA1OGhFXBHD9weXcEeSQNJlHgOT/pMSNNKf6grmkWV2ddkvGCvYs/07Iy+BxxhzFh6GPVjrpnLQeT4oSKQHMzhhfoti7delGJYRoWRfmhJt9pDJP+wJ+qyK0JKZwlYaqNCVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647804; c=relaxed/simple; bh=WjgiHsnx7xD3+T3kJmUxNPqoGxXpRrKhLzN+Kh6y/0k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TGpw3WaRyYyPtMtAwc8PGUmcGDIzMBjVNe18Mvef4BEUOvEumbHJS1WHdeaccxEeeytjI/Zm5ddRkFFzjo5IJPDWymkkqBgOttRocRmixmWPFBp5d3FUpoWFb5NffXVCwkeLOcPQpzPopptVPpo2xGKduOspSRgcniNxq/dmQZc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=JI77y5C8; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="JI77y5C8" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-cb5b8572b70so5339603a12.2 for ; Tue, 25 Aug 2026 01:50:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647802; x=1788252602; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Wvrg0bPIlMwg/Co6PQfYja34C6YJR4b4gHgQBVsm43s=; b=JI77y5C8k41he1m5KnQXosZiuKBX3PcFD0tVThOLBOqzFiMBAIgJ49GLIeZHZgG/gU U66K9z0mmZk0I4bsfq25MbfWnVA5Ow6h+10WRE9lVC/RS7aDfnqxNJ0hPnzGJYbphMLN s8foxDAjmjZ7nC3I51hDPt55exIg+JjVeFpUqyqyuRWMaEhrZ/XJ+WMv1rei20eA4+fk 5lHaZnlpYao0VewI167q/WIyE09GtcQDBecBGcgtflI+hqCnYcfXq9dpt64M6c+Gra9T 5qarRgXwfRgxasrzVshS0CvPUNm90E+TbyB8Z1EDjB2fRXaHtBgrKmtbbAR3iTK6AAMB YyIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647802; x=1788252602; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Wvrg0bPIlMwg/Co6PQfYja34C6YJR4b4gHgQBVsm43s=; b=jSzbAiwCRGEpF4VTFY7V4M7WSW24mcooQD5LtSab+U6T8fYFb9uo6gWzAXnw02e1lU dPOk6Jhri5sE0BSVcw2mZ8d6eei/8KvpOQWgVS3DBSILdCwFCY3mjuCmVLQzBMmHRVng KpowBbfamhS/bQKgGc65bdqBJflakpNFWfRgsGwGSd68I0ID7a1QlRdQ9NJlxGFV/HnK g7W0x1UOWlz7XiXY8tb+G8fHREmINlfdMawsqI4OvpKYI4dt5v+WwyIydcCVXPJtopSL FTEat1e579z0NcVc34yJCkrJRhl3iMf5eD1Y5tRoE5YvUFbr7LggFTC4tasKR69OjtWm Fiqw== X-Forwarded-Encrypted: i=1; AHgh+RqXCl1/yO2Ct73AEJ/cPMMPJhAxMxPZEGSBBEvqZ8c0CK7Qgvg5LpF4mHwERtDswrF/67q2bymr9DDaU8o=@vger.kernel.org X-Gm-Message-State: AFuF++kdc5ZRELsYWpw/ElT/iKwPsjK3ZM7w/9l4yFeGC10nA4QSaNjh bTjSj7T0lEjyJb4PhyEqGdsZsRdaHRbOjXznApEoHvoL/LSMEpQMAuqsSPhlDkHun2k= X-Gm-Gg: AR+sD11mJInwc9YsfRkiVltUrUJt9kGTAOsFiiuarqhhU+Kd2a4vL6lNP7vA0AoohiT J5s/NvlBNdOA0s/064W6/LTqLv9Z13U3qvcCz7jrnQjgPBKWd8qtstW2zQMXO1E/PCsE/HSjytC xbOIYxkDluBoWIYkwzSHh1ObTjzeMpsfbRmy74PwjnI+Xn8yMgjxPENRv9fNpASYEfDtWYUlXmr 3bRGJGewak1zkgBN+/B3P8fz9fuKDJn/OU+a6dh4F3LQBUifXLfXHFOJ07F8eUZ3yxAkLwTcyXF 6ig+SrLn/MVnrso9lK4zN13H3Pu627ToxoWWnabawj5Q3mT8akfWQbu5u1aVisUmqWKf/g85PHe qlhdwPip0+pQLBq2fmzvatj4CvjsydTQd0HFflBDvJW6Yqa0c8PXxcCwbWXD+utY9hnz7g7J66y XRORFAb9DvkEwa62uJKhSwb/PUeYdMtyWVQYard69CvnIt7FLUtzK7vPlTk84au0poVu9tHxTZA N3E2XWXTlfvBkABxC56Mnejcr0TPkRPdIWr X-Received: by 2002:a17:90b:520c:b0:37f:ed7e:7e42 with SMTP id 98e67ed59e1d1-395c3867ac5mr63066803a91.14.1787647802366; Tue, 25 Aug 2026 01:50:02 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.49.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:50:02 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 15/17] mm/hugetlb: remove HUGE_BOOTMEM_CMA Date: Tue, 25 Aug 2026 16:46:06 +0800 Message-ID: <20260825084608.47437-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Track early CMA hugetlb pages from the hstate instead of storing a redundant bootmem flag. This removes the unused helper and keeps the bootmem metadata minimal. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 1 - mm/hugetlb.c | 14 ++++---------- 2 files changed, 4 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3559041a5a57..255a258f11d1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -676,7 +676,6 @@ struct hstate { }; =20 #define HUGE_BOOTMEM_ZONES_VALID BIT(0) -#define HUGE_BOOTMEM_CMA BIT(1) =20 int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4bb5a25849b1..6822fb4159fb 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3122,7 +3122,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) */ INIT_LIST_HEAD(&m->list); m->hstate =3D h; - m->flags =3D hugetlb_early_cma(h) ? HUGE_BOOTMEM_CMA : 0; + m->flags =3D 0; =20 /* CMA pages: zone-crossing is validated in hugetlb_cma_reserve(). */ if (!hugetlb_early_cma(h) && @@ -3198,11 +3198,6 @@ static void __init hugetlb_folio_init_vmemmap(struct= folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } =20 -static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page = *m) -{ - return m->flags & HUGE_BOOTMEM_CMA; -} - /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABL= E) @@ -3293,9 +3288,6 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) folio_set_hugetlb_vmemmap_optimized(folio); section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); =20 - if (hugetlb_bootmem_page_earlycma(m)) - folio_set_hugetlb_cma(folio); - list_add(&folio->lru, &folio_list); =20 /* @@ -3306,7 +3298,9 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) * For CMA pages, this is done in init_cma_pageblock * (via hugetlb_bootmem_init_migratetype), so skip it here. */ - if (!folio_test_hugetlb_cma(folio)) + if (hugetlb_early_cma(h)) + folio_set_hugetlb_cma(folio); + else adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A86B3AA182 for ; Tue, 25 Aug 2026 08:50:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647809; cv=none; b=dkkJMv08z2BuzRDXsunuwsR9tHjA+Ct3jF/io/Z4UuEBK++0E/ZMoCrtpF1mAU/BZOSoL115yw6gnshYk9Cd8XZZFhRMu3Zu54t5pxHGs/lLleZsJNyNzR0Odpdgx+MBmFB6jd8bKKlVyYbltygQ4+7uZu7sTNSiB/dV35Aw2Vs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647809; c=relaxed/simple; bh=eU4KMCbszTTDR/hi5zsOw58LHjF/NkT3WAOR2vR37e0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JB2HBzivG5FPEWNjnthoVtV19W8SOjjwG+V1jkDYPB2plIDqUYExcVcTYED/KnYJYV0APT9JApoXaZSekzpTd/iv9DChL5CgIkyKfJkjAdvnevmdWVPppqVceAEj2DgLSaxYrz7UoGLIg2RwvZPPXJpfVKEdroUDsmeucBh4hFE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=XZybyYmf; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="XZybyYmf" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-38e58034d05so4014162a91.2 for ; Tue, 25 Aug 2026 01:50:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647807; x=1788252607; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=MkaNBeAytbvVs3m3mYf/1Z2YiyF8/bnD0d7KipoqAQ4=; b=XZybyYmfMoR3icJLrLfVDSNDkvZmH/l0VyG827DczQkoNXtf2/+9rP1xwHJz+oy6rN vNd0MnMeQyHhYukibfLXEihPPpJsqIVcOq5ijupqEy3a3p5DXSOA7u2+npiiaVCjxa3w 5KwQn5K6LXTUp/0OACcP8rI2lRjw0zP5XJV89GaXpGIBymd/tcfY77OyTAZ3hyZC8QSu NT7GA4WSnoxdLZRwKnKSAArOK6eC8b0mFPmY8hd2YvtSxhJ6Ed+AGHRh9Q/STJyk+RKd 8XyJYLmntIH1LFEtDVTDZPF1Gmli3H+pwGFCPxdJ++LJVVBXvMf0fnpTNes/uoPKdmZD /XCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647807; x=1788252607; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=MkaNBeAytbvVs3m3mYf/1Z2YiyF8/bnD0d7KipoqAQ4=; b=S8luh0g7qlP2krMr/AImyQ4d9N9ktF0HGuGDsFgfPV1pvowYgdrXlBhiOS1tyewrFM oO8MLVh00G41o8Ml8OsbF/IdrQNZN+By/K+SOR6C/ERIV3k13Z4yzv3ES6lnLS0x9jDt t/DaARRU9KM2BBPtDiYsnjybOGef7YVNtOh0YvX/0zQpTvJQ/ppJHwp2hyKfoekYdh3M Cmnct5MXOTTSKHlaU1LRnFMMimQLL/vSSnDdoDJHHhoCGd/36lrgd2oe4AohvnzJw2eR VR/LR7dmbmjg/Ib2Po7kRPQP1MxMK989fp2dvQg4c0SnB11+Yvdd4Wi4aWFC7qS/1ZoB cQfA== X-Forwarded-Encrypted: i=1; AHgh+Ro1dWfhKxtEDQXZkc5O1JW/khoIRw5WYFd8pl/ArTCTUWQhPPAvljWwJfA4hIPP0DxCXHoYNuxAldorUeE=@vger.kernel.org X-Gm-Message-State: AFuF++k6TdQCZYJgEEDN73mkW6ufvTsUHrbACOJ+Dp1qsd+Yj/Q3c4/m NAFZ1Oz2BcN55LN3K8qsHrV2VG/fQCuczbggsKbQx4LIbkuUPJgedaBVahHQTxqpHCY= X-Gm-Gg: AR+sD13I4o3AusSnZbf8AHlNzXiZ/Q7gpjY31riD36S5pI6W+4N679vrlZnW0WaQ/wK ym4pt54ZQNvbcxOUFSSEaJdvNewk0tEMfP7MqTLZhIBk+7h5MvRJt2EKuqkv88Cvzla8cgyJzmV e/rhmnODwh4aAZOHbtoWbFuKW9BtDYvm21vZjs3taiQlkLzXnC/DEqiabKWsrO9Doy1Y7Jti8Yq Xfd6nL94pypxxoLUoIRX0Y+j44rHvb+k3bw77wm9sDtHTRDVtHeALuKaA9MRg/3uZzL2PcVsa28 OZK6hiEFv/wM+5EzI/EFA7TWOEIsTEUjvewtem62aBJGZtXPiFW85+V1Urd6SV9QhXYBHNxKrTE fPh1Ig6/rVoXMf6v3NyQfO4SesTSDe6TFAtHfR/+Wqco/GawD4exrBVBoLRgdnTHvzVc/lvN+jY s9skuO2P6jlFev2NkQZ5kwi2A2rEiv25V8fhOr8SKc3UD2bpfnPD8hLPpBX1ZBI2ccj+IDGJDhL 9LCm6yWOYH0hs2ofcalUFLXNg== X-Received: by 2002:a17:90b:53d0:b0:38e:bbf1:de3f with SMTP id 98e67ed59e1d1-395c359a74bmr59356178a91.12.1787647806588; Tue, 25 Aug 2026 01:50:06 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.50.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:50:06 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 16/17] mm/hugetlb: localize struct huge_bootmem_page Date: Tue, 25 Aug 2026 16:46:07 +0800 Message-ID: <20260825084608.47437-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct huge_bootmem_page is only used by hugetlb boot-time allocation code, but its definition currently lives in mm/internal.h because hugetlb_vmemmap_optimize_bootmem_page() takes it as an argument. This exposes a hugetlb-specific internal type more broadly than needed. Change hugetlb_vmemmap_optimize_bootmem_page() to take the information it actually needs. With that interface, mm/hugetlb_vmemmap.h no longer needs to include mm/internal.h, and struct huge_bootmem_page can move into mm/hugetlb.c. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v2: - Pass PFN and order to hugetlb_vmemmap_optimize_bootmem_page() instead of struct hstate and bootmem metadata to simplify the code further - Collect Acked-by from Mike Rapoport --- mm/hugetlb.c | 8 +++++++- mm/hugetlb_vmemmap.c | 8 +++----- mm/hugetlb_vmemmap.h | 5 ++--- mm/internal.h | 7 ------- 4 files changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6822fb4159fb..51d386b8b5c1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -55,6 +55,12 @@ #include "sparse.h" #include =20 +struct huge_bootmem_page { + struct list_head list; + struct hstate *hstate; + unsigned long flags; +}; + int hugetlb_max_hstate __read_mostly; unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; @@ -3140,7 +3146,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) } else { list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |=3D HUGE_BOOTMEM_ZONES_VALID; - hugetlb_vmemmap_optimize_bootmem_page(m); + hugetlb_vmemmap_optimize_bootmem_page(pfn, huge_page_order(h)); /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index a25adc474351..eb339c4a71f4 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -19,6 +19,7 @@ #include #include "hugetlb_vmemmap.h" #include "sparse.h" +#include "internal.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -705,15 +706,12 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct h= state *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } =20 -void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page= *m) +void __init hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsig= ned int order) { - struct hstate *h =3D m->hstate; - unsigned long pfn =3D PHYS_PFN(__pa(m)); - if (!READ_ONCE(vmemmap_optimize_enabled)) return; =20 - section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); + section_set_order_range(pfn, 1UL << order, order); } =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 20eb03df542a..464192e32dec 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,7 +9,6 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include -#include "internal.h" =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -26,7 +25,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *= h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *= folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list= _head *folio_list); -void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); +void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsigned int= order); =20 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -77,7 +76,7 @@ static inline unsigned int hugetlb_vmemmap_optimizable_si= ze(const struct hstate return 0; } =20 -static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootm= em_page *m) +static inline void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn= , unsigned int order) { } #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c9..c37a468d53c1 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -23,13 +23,6 @@ #include "vma.h" =20 struct folio_batch; -struct hstate; - -struct huge_bootmem_page { - struct list_head list; - struct hstate *hstate; - unsigned long flags; -}; =20 /* mm/workingset.c */ bool workingset_test_recent(void *shadow, bool file, bool *workingset, --=20 2.54.0 From nobody Mon Sep 28 07:23:10 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D9C93EB0FA for ; Tue, 25 Aug 2026 08:50:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647812; cv=none; b=n7JwrWRDF3IubcvXTck8OEOoHVu8n99nELT52r21J56qDd3zUWEU9wdYWAUiHDmAZWKQhpH0RoRQWrjShXmSYq3mog8UjFM+yoB/roYVgntdDr08KwIBOU3ktuSPl0vO5TvWrroyAPR57twLMUmaXxn4WFwh228eZgV3avzLeyY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787647812; c=relaxed/simple; bh=15eB+MuDrFkwWDFgUWoSPNVdl4JsiQRyLX8rJnwDzUE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nF3fYQaS8hpQDYZn7VnCpQZDm2u3DFMacp94LWOgYBLsKNhI4nW4OyBj6XTJXVDaikFh0Ar1BMc3tz7NwwrX+QFVWDcRlkU3Ru7T0+PLk7xZD2wlKmgQ4isGKbUAkNasHZv3qarzlukSchzayE6xDgGDWYWocgVpaiJ8o+fnivU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=X+O37NsC; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="X+O37NsC" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2cf452def93so3916335ad.1 for ; Tue, 25 Aug 2026 01:50:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787647811; x=1788252611; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=imD7OJcy+PM8Ecas8hhVPwhNIQNnPaNcrXb98j5hJcc=; b=X+O37NsCjuBaNXHcwqHbnA3mWLRIiXvNAqP/KHTmz4rdudufQ0hLOrBEjpA4JAnjaj nhZX9QtO514mGZu5riEfK+XYs++lOrCTQoYn6sM5EP4yATw1QU5xx2w24zTpcnM52yg6 6nY/JHMkgAFsSCkzVcErhAG+hpBLTfxL3QcwaDntZ3SXVCuH7FuF7z8z/S/5qOc+hPCY QoLcamSbaAC4bWpfcMHKTaSAAyH6oh37tUZlu3t1L8k6RTWDXKg2wuutoWmOAUuGnnEi QNZYrr1HIqrDDW2lxe6O9dNscKTozD02xEm+UI9M28ZdfwHHM+/Dx2PZVnYDq+6MCOFK Rgdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787647811; x=1788252611; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=imD7OJcy+PM8Ecas8hhVPwhNIQNnPaNcrXb98j5hJcc=; b=fxTmCU2YlXvsb1ZEbrH9YlDX094Mo3fMZ5d1UrjiV3AUz2t1HmrM8IiSOxy8iGBL29 N9BrYsdyc+26m6G2qvpHTcWXVbgGg1w0PcA3727YNdeD+5+VjTgGDfrZFLAvt7iE/2BH fWL3VhPGiptyNu8fxJvIcG+A3o61sbbkGRTsVZvWDt2vmPqFvP++HbNV9DMzUskk2q+q 8cHvhs9kh8qW9Roj/hxYgfbOUYwSiaiiJjGWm/5Cwbj5Nsyqko0s387cdHCRwfh9aThw IoA5f5OHU7HdI/EKuvkDzQMo1Iuf3fGz2RKfSAHOHo2tu9xbSi2rd+6rlc5PoTR3Ht7k 12GA== X-Forwarded-Encrypted: i=1; AHgh+RrmZm9wRd1LHdpOvWOAcWHHzK6P8ctguTihqC9wNEOO/S5BhAe+eMooF1kmZRA76MV3iZyYoFRhcGJEtTA=@vger.kernel.org X-Gm-Message-State: AFuF++ntIL2liitBt43KZptvitSw7W3vpAtsoENOLW0FOu7j/obThYOX ozX5xE5WZTxgjaqRDWv7VLsZTv4FnX7lABmprE3jRLMFogTuEz6LLV7RpMcGhNkOQeE= X-Gm-Gg: AR+sD12hUcx1tGLPhYoyJ2ZIMg6/YPZHOnZePE/4OmnwRQHSdKrijkcnMBayB/D0HsD a1KzqYYDp8MCDr1Bpu29KVqSFXi2dxlx7WX8+mClQug9nldwN3v5fdTA6+IwKD7EDKUeuD+WzjL 6rO+YPpHyaSh3VHVAGGOiG41Iz+Nf5LNOQB/0U9QiQ2+XnOzjXEiVZYOvtIvJQXyWhJRDC4/OWh 30Hwc3NO8xrgMr+K2IN2TDGM8VD1CFZs4/Q7aNT4ZWk7mVJi9/FbImtWqgO+3tCTLH6YWM7iYUp VHyRFyyzjYX/kzzDfZfd1nBAoML4d8tujbTMU1ToFulvocUW/8Z1sK6ta2jcddVqbvNi2Ts87qo S3O/nEp/LTs58qSP9vsFZMzCogRYR55rz0IPK+7646sz4CGFD8KAUeyNgRxANfUSoFQ6nUlNEvl /euIntlS8mArjdTBY1JxVqhek+EKj6juURrl2LyTvivccafPILFMcquEsZfZk9T8NGxdYo79Kfb OEU/BgE5MrleEDyEvmvem/xAg== X-Received: by 2002:a17:90b:2543:b0:395:8124:ac53 with SMTP id 98e67ed59e1d1-39645aaa837mr5929896a91.6.1787647810763; Tue, 25 Aug 2026 01:50:10 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.7]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39645da08desm2873386a91.17.2026.08.25.01.50.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 25 Aug 2026 01:50:10 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , Qi Zheng , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v5 17/17] mm/hugetlb: localize HUGE_BOOTMEM_ZONES_VALID Date: Tue, 25 Aug 2026 16:46:08 +0800 Message-ID: <20260825084608.47437-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260825084608.47437-1-songmuchun@bytedance.com> References: <20260825084608.47437-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HUGE_BOOTMEM_ZONES_VALID is only used by the huge_bootmem_page flag handling in mm/hugetlb.c. Keep the definition next to that private data structure instead of exposing it through the public hugetlb header. No functional change is intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Qi Zheng --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 2 -- mm/hugetlb.c | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 255a258f11d1..900c95e346b2 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -675,8 +675,6 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; =20 -#define HUGE_BOOTMEM_ZONES_VALID BIT(0) - int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); void wait_for_freed_hugetlb_folios(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 51d386b8b5c1..3fb261d9214f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -55,6 +55,8 @@ #include "sparse.h" #include =20 +#define HUGE_BOOTMEM_ZONES_VALID BIT(0) + struct huge_bootmem_page { struct list_head list; struct hstate *hstate; --=20 2.54.0