From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9764B44BC91 for ; Wed, 19 Aug 2026 09:53:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133192; cv=none; b=G7EygYWnhRWm+GuzJ5YQ6mz4CftCtRtqLLfFZ/hI4ASkBI1uh3vjfuosrq0qiMQE8JNr9LjsajpaDT+vkH2y85LH4UxlqTMDmVfxQ5yIkxx5es4RrUp6rKofyrQo0Kr43BUFMSH0O3QK9DC3KBGTFi2XhcsBZdz5qq9oGgRUp5M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133192; c=relaxed/simple; bh=fb22Klqi5RrupQKeoji8R+9SJ4vTRSBLUMnS/YArHeY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qocOvBp8FHsZi9TRZGZMYWoeCLKH5e9c/+BV8/jlhIO7JPjRe97JOCQ/oCcL1Ix5zdfizsgdinU4hfqFue6LfDjQQ6ZrMmwgp1yz3zLdPmFN0wAHhsGvSOwAzcicvMbncUqzDB/MqhtwkWSJtk26PrjkhxJQSQK4KnLu0dUVs3M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PDbMZIQ8; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PDbMZIQ8" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2cab973140bso10376215ad.3 for ; Wed, 19 Aug 2026 02:53:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133183; x=1787737983; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=w63tlQxIXv6hFZNtC3XunmfoLQDKL59/Pnvl33aJ5Uo=; b=PDbMZIQ8I8jeSL10R+25YpU/ol2NopbstnPLmmLVpGW53Brrhc3kaT3iqoWtuUtGXz FGNl4bTRWgLE+eOlA4GlsoRg3pl96g7leiU7p1MJOCCk0txRBnvIflSo0Fx0ClrMm/0d 4UVn7avDxwvgdlFOGKfXjdnlHFcNjDJ3pwTFyBRgPvad93hbGto1sgQO3Ca+9jk/qWNU ZaEBkZ5IbrPVI34YlZ2G0BxVPT+itXEOS51iuCBlJpdAbOyXz3ENXtbPpPPrc5bo29xj zntA0y4oz7yNX72/Sx9RD7P1jzQa1/s9Ksz6kmIPNb8hLaTR8PI/tQQz/fHxSHnTe50D QwFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133183; x=1787737983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=w63tlQxIXv6hFZNtC3XunmfoLQDKL59/Pnvl33aJ5Uo=; b=DkrnyaN4ltQkpwkQ/liCNJa52MuE3SlJHyZJxI/81WcVsDVpw8l408UTha39JZHeoW HVOXueIMviYPei5farmA9Y0Pn1ECnro0KJw8dE4Fs8pJMHV+Iisot93g7rj8w+cTzsmg Ure3Yz+fOCQgQcPuVxwBCHc9pz/cwLcBLqd2ySB+Fu+qw3tfXUQQbajH/Biv0g25eoVK afT9ie9Deru18DHLZbR3YpOztctPK8UQVgj5hMXZr+SqnAvw63xXWyq7eGgm8TeZwCt+ 0xeQ41sUJm/UVvViY1lnLZWYmIW6xdkt/03m/xEUEXfiHkKoIG0QKmr0YotbqoLf2/cc RJsg== X-Forwarded-Encrypted: i=1; AHgh+Rpdbt+wvehIhbCB47TXEJDXKsuYWRV9K/UazSPHGGoFqVqJNoSh04KVKuKVoD+adnUNtDLDyAmv5t/pV4c=@vger.kernel.org X-Gm-Message-State: AFuF++mZnGuwhlA/moS3Ivuz9BjFCdPZnwT3AT67RHhqfjlf2uwuBBew JAIoRbbU/pjtWv9qrLYeWJwk6iI9NaSQyuIzkNlppGo3+fxi/B8jZS1iTVo+BmSLdQ5yRviRUVD sSb80 X-Gm-Gg: AR+sD10ymCtcCIwaGe7yV5bJm7NPHnhKjM2Tg1SWbloWEzgp24z1H7MQc4KEnCwJBLx S2D6CsClV4E6VEpUp0Y08P4SbdCt8E0NchagC8TtcCj2XbRdXNYWgWWRXYCdKeFTTvEZauXk225 22ZjBUG64bbV4Dnwh7Mdvz41LKBPk0V+kHUWoS+SrB2da7oxYlW9Cbgk0O/qfeFNLesD9ecUYfN t7aS2s8DkjWc9mqY82RJw46PF9ZQJrVOlG05tYikP4glM9alMTnCuTMDAtofVR/NVPoxZNIsNzf k4DgX7/ZKCeADRdEUy/tb1tOjQJL9ZOoLfb3oYVp14gb65qJFQQMVIoMbICuveZ0ljuV3XZBF5S hCeGpaotd3OFr3DC3BBV7UZLXV8edCFTtroRovLKBMw1F14+9NxX3yiqgwWK35P1PlOl8fueb/2 k/42dgEXC6Kdbe8AJn9cIgm22McWFoV3P7aA2+UvT3oLA86uFvcDlvlssUUwZmIiD+iITa5rkRc tSO0NlK1sGsGGrEuokvHwq3Ww== X-Received: by 2002:a17:902:ebc7:b0:2ca:e565:7b15 with SMTP id d9443c01a7336-2d5fd73d03dmr58751175ad.10.1787133182575; Wed, 19 Aug 2026 02:53:02 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.52.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:02 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 01/17] mm/sparse: relax struct mem_section size constraints Date: Wed, 19 Aug 2026 17:51:23 +0800 Message-ID: <20260819095140.17252-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct mem_section is currently forced to a power-of-2 size so the section-to-root lookup can use a mask instead of a modulo. That requirement makes future extensions harder than necessary: adding a small field can require configuration-dependent padding or layout checks just to preserve the lookup scheme. Keep the lookup correct for any struct mem_section size by using a plain modulo instead. Do not leave the layout entirely unconstrained, though. Keep struct mem_section double-word aligned so modest size changes, such as adding another word-sized field on 64-bit systems, still keep a compact and efficient layout. If future fields grow the structure beyond that sweet spot, the lookup remains correct; only the exact layout efficiency changes. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Align struct mem_section to power-of-2 where possible, but support non-power-of-2 lookup (suggested by David Laight) - Collect Acked-by from Mike Rapoport --- include/linux/mmzone.h | 11 +++-------- mm/sparse.c | 2 -- scripts/gdb/linux/mm.py | 6 ++---- 3 files changed, 5 insertions(+), 14 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff5416..0a2428714108 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2027,13 +2027,9 @@ struct mem_section { * section. (see page_ext.h about this.) */ struct page_ext *page_ext; - unsigned long pad; #endif - /* - * WARNING: mem_section must be a power-of-2 in size for the - * calculation and use of SECTION_ROOT_MASK to make sense. - */ -}; +/* Sacrifice minor padding space for efficient lookup. */ +} __aligned(2 * sizeof(unsigned long)); =20 #ifdef CONFIG_SPARSEMEM_EXTREME #define SECTIONS_PER_ROOT (PAGE_SIZE / sizeof (struct mem_section)) @@ -2043,7 +2039,6 @@ struct mem_section { =20 #define SECTION_NR_TO_ROOT(sec) ((sec) / SECTIONS_PER_ROOT) #define NR_SECTION_ROOTS DIV_ROUND_UP(NR_MEM_SECTIONS, SECTIONS_PER_ROOT) -#define SECTION_ROOT_MASK (SECTIONS_PER_ROOT - 1) =20 #ifdef CONFIG_SPARSEMEM_EXTREME extern struct mem_section **mem_section; @@ -2067,7 +2062,7 @@ static inline struct mem_section *__nr_to_section(uns= igned long nr) if (!mem_section || !mem_section[root]) return NULL; #endif - return &mem_section[root][nr & SECTION_ROOT_MASK]; + return &mem_section[root][nr % SECTIONS_PER_ROOT]; } =20 /* diff --git a/mm/sparse.c b/mm/sparse.c index 7c15406e77f5..c84b4c7b8c70 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -322,8 +322,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count =3D 1; int nid_begin; =20 - /* see include/linux/mmzone.h 'struct mem_section' definition */ - BUILD_BUG_ON(!is_power_of_2(sizeof(struct mem_section))); memblocks_present(); =20 if (compound_info_has_mask()) { diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index dffadccbb01d..da4e8e9655a6 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -70,7 +70,6 @@ class x86_page_ops(): self.SECTIONS_PER_ROOT =3D 1 =20 self.NR_SECTION_ROOTS =3D DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.= SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK =3D self.SECTIONS_PER_ROOT - 1 =20 try: self.SECTION_HAS_MEM_MAP =3D 1 << int(gdb.parse_and_eval('SECT= ION_HAS_MEM_MAP_BIT')) @@ -100,7 +99,7 @@ class x86_page_ops(): def __nr_to_section(self, nr): root =3D self.SECTION_NR_TO_ROOT(nr) mem_section =3D gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] =20 def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT @@ -249,7 +248,6 @@ class aarch64_page_ops(): self.SECTIONS_PER_ROOT =3D 1 =20 self.NR_SECTION_ROOTS =3D DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.= SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK =3D self.SECTIONS_PER_ROOT - 1 self.SUBSECTION_SHIFT =3D 21 self.SEBSECTION_SIZE =3D 1 << self.SUBSECTION_SHIFT self.PFN_SUBSECTION_SHIFT =3D self.SUBSECTION_SHIFT - self.PAGE_SH= IFT @@ -304,7 +302,7 @@ class aarch64_page_ops(): def __nr_to_section(self, nr): root =3D self.SECTION_NR_TO_ROOT(nr) mem_section =3D gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] =20 def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF04A442B31 for ; Wed, 19 Aug 2026 09:53:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133204; cv=none; b=E6AJiSRQBpBtuSOK/ghyCB2YhiWgrGQR2dV88O2+AilKUQf9qmGh+fnkaUEC3ijIz8KescKNAOkYC6o3VfO26+bDJLKqPypAtPd7iXYmiqLvDBPRdlPS0cThWYnsQ/Frda8E4THKmFCy/HJ2L1qz9+jfQ/fLSJBw+MLKS5/O+M4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133204; c=relaxed/simple; bh=MOJGF39gfm06Lr6EFYc39cHGviZ2sFH9xZv8FR1yhM8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K6+IWP5V5jUNHST8iKTx+2bI5xt1qixePI85fyh8YHlEJUmPo0RTM1MQB+YuLkmXYSoBMxYAeri1RZIr3KRv5cnoqifOYMCMOhza+PYmyucC2L0cvc1R7EL7rRvRaxvQgmwxoHjo6Nn8QAtpSMm0cZCVG4t/iresULrA9+SnUQI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BLeGc6R+; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BLeGc6R+" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2caced6038eso7247235ad.0 for ; Wed, 19 Aug 2026 02:53:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133186; x=1787737986; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=pICFP83t3xFRgZ5Z8POKVzeGGbOgsBvO3I/0k/S2WzI=; b=BLeGc6R+nrCs4ykuk08Myl4mHnJgxHJUJeov2CKExaOLHoc2mj2DqynUGpWY6L0Nv7 cx459mk3UrX5sHpRL2+oA4ki0sMz1xTtgm8fpzLcevA04Z+1vrC6Hv5Yi/iQ31mictsM jDL6tTzqOD9rJ09UAi3s6RRwAd0L0taPXA802GivUhueIqQ5VKl7VbUlG9W5uZfevv/Z WIsKg5mkC1ZcMezgIu/lutjTjVE2YiXXJsUQzM/Xj/g8OHZkft5Ltj4O4h/JER1qKXR8 /KFr1xD9J85DY4PjUc4jG4MghSdI6e5fVq6JD/x1Ska5GrncxKT8x5Zbhl7YGF9+41ej UCWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133186; x=1787737986; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=pICFP83t3xFRgZ5Z8POKVzeGGbOgsBvO3I/0k/S2WzI=; b=bKagTAg7EPZAxYLLyI4Yb6s7R84P+RaZeaRFS7ZxEkKXkfoDe+N25rXAsms8aFaYUD wNv9iNUHBDdvVOHJW3QtP3a3CEUEXBL6AKUka4SIe9u81BlOsn/x+L6myDLV6yxVME2d T/2/JDdC3+OvQC/dIAKN3Z7eSohNUjpXkw7eTwDwiyhwmSJdddFZhovZNr8B9KuiLzUj YDYOQHKmCsCpaeEHl/2St4I+HKynEGG/EjjATC4Ox3GZq1M39O1o2deTKn7P+5chMe8x fP1mOcRyuNOE0c9Mm8dhSXznNgj6CVKxF0KRA2zz1bRDz2owwMe0KCs0e1+eDSp5z3ta 6iAQ== X-Forwarded-Encrypted: i=1; AHgh+Rqnl2gudDykwxhLKbhze0BJHh+XmFKsBjd82QKYTvWbmZe29P1YVkArdUnLqaP/claGM6OFmyWmu5/8tko=@vger.kernel.org X-Gm-Message-State: AFuF++ll4/d/hy4L5Hb0MseXuRVzGzMHXOLq+jazR3O1Byfccp4vEuwa rX2XG1eXurWZHtm8vmt8FQa9QURbjYUJT5qq0XMmsSasYyoCUWa3EmQduv8XBKPHknE= X-Gm-Gg: AR+sD103uAMZj8PopjD7aSOH3qEyH8752oM1fiGphCPPgD3mSYr7228RCA9oIKw6S+o OXrX17Uik0Qg+oyO2j9Ebh3XttqoUvjw2eBtL/gOLODaAq6LBNRN0pMvVrw1Q2h4O9CKR3pG0b0 8llH6BaAlUQ/2YFDeCUpQH+E8Hy7M4AYWbteRfr3BfpXT4+F9dfjHNX0vPuwZIvFR0X0PQ3NHX5 DEFsjep/xY+avTHHyAQS0sGyljGwv+gBzAa+Rz94KT1gSL15ifkYVicxu3i/3Swzv8sLYf+1Dx0 h+ounAHrBSfY55ovwumXkb64KnwMl8M6Slf7ejFpT3h4qi5/ohrHAmNs1cSUj2J23jlgCRQogCP JrhAshWdN12yt1N/lELdLvLs3oDeXZGQdZ2qpiMmMc3dD9WTPnuRZlv516yCWg+We+eJEabskeP jXYA1IX00athEbO5WtX5M/AsJqAhjhh/JSD8QDV2YqhYtydnudE3JxKqrcNr4PqP9TRWLyU/peU e1x/lHJcKjfZ0A1/MYKPLTYMg== X-Received: by 2002:a17:902:da2d:b0:2d3:7122:833 with SMTP id d9443c01a7336-2d5fca53e5dmr41430365ad.15.1787133186380; Wed, 19 Aug 2026 02:53:06 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:06 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 02/17] mm/sparse-vmemmap: rename HVO order macros Date: Wed, 19 Aug 2026 17:51:24 +0800 Message-ID: <20260819095140.17252-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The macros VMEMMAP_TAIL_MIN_ORDER and NR_VMEMMAP_TAILS describe the order range where HVO can be applied, but their names tie that range to the tail-page cache implementation. Rename them with a VMEMMAP_OPTIMIZATION prefix and use the new names in the HVO paths. This makes the code describe the optimization requirements rather than the tail-page cache implementation detail. No functional change intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v3: - Collect Acked-by from Mike Rapoport v2: - Rename the macros with a VMEMMAP_OPTIMIZATION prefix (suggested by Mike Rapoport) - Drop intermediate optimized-folio size macros (suggested by Mike Rapoport) --- include/linux/mmzone.h | 17 +++++++++-------- mm/hugetlb.c | 4 ++-- mm/hugetlb_vmemmap.c | 2 +- mm/sparse-vmemmap.c | 4 ++-- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 0a2428714108..5fb9b37819d5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -107,13 +107,14 @@ is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 -/* - * vmemmap optimization (like HVO) is only possible for page orders that f= ill - * two or more pages with struct pages. - */ -#define VMEMMAP_TAIL_MIN_ORDER (ilog2(2 * PAGE_SIZE / sizeof(struct page))) -#define __NR_VMEMMAP_TAILS (MAX_FOLIO_ORDER - VMEMMAP_TAIL_MIN_ORDER + 1) -#define NR_VMEMMAP_TAILS (__NR_VMEMMAP_TAILS > 0 ? __NR_VMEMMAP_TAILS : 0) +/* The number of struct pages covered by the retained vmemmap pages with H= VO enabled. */ +#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (PAGE_SIZE / sizeof(struct pa= ge)) +#define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STR= UCT_PAGES) + 1) + +#define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ + (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1) +#define VMEMMAP_OPTIMIZATION_NR_ORDERS \ + (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS = : 0) =20 enum migratetype { MIGRATE_UNMOVABLE, @@ -1158,7 +1159,7 @@ struct zone { atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct page *vmemmap_tails[NR_VMEMMAP_TAILS]; + struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c015b6a65894..a018d0da4ca7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3344,7 +3344,7 @@ void __init hugetlb_bootmem_struct_page_init(void) struct zone *zone; =20 for_each_zone(zone) { - for (int i =3D 0; i < NR_VMEMMAP_TAILS; i++) { + for (int i =3D 0; i < VMEMMAP_OPTIMIZATION_NR_ORDERS; i++) { struct page *tail, *p; unsigned int order; =20 @@ -3352,7 +3352,7 @@ void __init hugetlb_bootmem_struct_page_init(void) if (!tail) continue; =20 - order =3D i + VMEMMAP_TAIL_MIN_ORDER; + order =3D i + VMEMMAP_OPTIMIZATION_MIN_ORDER; p =3D page_to_virt(tail); /* * prep_and_add_bootmem_folios() can access pageblock diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 917db0984143..ae8fdaa42118 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -494,7 +494,7 @@ static bool vmemmap_should_optimize_folio(const struct = hstate *h, struct folio * =20 static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) { - const unsigned int idx =3D order - VMEMMAP_TAIL_MIN_ORDER; + const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; struct page *tail, *p; int node =3D zone_to_nid(zone); =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 5a2469fb1838..aa6a4a2fae98 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -329,12 +329,12 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * unsigned int idx; int node =3D zone_to_nid(zone); =20 - if (WARN_ON_ONCE(order < VMEMMAP_TAIL_MIN_ORDER)) + if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) return NULL; if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) return NULL; =20 - idx =3D order - VMEMMAP_TAIL_MIN_ORDER; + idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; tail =3D zone->vmemmap_tails[idx]; if (tail) return tail; --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 575C844A403 for ; Wed, 19 Aug 2026 09:53:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133211; cv=none; b=uQ/RntKWIUEOVXF/6g3avmnK9eSmRTFYRuMLJfUQl2IgWYnTysgXYkHMWGZfJqkvrqCMI2i6DSg9MWguQ8KW8yOOiEhkhGF4pgKHXQr20gMxdKeMub86NPdWvoEZwq5aQQA4BQ2ArtO8jCPvz136tLX4rRA8MEHYVbe7iTdwQ6A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133211; c=relaxed/simple; bh=DgUQ1rM24/xeGG2KTVaP6pulO3dlsX+pLmHNSEY8Vb4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T6V0jnDvG3ppwPjAICkuYW+IbdP1iES//XOrfypRjw/BK67Qo+Nov6iBFvLF77FQa4GlvP3wco5nd3IadO1pEB1KWzi8WPdszLJTGBoMeRYFh57Izz2SxsIEE5msxvKPf1Ck5JVfucnBq7xT2zcFg4JiHq7AtHIyAU3LeF+DtAI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RoVqS6OY; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RoVqS6OY" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2cc891373e0so9416355ad.2 for ; Wed, 19 Aug 2026 02:53:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133190; x=1787737990; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=20tETBVEF5Pi+yuhDabHc5ZMMKGnCZY104Vd+m53HpU=; b=RoVqS6OYLUxmVO0rgcTfWN5+SwqRXzRiHGDG0Z63nRZkDCt27wkQPjOKHbTJ+tdab7 Ha2KHDNXMg6QGuSkutcjdyYKJrTBAx6E4QPENs6zcwDxc1qHRs8KPDX8MyKoTmiPKPm1 6QAblctZbo4OvTgX6XWnmbIAqMM7uzPku9YIsPUVb04P9eRha3ZB2/DsBN47//6BPBv1 iSDSqwAUZV6ci6iZmGTVeTqVrB9k1u4ruAdPRhAaiVf/sVjLtnpW045pShzCYvGZXj59 u/jrqqh9B0+EsJ8ynEOP4WqLSnG6RTNSnHSnsTf5SLhRfl2cyEnqqz1vtayM2Gn6x3G2 0sbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133190; x=1787737990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=20tETBVEF5Pi+yuhDabHc5ZMMKGnCZY104Vd+m53HpU=; b=mxG8UyrR/k5tCH3yvZJ4ehH7DJ2kJsMYz/8RZzkpU76XKnMyutZnXmDUKkMnNvDxNl dWhpwLFEfK5RaxKsZW4HjhKmp0iRV4GGphyFmFG5xV4o5RGEtTbtJcnyegjolSXOYpv6 fkL/CZPAmQtA5uA8HkvIkFRFTbMntEU9cqKimSCleT+XzAjkZaerss1kCUHbaFXBqqbC U9DBI0BkxRw1UvYBhy5FqNr+ufTfOOL5oBU2ZglVCelSdutrOATHnnDjQWobBBgX/TDQ ADBOe8jopQxZKhtCct7SyB2Lel/BnZTIfh3ec5mYQ+Utq3LpegorvkDR31Iuasvw7mCe 8M2Q== X-Forwarded-Encrypted: i=1; AHgh+RoqKCaEsOC0uIWl57nCIvD39gibpV20TSnvTu8D/0AzQsPwgYOyG4W5rC76obYmjsqjyDsulWj+dA1CrEs=@vger.kernel.org X-Gm-Message-State: AFuF++mM0SZSfHGnRSMaDwI7JANQdfHtkzc1JZtJZ4bVFACDaEdmFJBo mgPXZDmscZrJ6zVDvHWtPmYLhk1QRrolicpzgg3SW8KdIpL1akaSJiaswuaJn/7LUoQ= X-Gm-Gg: AR+sD11IX1caz+lzypNjHalw9HeKRZysUvfFNY2os4vQqBLEp0iI6s4VZsDK7IsXdfk YwB2G7gUGn7hRn90DWomzZBBvg1WV6PVDBVTiFWFhjlLPW3kJ/aNDx8QQmZhQjvXvVSiRKhD4Oi H1NEnoEJDR/ohaqtbVVaC1CwT0dTBFbra0s6y2YiiLr33m1qXOu1iHT8NdYzAIyXgqznZCP9YL6 fglz2kduTNoGI3FimFE4NbVPMaCxO1HKAHa6ylOuFx0od1A9wHPYdC8HNmY+o5KxVEwG80OLDy8 wvFpKhYAbguk9aaa2upfkwiE2TrMe19JvzC75G3udge1sUZ1tyVuEMhjtK9lmocei1IdgnvsQiK e5hLI8zuaNwWTRt1fRmSt+8Z/BHPvAoooC1OrPLSb/i6EJzVbWLGK/yv8Iaii0vAh8cKzyIofOG 3iFdYRJEij+RU4O7DtYJrULEr+tAeb+bCSkzNr36cvx2qiIOGt5UYaYL5EUobKKNuIAYKsHWqMp ilTyT6YpxKopmb17HpUgqMoehYI2Z64p4d4 X-Received: by 2002:a17:902:d2d0:b0:2c7:ebfb:618f with SMTP id d9443c01a7336-2d601985763mr48725705ad.14.1787133190083; Wed, 19 Aug 2026 02:53:10 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:09 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 03/17] mm/mm_init: skip initializing shared vmemmap tail pages Date: Wed, 19 Aug 2026 17:51:25 +0800 Message-ID: <20260819095140.17252-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memmap_init_range() initializes every struct page in the target range. For compound pages with vmemmap optimization, the tail struct pages are backed by a shared vmemmap page. Initializing those tail struct pages would overwrite the shared vmemmap page contents, requiring users such as HugeTLB to restore the metadata afterwards. Track the compound order for HVO-backed sections and use that metadata to detect struct pages that fall into the shared tail vmemmap range. Skip those shared tail pages in memmap_init_range(), then initialize pageblock migratetypes for the processed range with a helper after the per-page initialization loop. Keep direct mem_section access inside sparse helpers by exposing pfn_to_section_order() to users that only need the order associated with a PFN. This lets memmap_init_range() skip shared tail vmemmap pages without exposing __pfn_to_section() to !SPARSEMEM builds. This is a preparatory change for consolidating handling across users of vmemmap optimization, and it also avoids redundant initialization of shared tail vmemmap pages during early boot. Signed-off-by: Muchun Song --- v4: - Rename pfn_vmemmap_optimizable() to vmemmap_optimizable_pfn() for consistency with vmemmap_optimizable_order() v3: - Replace the !SPARSEMEM __pfn_to_section() stub with pfn_to_section_order() (suggested by Mike Rapoport) v2: - Fold section order tracking into the first user instead of keeping a standalone API-only patch (suggested by Mike Rapoport) - Rename page_vmemmap_optimizable() to pfn_vmemmap_optimizable() and pass a PFN directly (suggested by Mike Rapoport) - Initialize pageblock migratetypes from a helper after the per-page loop (suggested by Mike Rapoport) - Use a 1G PFN chunk for cond_resched() in the pageblock helper (suggested by Mike Rapoport) - Guard section_order() with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP so it returns 0 when HVO is disabled and lets the compiler optimize the code as much as possible (suggested by Mike Rapoport) - Explain why the !SPARSEMEM __pfn_to_section() stub belongs here (suggested by Mike Rapoport) --- include/linux/mmzone.h | 8 ++++++++ mm/mm_init.c | 34 +++++++++++++++++----------------- mm/sparse.h | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+), 17 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5fb9b37819d5..df31cac12311 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2022,6 +2022,14 @@ struct mem_section { unsigned long section_mem_map; =20 struct mem_section_usage *usage; +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + /* + * Normally, sections hold regular (order-0) pages. However, for + * sections with HVO enabled, this tracks the compound page order + * to enable deduplication of redundant vmemmap pages. + */ + unsigned int order; +#endif #ifdef CONFIG_PAGE_EXTENSION /* * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use diff --git a/mm/mm_init.c b/mm/mm_init.c index 1533aebafb68..05c09e755e0b 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -677,21 +678,19 @@ static inline void fixup_hashdist(void) static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ =20 -#if defined(CONFIG_ZONE_DEVICE) || defined(CONFIG_DEFERRED_STRUCT_PAGE_INI= T) static __meminit void pageblock_migratetype_init_range(unsigned long pfn, - unsigned long nr_pages, int migratetype, bool atomic) + unsigned long nr_pages, int migratetype, bool isolate, bool atomic) { const unsigned long end =3D pfn + nr_pages; =20 for (pfn =3D pageblock_align(pfn); pfn < end; pfn +=3D pageblock_nr_pages= ) { enum migratetype mt =3D kho_scratch_migratetype(pfn, migratetype); =20 - init_pageblock_migratetype(pfn_to_page(pfn), mt, false); - if (!atomic && IS_ALIGNED(pfn, PAGES_PER_SECTION)) + init_pageblock_migratetype(pfn_to_page(pfn), mt, isolate); + if (!atomic && IS_ALIGNED(pfn, PFN_DOWN(SZ_1G))) cond_resched(); } } -#endif =20 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) @@ -886,6 +885,13 @@ void __meminit memmap_init_range(unsigned long size, i= nt nid, unsigned long zone } } =20 + if (vmemmap_optimizable_pfn(pfn)) { + unsigned int order =3D pfn_to_section_order(pfn); + + pfn =3D min(ALIGN(pfn, 1UL << order), end_pfn); + continue; + } + page =3D pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); if (context =3D=3D MEMINIT_HOTPLUG) { @@ -897,19 +903,13 @@ void __meminit memmap_init_range(unsigned long size, = int nid, unsigned long zone __SetPageOffline(page); } =20 - /* - * Usually, we want to mark the pageblock MIGRATE_MOVABLE, - * such that unmovable allocations won't be scattered all - * over the place during system boot. - */ - if (pageblock_aligned(pfn)) { - enum migratetype mt =3D kho_scratch_migratetype(pfn, migratetype); - - init_pageblock_migratetype(page, mt, isolate_pageblock); + if (pageblock_aligned(pfn)) cond_resched(); - } pfn++; } + + pageblock_migratetype_init_range(start_pfn, pfn - start_pfn, migratetype, + isolate_pageblock, false); } =20 static void __init memmap_init_zone_range(struct zone *zone, @@ -1112,7 +1112,7 @@ void __ref memmap_init_zone_device(struct zone *zone, compound_nr_pages(pfn, altmap, pgmap)); } =20 - pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, fa= lse); + pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, fa= lse, false); =20 pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); @@ -1921,7 +1921,7 @@ static void __init deferred_free_pages(unsigned long = pfn, if (!nr_pages) return; =20 - pageblock_migratetype_init_range(pfn, nr_pages, MIGRATE_MOVABLE, true); + pageblock_migratetype_init_range(pfn, nr_pages, MIGRATE_MOVABLE, false, t= rue); =20 page =3D pfn_to_page(pfn); =20 diff --git a/mm/sparse.h b/mm/sparse.h index 3b744667a7e6..1fcda8a1c270 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -10,6 +10,39 @@ =20 #include =20 +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static inline unsigned int section_order(const struct mem_section *section) +{ + return section->order; +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return section_order(__pfn_to_section(pfn)); +} +#else +static inline unsigned int section_order(const struct mem_section *section) +{ + return 0; +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return 0; +} +#endif + +static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +{ + const unsigned int order =3D pfn_to_section_order(pfn); + const unsigned long nr_pages =3D 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + /* * mm/sparse.c */ --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 204DA44C4E1 for ; Wed, 19 Aug 2026 09:53:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133206; cv=none; b=XU8081sewFCEvFBc7oGxFXT4fmR6HKaJ6bv7aMIxqaeHTHMeGlmr1EepS8aTGA1Jwx6rwgM4CkQMs379kFL+tutoT4VDyT+41JIoEAokO11FXUoxJgHKRbhvRRVzgUONGoKuPbLjsCTNR6FucjsBeR5lTnBHdz3pkJUfe2Ckmeo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133206; c=relaxed/simple; bh=owQ+Elu7FhHy0RmBaGNrJ1F4/GpIAVuOEqAZSW69+JA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o7vD2aHnAmeW1Of5ccnWe2MkpBm4Vvg3psVJY/d/cHd+SKqTAOwPR/BSGCnfIL2ULTWZYMpI3Y1pnGPGu2qidA1SuvR6E4tXG6zo8WEWtLyDQA1VWgM5ApMtbTzuD77ChzKvVz5llvu592wpeLjUh3SdivAv2/KygORd1v7ax3w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=QwlN7GHC; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="QwlN7GHC" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2ce87c7e3bbso9354815ad.1 for ; Wed, 19 Aug 2026 02:53:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133194; x=1787737994; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bOzdkIPPNoupexe8+AVOrQT3DBRmXPJ6Kybkle+RLi4=; b=QwlN7GHCGuvH+rYYvuTWnUGRdJFq+B88j2Q0tOB5NP0i6+MgIqgUa8EO9N5IDzRZQA frrFmx9AgxnVyPQJCFAJib9PJQKbDeGndhkUnWdWcNlC+m98ufPoppJObZT45K/iH3e1 vUvpdJ40OEA/tXHh6craYNklO8sdtqBIQYelMeEJ6L4i4SHj6hkvdHpH1B+XAZexf/Mp TvI7hAn39bhXlPEo/TQSPbfSUMch7tkFmENjJqKIGOGKLT2mw0RQYz2jFCTNG863VgMM PET12qFv8sJDC05MjwiCZBLLmWFS/fw7zgGhxkddNt6+/BzHvvvp3xhJ8ovQoTBAPwzw TY2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133194; x=1787737994; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=bOzdkIPPNoupexe8+AVOrQT3DBRmXPJ6Kybkle+RLi4=; b=A5rHsG3yn2YGS70T+ETcmEXVuDWtLcRO7Nhhlo/xC2WzWsez1oON8LlY+rFD7rJbnh FsY0LQ9EQ8ggFhkyMf1x18Xz3Ex5x4Atoxlo72lyFSt4UlgHTJRRTED5FKp7eUXCVXoB 7uZu9xF2uCWTz1Waqm/OVfYfCSGLabPaI3D3IEUMZGS14/Dhk7zXop7L3pU7QMyctpZy Xcz0y2lQ2SuQrIb0s14AtRRqX2JU0aT/6FzK+XcqYB01MfuzCBrYBMeGteaLMTsP78IY yqVZhGcCyjfwhLj8XDYD/2JB/1g4u6o5Gju+4SEp5H5ehytM8hGgxAovlUQl7LpVZfx8 QCJg== X-Forwarded-Encrypted: i=1; AHgh+RrqGT6fGvZEg7sIkYzIXoGw/GoYb2zuf0W+B6E84fblHo+txswRVEYkg2rKMoPRTnBOckzD+SqD96yPjqs=@vger.kernel.org X-Gm-Message-State: AFuF++m2KdLCd3i1WO83cSoqlbU9uJ9DwkIlqQVFJebBP+muy6VLG/eG qwIpYG/YUgjNMDi2IlPCKhhfUpaS8+1edfqBxXga26cC1Ml01a3+sir7bxo34dg8UFw= X-Gm-Gg: AR+sD12M0W0xW9RAqlhkwDdxV40Ua/5yDMiuIm97O91zs+1xqDnpmgYsmT9mghcKERC HT5tZqelVUNirAUgsXKAIgsVe/KdYXhPOaNv60g/YbkWyknuc645CgwFURAWPZ4SOcHhYLdSdic DCzBCMFzIQQf2LbHkAplHqo0girtVRegCQKNj7dofbYynJYlI1CsN3VA9F3TdRjxb4pyHumEpPh SaPoGtq63z3WDTUAPVva2bXisDKNnvSP/8IZXWe9keZ4fBRoRm7PDYDYZd8VVRX7pM8rysyYXNb DVZcWAhEhQugBgiES401UzXYp3ue7n45r8krxyRlHfbvrUcE61V+gidE7BFkoGqG8vB+ATdrDSJ JsGu+ivLf8o4V1sIYhR6SPEY6dA7JS4nRdFqcpixyxYl0vIxfjCQmzfGWkmAeeiAfjaCnZX+oYY eI1t2trLjk0Z2eOac/0cV0eQAI2n+TGuTqpCbXLH+f/rSl8ClLrY52vRPEkto6xta3J4SlWyGrE 5489Z3YPmUHci0YR8aQB3ij0Q== X-Received: by 2002:a17:903:b4e:b0:2c8:4c29:afeb with SMTP id d9443c01a7336-2d5fd674a51mr64709405ad.8.1787133194077; Wed, 19 Aug 2026 02:53:14 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:13 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 04/17] mm/sparse-vmemmap: initialize shared tail vmemmap pages on allocation Date: Wed, 19 Aug 2026 17:51:26 +0800 Message-ID: <20260819095140.17252-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The shared tail vmemmap page allocated in vmemmap_get_tail() used to be left uninitialized, because memmap_init_range() would later overwrite it. That forced users such as HugeTLB to defer the initialization to their own setup paths. Now that memmap_init_range() skips shared tail vmemmap pages, initialize them immediately in vmemmap_get_tail() with init_compound_tail() instead. This moves the initialization to the point where the shared tail page is allocated and avoids relying on deferred handling in individual users. The remaining deferred initialization in HugeTLB will be removed once it switches to the section-based vmemmap optimization mechanism. Signed-off-by: Muchun Song --- mm/sparse-vmemmap.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aa6a4a2fae98..107215cf8488 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -338,19 +338,11 @@ static __meminit struct page *vmemmap_get_tail(unsign= ed int order, struct zone * tail =3D zone->vmemmap_tails[idx]; if (tail) return tail; - - /* - * Only allocate the page, but do not initialize it. - * - * Any initialization done here will be overwritten by memmap_init(). - * - * hugetlb_bootmem_struct_page_init() will take care of initialization - * after memmap_init(). - */ - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); if (!p) return NULL; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail(p + i, NULL, order, zone); =20 tail =3D virt_to_page(p); zone->vmemmap_tails[idx] =3D tail; --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D88A44A409 for ; Wed, 19 Aug 2026 09:53:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133213; cv=none; b=gcaL71ckIoEW/iGJnowI1qdCtH4c6+H/Xtk3IqMO5gNfU5kdBS/dEZGUkLXh0Ph9sscHEfWLkg2balTsYkQbTG6qoKVEqAZCqOZCDJjO5aMo0cCis8xQdgKVddLeF9MK5cVEAL993E2LP6BsSAt8Xg7SJSQmHoXhPPDg8skU2g8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133213; c=relaxed/simple; bh=HO2/UsE4/xA12G7GUVatbvK/9WV1SEM2PIF87wmx8wg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZOSCEvjEitGZFmlEivzLpvyepPKnkeMP+8Xq2vucS57aymHN+XFFx7qZpzwDYIxhUv3YbXraO8dIxkwJcBw6Uljnxh3cImXoYe/ZJSE0TjCuxGI2SPbH7QZzxTHdVHCPORH4V3TtRDKxyw60lj53HoocRdo0pgMZ7gUlhP4/mnw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BJ0jwLBr; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BJ0jwLBr" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2ceb096e675so9343175ad.0 for ; Wed, 19 Aug 2026 02:53:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133198; x=1787737998; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=s4XhcWcLOxh7vr8ZmuE0kXQ1fb1jPl0YDGFVXv1M41Y=; b=BJ0jwLBruYKG2cdNa0QDB5vwY/hjB6GASuKFwzYgCCgMDGognDWpNyZGBq43M5PQa5 m+3T/OQu9A0LY40oloInrmnGJwiJgScHEqK9a0tCrP12Wz30Ps2aokLKUaIS1FWHgWj+ 9GDktLsAFvYxVmbcU3ggl35Bi7wxE85eVCPSm++jxWM/9SnAOC8cZWYBxsGxY1aaDWhp e2qGeLkaFL4hs+VkbGKHoF/E98fKAZvtyIwsI7nh1jCDIS0Z+PNoe3cenBza0TaWw1d8 fdpMrOxbv8lB9WU3sTlRiTeMQ+/s4bAzLzH1/Z8cyf/wCbTwRl00qCDHfoChrzi/YrvS qdcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133198; x=1787737998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=s4XhcWcLOxh7vr8ZmuE0kXQ1fb1jPl0YDGFVXv1M41Y=; b=LeId12keGd43VfcpAMrBTwmiN1+Eoa/IG7drDtX2pgdwIhVY6ijSX2sYNBIRka1iOW iPAE7x9qkA9t5rlDObLe6yVmEMskLIegvTWTBaTpNwJWDRfkwJJrHqHtEyvy0k3W1bLD v7e/ldrl8Rwk2azMKQ9XhoTqIBJcA5Srd+ceXlHF0qBDb1UwTqWtPo3O6dsnuG4vJcoc c0eZfJVsDNTREPkspWqAkRTEAhd/QQUl55tcKX36TZKjxYLl8LPs4nIRV9Snmyi03dRx HfHOzaX9aNawofWZ+EZJ5w91qtlG2vEEFG8jHJxMcDwfqh0OuPiKZ3xhmxWFzZSDwtaF biVg== X-Forwarded-Encrypted: i=1; AHgh+RqKI6wrMocMvXQl+Z/gtpxn3tWecYoCf0wCXCE4rua5T1QF8V2kr1R1jHiVScONVtj6WFikN4t9wybLNSA=@vger.kernel.org X-Gm-Message-State: AFuF++nkKXkrTBR4E8lHtEyjmN571AcJnW9MHbhQFOHnbk+0NtPQALYz 2oHyrHxcIkhZRyaAcmH4EGMXZGfEGBYdqunuGiq0xZdyUZEf2dJfXgcmE+EwUyYFtAg= X-Gm-Gg: AR+sD13J1xQJ5mny+j5iX7XJ/O0FPWJppigNM4kRcZBdzua8XSc4I9RDO9iI4mUVK3d PMJNrTekrxQkcHF9Hkic3XCQ0Y3YNUERVNYk6ZJAd9kTO5zTiZ059AE2mPzujzE3Te5WSNIAnQH xnuB2EpvBne4R1L4u6sSY3rydb180vQV0KiiUnNyd586YDYwXJWOcvaJvT8oBUgNwDtSSlGyqto sIwN3WKX2e976nCUt9SfDkf3ZpOAjeQjRzhNSOdEX3Zvtt/oO4otfle7i+g5MpRy61Wax5NcMLl UwTcELLrDX4CM7LYRBnwBMAgOLwqR89grG0+ByfkMYHi9vH5FJdGNMdKBrAxPygjh2txF7LnayX EWHqZDjXf7nLAM5LL0K1IPsBENGnckjCnqch3ZSNRuYQrfWFbuu79B4G/SwJikZyWugruLwm/7K 54ph1shwEZ64Qq7XkAu3v1O2pXfPRQdeDZOSimB6BWoCOwISSry5DcnE2V1zpoXugQUbY5nweW0 UHXrih7r1Ekhtq/ZcxcWMRRFTlBWm3pmzPm X-Received: by 2002:a17:902:d4c4:b0:2d3:89fb:35fb with SMTP id d9443c01a7336-2d5fd3df2bbmr55030125ad.0.1787133197688; Wed, 19 Aug 2026 02:53:17 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:17 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 05/17] mm/sparse-vmemmap: support section-based vmemmap accounting Date: Wed, 19 Aug 2026 17:51:27 +0800 Message-ID: <20260819095140.17252-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_nr_vmemmap_pages() can account ordinary sections and DAX sections, but section-based vmemmap optimization keeps its compound order in struct mem_section and retains a different number of vmemmap pages. Teach section_nr_vmemmap_pages() to recognize section-based optimized sections and calculate their vmemmap page count from the section order and the HVO retained page count. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v3: - Add vmemmap_optimizable_order() for order-based optimization checks v2: - Remove an unnecessary vmemmap_can_optimize() call to simplify the code (suggested by Mike Rapoport). - Rewrite the commit message for better understanding. --- include/linux/mmzone.h | 6 ++++-- mm/sparse-vmemmap.c | 10 ++++++---- mm/sparse.h | 16 ++++++++++++++++ 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index df31cac12311..177455d98064 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -107,8 +107,10 @@ is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 -/* The number of struct pages covered by the retained vmemmap pages with H= VO enabled. */ -#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (PAGE_SIZE / sizeof(struct pa= ge)) +/* The number of retained vmemmap pages with HVO enabled. */ +#define VMEMMAP_OPTIMIZATION_PAGES 1 +#define VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES \ + (VMEMMAP_OPTIMIZATION_PAGES * PAGE_SIZE / sizeof(struct page)) #define VMEMMAP_OPTIMIZATION_MIN_ORDER (ilog2(VMEMMAP_OPTIMIZATION_NR_STR= UCT_PAGES) + 1) =20 #define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 107215cf8488..b7abc5494bb9 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -649,24 +649,26 @@ void offline_mem_sections(unsigned long start_pfn, un= signed long end_pfn) static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - const unsigned int order =3D pgmap ? pgmap->vmemmap_shift : 0; + const struct mem_section *ms =3D __pfn_to_section(pfn); + const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); + const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); =20 - if (!vmemmap_can_optimize(altmap, pgmap)) + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); =20 if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return VMEMMAP_RESERVE_NR * nr_pages / pages_per_compound; + return vmemmap_pages * nr_pages / pages_per_compound; } =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); =20 if (IS_ALIGNED(pfn, pages_per_compound)) - return VMEMMAP_RESERVE_NR; + return vmemmap_pages; =20 return 0; } diff --git a/mm/sparse.h b/mm/sparse.h index 1fcda8a1c270..02ed0f34eac8 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -43,6 +43,17 @@ static inline bool vmemmap_optimizable_pfn(unsigned long= pfn) return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; } =20 +static inline bool vmemmap_optimizable_order(unsigned int order) +{ + if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + return false; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; +} + /* * mm/sparse.c */ @@ -86,6 +97,11 @@ static inline size_t mem_section_usage_size(void) return struct_size_t(struct mem_section_usage, pageblock_flags, BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS)); } + +static inline bool section_vmemmap_optimizable(const struct mem_section *m= s) +{ + return vmemmap_optimizable_order(section_order(ms)); +} #else static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3515D44A408 for ; Wed, 19 Aug 2026 09:53:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133219; cv=none; b=pTzkxiOQM3uzgs4ji7hwYl9Nbmh/8xm5/V6+/3HOEHs+z8JpeTZFxfuENtQ4/PJGT03DbkomZ3NsaCHHsXB28V9EsdFR+CjRnc7AvXxRVGWN8iQf6g362f3xg0dZenRG7D45igHlcEexwdikFqMZnDi7mOfQrOiLboTC23wBV9s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133219; c=relaxed/simple; bh=Gd4i6Y9jxvfKQkb5cJnO042mb8rrqz1CmfeQEiUTg1I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qPLuZzVuLQCIEwmrPqbPANOU5r9gK+F+d6l7WXycEJsTEixfH9LLv6MZniVE1pGhDXfPkbUJkBM9RZhZlo+eoAcqNX2wb6fi8ssIsCl4LFzDkDXALoSw7ROKQqXaXygyfn1MbEgaHcqwTfBcAjMiZqNbUfWrjSG4H8rjgoxfr0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=B3QosVcJ; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="B3QosVcJ" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2d5335cf904so6448945ad.2 for ; Wed, 19 Aug 2026 02:53:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133201; x=1787738001; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=rJXoGFcGTrfgHmFDIP9742DptQ/jUwqkzOwo8zU38YU=; b=B3QosVcJxck2gh4WpS9Ls6NnLvGtD76GpIO3wUNPNevac/7kGQBnvN09pSy6dyq0Id tTxT7XahQBVjGIyzMQIMwTA9ttOt3mW2mP/tx6vaiF90uHEPMwWLR7bEoKmt4EO8ByTp qGBb3xB3TFvAXfF0BhBtZsRYY3HxhCJ30/0KrrB2Dz8Cwg/6GaFIceLMpLk80D8FBDqa 2bxjTVq0mJq1G3pmcKu6N/xTl7p1fDFLujnlJi1LDDRm5Am6bnRbqBpY7ryEDuV03Qt2 v89PVM52VanVVc++2NBFkhCkQlBczyOoCK7O5euBTVTAwmBs0+jbsbJUuHdrqOEIV8Yx yF7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133201; x=1787738001; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=rJXoGFcGTrfgHmFDIP9742DptQ/jUwqkzOwo8zU38YU=; b=h5qAq2uBmgYzoh0DhJ3RaQ+/Shk3og3AelUuJ8B3mYcOcnW9Wg8tyHeIwaBD42AcZw kT+ca6Q1IlmZ4P1PNDc9dnonPV3lDg8r4y+x2Q2d3WV/o7ptvcCTSaZKOggWCwpxH4KJ aNL7ABLZdXVDR26R4t9Rc8X/J+Y4nIHsE8OIpqbsXlq5TZ5OoSsyPBs4wQXk2lgXxgbk PLYAUEG3P1lzWQuhzz16WDdEX0RUI5s4XaevaNGQNGHOrgaGjWzI8YPUExxS9mNjIRR2 6ebY0gpB3G3vKkR9l4HgZJ6i3cnxnKt86iF2osCvIkBpE2aDPPjX7iEzQwOl2Bnqgkjd +2sQ== X-Forwarded-Encrypted: i=1; AHgh+RoBYXz0eFTBEsy1kGXFln7bXJymuqQo0DMbhT4xEvmHrYW2c7y3lrLBO7PyT+200jCRmjxu+5dNHvo5lnU=@vger.kernel.org X-Gm-Message-State: AFuF++kCo4wYNOnjfqFfZ1O/LAozZDreeTRXjJe0edIcv+kjx3f8/A6H hRRgDdIDoopDQwbqBdoFQdT7ryZPZA6ZNCjG5WIlXYpBZMVaEqPAEfzAwSx4BUoafPo= X-Gm-Gg: AR+sD13P6BmWip+/9y0a7kjIkYrYWWSOhDNfCDiqYeHXFXyjNc857wUXUmWyLvGIhbz 2m5o9X1+X+6P0n5GVb9XJbfYEwVwoPlAVa6k1ciJ+d0r607b/XmfHeGa4I8n92kCRlu0nuxI4Xv KygNM6wLhZgfnSzqKHixHRjQQhZ+3LsvIeOex0QOvovF9lc5nkqrTq/Xjvqqxgd96sLRxvum8Vf mNFtHW2DGDlylq0zkOf37h8QSBJ+umsmdKOKpMWJZMLjuMyQGfAB69Dx56aaUrxXs8ZGVpja/jM rCTKf2kNlZeyzTwxv6aMX/AbUFJdQC5VdsZ/RH/0DKrTA3LSOJqCDDvvETO3VB8VT6buGY/TjP5 I3NspdqLSvOVSultRIT35fsnOqAH3vmbPSPz3IYbIrIRVcth28aLBUltPApO/R68lGVeylQWIDI rQvRgPupKUn5FsJGt7ObRmAvgU3AiQjO2lwuIqPIKeljIe2Pjxd/ynWIG1/c/Co1urZVENmDiqL Ueg2aE018CpXPToi+fHtSAt0w== X-Received: by 2002:a17:903:b83:b0:2ca:e5c:7fcf with SMTP id d9443c01a7336-2d5fd5eaca9mr65885345ad.3.1787133201498; Wed, 19 Aug 2026 02:53:21 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:21 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 06/17] mm/mm_init: factor out pfn_to_zone() Date: Wed, 19 Aug 2026 17:51:28 +0800 Message-ID: <20260819095140.17252-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" pfn_to_zone() in hugetlb_vmemmap.c duplicates the zone lookup logic in __init_deferred_page(). Move it to mm_init.c, declare it in mm/mm_init.h, and reuse it from __init_deferred_page() and HugeTLB early vmemmap initialization instead of open-coding the zone walk there. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v3: - Fix the commit message to name mm/mm_init.h instead of mm/internal.h - Collect Acked-by from Mike Rapoport v2: - Move this preparatory patch before the sparse-vmemmap optimization changes (suggested by Mike Rapoport) --- mm/hugetlb_vmemmap.c | 17 ++--------------- mm/mm_init.c | 28 ++++++++++++++++++---------- mm/mm_init.h | 1 + 3 files changed, 21 insertions(+), 25 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index ae8fdaa42118..c48fcea076a5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -19,6 +19,7 @@ #include #include "hugetlb_vmemmap.h" #include "internal.h" +#include "mm_init.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -744,20 +745,6 @@ static bool vmemmap_should_optimize_bootmem_page(struc= t huge_bootmem_page *m) return true; } =20 -static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) -{ - struct zone *zone; - enum zone_type zone_type; - - for (zone_type =3D 0; zone_type < MAX_NR_ZONES; zone_type++) { - zone =3D &NODE_DATA(nid)->node_zones[zone_type]; - if (zone_spans_pfn(zone, pfn)) - return zone; - } - - return NULL; -} - /* * Initialize memmap section for a gigantic page, HVO-style. */ @@ -787,7 +774,7 @@ void __init hugetlb_vmemmap_init_early(int nid) map =3D pfn_to_page(pfn); start =3D (unsigned long)map; end =3D start + hugetlb_vmemmap_size(m->hstate); - zone =3D pfn_to_zone(nid, pfn); + zone =3D pfn_to_zone(pfn, nid); =20 if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) diff --git a/mm/mm_init.c b/mm/mm_init.c index 05c09e755e0b..3625de86d1d9 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -692,6 +692,20 @@ static __meminit void pageblock_migratetype_init_range= (unsigned long pfn, } } =20 +struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid) +{ + pg_data_t *pgdat =3D NODE_DATA(nid); + + for (enum zone_type zone_type =3D 0; zone_type < MAX_NR_ZONES; zone_type+= +) { + struct zone *zone =3D &pgdat->node_zones[zone_type]; + + if (zone_spans_pfn(zone, pfn)) + return zone; + } + + return NULL; +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) { @@ -750,20 +764,14 @@ defer_init(int nid, unsigned long pfn, unsigned long = end_pfn) =20 static void __meminit __init_deferred_page(unsigned long pfn, int nid) { - pg_data_t *pgdat =3D NODE_DATA(nid); - int zid; + struct zone *zone; =20 if (early_page_initialised(pfn, nid)) return; =20 - for (zid =3D 0; zid < MAX_NR_ZONES; zid++) { - struct zone *zone =3D &pgdat->node_zones[zid]; - - if (zone_spans_pfn(zone, pfn)) - break; - } - __init_single_page(pfn_to_page(pfn), pfn, zid, nid); - + zone =3D pfn_to_zone(pfn, nid); + __init_single_page(pfn_to_page(pfn), pfn, + zone ? zone_idx(zone) : MAX_NR_ZONES, nid); if (pageblock_aligned(pfn)) { enum migratetype mt =3D kho_scratch_migratetype(pfn, MIGRATE_MOVABLE); diff --git a/mm/mm_init.h b/mm/mm_init.h index 39f75df9be1c..c9fc35e7e9f1 100644 --- a/mm/mm_init.h +++ b/mm/mm_init.h @@ -39,6 +39,7 @@ void memmap_init_range(unsigned long size, int nid, unsig= ned long zone, enum meminit_context context, struct vmem_altmap *altmap, int migratetype, bool isolate_pageblock); +struct zone *pfn_to_zone(unsigned long pfn, int nid); =20 #if defined CONFIG_COMPACTION || defined CONFIG_CMA /* Free whole pageblock and set its migration type to MIGRATE_CMA. */ --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6015449B09 for ; Wed, 19 Aug 2026 09:53:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133217; cv=none; b=uZWk6QaUkep+kUYcyyzzHGDbCRPg+y2m556q/ATwVIV6WePfZxZVDqyYiMO5aIlhjNKJd1PRCdQQa9vYNiKR1Znhl1iAYnxQhVk7UYoMsbHNQOREgqgjXWdJw73GTS0vFh+H+hc7k/PzECkBi+cCy2yNrk8Ez3ivqYC8AN+zOYY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133217; c=relaxed/simple; bh=cNXcXDZgPY6FKuMArv5HC7EpvCTKLUm1kC82mrQQeO4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JnQliq5fWO7ofo4WLPw3Abhr9zXdX2ogYDVI+murBVeu09czAUlKdN6xFdkPzZwRyr6NGh26n0yPjw6VSJYjV6j19voxxYH0pFC5XxJ7eTa4rrHWxbizQiXb+RLvhCoPYReY++sXGRR9+pS1Axy3qW+Rc1YttQtI2p3O7Gsvh6A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BUyd2Mfl; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BUyd2Mfl" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2cc61541f8cso22580225ad.0 for ; Wed, 19 Aug 2026 02:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133205; x=1787738005; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qLDW1MKsRg3YxabDjaQCuEM3DeM9EuZ/ZJJJqQFbzN0=; b=BUyd2MflTqd+KLsAq1OQifyeS6FzrPtQaoHp8816G8CVLDEVQr0ykBt2NePhArySI/ 8yi/iOVXkLH+/IAiRhjjSeTDEisspvbVDPxbF6D8XdL9gk9q2LJOrmcC9nMVH0+nsTYm EyrlXu/YDRuNYaj10MZyYBXvkXyyjY0/isl5oQ8ss+gm2Zuth1FZ3PpmVzegY1W+hH/q wNTLFL5ma/94M1DiiWvU1tW8QAFoW/cAR++CLD7m95TgKFlyq+o46ppKm5cMmLJfHRIy GQWTbxytjs2qfsbcQ/YopSxDuEhe13JdWp2Eir3J/y/CRCOY6x9NQX4vZokFu+J3PSIz m46Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133205; x=1787738005; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=qLDW1MKsRg3YxabDjaQCuEM3DeM9EuZ/ZJJJqQFbzN0=; b=LVdQMYOmYhe/x8ODtiB1p9CuGcOnDrGUgotxD9T+J9dTz6RzdBY6EpPui/eCvp6h26 huWcuMOnRZcBw8FjXU8AdzgNKOZJGFwYJtan4n46nBdRHAhpKkf7bHgU8u9KUZ8z0ThA h9ubF8I+LOzlibX0C6iKBKaQeQKqMbujuAL4gaVw1oELW7qVJhGBLn6LU5OrKgnshq8e BBOyjzqMx4Eb3pEIfHAFQrksvRymKNbg6KwDe9lcJachHbz2w7mgLfrnxFC4kEpkzbcv WL8K5BKPCXZUq9XANEDc4tjVZ6vaJq4EntRh1/fAhnyiHwmHGQNf5SLGWgjaxv3t7wYa zDQg== X-Forwarded-Encrypted: i=1; AHgh+RotRiMjaiG7O36V3N3rATiNvfSlfchiR9h+LD/j0Qn5G0uyDbZEW2xQqyf1u584YC7sbwaNgFbY8yN3X9A=@vger.kernel.org X-Gm-Message-State: AFuF++ke5ndlMepcsIEE7ICHA/cH76Eb+IYIYA8zJIsTGdIDYhF60L8K BRenHLNGdivSaBQofw7Edbhlf/ZN7ZDf6Xb0u7ry2N4EoiR0Td1T60slWVwP42f8ia8= X-Gm-Gg: AR+sD10sxda5avMTIcgfJcTfgeLF25qXTvtDgScmK5PMEG37NXbG4D6KZtBaxMNX2Wm ENErR3otDt2fx3k1CS8co+Be4bXHMD35LdeZEJ3P8pgiy++YLHPAUQeJc48ne10Pgn+d4VmLyFQ 5383C/+HnqbQoJKWxLPV4L38zQ4wXKsWekG5owadjlVFmflJoUvxa0KKtEJw5MATJl8D+sZtDkB 29/70LsJhnBUksubGg6Zjpc4U37/+pJilUDtv+EbZUJkwUVkxkM73ijABpsaO571n5jsQU/cmp0 wU2nLDvJLsSVMlaARA1kdNs7GqQltYRLzQbthXEoDntyu/Z3+E5G1G4Gftae/YlOw2J+WjEfZIg +Nr18fouN067dF+DE+lMBOkcRXD6Nf1shBfYWk356YEjLoYSsK9V/MIqoTecm5MhZHMrSYF7i7X hP7wmzfv27AC5m9HGrbQt78Imk/B6mOUMM4Tfcm+5iy547KOh5sIeYghgeb1Gujd53roJ6/owCi 2Y2UgPtQVcOjhJpPy3dhcceqA== X-Received: by 2002:a17:903:15c8:b0:2cf:7db9:e13e with SMTP id d9443c01a7336-2d5fc8c65a8mr41853765ad.3.1787133205195; Wed, 19 Aug 2026 02:53:25 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:24 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 07/17] mm/sparse-vmemmap: move vmemmap_get_tail() before PTE population Date: Wed, 19 Aug 2026 17:51:29 +0800 Message-ID: <20260819095140.17252-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A follow-up change will call vmemmap_get_tail() from vmemmap_pte_populate(). Move it before the PTE population helpers to avoid adding a forward declaration. Move vmemmap_alloc_block_zero() with it because vmemmap_get_tail() depends on that helper. No functional change is intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Add this new patch to move vmemmap_get_tail() before PTE population (suggested by Mike Rapoport) --- mm/sparse-vmemmap.c | 78 +++++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 38 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b7abc5494bb9..b770fe2428fd 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -148,6 +148,46 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 +static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) +{ + void *p =3D vmemmap_alloc_block(size, node); + + if (!p) + return NULL; + memset(p, 0, size); + + return p; +} + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) +{ + struct page *p, *tail; + unsigned int idx; + int node =3D zone_to_nid(zone); + + if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) + return NULL; + if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + return NULL; + + idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; + tail =3D zone->vmemmap_tails[idx]; + if (tail) + return tail; + p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); + if (!p) + return NULL; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail(p + i, NULL, order, zone); + + tail =3D virt_to_page(p); + zone->vmemmap_tails[idx] =3D tail; + + return tail; +} +#endif + static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long ad= dr, int node, struct vmem_altmap *altmap, unsigned long ptpfn, unsigned long flags) @@ -181,17 +221,6 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *p= md, unsigned long addr, in return pte; } =20 -static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int n= ode) -{ - void *p =3D vmemmap_alloc_block(size, node); - - if (!p) - return NULL; - memset(p, 0, size); - - return p; -} - static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long ad= dr, int node) { pmd_t *pmd =3D pmd_offset(pud, addr); @@ -323,33 +352,6 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigne= d long end, } =20 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) -{ - struct page *p, *tail; - unsigned int idx; - int node =3D zone_to_nid(zone); - - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) - return NULL; - - idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail =3D zone->vmemmap_tails[idx]; - if (tail) - return tail; - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) - return NULL; - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - tail =3D virt_to_page(p); - zone->vmemmap_tails[idx] =3D tail; - - return tail; -} - int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, unsigned int order, struct zone *zone, unsigned long headsize) --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3788E449B01 for ; Wed, 19 Aug 2026 09:53:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133221; cv=none; b=HIcXkWWo98xGog48N/WyGfQ63Tg0mydiTbdqPrW9UcH60kVVlChjwBk864FOm3soStajqEHHxMvH7hYccdzeTp0f3tWj8OT7N07iV3v+uLyGvkMzsGLjYMTob2hIfPEQipCZtOck49CMKrYhtGVZHplk9zILa6HbIyrQ77SvMj0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133221; c=relaxed/simple; bh=Z7TQ1SBJopSg2m/uDqzifmTewvEx3Vp99MkMw8QQORE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vnx57UJcuufpVJvtww1zDQuJ5DhgAfcQ8VN8TyJbGFfvN4x+tkkycVCsQMTaBp5yZL8BLtuCXtpveCmYdpTdj3FUPLGlaaEKYJUXAWW0buqF+wn399sNz4u+fXABhC8tu7Y+efiRQMXIbkDPXWrw19sW8WXeGYq2M6EAf3AYTgM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=cYa4WKUo; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="cYa4WKUo" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2d5655cc850so9450375ad.3 for ; Wed, 19 Aug 2026 02:53:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133210; x=1787738010; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=671AwO9cxknmM2yc6/2dzWMjTOAP7N0nlaeCKAVGNAc=; b=cYa4WKUorAufLaPimnzekSMXp7vMGVNiEQvAiw087kp1UIlXwimwJLpdPnVYbk5jBL O97wFMM+ctyjhtQg8Ib+cBMmC+6zNnv/oBP2xAOyFTYGwG+WrdFG9DCKu7/QxWlVI3wt exqV8NoLTs5Brpady2ICfELKvhSNlm8S9ktk3DBtGg/Nu9h7AzfXa3Qw4fKahguyb//c QGckeo6Pz4NxYCgwTo0x5RxCqak96PLikEbgJa4qeULT8rWslPVHX1ZGjQRAaNfuJOCB A2MJTmqKBdFIDrbCBpcyypDOHNECFY/bR7vDZtme0/0r7UTevFa9WO394tWfylhzom4J BqAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133210; x=1787738010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=671AwO9cxknmM2yc6/2dzWMjTOAP7N0nlaeCKAVGNAc=; b=nwe7cTU8cB1IzF50TuZqrbfhwBy1kGh2STG35/izOPo8tAEemsANA/65bL9FjhuNAq qYYsEwa1/3VmzoPByQrmaefddcda2prtjZOrKuo7IJ35X0d/mVvPWUlQMig7pA4HTUkE BrWWIf2JUCufO69aUYO1OEMDqph/F2m0xRlDOaUsdKNima+gxwEsjzbZRrY8Ymm45+6p WLx7tg6ZXoE7O+fmLtGfNy4ayKMM9bJw5HoCMMAaOMiDtgWhVHN+/bt2im+7t/H2OD97 Jv2mQw6rDL23iros3IN5vIdnDfdUJtbpG/FWea5ggbxu9olArJm2QFu9eALdHTR8OQDI IYEQ== X-Forwarded-Encrypted: i=1; AHgh+RqRrx1dD5r2lVqeYWmAnjnPgFKRgmTbuvJgG5CgVGJkYAryup77+WfpUVmuWSEwPH7jlBeFg7azm2DdHqQ=@vger.kernel.org X-Gm-Message-State: AFuF++lNAC4AvW92KZnfzZpU9nRinf4watUxhDqI0bJDo3AUC0iRC9Ty GnGLAZy3YzXMrnzjM9Pmrl5Y+dYgG7GI7N0HfkcBeh3uuiVquOuQP8amvAtU6j3v9us= X-Gm-Gg: AR+sD12E/12XTqA76UlaPuADXmk+qkwRdkx4ZfTnMzYU2F2vr3shZWeqK5LzLCOOz/Q gq7Cy21fu7Z/OptHojef/PnjT5HMuzlZU8F9F7RLH0PPRfmJ2QUk5LCvMUo2e1sPIBQqQ+lFUSX 21HKp1UUjfoDOiT0KR7/n6KCA/LON+TTKCD6m/gawabzeh3huNHTxaAKV0zcO57QvkO+hcl1HyR lKRxvVi9FJ40lQcFT/+GWHFST0YB+kW3nzPATiZyfEpa5ugprYTKAdrkUZNS157EFqPPIPXzAH/ 3czcq//9kQiPkxRdHMaJbxrV3ZF4ITt0a8PUAP7F0Fb4wcooWUYga+gLX7b92trdAW8CZsgMJ1Z n+/lwOsya618p4g245eAIVwL/BOrxaE6+n29McZYxiwqxoPD5mPA/AA20AEzt4QUqU5BaDuGhKs P5UCEygZFCiUJlLyA5EHK+RR/KBPSwZHDgN7Rae0J+679x3nw6baGvFwHprzj2p6gOW0YQ1P2l4 ImwIiSQEp1jthOLDVVK1Rjqug== X-Received: by 2002:a17:902:cf08:b0:2d0:cc92:f7a3 with SMTP id d9443c01a7336-2d5fd5ec972mr66156725ad.2.1787133209699; Wed, 19 Aug 2026 02:53:29 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:28 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 08/17] mm/sparse-vmemmap: support section-based vmemmap optimization Date: Wed, 19 Aug 2026 17:51:30 +0800 Message-ID: <20260819095140.17252-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach sparse-vmemmap population code to use the compound page order when deciding whether a vmemmap page can be optimized. With this information, the common sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages directly instead of relying on HugeTLB-specific handling. This centralizes vmemmap optimization logic in the sparse-vmemmap code, based on section metadata, and prepares for sharing the same mechanism across different users of vmemmap optimization, including HugeTLB and DAX. Signed-off-by: Muchun Song --- v4: - Move section_nr_vmemmap_pages() outside CONFIG_MEMORY_HOTPLUG to fix CONFIG_MEMORY_HOTPLUG=3Dn builds (reported by kernel test robot) v2: - Keep vmemmap accounting and population logic in sparse-vmemmap.c (suggested by Mike Rapoport) - Move vmemmap_get_tail() before its first use instead of adding only a forward declaration in the previous patch (suggested by Mike Rapoport) - Simplify the PMD path handling for HVO-covered sections --- mm/sparse-vmemmap.c | 88 ++++++++++++++++++++++++++++----------------- mm/sparse.c | 4 +-- mm/sparse.h | 7 ++++ 3 files changed, 65 insertions(+), 34 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b770fe2428fd..737d50bdd3ef 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -186,6 +186,11 @@ static __meminit struct page *vmemmap_get_tail(unsigne= d int order, struct zone * =20 return tail; } +#else +static inline struct page *vmemmap_get_tail(unsigned int order, struct zon= e *zone) +{ + return NULL; +} #endif =20 static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long ad= dr, int node, @@ -193,12 +198,24 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *= pmd, unsigned long addr, in unsigned long ptpfn, unsigned long flags) { pte_t *pte =3D pte_offset_kernel(pmd, addr); + unsigned long pfn =3D page_to_pfn((struct page *)addr); + if (pte_none(ptep_get(pte))) { pte_t entry; - void *p; + + if (vmemmap_optimizable_pfn(pfn) && ptpfn =3D=3D (unsigned long)-1) { + unsigned int order =3D pfn_to_section_order(pfn); + struct zone *zone =3D pfn_to_zone(pfn, node); + struct page *page =3D vmemmap_get_tail(order, zone); + + if (!page) + return NULL; + ptpfn =3D page_to_pfn(page); + } =20 if (ptpfn =3D=3D (unsigned long)-1) { - p =3D vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + void *p =3D vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) return NULL; ptpfn =3D PHYS_PFN(__pa(p)); @@ -217,7 +234,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in } entry =3D pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); - } + } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) + return NULL; return pte; } =20 @@ -406,6 +424,9 @@ int __meminit vmemmap_populate_hugepages(unsigned long = start, unsigned long end, pmd_t *pmd; =20 for (addr =3D start; addr < end; addr =3D next) { + unsigned long pfn =3D page_to_pfn((struct page *)addr); + const struct mem_section *ms =3D __pfn_to_section(pfn); + next =3D pmd_addr_end(addr, end); =20 pgd =3D vmemmap_pgd_populate(addr, node); @@ -421,7 +442,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long = start, unsigned long end, return -ENOMEM; =20 pmd =3D pmd_offset(pud, addr); - if (pmd_none(pmdp_get(pmd))) { + if (pmd_none(pmdp_get(pmd)) && !section_vmemmap_optimizable(ms)) { void *p; =20 p =3D vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); @@ -439,8 +460,11 @@ int __meminit vmemmap_populate_hugepages(unsigned long= start, unsigned long end, */ return -ENOMEM; } - } else if (vmemmap_check_pmd(pmd, node, addr, next)) + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + if (WARN_ON_ONCE(section_vmemmap_optimizable(ms))) + return -EOPNOTSUPP; continue; + } if (vmemmap_populate_basepages(addr, next, node, altmap)) return -ENOMEM; } @@ -620,6 +644,33 @@ void __init sparse_init_subsection_map(void) sparse_init_subsection_map_range(start, end - start); } =20 +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + const struct mem_section *ms =3D __pfn_to_section(pfn); + const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); + const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; + const unsigned long pages_per_compound =3D 1UL << order; + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); + VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); + + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); + + if (order < PFN_SECTION_SHIFT) { + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); + return vmemmap_pages * nr_pages / pages_per_compound; + } + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); + + if (IS_ALIGNED(pfn, pages_per_compound)) + return vmemmap_pages; + + return 0; +} + #ifdef CONFIG_MEMORY_HOTPLUG =20 /* Mark all memory sections within the pfn range as online */ @@ -648,33 +699,6 @@ void offline_mem_sections(unsigned long start_pfn, uns= igned long end_pfn) } } =20 -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned = long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) -{ - const struct mem_section *ms =3D __pfn_to_section(pfn); - const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); - const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; - const unsigned long pages_per_compound =3D 1UL << order; - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); - VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) - return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); - - if (order < PFN_SECTION_SHIFT) { - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; - } - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); - - if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; - - return 0; -} - static struct page * __meminit populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) diff --git a/mm/sparse.c b/mm/sparse.c index c84b4c7b8c70..e6cb67ca9c8d 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -305,8 +305,8 @@ static void __init sparse_init_nid(int nid, unsigned lo= ng pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct pa= ge), - PAGE_SIZE)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); sparse_init_early_section(nid, map, pnum, 0); } } diff --git a/mm/sparse.h b/mm/sparse.h index 02ed0f34eac8..1569774270fb 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -111,8 +111,15 @@ static inline void sparse_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else static inline void sparse_init_subsection_map(void) {} +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); +} #endif /* CONFIG_SPARSEMEM_VMEMMAP */ =20 #endif /* __MM_SPARSE_H */ --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0B4E449980 for ; Wed, 19 Aug 2026 09:53:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133233; cv=none; b=Gowrb5IUP5wQwSSUK8u4AHSmjuXWW9IT0ybhqTdTtBlZTQGj+oFDllNRK9Y11w+tjAQndFxqQeT0RSACMUeznuAogmFJL3Sm80WdcxBW5js0Pxd5Xge5qxtIcv9YpMFm5FCTP8eoRhGEllchOOmvviLIvo80bvNbWGAbbCWP7ag= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133233; c=relaxed/simple; bh=MQxnkwtTutrKqM/1GLki1uygjx8sOpOKNNSpXx0PJiY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qcn9ntv7CI5bGGbbLvnHZzCNOWw+npJSHxedZnfl7Rk3gSoAQ5xVoFSa0cuowQXNn1cILkJpFvbDyulYbRtT9gxehZjOizCKTQ/Ufx+h2uzu4BKkPxX5uc9Wt++30er0i4+zx6UXHG+IUclLLQAWVjMD/4O5XHglkpSIK/8bWIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RXxZGbW5; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RXxZGbW5" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2cf50c6f235so8999255ad.0 for ; Wed, 19 Aug 2026 02:53:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133215; x=1787738015; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=K7eU7XEPiihNsb4nEfOM3x1W8CVVTfFcc0o6L0WYHdw=; b=RXxZGbW57lxA2H6whsF9hvdJxasVgGo3IagnnQWhGNa7zZ/tho6EGnShf17EDHrtCt SYd4ji8PGa9CAI1tpq5zk9PhXY2uvNWRCZV2xZzD1prVJUaHTYF5zdLV+GRpEwKbDXws 7cdygmyG5Dcxr6UzrF4z9ND66G6AfCuv/Z/elxJ3NK6ekGGztSLDrF3hJVuu1Ox034zi sd2ofNomgUS1RA9hNqMc0umy88yIHXP4g6L0h6JC9iIwDY8w1idUwVI4ufW2z6d8hi+z PWSIql7zFseHgUJ9bUsnn4bBfCi8vdRnqMevq2yRNTDk4oY1rwpR+235b1m4fXlB6nz0 GFxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133215; x=1787738015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=K7eU7XEPiihNsb4nEfOM3x1W8CVVTfFcc0o6L0WYHdw=; b=X5n/MpCkju5PSQOlAk5g3SGU/69CLCqpOaSIKxGz1f4isuJwylfIvXhpG4ytxYqXsS R1NP2S38WmV2NNNIBHgjbHEmomK/pWzrwXPlzaFVlGxePE9SWouoWQ7Fx27rX9vrmqz+ iCiRde6ULWmaWXFR8oMUNao8MLs3A2bQ6r5NpcAZ+Chd/W9cOVZpO0v6JglNh1oI2Rwb nUdHhn+R3cNFVmTtJ5EvATHTbsJoRvTxldrq24BH+cIyB4taNoqmVOIvpV/JHX4vLFih 1jmj7jiKwt/i2rRE5SuCcgJEEmXBnXZVSq4Fa5PUz2jjozMzvI3wK3PwOuyYiPg8NaHp rxyA== X-Forwarded-Encrypted: i=1; AHgh+Rq76ehd6OJSvo2w1QkOiwxBEqrfnSLPqGFFI9tcoI6X8BTBiOrUOWkbGrqrZyIAbrtD7fcG1GaO5Au9ShM=@vger.kernel.org X-Gm-Message-State: AFuF++lYJgM9/Iv0CG3960JIJYxfSYHOmwl8Lz2dXStUA+6QQOcnICmM BXVkgG0qfmgb5R6YV14V2Svtj9YzM3gZwp77Qtv/eNdgra16fP2MbnhsO3etqAl8XJE= X-Gm-Gg: AR+sD10sTMGptWz4MtTN+JFOOKCwY+xBEtH4rFJiBRwDOn/JqIqEOEfjcUNMjoCCiLb 9wTPqRpD8VPl2XN51bGQYSHVFUBnN1tL7Z8eeQ6ztEdC+lmUQ3hcVye9AnPKpi1oKdBj7yIqebK Wls7lfV1QCtM6Heriql1RauUQILOkUw6RSndCSZd1lAM1NOOLFdZ/WAVdAXarPeU0UJ8nhPau82 rggMdxH9PEtWoGEFSGf/Eoq4rsaubYWiWG3kdkUl+ehC6C1DQXX1opyY5FHJ3eDF7MGJBMKjQ9c fYtvvbkxzleC2GCJzmYX98EVQPr/iFuXnFx4/8y4C6EeZ2SL6tTaQMmdTLyCX5aENBx9OvyvN+M pMch5fc5ecDBSyqfkRsg6wBx1fH3ii4mLv5USb6JGWK8Yxhika38gsGznc69kKPnacNsfEAPeV2 Bu5gCcLV6Q2IWKMGBc4vDnzajyHPbreUEKZC22oI5jkHhgUMVSSL5qXSzUWY7cdamGNswi91FU/ 6YTq5lKJT/tCWroNHpdVDX9pgB9SVLfpV+J X-Received: by 2002:a17:902:d483:b0:2cc:f5aa:9513 with SMTP id d9443c01a7336-2d5fd6fb665mr56244815ad.10.1787133214623; Wed, 19 Aug 2026 02:53:34 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.30 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:34 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 09/17] mm/sparse: initialize memory sections earlier Date: Wed, 19 Aug 2026 17:51:31 +0800 Message-ID: <20260819095140.17252-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Upcoming HugeTLB bootmem changes need sparsemem section metadata before the HugeTLB bootmem allocation path runs. The memory sections are initialized from sparse_init(), which is called too late for that setup. Move the code that initializes sparsemem section metadata for memblock ranges into mm_core_init_early(), before free_area_init() and the HugeTLB bootmem setup. Rename the helper to sparse_sections_init() so the new caller describes the sparsemem-specific initialization step. This is a preparatory change. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) --- v2: - Rename the helper to sparse_sections_init() to describe the section metadata initialization (suggested by Mike Rapoport) - Fix the !SPARSEMEM stub name so SPARSEMEM=3Dn builds compile (reported by Sashiko) --- mm/mm_init.c | 1 + mm/sparse.c | 10 ++-------- mm/sparse.h | 2 ++ 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 3625de86d1d9..c37b15a3c69b 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2636,6 +2636,7 @@ void __init mm_core_init_early(void) { kho_memory_init_early(); =20 + sparse_sections_init(); free_area_init(); =20 hugetlb_cma_reserve(); diff --git a/mm/sparse.c b/mm/sparse.c index e6cb67ca9c8d..439802e6a6ad 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -191,12 +191,8 @@ static void __init memory_present(int nid, unsigned lo= ng start, unsigned long en } } =20 -/* - * Mark all memblocks as present using memory_present(). - * This is a convenience function that is useful to mark all of the systems - * memory as present during initialization. - */ -static void __init memblocks_present(void) +/* Initialize memory section metadata for all system memory. */ +void __init sparse_sections_init(void) { unsigned long start, end; int i, nid; @@ -322,8 +318,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count =3D 1; int nid_begin; =20 - memblocks_present(); - if (compound_info_has_mask()) { VM_WARN_ON_ONCE(!IS_ALIGNED((unsigned long) pfn_to_page(0), MAX_FOLIO_VMEMMAP_ALIGN)); diff --git a/mm/sparse.h b/mm/sparse.h index 1569774270fb..89d7ef91c041 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -59,6 +59,7 @@ static inline bool vmemmap_optimizable_order(unsigned int= order) */ #ifdef CONFIG_SPARSEMEM void sparse_init(void); +void sparse_sections_init(void); int sparse_index_init(unsigned long section_nr, int nid); =20 static inline void sparse_init_one_section(struct mem_section *ms, @@ -104,6 +105,7 @@ static inline bool section_vmemmap_optimizable(const st= ruct mem_section *ms) } #else static inline void sparse_init(void) {} +static inline void sparse_sections_init(void) {} #endif /* CONFIG_SPARSEMEM */ =20 /* --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 342BE44C519 for ; Wed, 19 Aug 2026 09:53:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133228; cv=none; b=o4sCHA7/1PsRZiQXa2I/TJ/dZrRgq88Oynv/1rV+5EjCwwPhi8qlcH3NVPyHyLJXzBWfvVVgVx2iijnIog8OODDnAAUwa9UBLeQbMjVfLjkklAAEaB0aeqDLyAyL5eDirDTEyegj4oUFBJ0K2gV6JvCpdlMTyiiOAeTeEc+y9is= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133228; c=relaxed/simple; bh=ocTzpOoeaKnkHzALRE5q+Qs8KijlUuUx4h175ynjPnk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mQJt7KRGNRjF+wuzJqhbAd4DCmtQmQAgs/ViAnT73ScWBPFuuRPd8EhxCw5LeLDPD95SUgL0YxvRfCuQPrjlLqbZ9G4F0M3w4zFAHHRh3neXncmcleUn6q/u9qWRGtBgVLIWB1+r+1+hhqWKya4GvW8RhzgEnihsUSNTEnBcJ54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=djuFscr8; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="djuFscr8" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-ca88130e09aso409225a12.3 for ; Wed, 19 Aug 2026 02:53:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133219; x=1787738019; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SWP1aSDIxPqpucFi2P6HiR4lR16ch7GDnI9FrMPGnfA=; b=djuFscr8fnE/hzWRkE9Yr5j3GvbhUpKsb7+m/6l26Jwwyinu2Ox8/4/dyhzV/45OTH 0bccoAxjXymD8JylikXLN9oRbAfd+fHwyqKFRgpFUxSrC9ZdSBjI2XTdmHgNbDneUtTd VVAvQwTioQwc4YNMMAidI25zS+EW1NqgEv1E3Eswu6HjBVT1xzkBRD81d563VY2qDSYI W+yIVUXHKceaWMr9/s2r+BPnGDptUL/SAm/rSl9ZfGHRXXHtang3IxvzwsSY8WBlm9kh c1gvHUbvebkWgoPab/4xegcvCXKSDaOYzaY1QDBPLsFV0qkh/utogKdShW67u26QgJvW wzCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133219; x=1787738019; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=SWP1aSDIxPqpucFi2P6HiR4lR16ch7GDnI9FrMPGnfA=; b=EuksN+qOPbQ0iQyxC5MFNILeB1AAqOLIDZKu6Zio35KU1lfDqq/wn6o16YCaHnnFVd NXiWP4M9YIFHIgxrrMQf8CkpgXaCyU2VZmBf30nW40en3fh5k57v3akz/o2UIE8NSU/2 l6IDWnTN/56fBprqVM1hmOrsvOcWvmPitB7V7KP2uxushtHpcxRluIAipqEECGZEVn/R LgHIGbDJRLf9rLX/7Q2AHqvat2IZj5lgCHA88nUNABef8SbJfSR671S0v+22f84zfQ3k r9ZResOa3WCNs1Bf1YV5z9TVy1VnRBoQzntz+CsNsA84d73X1ziMcRfla8p0aZe7Jg5T v9wQ== X-Forwarded-Encrypted: i=1; AHgh+RrFEy46lYKgm98/44q5KgdS3yQZc3QjjSMxmh+KdHurTcNbVyYujqhRk5nD0H1UsM1GLPHlOfHOtf9zkoQ=@vger.kernel.org X-Gm-Message-State: AFuF++mx4ETk0x2N88gmeFLexNhkFrSZmUcIfr1xNSRDPes24UZP4U4I X8ZG9loMn69NAPEirkx8+5EY9CG9t9tdi0qm2pF1rFlM3XRnOCARQXW2kjvLYCKhaQA= X-Gm-Gg: AR+sD13m6IKZhd+FbNG80PkZIchkJtX/kk/xUZYbujoP0zpSUzP2M0ZYERANo2Q6KJr FrJxZ20DiiaTPKYpIZTNwSuwYSAMaOOozfEX6iH0cYy8xkv6l1M71wih5Pc7Emp55R/RALmNj9l we47aNVNdrBExOh9Vq2MQKwsdoDqSvYQSNEAmhiHVxva4m8+CC/AkVK7uKiQKYPQzQyrKmjQRrr SZsoxrdsg/erVadcee0nm2EydpVQ0WkviQlal4h4sP5tLRR56dDNpdaCT430nuxEnw7Lmf1ZGxF Lo3LsEABccgApoDKQZ67fEqg3q71ja2pZKpQJm85sAJyKIiata+hp3DNYX9h0HpAbTAjK1ZPSE9 cJBMit/3o5Xe6/DpdxwbU/TdgZ7SfjraI5f1wYy+j7RaEHgbcir4QL0OSPdY1g6gRedTTqk5yKw Mys9HVRGeMaawg9DW7sEPtp+cYeE2mJn/iZRr0arSfCfEHSE3PC2qwq4++uQXKPUAsl+vCLyLMd M2LB0Z5D450BK4Jgw/vFmxE5g== X-Received: by 2002:a17:903:11c7:b0:2cc:864b:539 with SMTP id d9443c01a7336-2d60175df80mr48678865ad.6.1787133219241; Wed, 19 Aug 2026 02:53:39 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:38 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 10/17] mm/hugetlb: switch HugeTLB to section-based vmemmap optimization Date: Wed, 19 Aug 2026 17:51:32 +0800 Message-ID: <20260819095140.17252-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB bootmem vmemmap optimization still carries its own early setup path, including pre-populating optimized mappings before the generic sparse-vmemmap code runs. Now that the section-based vmemmap optimization can derive HugeTLB vmemmap deduplication from section metadata, HugeTLB only needs to mark the bootmem huge page range with the appropriate order. The generic sparse-vmemmap population path can then allocate and map the shared tail vmemmap pages without any HugeTLB-specific early population code. Do that by setting the section order when a bootmem huge page is allocated and dropping the dedicated pre-HVO helpers and related special-casing. This removes duplicate early setup logic and switches HugeTLB to the section-based vmemmap optimization path. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v3: - Use the order-based helper for the bootmem vmemmap-optimized check v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 1 - include/linux/mm.h | 3 -- mm/hugetlb.c | 30 ++------------ mm/hugetlb_vmemmap.c | 90 +++-------------------------------------- mm/hugetlb_vmemmap.h | 14 +++---- mm/sparse-vmemmap.c | 31 -------------- mm/sparse.h | 27 +++++++++++++ 7 files changed, 42 insertions(+), 154 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 16c4c4caa126..fe28f98e1b22 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -171,7 +171,6 @@ struct address_space *hugetlb_folio_mapping_lock_write(= struct folio *folio); =20 extern int movable_gigantic_pages __read_mostly; extern int sysctl_hugetlb_shm_group __read_mostly; -extern struct list_head huge_boot_pages[MAX_NUMNODES]; =20 void hugetlb_bootmem_struct_page_init(void); void hugetlb_bootmem_alloc(void); diff --git a/include/linux/mm.h b/include/linux/mm.h index 0829e0d3b2d1..2cad82cc466d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5159,9 +5159,6 @@ int vmemmap_populate_hugepages(unsigned long start, u= nsigned long end, int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); -int vmemmap_populate_hvo(unsigned long start, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize); void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int nod= e, unsigned long headsize); void vmemmap_populate_print_last(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a018d0da4ca7..fc579ba9ab39 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -52,6 +52,7 @@ #include "hugetlb_cma.h" #include "hugetlb_internal.h" #include "mm_init.h" +#include "sparse.h" #include =20 int hugetlb_max_hstate __read_mostly; @@ -59,7 +60,7 @@ unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; =20 __initdata nodemask_t hugetlb_bootmem_nodes; -__initdata struct list_head huge_boot_pages[MAX_NUMNODES]; +static struct list_head huge_boot_pages[MAX_NUMNODES] __initdata; =20 /* * Due to ordering constraints across the init code for various @@ -3137,6 +3138,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) } else { list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |=3D HUGE_BOOTMEM_ZONES_VALID; + hugetlb_vmemmap_optimize_bootmem_page(m); /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB @@ -3297,6 +3299,7 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) * this folio. */ folio_set_hugetlb_vmemmap_optimized(folio); + section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); =20 if (hugetlb_bootmem_page_earlycma(m)) folio_set_hugetlb_cma(folio); @@ -3340,31 +3343,6 @@ void __init hugetlb_bootmem_struct_page_init(void) .max_threads =3D num_node_state(N_MEMORY), .numa_aware =3D true, }; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct zone *zone; - - for_each_zone(zone) { - for (int i =3D 0; i < VMEMMAP_OPTIMIZATION_NR_ORDERS; i++) { - struct page *tail, *p; - unsigned int order; - - tail =3D zone->vmemmap_tails[i]; - if (!tail) - continue; - - order =3D i + VMEMMAP_OPTIMIZATION_MIN_ORDER; - p =3D page_to_virt(tail); - /* - * prep_and_add_bootmem_folios() can access pageblock - * flags on bootmem HugeTLB pages, so initialize the - * shared tail struct pages here before bootmem folios - * start using them. - */ - for (int j =3D 0; j < PAGE_SIZE / sizeof(struct page); j++) - init_compound_tail(p + j, NULL, order, zone); - } - } -#endif =20 padata_do_multithreaded(&job); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c48fcea076a5..7293706b532f 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -18,8 +18,7 @@ =20 #include #include "hugetlb_vmemmap.h" -#include "internal.h" -#include "mm_init.h" +#include "sparse.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -706,95 +705,18 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct h= state *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - -/* Return true of a bootmem allocated HugeTLB page should be pre-HVO-ed */ -static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page = *m) +void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page= *m) { - unsigned long section_size, psize, pmd_vmemmap_size; - phys_addr_t paddr; - - if (!READ_ONCE(vmemmap_optimize_enabled)) - return false; - - if (!hugetlb_vmemmap_optimizable(m->hstate)) - return false; - - psize =3D huge_page_size(m->hstate); - paddr =3D virt_to_phys(m); - - /* - * Pre-HVO only works if the bootmem huge page - * is aligned to the section size. - */ - section_size =3D (1UL << PA_SECTION_SHIFT); - if (!IS_ALIGNED(paddr, section_size) || - !IS_ALIGNED(psize, section_size)) - return false; - - /* - * The pre-HVO code does not deal with splitting PMDS, - * so the bootmem page must be aligned to the number - * of base pages that can be mapped with one vmemmap PMD. - */ - pmd_vmemmap_size =3D (PMD_SIZE / (sizeof(struct page))) << PAGE_SHIFT; - if (!IS_ALIGNED(paddr, pmd_vmemmap_size) || - !IS_ALIGNED(psize, pmd_vmemmap_size)) - return false; - - return true; -} - -/* - * Initialize memmap section for a gigantic page, HVO-style. - */ -void __init hugetlb_vmemmap_init_early(int nid) -{ - unsigned long psize, paddr, section_size; - unsigned long ns, i, pnum, pfn, nr_pages; - unsigned long start, end; - struct huge_bootmem_page *m =3D NULL; - void *map; + struct hstate *h =3D m->hstate; + unsigned long pfn =3D PHYS_PFN(__pa(m)); =20 if (!READ_ONCE(vmemmap_optimize_enabled)) return; =20 - section_size =3D (1UL << PA_SECTION_SHIFT); - - list_for_each_entry(m, &huge_boot_pages[nid], list) { - struct zone *zone; - - if (!vmemmap_should_optimize_bootmem_page(m)) - continue; - - nr_pages =3D pages_per_huge_page(m->hstate); - psize =3D nr_pages << PAGE_SHIFT; - paddr =3D virt_to_phys(m); - pfn =3D PHYS_PFN(paddr); - map =3D pfn_to_page(pfn); - start =3D (unsigned long)map; - end =3D start + hugetlb_vmemmap_size(m->hstate); - zone =3D pfn_to_zone(pfn, nid); - - if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), - zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) - panic("Failed to allocate memmap for HugeTLB page\n"); - memmap_boot_pages_add(DIV_ROUND_UP(HUGETLB_VMEMMAP_RESERVE_SIZE, PAGE_SI= ZE)); - - pnum =3D pfn_to_section_nr(pfn); - ns =3D psize / section_size; - - for (i =3D 0; i < ns; i++) { - sparse_init_early_section(nid, map, pnum, - SECTION_IS_VMEMMAP_PREINIT); - map +=3D section_map_size(); - pnum++; - } - + section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); + if (vmemmap_optimizable_order(pfn_to_section_order(pfn))) m->flags |=3D HUGE_BOOTMEM_HVO; - } } -#endif =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 7ac49c52457d..20eb03df542a 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,8 +9,7 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include -#include -#include +#include "internal.h" =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -27,10 +26,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate = *h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *= folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list= _head *folio_list); -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -void hugetlb_vmemmap_init_early(int nid); -#endif - +void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); =20 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -76,13 +72,13 @@ static inline void hugetlb_vmemmap_optimize_bootmem_fol= ios(struct hstate *h, { } =20 -static inline void hugetlb_vmemmap_init_early(int nid) +static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct h= state *h) { + return 0; } =20 -static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct h= state *h) +static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootm= em_page *m) { - return 0; } #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 737d50bdd3ef..97888f146571 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -32,8 +32,6 @@ #include #include =20 -#include "hugetlb_vmemmap.h" - /* * Flags for vmemmap_populate_range and friends. */ @@ -369,34 +367,6 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigne= d long end, } } =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize) -{ - unsigned long maddr; - struct page *tail; - pte_t *pte; - int node =3D zone_to_nid(zone); - - tail =3D vmemmap_get_tail(order, zone); - if (!tail) - return -ENOMEM; - - for (maddr =3D addr; maddr < addr + headsize; maddr +=3D PAGE_SIZE) { - pte =3D vmemmap_populate_address(maddr, node, NULL, -1, 0); - if (!pte) - return -ENOMEM; - } - - /* - * Reuse the last page struct page mapped above for the rest. - */ - return vmemmap_populate_range(maddr, end, node, NULL, - page_to_pfn(tail), 0); -} -#endif - void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, unsigned long addr, unsigned long next) { @@ -599,7 +569,6 @@ struct page * __meminit __populate_section_memmap(unsig= ned long pfn, */ void __init sparse_vmemmap_init_nid_early(int nid) { - hugetlb_vmemmap_init_early(nid); } #endif =20 diff --git a/mm/sparse.h b/mm/sparse.h index 89d7ef91c041..bd1538a0a61b 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -16,6 +16,24 @@ static inline unsigned int section_order(const struct me= m_section *section) return section->order; } =20 +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ + VM_WARN_ON(section_order(section) && order && section_order(section) !=3D= order); + section->order =3D order; +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ + unsigned long section_nr =3D pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_order(__nr_to_section(section_nr + i), order); +} + static inline unsigned int pfn_to_section_order(unsigned long pfn) { return section_order(__pfn_to_section(pfn)); @@ -26,6 +44,15 @@ static inline unsigned int section_order(const struct me= m_section *section) return 0; } =20 +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ +} + static inline unsigned int pfn_to_section_order(unsigned long pfn) { return 0; --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F6C72E06E6 for ; Wed, 19 Aug 2026 09:53:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133240; cv=none; b=r27WNzkXJA/tc+pxga9AqE26DHo9VpKwLAn05YWfKZVixXEUCfzwjVNEqLLgZqOZS3fFGUzKsr8mJ5yx8sdXDCxdGZFEwAlkhv9h7fJ/fDU6XfMvmqmS5h7UAMWEDYd/+KWN6P9nNFjiVMt3m6Q+OXgEu1DGz6b2FaXYn3JD0Lg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133240; c=relaxed/simple; bh=7AS74LsrnP1g9XUrGvmw9A3o1D235x3S9CpPnY07/wA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fh1u/NfJJlGa6xS6xBF3GrbRMgm3SMwqjSzbXa7GjeeW/FeTTUUDOJUFa+fZncJSlF2PXA5coq3ph5LspnYQfaNEKZ8PhxxLoRo+s4c5DuZu0CYwYZwlzouMbumVCH8BQpnw3NZ2UM/dmJDmc8IhK0iLWexjQqifvL9usLaUiQY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=FBkA7RfC; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="FBkA7RfC" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2ceb096e675so9346575ad.0 for ; Wed, 19 Aug 2026 02:53:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133224; x=1787738024; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8/kRQ1gy7AyeNmujGKY3KUegeF2WIa8rEbuBkgyiams=; b=FBkA7RfCYoNEoyukC9KtM5cKWcRozbXnarkjTSI9Xj2K9R52/h4nzwpGda+adQj4Qw w+Z3+/YXjwZLBCEQRpRUf5jKVkiSLu1MPrILUzU0UoTtmQ/D0+gx9j5YgYpKqd2BBw5b TySouGgfltZ0l0GPjg8+vmLSqCIXeeoGwV9j4oElfdUYheOeM16QRmdt2gjPoxJw7TTC IvSM5+RCOGnse1caN/H+rXE8RSFlhooN/OXfTV06UDUamDKmy6x7+sNmaJIHsgoZDkRt DAPoIatHMx2nmUTegnV7RkNh1117cGhIrunad5LZ/syEGmLt+NQP6NdHai6yvvGbAWAr 27+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133224; x=1787738024; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=8/kRQ1gy7AyeNmujGKY3KUegeF2WIa8rEbuBkgyiams=; b=RXvfdQHggvp1TWtAF1AHsg8kcyOIIXDuEWXOQi10Y5P2qhYh6vPsJdIkzChk5MgrmC 4eiUbJF/ZdoyQMMiAIQvcfSK/9e3qSJy92w7wCeVeWHEYzXIgXH/AVEg6MDJfKJu0pQp Fai2SKJtNoFz57Z4SYLsPTku/V7+LH/noB7Oc58UMBuXIx8mrpAYDFV9lhSf2I4j/MSm nBQ/R3nAZ1hHn7BTV67Mf0RRlWAjUxn4yuoe/hCKCxFJ5hJedthH67x/sd+MVehrhLF6 vpNsYkQXbI0IgazKaimVr6fhuyPstSHc7wMF54465Fii94sKbYcRWj5SGIaWvu3JGFMO 4mPg== X-Forwarded-Encrypted: i=1; AHgh+RqzKm1mWB58EaKG6NBp7oJQfxHoChJWgbpT0VVrUUcn6TSLMg93zXIePmS2HjSGESakFdiSMmT0DFE9wFc=@vger.kernel.org X-Gm-Message-State: AOJu0YwN8UBbcW0EtSCES9Uvxk1n4R3cgwkg9GzTtPpQmaZaI4Es0GCa G6EvP6uKstMpiXmV5XdKZZ+oKdN/G7DkqXaOAA8sBoMXwxhXrVsEEkNeZSyJemh4Lws= X-Gm-Gg: AR+sD13UNlhRPYU69gDRFAgZgKZkIrL9S/kYHFt7ttXnb/lTVnSQ1lSchbMj1TXVY2Q ISjTqv9t6VQH+Qm9voKgLH/VVeVKKalb8tcVTp4GFokv3nSClmKFp/o0qZSDuiiM7icXTtVy0Bn nL2SmldwgxuSHcoE+nIvGYd2bezmnWvFcqllgX44M16+7XOVJ93EvfrwBWisuCFCH95E4tY/cV2 aMyfSSHeGm0jBx+dJcfv9bnBCdfJ4Lptog/X1/XQnbOEAJcASqVW6J2SvdruoixZBKDunPQfY4y JYNcsuW4SZwk5Ut8nAANquLwt70Wq4zAd+7CyrL0vELzi/m3x77QljHaHhC/6v/zhx8gwTce6RK SyD43z4hQKfvb1bQl8r85s20PvcOz84Jktm9RDwOpBGLYC72/vQjc8iMPSoY8gRdgwkQwrW5cbF 4cWx7Xh4eIbQisdNSL/WW8fDeJ6or4CsaBiMatqDjRAPghxlqiB2WWpwv654smNRKwrB2qULZw8 VTi2E8lli1Lr0tzKMrfrAHkqQ== X-Received: by 2002:a17:903:17c5:b0:2d5:e3ce:3988 with SMTP id d9443c01a7336-2d5fd74019dmr65164905ad.11.1787133223772; Wed, 19 Aug 2026 02:53:43 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:43 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 11/17] mm/sparse-vmemmap: remove SPARSEMEM_VMEMMAP_PREINIT support Date: Wed, 19 Aug 2026 17:51:33 +0800 Message-ID: <20260819095140.17252-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" SPARSEMEM_VMEMMAP_PREINIT existed only to support HugeTLB's early vmemmap optimization setup. Now that HugeTLB bootmem vmemmap optimization uses the common section-based sparse-vmemmap path, the sparse initialization code no longer needs a separate pre-initialization mechanism for vmemmap population. Remove the related Kconfig symbols, section flag, and empty early hook, so present sections always go through the normal sparse setup path. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Collect Acked-by from Mike Rapoport --- arch/x86/Kconfig | 1 - fs/Kconfig | 1 - include/linux/mmzone.h | 25 ------------------------- mm/Kconfig | 5 ----- mm/sparse-vmemmap.c | 13 ------------- mm/sparse.c | 23 ++++++++--------------- 6 files changed, 8 insertions(+), 60 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7ac6c3173c20..db690f5e7dfe 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -149,7 +149,6 @@ config X86 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 - select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select ARCH_WANT_IRQS_OFF_ACTIVATE_MM diff --git a/fs/Kconfig b/fs/Kconfig index e05917adcd60..d1c210c6508f 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,7 +278,6 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP - select SPARSEMEM_VMEMMAP_PREINIT if ARCH_WANT_HUGETLB_VMEMMAP_PREINIT =20 config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 177455d98064..8bcb522645ba 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2095,9 +2095,6 @@ enum { SECTION_IS_EARLY_BIT, #ifdef CONFIG_ZONE_DEVICE SECTION_TAINT_ZONE_DEVICE_BIT, -#endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - SECTION_IS_VMEMMAP_PREINIT_BIT, #endif SECTION_MAP_LAST_BIT, }; @@ -2109,9 +2106,6 @@ enum { #ifdef CONFIG_ZONE_DEVICE #define SECTION_TAINT_ZONE_DEVICE BIT(SECTION_TAINT_ZONE_DEVICE_BIT) #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -#define SECTION_IS_VMEMMAP_PREINIT BIT(SECTION_IS_VMEMMAP_PREINIT_BIT) -#endif #define SECTION_MAP_MASK (~(BIT(SECTION_MAP_LAST_BIT) - 1)) #define SECTION_NID_SHIFT SECTION_MAP_LAST_BIT =20 @@ -2166,24 +2160,6 @@ static inline int online_device_section(const struct= mem_section *section) } #endif =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -static inline int preinited_vmemmap_section(const struct mem_section *sect= ion) -{ - return (section && - (section->section_mem_map & SECTION_IS_VMEMMAP_PREINIT)); -} - -void sparse_vmemmap_init_nid_early(int nid); -#else -static inline int preinited_vmemmap_section(const struct mem_section *sect= ion) -{ - return 0; -} -static inline void sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static inline int online_section_nr(unsigned long nr) { return online_section(__nr_to_section(nr)); @@ -2385,7 +2361,6 @@ static inline unsigned long next_present_section_nr(u= nsigned long section_nr) #endif =20 #else -#define sparse_vmemmap_init_nid_early(_nid) do {} while (0) #define pfn_in_present_section pfn_valid #endif /* CONFIG_SPARSEMEM */ =20 diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..2c385f8b2944 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -461,8 +461,6 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. =20 -config SPARSEMEM_VMEMMAP_PREINIT - bool # # Select this config option from the architecture Kconfig, if it is prefer= red # to enable the feature of HugeTLB/dev_dax vmemmap optimization. @@ -473,9 +471,6 @@ config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool =20 -config ARCH_WANT_HUGETLB_VMEMMAP_PREINIT - bool - config HAVE_MEMBLOCK_PHYS_MAP bool =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 97888f146571..0aa0ab4f8b11 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -559,19 +559,6 @@ struct page * __meminit __populate_section_memmap(unsi= gned long pfn, return pfn_to_page(pfn); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -/* - * This is called just before initializing sections for a NUMA node. - * Any special initialization that needs to be done before the - * generic initialization can be done from here. Sections that - * are initialized in hooks called from here will be skipped by - * the generic initialization. - */ -void __init sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static void subsection_mask_set(unsigned long *map, unsigned long pfn, unsigned long nr_pages) { diff --git a/mm/sparse.c b/mm/sparse.c index 439802e6a6ad..948839621f83 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -284,27 +284,20 @@ static void __init sparse_init_nid(int nid, unsigned = long pnum_begin, if (sparse_usage_init(nid, map_count)) panic("Failed to allocate usemap for node %d\n", nid); =20 - sparse_vmemmap_init_nid_early(nid); - for_each_present_section_nr(pnum_begin, pnum) { - struct mem_section *ms; unsigned long pfn =3D section_nr_to_pfn(pnum); + struct page *map; =20 if (pnum >=3D pnum_end) break; =20 - ms =3D __nr_to_section(pnum); - if (!preinited_vmemmap_section(ms)) { - struct page *map; - - map =3D __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) - panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); - } + map =3D __populate_section_memmap(pfn, PAGES_PER_SECTION, + nid, NULL, NULL); + if (!map) + panic("Failed to allocate memmap for section %lu\n", pnum); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); + sparse_init_early_section(nid, map, pnum, 0); } sparse_usage_fini(); } --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9129A44A417 for ; Wed, 19 Aug 2026 09:53:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133235; cv=none; b=Ycvt1NxXw31kjbqvP9RZUABKAHOtZ5WdLP3pcU1HGfL9+7fPzPxmZG1NA9MMM5rzXeK3s1vO7dMqGbNlgrE17CvdiAMlCImLTbr9qOe+he+le7qhD/SlIciu+pOjhK2qMzLE949ZfXbXW6Y57gvvbXXx11pQuV2HWi5+sLQCUrI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133235; c=relaxed/simple; bh=WvHs/YYIPDCuTcJlnFuFsuo6HzyLFJYgVm8kdN3/Vs4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cJ0Q5HDW7qGqLF6ouFpSsN/xUGGK7jGgfhPr1JL1kxcFUhSCie8Nhe+RYkrdxRoNXEpf1Lx7oyEdJpTNmDR2z4PcSliaAv78MH7OKLgPuo6F/rg50i7z5dtHHcQYJeLsdtU5nawaQPGERhAPX4x5Ndku2OG1M+H5E7Js52/XVVw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PxYf5xqc; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PxYf5xqc" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2cacf197759so11010335ad.2 for ; Wed, 19 Aug 2026 02:53:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133227; x=1787738027; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tfN6zKvJyOETRiPLuhLazGAgAbhX73SzFIauwVYcbkM=; b=PxYf5xqcElR2aoNb9/TJRIodRTqIE8s7esECuzAL5qtGdS0mhkPJaVjRohkAtUs94m erd9eukbBfr+rM2GlTuTwM+79lVvlx1k3RvG0taPR3O3k3tlvP6aFiNCS/qKBwNUXMIF ApBD/yxsj04g/ti1umhBUmg85YGAUJQlD1Cm2KZRENiAQqG8BAQGhm+TqBiEHR2hDtki UjQF1sIluqM4aabwM1JYRlNloSU/A9B7LiiZ67vseRJ4UEKg5xyrLywmHT6++p8+3QG6 QLUcsaPXx6S6YDdbrvw6EEt3MfwvJLRi2OmYzL+OCa58fv1QjHrE9xsGj2uUhmkYMgeO aWBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133227; x=1787738027; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=tfN6zKvJyOETRiPLuhLazGAgAbhX73SzFIauwVYcbkM=; b=r1c+uyijJ/AvTZSmhpOQebBWzxje8VWTdFlfAYzygw+5Xuy7vR/Oy55nwtHJX2N4EH KTWHa3INVPYyi6y3zLK/7qoZGln/OxxBgwZoCtWuWOm9yc+XhaHiarw+aBRNqrTu9Q2E YQ8JrAlkpZHRWTX91w5zCa+cvr5vjdtLyCwN3ehodvJlLSKgarRIyhoB3sYvh67j3xrv gNsN8oh9okFJ+jNAO1NQhuhps8uHzyLeK1jfplVMqzhLYUFsHocp3VifBRXYMo/eEQLi H2FF1kikQX2ZFw3j8z4HlvxLhiQYDiI8PZ3leRskDn+uAJMnCrOcQGAkKrKrh8tbVIES gNQw== X-Forwarded-Encrypted: i=1; AHgh+Ro8G4f9rBW9byqeiC1fSq0wYZltbAM9kMurEIK8LiXzNp/ltysXiZdXE8YUCMjj8f6VpAM8KpP2oq58dus=@vger.kernel.org X-Gm-Message-State: AOJu0YxhngMSAlIkxU3HkvdbTkBPx7o+s3L+e05JKR/EJ/gPePXbRbqD 56qIRBR8HkiAvrea0mkq0GeoCC5n2cIlu9/0UATrKarHnz4jgXd8wR4/1Ozi7+ayfHg= X-Gm-Gg: AR+sD12bb7uduSsF9AWkYyxbmvx6hk4ecdF1SpS14idOEB8m8XtuoxYJpMcrwW8E0PN +Fz+YW3U1JGlWUXJg7JBUhzGbucJKKuvxwohm5338FEGfF6j5wanNsCKk2OGjjnALhrGzfLzKn2 8cqAtxsp3cnAek/68B7aVlJprSOmm77wLLw8zEqvJ3tqrAClKj89v230MgU1wD8ZztuhMS355aZ YRgyCqeM9vT57D8HQWfoarTRhHuL0L4BDaaR6zUhqj3diObBqiLHxYiprs3YHF9KiC/I5gzTpz3 INadjlgCBvQLV4nYxG7ZusQ6GvTrytgu3Cx5/NbiqBNsmzsDywsooJ1+xAFZf5tfQDucOwkeoqq 6gWDf/6h3oKkLGHkmaudc3m3nToZ5mVo7DEuex7grWd3l7kmb2QvK2yNoQl8DiGT15Nx+Gkk6fa YbERkLdOfMN6DmJHqAT0lKWlpKhbkfMii7un/XQqPV3OFGDBjayEa5jHlGopwyvylMtUximYc90 tOdRE4QyZWreKCH9yLXg7BCC+CzHdGnDKXs X-Received: by 2002:a17:903:26cf:b0:2bd:c925:3a16 with SMTP id d9443c01a7336-2d5fd684b75mr61834225ad.2.1787133227279; Wed, 19 Aug 2026 02:53:47 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:46 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 12/17] mm/sparse: inline usemap allocation into sparse_init_nid() Date: Wed, 19 Aug 2026 17:51:34 +0800 Message-ID: <20260819095140.17252-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After removing SPARSEMEM_VMEMMAP_PREINIT, sparse_init_nid() no longer needs the transient sparse_usagebuf state and its helper wrappers. Allocate the usemap buffer directly in sparse_init_nid(), pass it to sparse_init_one_section(), and drop sparse_usage_init(), sparse_usage_fini(), and sparse_init_early_section(). Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/mmzone.h | 3 --- mm/sparse.c | 46 +++++++----------------------------------- 2 files changed, 7 insertions(+), 42 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8bcb522645ba..67c84a8a7258 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2223,9 +2223,6 @@ static inline bool pfn_section_first_valid(struct mem= _section *ms, unsigned long } #endif =20 -void sparse_init_early_section(int nid, struct page *map, unsigned long pn= um, - unsigned long flags); - #ifndef CONFIG_HAVE_ARCH_PFN_VALID /** * pfn_valid - check if there is a valid memory map entry for a PFN diff --git a/mm/sparse.c b/mm/sparse.c index 948839621f83..7096d8e13802 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -235,42 +235,6 @@ void __weak __meminit vmemmap_populate_print_last(void) { } =20 -static void *sparse_usagebuf __initdata; -static void *sparse_usagebuf_end __initdata; - -/* - * Helper function that is used for generic section initialization, and - * can also be used by any hooks added above. - */ -void __init sparse_init_early_section(int nid, struct page *map, - unsigned long pnum, unsigned long flags) -{ - BUG_ON(!sparse_usagebuf || sparse_usagebuf >=3D sparse_usagebuf_end); - sparse_init_one_section(__nr_to_section(pnum), pnum, map, - sparse_usagebuf, SECTION_IS_EARLY | flags); - sparse_usagebuf =3D (void *)sparse_usagebuf + mem_section_usage_size(); -} - -static int __init sparse_usage_init(int nid, unsigned long map_count) -{ - unsigned long size; - - size =3D mem_section_usage_size() * map_count; - sparse_usagebuf =3D memblock_alloc_node(size, SMP_CACHE_BYTES, nid); - if (!sparse_usagebuf) { - sparse_usagebuf_end =3D NULL; - return -ENOMEM; - } - - sparse_usagebuf_end =3D sparse_usagebuf + size; - return 0; -} - -static void __init sparse_usage_fini(void) -{ - sparse_usagebuf =3D sparse_usagebuf_end =3D NULL; -} - /* * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_= end) * And number of present sections in this node is map_count. @@ -280,8 +244,11 @@ static void __init sparse_init_nid(int nid, unsigned l= ong pnum_begin, unsigned long map_count) { unsigned long pnum; + struct mem_section_usage *usage; =20 - if (sparse_usage_init(nid, map_count)) + usage =3D memblock_alloc_node(map_count * mem_section_usage_size(), + SMP_CACHE_BYTES, nid); + if (!usage) panic("Failed to allocate usemap for node %d\n", nid); =20 for_each_present_section_nr(pnum_begin, pnum) { @@ -297,9 +264,10 @@ static void __init sparse_init_nid(int nid, unsigned l= ong pnum_begin, panic("Failed to allocate memmap for section %lu\n", pnum); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); + sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, + SECTION_IS_EARLY); + usage =3D (void *)usage + mem_section_usage_size(); } - sparse_usage_fini(); } =20 /* --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 978CB44998A for ; Wed, 19 Aug 2026 09:53:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133253; cv=none; b=uR95H9Mp0VTZ+DLDWTsCHg9hMwPwd1VsQ97V+4yyslGNakXHiVkHVivxTuPnBu9fDq+2UI6P/M45dumtjgbZGAdf/Kg3s21DGOpmV0dXWa9T8dH7USUGSof2qw6zJ+lGT0Z3WNsWowZ802dmAFmEPmiVOlQa/PWNY9VKP1Jd6Uw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133253; c=relaxed/simple; bh=crJ0um78If1rNLpzQ2AnPLefon4RNSDyfj7UqqZTZgE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JaGbb/BmpYVEY+GQxxQtsv5Knsw0pZx0EhZqhNXIKuVhpu5m2Hzsh60jDdjsoObWUwu0kWRlLheLMS9r6C054bUMgBzkbjuzAZuclN9Yr2ZgkAJnG8dgLYeAzUXc0KDwT42I8uxrWKlfpkJ3N4AOuBi0HZMr37ddQmEBDJ9NDRA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=FGLVlfgf; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="FGLVlfgf" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2ce7d2adef4so10712495ad.3 for ; Wed, 19 Aug 2026 02:53:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133231; x=1787738031; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=1Ij9dEqKGuikzoEaZ+Dz4+F6pvL226lB4Ua/B4QAWHQ=; b=FGLVlfgfmSWHDq6r3qIGumil+a0U/mY//eIRMaRCYid9WAllvU3EemPMAgF6uLWgKJ sQ6LkTHbd6S9elHWpxCTNXSWrOM78CrG9gsCgFtNHH9L+rfgElyZBxkyELiw6ysPvQBs BgEfp4RfR0LCC2qjqnDQwCOUkWepE0VFJewFeTImKYBlmBEJcFI77jv+C0Aix8NWhRHa kX+ay9f8juhoo1e4NGWslg8ZTC5LJoNfEwkMXDcK+JcJaeHoaNi2P2BxMN0VvAbhOTbl Wg0AVtLkT4vLVdg3A+RYpVgVADd27IgtFvM9isfqe5WdTZzWuIdQ9lD9AEf3oOxtoUsx kyDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133231; x=1787738031; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=1Ij9dEqKGuikzoEaZ+Dz4+F6pvL226lB4Ua/B4QAWHQ=; b=eFDnBm/LxHrcdULhacawl7a/Y2MdYGr2bYxz7rOl6Dqv44MpNgVK3S3o9yaQWX3q/9 FyIwxX2ljBcFQUsnyfgnqfN7OfGR5Dk6HryzfePfgNY2x+mDwQzMzPA3sY8YNVvP5/of fA8kQ/j+GVOSawCGrIdEcMLoT2ms4x6Qi7TzutT6dbTOGB3cVlLSlq6+6hmStejHDTN0 jbtsUAfMn5V1MOAn6ayf2OfLlB6Uhg3vqYjji17QIhqRwdR0P6sVEzcjeLAAlQoqDVjC phCPCHH+E+zqQXr0gzc7uR7bqjaYUo5Y7obeiFiQQr1pGJ397GI6fslm5nP0Q7rfHvHM 0IBg== X-Forwarded-Encrypted: i=1; AHgh+Rru+2MSs3O+dk5csJBoQJNasHmvm+naFumhSbDqjMuUwluxGHbhAF9Yzptf7jEGB1JiZ17TAPVyeaduC0g=@vger.kernel.org X-Gm-Message-State: AFuF++nrsxVtMlLWfrlYrFDjTG19XJcTSW7p5JSb8OUFeFwIiNDvdUN2 x8JGcTr4Mfea02gOYTTDsbP/f1yIYdEBSIJgOXlh5Ri1Rsl7JQnO6+5UjV+ShDQs3Eg= X-Gm-Gg: AR+sD102pJHu+roRdg+cc7wfynT+v92kk0VT22iB1QtafP4qI4AJX8oB3sS0kG1uepB ytPHAOcd5rdODcfzuZm0SneAJxnOn/XgzhSkFKhPCFryQfJOxLx86YBV1LUdPxzdAMPWpVC2Pt1 6HPh7FmO7Yr3/Rqd/kz+qLUQvW8Zybkp6PTGOg5lq1uJISR9RS7C+FTgSMHRUylycwIK3mn3SNn v8OQ+DJWHwFwaK2YANg2EhSyiC2yRfcQtI3Cm9fasSEUih4ifsHC75fD9q1zZgyXHekfzZyHLcW z4SLorp9tnw09djZa/Vdiznr6PLUwU4NIfBd0nSvqVk8G5D6tkkZpL4bsfqZKSC3GZ2fVj02vmX fAqxa5L3sg7ctoxfAvQ13apb+drJ6OYd2FHLD+GgSCtkzl6ViwfSBOdlYJJx5fiRv4qh1rhJWMK xfAZkBcmJWagwfzXfJP40DXy9CgqLGIHfMlfWOUKcLtL0BPHmAmgKpTfZFt/F0mrHDpfreaqygf 2apIE+V9FmvPJK759GLGtIBIC3NggIP6rwl6g== X-Received: by 2002:a17:903:1205:b0:2ca:d91d:d3a7 with SMTP id d9443c01a7336-2d6017649cbmr54047305ad.10.1787133231219; Wed, 19 Aug 2026 02:53:51 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:50 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 13/17] mm/sparse: remove section_map_size() Date: Wed, 19 Aug 2026 17:51:35 +0800 Message-ID: <20260819095140.17252-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_map_size() no longer provides any shared logic. After the sparse-vmemmap changes, its only remaining user is the !CONFIG_SPARSEMEM_VMEMMAP path in __populate_section_memmap(), which can compute the size inline with PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION). Remove section_map_size() and inline the remaining calculation. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/mm.h | 1 - mm/sparse.c | 15 ++------------- 2 files changed, 2 insertions(+), 14 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2cad82cc466d..be2865c00cf0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5140,7 +5140,6 @@ static inline void print_vma_addr(char *prefix, unsig= ned long rip) } #endif =20 -unsigned long section_map_size(void); struct page * __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); diff --git a/mm/sparse.c b/mm/sparse.c index 7096d8e13802..9349ed6326c0 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -209,23 +209,12 @@ void __init sparse_sections_init(void) memory_present(nid, start, end); } =20 -#ifdef CONFIG_SPARSEMEM_VMEMMAP -unsigned long __init section_map_size(void) -{ - return ALIGN(sizeof(struct page) * PAGES_PER_SECTION, PMD_SIZE); -} - -#else -unsigned long __init section_map_size(void) -{ - return PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION); -} - +#ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - unsigned long size =3D section_map_size(); + unsigned long size =3D PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION= ); =20 return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), nid, false); } --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B00044E67C for ; Wed, 19 Aug 2026 09:54:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133245; cv=none; b=OuPKHnUhlgSYhKn5SNUful+UqMXdASVVsP2rUq00AcH1sT73Ebkabqy9If0y6JAljcxwwxu6iTtFovqqlQkNCzMK2xvClTa4TPIskvTZgh/whyocpA9U3nWlYBldhalb7NGwhDqjMgBzk/mej6J2DXUwYk9kLggE4aoEaBmndpA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133245; c=relaxed/simple; bh=oaPO9eZTPKL7BIRxynS3nhDWKMopQaCn1lya/KaYfu8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LQxz6RS8rjmuETJ0mwOdr4v1VNm1Huu7Qj+sOwlIXDeFBdX170arPRVUub4S6CP8ygnctCnq3xV9zADD1FxogqQfw5v++4WATeDFvJAPgVgqi/TWj8EMw+utSCxp835URWzMk/BFa3DNFdQmNZ0ZoVfNiUFtPwhnRQl0Z++Ku1M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LtfERHcn; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LtfERHcn" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2cf6d65d8a7so9687885ad.0 for ; Wed, 19 Aug 2026 02:53:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133235; x=1787738035; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Q6Hg9s0CPg01GOWYZ58NmcFpF0VjmJuzkxO0BlFJkPg=; b=LtfERHcnHmKyHc3JeHQbdhcBL6dmKHSTrA9vFvSG1Q/IDXZmB4xPEhApGDwBekS7r7 3JeTqJ1zXhmbBi1nWKD8sKRI9vVfXlLvCSvvSmglEjsgUPaL+aQyu5Rf7B5Pw/7RgGlC RebzLvRR5rMcrs38Gfi0pYLBc5c+3bwCQq21+aTYcByRr995ssM9uvE4E26bxOxv+971 7TrFciSfC6zkOj67AbHb80h0vyVcQisG31m8CITt0rDRuqz9d5n3tmh8BWjoff3k7q2i 4p4xcKBwgcn5d/6hWsMYiW9D4vdVHXCzTT5VDCCAQO0vADVCMhFwFu+iMxXtLwkM8Q9v tu5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133235; x=1787738035; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Q6Hg9s0CPg01GOWYZ58NmcFpF0VjmJuzkxO0BlFJkPg=; b=HbfC6u0hjzsJpEvpPm6R6hHA3NxpeKvn7yKTXFgxEhRHT6xtGd1uzzye54CNEhIAVL eVLCiJTKM541CrZn8fqGVV8sZncGg2F7kZc8zpjgJSh4hAKrCVaapndVyd+kMxthIXQJ YKISeLeNOXH97psU3oCei/EZPBH/9Txa1tRdBIAgEFAJ6IIDKZt1rfXwafnlUNVSmitN LJYGTf5bOMhTDkniBD2a6X335G9VT6RcZBk0nZBfKQyF1kfWiFO0XPaDOX/Y1FhKqecQ M+Aouu+w7N7/27DPEpFuJaW+pNWG+zKNyADViLjYSn26Qh5bUd8YY83NzJYpMvB7CgPC fXWQ== X-Forwarded-Encrypted: i=1; AHgh+RqX43SKSEfbdTBSuZ4gLbI+Rc/jv+stYqoXqqd5WOP0oJaTP9oW8gTCFVe4s0UClL+WG2vM4yCKcl4Iu8o=@vger.kernel.org X-Gm-Message-State: AFuF++lIGhrsijeFNxKBol+GwFY/dtjap6whCceXvaLndeY2TaZ7whVV mWpvKmSZFrke4NHXePd29j3XVlGN3FlZ/bhNUDbYQouuomR/eua3KZiESYe5bx9uk3Q= X-Gm-Gg: AR+sD10eeyQtFgHyEoFY6o8yNLYtPai+0SqW+HKvCrEaQcPlw3pq/Fk36uIpyHzn2Sg 5rEUJe0rFtnfgdvwLNw3EaxRh/FSjwXssTmQitOFJsLPs2R62JzDzOqgPTbjEeH4P5XKK4o8nao 1dJPLCSf7+6Hlq3pqC/O3PVKwZBmnv8EZZXzBIyqur7LxYYvFffu0UZqibWPNgS9h7VeRMNIKeq aolGQ0SNJLX7PmL/G2B6R/Ms5FaF5JlUPxLm4OuR3pX9kgb20fsVNf8nsP9w1T4dnUB7z/73KV6 L8aYELYBEcJyOorKtY/tb50hFfmlo1GOX8IQM2HeVZp00wxbrgdbq3pb9Uzy5zz42I3J+iGtjn9 5ql9wmJWI/WtodO2G6OIiSAXGwFQQHNsGi7O3eajAjoB0B5wQfnjb7JA+LJndtp4ArkI/tmNvpR faV+ZYI/x0Hego1pdspFZFDXTfK2QunVb/zBrm4LZ+4GY/8OTsDxUoO5ecbYskkNNFczahWq6sH c1mda5JoSL381TwfqJRCxOduU3azmwO+0Zm X-Received: by 2002:a17:903:4b48:b0:2bf:dd0:c8b1 with SMTP id d9443c01a7336-2d5fd432110mr63480745ad.0.1787133235251; Wed, 19 Aug 2026 02:53:55 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:54 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 14/17] mm/hugetlb: remove HUGE_BOOTMEM_HVO Date: Wed, 19 Aug 2026 17:51:36 +0800 Message-ID: <20260819095140.17252-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The HUGE_BOOTMEM_HVO flag tracked whether a bootmem huge page had already gone through the old early vmemmap optimization path. Now that HugeTLB uses section-based vmemmap optimization, that state is already reflected in the section order. Remove HUGE_BOOTMEM_HVO and its helper, and use the section state directly when deciding whether to mark a folio as vmemmap-optimized. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v3: - Use the order-based helper for the bootmem vmemmap-optimized check v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 5 ++--- mm/hugetlb.c | 12 +----------- mm/hugetlb_vmemmap.c | 2 -- 3 files changed, 3 insertions(+), 16 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index fe28f98e1b22..3559041a5a57 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -675,9 +675,8 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; =20 -#define HUGE_BOOTMEM_HVO 0x0001 -#define HUGE_BOOTMEM_ZONES_VALID 0x0002 -#define HUGE_BOOTMEM_CMA 0x0004 +#define HUGE_BOOTMEM_ZONES_VALID BIT(0) +#define HUGE_BOOTMEM_CMA BIT(1) =20 int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fc579ba9ab39..ef41b82493f5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3196,11 +3196,6 @@ static void __init hugetlb_folio_init_vmemmap(struct= folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } =20 -static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) -{ - return m->flags & HUGE_BOOTMEM_HVO; -} - static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page = *m) { return m->flags & HUGE_BOOTMEM_CMA; @@ -3292,12 +3287,7 @@ static void __init gather_bootmem_prealloc_node(unsi= gned long nid) HUGETLB_VMEMMAP_RESERVE_PAGES); init_new_hugetlb_folio(folio); =20 - if (hugetlb_bootmem_page_prehvo(m)) - /* - * If pre-HVO was done, just set the - * flag, the HVO code will then skip - * this folio. - */ + if (vmemmap_optimizable_order(pfn_to_section_order(folio_pfn(folio)))) folio_set_hugetlb_vmemmap_optimized(folio); section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); =20 diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 7293706b532f..a25adc474351 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -714,8 +714,6 @@ void __init hugetlb_vmemmap_optimize_bootmem_page(struc= t huge_bootmem_page *m) return; =20 section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); - if (vmemmap_optimizable_order(pfn_to_section_order(pfn))) - m->flags |=3D HUGE_BOOTMEM_HVO; } =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C2C836AB5A for ; Wed, 19 Aug 2026 09:54:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133247; cv=none; b=PAU1X8T7V2JhfB42DWooSJ1n7U4Mt7JfqSl/3on/jRvJwBnwKsI5HMmoqajUdFqGk5BBWYZ31Zfal8Hj8iuw4fD3O8hIGu3K69rrRpGKqbvJT5IICVhzE9aTt/0ln5Ir+iuL0vglhO4BDKPkTFZT8lndAG3WHyD9xw6uPSGPH/0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133247; c=relaxed/simple; bh=86LBbKn8CxSH5COWHpLd0wRiT82VydbOzqKebBBrKjg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Wwnuxp7SiyM2/JhhIv3WB+YY9M9QkBF4aid8FltpDZFWt6cnMSqF63uCVa7vvhEaEvm+WZx0szEwsaujS1MTu5xB6THWh0FGdWy+lhm2itlNlPgsf4vPlxQ8AszIaEBVVsa5L5wLgQ4rrcEVI/v1tIhbuXMt8zke1w+M+zn53tc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=OBuPtUI0; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="OBuPtUI0" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2cf27856f9cso6643455ad.2 for ; Wed, 19 Aug 2026 02:54:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133239; x=1787738039; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=JaODolDH1UCHuAYQUYmSzn0eUVKO4ynwv1wpYt0umIM=; b=OBuPtUI0nxXiiHOJ8UCmIfZmXQyIzdu8qNDlX2y/Y671LOEdctMWimzCBIXnqVuafX ZyFf9JIG0s5l2ilYyg15aB+b+kTi5LVlhBtDmVXUeCrsgH46mQYPiOBjbOrqiacznS+s SCAvp4LSoQAxwkjCdnk006UT6Ms/lnvaNFITnppiibZSKzMBtWuOAv0r+GE1ZIS4xDdC yHNXfw5HsF4IsEFIyj2tv5GfrRXca5yBVHFYI8EOj9Dq9e2DGUl156L7y9NOW3bPGRmF r/m/LoaENe6JNt1U7vpQyIASuMa8FuEcu8IaPJpD0d3fuC8emrXlreMzsAqjstPJ4lqV pU/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133239; x=1787738039; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=JaODolDH1UCHuAYQUYmSzn0eUVKO4ynwv1wpYt0umIM=; b=NEZETig5ZK4029Y/C0z2QoEy8RNTzzevrePEyE0E5ML+Prt8IY7TVIuzUt0E/s33nW w5XOLfj2pYPQhXDWtzJDdyzr67+0G6c/ok7vYM5FOJ3oagvBK4Rq4b1NfrkzBHWsH2Xz ccUy8amBTo+3O74nWZN5MPAeoOlbwv2o+cx2DZDTVLbdExErXYEVFamXaPNQwo5BDahO y/50pjpBECs8XtsW59cUXVhWHEqP9bv+wEEtdP2/p7RPMmM0QNNDCtzTqIKh0zMj/+77 uy5zksOr0nA6hkx3ppFwtYJTHpys+rTjVTbGzo5vSAOfEOV4bgvUoGHbhjTh1Fcwvbnf gdqA== X-Forwarded-Encrypted: i=1; AHgh+Ro84O2sIoi2jl2qAH3NXBVxKb0qyZ4YUud4OnaHDMFD3DGkZicYGalQrtHDSpIImtbQCZTp1fUZUoZW9VI=@vger.kernel.org X-Gm-Message-State: AFuF++kzRElGNdfyDIigugVK2JxipqdRxwwhe1ZDD6tvQKCPwQbpLk8+ o0alcHWh2RJEyWK/2MTrowVsU1/jBY9cEWm7JlCq5vl5ZTo8eQFCIm3j5msGCPV52jU= X-Gm-Gg: AR+sD133yi+Mrb8hnFziYgPx19Xkz+5reyWh2LLuLZVFa3i2O/V/s37SCd6nSb7LdlV bmdkRrYJwMKHXwqw3SfvuHIpNSbJ9S8E9dHYc0uI4doB9itF9JFjAuwBOajN0Wx+o2wKLJy32I/ tkTYWX1zRidRRJRoYKZTAXKkx+h6802Lxe7S7pm4aMahZ8YPZmAZZSf6V62nIAXcGyV2ZaW6klP esxXMItQytKH1JnwP10hi9a8kARHhGIH3f9/tw9VEOgOIuDp+8XQA8F64FCxF7X6RikdwLhgptx bbcHesoqGOT3CyAifgCpovqMnN6pG7pWCv2Tj2+J1sQDoPtw/yrk9HZj05A9i4mGtijlW958ZnA wC+OOrt21p8HxjvYHyQcbrcfcL7Jh3HZF536AYXBY9YTz3JPot4qNvGvCKmibej9Sudva7a3PJO Vs3+FqHNvooMxyrcpkJ2a+A7xvmi7lgaGzLdgSfquzCG0eXQYREgNjD/G5a/QERYjqOU7kYTjf1 NiiSpZPzS9JyDG3JmFujsw4ng== X-Received: by 2002:a17:903:1987:b0:2ca:53e9:1277 with SMTP id d9443c01a7336-2d5fd7130edmr57180835ad.1.1787133239166; Wed, 19 Aug 2026 02:53:59 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:58 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 15/17] mm/hugetlb: remove HUGE_BOOTMEM_CMA Date: Wed, 19 Aug 2026 17:51:37 +0800 Message-ID: <20260819095140.17252-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Track early CMA hugetlb pages from the hstate instead of storing a redundant bootmem flag. This removes the unused helper and keeps the bootmem metadata minimal. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 1 - mm/hugetlb.c | 14 ++++---------- 2 files changed, 4 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3559041a5a57..255a258f11d1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -676,7 +676,6 @@ struct hstate { }; =20 #define HUGE_BOOTMEM_ZONES_VALID BIT(0) -#define HUGE_BOOTMEM_CMA BIT(1) =20 int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ef41b82493f5..03665ff92696 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3120,7 +3120,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) */ INIT_LIST_HEAD(&m->list); m->hstate =3D h; - m->flags =3D hugetlb_early_cma(h) ? HUGE_BOOTMEM_CMA : 0; + m->flags =3D 0; =20 /* CMA pages: zone-crossing is validated in hugetlb_cma_reserve(). */ if (!hugetlb_early_cma(h) && @@ -3196,11 +3196,6 @@ static void __init hugetlb_folio_init_vmemmap(struct= folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } =20 -static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page = *m) -{ - return m->flags & HUGE_BOOTMEM_CMA; -} - /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABL= E) @@ -3291,9 +3286,6 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) folio_set_hugetlb_vmemmap_optimized(folio); section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); =20 - if (hugetlb_bootmem_page_earlycma(m)) - folio_set_hugetlb_cma(folio); - list_add(&folio->lru, &folio_list); =20 /* @@ -3304,7 +3296,9 @@ static void __init gather_bootmem_prealloc_node(unsig= ned long nid) * For CMA pages, this is done in init_cma_pageblock * (via hugetlb_bootmem_init_migratetype), so skip it here. */ - if (!folio_test_hugetlb_cma(folio)) + if (hugetlb_early_cma(h)) + folio_set_hugetlb_cma(folio); + else adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B669F44A40F for ; Wed, 19 Aug 2026 09:54:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133253; cv=none; b=KwmZRyxdr7hN5frPLWQE8KMyoRdgnyAi0TzO6hP+svNq6arggvtxziEat8wM8t03KzOlGqs6hy85VPeLzEVBnW+X9+qsM8sCwogleK1mzvZF6MB6Es7DzzGBcuMKBZ2J/kd5hLSHTfTjf94/3R+p1tW34/lRC4Ba4fVNKK6kqAU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133253; c=relaxed/simple; bh=DPCkGNFDamp2cg3/ZuPNVC1FhkmJDgi2+6JTbf0jXLQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ss+97hTupZpR86Pe+K0dlQky7kp0psBRNDFOWW9PYIhFsAKpvjDNmkF8y6EkyXcBppcaCC519Vv2iBTOyzsQLRQXQRONLbBpL8ggkC+F+wJ3/EHRH3WTn7nUToPS/RwOEDEJyzgWk/PFKpPaMUHcfU2/80P5lqcY9wXLgdplaaE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dBRdLTRX; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dBRdLTRX" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2cf27856f9cso6643805ad.2 for ; Wed, 19 Aug 2026 02:54:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133243; x=1787738043; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9VHhccCkICJlyi9ZKauHubucoGPnSLODJt3qpXuHQto=; b=dBRdLTRXlkzhGsihH9OkQ6iqKXukbXEorLS9KzS3kSOzXfLD3XHF4J1yf+bMStwdAU fLzeI33nel1fMhrlBDnSejeo5bnjhaqGcjMuUoC411qExPkaVAUJcqaPmNvtY4XuLJMu r3E8TKZuwQUsQDP6cb22OLjK1yQTGOgNRLsnHMCXd4KQedaZaDwdnxur9k2NRCrBv9Cc WvZGexH4nQvDfGs16U6eFbFNSaaEXbFcuzQWwsoRgLsxiUrwdXP6T2ZIs6egAECnvMmw ac1SeFY8EhNe1wzOC0Fe4CrXTSdI0vqXz4aeJ5W1RBte1w5FsOWpLH2GAuTQt2QiZNYk WJ6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133243; x=1787738043; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=9VHhccCkICJlyi9ZKauHubucoGPnSLODJt3qpXuHQto=; b=n7QkSVZuFZhqxLBmY++ao78oOBr3CiEIUf5x/iVz8pBKdDXPAXPEV2lofvNl6IOpPC EfX4zZjVlQ65vXEbPvcBFF9FwyDblq+wVtfLQx1IWi2k9r0oZ9cxHTzlD9VYgffc9KBW Bn4l6tCuOqgSh8+Gm6Q3P8ov4brU7o6mkGiKKM7is5UIikOT3iXXZVWQT5SgLpo3kdNM OU3SXwsicOSkc89TIbTsUb+HZZW0qFlYEEtvzS+QTyhy2GmTskt7OWCEKa96EHuBZpEX xVMv2IVtofN40nGFzjHQ2OO9vqfmrB66eNCIzJ+vVbawt0BQ1XZCUceHerkqz5TZBYZr 7DBg== X-Forwarded-Encrypted: i=1; AHgh+RrEwQgqq7q54RvFvYrfUk5PaZau4o4M3/gk/Tb3jngmsvyjhL5rRdOwhdDdw9fOpv+hv5TglXY2jD45DfQ=@vger.kernel.org X-Gm-Message-State: AFuF++khlLdzK9VQ5m8JOG6JpTe0AyAGHp5LQa/3ZmTmrlSTd+DB9qD+ 6YSGy77wvP+IetbjLFbUODWwMW9EBddjjYzo/4VR9H/p6MB4t0JBKz/LgxUw8axzdd2PJw0MU5r todB5 X-Gm-Gg: AR+sD10PJg+k8z+UfC37378dGPCSDoZGPJ0xvwaFcH/cl+aUZE6499sRowaqNL1mYP+ 8Zepb4xBQqg3aksvOwU1UPKNWwbM9dgWA7RFTJKj0VKqdiycImEqb0bdBpTfb53zQiPTSZR5kbl WDEoqTVctVFxUW7n/tZEI5N9bQgft7evSrBkR69z5ydRtEFK9PjxA05P5Bv03v5BHxdyo8wdNfS 6LLoSI85tTUUFVfvrediAF3qUk/i8REGf/wp1EL7Ou93XJnBwhLLRKpJXcKWFhr86RMjeOFwoWA JCe3xK+ZmBXYW6xLD0C5haO6lLPH/ALYsq2z0Koy7xbLYxR85Yj5YM7+mw69yxYZGxqs7OWdM64 to6H8+R0HgTUM6gL2wJ9Q/tsxSIP49NGNVu56UqKCYcIviZRAtFWZcLcTxxIy80v8l+CNT2iYO5 ZUc7S+8QuHIKMd67RUaGgM7i7GBGYd7QygMUFD2z8++2rP2cPWNPt3NdoU9jHWgLrGzWvOJztEz 6M8i7SvRod3XwKOXTBb8T4t6V4qSVs= X-Received: by 2002:a17:902:e5c5:b0:2c9:e9db:8167 with SMTP id d9443c01a7336-2d601860803mr57208225ad.7.1787133243476; Wed, 19 Aug 2026 02:54:03 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:54:02 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 16/17] mm/hugetlb: localize struct huge_bootmem_page Date: Wed, 19 Aug 2026 17:51:38 +0800 Message-ID: <20260819095140.17252-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct huge_bootmem_page is only used by hugetlb boot-time allocation code, but its definition currently lives in mm/internal.h because hugetlb_vmemmap_optimize_bootmem_page() takes it as an argument. This exposes a hugetlb-specific internal type more broadly than needed. Change hugetlb_vmemmap_optimize_bootmem_page() to take the information it actually needs. With that interface, mm/hugetlb_vmemmap.h no longer needs to include mm/internal.h, and struct huge_bootmem_page can move into mm/hugetlb.c. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Pass PFN and order to hugetlb_vmemmap_optimize_bootmem_page() instead of struct hstate and bootmem metadata to simplify the code further - Collect Acked-by from Mike Rapoport --- mm/hugetlb.c | 8 +++++++- mm/hugetlb_vmemmap.c | 8 +++----- mm/hugetlb_vmemmap.h | 5 ++--- mm/internal.h | 7 ------- 4 files changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 03665ff92696..b2b80284c4ce 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -55,6 +55,12 @@ #include "sparse.h" #include =20 +struct huge_bootmem_page { + struct list_head list; + struct hstate *hstate; + unsigned long flags; +}; + int hugetlb_max_hstate __read_mostly; unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; @@ -3138,7 +3144,7 @@ static bool __init alloc_bootmem_huge_page(struct hst= ate *h, int nid) } else { list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |=3D HUGE_BOOTMEM_ZONES_VALID; - hugetlb_vmemmap_optimize_bootmem_page(m); + hugetlb_vmemmap_optimize_bootmem_page(pfn, huge_page_order(h)); /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index a25adc474351..eb339c4a71f4 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -19,6 +19,7 @@ #include #include "hugetlb_vmemmap.h" #include "sparse.h" +#include "internal.h" =20 /** * struct vmemmap_remap_walk - walk vmemmap page table @@ -705,15 +706,12 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct h= state *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } =20 -void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page= *m) +void __init hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsig= ned int order) { - struct hstate *h =3D m->hstate; - unsigned long pfn =3D PHYS_PFN(__pa(m)); - if (!READ_ONCE(vmemmap_optimize_enabled)) return; =20 - section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); + section_set_order_range(pfn, 1UL << order, order); } =20 static const struct ctl_table hugetlb_vmemmap_sysctls[] =3D { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 20eb03df542a..464192e32dec 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,7 +9,6 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include -#include "internal.h" =20 /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -26,7 +25,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *= h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *= folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *f= olio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list= _head *folio_list); -void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); +void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsigned int= order); =20 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -77,7 +76,7 @@ static inline unsigned int hugetlb_vmemmap_optimizable_si= ze(const struct hstate return 0; } =20 -static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootm= em_page *m) +static inline void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn= , unsigned int order) { } #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c9..c37a468d53c1 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -23,13 +23,6 @@ #include "vma.h" =20 struct folio_batch; -struct hstate; - -struct huge_bootmem_page { - struct list_head list; - struct hstate *hstate; - unsigned long flags; -}; =20 /* mm/workingset.c */ bool workingset_test_recent(void *shadow, bool file, bool *workingset, --=20 2.54.0 From nobody Mon Sep 28 17:49:02 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9267244E037 for ; Wed, 19 Aug 2026 09:54:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133259; cv=none; b=IhyF6iaxRhv2Z8MOKe4TIE+XxT4eqhw16QI7hCBuf//tsI3XWrh/ktZoZBmFzoEgeb4i9eAONacV+igvzA1UoPrA4HoCQG9tqI9TObSLehlK6roS8LzisuRl6aBO+yotk4gvDFeGgWaSRugUHyJPlNUF9N3Y6dDMVZIwG0zHtXc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133259; c=relaxed/simple; bh=9kF0wvrlfCuictwKhSOE01DRn+aGiGVHfw7YJ+eUOfI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S5N7HUndZHdMSR2OmJJ/WwE8mfDbfTi5DRxFG65qYZRMV/5LCE6uI8PJOhCGBa9uP2Y+TY6w7TMHkNVAnVxUMsp1Lngmb6k6nSdrv/LQsq1YA52yHsnGC+GUMMEwuGgNXXUZq70VKwgWDSmyMMYV8AKvO6UtJVYEu9C15iIANZQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=AzEdXI4m; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="AzEdXI4m" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2d5cad1a6baso6580595ad.3 for ; Wed, 19 Aug 2026 02:54:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133248; x=1787738048; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Dl94Y3Ocjmgq20YN+NWMCqSvUO2QWpmbB00opElt5jo=; b=AzEdXI4mhABleNGJ660C5F21347yydG/fwJGRuqfei15ViKpNuWSVnh5556Dr8bh+a FCJIGr+XrN4xFfbkgdmvXsaie0XDAI4uCj2kMqtVOzVJguKTieE7UATXYSJbl7esFQpk iFhknEE3CQ7zXshSpdrWY34DKKcrjmTfNb+euhyxMyb725NQT9sidgPdnL9RbfP1QwwI 3bm1pEq94cp1Ip3zb3jFrWa+eT/GYGhAA7hkqbrNhOQeq7/1q3Diy8f2ADBz6JsifgFv 5lzBLwFC5I8xv023b+igkydrvlSYcWQ1jvtNPtANfTANYtNC41RdxedR383CoOIXqVYo 657g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133248; x=1787738048; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Dl94Y3Ocjmgq20YN+NWMCqSvUO2QWpmbB00opElt5jo=; b=qFXHn/iI7Sw06qYbOuIYYaCeBGNsm5ZhspbR06R9J0BHCexDuj3EiAHFfkBakHwc25 ZfagOLH3H7088ZFYOd9RdX7FehVXTGaHDSuvQq74glfvHrvsF4pt/eQZ414hhxgcDI5l 3H5+rXzee/t3Kp1f1b+W38meSP4gE/bnhylKH+pgvA5YZxPJS68oL4lXEw4WgdZZ4/Xo E5IRB+ET0ateEy70jtkY7SDmVIFL/cifDTGvm5QjSVZUF/sWJ2dhlZVhT8X6Lh4SN6zd 21iB0pS6j5cOSeB7n1YyVvQIbFOkr0Y+TXnwDJS+v2UIpQRQpYqZRswuzBtVgJcK+4LJ gODQ== X-Forwarded-Encrypted: i=1; AHgh+Ro0in0Wkgvm3bqszmXv+dvPYff63dgJYv2UNTMksf7mGdc4KGBWZ3c4HpEjinmomax1d00aPmkSVW4taL8=@vger.kernel.org X-Gm-Message-State: AFuF++kQab8e74NvDPJHEQnjE1zhEGDO8Z+Dl/rnN295M7VbfzPUPiWf eSIS4yJsjoosMm6W3OdcuJS3r4tMN07w/tOa2/9kTXC7ZDlwnhajKV4ST/BPVfIbq5k= X-Gm-Gg: AR+sD13TvrwQiv+bEBjxK5IxM69aZFjqc228CALUJbGfsneXUi3WLzaRgTPUqR6VWl1 qctsuJcmrZ/FYnPotTmQ4zaRLyjaCt3qarLJSpKBda0Pm2egq2BoNTwuUdTcEQAisBG26QjME3Z XPVpmqyJ/xhUsgbc8qXZ4dc/LCtZjU1JZxa1C6Mk+c7OKS/bElQPZViSJO+OHuuX5uLnIu3jj2e OamzmPtHf9gKKg8dKQ8hlMtOqVG/uJ9fTdycTQGx+b3cbEC6GBxAPTVEduwBmao3mG0O/Eupza5 JhFULxaR4rSHv27TVySPPyKbzSauDcchfmDV5m2i/sFfV3A2kttowvkXX68AOILC2KfTLDBKOmB qFRCH0D43tRpePd3ZceSDoijMJmQKJZU2dyTaaC5FNcPWXO2OD3EydIzl+Mdbv5Z5/8oBHQj+zv 48p3fh1iTIKRg87OPyjW1mpFSB5dWKUmdJfHGDIdyvwe9Fc9IHGB72MNdE3/nPt19Fu96tWDduM q2I7znoGx2sBYbq3npZ+2TiCA== X-Received: by 2002:a17:903:468d:b0:2cf:afe8:b722 with SMTP id d9443c01a7336-2d601ea52fbmr64639885ad.11.1787133247576; Wed, 19 Aug 2026 02:54:07 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.54.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:54:07 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 17/17] mm/hugetlb: localize HUGE_BOOTMEM_ZONES_VALID Date: Wed, 19 Aug 2026 17:51:39 +0800 Message-ID: <20260819095140.17252-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HUGE_BOOTMEM_ZONES_VALID is only used by the huge_bootmem_page flag handling in mm/hugetlb.c. Keep the definition next to that private data structure instead of exposing it through the public hugetlb header. No functional change is intended. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) --- v2: - Collect Acked-by from Mike Rapoport --- include/linux/hugetlb.h | 2 -- mm/hugetlb.c | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 255a258f11d1..900c95e346b2 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -675,8 +675,6 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; =20 -#define HUGE_BOOTMEM_ZONES_VALID BIT(0) - int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *= list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long en= d_pfn); void wait_for_freed_hugetlb_folios(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b2b80284c4ce..3116e323fe31 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -55,6 +55,8 @@ #include "sparse.h" #include =20 +#define HUGE_BOOTMEM_ZONES_VALID BIT(0) + struct huge_bootmem_page { struct list_head list; struct hstate *hstate; --=20 2.54.0