From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52334371887 for ; Fri, 11 Sep 2026 05:03:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103014; cv=none; b=nlzGpG4DxLpKBv51bcjRWuPCbsPFU4pZRsUvSyYopFZhLenxFRNGAlohACgkvK0wwixs728Wq7jO+k6So8FvwrvgZerkZcQop4yNDaRNl5R4yVIj7mHMw4jWfCWdzVA1JKRjOtFE3o8atS5O4b25+ky98bzlBpIR9Ov4itV8gjc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103014; c=relaxed/simple; bh=bVtHGPuP9MfV6o4kuMI0O0hiuhqWujHd+5z3+uuKs1M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AcB7/BqVIRyCX2V2K4obF/XRKmcW0vn4hEtD2Eturk8746zWx2mtTVPxwB0XGCOZ7d7JDSRFDeD6vTn1zO2BJzjEjx3/2gkp3M5gq8IPlKuLlPeX8mWlZ2qOWDGePZxr/gxarxcRYoBYs/6DPfNnNc+vA+gSqd+JnZkMpKv/czg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=cd2hWQp6; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="cd2hWQp6" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2dd020a2e44so4692925ad.1 for ; Thu, 10 Sep 2026 22:03:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103012; x=1789707812; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EYLeiWTXnovMqWiY32MdZegt5FI43o4EiONe9mBHAYI=; b=cd2hWQp6iPN25inImnxnk2vcda6XyTukOu5BXf0v/bG5u0E1omeDFKgt24SpDtEaRP t2R36rz3E9slWOADT57/73+6KY8WPSjRDHYZ+I9OSctCkxREM9ASPz986IvkyVlUo5mV Pt0CtH/bYTGjVUfuK6PfoPpHhhMPrJK37rbs1ZAvF6mgxDCHDSkzpo4oSmzrfWUoDMkd AfUYbq6yOWnh/6PvG9+Qg7NkheIDrJCkiryoY9BWGqgzJNmk4cKtxTebCBw8QKlYSkJE lnFIvXkLfyiv5mBSVpCaPxyFyEpqnYL/ZzBlnKnuMT7QEo806XEihECHjK26VjVTevmO 0dVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103012; x=1789707812; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=EYLeiWTXnovMqWiY32MdZegt5FI43o4EiONe9mBHAYI=; b=kvW+cT1IqsguReK6m+XdmqbqDZ36mY5Xvnz4jr0f5MFqQVasa8BNOl0F36sSmzvw7q Bcuuu8Mi9L3/tRrCZDSjwbuJ0Hapc5rjhaNsawjNp6cPlFTvGV0Fxotium0gTWGnLV6b da3mAHQF2BU4VWeAWtYDsTbbLac3BPLH8p5xVUjjbDGV9/3ye0PPlCR9ODBZRG4Ffmoz DhHLaSvltb5mRR/pCBOkWGDejwySg50UQ8FtG26MFIcyLe3u3bGE2xmuFQrVwnqqAULy WJKVyi8ZZl92jrsrzhio3iAUmGdZJrNNQtbsmSECL3XjfxJxjVFy8hPLhbDq20WndYlq hz4g== X-Forwarded-Encrypted: i=1; AKwUvByEbw6aWLwH/cI5Iu3wmb04SMx2td771enhzAshFDXnENXFQOf8AVKwYMucwk8wBKDWSUP7ws9ELEkkZ64=@vger.kernel.org X-Gm-Message-State: AFuF++mU5ZewD2co5nQNKcntb4WQmnHDLd83NF3g+NbpB7PgFE1RY7/z wvvPtUL+kLaxUmSDYFQYKBrkUfUYWkvhwazgqNJA9EUgMiQ59hz24cpdUVNJTdDZegc= X-Gm-Gg: AYBFou2gEPRgagRwaxfB8W7qYHWm5y5Q0WK8OT+Rr/eZVcZX3E7eKoqGepsSNU8P7r0 A/draAyIi4TZTS5yEOmU+f7AAovHOTqNslgGPqtmrYLlI1dYEARRKXOQFLyqH0ThUIow55w4p+h 2clJTSZvQFa+WN1ei2Xf2u+4cvsHHrZ7pG4kUYsgSw1nlS6mclcpZHTfH5UxTTi2K0u5lRtUBtv lWk3CcCZViV6rALKOhfFZZkjOUI7auZrYsPd/cClB1eZ7dcYufMPXNb5c0eY64Lmvr+UmCPcIhW FuCTI7xcu88x0Dof5vtJnjISiYhNpuTYCPvb/HUU1FluVFfP9a1iiUMvS5VQ79eUqS/LEfDZz0Z WGqLPj1+fDWGZNWz7hkVdRFYQ8OpKQDl0hne8o/9MDqXX3B8Q/qkhzGyOoqxT/P4JjIucWPwiVj scPzodrgy2nk9JQbqvGaRwC4hn4sngww7rNXLaH4+9bEuKy//n/m/jw0xAqC9kgRRVlxjWZjQ9e 4spU2SS3k+tpAD9qWOJISJ4diLtp5tU5Rrq X-Received: by 2002:a17:90b:278c:b0:398:e6b6:acc2 with SMTP id 98e67ed59e1d1-39d9c1b4d9amr3909895a91.12.1789103011605; Thu, 10 Sep 2026 22:03:31 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.03.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:03:31 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 01/11] mm/sparse-vmemmap: introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION Date: Fri, 11 Sep 2026 13:02:18 +0800 Message-ID: <20260911050228.58884-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The section-based vmemmap optimization infrastructure is still guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, but it also can be used by device DAX. Introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION as a common config for the shared infrastructure. Select the new option from HUGETLB_PAGE_OPTIMIZE_VMEMMAP and from DEV_DAX when the architecture opts in to DAX vmemmap optimization, and use it to guard the generic sparse-vmemmap state and helpers. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) --- v2: - Fix SPARSEMEM_VMEMMAP_OPTIMIZATION being selected without SPARSEMEM_VMEMM= AP reported by Sashiko. - Add an explicit DEV_DAX dependency on ZONE_DEVICE - Collect Acked-by from Qi Zheng --- arch/x86/entry/vdso/vdso32/fake_32bit_build.h | 2 +- drivers/dax/Kconfig | 2 ++ fs/Kconfig | 1 + include/linux/mm.h | 3 +++ include/linux/mmzone.h | 13 +++++++------ include/linux/page-flags.h | 5 ++--- mm/Kconfig | 4 ++++ mm/sparse.h | 4 ++-- 8 files changed, 22 insertions(+), 12 deletions(-) diff --git a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h b/arch/x86/entry= /vdso/vdso32/fake_32bit_build.h index bc3e549795c3..5f8424eade2b 100644 --- a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h +++ b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h @@ -11,7 +11,7 @@ #undef CONFIG_PGTABLE_LEVELS #undef CONFIG_ILLEGAL_POINTER_VALUE #undef CONFIG_SPARSEMEM_VMEMMAP -#undef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#undef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION #undef CONFIG_NR_CPUS #undef CONFIG_PARAVIRT_XXL =20 diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index 602f9a0839a9..6250954b0fa7 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -8,6 +8,8 @@ if DAX config DEV_DAX tristate "Device DAX: direct access mapping device" depends on TRANSPARENT_HUGEPAGE + depends on ZONE_DEVICE + select SPARSEMEM_VMEMMAP_OPTIMIZATION if ARCH_WANT_OPTIMIZE_DAX_VMEMMAP help Support raw access to differentiated (persistence, bandwidth, latency...) memory via an mmap(2) capable character diff --git a/fs/Kconfig b/fs/Kconfig index d1c210c6508f..9b32ce79cc80 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,6 +278,7 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP + select SPARSEMEM_VMEMMAP_OPTIMIZATION =20 config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mm.h b/include/linux/mm.h index c49ef99b4413..a2ebe87e7654 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5175,6 +5175,9 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, unsigned long nr_pages; unsigned long nr_vmemmap_pages; =20 + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) + return false; + if (!pgmap || !is_power_of_2(sizeof(struct page))) return false; =20 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index acd94cecc0d3..97511f651ebc 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -102,9 +102,9 @@ * * HVO which is only active if the size of struct page is a power of 2. */ -#define MAX_FOLIO_VMEMMAP_ALIGN \ - (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && \ - is_power_of_2(sizeof(struct page)) ? \ +#define MAX_FOLIO_VMEMMAP_ALIGN \ + (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION) && \ + is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 /* The number of retained vmemmap pages with HVO enabled. */ @@ -116,7 +116,8 @@ #define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1) #define VMEMMAP_OPTIMIZATION_NR_ORDERS \ - (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS = : 0) + ((__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 && \ + IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) ? __VMEMMAP_OPTIMIZA= TION_NR_ORDERS : 0) =20 enum migratetype { MIGRATE_UNMOVABLE, @@ -1155,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; @@ -2019,7 +2020,7 @@ struct mem_section { unsigned long section_mem_map; =20 struct mem_section_usage *usage; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION /* * Normally, sections hold regular (order-0) pages. However, for * sections with HVO enabled, this tracks the compound page order diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 86dd0470da11..462e89e05548 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -208,14 +208,13 @@ enum pageflags { static __always_inline bool compound_info_has_mask(void) { /* - * Limit mask usage to HugeTLB vmemmap optimization (HVO) where it - * makes a difference. + * Limit mask usage to HVO where it makes a difference. * * The approach with mask would work in the wider set of conditions, * but it requires validating that struct pages are naturally aligned * for all orders up to the MAX_FOLIO_ORDER, which can be tricky. */ - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) return false; =20 return is_power_of_2(sizeof(struct page)); diff --git a/mm/Kconfig b/mm/Kconfig index bc7befafb47b..c180d40cd671 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -461,6 +461,10 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. =20 +config SPARSEMEM_VMEMMAP_OPTIMIZATION + bool + depends on SPARSEMEM_VMEMMAP + # # Select this config option from the architecture Kconfig, if it is prefer= red # to enable the feature of HugeTLB/dev_dax vmemmap optimization. diff --git a/mm/sparse.h b/mm/sparse.h index d3a71ef4fad0..3151d4db7575 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -10,7 +10,7 @@ =20 #include =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION static inline unsigned int section_compound_order(const struct mem_section= *section) { return section->compound_page_order; @@ -75,7 +75,7 @@ static inline bool vmemmap_optimizable_pfn(unsigned long = pfn) =20 static inline bool vmemmap_optimizable_order(unsigned int order) { - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) return false; =20 if (!is_power_of_2(sizeof(struct page))) --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A199351C11 for ; Fri, 11 Sep 2026 05:03:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103018; cv=none; b=DodBS3p9GS8tf7IXOxVkCYPrw0rzB6wct7C+WTUijd/XY21k4KGK7HH75VlVWLWP6vKvMusEEn0uEyA+LuiKv77Zh/n1e0EztkvtzRq7vk2qEEkS2KPhG8I5fW+XHUUxShMWeChX0kzHPG7gxaVeUIxLW5rQk4jqCW0NIzXOvFs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103018; c=relaxed/simple; bh=fil1wyB3n2qakOtRPZGMWnUUGnhepk7poZ7+sHybDhs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FZBHAQKIzymdnFe1uucig6DCh8/GDbekynAEoTYxvj1EzPtUaUyTOJ7MhC9TNxhpXbhyogMmy10z1Xl76aWwLLK8mF2EzA0jMLM/wISpTDRTa5mPeHBOV4Xxz8nv/wjQ1OmyfvdwH1gHJ2ORfpC/XZ1Jj0dMo/afBbosM9/3+zU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MABlb6Wm; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MABlb6Wm" Received: by mail-pj2-f12.google.com with SMTP id 98e67ed59e1d1-396ccafb74fso504354a91.3 for ; Thu, 10 Sep 2026 22:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103016; x=1789707816; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6bR1GMYxjFGrp68HZ8kexJBvgYVBJednZ0YGt4vUS+s=; b=MABlb6WmqgWHq8e1CKuMVOfOGwIqM7Y0uhz3rx/ZUqGvnaymUVG/EB6uwUkC1kTWlR 67DsjyIEYpnvfFnRutlcZ3W57kMVaHpxAB3fS8PLfpOPdJ8I67Zhjl7D/WocCcLSxdN1 zDt0H6MwRHVD5SodswUIWi41stFDSpCuYWrQpVfTfImFO4ySAyeg8MKEYlMBTJuWYJNa 5lKEx+1rxeSiargrHeBAn2+v4dymxXi/2VvuzLehtdewufz22ssofVP2qnGQjHW068N5 w3UGY66jPf6CxRCcLG22vovCi4WbxBgCUn5/p5mq2qK6Qlm6tY/1wnzLjT+51lOkmzdS P4lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103016; x=1789707816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6bR1GMYxjFGrp68HZ8kexJBvgYVBJednZ0YGt4vUS+s=; b=jpGZYp0uKg1ZFlsCMXduYWNePWjPQSx4OmsKf7n7N6C1ct4h6vP6FyKzNtVGIXQQER 3jROfELe86I364mhF6BsdFpJ6PK6/FX3bSHnBp5ETVCheHDdguhOAhsLZWoqWSGJObws Eme3UVmaHQMqlQBVrd9kQcJyLbIGnQmmRMOeGGYo6gzzGFSHnT98BohevLdr6tjNOz8L rZU9aC3FruaLTzxn3cDE+Xzi4kSt4RPUrqrpiTtblbXoz6sJnPMJ27aJsBVhMb7zHSYD idQ9Lwa/+U0uf3VAb4f9EpDKuLTBJFvMFi51I3TYkstG+mGwYdXGcUyVrI2wcbA4MxvD OqSg== X-Forwarded-Encrypted: i=1; AKwUvBxSVnKSkqWq/1NiUEEbIx62+c9ofAGTCXiz/Zax+fc119GtfM8iurkZYatBOplp+r7PKn1zjKyu+ihgRwk=@vger.kernel.org X-Gm-Message-State: AFuF++nE9rBDRvGC/cdxw9EfO2jhreKEzGxLuwjW0rBB2eef8eQETcwL JznA09ntAbOUsgxlnGVTAxpnHv/8m9Q5v6sIaQZhly6KH+nP60cKeL+dlUFQnRjmTWw= X-Gm-Gg: AYBFou2zxsXHPJcEMp0ijHkdZ4TB/oCc0gY/4vZWLcyemN+cVOu0tp60CzpZuRs8WJO 43Mdm0PERo5N6N+N/W7adq6Zl1vp8jBkYCzWnG62m6IE5K2f+XPkvKoCy3Md6f6mQkayepNIXqD t1ZZ07rR5y2x/5tnxGdu3vO572UkXsztvVvgKR2vZUQ8ahtQV4m+f1HOWp8mP9tem/ZsP9nSIjS RLSNNDQspkBb3if1TJwMZ8+Iogx9J6ct6BbLhJUQuZ/xkBJ4YaX9g8Vq/hivG0OBu3zThghi9ZF v/VIlQxMPU1u2WKJB7LFXVzzNd4V/uEiEmO3nQciKi6slXVK6n0EJA9+e+p4tIcmuWWL0x7rgGd yl4y4IThvqK9qUtda4Zr0S2a7LgWvSlGHEkl7k26X4H+RbkXaGObjwiCa3Aos4EZOtLUJ+OOnmM NGd7kBt0AOLgAlCib5Q63WMKvz9tXrKCU2Z83nfdi89Yc49/QYgwlxR6seMCHtgrGOeoc8+cMY6 Qght5NbVx8yFnkSZiqJIkZAWTvceWSRJg9zvw== X-Received: by 2002:a17:90b:48c7:b0:398:9be8:ea66 with SMTP id 98e67ed59e1d1-39d9c226c4dmr4442740a91.19.1789103016270; Thu, 10 Sep 2026 22:03:36 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.03.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:03:35 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 02/11] mm/sparse-vmemmap: factor out shared vmemmap tail page allocation Date: Fri, 11 Sep 2026 13:02:19 +0800 Message-ID: <20260911050228.58884-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB and sparse-vmemmap each have their own helper to allocate the shared vmemmap tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper. It allocates the page through vmemmap_alloc_block(), and uses cmpxchg() to install the per-zone shared page. Expose zone->vmemmap_tails under CONFIG_SPARSEMEM_VMEMMAP to match the shared helper's build condition. This avoids a !CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION stub; when optimization is disabled, the array has no entries and the compiler folds away the unused paths, so no storage or runtime overhead is added. This removes duplicate allocation logic while still handling both the early boot and runtime paths through the same helper. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) --- v2: - Collect Acked-by from Qi Zheng --- include/linux/mmzone.h | 2 +- mm/hugetlb_vmemmap.c | 28 +--------------- mm/sparse-vmemmap.c | 74 +++++++++++++++++------------------------- mm/sparse.h | 1 + 4 files changed, 33 insertions(+), 72 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 97511f651ebc..1a18c1c40c8c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1156,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +#ifdef CONFIG_SPARSEMEM_VMEMMAP struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index f977d0a7e002..5ddf06b83c96 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -493,32 +493,6 @@ static bool vmemmap_should_optimize_folio(const struct= hstate *h, struct folio * return true; } =20 -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - struct page *tail, *p; - int node =3D zone_to_nid(zone); - - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail =3D alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p =3D page_to_virt(tail); - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -535,7 +509,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struc= t hstate *h, return ret; =20 nid =3D folio_nid(folio); - vmemmap_tail =3D vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail =3D vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index f22d815d7af0..7388a5b5cce3 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -42,27 +42,13 @@ #include "mm_init.h" #include "sparse.h" =20 -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask =3D GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order =3D get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; =20 page =3D alloc_pages_node(node, gfp_mask, order); @@ -76,8 +62,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, = int node) } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } =20 static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -184,39 +169,40 @@ static void * __meminit vmemmap_alloc_block_zero(unsi= gned long size, int node) return p; } =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zon= e *zone) { - struct page *p, *tail; - unsigned int idx; - int node =3D zone_to_nid(zone); + void *addr; + struct page *page; + const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; =20 - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >=3D ARRAY_SIZE(zone->vmemmap_tails))) return NULL; =20 - idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail =3D zone->vmemmap_tails[idx]; - if (tail) - return tail; - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + if (likely(page)) + return page; + + addr =3D vmemmap_alloc_block(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); =20 - tail =3D virt_to_page(p); - zone->vmemmap_tails[idx] =3D tail; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page =3D (struct page *)addr + i; + mm_zero_struct_page(page); + init_compound_tail(page, NULL, order, zone); + } =20 - return tail; -} -#else -static inline struct page *vmemmap_get_tail(unsigned int order, struct zon= e *zone) -{ - return NULL; + page =3D virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) !=3D NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(addr, PAGE_SIZE); + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + } + + return page; } -#endif =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct vmem_altmap *altmap) @@ -229,7 +215,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long = pfn, int node, return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); - page =3D vmemmap_get_tail(order, zone); + page =3D vmemmap_shared_tail_page(order, zone); if (!page) return NULL; =20 diff --git a/mm/sparse.h b/mm/sparse.h index 3151d4db7575..a250bea4088e 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -142,6 +142,7 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 137A33101B2 for ; Fri, 11 Sep 2026 05:03:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103023; cv=none; b=D4eG1i3T9Wt8sXdPANzpC3269LvIW6ZorPs2buYw5ZOUwFW0Hr+vfz4IJEl8H0gl/iO6r7LLGE412Ml/wj1PrPHmZVYOmCCMMKjS3zm1xvf9zassQrT+4ziHSzlHtSq7ik828VgTlCiN3OVG8m/LextT37j/he+j8AVwStaXPLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103023; c=relaxed/simple; bh=o93Z9Y+DL/D2s+9gaMivcy0VdYP+V7A91VZF+QexHA8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jgqmZz8Iq7QJfIowuVwR5Xu8PyvExhSilfDuP756IKNRAaDg3kyyxOrZDRs9BaUCMYtakTUIKFGQ4ErcVlDwbzDmB3FO/dEziZadqrgVQ0Q3rgZiBLvmeDD+5Tdkgf+TybRV2UGvp8Y+e0FFmAGAlZk8ihewxvy8TlEtjjIcncI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=AURoCfLQ; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="AURoCfLQ" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-38ec1402b05so454677a91.2 for ; Thu, 10 Sep 2026 22:03:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103021; x=1789707821; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=otJyn0+LhJb8oZRc2DA2ivaMEPoP7qIjxigOuUE5KLU=; b=AURoCfLQJ8uQDhpBI1nyCRQ3l5kb5f6Osmb0qnvILyHTajkmIDE1x4Ge5hm6JAqP68 /wjJSG69CWv6JTEt8dw/GV91tIeBKamMnj/4XlG40k0PvDs2jigZd87k6Kle1bjzBr3B 9C9wtWPqmXSwXIJ45r2UpiHvXsDw9oCc7u9p9tWVDlQxUVcImI3ruCZ/h/n0CCuzg+05 l4Gnv32oOHt4mQt8fz+hkmhdf4cFMjJ0y6VRVzHk1y57X99AUw0heWibv7F1cJyk16pC /8OSW2974Beg++5SmUZphB/MtYOGD9bAFOQoShSzm4UMSGePMWdhcIzj6ciD8q1JQqN4 Lwlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103021; x=1789707821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=otJyn0+LhJb8oZRc2DA2ivaMEPoP7qIjxigOuUE5KLU=; b=IG90Z43oAZFAj5ZUZUfP3nJUPzuxh6mbqsrk7caq6pNKKSLyucnMxjTX7OEYgFEfst 7jsG2aYOmai+om7sGEIaH+xjZZVcDmt5YPCeaBk99zFq5FRa9ix/1O3AjOIdCXn/2LAI DtYrCC/vrJfgzDbkP6EDcy5Xfzd5uFDAWDIBOgB18rkVBDjdz/6C1JCXcD+f8wmjic/2 KkzjmqUEzNuuKaxNaZcWE9W5mzVLrD7nVJOuGhm3gfn5Czl5gOHE6XFGvzKGTnVNlXnQ ejpvUvchrv+ppRyb0tMd/p92pwlqDd/W4hzqnM5ovlqWJ5HNjOEdujKh2VImoJbEq7cT o1cg== X-Forwarded-Encrypted: i=1; AKwUvBzaHJjb/jBYYQv7kVOyswsIhfbzTbDRCo8z/3aHT2xg1jCg6UuEYieKtZEV/B2fU5NEPgpA3RqwF8rD0W0=@vger.kernel.org X-Gm-Message-State: AFuF++mucHFeE+8vcOdiQQOY307H/h7Hpt2EQZtvaPVCwsOhkwbIThRK tTf3j/DLAb+0xCC7Dwx8eSz0XRn/uvxFkIH0o+/TCiZmmXaFCIGJ9KIyquhPxo76lVM= X-Gm-Gg: AYBFou0LHszCv372xNzRTD/57meC05gmKNrXtRxI9D5dY2bVfaFnJ/s4vDb6DKIExwS 4DWHld2dh/Gv2a0mCvyWIQaZiLDiZT2AFSSJZPwTmOFo6IP/6eA3+feXMnunqMymvWKNIQx/6ef YNG+QAadLf6nZh1pDzHMq64m0FBK7IVLCWXiddIhB/3Bmn9vlJK8APAZ0LkcSTWSD6cCy8WrsNy tMUVcgIaJVpN4FmYLLbAPQU1Jw7AIKVEk7hNGxmSCaLtU62Han4gAV4U9M3EywzgEv3cW4a8lB3 +imOAFUWZPcaoE+SmpmNmZrDBu8faMSHQI3LIGrktxkSiAmqkR/MoYI4swxTLRiZNNRLzWvQ9Q1 05MviPPEFKbyRwQ2uCpPeAVdzhNkOjKyMuoOX39uibrb0EOuIGCr0iWklAIZKP1Sh0nyN3QrgL2 o7vw2K+U733+t7D3KDBpG1qIlJT9EmhdeSPW9GpvYHV6ctbJvBZhOMeh2UDilPj3CoECGlNdHVV Yuucaf3XMyAvaq0Eyz+zomU4PFItN8ZeGyI8Q== X-Received: by 2002:a17:90b:588d:b0:399:1f8b:d255 with SMTP id 98e67ed59e1d1-39d9bbc823fmr4102685a91.5.1789103021399; Thu, 10 Sep 2026 22:03:41 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.03.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:03:41 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 03/11] mm/sparse-vmemmap: open-code init_compound_tail() Date: Fri, 11 Sep 2026 13:02:20 +0800 Message-ID: <20260911050228.58884-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" init_compound_tail() is only used by vmemmap_shared_tail_page(), where the shared tail page setup intentionally passes NULL as the compound head. Keeping this helper in mm/internal.h exposes that special case to the rest of the MM code and can make the NULL head argument look generally valid. Open-code the initialization at the only call site so the special-case use stays local to sparse vmemmap optimization. No functional change intended. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: David Hildenbrand (Arm) Acked-by: Mike Rapoport (Microsoft) --- v3: - Collect Acked-by from David Hildenbrand v2: - Collect Acked-by from Qi Zheng --- mm/internal.h | 9 --------- mm/sparse-vmemmap.c | 5 ++++- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index da14c56fb24e..0dca33db068f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -786,15 +786,6 @@ static inline void prep_compound_tail(struct page *tai= l, VM_WARN_ON_ONCE(tail->private); } =20 -static inline void init_compound_tail(struct page *tail, - const struct page *head, unsigned int order, struct zone *zone) -{ - atomic_set(&tail->_mapcount, -1); - set_page_node(tail, zone_to_nid(zone)); - set_page_zone(tail, zone_idx(zone)); - prep_compound_tail(tail, head, order); -} - #if defined CONFIG_COMPACTION || defined CONFIG_CMA =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 7388a5b5cce3..861e09b2b096 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -189,7 +189,10 @@ struct page __ref *vmemmap_shared_tail_page(unsigned i= nt order, struct zone *zon for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { page =3D (struct page *)addr + i; mm_zero_struct_page(page); - init_compound_tail(page, NULL, order, zone); + atomic_set(&page->_mapcount, -1); + set_page_node(page, zone_to_nid(zone)); + set_page_zone(page, zone_idx(zone)); + prep_compound_tail(page, NULL, order); } =20 page =3D virt_to_page(addr); --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pz2-f12.google.com (mail-pz2-f12.google.com [74.125.228.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F9843A0E8E for ; Fri, 11 Sep 2026 05:03:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.228.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103028; cv=none; b=vCVNoOkOjkVb0gAH4pY2NGkVXYV3rRfD6FNCSo5Hz6F7q4RkohY9L2qWNVC1Tw0mBR9AN2UioR6OMotJ4kzAGo3JcGnww04tdce82eRMVhJbgQWzKP0I0P3k0xpVqlbwoZYaUCzhgh0hGqt6Ba3G3+MPg8H1Di+MN9q1w9CtOeI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103028; c=relaxed/simple; bh=4C/H0OXURhGuKf+braJ+uKcmMn4yaEgFfA4J4Js2eLc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ahJ7xLcHbcNScnwxafKT8MZzVkQ8GO+0txUbI7BKv5SZXE3T6OIBWW2PpgpSbkwLndK4IeDSg5BQfV+NysDDxA0AKYsv/5nzRe0kBackaIJ+Yei4nZ+GZDADR3CyCr5v2jjNUmia8NwItjj1HGcv/4bYzdqrqhgKxF9EN7ccI6o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=EId8OT3s; arc=none smtp.client-ip=74.125.228.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="EId8OT3s" Received: by mail-pz2-f12.google.com with SMTP id 41be03b00d2f7-cc4ae61629fso407472a12.2 for ; Thu, 10 Sep 2026 22:03:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103026; x=1789707826; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=jrSUwqa7xtL2ZBs1WlB04Z11tmz781BnHoaaNOznkvI=; b=EId8OT3ssGYVFUtSPjUI8PMyYQNJc793KFLxt4iaekVT4POWEWhqq4H/UM7GapqIAR ADTGlgJlDbChZb5eOAOS/DedQe/Xy4PJOLtT0A5TnzIHG3onKvtmv0glLdeTqiVqdFAG vWmv/0Ui3UCcVmp6nyqoFjqpHYBSpOJhzmwNKBBT9kQ0QckiVw135lxlIQuX3Db3ZWTu wAlKXU5IBrpMoKhn8PuS2tCPY7PvfyTTOke1N8803utTO6SPvimdzW8pfg7onWyLsOS7 7w9wvXH7aAGNuCKrNrupEIoeUdgCC/v+7GXAPeCQ7YaG/KUrdCQGQLNqx2lCuR5BqUcm X4Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103026; x=1789707826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=jrSUwqa7xtL2ZBs1WlB04Z11tmz781BnHoaaNOznkvI=; b=bI7FqpqAQfgRb90qxtoMnZgn6VDr67lKeIkBeLcoykbJMDsnLYvINOJiYU4v7+IbzB jXoacOxZy4YNg/h/wrAiZN0yrKevfmReE0y/LRwWGFXo0GuB372ZjkDTxs0qH+Ci4i73 XpFH9VGZL+5VTDvhqnLQpsc6YShQdYebFSLCVHvrGlVeStjAx5xydntTLmszSirYT8Uq NqQujs8GFFoTt3cds6LtlBlyKbmOaVaoI9955hNJjaLtxwHvMBKA7IEJxjxRVSHJcDt/ wW0eIYNTRb2L1XTo+onT2okf3nH9hDueiZvTTUcSmS0lDiwrTZjm+8AETyjU9oyVGHlP FUCA== X-Forwarded-Encrypted: i=1; AKwUvBwi1i0EcgA3nbskWgL/nDo88zm8Y3VrbV1OUwHnqAWqJK0IrCnjfCPXZhEAHwtaOb9hQQSk4BBwRoGtMqs=@vger.kernel.org X-Gm-Message-State: AFuF++lTGqFviFeUhIZ3tc5A9zke4ZmZSPbRtM7cG7mhM+gbSOvSP3Ir AKBbilSYjuvjZlYEebAgXIkImoNfQmrGnWKh+BivOu01n+6AtSdJRdXd7r41ILvwWEo= X-Gm-Gg: AYBFou08o/6b2j5H8FZB/2LS2NdqjBgKEJJaiJZWZXDt6rn4JdB+x4I8/gpK8ccFMav aCHPIpia+iRqxpcOrS/8bNP6UlrLBlmBSOnIgAvOOAGIymZRBIZo28HyFpZaFSp00p5CHdkm/0M warRB9tLUCYd4hxoFQ0qearxLFdMPX8aprTVb6n9xXvl0b9BTSCQnopEuP13CWzvI5uUzCNROqb kqPtFpuZYHFhYhaNFciBzSFwJbkKCzsPqenX7JKE42e4FoV/oFgL2avkaJOKbX+MSRPu/nZ+J7m 1Q6uOAgmMxv/jqVxPAPJYBIykxm7iRObMmi3LQII9AqtG8cfoRVZlXmQ91KinA6cWfgLLq4RSNL Q4uOe0NmyaZ5XyGHVp+AvwqjuWHlpL73HLiov1UvBQ9Bo3KdC5rLJtFX4nY3hQUzYqBx4udS2C4 c2Px4SejyUuFD/R16jHE3WhxX/gqHs56JF6EQbW7lBXcBRV1vFM7p7nZsvvvxxxMl7xEkkHqXoP Oy1JzueyKfLBYLrtsYXSbnSufmZa9S/MhsP X-Received: by 2002:a17:90b:390f:b0:38e:bbf1:de3f with SMTP id 98e67ed59e1d1-39d9c0b7047mr3845524a91.12.1789103026376; Thu, 10 Sep 2026 22:03:46 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.03.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:03:46 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 04/11] mm/sparse-vmemmap: prepare DAX vmemmap population for compound page orders Date: Fri, 11 Sep 2026 13:02:21 +0800 Message-ID: <20260911050228.58884-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX still uses vmemmap_populate_compound_pages() to populate its compound-page vmemmap mappings. That helper allocates the head and first tail vmemmap pages explicitly, then reuses the first tail page for the remaining tail page mappings. Device DAX is being moved to the section-based vmemmap optimization infrastructure, but it cannot switch to the generic section-based population path yet. Once a later patch records the DAX compound page order in section metadata, DAX head and first-tail PFNs can look optimizable to the generic helpers as well. Add a DAX-specific population flag for this transition. It keeps DAX head/first-tail allocations on the normal vmemmap allocation path, while preserving the existing page reference for reused DAX tail mappings. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Update the subject and commit message to use compound page order terminology v2: - Collect Acked-by from Qi Zheng --- mm/sparse-vmemmap.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 861e09b2b096..aa89c16f7fc7 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -35,8 +35,8 @@ /* * Flags for vmemmap_populate_range and friends. */ -/* Get a ref on the head page struct page, for ZONE_DEVICE compound pages = */ -#define VMEMMAP_POPULATE_PAGEREF 0x0001 +/* Vmemmap population for ZONE_DEVICE compound pages */ +#define VMEMMAP_POPULATE_DAX 0x0001 =20 #include "internal.h" #include "mm_init.h" @@ -208,13 +208,17 @@ struct page __ref *vmemmap_shared_tail_page(unsigned = int order, struct zone *zon } =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, unsigned long flags) { struct zone *zone; struct page *page; const unsigned int order =3D pfn_to_section_compound_order(pfn); =20 - if (!vmemmap_optimizable_pfn(pfn)) + /* + * Device DAX still relies on vmemmap_populate_compound_pages() for + * head/first-tail allocation and tail-page reuse. + */ + if (!vmemmap_optimizable_pfn(pfn) || flags & VMEMMAP_POPULATE_DAX) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); @@ -236,7 +240,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in pte_t entry; =20 if (ptpfn =3D=3D (unsigned long)-1) { - void *p =3D vmemmap_alloc_pte(pfn, node, altmap); + void *p =3D vmemmap_alloc_pte(pfn, node, altmap, flags); =20 if (!p) return NULL; @@ -251,7 +255,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in * and through vmemmap_populate_compound_pages() when * slab is available. */ - if (flags & VMEMMAP_POPULATE_PAGEREF) + if (flags & VMEMMAP_POPULATE_DAX) get_page(pfn_to_page(ptpfn)); } entry =3D pfn_pte(ptpfn, PAGE_KERNEL); @@ -511,6 +515,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, unsigned long size, addr; pte_t *pte; int rc; + unsigned long flags =3D VMEMMAP_POPULATE_DAX; =20 if (reuse_compound_section(start_pfn, pgmap)) { pte =3D compound_section_tail_page(start); @@ -522,8 +527,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, * with just tail struct pages. */ return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); } =20 size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); @@ -531,13 +535,13 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, unsigned long next, last =3D addr + size; =20 /* Populate the head page vmemmap page */ - pte =3D vmemmap_populate_address(addr, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(addr, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 /* Populate the tail pages vmemmap page */ next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 @@ -547,8 +551,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); if (rc) return -ENOMEM; } --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 034EA3101B2 for ; Fri, 11 Sep 2026 05:03:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103033; cv=none; b=c1PNrTGYcJEseqcKXGa3MpAWQFj809YgX97JN0INt2iZ9GgC1byyKg4U81bAT2q7mRQU2hiOkplNi9VsdXHUYtxlO4tTCD+sBYl3CRticrYNb+V2783hyk6VK/AaHhy92xudYB6EWkYpWGpeLqECmP1xRNpgKuo0pHVnFTdo4eM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103033; c=relaxed/simple; bh=q4sE4pbRykd1vhFCtA51ekVFkV6gQ95xL4RRztXC6GQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PGjbzTyW9hYlQflAzewAf1dpuJCuGLL6alTgl0Gaz0DmTV47a9YiD7ZCWUqt9fcZX5bWzR1w6obHLtAPFY5S/fQfFDAtBJs/Xf2prW0++d1A/dMXvo4JQL/471N/AUvwkZoKiQBRLRjvC0YS6fIR6h0dT4DD+SNjVkzP44g+EL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=X7rk64k0; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="X7rk64k0" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-cc4d03f95dfso3854a12.2 for ; Thu, 10 Sep 2026 22:03:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103031; x=1789707831; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6sM9zpfll+quDztLFFF0jfXeRCau4il9IJzrgeT16pM=; b=X7rk64k0BrCqji/e9/gqOIT/4LWlUxZI/XtJvgPah5bqpAebchlL+ey4lztPs5r7RL HB2yp1afNCL69uocoQFLPPpGT8kU7oUuhoV5KbH5WTX6hyg17LNhw/VTf3HfBWtGNQ91 6BrvQMRHrm6kSLPAmNIDpLVYq2o+m/q9lUixmH+aWJHxDJe/5I78kND/fM97LAwebpNx 2V2xANSVN1pC6i54FItRGqjV9O/OiTiYlEkswNCqze3o9seWh3pPvaFyptZDfhZ2Nkow 3dok0A5LSovEJuALUy0GMBsp2W0JexZAI1fmmQD649KkRa3SCjCfoADeYjJrV07MiaZI 1nHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103031; x=1789707831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6sM9zpfll+quDztLFFF0jfXeRCau4il9IJzrgeT16pM=; b=BjPLhjSayPpPN9t0oFAzrVpbSDdJhFzkWB5os2XgJYOlBmEHte26mC4XDWih059ug/ o5LjjyGwN+Lt5ZvJ6y43LQcc0MoeDywUuxDiheyJ78exIg6xnpIleNw+ZB0z1sCkpiwV fOGcbweN9CuWAzLZXj6wRwGpmRW4rjUxnkZlLQyjsNFcjlON0ydWvWxvD4Wq3FC6c6oy +kNIsE4MgT8hROt7UZnxh9JfZJhDGs4vK6Ia+uewVYJUToQHT947HeMpUomc1oFNuq0O EEVloryNVOVScgRnH6CJ5FfzA3pxeTMa1/Ej9CwLw40d1F0ngNY/vKKZCfzMajYFkMzw gU8Q== X-Forwarded-Encrypted: i=1; AKwUvBy0uvSRMMyfbRQh6wM3LtpSqhSm7PU7Zo4V4WE2t7Eyjw4yMqKl/FiW3TM3LNONsoJfshyyg9a+heQWSJQ=@vger.kernel.org X-Gm-Message-State: AFuF++mbEwILelHItia2DGB65ZwuO2kyu57JZmQXknMoqzOfnzJYVkv8 4QpouNIGo/QfC6PtIdRn7PN21RI87qhgPLIYRxSrcICVICCja44jFaqrfLpDEoSauuCiCgmODom yVe2H X-Gm-Gg: AYBFou0ay4eenuu3mwPPjhtrFw9GEa6lNzevW6F9oVV77M2bxQNajZopdwVGoy1W8hn YBjr2LCW5a86f/TRoK+/I/KHmAr50Ohueo1HknH3/zthZydBYbRsZPGozINBdE2eOn7c3YQN+tf XvYHkIe4zm5oCjlzehSjy120sFQOw6oUV9dbqzxAV+JNcbj3DwW81kCp7b8v2Lce3TSC3TqBR/H D8h0Vsx6e+soSMzRqR+hMalA103C+hhwL+g5mTu0K6S8+Zl8fZxHDDp2Dqqqn/WTR2gFa7qhDmn WA9kzrIrN5qqCWvT2KCmaRuQmzwOQXNlu6vbuj8NdckzrW6NhdUPZD+EetXMQ4yGyprTOcxzK1Y PgXuxA2EqbQFSo7b3fevAPXqmzYDCoO4DGb4hQQegSgLwKj5qSk8pIDNzqj7/Tr9t7dnTkU8MYt t0gPKpNR18iwZ/zADDbDQPWl1v5iRadlelzRZn+OzO8Y/lQ1qD35J4eq+UFiO7DNXwqT/vGvkIb xED2rX4gJGQfgtpLnDB/onDjn5Ixrta3WLd X-Received: by 2002:a17:90b:2b43:b0:398:a2a3:b631 with SMTP id 98e67ed59e1d1-39d9c21f8c9mr4085781a91.19.1789103031225; Thu, 10 Sep 2026 22:03:51 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.03.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:03:50 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 05/11] mm/sparse-vmemmap: set compound page order for device DAX Date: Fri, 11 Sep 2026 13:02:22 +0800 Message-ID: <20260911050228.58884-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX can use vmemmap optimization only when a full section is populated with a compound-page geometry. Record that geometry as the compound page order in section metadata before populating the section, so later vmemmap accounting and population decisions can use the section state directly. Clear the compound page order when the section becomes empty again. Also reject partial additions to a section that already has optimized vmemmap mappings. compound_nr_pages() determines how many struct pages to initialize with a section as the smallest granularity. A section therefore cannot safely mix optimized and ordinary vmemmap layouts. Partial additions continue to use ordinary vmemmap population, so they do not save vmemmap memory. Such additions are uncommon, and the lost saving is negligible. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Update the subject and commit message to use compound page order terminology - Use EOPNOTSUPP instead of ENOTSUPP v2: - Explain why optimized and ordinary layouts cannot share a section (suggested by Qi Zheng) - Collect Acked-by from Qi Zheng --- mm/mm_init.c | 15 +++++---------- mm/sparse-vmemmap.c | 16 ++++++++++++---- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 97e0158d2aca..efffa8609b85 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1049,16 +1049,11 @@ static void zone_device_page_init_from_template(str= uct page *page, * of an altmap. See vmemmap_populate_compound_pages(). */ static inline unsigned long compound_nr_pages(unsigned long pfn, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - /* - * If DAX memory is hot-plugged into an unoccupied subsection - * of an early section, the unoptimized boot memmap is reused. - * See section_activate(). - */ - if (early_section(__pfn_to_section(pfn)) || - !vmemmap_can_optimize(altmap, pgmap)) + const struct mem_section *ms =3D __pfn_to_section(pfn); + + if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); @@ -1144,7 +1139,7 @@ void __ref memmap_init_zone_device(struct zone *zone, memcpy(&template, page, sizeof(*page)); if (pfns_per_compound !=3D 1) memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); pfn +=3D pfns_per_compound; =20 /* Initialize the remaining head pages from template. */ @@ -1160,7 +1155,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; =20 memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); } =20 pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aa89c16f7fc7..02da1321197d 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -135,14 +135,14 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms =3D __pfn_to_section(pfn); - const int order =3D pgmap ? pgmap->vmemmap_shift : section_compound_order= (ms); + const int order =3D section_compound_order(ms); const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); =20 - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) + if (!section_vmemmap_optimizable(ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); =20 if (order < PFN_SECTION_SHIFT) { @@ -573,7 +573,7 @@ struct page * __meminit __populate_section_memmap(unsig= ned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; =20 - if (vmemmap_can_optimize(altmap, pgmap)) + if (pgmap && section_vmemmap_optimizable(__pfn_to_section(pfn))) r =3D vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r =3D vmemmap_populate(start, end, nid, altmap); @@ -792,8 +792,10 @@ static void section_deactivate(unsigned long pfn, unsi= gned long nr_pages, else if (memmap) free_map_bootmem(memmap); =20 - if (empty) + if (empty) { ms->section_mem_map =3D (unsigned long)NULL; + section_set_compound_order(ms, 0); + } } =20 static struct page * __meminit section_activate(int nid, unsigned long pfn, @@ -803,8 +805,13 @@ static struct page * __meminit section_activate(int ni= d, unsigned long pfn, struct mem_section *ms =3D __pfn_to_section(pfn); struct mem_section_usage *usage =3D NULL; struct page *memmap; + unsigned int order; int rc; =20 + order =3D vmemmap_can_optimize(altmap, pgmap) ? pgmap->vmemmap_shift : 0; + if (nr_pages < PAGES_PER_SECTION && section_compound_order(ms)) + return ERR_PTR(-EOPNOTSUPP); + if (!ms->usage) { usage =3D kzalloc(mem_section_usage_size(), GFP_KERNEL); if (!usage) @@ -830,6 +837,7 @@ static struct page * __meminit section_activate(int nid= , unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); =20 + section_set_compound_order_range(pfn, nr_pages, order); memmap =3D populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap, pgmap); --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1875A3101B2 for ; Fri, 11 Sep 2026 05:03:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103038; cv=none; b=hAXNRZIAcV0c1ml9SlZq3ny61yoDbm26GAjVdGjdwPQ/T2cpJDlnBY9Swn9Tfg1fITYgGByOTtTviUlDdcKaSWddkNeXYWjNclaws9ot8PHE+9jCXopxMWIz6wOp6ONH5/l1HFaNK+SVg88YyyygvhtJIbklpR2GvstAAeqB+cI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103038; c=relaxed/simple; bh=0llyyoMRgBuen3IFNQ8CcdCaoiuoaaDgEXEVe3ASyro=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=m49Y7p12zaoOpHsB35usFyTpGTbQsXcsfhJyJAo/Gq1lriaeyC1b96mgySa0jjBRq/YNB2RO+ZucNsitBpgD6cyvlDOvstlFDFJ5gDQooURCD+cUa0GMIAeZ9NDdRXXvHcJ0/zpw8Bt5NeiCHLdTO6BgQTBMvgWJHc+y4zEIBRQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lNggjDWR; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lNggjDWR" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-398e9698a70so556633a91.0 for ; Thu, 10 Sep 2026 22:03:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103036; x=1789707836; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4SoYJ7WMuU/pqlbOJzY7F/meqClQuD0NoJ+12TnNHPs=; b=lNggjDWR+N8v6zzi9sMsHr2ovH4lnFQb6hX405YNDT76vQ7bIl08YyhfnY65RQM9w2 EcDpMemkeo1hvQc8XDuhROUtJTYfAuChi7AYWOtBUfcZmsVb9iyASGlZqNudPJdhqdJ2 lYKY7EzIaM6d9oWz1VVWtNmvSBd198V6YVBMnPV4ANrpmTp/CY9iAX7VQ3CYGgmKSSvW f/rV6VitPJNaO27sszFQHs4j2Bc3vu8z1xArZbVGWqYNRB2s8kSRfLujf/VdGx18UPDs NIeH1UB3ZclHCZZA02wxGEpTr//bf7mHDt/PHFPePwQdKHWYgsHioRAKa/MHTnEdWeTV essA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103036; x=1789707836; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=4SoYJ7WMuU/pqlbOJzY7F/meqClQuD0NoJ+12TnNHPs=; b=bpUs5dYksLeT7CjotlCi1goPGZIGUrblj41G5Ytkq2DzIUx3Nt7nEi80BACfMVeEwY od09wqmtsJ2iLFyiaf6yAWU73/1mJfFNo2NIZP7T50a8dIqkGOlIx+FEfbKZkgkUU4Gi 7UIi81OOtAbE9GWlHeWOPgBiuF+HG8tt/nMRvlQyze6NtnIcyrZrKGNIZIKqCChTbESc +vGVbUFMkTmtTmpLC6XxOkPncJS1oYFGxtm4kN6d1WmVhkShyezkDmfNqfxi0oD2ah5w DRNtU9PihV95X2w56yxgcRzr+Z7d+MAbCpfOnoUl70MXSC6Dn1wwt61wMR8bRukvpOYF q+YA== X-Forwarded-Encrypted: i=1; AKwUvBwvFbPVsSyzbpn5rHxOkvvUonBvgDcOBttFGYxXHs++ED0sE+AZwl5kLgOdd9cnYbTa4zyEbSuSIrIh19A=@vger.kernel.org X-Gm-Message-State: AFuF++mZCjR/wLGJfVr4TTkU8EBFO8XFt65qfe13vG15w0OcDlfGj9PK oBo3cPWUfPCkOO96Ci6+gfRU1gp8PrItpy1RSuR6d95jRcEssfrb7s6tUOHhDQTNMQY= X-Gm-Gg: AYBFou24z2rcCcDCB/TBbfl+BgCmJRoOYsgOQnljkyR/DyTjCQ68Ur/0K9tZ72tS7k5 +p8qoI4XtGv3v/lWcQWO02kH9pP7x8qWYGwSV7YBIRiojsJX9jtHqA/lUR0PVnjNd3woSRCBTX8 O0j64TIu7qfEHGlL3NSE6jTmnX2grNWr2Bt4wL98NXVAta/ynW5W91DrHgAyawnNPIEno9UYq8N JyYuVTutw5+MXk7KYBzaIqCme7HPxo57eFC3rzWN+ASEmfY2XX8JJyLkaKNiPrQj2jcKHfeFiFM adWUEElSs6NLVVJah9sBSPI7FZEW11obyL2e730g0UlXvd+3N6ZfZZjqwHeySJGdmvRxGoP1Gad eP60yg6VRbUbT7MG+V2jlt5U0ejQ+TBcje3F4qi0ALz+dZmOczIPDhn9WOmbtho3U3B+y5Esx74 5fjYV52UpDuoITHaSz+MTUJzkSzGpo/JdcRurSdP4OrIkzlWcTBL1riySl6sPGVd/QfWs6fH9ar PQr9AkhVYVqzCU4nbjNhGQ1FTm0oWyzMSB8 X-Received: by 2002:a17:90b:2b50:b0:380:21b7:e727 with SMTP id 98e67ed59e1d1-39d9c1c199dmr3979201a91.14.1789103036192; Thu, 10 Sep 2026 22:03:56 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.03.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:03:55 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 06/11] mm/sparse-vmemmap: switch device DAX to shared tail vmemmap pages Date: Fri, 11 Sep 2026 13:02:23 +0800 Message-ID: <20260911050228.58884-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB vmemmap optimization now uses per-zone shared tail vmemmap pages. Device DAX has not been switched to that mechanism yet. Switch device DAX to vmemmap_shared_tail_page() as well. This aligns DAX with HugeTLB by using the common per-zone shared tail vmemmap page. The optimization is enabled only for DEV-DAX through pgmap->vmemmap_shift, which supplies the compound page order recorded in section metadata before vmemmap population. Unlike FS-DAX, DEV-DAX does not modify tail struct pages, so sharing them is safe. Since the shared tail page can now back ZONE_DEVICE vmemmap mappings, initialize its entries with PG_reserved for device zones. Also skip poisoning vmemmap-optimizable sections while their struct pages may be shared. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Move device_zone() after the definition of NODE_DATA() to fix non-NUMA builds. - Update the commit message to describe the compound page order stored in section metadata - Collect Acked-by from Qi Zheng v2: - Explain why sharing tail vmemmap pages is safe for DEV-DAX (suggested by Qi Zheng) --- include/linux/mmzone.h | 10 +++++++++ mm/memory_hotplug.c | 6 ++++-- mm/sparse-vmemmap.c | 47 ++++++++++++++---------------------------- 3 files changed, 29 insertions(+), 34 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1a18c1c40c8c..f8a2d823f7de 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2149,11 +2149,21 @@ static inline int online_device_section(const struc= t mem_section *section) =20 return section && ((section->section_mem_map & flags) =3D=3D flags); } + +static inline struct zone *device_zone(int nid) +{ + return &NODE_DATA(nid)->node_zones[ZONE_DEVICE]; +} #else static inline int online_device_section(const struct mem_section *section) { return 0; } + +static inline struct zone *device_zone(int nid) +{ + return NULL; +} #endif =20 static inline int online_section_nr(unsigned long nr) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index b428da66d279..d7a59167bec4 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -43,6 +43,7 @@ #include "mm_init.h" #include "page_alloc.h" #include "shuffle.h" +#include "sparse.h" =20 enum { MEMMAP_ON_MEMORY_DISABLE =3D 0, @@ -554,8 +555,9 @@ void remove_pfn_range_from_zone(struct zone *zone, /* Select all remaining pages up to the next section boundary */ cur_nr_pages =3D min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - page_init_poison(pfn_to_page(pfn), - sizeof(struct page) * cur_nr_pages); + if (!section_vmemmap_optimizable(__pfn_to_section(pfn))) + page_init_poison(pfn_to_page(pfn), + sizeof(struct page) * cur_nr_pages); } =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 02da1321197d..7b5b9ceec697 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -193,6 +193,8 @@ struct page __ref *vmemmap_shared_tail_page(unsigned in= t order, struct zone *zon set_page_node(page, zone_to_nid(zone)); set_page_zone(page, zone_idx(zone)); prep_compound_tail(page, NULL, order); + if (zone_is_zone_device(zone)) + __SetPageReserved(page); } =20 page =3D virt_to_page(addr); @@ -490,23 +492,6 @@ static bool __meminit reuse_compound_section(unsigned = long start_pfn, return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; } =20 -static pte_t * __meminit compound_section_tail_page(unsigned long addr) -{ - pte_t *pte; - - addr -=3D PAGE_SIZE; - - /* - * Assuming sections are populated sequentially, the previous section's - * page data can be reused. - */ - pte =3D pte_offset_kernel(pmd_off_k(addr), addr); - if (!pte) - return NULL; - - return pte; -} - static int __meminit vmemmap_populate_compound_pages(unsigned long start_p= fn, unsigned long start, unsigned long end, int node, @@ -516,21 +501,18 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, pte_t *pte; int rc; unsigned long flags =3D VMEMMAP_POPULATE_DAX; + struct page *page; + unsigned int order =3D pfn_to_section_compound_order(start_pfn); =20 - if (reuse_compound_section(start_pfn, pgmap)) { - pte =3D compound_section_tail_page(start); - if (!pte) - return -ENOMEM; + page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!page) + return -ENOMEM; =20 - /* - * Reuse the page that was populated in the prior iteration - * with just tail struct pages. - */ + if (reuse_compound_section(start_pfn, pgmap)) return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), flags); - } + page_to_pfn(page), flags); =20 - size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); + size =3D min(end - start, (1UL << order) * sizeof(struct page)); for (addr =3D start; addr < end; addr +=3D size) { unsigned long next, last =3D addr + size; =20 @@ -546,12 +528,12 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, return -ENOMEM; =20 /* - * Reuse the previous page for the rest of tail pages + * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), flags); + page_to_pfn(page), flags); if (rc) return -ENOMEM; } @@ -883,13 +865,14 @@ int __meminit sparse_add_section(int nid, unsigned lo= ng start_pfn, if (IS_ERR(memmap)) return PTR_ERR(memmap); =20 + ms =3D __nr_to_section(section_nr); /* * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - page_init_poison(memmap, sizeof(struct page) * nr_pages); + if (!section_vmemmap_optimizable(ms)) + page_init_poison(memmap, sizeof(struct page) * nr_pages); =20 - ms =3D __nr_to_section(section_nr); __section_mark_present(ms, section_nr); =20 /* Align memmap to section boundary in the subsection case */ --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCD7839E178 for ; Fri, 11 Sep 2026 05:04:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103043; cv=none; b=FltntzaQB5Hvsfqp6hRbuuk56W9dfIA8nqzOs5WGYqX/xtbAvCEXdb1OE5PInzH8UZv/xR26fidCTKZI5/ha1jowizZe0FhrpREXg9OmfKFNYEiKsRlbnjHWflf5V5m3cgY7XVGeQgx3TOftYKjqy9gBSGZyd02Uz/FQnsQ4Lb0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103043; c=relaxed/simple; bh=MxAOev2p2NgmPSdtM2IAWo2HVf04Xw3aSZERHz9cFF0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s3ANFmMh/CaGw4H5/C9AI+3q76SL86oxheDVQ4CZD1fnC2w32YeOmij+kCsgcE70jHuYoqZxX6jGDM8vv3umALF5ERXwYINFLYFGuOHIC1KTznIiXTygJv+Ag5tEEQvfQ+W8vqmA2N64ugktO4kEfqj28DGAd0kd5BMWigclsto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=kZRkvwL7; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="kZRkvwL7" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-3964dfb5b9aso622916a91.1 for ; Thu, 10 Sep 2026 22:04:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103041; x=1789707841; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9h5irsdiFbTSRTtgQf8+DpZjFaqFHIv3CtCmLZFKqxE=; b=kZRkvwL7aI/ZD8UrHtWmTILqO6SchcRnAQ3jlbTPfHRK0DrzcqToBswt71+7GGWY32 tQMrYXHmqs3p8g6daooCodxgGr1PNnD5rCpqS42iwMGph4QIC0DkUqUC7Xyh1rvAH7Ko ucVFHx55Q2e2jN7fLLWmZ+lIlQb9Z9qLi1kSW92ZGVv1gM6P98+lW3S9TQ6b8094IAj+ BuVXeJPHuRUg53ciebcrxmenMLPV+KbRxdcgDBmbAMKlfuhkV8Qwdd2lUOHzGRd5PEjL 4iSrMhTB7/X45RzD2BmVMOX3d2uvHvPzJBwRBaLkyQWPNe59ntL10zBag1+UFbTbFLkD iNaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103041; x=1789707841; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=9h5irsdiFbTSRTtgQf8+DpZjFaqFHIv3CtCmLZFKqxE=; b=D9rM5TnlUH33B2EKp8wSuT8lnWXEtVY0ftoSDKbWtIWfhlFzGu6l35RyAkcIBIVsg0 vBOIUhlzM8UxiEK3ky42vKiAPtAipzRfGYUz8jP4/uZMRGRvj2AIXpEYvj2eeazVtP5Y wAeJy5oasVC7V1se3T0tKCXvVN8SoqrhQxtGA7utFjxpblMLDANOyxM/F65pdIOkD0nJ EloxwdTJStom1nANDfsj+f7CRJMSw4G0NbMccSm7J8DQCVeaU7mN2ILpmxz7w4J+3wtR MQB1ZMlVz4+fZP+Xe0Mq4PQKRz0UrBXjPW7qAq8ELtUPatBhU5e9eJmU5+7DnCPoCBOq hP5A== X-Forwarded-Encrypted: i=1; AKwUvBwG+etjR9aTvsDE8pDvcM9CBR8s43CmuO3iqwywAvCRD4aSz3Q1dcuT8rbCJS4Zm1VVa0BMydSbcwqUwxs=@vger.kernel.org X-Gm-Message-State: AFuF++mXttX262G0pNtcWrFQdieR8gJY/8CI8r1tTmoE+J5taEHEYtt1 4+WCyslTCvU9tNkdk5pAQ0p353bCmDP4qUlj4O09p/JAep0RFCCwfyUOqK6UcSVCur8= X-Gm-Gg: AYBFou2Wnong5KXNVikCBbf8yfqodak9I0+8mwdXMpkv9UpQMz7YBJbA7lhdRwMuKlL p01/RXgFnAGyyAZWO1H8EjKatWT8CnWwdP+jcNBm5e6FH8BBW5vBAqmqktjhRxHoE4G1oK6jg9Y MFO0kvquk5rO2irw74ReBVrSUzS6758qy42DoLdYpEO2G5EjlXyZSMyGPJIeypoAFOPzwvS1BuL v8uLfQytH5U/wFYlc0iOd3lTqAOTpk1uxAIEA2hRsCXnvP/98/CaQYbLMqio+NJmpqCFGSFGIwg BMg+f/Z5+4S5joWkUea9uthe2gXlfVE0PEhUkPw7Qv+itu91hTE0xFf9iWW459JOj5KhsHQRB34 wxnOQRK+Us5xR8014cU1UOtLqgrGj2F35wza43U1XWQtT/idNcw3ztgbQ8y0Jd1UgkE8fk/d28p X6kbE+gEfHdSDVsy4WQSk802ZaZQoQqZWIMCAWzw4JWHi0j/a++a2s+jGZSLh2SrLTTLLjy2LdX wmRQQPPRzeJzllO1hHhWZK6XCB7WJAx7OvxpGzexg0aptY= X-Received: by 2002:a17:90b:1648:b0:36b:bec8:94c5 with SMTP id 98e67ed59e1d1-39d9bec50b9mr3746365a91.10.1789103041109; Thu, 10 Sep 2026 22:04:01 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.03.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:04:00 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 07/11] mm/sparse-vmemmap: move vmemmap optimization helpers to a public header Date: Fri, 11 Sep 2026 13:02:24 +0800 Message-ID: <20260911050228.58884-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The vmemmap optimization helpers currently live in mm/sparse.h, which is an internal MM header. That works for MM code, but prevents powerpc from using the same interfaces without including a private header. Move the declarations and inline helpers to include/linux/vmemmap-optimization.h. This is a preparatory change for powerpc, which has its own vmemmap optimization implementation and needs to use the common vmemmap optimization interfaces from architecture code. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) --- v3: - Update the subject and commit message to describe common vmemmap optimization helpers - Collect Acked-by from Qi Zheng v2: - Fix missing header dependencies. --- MAINTAINERS | 1 + include/linux/vmemmap-optimization.h | 94 ++++++++++++++++++++++++++++ mm/hugetlb.c | 2 +- mm/hugetlb_vmemmap.c | 2 +- mm/sparse.h | 76 +--------------------- 5 files changed, 98 insertions(+), 77 deletions(-) create mode 100644 include/linux/vmemmap-optimization.h diff --git a/MAINTAINERS b/MAINTAINERS index 3ef018673b5e..e4412c3d8d45 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12096,6 +12096,7 @@ F: Documentation/mm/hugetlbfs_reserv.rst F: Documentation/mm/vmemmap_dedup.rst F: fs/hugetlbfs/ F: include/linux/hugetlb.h +F: include/linux/vmemmap-optimization.h F: include/trace/events/hugetlbfs.h F: mm/hugetlb.c F: mm/hugetlb_cgroup.c diff --git a/include/linux/vmemmap-optimization.h b/include/linux/vmemmap-o= ptimization.h new file mode 100644 index 000000000000..2327929467f3 --- /dev/null +++ b/include/linux/vmemmap-optimization.h @@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * vmemmap-optimization.h + * + * Generic vmemmap optimization declarations. + * + * Author: Muchun Song + */ +#ifndef _LINUX_VMEMMAP_OPTIMIZATION_H +#define _LINUX_VMEMMAP_OPTIMIZATION_H + +#include +#include +#include +#include + +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +static inline unsigned int section_compound_order(const struct mem_section= *section) +{ + return section->compound_page_order; +} + +static inline void section_set_compound_order(struct mem_section *section, + unsigned int order) +{ + VM_WARN_ON(section_compound_order(section) && order && + section_compound_order(section) !=3D order); + section->compound_page_order =3D order; +} + +static inline void section_set_compound_order_range(unsigned long pfn, + unsigned long nr_pages, unsigned int order) +{ + unsigned long section_nr =3D pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_compound_order(__nr_to_section(section_nr + i), order); +} + +static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) +{ + return section_compound_order(__pfn_to_section(pfn)); +} +#else +static inline unsigned int section_compound_order(const struct mem_section= *section) +{ + return 0; +} + +static inline void section_set_compound_order(struct mem_section *section, + unsigned int order) +{ +} + +static inline void section_set_compound_order_range(unsigned long pfn, + unsigned long nr_pages, unsigned int order) +{ +} + +static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) +{ + return 0; +} +#endif /* CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION */ + +static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +{ + const unsigned int order =3D pfn_to_section_compound_order(pfn); + const unsigned long nr_pages =3D 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + +static inline bool vmemmap_optimizable_order(unsigned int order) +{ + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) + return false; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; +} + +#ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); +#endif /* CONFIG_SPARSEMEM_VMEMMAP */ +#endif /* _LINUX_VMEMMAP_OPTIMIZATION_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0d101ef934fc..7af55e1544e9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -38,6 +38,7 @@ #include #include #include +#include =20 #include #include @@ -52,7 +53,6 @@ #include "hugetlb_cma.h" #include "hugetlb_internal.h" #include "mm_init.h" -#include "sparse.h" #include =20 #define HUGE_BOOTMEM_ZONES_VALID BIT(0) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 5ddf06b83c96..a27f46fffb54 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -15,10 +15,10 @@ #include #include #include +#include =20 #include #include "hugetlb_vmemmap.h" -#include "sparse.h" #include "internal.h" =20 /** diff --git a/mm/sparse.h b/mm/sparse.h index a250bea4088e..a5111087ee3a 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -9,80 +9,7 @@ #define __MM_SPARSE_H =20 #include - -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION -static inline unsigned int section_compound_order(const struct mem_section= *section) -{ - return section->compound_page_order; -} - -static inline void section_set_compound_order(struct mem_section *section, - unsigned int order) -{ - VM_WARN_ON(section_compound_order(section) && order && - section_compound_order(section) !=3D order); - section->compound_page_order =3D order; -} - -static inline void section_set_compound_order_range(unsigned long pfn, - unsigned long nr_pages, unsigned int order) -{ - unsigned long section_nr =3D pfn_to_section_nr(pfn); - - if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) - return; - - for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) - section_set_compound_order(__nr_to_section(section_nr + i), order); -} - -static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) -{ - return section_compound_order(__pfn_to_section(pfn)); -} -#else -static inline unsigned int section_compound_order(const struct mem_section= *section) -{ - return 0; -} - -static inline void section_set_compound_order(struct mem_section *section, - unsigned int order) -{ -} - -static inline void section_set_compound_order_range(unsigned long pfn, - unsigned long nr_pages, unsigned int order) -{ -} - -static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) -{ - return 0; -} -#endif - -static inline bool vmemmap_optimizable_pfn(unsigned long pfn) -{ - const unsigned int order =3D pfn_to_section_compound_order(pfn); - const unsigned long nr_pages =3D 1UL << order; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; -} - -static inline bool vmemmap_optimizable_order(unsigned int order) -{ - if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) - return false; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; -} +#include =20 /* * mm/sparse.c @@ -142,7 +69,6 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP -struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9E173A6EF7 for ; Fri, 11 Sep 2026 05:04:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103048; cv=none; b=FQZJZF8lHOAu2o0Oi75hBxZdnGryzpsLotxnzSP94SESZHTmAOejs/BfZa90nT70Jphsc3f+JyDwcu9zjUNYSRp2msZOON15QOZDPOHwI28dg5cZdXBZ89aSg0+Ndpo9XMuAgoT2qXuTwz4osmud98AvwdjIs6ngXeTsoxugtf0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103048; c=relaxed/simple; bh=XXJonAu5v5lmenvG0H3fDY0XU6EMxiTGRHxJPTiZN5c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nOHQe8bNkwE0RY9+nrCYTu5YBm1UgJysf5vTwKhYKq8uK7MTtOERtbWN7Vf2cS83qHrS+nDXv3AenTnPYHVH3KK6f+DQ6KoWeYB9jMe7hISlMoPtbzGMJmPCSzh5Uu/st3/2EEETbOv5na1xcuGVYdEH0ftYGiwzwlbpkJPrFf4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=E57CwyRc; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="E57CwyRc" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-cc4c02ddd62so456229a12.0 for ; Thu, 10 Sep 2026 22:04:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103046; x=1789707846; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=TZbZ6WC7vZbHjIhgnid+8T29mCn5dRvITTk6eMnH6Vk=; b=E57CwyRcLDnZd7bIDcp1fJk8ryu52AKMO83H2R7foYaEl9FlsBHrCVFUcHt2e2mzIy EM3YUfrcOsveia8pE1AcqtyazgPk9PRrHUDW4AL4BDiYU0PUKZ8ovPsARF9vUorRtLfX wQ9BzO4zK7iM1aMBlAvQ0L3PZgaRB9K6vSJ4ZoLPA9Wes1qqrhNDSPva+CcpVAnWomQj wgsZXdosUltTSp62muiqS0idXLLZ8aFcs0YaOY7WRc2cusPWsWpkzCkfKnODeuUOboDL PlerVGL6chbJ/7/1nV/NUKsd7RqP9RMZ814hHosuqiQfXx+CrZX63p8pe4d7TbTULjWT 2bdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103046; x=1789707846; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=TZbZ6WC7vZbHjIhgnid+8T29mCn5dRvITTk6eMnH6Vk=; b=e8RPE8OLi6zAB1fM/LVtwBky/yyAgPVgtGfL+HodCylnVz1qVYViPTmd4kyc8+2Sh5 m4s5nPm4fKlZBzITZdbyJZovoNMFtcUnuNh1chaWpgJLjf/0Goi2cbONtDxipS2fjEZc tRQpeeO06FgeyTW4FDETIl7bih34LvLheC+WSEKoI9OKAdo+LkvYHI3mCzFKVYbG/EXb daklPaRbroVVkv0QLh9+uN4kPA5wHvdCFxhT2M6lPiYB15MYErP8yvsk9re3Lhvndxw0 NXlDN76bEijkLwHm6P/8ER0Oz/NxWmQTk4mlzIZ7VpfgXIlWptibhUbc+Su4lGOvVgqn Kp1Q== X-Forwarded-Encrypted: i=1; AKwUvBzSfv6x4sUxq3qGxtFcpa1W+Ee3pcMLYx/+7v+cv5x6z6z2MV6JM4sbNiUoU0FbO6oTI3u0R+H/w0w1ids=@vger.kernel.org X-Gm-Message-State: AFuF++nTJFuj4c8NGrriDz5/l4cP3Dbr4FMbFK1BQTFOquhYc9BD61Zf GZc9NgrQOLtCcfbI6APSI3R3s3G7eHC27arBocQVfTY5QXhbSxlmhN7k5o4wLKdkXhg= X-Gm-Gg: AYBFou2Nc2VvhB1ceMkrYHSDzJj0NvDGqGVrC/1OLE7LZSPVaSGdbOySzPsWF5yER3s skDZ2ynkJGI0E74fcQqiZVoKMUxJHcrJpDK8U4jh8WpHvtlaGW3JupVKj/qq6wHAsWl7X+C65YI ZlSJymx6nAK89x/4R8Jo/w22vBzRFe1KbxWJaev5brQhOduWOTAwjTtOyD5IVS3F6G+b1kjLAU1 LqOz15meK8dUG3ARNMuvrH3hSLBysBId2pxKY/ECLuNjYCi9EZGO/mWOsRT1qI6G8auF6ycIkJy mVfZjdlYh+kK66bv+RMMsn3FepQDEWGa9zsyqTN9L0qhI2xW5kfeMO5l9CWhkcjyJQkB3lDLWuG XhcknvOXfODyGj7w0lksWsNQaoORqFTcjgknYfnXfXQWoYO4AROI5L1t9BhjVl77SCWQ/4a1Cug Qp8hgO5S7Znzs1H5aZfVpSwV6/khjEUnbzeFJOtgQn3jAAPSuG4r2Xpbo+EuuZ4bu322mywUIUF nXJgQuFaw0JfLqlzfsXONl9tJT8FBzhLTQW X-Received: by 2002:a17:90a:fc48:b0:39a:e983:d4bd with SMTP id 98e67ed59e1d1-39d9c3527b5mr3643408a91.24.1789103045863; Thu, 10 Sep 2026 22:04:05 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.04.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:04:05 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 08/11] powerpc/mm: switch device DAX to shared tail vmemmap pages Date: Fri, 11 Sep 2026 13:02:25 +0800 Message-ID: <20260911050228.58884-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The powerpc radix compound vmemmap population path still finds a reusable tail page by walking the vmemmap page tables. Switch it to the common vmemmap_shared_tail_page() helper instead, so it can use the shared vmemmap page directly to simplify the code. This removes the powerpc-specific tail-page lookup and its fallback path and aligns the device DAX vmemmap optimization path with HugeTLB. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 80 +++--------------------- 1 file changed, 9 insertions(+), 71 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index cf692b2b5f7b..ee068f24a79f 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 #include #include @@ -1250,59 +1251,6 @@ static pte_t * __meminit radix__vmemmap_populate_add= ress(unsigned long addr, int return pte; } =20 -static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, - unsigned long pfn_offset, int node) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - unsigned long map_addr; - - /* the second vmemmap page which we use for duplication */ - map_addr =3D addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; - pgd =3D pgd_offset_k(map_addr); - p4d =3D p4d_offset(pgd, map_addr); - pud =3D vmemmap_pud_alloc(p4d, node, map_addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, map_addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, map_addr); - if (!pte) - return NULL; - /* - * Check if there exist a mapping to the left - */ - if (pte_none(*pte)) { - /* - * Populate the head page vmemmap page. - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL= , NULL); - if (!pte) - return NULL; - /* - * Populate the tail pages vmemmap page - */ - pte =3D radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); - if (!pte) - return NULL; - vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); - return pte; - } - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1320,6 +1268,12 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, pud_t *pud; pmd_t *pmd; pte_t *pte; + struct page *tail_page; + unsigned int order =3D pfn_to_section_compound_order(start_pfn); + + tail_page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!tail_page) + return -ENOMEM; =20 for (addr =3D start; addr < end; addr =3D next) { =20 @@ -1349,10 +1303,9 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + PAGE_SIZE; continue; } else { - unsigned long nr_pages =3D pgmap_vmemmap_nr(pgmap); + unsigned long nr_pages =3D 1UL << order; unsigned long addr_pfn =3D page_to_pfn((struct page *)addr); unsigned long pfn_offset =3D addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); - pte_t *tail_page_pte; =20 /* * if the address is aligned to huge page size it is the @@ -1377,23 +1330,8 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + 2 * PAGE_SIZE; continue; } - /* - * get the 2nd mapping details - * Also create it if that doesn't exist - */ - tail_page_pte =3D vmemmap_compound_tail_page(addr, pfn_offset, node); - if (!tail_page_pte) { - - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - if (!pte) - return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - next =3D addr + PAGE_SIZE; - continue; - } =20 - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*ta= il_page_pte)); + pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, tail_page); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0837E3A6EF7 for ; Fri, 11 Sep 2026 05:04:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103054; cv=none; b=jUeVLGrxz6nbpt94baMkHtT4n4rlMasGAuMkLLyAqw/yuetMGCnR73sfKZ8M4qoDJi4hUHMie4tJFpnBV3FjdvWiDZx1EDCsFe0FaEUAC5URbYAjvOQJVYX1i59RBRnnuIxUnHlrHyRTERDtgiRtSzhAgHerjA1BUW4fepK1GVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103054; c=relaxed/simple; bh=gShyvx+/cniTHn6nFfCC/e3ahAHIaLdaIffqkvBBwuY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=D9pCXZRFn99SqLIIYz7H7egkENni1ZgYjQw+5ioxxoin+OVRqwfGqFfOjeC/stRpkVctzM6e1ja197BljZXMiw9hNqrPsiAllFLaiar9sp7nLFbZRzrmb98yNr/UsmwRQanZ+HD/6sZeUWYlF61zO1KQxyJvOcH72hli3LksIn4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=L9t5b41U; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="L9t5b41U" Received: by mail-pj2-f12.google.com with SMTP id 98e67ed59e1d1-398beb616f5so163156a91.1 for ; Thu, 10 Sep 2026 22:04:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103051; x=1789707851; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=OZ9EEhriv4uQFVrwFElKF7ZgZH1tSjYdJIJZrcuS8I4=; b=L9t5b41UgRlQyLt86N4Bh05WiSlXemjLr+vEqxXdO1Nw7PO7kjG7/jRorIEjR+OnJS 0kJnlkI8ok2PLJ2NbPc6IDviOAmtQYcc4bUxxL6PC3pJuU43VJT9v2kriKhmpQwUHkQa ieOqQ8mZZaR21TcaJW6Gi21LL5xLB2afBL6wGuS8S1KSJTg1Gkj4Bk19MO/NODN1l/5h mvwYk2YuVdHxaqTD69JMot1oM5UDesZT6clPVTXKtxISGbstjQrDiQDnLQaDYKwq3sHJ 7xIyXqe5zNKgCMtW9s/I0jhj3lq4xOxi3pTpm38ndNFLpvqrThxm3myc1YWXQ2VGTEbq AxLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103051; x=1789707851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=OZ9EEhriv4uQFVrwFElKF7ZgZH1tSjYdJIJZrcuS8I4=; b=gLPc4Pb1De8/4ojTGW+RBANHCf3xjqISwy6xWXPv1BcOrakW8VEK3W7/efXE1VvDVN W1+SQEerXH6HKGeOSm1mHEAc3AcdOH17YnFozs8JM6V51/n/zhQqSHj9QYbqbHhEEaTx OlQD0gCEgc+C2sayMkoy2GyZXDQm1osor5mQ4pM96OBcTDwYWqBRaJePoM7WoaMw5C9S jdg4edt5XdkkJdxNAPzh2JsWBNojjQHZkkFpV6Q41asQpxLDR3kA8+9HrLdDsdUN01Qw WERBNkjCGaeQhe9OTiTobaXVruVz1OtdvtlxeVfD7KlLmYYc8R/l38VJa1MGk3qxQFqU Z7uQ== X-Forwarded-Encrypted: i=1; AKwUvBx73vSDdBvJvwOusXrM7T5lP+A+DCJAMPoN4HgZO229x2ZX3M2xOo+FQ1ECATjQ+nYc03hcpkAjY49mH0w=@vger.kernel.org X-Gm-Message-State: AFuF++loaYm5SqjuqfxwGwkh2Ym1AuPczoMaxmLJLznrHlk3KBRta8Lq oC5/zr63xUXn+dNsMO1AQ4k4aYnjj/ztrIiaB5L36VtbyGDf645RHo9Ik9Iqu8rqL7I= X-Gm-Gg: AYBFou2tzv264M45aED5uqos9HJwKkYROFjKjrLcMBqfFiw0r6fTPErv9Ebsxt4DIwa gIxZ6waHFDjCkhxs+seaOJA9itmcSrMHyc1kgGDfDq0G43v6kLtgm7PRbza9ZqKTMoZBtMZtUDk OOMtQwC2w3X6FevOkkw1VBTDV2zmblv4OloMwbtnRw7oyOLRgdkAZuZpFpztTqoBIRR1WSDD/uJ tpeCb2VxMv2P/YGy7koB4Ofv78WA1VzVmm4HzffdFFWWQa4lP7/U9/xQi3fd0wwCiPS0fJU45J0 YnDaBLBIbhaRnpB9YdSrqFLxYUWyUi1Mx+sDU5qtPifJ4PZYKtDQj5fz8rIIgx8exg5Hfw8oGos sQ16baAM8Td0OGE4l7SmFxCZhwmtG1doM0lF5GskJu5Bslt5EB0i8DxOZnNLPhA2u3jffpnxir2 j+sq2y1skpM/wktiTDUln0tF0CbXhK+8VOBEd458W0SdFxgOzERDipyeqe3O44cU+7wsjuwCjae /RmjyvRM/6SDycBYwOjnClanT7wRkj4KkAV X-Received: by 2002:a17:90b:4c4e:b0:38e:7297:a92e with SMTP id 98e67ed59e1d1-39d9805da27mr2729392a91.9.1789103051065; Thu, 10 Sep 2026 22:04:11 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.04.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:04:10 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 09/11] mm/sparse-vmemmap: drop the extra tail page from device DAX reservation Date: Fri, 11 Sep 2026 13:02:26 +0800 Message-ID: <20260911050228.58884-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The device DAX vmemmap population still reserves one extra tail vmemmap page after the head page. Drop that extra reservation and let the shared tail page cover all tail vmemmap pages after the head page, so DAX follows the same reservation model as HugeTLB. This reduces the reserved vmemmap pages for optimized DAX mappings to one and removes the now-unneeded first-tail population from the generic and powerpc paths to simplify the code as well. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Collect Acked-by from Qi Zheng --- arch/powerpc/mm/book3s64/radix_pgtable.c | 46 ++---------------------- include/linux/mm.h | 3 +- mm/mm_init.c | 2 +- mm/sparse-vmemmap.c | 13 ++----- 4 files changed, 7 insertions(+), 57 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index ee068f24a79f..9ca28e4a610a 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1218,39 +1218,6 @@ int __meminit radix__vmemmap_populate(unsigned long = start, unsigned long end, in return 0; } =20 -static pte_t * __meminit radix__vmemmap_populate_address(unsigned long add= r, int node, - struct vmem_altmap *altmap, - struct page *reuse) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - - pgd =3D pgd_offset_k(addr); - p4d =3D p4d_offset(pgd, addr); - pud =3D vmemmap_pud_alloc(p4d, node, addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, addr); - if (!pte) - return NULL; - radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1297,7 +1264,7 @@ int __meminit vmemmap_populate_compound_pages(unsigne= d long start_pfn, if (!pte_none(*pte)) { /* * This could be because we already have a compound - * page whose VMEMMAP_RESERVE_NR pages were mapped and + * page whose retained vmemmap page was mapped and * this request fall in those pages. */ next =3D addr + PAGE_SIZE; @@ -1318,16 +1285,7 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); =20 - /* - * Populate the tail pages vmemmap page - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, = NULL); - if (!pte) - return -ENOMEM; - - next =3D addr + 2 * PAGE_SIZE; + next =3D addr + PAGE_SIZE; continue; } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index a2ebe87e7654..969594074fd2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5167,7 +5167,6 @@ static inline void vmem_altmap_free(struct vmem_altma= p *altmap, } #endif =20 -#define VMEMMAP_RESERVE_NR 2 #ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) @@ -5187,7 +5186,7 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, * For vmemmap optimization with DAX we need minimum 2 vmemmap * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); + return !altmap && (nr_vmemmap_pages > VMEMMAP_OPTIMIZATION_PAGES); } /* * If we don't have an architecture override, use the generic rule diff --git a/mm/mm_init.c b/mm/mm_init.c index efffa8609b85..56bb4567a494 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1056,7 +1056,7 @@ static inline unsigned long compound_nr_pages(unsigne= d long pfn, if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 - return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); + return VMEMMAP_OPTIMIZATION_PAGES * (PAGE_SIZE / sizeof(struct page)); } =20 static void __ref memmap_init_compound(struct page *head, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 7b5b9ceec697..39f0cfeefc6a 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -136,7 +136,6 @@ int __meminit section_nr_vmemmap_pages(unsigned long pf= n, unsigned long nr_pages { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_compound_order(ms); - const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); @@ -147,13 +146,13 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages =20 if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; + return VMEMMAP_OPTIMIZATION_PAGES * nr_pages / pages_per_compound; } =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); =20 if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; + return VMEMMAP_OPTIMIZATION_PAGES; =20 return 0; } @@ -521,17 +520,11 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, if (!pte) return -ENOMEM; =20 - /* Populate the tail pages vmemmap page */ - next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); - if (!pte) - return -ENOMEM; - /* * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - next +=3D PAGE_SIZE; + next =3D addr + PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, page_to_pfn(page), flags); if (rc) --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5D3C39D6E5 for ; Fri, 11 Sep 2026 05:04:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103060; cv=none; b=L7NiSTGKQzda5VUjdiGduUvzj2kDPtgdZEiHChtK8YsSjURIFvZ7zjBe7tpcifExI7FLzFTFQdDKkDqSSsqSLz/Mgq7MkLLmpwuRbQvWISUBW5IZnxb4oqFkPOik+fc1rpSeE40qyxqeN1+IwXAWunz9mbZ5QAb1gxHZ6jmtoKw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103060; c=relaxed/simple; bh=o5WJM3/QFSVrnnx6cWLQ5UPoN7IuOCV+HHfC3SmFVvo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OEF1ksMbf+N7WXXYlwhBvKgUKiNdVx3q+xSqhCghcFbh1/gpCY4qDyLKI4F/oEg7XKNWAl/33foLuMKWMZYrLVGuJNoB14tQJl0yuqmbWH0sS/dYhtsQvevMQE9flu+5ZMOpbKgF+SmCg3zQV9YPqZ1RSQD2MyWJHuHy1Unw4UM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=AXRhn5PK; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="AXRhn5PK" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-39b9a99469fso468242a91.3 for ; Thu, 10 Sep 2026 22:04:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103056; x=1789707856; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iM8j8Wu3fwHIWd8x6lR8eMq2Esw45qvi78c+WOmPzrM=; b=AXRhn5PKXzYGtdB6OaHb4lTuZqqB4h+EoDqk/IJOlI6InFnWnM90fsx01O3JR/Nhvh XFgNuYI3lxy9BxipCENY0G0CXPVDjPeZYRoXpc15Ms7QpDiSZAJhisGA1vxgr8Md3j92 bBOOXxDM5KWKr2lfcwn5q9XAvc+IoWHNBw0HkFyjIZnBN4ZRhPCryeFy26+iKj00PLrw KbxVRoKC4Y9ijIS6NZs++3/3/hkfpBL++4FOIba3UknmxqshrxYpy8cobabXfdICRLs1 FsogvXRrjgfa3HG7M9AfZ42xmbtE3ZlCvBn5M9Al4WWVYsqWlFNRVM2r6MKoISBfV1rL zwfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103056; x=1789707856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iM8j8Wu3fwHIWd8x6lR8eMq2Esw45qvi78c+WOmPzrM=; b=qu6QVxEaUlRVYAlWHEn7BqfqY5fdWLh27Chxgfwjw4sLeTi/ScWB5hUWgBoveYj56B Cyri5fbsNvV4T4X3P08xK9TU4NMG93QvvJfOu5nqLpy/X3Xsg8z4xKHZ/A6PaIZsGJaK ezqeE/Nz8Y9cymrHVyxe56BHlakmDGA7U+9cLE7shbLThw9LUzKqXT9RitkTlzKdmzlZ NCGf1Hn60vIGkm44/YVN8Tgln7VzQE7lWpGKxShejlD9ynb7DpefVMu/vcl0fqMCAr60 vAGI28MC93asugY/bcl4gpb+30kB8gU7inVbgPkCbcnNCiyacZFPLEOxB5l2M6zPYAvw zslg== X-Forwarded-Encrypted: i=1; AKwUvBwE3BlITXFFQUUuHiGVVf/RuInXjwlYMElVtSbBO/OAcZSUxn8IAxTquQ58WhcekweXZ76AD1yK1U2NT9s=@vger.kernel.org X-Gm-Message-State: AFuF++kZr36kmDlUPIGOlf7o40qPmmkeQ/q7oaeMBvOP/3yjudpinAFA YpUFDO2zarbGG2v8LBHn2o1EKApy49RZ6wE3GyYRzQR1fQ7dIoYxB/Qn3pHNQ3usYnA= X-Gm-Gg: AYBFou03sopEjN7v70D+6QR16PITom6tDfVsf9cw1bunYT6kdnDHeXhV/4CMuIGHQn8 ndiS+ekXtGyQTnCJlKzv0IOTTQtYAhPNs8zUMjcZmWfUE2j9xzFe6CRRsPvxr8svULP5Q67bclL Tcxytk5D6bT5aGp/5CQS+J99irhQ+xc6l+nI2FO2MKVC1fdPoAFYCBL+vuCn5ODNpbQ8wzUd7Db kzsO9ly0SP7LxbcZ9RkSe9F5a7JcGVK8N+b02KUrlpYjeAnYBe1+iZFm+qpQl77lQN446flE7Qa CcUaFPq02DcCMoLBmRXeMmL4lYA84b/SrhL7HE7M3VHlhkczicon52qb5CPHxK6cMGJFH5Wt4ui KULSLW+ueUEfC78wPQV9VwhMO6O5BMDjuleGsfuPDVaP6Pq072xRZLgr1P5dTxDLCG3Z17FEjjo X6ldNE9rZAOrLqN1yy5V1C0TjNuxvp+T7PJesCXjfZT8CkC8pdK9XwDeDX89yOVpoNuTBw16PSp 22DaP8s//ZUJKZhvTRDM3AXvHbksb5Vrmgl X-Received: by 2002:a17:90b:3ecc:b0:380:540:d499 with SMTP id 98e67ed59e1d1-39d9bc382b7mr3533958a91.6.1789103056004; Thu, 10 Sep 2026 22:04:16 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.04.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:04:15 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 10/11] mm/sparse-vmemmap: drop unused section_nr_vmemmap_pages() arguments Date: Fri, 11 Sep 2026 13:02:27 +0800 Message-ID: <20260911050228.58884-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_nr_vmemmap_pages() no longer uses the altmap or pgmap arguments, so drop them from the helper and its callers. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Collect Acked-by from Qi Zheng --- mm/sparse-vmemmap.c | 10 ++++------ mm/sparse.c | 3 +-- mm/sparse.h | 6 ++---- 3 files changed, 7 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 39f0cfeefc6a..96506f594924 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -131,8 +131,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 -int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages) { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_compound_order(ms); @@ -637,7 +636,7 @@ static struct page * __meminit populate_section_memmap(= unsigned long pfn, struct page *page =3D __populate_section_memmap(pfn, nr_pages, nid, altma= p, pgmap); =20 - memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages)); =20 return page; } @@ -648,7 +647,7 @@ static void depopulate_section_memmap(unsigned long pfn= , unsigned long nr_pages, unsigned long start =3D (unsigned long) pfn_to_page(pfn); unsigned long end =3D start + nr_pages * sizeof(struct page); =20 - memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages)); vmemmap_free(start, end, altmap); } =20 @@ -658,8 +657,7 @@ static void free_map_bootmem(struct page *memmap) unsigned long end =3D (unsigned long)(memmap + PAGES_PER_SECTION); unsigned long pfn =3D page_to_pfn(memmap); =20 - memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); vmemmap_free(start, end, NULL); } =20 diff --git a/mm/sparse.c b/mm/sparse.c index cc28bb41fdb1..b75921c622ed 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -250,8 +250,7 @@ static void __init sparse_init_nid(int nid, unsigned lo= ng pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, SECTION_IS_EARLY); usage =3D (void *)usage + mem_section_usage_size(); diff --git a/mm/sparse.h b/mm/sparse.h index a5111087ee3a..530692cdd516 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -70,12 +70,10 @@ static inline void sparse_sections_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); -int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap); +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages); #else static inline void sparse_init_subsection_map(void) {} -static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages) { return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); } --=20 2.54.0 From nobody Fri Sep 25 15:14:23 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC67430AD05 for ; Fri, 11 Sep 2026 05:04:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103067; cv=none; b=aBopOB0c9grlZjRsuUg8zKAm6/sT/8xvrGNfwkpMEMsjUZqycfKVIeqzHqfdfwXiphTT5U87tLI/J6y0hp/FynR+7I2+sBFK5vPcWF7vAzz0Cw8xDIwvfI5yu2EGsJhj8cB5EU13NEXXyiop84rJRBIk8clCd8vx4W9YZSKw3FA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789103067; c=relaxed/simple; bh=SzfK6CJnam9Dg4BVTpagUZL4n3gDBm63Y+qtGXTWOqU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=arJ36d04ccQA6bzY4BIPq2OoLdyag6WcPxtUxw2yg4HC3AAyzdg5LRR4/3zSKmT9qO4y4rjX52mVNcealrK57DdQt49v+fsT0UgXV4LIyA8VuF9ua3wSSZQSc9COci3FoM7P8oU3NFnykn1wyIv/ph4pW4CaEiedUEMkPzP1s8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lMLbw6AX; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lMLbw6AX" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-382ef647e20so650089a91.1 for ; Thu, 10 Sep 2026 22:04:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789103061; x=1789707861; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=zUhtpwjDYsFhZVHLtQ8dC1wt8tqWIMzZ7omgfR51vXw=; b=lMLbw6AXCXte+pH4CB8ja5Xcp8wCeJnJUgn9rT13NJJ+YyUkB+lQ4+2kF5i/r3CCQn AVBMdveZPrIbPhEkP+0NzFZ+WAyzNJxT+sIhCorIC1cyxg8kJlVm9ltdksUHvs/DZNqf Aa9GN+z2nOhnBXpVwTtLhoWidVVbDXqRitQggPYqA2EgEY7jdZvAfbLhtkcHlYdMxxR+ mHWmhL4fWS+nWIBgo+kCOTdyZS8c+6RLrkExL0sadAzEnPyIbrKmHvCFgg1XG5bgF3Vr YhL/OPH4Q/Gmx+k7feyZrXC08NkkeCv1+FOBGePL0FQIOFqYzdOetBcV+FUeDWz2ZVO6 a9Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789103061; x=1789707861; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=zUhtpwjDYsFhZVHLtQ8dC1wt8tqWIMzZ7omgfR51vXw=; b=iP6Dcy7k3m3lfaF8pJWFbZsr429r9DXBtZX5WozlVXsgn/0qOfcafkwyQ1AHx+ueWB UR5y8EiZNrcHGTvO9maxdLODtbEPC3ihQ3B70UElSbCtRxP94ODuzHvmXcq54voBz1Ot QtkljGXLh8CA9KvzZMRUUIlb7RL0viSEjk/FC9dPYIGr7obLhbBErbbMOG0lwmnXN7bE YEgoJdfrwj7kWexfzrjnG1EfJ05Oagcy8gO1+hNZcaJI27a1ehA1suv2ZP5g3ZEC8DmD kdVpaOQFT2LWgP96HYs7xkr6Vt3jQmczW7AA5ZhFHvs+ddDBX1dMIh6i52sIWLTSvgil lkbA== X-Forwarded-Encrypted: i=1; AKwUvBwp9yLqfJmH2oQm//BLqjbMaWuSnMj3SQjTwSchHsX6wEB5eHueM1PbV/6DiXJJ2U2p3RT+/HZiPQzSwRQ=@vger.kernel.org X-Gm-Message-State: AFuF++kFv35jj4UWA9VPPsY5GooBI+cdlamJp0mdiCBX6L24PuTz6Au3 zT7OA4Kby6yirIcbI8N4nZRGxw1f0ITlES91tv9w1qqzAHI67dG7qBJFYrnOKES+qy0= X-Gm-Gg: AYBFou0KcTul1/DvmlgHhW2g6YLJQUa7vLIBRMz9qhjEpzWzx/KB4paCRkKRpyAljAr uU8EXAU77ZNE0lUESvZpWibVZq6JTOXZuQIsNz/YHOE2QeHiZ5sYeXYpbswqu6YrUtEqOCbHwh/ NXkMqud3FzjHLH5ICvDp9TOfKd27wWFW1f2MJX+AGQtSOulSbKEyU0JNlcG1Vi1ktKsYMzZqqZ9 ERjlJbZ4y1YemaAiFdm/OgqiL5hu3OppA06DIj8dR4yGmazhZKTMwkAkB12orUf2+9sCG61QfSP QsGw3ObvrVBI3WBUyXbQbt77aTUoLRI8DjEyRTFZfyhnTh3me7B5F2z9LKAUye5MPsI5+EU53mg 1PZbTW5tQxy+tKlAkvMyEnoCJCx1HsDT1DhZkx2vS0q2y0CWOQEpl0nxY7cK2H5LUkD1wKjO3N+ qSrKoVXIswegFXu37PhbKIIGN8s+hyYJOEYjVFw6v5OhSbK5/FYPdKlxuAnLPj6Bv62D0sIfrg7 d+Del4vptu6f79jVISxSmtyk07cfRN6RwUhG8lf+DUCfQA= X-Received: by 2002:a17:90b:2549:b0:398:dcfe:967a with SMTP id 98e67ed59e1d1-39d9c227fc1mr3740017a91.17.1789103060720; Thu, 10 Sep 2026 22:04:20 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.11]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-39d99531b16sm2651561a91.14.2026.09.10.22.04.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 10 Sep 2026 22:04:20 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v3 11/11] Documentation/mm: update DAX vmemmap deduplication docs Date: Fri, 11 Sep 2026 13:02:28 +0800 Message-ID: <20260911050228.58884-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260911050228.58884-1-songmuchun@bytedance.com> References: <20260911050228.58884-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX now uses the common per-zone shared tail page for vmemmap deduplication. The old documentation still described a DAX-specific layout with a separately populated tail vmemmap page and half the HugeTLB savings. Update the generic and powerpc documentation to describe the shared layout. In the powerpc document, keep the radix and 64K-specific details, drop the duplicated 4K PUD arithmetic, and replace the repeated device-dax diagrams with a single parameterized PMD/PUD diagram. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Collect Acked-by from Qi Zheng v2: - Clarify the commit message to state that the 4K PUD arithmetic is intentionally dropped reported by Sashiko. --- Documentation/arch/powerpc/vmemmap_dedup.rst | 90 ++++---------------- Documentation/mm/vmemmap_dedup.rst | 32 +------ 2 files changed, 21 insertions(+), 101 deletions(-) diff --git a/Documentation/arch/powerpc/vmemmap_dedup.rst b/Documentation/a= rch/powerpc/vmemmap_dedup.rst index dc4db59fdf87..8286acbca9bc 100644 --- a/Documentation/arch/powerpc/vmemmap_dedup.rst +++ b/Documentation/arch/powerpc/vmemmap_dedup.rst @@ -19,82 +19,28 @@ With 1G PUD level mapping, we require 16384 struct page= s and a single 64K vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Henc= e we require 16 64K pages in vmemmap to map the struct page for 1G PUD level ma= pping. =20 -Here's how things look like on device-dax after the sections are populated= :: - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 15 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - - With 4K page size, 2M PMD level mapping requires 512 struct pages and a si= ngle 4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapp= ing. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - -With 1G PUD level mapping, we require 262144 struct pages and a single 4K -vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we -require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level -mapping. - -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 4095 | -------------------------= -+ - | | +-----------+ +Here's how things look on device-dax after vmemmap-optimized sections are +populated. ``N`` is the number of vmemmap pages required by the DAX mapping +above:: + + Device DAX vmemmap pages (N pages) backing page= frames + +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= ----+ + | | | 0 | -------------> | 0 = | + | | +-----------+ +---------= ----+ + | | | 1 | ------+ + | | +-----------+ | + | | | 2 | ------+ + | | +-----------+ | + | | | . | ------+ +---------= ----+ + | PMD/PUD | +-----------+ | | A single= , | + | level | | . | ------+------> | per-zone= | + | mapping | +-----------+ | | shared t= ail | + | | | N - 1 | ------+ | page = | + | | +-----------+ +---------= ----+ | | | | | | diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_= dedup.rst index 9fa8642ded48..8c287ae3f86c 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -1,4 +1,3 @@ - .. SPDX-License-Identifier: GPL-2.0 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -192,32 +191,7 @@ to 4 on HugeTLB pages. =20 There's no remapping of vmemmap given that device-dax memory is not part of System RAM ranges initialized at boot. Thus the tail page deduplication -happens at a later stage when we populate the sections. HugeTLB reuses the -the head vmemmap page representing, whereas device-dax reuses the tail -vmemmap page. This results in only half of the savings compared to HugeTLB. - -Deduplicated tail pages are not mapped read-only. +happens at a later stage when we populate the sections. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ +Deduplicated tail pages are not mapped read-only. The mapping layout is th= e same +as HugeTLB. --=20 2.54.0