From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CDDF309F08 for ; Tue, 8 Sep 2026 03:04:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836646; cv=none; b=TS+0H1NtxZOaKv/1ZKDKYPKCkM1wro0b8r6P1XurSG/cHxdvjYJyGC+D3MOEuY+nKsSNpyQTcW6buNXG18SdH/Cb1lDiFFYpgbM+S+v6PbRQszO9UGZJFeMtxgJ943n1orgnqLHFHtexUepryDiQJuVhLYgw70uehThlPUaJJrY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836646; c=relaxed/simple; bh=ug15MoXQkYIwBETbu7RDyCK8xzJPchFY65UExPQwebU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EqC+3Qu0uqGtAZrr+fWqpokheTTvCFLXDZU4I1t7d6R+nKZ8JgIy8NTx7pDAPmd+kIH41S4Q0lldqnDrKItJuXOGy1kqwdORH+gNmcPB2SXmdOkoTBXdPVdo9DhT7Jrq9VhBGcnu37FvRTNeYXb0B++/ONZkOIPT9XnRZzrIU/4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=T5h6JRyn; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="T5h6JRyn" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-84e27035206so3755552b3a.3 for ; Mon, 07 Sep 2026 20:04:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836643; x=1789441443; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vjEcqTpfVMGQsEndFVux5pizoY7o64l8tWT5aAqDG0w=; b=T5h6JRynust9dNLK5JSxUbPtE9KuYNyd3wBn8HlofuZitgVAtzkx+ofwGmO82bEca7 ODG7bM668F+fp0Wy87GQwkQ8dbYqIF+oyXPWXRHW9ZHhc6tl/IWZWNjnl/prdxWQQxDD rQIXTpLXLLtNW4WoaAITeeHL5h7kebPe+8fHB0jAqS8XMsGScGPoKzFigxkU1+2y6f55 /NvQLFbqsZtWB0mGtNNhF67E01u18ql8Ye4MZCR94KwflM7kqPjdDlNbs+HTfrwfLp3p qABPBTFUoWBwTmjtzgvv8Q/n2G1ouYqoLM6EmPcPiPRbwP+rcYzo44kUfy453oaKF70S D2eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836643; x=1789441443; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=vjEcqTpfVMGQsEndFVux5pizoY7o64l8tWT5aAqDG0w=; b=tG+Nq02heAt9fmPpdtGC1ugPd67vvGnrxPQkr9Eiw85tQ2Xwy7e5hUaeSFHyliUwVy G2SAFG2mGRMDSac51IGU8P+xuWey1POvZpZRi9cM5DnNt4NIJEgmy55MJdODSkQyVOpJ 3DK/6YbZHPFXwx2Zrmw7GWYZ7l/wp/C7jNrnvu1Tcf+6iS3MJK5I1yHT4kwmBXklLXEp c/pebdSG2+GIKK//84IQL7KTKkbs+4gqrYasEbjv3qQHr+tRKapH30fUHm6M6aC3zlRw CC7V/jHOBgOFoLuF3XbramRjfy3L+vei6egxbkUNRYOez+ozM4OL6PsYsN/6Nu/lZLiB tttQ== X-Forwarded-Encrypted: i=1; AKwUvByhZKuf4SD/n3XAp80nQ7OsOdfYM5mKQCMDvFsod7mXAN3ZVdDQ4A8fNoeWS6KQqa8QsDoVTTTrD9PQtHY=@vger.kernel.org X-Gm-Message-State: AFuF++kFfFL4pwFUNhfp6gI1sA9OIIHtYpYkJ5X7Ea2/4UTYLwBS8V2Y heenIMeqp7+IOQc98EShkO5CwPLgTcv3U5q2LuD71KQX0GEb057KmymxtGB6B2IS7Kl+OeeNshT 3XuGc X-Gm-Gg: AYBFou0Foz1a+22/ZiZvlj+ilgndk3A5uyA+mbMQpoFweduNpfZpIty898A2M/x4e27 8iZXrC0uu+1rRSfMDf1HKpYABtaW03D8+j40geWzg1WeAI/bbl8lp6oZZ2fpIrrVmSbjFByYXEP j+mST+2bdkt2wjy9f2wkZmLI/kbBwBY4KPVWH0O6SewBzitnAPGcxWL9yOuFLdoQ3x0A4hJA1Fa rJrscXkhjezstaa2C2jlsnVaouzAdWG6Ki1S0lxF2ft9+k3Cd8clw3FMKcW+nFEhAYIDCd6XTAz /2vJ7XP5LgboAf0/K9dK7eOg1qGCGkX5+CERvFAMO8DjFAABrN5BmoL9pNcE/RkMUSoWftynVZN SH8oHCeTv/xWGNwdNrji0MAZJY0uw6JdYUD+b9BCOClCcrGZ4sUSaJ/09QdBXF//D/wPzZHLoyH kh4aDwliSKelUQ/y4Hh6XYb5PoH0ayFDGF9Zg4jRf8PZqxEMVCyNzUqj7tvKmqREh0ZveolfHiZ cePGwrPcwx8WvEnr8brwymR X-Received: by 2002:a05:6a00:808:b0:84e:d5e6:a085 with SMTP id d2e1a72fcca58-86164adae77mr35546667b3a.0.1788836643380; Mon, 07 Sep 2026 20:04:03 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.03.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:02 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 01/11] mm/sparse-vmemmap: introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION Date: Tue, 8 Sep 2026 11:03:25 +0800 Message-ID: <20260908030335.96549-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The section-based vmemmap optimization infrastructure is still guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, but it also can be used by device DAX. Introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION as a common config for the shared infrastructure. Select the new option from HUGETLB_PAGE_OPTIMIZE_VMEMMAP and from DEV_DAX when the architecture opts in to DAX vmemmap optimization, and use it to guard the generic sparse-vmemmap state and helpers. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v2: - Fix SPARSEMEM_VMEMMAP_OPTIMIZATION being selected without SPARSEMEM_VMEMM= AP reported by Sashiko. - Add an explicit DEV_DAX dependency on ZONE_DEVICE - Collect Acked-by from Qi Zheng --- arch/x86/entry/vdso/vdso32/fake_32bit_build.h | 2 +- drivers/dax/Kconfig | 2 ++ fs/Kconfig | 1 + include/linux/mm.h | 3 +++ include/linux/mmzone.h | 13 +++++++------ include/linux/page-flags.h | 5 ++--- mm/Kconfig | 4 ++++ mm/sparse.h | 4 ++-- 8 files changed, 22 insertions(+), 12 deletions(-) diff --git a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h b/arch/x86/entry= /vdso/vdso32/fake_32bit_build.h index bc3e549795c3..5f8424eade2b 100644 --- a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h +++ b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h @@ -11,7 +11,7 @@ #undef CONFIG_PGTABLE_LEVELS #undef CONFIG_ILLEGAL_POINTER_VALUE #undef CONFIG_SPARSEMEM_VMEMMAP -#undef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#undef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION #undef CONFIG_NR_CPUS #undef CONFIG_PARAVIRT_XXL =20 diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index 602f9a0839a9..6250954b0fa7 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -8,6 +8,8 @@ if DAX config DEV_DAX tristate "Device DAX: direct access mapping device" depends on TRANSPARENT_HUGEPAGE + depends on ZONE_DEVICE + select SPARSEMEM_VMEMMAP_OPTIMIZATION if ARCH_WANT_OPTIMIZE_DAX_VMEMMAP help Support raw access to differentiated (persistence, bandwidth, latency...) memory via an mmap(2) capable character diff --git a/fs/Kconfig b/fs/Kconfig index d1c210c6508f..9b32ce79cc80 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,6 +278,7 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP + select SPARSEMEM_VMEMMAP_OPTIMIZATION =20 config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mm.h b/include/linux/mm.h index c49ef99b4413..a2ebe87e7654 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5175,6 +5175,9 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, unsigned long nr_pages; unsigned long nr_vmemmap_pages; =20 + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) + return false; + if (!pgmap || !is_power_of_2(sizeof(struct page))) return false; =20 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c9ae7991a8b2..e9b54ea0eff0 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -102,9 +102,9 @@ * * HVO which is only active if the size of struct page is a power of 2. */ -#define MAX_FOLIO_VMEMMAP_ALIGN \ - (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && \ - is_power_of_2(sizeof(struct page)) ? \ +#define MAX_FOLIO_VMEMMAP_ALIGN \ + (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION) && \ + is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 /* The number of retained vmemmap pages with HVO enabled. */ @@ -116,7 +116,8 @@ #define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1) #define VMEMMAP_OPTIMIZATION_NR_ORDERS \ - (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS = : 0) + ((__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 && \ + IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) ? __VMEMMAP_OPTIMIZA= TION_NR_ORDERS : 0) =20 enum migratetype { MIGRATE_UNMOVABLE, @@ -1155,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; @@ -2019,7 +2020,7 @@ struct mem_section { unsigned long section_mem_map; =20 struct mem_section_usage *usage; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION /* * Normally, sections hold regular (order-0) pages. However, for * sections with HVO enabled, this tracks the compound page order diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 86dd0470da11..462e89e05548 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -208,14 +208,13 @@ enum pageflags { static __always_inline bool compound_info_has_mask(void) { /* - * Limit mask usage to HugeTLB vmemmap optimization (HVO) where it - * makes a difference. + * Limit mask usage to HVO where it makes a difference. * * The approach with mask would work in the wider set of conditions, * but it requires validating that struct pages are naturally aligned * for all orders up to the MAX_FOLIO_ORDER, which can be tricky. */ - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) return false; =20 return is_power_of_2(sizeof(struct page)); diff --git a/mm/Kconfig b/mm/Kconfig index 79163b7d795a..288b32dcc350 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -461,6 +461,10 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. =20 +config SPARSEMEM_VMEMMAP_OPTIMIZATION + bool + depends on SPARSEMEM_VMEMMAP + # # Select this config option from the architecture Kconfig, if it is prefer= red # to enable the feature of HugeTLB/dev_dax vmemmap optimization. diff --git a/mm/sparse.h b/mm/sparse.h index 049272aba84e..b408d15baf7b 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -10,7 +10,7 @@ =20 #include =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION static inline unsigned int section_order(const struct mem_section *section) { return section->order; @@ -72,7 +72,7 @@ static inline bool vmemmap_optimizable_pfn(unsigned long = pfn) =20 static inline bool vmemmap_optimizable_order(unsigned int order) { - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) return false; =20 if (!is_power_of_2(sizeof(struct page))) --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D01D3093D8 for ; Tue, 8 Sep 2026 03:04:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836653; cv=none; b=V1RRK5ccE6/XdgZzt6V6eYSb/4HhTICoB+XtFNtrI5BBkrAIKo5iVlwSRWyOqCnaETqc0rW1/6KxIsZsGBOImSk22I78Ca1bnb9uGtyhypGi1S1xvmwDo7dVIpnMhJ+B4N1j1ZEmHBZ63RwYMxStca9zMLd+8y0SDzt+FImdEzo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836653; c=relaxed/simple; bh=uFyg7N82s0QYKej53JA1qkjv6Rx4vM6gC6EvxnKAKxY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qL95VdV6dVjPJFC51Fi4D4aErjZSCHyI4tBw7aZeTKwX9rzYL56J9XS3HU35gIuC8e4k2kE7YXq18/Z27wZTMvq0x7ORU0eJXFPgjH1s/5VHRPybVLfvK4STk3i4e+E3czZ/a/hIdR3wloi2akdZH8/HqIYqCem6cLcscQ93uK0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aISbZEAr; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aISbZEAr" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-8557c3f270eso2196817b3a.3 for ; Mon, 07 Sep 2026 20:04:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836648; x=1789441448; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8ftmCWznJ9sK2TfrzbteNQ0qTXPtrb89VSG1QYG+6OE=; b=aISbZEAr9Q2ig8oma6YE4dFNas3WwZjrN+QKU8EZYhOjRwdySWiuPMJr+V8LiwUD6O ltaM0ynAFr2imSy+oVg7lSf8EcmV3LofG+DlmjakLWvwChgdu26dKdUoA0IUN35obFgz R14j2kRbpFIqBC4NCp8dpnTt+7ZeKGCJCJnhuqz30JuKyIEtYNAW55hO4L3wIbgFqfkx z4wNP/LSOsxi38h87jQT1zTbntz21E/is/o3PpQt713Ip0r8u3xVOVeajmqrnUOb+jfy saiWAipRgHTNaHjURkFvDtpn1bJ70FOyrrJjdXYAhciIarmcDH9/t5iWSO6zdi1TQktg 1CTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836648; x=1789441448; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=8ftmCWznJ9sK2TfrzbteNQ0qTXPtrb89VSG1QYG+6OE=; b=PxBobnBXbeVL+8La89Mg4s6ufz/idnKXygAJkyB7tfAvyhXD18vPf3V710jCJXe9CC TpNdUN3Mp9K0kTFYa0gwVbrXHHeR5wf/z79/6Mjl93WPL+AiNItXKvRjUBjOauhSsC+d qR03RajRKVpwClcTOJ5maCFPVvP/V/1Wz/7icNvUXCmHHljye7fQGsuN27f2xQ5WTVDt n/mFMoWCFPGZMpC7deUVcEjkxY3kpT0fXudKvbgHyoze8sHTsLEVPDfMOa2FF+QyrEe1 vJkNYOhv5a1psVhE/dhOf0lxT/Qg9LWJ0wPID00ZBysWDSXNBXKQu2C/DVwXbCTVGGNK Dmpg== X-Forwarded-Encrypted: i=1; AKwUvByDZSyPci4ATVBuWgU6keMnpp3nqqnb67cFEaphXLPPlYTgRhL0DHj+cZjfJzJI5MTWMDhhWaldRdjs3Oo=@vger.kernel.org X-Gm-Message-State: AFuF++ntA+d6gMuwoKusbyE+7IR7F6ckg7rPnb8u9Yy3eFcu1Eo3c+8t lHbbRvNZsfRP4JVOLJ6Ri2eJGH2Lg5n3NosjieWO9ORsqfmcTxWngYtgbzvfoQBUIBg= X-Gm-Gg: AYBFou1uEPTN0yaBXdClmyLciZSeRPh+kONAOo0PsS008t26H6oTdpDCexXlryOhtiH kHPS9RTTISqe07CCFtIL2zCTlmabJMrgps0FrdvgNuN3FNEhdhY72U3VLaPKlqlimFqhMrj7AfO bQHxJaIFrGRXuIXgPyUixqIf5oCZpILKTnAWaj5xwhbEDDg9ieSesvfzchbGPcUZpXhffzLlvEe jSFs0SaNFnPSWYAej8XJSoiRio7d3oG/8AM45s2D2VOPj4tcZa5ixcCEVi4BM/qt4HkBQvRC85t yVcdWnUW7VLIRI4DB+WtvHzvjOMkN8Y1RCKGpn4Tou06IpSd2RSrOzRiTF/U16V5ZtFnIZIhqJo sWhiWuaIeny/j+5rodmSsitl94djPAcI13eCeIRpIAZmbJ5BOSCspOVqmtikw6OU/Ik2zFXVU58 JQqypiMMTwq0un8+0+RWoh0daZcwTmjt7lnk0q1WEjih1zld9uPP6Ih8w07TET578IHTUoW9zVK BnIOC5957WLT5ngXRkvbyuo X-Received: by 2002:a05:6a00:98b:b0:842:2419:6c0b with SMTP id d2e1a72fcca58-86169c71e5cmr35224717b3a.10.1788836647750; Mon, 07 Sep 2026 20:04:07 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:07 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 02/11] mm/sparse-vmemmap: factor out shared vmemmap tail page allocation Date: Tue, 8 Sep 2026 11:03:26 +0800 Message-ID: <20260908030335.96549-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB and sparse-vmemmap each have their own helper to allocate the shared vmemmap tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper. It allocates the page through vmemmap_alloc_block(), and uses cmpxchg() to install the per-zone shared page. Expose zone->vmemmap_tails under CONFIG_SPARSEMEM_VMEMMAP to match the shared helper's build condition. This avoids a !CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION stub; when optimization is disabled, the array has no entries and the compiler folds away the unused paths, so no storage or runtime overhead is added. This removes duplicate allocation logic while still handling both the early boot and runtime paths through the same helper. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v2: - Collect Acked-by from Qi Zheng --- include/linux/mmzone.h | 2 +- mm/hugetlb_vmemmap.c | 28 +--------------- mm/sparse-vmemmap.c | 74 +++++++++++++++++------------------------- mm/sparse.h | 1 + 4 files changed, 33 insertions(+), 72 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index e9b54ea0eff0..d3778ba976a5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1156,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +#ifdef CONFIG_SPARSEMEM_VMEMMAP struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index eb339c4a71f4..4a57e6c3352c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -493,32 +493,6 @@ static bool vmemmap_should_optimize_folio(const struct= hstate *h, struct folio * return true; } =20 -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - struct page *tail, *p; - int node =3D zone_to_nid(zone); - - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail =3D alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p =3D page_to_virt(tail); - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -535,7 +509,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struc= t hstate *h, return ret; =20 nid =3D folio_nid(folio); - vmemmap_tail =3D vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail =3D vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e62e6aa07f12..70143dd8b579 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -42,27 +42,13 @@ #include "mm_init.h" #include "sparse.h" =20 -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask =3D GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order =3D get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; =20 page =3D alloc_pages_node(node, gfp_mask, order); @@ -76,8 +62,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, = int node) } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } =20 static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -184,39 +169,40 @@ static void * __meminit vmemmap_alloc_block_zero(unsi= gned long size, int node) return p; } =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zon= e *zone) { - struct page *p, *tail; - unsigned int idx; - int node =3D zone_to_nid(zone); + void *addr; + struct page *page; + const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; =20 - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >=3D ARRAY_SIZE(zone->vmemmap_tails))) return NULL; =20 - idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail =3D zone->vmemmap_tails[idx]; - if (tail) - return tail; - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + if (likely(page)) + return page; + + addr =3D vmemmap_alloc_block(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); =20 - tail =3D virt_to_page(p); - zone->vmemmap_tails[idx] =3D tail; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page =3D (struct page *)addr + i; + mm_zero_struct_page(page); + init_compound_tail(page, NULL, order, zone); + } =20 - return tail; -} -#else -static inline struct page *vmemmap_get_tail(unsigned int order, struct zon= e *zone) -{ - return NULL; + page =3D virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) !=3D NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(addr, PAGE_SIZE); + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + } + + return page; } -#endif =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct vmem_altmap *altmap) @@ -229,7 +215,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long = pfn, int node, return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); - page =3D vmemmap_get_tail(order, zone); + page =3D vmemmap_shared_tail_page(order, zone); if (!page) return NULL; =20 diff --git a/mm/sparse.h b/mm/sparse.h index b408d15baf7b..59b825df83b9 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -139,6 +139,7 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0EEB30D41D for ; Tue, 8 Sep 2026 03:04:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836656; cv=none; b=cMKhhsKfiwfN3YR0GWE8VOqTPAhAUOPma4T4IIp2PXc9Wuv3EeQcE8sPbwDFJDZlCElbHrajU54+lLRn3Jsc2KebgrlsDosfTc24YPJHWocPdm7ZVUdSEaXurTjmffPYaWa1UyPkZ28UhopMRixeMzi1jS9UzVNcsd3t8pCTweA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836656; c=relaxed/simple; bh=Spi40rx4E2u0S6TcpQdxQK6wcnBHvGkB92O2rN9o6oA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LmWiwBviNE6/DHulWLG54lXXkBihTOS5r0ixKV1HMHcdtWMyy1Bl7IXqNjh+hYlrsiLRm51Lzx9y9OKQ2U7J8ellSvVsiQq/x2APApUull4KR4fC03BXSJ2XRRyQpwZeysJ8rxSYT6fO4vYFLaNP6NChe/ZgsPhOC9HckWDb+60= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=GgA/dNcV; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="GgA/dNcV" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-8525efa7274so2750559b3a.2 for ; Mon, 07 Sep 2026 20:04:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836653; x=1789441453; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=w9QdH6WsgivxpcgrN4flEbJh0Z1iEZy30Xh6LMeUuak=; b=GgA/dNcVPjA/kt8Ca9lYm/Cl9ybINBvvzF4A/qxaTaay8Je+Mnzm3IX6zMyQ6PxZHh nR8nXPL0a+gdFLDKO68V203+qtknAueHptSNF0C/vJQQLrAPbtHnZAcIXltOpIaFdVyL q2pNJ+ujZiZuhxmFFiS7HTFNvsQn5WVYTTPJ6DAnhYwvCYdrCqC0B4acKVn1HH4yK3fI zWjlP+QDh5oVbeLo1oHQiWUspwNvKQBRg6ip2t41Rs8Y1VRZ3DmnsuTA7J8a0jWnR6V6 jGAVJ7TMDFNBu5bSmQZ6UgAsO3z1rQLypv4lYqvALjhnsGBMGYWZO+2M1gOA+0ckb4B2 VTPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836653; x=1789441453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=w9QdH6WsgivxpcgrN4flEbJh0Z1iEZy30Xh6LMeUuak=; b=elN8/We95gW40VlfJAG4tVJ9dQnEqHHaRuOI415jTLeKiUPagvwY7jnEkBXJsOBntz gkAO/AMF0nQdqlLntw2M2JiyyIYlx6ZbHHYcbOMve8u8mX272kr2U3N0XRJNtJINlLq6 mzD7D7BXgk9rcJz0yelIHVHEI/A+qR5u7oN5myl4e0aeKVv/RsG+ptgSFP8bP4CCY6Mk u+coJuWxldTzx4DYYMKTDEteITdJus1jE9ddSzDsZx9qLBfSDR4ekqZ9TQ0bgo+8WnmK +9j3yV7QCc/IIlW3LuY6M19mggVT+vX5OB8t19GO8VkDJipo+E3n8SA35N99QeDt39GA YB8g== X-Forwarded-Encrypted: i=1; AKwUvBwJSaScTC0N1fb88sME3thOJrudznW1lC8Sm2Qfkyu+HeKBrSMQQ3kIGgX3h2IFx4sL7biouMFAri4gkx4=@vger.kernel.org X-Gm-Message-State: AFuF++k9J+EIiJfXBsVSS8vJYFhgqiKLbFZ/ssjNBsBgS1B0lG0ngbZD lkgjbt2Sl3Ddd4nCWotX4dpZNl3cS+Pos/jn0MOEdXY/H2F9ko2qIIpcIbr5lTsL+6Y= X-Gm-Gg: AYBFou1fBVqhg9H6mzI26pQ5eoZev+rxo/BXLvp+jcqoCFK8mP3gituQsYYrWaDX9E3 KKZamr198A3nMY9hj7uuXGSGTnL2LtZiZcm25bkkDjSMTLmvZ0bdh1qmmb7C+wq3KEVvewH1VGX PgsniPeXU9z48pv0j4jdOIx6LQPu8KX6ongenuzEQxYSNgX5sWTHEE6QIR4xs8HWURrZm2FAWnW FGLhkjSLy2Ujtr8YdeMStBvTAxbnpTVwitCT3pdQduv4kiK3LRClYOm8ns39x958qd+JVWxRUju dIRDpVU1o7svXNn2GbSLW4iTfonTJKqShD1Wom5Jb5kx03caA/9I/9qQpqD+qhSLFO4lcFLBnRL hiLoI2EEx8z+mH60YVmE3okmiBHUMCYaA0cP74VeyeiZ42bv/hZrzmyryp74tWER3HYzriv0Wyt me6do7ReHuUmFG9maQj5U56Sg509ZaNaRBwN4X/Le80GyrBPBbwc9urxiqULlxfx/GtCf9fQyCX ZpR8tioeGpGuRR1Fgh4D8CI X-Received: by 2002:a05:6a00:3021:b0:857:727c:a1f3 with SMTP id d2e1a72fcca58-8616907fefamr35713322b3a.21.1788836652665; Mon, 07 Sep 2026 20:04:12 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.08 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:11 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 03/11] mm/sparse-vmemmap: open-code init_compound_tail() Date: Tue, 8 Sep 2026 11:03:27 +0800 Message-ID: <20260908030335.96549-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" init_compound_tail() is only used by vmemmap_shared_tail_page(), where the shared tail page setup intentionally passes NULL as the compound head. Keeping this helper in mm/internal.h exposes that special case to the rest of the MM code and can make the NULL head argument look generally valid. Open-code the initialization at the only call site so the special-case use stays local to sparse vmemmap optimization. No functional change intended. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: David Hildenbrand (Arm) --- v2: - Collect Acked-by from Qi Zheng --- mm/internal.h | 9 --------- mm/sparse-vmemmap.c | 5 ++++- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index da14c56fb24e..0dca33db068f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -786,15 +786,6 @@ static inline void prep_compound_tail(struct page *tai= l, VM_WARN_ON_ONCE(tail->private); } =20 -static inline void init_compound_tail(struct page *tail, - const struct page *head, unsigned int order, struct zone *zone) -{ - atomic_set(&tail->_mapcount, -1); - set_page_node(tail, zone_to_nid(zone)); - set_page_zone(tail, zone_idx(zone)); - prep_compound_tail(tail, head, order); -} - #if defined CONFIG_COMPACTION || defined CONFIG_CMA =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 70143dd8b579..e453ce4675a0 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -189,7 +189,10 @@ struct page __ref *vmemmap_shared_tail_page(unsigned i= nt order, struct zone *zon for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { page =3D (struct page *)addr + i; mm_zero_struct_page(page); - init_compound_tail(page, NULL, order, zone); + atomic_set(&page->_mapcount, -1); + set_page_node(page, zone_to_nid(zone)); + set_page_zone(page, zone_idx(zone)); + prep_compound_tail(page, NULL, order); } =20 page =3D virt_to_page(addr); --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pz2-f12.google.com (mail-pz2-f12.google.com [74.125.228.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54DDA30F92D for ; Tue, 8 Sep 2026 03:04:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.228.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836660; cv=none; b=hMg8M6dHqTTLir5EWaIhmdXYlJ0q3jp5X+a6gnGh+eVB5Y+yLuaw5gJSIV8AklDdOo2QTAKRPhH9gsXwtBYgFy3E8+Mdly+iS6ziV9euwBdOoE+JHbuYv0xLMT2TB47bEiVv6tM0n+h4cowjEc+9CsMXqP9YU/kORcDZ3gYsGXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836660; c=relaxed/simple; bh=yR8q1n1I8bNxoF0boV+yOULjBVMjjKAcPkyFRNw1MXI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AF5KF9F8eBGFeGVoivu2hikz1S9zybnvHed4CkltFZfq21uIhuTVhT2owKnDUBIBSn3mAJ31/VwWTogdNOt1jb8+CYvTtMYoJdhNwqbhjfHpkwC6BelcvZPwfKiBwRvcPfrNi3YIyEtae55waK/iz5KSTfjYPcVuYDoFiJcCnfM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=b8U0RpLL; arc=none smtp.client-ip=74.125.228.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="b8U0RpLL" Received: by mail-pz2-f12.google.com with SMTP id d2e1a72fcca58-85469e2254dso66642b3a.1 for ; Mon, 07 Sep 2026 20:04:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836658; x=1789441458; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=IUXKrxViB50EGCJ21IU1wh3SHWR1crLUNz3m8pEDEC8=; b=b8U0RpLLsH8UO+M+fWxmVl1cyEmSNsJbSmEspC8z4xq/KnTc/lgaeyPx6Elb6IKp8M TKPLaXVFXO3MmsLBHLCGf6X/rofDROFIetUZRNcfJk2VQRWSoNmIPnYHzMpNAM3wjho/ p7A7eV/O41VwZSoHhImRkwuGOGhxEbmGedJBXT41CaaYiq65v/SVK/D6H9E+hchYkoqN /UawtfWnZYxL4Pf4RRbEUpCNBYjDQnfdt9dxirkZ49eRyJ4qcDhB5CB/IWOC4sdqVg4V Csa8KpfDxbfCa9fz+Z+/O/lbjNiTrYgBJO5kQctis29bzrl3Vf5hCtWqCcm2lDGjwS+g HJSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836658; x=1789441458; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=IUXKrxViB50EGCJ21IU1wh3SHWR1crLUNz3m8pEDEC8=; b=KRq9WZr/iaTM0qBLx5HXgravnPgTdOKTo2j9tVyxFEZ2kMz3vC/Gn3r+WD8fPnUfzI 7HMk1cPDHKVMwBk633zDhzymTL0uEAOWBKd+mLBhtS950uGJic5teIJ+vKk2bCnUHUy/ caGbttlplWPLCqJQcOZzDIBEgunrH3PRxgrh0YC8JXB6a0+426kp4KGTnx4o9ssrzMiV 3qLbqdVRGrJuE0SDiHEuFFhHc4yu4UCiAMDHKiO132caYtPEkCF29VFyPLTuV5MD6rQ7 ARpRsv8tRTu0+QG3DPvOD0fZUlRBMFMDmEcCNGnTSidhg0z7QKERFtnDJf4ScyBpWV3N 5Qxg== X-Forwarded-Encrypted: i=1; AKwUvBwbej1HM7yV/9VwBmsQ1v1JFhP72v7/grmd3F7ZcTj3m9r4zRxWgpMyJyK3ZozjfoNHFFYvsdSoZHA0Sec=@vger.kernel.org X-Gm-Message-State: AFuF++mIfjpzB3VB5L9kPan/z89pUA+Ebutie5FMthjKUebAPENm+RZc /7s295L0AOijslpkycYyTDSc7x9LRNYcIZkb/4Cvlt/mlAFi14mnJ19+gidKe4aSuyI= X-Gm-Gg: AYBFou24LKzyv3OuknA8ihTou0jXuvI45WdS3kpHKv5JTpXhZGwnPqUgK30mvtVqLtj qRUvrDas5HLKBGKLUJOEGIRTIUm+nOBMEN668A0QAws2HssiXNN1JjyFzZ9+6/ldEcjil7Om/q1 TO/kpBXKAur+dpHhMrk8GHmya/agFVAR33Wa/bprmbvVMQVh1r00G/38t2P0FzExhuMNjuRMBqF cOhBzwnWqmVSpg8yvvL0rjRH7GHnSgxpICr3cOio2Nal5MB3bWMFWJoR6IRIIjJptpNeX/d2K8N laty7/Y0m+DiawmWjf30FVc8GvS5j8rBR29tlj/CDG9AvM6R8fNEv9r6YCtrOb4/Dw1m5TE0DMT 0S1YmJ3mGFBU+uSYR35ARDmmGuYWtJP4MAAiRdFURzmBEKazImWDGLuBuriHeWMYVJ3Z8QqIA7n oybJsw3ZMs8wpGzj3kV+MS5sT9ZCU0FBXOBd72+Dpo9h0ox0fF5jfLrJR9l0UM/xlevfO9FIaV4 nNdy5mRYF7NzlC535eS/OA/ X-Received: by 2002:a05:6a00:1807:b0:862:bb07:2ba1 with SMTP id d2e1a72fcca58-866988d9942mr4431251b3a.4.1788836657402; Mon, 07 Sep 2026 20:04:17 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:16 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 04/11] mm/sparse-vmemmap: prepare DAX vmemmap population for section orders Date: Tue, 8 Sep 2026 11:03:28 +0800 Message-ID: <20260908030335.96549-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX still uses vmemmap_populate_compound_pages() to populate its compound-page vmemmap mappings. That helper allocates the head and first tail vmemmap pages explicitly, then reuses the first tail page for the remaining tail page mappings. Device DAX is being moved to the section-based vmemmap optimization infrastructure, but it cannot switch to the generic section-based population path yet. Once a later patch records the DAX compound-page geometry in the section order, DAX head and first-tail PFNs can look optimizable to the generic helpers as well. Add a DAX-specific population flag for this transition. It keeps DAX head/first-tail allocations on the normal vmemmap allocation path, while preserving the existing page reference for reused DAX tail mappings. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v2: - Collect Acked-by from Qi Zheng --- mm/sparse-vmemmap.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e453ce4675a0..54ae8c284324 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -35,8 +35,8 @@ /* * Flags for vmemmap_populate_range and friends. */ -/* Get a ref on the head page struct page, for ZONE_DEVICE compound pages = */ -#define VMEMMAP_POPULATE_PAGEREF 0x0001 +/* Vmemmap population for ZONE_DEVICE compound pages */ +#define VMEMMAP_POPULATE_DAX 0x0001 =20 #include "internal.h" #include "mm_init.h" @@ -208,13 +208,17 @@ struct page __ref *vmemmap_shared_tail_page(unsigned = int order, struct zone *zon } =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, unsigned long flags) { struct zone *zone; struct page *page; const unsigned int order =3D pfn_to_section_order(pfn); =20 - if (!vmemmap_optimizable_pfn(pfn)) + /* + * Device DAX still relies on vmemmap_populate_compound_pages() for + * head/first-tail allocation and tail-page reuse. + */ + if (!vmemmap_optimizable_pfn(pfn) || flags & VMEMMAP_POPULATE_DAX) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); @@ -236,7 +240,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in pte_t entry; =20 if (ptpfn =3D=3D (unsigned long)-1) { - void *p =3D vmemmap_alloc_pte(pfn, node, altmap); + void *p =3D vmemmap_alloc_pte(pfn, node, altmap, flags); =20 if (!p) return NULL; @@ -251,7 +255,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in * and through vmemmap_populate_compound_pages() when * slab is available. */ - if (flags & VMEMMAP_POPULATE_PAGEREF) + if (flags & VMEMMAP_POPULATE_DAX) get_page(pfn_to_page(ptpfn)); } entry =3D pfn_pte(ptpfn, PAGE_KERNEL); @@ -511,6 +515,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, unsigned long size, addr; pte_t *pte; int rc; + unsigned long flags =3D VMEMMAP_POPULATE_DAX; =20 if (reuse_compound_section(start_pfn, pgmap)) { pte =3D compound_section_tail_page(start); @@ -522,8 +527,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, * with just tail struct pages. */ return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); } =20 size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); @@ -531,13 +535,13 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, unsigned long next, last =3D addr + size; =20 /* Populate the head page vmemmap page */ - pte =3D vmemmap_populate_address(addr, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(addr, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 /* Populate the tail pages vmemmap page */ next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 @@ -547,8 +551,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); if (rc) return -ENOMEM; } --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D2F930C171 for ; Tue, 8 Sep 2026 03:04:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836664; cv=none; b=Y/rIqau5hauIUCNZ7TuY8+qp55eeQu017GerTMqTbMpgd/nacCDoOh/cHAjJmkw4slctQ9Fc7UxzeOoCaNVkgX7y9T7WTiArN8cV1A77dTMYz+yRoaRQMbG6pa19zi4ssDb8sQSVH3ugDfKnKdwR23D4ZYOQ/c4cFg0cwNQrxJI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836664; c=relaxed/simple; bh=B2ejOMr2ZHP5Xylq8iU8Nr9jZObNOXowzyIFQOvCedM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N8kyA+EInVKxyVyGotNaj7ftFgj85fjvWCUzUWFOoBBU3RoZ1Ov/RueEoQdTEubfSwdQv1CGe+8ayKJPPI4x4OItpDctSkTI5RisKawKh/I12sJcO2F/Y6GJFPrmefzHnwPQui1W+7PRapuA53uUaDbl+ReVrYuujsTQTTRhozg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=SAUE0jeM; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="SAUE0jeM" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-848643382fcso4199154b3a.1 for ; Mon, 07 Sep 2026 20:04:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836662; x=1789441462; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Nco6X6VSmg3b4sCsw2GCdeq7qTzf+r3pWDgUiD6K2qM=; b=SAUE0jeMhBegQIV/hr1vx+uWXUsE2gT+IAOzYZCAraKBIweUy3MsU8U3nQfQGewLPA Eovsm5KAjB+eM1OzSACAjxfjwOJfMUNSRVP9b4Ap0nMBTCHxJONphiKJ1PdVDMicknLM 0RUO8rUdCp1mnqu8XV6Nvjb25wElJxHq/PV4Id7iOlpx8g6bvGf2Lj/QboeBKbafF/rl zJjLxiJ+r0L8x5uGtxLNK/CR5n1mb1s6EzGkzmBpdU/VIU7LONrr9fj9cwmVCipN8wu2 RoOZXdWiOgO1sXOS8kDKkZ/r94cAWj+RVsaBkU3MKP8FSSdMkCIliRZY4yh6sZeeokvF FVXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836662; x=1789441462; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Nco6X6VSmg3b4sCsw2GCdeq7qTzf+r3pWDgUiD6K2qM=; b=MStU61xyUcCoaN940rYPMeSJ2eKOL+ybdNYb3CXe9gJ2fX11G5jepmavTXkiFz/1Ry GGE+G+sLFoKUStMPo11VHwPTYhgsg6MvtVhZFxaUxumqtUPByQvFEJQQ+hNf0e31d2c2 He132//V6Lg+h9JEWrAJRCBVasDAZPQO42CShFFUPVmqjqbpXV+XPcnfuUfGs5uhWVCh HqjJNnImHbjEh5zPzW3X/r5Kl5wP52h9wgxTJasFB+zDIDpVy3YySm6fkesCN/btCC/a b5kCmu3Hvpa2kZqTqqLNScX1H69HWVnKrOVbiWkgMudtOveMUveBRKFPcFK/2Wl4hTkH OnJw== X-Forwarded-Encrypted: i=1; AKwUvBxcvSnJnXDa9Ny5Ug/lp7Np2kE2YzB/RYvscJEu2o8+U0IJe+9s5Fcwn7Zp2NTjnjt02pQIUXZvHsjMY3k=@vger.kernel.org X-Gm-Message-State: AFuF++lUdsMmtCggPTPcIgFH9QPBj8iLuaz9jRN2jxl2JaIRHbxsA+9D /M3y9ducq8rFBzBEo5s3QtIKdHLUxLJ5GSPEmP0gRLZq9qgjCzjg4bnq0bK8eDY8MEU= X-Gm-Gg: AYBFou0LJNlFEHyvtvKZ2P7KxfzMCxqqNn16FLUpfvMLJVEOT8W4G3RYNrsDRJW6hWL FVH35lExrSqFImSuCRrE0uROosr7t1Y2cZRCAvJ+b3/mkv/2wJnGozBizN7LBP8rW/UOkEgzhdi dsOMCRadwK89Vi5OIAg7K8+zDS6jV3hEU1vcYD0VJSxG0SFCjq7Yn64gSnCTB3pah3M5SFUE5rp Mw9FcuGtQkqzfvjAkjAq2BHH9LlDJhlgxxs/DA107/VjeXk8W2vXxTCk2AZ7kG3JT9Qv4Gc3faF ebqWxl+g3SY3HwbNHw02FjGvHpTBHdN3yXebh7Wu2ObWUQ/PXOZ3tMFbsEfhezapz2GsAFLJysx g+LYYF4WufzZOJqNiYDkO0991GjSvmFaCYcdJAC7IksbqGNjPwvpCmvI6mjW1rbAMtzFaVlljMW cssFlYkA35mY4+cbdxDKPLnrNPQAu4Pd1HZl5KQqHJe9IkFIuykdtj/d0yfAUIUsBR2RGjmljL0 7MSfB5o1EHXfLP5javo6WyrAg== X-Received: by 2002:a05:6a00:4ac6:b0:847:7ffd:ce35 with SMTP id d2e1a72fcca58-86168992df1mr41275963b3a.8.1788836661718; Mon, 07 Sep 2026 20:04:21 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:21 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 05/11] mm/sparse-vmemmap: set section order for device DAX Date: Tue, 8 Sep 2026 11:03:29 +0800 Message-ID: <20260908030335.96549-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX can use vmemmap optimization only when a full section is populated with a compound-page geometry. Record that geometry in the section order before populating the section, so later vmemmap accounting and population decisions can use the section state directly. Clear the section order when the section becomes empty again. Also reject partial additions to a section that already has optimized vmemmap mappings. compound_nr_pages() determines how many struct pages to initialize with a section as the smallest granularity. A section therefore cannot safely mix optimized and ordinary vmemmap layouts. Partial additions continue to use ordinary vmemmap population, so they do not save vmemmap memory. Such additions are uncommon, and the lost saving is negligible. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v2: - Explain why optimized and ordinary layouts cannot share a section (suggested by Qi Zheng) - Collect Acked-by from Qi Zheng --- mm/mm_init.c | 15 +++++---------- mm/sparse-vmemmap.c | 16 ++++++++++++---- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 9e8ffd01b4f7..7a2e58d631c2 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1049,16 +1049,11 @@ static void zone_device_page_init_from_template(str= uct page *page, * of an altmap. See vmemmap_populate_compound_pages(). */ static inline unsigned long compound_nr_pages(unsigned long pfn, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - /* - * If DAX memory is hot-plugged into an unoccupied subsection - * of an early section, the unoptimized boot memmap is reused. - * See section_activate(). - */ - if (early_section(__pfn_to_section(pfn)) || - !vmemmap_can_optimize(altmap, pgmap)) + const struct mem_section *ms =3D __pfn_to_section(pfn); + + if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); @@ -1144,7 +1139,7 @@ void __ref memmap_init_zone_device(struct zone *zone, memcpy(&template, page, sizeof(*page)); if (pfns_per_compound !=3D 1) memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); pfn +=3D pfns_per_compound; =20 /* Initialize the remaining head pages from template. */ @@ -1160,7 +1155,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; =20 memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); } =20 pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 54ae8c284324..aed1e7429daa 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -135,14 +135,14 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms =3D __pfn_to_section(pfn); - const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); + const int order =3D section_order(ms); const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); =20 - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) + if (!section_vmemmap_optimizable(ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); =20 if (order < PFN_SECTION_SHIFT) { @@ -573,7 +573,7 @@ struct page * __meminit __populate_section_memmap(unsig= ned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; =20 - if (vmemmap_can_optimize(altmap, pgmap)) + if (pgmap && section_vmemmap_optimizable(__pfn_to_section(pfn))) r =3D vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r =3D vmemmap_populate(start, end, nid, altmap); @@ -792,8 +792,10 @@ static void section_deactivate(unsigned long pfn, unsi= gned long nr_pages, else if (memmap) free_map_bootmem(memmap); =20 - if (empty) + if (empty) { ms->section_mem_map =3D (unsigned long)NULL; + section_set_order(ms, 0); + } } =20 static struct page * __meminit section_activate(int nid, unsigned long pfn, @@ -803,8 +805,13 @@ static struct page * __meminit section_activate(int ni= d, unsigned long pfn, struct mem_section *ms =3D __pfn_to_section(pfn); struct mem_section_usage *usage =3D NULL; struct page *memmap; + unsigned int order; int rc; =20 + order =3D vmemmap_can_optimize(altmap, pgmap) ? pgmap->vmemmap_shift : 0; + if (nr_pages < PAGES_PER_SECTION && section_order(ms)) + return ERR_PTR(-ENOTSUPP); + if (!ms->usage) { usage =3D kzalloc(mem_section_usage_size(), GFP_KERNEL); if (!usage) @@ -830,6 +837,7 @@ static struct page * __meminit section_activate(int nid= , unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); =20 + section_set_order_range(pfn, nr_pages, order); memmap =3D populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap, pgmap); --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pz2-f12.google.com (mail-pz2-f12.google.com [74.125.228.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 172C0311C2D for ; Tue, 8 Sep 2026 03:04:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.228.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836670; cv=none; b=ef2mtUDtob1Ie+B1TlPip8KUS0Fbc1zeTADzg8Pg0PtEcqWtN6rX9H3ds9hMhDeinh7If3nUqCiNngWp+qTTH/ruWcm0+FKN+XIp9KRqspWu4zvnRbO+0rsoRaAv8YOQW1B40Poidt7nH7xOPL9ok7VqNaiyA7nojmYBeoqFG7g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836670; c=relaxed/simple; bh=6iR+I4+CRd6OErCQvwrzJO+09x/z03jI1dFumA0VSuw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KQaurSDRpNnvFDSdskXQqbL3jrNq0KZKjs2p0FvQmaSqwMVyxD3qAs/WO21i6Y9Oe34lI/SLHNbP0aiOGXW7rrGQhLhNTgFWgEcJH8tLFIAUpwS+Z8gkbQn4idsyyL02PNXKUt1gXAhxIGMi9OhplrfPXIq/m20aDQxaKORq1cM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ffhFmlrJ; arc=none smtp.client-ip=74.125.228.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ffhFmlrJ" Received: by mail-pz2-f12.google.com with SMTP id d2e1a72fcca58-85469e2254dso66657b3a.1 for ; Mon, 07 Sep 2026 20:04:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836666; x=1789441466; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uEGgQCKJvggw1Narb1WaslYNRsxnnnRwg14YCHWkvvU=; b=ffhFmlrJMMvyXs9uH9zMivqVlVdM+B+314G62SuVXMeHJHMSmZi2yg4+zpLBsYRmqf mkWXkvXUvV10wJbODubnWBa0ccd2gV2ptQ0ZqpX2AOlj8s4OhNPxcYxx5FrS7aAo7TEq Ak7bUm/f6082b0GTvBTETBdvyXo6vsktAGhFWLdksiGAkWnvyO4eco4AEo5/b8+7MGLD qtCJBw8oGAXpbOdASHtWSfVp8ZLxU/c9II+oIs48uRN1m+s+ctVCdzBe4XHS7garOfva D87ZruF2lrb10eWw4qeKqnzZKOgG/HqPVC48+4rAmLGi/NrL4GH/aR011u11paXRLVV2 0D7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836666; x=1789441466; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=uEGgQCKJvggw1Narb1WaslYNRsxnnnRwg14YCHWkvvU=; b=Xu+iK3BOw+KYMlM0u0MR3zZUGp7Sf0n0JJHy75opRj1T4/Inbibk0bg7bZAoruh3iO TSif9c1PEEhJKYSQsIeAe09FzoLM5T/W5ych0kVJld8+46TFXwcZPmLN+nJ5/3Y9cdkJ wZOT/yUjyEPacstj1ebumOiOCDe28zDABB6jr62Eke5hkvS0kmDLWAR/LSNY8V5JDDMY 3f4MyXTlr8yQOcq/8yRZS5jkrOjRtM29xTI1ceZZwGzkEEanjFeIBQeifqMEte2dntOE m+cAu4WaeqJQVxHwKkcrjMHxf1xhf7sRYcLZSHbK522DQtrzT+OYzcaj34/nnxsOkDuU y0Gg== X-Forwarded-Encrypted: i=1; AKwUvBy9mNjINHVuotCCKFbfHwcXnZ+5puGfedpqZsjPY2aV8NH+6QRyByAPuHBnDD3PpaXshCrecwWFJezmDD4=@vger.kernel.org X-Gm-Message-State: AFuF++k81fVNlz53if5NE7+pqCk4UHWETSwGBCSt9QqpPHsuyxaEw/hJ 6lrOVWutIemgaNomiKh7V3Z3L7irhR4RV6anm464Gc3EeE2XegPwFCBeGgqwiIjmlv0= X-Gm-Gg: AYBFou0e+98mvDcSb030amq3ke2nZUAd+sTh8OoB87iwsN5ASHudTDbU5DrwnO7fCL2 v8aSByqhAd9ZWAXkhm/fp9iYQXemcV+fTfOCRAn5lPTpVeJena8+gxSpttSDhoRV6zQAnRx+L3k CQhZjPsY/lkoD3TgIDkp7mBIIOrtOlt8pH4bHek8OtibUzm6pizUVtwkz2ywWlI9w17UahCLcOT WDGoLTp+ebgrCCaytHsq/U38vINOpjtDHHTU235wU8y0XgCJk37ZdGqJgVTZM5+3qZVX3mQN7hB JRoQGk2c9olmU1514ov2wpREk/E242jcKJMiPJ11yhemCeVhJwRXc02lzqyiezpfsP28xADB7v3 Lz9SXEoAq4t5pfXBlmCNnWeCCFm1sswJHWFeXhvPULfRlThf3UMDkNlDv8ejzaucOCrWVFZ9QLB i9BhveaXaCuD7vcs6mWjga+0bCebi3HuX6lcrdFdXK7XgME0HbHl4l6qXEbjFbBaTKjYtqAJHfD CYV7UNw4N+wcW6/xVkXhyH9A3AlOuLjJWE= X-Received: by 2002:a05:6a00:1d9a:b0:847:852f:c28a with SMTP id d2e1a72fcca58-86698ee5c1bmr4207907b3a.5.1788836666149; Mon, 07 Sep 2026 20:04:26 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:25 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 06/11] mm/sparse-vmemmap: switch device DAX to shared tail vmemmap pages Date: Tue, 8 Sep 2026 11:03:30 +0800 Message-ID: <20260908030335.96549-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB vmemmap optimization now uses per-zone shared tail vmemmap pages. Device DAX has not been switched to that mechanism yet. Switch device DAX to vmemmap_shared_tail_page() as well. This aligns DAX with HugeTLB by using the common per-zone shared tail vmemmap page. The optimization is enabled only for DEV-DAX through pgmap->vmemmap_shift, which is assigned when a DEV-DAX device is initialized. Unlike FS-DAX, DEV-DAX does not modify tail struct pages, so sharing them is safe. Since the shared tail page can now back ZONE_DEVICE vmemmap mappings, initialize its entries with PG_reserved for device zones. Also skip poisoning vmemmap-optimizable sections while their struct pages may be shared. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v2: - Explain why sharing tail vmemmap pages is safe for DEV-DAX (suggested by Qi Zheng) --- include/linux/mmzone.h | 10 +++++++++ mm/memory_hotplug.c | 5 +++-- mm/sparse-vmemmap.c | 47 ++++++++++++++---------------------------- 3 files changed, 28 insertions(+), 34 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d3778ba976a5..03ed9232f186 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1686,11 +1686,21 @@ static inline bool zone_is_zone_device(const struct= zone *zone) { return zone_idx(zone) =3D=3D ZONE_DEVICE; } + +static inline struct zone *device_zone(int nid) +{ + return &NODE_DATA(nid)->node_zones[ZONE_DEVICE]; +} #else static inline bool zone_is_zone_device(const struct zone *zone) { return false; } + +static inline struct zone *device_zone(int nid) +{ + return NULL; +} #endif =20 /* diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index b428da66d279..0db0379826df 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -554,8 +554,9 @@ void remove_pfn_range_from_zone(struct zone *zone, /* Select all remaining pages up to the next section boundary */ cur_nr_pages =3D min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - page_init_poison(pfn_to_page(pfn), - sizeof(struct page) * cur_nr_pages); + if (!section_vmemmap_optimizable(__pfn_to_section(pfn))) + page_init_poison(pfn_to_page(pfn), + sizeof(struct page) * cur_nr_pages); } =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aed1e7429daa..0201877a7f80 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -193,6 +193,8 @@ struct page __ref *vmemmap_shared_tail_page(unsigned in= t order, struct zone *zon set_page_node(page, zone_to_nid(zone)); set_page_zone(page, zone_idx(zone)); prep_compound_tail(page, NULL, order); + if (zone_is_zone_device(zone)) + __SetPageReserved(page); } =20 page =3D virt_to_page(addr); @@ -490,23 +492,6 @@ static bool __meminit reuse_compound_section(unsigned = long start_pfn, return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; } =20 -static pte_t * __meminit compound_section_tail_page(unsigned long addr) -{ - pte_t *pte; - - addr -=3D PAGE_SIZE; - - /* - * Assuming sections are populated sequentially, the previous section's - * page data can be reused. - */ - pte =3D pte_offset_kernel(pmd_off_k(addr), addr); - if (!pte) - return NULL; - - return pte; -} - static int __meminit vmemmap_populate_compound_pages(unsigned long start_p= fn, unsigned long start, unsigned long end, int node, @@ -516,21 +501,18 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, pte_t *pte; int rc; unsigned long flags =3D VMEMMAP_POPULATE_DAX; + struct page *page; + unsigned int order =3D pfn_to_section_order(start_pfn); =20 - if (reuse_compound_section(start_pfn, pgmap)) { - pte =3D compound_section_tail_page(start); - if (!pte) - return -ENOMEM; + page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!page) + return -ENOMEM; =20 - /* - * Reuse the page that was populated in the prior iteration - * with just tail struct pages. - */ + if (reuse_compound_section(start_pfn, pgmap)) return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), flags); - } + page_to_pfn(page), flags); =20 - size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); + size =3D min(end - start, (1UL << order) * sizeof(struct page)); for (addr =3D start; addr < end; addr +=3D size) { unsigned long next, last =3D addr + size; =20 @@ -546,12 +528,12 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, return -ENOMEM; =20 /* - * Reuse the previous page for the rest of tail pages + * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), flags); + page_to_pfn(page), flags); if (rc) return -ENOMEM; } @@ -883,13 +865,14 @@ int __meminit sparse_add_section(int nid, unsigned lo= ng start_pfn, if (IS_ERR(memmap)) return PTR_ERR(memmap); =20 + ms =3D __nr_to_section(section_nr); /* * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - page_init_poison(memmap, sizeof(struct page) * nr_pages); + if (!section_vmemmap_optimizable(ms)) + page_init_poison(memmap, sizeof(struct page) * nr_pages); =20 - ms =3D __nr_to_section(section_nr); __section_mark_present(ms, section_nr); =20 /* Align memmap to section boundary in the subsection case */ --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC4CD30E0FE for ; Tue, 8 Sep 2026 03:04:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836673; cv=none; b=emOpfW0pBlV/++1tQcAdEcrNvBoAGyPbe7wxjcaiOBtVKAuozlSvdZ00DbflbcxUAZACoDONF/rAHx3zGsB8yES/YPFfXb3u6yrbmSgO0+O1HpxeaXiotkHt6RcW/j+O97tV2yV1hdTxOlk6WXGi4YFRyXuXhVC4LDW+ueKUYsQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836673; c=relaxed/simple; bh=dRLbFzgpgZgCYflbXCNRkhPaT+5HR8dXEGt9Utc4VhY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o1VF9SwaFYjwGpqdmym2nG5p/yVjatNHozc2EbWceq6WAIbneU/bmmiQu+3PJJF/8pjcFU9XP4H512yj9TKhpeGz8yI4+kkwHuZyYZbkX15jR4cqmiytMjv5k4DEgznixvWIaMhrmFyHKmcQSlXZjKaql+I5zzhLpkShVrhxNwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=TE//ASuY; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="TE//ASuY" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-cc1c9879395so2733405a12.1 for ; Mon, 07 Sep 2026 20:04:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836671; x=1789441471; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=CVT6j1P3zfNYXYnEjgM8kLQiWtuO+wz13pOr0ZtKdew=; b=TE//ASuYL2Y3Dff+qgLg9hcSpCfO9snNmNKd4D0FGpyNSGxSUKE8nc1/OGnt7/RRE8 FQpPa7BHQ6ZrJrmc0ArW+eEb/jizJ0EM6LG7/461IzCcKhUIaOsahhdnqld/ZT9N44qS f7N9b8ZqZrgjq/8o+UENxXyJpMnnF3ULpU5qtHHIthZsnYH0Si3Fkf36bCgX8BbA6huX CEg8kvUoLWOVjUdmJknGu7BNQDGSoRbtFQ/1ZCFO8owG4czBhKBX1y76fyhdMkZR3j4o b1xt9KqPCYDJbKtG+2yO8Fa11jDI1YVSdsF7IxDrISNx86rFVATHMiYYEYzRmEOox6/J UNYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836671; x=1789441471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=CVT6j1P3zfNYXYnEjgM8kLQiWtuO+wz13pOr0ZtKdew=; b=P+AMSI1TrJlUvDA4ssSSdGhGT05+3Cu4nUuCIbiJWfy5jVPD9KBTMk6z2tgo6KxRrL zScJoPj9c4IUwQPlZ9OdFliHsqiXfDv9OOG+nAJuBOWonQgbnfgiIGfe6uZSOm300IRc NUX56Trq8L9+fAfvCsKVJ9C2ViFnWkeFzOeJkxyr+RTIOToqjlGRa5VYPiZz4rxNr2tZ 4TfvRd+O2MK5Oj3DD/GvxR/WrVbGm1AFI9+UJ8x/pMeHeYKqOLnXSj+T21cpDdzD1hOa EoZdUbxf7bQOahn2lqgJ4dqeBrFlzVjc3zvOBO0P5i5hbVtngRAusK2ycJjrj9GGF1hd O8vQ== X-Forwarded-Encrypted: i=1; AKwUvBy3NqlpPRNc/X41VnR8OfiRbP6b7MrRGTByNj6UgfgGELzuaT5VImSVO1PYAD9PDqmvQpT1qTiqaax9dyY=@vger.kernel.org X-Gm-Message-State: AFuF++kEfhrd9k1QwwZQZNP7eyKHfa+lSdgB2f5kx4+/A4obYOqhABSW acirVy01Eub7oowrxvnyLfx6gN3EiI/O+ht4C9gbfn+I1a4wCiM8ZugDdMPc62njb7w= X-Gm-Gg: AYBFou3hRpdiJVFZQKCvYeESjZkSYdu/Rj4dQP9EL0+OChN4nQ+LD/wAOvnWcPvo4/Z i+/u12MNOSdpucSryHTeYn6YZKdGr34GzaSTJowG5QUaKkA0PIE8629WYRBZ9FxSTkd4GrqH//t ooLZbsvng2mbKQ7hOUszYbhvo4b/Fk5vZ24y2HDcHOtHIjfXQdXaKJnTa9oGEfhopGd+kzOK8kG X1JgJR+bgUHXDEFpcGV2jThbDFCsbEAjJz2MiV+1HWebppNbaNjDroNl39VcHwKs8QI9R6TqA/p QIAsAgDkfi0N65jIYM+e14HHXpWqhn0d8r8oD/wso5y5zGwWvsNdqi9aUj0ivrsuB2h5Ath5U7V I3lnbWf9QAAcR6PUmUjxtlPovoS0gqz1ajhe1CLwjA9YwSkrl3cQmUeny7NuqcurSV7lK0wfk+3 OmB/7afwb+LFTXGmFXQGqfD87AW7iiGiWy9KKGayiBJouexklytS4HBKP6sssEn/NohEiA2A+x8 CBs9vusowkApMOtS+7rwf5e X-Received: by 2002:a05:6a00:8c3:b0:856:30dd:91ba with SMTP id d2e1a72fcca58-86168e8bb34mr33801118b3a.2.1788836670753; Mon, 07 Sep 2026 20:04:30 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:30 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 07/11] mm/sparse-vmemmap: move HVO helpers to a public header Date: Tue, 8 Sep 2026 11:03:31 +0800 Message-ID: <20260908030335.96549-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The vmemmap optimization helpers currently live in mm/sparse.h, which is an internal MM header. That works for MM code, but prevents powerpc from using the same interfaces without including a private header. Move the declarations and inline helpers to include/linux/vmemmap-optimization.h. This is a preparatory change for powerpc, which has its own vmemmap optimization implementation and needs to use the HVO interfaces from architecture code. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v2: - Fix missing header dependencies. --- MAINTAINERS | 1 + include/linux/vmemmap-optimization.h | 91 ++++++++++++++++++++++++++++ mm/hugetlb.c | 2 +- mm/hugetlb_vmemmap.c | 2 +- mm/memory_hotplug.c | 1 + mm/sparse.h | 73 +--------------------- 6 files changed, 96 insertions(+), 74 deletions(-) create mode 100644 include/linux/vmemmap-optimization.h diff --git a/MAINTAINERS b/MAINTAINERS index c7aaa2e4790e..2985286cbbbf 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12096,6 +12096,7 @@ F: Documentation/mm/hugetlbfs_reserv.rst F: Documentation/mm/vmemmap_dedup.rst F: fs/hugetlbfs/ F: include/linux/hugetlb.h +F: include/linux/vmemmap-optimization.h F: include/trace/events/hugetlbfs.h F: mm/hugetlb.c F: mm/hugetlb_cgroup.c diff --git a/include/linux/vmemmap-optimization.h b/include/linux/vmemmap-o= ptimization.h new file mode 100644 index 000000000000..492dc662aae7 --- /dev/null +++ b/include/linux/vmemmap-optimization.h @@ -0,0 +1,91 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * vmemmap-optimization.h + * + * Generic vmemmap optimization declarations. + * + * Author: Muchun Song + */ +#ifndef _LINUX_VMEMMAP_OPTIMIZATION_H +#define _LINUX_VMEMMAP_OPTIMIZATION_H + +#include +#include +#include +#include + +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +static inline unsigned int section_order(const struct mem_section *section) +{ + return section->order; +} + +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ + VM_WARN_ON(section_order(section) && order && section_order(section) !=3D= order); + section->order =3D order; +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ + unsigned long section_nr =3D pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_order(__nr_to_section(section_nr + i), order); +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return section_order(__pfn_to_section(pfn)); +} +#else +static inline unsigned int section_order(const struct mem_section *section) +{ + return 0; +} + +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return 0; +} +#endif /* CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION */ + +static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +{ + const unsigned int order =3D pfn_to_section_order(pfn); + const unsigned long nr_pages =3D 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + +static inline bool vmemmap_optimizable_order(unsigned int order) +{ + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) + return false; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; +} + +#ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); +#endif /* CONFIG_SPARSEMEM_VMEMMAP */ +#endif /* _LINUX_VMEMMAP_OPTIMIZATION_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fda525bccf50..dff915e6c48b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -38,6 +38,7 @@ #include #include #include +#include =20 #include #include @@ -52,7 +53,6 @@ #include "hugetlb_cma.h" #include "hugetlb_internal.h" #include "mm_init.h" -#include "sparse.h" #include =20 #define HUGE_BOOTMEM_ZONES_VALID BIT(0) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4a57e6c3352c..25c4e7d2664c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -15,10 +15,10 @@ #include #include #include +#include =20 #include #include "hugetlb_vmemmap.h" -#include "sparse.h" #include "internal.h" =20 /** diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 0db0379826df..d7a59167bec4 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -43,6 +43,7 @@ #include "mm_init.h" #include "page_alloc.h" #include "shuffle.h" +#include "sparse.h" =20 enum { MEMMAP_ON_MEMORY_DISABLE =3D 0, diff --git a/mm/sparse.h b/mm/sparse.h index 59b825df83b9..e511d99fc26b 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -9,77 +9,7 @@ #define __MM_SPARSE_H =20 #include - -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION -static inline unsigned int section_order(const struct mem_section *section) -{ - return section->order; -} - -static inline void section_set_order(struct mem_section *section, unsigned= int order) -{ - VM_WARN_ON(section_order(section) && order && section_order(section) !=3D= order); - section->order =3D order; -} - -static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, - unsigned int order) -{ - unsigned long section_nr =3D pfn_to_section_nr(pfn); - - if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) - return; - - for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) - section_set_order(__nr_to_section(section_nr + i), order); -} - -static inline unsigned int pfn_to_section_order(unsigned long pfn) -{ - return section_order(__pfn_to_section(pfn)); -} -#else -static inline unsigned int section_order(const struct mem_section *section) -{ - return 0; -} - -static inline void section_set_order(struct mem_section *section, unsigned= int order) -{ -} - -static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, - unsigned int order) -{ -} - -static inline unsigned int pfn_to_section_order(unsigned long pfn) -{ - return 0; -} -#endif - -static inline bool vmemmap_optimizable_pfn(unsigned long pfn) -{ - const unsigned int order =3D pfn_to_section_order(pfn); - const unsigned long nr_pages =3D 1UL << order; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; -} - -static inline bool vmemmap_optimizable_order(unsigned int order) -{ - if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) - return false; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; -} +#include =20 /* * mm/sparse.c @@ -139,7 +69,6 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP -struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC50F2EEE91 for ; Tue, 8 Sep 2026 03:04:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836677; cv=none; b=UdQIxuowCU0EKH2fjmArSQ9RkOS0wtOFlpQXLVUdkyuSbdmCivj4WdQwVaNrLw+KSzvWGmOoc0tl6p8soSjh8X4GG2IAvowI4GbugbBvKo9ETWu8GrLO/4BZqoN1o+OcisgRj9jcd8f1vLgqqlX72y5AWqkr32ZOu9560uNHPQ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836677; c=relaxed/simple; bh=Vxc/pXcxmB+iFgkrcwmPIqJ1huzGQ8x23bFD3zc2UMc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qY7K5Iu5X9EsfggzbcsFnW2s6RUkEKwVTzIapS8bEd8Ri4V24K/aXiekj4lidMAzazvHx6nENirVrdnaYx1dA8Kh5EBif0dq7jE+p4FOBW2/dKETHFEQ7gbhFH0oBkvBzyxdmPCnN/z8UoSmT8I99g8F9Sa9jkwfXOxa2fvzKj4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=euVwC79H; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="euVwC79H" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-84e84a6c4bfso3529321b3a.1 for ; Mon, 07 Sep 2026 20:04:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836675; x=1789441475; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=l5VVmQasiv9FjgFp8hyWKR9WL3WpfimdUt+OCxubG2c=; b=euVwC79HPl0mbIU7hKHQtRFJlCDo0y/Zx1XaLx8vVJCgRFep6mQtHhf3Gw/b5gYWs9 UM9w1Y8Qmjw2aRAbhb9RBXuvQ51FklYUg8G4tgzKFClef9jXCEsvRp+VPplAuqUBFKaw g4pkLPIev0DbmB4FplkS0e/lyJ4g4Q6imLBggD/Pzyr4WonaWKgOI1sqJsHhhdCD42Y7 TWLkASY3j3c1tH+4HAZcCjy3JVsmeP306JZ4KaMY5pKd60Dk9UN7RNNDZWPsrAB8N4eA ZqkRUT3HUV3AtBFYRI6haKOfIarLF289O+5mHmnAytVVThYMMlMr96/+42++AKGlY83K yqlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836675; x=1789441475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=l5VVmQasiv9FjgFp8hyWKR9WL3WpfimdUt+OCxubG2c=; b=pEsBvDe7xfotCaULvzEMMrZ7foCSOCCLmfCQlIGmJnsNGO5fmo68VlL3kMJeq4z+6y yCOuf05pUbrtlDnjzESdSpLcKOs4cdJIYn4H15EkvatRPasU8G39cIyUgoXp7mt/YKgp 308cZqO0IFqhAua8HRbzTGWJtK8lPA3QCHtB9JLZpHOFmoa+yAQAJLxOLqtPPNNQTNyZ XrTc5NMGfGgyNw+9ejJWrlyXH/CE7jVXTk3Pjl3GjXqWKLsRmR1Rv4ztu0Rm/oU/WjEI 4vP+RvqrZijGqHIOwWFZNna2IdE1SWRWuZhDiStETyO0wvqbYkHag5lBS6QvOKfhrqhD 75vg== X-Forwarded-Encrypted: i=1; AKwUvBx73Zga4qa32Dtaofh8ejzEfYuQZ2Fz0zBG62b9XmyqIyNSIdV63pOUw8RsSFSb5VtM+LO5bEeqRzDoCHs=@vger.kernel.org X-Gm-Message-State: AFuF++lmpQxpMZz3M6hGJGWw9GejW4wJtYDZpR9tVLOTNQKoWHsVDl9L sdgH1Nt37WH0NtwYkDab8YPRtLxuDWGgVtxrv/0C8kfYB3JsPqSKlMWt81i0PVAtSEY= X-Gm-Gg: AYBFou3hIaRw1X6FzsoMXGfiZqqp+Q8ygoNucIs/z3qFUSO8ZvdZG6a5iZXrGTQHP1d nxxoQawBg9gPKXcJAZ9Ph/amNbj/OU7v/g+GM7Snz5gBzrYEEWbJungcqYjRkEf7zRrYdTsw1eF JP+bY+YXhL82Bm9CDjSpZAT0JQLlLNb7UUIA0ws/hDWS0fcOH1QzbybeknO/+G6uNlyV5tUBlMr 7SmAtrE3gi7stC2Bc3nOFIM3WKa255IdMFMWotILcBfLawThNRfJIBmraSoJ9sbR/vEO52g0e3+ 3me0M6h4+Mzr6PH8YqKf0YER5fSJP4oSOIKc53TvVECneDfA8FTnErgNTwoon/fFvDAcgnfCTtM 7SEw8/5Uypsz/85/Xf6XL5fmW/WHKpJzePC9n5Soa3KTiUaQEqJ5NnIDlXGUhiHjaf6xWSAl17W 4llQDXJ4WbEXL4GUtH82UYa2tE/Or00Q0I2u5e4uH6o/d3ycIVVxpe57vVXZkLoG/6rp/3sIsr8 w6ZVHIEZuKVheWsECTQ5Jsd X-Received: by 2002:a05:6a00:991:b0:848:4c3a:a359 with SMTP id d2e1a72fcca58-85ffeadd0famr28321527b3a.19.1788836675326; Mon, 07 Sep 2026 20:04:35 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:34 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 08/11] powerpc/mm: switch device DAX to shared tail vmemmap pages Date: Tue, 8 Sep 2026 11:03:32 +0800 Message-ID: <20260908030335.96549-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The powerpc radix compound vmemmap population path still finds a reusable tail page by walking the vmemmap page tables. Switch it to the common vmemmap_shared_tail_page() helper instead, so it can use the shared vmemmap page directly to simplify the code. This removes the powerpc-specific tail-page lookup and its fallback path and aligns the device DAX vmemmap optimization path with HugeTLB. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 80 +++--------------------- 1 file changed, 9 insertions(+), 71 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index cf692b2b5f7b..831c231a4a18 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 #include #include @@ -1250,59 +1251,6 @@ static pte_t * __meminit radix__vmemmap_populate_add= ress(unsigned long addr, int return pte; } =20 -static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, - unsigned long pfn_offset, int node) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - unsigned long map_addr; - - /* the second vmemmap page which we use for duplication */ - map_addr =3D addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; - pgd =3D pgd_offset_k(map_addr); - p4d =3D p4d_offset(pgd, map_addr); - pud =3D vmemmap_pud_alloc(p4d, node, map_addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, map_addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, map_addr); - if (!pte) - return NULL; - /* - * Check if there exist a mapping to the left - */ - if (pte_none(*pte)) { - /* - * Populate the head page vmemmap page. - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL= , NULL); - if (!pte) - return NULL; - /* - * Populate the tail pages vmemmap page - */ - pte =3D radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); - if (!pte) - return NULL; - vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); - return pte; - } - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1320,6 +1268,12 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, pud_t *pud; pmd_t *pmd; pte_t *pte; + struct page *tail_page; + unsigned int order =3D pfn_to_section_order(start_pfn); + + tail_page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!tail_page) + return -ENOMEM; =20 for (addr =3D start; addr < end; addr =3D next) { =20 @@ -1349,10 +1303,9 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + PAGE_SIZE; continue; } else { - unsigned long nr_pages =3D pgmap_vmemmap_nr(pgmap); + unsigned long nr_pages =3D 1UL << order; unsigned long addr_pfn =3D page_to_pfn((struct page *)addr); unsigned long pfn_offset =3D addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); - pte_t *tail_page_pte; =20 /* * if the address is aligned to huge page size it is the @@ -1377,23 +1330,8 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + 2 * PAGE_SIZE; continue; } - /* - * get the 2nd mapping details - * Also create it if that doesn't exist - */ - tail_page_pte =3D vmemmap_compound_tail_page(addr, pfn_offset, node); - if (!tail_page_pte) { - - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - if (!pte) - return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - next =3D addr + PAGE_SIZE; - continue; - } =20 - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*ta= il_page_pte)); + pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, tail_page); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92CD730F80C for ; Tue, 8 Sep 2026 03:04:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836682; cv=none; b=g1CkBToZlQT9xfLAV8Tanz/3vuih/7kQ15iNyGlXMk3BMxw9x1CkOmC5MkM6YWTM0Uggvre1Ga+FfU8nIAZPtQi9nBcl5Tg4Rco6AfOB+FsguIn/8c/ssIYSUaJf/dtZJqZ5dw+J9fgSAMp+38M5i/x379w6zDddjkZaXupj7Cs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836682; c=relaxed/simple; bh=gtGPqvfkgJXa2d045dU4tvwglGiVXOWcJDVT4mIHLzU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AvGLDNdnAhvEsjcjlvUqOLoNZcYdvjHjKxwYHtlb+QnRQL2OYyw+nP+FSG+MNwYLEt300LmW9I0cdvg/dfVTN6YKQypfhCbcNGZvl0a7mImnXy/N+i5q3xSr6x4DtMTRe+Ot1aljN+nTnkX4veW4F0LnGMNlMqbN/iQrrb4E3eQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CF9EQOPF; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CF9EQOPF" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-85339ed040aso3211995b3a.1 for ; Mon, 07 Sep 2026 20:04:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836680; x=1789441480; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wfKr3rCUX839N8PLHteC+GkG6En7LO38FWq77XHuH2Y=; b=CF9EQOPFzWd/lfo4G7zW6j3t8tVX397cTn+xqHuGxIE5PgNmWa98su+XIVpY6elH74 im346FpKbuM5bQEtDpuvX9RV8CsXkviCsb9/W9G2NMiF+/oUBs1Fr1tNvdNWA/jHQN7x rsJ9DWBJpcmNtJOU1XKc74UEhOWmVLkkWK9qpQNyVB9xs2lnODThkS0Ctdl3q199khOY 8kVZJIdoAOG4CUCcNLoNVfSgjr7zVArhZru5+DgszPJkw0Dh8zsunKk2MWs9FWTM4NXj r5wznQDVkJxaWeCi5jpXgQBfB4I5yjj7i1h/FCReMk8YTRVN2S0GJEP8B8u9WxGXiNFN cQxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836680; x=1789441480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wfKr3rCUX839N8PLHteC+GkG6En7LO38FWq77XHuH2Y=; b=SU5ZvP/FQIq7E2Vw9F8MZ/H3LNCrBzojjlMA39f8LWhko+QXyde+iXSfVxcg05CcS1 Gbfjsf/LCiF2effScmkHqEViUwXKZzUioW6z3JDter503iE8g1Fi3iPiNQ7T0xusauAo uC/Uahp+6B7G/SM5V7R/h8rWWSreDWviwVKyq6U26oeY+xD/kRrrzDaFHbOG+7KVqJ9U C+b6c9z1ZfNf2/hvSxqn37Z3QULjuL6kNRB+6QuIkIZakZsezlILZ6OyNM592psUjSj0 RQUay+2KuaIf8yniElAY0qKVGLRtkIWHX7N8DI4vo5RegL1C6Mr1Tc3x8Ag/QyQasAHu kFzg== X-Forwarded-Encrypted: i=1; AKwUvByGEwkXKXWhRrNB0FOSnd7Z1PYDPDI6djvIl2XyaU3GNwzgOi6uKrsX0004W0gnQMdFavMERogr04/xh5s=@vger.kernel.org X-Gm-Message-State: AFuF++lnlBLcHBfe6VbcnnTrEi+7ZqGPH2HXcNfGb/uvVhPrZ8isirQ6 kAn3m3vmdKYMz3DX7n75mkUIDLj494LH9bns+rXulhStadUxvBzfYNH+c98P/spt3PM= X-Gm-Gg: AYBFou06v/LBJjVE15h0+w9i7mAhUmcbBYMHyF/TKsH4iVIJt3p0deVI97Nyctj6TTk pBgXj3DdXxf4PRS8mfBIpXgd5A1TZG6bgA9eQnKXb0nCHmRxK2Ila/wprSU5P+65cdArkIlqSiL o5GW13W1r76ggFLLTKwNMXbufN1kO65QP1PevQ66PY02KoDGG4dyxE1o5MoKpUNKzp8Rw7AQQXH U1qnErdFBwIMhw60kQ9IOFmgiXdTgIoB7aGr8v7JdUSuj3xgWyurc28TwWn7/I5jeqRSQ4XeptL pHChQM1zQvnjCqtorqxUJfWHVVp04fleFLsQESxTxvrSCqu9YlfISZ657TDzojBFIKlnhcN4Mdt 8Z1CiA3Zp+1OYsAUCpF9FZPsUdr7A5m0SnRzN2dQnvs2+N5VTXDH1Hb7IpkseYYYCUC0Ad4KFDX tY1gjZwyTogjAl7yfwKoydpvrb1Y4XUthlk204Ol63OIiGY1kvCNkZR8rEKrAhJ4cwWDzBa0nU+ ueW+unCieGT2+LYqdrCtgMT X-Received: by 2002:a05:6a00:4f81:b0:845:3033:6cb7 with SMTP id d2e1a72fcca58-86166bc01c9mr34022528b3a.9.1788836679806; Mon, 07 Sep 2026 20:04:39 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:39 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 09/11] mm/sparse-vmemmap: drop the extra tail page from device DAX reservation Date: Tue, 8 Sep 2026 11:03:33 +0800 Message-ID: <20260908030335.96549-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The device DAX vmemmap population still reserves one extra tail vmemmap page after the head page. Drop that extra reservation and let the shared tail page cover all tail vmemmap pages after the head page, so DAX follows the same reservation model as HugeTLB. This reduces the reserved vmemmap pages for optimized DAX mappings to one and removes the now-unneeded first-tail population from the generic and powerpc paths to simplify the code as well. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- arch/powerpc/mm/book3s64/radix_pgtable.c | 46 ++---------------------- include/linux/mm.h | 3 +- mm/mm_init.c | 2 +- mm/sparse-vmemmap.c | 13 ++----- 4 files changed, 7 insertions(+), 57 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index 831c231a4a18..e7e751c48dd2 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1218,39 +1218,6 @@ int __meminit radix__vmemmap_populate(unsigned long = start, unsigned long end, in return 0; } =20 -static pte_t * __meminit radix__vmemmap_populate_address(unsigned long add= r, int node, - struct vmem_altmap *altmap, - struct page *reuse) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - - pgd =3D pgd_offset_k(addr); - p4d =3D p4d_offset(pgd, addr); - pud =3D vmemmap_pud_alloc(p4d, node, addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, addr); - if (!pte) - return NULL; - radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1297,7 +1264,7 @@ int __meminit vmemmap_populate_compound_pages(unsigne= d long start_pfn, if (!pte_none(*pte)) { /* * This could be because we already have a compound - * page whose VMEMMAP_RESERVE_NR pages were mapped and + * page whose retained vmemmap page was mapped and * this request fall in those pages. */ next =3D addr + PAGE_SIZE; @@ -1318,16 +1285,7 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); =20 - /* - * Populate the tail pages vmemmap page - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, = NULL); - if (!pte) - return -ENOMEM; - - next =3D addr + 2 * PAGE_SIZE; + next =3D addr + PAGE_SIZE; continue; } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index a2ebe87e7654..969594074fd2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5167,7 +5167,6 @@ static inline void vmem_altmap_free(struct vmem_altma= p *altmap, } #endif =20 -#define VMEMMAP_RESERVE_NR 2 #ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) @@ -5187,7 +5186,7 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, * For vmemmap optimization with DAX we need minimum 2 vmemmap * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); + return !altmap && (nr_vmemmap_pages > VMEMMAP_OPTIMIZATION_PAGES); } /* * If we don't have an architecture override, use the generic rule diff --git a/mm/mm_init.c b/mm/mm_init.c index 7a2e58d631c2..629420a83891 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1056,7 +1056,7 @@ static inline unsigned long compound_nr_pages(unsigne= d long pfn, if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 - return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); + return VMEMMAP_OPTIMIZATION_PAGES * (PAGE_SIZE / sizeof(struct page)); } =20 static void __ref memmap_init_compound(struct page *head, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 0201877a7f80..e655d9d1348f 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -136,7 +136,6 @@ int __meminit section_nr_vmemmap_pages(unsigned long pf= n, unsigned long nr_pages { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_order(ms); - const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); @@ -147,13 +146,13 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages =20 if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; + return VMEMMAP_OPTIMIZATION_PAGES * nr_pages / pages_per_compound; } =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); =20 if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; + return VMEMMAP_OPTIMIZATION_PAGES; =20 return 0; } @@ -521,17 +520,11 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, if (!pte) return -ENOMEM; =20 - /* Populate the tail pages vmemmap page */ - next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); - if (!pte) - return -ENOMEM; - /* * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - next +=3D PAGE_SIZE; + next =3D addr + PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, page_to_pfn(page), flags); if (rc) --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28802311592 for ; Tue, 8 Sep 2026 03:04:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836686; cv=none; b=Z8smdQClP19KOrlBgxXeMg97aG4zXQuATPsb0RE9+n6lAPQQMx+DX0vfjhL14Fou8KNpQM1d60truamNW/PZ0cDWB9Fnpo4wZiuZ13ydKq6DSyHdwDrciHsqOlXQZArX6sHvNnUUIKIHad+yJkEtoVGbWwfbq9pF9kOfXsgWB44= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836686; c=relaxed/simple; bh=0craoejHz+yL27FxB85ch1A7TZcJ/6nPjXoONqTOYGE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=k63opLgBnCEZ21G4ToqHsI/j8++Xy6En6zB3UonVjw3oe101QDkeWNP++o5tCjRYnEBCm/i42AGGdHtiYVxgKKvV7c1OaIplDcn8Rk8AXvcmFNT1hmdLAy9cpVd0FsQAZr/5OMh3alqE6UimdZw7yKPfdC1S0WteES/Ul8GN3Ls= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZmcYB/kI; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZmcYB/kI" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-84e27035206so3756009b3a.3 for ; Mon, 07 Sep 2026 20:04:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836684; x=1789441484; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8Uns/ZJlTRrXsZn1DdaJluablabo4P2O8/HR8HKp5bE=; b=ZmcYB/kIbdeQGeaxzrQXuGgGp2FqSs8Hrh3it9lidPD6Qxv3OhZQ4VDw5kuVC/71GK k6zJNDMcscltrPZvF/1Vae/y1wbZHX3IsZ62GXjNePc+EdEmviv9W9KX/QgXcQToq2ok hR692KkqMoZ0pBgAydYaq8bWM6KBSDMp6UJYd28QzfE/Wp/tDsJaAYx615DANiW4Qy4z Yx81BlHQ+YhNrbhcoxTzD7iEOqT8i8PSG5PSH0q0y28sm1EOK0wzhHhDTk1claUtc2b2 Uuqa+WctIKl7qxgimbVBTN+lDxTp2qHpdpGJpoWas8SF925tBlxLcVyF90KSgk9+Lv12 DbMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836684; x=1789441484; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=8Uns/ZJlTRrXsZn1DdaJluablabo4P2O8/HR8HKp5bE=; b=Gqoz84UdTqLOF6mBj5gKk0HK2ralqAuPo0Pfw/L7nlkc9ifcysVGi3EOQjCrZCaU79 HuSNjivQQaSa6UxiO6/PtDP0+xsGfgmESktrqeFBzm5H2LRi/w9aJeYz0J9wxb5S9+If YBDT7y4qRgWxFARRNMSQZQ03IyLIBiGcWUkM6/EWetnlt7vrm9tmkYbLpR86V9Qolhm7 3fXTppv++l20E2PfuxjxBsCaeq9+vC+jP1KXPNWKuijRtLNs5yFI8AchYJyGobXW+bD/ atDFl6WXq8/Yy6K26Ce6vLinahtRECp53jH4ZPMXrzeW+9ujK3RMucCbopifbcoYv8aq O5IQ== X-Forwarded-Encrypted: i=1; AKwUvBzRg3xgJzuMQ06fNVFZSCpk+iGrkpgSEeLKj6bLOPRAMo7vIClQiDJDzcPQNPm3XgSA0iHzS5kugwRArLs=@vger.kernel.org X-Gm-Message-State: AFuF++mDM4zcBY46Amtg6C817d/dvCWC9dk1g7qjWTx9CaDt7KTfXaCC GDefptGgO+CH2Kot5A54yLe+LTRgkeE/XPi2yfFq1abARbEGlXBZw4JawmII/jT3Hug= X-Gm-Gg: AYBFou1rfU+YztsEwP2qxqtz25tUUU84p90vCdCobfkz+PwrT3j4P7BF5FkKbS+ySXf Y2nOaxVmIdY8o6diXIGvyuRpdCT/+8y50ZAxfU7f8cwRSc3iBUwsbjOz8e4xXz6ssdTTdWDDTJ/ dCT9onaAuYegjD6/gLcAnNT1IRik3O6Uh40xMUGq3gb++2DeaNP3Sk1F29pXzW/Af2YIX9ZzC2i Kgu5HPjph3iNal3MthnP//baGQhCerp+6FUHz4KRdDtqEzi8JR2i7KFK7tMyRfIwwZc9UqrNcid YDlV8wTt0Kl9rNWN376I+hfEriDWRtV9rwxxpvtwcQGAJa+2txuuZrOMan/ndBSiCbEK+7OH2wG 46eAgmKCHGpVAImVbPYCJ3YpHNhNIgfM2srsBJIvmCzO5iVMXjHPumqsIGEdxUEBTWFzkou3Qbe aHQwvjhQV/9PO22m+IQD/VeXxQ+8TapPa6uHKhZ8cye2Dg3MbgwA21FJS6D68M4R7oQU6L7Kwk0 vYq8X8AcAsv3hHMscJo6XHQDw== X-Received: by 2002:a05:6a00:3494:b0:857:726d:2e95 with SMTP id d2e1a72fcca58-8616a96c426mr38438397b3a.18.1788836684381; Mon, 07 Sep 2026 20:04:44 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:43 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 10/11] mm/sparse-vmemmap: drop unused section_nr_vmemmap_pages() arguments Date: Tue, 8 Sep 2026 11:03:34 +0800 Message-ID: <20260908030335.96549-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_nr_vmemmap_pages() no longer uses the altmap or pgmap arguments, so drop them from the helper and its callers. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- mm/sparse-vmemmap.c | 10 ++++------ mm/sparse.c | 3 +-- mm/sparse.h | 6 ++---- 3 files changed, 7 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e655d9d1348f..4950ac2a1c01 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -131,8 +131,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 -int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages) { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_order(ms); @@ -637,7 +636,7 @@ static struct page * __meminit populate_section_memmap(= unsigned long pfn, struct page *page =3D __populate_section_memmap(pfn, nr_pages, nid, altma= p, pgmap); =20 - memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages)); =20 return page; } @@ -648,7 +647,7 @@ static void depopulate_section_memmap(unsigned long pfn= , unsigned long nr_pages, unsigned long start =3D (unsigned long) pfn_to_page(pfn); unsigned long end =3D start + nr_pages * sizeof(struct page); =20 - memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages)); vmemmap_free(start, end, altmap); } =20 @@ -658,8 +657,7 @@ static void free_map_bootmem(struct page *memmap) unsigned long end =3D (unsigned long)(memmap + PAGES_PER_SECTION); unsigned long pfn =3D page_to_pfn(memmap); =20 - memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); vmemmap_free(start, end, NULL); } =20 diff --git a/mm/sparse.c b/mm/sparse.c index 9349ed6326c0..adf057f54c0f 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -251,8 +251,7 @@ static void __init sparse_init_nid(int nid, unsigned lo= ng pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, SECTION_IS_EARLY); usage =3D (void *)usage + mem_section_usage_size(); diff --git a/mm/sparse.h b/mm/sparse.h index e511d99fc26b..5d0f407bde7e 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -70,12 +70,10 @@ static inline void sparse_sections_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); -int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap); +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages); #else static inline void sparse_init_subsection_map(void) {} -static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages) { return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); } --=20 2.54.0 From nobody Fri Sep 25 22:19:33 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1165312814 for ; Tue, 8 Sep 2026 03:04:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836691; cv=none; b=MuDGmp3JonNseUQkyVP7L8yjDCVPsi/MafmJSL4LX5GPZXssnfsSpjhNEXRmS5jmSPLUvlZp1vCpSoZsVhnqESG7UrS2I0wfNbZULhCfH/uSwbAuaV5S8hplsE/eYK4FtskpESaa0PhVAw/jZprf6HhzXTIR+zhPSymPpKrn414= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788836691; c=relaxed/simple; bh=4wufTkSeOR/+Z2FWN8LbWpxz3ii+Qo8NagFu/L3vU5Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fZumEkWfVaE7WrrCZXVvcM1MMJ1yLJIJXSe/dbP8olKb2lPCaJtSo1MADEGY+wD4trd/ftkY2La/0QgdD5IyVxKcxaH6TUD4xqT0zIL19mNopLl5C5laear5//CRPrayMDZqM3mNijvxytEhAojyFTHK+v8vIZPhyHwJkj1CXO4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=UJjN2+ak; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="UJjN2+ak" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-85339ed040aso3212078b3a.1 for ; Mon, 07 Sep 2026 20:04:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788836689; x=1789441489; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=IQIwuT52UGUULzXnvd9jyK7/apm1xntD2qx3gMLc028=; b=UJjN2+akjtAsORY2Djv1wHs4JESj+kBU/dsb6OD7u8mKRLqCvx+bLYey7dfoSo62wz HRPpFRAvnD5l8WkyDzY/zVXvxgsWUTonLJ0hckip8Tpsb7xwICQRt6nTIo+PaC171p25 0yHFq8qAxCoptDH5Nx7wJpBlZGSFBxlXbaJPXm7o80UW2vUv+Nye4Y8bwQRFT7REoboJ lc9Yhi/o05V+CLXeO/9f1JP0yE7R0cFpZxOQ0XyujHJQnK78gCRW6/L7FVVSMczLRneF J/9tvepLT+Nh5qR+RPk6i7iPMjgCgPGWlt1k2C6jmDFG2a4WxNlO7pDR3AmtDkKLaKsR Cp9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788836689; x=1789441489; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=IQIwuT52UGUULzXnvd9jyK7/apm1xntD2qx3gMLc028=; b=KUkP5NDBcFvgGnL5oEh8MZ46wKmqj7Tnc8vSTTgTCr2ILYrOLb7DlnUaSrNtMNHq+C +QOoHYXyrCLP5BWD+KTrkrwwb7XGp6SgKMv63pYpb+K/TpwG/GxOwWHtkk1WJ7ZZMaVW B6oIBaEcce3NIoEx5dcq2eEoy0s200uTwfY0o2RqKjXUDY83BfFa3vYsYowXhTTp4C5O bAnRrQHY9KZHSwqbFf5sikv1XaKaHnhgS1/+ZNJgdxsZAuvQ1dV130VqSEWDchKNqxzV 7zkg2J2egvJMHwMPlkWEkVSfWXHTWevyjmNznNhRXHGozrlsWL7bHIIx4lG6BScPy0nK X5hA== X-Forwarded-Encrypted: i=1; AKwUvBwdEQWjVP3nOh9yKI9Raq0Z6T1/yFy5SlGYA1csWAF09hGsUZW3D3Jjc11wLME8FQqPo1oGuTcjbuY1LUo=@vger.kernel.org X-Gm-Message-State: AFuF++km48w0sx6Wq8gxaMH9vh7dY2EMdAGr5yAjZS1A3NOMMzwW0bNP TNvfNiTypmJEtM2qN+ruDsEK6EWA6Xgd0gG982x2NzuT93lg8MzpUo/A+F4I/W0C0XU= X-Gm-Gg: AYBFou0jPebD7iH2SFg9CKE4lPSxu8JkoYh4VCBv0BLCjoJsVh4p6mtCK4A6gzZKTp1 EkxcPHGjlNaMOKXJEMnw7gRehmcFHtr1cpbzcU3HENSYVcM9MiHUW64dd7HQhM1QXUx0ke6q/XJ ZDqsWLnmeTHSi3Fq1e/2WmwjO69VUortbUlNweK6/HMg5VPCAKaUZzWImyHMYK5ezzPC8cxZ693 yMOHT7evsdgp5ZHFh5AKaTpTt8+H5/8b3PPmKvHU9MtLQOg0695VIGeBgV7V5HJGRubRzlj1Sf1 ba7pxAcTuoGCeTQT/aeL48MqItxAUwY3YnBvtDuMeo05psvkRqp2DXxFZ0IM/MZVWx91syCbtJn B0BquJarXDEqObzNeW5Vu+9EPZvVB/gzsx5bvLIDnZp3cKGQYQl85J4NqH1QDM0BvzgmPJkACmo DzuLuMeXW5IGUXtfRiBwPpmWZdSmv3Ls1aI/vTisl0YW6HSIXANKAm6Yxz9e/4AslkLaLehYVwC gcSkROaFGzUpW90AOT3wL/gYltYLUw2Bys= X-Received: by 2002:a05:6a00:400e:b0:84e:e741:174f with SMTP id d2e1a72fcca58-861669c6df9mr39676733b3a.7.1788836688830; Mon, 07 Sep 2026 20:04:48 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-86152a358a2sm4868234b3a.29.2026.09.07.20.04.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 07 Sep 2026 20:04:48 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v2 11/11] Documentation/mm: update DAX vmemmap deduplication docs Date: Tue, 8 Sep 2026 11:03:35 +0800 Message-ID: <20260908030335.96549-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260908030335.96549-1-songmuchun@bytedance.com> References: <20260908030335.96549-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX now uses the common per-zone shared tail page for vmemmap deduplication. The old documentation still described a DAX-specific layout with a separately populated tail vmemmap page and half the HugeTLB savings. Update the generic and powerpc documentation to describe the shared layout. In the powerpc document, keep the radix and 64K-specific details, drop the duplicated 4K PUD arithmetic, and replace the repeated device-dax diagrams with a single parameterized PMD/PUD diagram. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v2: - Clarify the commit message to state that the 4K PUD arithmetic is intentionally dropped reported by Sashiko. --- Documentation/arch/powerpc/vmemmap_dedup.rst | 90 ++++---------------- Documentation/mm/vmemmap_dedup.rst | 32 +------ 2 files changed, 21 insertions(+), 101 deletions(-) diff --git a/Documentation/arch/powerpc/vmemmap_dedup.rst b/Documentation/a= rch/powerpc/vmemmap_dedup.rst index dc4db59fdf87..8286acbca9bc 100644 --- a/Documentation/arch/powerpc/vmemmap_dedup.rst +++ b/Documentation/arch/powerpc/vmemmap_dedup.rst @@ -19,82 +19,28 @@ With 1G PUD level mapping, we require 16384 struct page= s and a single 64K vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Henc= e we require 16 64K pages in vmemmap to map the struct page for 1G PUD level ma= pping. =20 -Here's how things look like on device-dax after the sections are populated= :: - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 15 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - - With 4K page size, 2M PMD level mapping requires 512 struct pages and a si= ngle 4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapp= ing. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - -With 1G PUD level mapping, we require 262144 struct pages and a single 4K -vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we -require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level -mapping. - -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 4095 | -------------------------= -+ - | | +-----------+ +Here's how things look on device-dax after vmemmap-optimized sections are +populated. ``N`` is the number of vmemmap pages required by the DAX mapping +above:: + + Device DAX vmemmap pages (N pages) backing page= frames + +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= ----+ + | | | 0 | -------------> | 0 = | + | | +-----------+ +---------= ----+ + | | | 1 | ------+ + | | +-----------+ | + | | | 2 | ------+ + | | +-----------+ | + | | | . | ------+ +---------= ----+ + | PMD/PUD | +-----------+ | | A single= , | + | level | | . | ------+------> | per-zone= | + | mapping | +-----------+ | | shared t= ail | + | | | N - 1 | ------+ | page = | + | | +-----------+ +---------= ----+ | | | | | | diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_= dedup.rst index 9fa8642ded48..8c287ae3f86c 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -1,4 +1,3 @@ - .. SPDX-License-Identifier: GPL-2.0 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -192,32 +191,7 @@ to 4 on HugeTLB pages. =20 There's no remapping of vmemmap given that device-dax memory is not part of System RAM ranges initialized at boot. Thus the tail page deduplication -happens at a later stage when we populate the sections. HugeTLB reuses the -the head vmemmap page representing, whereas device-dax reuses the tail -vmemmap page. This results in only half of the savings compared to HugeTLB. - -Deduplicated tail pages are not mapped read-only. +happens at a later stage when we populate the sections. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ +Deduplicated tail pages are not mapped read-only. The mapping layout is th= e same +as HugeTLB. --=20 2.54.0