From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC3593C98B5 for ; Mon, 31 Aug 2026 07:54:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162875; cv=none; b=btYJ5+xoQYLnLgBcr+zXnlNC97Q0fmd0PyB+vELrT35LSj2fZPPEZb/uHX6ltho+ys4X8jO69RtkV5OPedZwnh92IFkQVPlPx+zSmaHJ0PToWas6yPMOSWnczzqpLizF5bBmQSWdAYWrp2/IFpeC0H5qndC/ibvOTnhosO6OzEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162875; c=relaxed/simple; bh=o8Aks7xfsNOUvUy+GMfdpXSPyjdlXTmKg+NBN/YdiZ8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PbT7oKX4GJ/GfOUhTWQPeY10H7+Cxg7dJd+un7oT+iiIMa+ZL+HPICazsaX5T+aiy1dIF025m70kSCLHWv+TSgvuiHaRxiJ+0gZyy/0dY3zPtPTZBxE+5gmsB0Lq2/1Uc5+5QHOTd8SiiWmcJR4BSiuI6w9dM+mHPGDPwQ7j3x8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gDXK/fxu; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gDXK/fxu" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-8485b358552so3095246b3a.2 for ; Mon, 31 Aug 2026 00:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162872; x=1788767672; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XjT0Dc9Rc0+WQPw0AxlsVqG1ooKsABjb50Q/PiKd2Bg=; b=gDXK/fxujBhG1uzwCam1wU7ulkvRcM2+skkB0CV+uulDP67k7i1cPv2JOJecQyqjdV 5ZOAivH+rdpARi53krkJAv6HAwpqPQEhYxZaQiB+sCiALqnHg9s4iB7BapQLFG5T/lk9 UIWBp3QlLLW+xAJYt/DMnzuw8/mlz8OW8G/OQgZ4noGaCD0dZOnHvxH+WpXa+iUiTLBG y37Lsai6hpGKA1W1fxC4kePWvevY7v8JeVeX3EAbhk+XjxGER8cyMd22TDHWCMrkIcg1 Jj2B8S4Rjr7Oc/PMInWFDvfm7WwIX+yRLqIG/+mfMa6eTIKqeN4X6q2fPcljjwKL7TVh DpxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162872; x=1788767672; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=XjT0Dc9Rc0+WQPw0AxlsVqG1ooKsABjb50Q/PiKd2Bg=; b=g7uxNJsVAe+1afU0KFLaAx/0Kjjw0yUyI00ILpu9A6S30kELWEAxXnsOFOltPHbgrT C6HdnId0qU6s7faxRZxGjTjvpu10V4291gaSBxUkLZfsvMYwaDGnn5Zik/LWGKoUMFMb TXLR2g1/8JkvwBe1Ot+tELAZjAJ1BevEdENuBJMeUfR96l5hTQJ7bLYk4Zt5E1B5m2Pd pJkVWPZI9+oWLvWol1VABSF5TIBMWYX77YSZv+ZLkgK5BYhWBcXwc9AxzS5ZnAMKIUH3 z6RI7R1Dgaeg4IqJzscqZpHaDLs/jtVrqgQe9Le3wyWiyzmia1r7Lf6N3m1Qt5ebgin/ KujQ== X-Forwarded-Encrypted: i=1; AHgh+RrzcgidAgBfrP4UoTD2B0p9P/NBNwLznBO5+sVIjKqyLuaOOP8uexvsLDou7sQb0EhO4FPxxE/Y7VeLf7E=@vger.kernel.org X-Gm-Message-State: AFuF++nmxl44/Ee9OvkgXTRAEm8SuE1hv5/7l05/DE7vfqIK6KXtSMAM +Sc29SE8hhJx9IM/w4bZVHXcVQUjhqQwX0/VrVeZPQZPDKwft927sGauSctMKhpB3yc= X-Gm-Gg: AR+sD122v/SHzQ5vuqYXN1gIRsA+gImqIh/DEUCeRfdOoSneSbbbAu3/marbvBjVcdG 5XmEbHZDux0NnMvkXwJUULSSK+A6nezQ3MgzPmaAYeuZ1xe27qrWjLUgNYHq9q+fc0nqAFm+CuV 4eLbokhGEIQbPkfUBk0f2lh8OAZahgKkYv/LSMld51Dmrrmb8hrbT+Qt+jKoCwv+WZnbPbf7jiu bEo7qhJEwgP4zgMlbfKBrT6p0dL4sisaFl30QAdkVxugdctJAautLoaGNG8Y4aOtcga4eL9Aswm xkfivERCX+Or1DiB+gGYqmoGAQE0uNiZodsvnyUZ3kKpwPPTXvIJnQVmHArGiCGhV0QRuHZb9+N 61PpAYV7kSNCOl+gxs4YNlsWawEtL5sfIfKcXIKT/g0yLxPGrDMKDG6VdCr+KjxTuBeSb6zYx+x 1y96cj58Ecz3sa9ZeT9thQh6hId6wuqgF0yUdd+A4weNozfyIC+Q877JgRkTgGsnIJOdeDkm4H8 HFNw6GAa6w1rDPuZFfWZeGr X-Received: by 2002:a05:6a00:3c8b:b0:857:726d:2e98 with SMTP id d2e1a72fcca58-857726d2fc5mr22906110b3a.21.1788162872037; Mon, 31 Aug 2026 00:54:32 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.54.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:54:31 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 01/11] mm/sparse-vmemmap: introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION Date: Mon, 31 Aug 2026 15:53:32 +0800 Message-ID: <20260831075342.57563-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The section-based vmemmap optimization infrastructure is still guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, but it also can be used by device DAX. Introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION as a common config for the shared infrastructure. Select the new option from HUGETLB_PAGE_OPTIMIZE_VMEMMAP and from DEV_DAX when the architecture opts in to DAX vmemmap optimization, and use it to guard the generic sparse-vmemmap state and helpers. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- arch/x86/entry/vdso/vdso32/fake_32bit_build.h | 2 +- drivers/dax/Kconfig | 1 + fs/Kconfig | 1 + include/linux/mm.h | 3 +++ include/linux/mmzone.h | 13 +++++++------ include/linux/page-flags.h | 5 ++--- mm/Kconfig | 3 +++ mm/sparse.h | 4 ++-- 8 files changed, 20 insertions(+), 12 deletions(-) diff --git a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h b/arch/x86/entry= /vdso/vdso32/fake_32bit_build.h index bc3e549795c3..5f8424eade2b 100644 --- a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h +++ b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h @@ -11,7 +11,7 @@ #undef CONFIG_PGTABLE_LEVELS #undef CONFIG_ILLEGAL_POINTER_VALUE #undef CONFIG_SPARSEMEM_VMEMMAP -#undef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#undef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION #undef CONFIG_NR_CPUS #undef CONFIG_PARAVIRT_XXL =20 diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index 602f9a0839a9..85ad4c135cdd 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -8,6 +8,7 @@ if DAX config DEV_DAX tristate "Device DAX: direct access mapping device" depends on TRANSPARENT_HUGEPAGE + select SPARSEMEM_VMEMMAP_OPTIMIZATION if ARCH_WANT_OPTIMIZE_DAX_VMEMMAP help Support raw access to differentiated (persistence, bandwidth, latency...) memory via an mmap(2) capable character diff --git a/fs/Kconfig b/fs/Kconfig index d1c210c6508f..9b32ce79cc80 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,6 +278,7 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP + select SPARSEMEM_VMEMMAP_OPTIMIZATION =20 config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mm.h b/include/linux/mm.h index a9fbe26536f4..edadd7549b72 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5188,6 +5188,9 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, unsigned long nr_pages; unsigned long nr_vmemmap_pages; =20 + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) + return false; + if (!pgmap || !is_power_of_2(sizeof(struct page))) return false; =20 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c9ae7991a8b2..e9b54ea0eff0 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -102,9 +102,9 @@ * * HVO which is only active if the size of struct page is a power of 2. */ -#define MAX_FOLIO_VMEMMAP_ALIGN \ - (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && \ - is_power_of_2(sizeof(struct page)) ? \ +#define MAX_FOLIO_VMEMMAP_ALIGN \ + (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION) && \ + is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 /* The number of retained vmemmap pages with HVO enabled. */ @@ -116,7 +116,8 @@ #define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1) #define VMEMMAP_OPTIMIZATION_NR_ORDERS \ - (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS = : 0) + ((__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 && \ + IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) ? __VMEMMAP_OPTIMIZA= TION_NR_ORDERS : 0) =20 enum migratetype { MIGRATE_UNMOVABLE, @@ -1155,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; @@ -2019,7 +2020,7 @@ struct mem_section { unsigned long section_mem_map; =20 struct mem_section_usage *usage; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION /* * Normally, sections hold regular (order-0) pages. However, for * sections with HVO enabled, this tracks the compound page order diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index ae2ebaed6d4d..de3c06062bc6 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -208,14 +208,13 @@ enum pageflags { static __always_inline bool compound_info_has_mask(void) { /* - * Limit mask usage to HugeTLB vmemmap optimization (HVO) where it - * makes a difference. + * Limit mask usage to HVO where it makes a difference. * * The approach with mask would work in the wider set of conditions, * but it requires validating that struct pages are naturally aligned * for all orders up to the MAX_FOLIO_ORDER, which can be tricky. */ - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) return false; =20 return is_power_of_2(sizeof(struct page)); diff --git a/mm/Kconfig b/mm/Kconfig index c1ddf59c0d71..b5f8372cd164 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. =20 +config SPARSEMEM_VMEMMAP_OPTIMIZATION + bool + # # Select this config option from the architecture Kconfig, if it is prefer= red # to enable the feature of HugeTLB/dev_dax vmemmap optimization. diff --git a/mm/sparse.h b/mm/sparse.h index 049272aba84e..b408d15baf7b 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -10,7 +10,7 @@ =20 #include =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION static inline unsigned int section_order(const struct mem_section *section) { return section->order; @@ -72,7 +72,7 @@ static inline bool vmemmap_optimizable_pfn(unsigned long = pfn) =20 static inline bool vmemmap_optimizable_order(unsigned int order) { - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) return false; =20 if (!is_power_of_2(sizeof(struct page))) --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B4933BFAFB for ; Mon, 31 Aug 2026 07:54:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162879; cv=none; b=KQcQjQqjYAV3243LaQH4tF2oWievtUNAU7x3rhA+W5ngysvo4oDhV7ZMgCv50nTJ20MYca2OcB0Z20H+SvL5tQ7kaVqhkzuksO4bGvfo8l4KvIn/FcwkpXW1qQAq2uBdiyqjvRvxxvfq5ULGE8IQ4nq89VnSpl8gv4i3QrvUuAY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162879; c=relaxed/simple; bh=NbNECtNzYk+UHZ/PxBOh30rHMR89K1eknJE2XvFAA94=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=djigFxEdRF+9Q6Q8Zrp+vEsf+L2Q+9B+LVZyBbIrpuGSD6OtH4Qq6QY1TBABzE6wwxAceoIVfQGiApmacmdewrOpPa76zE6RHhIkqQTB2RNH8icGZ6dFJAN54YgiR9TKX4sOuEgz/WVretiZ5qi2O2EGSzRKc0x1/y/htONFoAQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gM2ribIR; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gM2ribIR" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-cc1cb472b76so2777466a12.0 for ; Mon, 31 Aug 2026 00:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162877; x=1788767677; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4jDvtv3kIRHCHEO8kcmf2tbgWKyzjUPm6N9KUwqqy5o=; b=gM2ribIRiWiEPBT9ra6LIelFw9BMOo2m8PRRdeMUJTLmn0wmYlj3i7Gloa7y2oWQXs OhF3khhEGOtbylg23Or+2Zht2T+NyY5wb/0z7KjtaX6pv1kttHk7Rn9bP99oS6Nc80k7 WL/0IlZ+CFoACoSV7TLivQfvIvGHS7rdh0Ai3qOUW7X9BijqHH/8g2OeAAc23OCNzfFS Axoar1YNwCYQadPoJYGtgbZJmmS/RrKwkY5aBGxRHlMEqR53pJRRz3xsZkQ8Z6Q3NsI2 FE00VLtdzVk0xH2ytluwXmx9MHGwxGKw6rpcS2CiDbHgFaZt2cE6oekNyuSSAyGt4pa+ 0oOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162877; x=1788767677; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=4jDvtv3kIRHCHEO8kcmf2tbgWKyzjUPm6N9KUwqqy5o=; b=ULBts8s8BtYQRX1IQZjYqYFauLQbxu8ecfiDehDTfQp1ZmTIKjFbstADaa+zgRFsaU EdKYtWSt88cn8Btgxq3xj+VcNkO0mHfEN8AAct4sJASGOa/23htLQztKh7VSIfFbQKPj 4W/KqEQ1qJpNI3YihGJmsqTBX4ufqHvWj2Ixp77TUU4iFhWIt17QWl7st7uKgQqXzqz2 Myn3Qlfh0OQegn75EyVQ9udxaeq6tnhBREQnQYA+I9TP1Ur6LmhW+0Vgg1IPLpZbxGqV NDvhRiYFsMPEwmYG/L+ZjBLTqoHP5kiPnd5gaf7p43XhYH+Bj5DQtjcjDSI+21PFnKWk 30CQ== X-Forwarded-Encrypted: i=1; AHgh+RosqK3pvdCdrvqvQhaj3YZ/Q2e6LaTMBbNA5PYLqWmnW7gOoXgcBlsu4LYdRjU+Ada8HWl208mqXsHrzms=@vger.kernel.org X-Gm-Message-State: AFuF++kn/2i9nF8FcfG5Hh2Vb3vV2R7mHH3Sixu4g1R9VmbPKcrm+wjh 6ZgnEZcnwpU139GRO9qN1BfJeK4cLP25p1pY9TaOnv4exjr/E+kXe2YHyzlIr3mdL6k= X-Gm-Gg: AR+sD13fK6rtrNq3teQIPsl8j5qC+FC7lEy4aOXFSSgHG6e/vx8krU9+ThnnzTVtx0o lOmJZ8waPMhtbkgelH+0tK41lVLEgieXX7XgrcL1GxTqNAdb/nQ2WMbOAFUFRKe3Ik9dZWGZS9x S9qZeptIvBwEUMnYVw/lEUAMhdvNqLM8iP/TlhXaj7Bttzxyc3/LfXF+R4KZfgWov9Cqc8s4zq8 0x95VFJWz3UB40/LBObfcLhHdlfEPivxkS//QceuT9mbI0z8Gt0UeUCm2zwLJPxGogarRrisC8a 9FHBxt3wuNN9dpT8YLm7h0vqCA2kC8lEfatw0DUFZLWKiRNJ2UPrn4KS0MJ74WPZoq8LR4MKt+l uD/hQ3mjxpFWBn/H1L4c2F6JjdwAx28jPmNfMibMMojRbj1RgbXSWbdGEAj8K5N9HR+j8N+7A2S JkO5j9L0QtWE3fzdDooOdZEHhXwljCbegsoA7Hozwt83ZAdBkA3DnvAjRZqvj1hUNrFXhQ5owNY jc6r6CKRaEiWsU/da/C7d5P X-Received: by 2002:a05:6a20:6a15:b0:3c4:4272:b400 with SMTP id adf61e73a8af0-3d2688bd9c8mr37731922637.18.1788162876874; Mon, 31 Aug 2026 00:54:36 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.54.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:54:36 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 02/11] mm/sparse-vmemmap: factor out shared vmemmap tail page allocation Date: Mon, 31 Aug 2026 15:53:33 +0800 Message-ID: <20260831075342.57563-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB and sparse-vmemmap each have their own helper to allocate the shared vmemmap tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper. It allocates the page through vmemmap_alloc_block(), and uses cmpxchg() to install the per-zone shared page. Expose zone->vmemmap_tails under CONFIG_SPARSEMEM_VMEMMAP to match the shared helper's build condition. This avoids a !CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION stub; when optimization is disabled, the array has no entries and the compiler folds away the unused paths, so no storage or runtime overhead is added. This removes duplicate allocation logic while still handling both the early boot and runtime paths through the same helper. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- include/linux/mmzone.h | 2 +- mm/hugetlb_vmemmap.c | 28 +--------------- mm/sparse-vmemmap.c | 74 +++++++++++++++++------------------------- mm/sparse.h | 1 + 4 files changed, 33 insertions(+), 72 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index e9b54ea0eff0..d3778ba976a5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1156,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +#ifdef CONFIG_SPARSEMEM_VMEMMAP struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index eb339c4a71f4..4a57e6c3352c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -493,32 +493,6 @@ static bool vmemmap_should_optimize_folio(const struct= hstate *h, struct folio * return true; } =20 -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - struct page *tail, *p; - int node =3D zone_to_nid(zone); - - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail =3D alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p =3D page_to_virt(tail); - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -535,7 +509,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struc= t hstate *h, return ret; =20 nid =3D folio_nid(folio); - vmemmap_tail =3D vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail =3D vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e62e6aa07f12..70143dd8b579 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -42,27 +42,13 @@ #include "mm_init.h" #include "sparse.h" =20 -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask =3D GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order =3D get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; =20 page =3D alloc_pages_node(node, gfp_mask, order); @@ -76,8 +62,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, = int node) } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } =20 static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -184,39 +169,40 @@ static void * __meminit vmemmap_alloc_block_zero(unsi= gned long size, int node) return p; } =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zon= e *zone) { - struct page *p, *tail; - unsigned int idx; - int node =3D zone_to_nid(zone); + void *addr; + struct page *page; + const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; =20 - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >=3D ARRAY_SIZE(zone->vmemmap_tails))) return NULL; =20 - idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail =3D zone->vmemmap_tails[idx]; - if (tail) - return tail; - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + if (likely(page)) + return page; + + addr =3D vmemmap_alloc_block(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); =20 - tail =3D virt_to_page(p); - zone->vmemmap_tails[idx] =3D tail; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page =3D (struct page *)addr + i; + mm_zero_struct_page(page); + init_compound_tail(page, NULL, order, zone); + } =20 - return tail; -} -#else -static inline struct page *vmemmap_get_tail(unsigned int order, struct zon= e *zone) -{ - return NULL; + page =3D virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) !=3D NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(addr, PAGE_SIZE); + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + } + + return page; } -#endif =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct vmem_altmap *altmap) @@ -229,7 +215,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long = pfn, int node, return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); - page =3D vmemmap_get_tail(order, zone); + page =3D vmemmap_shared_tail_page(order, zone); if (!page) return NULL; =20 diff --git a/mm/sparse.h b/mm/sparse.h index b408d15baf7b..59b825df83b9 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -139,6 +139,7 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3308F3B47E2 for ; Mon, 31 Aug 2026 07:54:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162883; cv=none; b=Rf1at0BBdlFvQNXIPeF8VEtwlSB9VGDi9clvHPnt+JUd01MdtGMgJ0a3POXkxIeU9hx9enl8Cil0gNkX8cJkSRRphKvknWbtfrqmAUR7ckb1K4QBpe7cYRf/ayoAmHcYVRJvr0Aw/Wd7KhVk/MJFZ9+J97RjXLz6IvmFGcUirUM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162883; c=relaxed/simple; bh=9VZFInChYy4Z6j6P4cJOG0K1lF4h/rPu4QvJGIxjGyM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qJaHdFPDsf0YnYYEtZ3GRmIq2tqyZJC80FlrrtmaA454UdGJEnTQBn0ablkxirvZDhRzPsJSr2dudIeeLGC4Om1vkPVqCsPESw3R49RMyX0RUYoWRMWXwowdnsjSDsuEKyux+LLX+PgI7NrPUKc3ZdQtyD+0/U3+qT4jmB6oirY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=G1xxSAWX; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="G1xxSAWX" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-84e84a6c4bfso2870596b3a.1 for ; Mon, 31 Aug 2026 00:54:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162881; x=1788767681; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Ydg9K+ixOLWYvkyYLVa0DgkFjsmj2k1/+Q4MRUoiR6A=; b=G1xxSAWXM401imJZB405HrF3O3XW/X2Lj4k0FPMxnSoSeWhRs9tFIHd18E9vQXgAeb wX8/tdaLYQhOd5zCs50tXU8/P9071djjpK40BKsGoQfw1aIA0/sGpHX1KGtUJr6Tee26 zzI0cDyd5ukwKTDhasHSVH9/nGBo6QcmE1uvy26lADxOo5aQKUA2EVUkxhhsX8gPkm0m gWe0+lwBiA5FVwIa/LsDGnuKhhN6Jwl3lCPexkkNqkWFqudMXVqx3uc6wMTwSff9nHuQ APSANxk4haBQx6MtlNNvT5TraRvrMm0Fz+S4C1RqXvx4yC+aAdjwXfWP8ATG7GTuOk/y 7ogg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162881; x=1788767681; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Ydg9K+ixOLWYvkyYLVa0DgkFjsmj2k1/+Q4MRUoiR6A=; b=d8i4eXcu+sdsbWfmHDpBxr71ndPODoOY0QOw/J4J1pPQzVmwe4UpBHOBJGsM0EaCc0 iE3uRoQ7q1HwgNRycuDIPDjBQ+ubYYLMTpg8PtmQWN8Vxf9RcQISxzv6g6YQZ0Zphi7K GTNazF7xyoZ8/Xu0k6/MDcwkln0bnpwZYGK69dNvd5PCTBrqAJSH/8l2N+PSD/Xpibjb 5yvd++71ePx/gZSpB4TUOcnCXKrdLuQ9PW4kUEFJSoTBNy6Nk8Djy6feOog5wlelUJwt 0aRe8p/Xda/ei+35QgbXBtAIP5SQ5OgEE8GQamiFez9LmS7CwS25KDaTr0n/tFEC/dao PFbA== X-Forwarded-Encrypted: i=1; AHgh+RrNpWm5VjJgd3JIL7A6jhqTex3kr7gmo81BAB4OWx+AqAZWRjIiz2IscgBLzZDrDpm7Fb7ho8yBuoDcqdo=@vger.kernel.org X-Gm-Message-State: AFuF++nBqQ661bsJA27FILRNRZDU0t6nSd1mAOh88HYcpgAuc6nirH7t ETNcbxsLNVXF1p3vrsUmzHH5IJ5u827N6N2lonMzXI1Q2LkUNX6iMD8tbWcGbCEOLUQ= X-Gm-Gg: AR+sD12Yj4VAw8UFPSywIPKWOQUXUY0Au8ay5/WMWBkUDOctufvLwUPXn0Ktp1bMxjT nU5hTB82sGc1BLCgvLKYtYRN4Je8W5wYqrjUfkIu4iNuIwEVaw/maHa+MFxMnYq1+SLGu1rj1Rp WndI8GbDkAiVtN0zcFi4A2ss+1RX5yHJlJmb9YsxEw6WyCIkmbCfvSVl/esbJDMhPT3B+HPzq47 Vf/Ol0tzw+5duOG3VV1quozr6Y6c5+9aRqyC2KrabproOm1HlO5xnciKBi75L+jVhbBT6KfhM2I wgf16ouqfpBA/zj+67Ya5669u2G86JAJR2b4sBV/h0wz6xRkWbvv/r55qIkPSZ9wNFX6O/cqCKn 6olnNN+NzLONP9+dxiWWKnwnU6cop+jMOSdGxGdJ2INY4Ued9pYWJQ8Z1FrRb4AxGJZ1aAUv+er guSbnFORs0Po3jpLouMKAPNdqLE5pvZoYWqz1yjKx+koE8ALbk71AXHWYu/C7CbCV2KyIXEjdhp XD2lzuQaiC/9bpoQ92v35RV X-Received: by 2002:a05:6a00:3487:b0:842:3c77:5996 with SMTP id d2e1a72fcca58-85994491148mr5360786b3a.1.1788162881545; Mon, 31 Aug 2026 00:54:41 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.54.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:54:41 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 03/11] mm/sparse-vmemmap: open-code init_compound_tail() Date: Mon, 31 Aug 2026 15:53:34 +0800 Message-ID: <20260831075342.57563-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" init_compound_tail() is only used by vmemmap_shared_tail_page(), where the shared tail page setup intentionally passes NULL as the compound head. Keeping this helper in mm/internal.h exposes that special case to the rest of the MM code and can make the NULL head argument look generally valid. Open-code the initialization at the only call site so the special-case use stays local to sparse vmemmap optimization. No functional change intended. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- mm/internal.h | 9 --------- mm/sparse-vmemmap.c | 5 ++++- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index e16f1250b25c..5ddb49680b06 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -783,15 +783,6 @@ static inline void prep_compound_tail(struct page *tai= l, VM_WARN_ON_ONCE(tail->private); } =20 -static inline void init_compound_tail(struct page *tail, - const struct page *head, unsigned int order, struct zone *zone) -{ - atomic_set(&tail->_mapcount, -1); - set_page_node(tail, zone_to_nid(zone)); - set_page_zone(tail, zone_idx(zone)); - prep_compound_tail(tail, head, order); -} - #if defined CONFIG_COMPACTION || defined CONFIG_CMA =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 70143dd8b579..e453ce4675a0 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -189,7 +189,10 @@ struct page __ref *vmemmap_shared_tail_page(unsigned i= nt order, struct zone *zon for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { page =3D (struct page *)addr + i; mm_zero_struct_page(page); - init_compound_tail(page, NULL, order, zone); + atomic_set(&page->_mapcount, -1); + set_page_node(page, zone_to_nid(zone)); + set_page_zone(page, zone_idx(zone)); + prep_compound_tail(page, NULL, order); } =20 page =3D virt_to_page(addr); --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B5C327E1DC for ; Mon, 31 Aug 2026 07:54:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162888; cv=none; b=DKa9Xj32QMIz+qoS61Ctpa1fE4xekWkYWmw1WJNGslfaBs08Q6WcoQOT31RSQjDgP8iPp+sTEQGHClTV3xn335wZGEB6CaDVT1hJdLOla1672aoCUICLSP7kb2/Ly01OxAy4X+p40eTBKUNZVL6ZYrj4smhqJctPytdo6SmEuIE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162888; c=relaxed/simple; bh=mY7MmVnPZ0lPBAqPB17Huo/jDkdlxRi5TewX6ZYgDYc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=onZKsZQaNkpYfyMwy9+gKGZiZAGL8HCjvmrSx2l+BVDR+wMQe1/yYsaIgqgYg9hsewfC4TpfkigobOCvlQmar0tZaVeWpH1hxUb2UO1ZvR1ZL++uIhg3fWQud6sJqEplfXconl408BGgT0z33cNttigGyOU/42b+YSqfAC+86GY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=AmL9Fh3H; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="AmL9Fh3H" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-84eb992a881so2783749b3a.2 for ; Mon, 31 Aug 2026 00:54:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162886; x=1788767686; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YVp9a6es5dGYiwq82JtXpVvKTCPlSq+5R8j2MYX8xm8=; b=AmL9Fh3H98UklS9mymoy6+g8qXT9925TQgnkQs4OL2up4icZto/erQGtDQZkiRFt0h HK9pXgFiH3xZMmQIA8ieMcGITKuuiPEisIfNgANVmXvOa4FQOb+I27bXHPEV4TfFHXIk 27U4wByn8zux5v5kABMERyIOIyzyk3GrBpBf7Qcw5EzGaUUd2DGbfR30uEqpBoNd7nyR kHulIgii41k64Qg/0P4mNJ+gEAnCTN9AqdBRsuFfb90Jj24aehZx5EElU6AK/tPNWBJb BRhPRzHc735yPaXC0BcDc+KXl3iAScoJNJdb3ON33D4vWkxcDwh4MMCYgKUcU9pD2f1e 2sTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162886; x=1788767686; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=YVp9a6es5dGYiwq82JtXpVvKTCPlSq+5R8j2MYX8xm8=; b=RptQmCDotPLtWfLCCIb7CqzOztvnC0kHIWTAd3vrA1Dkvn91EaartdJ6R8AA4m39pP cDHI1kHcpHq3w9Zo0WSmxXeHG/dYROkNx0TVQbMbF4r1NwMM5KJxYOldtI+qZMsNujo6 YlAX4w5CHtX+2Jxp+hB84hi/P3bcGZ2pJ3g2Kmx9qi6vtwYbYWo6ce4TPD1N3e6VSjcz J7e4H7slnqFxbpWvlqOdaC5ChRx0Lm4pqYDbjvM5BuchBz8y0FZnrpoA/MuKfjmLT+zn kZ4wsLEJG8pgkcM2HOjqWkiw9A7NToB96DmbPTV+GjH37X77aeRjmqSxF5KaD6xF20FN Z5jg== X-Forwarded-Encrypted: i=1; AHgh+RppCArlUNzooB712vzPoKK5ZPeBaTPYbDqrWe9vvqqGvCrxhTwDrSbe2wxxOnTYEH0lN82bMkkMgWvuDx0=@vger.kernel.org X-Gm-Message-State: AFuF++lCgj78cBPW0mP6EukwuW5ThpubU50JqZo6l36EdEWA0uOnnIvO DGA7HKsoo7wNDx0qrG7WpMK8khZRmIeyz9OcdeWdNT0aasNVA4yqb//p9s7b1HVbPlE= X-Gm-Gg: AR+sD11x6i0c8m54ETl4OAS3ALubCu/Mf4MmOGHd5AM2A5cQVHZ//JmgtD0twZ+xR28 uDw9ZYMgQbNu/wmsJ7EQe/PSxaUPChjnRlz5fb5xOcFHE3k/voeC/vOa1L/ek5GIcdf1VhtVRqv gR19jO44lQTccg4OwhZON0Df7VykepjUNz3AeT9L4wN6OwifOal2EcFMjObimAEbbpKU6Zw/4by k5o719YwhlncatQKLKcOYWik2WolyuFV3fkVqYIRNGApqIda1DR3GCryFjwVhGvhUfJjbZxFDeU mzaGajaBiuFo1MoKH3l7PSdHrOqn4cS8UgXGCuCzc/na/wBORUboEU60E/d1bz9pgIPK09npyss QmWwaDyIdv+AVpP5sZK0+RIvtiTLU+8hZ3JzIfd0yzUKwPAW5dPuxsaww+IDvoX66ljjG2/k63R k2idyRNUb95by7p1XdiawEIXv6HjrIcgSRj6FvrckkDETWOTBAG+G0OedbzwFBjejqdlujiENTy e7vlXR9F3T4t6QmOxWAD3hF X-Received: by 2002:a05:6a00:1706:b0:851:c1d2:c48d with SMTP id d2e1a72fcca58-8562a1f8fd6mr37313481b3a.8.1788162886422; Mon, 31 Aug 2026 00:54:46 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.54.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:54:46 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 04/11] mm/sparse-vmemmap: prepare DAX vmemmap population for section orders Date: Mon, 31 Aug 2026 15:53:35 +0800 Message-ID: <20260831075342.57563-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX still uses vmemmap_populate_compound_pages() to populate its compound-page vmemmap mappings. That helper allocates the head and first tail vmemmap pages explicitly, then reuses the first tail page for the remaining tail page mappings. Device DAX is being moved to the section-based vmemmap optimization infrastructure, but it cannot switch to the generic section-based population path yet. Once a later patch records the DAX compound-page geometry in the section order, DAX head and first-tail PFNs can look optimizable to the generic helpers as well. Add a DAX-specific population flag for this transition. It keeps DAX head/first-tail allocations on the normal vmemmap allocation path, while preserving the existing page reference for reused DAX tail mappings. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- mm/sparse-vmemmap.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e453ce4675a0..54ae8c284324 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -35,8 +35,8 @@ /* * Flags for vmemmap_populate_range and friends. */ -/* Get a ref on the head page struct page, for ZONE_DEVICE compound pages = */ -#define VMEMMAP_POPULATE_PAGEREF 0x0001 +/* Vmemmap population for ZONE_DEVICE compound pages */ +#define VMEMMAP_POPULATE_DAX 0x0001 =20 #include "internal.h" #include "mm_init.h" @@ -208,13 +208,17 @@ struct page __ref *vmemmap_shared_tail_page(unsigned = int order, struct zone *zon } =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, unsigned long flags) { struct zone *zone; struct page *page; const unsigned int order =3D pfn_to_section_order(pfn); =20 - if (!vmemmap_optimizable_pfn(pfn)) + /* + * Device DAX still relies on vmemmap_populate_compound_pages() for + * head/first-tail allocation and tail-page reuse. + */ + if (!vmemmap_optimizable_pfn(pfn) || flags & VMEMMAP_POPULATE_DAX) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); @@ -236,7 +240,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in pte_t entry; =20 if (ptpfn =3D=3D (unsigned long)-1) { - void *p =3D vmemmap_alloc_pte(pfn, node, altmap); + void *p =3D vmemmap_alloc_pte(pfn, node, altmap, flags); =20 if (!p) return NULL; @@ -251,7 +255,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in * and through vmemmap_populate_compound_pages() when * slab is available. */ - if (flags & VMEMMAP_POPULATE_PAGEREF) + if (flags & VMEMMAP_POPULATE_DAX) get_page(pfn_to_page(ptpfn)); } entry =3D pfn_pte(ptpfn, PAGE_KERNEL); @@ -511,6 +515,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, unsigned long size, addr; pte_t *pte; int rc; + unsigned long flags =3D VMEMMAP_POPULATE_DAX; =20 if (reuse_compound_section(start_pfn, pgmap)) { pte =3D compound_section_tail_page(start); @@ -522,8 +527,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, * with just tail struct pages. */ return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); } =20 size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); @@ -531,13 +535,13 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, unsigned long next, last =3D addr + size; =20 /* Populate the head page vmemmap page */ - pte =3D vmemmap_populate_address(addr, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(addr, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 /* Populate the tail pages vmemmap page */ next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 @@ -547,8 +551,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); if (rc) return -ENOMEM; } --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C2963C9EF6 for ; Mon, 31 Aug 2026 07:54:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162896; cv=none; b=lKFy5lW/R5N48gmXp6V7bCUybVe4NEYBUjNbHs4Ih38IrVq2KAz9BgQfhZuaGcOC2pUPypJNcpgOmzRw/panduycGP3nzL8ELiF4GXf4pWj7N0XNo82i2pY996iO/jRip8B72maklliCVERvnNS0qVgyOfWFW6gTaMDMsq1Tf+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162896; c=relaxed/simple; bh=Yhk2qT4J6yuXuggajr5zs+V+I0LBgnI5qbnHpBCNQE8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JvJ1wdI+c8GMi9MUMs8vdC9zZl8tf89ZdTrWOmiLqamX4zrkDu5kxMQ0FiCH2rpLhzWRIntsNdg3ecGBWNgRJi9fnl0595k5P6I56Htjic6trYt+aDNViJ13wa2JMvl52bFBRtoevjSGZU9WALQZuCijOLTwmvxmaGp08NM3//w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=fcPI9HWq; arc=none smtp.client-ip=209.85.216.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="fcPI9HWq" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-38dc4553f62so4488842a91.0 for ; Mon, 31 Aug 2026 00:54:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162891; x=1788767691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=d85H49jI1quBgxYs+PQ2R3ebCcg9h4JyI/UtxSGskms=; b=fcPI9HWqYXp4dM9ElRdrypDs3QU5JD2kSaGVRfeb9GLY/5dqGftchpwoQ4wioV0NJK hYeZwUuH77pRyovM0aPVH6+sORs332Ri9L+XNBL4WqqhoQlYXAMnBZGa+5HnBEKNctMM yCvvX2mUbrtKBsKpexI8AlTpP1p6Q2SWtWY/pKbjbzPKAvaspQm+MZjj3qdTpbbDEqVR AogoBSfkHezxxlesfuPp0sm7Bl2s8FViLCtporgIIbK7e6WEkoAZwhJ0F+AIfpEDiyhE kEM3b07/X2HdSfVq05fnWNhQHyOgVpfC2D6RmhKJ7trNWpfIXUGfc66CuWvyNMLR/KmZ jZJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162891; x=1788767691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=d85H49jI1quBgxYs+PQ2R3ebCcg9h4JyI/UtxSGskms=; b=Npl1jDe4LBNA5ebJuqJxyUSHfmmx3CS+EqjBv12OXnUqcrl7t+tjMVNihVXyvYwGoZ 45WvxdWYveD+m3GuWqgS0fBWrYVVsqne/BUxc/1NEtA1kGku5AwtVCImreuTZFKyYEmu AYzm6vBpPzkT3RKvc8pu/jF5NL4DH2NcWzJ/rUhN063eolTiTMO4s1mQvMwN+Dhx9EPk 7v+ZB3XmjnIPT+K/AIj9FwVZIE05ihXKaep8DntvYSFSCT84HRMGDT6ddvhFeD3VTysd j3Vk7eTKL+/BdpB/xxOVIvSmZRTLE5spI3LiV+ODMyOT879B1ZS2qG6kW+AtCCgma3EA /UJQ== X-Forwarded-Encrypted: i=1; AHgh+RrKfKDyiDgZiL+SiGXE/VaSRAApFeGscuKSEFbadwKf9afoJpRgj4umB1E8nC8VGQVOH8TZAVlJ/sKr/yQ=@vger.kernel.org X-Gm-Message-State: AFuF++nxW/vXts5Yq4NOfdKr1J0S7UGzTU4Zu8eABX8qeBBSV3+8lYpu oBT7sCIcRqF9/NMV2sOdsDh5mWWxeFRhyykJ9joNlQ+Ib+1sn0EKWvyPEKCWgjCkkTw= X-Gm-Gg: AR+sD12vV5nvlynwCC3MKvbMb6IMVKqe0qt+SbKlFEjxtZN01GJUsIKDKqQ0NzGhzp8 NF6RwX0hA4LTh6flr+z5AGs7WSUmHBiv+lnXXaZvfCrNl4TGulc0BZWPBhM9ZkvqgWo16TkS2n+ AE25Rh0i38i9gdcK7rU6Amv2SN4PJBQUst08FWX1E5YQK6jfPzCS/rKpZg/4s56u0gzWFYgbjGi lC9pULXQUvG2nkDc9aKNSYVZzgLXhCqUYUgOLwJsqxDoXm3623EnY0anjEigCx80qzqnoqkkOBX Ek/832HupnKELCW+xyZkdjSjHa6AoGIKGhFH8tjQKEZMM6EIRVEHNZdGB5qm1yd0D1868E07YtC hugtC2yzmbT1UH/x9vNrzsO5GfEnHEQYBPrKuQuI4+ZI57tYCPO7nRFUMfBwZonEWblU0lwywny FcYuwllbtGDDBn4NRAhyHkjE+b9Eiiyd7xVFHde3XKiQTzjJF+NWAO2vqSHEM8xFbHwiYLuyhyR Wx/Pp8wMkyt8F+xFFIP2jY1 X-Received: by 2002:a05:6300:220d:b0:3cc:faf3:331a with SMTP id adf61e73a8af0-3d2686aa479mr43643171637.10.1788162891372; Mon, 31 Aug 2026 00:54:51 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.54.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:54:50 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 05/11] mm/sparse-vmemmap: set section order for device DAX Date: Mon, 31 Aug 2026 15:53:36 +0800 Message-ID: <20260831075342.57563-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX can use vmemmap optimization only when a full section is populated with a compound-page geometry. Record that geometry in the section order before populating the section, so later vmemmap accounting and population decisions can use the section state directly. Clear the section order when the section becomes empty again. Also reject partial additions to a section that already has optimized vmemmap mappings, because a section cannot safely mix optimized and ordinary vmemmap layouts. Partial additions continue to use ordinary vmemmap population, so they do not save vmemmap memory. Such additions are uncommon, and the lost saving is negligible. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- mm/mm_init.c | 13 ++++--------- mm/sparse-vmemmap.c | 16 ++++++++++++---- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 33ff95141adb..c4cd61978ce8 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1042,16 +1042,11 @@ static void __ref __init_zone_device_page(struct pa= ge *page, unsigned long pfn, * of an altmap. See vmemmap_populate_compound_pages(). */ static inline unsigned long compound_nr_pages(unsigned long pfn, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - /* - * If DAX memory is hot-plugged into an unoccupied subsection - * of an early section, the unoptimized boot memmap is reused. - * See section_activate(). - */ - if (early_section(__pfn_to_section(pfn)) || - !vmemmap_can_optimize(altmap, pgmap)) + const struct mem_section *ms =3D __pfn_to_section(pfn); + + if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); @@ -1121,7 +1116,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; =20 memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); } =20 pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 54ae8c284324..aed1e7429daa 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -135,14 +135,14 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms =3D __pfn_to_section(pfn); - const int order =3D pgmap ? pgmap->vmemmap_shift : section_order(ms); + const int order =3D section_order(ms); const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); =20 - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) + if (!section_vmemmap_optimizable(ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); =20 if (order < PFN_SECTION_SHIFT) { @@ -573,7 +573,7 @@ struct page * __meminit __populate_section_memmap(unsig= ned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; =20 - if (vmemmap_can_optimize(altmap, pgmap)) + if (pgmap && section_vmemmap_optimizable(__pfn_to_section(pfn))) r =3D vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r =3D vmemmap_populate(start, end, nid, altmap); @@ -792,8 +792,10 @@ static void section_deactivate(unsigned long pfn, unsi= gned long nr_pages, else if (memmap) free_map_bootmem(memmap); =20 - if (empty) + if (empty) { ms->section_mem_map =3D (unsigned long)NULL; + section_set_order(ms, 0); + } } =20 static struct page * __meminit section_activate(int nid, unsigned long pfn, @@ -803,8 +805,13 @@ static struct page * __meminit section_activate(int ni= d, unsigned long pfn, struct mem_section *ms =3D __pfn_to_section(pfn); struct mem_section_usage *usage =3D NULL; struct page *memmap; + unsigned int order; int rc; =20 + order =3D vmemmap_can_optimize(altmap, pgmap) ? pgmap->vmemmap_shift : 0; + if (nr_pages < PAGES_PER_SECTION && section_order(ms)) + return ERR_PTR(-ENOTSUPP); + if (!ms->usage) { usage =3D kzalloc(mem_section_usage_size(), GFP_KERNEL); if (!usage) @@ -830,6 +837,7 @@ static struct page * __meminit section_activate(int nid= , unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); =20 + section_set_order_range(pfn, nr_pages, order); memmap =3D populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap, pgmap); --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE7ED3C9ED6 for ; Mon, 31 Aug 2026 07:54:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162898; cv=none; b=q7VfVbTaVd+qy/S7teLqX2paClqH92+PxrAMQyUADRq76AVRfjaihiy+pSnQW9XdGwGaqyD8bl0LcRE64aojfOqYdUojXK/WUelfV1Vt1+L8l+stmZcIaS5d1bh4d82V0LX330oK5Bhsrp1fHoAKzOa3JuzoASYRN6y77iQHR18= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162898; c=relaxed/simple; bh=01t7KBgXm2nUEn5QJqpbkkRv+YguN+ZQDkImEkyxlKY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nwSOezsTE5wnZFzCW9swrxj0qvTVsFwC+cRr0nYmskcP6NSIQBGpe8O2lvc2UhNIbCKgUNWgjbdCYgJOjFUOEfc/TYYN6KFh8L04kH8B8G8Y8UKvLXciHLXNlNkIsv0Wzbu1FJpJMwZIsxzGTqANwaOBuDK/Fkp5nb9AyG1aXyE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=lYhd7iKU; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="lYhd7iKU" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-8556ec44e9aso2895109b3a.3 for ; Mon, 31 Aug 2026 00:54:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162896; x=1788767696; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=RFw62PZ7fKSG6PmMZYyDraeWannjZPtyw6lLCfcFAXM=; b=lYhd7iKUPS2I3n+cVUUaAp3KpgZRlpa8Qf6Rfln4oMb7KPYXBK3PNb03S4QLHIh2gC zt1WnmPVDk93/01ux/sMB2UOTkjAfQ+mBkKI1kFWyYWABrRssBdEfm1ni/Yfla0iZFeq 7/yGCKqcyf86Mgi/ATGuu5eQzB/WgF17a+2sE/W7pwS13l/StI5rnJiv7KgPGNO/Z5oL aRs4BTXTHc6bNH5GIMrs2Au6NQLU2Zb4hH2dbELhCGN4RYQvRxhmtWoIXgsr3r2A7u06 yqUTxYs68E2T2FUPwmX+yCM6pOkz9q6bV1nCMeRaEplZtu6jQJA8r5whMinJJb0D0JPG jp/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162896; x=1788767696; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=RFw62PZ7fKSG6PmMZYyDraeWannjZPtyw6lLCfcFAXM=; b=flyr6A7CSt/UlNVK49p+h9q0qeqR+rDC8+p7YvQABIIJS5fPU2HXbJPDqmkJ5zbJdn lxTFChkO79XO4bbr/10y744LVUmFePg+b2mrPcvzQEbZSLy0bo9iqwUUd2jTeFUX/lTl HOdRgi/gBhNwcYBC4a6qpo8N2zNjrN9PQ1ZssZAHL5mV9AZsoLHbeVI0bwW+6BMZVzIU Ya0MLY/ideYVduKLIabM7OktR2q8u0f5aL9JiIo8pRs28MxULQlHWufYLSeNgJ+4AVwL FeFTC3VAH+nnX/ayWENRweBPWTXzdG1oDwTPhEurRC4HPlonpsgnmoEABprhs/ywFBie KWJg== X-Forwarded-Encrypted: i=1; AHgh+Rp+Hd//bnI2DmZWI6sEDLaW2QAZEDokkgpk60mJbPNgzY6ckgPm8DZ/HWxkTLhcZB0P2YMyDFe8P03sAZ8=@vger.kernel.org X-Gm-Message-State: AFuF++lrwXtgc6l4jZHyh7xkV10Os8ePRPTKPcaTG0lO8Ieoj+Id0B4e LlX0yDx1OuMDLe60Uo0BJGR6oKr10MCiBKLHxBEskL6Ye/UOPP0PvWZDryPXWK6Wk4c= X-Gm-Gg: AR+sD11S9QN8K0O2C+eOByjgtOffpZmlk1uyVi/hwTud3Yo4yJai1feKl4IH3NLM3Qr wMxIDUOAzVpifo8mou8FTt/5VKKeondfDR+1Ou3aKEB3vObsfS8vEwm57gW68x0U1p6Qga6FLcL 2jRXZP/TfBikkJhGOzQMuzhDsEDAowpPL9tekn4pH6utpTRt3JLB7rCUIho3dvrwTG5/jno0VSt 0OrqISATYBHYsBkkiESXG+m0rmheN9i2fGVhVZaPIP+EZpe+J/NfpEEXNlT2AnwIAAC1IZoMapi KHrV3xrHq1yw9fNSaKFKbfh0n7SLMDHhEZOaFeHALS36uuV1/9+3t6rzi+ezbq43vqn5MdkSOAa U8Xzyl5s37MieAhLoLTE7/NYmjJpyierWK8L3LhBL/YSFyJYvHpvZR/vH6DxloztS02B419PlMT 3xRLDzdGlQqFHrIzHQBJOMnvS41z9Hed1cI3y7Zf+id7XPCURtMvAvc+tL6ZZBbqfKmlKr/qMph EKhBeqJWEboxQPmZsECs2D+Gy+LrE/wvuI= X-Received: by 2002:aa7:888a:0:b0:84f:dd30:1890 with SMTP id d2e1a72fcca58-85adb7bdb1dmr2240316b3a.15.1788162896019; Mon, 31 Aug 2026 00:54:56 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.54.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:54:55 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 06/11] mm/sparse-vmemmap: switch device DAX to shared tail vmemmap pages Date: Mon, 31 Aug 2026 15:53:37 +0800 Message-ID: <20260831075342.57563-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB vmemmap optimization now uses per-zone shared tail vmemmap pages. Device DAX has not been switched to that mechanism yet. Switch device DAX to vmemmap_shared_tail_page() as well. This aligns DAX with HugeTLB by using the common per-zone shared tail vmemmap page. Since the shared tail page can now back ZONE_DEVICE vmemmap mappings, initialize its entries with PG_reserved for device zones. Also skip poisoning vmemmap-optimizable sections while their struct pages may be shared. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 10 +++++++++ mm/memory_hotplug.c | 5 +++-- mm/sparse-vmemmap.c | 47 ++++++++++++++---------------------------- 3 files changed, 28 insertions(+), 34 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d3778ba976a5..03ed9232f186 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1686,11 +1686,21 @@ static inline bool zone_is_zone_device(const struct= zone *zone) { return zone_idx(zone) =3D=3D ZONE_DEVICE; } + +static inline struct zone *device_zone(int nid) +{ + return &NODE_DATA(nid)->node_zones[ZONE_DEVICE]; +} #else static inline bool zone_is_zone_device(const struct zone *zone) { return false; } + +static inline struct zone *device_zone(int nid) +{ + return NULL; +} #endif =20 /* diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 226ab9cb078a..d28bafb6fd53 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -554,8 +554,9 @@ void remove_pfn_range_from_zone(struct zone *zone, /* Select all remaining pages up to the next section boundary */ cur_nr_pages =3D min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - page_init_poison(pfn_to_page(pfn), - sizeof(struct page) * cur_nr_pages); + if (!section_vmemmap_optimizable(__pfn_to_section(pfn))) + page_init_poison(pfn_to_page(pfn), + sizeof(struct page) * cur_nr_pages); } =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aed1e7429daa..0201877a7f80 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -193,6 +193,8 @@ struct page __ref *vmemmap_shared_tail_page(unsigned in= t order, struct zone *zon set_page_node(page, zone_to_nid(zone)); set_page_zone(page, zone_idx(zone)); prep_compound_tail(page, NULL, order); + if (zone_is_zone_device(zone)) + __SetPageReserved(page); } =20 page =3D virt_to_page(addr); @@ -490,23 +492,6 @@ static bool __meminit reuse_compound_section(unsigned = long start_pfn, return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; } =20 -static pte_t * __meminit compound_section_tail_page(unsigned long addr) -{ - pte_t *pte; - - addr -=3D PAGE_SIZE; - - /* - * Assuming sections are populated sequentially, the previous section's - * page data can be reused. - */ - pte =3D pte_offset_kernel(pmd_off_k(addr), addr); - if (!pte) - return NULL; - - return pte; -} - static int __meminit vmemmap_populate_compound_pages(unsigned long start_p= fn, unsigned long start, unsigned long end, int node, @@ -516,21 +501,18 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, pte_t *pte; int rc; unsigned long flags =3D VMEMMAP_POPULATE_DAX; + struct page *page; + unsigned int order =3D pfn_to_section_order(start_pfn); =20 - if (reuse_compound_section(start_pfn, pgmap)) { - pte =3D compound_section_tail_page(start); - if (!pte) - return -ENOMEM; + page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!page) + return -ENOMEM; =20 - /* - * Reuse the page that was populated in the prior iteration - * with just tail struct pages. - */ + if (reuse_compound_section(start_pfn, pgmap)) return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), flags); - } + page_to_pfn(page), flags); =20 - size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); + size =3D min(end - start, (1UL << order) * sizeof(struct page)); for (addr =3D start; addr < end; addr +=3D size) { unsigned long next, last =3D addr + size; =20 @@ -546,12 +528,12 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, return -ENOMEM; =20 /* - * Reuse the previous page for the rest of tail pages + * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), flags); + page_to_pfn(page), flags); if (rc) return -ENOMEM; } @@ -883,13 +865,14 @@ int __meminit sparse_add_section(int nid, unsigned lo= ng start_pfn, if (IS_ERR(memmap)) return PTR_ERR(memmap); =20 + ms =3D __nr_to_section(section_nr); /* * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - page_init_poison(memmap, sizeof(struct page) * nr_pages); + if (!section_vmemmap_optimizable(ms)) + page_init_poison(memmap, sizeof(struct page) * nr_pages); =20 - ms =3D __nr_to_section(section_nr); __section_mark_present(ms, section_nr); =20 /* Align memmap to section boundary in the subsection case */ --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68D043B2FC8 for ; Mon, 31 Aug 2026 07:55:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162903; cv=none; b=IAYEYgOUZ4QiRTZIwkrFpDDP93ut/TAX68vDIznRUl5riYO83YZAxkEkk597+hiARVuJytPW5uLz7+YPQeIBCAYuEhFNpJb4ofpwPzxpx1eOYd08wSa0sP7I3ZrJ+wGRPEoCHQ1+mRUxBesFqyS7VWYIe1dpEo3AfiaXuZPfdZM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162903; c=relaxed/simple; bh=0fzWxzOVe0HaP6mmmUOGPsdZyD1efaBndlEqFksa+Qc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hOM2M1m5O8TsoZ3kjhRaomYeNfssvBkQND9vHD7pr3P19TXDwvJlX9WPU+10HseDVqQCOrT8KR+U69fH15/QwquSv1i4WxWO/HX3rke79IsScg0E6gZ8hT4oEfNap1vjJ58J6AEkgvFXUFFlK90BGYT3HECvo20lXvfgRvgHNek= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=cZJrTZgn; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="cZJrTZgn" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-cb5b8572b70so3268215a12.2 for ; Mon, 31 Aug 2026 00:55:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162901; x=1788767701; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4fLjmUTfKT3TRPIaKPcH4qoh94gQxw3yJaaLiOmIzFs=; b=cZJrTZgnEHnLecBRf0PB4/2x3FsLuH/pnMShQAmWNHYD55VzoqNU+iQ5uQmDWp64Sv fiuFEhCyNimA8bo9BITSOy16AXKCDWRYIi/MPz2kmlfEWVLnx5u/DeQxA+mLWxY18HF1 Akammz2UppYDXnQfaBLKs2m+auisWlbmtY62FDE5frKpYoF4U47GPkmROe73cVGgE4F+ cX3CgUP3CvDr9Z29PwFFfqOcTTYf3W/s09UQoWzsy8b1PSX5gmbzyE1ar8gvEx1q8RRb kLPNBmb2d0R6rW9tO6Zwqnxyud/4pu37oar9gm9IpOHjmEowQI095K8/73wBk33jkYxe hEqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162901; x=1788767701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=4fLjmUTfKT3TRPIaKPcH4qoh94gQxw3yJaaLiOmIzFs=; b=LXpvd5bpQESTCaXuqPisVBT4pk0H6GQIZDOyPQU8QMLBlWXAoNMpkTo/m6CiozrmH3 xNHzlvxOFWtewJG0OxUwbKaCvmb48zr9B5UHXLfDjfsBX3SO0GeXV2wG6SLWn+1ReCvO ey3PYzoMROr8+XgxmPv0MqP5fZchdV0y8v4iDxYYMPePKY54OVATQ1c6TX4L7ANMQYkv JJraMVZ9f4AUv0M4hgeZMeCBuTVG+irvxwbI2ukqPRzpDzHRo5Tu+cUjPTjJZCpehIvI lmuPToVbiLk3Ot30UvZpo56MOIP4su8Jy73OOSSIZ8OB/Wpg9J1BSGd+xF3MAIx9R/PI gzng== X-Forwarded-Encrypted: i=1; AHgh+RoQOMPHsJDXWIgPRNHnNziSYsinDYKlkZOz1RrxIbQKkwzD99j9gEedPM6GBiiyYVOzdroBnMj0DxYZ+jI=@vger.kernel.org X-Gm-Message-State: AFuF++k2QTnuQeeqCv6AtjwLFtK8f9oxzpjzVAPAIkTrCp2rlDIDD3S2 11RVpofbUgq31eV+gkDJ66afRI4m38Dj2puA+2kUQ+lLsIRjTys+KqLubmcWOtnEt5s= X-Gm-Gg: AR+sD10t35KfVH+fuGRicJ9FDfFEt9MYxoigSxblboSOQrW5HaBmRxNOVfPOs/h/ZIM YB8Ad17tFyp9pXUeIhyUD8aeCUETiv5Qk+Ay+OwiZPsEXXU/Vbk/YM/ByqPO7Dv7uKsuiH783nc vhITAAVkBumqGAlDg99rTdPT0J84gYflYRKbvX9MOwLq17P1Y3r2B/tw6NmQQj07zyBUkl/gWrh bv+Nx/L4sTgJHv7DWxt25sww2rz1pPQsFHAOn8SfgwHAU8Gt8picOM8wiQ2VUdim/P/hX604gwg XPLbQi9l2+EvZdzoRb3H1hKI+zPVRjtFCqqxdpDEE1Y0Go2MHGKRDBpJZRdcUg6bMToYUQfb7yO 1krAO6ehTODK37h+KoJldiQ3JU08Gb/dJxukKlPKF+ugfm/rryeTQPxvHMVzhlSOhKoQFuZ+0z2 SQCSAOZf6ROtb+k5nZBP7irZEQBbS/MnNmOjqz5YAoM5FS51G6E0n0gJlx5fzwdSfwpQVliQvun h17TlCH60XZI4aCji/dUH3t X-Received: by 2002:a05:6a21:a09:b0:3b4:61f:1fec with SMTP id adf61e73a8af0-3d26668474amr44495998637.2.1788162900583; Mon, 31 Aug 2026 00:55:00 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.54.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:55:00 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 07/11] mm/sparse-vmemmap: move HVO helpers to a public header Date: Mon, 31 Aug 2026 15:53:38 +0800 Message-ID: <20260831075342.57563-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The vmemmap optimization helpers currently live in mm/sparse.h, which is an internal MM header. That works for MM code, but prevents powerpc from using the same interfaces without including a private header. Move the declarations and inline helpers to include/linux/vmemmap-optimization.h. This is a preparatory change for powerpc, which has its own vmemmap optimization implementation and needs to use the HVO interfaces from architecture code. Signed-off-by: Muchun Song --- MAINTAINERS | 1 + include/linux/vmemmap-optimization.h | 88 ++++++++++++++++++++++++++++ mm/hugetlb.c | 2 +- mm/hugetlb_vmemmap.c | 2 +- mm/memory_hotplug.c | 1 + mm/sparse.h | 73 +---------------------- 6 files changed, 93 insertions(+), 74 deletions(-) create mode 100644 include/linux/vmemmap-optimization.h diff --git a/MAINTAINERS b/MAINTAINERS index a3167bcbe589..ccc7f75bbeb0 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12099,6 +12099,7 @@ F: Documentation/mm/hugetlbfs_reserv.rst F: Documentation/mm/vmemmap_dedup.rst F: fs/hugetlbfs/ F: include/linux/hugetlb.h +F: include/linux/vmemmap-optimization.h F: include/trace/events/hugetlbfs.h F: mm/hugetlb.c F: mm/hugetlb_cgroup.c diff --git a/include/linux/vmemmap-optimization.h b/include/linux/vmemmap-o= ptimization.h new file mode 100644 index 000000000000..171553fabd47 --- /dev/null +++ b/include/linux/vmemmap-optimization.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * vmemmap-optimization.h + * + * Generic vmemmap optimization declarations. + * + * Author: Muchun Song + */ +#ifndef _LINUX_VMEMMAP_OPTIMIZATION_H +#define _LINUX_VMEMMAP_OPTIMIZATION_H + +#include + +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +static inline unsigned int section_order(const struct mem_section *section) +{ + return section->order; +} + +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ + VM_WARN_ON(section_order(section) && order && section_order(section) !=3D= order); + section->order =3D order; +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ + unsigned long section_nr =3D pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_order(__nr_to_section(section_nr + i), order); +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return section_order(__pfn_to_section(pfn)); +} +#else +static inline unsigned int section_order(const struct mem_section *section) +{ + return 0; +} + +static inline void section_set_order(struct mem_section *section, unsigned= int order) +{ +} + +static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, + unsigned int order) +{ +} + +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return 0; +} +#endif /* CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION */ + +static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +{ + const unsigned int order =3D pfn_to_section_order(pfn); + const unsigned long nr_pages =3D 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + +static inline bool vmemmap_optimizable_order(unsigned int order) +{ + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) + return false; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; +} + +#ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); +#endif /* CONFIG_SPARSEMEM_VMEMMAP */ +#endif /* _LINUX_VMEMMAP_OPTIMIZATION_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8fa1bafa03d9..5aa9c0eaf2d0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -38,6 +38,7 @@ #include #include #include +#include =20 #include #include @@ -52,7 +53,6 @@ #include "hugetlb_cma.h" #include "hugetlb_internal.h" #include "mm_init.h" -#include "sparse.h" #include =20 #define HUGE_BOOTMEM_ZONES_VALID BIT(0) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4a57e6c3352c..25c4e7d2664c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -15,10 +15,10 @@ #include #include #include +#include =20 #include #include "hugetlb_vmemmap.h" -#include "sparse.h" #include "internal.h" =20 /** diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index d28bafb6fd53..4f3387632883 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -43,6 +43,7 @@ #include "mm_init.h" #include "page_alloc.h" #include "shuffle.h" +#include "sparse.h" =20 enum { MEMMAP_ON_MEMORY_DISABLE =3D 0, diff --git a/mm/sparse.h b/mm/sparse.h index 59b825df83b9..e511d99fc26b 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -9,77 +9,7 @@ #define __MM_SPARSE_H =20 #include - -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION -static inline unsigned int section_order(const struct mem_section *section) -{ - return section->order; -} - -static inline void section_set_order(struct mem_section *section, unsigned= int order) -{ - VM_WARN_ON(section_order(section) && order && section_order(section) !=3D= order); - section->order =3D order; -} - -static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, - unsigned int order) -{ - unsigned long section_nr =3D pfn_to_section_nr(pfn); - - if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) - return; - - for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) - section_set_order(__nr_to_section(section_nr + i), order); -} - -static inline unsigned int pfn_to_section_order(unsigned long pfn) -{ - return section_order(__pfn_to_section(pfn)); -} -#else -static inline unsigned int section_order(const struct mem_section *section) -{ - return 0; -} - -static inline void section_set_order(struct mem_section *section, unsigned= int order) -{ -} - -static inline void section_set_order_range(unsigned long pfn, unsigned lon= g nr_pages, - unsigned int order) -{ -} - -static inline unsigned int pfn_to_section_order(unsigned long pfn) -{ - return 0; -} -#endif - -static inline bool vmemmap_optimizable_pfn(unsigned long pfn) -{ - const unsigned int order =3D pfn_to_section_order(pfn); - const unsigned long nr_pages =3D 1UL << order; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; -} - -static inline bool vmemmap_optimizable_order(unsigned int order) -{ - if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) - return false; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; -} +#include =20 /* * mm/sparse.c @@ -139,7 +69,6 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP -struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCBE63CAA3A for ; Mon, 31 Aug 2026 07:55:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162907; cv=none; b=CP1ghql/g/MPPt+gakoXyaD/1IHLkmBMWTvYufByxFHnxQ4ViKAAbZuNj9pQnHw1glr4sTqTFftMxUBRLSVhj6yn9yOyCBpT7X4KRhpKkLOgd/u1eEB+DEi32gecuetIZy67IlRl0p8t8Sm+ixtaroz/ZHBbiyN3hwiFE/ClN10= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162907; c=relaxed/simple; bh=Vxc/pXcxmB+iFgkrcwmPIqJ1huzGQ8x23bFD3zc2UMc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P6h+RwvIgO9xqIKCYYj4Vr1Xe2Vcgn5wE+kzI0J+rGHK05kLhDNDZALU4V5wBZpK/pMvUNE+3dJX8AG9jE8nm9QceoSMcvWOHvCc5W1ueNy+GIsp3HUvzf7lwgYwIlwHQIfdpp3E/2DXlwD/zSsGIYHwtZVXvljj7kP5z1eX0WQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=GZoX5PrO; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="GZoX5PrO" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-8557c3f270eso2303245b3a.3 for ; Mon, 31 Aug 2026 00:55:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162905; x=1788767705; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=l5VVmQasiv9FjgFp8hyWKR9WL3WpfimdUt+OCxubG2c=; b=GZoX5PrOPy//UEPS6/+2ocw20SMN1kOhzOIwfTcGXfNj96LwjutBuNoKEymEKKYrlG /H+O8Y7KDcuJyCsdMBUjLbK8uFVeu60ZmLcFFZeTBDlOdSKHU5vhbbFI1ZwD/sWCYQ+Q UqkEbh7BGBnRCtbpbvpI+DYSzjZG/oPlhOGQ+RqRc16O1bMvYcsLk8JzE4OT1pW+/N96 cqvvjuTI0Oe1cZPlkOaC70+2e8WOLd/Y4/VPc9jSibpqpOHPh9fuy1ptcF+5M5f6ZELx EMESunyA7vVAjHyaTHOXSi//cSCTLbtSUqtIKokGGtY0ysJkSB2EiQk59423ZQkQvefc Fg7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162905; x=1788767705; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=l5VVmQasiv9FjgFp8hyWKR9WL3WpfimdUt+OCxubG2c=; b=hnTivaCEGOsJ8hBboo2iBCwHmxa3PwxqT0/A7R555zMXBL+fOcA0EzaRsufuhWpDM4 VFhUxils6ofS3pxi6RcKHLUX5/c/c1RyAna2vRLioLqUVBxaTGsqFEwzF7DUyIhy0Yk9 MHb3FE0m2DelBtu8HxOpnxCvzV8AjrK+PwyEgdBr0BYi0Qt0TX5ogicCuzE+6NAeF0P1 Ww7H/Aix2DG6Q7Z2IY3j2Ry3wDeOEfa3cLHhzrB6sy1I7YshYJ+CROMwo2JOZ/Lz8OYw 4hpL4eQeqdDN7xtXOGkdEcg4eeYbYldqPhrb190e2T5TleDgzP4DBpKisqfWMYGuJHa5 NbCA== X-Forwarded-Encrypted: i=1; AHgh+RpI+6T2aCsbKRYSPUw5ulHnoHxNwgSuBbNYo09I6T7GIHh3Hv5EyxSlI2M89UAmiwslqztglb5zKVaznZc=@vger.kernel.org X-Gm-Message-State: AFuF++nbxI68fmxt0MCIu7Xg4S8MuMGqXrK98tb2tUxE9qESYB7DsyJ4 OMAavTo1i+y/dAuQ4sWAvZYgrEY7gSkSskdRnh+EhfU4IH4wYa2yTGFcM0WuKNQ3JD2mQ8Qv+Mg 4zaC2 X-Gm-Gg: AR+sD12Lv5V8Ssy9taCq0AMe3h/B71+JIod1SRLTGbtqY/eHjCapX3G21Kmj9vohfQ0 rHec4UcQEwX4BGwSiQOe8/OcxG4C2FfOTmYKabBemriiG5D4yDvsyrjZJG4EZyIpl42Fu5YlinC cBEJR9y5+dp8tY4abZrTIPkVKP/8+uLQxnLTvYpSQrsH3yP8zMg63WkDmpNu/Yg6CKFECy8xHIu bnlk9fnMX3CawHInehEPN+n9tucE+NIKNJlOWuzRUTJb3n+CZFk0i8qXehmRYT8YntodBNbt68V nI1SL17k7MK1qvlvvuxJYoq1Jl3Rc+HDao4X/WCpFAenDmuWs0uTXDq8xR+XMCGPvT/v2U71mcg 855g9WImXGHjMqi4skO66Ee/h1qH/6KgEUBAaVDfHWxiqpQmpRbGlIL3YnwxwBxkle7CNIROygy VLf8OM829PvtJRPV5C0fOWjD0L6szKE8QyYcYMxBrr3zrUJpjkwucxFS6wyEpYwABB6W1//585z v+R5teI8bvd2SL/tIUkxZXg X-Received: by 2002:a05:6a00:c4d1:b0:856:30dd:91ba with SMTP id d2e1a72fcca58-85630dd920dmr21751321b3a.2.1788162905187; Mon, 31 Aug 2026 00:55:05 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.55.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:55:04 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 08/11] powerpc/mm: switch device DAX to shared tail vmemmap pages Date: Mon, 31 Aug 2026 15:53:39 +0800 Message-ID: <20260831075342.57563-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The powerpc radix compound vmemmap population path still finds a reusable tail page by walking the vmemmap page tables. Switch it to the common vmemmap_shared_tail_page() helper instead, so it can use the shared vmemmap page directly to simplify the code. This removes the powerpc-specific tail-page lookup and its fallback path and aligns the device DAX vmemmap optimization path with HugeTLB. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 80 +++--------------------- 1 file changed, 9 insertions(+), 71 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index cf692b2b5f7b..831c231a4a18 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 #include #include @@ -1250,59 +1251,6 @@ static pte_t * __meminit radix__vmemmap_populate_add= ress(unsigned long addr, int return pte; } =20 -static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, - unsigned long pfn_offset, int node) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - unsigned long map_addr; - - /* the second vmemmap page which we use for duplication */ - map_addr =3D addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; - pgd =3D pgd_offset_k(map_addr); - p4d =3D p4d_offset(pgd, map_addr); - pud =3D vmemmap_pud_alloc(p4d, node, map_addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, map_addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, map_addr); - if (!pte) - return NULL; - /* - * Check if there exist a mapping to the left - */ - if (pte_none(*pte)) { - /* - * Populate the head page vmemmap page. - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL= , NULL); - if (!pte) - return NULL; - /* - * Populate the tail pages vmemmap page - */ - pte =3D radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); - if (!pte) - return NULL; - vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); - return pte; - } - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1320,6 +1268,12 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, pud_t *pud; pmd_t *pmd; pte_t *pte; + struct page *tail_page; + unsigned int order =3D pfn_to_section_order(start_pfn); + + tail_page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!tail_page) + return -ENOMEM; =20 for (addr =3D start; addr < end; addr =3D next) { =20 @@ -1349,10 +1303,9 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + PAGE_SIZE; continue; } else { - unsigned long nr_pages =3D pgmap_vmemmap_nr(pgmap); + unsigned long nr_pages =3D 1UL << order; unsigned long addr_pfn =3D page_to_pfn((struct page *)addr); unsigned long pfn_offset =3D addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); - pte_t *tail_page_pte; =20 /* * if the address is aligned to huge page size it is the @@ -1377,23 +1330,8 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + 2 * PAGE_SIZE; continue; } - /* - * get the 2nd mapping details - * Also create it if that doesn't exist - */ - tail_page_pte =3D vmemmap_compound_tail_page(addr, pfn_offset, node); - if (!tail_page_pte) { - - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - if (!pte) - return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - next =3D addr + PAGE_SIZE; - continue; - } =20 - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*ta= il_page_pte)); + pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, tail_page); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6D933B2FC8 for ; Mon, 31 Aug 2026 07:55:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162912; cv=none; b=KD4M5pu/NuBoqkuGre4u10LWkmNX1eDc8edBDBhptWf/ezIumZmLS2I5SYuOwz44r9hBl6/KH1/wgntItl3/jm6pFESiuk4tjIng9MBBxqLShdZCfKL0m9ud6VT14Kembn9FvTLRBqFHV2Q/YcbmN8twjQOBWGR0NJFbsLAEv08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162912; c=relaxed/simple; bh=+I09y+q9sr67CNOypLQdIUiE7TdXVc7M6dwiHdriwd8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=J8AaqCDoBBcnyray5oUpYzwx5ke+tQLzdQnkNyEVaKZ6nOMlw59kS+gOPUvwVskQmhSHMZf7I5XmWu7+jcb3N1z5+DUqLXSKuicC8p83xZTDpTRVJ7Z7Qf/HFcu96lC/Hf19zv1nGvzl6TJIJrT5axl1s/Z8L5GX+hrRwIhpHTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hy9ZhfI+; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hy9ZhfI+" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-84864086bfeso3303229b3a.1 for ; Mon, 31 Aug 2026 00:55:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162910; x=1788767710; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0tkr6D1mJkSgu6ygeW9vA9keYBBS8zHk8hkf8RCBkSI=; b=hy9ZhfI+FjnCX4moJq98A5i2Qx1SZXoib8KAt6SxtUOO1eEEwoaSLlv1CeVeOmT5q6 LWSfc4hvpGgKUaMbKM0BAewLRhMt2+3RRnG7aFjA/94pEzBBBXLlHqsMoiG1mmfBdNvv 4VAhIRXuezPbAo6wxjDJJiIpqD3fQzCN5aDk3iK5n4/VcAIMqjXBR/FbJn5haQJppYrO LBzm5Se7G3mTrTnUilSetn7xrV37H+Eo+02LP2Ka7sIU4z4lw42KJnr/TVkHX7mMUB7z BrXx8+kaGKFVxnCrZALAdEONkCr5/m2l23YHAwa6EgVKP3ecbHB62Ow8u5VroWPBS18t Z1hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162910; x=1788767710; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=0tkr6D1mJkSgu6ygeW9vA9keYBBS8zHk8hkf8RCBkSI=; b=BFqeYS97LYRa0aORaqgIU/iaw9xw3Hc6Fp//FsxvDkZSMVwatdhkr06BJu25jIHqpB jUMGuip2BMx5FkLgYDgoBz0JTPOxJcrh+JX0VR7d4elICdcoqr2S+TP7NSOW7W61Lz0R bZ2reIL90uEd26eq/pV0G4K6AIHqDn5mTYimE1y1ENuxj6NtI9gg/YGK6F56y+RpWL6X Lm435JHvSFbc/opW9jeq/Jmuz5iPlvVrj4gvo/ZGFmfqdXK/rBjzde+ecbUjx2nnyv/v pR4IJTIjzXh+pm/cS4iETgwh18P23KMbAC7Pq4yd+JBDoJ2NLC/gxsY2Fmq4Gjnfe6zd h1cw== X-Forwarded-Encrypted: i=1; AHgh+RoGUQJIwbLdz6l/vRg89DeTHgKdMD9I1icGQXKdJcIsxoBLxRD+vPFlVoc9E+vNt3nI6PN2ZXr9Hb9ptFg=@vger.kernel.org X-Gm-Message-State: AFuF++kaCf1D9sehCcAsU2TlyxeHUitF0uBVX//0ksq239TQQsfLktmr F3zvlC7nUaDqNT8RwYZqRp0V3pRKHLKHvOfeoU9S0NaXQh6gYuft6GuT34W7U9PGWsk= X-Gm-Gg: AR+sD13t1HnKyZfYHbzd2bCHAWFyO+1Vt7KHJ4/hKYb9NpZylE7a56Fd4AeLYDh/RZY ASIzfPhpPZUOYOPGOcvzfS3Jva0w3kwPGVUnjy86HSEIYLROJuTc0igL1QzG+zlus1ZNe8N+pF0 ScwkMjeiOzkFgaVtXAGjDEOXp7RVVxLIo5FngiPmyYpGod0jyNvv5EXkLCDAq9qpiKim7VjTMnk rLrk0nNgpVvpxYPY0H0i1BdGkuhhMXFdI1rZiCu2XDWXpR8IBRYxWWw25sMePlI6YdtZe/v4dVg eN/PsInVXFZODfCpGj8yfnY+9EMk2OpcjgsAu0AbqfYHK09sm5GyfiGKRZXxjmoz6YY6PugeVif Q+qRJUrqw6f9K0DGYR7UIN0z1cA2jEQdGaALrSynUNXR8uirgVj2E1esC9lvC6ax56fgXfuV06L I+y9zC8Lp5YEy6NQPuAZvvZgG0kYL+cOVlap6UFZV/jt3Cg9gTLsrnkt+VHsFmj6e9NEseNTj8u W7cVdEAUDDiDO0d3tY+eebx X-Received: by 2002:a05:6a00:1c9a:b0:84e:9257:d0c2 with SMTP id d2e1a72fcca58-8562a00149emr40654262b3a.9.1788162909920; Mon, 31 Aug 2026 00:55:09 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.55.05 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:55:09 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 09/11] mm/sparse-vmemmap: drop the extra tail page from device DAX reservation Date: Mon, 31 Aug 2026 15:53:40 +0800 Message-ID: <20260831075342.57563-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The device DAX vmemmap population still reserves one extra tail vmemmap page after the head page. Drop that extra reservation and let the shared tail page cover all tail vmemmap pages after the head page, so DAX follows the same reservation model as HugeTLB. This reduces the reserved vmemmap pages for optimized DAX mappings to one and removes the now-unneeded first-tail population from the generic and powerpc paths to simplify the code as well. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 46 ++---------------------- include/linux/mm.h | 3 +- mm/mm_init.c | 2 +- mm/sparse-vmemmap.c | 13 ++----- 4 files changed, 7 insertions(+), 57 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index 831c231a4a18..e7e751c48dd2 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1218,39 +1218,6 @@ int __meminit radix__vmemmap_populate(unsigned long = start, unsigned long end, in return 0; } =20 -static pte_t * __meminit radix__vmemmap_populate_address(unsigned long add= r, int node, - struct vmem_altmap *altmap, - struct page *reuse) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - - pgd =3D pgd_offset_k(addr); - p4d =3D p4d_offset(pgd, addr); - pud =3D vmemmap_pud_alloc(p4d, node, addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, addr); - if (!pte) - return NULL; - radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1297,7 +1264,7 @@ int __meminit vmemmap_populate_compound_pages(unsigne= d long start_pfn, if (!pte_none(*pte)) { /* * This could be because we already have a compound - * page whose VMEMMAP_RESERVE_NR pages were mapped and + * page whose retained vmemmap page was mapped and * this request fall in those pages. */ next =3D addr + PAGE_SIZE; @@ -1318,16 +1285,7 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); =20 - /* - * Populate the tail pages vmemmap page - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, = NULL); - if (!pte) - return -ENOMEM; - - next =3D addr + 2 * PAGE_SIZE; + next =3D addr + PAGE_SIZE; continue; } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index edadd7549b72..edc7b9ce9e79 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5180,7 +5180,6 @@ static inline void vmem_altmap_free(struct vmem_altma= p *altmap, } #endif =20 -#define VMEMMAP_RESERVE_NR 2 #ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) @@ -5200,7 +5199,7 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, * For vmemmap optimization with DAX we need minimum 2 vmemmap * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); + return !altmap && (nr_vmemmap_pages > VMEMMAP_OPTIMIZATION_PAGES); } /* * If we don't have an architecture override, use the generic rule diff --git a/mm/mm_init.c b/mm/mm_init.c index c4cd61978ce8..d520fd8de0df 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1049,7 +1049,7 @@ static inline unsigned long compound_nr_pages(unsigne= d long pfn, if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 - return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); + return VMEMMAP_OPTIMIZATION_PAGES * (PAGE_SIZE / sizeof(struct page)); } =20 static void __ref memmap_init_compound(struct page *head, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 0201877a7f80..e655d9d1348f 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -136,7 +136,6 @@ int __meminit section_nr_vmemmap_pages(unsigned long pf= n, unsigned long nr_pages { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_order(ms); - const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); @@ -147,13 +146,13 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages =20 if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; + return VMEMMAP_OPTIMIZATION_PAGES * nr_pages / pages_per_compound; } =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); =20 if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; + return VMEMMAP_OPTIMIZATION_PAGES; =20 return 0; } @@ -521,17 +520,11 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, if (!pte) return -ENOMEM; =20 - /* Populate the tail pages vmemmap page */ - next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); - if (!pte) - return -ENOMEM; - /* * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - next +=3D PAGE_SIZE; + next =3D addr + PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, page_to_pfn(page), flags); if (rc) --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 293C53C9EEE for ; Mon, 31 Aug 2026 07:55:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162916; cv=none; b=A+5mpetIqMclkU/VHnpUGKCD6UGChkimdLJ4/WKDHV/AcGK5afVRi/mIV+bnXxwoWgRsJcQtlLRSdFeGdmdAZqgMQK8QXamasIJ32JtA2sJRqV6c5EXaMZ2OhvNnTHgE5Y5caHWgRf2qUoOy7LN2ETQvm4uKhcHt/mTrbMlF3+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162916; c=relaxed/simple; bh=0craoejHz+yL27FxB85ch1A7TZcJ/6nPjXoONqTOYGE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Nj/D3e6GHEAf6crLKbNk0U+pUAtvYE/kuCw58kIX2ZOeCh1orTqdTPU6mDq9YDP4dZZkXMRMPGi1fDhdDGwigCGwPpts/Qw+jwACYjZryJ7qb5XimB5WdL/XskJwjCEetrprj+WBFlp7ba8Di4MI03aS++MsMEv/ACWq/y0eAdc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DFM58fR5; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DFM58fR5" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-84fa3b14ee1so2825370b3a.0 for ; Mon, 31 Aug 2026 00:55:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162914; x=1788767714; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8Uns/ZJlTRrXsZn1DdaJluablabo4P2O8/HR8HKp5bE=; b=DFM58fR5HSPLFJhOX23ETEp/9cXZuuukBeywrABjaW3Wh0ls4YBzOhMOeBR9Qq5/85 bIg8H9pV6pT2SXGMJsdZlxtwWRqa2VuMoex+gQdTqnRmWWsHQv/IpfiAXb5xLLTRGdjR eBOusAYJ+0anPgnymRH4i/4/TBgisTGdNABwe25v/37FF4UT9kRckXXMQMG09XvdxSqm 1DS2BvpGTNw6kmH8NKIE1BD3NhAtwxO1rtVu+JRjQ+LRf80RY4y8a2jodicUkuHA7yUJ as9wWtkF/ZPqDII32qwPqPbHGTE3g2SRbTWf9KyTxGqs5XPQ3UizdnrQdN3nO4FXB4wU U7cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162914; x=1788767714; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=8Uns/ZJlTRrXsZn1DdaJluablabo4P2O8/HR8HKp5bE=; b=f/oUXavaPCEbX87iFUQ0fEzZu7ZSi8eAdxxkNYRExDuqu2D8qpw9I4Q92uihDZ33Oo pPg5vShvrNgYylftBDtqWwLiFL5S/DFwVj9I7vhNZ3TjRQYJE0aXNdZjnmMoIs+H8730 yyJkysqd+ZRcMbaIIsdBF5Rqrhiozb33bgZLucFLhytnocvwqhWuCbAmrzRQ9V50IS1L K0kqkV2rNxR+Hzs8dCOddqo/78k4akAnO/lQiLxkgs9wicFfiH//yQ7A6fikCT62iTA2 7FjwRzPfQOtX447Y3rJioFXxe5BWF7e416KuxqP0MZ7l+HKO4PToXgpqFmlJ4nzIBzcz ULKg== X-Forwarded-Encrypted: i=1; AHgh+RozcgO6Ga/pnbFr+I3hPeGLGX6Btk0Ep3CjRhms8zkPuZ8i4lpOUcpH0Pvwr55sr1XJ5xHsAVv1mwK+fR8=@vger.kernel.org X-Gm-Message-State: AFuF++keDX4pmPt3vzF9342bT2JYLRMzkpwIVDCa7pgyEWHoV+T7un9v y4MgSQpZ13XsC3TD+JQTY6ShxwJEzBDPvWMgejd0VWou7+z54kwIWSV6byWpLh+uqhw= X-Gm-Gg: AR+sD10X4sTM6toCepQyFA2OsCV0OrYLOMDQoSUo6wm1o2LKSAtsiagDMSWiJkxZWk6 gY/b0PPVfs5GQo7rA07K+r5JyFPukrvbnSYaxL/09mVmpG3JEKhmFXNgaxxWaTUBHe0Dfd7OQcE qcfjlBlEVPKEJsb0PzorKK/a6KrS2uOwdPNGgEkppc/DJI69hqUH5gHeLDfNcEuP+WvGewtUoDi MF+o5zEpFMEbhgnsJ6AwOizXcrNBPZqG2aCL/Kzz+vhZ15c15y6REnLgNstxoLJB/XlCih5RrQ7 cLT/tO0CBzI1W+UEf87g/Ljl2p42ShH1ugVg1x/voHIxebycNNWYvu+082HKSnXyyZ/l3Ghz1lY MZxy8Iy4MZgWMU6tVhxdnfoLHiWohKieFXmlkzIyvOLmNsd08tnDffbta5HR0gv7iPIKj3akGQb zu6dv3Prz4WdeOU7CpgLXMczawz3Sts99QLkIMnsiT/czboRfyS8W7xcoCsXLb91YAEmpU3H1i+ H0DPdl2SFy4p6UqdC2lUwkN X-Received: by 2002:a05:6a00:845:b0:857:72ba:ff11 with SMTP id d2e1a72fcca58-85adca7066cmr2156632b3a.25.1788162914397; Mon, 31 Aug 2026 00:55:14 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.55.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:55:14 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 10/11] mm/sparse-vmemmap: drop unused section_nr_vmemmap_pages() arguments Date: Mon, 31 Aug 2026 15:53:41 +0800 Message-ID: <20260831075342.57563-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_nr_vmemmap_pages() no longer uses the altmap or pgmap arguments, so drop them from the helper and its callers. Signed-off-by: Muchun Song --- mm/sparse-vmemmap.c | 10 ++++------ mm/sparse.c | 3 +-- mm/sparse.h | 6 ++---- 3 files changed, 7 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index e655d9d1348f..4950ac2a1c01 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -131,8 +131,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 -int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages) { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_order(ms); @@ -637,7 +636,7 @@ static struct page * __meminit populate_section_memmap(= unsigned long pfn, struct page *page =3D __populate_section_memmap(pfn, nr_pages, nid, altma= p, pgmap); =20 - memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages)); =20 return page; } @@ -648,7 +647,7 @@ static void depopulate_section_memmap(unsigned long pfn= , unsigned long nr_pages, unsigned long start =3D (unsigned long) pfn_to_page(pfn); unsigned long end =3D start + nr_pages * sizeof(struct page); =20 - memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages)); vmemmap_free(start, end, altmap); } =20 @@ -658,8 +657,7 @@ static void free_map_bootmem(struct page *memmap) unsigned long end =3D (unsigned long)(memmap + PAGES_PER_SECTION); unsigned long pfn =3D page_to_pfn(memmap); =20 - memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); vmemmap_free(start, end, NULL); } =20 diff --git a/mm/sparse.c b/mm/sparse.c index 9349ed6326c0..adf057f54c0f 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -251,8 +251,7 @@ static void __init sparse_init_nid(int nid, unsigned lo= ng pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, SECTION_IS_EARLY); usage =3D (void *)usage + mem_section_usage_size(); diff --git a/mm/sparse.h b/mm/sparse.h index e511d99fc26b..5d0f407bde7e 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -70,12 +70,10 @@ static inline void sparse_sections_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); -int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap); +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages); #else static inline void sparse_init_subsection_map(void) {} -static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages) { return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); } --=20 2.54.0 From nobody Sat Sep 26 19:35:31 2026 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6A783C1D48 for ; Mon, 31 Aug 2026 07:55:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162921; cv=none; b=d1/5/0NsxtVqJkLwp5GjejsM0Od2tSTmOtebEiCdkU0PxS4azAM/ibdatNdPg/3RKjNgZhDdkqiE7nGOcUG31ns5vRIK5wTSoP5JbGLtsNYdEJo9OUtO1+A6C9K5bZgqqCld/L4ZrRc+imrp4WF3zQG0gmC+Rz64EWyKcboYbd8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788162921; c=relaxed/simple; bh=GlAJ1WSK7MS4JWW3qgmtSMSpzYqrlPQ5xWGNOhjuiDU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nkQhmPMhJ2Pp0leMeNmQCcHGztykVGQzMD4pzzHmTGeT361/avLYQWwlRMYZv/9+KTzoFhjt+aUZaHYbaH6x+mNVScDEuAWdHDWt1aetiCLL8fcqi9IAoO4jq4BnAi6x/hhF/eJ3AaCSEMQdRnx2DMCEFOHQXvI973OvLMWgJvs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Bqa4S0km; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Bqa4S0km" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-85377c8bc96so2855623b3a.3 for ; Mon, 31 Aug 2026 00:55:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1788162919; x=1788767719; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=UUo/y6IjpqbcIL2TFvU1ITTKCpXfhtvRzwNLV1+yihc=; b=Bqa4S0kmsSeiBX9zprRiGHLdDs6SGrmULANisb+wFZsc78t5kK9jSoEpHaxFNPjuc/ CszefzXJi2pjx8ogQvZb4nxcZVNaLy0vUtwiK1k8sX5Qitef8ba9LOalb4bVvpMBwL+v n7coQl+SLVaEI3Et8fQrhRhqc0aoBDET1BZTJ5Ny/k8Mkv52mUnwel8SYQc2076JngHD jD3qxIcHVdN4AD5aXewWaCLl7QvSCu1Bz3E286VMtZ8GUXuYDDgdqmXg8QabldcCDKgo tzk0vkMIiX8+ySpyCjTl8Xpd+BvXm24h74m4D3ay9ijCafkfsp/Ipd+wF8eiha7pXyoP y0Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788162919; x=1788767719; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=UUo/y6IjpqbcIL2TFvU1ITTKCpXfhtvRzwNLV1+yihc=; b=ec9qGEg9BkFFVWkpR9tI+MyralD0Aa0aR+iyKMDzLBTSU3nznd7emAkyfklHr3dJ2c Q4KheTbCWrSgDpECX1i2oXfzWmUe4RTwlcoUl5CkOJN1Qwgx4mWW+JZoTN8zZi4jytS5 W5D08xuGhUJLqA7dclHqbq8SdjvHuBuZtjK1F3TxwtiR+ZTZweaTtDQUPOl/Rn9/XPrA 7M4VfWAzQEa0Pf4pQHXP0YIKu5TnV/okmw04XMHI6+W/CdbNyKURMaP70LxYZw6tPXz9 rInUjwT7mIQevR7RuQmBz8OryUFNgSjcdkUgQmI593GUg/M6bl9sVtHd+wyqfYx3T76m W6pg== X-Forwarded-Encrypted: i=1; AHgh+RoxNfCznKSA6IEtRKtRxWNRJuvakUZuDrImdUSoqQ94PJzeX6gr5pq7cgQVtagvoZWWLw69hcSxkz/zb+s=@vger.kernel.org X-Gm-Message-State: AFuF++kpB8uLehvOP32fajSbOpq/Mlr60GqUV4ITbM+avjAkcx46BNPk xODceGx8orAJtIgJ4h2IO9N6oyIERQlzwqwo6Vgsb0aDRjpo8iaUOFrmPtrnzsujllg= X-Gm-Gg: AR+sD13URnuNesWRyQKdnxz7XTwvAg2hh1tfVSJtsHdt21QwfRGnAHB168XjwtAeiM1 O0tNHYsR4JmbUT+pwuhlOgx1cMcT5dAuec/u8kbZ/hA8092N9b6Xjhzw8egQ5X5cHPUSkASvXWR Wwud9ERNgrvoY8gl2j8Ppq5GWa6fn6vaBgfhxlNnNmgFd8HGsJ7+HnuAxIQV9rZM2LEYhkikmal dY9WBST0i+jmexZC1EiLBQQRFn0iMPCzGD0k9wRG7ApJSTw6wsXsP/ZH3jphs9vsZ/Ewtkaix5N E+0BoNd8sJAzkhcQCg+J1VER/Ih7+CMRCCw4ugOOKiJgVo2a7wLD+/cKqK4HY8DSkxpqD+83vit hVUYfQ25dktjSHYZ/l4AExdRGl1vS8zVlqBeM+PlmjZsGFCeopCBy4jv8CazpUiwmjHJNNfpZzs ZikZjidGyQJxHwugAef8flCQl3dOCFMEapLJijxZK7Y9wbYNm/rG2OoaOR6y0Gxc2Tt7OFm15Ce uvPZfHf+u7/AZDAKWP7sIvj X-Received: by 2002:a05:6a00:f0a:b0:848:4859:a45 with SMTP id d2e1a72fcca58-85628988a7cmr40655921b3a.2.1788162918893; Mon, 31 Aug 2026 00:55:18 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.9]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-cc1f330f88csm3590256a12.10.2026.08.31.00.55.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 31 Aug 2026 00:55:18 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH 11/11] Documentation/mm: update DAX vmemmap deduplication docs Date: Mon, 31 Aug 2026 15:53:42 +0800 Message-ID: <20260831075342.57563-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260831075342.57563-1-songmuchun@bytedance.com> References: <20260831075342.57563-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX now uses the common per-zone shared tail page for vmemmap deduplication. The old documentation still described a DAX-specific layout with a separately populated tail vmemmap page and half the HugeTLB savings. Update the generic and powerpc documentation to describe the shared layout and keep the powerpc-specific size calculations while avoiding duplicate diagrams. Signed-off-by: Muchun Song --- Documentation/arch/powerpc/vmemmap_dedup.rst | 90 ++++---------------- Documentation/mm/vmemmap_dedup.rst | 32 +------ 2 files changed, 21 insertions(+), 101 deletions(-) diff --git a/Documentation/arch/powerpc/vmemmap_dedup.rst b/Documentation/a= rch/powerpc/vmemmap_dedup.rst index dc4db59fdf87..8286acbca9bc 100644 --- a/Documentation/arch/powerpc/vmemmap_dedup.rst +++ b/Documentation/arch/powerpc/vmemmap_dedup.rst @@ -19,82 +19,28 @@ With 1G PUD level mapping, we require 16384 struct page= s and a single 64K vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Henc= e we require 16 64K pages in vmemmap to map the struct page for 1G PUD level ma= pping. =20 -Here's how things look like on device-dax after the sections are populated= :: - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 15 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - - With 4K page size, 2M PMD level mapping requires 512 struct pages and a si= ngle 4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapp= ing. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - -With 1G PUD level mapping, we require 262144 struct pages and a single 4K -vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we -require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level -mapping. - -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 4095 | -------------------------= -+ - | | +-----------+ +Here's how things look on device-dax after vmemmap-optimized sections are +populated. ``N`` is the number of vmemmap pages required by the DAX mapping +above:: + + Device DAX vmemmap pages (N pages) backing page= frames + +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= ----+ + | | | 0 | -------------> | 0 = | + | | +-----------+ +---------= ----+ + | | | 1 | ------+ + | | +-----------+ | + | | | 2 | ------+ + | | +-----------+ | + | | | . | ------+ +---------= ----+ + | PMD/PUD | +-----------+ | | A single= , | + | level | | . | ------+------> | per-zone= | + | mapping | +-----------+ | | shared t= ail | + | | | N - 1 | ------+ | page = | + | | +-----------+ +---------= ----+ | | | | | | diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_= dedup.rst index 9fa8642ded48..8c287ae3f86c 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -1,4 +1,3 @@ - .. SPDX-License-Identifier: GPL-2.0 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -192,32 +191,7 @@ to 4 on HugeTLB pages. =20 There's no remapping of vmemmap given that device-dax memory is not part of System RAM ranges initialized at boot. Thus the tail page deduplication -happens at a later stage when we populate the sections. HugeTLB reuses the -the head vmemmap page representing, whereas device-dax reuses the tail -vmemmap page. This results in only half of the savings compared to HugeTLB. - -Deduplicated tail pages are not mapped read-only. +happens at a later stage when we populate the sections. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ +Deduplicated tail pages are not mapped read-only. The mapping layout is th= e same +as HugeTLB. --=20 2.54.0