From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f20.google.com (mail-pj2-f20.google.com [74.125.227.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44F2B3EC687 for ; Wed, 16 Sep 2026 06:43:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.148 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541037; cv=none; b=q+aftKdGVeJHT9jHidlx8hcaacGNFNO2PelZdSYj3PQEg2zPZ3CgnaG8W+zjPiQevIxJxRnaHUllVhtWVXoveLCxFzQl+R8TyKDXiUg6Sl8nV8srSHHh4vNTF+4zIr8vH+lUJP/l63XzMbDwyawyDFU4E+Sg8y2HfRWl3F/xAnE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541037; c=relaxed/simple; bh=Ko3YaiKEa0v5sLcORxbisHb1mdErqj3NkKNU1hORGAQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cDtAJ/UQr5iZ13QZ8oF4rSszFujDAHfZHMsjRQLZwgR/u06aVw1shGMB2BJvvieLgfCb+kGXs4RP1ZVleJZxv4yCgdzU8RQ3NlZvpxIThwG3V5VMemyGqQWQZqZogHJVEh3QmbrvC5h16Oax+WPmoW/HupfeMy55Ajc6pqHXgEE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RQNQC8nm; arc=none smtp.client-ip=74.125.227.148 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RQNQC8nm" Received: by mail-pj2-f20.google.com with SMTP id d9443c01a7336-2d747ed9866so5612835ad.2 for ; Tue, 15 Sep 2026 23:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541034; x=1790145834; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=1OKh9ewKICcZ4B1bMy6j7dzmXfE4aXD+pSJoYMr6des=; b=RQNQC8nmFNmN5M/X893TBi5w7ZfhmalW8txi4/R6MAIlre9O8TgIwmCwRT7xkAASQT vEEJNJNW67GRBHszJtutw4mZ7mAjVszahKrwetfFIU8A3Bp1j45/QsiHvnyX7xBjMek6 HPgT12cTsxRZwiJGmV8UO2Rtj28eaIDXPg4/pzrgsFMyIcwsEuu62X/L4ZBvkyAGc4nS LD06D8GOSXTVgIx5EbzvxjcorEPTZlSXp0g2Vml/D1facW8nXy7Wh+l8M3HIhPWAiaa8 o9iXi5ATY0WyTGtLH/i5rFgJQqJTXCLkrLXazrWHw4moO3vX1k5Wvq89wTUUPZJybssG UiZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541034; x=1790145834; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=1OKh9ewKICcZ4B1bMy6j7dzmXfE4aXD+pSJoYMr6des=; b=btUyMngtv7JiQqpV2yApqjA1/Q4ydF3YT6T5C+FA6YN+1WRF0iGE0rAjYQUJBTmH0O tLuqfc4RFNt83+KDQyXhCNOeOAwpX4+A2vKigtDMzRDRUqqd8mwgEruZu75EBw5JHFEJ 7cZTm/MZ1yJh5KZnFTzb6G2ZpIG9jB9xINiOLhL65GL7A4NkIIcchnXIuXayAGVksh0F vDN6rdw5RKqKxsWv6eg/jqCtTj1IJa8VCzjQOhNkZbbK75hlYZDQm9lo6uoFhOEbXdzK P9rA/sJoF226MSrj+8YpI11/WqlltzPUlHcFLUKF11TjJqbASkZVq4K2M+9btejcR8Vt YOow== X-Forwarded-Encrypted: i=1; AKwUvBxP6/l6yPULLur4S4aNv+GIOpLDazsX//+Kq/5JDTo+OJFrG+xba6jX9voqems0uIqc6gd+G23h22Q1dSo=@vger.kernel.org X-Gm-Message-State: AFuF++mDm7JpjqdIZnvnS19qyIX6dUqoxLeyko8FVNyuvo8223HYB52p 4CrvKKKWlp3phuIa5S6f6Z+GwY8kPMmGq095tf2C103FcInmD3Udx3vEOhJo23yP/ps= X-Gm-Gg: AYBFou2aX20/DI7McVMpwVDNBUZMFGbMPLT3sNZHCKQeTvK8YjIn9fv+8+lLMBGftmg DoTTuVGGWIT7M6qUJguQhQ2o9FfONSK88xXCV/9ka//+UwFACR5QJILdOhlBkJIP6LgXwGpw+4m ZYmSSdN8ZC7lcrncmDmll3V0jFniSIDQHCvAu1NP4sF+hwlJvJo3l8JE1Wwq17vKKEUxRsA6jC7 q6Eb+t8yybwg+rE3uOjlazWBtPIWszcQXlM8f0/JaLVvL1SmV/4wM0Qym1p8kdhqKmnIPyvlUh/ 7/HsZVgmzbWJa131cCmFfTotkH5Ds7ZTUrMstsvqRNClTq5LztAsxW4P3R6ByJK9GPHDmzTIUBm dGANppYkWOejTAGMBI+sDWjA7A+xlbTreLEOjzeJVPH4gTdbQVHmUM9GuHrNLI1sgjcldF1uuYv iEbuyjpYeJK5QBlBo+MtL5vwpVYoccx81Mxz9G5N5rAK3Atxg1lNVOr4ncBYeyyFXQeWH9Xagaq InEjqSLbG5sK6U+ZJrxtgrvmc+nVw== X-Received: by 2002:a17:903:1983:b0:2d9:4871:393d with SMTP id d9443c01a7336-2dd8e77e51fmr32018845ad.21.1789541034351; Tue, 15 Sep 2026 23:43:54 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.43.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:43:53 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 01/11] mm/sparse-vmemmap: introduce CONFIG_VMEMMAP_OPTIMIZATION Date: Wed, 16 Sep 2026 14:43:31 +0800 Message-ID: <20260916064341.1825793-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The section-based vmemmap optimization infrastructure is still guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, but it also can be used by device DAX. Introduce CONFIG_VMEMMAP_OPTIMIZATION as a common config for the shared infrastructure. Select the new option from HUGETLB_PAGE_OPTIMIZE_VMEMMAP and from DEV_DAX when the architecture opts in to DAX vmemmap optimization, and use it to guard the generic sparse-vmemmap state and helpers. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) --- v4: - Rename SPARSEMEM_VMEMMAP_OPTIMIZATION to VMEMMAP_OPTIMIZATION (suggested by Mike Rapoport) - Collect Acked-by from Mike Rapoport v2: - Fix SPARSEMEM_VMEMMAP_OPTIMIZATION being selected without SPARSEMEM_VMEMM= AP reported by Sashiko. - Add an explicit DEV_DAX dependency on ZONE_DEVICE - Collect Acked-by from Qi Zheng --- arch/x86/entry/vdso/vdso32/fake_32bit_build.h | 2 +- drivers/dax/Kconfig | 2 ++ fs/Kconfig | 1 + include/linux/mm.h | 3 +++ include/linux/mmzone.h | 13 +++++++------ include/linux/page-flags.h | 5 ++--- mm/Kconfig | 4 ++++ mm/sparse.h | 4 ++-- 8 files changed, 22 insertions(+), 12 deletions(-) diff --git a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h b/arch/x86/entry= /vdso/vdso32/fake_32bit_build.h index bc3e549795c3..72a92cb9b53d 100644 --- a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h +++ b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h @@ -11,7 +11,7 @@ #undef CONFIG_PGTABLE_LEVELS #undef CONFIG_ILLEGAL_POINTER_VALUE #undef CONFIG_SPARSEMEM_VMEMMAP -#undef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#undef CONFIG_VMEMMAP_OPTIMIZATION #undef CONFIG_NR_CPUS #undef CONFIG_PARAVIRT_XXL =20 diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index 602f9a0839a9..f50c6b32d826 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -8,6 +8,8 @@ if DAX config DEV_DAX tristate "Device DAX: direct access mapping device" depends on TRANSPARENT_HUGEPAGE + depends on ZONE_DEVICE + select VMEMMAP_OPTIMIZATION if ARCH_WANT_OPTIMIZE_DAX_VMEMMAP help Support raw access to differentiated (persistence, bandwidth, latency...) memory via an mmap(2) capable character diff --git a/fs/Kconfig b/fs/Kconfig index d1c210c6508f..1454b7fe9641 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,6 +278,7 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP + select VMEMMAP_OPTIMIZATION =20 config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mm.h b/include/linux/mm.h index c49ef99b4413..070ce27e9cd3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5175,6 +5175,9 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, unsigned long nr_pages; unsigned long nr_vmemmap_pages; =20 + if (!IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION)) + return false; + if (!pgmap || !is_power_of_2(sizeof(struct page))) return false; =20 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index acd94cecc0d3..56403841e887 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -102,9 +102,9 @@ * * HVO which is only active if the size of struct page is a power of 2. */ -#define MAX_FOLIO_VMEMMAP_ALIGN \ - (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && \ - is_power_of_2(sizeof(struct page)) ? \ +#define MAX_FOLIO_VMEMMAP_ALIGN \ + (IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION) && \ + is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) =20 /* The number of retained vmemmap pages with HVO enabled. */ @@ -116,7 +116,8 @@ #define __VMEMMAP_OPTIMIZATION_NR_ORDERS \ (MAX_FOLIO_ORDER - VMEMMAP_OPTIMIZATION_MIN_ORDER + 1) #define VMEMMAP_OPTIMIZATION_NR_ORDERS \ - (__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 ? __VMEMMAP_OPTIMIZATION_NR_ORDERS = : 0) + ((__VMEMMAP_OPTIMIZATION_NR_ORDERS > 0 && \ + IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION)) ? __VMEMMAP_OPTIMIZATION_NR_OR= DERS : 0) =20 enum migratetype { MIGRATE_UNMOVABLE, @@ -1155,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_VMEMMAP_OPTIMIZATION struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; @@ -2019,7 +2020,7 @@ struct mem_section { unsigned long section_mem_map; =20 struct mem_section_usage *usage; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_VMEMMAP_OPTIMIZATION /* * Normally, sections hold regular (order-0) pages. However, for * sections with HVO enabled, this tracks the compound page order diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 86dd0470da11..7080a6a1a79e 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -208,14 +208,13 @@ enum pageflags { static __always_inline bool compound_info_has_mask(void) { /* - * Limit mask usage to HugeTLB vmemmap optimization (HVO) where it - * makes a difference. + * Limit mask usage to HVO where it makes a difference. * * The approach with mask would work in the wider set of conditions, * but it requires validating that struct pages are naturally aligned * for all orders up to the MAX_FOLIO_ORDER, which can be tricky. */ - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION)) return false; =20 return is_power_of_2(sizeof(struct page)); diff --git a/mm/Kconfig b/mm/Kconfig index bc7befafb47b..0fa2eb76e4f2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -461,6 +461,10 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. =20 +config VMEMMAP_OPTIMIZATION + bool + depends on SPARSEMEM_VMEMMAP + # # Select this config option from the architecture Kconfig, if it is prefer= red # to enable the feature of HugeTLB/dev_dax vmemmap optimization. diff --git a/mm/sparse.h b/mm/sparse.h index d3a71ef4fad0..d8c08a388e4e 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -10,7 +10,7 @@ =20 #include =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_VMEMMAP_OPTIMIZATION static inline unsigned int section_compound_order(const struct mem_section= *section) { return section->compound_page_order; @@ -75,7 +75,7 @@ static inline bool vmemmap_optimizable_pfn(unsigned long = pfn) =20 static inline bool vmemmap_optimizable_order(unsigned int order) { - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION)) return false; =20 if (!is_power_of_2(sizeof(struct page))) --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f38.google.com (mail-pj2-f38.google.com [74.125.227.166]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 133603D75D3 for ; Wed, 16 Sep 2026 06:44:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.166 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541042; cv=none; b=VdKwem2J858Af6IOgW8n+6bFodD99yxM/RCYwzwh4+bNeAEii75hPSGpEFi5GStWylhVYMXquGrbNFJKL4D1pYWNWMEIM8tn3pW8TQ4N1vd1+8y+LWeCYwR3boc/xoG3ZManjcjfXAZGsNWDmwhTqGjprv0MzSb5WrpVuEBM498= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541042; c=relaxed/simple; bh=SKt6xw4rOqcfu23Q5IMqrvmQXBfEpcgpGIEn3KpHQQU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KtjvI9hQZMBekAS226G+ZNa8Inh+jFqx6ahnGy7pdTNVWYpFSteyDTijJT8ijcNtm5tRIwymgR0VH6SoECTVMZG6WwSt3463pCAUsfifjQgOv8A3nq/eGx937/iMcR17p269doEpweDnZoaV4uOT0ZvALSxORyIbUHXIdVmqWI0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=XUHJTDzb; arc=none smtp.client-ip=74.125.227.166 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="XUHJTDzb" Received: by mail-pj2-f38.google.com with SMTP id d9443c01a7336-2d747ec6185so4276525ad.0 for ; Tue, 15 Sep 2026 23:44:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541040; x=1790145840; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uBBuOi4IklhpEQjilPGFjs1/1RKNgj5LsNpTgitcyy0=; b=XUHJTDzb+pcB/LHIsTZogG5vKSYWdIuyCLEZlE17wYCr72fst4juIM9WtjEkk/7Epo 2mN4l5nVFwet8+6/a+xOAkOQxHOhVj5GRK/aiSxxEy6s2cO234gR4h4kAVSYE9KPueF2 Sw7yN47i4M6yLTWFthe71fGKLhYsSLqMUvnsWL8FtEWuI8Wj3IsGQMW4y55yPbpPXtDg Vq6hTBUwGeuS5UNG3BfpfGMh96gYFxmMSaSc+Uk+rsjMWfeWeFMNroZ1PJso+MhFCIFw 8hVN9UWw7Z2+PgRKHIv6SQTIRv3EOy1MHWCw6nK2M5jZX0uenz/WmEG823veM+TT5Hfk 7mrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541040; x=1790145840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=uBBuOi4IklhpEQjilPGFjs1/1RKNgj5LsNpTgitcyy0=; b=b5tDsdbEulQYSWj0fI+KcfoqoSTQ5bnUQded9hY2zPf1QyW93QwxxSSAyjRgoQAFyz tfFYU3P6Tq8VVDRlB7ikzN4gC04eJMTgmi5hRWNSwlYhDmRjw/9GZFKYilYzEi9qpRR9 xzNC9o6MWgEDGityTCmj/ofuxNE2GKxWcGMpGwt1xwN9eiVS3IYEe32hrIegkhwJ0fHJ SEUjj9c7igHKgcKWjmmKHlgs21NtjsfbAWjVCTX+GHUnt9/QcajpQ8uKeqRQFvEo3Xs2 mF11E5GKin/pEOTYmNe+4V19+KJVfneY4/zKUmp1cIoTT+nafOHOCor9lW/wVF7sPrWY QdEg== X-Forwarded-Encrypted: i=1; AKwUvBxVDV9CqzXBXUvCZUYR2VzXUyiYcilNN1RfTyL2VcQ5d1JBWG/Q+WNaa+7JWZGcecmpwedRBhSGzq09+PU=@vger.kernel.org X-Gm-Message-State: AFuF++kHm1qZ1NugDeRV9jpR1/d5HGNz7/8kHrOpAUnQj/u4cNOSfYUi 2FEsHSjjztG38pvntQqb3Q+c007csoHco5KWC9nWcpYMac+A6MvFfIN5qoZQTZ6zxQs= X-Gm-Gg: AYBFou0bjVMTKieK+wtO+B83lYhiQvDZZOq5bRWcHXYwlNbMSHVGWxeRVwwvNyNy0ex JvqlRCmcyz9Fuse+Cf7cqVyRxVLw1QbhGNLphpDmaNJXhEl6dob+He60u5afibjz60zKqeXeDBF EImIVDx+NuhC7FsGEedbMjqW3xnSJDL6UNHla0hWoegrT/vKseEse8XxVSxp8R4+DgHNiueudOf eBvcDfd1tOrrbHV77dD3aM0cWNUd/7qyCgI33pq+RXJQk4agaUrYBWEjNWFiMI1hm5ObryN6/gz vjiYSxpx799LspZ5jKwOIH7FKYWtx50nbjDxxMCDPXqYCRvKAE1uPKDCOJ3yhKPqOjQYfSFem3t 0kivl9+wcjdswsxRGAemgrdc562fsGlfMhnSUfgiUyidz6/Z/toXFa8vs0Db4M1rSR08mBWVEr2 wQ78q0JydC6XWaa43x0Vh8lOmKJxTvOftIQsWv9qMci4G9FX74YDuuh/WssTHdz9mNCSXqAZ6ep QH/q1AQ2RVwn6U2Ek5T X-Received: by 2002:a17:902:ffcb:b0:2bf:27b2:4b80 with SMTP id d9443c01a7336-2dd8e40607cmr31031175ad.14.1789541040200; Tue, 15 Sep 2026 23:44:00 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.43.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:43:59 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 02/11] mm/sparse-vmemmap: factor out shared vmemmap tail page allocation Date: Wed, 16 Sep 2026 14:43:32 +0800 Message-ID: <20260916064341.1825793-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB and sparse-vmemmap each have their own helper to allocate the shared vmemmap tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper. It allocates the page through vmemmap_alloc_block(), and uses cmpxchg() to install the per-zone shared page. Expose zone->vmemmap_tails under CONFIG_SPARSEMEM_VMEMMAP to match the shared helper's build condition. This avoids a !CONFIG_VMEMMAP_OPTIMIZATION stub; when optimization is disabled, the array has no entries and the compiler folds away the unused paths, so no storage or runtime overhead is added. This removes duplicate allocation logic while still handling both the early boot and runtime paths through the same helper. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) --- v4: - Update the commit message for the renamed VMEMMAP_OPTIMIZATION config - Collect Acked-by from Mike Rapoport v2: - Collect Acked-by from Qi Zheng --- include/linux/mmzone.h | 2 +- mm/hugetlb_vmemmap.c | 28 +--------------- mm/sparse-vmemmap.c | 74 +++++++++++++++++------------------------- mm/sparse.h | 1 + 4 files changed, 33 insertions(+), 72 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 56403841e887..97147c764658 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1156,7 +1156,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_VMEMMAP_OPTIMIZATION +#ifdef CONFIG_SPARSEMEM_VMEMMAP struct page *vmemmap_tails[VMEMMAP_OPTIMIZATION_NR_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index f977d0a7e002..5ddf06b83c96 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -493,32 +493,6 @@ static bool vmemmap_should_optimize_folio(const struct= hstate *h, struct folio * return true; } =20 -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - struct page *tail, *p; - int node =3D zone_to_nid(zone); - - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail =3D alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p =3D page_to_virt(tail); - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail =3D READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -535,7 +509,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struc= t hstate *h, return ret; =20 nid =3D folio_nid(folio); - vmemmap_tail =3D vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail =3D vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; =20 diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index f22d815d7af0..7388a5b5cce3 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -42,27 +42,13 @@ #include "mm_init.h" #include "sparse.h" =20 -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask =3D GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order =3D get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; =20 page =3D alloc_pages_node(node, gfp_mask, order); @@ -76,8 +62,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, = int node) } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } =20 static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -184,39 +169,40 @@ static void * __meminit vmemmap_alloc_block_zero(unsi= gned long size, int node) return p; } =20 -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct = zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zon= e *zone) { - struct page *p, *tail; - unsigned int idx; - int node =3D zone_to_nid(zone); + void *addr; + struct page *page; + const unsigned int idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; =20 - if (WARN_ON_ONCE(order < VMEMMAP_OPTIMIZATION_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >=3D ARRAY_SIZE(zone->vmemmap_tails))) return NULL; =20 - idx =3D order - VMEMMAP_OPTIMIZATION_MIN_ORDER; - tail =3D zone->vmemmap_tails[idx]; - if (tail) - return tail; - p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + if (likely(page)) + return page; + + addr =3D vmemmap_alloc_block(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); =20 - tail =3D virt_to_page(p); - zone->vmemmap_tails[idx] =3D tail; + for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page =3D (struct page *)addr + i; + mm_zero_struct_page(page); + init_compound_tail(page, NULL, order, zone); + } =20 - return tail; -} -#else -static inline struct page *vmemmap_get_tail(unsigned int order, struct zon= e *zone) -{ - return NULL; + page =3D virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) !=3D NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(addr, PAGE_SIZE); + page =3D READ_ONCE(zone->vmemmap_tails[idx]); + } + + return page; } -#endif =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct vmem_altmap *altmap) @@ -229,7 +215,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long = pfn, int node, return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); - page =3D vmemmap_get_tail(order, zone); + page =3D vmemmap_shared_tail_page(order, zone); if (!page) return NULL; =20 diff --git a/mm/sparse.h b/mm/sparse.h index d8c08a388e4e..8f6ef0b50b4f 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -142,6 +142,7 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 269E93E7174 for ; Wed, 16 Sep 2026 06:44:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541047; cv=none; b=Q/Q/CkmJWV58rm6S1gPToPaOAo4Wj6KZybpU4Z2XEr63ti2KlFmGi3X5wrqnUcQtVBXc16AJEY7UePpAPBkLn0VYdGuNRZgSHGsyHjMxTMnZ+LTHzLbYRzAmpsVrt8MFd+fx1eAlrwlK8zeW41onylgi3DH1X5p9c3MGBFJvVrI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541047; c=relaxed/simple; bh=yuRP77nW6fuC094S8aFOWQK8lAaxmYpqi+IRrf5Gu/8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qgK5+qySImWOOBlhryLSxOIrBmeM1cgiG5Uo754lDv4Z2r6ex6VUkV646g0doqGN4adB/R9WvpuZ4kQ9gDotJBwJ8vIP/RHjikcoihiNd43wD0blxKK80Px02u2shwwK+ba3u8CyyhkGtWSlGxhGzpHr0RMcaVqHBv7OqhMd2ko= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hsJqX4/Y; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hsJqX4/Y" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2d747eefae4so1578365ad.0 for ; Tue, 15 Sep 2026 23:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541045; x=1790145845; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=PDFzpc+a9bATnmd50P4M+vPlS8YdthhE8ho3W/sX1lQ=; b=hsJqX4/YYPHcbizwIaHCQv/jLgIxStbze3exY8AH5Wk2aAh6X9ffSNrMV5sCi7RJF9 vmEewRNqadtOeQf28Vlqa62M+o1oQHfUL59w8vUMHLeMiT612fUU7gcBfpinPL009Kkr Nkx3m27Qt6Voy/4IooArfEImgDnSlobcLbnim4ZO9/bMm3fo/Mee5F+b0itt03zIv1vl hUgUfdCYe0CozezckpgWNIRqSCkLMeQoKBB+Rpi4bW/s22R25pmoQi5+keYJhL3mQG4E Eqm3+LoZsp7PgdXfY2jgcma5N6L8BWx2t8ckohWtOHclSTSu0g0HzXo+VFR+b2K7XPUG /9EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541045; x=1790145845; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=PDFzpc+a9bATnmd50P4M+vPlS8YdthhE8ho3W/sX1lQ=; b=E6yLLJyn5DCwU/ODxUkxj1hlRd9J/FPB84pQZGze+ZOaIJRDdp5qHPFQi4IDxfqoui XatQR3DsvJFCr9cKsxRwmY8Avj4ULt4YwnyyPuGKSCKUiq/Sr5twn2sCnOeJsdmTRqKC 9zD35bfoPoKOR/hy8cBHT18+TN3R5h4Syi32+Eyods10PfshzOFsCiMTqzH2Bp+35Cxa nzcpWFmfivMojEiwO4N7ozI1MD4lHBBQK8K4n/Vk2Z/xgOGW9J97v6ag/TcNGSKrvl0t vg9VcRkpSIJSLp0SQiUuInUrNZLPaEyoaoeVZdFI7I6gi9H+5EY0SXyXeyFC2199dOEc E/CQ== X-Forwarded-Encrypted: i=1; AKwUvBxv3WikGiFA40DNfg8YS7OBsK3hauOCect4GywbnXXlYgwk9DjYuyPZ48JMryKzL5mdlPK8ecf8wM1F3vk=@vger.kernel.org X-Gm-Message-State: AFuF++mpeYBn3NuvH9DKPPYGuKidq7EoX7ZdZsOhx/esuhtdrywwjJrI ABQMqvVx7GHuT7L5oYxcVYyCqbb6rGyTYz9Mj+c/jAn8WjlI1V+Pq/MhWguMeOcWx2I= X-Gm-Gg: AYBFou3Cem0U3/TlKCdV++kMPKNpxVk0f7oALtYlJyV35fu2RSdSJxlrUuTFGMuxyT3 /F3Vt7o+N0nWIzdxO0f5uHuKR0GVeFOllROK5Wx6XOsxs+RF5p+U3s7+uDW6m4sUkaqb2INWv6J RFjtpqfriA+peRpFoFiZlEQhax8Rxlo3CvK2fB2QF2Nr+uJtcpXk22VnIUg16elq04KJHwuFYCC vK2TJdgnjNOZ7ikWoLeg7NyvpMgNnv2jDvUaas+fIIk5UqE/A2H5UiwfT+dTG+5+U36ZfMTtMQG hZFd9QGlt1UDxOy5iJbqJ09txp8avIBUk+dGpWWcLRc5JzwD+s078FgyGE951zQlQukd4MT70xg dCIZEAUVnQZOWhedvu5DDnuTRvpvIm6f5IVCEG4PdRvB/cxT55WTKTW4QiLTX+vzezevJt8Ihk6 WO0RWQ7cDEGbGwKJHChD8y93oXxYTz6numzOXWrxG44WhB9nJaiITe/yIq++u0bA5P0uFyRiSO6 01sXsxoivprgbOQc5M= X-Received: by 2002:a17:903:3b8f:b0:2d7:4bc8:41a4 with SMTP id d9443c01a7336-2dd8ea1cd1dmr17328145ad.10.1789541045356; Tue, 15 Sep 2026 23:44:05 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:05 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 03/11] mm/sparse-vmemmap: open-code init_compound_tail() Date: Wed, 16 Sep 2026 14:43:33 +0800 Message-ID: <20260916064341.1825793-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" init_compound_tail() is only used by vmemmap_shared_tail_page(), where the shared tail page setup intentionally passes NULL as the compound head. Keeping this helper in mm/internal.h exposes that special case to the rest of the MM code and can make the NULL head argument look generally valid. Open-code the initialization at the only call site so the special-case use stays local to sparse vmemmap optimization. No functional change intended. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: David Hildenbrand (Arm) Acked-by: Mike Rapoport (Microsoft) --- v4: - Collect Acked-by from Mike Rapoport v3: - Collect Acked-by from David Hildenbrand v2: - Collect Acked-by from Qi Zheng --- mm/internal.h | 9 --------- mm/sparse-vmemmap.c | 5 ++++- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index da14c56fb24e..0dca33db068f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -786,15 +786,6 @@ static inline void prep_compound_tail(struct page *tai= l, VM_WARN_ON_ONCE(tail->private); } =20 -static inline void init_compound_tail(struct page *tail, - const struct page *head, unsigned int order, struct zone *zone) -{ - atomic_set(&tail->_mapcount, -1); - set_page_node(tail, zone_to_nid(zone)); - set_page_zone(tail, zone_idx(zone)); - prep_compound_tail(tail, head, order); -} - #if defined CONFIG_COMPACTION || defined CONFIG_CMA =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 7388a5b5cce3..861e09b2b096 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -189,7 +189,10 @@ struct page __ref *vmemmap_shared_tail_page(unsigned i= nt order, struct zone *zon for (int i =3D 0; i < PAGE_SIZE / sizeof(struct page); i++) { page =3D (struct page *)addr + i; mm_zero_struct_page(page); - init_compound_tail(page, NULL, order, zone); + atomic_set(&page->_mapcount, -1); + set_page_node(page, zone_to_nid(zone)); + set_page_zone(page, zone_idx(zone)); + prep_compound_tail(page, NULL, order); } =20 page =3D virt_to_page(addr); --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18F2D3E559C for ; Wed, 16 Sep 2026 06:44:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541054; cv=none; b=KYma27yj6BOSijE5jkvt3IidedriqqipsL+gt7RwZf6blpEaCHYx8ZjT0M+K2CcIrUzt19xxJ6PWmdYUXSRBcyQjxKIO4QOganehjLwEnW0uvqmJfNZbLLVqAa2+hDnmHtuDVIIh9ZI95yLaoNvrsEgM+ho1aBNOk6St3mMT1Tw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541054; c=relaxed/simple; bh=4C/H0OXURhGuKf+braJ+uKcmMn4yaEgFfA4J4Js2eLc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LyL3SfHnY+T/eM2fqbuWEa2GPAU5gXuQ6kWnQ5rykskduTrzzKJgEQlfhR/vpdvMXFR/5T8Z/HiTMP8UT+7p40/AKcz74QGGztOzG0X6iS09KjRNZ8T5PZb0OdwXNhyDDVMredyAaJGI5cbP7XcF24AR+xZ2IpNEVu0PRl9hYFI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IKPamaUM; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IKPamaUM" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2db18fe433fso5142495ad.2 for ; Tue, 15 Sep 2026 23:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541052; x=1790145852; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=jrSUwqa7xtL2ZBs1WlB04Z11tmz781BnHoaaNOznkvI=; b=IKPamaUM5+ChDYemNMjT2VaTTngBn1onpMDClpXULJ+Cg7kn6q6L6ripwuicO81emm dsXpYSm0ZpwPVcLDaqYIUek6eAZj4aVrGxoBN8TTDiWZUdXtpbwS2a34pY55/6c6hgUn wDMZBmn4ZUhSn4OlzfesIhiwLoCrPQvi2DqmI9uyxSAYSgmKam0xdSVcpQd1UZmxzmiz X/cC/QJs5f0s0kMdvVGnCaPtX6EV3ZwSG1tQc0EFbDotEM+cTOwqG7NJMMpJdtOpKe3b 2ys3LosuaSghvDwVNcEjgIji2OYh5BQoH5a7vdhbFvLo5QyjLl95RmX0bJXYN/06EFNx R9cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541052; x=1790145852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=jrSUwqa7xtL2ZBs1WlB04Z11tmz781BnHoaaNOznkvI=; b=yAPs31Cj38MqTmPav4mTPr4z2Fj67zRUr1X909owf8+ATW7VkwlBgEqRCk8yximdhD jGK794KARsqihJ53HijyZrRcCRUWQ46iUwaGiWaQy/ep6MId0KLBxvNGyK/1iUyqW0cK WRmPQvhW27N9lW6DGcyMGhLBP0Mpulz4U2fabxpHTcGd2MQhs51IFjhfsCpukdsms660 dpbc7c9U6oJZ9SqgTTVX79aQGh0JRInhF5f8mD27WjSApSxbcewIKHQIEVH3hcdNbGz4 ji7iZYMox5RiG5nT7iG4A59EhlsHgINpSOzmUDS7QFRFA7I9GrIU8rMN+LuDTUBv8+YD +S1w== X-Forwarded-Encrypted: i=1; AKwUvBz3xOi3g4XAaiR7UdZLAQLPA+gt8Gzq+L3egAvPQfix4MRV2pwjB2G7vFLoZp3HlA3E3WqS0uTP3mhycjY=@vger.kernel.org X-Gm-Message-State: AFuF++nhNuD/6uTQRC1zOMA9HUjB6uztDDFErkZZ7P9+qPGeuknkzNbP V/6aAmIaL5Zwo2fEXfb+SEAjJp/KdxmdkhXq02shEobUNrAatfUVQheGpcezzeLqgm0= X-Gm-Gg: AYBFou36CjisdWjEQuzFED+NHgJ9L2pNme0jRQ2wLihGH8DpFg4+rqjR9qd6N5TMAL+ uWdioefIa2oPBElGQT/KiAInbXGMXI1d45bbGyy+jIidJblzc9hnTVn1goctTDLB6Gqnk4mC/Jn ny/XMKqPOKZveg2hRxBjZPmvLAx1vdwWT1vb9vC6Mc1oPwgqv06YFFWV+ESlC1bZdKglnpaJ4Sr yEb6HM5l8TH+aMqNXI+SS5US/S/XbUCu0pj596eC+285r5JJYj9oy/gYQ97Z1IFm83lydSyLITK ulkSXr/gHZOcWutxrU1mEwHFif0xoXAl8QVK6NCcPmJ/PIqM6Dkup3ipicjTq4840siKcIh6odS ku6u3zxk1bYJ1fV2l8xoki5z3rCaXDIhSyDiT7LNuiatZIkbp6yqK15H07cvuAkn4nU0rJir/0V /F73WszEQ3syohYdcUMtOJapygdAYwRdopYXquhBw77UmyzqpNytDLaQk6MG1wFYi1mZ3i4XFf/ 61HgbzztgwzkmPxUNM= X-Received: by 2002:a17:903:1251:b0:2d8:d4d0:792c with SMTP id d9443c01a7336-2dd8e752f13mr29866745ad.16.1789541050075; Tue, 15 Sep 2026 23:44:10 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:09 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 04/11] mm/sparse-vmemmap: prepare DAX vmemmap population for compound page orders Date: Wed, 16 Sep 2026 14:43:34 +0800 Message-ID: <20260916064341.1825793-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX still uses vmemmap_populate_compound_pages() to populate its compound-page vmemmap mappings. That helper allocates the head and first tail vmemmap pages explicitly, then reuses the first tail page for the remaining tail page mappings. Device DAX is being moved to the section-based vmemmap optimization infrastructure, but it cannot switch to the generic section-based population path yet. Once a later patch records the DAX compound page order in section metadata, DAX head and first-tail PFNs can look optimizable to the generic helpers as well. Add a DAX-specific population flag for this transition. It keeps DAX head/first-tail allocations on the normal vmemmap allocation path, while preserving the existing page reference for reused DAX tail mappings. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Update the subject and commit message to use compound page order terminology v2: - Collect Acked-by from Qi Zheng --- mm/sparse-vmemmap.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 861e09b2b096..aa89c16f7fc7 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -35,8 +35,8 @@ /* * Flags for vmemmap_populate_range and friends. */ -/* Get a ref on the head page struct page, for ZONE_DEVICE compound pages = */ -#define VMEMMAP_POPULATE_PAGEREF 0x0001 +/* Vmemmap population for ZONE_DEVICE compound pages */ +#define VMEMMAP_POPULATE_DAX 0x0001 =20 #include "internal.h" #include "mm_init.h" @@ -208,13 +208,17 @@ struct page __ref *vmemmap_shared_tail_page(unsigned = int order, struct zone *zon } =20 static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, unsigned long flags) { struct zone *zone; struct page *page; const unsigned int order =3D pfn_to_section_compound_order(pfn); =20 - if (!vmemmap_optimizable_pfn(pfn)) + /* + * Device DAX still relies on vmemmap_populate_compound_pages() for + * head/first-tail allocation and tail-page reuse. + */ + if (!vmemmap_optimizable_pfn(pfn) || flags & VMEMMAP_POPULATE_DAX) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); =20 zone =3D pfn_to_zone(pfn, node); @@ -236,7 +240,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in pte_t entry; =20 if (ptpfn =3D=3D (unsigned long)-1) { - void *p =3D vmemmap_alloc_pte(pfn, node, altmap); + void *p =3D vmemmap_alloc_pte(pfn, node, altmap, flags); =20 if (!p) return NULL; @@ -251,7 +255,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pm= d, unsigned long addr, in * and through vmemmap_populate_compound_pages() when * slab is available. */ - if (flags & VMEMMAP_POPULATE_PAGEREF) + if (flags & VMEMMAP_POPULATE_DAX) get_page(pfn_to_page(ptpfn)); } entry =3D pfn_pte(ptpfn, PAGE_KERNEL); @@ -511,6 +515,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, unsigned long size, addr; pte_t *pte; int rc; + unsigned long flags =3D VMEMMAP_POPULATE_DAX; =20 if (reuse_compound_section(start_pfn, pgmap)) { pte =3D compound_section_tail_page(start); @@ -522,8 +527,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, * with just tail struct pages. */ return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); } =20 size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); @@ -531,13 +535,13 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, unsigned long next, last =3D addr + size; =20 /* Populate the head page vmemmap page */ - pte =3D vmemmap_populate_address(addr, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(addr, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 /* Populate the tail pages vmemmap page */ next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, 0); + pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); if (!pte) return -ENOMEM; =20 @@ -547,8 +551,7 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte)), flags); if (rc) return -ENOMEM; } --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A1313D413F for ; Wed, 16 Sep 2026 06:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541057; cv=none; b=NnGyY7lPj2f0swIBSbY/s6XtUZIpHU3JiQMlPwb8n9SMvhsE4HF1SbQdiOoE1rUgUs92d5qa9M20ZqkTorKU9EK7TpREr9Ve2S1EjZPRSUrZRvsYmTfptfDSBAh77bTyhOnR/NDDcCcHV/CPeT4ARgOC9Tsn1H348fIQYT0iVv0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541057; c=relaxed/simple; bh=q4sE4pbRykd1vhFCtA51ekVFkV6gQ95xL4RRztXC6GQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ucGBy2mlY2nz2B8pdah8YaBoQ7A0J4eBbOioTxQ5BpiIDUpzntHdsL9tgHe5w+8pkqgdA84O6ZqKjku7ZXZ2XiYfK186SYtItPjFMfd91kqCyzv31mjPj8ZPm60fnXzQiF+KTvY0PoLaWLMYycesMtnbUujMAwQU0OMOw7dEWV8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dhFRSzAh; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dhFRSzAh" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2dd88a115c1so6145925ad.3 for ; Tue, 15 Sep 2026 23:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541055; x=1790145855; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6sM9zpfll+quDztLFFF0jfXeRCau4il9IJzrgeT16pM=; b=dhFRSzAhBYhUpFbaQND345zR2MbdFHFLnZjvDC4/8NFXgiBSdit8QQ5ed9J41+mrWS xeUptWNmzW+2pdF9OH8Hu8YIooZqh6cSGoURI1aWSWHvhkaR7WajWJPt3+/k4pcKuRMr DoPujnc18aQ96e8ArTku22Hsb6afURyuM9Y6H8HYxYvTfiUEfPpuOPCydHJ/NySopAX9 eNi2akAJcqx5qC9kaY5oQtdK9P1FThXwpX7UhlEyj8qe2Gfc7atiY/Qk09J+lC/lP7IF JKRqb8IQdqLJ24jDrMEfs7spBA7MZUmS9pJ/Sltz8vZylaBxdu04mRy+9WQncZMcolw6 xFNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541055; x=1790145855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6sM9zpfll+quDztLFFF0jfXeRCau4il9IJzrgeT16pM=; b=vHc8C0Mm1JVC7h59q7xbeQ5Ib3w9GNHF+WL577Xv9wqLLNc2zLLaSqJKiveKCMKQNv XWoI/p2g+BUqxQaIIN3IBfPJ1an5/2kXcAwgREkmPmxl+CfWUCAtZpCzELfubtTa0Vwd hxpZreFKDqA9UqHpKAigjY948cS6zrrMfSZIU0ziCAv2FnUB1hF5tCX5kAm6e4Huuv8Y TUgOfd4fHQEtz6PMZxk3Bwnm2ZQGgbOa7nTKMEImAsl/bpdNYRvdqa5e6RTBDNhIcECB y3n4Qzjr2Hzf7Tt+rNIXbPfOeB9Ca1RHmZQXUIKYWw7j2AzhiydZTxBONoql7mVQpzXX rMXw== X-Forwarded-Encrypted: i=1; AKwUvBzauf6fO+s3cJBaxv+EWpRUvqmeXyXF2i/hiSXaBQaiYSnM1NMzfQTcGg5/rHonBRcOkwQFRZeX/dDjFpA=@vger.kernel.org X-Gm-Message-State: AFuF++ktjZjyrbVHOEao/ZUDgPGKXVBt0JCQ1ocVTb1zMwZ6lgQYR6kQ 5V68hSDbRFSIrnhrvS+t8GY/gzye8p6+dfb7zQsrDv5GyH/3g3dlGGaE+PviblsnRG8= X-Gm-Gg: AYBFou0HCZ7f8hrYXfPnjgVGqPbo7LScLW9UITfykNJvQ5QzW5ban9/KjflEN+ZHPXj XPSx1HmTinwqU3q6gGWtbd4V8NMxLtiEUCeJdq1eCr6DR4u4Tiru8bOmAI9i+nmoNkNTW/mMs2G xQqRcWAZ8eXjTwJH+GxPXS7h8fjjnvH6Xu5I6GcfS80Go/hMyKUOvvkykgqHhprYWNG+NLch38/ 5mUOZ0LZ/avz6qgHYlkQ3a1f2gDq03htMYNpVGspM6qvDN3qEzipHuXQpyHxqngDc5/drpAPnXx 5nHfBtfXrOE6+JAXWzrrBvHxN4ql8qxIGczoxMQObUXUao18YiHiL9R6Hr97BsMVT6YqxWe+if6 h8dENvrjRmVsOsiEdCiyRgXFMBmIZRygQyx88E3JB82VoD9dLY2YN38Kv/agxWjG8FlW+fX6UvT B0FPZWE79cpxZcA+w865ZSv1OUghXt6ALNWoAJClwlfPH0fnxsiwz9ZXRW8hgJV/tBd+CksX0Jv e40sCQaRm/Qyk09qqPC X-Received: by 2002:a17:903:1b2f:b0:2d5:2f49:b3f4 with SMTP id d9443c01a7336-2dd8dbf6ed0mr31915195ad.2.1789541055247; Tue, 15 Sep 2026 23:44:15 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:14 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 05/11] mm/sparse-vmemmap: set compound page order for device DAX Date: Wed, 16 Sep 2026 14:43:35 +0800 Message-ID: <20260916064341.1825793-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX can use vmemmap optimization only when a full section is populated with a compound-page geometry. Record that geometry as the compound page order in section metadata before populating the section, so later vmemmap accounting and population decisions can use the section state directly. Clear the compound page order when the section becomes empty again. Also reject partial additions to a section that already has optimized vmemmap mappings. compound_nr_pages() determines how many struct pages to initialize with a section as the smallest granularity. A section therefore cannot safely mix optimized and ordinary vmemmap layouts. Partial additions continue to use ordinary vmemmap population, so they do not save vmemmap memory. Such additions are uncommon, and the lost saving is negligible. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Update the subject and commit message to use compound page order terminology - Use EOPNOTSUPP instead of ENOTSUPP v2: - Explain why optimized and ordinary layouts cannot share a section (suggested by Qi Zheng) - Collect Acked-by from Qi Zheng --- mm/mm_init.c | 15 +++++---------- mm/sparse-vmemmap.c | 16 ++++++++++++---- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 97e0158d2aca..efffa8609b85 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1049,16 +1049,11 @@ static void zone_device_page_init_from_template(str= uct page *page, * of an altmap. See vmemmap_populate_compound_pages(). */ static inline unsigned long compound_nr_pages(unsigned long pfn, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - /* - * If DAX memory is hot-plugged into an unoccupied subsection - * of an early section, the unoptimized boot memmap is reused. - * See section_activate(). - */ - if (early_section(__pfn_to_section(pfn)) || - !vmemmap_can_optimize(altmap, pgmap)) + const struct mem_section *ms =3D __pfn_to_section(pfn); + + if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); @@ -1144,7 +1139,7 @@ void __ref memmap_init_zone_device(struct zone *zone, memcpy(&template, page, sizeof(*page)); if (pfns_per_compound !=3D 1) memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); pfn +=3D pfns_per_compound; =20 /* Initialize the remaining head pages from template. */ @@ -1160,7 +1155,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; =20 memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); } =20 pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aa89c16f7fc7..02da1321197d 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -135,14 +135,14 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms =3D __pfn_to_section(pfn); - const int order =3D pgmap ? pgmap->vmemmap_shift : section_compound_order= (ms); + const int order =3D section_compound_order(ms); const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); =20 - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(= ms)) + if (!section_vmemmap_optimizable(ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); =20 if (order < PFN_SECTION_SHIFT) { @@ -573,7 +573,7 @@ struct page * __meminit __populate_section_memmap(unsig= ned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; =20 - if (vmemmap_can_optimize(altmap, pgmap)) + if (pgmap && section_vmemmap_optimizable(__pfn_to_section(pfn))) r =3D vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r =3D vmemmap_populate(start, end, nid, altmap); @@ -792,8 +792,10 @@ static void section_deactivate(unsigned long pfn, unsi= gned long nr_pages, else if (memmap) free_map_bootmem(memmap); =20 - if (empty) + if (empty) { ms->section_mem_map =3D (unsigned long)NULL; + section_set_compound_order(ms, 0); + } } =20 static struct page * __meminit section_activate(int nid, unsigned long pfn, @@ -803,8 +805,13 @@ static struct page * __meminit section_activate(int ni= d, unsigned long pfn, struct mem_section *ms =3D __pfn_to_section(pfn); struct mem_section_usage *usage =3D NULL; struct page *memmap; + unsigned int order; int rc; =20 + order =3D vmemmap_can_optimize(altmap, pgmap) ? pgmap->vmemmap_shift : 0; + if (nr_pages < PAGES_PER_SECTION && section_compound_order(ms)) + return ERR_PTR(-EOPNOTSUPP); + if (!ms->usage) { usage =3D kzalloc(mem_section_usage_size(), GFP_KERNEL); if (!usage) @@ -830,6 +837,7 @@ static struct page * __meminit section_activate(int nid= , unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); =20 + section_set_compound_order_range(pfn, nr_pages, order); memmap =3D populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap, pgmap); --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3454D3EC817 for ; Wed, 16 Sep 2026 06:44:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541063; cv=none; b=YI9PT/Li7aZeu4HJ1kMd4eLrbHK+wiy6yH5dMdfSqlYcNBQy8sUdA3YnJyPknyKiwy4atQQ0uMRc7ERD2Zfw49v6VqUnE7PCAefM2fR++GA6oZJgzP/ASPQC0TArHw06nay6+R6P0gzFvmcoW1IMtaszvnT0DbHfsALT34N+KBk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541063; c=relaxed/simple; bh=QgI9M8JCvMkxfWw24tOIx4mGI9U98UyteMWum8iZNHU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L0mj9/IiTdnxGNIOKj3x8EppPXMXOmCsD4TAvQUOTpqrOMGfWPwOIAu+r5fLTS0SkXpSu6kFVe/KUu11T6CyfsygSGzgzTCwPPmkJzoNZVFi+NwP/9BSVCcbhB11V3VhKK9VIiGQ27alDTcR4tQz4L0QTMfkhvFMKYaOqf13YyI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RlRsxxoH; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RlRsxxoH" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2d747ed6d6eso4852335ad.2 for ; Tue, 15 Sep 2026 23:44:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541060; x=1790145860; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=b1EqIpH0CyAwdpMqjXcOe7F5BCJx4eubUzqhI5rvq2s=; b=RlRsxxoHMYYB4E0v6O/yoE7vr2B87JzmbdtVqkReLV3phahaQyccRhkLLjVttQ6dF9 QgE6fwyASHHBR2XW+OHoQ6a3Z/Y8QQmVbsr/BxrcgVABCucMIoSbOpDtInU7Fxv/yTHo w1h5du32nsnyFLnWQTJmr9COUe8GMxSLcZkHE8vt9VheN+a21IY8QUA7XsXkA3XtP2t8 gUXs3peKNpGuLqnfXs5k2dxsIRyVGQSJGEFBSCfuS5aOeGMWBQr2LP/mF6Vecx6WgXjX eeEvrW9DatCYOGyOkTrGXeQwntRdTKhor/NdyamMfBYcg3MplMiJAQPH8G3T/mEt3vP3 GeYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541060; x=1790145860; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=b1EqIpH0CyAwdpMqjXcOe7F5BCJx4eubUzqhI5rvq2s=; b=EXTpp7XNmSw4MtHKhLeexiJ8KwbuImBB36XHJBLn3kJAc2bXgdpIJbviSjsX4YG2t0 vKmJPWhs/0TuWRpOxNlHZBzqXBC03ZTTpbz66v30ydwkbInvk+b47xihAgz4IE0y1FWf cVo9sWrkfPykefqvS8hRd91chrOc9o8i53eOTPVAJAqGacUjCWX6iy0CpkLq84BAU6Gq HD5g8UT80KDJwlbfgXq4Kifz4eDQOz2mCzDnQnwp4WqmBY5HgYLSD21Y4l69Fpghf+xq EvPyqc4CCr7GNYH9fHjjLy4vdCJKbhwxsoS/zxBgNDtPA2+OZwjFjtfE8ijv7MzlcMyb Xs8Q== X-Forwarded-Encrypted: i=1; AKwUvBwtS2fsmd3U2ox55BB0CqXV8V7YtPJU5mZiAmt6ap4Yep/F4oxnR53j+EH5efpWFSFWD5k4z9H+CHYETrQ=@vger.kernel.org X-Gm-Message-State: AFuF++lHVH+qip4tnV0aDFsH6IDbv7Zra/P57Mtmj7oCBRwZWgi8EcJl Hj6VodEWGVZ1nnyBAqF1Tdja1NnRD5Gys5kmxMJVLfkvd5dn6U48zQX39rJE663M/+w= X-Gm-Gg: AYBFou0UEGiWsOezH8zY338RVviSMLaiI3yyWBDZ+QQN0O1ecK0Hn1L0ZZOj3iZERb2 v061es3noCtbmw8Si+bVLTYAGknq7VMUB55WXS8IgLCXbbdjNmj8S5GfYC8tjnpW8VNTSfm5JyH ki80rSsKSKeOpY4FOCEbip7X8+V6JJheTltCzGBi48DM6NIEEzJRToRs4HdFJSl286XuoV0nFkw 7/i/5mIMqxzYun7WyzZwRKszpHPyUqTVwGbZJYZof20pU1OL3uWu9zk4BGSEe73VqWoI8HF5LVN MaQKuQpNkrRo1alSx4EgDod7doyWUh6X3JagdvBbqfD/dzbjHfu+cFb+5rPyIaDvdE5V/+9C6Fc Kaj6qk9/TtL6v+rb38ZBf1Ebxppu/RLYKhBW2Hi52nzlvXwxjYJLwL6R1SFHZQ7IQnlietJCMdV QgpCfsz9teOMn9s9d1yfUhJhnYuSq+vpH/X/mgmKwG5pflg7DCCQB4024b3PdJQTct2nw5dzJOi wjFHqTYBWCQF1sZblnE X-Received: by 2002:a17:902:c401:b0:2dc:fcfc:bdeb with SMTP id d9443c01a7336-2dd8e76f78dmr28974185ad.22.1789541060351; Tue, 15 Sep 2026 23:44:20 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:19 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 06/11] mm/sparse-vmemmap: switch device DAX to shared tail vmemmap pages Date: Wed, 16 Sep 2026 14:43:36 +0800 Message-ID: <20260916064341.1825793-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" HugeTLB vmemmap optimization now uses per-zone shared tail vmemmap pages. Device DAX has not been switched to that mechanism yet. Switch device DAX to vmemmap_shared_tail_page() as well. This aligns DAX with HugeTLB by using the common per-zone shared tail vmemmap page. The optimization is enabled only for DEV-DAX through pgmap->vmemmap_shift, which supplies the compound page order recorded in section metadata before vmemmap population. Unlike FS-DAX, DEV-DAX does not modify tail struct pages, so sharing them is safe. Since the shared tail page can now back ZONE_DEVICE vmemmap mappings, initialize its entries with PG_reserved for device zones. Also skip poisoning vmemmap-optimizable sections while their struct pages may be shared. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Move device_zone() after the definition of NODE_DATA() to fix non-NUMA builds. - Update the commit message to describe the compound page order stored in section metadata - Collect Acked-by from Qi Zheng v2: - Explain why sharing tail vmemmap pages is safe for DEV-DAX (suggested by Qi Zheng) --- include/linux/mmzone.h | 10 +++++++++ mm/memory_hotplug.c | 6 ++++-- mm/sparse-vmemmap.c | 47 ++++++++++++++---------------------------- 3 files changed, 29 insertions(+), 34 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 97147c764658..ebbda6f31139 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2149,11 +2149,21 @@ static inline int online_device_section(const struc= t mem_section *section) =20 return section && ((section->section_mem_map & flags) =3D=3D flags); } + +static inline struct zone *device_zone(int nid) +{ + return &NODE_DATA(nid)->node_zones[ZONE_DEVICE]; +} #else static inline int online_device_section(const struct mem_section *section) { return 0; } + +static inline struct zone *device_zone(int nid) +{ + return NULL; +} #endif =20 static inline int online_section_nr(unsigned long nr) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index b428da66d279..d7a59167bec4 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -43,6 +43,7 @@ #include "mm_init.h" #include "page_alloc.h" #include "shuffle.h" +#include "sparse.h" =20 enum { MEMMAP_ON_MEMORY_DISABLE =3D 0, @@ -554,8 +555,9 @@ void remove_pfn_range_from_zone(struct zone *zone, /* Select all remaining pages up to the next section boundary */ cur_nr_pages =3D min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - page_init_poison(pfn_to_page(pfn), - sizeof(struct page) * cur_nr_pages); + if (!section_vmemmap_optimizable(__pfn_to_section(pfn))) + page_init_poison(pfn_to_page(pfn), + sizeof(struct page) * cur_nr_pages); } =20 /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 02da1321197d..7b5b9ceec697 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -193,6 +193,8 @@ struct page __ref *vmemmap_shared_tail_page(unsigned in= t order, struct zone *zon set_page_node(page, zone_to_nid(zone)); set_page_zone(page, zone_idx(zone)); prep_compound_tail(page, NULL, order); + if (zone_is_zone_device(zone)) + __SetPageReserved(page); } =20 page =3D virt_to_page(addr); @@ -490,23 +492,6 @@ static bool __meminit reuse_compound_section(unsigned = long start_pfn, return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; } =20 -static pte_t * __meminit compound_section_tail_page(unsigned long addr) -{ - pte_t *pte; - - addr -=3D PAGE_SIZE; - - /* - * Assuming sections are populated sequentially, the previous section's - * page data can be reused. - */ - pte =3D pte_offset_kernel(pmd_off_k(addr), addr); - if (!pte) - return NULL; - - return pte; -} - static int __meminit vmemmap_populate_compound_pages(unsigned long start_p= fn, unsigned long start, unsigned long end, int node, @@ -516,21 +501,18 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, pte_t *pte; int rc; unsigned long flags =3D VMEMMAP_POPULATE_DAX; + struct page *page; + unsigned int order =3D pfn_to_section_compound_order(start_pfn); =20 - if (reuse_compound_section(start_pfn, pgmap)) { - pte =3D compound_section_tail_page(start); - if (!pte) - return -ENOMEM; + page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!page) + return -ENOMEM; =20 - /* - * Reuse the page that was populated in the prior iteration - * with just tail struct pages. - */ + if (reuse_compound_section(start_pfn, pgmap)) return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), flags); - } + page_to_pfn(page), flags); =20 - size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); + size =3D min(end - start, (1UL << order) * sizeof(struct page)); for (addr =3D start; addr < end; addr +=3D size) { unsigned long next, last =3D addr + size; =20 @@ -546,12 +528,12 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, return -ENOMEM; =20 /* - * Reuse the previous page for the rest of tail pages + * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), flags); + page_to_pfn(page), flags); if (rc) return -ENOMEM; } @@ -883,13 +865,14 @@ int __meminit sparse_add_section(int nid, unsigned lo= ng start_pfn, if (IS_ERR(memmap)) return PTR_ERR(memmap); =20 + ms =3D __nr_to_section(section_nr); /* * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - page_init_poison(memmap, sizeof(struct page) * nr_pages); + if (!section_vmemmap_optimizable(ms)) + page_init_poison(memmap, sizeof(struct page) * nr_pages); =20 - ms =3D __nr_to_section(section_nr); __section_mark_present(ms, section_nr); =20 /* Align memmap to section boundary in the subsection case */ --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C5D63EC832 for ; Wed, 16 Sep 2026 06:44:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541067; cv=none; b=HNZaGQjp0mCy67kZezB4czujNGk0b0wGGioMY6JdLWkI27Hi52Ey2CXRaFAYoMn+O4RifFfuoqUVHNHqHx/87eRTTTMk7mCuA/R9D+YNDM7JR8h0P/AdTn5Qv5JEQtiZCSIRXfiMOM+RsK5CfiM9lAN7vFkb2/z+I16WzM9S7VA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541067; c=relaxed/simple; bh=41ZOaSS795Zy+Ullt5d+75xCPMwdSbCwgTHk1Zi0gZY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MC9hNSqK6NuU3g4V6no0Doa2yusCtPLCRxLUAodVrSM6HXFHV+QnQ+FhaVE/1O2P3EFJOTOl1PRg732pxkvLcPKqW4hGPWTGw9OriCpONRQ8dQjx/8yC8y4Kq1wMrm46H3LnRGFP833psGsmW1ZPM0M9FQxYBXo3v3vDZvxxEnA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MxLsaHmi; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MxLsaHmi" Received: by mail-pj2-f12.google.com with SMTP id d9443c01a7336-2d747eb79f6so3862125ad.0 for ; Tue, 15 Sep 2026 23:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541065; x=1790145865; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=nO/SIiOPBpUCTQXEHAeYL+mDwPc1g15QDptSjej1KB0=; b=MxLsaHmiL5KVxCL53JtvdKNRjY/t6kfHMpxYQk55we1Nv0AI1XOVBfcpDTPq0OvFco kELkAciPwKJr7wkY9WWxk/zy8pHahVHD00l+xONGFN0U/egYiL6P1Eo0Bn6heioGAvTN 75nZsnAIqeNIY0rZYn41RguriNaqeYS0d2sYZzfmY+S2mEOz6sXe4K/oNO90eNSSUIqp 1Q8d3cVSrynAI4xfaXLSfPOGVhtyNd3ozKhCE1V/OL9AMl6yxl5YTEOwuVY12R/P1j4L //F7erOyWMdhiKCK8N7d22EtBUWoV4MU0CXrFacNssS6F5n5hwrJTG0zy9ZuVNHlU+fY Jmhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541065; x=1790145865; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=nO/SIiOPBpUCTQXEHAeYL+mDwPc1g15QDptSjej1KB0=; b=ww/qzrnaVauenU/p+DXlOuzOBb1LFU1j2hvsW9JOr5X5JmdzfbeNntqa1oPwRM2+3n IY2GcpEpujiNX3CMvSrrUc0qE515yaDhmou4OmNrOxdDKCUWF+Drhg6SlYsQALrvduIy vvRSwjkhzwFc4AgoZod+m4nz0em2fS7knhmBLoF/VU6W8zeJ7vie/gz5yzWmugdMj/F4 /bfOv4qIvk58xucOeYxAEmUgyUcX8UA69omhSXTjy84/ZAxeMbpgQfG7APNQw18Ci5ff TKynI6r2xVdgRvSHf7yk+78QWxqSqCfq/s1qRFd28TvkkuqWQakDpgmXjcNfUE9BLyMj imPg== X-Forwarded-Encrypted: i=1; AKwUvBwMtJq5iimNWe0b2qLbH8pCXpLdFaGXeUiI3Mm+6qPsVEepTpjbRyN7UUVnT8ATpXWZzVqXKc3/Le1YR50=@vger.kernel.org X-Gm-Message-State: AFuF++kd/GAgtOsfdW4s5vB4QlncuXmKVkW9G7M285X1/HYRgWxjD3Q4 b+gsiM97cYOe6vCNtJyVCzGhCmIEd6p1LqP+hg1wkLrinN7qxsElk7YbWJRkiMadQOOgYzpDy5o PJV/N X-Gm-Gg: AYBFou0+KrlAGbER2oYf+O96xp7VYZFH9aBo44ochT2uBaZmNhIpLX9DHJL28j8w57r vHhLJ2PfDaWtz8aXU3JN2CDJf/9kg0VCnVpc6f0E8Tbri15FQwaxytWk0SN5uLsaQQ5bBcjsbuy GroGRzmjp0laIaVPzQcK4IzeVJgTj3aDIq6QJuCyLzOXZslC9degj17nKZEP8d7gR+sT1qc0Ez1 0rNNzCZ/oko4UB42w8RbeCnBOjkRhyr7UFGO+1WAbwuDqJNtOXAdTOm3YTUF4864EPiuhlioNN4 3BovIV7VXQgia0ZkD6uPyvkKXKj+1msL4dveMJ3sR1lFnsBDPdKHmThRh0t4wI9o6rRJTPRDaiD qHAXOceAUswSJuPVvp6yf+geOKcehRJ7NWiN11cDA8OCs5rdwCNVq0h7vtVvoeEiOI4/kY02evw vmdDmCym5X06TMrfTX+lJK2DggGyJJ9is+4tD7Xy/6pDiQpHvvfRR9ZWE7qCijFqR2AR0ieD3wQ XtQONVKXHAty7mkc48= X-Received: by 2002:a17:902:ce92:b0:2db:5f8c:b81c with SMTP id d9443c01a7336-2dd8e52ed65mr27165495ad.19.1789541064774; Tue, 15 Sep 2026 23:44:24 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:24 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 07/11] mm/sparse-vmemmap: move vmemmap optimization helpers to a public header Date: Wed, 16 Sep 2026 14:43:37 +0800 Message-ID: <20260916064341.1825793-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The vmemmap optimization helpers currently live in mm/sparse.h, which is an internal MM header. That works for MM code, but prevents powerpc from using the same interfaces without including a private header. Move the declarations and inline helpers to include/linux/vmemmap-optimization.h. This is a preparatory change for powerpc, which has its own vmemmap optimization implementation and needs to use the common vmemmap optimization interfaces from architecture code. Signed-off-by: Muchun Song Acked-by: Qi Zheng Acked-by: Mike Rapoport (Microsoft) --- v4: - Use the renamed VMEMMAP_OPTIMIZATION config in the public header - Collect Acked-by from Mike Rapoport v3: - Update the subject and commit message to describe common vmemmap optimization helpers - Collect Acked-by from Qi Zheng v2: - Fix missing header dependencies. --- MAINTAINERS | 1 + include/linux/vmemmap-optimization.h | 94 ++++++++++++++++++++++++++++ mm/hugetlb.c | 2 +- mm/hugetlb_vmemmap.c | 2 +- mm/sparse.h | 76 +--------------------- 5 files changed, 98 insertions(+), 77 deletions(-) create mode 100644 include/linux/vmemmap-optimization.h diff --git a/MAINTAINERS b/MAINTAINERS index 3ef018673b5e..e4412c3d8d45 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12096,6 +12096,7 @@ F: Documentation/mm/hugetlbfs_reserv.rst F: Documentation/mm/vmemmap_dedup.rst F: fs/hugetlbfs/ F: include/linux/hugetlb.h +F: include/linux/vmemmap-optimization.h F: include/trace/events/hugetlbfs.h F: mm/hugetlb.c F: mm/hugetlb_cgroup.c diff --git a/include/linux/vmemmap-optimization.h b/include/linux/vmemmap-o= ptimization.h new file mode 100644 index 000000000000..0f9da22bd8d0 --- /dev/null +++ b/include/linux/vmemmap-optimization.h @@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * vmemmap-optimization.h + * + * Generic vmemmap optimization declarations. + * + * Author: Muchun Song + */ +#ifndef _LINUX_VMEMMAP_OPTIMIZATION_H +#define _LINUX_VMEMMAP_OPTIMIZATION_H + +#include +#include +#include +#include + +#ifdef CONFIG_VMEMMAP_OPTIMIZATION +static inline unsigned int section_compound_order(const struct mem_section= *section) +{ + return section->compound_page_order; +} + +static inline void section_set_compound_order(struct mem_section *section, + unsigned int order) +{ + VM_WARN_ON(section_compound_order(section) && order && + section_compound_order(section) !=3D order); + section->compound_page_order =3D order; +} + +static inline void section_set_compound_order_range(unsigned long pfn, + unsigned long nr_pages, unsigned int order) +{ + unsigned long section_nr =3D pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_compound_order(__nr_to_section(section_nr + i), order); +} + +static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) +{ + return section_compound_order(__pfn_to_section(pfn)); +} +#else +static inline unsigned int section_compound_order(const struct mem_section= *section) +{ + return 0; +} + +static inline void section_set_compound_order(struct mem_section *section, + unsigned int order) +{ +} + +static inline void section_set_compound_order_range(unsigned long pfn, + unsigned long nr_pages, unsigned int order) +{ +} + +static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) +{ + return 0; +} +#endif /* CONFIG_VMEMMAP_OPTIMIZATION */ + +static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +{ + const unsigned int order =3D pfn_to_section_compound_order(pfn); + const unsigned long nr_pages =3D 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + +static inline bool vmemmap_optimizable_order(unsigned int order) +{ + if (!IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION)) + return false; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; +} + +#ifdef CONFIG_SPARSEMEM_VMEMMAP +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); +#endif /* CONFIG_SPARSEMEM_VMEMMAP */ +#endif /* _LINUX_VMEMMAP_OPTIMIZATION_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fd00141b089a..2003439ea13c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -38,6 +38,7 @@ #include #include #include +#include =20 #include #include @@ -52,7 +53,6 @@ #include "hugetlb_cma.h" #include "hugetlb_internal.h" #include "mm_init.h" -#include "sparse.h" #include =20 #define HUGE_BOOTMEM_ZONES_VALID BIT(0) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 5ddf06b83c96..a27f46fffb54 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -15,10 +15,10 @@ #include #include #include +#include =20 #include #include "hugetlb_vmemmap.h" -#include "sparse.h" #include "internal.h" =20 /** diff --git a/mm/sparse.h b/mm/sparse.h index 8f6ef0b50b4f..a5111087ee3a 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -9,80 +9,7 @@ #define __MM_SPARSE_H =20 #include - -#ifdef CONFIG_VMEMMAP_OPTIMIZATION -static inline unsigned int section_compound_order(const struct mem_section= *section) -{ - return section->compound_page_order; -} - -static inline void section_set_compound_order(struct mem_section *section, - unsigned int order) -{ - VM_WARN_ON(section_compound_order(section) && order && - section_compound_order(section) !=3D order); - section->compound_page_order =3D order; -} - -static inline void section_set_compound_order_range(unsigned long pfn, - unsigned long nr_pages, unsigned int order) -{ - unsigned long section_nr =3D pfn_to_section_nr(pfn); - - if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) - return; - - for (unsigned long i =3D 0; i < nr_pages / PAGES_PER_SECTION; i++) - section_set_compound_order(__nr_to_section(section_nr + i), order); -} - -static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) -{ - return section_compound_order(__pfn_to_section(pfn)); -} -#else -static inline unsigned int section_compound_order(const struct mem_section= *section) -{ - return 0; -} - -static inline void section_set_compound_order(struct mem_section *section, - unsigned int order) -{ -} - -static inline void section_set_compound_order_range(unsigned long pfn, - unsigned long nr_pages, unsigned int order) -{ -} - -static inline unsigned int pfn_to_section_compound_order(unsigned long pfn) -{ - return 0; -} -#endif - -static inline bool vmemmap_optimizable_pfn(unsigned long pfn) -{ - const unsigned int order =3D pfn_to_section_compound_order(pfn); - const unsigned long nr_pages =3D 1UL << order; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return (pfn & (nr_pages - 1)) >=3D VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; -} - -static inline bool vmemmap_optimizable_order(unsigned int order) -{ - if (!IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION)) - return false; - - if (!is_power_of_2(sizeof(struct page))) - return false; - - return order >=3D VMEMMAP_OPTIMIZATION_MIN_ORDER; -} +#include =20 /* * mm/sparse.c @@ -142,7 +69,6 @@ static inline void sparse_sections_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP -struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zon= e); void sparse_init_subsection_map(void); int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6C193EC827 for ; Wed, 16 Sep 2026 06:44:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541072; cv=none; b=XRXJzsK29gGgvdH17bpirXgK/P4fsMu2llG9thS9xqb7W67zINNLM+dyu9sy7oWVlLcGHfXmB6fu6NPcemkQcT4/5FkEx0hu3wL/72AHF3fFzLR7uDaLRvLUgcQfGiSfMklJIO50OSK8JxEPHCLfzna/tQdGMDyhlka/XDGSXcU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541072; c=relaxed/simple; bh=XXJonAu5v5lmenvG0H3fDY0XU6EMxiTGRHxJPTiZN5c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fWE3QIkfXc5/M8fd99ZUe/lR0joTGGXuMUyr4yvskjEyej666MGDQnWo6YZ7dab1Jf8mysl9iIKt4WQIiQZvidonalc8HEDCMTKI+79osPYbhzEusUfimgftGkSBErL0YAOrGxeynBtQg/RssYC+W8QmY716ptOPYtjuu34aasY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=WaoFARB5; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="WaoFARB5" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2dd88a115ebso4663595ad.2 for ; Tue, 15 Sep 2026 23:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541070; x=1790145870; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=TZbZ6WC7vZbHjIhgnid+8T29mCn5dRvITTk6eMnH6Vk=; b=WaoFARB5SK+WFgGhp7/NZvS/mlqRN2nUPm+MimiVkvBANvdftB2/MUn826bOm2txcu svkK0QBrBzjUuzPNFkH3B6NcPFJFdEmJQR3dxzsl96VEfrJuUYsFNN1F8Tw9D6xvGSdf cPMIut6SCFlhNt58zKxNkwRwEx9/6jgNDeNsE1CtfpDqCu+0z00TLRe2TtQ3Qg/oCmvZ eqHc/rCBJjpwvk/aWdfhcbn94IzszTasK+ijMIaKIc2qlqNTFwd74wKhjVUWzZ8+jSA6 GwjPGFj9TFa1ShuiJfcv05D02LRjwrzml6d8zfj7h+ZQhJYC4GOObQ5xEAfkg4q5ymIv JXew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541070; x=1790145870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=TZbZ6WC7vZbHjIhgnid+8T29mCn5dRvITTk6eMnH6Vk=; b=X4/tO0Ly82XrfeYo28YO8+n3w4FpS/ZkTwWHQib0Pou/kNevDz8cIiL0GtITy5v+A4 zqUTdzjt3RV8k06rr31bWK0lcD2xwdRXpSm5T2NZzuhuYnxG7QN+0nJL4QmsNOmBejo8 X0J05I4RZIShnCAleu3+up3VY4c8KbTeAd73PJPnsh8Wsnn7FezETnq0xRJ8QhvnVd2j y8TfK2VPFUy7NIf94Nie4pE1w6eKTPJxoWCYaqElRqsoMgggAt01IMkdVoV5RCmuEPdE 9RhcG1Syuf5PEjEBU4Jdeevc/VSAP5Hx07y3rMXPVwsYRFlnQf6BwNOdsjk78NoDehv8 U80w== X-Forwarded-Encrypted: i=1; AKwUvBxynKpm0i0cN5LCCxxXcpQxiOw5uMY7GUA+iH08pIyiHYLfGAy++VnP1ek4UJAdEeilDIehmTrgNcJjtic=@vger.kernel.org X-Gm-Message-State: AFuF++mlhgkVK/5tsQLSRoErtzI63Og9p9ogb2zbMX6TKs9fcTdkBxIW KckpzPbnCS81ebTa+qrHe0tXexA4UfOnevo4FFO8USx0fPqLGSH/FCMIFvNuKnkHr5k= X-Gm-Gg: AYBFou0vv/VdDc037tdb1JSiLzjDPWX+ax2u5TM7tY5PsjQXfGKgRjznIIDlerLlelp oe/voS14RgOrlUH4EMFYK3RAD/GXLBPXSI7d1Gq17BidCXRiMd05f8bmyGjmraOVAHic6nG4TVK bqieE2yKv5q0QbGZKTOdL6f/iBOqDkebGd/GgCqzxvVkrLQHtt7xHBxgH/+iymwhLvbHtw4SJDu Pv69kpEUwxJtArKI9YrCSxGrKOBN8bYFkVH328M1cllXRNr5WHPCDePXVSBla3asu3HTB1ZEkDT RH9yKG/Gl+PFUXFT1Re5o27lqmDj1WqPH/VNt5sWD0QEo03MBghsGRo8Ty9Ozr/cpTe11T4Qrkf AkgZ2PifdA3fPApN1X5uBXXmm5zE02hTSSIrnt9pDynMgaQV5Ga9zPMdH1qkSad/zN2aLaR01aJ 6mphQDhRVJ0RSbWh/8OPZ+HHMTB7BQ1o4WIUpOI605n9RMDItx1eL/LjLx5Txa7eWV36EPkENUt K9ZtaDZew2GHjUVMoc= X-Received: by 2002:a17:902:ce8f:b0:2da:e967:7953 with SMTP id d9443c01a7336-2dd8e404066mr28394075ad.12.1789541069935; Tue, 15 Sep 2026 23:44:29 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:29 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 08/11] powerpc/mm: switch device DAX to shared tail vmemmap pages Date: Wed, 16 Sep 2026 14:43:38 +0800 Message-ID: <20260916064341.1825793-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The powerpc radix compound vmemmap population path still finds a reusable tail page by walking the vmemmap page tables. Switch it to the common vmemmap_shared_tail_page() helper instead, so it can use the shared vmemmap page directly to simplify the code. This removes the powerpc-specific tail-page lookup and its fallback path and aligns the device DAX vmemmap optimization path with HugeTLB. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 80 +++--------------------- 1 file changed, 9 insertions(+), 71 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index cf692b2b5f7b..ee068f24a79f 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 #include #include @@ -1250,59 +1251,6 @@ static pte_t * __meminit radix__vmemmap_populate_add= ress(unsigned long addr, int return pte; } =20 -static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, - unsigned long pfn_offset, int node) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - unsigned long map_addr; - - /* the second vmemmap page which we use for duplication */ - map_addr =3D addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; - pgd =3D pgd_offset_k(map_addr); - p4d =3D p4d_offset(pgd, map_addr); - pud =3D vmemmap_pud_alloc(p4d, node, map_addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, map_addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, map_addr); - if (!pte) - return NULL; - /* - * Check if there exist a mapping to the left - */ - if (pte_none(*pte)) { - /* - * Populate the head page vmemmap page. - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL= , NULL); - if (!pte) - return NULL; - /* - * Populate the tail pages vmemmap page - */ - pte =3D radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); - if (!pte) - return NULL; - vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); - return pte; - } - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1320,6 +1268,12 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, pud_t *pud; pmd_t *pmd; pte_t *pte; + struct page *tail_page; + unsigned int order =3D pfn_to_section_compound_order(start_pfn); + + tail_page =3D vmemmap_shared_tail_page(order, device_zone(node)); + if (!tail_page) + return -ENOMEM; =20 for (addr =3D start; addr < end; addr =3D next) { =20 @@ -1349,10 +1303,9 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + PAGE_SIZE; continue; } else { - unsigned long nr_pages =3D pgmap_vmemmap_nr(pgmap); + unsigned long nr_pages =3D 1UL << order; unsigned long addr_pfn =3D page_to_pfn((struct page *)addr); unsigned long pfn_offset =3D addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); - pte_t *tail_page_pte; =20 /* * if the address is aligned to huge page size it is the @@ -1377,23 +1330,8 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, next =3D addr + 2 * PAGE_SIZE; continue; } - /* - * get the 2nd mapping details - * Also create it if that doesn't exist - */ - tail_page_pte =3D vmemmap_compound_tail_page(addr, pfn_offset, node); - if (!tail_page_pte) { - - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - if (!pte) - return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - next =3D addr + PAGE_SIZE; - continue; - } =20 - pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*ta= il_page_pte)); + pte =3D radix__vmemmap_pte_populate(pmd, addr, node, NULL, tail_page); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f33.google.com (mail-pj2-f33.google.com [74.125.227.161]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AD993ED3BB for ; Wed, 16 Sep 2026 06:44:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.161 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541077; cv=none; b=Q6remOCEM+L7kzBVU/LsEIe5DWi7xeczE47T3yZP7FZ66CflAoGJ2gnSRh1Iinx8Xy3njBH08Y/KQlQCjB5Vw17DZWzIG9+8wNOPbkE17/QNsqwZ/yWKcFh6h2z/lwWZZG2siD6skaP7/aGiohqYbT6GaHlu6NjYJMXxJAj1T3E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541077; c=relaxed/simple; bh=bkDO7GMEwrirAzyCc6E/9aYGNJDW7onGIUPfcccFqr4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JMZuUR77P0+HiA3CEJfpbRnMzIDoN34j2QQpsBDfB+inCpM+cwmkV9GDLXSfRk9URKjwSn6NNLqdVv3gjnMgo/hXZukKk/cIajE7zbrC20TAfQf/yPq6QhgOlMWr0vuLoFqsu6WcPGqEFQ80acT/2V/OgTVJBu2uDZgnNhgaCQg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=KP3teCTl; arc=none smtp.client-ip=74.125.227.161 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="KP3teCTl" Received: by mail-pj2-f33.google.com with SMTP id d9443c01a7336-2d747ed1368so6211205ad.1 for ; Tue, 15 Sep 2026 23:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541075; x=1790145875; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iVLitPRwuorHo3p/kYxhVkChlufVFy2kYjbuHRdLiho=; b=KP3teCTl82A6y9p5xByVBUbG4UaAOVHPHblHrdH8CRWC13ZN3FCQ3/XOhN+GbUB+hu omhL31ia9BfjXKfAZAxFTC09ro3mIIjL4wcpiDns425AVXD3lUVrQaDrO0v7pbr8lUlw Myk1OwPsBwd35GqH5snP/y2w1QLWobPpBL1Gyl4EXjafgDaYWttvvNHCRD2H91BLrpMO QEM7HJM0fmnp6LRAlgl8dDUon2RIQnRmKwThbV5SGkyFkPn1Q5DAcnDz4yoPdCvBE+SK /bNQg85zFUzKQE3rFUlo45+Bg8ycv1p6Jz0M91jVmaP3c/N792nA1zcJxp6CWFgFNji2 aqRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541075; x=1790145875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iVLitPRwuorHo3p/kYxhVkChlufVFy2kYjbuHRdLiho=; b=F6f2kXNCIXERruWasSb3UQ1y7VgUpIxwDxkGOTq484AWX15rdvcCTGIChafcLpiciu QDIbLcrdXd3Ywvkvb9tlkMBoGGmAxXmdZbT2UF0Qwb8G28vAK7CxpamuYB4M8v8oAUTf u1t665to1BjaE5RyDydoNjb6bfGt6KTiHeSeIRoXJtdI9MXKd+q893mU8AJp+D0k2oxJ 4tpwDODyINh7Q5L8rNY3R3H3mXYZ/NrCMvzuXtw1HaQehhPodJMu6fq/eoBZqN8Hy3UJ YcH7fENVcPH+YN9z2xSC3YRApnLw+8UCC51UkeCi7bgNJfspD5232JotEP2vX96fLc3y vE2A== X-Forwarded-Encrypted: i=1; AKwUvBx0uDLoMz7m0dTJARlxByZlG92t5eB3HIcXMH4I0SmUsAiCJX3e1PXmKf+Y7qqaomc7htf908YKEgypW3Y=@vger.kernel.org X-Gm-Message-State: AFuF++kcjYddeKJoz8Z4bSjPqtTbrFjxlVd525EGDEc5gFmMSZMydFt5 eRoIdtcPnkRGlvPUjfnCjqqDpzq4P04E/xdMnQqsJTOMRW97+s+DBANTCOMFyL3unaE= X-Gm-Gg: AYBFou2Sx7vupApk96wi0h0i1VmfkFsOWXp3DamrTxbkIdd3WTziVTpR7kqFCOqx8FA 9adUX36+XlKeBnKzVeiewNO/R3TRNoiHxnXH1SZP4sn13YJezescure1pg9lzTW3wSVzkF/Jymf HOSlN+O399tE6hzoiJQLYuVrtsqhy7nc3/XzxwrXdHOiae91LDhUEaAQ+M68eBGlOBnMml0ejIL fr4liqSx8QWBTM5xyEMcgpWZ4Jq20N4HGpNJmMwgRG/QgbjaBZKiEgAZRtBRGAXtucAELltmPh7 ddrXA1+x9VToHquTvMquROL6ws3sAdZjcD+RDoUxRDL24AgebTbhBC8v+9BqhYm0uyZwiMBPo6q vJWrZ4OQ7rqBnMFRZgPfNvQRDByBG5VvhitOj4NQqWGaKxqajQkNKZYoDBtJwWbbaJhDq409Lec zU01orKedBaXw2q8nUkMS9ZOTizMt+UUzkoKEGA9m8FCZRo3fdC5aC3PeifZ2uCgVmaVhpkCfqC OAvxOOlf8IzvsL0B9BF4gXd7Hf+Ew== X-Received: by 2002:a17:902:eac9:b0:2db:5c0a:f18c with SMTP id d9443c01a7336-2dd8e00ecdamr19914175ad.6.1789541074649; Tue, 15 Sep 2026 23:44:34 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:34 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 09/11] mm/sparse-vmemmap: drop the extra tail page from device DAX reservation Date: Wed, 16 Sep 2026 14:43:39 +0800 Message-ID: <20260916064341.1825793-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The device DAX vmemmap population still reserves one extra tail vmemmap page after the head page. Drop that extra reservation and let the shared tail page cover all tail vmemmap pages after the head page, so DAX follows the same reservation model as HugeTLB. This reduces the reserved vmemmap pages for optimized DAX mappings to one and removes the now-unneeded first-tail population from the generic and powerpc paths to simplify the code as well. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Collect Acked-by from Qi Zheng --- arch/powerpc/mm/book3s64/radix_pgtable.c | 46 ++---------------------- include/linux/mm.h | 3 +- mm/mm_init.c | 2 +- mm/sparse-vmemmap.c | 13 ++----- 4 files changed, 7 insertions(+), 57 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index ee068f24a79f..9ca28e4a610a 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1218,39 +1218,6 @@ int __meminit radix__vmemmap_populate(unsigned long = start, unsigned long end, in return 0; } =20 -static pte_t * __meminit radix__vmemmap_populate_address(unsigned long add= r, int node, - struct vmem_altmap *altmap, - struct page *reuse) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - - pgd =3D pgd_offset_k(addr); - p4d =3D p4d_offset(pgd, addr); - pud =3D vmemmap_pud_alloc(p4d, node, addr); - if (!pud) - return NULL; - pmd =3D vmemmap_pmd_alloc(pud, node, addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte =3D vmemmap_pte_alloc(pmd, node, addr); - if (!pte) - return NULL; - radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1297,7 +1264,7 @@ int __meminit vmemmap_populate_compound_pages(unsigne= d long start_pfn, if (!pte_none(*pte)) { /* * This could be because we already have a compound - * page whose VMEMMAP_RESERVE_NR pages were mapped and + * page whose retained vmemmap page was mapped and * this request fall in those pages. */ next =3D addr + PAGE_SIZE; @@ -1318,16 +1285,7 @@ int __meminit vmemmap_populate_compound_pages(unsign= ed long start_pfn, return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); =20 - /* - * Populate the tail pages vmemmap page - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte =3D radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, = NULL); - if (!pte) - return -ENOMEM; - - next =3D addr + 2 * PAGE_SIZE; + next =3D addr + PAGE_SIZE; continue; } =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index 070ce27e9cd3..66d384da4433 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5167,7 +5167,6 @@ static inline void vmem_altmap_free(struct vmem_altma= p *altmap, } #endif =20 -#define VMEMMAP_RESERVE_NR 2 #ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) @@ -5187,7 +5186,7 @@ static inline bool __vmemmap_can_optimize(struct vmem= _altmap *altmap, * For vmemmap optimization with DAX we need minimum 2 vmemmap * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); + return !altmap && (nr_vmemmap_pages > VMEMMAP_OPTIMIZATION_PAGES); } /* * If we don't have an architecture override, use the generic rule diff --git a/mm/mm_init.c b/mm/mm_init.c index efffa8609b85..56bb4567a494 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1056,7 +1056,7 @@ static inline unsigned long compound_nr_pages(unsigne= d long pfn, if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); =20 - return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); + return VMEMMAP_OPTIMIZATION_PAGES * (PAGE_SIZE / sizeof(struct page)); } =20 static void __ref memmap_init_compound(struct page *head, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 7b5b9ceec697..39f0cfeefc6a 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -136,7 +136,6 @@ int __meminit section_nr_vmemmap_pages(unsigned long pf= n, unsigned long nr_pages { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_compound_order(ms); - const int vmemmap_pages =3D pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZA= TION_PAGES; const unsigned long pages_per_compound =3D 1UL << order; =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); @@ -147,13 +146,13 @@ int __meminit section_nr_vmemmap_pages(unsigned long = pfn, unsigned long nr_pages =20 if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; + return VMEMMAP_OPTIMIZATION_PAGES * nr_pages / pages_per_compound; } =20 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); =20 if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; + return VMEMMAP_OPTIMIZATION_PAGES; =20 return 0; } @@ -521,17 +520,11 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, if (!pte) return -ENOMEM; =20 - /* Populate the tail pages vmemmap page */ - next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, -1, flags); - if (!pte) - return -ENOMEM; - /* * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - next +=3D PAGE_SIZE; + next =3D addr + PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, page_to_pfn(page), flags); if (rc) --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9176B3EC817 for ; Wed, 16 Sep 2026 06:44:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541084; cv=none; b=qJ8n4TSg78phjQc/WeXi/XC2I74+Rn2Na/edyEvO2XiwaDj0WNT39f71B9rmnfZQTWdlQUHEvbMBx6pbMkHZYt6Pb4wBhFMGQ1WFXlmEtfWWZHfvjFYky+RKXvwVKu9ycEjvRPwhDO/Wu39X1egD4yGTq32kbx8emRApdjGH3p8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541084; c=relaxed/simple; bh=o5WJM3/QFSVrnnx6cWLQ5UPoN7IuOCV+HHfC3SmFVvo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bRdrCS/JRfNL3dTDJsP0wZFcNc39cjyx6zoMh2V/qUFp2Erpe1ypj4BAWevReM83nMbLcSf5NxX2vVUT4wKU5cCZr5CGLShyJtZLgQIZQAfi0opcoeM6jxuR8svhwOs95B3XLanqeHYsObnGe1uLqBtX7AktbHrfiM0FO//FrSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BbqzPG7G; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BbqzPG7G" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2d8fe517774so7193265ad.0 for ; Tue, 15 Sep 2026 23:44:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541080; x=1790145880; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iM8j8Wu3fwHIWd8x6lR8eMq2Esw45qvi78c+WOmPzrM=; b=BbqzPG7GWtNylKccSg71VixFuJecxTQ3lf0YDea9jtfstudOs3EP5CzM468Aw2krl4 6sz/apRIEAFfPPtKqTfPv2fhrbYqR+O9I+Kx17I3GVceIZKlWD7jbs87t9f6QR+tMoI8 KgA88X4cLLcYxSLLIJ8GTZ0k/M5MtaNkOSglQ487/Nf9+VKh5se42taGsfEmiF42UUTi znlLbNxAoVCvx9xyBGgb+pAhIrShQqSUyVmeloqQll6hSeWV7JZqeGg9LEEM2Ft+Yg2F Ds38Vyj42FZpz0LkSlgdWA4cZ88kn9gHwoLTBRnVhzz+V1PY+8US6uwu1WtsuH5TV6Lg BHfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541080; x=1790145880; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iM8j8Wu3fwHIWd8x6lR8eMq2Esw45qvi78c+WOmPzrM=; b=dTuNfdT0SSwgooMwftSuFr+lz7maqm7KkdnFO+/gSUazB707wXVih/O/0tdXPG50Wi f5YMi8oYEdhvDunB1gPh/tMUGhiMCh16IO7caWPeBmae/yX7kRzkTfU8uIiA6YanAov0 CA9YwaWuicXerlcUG6es7ugnzuhahAeU2i+PuYEbTey87MmTDoX0ItGct4QSr2kTJW4S T56vucajazzW6A2EJaQpH3n++ArTv8B7Yx1aH4qCgUoViZXizceU1HoqHQDH63xEw1FG t3vc/EKPBoo7q4h8BNo8eCVS5Ybr5a4r3IiiXu9ks/r+0fPLNwBzkBj3fCE6BfqJbjz7 KPtA== X-Forwarded-Encrypted: i=1; AKwUvBwGdzqobmhdymWGGKMxX1msuUTdtNYYxoYzbcJ/TGiUbCcM/98wdsXETSYG4Kd7GgGE1M4/CKYSWvGjpb8=@vger.kernel.org X-Gm-Message-State: AFuF++lKZPX9tHQVuSD9aCPkqt5L6NydJtG2FMMZzJfN8n10w5TJor3q vcRrNYggIsa2oLESbx0MELR39zQua9FtgZ1GKRmYXIHjfvDZxm1Lq8mgC1dRHv+Zgvc= X-Gm-Gg: AYBFou0Q50Cd0ozaCGa7a2xzRZXRfvYzZRG2uFfIn4hOwxEYPYDQCiUTf46YdvUL6DI bgTT6fvq9oVjrDuI9GUQxXWsQ4/6RObl/4/NCqD/RHrZgvFaSvifZ5pkTUysTi5y2gLC455Mu2A xfHhTB0x7aoxBZuIqBL5R4Egwgwmbb8bUBBAIYFSo6x3moIPgK1axJyWjvwc0t3fSOUXlxMRnEk rRNlwvUSYoN6E4vHJImSLv4bzCV/4Gqn8tSxA54nEDdViVKE8vE3SeWCDrKI/iOmtnv8DukoWwl FF9qcJ60izPxsCfsB+A+BRaYPrOddwS/1POlbWhmYQr3HA74I27uXd21VfLgH/fMbSzpIGtTE7V jfKwxm8EwF5OxMvcQv3o0tYfkrMdh2CLv4I5LqL9X2lfq77QSVg9fsKxl/3XJlSwNtYwiVmr6F9 qsO6Fb6RJ38PBQcvpfxrW7l2RGJpS+W0Kq5/KPCALSF1nedJWNvX1jHG9xxAO3Gop9yLv2lqfds FC16DLdiIZiDt9ju0HOQUho9emlDg== X-Received: by 2002:a17:902:c94a:b0:2d8:d4cc:be68 with SMTP id d9443c01a7336-2dd8e7b5ca2mr29021055ad.21.1789541079708; Tue, 15 Sep 2026 23:44:39 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:39 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 10/11] mm/sparse-vmemmap: drop unused section_nr_vmemmap_pages() arguments Date: Wed, 16 Sep 2026 14:43:40 +0800 Message-ID: <20260916064341.1825793-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" section_nr_vmemmap_pages() no longer uses the altmap or pgmap arguments, so drop them from the helper and its callers. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Collect Acked-by from Qi Zheng --- mm/sparse-vmemmap.c | 10 ++++------ mm/sparse.c | 3 +-- mm/sparse.h | 6 ++---- 3 files changed, 7 insertions(+), 12 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 39f0cfeefc6a..96506f594924 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -131,8 +131,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } =20 -int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr= _pages) { const struct mem_section *ms =3D __pfn_to_section(pfn); const int order =3D section_compound_order(ms); @@ -637,7 +636,7 @@ static struct page * __meminit populate_section_memmap(= unsigned long pfn, struct page *page =3D __populate_section_memmap(pfn, nr_pages, nid, altma= p, pgmap); =20 - memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages)); =20 return page; } @@ -648,7 +647,7 @@ static void depopulate_section_memmap(unsigned long pfn= , unsigned long nr_pages, unsigned long start =3D (unsigned long) pfn_to_page(pfn); unsigned long end =3D start + nr_pages * sizeof(struct page); =20 - memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages)); vmemmap_free(start, end, altmap); } =20 @@ -658,8 +657,7 @@ static void free_map_bootmem(struct page *memmap) unsigned long end =3D (unsigned long)(memmap + PAGES_PER_SECTION); unsigned long pfn =3D page_to_pfn(memmap); =20 - memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); vmemmap_free(start, end, NULL); } =20 diff --git a/mm/sparse.c b/mm/sparse.c index cc28bb41fdb1..b75921c622ed 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -250,8 +250,7 @@ static void __init sparse_init_nid(int nid, unsigned lo= ng pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, SECTION_IS_EARLY); usage =3D (void *)usage + mem_section_usage_size(); diff --git a/mm/sparse.h b/mm/sparse.h index a5111087ee3a..530692cdd516 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -70,12 +70,10 @@ static inline void sparse_sections_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); -int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap); +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages); #else static inline void sparse_init_subsection_map(void) {} -static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned lon= g nr_pages) { return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); } --=20 2.54.0 From nobody Fri Sep 25 06:03:56 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B57A03EDAA8 for ; Wed, 16 Sep 2026 06:44:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541086; cv=none; b=XHF2oAR1C/S+AIk9eO+MLlNFU2CGPwKV2LoQvxZfWRxTI+YuIdpZMN3fgRaOxEzGgdf9GwpHA5/li/0OwjdCif5/36Tf1AEtwLz4t0P0SHICFIhiPGCqLVPbj45ZzQfKEzb+b1Us0jp+7eJO4B3kOG6wZzDQCscOAe9u8AuyJ30= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789541086; c=relaxed/simple; bh=SzfK6CJnam9Dg4BVTpagUZL4n3gDBm63Y+qtGXTWOqU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T7ckDSUszB30VmyN/B7BF6rGCQHTPDcpenDIs5RHkH17hgdk5xwO30AymN9xupkHMiZYh8Csdwvn5wF64w0ZrsbOeupS0Q8CVXq3MOCliey3lB4B+nr05QErmNDAHg0/bkrlYoPxB+wAlxpdxyPZt07903u5PlytnSvioYB3fSU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=COkqCTcV; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="COkqCTcV" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2dd4b43b20bso3753905ad.1 for ; Tue, 15 Sep 2026 23:44:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1789541084; x=1790145884; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=zUhtpwjDYsFhZVHLtQ8dC1wt8tqWIMzZ7omgfR51vXw=; b=COkqCTcVOE3gTa2CetFt3I6Y1J8mj8yCNrj942/DylYRwBEB1pMhn+tL9Ebs5/O+NB XIgnZeM8u/TwFo5TMLqW+G4Tn9Mj2YQdn/rZ2HVaZPOZqSMKy1ZUCdCnZFqPDeRrskxH G57muZ6E9WM0lQUvf2/oNwQKPWQKN7MlW9XINf8nTSN4tJo/3dOFNsOgdbxhTns8/spw wgqwTc0iD85ETBkg3P4T2vfRWo2TYf9h/Kf/KRT+X7XqJsa+52K0ZfRDDX+qUcUb2IZD wrX1cil1nWHajwEVLL5dvFml8OUwSN3P0vAluKqA4/+n+qfR+GhZn6W0QI7U+bqPf8Nv vH1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789541084; x=1790145884; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=zUhtpwjDYsFhZVHLtQ8dC1wt8tqWIMzZ7omgfR51vXw=; b=eudZ9bwIIeMBGiL87MgY6X2emBgGZCd6mh8F5ym92pSQzyCPbofgNw9HVYQYSFOIWi ni4sSNEPGpKJZmdX9ey8Z35taDL6RC0kYIPwQyLU2i9OzN7BY86Ux2vwkP+Bueg1qzs4 0r48BV0i9pUAHNgLrvN0RF2IkzOVpS61SNoiWL2T7v1z3G+lnSrh74p0le+tqSbY5gWV Xw45Zlr4DQ2x5bOqHkqEkHVWq1jh08rkCqMiZmtOjdb8BUwCLOGQq/Dzp+AkdFG1tdWG g5EX1FNpghnOONtOfAoAjGNr5psmr7QJidW2wkdnrpqd+/sNniyXi2ez6HEJBfvlNOjc ooaA== X-Forwarded-Encrypted: i=1; AKwUvBz5XA6r/nwLB8ke1WtMh7tnBrjYmSoMyEaiD0MUfH9MBxMJAdqIlssmOUmKEjzFhtS472KSqRFdF2B4dFw=@vger.kernel.org X-Gm-Message-State: AFuF++lGyjPMEjndN+pfzkRBxqBoMWMGQjXWwwRrurkNWHh5czfxO5fO 3WV2NzSStPcTKUKwPUQBGpLpw1kp9+KVBLkQqmEg3OV6QztTH42Em8Sq8GbPo3KwI3s= X-Gm-Gg: AYBFou0+p5gVCO6QJRTnKDas2S4tASgUCcrigSvDbZCiVOhTIm3svVvXwbdtFfOxYzB v3m6HdaXAjR3akTNKOn3Y6MrHKKi77DdTjzaXeS0uegbI+0CZsI7B6dVM7rd4NsyCxIDwzR/nMh 8xl4oLGq0Wz6DJ0tMQcpghrdPqJkrhmnJFUPUPDaaHDGNQFv50261r/sZ1fOn/LofanzKprTDTg OT91QNpR9kqbROizWT2hTGalM3Y7IwMpMF6VeLsfmxqjJq9BZ3KWlxK3XtlfBMNICgOWuV8vOhJ H9g77nJt8PdxlOX1GFcsNBsK9farhDW8Whq+UOYqjcBiU1HCmXha6r5Vyp3xfn7DUtKvFfLZevF 9L8Um0BVegMQRoyaNosIkgwhBfmskj0JkrSx0vNEQGe8V7sb/pMcxcQfznhquvVFwvaSX/IJ/dY DunGz0yU0s0plnl5sgfW33F9vza417s2AY3WYzMFZhT0ErYMF/RBwNRE5UhG2KBxdhlXdIaXR5V /L+6S75lA5KJHYsqPQ= X-Received: by 2002:a17:902:f547:b0:2d8:df44:a5c2 with SMTP id d9443c01a7336-2dd8e766fd2mr29327755ad.20.1789541084051; Tue, 15 Sep 2026 23:44:44 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.98]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2dd89f0592asm6074755ad.62.2026.09.15.23.44.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 23:44:43 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Oscar Salvador , Madhavan Srinivasan , Michael Ellerman , Jonathan Corbet Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-doc@vger.kernel.org, Muchun Song , Lorenzo Stoakes , Mike Rapoport , Qi Zheng , Nicholas Piggin , Christophe Leroy , Randy Dunlap , Muchun Song Subject: [PATCH v4 11/11] Documentation/mm: update DAX vmemmap deduplication docs Date: Wed, 16 Sep 2026 14:43:41 +0800 Message-ID: <20260916064341.1825793-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260916064341.1825793-1-songmuchun@bytedance.com> References: <20260916064341.1825793-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Device DAX now uses the common per-zone shared tail page for vmemmap deduplication. The old documentation still described a DAX-specific layout with a separately populated tail vmemmap page and half the HugeTLB savings. Update the generic and powerpc documentation to describe the shared layout. In the powerpc document, keep the radix and 64K-specific details, drop the duplicated 4K PUD arithmetic, and replace the repeated device-dax diagrams with a single parameterized PMD/PUD diagram. Signed-off-by: Muchun Song Acked-by: Qi Zheng --- v3: - Collect Acked-by from Qi Zheng v2: - Clarify the commit message to state that the 4K PUD arithmetic is intentionally dropped reported by Sashiko. --- Documentation/arch/powerpc/vmemmap_dedup.rst | 90 ++++---------------- Documentation/mm/vmemmap_dedup.rst | 32 +------ 2 files changed, 21 insertions(+), 101 deletions(-) diff --git a/Documentation/arch/powerpc/vmemmap_dedup.rst b/Documentation/a= rch/powerpc/vmemmap_dedup.rst index dc4db59fdf87..8286acbca9bc 100644 --- a/Documentation/arch/powerpc/vmemmap_dedup.rst +++ b/Documentation/arch/powerpc/vmemmap_dedup.rst @@ -19,82 +19,28 @@ With 1G PUD level mapping, we require 16384 struct page= s and a single 64K vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Henc= e we require 16 64K pages in vmemmap to map the struct page for 1G PUD level ma= pping. =20 -Here's how things look like on device-dax after the sections are populated= :: - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 15 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - - With 4K page size, 2M PMD level mapping requires 512 struct pages and a si= ngle 4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapp= ing. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ - -With 1G PUD level mapping, we require 262144 struct pages and a single 4K -vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we -require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level -mapping. - -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PUD | +-----------+ | |= | - | level | | . | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | . | ------------------------+= | - | | +-----------+ = | - | | | 4095 | -------------------------= -+ - | | +-----------+ +Here's how things look on device-dax after vmemmap-optimized sections are +populated. ``N`` is the number of vmemmap pages required by the DAX mapping +above:: + + Device DAX vmemmap pages (N pages) backing page= frames + +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= ----+ + | | | 0 | -------------> | 0 = | + | | +-----------+ +---------= ----+ + | | | 1 | ------+ + | | +-----------+ | + | | | 2 | ------+ + | | +-----------+ | + | | | . | ------+ +---------= ----+ + | PMD/PUD | +-----------+ | | A single= , | + | level | | . | ------+------> | per-zone= | + | mapping | +-----------+ | | shared t= ail | + | | | N - 1 | ------+ | page = | + | | +-----------+ +---------= ----+ | | | | | | diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_= dedup.rst index 9fa8642ded48..8c287ae3f86c 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -1,4 +1,3 @@ - .. SPDX-License-Identifier: GPL-2.0 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -192,32 +191,7 @@ to 4 on HugeTLB pages. =20 There's no remapping of vmemmap given that device-dax memory is not part of System RAM ranges initialized at boot. Thus the tail page deduplication -happens at a later stage when we populate the sections. HugeTLB reuses the -the head vmemmap page representing, whereas device-dax reuses the tail -vmemmap page. This results in only half of the savings compared to HugeTLB. - -Deduplicated tail pages are not mapped read-only. +happens at a later stage when we populate the sections. =20 -Here's how things look like on device-dax after the sections are populated= :: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +---------= --+ - | | | 0 | -------------> | 0 = | - | | +-----------+ +---------= --+ - | | | 1 | -------------> | 1 = | - | | +-----------+ +---------= --+ - | | | 2 | ----------------^ ^ ^ ^ ^= ^ - | | +-----------+ | | | |= | - | | | 3 | ------------------+ | | |= | - | | +-----------+ | | |= | - | | | 4 | --------------------+ | |= | - | PMD | +-----------+ | |= | - | level | | 5 | ----------------------+ |= | - | mapping | +-----------+ |= | - | | | 6 | ------------------------+= | - | | +-----------+ = | - | | | 7 | -------------------------= -+ - | | +-----------+ - | | - | | - | | - +-----------+ +Deduplicated tail pages are not mapped read-only. The mapping layout is th= e same +as HugeTLB. --=20 2.54.0