memmap_init_zone_device() can take a noticeable amount of time when large
pmem namespaces are bound or rebound, because it initializes nearly
identical struct page descriptors one PFN at a time. This series reduces
that ZONE_DEVICE memmap initialization overhead by reusing prepared
struct page templates and, on x86, using memcpy_nontemporal() for the
template copy path.

The main target is large fsdax/devdax pmem configurations, where the
cost of initializing the memmap shows up directly in nd_pmem/dax_pmem
bind and rebind latency.

Patches 1-3 are preparatory cleanups and helper extraction. Patches 4-5
add the template-copy path for head pages and compound tails. Patches
6-8 introduce memcpy_nontemporal(), extend the x86 fixed-size
memcpy_flushcache() inline cases used by that helper, and switch the
template-copy path over to memcpy_nontemporal(). Patch 9 removes the
remaining local opt-out predicate and always uses the template path after
the first real page seeds the reusable template.

Architectures without a specialized memcpy_nontemporal() backend fall
back to memcpy(), so the template-copy optimization remains available
without arch-specific support. On x86, memcpy_nontemporal() maps to the
existing memcpy_flushcache() backend and can use the fixed-size MOVNTI
paths added by this series for struct page sized copies.

This version also incorporates a round of review comments from Sashiko
on patches 6-8.

For patch 6, the generic memcpy_nontemporal() fallback is a macro alias
to memcpy(), rather than an inline wrapper, so architectures without a
specialized backend keep the usual memcpy() FORTIFY coverage when object
sizes remain visible at the original call site.

For patch 7, I re-checked whether the new 32/48/64/80/96-byte MOVNTI
cases should add a separate alignment check before staying on the inline
fast path. I re-checked the Intel SDM together with the existing
memcpy_flushcache() behavior, and I did not find an architectural
statement that would make those new cases require a different
correctness rule from the long-standing 4/8/16-byte fast paths. This
series therefore keeps the new fixed-size cases inline without adding a
separate alignment check.

For patch 8, I re-checked whether the template-copy path needs a
helper-level drain after the MOVNTI-based copies. This series keeps
memcpy_nontemporal() as a copy primitive only. Callers that use it for a
producer-consumer or device-visible handoff must provide the required
ordering. The ZONE_DEVICE template-copy path uses it only while
initializing struct page metadata, so the copy primitive itself does not
grow a separate drain contract.

The numbers below measure the time spent in memmap_init_zone_device()
during driver bind/rebind. They are not measurements of the full
nd_pmem or dax_pmem bind/rebind operation.

Tested in a VM with a 100 GB fsdax namespace device configured with
map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
server.

Test procedure:
Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
initialization time from the pr_debug() output of
memmap_init_zone_device().

Base(v7.2-rc1):
  First binding for nd_pmem driver: 1456 ms
  Average of subsequent rebinds: 244.28 ms

  First binding for dax_pmem driver: 1462 ms
  Average of subsequent rebinds: 273.31 ms

With this series applied:
  First binding for nd_pmem driver: 1272 ms
  Average of subsequent rebinds: 96.79 ms

  First binding for dax_pmem driver: 1354 ms
  Average of subsequent rebinds: 119.04 ms

This reduces the average memmap initialization time measured during
rebind by about 60.4% for nd_pmem and 56.4% for dax_pmem.

As an additional data point, I also ran a smaller set of measurements on
the same physical x86_64 host with a 100 GB PMEM region created via the
memmap= kernel command line, configured as fsdax and devdax namespaces
with map=dev and 2 MiB alignment.

For brevity, the individual patches keep only the VM results rather than
including a second set of physical-host measurements throughout the
series. The physical-host numbers below are included only as
supplemental evidence that the same optimization also provides a similar
benefit on a non-virtualized system.

Test procedure:
Reconfigure the namespace mode, rebind the nd_pmem or dax_pmem driver
once, and collect the memmap initialization time from the pr_debug()
output of memmap_init_zone_device().

Base (v7.2-rc1):
  nd_pmem / fsdax: 179 ms
  dax_pmem / devdax: 264 ms

With this series applied:
  nd_pmem / fsdax: 82 ms
  dax_pmem / devdax: 113 ms

This reduces the measured memmap initialization time during rebind by
about 54.2% for nd_pmem and 57.2% for dax_pmem on that setup, which is
broadly consistent with the VM results above.

As another supplemental data point, I also measured the test_hmm.ko
module on the same physical x86_64 host, using the test_hmm.ko setup
from the previous discussion that times ten 64 GB
memremap_pages()/memunmap_pages() iterations during module insertion[1].
By default, module insertion initializes two DEVICE_PRIVATE dmirror
devices, so two avg memremap values are reported; each value is the
average for one 64 GB chunk.

This is not the primary target workload of the series, but it exercises
the same large ZONE_DEVICE memmap initialization path and shows the same
direction of improvement.

Base (v7.2-rc1):
  avg memremap reported during module insertion: 116689362 ns, 116539263 ns

With this series applied:
  avg memremap reported during module insertion: 54607108 ns, 54458236 ns

This corresponds to about a 53.2% reduction based on the mean of the
reported values, which is again consistent with the pmem bind/rebind
results above.

[1] https://lore.kernel.org/all/aiEoByaQdRR3xtM5@nvdebian.thelocal/

Li Zhe (9):
  mm: fix stale ZONE_DEVICE refcount comment
  mm: factor zone-device page init helpers out of
    __init_zone_device_page
  mm: add a set_page_section_from_pfn() helper
  mm: add a template-based fast path for zone-device page init
  mm: extend the template fast path to zone-device compound tails
  string: introduce memcpy_nontemporal()
  x86/string: extend memcpy_flushcache() fixed-size fastpaths
  mm: use memcpy_nontemporal() in zone-device template copies
  mm: always use the zone-device template init path

 arch/x86/include/asm/string_64.h |  68 +++++++++++++++-
 include/linux/mm.h               |  15 +++-
 include/linux/string.h           |  12 +++
 mm/mm_init.c                     | 131 +++++++++++++++++++++++++------
 4 files changed, 198 insertions(+), 28 deletions(-)

---
v6: https://lore.kernel.org/all/20260709112520.24857-1-lizhe.67@bytedance.com/
v5: https://lore.kernel.org/all/20260701090553.62691-1-lizhe.67@bytedance.com/
v4: https://lore.kernel.org/all/20260603080152.64728-1-lizhe.67@bytedance.com/
v3: https://lore.kernel.org/all/20260527033636.28231-1-lizhe.67@bytedance.com/
v2: https://lore.kernel.org/all/20260521040124.10608-1-lizhe.67@bytedance.com/
v1: https://lore.kernel.org/all/20260515082045.63029-1-lizhe.67@bytedance.com/

Changelogs:

v6->v7:
- Rename memcpy_nt() to memcpy_nontemporal() so the generic helper name
  is explicit rather than abbreviated. Suggested by Muchun Song.
- Drop the separate memcpy_nt_drain() helper and remove the drain calls
  from the ZONE_DEVICE template-copy path. Suggested by Muchun Song.
- Rework patch 6 so the generic memcpy_nontemporal() fallback uses
  '#define memcpy_nontemporal memcpy', preserving the usual memcpy()
  FORTIFY coverage when object sizes remain visible at the original call
  site.
- Summarize the off-list Sashiko review comments relayed by Andrew on
  the memcpy helper and x86 memcpy_flushcache() pieces.
- Add patch 9 to remove the local template-enable predicate and the
  remaining non-template fallback path, as suggested by Muchun. This
  also means KASAN/KMSAN builds no longer force the per-page
  initialization path, and page_ref_set no longer observes every
  initialization-time refcount assignment.
- Re-ran the VM performance test for v7. The results were close to the
  previously reported v5 numbers, so keep the existing performance data
  unchanged.
- Rename pagemap_resets_refcount() to pagemap_requires_refcount_reset()
  and invert the predicate accordingly. Suggested by Balbir Singh.

For changelogs of earlier revisions, please refer to the v6 cover letter.

-- 
2.20.1

On Mon, 20 Jul 2026 20:02:49 +0800 "Li Zhe" <lizhe.67@bytedance.com> wrote:

> memmap_init_zone_device() can take a noticeable amount of time when large
> pmem namespaces are bound or rebound, because it initializes nearly
> identical struct page descriptors one PFN at a time. This series reduces
> that ZONE_DEVICE memmap initialization overhead by reusing prepared
> struct page templates and, on x86, using memcpy_nontemporal() for the
> template copy path.

What Dave said - more details on the real-world userspace benefit would
be helpful, thanks.

AI review might have found some things, one "critical":
	https://sashiko.dev/#/patchset/20260720120259.1545-1-lizhe.67@bytedance.com

On Mon, 20 Jul 2026 15:43:34 -0700, akpm@linux-foundation.org wrote:

> AI review might have found some things, one "critical":
> 	https://sashiko.dev/#/patchset/20260720120259.1545-1-lizhe.67@bytedance.com
> 
> --- Patch [6]: [PATCH v7 6/9] string: introduce memcpy_nontemporal() ---
> commit 48cf815dd31950a95bb4752e1e8e62594a090364
> Author: Li Zhe <lizhe.67@bytedance.com>
> string: introduce memcpy_nontemporal()
> Introduce memcpy_nontemporal() for write-once copy sites that want a
> named non-temporal copy primitive.
> > diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
> > index 4635616863f53..21ae515ae35a3 100644
> > --- a/arch/x86/include/asm/string_64.h
> > +++ b/arch/x86/include/asm/string_64.h
> > @@ -100,6 +100,18 @@ static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t
> >  	}
> >  	__memcpy_flushcache(dst, src, cnt);
> >  }
> > +
> > +#define memcpy_nontemporal memcpy_nontemporal
> > +/*
> > + * Reuse the existing x86 flushcache backend as the non-temporal copy
> > + * primitive.
> > + */
> > +static __always_inline void memcpy_nontemporal(void *dst, const void *src,
> > +		size_t cnt)
> > +{
> > +	memcpy_flushcache(dst, src, cnt);
> > +}
> Does this missing fencing introduce a memory ordering hazard on x86_64?
> Because memcpy_flushcache() uses weakly-ordered movnti instructions, it
> bypasses the x86 Total Store Order (TSO) guarantees. If generic code
> follows memcpy_nontemporal() with smp_store_release() or smp_wmb(), it
> could result in data corruption since those standard SMP barriers only
> compile to compiler barriers on x86_64.
> Is there a plan to add a built-in sfence here, or update the
> documentation to warn callers that wmb() is strictly required?

On x86 it intentionally follows the existing memcpy_flushcache() style:
it performs the non-temporal copy, but it does not imply a drain or a
publication barrier.

Callers that use memcpy_nontemporal() as part of a producer-consumer or
device-visible handoff must add the required ordering at the handoff
point. On x86, that means a real write barrier such as wmb() or pmem_wmb();
smp_wmb() and smp_store_release() are not sufficient for ordering
non-temporal stores before such a handoff.

This patchset does not need such a drain in the ZONE_DEVICE template-copy
path. The copy is followed by more local struct page initialization on
the same CPU, not by a publication store to another CPU or device.

> Will this bypass KASAN memory safety instrumentation and FORTIFY_SOURCE
> bounds checking on x86_64?
> The inline wrapper doesn't use __FORTIFY_INLINE, and calling
> memcpy_flushcache() directly skips the usual kasan_check_write() and
> kasan_check_read() protections that the generic memcpy macro preserves.

On x86, memcpy_nontemporal() is only a thin wrapper around the existing
memcpy_flushcache() backend. I would prefer to keep the wrapper at that
level and not add a separate FORTIFY_SOURCE layer here, since the backend
it wraps does not provide one either.

For this series, bypassing KASAN/KMSAN instrumentation in the final
ZONE_DEVICE template-copy path is expected and is called out in the final
cleanup patch.

> 
> > diff --git a/include/linux/string.h b/include/linux/string.h
> > index 5702daca4326b..9815c08867d72 100644
> > --- a/include/linux/string.h
> > +++ b/include/linux/string.h
> [ ... ]
> > @@ -278,6 +278,18 @@ static inline void memcpy_flushcache(void *dst, const void *src, size_t cnt)
> >  }
> >  #endif
> >  
> > +#ifndef memcpy_nontemporal
> > +/*
> > + * memcpy_nontemporal() requests a non-temporal copy when the
> > + * architecture has a suitable backend. Architectures without a
> > + * specialized backend fall back to memcpy(). Keep this as a macro alias
> > + * so the compiler can still see the original memcpy() call site and
> > + * preserve the usual FORTIFY coverage when object sizes remain visible
> > + * there.
> > + */
> > +#define memcpy_nontemporal memcpy
> > +#endif
> Could this macro definition cause build failures due to a return type
> mismatch across architectures?
> The generic fallback here aliases to memcpy, which implicitly returns
> void *, while the x86_64 backend explicitly returns void. If a caller
> attempts to use the return value on x86_64, the build will fail with a
> void value error, whereas it will compile cleanly on fallback
> architectures.
> Should this macro cast the result to void to enforce a uniform API?
> For example: #define memcpy_nontemporal(d, s, l) ((void)memcpy(d, s, l))

Agreed. The return value should not be part of the memcpy_nontemporal()
API. I will make the generic fallback a void function-like macro. This
keeps the API consistent with memcpy_flushcache(), which also returns
void, and makes accidental use of a return value fail consistently
across architectures.

> --- Patch [7]: [PATCH v7 7/9] x86/string: extend memcpy_flushcache() fixed-size fastpaths ---
> commit 882d29aec39f30f4e6b01548ec298e08ca1ab787
> Author: Li Zhe <lizhe.67@bytedance.com>
> x86/string: extend memcpy_flushcache() fixed-size fastpaths
> This commit adds larger fixed-size fast paths (32 to 96 bytes) to
> memcpy_flushcache() using inline movnti helpers. It also explicitly removes
> the 8-byte destination alignment check for these sizes based on microbenchmark
> results showing improved performance for unaligned copies.
> > diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
> > --- a/arch/x86/include/asm/string_64.h
> > +++ b/arch/x86/include/asm/string_64.h
> [ ... ]
> > +static __always_inline void movnti_8(void *dst, const void *src)
> > +{
> > +	asm volatile("movntiq %1, %0"
> > +		     : "=m"(*(u64 *)dst)
> > +		     : "r"(*(const u64 *)src)
> Does casting an unaligned void pointer to a strictly-aligned u64 pointer and
> dereferencing it invoke undefined behavior here?
> Since this fast path handles unaligned addresses, the compiler might assume the
> memory operand is naturally 8-byte aligned based on the u64 type. Could this
> allow the compiler to incorrectly optimize away subsequent alignment checks or
> emit invalid aligned optimizations?

I do not see an 8-byte alignment requirement for MOVNTI in the Intel SDM.
This also matches the existing memcpy_flushcache() fixed-size cases: the
8-byte and 16-byte cases already use MOVNTI without a separate alignment
check.

So I do not think this series needs to add an alignment fallback for the
new 32/48/64/80/96-byte fixed-size cases. Please correct me if I am wrong.

> > +		     : "memory");
> Is this global "memory" clobber necessary given the specific "=m" output
> constraint?
> When this helper is nested to create larger copies like movnti_64(), it
> generates up to 8 consecutive full compiler memory barriers. Does this force
> the compiler to unnecessarily spill and reload locally cached variables from
> registers, preventing instruction scheduling across each 8-byte chunk?

The "memory" clobber is intentional here. The goal is to keep the
fixed-size copy as a compact sequence of MOVNTI stores and avoid the
compiler moving other memory accesses into the middle of that streaming
store sequence. The "=m" operand describes the destination memory
operand, but it does not express that broader scheduling constraint.

The largest fixed-size copy added by this patch is still small, and the
measured result for the final memcpy_flushcache() path is still favorable
for the ZONE_DEVICE struct page copies that motivated this series. So I
would prefer to keep the conservative clobber in this version unless
there is a demonstrated code generation or performance regression from
it. Please correct me if I am wrong.

> > +}
> [ ... ]
> > +static __always_inline void memcpy_flushcache(void *dst, const void *src,
> > +					      size_t cnt)
> >  {
> [ ... ]
> > +			case 32:
> > +				movnti_32(dst, src);
> > +				return;
> Does skipping the 8-byte alignment check here break the cache bypass guarantee
> required for persistent memory (PMEM) durability?
> The memcpy_flushcache() API guarantees that written data bypasses the CPU cache
> so PMEM updates are durable without requiring manual clflush. On x86, unaligned
> non-temporal stores (MOVNTI) crossing cache lines silently fall back to
> temporal Read-For-Ownership cache allocations.
> Because this inline fast path bypasses __memcpy_flushcache() (which correctly
> aligns the destination using normal memcpy and clflushopt), will the unaligned
> data silently sit in the CPU L1/L2 cache instead of reaching the PMEM module?
> Is it possible that the microbenchmark speedup mentioned in the commit message
> (0.410us vs 0.962us for unaligned offsets) was achieved precisely because the
> stores hit the fast L1 cache instead of actually flushing to memory?

I do not see anything in the Intel SDM saying that an unaligned
non-temporal MOVNTI store that crosses a cache line silently falls back
to temporal RFO cache allocations. The existing 4/8/16-byte
memcpy_flushcache() fixed-size cases already use MOVNTI without such an
alignment check. Please correct me if I am wrong.

The microbenchmark is only used as the performance motivation for keeping
these small constant-size copies in the inline path. It is not intended
as a durability proof.

> --- Patch [8]: [PATCH v7 8/9] mm: use memcpy_nontemporal() in zone-device template copies ---
> commit ebc6b81ea2e4b12a555515b9468c89fd0bdfe4f1
> Author: Li Zhe <lizhe.67@bytedance.com>
> mm: use memcpy_nontemporal() in zone-device template copies
> The template fast path currently uses memcpy() for the actual struct page
> copy. Switch zone_device_page_init_from_template() to memcpy_nontemporal()
> to reduce write-allocate cache traffic.
> > diff --git a/mm/mm_init.c b/mm/mm_init.c
> > index 2668acef4fe60..2a723a518f413 100644
> > --- a/mm/mm_init.c
> > +++ b/mm/mm_init.c
> [ ... ]
> > @@ -1110,7 +1120,7 @@ static void zone_device_page_init_from_template(struct page *page,
> >  	 * to the destination page.
> >  	 */
> >  	zone_device_page_update_template(template, pfn);
> > -	memcpy(page, template, sizeof(*page));
> > +	memcpy_nontemporal(page, template, sizeof(*page));
> Could this mix of non-temporal and temporal stores to the same memory location
> cause data corruption on x86?
> During DAX device binding, memmap_init_zone_device() copies the template to
> the head page using this memcpy_nontemporal() call, and then immediately
> calls memmap_init_compound():
> mm/mm_init.c:memmap_init_zone_device() {
>     ...
> 		} else {
> 			zone_device_page_init_from_template(page, pfn,
> 							    &template);
> 		}
>     ...
> 		memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
> 				     compound_nr_pages(pfn, altmap, pgmap),
> 				     use_template);
>     ...
> }
> Inside memmap_init_compound(), a normal temporal store is applied to the
> head page:
> mm/mm_init.c:memmap_init_compound() {
>     ...
> 	/*
> 	 * We have to initialize the pages, including setting up page links.
> 	 * prep_compound_page() does not take care of that, so instead we
> 	 * open-code prep_compound_page() so we can take care of initializing
> 	 * the pages in the same go.
> 	 */
> 	__SetPageHead(head);
>     ...
> }
> The commit message states:
> "The ordinary stores that follow in this path, such as compound-page setup,
> are part of the same CPU's initialization sequence; they are not used as a
> publication store that tells another CPU or device to consume data written
> by the non-temporal copy. Therefore this call site does not need a helper-
> level drain for correctness."
> However, because x86 does not maintain coherency between write-combining (WC)
> buffers and temporal cache lines for the same address, a pending WC buffer
> flush can unpredictably overwrite the L1 cache updates from the same CPU. This
> could cause the PG_head bit to be lost.
> Does this path actually require an intervening memory barrier (like sfence)
> before the temporal writes to guarantee correctness?

I do not think an sfence is required between the template copy and the
following compound-page setup.

The weak-ordering concern for non-temporal stores is about when those
stores become visible to other agents, and about ordering them before an
external handoff. The following __SetPageHead() store is not such a
handoff. It is still part of the same CPU's construction of the same
struct page metadata.

I do not think the x86 memory-ordering rules imply that a later
same-CPU temporal update to the same struct page can be silently lost
behind an earlier non-temporal store. The required fence is for ordering
non-temporal stores before an external observer or handoff.

So the relevant ordering point would be before publishing data written by
non-temporal stores to another CPU or device. This path does not do that
between the memcpy_nontemporal() call and the following compound-page
stores. It is only continuing local initialization of the struct page.

Thanks,
Zhe

On 7/20/26 05:02, Li Zhe wrote:
> This reduces the average memmap initialization time measured during
> rebind by about 60.4% for nd_pmem and 56.4% for dax_pmem.

There are lots of things in the kernel we could optimize.

Why is this important?

On 7/20/26 9:05 PM, Dave Hansen wrote:
> On 7/20/26 05:02, Li Zhe wrote:
>> This reduces the average memmap initialization time measured during
>> rebind by about 60.4% for nd_pmem and 56.4% for dax_pmem.
> There are lots of things in the kernel we could optimize.
>
> Why is this important?
This is in the synchronous probe/bind path for large DAX/PMEM
ZONE_DEVICE mappings. Userspace operations such as enabling or
reconfiguring nd_pmem/dax_pmem namespaces, or bringing a hot-added PMEM
device online, will benefit from the lower synchronous initialization
latency.

The numbers in the cover letter are only for memmap_init_zone_device(),
not for the full driver bind operation. But that time is still a
blocking part of the user-visible operation. On the 100 GB setup it is
already a few hundred milliseconds per bind/rebind, and the cost grows
with the amount of ZONE_DEVICE memmap initialized. With larger devices,
or several namespaces being brought online, that becomes visible
provisioning and recovery latency.

So the benefit is lower latency for provisioning, hot-add, and recovery
or rebind of large DAX/PMEM devices.

Thanks,
Zhe

memmap_init_zone_device() can take a noticeable amount of time when large
pmem namespaces are bound or rebound, because it initializes nearly
identical struct page descriptors one PFN at a time. This series reduces
that ZONE_DEVICE memmap initialization overhead by reusing prepared
struct page templates and, on x86, using memcpy_nontemporal() for the
template copy path.

The main target is large fsdax/devdax pmem configurations, where the
cost of initializing the memmap shows up directly in nd_pmem/dax_pmem
bind and rebind latency.

Patches 1-3 are preparatory cleanups and helper extraction. Patches 4-5
add the template-copy path for head pages and compound tails. Patches
6-8 introduce memcpy_nontemporal(), extend the x86 fixed-size
memcpy_flushcache() inline cases used by that helper, and switch the
template-copy path over to memcpy_nontemporal(). Patch 9 removes the
remaining local opt-out predicate and always uses the template path after
the first real page seeds the reusable template.

Architectures without a specialized memcpy_nontemporal() backend fall
back to memcpy(), so the template-copy optimization remains available
without arch-specific support. On x86, memcpy_nontemporal() maps to the
existing memcpy_flushcache() backend and can use the fixed-size MOVNTI
paths added by this series for struct page sized copies.

This version also incorporates a round of review comments from Sashiko
on patches 6-8.

For patch 6, the generic memcpy_nontemporal() fallback is a macro alias
to memcpy(), rather than an inline wrapper, so architectures without a
specialized backend keep the usual memcpy() FORTIFY coverage when object
sizes remain visible at the original call site.

For patch 7, I re-checked whether the new 32/48/64/80/96-byte MOVNTI
cases should add a separate alignment check before staying on the inline
fast path. I re-checked the Intel SDM together with the existing
memcpy_flushcache() behavior, and I did not find an architectural
statement that would make those new cases require a different
correctness rule from the long-standing 4/8/16-byte fast paths. This
series therefore keeps the new fixed-size cases inline without adding a
separate alignment check.

For patch 8, I re-checked whether the template-copy path needs a
helper-level drain after the MOVNTI-based copies. This series keeps
memcpy_nontemporal() as a copy primitive only. Callers that use it for a
producer-consumer or device-visible handoff must provide the required
ordering. The ZONE_DEVICE template-copy path uses it only while
initializing struct page metadata, so the copy primitive itself does not
grow a separate drain contract.

The numbers below measure the time spent in memmap_init_zone_device()
during driver bind/rebind. They are not measurements of the full
nd_pmem or dax_pmem bind/rebind operation.

Tested in a VM with a 100 GB fsdax namespace device configured with
map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
server.

Test procedure:
Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
initialization time from the pr_debug() output of
memmap_init_zone_device().

Base(v7.2-rc1):
  First binding for nd_pmem driver: 1456 ms
  Average of subsequent rebinds: 244.28 ms

  First binding for dax_pmem driver: 1462 ms
  Average of subsequent rebinds: 273.31 ms

With this series applied:
  First binding for nd_pmem driver: 1272 ms
  Average of subsequent rebinds: 96.79 ms

  First binding for dax_pmem driver: 1354 ms
  Average of subsequent rebinds: 119.04 ms

This reduces the average memmap initialization time measured during
rebind by about 60.4% for nd_pmem and 56.4% for dax_pmem.

As an additional data point, I also ran a smaller set of measurements on
the same physical x86_64 host with a 100 GB PMEM region created via the
memmap= kernel command line, configured as fsdax and devdax namespaces
with map=dev and 2 MiB alignment.

For brevity, the individual patches keep only the VM results rather than
including a second set of physical-host measurements throughout the
series. The physical-host numbers below are included only as
supplemental evidence that the same optimization also provides a similar
benefit on a non-virtualized system.

Test procedure:
Reconfigure the namespace mode, rebind the nd_pmem or dax_pmem driver
once, and collect the memmap initialization time from the pr_debug()
output of memmap_init_zone_device().

Base (v7.2-rc1):
  nd_pmem / fsdax: 179 ms
  dax_pmem / devdax: 264 ms

With this series applied:
  nd_pmem / fsdax: 82 ms
  dax_pmem / devdax: 113 ms

This reduces the measured memmap initialization time during rebind by
about 54.2% for nd_pmem and 57.2% for dax_pmem on that setup, which is
broadly consistent with the VM results above.

As another supplemental data point, I also measured the test_hmm.ko
module on the same physical x86_64 host, using the test_hmm.ko setup
from the previous discussion that times ten 64 GB
memremap_pages()/memunmap_pages() iterations during module insertion[1].
By default, module insertion initializes two DEVICE_PRIVATE dmirror
devices, so two avg memremap values are reported; each value is the
average for one 64 GB chunk.

This is not the primary target workload of the series, but it exercises
the same large ZONE_DEVICE memmap initialization path and shows the same
direction of improvement.

Base (v7.2-rc1):
  avg memremap reported during module insertion: 116689362 ns, 116539263 ns

With this series applied:
  avg memremap reported during module insertion: 54607108 ns, 54458236 ns

This corresponds to about a 53.2% reduction based on the mean of the
reported values, which is again consistent with the pmem bind/rebind
results above.

[1] https://lore.kernel.org/all/aiEoByaQdRR3xtM5@nvdebian.thelocal/

Li Zhe (9):
  mm: fix stale ZONE_DEVICE refcount comment
  mm: factor zone-device page init helpers out of
    __init_zone_device_page
  mm: add a set_page_section_from_pfn() helper
  mm: add a template-based fast path for zone-device page init
  mm: extend the template fast path to zone-device compound tails
  string: introduce memcpy_nontemporal()
  x86/string: extend memcpy_flushcache() fixed-size fastpaths
  mm: use memcpy_nontemporal() in zone-device template copies
  mm: always use the zone-device template init path

 arch/x86/include/asm/string_64.h |  68 +++++++++++++++-
 include/linux/mm.h               |  15 +++-
 include/linux/string.h           |  12 +++
 mm/mm_init.c                     | 131 +++++++++++++++++++++++++------
 4 files changed, 198 insertions(+), 28 deletions(-)

---
v6: https://lore.kernel.org/all/20260709112520.24857-1-lizhe.67@bytedance.com/
v5: https://lore.kernel.org/all/20260701090553.62691-1-lizhe.67@bytedance.com/
v4: https://lore.kernel.org/all/20260603080152.64728-1-lizhe.67@bytedance.com/
v3: https://lore.kernel.org/all/20260527033636.28231-1-lizhe.67@bytedance.com/
v2: https://lore.kernel.org/all/20260521040124.10608-1-lizhe.67@bytedance.com/
v1: https://lore.kernel.org/all/20260515082045.63029-1-lizhe.67@bytedance.com/

Changelogs:

v6->v7:
- Rename memcpy_nt() to memcpy_nontemporal() so the generic helper name
  is explicit rather than abbreviated. Suggested by Muchun Song.
- Drop the separate memcpy_nt_drain() helper and remove the drain calls
  from the ZONE_DEVICE template-copy path. Suggested by Muchun Song.
- Rework patch 6 so the generic memcpy_nontemporal() fallback uses
  '#define memcpy_nontemporal memcpy', preserving the usual memcpy()
  FORTIFY coverage when object sizes remain visible at the original call
  site.
- Summarize the off-list Sashiko review comments relayed by Andrew on
  the memcpy helper and x86 memcpy_flushcache() pieces.
- Add patch 9 to remove the local template-enable predicate and the
  remaining non-template fallback path, as suggested by Muchun. This
  also means KASAN/KMSAN builds no longer force the per-page
  initialization path, and page_ref_set no longer observes every
  initialization-time refcount assignment.
- Re-ran the VM performance test for v7. The results were close to the
  previously reported v5 numbers, so keep the existing performance data
  unchanged.
- Rename pagemap_resets_refcount() to pagemap_requires_refcount_reset()
  and invert the predicate accordingly. Suggested by Balbir Singh.

For changelogs of earlier revisions, please refer to the v6 cover letter.

-- 
2.20.1