[v1] mm/memremap: fix spurious large folio warning for FS-DAX

[PATCH] mm/memremap: fix spurious large folio warning for FS-DAX

Posted by John Groves 1 month, 3 weeks ago

From: John Groves <John@Groves.net>

This patch addresses a warning that I discovered while working on famfs,
which is an fs-dax file system that virtually always does PMD faults
(next famfs patch series coming after the holidays).

However, XFS also does PMD faults in fs-dax mode, and it also triggers
the warning. It takes some effort to get XFS to do a PMD fault, but
instructions to reproduce it are below.

The VM_WARN_ON_ONCE(folio_test_large(folio)) check in
free_zone_device_folio() incorrectly triggers for MEMORY_DEVICE_FS_DAX
when PMD (2MB) mappings are used.

FS-DAX legitimately creates large file-backed folios when handling PMD
faults. This is a core feature of FS-DAX that provides significant
performance benefits by mapping 2MB regions directly to persistent
memory. When these mappings are unmapped, the large folios are freed
through free_zone_device_folio(), which triggers the spurious warning.

The warning was introduced by commit that added support for large zone
device private folios. However, that commit did not account for FS-DAX
file-backed folios, which have always supported large (PMD-sized)
mappings.

The check distinguishes between anonymous folios (which clear
AnonExclusive flags for each sub-page) and file-backed folios. For
file-backed folios, it assumes large folios are unexpected - but this
assumption is incorrect for FS-DAX.

The fix is to exempt MEMORY_DEVICE_FS_DAX from the large folio warning,
allowing FS-DAX to continue using PMD mappings without triggering false
warnings.

Signed-off-by: John Groves <john@groves.net>
---
=== How to reproduce ===

A reproducer is available at:

    git clone https://github.com/jagalactic/dax-pmd-test.git
    cd xfs-dax-test
    make
    sudo make test

This will set up XFS on pmem with 2MB stripe alignment and run a test
that triggers the warning.

Alternatively, follow the manual steps below.

Prerequisites:
  - Linux kernel with FS-DAX support and CONFIG_DEBUG_VM=y
  - A pmem device (real or emulated)
  - An fsdax namespace configured via ndctl as /dev/pmem0

Manual steps:

1. Create an fsdax namespace (if not already present):
   # ndctl create-namespace -m fsdax -e namespace0.0

2. Create XFS with 2MB stripe alignment:
   # mkfs.xfs -f -d su=2m,sw=1 /dev/pmem0
   # mount -o dax /dev/pmem0 /mnt/pmem

3. Compile and run the reproducer:
   # gcc -Wall -O2 -o dax_pmd_test dax_pmd_test.c
   # ./dax_pmd_test /mnt/pmem/testfile

4. Check dmesg for the warning:
   WARNING: mm/memremap.c:431 at free_zone_device_folio+0x.../0x...

Note: The 2MB stripe alignment (-d su=2m,sw=1) is critical. XFS normally
allocates blocks at arbitrary offsets, causing PMD faults to fall back
to PTE faults. The stripe alignment forces 2MB-aligned allocations,
allowing PMD faults to succeed and exposing this bug.

=== Proposed fix ===

mm/memremap.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/memremap.c b/mm/memremap.c
index 4c2e0d68eb27..af37c3b4e39b 100644
--- a/mm/memremap.c
+++ b/mm/memremap.c
@@ -428,7 +428,12 @@ void free_zone_device_folio(struct folio *folio)
 		for (i = 0; i < nr; i++)
 			__ClearPageAnonExclusive(folio_page(folio, i));
 	} else {
-		VM_WARN_ON_ONCE(folio_test_large(folio));
+		/*
+		 * FS_DAX legitimately uses large file-mapped folios for
+		 * PMD mappings, so only warn for other device types.
+		 */
+		VM_WARN_ON_ONCE(pgmap->type != MEMORY_DEVICE_FS_DAX &&
+				folio_test_large(folio));
 	}
 
 	/*

base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
-- 
2.49.0

Re: [PATCH] mm/memremap: fix spurious large folio warning for FS-DAX

Posted by dan.j.williams@intel.com 1 month, 3 weeks ago

John Groves wrote:
> From: John Groves <John@Groves.net>
> 
> This patch addresses a warning that I discovered while working on famfs,
> which is an fs-dax file system that virtually always does PMD faults
> (next famfs patch series coming after the holidays).
> 
> However, XFS also does PMD faults in fs-dax mode, and it also triggers
> the warning. It takes some effort to get XFS to do a PMD fault, but
> instructions to reproduce it are below.
> 
> The VM_WARN_ON_ONCE(folio_test_large(folio)) check in
> free_zone_device_folio() incorrectly triggers for MEMORY_DEVICE_FS_DAX
> when PMD (2MB) mappings are used.
> 
> FS-DAX legitimately creates large file-backed folios when handling PMD
> faults. This is a core feature of FS-DAX that provides significant
> performance benefits by mapping 2MB regions directly to persistent
> memory. When these mappings are unmapped, the large folios are freed
> through free_zone_device_folio(), which triggers the spurious warning.
> 
> The warning was introduced by commit that added support for large zone
> device private folios. However, that commit did not account for FS-DAX
> file-backed folios, which have always supported large (PMD-sized)
> mappings.

Oh, I was not copied on:

d245f9b4ab80 mm/zone_device: support large zone device private folios

...I should probably add myself as a reviewer to the MEMORY HOT(UN)PLUG
entry in MAINTAINERS at least for the mm/mememap.c bits.

Now, why is the warning there in the first place?

I.e. what is the risk of just doing this fixup:

diff --git a/mm/memremap.c b/mm/memremap.c
index 4c2e0d68eb27..63c6ab4fdf08 100644
--- a/mm/memremap.c
+++ b/mm/memremap.c
@@ -427,8 +427,6 @@ void free_zone_device_folio(struct folio *folio)
        if (folio_test_anon(folio)) {
                for (i = 0; i < nr; i++)
                        __ClearPageAnonExclusive(folio_page(folio, i));
-       } else {
-               VM_WARN_ON_ONCE(folio_test_large(folio));
        }
 
        /*

Re: [PATCH] mm/memremap: fix spurious large folio warning for FS-DAX

Posted by Alistair Popple 1 month, 3 weeks ago

On 2025-12-18 at 10:59 +1100, dan.j.williams@intel.com wrote...
> John Groves wrote:
> > From: John Groves <John@Groves.net>
> > 
> > This patch addresses a warning that I discovered while working on famfs,
> > which is an fs-dax file system that virtually always does PMD faults
> > (next famfs patch series coming after the holidays).
> > 
> > However, XFS also does PMD faults in fs-dax mode, and it also triggers
> > the warning. It takes some effort to get XFS to do a PMD fault, but
> > instructions to reproduce it are below.
> > 
> > The VM_WARN_ON_ONCE(folio_test_large(folio)) check in
> > free_zone_device_folio() incorrectly triggers for MEMORY_DEVICE_FS_DAX
> > when PMD (2MB) mappings are used.
> > 
> > FS-DAX legitimately creates large file-backed folios when handling PMD
> > faults. This is a core feature of FS-DAX that provides significant
> > performance benefits by mapping 2MB regions directly to persistent
> > memory. When these mappings are unmapped, the large folios are freed
> > through free_zone_device_folio(), which triggers the spurious warning.
> > 
> > The warning was introduced by commit that added support for large zone
> > device private folios. However, that commit did not account for FS-DAX
> > file-backed folios, which have always supported large (PMD-sized)
> > mappings.
> 
> Oh, I was not copied on:
> 
> d245f9b4ab80 mm/zone_device: support large zone device private folios
> 
> ...I should probably add myself as a reviewer to the MEMORY HOT(UN)PLUG
> entry in MAINTAINERS at least for the mm/mememap.c bits.
> 
> Now, why is the warning there in the first place?

Lets wait for Balbir to comment but I suspect it's just a mistake in
d245f9b4ab80 ("mm/zone_device: support large zone device private folios"):

-       /*
-        * Note: we don't expect anonymous compound pages yet. Once supported
-        * and we could PTE-map them similar to THP, we'd have to clear
-        * PG_anon_exclusive on all tail pages.
-        */
        if (folio_test_anon(folio)) {
-               VM_BUG_ON_FOLIO(folio_test_large(folio), folio);
-               __ClearPageAnonExclusive(folio_page(folio, 0));
+               for (i = 0; i < nr; i++)
+                       __ClearPageAnonExclusive(folio_page(folio, i));
+       } else {
+               VM_WARN_ON_ONCE(folio_test_large(folio));
        }

The warning never applied to !folio_test_anon() folios and was just a reminder
for the comment above which was deleted because it was fixed. Therefore the
warning should also have been deleted.

> I.e. what is the risk of just doing this fixup:

None, I think that is the correct fix.

> diff --git a/mm/memremap.c b/mm/memremap.c
> index 4c2e0d68eb27..63c6ab4fdf08 100644
> --- a/mm/memremap.c
> +++ b/mm/memremap.c
> @@ -427,8 +427,6 @@ void free_zone_device_folio(struct folio *folio)
>         if (folio_test_anon(folio)) {
>                 for (i = 0; i < nr; i++)
>                         __ClearPageAnonExclusive(folio_page(folio, i));
> -       } else {
> -               VM_WARN_ON_ONCE(folio_test_large(folio));
>         }
>  
>         /*
>

Re: [PATCH] mm/memremap: fix spurious large folio warning for FS-DAX

Posted by Alistair Popple 1 month, 3 weeks ago

On 2025-12-18 at 08:13 +1100, John Groves <John@Groves.net> wrote...
> From: John Groves <John@Groves.net>
> 
> This patch addresses a warning that I discovered while working on famfs,
> which is an fs-dax file system that virtually always does PMD faults
> (next famfs patch series coming after the holidays).
> 
> However, XFS also does PMD faults in fs-dax mode, and it also triggers
> the warning. It takes some effort to get XFS to do a PMD fault, but
> instructions to reproduce it are below.
> 
> The VM_WARN_ON_ONCE(folio_test_large(folio)) check in
> free_zone_device_folio() incorrectly triggers for MEMORY_DEVICE_FS_DAX
> when PMD (2MB) mappings are used.
> 
> FS-DAX legitimately creates large file-backed folios when handling PMD
> faults. This is a core feature of FS-DAX that provides significant
> performance benefits by mapping 2MB regions directly to persistent
> memory. When these mappings are unmapped, the large folios are freed
> through free_zone_device_folio(), which triggers the spurious warning.

Yep, and I'm pretty sure devdax can also create large folios so we might need
a similar fix there. In fact looking at old vs. new code it seems we only ever
used to have this warning for anon folios, which I think could only ever be true
for DEVICE_PRIVATE or DEVICE_COHERENT folios.

So I suspect the proper fix is to just remove the warning entirely now that they
also support compound sizes.

> The warning was introduced by commit that added support for large zone
> device private folios. However, that commit did not account for FS-DAX
> file-backed folios, which have always supported large (PMD-sized)
> mappings.

Right, one of the nice side-effects (other than delaying fam-fs, sorry! :-/) of
fixing the refcounting was that these started looking like normal large folios.

> The check distinguishes between anonymous folios (which clear
> AnonExclusive flags for each sub-page) and file-backed folios. For
> file-backed folios, it assumes large folios are unexpected - but this
> assumption is incorrect for FS-DAX.
> 
> The fix is to exempt MEMORY_DEVICE_FS_DAX from the large folio warning,
> allowing FS-DAX to continue using PMD mappings without triggering false
> warnings.

As this is a fix you will want a "Fixes:" tag.

> Signed-off-by: John Groves <john@groves.net>
> ---
> === How to reproduce ===
> 
> A reproducer is available at:
> 
>     git clone https://github.com/jagalactic/dax-pmd-test.git
>     cd xfs-dax-test
>     make
>     sudo make test
> 
> This will set up XFS on pmem with 2MB stripe alignment and run a test
> that triggers the warning.
> 
> Alternatively, follow the manual steps below.
> 
> Prerequisites:
>   - Linux kernel with FS-DAX support and CONFIG_DEBUG_VM=y
>   - A pmem device (real or emulated)
>   - An fsdax namespace configured via ndctl as /dev/pmem0
> 
> Manual steps:
> 
> 1. Create an fsdax namespace (if not already present):
>    # ndctl create-namespace -m fsdax -e namespace0.0
> 
> 2. Create XFS with 2MB stripe alignment:
>    # mkfs.xfs -f -d su=2m,sw=1 /dev/pmem0
>    # mount -o dax /dev/pmem0 /mnt/pmem
> 
> 3. Compile and run the reproducer:
>    # gcc -Wall -O2 -o dax_pmd_test dax_pmd_test.c
>    # ./dax_pmd_test /mnt/pmem/testfile
> 
> 4. Check dmesg for the warning:
>    WARNING: mm/memremap.c:431 at free_zone_device_folio+0x.../0x...
> 
> Note: The 2MB stripe alignment (-d su=2m,sw=1) is critical. XFS normally
> allocates blocks at arbitrary offsets, causing PMD faults to fall back
> to PTE faults. The stripe alignment forces 2MB-aligned allocations,
> allowing PMD faults to succeed and exposing this bug.

Thanks for the detailed repro instructions. Not always neccessary but definitely
nice to have.

> === Proposed fix ===
> 
> mm/memremap.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/memremap.c b/mm/memremap.c
> index 4c2e0d68eb27..af37c3b4e39b 100644
> --- a/mm/memremap.c
> +++ b/mm/memremap.c
> @@ -428,7 +428,12 @@ void free_zone_device_folio(struct folio *folio)
>  		for (i = 0; i < nr; i++)
>  			__ClearPageAnonExclusive(folio_page(folio, i));
>  	} else {
> -		VM_WARN_ON_ONCE(folio_test_large(folio));
> +		/*
> +		 * FS_DAX legitimately uses large file-mapped folios for
> +		 * PMD mappings, so only warn for other device types.
> +		 */
> +		VM_WARN_ON_ONCE(pgmap->type != MEMORY_DEVICE_FS_DAX &&
> +				folio_test_large(folio));
>  	}
>  
>  	/*
> 
> base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
> -- 
> 2.49.0
>

Re: [PATCH] mm/memremap: fix spurious large folio warning for FS-DAX

Posted by Andrew Morton 1 month, 3 weeks ago

On Thu, 18 Dec 2025 09:58:02 +1100 Alistair Popple <apopple@nvidia.com> wrote:

> On 2025-12-18 at 08:13 +1100, John Groves <John@Groves.net> wrote...
> > From: John Groves <John@Groves.net>
> > 
> > This patch addresses a warning that I discovered while working on famfs,
> > which is an fs-dax file system that virtually always does PMD faults
> > (next famfs patch series coming after the holidays).
> > 
> > However, XFS also does PMD faults in fs-dax mode, and it also triggers
> > the warning. It takes some effort to get XFS to do a PMD fault, but
> > instructions to reproduce it are below.
> > 
> > The VM_WARN_ON_ONCE(folio_test_large(folio)) check in
> > free_zone_device_folio() incorrectly triggers for MEMORY_DEVICE_FS_DAX
> > when PMD (2MB) mappings are used.
> > 
> > FS-DAX legitimately creates large file-backed folios when handling PMD
> > faults. This is a core feature of FS-DAX that provides significant
> > performance benefits by mapping 2MB regions directly to persistent
> > memory. When these mappings are unmapped, the large folios are freed
> > through free_zone_device_folio(), which triggers the spurious warning.
> 
> Yep, and I'm pretty sure devdax can also create large folios so we might need
> a similar fix there. In fact looking at old vs. new code it seems we only ever
> used to have this warning for anon folios, which I think could only ever be true
> for DEVICE_PRIVATE or DEVICE_COHERENT folios.
> 
> So I suspect the proper fix is to just remove the warning entirely now that they
> also support compound sizes.

So I'm assuming we can expect an updated version of this fix.

> > The warning was introduced by commit that added support for large zone
> > device private folios. However, that commit did not account for FS-DAX
> > file-backed folios, which have always supported large (PMD-sized)
> > mappings.
> 
> Right, one of the nice side-effects (other than delaying fam-fs, sorry! :-/) of
> fixing the refcounting was that these started looking like normal large folios.
> 
> > The check distinguishes between anonymous folios (which clear
> > AnonExclusive flags for each sub-page) and file-backed folios. For
> > file-backed folios, it assumes large folios are unexpected - but this
> > assumption is incorrect for FS-DAX.
> > 
> > The fix is to exempt MEMORY_DEVICE_FS_DAX from the large folio warning,
> > allowing FS-DAX to continue using PMD mappings without triggering false
> > warnings.
> 
> As this is a fix you will want a "Fixes:" tag.

Someone (possibly me) already added

Fixes: d245f9b4ab80 ("mm/zone_device: support large zone device private folios")

Re: [PATCH] mm/memremap: fix spurious large folio warning for FS-DAX

Posted by John Groves 1 month, 3 weeks ago

On 25/12/18 04:03PM, Andrew Morton wrote:
> On Thu, 18 Dec 2025 09:58:02 +1100 Alistair Popple <apopple@nvidia.com> wrote:
> 
> > On 2025-12-18 at 08:13 +1100, John Groves <John@Groves.net> wrote...
> > > From: John Groves <John@Groves.net>
> > > 
> > > This patch addresses a warning that I discovered while working on famfs,
> > > which is an fs-dax file system that virtually always does PMD faults
> > > (next famfs patch series coming after the holidays).
> > > 
> > > However, XFS also does PMD faults in fs-dax mode, and it also triggers
> > > the warning. It takes some effort to get XFS to do a PMD fault, but
> > > instructions to reproduce it are below.
> > > 
> > > The VM_WARN_ON_ONCE(folio_test_large(folio)) check in
> > > free_zone_device_folio() incorrectly triggers for MEMORY_DEVICE_FS_DAX
> > > when PMD (2MB) mappings are used.
> > > 
> > > FS-DAX legitimately creates large file-backed folios when handling PMD
> > > faults. This is a core feature of FS-DAX that provides significant
> > > performance benefits by mapping 2MB regions directly to persistent
> > > memory. When these mappings are unmapped, the large folios are freed
> > > through free_zone_device_folio(), which triggers the spurious warning.
> > 
> > Yep, and I'm pretty sure devdax can also create large folios so we might need
> > a similar fix there. In fact looking at old vs. new code it seems we only ever
> > used to have this warning for anon folios, which I think could only ever be true
> > for DEVICE_PRIVATE or DEVICE_COHERENT folios.
> > 
> > So I suspect the proper fix is to just remove the warning entirely now that they
> > also support compound sizes.
> 
> So I'm assuming we can expect an updated version of this fix.

I'll send an update Friday morning

<snip>

Thanks Alistair, Dan and Andrew!

John