This series adds uapi bits for private memory NUMA node [0] control.

It is not intended for merge - just discussion.

I have been using this to extensively test mm/ behaviors with private
nodes [1] and am wondering whether some of this is more generally
useful for coherent memory devices if private nodes lands.

For now, this is useful for testing, but locks in UAPI, so this RFC
is not intended as an intent to upstream - just sharing my testing api.
  
  Question 1: Do we want dax control of private node settings?
              1a) External? (uapi like this?)
	      1b) Internal-only? (plumbed through module apis)

  Question 2: If we do want this, do we want it in kmem or a
              separate dax driver? (dax/private_mem ?)

  Question 3: are dax_file and adistance useful regardless?

This adds 9 interfaces:

  dax0.0/dax_file
     this makes /dev/dax0.0 mmap-able when online as kmem
     it essentially attaches a default mempolicy to the vma
     that binds it to the dax device's node.

  dax0.0/private
     this switches whether hotplug sets N_MEMORY or N_MEMORY_PRIVATE

  dax0.0/adistance
     this allows overriding the default adistance to for the node
     into a specific tier.  That allows for testing tiering in both
     directions (demotion from/to the node)

  dax0.0/reclaim
  dax0.0/user_numa
  dax0.0/hotunplug
  dax0.0/demotion
  dax0.0/numa_balancing
  dax0.0/ltpin
     these bits all enable their respective NODE_PRIVATE_CAP_ bit

     reclaim   - enables reclaim on the node
     user_numa - enables mbind/mempolicy/move_pages/migrate_pages
     hotunplug - enables hotunplug to migrate memory
     demotion  - enables reclaim to demote from or to the node
     numa_balancing - allows prot_none injection on the node's folios
     ltpin     - allows longterm pin on node's folios to work

[0] private node series
https://lore.kernel.org/linux-mm/20260720193431.3841992-1-gourry@gourry.net/

[1] selftests/dax extensions for all of this that show use cases and
    testing for the private node series - including all user api's
    like mbind, set_mempolicy, move_pages, migrate_privates - as well
    as reclaim behaviors.
https://github.com/gourryinverse/linux/commits/scratch/gourry/managed_nodes/dax_private-mm-new/

Gregory Price (8):
  dax/kmem: add dax_file= to expose an anonymous-fault char device
  dax/kmem: add private= to online memory as an N_MEMORY_PRIVATE node
  dax/kmem: add the reclaim opt-in (CAP_RECLAIM) for private nodes
  dax/kmem: add the user_numa opt-in (CAP_USER_NUMA) for private nodes
  dax/kmem: add the hotunplug opt-in (CAP_HOTUNPLUG) for private nodes
  dax/kmem: add the demotion and numa_balancing opt-ins for private
    nodes
  dax/kmem: add the ltpin opt-in (CAP_LTPIN) for private nodes
  dax/kmem: add dax/adistance to make private node adistance
    configurable

 drivers/dax/kmem.c | 368 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 366 insertions(+), 2 deletions(-)

-- 
2.53.0-Meta

On Wed, Jul 22, 2026 at 02:35:11PM +0800, Gregory Price wrote:
> This series adds uapi bits for private memory NUMA node [0] control.
> 
> It is not intended for merge - just discussion.
> 
> I have been using this to extensively test mm/ behaviors with private
> nodes [1] and am wondering whether some of this is more generally
> useful for coherent memory devices if private nodes lands.
> 
> For now, this is useful for testing, but locks in UAPI, so this RFC
> is not intended as an intent to upstream - just sharing my testing api.
>   
>   Question 1: Do we want dax control of private node settings?
>               1a) External? (uapi like this?)
> 	      1b) Internal-only? (plumbed through module apis)
> 
>   Question 2: If we do want this, do we want it in kmem or a
>               separate dax driver? (dax/private_mem ?)
> 
>   Question 3: are dax_file and adistance useful regardless?
> 
> This adds 9 interfaces:
> 
>   dax0.0/dax_file
>      this makes /dev/dax0.0 mmap-able when online as kmem
>      it essentially attaches a default mempolicy to the vma
>      that binds it to the dax device's node.
> 
>   dax0.0/private
>      this switches whether hotplug sets N_MEMORY or N_MEMORY_PRIVATE
> 
>   dax0.0/adistance
>      this allows overriding the default adistance to for the node
>      into a specific tier.  That allows for testing tiering in both
>      directions (demotion from/to the node)
> 
>   dax0.0/reclaim
>   dax0.0/user_numa
>   dax0.0/hotunplug
>   dax0.0/demotion
>   dax0.0/numa_balancing
>   dax0.0/ltpin
>      these bits all enable their respective NODE_PRIVATE_CAP_ bit
> 
>      reclaim   - enables reclaim on the node
>      user_numa - enables mbind/mempolicy/move_pages/migrate_pages
>      hotunplug - enables hotunplug to migrate memory
>      demotion  - enables reclaim to demote from or to the node
>      numa_balancing - allows prot_none injection on the node's folios
>      ltpin     - allows longterm pin on node's folios to work
>

Hi Gregory,

I have some thoughts especially for ownership and configs.

I think it would be good to expose dax control of private node, just with
the following guarantees.

Let's make it clear that userspace should never become the owner
of private node, they should be driver-owned and 1-to-1 mapping relationship.

A userspace tool can request to reconfigure some capability of the DAX device, and
maybe how the driver expose it.

A dax range may have several deployment rules, e.g. devdax, system-ram or private, with
the patch series' tool, it looks like a userspace tool can convert a DAX device between
dax and system-ram mode, I think that's unsafe especially when the region is online as
system-ram.

And it will cause coherence issue to be resolved, if no special or strong use cases, I would
prevent having such ability to convert the rules in runtime, especially for fixed-purpose device.

For capability bitmap part, I think it's reasonable to expose it, but with 2 concepts
- supported_caps : immutable, supplied by the provider/vendor
- enabled-caps : selected policy, subset of supported_caps

The invariant should always hold
"""
enabled_caps & ~supported_caps == 0
"""

This way we can prevent userspace from touching things that they shouldn't.

Maybe a demo with a driver would be nice ?

This is my initial thoughts of this, thanks.

Best regards,
Richard Cheng.

> [0] private node series
> https://lore.kernel.org/linux-mm/20260720193431.3841992-1-gourry@gourry.net/
> 
> [1] selftests/dax extensions for all of this that show use cases and
>     testing for the private node series - including all user api's
>     like mbind, set_mempolicy, move_pages, migrate_privates - as well
>     as reclaim behaviors.
> https://github.com/gourryinverse/linux/commits/scratch/gourry/managed_nodes/dax_private-mm-new/
> 
> Gregory Price (8):
>   dax/kmem: add dax_file= to expose an anonymous-fault char device
>   dax/kmem: add private= to online memory as an N_MEMORY_PRIVATE node
>   dax/kmem: add the reclaim opt-in (CAP_RECLAIM) for private nodes
>   dax/kmem: add the user_numa opt-in (CAP_USER_NUMA) for private nodes
>   dax/kmem: add the hotunplug opt-in (CAP_HOTUNPLUG) for private nodes
>   dax/kmem: add the demotion and numa_balancing opt-ins for private
>     nodes
>   dax/kmem: add the ltpin opt-in (CAP_LTPIN) for private nodes
>   dax/kmem: add dax/adistance to make private node adistance
>     configurable
> 
>  drivers/dax/kmem.c | 368 ++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 366 insertions(+), 2 deletions(-)
> 
> -- 
> 2.53.0-Meta
> 
>

On Fri, Jul 24, 2026 at 03:45:23PM +0800, Richard Cheng wrote:
> On Wed, Jul 22, 2026 at 02:35:11PM +0800, Gregory Price wrote:
> 
> Hi Gregory,
> 
> I have some thoughts especially for ownership and configs.
> 
> I think it would be good to expose dax control of private node, just with
> the following guarantees.
> 
> Let's make it clear that userspace should never become the owner
> of private node, they should be driver-owned and 1-to-1 mapping relationship.
> 

The intent here is 100% 1-to-1 driver:node mappings.

Technically there's nothing unreasonable about single-owner multi-node, but
really we just don't want multiple devices on one node.

> A userspace tool can request to reconfigure some capability of the DAX device, and
> maybe how the driver expose it.
> 
> A dax range may have several deployment rules, e.g. devdax, system-ram or private, with
> the patch series' tool, it looks like a userspace tool can convert a DAX device between
> dax and system-ram mode, I think that's unsafe especially when the region is online as
> system-ram.
> 
> And it will cause coherence issue to be resolved, if no special or strong use cases, I would
> prevent having such ability to convert the rules in runtime, especially for fixed-purpose device.
> 

Right now we can't really prevent a user from doing:

echo dax0.0 > dax/drivers/devdax/unbind
echo dax0.0 > dax/drivers/kmem/bind

And in fact we depend on kmem being the default target for CXL to
auto-probe through to hotplug.  For CXL accelerators, the current
intended auto-probe behavior is to terminate at an offline kmem dax
binding, and then expect the user to either rebind the dax device, or
rebind the entire cxl device:

echo $DEVICE > ../drivers/cxl_pci/unbind
echo $DEVICE > ../pci/my_cxl_device_driver/bind

I have some old patches that augment how drivers select the dax mode,
I can see about reviving that and maybe letting the source driver
also dictate which dax modes are valid according to the source.

I'll look into that.

> For capability bitmap part, I think it's reasonable to expose it, but with 2 concepts
> - supported_caps : immutable, supplied by the provider/vendor
> - enabled-caps : selected policy, subset of supported_caps
> 
> The invariant should always hold
> """
> enabled_caps & ~supported_caps == 0
> """
> 
> This way we can prevent userspace from touching things that they shouldn't.
> 

This I like, and i think you've convinced me that this should be a
separate dax device driver so we're not overloading kmem.

That said, for a dax device that was not created by a driver, but
instead created via something like "memmap=" boot parameter, i'm we
can't really dictate supported caps - but for a driver that *does* want
to limit support, I think we can manage that.

> Maybe a demo with a driver would be nice ?
> 

Sure! I think i'm going to include my anon-memory only compressed ram
service back into v6 w/ the dax driver extensions you've suggested.

~Gregory