This is the first batch of patches continuing the x86 Address Space
Isolation (ASI) work that Roger posted as "x86: adventures in Address
Space Isolation" (v1 [1], v2 [2]).  I've taken over finishing it up
and getting it upstream.

Rather than re-posting the whole stack (nearly 60 patches) each time,
I'd like to run this as a rolling series: post a reviewable slice from
the front, drop patches as they are committed, and append the next
ones as they mature.  Each batch should stand on its own; the cover
letter of each will say where it sits in the larger picture.

A "map" of the entire series -- grouped into logical chunks, with the
dependencies between patches -- is maintained here:

https://xenbits.xenproject.org/people/gdunlap/asi-series-deps.html

Note that the graph above is a work in progress; dependency lines may
change as more of the series is vetted.  Note also that the full
series includes a design doc as patch 1; that's not ready for
publication yet, so patches 1-7 of this series correspond to nodes 2-8
of the graph.

The problem this slice addresses: the per-domain area already has
central machinery for building its page-tables and for managing the
backing pages it owns itself (create_perdomain_mapping() and friends);
what it lacks is a way to install a caller's own pages at a chosen
address.  The PV GDT and LDT code fills that gap privately, by having
create_perdomain_mapping() hand back aliases of the L1 tables it
builds and stashing them in d->arch.pv.gdt_ldt_l1tab; mapping updates
are then written through the stash, bypassing the interface.  That
arrangement assumes a single, domain-wide set of per-domain
page-tables, which stops holding once the per-domain area becomes
per-vCPU (the next slices).

This slice closes the gap centrally:

 - Patch 1 moves the per-domain page-table allocations from the
   domheap to the xenheap.  The page-tables (not the data pages they
   map) are then reachable through their always-mapped alias from any
   context, so walking them needs no mapcache -- including from the
   context switch.
 - Patch 2 introduces populate_perdomain_mapping() on top: a single
   writer for the per-domain area, installing caller-owned pages at a
   chosen address by walking the always-mapped page-tables.
 - Patches 3-5 convert the Xen-GDT slot, the guest GDT, and the guest
   LDT paths to it.
 - Patch 6 removes the stash.  Patch 7 simplifies
   create_perdomain_mapping(), whose L1-capture mode existed only to
   build the stash.

One point reviewers may want to look at specifically: patch 1 changes
where the per-domain page-tables are allocated from, and its commit
message discusses the (minor) NUMA-placement consequence.

Relative to v2: patch 1 is new -- v2 kept the page-tables in the
domheap and walked them through map_domain_page(), with a linear-map
fast path for the currently-running vCPU; making the page-tables
always-mapped lets one plain walk serve every caller and context, and
the context switch keeps its existing structure.  The populate patch
is split from its first user; the LDT demand-map now goes through
populate_perdomain_mapping() rather than writing linear entries;
pv_destroy_gdt() keeps mapping torn-down slots read-only to the zero
page (in v2 they became empty -- a guest-visible partial revert of
cf6d39f819); and the domain -> vCPU parameter switches move to the
next slice.  Per-patch changes are noted below each patch's "---".

Testing:
 - Each patch builds (x86_64, CONFIG_DEBUG=y); tier-1 qemu boot at
   the tip.
 - The series passes the Xen GitLab CI pipeline, including the
   hardware runner:
    https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/2776674713
 - On an Intel NUC, debug build: XTF pv64 and pv32pae suites (the
   latter with cet=no-shstk,no-ibt pv=32, since CET disables PV32);
   plus an LDT exerciser in a PV Linux guest (modify_ldt() with 1-16
   page LDTs, demand-faulting every page, LAR beyond the limit,
   shrinking, teardown; also with the guest's vCPUs bounced across
   pCPUs) -- thousands of rounds, no assertions or "unable to map"
   reports.

[1] https://lore.kernel.org/xen-devel/20240726152206.28411-1-roger.pau@citrix.com/
[2] https://lore.kernel.org/xen-devel/20250108142659.99490-1-roger.pau@citrix.com/

George Dunlap (1):
  x86/mm: allocate the per-domain page-tables from the xenheap

Roger Pau Monné (6):
  x86/mm: introduce populate_perdomain_mapping()
  x86/pv: use populate_perdomain_mapping() to map the Xen GDT
  x86/pv: set/clear guest GDT mappings using
    populate_perdomain_mapping()
  x86/pv: update guest LDT mappings using
    {populate,destroy}_perdomain_mapping()
  x86/pv: remove stashing of GDT/LDT L1 page-tables
  x86/mm: simplify create_perdomain_mapping() interface

 xen/arch/x86/domain.c               |  17 ++-
 xen/arch/x86/domain_page.c          |  10 +-
 xen/arch/x86/hvm/hvm.c              |   2 +-
 xen/arch/x86/include/asm/desc.h     |   6 +-
 xen/arch/x86/include/asm/domain.h   |  14 +-
 xen/arch/x86/include/asm/mm.h       |   9 +-
 xen/arch/x86/mm.c                   | 225 +++++++++++++++++-----------
 xen/arch/x86/pv/descriptor-tables.c |  57 ++++---
 xen/arch/x86/pv/domain.c            |  16 +-
 xen/arch/x86/pv/mm.c                |  16 +-
 xen/arch/x86/smpboot.c              |  14 +-
 xen/arch/x86/traps.c                |   4 +-
 xen/arch/x86/x86_64/mm.c            |   3 +-
 13 files changed, 221 insertions(+), 172 deletions(-)


base-commit: 669f8c502aeaa538f8407013ba04461e71361ed4
-- 
2.55.0

On 20.08.2026 19:43, George Dunlap wrote:
> One point reviewers may want to look at specifically: patch 1 changes
> where the per-domain page-tables are allocated from, and its commit
> message discusses the (minor) NUMA-placement consequence.

While I don't recall which recent patch (series) it was, I can't very well
say "no new xenheap allocations please" there without also saying so here.
I've read over patch 1's description, and while it tries to justify this
accordingly, I still remain concerned. I think we simply have to accept
the mapping overhead, to avoid allocating from a pool which - over time -
is representing a decreasing portion of total memory systems have (on
average, and not even considering systems with extremely sparse memory
layouts, and with perhaps PDX compression not doing good enough to
compensate).

Jan

On Fri, Aug 21, 2026 at 9:45 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 20.08.2026 19:43, George Dunlap wrote:
> > One point reviewers may want to look at specifically: patch 1 changes
> > where the per-domain page-tables are allocated from, and its commit
> > message discusses the (minor) NUMA-placement consequence.
>
> While I don't recall which recent patch (series) it was, I can't very well
> say "no new xenheap allocations please" there without also saying so here.
> I've read over patch 1's description, and while it tries to justify this
> accordingly, I still remain concerned. I think we simply have to accept
> the mapping overhead, to avoid allocating from a pool which - over time -
> is representing a decreasing portion of total memory systems have (on
> average, and not even considering systems with extremely sparse memory
> layouts, and with perhaps PDX compression not doing good enough to
> compensate).

You should certainly have the same resistance to adding new xenheap
allocations.  But looking at the numbers, I don't see that we're
anywhere near the point where we say, "Absolutely no new xenheap
allocations, regardless of the cost."  domheap+vmap looks like it was
cheap and easy alternative for Teddy, but the alternatives here
aren't, compared to the cost of extra xenheap allocations.

So let's lay everything out.

My understanding is that we have the following two issues allocating
things in the xenheap on systems larger than 4T:

- The total amount of xenheap space is limited to 4 TiB of virtual
  address space.  On some systems, this may correspond to 4 TiB of
  actual RAM; but on a machine whose RAM layout is sparser, the
  actual RAM addressable in this window may be far less

- It's not symmetric NUMA-wise; so the larger the system, the more of
  the xenheap will end up being from the same NUMA node.  This will
  limit Xen's ability to have NUMA-local data structures, and its
  ability to give NUMA-local data to guests running on node 0.

Looking at this series as a whole, although the first patch adds pages
to the xenheap, the end goal of the rest of the work is to remove
pages from the xenheap. Things added in:

- Making the perdomain area per-vCPU, with its pagetables allocated
  from the xenheap, adds a per-vCPU L3 plus an L2+L1 pair for each
  slot in use.  This totals 5 pages/vCPU for HVM guests and 8 pages/vCPU
  for PV guests.

  (Note that the GDT/LDT L1s are already allocated from the xenheap
  today, but per-domain rather than per-vCPU.)

Things removed:

- Per-pCPU stacks -- 8 xenheap pages / pCPU

- AMD VMCB - one xenheap page / vCPU

- VMX guest MSR area: 1 page per vCPU

- sub-page XSAVE areas (~2.7 KiB/vCPU of xmalloc pool today; planned
  follow-on work aggregating other miscellaneous xmalloc'd guest state
  should take this to about a page per vCPU)

To do some math: current security-supported limits for x86 are 4096
pCPUs on a 12TiB system.  Suppose we have an 8:1 vCPU:pCPU ratio, and
an average of 8 vcpus per domain.  So 32768 total vCPUs and 4096
domains.  On a Full ASI system, vcpu-pt on all domains, per-CPU stacks
on, all Intel HVM domains, we get numbers like the following:

Added to xenheap:

- Per-vCPU tables, 5/vCPU (L3; mapcache L2+L1; state-window L2+L1):
  5 × 32,768 = 163,840 pages = 640 MiB
- Per-pCPU stack tables, 2/pCPU: 2 × 4,096 = 8,192 pages = 32 MiB
  (→ 0: these are only written at CPU bring-up and tear-down, so we
  have already moved them to the domheap in the working branch --
  which also makes them NUMA-local unconditionally)
- Per-domain tables: replaced by the per-vCPU sets in vcpu-pt mode → 0
- Total added: 172,032 pages = 672 MiB

Removed from xenheap:

- Stacks, 8/pCPU: 8 × 4,096 = 32,768 pages = 128 MiB
- XSAVE, ~2.7 KiB/vCPU from the xmalloc pools: 32,768 × 2.7 KiB ≈
  21,600 pages ≈ 86 MiB (0 if guests get AMX — those areas are domheap
  today)
- VMX guest MSR page: lazily allocated, typically absent → 0 (upper
  bound 128 MiB if every vCPU used one)
- Total removed: ≈ 54,400 pages ≈ 214 MiB

Net: +117,600 pages ~ +458 MiB — against a 4 TiB window (0.011%), on
a 12 TiB host (0.0036%).

An AMD HVM fleet would include VMCB removal (32,768 pages = 128 MiB) →
net +330 MiB. A PV fleet is the worst case — 8 tables/vCPU (add
GDT/LDT L2+L1 and the per-vCPU root) → 1,056 MiB added, 214 removed,
net +842 MiB.

I have explored a number of other options, to various levels of depth.

One is map_domain_page_irqoff(): If the caller promises to keep
interrupts disabled until unmap_domain_page_irqoff(), we can safely
perform maps in a context switch without having to worry about
sync_lazy_execstate.  (This was actually implemented and almost sent
on Tuesday evening, when I noticed your review of Roger's v2 saying,
"Question is whether it's a good idea in the first place to start
using map_domain_page() from the context switch path.  Surely there
are possible alternatives.")  This maps all vcpu pages from the
domheap, adding nothing to the xenheap *or* the vmap area.  But it
costs 9 map/unmap pairs *per context switch*.

I absolutely reject the idea that because on a 12TiB system with 32k
PV vCPUs, we take up an extra 0.02% of the xenheap area, that a laptop
running QubesOS has to do 9 maps and unmaps per context switch.  That
is not a valid cost/benefits tradeoff.  In the worst case we could
just add a switch to such a system, allowing people who find their
xenheap too full to use the mapcache version instead.  (We could even
turn this on automatically at boot based on projected xenheap
utilization.)

There are other options I've explored:

- domheap + vmap; basically, allocate from domheap, map in the vmap
  area.  On paper this sounds like the same thing; the problem is that
  we don't have a simple MFN -> VA mapping, as we do in the xenheap
  case, so the walk is a lot harder; we start to have to do lookups,
  significantly increasing the cost over simple memory reads and math.
  (This is the difference from the intremap table on the VT-d thread:
  that's a leaf structure reached from a single pointer, so a
  permanent vmap costs nothing there.  Pagetable hierarchies are
  exactly the case where the MFN -> VA step is critical: each entry
  read yields an MFN, which the walk has to turn into the next VA.)
  And if we're concerned about "xenheap creep", when we have a 4 TiB
  ceiling, shouldn't we also be worried about "vmap creep", when we
  have a 64 GiB ceiling?

- Stash everything we need; basically, an extension of the current
  gdt_ldt_l1tab functionality.  Allocate everything from the domheap,
  map it in the vmap area (moving gdt_ldt_l1tab there as well), keep
  pointers to all the things we need to modify on context switch, so
  we don't need to walk the tables.  This would basically be, three
  pointers per vCPU: a pointer to its GDT/LDT L1, a pointer to its
  per-vCPU L3, and a pointer to the per-vCPU root_pgt.  (This would
  put ~384 MiB of mappings into the 64 GiB vmap region -- 0.6%, shared
  with ioremap and the fixmap -- to avoid 0.02% of the xenheap
  window.)

Both the vmap options have two complications, compared to the posted
option.  One thing to worry about here would be the additional stress
on the vmap allocator: It's a linear bitmap scan under one global
lock, designed for dozens-to-hundreds of ioremaps, not ~100k
long-lived single-page mappings (32k vCPUs x 3 pages per vCPU in the
"stash everything" case).

The second is that we begin to run into bootstrapping issues.  With
the xenheap approach, we can begin building and walking pagetables
very early in boot in the same manner in which they'll be walked
throughout Xen's lifecycle.  With the vmap approach, we need to deal
with the fact that the vmap area itself isn't up until later.

The final option I looked at was mapping the incoming vcpu's linear
map to edit it ("altlinmap").  That still adds a map/unmap per context
switch, and requires some additional complication to handle
ASI/non-ASI systems.

Xen already consistently allocates its page tables from the xenheap
whenever it needs to access them during a context switch:
alloc_xen_pagetable() has allocated from the domheap since Hongyan's
directmap-removal preparation (those tables are only ever walked in
contexts where map_domain_page() works), but XPTI's per-CPU root_pgt
is alloc_xenheap_page(), precisely because it has to be written on the
context-switch path.  The same for the PV GDT / LDT L1 tables.  The
series follows the same rule for the same reason.

Ultimately, I think there's a lot of wisdom in the saying, "Premature
optimization is the root of all evil."  As I said, it's certainly
right to be on our guard against adding things to xenheap, and look at
alternatives; but we're nowhere near the point where we need to say,
"Absolutely nothing added, regardless of the cost."  The design here
is not locking us into the pages long-term; alternate designs have a
significant cost in terms of authoring, reviewing, code complexity and
maintenance, and code performance.  At such time as we find systems
where the xenheap allocations introduced in this series become a
problem, we have a number of potential ways to mitigate the problem,
including switching to mapcache *on systems with the problem*, or
switching to a number of the other more complicated approaches.

 -George

On 21.08.2026 17:17, George Dunlap wrote:
> On Fri, Aug 21, 2026 at 9:45 AM Jan Beulich <jbeulich@suse.com> wrote:
>> On 20.08.2026 19:43, George Dunlap wrote:
>>> One point reviewers may want to look at specifically: patch 1 changes
>>> where the per-domain page-tables are allocated from, and its commit
>>> message discusses the (minor) NUMA-placement consequence.
>>
>> While I don't recall which recent patch (series) it was, I can't very well
>> say "no new xenheap allocations please" there without also saying so here.
>> I've read over patch 1's description, and while it tries to justify this
>> accordingly, I still remain concerned. I think we simply have to accept
>> the mapping overhead, to avoid allocating from a pool which - over time -
>> is representing a decreasing portion of total memory systems have (on
>> average, and not even considering systems with extremely sparse memory
>> layouts, and with perhaps PDX compression not doing good enough to
>> compensate).
> 
> You should certainly have the same resistance to adding new xenheap
> allocations.  But looking at the numbers, I don't see that we're
> anywhere near the point where we say, "Absolutely no new xenheap
> allocations, regardless of the cost."

Well, that depends, and in part on the longer term plans with ASI. It has
been my (silent) assumption that eventually the directmap would go away
altogether when ASI is in use, with the VA space freed (almost?) all
becoming available for vmap(). With the disappearance of directmap, the
xenheap would naturally disappear as well. Hence putting stuff there in
new work actually adds to our technical debt.

>  domheap+vmap looks like it was
> cheap and easy alternative for Teddy, but the alternatives here
> aren't, compared to the cost of extra xenheap allocations.
> 
> So let's lay everything out.
> 
> My understanding is that we have the following two issues allocating
> things in the xenheap on systems larger than 4T:
> 
> - The total amount of xenheap space is limited to 4 TiB of virtual
>   address space.  On some systems, this may correspond to 4 TiB of
>   actual RAM; but on a machine whose RAM layout is sparser, the
>   actual RAM addressable in this window may be far less
> 
> - It's not symmetric NUMA-wise; so the larger the system, the more of
>   the xenheap will end up being from the same NUMA node.  This will
>   limit Xen's ability to have NUMA-local data structures, and its
>   ability to give NUMA-local data to guests running on node 0.
> 
> Looking at this series as a whole, although the first patch adds pages
> to the xenheap, the end goal of the rest of the work is to remove
> pages from the xenheap. Things added in:
> 
> - Making the perdomain area per-vCPU, with its pagetables allocated
>   from the xenheap, adds a per-vCPU L3 plus an L2+L1 pair for each
>   slot in use.  This totals 5 pages/vCPU for HVM guests and 8 pages/vCPU
>   for PV guests.
> 
>   (Note that the GDT/LDT L1s are already allocated from the xenheap
>   today, but per-domain rather than per-vCPU.)
> 
> Things removed:
> 
> - Per-pCPU stacks -- 8 xenheap pages / pCPU
> 
> - AMD VMCB - one xenheap page / vCPU
> 
> - VMX guest MSR area: 1 page per vCPU
> 
> - sub-page XSAVE areas (~2.7 KiB/vCPU of xmalloc pool today; planned
>   follow-on work aggregating other miscellaneous xmalloc'd guest state
>   should take this to about a page per vCPU)
> 
> To do some math: current security-supported limits for x86 are 4096
> pCPUs on a 12TiB system.  Suppose we have an 8:1 vCPU:pCPU ratio, and
> an average of 8 vcpus per domain.  So 32768 total vCPUs and 4096
> domains.  On a Full ASI system, vcpu-pt on all domains, per-CPU stacks
> on, all Intel HVM domains, we get numbers like the following:
> 
> Added to xenheap:
> 
> - Per-vCPU tables, 5/vCPU (L3; mapcache L2+L1; state-window L2+L1):
>   5 × 32,768 = 163,840 pages = 640 MiB
> - Per-pCPU stack tables, 2/pCPU: 2 × 4,096 = 8,192 pages = 32 MiB
>   (→ 0: these are only written at CPU bring-up and tear-down, so we
>   have already moved them to the domheap in the working branch --
>   which also makes them NUMA-local unconditionally)
> - Per-domain tables: replaced by the per-vCPU sets in vcpu-pt mode → 0
> - Total added: 172,032 pages = 672 MiB
> 
> Removed from xenheap:
> 
> - Stacks, 8/pCPU: 8 × 4,096 = 32,768 pages = 128 MiB
> - XSAVE, ~2.7 KiB/vCPU from the xmalloc pools: 32,768 × 2.7 KiB ≈
>   21,600 pages ≈ 86 MiB (0 if guests get AMX — those areas are domheap
>   today)
> - VMX guest MSR page: lazily allocated, typically absent → 0 (upper
>   bound 128 MiB if every vCPU used one)
> - Total removed: ≈ 54,400 pages ≈ 214 MiB
> 
> Net: +117,600 pages ~ +458 MiB — against a 4 TiB window (0.011%), on
> a 12 TiB host (0.0036%).

While these percentiles in particular of course look very tiny, they are
applicable only on systems having no meaningful gaps in the physical
address map. And even more generally I find all of these calculations
only partly convincing, not the least because you start out from numbers
which look pretty contrived when comparing to actual systems which would
run the new code. (Using more realistic real-system values may end up
going in favor of what you want to convey, or it may not.)

> I have explored a number of other options, to various levels of depth.
> 
> One is map_domain_page_irqoff(): If the caller promises to keep
> interrupts disabled until unmap_domain_page_irqoff(), we can safely
> perform maps in a context switch without having to worry about
> sync_lazy_execstate.  (This was actually implemented and almost sent
> on Tuesday evening, when I noticed your review of Roger's v2 saying,
> "Question is whether it's a good idea in the first place to start
> using map_domain_page() from the context switch path.  Surely there
> are possible alternatives.")  This maps all vcpu pages from the
> domheap, adding nothing to the xenheap *or* the vmap area.  But it
> costs 9 map/unmap pairs *per context switch*.

But why would not using vmap() be a necessary conclusion of my initial
comment? All I'm objecting to are new uses of the xenheap.

> I absolutely reject the idea that because on a 12TiB system with 32k
> PV vCPUs, we take up an extra 0.02% of the xenheap area, that a laptop
> running QubesOS has to do 9 maps and unmaps per context switch.  That
> is not a valid cost/benefits tradeoff.  In the worst case we could
> just add a switch to such a system, allowing people who find their
> xenheap too full to use the mapcache version instead.  (We could even
> turn this on automatically at boot based on projected xenheap
> utilization.)

Maybe, yet extra overhead may be a necessary (but hopefully only
transient) price to pay in the course of the transformation.

> There are other options I've explored:
> 
> - domheap + vmap; basically, allocate from domheap, map in the vmap
>   area.  On paper this sounds like the same thing; the problem is that
>   we don't have a simple MFN -> VA mapping, as we do in the xenheap
>   case, so the walk is a lot harder; we start to have to do lookups,
>   significantly increasing the cost over simple memory reads and math.
>   (This is the difference from the intremap table on the VT-d thread:
>   that's a leaf structure reached from a single pointer, so a
>   permanent vmap costs nothing there.  Pagetable hierarchies are
>   exactly the case where the MFN -> VA step is critical: each entry
>   read yields an MFN, which the walk has to turn into the next VA.)

The pages used here are entirely private to logic handling those page
tables. Hence a struct page_info field can very likely be used to stash
the VA of a permanent mapping. (Feels like similarly I must have
suggested this somewhere else recently, yet I don't recall the context.)

>   And if we're concerned about "xenheap creep", when we have a 4 TiB
>   ceiling, shouldn't we also be worried about "vmap creep", when we
>   have a 64 GiB ceiling?

Absolutely, and I have been mentioning the need to consider growing this
area in a number of situations (one iirc again pretty recently).

> - Stash everything we need; basically, an extension of the current
>   gdt_ldt_l1tab functionality.  Allocate everything from the domheap,
>   map it in the vmap area (moving gdt_ldt_l1tab there as well), keep
>   pointers to all the things we need to modify on context switch, so
>   we don't need to walk the tables.  This would basically be, three
>   pointers per vCPU: a pointer to its GDT/LDT L1, a pointer to its
>   per-vCPU L3, and a pointer to the per-vCPU root_pgt.  (This would
>   put ~384 MiB of mappings into the 64 GiB vmap region -- 0.6%, shared
>   with ioremap and the fixmap -- to avoid 0.02% of the xenheap
>   window.)
> 
> Both the vmap options have two complications, compared to the posted
> option.  One thing to worry about here would be the additional stress
> on the vmap allocator: It's a linear bitmap scan under one global
> lock, designed for dozens-to-hundreds of ioremaps, not ~100k
> long-lived single-page mappings (32k vCPUs x 3 pages per vCPU in the
> "stash everything" case).

Indeed, heavier use of that allocator may require work to be done there.

> The second is that we begin to run into bootstrapping issues.  With
> the xenheap approach, we can begin building and walking pagetables
> very early in boot in the same manner in which they'll be walked
> throughout Xen's lifecycle.  With the vmap approach, we need to deal
> with the fact that the vmap area itself isn't up until later.

Valid concern, yet surely possible to deal with.

> The final option I looked at was mapping the incoming vcpu's linear
> map to edit it ("altlinmap").  That still adds a map/unmap per context
> switch, and requires some additional complication to handle
> ASI/non-ASI systems.
> 
> Xen already consistently allocates its page tables from the xenheap
> whenever it needs to access them during a context switch:
> alloc_xen_pagetable() has allocated from the domheap since Hongyan's
> directmap-removal preparation (those tables are only ever walked in
> contexts where map_domain_page() works), but XPTI's per-CPU root_pgt
> is alloc_xenheap_page(), precisely because it has to be written on the
> context-switch path.  The same for the PV GDT / LDT L1 tables.  The
> series follows the same rule for the same reason.

"Rule" is a strong word. XPTI at the time needed to be done quickly.
The inability to map_domain_page() from the context switch path left
xenheap as the only viable option. Whereas with ASI, as said at the
top, phasing out directmap (and hence xenheap) as a concept is (imo) a
mid- to long-term goal.

> Ultimately, I think there's a lot of wisdom in the saying, "Premature
> optimization is the root of all evil."  As I said, it's certainly
> right to be on our guard against adding things to xenheap, and look at
> alternatives; but we're nowhere near the point where we need to say,
> "Absolutely nothing added, regardless of the cost."  The design here
> is not locking us into the pages long-term; alternate designs have a
> significant cost in terms of authoring, reviewing, code complexity and
> maintenance, and code performance.  At such time as we find systems
> where the xenheap allocations introduced in this series become a
> problem, we have a number of potential ways to mitigate the problem,
> including switching to mapcache *on systems with the problem*, or
> switching to a number of the other more complicated approaches.

I'm a little puzzled by you talking of "optimization" (premature or
not) here. In my initial reply I did point out a functional aspect, and
I made clear that I'm aware that this is going to have a performance
impact. I.e. quite the opposite of "optimization".

Jan

On Mon, Aug 24, 2026 at 10:02 AM Jan Beulich <jbeulich@suse.com> wrote:
> On 21.08.2026 17:17, George Dunlap wrote:
> > On Fri, Aug 21, 2026 at 9:45 AM Jan Beulich <jbeulich@suse.com> wrote:
> >> On 20.08.2026 19:43, George Dunlap wrote:
> >>> One point reviewers may want to look at specifically: patch 1 changes
> >>> where the per-domain page-tables are allocated from, and its commit
> >>> message discusses the (minor) NUMA-placement consequence.
> >>
> >> While I don't recall which recent patch (series) it was, I can't very well
> >> say "no new xenheap allocations please" there without also saying so here.
> >> I've read over patch 1's description, and while it tries to justify this
> >> accordingly, I still remain concerned. I think we simply have to accept
> >> the mapping overhead, to avoid allocating from a pool which - over time -
> >> is representing a decreasing portion of total memory systems have (on
> >> average, and not even considering systems with extremely sparse memory
> >> layouts, and with perhaps PDX compression not doing good enough to
> >> compensate).
> >
> > You should certainly have the same resistance to adding new xenheap
> > allocations.  But looking at the numbers, I don't see that we're
> > anywhere near the point where we say, "Absolutely no new xenheap
> > allocations, regardless of the cost."
>
> Well, that depends, and in part on the longer term plans with ASI. It has
> been my (silent) assumption that eventually the directmap would go away
> altogether when ASI is in use, with the VA space freed (almost?) all
> becoming available for vmap(). With the disappearance of directmap, the
> xenheap would naturally disappear as well. Hence putting stuff there in
> new work actually adds to our technical debt.

I do think that makes sense as a long-term goal.  Actually, I asked
Fable to do an audit of xenheap allocations, asking it to classify
them as to whether they needed xenheap's key properties (easy mfn <->
va conversion, available in early boot), and it reckoned only about 4%
of the allocations (by volume) in the example system I did numbers for
needed xenheap-specific properties.  Lots of things just need *some*
global mapping somewhere, and would probably actually overall benefit
from being moved to the vmap, so they wouldn't be restricted to a
single NUMA node.  Grant frames were the biggest chunk.

The xenheap / vmap split is certainly something I would now consider a
large piece of technical debt; moving towards paying that off is
certainly something I think worth achieving, provided the rest of the
maintainers are on board.

(Imagine how differently this conversation would have gone, if in the
first email you had said, "Actually, I thought one of the main goals
of this series was to get rid of the xenheap altogether, so that we
could switch to having a single large vmap area instead?")

> While these percentiles in particular of course look very tiny, they are
> applicable only on systems having no meaningful gaps in the physical
> address map. And even more generally I find all of these calculations
> only partly convincing, not the least because you start out from numbers
> which look pretty contrived when comparing to actual systems which would
> run the new code. (Using more realistic real-system values may end up
> going in favor of what you want to convey, or it may not.)

To be honest, I'm inclined to think that they're not very convincing
because you don't actually have an idea what the problem is.  You
didn't specify what you were worried about, so I tried to guess a
scenario that I considered 95th-percentile worse case.  I don't know
what kinds of sparse memory layout machines you have in mind -- are
they written down anywhere, so that contributors can read and
understand what they need to consider *before* implementing?  Even now
you haven't even said what about my scenario you consider unrealistic,
much less told me parameters you think are more realistic.

I don't even know exactly what failure mode you're worried about.  Two
kinds of potential failures I know about:
 - Performance impacted because pages can't be NUMA-local
 - Toolstack operations (including domain creation) fail because
xenheap has been exhausted.

If you're willing to accept 9 map/unmap operations on *all* systems,
then NUMA-non-local accesses can't be that big of an issue for you.

On the fairly largish system / load that I tried to estimate, the
total xenheap usage was less than 3GiB in the worst case.  Let's
double that just for safety sake: Do there exist systems whose memory
is so sparse that even with PDX compression, they can't even scrounge
together 6GiB below the 4TiB limit?  If so, I think a much better
solution would be to document that such systems may be able to support
a lower degree of oversubscribing than most systems, and leave it at
that.

In short: I can't imagine a scenario where a larger xenheap is an
issue we should be concerned about.

It's not up to me to guess what sorts of numbers would allay your
concern.  If you want me to consider a large xenheap to be a problem
on its own, it is now your job to articulate, first, at least one
target system (hardware and configuration) you think would be
problematic;  and secondly, exactly what bad thing you're worried
about happening.  Only then do I have any hope of addressing your
concerns.  Until that time, I don't consider "the xenheap is getting
too large" objection to be valid.

Objections I will consider:
- xenheap has poorer NUMA locality
- The xenheap/vmap split is a big ugly unnecessary bit of technical
debt; Xen would be far better if we could get rid of the xenheap
altogether.  Every additional user of xenheap is another patch in a
series converting xenheap to vmap.

> > One is map_domain_page_irqoff(): If the caller promises to keep
> > interrupts disabled until unmap_domain_page_irqoff(), we can safely
> > perform maps in a context switch without having to worry about
> > sync_lazy_execstate.  (This was actually implemented and almost sent
> > on Tuesday evening, when I noticed your review of Roger's v2 saying,
> > "Question is whether it's a good idea in the first place to start
> > using map_domain_page() from the context switch path.  Surely there
> > are possible alternatives.")  This maps all vcpu pages from the
> > domheap, adding nothing to the xenheap *or* the vmap area.  But it
> > costs 9 map/unmap pairs *per context switch*.
>
> But why would not using vmap() be a necessary conclusion of my initial
> comment? All I'm objecting to are new uses of the xenheap.

I'm trying to list all the advantages and disadvantages of the various
options I've explored.  You've agreed that growing the vmap region is
*also* something we need to worry about; and that the vmap allocator
may not be ready to become a performance-critical part of the system.
Furthermore, as Roger pointed out privately, regardless of where the
global mapping lives (vmap or sparsely-mapped xenheap), having a
global mapping at all means global TLB flushes whenever we destroy a
vCPU; and in any case, in principle we'd like to avoid exposing any
data whatsoever.  Using the mapcache avoids all those problems, for a
different cost.

> > Ultimately, I think there's a lot of wisdom in the saying, "Premature
> > optimization is the root of all evil."
> > ...
> I'm a little puzzled by you talking of "optimization" (premature or
> not) here. In my initial reply I did point out a functional aspect, and
> I made clear that I'm aware that this is going to have a performance
> impact. I.e. quite the opposite of "optimization".

"Optimize" in the terms of "improve", not necessarily in terms of cycle count.

The point of the principle is to say this:  First, build it correctly,
in a way that is simple, clear, robust, and easy to write, review, and
maintain.  *Then*, after you've measured that there is a problem,
where there is a problem, and so on, should you put in extra effort
and add extra complication, only to areas where you know there will be
some material benefit.

You've argued that we should avoid using xenheap because it will have
some negative impacts on large systems with sparse memory layouts; in
other words, you're asking me to *optimize* for those use cases, at
the expense of more typical systems.  In isolation, this principle
would say: take the xenheap option first, as it's clean and fast in
the common case, and measure it on a target systems (or at least,
estimate what the impact would be based on modeling).  Once you have
reason to believe there will be a problem, then introduce code
complications based on the actual issue you find.

> > There are other options I've explored:
> >
> > - domheap + vmap; basically, allocate from domheap, map in the vmap
> >   area.  On paper this sounds like the same thing; the problem is that
> >   we don't have a simple MFN -> VA mapping, as we do in the xenheap
> >   case, so the walk is a lot harder; we start to have to do lookups,
> >   significantly increasing the cost over simple memory reads and math.
> >   (This is the difference from the intremap table on the VT-d thread:
> >   that's a leaf structure reached from a single pointer, so a
> >   permanent vmap costs nothing there.  Pagetable hierarchies are
> >   exactly the case where the MFN -> VA step is critical: each entry
> >   read yields an MFN, which the walk has to turn into the next VA.)
>
> The pages used here are entirely private to logic handling those page
> tables. Hence a struct page_info field can very likely be used to stash
> the VA of a permanent mapping. (Feels like similarly I must have
> suggested this somewhere else recently, yet I don't recall the context.)

This is an interesting idea, particularly for a full xenheap -> vmap
change.  Probably too complicated for this series (see below).

> Absolutely, and I have been mentioning the need to consider growing this
> area in a number of situations (one iirc again pretty recently).
...
> Indeed, heavier use of that allocator may require work to be done there.
...
> Valid concern, yet surely possible to deal with.

One thing you do need to consider:  There are at least 45 patches to
get to the most basic form of extra security (no direct-map, FPU/XSAVE
moved to domheap); and another 13 after that to move to per-cpu
stacks.  I'm engaged until November to work on this.  If we don't have
significant progress by then, there may be no ASI at all (at least for
a long time), and thus no hope of getting rid of the xenheap.  If
every batch of 7 patches takes a month to get through, we're not going
to be anywhere close by November.  So you need to be strategic about
what kinds of additional work you ask me to do: what does a solution
look like that is both technically acceptable, and achieves measurable
progress by November?

If we had all the time in the world, we could consider trying to
convert the entire xenheap to vmap as the first step.  (Even on the
fairly large system I tried to describe, the xenheap was only around
3GiB; still plenty of room in the vmap area to get us by until the
direct map is gone.)  I don't think that's really viable, as there's
quite a long tail of allocations that would probably end up being
haggled over before we even began the ASI series itself.

So let's try to take stock.  We can't safely remove the direct-map
unless we have per-vcpu mapcaches.  We can't really say we've isolated
the system while all pCPU stacks, with random bits of guest state, are
visible to all other pCPUs. We can't have per-vcpu mapcaches or
per-CPU stack maps unless we have per-cpu root pagetables for PV
guests.  Both require modifying per-pCPU bits of pagetables of the
incoming vCPU on a context switch.

We have four ways of mapping in general: xenheap, vmap, mapcache, or
(for the pagetables) the linear map.

For the first three, we have several different ways of arriving at the
entries.  Both Roger's v2 and my v1 start at the top and walk down the
pagetables.  For the mapcache, this seems relatively heavy.  I thought
xenheap would be just simple math, but with PDX on all the time,
that's more expensive than it looks.  vmap would require looking into
stashing a pointer into an unused (by xenheap pages) portion of the
struct page_info.

But the other approach is to stash references to just the page we need
to modify -- basically, rather than get rid of gdt_ldt_l1tab, add two
more instances.  For xenheap or vmap, this would be pointers to the
virtual addresses; but it's also possible to do in the mapcache
version, by stashing the mfn of the exact table we need to map.  In
all cases, for the context switch, it's just three references.

Both global-mapping options expose Xen pagetables.  We've agreed these
are not sensitive, but also in general our posture is that we
shouldn't reveal anything unless it buys us something.  They also both
require host-wide TLB shootdowns on vCPU tear-down.

In the spirit of "measure before optimizing", I did some tests of the
mapcache-walk variant.  On my NUC, at the end of the series, I get:

- baseline: 1480 cycles / context switch
- xenheap-walk: 2460 cycles / context switch
- mapcache-walk: 6440 cycles / context switch

By default Xen has a context switch rate limit of 1ms, so the
difference isn't measurable.  If you disable the ratelimit and do a
"ping flood" microbenchmark, the mapcache-walk reduces performance by
a whopping 70% (41k pings per second -> 12k pings per second).

If it weren't for the general intent to move away form xenheap, I'd
argue more strenuously that we should take the series as I've posted
it.  As it stands, I think the performance of mapcache-walk is
acceptable enough for a first cut, particularly given that we have two
potential optimizations already (caching MFNs rather than walking
pagetables as an easy option, switching to vmap as a slightly more
complicated one).

It's annoying that Tuesday evening I didn't know that you hated
xenheap allocations, and had only a mild distaste for mapping in a
context switch, or I might have implemented vmap instead.  At any
rate, I'll move forward with mapcache-walk, which we can later look at
optimizing by stashing the relevant MFNs so we can avoid the walk.

If anyone doesn't like that, let me know sooner rather than later, so
we can avoid wasting more time.

 -George

 -George

On 25.08.2026 13:42, George Dunlap wrote:
> On Mon, Aug 24, 2026 at 10:02 AM Jan Beulich <jbeulich@suse.com> wrote:
>> While these percentiles in particular of course look very tiny, they are
>> applicable only on systems having no meaningful gaps in the physical
>> address map. And even more generally I find all of these calculations
>> only partly convincing, not the least because you start out from numbers
>> which look pretty contrived when comparing to actual systems which would
>> run the new code. (Using more realistic real-system values may end up
>> going in favor of what you want to convey, or it may not.)
> 
> To be honest, I'm inclined to think that they're not very convincing
> because you don't actually have an idea what the problem is.  You
> didn't specify what you were worried about, so I tried to guess a
> scenario that I considered 95th-percentile worse case.  I don't know
> what kinds of sparse memory layout machines you have in mind -- are
> they written down anywhere, so that contributors can read and
> understand what they need to consider *before* implementing?  Even now
> you haven't even said what about my scenario you consider unrealistic,
> much less told me parameters you think are more realistic.

What I specifically considered unrealistic is that you use huge pCPU and
vCPU counts. Yes, you're trying to do a worst case estimate, yet at the
same time you're assuming huge amounts of memory to be available (which
doesn't represent a "worst case").

As to sparse layouts - ones which have led to the two forms of PDX
compression are well known (I think). The need for more recent (offset)
form is a good example of what could go wrong here: New machines can
always come with new layouts, potentially requiring new compressions
approaches. So what I'm concerned about is effectively _any_ sparse
layout that we may encounter without having a suitable PDX compression
method readily available.

> I don't even know exactly what failure mode you're worried about.  Two
> kinds of potential failures I know about:
>  - Performance impacted because pages can't be NUMA-local
>  - Toolstack operations (including domain creation) fail because
> xenheap has been exhausted.

One thing I can't help thinking you keep overlooking throughout your
reply: xenheap and domheap aren't separate. There being only a
relatively small part of it needed for the worst case estimate you did
means nothing as to exhausting the xenheap in practice: Almost the
entirety of it (with the DMA reserve being somewhat protected) can be
used to build domains. Once in that state, allocations would fail no
matter that large swathes of domheap might (have become) available
(again).

That said, with what you indicated at the very bottom of your reply,
it looks like this part of the discussion has become largely moot.

Jan

On Tue, Aug 25, 2026 at 2:28 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 25.08.2026 13:42, George Dunlap wrote:
> > On Mon, Aug 24, 2026 at 10:02 AM Jan Beulich <jbeulich@suse.com> wrote:
> >> While these percentiles in particular of course look very tiny, they are
> >> applicable only on systems having no meaningful gaps in the physical
> >> address map. And even more generally I find all of these calculations
> >> only partly convincing, not the least because you start out from numbers
> >> which look pretty contrived when comparing to actual systems which would
> >> run the new code. (Using more realistic real-system values may end up
> >> going in favor of what you want to convey, or it may not.)
> >
> > To be honest, I'm inclined to think that they're not very convincing
> > because you don't actually have an idea what the problem is.  You
> > didn't specify what you were worried about, so I tried to guess a
> > scenario that I considered 95th-percentile worse case.  I don't know
> > what kinds of sparse memory layout machines you have in mind -- are
> > they written down anywhere, so that contributors can read and
> > understand what they need to consider *before* implementing?  Even now
> > you haven't even said what about my scenario you consider unrealistic,
> > much less told me parameters you think are more realistic.
>
> What I specifically considered unrealistic is that you use huge pCPU and
> vCPU counts. Yes, you're trying to do a worst case estimate, yet at the
> same time you're assuming huge amounts of memory to be available (which
> doesn't represent a "worst case").

Let me point out that you still haven't named exact numbers -- you're
still offloading that to me to try to guess or imagine.

The v1 series I posted adds a few pages per vCPU and a few pages per
pCPU into the xenheap. The problem is using up too much of the xenheap
address space.  So obviously to make a reasonable worst-case that
you're not going to dismiss as contrived, I need to maximize my pCPU
count and vCPU count.  pCPUs is easy -- we're documented as supporting
4096.  How many is a reasonable number of domains and vcpus?  Well, in
general, pCPUs are an effective limit to how many vCPUs you have total
on the system; an 8:1 vCPU overcommit is high, but not preposterously
high.

I don't understand your point about huge amounts of memory.  If you're
talking about *total RAM used*, it doesn't matter whether it comes
from the domheap or the xenheap.  The only possible reason to say
domheap is OK but xenheap is not is if you're concerned about RAM
above the 4TiB boundary.  Which can only happen on system with large
amounts of RAM, or systems with really sparse memory layouts.  Does
the analysis really change at all whether you're using 12TiB or 6TiB?

> As to sparse layouts - ones which have led to the two forms of PDX
> compression are well known (I think). The need for more recent (offset)
> form is a good example of what could go wrong here: New machines can
> always come with new layouts, potentially requiring new compressions
> approaches. So what I'm concerned about is effectively _any_ sparse
> layout that we may encounter without having a suitable PDX compression
> method readily available.

"There may be some new layout that doesn't compress well" -- it's not
uncommon for random bits of new hardware not to work well until we
supply a patch to fix it.  The position you're supporting is
effectively: "We must absolutely avoid a situation where some unknown
system is temporarily restricted in how many vCPUs it can create due
to a sparse address space, even if it means making the context switch
4x as expensive for every single current user."  I just don't think
that's a reasonable position in any shape or form.

> > I don't even know exactly what failure mode you're worried about.  Two
> > kinds of potential failures I know about:
> >  - Performance impacted because pages can't be NUMA-local
> >  - Toolstack operations (including domain creation) fail because
> > xenheap has been exhausted.
>
> One thing I can't help thinking you keep overlooking throughout your
> reply: xenheap and domheap aren't separate. There being only a
> relatively small part of it needed for the worst case estimate you did
> means nothing as to exhausting the xenheap in practice: Almost the
> entirety of it (with the DMA reserve being somewhat protected) can be
> used to build domains. Once in that state, allocations would fail no
> matter that large swathes of domheap might (have become) available
> (again).

Right, so if Xen allocates too much domheap from the directmap region,
the xenheap may be not be able to allocate any more, even if there's
plenty of memory.

So something like the following:

We have 8TiB of RAM, 4 nodes, 2TiB per node.  The user wants to start
4 2TiB guests, one pinned to each node; so she starts d1 on node 1, d2
on node 2, then tries d3 and can't start it because although there's
still 4TiB of RAM left, all the memory below 4TiB was handed out to
guests already.

Is that what you had in mind?

If I didn't agree that the xenheap represents technical debt that
needs to be removed anyway, I'd say a simpler solution would be to do
do some simple xenheap reservation, based on various factors
(including number of pCPUs, and the total amount of RAM).  Reserving
6GiB on an 8TiB system would have very little impact (the first guest
would either need to be a bit smaller, or have, and would make the
whole problem go away essentially.  The first guest would either need
to be less than 0.1% smaller, or have 0.1% of its pages on a different
node.

> That said, with what you indicated at the very bottom of your reply,
> it looks like this part of the discussion has become largely moot.

Yes, but I also want to challenge your operating principles -- to get
you to state more clearly what you're concerned about.  Also, in order
to either get you to relax a bit about the xenheap growing, or to help
you articulate more clearly what problems which contributors need to
address.

 -George

On 25.08.26 13:42, George Dunlap wrote:
> On Mon, Aug 24, 2026 at 10:02 AM Jan Beulich <jbeulich@suse.com> wrote:
>> On 21.08.2026 17:17, George Dunlap wrote:
>>> On Fri, Aug 21, 2026 at 9:45 AM Jan Beulich <jbeulich@suse.com> wrote:
>>>> On 20.08.2026 19:43, George Dunlap wrote:
>>>>> One point reviewers may want to look at specifically: patch 1 changes
>>>>> where the per-domain page-tables are allocated from, and its commit
>>>>> message discusses the (minor) NUMA-placement consequence.
>>>>
>>>> While I don't recall which recent patch (series) it was, I can't very well
>>>> say "no new xenheap allocations please" there without also saying so here.
>>>> I've read over patch 1's description, and while it tries to justify this
>>>> accordingly, I still remain concerned. I think we simply have to accept
>>>> the mapping overhead, to avoid allocating from a pool which - over time -
>>>> is representing a decreasing portion of total memory systems have (on
>>>> average, and not even considering systems with extremely sparse memory
>>>> layouts, and with perhaps PDX compression not doing good enough to
>>>> compensate).
>>>
>>> You should certainly have the same resistance to adding new xenheap
>>> allocations.  But looking at the numbers, I don't see that we're
>>> anywhere near the point where we say, "Absolutely no new xenheap
>>> allocations, regardless of the cost."
>>
>> Well, that depends, and in part on the longer term plans with ASI. It has
>> been my (silent) assumption that eventually the directmap would go away
>> altogether when ASI is in use, with the VA space freed (almost?) all
>> becoming available for vmap(). With the disappearance of directmap, the
>> xenheap would naturally disappear as well. Hence putting stuff there in
>> new work actually adds to our technical debt.
> 
> I do think that makes sense as a long-term goal.  Actually, I asked
> Fable to do an audit of xenheap allocations, asking it to classify
> them as to whether they needed xenheap's key properties (easy mfn <->
> va conversion, available in early boot), and it reckoned only about 4%
> of the allocations (by volume) in the example system I did numbers for
> needed xenheap-specific properties.  Lots of things just need *some*
> global mapping somewhere, and would probably actually overall benefit
> from being moved to the vmap, so they wouldn't be restricted to a
> single NUMA node.  Grant frames were the biggest chunk.

Please note that I'm currently working on a patch series which will need
to convert grant frames and some other xenheap allocations to use domheap
and vmap().

Currently this is just a proof of concept for Xen summit, but I expect
this to become mature after some feedback I hope to get there.


Juergen

On Fri, Aug 21, 2026 at 4:17 PM George Dunlap <gwd@xenproject.org> wrote:

> Ultimately, I think there's a lot of wisdom in the saying, "Premature
> optimization is the root of all evil."  As I said, it's certainly
> right to be on our guard against adding things to xenheap, and look at
> alternatives; but we're nowhere near the point where we need to say,
> "Absolutely nothing added, regardless of the cost."  The design here
> is not locking us into the pages long-term; alternate designs have a
> significant cost in terms of authoring, reviewing, code complexity and
> maintenance, and code performance.

To get a sense of how much this approach simplifies things, look at
how patch 1 of this series simplified create_perdomain_mapping; and
look at how much simpler patch 2 is compared to Roger's implementation
[1].  The xenheap option is way easier to review and maintain.

 -George

[1] https://marc.info/?l=xen-devel&m=173634640624259