.:: Intro
This code illustrates the idea I'm proposing at LSF/MM/BPF [0].
Sorry it's so close to the conference, I was initially quite ambitious
in what I wanted to show here and tried to implement a more complete
patch series. Now I've run out of time and I've had to reduce the scope
and just hack some minimal stuff together. Now, this series is _only_
supposed to be about page_alloc.c, everything else is just there as
scaffolding so that allocator code can be discussed.
I've marked the most incomplete patches with [HACKS] in the title to
illustrate what aspects are less worthy of attention.
See [0] and also [1] for broader context on the ASI/page_alloc topic.
See [2] for context about ASI itself. For this RFC the most important
fact is: ASI requires creating another kernel address space (the
"restricted address space") that is a subset of that normal one (i.e.
the "unrestricted address space"). That is, an address space just like
the normal one, but with holes in it. Pages that are unmapped from the
restricted address space are called "sensitive".
.:: The Idea
What is sensitive (i.e. where the holes are) is decided at allocation
time. This illustrates an initial implementation of that capability for
the direct map. The basic idea of this implementation is to operate at
pageblock-granularity, and use migratetypes to track sensitivity. The
key advantages of this approach are:
- Migratetypes exist to avoid fragmentation. Using them to index pages
by sensitivity takes advantage of this, so that the physmap doesn't
get fragmented with respect to sensitivity. This means we can use
large TLB entries for the restricted physmap.
- Since pageblocks are never smaller than a PMD mapping, if the
restricted physmap is always made of PMDs, we never have to break down
mappings while changing sensitivity. This means we don't have
difficulties with needing to allocate pagetables in the middle of the
allocator.
- Migratetypes already offer indexing capability - that is, there are
separate freelists for each migratetype. This means when the user
allocates a page with a given sensitivity, all the infrastructure is
already in place to look up a page that is already mapped/unmapped as
needed (if it exists). This minimizes unnecessary TLB flushes.
This differs from Mike Rapoport's work on __GFP_UNMAPPED [3] in that,
instead of having a totally separate free area for the pages that are
unmapped, it aims to pervade the allocator. If it turns out that for all
nonsensitive (or all sensitive, which seems highly unlikely) pages, a
access to the full feature set of the page allocator is not needed for a
performant system, we could certainly do something like Mike's patchset.
But we don't have any reason to expect a correlation between
sensitivity and performance needs.
.:: Patchset overview
- Patch 1 adds a minimal subset of the base ASI framework that was
introduced by the RFCv2 [2].
- Patches 2-5 add the necessary framework for creating and manipulating
the ASI physmap. This is the area where I have had to reduce the scope
of this series, I had hoped to present a proper integration here. But
instead I've had to just hack something together that kinda works.
You can probably skip over this section.
- Patches 6-8 are preparatory hacks and changes to the generic mm code.
- Patches 9-11 are the important bit. The new migratetypes are created.
Then logic is added to create nonsensitive pageblocks when needed.
Then logic is added to change them back to sensitive pageblocks when
needed.
.:: TODOs
- This doesn't let you allocate from MIGRATE_HIGHATOMIC pageblocks
unless you have __GFP_SENSITIVE. We probably need to make the
pageblock type and per-freelist logic more advanced to be able to
account for this.
- When pages transition from sensitive to nonsensitive, they need to be
zeroed to prevent any leftover data being leaked. This series doesn't
address that requirement at all.
- Although I think the abstract design is OK, the actual implementation
of calling asi_map()/asi_unmap() from page_alloc.c is pretty
confusing: asi_map() is implicit when calling
set_pageblock_migratetype() but asi_unmap() is up to the caller. This
requires some refactoring.
- Changes to the unrestricted physmap (page protection changes, memory
hotplug) are not properly mirrored into the restricted physmap.
- There's no integration with CMA. The branch at [4] has some minimal
integration into alloc_contig_range().
.:: References
[0] https://lore.kernel.org/linux-mm/CA+i-1C169s8pyqZDx+iSnFmftmGfssdQA29+pYm-gqySAYWgpg@mail.gmail.com/
[1] Some slides I presented in an earlier discussion of this topic:
https://docs.google.com/presentation/d/1Ozuan7E4z2YWm4V6uE_fe7YoF2BdS3m5jXjDKO7DVy0/edit#slide=id.g32d28ea451a_0_43
[2] https://lore.kernel.org/linux-mm/20250110-asi-rfc-v2-v2-0-8419288bc805@google.com/
[3] https://lore.kernel.org/all/20230308094106.227365-1-rppt@kernel.org/
[5] https://lore.kernel.org/linux-mm/20250129144320.2675822-1-jackmanb@google.com/
This series is available as a branch with some additional testing here:
[4] https://github.com/bjackman/linux/tree/asi/page-alloc-lsfmmbpf25
This applies to mm-unstable.
Signed-off-by: Brendan Jackman <jackmanb@google.com>
---
Brendan Jackman (11):
x86/mm: Bare minimum ASI API for page_alloc integration
x86/mm: Factor out phys_pgd_init()
x86/mm: Add lookup_pgtable_in_pgd()
x86/mm/asi: Sync physmap into ASI_GLOBAL_NONSENSITIVE
[RFC HACKS] Add asi_map() and asi_unmap()
mm/page_alloc: Add __GFP_SENSITIVE and always set it
[RFC HACKS] mm/slub: Set __GFP_SENSITIVE for reclaimable slabs
[RFC HACKS] mm/page_alloc: Simplify gfp_migratetype()
mm/page_alloc: Split MIGRATE_UNMOVABLE by sensitivity
mm/page_alloc: Add support for nonsensitive allocations
mm/page_alloc: Add support for ASI-unmapping pages
arch/Kconfig | 14 ++++
arch/x86/Kconfig | 1 +
arch/x86/include/asm/asi.h | 36 ++++++++
arch/x86/include/asm/pgtable_types.h | 2 +
arch/x86/mm/Makefile | 1 +
arch/x86/mm/asi.c | 85 +++++++++++++++++++
arch/x86/mm/init.c | 3 +-
arch/x86/mm/init_64.c | 53 ++++++++++--
arch/x86/mm/pat/set_memory.c | 34 ++++++++
include/linux/asi.h | 20 +++++
include/linux/gfp.h | 30 ++++---
include/linux/gfp_types.h | 15 +++-
include/linux/mmzone.h | 19 ++++-
include/linux/vmalloc.h | 4 +
mm/internal.h | 5 ++
mm/memory_hotplug.c | 2 +-
mm/page_alloc.c | 158 +++++++++++++++++++++++++++++++----
mm/show_mem.c | 13 +--
mm/slub.c | 6 +-
mm/vmalloc.c | 32 ++++---
20 files changed, 475 insertions(+), 58 deletions(-)
---
base-commit: 5ee93e1a769230377c3d44edd4917e8df77be566
change-id: 20250310-asi-page-alloc-80ea1f8307d0
Best regards,
--
Brendan Jackman <jackmanb@google.com>