Migrating a PTE-mapped large folio currently walks the folio's rmap
one page at a time. It does so twice per migration:
1. try_to_migrate_one(): Replace present PTEs with migration entries
2. remove_migration_pte(): Restore migration entries to working PTEs.
For a folio with N contiguous PTEs, each walk repeats page_vma_mapped_walk()
and the associated operation per subpage of the folio. (256 iterations
for 1M folio).
This series enables both walks to process a batch of consecutive PTEs
that map the same large folio and do some refactoring and clean up on
the way.
Performance:
AMD EPYC ZEN 3, 2-socket system (1 node per socket).
move_pages() migrating 1 GB from node 0 -> node 1, throughput
in GB/s:
batch- batch-restore
folio vanilla restore-only + unmap restore% restore+unmap%
4K 2.96±0.15 3.02±0.14 2.99±0.17 +2.2% +1.0%
64K 5.99±0.10 6.12±0.08 7.65±0.16 +2.2% +27.7%
256K 6.48±0.09 6.67±0.08 9.23±0.14 +2.9% +42.4%
1M 6.76±0.10 6.98±0.10 9.74±0.19 +3.1% +44.0%
2M 10.50±0.20 10.44±0.16 10.67±0.21 -0.5% +1.6%
In plain migration, the folio copy itself dominates the cost, so these rmap walk
savings are only partly visible end-to-end.
When the folio copy is batched-copy and offloaded to DMA engine [1], the copy is
no longer the bottleneck and rmap-walk batching translates into much larger gains:
This series + [1], DMA Offload on PTDMA (DCBM), 16 channels:
batch-copy dcbm+restore
folio vanilla -offload(dcbm) dcbm+restore +unmap (rmap batch impact)
4K 3.05±0.13 2.76±0.11 2.77±0.12 2.74±0.09
64K 5.85±0.11 10.66±0.27 11.56±1.45 17.55±0.56
256K 6.26±0.08 13.64±2.73 13.68±0.16 34.76±0.57 (2.54x)
1M 6.58±0.08 14.25±0.07 15.27±0.77 44.28±0.27 (3.11x)
2M 10.61±0.24 65.26±0.96 65.25±1.65 65.66±0.87
Applies cleanly on mm-new+[2].
[1] https://lore.kernel.org/linux-mm/20260630-shivank-batch-migrate-offload-v6-0-da95d7e8b8a2@amd.com
[2] https://lore.kernel.org/linux-mm/20260723070905.3422276-1-dev.jain@arm.com
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
Changes in v2:
- Rebase onto mm-new, drop the borrowed folio_try_share_anon_rmap_ptes() patch
and add it as dependency on Dev's series and adapt to recent migrate/rmap changes.
- Switch to recently added batched helpers - finish_folio_unmap, set_softleaf_ptes,
and page_anon_exclusive_batch.
- Rename migration_softleaf_entry_to_pte() to migration_entry_to_pte(). (David)
- Split PTE construction from hugetlb handling into separate patch. (David)
Mirror the split in mm/rmap.c patch as well.
- Rename make_migration_swp_pte() to make_migration_pte().
- Early exit the single-entry hugetlb page_vma_mapped_walk() paths - removes
unnecessary nesting.
- Add the missing page_vma_mapped_walk_done() in remove_migration_pte_hugetlb(),
which leaked the huge-PTE lock. (syzbot ci, Sashiko)
- Fix handling of hwpoisoned subpages in the batched unmap. Read PageHWPoison()
once and skip batching when mapped page is poisoned, and use same result for
hwpoison handling. (Sashiko)
- Remove the pte_unused(pteval) handling from try_to_migrate_hugetlb_one() (Sashiko).
- Link to v1: https://lore.kernel.org/r/20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com
---
Shivank Garg (7):
mm: factor out generic PTE batch detection from swap_pte_batch()
mm/migrate: factor out migration PTE construction
mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte()
mm/migrate: batch the restore-side migration rmap walk
mm/rmap: factor out migration PTE construction
mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one()
mm/rmap: batch the unmap of large folios in try_to_migrate_one()
mm/internal.h | 36 ++---
mm/migrate.c | 191 ++++++++++++++++++--------
mm/rmap.c | 419 ++++++++++++++++++++++++++++++++++++----------------------
3 files changed, 416 insertions(+), 230 deletions(-)
---
base-commit: 480a31230b426efb005b6e71a14ef80f405f18b6
change-id: 20260701-migrate-rmap-batch-8ecc04de1905
prerequisite-message-id: 20260723070905.3422276-1-dev.jain@arm.com
prerequisite-patch-id: 5fdb03c3ea4a037f66fbd1c9b5d04e24e86406eb
prerequisite-patch-id: 033980575a1ef89d70e43a67c5530e2cf6cf97aa
prerequisite-patch-id: 6f7508334fc2bfc0df104ac50d36ed88072487e4
prerequisite-patch-id: 3f1aaae36e02d8b9343f7e719f732f10c1767572
prerequisite-patch-id: 98bc99c6bc274594946d022b3998f5f1dc70012f
prerequisite-patch-id: dd35ec00b82bf235f2e41e286d16eab6a3fd1e9e
prerequisite-patch-id: 552ee939dc6a7b3cce51373f4e75fa01a9d2b979
prerequisite-patch-id: 485cdbbe4e65b13494dc5104d4b8b7d0d23911a4
Best regards,
--
Shivank Garg <shivankg@amd.com>