From: Lance Yang <lance.yang@linux.dev>
Embed the tlb_table_flush_implies_ipi_broadcast() check directly inside
tlb_remove_table_sync_one() instead of requiring every caller to check
it explicitly. This relies on callers to do the right thing: flush with
freed_tables=true or unshared_tables=true beforehand.
All existing callers satisfy this requirement:
1. mm/khugepaged.c:1188 (collapse_huge_page):
pmdp_collapse_flush(vma, address, pmd)
-> flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE)
-> flush_tlb_mm_range(mm, ..., freed_tables = true)
-> flush_tlb_multi(mm_cpumask(mm), info)
So freed_tables=true before calling tlb_remove_table_sync_one().
2. include/asm-generic/tlb.h:861 (tlb_flush_unshared_tables):
tlb_flush_mmu_tlbonly(tlb)
-> tlb_flush(tlb)
-> flush_tlb_mm_range(mm, ..., unshared_tables = true)
-> flush_tlb_multi(mm_cpumask(mm), info)
unshared_tables=true (equivalent to freed_tables for sending IPIs).
3. mm/mmu_gather.c:341 (__tlb_remove_table_one):
When we can't allocate a batch page in tlb_remove_table(), we do:
tlb_table_invalidate(tlb)
-> tlb_flush_mmu_tlbonly(tlb)
-> flush_tlb_mm_range(mm, ..., freed_tables = true)
-> flush_tlb_multi(mm_cpumask(mm), info)
Then:
tlb_remove_table_one(table)
-> __tlb_remove_table_one(table) // if !CONFIG_PT_RECLAIM
-> tlb_remove_table_sync_one()
freed_tables=true, and this should work too.
Why is tlb->freed_tables guaranteed? Because callers like
pte_free_tlb() (via free_pte_range) set freed_tables=true before
calling __pte_free_tlb(), which then calls tlb_remove_table().
We cannot free page tables without freed_tables=true.
Note that tlb_remove_table_sync_one() was a NOP on bare metal x86
(CONFIG_MMU_GATHER_RCU_TABLE_FREE=n) before commit a37259732a7d
("x86/mm: Make MMU_GATHER_RCU_TABLE_FREE unconditional").
4-5. mm/khugepaged.c:1683,1819 (pmdp_get_lockless_sync macro):
Same as #1. These also use pmdp_collapse_flush() beforehand.
Suggested-by: David Hildenbrand (Red Hat) <david@kernel.org>
Signed-off-by: Lance Yang <lance.yang@linux.dev>
---
mm/mmu_gather.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c
index 7468ec388455..7b588643cbae 100644
--- a/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -276,6 +276,10 @@ static void tlb_remove_table_smp_sync(void *arg)
void tlb_remove_table_sync_one(void)
{
+ /* Skip the IPI if the TLB flush already synchronized with other CPUs. */
+ if (tlb_table_flush_implies_ipi_broadcast())
+ return;
+
/*
* This isn't an RCU grace period and hence the page-tables cannot be
* assumed to be actually RCU-freed.
--
2.49.0