From nobody Wed Jun 10 12:27:04 2026 Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [95.215.58.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 32CA9331A64 for ; Fri, 24 Apr 2026 06:26:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777011989; cv=none; b=Gz6ekconUxVvFJfXwsKL3Nj0Ul6iYMntoTG5reZVAL/hWVVXzJsqapVscU16z6V5bRmouqmvGVJQW3cUXcRl8wPwI5V3PjkzhAHVGnhugomAt43OMXbyoM8CAMQhgxhdxp1pLMiJ/zwY4kouNy6TKYW/CDWSPKysWmbAOorPVwA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777011989; c=relaxed/simple; bh=B80OrE2TyDwgJ0jOPDUtK8dp4/gddcy+8J6bsgw0WhI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Zr0hSQYWuY+kvdYV+6dzLl/EhnmsADQsEnR9ZJoNqyO6v3h8QxnU7DJzSBmdemxShcmj1xqXsi7QrtFmb9YNySC3T5esRti/U1eMpAiOd9qw9JXuiETpYkm5/lbrTohTc/gmVx7zgpxNRfRAFq7mIc0BWltQIYYUx7W6LjMHtwo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=fB+5sM7g; arc=none smtp.client-ip=95.215.58.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="fB+5sM7g" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1777011976; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r4i42uUk09a42HzxE9DZCh9MNHfSJlOmtFh/Ez0FeKo=; b=fB+5sM7gMx2on5aLQHGonIkRBU1HzC1Qr4OzNjxhHU77wNinnNBc9rippSgLMbbXX6/ENR YhqKF5dOnXSibRmTfsDJJHzi8TgFN+umza8gWguEhfnrZtGvFoHTJ/0qtb96e7al7099hI 0uh8GEjClEWTmvJCfoi+JokTjJch4VE= From: Lance Yang To: akpm@linux-foundation.org Cc: peterz@infradead.org, david@kernel.org, dave.hansen@intel.com, dave.hansen@linux.intel.com, ypodemsk@redhat.com, hughd@google.com, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, arnd@arndb.de, ljs@kernel.org, ziy@nvidia.com, baolin.wang@linux.alibaba.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, shy828301@gmail.com, riel@surriel.com, jannh@google.com, jgross@suse.com, seanjc@google.com, pbonzini@redhat.com, boris.ostrovsky@oracle.com, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, ioworker0@gmail.com, Lance Yang Subject: [PATCH 7.2 v10 1/2] mm/mmu_gather: prepare to skip redundant sync IPIs Date: Fri, 24 Apr 2026 14:25:27 +0800 Message-ID: <20260424062528.71951-2-lance.yang@linux.dev> In-Reply-To: <20260424062528.71951-1-lance.yang@linux.dev> References: <20260424062528.71951-1-lance.yang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" From: Lance Yang When page table operations require synchronization with software/lockless walkers, they call tlb_remove_table_sync_{one,rcu}() after flushing the TLB (tlb->freed_tables or tlb->unshared_tables). On architectures where the TLB flush already sends IPIs to all target CPUs, the subsequent sync IPI broadcast is redundant. This is not only costly on large systems where it disrupts all CPUs even for single-process page table operations, but has also been reported to hurt RT workloads[1]. Introduce tlb_table_flush_implies_ipi_broadcast() to check if the prior TLB flush already provided the necessary synchronization. When true, the sync calls can early-return. A few cases rely on this synchronization: 1) hugetlb PMD unshare[2]: The problem is not the freeing but the reuse of the PMD table for other purposes in the last remaining user after unsharing. 2) khugepaged collapse[3]: Ensure no concurrent GUP-fast before collapsing and (possibly) freeing the page table / re-depositing it. Currently always returns false (no behavior change). The follow-up patch will enable the optimization for x86. [1] https://lore.kernel.org/linux-mm/1b27a3fa-359a-43d0-bdeb-c31341749367@k= ernel.org/ [2] https://lore.kernel.org/linux-mm/6a364356-5fea-4a6c-b959-ba3b22ce9c88@k= ernel.org/ [3] https://lore.kernel.org/linux-mm/2cb4503d-3a3f-4f6c-8038-7b3d1c74b3c2@k= ernel.org/ Suggested-by: David Hildenbrand (Arm) Acked-by: David Hildenbrand (Arm) Signed-off-by: Lance Yang --- include/asm-generic/tlb.h | 17 +++++++++++++++++ mm/mmu_gather.c | 15 +++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index bdcc2778ac64..cb41cc6a0024 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -240,6 +240,23 @@ static inline void tlb_remove_table(struct mmu_gather = *tlb, void *table) } #endif /* CONFIG_MMU_GATHER_TABLE_FREE */ =20 +/** + * tlb_table_flush_implies_ipi_broadcast - does TLB flush imply IPI sync + * + * When page table operations require synchronization with software/lockle= ss + * walkers, they flush the TLB (tlb->freed_tables or tlb->unshared_tables) + * then call tlb_remove_table_sync_{one,rcu}(). If the flush already sent + * IPIs to all CPUs, the sync call is redundant. + * + * Returns false by default. Architectures can override by defining this. + */ +#ifndef tlb_table_flush_implies_ipi_broadcast +static inline bool tlb_table_flush_implies_ipi_broadcast(void) +{ + return false; +} +#endif + #ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE /* * This allows an architecture that does not use the linux page-tables for diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 3985d856de7f..37a6a711c37e 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -283,6 +283,14 @@ void tlb_remove_table_sync_one(void) * It is however sufficient for software page-table walkers that rely on * IRQ disabling. */ + + /* + * Skip IPI if the preceding TLB flush already synchronized with + * all CPUs that could be doing software/lockless page table walks. + */ + if (tlb_table_flush_implies_ipi_broadcast()) + return; + smp_call_function(tlb_remove_table_smp_sync, NULL, 1); } =20 @@ -312,6 +320,13 @@ static void tlb_remove_table_free(struct mmu_table_bat= ch *batch) */ void tlb_remove_table_sync_rcu(void) { + /* + * Skip RCU wait if the preceding TLB flush already synchronized + * with all CPUs that could be doing software/lockless page table walks. + */ + if (tlb_table_flush_implies_ipi_broadcast()) + return; + synchronize_rcu(); } =20 --=20 2.49.0 From nobody Wed Jun 10 12:27:04 2026 Received: from out-174.mta0.migadu.com (out-174.mta0.migadu.com [91.218.175.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E87D35AC15 for ; Fri, 24 Apr 2026 06:26:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777011995; cv=none; b=BaOleq87LmbZvNhMSrVrgt9PvLb4p2A7oGRSOOKgkClj1ceSOCdJUR3FQcdgBynibirmBtIDXBiklTf3yQfqorRtGb3ZlBBnCRphqBgDUIs9pFYO8sAdl7JpVbDZ6gLBgbAeHTETu1rvOxgK4Nex3Zb1jDJbdN1/KCRlfFabbTs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777011995; c=relaxed/simple; bh=r3cX9/cKucEKI7jeSBF7ovZVeEBlaU54h0dr4Se14LI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a37BOL01RMe/cavgZC6CyG1GbGA0ppcgT3+vYSGLzhGaKyDoUlEwF4QZ/WD8t013TlXmq/H99N3JL5jcVMW6iK27ypENnpFRJ3+zrEO/WT/lU9fN76R2UML1pn5+YxVMAAsWoMM19FDLFq434NyrORuZayohP651fq1Qjxj6ktg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=lXFN78sr; arc=none smtp.client-ip=91.218.175.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="lXFN78sr" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1777011989; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rTh+Yvx/G+4eA9lbJw6qEK4XyieG3LR31VvrGkRr96o=; b=lXFN78sryZ4PB+yHSBlA0WPl5M4L4ogY1eH/biaBzsnxm7WNDoMkzHzGdk7QKgYfxNeOy0 5st7Qf4OgYppgPfZFHjFoqGEV8pmnNmBKnkB0JO8npFF6lWb+nYs/1TlvCaH5DfMx0R1aR DISGE9gPZLPERc1OKnbM0iSsegCiaus= From: Lance Yang To: akpm@linux-foundation.org Cc: peterz@infradead.org, david@kernel.org, dave.hansen@intel.com, dave.hansen@linux.intel.com, ypodemsk@redhat.com, hughd@google.com, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, arnd@arndb.de, ljs@kernel.org, ziy@nvidia.com, baolin.wang@linux.alibaba.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, shy828301@gmail.com, riel@surriel.com, jannh@google.com, jgross@suse.com, seanjc@google.com, pbonzini@redhat.com, boris.ostrovsky@oracle.com, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, ioworker0@gmail.com, Lance Yang Subject: [PATCH 7.2 v10 2/2] x86/tlb: skip redundant sync IPIs for native TLB flush Date: Fri, 24 Apr 2026 14:25:28 +0800 Message-ID: <20260424062528.71951-3-lance.yang@linux.dev> In-Reply-To: <20260424062528.71951-1-lance.yang@linux.dev> References: <20260424062528.71951-1-lance.yang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" From: Lance Yang Some page table operations need to synchronize with software/lockless walkers after a TLB flush by calling tlb_remove_table_sync_{one,rcu}(). On x86, that extra synchronization is redundant when the preceding TLB flush already broadcast IPIs to all relevant CPUs. native_pv_tlb_init() checks whether native_flush_tlb_multi() is in use. On CONFIG_PARAVIRT systems, it checks pv_ops; on non-PARAVIRT, native flush is always in use. It decides once at boot whether to enable the optimization: if using native TLB flush and INVLPGB is not supported, we know IPIs were sent and can skip the redundant sync. The decision is fixed via a static key as Peter suggested[1]. PV backends (KVM, Xen, Hyper-V) typically have their own implementations and don't call native_flush_tlb_multi() directly, so they cannot be trusted to provide the IPI guarantees we need. Also rename the x86 flush_tlb_info bit from freed_tables to wake_lazy_cpus, as Dave suggested[2], to match the behavior it controls: whether the remote flush may skip CPUs in lazy TLB mode. Both freed_tables and unshared_tables set it, because lazy-TLB CPUs must receive IPIs before page tables can be freed or reused. With that guarantee in place, tlb_table_flush_implies_ipi_broadcast() can safely skip the later sync IPI. Two-step plan as David suggested[3]: Step 1 (this patch): Skip redundant sync when we're 100% certain the TLB flush sent IPIs. INVLPGB is excluded because when supported, we cannot guarantee IPIs were sent, keeping it clean and simple. Step 2 (future work): Send targeted IPIs only to CPUs actually doing software/lockless page table walks, benefiting all architectures. Regarding Step 2, it obviously only applies to setups where Step 1 does not apply: like x86 with INVLPGB or arm64. [1] https://lore.kernel.org/linux-mm/20260302145652.GH1395266@noisy.program= ming.kicks-ass.net/ [2] https://lore.kernel.org/linux-mm/f856051b-10c7-4d65-9dbe-6b1677af74bd@i= ntel.com/ [3] https://lore.kernel.org/linux-mm/bbfdf226-4660-4949-b17b-0d209ee4ef8c@k= ernel.org/ Suggested-by: Dave Hansen Suggested-by: Peter Zijlstra Suggested-by: David Hildenbrand (Arm) Signed-off-by: Lance Yang --- arch/x86/hyperv/mmu.c | 4 ++-- arch/x86/include/asm/tlb.h | 19 +++++++++++++++- arch/x86/include/asm/tlbflush.h | 6 +++-- arch/x86/kernel/smpboot.c | 1 + arch/x86/mm/tlb.c | 39 +++++++++++++++++++++++---------- 5 files changed, 52 insertions(+), 17 deletions(-) diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c index cfcb60468b01..2cf1eeaffd6f 100644 --- a/arch/x86/hyperv/mmu.c +++ b/arch/x86/hyperv/mmu.c @@ -63,7 +63,7 @@ static void hyperv_flush_tlb_multi(const struct cpumask *= cpus, struct hv_tlb_flush *flush; u64 status; unsigned long flags; - bool do_lazy =3D !info->freed_tables; + bool do_lazy =3D !info->wake_lazy_cpus; =20 trace_hyperv_mmu_flush_tlb_multi(cpus, info); =20 @@ -198,7 +198,7 @@ static u64 hyperv_flush_tlb_others_ex(const struct cpum= ask *cpus, =20 flush->hv_vp_set.format =3D HV_GENERIC_SET_SPARSE_4K; nr_bank =3D cpumask_to_vpset_skip(&flush->hv_vp_set, cpus, - info->freed_tables ? NULL : cpu_is_lazy); + info->wake_lazy_cpus ? NULL : cpu_is_lazy); if (nr_bank < 0) return HV_STATUS_INVALID_PARAMETER; =20 diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 866ea78ba156..fb256fd95f95 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -5,22 +5,39 @@ #define tlb_flush tlb_flush static inline void tlb_flush(struct mmu_gather *tlb); =20 +#define tlb_table_flush_implies_ipi_broadcast tlb_table_flush_implies_ipi_= broadcast +static inline bool tlb_table_flush_implies_ipi_broadcast(void); + #include #include #include #include =20 +DECLARE_STATIC_KEY_FALSE(tlb_ipi_broadcast_key); + +static inline bool tlb_table_flush_implies_ipi_broadcast(void) +{ + return static_branch_likely(&tlb_ipi_broadcast_key); +} + static inline void tlb_flush(struct mmu_gather *tlb) { unsigned long start =3D 0UL, end =3D TLB_FLUSH_ALL; unsigned int stride_shift =3D tlb_get_unmap_shift(tlb); =20 + /* + * Both freed_tables and unshared_tables must wake lazy-TLB CPUs, so + * they receive IPIs before reusing or freeing page tables, allowing + * us to safely implement tlb_table_flush_implies_ipi_broadcast(). + */ + bool wake_lazy_cpus =3D tlb->freed_tables || tlb->unshared_tables; + if (!tlb->fullmm && !tlb->need_flush_all) { start =3D tlb->start; end =3D tlb->end; } =20 - flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); + flush_tlb_mm_range(tlb->mm, start, end, stride_shift, wake_lazy_cpus); } =20 static inline void invlpg(unsigned long addr) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflus= h.h index 5a3cdc439e38..39b9454781c3 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -18,6 +18,8 @@ =20 DECLARE_PER_CPU(u64, tlbstate_untag_mask); =20 +void __init native_pv_tlb_init(void); + void __flush_tlb_all(void); =20 #define TLB_FLUSH_ALL -1UL @@ -225,7 +227,7 @@ struct flush_tlb_info { u64 new_tlb_gen; unsigned int initiating_cpu; u8 stride_shift; - u8 freed_tables; + u8 wake_lazy_cpus; u8 trim_cpumask; }; =20 @@ -315,7 +317,7 @@ static inline bool mm_in_asid_transition(struct mm_stru= ct *mm) { return false; } extern void flush_tlb_all(void); extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, - bool freed_tables); + bool wake_lazy_cpus); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); =20 static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned lon= g a) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 294a8ea60298..df776b645a9c 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1256,6 +1256,7 @@ void __init native_smp_prepare_boot_cpu(void) switch_gdt_and_percpu_base(me); =20 native_pv_lock_init(); + native_pv_tlb_init(); } =20 void __init native_smp_cpus_done(unsigned int max_cpus) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 621e09d049cb..3ce254a3982c 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -26,6 +26,8 @@ =20 #include "mm_internal.h" =20 +DEFINE_STATIC_KEY_FALSE(tlb_ipi_broadcast_key); + #ifdef CONFIG_PARAVIRT # define STATIC_NOPV #else @@ -1360,16 +1362,16 @@ STATIC_NOPV void native_flush_tlb_multi(const struc= t cpumask *cpumask, (info->end - info->start) >> PAGE_SHIFT); =20 /* - * If no page tables were freed, we can skip sending IPIs to - * CPUs in lazy TLB mode. They will flush the CPU themselves - * at the next context switch. + * If lazy-TLB CPUs do not need to be woken, we can skip sending + * IPIs to them. They will flush themselves at the next context + * switch. * - * However, if page tables are getting freed, we need to send the - * IPI everywhere, to prevent CPUs in lazy TLB mode from tripping - * up on the new contents of what used to be page tables, while - * doing a speculative memory access. + * However, if page tables are getting freed or unshared, we need + * to send the IPI everywhere, to prevent CPUs in lazy TLB mode + * from tripping up on the new contents of what used to be page + * tables, while doing a speculative memory access. */ - if (info->freed_tables || mm_in_asid_transition(info->mm)) + if (info->wake_lazy_cpus || mm_in_asid_transition(info->mm)) on_each_cpu_mask(cpumask, flush_tlb_func, (void *)info, true); else on_each_cpu_cond_mask(should_flush_tlb, flush_tlb_func, @@ -1402,7 +1404,7 @@ static DEFINE_PER_CPU(unsigned int, flush_tlb_info_id= x); =20 static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm, unsigned long start, unsigned long end, - unsigned int stride_shift, bool freed_tables, + unsigned int stride_shift, bool wake_lazy_cpus, u64 new_tlb_gen) { struct flush_tlb_info *info =3D this_cpu_ptr(&flush_tlb_info); @@ -1429,7 +1431,7 @@ static struct flush_tlb_info *get_flush_tlb_info(stru= ct mm_struct *mm, info->end =3D end; info->mm =3D mm; info->stride_shift =3D stride_shift; - info->freed_tables =3D freed_tables; + info->wake_lazy_cpus =3D wake_lazy_cpus; info->new_tlb_gen =3D new_tlb_gen; info->initiating_cpu =3D smp_processor_id(); info->trim_cpumask =3D 0; @@ -1448,7 +1450,7 @@ static void put_flush_tlb_info(void) =20 void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, - bool freed_tables) + bool wake_lazy_cpus) { struct flush_tlb_info *info; int cpu =3D get_cpu(); @@ -1457,7 +1459,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigne= d long start, /* This is also a barrier that synchronizes with switch_mm(). */ new_tlb_gen =3D inc_mm_tlb_gen(mm); =20 - info =3D get_flush_tlb_info(mm, start, end, stride_shift, freed_tables, + info =3D get_flush_tlb_info(mm, start, end, stride_shift, wake_lazy_cpus, new_tlb_gen); =20 /* @@ -1834,3 +1836,16 @@ static int __init create_tlb_single_page_flush_ceili= ng(void) return 0; } late_initcall(create_tlb_single_page_flush_ceiling); + +void __init native_pv_tlb_init(void) +{ +#ifdef CONFIG_PARAVIRT + if (pv_ops.mmu.flush_tlb_multi !=3D native_flush_tlb_multi) + return; +#endif + + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return; + + static_branch_enable(&tlb_ipi_broadcast_key); +} --=20 2.49.0