From nobody Mon Apr 6 21:21:36 2026 Received: from va-1-112.ptr.blmpb.com (va-1-112.ptr.blmpb.com [209.127.230.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 237433002AB for ; Wed, 18 Mar 2026 04:59:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773809990; cv=none; b=b1QRXL9xNTn7z0/zU6xvntFHqXZBrhVym0ZNVQi/++5aIwDkeHYQZ0SMkTJNB9COQF3neSLaR305etfdAxw3TOc9NHDcBEU17aU4G1nyWgXMdHTZCcBnWKhAg17Vm3chF7n/mQ/jIhyd30ND7l3LOppu17rVpx52ipI7B+/sAPE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773809990; c=relaxed/simple; bh=iu2tR5++cx1DsR8djRRme5fVzstGkXmTeBUTeU7U+u0=; h=Subject:Message-Id:References:To:Cc:Mime-Version:In-Reply-To:From: Date:Content-Type; b=tpe4+cGZ4S9XSHZ59+KzjBO9qSArofs1rPsQSDs8CnekOdad2K1tH2me6juMccNdfRmtpsWtML3kih3dUjTdIh4fgKbcW+LcXMiU0fkIDrZJyFer+IA8H8aAsIH3nL9unVu1l+soIXwZwOmHMZdpBqoz30VwN5lJQT8zAMdOHLY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=X1K4ABaD; arc=none smtp.client-ip=209.127.230.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="X1K4ABaD" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1773809983; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=XM8LPX7mDbn8GUhfaBl8sGfw5chH9onLSnGyii/rkUQ=; b=X1K4ABaDtVPtTHZ5CfeAFuPyRZFGBd1fEqWUeJZoh4HYWknrQuZR+FBqA1hO5PPzEMwFVY 9s97PBzBXjFmF0OkWTRuGA+SVAA3b+1RHZaFhXGPagWI7BuCmaPm+KReXx+AdwI2ziNt7z XUSAxxo+rfQYHr+RN0HiLM+8rxNzXGFzhFHxl9tDPV6ersfTE3ZCDInzF0obvN5LUv68fz fR9CQePjJI52uWW9akWftKlmFsU+L9ygRjWRQPrDpBSafLful1t9L8ofSi2jd6/fR1sD2t maTOSpJvsxfprydn04XQVV/4M69RJtW16Ntfjz9FDhOe+F1etGR5bgwjBeMMXg== Subject: [PATCH v3 11/12] x86/mm: Enable preemption during native_flush_tlb_multi Message-Id: <20260318045638.1572777-12-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou References: <20260318045638.1572777-1-zhouchuyi@bytedance.com> X-Lms-Return-Path: To: , , , , , , , , , , , Cc: , "Chuyi Zhou" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 In-Reply-To: <20260318045638.1572777-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 From: "Chuyi Zhou" Date: Wed, 18 Mar 2026 12:56:37 +0800 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" native_flush_tlb_multi() may be frequently called by flush_tlb_mm_range() and arch_tlbbatch_flush() in production environments. When pages are reclaimed or process exit, native_flush_tlb_multi() sends IPIs to remote CPUs and waits for all remote CPUs to complete their local TLB flushes. The overall latency may reach tens of milliseconds due to a large number of remote CPUs and other factors (such as interrupts being disabled). Since flush_tlb_mm_range() and arch_tlbbatch_flush() always disable preemption, which may cause increased scheduling latency for other threads on the current CPU. Previous patch converted flush_tlb_info from per-cpu variable to on-stack variable. Additionally, it's no longer necessary to explicitly disable preemption before calling smp_call*() since they internally handle the preemption logic. Now it's safe to enable preemption during native_flush_tlb_multi(). Signed-off-by: Chuyi Zhou --- arch/x86/kernel/kvm.c | 4 +++- arch/x86/mm/tlb.c | 9 +++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 3bc062363814..4f7f4c1149b9 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -668,8 +668,10 @@ static void kvm_flush_tlb_multi(const struct cpumask *= cpumask, u8 state; int cpu; struct kvm_steal_time *src; - struct cpumask *flushmask =3D this_cpu_cpumask_var_ptr(__pv_cpu_mask); + struct cpumask *flushmask; =20 + guard(preempt)(); + flushmask =3D this_cpu_cpumask_var_ptr(__pv_cpu_mask); cpumask_copy(flushmask, cpumask); /* * We have to call flush only on online vCPUs. And diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 4704200de3f0..73500376d185 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1406,9 +1406,11 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsign= ed long start, if (mm_global_asid(mm)) { broadcast_tlb_flush(&info); } else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + put_cpu(); info.trim_cpumask =3D should_trim_cpumask(mm); flush_tlb_multi(mm_cpumask(mm), &info); consider_global_asid(mm); + goto invalidate; } else if (mm =3D=3D this_cpu_read(cpu_tlbstate.loaded_mm)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1417,6 +1419,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigne= d long start, } =20 put_cpu(); +invalidate: mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } =20 @@ -1697,7 +1700,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_b= atch *batch) invlpgb_flush_all_nonglobals(); batch->unmapped_pages =3D false; } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + put_cpu(); flush_tlb_multi(&batch->cpumask, &info); + goto clear; } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1705,9 +1710,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_b= atch *batch) local_irq_enable(); } =20 - cpumask_clear(&batch->cpumask); - put_cpu(); +clear: + cpumask_clear(&batch->cpumask); } =20 /* --=20 2.20.1