From nobody Thu Sep 24 20:03:35 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAA9F46A5FE for ; Mon, 21 Sep 2026 09:54:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984489; cv=none; b=E3rAOnf886zNMa4JJeJB2uTGNqBYu6D6U9ojUQXmx5k8f750SgVmQHdG4jws7Q/2T/340ACZj2SGrFt01k1ny6rP85bea8X97S8oTROX94G1URn+U4zlcWz1sFdgG0zryQPfpLBAcNF15xgMJMoGHNZ+5XszSVWcHERQUW09aFk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984489; c=relaxed/simple; bh=D2H32IcNGCfn9AdDj0sZ4KdyUBvOQiBM5LH98Jkhw80=; h=Date:Content-Type:To:From:Mime-Version:Cc:Subject:Message-Id: In-Reply-To:References; b=NZD6xDpplwq/0P3BuoYzyrLx0XVAqfRhcpZo4LaC/zWzc3vu0PRl6FWyTByqLZ1w00HWYbtoIQRfcvWov9KjzGzQdo57ZLJVrEAGvntVKYEZn4hpzpu9NMRoQzqit1wK8dsBehDFOL+Wkvs5ppG4GOEvISDGV9N+4PIPuzuOMLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=exdE4D/V; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="exdE4D/V" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1789984476; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=OcmkMLtMui+Hzv0wD+gl6HVDauRdtJSikt6X5IDzSTQ=; b=exdE4D/VHZw4M/99nPnAKSLcZEEtEL3jNijToIp72RMqNCg/7u5pJ3K3BZHT1RLYE33Yw6 c/Ec5HaBcsp6dpiOqbQv53jtvc8FoBr2Ks+/sfU4rGkjfZ9HGwrt04CNlZaxoy9L2qyHSB nrbLdj+uKj5cmrQlybt1yXof+ACAgTCwAiQXhU1vAf/ObY6EiWNF/+mkUIHvyN3eRWy0KV BUiy2IzrvBRx/oLi0Ykxn4TCR6BdChcWTWdseNY9Y7TSXuv6LIxciMWebRadsHUkh+RFuf 1etVBg5tleoV09ckwOtG3v1nEYWEhabRWHQ6FrecIjWxbvzFn5Un1aixdk16kw== Date: Mon, 21 Sep 2026 17:53:55 +0800 X-Lms-Return-Path: Content-Transfer-Encoding: quoted-printable To: , , , , , , , , , , , , From: "Chuyi Zhou" X-Original-From: Chuyi Zhou X-Mailer: git-send-email 2.20.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Cc: , "Chuyi Zhou" Subject: [PATCH 1/5] x86/mm: Account for remote kernel TLB flush requests Message-Id: <20260921095359.3784458-2-zhouchuyi@bytedance.com> In-Reply-To: <20260921095359.3784458-1-zhouchuyi@bytedance.com> References: <20260921095359.3784458-1-zhouchuyi@bytedance.com> Content-Type: text/plain; charset="utf-8" NR_TLB_REMOTE_FLUSH counts requests to invalidate TLB entries on other CPUs. flush_tlb_all() and native_flush_tlb_multi() account for these requests, but flush_tlb_kernel_range() bypasses both functions and does not update the counter. Kernel range flushes are therefore missing from the sender-side statistics, including ranges promoted to a full flush. Count each request in kernel_tlb_flush_all() and kernel_tlb_flush_range(). Account for both IPI and INVLPGB backends, as flush_tlb_all() already does, and count once per request regardless of the number of target CPUs or invalidation instructions. Signed-off-by: Chuyi Zhou --- arch/x86/mm/tlb.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 0c4da320831c..da7c408f921c 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1488,6 +1488,8 @@ static void do_kernel_range_flush(void *info) =20 static void kernel_tlb_flush_all(struct flush_tlb_info *info) { + count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_flush_all(); else @@ -1496,6 +1498,8 @@ static void kernel_tlb_flush_all(struct flush_tlb_inf= o *info) =20 static void kernel_tlb_flush_range(struct flush_tlb_info *info) { + count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_kernel_range_flush(info); else base-commit: e81ee06308379a5f2ededf997bcf17551bce5db7 --=20 2.20.1 From nobody Thu Sep 24 20:03:35 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D52047140E for ; Mon, 21 Sep 2026 09:54:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984499; cv=none; b=RQrOVNmdGYtO/DSDumYhWs0OiSyW7OyOux6okohmFSXNDOlAM7ZQ/AsOXZIzWyA9N3ppilk7f4vJWvSpskTB13XaJPnY90lFDSlTw5YAECTZek+nz/3kqT29pj/Ebv+Zn6u3vTOrH6HRKqW9oxrMLAsmfZtf6emXGL7nGvNvOyc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984499; c=relaxed/simple; bh=IVPTkpJIClKUpnACjBCDVcvOd5PdrF+IXOfA9atO/XY=; h=Subject:Date:To:From:Message-Id:Mime-Version:In-Reply-To:Cc: References:Content-Type; b=qJi9i/XcvePRTlAPzhpuqJKB7/mWdVHRRaaXPzmBRFoSgu9Mr+4Dd6FLZ/pRuZ4lqLQzpeSvDzRW64n6VZyHmqjt75bfM5MwXLMJ65NxLa9ZhhLaENrECyFvUmwtyscFxZC3yegadIAC5ggno9OxMSPazvzNddK5B5P0NbKG/Qo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CKXp18in; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CKXp18in" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1789984492; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=zRismljpTyFugrrIt8llp3KUsGKfOV8Fp0taHktktcI=; b=CKXp18inONyREBEfdzQCUKsHwjkL4/gAkfesVcSQR57mETnt/k1qhCF0iR5Kcvv4t+dUlB LgVFPwlGA0w6Q77l1EwxYHOYysGnEMedVtRDSIAm/ftLcLiaTZoG8fTycCj6C3KbwVrlID v1XgUn8cJIH+/RKA0krlXspA4cOSxNEiAdcfVjvIbFjpWxepvq36Atnz6bCV7vtSOv1xgl YnP/fFMNGRqr5xYIK++zwty4HEKuFohdRV5GiJRaQi5WUNHPAuDnlsGDYSgVs5Eh3MnrXF /ZMeSAMWilvHKfOl4bIG3viG/H0OHT5PLie0WdRowhq3QgsDMcQlEJ9L20bTcQ== Subject: [PATCH 2/5] x86/mm: Share the full TLB flush dispatch Date: Mon, 21 Sep 2026 17:53:56 +0800 X-Lms-Return-Path: X-Original-From: Chuyi Zhou X-Mailer: git-send-email 2.20.1 To: , , , , , , , , , , , , From: "Chuyi Zhou" Message-Id: <20260921095359.3784458-3-zhouchuyi@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable In-Reply-To: <20260921095359.3784458-1-zhouchuyi@bytedance.com> Cc: , "Chuyi Zhou" References: <20260921095359.3784458-1-zhouchuyi@bytedance.com> Content-Type: text/plain; charset="utf-8" flush_tlb_all() and kernel_tlb_flush_all() select the same INVLPGB or IPI backend and account for the same sender-side event. Keeping two implementations duplicates the backend selection and accounting. Use kernel_tlb_flush_all() for both the public full-flush interface and kernel ranges promoted to a full flush. Drop its unused flush_tlb_info argument and keep the event accounting in the shared helper so each request is counted once. Signed-off-by: Chuyi Zhou --- arch/x86/mm/tlb.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index da7c408f921c..5e9d1689f605 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1445,7 +1445,7 @@ static void do_flush_tlb_all(void *info) __flush_tlb_all(); } =20 -void flush_tlb_all(void) +static void kernel_tlb_flush_all(void) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); =20 @@ -1457,6 +1457,11 @@ void flush_tlb_all(void) on_each_cpu(do_flush_tlb_all, NULL, 1); } =20 +void flush_tlb_all(void) +{ + kernel_tlb_flush_all(); +} + /* Flush an arbitrarily large range of memory with INVLPGB. */ static void invlpgb_kernel_range_flush(struct flush_tlb_info *info) { @@ -1486,16 +1491,6 @@ static void do_kernel_range_flush(void *info) flush_tlb_one_kernel(addr); } =20 -static void kernel_tlb_flush_all(struct flush_tlb_info *info) -{ - count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); - - if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) - invlpgb_flush_all(); - else - on_each_cpu(do_flush_tlb_all, NULL, 1); -} - static void kernel_tlb_flush_range(struct flush_tlb_info *info) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); @@ -1515,7 +1510,7 @@ void flush_tlb_kernel_range(unsigned long start, unsi= gned long end) TLB_GENERATION_INVALID); =20 if (info.end =3D=3D TLB_FLUSH_ALL) - kernel_tlb_flush_all(&info); + kernel_tlb_flush_all(); else kernel_tlb_flush_range(&info); } --=20 2.20.1 From nobody Thu Sep 24 20:03:35 2026 Received: from va-1-113.ptr.blmpb.com (va-1-113.ptr.blmpb.com [209.127.230.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3C6D468C38 for ; Mon, 21 Sep 2026 09:55:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.113 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984526; cv=none; b=gNW1GuKr3Hk/V6SxU2OUXg+TB0la0l+IoKebya0dZr9ku8Sxhib867HhL7sRXoTbZQpe5MkCv9b2bhkhzl0lJ72jRpMP17Z0wQ5JUZxGaMN2NtSF92MkXDTmH3FcShMAw35X2eNRo2EaWDDcRa2/m3TgvcJFFBkxNIFdd1OKmM0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984526; c=relaxed/simple; bh=t61+VFVtE8EOj6JpiJSOYzszRz+2eirW6nwllU9BOYE=; h=To:Cc:From:Mime-Version:Date:References:Content-Type:Subject: Message-Id:In-Reply-To; b=QADGIpbQPRzyfMkPA5k33IHSH2TfC12X5CCt5gxIQUp2AH3luJReYICsrelg79cJl0WBbztz7TMDEAV9AEYS1gpJwlbTSBol1rNvt2lux3cGfllq1KkQ0XIvMGckqVUsVCtz2A+/vx19zI2VlcXkMmoTki0BLGSPMsvZyVJkFiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Gb05Dh+B; arc=none smtp.client-ip=209.127.230.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Gb05Dh+B" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1789984509; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=LgWZ+OUTM6vzTgLOIOm4srpsdcihjDcgdCtvNTUyPDA=; b=Gb05Dh+BqN83S53qmprLCT7W6ZzSvlUnt1J/ygfmm4+0Fawd4uHRYJdKVxY8/cdjNj5OQw 8xifwf91ZJ+MxB2xcNgKkCk+A1qfKYO1xHl8tGEpXUjXmtZXz3OnuUKt/MNlEHRAYkPHCn erzRypjpJG5SDFKUOYAq6Yiwkgtg6uzapDL/OZcelSzPtuuCd4ix38sRZsuaLs/iDlqEOu 2eFQVr6VDXQDU/AT9NBo571cFMl2GYM5wF55vwmPlnBxup7ZnEtmc+1GLD4lvj5zqyE9z3 C7pSepbXJxV9xFGsuYgDDJyNAScke0uXtAuCCYgdVeMclXofKlHRvr1uTF4zMQ== To: , , , , , , , , , , , , Cc: , "Chuyi Zhou" From: "Chuyi Zhou" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.20.1 Date: Mon, 21 Sep 2026 17:53:57 +0800 Content-Transfer-Encoding: quoted-printable References: <20260921095359.3784458-1-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou Subject: [PATCH 3/5] x86/mm: Extract the TLB range flush threshold check Message-Id: <20260921095359.3784458-4-zhouchuyi@bytedance.com> X-Lms-Return-Path: In-Reply-To: <20260921095359.3784458-1-zhouchuyi@bytedance.com> Content-Type: text/plain; charset="utf-8" The decision to replace a range flush with a full flush is embedded in init_flush_tlb_info(). Both mm and kernel flushes use this policy, but the kernel path only needs the range and the decision, without the mm-specific descriptor initialization. Extract the threshold predicate into tlb_range_exceeds_ceiling() and use it in init_flush_tlb_info(). Preserve the unsigned range arithmetic, right-shift rounding and strict greater-than comparison. Descriptor initialization and both flush paths remain unchanged. Signed-off-by: Chuyi Zhou --- arch/x86/mm/tlb.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 5e9d1689f605..f0dfeb271c66 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1373,6 +1373,12 @@ void flush_tlb_multi(const struct cpumask *cpumask, */ unsigned long tlb_single_page_flush_ceiling __read_mostly =3D 33; =20 +static bool tlb_range_exceeds_ceiling(unsigned long start, unsigned long e= nd, + unsigned int stride_shift) +{ + return ((end - start) >> stride_shift) > tlb_single_page_flush_ceiling; +} + static void init_flush_tlb_info(struct flush_tlb_info *info, struct mm_struct *mm, unsigned long start, unsigned long end, @@ -1383,7 +1389,7 @@ static void init_flush_tlb_info(struct flush_tlb_info= *info, * If the number of flushes is so large that a full flush * would be faster, do a full flush. */ - if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) { + if (tlb_range_exceeds_ceiling(start, end, stride_shift)) { start =3D 0; end =3D TLB_FLUSH_ALL; } --=20 2.20.1 From nobody Thu Sep 24 20:03:35 2026 Received: from va-1-112.ptr.blmpb.com (va-1-112.ptr.blmpb.com [209.127.230.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41A2148664F for ; Mon, 21 Sep 2026 09:55:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984534; cv=none; b=Fo+WoYXmazxEp7vFIN4NT/y9fTEj59CoyHqC7kYR5ke7/tdv6ryp+3wz9KofXhwjA+03ZOmoEDrsf2LjPTs7sZ/BQ+gO/Ker4TMorRMD7CLZOiq1o5/hO+7+gAIXBHHcrpVMiw3HTSi4JIEps038yFB/Jl5KZbuALwtzHWJtrJE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984534; c=relaxed/simple; bh=tdJ4y6W831MfrcRxzWcbmDc9h7zfEQgucTZ56tqe1KY=; h=In-Reply-To:References:Date:Message-Id:Mime-Version:Content-Type: Cc:From:Subject:To; b=GO+MGWMlciBsdVjr6hvMIDiESwuhOtj/UCP24twaqnDfGuAg97auNcHSJv0U052WbHsPVW+Z6x7+dnzWnRnGe1J6WXQvh2DXAUX+aA9QUpE7QRRvVkJbItVrP1KRf6GWNIAhZKcBbaLtHqN9CyRaRLIBBzeKtDFWPRkH0aM5u1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=WKSB034A; arc=none smtp.client-ip=209.127.230.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="WKSB034A" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1789984523; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=pYrQpTWYHD0kCaq3gvxtuTIuMINfoyz5/m2rYD7GMAg=; b=WKSB034ALjcT2M3oZVB41lwzw63UGV4NDZ1DVBq0SEgsxhUOUWROepyweO67MbqCsLP2st NfbUkk5/1jcMz0bVVcjTf9YDbBz4SfThACuuRs6yu/Mkza99sysG5Q1XQydmIohVQ4Va2F 87s9OgeU9Z31hkb+aZYyvhJEv8qa2Y8o7phcQl5jOTnNjwdAqohK+D/tVoTBvO+Y9XW/UA LoEM4YIueYoiDskgzxu4rscYx+2m+E3wkJ/HC3c++ZEXZkzQ7wHjExaz48f2nbpdXq7g/8 mPB57TP6cmKY865MxoVyTCmppTUFZkn0sibfOoZNlZS0SEqgqJaHPASPTqB0lQ== X-Original-From: Chuyi Zhou In-Reply-To: <20260921095359.3784458-1-zhouchuyi@bytedance.com> References: <20260921095359.3784458-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 Date: Mon, 21 Sep 2026 17:53:58 +0800 Message-Id: <20260921095359.3784458-5-zhouchuyi@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Cc: , "Chuyi Zhou" From: "Chuyi Zhou" Subject: [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info Content-Transfer-Encoding: quoted-printable To: , , , , , , , , , , , , X-Lms-Return-Path: Content-Type: text/plain; charset="utf-8" Kernel range flushes only need the start and end addresses, but reuse flush_tlb_info and its initialization of mm state, TLB generations and the initiating CPU. None of those fields is consumed by the kernel flush callbacks. In particular, initializing initiating_cpu imposes a CPU-pinning requirement on a path that does not use it. Select the full or range flush directly in flush_tlb_kernel_range(), using the shared threshold predicate and an explicit TLB_FLUSH_ALL check. Keep init_flush_tlb_info() and its smp_processor_id() check for the mm paths. Pass start and end directly through the kernel range helpers. Package them in a private kernel_tlb_range only for the IPI callback, retaining the existing payload alignment. The synchronous on_each_cpu() call keeps the stack descriptor alive until all callbacks have completed. Retain the outer preemption guard in flush_tlb_kernel_range() so the descriptor refactoring does not change the preemption behavior. Signed-off-by: Chuyi Zhou Link: https://lore.kernel.org/20260522104818.CbT5fyN8@linutronix.de/ --- arch/x86/mm/tlb.c | 41 +++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index f0dfeb271c66..4f9f0a18dbbc 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1469,12 +1469,12 @@ void flush_tlb_all(void) } =20 /* Flush an arbitrarily large range of memory with INVLPGB. */ -static void invlpgb_kernel_range_flush(struct flush_tlb_info *info) +static void invlpgb_kernel_range_flush(unsigned long start, unsigned long = end) { unsigned long addr, nr; =20 - for (addr =3D info->start; addr < info->end; addr +=3D nr << PAGE_SHIFT) { - nr =3D (info->end - addr) >> PAGE_SHIFT; + for (addr =3D start; addr < end; addr +=3D nr << PAGE_SHIFT) { + nr =3D (end - addr) >> PAGE_SHIFT; =20 /* * INVLPGB has a limit on the size of ranges it can @@ -1487,38 +1487,47 @@ static void invlpgb_kernel_range_flush(struct flush= _tlb_info *info) __tlbsync(); } =20 +/* Preserve the alignment of the IPI payload shared with remote CPUs. */ +struct kernel_tlb_range { + unsigned long start; + unsigned long end; +} __aligned(FLUSH_TLB_INFO_ALIGN); + static void do_kernel_range_flush(void *info) { - struct flush_tlb_info *f =3D info; + const struct kernel_tlb_range *range =3D info; unsigned long addr; =20 /* flush range by one by one 'invlpg' */ - for (addr =3D f->start; addr < f->end; addr +=3D PAGE_SIZE) + for (addr =3D range->start; addr < range->end; addr +=3D PAGE_SIZE) flush_tlb_one_kernel(addr); } =20 -static void kernel_tlb_flush_range(struct flush_tlb_info *info) +static void kernel_tlb_flush_range(unsigned long start, unsigned long end) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); =20 - if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) - invlpgb_kernel_range_flush(info); - else - on_each_cpu(do_kernel_range_flush, info, 1); + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_kernel_range_flush(start, end); + } else { + struct kernel_tlb_range range =3D { + .start =3D start, + .end =3D end, + }; + + on_each_cpu(do_kernel_range_flush, &range, 1); + } } =20 void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - struct flush_tlb_info info; - guard(preempt)(); - init_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false, - TLB_GENERATION_INVALID); =20 - if (info.end =3D=3D TLB_FLUSH_ALL) + if (end =3D=3D TLB_FLUSH_ALL || + tlb_range_exceeds_ceiling(start, end, PAGE_SHIFT)) kernel_tlb_flush_all(); else - kernel_tlb_flush_range(&info); + kernel_tlb_flush_range(start, end); } =20 /* --=20 2.20.1 From nobody Thu Sep 24 20:03:35 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F7242D8DA8 for ; Mon, 21 Sep 2026 10:00:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984841; cv=none; b=g19/XtdUfjPN2oWrdY8GS/Q62KFdpO7GYn0Y990qbnLU+ZMyipvhyCbX8JGEwDwGmrqilGGuE/4WHWDPabSeI1y3pZaIVeqnKqTnmHABQOZxCPWNUyqLL5h+YvzEH9QdoOruti0H7ZVMMuGVOaeWar5qlJil2P9DTGS4V7So6Ow= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789984841; c=relaxed/simple; bh=JMon/sqgQsJexCEWq8SVfFoEgmuxLqowCdCV90yuWac=; h=Cc:Date:Mime-Version:Subject:Message-Id:References:In-Reply-To: Content-Type:To:From; b=OHUwadUvK6xK2A0tIle+FUs+2lBE+lbgzewuiZdp8UvF9cnKD9RPFfH7uaLAITU+1HhoVC/BSzbPveewqOISrAzkPl6q6rvcFB5w64kUW9qiQI8QoHqrMHVV+DI6vF4hAYaNwX7f4UjhaYqXDq+1lOMRRu6xa/rnGyJFTXJhJHc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PlgjNYoQ; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PlgjNYoQ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1789984828; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=FEEUMLjS98DKCdGbmaNycCrOOVlmaBW7hZVkLLaDhkk=; b=PlgjNYoQVQagATxgJHR6AjdwljuXBBBZWWaWfBfVEWsLxPadn/z14hrfSc6h5IcoZmR4Vx m6PF7kc3V/PVDHrhespX0dahPIBeOY3XrzJH1ALoMYsEU4S6XPUzy+ozrowy+9en/ENcah j+KZrgF6PaFb0qKwwlEKqjX1P5sESw7iQ1KHa/sAMe3/4d8ZKyqvqPsbtOC+SeDmnOGLe7 YxMAsEUZTHkguwLFQKwZxuJj3PDJ5HAtclOvgKbB1n9uGiq8zCecHiuhdcXIE2qMqGxqLZ jnyVf9u2zrDWaXv2qUnT6tH4sraXXH1dEn8RxnTCAH+7tBufbH9atJWi3TQ9Rw== Cc: , "Chuyi Zhou" Date: Mon, 21 Sep 2026 17:59:59 +0800 X-Original-From: Chuyi Zhou Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Lms-Return-Path: Subject: [PATCH 5/5] x86/mm: Re-enable preemption before waiting for kernel TLB flushes Message-Id: <20260921095959.3787573-1-zhouchuyi@bytedance.com> Content-Transfer-Encoding: quoted-printable References: <20260921095359.3784458-1-zhouchuyi@bytedance.com> In-Reply-To: <20260921095359.3784458-1-zhouchuyi@bytedance.com> To: , , , , , , , , , , , , From: "Chuyi Zhou" X-Mailer: git-send-email 2.20.1 Content-Type: text/plain; charset="utf-8" flush_tlb_kernel_range() uses on_each_cpu() to synchronously flush kernel mappings on all online CPUs when using the IPI backend. The synchronous wait can become longer as the number of online CPUs grows, and a remote CPU with interrupts disabled can further delay completion. Preemption remains disabled throughout this wait, delaying higher-priority tasks on the initiating CPU. The outer guard prevents the SMP layer from making its final completion wait preemptible. The kernel range descriptor is private stack storage and remains valid until on_each_cpu() returns. The SMP layer protects CPU selection, IPI queueing and local callback execution, so the kernel flush caller does not need to remain pinned while waiting for remote completion. INVLPGB requires separate protection: TLBSYNC only waits for operations issued on the same CPU. Disable preemption inside invlpgb_kernel_range_flush() across the broadcast loop and TLBSYNC. The full-flush helper invlpgb_flush_all() already provides this protection. Remove the outer preemption guard from flush_tlb_kernel_range() so the IPI completion wait can be preempted when the caller's context allows it. Keep both flush backends synchronous. Signed-off-by: Chuyi Zhou --- arch/x86/mm/tlb.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 4f9f0a18dbbc..f7773ba4a152 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1473,6 +1473,9 @@ static void invlpgb_kernel_range_flush(unsigned long = start, unsigned long end) { unsigned long addr, nr; =20 + /* Keep the INVLPGB operations and TLBSYNC on the same CPU. */ + guard(preempt)(); + for (addr =3D start; addr < end; addr +=3D nr << PAGE_SHIFT) { nr =3D (end - addr) >> PAGE_SHIFT; =20 @@ -1521,8 +1524,6 @@ static void kernel_tlb_flush_range(unsigned long star= t, unsigned long end) =20 void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - guard(preempt)(); - if (end =3D=3D TLB_FLUSH_ALL || tlb_range_exceeds_ceiling(start, end, PAGE_SHIFT)) kernel_tlb_flush_all(); --=20 2.20.1