From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-112.ptr.blmpb.com (va-1-112.ptr.blmpb.com [209.127.230.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44CBA421EE2 for ; Wed, 13 May 2026 12:46:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676371; cv=none; b=hLzYR3JWqOB8ze5wqxbJo5ysjxm4qeAgHBO36whwbakBRaT/S6MMaWIBfFbguWXTRdho3ji04OEkDBAAWSkODc9eknZCdMh1UijQ+FFRsqjbJq0EbhJW4xH6AfGFBUDKLhaDLKiVKz/LchG1m03lV/dnIQ+YCKIR3su20Q4rBuA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676371; c=relaxed/simple; bh=Jh+uClddCca3FSI+mbTK83nxf3SnGFzmGxb6aJS9x14=; h=Mime-Version:In-Reply-To:Content-Type:Cc:References:To:From:Date: Message-Id:Subject; b=SAiHYNaGk6BcB0AilcqzrQI8S/Q39ZDDSwJLJ1SoEOX4Z7aKvR8595h8RHIueq4Br2ipBAHPx23zIN4N16i7STy2txrQMzUVGcRwSdKE2yy+6RDWqfnNQnK09PZMpPuvUn7TR6rS1zTjmPEimpiOIvnNSN4MYCz/+vWlYznj+yM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=VVyBh2fR; arc=none smtp.client-ip=209.127.230.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="VVyBh2fR" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676364; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=AQ5azY1la9Qq56fewVs7n4m/FZ8PJQBk4PtZGiWCpJI=; b=VVyBh2fRbVR/PjFAOve9jW79vzeDZqqQgDQ19Qis1pTpt3azwObH4wVGxQZHRSF9e+1DTg kpJ0++xSfpXXQ6ecxNBsJukoKuog9T1ZG3a7ihrZzKwnLyqGscw1GHvQtxZd6JHNrMW/Dm xM/aQSu4n+n9syzOUyabcq7N7rAq15VhGHlhfSf23m3LV7E720HFY6QSnbznOVO+bRuQXa 0Ab1gyIWHCM3y+YGIA+ZzuSfoF3kuZoiAG1HKRalWnwUyMD1KJQY0cZ+9fy4RlhJ4z9Z0E +mEx/v1EAqhrY0x/aZ7Wg0/7a4hk9UyDXAOam5ytCWJrW+UQctHZn4QsGSXY+A== Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Cc: , "Chuyi Zhou" X-Lms-Return-Path: References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Content-Transfer-Encoding: quoted-printable To: , , , , , , , , , , , , From: "Chuyi Zhou" Date: Wed, 13 May 2026 20:45:13 +0800 Message-Id: <20260513124524.2569867-2-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou X-Mailer: git-send-email 2.20.1 Subject: [RESEND PATCH v5 01/12] smp: Disable preemption explicitly in __csd_lock_wait Content-Type: text/plain; charset="utf-8" The latter patches will enable preemption before csd_lock_wait(), which could break csdlock_debug. Because the slice of other tasks on the CPU may be accounted between ktime_get_mono_fast_ns() calls, disable preemption explicitly in __csd_lock_wait(). This is a preparation for the next patches. Signed-off-by: Chuyi Zhou Acked-by: Muchun Song Reviewed-by: Steven Rostedt (Google) --- kernel/smp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/smp.c b/kernel/smp.c index a0bb56bd8dda..b58975480e11 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -323,6 +323,8 @@ static void __csd_lock_wait(call_single_data_t *csd) int bug_id =3D 0; u64 ts0, ts1; =20 + guard(preempt)(); + ts1 =3D ts0 =3D ktime_get_mono_fast_ns(); for (;;) { if (csd_lock_wait_toolong(csd, ts0, &ts1, &bug_id, &nmessages)) --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-114.ptr.blmpb.com (va-1-114.ptr.blmpb.com [209.127.230.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F230289E13 for ; Wed, 13 May 2026 12:46:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.114 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676389; cv=none; b=ZxH6pb8HIQZNCEtvg1Ilc801fyZJRQAE++nyR6cymnovu4iWm2fe1bvUAiXNJMenTfh3ZxTqoRsRRX9+WD3SlisF258w0hT4r+thNFaIbVTFPNjoGNUn+C7fxoEim33Dgn/b8AQSaAjnSKBo7sqFi+3BMm/5gZMDIVXNHoNPN/A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676389; c=relaxed/simple; bh=vE5jQFy5pFyzEGryo9XJ36yYIHZDePx/ABJ6N57nt/A=; h=Cc:References:Content-Type:To:Mime-Version:In-Reply-To:From: Subject:Date:Message-Id; b=A4pVnTRIiDd/Go7VLn3O5RuH57/jfNhpqyyw3QLtEdZUaXoLGbls5IzehF7JJWsnFPnXc4tS084V+GQ8rXEvWOJ9zCuz5ILUv7s+bZODzSN5g150x5k+gncBmt0TESoPFVLA1dgcozg4omHvWM+rmpibVVtytQbAoE4/3XQ7xHg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=W7+I9Ou/; arc=none smtp.client-ip=209.127.230.114 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="W7+I9Ou/" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676377; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=L2M7ctaod3d46KXdrMlsb/SZhOyJC5OxK8ymMPI71u4=; b=W7+I9Ou/nI3c20NqQiT0nV0mYLI1aFnULdYoqXjYButIjG5AaNiRSCC/OPBoseWAVY929M jbVTayt2SDuxVx86f7tyw24e09DEFPDFBCMyaAptFfX18izkSvNCLS2Q3neXX04AXgVctl s5SF3TSSVZf1uM77t5vZ8rF7g7XeFiHp7N/o1qhDXirMvzQHipCYw/4vpbDSwK2d287g+m XMvr3uR+AXHF/oVzBN8hcP4UkM4G0tkyjwPkdWMmIkmiL+tYa58w95vBWaWgo759E+z6CC UMbJi1NNMS2hAuAPmsQWkrDVWYfPisXWiak77l0cjjNv2oJ49TY7dsivK2mPKQ== Cc: , "Chuyi Zhou" References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 X-Lms-Return-Path: Content-Transfer-Encoding: quoted-printable To: , , , , , , , , , , , , Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou From: "Chuyi Zhou" Subject: [RESEND PATCH v5 02/12] smp: Enable preemption early in smp_call_function_single Date: Wed, 13 May 2026 20:45:14 +0800 Message-Id: <20260513124524.2569867-3-zhouchuyi@bytedance.com> Content-Type: text/plain; charset="utf-8" Now smp_call_function_single() disables preemption mainly for the following reasons: - To protect the per-cpu csd_data from concurrent modification by other tasks on the current CPU in the !wait case. For the wait case, synchronization is not a concern as on-stack csd is used. - To prevent the remote online CPU from being offlined. Specifically, we want to ensure that no new IPIs are queued after smpcfd_dying_cpu() has finished. Disabling preemption for the entire execution is unnecessary, especially csd_lock_wait() part does not require preemption protection. This patch enables preemption before csd_lock_wait() to reduce the preemption-disabled critical section. Signed-off-by: Chuyi Zhou Reviewed-by: Muchun Song Reviewed-by: Steven Rostedt (Google) --- kernel/smp.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index b58975480e11..292eefadddbc 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -700,11 +700,16 @@ int smp_call_function_single(int cpu, smp_call_func_t= func, void *info, =20 err =3D generic_exec_single(cpu, csd); =20 + /* + * @csd is stack-allocated when @wait is true. No concurrent access + * except from the IPI completion path, so we can re-enable preemption + * early to reduce latency. + */ + put_cpu(); + if (wait) csd_lock_wait(csd); =20 - put_cpu(); - return err; } EXPORT_SYMBOL(smp_call_function_single); --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-112.ptr.blmpb.com (va-1-112.ptr.blmpb.com [209.127.230.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C269C289E13 for ; Wed, 13 May 2026 12:46:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676397; cv=none; b=TB5MxXQQVq7NXuKEYn5ULMcfvHl54PZ2z30hS7tRm2ypnJbjZodpC6Qh5J0ZFuLCsj13dJZ/Cp+6g4M3OBSHajapIfYJekQ82mYXoxPHxW4LStV7zJSRgWHUqhzU3Vcvocn3dQTgnV26c5sQuQbc0VrL0t6WvSyfTC/Nj0YaXBU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676397; c=relaxed/simple; bh=BlCnDSwEeT4gZlf6A057p3rCOnc2R26LYrNslYlSuBA=; h=To:Subject:In-Reply-To:Cc:From:References:Content-Type:Date: Mime-Version:Message-Id; b=a5sRoGni96CZOJbL3Hwyn7niOjhInoGfS9IM3KMFcnaEPgGcM2bZB2mVwhrRj4isV/daaM5XjHLhwCKzPxgQc9qvD90hJXu4DA6WcZv16CBIHAsbJbMxc8+uQv0vKgYIl1zPuXNkiQHBe9PO8/SjdpDrEbF2whFxlVJ0wEs3+9E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DlP3RZFY; arc=none smtp.client-ip=209.127.230.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DlP3RZFY" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676390; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=usG1cQBZw0E06CdxXRiHWsnG/XND5onF/+2nGw7t910=; b=DlP3RZFYpK6a7wDLYisZmXErWgX7QgING+2dcLfbfrulHEfHtXlI+wy4L2U9Dxb6fuhYjd /Ch90H2rJPmPtQqrkssVxScMtNPaBZbPqULqahP0jKhNLr2FDx+9o/XNbOIR86YZklwsL8 cHiFOCLws9y9UGISHveVD1FuQXamDVndMCLI6XLN71pMCXmo4QErFWRJq+ykbr42QnWesK EXJGIDlwiWjMLacB9cdiAG6ZkF1+v9dGnr55ipr4CofMbB2ut976UOt4Fr4yonvU3NE/G6 Huigla07FyvbzylMV2jJghNjDWUNAACER0cEMm/1IK9lhDWV5lqRdgxm+rZ+5w== To: , , , , , , , , , , , , Subject: [RESEND PATCH v5 03/12] smp: Refactor remote CPU selection in smp_call_function_any() In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou Cc: , "Chuyi Zhou" From: "Chuyi Zhou" References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Date: Wed, 13 May 2026 20:45:15 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Lms-Return-Path: Message-Id: <20260513124524.2569867-4-zhouchuyi@bytedance.com> Content-Transfer-Encoding: quoted-printable X-Mailer: git-send-email 2.20.1 Content-Type: text/plain; charset="utf-8" Currently, smp_call_function_any() disables preemption across the entire process of picking a target CPU, enqueueing the IPI, and synchronously waiting for the remote CPU. Since smp_call_function_single() has already been optimized to re-enable preemption before the synchronous csd_lock_wait(), callers of smp_call_function_any() should also benefit from this optimization to reduce the preemption-disabled critical section. A naive approach would be to simply remove get_cpu() and put_cpu() from smp_call_function_any(), leaving the preemption disablement entirely to smp_call_function_single(). However, doing so opens a dangerous preemption window between picking the remote CPU (e.g., via sched_numa_find_nth_cpu()) and dispatching the IPI inside smp_call_function_single(). If the selected remote CPU is fully offlined during this window, smp_call_function_single() will fail its cpu_online() check and return -ENXIO directly to the caller, violating the guarantee to execute on *any* online CPU in the mask. To safely enable this optimization, this patch refactors the logic of smp_call_function_any() and smp_call_function_single(). By moving the random remote CPU selection into a common __smp_call_function_single(), and keep the entire selection and IPI dispatch process within a single preemption-disabled region. Signed-off-by: Chuyi Zhou --- kernel/smp.c | 48 ++++++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 22 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 292eefadddbc..9e9dab3b0d51 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -641,17 +641,8 @@ void flush_smp_call_function_queue(void) local_irq_restore(flags); } =20 -/** - * smp_call_function_single - Run a function on a specific CPU - * @cpu: Specific target CPU for this function. - * @func: The function to run. This must be fast and non-blocking. - * @info: An arbitrary pointer to pass to the function. - * @wait: If true, wait until function has completed on other CPUs. - * - * Returns: %0 on success, else a negative status code. - */ -int smp_call_function_single(int cpu, smp_call_func_t func, void *info, - int wait) +static int __smp_call_function_single(int cpu, smp_call_func_t func, + void *info, const struct cpumask *mask, int wait) { call_single_data_t *csd; call_single_data_t csd_stack =3D { @@ -668,6 +659,14 @@ int smp_call_function_single(int cpu, smp_call_func_t = func, void *info, */ this_cpu =3D get_cpu(); =20 + if (mask) { + /* Try for same CPU (cheapest) */ + if (!cpumask_test_cpu(this_cpu, mask)) + cpu =3D sched_numa_find_nth_cpu(mask, 0, cpu_to_node(this_cpu)); + else + cpu =3D this_cpu; + } + /* * Can deadlock when called with interrupts disabled. * We allow cpu's that are not yet online though, as no one else can @@ -712,6 +711,21 @@ int smp_call_function_single(int cpu, smp_call_func_t = func, void *info, =20 return err; } + +/** + * smp_call_function_single - Run a function on a specific CPU + * @cpu: Specific target CPU for this function. + * @func: The function to run. This must be fast and non-blocking. + * @info: An arbitrary pointer to pass to the function. + * @wait: If true, wait until function has completed on other CPUs. + * + * Returns: %0 on success, else a negative status code. + */ +int smp_call_function_single(int cpu, smp_call_func_t func, void *info, + int wait) +{ + return __smp_call_function_single(cpu, func, info, NULL, wait); +} EXPORT_SYMBOL(smp_call_function_single); =20 /** @@ -776,17 +790,7 @@ EXPORT_SYMBOL_GPL(smp_call_function_single_async); int smp_call_function_any(const struct cpumask *mask, smp_call_func_t func, void *info, int wait) { - unsigned int cpu; - int ret; - - /* Try for same CPU (cheapest) */ - cpu =3D get_cpu(); - if (!cpumask_test_cpu(cpu, mask)) - cpu =3D sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu)); - - ret =3D smp_call_function_single(cpu, func, info, wait); - put_cpu(); - return ret; + return __smp_call_function_single(-1, func, info, mask, wait); } EXPORT_SYMBOL_GPL(smp_call_function_any); =20 --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-113.ptr.blmpb.com (va-1-113.ptr.blmpb.com [209.127.230.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E533421F17 for ; Wed, 13 May 2026 12:46:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.113 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676412; cv=none; b=XP411rqSnl4km8+yzAt2NlrXseqslRCLRkvsbkX7TyyicJGy0ZHlbFR2bKNLZat8jYV0nrnHzXZvrEjJke7bHkS7+R5Dy8qu44bYso9cqaKm4R2J/Ci7MVqeBcDhYEGPw3qvdIAhb/BmUUMjIl4GoPOf1KROSyZcTooj1q63mJU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676412; c=relaxed/simple; bh=u79xGmflwucI+YEOnG0UZY0wDGbdoqD3LQdBnD8y8jc=; h=From:Message-Id:References:In-Reply-To:Content-Type:To:Subject: Date:Cc:Mime-Version; b=WN2OqXuFEHzqntKLbbv2yBI49Ih5/j6/YRo7PNnbRvb4XIgSEx03T3WTo1ijfo2GBxzIFRmzfshZX7ArqFDe7XxhpisrV865RI0bNDTyvlynK2AJHbERt6G3BQOkI8VqQQIVvVrFFCXmmRfQ5zxNoaYyUe104b9RRHwHo/6+4UI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=nXXen8xS; arc=none smtp.client-ip=209.127.230.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="nXXen8xS" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676406; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=OglEIZT1ufBPtbBBcdcHmPqq6F3370jITSD7W3xD394=; b=nXXen8xSUSnV8wUAAkTaOi3QY6FKj5YCSDyH+77Y7d7Jnd6L7AAhJyZFVNJh1fQLVuTrWh yc9oO5RmPU4ZziL+expAatxxJt+bk5VEVy0UVh54yoMHNK1QGABKxXPX3L9vR4ebghbnJo Bsbd/NJZdEhhjjZhOuSyKCv6sVSaOxcGbmnO7TvEKTS8v0CO5glkvHqVcf/J/uj8QtZImm Kk0pR7J9EdvlVte/BG15+8rsPo9e4CxrkdHCncZMIclpCM6z8sVZtWVBF3pK1iJQrC+iYv xP22kVLFusvDEV6TpundN1BSaxpOjfzM0kpbW+oMFvlb+yerPf/EnnIiuEWGHg== From: "Chuyi Zhou" Message-Id: <20260513124524.2569867-5-zhouchuyi@bytedance.com> X-Lms-Return-Path: X-Mailer: git-send-email 2.20.1 Content-Transfer-Encoding: quoted-printable References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou To: , , , , , , , , , , , , Subject: [RESEND PATCH v5 04/12] smp: Use task-local IPI cpumask in smp_call_function_many_cond() Date: Wed, 13 May 2026 20:45:16 +0800 Cc: , "Chuyi Zhou" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch prepares the task-local IPI cpumask during thread creation, and uses the local cpumask to replace the percpu cfd cpumask in smp_call_function_many_cond(). We will enable preemption during csd_lock_wait() later, and this can prevent concurrent access to the cfd->cpumask from other tasks on the current CPU. For cases where cpumask_size() is smaller than or equal to the pointer size, it tries to stash the cpumask in the pointer itself to avoid extra memory allocations. Signed-off-by: Chuyi Zhou --- include/linux/sched.h | 6 +++++ include/linux/smp.h | 20 +++++++++++++++ kernel/fork.c | 9 ++++++- kernel/smp.c | 59 ++++++++++++++++++++++++++++++++++++++----- 4 files changed, 87 insertions(+), 7 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 368c7b4d7cb5..bb2c53279412 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1348,6 +1348,12 @@ struct task_struct { struct list_head perf_event_list; struct perf_ctx_data __rcu *perf_ctx_data; #endif +#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPTION) + union { + cpumask_t *ipi_mask_ptr; + unsigned long ipi_mask_val; + }; +#endif #ifdef CONFIG_DEBUG_PREEMPT unsigned long preempt_disable_ip; #endif diff --git a/include/linux/smp.h b/include/linux/smp.h index 6925d15ccaa7..b2ecb9a5edd0 100644 --- a/include/linux/smp.h +++ b/include/linux/smp.h @@ -167,6 +167,12 @@ void smp_call_function_many(const struct cpumask *mask, int smp_call_function_any(const struct cpumask *mask, smp_call_func_t func, void *info, int wait); =20 +#ifdef CONFIG_PREEMPTION +int smp_task_ipi_mask_alloc(struct task_struct *task); +void smp_task_ipi_mask_free(struct task_struct *task); +cpumask_t *smp_task_ipi_mask(struct task_struct *cur); +#endif + void kick_all_cpus_sync(void); void wake_up_all_idle_cpus(void); bool cpus_peek_for_pending_ipi(const struct cpumask *mask); @@ -310,4 +316,18 @@ bool csd_lock_is_stuck(void); static inline bool csd_lock_is_stuck(void) { return false; } #endif =20 +#if !defined(CONFIG_SMP) || !defined(CONFIG_PREEMPTION) +static inline int smp_task_ipi_mask_alloc(struct task_struct *task) +{ + return 0; +} +static inline void smp_task_ipi_mask_free(struct task_struct *task) +{ +} +static inline cpumask_t *smp_task_ipi_mask(struct task_struct *cur) +{ + return NULL; +} +#endif + #endif /* __LINUX_SMP_H */ diff --git a/kernel/fork.c b/kernel/fork.c index 5f3fdfdb14c7..bf485c51c447 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -535,6 +535,7 @@ void free_task(struct task_struct *tsk) #endif release_user_cpus_ptr(tsk); scs_release(tsk); + smp_task_ipi_mask_free(tsk); =20 #ifndef CONFIG_THREAD_INFO_IN_TASK /* @@ -932,10 +933,14 @@ static struct task_struct *dup_task_struct(struct tas= k_struct *orig, int node) #endif account_kernel_stack(tsk, 1); =20 - err =3D scs_prepare(tsk, node); + err =3D smp_task_ipi_mask_alloc(tsk); if (err) goto free_stack; =20 + err =3D scs_prepare(tsk, node); + if (err) + goto free_ipi_mask; + #ifdef CONFIG_SECCOMP /* * We must handle setting up seccomp filters once we're under @@ -1006,6 +1011,8 @@ static struct task_struct *dup_task_struct(struct tas= k_struct *orig, int node) #endif return tsk; =20 +free_ipi_mask: + smp_task_ipi_mask_free(tsk); free_stack: exit_task_stack_account(tsk); free_thread_stack(tsk); diff --git a/kernel/smp.c b/kernel/smp.c index 9e9dab3b0d51..630c8e5a635d 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -794,6 +794,44 @@ int smp_call_function_any(const struct cpumask *mask, } EXPORT_SYMBOL_GPL(smp_call_function_any); =20 +static DEFINE_STATIC_KEY_FALSE(ipi_mask_inlined); + +#ifdef CONFIG_PREEMPTION + +int smp_task_ipi_mask_alloc(struct task_struct *task) +{ + if (static_branch_unlikely(&ipi_mask_inlined)) + return 0; + + task->ipi_mask_ptr =3D kmalloc(cpumask_size(), GFP_KERNEL); + if (!task->ipi_mask_ptr) + return -ENOMEM; + + return 0; +} + +void smp_task_ipi_mask_free(struct task_struct *task) +{ + if (static_branch_unlikely(&ipi_mask_inlined)) + return; + + kfree(task->ipi_mask_ptr); +} + +cpumask_t *smp_task_ipi_mask(struct task_struct *cur) +{ + /* + * If cpumask_size() is smaller than or equal to the pointer + * size, it stashes the cpumask in the pointer itself to + * avoid extra memory allocations. + */ + if (static_branch_unlikely(&ipi_mask_inlined)) + return (cpumask_t *)&cur->ipi_mask_val; + + return cur->ipi_mask_ptr; +} +#endif + /* * Flags to be used as scf_flags argument of smp_call_function_many_cond(). * @@ -811,11 +849,18 @@ static void smp_call_function_many_cond(const struct = cpumask *mask, int cpu, last_cpu, this_cpu =3D smp_processor_id(); struct call_function_data *cfd; bool wait =3D scf_flags & SCF_WAIT; + struct cpumask *cpumask, *task_mask; + bool preemptible_wait; int nr_cpus =3D 0; bool run_remote =3D false; =20 lockdep_assert_preemption_disabled(); =20 + task_mask =3D smp_task_ipi_mask(current); + preemptible_wait =3D task_mask; + cfd =3D this_cpu_ptr(&cfd_data); + cpumask =3D preemptible_wait ? task_mask : cfd->cpumask; + /* * Can deadlock when called with interrupts disabled. * We allow cpu's that are not yet online though, as no one else can @@ -836,16 +881,15 @@ static void smp_call_function_many_cond(const struct = cpumask *mask, =20 /* Check if we need remote execution, i.e., any CPU excluding this one. */ if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) { - cfd =3D this_cpu_ptr(&cfd_data); - cpumask_and(cfd->cpumask, mask, cpu_online_mask); - __cpumask_clear_cpu(this_cpu, cfd->cpumask); + cpumask_and(cpumask, mask, cpu_online_mask); + __cpumask_clear_cpu(this_cpu, cpumask); =20 cpumask_clear(cfd->cpumask_ipi); - for_each_cpu(cpu, cfd->cpumask) { + for_each_cpu(cpu, cpumask) { call_single_data_t *csd =3D per_cpu_ptr(cfd->csd, cpu); =20 if (cond_func && !cond_func(cpu, info)) { - __cpumask_clear_cpu(cpu, cfd->cpumask); + __cpumask_clear_cpu(cpu, cpumask); continue; } =20 @@ -896,7 +940,7 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, } =20 if (run_remote && wait) { - for_each_cpu(cpu, cfd->cpumask) { + for_each_cpu(cpu, cpumask) { call_single_data_t *csd; =20 csd =3D per_cpu_ptr(cfd->csd, cpu); @@ -1010,6 +1054,9 @@ EXPORT_SYMBOL(nr_cpu_ids); void __init setup_nr_cpu_ids(void) { set_nr_cpu_ids(find_last_bit(cpumask_bits(cpu_possible_mask), NR_CPUS) + = 1); + + if (IS_ENABLED(CONFIG_PREEMPTION) && cpumask_size() <=3D sizeof(unsigned = long)) + static_branch_enable(&ipi_mask_inlined); } =20 /* Called by boot processor to activate the rest. */ --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE072402B83 for ; Wed, 13 May 2026 12:47:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676428; cv=none; b=LDYmQKHqMkYXQFQ0MEbcrQiQkG8MfxjQ2ttiIiqojTWBaptVuN1FLjDfa8JZka+iS3A1VA7qHwXw+Uog9vJ67aY7CRkqm8O0rOGtYP+n5gc+SK8hjOqI+pB3fqlYQ5daZoTIwxUMtAiXL+zTF7NLxCHPoOHgUuMgUNvK2Qfv96U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676428; c=relaxed/simple; bh=GxBMfxkAummKY4JjXv/+hvPrFEfGANZDM6PvoaAJIwA=; h=From:Subject:Mime-Version:To:Cc:Message-Id:In-Reply-To:Date: References:Content-Type; b=mvHznnOspurVWFSXMvMPhiOYTOlCRU9uBWDEB25JXxNVQUhpTOYKqgXm0FLnYKjAV3s0Bl5AVV0ccXO/FP+lGlGjup8EkBraojkGC6rmTSB7nVdD6Wbuw4Ww21//tscUwNw0W6IU+d+IpTJ33DN9wN9k2eeITU6eNbysOpijsQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=e/8K8+QB; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="e/8K8+QB" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676419; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=Gj20M9mnplaipkImWhK7gz46CRqEXvYt8wjCUsqg14M=; b=e/8K8+QBEX/7r6GZALTF1AeNwW2MWQFkIWvf7F4e/SP+tsBCbKTViEzZfyvsseaZrJOkug HuIsbn1aW6JawMjsXCOxwDWLQ67V1UXy9VjwH06Bu+VXpiodImRy6y5XYdgi+B0RGUoqTZ ydvtWKe4M413jF0XIqrBhGi7pnUS+YqKr5aTGWp5L/x6b2keVXDwKj/dQVfiVq/ayohtNz RQkc4mTmZWf7li+o98gKO6IGobHjL/468f8N1n6ptPbU0NmvTLP8h3j1+hqQUqxQ9YlN4e R9/5VNHn11xBck9fPTpY7lqYZIEEQUp9qIGc7Ka1iAYhv1koeyrrLI6lsHU02Q== From: "Chuyi Zhou" Subject: [RESEND PATCH v5 05/12] smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 To: , , , , , , , , , , , , Cc: , "Chuyi Zhou" Message-Id: <20260513124524.2569867-6-zhouchuyi@bytedance.com> In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Date: Wed, 13 May 2026 20:45:17 +0800 X-Mailer: git-send-email 2.20.1 X-Original-From: Chuyi Zhou References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Lms-Return-Path: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Later patch would enable preemption during csd_lock_wait() in smp_call_function_many_cond(), which may cause access cfd->csd data that has already been freed in smpcfd_dead_cpu(). One way to fix the above issue is to use the RCU mechanism to protect the csd data and wait for all read critical sections to exit before freeing the memory in smpcfd_dead_cpu(), but this could delay CPU shutdown. This patch chooses a simpler approach: allocate the percpu csd on the UP side only once and skip freeing the csd memory in smpcfd_dead_cpu(). Suggested-by: Sebastian Andrzej Siewior Signed-off-by: Chuyi Zhou Acked-by: Muchun Song --- kernel/smp.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 630c8e5a635d..ab22a0697db2 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -63,7 +63,15 @@ int smpcfd_prepare_cpu(unsigned int cpu) free_cpumask_var(cfd->cpumask); return -ENOMEM; } - cfd->csd =3D alloc_percpu(call_single_data_t); + + /* + * The percpu csd is allocated only once and never freed. + * This ensures that smp_call_function_many_cond() can safely + * access the csd of an offlined CPU if it gets preempted + * during csd_lock_wait(). + */ + if (!cfd->csd) + cfd->csd =3D alloc_percpu(call_single_data_t); if (!cfd->csd) { free_cpumask_var(cfd->cpumask); free_cpumask_var(cfd->cpumask_ipi); @@ -79,7 +87,6 @@ int smpcfd_dead_cpu(unsigned int cpu) =20 free_cpumask_var(cfd->cpumask); free_cpumask_var(cfd->cpumask_ipi); - free_percpu(cfd->csd); return 0; } =20 --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-113.ptr.blmpb.com (va-1-113.ptr.blmpb.com [209.127.230.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 691F640DFBE for ; Wed, 13 May 2026 12:47:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.113 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676439; cv=none; b=IHRVWPkGRYK54vrFtm1cetsbknZ3Cje+UwVyD+pmWkcOSA5bXX8WUPgpA37Ri6Kt+pZXU8ulgyJQs2iebj19GP+m9VErdbScSebTawKZEilK0b1BbdBGoyZazWf1UJmlx1NAMNhiiShpU5Lu6Sb1QtYdzLB8aUZSOgbJ/nlWN6U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676439; c=relaxed/simple; bh=Yg6+ysqNpKUAvKlsN+fcz5tkal+wkG1yuZtlq9dSwmc=; h=Mime-Version:Content-Type:Cc:References:From:Subject:Date:To: Message-Id:In-Reply-To; b=CxJP3q/MHPwnUWiDHX+ikaCjQxD2SRv6vHSqbSReLoqRAOS3ns7OPdxK18EjGTmRVchx4YNXtOCOCZyX9ka3PHkszFS6huE89B0KFQQbzCc/3l2Kw69SH13Pm9zAta0o10eY0FcF7z4M+f3V+8dbfAqEqXyg3Bd8/Wq94+V+Ml0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=WEKeWTNN; arc=none smtp.client-ip=209.127.230.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="WEKeWTNN" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676432; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=BXpZPH4dp2a8rB0CKaTWsQkpCpkV/kf2CjnJVZI7f/8=; b=WEKeWTNNwHLFG736CjkskUHeMtXX5UhfLBfEqYvBZ9w0D4XvoEPoSRoU2Q3Z6z7H8L+wv6 a3NHAIV+BnlUXk7ZyZeAgHjj1isM25z/0s725L2+jc4MpgJtsO3YafY9ahLHoBnHxT12RG +Ovifdz20lPZfUrsaTUxhIIgVI+NWOg8QRHLEAKH3WGzQrvpUt0jQyQihQTb11l98SO775 NwhtYf68KvP2T2McaxnyCMbJHkiESAxpbcy2kMgvfOrP1mn5NaKX/Dz29faI+x7kCUsHoa MD4G1nDjSyf2K4h1hTUcad/auaqbQ3kxKjdlLb5r67iCg/ziK9mfYH8n0Wp5nQ== Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Cc: , "Chuyi Zhou" Content-Transfer-Encoding: quoted-printable X-Original-From: Chuyi Zhou References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 From: "Chuyi Zhou" Subject: [RESEND PATCH v5 06/12] smp: Enable preemption early in smp_call_function_many_cond Date: Wed, 13 May 2026 20:45:18 +0800 To: , , , , , , , , , , , , Message-Id: <20260513124524.2569867-7-zhouchuyi@bytedance.com> In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Lms-Return-Path: Content-Type: text/plain; charset="utf-8" Disabling preemption entirely during smp_call_function_many_cond() was primarily for the following reasons: - To prevent the remote online CPU from going offline. Specifically, we want to ensure that no new csds are queued after smpcfd_dying_cpu() has finished. Therefore, preemption must be disabled until all necessary IPIs are sent. - To prevent current CPU from going offline. Being migrated to another CPU and calling csd_lock_wait() may cause UAF due to smpcfd_dead_cpu() during the current CPU offline process. - To protect the per-cpu cfd_data from concurrent modification by other tasks on the current CPU. cfd_data contains cpumasks and per-cpu csds. Before enqueueing a csd, we block on the csd_lock() to ensure the previous async csd->func() has completed, and then initialize csd->func and csd->info. After sending the IPI, we spin-wait for the remote CPU to call csd_unlock(). Actually the csd_lock mechanism already guarantees csd serialization. If preemption occurs during csd_lock_wait, other concurrent smp_call_function_many_cond calls will simply block until the previous csd->func() completes: task A task B sd->func =3D fun_a send ipis preempted by B ---------------> csd_lock(csd); // block until last // fun_a finished csd->func =3D func_b; csd->info =3D info; ... send ipis switch back to A <--------------- csd_lock_wait(csd); // block until remote finish func_* Previous patches replaced the per-cpu cfd->cpumask with task-local cpumask, and the percpu csd is allocated only once and is never freed to ensure we can safely access csd. Now we can enable preemption before csd_lock_wait() which makes the potentially unpredictable csd_lock_wait() preemptible and migratable. Signed-off-by: Chuyi Zhou --- kernel/smp.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index ab22a0697db2..a55919f39cf4 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -853,7 +853,7 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, unsigned int scf_flags, smp_cond_func_t cond_func) { - int cpu, last_cpu, this_cpu =3D smp_processor_id(); + int cpu, last_cpu, this_cpu; struct call_function_data *cfd; bool wait =3D scf_flags & SCF_WAIT; struct cpumask *cpumask, *task_mask; @@ -861,10 +861,10 @@ static void smp_call_function_many_cond(const struct = cpumask *mask, int nr_cpus =3D 0; bool run_remote =3D false; =20 - lockdep_assert_preemption_disabled(); - task_mask =3D smp_task_ipi_mask(current); - preemptible_wait =3D task_mask; + preemptible_wait =3D task_mask && preemptible(); + + this_cpu =3D get_cpu(); cfd =3D this_cpu_ptr(&cfd_data); cpumask =3D preemptible_wait ? task_mask : cfd->cpumask; =20 @@ -946,6 +946,19 @@ static void smp_call_function_many_cond(const struct c= pumask *mask, local_irq_restore(flags); } =20 + /* + * We may block in csd_lock_wait() for a significant amount of time, + * especially when interrupts are disabled or with a large number of + * remote CPUs. Try to enable preemption before csd_lock_wait(). + * + * Use the task_mask instead of cfd->cpumask to avoid concurrency + * modification from tasks on the same cpu. If preemption occurs during + * csd_lock_wait, other concurrent smp_call_function_many_cond() calls + * will simply block until the previous csd->func() completes. + */ + if (preemptible_wait) + put_cpu(); + if (run_remote && wait) { for_each_cpu(cpu, cpumask) { call_single_data_t *csd; @@ -954,6 +967,9 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, csd_lock_wait(csd); } } + + if (!preemptible_wait) + put_cpu(); } =20 /** @@ -965,8 +981,7 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, * on other CPUs. * * You must not call this function with disabled interrupts or from a - * hardware interrupt handler or from a bottom half handler. Preemption - * must be disabled when calling this function. + * hardware interrupt handler or from a bottom half handler. * * @func is not called on the local CPU even if @mask contains it. Consid= er * using on_each_cpu_cond_mask() instead if this is not desirable. --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A98CF289E13 for ; Wed, 13 May 2026 12:47:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676451; cv=none; b=cbvvhse0lYiocXPvvxn2q1IOPxo1af97C9PVeUpeEZR7vtADM6hsA68FTteeD8LDWhy8/NAod/EcfuRBTGS5QSbCs7cz3ODVhTni+Yh2BGxy3mEzgA3EYnGMc7O7e5NHccR7WenI/GKmS8reFHDVePx06WCo8b+lxEGxm7oX7ms= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676451; c=relaxed/simple; bh=+xc6B2ARcpxzfjvdfu52u9M+OGCrqIv+iyguFipkTkw=; h=To:Subject:Mime-Version:In-Reply-To:From:Date:References: Content-Type:Cc:Message-Id; b=SZPuVNb6FqsPbUhQuhL3fYwmgttElK3u7rFPq9NXF3pAeNNcrBA5e18RYAxzRcrkl4XGEjaNvE29ub5Co8ZLMA0j4nd6q1c1FQJdsTdF9vveRBINGWr6Y/0VuInxZ3c6fbnEDApHccL0v85MHmvw/j822TTR4T1paNvxY5pddm4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=mrD5fxVZ; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="mrD5fxVZ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676446; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=83sW+8PHtjtFZakm04qQMgkHX/OimsM0Gnb646vRzsA=; b=mrD5fxVZYxqgNsseFrPYWPz0RNVHcWc2EDRJFdvm9bFEHAMhJMLsbHmdzcckUBwn7Rtebz 7baLq/sRVbhF4F5SX/E9wvp7i3/KBmOXfKsmdF0H6K2vIRNRg9jq7sVTDKrqrf6smtFjP0 +gTkdPuWYB7/dn4dmo+15DN4Nt3u2Y3E1yVPLcyPgeQeCjzz5aWCfT36XuZANVq4dJBHSW xistgvXxPMGQn1/mYl1CfmJl8/IKS9+MQNUnvOhVv4xw5bjmKx8xIci2ZtHC+pqloRyJyS O8EYXwEDYcOawJhC8M9DOjimKt6jHDQ/zzZXI8utLUIPw73X0mmRjM/WMwRYbA== X-Lms-Return-Path: To: , , , , , , , , , , , , Subject: [RESEND PATCH v5 07/12] smp: Remove preempt_disable from smp_call_function Content-Transfer-Encoding: quoted-printable Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> From: "Chuyi Zhou" Date: Wed, 13 May 2026 20:45:19 +0800 References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 X-Original-From: Chuyi Zhou Cc: , "Chuyi Zhou" Message-Id: <20260513124524.2569867-8-zhouchuyi@bytedance.com> Content-Type: text/plain; charset="utf-8" Now smp_call_function_many_cond() internally handles the preemption logic, so smp_call_function() does not need to explicitly disable preemption. Remove preempt_{enable, disable} from smp_call_function(). Signed-off-by: Chuyi Zhou Reviewed-by: Muchun Song --- kernel/smp.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index a55919f39cf4..fa35c5725c98 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -1008,9 +1008,8 @@ EXPORT_SYMBOL(smp_call_function_many); */ void smp_call_function(smp_call_func_t func, void *info, int wait) { - preempt_disable(); - smp_call_function_many(cpu_online_mask, func, info, wait); - preempt_enable(); + smp_call_function_many_cond(cpu_online_mask, func, info, + wait ? SCF_WAIT : 0, NULL); } EXPORT_SYMBOL(smp_call_function); =20 --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-111.ptr.blmpb.com (va-1-111.ptr.blmpb.com [209.127.230.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC275245031 for ; Wed, 13 May 2026 12:47:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.111 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676467; cv=none; b=k8FZEr+ERyk03roZ7Gly41chyWXI4TK0SXAgeCmCY4lDnNFvu5L98vyOH42Kr7BVBTseKkIAXk1LLQl9FZs4jBm7uKrt9BM3aemXJJztcZj3tdejhURvTusS2PcBosdbJvo0Kumhh250zVXLUEEH+xQ4SIP8+jcxlub7KvuHdEs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676467; c=relaxed/simple; bh=4SHqM3yOPN4P2ufCvEjea0q5MBnAhIcC3dgoFolcwqg=; h=Content-Type:From:References:To:In-Reply-To:Subject:Mime-Version: Cc:Date:Message-Id; b=j86d19irE+kd7CJOc3Yo0SkvvQwaFP3pu1S/Y9jZS4vllas+BHelkwnpo8pFl/QiBT65gX8X28d75s0wUZo4Vfa3dquLd6ntX/JDJFdEvruzhuNDOIQvDDFzFnEnEMT2W7ZcVbK+1EhPXXHE4wwDmTIM7arn9lqMLhOLlcNDFHc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=piS+L5Oj; arc=none smtp.client-ip=209.127.230.111 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="piS+L5Oj" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676460; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=Ak7Dzg2NCPfamL2k7vvS6ThY1nprXzUendPOHV7QBkU=; b=piS+L5Oj23XkjvgK7gEIhCmtUx3yNYClQgRuiqSawHo3CPZiigzSV3Dy/l8D5kQG5VWjTk Ov0h+fiutNILfgXvbzkSAISUfE0jloJiJ3o6s4MRDbGUumIs5cqck/gp48e9BBNBQ2DQqY z/RZmqfKzv22tik6kXbIGfrsaHIG3t3GeDrjAM01yyTvNMEnZXgEnDhCBgJ/RyMjQAis9R DbVh9h/LJZW/lKIhzm8ou+5knKXT0aOj75sOz228qr3XazCUi9oHZon59YWO103rReGUHq ySOmc2sIvMBCLAzstF6ES/QP3tPxH24WJf2VIBJNtimLluHeMudByBn2Y1HYPw== From: "Chuyi Zhou" Content-Transfer-Encoding: quoted-printable References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Lms-Return-Path: X-Mailer: git-send-email 2.20.1 To: , , , , , , , , , , , , In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou Subject: [RESEND PATCH v5 08/12] smp: Remove preempt_disable from on_each_cpu_cond_mask Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Cc: , "Chuyi Zhou" Date: Wed, 13 May 2026 20:45:20 +0800 Message-Id: <20260513124524.2569867-9-zhouchuyi@bytedance.com> Content-Type: text/plain; charset="utf-8" Now smp_call_function_many_cond() internally handles the preemption logic, so on_each_cpu_cond_mask does not need to explicitly disable preemption. Remove preempt_{enable, disable} from on_each_cpu_cond_mask(). Signed-off-by: Chuyi Zhou Reviewed-by: Muchun Song --- kernel/smp.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index fa35c5725c98..145f097a028b 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -1132,9 +1132,7 @@ void on_each_cpu_cond_mask(smp_cond_func_t cond_func,= smp_call_func_t func, if (wait) scf_flags |=3D SCF_WAIT; =20 - preempt_disable(); smp_call_function_many_cond(mask, func, info, scf_flags, cond_func); - preempt_enable(); } EXPORT_SYMBOL(on_each_cpu_cond_mask); =20 --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-114.ptr.blmpb.com (va-1-114.ptr.blmpb.com [209.127.230.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E70AF423147 for ; Wed, 13 May 2026 12:47:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.114 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676480; cv=none; b=Pqrd0Ik+Eg4VTa0RsFHyljV2F5X1mgdcrURK/ZmErvoMc/jOU8I75JDwBXQccS9SKL4xaTTqAuUa5+yE5G9MdjVuT49kG+GEcOpe8TCWBx72Hgl9m2Ipt9VCrTCB5bwKpCH8HpFdp/L0mXyjdSiaxlQ/sQhie1KLcPQrfuGNB80= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676480; c=relaxed/simple; bh=fF5rNKBntigPfaLSBL5JPchOMBc9gOsuc0c/gcdz6x4=; h=From:Subject:Message-Id:References:To:Cc:Mime-Version: Content-Type:In-Reply-To:Date; b=q8rlDg3ktkAkHmdRAAbHCo/UT0i5rFW/PP5zv+QRZ1AYRNNQlUUM+REqHFJ9Ynohgf/8rqeV88OmrNUcaKSUgn/OuE8O6mAhzIQY9Q1ys1rm9Y+kKwPR27AqfJ7HQ2cvqZD/Rorv3GZR4La8OzCnJdq/cpvD0Dk3aN/mrzDAYDA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=VbDE/GHp; arc=none smtp.client-ip=209.127.230.114 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="VbDE/GHp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676474; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=ICWN1OsCdHFaPjXkwVSuhRMSca3SDV1TAbqsiUSfFFA=; b=VbDE/GHpEi+ZLHIMQW8N8NdbM7LLXfcsUuNTpDYypeQeUyeDzljE1hTRjxFkSE6Z+ftvfc GII4K0tDT6xpsExFila6ovUX9nxVC0vxo8a5PS4/F44hrL6Ly9805ztlm4kcSPIKFfmWUI KN5ESNQSobOcs4pxZS7Wk4xIVe+dVlA8ORrL1WoxjhyQyot3AfCgnKmSoSTrhNRprAgsjm Ij4olD9Bz7dvRMwUXfQXFaH0PXA4u6y3mWAbQ0NhZOLz93L0NW20/wbJi/UKpSxxv+O55k T9n+od58q06gpzkF9wUNuqHliDQN+34DgfCih5poCaHlhEqIQU4PTRAQ4brfsA== Content-Transfer-Encoding: quoted-printable From: "Chuyi Zhou" Subject: [RESEND PATCH v5 09/12] scftorture: Remove preempt_disable in scftorture_invoke_one Message-Id: <20260513124524.2569867-10-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> To: , , , , , , , , , , , , Cc: , "Chuyi Zhou" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Lms-Return-Path: In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Date: Wed, 13 May 2026 20:45:21 +0800 X-Original-From: Chuyi Zhou Content-Type: text/plain; charset="utf-8" Previous patches make smp_call*() functions handle preemption logic internally. Thus, the explicit preempt_disable() surrounding these calls becomes unnecessary. Furthermore, keeping the external preempt_disable() would prevent scftorture from exercising the newly narrowed internal preemption-disabled regions during IPI dispatch. This patch removes the preempt_{enable, disable} pairs in scftorture_invoke_one(). Removing this preemption protection could expose a race condition with CPU hotplug when use_cpus_read_lock is false. Specifically, for multi-cast operations (SCF_PRIM_MANY or SCF_PRIM_ALL), if only 1 CPU is online, smp_call_function_many() correctly skips sending IPIs and leaves scfc_out as false. Without preemption disabled, a CPU hotplug thread could preempt the test thread, bring a second CPU online, and increment num_online_cpus(). When the test thread resumes, the validation check would see num_online_cpus() > 1 and falsely trigger the memory-ordering warning, leaking the scfcp structure. To avoid this potential false positive, restrict the num_online_cpus() > 1 condition to only apply when use_cpus_read_lock is true, ensuring the CPU count remains stable during evaluation. Signed-off-by: Chuyi Zhou --- kernel/scftorture.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index 327c315f411c..2082f9b44370 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -348,6 +348,8 @@ static void scftorture_invoke_one(struct scf_statistics= *scfp, struct torture_ra int ret =3D 0; struct scf_check *scfcp =3D NULL; struct scf_selector *scfsp =3D scf_sel_rand(trsp); + bool is_single =3D (scfsp->scfs_prim =3D=3D SCF_PRIM_SINGLE || + scfsp->scfs_prim =3D=3D SCF_PRIM_SINGLE_RPC); =20 if (scfsp->scfs_prim =3D=3D SCF_PRIM_SINGLE || scfsp->scfs_wait) { scfcp =3D kmalloc_obj(*scfcp, GFP_ATOMIC); @@ -364,8 +366,6 @@ static void scftorture_invoke_one(struct scf_statistics= *scfp, struct torture_ra } if (use_cpus_read_lock) cpus_read_lock(); - else - preempt_disable(); switch (scfsp->scfs_prim) { case SCF_PRIM_RESCHED: if (IS_BUILTIN(CONFIG_SCF_TORTURE_TEST)) { @@ -411,13 +411,10 @@ static void scftorture_invoke_one(struct scf_statisti= cs *scfp, struct torture_ra if (!ret) { if (use_cpus_read_lock) cpus_read_unlock(); - else - preempt_enable(); + wait_for_completion(&scfcp->scfc_completion); if (use_cpus_read_lock) cpus_read_lock(); - else - preempt_disable(); } else { scfp->n_single_rpc_ofl++; scf_add_to_free_list(scfcp); @@ -452,7 +449,7 @@ static void scftorture_invoke_one(struct scf_statistics= *scfp, struct torture_ra scfcp->scfc_out =3D true; } if (scfcp && scfsp->scfs_wait) { - if (WARN_ON_ONCE((num_online_cpus() > 1 || scfsp->scfs_prim =3D=3D SCF_P= RIM_SINGLE) && + if (WARN_ON_ONCE(((use_cpus_read_lock && num_online_cpus() > 1) || is_si= ngle) && !scfcp->scfc_out)) { pr_warn("%s: Memory-ordering failure, scfs_prim: %d.\n", __func__, scfs= p->scfs_prim); atomic_inc(&n_mb_out_errs); // Leak rather than trash! @@ -463,8 +460,6 @@ static void scftorture_invoke_one(struct scf_statistics= *scfp, struct torture_ra } if (use_cpus_read_lock) cpus_read_unlock(); - else - preempt_enable(); if (allocfail) schedule_timeout_idle((1 + longwait) * HZ); // Let no-wait handlers com= plete. else if (!(torture_random(trsp) & 0xfff)) --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-111.ptr.blmpb.com (va-1-111.ptr.blmpb.com [209.127.230.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6AD22401A1D for ; Wed, 13 May 2026 12:48:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.111 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676494; cv=none; b=OQ6fbyysagFP2jUYG1oRZUDQAYTfyC1yGZQhPEz23wZsvDbcq0tDrkfmkRWV8fha9yhrxlKCO2Ct/k3GXO6r2y8fdPIi+KlVS0Nbo19M8H77grwloHfvTIl9IhugHnx8YKaWkSDA6l25vHh1TGiYx30yMNW5tn/oJlnria16oD4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676494; c=relaxed/simple; bh=yKMOFqHHPiGM6QdNGInGyQVsW7G2+Z6vzitvihpbaCE=; h=Subject:Message-Id:References:To:From:Date:Content-Type: In-Reply-To:Cc:Mime-Version; b=j+5e7sK82KYco2QogndZziysTjS4LS/UPlZInHNtOOKPPs4nDDHRcav/Y3tkPeYws0NVu/HeQICXtTowpfLEBxcquq7cP5iGp4U+SIfUM+duagFp1pJLWlNbI1lQQbMCE09QMr+zl8HEpOPg8E1DX6WSlkyOM2dK//b5BbbyGm8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=NFcIjy1g; arc=none smtp.client-ip=209.127.230.111 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="NFcIjy1g" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676487; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=aUK3r6pXI/HwsoVBOCdW4m6bVqpLkwnx6XaS+jnRFrM=; b=NFcIjy1gD8w8Ui+0XjDvp36YgXB8MDZY8V9iMxuNsxduSkRug8aw/JZwVtEa3eka+hyglz TWMHzpY7k5rhZyhzaHVmXkGRR3mCDjRnDZ40Yhn+KikoMWsPh+GbqbxC4oyMx2yr2IQ0xV oCM1VjbwAtvXLv4xPJGihTgcVIMak55Djos57y7n47XxP8x0c+prqs3is6wEd+gvAoYDOC 7MS45NjFSXBd/cJrv2VzBzVN+L7JM2GRLk3cxO+z3rFD83WLAaxkW8ft6lg+eRLNzqbwoo NOfSeZsMc/3p4MrwhD2DJM4s96d8NJ7OJd2+Didlq+z/ZZJRoWfKzFxZnXtJLQ== Subject: [RESEND PATCH v5 10/12] x86/mm: Move flush_tlb_info back to the stack Message-Id: <20260513124524.2569867-11-zhouchuyi@bytedance.com> References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Lms-Return-Path: X-Original-From: Chuyi Zhou To: , , , , , , , , , , , , From: "Chuyi Zhou" Date: Wed, 13 May 2026 20:45:22 +0800 Content-Transfer-Encoding: quoted-printable X-Mailer: git-send-email 2.20.1 In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Cc: , "Chuyi Zhou" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Commit 3db6d5a5ecaf ("x86/mm/tlb: Remove 'struct flush_tlb_info' from the stack") converted flush_tlb_info from stack variable to per-CPU variable. This brought about a performance improvement of around 3% in extreme test. However, it also required that all flush_tlb* operations keep preemption disabled entirely to prevent concurrent modifications of flush_tlb_info. flush_tlb* needs to send IPIs to remote CPUs and synchronously wait for all remote CPUs to complete their local TLB flushes. The process could take tens of milliseconds when interrupts are disabled or with a large number of remote CPUs. From the perspective of improving kernel real-time performance, this patch reverts flush_tlb_info back to stack variables and align it with SMP_CACHE_BYTES. In certain configurations, SMP_CACHE_BYTES may be large, so the alignment size is limited to 64. This is a preparation for enabling preemption during TLB flush in next patch. To evaluate the performance impact of this patch, use the following script to reproduce the microbenchmark mentioned in commit 3db6d5a5ecaf ("x86/mm/tlb: Remove 'struct flush_tlb_info' from the stack"). The test environment is an Ice Lake system (Intel(R) Xeon(R) Platinum 8336C) with 128 CPUs and 2 NUMA nodes. During the test, the threads were bound to specific CPUs, and both pti and mitigations were disabled: #include #include #include #include #include #include #define NUM_OPS 1000000 #define NUM_THREADS 3 #define NUM_RUNS 5 #define PAGE_SIZE 4096 volatile int stop_threads =3D 0; void *busy_wait_thread(void *arg) { while (!stop_threads) { __asm__ volatile ("nop"); } return NULL; } long long get_usec() { struct timeval tv; gettimeofday(&tv, NULL); return tv.tv_sec * 1000000LL + tv.tv_usec; } int main() { pthread_t threads[NUM_THREADS]; char *addr; int i, r; addr =3D mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr =3D=3D MAP_FAILED) { perror("mmap"); exit(1); } for (i =3D 0; i < NUM_THREADS; i++) { if (pthread_create(&threads[i], NULL, busy_wait_thread, NULL)) exit(1); } printf("Running benchmark: %d runs, %d ops each, %d background\n" "threads\n", NUM_RUNS, NUM_OPS, NUM_THREADS); for (r =3D 0; r < NUM_RUNS; r++) { long long start, end; start =3D get_usec(); for (i =3D 0; i < NUM_OPS; i++) { addr[0] =3D 1; if (madvise(addr, PAGE_SIZE, MADV_DONTNEED)) { perror("madvise"); exit(1); } } end =3D get_usec(); double duration =3D (double)(end - start); double avg_lat =3D duration / NUM_OPS; printf("Run %d: Total time %.2f us, Avg latency %.4f us/op\n", r + 1, duration, avg_lat); } stop_threads =3D 1; for (i =3D 0; i < NUM_THREADS; i++) pthread_join(threads[i], NULL); munmap(addr, PAGE_SIZE); return 0; } base on-stack-aligned on-stack-not-aligned ---- --------- ----------- avg (usec/op) 2.5278 2.5261 2.5508 stddev 0.0007 0.0027 0.0023 The benchmark results show that the average latency difference between the baseline (base) and the properly aligned stack variable (on-stack-aligned) is within the standard deviation (stddev). This indicates that the variations are caused by testing noise, and reverting to a stack variable with proper alignment causes no performance regression compared to the per-CPU implementation. The unaligned version (on-stack-not-aligned) shows a minor performance drop. This demonstrates that we can improve the real-time performance without sacrificing performance. Suggested-by: Sebastian Andrzej Siewior Suggested-by: Nadav Amit Signed-off-by: Chuyi Zhou --- arch/x86/include/asm/tlbflush.h | 8 +++- arch/x86/mm/tlb.c | 72 +++++++++------------------------ 2 files changed, 27 insertions(+), 53 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflus= h.h index 0545fe75c3fa..f4e4505d4ece 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -211,6 +211,12 @@ extern u16 invlpgb_count_max; =20 extern void initialize_tlbstate_and_flush(void); =20 +#if SMP_CACHE_BYTES > 64 +#define FLUSH_TLB_INFO_ALIGN 64 +#else +#define FLUSH_TLB_INFO_ALIGN SMP_CACHE_BYTES +#endif + /* * TLB flushing: * @@ -249,7 +255,7 @@ struct flush_tlb_info { u8 stride_shift; u8 freed_tables; u8 trim_cpumask; -}; +} __aligned(FLUSH_TLB_INFO_ALIGN); =20 void flush_tlb_local(void); void flush_tlb_one_user(unsigned long addr); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index af43d177087e..cfc3a72477f5 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1373,28 +1373,12 @@ void flush_tlb_multi(const struct cpumask *cpumask, */ unsigned long tlb_single_page_flush_ceiling __read_mostly =3D 33; =20 -static DEFINE_PER_CPU_SHARED_ALIGNED(struct flush_tlb_info, flush_tlb_info= ); - -#ifdef CONFIG_DEBUG_VM -static DEFINE_PER_CPU(unsigned int, flush_tlb_info_idx); -#endif - -static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm, - unsigned long start, unsigned long end, - unsigned int stride_shift, bool freed_tables, - u64 new_tlb_gen) +static void get_flush_tlb_info(struct flush_tlb_info *info, + struct mm_struct *mm, + unsigned long start, unsigned long end, + unsigned int stride_shift, bool freed_tables, + u64 new_tlb_gen) { - struct flush_tlb_info *info =3D this_cpu_ptr(&flush_tlb_info); - -#ifdef CONFIG_DEBUG_VM - /* - * Ensure that the following code is non-reentrant and flush_tlb_info - * is not overwritten. This means no TLB flushing is initiated by - * interrupt handlers and machine-check exception handlers. - */ - BUG_ON(this_cpu_inc_return(flush_tlb_info_idx) !=3D 1); -#endif - /* * If the number of flushes is so large that a full flush * would be faster, do a full flush. @@ -1412,32 +1396,22 @@ static struct flush_tlb_info *get_flush_tlb_info(st= ruct mm_struct *mm, info->new_tlb_gen =3D new_tlb_gen; info->initiating_cpu =3D smp_processor_id(); info->trim_cpumask =3D 0; - - return info; -} - -static void put_flush_tlb_info(void) -{ -#ifdef CONFIG_DEBUG_VM - /* Complete reentrancy prevention checks */ - barrier(); - this_cpu_dec(flush_tlb_info_idx); -#endif } =20 void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables) { - struct flush_tlb_info *info; + struct flush_tlb_info _info; + struct flush_tlb_info *info =3D &_info; int cpu =3D get_cpu(); u64 new_tlb_gen; =20 /* This is also a barrier that synchronizes with switch_mm(). */ new_tlb_gen =3D inc_mm_tlb_gen(mm); =20 - info =3D get_flush_tlb_info(mm, start, end, stride_shift, freed_tables, - new_tlb_gen); + get_flush_tlb_info(&_info, mm, start, end, stride_shift, freed_tables, + new_tlb_gen); =20 /* * flush_tlb_multi() is not optimized for the common case in which only @@ -1457,7 +1431,6 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigne= d long start, local_irq_enable(); } =20 - put_flush_tlb_info(); put_cpu(); mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } @@ -1527,19 +1500,16 @@ static void kernel_tlb_flush_range(struct flush_tlb= _info *info) =20 void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - struct flush_tlb_info *info; + struct flush_tlb_info info; =20 guard(preempt)(); + get_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false, + TLB_GENERATION_INVALID); =20 - info =3D get_flush_tlb_info(NULL, start, end, PAGE_SHIFT, false, - TLB_GENERATION_INVALID); - - if (info->end =3D=3D TLB_FLUSH_ALL) - kernel_tlb_flush_all(info); + if (info.end =3D=3D TLB_FLUSH_ALL) + kernel_tlb_flush_all(&info); else - kernel_tlb_flush_range(info); - - put_flush_tlb_info(); + kernel_tlb_flush_range(&info); } =20 /* @@ -1707,12 +1677,11 @@ EXPORT_SYMBOL_FOR_KVM(__flush_tlb_all); =20 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { - struct flush_tlb_info *info; + struct flush_tlb_info info; =20 int cpu =3D get_cpu(); - - info =3D get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false, - TLB_GENERATION_INVALID); + get_flush_tlb_info(&info, NULL, 0, TLB_FLUSH_ALL, 0, false, + TLB_GENERATION_INVALID); /* * flush_tlb_multi() is not optimized for the common case in which only * a local TLB flush is needed. Optimize this use-case by calling @@ -1722,17 +1691,16 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap= _batch *batch) invlpgb_flush_all_nonglobals(); batch->unmapped_pages =3D false; } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { - flush_tlb_multi(&batch->cpumask, info); + flush_tlb_multi(&batch->cpumask, &info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled(); local_irq_disable(); - flush_tlb_func(info); + flush_tlb_func(&info); local_irq_enable(); } =20 cpumask_clear(&batch->cpumask); =20 - put_flush_tlb_info(); put_cpu(); } =20 --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C93A40F8D0 for ; Wed, 13 May 2026 12:48:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676507; cv=none; b=SURbp3lolTc/PBg2Kcu2xp6N9q6gx9VIiHWDBEEhiLbwXVybIapyXjE+JGP5Ykk6F/Rylof1DKbVcMPMj1/jJwRl9kPAx+I55e0GFdhYINXhF3jDCYueZ2MI2tVxPa9XXFkyNwiKnaLVjYh1qIdEdGNCYZ/aI8qCrtCedBCebaw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676507; c=relaxed/simple; bh=l+a51GjDdeY3iCdCuc5+dubAJSu0U2am6FsfO42Po+0=; h=Message-Id:Date:In-Reply-To:Content-Type:Mime-Version:To:From: Subject:References:Cc; b=At8X9aGgrEQdNR/xJFJm8QhyquDAe3Y4O/ojLRVz3uByvAWy0/NXjq1couxTPx3A0OFZFG5Bnw2LFlKam2wLejlAAlwgbV5Zp1wNtILBGiAUQ26qIdwwQwy3NiZzssicdzvW/jhOeoI1y1+/k8i1FmiFZIv98utB5CMisnvk/WY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=eQ4TwbxL; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="eQ4TwbxL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676501; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=1AfQ0YrwAict/NhK9ZBJRyFni9dP4FKTse0qgdYBtJo=; b=eQ4TwbxLTYR+a74pJA5qgAVXlcgYwHxjR9oba+qOip0vXCi4K/xcouE8rtFJ0ZLGL+UJew Y0agTlvSMTQrOl01zSDlX+m61+aAyMjVCZb3mNwgHyifE8OapN4TeDFfG3gHbQzOlciIU3 7t4ojHFMsyhmvtBq8o9EhylpkfnlyOx8PzBR+OxlV+DxBk8XgzikScg2UfNXunMEIjeXRT yIlbB/YIidB+ms/kxQaTlnyCT1pleaS/ccPPQUkrcooBr9GFWsYPMoatXB8Uu/W4zT4aAD owYQjw9zPcmOqDy1e7aEa91BAadTbnNUYJzGVS3VPyEdk2Aq93v6Mxual8GgHA== Message-Id: <20260513124524.2569867-12-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou Date: Wed, 13 May 2026 20:45:23 +0800 In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 To: , , , , , , , , , , , , From: "Chuyi Zhou" Subject: [RESEND PATCH v5 11/12] x86/mm: Enable preemption during native_flush_tlb_multi Content-Transfer-Encoding: quoted-printable X-Lms-Return-Path: References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Cc: , "Chuyi Zhou" Content-Type: text/plain; charset="utf-8" native_flush_tlb_multi() may be frequently called by flush_tlb_mm_range() and arch_tlbbatch_flush() in production environments. When pages are reclaimed or process exit, native_flush_tlb_multi() sends IPIs to remote CPUs and waits for all remote CPUs to complete their local TLB flushes. The overall latency may reach tens of milliseconds due to a large number of remote CPUs and other factors (such as interrupts being disabled). Since flush_tlb_mm_range() and arch_tlbbatch_flush() always disable preemption, which may cause increased scheduling latency for other threads on the current CPU. Previous patch converted flush_tlb_info from per-cpu variable to on-stack variable. Additionally, it's no longer necessary to explicitly disable preemption before calling smp_call*() since they internally handle the preemption logic. Now it's safe to enable preemption during native_flush_tlb_multi(). Signed-off-by: Chuyi Zhou --- arch/x86/kernel/kvm.c | 4 +++- arch/x86/mm/tlb.c | 9 +++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 29226d112029..d540f54f4d16 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -662,8 +662,10 @@ static void kvm_flush_tlb_multi(const struct cpumask *= cpumask, u8 state; int cpu; struct kvm_steal_time *src; - struct cpumask *flushmask =3D this_cpu_cpumask_var_ptr(__pv_cpu_mask); + struct cpumask *flushmask; =20 + guard(preempt)(); + flushmask =3D this_cpu_cpumask_var_ptr(__pv_cpu_mask); cpumask_copy(flushmask, cpumask); /* * We have to call flush only on online vCPUs. And diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index cfc3a72477f5..58c6f3d2f993 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1421,9 +1421,11 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsign= ed long start, if (mm_global_asid(mm)) { broadcast_tlb_flush(info); } else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + put_cpu(); info->trim_cpumask =3D should_trim_cpumask(mm); flush_tlb_multi(mm_cpumask(mm), info); consider_global_asid(mm); + goto invalidate; } else if (mm =3D=3D this_cpu_read(cpu_tlbstate.loaded_mm)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1432,6 +1434,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigne= d long start, } =20 put_cpu(); +invalidate: mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } =20 @@ -1691,7 +1694,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_b= atch *batch) invlpgb_flush_all_nonglobals(); batch->unmapped_pages =3D false; } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + put_cpu(); flush_tlb_multi(&batch->cpumask, &info); + goto clear; } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1699,9 +1704,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_b= atch *batch) local_irq_enable(); } =20 - cpumask_clear(&batch->cpumask); - put_cpu(); +clear: + cpumask_clear(&batch->cpumask); } =20 /* --=20 2.20.1 From nobody Fri Jun 12 17:18:00 2026 Received: from va-1-111.ptr.blmpb.com (va-1-111.ptr.blmpb.com [209.127.230.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96F95402426 for ; Wed, 13 May 2026 12:48:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.111 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676522; cv=none; b=d36hNi0cb9zaBYOCaUFBEHCIQimcBBZoO8js2ne6vFihEULxN7GQaemq2paF6uO0XoqsSPk9EY6txLjKJA0YCHFVoftWFFJVibSmTSlopnnJ24j/CJmOHW9wGgFuXATh1OMVQcSLiqgIrmsNjtW4G3P8lbfl1qTbVpyABvl+/fo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778676522; c=relaxed/simple; bh=9HZ6YSDto50uyHsiSF3PXW23dmlhrFNA7PdEkwEU9pA=; h=Mime-Version:In-Reply-To:References:To:Message-Id:Cc:Subject: Content-Type:From:Date; b=sw3lr4sbId71fr+XgW0tX452MHM3rChA9Q5HGO9+q/eWnfrvOdPPMdXKdOFWrqp2A7qb2TPIw1p/gk00AtHmyiI7RzSi7LUe5grnkOPQvQI3oybgKKOlQgO+bGHmnD071oeXUuvaeewbyVxlCs07UDWchG0h6zKBz+n+geJqfnE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LPgUpDlj; arc=none smtp.client-ip=209.127.230.111 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LPgUpDlj" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1778676515; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=OwZiM/rvVevkD2sWSEluAauZ9Gw9rO6C52AV1cynZb8=; b=LPgUpDljDMFfh13rF4ii5ml3VCdWEzZhRM6UdkL3vyKAI+wkPVo4kQd2JOOCny5ZvAjPZ/ 6lj2IxPXp4sjTqBMbMX+7q3bxlYBXxZPkYm1rX8/QcHv1tIF14IcWCNgjaW9d4JyPzyfO0 nl9R3yAvxKdDkgzsLyRs8eAe5gPeRiaX+oTDTRgtaPlUGYQCQ8j5c+lIZjwPEP8ZKBMT3D XAZaSIqgSn3iQ+MhdiNFra7qzuB7We0fuXgAarko2rNUlws22D6/WM4q8fKWb2gGpmQxY+ zY/0HgAN5exCqsw8wyZ/w5OePqzf0xG/KLpmpAp+QR3zZei8/jrm3sqvT3RROg== Content-Transfer-Encoding: quoted-printable Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Original-From: Chuyi Zhou In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 X-Lms-Return-Path: To: , , , , , , , , , , , , Message-Id: <20260513124524.2569867-13-zhouchuyi@bytedance.com> Cc: , "Chuyi Zhou" Subject: [RESEND PATCH v5 12/12] x86/mm: Enable preemption during flush_tlb_kernel_range From: "Chuyi Zhou" Date: Wed, 13 May 2026 20:45:24 +0800 Content-Type: text/plain; charset="utf-8" flush_tlb_kernel_range() is invoked when kernel memory mapping changes. On x86 platforms without the INVLPGB feature enabled, we need to send IPIs to every online CPU and synchronously wait for them to complete do_kernel_range_flush(). This process can be time-consuming due to factors such as a large number of CPUs or other issues (like interrupts being disabled). flush_tlb_kernel_range() always disables preemption, this may affect the scheduling latency of other tasks on the current CPU. Previous patch converted flush_tlb_info from per-cpu variable to on-stack variable. Additionally, it's no longer necessary to explicitly disable preemption before calling smp_call*() since they internally handles the preemption logic. Now it's safe to enable preemption during flush_tlb_kernel_range(). Additionally, in get_flush_tlb_info() use raw_smp_processor_id() to avoid warnings from check_preemption_disabled(). Signed-off-by: Chuyi Zhou --- arch/x86/mm/tlb.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 58c6f3d2f993..c37cc9845abc 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1394,7 +1394,7 @@ static void get_flush_tlb_info(struct flush_tlb_info = *info, info->stride_shift =3D stride_shift; info->freed_tables =3D freed_tables; info->new_tlb_gen =3D new_tlb_gen; - info->initiating_cpu =3D smp_processor_id(); + info->initiating_cpu =3D raw_smp_processor_id(); info->trim_cpumask =3D 0; } =20 @@ -1461,6 +1461,8 @@ static void invlpgb_kernel_range_flush(struct flush_t= lb_info *info) { unsigned long addr, nr; =20 + guard(preempt)(); + for (addr =3D info->start; addr < info->end; addr +=3D nr << PAGE_SHIFT) { nr =3D (info->end - addr) >> PAGE_SHIFT; =20 @@ -1505,7 +1507,6 @@ void flush_tlb_kernel_range(unsigned long start, unsi= gned long end) { struct flush_tlb_info info; =20 - guard(preempt)(); get_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false, TLB_GENERATION_INVALID); =20 --=20 2.20.1