From nobody Mon Jun 8 20:41:25 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46E8E421A13 for ; Tue, 26 May 2026 18:08:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779818897; cv=none; b=CaxbW9xbnJNMApXBXaISeRH24LwTFQ7VZpHIGUudW3DCMIchOxtqFFeyd1DAPv5V0u6L7NfjdWmRCWF/1qx9gjMKogb7gENvqJk5oYuc5dkI6jmNfi5MxBat2t9ke4JGg5Lz8ru9fXwi1A1Jyk3jQ6TCT8mJXawr5noBuzCaJsU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779818897; c=relaxed/simple; bh=Sjago3jtBD4V02QbFl9FT/wPn0DLucs0jnujBLxQpfg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=UriVCLjj79f8gXUdRWO+Rl+y0QuEa0ciayjXJIQlKa6Dr+sUD6fynVpzldERKEXEAyodvK1EUYQt7495y5Iz3hokbU9Gd7ZLQZW4qfYhFTG4kJ1x5o+WZ8j4bBQCQjNvOYfecWMezZx+sLIq8uhtBMo4eUs5hPpqc3HRbimfJTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=ivRcFh/H; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="ivRcFh/H" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=RBkg9puqnsn/oMOz1istftb0PbC8C6//C02CGsLpVh4=; b=ivRcFh/HrEpc8MfwQ2MhoGrgBQ qxCCcC3B5wBTZ++VN+DQ1hPe6ueJ6mt7JS4Bf+OSVWKcuSeuwBPGbeynkehTMx2s01GVOAMonNG/3 D+fYmPmd4T3GUekKIUttKv4rnhZrY42rT+8rvsefHsB/VxMG8f7cRFmAzSfPqJBlJADTj0t8rHSeF ZtujQZKHtihiZR7yWKu31UWEM9XC1EN7S1O/MNyAJj4H9Mvh+FQPeXWtO7IFWNHKC0hT+//ierSAD YnMpASUZAIQB5Hm6LkF3b/pcdeUwpVeAauvsgY7NP2oaPYyfhRXSuSrcc7bDQCVO4RT9fGuzBze7/ JLbdGHLQ==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wRwCI-002Wtw-0j; Tue, 26 May 2026 18:08:10 +0000 From: Breno Leitao Date: Tue, 26 May 2026 14:08:05 -0400 Subject: [PATCH 1/2] workqueue: split kick_pool() into kick_pool_pick() + wake_up_process() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260526-fastwake-v1-1-e69ad86923e6@debian.org> References: <20260526-fastwake-v1-0-e69ad86923e6@debian.org> In-Reply-To: <20260526-fastwake-v1-0-e69ad86923e6@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.16-dev-d5d98 X-Developer-Signature: v=1; a=openpgp-sha256; l=2794; i=leitao@debian.org; h=from:subject:message-id; bh=Sjago3jtBD4V02QbFl9FT/wPn0DLucs0jnujBLxQpfg=; b=owEBbQKS/ZANAwAKATWjk5/8eHdtAcsmYgBqFeGHr5sb7EUvoGFb3hUdkE4Tc5pAZ3h5OxsHm bqJVUlsKg+JAjMEAAEKAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCahXhhwAKCRA1o5Of/Hh3 bbseEACag5ZzGaYp227K16rVYxgWc50swp6k242DDEoisI9yGetbLmWLOhbsMTtJbp5/oivgcFb RP29dri+H21sF82Eo6OGFTmYYz4DIaXHgYGUeNmEH7J6Vp8uFNwvJkpo8KpybjUwmVqktBa2rcB ws6gd5hfr5/iy2Xa2fD8c7+PD5fXK+9S+QzUu27xeDteHW3MpXfOykeWkeO6WPx7VaSYmDSTmSs uKZwmEHhutEfRDmaOQ/AQ/X5hmHntieS7m6aADVhMPNqE/RpGTiVLFbzH8kEyA1eR7tRIBWJOYu DIphGMbn+PdBLl/jZ6GwvKzNwmS5HesQgViBbHPgm5Z42Fp9AYYZuxJZDPbyPwE0Gs6gEBKP43f 5dD97O54GD+/FpdhaQKuT8bBY7KWVEYZFTWdKviAAWu/K80wV+uROaFLb9N1Nxv/w59Hbq5nqEX M9hTVKBuYUguRzHyvqsgu9jF1A17c1wz9rD1WNRp7ZZBHbnStdEKpVxP+xAhoSOlxf4MXGd1h8I 8SbXiSmzi9rNdV+3JaD4IVqOZuxSuQuq59Tqml4TiahzwkGhfg+n++1vrCPEkmV3oc9tfz50IMU O38N5+HI/Df0Yik3zkPWwoXIoTba/jnNIE5ej8ToUTT8GW2ZsvkvU3Z3jth1s0Wm/1bEX3a3Vgr 9Lm4iZtsdERfzJQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao Factor the worker-selection part of kick_pool() out into a new helper kick_pool_pick() that picks the worker to wake (and applies the wake_cpu / repatriation hint that requires pool->lock) but does not call wake_up_process() on it. It returns the task_struct of the worker to wake, or NULL when no wakeup is needed. kick_pool() becomes a thin wrapper that calls kick_pool_pick() and then wake_up_process() on the returned task, so all existing callers keep their current behavior. This is a pure refactor with no functional change. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 36 ++++++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 8df671066dd1..b788d7c44ac0 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1258,13 +1258,21 @@ static void kick_bh_pool(struct worker_pool *pool) } =20 /** - * kick_pool - wake up an idle worker if necessary + * kick_pool_pick - select a worker to wake up but defer the wakeup * @pool: pool to kick * - * @pool may have pending work items. Wake up worker if necessary. Returns - * whether a worker was woken up. + * Same selection logic as kick_pool() but returns the task that should be + * woken instead of calling wake_up_process() on it. The caller is + * responsible for calling wake_up_process() on the returned task (typical= ly + * after dropping @pool->lock to shorten the locked region). + * + * Returns the task_struct to wake, or NULL if no wakeup is needed (e.g. t= he + * pool was empty, no idle worker was available, or it was a BH pool which + * was already kicked synchronously). + * + * Must be called with @pool->lock held. */ -static bool kick_pool(struct worker_pool *pool) +static struct task_struct *kick_pool_pick(struct worker_pool *pool) { struct worker *worker =3D first_idle_worker(pool); struct task_struct *p; @@ -1272,11 +1280,11 @@ static bool kick_pool(struct worker_pool *pool) lockdep_assert_held(&pool->lock); =20 if (!need_more_worker(pool) || !worker) - return false; + return NULL; =20 if (pool->flags & POOL_BH) { kick_bh_pool(pool); - return true; + return NULL; } =20 p =3D worker->task; @@ -1310,6 +1318,22 @@ static bool kick_pool(struct worker_pool *pool) } } #endif + return p; +} + +/** + * kick_pool - wake up an idle worker if necessary + * @pool: pool to kick + * + * @pool may have pending work items. Wake up worker if necessary. Retur= ns + * whether a worker was woken up. + */ +static bool kick_pool(struct worker_pool *pool) +{ + struct task_struct *p =3D kick_pool_pick(pool); + + if (!p) + return false; wake_up_process(p); return true; } --=20 2.51.0 From nobody Mon Jun 8 20:41:25 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46FCA421A19 for ; Tue, 26 May 2026 18:08:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779818898; cv=none; b=B4eYD8wkCDiWdZBVsCjQfZwKRR3CH+VNoJHa+1ljTbxlv/xYT+PSz+9jw1K7GTowQ9NAtpm/vmccaSio99KL6UkxIvdkuQjJWI8PPF18hWMlaqRN3aR5W1hhRWNIzKgXESolWseQSbDvMNZNB7/h4heAVNx7Qe3L4tA7NymZcJI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779818898; c=relaxed/simple; bh=OHFpmGg6eK2vffd/eDnZJyfgvVD70ayi3QDPz+fZYSo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hnkQ1GLCCiB72ceCV4Py02RiaiBhLPAHSmhugRma7pLH04UayZLRYbHQv15Zk6vhbjX6fBi+56KGDmSQU0uYJXaOBQhVdoFgDqOfubE6BNNuZkRTCaRxrw7vzbFF5sP3gZpEwo7lyqX3VBmsx5fzpRJyQ7Lw/Nd2d9Jptcf6m28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=QMrHnuzL; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="QMrHnuzL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=p6PRWoqmsxSGKSyXhbzegHEoF9nWLbHQq8uk2FbvmCI=; b=QMrHnuzLbun0FFk0ZfJQGHpjwZ DXqHOdugwf2NLrifwrULOFwwIeoiaOAMg6BKohEbZXsjdQ31MC3u/L5boCuWJbWlMVa42kHGwanvv o2HFdOGuRda9asN/0QtKIOrqFuCj1NlZFi1a7cvN6+TTvSydF0KlYmjsGM8jDzml5UgMHKKi0Jxkh /VXPgvDGCr/7Ct834uNFFnIwM9mO/Y3pPKmoqBNwgOaUDNv0qu8Rw5rvR7UXyo2t6AIWnZyjOkwrd rGIMxfn864pY8WfYWXVrLEIf+gqdKBgcK4te6fJI6RQb7lYWTbZ3GA6XC4OxzO7SY03SWIhK92l9f 6RVHcHAQ==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wRwCJ-002Wty-1I; Tue, 26 May 2026 18:08:11 +0000 From: Breno Leitao Date: Tue, 26 May 2026 14:08:06 -0400 Subject: [PATCH 2/2] workqueue: defer wake_up_process() outside pool->lock on hot paths Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260526-fastwake-v1-2-e69ad86923e6@debian.org> References: <20260526-fastwake-v1-0-e69ad86923e6@debian.org> In-Reply-To: <20260526-fastwake-v1-0-e69ad86923e6@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.16-dev-d5d98 X-Developer-Signature: v=1; a=openpgp-sha256; l=4350; i=leitao@debian.org; h=from:subject:message-id; bh=OHFpmGg6eK2vffd/eDnZJyfgvVD70ayi3QDPz+fZYSo=; b=owEBbQKS/ZANAwAKATWjk5/8eHdtAcsmYgBqFeGHp2tCYDsdnUfbqKDq18COGd6ap/Qs5n5v3 Xk0EvJHODmJAjMEAAEKAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCahXhhwAKCRA1o5Of/Hh3 bbVKD/97brpwlAm+HuM8f7tITpHvHr75pPebuGC8YQLZeHXAkda77TMMTyQsr6lQIOrOgWTDhMs r42ZkZCba7AchaBwDzoL0G4bpgP315eN4VrhZXGUzYx+1372lmaUIg4Zj6uT7XxUloz65nXpjBD uJI3vnaHBmT7zhwqq8LRBqlMpU9qAmT3sqwyVjMaP9ALgnCVflYVZoSd07oriZ43qocBl2Bf1m5 dL6rphcOGBHVFvEn91ACU5GvIAtQ/GfRnrW/4sTEypmrA2LWCLsgzurvYWGROekeow5yOWVEsvk hqUY8FqciOIIYBe3ZhG0vFejA54gTDAB0uUtkQvqF6lTyUK+CcIHho6LpdmoXRDiVH/S1ICwFrB /eQhNAr0Lfk6tD2LI/teOZ5hmFnyTbytXQhOtAIfIbNFMbK5mBPoMqKloaYZuJsyZa6AaE9aJ9/ bXiLGwyg6xGFM0ckfQ85BsVmAYorNb8elOOYotUDhDIG1bIBJlok3r29G+e2/rGa87pgiAvQZI7 FGkY3p9p6bwS+NKri85QygWx+v130wdfTHZmpxtFEziL/bBb0kYzZyN8H2Ow4j4pIeGNYbjYhvU E1ytnLi563kBEwR73omi9PQRIursSe6eazOMNZE8yHjKECye80QKY3J8FocEOGj8FZ71gCJhkI2 mv6jRtmpRJCfUNQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao Both __queue_work() (enqueue) and process_one_work() (per-work chain kick on unbound/CPU_INTENSIVE pools) call kick_pool() while holding pool->lock. kick_pool() ends in wake_up_process(), which takes the target task's rq->lock. Holding pool->lock across that runqueue lock acquisition lengthens the locked region on the two hottest paths of a contended unbound workqueue. Use the new kick_pool_pick() helper to select the worker to wake while holding pool->lock, then call wake_up_process() after pool->lock is released. All state that requires pool->lock (worker selection, wake_cpu adjustment, BH-pool fast path) is still done under the lock; only the unrelated rq->lock acquisition is moved out. Measured on a CONFIG_SMP arm64 VM (8 vCPUs) with the test_workqueue benchmark (lib/test_workqueue.c) using a batched-submit mode (8 producer kthreads, 200000 work items each, WQ_UNBOUND). Averages of five runs per scope: affinity_scope baseline (items/s) patched (items/s) gain -------------- ------------------ ----------------- ---- cpu 1,419,973 1,413,896 -0.4 % (no c= ontention) smt 1,442,921 1,437,164 -0.4 % (no c= ontention) cache_shard 1,184,058 1,279,184 +8.0 % cache 1,167,603 1,271,341 +8.9 % numa 1,163,617 1,285,427 +10.5 % system 1,175,933 1,255,227 +6.7 % Enqueue latency on the contended scopes also drops (p50 ~2875 -> ~2625 ns, p99 ~5000 -> ~4200 ns). The cpu/smt scopes use per-CPU pools with no producer/consumer contention, so as expected they are unchanged. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index b788d7c44ac0..1403a4b195a3 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -2301,6 +2301,7 @@ static void __queue_work(int cpu, struct workqueue_st= ruct *wq, { struct pool_workqueue *pwq; struct worker_pool *last_pool, *pool; + struct task_struct *wake_p =3D NULL; unsigned int work_flags; unsigned int req_cpu =3D cpu; =20 @@ -2415,7 +2416,7 @@ static void __queue_work(int cpu, struct workqueue_st= ruct *wq, =20 trace_workqueue_activate_work(work); insert_work(pwq, work, &pool->worklist, work_flags); - kick_pool(pool); + wake_p =3D kick_pool_pick(pool); } else { work_flags |=3D WORK_STRUCT_INACTIVE; insert_work(pwq, work, &pwq->inactive_works, work_flags); @@ -2423,6 +2424,15 @@ static void __queue_work(int cpu, struct workqueue_s= truct *wq, =20 out: raw_spin_unlock(&pool->lock); + /* + * Issue the wakeup after dropping pool->lock to shorten the + * locked region on this hot enqueue path. kick_pool_pick() did all + * of the work that required the lock (worker selection and + * wake_cpu setup); the wake_up_process() itself only needs to + * take the target rq->lock. + */ + if (wake_p) + wake_up_process(wake_p); rcu_read_unlock(); } =20 @@ -3243,6 +3253,7 @@ __acquires(&pool->lock) { struct pool_workqueue *pwq =3D get_work_pwq(work); struct worker_pool *pool =3D worker->pool; + struct task_struct *wake_p; unsigned long work_data; int lockdep_start_depth, rcu_start_depth; bool bh_draining =3D pool->flags & POOL_BH_DRAINING; @@ -3296,8 +3307,11 @@ __acquires(&pool->lock) * since nr_running would always be >=3D 1 at this point. This is used to * chain execution of the pending work items for WORKER_NOT_RUNNING * workers such as the UNBOUND and CPU_INTENSIVE ones. + * + * Select the worker to wake while holding pool->lock, but defer the + * actual wake_up_process() until after the lock is dropped below. */ - kick_pool(pool); + wake_p =3D kick_pool_pick(pool); =20 /* * Record the last pool and clear PENDING which should be the last @@ -3310,6 +3324,9 @@ __acquires(&pool->lock) pwq->stats[PWQ_STAT_STARTED]++; raw_spin_unlock_irq(&pool->lock); =20 + if (wake_p) + wake_up_process(wake_p); + rcu_start_depth =3D rcu_preempt_depth(); lockdep_start_depth =3D lockdep_depth(current); /* see drain_dead_softirq_workfn() */ --=20 2.51.0