From nobody Mon Jun 8 20:41:43 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8C0430C173 for ; Wed, 3 Jun 2026 13:40:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494044; cv=none; b=S9rEifT3a3xFmnQEybRB7mqFjqubc6p9yN4SMosV5MqM+bsFG59qkXAiZGektvg8ZVRy+FnIRBkkg9J2J4DgbU7lD8Nb/TS5IkvRxkBpTAVc4GXjT73+epuh8e5Q9U+9rvGin/9+H8cum/jZOFWHuhrltTrTazSDaMz1kxSIM3I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494044; c=relaxed/simple; bh=jWdNE/KMeZJTSiqSHYYEzG4CEGJkmcoUePz3s6zuso4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=XzVoUoh9me/vYVMRvu9GLNj8yqMf5DMSnGrXtUuYByF7bb45CmBqbEFBuTaTDqxXP6t3LRjlmZHOJkegxQm+Y8K/6O5uc/9h8erWwC5yTcPEjuBB0Bc24/2ZTxMpenSQlyzU0sQPo/cftqjbEmT51HBki0h95F6LNn1+ixYBB6I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=wG8dWCwz; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="wG8dWCwz" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=mmRjFmnHa4OtVYeUn9gDh8wWdcqmWzYGmuxNKJfY0k8=; b=wG8dWCwzuJBrKnMxVs3yMzwJGt 1LSn7r0JfPkj5CQOW2TuOABKRGP2F3s46A7Wwb+K7Inkl4plFHaj31KAAaAePfFeGtfcly0yDbAaU zgFDOEEA2JNBdrd62bTPR8OmklugooeEc5fSFDN2br2Xakci2LkH8ZVYXFBoiW8/wkIBg4E3Nb31x DqSIzWAmXVfxQzmsErv1BZMZGhvqahscll3p4StTuQ+VgGoeCXupI6ka4r7YMKfdUuFquhQzcNrcC t8wlI60zJc1ku5IbBVcVFPNPYVjIFzVpa78D8L3SByp301YTKNzSvnwZ4JgE6Eo84KiLJjiDBEc8p VYu4qg0g==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpX-003m3g-2H; Wed, 03 Jun 2026 13:40:24 +0000 From: Breno Leitao Date: Wed, 03 Jun 2026 06:40:08 -0700 Subject: [PATCH v2 1/4] workqueue: park kicked worker on pool->kicked_list Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260603-fastwake-v2-1-2977512fe7fa@debian.org> References: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> In-Reply-To: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=4239; i=leitao@debian.org; h=from:subject:message-id; bh=jWdNE/KMeZJTSiqSHYYEzG4CEGJkmcoUePz3s6zuso4=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+GH0/6Rkqt7MeHKoUzuGzTTrAHwNOgLiZj 0+E9FNyUtKJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bcDBD/9Ez8Z7D5jmDDLPNF+lq+JaRG+7fu8hHDCEJtWZW5GJHFvcWh1bOmZmrj1YUeeQfuaQuY8 AE7TPfWn1dJQpGEZ2/G3ubqYG7HmMFwg+KvWdr98COeKZyCzb52KsYTg/zuGnBvaobhosdk5gy/ VJkMadigouxT8ppTn5ev5JpK0CR+ROlM1AMHBb3uGYvhF1d0/luX35YlNlHpYKOpGzy2kdD7wQC 9RIsQ+7AhymfRrHe5SfnX1yo5KjYqBfU3fkuZUyWbO/oJp2PnhoWGeXxvGaUlgXrK/KyW1A1dVq UBOJVbWX6mzsUwQy7mep4pVvgkJM9hM/p9A+9/3QmFP2lr0WFZTHMlfYQyXyKJNR5pbLhmGbIga OYbxobZSpZ1PuzfFbiMSbr0aoFc9asWuFiKsL/yppwLbevS0AIDURMQIY2hAaH8VFTiwrpugcqk TOiOz7yKI0gghMZh1duNQPEMkQ3g9QKkYvC8Gh9jOAgbm/wKUmFYeNCu0M0sr07c7a2cl+RL803 GDMoHVxFD2VLeuTt4A4n/p4cXHMJl86Zh2dfwH1cCZgsKh/qttUcYj1TMP2FOvYu1IaTl8sgHu3 /VI4QeDn2L97pMg0r/c4EAbVYS/3dYP9kDOZRM1NkQ32GN2jlZpW3mvfiTVUBR4ikelg3hMTcKg lVJ74nkt25b7mmg== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao kick_pool() picks an idle worker and wakes it, but leaves it WORKER_IDLE on pool->idle_list until the woken kthread schedules in and runs worker_leave_idle(). idle_cull_fn() only checks WORKER_IDLE, not the task state, so a kicked-but-not-yet-scheduled worker is still a valid cull victim -- the cull can reap it before it consumes the just-enqueued work, stranding the item. The window is narrow today but later patches in this series defer the wakeup outside pool->lock, widening it. Move the picked worker from pool->idle_list to a new pool->kicked_list under pool->lock so the cull path -- which walks idle_list only -- cannot reach it. worker_leave_idle() already does list_del_init(), so it correctly removes the worker from kicked_list when it actually runs; worker_enter_idle() puts it back onto idle_list on completion. No extra list ops on the worker side. LIFO coalescing of back-to-back kicks onto the same cache-hot worker is preserved by having first_idle_worker() peek kicked_list before idle_list: the second kick lands on the already-kicked worker, the duplicate wakeup is a no-op, and the worker drains both items when it runs. Why not creating a new WORKER_KICKED flag instead, you might ask. I've tried it and the numbers decreased. Compared to tagging the worker with a new WORKER_KICKED flag, list_move() writes to worker->entry (offset 0 of struct worker), which the producer already dirties when reading the idle_list head; no new cross-CPU cacheline is introduced. Tagging worker->flags would have put a producer-side write on an otherwise worker-private cacheline, causing a coherence bounce on every kick. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 8df671066dd1..b3f8b86cb52f 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -216,12 +216,13 @@ struct worker_pool { int nr_idle; /* L: currently idle workers */ =20 struct list_head idle_list; /* L: list of idle workers */ + struct list_head kicked_list; /* L: workers kicked but not yet running */ struct timer_list idle_timer; /* L: worker idle timeout */ struct work_struct idle_cull_work; /* L: worker idle cleanup */ =20 struct timer_list mayday_timer; /* L: SOS timer for workers */ =20 - /* a workers is either on busy_hash or idle_list, or the manager */ + /* a worker is either on busy_hash, idle_list, kicked_list, or the manage= r */ DECLARE_HASHTABLE(busy_hash, BUSY_WORKER_HASH_ORDER); /* L: hash of busy workers */ =20 @@ -1031,6 +1032,13 @@ static inline void worker_clr_flags(struct worker *w= orker, unsigned int flags) /* Return the first idle worker. Called with pool->lock held. */ static struct worker *first_idle_worker(struct worker_pool *pool) { + /* + * Prefer an already-kicked worker so back-to-back kicks coalesce + * onto the same cache-hot worker (LIFO reuse). + */ + if (!list_empty(&pool->kicked_list)) + return list_first_entry(&pool->kicked_list, struct worker, entry); + if (unlikely(list_empty(&pool->idle_list))) return NULL; =20 @@ -1310,6 +1318,16 @@ static bool kick_pool(struct worker_pool *pool) } } #endif + /* + * Move @worker to pool->kicked_list so a concurrent idle_cull_fn() + * (which only walks pool->idle_list) cannot reap it before it + * consumes the just-enqueued work. worker_leave_idle() removes the + * worker from whichever list it sits on; worker_enter_idle() puts + * it back on pool->idle_list on completion. first_idle_worker() + * peeks kicked_list first, so back-to-back kicks still coalesce + * onto the same cache-hot worker (LIFO reuse). + */ + list_move(&worker->entry, &pool->kicked_list); wake_up_process(p); return true; } @@ -4896,6 +4914,7 @@ static int init_worker_pool(struct worker_pool *pool) pool->last_progress_ts =3D jiffies; INIT_LIST_HEAD(&pool->worklist); INIT_LIST_HEAD(&pool->idle_list); + INIT_LIST_HEAD(&pool->kicked_list); hash_init(pool->busy_hash); =20 timer_setup(&pool->idle_timer, idle_worker_timeout, TIMER_DEFERRABLE); --=20 2.54.0 From nobody Mon Jun 8 20:41:43 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8B5030C145 for ; Wed, 3 Jun 2026 13:40:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494044; cv=none; b=QDYKZIdUqnsD6jsnohz/+AZUverlOy/431VW1diTkQ+u7sO9QF/89/d0X17V8+VhEg9VQjEPH5ifYCTq926OICaHcdiNFBfilX26JULYfJh11MfeP/ptmoOsI5r84tQyc6TERXooGBOsQevLp9NIBqZQFITpFdm7+yiqwcMlxN4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494044; c=relaxed/simple; bh=5zuCILowJvwKt2iMqBvXcW0/SCwgdBL2D6Kzu8fhPPI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=BBqYqmE9BdAQ2HS88aN5UTq1dfE+4vJPHBja0uP/usmwBp7dB6FBtrdx2cnW2fc5DRVyRj+fdRY+fAOXWJsNiBohunHsmrFYuoyyPFghEgylqmDGORUY1dNvDEbIwcGyizzbzbFhu3vlIYNQX+o1LXfVLdXIAhgtQerLjjul6rw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=TiMH4afT; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="TiMH4afT" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=IboW8XcmfP5WrWPuXkFKLhoyJ02H6sqZ2ovSjXYM4rk=; b=TiMH4afTeQCnmBncYWvSlwsyq9 A7bfLEMM22WYv9DiKk7chFPAAYuMPu9VI7pR4I0Ev7JT4+6Pq9EgkJ49u0MBOBayjXIpUolRb6XrZ vLWC1SazXngh9XCIQue3ifXXYCwAYN0BA6lwGvvttuGSr2vOfkZouIv0QwaDGm/KlzYga/8ClEJpm TIrMK6CPCrAxPHB3iUBIfBh/q/r6fDVCpr15KaMOPdS4MXOdNQn5JGLeee7U9L9aXhF9iOxPgU6Wv Dty9Mn0FPO5hdWo4b+Rb2/t3dtmXsl0TBBNoaG679erIDjDE4g5O2/yEOOK58C3IQ5+4Ax7/zO0zO HXa1F89w==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpb-003m3i-2z; Wed, 03 Jun 2026 13:40:28 +0000 From: Breno Leitao Date: Wed, 03 Jun 2026 06:40:09 -0700 Subject: [PATCH v2 2/4] workqueue: split kick_pool() into kick_pool_pick() + wake_up_q() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260603-fastwake-v2-2-2977512fe7fa@debian.org> References: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> In-Reply-To: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=2670; i=leitao@debian.org; h=from:subject:message-id; bh=5zuCILowJvwKt2iMqBvXcW0/SCwgdBL2D6Kzu8fhPPI=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+Yk1Rky+cgXyoiEq8w5dqJENvLAN9Mm7AW ldV2jM/vLaJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bevTD/0V/Hauv0oijRL9sDMnK+ZKToT3C1d+4+tdOwDkUj8uSekGk7izOkeOe/8/0q6YUKdCKu3 +HBXGU0G8eXqhjMSKc4Kc8XI0Hwhcf14LDRSy2+LbTcAw2YmTbfkv9C1jj4SxzA/bJHXYdnHJlX w7soNoKWHUXkII8JwGFg2j9OkggH4Ofsgq5y0qfDNNvd6lkWPKdSqDM26Wgn8kJxOTvHqSyqIaJ jmxpNDQQYBgauLiI8UccRtFpwEicWcVu7a6t3fYwbIVZP0pzs/MeTQYUEJ8L6+0e+GoVZ9UMwL8 2DxbVSkpKIX5V/up5p6noFsdH7+K65fuLAv+8cvxGqp+WjQGN7hRWRxZQHaBWjpl4nXZywkkkWC Sya9FXOWbkxe1s0MxUkACn1IQQWNfMJndADDAs3Dc0/9r1/uY4Z7lYTQCF1cKiF9qDC7RMsQ392 /pQZ7uUB5XjJ2DoIrdT3YmmIuHReYNFDIE6Ox94RqpLTqCCam8uDwm+kD3OF/aTL2NdMRGqEcgX xjRbbUHHA7xWwL+hKHPu4myr5roPAu8pwRs5j7V09CjR15XYWe6V7zaMmzkAhPjjsC+/IOicmbt lZ4/jJUlBrrbJ8aVQ+QdgLh8m88fLRz/QZOmO92K5MzYqhq7g2suk3Ao2cZosGKpel6DaTz7x+B sJUj5fJ67hlczrw== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao Factor the worker selection out of kick_pool() into kick_pool_pick(), which picks and claims the worker under pool->lock but, instead of waking it, queues it on a caller-provided wake_q via wake_q_add(). The caller issues the wakeup later with wake_up_q(). wake_q_add() is safe under the lock (cmpxchg + get_task_struct()); only wake_up_q() takes rq->lock. kick_pool() becomes a thin wrapper using a local wake_q, so all existing callers keep waking under pool->lock. Pure refactor, no functional change. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index b3f8b86cb52f..2811ada6dec6 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -1266,13 +1267,17 @@ static void kick_bh_pool(struct worker_pool *pool) } =20 /** - * kick_pool - wake up an idle worker if necessary + * kick_pool_pick - select and claim an idle worker, deferring the wakeup * @pool: pool to kick + * @wakeq: wake_q to queue the selected worker on * - * @pool may have pending work items. Wake up worker if necessary. Returns - * whether a worker was woken up. + * Like kick_pool() but queues the picked worker on @wakeq (wake_q_add()) + * instead of waking it, so the caller can wake_up_q(@wakeq) after dropping + * pool->lock. Returns whether a worker was selected. + * + * Must be called with @pool->lock held. */ -static bool kick_pool(struct worker_pool *pool) +static bool kick_pool_pick(struct worker_pool *pool, struct wake_q_head *w= akeq) { struct worker *worker =3D first_idle_worker(pool); struct task_struct *p; @@ -1328,10 +1333,27 @@ static bool kick_pool(struct worker_pool *pool) * onto the same cache-hot worker (LIFO reuse). */ list_move(&worker->entry, &pool->kicked_list); - wake_up_process(p); + wake_q_add(wakeq, p); return true; } =20 +/** + * kick_pool - wake up an idle worker if necessary + * @pool: pool to kick + * + * @pool may have pending work items. Wake up worker if necessary. Returns + * whether a worker was woken up. + */ +static bool kick_pool(struct worker_pool *pool) +{ + DEFINE_WAKE_Q(wakeq); + bool kicked; + + kicked =3D kick_pool_pick(pool, &wakeq); + wake_up_q(&wakeq); + return kicked; +} + #ifdef CONFIG_WQ_CPU_INTENSIVE_REPORT =20 /* --=20 2.54.0 From nobody Mon Jun 8 20:41:43 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50F33349CDD for ; Wed, 3 Jun 2026 13:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; cv=none; b=EzrM7ZMVZ5G6aY4phoUyakNh51oaTqlzRjNbaxVlvowFP6/v3VyU6x+sMNfydWX2FnTBsq1ls/uSrAE2iqcZYeT9G9MRwfLkKHIWGIyPz5ogYaLwYvCE2mBO6+SnpRKo74yIUavxDWi83VGH8n9bNpb1x/Mk8aIyVbZJBA+j0M0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; c=relaxed/simple; bh=Rh8NcshA+JKySXxueL/MKaq6iIyPz5j3yokH2xCJHdU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ewsaMPvGZx1ZB3EzkevbnsoVazcj2+1wlSLcf1gCZYJ4nahKg3lRmB1qNcdwBFzUkLUJSLY7km0J2jSLB/x7vRo8My19DnMdEP+74vEVHsO19xMuLXzhxaNc7PzxUs9uGVitVPSMqGil6ayhrB4/XJfkEXyIZ8ywAd4AKKy4qtE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=PmcYLo6e; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="PmcYLo6e" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=hrswI0VJOBYu3MvyNJ2d8q2GqckwRefPMcCP+neOB8w=; b=PmcYLo6eVkAvHThojyqtMI+euo yF+3/pbWlUwCHRo1PGCGjR15sKwCt7RwPWju0l80uHP8UMwUUMmM3wwpcE9HFbatz0K9liOoICIqE 1BCcSK8Piyi6SCfo6W4rzteYWP37cDIxdeyngGENhrHwTsP0+czNyzqejlOHt9q4wFBYaRiVTbum/ V0nh7KPOyoTBWspbL3n5iXNjyq7Xv2Lt322d7I3BSUiMeVQL6BubqFtendZgBPZjOSD9+40kXGWlz /ytwL03mXjUVELkf41xmc17S53LUMTcB5FOAjc5pEU7Ity6PgLa9OHP7VmCTEwr+CX5/Ntg5cEqlR ugyQ8hEw==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpg-003m3x-06; Wed, 03 Jun 2026 13:40:32 +0000 From: Breno Leitao Date: Wed, 03 Jun 2026 06:40:10 -0700 Subject: [PATCH v2 3/4] workqueue: defer the worker wakeup outside pool->lock in __queue_work() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260603-fastwake-v2-3-2977512fe7fa@debian.org> References: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> In-Reply-To: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=1744; i=leitao@debian.org; h=from:subject:message-id; bh=Rh8NcshA+JKySXxueL/MKaq6iIyPz5j3yokH2xCJHdU=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+8QU697uyoPjgWM5obezKLndTtXQ61Tb+3 R7cjgJnmV2JAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bev/D/9h6jnT4YWF5x/kbz1QiR4Loc9i1HbxYqYyYIfyarik0nzu6GEYQcbSaN0llPeNYk8Nr6Z K8IavXzF4N4KCEMfqbg3RsTKFQzKSCXY8f7DhqrDTh0hrxtEYrO/YP0M4Y8UK2E8WutQ5EGsXlD g+272cYXq3eZpvPlxDtuGyz8ylGTaPOmU0cNzpry4GdNisaaoQTylFk9vPdj2XX0zc4LFFMjZvn nmK4uPzEmt7NE+HSCN6/vAF9Z94UKjWbK8PlMHTQq397Iolenq0WuXUsTYhxm4+3KlMFFlcmeqe xWJz21l/hNYK6W8UPYI4GVn2kE9X8WB8ay5V27+PN2Z2y6NFnSFEnhfZGHWsGxaTYEVRp0tCZMj VDjFUmD4Ivio9HyXilAW202esfhcNZfPfKTY2IflVfb9QMK66BjTwXoVuX5z8ZRBITTTcsssi2P RZXdyRNyXKKzymHgp+oBvBfNqqwO7aJHtiZ8/mP5/izUmQbaTWiTRMgi3DQwZebBbVIGC1K4RSg PXeW2Z2sO0vYDPzY1SJxvTmD3m/zfi0XQAnIHgis3NWfzVZaQGF5ipK8Yn/37+bWg1G4tORAP4w tD2zZHxSaf5VdxHE2EtULDFNwEjhemwF0WCBJiCbqlGi9Sn+tQ7P1U9jpYVf09e82g3EPMyB1r2 ogZGR2DwdhiwChg== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao __queue_work() is the enqueue hot path: it inserts the work item and calls kick_pool() while holding pool->lock. kick_pool() ends in a wakeup, which takes the target task's rq->lock, so rq->lock nests under pool->lock on every enqueue that wakes a worker on a contended unbound pool. Use kick_pool_pick() to select and claim the worker under pool->lock, queue it on an on-stack wake_q, and issue the wakeup with wake_up_q() right after dropping the lock via raw_spin_unlock_wake(). Worker selection, wake_cpu setup and claiming the worker off pool->idle_list still happen under the lock; only the rq->lock acquisition moves out. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 2811ada6dec6..b4246a801dd8 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -2317,6 +2317,7 @@ static void __queue_work(int cpu, struct workqueue_st= ruct *wq, { struct pool_workqueue *pwq; struct worker_pool *last_pool, *pool; + DEFINE_WAKE_Q(wakeq); unsigned int work_flags; unsigned int req_cpu =3D cpu; =20 @@ -2431,14 +2432,15 @@ static void __queue_work(int cpu, struct workqueue_= struct *wq, =20 trace_workqueue_activate_work(work); insert_work(pwq, work, &pool->worklist, work_flags); - kick_pool(pool); + kick_pool_pick(pool, &wakeq); } else { work_flags |=3D WORK_STRUCT_INACTIVE; insert_work(pwq, work, &pwq->inactive_works, work_flags); } =20 out: - raw_spin_unlock(&pool->lock); + /* deferred kick_pool_pick() wakeup, issued outside pool->lock */ + raw_spin_unlock_wake(&pool->lock, &wakeq); rcu_read_unlock(); } =20 --=20 2.54.0 From nobody Mon Jun 8 20:41:43 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2623C333434 for ; Wed, 3 Jun 2026 13:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; cv=none; b=Tr3jJpiAfRn2AFDoJqEIF1+Cc+mYq6xbKo2l8wG9wFylwb2IorEwerJTLvDj4t6cox9uyWDFOTkt1vzrkOkcjz8g1a6aNhuFMtRxDZFyTANne1FwhTNxhR7K3jTlDEEmHiYHFfvc+pU+m7PYfjzn3VCt24NyL4HWSUSaFZGDQok= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780494047; c=relaxed/simple; bh=PP8L3V9RvwIe7/3EQHRj1yOGzLt+tldSrZcyiaE/5Q4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rGdPvlLa2zHCr6pxU1BXFS9oxzAFe9yQhxnBiVAHlA9U/Y6FfRiN4dfTBLjxj5aZearSpsnDSe4MWD4Udwt0YVamCMaVHQ7+x4fPxKYd7oRQ1yUsGQYRPYTfjxFVvDTJY8CFk42nFH9aMUxEvV4LQrmLmBquZhdMVJDRUvH6IcU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=ULHLQ9Wy; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="ULHLQ9Wy" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=84V6l6lJO/YvuiFUAqRuwGACP7KgIG4dKLq8Inlqnk4=; b=ULHLQ9Wyga/i2FCNnBSdBxTE3N w+rd581bxqsHXE5uT9ZI2WhZ3NRmFIzIMI/qv+LhRs1cS3u9uzdKPNl7dwTPPgKSHvewH9JRZ1UJ1 SLWr3LufJm22ejmwBp8R2lPEVGi7OB8W5FGHOidwhLb1ngnEWEGBBlIRE5wIjxYE6yYm5bnz8YuSb IoMebcZf32Wdj/pgAiIJGHmGrlgff/poOyE1EtthJy1cyRYOqKR03ImiD1j+qpddWwf3sbf+l9kSi pCWUAGcFfLcExTESeBoMTKd2uP2/+sEhtRTI1J9JnhkQ1nFltOK6DmiPi99/K/sEfbZZBsQR9vbti LwLtv8BA==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wUlpk-003m4A-0e; Wed, 03 Jun 2026 13:40:36 +0000 From: Breno Leitao Date: Wed, 03 Jun 2026 06:40:11 -0700 Subject: [PATCH v2 4/4] workqueue: defer the worker wakeup outside pool->lock in process_one_work() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260603-fastwake-v2-4-2977512fe7fa@debian.org> References: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> In-Reply-To: <20260603-fastwake-v2-0-2977512fe7fa@debian.org> To: Tejun Heo , Lai Jiangshan Cc: linux-kernel@vger.kernel.org, marco.crivellari@suse.com, frederic@kernel.org, bigeasy@linutronix.de, Hillf Danton , Breno Leitao , kernel-team@meta.com X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=2998; i=leitao@debian.org; h=from:subject:message-id; bh=PP8L3V9RvwIe7/3EQHRj1yOGzLt+tldSrZcyiaE/5Q4=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqIC6+gn7OlcJ5+su4WhUscE3a0LyIguogbXHOD lyvDB5JaaCJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaiAuvgAKCRA1o5Of/Hh3 bV8QEACLUuDiz2F8pD+QBJ0o4tO1l2dF9RN4yzKLh4hT0mKArVcXfsPwSIiyGmdiaRLkzR/v9Mp 5Pm2zV0C02Mgpn/YJx0PWuJdQbmYt45Qy24HttIiZkKx3mUqjs+6Z9j9chVGJz823UTq8HKESht QnG56libGEQO8kyqrxCF2MxYeiIT9I0V828NEp2+MFvsqb1P24YdxvRx54FYHcU/VXSIA/ZcLaZ 4h+qQ7l1zqPrDPIHhJZskiCxtQ0qB1yk7fp7W26k4OWn5wElkqsloYjSN+YqcCJYf80zNiz/k62 eXJMM11bRoGO+wi/rNR3CnSGQ0pRpydGNhvZWYzqXfWeK3kISBY6lwWqeuR/SwMEqbsigjrM/PN 4ShQab4ow2XtK41CodcylKD6KRhl95aunFz3bzcnEyIjB0goTLefCqICAN4KuGgjXhPYEe+MLXp Kkwbc5L6Uq5X5TcjTJQnvd4YpKHnalOiHCKX0wdTX1yYw3adSrg5sj/DY1A56T9/2oiRLkfxYd6 HvpXnx7ymrNwPJEvoSY56HaTUYaBlMMeHZgm1UkJJXf7GWucx1dqEjxmSxz0uDC4cKmC/PELMRL T+/wmjTXd/yOkMfrbqYM58V9N19qz64JIDA/42dbTTpQWDSAw/xeupzjS/Obc8JxX/7MaZc4gao oCKFs/EtrL5DjZA== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao process_one_work() kicks the pool to chain execution of the remaining work items on WORKER_NOT_RUNNING pools (the UNBOUND and CPU_INTENSIVE ones), calling kick_pool() while holding pool->lock. As in the enqueue path, the wakeup pulls the target rq->lock in under pool->lock. Use kick_pool_pick() to select and claim the worker under pool->lock and issue the wakeup with wake_up_q() after the lock is dropped via raw_spin_unlock_irq_wake(). With both hot paths converted, measured on a CONFIG_SMP x86 VM (8 vCPUs) with the in-tree test_workqueue benchmark (lib/test_workqueue.c; each of 8 producers queues 200000 work items one at a time on a WQ_UNBOUND workqueue, waiting for each to complete), medians of five boots per scope: affinity_scope baseline patched tput p95 (items/s) (items/s) gain drop -------------- --------- --------- ------ ------ cpu 3,611,591 3,568,433 -1.2% +4.6% smt 3,601,697 3,550,632 -1.4% +6.1% cache_shard 341,913 401,213 +17.3% -36.8% cache 320,607 400,560 +24.9% -41.9% numa 324,909 389,202 +19.8% -38.0% system 314,510 392,278 +24.7% -37.5% (p95 drop is the change in the p95 enqueue latency; negative is better.) cpu/smt use per-CPU pools with no producer/consumer contention and are essentially unchanged. On the contended scopes the shorter pool->lock hold time cuts p95 enqueue latency by ~40%, and because this workload is bound by the producer<->worker round-trip that latency reduction also lifts throughput by ~20%. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index b4246a801dd8..238b02edd01d 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -3261,6 +3261,7 @@ __acquires(&pool->lock) { struct pool_workqueue *pwq =3D get_work_pwq(work); struct worker_pool *pool =3D worker->pool; + DEFINE_WAKE_Q(wakeq); unsigned long work_data; int lockdep_start_depth, rcu_start_depth; bool bh_draining =3D pool->flags & POOL_BH_DRAINING; @@ -3315,7 +3316,7 @@ __acquires(&pool->lock) * chain execution of the pending work items for WORKER_NOT_RUNNING * workers such as the UNBOUND and CPU_INTENSIVE ones. */ - kick_pool(pool); + kick_pool_pick(pool, &wakeq); =20 /* * Record the last pool and clear PENDING which should be the last @@ -3326,7 +3327,8 @@ __acquires(&pool->lock) set_work_pool_and_clear_pending(work, pool->id, pool_offq_flags(pool)); =20 pwq->stats[PWQ_STAT_STARTED]++; - raw_spin_unlock_irq(&pool->lock); + /* deferred kick_pool_pick() wakeup, issued outside pool->lock */ + raw_spin_unlock_irq_wake(&pool->lock, &wakeq); =20 rcu_start_depth =3D rcu_preempt_depth(); lockdep_start_depth =3D lockdep_depth(current); --=20 2.54.0