From nobody Sat Apr 18 03:25:59 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C8C11B983F for ; Wed, 11 Feb 2026 12:32:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813162; cv=none; b=PHDKl+tu0jk2oeHy8WR/QMKDKmdPJDOE/7vW3kNw3O1Mbj0427W2CWBf1ndHfN94ec4nvErWYSV5Yt0U3+8RhS/09xlGJM03+eh0UWcfRj+1S6P4StI7g8SV6xEeWKbsfbQDOXQRwMjZy/dZNYMFfgX69DdH0LMIj+bYzL3rMEc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813162; c=relaxed/simple; bh=KPZEFqB87t2rBUIKQUeiu6xy1f+lMRscLe8C4JBNEpM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rcefstK/vSC7eeNezTNcY5SuVamRRHRkwRt1LLROigDvV39T5tQgq3ZamTFWDT4VPzYE4WBey5h4Jry1flJ8mRj1GEiqf3Ttbden53Vr21pK9slqgxr0a0dD3gHu/hZdJXhrlA21BF9JZpdoUtjM1dhpr60Y/sTHnGTxZshg78c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=a35bSUn8; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="a35bSUn8" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=AJlHlAwBVx/FIfh9aEfWIHI96KrrW5jCmJlWcYeF0cY=; b=a35bSUn89igwmhwOpcr0oWKEQp eqljT4TV3LmiE1c70mOkPj6pvWMTzdK/XI9RSsMlNaqEEavJvPHXBRo8yaphz3R53j0Caf2acUGdT ZWaFE5nJgrp+BRI21Ca2DF+bS4yuA7rI/fmm1BliqiNlcdN3c677nxtFd2cLQWeBZB6Ng6r+c8OCE v5m7tSARFL4L4PbqIyzjYibRozjvWlvQ6r0jq2U+bsH8Cx+BdiwGGOLP7mSB7I0Vv80saZYygU9qR ZvfTtk1+75onStrxevAvfQSM1HtqqcA2JMR+MZd1bOsOQ1yq+yxOJ/gu+dvH2yCumKHTShg95rc6Q jXA8dCbw==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1vq9OW-00B22a-Fh; Wed, 11 Feb 2026 12:32:36 +0000 From: Breno Leitao Date: Wed, 11 Feb 2026 04:29:15 -0800 Subject: [PATCH 1/4] workqueue: Use POOL_BH instead of WQ_BH when checking pool flags Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260211-wqstall_start-at-v1-1-bd9499a18c19@debian.org> References: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> In-Reply-To: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, Omar Sandoval , kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.15-dev-f4305 X-Developer-Signature: v=1; a=openpgp-sha256; l=1075; i=leitao@debian.org; h=from:subject:message-id; bh=KPZEFqB87t2rBUIKQUeiu6xy1f+lMRscLe8C4JBNEpM=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpjHbc73eiFJMNLeu1kwHJoA4nysg8SdOayjXhp RQN/fLPUZCJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaYx23AAKCRA1o5Of/Hh3 bfYED/0Y6gm3JFM3nqdhhM/oA6JSrEuclbO3/8HfYpCJWX+fGpir23HCtd8oQjcpINp+WGe0Zzw 1hGUxYcTDBhDxs6wzejvNwOzPqERvchys81rENTlHpP/ldBe/gWpm7KZo8VT1758oY5b8bxaJdZ MwqlA0qmMlSfovm/eQs1Ju8BOLOcVq1yVVB891OZWfA8fKNAId1vyZcjtP0GRdjp2aLzO1m5h28 b1birxtJJ0P2klNzGboORnE9SPfgaTmZb6qGo1UmDKtyb5PZGLtjYuRjd3b1GPs1kn8c2YFEuYK 1ZoTzIi1zuN3cuQtHfSNJGZqs7XOAq1LrhrOEg8pGig7t2pAdE0Gzd61eSYDAwbBnIE/vvFxfNJ kXBp1R6R1Hii210MsEOSmS+SEWdTAoq2qfZNLDjBa5qz13I7S8l91XOVKPIc6GVWjBzT40OYhDE H0albclhkSd3IU866PL91Y2VE96hbe58TQJvuGyNqcXWa807cnm1FeMZWZx1Jbt0oG86JRhm6pj iKX4q1y50eBdqW9t4j4PHdi1E3ngW4D4q30gBT/BA43fPjgIMPHURN7CookxNBpi8JS73INgbWu 3Jzka74tSqZMUl7ziLDa+2oxIrxLiPiOTVQAduc8492h3VN1DtFReVHciwmhs6mhWuD1l3YI3lh qHelm9QmxsYBBMw== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao pr_cont_worker_id() checks pool->flags against WQ_BH, which is a workqueue-level flag (defined in workqueue.h). Pool flags use a separate namespace with POOL_* constants (defined in workqueue.c). The correct constant is POOL_BH. Both WQ_BH and POOL_BH are defined as (1 << 0) so this has no behavioral impact, but it is semantically wrong and inconsistent with every other pool-level BH check in the file. Fixes: 4cb1ef64609f ("workqueue: Implement BH workqueues to eventually repl= ace tasklets") Signed-off-by: Breno Leitao --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index e6c249f2fb46b..265d841e1b81c 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -6274,7 +6274,7 @@ static void pr_cont_worker_id(struct worker *worker) { struct worker_pool *pool =3D worker->pool; =20 - if (pool->flags & WQ_BH) + if (pool->flags & POOL_BH) pr_cont("bh%s", pool->attrs->nice =3D=3D HIGHPRI_NICE_LEVEL ? "-hi" : ""); else --=20 2.47.3 From nobody Sat Apr 18 03:25:59 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEB3E28F50F for ; Wed, 11 Feb 2026 12:32:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813166; cv=none; b=C3lRpCY15yw9wigbheQ5R8K9L0y7GhTMEbcc764hsbsJjO+KL2ygHE8UMqKzWbxoOnlGkS8z7gvadxjBTbyHoeMqcrh3WgoWDaKWsiXJL3Qac5g4++jzQw9jsLsATzIU8YhQPx2eDVRpQZbKz3yTyGknRv2GH32FojDksirwMrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813166; c=relaxed/simple; bh=PaMKfS3oOVI4Hu9v3A8dH98Yf18DSDx4TRNt2oPK+Zs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=sdoN1BftGyNw4lzdGY8DRcn/NEzoqUOE0IWrzoxLo7t4pEh1jDzM6H/41u2a7alaSpdDwLJVuehKv+/Q+0OzpPIWy1xMy018b+33hxiqsgvr2lW1DIWEhlAUjHJExJ8VWl+LG7KZc7RDLyjV0qgWespbzEvLu+hgVo+TWXzMPok= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=M6UY8LD0; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="M6UY8LD0" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=iDDy54e0brmx47WRdslaCKq/+Ha+VGBjeV2WAK1uklY=; b=M6UY8LD0zmnorNFs+AqCRVY3Cu rrN3gv0ruEkYAZej6Ut1pPOcESt2df1KRzxNI+XuKpWfMCQ5D8/Yu5yYUy+NH2UaLjAwiNzxKn+Me qrBcNb7DNvFfzHDiwGL2VM72YiJoLll9KC4JKj2GCupG8Y+9i2QXZeezTDZOuAAkYG5wkW6dWCjTe BvwoDhAHXGX7TkB93UCikEAtRyiKzdhpRe2QXntjJwb4wvkfb6atG7jkerLfhT9hJllFJ2BzqwVWL ULqnmCmUIv/eduqie+9avCKtJYECR9ZXpBUB2sQFAC/NDAFwweMFL+LptqcT7hN9m8Sn2bk/TkNUd s1fynMQw==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1vq9OZ-00B22k-TT; Wed, 11 Feb 2026 12:32:40 +0000 From: Breno Leitao Date: Wed, 11 Feb 2026 04:29:16 -0800 Subject: [PATCH 2/4] workqueue: Rename pool->watchdog_ts to pool->last_progress_ts Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260211-wqstall_start-at-v1-2-bd9499a18c19@debian.org> References: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> In-Reply-To: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, Omar Sandoval , kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.15-dev-f4305 X-Developer-Signature: v=1; a=openpgp-sha256; l=3162; i=leitao@debian.org; h=from:subject:message-id; bh=PaMKfS3oOVI4Hu9v3A8dH98Yf18DSDx4TRNt2oPK+Zs=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpjHbcobf5wkSpOXDRatcxIb1h0P6h1/AosZEjY ZNXhRojLx6JAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaYx23AAKCRA1o5Of/Hh3 bfW4D/44/CZCBhK2uwXoKlFJt3Z8t7uHvHtuvtVTlIRtFQFfp40/fqjf4bCl6ws4Yt82DdnNiE2 awAxgvgPAkKTS4XLQY7IdsOs3AATMIB1+/1aF44txxW3+QGuyWRwgqRi4gI1nHubuegy7+CdDrh c51NFCAcGkhovt7IqHqRIxjwy2d6X86nH8LTSNDOKuF61EQ1nvUd6p6UKfrYkLfxcnnqe5pmadL CxOgbRW3CFey6bIbr7cwfy1Z3Ps/+KcVtOUrrUYkF59544E6RTaFUZZh91/RalWQu+sZzSOmyE1 3LOQXmJAI3eGd9fDypBACxiU0iy9yYOwf0ndn62B3NZxUYkaSJCTHUWO/eyGjDQsknsOmjl3HSZ DjuotkcIso7OHbcGm0sAEgAbYDhfc4q5X6em9eIb+vWU1JOLPFwM/wR1vSc3+CZzbT06TeEkTEH k72jOi53/FLjJoNSvgJ0mkCvpHAsNehAk+QLovUd2/vWfBrVM/4TOne+si4pxfV2kikCZlaNF5N DOb/nkOBes36WnpoEE35DzcQP5HBFtrv/FHn5nugObBH5mMk3QFZEWflexlZl5S0nJqVh+U9Ea0 ptMUvhar8FryltBN8RirQjefid1Sy7TIXEFaDXxckeLebfGqKIDMWfgN6g8jnz9uWCpe5lyAarf FUBOZ3uz5FDENKw== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The watchdog_ts name doesn't convey what the timestamp actually tracks. This field tracks the last time a workqueue got progress. Rename it to last_progress_ts to make it clear that it records when the pool last made forward progress (started processing new work items). No functional change. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 265d841e1b81c..b3ba739cf493a 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -190,7 +190,7 @@ struct worker_pool { int id; /* I: pool ID */ unsigned int flags; /* L: flags */ =20 - unsigned long watchdog_ts; /* L: watchdog timestamp */ + unsigned long last_progress_ts; /* L: last forward progress timestamp */ bool cpu_stall; /* WD: stalled cpu bound pool */ =20 /* @@ -1697,7 +1697,7 @@ static void __pwq_activate_work(struct pool_workqueue= *pwq, WARN_ON_ONCE(!(*wdb & WORK_STRUCT_INACTIVE)); trace_workqueue_activate_work(work); if (list_empty(&pwq->pool->worklist)) - pwq->pool->watchdog_ts =3D jiffies; + pwq->pool->last_progress_ts =3D jiffies; move_linked_works(work, &pwq->pool->worklist, NULL); __clear_bit(WORK_STRUCT_INACTIVE_BIT, wdb); } @@ -2348,7 +2348,7 @@ static void __queue_work(int cpu, struct workqueue_st= ruct *wq, */ if (list_empty(&pwq->inactive_works) && pwq_tryinc_nr_active(pwq, false))= { if (list_empty(&pool->worklist)) - pool->watchdog_ts =3D jiffies; + pool->last_progress_ts =3D jiffies; =20 trace_workqueue_activate_work(work); insert_work(pwq, work, &pool->worklist, work_flags); @@ -3352,7 +3352,7 @@ static void process_scheduled_works(struct worker *wo= rker) while ((work =3D list_first_entry_or_null(&worker->scheduled, struct work_struct, entry))) { if (first) { - worker->pool->watchdog_ts =3D jiffies; + worker->pool->last_progress_ts =3D jiffies; first =3D false; } process_one_work(worker, work); @@ -4850,7 +4850,7 @@ static int init_worker_pool(struct worker_pool *pool) pool->cpu =3D -1; pool->node =3D NUMA_NO_NODE; pool->flags |=3D POOL_DISASSOCIATED; - pool->watchdog_ts =3D jiffies; + pool->last_progress_ts =3D jiffies; INIT_LIST_HEAD(&pool->worklist); INIT_LIST_HEAD(&pool->idle_list); hash_init(pool->busy_hash); @@ -6462,7 +6462,7 @@ static void show_one_worker_pool(struct worker_pool *= pool) =20 /* How long the first pending work is waiting for a worker. */ if (!list_empty(&pool->worklist)) - hung =3D jiffies_to_msecs(jiffies - pool->watchdog_ts) / 1000; + hung =3D jiffies_to_msecs(jiffies - pool->last_progress_ts) / 1000; =20 /* * Defer printing to avoid deadlocks in console drivers that @@ -7688,7 +7688,7 @@ static void wq_watchdog_timer_fn(struct timer_list *u= nused) touched =3D READ_ONCE(per_cpu(wq_watchdog_touched_cpu, pool->cpu)); else touched =3D READ_ONCE(wq_watchdog_touched); - pool_ts =3D READ_ONCE(pool->watchdog_ts); + pool_ts =3D READ_ONCE(pool->last_progress_ts); =20 if (time_after(pool_ts, touched)) ts =3D pool_ts; --=20 2.47.3 From nobody Sat Apr 18 03:25:59 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63CF23595D for ; Wed, 11 Feb 2026 12:32:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813169; cv=none; b=DUwZCTNcUazxxejWqUdyd1XQ36mqOMrUVSu87sG9HqdLQnGhBlutmYFpQ7IqxTz45smBvuuW9jrVJUJ99YmpyttVlTACDUWFykyjpAmIYtHpo1e+QV3XF7Q8qr43FAUVmNaTKQPMb7ZaKuFPGpij2RXfyrHlBseNGB7e5YZB4tQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813169; c=relaxed/simple; bh=GJemNy6Ki2QA5V3h7PH87surwvD6XefdRUU/bYZ4Isc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=jC9halHsRce8pMuhGcUiMmW4sjAuU7/+nciZnHkVknpidLccm2ZSdGBMOEspFnzQZEITMWMfqu2PZKiTW0GpYYYgGnXN0r+pcT3V3lkzfbGtBIoX0bJ7greMOiKklMjdxUH88i6BtTF+22xYuKj3Kiy9lsIWrhTXKG+khkfQbYA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=WLH1vfXX; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="WLH1vfXX" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=eyHDShenyeYpvpIVQXSORsBn0Sb9ydGp35ahaTlKk+8=; b=WLH1vfXXOtKJzJBqibSLIgbGL9 MzgCRDWWP4GR/5kDds/qKauAI3/GNxFrrFnm+lmKOptF4JGzVLNXwZNqt1Ay7hT8Gl+xCBodO2YWs KP0BQAXnMkOskBjmoK5zlXGs5/Ln9TcT+3Z3OcOI8nEwAFaFXQHglsX6qbCWboSwMQr+6B/IWbKHU cTx/gb8PjrTTy4Obr4fynFkv38gyGFQKDs5LfgxAL0EwKcOS4roMsySvk8qFIxIMB8S9m+ImcRbzw N1GC5J3R3w9fZly8z6zwq84mNwH9hb/WT5WEbj07qYEVkrnzFoqzW0dxRE3efyW42K45blso8JxcB qSiAd6/Q==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1vq9Od-00B22u-Af; Wed, 11 Feb 2026 12:32:43 +0000 From: Breno Leitao Date: Wed, 11 Feb 2026 04:29:17 -0800 Subject: [PATCH 3/4] workqueue: Show in-flight work item duration in stall diagnostics Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260211-wqstall_start-at-v1-3-bd9499a18c19@debian.org> References: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> In-Reply-To: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, Omar Sandoval , kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.15-dev-f4305 X-Developer-Signature: v=1; a=openpgp-sha256; l=2206; i=leitao@debian.org; h=from:subject:message-id; bh=GJemNy6Ki2QA5V3h7PH87surwvD6XefdRUU/bYZ4Isc=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpjHbcgG/l34DQUyH7MN7zmFTEWQ6ql/Ejd8GCn DxXLewIN7uJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaYx23AAKCRA1o5Of/Hh3 bXHyD/4xK1UGcGP+PvrTxxiuF94Oyjm88y0isqSASjwXgQucX2Q1piRIEHlTZHwpjH1KldsyWlH lbdOcT5pQ6cDlTRTEH2qzk/m94Z5V2Vav4do6nTYfHePRjj3BNzZXNYY4bvtY9wWe6WNQuz1YLZ oY0yUb9NeCCEgI8Ddn5WBpECDLGh3G5UskS1FZ9H+WsEvRBr7GE8/HQRITCGdQB3/FR/af5hi1E lLcbibbHBv7UNE5BZckqGsHTOL8ky6Ca0pvZGq1Vk7x/O2Z+cQKsUHcmL5EUKt7oaiWSVYqEcA6 kqJP29Cg1rK9Mku7nlWLmP+5rocUIhls1KszPXhffAEINbvAGYQwzMRYEli916GD7nXkXE8e4HA b7ZvTE6uHc1/N7uFJ4/yKDE5AZoo8Y/BXeX3kmT7xLGbowb8jsCH3bufDWl3GRPKW1OYnQZOSpl Xnlimrw0pvwENYXgo9Db4UKm0T9UdJqUrU+UAj16DWJiCfvIKWoxDxSR4Pa8jn4hA5E/NzwlG/q WgacWLZo1NJeKoRy15UUty2nDL7qEi+RrJ3EA+1F5Py4W91fz6KmxgbWJgpUan97ckM1ETlRZlM tq1AhcYJtYq0ptfTPHmZ8oOaTaLp1aANxVIWFXpk+nPPQ3Uc4lWmwANcuVrvDQbkclaqgyzeAga 4cc7GQfsOOfRCTQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao When diagnosing workqueue stalls, knowing how long each in-flight work item has been executing is valuable. Add a current_start timestamp (jiffies) to struct worker, set it when a work item begins execution in process_one_work(), and print the elapsed wall-clock time in show_pwq(). Unlike current_at (which tracks CPU runtime and resets on wakeup for CPU-intensive detection), current_start is never reset because the diagnostic cares about total wall-clock time including sleeps. Before: in-flight: 165:stall_work_fn [wq_stall] After: in-flight: 165:stall_work_fn [wq_stall] for 100s Signed-off-by: Breno Leitao --- kernel/workqueue.c | 3 +++ kernel/workqueue_internal.h | 1 + 2 files changed, 4 insertions(+) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index b3ba739cf493a..e527e763162e6 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -3204,6 +3204,7 @@ __acquires(&pool->lock) worker->current_pwq =3D pwq; if (worker->task) worker->current_at =3D worker->task->se.sum_exec_runtime; + worker->current_start =3D jiffies; work_data =3D *work_data_bits(work); worker->current_color =3D get_work_color(work_data); =20 @@ -6359,6 +6360,8 @@ static void show_pwq(struct pool_workqueue *pwq) pr_cont(" %s", comma ? "," : ""); pr_cont_worker_id(worker); pr_cont(":%ps", worker->current_func); + pr_cont(" for %us", + jiffies_to_msecs(jiffies - worker->current_start) / 1000); list_for_each_entry(work, &worker->scheduled, entry) pr_cont_work(false, work, &pcws); pr_cont_work_flush(comma, (work_func_t)-1L, &pcws); diff --git a/kernel/workqueue_internal.h b/kernel/workqueue_internal.h index f6275944ada77..8def1ddc5a1bf 100644 --- a/kernel/workqueue_internal.h +++ b/kernel/workqueue_internal.h @@ -32,6 +32,7 @@ struct worker { work_func_t current_func; /* K: function */ struct pool_workqueue *current_pwq; /* K: pwq */ u64 current_at; /* K: runtime at start or last wakeup */ + unsigned long current_start; /* K: start time of current work item */ unsigned int current_color; /* K: color */ =20 int sleeping; /* S: is worker sleeping? */ --=20 2.47.3 From nobody Sat Apr 18 03:25:59 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7567E246798 for ; Wed, 11 Feb 2026 12:32:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813172; cv=none; b=KzJEO4UPLqjDTpiezqXqZ3jnKQMtDj/8hGmwJ52UBiBbPv7DdHz5sp/bQW4EUK7OI+wtxFevFdTLLuPnX041snNGRNiKbcBkaL3Rd/gncwjoUJcXcY8BjDt0ZP0ofUETnzo6Ym7LEtXe1ChrC24BcXJ1uiFkP+TiFYd8rmY9Rh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770813172; c=relaxed/simple; bh=t77vSLWgXMMeEcQf9E3d/8zqaT20FgoqfU8sZDvzcVw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=khqQe5l2GjQFySSSZ8p5uZaEbf8CMxszSwzVIiz5d8wbTNwtWcLp8NUOSr2mrC9t7H7im9v6I31gOjSe23mquXl552xabe/Hxi2XoRDDwlVKYzSUxwbbzicDI1cREJ2+t3sA818enGCLsnicNQsqi6ixdXgM77WTxQNuPF9M/cw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=jvSswlwv; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="jvSswlwv" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=vzGfRiXeh6X8RFTmty5cMTgqMHPVmYvoXIEuiBDVbgk=; b=jvSswlwvMZi429+mqVZYNGAKaQ MzlwpofH08012DkltNTEsB/WRnFZj3gEF0oeQAxPE9TevaF7QSgpGRlWCCoknQiDrEJ4W7r8Zfwnq pu+v89qBtAAGPK5IoLEcxOslxwTy1tO/FY6dQmABE4ZE15TkgEYWM6BBqdAM0bcl3pbh6LbOAeLGF rwJc0mw7lCHRicSoGN6cfUYcKVtUjApAhzKN06QWuNinR6R6e+KFftZ8+tJEfh1PfNz1ZZ+53oHNg XEkL8xidiSp7IykIK5kR2zOJoDNiBliGYuOBGVRF07YIAFfxPLF26+tAV7Hk9tBQdmSsU7xkzjzj0 UJWTlT7A==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1vq9Og-00B23B-MP; Wed, 11 Feb 2026 12:32:47 +0000 From: Breno Leitao Date: Wed, 11 Feb 2026 04:29:18 -0800 Subject: [PATCH 4/4] workqueue: Detect stalled in-flight work items with empty worklist Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260211-wqstall_start-at-v1-4-bd9499a18c19@debian.org> References: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> In-Reply-To: <20260211-wqstall_start-at-v1-0-bd9499a18c19@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, Omar Sandoval , kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.15-dev-f4305 X-Developer-Signature: v=1; a=openpgp-sha256; l=5044; i=leitao@debian.org; h=from:subject:message-id; bh=t77vSLWgXMMeEcQf9E3d/8zqaT20FgoqfU8sZDvzcVw=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpjHbcKPhWrwIYvHDyv8c7iswhohrChgddiukIr 5f/o67TAiOJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCaYx23AAKCRA1o5Of/Hh3 bQRXEACSlLnvopofkkSd4WBguhrh3VvU7H3yvBwsWxj3Jx95/ias31MgH3mjjaKRLhdrqpZHsRk KbZ6/ZaF+lOOpKEKUkzRFtR8EMyhQ3XePtWVpcNmp7OwOFBYZqXKzu0LIiV0hHmXg926FNwUcrG ljVr2rZu2VpF6JCPIEaO8dzxTmHBGKrNeacqc81WDwrq5CRJDMXDbhbU/vsn/f/AdDVMfrepcbD FoMnQtw5+CimZADW3PZQ0dXMtgd7usLBtTTw+hZhWZe9VvS5SxgOxVaSbMYu6r4+UbxSbg/mNNI Cgp/A802X17lDF9B3o/RU222K/ufRnnPF9jr0ex/7Qi1wKQecvjks5dbCpCbMPCdQx/G9r+EedO /kcH6g7egdYjs5NBfNMKpHAt1pFwYjelg68p7VJjjcMVczg6+oCjyGaBabSkZ4ZBCiNshkysv1m TYWU9J9b7fhfUplGR5NyAzWd+F35Sh9zQQMHz263jtjZzuHkFYhkFcvlY79x5iAHOrLW3rVsJD7 T+n2N7mC7RmfQdxxboXvTMdxSXGEhfq7drJiWrWVxYuid5+bkYC1tYbbZKnWBk3+jZHXUUywY+D ECvLfY0UvFYl110Cem33EZSTWu+/L3Tn2ufPtgHplBC0X8flAr8lijk9+AES4f8kYN87I1UwAhX k+p9MxoWFuQWCvQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The workqueue watchdog skips pools with an empty worklist, assuming no work is pending. However, a single work item that was dequeued and is now executing on a worker will leave the worklist empty while the worker is stuck. This means a pool with one hogged worker and no pending work is invisible to the watchdog. An example is something like: static void stall_work_fn(struct work_struct *work) { for (;;) { mdelay(1000); cond_resched(); } } Fix this by scanning the pool's busy_hash for workers whose current_start timestamp exceeds the watchdog threshold, independent of worklist state. The new report_stalled_workers() function iterates all in-flight workers in a pool and reports each one that has exceeded the threshold, running as a separate detection path alongside the existing pool-level last_progress_ts check. This is an example of the report: BUG: workqueue lockup - worker 365:stall_work_fn [wq_stall] stuck in pool= cpus=3D9 node=3D0 flags=3D0x0 nice=3D0 for 33s! Showing busy workqueues and worker pools: ... The feature is gated behind a new CONFIG_WQ_WATCHDOG_WORKERS option (disabled by default) under CONFIG_WQ_WATCHDOG. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++= -- lib/Kconfig.debug | 12 ++++++++++++ 2 files changed, 62 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index e527e763162e6..719e14aa4ac56 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -7659,6 +7659,49 @@ static void wq_watchdog_reset_touched(void) per_cpu(wq_watchdog_touched_cpu, cpu) =3D jiffies; } =20 +#ifdef CONFIG_WQ_WATCHDOG_WORKERS +/* + * Scan all in-flight workers in @pool for stalls. A worker is considered + * stalled if its current work item has been executing for longer than @th= resh + * based on its current_start timestamp. This catches workers that are stu= ck + * regardless of the pool's worklist state or last_progress_ts. + */ +static bool report_stalled_workers(struct worker_pool *pool, + unsigned long now, + unsigned long thresh) +{ + struct worker *worker; + bool stall =3D false; + int bkt; + + /* + * Iterate busy_hash without pool->lock. This is intentionally + * lockless to avoid contention in the watchdog timer path. + * Workers that have been stalled for thresh (typically 30s) are + * unlikely to be transitioning in/out of busy_hash concurrently. + */ + hash_for_each(pool->busy_hash, bkt, worker, hentry) { + if (time_after(now, worker->current_start + thresh)) { + pr_emerg("BUG: workqueue lockup - worker "); + pr_cont_worker_id(worker); + pr_cont(":%ps stuck in pool", + worker->current_func); + pr_cont_pool_info(pool); + pr_cont(" for %us!\n", + jiffies_to_msecs(now - worker->current_start) / 1000); + stall =3D true; + } + } + return stall; +} +#else +static bool report_stalled_workers(struct worker_pool *pool, + unsigned long now, unsigned long thresh) +{ + return false; +} +#endif /* CONFIG_WQ_WATCHDOG_WORKERS */ + static void wq_watchdog_timer_fn(struct timer_list *unused) { unsigned long thresh =3D READ_ONCE(wq_watchdog_thresh) * HZ; @@ -7677,8 +7720,6 @@ static void wq_watchdog_timer_fn(struct timer_list *u= nused) unsigned long pool_ts, touched, ts; =20 pool->cpu_stall =3D false; - if (list_empty(&pool->worklist)) - continue; =20 /* * If a virtual machine is stopped by the host it can look to @@ -7686,6 +7727,13 @@ static void wq_watchdog_timer_fn(struct timer_list *= unused) */ kvm_check_and_clear_guest_paused(); =20 + /* Check for individual stalled workers in this pool. */ + if (report_stalled_workers(pool, now, thresh)) + lockup_detected =3D true; + + if (list_empty(&pool->worklist)) + continue; + /* get the latest of pool and touched timestamps */ if (pool->cpu >=3D 0) touched =3D READ_ONCE(per_cpu(wq_watchdog_touched_cpu, pool->cpu)); diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce25a8faf6e9e..dc4bb546b2033 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1320,6 +1320,18 @@ config BOOTPARAM_WQ_STALL_PANIC This setting can be overridden at runtime via the workqueue.panic_on_stall kernel parameter. =20 +config WQ_WATCHDOG_WORKERS + bool "Detect individual stalled workqueue workers" + depends on WQ_WATCHDOG + default n + help + Say Y here to enable per-worker stall detection. When enabled, + the workqueue watchdog scans all in-flight workers in each pool + and reports any whose current work item has been executing for + longer than the watchdog threshold. This catches stalled workers + even when the pool's worklist is empty or the pool has recently + made forward progress on other work items. + config WQ_CPU_INTENSIVE_REPORT bool "Report per-cpu work items which hog CPU for too long" depends on DEBUG_KERNEL --=20 2.47.3