1. Patchew
  2. linux
  3. View series

[PATCH] workqueue: Clear worker->pool in the worker thread context

Lai Jiangshan posted 1 patch 2 months, 2 weeks ago 
kernel/workqueue.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
[PATCH] workqueue: Clear worker->pool in the worker thread context
Posted by Lai Jiangshan 2 months, 2 weeks ago 
From: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Marc Hartmayer reported:
        [   23.133876] Unable to handle kernel pointer dereference in virtual kernel address space
        [   23.133950] Failing address: 0000000000000000 TEID: 0000000000000483
        [   23.133954] Fault in home space mode while using kernel ASCE.
        [   23.133957] AS:000000001b8f0007 R3:0000000056cf4007 S:0000000056cf3800 P:000000000000003d
        [   23.134207] Oops: 0004 ilc:2 [#1] SMP
	(snip)
        [   23.134516] Call Trace:
        [   23.134520]  [<0000024e326caf28>] worker_thread+0x48/0x430
        [   23.134525] ([<0000024e326caf18>] worker_thread+0x38/0x430)
        [   23.134528]  [<0000024e326d3a3e>] kthread+0x11e/0x130
        [   23.134533]  [<0000024e3264b0dc>] __ret_from_fork+0x3c/0x60
        [   23.134536]  [<0000024e333fb37a>] ret_from_fork+0xa/0x38
        [   23.134552] Last Breaking-Event-Address:
        [   23.134553]  [<0000024e333f4c04>] mutex_unlock+0x24/0x30
        [   23.134562] Kernel panic - not syncing: Fatal exception: panic_on_oops

With debuging and analysis, worker_thread() accesses to the nullified
worker->pool when the newly created worker is destroyed before being
waken-up, in which case worker_thread() can see the result detach_worker()
reseting worker->pool to NULL at the begining.

Move the code "worker->pool = NULL;" out from detach_worker() to fix the
problem.

worker->pool had been designed to be constant for regular workers and
changeable for rescuer. To share attaching/detaching code for regular
and rescuer workers and to avoid worker->pool being accessed inadvertently
when the worker has been detached, worker->pool is reset to NULL when
detached no matter the worker is rescuer or not.

To maintain worker->pool being reset after detached, move the code
"worker->pool = NULL;" in the worker thread context after detached.

It is either be in the regular worker thread context after PF_WQ_WORKER
is cleared or in rescuer worker thread context with wq_pool_attach_mutex
held. So it is safe to do so.

Cc: Marc Hartmayer <mhartmay@linux.ibm.com>
Link: https://lore.kernel.org/lkml/87wmjj971b.fsf@linux.ibm.com/
Reported-by: Marc Hartmayer <mhartmay@linux.ibm.com>
Fixes: f4b7b53c94af ("workqueue: Detach workers directly in idle_cull_fn()")
Cc: stable@vger.kernel.org # v6.11+
Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
---
 kernel/workqueue.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index e7b005ff3750..6f2545037e57 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -2709,7 +2709,6 @@ static void detach_worker(struct worker *worker)
 
 	unbind_worker(worker);
 	list_del(&worker->node);
-	worker->pool = NULL;
 }
 
 /**
@@ -2729,6 +2728,7 @@ static void worker_detach_from_pool(struct worker *worker)
 
 	mutex_lock(&wq_pool_attach_mutex);
 	detach_worker(worker);
+	worker->pool = NULL;
 	mutex_unlock(&wq_pool_attach_mutex);
 
 	/* clear leftover flags without pool->lock after it is detached */
@@ -3349,7 +3349,11 @@ static int worker_thread(void *__worker)
 	if (unlikely(worker->flags & WORKER_DIE)) {
 		raw_spin_unlock_irq(&pool->lock);
 		set_pf_worker(false);
-
+		/*
+		 * The worker is dead and PF_WQ_WORKER is cleared, worker->pool
+		 * shouldn't be accessed, reset it to NULL in case otherwise.
+		 */
+		worker->pool = NULL;
 		ida_free(&pool->worker_ida, worker->id);
 		return 0;
 	}
-- 
2.19.1.6.gb485710b
Re: [PATCH] workqueue: Clear worker->pool in the worker thread context
Posted by Marc Hartmayer 2 months, 2 weeks ago 
On Thu, Sep 12, 2024 at 11:23 AM +0800, Lai Jiangshan <jiangshanlai@gmail.com> wrote:
> From: Lai Jiangshan <jiangshan.ljs@antgroup.com>
>
> Marc Hartmayer reported:
>         [   23.133876] Unable to handle kernel pointer dereference in virtual kernel address space
>         [   23.133950] Failing address: 0000000000000000 TEID: 0000000000000483
>         [   23.133954] Fault in home space mode while using kernel ASCE.
>         [   23.133957] AS:000000001b8f0007 R3:0000000056cf4007 S:0000000056cf3800 P:000000000000003d
>         [   23.134207] Oops: 0004 ilc:2 [#1] SMP
> 	(snip)
>         [   23.134516] Call Trace:
>         [   23.134520]  [<0000024e326caf28>] worker_thread+0x48/0x430
>         [   23.134525] ([<0000024e326caf18>] worker_thread+0x38/0x430)
>         [   23.134528]  [<0000024e326d3a3e>] kthread+0x11e/0x130
>         [   23.134533]  [<0000024e3264b0dc>] __ret_from_fork+0x3c/0x60
>         [   23.134536]  [<0000024e333fb37a>] ret_from_fork+0xa/0x38
>         [   23.134552] Last Breaking-Event-Address:
>         [   23.134553]  [<0000024e333f4c04>] mutex_unlock+0x24/0x30
>         [   23.134562] Kernel panic - not syncing: Fatal exception: panic_on_oops
>
> With debuging and analysis, worker_thread() accesses to the nullified
> worker->pool when the newly created worker is destroyed before being
> waken-up, in which case worker_thread() can see the result detach_worker()
> reseting worker->pool to NULL at the begining.
>
> Move the code "worker->pool = NULL;" out from detach_worker() to fix the
> problem.
>
> worker->pool had been designed to be constant for regular workers and
> changeable for rescuer. To share attaching/detaching code for regular
> and rescuer workers and to avoid worker->pool being accessed inadvertently
> when the worker has been detached, worker->pool is reset to NULL when
> detached no matter the worker is rescuer or not.
>
> To maintain worker->pool being reset after detached, move the code
> "worker->pool = NULL;" in the worker thread context after detached.
>
> It is either be in the regular worker thread context after PF_WQ_WORKER
> is cleared or in rescuer worker thread context with wq_pool_attach_mutex
> held. So it is safe to do so.
>
> Cc: Marc Hartmayer <mhartmay@linux.ibm.com>
> Link: https://lore.kernel.org/lkml/87wmjj971b.fsf@linux.ibm.com/
> Reported-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> Fixes: f4b7b53c94af ("workqueue: Detach workers directly in idle_cull_fn()")
> Cc: stable@vger.kernel.org # v6.11+
> Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
> ---

[…snip…]

Hi, Lai,

so I tested several things:

1. f4b7b53c94af ("workqueue: Detach workers directly in
   idle_cull_fn()") is the commit that introduced the bug
2. with your fix applied I cannot reproduce the bug

Therefore

Tested-by: Marc Hartmayer <mhartmay@linux.ibm.com>

Thanks for the quick fix!

---
Kind regards / Beste Grüße
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Wolfgang Wendt
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
Re: [PATCH] workqueue: Clear worker->pool in the worker thread context
Posted by Tejun Heo 2 months, 2 weeks ago 
On Thu, Sep 12, 2024 at 11:23:29AM +0800, Lai Jiangshan wrote:
> From: Lai Jiangshan <jiangshan.ljs@antgroup.com>
> 
> Marc Hartmayer reported:
>         [   23.133876] Unable to handle kernel pointer dereference in virtual kernel address space
>         [   23.133950] Failing address: 0000000000000000 TEID: 0000000000000483
>         [   23.133954] Fault in home space mode while using kernel ASCE.
>         [   23.133957] AS:000000001b8f0007 R3:0000000056cf4007 S:0000000056cf3800 P:000000000000003d
>         [   23.134207] Oops: 0004 ilc:2 [#1] SMP
> 	(snip)
>         [   23.134516] Call Trace:
>         [   23.134520]  [<0000024e326caf28>] worker_thread+0x48/0x430
>         [   23.134525] ([<0000024e326caf18>] worker_thread+0x38/0x430)
>         [   23.134528]  [<0000024e326d3a3e>] kthread+0x11e/0x130
>         [   23.134533]  [<0000024e3264b0dc>] __ret_from_fork+0x3c/0x60
>         [   23.134536]  [<0000024e333fb37a>] ret_from_fork+0xa/0x38
>         [   23.134552] Last Breaking-Event-Address:
>         [   23.134553]  [<0000024e333f4c04>] mutex_unlock+0x24/0x30
>         [   23.134562] Kernel panic - not syncing: Fatal exception: panic_on_oops
> 
> With debuging and analysis, worker_thread() accesses to the nullified
> worker->pool when the newly created worker is destroyed before being
> waken-up, in which case worker_thread() can see the result detach_worker()
> reseting worker->pool to NULL at the begining.
> 
> Move the code "worker->pool = NULL;" out from detach_worker() to fix the
> problem.
> 
> worker->pool had been designed to be constant for regular workers and
> changeable for rescuer. To share attaching/detaching code for regular
> and rescuer workers and to avoid worker->pool being accessed inadvertently
> when the worker has been detached, worker->pool is reset to NULL when
> detached no matter the worker is rescuer or not.
> 
> To maintain worker->pool being reset after detached, move the code
> "worker->pool = NULL;" in the worker thread context after detached.
> 
> It is either be in the regular worker thread context after PF_WQ_WORKER
> is cleared or in rescuer worker thread context with wq_pool_attach_mutex
> held. So it is safe to do so.
> 
> Cc: Marc Hartmayer <mhartmay@linux.ibm.com>
> Link: https://lore.kernel.org/lkml/87wmjj971b.fsf@linux.ibm.com/
> Reported-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> Fixes: f4b7b53c94af ("workqueue: Detach workers directly in idle_cull_fn()")
> Cc: stable@vger.kernel.org # v6.11+
> Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>

Applied to wq/for-6.11-fxes.

Thanks.

-- 
tejun

Patchew 2.1-devel-b67e4c2

© 2016 - 2024 Red Hat, Inc.