dm: remove fake timeout to avoid leak request

[PATCH] dm: remove fake timeout to avoid leak request

Posted by Ding Hui 1 month, 2 weeks ago

Since commit 15f73f5b3e59 ("blk-mq: move failure injection out of
blk_mq_complete_request"), drivers are responsible for calling
blk_should_fake_timeout() at appropriate code paths and opportunities.

However, the dm driver does not implement its own timeout handler and
relies on the timeout handling of its slave devices.

If an io-timeout-fail error is injected to a dm device, the request
will be leaked and never completed, causing tasks to hang indefinitely.

Reproduce:
1. prepare dm which has iscsi slave device
2. inject io-timeout-fail to dm
   echo 1 >/sys/class/block/dm-0/io-timeout-fail
   echo 100 >/sys/kernel/debug/fail_io_timeout/probability
   echo 10 >/sys/kernel/debug/fail_io_timeout/times
3. read/write dm
4. iscsiadm -m node -u

Result: hang task like below
[  862.243768] INFO: task kworker/u514:2:151 blocked for more than 122 seconds.
[  862.244133]       Tainted: G            E       6.19.0-rc1+ #51
[  862.244337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.244718] task:kworker/u514:2  state:D stack:0     pid:151   tgid:151   ppid:2      task_flags:0x4288060 flags:0x00080000
[  862.245024] Workqueue: iscsi_ctrl_3:1 __iscsi_unbind_session [scsi_transport_iscsi]
[  862.245264] Call Trace:
[  862.245587]  <TASK>
[  862.245814]  __schedule+0x810/0x15c0
[  862.246557]  schedule+0x69/0x180
[  862.246760]  blk_mq_freeze_queue_wait+0xde/0x120
[  862.247688]  elevator_change+0x16d/0x460
[  862.247893]  elevator_set_none+0x87/0xf0
[  862.248798]  blk_unregister_queue+0x12e/0x2a0
[  862.248995]  __del_gendisk+0x231/0x7e0
[  862.250143]  del_gendisk+0x12f/0x1d0
[  862.250339]  sd_remove+0x85/0x130 [sd_mod]
[  862.250650]  device_release_driver_internal+0x36d/0x530
[  862.250849]  bus_remove_device+0x1dd/0x3f0
[  862.251042]  device_del+0x38a/0x930
[  862.252095]  __scsi_remove_device+0x293/0x360
[  862.252291]  scsi_remove_target+0x486/0x760
[  862.252654]  __iscsi_unbind_session+0x18a/0x3e0 [scsi_transport_iscsi]
[  862.252886]  process_one_work+0x633/0xe50
[  862.253101]  worker_thread+0x6df/0xf10
[  862.253647]  kthread+0x36d/0x720
[  862.254533]  ret_from_fork+0x2a6/0x470
[  862.255852]  ret_from_fork_asm+0x1a/0x30
[  862.256037]  </TASK>

Remove the blk_should_fake_timeout() check from dm, as dm has no
native timeout handling and should not attempt to fake timeouts.

Signed-off-by: Ding Hui <dinghui@sangfor.com.cn>
---
 drivers/md/dm-rq.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
index a6ca92049c10..5e0854669614 100644
--- a/drivers/md/dm-rq.c
+++ b/drivers/md/dm-rq.c
@@ -278,8 +278,7 @@ static void dm_complete_request(struct request *rq, blk_status_t error)
 	struct dm_rq_target_io *tio = tio_from_request(rq);
 
 	tio->error = error;
-	if (likely(!blk_should_fake_timeout(rq->q)))
-		blk_mq_complete_request(rq);
+	blk_mq_complete_request(rq);
 }
 
 /*
-- 
2.17.1

Re: [PATCH] dm: remove fake timeout to avoid leak request

Posted by Ding Hui 3 weeks, 5 days ago

Hi,

Sorry for this ping mail as there is no feedback from maintainers, just in
case the patch was forgotten somewhere.

And it has already received a Reviewed-by from Christoph last month.

On 2025/12/20 20:03, Ding Hui wrote:
> Since commit 15f73f5b3e59 ("blk-mq: move failure injection out of
> blk_mq_complete_request"), drivers are responsible for calling
> blk_should_fake_timeout() at appropriate code paths and opportunities.
> 
> However, the dm driver does not implement its own timeout handler and
> relies on the timeout handling of its slave devices.
> 
> If an io-timeout-fail error is injected to a dm device, the request
> will be leaked and never completed, causing tasks to hang indefinitely.
> 
> Reproduce:
> 1. prepare dm which has iscsi slave device
> 2. inject io-timeout-fail to dm
>     echo 1 >/sys/class/block/dm-0/io-timeout-fail
>     echo 100 >/sys/kernel/debug/fail_io_timeout/probability
>     echo 10 >/sys/kernel/debug/fail_io_timeout/times
> 3. read/write dm
> 4. iscsiadm -m node -u
> 
> Result: hang task like below
> [  862.243768] INFO: task kworker/u514:2:151 blocked for more than 122 seconds.
> [  862.244133]       Tainted: G            E       6.19.0-rc1+ #51
> [  862.244337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [  862.244718] task:kworker/u514:2  state:D stack:0     pid:151   tgid:151   ppid:2      task_flags:0x4288060 flags:0x00080000
> [  862.245024] Workqueue: iscsi_ctrl_3:1 __iscsi_unbind_session [scsi_transport_iscsi]
> [  862.245264] Call Trace:
> [  862.245587]  <TASK>
> [  862.245814]  __schedule+0x810/0x15c0
> [  862.246557]  schedule+0x69/0x180
> [  862.246760]  blk_mq_freeze_queue_wait+0xde/0x120
> [  862.247688]  elevator_change+0x16d/0x460
> [  862.247893]  elevator_set_none+0x87/0xf0
> [  862.248798]  blk_unregister_queue+0x12e/0x2a0
> [  862.248995]  __del_gendisk+0x231/0x7e0
> [  862.250143]  del_gendisk+0x12f/0x1d0
> [  862.250339]  sd_remove+0x85/0x130 [sd_mod]
> [  862.250650]  device_release_driver_internal+0x36d/0x530
> [  862.250849]  bus_remove_device+0x1dd/0x3f0
> [  862.251042]  device_del+0x38a/0x930
> [  862.252095]  __scsi_remove_device+0x293/0x360
> [  862.252291]  scsi_remove_target+0x486/0x760
> [  862.252654]  __iscsi_unbind_session+0x18a/0x3e0 [scsi_transport_iscsi]
> [  862.252886]  process_one_work+0x633/0xe50
> [  862.253101]  worker_thread+0x6df/0xf10
> [  862.253647]  kthread+0x36d/0x720
> [  862.254533]  ret_from_fork+0x2a6/0x470
> [  862.255852]  ret_from_fork_asm+0x1a/0x30
> [  862.256037]  </TASK>
> 
> Remove the blk_should_fake_timeout() check from dm, as dm has no
> native timeout handling and should not attempt to fake timeouts.
> 
> Signed-off-by: Ding Hui <dinghui@sangfor.com.cn>
> ---
>   drivers/md/dm-rq.c | 3 +--
>   1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
> index a6ca92049c10..5e0854669614 100644
> --- a/drivers/md/dm-rq.c
> +++ b/drivers/md/dm-rq.c
> @@ -278,8 +278,7 @@ static void dm_complete_request(struct request *rq, blk_status_t error)
>   	struct dm_rq_target_io *tio = tio_from_request(rq);
>   
>   	tio->error = error;
> -	if (likely(!blk_should_fake_timeout(rq->q)))
> -		blk_mq_complete_request(rq);
> +	blk_mq_complete_request(rq);
>   }
>   
>   /*

-- 
Thanks,
- Ding Hui

Re: [PATCH] dm: remove fake timeout to avoid leak request

Posted by Mikulas Patocka 3 weeks, 4 days ago


On Mon, 12 Jan 2026, Ding Hui wrote:

> Hi,
> 
> Sorry for this ping mail as there is no feedback from maintainers, just in
> case the patch was forgotten somewhere.
> 
> And it has already received a Reviewed-by from Christoph last month.

Hi

I committed the patch - it is in the linux-dm git, branch for-next.

Mikulas

> On 2025/12/20 20:03, Ding Hui wrote:
> > Since commit 15f73f5b3e59 ("blk-mq: move failure injection out of
> > blk_mq_complete_request"), drivers are responsible for calling
> > blk_should_fake_timeout() at appropriate code paths and opportunities.
> > 
> > However, the dm driver does not implement its own timeout handler and
> > relies on the timeout handling of its slave devices.
> > 
> > If an io-timeout-fail error is injected to a dm device, the request
> > will be leaked and never completed, causing tasks to hang indefinitely.
> > 
> > Reproduce:
> > 1. prepare dm which has iscsi slave device
> > 2. inject io-timeout-fail to dm
> >     echo 1 >/sys/class/block/dm-0/io-timeout-fail
> >     echo 100 >/sys/kernel/debug/fail_io_timeout/probability
> >     echo 10 >/sys/kernel/debug/fail_io_timeout/times
> > 3. read/write dm
> > 4. iscsiadm -m node -u
> > 
> > Result: hang task like below
> > [  862.243768] INFO: task kworker/u514:2:151 blocked for more than 122
> > seconds.
> > [  862.244133]       Tainted: G            E       6.19.0-rc1+ #51
> > [  862.244337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> > this message.
> > [  862.244718] task:kworker/u514:2  state:D stack:0     pid:151   tgid:151
> > ppid:2      task_flags:0x4288060 flags:0x00080000
> > [  862.245024] Workqueue: iscsi_ctrl_3:1 __iscsi_unbind_session
> > [scsi_transport_iscsi]
> > [  862.245264] Call Trace:
> > [  862.245587]  <TASK>
> > [  862.245814]  __schedule+0x810/0x15c0
> > [  862.246557]  schedule+0x69/0x180
> > [  862.246760]  blk_mq_freeze_queue_wait+0xde/0x120
> > [  862.247688]  elevator_change+0x16d/0x460
> > [  862.247893]  elevator_set_none+0x87/0xf0
> > [  862.248798]  blk_unregister_queue+0x12e/0x2a0
> > [  862.248995]  __del_gendisk+0x231/0x7e0
> > [  862.250143]  del_gendisk+0x12f/0x1d0
> > [  862.250339]  sd_remove+0x85/0x130 [sd_mod]
> > [  862.250650]  device_release_driver_internal+0x36d/0x530
> > [  862.250849]  bus_remove_device+0x1dd/0x3f0
> > [  862.251042]  device_del+0x38a/0x930
> > [  862.252095]  __scsi_remove_device+0x293/0x360
> > [  862.252291]  scsi_remove_target+0x486/0x760
> > [  862.252654]  __iscsi_unbind_session+0x18a/0x3e0 [scsi_transport_iscsi]
> > [  862.252886]  process_one_work+0x633/0xe50
> > [  862.253101]  worker_thread+0x6df/0xf10
> > [  862.253647]  kthread+0x36d/0x720
> > [  862.254533]  ret_from_fork+0x2a6/0x470
> > [  862.255852]  ret_from_fork_asm+0x1a/0x30
> > [  862.256037]  </TASK>
> > 
> > Remove the blk_should_fake_timeout() check from dm, as dm has no
> > native timeout handling and should not attempt to fake timeouts.
> > 
> > Signed-off-by: Ding Hui <dinghui@sangfor.com.cn>
> > ---
> >   drivers/md/dm-rq.c | 3 +--
> >   1 file changed, 1 insertion(+), 2 deletions(-)
> > 
> > diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
> > index a6ca92049c10..5e0854669614 100644
> > --- a/drivers/md/dm-rq.c
> > +++ b/drivers/md/dm-rq.c
> > @@ -278,8 +278,7 @@ static void dm_complete_request(struct request *rq,
> > blk_status_t error)
> >   	struct dm_rq_target_io *tio = tio_from_request(rq);
> >     	tio->error = error;
> > -	if (likely(!blk_should_fake_timeout(rq->q)))
> > -		blk_mq_complete_request(rq);
> > +	blk_mq_complete_request(rq);
> >   }
> >     /*
> 
> -- 
> Thanks,
> - Ding Hui
>

Re: [PATCH] dm: remove fake timeout to avoid leak request

Posted by Christoph Hellwig 1 month, 2 weeks ago

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>