From nobody Sat Sep 26 21:12:53 2026 Received: from mta0.migadu.com (out-3.mta0.migadu.com [91.218.175.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C09EA2E03F1 for ; Sun, 30 Aug 2026 01:20:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788052824; cv=none; b=qycK8K0OUA6lihnL5pjTvq6xftQK9kaLQpEZpfdS7O7cYKDGvHIr7EjPXJ8hfw2OBVcKiMv9FVycabz//u4MIZJJVrN7ATkCccfQaHDdNPWkXwezmZTVKZNC3aXYqz+IkWbVvLT2FttjkoSvBrwNom/TOU9+kki/gkWlWB9xxDU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788052824; c=relaxed/simple; bh=y0l2JK7SwE0igvljwsQtBdD/9LULsWF21doXn8fbuHo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VgMiXnIBz7lbUu2qOpD0acGiZmsQPKnmZ7073GG7IVp/NQ+WhfDEzsHTaEqRG1n1p/ooY6YUtcz+JJEMtbc72AUzR6pPvsdjVkMDL5VZSZs7DeUnDLV3lpI9iMr8KpkULTg3SisQ6L8v4Z03vFQYLcwC6MGbcM/ieGaesWKM/1A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=jd3Y3GLL; arc=none smtp.client-ip=91.218.175.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="jd3Y3GLL" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=y0l2JK7SwE0igvljwsQtBdD/9LULsWF21doXn8fbuHo=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788052820; v=1; x=1788657620; b=jd3Y3GLLidaY5WTX8/F9N/r2/Ez1P+H3LJnHfLVYLhYbnElDSA9/0anruqd8r5cbnu8Y8lP/ ifIJsa3nWrC34/v8Mneb9UGX/UA1GEFh0vOPkImmdo8b9NmOEjDrssVbxXTZRmgAE7Vyj0RISx0 +2Vs01wsYQNoE4AlVTvhZN5A= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 1edf6a99255e172b; Sun, 30 Aug 2026 01:20:20 +0000 X-Mizu-Trace-ID: 1edf6a99255e172b X-Migadu-Flow: FLOW_OUT From: Md Haris Iqbal To: Jens Axboe , linux-block@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Christoph Hellwig , Keith Busch , Jonathan Corbet , linux-doc@vger.kernel.org, Md Haris Iqbal Subject: [v2 for-next 1/3] block: reject unknown status tags in error injection rules Date: Sun, 30 Aug 2026 03:20:00 +0200 Message-ID: <20260830012002.80275-2-haris.iqbal@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260830012002.80275-1-haris.iqbal@linux.dev> References: <20260830012002.80275-1-haris.iqbal@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" tag_to_blk_status() returns BLK_STS_OK both for the "OK" tag and for a tag it does not recognise, so a caller cannot tell the two apart. match_status() leaves *status at BLK_STS_OK for an unknown tag and relies on error_inject_add() rejecting BLK_STS_OK. That holds only while a rule without a status is meaningless. The delay option added next makes such a rule valid, and "OK" is what blk_error_injection_show() prints for one, so the two cases have to be told apart. Return the status through a pointer and report an unknown tag as -EINVAL. There is no spare blk_status_t to encode "not found" in: every value in the table has a tag that can be typed, and anything outside the table trips the WARN_ON_ONCE() in blk_status_to_str(), blk_status_to_tag() and blk_status_to_errno(). For a single status=3D this does not change behaviour: an unknown tag still fails the write with -EINVAL. A repeated status=3D where an invalid tag comes first is now rejected instead of being overridden by the later one. Cc: Christoph Hellwig Signed-off-by: Md Haris Iqbal --- block/blk-core.c | 14 ++++++-------- block/blk.h | 2 +- block/error-injection.c | 7 ++++--- 3 files changed, 11 insertions(+), 12 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 196bccf27f58..29c86addb8f2 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -225,21 +225,19 @@ const char *blk_status_to_tag(blk_status_t status) return blk_errors[idx].tag; } =20 -blk_status_t tag_to_blk_status(const char *tag) +int tag_to_blk_status(const char *tag, blk_status_t *status) { int i; =20 for (i =3D 0; i < ARRAY_SIZE(blk_errors); i++) { if (blk_errors[i].tag && - !strcmp(blk_errors[i].tag, tag)) - return (__force blk_status_t)i; + !strcmp(blk_errors[i].tag, tag)) { + *status =3D (__force blk_status_t)i; + return 0; + } } =20 - /* - * Return BLK_STS_OK for mismatches as this function is intended to - * parse error status values. - */ - return BLK_STS_OK; + return -EINVAL; } =20 /** diff --git a/block/blk.h b/block/blk.h index 50abfd932886..8a8ab961528d 100644 --- a/block/blk.h +++ b/block/blk.h @@ -52,7 +52,7 @@ void blk_free_flush_queue(struct blk_flush_queue *q); =20 const char *blk_status_to_str(blk_status_t status); const char *blk_status_to_tag(blk_status_t status); -blk_status_t tag_to_blk_status(const char *tag); +int tag_to_blk_status(const char *tag, blk_status_t *status); enum req_op str_to_blk_op(const char *op); =20 bool __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic); diff --git a/block/error-injection.c b/block/error-injection.c index e14bc4b723ef..41ee8f788bb5 100644 --- a/block/error-injection.c +++ b/block/error-injection.c @@ -171,15 +171,16 @@ static int match_op(substring_t *args, enum req_op *o= p) static int match_status(substring_t *args, blk_status_t *status) { const char *tag; + int ret; =20 tag =3D match_strdup(args); if (!tag) return -ENOMEM; - *status =3D tag_to_blk_status(tag); - if (!*status) + ret =3D tag_to_blk_status(tag, status); + if (ret) pr_warn("invalid status '%s'\n", tag); kfree(tag); - return 0; + return ret; } =20 static ssize_t blk_error_injection_parse_options(struct gendisk *disk, --=20 2.53.0 From nobody Sat Sep 26 21:12:53 2026 Received: from mta1.migadu.com (out-228.mta1.migadu.com [95.215.58.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 910FB2D73B6 for ; Sun, 30 Aug 2026 01:20:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.228 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788052825; cv=none; b=GzxjVpIkACXEQWQ7YmykqvLzclRReMZj3RyFoeO/JwKFaPqbMxB7WNdtiMfkDVzMGIAGJ2IBFq1P5xZrvtrfbAmTLBWaeM9tvzWeyNPec0nWGbaLK8tb1SDFsdh3J8km4YfmRaPDgpVsh40Z/eIadtrlmS+Xf0/PaRwQxqzTp+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788052825; c=relaxed/simple; bh=IFzXOFd5TrUmv1oJ69PwzfP7yslWZ8DlDbTIq3MMXdg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tnGaqweMf65l37hlvAzsfYW6HsMw+glMiab8jWMKmV94LXKeoyzfTjmwxSTT9ZQeNEROa3J1QSOMVsSloGIVpJ/QoqfhNVI8XYCCmcIthcZ17q/ut0IPOZnzTQS5z0sDxH5DzdYW+QPmDVU1iVNr2fKRCHgPhXAl1LDoBAKu+gw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=s5HbhGgt; arc=none smtp.client-ip=95.215.58.228 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="s5HbhGgt" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=IFzXOFd5TrUmv1oJ69PwzfP7yslWZ8DlDbTIq3MMXdg=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788052821; v=1; x=1788657621; b=s5HbhGgtBWsr2grzryKEDqwSyrAWPu/WOzwtdylvlcwTZhf2uAs9FokVqbzqLVm3blYV4Fpf poUg9GH2eRmULbcIUHuJ2ysreZO+oYDPAlFeSopkSNlHTy4X+83qZRczu1LGeA9LVrONkQbgfoz ObQ0srXwy9GtGbFwcQKlkW2U= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id ac1466028375361f; Sun, 30 Aug 2026 01:20:21 +0000 X-Mizu-Trace-ID: ac1466028375361f X-Migadu-Flow: FLOW_OUT From: Md Haris Iqbal To: Jens Axboe , linux-block@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Christoph Hellwig , Keith Busch , Jonathan Corbet , linux-doc@vger.kernel.org, Md Haris Iqbal Subject: [v2 for-next 2/3] block: allow error injection rules to delay bios Date: Sun, 30 Aug 2026 03:20:01 +0200 Message-ID: <20260830012002.80275-3-haris.iqbal@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260830012002.80275-1-haris.iqbal@linux.dev> References: <20260830012002.80275-1-haris.iqbal@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Error injection can only fail a bio today. Add a delay_us option so that a rule can hold a bio back first, to model a slow device. If a matching rule has delay_us set, the bio is held for that long. It is then failed if the rule also has a status, or resubmitted below the injection hook so that the rules are not applied to it again. Submitting below the hook is not enough on its own. A bio is split below the hook, and bio_submit_split_bioset() resubmits the remainder through submit_bio_noacct_nocheck(), which is above it. bio_split() advances the original bio and returns a clone of the front piece, so the remainder is the same bio on a range the rule still covers and would be delayed once per split. Mark a delayed bio with BIO_ERROR_INJECTED instead and skip the hook for a bio that has it. __bio_clone() does not propagate the flag, so the front pieces and the clones a stacking driver aims at a lower device are still evaluated. The bio is submitted from a workqueue rather than from the timer, because submitting a bio can sleep. Bios with REQ_NOWAIT are never delayed, and values above 600 seconds are rejected. Cc: Christoph Hellwig Signed-off-by: Md Haris Iqbal --- block/blk-core.c | 13 +++- block/blk.h | 1 + block/error-injection.c | 158 ++++++++++++++++++++++++++++++++++---- block/error-injection.h | 1 + include/linux/blk_types.h | 1 + 5 files changed, 155 insertions(+), 19 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 29c86addb8f2..6ba21fd37b6c 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -757,11 +757,8 @@ static void __submit_bio_noacct_mq(struct bio *bio) current->bio_list =3D NULL; } =20 -void submit_bio_noacct_nocheck(struct bio *bio, bool split) +void __submit_bio_noacct_nocheck(struct bio *bio, bool split) { - if (unlikely(blk_error_inject(bio))) - return; - blk_cgroup_bio_start(bio); =20 if (!bio_flagged(bio, BIO_TRACE_COMPLETION)) { @@ -791,6 +788,14 @@ void submit_bio_noacct_nocheck(struct bio *bio, bool s= plit) } } =20 +void submit_bio_noacct_nocheck(struct bio *bio, bool split) +{ + if (unlikely(blk_error_inject(bio))) + return; + + __submit_bio_noacct_nocheck(bio, split); +} + static blk_status_t blk_validate_atomic_write_op_size(struct request_queue= *q, struct bio *bio) { diff --git a/block/blk.h b/block/blk.h index 8a8ab961528d..389c9a487067 100644 --- a/block/blk.h +++ b/block/blk.h @@ -61,6 +61,7 @@ bool __blk_freeze_queue_start(struct request_queue *q, struct task_struct *owner); int __bio_queue_enter(struct request_queue *q, struct bio *bio); void submit_bio_noacct_nocheck(struct bio *bio, bool split); +void __submit_bio_noacct_nocheck(struct bio *bio, bool split); int bio_submit_or_kill(struct bio *bio, unsigned int flags); =20 static inline bool blk_try_enter_queue(struct request_queue *q, bool pm) diff --git a/block/error-injection.c b/block/error-injection.c index 41ee8f788bb5..db8384c7c6f9 100644 --- a/block/error-injection.c +++ b/block/error-injection.c @@ -6,9 +6,17 @@ #include #include #include +#include #include "blk.h" #include "error-injection.h" =20 +/* + * Cap the delay so that a typo can't wedge a device for good. This is st= ill + * well beyond the default hung task timeout, which is one of the things a + * delay is useful for triggering. + */ +#define BLK_ERROR_INJECT_MAX_DELAY_US (600 * USEC_PER_SEC) + struct blk_error_inject { struct list_head entry; sector_t start; @@ -18,14 +26,102 @@ struct blk_error_inject { =20 /* only inject every 1 / chance times */ unsigned int chance; + + /* hold the bio for this long before submitting or failing it */ + unsigned int delay_us; }; =20 +/* + * A bio held by a delay rule. This is self-contained on purpose: it does= not + * point back at the rule, so rules can be removed while delayed bios are + * outstanding, and it does not point at the gendisk, so nothing has to be + * cleaned up when the disk goes away. A delayed bio holds no queue usage + * counter reference either, so one that outlives its disk is failed by the + * GD_DEAD check in __bio_queue_enter() once it is finally submitted. + */ +struct blk_error_inject_delay { + struct delayed_work dwork; + struct bio *bio; + blk_status_t status; +}; + +static struct workqueue_struct *blk_error_inject_wq; + DEFINE_STATIC_KEY_FALSE(blk_error_injection_enabled); =20 +static void blk_error_inject_delay_work(struct work_struct *work) +{ + struct blk_error_inject_delay *d =3D container_of(to_delayed_work(work), + struct blk_error_inject_delay, dwork); + struct bio *bio =3D d->bio; + blk_status_t status =3D d->status; + + kfree(d); + + if (status !=3D BLK_STS_OK) { + bio->bi_status =3D status; + bio_endio(bio); + } else { + /* + * Submit below the injection hook. Together with + * BIO_ERROR_INJECTED, which also covers the resubmission of + * the remainder of a split, this means a bio that was delayed + * once skips error injection entirely from here on, including + * any other rule that covers it. + */ + __submit_bio_noacct_nocheck(bio, false); + } +} + +/* + * Hand the bio to a workqueue that submits or fails it once the delay has + * expired. Both blk_mq_submit_bio() and ->submit_bio can sleep, so this = can't + * be completed from the timer itself. + * + * Returns false if the bio can't be delayed, in which case the caller han= dles + * it immediately instead. + */ +static bool blk_error_inject_delay(struct gendisk *disk, struct bio *bio, + blk_status_t status, unsigned int delay_us) +{ + struct blk_error_inject_delay *d; + + /* never block a bio that asked not to be blocked */ + if (bio->bi_opf & REQ_NOWAIT) + return false; + + d =3D kmalloc_obj(*d, GFP_NOIO); + if (!d) + return false; + + pr_info_ratelimited("%pg: delaying %s at sector %llu:%u by %uus\n", + disk->part0, blk_op_str(bio_op(bio)), + bio->bi_iter.bi_sector, bio_sectors(bio), delay_us); + + d->bio =3D bio; + d->status =3D status; + INIT_DELAYED_WORK(&d->dwork, blk_error_inject_delay_work); + + /* + * Mark the bio before queueing the work, which can complete it as soon + * as it is queued. Splitting happens below the injection hook, but + * bio_submit_split_bioset() resubmits the remainder through the hook + * again, and as bio_split() only advances the original bio that + * remainder still matches the same rule. Without this a bio would be + * delayed once per split. + */ + bio_set_flag(bio, BIO_ERROR_INJECTED); + queue_delayed_work(blk_error_inject_wq, &d->dwork, + usecs_to_jiffies(delay_us)); + return true; +} + bool __blk_error_inject(struct bio *bio) { struct gendisk *disk =3D bio->bi_bdev->bd_disk; struct blk_error_inject *inj; + blk_status_t status =3D BLK_STS_OK; + unsigned int delay_us =3D 0; =20 rcu_read_lock(); list_for_each_entry_rcu(inj, &disk->error_injection_list, entry) { @@ -45,29 +141,38 @@ bool __blk_error_inject(struct bio *bio) if (inj->chance > 1 && (get_random_u32() % inj->chance) !=3D 0) continue; =20 - pr_info_ratelimited("%pg: injecting %s error for %s at sector %llu:%u\n", - disk->part0, blk_status_to_str(inj->status), - blk_op_str(inj->op), bio->bi_iter.bi_sector, - bio_sectors(bio)); - bio->bi_status =3D inj->status; - rcu_read_unlock(); - bio_endio(bio); - return true; + status =3D inj->status; + delay_us =3D inj->delay_us; + break; } rcu_read_unlock(); - return false; + + if (delay_us && blk_error_inject_delay(disk, bio, status, delay_us)) + return true; + if (status =3D=3D BLK_STS_OK) + return false; + + pr_info_ratelimited("%pg: injecting %s error for %s at sector %llu:%u\n", + disk->part0, blk_status_to_str(status), + blk_op_str(bio_op(bio)), bio->bi_iter.bi_sector, + bio_sectors(bio)); + bio->bi_status =3D status; + bio_endio(bio); + return true; } =20 static int error_inject_add(struct gendisk *disk, enum req_op op, sector_t start, u64 nr_sectors, blk_status_t status, - unsigned int chance) + unsigned int chance, unsigned int delay_us) { struct blk_error_inject *inj; int error =3D -EINVAL; =20 if (op =3D=3D REQ_OP_LAST) return -EINVAL; - if (status =3D=3D BLK_STS_OK) + if (status =3D=3D BLK_STS_OK && !delay_us) + return -EINVAL; + if (delay_us > BLK_ERROR_INJECT_MAX_DELAY_US) return -EINVAL; =20 inj =3D kzalloc_obj(*inj); @@ -86,6 +191,7 @@ static int error_inject_add(struct gendisk *disk, enum r= eq_op op, inj->start =3D start; inj->status =3D status; inj->chance =3D chance; + inj->delay_us =3D delay_us; =20 pr_debug_ratelimited("%pg: adding %s injection for %s at sector %llu:%llu= \n", disk->part0, blk_status_to_str(status), @@ -139,6 +245,7 @@ enum options { Opt_nr_sectors =3D (1u << 18), Opt_status =3D (1u << 19), Opt_chance =3D (1u << 20), + Opt_delay_us =3D (1u << 21), =20 Opt_invalid, }; @@ -151,6 +258,7 @@ static const match_table_t opt_tokens =3D { { Opt_nr_sectors, "nr_sectors=3D%u" }, { Opt_status, "status=3D%s" }, { Opt_chance, "chance=3D%u" }, + { Opt_delay_us, "delay_us=3D%u" }, { Opt_invalid, NULL, }, }; =20 @@ -187,7 +295,7 @@ static ssize_t blk_error_injection_parse_options(struct= gendisk *disk, char *options) { enum { Unset, Add, Removeall } action =3D Unset; - unsigned int option_mask =3D 0, chance =3D 1; + unsigned int option_mask =3D 0, chance =3D 1, delay_us =3D 0; enum req_op op =3D REQ_OP_LAST; u64 start =3D 0, nr_sectors =3D 0; blk_status_t status =3D BLK_STS_OK; @@ -230,6 +338,9 @@ static ssize_t blk_error_injection_parse_options(struct= gendisk *disk, if (!error && chance =3D=3D 0) error =3D -EINVAL; break; + case Opt_delay_us: + error =3D match_uint(args, &delay_us); + break; default: pr_warn("unknown parameter or missing value '%s'\n", p); error =3D -EINVAL; @@ -241,7 +352,7 @@ static ssize_t blk_error_injection_parse_options(struct= gendisk *disk, switch (action) { case Add: return error_inject_add(disk, op, start, nr_sectors, status, - chance); + chance, delay_us); case Removeall: if (option_mask & ~Opt_removeall) return -EINVAL; @@ -277,10 +388,11 @@ static int blk_error_injection_show(struct seq_file *= s, void *private) =20 rcu_read_lock(); list_for_each_entry_rcu(inj, &disk->error_injection_list, entry) { - seq_printf(s, "%llu:%llu op=3D%s,status=3D%s,chance=3D%u", + seq_printf(s, "%llu:%llu op=3D%s,status=3D%s,chance=3D%u,delay_us=3D%u", inj->start, inj->end, blk_op_str(inj->op), - blk_status_to_tag(inj->status), inj->chance); + blk_status_to_tag(inj->status), inj->chance, + inj->delay_us); seq_putc(s, '\n'); } rcu_read_unlock(); @@ -315,3 +427,19 @@ void blk_error_injection_exit(struct gendisk *disk) { error_inject_removeall(disk); } + +static int __init blk_error_injection_init_wq(void) +{ + /* + * WQ_MEM_RECLAIM so that a delayed bio on the reclaim path can still + * find a worker under memory pressure. Note that this only guarantees + * a worker exists, not that it is free: submitting a bio can block on + * a queue freeze or on tag allocation, so a delayed bio can still be + * held up behind another one. + */ + blk_error_inject_wq =3D alloc_workqueue("blk_error_inject", WQ_MEM_RECLAI= M | WQ_UNBOUND, 0); + if (!blk_error_inject_wq) + panic("Failed to create blk_error_inject wq\n"); + return 0; +} +subsys_initcall(blk_error_injection_init_wq); diff --git a/block/error-injection.h b/block/error-injection.h index 9821d773abab..8b3809d85e83 100644 --- a/block/error-injection.h +++ b/block/error-injection.h @@ -13,6 +13,7 @@ static inline bool blk_error_inject(struct bio *bio) { if (IS_ENABLED(CONFIG_BLK_ERROR_INJECTION) && static_branch_unlikely(&blk_error_injection_enabled) && + !bio_flagged(bio, BIO_ERROR_INJECTED) && test_bit(GD_ERROR_INJECT, &bio->bi_bdev->bd_disk->state)) return __blk_error_inject(bio); return false; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 98e21b4cbf32..50e28dc0f1f9 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -323,6 +323,7 @@ enum { BIO_ZONE_WRITE_PLUGGING, /* bio handled through zone write plugging */ BIO_EMULATES_ZONE_APPEND, /* bio emulates a zone append operation */ BIO_COMPLETE_IN_TASK, /* complete bi_end_io() in task context */ + BIO_ERROR_INJECTED, /* error injection rules already applied */ BIO_FLAG_LAST }; =20 --=20 2.53.0 From nobody Sat Sep 26 21:12:53 2026 Received: from mta0.migadu.com (out-9.mta0.migadu.com [91.218.175.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A611E27603A for ; Sun, 30 Aug 2026 01:20:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788052830; cv=none; b=p969X+Wvx6PrASL7jvUKuvftH1ahCTXZSEeC4pyZcxWG0RKLhaFQ22ifJu2hnMPdAR/f5oAN5UQ0p0R2VsBEqlzqH4PSIhWTW80f08A341cnOEk4EBqN0j0j3fgrd6+d8GHOvbKdDTrcL/6GivErQKdpI9DCdunKTWhTsVRjWvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788052830; c=relaxed/simple; bh=cXHnugjrYFIs1UwN0nVV8md0Hm/sTFj0FoTfSt8CKkQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FDfsgehHNdMaAry9G0p20CmePKN+lF7mhUhCVOjj6/7RtZdFiMYnfIH5IGt/WYTDCHOUChcixxSb2AQDMZuBgPXAakOZzEjhC/iyp1tL6iMkeLXmlfqqAucYG+ZmRc+BzRlBfa+gXtETjh/3y7pX2UhPHfkmUm+NFeS2IDd6H50= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=uj+g9NvU; arc=none smtp.client-ip=91.218.175.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="uj+g9NvU" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=cXHnugjrYFIs1UwN0nVV8md0Hm/sTFj0FoTfSt8CKkQ=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1788052825; v=1; x=1788657625; b=uj+g9NvUax7kkrmIpYg5SAngii4rI5NgZzTsmimdl7DRGWtZKrBbA4vzOltKw3RmjP45oemd c4Hn8XvTb+1yIkyINTqftazhZ8oPSr+ZUCLw2B2zLD+nJPvNohSU6CBT06lVB0SRDhdzxYx//6L nQJ2sM9NxgDidSbFkEHYpY58= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 0ca15c6cd765ee62; Sun, 30 Aug 2026 01:20:25 +0000 X-Mizu-Trace-ID: 0ca15c6cd765ee62 X-Migadu-Flow: FLOW_OUT From: Md Haris Iqbal To: Jens Axboe , linux-block@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Christoph Hellwig , Keith Busch , Jonathan Corbet , linux-doc@vger.kernel.org, Md Haris Iqbal Subject: [v2 for-next 3/3] Documentation: block: document error injection delays Date: Sun, 30 Aug 2026 03:20:02 +0200 Message-ID: <20260830012002.80275-4-haris.iqbal@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260830012002.80275-1-haris.iqbal@linux.dev> References: <20260830012002.80275-1-haris.iqbal@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Document the delay_us option: the delay happens above the driver, a delayed bio is not run through the rules again, and holding a bio back reorders it against bios submitted later. Cc: Christoph Hellwig Signed-off-by: Md Haris Iqbal --- Documentation/block/error-injection.rst | 58 ++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 2 deletions(-) diff --git a/Documentation/block/error-injection.rst b/Documentation/block/= error-injection.rst index 81f31af82e65..a1f50d41211d 100644 --- a/Documentation/block/error-injection.rst +++ b/Documentation/block/error-injection.rst @@ -9,7 +9,8 @@ Overview =20 Configurable error injection allows injecting specific block layer status = codes for sector ranges of a block device. Errors can be injected unconditional= ly, or -with a given probability. +with a given probability. Instead of, or before, failing a bio it can als= o be +held back for a while to model a slow device. =20 To use configurable error injection, CONFIG_BLK_ERROR_INJECTION must be en= abled. =20 @@ -34,15 +35,60 @@ op=3D block layer operation this rule applies = to. This uses the XYZ for each REQ_OP_XYZ operation, e.g. READ, WRITE or DISCARD. Mandatory. status=3D Status to return. This uses XYZ for each BLK_STS_XYZ - code, e.g. IOERR or MEDIUM. Mandatory. + code, e.g. IOERR or MEDIUM. Mandatory unless delay_us + is given. start=3D First block layer sector the rule applies to. Optional, defaults to 0. nr_sectors=3D Number of sectors this rule applies. Optional, defaults to the remainder of the device. chance=3D Only return a failure with a likelihood of 1/chance. Optional, defaults to 1 (always). +delay_us=3D Hold the bio back for this many microseconds. Without + status the bio is then submitted to the device as + usual, with status it is failed once the delay has + expired. Optional, defaults to 0 (no delay). + Values above 600 seconds are rejected. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 +Delays +------ + +A delayed bio is held before it is submitted, so the device itself never s= ees a +slow I/O: the delay is not visible to the driver, to the I/O statistics, o= r to +anything else below submission such as writeback throttling. Throttling by +blk-throttle happens before a bio can be delayed, so it is not affected ei= ther. +What it does exercise is everything waiting above the block layer, for ins= tance +io_uring cancellation, hung task detection, and filesystem or userspace +timeouts. Because the low level driver is not involved, a delay does not = reach +the blk-mq timeout handler or SCSI error handling. + +Once a bio has been delayed no rule is evaluated for it a second time, not= when +its delay expires and it is submitted below the injection hook, and not wh= en +the block layer splits it and resubmits the remainder above the hook. A b= io +that matched a delay rule therefore never gets an error from another rule,= even +one covering the same sectors, and is held for the delay once rather than = once +per split. Put the delay and the status in a single rule to fail a bio af= ter +holding it back. + +A delayed bio is issued after bios submitted while it was held, which reor= ders +the I/O stream. On zoned devices this breaks sequential write ordering: z= one +write plugging happens below the injection hook, so the writes issued whil= e a +write is held reach the zone out of order and are failed as misaligned. O= nly +delay reads there. + +Bios that must not block are never delayed. A bio with REQ_NOWAIT set is +submitted, or failed with the rule's status, immediately. + +The delay is a lower bound for anything longer than a timer tick, and the = timer +wheel adds further slack as the delay grows. Values shorter than a tick a= re of +little use: they expire on the next tick, which is anywhere between now an= d one +tick away. + +Removing rules does not release bios that are already being delayed by the= m; +those run out on their own. A delayed bio whose disk is removed in the me= antime +is not submitted until its delay expires, by which point the queue no long= er +accepts I/O, so it fails with EIO. + Example ------- =20 @@ -54,6 +100,14 @@ Return BLK_STS_MEDIUM for every write to /dev/nvme0n1: =20 $ echo 'add,op=3DWRITE,start=3D0,status=3DMEDIUM' > /sys/kernel/debug/blo= ck/nvme0n1/error_injection =20 +Delay every read of /dev/nvme0n1 by 10 milliseconds, then issue it normall= y: + + $ echo 'add,op=3DREAD,delay_us=3D10000' > /sys/kernel/debug/block/nvme0n1= /error_injection + +Fail one in 100 writes with BLK_STS_TIMEOUT, but only after 30 seconds: + + $ echo 'add,op=3DWRITE,status=3DTIMEOUT,chance=3D100,delay_us=3D30000000'= > /sys/kernel/debug/block/nvme0n1/error_injection + Remove all rules for /dev/nvme0n1: =20 $ echo 'removeall' > /sys/kernel/debug/block/nvme0n1/error_injection --=20 2.53.0