From nobody Fri Oct 3 02:03:25 2025 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF71C2EB5BF; Mon, 8 Sep 2025 09:29:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323771; cv=none; b=IfseIv4Je2DWNc0qfqGZpB9UYeLFpcranyhgdbYIvX9YgWFvROSSk4dLpYwMYc0gsVFOSk3jYMT8VgztVQAh+Hy98/ePxoAj+TNtOHZ3agakWcd4JoihWVcH9aha9lOft8sy0KVtG0p4wuLQLndiifCwoAiimoM3KXGW10SG/0Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323771; c=relaxed/simple; bh=zcUoMdH7hBNz7MSsR7/dhT0RSHn4wSbUyGgnmqAeiNs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=f7ZbRV6nEpZ/8ZKI4X6r2cbt2bi2sbFjuNyul212MmR8qalVlnYrJCHEmDJAyHj62qgH6hVB4CzlNMdCsQ8/izjw5tfLfMj3XMFM0B5YFhQ4xdChwEnb1WR8TCpxU8W3efgaHlq5lhxrwMvlPTG0i7pe+le3igAu6lSEXuGcxhc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.93.142]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4cL1qp4pNVzKHN7F; Mon, 8 Sep 2025 17:29:26 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.128]) by mail.maildlp.com (Postfix) with ESMTP id BE88F1A0E98; Mon, 8 Sep 2025 17:29:26 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.104.67]) by APP4 (Coremail) with SMTP id gCh0CgAXYIzzob5oGjMyBw--.46699S5; Mon, 08 Sep 2025 17:29:26 +0800 (CST) From: Yu Kuai To: dlemoal@kernel.org, hare@suse.de, bvanassche@acm.org, ming.lei@redhat.com, tj@kernel.org, josef@toxicpanda.com, axboe@kernel.dk, yukuai3@huawei.com Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai1@huaweicloud.com, yi.zhang@huawei.com, yangerkun@huawei.com, johnny.chenyi@huawei.com Subject: [PATCH v4 for-6.18/block 1/5] blk-mq-sched: introduce high level elevator lock Date: Mon, 8 Sep 2025 17:20:03 +0800 Message-Id: <20250908092007.3796967-2-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250908092007.3796967-1-yukuai1@huaweicloud.com> References: <20250908092007.3796967-1-yukuai1@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: gCh0CgAXYIzzob5oGjMyBw--.46699S5 X-Coremail-Antispam: 1UD129KBjvJXoWxXw1UJw47Gr1fGr1kCr1kGrg_yoWrGF45pF 4rGa98Kw1kXF42gF97A3W3Xw43K34v9r9xZrWxZ3y7tFn7Jr4fW3W8GF1UZF4UZr4fGFsF qr18taykWay7G3DanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmY14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jr4l82xGYIkIc2 x26xkF7I0E14v26r4j6ryUM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJw A2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2kIc2 xKxwCY1x0262kKe7AKxVWUtVW8ZwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWU JVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67AF67 kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY 6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0x vEx4A2jsIE14v26r4j6F4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Gr1j6F4UJbIYCTnIWIev Ja73UjIFyTuYvjfU8XdbUUUUU X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ Content-Type: text/plain; charset="utf-8" From: Yu Kuai Currently, both mq-deadline and bfq have global spin lock that will be grabbed inside elevator methods like dispatch_request, insert_requests, and bio_merge. And the global lock is the main reason mq-deadline and bfq can't scale very well. While dispatching request, blk_mq_get_disatpch_budget() and blk_mq_get_driver_tag() must be called, and they are not ready to be called inside elevator methods, hence introduce a new method like dispatch_requests is not possible. Hence introduce a new high level elevator lock, currently it is protecting dispatch_request only. Following patches will convert mq-deadline and bfq to use this lock and finally support request batch dispatching by calling the method multiple time while holding the lock. Signed-off-by: Yu Kuai --- block/blk-mq-sched.c | 9 ++++++++- block/elevator.c | 1 + block/elevator.h | 31 +++++++++++++++++++++++++++++-- 3 files changed, 38 insertions(+), 3 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index d06bb137a743..6f2b0573a8ff 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -113,7 +113,14 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_hw= _ctx *hctx) if (budget_token < 0) break; =20 - rq =3D e->type->ops.dispatch_request(hctx); + if (blk_queue_sq_sched(q)) { + elevator_dispatch_lock(e); + rq =3D e->type->ops.dispatch_request(hctx); + elevator_dispatch_unlock(e); + } else { + rq =3D e->type->ops.dispatch_request(hctx); + } + if (!rq) { blk_mq_put_dispatch_budget(q, budget_token); /* diff --git a/block/elevator.c b/block/elevator.c index e2ebfbf107b3..e132a24f9ce3 100644 --- a/block/elevator.c +++ b/block/elevator.c @@ -146,6 +146,7 @@ struct elevator_queue *elevator_alloc(struct request_qu= eue *q, eq->type =3D e; kobject_init(&eq->kobj, &elv_ktype); mutex_init(&eq->sysfs_lock); + spin_lock_init(&eq->lock); hash_init(eq->hash); eq->et =3D et; =20 diff --git a/block/elevator.h b/block/elevator.h index c4d20155065e..4864add27136 100644 --- a/block/elevator.h +++ b/block/elevator.h @@ -119,13 +119,13 @@ struct request *elv_rqhash_find(struct request_queue = *q, sector_t offset); /* * each queue has an elevator_queue associated with it */ -struct elevator_queue -{ +struct elevator_queue { struct elevator_type *type; struct elevator_tags *et; void *elevator_data; struct kobject kobj; struct mutex sysfs_lock; + spinlock_t lock; unsigned long flags; DECLARE_HASHTABLE(hash, ELV_HASH_BITS); }; @@ -133,6 +133,33 @@ struct elevator_queue #define ELEVATOR_FLAG_REGISTERED 0 #define ELEVATOR_FLAG_DYING 1 #define ELEVATOR_FLAG_ENABLE_WBT_ON_EXIT 2 +#define ELEVATOR_FLAG_DISPATCH_IRQ 3 + +#define elevator_lock(e) spin_lock(&(e)->lock) +#define elevator_unlock(e) spin_unlock(&(e)->lock) +#define elevator_lock_irq(e) spin_lock_irq(&(e)->lock) +#define elevator_unlock_irq(e) spin_unlock_irq(&(e)->lock) +#define elevator_lock_irqsave(e, flags) \ + spin_lock_irqsave(&(e)->lock, flags) +#define elevator_unlock_irqrestore(e, flags) \ + spin_unlock_irqrestore(&(e)->lock, flags) +#define elevator_lock_assert_held(e) lockdep_assert_held(&(e)->lock) + +static inline void elevator_dispatch_lock(struct elevator_queue *eq) +{ + if (test_bit(ELEVATOR_FLAG_DISPATCH_IRQ, &eq->flags)) + elevator_lock_irq(eq); + else + elevator_lock(eq); +} + +static inline void elevator_dispatch_unlock(struct elevator_queue *eq) +{ + if (test_bit(ELEVATOR_FLAG_DISPATCH_IRQ, &eq->flags)) + elevator_unlock_irq(eq); + else + elevator_unlock(eq); +} =20 /* * block elevator interface --=20 2.39.2 From nobody Fri Oct 3 02:03:25 2025 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 831E32F068E; Mon, 8 Sep 2025 09:29:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323771; cv=none; b=aYaSa+saQM3tnol1J4cshzfzhaD6ProxDPLCVLb5W0JZQNUqtNz2Owxk+9lIGdz4cQ8SwkvaJ00HsSsCwgxangd58hPaApzXJL+3IhSt+9AmgR+Yd+nZMNcL6mxmk8rzN4VXuv4rNqvWPcZGZwCcQhhTD6iWZw1txo/v6gb2+0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323771; c=relaxed/simple; bh=Syo51umTQhqv1HC0LouVUaYiyrIvpOJaTmFlhLQhDNo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AxiQ8LCv+5OiWtAniDnZ+r3khqIfGQN7oxtYMbb5dzs5aYdF/PyZiG5gZXDGq3ijy7QkVf/DXbfqCN7tYS5EU5Z31Rfd3K9Q6wv6k8bEhRgyl1JfJaLyGKdxpihygMEKDYUAKAcY52QtTI+R+S2vvF0+XFjWuiltQeZXEjofDoc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTPS id 4cL1qr5zwWzYQvYf; Mon, 8 Sep 2025 17:29:28 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.128]) by mail.maildlp.com (Postfix) with ESMTP id 565061A174E; Mon, 8 Sep 2025 17:29:27 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.104.67]) by APP4 (Coremail) with SMTP id gCh0CgAXYIzzob5oGjMyBw--.46699S6; Mon, 08 Sep 2025 17:29:27 +0800 (CST) From: Yu Kuai To: dlemoal@kernel.org, hare@suse.de, bvanassche@acm.org, ming.lei@redhat.com, tj@kernel.org, josef@toxicpanda.com, axboe@kernel.dk, yukuai3@huawei.com Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai1@huaweicloud.com, yi.zhang@huawei.com, yangerkun@huawei.com, johnny.chenyi@huawei.com Subject: [PATCH v4 for-6.18/block 2/5] mq-deadline: switch to use elevator lock Date: Mon, 8 Sep 2025 17:20:04 +0800 Message-Id: <20250908092007.3796967-3-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250908092007.3796967-1-yukuai1@huaweicloud.com> References: <20250908092007.3796967-1-yukuai1@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: gCh0CgAXYIzzob5oGjMyBw--.46699S6 X-Coremail-Antispam: 1UD129KBjvJXoW3XryrGFyfKFWrZFW8ZF18Zrb_yoW3ur48pF W5Ka90yr4rXFsrZF1DJa9rZr4agw4I9342qr93Kw4fKFn7Jr9rW3WjkF10vrs3Jr9rCFsI gF4qga98JF17Jw7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmY14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jryl82xGYIkIc2 x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJw A2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2kIc2 xKxwCY1x0262kKe7AKxVWUtVW8ZwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWU JVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67AF67 kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY 6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0x vEx4A2jsIE14v26r4j6F4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Gr1j6F4UJbIYCTnIWIev Ja73UjIFyTuYvjfUOdgAUUUUU X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ Content-Type: text/plain; charset="utf-8" From: Yu Kuai Replace the internal spinlock 'dd->lock' with the new spinlock in elevator_queue, things will keep working the same way as before. Signed-off-by: Yu Kuai --- block/mq-deadline.c | 60 ++++++++++++++++++++------------------------- 1 file changed, 27 insertions(+), 33 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 2e689b2c4021..2f394571a6ba 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -101,7 +101,7 @@ struct deadline_data { u32 async_depth; int prio_aging_expire; =20 - spinlock_t lock; + struct elevator_queue *e; }; =20 /* Maps an I/O priority class to a deadline scheduler priority. */ @@ -213,7 +213,7 @@ static void dd_merged_requests(struct request_queue *q,= struct request *req, const u8 ioprio_class =3D dd_rq_ioclass(next); const enum dd_prio prio =3D ioprio_class_to_prio[ioprio_class]; =20 - lockdep_assert_held(&dd->lock); + elevator_lock_assert_held(dd->e); =20 dd->per_prio[prio].stats.merged++; =20 @@ -253,7 +253,7 @@ static u32 dd_queued(struct deadline_data *dd, enum dd_= prio prio) { const struct io_stats_per_prio *stats =3D &dd->per_prio[prio].stats; =20 - lockdep_assert_held(&dd->lock); + elevator_lock_assert_held(dd->e); =20 return stats->inserted - atomic_read(&stats->completed); } @@ -323,7 +323,7 @@ static struct request *__dd_dispatch_request(struct dea= dline_data *dd, enum dd_prio prio; u8 ioprio_class; =20 - lockdep_assert_held(&dd->lock); + elevator_lock_assert_held(dd->e); =20 if (!list_empty(&per_prio->dispatch)) { rq =3D list_first_entry(&per_prio->dispatch, struct request, @@ -434,7 +434,7 @@ static struct request *dd_dispatch_prio_aged_requests(s= truct deadline_data *dd, enum dd_prio prio; int prio_cnt; =20 - lockdep_assert_held(&dd->lock); + elevator_lock_assert_held(dd->e); =20 prio_cnt =3D !!dd_queued(dd, DD_RT_PRIO) + !!dd_queued(dd, DD_BE_PRIO) + !!dd_queued(dd, DD_IDLE_PRIO); @@ -466,10 +466,9 @@ static struct request *dd_dispatch_request(struct blk_= mq_hw_ctx *hctx) struct request *rq; enum dd_prio prio; =20 - spin_lock(&dd->lock); rq =3D dd_dispatch_prio_aged_requests(dd, now); if (rq) - goto unlock; + return rq; =20 /* * Next, dispatch requests in priority order. Ignore lower priority @@ -481,9 +480,6 @@ static struct request *dd_dispatch_request(struct blk_m= q_hw_ctx *hctx) break; } =20 -unlock: - spin_unlock(&dd->lock); - return rq; } =20 @@ -528,9 +524,9 @@ static void dd_exit_sched(struct elevator_queue *e) WARN_ON_ONCE(!list_empty(&per_prio->fifo_list[DD_READ])); WARN_ON_ONCE(!list_empty(&per_prio->fifo_list[DD_WRITE])); =20 - spin_lock(&dd->lock); + elevator_lock(e); queued =3D dd_queued(dd, prio); - spin_unlock(&dd->lock); + elevator_unlock(e); =20 WARN_ONCE(queued !=3D 0, "statistics for priority %d: i %u m %u d %u c %u\n", @@ -571,7 +567,7 @@ static int dd_init_sched(struct request_queue *q, struc= t elevator_queue *eq) dd->last_dir =3D DD_WRITE; dd->fifo_batch =3D fifo_batch; dd->prio_aging_expire =3D prio_aging_expire; - spin_lock_init(&dd->lock); + dd->e =3D eq; =20 /* We dispatch from request queue wide instead of hw queue */ blk_queue_flag_set(QUEUE_FLAG_SQ_SCHED, q); @@ -620,13 +616,12 @@ static int dd_request_merge(struct request_queue *q, = struct request **rq, static bool dd_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { - struct deadline_data *dd =3D q->elevator->elevator_data; struct request *free =3D NULL; bool ret; =20 - spin_lock(&dd->lock); + elevator_lock(q->elevator); ret =3D blk_mq_sched_try_merge(q, bio, nr_segs, &free); - spin_unlock(&dd->lock); + elevator_unlock(q->elevator); =20 if (free) blk_mq_free_request(free); @@ -648,7 +643,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hct= x, struct request *rq, struct dd_per_prio *per_prio; enum dd_prio prio; =20 - lockdep_assert_held(&dd->lock); + elevator_lock_assert_held(dd->e); =20 prio =3D ioprio_class_to_prio[ioprio_class]; per_prio =3D &dd->per_prio[prio]; @@ -689,10 +684,9 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *h= ctx, blk_insert_t flags) { struct request_queue *q =3D hctx->queue; - struct deadline_data *dd =3D q->elevator->elevator_data; LIST_HEAD(free); =20 - spin_lock(&dd->lock); + elevator_lock(q->elevator); while (!list_empty(list)) { struct request *rq; =20 @@ -700,7 +694,7 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hc= tx, list_del_init(&rq->queuelist); dd_insert_request(hctx, rq, flags, &free); } - spin_unlock(&dd->lock); + elevator_unlock(q->elevator); =20 blk_mq_free_requests(&free); } @@ -816,13 +810,13 @@ static const struct elv_fs_entry deadline_attrs[] =3D= { #define DEADLINE_DEBUGFS_DDIR_ATTRS(prio, data_dir, name) \ static void *deadline_##name##_fifo_start(struct seq_file *m, \ loff_t *pos) \ - __acquires(&dd->lock) \ + __acquires(&dd->e->lock) \ { \ struct request_queue *q =3D m->private; \ struct deadline_data *dd =3D q->elevator->elevator_data; \ struct dd_per_prio *per_prio =3D &dd->per_prio[prio]; \ \ - spin_lock(&dd->lock); \ + elevator_lock(dd->e); \ return seq_list_start(&per_prio->fifo_list[data_dir], *pos); \ } \ \ @@ -837,12 +831,12 @@ static void *deadline_##name##_fifo_next(struct seq_f= ile *m, void *v, \ } \ \ static void deadline_##name##_fifo_stop(struct seq_file *m, void *v) \ - __releases(&dd->lock) \ + __releases(&d->e->lock) \ { \ struct request_queue *q =3D m->private; \ struct deadline_data *dd =3D q->elevator->elevator_data; \ \ - spin_unlock(&dd->lock); \ + elevator_unlock(dd->e); \ } \ \ static const struct seq_operations deadline_##name##_fifo_seq_ops =3D { \ @@ -908,11 +902,11 @@ static int dd_queued_show(void *data, struct seq_file= *m) struct deadline_data *dd =3D q->elevator->elevator_data; u32 rt, be, idle; =20 - spin_lock(&dd->lock); + elevator_lock(dd->e); rt =3D dd_queued(dd, DD_RT_PRIO); be =3D dd_queued(dd, DD_BE_PRIO); idle =3D dd_queued(dd, DD_IDLE_PRIO); - spin_unlock(&dd->lock); + elevator_unlock(dd->e); =20 seq_printf(m, "%u %u %u\n", rt, be, idle); =20 @@ -924,7 +918,7 @@ static u32 dd_owned_by_driver(struct deadline_data *dd,= enum dd_prio prio) { const struct io_stats_per_prio *stats =3D &dd->per_prio[prio].stats; =20 - lockdep_assert_held(&dd->lock); + elevator_lock_assert_held(dd->e); =20 return stats->dispatched + stats->merged - atomic_read(&stats->completed); @@ -936,11 +930,11 @@ static int dd_owned_by_driver_show(void *data, struct= seq_file *m) struct deadline_data *dd =3D q->elevator->elevator_data; u32 rt, be, idle; =20 - spin_lock(&dd->lock); + elevator_lock(dd->e); rt =3D dd_owned_by_driver(dd, DD_RT_PRIO); be =3D dd_owned_by_driver(dd, DD_BE_PRIO); idle =3D dd_owned_by_driver(dd, DD_IDLE_PRIO); - spin_unlock(&dd->lock); + elevator_unlock(dd->e); =20 seq_printf(m, "%u %u %u\n", rt, be, idle); =20 @@ -950,13 +944,13 @@ static int dd_owned_by_driver_show(void *data, struct= seq_file *m) #define DEADLINE_DISPATCH_ATTR(prio) \ static void *deadline_dispatch##prio##_start(struct seq_file *m, \ loff_t *pos) \ - __acquires(&dd->lock) \ + __acquires(&dd->e->lock) \ { \ struct request_queue *q =3D m->private; \ struct deadline_data *dd =3D q->elevator->elevator_data; \ struct dd_per_prio *per_prio =3D &dd->per_prio[prio]; \ \ - spin_lock(&dd->lock); \ + elevator_lock(dd->e); \ return seq_list_start(&per_prio->dispatch, *pos); \ } \ \ @@ -971,12 +965,12 @@ static void *deadline_dispatch##prio##_next(struct se= q_file *m, \ } \ \ static void deadline_dispatch##prio##_stop(struct seq_file *m, void *v) \ - __releases(&dd->lock) \ + __releases(&dd->e->lock) \ { \ struct request_queue *q =3D m->private; \ struct deadline_data *dd =3D q->elevator->elevator_data; \ \ - spin_unlock(&dd->lock); \ + elevator_unlock(dd->e); \ } \ \ static const struct seq_operations deadline_dispatch##prio##_seq_ops =3D {= \ --=20 2.39.2 From nobody Fri Oct 3 02:03:25 2025 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 746BA2F0C6E; Mon, 8 Sep 2025 09:29:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323773; cv=none; b=t2OfyBFoJp7+6758h0tswAJjKnoezLVqTSueI+jNUU2O4dHak5iBFVOABFqh6adZUxIFGPkyyluJZbYUvYO94LEVPuwi3pluSDTSePiS1aDnC71CtF9BAwKm/eo5cxvKKrHFG0czIHj9oMYC63eTcH4JF1vpNGeFNhYNnBmyHDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323773; c=relaxed/simple; bh=5nT5qTgPqOjgIx3R+4wrwCpD1aE4UDBiV1OP44fCvx8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=u2VQcnn7oiHANBICFEIutGmkVoejoBBzVCH6lmm69E7GeZP6adGeWjqyw2lzVZi0q8LnpnhbJcEB7uKtKSaZGAPX8yQ+Xrdi7QAK3SH08CxqjdbggNdcWO8IshDOpMrfTuBhtx50PM+gLR6BrZ9Z7OvFRR7cPWtHnhTbm7/PkdU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.93.142]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4cL1qq5y8wzKHN0s; Mon, 8 Sep 2025 17:29:27 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.128]) by mail.maildlp.com (Postfix) with ESMTP id E5F771A0E98; Mon, 8 Sep 2025 17:29:27 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.104.67]) by APP4 (Coremail) with SMTP id gCh0CgAXYIzzob5oGjMyBw--.46699S7; Mon, 08 Sep 2025 17:29:27 +0800 (CST) From: Yu Kuai To: dlemoal@kernel.org, hare@suse.de, bvanassche@acm.org, ming.lei@redhat.com, tj@kernel.org, josef@toxicpanda.com, axboe@kernel.dk, yukuai3@huawei.com Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai1@huaweicloud.com, yi.zhang@huawei.com, yangerkun@huawei.com, johnny.chenyi@huawei.com Subject: [PATCH v4 for-6.18/block 3/5] block, bfq: switch to use elevator lock Date: Mon, 8 Sep 2025 17:20:05 +0800 Message-Id: <20250908092007.3796967-4-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250908092007.3796967-1-yukuai1@huaweicloud.com> References: <20250908092007.3796967-1-yukuai1@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: gCh0CgAXYIzzob5oGjMyBw--.46699S7 X-Coremail-Antispam: 1UD129KBjvJXoW3uF18CFWrtF45Zw17Jr13Arb_yoWDtry7pa 1fKan0yw48Xr4Fvr4jkw4qqw47Kwsag3sFgryxX3yftFWxJrnxX3ZYyFyFvF4SqFn7CFs0 gr1jgayvyF1UJaUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmF14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_JrWl82xGYIkIc2 x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJw A2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2kIc2 xKxwCY1x0262kKe7AKxVWUtVW8ZwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWU JVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67AF67 kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY 6xIIjxv20xvEc7CjxVAFwI0_Cr0_Gr1UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42 IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4UJVWxJrUvcSsGvfC2 KfnxnUUI43ZEXa7VUbpwZ7UUUUU== X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ Content-Type: text/plain; charset="utf-8" From: Yu Kuai Replace the internal spinlock 'bfqd->lock' with the new spinlock in elevator_queue, things will keep working the same way as before. Signed-off-by: Yu Kuai --- block/bfq-cgroup.c | 6 ++--- block/bfq-iosched.c | 54 +++++++++++++++++++++------------------------ block/bfq-iosched.h | 2 -- 3 files changed, 28 insertions(+), 34 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index 9fb9f3533150..a23622cc2be2 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -746,7 +746,7 @@ static void bfq_sync_bfqq_move(struct bfq_data *bfqd, * @bic: the bic to move. * @bfqg: the group to move to. * - * Move bic to blkcg, assuming that bfqd->lock is held; which makes + * Move bic to blkcg, assuming that elevator lock is held; which makes * sure that the reference to cgroup is valid across the call (see * comments in bfq_bic_update_cgroup on this issue) */ @@ -878,7 +878,7 @@ static void bfq_pd_offline(struct blkg_policy_data *pd) unsigned long flags; int i; =20 - spin_lock_irqsave(&bfqd->lock, flags); + elevator_lock_irqsave(bfqd->queue->elevator, flags); =20 if (!entity) /* root group */ goto put_async_queues; @@ -923,7 +923,7 @@ static void bfq_pd_offline(struct blkg_policy_data *pd) put_async_queues: bfq_put_async_queues(bfqd, bfqg); =20 - spin_unlock_irqrestore(&bfqd->lock, flags); + elevator_unlock_irqrestore(bfqd->queue->elevator, flags); /* * @blkg is going offline and will be ignored by * blkg_[rw]stat_recursive_sum(). Transfer stats to the parent so diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 4a8d3d96bfe4..729a1c2ada6d 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -466,7 +466,7 @@ static struct bfq_io_cq *bfq_bic_lookup(struct request_= queue *q) */ void bfq_schedule_dispatch(struct bfq_data *bfqd) { - lockdep_assert_held(&bfqd->lock); + elevator_lock_assert_held(bfqd->queue->elevator); =20 if (bfqd->queued !=3D 0) { bfq_log(bfqd, "schedule dispatch"); @@ -591,7 +591,7 @@ static bool bfqq_request_over_limit(struct bfq_data *bf= qd, int level; =20 retry: - spin_lock_irq(&bfqd->lock); + elevator_lock_irq(bfqd->queue->elevator); bfqq =3D bic_to_bfqq(bic, op_is_sync(opf), act_idx); if (!bfqq) goto out; @@ -603,7 +603,7 @@ static bool bfqq_request_over_limit(struct bfq_data *bf= qd, /* +1 for bfqq entity, root cgroup not included */ depth =3D bfqg_to_blkg(bfqq_group(bfqq))->blkcg->css.cgroup->level + 1; if (depth > alloc_depth) { - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(bfqd->queue->elevator); if (entities !=3D inline_entities) kfree(entities); entities =3D kmalloc_array(depth, sizeof(*entities), GFP_NOIO); @@ -661,7 +661,7 @@ static bool bfqq_request_over_limit(struct bfq_data *bf= qd, } } out: - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(bfqd->queue->elevator); if (entities !=3D inline_entities) kfree(entities); return ret; @@ -2215,7 +2215,7 @@ static void bfq_add_request(struct request *rq) bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq)); bfqq->queued[rq_is_sync(rq)]++; /* - * Updating of 'bfqd->queued' is protected by 'bfqd->lock', however, it + * Updating of 'bfqd->queued' is protected by elevator lock, however, it * may be read without holding the lock in bfq_has_work(). */ WRITE_ONCE(bfqd->queued, bfqd->queued + 1); @@ -2395,7 +2395,7 @@ static void bfq_remove_request(struct request_queue *= q, list_del_init(&rq->queuelist); bfqq->queued[sync]--; /* - * Updating of 'bfqd->queued' is protected by 'bfqd->lock', however, it + * Updating of 'bfqd->queued' is protected by elevator lock, however, it * may be read without holding the lock in bfq_has_work(). */ WRITE_ONCE(bfqd->queued, bfqd->queued - 1); @@ -2452,7 +2452,7 @@ static bool bfq_bio_merge(struct request_queue *q, st= ruct bio *bio, struct request *free =3D NULL; bool ret; =20 - spin_lock_irq(&bfqd->lock); + elevator_lock_irq(q->elevator); =20 if (bic) { /* @@ -2470,7 +2470,7 @@ static bool bfq_bio_merge(struct request_queue *q, st= ruct bio *bio, =20 ret =3D blk_mq_sched_try_merge(q, bio, nr_segs, &free); =20 - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(q->elevator); if (free) blk_mq_free_request(free); =20 @@ -2645,7 +2645,7 @@ static void bfq_end_wr(struct bfq_data *bfqd) struct bfq_queue *bfqq; int i; =20 - spin_lock_irq(&bfqd->lock); + elevator_lock_irq(bfqd->queue->elevator); =20 for (i =3D 0; i < bfqd->num_actuators; i++) { list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list) @@ -2655,7 +2655,7 @@ static void bfq_end_wr(struct bfq_data *bfqd) bfq_bfqq_end_wr(bfqq); bfq_end_wr_async(bfqd); =20 - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(bfqd->queue->elevator); } =20 static sector_t bfq_io_struct_pos(void *io_struct, bool request) @@ -5301,8 +5301,6 @@ static struct request *bfq_dispatch_request(struct bl= k_mq_hw_ctx *hctx) struct bfq_queue *in_serv_queue; bool waiting_rq, idle_timer_disabled =3D false; =20 - spin_lock_irq(&bfqd->lock); - in_serv_queue =3D bfqd->in_service_queue; waiting_rq =3D in_serv_queue && bfq_bfqq_wait_request(in_serv_queue); =20 @@ -5312,7 +5310,6 @@ static struct request *bfq_dispatch_request(struct bl= k_mq_hw_ctx *hctx) waiting_rq && !bfq_bfqq_wait_request(in_serv_queue); } =20 - spin_unlock_irq(&bfqd->lock); bfq_update_dispatch_stats(hctx->queue, rq, idle_timer_disabled ? in_serv_queue : NULL, idle_timer_disabled); @@ -5490,9 +5487,9 @@ static void bfq_exit_icq(struct io_cq *icq) * this is the last time these queues are accessed. */ if (bfqd) { - spin_lock_irqsave(&bfqd->lock, flags); + elevator_lock_irqsave(bfqd->queue->elevator, flags); _bfq_exit_icq(bic, bfqd->num_actuators); - spin_unlock_irqrestore(&bfqd->lock, flags); + elevator_unlock_irqrestore(bfqd->queue->elevator, flags); } else { _bfq_exit_icq(bic, BFQ_MAX_ACTUATORS); } @@ -6247,10 +6244,10 @@ static void bfq_insert_request(struct blk_mq_hw_ctx= *hctx, struct request *rq, if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio) bfqg_stats_update_legacy_io(q, rq); #endif - spin_lock_irq(&bfqd->lock); + elevator_lock_irq(q->elevator); bfqq =3D bfq_init_rq(rq); if (blk_mq_sched_try_insert_merge(q, rq, &free)) { - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(q->elevator); blk_mq_free_requests(&free); return; } @@ -6283,7 +6280,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *= hctx, struct request *rq, * merge). */ cmd_flags =3D rq->cmd_flags; - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(q->elevator); =20 bfq_update_insert_stats(q, bfqq, idle_timer_disabled, cmd_flags); @@ -6664,7 +6661,7 @@ static void bfq_finish_requeue_request(struct request= *rq) rq->io_start_time_ns, rq->cmd_flags); =20 - spin_lock_irqsave(&bfqd->lock, flags); + elevator_lock_irqsave(bfqd->queue->elevator, flags); if (likely(rq->rq_flags & RQF_STARTED)) { if (rq =3D=3D bfqd->waited_rq) bfq_update_inject_limit(bfqd, bfqq); @@ -6674,7 +6671,7 @@ static void bfq_finish_requeue_request(struct request= *rq) bfqq_request_freed(bfqq); bfq_put_queue(bfqq); RQ_BIC(rq)->requests--; - spin_unlock_irqrestore(&bfqd->lock, flags); + elevator_unlock_irqrestore(bfqd->queue->elevator, flags); =20 /* * Reset private fields. In case of a requeue, this allows @@ -7005,7 +7002,7 @@ bfq_idle_slice_timer_body(struct bfq_data *bfqd, stru= ct bfq_queue *bfqq) enum bfqq_expiration reason; unsigned long flags; =20 - spin_lock_irqsave(&bfqd->lock, flags); + elevator_lock_irqsave(bfqd->queue->elevator, flags); =20 /* * Considering that bfqq may be in race, we should firstly check @@ -7015,7 +7012,7 @@ bfq_idle_slice_timer_body(struct bfq_data *bfqd, stru= ct bfq_queue *bfqq) * been cleared in __bfq_bfqd_reset_in_service func. */ if (bfqq !=3D bfqd->in_service_queue) { - spin_unlock_irqrestore(&bfqd->lock, flags); + elevator_unlock_irqrestore(bfqd->queue->elevator, flags); return; } =20 @@ -7043,7 +7040,7 @@ bfq_idle_slice_timer_body(struct bfq_data *bfqd, stru= ct bfq_queue *bfqq) =20 schedule_dispatch: bfq_schedule_dispatch(bfqd); - spin_unlock_irqrestore(&bfqd->lock, flags); + elevator_unlock_irqrestore(bfqd->queue->elevator, flags); } =20 /* @@ -7156,10 +7153,10 @@ static void bfq_exit_queue(struct elevator_queue *e) =20 hrtimer_cancel(&bfqd->idle_slice_timer); =20 - spin_lock_irq(&bfqd->lock); + elevator_lock_irq(e); list_for_each_entry_safe(bfqq, n, &bfqd->idle_list, bfqq_list) bfq_deactivate_bfqq(bfqd, bfqq, false, false); - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(e); =20 for (actuator =3D 0; actuator < bfqd->num_actuators; actuator++) WARN_ON_ONCE(bfqd->rq_in_driver[actuator]); @@ -7173,10 +7170,10 @@ static void bfq_exit_queue(struct elevator_queue *e) #ifdef CONFIG_BFQ_GROUP_IOSCHED blkcg_deactivate_policy(bfqd->queue->disk, &blkcg_policy_bfq); #else - spin_lock_irq(&bfqd->lock); + elevator_lock_irq(e); bfq_put_async_queues(bfqd, bfqd->root_group); kfree(bfqd->root_group); - spin_unlock_irq(&bfqd->lock); + elevator_unlock_irq(e); #endif =20 blk_stat_disable_accounting(bfqd->queue); @@ -7335,8 +7332,6 @@ static int bfq_init_queue(struct request_queue *q, st= ruct elevator_queue *eq) /* see comments on the definition of next field inside bfq_data */ bfqd->actuator_load_threshold =3D 4; =20 - spin_lock_init(&bfqd->lock); - /* * The invocation of the next bfq_create_group_hierarchy * function is the head of a chain of function calls @@ -7359,6 +7354,7 @@ static int bfq_init_queue(struct request_queue *q, st= ruct elevator_queue *eq) bfq_init_entity(&bfqd->oom_bfqq.entity, bfqd->root_group); bfq_depth_updated(q); =20 + set_bit(ELEVATOR_FLAG_DISPATCH_IRQ, &eq->flags); /* We dispatch from request queue wide instead of hw queue */ blk_queue_flag_set(QUEUE_FLAG_SQ_SCHED, q); =20 diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 34a498e6b2a5..7935663988c3 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -795,8 +795,6 @@ struct bfq_data { /* fallback dummy bfqq for extreme OOM conditions */ struct bfq_queue oom_bfqq; =20 - spinlock_t lock; - /* * bic associated with the task issuing current bio for * merging. This and the next field are used as a support to --=20 2.39.2 From nobody Fri Oct 3 02:03:25 2025 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7463B2EBDD9; Mon, 8 Sep 2025 09:29:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323772; cv=none; b=mlmRfgGpO85UbhYzhbur7aHTbv2sjM52naoCa1DPWlAt7FjKnEWG/PXWFq6QGxLplS6HbOdc9sNBTKU5bnUaArsIb3o0T9mHkqj1ars8dE8LP8mtNknGMXGNiAT6B7RsIpMANOEYkeZe4GqhZoKaZrPLGatpTf5pOqC+GYiilmA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323772; c=relaxed/simple; bh=KwWvbVIcttFb/phNvSQXEynyrXAMQeq4hE88O8a3w+k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ifN5v49r5m9HVvsAIHuVAhuV89KRHCvza7GXrL7qt76q2hgmpf6W4sEDRgXb//+1iGxSQ8m2PHxwAVZUjVQK6qUPBKgulw3oCWt3UMdeAYLPHKGySYMSAI6avXYgQr3EMJlsYREH8YE/fytKVlBYoeIcIHs6NcsnuyEQDlgBcbI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4cL1qr2x9tzKHMxy; Mon, 8 Sep 2025 17:29:28 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.128]) by mail.maildlp.com (Postfix) with ESMTP id 801211A2B2D; Mon, 8 Sep 2025 17:29:28 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.104.67]) by APP4 (Coremail) with SMTP id gCh0CgAXYIzzob5oGjMyBw--.46699S8; Mon, 08 Sep 2025 17:29:28 +0800 (CST) From: Yu Kuai To: dlemoal@kernel.org, hare@suse.de, bvanassche@acm.org, ming.lei@redhat.com, tj@kernel.org, josef@toxicpanda.com, axboe@kernel.dk, yukuai3@huawei.com Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai1@huaweicloud.com, yi.zhang@huawei.com, yangerkun@huawei.com, johnny.chenyi@huawei.com Subject: [PATCH v4 for-6.18/block 4/5] blk-mq-sched: refactor __blk_mq_do_dispatch_sched() Date: Mon, 8 Sep 2025 17:20:06 +0800 Message-Id: <20250908092007.3796967-5-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250908092007.3796967-1-yukuai1@huaweicloud.com> References: <20250908092007.3796967-1-yukuai1@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: gCh0CgAXYIzzob5oGjMyBw--.46699S8 X-Coremail-Antispam: 1UD129KBjvJXoWxuw4UuFy5GFWrAw1UGFyDKFg_yoW3XryUpF 4fGF4ay395XF4jqFyIvw43Jw1fA34xuasrWryrKrW3twn0qrs8Jrn5JFyUAFs7Jr95uFZF 9r4DWrZ8CFs2qrDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUma14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_JF0E3s1l82xGYI kIc2x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2 z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F 4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq 3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7 IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4U M4x0Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2 kIc2xKxwCY1x0262kKe7AKxVWUtVW8ZwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkE bVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67 AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI 42IY6xIIjxv20xvEc7CjxVAFwI0_Cr0_Gr1UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCw CI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4UJVWxJrUvcSsG vfC2KfnxnUUI43ZEXa7VUbPC7UUUUUU== X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ Content-Type: text/plain; charset="utf-8" From: Yu Kuai Introduce struct sched_dispatch_ctx, and split the helper into elevator_dispatch_one_request() and elevator_finish_dispatch(). Also and comments about the non-error return value. Make code cleaner, and make it easier to add a new branch to dispatch a batch of requests at a time in the next patch. Signed-off-by: Yu Kuai --- block/blk-mq-sched.c | 196 ++++++++++++++++++++++++++----------------- 1 file changed, 118 insertions(+), 78 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 6f2b0573a8ff..e6305b680db9 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -16,6 +16,16 @@ #include "blk-mq-sched.h" #include "blk-wbt.h" =20 +struct sched_dispatch_ctx { + struct blk_mq_hw_ctx *hctx; + struct list_head rq_list; + + int count; + bool multi_hctxs; + bool run_queue; + bool busy; +}; + /* * Mark a hardware queue as needing a restart. */ @@ -74,92 +84,92 @@ static bool blk_mq_dispatch_hctx_list(struct list_head = *rq_list) =20 #define BLK_MQ_BUDGET_DELAY 3 /* ms units */ =20 -/* - * Only SCSI implements .get_budget and .put_budget, and SCSI restarts - * its queue by itself in its completion handler, so we don't need to - * restart queue if .get_budget() fails to get the budget. - * - * Returns -EAGAIN if hctx->dispatch was found non-empty and run_work has = to - * be run again. This is necessary to avoid starving flushes. - */ -static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) +static bool blk_mq_should_dispatch(struct sched_dispatch_ctx *ctx) { - struct request_queue *q =3D hctx->queue; - struct elevator_queue *e =3D q->elevator; - bool multi_hctxs =3D false, run_queue =3D false; - bool dispatched =3D false, busy =3D false; - unsigned int max_dispatch; - LIST_HEAD(rq_list); - int count =3D 0; + struct elevator_queue *e =3D ctx->hctx->queue->elevator; =20 - if (hctx->dispatch_busy) - max_dispatch =3D 1; - else - max_dispatch =3D hctx->queue->nr_requests; + if (e->type->ops.has_work && !e->type->ops.has_work(ctx->hctx)) + return false; =20 - do { - struct request *rq; - int budget_token; + if (!list_empty_careful(&ctx->hctx->dispatch)) { + ctx->busy =3D true; + return false; + } =20 - if (e->type->ops.has_work && !e->type->ops.has_work(hctx)) - break; + return true; +} =20 - if (!list_empty_careful(&hctx->dispatch)) { - busy =3D true; - break; - } +static bool blk_mq_dispatch_one_request(struct sched_dispatch_ctx *ctx) +{ + struct request_queue *q =3D ctx->hctx->queue; + struct elevator_queue *e =3D q->elevator; + struct request *rq; + int budget_token; =20 - budget_token =3D blk_mq_get_dispatch_budget(q); - if (budget_token < 0) - break; + if (!blk_mq_should_dispatch(ctx)) + return false; =20 - if (blk_queue_sq_sched(q)) { - elevator_dispatch_lock(e); - rq =3D e->type->ops.dispatch_request(hctx); - elevator_dispatch_unlock(e); - } else { - rq =3D e->type->ops.dispatch_request(hctx); - } + budget_token =3D blk_mq_get_dispatch_budget(q); + if (budget_token < 0) + return false; =20 - if (!rq) { - blk_mq_put_dispatch_budget(q, budget_token); - /* - * We're releasing without dispatching. Holding the - * budget could have blocked any "hctx"s with the - * same queue and if we didn't dispatch then there's - * no guarantee anyone will kick the queue. Kick it - * ourselves. - */ - run_queue =3D true; - break; - } - - blk_mq_set_rq_budget_token(rq, budget_token); + if (blk_queue_sq_sched(q)) { + elevator_dispatch_lock(e); + rq =3D e->type->ops.dispatch_request(ctx->hctx); + elevator_dispatch_unlock(e); + } else { + rq =3D e->type->ops.dispatch_request(ctx->hctx); + } =20 + if (!rq) { + blk_mq_put_dispatch_budget(q, budget_token); /* - * Now this rq owns the budget which has to be released - * if this rq won't be queued to driver via .queue_rq() - * in blk_mq_dispatch_rq_list(). + * We're releasing without dispatching. Holding the + * budget could have blocked any "hctx"s with the + * same queue and if we didn't dispatch then there's + * no guarantee anyone will kick the queue. Kick it + * ourselves. */ - list_add_tail(&rq->queuelist, &rq_list); - count++; - if (rq->mq_hctx !=3D hctx) - multi_hctxs =3D true; + ctx->run_queue =3D true; + return false; + } =20 - /* - * If we cannot get tag for the request, stop dequeueing - * requests from the IO scheduler. We are unlikely to be able - * to submit them anyway and it creates false impression for - * scheduling heuristics that the device can take more IO. - */ - if (!blk_mq_get_driver_tag(rq)) - break; - } while (count < max_dispatch); + blk_mq_set_rq_budget_token(rq, budget_token); =20 - if (!count) { - if (run_queue) - blk_mq_delay_run_hw_queues(q, BLK_MQ_BUDGET_DELAY); - } else if (multi_hctxs) { + /* + * Now this rq owns the budget which has to be released + * if this rq won't be queued to driver via .queue_rq() + * in blk_mq_dispatch_rq_list(). + */ + list_add_tail(&rq->queuelist, &ctx->rq_list); + ctx->count++; + if (rq->mq_hctx !=3D ctx->hctx) + ctx->multi_hctxs =3D true; + + /* + * If we cannot get tag for the request, stop dequeueing + * requests from the IO scheduler. We are unlikely to be able + * to submit them anyway and it creates false impression for + * scheduling heuristics that the device can take more IO. + */ + return blk_mq_get_driver_tag(rq); +} + +/* + * Returns -EAGAIN if hctx->dispatch was found non-empty and run_work has = to + * be run again. This is necessary to avoid starving flushes. + * Return 0 if no request is dispatched. + * Return 1 if at least one request is dispatched. + */ +static int blk_mq_finish_dispatch(struct sched_dispatch_ctx *ctx) +{ + bool dispatched =3D false; + + if (!ctx->count) { + if (ctx->run_queue) + blk_mq_delay_run_hw_queues(ctx->hctx->queue, + BLK_MQ_BUDGET_DELAY); + } else if (ctx->multi_hctxs) { /* * Requests from different hctx may be dequeued from some * schedulers, such as bfq and deadline. @@ -167,19 +177,49 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_h= w_ctx *hctx) * Sort the requests in the list according to their hctx, * dispatch batching requests from same hctx at a time. */ - list_sort(NULL, &rq_list, sched_rq_cmp); + list_sort(NULL, &ctx->rq_list, sched_rq_cmp); do { - dispatched |=3D blk_mq_dispatch_hctx_list(&rq_list); - } while (!list_empty(&rq_list)); + dispatched |=3D blk_mq_dispatch_hctx_list(&ctx->rq_list); + } while (!list_empty(&ctx->rq_list)); } else { - dispatched =3D blk_mq_dispatch_rq_list(hctx, &rq_list, false); + dispatched =3D blk_mq_dispatch_rq_list(ctx->hctx, &ctx->rq_list, + false); } =20 - if (busy) + if (ctx->busy) return -EAGAIN; + return !!dispatched; } =20 +/* + * Only SCSI implements .get_budget and .put_budget, and SCSI restarts + * its queue by itself in its completion handler, so we don't need to + * restart queue if .get_budget() fails to get the budget. + * + * See blk_mq_finish_dispatch() for return values. + */ +static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) +{ + unsigned int max_dispatch; + struct sched_dispatch_ctx ctx =3D { + .hctx =3D hctx, + .rq_list =3D LIST_HEAD_INIT(ctx.rq_list), + }; + + if (hctx->dispatch_busy) + max_dispatch =3D 1; + else + max_dispatch =3D hctx->queue->nr_requests; + + do { + if (!blk_mq_dispatch_one_request(&ctx)) + break; + } while (ctx.count < max_dispatch); + + return blk_mq_finish_dispatch(&ctx); +} + static int blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) { unsigned long end =3D jiffies + HZ; --=20 2.39.2 From nobody Fri Oct 3 02:03:25 2025 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 004262F069C; Mon, 8 Sep 2025 09:29:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323773; cv=none; b=iVHIXENXwQZzjZlyIqrfsal/jeolw28BKlydwIpPlrNhbDJfXVpCFcJQWrNsfO3QSX/wnYVzaz4mg8sc+j0TTT6tzTq0CyqblIrlVnqIUNpJya55WwvtypP7XZEIxCIm+ZR75O6+/cekt38XSv0jWaT45W7u+/ZsuHmh9XiDko8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757323773; c=relaxed/simple; bh=WPz79VF6Fr9zr+SPrvDTdY9SUdYOoerXVi6P+YlD9ms=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PNXpYkyDabvD32ngjFdwsq/gw3hJ4pykPhgrAKpCzgXbHGAB9qi8yBLle4jN0dA05mK+LwUIf9MLIUn0MIlHdpVc3+EU6iW8PUcyvXr6+EwP0EwUGV7JfLr0EoaeVrTYQkXX5vkTIl0Jha2hzqO/NHMnqsfuEYYnUuqbXWnz+gE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=none smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTPS id 4cL1qt4JcGzYQvZ6; Mon, 8 Sep 2025 17:29:30 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.128]) by mail.maildlp.com (Postfix) with ESMTP id 1BFC31A174E; Mon, 8 Sep 2025 17:29:29 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.104.67]) by APP4 (Coremail) with SMTP id gCh0CgAXYIzzob5oGjMyBw--.46699S9; Mon, 08 Sep 2025 17:29:28 +0800 (CST) From: Yu Kuai To: dlemoal@kernel.org, hare@suse.de, bvanassche@acm.org, ming.lei@redhat.com, tj@kernel.org, josef@toxicpanda.com, axboe@kernel.dk, yukuai3@huawei.com Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai1@huaweicloud.com, yi.zhang@huawei.com, yangerkun@huawei.com, johnny.chenyi@huawei.com Subject: [PATCH v4 for-6.18/block 5/5] blk-mq-sched: support request batch dispatching for sq elevator Date: Mon, 8 Sep 2025 17:20:07 +0800 Message-Id: <20250908092007.3796967-6-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250908092007.3796967-1-yukuai1@huaweicloud.com> References: <20250908092007.3796967-1-yukuai1@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: gCh0CgAXYIzzob5oGjMyBw--.46699S9 X-Coremail-Antispam: 1UD129KBjvJXoWxWFWkXFyfCF1kuF15XFy7Wrg_yoWrAF4kpF 48Ja1YyrW0qasFqF93Cw47Jw15Aw4xuF9rGryfKr43tan7XrsxAr1rJFyUZF4xAr4fCFsr ur4DWrykW3WIq37anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUma14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_JF0E3s1l82xGYI kIc2x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2 z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F 4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq 3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7 IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4U M4x0Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2 kIc2xKxwCY1x0262kKe7AKxVWUtVW8ZwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkE bVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67 AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67AKxVWUCVW8JwCI 42IY6xIIjxv20xvEc7CjxVAFwI0_Cr0_Gr1UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCw CI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4UJVWxJrUvcSsG vfC2KfnxnUUI43ZEXa7VUbPC7UUUUUU== X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ Content-Type: text/plain; charset="utf-8" From: Yu Kuai For dispatch_request method, current behavior is dispatching one request at a time. In the case of multiple dispatching contexts, This behavior, on the one hand, introduce intense lock contention: t1: t2: t3: lock lock lock // grab lock ops.dispatch_request unlock // grab lock ops.dispatch_request unlock // grab lock ops.dispatch_request unlock on the other hand, messing up the requests dispatching order: t1: lock rq1 =3D ops.dispatch_request unlock t2: lock rq2 =3D ops.dispatch_request unlock lock rq3 =3D ops.dispatch_request unlock lock rq4 =3D ops.dispatch_request unlock //rq1,rq3 issue to disk // rq2, rq4 issue to disk In this case, the elevator dispatch order is rq 1-2-3-4, however, such order in disk is rq 1-3-2-4, the order for rq2 and rq3 is inversed. Fix those problems by introducing elevator_dispatch_requests(), this helper will grab the lock and dispatch a batch of requests while holding the lock. Signed-off-by: Yu Kuai --- block/blk-mq-sched.c | 65 +++++++++++++++++++++++++++++++++++++++++--- block/blk-mq.h | 18 ++++++++++++ 2 files changed, 79 insertions(+), 4 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index e6305b680db9..3809ad880d49 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -192,6 +192,59 @@ static int blk_mq_finish_dispatch(struct sched_dispatc= h_ctx *ctx) return !!dispatched; } =20 +static void blk_mq_dispatch_requests(struct sched_dispatch_ctx *ctx) +{ + struct request_queue *q =3D ctx->hctx->queue; + struct elevator_queue *e =3D q->elevator; + bool has_get_budget =3D q->mq_ops->get_budget !=3D NULL; + int budget_token[BUDGET_TOKEN_BATCH]; + int count =3D q->nr_requests; + int i; + + while (true) { + if (!blk_mq_should_dispatch(ctx)) + return; + + if (has_get_budget) { + count =3D blk_mq_get_dispatch_budgets(q, budget_token); + if (count <=3D 0) + return; + } + + elevator_dispatch_lock(e); + for (i =3D 0; i < count; ++i) { + struct request *rq =3D + e->type->ops.dispatch_request(ctx->hctx); + + if (!rq) { + ctx->run_queue =3D true; + goto err_free_budgets; + } + + if (has_get_budget) + blk_mq_set_rq_budget_token(rq, budget_token[i]); + + list_add_tail(&rq->queuelist, &ctx->rq_list); + ctx->count++; + + if (rq->mq_hctx !=3D ctx->hctx) + ctx->multi_hctxs =3D true; + + if (!blk_mq_get_driver_tag(rq)) { + i++; + goto err_free_budgets; + } + } + elevator_dispatch_unlock(e); + } + +err_free_budgets: + elevator_dispatch_unlock(e); + if (has_get_budget) + for (; i < count; ++i) + blk_mq_put_dispatch_budget(q, budget_token[i]); +} + /* * Only SCSI implements .get_budget and .put_budget, and SCSI restarts * its queue by itself in its completion handler, so we don't need to @@ -212,10 +265,14 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_h= w_ctx *hctx) else max_dispatch =3D hctx->queue->nr_requests; =20 - do { - if (!blk_mq_dispatch_one_request(&ctx)) - break; - } while (ctx.count < max_dispatch); + if (!hctx->dispatch_busy && blk_queue_sq_sched(hctx->queue)) { + blk_mq_dispatch_requests(&ctx); + } else { + do { + if (!blk_mq_dispatch_one_request(&ctx)) + break; + } while (ctx.count < max_dispatch); + } =20 return blk_mq_finish_dispatch(&ctx); } diff --git a/block/blk-mq.h b/block/blk-mq.h index af09eb617d11..1f35f9ec146d 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -38,6 +38,7 @@ enum { }; =20 #define BLK_MQ_CPU_WORK_BATCH (8) +#define BUDGET_TOKEN_BATCH (8) =20 typedef unsigned int __bitwise blk_insert_t; #define BLK_MQ_INSERT_AT_HEAD ((__force blk_insert_t)0x01) @@ -274,6 +275,23 @@ static inline int blk_mq_get_dispatch_budget(struct re= quest_queue *q) return 0; } =20 +static inline int blk_mq_get_dispatch_budgets(struct request_queue *q, + int *budget_token) +{ + int count =3D 0; + + while (count < BUDGET_TOKEN_BATCH) { + int token =3D q->mq_ops->get_budget(q); + + if (token < 0) + return count; + + budget_token[count++] =3D token; + } + + return count; +} + static inline void blk_mq_set_rq_budget_token(struct request *rq, int toke= n) { if (token < 0) --=20 2.39.2