From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAD36C433EF for ; Mon, 10 Jan 2022 09:12:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242765AbiAJJMj (ORCPT ); Mon, 10 Jan 2022 04:12:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242653AbiAJJMO (ORCPT ); Mon, 10 Jan 2022 04:12:14 -0500 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 055C9C061212; Mon, 10 Jan 2022 01:12:08 -0800 (PST) Received: by mail-pf1-x42d.google.com with SMTP id t19so10070964pfg.9; Mon, 10 Jan 2022 01:12:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0AjmjFBGavsvtxiPwAj3xUkJ0oQuucx2gbTAmMmtGEI=; b=qBB4XiRmOUg7Fhsgm27ytLlait1aSQJ55NYbveAGQ8KlfJ43zppL6+ZU10iiwoZtQN rnOcN9ZJrl5qfh21UySN0zRBFka/magri+itMAODk4yi6TMZ8NybMM7nyKxlinAIZZk6 lAtDDaEX8q69XigGoD+W8gXOWpa6/qFxP+esDz+fpP33zTmWVJH2Iuba8k+FYHwE59ta zi+XSul789Bx/DPrhC5N7+4CYbajAEsQihag4sVutDSWnQFMqbUScZk/hLB3R+FJX7/b W3L72imBZYo3zbGkYTCxqCVB47Xyz9RvxUqZQ1bB6nJr7G8ejAdgIH77c4wC+v//kKHw KTIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0AjmjFBGavsvtxiPwAj3xUkJ0oQuucx2gbTAmMmtGEI=; b=s5GxH0z6nGQAKhniAgNICTi1hdve9VKSxqgeSaDXz6DXk1yCeOM0mpq8xlYACq8ZGG 6uTjNjcp+K68gIcNvn5zOr9aS1ZmdyFDQpsxkEWQUYDsKH2eqmOyV8PJ7JviF4qtu9kx +TvtwVqVaF64IyDX/q+AwNvcvggaYST0Kze3xDe8MHntu/ZYQPWsUO30pxKiRp33h81t WqAmLh7lyJMffE69O/wXsYa3RXDO4MjZkhpQvpK7EFWXyHlqyamXwJ226BNbQ5icvw/i k8QdWNi6g+BvSRZ9uli5R25MnEUPNgwNfS+OX5TF6K/TqMYnLINfIJJ/6wgUHetuXN3t Yqcw== X-Gm-Message-State: AOAM533tldmgN8meMO1kfvofACRJF0aLbw3KoQvLU4ucXw+PU3RMQih1 HvQU49wnpyKP+gzRJFYMM74= X-Google-Smtp-Source: ABdhPJzOD1APLre3OAuDoVPSCNs0ASHxcx0YcINNGcOHqWAZlwSn+cFQ4S7ZO/t2peezlzvk7yVJuA== X-Received: by 2002:a63:455f:: with SMTP id u31mr65643828pgk.537.1641805928378; Mon, 10 Jan 2022 01:12:08 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:08 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/13] blk: make blk-rq-qos support pluggable and modular policy Date: Mon, 10 Jan 2022 17:10:34 +0800 Message-Id: <20220110091046.17010-2-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao blk-rq-qos is a standalone framework out of io-sched and can be used to control or observe the IO progress in block-layer with hooks. blk-rq-qos is a great design but right now, it is totally fixed and built-in and shut out peoples who want to use it with external module. This patch make blk-rq-qos policies pluggable and modular. (1) Add code to maintain the rq_qos_ops. A rq-qos module need to register itself with rq_qos_register(). The original enum rq_qos_id will be removed in following patch. They will use a dynamic id maintained by rq_qos_ida. (2) Add .init callback into rq_qos_ops. We use it to initialize the resource. (3) Add /sys/block/x/queue/qos We can use '+name' or "-name" to open or close the blk-rq-qos policy. Because the rq-qos list can be modified at anytime, rq_qos_id() which has been renamed to rq_qos_by_id() has to iterate the list under sysfs_lock or queue_lock. This patch adapts the code for this. More details, please refer to the comment above rq_qos_get(), And the rq_qos_exit() is moved to blk_cleanup_queue. Except for these modification, there is no other functional change here. Following patches will adpat the code of wbt, iolatency, iocost and ioprio to make them pluggable and modular one by one. Signed-off-by: Wang Jianchao Reported-by: kernel test robot --- block/blk-core.c | 2 + block/blk-iocost.c | 20 ++- block/blk-mq-debugfs.c | 4 +- block/blk-rq-qos.c | 312 ++++++++++++++++++++++++++++++++++++++++- block/blk-rq-qos.h | 55 +++++++- block/blk-sysfs.c | 2 + block/blk-wbt.c | 6 +- block/elevator.c | 3 + block/genhd.c | 3 - include/linux/blkdev.h | 4 + 10 files changed, 394 insertions(+), 17 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 1378d084c770..2847ab514c1f 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -51,6 +51,7 @@ #include "blk-mq-sched.h" #include "blk-pm.h" #include "blk-throttle.h" +#include "blk-rq-qos.h" =20 struct dentry *blk_debugfs_root; =20 @@ -377,6 +378,7 @@ void blk_cleanup_queue(struct request_queue *q) * it is safe to free requests now. */ mutex_lock(&q->sysfs_lock); + rq_qos_exit(q); if (q->elevator) blk_mq_sched_free_rqs(q); mutex_unlock(&q->sysfs_lock); diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 769b64394298..cfc0e305c32e 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -662,7 +662,7 @@ static struct ioc *rqos_to_ioc(struct rq_qos *rqos) =20 static struct ioc *q_to_ioc(struct request_queue *q) { - return rqos_to_ioc(rq_qos_id(q, RQ_QOS_COST)); + return rqos_to_ioc(rq_qos_by_id(q, RQ_QOS_COST)); } =20 static const char *q_name(struct request_queue *q) @@ -3162,6 +3162,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file = *of, char *input, size_t nbytes, loff_t off) { struct block_device *bdev; + struct rq_qos *rqos; struct ioc *ioc; u32 qos[NR_QOS_PARAMS]; bool enable, user; @@ -3172,14 +3173,15 @@ static ssize_t ioc_qos_write(struct kernfs_open_fil= e *of, char *input, if (IS_ERR(bdev)) return PTR_ERR(bdev); =20 - ioc =3D q_to_ioc(bdev_get_queue(bdev)); - if (!ioc) { + rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); + if (!rqos) { ret =3D blk_iocost_init(bdev_get_queue(bdev)); if (ret) goto err; - ioc =3D q_to_ioc(bdev_get_queue(bdev)); + rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); } =20 + ioc =3D rqos_to_ioc(rqos); spin_lock_irq(&ioc->lock); memcpy(qos, ioc->params.qos, sizeof(qos)); enable =3D ioc->enabled; @@ -3272,10 +3274,12 @@ static ssize_t ioc_qos_write(struct kernfs_open_fil= e *of, char *input, ioc_refresh_params(ioc, true); spin_unlock_irq(&ioc->lock); =20 + rq_qos_put(rqos); blkdev_put_no_open(bdev); return nbytes; einval: ret =3D -EINVAL; + rq_qos_put(rqos); err: blkdev_put_no_open(bdev); return ret; @@ -3329,6 +3333,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_ope= n_file *of, char *input, size_t nbytes, loff_t off) { struct block_device *bdev; + struct rq_qos *rqos; struct ioc *ioc; u64 u[NR_I_LCOEFS]; bool user; @@ -3339,14 +3344,15 @@ static ssize_t ioc_cost_model_write(struct kernfs_o= pen_file *of, char *input, if (IS_ERR(bdev)) return PTR_ERR(bdev); =20 - ioc =3D q_to_ioc(bdev_get_queue(bdev)); + rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); if (!ioc) { ret =3D blk_iocost_init(bdev_get_queue(bdev)); if (ret) goto err; - ioc =3D q_to_ioc(bdev_get_queue(bdev)); + rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); } =20 + ioc =3D rqos_to_ioc(rqos); spin_lock_irq(&ioc->lock); memcpy(u, ioc->params.i_lcoefs, sizeof(u)); user =3D ioc->user_cost_model; @@ -3397,11 +3403,13 @@ static ssize_t ioc_cost_model_write(struct kernfs_o= pen_file *of, char *input, ioc_refresh_params(ioc, true); spin_unlock_irq(&ioc->lock); =20 + rq_qos_put(rqos); blkdev_put_no_open(bdev); return nbytes; =20 einval: ret =3D -EINVAL; + rq_qos_put(rqos); err: blkdev_put_no_open(bdev); return ret; diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 4f2cf8399f3d..e3e8d54c836f 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -841,7 +841,9 @@ void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos) void blk_mq_debugfs_register_rqos(struct rq_qos *rqos) { struct request_queue *q =3D rqos->q; - const char *dir_name =3D rq_qos_id_to_name(rqos->id); + const char *dir_name; + + dir_name =3D rqos->ops->name ? rqos->ops->name : rq_qos_id_to_name(rqos->= id); =20 if (rqos->debugfs_dir || !rqos->ops->debugfs_attrs) return; diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index e83af7bc7591..a94ff872722b 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -2,6 +2,11 @@ =20 #include "blk-rq-qos.h" =20 +static DEFINE_IDA(rq_qos_ida); +static int nr_rqos_blkcg_pols; +static DEFINE_MUTEX(rq_qos_mutex); +static LIST_HEAD(rq_qos_list); + /* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -294,11 +299,316 @@ void rq_qos_wait(struct rq_wait *rqw, void *private_= data, =20 void rq_qos_exit(struct request_queue *q) { - blk_mq_debugfs_unregister_queue_rqos(q); + WARN_ON(!mutex_is_locked(&q->sysfs_lock)); =20 while (q->rq_qos) { struct rq_qos *rqos =3D q->rq_qos; q->rq_qos =3D rqos->next; + if (rqos->ops->owner) + module_put(rqos->ops->owner); rqos->ops->exit(rqos); } + blk_mq_debugfs_unregister_queue_rqos(q); +} + +/* + * After the pluggable blk-qos, rqos's life cycle become complicated, + * qos switching path can add/delete rqos to/from request_queue + * under sysfs_lock and queue_lock. There are following places + * may access rqos through rq_qos_by_id() concurrently: + * (1) normal IO path, under q_usage_counter, + * (2) queue sysfs interfaces, under sysfs_lock, + * (3) blkg_create, the .pd_init_fn() may access rqos, under queue_lock, + * (4) cgroup file, such as ioc_cost_model_write, + * + * (1)(2)(3) are definitely safe. case (4) is tricky. rq_qos_get() is + * for the case. + */ +struct rq_qos *rq_qos_get(struct request_queue *q, int id) +{ + struct rq_qos *rqos; + + spin_lock_irq(&q->queue_lock); + rqos =3D rq_qos_by_id(q, id); + if (rqos && rqos->dying) + rqos =3D NULL; + if (rqos) + refcount_inc(&rqos->ref); + spin_unlock_irq(&q->queue_lock); + return rqos; +} +EXPORT_SYMBOL_GPL(rq_qos_get); + +void rq_qos_put(struct rq_qos *rqos) +{ + struct request_queue *q =3D rqos->q; + + spin_lock_irq(&q->queue_lock); + refcount_dec(&rqos->ref); + if (rqos->dying) + wake_up(&rqos->waitq); + spin_unlock_irq(&q->queue_lock); +} +EXPORT_SYMBOL_GPL(rq_qos_put); + +void rq_qos_activate(struct request_queue *q, + struct rq_qos *rqos, const struct rq_qos_ops *ops) +{ + struct rq_qos *pos; + bool rq_alloc_time =3D false; + + WARN_ON(!mutex_is_locked(&q->sysfs_lock)); + + rqos->dying =3D false; + refcount_set(&rqos->ref, 1); + init_waitqueue_head(&rqos->waitq); + rqos->id =3D ops->id; + rqos->ops =3D ops; + rqos->q =3D q; + rqos->next =3D NULL; + + spin_lock_irq(&q->queue_lock); + pos =3D q->rq_qos; + if (pos) { + while (pos->next) { + if (pos->ops->flags & RQOS_FLAG_RQ_ALLOC_TIME) + rq_alloc_time =3D true; + pos =3D pos->next; + } + pos->next =3D rqos; + } else { + q->rq_qos =3D rqos; + } + if (ops->flags & RQOS_FLAG_RQ_ALLOC_TIME && + !rq_alloc_time) + blk_queue_flag_set(QUEUE_FLAG_RQ_ALLOC_TIME, q); + + spin_unlock_irq(&q->queue_lock); + + if (rqos->ops->debugfs_attrs) + blk_mq_debugfs_register_rqos(rqos); +} +EXPORT_SYMBOL_GPL(rq_qos_activate); + +void rq_qos_deactivate(struct rq_qos *rqos) +{ + struct request_queue *q =3D rqos->q; + struct rq_qos **cur, *pos; + bool rq_alloc_time =3D false; + + WARN_ON(!mutex_is_locked(&q->sysfs_lock)); + + spin_lock_irq(&q->queue_lock); + rqos->dying =3D true; + /* + * Drain all of the usage of get/put_rqos() + */ + wait_event_lock_irq(rqos->waitq, + refcount_read(&rqos->ref) =3D=3D 1, q->queue_lock); + for (cur =3D &q->rq_qos; *cur; cur =3D &(*cur)->next) { + if (*cur =3D=3D rqos) { + *cur =3D rqos->next; + break; + } + } + + pos =3D q->rq_qos; + while (pos && pos->next) { + if (pos->ops->flags & RQOS_FLAG_RQ_ALLOC_TIME) + rq_alloc_time =3D true; + pos =3D pos->next; + } + + if (rqos->ops->flags & RQOS_FLAG_RQ_ALLOC_TIME && + !rq_alloc_time) + blk_queue_flag_clear(QUEUE_FLAG_RQ_ALLOC_TIME, q); + + spin_unlock_irq(&q->queue_lock); + blk_mq_debugfs_unregister_rqos(rqos); +} +EXPORT_SYMBOL_GPL(rq_qos_deactivate); + +static struct rq_qos_ops *rq_qos_find_by_name(const char *name) +{ + struct rq_qos_ops *pos; + + list_for_each_entry(pos, &rq_qos_list, node) { + if (!strncmp(pos->name, name, strlen(pos->name))) + return pos; + } + + return NULL; +} + +int rq_qos_register(struct rq_qos_ops *ops) +{ + int ret, start; + + mutex_lock(&rq_qos_mutex); + + if (rq_qos_find_by_name(ops->name)) { + ret =3D -EEXIST; + goto out; + } + + if (ops->flags & RQOS_FLAG_CGRP_POL && + nr_rqos_blkcg_pols >=3D (BLKCG_MAX_POLS - BLKCG_NON_RQOS_POLS)) { + ret =3D -ENOSPC; + goto out; + } + + start =3D RQ_QOS_IOPRIO + 1; + ret =3D ida_simple_get(&rq_qos_ida, start, INT_MAX, GFP_KERNEL); + if (ret < 0) + goto out; + + if (ops->flags & RQOS_FLAG_CGRP_POL) + nr_rqos_blkcg_pols++; + + ops->id =3D ret; + ret =3D 0; + INIT_LIST_HEAD(&ops->node); + list_add_tail(&ops->node, &rq_qos_list); +out: + mutex_unlock(&rq_qos_mutex); + return ret; +} +EXPORT_SYMBOL_GPL(rq_qos_register); + +void rq_qos_unregister(struct rq_qos_ops *ops) +{ + mutex_lock(&rq_qos_mutex); + + if (ops->flags & RQOS_FLAG_CGRP_POL) + nr_rqos_blkcg_pols--; + list_del_init(&ops->node); + ida_simple_remove(&rq_qos_ida, ops->id); + mutex_unlock(&rq_qos_mutex); +} +EXPORT_SYMBOL_GPL(rq_qos_unregister); + +ssize_t queue_qos_show(struct request_queue *q, char *buf) +{ + struct rq_qos_ops *ops; + struct rq_qos *rqos; + int ret =3D 0; + + mutex_lock(&rq_qos_mutex); + /* + * Show the policies in the order of being invoked + */ + for (rqos =3D q->rq_qos; rqos; rqos =3D rqos->next) { + if (!rqos->ops->name) + continue; + ret +=3D sprintf(buf + ret, "[%s] ", rqos->ops->name); + } + list_for_each_entry(ops, &rq_qos_list, node) { + if (!rq_qos_by_name(q, ops->name)) + ret +=3D sprintf(buf + ret, "%s ", ops->name); + } + + ret--; /* overwrite the last space */ + ret +=3D sprintf(buf + ret, "\n"); + mutex_unlock(&rq_qos_mutex); + + return ret; +} + +int rq_qos_switch(struct request_queue *q, + const struct rq_qos_ops *ops, + struct rq_qos *rqos) +{ + int ret; + + WARN_ON(!mutex_is_locked(&q->sysfs_lock)); + + blk_mq_freeze_queue(q); + if (!rqos) { + ret =3D ops->init(q); + } else { + ops->exit(rqos); + ret =3D 0; + } + blk_mq_unfreeze_queue(q); + + return ret; +} + +ssize_t queue_qos_store(struct request_queue *q, const char *page, + size_t count) +{ + const struct rq_qos_ops *ops; + struct rq_qos *rqos; + const char *qosname; + char *buf; + bool add; + int ret; + + buf =3D kstrdup(page, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + buf =3D strim(buf); + if (buf[0] !=3D '+' && buf[0] !=3D '-') { + ret =3D -EINVAL; + goto out; + } + + add =3D buf[0] =3D=3D '+'; + qosname =3D buf + 1; + + rqos =3D rq_qos_by_name(q, qosname); + if ((buf[0] =3D=3D '+' && rqos)) { + ret =3D -EEXIST; + goto out; + } + + if ((buf[0] =3D=3D '-' && !rqos)) { + ret =3D -ENODEV; + goto out; + } + + mutex_lock(&rq_qos_mutex); + if (add) { + ops =3D rq_qos_find_by_name(qosname); + if (!ops) { + /* + * module_init callback may request this mutex + */ + mutex_unlock(&rq_qos_mutex); + request_module("%s", qosname); + mutex_lock(&rq_qos_mutex); + ops =3D rq_qos_find_by_name(qosname); + } + } else { + ops =3D rqos->ops; + } + + if (!ops) { + ret =3D -EINVAL; + } else if (ops->owner && !try_module_get(ops->owner)) { + ops =3D NULL; + ret =3D -EAGAIN; + } + mutex_unlock(&rq_qos_mutex); + + if (!ops) + goto out; + + if (add) { + ret =3D rq_qos_switch(q, ops, NULL); + if (!ret && ops->owner) + __module_get(ops->owner); + } else { + rq_qos_switch(q, ops, rqos); + ret =3D 0; + if (ops->owner) + module_put(ops->owner); + } + + if (ops->owner) + module_put(ops->owner); +out: + kfree(buf); + return ret ? ret : count; } diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index 3cfbc8668cba..c2b9b41f8fd4 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -26,7 +26,10 @@ struct rq_wait { }; =20 struct rq_qos { - struct rq_qos_ops *ops; + refcount_t ref; + wait_queue_head_t waitq; + bool dying; + const struct rq_qos_ops *ops; struct request_queue *q; enum rq_qos_id id; struct rq_qos *next; @@ -35,7 +38,17 @@ struct rq_qos { #endif }; =20 +enum { + RQOS_FLAG_CGRP_POL =3D 1 << 0, + RQOS_FLAG_RQ_ALLOC_TIME =3D 1 << 1 +}; + struct rq_qos_ops { + struct list_head node; + struct module *owner; + const char *name; + int flags; + int id; void (*throttle)(struct rq_qos *, struct bio *); void (*track)(struct rq_qos *, struct request *, struct bio *); void (*merge)(struct rq_qos *, struct request *, struct bio *); @@ -46,6 +59,7 @@ struct rq_qos_ops { void (*cleanup)(struct rq_qos *, struct bio *); void (*queue_depth_changed)(struct rq_qos *); void (*exit)(struct rq_qos *); + int (*init)(struct request_queue *); const struct blk_mq_debugfs_attr *debugfs_attrs; }; =20 @@ -59,10 +73,12 @@ struct rq_depth { unsigned int default_depth; }; =20 -static inline struct rq_qos *rq_qos_id(struct request_queue *q, - enum rq_qos_id id) +static inline struct rq_qos *rq_qos_by_id(struct request_queue *q, int id) { struct rq_qos *rqos; + + WARN_ON(!mutex_is_locked(&q->sysfs_lock) && !spin_is_locked(&q->queue_loc= k)); + for (rqos =3D q->rq_qos; rqos; rqos =3D rqos->next) { if (rqos->id =3D=3D id) break; @@ -72,12 +88,12 @@ static inline struct rq_qos *rq_qos_id(struct request_q= ueue *q, =20 static inline struct rq_qos *wbt_rq_qos(struct request_queue *q) { - return rq_qos_id(q, RQ_QOS_WBT); + return rq_qos_by_id(q, RQ_QOS_WBT); } =20 static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q) { - return rq_qos_id(q, RQ_QOS_LATENCY); + return rq_qos_by_id(q, RQ_QOS_LATENCY); } =20 static inline void rq_wait_init(struct rq_wait *rq_wait) @@ -132,6 +148,35 @@ static inline void rq_qos_del(struct request_queue *q,= struct rq_qos *rqos) blk_mq_debugfs_unregister_rqos(rqos); } =20 +int rq_qos_register(struct rq_qos_ops *ops); +void rq_qos_unregister(struct rq_qos_ops *ops); +void rq_qos_activate(struct request_queue *q, + struct rq_qos *rqos, const struct rq_qos_ops *ops); +void rq_qos_deactivate(struct rq_qos *rqos); +ssize_t queue_qos_show(struct request_queue *q, char *buf); +ssize_t queue_qos_store(struct request_queue *q, const char *page, + size_t count); +struct rq_qos *rq_qos_get(struct request_queue *q, int id); +void rq_qos_put(struct rq_qos *rqos); + +static inline struct rq_qos *rq_qos_by_name(struct request_queue *q, + const char *name) +{ + struct rq_qos *rqos; + + WARN_ON(!mutex_is_locked(&q->sysfs_lock)); + + for (rqos =3D q->rq_qos; rqos; rqos =3D rqos->next) { + if (!rqos->ops->name) + continue; + + if (!strncmp(rqos->ops->name, name, + strlen(rqos->ops->name))) + return rqos; + } + return NULL; +} + typedef bool (acquire_inflight_cb_t)(struct rq_wait *rqw, void *private_da= ta); typedef void (cleanup_cb_t)(struct rq_wait *rqw, void *private_data); =20 diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index cd75b0f73dc6..91f980985b1b 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -573,6 +573,7 @@ QUEUE_RO_ENTRY(queue_max_segments, "max_segments"); QUEUE_RO_ENTRY(queue_max_integrity_segments, "max_integrity_segments"); QUEUE_RO_ENTRY(queue_max_segment_size, "max_segment_size"); QUEUE_RW_ENTRY(elv_iosched, "scheduler"); +QUEUE_RW_ENTRY(queue_qos, "qos"); =20 QUEUE_RO_ENTRY(queue_logical_block_size, "logical_block_size"); QUEUE_RO_ENTRY(queue_physical_block_size, "physical_block_size"); @@ -632,6 +633,7 @@ static struct attribute *queue_attrs[] =3D { &queue_max_integrity_segments_entry.attr, &queue_max_segment_size_entry.attr, &elv_iosched_entry.attr, + &queue_qos_entry.attr, &queue_hw_sector_size_entry.attr, &queue_logical_block_size_entry.attr, &queue_physical_block_size_entry.attr, diff --git a/block/blk-wbt.c b/block/blk-wbt.c index 0c119be0e813..88265ae4fa41 100644 --- a/block/blk-wbt.c +++ b/block/blk-wbt.c @@ -628,9 +628,13 @@ static void wbt_requeue(struct rq_qos *rqos, struct re= quest *rq) =20 void wbt_set_write_cache(struct request_queue *q, bool write_cache_on) { - struct rq_qos *rqos =3D wbt_rq_qos(q); + struct rq_qos *rqos; + + spin_lock_irq(&q->queue_lock); + rqos =3D wbt_rq_qos(q); if (rqos) RQWB(rqos)->wc =3D write_cache_on; + spin_unlock_irq(&q->queue_lock); } =20 /* diff --git a/block/elevator.c b/block/elevator.c index 19a78d5516ba..fe664674c14d 100644 --- a/block/elevator.c +++ b/block/elevator.c @@ -701,12 +701,15 @@ void elevator_init_mq(struct request_queue *q) * requests, then no need to quiesce queue which may add long boot * latency, especially when lots of disks are involved. */ + + mutex_lock(&q->sysfs_lock); blk_mq_freeze_queue(q); blk_mq_cancel_work_sync(q); =20 err =3D blk_mq_init_sched(q, e); =20 blk_mq_unfreeze_queue(q); + mutex_unlock(&q->sysfs_lock); =20 if (err) { pr_warn("\"%s\" elevator initialization failed, " diff --git a/block/genhd.c b/block/genhd.c index 30362aeacac4..af2e8ebce46e 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -27,7 +27,6 @@ #include =20 #include "blk.h" -#include "blk-rq-qos.h" =20 static struct kobject *block_depr; =20 @@ -621,8 +620,6 @@ void del_gendisk(struct gendisk *disk) device_del(disk_to_dev(disk)); =20 blk_mq_freeze_queue_wait(q); - - rq_qos_exit(q); blk_sync_queue(q); blk_flush_integrity(); /* diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index bd4370baccca..e7dce2232814 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -43,6 +43,10 @@ struct blk_crypto_profile; * Defined here to simplify include dependency. */ #define BLKCG_MAX_POLS 6 +/* + * Non blk-rq-qos blkcg policies include blk-throttle and bfq + */ +#define BLKCG_NON_RQOS_POLS 2 =20 static inline int blk_validate_block_size(unsigned int bsize) { --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83A09C433F5 for ; Mon, 10 Jan 2022 09:15:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242904AbiAJJO7 (ORCPT ); Mon, 10 Jan 2022 04:14:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242714AbiAJJM3 (ORCPT ); Mon, 10 Jan 2022 04:12:29 -0500 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0CDCC0611FD; Mon, 10 Jan 2022 01:12:11 -0800 (PST) Received: by mail-pg1-x533.google.com with SMTP id y9so10592173pgr.11; Mon, 10 Jan 2022 01:12:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=79z+e8U2E7T+KFfzG2ZOpnjLcxockDrhWWDyySilJSA=; b=Xs7/ZrRahdVQa7Nfoz3k+oUc/a8U+WPJCQU5VYGjw2t/JxZv9mct6lTWThtLnxLzkY T/yovAq17YyK0Uf72Q9Ai3+32BYsy3NmutJ1hfrx8/0KhTVc+O6UbP8oDWIR7xdNc53b rAzXSj6r0jLIs2r3y93FkYkIJ5XCi3hkllrSGsF0pVE8U+5rdndsqKlapBVeVCIQOODb LlIRbaZXRs96X3R14jHep/ePxdU8vlB566/T/LVUEKEnb2pH+Dufl9tPG8H/BRHURH+B izYTydxJ788WF42awsAfkcs0FPLsmyTQJvN/lbjmlLeSEGRiuX6gHzRQKTw+47cNv986 2m8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=79z+e8U2E7T+KFfzG2ZOpnjLcxockDrhWWDyySilJSA=; b=N7NkeA1q/LbTAoNyL3x2PzjaDkPHKNiz5zxcV/VbjLwR1QnPoaVSKx9Gm68qlV9cJk dmWhiZN+PSc/sGdwno9eqYnvceQ9Hn4csy1F9JjQXHEVF1ZQEHaw/z0r1+VMJq2m2s2k XIwP6xR9ClsoW89moHwqnkH656N+1qEV468q7uPdT72sz5HUnygbtAMgZ3Gs+jcNjPHx O66bOGd8a3DhYpquZO+IKnDYSuftodRAfCSoAr3uXDxIvepb3l5K29rmrdBNxa7opFoK avvlD2QRZhCkzmebV4RZJvNV/5GWD1YXDwlf6rjxu3Q8D/pK2L//gSBs1NQLAQcgmCEZ XYeA== X-Gm-Message-State: AOAM531bJb165WuPwJ/wNJ84eYCoDg1c/NMJSVxmXWqGrBRHilYhJO0n AclPvYHAWqH7uyxXn8T5er8= X-Google-Smtp-Source: ABdhPJySGRqxk+h5i1Ul+jsHfIYmcDThvDBfPN61ob4u2MUl8JioAd7588dD6VgxCaeKA/jDBlmFoA== X-Received: by 2002:a63:cd49:: with SMTP id a9mr65556903pgj.607.1641805931117; Mon, 10 Jan 2022 01:12:11 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:10 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/13] blk-wbt: make wbt pluggable Date: Mon, 10 Jan 2022 17:10:35 +0800 Message-Id: <20220110091046.17010-3-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao This patch makes wbt pluggable through /sys/block/xxx/queue/qos. Signed-off-by: Wang Jianchao --- block/blk-mq-debugfs.c | 2 -- block/blk-rq-qos.h | 8 ++------ block/blk-sysfs.c | 7 ++----- block/blk-wbt.c | 30 +++++++++++++++++++++++++----- block/blk-wbt.h | 8 ++++---- 5 files changed, 33 insertions(+), 22 deletions(-) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index e3e8d54c836f..e225db3c271f 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -820,8 +820,6 @@ void blk_mq_debugfs_unregister_sched(struct request_que= ue *q) static const char *rq_qos_id_to_name(enum rq_qos_id id) { switch (id) { - case RQ_QOS_WBT: - return "wbt"; case RQ_QOS_LATENCY: return "latency"; case RQ_QOS_COST: diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index c2b9b41f8fd4..de82eb951bdd 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -14,7 +14,6 @@ struct blk_mq_debugfs_attr; =20 enum rq_qos_id { - RQ_QOS_WBT, RQ_QOS_LATENCY, RQ_QOS_COST, RQ_QOS_IOPRIO, @@ -86,11 +85,6 @@ static inline struct rq_qos *rq_qos_by_id(struct request= _queue *q, int id) return rqos; } =20 -static inline struct rq_qos *wbt_rq_qos(struct request_queue *q) -{ - return rq_qos_by_id(q, RQ_QOS_WBT); -} - static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q) { return rq_qos_by_id(q, RQ_QOS_LATENCY); @@ -158,6 +152,8 @@ ssize_t queue_qos_store(struct request_queue *q, const = char *page, size_t count); struct rq_qos *rq_qos_get(struct request_queue *q, int id); void rq_qos_put(struct rq_qos *rqos); +int rq_qos_switch(struct request_queue *q, const struct rq_qos_ops *ops, + struct rq_qos *rqos); =20 static inline struct rq_qos *rq_qos_by_name(struct request_queue *q, const char *name) diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 91f980985b1b..12399e491670 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -482,11 +482,8 @@ static ssize_t queue_wb_lat_store(struct request_queue= *q, const char *page, return -EINVAL; =20 rqos =3D wbt_rq_qos(q); - if (!rqos) { - ret =3D wbt_init(q); - if (ret) - return ret; - } + if (!rqos) + return -EOPNOTSUPP; =20 if (val =3D=3D -1) val =3D wbt_default_latency_nsec(q); diff --git a/block/blk-wbt.c b/block/blk-wbt.c index 88265ae4fa41..ce4b41e50564 100644 --- a/block/blk-wbt.c +++ b/block/blk-wbt.c @@ -31,6 +31,13 @@ #define CREATE_TRACE_POINTS #include =20 +static struct rq_qos_ops wbt_rqos_ops; + +struct rq_qos *wbt_rq_qos(struct request_queue *q) +{ + return rq_qos_by_id(q, wbt_rqos_ops.id); +} + static inline void wbt_clear_state(struct request *rq) { rq->wbt_flags =3D 0; @@ -656,7 +663,7 @@ void wbt_enable_default(struct request_queue *q) return; =20 if (queue_is_mq(q) && IS_ENABLED(CONFIG_BLK_WBT_MQ)) - wbt_init(q); + rq_qos_switch(q, &wbt_rqos_ops, NULL); } EXPORT_SYMBOL_GPL(wbt_enable_default); =20 @@ -696,6 +703,7 @@ static void wbt_exit(struct rq_qos *rqos) struct rq_wb *rwb =3D RQWB(rqos); struct request_queue *q =3D rqos->q; =20 + rq_qos_deactivate(rqos); blk_stat_remove_callback(q, rwb->cb); blk_stat_free_callback(rwb->cb); kfree(rwb); @@ -806,7 +814,9 @@ static const struct blk_mq_debugfs_attr wbt_debugfs_att= rs[] =3D { }; #endif =20 +int wbt_init(struct request_queue *q); static struct rq_qos_ops wbt_rqos_ops =3D { + .name =3D "wbt", .throttle =3D wbt_wait, .issue =3D wbt_issue, .track =3D wbt_track, @@ -815,6 +825,7 @@ static struct rq_qos_ops wbt_rqos_ops =3D { .cleanup =3D wbt_cleanup, .queue_depth_changed =3D wbt_queue_depth_changed, .exit =3D wbt_exit, + .init =3D wbt_init, #ifdef CONFIG_BLK_DEBUG_FS .debugfs_attrs =3D wbt_debugfs_attrs, #endif @@ -838,9 +849,6 @@ int wbt_init(struct request_queue *q) for (i =3D 0; i < WBT_NUM_RWQ; i++) rq_wait_init(&rwb->rq_wait[i]); =20 - rwb->rqos.id =3D RQ_QOS_WBT; - rwb->rqos.ops =3D &wbt_rqos_ops; - rwb->rqos.q =3D q; rwb->last_comp =3D rwb->last_issue =3D jiffies; rwb->win_nsec =3D RWB_WINDOW_NSEC; rwb->enable_state =3D WBT_STATE_ON_DEFAULT; @@ -850,7 +858,7 @@ int wbt_init(struct request_queue *q) /* * Assign rwb and add the stats callback. */ - rq_qos_add(q, &rwb->rqos); + rq_qos_activate(q, &rwb->rqos, &wbt_rqos_ops); blk_stat_add_callback(q, rwb->cb); =20 rwb->min_lat_nsec =3D wbt_default_latency_nsec(q); @@ -860,3 +868,15 @@ int wbt_init(struct request_queue *q) =20 return 0; } + +static __init int wbt_mod_init(void) +{ + return rq_qos_register(&wbt_rqos_ops); +} + +static __exit void wbt_mod_exit(void) +{ + return rq_qos_unregister(&wbt_rqos_ops); +} +module_init(wbt_mod_init); +module_exit(wbt_mod_exit); diff --git a/block/blk-wbt.h b/block/blk-wbt.h index 2eb01becde8c..72e9602df330 100644 --- a/block/blk-wbt.h +++ b/block/blk-wbt.h @@ -88,7 +88,7 @@ static inline unsigned int wbt_inflight(struct rq_wb *rwb) =20 #ifdef CONFIG_BLK_WBT =20 -int wbt_init(struct request_queue *); +struct rq_qos *wbt_rq_qos(struct request_queue *q); void wbt_disable_default(struct request_queue *); void wbt_enable_default(struct request_queue *); =20 @@ -101,12 +101,12 @@ u64 wbt_default_latency_nsec(struct request_queue *); =20 #else =20 -static inline void wbt_track(struct request *rq, enum wbt_flags flags) +static inline struct rq_qos *wbt_rq_qos(struct request_queue *q) { + return NULL; } -static inline int wbt_init(struct request_queue *q) +static inline void wbt_track(struct request *rq, enum wbt_flags flags) { - return -EINVAL; } static inline void wbt_disable_default(struct request_queue *q) { --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B88C0C433F5 for ; Mon, 10 Jan 2022 09:14:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243303AbiAJJOs (ORCPT ); Mon, 10 Jan 2022 04:14:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242715AbiAJJM3 (ORCPT ); Mon, 10 Jan 2022 04:12:29 -0500 Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1EF7AC061751; Mon, 10 Jan 2022 01:12:14 -0800 (PST) Received: by mail-pg1-x530.google.com with SMTP id v25so10608430pge.2; Mon, 10 Jan 2022 01:12:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mye8NrrngBW+LTEJOPV5rC/ZWn3FGUdytNp+4GKYHYE=; b=WivIciOSaQFS3IEPnDGqfT0VeQwufCr5Cw2IZWjxnUPIFReohYsNsunLXo3v6kLfJv x/nArzg8Y/DMxpnxdQH9SODoNPqPOz8KyHv18qt0tO4LfAyy8MUHh34qZXNE3NhrB1jo Ar493F/EdStR5iGrxtvftkhej8aQ6ag6+BtuEvxXJ8TWtiGA2ZryVrfMw5IbsN1O2ccW ozkEvYIjOZpxrISOZ/Ny8Wslvs1C/addTshfclC28tnH6hKHtEH4/VUNeZZD2twb0GLK AoGPRIwM27GOuc7iuwalW6AFIXRRW9vZn0+YUUtl1VeuyNjLVM87EAhMzi61f+uVmD+Y w7tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mye8NrrngBW+LTEJOPV5rC/ZWn3FGUdytNp+4GKYHYE=; b=gicGX3bqOM5o3VmQ6aXGGm+c3tByWW7G/DXSnVDajMFE+dXg1c1y8Or/aPNBc0PJi2 4tnyMxyea8J9qTyGyy17m1bYb8b0sUgde8yWgJRVMcsbr2XNN6q9xPhRj5oNxdHT8BQP YCIlIDN9uFU0Cj5Nk7A+ewmlbMu6Puq3d+JTprNe5E+JaPNdyvgDtpelt6sZHJy6Q8ty fwSvZARWWwnryMAKEzSOTZSxLXIxgjivInFd1hjDRTiSx06bACNVyY+zv9xib5Vn++ic i/rjOrpCBVinEw1MDGFpFPknKqw7zFY7r2DsvRMuEZE0Ct0WWzRnD/VzCduozUEFZvyG XX+A== X-Gm-Message-State: AOAM533df3TrmhCPHaaucT2VMBZ/tJYqoDZPLrE3wsUQrpJTbmKWZbf5 XW6J3XTOyO2JqZwB1Pbo2DPsNJcxKIX4hn+R X-Google-Smtp-Source: ABdhPJzA0wzm5mUwsMaMKwmPjz/g2UVPNLMf9OkAcZmCzqZLrXVI/9Jp2ru1duFdtlM5AUBpK2xTiw== X-Received: by 2002:a62:5fc4:0:b0:4bb:2472:5147 with SMTP id t187-20020a625fc4000000b004bb24725147mr74946037pfb.74.1641805933690; Mon, 10 Jan 2022 01:12:13 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:13 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/13] blk: export following interfaces Date: Mon, 10 Jan 2022 17:10:36 +0800 Message-Id: <20220110091046.17010-4-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao This is a preparation for making blk-rq-qos policyies modular, there is no functional change. Signed-off-by: Wang Jianchao --- block/bdev.c | 5 ----- block/blk-cgroup.c | 12 ++++++++++++ block/blk-mq-debugfs.c | 2 ++ block/blk-rq-qos.c | 2 ++ block/blk-stat.c | 30 ------------------------------ block/blk-stat.h | 31 ++++++++++++++++++++++++++++--- include/linux/blk-cgroup.h | 1 + include/linux/blkdev.h | 5 ++++- 8 files changed, 49 insertions(+), 39 deletions(-) diff --git a/block/bdev.c b/block/bdev.c index b1d087e5e205..35d8c71be741 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -761,11 +761,6 @@ struct block_device *blkdev_get_no_open(dev_t dev) return bdev; } =20 -void blkdev_put_no_open(struct block_device *bdev) -{ - put_device(&bdev->bd_device); -} - /** * blkdev_get_by_dev - open a block device by device number * @dev: device number of block device to open diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 663aabfeba18..fb56d74f1c8e 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -55,10 +55,18 @@ static struct blkcg_policy *blkcg_policy[BLKCG_MAX_POLS= ]; static LIST_HEAD(all_blkcgs); /* protected by blkcg_pol_mutex */ =20 bool blkcg_debug_stats =3D false; +EXPORT_SYMBOL_GPL(blkcg_debug_stats); + static struct workqueue_struct *blkcg_punt_bio_wq; =20 #define BLKG_DESTROY_BATCH_SIZE 64 =20 +bool blkcg_debug_stats_enabled(void) +{ + return blkcg_debug_stats; +} +EXPORT_SYMBOL_GPL(blkcg_debug_stats_enabled); + static bool blkcg_policy_enabled(struct request_queue *q, const struct blkcg_policy *pol) { @@ -494,6 +502,7 @@ const char *blkg_dev_name(struct blkcg_gq *blkg) return NULL; return bdi_dev_name(blkg->q->disk->bdi); } +EXPORT_SYMBOL_GPL(blkg_dev_name); =20 /** * blkcg_print_blkgs - helper for printing per-blkg data @@ -606,6 +615,7 @@ struct block_device *blkcg_conf_open_bdev(char **inputp) *inputp =3D input; return bdev; } +EXPORT_SYMBOL_GPL(blkcg_conf_open_bdev); =20 /** * blkg_conf_prep - parse and prepare for per-blkg config update @@ -1778,6 +1788,7 @@ void blkcg_schedule_throttle(struct request_queue *q,= bool use_memdelay) current->use_memdelay =3D use_memdelay; set_notify_resume(current); } +EXPORT_SYMBOL_GPL(blkcg_schedule_throttle); =20 /** * blkcg_add_delay - add delay to this blkg @@ -1795,6 +1806,7 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, = u64 delta) blkcg_scale_delay(blkg, now); atomic64_add(delta, &blkg->delay_nsec); } +EXPORT_SYMBOL_GPL(blkcg_add_delay); =20 /** * blkg_tryget_closest - try and get a blkg ref on the closet blkg diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index e225db3c271f..9c786b63c847 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -835,6 +835,7 @@ void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos) debugfs_remove_recursive(rqos->debugfs_dir); rqos->debugfs_dir =3D NULL; } +EXPORT_SYMBOL_GPL(blk_mq_debugfs_unregister_rqos); =20 void blk_mq_debugfs_register_rqos(struct rq_qos *rqos) { @@ -855,6 +856,7 @@ void blk_mq_debugfs_register_rqos(struct rq_qos *rqos) =20 debugfs_create_files(rqos->debugfs_dir, rqos, rqos->ops->debugfs_attrs); } +EXPORT_SYMBOL_GPL(blk_mq_debugfs_register_rqos); =20 void blk_mq_debugfs_unregister_queue_rqos(struct request_queue *q) { diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index a94ff872722b..08ccd4a4e913 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -33,6 +33,7 @@ bool rq_wait_inc_below(struct rq_wait *rq_wait, unsigned = int limit) { return atomic_inc_below(&rq_wait->inflight, limit); } +EXPORT_SYMBOL_GPL(rq_wait_inc_below); =20 void __rq_qos_cleanup(struct rq_qos *rqos, struct bio *bio) { @@ -296,6 +297,7 @@ void rq_qos_wait(struct rq_wait *rqw, void *private_dat= a, } while (1); finish_wait(&rqw->wait, &data.wq); } +EXPORT_SYMBOL_GPL(rq_qos_wait); =20 void rq_qos_exit(struct request_queue *q) { diff --git a/block/blk-stat.c b/block/blk-stat.c index ae3dd1fb8e61..2b0c530b0c9c 100644 --- a/block/blk-stat.c +++ b/block/blk-stat.c @@ -18,36 +18,6 @@ struct blk_queue_stats { bool enable_accounting; }; =20 -void blk_rq_stat_init(struct blk_rq_stat *stat) -{ - stat->min =3D -1ULL; - stat->max =3D stat->nr_samples =3D stat->mean =3D 0; - stat->batch =3D 0; -} - -/* src is a per-cpu stat, mean isn't initialized */ -void blk_rq_stat_sum(struct blk_rq_stat *dst, struct blk_rq_stat *src) -{ - if (!src->nr_samples) - return; - - dst->min =3D min(dst->min, src->min); - dst->max =3D max(dst->max, src->max); - - dst->mean =3D div_u64(src->batch + dst->mean * dst->nr_samples, - dst->nr_samples + src->nr_samples); - - dst->nr_samples +=3D src->nr_samples; -} - -void blk_rq_stat_add(struct blk_rq_stat *stat, u64 value) -{ - stat->min =3D min(stat->min, value); - stat->max =3D max(stat->max, value); - stat->batch +=3D value; - stat->nr_samples++; -} - void blk_stat_add(struct request *rq, u64 now) { struct request_queue *q =3D rq->q; diff --git a/block/blk-stat.h b/block/blk-stat.h index 17b47a86eefb..2642969594d8 100644 --- a/block/blk-stat.h +++ b/block/blk-stat.h @@ -164,8 +164,33 @@ static inline void blk_stat_activate_msecs(struct blk_= stat_callback *cb, mod_timer(&cb->timer, jiffies + msecs_to_jiffies(msecs)); } =20 -void blk_rq_stat_add(struct blk_rq_stat *, u64); -void blk_rq_stat_sum(struct blk_rq_stat *, struct blk_rq_stat *); -void blk_rq_stat_init(struct blk_rq_stat *); +static inline void blk_rq_stat_init(struct blk_rq_stat *stat) +{ + stat->min =3D -1ULL; + stat->max =3D stat->nr_samples =3D stat->mean =3D 0; + stat->batch =3D 0; +} + +/* src is a per-cpu stat, mean isn't initialized */ +static inline void blk_rq_stat_sum(struct blk_rq_stat *dst, struct blk_rq_= stat *src) +{ + if (!src->nr_samples) + return; + + dst->min =3D min(dst->min, src->min); + dst->max =3D max(dst->max, src->max); =20 + dst->mean =3D div_u64(src->batch + dst->mean * dst->nr_samples, + dst->nr_samples + src->nr_samples); + + dst->nr_samples +=3D src->nr_samples; +} + +static inline void blk_rq_stat_add(struct blk_rq_stat *stat, u64 value) +{ + stat->min =3D min(stat->min, value); + stat->max =3D max(stat->max, value); + stat->batch +=3D value; + stat->nr_samples++; +} #endif diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index b4de2010fba5..b87a1bdde675 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -179,6 +179,7 @@ struct blkcg_policy { extern struct blkcg blkcg_root; extern struct cgroup_subsys_state * const blkcg_root_css; extern bool blkcg_debug_stats; +bool blkcg_debug_stats_enabled(void); =20 struct blkcg_gq *blkg_lookup_slowpath(struct blkcg *blkcg, struct request_queue *q, bool update_hint); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index e7dce2232814..ed30b3c3fee7 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1297,7 +1297,10 @@ void blkdev_put(struct block_device *bdev, fmode_t m= ode); =20 /* just for blk-cgroup, don't use elsewhere */ struct block_device *blkdev_get_no_open(dev_t dev); -void blkdev_put_no_open(struct block_device *bdev); +static inline void blkdev_put_no_open(struct block_device *bdev) +{ + put_device(&bdev->bd_device); +} =20 struct block_device *bdev_alloc(struct gendisk *disk, u8 partno); void bdev_add(struct block_device *bdev, dev_t dev); --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E937C433F5 for ; Mon, 10 Jan 2022 09:14:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243282AbiAJJOp (ORCPT ); Mon, 10 Jan 2022 04:14:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242724AbiAJJMb (ORCPT ); Mon, 10 Jan 2022 04:12:31 -0500 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91E4BC061756; Mon, 10 Jan 2022 01:12:16 -0800 (PST) Received: by mail-pj1-x102e.google.com with SMTP id r16-20020a17090a0ad000b001b276aa3aabso21771817pje.0; Mon, 10 Jan 2022 01:12:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BKTMUEDPmhQm52YOgdwkcpYTtV0KPgGhD+PMcaIjPHo=; b=i8Lo0Yw1U8tKhsezqg+GEtufsEuozr5Xsl3OySgcCoXxFzRoFHOwxGT9wyvEeaymVs GWDs/N6TUlNfrFfU/3uCybGeCeRYPHAvsnRt9FI8ELikS2W68ccrDert0r58e28Ic/nP 24n9sT4BCgqrO7337rwgp7sBmKvK2rwr97NWW/M1YzwUV23EZMT/ILzv9/PMvMs8lto8 KyMXV53IAsbmmu86sTEj0QIt5yoZRGLSm7Xi8IYN/6isqfM44Q+iFMuO6KSxf2qFHsx5 zrHenpPDRJZGR2+XncacuRkcI3WCwkVBONesHznbVGmansTWS7m3k8Tt86gjpm0mBkaK tX7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BKTMUEDPmhQm52YOgdwkcpYTtV0KPgGhD+PMcaIjPHo=; b=1mRC5mKL2TzQHIjQuSapxmVZYB+Pm89tJIYalxG4bKf4VJwXBTukFoua45z67Hb06V fxEeIlta/1Jy2bBfXTmsl1sjGPNE0d+WCh0NNULyRxOCfo9cwL6gaYjUhcoP40gRYiJP YGatHSnxzh7eG9rJc52Ld5bPLM4RjRUEStZGhe2k4Wf+Dcxma+iscXf4gtFeTbdG8CC3 dumE6n6yonv4LRmWCd/dwjfpPEQKx33J3+FUfs5W1c0h20Qq1vO34Fa8YHkIZi3MyHRq F4fMbw28C5QSM9bFUbctZJW0qB42t4fRRfC3/L/xetjVnJHh4FIj0oAzrfnbc8hU0PXZ D5eg== X-Gm-Message-State: AOAM531Ua32qr2aECEBHYBbM/z+UcrIROKo/8T0+/XpiV0sgbmUthcoN 0W5NJ0BcIfFkafdrU8MJON4= X-Google-Smtp-Source: ABdhPJxkPQeT/5kSNCySOOU6xiXE6T4y8PsZtgghWGm9d/n6HyyARjeVFsxfXNFusu+/510EGjgTDQ== X-Received: by 2002:a17:90b:4a48:: with SMTP id lb8mr476251pjb.54.1641805936211; Mon, 10 Jan 2022 01:12:16 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:15 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/13] cgroup: export following two interfaces Date: Mon, 10 Jan 2022 17:10:37 +0800 Message-Id: <20220110091046.17010-5-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao This is a preparation for making blk-rq-qos modular, there is no functional change, but just export interfaces pr_cont_cgroup_path and cgroup_parse_float. Signed-off-by: Wang Jianchao --- include/linux/cgroup.h | 5 +---- kernel/cgroup/cgroup.c | 7 +++++++ 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 75c151413fda..1a67b0db00db 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -666,10 +666,7 @@ static inline void pr_cont_cgroup_name(struct cgroup *= cgrp) pr_cont_kernfs_name(cgrp->kn); } =20 -static inline void pr_cont_cgroup_path(struct cgroup *cgrp) -{ - pr_cont_kernfs_path(cgrp->kn); -} +void pr_cont_cgroup_path(struct cgroup *cgrp); =20 static inline struct psi_group *cgroup_psi(struct cgroup *cgrp) { diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 919194de39c8..f358d5122033 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6597,6 +6597,13 @@ int cgroup_parse_float(const char *input, unsigned d= ec_shift, s64 *v) *v =3D whole * power_of_ten(dec_shift) + frac; return 0; } +EXPORT_SYMBOL_GPL(cgroup_parse_float); + +void pr_cont_cgroup_path(struct cgroup *cgrp) +{ + pr_cont_kernfs_path(cgrp->kn); +} +EXPORT_SYMBOL_GPL(pr_cont_cgroup_path); =20 /* * sock->sk_cgrp_data handling. For more info, see sock_cgroup_data --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E2BCC433EF for ; Mon, 10 Jan 2022 09:14:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243243AbiAJJOh (ORCPT ); Mon, 10 Jan 2022 04:14:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242563AbiAJJMb (ORCPT ); Mon, 10 Jan 2022 04:12:31 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16189C061757; Mon, 10 Jan 2022 01:12:19 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id m13so12667588pji.3; Mon, 10 Jan 2022 01:12:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=4lE5PHQYmUvEqvSpisgi3VasZwT8tgz/FJ17TpKqMMY=; b=hYISV7/FT/xXa9WAfxYaQavkfEd79VcEhIpJzFbQZnLGItS+7h2qG0tmAQA3UQA+Gg 4jnPoex++S4yCyOKFmUUBPRN0aRobJkhbW4cNfI4Q3xwMrfj2yqyvlb9wc4gIv5wmDFf bibZN2gN0UwJd8GErMrdxAdj8iDWHyeSX9tK8Ka3TEnUor7HUapIMwO94pvA91DFfoQc iT736brWPhy4FeQEESZpjBXNkMmbh1QH8sKNtJUIQX2GIp9mOLYRKuBLA6gvvfKsbNkP 76vmevuyPLHkVxjo/1NsLDRZSDMFJ/3DBVe5SDNaS/gpBKzTfRlfIiyNA47sULI94NRl wVuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=4lE5PHQYmUvEqvSpisgi3VasZwT8tgz/FJ17TpKqMMY=; b=HM765KrvcBM97pKC0VQ5G7F93i2yFW1T9FwSPxyGcBm46hOgUv1nQwd8P+LL5+dYnr kqJp5q/L+2SpJFvBbS5qkyTBiqBRwkClOZKH/xZ1Q33oY5kjORXdoXBVpaIuqOcOPuCj I7UTG5rnsWPw6m/Qlb8ni/i2xoKe6WyFThLm2Tmn8MvVoT/DMeePeeAjDIUWYNhe9lYm A7OcMJstT/wv6V9DTOirxYMSIkZqtYo98WaSZerUia4KR/1hUql/c4sVcjEIb0ig6dxN eVSUGOnLEKjByH4lX4Q1W8opbvdv3RZQ29foGyCE/HDm5xVEGSWYjgM47PfzIhqX5IjT S+jg== X-Gm-Message-State: AOAM532g24gQ2Sk6TB2y0+SCuP5nq6gzVqZbJR6vXqSVpP9t8UQDsUGe rMsD9S4R1HCu2PCLDVzltZQ= X-Google-Smtp-Source: ABdhPJzVCbbXY+GPZ285lAOvjIJJgS49A7vVnkOb4wPTmUy8WPlrMcdMWKhLvEPuP5tAW107PE7tOg== X-Received: by 2002:a17:903:3015:b0:14a:2206:8cdc with SMTP id o21-20020a170903301500b0014a22068cdcmr7666910pla.55.1641805938642; Mon, 10 Jan 2022 01:12:18 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:18 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/13] blk-iolatency: make iolatency pluggable and modular Date: Mon, 10 Jan 2022 17:10:38 +0800 Message-Id: <20220110091046.17010-6-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao Make blk-iolatency pluggable and modular. Then we can close or open it through /sys/block/xxx/queue/qos and rmmod the module if we don't need it which can release one blkcg policy slot. Signed-off-by: Wang Jianchao --- block/Kconfig | 2 +- block/Makefile | 4 +++- block/blk-cgroup.c | 6 ------ block/blk-iolatency.c | 39 +++++++++++++++++++++++++++++++-------- block/blk-mq-debugfs.c | 2 -- block/blk-rq-qos.h | 6 ------ block/blk.h | 6 ------ 7 files changed, 35 insertions(+), 30 deletions(-) diff --git a/block/Kconfig b/block/Kconfig index c6ce41a5e5b2..1c0d05df2aec 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -111,7 +111,7 @@ config BLK_WBT_MQ Enable writeback throttling by default for request-based block devices. =20 config BLK_CGROUP_IOLATENCY - bool "Enable support for latency based cgroup IO protection" + tristate "Enable support for latency based cgroup IO protection" depends on BLK_CGROUP help Enabling this option enables the .latency interface for IO throttling. diff --git a/block/Makefile b/block/Makefile index 44df57e562bf..ccf61c57e1d4 100644 --- a/block/Makefile +++ b/block/Makefile @@ -18,7 +18,9 @@ obj-$(CONFIG_BLK_CGROUP) +=3D blk-cgroup.o obj-$(CONFIG_BLK_CGROUP_RWSTAT) +=3D blk-cgroup-rwstat.o obj-$(CONFIG_BLK_DEV_THROTTLING) +=3D blk-throttle.o obj-$(CONFIG_BLK_CGROUP_IOPRIO) +=3D blk-ioprio.o -obj-$(CONFIG_BLK_CGROUP_IOLATENCY) +=3D blk-iolatency.o +iolat-y :=3D blk-iolatency.o +obj-$(CONFIG_BLK_CGROUP_IOLATENCY) +=3D iolat.o + obj-$(CONFIG_BLK_CGROUP_IOCOST) +=3D blk-iocost.o obj-$(CONFIG_MQ_IOSCHED_DEADLINE) +=3D mq-deadline.o obj-$(CONFIG_MQ_IOSCHED_KYBER) +=3D kyber-iosched.o diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index fb56d74f1c8e..fd874dfd38ed 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1212,12 +1212,6 @@ int blkcg_init_queue(struct request_queue *q) if (ret) goto err_destroy_all; =20 - ret =3D blk_iolatency_init(q); - if (ret) { - blk_throtl_exit(q); - goto err_destroy_all; - } - return 0; =20 err_destroy_all: diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 6593c7123b97..6aaf0775e484 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -90,6 +90,12 @@ struct blk_iolatency { atomic_t enabled; }; =20 +static struct rq_qos_ops blkcg_iolatency_ops; +static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q) +{ + return rq_qos_by_id(q, blkcg_iolatency_ops.id); +} + static inline struct blk_iolatency *BLKIOLATENCY(struct rq_qos *rqos) { return container_of(rqos, struct blk_iolatency, rqos); @@ -646,13 +652,21 @@ static void blkcg_iolatency_exit(struct rq_qos *rqos) =20 del_timer_sync(&blkiolat->timer); blkcg_deactivate_policy(rqos->q, &blkcg_policy_iolatency); + rq_qos_deactivate(rqos); kfree(blkiolat); } =20 +static int blk_iolatency_init(struct request_queue *q); static struct rq_qos_ops blkcg_iolatency_ops =3D { +#if IS_MODULE(CONFIG_BLK_CGROUP_IOLATENCY) + .owner =3D THIS_MODULE, +#endif + .name =3D "iolat", + .flags =3D RQOS_FLAG_CGRP_POL, .throttle =3D blkcg_iolatency_throttle, .done_bio =3D blkcg_iolatency_done_bio, .exit =3D blkcg_iolatency_exit, + .init =3D blk_iolatency_init, }; =20 static void blkiolatency_timer_fn(struct timer_list *t) @@ -727,15 +741,10 @@ int blk_iolatency_init(struct request_queue *q) return -ENOMEM; =20 rqos =3D &blkiolat->rqos; - rqos->id =3D RQ_QOS_LATENCY; - rqos->ops =3D &blkcg_iolatency_ops; - rqos->q =3D q; - - rq_qos_add(q, rqos); - + rq_qos_activate(q, rqos, &blkcg_iolatency_ops); ret =3D blkcg_activate_policy(q, &blkcg_policy_iolatency); if (ret) { - rq_qos_del(q, rqos); + rq_qos_deactivate(rqos); kfree(blkiolat); return ret; } @@ -1046,13 +1055,27 @@ static struct blkcg_policy blkcg_policy_iolatency = =3D { =20 static int __init iolatency_init(void) { - return blkcg_policy_register(&blkcg_policy_iolatency); + int ret; + + ret =3D rq_qos_register(&blkcg_iolatency_ops); + if (ret) + return ret; + + ret =3D blkcg_policy_register(&blkcg_policy_iolatency); + if (ret) + rq_qos_unregister(&blkcg_iolatency_ops); + + return ret; } =20 static void __exit iolatency_exit(void) { blkcg_policy_unregister(&blkcg_policy_iolatency); + rq_qos_unregister(&blkcg_iolatency_ops); } =20 module_init(iolatency_init); module_exit(iolatency_exit); +MODULE_AUTHOR("Josef Bacik"); +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Latency based cgroup IO protection"); diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 9c786b63c847..57c33f4730f2 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -820,8 +820,6 @@ void blk_mq_debugfs_unregister_sched(struct request_que= ue *q) static const char *rq_qos_id_to_name(enum rq_qos_id id) { switch (id) { - case RQ_QOS_LATENCY: - return "latency"; case RQ_QOS_COST: return "cost"; case RQ_QOS_IOPRIO: diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index de82eb951bdd..6ca46c69e325 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -14,7 +14,6 @@ struct blk_mq_debugfs_attr; =20 enum rq_qos_id { - RQ_QOS_LATENCY, RQ_QOS_COST, RQ_QOS_IOPRIO, }; @@ -85,11 +84,6 @@ static inline struct rq_qos *rq_qos_by_id(struct request= _queue *q, int id) return rqos; } =20 -static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q) -{ - return rq_qos_by_id(q, RQ_QOS_LATENCY); -} - static inline void rq_wait_init(struct rq_wait *rq_wait) { atomic_set(&rq_wait->inflight, 0); diff --git a/block/blk.h b/block/blk.h index ccde6e6f1736..e2e4fbb9a58d 100644 --- a/block/blk.h +++ b/block/blk.h @@ -436,12 +436,6 @@ static inline void blk_queue_bounce(struct request_que= ue *q, struct bio **bio) __blk_queue_bounce(q, bio);=09 } =20 -#ifdef CONFIG_BLK_CGROUP_IOLATENCY -extern int blk_iolatency_init(struct request_queue *q); -#else -static inline int blk_iolatency_init(struct request_queue *q) { return 0; } -#endif - struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp= ); =20 #ifdef CONFIG_BLK_DEV_ZONED --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CE3FC433F5 for ; Mon, 10 Jan 2022 09:14:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243111AbiAJJOE (ORCPT ); Mon, 10 Jan 2022 04:14:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242619AbiAJJMe (ORCPT ); Mon, 10 Jan 2022 04:12:34 -0500 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DEAEC034000; Mon, 10 Jan 2022 01:12:21 -0800 (PST) Received: by mail-pl1-x62d.google.com with SMTP id q14so11283658plx.4; Mon, 10 Jan 2022 01:12:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=l3XXcURerpYJD7VuNrikwmg4jwTKIhd5e7FpXq8T7WM=; b=fHcMn2z/pwtFurWbFk+PvvXhhsECkSUJRSeDouKx2K3qSyHOeQCx0ODMAbPN8El0QP telhh3eIadEfgXcfabxrMoHxBWtwQEDxUFPnHSUoMYuQjl7Q/wKbKzg/vr3frL4TnMCK 1F9bYuidAwClzGUOBSFhx48y84eJB2JCeH3ykYA9nnFa4i1cqitYiY5aV7j0FvaHbI/W ymVmZM5lJT1KN0Yzwazb/8vV/3HpufljjxhuMgjMHnC9PKYGKkExWuZUjl27XukI79gK 04kHEA5/0klFJaYbrotzHyc2S+sMq/B8Q2u6qA7ngHmvghmRfFHsU9qkkkEec+Km4ERj KY/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=l3XXcURerpYJD7VuNrikwmg4jwTKIhd5e7FpXq8T7WM=; b=3vKVTuv8qYCyY0nzVkGe1Z0clwKDN92burt3D/3Z3uwwmgh0fy8oF47DsvrIfBT/NE T3c5t+mnCoeBSC8SubLak9QqNojAceQRvXNwVhcZ4Oow/nXvpQmSKO0re0h+pJALwDWn bnXLqTlwnZ3ATk9xfaAqTRpoQd56EajWe9DQiA3H0hc4702QiS/DiQFxLI661GWlc6sf +2sp8RkZBI5MO1hb5I1hvdL8cBhYJTt6esOB8cRuEGqOqNHsQ4v37uT1+aUOyXc2k0B2 tq1V8tdCXTEW4+6QzIob9xUM0q4jHbKHyp6LtaK6t2t8InKB5+GcQufC8a3AFmU30RkL 4CEA== X-Gm-Message-State: AOAM530GSTjZ2hYBxSPB+0vUICGMTfkgx5LQv3J3QJ7Y5QfVShTVmijc TnJRXY6dlWFxRxUa67Db3fpdPLVqNPRM7AH7 X-Google-Smtp-Source: ABdhPJyvtR5CwApsfmKKBwNyqfQ/icTdLXB/W5BcF2uOaQo9PRlbBZ3mJELp6oUzlr39glr+NV+RKg== X-Received: by 2002:a17:90a:bb91:: with SMTP id v17mr28008022pjr.238.1641805941140; Mon, 10 Jan 2022 01:12:21 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:20 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/13] blk: remove unused BLK_RQ_IO_DATA_LEN Date: Mon, 10 Jan 2022 17:10:39 +0800 Message-Id: <20220110091046.17010-7-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao Remove it as nobody use it any more. Signed-off-by: Wang Jianchao --- block/Kconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/block/Kconfig b/block/Kconfig index 1c0d05df2aec..50cc1b56852c 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -133,7 +133,6 @@ config BLK_CGROUP_FC_APPID config BLK_CGROUP_IOCOST bool "Enable support for cost model based cgroup IO controller" depends on BLK_CGROUP - select BLK_RQ_IO_DATA_LEN select BLK_RQ_ALLOC_TIME help Enabling this option enables the .weight interface for cost --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21419C433EF for ; Mon, 10 Jan 2022 09:14:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243167AbiAJJOS (ORCPT ); Mon, 10 Jan 2022 04:14:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242625AbiAJJMe (ORCPT ); Mon, 10 Jan 2022 04:12:34 -0500 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07AE8C034001; Mon, 10 Jan 2022 01:12:24 -0800 (PST) Received: by mail-pj1-x1031.google.com with SMTP id pj2so11473542pjb.2; Mon, 10 Jan 2022 01:12:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MA5aJkYexI+3gxXmnqrDt6y1iMXmwH2sipH7h3l2T/8=; b=LAyP3ZHXITzhDzYrLuXKkTVxF00+FnKh0EFu7O1GsRVgkdsWl6NC4I3bYui1OB5PiA vmQ360Da6YlViHz3ighBuNkBJ0JCefSa91GppCG/0whdjDywOCmZX5to/KbPnrA4ZWsi fDbE4xsTDoB7Iqo/vH2YUYN+RjhjcS67evELHnK5yAgm3DvswXe/903yrLDpxmnJD2o9 S77ewUuBzlSFF0X+zUyCs0f2MXISJ9r7mYzyYUJ1vKexRvVmFeVLgXGkVal9mP115phl Evl0luNiizt7Ce2ClFW1AroanadHcdWp5t4oxl2SXkUuozYufxvdrsn8DpLD+FgNYk1G ANBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MA5aJkYexI+3gxXmnqrDt6y1iMXmwH2sipH7h3l2T/8=; b=p4x6/3ed0CjTgk47/YyT/2LqO8kfUufTPxdBKUC3553FxCiT3AXrtXT8/rWWaXplrC ZDMKN51p7fvK5EOLCSoWSz9d7/NgenjtuuMkiHPsdZyzLHybxnDbdSvVCeVvov2TgUZX QIOXYfQuDlVt7OYJcdqiClurGH1nEB2m9hutP9xQXI3uI9mecav3dzvTAHSLJ1yIU78L ULpXJU+rNs060exgUU81mFjvMQTRMg8RkNRM+v1YhIzqv+MoNLUnAfQx72gx9HsW4UbO FyqXPbOP3h1v1/gTeSM+WjaFwuW+yVam5tP8NNRAvKoGMsOoXPqXPJSSfTPJ42S0dhYA EDRA== X-Gm-Message-State: AOAM533GRpJOoKUaG1ivfVfJkzGOuWFMfPVU2xetDpfEzANUMH6HiaCF PMxXQg7iC987h0T9OijT0Ls= X-Google-Smtp-Source: ABdhPJyzY1KDd5ZB8tkJakiVIfPqn83Mo3LLd11awzreh+qXGyl17XBw7GTo2pOGUP0HY4IKCzzAdw== X-Received: by 2002:a17:90a:4489:: with SMTP id t9mr28782900pjg.223.1641805943631; Mon, 10 Jan 2022 01:12:23 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:23 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/13] blk: use standalone macro to control bio.bi_iocost_cost Date: Mon, 10 Jan 2022 17:10:40 +0800 Message-Id: <20220110091046.17010-8-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao This is a preparation to make iocost modular Signed-off-by: Wang Jianchao --- block/Kconfig | 4 ++++ block/bio.c | 2 +- include/linux/blk_types.h | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/block/Kconfig b/block/Kconfig index 50cc1b56852c..e1b1bff5c1e9 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -26,6 +26,9 @@ menuconfig BLOCK =20 if BLOCK =20 +config BLK_BIO_IOCOST + bool + config BLK_RQ_ALLOC_TIME bool =20 @@ -134,6 +137,7 @@ config BLK_CGROUP_IOCOST bool "Enable support for cost model based cgroup IO controller" depends on BLK_CGROUP select BLK_RQ_ALLOC_TIME + select BLK_BIO_IOCOST help Enabling this option enables the .weight interface for cost model based proportional IO control. The IO controller diff --git a/block/bio.c b/block/bio.c index 15ab0d6d1c06..a9e2347b0021 100644 --- a/block/bio.c +++ b/block/bio.c @@ -268,7 +268,7 @@ void bio_init(struct bio *bio, struct bio_vec *table, #ifdef CONFIG_BLK_CGROUP bio->bi_blkg =3D NULL; bio->bi_issue.value =3D 0; -#ifdef CONFIG_BLK_CGROUP_IOCOST +#ifdef CONFIG_BLK_BIO_IOCOST bio->bi_iocost_cost =3D 0; #endif #endif diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index fe065c394fff..495ffc29bab0 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -261,7 +261,7 @@ struct bio { */ struct blkcg_gq *bi_blkg; struct bio_issue bi_issue; -#ifdef CONFIG_BLK_CGROUP_IOCOST +#ifdef CONFIG_BLK_BIO_IOCOST u64 bi_iocost_cost; #endif #endif --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C89FC433FE for ; Mon, 10 Jan 2022 09:13:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242759AbiAJJNL (ORCPT ); Mon, 10 Jan 2022 04:13:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242630AbiAJJMe (ORCPT ); Mon, 10 Jan 2022 04:12:34 -0500 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F525C034003; Mon, 10 Jan 2022 01:12:26 -0800 (PST) Received: by mail-pj1-x1035.google.com with SMTP id r16-20020a17090a0ad000b001b276aa3aabso21772344pje.0; Mon, 10 Jan 2022 01:12:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QjwJsbeo9uruskPn1+c6wYleKO50y4VIL3hXZvxsfXw=; b=pDEp4cMPc15aOxBiVGfQ2kYpyXV6sApSCvQAi/bjfVt9RR3uLDYSYYlC/7oQAKXkPR nJ/aXZXEtNncN/QuW9J1PG4g/BilOgBqAkfPNxKDKG+oYkpsUKrrka57SEjvKk9+h3PG 8oMG6CSirJW+sfC8a4V6OEeTcT04bKq6jm/9CoF77mqhtaZW+UqBzhYuTZ7rJ6xbvYjv U7C8AnQmm//Qs41GGwjdpYARTkn7uWda7bFphipUQKedULWkrJOXyLCNapyQSbr3WAj6 XnvqACxcdjtQLrBNDBOPztnTjBS63qVWqSugAD7joAPTYZZBV4J8T8RJvQypIUkFf0bs H3+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=QjwJsbeo9uruskPn1+c6wYleKO50y4VIL3hXZvxsfXw=; b=B4fM7kDiEKj7SCevx/M9oawwaHO5xdDbDoxFGbOh0ZOAFhl8QPmv5FL5r9Qg90PQB4 +6CDAtRe8q6uVd6e7NDiIGPHR60pWj9pJ8mdrQ8nRzqob9lvghE/mEL/q2sqegNh4Soo 2EaBqPaJo8fInbzKElSUzqe6I1OpswcGgCpELycXYwp2I2pLvtUO7ywj/Q1uLqRV6iVj UEll+mZyyYKutOZNNvikEPvp4drhRe6S/lb079VCuh0Pdze5OXLshb56QXOo/mJRo2Mh e4NQidrBxKnfAMqXUXoYVkLKOQFchCknIQCUqZa6HF+yol/5zZ1U3P5dQ5k324vo1dX/ KXlQ== X-Gm-Message-State: AOAM530jLzugSXn5viF96tsl9HlXiOpZsRO71/gT5dC7kHcBFej0humi aNdNY0HAy5o2kjg4LyXpxcA= X-Google-Smtp-Source: ABdhPJzsX9cn3Gn+g4gyD5G4S35HOBCNDc3n08Ncy9yx+YyV808IIIbCHLq6OizQJbZ/8SJTD9K4Eg== X-Received: by 2002:a17:902:e74e:b0:149:7c20:c69b with SMTP id p14-20020a170902e74e00b001497c20c69bmr64501735plf.82.1641805946151; Mon, 10 Jan 2022 01:12:26 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:25 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/13] blk-iocost: make iocost pluggable and modular Date: Mon, 10 Jan 2022 17:10:41 +0800 Message-Id: <20220110091046.17010-9-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao Make blk-iocost pluggable and modular. Then we can close or open it through /sys/block/xxx/queue/qos and rmmod the module if we don't need it which can release one blkcg policy slot. Signed-off-by: Wang Jianchao --- block/Kconfig | 2 +- block/Makefile | 4 ++-- block/blk-iocost.c | 53 ++++++++++++++++++++++++++---------------- block/blk-mq-debugfs.c | 2 -- block/blk-rq-qos.h | 1 - 5 files changed, 36 insertions(+), 26 deletions(-) diff --git a/block/Kconfig b/block/Kconfig index e1b1bff5c1e9..3e1a3487b55a 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -134,7 +134,7 @@ config BLK_CGROUP_FC_APPID application specific identification into the FC frame. =20 config BLK_CGROUP_IOCOST - bool "Enable support for cost model based cgroup IO controller" + tristate "Enable support for cost model based cgroup IO controller" depends on BLK_CGROUP select BLK_RQ_ALLOC_TIME select BLK_BIO_IOCOST diff --git a/block/Makefile b/block/Makefile index ccf61c57e1d4..8950913cbcc9 100644 --- a/block/Makefile +++ b/block/Makefile @@ -20,8 +20,8 @@ obj-$(CONFIG_BLK_DEV_THROTTLING) +=3D blk-throttle.o obj-$(CONFIG_BLK_CGROUP_IOPRIO) +=3D blk-ioprio.o iolat-y :=3D blk-iolatency.o obj-$(CONFIG_BLK_CGROUP_IOLATENCY) +=3D iolat.o - -obj-$(CONFIG_BLK_CGROUP_IOCOST) +=3D blk-iocost.o +iocost-y :=3D blk-iocost.o +obj-$(CONFIG_BLK_CGROUP_IOCOST) +=3D iocost.o obj-$(CONFIG_MQ_IOSCHED_DEADLINE) +=3D mq-deadline.o obj-$(CONFIG_MQ_IOSCHED_KYBER) +=3D kyber-iosched.o bfq-y :=3D bfq-iosched.o bfq-wf2q.o bfq-cgroup.o diff --git a/block/blk-iocost.c b/block/blk-iocost.c index cfc0e305c32e..afa52354d42b 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -660,9 +660,10 @@ static struct ioc *rqos_to_ioc(struct rq_qos *rqos) return container_of(rqos, struct ioc, rqos); } =20 +static struct rq_qos_ops ioc_rqos_ops; static struct ioc *q_to_ioc(struct request_queue *q) { - return rqos_to_ioc(rq_qos_by_id(q, RQ_QOS_COST)); + return rqos_to_ioc(rq_qos_by_id(q, ioc_rqos_ops.id)); } =20 static const char *q_name(struct request_queue *q) @@ -2810,6 +2811,7 @@ static void ioc_rqos_exit(struct rq_qos *rqos) struct ioc *ioc =3D rqos_to_ioc(rqos); =20 blkcg_deactivate_policy(rqos->q, &blkcg_policy_iocost); + rq_qos_deactivate(rqos); =20 spin_lock_irq(&ioc->lock); ioc->running =3D IOC_STOP; @@ -2820,13 +2822,20 @@ static void ioc_rqos_exit(struct rq_qos *rqos) kfree(ioc); } =20 +static int blk_iocost_init(struct request_queue *q); static struct rq_qos_ops ioc_rqos_ops =3D { +#if IS_MODULE(CONFIG_BLK_CGROUP_IOCOST) + .owner =3D THIS_MODULE, +#endif + .name =3D "iocost", + .flags =3D RQOS_FLAG_CGRP_POL | RQOS_FLAG_RQ_ALLOC_TIME, .throttle =3D ioc_rqos_throttle, .merge =3D ioc_rqos_merge, .done_bio =3D ioc_rqos_done_bio, .done =3D ioc_rqos_done, .queue_depth_changed =3D ioc_rqos_queue_depth_changed, .exit =3D ioc_rqos_exit, + .init =3D blk_iocost_init, }; =20 static int blk_iocost_init(struct request_queue *q) @@ -2856,10 +2865,7 @@ static int blk_iocost_init(struct request_queue *q) } =20 rqos =3D &ioc->rqos; - rqos->id =3D RQ_QOS_COST; - rqos->ops =3D &ioc_rqos_ops; - rqos->q =3D q; - + rq_qos_activate(q, rqos, &ioc_rqos_ops); spin_lock_init(&ioc->lock); timer_setup(&ioc->timer, ioc_timer_fn, 0); INIT_LIST_HEAD(&ioc->active_iocgs); @@ -2883,10 +2889,9 @@ static int blk_iocost_init(struct request_queue *q) * called before policy activation completion, can't assume that the * target bio has an iocg associated and need to test for NULL iocg. */ - rq_qos_add(q, rqos); ret =3D blkcg_activate_policy(q, &blkcg_policy_iocost); if (ret) { - rq_qos_del(q, rqos); + rq_qos_deactivate(rqos); free_percpu(ioc->pcpu_stat); kfree(ioc); return ret; @@ -3173,12 +3178,10 @@ static ssize_t ioc_qos_write(struct kernfs_open_fil= e *of, char *input, if (IS_ERR(bdev)) return PTR_ERR(bdev); =20 - rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); + rqos =3D rq_qos_get(bdev_get_queue(bdev), ioc_rqos_ops.id); if (!rqos) { - ret =3D blk_iocost_init(bdev_get_queue(bdev)); - if (ret) - goto err; - rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); + ret =3D -EOPNOTSUPP; + goto err; } =20 ioc =3D rqos_to_ioc(rqos); @@ -3257,10 +3260,8 @@ static ssize_t ioc_qos_write(struct kernfs_open_file= *of, char *input, =20 if (enable) { blk_stat_enable_accounting(ioc->rqos.q); - blk_queue_flag_set(QUEUE_FLAG_RQ_ALLOC_TIME, ioc->rqos.q); ioc->enabled =3D true; } else { - blk_queue_flag_clear(QUEUE_FLAG_RQ_ALLOC_TIME, ioc->rqos.q); ioc->enabled =3D false; } =20 @@ -3344,12 +3345,10 @@ static ssize_t ioc_cost_model_write(struct kernfs_o= pen_file *of, char *input, if (IS_ERR(bdev)) return PTR_ERR(bdev); =20 - rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); + rqos =3D rq_qos_get(bdev_get_queue(bdev), ioc_rqos_ops.id); if (!ioc) { - ret =3D blk_iocost_init(bdev_get_queue(bdev)); - if (ret) - goto err; - rqos =3D rq_qos_get(bdev_get_queue(bdev), RQ_QOS_COST); + ret =3D -EOPNOTSUPP; + goto err; } =20 ioc =3D rqos_to_ioc(rqos); @@ -3449,13 +3448,27 @@ static struct blkcg_policy blkcg_policy_iocost =3D { =20 static int __init ioc_init(void) { - return blkcg_policy_register(&blkcg_policy_iocost); + int ret; + + ret =3D rq_qos_register(&ioc_rqos_ops); + if (ret) + return ret; + + ret =3D blkcg_policy_register(&blkcg_policy_iocost); + if (ret) + rq_qos_unregister(&ioc_rqos_ops); + + return ret; } =20 static void __exit ioc_exit(void) { blkcg_policy_unregister(&blkcg_policy_iocost); + rq_qos_unregister(&ioc_rqos_ops); } =20 module_init(ioc_init); module_exit(ioc_exit); +MODULE_AUTHOR("Tejun Heo"); +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Cost model based cgroup IO controller"); diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 57c33f4730f2..14fda9a5e552 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -820,8 +820,6 @@ void blk_mq_debugfs_unregister_sched(struct request_que= ue *q) static const char *rq_qos_id_to_name(enum rq_qos_id id) { switch (id) { - case RQ_QOS_COST: - return "cost"; case RQ_QOS_IOPRIO: return "ioprio"; } diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index 6ca46c69e325..4eef53f2c290 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -14,7 +14,6 @@ struct blk_mq_debugfs_attr; =20 enum rq_qos_id { - RQ_QOS_COST, RQ_QOS_IOPRIO, }; =20 --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A23CBC433EF for ; Mon, 10 Jan 2022 09:14:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243123AbiAJJOG (ORCPT ); Mon, 10 Jan 2022 04:14:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242685AbiAJJMe (ORCPT ); Mon, 10 Jan 2022 04:12:34 -0500 Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19625C06175A; Mon, 10 Jan 2022 01:12:29 -0800 (PST) Received: by mail-pg1-x52f.google.com with SMTP id v25so10608824pge.2; Mon, 10 Jan 2022 01:12:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5uGAFhwG1PalEv/hF+X/ZtSoGm0z/W3W27BE5YAO4Hk=; b=HclcVXbL/P/hD+zMRRRq3PMc7Z0zkztb27qGLwdXrpjH168r+dmIDTe1bvQzBM6PN0 CoqL+m6dM+vhPWm1ofsZfpwHjIhJu4EgZCJKZS2HCMxZ3SeEybPC/w+FNXugUmVkFzu4 uEPnw2eXhxngyTVqLfNGMhCPTQBA+VFk3zdl/5nSQIJEWuDoDd7NDmiiBa/1nus/Z3QB B3Ta5CL/HOyXNOMI+RBvPgEjRm5JYJrRFNJrG3NI7f5MQB+C8bW6OZ23oJL5WdJyqPBT D3oeHOOlCOeJdj13Qv7kx9cRdNG4Kshbif/lq0n2m3qQC5B3nIq0a1MenIaK0r89IbnR 0JZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5uGAFhwG1PalEv/hF+X/ZtSoGm0z/W3W27BE5YAO4Hk=; b=UTdC1MQIIgTH/O62JqGiJT4UmRmKau7Bq1r/tKTjTrUlo7LD1X5ENtBEoF0cp/BdRs O1aYK6P7UpAWA1bDw2Zzz3twbKs9wvZZOSvGNkfjVTcBTeowxzyuO1Si9p2F0Yoly2F6 5ssEuaCgQUqcYgWjY0Rh0dElwHjJHb2R/NQlCV0CVGBCctePpfhPLQDSnCk0x1B2S7fn dytuXTfII3RUrxY+XCxmYgx/t53dFH5pKkWAjq3qrYvFSFFV2EUfkVe4j//w+wwKDt8x g+6kpus9RWdf/ujOMqUHWwca8ORuhMUTH4SX8SKMXLYCav3zzBGN7hJDK+oJEDFtv21w /QSQ== X-Gm-Message-State: AOAM530VOoHr4T5VjHCYtU95rYsBrAEZbsDDLEy0Im8gYLHrBdvBE4OD x3610CldFUoGaVvRPWgnO9Q= X-Google-Smtp-Source: ABdhPJxvOYj/fwaErfsO6GCO2pBBDv3M4enh4tzXVYxkLKfbV68E2GHA1K7sLhRddzhPPP1seAlZaQ== X-Received: by 2002:a63:7014:: with SMTP id l20mr1684639pgc.384.1641805948672; Mon, 10 Jan 2022 01:12:28 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:28 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/13] blk: rename ioprio.c to ioprio-common.c Date: Mon, 10 Jan 2022 17:10:42 +0800 Message-Id: <20220110091046.17010-10-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao In next patch, blk-ioprio.c is changed to a module named ioprio.ko. Rename ioprio.c to ioprio-common.c to avoid same ioprio.o in Makefile Signed-off-by: Wang Jianchao --- block/Makefile | 2 +- block/{ioprio.c =3D> ioprio-common.c} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename block/{ioprio.c =3D> ioprio-common.c} (100%) diff --git a/block/Makefile b/block/Makefile index 8950913cbcc9..beacc3a03c8b 100644 --- a/block/Makefile +++ b/block/Makefile @@ -8,7 +8,7 @@ obj-y :=3D bdev.o fops.o bio.o elevator.o blk-core.o blk-s= ysfs.o \ blk-exec.o blk-merge.o blk-timeout.o \ blk-lib.o blk-mq.o blk-mq-tag.o blk-stat.o \ blk-mq-sysfs.o blk-mq-cpumap.o blk-mq-sched.o ioctl.o \ - genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \ + genhd.o ioprio-common.o badblocks.o partitions/ blk-rq-qos.o \ disk-events.o blk-ia-ranges.o =20 obj-$(CONFIG_BOUNCE) +=3D bounce.o diff --git a/block/ioprio.c b/block/ioprio-common.c similarity index 100% rename from block/ioprio.c rename to block/ioprio-common.c --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5308FC433F5 for ; Mon, 10 Jan 2022 09:13:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242674AbiAJJNX (ORCPT ); Mon, 10 Jan 2022 04:13:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242636AbiAJJMe (ORCPT ); Mon, 10 Jan 2022 04:12:34 -0500 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE048C06175C; Mon, 10 Jan 2022 01:12:31 -0800 (PST) Received: by mail-pj1-x102f.google.com with SMTP id ie23-20020a17090b401700b001b38a5318easo6024503pjb.2; Mon, 10 Jan 2022 01:12:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dzt0bzuRBlkO35+KYftsgi07H/oE3AzdNVwuKYsXxZs=; b=CPwAQj1mCmlC04HOlUdlK38+pfOGOXLbdnDpR6cnyfK+vlFQ6C+Rur4MHpt4ZN7nmC Bf0k4Mam86x141bOIzt1afIjzr/2ycQlpqZfSR1gnpZcG1BQovpPemlOelj9CKEysncp 5sS5dn8aplErQDATGbAQNPHwu/+6CnBvMLcQcQ1V5awK7TICjP7YxpD/8UPDzF9ko+U9 crEKi4qsmxX3wQ0P+k6Ua+ZcXv5sC/j/4AV5qsVY92pdA65arnpxJ9z1nCyp7URZtHuT w56TOKf/OErO+T6TIlbTMn3B3S4VrlSepmChenyGsedIZvK6kQmfilLPznItFZqvHs74 5KEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dzt0bzuRBlkO35+KYftsgi07H/oE3AzdNVwuKYsXxZs=; b=Hf9jiHb6QOFDZwbrc1rAFuNxa4ao446miAkFrsyU3qjAU833sdsMB4UgdZt7Y+Xj5r ZNTNXQlqx5LRXuwUhlfreZqart7SFi4roS4MaCcxE0se4gO/QRX9C+E+lLvuhVcz6O/p SF49r8Ki6OXuVZ8Hh5AUNsTkEMmrgGFpkH6+1/1X63rMneRQvdu23wfRA4CuM+w+x0FM hTpeFQXVZXONiT89n3OrwtU0FZl2cptnlj6tSIqEM98+hZFBgg9cV4mtFaKBaE40/AHw HHlYGScNcpuXfbVJVAc0n2B5PdjW1KsGK6JaNMAHi5Tdx0ao+oUamHiWjEN3m9AIA/1B 1PLA== X-Gm-Message-State: AOAM531FuA/F795A9ussKNpX53Sf3BJHcT4AfCSbg6azBPgQmazOFvCD kKXT51cFB5dwk4FUj0zt/yc= X-Google-Smtp-Source: ABdhPJy/eQapQuf+wbd8rg84L547vr+mzHjsjZ3/v13se8TuJJhFP7rAhcNiXVt1W02SsTw3vrF5fQ== X-Received: by 2002:a17:902:744b:b0:14a:3d99:b524 with SMTP id e11-20020a170902744b00b0014a3d99b524mr2940153plt.27.1641805951398; Mon, 10 Jan 2022 01:12:31 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:30 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/13] blk-ioprio: make ioprio pluggable and modular Date: Mon, 10 Jan 2022 17:10:43 +0800 Message-Id: <20220110091046.17010-11-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao Make blk-ioprio pluggable and modular. Then we can close or open it through /sys/block/xxx/queue/qos and rmmod the module if we don't need it which can release one blkcg policy slot. Signed-off-by: Wang Jianchao --- block/Kconfig | 2 +- block/Makefile | 3 ++- block/blk-cgroup.c | 5 ----- block/blk-ioprio.c | 50 ++++++++++++++++++++++++++++-------------- block/blk-ioprio.h | 19 ---------------- block/blk-mq-debugfs.c | 4 ---- block/blk-rq-qos.c | 2 +- block/blk-rq-qos.h | 2 +- 8 files changed, 38 insertions(+), 49 deletions(-) delete mode 100644 block/blk-ioprio.h diff --git a/block/Kconfig b/block/Kconfig index 3e1a3487b55a..b3a2c656a64b 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -145,7 +145,7 @@ config BLK_CGROUP_IOCOST their share of the overall weight distribution. =20 config BLK_CGROUP_IOPRIO - bool "Cgroup I/O controller for assigning an I/O priority class" + tristate "Cgroup I/O controller for assigning an I/O priority class" depends on BLK_CGROUP help Enable the .prio interface for assigning an I/O priority class to diff --git a/block/Makefile b/block/Makefile index beacc3a03c8b..3f76836076b2 100644 --- a/block/Makefile +++ b/block/Makefile @@ -17,7 +17,8 @@ obj-$(CONFIG_BLK_DEV_BSGLIB) +=3D bsg-lib.o obj-$(CONFIG_BLK_CGROUP) +=3D blk-cgroup.o obj-$(CONFIG_BLK_CGROUP_RWSTAT) +=3D blk-cgroup-rwstat.o obj-$(CONFIG_BLK_DEV_THROTTLING) +=3D blk-throttle.o -obj-$(CONFIG_BLK_CGROUP_IOPRIO) +=3D blk-ioprio.o +ioprio-y :=3D blk-ioprio.o +obj-$(CONFIG_BLK_CGROUP_IOPRIO) +=3D ioprio.o iolat-y :=3D blk-iolatency.o obj-$(CONFIG_BLK_CGROUP_IOLATENCY) +=3D iolat.o iocost-y :=3D blk-iocost.o diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index fd874dfd38ed..c5dc44194314 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -31,7 +31,6 @@ #include #include #include "blk.h" -#include "blk-ioprio.h" #include "blk-throttle.h" =20 /* @@ -1204,10 +1203,6 @@ int blkcg_init_queue(struct request_queue *q) if (preloaded) radix_tree_preload_end(); =20 - ret =3D blk_ioprio_init(q); - if (ret) - goto err_destroy_all; - ret =3D blk_throtl_init(q); if (ret) goto err_destroy_all; diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c index 332a07761bf8..93d8ba698942 100644 --- a/block/blk-ioprio.c +++ b/block/blk-ioprio.c @@ -17,7 +17,6 @@ #include #include #include -#include "blk-ioprio.h" #include "blk-rq-qos.h" =20 /** @@ -209,15 +208,23 @@ static void blkcg_ioprio_exit(struct rq_qos *rqos) container_of(rqos, typeof(*blkioprio_blkg), rqos); =20 blkcg_deactivate_policy(rqos->q, &ioprio_policy); + rq_qos_deactivate(rqos); kfree(blkioprio_blkg); } =20 +static int blk_ioprio_init(struct request_queue *q); static struct rq_qos_ops blkcg_ioprio_ops =3D { +#if IS_MODULE(CONFIG_BLK_CGROUP_IOPRIO) + .owner =3D THIS_MODULE, +#endif + .flags =3D RQOS_FLAG_CGRP_POL, + .name =3D "ioprio", .track =3D blkcg_ioprio_track, .exit =3D blkcg_ioprio_exit, + .init =3D blk_ioprio_init, }; =20 -int blk_ioprio_init(struct request_queue *q) +static int blk_ioprio_init(struct request_queue *q) { struct blk_ioprio *blkioprio_blkg; struct rq_qos *rqos; @@ -227,36 +234,45 @@ int blk_ioprio_init(struct request_queue *q) if (!blkioprio_blkg) return -ENOMEM; =20 + /* + * No need to worry ioprio_blkcg_from_css return NULL as + * the queue is frozen right now. + */ + rqos =3D &blkioprio_blkg->rqos; + rq_qos_activate(q, rqos, &blkcg_ioprio_ops); + ret =3D blkcg_activate_policy(q, &ioprio_policy); if (ret) { + rq_qos_deactivate(rqos); kfree(blkioprio_blkg); - return ret; } =20 - rqos =3D &blkioprio_blkg->rqos; - rqos->id =3D RQ_QOS_IOPRIO; - rqos->ops =3D &blkcg_ioprio_ops; - rqos->q =3D q; - - /* - * Registering the rq-qos policy after activating the blk-cgroup - * policy guarantees that ioprio_blkcg_from_bio(bio) !=3D NULL in the - * rq-qos callbacks. - */ - rq_qos_add(q, rqos); - - return 0; + return ret; } =20 static int __init ioprio_init(void) { - return blkcg_policy_register(&ioprio_policy); + int ret; + + ret =3D rq_qos_register(&blkcg_ioprio_ops); + if (ret) + return ret; + + ret =3D blkcg_policy_register(&ioprio_policy); + if (ret) + rq_qos_unregister(&blkcg_ioprio_ops); + + return ret; } =20 static void __exit ioprio_exit(void) { blkcg_policy_unregister(&ioprio_policy); + rq_qos_unregister(&blkcg_ioprio_ops); } =20 module_init(ioprio_init); module_exit(ioprio_exit); +MODULE_AUTHOR("Bart Van Assche"); +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Cgroup I/O controller for assigning an I/O priority cl= ass"); diff --git a/block/blk-ioprio.h b/block/blk-ioprio.h deleted file mode 100644 index a7785c2f1aea..000000000000 --- a/block/blk-ioprio.h +++ /dev/null @@ -1,19 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ - -#ifndef _BLK_IOPRIO_H_ -#define _BLK_IOPRIO_H_ - -#include - -struct request_queue; - -#ifdef CONFIG_BLK_CGROUP_IOPRIO -int blk_ioprio_init(struct request_queue *q); -#else -static inline int blk_ioprio_init(struct request_queue *q) -{ - return 0; -} -#endif - -#endif /* _BLK_IOPRIO_H_ */ diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 14fda9a5e552..90610a0cd25a 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -819,10 +819,6 @@ void blk_mq_debugfs_unregister_sched(struct request_qu= eue *q) =20 static const char *rq_qos_id_to_name(enum rq_qos_id id) { - switch (id) { - case RQ_QOS_IOPRIO: - return "ioprio"; - } return "unknown"; } =20 diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index 08ccd4a4e913..15852147ba73 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -459,7 +459,7 @@ int rq_qos_register(struct rq_qos_ops *ops) goto out; } =20 - start =3D RQ_QOS_IOPRIO + 1; + start =3D 1; ret =3D ida_simple_get(&rq_qos_ida, start, INT_MAX, GFP_KERNEL); if (ret < 0) goto out; diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index 4eef53f2c290..ee396367a5b2 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -14,7 +14,7 @@ struct blk_mq_debugfs_attr; =20 enum rq_qos_id { - RQ_QOS_IOPRIO, + RQ_QOS_UNUSED, }; =20 struct rq_wait { --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8209C433EF for ; Mon, 10 Jan 2022 09:13:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242805AbiAJJN3 (ORCPT ); Mon, 10 Jan 2022 04:13:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242750AbiAJJMg (ORCPT ); Mon, 10 Jan 2022 04:12:36 -0500 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5399DC06173F; Mon, 10 Jan 2022 01:12:34 -0800 (PST) Received: by mail-pf1-x42a.google.com with SMTP id x27so1437790pfu.3; Mon, 10 Jan 2022 01:12:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Havnqspezv4jUABRFsufu3DjbG7hCfaqkFuMwWHgcgI=; b=Rs/WXbRqQzbODcPTUCom8Rp8jiyXqERj3Qwq0AOd3oQMgAQUtl6TWYHwE+UQWuf/Q7 9UXhQAEdeL7wXYpH5WpW9ZrtGiXLv+lvDVLmPqITAf8xTPj0pdtksmvH1L/ocABmJ+WA tVjnEFiYTBmskR4hthg86v/Lww1NqKTJyknWfC+zJYJFF9uNOV+oSwaBBxXX+b/Q6kVj WHVPQvya/fGtatrLi0i50Cr6jb3JwshZwcCH+ujys2SNOOa3x5q8P5iCWHXWHS2NKymG srSa5bzT5MxqJALe4g1QkUnib81gS48Q3WBzBJZ4+VqpDcdrgKh+PtObINy0tdBpl5Ij RVOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Havnqspezv4jUABRFsufu3DjbG7hCfaqkFuMwWHgcgI=; b=25LTrqwg1DB7l8NeFNcK6tDLClyKWgPFd5vmw2A8WWVXwd414aRviT+TIoS6WSN/Tq Amyz2gPIbucxiaV9VMjxQTPoLx4oRf45p22u/tw956/qsqTxymmytEM1Opa6eQX22g/5 9FXE+1fbIdMlb7A90vXoU+bs2bgcmFvE6B3azbVrk6ct4ExwW/wDaxI7OvEcVWVVnubN wB0NlVTTwwBzKEwi9XKhWT3L/IWVr1tZ3yMPnheZpq+YozCe4kZKfugtXr3gc2y97xE7 Cyh+8+910JdvKNWjOIXjviKJ9ZQuTEbWLgWN5hn64lTfpmk/MLjEvO6x1twyrOXmTg6R xaWw== X-Gm-Message-State: AOAM530xNugAJGvlvMoPhkVeGDSBGn+hTnYP+/twNbpoDAd0+h5MC2qI blT3vvz0x1kXrMgWxhW2n/k= X-Google-Smtp-Source: ABdhPJweU+mbCV3mYSnAIFT38eF6SYRj5iJJRFijz+mK/sNuVDOKii8PmBvA6cGbe4HOdBCSunZGPw== X-Received: by 2002:a63:36c2:: with SMTP id d185mr2932196pga.612.1641805953910; Mon, 10 Jan 2022 01:12:33 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:33 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 11/13] blk: remove unused interfaces of blk-rq-qos Date: Mon, 10 Jan 2022 17:10:44 +0800 Message-Id: <20220110091046.17010-12-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao No functional changes here Signed-off-by: Wang Jianchao --- block/blk-mq-debugfs.c | 10 +------- block/blk-rq-qos.h | 52 +----------------------------------------- 2 files changed, 2 insertions(+), 60 deletions(-) diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 90610a0cd25a..f4f5ca1953f3 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -817,11 +817,6 @@ void blk_mq_debugfs_unregister_sched(struct request_qu= eue *q) q->sched_debugfs_dir =3D NULL; } =20 -static const char *rq_qos_id_to_name(enum rq_qos_id id) -{ - return "unknown"; -} - void blk_mq_debugfs_unregister_rqos(struct rq_qos *rqos) { debugfs_remove_recursive(rqos->debugfs_dir); @@ -832,9 +827,6 @@ EXPORT_SYMBOL_GPL(blk_mq_debugfs_unregister_rqos); void blk_mq_debugfs_register_rqos(struct rq_qos *rqos) { struct request_queue *q =3D rqos->q; - const char *dir_name; - - dir_name =3D rqos->ops->name ? rqos->ops->name : rq_qos_id_to_name(rqos->= id); =20 if (rqos->debugfs_dir || !rqos->ops->debugfs_attrs) return; @@ -843,7 +835,7 @@ void blk_mq_debugfs_register_rqos(struct rq_qos *rqos) q->rqos_debugfs_dir =3D debugfs_create_dir("rqos", q->debugfs_dir); =20 - rqos->debugfs_dir =3D debugfs_create_dir(dir_name, + rqos->debugfs_dir =3D debugfs_create_dir(rqos->ops->name, rqos->q->rqos_debugfs_dir); =20 debugfs_create_files(rqos->debugfs_dir, rqos, rqos->ops->debugfs_attrs); diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index ee396367a5b2..123b6b100355 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -13,10 +13,6 @@ =20 struct blk_mq_debugfs_attr; =20 -enum rq_qos_id { - RQ_QOS_UNUSED, -}; - struct rq_wait { wait_queue_head_t wait; atomic_t inflight; @@ -28,7 +24,7 @@ struct rq_qos { bool dying; const struct rq_qos_ops *ops; struct request_queue *q; - enum rq_qos_id id; + int id; struct rq_qos *next; #ifdef CONFIG_BLK_DEBUG_FS struct dentry *debugfs_dir; @@ -89,52 +85,6 @@ static inline void rq_wait_init(struct rq_wait *rq_wait) init_waitqueue_head(&rq_wait->wait); } =20 -static inline void rq_qos_add(struct request_queue *q, struct rq_qos *rqos) -{ - /* - * No IO can be in-flight when adding rqos, so freeze queue, which - * is fine since we only support rq_qos for blk-mq queue. - * - * Reuse ->queue_lock for protecting against other concurrent - * rq_qos adding/deleting - */ - blk_mq_freeze_queue(q); - - spin_lock_irq(&q->queue_lock); - rqos->next =3D q->rq_qos; - q->rq_qos =3D rqos; - spin_unlock_irq(&q->queue_lock); - - blk_mq_unfreeze_queue(q); - - if (rqos->ops->debugfs_attrs) - blk_mq_debugfs_register_rqos(rqos); -} - -static inline void rq_qos_del(struct request_queue *q, struct rq_qos *rqos) -{ - struct rq_qos **cur; - - /* - * See comment in rq_qos_add() about freezing queue & using - * ->queue_lock. - */ - blk_mq_freeze_queue(q); - - spin_lock_irq(&q->queue_lock); - for (cur =3D &q->rq_qos; *cur; cur =3D &(*cur)->next) { - if (*cur =3D=3D rqos) { - *cur =3D rqos->next; - break; - } - } - spin_unlock_irq(&q->queue_lock); - - blk_mq_unfreeze_queue(q); - - blk_mq_debugfs_unregister_rqos(rqos); -} - int rq_qos_register(struct rq_qos_ops *ops); void rq_qos_unregister(struct rq_qos_ops *ops); void rq_qos_activate(struct request_queue *q, --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 314BDC433F5 for ; Mon, 10 Jan 2022 09:14:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243088AbiAJJN6 (ORCPT ); Mon, 10 Jan 2022 04:13:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242696AbiAJJMh (ORCPT ); Mon, 10 Jan 2022 04:12:37 -0500 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD787C06175B; Mon, 10 Jan 2022 01:12:36 -0800 (PST) Received: by mail-pg1-x536.google.com with SMTP id f5so10590908pgk.12; Mon, 10 Jan 2022 01:12:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bSUEh0XLtqhfgkfkzajXuKpmq20fHb9Sj/+2QGQC9ew=; b=jzwrhKF0NlGNkrs0lKhKEQVXqGfdbsnXa/4dzpcDhN3rFtajTCp9xVy86nUMHwXCtr iql48w/CwdJX9x5rpOvpnLNHFMSGu8KD5a8CpHcdSLM43GkkGH0fRIsH26NAn4ElAVHw hpmFOSyyjy1fchoAZGBvYHCaENHaFJhZZ6V/1ELvUleV1rtbHgxldxoDN+InaHkbE/MK hB1myc/KO6znAfJcTSQf5Wz/EIq14zITjIm+50KHSLODbabX8bXVgpCz3c34bWBdCuCi 1zCj40tb1hDNMuBJX4PWb6KpKs5M/o+v+AY6yql8ktoTeUWSSsgy35b2pC7WN90kF8PQ Jtlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bSUEh0XLtqhfgkfkzajXuKpmq20fHb9Sj/+2QGQC9ew=; b=u192hYkONIaUPAb3c42BRcolRzFUsUrxZpt0/1sqH7dd8VqDKNqkOAV7Db5BKUA2l4 mY0JZQaGG2heEKhuFmiolRg+w4Y/jt5mTkSMR/wRUm5MILDyDzm+HV5pObWNzzs3Q3+y ahoZiQd7ttOw/+uaNHdImEiGb6z8Y3/dq/Q321xXeIpuN210MS2BcixXLD0OTKFTmaun 7zmBFUxTvU/GPF8dFzy1TZ3EUkvD0LBJZVOeuhPv779lhMk62im/zBKz6KhMuuUfaZVl sv1KKzEMVHZyqwIQm2QNEBDr/l45nigS+vv+oVX1SAUAAsSdRDTGu8Xp4Wu5nIOq1ngD 3JWA== X-Gm-Message-State: AOAM532gys9LwjPGxcevQ0RFFBh31LJSEt9V8bsa83CiQLIIwKNH1LGo CrP4EaBzU1kshQWFxpGJfIM= X-Google-Smtp-Source: ABdhPJzsaxURA5vI16ldXhVbMjcAnwUgXVGx9P+pyAfurtLyGpwIdEoqiLlJH/5hXDN5H9EapuNNhA== X-Received: by 2002:a63:33cd:: with SMTP id z196mr42921496pgz.78.1641805956416; Mon, 10 Jan 2022 01:12:36 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:36 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 12/13] blk: make request able to carry blkcg_gq Date: Mon, 10 Jan 2022 17:10:45 +0800 Message-Id: <20220110091046.17010-13-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao After blk_update_request, the bios can be gone. We cannot track the req in cgroup fashion in following IO completion path. This patch add blkcg_gq into request, get it when install bio, put it before request is released. Signed-off-by: Wang Jianchao --- block/Kconfig | 3 +++ block/blk-core.c | 6 +++++- block/blk-merge.c | 9 +++++++++ block/blk-mq.c | 14 ++++++++++++++ include/linux/blk-mq.h | 4 +++- 5 files changed, 34 insertions(+), 2 deletions(-) diff --git a/block/Kconfig b/block/Kconfig index b3a2c656a64b..ea612cb5c8ee 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -32,6 +32,9 @@ config BLK_BIO_IOCOST config BLK_RQ_ALLOC_TIME bool =20 +config BLK_RQ_BLKCG_GQ + bool + config BLK_CGROUP_RWSTAT bool =20 diff --git a/block/blk-core.c b/block/blk-core.c index 2847ab514c1f..083160895125 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1462,7 +1462,11 @@ int blk_rq_prep_clone(struct request *rq, struct req= uest *rq_src, } rq->nr_phys_segments =3D rq_src->nr_phys_segments; rq->ioprio =3D rq_src->ioprio; - +#ifdef CONFIG_BLK_RQ_BLKCG_GQ + if (rq_src->blkg) + blkg_get(rq_src->blkg); + rq->blkg =3D rq_src->blkg; +#endif if (rq->bio && blk_crypto_rq_bio_prep(rq, rq->bio, gfp_mask) < 0) goto free_and_out; =20 diff --git a/block/blk-merge.c b/block/blk-merge.c index 893c1a60b701..cf5d0e5ce04f 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -793,6 +793,10 @@ static struct request *attempt_merge(struct request_qu= eue *q, if (req->ioprio !=3D next->ioprio) return NULL; =20 +#ifdef CONFIG_BLK_RQ_BLKCG_GQ + if (req->blkg !=3D next->blkg) + return NULL; +#endif /* * If we are allowed to merge, then append bio list * from next to rq and release next. merge_requests_fn @@ -930,6 +934,11 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *b= io) if (rq->ioprio !=3D bio_prio(bio)) return false; =20 +#ifdef CONFIG_BLK_RQ_BLKCG_GQ + if (rq->blkg !=3D bio->bi_blkg) + return false; +#endif + return true; } =20 diff --git a/block/blk-mq.c b/block/blk-mq.c index 8874a63ae952..131845bca5de 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -28,6 +28,7 @@ #include #include #include +#include =20 #include =20 @@ -369,6 +370,9 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq= _alloc_data *data, rq->nr_phys_segments =3D 0; #if defined(CONFIG_BLK_DEV_INTEGRITY) rq->nr_integrity_segments =3D 0; +#endif +#ifdef CONFIG_BLK_RQ_BLKCG_GQ + rq->blkg =3D NULL; #endif rq->end_io =3D NULL; rq->end_io_data =3D NULL; @@ -600,6 +604,10 @@ static void __blk_mq_free_request(struct request *rq) struct blk_mq_hw_ctx *hctx =3D rq->mq_hctx; const int sched_tag =3D rq->internal_tag; =20 +#ifdef CONFIG_BLK_RQ_BLKCG_GQ + if (rq->blkg) + blkg_put(rq->blkg); +#endif blk_crypto_free_request(rq); blk_pm_mark_last_busy(rq); rq->mq_hctx =3D NULL; @@ -2305,6 +2313,12 @@ static void blk_mq_bio_to_request(struct request *rq= , struct bio *bio, rq->__sector =3D bio->bi_iter.bi_sector; rq->write_hint =3D bio->bi_write_hint; blk_rq_bio_prep(rq, bio, nr_segs); +#ifdef CONFIG_BLK_RQ_BLKCG_GQ + if (bio->bi_blkg) { + blkg_get(bio->bi_blkg); + rq->blkg =3D bio->bi_blkg; + } +#endif =20 /* This can't fail, since GFP_NOIO includes __GFP_DIRECT_RECLAIM. */ err =3D blk_crypto_rq_bio_prep(rq, bio, GFP_NOIO); diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 2949d9ac7484..f9cc6f6b8d63 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -110,7 +110,9 @@ struct request { u64 start_time_ns; /* Time that I/O was submitted to the device. */ u64 io_start_time_ns; - +#ifdef CONFIG_BLK_RQ_BLKCG_GQ + struct blkcg_gq *blkg; +#endif #ifdef CONFIG_BLK_WBT unsigned short wbt_flags; #endif --=20 2.17.1 From nobody Tue Jun 30 20:10:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 187FAC433F5 for ; Mon, 10 Jan 2022 09:14:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243190AbiAJJO1 (ORCPT ); Mon, 10 Jan 2022 04:14:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242771AbiAJJMj (ORCPT ); Mon, 10 Jan 2022 04:12:39 -0500 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F448C061751; Mon, 10 Jan 2022 01:12:39 -0800 (PST) Received: by mail-pf1-x42a.google.com with SMTP id x27so1437935pfu.3; Mon, 10 Jan 2022 01:12:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pUKMXuo3xVxEWKXjp7ogKF7zeSoPw2PTqo2WG9e7EOA=; b=Di3cCvIIY239IGt/c0tGqMqgvAASPuPHbHmpXojKoUTCVBMW1LRI5K5Roe/rkjJJF5 J9zueerHJSCCjyTJL7jLXgEl3GnSNljmqfAhm5b86Un2Lhyu1CZI+IoZI+cLDaREx7ri 7DOjA8EMIZ9c4wsmiRsBmKHm1YQC05orPwzfSvK4qdSEeaHiZqm0Yb7LfuRfcqAXiQoQ YoNBwOAsaXc8AfJLJ/mhZlZJeGB/6uev3KQXCfIt95hf8/gYJGcNrIWmWKncgMK6cTrD 9d/3iH2t7PymWGE2CFZkWszqlDUNpTYE9FX/H1+UVFS+ZMOLoKzrJzdjGjxEpJ/L0WSF mHRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pUKMXuo3xVxEWKXjp7ogKF7zeSoPw2PTqo2WG9e7EOA=; b=sYKBh/sQBKE65L7dT9XjRuXBUOE5JVe9FKCkaDDZgRXG8n6eovY9iVPzH5ggXyLD8d ZWEpXW2ZmkT1umkg1cUFlZ3E9hN9RLeJFrnOY7IJUR0xnHEpZUbW0RUVIWXYykTqDikw w6P3CR2nZHNtkvDeyNlUwmelQ34LiXwxtCP3x1+wP2QymBa4rEzFvXFePMgEvL5nqQw6 6c23goeFyqhoeMuQ+uEfNOPFUDqQ84zQ031+2C2h+A70cpDcnSgtildcxQDYZzff4MRA c+9UJxrI4CKohFOkjC4P2uJdi04ISYTQDOTQeSaLoMnphEqdyVtKQblFGxUqeyauO1s5 s6fw== X-Gm-Message-State: AOAM5327DVwdJJSAKID+/bOH/Q9WGE1XI3hUekDKRJHd3Ctx6saRv3/R FcEoq8Wdb9+C0OawNbGtNCQ= X-Google-Smtp-Source: ABdhPJzcUoj3Y6lu927itTNWfKF9jZHJw6//ZclzBXChG2+gGlTakfZDxB2artybUeJ0b5BbTvueRw== X-Received: by 2002:a05:6a00:1351:b0:4bc:edbf:7acd with SMTP id k17-20020a056a00135100b004bcedbf7acdmr18006227pfu.41.1641805958867; Mon, 10 Jan 2022 01:12:38 -0800 (PST) Received: from localhost.localdomain ([162.219.34.250]) by smtp.gmail.com with ESMTPSA id j22sm6293910pfj.29.2022.01.10.01.12.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 01:12:38 -0800 (PST) From: Wang Jianchao To: axboe@kernel.dk Cc: jbacik@fb.com, tj@kernel.org, bvanassche@acm.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 13/13] blk: introduce iostat per cgroup module Date: Mon, 10 Jan 2022 17:10:46 +0800 Message-Id: <20220110091046.17010-14-jianchao.wan9@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220110091046.17010-1-jianchao.wan9@gmail.com> References: <20220110091046.17010-1-jianchao.wan9@gmail.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Wang Jianchao iostat can only track the whole device's io statistics. This patch introduces iostat per cgroup based on blk-rq-qos framework which can track bw, iops, queue latency and device latency and distinguish regular or meta data. The blkio.iostat per cgroup output in following format, vda-data bytes iops queue_lat dev_lat [ditto] [ditto] meta \___________ ______________/ | | v v v read write discard In particular, the blkio.iostat of root only output the statistics of IOs from root cgroup. However, the non-root blkio.iostat outputs all of the children cgroups. With meta stats in root cgroup, hope to observe the performace of fs metadata. Signed-off-by: Wang Jianchao --- block/Kconfig | 9 ++ block/Makefile | 2 + block/blk-iostat.c | 356 +++++++++++++++++++++++++++++++++++++++++ include/linux/blkdev.h | 2 +- 4 files changed, 368 insertions(+), 1 deletion(-) create mode 100644 block/blk-iostat.c diff --git a/block/Kconfig b/block/Kconfig index ea612cb5c8ee..35f24db3ec92 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -156,6 +156,15 @@ config BLK_CGROUP_IOPRIO scheduler and block devices process requests. Only some I/O schedulers and some block devices support I/O priorities. =20 +config BLK_CGROUP_IOSTAT + tristate "IO statistics monitor per cgroup" + select BLK_RQ_BLKCG_GQ + select BLK_RQ_ALLOC_TIME + depends on BLK_CGROUP + help + Monitor IO statistics, including bw, iops, queue latency and device + latency, in per-cgroup level. + config BLK_DEBUG_FS bool "Block layer debugging information in debugfs" default y diff --git a/block/Makefile b/block/Makefile index 3f76836076b2..ad89015e37ce 100644 --- a/block/Makefile +++ b/block/Makefile @@ -23,6 +23,8 @@ iolat-y :=3D blk-iolatency.o obj-$(CONFIG_BLK_CGROUP_IOLATENCY) +=3D iolat.o iocost-y :=3D blk-iocost.o obj-$(CONFIG_BLK_CGROUP_IOCOST) +=3D iocost.o +iostat-y :=3D blk-iostat.o +obj-$(CONFIG_BLK_CGROUP_IOSTAT) +=3D iostat.o obj-$(CONFIG_MQ_IOSCHED_DEADLINE) +=3D mq-deadline.o obj-$(CONFIG_MQ_IOSCHED_KYBER) +=3D kyber-iosched.o bfq-y :=3D bfq-iosched.o bfq-wf2q.o bfq-cgroup.o diff --git a/block/blk-iostat.c b/block/blk-iostat.c new file mode 100644 index 000000000000..3c6bcb6ab055 --- /dev/null +++ b/block/blk-iostat.c @@ -0,0 +1,356 @@ +#include +#include +#include +#include +#include +#include + +#include "blk.h" +#include "blk-rq-qos.h" + +enum { + IOSTAT_READ =3D 0, + IOSTAT_WRITE, + IOSTAT_DISCARD, + IOSTAT_MAX, +}; + +struct iostat_data { + u64 bytes[IOSTAT_MAX]; + u64 ios[IOSTAT_MAX]; + u64 queue_lat[IOSTAT_MAX]; + u64 dev_lat[IOSTAT_MAX]; +}; + +struct iostat_queue { + struct rq_qos rqos; +}; + +struct iostat_gq { + struct blkg_policy_data pd; + char disk_name[DISK_NAME_LEN]; + struct { + struct iostat_data __percpu *data; + struct iostat_data __percpu *meta; + } stat; +}; + +struct iostat_cgrp { + struct blkcg_policy_data cpd; +}; + +DEFINE_MUTEX(iostat_mutex); + +static struct blkcg_policy blkcg_policy_iostat; + +static inline struct iostat_gq *pd_to_ist(struct blkg_policy_data *pd) +{ + return pd ? container_of(pd, struct iostat_gq, pd) : NULL; +} + +static inline struct iostat_gq *blkg_to_ist(struct blkcg_gq *blkg) +{ + return pd_to_ist(blkg_to_pd(blkg, &blkcg_policy_iostat)); +} + +static inline bool req_is_meta(struct request *req) +{ + return req->cmd_flags & REQ_META; +} + +static inline int iostat_op(struct request *req) +{ + int op; + + if (unlikely(op_is_discard(req_op(req)))) + op =3D IOSTAT_DISCARD; + else if (op_is_write(req_op(req))) + op =3D IOSTAT_WRITE; + else + op =3D IOSTAT_READ; + + return op; +} + +static void __iostat_issue(struct rq_qos *rqos, + struct iostat_gq *is, struct request *req) +{ + struct iostat_data *stat; + int op =3D iostat_op(req); + + /* + * blk_mq_start_request() inherents bio_issue_time() when BLK_CGROUP + * to avoid overhead of readtsc. + */ + req->io_start_time_ns =3D ktime_get_ns(); + if (req_is_meta(req)) + stat =3D get_cpu_ptr(is->stat.meta); + else + stat =3D get_cpu_ptr(is->stat.data); + /* + * alloc_time_ns is get before get tag, we use it monitor depth, + * tag waits and in queue time. + */ + stat->queue_lat[op] +=3D req->io_start_time_ns - req->alloc_time_ns; + stat->ios[op]++; + stat->bytes[op] +=3D blk_rq_bytes(req); + put_cpu_ptr(stat); +} + +static void iostat_issue(struct rq_qos *rqos, struct request *req) +{ + struct iostat_gq *is; + + if (unlikely(!req->bio)) + return; + + is =3D blkg_to_ist(req->blkg); + /* + * Most of time, bios from submit_bio would have the valid bi_blkg, + * however, blk_execute_rq case is an exception. + */ + if (is) + __iostat_issue(rqos, is, req); +} + +static void __iostat_done(struct rq_qos *rq_qos, + struct iostat_gq *is, struct request *req) +{ + struct iostat_data *stat; + int op =3D iostat_op(req); + + if (req_is_meta(req)) + stat =3D get_cpu_ptr(is->stat.meta); + else + stat =3D get_cpu_ptr(is->stat.data); + if (req->io_start_time_ns) + stat->dev_lat[op] +=3D ktime_get_ns() - req->io_start_time_ns; + put_cpu_ptr(stat); +} + +static void iostat_done(struct rq_qos *rqos, struct request *req) +{ + struct iostat_gq *is =3D blkg_to_ist(req->blkg); + + if (is) + __iostat_done(rqos, is, req); +} + +static void iostat_exit(struct rq_qos *rqos) +{ + struct iostat_queue *isq =3D container_of(rqos, struct iostat_queue, rqos= ); + + blkcg_deactivate_policy(rqos->q, &blkcg_policy_iostat); + rq_qos_deactivate(rqos); + kfree(isq); +} + +static int iostat_init(struct request_queue *q); + +struct rq_qos_ops iostat_rq_ops =3D { +#if IS_MODULE(CONFIG_BLK_CGROUP_IOLATENCY) + .owner =3D THIS_MODULE, +#endif + .name =3D "iostat", + .flags =3D RQOS_FLAG_CGRP_POL | RQOS_FLAG_RQ_ALLOC_TIME, + .issue =3D iostat_issue, + .done =3D iostat_done, + .exit =3D iostat_exit, + .init =3D iostat_init, +}; + +static int iostat_init(struct request_queue *q) +{ + struct iostat_queue *isq; + struct rq_qos *rqos; + int ret; + + isq =3D kzalloc_node(sizeof(*isq), GFP_KERNEL, q->node); + if (!isq) { + ret =3D -ENOMEM; + goto out; + } + + blk_queue_flag_set(QUEUE_FLAG_RQ_ALLOC_TIME, q); + rqos =3D &isq->rqos; + rq_qos_activate(q, rqos, &iostat_rq_ops); + + ret =3D blkcg_activate_policy(q, &blkcg_policy_iostat); + if (ret) { + rq_qos_deactivate(rqos); + kfree(isq); + } +out: + return ret; +} + +static void iostat_sum(struct blkcg_gq *blkg, + struct iostat_data *sum, bool meta) +{ + struct iostat_gq *is =3D blkg_to_ist(blkg); + struct iostat_data *stat; + int cpu, i; + + for_each_possible_cpu(cpu) { + if (meta) + stat =3D per_cpu_ptr(is->stat.meta, cpu); + else + stat =3D per_cpu_ptr(is->stat.data, cpu); + for (i =3D 0; i < IOSTAT_MAX; i++) { + sum->bytes[i] +=3D stat->bytes[i]; + sum->ios[i] +=3D stat->ios[i]; + sum->dev_lat[i] +=3D stat->dev_lat[i]; + sum->queue_lat[i] +=3D stat->queue_lat[i]; + } + } +} + +static int iostat_show(struct seq_file *sf, void *v) +{ + struct blkcg *blkcg =3D css_to_blkcg(seq_css(sf)); + struct cgroup_subsys_state *pos_css; + struct iostat_gq *is; + struct blkcg_gq *blkg, *pos_blkg; + struct iostat_data data_sum, meta_sum; + int i; + + rcu_read_lock(); + hlist_for_each_entry_rcu(blkg, &blkcg->blkg_list, blkcg_node) { + is =3D blkg_to_ist(blkg); + /* + * The is activated on demand so iostat may be NULL + */ + if (!is) + continue; + + memset(&data_sum, 0, sizeof(data_sum)); + memset(&meta_sum, 0, sizeof(meta_sum)); + if (blkg =3D=3D blkg->q->root_blkg) { + iostat_sum(blkg, &data_sum, false); + iostat_sum(blkg, &meta_sum, true); + } else { + /* + * Iterate every children blkg to agregate statistics + */ + blkg_for_each_descendant_pre(pos_blkg, pos_css, blkg) { + if (!pos_blkg->online) + continue; + iostat_sum(pos_blkg, &data_sum, false); + iostat_sum(pos_blkg, &meta_sum, true); + } + } + + seq_printf(sf, "%s-data ", is->disk_name); + for (i =3D 0; i < IOSTAT_MAX; i++) + seq_printf(sf, "%llu %llu %llu %llu ", + data_sum.bytes[i], data_sum.ios[i], + data_sum.queue_lat[i], data_sum.dev_lat[i]); + seq_printf(sf, "\n"); + seq_printf(sf, "%s-meta ", is->disk_name); + for (i =3D 0; i < IOSTAT_MAX; i++) + seq_printf(sf, "%llu %llu %llu %llu ", + meta_sum.bytes[i], meta_sum.ios[i], + meta_sum.queue_lat[i], meta_sum.dev_lat[i]); + seq_printf(sf, "\n"); + } + rcu_read_unlock(); + + return 0; +} + +static struct cftype iostat_files[] =3D { + { + .name =3D "iostat", + .seq_show =3D iostat_show, + }, + {} +}; + +static struct cftype iostat_legacy_files[] =3D { + { + .name =3D "iostat", + .seq_show =3D iostat_show, + }, + {} +}; + +static void iostat_pd_free(struct blkg_policy_data *pd) +{ + struct iostat_gq *is =3D pd_to_ist(pd); + + if (is->stat.data) + free_percpu(is->stat.data); + + if (is->stat.meta) + free_percpu(is->stat.meta); + + kfree(is); +} + +static struct blkg_policy_data *iostat_pd_alloc(gfp_t gfp, + struct request_queue *q, struct blkcg *blkcg) +{ + struct iostat_gq *is; + + is =3D kzalloc_node(sizeof(*is), gfp, q->node); + if (!is) + return NULL; + + is->stat.data =3D __alloc_percpu_gfp(sizeof(struct iostat_data), + __alignof__(struct iostat_data), gfp); + if (!is->stat.data) + goto out_free; + + is->stat.meta =3D __alloc_percpu_gfp(sizeof(struct iostat_data), + __alignof__(struct iostat_data), gfp); + if (!is->stat.meta) + goto out_free; + /* + * request_queue.kobj's parent is gendisk + */ + strlcpy(is->disk_name, + kobject_name(q->kobj.parent), + DISK_NAME_LEN); + return &is->pd; +out_free: + if (is->stat.data) + free_percpu(is->stat.data); + iostat_pd_free(&is->pd); + return NULL; +} + +static struct blkcg_policy blkcg_policy_iostat =3D { + .dfl_cftypes =3D iostat_files, + .legacy_cftypes =3D iostat_legacy_files, + .pd_alloc_fn =3D iostat_pd_alloc, + .pd_free_fn =3D iostat_pd_free, +}; + +static int __init iostat_mod_init(void) +{ + int ret; + + ret =3D rq_qos_register(&iostat_rq_ops); + if (ret) + return ret; + + ret =3D blkcg_policy_register(&blkcg_policy_iostat); + if (ret) { + rq_qos_unregister(&iostat_rq_ops); + return ret; + } + + return 0; +} + +static void __exit iostat_mod_exit(void) +{ + rq_qos_unregister(&iostat_rq_ops); + blkcg_policy_unregister(&blkcg_policy_iostat); +} + +module_init(iostat_mod_init); +module_exit(iostat_mod_exit); +MODULE_AUTHOR("Wang Jianchao"); +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Block Statistics per Cgroup"); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index ed30b3c3fee7..75026cf54384 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -42,7 +42,7 @@ struct blk_crypto_profile; * Maximum number of blkcg policies allowed to be registered concurrently. * Defined here to simplify include dependency. */ -#define BLKCG_MAX_POLS 6 +#define BLKCG_MAX_POLS 7 /* * Non blk-rq-qos blkcg policies include blk-throttle and bfq */ --=20 2.17.1