From nobody Fri Sep 25 16:50:48 2026 Received: from mta1.migadu.com (out-67.mta1.migadu.com [95.215.58.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24F3148D89A for ; Thu, 10 Sep 2026 12:58:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.67 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045123; cv=none; b=EQbOwo6GaEI0Lk/HZZ3VQTDA0LuO/aoN05IHuX4E7eliSs98USz006LprqP866udhyq8FKjd1Gc6Vx42hFbjWc7QdQ5otRA4hpgiuja8CMrJFxRA/ntuXHZUnDqzhzkYJoRpU17tKAtQT+OE2j2X8JQr+AbeMazf+MtFs2hUrxM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045123; c=relaxed/simple; bh=MJ++yjDOPO4NFzAUTelL4zZK30pEBPrATP/d1Uv+VbY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rB3I8FGhh7gq47bCKD56/KSfl5Ci1pw8QyekKySn+LGNxKTPZ7P/eEV/lYRVck22OoLgb8pLTEsDZf4BDB3OIfIXIZgjJRpX3ZuQ0DjAmElsYOS/Ng/AZ4KTvTyYrMC6ys6+InmhJKLBgJiZkqZiwgAB2f7cMBNcj6CuBlNpEqs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Em7gdEXR; arc=none smtp.client-ip=95.215.58.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Em7gdEXR" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=MJ++yjDOPO4NFzAUTelL4zZK30pEBPrATP/d1Uv+VbY=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789045114; v=1; x=1789649914; b=Em7gdEXRwVWcAmDsBKIQp8XHyWVYh5+fKfCh0VlYmPQNGICouASge53Du3Q8G/ncmLDgYR/k L8KPM4ZWrBZ2B4AfzXwvE/4kb8tZZWZx6vR35GfMaicmani5c9SAedVAGyQUmGRKjiSHRtu5wi6 Gx2fnJ72fu6DfdE439mWY7u8= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 92caa6535c2126fa; Thu, 10 Sep 2026 12:58:34 +0000 X-Mizu-Trace-ID: 92caa6535c2126fa X-Migadu-Flow: FLOW_OUT From: Tao Cui To: tj@kernel.org, josef@toxicopanda.com, axboe@kernel.dk Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net, linux-kselftest@vger.kernel.org, cui.tao@linux.dev, cuitao@kylinos.cn Subject: [RFC PATCH v2 1/5] blk-iocost: add BPF struct_ops cost model support Date: Thu, 10 Sep 2026 20:58:13 +0800 Message-ID: <20260910125817.223354-2-cui.tao@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260910125817.223354-1-cui.tao@linux.dev> References: <20260910125817.223354-1-cui.tao@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tao Cui Add the iocost_model_ops struct_ops: a bound BPF model fully replaces the builtin linear model on a device. calc_cost() receives the full bio->bi_opf (including REQ_PREFLUSH and REQ_FUA), the IO size, the start sector (sector_t), the issuing blkcg and the iocost-specific call metadata (the merge-path indicator), and is called from both the bio charging path and the request-level sizing path, so a model owns pricing for every IO on the device. The builtin cursor is not exposed: a model is expected to track its own stream state. The registration and binding model follows the TCP congestion model registration pattern: registering a struct_ops makes the model available by its name (char name[16], validated at init_member), while io.cost.model binds one registered model to a device with "model=3D" and unbinds with "model=3Dlinear" or "ctrl=3Dauto/user". Unregistering a model removes it from the registry so it can no longer be selected by name; devices already using the model continue to do so until switched back to the builtin model. References are taken with bpf_struct_ops_get()/put() on the kdata and released when the device switches back to the builtin model. calc_cost() runs under RCU read lock; sleepable programs are rejected in .check_member. blkcg_online()/blkcg_offline() callbacks mirroring the blkcg css lifecycle let models manage per-cgroup state. Registered and bound models coexist with the builtin model: devices which are not bound keep the builtin linear model unchanged. Signed-off-by: Tao Cui --- block/Kconfig | 9 ++ block/Makefile | 1 + block/blk-cgroup.c | 3 + block/blk-iocost-bpf.c | 250 +++++++++++++++++++++++++++++++++++++ block/blk-iocost.c | 133 ++++++++++++++++++-- include/linux/blk-iocost.h | 82 ++++++++++++ 6 files changed, 471 insertions(+), 7 deletions(-) create mode 100644 block/blk-iocost-bpf.c create mode 100644 include/linux/blk-iocost.h diff --git a/block/Kconfig b/block/Kconfig index 70e4a66d941f..91e808f86d28 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -231,4 +231,13 @@ config BLK_ERROR_INJECTION =20 source "block/Kconfig.iosched" =20 +config BLK_CGROUP_IOCOST_BPF + bool "Enable BPF pluggable cost model support for the cost IO controller" + depends on BLK_CGROUP_IOCOST && BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF + help + Enabling this option registers the "iocost_model_ops" BPF + struct_ops type, which allows a BPF program to fully replace + the builtin linear cost model on a device it is bound to + through io.cost.model. + endif # BLOCK diff --git a/block/Makefile b/block/Makefile index e7bd320e3d69..ee5cebeea006 100644 --- a/block/Makefile +++ b/block/Makefile @@ -39,3 +39,4 @@ obj-$(CONFIG_BLK_INLINE_ENCRYPTION) +=3D blk-crypto.o blk= -crypto-profile.o \ blk-crypto-sysfs.o obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) +=3D blk-crypto-fallback.o obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED) +=3D holder.o +obj-$(CONFIG_BLK_CGROUP_IOCOST_BPF) +=3D blk-iocost-bpf.o diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 2b5c29434e42..872871045351 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -32,6 +32,7 @@ #include #include "blk.h" #include "blk-cgroup.h" +#include #include "blk-ioprio.h" #include "blk-throttle.h" =20 @@ -1341,6 +1342,7 @@ void blkcg_unpin_online(struct cgroup_subsys_state *b= lkcg_css) */ static void blkcg_css_offline(struct cgroup_subsys_state *css) { + iocost_notify_blkcg_offline(css_to_blkcg(css)); /* this prevents anyone from attaching or migrating to this blkcg */ wb_blkcg_offline(css); =20 @@ -1445,6 +1447,7 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_cs= s) =20 static int blkcg_css_online(struct cgroup_subsys_state *css) { + iocost_notify_blkcg_online(css_to_blkcg(css)); struct blkcg *parent =3D blkcg_parent(css_to_blkcg(css)); =20 /* diff --git a/block/blk-iocost-bpf.c b/block/blk-iocost-bpf.c new file mode 100644 index 000000000000..aec6df279599 --- /dev/null +++ b/block/blk-iocost-bpf.c @@ -0,0 +1,250 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * blk-iocost: BPF struct_ops plumbing for pluggable cost models. + * + * Registers the "iocost_model_ops" struct_ops type and maintains the + * name registry of registered models. A registered model is bound to + * a device through io.cost.model; see include/linux/blk-iocost.h. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static DEFINE_MUTEX(iocost_bpf_reg_lock); +static LIST_HEAD(iocost_bpf_models); + +/* + * The registry holds a bpf_struct_ops_get() reference obtained in .reg; + * .unreg drops it, so the kdata of an unregistered model stays alive + * while any device is still bound to it. + */ +struct iocost_bpf_model { + struct list_head list; + const struct iocost_model_ops *ops; +}; + +/* + * Look up a registered model by name and acquire a reference on it. + * The registry lock is held across lookup and bpf_struct_ops_get() so + * the model cannot be unregistered in between. + */ +int iocost_bpf_model_get(const char *name, + const struct iocost_model_ops **opsp) +{ + struct iocost_bpf_model *m; + int ret =3D -ENOENT; + + mutex_lock(&iocost_bpf_reg_lock); + list_for_each_entry(m, &iocost_bpf_models, list) { + if (!strcmp(m->ops->name, name)) { + if (bpf_struct_ops_get(m->ops)) { + *opsp =3D m->ops; + ret =3D 0; + } + break; + } + } + mutex_unlock(&iocost_bpf_reg_lock); + return ret; +} + +void iocost_bpf_model_put(const struct iocost_model_ops *ops) +{ + bpf_struct_ops_put(ops); +} + +static struct iocost_bpf_model * +iocost_bpf_model_lookup(const struct iocost_model_ops *ops) +{ + struct iocost_bpf_model *m; + + list_for_each_entry(m, &iocost_bpf_models, list) { + if (m->ops =3D=3D ops) + return m; + } + return NULL; +} + +void iocost_notify_blkcg_online(struct blkcg *blkcg) +{ + struct iocost_bpf_model *m; + + guard(mutex)(&iocost_bpf_reg_lock); + list_for_each_entry(m, &iocost_bpf_models, list) { + if (m->ops->blkcg_online) + m->ops->blkcg_online(blkcg); + } +} + +void iocost_notify_blkcg_offline(struct blkcg *blkcg) +{ + struct iocost_bpf_model *m; + + guard(mutex)(&iocost_bpf_reg_lock); + list_for_each_entry(m, &iocost_bpf_models, list) { + if (m->ops->blkcg_offline) + m->ops->blkcg_offline(blkcg); + } +} + +static int bpf_iocost_model_init(struct btf *btf) +{ + s32 type_id; + + type_id =3D btf_find_by_name_kind(btf, "iocost_model_ops", BTF_KIND_STRUC= T); + if (type_id < 0) + return -EINVAL; + return 0; +} + +static bool bpf_iocost_is_valid_access(int off, int size, + enum bpf_access_type type, + const struct bpf_prog *prog, + struct bpf_insn_access_aux *info) +{ + return bpf_tracing_btf_ctx_access(off, size, type, prog, info); +} + +static const struct bpf_func_proto * +bpf_iocost_get_func_proto(enum bpf_func_id func_id, + const struct bpf_prog *prog) +{ + switch (func_id) { +#ifdef CONFIG_CGROUPS + case BPF_FUNC_cgrp_storage_get: + return &bpf_cgrp_storage_get_proto; +#endif + default: + return bpf_base_func_proto(func_id, prog); + } +} + +static int bpf_iocost_check_member(const struct btf_type *t, + const struct btf_member *member, + const struct bpf_prog *prog) +{ + /* calc_cost() is called with RCU read lock held */ + if (prog->sleepable) + return -EINVAL; + return 0; +} + +static int bpf_iocost_init_member(const struct btf_type *t, + const struct btf_member *member, + void *kdata, const void *udata) +{ + struct iocost_model_ops *ops =3D kdata; + const struct iocost_model_ops *uops =3D udata; + u32 moff =3D __btf_member_bit_offset(t, member) / 8; + + switch (moff) { + case offsetof(struct iocost_model_ops, name): + if (bpf_obj_name_cpy(ops->name, uops->name, + sizeof(ops->name)) <=3D 0) + return -EINVAL; + return 1; + } + + return 0; +} + +static int bpf_iocost_validate(void *kdata) +{ + struct iocost_model_ops *ops =3D kdata; + + return ops->calc_cost ? 0 : -EINVAL; +} + +static int bpf_iocost_reg(void *kdata, struct bpf_link *link) +{ + struct iocost_model_ops *ops =3D kdata; + struct iocost_bpf_model *m; + int ret =3D 0; + + if (!bpf_struct_ops_get(ops)) + return -ENOENT; + + m =3D kzalloc(sizeof(*m), GFP_KERNEL); + if (!m) { + bpf_struct_ops_put(ops); + return -ENOMEM; + } + + mutex_lock(&iocost_bpf_reg_lock); + { + struct iocost_bpf_model *other; + + list_for_each_entry(other, &iocost_bpf_models, list) { + if (!strcmp(other->ops->name, ops->name)) { + ret =3D -EEXIST; + break; + } + } + } + if (!ret) { + m->ops =3D ops; + list_add(&m->list, &iocost_bpf_models); + } + mutex_unlock(&iocost_bpf_reg_lock); + + if (ret) { + bpf_struct_ops_put(ops); + kfree(m); + } + return ret; +} + +static void bpf_iocost_unreg(void *kdata, struct bpf_link *link) +{ + struct iocost_model_ops *ops =3D kdata; + struct iocost_bpf_model *m; + + mutex_lock(&iocost_bpf_reg_lock); + m =3D iocost_bpf_model_lookup(ops); + if (m) { + list_del(&m->list); + bpf_struct_ops_put(ops); + kfree(m); + } + mutex_unlock(&iocost_bpf_reg_lock); +} + +static const struct bpf_verifier_ops bpf_iocost_verifier_ops =3D { + .get_func_proto =3D bpf_iocost_get_func_proto, + .is_valid_access =3D bpf_iocost_is_valid_access, +}; + +static u64 bpf_iocost_calc_cost_stub(u64 opf, u64 nbytes, u64 sector, + struct blkcg *blkcg, u64 flags) +{ + return 0; +} + +static struct iocost_model_ops __bpf_ops_iocost_model_ops =3D { + .calc_cost =3D bpf_iocost_calc_cost_stub, +}; + +static struct bpf_struct_ops bpf_iocost_model_ops =3D { + .verifier_ops =3D &bpf_iocost_verifier_ops, + .init =3D bpf_iocost_model_init, + .check_member =3D bpf_iocost_check_member, + .init_member =3D bpf_iocost_init_member, + .validate =3D bpf_iocost_validate, + .reg =3D bpf_iocost_reg, + .unreg =3D bpf_iocost_unreg, + .name =3D "iocost_model_ops", + .cfi_stubs =3D &__bpf_ops_iocost_model_ops, + .owner =3D THIS_MODULE, +}; + +static int __init bpf_iocost_init(void) +{ + return register_bpf_struct_ops(&bpf_iocost_model_ops, iocost_model_ops); +} +late_initcall(bpf_iocost_init); diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 2745bffcd5ee..182601ad783f 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -177,6 +177,7 @@ #include #include #include +#include #include #include #include @@ -445,6 +446,11 @@ struct ioc { int autop_idx; bool user_qos_params:1; bool user_cost_model:1; + +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF + /* bound BPF cost model, NULL =3D builtin linear model */ + const struct iocost_model_ops __rcu *model; +#endif }; =20 struct iocg_pcpu_stat { @@ -2571,10 +2577,28 @@ static void calc_vtime_cost_builtin(struct bio *bio= , struct ioc_gq *iocg, =20 static u64 calc_vtime_cost(struct bio *bio, struct ioc_gq *iocg, bool is_m= erge) { +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF + const struct iocost_model_ops *model; u64 cost; =20 - calc_vtime_cost_builtin(bio, iocg, is_merge, &cost); - return cost; + rcu_read_lock(); + model =3D rcu_dereference(iocg->ioc->model); + if (model) { + cost =3D model->calc_cost(bio->bi_opf, bio->bi_iter.bi_size, + bio->bi_iter.bi_sector, + iocg_to_blkg(iocg)->blkcg, + is_merge ? IOCOST_COST_F_MERGE : 0); + rcu_read_unlock(); + return min(cost, VTIME_PER_SEC); + } + rcu_read_unlock(); +#endif + { + u64 cost; + + calc_vtime_cost_builtin(bio, iocg, is_merge, &cost); + return cost; + } } =20 static void calc_size_vtime_cost_builtin(struct request *rq, struct ioc *i= oc, @@ -2596,10 +2620,28 @@ static void calc_size_vtime_cost_builtin(struct req= uest *rq, struct ioc *ioc, =20 static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc) { - u64 cost; +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF + const struct iocost_model_ops *model; =20 - calc_size_vtime_cost_builtin(rq, ioc, &cost); - return cost; + rcu_read_lock(); + model =3D rcu_dereference(ioc->model); + if (model && rq->bio && rq->bio->bi_blkg) { + u64 cost; + + cost =3D model->calc_cost(rq->cmd_flags, blk_rq_bytes(rq), + blk_rq_pos(rq), + rq->bio->bi_blkg->blkcg, 0); + rcu_read_unlock(); + return min(cost, VTIME_PER_SEC); + } + rcu_read_unlock(); +#endif + { + u64 cost; + + calc_size_vtime_cost_builtin(rq, ioc, &cost); + return cost; + } } =20 enum over_budget_action { @@ -2900,6 +2942,19 @@ static void ioc_rqos_exit(struct rq_qos *rqos) =20 timer_shutdown_sync(&ioc->timer); free_percpu(ioc->pcpu_stat); +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF + { + const struct iocost_model_ops *model; + + spin_lock_irq(&ioc->lock); + model =3D rcu_dereference_protected(ioc->model, + lockdep_is_held(&ioc->lock)); + rcu_assign_pointer(ioc->model, NULL); + spin_unlock_irq(&ioc->lock); + if (model) + iocost_bpf_model_put(model); + } +#endif kfree(ioc); } =20 @@ -3438,12 +3493,30 @@ static u64 ioc_cost_model_prfill(struct seq_file *s= f, return 0; =20 spin_lock_irq(&ioc->lock); +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF + { + const struct iocost_model_ops *model =3D + rcu_dereference_protected(ioc->model, + lockdep_is_held(&ioc->lock)); + + seq_printf(sf, "%s ctrl=3D%s model=3D%s " + "rbps=3D%llu rseqiops=3D%llu rrandiops=3D%llu " + "wbps=3D%llu wseqiops=3D%llu wrandiops=3D%llu\n", + dname, model ? "bpf" : + ioc->user_cost_model ? "user" : "auto", + model ? model->name : "linear", + u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], + u[I_LCOEF_RRANDIOPS], u[I_LCOEF_WBPS], + u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS]); + } +#else seq_printf(sf, "%s ctrl=3D%s model=3Dlinear " "rbps=3D%llu rseqiops=3D%llu rrandiops=3D%llu " "wbps=3D%llu wseqiops=3D%llu wrandiops=3D%llu\n", dname, ioc->user_cost_model ? "user" : "auto", u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], u[I_LCOEF_RRANDIOPS], u[I_LCOEF_WBPS], u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS]); +#endif spin_unlock_irq(&ioc->lock); return 0; } @@ -3457,6 +3530,37 @@ static int ioc_cost_model_show(struct seq_file *sf, = void *v) return 0; } =20 +/* + * Bind @name (empty =3D builtin linear model) as the active cost model of + * @ioc. The registry lookup and reference management happen outside + * ioc->lock; the pointer swap happens under it. + */ +static int ioc_bpf_model_bind(struct ioc *ioc, const char *name) +{ +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF + const struct iocost_model_ops *new =3D NULL, *old; + int ret; + + if (name[0]) { + ret =3D iocost_bpf_model_get(name, &new); + if (ret) + return ret; + } + + spin_lock_irq(&ioc->lock); + old =3D rcu_dereference_protected(ioc->model, + lockdep_is_held(&ioc->lock)); + rcu_assign_pointer(ioc->model, new); + spin_unlock_irq(&ioc->lock); + + if (old) + iocost_bpf_model_put(old); + return 0; +#else + return name[0] ? -ENOENT : 0; +#endif +} + static const match_table_t cost_ctrl_tokens =3D { { COST_CTRL, "ctrl=3D%s" }, { COST_MODEL, "model=3D%s" }, @@ -3482,6 +3586,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_ope= n_file *of, char *input, struct ioc *ioc; u64 u[NR_I_LCOEFS]; bool user; + char bpf_model[IOCOST_MODEL_NAME_LEN]; char *body, *p; int ret; =20 @@ -3512,6 +3617,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_ope= n_file *of, char *input, spin_lock_irq(&ioc->lock); memcpy(u, ioc->params.i_lcoefs, sizeof(u)); user =3D ioc->user_cost_model; + bpf_model[0] =3D '\0'; =20 ret =3D -EINVAL; =20 @@ -3533,11 +3639,16 @@ static ssize_t ioc_cost_model_write(struct kernfs_o= pen_file *of, char *input, user =3D true; else goto unlock; + bpf_model[0] =3D '\0'; continue; case COST_MODEL: match_strlcpy(buf, &args[0], sizeof(buf)); - if (strcmp(buf, "linear")) - goto unlock; + if (!strcmp(buf, "linear")) { + /* back to the builtin linear model */ + bpf_model[0] =3D '\0'; + continue; + } + match_strlcpy(bpf_model, &args[0], sizeof(bpf_model)); continue; } =20 @@ -3563,6 +3674,14 @@ static ssize_t ioc_cost_model_write(struct kernfs_op= en_file *of, char *input, unlock: spin_unlock_irq(&ioc->lock); =20 + /* + * Bind the BPF model outside ioc->lock: the registry lookup + * takes the registration mutex and the old model's reference + * is dropped after the swap. + */ + if (!ret) + ret =3D ioc_bpf_model_bind(ioc, bpf_model); + blk_mq_unquiesce_queue(q); blk_mq_unfreeze_queue(q, memflags); =20 diff --git a/include/linux/blk-iocost.h b/include/linux/blk-iocost.h new file mode 100644 index 000000000000..3a0855efb610 --- /dev/null +++ b/include/linux/blk-iocost.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_BLK_IOCOST_H +#define _LINUX_BLK_IOCOST_H + +#include +#include + +#define IOCOST_MODEL_NAME_LEN 16 + +#ifdef CONFIG_BLK_CGROUP_IOCOST_BPF + +struct blkcg; + +/* + * Pluggable cost model interface for blk-iocost. + * + * A BPF struct_ops implementation registered against "iocost_model_ops" + * fully replaces the builtin linear model on the devices it is bound to + * through io.cost.model. The model owns pricing for every IO on a bound + * device: it prices all operations, including flushes, and both the bio + * charging path and the request-level sizing path consult it. + * + * calc_cost() is called from the IO submission path with RCU read lock + * held and must not sleep. It returns the cost of the IO in vtime + * units, where 1 second of device time equals VTIME_PER_SEC (2^37, + * available to BPF programs through vmlinux.h). The returned value is + * clamped to 1 second of device time per IO. + * + * The model is passed the blkcg of the issuing cgroup so it can keep + * per-cgroup state. blkcg_online()/blkcg_offline() are optional + * callbacks mirroring the blkcg css lifecycle: state created on online + * (or lazily on first use) must be released on offline. + * + * The registration and binding model follows the TCP congestion + * control framework: registering a struct_ops makes the model available + * by its name, while io.cost.model binds one registered model to a + * device. Unregistering removes the name from the registry; devices + * already bound keep using it until switched back to the builtin + * model. + */ + +#define IOCOST_MODEL_NAME_LEN 16 + +/* + * iocost-specific call metadata for calc_cost()'s model_flags + * argument; everything else, including REQ_PREFLUSH/REQ_FUA, is + * already present in the opf argument + */ +#define IOCOST_COST_F_MERGE (1ULL << 0) /* called from merge path */ + +struct iocost_model_ops { + u64 (*calc_cost)(u64 opf, u64 nbytes, sector_t sector, + struct blkcg *blkcg, u64 model_flags); + void (*blkcg_online)(struct blkcg *blkcg); + void (*blkcg_offline)(struct blkcg *blkcg); + + /* model name, used to select the model through io.cost.model */ + char name[16]; +}; + +int iocost_bpf_model_get(const char *name, + const struct iocost_model_ops **opsp); +void iocost_bpf_model_put(const struct iocost_model_ops *ops); +void iocost_notify_blkcg_online(struct blkcg *blkcg); +void iocost_notify_blkcg_offline(struct blkcg *blkcg); + +#else /* CONFIG_BLK_CGROUP_IOCOST_BPF */ + +struct blkcg; +struct iocost_model_ops; + +static inline int iocost_bpf_model_get(const char *name, + const struct iocost_model_ops **opsp) +{ + return -EOPNOTSUPP; +} +static inline void iocost_bpf_model_put(const struct iocost_model_ops *ops= ) { } +static inline void iocost_notify_blkcg_online(struct blkcg *blkcg) { } +static inline void iocost_notify_blkcg_offline(struct blkcg *blkcg) { } + +#endif /* CONFIG_BLK_CGROUP_IOCOST_BPF */ +#endif /* _LINUX_BLK_IOCOST_H */ --=20 2.43.0 From nobody Fri Sep 25 16:50:48 2026 Received: from mta0.migadu.com (out-167.mta0.migadu.com [91.218.175.167]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F031B48EC9C for ; Thu, 10 Sep 2026 12:58:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.167 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045125; cv=none; b=F8QkUAh/2H/hoqm6Gilgyk/fHudfvhF8AvIu8fYo3ydj42CSaM2aJN318QJu7tXLHA1IRYBzyZTTM9gBo1SzCmbkciFTEhzqDj3wZlbAq0LABcEHOW9uW5r82phg0ymLduHK0l8qAwOhkZBgOa7IBJTDqnlL3vwC3Z0GYtWkjL4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045125; c=relaxed/simple; bh=Se0UEuEkn7sxV61bxAAIjwfnVXML7LXljmTxrF6ZABc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VHsmNxFvzgg0en6c2Whh5pXZvAy8Ro+7ixVtl7jUVoOP0urtiMi8wIB//qXJPZXuZQW7wSllNLbetD/dgGY/NAhcPmlLyPPHZnkawK73YmAhVFxYyHc8h/MbCl1c1D5h5iNqJVDWF79JrHgfBuk/oHaJh+efjCBVqT/NgeCFzco= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=PBesXywy; arc=none smtp.client-ip=91.218.175.167 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="PBesXywy" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=Se0UEuEkn7sxV61bxAAIjwfnVXML7LXljmTxrF6ZABc=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789045119; v=1; x=1789649919; b=PBesXywyCmuIu4NJ3ekT0u+54gg4KasAp6ZuQ1IBUbRd0p+7yPv4G2gzWBf4E6fJJo+GyZb7 ykL9jsMtNuZRhUAP7K9CT1ULZCjvd+KHxQjzLR88xtQBq6O0L4b7iqMHGzjMvaBeuRxHCAGxv2p A9cnnpiXSW3tIzeWH8PyAz1U= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 315279177ac67947; Thu, 10 Sep 2026 12:58:38 +0000 X-Mizu-Trace-ID: 315279177ac67947 X-Migadu-Flow: FLOW_OUT From: Tao Cui To: tj@kernel.org, josef@toxicopanda.com, axboe@kernel.dk Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net, linux-kselftest@vger.kernel.org, cui.tao@linux.dev, cuitao@kylinos.cn Subject: [RFC PATCH v2 2/5] selftests/bpf: add iocost cost model test Date: Thu, 10 Sep 2026 20:58:14 +0800 Message-ID: <20260910125817.223354-3-cui.tao@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260910125817.223354-1-cui.tao@linux.dev> References: <20260910125817.223354-1-cui.tao@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tao Cui Add an example cost model implementing the full builtin linear HDD formula at double cost, including flush pricing, and a runner which registers it as a struct_ops and binds it to a device through io.cost.model with "model=3Diocost_2x", verifying the readback and restoring "model=3Dlinear" afterwards, checking the write error and errno of every step including the restoration. Binding an unknown model name is verified to fail with ENOENT. Under the same workload the doubled model charges exactly twice the builtin model (verified 2882us -> 5722us per IO, completed IO count halved). Per-cgroup stream state uses a CGRP_STORAGE map keyed by the cgroup of the blkcg argument, so the model inherits the cgroup lifetime and never leaks or reuses stale state. opf carries the full bio->bi_opf including REQ_* flag bits, so the operation must be extracted with a mask, not compared for equality. All writes go through write(2) rather than stdio, since the kernel's rejection happens in the write() syscall, not in the userspace buffer copy. CONFIG_BLK_CGROUP_IOCOST and CONFIG_BLK_CGROUP_IOCOST_BPF are added to the selftest kernel config: without them vmlinux.h does not contain iocost_model_ops and the skeletons fail to build; the runtime skip cannot avoid a build dependency. The runner is skipped unless $IOCOST_TEST_DEV gives a major:minor of a device with iocost enabled. Signed-off-by: Tao Cui --- tools/testing/selftests/bpf/config | 2 + .../selftests/bpf/prog_tests/iocost_model.c | 193 ++++++++++++++++++ .../selftests/bpf/progs/iocost_model.c | 116 +++++++++++ 3 files changed, 311 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/iocost_model.c create mode 100644 tools/testing/selftests/bpf/progs/iocost_model.c diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/b= pf/config index 2f79688dcf7c..67a630cb5614 100644 --- a/tools/testing/selftests/bpf/config +++ b/tools/testing/selftests/bpf/config @@ -138,3 +138,5 @@ CONFIG_SMC_HS_CTRL_BPF=3Dy CONFIG_DIBS=3Dy CONFIG_DIBS_LO=3Dy CONFIG_PM_WAKELOCKS=3Dy +CONFIG_BLK_CGROUP_IOCOST=3Dy +CONFIG_BLK_CGROUP_IOCOST_BPF=3Dy diff --git a/tools/testing/selftests/bpf/prog_tests/iocost_model.c b/tools/= testing/selftests/bpf/prog_tests/iocost_model.c new file mode 100644 index 000000000000..e9344b13fba9 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/iocost_model.c @@ -0,0 +1,193 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include "iocost_model.skel.h" +#include "iocost_ms.skel.h" + +/* + * Write a line to io.cost.model with write(2) and return the errno of + * the failed write, or 0 on success. stdio is not used here on + * purpose: the kernel's rejection happens in the write() syscall, + * not in the userspace buffer copy, and every write, including + * the error paths of the callers below, is checked. + */ +static int write_cost_model(const char *buf) +{ + int fd, err =3D 0; + ssize_t n; + + fd =3D open("/sys/fs/cgroup/io.cost.model", O_WRONLY); + if (fd < 0) + return errno; + n =3D write(fd, buf, strlen(buf)); + if (n < 0) + err =3D errno; + close(fd); + return err; +} + +/* + * Read back the io.cost.model line of dev and copy the model=3D value + * into @model. Returns 0 on success. + */ +static int readback_model(const char *dev, char *model, size_t model_sz) +{ + char line[256], word[256], *m, *end; + FILE *fp; + int found =3D 0; + + fp =3D fopen("/sys/fs/cgroup/io.cost.model", "r"); + if (!fp) + return -1; + while (fgets(line, sizeof(line), fp)) { + if (sscanf(line, "%255s", word) =3D=3D 1 && !strcmp(word, dev)) { + found =3D 1; + break; + } + } + fclose(fp); + if (!found) + return -1; + + m =3D strstr(line, "model=3D"); + if (!m) + return -1; + m +=3D strlen("model=3D"); + end =3D m; + while (*end && !isspace(*end)) + end++; + snprintf(model, model_sz, "%.*s", (int)(end - m), m); + return 0; +} + +/* + * Bind the named model to a device with "model=3D", verify the + * readback and restore the builtin model. Returns 0 on success. + */ +static int bind_model(const char *dev, const char *name) +{ + char buf[300], got[64]; + int err; + + snprintf(buf, sizeof(buf), "%s model=3D%s\n", dev, name); + err =3D write_cost_model(buf); + if (err) { + CHECK(false, "write_model", "write model=3D%s: %s\n", name, + strerror(err)); + return -1; + } + err =3D readback_model(dev, got, sizeof(got)); + if (err || strcmp(got, name)) { + CHECK(false, "readback_model", "got model=3D%s want %s\n", + err ? "(none)" : got, name); + return -1; + } + + /* restore the builtin linear model; the write is checked too */ + snprintf(buf, sizeof(buf), "%s model=3Dlinear\n", dev); + err =3D write_cost_model(buf); + if (err) { + CHECK(false, "restore_linear", "write model=3Dlinear: %s\n", + strerror(err)); + return -1; + } + return 0; +} + +/* + * The dev argument must be present in io.cost.qos already, which + * means iocost is enabled for it. + */ +static int dev_has_iocost(const char *dev) +{ + char line[256], word[256]; + FILE *fp; + int found =3D 0; + + fp =3D fopen("/sys/fs/cgroup/io.cost.qos", "r"); + if (!fp) + return 0; + while (fgets(line, sizeof(line), fp)) { + if (sscanf(line, "%255s", word) =3D=3D 1 && !strcmp(word, dev)) { + found =3D 1; + break; + } + } + fclose(fp); + return found; +} + +/* + * Bind the 2x example model and verify the io.cost.model readback. + * IO accounting itself is not checked here; it needs a device doing + * real IO under iocost and is covered by the kernel-side validation + * described in the cover letter. + * + * Requires root, cgroup v2 and a device with iocost support. The + * device must be given as major:minor in $IOCOST_TEST_DEV, otherwise + * the test is skipped. + */ +void serial_test_iocost_model(void) +{ + struct iocost_model *skel; + char buf[300], *dev; + int err; + + dev =3D getenv("IOCOST_TEST_DEV"); + if (!dev || geteuid() !=3D 0) { + test__skip(); + return; + } + if (!ASSERT_TRUE(dev_has_iocost(dev), "iocost_mounted")) + return; + + /* + * negative: binding an unknown model name must be rejected, + * so a typo cannot silently disable cost model updates + */ + snprintf(buf, sizeof(buf), "%s model=3Dno_such_model\n", dev); + err =3D write_cost_model(buf); + ASSERT_EQ(err, ENOENT, "unknown_model_rejected"); + + skel =3D iocost_model__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open_load")) + return; + + /* attaching the struct_ops registers the model by name */ + err =3D iocost_model__attach(skel); + if (ASSERT_OK(err, "attach")) + ASSERT_OK(bind_model(dev, "iocost_2x"), "bind_and_readback"); + + iocost_model__destroy(skel); +} + +/* + * Same check for the multi-stream example model. Only one model can + * be bound to a device at a time; both tests bind and restore, so + * they are serial and independent. + */ +void serial_test_iocost_model_streams(void) +{ + struct iocost_ms *skel; + char *dev; + int err; + + dev =3D getenv("IOCOST_TEST_DEV"); + if (!dev || geteuid() !=3D 0) { + test__skip(); + return; + } + if (!ASSERT_TRUE(dev_has_iocost(dev), "iocost_mounted")) + return; + + skel =3D iocost_ms__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open_load")) + return; + + err =3D iocost_ms__attach(skel); + if (ASSERT_OK(err, "attach")) + ASSERT_OK(bind_model(dev, "iocost_ms"), "bind_and_readback"); + + iocost_ms__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/iocost_model.c b/tools/testi= ng/selftests/bpf/progs/iocost_model.c new file mode 100644 index 000000000000..f8ad417b3584 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/iocost_model.c @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Example iocost cost model: the builtin linear HDD formula with all + * costs doubled. + * + * The constants mirror what calc_lcoefs() derives from the AUTOP_HDD + * defaults (rbps=3D174019176 rseqiops=3D41708 rrandiops=3D370, w-side + * analog) in vtime units where 1s =3D=3D 2^37, expressed with the same + * round-up divisions so they cannot drift from the kernel. A device + * bound to this model through io.cost.model charges exactly twice the + * builtin model under the same workload, which makes it a convenient + * way to verify that accounting goes through the BPF path. + * + * The model implements the full linear formula itself, including + * flushes: there is no fallback to the builtin model, a dataless + * WRITE|REQ_PREFLUSH is priced as a one-page write. + */ + +#include "vmlinux.h" +#include +#include + +/* VTIME_PER_SEC comes from vmlinux.h (a BTF enum constant) */ +#define IOC_PAGE_SIZE 4096 +#define IOC_SECT_TO_PAGE_SHIFT 3 /* 512B sectors to 4kB pages */ +#define LCOEF_RANDIO_PAGES 4096 /* 16MB seek threshold */ +#define IOCOST_COST_F_MERGE (1ULL << 0) /* not in BTF: a plain macro */ +#define IOCOST_REQ_OP_MASK 0xff /* REQ_OP_MASK, not in BTF */ + +/* + * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at + * compile time + */ +#define RU(x, y) ((x) / (y) + (((x) % (y)) ? 1 : 0)) + +#define RBPS 174019176ULL +#define RSEQIOPS 41708ULL +#define RRANDIOPS 370ULL +#define WBPS 178075866ULL +#define WSEQIOPS 42705ULL +#define WRANDIOPS 378ULL + +#define RPAGE (RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE))) +#define RSEQIO (RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE) +#define RRANDIO (RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE) +#define WPAGE (RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE))) +#define WSEQIO (RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE) +#define WRANDIO (RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE) + +/* + * per-cgroup cursor storage: keyed by the cgroup, freed with it, so + * the model never leaks or reuses stale per-cgroup state + */ +struct { + __uint(type, BPF_MAP_TYPE_CGRP_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, __u64); +} cursor_store SEC(".maps"); + +SEC("struct_ops") +u64 BPF_PROG(iocost_2x_calc_cost, u64 opf, u64 nbytes, u64 sector, + struct blkcg *blkcg, u64 model_flags) +{ + u64 pages, seek_pages =3D 0, base, coef_page, randio, cost; + + if ((opf & IOCOST_REQ_OP_MASK) =3D=3D REQ_OP_READ) { + base =3D RSEQIO; coef_page =3D RPAGE; randio =3D RRANDIO; + } else if ((opf & IOCOST_REQ_OP_MASK) =3D=3D REQ_OP_WRITE) { + base =3D WSEQIO; coef_page =3D WPAGE; randio =3D WRANDIO; + } else { + /* + * a fully owning model must price every op; unknown + * ops are priced as a single page write + */ + base =3D 0; coef_page =3D WPAGE; randio =3D 0; + } + + /* + * mirror the builtin single per-cgroup cursor: the model keeps + * its own cursor keyed by the blkcg argument + */ + { + __u64 *cursor, cur; + + cursor =3D bpf_cgrp_storage_get(&cursor_store, + blkcg->css.cgroup, NULL, + BPF_LOCAL_STORAGE_GET_F_CREATE); + if (!cursor) + return 2 * (base + RU(nbytes, IOC_PAGE_SIZE) * coef_page); + cur =3D *cursor; + seek_pages =3D sector > cur ? sector - cur : cur - sector; + seek_pages >>=3D IOC_SECT_TO_PAGE_SHIFT; + if (seek_pages > LCOEF_RANDIO_PAGES) + base =3D randio; + if (!(model_flags & IOCOST_COST_F_MERGE)) + *cursor =3D sector + RU(nbytes, 512); + } + + pages =3D RU(nbytes, IOC_PAGE_SIZE); + if (!pages) + pages =3D 1; /* dataless flush: one page */ + if (model_flags & IOCOST_COST_F_MERGE) + base =3D 0; + + cost =3D 2 * (base + pages * coef_page); + return cost; +} + +SEC(".struct_ops") +struct iocost_model_ops iocost_2x =3D { + .calc_cost =3D (void *)iocost_2x_calc_cost, + .name =3D "iocost_2x", +}; + +char LICENSE[] SEC("license") =3D "GPL"; --=20 2.43.0 From nobody Fri Sep 25 16:50:48 2026 Received: from mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6341848F820 for ; Thu, 10 Sep 2026 12:58:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045129; cv=none; b=jqof3+XneIMI+Sv50J2eiIT58kzk+C2QQC7Soyl9tfxdCmM+mJkZXLYd2B+XB7nPxGydDSbQj1fxHobO4GdRbVDfc464l0i4Rn/4YyAY3RWKBnQ/d6dDjjTVavvL0nf2deQFh8O5HkNhqeRkXKw2TXLgMBf8SS4vx+g9KB4em3s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045129; c=relaxed/simple; bh=0fJmoDVgXoCT6wg71trk+r4GMKglV+8D+hnxVKMiMh8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gZrbdwtcPOh+8Y+5BvzRjNElkYIYspozVBxSbCxUj/Th7J3towrfBXYEHhH023pUvsebT6ULvHMbHBStWFTF/Oqs9Ddautv+b0S5YuXBHWp9VUDFMZ7atGnk8zQs7PC/23uDStO9IbG4mcJ38LAQX7SFqQC79kqDnB6ALa0R1e0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=j6axNBtV; arc=none smtp.client-ip=91.218.175.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="j6axNBtV" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=0fJmoDVgXoCT6wg71trk+r4GMKglV+8D+hnxVKMiMh8=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789045123; v=1; x=1789649923; b=j6axNBtVTV97ABwliYcSHsaak+4cZhBU4WSPGel1gWMOo3ldJzE9nlcG2ldSXxj8kz3F5MEM z594oHJBO6Yqv9UgLriMM0qEUhIq1CH4uMnQlDbT4PN8o4zFGBN/9aEPHsbcKZLDA0VMglRUXZp N8TBE7oHd+qGYsfZmPLFe1+s= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 4fa7f97bbfba81e3; Thu, 10 Sep 2026 12:58:43 +0000 X-Mizu-Trace-ID: 4fa7f97bbfba81e3 X-Migadu-Flow: FLOW_OUT From: Tao Cui To: tj@kernel.org, josef@toxicopanda.com, axboe@kernel.dk Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net, linux-kselftest@vger.kernel.org, cui.tao@linux.dev, cuitao@kylinos.cn Subject: [RFC PATCH v2 3/5] blk-iocost: add iocost_ioc_tick tracepoint for per-period device summary Date: Thu, 10 Sep 2026 20:58:15 +0800 Message-ID: <20260910125817.223354-4-cui.tao@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260910125817.223354-1-cui.tao@linux.dev> References: <20260910125817.223354-1-cui.tao@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tao Cui The existing iocost tracepoints are state-change driven: vrate_adj fires only when the adjustment logic runs, inuse_* only on surplus state transitions, activate/idle only on cgroup state changes. In a steady state none of them fire. The only other way to observe the controller (period length, vrate, busy level, active cgroup count, device utilization) is iocost_monitor.py, which reads kernel memory through drgn and is not usable in most production environments. Add iocost_ioc_tick, emitted once per period from the tail of ioc_timer_fn() with the overall controller state: period_us, vrate, busy_level, active iocg count, usage percentage and running state. It fires every period the controller runs, including steady states, plus one final tick before the controller goes idle, which makes dormancy (e.g. a device saturated entirely by uncharged IO) directly visible. At the default period this is a couple of events per second per device; the cost is zero while the static key is off. Signed-off-by: Tao Cui --- block/blk-iocost.c | 5 +++++ include/trace/events/iocost.h | 40 +++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 182601ad783f..bccdbd2496d8 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -2244,6 +2244,7 @@ static void ioc_timer_fn(struct timer_list *timer) struct ioc_now now; LIST_HEAD(surpluses); int nr_debtors, nr_shortages =3D 0, nr_lagging =3D 0; + int nr_active =3D 0; u64 usage_us_sum =3D 0; u32 ppm_rthr; u32 ppm_wthr; @@ -2280,6 +2281,8 @@ static void ioc_timer_fn(struct timer_list *timer) u64 vdone, vtime, usage_us; u32 hw_active, hw_inuse; =20 + nr_active++; + /* * Collect unused and wind vtime closer to vnow to prevent * iocgs from accumulating a large amount of budget. @@ -2466,6 +2469,8 @@ static void ioc_timer_fn(struct timer_list *timer) ioc_refresh_vrate(ioc, &now); } =20 + trace_iocost_ioc_tick(ioc, nr_active, usage_us_sum); + spin_unlock_irq(&ioc->lock); } =20 diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h index e772b1bc60d6..2b9ff348a4f5 100644 --- a/include/trace/events/iocost.h +++ b/include/trace/events/iocost.h @@ -178,6 +178,46 @@ TRACE_EVENT(iocost_ioc_vrate_adj, ) ); =20 +/* + * Periodic per-device summary, emitted once per period from the tail of + * ioc_timer_fn(). Unlike the state-change events above, this fires every + * period the controller is running, including steady states, and carries + * the overall controller state so basic monitoring doesn't require drgn. + */ +TRACE_EVENT(iocost_ioc_tick, + + TP_PROTO(struct ioc *ioc, int nr_active, u64 usage_us_sum), + + TP_ARGS(ioc, nr_active, usage_us_sum), + + TP_STRUCT__entry ( + __string(devname, ioc_name(ioc)) + __field(u32, period_us) + __field(u64, vrate) + __field(int, busy_level) + __field(int, nr_active) + __field(u32, usage_pct) + __field(int, running) + ), + + TP_fast_assign( + __assign_str(devname); + __entry->period_us =3D ioc->period_us; + __entry->vrate =3D ioc->vtime_base_rate; + __entry->busy_level =3D ioc->busy_level; + __entry->nr_active =3D nr_active; + __entry->usage_pct =3D ioc->period_us ? + div_u64(usage_us_sum * 100, ioc->period_us) : 0; + __entry->running =3D ioc->running; + ), + + TP_printk("[%s] period=3D%uus vrate=3D%llu busy=3D%d active=3D%d usage=3D= %u%% running=3D%d", + __get_str(devname), __entry->period_us, __entry->vrate, + __entry->busy_level, __entry->nr_active, __entry->usage_pct, + __entry->running + ) +); + TRACE_EVENT(iocost_iocg_forgive_debt, =20 TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now, --=20 2.43.0 From nobody Fri Sep 25 16:50:48 2026 Received: from mta1.migadu.com (out-84.mta1.migadu.com [95.215.58.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07957492E22 for ; Thu, 10 Sep 2026 12:58:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.84 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045133; cv=none; b=iWG3NmUzL1T9pcCPNZPffHejGRBht8IJDM5V0yHtutVtorJ3MQkUF5uhU9iBvazoEoRhVwxE9p1TJjKDKE5m4Y7deVYO1P8EpRH1ob+lJsjBs0Iybs3HWPUXCBR00Z/fpWGBtWgu7EJchnInOJsI1umazZETDjnTCI2OpqeNl2w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045133; c=relaxed/simple; bh=kuqHBkMBo3V9YxQhfmvAdSRAURGFEJi13xW6az8I9eQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=moWSt5WQqu4+raD6yBYTE4b1mJ1ViPOEKFa3ldjag5oZEZVysa+w1b7967QojGAfvoHwuTDRes6XNLAPCTUXnQUCYhm/XALJqWUx2KNVW1jN8uWZOp6WwNDr37itHNJ5T8VY+TB7Yw+FsmtiNMBowxgrOFH+gb0gENb3s0Q6CCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=c949H3Q9; arc=none smtp.client-ip=95.215.58.84 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="c949H3Q9" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=kuqHBkMBo3V9YxQhfmvAdSRAURGFEJi13xW6az8I9eQ=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789045128; v=1; x=1789649928; b=c949H3Q9PbRB72Uy+dGsmRmUNkOAaYUX8Sc8tbx/AzkaRmCLQc+oDkIRVqn79FjgGfUtKsHk yecC27LjK9cmkO81mYQz1/CHuvfBaYK33JsAFh8yekESXLdBELFhP1zXmMGO+ow4CnFRz+IJX7M ePvdgBtn068YrWEYRIuX6Rto= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id d248a60d230e97a6; Thu, 10 Sep 2026 12:58:48 +0000 X-Mizu-Trace-ID: d248a60d230e97a6 X-Migadu-Flow: FLOW_OUT From: Tao Cui To: tj@kernel.org, josef@toxicopanda.com, axboe@kernel.dk Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net, linux-kselftest@vger.kernel.org, cui.tao@linux.dev, cuitao@kylinos.cn Subject: [RFC PATCH v2 4/5] selftests/bpf: add multi-stream sequentiality example model Date: Thu, 10 Sep 2026 20:58:16 +0800 Message-ID: <20260910125817.223354-5-cui.tao@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260910125817.223354-1-cui.tao@linux.dev> References: <20260910125817.223354-1-cui.tao@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tao Cui Add a second example cost model which replaces the builtin single-cursor sequentiality heuristic with a per-cgroup table of stream slots: an IO is sequential iff its sector matches the expected next sector of any tracked stream. Interleaved sequential readers in one cgroup keep their own slots instead of ping-ponging a single cursor, and random IO inside a hot window rarely matches a moving expectation. Merged bios skip the base cost but still advance the matched stream position, so a merge at the expected sector does not make the following new IO look random. Stream state lives in a CGRP_STORAGE map keyed by the cgroup of the blkcg argument, following the cgroup lifetime; there is no fixed-size registry to exhaust. Measured (QEMU, virtio-blk with the HDD profile, 4k IOs, w=3D1000): two sequential readers in one cgroup are priced 1961us/op by the builtin model (judged random) and 23us/op by this model (judged sequential), the completed IO count rises from 4495 to 207505; random IO inside an 8M window is priced 24us/op by builtin (undercharge) and 2607us/op by this model; single-stream sequential and whole-disk random pricing are unchanged. --- tools/testing/selftests/bpf/progs/iocost_ms.c | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/iocost_ms.c diff --git a/tools/testing/selftests/bpf/progs/iocost_ms.c b/tools/testing/= selftests/bpf/progs/iocost_ms.c new file mode 100644 index 000000000000..3cc57c03d3c8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/iocost_ms.c @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Example multi-stream sequentiality detection cost model. + * + * The builtin model keeps a single cursor per cgroup, so two + * interleaved sequential readers in one cgroup are all priced random + * (measured 89x overcharge, 12.9x throughput collapse), while random + * IO inside a hot window smaller than the 16MB seek threshold is + * priced sequential (measured 107x undercharge). This model replaces + * the single cursor with a per-cgroup table of stream slots: an IO is + * sequential iff its sector matches the expected next sector of any + * tracked stream. Interleaved streams keep their own slots, and + * windowed random IO rarely matches a moving expectation. + * + * Stream state lives in a CGRP_STORAGE map, so it is created and + * freed with the cgroup. The model implements the full builtin + * linear formula itself, including flush pricing. + */ +#include "vmlinux.h" +#include +#include + +/* VTIME_PER_SEC comes from vmlinux.h (a BTF enum constant) */ +#define IOC_PAGE_SIZE 4096 +#define IOC_SECT_TO_PAGE_SHIFT 3 /* 512B sectors to 4kB pages */ +#define LCOEF_RANDIO_PAGES 4096 /* 16MB seek threshold */ +#define IOCOST_COST_F_MERGE (1ULL << 0) /* not in BTF: a plain macro */ +#define IOCOST_REQ_OP_MASK 0xff /* REQ_OP_MASK, not in BTF */ + +/* + * DIV64_U64_ROUND_UP / DIV_ROUND_UP_ULL equivalents, folded at + * compile time + */ +#define RU(x, y) ((x) / (y) + (((x) % (y)) ? 1 : 0)) + +#define RBPS 174019176ULL +#define RSEQIOPS 41708ULL +#define RRANDIOPS 370ULL +#define WBPS 178075866ULL +#define WSEQIOPS 42705ULL +#define WRANDIOPS 378ULL + +#define RPAGE (RU(VTIME_PER_SEC, RU(RBPS, IOC_PAGE_SIZE))) +#define RSEQIO (RU(VTIME_PER_SEC, RSEQIOPS) - RPAGE) +#define RRANDIO (RU(VTIME_PER_SEC, RRANDIOPS) - RPAGE) +#define WPAGE (RU(VTIME_PER_SEC, RU(WBPS, IOC_PAGE_SIZE))) +#define WSEQIO (RU(VTIME_PER_SEC, WSEQIOPS) - WPAGE) +#define WRANDIO (RU(VTIME_PER_SEC, WRANDIOPS) - WPAGE) + +#define NSLOTS 4 + +struct streams { + __u64 expected[NSLOTS]; /* next expected sector, per stream */ + __u64 stamp[NSLOTS]; /* LRU stamp, 0 =3D empty */ +}; + +/* + * per-cgroup stream table: keyed by the cgroup, freed with it + */ +struct { + __uint(type, BPF_MAP_TYPE_CGRP_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct streams); +} stream_tab SEC(".maps"); + +SEC("struct_ops") +u64 BPF_PROG(iocost_ms_calc_cost, u64 opf, u64 nbytes, u64 sector, + struct blkcg *blkcg, u64 model_flags) +{ + struct streams *s; + u64 pages, base, coef_page, randio, advance, now; + u32 i, victim =3D 0, found =3D 0xFFFFFFFF; + + if ((opf & IOCOST_REQ_OP_MASK) =3D=3D REQ_OP_READ) { + base =3D RSEQIO; coef_page =3D RPAGE; randio =3D RRANDIO; + } else if ((opf & IOCOST_REQ_OP_MASK) =3D=3D REQ_OP_WRITE) { + base =3D WSEQIO; coef_page =3D WPAGE; randio =3D WRANDIO; + } else { + /* + * a fully owning model must price every op; unknown + * ops are priced as a single page write + */ + base =3D 0; coef_page =3D WPAGE; randio =3D 0; + } + advance =3D RU(nbytes, 512); /* sectors */ + + s =3D bpf_cgrp_storage_get(&stream_tab, blkcg->css.cgroup, NULL, + BPF_LOCAL_STORAGE_GET_F_CREATE); + if (!s) + return base + RU(nbytes, IOC_PAGE_SIZE) * coef_page; + + now =3D bpf_ktime_get_ns(); + for (i =3D 0; i < NSLOTS; i++) { + if (s->expected[i] =3D=3D sector && s->stamp[i]) { + found =3D i; + break; + } + } + if (found !=3D 0xFFFFFFFF) { + /* sequential: keep the seq base from the op branch */ + s->expected[found] =3D sector + advance; + s->stamp[found] =3D now; + } else { + base =3D randio; + for (i =3D 1; i < NSLOTS; i++) { + if (s->stamp[i] < s->stamp[victim]) + victim =3D i; + } + s->expected[victim] =3D sector + advance; + s->stamp[victim] =3D now; + } + + pages =3D RU(nbytes, IOC_PAGE_SIZE); + if (!pages) + pages =3D 1; /* dataless flush: one page */ + if (model_flags & IOCOST_COST_F_MERGE) { + /* + * merged bios skip the base cost but still advance + * the stream position above, so a merge at the + * expected sector does not make the following new IO + * look random + */ + base =3D 0; + } + + return base + pages * coef_page; +} + +SEC(".struct_ops") +struct iocost_model_ops iocost_ms =3D { + .calc_cost =3D (void *)iocost_ms_calc_cost, + .name =3D "iocost_ms", +}; + +char LICENSE[] SEC("license") =3D "GPL"; --=20 2.43.0 From nobody Fri Sep 25 16:50:48 2026 Received: from mta0.migadu.com (out-192.mta0.migadu.com [91.218.175.192]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96D62493D3C for ; Thu, 10 Sep 2026 12:58:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.192 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045138; cv=none; b=X08+yd7y6z+1Si23lrlB7v8Wo2HotP0LnaK/s3qRoVJaNVD5lhh47yReEWAqGYv7IviK2LvvFhsTEM08+MNLWphzlqLqtECnCLE/vEezDAoB5Dbda0g7OclQ+92/Dcic03ocW0Y40Qv/E2XlouU+N7/5xvFYJfmshKFyn4y/xPM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789045138; c=relaxed/simple; bh=FY3LXKu+5c2SOYK7Zse2tGCHfSqhOu/bDt33iTqGd4Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G6y4Emh2whsO+vu5rm5Cyu6eEgcuhfBifJfQmO87PlrrcwcUrESuEZhRawyG6fKBEsia0MHDH/Nb5cvdo4iRwWuixka3hTlC+rzPYDchZqYDQbBfrpMVg5INIW9DMIaQ3xWymxbDuPxUG6VigfoG8adzrP1dTsIP3GV0fVWeCXk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=LtkEc5vW; arc=none smtp.client-ip=91.218.175.192 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="LtkEc5vW" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=FY3LXKu+5c2SOYK7Zse2tGCHfSqhOu/bDt33iTqGd4Y=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1789045133; v=1; x=1789649933; b=LtkEc5vW2sC7ZA6b9E60TcM5egp/NT3jLhuj+fONnmi03fSWJqSaop31plixNfPFFCyLvzsp QW7q4ZwPNLlGlIuhkxr8vz+skvWY+aXBztscPurVNK5mfeGoAs829Z1MEEF9FyYfScKW7u/XLvJ G2wS4bT7nv3gvIdG1NFdGToY= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 01bb2cbd500ae1c4; Thu, 10 Sep 2026 12:58:53 +0000 X-Mizu-Trace-ID: 01bb2cbd500ae1c4 X-Migadu-Flow: FLOW_OUT From: Tao Cui To: tj@kernel.org, josef@toxicopanda.com, axboe@kernel.dk Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net, linux-kselftest@vger.kernel.org, cui.tao@linux.dev, cuitao@kylinos.cn Subject: [RFC PATCH v2 5/5] docs: cgroup-v2: document io.cost model= binding Date: Thu, 10 Sep 2026 20:58:17 +0800 Message-ID: <20260910125817.223354-6-cui.tao@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260910125817.223354-1-cui.tao@linux.dev> References: <20260910125817.223354-1-cui.tao@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tao Cui Document the named-model binding of io.cost.model in the io.cost.model section of the cgroup v2 documentation: the binding and restore syntax, the unknown-name rejection, the full-replacement semantics of a bound model and the unregister lifetime. Signed-off-by: Tao Cui --- Documentation/admin-guide/cgroup-v2.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-= guide/cgroup-v2.rst index 8d2603751c51..a6910ad7c2d0 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2124,6 +2124,17 @@ IO Interface Files parameters are written to, "ctrl" become "user" and the automatic changes are disabled. =20 + When CONFIG_BLK_CGROUP_IOCOST_BPF is enabled, "model" also accepts + the name of a registered iocost_model_ops BPF struct_ops model: + "model=3D" binds the model to the device and the model fully + replaces the builtin linear pricing, for every operation including + flushes; "model=3Dlinear" (or "ctrl=3Dauto/user") restores the builtin + model. Writing an unknown name fails with ENOENT. Unregistering + a model removes its name; devices already bound keep using it + until switched back to the builtin model, following the lifetime + model of TCP congestion control. See include/linux/blk-iocost.h + for the model interface. + When "model" is "linear", the following model parameters are defined. =20 --=20 2.43.0