From nobody Fri Oct 2 07:45:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BF3E3128D9; Tue, 4 Aug 2026 06:53:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785826432; cv=none; b=WkEtYX/EVnZNiyfFaq6gbaOzyoARjzCOCscxjOpHmtTGpqXVbIBmdtZgMR2c5fOkxngU+Wn/pm/27dIjBuoZWni4e5JD7BIoP2ogC6pkflRRGzVtTkfLzrnJJgpEeZBkBWhRBwxhhtXbdcLbCP13zy4eFwiAirSUf9BazTgNj0o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785826432; c=relaxed/simple; bh=6ktP5g+m6fZwS2nBQas1Y3Vp8ODAtg9LuSW0Od/w7NE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ADRLZPYQuENGhJtdhYiuYWYon14htnH+IX6m/rBwGcbdlT2/NqdD0f9qUEyt5d9vUQI74DR9BpbmHUg+uRARQ7wxTGOyXeLLTBa8kgmWhsmODAr83qKEQulm1Xg+jVk3CR7m9JXFXr1tpGIDk13wJ2qZmkvAu/9wXIy2ohkgPik= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YSr1DG8h; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YSr1DG8h" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A2D8F1F00A3A; Tue, 4 Aug 2026 06:53:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1785826430; bh=3ikR+jaOP+v8a0i8d+lZkMdQtKeVSnS18koPO57INZE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=YSr1DG8hVF4pQdkDHS7LCxgbsxsVCvFRBUsjgo3Db8n7uFJyplwAkALeMcFvmWzk1 +13wyQA/LAUsZMuRo4eH7Z0tNi7CdmNj9zjAT14JbbglkURPFnRI45aaIyVNf1SBW5 nXr6m5KniRl07ybPsZL6efAEsuvu3EmqgLu9yYcbY3/tSJ69UzKc67Sdek49skf0V4 smyBic7YC8M16wy3X9KqauwmPmBNQNwm3tBv8VBeujwQI1xEuhJFu/8jxeDwaHw/yD AuRkDT4IAZb+PGr3fjM72Y0TrcXbxdlKWhAyLcbZb50Zkc0+ukfw3GZWRc86Kk4s2w wGlt/BoilTG+g== From: Yu Kuai To: Jens Axboe Cc: Tejun Heo , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Jonathan Corbet , Yu Kuai , Josef Bacik , Coly Li , Kent Overstreet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , Song Liu , Dan Williams , Vishal Verma , Dave Jiang , Alison Schofield , Pankaj Gupta , Andreas Gruenbacher , Matthew Wilcox , Jan Kara , Andrew Morton , Chris Li , Kairui Song , Christoph Hellwig , Nilay Shroff , Tao Cui , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-bcache@vger.kernel.org, dm-devel@lists.linux.dev, linux-raid@vger.kernel.org, nvdimm@lists.linux.dev, virtualization@lists.linux.dev, gfs2@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 1/3] blk-cgroup: add helpers for bio cgroup state Date: Tue, 4 Aug 2026 14:53:11 +0800 Message-ID: <20260804065313.2092022-2-yukuai@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260804065313.2092022-1-yukuai@kernel.org> References: <20260804065313.2092022-1-yukuai@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yu Kuai blk-cgroup users open-code bio->bi_blkg throughout the policy and accounting paths. Some users need the blkg itself, while others immediately dereference it to get the associated blkcg. The bio release paths also open-code the CONFIG_BLK_CGROUP guarded blkg reference drop and field clear. Add bio_blkg(), bio_blkcg() and bio_clear_blkg() helpers. Convert the read-side users to use the accessors, and use bio_clear_blkg() from bio_uninit() and bio_endio() so the release path no longer needs to know about CONFIG_BLK_CGROUP. Keep the direct bio->bi_blkg stores in the association and initialization paths, as those paths still assign the stored association. This keeps the current behavior unchanged while preparing for changing what cgroup state a bio stores internally. Signed-off-by: Yu Kuai --- block/bfq-cgroup.c | 4 ++-- block/bio.c | 14 ++------------ block/blk-cgroup-fc-appid.c | 11 +++++++++-- block/blk-cgroup.c | 19 ++++++++++--------- block/blk-cgroup.h | 19 ++++++++++++++++++- block/blk-iocost.c | 6 +++--- block/blk-iolatency.c | 6 +++--- block/blk-ioprio.c | 2 +- block/blk-throttle.c | 2 +- block/blk-throttle.h | 2 +- include/linux/bio.h | 10 ++++++++++ 11 files changed, 60 insertions(+), 35 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index e82ff03bda02..7e65fe6844ee 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -363,7 +363,7 @@ void bfqg_and_blkg_put(struct bfq_group *bfqg) =20 void bfqg_stats_update_legacy_io(struct request_queue *q, struct request *= rq) { - struct bfq_group *bfqg =3D blkg_to_bfqg(rq->bio->bi_blkg); + struct bfq_group *bfqg =3D blkg_to_bfqg(bio_blkg(rq->bio)); =20 if (!bfqg) return; @@ -606,7 +606,7 @@ static void bfq_link_bfqg(struct bfq_data *bfqd, struct= bfq_group *bfqg) =20 struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio) { - struct blkcg_gq *blkg =3D bio->bi_blkg; + struct blkcg_gq *blkg =3D bio_blkg(bio); struct bfq_group *bfqg; =20 while (blkg) { diff --git a/block/bio.c b/block/bio.c index 6a2f6fc3413e..c207b248edba 100644 --- a/block/bio.c +++ b/block/bio.c @@ -179,12 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp) =20 void bio_uninit(struct bio *bio) { -#ifdef CONFIG_BLK_CGROUP - if (bio->bi_blkg) { - blkg_put(bio->bi_blkg); - bio->bi_blkg =3D NULL; - } -#endif + bio_clear_blkg(bio); if (bio_integrity(bio)) bio_integrity_free(bio); =20 @@ -1803,17 +1798,12 @@ void bio_endio(struct bio *bio) goto again; } =20 -#ifdef CONFIG_BLK_CGROUP /* * Release cgroup info. We shouldn't have to do this here, but quite * a few callers of bio_init fail to call bio_uninit, so we cover up * for that here at least for now. */ - if (bio->bi_blkg) { - blkg_put(bio->bi_blkg); - bio->bi_blkg =3D NULL; - } -#endif + bio_clear_blkg(bio); =20 if (bio->bi_end_io) bio->bi_end_io(bio); diff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c index 3ec21333f393..b2e16e9a7a6c 100644 --- a/block/blk-cgroup-fc-appid.c +++ b/block/blk-cgroup-fc-appid.c @@ -50,8 +50,15 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid); */ char *blkcg_get_fc_appid(struct bio *bio) { - if (!bio->bi_blkg || bio->bi_blkg->blkcg->fc_app_id[0] =3D=3D '\0') + struct blkcg *blkcg; + + if (!bio_blkg(bio)) return NULL; - return bio->bi_blkg->blkcg->fc_app_id; + + blkcg =3D bio_blkcg(bio); + if (blkcg->fc_app_id[0] =3D=3D '\0') + return NULL; + + return blkcg->fc_app_id; } EXPORT_SYMBOL_GPL(blkcg_get_fc_appid); diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 354637f3b158..753a3bdd0e8c 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -239,7 +239,7 @@ static void blkg_async_bio_workfn(struct work_struct *w= ork) */ void blkcg_punt_bio_submit(struct bio *bio) { - struct blkcg_gq *blkg =3D bio->bi_blkg; + struct blkcg_gq *blkg =3D bio_blkg(bio); =20 if (blkg->parent) { spin_lock(&blkg->async_bio_lock); @@ -275,9 +275,9 @@ subsys_initcall(blkcg_punt_bio_init); */ struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio) { - if (!bio || !bio->bi_blkg) + if (!bio || !bio_blkg(bio)) return NULL; - return &bio->bi_blkg->blkcg->css; + return &bio_blkcg(bio)->css; } EXPORT_SYMBOL_GPL(bio_blkcg_css); =20 @@ -2118,8 +2118,8 @@ static inline struct blkcg_gq *blkg_tryget_closest(st= ruct bio *bio, void bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css) { - if (bio->bi_blkg) - blkg_put(bio->bi_blkg); + if (bio_blkg(bio)) + blkg_put(bio_blkg(bio)); =20 if (css && css->parent) { bio->bi_blkg =3D blkg_tryget_closest(bio, css); @@ -2146,7 +2146,7 @@ void bio_associate_blkg(struct bio *bio) if (blk_op_is_passthrough(bio->bi_opf)) return; =20 - if (bio->bi_blkg) { + if (bio_blkg(bio)) { css =3D bio_blkcg_css(bio); bio_associate_blkg_from_css(bio, css); } else { @@ -2170,7 +2170,7 @@ EXPORT_SYMBOL_GPL(bio_associate_blkg); */ void bio_clone_blkg_association(struct bio *dst, struct bio *src) { - if (src->bi_blkg) + if (bio_blkg(src)) bio_associate_blkg_from_css(dst, bio_blkcg_css(src)); } EXPORT_SYMBOL_GPL(bio_clone_blkg_association); @@ -2186,7 +2186,8 @@ static int blk_cgroup_io_type(struct bio *bio) =20 void blk_cgroup_bio_start(struct bio *bio) { - struct blkcg *blkcg =3D bio->bi_blkg->blkcg; + struct blkcg_gq *blkg =3D bio_blkg(bio); + struct blkcg *blkcg =3D bio_blkcg(bio); int rwd =3D blk_cgroup_io_type(bio), cpu; struct blkg_iostat_set *bis; unsigned long flags; @@ -2199,7 +2200,7 @@ void blk_cgroup_bio_start(struct bio *bio) return; =20 cpu =3D get_cpu(); - bis =3D per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu); + bis =3D per_cpu_ptr(blkg->iostat_cpu, cpu); flags =3D u64_stats_update_begin_irqsave(&bis->sync); =20 /* diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 615390f751aa..1e80b0a73233 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -124,6 +124,11 @@ static inline struct blkcg *css_to_blkcg(struct cgroup= _subsys_state *css) return css ? container_of(css, struct blkcg, css) : NULL; } =20 +static inline struct blkcg *bio_blkcg(struct bio *bio) +{ + return bio_blkg(bio)->blkcg; +} + /* * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a * request_queue (q). This is used by blkcg policies which need to track @@ -343,6 +348,16 @@ static inline void blkg_put(struct blkcg_gq *blkg) percpu_ref_put(&blkg->refcnt); } =20 +static inline void bio_clear_blkg(struct bio *bio) +{ + struct blkcg_gq *blkg =3D bio_blkg(bio); + + if (blkg) { + blkg_put(blkg); + bio->bi_blkg =3D NULL; + } +} + /** * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants * @d_blkg: loop cursor pointing to the current descendant @@ -455,7 +470,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq *b= lkg) */ static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bi= o) { - return rq->bio->bi_blkg =3D=3D bio->bi_blkg && + return bio_blkg(rq->bio) =3D=3D bio_blkg(bio) && bio_issue_as_root_blkg(rq->bio) =3D=3D bio_issue_as_root_blkg(bio); } =20 @@ -481,6 +496,7 @@ struct blkcg_policy { struct blkcg { }; =20 +static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; } static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key)= { return NULL; } static inline void blkg_init_queue(struct request_queue *q) { } static inline int blkcg_init_disk(struct gendisk *disk) { return 0; } @@ -497,6 +513,7 @@ static inline struct blkg_policy_data *blkg_to_pd(struc= t blkcg_gq *blkg, static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) { r= eturn NULL; } static inline void blkg_get(struct blkcg_gq *blkg) { } static inline void blkg_put(struct blkcg_gq *blkg) { } +static inline void bio_clear_blkg(struct bio *bio) { } static inline void blk_cgroup_bio_start(struct bio *bio) { } static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bi= o) { return true; } =20 diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 8b2aeba2e1e3..d4470476bcd0 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -2686,7 +2686,7 @@ iocg_handle_over_budget(struct rq_qos *rqos, struct i= oc_gq *iocg, =20 static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio) { - struct blkcg_gq *blkg =3D bio->bi_blkg; + struct blkcg_gq *blkg =3D bio_blkg(bio); struct ioc *ioc =3D rqos_to_ioc(rqos); struct ioc_gq *iocg =3D blkg_to_iocg(blkg); struct ioc_now now; @@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, st= ruct bio *bio) static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq, struct bio *bio) { - struct ioc_gq *iocg =3D blkg_to_iocg(bio->bi_blkg); + struct ioc_gq *iocg =3D blkg_to_iocg(bio_blkg(bio)); struct ioc *ioc =3D rqos_to_ioc(rqos); sector_t bio_end =3D bio_end_sector(bio); struct ioc_now now; @@ -2833,7 +2833,7 @@ static void ioc_rqos_merge(struct rq_qos *rqos, struc= t request *rq, =20 static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio) { - struct ioc_gq *iocg =3D blkg_to_iocg(bio->bi_blkg); + struct ioc_gq *iocg =3D blkg_to_iocg(bio_blkg(bio)); =20 if (iocg && bio->bi_iocost_cost) atomic64_add(bio->bi_iocost_cost, &iocg->done_vtime); diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index cef02b6c5fa9..c0d8d5f6bdba 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -463,7 +463,7 @@ static void check_scale_change(struct iolatency_grp *io= lat) static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio) { struct blk_iolatency *blkiolat =3D BLKIOLATENCY(rqos); - struct blkcg_gq *blkg =3D bio->bi_blkg; + struct blkcg_gq *blkg =3D bio_blkg(bio); bool issue_as_root =3D bio_issue_as_root_blkg(bio); =20 if (!blkiolat->enabled) @@ -590,11 +590,11 @@ static void blkcg_iolatency_done_bio(struct rq_qos *r= qos, struct bio *bio) bool issue_as_root =3D bio_issue_as_root_blkg(bio); int inflight =3D 0; =20 - blkg =3D bio->bi_blkg; + blkg =3D bio_blkg(bio); if (!blkg || !bio_flagged(bio, BIO_QOS_THROTTLED)) return; =20 - iolat =3D blkg_to_lat(bio->bi_blkg); + iolat =3D blkg_to_lat(blkg); if (!iolat) return; =20 diff --git a/block/blk-ioprio.c b/block/blk-ioprio.c index 8fa8bca35062..5e3f8e49550d 100644 --- a/block/blk-ioprio.c +++ b/block/blk-ioprio.c @@ -132,7 +132,7 @@ static struct blkcg_policy ioprio_policy =3D { =20 void blkcg_set_ioprio(struct bio *bio) { - struct ioprio_blkcg *blkcg =3D blkcg_to_ioprio_blkcg(bio->bi_blkg->blkcg); + struct ioprio_blkcg *blkcg =3D blkcg_to_ioprio_blkcg(bio_blkcg(bio)); u16 prio; =20 if (!blkcg || blkcg->prio_policy =3D=3D POLICY_NO_CHANGE) diff --git a/block/blk-throttle.c b/block/blk-throttle.c index ffc3b70065d4..3828c3857900 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -1792,7 +1792,7 @@ static bool tg_within_limit(struct throtl_grp *tg, st= ruct bio *bio, bool rw) bool __blk_throtl_bio(struct bio *bio) { struct request_queue *q =3D bdev_get_queue(bio->bi_bdev); - struct blkcg_gq *blkg =3D bio->bi_blkg; + struct blkcg_gq *blkg =3D bio_blkg(bio); struct throtl_qnode *qn =3D NULL; struct throtl_grp *tg =3D blkg_to_tg(blkg); struct throtl_service_queue *sq; diff --git a/block/blk-throttle.h b/block/blk-throttle.h index 9d7a42c039a1..609a126c7ccb 100644 --- a/block/blk-throttle.h +++ b/block/blk-throttle.h @@ -173,7 +173,7 @@ static inline bool blk_should_throtl(struct bio *bio) if (!blk_throtl_activated(bio->bi_bdev->bd_queue)) return false; =20 - tg =3D blkg_to_tg(bio->bi_blkg); + tg =3D blkg_to_tg(bio_blkg(bio)); if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) { if (!bio_flagged(bio, BIO_CGROUP_ACCT)) { bio_set_flag(bio, BIO_CGROUP_ACCT); diff --git a/include/linux/bio.h b/include/linux/bio.h index 8f33f717b14f..dc4baa3602b7 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -503,12 +503,22 @@ static inline void bio_release_pages(struct bio *bio,= bool mark_dirty) disk_devt((bio)->bi_bdev->bd_disk) =20 #ifdef CONFIG_BLK_CGROUP +static inline struct blkcg_gq *bio_blkg(struct bio *bio) +{ + return bio->bi_blkg; +} + void bio_associate_blkg(struct bio *bio); void bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css); void bio_clone_blkg_association(struct bio *dst, struct bio *src); void blkcg_punt_bio_submit(struct bio *bio); #else /* CONFIG_BLK_CGROUP */ +static inline struct blkcg_gq *bio_blkg(struct bio *bio) +{ + return NULL; +} + static inline void bio_associate_blkg(struct bio *bio) { } static inline void bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css) --=20 2.51.0 From nobody Fri Oct 2 07:45:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC6C03F8714; Tue, 4 Aug 2026 06:54:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785826449; cv=none; b=B/7YzCKYdzNRJa/l2CI/F/omGRlgF5uG1AIQX/cLsxebrMkizpGiVukKebw/dR5tQl7sxaRoq76EMe2nnRzgrAKE6kRxyrYUwG9IW4V++E3tW6WjW4WgkWqeGAwUjLWOvLsJ8qafiVl4MuG0Jm1FQ5gxucYX8GevXPCBDZOfqck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785826449; c=relaxed/simple; bh=dhmnZcrOpC/RymjrtmWO6vFjGzu8mO222pyanalsQ98=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S1r4kZXhG3AvXefFD7SKhkSikb976DeRa53DzhEWubmwfd0TW0uuwJGg58pCJPEKuC4to6Sm20HZZ00a54MXM7FbMnMFZmL5ZBQ0OPc1RTwE6kobAKjQ+DNYon7GeAztnkVBezZ4yNjvwZbTj1JCq3ykZfJgW5kr9KaQKsvRxTM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Xy/tDSP+; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Xy/tDSP+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A6DCC1F000E9; Tue, 4 Aug 2026 06:53:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1785826446; bh=X0DWde6zYk+gUKI+dAPZqUpQqQiQHHuCy61d3LhBkcw=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Xy/tDSP+geeVWx3qptQzY67BXWki2gi40Z0f0byBNGTfRigSpyjS3NxrzuGXkN/GQ rF9rhl/cA/ScPI7nfdLip7m5mJ/ds0sFBA+CMliue05AK1bhUzkcZth/eAo0h2dapv PpcHMW6gwBsevSS5ZNn6KcbJArRWseWuaDEIeNrVo/HL77sxCYb+UB/ln44wJFtOk3 IpH42F2UG5gzZWH4YMFoPVHekndWSAuzihG8vOb3ZTJOQIzelgdfYvlGgmD+3At/4f 44Fpjzm+RkIQ71kZFtIYunQCPVVJIHG+lhFJ88wvoAZE+NkWcFCwWZgaj0x48MZRQp AKSut6EtJcZRA== From: Yu Kuai To: Jens Axboe Cc: Tejun Heo , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Jonathan Corbet , Yu Kuai , Josef Bacik , Coly Li , Kent Overstreet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , Song Liu , Dan Williams , Vishal Verma , Dave Jiang , Alison Schofield , Pankaj Gupta , Andreas Gruenbacher , Matthew Wilcox , Jan Kara , Andrew Morton , Chris Li , Kairui Song , Christoph Hellwig , Nilay Shroff , Tao Cui , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-bcache@vger.kernel.org, dm-devel@lists.linux.dev, linux-raid@vger.kernel.org, nvdimm@lists.linux.dev, virtualization@lists.linux.dev, gfs2@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 2/3] blk-cgroup: store blkcg in bio instead of blkg Date: Tue, 4 Aug 2026 14:53:12 +0800 Message-ID: <20260804065313.2092022-3-yukuai@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260804065313.2092022-1-yukuai@kernel.org> References: <20260804065313.2092022-1-yukuai@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yu Kuai A bio currently stores a queue-local blkg reference. This forces bio association and remap paths to look up or create a blkg even when the bio will never enter a blkcg policy. Store the blkcg css association in the bio instead, and derive the blkg from the bio's blkcg and current bdev when a policy needs it. The first successful policy lookup pins the blkg, records the pin with BIO_BLKG_REF, and drops it from bio_clear_blkcg() or when bio_set_dev() changes the lookup key. Keep lookup-only users from creating missing blkgs by using bio_blkg_lookup(), and rename the bio cgroup association helpers to match the stored blkcg state. Signed-off-by: Yu Kuai --- Documentation/admin-guide/cgroup-v2.rst | 2 +- block/bfq-cgroup.c | 14 +- block/bfq-iosched.c | 18 ++- block/bio.c | 12 +- block/blk-cgroup-fc-appid.c | 5 +- block/blk-cgroup.c | 206 +++++++++++++++--------- block/blk-cgroup.h | 23 ++- block/blk-crypto-fallback.c | 2 +- block/blk-iocost.c | 10 +- block/blk-iolatency.c | 7 +- drivers/md/bcache/request.c | 2 +- drivers/md/dm.c | 2 +- drivers/md/md.c | 2 +- drivers/nvdimm/nd_virtio.c | 2 +- fs/gfs2/lops.c | 3 +- include/linux/bio.h | 30 ++-- include/linux/blk_types.h | 9 +- include/linux/writeback.h | 2 +- mm/page_io.c | 10 +- 19 files changed, 218 insertions(+), 143 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-= guide/cgroup-v2.rst index 14b8c571c0d1..bbd79931d5ab 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -3235,7 +3235,7 @@ the configuration, the bio may be executed at a lower= priority and if the writeback session is holding shared resources, e.g. a journal entry, may lead to priority inversion. There is no one easy solution for the problem. Filesystems can try to work around specific problem -cases by skipping wbc_init_bio() and using bio_associate_blkg() +cases by skipping wbc_init_bio() and using bio_associate_blkcg() directly. =20 =20 diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index 7e65fe6844ee..3ac3b4c05402 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -363,11 +363,13 @@ void bfqg_and_blkg_put(struct bfq_group *bfqg) =20 void bfqg_stats_update_legacy_io(struct request_queue *q, struct request *= rq) { - struct bfq_group *bfqg =3D blkg_to_bfqg(bio_blkg(rq->bio)); + struct blkcg_gq *blkg =3D bio_blkg_lookup(rq->bio); + struct bfq_group *bfqg; =20 - if (!bfqg) + if (!blkg) return; =20 + bfqg =3D blkg_to_bfqg(blkg); blkg_rwstat_add(&bfqg->stats.bytes, rq->cmd_flags, blk_rq_bytes(rq)); blkg_rwstat_add(&bfqg->stats.ios, rq->cmd_flags, 1); } @@ -606,7 +608,7 @@ static void bfq_link_bfqg(struct bfq_data *bfqd, struct= bfq_group *bfqg) =20 struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio) { - struct blkcg_gq *blkg =3D bio_blkg(bio); + struct blkcg_gq *blkg =3D bio_blkg_lookup(bio); struct bfq_group *bfqg; =20 while (blkg) { @@ -614,14 +616,16 @@ struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd,= struct bio *bio) blkg =3D blkg->parent; continue; } + bfqg =3D blkg_to_bfqg(blkg); if (bfqg->pd.online) { - bio_associate_blkg_from_css(bio, &blkg->blkcg->css); + bio_associate_blkcg_from_css(bio, &blkg->blkcg->css); return bfqg; } blkg =3D blkg->parent; } - bio_associate_blkg_from_css(bio, + + bio_associate_blkcg_from_css(bio, &bfqg_to_blkg(bfqd->root_group)->blkcg->css); return bfqd->root_group; } diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 0f75301b3115..3d51d743552c 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -128,6 +128,7 @@ =20 #include "elevator.h" #include "blk.h" +#include "blk-cgroup.h" #include "blk-mq.h" #include "blk-mq-sched.h" #include "bfq-iosched.h" @@ -2452,15 +2453,15 @@ static bool bfq_bio_merge(struct request_queue *q, = struct bio *bio, struct request *free =3D NULL; bool ret; =20 +#ifdef CONFIG_BFQ_GROUP_IOSCHED + if (bic && bio_blkg_lookup(bio) =3D=3D NULL) + return false; +#endif + spin_lock_irq(&bfqd->lock); =20 if (bic) { - /* - * Make sure cgroup info is uptodate for current process before - * considering the merge. - */ bfq_bic_update_cgroup(bic, bio); - bfqd->bio_bfqq =3D bic_to_bfqq(bic, op_is_sync(bio->bi_opf), bfq_actuator_index(bfqd, bio)); } else { @@ -6245,6 +6246,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx = *hctx, struct request *rq, LIST_HEAD(free); =20 #ifdef CONFIG_BFQ_GROUP_IOSCHED + /* + * Pin the blkg used to look up bfqg. If this is the first IO for + * the blkcg on this queue, create the bfqg before holding bfqd->lock. + */ + if (rq->bio && !bio_flagged(rq->bio, BIO_BLKG_REF)) + bio_blkg(rq->bio); + if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio) bfqg_stats_update_legacy_io(q, rq); #endif diff --git a/block/bio.c b/block/bio.c index c207b248edba..db33c993c296 100644 --- a/block/bio.c +++ b/block/bio.c @@ -179,7 +179,7 @@ static inline gfp_t try_alloc_gfp(gfp_t gfp) =20 void bio_uninit(struct bio *bio) { - bio_clear_blkg(bio); + bio_clear_blkcg(bio); if (bio_integrity(bio)) bio_integrity_free(bio); =20 @@ -228,10 +228,10 @@ void bio_init(struct bio *bio, struct block_device *b= dev, struct bio_vec *table, bio->bi_end_io =3D NULL; bio->bi_private =3D NULL; #ifdef CONFIG_BLK_CGROUP - bio->bi_blkg =3D NULL; + bio->bi_blkcg =3D NULL; bio->issue_time_ns =3D 0; if (bdev) - bio_associate_blkg(bio); + bio_associate_blkcg(bio); #ifdef CONFIG_BLK_CGROUP_IOCOST bio->bi_iocost_cost =3D 0; #endif @@ -276,7 +276,7 @@ void bio_reset(struct bio *bio, struct block_device *bd= ev, blk_opf_t opf) bio->bi_io_vec =3D bv; bio->bi_bdev =3D bdev; if (bio->bi_bdev) - bio_associate_blkg(bio); + bio_associate_blkcg(bio); bio->bi_opf =3D opf; } EXPORT_SYMBOL(bio_reset); @@ -860,7 +860,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio= _src, gfp_t gfp) if (bio->bi_bdev =3D=3D bio_src->bi_bdev && bio_flagged(bio_src, BIO_REMAPPED)) bio_set_flag(bio, BIO_REMAPPED); - bio_clone_blkg_association(bio, bio_src); + bio_clone_blkcg_association(bio, bio_src); } =20 if (bio_crypt_clone(bio, bio_src, gfp) < 0) @@ -1803,7 +1803,7 @@ void bio_endio(struct bio *bio) * a few callers of bio_init fail to call bio_uninit, so we cover up * for that here at least for now. */ - bio_clear_blkg(bio); + bio_clear_blkcg(bio); =20 if (bio->bi_end_io) bio->bi_end_io(bio); diff --git a/block/blk-cgroup-fc-appid.c b/block/blk-cgroup-fc-appid.c index b2e16e9a7a6c..7589c6209989 100644 --- a/block/blk-cgroup-fc-appid.c +++ b/block/blk-cgroup-fc-appid.c @@ -50,12 +50,11 @@ EXPORT_SYMBOL_GPL(blkcg_set_fc_appid); */ char *blkcg_get_fc_appid(struct bio *bio) { - struct blkcg *blkcg; + struct blkcg *blkcg =3D bio_blkcg(bio); =20 - if (!bio_blkg(bio)) + if (!blkcg) return NULL; =20 - blkcg =3D bio_blkcg(bio); if (blkcg->fc_app_id[0] =3D=3D '\0') return NULL; =20 diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 753a3bdd0e8c..93ab57e0a9f1 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -241,13 +241,13 @@ void blkcg_punt_bio_submit(struct bio *bio) { struct blkcg_gq *blkg =3D bio_blkg(bio); =20 - if (blkg->parent) { + if (blkg && blkg->parent) { spin_lock(&blkg->async_bio_lock); bio_list_add(&blkg->async_bios, bio); spin_unlock(&blkg->async_bio_lock); queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work); } else { - /* never bounce for the root cgroup */ + /* Never bounce if there is no non-root blkg to queue on. */ submit_bio(bio); } } @@ -275,7 +275,7 @@ subsys_initcall(blkcg_punt_bio_init); */ struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio) { - if (!bio || !bio_blkg(bio)) + if (!bio || !bio_blkcg(bio)) return NULL; return &bio_blkcg(bio)->css; } @@ -2051,129 +2051,181 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 n= ow, u64 delta) atomic64_add(delta, &blkg->delay_nsec); } =20 -static inline struct blkcg_gq *blkg_lookup_tryget(struct blkcg_gq *blkg) +/* + * Return the blkg pinned by @bio through BIO_BLKG_REF. The returned blkg= is + * already owned by @bio and no extra reference is acquired. If the pinned + * blkg can't be found, fall back to the root blkg. + */ +static struct blkcg_gq *bio_pinned_blkg(struct bio *bio) { -retry: - if (blkg_tryget(blkg)) - return blkg; + struct request_queue *q =3D bdev_get_queue(bio->bi_bdev); + struct blkcg_gq *blkg; =20 - blkg =3D blkg->parent; - if (blkg) - goto retry; + rcu_read_lock(); + blkg =3D blkg_lookup(bio_blkcg(bio), q); + rcu_read_unlock(); =20 - return NULL; + if (WARN_ON_ONCE(!blkg)) + return q->root_blkg; + return blkg; } + /** - * blkg_tryget_closest - try and get a blkg ref on the closet blkg + * bio_blkg_lookup - look up a blkg associated with a bio * @bio: target bio - * @css: target css * - * As the failure mode here is to walk up the blkg tree, this ensure that = the - * blkg->parent pointers are always valid. This returns the blkg that it = ended - * up taking a reference on or %NULL if no reference was taken. + * Look up the queue-local blkg for @bio's current device and blkcg without + * creating a missing blkg. The first successful lookup pins the blkg to = @bio; + * later lookups reuse the bio-owned reference. */ -static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio, - struct cgroup_subsys_state *css) +struct blkcg_gq *bio_blkg_lookup(struct bio *bio) { - struct request_queue *q =3D bio->bi_bdev->bd_queue; - struct blkcg *blkcg =3D css_to_blkcg(css); + struct blkcg *blkcg =3D bio_blkcg(bio); + struct request_queue *q; struct blkcg_gq *blkg; =20 + if (bio_flagged(bio, BIO_BLKG_REF)) + return bio_pinned_blkg(bio); + + if (!blkcg || !bio->bi_bdev) + return NULL; + + q =3D bdev_get_queue(bio->bi_bdev); rcu_read_lock(); blkg =3D blkg_lookup(blkcg, q); - if (likely(blkg)) - blkg =3D blkg_lookup_tryget(blkg); + if (blkg && blkg_tryget(blkg)) + bio_set_flag(bio, BIO_BLKG_REF); + else + blkg =3D NULL; rcu_read_unlock(); =20 - if (blkg) - return blkg; + return blkg; +} +EXPORT_SYMBOL_GPL(bio_blkg_lookup); + +/** + * bio_put_blkg_ref - drop the blkg reference pinned by a bio + * @bio: target bio + * + * Drop the bio-owned blkg reference acquired by bio_blkg(), if any. + */ +void bio_put_blkg_ref(struct bio *bio) +{ + if (bio_flagged(bio, BIO_BLKG_REF)) { + struct blkcg_gq *blkg =3D bio_pinned_blkg(bio); + + blkg_put(blkg); + bio_clear_flag(bio, BIO_BLKG_REF); + } +} +EXPORT_SYMBOL_GPL(bio_put_blkg_ref); + +/** + * bio_blkg - look up the blkg associated with a bio + * @bio: target bio + * + * Look up the queue-local blkg for @bio's current device and blkcg. If t= his + * is the first policy use of @bio, create the missing blkg hierarchy if + * necessary, pin the exact blkg, and mark @bio so bio_clear_blkcg() can d= rop + * the reference when the bio completes. + */ +struct blkcg_gq *bio_blkg(struct bio *bio) +{ + struct blkcg *blkcg =3D bio_blkcg(bio); + struct gendisk *disk; + struct request_queue *q; + struct blkcg_gq *blkg; + + if (!blkcg || !bio->bi_bdev) + return NULL; + + if (bio_flagged(bio, BIO_BLKG_REF)) + return bio_pinned_blkg(bio); + + disk =3D bio->bi_bdev->bd_disk; + q =3D disk->queue; =20 - /* - * Fast path failed, we're probably issuing IO in this cgroup the first - * time, hold lock to create new blkg. - */ spin_lock_irq(&q->queue_lock); - blkg =3D blkg_lookup_create(blkcg, bio->bi_bdev->bd_disk); - if (blkg) - blkg =3D blkg_lookup_tryget(blkg); + blkg =3D blkg_lookup_create(blkcg, disk); + if (blkg && blkg->blkcg =3D=3D blkcg && blkg_tryget(blkg)) + bio_set_flag(bio, BIO_BLKG_REF); + else + blkg =3D NULL; spin_unlock_irq(&q->queue_lock); =20 return blkg; } +EXPORT_SYMBOL_GPL(bio_blkg); =20 /** - * bio_associate_blkg_from_css - associate a bio with a specified css + * bio_associate_blkcg_from_css - associate a bio with a specified css * @bio: target bio * @css: target css * - * Associate @bio with the blkg found by combining the css's blkg and the - * request_queue of the @bio. An association failure is handled by walkin= g up - * the blkg tree. Therefore, the blkg associated can be anything between = @blkg - * and q->root_blkg. This situation only happens when a cgroup is dying a= nd - * then the remaining bios will spill to the closest alive blkg. + * Associate @bio with the blkcg found from @css. The queue-local blkg is + * created and pinned by bio_blkg() when blkcg policies need it. * - * A reference will be taken on the blkg and will be released when @bio is + * A reference will be taken on the blkcg and will be released when @bio is * freed. */ -void bio_associate_blkg_from_css(struct bio *bio, +void bio_associate_blkcg_from_css(struct bio *bio, struct cgroup_subsys_state *css) { - if (bio_blkg(bio)) - blkg_put(bio_blkg(bio)); + struct blkcg *blkcg; =20 - if (css && css->parent) { - bio->bi_blkg =3D blkg_tryget_closest(bio, css); - } else { - blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg); - bio->bi_blkg =3D bdev_get_queue(bio->bi_bdev)->root_blkg; - } + if (!css || !css->parent) + css =3D &blkcg_root.css; + + blkcg =3D css_to_blkcg(css); + if (bio_blkcg(bio) =3D=3D blkcg) + return; + + css_get(css); + bio_clear_blkcg(bio); + bio->bi_blkcg =3D blkcg; } -EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css); +EXPORT_SYMBOL_GPL(bio_associate_blkcg_from_css); =20 /** - * bio_associate_blkg - associate a bio with a blkg + * bio_associate_blkcg - associate a bio with a blkcg * @bio: target bio * - * Associate @bio with the blkg found from the bio's css and request_queue. - * If one is not found, bio_lookup_blkg() creates the blkg. If a blkg is - * already associated, the css is reused and association redone as the - * request_queue may have changed. + * Associate @bio with the blkcg found from the bio's css. If a blkcg is + * already associated, keep it as blkcg association is not queue-local. */ -void bio_associate_blkg(struct bio *bio) +void bio_associate_blkcg(struct bio *bio) { struct cgroup_subsys_state *css; =20 if (blk_op_is_passthrough(bio->bi_opf)) return; =20 - if (bio_blkg(bio)) { - css =3D bio_blkcg_css(bio); - bio_associate_blkg_from_css(bio, css); - } else { - rcu_read_lock(); - css =3D blkcg_css(); - if (!css_tryget_online(css)) - css =3D NULL; - rcu_read_unlock(); + if (bio_blkcg(bio)) + return; =20 - bio_associate_blkg_from_css(bio, css); - if (css) - css_put(css); - } + rcu_read_lock(); + css =3D blkcg_css(); + if (!css_tryget_online(css)) + css =3D NULL; + rcu_read_unlock(); + + bio_associate_blkcg_from_css(bio, css); + if (css) + css_put(css); } -EXPORT_SYMBOL_GPL(bio_associate_blkg); +EXPORT_SYMBOL_GPL(bio_associate_blkcg); =20 /** - * bio_clone_blkg_association - clone blkg association from src to dst bio + * bio_clone_blkcg_association - clone blkcg association from src to dst b= io * @dst: destination bio * @src: source bio */ -void bio_clone_blkg_association(struct bio *dst, struct bio *src) +void bio_clone_blkcg_association(struct bio *dst, struct bio *src) { - if (bio_blkg(src)) - bio_associate_blkg_from_css(dst, bio_blkcg_css(src)); + if (bio_blkcg(src)) + bio_associate_blkcg_from_css(dst, bio_blkcg_css(src)); } -EXPORT_SYMBOL_GPL(bio_clone_blkg_association); +EXPORT_SYMBOL_GPL(bio_clone_blkcg_association); =20 static int blk_cgroup_io_type(struct bio *bio) { @@ -2186,19 +2238,25 @@ static int blk_cgroup_io_type(struct bio *bio) =20 void blk_cgroup_bio_start(struct bio *bio) { - struct blkcg_gq *blkg =3D bio_blkg(bio); struct blkcg *blkcg =3D bio_blkcg(bio); + struct blkcg_gq *blkg; int rwd =3D blk_cgroup_io_type(bio), cpu; struct blkg_iostat_set *bis; unsigned long flags; =20 if (!cgroup_subsys_on_dfl(io_cgrp_subsys)) return; + if (!blkcg) + return; =20 /* Root-level stats are sourced from system-wide IO stats */ if (!cgroup_parent(blkcg->css.cgroup)) return; =20 + blkg =3D bio_blkg_lookup(bio); + if (!blkg) + return; + cpu =3D get_cpu(); bis =3D per_cpu_ptr(blkg->iostat_cpu, cpu); flags =3D u64_stats_update_begin_irqsave(&bis->sync); diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 1e80b0a73233..b6fb85db4d3d 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -126,7 +126,7 @@ static inline struct blkcg *css_to_blkcg(struct cgroup_= subsys_state *css) =20 static inline struct blkcg *bio_blkcg(struct bio *bio) { - return bio_blkg(bio)->blkcg; + return bio->bi_blkcg; } =20 /* @@ -281,6 +281,9 @@ static inline struct blkcg_gq *blkg_lookup(struct blkcg= *blkcg, return blkg; } =20 +struct blkcg_gq *bio_blkg_lookup(struct bio *bio); +struct blkcg_gq *bio_blkg(struct bio *bio); + /** * blkg_to_pd - get policy private data * @blkg: blkg of interest @@ -348,13 +351,15 @@ static inline void blkg_put(struct blkcg_gq *blkg) percpu_ref_put(&blkg->refcnt); } =20 -static inline void bio_clear_blkg(struct bio *bio) +static inline void bio_clear_blkcg(struct bio *bio) { - struct blkcg_gq *blkg =3D bio_blkg(bio); + struct blkcg *blkcg =3D bio_blkcg(bio); + + bio_put_blkg_ref(bio); =20 - if (blkg) { - blkg_put(blkg); - bio->bi_blkg =3D NULL; + if (blkcg) { + css_put(&blkcg->css); + bio->bi_blkcg =3D NULL; } } =20 @@ -470,7 +475,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq *b= lkg) */ static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bi= o) { - return bio_blkg(rq->bio) =3D=3D bio_blkg(bio) && + return bio_blkcg(rq->bio) =3D=3D bio_blkcg(bio) && bio_issue_as_root_blkg(rq->bio) =3D=3D bio_issue_as_root_blkg(bio); } =20 @@ -497,6 +502,8 @@ struct blkcg { }; =20 static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; } +static inline struct blkcg_gq *bio_blkg_lookup(struct bio *bio) { return N= ULL; } +static inline struct blkcg_gq *bio_blkg(struct bio *bio) { return NULL; } static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key)= { return NULL; } static inline void blkg_init_queue(struct request_queue *q) { } static inline int blkcg_init_disk(struct gendisk *disk) { return 0; } @@ -513,7 +520,7 @@ static inline struct blkg_policy_data *blkg_to_pd(struc= t blkcg_gq *blkg, static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) { r= eturn NULL; } static inline void blkg_get(struct blkcg_gq *blkg) { } static inline void blkg_put(struct blkcg_gq *blkg) { } -static inline void bio_clear_blkg(struct bio *bio) { } +static inline void bio_clear_blkcg(struct bio *bio) { } static inline void blk_cgroup_bio_start(struct bio *bio) { } static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bi= o) { return true; } =20 diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c index 2a5c52ab74b4..5ef4baab444b 100644 --- a/block/blk-crypto-fallback.c +++ b/block/blk-crypto-fallback.c @@ -187,7 +187,7 @@ static struct bio *blk_crypto_alloc_enc_bio(struct bio = *bio_src, bio->bi_write_hint =3D bio_src->bi_write_hint; bio->bi_write_stream =3D bio_src->bi_write_stream; bio->bi_iter.bi_sector =3D bio_src->bi_iter.bi_sector; - bio_clone_blkg_association(bio, bio_src); + bio_clone_blkcg_association(bio, bio_src); =20 /* * Move page array up in the allocated memory for the bio vecs as far as diff --git a/block/blk-iocost.c b/block/blk-iocost.c index d4470476bcd0..62ffd759bb95 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -2775,7 +2775,7 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, st= ruct bio *bio) static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq, struct bio *bio) { - struct ioc_gq *iocg =3D blkg_to_iocg(bio_blkg(bio)); + struct ioc_gq *iocg =3D blkg_to_iocg(bio_blkg_lookup(bio)); struct ioc *ioc =3D rqos_to_ioc(rqos); sector_t bio_end =3D bio_end_sector(bio); struct ioc_now now; @@ -2833,9 +2833,13 @@ static void ioc_rqos_merge(struct rq_qos *rqos, stru= ct request *rq, =20 static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio) { - struct ioc_gq *iocg =3D blkg_to_iocg(bio_blkg(bio)); + struct ioc_gq *iocg; + + if (!bio->bi_iocost_cost) + return; =20 - if (iocg && bio->bi_iocost_cost) + iocg =3D blkg_to_iocg(bio_blkg_lookup(bio)); + if (iocg) atomic64_add(bio->bi_iocost_cost, &iocg->done_vtime); } =20 diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index c0d8d5f6bdba..7ad18a538d7e 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -590,8 +590,11 @@ static void blkcg_iolatency_done_bio(struct rq_qos *rq= os, struct bio *bio) bool issue_as_root =3D bio_issue_as_root_blkg(bio); int inflight =3D 0; =20 - blkg =3D bio_blkg(bio); - if (!blkg || !bio_flagged(bio, BIO_QOS_THROTTLED)) + if (!bio_flagged(bio, BIO_QOS_THROTTLED)) + return; + + blkg =3D bio_blkg_lookup(bio); + if (!blkg) return; =20 iolat =3D blkg_to_lat(blkg); diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index 3fa3b13a410f..c0f945b8d941 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -848,7 +848,7 @@ static CLOSURE_CALLBACK(cached_dev_read_done) s->iop.bio->bi_iter.bi_sector =3D s->cache_miss->bi_iter.bi_sector; s->iop.bio->bi_iter.bi_size =3D s->insert_bio_sectors << 9; - bio_clone_blkg_association(s->iop.bio, s->cache_miss); + bio_clone_blkcg_association(s->iop.bio, s->cache_miss); bch_bio_map(s->iop.bio, NULL); =20 bio_copy_data(s->cache_miss, s->iop.bio); diff --git a/drivers/md/dm.c b/drivers/md/dm.c index d413bfaf3527..cd68eec77f5a 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1373,7 +1373,7 @@ void dm_submit_bio_remap(struct bio *clone, struct bi= o *tgt_clone) if (!tgt_clone) tgt_clone =3D clone; =20 - bio_clone_blkg_association(tgt_clone, io->orig_bio); + bio_clone_blkcg_association(tgt_clone, io->orig_bio); =20 /* * Account io->origin_bio to DM dev on behalf of target diff --git a/drivers/md/md.c b/drivers/md/md.c index d1465bcd86c8..af55f8efa46b 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -9355,7 +9355,7 @@ void md_submit_discard_bio(struct mddev *mddev, struc= t md_rdev *rdev, return; =20 bio_chain(discard_bio, bio); - bio_clone_blkg_association(discard_bio, bio); + bio_clone_blkcg_association(discard_bio, bio); mddev_trace_remap(mddev, discard_bio, bio->bi_iter.bi_sector); submit_bio_noacct(discard_bio); } diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c index 4176046627be..54e4adb0ccb7 100644 --- a/drivers/nvdimm/nd_virtio.c +++ b/drivers/nvdimm/nd_virtio.c @@ -121,7 +121,7 @@ int async_pmem_flush(struct nd_region *nd_region, struc= t bio *bio) =20 if (!child) return -ENOMEM; - bio_clone_blkg_association(child, bio); + bio_clone_blkcg_association(child, bio); child->bi_iter.bi_sector =3D -1; bio_chain(child, bio); submit_bio(child); diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index 6dabe73ad790..6512dbd9516f 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -484,7 +484,7 @@ static struct bio *gfs2_chain_bio(struct bio *prev, uns= igned int nr_iovecs, struct bio *new; =20 new =3D bio_alloc(prev->bi_bdev, nr_iovecs, opf, GFP_NOIO); - bio_clone_blkg_association(new, prev); + bio_clone_blkcg_association(new, prev); new->bi_iter.bi_sector =3D sector; bio_chain(new, prev); submit_bio(prev); @@ -1114,4 +1114,3 @@ const struct gfs2_log_operations *gfs2_log_ops[] =3D { &gfs2_revoke_lops, NULL, }; - diff --git a/include/linux/bio.h b/include/linux/bio.h index dc4baa3602b7..e5799fdf431d 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -503,28 +503,20 @@ static inline void bio_release_pages(struct bio *bio,= bool mark_dirty) disk_devt((bio)->bi_bdev->bd_disk) =20 #ifdef CONFIG_BLK_CGROUP -static inline struct blkcg_gq *bio_blkg(struct bio *bio) -{ - return bio->bi_blkg; -} - -void bio_associate_blkg(struct bio *bio); -void bio_associate_blkg_from_css(struct bio *bio, +void bio_associate_blkcg(struct bio *bio); +void bio_associate_blkcg_from_css(struct bio *bio, struct cgroup_subsys_state *css); -void bio_clone_blkg_association(struct bio *dst, struct bio *src); +void bio_clone_blkcg_association(struct bio *dst, struct bio *src); +void bio_put_blkg_ref(struct bio *bio); void blkcg_punt_bio_submit(struct bio *bio); #else /* CONFIG_BLK_CGROUP */ -static inline struct blkcg_gq *bio_blkg(struct bio *bio) -{ - return NULL; -} - -static inline void bio_associate_blkg(struct bio *bio) { } -static inline void bio_associate_blkg_from_css(struct bio *bio, +static inline void bio_associate_blkcg(struct bio *bio) { } +static inline void bio_associate_blkcg_from_css(struct bio *bio, struct cgroup_subsys_state *css) { } -static inline void bio_clone_blkg_association(struct bio *dst, +static inline void bio_clone_blkcg_association(struct bio *dst, struct bio *src) { } +static inline void bio_put_blkg_ref(struct bio *bio) { } static inline void blkcg_punt_bio_submit(struct bio *bio) { submit_bio(bio); @@ -534,10 +526,12 @@ static inline void blkcg_punt_bio_submit(struct bio *= bio) static inline void bio_set_dev(struct bio *bio, struct block_device *bdev) { bio_clear_flag(bio, BIO_REMAPPED); - if (bio->bi_bdev !=3D bdev) + if (bio->bi_bdev !=3D bdev) { + bio_put_blkg_ref(bio); bio_clear_flag(bio, BIO_BPS_THROTTLED); + } bio->bi_bdev =3D bdev; - bio_associate_blkg(bio); + bio_associate_blkcg(bio); } =20 /* diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 8808ee76e73c..5f95c2e0e90b 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -246,12 +246,10 @@ struct bio { void *bi_private; #ifdef CONFIG_BLK_CGROUP /* - * Represents the association of the css and request_queue for the bio. - * If a bio goes direct to device, it will not have a blkg as it will - * not have a request_queue associated with it. The reference is put - * on release of the bio. + * Represents the blkcg css association for the bio. The reference is + * put on release of the bio. */ - struct blkcg_gq *bi_blkg; + struct blkcg *bi_blkcg; /* Time that this bio was issued. */ u64 issue_time_ns; #ifdef CONFIG_BLK_CGROUP_IOCOST @@ -309,6 +307,7 @@ enum { BIO_TRACE_COMPLETION, /* bio_endio() should trace the final completion * of this bio. */ BIO_CGROUP_ACCT, /* has been accounted to a cgroup */ + BIO_BLKG_REF, /* bio pins the associated blkg */ BIO_QOS_THROTTLED, /* bio went through rq_qos throttle path */ /* * This bio has completed bps throttling at the single tg granularity, diff --git a/include/linux/writeback.h b/include/linux/writeback.h index 62552a2ce5b9..4f869fe9cc90 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -262,7 +262,7 @@ static inline void wbc_init_bio(struct writeback_contro= l *wbc, struct bio *bio) * regular writeback instead of writing things out itself. */ if (wbc->wb) - bio_associate_blkg_from_css(bio, wbc->wb->blkcg_css); + bio_associate_blkcg_from_css(bio, wbc->wb->blkcg_css); } =20 void inode_switch_wbs_work_fn(struct work_struct *work); diff --git a/mm/page_io.c b/mm/page_io.c index b23f494fcc83..112e50475605 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -316,7 +316,7 @@ static inline void count_swpout_vm_event(struct folio *= folio) } =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) -static void bio_associate_blkg_from_page(struct bio *bio, struct folio *fo= lio) +static void bio_associate_blkcg_from_page(struct bio *bio, struct folio *f= olio) { struct cgroup_subsys_state *css; struct mem_cgroup *memcg; @@ -331,12 +331,12 @@ static void bio_associate_blkg_from_page(struct bio *= bio, struct folio *folio) css =3D NULL; rcu_read_unlock(); =20 - bio_associate_blkg_from_css(bio, css); + bio_associate_blkcg_from_css(bio, css); if (css) css_put(css); } #else -#define bio_associate_blkg_from_page(bio, folio) do { } while (0) +#define bio_associate_blkcg_from_page(bio, folio) do { } while (0) #endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */ =20 struct swap_iocb { @@ -436,7 +436,7 @@ static void swap_writepage_bdev_sync(struct folio *foli= o, bio.bi_iter.bi_sector =3D swap_folio_sector(folio); bio_add_folio_nofail(&bio, folio, folio_size(folio), 0); =20 - bio_associate_blkg_from_page(&bio, folio); + bio_associate_blkcg_from_page(&bio, folio); count_swpout_vm_event(folio); =20 folio_start_writeback(folio); @@ -456,7 +456,7 @@ static void swap_writepage_bdev_async(struct folio *fol= io, bio->bi_end_io =3D end_swap_bio_write; bio_add_folio_nofail(bio, folio, folio_size(folio), 0); =20 - bio_associate_blkg_from_page(bio, folio); + bio_associate_blkcg_from_page(bio, folio); count_swpout_vm_event(folio); folio_start_writeback(folio); folio_unlock(folio); --=20 2.51.0 From nobody Fri Oct 2 07:45:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4B8A3F8714; Tue, 4 Aug 2026 06:54:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785826464; cv=none; b=TiLAAiaL2FlU5rWlY8Di2BydzfTljNme6A3Gf4jLlAEc1T1rrG7n4q0lizhohezWoT6kP1cL9yMVbOO35bBI9P/gFFqJehEtPv8Ks0bTnlZQa9ZWbydOLaEGChridGP/Wb9GGxaZlSO52f5QQ4KYE2QFArRoldGkCeRsZo/JfIE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785826464; c=relaxed/simple; bh=LGmf2KEi1vOYJnUZpZkGgA11jUahqBwgWVCT9Sl6bzQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kyT35DAmIoo90yH+0IvlSEjlbFsGWWaIQyDweHO9KlJ8O5mfjbvJ7MJo4iIEMGdpl5Ib7Wvki9jOtzsv81InoJVwTqZE9tevkf5+eiwvEtNxNdd82JyCuC5/O4ptr7bMuA6B4XwUQxY8TFmwEDL8s6/FK2Xvl3eIlPpXlv1ArBo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EhpRjq1T; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EhpRjq1T" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1950E1F00A3A; Tue, 4 Aug 2026 06:54:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1785826462; bh=0W+d2BSTkxNT+SthfX82FpSr/h0ZeS2Jo82xnGDi01I=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=EhpRjq1T7kHiQezjtfNCoscixE9MILZKN5AQAeuLfpl0CdFGwVYQTGV+wNKnrHM1r JrCkeWZ6d06vYOvkvnShsD053dLwqlqovCi8VdrTs6xOXSWs5xZyvvJv0+eeTuvsrr UjxsnEfXKMZbAY4xrGUC1ZY81kkxGCs6MR03zQf4P0zM/zg8Ufripxogr4rVuRBK6o J4vbWfQENrvlcBCqfeojeulJL4POIHcg1HvJbPr8DzbRolG6hlwUW458Nvj71gn44d NxB5XhSONuT9fI07DPIJdGnu5q8FNDVjyU1hLJPrJ5um9XXuJiqETrmLraKLEI7cFj xqC7LknjuY70A== From: Yu Kuai To: Jens Axboe Cc: Tejun Heo , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Jonathan Corbet , Yu Kuai , Josef Bacik , Coly Li , Kent Overstreet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , Song Liu , Dan Williams , Vishal Verma , Dave Jiang , Alison Schofield , Pankaj Gupta , Andreas Gruenbacher , Matthew Wilcox , Jan Kara , Andrew Morton , Chris Li , Kairui Song , Christoph Hellwig , Nilay Shroff , Tao Cui , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-bcache@vger.kernel.org, dm-devel@lists.linux.dev, linux-raid@vger.kernel.org, nvdimm@lists.linux.dev, virtualization@lists.linux.dev, gfs2@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 3/3] blk-cgroup: move async bio punt state to blkcg Date: Tue, 4 Aug 2026 14:53:13 +0800 Message-ID: <20260804065313.2092022-4-yukuai@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260804065313.2092022-1-yukuai@kernel.org> References: <20260804065313.2092022-1-yukuai@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yu Kuai blkcg_punt_bio_submit() currently queues punted bios on blkg->async_bios, so it has to call bio_blkg() to find or create a queue-local blkg. Bios now carry and pin the blkcg css, so punted bio lifetime no longer needs to be anchored by a blkg. Keeping the punt state in blkg can instantiate a blkg even when no blkcg policy is enabled, just to bounce submission from a shared kthread. Move async_bio_lock, async_bios and async_bio_work to struct blkcg, and queue punted bios on bio_blkcg() for non-root cgroups. Root or unassociated bios are submitted directly. This preserves the priority-inversion avoidance while preventing blkcg_punt_bio_submit() from creating blkgs that are not needed by any policy. Signed-off-by: Yu Kuai --- block/blk-cgroup.c | 44 +++++++++++++++++++++----------------------- block/blk-cgroup.h | 14 ++++++-------- 2 files changed, 27 insertions(+), 31 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 93ab57e0a9f1..236220db8578 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -170,10 +170,6 @@ static void __blkg_release(struct rcu_head *rcu) { struct blkcg_gq *blkg =3D container_of(rcu, struct blkcg_gq, rcu_head); =20 -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - WARN_ON(!bio_list_empty(&blkg->async_bios)); -#endif - blkg_free(blkg); } =20 @@ -206,19 +202,18 @@ static void blkg_release(struct percpu_ref *ref) #ifdef CONFIG_BLK_CGROUP_PUNT_BIO static struct workqueue_struct *blkcg_punt_bio_wq; =20 -static void blkg_async_bio_workfn(struct work_struct *work) +static void blkcg_async_bio_workfn(struct work_struct *work) { - struct blkcg_gq *blkg =3D container_of(work, struct blkcg_gq, - async_bio_work); + struct blkcg *blkcg =3D container_of(work, struct blkcg, async_bio_work); struct bio_list bios =3D BIO_EMPTY_LIST; struct bio *bio; struct blk_plug plug; bool need_plug =3D false; =20 - /* as long as there are pending bios, @blkg can't go away */ - spin_lock(&blkg->async_bio_lock); - bio_list_merge_init(&bios, &blkg->async_bios); - spin_unlock(&blkg->async_bio_lock); + /* as long as there are pending bios, @blkcg can't go away */ + spin_lock(&blkcg->async_bio_lock); + bio_list_merge_init(&bios, &blkcg->async_bios); + spin_unlock(&blkcg->async_bio_lock); =20 /* start plug only when bio_list contains at least 2 bios */ if (bios.head && bios.head->bi_next) { @@ -239,15 +234,15 @@ static void blkg_async_bio_workfn(struct work_struct = *work) */ void blkcg_punt_bio_submit(struct bio *bio) { - struct blkcg_gq *blkg =3D bio_blkg(bio); + struct blkcg *blkcg =3D bio_blkcg(bio); =20 - if (blkg && blkg->parent) { - spin_lock(&blkg->async_bio_lock); - bio_list_add(&blkg->async_bios, bio); - spin_unlock(&blkg->async_bio_lock); - queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work); + if (blkcg && cgroup_parent(blkcg->css.cgroup)) { + spin_lock(&blkcg->async_bio_lock); + bio_list_add(&blkcg->async_bios, bio); + spin_unlock(&blkcg->async_bio_lock); + queue_work(blkcg_punt_bio_wq, &blkcg->async_bio_work); } else { - /* Never bounce if there is no non-root blkg to queue on. */ + /* Never bounce if there is no non-root blkcg to queue on. */ submit_bio(bio); } } @@ -325,11 +320,6 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg= , struct gendisk *disk, INIT_LIST_HEAD(&blkg->q_node); blkg->blkcg =3D blkcg; blkg->iostat.blkg =3D blkg; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - spin_lock_init(&blkg->async_bio_lock); - bio_list_init(&blkg->async_bios); - INIT_WORK(&blkg->async_bio_work, blkg_async_bio_workfn); -#endif =20 u64_stats_init(&blkg->iostat.sync); for_each_possible_cpu(cpu) { @@ -1360,6 +1350,9 @@ static void blkcg_css_free(struct cgroup_subsys_state= *css) =20 mutex_unlock(&blkcg_pol_mutex); =20 +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + WARN_ON(!bio_list_empty(&blkcg->async_bios)); +#endif free_percpu(blkcg->lhead); kfree(blkcg); } @@ -1409,6 +1402,11 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_c= ss) refcount_set(&blkcg->online_pin, 1); INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_NOWAIT); INIT_HLIST_HEAD(&blkcg->blkg_list); +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + spin_lock_init(&blkcg->async_bio_lock); + bio_list_init(&blkcg->async_bios); + INIT_WORK(&blkcg->async_bio_work, blkcg_async_bio_workfn); +#endif #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&blkcg->cgwb_list); #endif diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index b6fb85db4d3d..b457c2b4ac7c 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -73,14 +73,7 @@ struct blkcg_gq { struct blkg_iostat_set iostat; =20 struct blkg_policy_data *pd[BLKCG_MAX_POLS]; -#ifdef CONFIG_BLK_CGROUP_PUNT_BIO - spinlock_t async_bio_lock; - struct bio_list async_bios; -#endif - union { - struct work_struct async_bio_work; - struct work_struct free_work; - }; + struct work_struct free_work; =20 atomic_t use_delay; atomic64_t delay_nsec; @@ -111,6 +104,11 @@ struct blkcg { */ struct llist_head __percpu *lhead; =20 +#ifdef CONFIG_BLK_CGROUP_PUNT_BIO + spinlock_t async_bio_lock; + struct bio_list async_bios; + struct work_struct async_bio_work; +#endif #ifdef CONFIG_BLK_CGROUP_FC_APPID char fc_app_id[FC_APPID_LEN]; #endif --=20 2.51.0