From nobody Mon Sep 28 23:51:20 2026 Received: from mta0.migadu.com (out-102.mta0.migadu.com [91.218.175.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D3134749C0 for ; Fri, 14 Aug 2026 16:57:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.102 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786726641; cv=none; b=jfPnlMR0OKX4Hgv7KFjXfNfMR4kXDXeBxKZFUvDUwv1gPMeC5ZMEMC6bzeR/AQHnWCCfp/5aKPtgnxzojrfIg75RhtQ3L3yYt2PpDTTmOZFkstkXP/oAMN5tB0YebAhzHnCbWl9iy9v7/SVSw69V2y2gG/jx+HowdONYXhqCVoI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786726641; c=relaxed/simple; bh=Q5/zdQWz0Amn1o+/OUx4rATy9xjEBYqtx8W87l2OgSc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ETqZSExUO4A8iWODcQZP9jYe28c36BxIY9FB/AsnwBMnbCz6rOBGNg80zJT6gFAxB1sSDGmDT+aA8RkOueRCCDJxglP7HPFHhIJbKGy9A2O7hv9M6US9XfJAHHVDvJG5s3Y6ZdnVt59tlyuhsMpst7yw2TvVJO9645sFCP1UMwc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=IUQDuDEj; arc=none smtp.client-ip=91.218.175.102 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="IUQDuDEj" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=Q5/zdQWz0Amn1o+/OUx4rATy9xjEBYqtx8W87l2OgSc=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1786726637; v=1; x=1787331437; b=IUQDuDEjt3YDpnQfnPH1DMRHAKcWm9gVxWDavNl8KQW7Pe9vU/HMX75Pm3hd19A90JjFnKGk fL23eV2TTw8JBGoQgzdCMCIlhjX/exHb/xYEhkELr6zD2VtXii+/aHLDNSKMiPIBEceSSzjyV9l BgkJPQpcNwou+DEWWBWfnee0= X-Envelope-To: linux-kernel@vger.kernel.org Received: from localhost (2a03:2880:10ff:51::) by mta10.migadu.com with ESMTPS id f3759c31d8a1f550; Fri, 14 Aug 2026 16:57:17 +0000 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: tj@kernel.org, axboe@kernel.dk, cgroups@vger.kernel.org, josef@toxicpanda.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: shakeel.butt@linux.dev, hannes@cmpxchg.org, riel@surriel.com, kernel-team@meta.com, Usama Arif Subject: [PATCH v2 1/3] blk-iolatency: clear delay state when freeing policy data Date: Fri, 14 Aug 2026 09:56:37 -0700 Message-ID: <20260814165712.510132-2-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260814165712.510132-1-usama.arif@linux.dev> References: <20260814165712.510132-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" io.latency can throttle a group which has no latency target of its own. When a sibling misses its target, check_scale_change() scales down its peers, and a peer that reaches queue depth one gets blkcg_use_delay() called on it on every further scale-down, even with min_lat_nsec =3D=3D 0. iolatency_pd_offline() resets the target through iolatency_set_min_lat_nsec(), which clears the delay only on a nonzero to zero transition, so it never clears such a peer. Freeing the policy data then leaves blkg->use_delay set and blkcg->congestion_count elevated with nothing left that can drop it. blk_cgroup_congested() then returns true for every task in that cgroup and its descendants for as long as the cgroup lives: page_cache_sync_ra() cuts readahead to a single page, page_cache_async_ra() skips it altogether, and __folio_throttle_swaprate() takes swap_avail_lock and schedules a throttle on anonymous folio allocation. Clear the delay in iolatency_pd_free(). By then bio-held blkg references have drained, or the queue is frozen for policy deactivation, so check_scale_change() cannot re-arm it. The free callback can also see policy data which was never attached to a blkg, hence the pd->blkg check. Fixes: d70675121546 ("block: introduce blk-iolatency io controller") Signed-off-by: Usama Arif Acked-by: Tejun Heo --- block/blk-iolatency.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 9eb69010c34e6..2caa79a008ad1 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -1043,6 +1043,15 @@ static void iolat_release(struct rcu_head *rcu) =20 static void iolatency_pd_free(struct blkg_policy_data *pd) { + struct blkcg_gq *blkg =3D pd_to_blkg(pd); + + /* + * Groups throttled as collateral have min_lat_nsec =3D=3D 0, so + * iolatency_pd_offline() leaves their delay set. Drop it here, where + * no in-flight bio can re-arm it via check_scale_change(). + */ + if (blkg) + blkcg_clear_delay(blkg); call_rcu(&pd->rcu_head, iolat_release); } =20 --=20 2.53.0-Meta From nobody Mon Sep 28 23:51:20 2026 Received: from mta1.migadu.com (out-110.mta1.migadu.com [95.215.58.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D77348D873 for ; Fri, 14 Aug 2026 16:57:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.110 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786726646; cv=none; b=n/uahXOVIJ38l7HeslD3YnuM+I51kOxyio10hnIJo/k2p3135mXH28suinLFh7M46KUs5ATAvj+BvtJxonKBKdmr3jEghxbyTk65uMDPLK22PaTvc4MINAV5j+Z1n5jBBx1BnlwI+ahxWNE+Mcj2L7LKXwZs3ZWdGC5ymmpL7UE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786726646; c=relaxed/simple; bh=MgdIR1gKCaB7HmkjICzTGmBtenY5gqEIgSXJG/amQ6w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eRLr+i0Q6fTSrCIZjOg/vz2hjP441aQSXYrRaB5Vil/UdPHmreprNsI4hERafjmm6DjgfetrNQQffTG+/HEH66ClRofv0EQZIwEASqZtSs6+N4dCh97rZItc/4Xpx7c8KQz0dk/R/ZihNjTTH2cfjGM9a37OqS7t39avE+fkDL4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=DuE+Ts+m; arc=none smtp.client-ip=95.215.58.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="DuE+Ts+m" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=MgdIR1gKCaB7HmkjICzTGmBtenY5gqEIgSXJG/amQ6w=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1786726642; v=1; x=1787331442; b=DuE+Ts+mGA+aggzm3O/Vs4/nRh7eJTlNszT7Bkhn1b75MZUTOdPz72Ue3jBxpTbKSs+qEinx PyPr/P85vepOf1NxoaCIW8GwsiJVt4Lr3DxN2oKBELRwy4QupF8MaHp55v+IjHSpzZXeoOKM4Kd 2YQHZzxWBUizSxWywO6LNbFo= X-Envelope-To: linux-kernel@vger.kernel.org Received: from localhost (2a03:2880:10ff:24::) by mta11.migadu.com with ESMTPS id a2c8de5b93f351fd; Fri, 14 Aug 2026 16:57:22 +0000 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: tj@kernel.org, axboe@kernel.dk, cgroups@vger.kernel.org, josef@toxicpanda.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: shakeel.butt@linux.dev, hannes@cmpxchg.org, riel@surriel.com, kernel-team@meta.com, Usama Arif Subject: [PATCH v2 2/3] blk-iocost: clear delay state when freeing policy data Date: Fri, 14 Aug 2026 09:56:38 -0700 Message-ID: <20260814165712.510132-3-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260814165712.510132-1-usama.arif@linux.dev> References: <20260814165712.510132-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" iocg_kick_delay() turns sufficiently large debt into an explicit block-cgroup delay with blkcg_set_delay(), setting blkg->use_delay to -1 and incrementing blkcg->congestion_count. Clearing it again depends on iocg_kick_delay() running from the period timer, the waitq timer or the issue path. ioc_pd_free() removes the iocg from active_iocgs and cancels its waitq timer, and no further bios can arrive, so once it has run nothing is left which can reduce the debt and clear the delay. The blkcg stays marked congested for the rest of its life. blk_cgroup_congested() then returns true for every task in that cgroup and its descendants: page_cache_sync_ra() cuts readahead to a single page, page_cache_async_ra() skips it altogether, and __folio_throttle_swaprate() takes swap_avail_lock and schedules a throttle on anonymous folio allocation. Clear it explicitly, after the list removal and the synchronous hrtimer_cancel() so that neither timer processing nor an I/O path can re-arm it. The free callback can also see policy data which was never attached to a blkg, hence the pd->blkg check. Fixes: 7caa47151ab2 ("blkcg: implement blk-iocost") Signed-off-by: Usama Arif Acked-by: Tejun Heo --- block/blk-iocost.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/block/blk-iocost.c b/block/blk-iocost.c index b60625613e095..64b92aa3e5d48 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -3063,6 +3063,7 @@ static void iocg_release(struct rcu_head *rcu) static void ioc_pd_free(struct blkg_policy_data *pd) { struct ioc_gq *iocg =3D pd_to_iocg(pd); + struct blkcg_gq *blkg =3D pd_to_blkg(pd); struct ioc *ioc =3D iocg->ioc; unsigned long flags; =20 @@ -3085,6 +3086,12 @@ static void ioc_pd_free(struct blkg_policy_data *pd) hrtimer_cancel(&iocg->waitq_timer); } =20 + /* off ->active_iocgs and timer gone, so nothing can re-arm the delay */ + iocg->delay =3D 0; + iocg->indelay_since =3D 0; + if (blkg) + blkcg_clear_delay(blkg); + call_rcu(&pd->rcu_head, iocg_release); } =20 --=20 2.53.0-Meta From nobody Mon Sep 28 23:51:20 2026 Received: from mta0.migadu.com (out-111.mta0.migadu.com [91.218.175.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C41CA4908C2 for ; Fri, 14 Aug 2026 16:57:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.111 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786726652; cv=none; b=Kkz7SvslnczmWZA4dcKoF8pljS8kVF1coylvYcv1TcsVcsmvJAJkaPCZ4AaVzLOKHAwikIBUwUtb2hUwpCoIQNz5A/4B5Vv8varShP/q1yix89n4OA/cI6UlLzFuiGmyqXOTtsJhUPusbjnN7cBzsPnrD8DhuXl3yhsv8fn0p7Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786726652; c=relaxed/simple; bh=X1KnkGIfM+vcK1hFOo1CDZ/cx/dJzFBqYtJogCIxOPE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qaPjDaB7e/l54319+D8Yw152kLCqmuYfcETrQw5oWQ0qmEZuJOA6xE4TfYkfO7yivDaBpBC6sWd0PWWZtoBgcGvIpXvzbRylhuCq5/cZty0WffDeb60zbd2BMlBfu96cMswYnFPhs/x3DTFl5D4jPww8+4zgNTHmkMIFebAiar8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Zi9PycuR; arc=none smtp.client-ip=91.218.175.111 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Zi9PycuR" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=X1KnkGIfM+vcK1hFOo1CDZ/cx/dJzFBqYtJogCIxOPE=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1786726648; v=1; x=1787331448; b=Zi9PycuRO/1ddpZJQJTkDE12CSExr7KiCxGz4hSZLizB7lVMkrwh5IwCX/5crBsEdUEtQ28d Lo4UdVbqYVExRrXYIdAnnQ+F9EKPMX4pW2D+V1F1DhhykhO4DNeqK+KdrUfmlIy6+qhoi8QufXl SPyej7fIhfClQj56GGdb6j8w= X-Envelope-To: linux-kernel@vger.kernel.org Received: from localhost (2a03:2880:10ff:1a::) by mta12.migadu.com with ESMTPS id 31092de03c9f15a1; Fri, 14 Aug 2026 16:57:28 +0000 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: tj@kernel.org, axboe@kernel.dk, cgroups@vger.kernel.org, josef@toxicpanda.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: shakeel.butt@linux.dev, hannes@cmpxchg.org, riel@surriel.com, kernel-team@meta.com, Usama Arif Subject: [PATCH v2 3/3] block: skip blkcg walk in blk_cgroup_congested() when nothing throttled Date: Fri, 14 Aug 2026 09:56:39 -0700 Message-ID: <20260814165712.510132-4-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260814165712.510132-1-usama.arif@linux.dev> References: <20260814165712.510132-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" blk_cgroup_congested() walks the current task's blkcg ancestor chain on every readahead decision and, once swap is in use, on every anonymous and shmem folio allocation. The answer is almost always "no", but finding that out costs two loads per level on two cold cache lines, plus an out-of-line kthread_blkcg() and an RCU read-side pair. On a fleet profile of hosts running containers with 5-10 level hierarchies it costs about as much as all of mutex_lock(), 99.4% of it under __folio_throttle_swaprate(). Gate the walk on a global count of blkcgs with a non-zero congestion_count. The counter only moves on the 0 <-> 1 transitions of each blkcg's congestion_count, so the extra atomic stays in the throttle arm/disarm paths and never appears in steady state. When something is throttled the counter is non-zero and the walk runs as before. Signed-off-by: Usama Arif Acked-by: Tejun Heo --- block/blk-cgroup.c | 15 ++++++++++++++- block/blk-cgroup.h | 25 +++++++++++++++++++++---- include/linux/blk-cgroup.h | 23 ++++++++++++++++++++++- 3 files changed, 57 insertions(+), 6 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 354637f3b158c..2b5c29434e426 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -53,6 +53,9 @@ EXPORT_SYMBOL_GPL(blkcg_root); struct cgroup_subsys_state * const blkcg_root_css =3D &blkcg_root.css; EXPORT_SYMBOL_GPL(blkcg_root_css); =20 +/* number of blkcgs with a non-zero congestion_count */ +atomic_t blkcg_nr_congested __read_mostly =3D ATOMIC_INIT(0); + static struct blkcg_policy *blkcg_policy[BLKCG_MAX_POLS]; =20 static LIST_HEAD(all_blkcgs); /* protected by blkcg_pol_mutex */ @@ -1350,6 +1353,16 @@ static void blkcg_css_free(struct cgroup_subsys_stat= e *css) struct blkcg *blkcg =3D css_to_blkcg(css); int i; =20 + /* + * Every blkg holds a reference on this css and drops any delay it + * still has from pd_free_fn(), so this is expected to be zero. Should + * a policy ever leave one behind, drop it here rather than let it pin + * blkcg_nr_congested and disable the fast path for the rest of the + * boot. Nothing can race with us at this point. + */ + if (WARN_ON_ONCE(atomic_xchg(&blkcg->congestion_count, 0) > 0)) + atomic_dec(&blkcg_nr_congested); + mutex_lock(&blkcg_pol_mutex); =20 list_del(&blkcg->all_blkcgs_node); @@ -2228,7 +2241,7 @@ void blk_cgroup_bio_start(struct bio *bio) put_cpu(); } =20 -bool blk_cgroup_congested(void) +bool __blk_cgroup_congested(void) { struct blkcg *blkcg; bool ret =3D false; diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 615390f751aa2..e67c698391294 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -375,12 +375,29 @@ static inline void blkg_put(struct blkcg_gq *blkg) if (((d_blkg) =3D blkg_lookup(css_to_blkcg(pos_css), \ (p_blkg)->q))) =20 +/* + * blkcg_nr_congested gates the hierarchy walk in blk_cgroup_congested(). + * These two helpers keep it in step with each blkcg's congestion_count in + * normal operation; blkcg_css_free() drops a residual count as a backstop. + */ +static inline void blkcg_inc_congestion_count(struct blkcg *blkcg) +{ + if (atomic_inc_return(&blkcg->congestion_count) =3D=3D 1) + atomic_inc(&blkcg_nr_congested); +} + +static inline void blkcg_dec_congestion_count(struct blkcg *blkcg) +{ + if (atomic_dec_and_test(&blkcg->congestion_count)) + atomic_dec(&blkcg_nr_congested); +} + static inline void blkcg_use_delay(struct blkcg_gq *blkg) { if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0)) return; if (atomic_add_return(1, &blkg->use_delay) =3D=3D 1) - atomic_inc(&blkg->blkcg->congestion_count); + blkcg_inc_congestion_count(blkg->blkcg); } =20 static inline int blkcg_unuse_delay(struct blkcg_gq *blkg) @@ -405,7 +422,7 @@ static inline int blkcg_unuse_delay(struct blkcg_gq *bl= kg) if (old =3D=3D 0) return 0; if (old =3D=3D 1) - atomic_dec(&blkg->blkcg->congestion_count); + blkcg_dec_congestion_count(blkg->blkcg); return 1; } =20 @@ -424,7 +441,7 @@ static inline void blkcg_set_delay(struct blkcg_gq *blk= g, u64 delay) =20 /* We only want 1 person setting the congestion count for this blkg. */ if (!old && atomic_try_cmpxchg(&blkg->use_delay, &old, -1)) - atomic_inc(&blkg->blkcg->congestion_count); + blkcg_inc_congestion_count(blkg->blkcg); =20 atomic64_set(&blkg->delay_nsec, delay); } @@ -441,7 +458,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq *b= lkg) =20 /* We only want 1 person clearing the congestion count for this blkg. */ if (old && atomic_try_cmpxchg(&blkg->use_delay, &old, 0)) - atomic_dec(&blkg->blkcg->congestion_count); + blkcg_dec_congestion_count(blkg->blkcg); } =20 /** diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index dd5841a42c331..58abde49f8c59 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -14,6 +14,8 @@ * Nauman Rafique */ =20 +#include +#include #include =20 struct bio; @@ -24,10 +26,29 @@ struct gendisk; =20 #ifdef CONFIG_BLK_CGROUP extern struct cgroup_subsys_state * const blkcg_root_css; +extern atomic_t blkcg_nr_congested; =20 void blkcg_schedule_throttle(struct gendisk *disk, bool use_memdelay); void blkcg_maybe_throttle_current(void); -bool blk_cgroup_congested(void); +bool __blk_cgroup_congested(void); + +/** + * blk_cgroup_congested - is the current task in a throttled blkcg? + * + * Called from mm hot paths where the answer is almost always false, so ke= ep + * that case to a load and a branch and only walk the hierarchy out of line + * when something in the system really is throttled. + * + * Return: %true if the current task's blkcg or any of its ancestors is + * throttled, %false otherwise. + */ +static inline bool blk_cgroup_congested(void) +{ + if (likely(!atomic_read(&blkcg_nr_congested))) + return false; + return __blk_cgroup_congested(); +} + void blkcg_pin_online(struct cgroup_subsys_state *blkcg_css); void blkcg_unpin_online(struct cgroup_subsys_state *blkcg_css); struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css); --=20 2.53.0-Meta