From nobody Mon Sep 28 20:05:23 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D572D47278D for ; Mon, 17 Aug 2026 21:42:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787002945; cv=none; b=SAVCFo1p2mLg3+5EYRcGz+cjUw6pHVYikT0N13oh4HKlsCBDasLkzbwD0kEXT7OcfQRI9A3T9qTAz5tj/tsgrqDuTiG4j7tBHnTAzmwSXs7dv1Ox7nCswBU6WleQkhbhTV9/6hOJBcoIfFbxPM1k6idD4sDWiWzySUnxxVB+FHc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787002945; c=relaxed/simple; bh=0z8PWGduURq5jdmaL9vW7E71Pmvd9PeAhi1Jwg4AAbc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EsN53pzGeBg7hE1RtA+A+yRyJDxRP6rI1wSN/bjHdEXDBR6MM+Ja7db65Gs4y/bbceum1DO9ZgkYgmyyoSi6Gj/tyYfUml6Inu53GLGtotrJIxDy0w1w8h6prhDvRUg9n3jxnbrsy+e6v0bIoRtXQZ0GUvHeTwU0faD7UN5qoD4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aDvHJwq6; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aDvHJwq6" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2cf6d65d8a7so50842245ad.0 for ; Mon, 17 Aug 2026 14:42:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787002942; x=1787607742; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0OAypTyGdu6BgXD9Bv7vBEYzg6snlB0upRlQ0JuzemU=; b=aDvHJwq6i2tIzrgcGv3KFgI63iLRqPNOGoImgz9BSed+93i79zMgB1VS6ai8Tu4bFM wBqWZxX4eB+pxJLBomOQe84DLzewCWWE/UJcTPpW7YybyEgQt7XSrguAFm41fdEbeVpc 1zVNVqPyzZsdtsh6x8z7NqPMGsGpej6q+AF2j5pEz82uKIVVhenN5npntDflLLuXTv9O b559TnxIjIlYTyOv8b3pYHCdhIrb5qVkMmAa2139dNsS52zf8QmM1SXhZScHKvuTHxLO nVM//LV1htAEZ+vpsHWzhkm4KCTVLFxSk042Dv+ZGA3rN/NefmAniF1oy5WvYOyb0/hJ J05Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787002942; x=1787607742; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=0OAypTyGdu6BgXD9Bv7vBEYzg6snlB0upRlQ0JuzemU=; b=sU+NskaO2nKQQYsbGkDHnzMg0Nk9TUNQGAq3NjrVizuiNdRZCwC3p8GEVbDcISzT8i aJSi5x77Kb8PIbduVe0mPbUiBg81XfOMXNbB5htALAH4B0jU9c8+FzhwkMtLSO7z9c0J sljQMM7MGQAL8xOGxn/re2H8XM71l+6FAXcTDJjCs3ye8aqXolsCMXIIyWJ9URZilb7O nvntOBhdXk9ljuTfE9rex+YafCq656xBYuKspyWERt/QtZAiEVFA3ePyel87/PwwOHE6 Dmd9P9dnJ2kYIoO75n6WDusv5V7PJ0rATntnXOAOPOtOqp3bb336tUKsQ9J5sFYL+J+f jDTA== X-Forwarded-Encrypted: i=1; AHgh+RoqkkbJtVb2Cct1laXdFzT6wSqzj1m5FGbW9Ad8a5XaQ0z+MpDy9GVg8oRqiAFBuCufxxNRKtMsvRU9nHs=@vger.kernel.org X-Gm-Message-State: AOJu0YxuamtC4aaGiGysM+4eSPIYmWMoaH6ouARhpCltYnmPPrtGCmeG xLm5NvVUKGAvMTjtP5D4t8SS5Sfo5wzOlfdT87rILHWgwR79wL3U0OKd X-Gm-Gg: AR+sD10mlCpgeba1Q6xomXGNudfym9aVxGMF/REk17xiuh8BdKBRwVv57YIH/rudztG yaTzDU4mVAkksL5N6ZdBefRy/l87yqz9OuwgOjOCkjfIf5Sf4KwIDcAlAIPV3sHs09dI15o93W0 rVDLb/E8hwQu+Gwyb8udexlonl0+3EVhHa2YsxfUvaaQwjJyG9837GlFYeaRJamYp4vsGUBbUen HkevrMQ0hMMxmHtG6Yr4bLevDCzBnaAAaqB5KFm9UY1FpfTEGWTrFKxT2SRIu04fxu/xLZGLsXb QeqNA9gXQDS+HyPTTRN2uIJp/EgY8aRTs46CDtFQNX0QigKmvOrqIiEKLotiD/1IRoV7+SKRVVe I7T9aKwcOx+oLQ7f2J6T5T/ZXDpp/+kjy6M+DSTckSKrk9+BiWvzQSDQOBq7TpRQw0aJtMJX/Yu lZ8ez0f4kwXDbBM+YjoHrtPY7SBXtKKKOvDXbkv/3bzIOB99F6W1+jlLc= X-Received: by 2002:a17:902:f605:b0:2ca:4f33:e86f with SMTP id d9443c01a7336-2d3b0c5e569mr339655925ad.12.1787002942282; Mon, 17 Aug 2026 14:42:22 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:42::]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3267c897e37sm10556279eec.28.2026.08.17.14.42.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 14:42:21 -0700 (PDT) From: Ziyang Men To: kernel-team@meta.com, Jens Axboe , Tejun Heo , Josef Bacik , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Shuah Khan , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Roman Gushchin , Shakeel Butt , JP Kobryn , Mykola Lysenko , Ziyang Men , linux-block@vger.kernel.org, bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/2] block: add BPF kfuncs to read blkcg io.stat Date: Mon, 17 Aug 2026 14:42:04 -0700 Message-ID: <20260817214205.723267-2-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260817214205.723267-1-ziyang.meme@gmail.com> References: <20260817214205.723267-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Collecting cgroup statistics is expensive: the existing method is to open and parse a cgroup file for every cgroup of interest. memcg already has an efficient alternative through BPF; this series extends that idea to block. This series exposes the block I/O controller's per-device io.stat to BPF. The flush is sleepable and takes a cgroup. It pins the I/O css before leaving RCU, then flushes it. The iterator takes the RCU-protected css and remains block-specific because each block device has its own blkg. The behavior mirrows the blkcg_print_stat(). The blkg device iterator take a RCU css. No kfuncs are added to read the blkcg counters since user can read it using the BPF_CORE_READ. Suggested-by: Shakeel Butt Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- MAINTAINERS | 1 + block/Makefile | 3 + block/blk-cgroup.c | 2 +- block/blk-cgroup.h | 1 + block/bpf_blkcg.c | 154 +++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 160 insertions(+), 1 deletion(-) create mode 100644 block/bpf_blkcg.c diff --git a/MAINTAINERS b/MAINTAINERS index 2f9472c1a090..87c56e955577 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -6617,6 +6617,7 @@ F: block/blk-cgroup.c F: block/blk-iocost.c F: block/blk-iolatency.c F: block/blk-throttle.c +F: block/bpf_blkcg.c F: include/linux/blk-cgroup.h =20 CONTROL GROUP - CPUSET diff --git a/block/Makefile b/block/Makefile index e7bd320e3d69..572e49988c8e 100644 --- a/block/Makefile +++ b/block/Makefile @@ -17,6 +17,9 @@ obj-$(CONFIG_BLK_ERROR_INJECTION) +=3D error-injection.o obj-$(CONFIG_BLK_DEV_BSG_COMMON) +=3D bsg.o obj-$(CONFIG_BLK_DEV_BSGLIB) +=3D bsg-lib.o obj-$(CONFIG_BLK_CGROUP) +=3D blk-cgroup.o +ifdef CONFIG_BPF_SYSCALL +obj-$(CONFIG_BLK_CGROUP) +=3D bpf_blkcg.o +endif obj-$(CONFIG_BLK_CGROUP_RWSTAT) +=3D blk-cgroup-rwstat.o obj-$(CONFIG_BLK_CGROUP_FC_APPID) +=3D blk-cgroup-fc-appid.o obj-$(CONFIG_BLK_DEV_THROTTLING) +=3D blk-throttle.o diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index d9676126c5b5..8d538ad4e861 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1086,7 +1086,7 @@ static void blkcg_rstat_flush(struct cgroup_subsys_st= ate *css, int cpu) * flushing the root cgroup's stats by explicitly filling in the iostat * with disk level statistics. */ -static void blkcg_fill_root_iostats(void) +void blkcg_fill_root_iostats(void) { struct class_dev_iter iter; struct device *dev; diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 615390f751aa..8c9c2a1adfaa 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -205,6 +205,7 @@ void blkcg_deactivate_policy(struct gendisk *disk, const struct blkcg_policy *pol); =20 const char *blkg_dev_name(struct blkcg_gq *blkg); +void blkcg_fill_root_iostats(void); void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg, u64 (*prfill)(struct seq_file *, struct blkg_policy_data *, int), diff --git a/block/bpf_blkcg.c b/block/bpf_blkcg.c new file mode 100644 index 000000000000..25c809f5091c --- /dev/null +++ b/block/bpf_blkcg.c @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Block I/O Controller-related BPF kfuncs and auxiliary code + */ + +#include "blk-cgroup.h" + +#include +#include +#include + +__bpf_kfunc_start_defs(); + +/** + * bpf_blkcg_flush_stats - Flush a block cgroup's io statistics + * @cgrp: cgroup to flush + * + * Propagate I/O statistics up the cgroup tree. Root statistics come from + * block devices and include all cgroups' I/O. + */ +__bpf_kfunc void bpf_blkcg_flush_stats(struct cgroup *cgrp) +{ + struct cgroup_subsys_state *css; + + /* Pin the css for the sleepable flush. */ + rcu_read_lock(); + css =3D cgroup_css(cgrp, &io_cgrp_subsys); + if (css && !css_tryget(css)) + css =3D NULL; + rcu_read_unlock(); + + if (!css) + return; + + if (!css->parent) + blkcg_fill_root_iostats(); + else + css_rstat_flush(css); + + css_put(css); +} + +struct bpf_iter_blkg { + __u64 __opaque[2]; +} __aligned(8); + +struct bpf_iter_blkg_kern { + struct blkcg *blkcg; + struct blkcg_gq *pos; +} __aligned(8); + +/** + * bpf_iter_blkg_new - Start iterating a block cgroup's per-device blkgs + * @it: iterator to initialize + * @css: the io controller's css + * + * Each blkg holds one device's io.stat counters. Offline blkgs are skippe= d. + * A blkg without a disk can be returned. Must run under RCU. + * + * Return: 0 on success, -EINVAL if @css is not the io controller's. + */ +__bpf_kfunc int bpf_iter_blkg_new(struct bpf_iter_blkg *it, + struct cgroup_subsys_state *css) +{ + struct bpf_iter_blkg_kern *kit =3D (void *)it; + + BUILD_BUG_ON(sizeof(struct bpf_iter_blkg_kern) > sizeof(struct bpf_iter_b= lkg)); + BUILD_BUG_ON(__alignof__(struct bpf_iter_blkg_kern) !=3D + __alignof__(struct bpf_iter_blkg)); + + kit->pos =3D NULL; + + if (css->ss !=3D &io_cgrp_subsys) { + kit->blkcg =3D NULL; + return -EINVAL; + } + + kit->blkcg =3D css_to_blkcg(css); + return 0; +} + +/** + * bpf_iter_blkg_next - Return the next online blkg of the iterated block = cgroup + * @it: iterator + * + * Return: the next online blkg, or NULL when the walk is done. + */ +__bpf_kfunc struct blkcg_gq *bpf_iter_blkg_next(struct bpf_iter_blkg *it) +{ + struct bpf_iter_blkg_kern *kit =3D (void *)it; + struct blkcg_gq *blkg =3D kit->pos; + struct hlist_node *node; + + if (!kit->blkcg) + return NULL; + + if (!blkg) + node =3D rcu_dereference(hlist_first_rcu(&kit->blkcg->blkg_list)); + else + node =3D rcu_dereference(hlist_next_rcu(&blkg->blkcg_node)); + + /* Skip offline blkgs, matching io.stat. */ + while (node) { + blkg =3D hlist_entry(node, struct blkcg_gq, blkcg_node); + /* A race only changes whether this blkg is returned. */ + if (data_race(blkg->online)) { + kit->pos =3D blkg; + return blkg; + } + node =3D rcu_dereference(hlist_next_rcu(&blkg->blkcg_node)); + } + + /* The iterator must keep returning NULL after completion. */ + kit->pos =3D NULL; + kit->blkcg =3D NULL; + return NULL; +} + +/** + * bpf_iter_blkg_destroy - Tear down a blkg iterator + * @it: iterator + */ +__bpf_kfunc void bpf_iter_blkg_destroy(struct bpf_iter_blkg *it) +{ +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(bpf_blkcg_kfuncs) +BTF_ID_FLAGS(func, bpf_blkcg_flush_stats, KF_SLEEPABLE) + +BTF_ID_FLAGS(func, bpf_iter_blkg_new, + KF_ITER_NEW | KF_RCU | KF_RCU_PROTECTED) +BTF_ID_FLAGS(func, bpf_iter_blkg_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_blkg_destroy, KF_ITER_DESTROY) +BTF_KFUNCS_END(bpf_blkcg_kfuncs) + +static const struct btf_kfunc_id_set bpf_blkcg_kfunc_set =3D { + .owner =3D THIS_MODULE, + .set =3D &bpf_blkcg_kfuncs, +}; + +static int __init bpf_blkcg_init(void) +{ + int err; + + err =3D register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, + &bpf_blkcg_kfunc_set); + if (err) + pr_warn("error while registering bpf blkcg kfuncs: %d\n", err); + + return err; +} +late_initcall(bpf_blkcg_init); --=20 2.53.0-Meta From nobody Mon Sep 28 20:05:23 2026 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D515E47255B for ; Mon, 17 Aug 2026 21:42:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787002946; cv=none; b=s5MhvCA2mVQ/qyudE/QOaNKScGEqhdJH/IjFEjXvVflT3vkH1AWcvFy1MLcp6qJdlGk0VrI2Zs+5lv37ViRdTBn3wkoTBrXkOqJxFgv7C9h9OuIFjdMG39LPOFGxEmbvrwZVR1B3xwSZiLHAAsqrQkuFo4F4Vjz9jqrMA3/1MVU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787002946; c=relaxed/simple; bh=+BrwSqK/iZEWjOi5n/aAk9jdHYV1SOICLEzhhQHfrdM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mJcL5Q/hfgpNual6G57wZSoU7raJialYK5sBO4fgIXTx/LZ6kd99YEtgCdpmnac0vuIAW2tmpHedbem5Uyyn8HqT7BtIQn9kQT7xUndcF8sRxVdFDU8VnZqzzqI5dzwsFQg/hAk/VpTYheUXLKSADbrMyE5/4neHt14goPEBlgU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WJpTGgkh; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WJpTGgkh" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-cbeea240819so2186629a12.1 for ; Mon, 17 Aug 2026 14:42:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787002944; x=1787607744; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Z8oHY1efdTZ0mrkkqnpEhOi9h4e/ZaN6Y4vux4nxnrw=; b=WJpTGgkhDAn8mSHKKOrK8V5g+WWW6/bwkGhgTjq5Gb8jBGVDTncw2rtji2w0Qv3Fyf uQ3f57E8lIJf30l39cXbSjdbbfXq9YnRjoJXu9CfeqOVB2wzoWlpzDyNF8hk+mZ/AX47 2G7dYkdllBvkTSSfeeWRpZByTdKcdPkRvWmyYNp9xdpyvXcv/hIkj7PmmOIZW4exYjLS o7P5w4I4pZ7LlFkprFtawpu9qeB1WSxbChM1mttvtSx3XFtZmo5STVDbSLTMUj1kNmmx 4aXIUhLwxcFyveUTyGe+z37LEckyyxG52H5OXAhHDSLEE1QNZMRvKESk5jPDnCTU6dmh hhEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787002944; x=1787607744; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Z8oHY1efdTZ0mrkkqnpEhOi9h4e/ZaN6Y4vux4nxnrw=; b=h/e/YBJhJj4STdY7AgXP0NfEt9fit3obtt5wMlg0LFfTrFtxYf/n3ZHCdU9pig8vEW FxvzA3X4OwBxEbQPCMbARBfg4aXQHroXwiu1ePi/opMP+v+7ZYfSJ7N1H1SgQUXQri9L z3hK+fIuxP/dYKipkQh8eive7QlLdvZL+Sf1L0VChlP5iNIgcU3X9qgNxZuPtU6EUSft WGeo8qa6wPHmnvc4TNeiLBUGEElhfcZjFbitfl4trQRh5mSCPLeHeuFxcdJMlOV3QIYP 0Uj5DMRBQweh9k4TnCH7a8ZaVIdPnMneOuDaZIjtaJEJ9BAL9t8R4h9ICofpFB5De3DR 4eNQ== X-Forwarded-Encrypted: i=1; AHgh+RpEAq9vthaauWRzw+jayfRJ4G0vgEO6i7PQvUTV1w7Q5vhu8Mdmqc53FKgpLOUslnVjUtLaoz9dUC91ZI8=@vger.kernel.org X-Gm-Message-State: AOJu0YzGJQh16/4VEGRHefHZO21gDz1Gm0+pewDUQ25cf9B6UV4ELS/B kvG+ATM2KkLfDOaS1dgPsXTX+79xFDqC2wcbJjXJukX+Nx7WZtshZY8I X-Gm-Gg: AR+sD11RoKfEwLBLKavHPts9UGNxcDRc7AFj7byZnrdG3euKikWwDhE86yilsX+CROn ExuPFyX1NaDdp8YQ3JheE9aUEKcW10v3E6+23SA2m597e/gaYIVpSMR4clkd2Vh8BWGczkGvQiG 8tiL3EMFNj9kNqykbrX38YrJ7d3Xe9LEc511T+7Jx6xK/qkNrkEQjIAYqGt6CZu3u9TRIt5PR7z q/ABwyPfjnpQOcr3tq7Z7kBhxK39OfUfLPO3X4KrLH7mo/FuWC7lC9dhfv32ebH7bENMUxTED0h rr5uW36CcHF0wxNRVoeuNwNvx4flH4OS0b/mCsGRPxIzEqNZNmvencNtQKVaxxaGB5Gd4hiYBPd 7R415yGD5X/rWzHGbva6a09OEkMFA6Giii60JyK4pTtp+ydLuHnxw08Km6EfymRK8ZIAEdzvkbV 1WNTjiJyisEVdw3/2Cfo30cI0lInCvUWI7QJNP2gxpTJ4dFDmEQWYcayc= X-Received: by 2002:a05:6a21:9087:b0:3c3:9aff:7a67 with SMTP id adf61e73a8af0-3ccda66c136mr3576310637.17.1787002943840; Mon, 17 Aug 2026 14:42:23 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:6b::]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-326794c0c94sm18048820eec.9.2026.08.17.14.42.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 14:42:23 -0700 (PDT) From: Ziyang Men To: kernel-team@meta.com, Jens Axboe , Tejun Heo , Josef Bacik , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Shuah Khan , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Roman Gushchin , Shakeel Butt , JP Kobryn , Mykola Lysenko , Ziyang Men , linux-block@vger.kernel.org, bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/2] selftests/bpf: add test for blkcg io.stat BPF kfuncs Date: Mon, 17 Aug 2026 14:42:05 -0700 Message-ID: <20260817214205.723267-3-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260817214205.723267-1-ziyang.meme@gmail.com> References: <20260817214205.723267-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add cgroup_iter_io to test the blkcg io.stat BPF kfuncs. The BPF program flushes the statistics, looks up the I/O css under RCU, walks its blkgs, and reads one device's counters with BPF_CORE_READ(). The test performs O_DIRECT I/O on a private loop device. It checks the device ID and counters against io.stat. Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- tools/testing/selftests/bpf/cgroup_iter_io.h | 17 ++ tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cgroup_iter_io.c | 277 ++++++++++++++++++ .../selftests/bpf/progs/cgroup_iter_io.c | 99 +++++++ 4 files changed, 394 insertions(+) create mode 100644 tools/testing/selftests/bpf/cgroup_iter_io.h create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_io.c diff --git a/tools/testing/selftests/bpf/cgroup_iter_io.h b/tools/testing/s= elftests/bpf/cgroup_iter_io.h new file mode 100644 index 000000000000..f4bbaaccdf71 --- /dev/null +++ b/tools/testing/selftests/bpf/cgroup_iter_io.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#ifndef __CGROUP_ITER_IO_H +#define __CGROUP_ITER_IO_H + +struct io_query { + /* one device's io.stat counters */ + __u64 rbytes; + __u64 wbytes; + __u64 rios; + __u64 wios; + __u64 dbytes; + __u64 dios; + __u64 dev; /* dev_t of the device the counters belong to */ +}; + +#endif /* __CGROUP_ITER_IO_H */ diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/b= pf/config index ea7044f30adc..270e6bf9194d 100644 --- a/tools/testing/selftests/bpf/config +++ b/tools/testing/selftests/bpf/config @@ -1,3 +1,4 @@ +CONFIG_BLK_CGROUP=3Dy CONFIG_BLK_DEV_LOOP=3Dy CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=3Dy CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=3D1 diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c b/tool= s/testing/selftests/bpf/prog_tests/cgroup_iter_io.c new file mode 100644 index 000000000000..5d27e5d28379 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c @@ -0,0 +1,277 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "cgroup_helpers.h" +#include "cgroup_iter_io.h" +#include "cgroup_iter_io.skel.h" + +#define IO_SIZE (4 * 1024 * 1024) + +static int read_stats(struct bpf_link *link) +{ + int fd, ret =3D 0; + ssize_t bytes; + + fd =3D bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_OK_FD(fd, "bpf_iter_create")) + return 1; + + /* Results land in skel->data_query; the read itself returns no data. */ + bytes =3D read(fd, NULL, 0); + if (!ASSERT_EQ(bytes, 0, "read fd")) + ret =3D 1; + + close(fd); + return ret; +} + +/* Set up a loop device for cgroup-charged I/O. */ +static int loop_setup(char *loop_path, size_t sz, int *ctl_fd, int *loop_f= d, + int *back_fd) +{ + char back_path[] =3D "/tmp/cgroup_iter_io.XXXXXX"; + int nr; + + *ctl_fd =3D *loop_fd =3D *back_fd =3D -1; + + *ctl_fd =3D open("/dev/loop-control", O_RDWR | O_CLOEXEC); + if (*ctl_fd < 0) + return -1; + + nr =3D ioctl(*ctl_fd, LOOP_CTL_GET_FREE); + if (nr < 0) + goto err; + snprintf(loop_path, sz, "/dev/loop%d", nr); + + *back_fd =3D mkstemp(back_path); + if (*back_fd < 0) + goto err; + unlink(back_path); + if (ftruncate(*back_fd, (off_t)IO_SIZE * 4)) + goto err; + + *loop_fd =3D open(loop_path, O_RDWR | O_CLOEXEC); + if (*loop_fd < 0) + goto err; + if (ioctl(*loop_fd, LOOP_SET_FD, *back_fd)) + goto err; + + return 0; +err: + if (*loop_fd >=3D 0) + close(*loop_fd); + if (*back_fd >=3D 0) + close(*back_fd); + close(*ctl_fd); + *ctl_fd =3D *loop_fd =3D *back_fd =3D -1; + return -1; +} + +static void loop_teardown(const char *loop_path, int ctl_fd, int loop_fd, + int back_fd) +{ + int nr =3D -1; + + if (loop_fd >=3D 0) { + ioctl(loop_fd, LOOP_CLR_FD, 0); + close(loop_fd); + } + if (back_fd >=3D 0) + close(back_fd); + if (ctl_fd >=3D 0) { + if (sscanf(loop_path, "/dev/loop%d", &nr) =3D=3D 1 && nr >=3D 0) + ioctl(ctl_fd, LOOP_CTL_REMOVE, nr); + close(ctl_fd); + } +} + +/* O_DIRECT keeps I/O charged to the current cgroup. */ +static int do_direct_io(const char *loop_path) +{ + void *buf; + int fd, ret =3D -1; + + fd =3D open(loop_path, O_RDWR | O_DIRECT | O_CLOEXEC); + if (fd < 0) + return -1; + if (posix_memalign(&buf, 4096, IO_SIZE)) + goto out_fd; + memset(buf, 0xab, IO_SIZE); + + if (pwrite(fd, buf, IO_SIZE, 0) !=3D IO_SIZE) + goto out_buf; + fsync(fd); + if (pread(fd, buf, IO_SIZE, 0) !=3D IO_SIZE) + goto out_buf; + ret =3D 0; +out_buf: + free(buf); +out_fd: + close(fd); + return ret; +} + +/* Read @dev's io.stat counters. @dev uses kernel dev_t encoding. */ +static int parse_io_stat(int cgroup_fd, __u64 dev, struct io_query *out) +{ + unsigned int want_maj =3D dev >> 20, want_min =3D dev & ((1U << 20) - 1); + char buf[4096], *line, *saveptr; + int fd, n, ret =3D -1; + + fd =3D openat(cgroup_fd, "io.stat", O_RDONLY); + if (fd < 0) + return -1; + n =3D read(fd, buf, sizeof(buf) - 1); + close(fd); + if (n <=3D 0) + return -1; + buf[n] =3D '\0'; + + for (line =3D strtok_r(buf, "\n", &saveptr); line; + line =3D strtok_r(NULL, "\n", &saveptr)) { + unsigned long long rb =3D 0, wb =3D 0, ri =3D 0, wi =3D 0, db =3D 0, di = =3D 0; + unsigned int maj, min; + + /* Only the device id is required; missing counters stay zero. */ + if (sscanf(line, + "%u:%u rbytes=3D%llu wbytes=3D%llu rios=3D%llu wios=3D%llu dbytes=3D= %llu dios=3D%llu", + &maj, &min, &rb, &wb, &ri, &wi, &db, &di) < 2) + continue; + if (maj !=3D want_maj || min !=3D want_min) + continue; + + out->rbytes =3D rb; + out->wbytes =3D wb; + out->rios =3D ri; + out->wios =3D wi; + out->dbytes =3D db; + out->dios =3D di; + ret =3D 0; + break; + } + return ret; +} + +void test_cgroup_iter_io(void) +{ + char *cgroup_rel_path =3D "/cgroup_iter_io_test"; + int ctl_fd =3D -1, loop_fd =3D -1, back_fd =3D -1; + struct cgroup_iter_io *skel =3D NULL; + struct bpf_link *link =3D NULL; + char loop_path[64]; + struct io_query *q; + int cgroup_fd; + + cgroup_fd =3D cgroup_setup_and_join(cgroup_rel_path); + if (!ASSERT_OK_FD(cgroup_fd, "cgroup_setup_and_join")) + return; + + if (loop_setup(loop_path, sizeof(loop_path), &ctl_fd, &loop_fd, &back_fd)= ) { + test__skip(); /* needs root + CONFIG_BLK_DEV_LOOP */ + goto cleanup_cgroup_fd; + } + + skel =3D cgroup_iter_io__open_and_load(); + if (!ASSERT_OK_PTR(skel, "cgroup_iter_io__open_and_load")) + goto cleanup_loop; + + /* Convert glibc st_rdev to kernel dev_t format. */ + { + struct stat lst; + + if (!ASSERT_OK(fstat(loop_fd, &lst), "fstat loop")) + goto cleanup_skel; + skel->data_query->target_dev =3D + ((__u64)major(lst.st_rdev) << 20) | minor(lst.st_rdev); + } + + DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts); + union bpf_iter_link_info linfo =3D { + .cgroup.cgroup_fd =3D cgroup_fd, + .cgroup.order =3D BPF_CGROUP_ITER_SELF_ONLY, + }; + opts.link_info =3D &linfo; + opts.link_info_len =3D sizeof(linfo); + + link =3D bpf_program__attach_iter(skel->progs.cgroup_io_query, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto cleanup_skel; + + /* This process is in the test cgroup, so the loop I/O is charged here. */ + if (!ASSERT_OK(do_direct_io(loop_path), "do_direct_io")) + goto cleanup_link; + + if (!ASSERT_OK(read_stats(link), "read stats")) + goto cleanup_link; + + q =3D &skel->data_query->io_query; + if (test__start_subtest("cgroup_iter_io__write")) { + ASSERT_GT(q->wbytes, 0, "wbytes"); + ASSERT_GT(q->wios, 0, "wios"); + } + if (test__start_subtest("cgroup_iter_io__read")) { + ASSERT_GT(q->rbytes, 0, "rbytes"); + ASSERT_GT(q->rios, 0, "rios"); + } + if (test__start_subtest("cgroup_iter_io__dev")) + ASSERT_GT(q->dev, 0, "dev"); + + /* Compare with io.stat without I/O between the reads. */ + if (test__start_subtest("cgroup_iter_io__match")) { + struct io_query filev =3D {}; + + if (ASSERT_OK(read_stats(link), "read stats") && + ASSERT_OK(parse_io_stat(cgroup_fd, q->dev, &filev), + "parse io.stat")) { + ASSERT_EQ(q->rbytes, filev.rbytes, "rbytes"); + ASSERT_EQ(q->wbytes, filev.wbytes, "wbytes"); + ASSERT_EQ(q->rios, filev.rios, "rios"); + ASSERT_EQ(q->wios, filev.wios, "wios"); + ASSERT_EQ(q->dbytes, filev.dbytes, "dbytes"); + ASSERT_EQ(q->dios, filev.dios, "dios"); + } + } + + /* Root statistics include this cgroup's I/O. */ + if (test__start_subtest("cgroup_iter_io__root")) { + struct bpf_link *root_link; + struct io_query *r; + + skel->data_query->got_root_css =3D 0; + root_link =3D bpf_program__attach_iter(skel->progs.cgroup_root_io_query, + &opts); + if (ASSERT_OK_PTR(root_link, "attach root iter")) { + if (ASSERT_OK(read_stats(root_link), "read root stats")) { + r =3D &skel->data_query->root_query; + ASSERT_EQ(skel->data_query->got_root_css, 1, + "got_root_css"); + ASSERT_EQ(r->dev, q->dev, "root dev"); + ASSERT_GE(r->wbytes, q->wbytes, "root wbytes"); + ASSERT_GE(r->wios, q->wios, "root wios"); + ASSERT_GE(r->rbytes, q->rbytes, "root rbytes"); + ASSERT_GE(r->rios, q->rios, "root rios"); + } + bpf_link__destroy(root_link); + } + } + +cleanup_link: + bpf_link__destroy(link); +cleanup_skel: + cgroup_iter_io__destroy(skel); +cleanup_loop: + loop_teardown(loop_path, ctl_fd, loop_fd, back_fd); +cleanup_cgroup_fd: + close(cgroup_fd); + cleanup_cgroup_environment(); +} diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_io.c b/tools/tes= ting/selftests/bpf/progs/cgroup_iter_io.c new file mode 100644 index 000000000000..2cd538068987 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c @@ -0,0 +1,99 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include "bpf_experimental.h" +#include "cgroup_iter_io.h" + +char _license[] SEC("license") =3D "GPL"; + +struct io_query io_query SEC(".data.query"); + +struct io_query root_query SEC(".data.query"); + +__u64 got_root_css SEC(".data.query"); + +/* Device selected by userspace in kernel dev_t format. */ +__u64 target_dev SEC(".data.query"); + +/* Keep inline: RCU and open-coded iterators cannot cross a BPF call. */ +static __always_inline int read_target_dev(struct cgroup *cgrp, + struct io_query *out) +{ + struct cgroup_subsys_state *css; + struct blkcg_gq *pos; + __u64 dev; + int ssid; + + /* The flush can sleep, so run it before the RCU section. */ + bpf_blkcg_flush_stats(cgrp); + + bpf_rcu_read_lock(); + ssid =3D bpf_core_enum_value(enum cgroup_subsys_id, io_cgrp_id); + + /* + * subsys[] is __rcu, so this read gives an RCU pointer the iterator + * accepts. BPF_CORE_READ() would return a plain value instead. + */ + css =3D cgrp->subsys[ssid]; + if (!css) { + bpf_rcu_read_unlock(); + return 0; + } + + bpf_for_each(blkg, pos, css) { + dev =3D BPF_CORE_READ(pos, q, disk, part0, bd_dev); + if (dev !=3D target_dev) + continue; + + out->dev =3D dev; + out->rbytes =3D BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_READ]); + out->wbytes =3D BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_WRITE]); + out->rios =3D BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_READ]); + out->wios =3D BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_WRITE]); + out->dbytes =3D BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_DISCARD]= ); + out->dios =3D BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_DISCARD]); + break; + } + bpf_rcu_read_unlock(); + return 1; +} + +SEC("iter.s/cgroup") +int cgroup_io_query(struct bpf_iter__cgroup *ctx) +{ + struct cgroup *cgrp =3D ctx->cgroup; + + if (!cgrp) + return 1; + + /* Start fresh so a device that is not found stays all-zero. */ + __builtin_memset(&io_query, 0, sizeof(io_query)); + + read_target_dev(cgrp, &io_query); + return 0; +} + +SEC("iter.s/cgroup") +int cgroup_root_io_query(struct bpf_iter__cgroup *ctx) +{ + struct cgroup *root; + + if (!ctx->cgroup) + return 1; + + __builtin_memset(&root_query, 0, sizeof(root_query)); + + /* The root cgroup always has id 1. */ + root =3D bpf_cgroup_from_id(1); + if (!root) + return 0; + + /* Root counters include all cgroups' I/O. */ + if (read_target_dev(root, &root_query)) + got_root_css =3D 1; + + bpf_cgroup_release(root); + return 0; +} --=20 2.53.0-Meta