From nobody Mon Sep 28 14:46:59 2026 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 118E7233954 for ; Thu, 20 Aug 2026 21:18:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260691; cv=none; b=NxkRGf9wEnfcIjnDnSIQuepSIUgEabfIA9rV5D1r7X208DkyJq8XOr9/O1bSLaDZUdVOEoAD7i3fzbjQVkArUhjCg7lGaq6HwApx60sQgpAsVF6aokxk5+RjnWYra9dWEovaNJOqs7OV7ozEuEPkYwSeMont3PaoX6VzNX0K56M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260691; c=relaxed/simple; bh=YjqdsGJuMeIq7n8PzjUvmgIYfZOosAHqNR3Q2LYldfs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u1q5wstN38E+Dpxfmo9qLrNX0TTiMASUDOwDBcNkaUp2Y+5D+N8SNkboYFVpuUQJpJLNZ/2qWMiZcwC3hRmxNBmEkU8pitxJh9IqiPZIngHF4dgkHe5DD7cNdrqJpR289PL482FuEznRToCqsEq6Q5QENR8diF0aZPpWsndZrgg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JbQuxgBK; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JbQuxgBK" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-cc149372c14so279964a12.1 for ; Thu, 20 Aug 2026 14:18:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787260683; x=1787865483; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fNnF6T8sz4WljkkHVGCRYU0uuUkbDNh6HBGKfIf2U2g=; b=JbQuxgBKFQmH25J3svttJ5EyFkABkSSDF8T4fxc5zuOmxOhIDPuxZj/iBDwmsKwWwF yg/wUGCvnPmDZ4ghB6GSFJauxMGIm4Crw3oPttlhcmg1+5h9TGwYZl9P8fACiaG88Nr4 niV2XR8Zh7SYcMcJl/9oXnzTfivZ+T8KSCq2Ni1vQSI7MQ7yJAyrdE2tjAXL/Ny8rYjN yPkIK24exLqzRYleEpa20U3T14LHE6XUykgKlz7kEuEoXza+C5ibFYH0dWo8zzue2Z++ HojkILr0m348OeH1PAbv6ma6mGXFNpWOc79+DDRkepKbcPKOWjRVQzzpDmdl1dvtMW4e pnQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787260683; x=1787865483; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=fNnF6T8sz4WljkkHVGCRYU0uuUkbDNh6HBGKfIf2U2g=; b=q7o6lwVXFBaDKDq7YuPEiNyA8c7yyZVgdOeCrFyKiMyfBeBOJihvad2FbAhSZSRlTX qt/lkNymSxRMCs0EpM3CuZkkkc5XnHfn+mgf4N8yv8p7KKmgRhGZW7NPujjq3v3E3oS0 OCKy2HH5Kec2oJRu0T/e70duJofXw0kxz2+1sdnFGnkCQt7pLyyxTStVn3zeB0W3jblB zFTPaSU24rN9O0NgyCXP7HkZbd79Rmvk/sXO2iutQ3BI27y1y3Ml6POKseJbfbfdpWVu piahkJCCMORFf9QT/AJrlfkj/CbetReNO4VuLNimN97FbeO1lfTCI/5ucBzc6ix5/h8e qE6A== X-Forwarded-Encrypted: i=1; AHgh+RphaR2KJ63mYWGy23/zOQkQbQkxaGuxlaq7uImdkGe2dsQqS0cuVDXVZa+wkBQdyS8M26IUJoDDx4koHXQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxTTAi2Aun7ZPXt3dtoVx0nPNGF5Bi2T6uRSBUdfETpF/YmRDqX dSKbuwXkdg4M7Hd5LMJD0xRioHs0XdyJsxa1smlGDZRgU/uxlgEJBHJP X-Gm-Gg: AR+sD10rIyJ3f4y4LVprSPq1+mgYANh+8yKp+FFvSSQldyj3OPxMgG9mKwUPB3bNq0N U8IDkD5SG1FmZGy/gy6/COQJIqzWPyWRsex/mfkkty47TO+ghlFX2XH5n+j+k4jwaCs5Pcd5GnF eFYdnL9CIDcZps0epLufS0ngihnIjLzKll0gJ6E4UuRJndNybDydZApVn0Zvp5kyHp1XCcqoXLh zIVwr2FE2ReOB3kYyjHsaWgxbcQULt8TA6u0aGbcLlL+Yy+J6ossyhVjyWPOzxuGd10nImlakwF m6pL5jrMx5+TUIGn7Nl2dj7hyPNxH0QZL4Cm0xU1JJjoGGOVBuAxXO6hsWZUxD46RxpkdctDCm/ S/npNjwlFFoVUKtEPy5LcsnmTIh84oMeKte6kCFTRkaiwrJeYuV4amYNGYLPv8Q2pFMdP7MD0C9 c8x+zZcG11CTQBBHoQkMmT9q/QFzueDmrqbXek0mirR7KXxcClzPiPw8Q= X-Received: by 2002:a05:6a20:450c:b0:3bf:6d96:ac40 with SMTP id adf61e73a8af0-3cd3007ce9bmr2646024637.12.1787260682945; Thu, 20 Aug 2026 14:18:02 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:72::]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-327bef3993dsm17917418eec.5.2026.08.20.14.18.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 14:18:02 -0700 (PDT) From: Ziyang Men To: kernel-team@meta.com, Jens Axboe , Tejun Heo , Josef Bacik , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Shuah Khan Cc: Ingo Molnar , Peter Zijlstra , Vincent Guittot , Ben Segall , Dietmar Eggemann , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Roman Gushchin , Shakeel Butt , JP Kobryn , Mykola Lysenko , Ziyang Men , linux-block@vger.kernel.org, bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 1/4] cgroup: add BPF kfuncs to read a cpu cgroup's stats Date: Thu, 20 Aug 2026 14:17:55 -0700 Message-ID: <20260820211758.3393984-2-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260820211758.3393984-1-ziyang.meme@gmail.com> References: <20260820211758.3393984-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Collecting cgroup statistics is expensive because the existing method opens and parses a cgroup file. memcg already provides an efficient BPF interface; extend that model to the CPU controller. Register css_rstat_flush() as a common kfunc and add bpf_cgroup_base_stat(). The latter returns cgroup_base_stat after the same cputime adjustment used by cpu.stat. The BPF program reads the plain CFS bandwidth counters directly. Add bpf_css_to_task_group() to check the controller and give the verifier a typed task_group pointer for bpf_per_cpu_ptr(). css_rstat_flush() may reschedule and requires a sleepable program. bpf_cgroup_base_stat() only takes locks and is not marked sleepable, but those locks are not NMI-safe. Suggested-by: Shakeel Butt Suggested-by: Tejun Heo Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- kernel/cgroup/Makefile | 2 ++ kernel/cgroup/bpf_cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++ kernel/cgroup/rstat.c | 54 ++++++++++++++++++++++++++++++++--- 3 files changed, 110 insertions(+), 4 deletions(-) create mode 100644 kernel/cgroup/bpf_cgroup.c diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile index ede31601a363..29f29228865b 100644 --- a/kernel/cgroup/Makefile +++ b/kernel/cgroup/Makefile @@ -1,6 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 obj-y :=3D cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o =20 +obj-$(CONFIG_BPF_SYSCALL) +=3D bpf_cgroup.o + obj-$(CONFIG_CGROUP_FREEZER) +=3D legacy_freezer.o obj-$(CONFIG_CGROUP_PIDS) +=3D pids.o obj-$(CONFIG_CGROUP_RDMA) +=3D rdma.o diff --git a/kernel/cgroup/bpf_cgroup.c b/kernel/cgroup/bpf_cgroup.c new file mode 100644 index 000000000000..cd28c838dc7b --- /dev/null +++ b/kernel/cgroup/bpf_cgroup.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Cgroup BPF kfuncs + * + * Author: Ziyang Men + */ + +#include +#include +#include + +#include "../sched/sched.h" + +#ifdef CONFIG_CGROUP_SCHED +__bpf_kfunc_start_defs(); + +/** + * bpf_css_to_task_group - Cast a CPU controller css to its task group + * @css: CPU controller css + * + * Must be called under RCU. The kfunc gives BPF a typed task_group pointe= r. + * + * Return: The task group, or NULL if @css belongs to another controller. + */ +__bpf_kfunc struct task_group * +bpf_css_to_task_group(struct cgroup_subsys_state *css) +{ + if (unlikely(css->ss !=3D &cpu_cgrp_subsys)) + return NULL; + + return container_of(css, struct task_group, css); +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(bpf_cpu_cgroup_kfunc_ids) +BTF_ID_FLAGS(func, bpf_css_to_task_group, + KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL) +BTF_KFUNCS_END(bpf_cpu_cgroup_kfunc_ids) + +static const struct btf_kfunc_id_set bpf_cpu_cgroup_kfunc_set =3D { + .owner =3D THIS_MODULE, + .set =3D &bpf_cpu_cgroup_kfunc_ids, +}; + +static int __init bpf_cpu_cgroup_kfunc_init(void) +{ + int err; + + err =3D register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, + &bpf_cpu_cgroup_kfunc_set); + if (err) + pr_warn("error while registering cpu cgroup kfuncs: %d\n", err); + + return err; +} +late_initcall(bpf_cpu_cgroup_kfunc_init); +#endif /* CONFIG_CGROUP_SCHED */ diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index de816a43db9f..5db72504a8a5 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -752,10 +752,49 @@ void cgroup_base_stat_cputime_show(struct seq_file *s= eq) cgroup_force_idle_show(seq, &bstat); } =20 -/* Add bpf kfuncs for css_rstat_updated() and css_rstat_flush() */ +#ifdef CONFIG_BPF_SYSCALL + +__bpf_kfunc_start_defs(); + +/** + * bpf_cgroup_base_stat - Read a cgroup's base statistics + * @cgrp: cgroup to read from + * @out: zero-initialized output in nanoseconds + * + * CPU time is adjusted as for cpu.stat. + */ +__bpf_kfunc void bpf_cgroup_base_stat(struct cgroup *cgrp, + struct cgroup_base_stat *out) +{ + if (cgroup_parent(cgrp)) { + __css_rstat_lock(&cgrp->self, -1); + *out =3D cgrp->bstat; + cputime_adjust(&cgrp->bstat.cputime, &cgrp->prev_cputime, + &out->cputime.utime, &out->cputime.stime); + __css_rstat_unlock(&cgrp->self, -1); + } else { + root_cgroup_cputime(out); + } +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(bpf_rstat_common_kfunc_ids) +BTF_ID_FLAGS(func, css_rstat_flush, KF_SLEEPABLE) +/* The reader does not sleep, but its locks are not NMI-safe. */ +BTF_ID_FLAGS(func, bpf_cgroup_base_stat) +BTF_KFUNCS_END(bpf_rstat_common_kfunc_ids) + +static const struct btf_kfunc_id_set bpf_rstat_common_kfunc_set =3D { + .owner =3D THIS_MODULE, + .set =3D &bpf_rstat_common_kfunc_ids, +}; + +#endif /* CONFIG_BPF_SYSCALL */ + +/* Add a bpf kfunc for css_rstat_updated(). */ BTF_KFUNCS_START(bpf_rstat_kfunc_ids) BTF_ID_FLAGS(func, css_rstat_updated) -BTF_ID_FLAGS(func, css_rstat_flush, KF_SLEEPABLE) BTF_KFUNCS_END(bpf_rstat_kfunc_ids) =20 static const struct btf_kfunc_id_set bpf_rstat_kfunc_set =3D { @@ -765,7 +804,14 @@ static const struct btf_kfunc_id_set bpf_rstat_kfunc_s= et =3D { =20 static int __init bpf_rstat_kfunc_init(void) { - return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, - &bpf_rstat_kfunc_set); + int ret; + + ret =3D register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, + &bpf_rstat_kfunc_set); +#ifdef CONFIG_BPF_SYSCALL + ret =3D ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, + &bpf_rstat_common_kfunc_set); +#endif + return ret; } late_initcall(bpf_rstat_kfunc_init); --=20 2.53.0-Meta From nobody Mon Sep 28 14:46:59 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 108F9485502 for ; Thu, 20 Aug 2026 21:18:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260694; cv=none; b=QHl92ycRxKeucrC2fCEo8053uzy0gyDEaEEmXp+dARe1A26RufxezNJ7+3B+CNoOa1z1KDaJT7mzwnk07jY0taaXiwCOc+4woNeSGjD1LdLokZ0caBtdSl25Kz/Yf71rkz3XT0CxffkGbRP4HC04xQCe7aDlKmfzapx5XlGv1ys= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260694; c=relaxed/simple; bh=OJSpf69yMcsfGIyak4mMels4G2wsnfckBxcquoS3tvs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bGEyNYbdw1LQr5sxHuX+YHjrhEgbaDhH3EheFhwjnjvMXZuGCUyG85hefVzqbXyo414ySxttWO8Lj4bVsQ9NKwyBv/z8b7RLvipxLDUDfU3xtT3+LuCbo8fh4elaxyL3SVGxLlwP3jl3PrXEygxZn1MYtQ3jSEGjYkcMFcwgbZE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aTiCEloJ; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aTiCEloJ" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2caea3f742bso6019705ad.0 for ; Thu, 20 Aug 2026 14:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787260685; x=1787865485; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=si8wyumoq2U2f4OwgM7HXzPkBFFrINH56EYQucTf6UU=; b=aTiCEloJfUL19anIQoysndVlNACXvFQ7bLbZDTIqCENAihLneesxO5x0U2dhVqIWmO iw/Jb93aJEHauKy+oLgQneSyONL4svUbByBxlEaJnOViSiIdVUEk5gInBdmY7wWjl0vT HyQkL+aVsFU6MaF8h6F/anXaU+mY09leGh6DCwNyYVrgIXjfWVDvQbiZmZSizy7Xo+UB g5h3w45PWhhIemE+rMOaWdBhChR9dmY0DEWkw/A9K2eK2dpsG6yEAWIZZxgCTxSvsFmf Mx/Ri+Q4eRR91BUdymWaV7snIRjI4fYuP/5GIArJyaa+dZ5u0U+YA9Jf14Q2rQUHKy84 7DvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787260685; x=1787865485; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=si8wyumoq2U2f4OwgM7HXzPkBFFrINH56EYQucTf6UU=; b=a1S9RyDAOHTkH0oYtT8Xk5tu6nY9lGi1PvMbhXGkBCjof//6Ron42eeZHzPDReJfxd l2AAzdRrBCr1Q/MBM2ZI6FJXaRtdL3W6pNsQMYPuikzB6MhLiniVmLaRrd8WoooCGahy zi1sz1lhZG26cfmAvHS+rsy39BG4I2Vn9740lPZ0+5VUUM2OAF9co1Xi5LNF/9RH8ddZ suR2jNXiIVM091kss0RLk7Qk/+gB6TINEgP+0jZT7ltT4RbmE/JhtHShhKfBkiaC5h8x jIo5rB6HP7/f0AErzQ3fsvcW0nCZrDary5As80R+6uj1ub2sIZwMoyngZGdnta7Ji/eT 85bg== X-Forwarded-Encrypted: i=1; AHgh+RqoKNmOIG1txre4x1dE5Tg3Igp7x6gcZXtz0Hrp/RJB3b094tICiI+++aQaVx+Ii6NRfFI4DXBqCJsBn5g=@vger.kernel.org X-Gm-Message-State: AOJu0YxIZ+VlRIjvAYSETpUzrmWS09ikOYphDGogKEv/agfBNoH9wUs8 VDl1aUq82YdtI9bWopWsKmN8wC+OQnagSsSlFD973Boa0RI9CB9CaSgg X-Gm-Gg: AR+sD13C1HoB4rk3SxVJHLL8LVNOrADhMd1eteboO1Gy8VoA/xQveTs3/UPgTP+0fsl C1Rf+tPo7Koxsm3hIaxNEMJ8SCGQyyjdzcLBR6APKZrrl6gryq9Zs6jTWLquu7XXwtZ3PyFt7nD YvipTBXXXwEJOzBidFh0hNnjO5qPmW1Mahk0aIWK3cfl+xLIXcrYYqSmo2ar132uFDhDIlrjCwk zyc8vqibqTiztkebxKWqQl/wQJjbcMzd9vqrYAkiEwVRxEcowynmvgYTWy3hBwdflzmCQuo/v/u nX1BWSTKGdXBdWdsir6ghfktfHFo9UNoafng0y3WS+emMba89dfA3A4AVs2tMcCkldd0MzBCoLf vaRGGtKEiWJcyqwiXXCKxgDRmqqfZi5HK/VBr1lB/IYEZ9AtkPG9nsVonBmmlWTVBpaOsgX4XaT BoM5nlVBGzquZoRGVDmf4OXzBuPAcCkpQ+LbEplGKk7QrLmHx2PYJjHjLF45TsjyYOfQ== X-Received: by 2002:a05:6a21:790:b0:3c3:7ac4:dac0 with SMTP id adf61e73a8af0-3cd300d467dmr2643686637.13.1787260684565; Thu, 20 Aug 2026 14:18:04 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:40::]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-1416ae1416dsm26285932c88.11.2026.08.20.14.18.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 14:18:04 -0700 (PDT) From: Ziyang Men To: kernel-team@meta.com, Jens Axboe , Tejun Heo , Josef Bacik , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Shuah Khan Cc: Ingo Molnar , Peter Zijlstra , Vincent Guittot , Ben Segall , Dietmar Eggemann , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Roman Gushchin , Shakeel Butt , JP Kobryn , Mykola Lysenko , Ziyang Men , linux-block@vger.kernel.org, bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 2/4] selftests/bpf: add cgroup_iter_cpu test for cpu cgroup kfuncs Date: Thu, 20 Aug 2026 14:17:56 -0700 Message-ID: <20260820211758.3393984-3-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260820211758.3393984-1-ziyang.meme@gmail.com> References: <20260820211758.3393984-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add cgroup_iter_cpu, a selftest for the CPU controller BPF kfuncs. The userspace side runs a CPU hog in a test cgroup with cpu.max settled then: - checks the CPU-time and throttling counters are nonzero, - compares whether all values the program read are same as those reading from cgroup file. Enable CONFIG_CGROUP_SCHED, CONFIG_FAIR_GROUP_SCHED and CONFIG_CFS_BANDWIDTH in the selftest config. Tested on VM with v7.2-rc5. Suggested-by: Shakeel Butt Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- tools/testing/selftests/bpf/cgroup_iter_cpu.h | 22 ++ tools/testing/selftests/bpf/config | 3 + .../bpf/prog_tests/cgroup_iter_cpu.c | 259 ++++++++++++++++++ .../selftests/bpf/progs/cgroup_iter_cpu.c | 113 ++++++++ 4 files changed, 397 insertions(+) create mode 100644 tools/testing/selftests/bpf/cgroup_iter_cpu.h create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c diff --git a/tools/testing/selftests/bpf/cgroup_iter_cpu.h b/tools/testing/= selftests/bpf/cgroup_iter_cpu.h new file mode 100644 index 000000000000..74599a5c0e4d --- /dev/null +++ b/tools/testing/selftests/bpf/cgroup_iter_cpu.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#ifndef __CGROUP_ITER_CPU_H +#define __CGROUP_ITER_CPU_H + +struct cpu_query { + /* base cpu time, from cpu.stat */ + __u64 usage_usec; + __u64 user_usec; + __u64 system_usec; + __u64 nice_usec; + __u64 forceidle_usec; + /* CFS bandwidth throttling, from cpu.stat and cpu.stat.local */ + __u64 nr_periods; + __u64 nr_throttled; + __u64 throttled_usec; + __u64 nr_bursts; + __u64 burst_usec; + __u64 throttled_self_usec; +}; + +#endif /* __CGROUP_ITER_CPU_H */ diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/b= pf/config index ea7044f30adc..482b40dde2f9 100644 --- a/tools/testing/selftests/bpf/config +++ b/tools/testing/selftests/bpf/config @@ -11,6 +11,9 @@ CONFIG_BPF_STREAM_PARSER=3Dy CONFIG_BPF_SYSCALL=3Dy # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set CONFIG_CGROUP_BPF=3Dy +CONFIG_CGROUP_SCHED=3Dy +CONFIG_FAIR_GROUP_SCHED=3Dy +CONFIG_CFS_BANDWIDTH=3Dy CONFIG_CRYPTO_HMAC=3Dy CONFIG_CRYPTO_SHA256=3Dy CONFIG_CRYPTO_USER_API=3Dy diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c b/too= ls/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c new file mode 100644 index 000000000000..cd7e92ababfb --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c @@ -0,0 +1,259 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include +#include +#include +#include "cgroup_helpers.h" +#include "cgroup_iter_cpu.h" +#include "cgroup_iter_cpu.skel.h" + +static int read_stats(struct bpf_link *link) +{ + int fd, ret =3D 0; + ssize_t bytes; + + fd =3D bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_OK_FD(fd, "bpf_iter_create")) + return 1; + + bytes =3D read(fd, NULL, 0); + if (!ASSERT_EQ(bytes, 0, "read fd")) + ret =3D 1; + + close(fd); + return ret; +} + +/* Read cgroup file @name into @buf. */ +static int read_cgroup_file(int cgroup_fd, const char *name, char *buf, + size_t size) +{ + ssize_t n; + int fd; + + fd =3D openat(cgroup_fd, name, O_RDONLY); + if (fd < 0) + return -1; + n =3D read(fd, buf, size - 1); + close(fd); + if (n <=3D 0) + return -1; + buf[n] =3D '\0'; + return 0; +} + +/* Parse the "cpu.stat" file into @out. */ +static int parse_cpu_stat(int cgroup_fd, struct cpu_query *out) +{ + char buf[4096], *line, *sp; + unsigned long long v; + + if (read_cgroup_file(cgroup_fd, "cpu.stat", buf, sizeof(buf))) + return -1; + + for (line =3D strtok_r(buf, "\n", &sp); line; + line =3D strtok_r(NULL, "\n", &sp)) { + if (sscanf(line, "usage_usec %llu", &v) =3D=3D 1) + out->usage_usec =3D v; + else if (sscanf(line, "user_usec %llu", &v) =3D=3D 1) + out->user_usec =3D v; + else if (sscanf(line, "system_usec %llu", &v) =3D=3D 1) + out->system_usec =3D v; + else if (sscanf(line, "nice_usec %llu", &v) =3D=3D 1) + out->nice_usec =3D v; + else if (sscanf(line, "core_sched.force_idle_usec %llu", &v) =3D=3D 1) + out->forceidle_usec =3D v; + else if (sscanf(line, "nr_periods %llu", &v) =3D=3D 1) + out->nr_periods =3D v; + else if (sscanf(line, "nr_throttled %llu", &v) =3D=3D 1) + out->nr_throttled =3D v; + else if (sscanf(line, "throttled_usec %llu", &v) =3D=3D 1) + out->throttled_usec =3D v; + else if (sscanf(line, "nr_bursts %llu", &v) =3D=3D 1) + out->nr_bursts =3D v; + else if (sscanf(line, "burst_usec %llu", &v) =3D=3D 1) + out->burst_usec =3D v; + } + return 0; +} + +/* + * Parse the "cpu.stat.local" file into @out. + */ +static int parse_cpu_stat_local(int cgroup_fd, struct cpu_query *out) +{ + unsigned long long v; + char buf[256]; + + if (read_cgroup_file(cgroup_fd, "cpu.stat.local", buf, sizeof(buf))) + return -1; + if (sscanf(buf, "throttled_usec %llu", &v) !=3D 1) + return -1; + out->throttled_self_usec =3D v; + return 0; +} + +/* Read file value the bpf program reads. */ +static int parse_stats(int cgroup_fd, struct cpu_query *out, bool have_bw) +{ + if (parse_cpu_stat(cgroup_fd, out)) + return -1; + if (have_bw && parse_cpu_stat_local(cgroup_fd, out)) + return -1; + return 0; +} + +/* + * Check whether this kernel accounts CFS bandwidth. + */ +static bool cgroup_has_bw_stat(int cgroup_fd) +{ + char buf[4096]; + + if (read_cgroup_file(cgroup_fd, "cpu.stat", buf, sizeof(buf))) + return false; + return strstr(buf, "nr_periods "); +} + +/* Fork a child that spins in the current cgroup, kill it if the test exit= s. */ +static pid_t spawn_cpu_hog(void) +{ + pid_t pid =3D fork(); + + if (pid =3D=3D 0) { + prctl(PR_SET_PDEATHSIG, SIGKILL); + while (1) + ; + } + return pid; +} + +void test_cgroup_iter_cpu(void) +{ + char *cgroup_rel_path =3D "/cgroup_iter_cpu_test"; + struct cgroup_iter_cpu *skel; + struct cpu_query *q; + struct bpf_link *link; + bool wrote_max, have_bw; + int cgroup_fd; + pid_t hog; + + cgroup_fd =3D cgroup_setup_and_join(cgroup_rel_path); + if (!ASSERT_OK_FD(cgroup_fd, "cgroup_setup_and_join")) + return; + + wrote_max =3D !write_cgroup_file(cgroup_rel_path, "cpu.max", "10000 10000= 0"); + + skel =3D cgroup_iter_cpu__open_and_load(); + if (!ASSERT_OK_PTR(skel, "cgroup_iter_cpu__open_and_load")) + goto cleanup_cgroup_fd; + + DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts); + union bpf_iter_link_info linfo =3D { + .cgroup.cgroup_fd =3D cgroup_fd, + .cgroup.order =3D BPF_CGROUP_ITER_SELF_ONLY, + }; + opts.link_info =3D &linfo; + opts.link_info_len =3D sizeof(linfo); + + link =3D bpf_program__attach_iter(skel->progs.cgroup_cpu_query, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto cleanup_skel; + + q =3D &skel->data_query->cpu_query; + + hog =3D spawn_cpu_hog(); + if (!ASSERT_GT(hog, 0, "spawn_cpu_hog")) + goto cleanup_link; + + sleep(1); + + /* Run the bpf program before anything here reads cpu.stat. */ + if (!ASSERT_OK(read_stats(link), "read stats")) + goto cleanup_hog; + + have_bw =3D wrote_max && cgroup_has_bw_stat(cgroup_fd); + + if (test__start_subtest("cgroup_iter_cpu__cputime")) { + ASSERT_GT(q->usage_usec, 0, "usage_usec"); + ASSERT_GT(q->user_usec + q->system_usec, 0, "user+system_usec"); + } + if (test__start_subtest("cgroup_iter_cpu__throttling")) { + if (!have_bw) { + test__skip(); + } else { + ASSERT_GT(q->nr_periods, 0, "nr_periods"); + ASSERT_GT(q->nr_throttled, 0, "nr_throttled"); + ASSERT_GT(q->throttled_usec, 0, "throttled_usec"); + ASSERT_GT(q->throttled_self_usec, 0, "throttled_self_usec"); + } + } + + /* + * cpu.stat cputime grows on every tick a task in the cgroup runs, so + * stop them all before comparing + */ + if (test__start_subtest("cgroup_iter_cpu__match")) { + struct cpu_query filev =3D {}; + int i, stable =3D 0; + + kill(hog, SIGSTOP); + waitpid(hog, NULL, WUNTRACED); + if (!ASSERT_OK(join_root_cgroup(), "join_root_cgroup")) + goto cleanup_hog; + + /* + * The period timer keeps adding to nr_periods for a while + * after the hog stops + */ + for (i =3D 0; i < 20; i++) { + struct cpu_query before =3D {}, after =3D {}; + + if (!ASSERT_OK(parse_stats(cgroup_fd, &before, have_bw), "cpu.stat") || + !ASSERT_OK(read_stats(link), "read stats") || + !ASSERT_OK(parse_stats(cgroup_fd, &after, have_bw), "cpu.stat")) + goto cleanup_hog; + + if (!memcmp(&before, &after, sizeof(before))) { + filev =3D before; + stable =3D 1; + break; + } + usleep(100000); + } + + if (!ASSERT_TRUE(stable, "cpu.stat stable")) + goto cleanup_hog; + + ASSERT_EQ(q->usage_usec, filev.usage_usec, "usage_usec"); + ASSERT_EQ(q->user_usec, filev.user_usec, "user_usec"); + ASSERT_EQ(q->system_usec, filev.system_usec, "system_usec"); + ASSERT_EQ(q->nice_usec, filev.nice_usec, "nice_usec"); + ASSERT_EQ(q->forceidle_usec, filev.forceidle_usec, "forceidle_usec"); + + if (have_bw) { + ASSERT_EQ(q->nr_periods, filev.nr_periods, "nr_periods"); + ASSERT_EQ(q->nr_throttled, filev.nr_throttled, "nr_throttled"); + ASSERT_EQ(q->throttled_usec, filev.throttled_usec, "throttled_usec"); + ASSERT_EQ(q->nr_bursts, filev.nr_bursts, "nr_bursts"); + ASSERT_EQ(q->burst_usec, filev.burst_usec, "burst_usec"); + ASSERT_EQ(q->throttled_self_usec, filev.throttled_self_usec, + "throttled_self_usec"); + } + } + +cleanup_hog: + kill(hog, SIGKILL); + waitpid(hog, NULL, 0); +cleanup_link: + bpf_link__destroy(link); +cleanup_skel: + cgroup_iter_cpu__destroy(skel); +cleanup_cgroup_fd: + close(cgroup_fd); + cleanup_cgroup_environment(); +} diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c b/tools/te= sting/selftests/bpf/progs/cgroup_iter_cpu.c new file mode 100644 index 000000000000..0e916234e7eb --- /dev/null +++ b/tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include "cgroup_iter_cpu.h" + +char _license[] SEC("license") =3D "GPL"; + +struct cpu_query cpu_query SEC(".data.query"); + +extern const void __cpu_possible_mask __ksym; + +struct cgroup_base_stat___local { + struct task_cputime cputime; + __u64 forceidle_sum; + __u64 ntime; +} __attribute__((preserve_access_index)); + +static __always_inline __u64 read_throttled_self(struct task_group *tg, __= u32 cpu) +{ + struct cfs_rq *cfs_rq; + + cfs_rq =3D bpf_per_cpu_ptr(tg->cfs_rq, cpu); + if (!cfs_rq) + return 0; + + return BPF_CORE_READ(cfs_rq, throttled_clock_self_time); +} + +SEC("iter.s/cgroup") +int cgroup_cpu_query(struct bpf_iter__cgroup *ctx) +{ + struct cgroup_base_stat___local bstat =3D {}; + struct cgroup *cgrp =3D ctx->cgroup; + struct cgroup_subsys_state *css; + struct task_group *tg; + __u64 throttled_self =3D 0; + int ssid; + + if (!cgrp) + return 1; + + css_rstat_flush(&cgrp->self); + bpf_cgroup_base_stat(cgrp, (struct cgroup_base_stat *)&bstat); + + cpu_query.usage_usec =3D bstat.cputime.sum_exec_runtime / 1000; + cpu_query.user_usec =3D bstat.cputime.utime / 1000; + cpu_query.system_usec =3D bstat.cputime.stime / 1000; + cpu_query.nice_usec =3D bstat.ntime / 1000; + cpu_query.forceidle_usec =3D 0; + if (bpf_core_field_exists(bstat.forceidle_sum)) + cpu_query.forceidle_usec =3D bstat.forceidle_sum / 1000; + + bpf_rcu_read_lock(); + if (!bpf_core_enum_value_exists(enum cgroup_subsys_id, cpu_cgrp_id) || + !bpf_ksym_exists(bpf_css_to_task_group)) + goto unlock; + + ssid =3D bpf_core_enum_value(enum cgroup_subsys_id, cpu_cgrp_id); + css =3D cgrp->subsys[ssid]; + if (!css) + goto unlock; + + tg =3D bpf_css_to_task_group(css); + if (tg && bpf_core_field_exists(tg->cfs_bandwidth.nr_periods)) { + cpu_query.nr_periods =3D + (__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_periods); + cpu_query.nr_throttled =3D + (__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_throttled); + cpu_query.throttled_usec =3D + BPF_CORE_READ(tg, cfs_bandwidth.throttled_time) / 1000; + cpu_query.nr_bursts =3D + (__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_burst); + cpu_query.burst_usec =3D + BPF_CORE_READ(tg, cfs_bandwidth.burst_time) / 1000; + } + + if (tg && bpf_core_field_exists(tg->cfs_rq) && + bpf_core_field_exists(struct cfs_rq, throttled_clock_self_time)) { + __u32 mask_bytes =3D bpf_core_type_size(struct cpumask); + __u32 full_words =3D mask_bytes / sizeof(__u64); + int *cpu; + + if (full_words) + bpf_for_each(bits, cpu, + (const __u64 *)&__cpu_possible_mask, + full_words) + throttled_self +=3D read_throttled_self(tg, *cpu); + + if (mask_bytes & (sizeof(__u64) - 1)) { + __u32 tail =3D 0; + const void *src =3D (const char *)&__cpu_possible_mask + + full_words * sizeof(__u64); + int bit; + + if (!bpf_probe_read_kernel(&tail, sizeof(tail), src)) + bpf_for(bit, 0, 32) + if (tail & (1U << bit)) { + __u32 tail_cpu =3D full_words * 64 + bit; + + throttled_self +=3D + read_throttled_self(tg, tail_cpu); + } + } + } + +unlock: + bpf_rcu_read_unlock(); + cpu_query.throttled_self_usec =3D throttled_self / 1000; + + return 0; +} --=20 2.53.0-Meta From nobody Mon Sep 28 14:46:59 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 012ED485CF9 for ; Thu, 20 Aug 2026 21:18:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260694; cv=none; b=Eq8u3GcPfI2AYVQKepO1vx3w13VggPH6En/TAmjb+xmwMLrTKec1R9erCgW+0K4z531CFpYphwgCkVrOmCTYIz9skOT5UnfoUSZVfBX2G3loQ/PJkfXRBkkBtGpAUBQISZvH2EvEQGc3MnifpZvb2EwSrfVyNNDkqp9oPfBFafM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260694; c=relaxed/simple; bh=TDKffd9Fkjz0NiAHzS4kTGbp08DKmog+dRzwJOqZg8Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=paVW3n5Uot1wcUMkIK/vCOIHLtkK7hUmRwu0p0vwK6/mQnuQWcxSsOT3vzgh0RgtYOApYYU/8C/UFrFo+us5T5fOolSGNJ4bG9d9uG4D1ktgL87ap+XqxtBMFUNskFCN48e5VCPzNN9D64cr2KKrcKQLuRuMGxRDBviy2PebPcg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EDxkV+Yi; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EDxkV+Yi" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2ce87c7e3bbso3930485ad.1 for ; Thu, 20 Aug 2026 14:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787260686; x=1787865486; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=usT9UbJbbjXSkfskstVIX1ccjxiE/NzHiKaVTFkNt1M=; b=EDxkV+YiM+3P8ZsBGZDQBhg34IhQD3pdLX01CwkZRuHWiqwV6uUhmTaR5c074TqfF/ wh2eypQ4eZHNMUbJWR6QIESpaU0ftq+n7JwoVVPjMWLFNjFE9jQj05qmLMm21K48PjmC LRD8R1LFRLSQ6ddKD0fbuh9Otx4U4mgF89PX92i6A8HtHZgBqnYxRV2k5GynMDJ8OrVL u4astMIfbANf0oNNWuod6lk/iBwsGKirSxQCEHBNEf9jOBWk57ncuGk9iSHdQIJRbauD iut0DDwSuayt6koBJQFZkeiGSW96wXBQ9BTdo65+MKPMo4IJ26MBfyIKC1eedNS7EMxU sfiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787260686; x=1787865486; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=usT9UbJbbjXSkfskstVIX1ccjxiE/NzHiKaVTFkNt1M=; b=VmFRVHtUFAfQiDLTomD9bBKOqTTVB+SK2n4VJcG8Dk462Yl6gIKIm0Y5y3nK6d5USh LNXKhpEcGmFW9VVGziRbBosIlcOJ1tFG7cn/z95pdjomhk6ttfqqitlNRPjkyo5rczuL J8sxUIRZaEe5U8BQEfsk5JhSTHLzrkkqSmPGF3LpK+WADNg7THGZIUxI47LG5DHrXOkc P+0om3QLfc0A1qrUl6dMnJUZIxBo3pV4Eaql8UoZ1Ph/ocuA2LiEvC8wTAGgBnGGbsv5 P9GCDBiQLy7BRFy1wYs0dAP6Fet1nv1XgR2GOe7Oqla9TJ/1rTZIs53IWYCllY7V7zlt j8QQ== X-Forwarded-Encrypted: i=1; AHgh+Ro2/0QGS85eIU4Gwh9FBsYjecPFW+6/VNYM66A2dlgQPZYJ9nFP2FCXiiFnNV/4GVYp96Koru7zjz648Hc=@vger.kernel.org X-Gm-Message-State: AOJu0YzAf0Yh6zYi356WXwEsuktfboS7fEZdnEHHMxtmqEYBRr+7uiPw WswhXARb6tGSLFcjXgoWuM2OzG4FBRna1aCmKGi3+qU5qCWuD9XH0sHGAfvCWk0V X-Gm-Gg: AR+sD13Tv9FOlxKbDIT1ie7X67F4FXtZrSVi7tDHf1PVHV/SaMiZkWqC1EkEN55tDKs gDRoTBvSk0xIYCxK5BkGfDQqd0q+eIpynHZMzSOqLkhbUS6ScH5oE9k39uf0ZvrOsW0Gkg2pVdQ 0TK3wnY+HNfXi57dNlGLpUYw3qvUzrIKd/iJhmYIwxH8OUu2HfuX/huYztAnd5SDxEAW1MaivG4 Zw9P3BWa9AzDk27KCCMoXRhzOPDYmfZsktCKIO7cySToifcHpUj4yYNT3B37yOW2019hRGuF+xm PS0h1JiXd9umMv30POoUPK5XgGlgXQv3Vxq6z4Bz2MLOrYKROTDD0MoEMt2RA4nN6RpSOlVbTOF 95m8dMEG+Ta5k3d5DAqpIpGxd+W1vWj71iuiotYwIrnxiuRwLtZxVT3YZSQwbuCHPu5QPzWD7hW heNV8qxDzNbJJR0yM6xjdAVJuZU/fdm/F7lYqccIVb/Vo8zFKOMe0lCuk= X-Received: by 2002:a17:903:b07:b0:2d0:cc92:f7a3 with SMTP id d9443c01a7336-2d64ada8eb8mr29906555ad.2.1787260686112; Thu, 20 Aug 2026 14:18:06 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:66::]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-327d01ff056sm11599824eec.19.2026.08.20.14.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 14:18:05 -0700 (PDT) From: Ziyang Men To: kernel-team@meta.com, Jens Axboe , Tejun Heo , Josef Bacik , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Shuah Khan Cc: Ingo Molnar , Peter Zijlstra , Vincent Guittot , Ben Segall , Dietmar Eggemann , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Roman Gushchin , Shakeel Butt , JP Kobryn , Mykola Lysenko , Ziyang Men , linux-block@vger.kernel.org, bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 3/4] block: add BPF kfuncs to read blkcg io.stat Date: Thu, 20 Aug 2026 14:17:57 -0700 Message-ID: <20260820211758.3393984-4-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260820211758.3393984-1-ziyang.meme@gmail.com> References: <20260820211758.3393984-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Collecting cgroup statistics is expensive because the existing method opens and parses a cgroup file for every cgroup. memcg already provides an efficient BPF interface; extend that model to the block controller. Add bpf_cgroup_css() and bpf_css_release() to acquire a controller's css from a cgroup. The reference keeps the css alive across the sleepable css_rstat_flush(). Add bpf_css_to_blkcg() as a checked RCU-protected css-to-blkcg conversion and an open-coded iterator for the per-device blkgs. Suggested-by: Shakeel Butt Suggested-by: Tejun Heo Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- MAINTAINERS | 1 + block/Makefile | 3 + block/bpf_blkcg.c | 138 +++++++++++++++++++++++++++++++++++++ kernel/cgroup/bpf_cgroup.c | 62 ++++++++++++++--- 4 files changed, 194 insertions(+), 10 deletions(-) create mode 100644 block/bpf_blkcg.c diff --git a/MAINTAINERS b/MAINTAINERS index 2f9472c1a090..87c56e955577 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -6617,6 +6617,7 @@ F: block/blk-cgroup.c F: block/blk-iocost.c F: block/blk-iolatency.c F: block/blk-throttle.c +F: block/bpf_blkcg.c F: include/linux/blk-cgroup.h =20 CONTROL GROUP - CPUSET diff --git a/block/Makefile b/block/Makefile index e7bd320e3d69..572e49988c8e 100644 --- a/block/Makefile +++ b/block/Makefile @@ -17,6 +17,9 @@ obj-$(CONFIG_BLK_ERROR_INJECTION) +=3D error-injection.o obj-$(CONFIG_BLK_DEV_BSG_COMMON) +=3D bsg.o obj-$(CONFIG_BLK_DEV_BSGLIB) +=3D bsg-lib.o obj-$(CONFIG_BLK_CGROUP) +=3D blk-cgroup.o +ifdef CONFIG_BPF_SYSCALL +obj-$(CONFIG_BLK_CGROUP) +=3D bpf_blkcg.o +endif obj-$(CONFIG_BLK_CGROUP_RWSTAT) +=3D blk-cgroup-rwstat.o obj-$(CONFIG_BLK_CGROUP_FC_APPID) +=3D blk-cgroup-fc-appid.o obj-$(CONFIG_BLK_DEV_THROTTLING) +=3D blk-throttle.o diff --git a/block/bpf_blkcg.c b/block/bpf_blkcg.c new file mode 100644 index 000000000000..d8ab8006bc57 --- /dev/null +++ b/block/bpf_blkcg.c @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Block I/O Controller-related BPF kfuncs and auxiliary code + */ + +#include "blk-cgroup.h" + +#include +#include +#include + +__bpf_kfunc_start_defs(); + +/** + * bpf_css_to_blkcg - Cast an io controller css to its block cgroup + * @css: io controller css + * + * Must be called under RCU. + * + * Return: The block cgroup, or NULL if @css belongs to another controller. + */ +__bpf_kfunc struct blkcg * +bpf_css_to_blkcg(struct cgroup_subsys_state *css) +{ + if (unlikely(css->ss !=3D &io_cgrp_subsys)) + return NULL; + + return css_to_blkcg(css); +} + +struct bpf_iter_blkg { + __u64 __opaque[2]; +} __aligned(8); + +struct bpf_iter_blkg_kern { + struct blkcg *blkcg; + struct blkcg_gq *pos; +} __aligned(8); + +/** + * bpf_iter_blkg_new - Start iterating a block cgroup's per-device blkgs + * @it: iterator to initialize + * @blkcg: block cgroup to iterate + * + * Each blkg holds one device's io.stat counters. Offline blkgs are skippe= d. + * A blkg without a disk can be returned. Root blkgs do not contain the + * system-wide statistics shown by root io.stat. Must run under RCU. + * + * Return: 0 on success. + */ +__bpf_kfunc int bpf_iter_blkg_new(struct bpf_iter_blkg *it, + struct blkcg *blkcg) +{ + struct bpf_iter_blkg_kern *kit =3D (void *)it; + + BUILD_BUG_ON(sizeof(struct bpf_iter_blkg_kern) > sizeof(struct bpf_iter_b= lkg)); + BUILD_BUG_ON(__alignof__(struct bpf_iter_blkg_kern) !=3D + __alignof__(struct bpf_iter_blkg)); + + kit->pos =3D NULL; + kit->blkcg =3D blkcg; + return 0; +} + +/** + * bpf_iter_blkg_next - Return the next online blkg of the iterated block = cgroup + * @it: iterator + * + * Return: the next online blkg, or NULL when the walk is done. + */ +__bpf_kfunc struct blkcg_gq *bpf_iter_blkg_next(struct bpf_iter_blkg *it) +{ + struct bpf_iter_blkg_kern *kit =3D (void *)it; + struct blkcg_gq *blkg =3D kit->pos; + struct hlist_node *node; + + if (!kit->blkcg) + return NULL; + + if (!blkg) + node =3D rcu_dereference(hlist_first_rcu(&kit->blkcg->blkg_list)); + else + node =3D rcu_dereference(hlist_next_rcu(&blkg->blkcg_node)); + + /* Skip offline blkgs, matching io.stat. */ + while (node) { + blkg =3D hlist_entry(node, struct blkcg_gq, blkcg_node); + /* A race only changes whether this blkg is returned. */ + if (data_race(blkg->online)) { + kit->pos =3D blkg; + return blkg; + } + node =3D rcu_dereference(hlist_next_rcu(&blkg->blkcg_node)); + } + + /* The iterator must keep returning NULL after completion. */ + kit->pos =3D NULL; + kit->blkcg =3D NULL; + return NULL; +} + +/** + * bpf_iter_blkg_destroy - Tear down a blkg iterator + * @it: iterator + */ +__bpf_kfunc void bpf_iter_blkg_destroy(struct bpf_iter_blkg *it) +{ +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(bpf_blkcg_kfuncs) +BTF_ID_FLAGS(func, bpf_css_to_blkcg, + KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL) + +BTF_ID_FLAGS(func, bpf_iter_blkg_new, + KF_ITER_NEW | KF_RCU | KF_RCU_PROTECTED) +BTF_ID_FLAGS(func, bpf_iter_blkg_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_blkg_destroy, KF_ITER_DESTROY) +BTF_KFUNCS_END(bpf_blkcg_kfuncs) + +static const struct btf_kfunc_id_set bpf_blkcg_kfunc_set =3D { + .owner =3D THIS_MODULE, + .set =3D &bpf_blkcg_kfuncs, +}; + +static int __init bpf_blkcg_init(void) +{ + int err; + + err =3D register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, + &bpf_blkcg_kfunc_set); + if (err) + pr_warn("error while registering bpf blkcg kfuncs: %d\n", err); + + return err; +} +late_initcall(bpf_blkcg_init); diff --git a/kernel/cgroup/bpf_cgroup.c b/kernel/cgroup/bpf_cgroup.c index cd28c838dc7b..e253633e8278 100644 --- a/kernel/cgroup/bpf_cgroup.c +++ b/kernel/cgroup/bpf_cgroup.c @@ -8,12 +8,50 @@ #include #include #include +#include =20 +#ifdef CONFIG_CGROUP_SCHED #include "../sched/sched.h" +#endif =20 -#ifdef CONFIG_CGROUP_SCHED __bpf_kfunc_start_defs(); =20 +/** + * bpf_cgroup_css - Get a reference to one controller's css + * @cgrp: cgroup to look in + * @ssid: controller ID + * + * The returned css must be released with bpf_css_release(). + * + * Return: The referenced css, or NULL. + */ +__bpf_kfunc struct cgroup_subsys_state * +bpf_cgroup_css(struct cgroup *cgrp, int ssid) +{ + struct cgroup_subsys_state *css; + + if (unlikely(ssid < 0 || ssid >=3D CGROUP_SUBSYS_COUNT)) + return NULL; + + rcu_read_lock(); + css =3D rcu_dereference(cgrp->subsys[ssid]); + if (css && !css_tryget(css)) + css =3D NULL; + rcu_read_unlock(); + + return css; +} + +/** + * bpf_css_release - Release a css reference + * @css: css to release + */ +__bpf_kfunc void bpf_css_release(struct cgroup_subsys_state *css) +{ + css_put(css); +} + +#ifdef CONFIG_CGROUP_SCHED /** * bpf_css_to_task_group - Cast a CPU controller css to its task group * @css: CPU controller css @@ -30,29 +68,33 @@ bpf_css_to_task_group(struct cgroup_subsys_state *css) =20 return container_of(css, struct task_group, css); } +#endif /* CONFIG_CGROUP_SCHED */ =20 __bpf_kfunc_end_defs(); =20 -BTF_KFUNCS_START(bpf_cpu_cgroup_kfunc_ids) +BTF_KFUNCS_START(bpf_cgroup_kfunc_ids) +BTF_ID_FLAGS(func, bpf_cgroup_css, KF_ACQUIRE | KF_RCU | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_css_release, KF_RELEASE) +#ifdef CONFIG_CGROUP_SCHED BTF_ID_FLAGS(func, bpf_css_to_task_group, KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL) -BTF_KFUNCS_END(bpf_cpu_cgroup_kfunc_ids) +#endif +BTF_KFUNCS_END(bpf_cgroup_kfunc_ids) =20 -static const struct btf_kfunc_id_set bpf_cpu_cgroup_kfunc_set =3D { +static const struct btf_kfunc_id_set bpf_cgroup_kfunc_set =3D { .owner =3D THIS_MODULE, - .set =3D &bpf_cpu_cgroup_kfunc_ids, + .set =3D &bpf_cgroup_kfunc_ids, }; =20 -static int __init bpf_cpu_cgroup_kfunc_init(void) +static int __init bpf_cgroup_kfunc_init(void) { int err; =20 err =3D register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, - &bpf_cpu_cgroup_kfunc_set); + &bpf_cgroup_kfunc_set); if (err) - pr_warn("error while registering cpu cgroup kfuncs: %d\n", err); + pr_warn("error while registering cgroup kfuncs: %d\n", err); =20 return err; } -late_initcall(bpf_cpu_cgroup_kfunc_init); -#endif /* CONFIG_CGROUP_SCHED */ +late_initcall(bpf_cgroup_kfunc_init); --=20 2.53.0-Meta From nobody Mon Sep 28 14:46:59 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D628486E78 for ; Thu, 20 Aug 2026 21:18:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260697; cv=none; b=bmgoh9dSoUVZnm4jNniwRFuauqL05WgH7aAiCvWA1RrO1rRudJRXN9dNVu9duef/PLdhjmtPtEK7OLl7/YvKlgG0zcXlrKvNyY2+qsB5bfLbc663h7yuarUT20aU4WwwlADgLwP6aXZV2/vqGczgDaOMwArCCYhEQULyw9Jq8oI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787260697; c=relaxed/simple; bh=KGpOGxE5aDGI+YJk6SVQmRM6Iyv3svPXmm0zCZ8M3JM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RQTFbNnhiv1QTjm0+DV+mhEbgnjYOr73zZ+Znq5cxVYTOGT8cfz50eRVNIFf02WRJkZymMA0mP/Ux3ZPzeIYr5BUcf6v55AuTfVaH5UfG3PFTmBcETN4QSd9t6hqVKlvtUSazXI8gYAAdFOEGNRj7yb5L50P5zih36/GfQtNbNo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CvGTBne3; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CvGTBne3" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-3811f512167so304704a91.3 for ; Thu, 20 Aug 2026 14:18:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787260688; x=1787865488; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=pTVI5fQqcRejLtqxc3simDI2MaSc8wcnSCEjipsJOUc=; b=CvGTBne3x3dy5StvYbKotfPM5CN0GGYIGCQPgCOQZ1TcPV6eoK2rUX9qFwEDyPrF9w NB9eC9mny7lv7Sis4mQmIIg81RhW5W5mpHt1/dom96+hdQWxR4uijPnIJitTtVSeSK3F Pp3Hk6DysR+0IPIY4mrMrC4hoFowXtY0PJMPQ6zBNpoXtCA/Hd613mQ7UObIlnz4urds GEky3vPDsd9ZNLzEuK2u11C4MkixiQA6f1afkOodNilaPQ3ePYtYm/u64G2PXKPYNeyi uuFLvwpw5n3EK5y9XVs+G9T+8ujXlXG4Aheqh42ClzAt8rz1v9sRtuCUbymYk/ZmuoDO gHVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787260688; x=1787865488; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=pTVI5fQqcRejLtqxc3simDI2MaSc8wcnSCEjipsJOUc=; b=fnwK/GBbvH7pQTMN0qnHju9F2XL1JxofENGCcaj8yDuiz+nocTK/WRQC2N736PgDkC zF32tssGFsxx0l095k5jYFP9c3Aqkr6MzTErq+/a/SrEp1w320MnOGRAvbCUe0OYT6Oi 9qpHPZafVS+l2OBV4Gn10sJDeAYxOemrka4OhWHONH2vI1SpW+j8QUz4PUUNcVy8aaJ4 1h3GbX2PrPI7OTWbZERul3SgQlH/AM9jAnwFy1ueuWoPJx27KIHTlaoG53ztuknq8qNo lL2F/AlaBVUI2xW5g3GO31gTyRqWA09TdQs7YJjdKYns6jANAzRv4yTu9vkxvxFXqyuD AeTw== X-Forwarded-Encrypted: i=1; AHgh+RqIengJZmouaaPyC1Itn4CfWabdFeoCOuZawFjAwzI9B+56cuZH1iHVHRddc8AEB8DJBiHShs17k4oJejU=@vger.kernel.org X-Gm-Message-State: AFuF++nq8Ew5lwLe578MJaW/MHENqgMrxyZZifzQTtE2psxc/h5VCL6w Rl8kp89atQ94Fh4wj057CuKHvcz59Lib+R8ewtz/YEBoGz/ADgHl0jUm X-Gm-Gg: AR+sD116tw/Cq0OTIk+anfTH4RRiQ/5aZmqbqkv3M2ToBPqGmKDectsl3RP1TZAz52c IBCJlKIG3PgOCNMuNXPFUmZzgVvSTmednCslNs2l2Q5MDMqHaiZryNdikVTmN3EulxOq6wyXb/E eIVAETavZybVHymCw0Gqh9gG5w1Ysiym4hiYLGwPzl/FZyq2b5ep4GUlL9QjtuaYNLLiCtWSo1S 6CHBsrjvrorgEhC+ExcWTGB9/yWiy1XoTqWjdDiGkLl0fu5WvK+WxPsdKyp9ZTvkr0AeMXL7YlP t9FH0h2TG2EMWE9m26AVp7XfG0Ex1+K4oWupubG6hIBshjholcPDbYlJ7gu/4xEmriThO2ZL0mk Ls5aa6eq65W5lRn/tHoZE33HIe5eC/c4EBKrMRIMrohC+mdLRtAdF80exFzyqvKS1E8p9i7HQ/f WehDmTChjIdPvQx8XZ2Tut0gy3zansYzo3smZvfVVj+qW1w+XCU01eNkw= X-Received: by 2002:a17:90b:1c90:b0:393:19a3:4f1 with SMTP id 98e67ed59e1d1-395c351d788mr2396492a91.6.1787260687664; Thu, 20 Aug 2026 14:18:07 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:40::]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-1416ad53874sm30447128c88.4.2026.08.20.14.18.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 14:18:07 -0700 (PDT) From: Ziyang Men To: kernel-team@meta.com, Jens Axboe , Tejun Heo , Josef Bacik , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Shuah Khan Cc: Ingo Molnar , Peter Zijlstra , Vincent Guittot , Ben Segall , Dietmar Eggemann , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Roman Gushchin , Shakeel Butt , JP Kobryn , Mykola Lysenko , Ziyang Men , linux-block@vger.kernel.org, bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 4/4] selftests/bpf: add test for blkcg io.stat BPF kfuncs Date: Thu, 20 Aug 2026 14:17:58 -0700 Message-ID: <20260820211758.3393984-5-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260820211758.3393984-1-ziyang.meme@gmail.com> References: <20260820211758.3393984-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add cgroup_iter_io to test the blkcg io.stat BPF kfuncs. The BPF program acquires the I/O css, flushes its statistics, converts it to a typed blkcg under RCU, walks the blkgs, and releases the css reference. The test performs O_DIRECT I/O on a private loop device. It checks the device ID and counters against io.stat. Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- tools/testing/selftests/bpf/cgroup_iter_io.h | 17 ++ tools/testing/selftests/bpf/config | 1 + .../selftests/bpf/prog_tests/cgroup_iter_io.c | 254 ++++++++++++++++++ .../selftests/bpf/progs/cgroup_iter_io.c | 74 +++++ 4 files changed, 346 insertions(+) create mode 100644 tools/testing/selftests/bpf/cgroup_iter_io.h create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_io.c diff --git a/tools/testing/selftests/bpf/cgroup_iter_io.h b/tools/testing/s= elftests/bpf/cgroup_iter_io.h new file mode 100644 index 000000000000..f4bbaaccdf71 --- /dev/null +++ b/tools/testing/selftests/bpf/cgroup_iter_io.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#ifndef __CGROUP_ITER_IO_H +#define __CGROUP_ITER_IO_H + +struct io_query { + /* one device's io.stat counters */ + __u64 rbytes; + __u64 wbytes; + __u64 rios; + __u64 wios; + __u64 dbytes; + __u64 dios; + __u64 dev; /* dev_t of the device the counters belong to */ +}; + +#endif /* __CGROUP_ITER_IO_H */ diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/b= pf/config index 482b40dde2f9..0e10d625de6e 100644 --- a/tools/testing/selftests/bpf/config +++ b/tools/testing/selftests/bpf/config @@ -1,3 +1,4 @@ +CONFIG_BLK_CGROUP=3Dy CONFIG_BLK_DEV_LOOP=3Dy CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=3Dy CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=3D1 diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c b/tool= s/testing/selftests/bpf/prog_tests/cgroup_iter_io.c new file mode 100644 index 000000000000..2267780912d8 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c @@ -0,0 +1,254 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "cgroup_helpers.h" +#include "cgroup_iter_io.h" +#include "cgroup_iter_io.skel.h" + +#define IO_SIZE (4 * 1024 * 1024) + +static int read_stats(struct bpf_link *link) +{ + int fd, ret =3D 0; + ssize_t bytes; + + fd =3D bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_OK_FD(fd, "bpf_iter_create")) + return 1; + + /* Results land in skel->data_query; the read itself returns no data. */ + bytes =3D read(fd, NULL, 0); + if (!ASSERT_EQ(bytes, 0, "read fd")) + ret =3D 1; + + close(fd); + return ret; +} + +/* Set up a loop device for cgroup-charged I/O. */ +static int loop_setup(char *loop_path, size_t sz, int *ctl_fd, int *loop_f= d, + int *back_fd) +{ + char back_path[] =3D "/tmp/cgroup_iter_io.XXXXXX"; + int nr; + + *ctl_fd =3D *loop_fd =3D *back_fd =3D -1; + + *ctl_fd =3D open("/dev/loop-control", O_RDWR | O_CLOEXEC); + if (*ctl_fd < 0) + return -1; + + nr =3D ioctl(*ctl_fd, LOOP_CTL_GET_FREE); + if (nr < 0) + goto err; + snprintf(loop_path, sz, "/dev/loop%d", nr); + + *back_fd =3D mkstemp(back_path); + if (*back_fd < 0) + goto err; + unlink(back_path); + if (ftruncate(*back_fd, (off_t)IO_SIZE * 4)) + goto err; + + *loop_fd =3D open(loop_path, O_RDWR | O_CLOEXEC); + if (*loop_fd < 0) + goto err; + if (ioctl(*loop_fd, LOOP_SET_FD, *back_fd)) + goto err; + + return 0; +err: + if (*loop_fd >=3D 0) + close(*loop_fd); + if (*back_fd >=3D 0) + close(*back_fd); + close(*ctl_fd); + *ctl_fd =3D *loop_fd =3D *back_fd =3D -1; + return -1; +} + +static void loop_teardown(const char *loop_path, int ctl_fd, int loop_fd, + int back_fd) +{ + int nr =3D -1; + + if (loop_fd >=3D 0) { + ioctl(loop_fd, LOOP_CLR_FD, 0); + close(loop_fd); + } + if (back_fd >=3D 0) + close(back_fd); + if (ctl_fd >=3D 0) { + if (sscanf(loop_path, "/dev/loop%d", &nr) =3D=3D 1 && nr >=3D 0) + ioctl(ctl_fd, LOOP_CTL_REMOVE, nr); + close(ctl_fd); + } +} + +/* O_DIRECT keeps I/O charged to the current cgroup. */ +static int do_direct_io(const char *loop_path) +{ + void *buf; + int fd, ret =3D -1; + + fd =3D open(loop_path, O_RDWR | O_DIRECT | O_CLOEXEC); + if (fd < 0) + return -1; + if (posix_memalign(&buf, 4096, IO_SIZE)) + goto out_fd; + memset(buf, 0xab, IO_SIZE); + + if (pwrite(fd, buf, IO_SIZE, 0) !=3D IO_SIZE) + goto out_buf; + fsync(fd); + if (pread(fd, buf, IO_SIZE, 0) !=3D IO_SIZE) + goto out_buf; + ret =3D 0; +out_buf: + free(buf); +out_fd: + close(fd); + return ret; +} + +/* Read @dev's io.stat counters. @dev uses kernel dev_t encoding. */ +static int parse_io_stat(int cgroup_fd, __u64 dev, struct io_query *out) +{ + unsigned int want_maj =3D dev >> 20, want_min =3D dev & ((1U << 20) - 1); + char buf[4096], *line, *saveptr; + int fd, n, ret =3D -1; + + fd =3D openat(cgroup_fd, "io.stat", O_RDONLY); + if (fd < 0) + return -1; + n =3D read(fd, buf, sizeof(buf) - 1); + close(fd); + if (n <=3D 0) + return -1; + buf[n] =3D '\0'; + + for (line =3D strtok_r(buf, "\n", &saveptr); line; + line =3D strtok_r(NULL, "\n", &saveptr)) { + unsigned long long rb =3D 0, wb =3D 0, ri =3D 0, wi =3D 0, db =3D 0, di = =3D 0; + unsigned int maj, min; + + /* Only the device id is required; missing counters stay zero. */ + if (sscanf(line, + "%u:%u rbytes=3D%llu wbytes=3D%llu rios=3D%llu wios=3D%llu dbytes=3D= %llu dios=3D%llu", + &maj, &min, &rb, &wb, &ri, &wi, &db, &di) < 2) + continue; + if (maj !=3D want_maj || min !=3D want_min) + continue; + + out->rbytes =3D rb; + out->wbytes =3D wb; + out->rios =3D ri; + out->wios =3D wi; + out->dbytes =3D db; + out->dios =3D di; + ret =3D 0; + break; + } + return ret; +} + +void test_cgroup_iter_io(void) +{ + char *cgroup_rel_path =3D "/cgroup_iter_io_test"; + int ctl_fd =3D -1, loop_fd =3D -1, back_fd =3D -1; + struct cgroup_iter_io *skel =3D NULL; + struct bpf_link *link =3D NULL; + char loop_path[64]; + struct io_query *q; + int cgroup_fd; + + cgroup_fd =3D cgroup_setup_and_join(cgroup_rel_path); + if (!ASSERT_OK_FD(cgroup_fd, "cgroup_setup_and_join")) + return; + + if (loop_setup(loop_path, sizeof(loop_path), &ctl_fd, &loop_fd, &back_fd)= ) { + test__skip(); /* needs root + CONFIG_BLK_DEV_LOOP */ + goto cleanup_cgroup_fd; + } + + skel =3D cgroup_iter_io__open_and_load(); + if (!ASSERT_OK_PTR(skel, "cgroup_iter_io__open_and_load")) + goto cleanup_loop; + + /* Convert glibc st_rdev to kernel dev_t format. */ + { + struct stat lst; + + if (!ASSERT_OK(fstat(loop_fd, &lst), "fstat loop")) + goto cleanup_skel; + skel->data_query->target_dev =3D + ((__u64)major(lst.st_rdev) << 20) | minor(lst.st_rdev); + } + + DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts); + union bpf_iter_link_info linfo =3D { + .cgroup.cgroup_fd =3D cgroup_fd, + .cgroup.order =3D BPF_CGROUP_ITER_SELF_ONLY, + }; + opts.link_info =3D &linfo; + opts.link_info_len =3D sizeof(linfo); + + link =3D bpf_program__attach_iter(skel->progs.cgroup_io_query, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto cleanup_skel; + + /* This process is in the test cgroup, so the loop I/O is charged here. */ + if (!ASSERT_OK(do_direct_io(loop_path), "do_direct_io")) + goto cleanup_link; + + if (!ASSERT_OK(read_stats(link), "read stats")) + goto cleanup_link; + + q =3D &skel->data_query->io_query; + if (test__start_subtest("cgroup_iter_io__write")) { + ASSERT_GT(q->wbytes, 0, "wbytes"); + ASSERT_GT(q->wios, 0, "wios"); + } + if (test__start_subtest("cgroup_iter_io__read")) { + ASSERT_GT(q->rbytes, 0, "rbytes"); + ASSERT_GT(q->rios, 0, "rios"); + } + if (test__start_subtest("cgroup_iter_io__dev")) + ASSERT_GT(q->dev, 0, "dev"); + + /* Compare with io.stat without I/O between the reads. */ + if (test__start_subtest("cgroup_iter_io__match")) { + struct io_query filev =3D {}; + + if (ASSERT_OK(read_stats(link), "read stats") && + ASSERT_OK(parse_io_stat(cgroup_fd, q->dev, &filev), + "parse io.stat")) { + ASSERT_EQ(q->rbytes, filev.rbytes, "rbytes"); + ASSERT_EQ(q->wbytes, filev.wbytes, "wbytes"); + ASSERT_EQ(q->rios, filev.rios, "rios"); + ASSERT_EQ(q->wios, filev.wios, "wios"); + ASSERT_EQ(q->dbytes, filev.dbytes, "dbytes"); + ASSERT_EQ(q->dios, filev.dios, "dios"); + } + } + +cleanup_link: + bpf_link__destroy(link); +cleanup_skel: + cgroup_iter_io__destroy(skel); +cleanup_loop: + loop_teardown(loop_path, ctl_fd, loop_fd, back_fd); +cleanup_cgroup_fd: + close(cgroup_fd); + cleanup_cgroup_environment(); +} diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_io.c b/tools/tes= ting/selftests/bpf/progs/cgroup_iter_io.c new file mode 100644 index 000000000000..0e9c9cd6e33a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include "bpf_experimental.h" +#include "cgroup_iter_io.h" + +char _license[] SEC("license") =3D "GPL"; + +struct io_query io_query SEC(".data.query"); + +/* Device selected by userspace in kernel dev_t format. */ +__u64 target_dev SEC(".data.query"); + +/* Keep inline: RCU and open-coded iterators cannot cross a BPF call. */ +static __always_inline int read_target_dev(struct cgroup *cgrp, + struct io_query *out) +{ + struct cgroup_subsys_state *css; + struct blkcg *blkcg; + struct blkcg_gq *pos; + __u64 dev; + int ssid; + + ssid =3D bpf_core_enum_value(enum cgroup_subsys_id, io_cgrp_id); + css =3D bpf_cgroup_css(cgrp, ssid); + if (!css) + return 0; + + css_rstat_flush(css); + + bpf_rcu_read_lock(); + + blkcg =3D bpf_css_to_blkcg(css); + if (!blkcg) { + bpf_rcu_read_unlock(); + bpf_css_release(css); + return 0; + } + + bpf_for_each(blkg, pos, blkcg) { + dev =3D BPF_CORE_READ(pos, q, disk, part0, bd_dev); + if (dev !=3D target_dev) + continue; + + out->dev =3D dev; + out->rbytes =3D BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_READ]); + out->wbytes =3D BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_WRITE]); + out->rios =3D BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_READ]); + out->wios =3D BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_WRITE]); + out->dbytes =3D BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_DISCARD]= ); + out->dios =3D BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_DISCARD]); + break; + } + bpf_rcu_read_unlock(); + bpf_css_release(css); + return 1; +} + +SEC("iter.s/cgroup") +int cgroup_io_query(struct bpf_iter__cgroup *ctx) +{ + struct cgroup *cgrp =3D ctx->cgroup; + + if (!cgrp) + return 1; + + /* Start fresh so a device that is not found stays all-zero. */ + __builtin_memset(&io_query, 0, sizeof(io_query)); + + read_target_dev(cgrp, &io_query); + return 0; +} --=20 2.53.0-Meta