From nobody Mon Sep 28 20:50:57 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE97F202C29 for ; Tue, 18 Aug 2026 00:24:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787012697; cv=none; b=VI25HC53yqpvaSVRhHTMVV2qIKItxnqoHJhOqJaQE7yRONko7Jt+ZyHYqOwLZLO7Tc0iVEi24BiKS6VHD3hhBIvuLUYaGsbfeYI3naBQ4QxXH4WccWfGfJ4r2C/B9iq7Bnm33WFHfsuHmQhoT4aw4/5sh35bFZDElu35MkiPcIU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787012697; c=relaxed/simple; bh=U0cdwRkNU8DDkTZBzpmBkqOIyFGgDUIy3LahHB12YmI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ipSyOL8tolFObPpS9jCaA8YLiCVthjhrHGStq4VOZ6Z9E9rBmhO5XDxUJwBleNHlQPZ4xW/pe8m9ztMuGysM0zAssvngfVyQx1Rv8a0yxOpmzBRnwQb1KtSSVZNFda3AiFCyRys/7SN2u1Qd9/PFFDXqU0BC32NTkszQCaZn5FU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NvZ8/ci7; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NvZ8/ci7" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-cbe6295f05bso315119a12.1 for ; Mon, 17 Aug 2026 17:24:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787012695; x=1787617495; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Dqe2p//IXwPRRWOgvWYahO9x2ZMS0WvCKoSIYY1NM34=; b=NvZ8/ci77f42Sts/5nli7p1aZJwGPjbZgu2OsgTl6nNAuL6Pyw2e4AgWk4NjM/1yXL GqPR5cLT9qzjK26zo5txgSE5KUyK8E0266a6J0CVQYKEzn/r+hGVqBNHJ05erf6Z9J0R nMaXROf83vhBLsykD+jkwIqWy1193VriXXQVkWgreSbLEFMXb+CmX9mvZPB9WQUNZF1j Z2zPLC0JVxD4UITe3hhiZk9UnbgTQaRtPO82ZLZ0E9HhAWlbHrrGs4VhF25fcDidCbhp UUlT1LgnZEYl0R0W1NlxhsKNx+A014xZ3n1ysQz2l1xYAOTrZUQPWpVS49CwIWCEfgTc LqVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787012695; x=1787617495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Dqe2p//IXwPRRWOgvWYahO9x2ZMS0WvCKoSIYY1NM34=; b=l4jrmj2BtOFR/XzauI+SzfEqze3jSeCgALRimgHj0IYIilfm3ZMarD6zjy1dZeX4z7 rPMoacpBpO16B2cc7mURgt6WJV+fXtmRL/86L4FZzmRtKVEvq9uDsqnYPBD+csfkG0SH ZPwYCdTDcGfTKH3KJ+dvYRg/NLbH/8Am1t2cSycFfnUUtx7A20zeevsQdoQAsR4y6XQ/ +f+CGsh7C7/f5KGo8w0tfRQitSHdJ6XI7yjTszQROrDw24dK+DyOkjLKsXy32psXlDjy kiQ/tJCGxRDFAn73ErCIGEZ7o96GFMNGpqMGbZcrHf/x9MaLo+AqJpvQEqVBayVvy7hm JikQ== X-Forwarded-Encrypted: i=1; AHgh+RpoVrl23mJ8R++OGccGNHb3nOEXMrbZcP/G+WX5D4upm0PkAaMHgCAh7eZcoiZ1QECyaFfAE47jZpViOto=@vger.kernel.org X-Gm-Message-State: AOJu0Yx0D89oTi/0Uz0Grt8IiCYO3qwpWeDi9Wwb/NwJpZz7ClgN2y1G VUihOUm26sCRAwwpjxs6Qga6EtcJAXqFEXlZ09b+DnMsDnjmJQ6QEpL1 X-Gm-Gg: AR+sD11LStH0D6TaB4WHS2R+ql5w8VJb/Z+Iv0Dzx/K+WaB3oe87Yw5dvEhjppwrZB1 puItkSUBK1tdK8pIWdunPXPhjPBQ9TBRNSaEpLCSTZgAKE9cPlQ0iF8CnUKZWXa4rWXCvF0RAJg KevGviLs0PvwcRudnF9Zi/XtnG4fqCpXNdbIF1s+tRP1O3ce5WzeIEbl64Zwns9YBT2y+c2DwO3 SdXuZxYyflNCA0Nj4fD3hnBWuuEcgp8tHuj5Qnjyo5Cbn8Y+h8l7oN4VAbwARuCfzYTeel8jIcu BRZrpLPGZhrHyRVC4uaqsGMz2D5noewV268akaru12k5yDfdUbu3aNNMsM5ogTdvWu8x1hIcb29 G9AY7uOd4+wak3l/3Ej4CQwRKkiIQBMS3i8LJ29rUK1Immfth44oeXPjPD76m/HFb1OyuNv2faA 82A0XmYR4cmOYOa2NWYNFr5EMfqkEqwoK7Qu8tl7Tj4DHf2d0nWGr3T8o= X-Received: by 2002:a17:90b:17ce:b0:38e:7e9b:5fbc with SMTP id 98e67ed59e1d1-3955f0e4554mr2145187a91.7.1787012695192; Mon, 17 Aug 2026 17:24:55 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:6b::]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-32678dcedf4sm10262056eec.5.2026.08.17.17.24.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 17:24:54 -0700 (PDT) From: Ziyang Men To: Tejun Heo , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Shuah Khan , kernel-team@meta.com Cc: Ingo Molnar , Peter Zijlstra , Vincent Guittot , Ben Segall , Dietmar Eggemann , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Roman Gushchin , Shakeel Butt , JP Kobryn , Ziyang Men , bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/2] cgroup: add BPF kfuncs to read a cpu cgroup's stats Date: Mon, 17 Aug 2026 17:24:49 -0700 Message-ID: <20260818002450.3071325-2-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260818002450.3071325-1-ziyang.meme@gmail.com> References: <20260818002450.3071325-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Collecting cgroup statistics is expensive: the existing method is to open and parse a cgroup file. memcg already has an efficient alternative through BPF; this series extends that idea to cpu. Expose the CPU controller's per-cgroup statistics to BPF, following the memory controller kfuncs in mm/bpf_memcontrol.c. Design: - Add bpf_css_flush_rstat() and bpf_cgroup_base_stat() to cgroup rstat. The second kfunc returns the raw cgroup_base_stat after the same cputime adjustment used by cpu.stat. - Leave reading the CFS bandwidth counters to the BPF program. They are plain fields of tg->cfs_bandwidth, so they need no kernel code. - Add bpf_css_to_task_group(), which returns a checked RCU pointer of the task_group*. This is usefule to compute the throttled time in bpf side since the bpf_per_cpu_ptr() requires a verifier-known struct task_group * and a pointer carrying the MEM_PERCPU property.=20 This use the convention that task_group embeds its css at offset zero, so no scheduler helper or scheduler source change is needed. Suggested-by: Shakeel Butt Suggested-by: Tejun Heo Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- kernel/cgroup/Makefile | 2 ++ kernel/cgroup/bpf_cpu.c | 61 +++++++++++++++++++++++++++++++++++++++++ kernel/cgroup/rstat.c | 59 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 120 insertions(+), 2 deletions(-) create mode 100644 kernel/cgroup/bpf_cpu.c diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile index ede31601a363..0ba59b7eef48 100644 --- a/kernel/cgroup/Makefile +++ b/kernel/cgroup/Makefile @@ -1,6 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 obj-y :=3D cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o =20 +obj-$(CONFIG_BPF_SYSCALL) +=3D bpf_cpu.o + obj-$(CONFIG_CGROUP_FREEZER) +=3D legacy_freezer.o obj-$(CONFIG_CGROUP_PIDS) +=3D pids.o obj-$(CONFIG_CGROUP_RDMA) +=3D rdma.o diff --git a/kernel/cgroup/bpf_cpu.c b/kernel/cgroup/bpf_cpu.c new file mode 100644 index 000000000000..ac165d0b79ef --- /dev/null +++ b/kernel/cgroup/bpf_cpu.c @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * CPU controller BPF kfuncs + * + * Author: Ziyang Men + */ + +#include +#include +#include + +#ifdef CONFIG_CGROUP_SCHED +struct task_group; + +__bpf_kfunc_start_defs(); + +/** + * bpf_css_to_task_group - Cast a CPU controller css to its task group + * @css: CPU controller css + * + * Must be called under RCU. + * A C cast does not give the verifier a task_group pointer. This kfunc + * preserves the task_group and per-CPU types needed to read cfs_rq. + * + * Return: The task group, or NULL if @css belongs to another controller. + */ +__bpf_kfunc struct task_group * +bpf_css_to_task_group(struct cgroup_subsys_state *css) +{ + if (css->ss !=3D &cpu_cgrp_subsys) + return NULL; + + /* task_group embeds css at offset zero. */ + return (struct task_group *)css; +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(bpf_cpu_cgroup_kfunc_ids) +BTF_ID_FLAGS(func, bpf_css_to_task_group, + KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL) +BTF_KFUNCS_END(bpf_cpu_cgroup_kfunc_ids) + +static const struct btf_kfunc_id_set bpf_cpu_cgroup_kfunc_set =3D { + .owner =3D THIS_MODULE, + .set =3D &bpf_cpu_cgroup_kfunc_ids, +}; + +static int __init bpf_cpu_cgroup_kfunc_init(void) +{ + int err; + + err =3D register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, + &bpf_cpu_cgroup_kfunc_set); + if (err) + pr_warn("error while registering cpu cgroup kfuncs: %d\n", err); + + return err; +} +late_initcall(bpf_cpu_cgroup_kfunc_init); +#endif /* CONFIG_CGROUP_SCHED */ diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index de816a43db9f..46c322c4858b 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -752,6 +752,54 @@ void cgroup_base_stat_cputime_show(struct seq_file *se= q) cgroup_force_idle_show(seq, &bstat); } =20 +#ifdef CONFIG_BPF_SYSCALL + +__bpf_kfunc_start_defs(); + +/** + * bpf_css_flush_rstat - Flush a cgroup subsystem's rstat data + * @css: cgroup subsystem state to flush + */ +__bpf_kfunc void bpf_css_flush_rstat(struct cgroup_subsys_state *css) +{ + css_rstat_flush(css); +} + +/** + * bpf_cgroup_base_stat - Read a cgroup's base statistics + * @cgrp: cgroup to read from + * @out: zero-initialized output in nanoseconds + * + * CPU time is adjusted as for cpu.stat. + */ +__bpf_kfunc void bpf_cgroup_base_stat(struct cgroup *cgrp, + struct cgroup_base_stat *out) +{ + if (cgroup_parent(cgrp)) { + __css_rstat_lock(&cgrp->self, -1); + *out =3D cgrp->bstat; + cputime_adjust(&cgrp->bstat.cputime, &cgrp->prev_cputime, + &out->cputime.utime, &out->cputime.stime); + __css_rstat_unlock(&cgrp->self, -1); + } else { + root_cgroup_cputime(out); + } +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(bpf_rstat_common_kfunc_ids) +BTF_ID_FLAGS(func, bpf_css_flush_rstat, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_cgroup_base_stat, KF_SLEEPABLE) +BTF_KFUNCS_END(bpf_rstat_common_kfunc_ids) + +static const struct btf_kfunc_id_set bpf_rstat_common_kfunc_set =3D { + .owner =3D THIS_MODULE, + .set =3D &bpf_rstat_common_kfunc_ids, +}; + +#endif /* CONFIG_BPF_SYSCALL */ + /* Add bpf kfuncs for css_rstat_updated() and css_rstat_flush() */ BTF_KFUNCS_START(bpf_rstat_kfunc_ids) BTF_ID_FLAGS(func, css_rstat_updated) @@ -765,7 +813,14 @@ static const struct btf_kfunc_id_set bpf_rstat_kfunc_s= et =3D { =20 static int __init bpf_rstat_kfunc_init(void) { - return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, - &bpf_rstat_kfunc_set); + int ret; + + ret =3D register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, + &bpf_rstat_kfunc_set); +#ifdef CONFIG_BPF_SYSCALL + ret =3D ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, + &bpf_rstat_common_kfunc_set); +#endif + return ret; } late_initcall(bpf_rstat_kfunc_init); --=20 2.53.0-Meta From nobody Mon Sep 28 20:50:57 2026 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D71451E492D for ; Tue, 18 Aug 2026 00:24:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787012702; cv=none; b=a3anzQa804PD7EZy8VO6aKnJBOrlo+rsSUGN2fy/zLDOHeSKsO5quUJJIK0IUJ14hWPU0t1tgpdoIM0hOJZEMUK8lNg9Be1Oc/V6EG/dkuyftEu78w+kdDfynMyRlcwp68FL1jamppaSLwh3lGAAh507U4hrnA/05o3iglcsEhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787012702; c=relaxed/simple; bh=unazk+OrwPgf7mLcnksv0A15zNlFkIg7Ewpx4cjJO+0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H6ZBCMqTCzyub3+MWwULDoAOxT3tgVNxx+qy4M8FcvcwvzgDj6z48eNFWlJgocjMZPs25AquYaUiyY/u4AuQ2eWuQpCO6YiXKumj/lWVyZnpz0FUQAaLzHcVWA8AsVtdh92/IrbpGK2pzuts2HHDaeuEjrueaWLZYmAeKQz207M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ifj5/ECY; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ifj5/ECY" Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-c9ef3e1337fso3216068a12.2 for ; Mon, 17 Aug 2026 17:24:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787012697; x=1787617497; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=QTdgOc/sEwYC2pU7oMn2+zaXVXAdEvolPRa+1xSal4o=; b=Ifj5/ECYAyEyxp7WxHB3PTbKm/NP3V1enyUuR4FVqkFp1jSiPpv7X7DpOF7iJ3cNt0 0jSb3Xn0zTfqQJPDD7JzgPBnsmijpD4oFeiR1WB40W6RR7RdxWVHDpLVTlHfBbBLweKE 4vkjKu8Z9Xqsk6PJLvya0lXBsrtKIP4V+Zs7rBDAICXOkgukAPGyN/2jaUZE/JdbZ/Ms cZEBK9tru8xZeFJ5AH8McTKH4uCIzgDXgufyu4+/wwFbCIW4Z2412ORx7UIYG8tx26yK UT8Zd7H5CnncIJDM3cO3wB+boWyLCLa/3fjUWUHZe+KLVlSQF/F/73DddSUI8MU0Bpyu WV8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787012697; x=1787617497; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=QTdgOc/sEwYC2pU7oMn2+zaXVXAdEvolPRa+1xSal4o=; b=WCMiqREJ2oyZfAAems2FY2jpKJP0XDiMrEgL2Vt1YqYAYJFDK99/VNVrPeIPpf6MKu Vvx5m7e1I3PCulcXPICEtureH1O2+7f6OECR5vNJKE35ZgalQKQIOt2mIqXK1AyRgcIA z48Lr7xFj5EWMxSenVDGg7LjyV5DT3IY96GRb5UeJQ98JCPlAvEDsM8ViTZSgIH9/3uG JRoJHcSwD5RpBM/HQ/jduQFY+fxtQVlfmBuqtBbI7eg5db7SNg3CuXMnvZC1KcdDieCA Z5TmlKW/1j7gMGMF5Glol3Y+sdgwI6rDdcJzbLZLvos7ESYexmUPgXdZsbCLX1sRLnS2 1OOA== X-Forwarded-Encrypted: i=1; AHgh+RppKkt1BW1YLS3vVV5B1qTFlIgh+af3pvxHlxDdgF1elmu9JtwIvW3qCr0NgoGXIBQToFzVuiC9RZz0PD8=@vger.kernel.org X-Gm-Message-State: AOJu0Yyj6mJ8o7WBEDeiAyY6IXqz3EG16H0xNc1m/demGgIfPT/lFLmZ 11O6uJw7A9qMPALRneEcT3MbcbvLfgeJXI+ViLpThlnL3cA7kRO424BJ X-Gm-Gg: AR+sD11Sr3/TKmDe6GhKLnJHsMoeCux3mTCSh+z/uDf1sacLLvTAfnK5UKcmAjFl81h U48KxN3UYCNx1VaWo/3Mbs5fW6++7gUEMHSXB1EokDp1twWDQoKTJzRNcBu9ssFuVkoDagzyIiu PX9XKYAcoRTcOOGnSQ8WfruUpYoI6DaI4/FX1ELERgTWbsubfsnrnA3fK3rkM31S47nlKeGl9d8 UpElyxhfdd2RwgzBesOL9mUVzFqGzl2Yc4ivdEkqsxFDW+VGAb6MAvHbiHRvBlSl7wkG02G6cMw Vil2pW1DP/JMmPqQhQngrNr3TOEoSJZ2Jyh/WLUEXCCCspcW50R4bM5VOFOjoUwgOKsRRg+Crrd BBtHB26KFffy6w4PTUxFwls/xVt01EmYi0zL/DJH6CCy0WV+AcHyz56dzl9jhgcM+UHmfszxnSE B3gqgqmB7/Weum56mI0zhk+1ZQjAcM+HMJDCXAQW9QN7I+wsCH8SzRz4k= X-Received: by 2002:a05:6a20:6a27:b0:3c3:8d86:9856 with SMTP id adf61e73a8af0-3ccda9f7871mr4239351637.14.1787012696932; Mon, 17 Aug 2026 17:24:56 -0700 (PDT) Received: from localhost ([2a03:2880:9ff:62::]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-32678dd26a0sm12458707eec.2.2026.08.17.17.24.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 17:24:56 -0700 (PDT) From: Ziyang Men To: Tejun Heo , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Shuah Khan , kernel-team@meta.com Cc: Ingo Molnar , Peter Zijlstra , Vincent Guittot , Ben Segall , Dietmar Eggemann , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Roman Gushchin , Shakeel Butt , JP Kobryn , Ziyang Men , bpf@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/2] selftests/bpf: add cgroup_iter_cpu test for cpu cgroup kfuncs Date: Mon, 17 Aug 2026 17:24:50 -0700 Message-ID: <20260818002450.3071325-3-ziyang.meme@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260818002450.3071325-1-ziyang.meme@gmail.com> References: <20260818002450.3071325-1-ziyang.meme@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add cgroup_iter_cpu, a selftest for the CPU controller BPF kfuncs. The userspace side runs a CPU hog in a test cgroup with cpu.max settled then: - checks the CPU-time and throttling counters are nonzero, - compares whether all values the program read are same as those reading from cgroup file. Enable CONFIG_CGROUP_SCHED, CONFIG_FAIR_GROUP_SCHED and CONFIG_CFS_BANDWIDTH in the selftest config. Tested on VM with v7.2-rc5.=20 Suggested-by: Shakeel Butt Assisted-by: Claude:claude-opus-5 Signed-off-by: Ziyang Men --- tools/testing/selftests/bpf/cgroup_iter_cpu.h | 22 ++ tools/testing/selftests/bpf/config | 3 + .../bpf/prog_tests/cgroup_iter_cpu.c | 259 ++++++++++++++++++ .../selftests/bpf/progs/cgroup_iter_cpu.c | 113 ++++++++ 4 files changed, 397 insertions(+) create mode 100644 tools/testing/selftests/bpf/cgroup_iter_cpu.h create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c diff --git a/tools/testing/selftests/bpf/cgroup_iter_cpu.h b/tools/testing/= selftests/bpf/cgroup_iter_cpu.h new file mode 100644 index 000000000000..74599a5c0e4d --- /dev/null +++ b/tools/testing/selftests/bpf/cgroup_iter_cpu.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#ifndef __CGROUP_ITER_CPU_H +#define __CGROUP_ITER_CPU_H + +struct cpu_query { + /* base cpu time, from cpu.stat */ + __u64 usage_usec; + __u64 user_usec; + __u64 system_usec; + __u64 nice_usec; + __u64 forceidle_usec; + /* CFS bandwidth throttling, from cpu.stat and cpu.stat.local */ + __u64 nr_periods; + __u64 nr_throttled; + __u64 throttled_usec; + __u64 nr_bursts; + __u64 burst_usec; + __u64 throttled_self_usec; +}; + +#endif /* __CGROUP_ITER_CPU_H */ diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/b= pf/config index ea7044f30adc..482b40dde2f9 100644 --- a/tools/testing/selftests/bpf/config +++ b/tools/testing/selftests/bpf/config @@ -11,6 +11,9 @@ CONFIG_BPF_STREAM_PARSER=3Dy CONFIG_BPF_SYSCALL=3Dy # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set CONFIG_CGROUP_BPF=3Dy +CONFIG_CGROUP_SCHED=3Dy +CONFIG_FAIR_GROUP_SCHED=3Dy +CONFIG_CFS_BANDWIDTH=3Dy CONFIG_CRYPTO_HMAC=3Dy CONFIG_CRYPTO_SHA256=3Dy CONFIG_CRYPTO_USER_API=3Dy diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c b/too= ls/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c new file mode 100644 index 000000000000..cd7e92ababfb --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c @@ -0,0 +1,259 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include +#include +#include +#include "cgroup_helpers.h" +#include "cgroup_iter_cpu.h" +#include "cgroup_iter_cpu.skel.h" + +static int read_stats(struct bpf_link *link) +{ + int fd, ret =3D 0; + ssize_t bytes; + + fd =3D bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_OK_FD(fd, "bpf_iter_create")) + return 1; + + bytes =3D read(fd, NULL, 0); + if (!ASSERT_EQ(bytes, 0, "read fd")) + ret =3D 1; + + close(fd); + return ret; +} + +/* Read cgroup file @name into @buf. */ +static int read_cgroup_file(int cgroup_fd, const char *name, char *buf, + size_t size) +{ + ssize_t n; + int fd; + + fd =3D openat(cgroup_fd, name, O_RDONLY); + if (fd < 0) + return -1; + n =3D read(fd, buf, size - 1); + close(fd); + if (n <=3D 0) + return -1; + buf[n] =3D '\0'; + return 0; +} + +/* Parse the "cpu.stat" file into @out. */ +static int parse_cpu_stat(int cgroup_fd, struct cpu_query *out) +{ + char buf[4096], *line, *sp; + unsigned long long v; + + if (read_cgroup_file(cgroup_fd, "cpu.stat", buf, sizeof(buf))) + return -1; + + for (line =3D strtok_r(buf, "\n", &sp); line; + line =3D strtok_r(NULL, "\n", &sp)) { + if (sscanf(line, "usage_usec %llu", &v) =3D=3D 1) + out->usage_usec =3D v; + else if (sscanf(line, "user_usec %llu", &v) =3D=3D 1) + out->user_usec =3D v; + else if (sscanf(line, "system_usec %llu", &v) =3D=3D 1) + out->system_usec =3D v; + else if (sscanf(line, "nice_usec %llu", &v) =3D=3D 1) + out->nice_usec =3D v; + else if (sscanf(line, "core_sched.force_idle_usec %llu", &v) =3D=3D 1) + out->forceidle_usec =3D v; + else if (sscanf(line, "nr_periods %llu", &v) =3D=3D 1) + out->nr_periods =3D v; + else if (sscanf(line, "nr_throttled %llu", &v) =3D=3D 1) + out->nr_throttled =3D v; + else if (sscanf(line, "throttled_usec %llu", &v) =3D=3D 1) + out->throttled_usec =3D v; + else if (sscanf(line, "nr_bursts %llu", &v) =3D=3D 1) + out->nr_bursts =3D v; + else if (sscanf(line, "burst_usec %llu", &v) =3D=3D 1) + out->burst_usec =3D v; + } + return 0; +} + +/* + * Parse the "cpu.stat.local" file into @out. + */ +static int parse_cpu_stat_local(int cgroup_fd, struct cpu_query *out) +{ + unsigned long long v; + char buf[256]; + + if (read_cgroup_file(cgroup_fd, "cpu.stat.local", buf, sizeof(buf))) + return -1; + if (sscanf(buf, "throttled_usec %llu", &v) !=3D 1) + return -1; + out->throttled_self_usec =3D v; + return 0; +} + +/* Read file value the bpf program reads. */ +static int parse_stats(int cgroup_fd, struct cpu_query *out, bool have_bw) +{ + if (parse_cpu_stat(cgroup_fd, out)) + return -1; + if (have_bw && parse_cpu_stat_local(cgroup_fd, out)) + return -1; + return 0; +} + +/* + * Check whether this kernel accounts CFS bandwidth. + */ +static bool cgroup_has_bw_stat(int cgroup_fd) +{ + char buf[4096]; + + if (read_cgroup_file(cgroup_fd, "cpu.stat", buf, sizeof(buf))) + return false; + return strstr(buf, "nr_periods "); +} + +/* Fork a child that spins in the current cgroup, kill it if the test exit= s. */ +static pid_t spawn_cpu_hog(void) +{ + pid_t pid =3D fork(); + + if (pid =3D=3D 0) { + prctl(PR_SET_PDEATHSIG, SIGKILL); + while (1) + ; + } + return pid; +} + +void test_cgroup_iter_cpu(void) +{ + char *cgroup_rel_path =3D "/cgroup_iter_cpu_test"; + struct cgroup_iter_cpu *skel; + struct cpu_query *q; + struct bpf_link *link; + bool wrote_max, have_bw; + int cgroup_fd; + pid_t hog; + + cgroup_fd =3D cgroup_setup_and_join(cgroup_rel_path); + if (!ASSERT_OK_FD(cgroup_fd, "cgroup_setup_and_join")) + return; + + wrote_max =3D !write_cgroup_file(cgroup_rel_path, "cpu.max", "10000 10000= 0"); + + skel =3D cgroup_iter_cpu__open_and_load(); + if (!ASSERT_OK_PTR(skel, "cgroup_iter_cpu__open_and_load")) + goto cleanup_cgroup_fd; + + DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts); + union bpf_iter_link_info linfo =3D { + .cgroup.cgroup_fd =3D cgroup_fd, + .cgroup.order =3D BPF_CGROUP_ITER_SELF_ONLY, + }; + opts.link_info =3D &linfo; + opts.link_info_len =3D sizeof(linfo); + + link =3D bpf_program__attach_iter(skel->progs.cgroup_cpu_query, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto cleanup_skel; + + q =3D &skel->data_query->cpu_query; + + hog =3D spawn_cpu_hog(); + if (!ASSERT_GT(hog, 0, "spawn_cpu_hog")) + goto cleanup_link; + + sleep(1); + + /* Run the bpf program before anything here reads cpu.stat. */ + if (!ASSERT_OK(read_stats(link), "read stats")) + goto cleanup_hog; + + have_bw =3D wrote_max && cgroup_has_bw_stat(cgroup_fd); + + if (test__start_subtest("cgroup_iter_cpu__cputime")) { + ASSERT_GT(q->usage_usec, 0, "usage_usec"); + ASSERT_GT(q->user_usec + q->system_usec, 0, "user+system_usec"); + } + if (test__start_subtest("cgroup_iter_cpu__throttling")) { + if (!have_bw) { + test__skip(); + } else { + ASSERT_GT(q->nr_periods, 0, "nr_periods"); + ASSERT_GT(q->nr_throttled, 0, "nr_throttled"); + ASSERT_GT(q->throttled_usec, 0, "throttled_usec"); + ASSERT_GT(q->throttled_self_usec, 0, "throttled_self_usec"); + } + } + + /* + * cpu.stat cputime grows on every tick a task in the cgroup runs, so + * stop them all before comparing + */ + if (test__start_subtest("cgroup_iter_cpu__match")) { + struct cpu_query filev =3D {}; + int i, stable =3D 0; + + kill(hog, SIGSTOP); + waitpid(hog, NULL, WUNTRACED); + if (!ASSERT_OK(join_root_cgroup(), "join_root_cgroup")) + goto cleanup_hog; + + /* + * The period timer keeps adding to nr_periods for a while + * after the hog stops + */ + for (i =3D 0; i < 20; i++) { + struct cpu_query before =3D {}, after =3D {}; + + if (!ASSERT_OK(parse_stats(cgroup_fd, &before, have_bw), "cpu.stat") || + !ASSERT_OK(read_stats(link), "read stats") || + !ASSERT_OK(parse_stats(cgroup_fd, &after, have_bw), "cpu.stat")) + goto cleanup_hog; + + if (!memcmp(&before, &after, sizeof(before))) { + filev =3D before; + stable =3D 1; + break; + } + usleep(100000); + } + + if (!ASSERT_TRUE(stable, "cpu.stat stable")) + goto cleanup_hog; + + ASSERT_EQ(q->usage_usec, filev.usage_usec, "usage_usec"); + ASSERT_EQ(q->user_usec, filev.user_usec, "user_usec"); + ASSERT_EQ(q->system_usec, filev.system_usec, "system_usec"); + ASSERT_EQ(q->nice_usec, filev.nice_usec, "nice_usec"); + ASSERT_EQ(q->forceidle_usec, filev.forceidle_usec, "forceidle_usec"); + + if (have_bw) { + ASSERT_EQ(q->nr_periods, filev.nr_periods, "nr_periods"); + ASSERT_EQ(q->nr_throttled, filev.nr_throttled, "nr_throttled"); + ASSERT_EQ(q->throttled_usec, filev.throttled_usec, "throttled_usec"); + ASSERT_EQ(q->nr_bursts, filev.nr_bursts, "nr_bursts"); + ASSERT_EQ(q->burst_usec, filev.burst_usec, "burst_usec"); + ASSERT_EQ(q->throttled_self_usec, filev.throttled_self_usec, + "throttled_self_usec"); + } + } + +cleanup_hog: + kill(hog, SIGKILL); + waitpid(hog, NULL, 0); +cleanup_link: + bpf_link__destroy(link); +cleanup_skel: + cgroup_iter_cpu__destroy(skel); +cleanup_cgroup_fd: + close(cgroup_fd); + cleanup_cgroup_environment(); +} diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c b/tools/te= sting/selftests/bpf/progs/cgroup_iter_cpu.c new file mode 100644 index 000000000000..22b9bb62d910 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include "cgroup_iter_cpu.h" + +char _license[] SEC("license") =3D "GPL"; + +struct cpu_query cpu_query SEC(".data.query"); + +extern const void __cpu_possible_mask __ksym; + +struct cgroup_base_stat___local { + struct task_cputime cputime; + __u64 forceidle_sum; + __u64 ntime; +} __attribute__((preserve_access_index)); + +static __always_inline __u64 read_throttled_self(struct task_group *tg, __= u32 cpu) +{ + struct cfs_rq *cfs_rq; + + cfs_rq =3D bpf_per_cpu_ptr(tg->cfs_rq, cpu); + if (!cfs_rq) + return 0; + + return BPF_CORE_READ(cfs_rq, throttled_clock_self_time); +} + +SEC("iter.s/cgroup") +int cgroup_cpu_query(struct bpf_iter__cgroup *ctx) +{ + struct cgroup_base_stat___local bstat =3D {}; + struct cgroup *cgrp =3D ctx->cgroup; + struct cgroup_subsys_state *css; + struct task_group *tg; + __u64 throttled_self =3D 0; + int ssid; + + if (!cgrp) + return 1; + + bpf_css_flush_rstat(&cgrp->self); + bpf_cgroup_base_stat(cgrp, (struct cgroup_base_stat *)&bstat); + + cpu_query.usage_usec =3D bstat.cputime.sum_exec_runtime / 1000; + cpu_query.user_usec =3D bstat.cputime.utime / 1000; + cpu_query.system_usec =3D bstat.cputime.stime / 1000; + cpu_query.nice_usec =3D bstat.ntime / 1000; + cpu_query.forceidle_usec =3D 0; + if (bpf_core_field_exists(bstat.forceidle_sum)) + cpu_query.forceidle_usec =3D bstat.forceidle_sum / 1000; + + bpf_rcu_read_lock(); + if (!bpf_core_enum_value_exists(enum cgroup_subsys_id, cpu_cgrp_id) || + !bpf_ksym_exists(bpf_css_to_task_group)) + goto unlock; + + ssid =3D bpf_core_enum_value(enum cgroup_subsys_id, cpu_cgrp_id); + css =3D cgrp->subsys[ssid]; + if (!css) + goto unlock; + + tg =3D bpf_css_to_task_group(css); + if (tg && bpf_core_field_exists(tg->cfs_bandwidth.nr_periods)) { + cpu_query.nr_periods =3D + (__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_periods); + cpu_query.nr_throttled =3D + (__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_throttled); + cpu_query.throttled_usec =3D + BPF_CORE_READ(tg, cfs_bandwidth.throttled_time) / 1000; + cpu_query.nr_bursts =3D + (__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_burst); + cpu_query.burst_usec =3D + BPF_CORE_READ(tg, cfs_bandwidth.burst_time) / 1000; + } + + if (tg && bpf_core_field_exists(tg->cfs_rq) && + bpf_core_field_exists(struct cfs_rq, throttled_clock_self_time)) { + __u32 mask_bytes =3D bpf_core_type_size(struct cpumask); + __u32 full_words =3D mask_bytes / sizeof(__u64); + int *cpu; + + if (full_words) + bpf_for_each(bits, cpu, + (const __u64 *)&__cpu_possible_mask, + full_words) + throttled_self +=3D read_throttled_self(tg, *cpu); + + if (mask_bytes & (sizeof(__u64) - 1)) { + __u32 tail =3D 0; + const void *src =3D (const char *)&__cpu_possible_mask + + full_words * sizeof(__u64); + int bit; + + if (!bpf_probe_read_kernel(&tail, sizeof(tail), src)) + bpf_for(bit, 0, 32) + if (tail & (1U << bit)) { + __u32 tail_cpu =3D full_words * 64 + bit; + + throttled_self +=3D + read_throttled_self(tg, tail_cpu); + } + } + } + +unlock: + bpf_rcu_read_unlock(); + cpu_query.throttled_self_usec =3D throttled_self / 1000; + + return 0; +} --=20 2.53.0-Meta