From nobody Mon Jun 8 05:25:25 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 315B838F654; Fri, 5 Jun 2026 12:43:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663417; cv=none; b=Qr861JOwxRGfBBC4DGeia+PQhQoTeeNPYQq9XB1qSFxIFhz6xgORoss+A/YoUo3dPKuuyxAAeDnVKScBDEw6BYmTTXQXUG1/glkbKONr2DhwWVgbav10oOI2e/rQ57PYHw3i7igFe5cYNp1lQPk1Y1f81fCcWAzFmTLGrnRQ0gw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663417; c=relaxed/simple; bh=ESn+jy1pu/UCj/USFRLlenpiPmXSDjwbSckkIfRJE50=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=QL7RNfl2Shg9nPKI+fsiKV83IC0YjdeTfy21BjaxRSATu2qMz8fRivxhsCmJpnkDOpdzLLhaP7o3/g9Poo0BQc9KPmTZNY5wDzVc0yGyCrWPZwB5qOS/AV4uXvUTPevdrwQJCQ0poeNFno4QHXooOl9zqBdLhTgt2VcNXl/sDbo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=vQzhqwkT; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="vQzhqwkT" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=FvXbdK8gbx0YRdwnPl5AOZZaCXWHI0fEhG+nZSugnvU=; b=vQzhqwkTOdFuzD2Sb3CPz0e+2V RZraMv6wn6QMoL44dQzv2L9VyhKx/1JiKdROcfLKe/XJ8kXjrpAoinCuWhe6JrCLjDl4Ez2TLgvh5 Hz3M6rkTiam+3eEOqeX/pteq/T7BYfDo59rJRj1daQk+tVseqEBh62mfJ59qVUvQ4hPvCM4A/O9ko CY1Buh7/r4hbWU4EQ76kXfv73aqzbhH55ypOsJE/E/SbU61+wUIvbtVhcT635DoDuWZtamrtNGKmO 3BSWKPzPTvn0lNQktwAT3LlnmZUf1a7y/dxSXyeXArJSqug8sCIzsuhTSmGAPQZ6VM8oonFMQix5g 49IHoNsg==; Received: from 2001-1c00-8d85-4b00-266e-96ff-fe07-7dcc.cable.dynamic.v6.ziggo.nl ([2001:1c00:8d85:4b00:266e:96ff:fe07:7dcc] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.99.1 #2 (Red Hat Linux)) id 1wVTtH-00000007val-3swD; Fri, 05 Jun 2026 12:43:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 24265300642; Fri, 05 Jun 2026 14:43:11 +0200 (CEST) Message-ID: <20260605124051.338602724@infradead.org> User-Agent: quilt/0.68 Date: Fri, 05 Jun 2026 14:40:14 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: longman@redhat.com, chenridong@huaweicloud.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io Subject: [PATCH v3 1/7] sched/fair: Add cgroup_mode switch References: <20260605105513.354837583@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The effective task weight (W_t') for a task in cgroup g on CPU n is given b= y: W_t W_t' =3D W_g * F_g_n * ---------- \Sum W_t_n Where W_g is the group's weight (cpu.weight), F_g_n is the fraction of the group weight for CPU n and W_t/W is the relative weight of this task against all other tasks in the same group on the same CPU. Furthermore, this makes: \Sum W_t_n F_g_n =3D ---------- \Sum W_t The fraction of weight inside the group of CPU n against the whole group. The problem is with F_g_n, the primary goal of this fraction is to make sure that the relative weight of tasks, when distributed over CPUs is maintained. For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3 distribution, then if F_g_n would simply be 1 (no weight re-distribution) t= he effective relative weights (W_t') of the tasks in our group would be: CPU0 CPU1 W_g W_g/3 W_g/3 W_g/3 IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 ta= sks on CPU1 combined. However, with the weight redistribution, this becomes: CPU0 CPU1 W_g/4 W_g/4 W_g/4 W_g/4 All tasks are equal weight (as intended). However, as is already evident fr= om this example, the more CPUs you add, the smaller F_g_n becomes, which creat= es a disparity against tasks not in our group. Specifically: avg(F_g_n) ~ 1/N This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete fairly across hierarchy levels. *Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that wh= en there are at least N runnable tasks, the average of this fraction tends to = 1/N. For a hierarchy of depth d, this gets even worse, since that gets terms on = the order of: avg(F_g_n)^d ~ 1/(N^d) Given fixed point arithmetic, this also leads to numerical trouble. However, the meaning of "cpu.weight" is simple and intiutive: the total wei= ght of the cgroup. But as explored above, there is deception in this simplicity. Prepare to add a few alternative methods for distributing weight. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/debug.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++= +++++ 1 file changed, 74 insertions(+) --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -633,6 +633,76 @@ static void debugfs_fair_server_init(voi } } =20 +#ifdef CONFIG_FAIR_GROUP_SCHED +static int cgroup_mode =3D 0; + +static const char *cgroup_mode_str[] =3D { + "smp", +}; + +static int sched_cgroup_mode(const char *str) +{ + for (int i =3D 0; i < ARRAY_SIZE(cgroup_mode_str); i++) { + if (!strcmp(str, cgroup_mode_str[i])) + return i; + } + return -EINVAL; +} + +static ssize_t sched_cgroup_write(struct file *filp, const char __user *ub= uf, + size_t cnt, loff_t *ppos) +{ + char buf[16]; + int mode; + + if (cnt > 15) + cnt =3D 15; + + if (copy_from_user(buf, ubuf, cnt)) + return -EFAULT; + + buf[cnt] =3D 0; + mode =3D sched_cgroup_mode(strstrip(buf)); + if (mode < 0) + return mode; + + WRITE_ONCE(cgroup_mode, mode); + + *ppos +=3D cnt; + return cnt; +} + +static int sched_cgroup_show(struct seq_file *m, void *v) +{ + int mode =3D READ_ONCE(cgroup_mode); + + for (int i =3D 0; i < ARRAY_SIZE(cgroup_mode_str); i++) { + if (mode =3D=3D i) + seq_puts(m, "("); + seq_puts(m, cgroup_mode_str[i]); + if (mode =3D=3D i) + seq_puts(m, ")"); + + seq_puts(m, " "); + } + seq_puts(m, "\n"); + return 0; +} + +static int sched_cgroup_open(struct inode *inode, struct file *filp) +{ + return single_open(filp, sched_cgroup_show, NULL); +} + +static const struct file_operations sched_cgroup_fops =3D { + .open =3D sched_cgroup_open, + .write =3D sched_cgroup_write, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D single_release, +}; +#endif + static __init int sched_init_debug(void) { struct dentry __maybe_unused *numa, *llc; @@ -686,6 +756,10 @@ static __init int sched_init_debug(void) =20 debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops= ); =20 +#ifdef CONFIG_FAIR_GROUP_SCHED + debugfs_create_file("cgroup_mode", 0644, debugfs_sched, NULL, &sched_cgro= up_fops); +#endif + debugfs_fair_server_init(); #ifdef CONFIG_SCHED_CLASS_EXT debugfs_ext_server_init(); From nobody Mon Jun 8 05:25:25 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B3473CCFAF; Fri, 5 Jun 2026 12:43:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663419; cv=none; b=hoy9ElROiZUcpjbIterwhiJsVoLVygYwQmNZQladcwBJXqfVMsB2ixM1VLwp5fXsPAqPfKzvndF1J5Du/RZYCgwN13f55mQ5wnHHEdytyuADzVMvSIz6ERwa1Ms3LHRjLqxxxFckUaqfNONXgKAaLSsuF6Fc6CpBFD6jwqfCPz0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663419; c=relaxed/simple; bh=lbzlG6CnYK/XlgVM9FGh8L8DRqCeQViAtJK/BYTj3Ck=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Qi8kJnF4XywpBrnURy70NG6m/aFZfiq8NlFOL/v7Mm+FUwRELkL96yT5H4I/m4J3T0XkXkKJ/9eyJzQwcXkeT5cwodfrEFbrXJpfZwc/sNjnNpOHUtsMxMJZcJWKiMBuqpKerK7GlMCpyb/EdvEIDolWGiGIoRdstHZfufMhVX0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=fy0qlg1+; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="fy0qlg1+" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Cok1Asq6labJWXMXmPjojFgpwVmhgYai3/1aGF0Onxk=; b=fy0qlg1+Q3tsdt8eA4ds4ROGZi 1O8Dupxx1PBpvPgSHWPa8WP+zEMG7FknkNH/DTMdWkCLGTILaU/7VWYmBsQM6ym5rKQFMcZbJksew vPjEWw4YEt5UR4sz4KT6EYsDT2qSfvGPns0hgur9R7SL8DmOPTqTbgXcEKhewG/xcSVGG64nigoTL Ep7iuy9u0YeGpx7knY78sGL0JAl/oalUVA0A4Dr4j7Y14X8GAAWpJdBI5Y30FZgyBLWWE48XG/NWj CD5xft0D6ZTpB4MWvh9KT3VbzuQPWKzwfeRwTnTJ9MEsVNT0dAO1IItJ5+PqtbcuW/aiqaVSSCGTd yUXbdkSg==; Received: from 2001-1c00-8d85-4b00-266e-96ff-fe07-7dcc.cable.dynamic.v6.ziggo.nl ([2001:1c00:8d85:4b00:266e:96ff:fe07:7dcc] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.99.2 #2 (Red Hat Linux)) id 1wVTtJ-0000000FbIG-1Any; Fri, 05 Jun 2026 12:43:13 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 28653302D6E; Fri, 05 Jun 2026 14:43:11 +0200 (CEST) Message-ID: <20260605124051.450303977@infradead.org> User-Agent: quilt/0.68 Date: Fri, 05 Jun 2026 14:40:15 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: longman@redhat.com, chenridong@huaweicloud.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io Subject: [PATCH v3 2/7] sched/fair: Add cgroup_mode: up References: <20260605105513.354837583@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of calculating the proportional fraction of the group weight for ea= ch CPU, just give each CPU the full measure, ignoring these pesky SMP problems. This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single task in a cgroup competes on equal footing to a task in a level above. However, as already explored, this is not a very good policy because it gets the SMP weight distribution wrong. Included for completeness. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/debug.c | 5 ++++- kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 3 files changed, 34 insertions(+), 3 deletions(-) --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -271,6 +271,7 @@ static ssize_t sched_dynamic_write(struc if (mode < 0) return mode; =20 + __sched_cgroup_mode_update(mode); sched_dynamic_update(mode); =20 *ppos +=3D cnt; @@ -634,9 +635,11 @@ static void debugfs_fair_server_init(voi } =20 #ifdef CONFIG_FAIR_GROUP_SCHED -static int cgroup_mode =3D 0; +static int cgroup_mode =3D 1; =20 +/* See __sched_cgroup_mode_update(). */ static const char *cgroup_mode_str[] =3D { + "up", "smp", }; =20 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -38,6 +38,7 @@ #include #include #include +#include =20 #include #include @@ -4800,7 +4801,7 @@ static inline int throttled_hierarchy(st * * hence icky! */ -static long calc_group_shares(struct cfs_rq *cfs_rq) +static long calc_smp(struct cfs_rq *cfs_rq) { long tg_weight, tg_shares, load, shares; struct task_group *tg =3D cfs_rq->tg; @@ -4835,6 +4836,32 @@ static long calc_group_shares(struct cfs } =20 /* + * Ignore this pesky SMP stuff, use (4). + */ +static long calc_up_shares(struct cfs_rq *cfs_rq) +{ + struct task_group *tg =3D cfs_rq->tg; + return READ_ONCE(tg->shares); +} + +DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares); + +void __sched_cgroup_mode_update(int mode) +{ + long (*func)(struct cfs_rq *); + switch (mode) { + case 0: + func =3D &calc_up_shares; + break; + case 1: + default: + func =3D &calc_smp_shares; + break; + } + static_call_update(calc_group_shares, func); +} + +/* * Recomputes the group entity based on the current state of its group * runqueue. */ @@ -4850,7 +4877,7 @@ static void update_cfs_group(struct sche if (!gcfs_rq || !gcfs_rq->load.weight) return; =20 - shares =3D calc_group_shares(gcfs_rq); + shares =3D static_call(calc_group_shares)(gcfs_rq); if (unlikely(se->load.weight !=3D shares)) reweight_entity(cfs_rq_of(se), se, shares); } --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -571,6 +571,7 @@ extern void free_fair_sched_group(struct extern int alloc_fair_sched_group(struct task_group *tg, struct task_group= *parent); extern void online_fair_sched_group(struct task_group *tg); extern void unregister_fair_sched_group(struct task_group *tg); +extern void __sched_cgroup_mode_update(int mode); #else /* !CONFIG_FAIR_GROUP_SCHED: */ static inline void free_fair_sched_group(struct task_group *tg) { } static inline int alloc_fair_sched_group(struct task_group *tg, struct tas= k_group *parent) From nobody Mon Jun 8 05:25:25 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1A6A3BE628; Fri, 5 Jun 2026 12:43:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663417; cv=none; b=oCZ3iqHLqH8uPOo/EX/WyOI7a+CpuTdqg2T+9AFRIZvEYv+rd0mW+GrFef0ATmagnTu1KsjP/EkbBenwyZhPupMkIidR0e2g4gvcQbb1sjVnkrXMD7IUdoJEVTe/c2W7VEtEKcHscSo/XNqNtYAY0JUp0Qd+RkN7MayCmmortp8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663417; c=relaxed/simple; bh=c4VeD7Zsnsh/unQbMVpMRZY4PeiOUexoPYPgs57H0qE=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=SZNiGiFJ53QZsdUFhT126vlpPVtiOVdlCD4NCcN5Ucc7fXTErMc82N4vpWDsb1k+MGZi7o2SJqtnU3WkdgLPJgiQKFG27M0CMT4UiuEqxQ0WN601S7mu50JpdvLXIyTq6QtTC0qHa39qCO3M3y9QBRb/MO97YEy/DaydFwqd92c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=DbEBAgkJ; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="DbEBAgkJ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=+g4JhBB4PfZbxdVKSRhV8vLIidbqMDTAQZUEIvo+pog=; b=DbEBAgkJFcX1rGQxNeqWt4CQ++ 5eWJMCWSzlMl0d5Y693X2WbPWN3RA5AO8PUlfoZZOv2NAntEaoIfn8vWMHmwuvFJm38IFYA5rnJLT sCEuwbJfU4i3B6pcfndeeZHCEpa3+VuVFlmTA/5RD4mgcettt4XmPt3vgPmo4RxlqVbUQ7XraATUa 6YFqPWayrYQ5NmrFBEk/+Kw9zOjbc8WDh3Zw9x37epjNBx4V8MQRedQhgcWK8guTtUV4L3zPyNMzQ tlNEg3SxWe/oK1vZVBcOU5uJw0djOv1jryZl7m1GvA4ZXZxD42rkyLAclYTjwShG9QyXQIJtqz61J ROWe2HLA==; Received: from 2001-1c00-8d85-4b00-266e-96ff-fe07-7dcc.cable.dynamic.v6.ziggo.nl ([2001:1c00:8d85:4b00:266e:96ff:fe07:7dcc] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.99.2 #2 (Red Hat Linux)) id 1wVTtJ-0000000FbIH-1CPx; Fri, 05 Jun 2026 12:43:13 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 2D0DC302F8E; Fri, 05 Jun 2026 14:43:11 +0200 (CEST) Message-ID: <20260605124051.589618504@infradead.org> User-Agent: quilt/0.68 Date: Fri, 05 Jun 2026 14:40:16 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: longman@redhat.com, chenridong@huaweicloud.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io Subject: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max References: <20260605105513.354837583@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N', assume each cgroup is maximally concurrent and distrubute 'N*weight', such that: F_g_n' =3D N * F_g_n Giving: avg(F_g_n') =3D N*avg(F_g_n) ~ N * 1/N =3D 1 And while this sounds like it solves things, remember what that ~ meant. Th= ere is the corner case when a cgroup is minimally loaded, eg a single runnable task, therefore limit the CPU fraction to that of a nice -20 task to avoid getting too much load. This last bit is what makes it different from a previous proposal to allow raising cpu.weight to '100 * N', that would not limit the mininal concurren= cy case and results in a very large F_g_n. And just like F_g_n << 1 is problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the kthreads, but it also risks overflowing the load values). So while this might appear to be a better scheme than the current default scheme, it doesn't really handle less than maximal concurrency nicely -- it clips and introduces artificially large weights. So where the traditional S= MP mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that reg= ime and vice-versa. The meaning of "cpu.weight" would be: weight per allowed CPU. Included for completeness (and infrastructure). Signed-off-by: Peter Zijlstra (Intel) --- include/linux/cpuset.h | 6 +++++ kernel/cgroup/cpuset.c | 15 ++++++++++++++ kernel/sched/debug.c | 1=20 kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++= ----- 4 files changed, 69 insertions(+), 5 deletions(-) --- a/include/linux/cpuset.h +++ b/include/linux/cpuset.h @@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpuma= sk *mask); extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mas= k); extern bool cpuset_cpus_allowed_fallback(struct task_struct *p); +extern int cpuset_num_cpus(struct cgroup *cgroup); extern nodemask_t cpuset_mems_allowed(struct task_struct *p); #define cpuset_current_mems_allowed (current->mems_allowed) void cpuset_init_current_mems_allowed(void); @@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f return false; } =20 +static inline int cpuset_num_cpus(struct cgroup *cgroup) +{ + return num_online_cpus(); +} + static inline nodemask_t cpuset_mems_allowed(struct task_struct *p) { return node_possible_map; --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct return changed; } =20 +int cpuset_num_cpus(struct cgroup *cgrp) +{ + int nr =3D num_online_cpus(); + struct cpuset *cs; + + if (is_in_v2_mode()) { + guard(rcu)(); + cs =3D css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys)); + if (cs) + nr =3D cpumask_weight(cs->effective_cpus); + } + + return nr; +} + void __init cpuset_init_current_mems_allowed(void) { nodes_setall(current->mems_allowed); --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -641,6 +641,7 @@ static int cgroup_mode =3D 1; static const char *cgroup_mode_str[] =3D { "up", "smp", + "max", }; =20 static int sched_cgroup_mode(const char *str) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4801,12 +4801,10 @@ static inline int throttled_hierarchy(st * * hence icky! */ -static long calc_smp(struct cfs_rq *cfs_rq) +static long __calc_smp_shares(struct cfs_rq *cfs_rq, long tg_shares, long = shares_max) { - long tg_weight, tg_shares, load, shares; struct task_group *tg =3D cfs_rq->tg; - - tg_shares =3D READ_ONCE(tg->shares); + long tg_weight, load, shares; =20 load =3D max(scale_load_down(cfs_rq->load.weight), cfs_rq->avg.load_avg); =20 @@ -4832,7 +4830,48 @@ static long calc_smp(struct cfs_rq *cfs_ * case no task is runnable on a CPU MIN_SHARES=3D2 should be returned * instead of 0. */ - return clamp_t(long, shares, MIN_SHARES, tg_shares); + return clamp_t(long, shares, MIN_SHARES, shares_max); +} + +static int tg_cpus(struct task_group *tg) +{ + int nr =3D num_online_cpus(); + + if (cpusets_enabled()) { + struct cgroup *cgrp =3D tg->css.cgroup; + if (cgrp) + nr =3D cpuset_num_cpus(cgrp); + } + + return nr; +} + +/* + * Func: min(fraction(nr_cpus * tg->shares), nice -20) + * + * Scale tg->shares by the maximal number of CPUs; but clip the max shares= at + * nice -20, otherwise a single spinner on a 512 CPU machine would result = in + * 512*NICE_0_LOAD, which is also crazy. + */ +static long calc_max_shares(struct cfs_rq *cfs_rq) +{ + struct task_group *tg =3D cfs_rq->tg; + int nr =3D tg_cpus(tg); + long tg_shares =3D READ_ONCE(tg->shares); + long max_shares =3D scale_load(sched_prio_to_weight[0]); + return __calc_smp_shares(cfs_rq, tg_shares * nr, max_shares); +} + +/* + * Func: fraction(tg->shares) + * + * This infamously results in tiny shares when you have many CPUs. + */ +static long calc_smp_shares(struct cfs_rq *cfs_rq) +{ + struct task_group *tg =3D cfs_rq->tg; + long tg_shares =3D READ_ONCE(tg->shares); + return __calc_smp_shares(cfs_rq, tg_shares, tg_shares); } =20 /* @@ -4857,6 +4896,9 @@ void __sched_cgroup_mode_update(int mode default: func =3D &calc_smp_shares; break; + case 2: + func =3D &calc_max_shares; + break; } static_call_update(calc_group_shares, func); } From nobody Mon Jun 8 05:25:25 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 576CB3DEFE2; Fri, 5 Jun 2026 12:43:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663421; cv=none; b=hVleOokTnR3yVFAIALxE8qFf/5f7zAIrvQrraXbfOCxBSEpH4dbqzFxqPY5433xvsqwWeCgiXx9VftEdc1qMNW2F4FAZJzEdN53yOHiU3dmBjt/GgGiBltm63396amh8YbgYs3b57obcPNiVPgUWJUH7Qw7ezp2FomqDmnGlRZs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663421; c=relaxed/simple; bh=bK9TxAbVAPg5ND7kKzBB2mFvH0tsbKgba0QPSCsn2ko=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=EoydFpkDlmVMAqZkQWUcFRVY3gVgjnibCXSGVFXqUoJDdA/EfVBxS9n5fgQH0RwoMWpkPdnCdhUfID0YJVePrdXC8mG0FaLsipxYkw5lrGtK3S1nYei05Qb+eCtT72UkGVL1eOwdaI4X7vz2Dsw2yy09j+VoullbQOe5Z5BjFVk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=fLyyAt+9; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="fLyyAt+9" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=hjfLK/hrZVZ1NOOmYGZm3Pn+gttYY82Cdw6XleRZwT8=; b=fLyyAt+91eXxjLz7ZQw8SZ7r5d DIOJAVKGQtHt8uDYe9EbrRFsXjaSBG5iEDh6Rji2lQZXTEwnboRUfq3woUrDgrkZC0u8X3QRvDDB8 27Sk9Guz8iI9u5o89EvApSAO7/PhBbFRq93oP4I5l64BQH9bcCIeJFlkD0Rb3zsS7bxI3bH0GWcaL jq15XRqyIzTK6npwAMlpXP/+8xVgeCHuNYdUnBQmPdY+bnb5vMGMd6BsrZHRCjgVXir9aB8KUfsZI ucl0UUMvRlEYTCY4lCi4O/AVluadNHzdX9M6+WwBCgpg4RvvpQTiTA4Tc60vJatBCGtGgEGun+hLT HqrHtCxA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.99.1 #2 (Red Hat Linux)) id 1wVTtH-00000007vam-3n3X; Fri, 05 Jun 2026 12:43:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 3229D30300D; Fri, 05 Jun 2026 14:43:11 +0200 (CEST) Message-ID: <20260605124051.740585993@infradead.org> User-Agent: quilt/0.68 Date: Fri, 05 Jun 2026 14:40:17 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: longman@redhat.com, chenridong@huaweicloud.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io Subject: [PATCH v3 4/7] sched/fair: Add cgroup_mode: concur References: <20260605105513.354837583@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Improve upon the previous scheme ("max") by no longer assuming maximal concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles the low concurrency cases more gracefully: F_g_n' =3D min(M, N) * F_g_n Notably this is the first mode where: avg(F_g_n) =3D 1 In the single task case it reduces to ("smp") and then it nicely scales up until it hits N, where it behaves like ("max"). This is no longer clipped at nice -20. Strictly speaking it isn't different from the normal SMP scenario where all tasks are extremely unbalanced. There are no unnatural inflations in this scheme. The meaning of "cpu.weight" would be: weight per active CPU. NOTE: Compute the group wide number of tasks by extending the tg->load_avg computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on cfs_rq->h_nr_running. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 43 ++++++++++++++++++++++++++++++++++++------- kernel/sched/sched.h | 3 +++ 3 files changed, 40 insertions(+), 7 deletions(-) --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -641,6 +641,7 @@ static int cgroup_mode =3D 1; static const char *cgroup_mode_str[] =3D { "up", "smp", + "concur", "max", }; =20 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4846,6 +4846,11 @@ static int tg_cpus(struct task_group *tg return nr; } =20 +static inline int tg_tasks(struct task_group *tg) +{ + return max(1, atomic_long_read(&tg->runnable_avg) >> SCHED_CAPACITY_SHIFT= ); +} + /* * Func: min(fraction(nr_cpus * tg->shares), nice -20) * @@ -4863,6 +4868,20 @@ static long calc_max_shares(struct cfs_r } =20 /* + * Func: fraction(nr * tg->shares); nr =3D min(nr_tasks, nr_cpus) + * + * Scales between "smp" and "max" in a natural way. No longer needs clippi= ng + * since there are no unnatural inflations like with "max". + */ +static long calc_concur_shares(struct cfs_rq *cfs_rq) +{ + struct task_group *tg =3D cfs_rq->tg; + int nr =3D min(tg_tasks(tg), tg_cpus(tg)); + long tg_shares =3D READ_ONCE(tg->shares); + return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares); +} + +/* * Func: fraction(tg->shares) * * This infamously results in tiny shares when you have many CPUs. @@ -4897,6 +4916,9 @@ void __sched_cgroup_mode_update(int mode func =3D &calc_smp_shares; break; case 2: + func =3D &calc_concur_shares; + break; + case 3: func =3D &calc_max_shares; break; } @@ -5043,7 +5065,7 @@ static inline bool cfs_rq_is_decayed(str */ static inline void update_tg_load_avg(struct cfs_rq *cfs_rq) { - long delta; + long dl, dr; u64 now; =20 /* @@ -5064,17 +5086,21 @@ static inline void update_tg_load_avg(st if (now - cfs_rq->last_update_tg_load_avg < NSEC_PER_MSEC) return; =20 - delta =3D cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib; - if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) { - atomic_long_add(delta, &cfs_rq->tg->load_avg); + dl =3D cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib; + dr =3D cfs_rq->avg.runnable_avg - cfs_rq->tg_runnable_avg_contrib; + if (abs(dl) > cfs_rq->tg_load_avg_contrib / 64 || + abs(dr) > cfs_rq->tg_runnable_avg_contrib / 64) { + atomic_long_add(dl, &cfs_rq->tg->load_avg); + atomic_long_add(dr, &cfs_rq->tg->runnable_avg); cfs_rq->tg_load_avg_contrib =3D cfs_rq->avg.load_avg; + cfs_rq->tg_runnable_avg_contrib =3D cfs_rq->avg.runnable_avg; cfs_rq->last_update_tg_load_avg =3D now; } } =20 static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq) { - long delta; + long dl, dr; u64 now; =20 /* @@ -5084,9 +5110,12 @@ static inline void clear_tg_load_avg(str return; =20 now =3D rq_clock(rq_of(cfs_rq)); - delta =3D 0 - cfs_rq->tg_load_avg_contrib; - atomic_long_add(delta, &cfs_rq->tg->load_avg); + dl =3D 0 - cfs_rq->tg_load_avg_contrib; + dr =3D 0 - cfs_rq->tg_runnable_avg_contrib; + atomic_long_add(dl, &cfs_rq->tg->load_avg); + atomic_long_add(dr, &cfs_rq->tg->runnable_avg); cfs_rq->tg_load_avg_contrib =3D 0; + cfs_rq->tg_runnable_avg_contrib =3D 0; cfs_rq->last_update_tg_load_avg =3D now; } =20 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -493,6 +493,8 @@ struct task_group { * will also be accessed at each tick. */ atomic_long_t load_avg ____cacheline_aligned; + atomic_long_t runnable_avg; + #endif /* CONFIG_FAIR_GROUP_SCHED */ =20 #ifdef CONFIG_RT_GROUP_SCHED @@ -722,6 +724,7 @@ struct cfs_rq { #ifdef CONFIG_FAIR_GROUP_SCHED u64 last_update_tg_load_avg; unsigned long tg_load_avg_contrib; + unsigned long tg_runnable_avg_contrib; long propagate; long prop_runnable_sum; From nobody Mon Jun 8 05:25:25 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D19D33BE178; Fri, 5 Jun 2026 12:43:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663417; cv=none; b=r2eKlBP6P8eY6KwTEDbp0X2xub1EomXzr8s+msrcKSKVbJDXurk/wAkN2nHOcVbVSgkY/M7S9C7zPzIp2HGAxoFDgOX9U0uUFcwm8S/jgyJNgCAJ2qPE9JIZdXfUJy06xAgEkxHKXBSlNycFSb+1s/PNi8u2wmQR0IrLUvmhOh4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663417; c=relaxed/simple; bh=wPXmGodprzDP+VkF82WYp0drWRnJQcEqtntT6G1+gto=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=sC6HKoyZ+NnBcwyT4UwRsLp9S9oWRTAZshS9VGM0juEvj56eR3HOluu1AmHaMZF3zDFE7stPm4BkiINA3j+7KQwemUyjDGzsKoxeT2B+USShsc27kprGtjDxZNj02/YypFTbnWYcsLikBc6WrOHDiIyZ9yA7fIFBGww3t63bZ64= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=CH/VSXrf; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="CH/VSXrf" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=DFUPfbO4QZ1wRkdXk7p/tFg3ovHGoKUf1O97CDPHAEI=; b=CH/VSXrfdeRZMoyKl1RKWY7GSY jYART92EHPk5R+WquOVRNPPzZhBqCfNoL6B/biXOpuvErHkt8t9dl5TkDUwuETV8Vzd0/LtlXqqf+ nOeZqsFzKGS8PfzphN5D4CrRy+K/O+OeoTExW7d9snvHqGVFPNBZeD2OufcvVHRjPlepn3p/sgHlA 9h9FHMnp8n+v8RZ1tLegXdCUV/t8IbHm7uGc1O1DgRNiLTFA3523gm3aSzg7BDy/7mAgt9oIe5VuT Rp95sUuFp0owkkeYU55JZMLz7izQCon95o/kdXCxlPdFfry7oyWrM1wmAxCxM7HGN7jD61HXnjzV4 hxnCnMvw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.99.2 #2 (Red Hat Linux)) id 1wVTtJ-0000000FbIK-3GTu; Fri, 05 Jun 2026 12:43:13 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 3749B3030E7; Fri, 05 Jun 2026 14:43:11 +0200 (CEST) Message-ID: <20260605124051.921991975@infradead.org> User-Agent: quilt/0.68 Date: Fri, 05 Jun 2026 14:40:18 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: longman@redhat.com, chenridong@huaweicloud.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io Subject: [PATCH v3 5/7] sched/fair: Add cgroup_mode: tasks References: <20260605105513.354837583@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since we are exploring this space; include a scheme that scales by total nu= mber of runnable tasks. This results in: F_g_n' =3D M * F_g_n This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact). And while that sounds odd, it actually has a fairly straight foward meaning= for "cpu.weight": average weight per member task. This is an entirely valid and workable option, it is however wildly differe= nt from the traditional meaning. Included for completeness (and curiosity). Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+) --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -643,6 +643,7 @@ static const char *cgroup_mode_str[] =3D { "smp", "concur", "max", + "tasks", }; =20 static int sched_cgroup_mode(const char *str) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4852,6 +4852,19 @@ static inline int tg_tasks(struct task_g } =20 /* + * Func: fraction(nr_tasks * tg->shares) + * + * Scale tg->shares by the number of tasks. + */ +static long calc_tasks_shares(struct cfs_rq *cfs_rq) +{ + struct task_group *tg =3D cfs_rq->tg; + int nr =3D tg_tasks(tg); + long tg_shares =3D READ_ONCE(tg->shares); + return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares); +} + +/* * Func: min(fraction(nr_cpus * tg->shares), nice -20) * * Scale tg->shares by the maximal number of CPUs; but clip the max shares= at @@ -4921,6 +4934,9 @@ void __sched_cgroup_mode_update(int mode case 3: func =3D &calc_max_shares; break; + case 4: + func =3D &calc_tasks_shares; + break; } static_call_update(calc_group_shares, func); } From nobody Mon Jun 8 05:25:25 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65FF43C414F; Fri, 5 Jun 2026 12:43:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663418; cv=none; b=RnGHZlww4ZkoHLgTicdkiYN5K0DdSPXWGBapYeqyBxEHJ+Jmo4uEvyubnAl6souJGhIsgq6zRNcG9ONnO91hMsmCqBtmBFVAI4vdU+hyPMMd0onfN6b6CuI63Ren1ulMkJ15VjD940N1TYC5nzgso9DdVT1Kqj0euccHX2ad9eg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663418; c=relaxed/simple; bh=Ynt7i9S9WcoxmiTkpFWjBbZk4A2pTNZaEtkFTUFzxbY=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=dnFzMCPa4Rd4aE6eSVZRM4kI75HwLPYhjopqjALh9xEhXgQbpxKRM9PkBQ4pVYbZiGmWYhE4mYPdvFhPGT483WpyW6iPqcaQ0iAy009GDANHdeO2o2wd2eHaaEAU55wljdIzYFbpT5CVDypNvYiGQPcguYVmgwehVM4eY7GzFZs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=cMpwb87j; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="cMpwb87j" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=iagqKCq+Q/VlKH7MvQ1XZ0bf/xG9PDjiUd+QpNkj100=; b=cMpwb87j98ztIVKkVkYzhzizTc wK1pXB1k3vKwDP7ALrNE1LK7LRXDhhe5Lxuq4Zoy7UzbM1rPtHpW5k+IfzOaHQKxXheXJe8k43Yva JZ/0VbMAesN+6YfUYvWkVF7T2TfIYCXh0dkWna63CQRDgLVO87vw2wLF49YpG0Rd1jSS0MM6YrBGB JfqsZ6wc0cwfIqUejDmmF5yo2z4TOWPpv8QoVFa7J1JL0211vDQ57RTCRPFuHlghwc40xzRw+VYzj YXLL2/Y7G/MWVdk+5xaHrR5E6FjD2GeIH+k610uqWxVMs9FFvBEESAHlqo1+PlLgaVKR+WjaOqMOm QfV1ThhQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.99.1 #2 (Red Hat Linux)) id 1wVTtI-00000007var-28pa; Fri, 05 Jun 2026 12:43:13 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 3BA2130325A; Fri, 05 Jun 2026 14:43:11 +0200 (CEST) Message-ID: <20260605124052.080482755@infradead.org> User-Agent: quilt/0.68 Date: Fri, 05 Jun 2026 14:40:19 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: longman@redhat.com, chenridong@huaweicloud.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io Subject: [PATCH v3 6/7] sched/fair: Change the default cgroup_mode to concur References: <20260605105513.354837583@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For all the reasons described in the preceding patches, the way cgroup weight is computed is problematic. However, changing it is bound to also lead to trouble. Esp. since people might have taken to inflating the weight value where they can. Since things are configurable, change the default and hope this serves more people than it hurts, esp. in the longer run. Specifically, this prepares for a flattened runqueue, where the hierarchical weight becomes far more important (F_g^d terms), so getting rid of small F_= g is imperative. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -635,7 +635,7 @@ static void debugfs_fair_server_init(voi } =20 #ifdef CONFIG_FAIR_GROUP_SCHED -static int cgroup_mode =3D 1; +static int cgroup_mode =3D 2; =20 /* See __sched_cgroup_mode_update(). */ static const char *cgroup_mode_str[] =3D { --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4921,7 +4921,7 @@ static long calc_up_shares(struct cfs_rq return READ_ONCE(tg->shares); } =20 -DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares); +DEFINE_STATIC_CALL(calc_group_shares, calc_concur_shares); =20 void __sched_cgroup_mode_update(int mode) { @@ -4931,10 +4931,10 @@ void __sched_cgroup_mode_update(int mode func =3D &calc_up_shares; break; case 1: - default: func =3D &calc_smp_shares; break; case 2: + default: func =3D &calc_concur_shares; break; case 3: From nobody Mon Jun 8 05:25:25 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6A2B21770B; Fri, 5 Jun 2026 12:43:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663420; cv=none; b=l0G17kV4zrNURi2FL5TxMXHbsjmDqRmLX3fUvKPEi6kzQSxjXfeqMtM2Ya3Jadn64u0fzebb9lhtH5nrzeGDZAf5+Gf7MtqW9/9Kp+1UwE06W7bvbi9l2pjVseKo5+95H+1mzuVVC+Stxre3TOShKisvo0CandxwWoX/JxmgfDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780663420; c=relaxed/simple; bh=FpwRrBkwKSXkxnjoSnBO3oIeUunERLqzAklG8a93z6U=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=FWxPTTPkWlj0Ew8ruQP22F5j/VmtZggaKRL7jHWs+ZNjz4MHU0Y+Hzokx0n4Iutd3xAfjwk0IhVXTKkcZpBJxj1d7e5qaf558RVX8IsFclPWWqcVWfzq5ls/1McgxsPqGLcg1rvlewoCW5LJzsR+6icuHDdfPXIZE/uZCqhuHOw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=FvAgik7y; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="FvAgik7y" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=QENbhsT42wygOuhjLKNvfeYB+Nqy2NYCzFToxy05sAU=; b=FvAgik7ytJT9aNnBLbmOYdB7wX ss5xFsKXEC2euaEF9o3naxsZisaJ5w+iLSCAZ7wmd2BatyM2z/J9UMfjW+MOQCmzU8bLwaXtVkhTw 8RIpgmx8etIAVwqvec4EOL9s6UEqSmp2cpsg2EmSsBEOjqW1GvI6T4cOAwRNAVJVI4wyL6nLIvns9 exdxN51gjPGSvulcxwz6s1mlGWjkcaCA3ikKuj7BsqsOCH2i5JV6asyEOeSUveJ3hZ1StWV87umJf lCewxlB3YmnJe7Jefb2Dlg289gm74mAPvqr0Kj6B3XLubZEcI2fC+tY8v5bbmMKLQPvY0tbKv26XB IAysyAIg==; Received: from 2001-1c00-8d85-4b00-266e-96ff-fe07-7dcc.cable.dynamic.v6.ziggo.nl ([2001:1c00:8d85:4b00:266e:96ff:fe07:7dcc] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.99.1 #2 (Red Hat Linux)) id 1wVTtI-00000007vas-2Ads; Fri, 05 Jun 2026 12:43:13 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 40CDD30325C; Fri, 05 Jun 2026 14:43:11 +0200 (CEST) Message-ID: <20260605124052.227463677@infradead.org> User-Agent: quilt/0.68 Date: Fri, 05 Jun 2026 14:40:20 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: longman@redhat.com, chenridong@huaweicloud.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, jstultz@google.com, kprateek.nayak@amd.com, qyousef@layalina.io Subject: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue References: <20260605105513.354837583@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Change fair/cgroup to a single runqueue. Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks. The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved. That is, given a hierarchy like: R | se--G1 / \ G2--se se--G3 / \ | T1--se se--T2 se--T3 This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like: _R_ / | \ T1 T2 T3 Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick. Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load. Since EEVDF is now exclusively operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup. One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr). As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away. Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues. TODO: split struct cfs_rq / struct sched_entity TODO: try and get rid of h_curr Signed-off-by: Peter Zijlstra (Intel) --- include/linux/sched.h | 1=20 kernel/sched/core.c | 5=20 kernel/sched/debug.c | 9=20 kernel/sched/fair.c | 803 +++++++++++++++++++++------------------------= ----- kernel/sched/pelt.c | 6=20 kernel/sched/sched.h | 26 - 6 files changed, 375 insertions(+), 475 deletions(-) --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -574,6 +574,7 @@ struct sched_statistics { struct sched_entity { /* For load-balancing: */ struct load_weight load; + struct load_weight h_load; struct rb_node run_node; u64 deadline; u64 min_vruntime; --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5652,11 +5652,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat); */ static inline void prefetch_curr_exec_start(struct task_struct *p) { -#ifdef CONFIG_FAIR_GROUP_SCHED - struct sched_entity *curr =3D p->se.cfs_rq->curr; -#else struct sched_entity *curr =3D task_rq(p)->cfs.curr; -#endif + prefetch(curr); prefetch(&curr->exec_start); } --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rq else SEQ_printf(m, " %c", task_state_to_char(p)); =20 - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.= %06ld %9Ld %5d ", + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld= %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), + p->se.h_load.weight, SPLIT_NS(p->se.vruntime), - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice), @@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m, =20 SEQ_printf(m, "\n"); SEQ_printf(m, "runnable tasks:\n"); - SEQ_printf(m, " S task PID vruntime eligible " + SEQ_printf(m, " S task PID weight vruntime eligi= ble " "deadline slice sum-exec switches " "prio wait-time sum-sleep sum-block" #ifdef CONFIG_NUMA_BALANCING @@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, in cfs_rq->tg_load_avg_contrib); SEQ_printf(m, " .%-30s: %ld\n", "tg_load_avg", atomic_long_read(&cfs_rq->tg->load_avg)); + SEQ_printf(m, " .%-30s: %lu\n", "h_load", + cfs_rq->h_load); #endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_CFS_BANDWIDTH SEQ_printf(m, " .%-30s: %d\n", "throttled", --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -297,8 +297,8 @@ static u64 __calc_delta(u64 delta_exec, */ static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se) { - if (unlikely(se->load.weight !=3D NICE_0_LOAD)) - delta =3D __calc_delta(delta, NICE_0_LOAD, &se->load); + if (se->h_load.weight !=3D NICE_0_LOAD) + delta =3D __calc_delta(delta, NICE_0_LOAD, &se->h_load); =20 return delta; } @@ -428,38 +428,6 @@ static inline struct sched_entity *paren return se->parent; } =20 -static void -find_matching_se(struct sched_entity **se, struct sched_entity **pse) -{ - int se_depth, pse_depth; - - /* - * preemption test can be made between sibling entities who are in the - * same cfs_rq i.e who have a common parent. Walk up the hierarchy of - * both tasks until we find their ancestors who are siblings of common - * parent. - */ - - /* First walk up until both entities are at same depth */ - se_depth =3D (*se)->depth; - pse_depth =3D (*pse)->depth; - - while (se_depth > pse_depth) { - se_depth--; - *se =3D parent_entity(*se); - } - - while (pse_depth > se_depth) { - pse_depth--; - *pse =3D parent_entity(*pse); - } - - while (!is_same_group(*se, *pse)) { - *se =3D parent_entity(*se); - *pse =3D parent_entity(*pse); - } -} - static int tg_is_idle(struct task_group *tg) { return tg->idle > 0; @@ -503,11 +471,6 @@ static inline struct sched_entity *paren return NULL; } =20 -static inline void -find_matching_se(struct sched_entity **se, struct sched_entity **pse) -{ -} - static inline int tg_is_idle(struct task_group *tg) { return 0; @@ -686,7 +649,7 @@ static inline unsigned long avg_vruntime static inline void __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se) { - unsigned long weight =3D avg_vruntime_weight(cfs_rq, se->load.weight); + unsigned long weight =3D avg_vruntime_weight(cfs_rq, se->h_load.weight); s64 w_vruntime, key =3D entity_key(cfs_rq, se); =20 w_vruntime =3D key * weight; @@ -703,7 +666,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r s64 key, tmp; =20 again: - weight =3D avg_vruntime_weight(cfs_rq, se->load.weight); + weight =3D avg_vruntime_weight(cfs_rq, se->h_load.weight); key =3D entity_key(cfs_rq, se); =20 if (check_mul_overflow(key, weight, &key)) @@ -749,7 +712,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq static void sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se) { - unsigned long weight =3D avg_vruntime_weight(cfs_rq, se->load.weight); + unsigned long weight =3D avg_vruntime_weight(cfs_rq, se->h_load.weight); s64 key =3D entity_key(cfs_rq, se); =20 cfs_rq->sum_w_vruntime -=3D key * weight; @@ -791,7 +754,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq) s64 runtime =3D cfs_rq->sum_w_vruntime; =20 if (curr) { - unsigned long w =3D avg_vruntime_weight(cfs_rq, curr->load.weight); + unsigned long w =3D avg_vruntime_weight(cfs_rq, curr->h_load.weight); =20 runtime +=3D entity_key(cfs_rq, curr) * w; weight +=3D w; @@ -862,8 +825,6 @@ bool update_entity_lag(struct cfs_rq *cf u64 avruntime =3D avg_vruntime(cfs_rq); s64 vlag =3D entity_lag(cfs_rq, se, avruntime); =20 - WARN_ON_ONCE(!se->on_rq); - if (se->sched_delayed) { /* previous vlag < 0 otherwise se would not be delayed */ vlag =3D max(vlag, se->vlag); @@ -899,7 +860,7 @@ static int vruntime_eligible(struct cfs_ long load =3D cfs_rq->sum_weight; =20 if (curr && curr->on_rq) { - unsigned long weight =3D avg_vruntime_weight(cfs_rq, curr->load.weight); + unsigned long weight =3D avg_vruntime_weight(cfs_rq, curr->h_load.weight= ); =20 avg +=3D entity_key(cfs_rq, curr) * weight; load +=3D weight; @@ -1040,6 +1001,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim */ static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *s= e) { + WARN_ON_ONCE(&rq_of(cfs_rq)->cfs !=3D cfs_rq); + WARN_ON_ONCE(!entity_is_task(se)); + sum_w_vruntime_add(cfs_rq, se); se->min_vruntime =3D se->vruntime; se->min_slice =3D se->slice; @@ -1049,6 +1013,9 @@ static void __enqueue_entity(struct cfs_ =20 static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *s= e) { + WARN_ON_ONCE(&rq_of(cfs_rq)->cfs !=3D cfs_rq); + WARN_ON_ONCE(!entity_is_task(se)); + rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline, &min_vruntime_cb); sum_w_vruntime_sub(cfs_rq, se); @@ -1145,7 +1112,7 @@ static struct sched_entity *pick_eevdf(s * We can safely skip eligibility check if there is only one entity * in this cfs_rq, saving some cycles. */ - if (cfs_rq->nr_queued =3D=3D 1) + if (cfs_rq->h_nr_queued =3D=3D 1) return curr && curr->on_rq ? curr : se; =20 /* @@ -1395,8 +1362,6 @@ static s64 update_se(struct rq *rq, stru return delta_exec; } =20 -static void set_next_buddy(struct sched_entity *se); - #ifdef CONFIG_SCHED_CACHE =20 /* @@ -1991,7 +1956,7 @@ static void update_curr(struct cfs_rq *c * not necessarily be the actual task running * (rq->curr.se). This is easy to confuse! */ - struct sched_entity *curr =3D cfs_rq->curr; + struct sched_entity *curr =3D cfs_rq->h_curr; struct rq *rq =3D rq_of(cfs_rq); s64 delta_exec; bool resched; @@ -2003,26 +1968,29 @@ static void update_curr(struct cfs_rq *c if (unlikely(delta_exec <=3D 0)) return; =20 + account_cfs_rq_runtime(cfs_rq, delta_exec); + + if (!entity_is_task(curr)) + return; + + cfs_rq =3D &rq->cfs; + curr->vruntime +=3D calc_delta_fair(delta_exec, curr); resched =3D update_deadline(cfs_rq, curr); =20 - if (entity_is_task(curr)) { - /* - * If the fair_server is active, we need to account for the - * fair_server time whether or not the task is running on - * behalf of fair_server or not: - * - If the task is running on behalf of fair_server, we need - * to limit its time based on the assigned runtime. - * - Fair task that runs outside of fair_server should account - * against fair_server such that it can account for this time - * and possibly avoid running this period. - */ - dl_server_update(&rq->fair_server, delta_exec); - } - - account_cfs_rq_runtime(cfs_rq, delta_exec); + /* + * If the fair_server is active, we need to account for the + * fair_server time whether or not the task is running on + * behalf of fair_server or not: + * - If the task is running on behalf of fair_server, we need + * to limit its time based on the assigned runtime. + * - Fair task that runs outside of fair_server should account + * against fair_server such that it can account for this time + * and possibly avoid running this period. + */ + dl_server_update(&rq->fair_server, delta_exec); =20 - if (cfs_rq->nr_queued =3D=3D 1) + if (cfs_rq->h_nr_queued =3D=3D 1) return; =20 if (resched || !protect_slice(curr)) { @@ -2033,7 +2001,10 @@ static void update_curr(struct cfs_rq *c =20 static void update_curr_fair(struct rq *rq) { - update_curr(cfs_rq_of(&rq->donor->se)); + struct sched_entity *se =3D &rq->donor->se; + + for_each_sched_entity(se) + update_curr(cfs_rq_of(se)); } =20 static inline void @@ -2109,7 +2080,7 @@ update_stats_enqueue_fair(struct cfs_rq * Are we enqueueing a waiting task? (for current tasks * a dequeue/enqueue event is a NOP) */ - if (se !=3D cfs_rq->curr) + if (se !=3D cfs_rq->h_curr) update_stats_wait_start_fair(cfs_rq, se); =20 if (flags & ENQUEUE_WAKEUP) @@ -2127,7 +2098,7 @@ update_stats_dequeue_fair(struct cfs_rq * Mark the end of the wait period if dequeueing a * waiting task: */ - if (se !=3D cfs_rq->curr) + if (se !=3D cfs_rq->h_curr) update_stats_wait_end_fair(cfs_rq, se); =20 if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) { @@ -4468,6 +4439,7 @@ static inline void update_scan_period(st static void account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) { + WARN_ON_ONCE(cfs_rq !=3D cfs_rq_of(se)); update_load_add(&cfs_rq->load, se->load.weight); if (entity_is_task(se)) { struct task_struct *p =3D task_of(se); @@ -4483,6 +4455,7 @@ account_entity_enqueue(struct cfs_rq *cf static void account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) { + WARN_ON_ONCE(cfs_rq !=3D cfs_rq_of(se)); update_load_sub(&cfs_rq->load, se->load.weight); if (entity_is_task(se)) { struct task_struct *p =3D task_of(se); @@ -4564,7 +4537,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq, static void rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vpr= ot) { - unsigned long old_weight =3D se->load.weight; + long old_weight =3D se->h_load.weight; =20 /* * VRUNTIME @@ -4664,16 +4637,17 @@ rescale_entity(struct sched_entity *se, se->vprot =3D div64_long(se->vprot * old_weight, weight); } =20 -static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, - unsigned long weight) +static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se, + unsigned long weight, bool on_rq) { bool curr =3D cfs_rq->curr =3D=3D se; bool rel_vprot =3D false; u64 avruntime =3D 0; =20 - if (se->on_rq) { - /* commit outstanding execution time */ - update_curr(cfs_rq); + if (se->h_load.weight =3D=3D weight) + return; + + if (on_rq) { avruntime =3D avg_vruntime(cfs_rq); se->vlag =3D entity_lag(cfs_rq, se, avruntime); se->deadline -=3D avruntime; @@ -4683,46 +4657,90 @@ static void reweight_entity(struct cfs_r rel_vprot =3D true; } =20 - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--; if (!curr) __dequeue_entity(cfs_rq, se); - update_load_sub(&cfs_rq->load, se->load.weight); } - dequeue_load_avg(cfs_rq, se); =20 rescale_entity(se, weight, rel_vprot); =20 - update_load_set(&se->load, weight); + update_load_set(&se->h_load, weight); =20 - do { - u32 divider =3D get_pelt_divider(&se->avg); - se->avg.load_avg =3D div_u64(se_weight(se) * se->avg.load_sum, divider); - } while (0); - - enqueue_load_avg(cfs_rq, se); - if (se->on_rq) { + if (on_rq) { if (rel_vprot) se->vprot +=3D avruntime; se->deadline +=3D avruntime; se->rel_deadline =3D 0; se->vruntime =3D avruntime - se->vlag; =20 - update_load_add(&cfs_rq->load, se->load.weight); if (!curr) __enqueue_entity(cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; } } =20 +static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, + unsigned long weight) +{ + if (se->load.weight =3D=3D weight) + return; + + if (se->on_rq) { + WARN_ON_ONCE(cfs_rq !=3D cfs_rq_of(se)); + update_load_sub(&cfs_rq->load, se->load.weight); + } + dequeue_load_avg(cfs_rq, se); + + update_load_set(&se->load, weight); + + do { + u32 divider =3D get_pelt_divider(&se->avg); + se->avg.load_avg =3D div_u64(se_weight(se) * se->avg.load_sum, divider); + } while (0); + + enqueue_load_avg(cfs_rq, se); + + if (se->on_rq) + update_load_add(&cfs_rq->load, se->load.weight); +} + +/* + * weight =3D NICE_0_LOAD; + * for_each_entity_se(se) + * weight =3D __calc_prop_weight(cfs_rq_of(se), se, weight); + */ +static __always_inline +unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entit= y *se, + unsigned long weight) +{ + weight *=3D se->load.weight; + if (parent_entity(se)) + weight /=3D cfs_rq->load.weight; + else + weight /=3D NICE_0_LOAD; + + return max(weight, MIN_SHARES); +} + static void reweight_task_fair(struct rq *rq, struct task_struct *p, const struct load_weight *lw) { struct sched_entity *se =3D &p->se; - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - struct load_weight *load =3D &se->load; + unsigned long weight =3D NICE_0_LOAD; =20 - reweight_entity(cfs_rq, se, lw->weight); - load->inv_weight =3D lw->inv_weight; + if (se->on_rq) + update_curr_fair(rq); + + reweight_entity(cfs_rq_of(se), se, lw->weight); + se->load.inv_weight =3D lw->inv_weight; + + if (!se->on_rq) + return; + + for_each_sched_entity(se) + weight =3D __calc_prop_weight(cfs_rq_of(se), se, weight); + + reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq); } =20 static inline int throttled_hierarchy(struct cfs_rq *cfs_rq); @@ -4958,8 +4976,7 @@ static void update_cfs_group(struct sche return; =20 shares =3D static_call(calc_group_shares)(gcfs_rq); - if (unlikely(se->load.weight !=3D shares)) - reweight_entity(cfs_rq_of(se), se, shares); + reweight_entity(cfs_rq_of(se), se, shares); } =20 #else /* !CONFIG_FAIR_GROUP_SCHED: */ @@ -5077,7 +5094,7 @@ static inline bool cfs_rq_is_decayed(str * differential update where we store the last value we propagated. This in * turn allows skipping updates if the differential is 'small'. * - * Updating tg's load_avg is necessary before update_cfs_share(). + * Updating tg's load_avg is necessary before update_cfs_group(). */ static inline void update_tg_load_avg(struct cfs_rq *cfs_rq) { @@ -5544,7 +5561,7 @@ static void migrate_se_pelt_lag(struct s * The cfs_rq avg is the direct sum of all its entities (blocked and runna= ble) * avg. The immediate corollary is that all (fair) tasks must be attached. * - * cfs_rq->avg is used for task_h_load() and update_cfs_share() for exampl= e. + * cfs_rq->avg is used for task_h_load() and update_cfs_group() for exampl= e. * * Return: true if the load decayed or we removed load. * @@ -6082,6 +6099,7 @@ static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { u64 vslice, vruntime =3D avg_vruntime(cfs_rq); + unsigned int nr_queued =3D cfs_rq->h_nr_queued; bool update_zero =3D false; s64 lag =3D 0; =20 @@ -6089,6 +6107,9 @@ place_entity(struct cfs_rq *cfs_rq, stru se->slice =3D sysctl_sched_base_slice; vslice =3D calc_delta_fair(se->slice, se); =20 + if (flags & ENQUEUE_QUEUED) + nr_queued -=3D 1; + /* * Due to how V is constructed as the weighted average of entities, * adding tasks with positive lag, or removing tasks with negative lag @@ -6097,7 +6118,7 @@ place_entity(struct cfs_rq *cfs_rq, stru * * EEVDF: placement strategy #1 / #2 */ - if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) { + if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) { struct sched_entity *curr =3D cfs_rq->curr; long load, weight; =20 @@ -6157,9 +6178,9 @@ place_entity(struct cfs_rq *cfs_rq, stru */ load =3D cfs_rq->sum_weight; if (curr && curr->on_rq) - load +=3D avg_vruntime_weight(cfs_rq, curr->load.weight); + load +=3D avg_vruntime_weight(cfs_rq, curr->h_load.weight); =20 - weight =3D avg_vruntime_weight(cfs_rq, se->load.weight); + weight =3D avg_vruntime_weight(cfs_rq, se->h_load.weight); lag *=3D load + weight; if (WARN_ON_ONCE(!load)) load =3D 1; @@ -6218,22 +6239,8 @@ static void check_enqueue_throttle(struc static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq); =20 static void -requeue_delayed_entity(struct sched_entity *se); - -static void enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { - bool curr =3D cfs_rq->curr =3D=3D se; - - /* - * If we're the current task, we must renormalise before calling - * update_curr(). - */ - if (curr) - place_entity(cfs_rq, se, flags); - - update_curr(cfs_rq); - /* * When enqueuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. @@ -6252,13 +6259,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st */ update_cfs_group(se); =20 - /* - * XXX now that the entity has been re-weighted, and it's lag adjusted, - * we can place the entity. - */ - if (!curr) - place_entity(cfs_rq, se, flags); - account_entity_enqueue(cfs_rq, se); =20 /* Entity has migrated, no longer consider this task hot */ @@ -6267,8 +6267,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st =20 check_schedstat_required(); update_stats_enqueue_fair(cfs_rq, se, flags); - if (!curr) - __enqueue_entity(cfs_rq, se); se->on_rq =3D 1; =20 if (cfs_rq->nr_queued =3D=3D 1) { @@ -6286,21 +6284,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st } } =20 -static void __clear_buddies_next(struct sched_entity *se) +static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se) { - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - if (cfs_rq->next !=3D se) - break; - - cfs_rq->next =3D NULL; - } + if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed)) + return; + if (se_is_idle(se)) + return; + cfs_rq->next =3D se; } =20 static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se) { if (cfs_rq->next =3D=3D se) - __clear_buddies_next(se); + cfs_rq->next =3D NULL; } =20 static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); @@ -6311,7 +6307,7 @@ static void set_delayed(struct sched_ent =20 /* * Delayed se of cfs_rq have no tasks queued on them. - * Do not adjust h_nr_runnable since dequeue_entities() + * Do not adjust h_nr_runnable since __dequeue_task() * will account it for blocked tasks. */ if (!entity_is_task(se)) @@ -6344,45 +6340,16 @@ static void clear_delayed(struct sched_e } } =20 -static bool +static void dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { - bool sleep =3D flags & DEQUEUE_SLEEP; - int action =3D 0; - - update_curr(cfs_rq); - clear_buddies(cfs_rq, se); - - if (flags & DEQUEUE_DELAYED) { - WARN_ON_ONCE(!se->sched_delayed); - } else { - bool delay =3D sleep; - /* - * DELAY_DEQUEUE relies on spurious wakeups, special task - * states must not suffer spurious wakeups, excempt them. - */ - if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE)) - delay =3D false; - - WARN_ON_ONCE(delay && se->sched_delayed); + int action =3D UPDATE_TG; =20 - if (sched_feat(DELAY_DEQUEUE) && delay && - !entity_eligible(cfs_rq, se)) { - if (entity_is_task(se)) - action |=3D UPDATE_UTIL_EST; - update_load_avg(cfs_rq, se, action); - update_entity_lag(cfs_rq, se); - set_delayed(se); - return false; - } - } - - action =3D UPDATE_TG; if (entity_is_task(se)) { if (task_on_rq_migrating(task_of(se))) action |=3D DO_DETACH; =20 - if (sleep && !(flags & DEQUEUE_DELAYED)) + if ((flags & DEQUEUE_SLEEP) && !(flags & DEQUEUE_DELAYED)) action |=3D UPDATE_UTIL_EST; } =20 @@ -6400,14 +6367,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st =20 update_stats_dequeue_fair(cfs_rq, se, flags); =20 - update_entity_lag(cfs_rq, se); - if (sched_feat(PLACE_REL_DEADLINE) && !sleep) { - se->deadline -=3D se->vruntime; - se->rel_deadline =3D 1; - } - - if (se !=3D cfs_rq->curr) - __dequeue_entity(cfs_rq, se); se->on_rq =3D 0; account_entity_dequeue(cfs_rq, se); =20 @@ -6416,9 +6375,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st =20 update_cfs_group(se); =20 - if (flags & DEQUEUE_DELAYED) - clear_delayed(se); - if (cfs_rq->nr_queued =3D=3D 0) { update_idle_cfs_rq_clock_pelt(cfs_rq); #ifdef CONFIG_CFS_BANDWIDTH @@ -6431,15 +6387,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st } #endif } - - return true; } =20 static void -set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first) +set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) { - clear_buddies(cfs_rq, se); - /* 'current' is not kept within the tree. */ if (se->on_rq) { /* @@ -6448,16 +6400,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s * runqueue. */ update_stats_wait_end_fair(cfs_rq, se); - __dequeue_entity(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); - - if (first) - set_protect_slice(cfs_rq, se); } =20 update_stats_curr_start(cfs_rq, se); - WARN_ON_ONCE(cfs_rq->curr); - cfs_rq->curr =3D se; + WARN_ON_ONCE(cfs_rq->h_curr); + cfs_rq->h_curr =3D se; =20 /* * Track our maximum slice length, if the CPU's load is at @@ -6477,23 +6425,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s se->prev_sum_exec_runtime =3D se->sum_exec_runtime; } =20 -static int dequeue_entities(struct rq *rq, struct sched_entity *se, int fl= ags); +static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags= ); =20 -/* - * Pick the next process, keeping these things in mind, in this order: - * 1) keep things fair between processes/task groups - * 2) pick the "next" process, since someone really wants that to run - * 3) pick the "last" process, for cache locality - * 4) do not run the "skip" process, if something else is available - */ static struct sched_entity * -pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect) +pick_next_entity(struct rq *rq, bool protect) { + struct cfs_rq *cfs_rq =3D &rq->cfs; struct sched_entity *se; =20 se =3D pick_eevdf(cfs_rq, protect); if (se->sched_delayed) { - dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED); + __dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED); /* * Must not reference @se again, see __block_task(). */ @@ -6513,13 +6455,11 @@ static void put_prev_entity(struct cfs_r =20 if (prev->on_rq) { update_stats_wait_start_fair(cfs_rq, prev); - /* Put 'current' back into the tree. */ - __enqueue_entity(cfs_rq, prev); /* in !on_rq case, update occurred at dequeue */ update_load_avg(cfs_rq, prev, 0); } - WARN_ON_ONCE(cfs_rq->curr !=3D prev); - cfs_rq->curr =3D NULL; + WARN_ON_ONCE(cfs_rq->h_curr !=3D prev); + cfs_rq->h_curr =3D NULL; } =20 static void @@ -7074,7 +7014,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf assert_list_leaf_cfs_rq(rq); =20 /* Determine whether we need to wake up potentially idle CPU: */ - if (rq->curr =3D=3D rq->idle && rq->cfs.nr_queued) + if (rq->curr =3D=3D rq->idle && rq->cfs.h_nr_queued) resched_curr(rq); } =20 @@ -7409,7 +7349,7 @@ static void check_enqueue_throttle(struc return; =20 /* an active group must be handled by the update_curr() path */ - if (!cfs_rq->runtime_enabled || cfs_rq->curr) + if (!cfs_rq->runtime_enabled || cfs_rq->h_curr) return; =20 /* ensure the group is not already throttled */ @@ -7781,7 +7721,7 @@ static void hrtick_start_fair(struct rq resched_curr(rq); return; } - delta =3D (se->load.weight * vdelta) / NICE_0_LOAD; + delta =3D (se->h_load.weight * vdelta) / NICE_0_LOAD; =20 /* * Correct for instantaneous load of other classes. @@ -7881,10 +7821,8 @@ static int choose_idle_cpu(int cpu, stru } =20 static void -requeue_delayed_entity(struct sched_entity *se) +requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - /* * se->sched_delayed should imply: se->on_rq =3D=3D 1. * Because a delayed entity is one that is still on @@ -7894,19 +7832,58 @@ requeue_delayed_entity(struct sched_enti WARN_ON_ONCE(!se->on_rq); =20 if (update_entity_lag(cfs_rq, se)) { - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--; if (se !=3D cfs_rq->curr) __dequeue_entity(cfs_rq, se); place_entity(cfs_rq, se, 0); if (se !=3D cfs_rq->curr) __enqueue_entity(cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; } =20 update_load_avg(cfs_rq, se, 0); clear_delayed(se); } =20 +static unsigned long enqueue_hierarchy(struct task_struct *p, int flags) +{ + unsigned long weight =3D NICE_0_LOAD; + int task_new =3D !(flags & ENQUEUE_WAKEUP); + struct sched_entity *se =3D &p->se; + int h_nr_idle =3D task_has_idle_policy(p); + int h_nr_runnable =3D 1; + + if (task_new && se->sched_delayed) + h_nr_runnable =3D 0; + + for_each_sched_entity(se) { + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + + update_curr(cfs_rq); + + if (!se->on_rq) { + enqueue_entity(cfs_rq, se, flags); + } else { + update_load_avg(cfs_rq, se, UPDATE_TG); + se_update_runnable(se); + update_cfs_group(se); + } + + cfs_rq->h_nr_runnable +=3D h_nr_runnable; + cfs_rq->h_nr_queued++; + cfs_rq->h_nr_idle +=3D h_nr_idle; + + if (cfs_rq_is_idle(cfs_rq)) + h_nr_idle =3D 1; + + weight =3D __calc_prop_weight(cfs_rq, se, weight); + + flags =3D ENQUEUE_WAKEUP; + } + + return weight; +} + /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -7915,13 +7892,12 @@ requeue_delayed_entity(struct sched_enti static void enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) { - struct cfs_rq *cfs_rq; - struct sched_entity *se =3D &p->se; - int h_nr_idle =3D task_has_idle_policy(p); - int h_nr_runnable =3D 1; - int task_new =3D !(flags & ENQUEUE_WAKEUP); int rq_h_nr_queued =3D rq->cfs.h_nr_queued; - u64 slice =3D 0; + int task_new =3D !(flags & ENQUEUE_WAKEUP); + struct sched_entity *se =3D &p->se; + struct cfs_rq *cfs_rq =3D &rq->cfs; + unsigned long weight; + bool curr; =20 if (task_is_throttled(p) && enqueue_throttled_task(p)) return; @@ -7933,10 +7909,10 @@ enqueue_task_fair(struct rq *rq, struct * estimated utilization, before we update schedutil. */ if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED)) - util_est_enqueue(&rq->cfs, p); + util_est_enqueue(cfs_rq, p); =20 if (flags & ENQUEUE_DELAYED) { - requeue_delayed_entity(se); + requeue_delayed_entity(cfs_rq, se); return; } =20 @@ -7948,57 +7924,22 @@ enqueue_task_fair(struct rq *rq, struct if (p->in_iowait) cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT); =20 - if (task_new && se->sched_delayed) - h_nr_runnable =3D 0; - - for_each_sched_entity(se) { - if (se->on_rq) { - if (se->sched_delayed) - requeue_delayed_entity(se); - break; - } - cfs_rq =3D cfs_rq_of(se); - - /* - * Basically set the slice of group entries to the min_slice of - * their respective cfs_rq. This ensures the group can service - * its entities in the desired time-frame. - */ - if (slice) { - se->slice =3D slice; - se->custom_slice =3D 1; - } - enqueue_entity(cfs_rq, se, flags); - slice =3D cfs_rq_min_slice(cfs_rq); - - cfs_rq->h_nr_runnable +=3D h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle +=3D h_nr_idle; - - if (cfs_rq_is_idle(cfs_rq)) - h_nr_idle =3D 1; - - flags =3D ENQUEUE_WAKEUP; - } - - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); + /* + * XXX comment on the curr thing + */ + curr =3D (cfs_rq->curr =3D=3D se); + if (curr) + place_entity(cfs_rq, se, flags); =20 - update_load_avg(cfs_rq, se, UPDATE_TG); - se_update_runnable(se); - update_cfs_group(se); + if (se->on_rq && se->sched_delayed) + requeue_delayed_entity(cfs_rq, se); =20 - se->slice =3D slice; - if (se !=3D cfs_rq->curr) - min_vruntime_cb_propagate(&se->run_node, NULL); - slice =3D cfs_rq_min_slice(cfs_rq); + weight =3D enqueue_hierarchy(p, flags); =20 - cfs_rq->h_nr_runnable +=3D h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle +=3D h_nr_idle; - - if (cfs_rq_is_idle(cfs_rq)) - h_nr_idle =3D 1; + if (!curr) { + reweight_eevdf(cfs_rq, se, weight, false); + place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED); + __enqueue_entity(cfs_rq, se); } =20 if (!rq_h_nr_queued && rq->cfs.h_nr_queued) @@ -8029,105 +7970,109 @@ enqueue_task_fair(struct rq *rq, struct hrtick_update(rq); } =20 -/* - * Basically dequeue_task_fair(), except it can deal with dequeue_entity() - * failing half-way through and resume the dequeue later. - * - * Returns: - * -1 - dequeue delayed - * 0 - dequeue throttled - * 1 - dequeue complete - */ -static int dequeue_entities(struct rq *rq, struct sched_entity *se, int fl= ags) +static void dequeue_hierarchy(struct task_struct *p, int flags) { - bool was_sched_idle =3D sched_idle_rq(rq); + struct sched_entity *se =3D &p->se; bool task_sleep =3D flags & DEQUEUE_SLEEP; bool task_delayed =3D flags & DEQUEUE_DELAYED; bool task_throttled =3D flags & DEQUEUE_THROTTLE; - struct task_struct *p =3D NULL; - int h_nr_idle =3D 0; - int h_nr_queued =3D 0; int h_nr_runnable =3D 0; - struct cfs_rq *cfs_rq; - u64 slice =3D 0; + int h_nr_idle =3D task_has_idle_policy(p); + bool dequeue =3D true; =20 - if (entity_is_task(se)) { - p =3D task_of(se); - h_nr_queued =3D 1; - h_nr_idle =3D task_has_idle_policy(p); - if (task_sleep || task_delayed || !se->sched_delayed) - h_nr_runnable =3D 1; - } + if (task_sleep || task_delayed || !se->sched_delayed) + h_nr_runnable =3D 1; =20 for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 - if (!dequeue_entity(cfs_rq, se, flags)) { - if (p && &p->se =3D=3D se) - return -1; + update_curr(cfs_rq); =20 - slice =3D cfs_rq_min_slice(cfs_rq); - break; + if (dequeue) { + dequeue_entity(cfs_rq, se, flags); + /* Don't dequeue parent if it has other entities besides us */ + if (cfs_rq->load.weight) + dequeue =3D false; + } else { + update_load_avg(cfs_rq, se, UPDATE_TG); + se_update_runnable(se); + update_cfs_group(se); } =20 cfs_rq->h_nr_runnable -=3D h_nr_runnable; - cfs_rq->h_nr_queued -=3D h_nr_queued; + cfs_rq->h_nr_queued--; cfs_rq->h_nr_idle -=3D h_nr_idle; =20 if (cfs_rq_is_idle(cfs_rq)) - h_nr_idle =3D h_nr_queued; + h_nr_idle =3D 1; =20 if (throttled_hierarchy(cfs_rq) && task_throttled) record_throttle_clock(cfs_rq); =20 - /* Don't dequeue parent if it has other entities besides us */ - if (cfs_rq->load.weight) { - slice =3D cfs_rq_min_slice(cfs_rq); - - /* Avoid re-evaluating load for this entity: */ - se =3D parent_entity(se); - /* - * Bias pick_next to pick a task from this cfs_rq, as - * p is sleeping when it is within its sched_slice. - */ - if (task_sleep && se) - set_next_buddy(se); - break; - } flags |=3D DEQUEUE_SLEEP; flags &=3D ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL); } +} =20 - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); +/* + * The part of dequeue_task_fair() that is needed to dequeue delayed tasks. + * + * Returns: + * true - dequeued + * false - delayed + */ +static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) +{ + struct sched_entity *se =3D &p->se; + struct cfs_rq *cfs_rq =3D &rq->cfs; + bool was_sched_idle =3D sched_idle_rq(rq); + bool task_sleep =3D flags & DEQUEUE_SLEEP; + bool task_delayed =3D flags & DEQUEUE_DELAYED; =20 - update_load_avg(cfs_rq, se, UPDATE_TG); - se_update_runnable(se); - update_cfs_group(se); + clear_buddies(cfs_rq, se); =20 - se->slice =3D slice; - if (se !=3D cfs_rq->curr) - min_vruntime_cb_propagate(&se->run_node, NULL); - slice =3D cfs_rq_min_slice(cfs_rq); + update_curr(cfs_rq_of(se)); + update_entity_lag(cfs_rq, se); =20 - cfs_rq->h_nr_runnable -=3D h_nr_runnable; - cfs_rq->h_nr_queued -=3D h_nr_queued; - cfs_rq->h_nr_idle -=3D h_nr_idle; + if (flags & DEQUEUE_DELAYED) { + WARN_ON_ONCE(!se->sched_delayed); + } else { + bool delay =3D task_sleep; + /* + * DELAY_DEQUEUE relies on spurious wakeups, special task + * states must not suffer spurious wakeups, excempt them. + */ + if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE)) + delay =3D false; =20 - if (cfs_rq_is_idle(cfs_rq)) - h_nr_idle =3D h_nr_queued; + WARN_ON_ONCE(delay && se->sched_delayed); =20 - if (throttled_hierarchy(cfs_rq) && task_throttled) - record_throttle_clock(cfs_rq); + if (sched_feat(DELAY_DEQUEUE) && delay && + !entity_eligible(cfs_rq, se)) { + update_load_avg(cfs_rq_of(se), se, UPDATE_UTIL_EST); + set_delayed(se); + return false; + } } =20 - sub_nr_running(rq, h_nr_queued); + dequeue_hierarchy(p, flags); + + if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) { + se->deadline -=3D se->vruntime; + se->rel_deadline =3D 1; + } + if (se !=3D cfs_rq->curr) + __dequeue_entity(cfs_rq, se); + + sub_nr_running(rq, 1); =20 /* balance early to pull high priority tasks */ if (unlikely(!was_sched_idle && sched_idle_rq(rq))) rq->next_balance =3D jiffies; =20 - if (p && task_delayed) { + if (task_delayed) { + clear_delayed(se); + WARN_ON_ONCE(!task_sleep); WARN_ON_ONCE(p->on_rq !=3D 1); =20 @@ -8139,7 +8084,7 @@ static int dequeue_entities(struct rq *r __block_task(rq, p); } =20 - return 1; + return true; } =20 /* @@ -8157,11 +8102,11 @@ static bool dequeue_task_fair(struct rq if (!p->se.sched_delayed) util_est_dequeue(&rq->cfs, p); =20 - if (dequeue_entities(rq, &p->se, flags) < 0) + if (!__dequeue_task(rq, p, flags)) return false; =20 /* - * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED). + * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED). */ return true; } @@ -9749,19 +9694,6 @@ static void migrate_task_rq_fair(struct static void task_dead_fair(struct task_struct *p) { struct sched_entity *se =3D &p->se; - - if (se->sched_delayed) { - struct rq_flags rf; - struct rq *rq; - - rq =3D task_rq_lock(p, &rf); - if (se->sched_delayed) { - update_rq_clock(rq); - dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED); - } - task_rq_unlock(rq, p, &rf); - } - remove_entity_load_avg(se); } =20 @@ -9795,21 +9727,10 @@ static void set_cpus_allowed_fair(struct set_task_max_allowed_capacity(p); } =20 -static void set_next_buddy(struct sched_entity *se) -{ - for_each_sched_entity(se) { - if (WARN_ON_ONCE(!se->on_rq)) - return; - if (se_is_idle(se)) - return; - cfs_rq_of(se)->next =3D se; - } -} - enum preempt_wakeup_action { PREEMPT_WAKEUP_NONE, /* No preemption. */ PREEMPT_WAKEUP_SHORT, /* Ignore slice protection. */ - PREEMPT_WAKEUP_PICK, /* Let __pick_eevdf() decide. */ + PREEMPT_WAKEUP_PICK, /* Let pick_eevdf() decide. */ PREEMPT_WAKEUP_RESCHED, /* Force reschedule. */ }; =20 @@ -9826,7 +9747,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq, if (cfs_rq->next && entity_before(cfs_rq->next, pse)) return false; =20 - set_next_buddy(pse); + set_next_buddy(cfs_rq, pse); return true; } =20 @@ -9879,7 +9800,7 @@ static void wakeup_preempt_fair(struct r enum preempt_wakeup_action preempt_action =3D PREEMPT_WAKEUP_PICK; struct task_struct *donor =3D rq->donor; struct sched_entity *nse, *se =3D &donor->se, *pse =3D &p->se; - struct cfs_rq *cfs_rq =3D task_cfs_rq(donor); + struct cfs_rq *cfs_rq =3D &rq->cfs; int cse_is_idle, pse_is_idle; =20 /* @@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r if (!sched_feat(WAKEUP_PREEMPTION)) return; =20 - find_matching_se(&se, &pse); WARN_ON_ONCE(!pse); =20 cse_is_idle =3D se_is_idle(se); @@ -9944,8 +9864,7 @@ static void wakeup_preempt_fair(struct r if (unlikely(!normal_policy(p->policy))) return; =20 - cfs_rq =3D cfs_rq_of(se); - update_curr(cfs_rq); + update_curr_fair(rq); /* * If @p has a shorter slice than current and @p is eligible, override * current's slice protection in order to allow preemption. @@ -9989,18 +9908,15 @@ static void wakeup_preempt_fair(struct r } =20 pick: - nse =3D pick_next_entity(rq, cfs_rq, preempt_action !=3D PREEMPT_WAKEUP_S= HORT); - /* If @p has become the most eligible task, force preemption */ - if (nse =3D=3D pse) - goto preempt; - - /* - * Because p is enqueued, nse being null can only mean that we - * dequeued a delayed task. If there are still entities queued in - * cfs, check if the next one will be p. - */ - if (!nse && cfs_rq->nr_queued) - goto pick; + if (cfs_rq->h_nr_queued) { + nse =3D pick_next_entity(rq, preempt_action !=3D PREEMPT_WAKEUP_SHORT); + if (unlikely(!nse)) + goto pick; + + /* If @p has become the most eligible task, force preemption */ + if (nse =3D=3D pse) + goto preempt; + } =20 if (sched_feat(RUN_TO_PARITY)) update_protect_slice(cfs_rq, se); @@ -10019,33 +9935,24 @@ static void wakeup_preempt_fair(struct r struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) __must_hold(__rq_lockp(rq)) { + struct cfs_rq *cfs_rq =3D &rq->cfs; struct sched_entity *se; - struct cfs_rq *cfs_rq; struct task_struct *p; - bool throttled; int new_tasks; =20 again: - cfs_rq =3D &rq->cfs; - if (!cfs_rq->nr_queued) + if (!cfs_rq->h_nr_queued) goto idle; =20 - throttled =3D false; - - do { - /* Might not have done put_prev_entity() */ - if (cfs_rq->curr && cfs_rq->curr->on_rq) - update_curr(cfs_rq); + /* Might not have done put_prev_entity() */ + if (cfs_rq->curr && cfs_rq->curr->on_rq) + update_curr(cfs_rq); =20 - se =3D pick_next_entity(rq, cfs_rq, true); - if (!se) - goto again; - cfs_rq =3D group_cfs_rq(se); - } while (cfs_rq); + se =3D pick_next_entity(rq, true); + if (!se) + goto again; =20 p =3D task_of(se); - if (unlikely(throttled)) - task_throttle_setup_work(p); return p; =20 idle: @@ -10079,7 +9986,7 @@ void fair_server_init(struct rq *rq) static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, st= ruct task_struct *next) { struct sched_entity *se =3D &prev->se; - struct cfs_rq *cfs_rq; + struct cfs_rq *cfs_rq =3D &rq->cfs; struct sched_entity *nse =3D NULL; =20 #ifdef CONFIG_FAIR_GROUP_SCHED @@ -10089,7 +9996,7 @@ static void put_prev_task_fair(struct rq =20 while (se) { cfs_rq =3D cfs_rq_of(se); - if (!nse || cfs_rq->curr) + if (!nse || cfs_rq->h_curr) put_prev_entity(cfs_rq, se); #ifdef CONFIG_FAIR_GROUP_SCHED if (nse) { @@ -10108,6 +10015,14 @@ static void put_prev_task_fair(struct rq #endif se =3D parent_entity(se); } + + /* Put 'current' back into the tree. */ + cfs_rq =3D &rq->cfs; + se =3D &prev->se; + WARN_ON_ONCE(cfs_rq->curr !=3D se); + cfs_rq->curr =3D NULL; + if (se->on_rq) + __enqueue_entity(cfs_rq, se); } =20 /* @@ -10116,8 +10031,8 @@ static void put_prev_task_fair(struct rq static void yield_task_fair(struct rq *rq) { struct task_struct *curr =3D rq->donor; - struct cfs_rq *cfs_rq =3D task_cfs_rq(curr); struct sched_entity *se =3D &curr->se; + struct cfs_rq *cfs_rq =3D &rq->cfs; =20 /* * Are we the only task in the tree? @@ -10158,11 +10073,11 @@ static bool yield_to_task_fair(struct rq struct sched_entity *se =3D &p->se; =20 /* !se->on_rq also covers throttled task */ - if (!se->on_rq) + if (!se->on_rq || se->sched_delayed) return false; =20 /* Tell the scheduler that we'd really like se to run next. */ - set_next_buddy(se); + set_next_buddy(&task_rq(p)->cfs, se); =20 yield_task_fair(rq); =20 @@ -10501,15 +10416,10 @@ static inline long migrate_degrades_loca */ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int= dest_cpu) { - struct cfs_rq *dst_cfs_rq; + struct cfs_rq *dst_cfs_rq =3D &cpu_rq(dest_cpu)->cfs; =20 -#ifdef CONFIG_FAIR_GROUP_SCHED - dst_cfs_rq =3D tg_cfs_rq(task_group(p), dest_cpu); -#else - dst_cfs_rq =3D &cpu_rq(dest_cpu)->cfs; -#endif - if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued && - !entity_eligible(task_cfs_rq(p), &p->se)) + if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued && + !entity_eligible(&task_rq(p)->cfs, &p->se)) return 1; =20 return 0; @@ -11292,7 +11202,7 @@ static void update_cfs_rq_h_load(struct while ((se =3D READ_ONCE(cfs_rq->h_load_next)) !=3D NULL) { load =3D cfs_rq->h_load; load =3D div64_ul(load * se->avg.load_avg, - cfs_rq_load_avg(cfs_rq) + 1); + cfs_rq_load_avg(cfs_rq) + 1); cfs_rq =3D group_cfs_rq(se); cfs_rq->h_load =3D load; cfs_rq->last_h_load_update =3D now; @@ -14684,7 +14594,7 @@ static inline void task_tick_core(struct * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check * if we need to give up the CPU. */ - if (rq->core->core_forceidle_count && rq->cfs.nr_queued =3D=3D 1 && + if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued =3D=3D 1 && __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE)) resched_curr(rq); } @@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str =20 WARN_ON_ONCE(task_rq(b)->core !=3D rq->core); =20 -#ifdef CONFIG_FAIR_GROUP_SCHED - /* - * Find an se in the hierarchy for tasks a and b, such that the se's - * are immediate siblings. - */ - while (sea->cfs_rq->tg !=3D seb->cfs_rq->tg) { - int sea_depth =3D sea->depth; - int seb_depth =3D seb->depth; - - if (sea_depth >=3D seb_depth) - sea =3D parent_entity(sea); - if (sea_depth <=3D seb_depth) - seb =3D parent_entity(seb); - } - - se_fi_update(sea, rq->core->core_forceidle_seq, in_fi); - se_fi_update(seb, rq->core->core_forceidle_seq, in_fi); - - cfs_rqa =3D sea->cfs_rq; - cfs_rqb =3D seb->cfs_rq; -#else /* !CONFIG_FAIR_GROUP_SCHED: */ cfs_rqa =3D &task_rq(a)->cfs; cfs_rqb =3D &task_rq(b)->cfs; -#endif /* !CONFIG_FAIR_GROUP_SCHED */ =20 /* * Find delta after normalizing se's vruntime with its cfs_rq's @@ -14955,11 +14843,20 @@ static inline void task_tick_core(struct static void task_tick_fair(struct rq *rq, struct task_struct *curr, int qu= eued) { struct sched_entity *se =3D &curr->se; - struct cfs_rq *cfs_rq; =20 - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); - entity_tick(cfs_rq, se, queued); + if (se->on_rq) { + unsigned long weight =3D NICE_0_LOAD; + struct cfs_rq *cfs_rq; + + for_each_sched_entity(se) { + cfs_rq =3D cfs_rq_of(se); + entity_tick(cfs_rq, se, queued); + + weight =3D __calc_prop_weight(cfs_rq, se, weight); + } + + se =3D &curr->se; + reweight_eevdf(cfs_rq, se, weight, se->on_rq); } =20 if (queued) @@ -14999,7 +14896,7 @@ prio_changed_fair(struct rq *rq, struct if (p->prio =3D=3D oldprio) return; =20 - if (rq->cfs.nr_queued =3D=3D 1) + if (rq->cfs.h_nr_queued =3D=3D 1) return; =20 /* @@ -15128,33 +15025,44 @@ static void switched_to_fair(struct rq * } } =20 -/* - * Account for a task changing its policy or group. - * - * This routine is mostly called to set cfs_rq->curr field when a task - * migrates between groups/classes. - */ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool = first) { struct sched_entity *se =3D &p->se; bool throttled =3D false; + struct cfs_rq *cfs_rq =3D &rq->cfs; + unsigned long weight =3D NICE_0_LOAD; + bool on_rq =3D se->on_rq; + + clear_buddies(cfs_rq, se); + + if (on_rq) + __dequeue_entity(cfs_rq, se); =20 for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + cfs_rq =3D cfs_rq_of(se); =20 - if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) && - first && cfs_rq->curr) - break; + if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) || + !first || !cfs_rq->h_curr) + set_next_entity(cfs_rq, se); =20 - set_next_entity(cfs_rq, se, first); /* ensure bandwidth has been allocated on our new cfs_rq */ throttled |=3D account_cfs_rq_runtime(cfs_rq, 0); + + if (on_rq) + weight =3D __calc_prop_weight(cfs_rq, se, weight); } =20 if (throttled) task_throttle_setup_work(p); =20 se =3D &p->se; + cfs_rq->curr =3D se; + + if (on_rq) { + reweight_eevdf(cfs_rq, se, weight, se->on_rq); + if (first) + set_protect_slice(cfs_rq, se); + } =20 if (task_on_rq_queued(p)) { /* @@ -15267,17 +15175,8 @@ void unregister_fair_sched_group(struct struct sched_entity *se =3D tg_se(tg, cpu); struct rq *rq =3D cpu_rq(cpu); =20 - if (se) { - if (se->sched_delayed) { - guard(rq_lock_irqsave)(rq); - if (se->sched_delayed) { - update_rq_clock(rq); - dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED); - } - list_del_leaf_cfs_rq(cfs_rq); - } + if (se) remove_entity_load_avg(se); - } =20 /* * Only empty task groups can be destroyed; so we can speculatively --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched /* * running is a subset of runnable (weight) so running can't be set if * runnable is clear. But there are some corner cases where the current - * se has been already dequeued but cfs_rq->curr still points to it. + * se has been already dequeued but cfs_rq->h_curr still points to it. * This means that weight will be 0 but not running for a sched_entity * but also for a cfs_rq if the latter becomes idle. As an example, * this happens during sched_balance_newidle() which calls @@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_enti= ty *se) { if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se), - cfs_rq->curr =3D=3D se)) { + cfs_rq->h_curr =3D=3D se)) { =20 ___update_load_avg(&se->avg, se_weight(se)); cfs_se_util_change(&se->avg); @@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st if (___update_load_sum(now, &cfs_rq->avg, scale_load_down(cfs_rq->load.weight), cfs_rq->h_nr_runnable, - cfs_rq->curr !=3D NULL)) { + cfs_rq->h_curr !=3D NULL)) { =20 ___update_load_avg(&cfs_rq->avg, 1); trace_pelt_cfs_tp(cfs_rq); --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -530,21 +530,8 @@ struct task_group { =20 }; =20 -#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD =20 -/* - * A weight of 0 or 1 can cause arithmetics problems. - * A weight of a cfs_rq is the sum of weights of which entities - * are queued on this cfs_rq, so a weight of a entity should not be - * too large, so as the shares value of a task group. - * (The default weight is 1024 - so there's no practical - * limitation from this.) - */ -#define MIN_SHARES (1UL << 1) -#define MAX_SHARES (1UL << 18) -#endif - typedef int (*tg_visitor)(struct task_group *, void *); =20 extern int walk_tg_tree_from(struct task_group *from, @@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrain =20 #endif /* !CONFIG_CGROUP_SCHED */ =20 +/* + * A weight of 0 or 1 can cause arithmetics problems. + * A weight of a cfs_rq is the sum of weights of which entities + * are queued on this cfs_rq, so a weight of a entity should not be + * too large, so as the shares value of a task group. + * (The default weight is 1024 - so there's no practical + * limitation from this.) + */ +#define MIN_SHARES (1UL << 1) +#define MAX_SHARES (1UL << 18) + extern void unregister_rt_sched_group(struct task_group *tg); extern void free_rt_sched_group(struct task_group *tg); extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *= parent); @@ -709,6 +707,7 @@ struct cfs_rq { /* * CFS load tracking */ + struct sched_entity *h_curr; struct sched_avg avg; #ifndef CONFIG_64BIT u64 last_update_time_copy; @@ -2575,6 +2574,7 @@ extern const u32 sched_prio_to_wmult[40 #define ENQUEUE_MIGRATED 0x00040000 #define ENQUEUE_INITIAL 0x00080000 #define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_QUEUED 0x00200000 =20 #define RETRY_TASK ((void *)-1UL)