From nobody Wed Apr 15 00:02:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8764C00144 for ; Fri, 29 Jul 2022 11:13:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232723AbiG2LNe (ORCPT ); Fri, 29 Jul 2022 07:13:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235834AbiG2LN3 (ORCPT ); Fri, 29 Jul 2022 07:13:29 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7B1BF7B34C for ; Fri, 29 Jul 2022 04:13:28 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DA723106F; Fri, 29 Jul 2022 04:13:28 -0700 (PDT) Received: from localhost.localdomain (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 339D73F73D; Fri, 29 Jul 2022 04:13:26 -0700 (PDT) From: Dietmar Eggemann To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Steven Rostedt Cc: Daniel Bristot de Oliveira , Valentin Schneider , Mel Gorman , Ben Segall , Luca Abeni , linux-kernel@vger.kernel.org Subject: [PATCH v2 1/3] sched: Introduce sched_asym_cpucap_active() Date: Fri, 29 Jul 2022 13:13:03 +0200 Message-Id: <20220729111305.1275158-2-dietmar.eggemann@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729111305.1275158-1-dietmar.eggemann@arm.com> References: <20220729111305.1275158-1-dietmar.eggemann@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Create an inline helper for conditional code to be only executed on asymmetric CPU capacity systems. This makes these (currently ~10 and future) conditions a lot more readable. Signed-off-by: Dietmar Eggemann --- kernel/sched/cpudeadline.c | 2 +- kernel/sched/deadline.c | 4 ++-- kernel/sched/fair.c | 8 ++++---- kernel/sched/rt.c | 4 ++-- kernel/sched/sched.h | 5 +++++ 5 files changed, 14 insertions(+), 9 deletions(-) diff --git a/kernel/sched/cpudeadline.c b/kernel/sched/cpudeadline.c index 02d970a879ed..57c92d751bcd 100644 --- a/kernel/sched/cpudeadline.c +++ b/kernel/sched/cpudeadline.c @@ -123,7 +123,7 @@ int cpudl_find(struct cpudl *cp, struct task_struct *p, unsigned long cap, max_cap =3D 0; int cpu, max_cpu =3D -1; =20 - if (!static_branch_unlikely(&sched_asym_cpucapacity)) + if (!sched_asym_cpucap_active()) return 1; =20 /* Ensure the capacity of the CPUs fits the task. */ diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 5867e186c39a..3f9d90b8a8b6 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -144,7 +144,7 @@ static inline unsigned long __dl_bw_capacity(int i) */ static inline unsigned long dl_bw_capacity(int i) { - if (!static_branch_unlikely(&sched_asym_cpucapacity) && + if (!sched_asym_cpucap_active() && capacity_orig_of(i) =3D=3D SCHED_CAPACITY_SCALE) { return dl_bw_cpus(i) << SCHED_CAPACITY_SHIFT; } else { @@ -1846,7 +1846,7 @@ select_task_rq_dl(struct task_struct *p, int cpu, int= flags) * Take the capacity of the CPU into account to * ensure it fits the requirement of the task. */ - if (static_branch_unlikely(&sched_asym_cpucapacity)) + if (sched_asym_cpucap_active()) select_rq |=3D !dl_task_fits_capacity(p, cpu); =20 if (select_rq) { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2fc47257ae91..3b186c9c4ea1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4262,7 +4262,7 @@ static inline int task_fits_capacity(struct task_stru= ct *p, =20 static inline void update_misfit_status(struct task_struct *p, struct rq *= rq) { - if (!static_branch_unlikely(&sched_asym_cpucapacity)) + if (!sched_asym_cpucap_active()) return; =20 if (!p || p->nr_cpus_allowed =3D=3D 1) { @@ -6506,7 +6506,7 @@ select_idle_capacity(struct task_struct *p, struct sc= hed_domain *sd, int target) =20 static inline bool asym_fits_capacity(unsigned long task_util, int cpu) { - if (static_branch_unlikely(&sched_asym_cpucapacity)) + if (sched_asym_cpucap_active()) return fits_capacity(task_util, capacity_of(cpu)); =20 return true; @@ -6526,7 +6526,7 @@ static int select_idle_sibling(struct task_struct *p,= int prev, int target) * On asymmetric system, update task utilization because we will check * that the task fits with cpu's capacity. */ - if (static_branch_unlikely(&sched_asym_cpucapacity)) { + if (sched_asym_cpucap_active()) { sync_entity_load_avg(&p->se); task_util =3D uclamp_task_util(p); } @@ -6580,7 +6580,7 @@ static int select_idle_sibling(struct task_struct *p,= int prev, int target) * For asymmetric CPU capacity systems, our domain of interest is * sd_asym_cpucapacity rather than sd_llc. */ - if (static_branch_unlikely(&sched_asym_cpucapacity)) { + if (sched_asym_cpucap_active()) { sd =3D rcu_dereference(per_cpu(sd_asym_cpucapacity, target)); /* * On an asymmetric CPU capacity system where an exclusive diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 55f39c8f4203..054b6711e961 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -509,7 +509,7 @@ static inline bool rt_task_fits_capacity(struct task_st= ruct *p, int cpu) unsigned int cpu_cap; =20 /* Only heterogeneous systems can benefit from this check */ - if (!static_branch_unlikely(&sched_asym_cpucapacity)) + if (!sched_asym_cpucap_active()) return true; =20 min_cap =3D uclamp_eff_value(p, UCLAMP_MIN); @@ -1897,7 +1897,7 @@ static int find_lowest_rq(struct task_struct *task) * If we're on asym system ensure we consider the different capacities * of the CPUs when searching for the lowest_mask. */ - if (static_branch_unlikely(&sched_asym_cpucapacity)) { + if (sched_asym_cpucap_active()) { =20 ret =3D cpupri_find_fitness(&task_rq(task)->rd->cpupri, task, lowest_mask, diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 73ae32898f25..72704b2b4a45 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1808,6 +1808,11 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym= _packing); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); extern struct static_key_false sched_asym_cpucapacity; =20 +static __always_inline bool sched_asym_cpucap_active(void) +{ + return static_branch_unlikely(&sched_asym_cpucapacity); +} + struct sched_group_capacity { atomic_t ref; /* --=20 2.25.1 From nobody Wed Apr 15 00:02:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90D36C00144 for ; Fri, 29 Jul 2022 11:13:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235895AbiG2LNg (ORCPT ); Fri, 29 Jul 2022 07:13:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235871AbiG2LNc (ORCPT ); Fri, 29 Jul 2022 07:13:32 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 3CD5A7B34C for ; Fri, 29 Jul 2022 04:13:31 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A061B1570; Fri, 29 Jul 2022 04:13:31 -0700 (PDT) Received: from localhost.localdomain (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A39E33F73D; Fri, 29 Jul 2022 04:13:28 -0700 (PDT) From: Dietmar Eggemann To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Steven Rostedt Cc: Daniel Bristot de Oliveira , Valentin Schneider , Mel Gorman , Ben Segall , Luca Abeni , linux-kernel@vger.kernel.org Subject: [PATCH v2 2/3] sched/deadline: Make dl_cpuset_cpumask_can_shrink() capacity-aware Date: Fri, 29 Jul 2022 13:13:04 +0200 Message-Id: <20220729111305.1275158-3-dietmar.eggemann@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729111305.1275158-1-dietmar.eggemann@arm.com> References: <20220729111305.1275158-1-dietmar.eggemann@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" dl_cpuset_cpumask_can_shrink() is used to validate whether there is still enough CPU capacity for DL tasks in the reduced cpuset. Currently it still operates on `# remaining CPUs in the cpuset` (1). Change this to use the already capacity-aware DL admission control __dl_overflow() for the `cpumask can shrink` test. dl_b->bw =3D sched_rt_period << BW_SHIFT / sched_rt_period dl_b->bw * (1) >=3D currently allocated bandwidth in root_domain (rd) Replace (1) w/ `\Sum CPU capacity in rd >> SCHED_CAPACITY_SHIFT` Adapt __dl_bw_capacity() to take a cpumask instead of a CPU number argument so that `rd->span` and `cpumask of the reduced cpuset` can be used here. Signed-off-by: Dietmar Eggemann --- kernel/sched/deadline.c | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 3f9d90b8a8b6..34de6060dea6 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -124,15 +124,12 @@ static inline int dl_bw_cpus(int i) return cpus; } =20 -static inline unsigned long __dl_bw_capacity(int i) +static inline unsigned long __dl_bw_capacity(const struct cpumask *mask) { - struct root_domain *rd =3D cpu_rq(i)->rd; unsigned long cap =3D 0; + int i; =20 - RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), - "sched RCU must be held"); - - for_each_cpu_and(i, rd->span, cpu_active_mask) + for_each_cpu_and(i, mask, cpu_active_mask) cap +=3D capacity_orig_of(i); =20 return cap; @@ -148,7 +145,10 @@ static inline unsigned long dl_bw_capacity(int i) capacity_orig_of(i) =3D=3D SCHED_CAPACITY_SCALE) { return dl_bw_cpus(i) << SCHED_CAPACITY_SHIFT; } else { - return __dl_bw_capacity(i); + RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held(), + "sched RCU must be held"); + + return __dl_bw_capacity(cpu_rq(i)->rd->span); } } =20 @@ -3004,17 +3004,15 @@ bool dl_param_changed(struct task_struct *p, const = struct sched_attr *attr) int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial) { - int ret =3D 1, trial_cpus; + unsigned long flags, cap; struct dl_bw *cur_dl_b; - unsigned long flags; + int ret =3D 1; =20 rcu_read_lock_sched(); cur_dl_b =3D dl_bw_of(cpumask_any(cur)); - trial_cpus =3D cpumask_weight(trial); - + cap =3D __dl_bw_capacity(trial); raw_spin_lock_irqsave(&cur_dl_b->lock, flags); - if (cur_dl_b->bw !=3D -1 && - cur_dl_b->bw * trial_cpus < cur_dl_b->total_bw) + if (__dl_overflow(cur_dl_b, cap, 0, 0)) ret =3D 0; raw_spin_unlock_irqrestore(&cur_dl_b->lock, flags); rcu_read_unlock_sched(); --=20 2.25.1 From nobody Wed Apr 15 00:02:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDFC3C00144 for ; Fri, 29 Jul 2022 11:13:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235951AbiG2LNk (ORCPT ); Fri, 29 Jul 2022 07:13:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235560AbiG2LNf (ORCPT ); Fri, 29 Jul 2022 07:13:35 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E88067AC0E for ; Fri, 29 Jul 2022 04:13:33 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 572501576; Fri, 29 Jul 2022 04:13:34 -0700 (PDT) Received: from localhost.localdomain (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 558AF3F73D; Fri, 29 Jul 2022 04:13:31 -0700 (PDT) From: Dietmar Eggemann To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Steven Rostedt Cc: Daniel Bristot de Oliveira , Valentin Schneider , Mel Gorman , Ben Segall , Luca Abeni , linux-kernel@vger.kernel.org Subject: [PATCH v2 3/3] sched/deadline: Use sched_dl_entity's dl_density in dl_task_fits_capacity() Date: Fri, 29 Jul 2022 13:13:05 +0200 Message-Id: <20220729111305.1275158-4-dietmar.eggemann@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729111305.1275158-1-dietmar.eggemann@arm.com> References: <20220729111305.1275158-1-dietmar.eggemann@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Save a multiplication in dl_task_fits_capacity() by using already maintained per-sched_dl_entity (i.e. per-task) `dl_runtime/dl_deadline` (dl_density). cap_scale(dl_deadline, cap) >=3D dl_runtime dl_deadline * cap >> SCHED_CAPACITY_SHIFT >=3D dl_runtime cap >=3D dl_runtime << SCHED_CAPACITY_SHIFT / dl_deadline cap >=3D (dl_runtime << BW_SHIFT / dl_deadline) >> BW_SHIFT - SCHED_CAPACITY_SHIFT cap >=3D dl_density >> BW_SHIFT - SCHED_CAPACITY_SHIFT __sched_setscheduler()->__checkparam_dl() ensures that the 2 corner cases (if conditions) `runtime =3D=3D RUNTIME_INF (-1)` and `period =3D=3D = 0` of to_ratio(deadline, runtime) are not met when setting dl_density in __sched_setscheduler()-> __setscheduler_params()->__setparam_dl(). Signed-off-by: Dietmar Eggemann --- kernel/sched/sched.h | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 72704b2b4a45..cdfe52cd8d39 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -320,21 +320,6 @@ struct dl_bw { u64 total_bw; }; =20 -/* - * Verify the fitness of task @p to run on @cpu taking into account the - * CPU original capacity and the runtime/deadline ratio of the task. - * - * The function will return true if the CPU original capacity of the - * @cpu scaled by SCHED_CAPACITY_SCALE >=3D runtime/deadline ratio of the - * task and false otherwise. - */ -static inline bool dl_task_fits_capacity(struct task_struct *p, int cpu) -{ - unsigned long cap =3D arch_scale_cpu_capacity(cpu); - - return cap_scale(p->dl.dl_deadline, cap) >=3D p->dl.dl_runtime; -} - extern void init_dl_bw(struct dl_bw *dl_b); extern int sched_dl_global_validate(void); extern void sched_dl_do_global(void); @@ -2894,6 +2879,21 @@ unsigned long effective_cpu_util(int cpu, unsigned l= ong util_cfs, enum cpu_util_type type, struct task_struct *p); =20 +/* + * Verify the fitness of task @p to run on @cpu taking into account the + * CPU original capacity and the runtime/deadline ratio of the task. + * + * The function will return true if the original capacity of @cpu is + * greater than or equal to task's deadline density right shifted by + * (BW_SHIFT - SCHED_CAPACITY_SHIFT) and false otherwise. + */ +static inline bool dl_task_fits_capacity(struct task_struct *p, int cpu) +{ + unsigned long cap =3D arch_scale_cpu_capacity(cpu); + + return cap >=3D p->dl.dl_density >> (BW_SHIFT - SCHED_CAPACITY_SHIFT); +} + static inline unsigned long cpu_bw_dl(struct rq *rq) { return (rq->dl.running_bw * SCHED_CAPACITY_SCALE) >> BW_SHIFT; --=20 2.25.1