[PATCH v5 00/14] Add a deadline server for sched_ext tasks

Joel Fernandes posted 14 patches 3 months, 2 weeks ago
There is a newer version of this series
include/linux/sched.h                         |   2 +-
kernel/sched/core.c                           |  19 +-
kernel/sched/deadline.c                       |  84 +++--
kernel/sched/debug.c                          | 171 +++++++++--
kernel/sched/ext.c                            | 120 +++++++-
kernel/sched/fair.c                           |  15 +-
kernel/sched/idle.c                           |   4 +-
kernel/sched/rt.c                             |   2 +-
kernel/sched/sched.h                          |  13 +-
kernel/sched/stop_task.c                      |   2 +-
tools/testing/selftests/sched_ext/Makefile    |   2 +
.../selftests/sched_ext/rt_stall.bpf.c        |  23 ++
tools/testing/selftests/sched_ext/rt_stall.c  | 213 +++++++++++++
tools/testing/selftests/sched_ext/total_bw.c  | 286 ++++++++++++++++++
14 files changed, 872 insertions(+), 84 deletions(-)
create mode 100644 tools/testing/selftests/sched_ext/rt_stall.bpf.c
create mode 100644 tools/testing/selftests/sched_ext/rt_stall.c
create mode 100644 tools/testing/selftests/sched_ext/total_bw.c
[PATCH v5 00/14] Add a deadline server for sched_ext tasks
Posted by Joel Fernandes 3 months, 2 weeks ago
sched_ext tasks currently are starved by RT hoggers especially since RT
throttling was replaced by deadline servers to boost only CFS tasks. Several
users in the community have reported issues with RT stalling sched_ext tasks.
Add a sched_ext deadline server as well so that sched_ext tasks are also
boosted and do not suffer starvation.

A kselftest is also provided to verify the starvation issues are now fixed.

Btw, there is still something funky going on with CPU hotplug and the
relinquish patch. Sometimes the sched_ext's hotplug self-test locks up
(./runner -t hotplug). Reverting that patch fixes it, so I am suspecting
something is off in dl_server_remove_params() when it is being called on
offline CPUs.

v4->v5:
-  Added a kselftest (total_bw) to sched_ext to verify bandwidth values
   from debugfs.
- Address comment from Andrea about redundant rq clock invalidation.

v3->v4:
 - Fixed issues with hotplugged CPUs having their DL server bandwidth
   altered due to loading SCX.
 - Fixed other issues.
 - Rebased on Linus master.
 - All sched_ext kselftests reliably pass now, also verified that
   the total_bw in debugfs (CONFIG_SCHED_DEBUG) is conserved with
   these patches.

v2->v3:
 - Removed code duplication in debugfs. Made ext interface separate.
 - Fixed issue where rq_lock_irqsave was not used in the relinquish patch.
 - Fixed running bw accounting issue in dl_server_remove_params.

Link to v1: https://lore.kernel.org/all/20250315022158.2354454-1-joelagnelf@nvidia.com/
Link to v2: https://lore.kernel.org/all/20250602180110.816225-1-joelagnelf@nvidia.com/
Link to v3: https://lore.kernel.org/all/20250613051734.4023260-1-joelagnelf@nvidia.com/
Link to v4: https://lore.kernel.org/all/20250617200523.1261231-1-joelagnelf@nvidia.com/

Andrea Righi (2):
  sched/deadline: Add support to remove DLserver's bandwidth
    contribution
  selftests/sched_ext: Add test for sched_ext dl_server

Joel Fernandes (12):
  sched/debug: Fix updating of ppos on server write ops
  sched/debug: Stop and start server based on if it was active
  sched/deadline: Clear the defer params
  sched/deadline: Prevent setting server as started if params couldn't
    be applied
  sched/deadline: Return EBUSY if dl_bw_cpus is zero
  sched: Add support to pick functions to take rf
  sched: Add a server arg to dl_server_update_idle_time()
  sched/ext: Add a DL server for sched_ext tasks
  sched/debug: Add support to change sched_ext server params
  sched/ext: Relinquish DL server reservations when not needed
  sched/deadline: Fix DL server crash in inactive_timer callback
  selftests/sched_ext: Add test for DL server total_bw consistency

 include/linux/sched.h                         |   2 +-
 kernel/sched/core.c                           |  19 +-
 kernel/sched/deadline.c                       |  84 +++--
 kernel/sched/debug.c                          | 171 +++++++++--
 kernel/sched/ext.c                            | 120 +++++++-
 kernel/sched/fair.c                           |  15 +-
 kernel/sched/idle.c                           |   4 +-
 kernel/sched/rt.c                             |   2 +-
 kernel/sched/sched.h                          |  13 +-
 kernel/sched/stop_task.c                      |   2 +-
 tools/testing/selftests/sched_ext/Makefile    |   2 +
 .../selftests/sched_ext/rt_stall.bpf.c        |  23 ++
 tools/testing/selftests/sched_ext/rt_stall.c  | 213 +++++++++++++
 tools/testing/selftests/sched_ext/total_bw.c  | 286 ++++++++++++++++++
 14 files changed, 872 insertions(+), 84 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/rt_stall.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/rt_stall.c
 create mode 100644 tools/testing/selftests/sched_ext/total_bw.c

-- 
2.43.0
Re: [PATCH v5 00/14] Add a deadline server for sched_ext tasks
Posted by Tejun Heo 3 months, 2 weeks ago
On Fri, Jun 20, 2025 at 04:32:15PM -0400, Joel Fernandes wrote:
> sched_ext tasks currently are starved by RT hoggers especially since RT
> throttling was replaced by deadline servers to boost only CFS tasks. Several
> users in the community have reported issues with RT stalling sched_ext tasks.
> Add a sched_ext deadline server as well so that sched_ext tasks are also
> boosted and do not suffer starvation.

I left some minor comments and had a question about adding @rf to
->pick_task() but the patchset generally looks great to me otherwise.

Thanks.

-- 
tejun
Re: [PATCH v5 00/14] Add a deadline server for sched_ext tasks
Posted by Joel Fernandes 3 months, 2 weeks ago

On 6/20/2025 4:32 PM, Joel Fernandes wrote:
> sched_ext tasks currently are starved by RT hoggers especially since RT
> throttling was replaced by deadline servers to boost only CFS tasks. Several
> users in the community have reported issues with RT stalling sched_ext tasks.
> Add a sched_ext deadline server as well so that sched_ext tasks are also
> boosted and do not suffer starvation.
> 
> A kselftest is also provided to verify the starvation issues are now fixed.
> 
> Btw, there is still something funky going on with CPU hotplug and the
> relinquish patch. Sometimes the sched_ext's hotplug self-test locks up
> (./runner -t hotplug). Reverting that patch fixes it, so I am suspecting
> something is off in dl_server_remove_params() when it is being called on
> offline CPUs.

Sorry that I forgot to delete this last paragraph. The hotplug issue is
completely fixed now (since v4).

There are no more open issues and all tests on mine and Andrea's side are
looking good now.

Thanks.