1. Patchew
  2. linux
  3. View series

[PATCH v2] sched/rt: Skip currently executing CPU in rto_next_cpu()

Chen Jinghuang posted 1 patch 6 days, 14 hours ago
There is a newer version of this series
kernel/sched/rt.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
[PATCH v2] sched/rt: Skip currently executing CPU in rto_next_cpu()
Posted by Chen Jinghuang 6 days, 14 hours ago 
CPU0 becomes overloaded when hosting a CPU-bound RT task, a non-CPU-bound
RT task, and a CFS task stuck in kernel space. When other CPUs switch from
RT to non-RT tasks, RT load balancing (LB) is triggered; with
HAVE_RT_PUSH_IPI enabled, they send IPIs to CPU0 to drive the execution
of rto_push_irq_work_func. During push_rt_task on CPU0,
if next_task->prio < rq->donor->prio, resched_curr() sets NEED_RESCHED
and after the push operation completes, CPU0 calls rto_next_cpu().
Since only CPU0 is overloaded in this scenario, rto_next_cpu() should
ideally return -1 (no further IPI needed).

However, multiple CPUs invoking tell_cpu_to_push() during LB increments
rd->rto_loop_next. Even when rd->rto_cpu is set to -1, the mismatch between
rd->rto_loop and rd->rto_loop_next forces rto_next_cpu() to restart its
search from -1. With CPU0 remaining overloaded (satisfying rt_nr_migratory
&& rt_nr_total > 1), it gets reselected, causing CPU0 to queue irq_work to
itself and send self-IPIs repeatedly. As long as CPU0 stays overloaded and
other CPUs run pull_rt_tasks(), it falls into an infinite self-IPI loop,
wasting CPU cycles on unnecessary interrupt handling.

The triggering scenario is as follows:

         cpu0	        	   cpu1               	      cpu2
                   	        pull_rt_task
	                      tell_cpu_to_push
                 <------------irq_work_queue_on
rto_push_irq_work_func
       push_rt_task
    resched_curr(rq)                                      pull_rt_task
    rto_next_cpu                                        tell_cpu_to_push
     			 <-------------------------- atomic_inc(rto_loop_next)
rd->rto_loop != next
     rto_next_cpu
   irq_work_queue_on
rto_push_irq_work_func

Fix redundant self-IPI by filtering the initiating CPU in rto_next_cpu().

Fixes: 4bdced5c9a292 ("sched/rt: Simplify the IPI based RT balancing logic")
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Chen Jinghuang <chenjinghuang2@huawei.com>
---
v1-->v2:
	- Replace the original "check NEED_RESCHED on target CPU"
	  logic with "skip the currently executing CPU"
	- This modification eliminates self-IPIS
---
---
 kernel/sched/rt.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 7936d4333731..dc0d583aa59a 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2100,6 +2100,7 @@ static void push_rt_tasks(struct rq *rq)
  */
 static int rto_next_cpu(struct root_domain *rd)
 {
+	int this_cpu = smp_processor_id();
 	int next;
 	int cpu;
 
@@ -2118,14 +2119,17 @@ static int rto_next_cpu(struct root_domain *rd)
 	 */
 	for (;;) {
 
-		/* When rto_cpu is -1 this acts like cpumask_first() */
-		cpu = cpumask_next(rd->rto_cpu, rd->rto_mask);
-
-		rd->rto_cpu = cpu;
+		do {
+			/* When rto_cpu is -1 this acts like cpumask_first() */
+			cpu = cpumask_next(rd->rto_cpu, rd->rto_mask);
+			rd->rto_cpu = cpu;
+			/* Do not send IPI to self */
+		} while (cpu == this_cpu);
 
 		if (cpu < nr_cpu_ids)
 			return cpu;
 
+
 		rd->rto_cpu = -1;
 
 		/*
-- 
2.34.1
Re: [PATCH v2] sched/rt: Skip currently executing CPU in rto_next_cpu()
Posted by Steven Rostedt 6 days, 7 hours ago 
On Tue, 25 Nov 2025 08:36:49 +0000
Chen Jinghuang <chenjinghuang2@huawei.com> wrote:

> CPU0 becomes overloaded when hosting a CPU-bound RT task, a non-CPU-bound
> RT task, and a CFS task stuck in kernel space. When other CPUs switch from
> RT to non-RT tasks, RT load balancing (LB) is triggered; with
> HAVE_RT_PUSH_IPI enabled, they send IPIs to CPU0 to drive the execution
> of rto_push_irq_work_func. During push_rt_task on CPU0,
> if next_task->prio < rq->donor->prio, resched_curr() sets NEED_RESCHED
> and after the push operation completes, CPU0 calls rto_next_cpu().
> Since only CPU0 is overloaded in this scenario, rto_next_cpu() should
> ideally return -1 (no further IPI needed).
> 
> However, multiple CPUs invoking tell_cpu_to_push() during LB increments
> rd->rto_loop_next. Even when rd->rto_cpu is set to -1, the mismatch between
> rd->rto_loop and rd->rto_loop_next forces rto_next_cpu() to restart its
> search from -1. With CPU0 remaining overloaded (satisfying rt_nr_migratory
> && rt_nr_total > 1), it gets reselected, causing CPU0 to queue irq_work to
> itself and send self-IPIs repeatedly. As long as CPU0 stays overloaded and
> other CPUs run pull_rt_tasks(), it falls into an infinite self-IPI loop,
> wasting CPU cycles on unnecessary interrupt handling.
> 
> The triggering scenario is as follows:
> 
>          cpu0	        	   cpu1               	      cpu2
>                    	        pull_rt_task
> 	                      tell_cpu_to_push
>                  <------------irq_work_queue_on
> rto_push_irq_work_func
>        push_rt_task
>     resched_curr(rq)                                      pull_rt_task
>     rto_next_cpu                                        tell_cpu_to_push
>      			 <-------------------------- atomic_inc(rto_loop_next)
> rd->rto_loop != next
>      rto_next_cpu
>    irq_work_queue_on
> rto_push_irq_work_func
> 
> Fix redundant self-IPI by filtering the initiating CPU in rto_next_cpu().
> 
> Fixes: 4bdced5c9a292 ("sched/rt: Simplify the IPI based RT balancing logic")
> Suggested-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Chen Jinghuang <chenjinghuang2@huawei.com>
> ---
> v1-->v2:
> 	- Replace the original "check NEED_RESCHED on target CPU"
> 	  logic with "skip the currently executing CPU"
> 	- This modification eliminates self-IPIS
> ---
> ---
>  kernel/sched/rt.c | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
> index 7936d4333731..dc0d583aa59a 100644
> --- a/kernel/sched/rt.c
> +++ b/kernel/sched/rt.c
> @@ -2100,6 +2100,7 @@ static void push_rt_tasks(struct rq *rq)
>   */
>  static int rto_next_cpu(struct root_domain *rd)
>  {
> +	int this_cpu = smp_processor_id();
>  	int next;
>  	int cpu;
>  
> @@ -2118,14 +2119,17 @@ static int rto_next_cpu(struct root_domain *rd)
>  	 */
>  	for (;;) {
>  
> -		/* When rto_cpu is -1 this acts like cpumask_first() */
> -		cpu = cpumask_next(rd->rto_cpu, rd->rto_mask);
> -
> -		rd->rto_cpu = cpu;
> +		do {
> +			/* When rto_cpu is -1 this acts like cpumask_first() */
> +			cpu = cpumask_next(rd->rto_cpu, rd->rto_mask);
> +			rd->rto_cpu = cpu;
> +			/* Do not send IPI to self */
> +		} while (cpu == this_cpu);

So this fixes the issue you see too! Great!

>  
>  		if (cpu < nr_cpu_ids)
>  			return cpu;
>  
> +

Unneeded extra whitespace.

Other than that:

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

-- Steve

>  		rd->rto_cpu = -1;
>  
>  		/*
[PATCH v3] sched/rt: Skip currently executing CPU in rto_next_cpu()
Posted by Chen Jinghuang 5 days, 17 hours ago 
CPU0 becomes overloaded when hosting a CPU-bound RT task, a non-CPU-bound
RT task, and a CFS task stuck in kernel space. When other CPUs switch from
RT to non-RT tasks, RT load balancing (LB) is triggered; with
HAVE_RT_PUSH_IPI enabled, they send IPIs to CPU0 to drive the execution
of rto_push_irq_work_func. During push_rt_task on CPU0,
if next_task->prio < rq->donor->prio, resched_curr() sets NEED_RESCHED
and after the push operation completes, CPU0 calls rto_next_cpu().
Since only CPU0 is overloaded in this scenario, rto_next_cpu() should
ideally return -1 (no further IPI needed).

However, multiple CPUs invoking tell_cpu_to_push() during LB increments
rd->rto_loop_next. Even when rd->rto_cpu is set to -1, the mismatch between
rd->rto_loop and rd->rto_loop_next forces rto_next_cpu() to restart its
search from -1. With CPU0 remaining overloaded (satisfying rt_nr_migratory
&& rt_nr_total > 1), it gets reselected, causing CPU0 to queue irq_work to
itself and send self-IPIs repeatedly. As long as CPU0 stays overloaded and
other CPUs run pull_rt_tasks(), it falls into an infinite self-IPI loop,
wasting CPU cycles on unnecessary interrupt handling.

The triggering scenario is as follows:

         cpu0	        	   cpu1               	      cpu2
                   	        pull_rt_task
	                      tell_cpu_to_push
                 <------------irq_work_queue_on
rto_push_irq_work_func
       push_rt_task
    resched_curr(rq)                                      pull_rt_task
    rto_next_cpu                                        tell_cpu_to_push
     			 <-------------------------- atomic_inc(rto_loop_next)
rd->rto_loop != next
     rto_next_cpu
   irq_work_queue_on
rto_push_irq_work_func

Fix redundant self-IPI by filtering the initiating CPU in rto_next_cpu().

Fixes: 4bdced5c9a292 ("sched/rt: Simplify the IPI based RT balancing logic")
Suggested-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Chen Jinghuang <chenjinghuang2@huawei.com>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
v1-->v2:
- Replace the original "check NEED_RESCHED on target CPU"
  logic with "skip the currently executing CPU"
- This modification eliminates self-IPIS
v2-->v3:
- Remove unneeded extra whitespace
- Add Reviewed-by tag from Steven Rostedt
---
---
 kernel/sched/rt.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 7936d4333731..f6af41e27508 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2100,6 +2100,7 @@ static void push_rt_tasks(struct rq *rq)
  */
 static int rto_next_cpu(struct root_domain *rd)
 {
+	int this_cpu = smp_processor_id();
 	int next;
 	int cpu;
 
@@ -2118,10 +2119,12 @@ static int rto_next_cpu(struct root_domain *rd)
 	 */
 	for (;;) {
 
-		/* When rto_cpu is -1 this acts like cpumask_first() */
-		cpu = cpumask_next(rd->rto_cpu, rd->rto_mask);
-
-		rd->rto_cpu = cpu;
+		do {
+			/* When rto_cpu is -1 this acts like cpumask_first() */
+			cpu = cpumask_next(rd->rto_cpu, rd->rto_mask);
+			rd->rto_cpu = cpu;
+			/* Do not send IPI to self */
+		} while (cpu == this_cpu);
 
 		if (cpu < nr_cpu_ids)
 			return cpu;
-- 
2.34.1

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.