[v1] sched/rt: Skip currently executing CPU in rto_next_cpu()

[RESEND] sched/rt: Skip currently executing CPU in rto_next_cpu()

Posted by Chen Jinghuang 2 weeks, 2 days ago

CPU0 becomes overloaded when hosting a CPU-bound RT task, a non-CPU-bound
RT task, and a CFS task stuck in kernel space. When other CPUs switch from
RT to non-RT tasks, RT load balancing (LB) is triggered; with
HAVE_RT_PUSH_IPI enabled, they send IPIs to CPU0 to drive the execution
of rto_push_irq_work_func. During push_rt_task on CPU0,
if next_task->prio < rq->donor->prio, resched_curr() sets NEED_RESCHED
and after the push operation completes, CPU0 calls rto_next_cpu().
Since only CPU0 is overloaded in this scenario, rto_next_cpu() should
ideally return -1 (no further IPI needed).

However, multiple CPUs invoking tell_cpu_to_push() during LB increments
rd->rto_loop_next. Even when rd->rto_cpu is set to -1, the mismatch between
rd->rto_loop and rd->rto_loop_next forces rto_next_cpu() to restart its
search from -1. With CPU0 remaining overloaded (satisfying rt_nr_migratory
&& rt_nr_total > 1), it gets reselected, causing CPU0 to queue irq_work to
itself and send self-IPIs repeatedly. As long as CPU0 stays overloaded and
other CPUs run pull_rt_tasks(), it falls into an infinite self-IPI loop,
which triggers a CPU hardlockup due to continuous self-interrupts.

The trigging scenario is as follows:

         cpu0                      cpu1                    cpu2
                                pull_rt_task
                              tell_cpu_to_push
                 <------------irq_work_queue_on
rto_push_irq_work_func
       push_rt_task
    resched_curr(rq)                                   pull_rt_task
    rto_next_cpu                                     tell_cpu_to_push
                      <-------------------------- atomic_inc(rto_loop_next)
rd->rto_loop != next
     rto_next_cpu
   irq_work_queue_on
rto_push_irq_work_func

Fix redundant self-IPI by filtering the initiating CPU in rto_next_cpu().
This solution has been verified to effectively eliminate spurious self-IPIs
and prevent CPU hardlockup scenarios.

Fixes: 4bdced5c9a29 ("sched/rt: Simplify the IPI based RT balancing logic")
Suggested-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Chen Jinghuang <chenjinghuang2@huawei.com>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Valentin Schneider <vschneid@redhat.com>
---
Changes since v4: https://lore.kernel.org/all/20260108070355.259530-1-chenjinghuang2@huawei.com/
- Add Reviewed-by tag from Valentin Schneider

Changes since v3: https://lore.kernel.org/all/20260105034012.3196947-1-chenjinghuang2@huawei.com/
- Replace the do-while loop with a continue statement to simplify the loop logic
- Add Suggested-by tag from K Prateek Nayak

Changes since v2: https://lore.kernel.org/all/20251125083649.1814558-1-chenjinghuang2@huawei.com/
- Replace the original "check NEED_RESCHED on target CPU"
  logic with "skip the currently executing CPU"
- This modification eliminates self-IPIS

Changes since v1: https://lore.kernel.org/all/20251121014004.564508-1-chenjinghuang2@huawei.com/
- Remove unneeded extra whitespace
- Add Reviewed-by tag from Steven Rostedt
---
 kernel/sched/rt.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index f1867fe8e5c5..e0ff90905019 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2100,6 +2100,7 @@ static void push_rt_tasks(struct rq *rq)
  */
 static int rto_next_cpu(struct root_domain *rd)
 {
+	int this_cpu = smp_processor_id();
 	int next;
 	int cpu;
 
@@ -2123,6 +2124,10 @@ static int rto_next_cpu(struct root_domain *rd)
 
 		rd->rto_cpu = cpu;
 
+		/* Do not send IPI to self */
+		if (cpu == this_cpu)
+			continue;
+
 		if (cpu < nr_cpu_ids)
 			return cpu;
 
-- 
2.34.1

Re: [RESEND] sched/rt: Skip currently executing CPU in rto_next_cpu()

Posted by Peter Zijlstra 2 weeks, 1 day ago

On Thu, Jan 22, 2026 at 01:25:33AM +0000, Chen Jinghuang wrote:
> CPU0 becomes overloaded when hosting a CPU-bound RT task, a non-CPU-bound
> RT task, and a CFS task stuck in kernel space. When other CPUs switch from
> RT to non-RT tasks, RT load balancing (LB) is triggered; with
> HAVE_RT_PUSH_IPI enabled, they send IPIs to CPU0 to drive the execution
> of rto_push_irq_work_func. During push_rt_task on CPU0,
> if next_task->prio < rq->donor->prio, resched_curr() sets NEED_RESCHED
> and after the push operation completes, CPU0 calls rto_next_cpu().
> Since only CPU0 is overloaded in this scenario, rto_next_cpu() should
> ideally return -1 (no further IPI needed).
> 
> However, multiple CPUs invoking tell_cpu_to_push() during LB increments
> rd->rto_loop_next. Even when rd->rto_cpu is set to -1, the mismatch between
> rd->rto_loop and rd->rto_loop_next forces rto_next_cpu() to restart its
> search from -1. With CPU0 remaining overloaded (satisfying rt_nr_migratory
> && rt_nr_total > 1), it gets reselected, causing CPU0 to queue irq_work to
> itself and send self-IPIs repeatedly. As long as CPU0 stays overloaded and
> other CPUs run pull_rt_tasks(), it falls into an infinite self-IPI loop,
> which triggers a CPU hardlockup due to continuous self-interrupts.
> 
> The trigging scenario is as follows:
> 
>          cpu0                      cpu1                    cpu2
>                                 pull_rt_task
>                               tell_cpu_to_push
>                  <------------irq_work_queue_on
> rto_push_irq_work_func
>        push_rt_task
>     resched_curr(rq)                                   pull_rt_task
>     rto_next_cpu                                     tell_cpu_to_push
>                       <-------------------------- atomic_inc(rto_loop_next)
> rd->rto_loop != next
>      rto_next_cpu
>    irq_work_queue_on
> rto_push_irq_work_func
> 
> Fix redundant self-IPI by filtering the initiating CPU in rto_next_cpu().
> This solution has been verified to effectively eliminate spurious self-IPIs
> and prevent CPU hardlockup scenarios.
> 
> Fixes: 4bdced5c9a29 ("sched/rt: Simplify the IPI based RT balancing logic")
> Suggested-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> Signed-off-by: Chen Jinghuang <chenjinghuang2@huawei.com>
> Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> Reviewed-by: Valentin Schneider <vschneid@redhat.com>
> ---

Thanks!

[tip: sched/core] sched/rt: Skip currently executing CPU in rto_next_cpu()

Posted by tip-bot2 for Chen Jinghuang 3 days, 22 hours ago

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     94894c9c477e53bcea052e075c53f89df3d2a33e
Gitweb:        https://git.kernel.org/tip/94894c9c477e53bcea052e075c53f89df3d2a33e
Author:        Chen Jinghuang <chenjinghuang2@huawei.com>
AuthorDate:    Thu, 22 Jan 2026 01:25:33 
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 03 Feb 2026 12:04:19 +01:00

sched/rt: Skip currently executing CPU in rto_next_cpu()

CPU0 becomes overloaded when hosting a CPU-bound RT task, a non-CPU-bound
RT task, and a CFS task stuck in kernel space. When other CPUs switch from
RT to non-RT tasks, RT load balancing (LB) is triggered; with
HAVE_RT_PUSH_IPI enabled, they send IPIs to CPU0 to drive the execution
of rto_push_irq_work_func. During push_rt_task on CPU0,
if next_task->prio < rq->donor->prio, resched_curr() sets NEED_RESCHED
and after the push operation completes, CPU0 calls rto_next_cpu().
Since only CPU0 is overloaded in this scenario, rto_next_cpu() should
ideally return -1 (no further IPI needed).

However, multiple CPUs invoking tell_cpu_to_push() during LB increments
rd->rto_loop_next. Even when rd->rto_cpu is set to -1, the mismatch between
rd->rto_loop and rd->rto_loop_next forces rto_next_cpu() to restart its
search from -1. With CPU0 remaining overloaded (satisfying rt_nr_migratory
&& rt_nr_total > 1), it gets reselected, causing CPU0 to queue irq_work to
itself and send self-IPIs repeatedly. As long as CPU0 stays overloaded and
other CPUs run pull_rt_tasks(), it falls into an infinite self-IPI loop,
which triggers a CPU hardlockup due to continuous self-interrupts.

The trigging scenario is as follows:

         cpu0                      cpu1                    cpu2
                                pull_rt_task
                              tell_cpu_to_push
                 <------------irq_work_queue_on
rto_push_irq_work_func
       push_rt_task
    resched_curr(rq)                                   pull_rt_task
    rto_next_cpu                                     tell_cpu_to_push
                      <-------------------------- atomic_inc(rto_loop_next)
rd->rto_loop != next
     rto_next_cpu
   irq_work_queue_on
rto_push_irq_work_func

Fix redundant self-IPI by filtering the initiating CPU in rto_next_cpu().
This solution has been verified to effectively eliminate spurious self-IPIs
and prevent CPU hardlockup scenarios.

Fixes: 4bdced5c9a29 ("sched/rt: Simplify the IPI based RT balancing logic")
Suggested-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Suggested-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Chen Jinghuang <chenjinghuang2@huawei.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Valentin Schneider <vschneid@redhat.com>
Link: https://patch.msgid.link/20260122012533.673768-1-chenjinghuang2@huawei.com
---
 kernel/sched/rt.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 0a9b2cd..a768047 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2106,6 +2106,7 @@ static void push_rt_tasks(struct rq *rq)
  */
 static int rto_next_cpu(struct root_domain *rd)
 {
+	int this_cpu = smp_processor_id();
 	int next;
 	int cpu;
 
@@ -2129,6 +2130,10 @@ static int rto_next_cpu(struct root_domain *rd)
 
 		rd->rto_cpu = cpu;
 
+		/* Do not send IPI to self */
+		if (cpu == this_cpu)
+			continue;
+
 		if (cpu < nr_cpu_ids)
 			return cpu;