Task's "blocked_on" relationship can be cleared remotely without
acquiring the task's runqueue lock and can lead to a blocked donor being
picked to run without having a wakeup.
Although this is not a problem since the task blocks back again if it
wasn't woken up in time, it cause problem from stats acccounting point,
especially for PSI since psi_enqueue() tries to reset the runnable
signals for a proxy task that wasn't fully block, and psi_sched_switch()
assumes a blocking task is always runnable before being switched out.
When investigating PSI task state corruption, the following transitions
were observed with CONFIG_SCHED_PROXY_EXEC=y:
... [141] ...: psi_flags_change: task=4358:... cpu=141 psi_flags=14 clear=14 set=0 queued=1 delayed=0 blocked=1 psi_bug=0
# Task is retained on the rq for proxy. All runnable signals are
# cleared for the task.
... [141] ...: psi_flags_change: task=4358:... cpu=141 psi_flags=0 clear=0 set=10 queued=1 delayed=0 blocked=0 psi_bug=0
# Task is picked and forced to run since task_is_blocked() returns
# false.
... [141] ...: psi_dequeue: task=4358:... cpu=141 psi_flags=10
# Task blocks again. Flag modifications are deferred to
# psi_sched_switch() since DEQUEUE_SLEEP.
... [141] ...: psi_flags_change: task=4358:... cpu=141 psi_flags=10 clear=14 set=0 queued=1 delayed=1 blocked=0 psi_bug=(0 -> 1)
# psi_sched_switch() tries to clear TSK_ONCPU and TSK_RUNNING.
# TSK_RUNNING was never set since task was never woken up.
# !! PSI inconsistent state waring triggered !!
... [014] ...: try_to_wake_up: wakeup: task=4358: psi_flags=0 queued=0 delayed=0 blocked=0
# Task wakes up and is finally runnable.
To prevent any inconsistencies from running a task that was supposed to
be blocked, deactivate a potential donor task if it was picked without
having the "sched_proxy" indicator cleared. A pending wakeup would queue
the task back again when it turns runnable again.
Fixes: be41bde4c3a8 ("sched: Add an initial sketch of the find_proxy_task() function")
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
kernel/sched/core.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 52a744beeca9..d6265f38e93a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6679,6 +6679,7 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
bool is_proxy_task(struct task_struct *p) { return false; }
static inline void set_task_proxy(struct task_struct *p) { }
static inline void clear_task_proxy(p) { }
+static bool __proxy_deactivate(struct rq *rq, struct task_struct *donor) { return false; }
static struct task_struct *
find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf)
{
@@ -6839,6 +6840,25 @@ static void __sched notrace __schedule(int sched_mode)
if (next == rq->idle)
goto keep_resched;
}
+ if (unlikely(is_proxy_task(next))) {
+ /*
+ * It is possible for a remote CPU to clear task
+ * "blocked_on" without acquiring the task rq lock.
+ *
+ * This can lead to a blocked task retained for proxy to
+ * be forced on CPU without the task being woken up
+ * since task_is_blocked(next) above returns false.
+ *
+ * Since "sched_proxy" is only cleared on wakeup,
+ * is_proxy_task() returning true indicates that the
+ * task hasn't woken up yet.
+ *
+ * Block the task and wait for wakeup to queue it back
+ * when it is runnable again.
+ */
+ if (__proxy_deactivate(rq, next))
+ goto pick_again;
+ }
picked:
clear_tsk_need_resched(prev);
clear_preempt_need_resched();
--
2.34.1