From: Peter Oskolkov <posk@google.com>

Add WF_CURRENT_CPU wake flag that advices the scheduler to
move the wakee to the current CPU. This is useful for fast on-CPU
context switching use cases such as UMCG.

In addition, make ttwu external rather than static so that
the flag could be passed to it from outside of sched/core.c.

Signed-off-by: Peter Oskolkov <posk@google.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
---
 kernel/sched/core.c  |  3 +--
 kernel/sched/fair.c  |  4 ++++
 kernel/sched/sched.h | 13 ++++++++-----
 3 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5800b0623ff3..cffa8f314c9a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4039,8 +4039,7 @@ bool ttwu_state_match(struct task_struct *p, unsigned int state, int *success)
  * Return: %true if @p->state changes (an actual wakeup was done),
  *	   %false otherwise.
  */
-static int
-try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
+int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 {
 	unsigned long flags;
 	int cpu, success = 0;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e4a0b8bd941c..4ebe7222664c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7204,6 +7204,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	if (wake_flags & WF_TTWU) {
 		record_wakee(p);
 
+		if ((wake_flags & WF_CURRENT_CPU) &&
+		    cpumask_test_cpu(cpu, p->cpus_ptr))
+			return cpu;
+
 		if (sched_energy_enabled()) {
 			new_cpu = find_energy_efficient_cpu(p, prev_cpu);
 			if (new_cpu >= 0)
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1644242ecd11..ee24141c4942 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2071,12 +2071,13 @@ static inline int task_on_rq_migrating(struct task_struct *p)
 }
 
 /* Wake flags. The first three directly map to some SD flag value */
-#define WF_EXEC     0x02 /* Wakeup after exec; maps to SD_BALANCE_EXEC */
-#define WF_FORK     0x04 /* Wakeup after fork; maps to SD_BALANCE_FORK */
-#define WF_TTWU     0x08 /* Wakeup;            maps to SD_BALANCE_WAKE */
+#define WF_EXEC         0x02 /* Wakeup after exec; maps to SD_BALANCE_EXEC */
+#define WF_FORK         0x04 /* Wakeup after fork; maps to SD_BALANCE_FORK */
+#define WF_TTWU         0x08 /* Wakeup;            maps to SD_BALANCE_WAKE */
 
-#define WF_SYNC     0x10 /* Waker goes to sleep after wakeup */
-#define WF_MIGRATED 0x20 /* Internal use, task got migrated */
+#define WF_SYNC         0x10 /* Waker goes to sleep after wakeup */
+#define WF_MIGRATED     0x20 /* Internal use, task got migrated */
+#define WF_CURRENT_CPU  0x40 /* Prefer to move the wakee to the current CPU. */
 
 #ifdef CONFIG_SMP
 static_assert(WF_EXEC == SD_BALANCE_EXEC);
@@ -3161,6 +3162,8 @@ static inline bool is_per_cpu_kthread(struct task_struct *p)
 extern void swake_up_all_locked(struct swait_queue_head *q);
 extern void __prepare_to_swait(struct swait_queue_head *q, struct swait_queue *wait);
 
+extern int try_to_wake_up(struct task_struct *tsk, unsigned int state, int wake_flags);
+
 #ifdef CONFIG_PREEMPT_DYNAMIC
 extern int preempt_dynamic_mode;
 extern int sched_dynamic_mode(const char *str);
-- 
2.37.2

On October 19, 2022 6:10:45 PM PDT, Andrei Vagin <avagin@gmail.com> wrote:
>From: Peter Oskolkov <posk@google.com>
>
>Add WF_CURRENT_CPU wake flag that advices the scheduler to
>move the wakee to the current CPU. This is useful for fast on-CPU
>context switching use cases such as UMCG.

UMCG is https://lwn.net/Articles/879398/ ?

>In addition, make ttwu external rather than static so that
>the flag could be passed to it from outside of sched/core.c.
>
>Signed-off-by: Peter Oskolkov <posk@google.com>
>Signed-off-by: Andrei Vagin <avagin@gmail.com>
>---
> kernel/sched/core.c  |  3 +--
> kernel/sched/fair.c  |  4 ++++
> kernel/sched/sched.h | 13 ++++++++-----
> 3 files changed, 13 insertions(+), 7 deletions(-)

This would need an Ack from the sched maintainers...


-- 
Kees Cook

On Wed, Oct 19, 2022 at 09:54:15PM -0700, Kees Cook wrote:
> On October 19, 2022 6:10:45 PM PDT, Andrei Vagin <avagin@gmail.com> wrote:
> >From: Peter Oskolkov <posk@google.com>
> >
> >Add WF_CURRENT_CPU wake flag that advices the scheduler to
> >move the wakee to the current CPU. This is useful for fast on-CPU
> >context switching use cases such as UMCG.
> 
> UMCG is https://lwn.net/Articles/879398/ ?
> 

Yes, this is it. https://lkml.org/lkml/2021/11/4/830 is the most recent
version that I've seen.

> >In addition, make ttwu external rather than static so that
> >the flag could be passed to it from outside of sched/core.c.
> >
> >Signed-off-by: Peter Oskolkov <posk@google.com>
> >Signed-off-by: Andrei Vagin <avagin@gmail.com>
> >---
> > kernel/sched/core.c  |  3 +--
> > kernel/sched/fair.c  |  4 ++++
> > kernel/sched/sched.h | 13 ++++++++-----
> > 3 files changed, 13 insertions(+), 7 deletions(-)
> 
> This would need an Ack from the sched maintainers...