From nobody Sat Jul 25 20:47:24 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACD6F356747 for ; Mon, 13 Jul 2026 19:46:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783971978; cv=none; b=B7NSepOUF51qyBJQYqX8eSTknflmY18qGGpOQ6OlMDZZfhOch9HbINUPqFyOHYz+4ikZIG+IpmZsQORC5wo1axNpjMvqOKdqKttzD6RihtHbYuqqY2QLaxRMTvuPlXcVzPvQTlq9tYyVuDnSNv1Xvp9d97PxKnhWyAv3J5iJ6hE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783971978; c=relaxed/simple; bh=qCNUoqcUcw7avE7Mbd/YBIuA9CB7KU8dqhvUBBQvXm8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=DWoK9JhxZI8FQVM2BSNHue5D8F4GDREPqXosMlEM8861CCdjhhFaD0EHj2Kfe+wk4gPtF1YXt2OivOpTVUWiJX8+1wrV7KslWrKHoUMfHUjq9JYPT/NGN3slGFA87CTAdbJB7MEMs/gD6Br/Xpk/MQXEpl9La7iyJ0NyHpajmPo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=EewENQrW; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="EewENQrW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1783971975; bh=qCNUoqcUcw7avE7Mbd/YBIuA9CB7KU8dqhvUBBQvXm8=; h=From:Date:Subject:To:Cc:From; b=EewENQrWsmvhVtJqeNOclV6nE1wexNX8DpRpyaFbfOJ6HkCd/sgRnv7/U4ksDEP7P zozSVWxDI2qzEuM9Z+We49O5XH/ZAdaAwi/TpLL0hABsH/GSgOzN9lqRIyuPTbmVvq TDQkXdB4sP2lZy5UE5ZUwUcDOaKfM5ThY4S6A1ts= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id A0229406E3; Mon, 13 Jul 2026 12:46:15 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Mon, 13 Jul 2026 12:46:10 -0700 Subject: [PATCH v2] sched/core: Preserve wake flags across queued wakeups Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260713-b4-sched-ttwu-wake-flags-v2-1-76e23a8cc313@gentwo.org> X-B4-Tracking: v=1; b=H4sIAIFAVWoC/42NSw7CIBRFt9K8sc8A/dA4ch+mA2xfgWhKA0g1D XsXuwKH5yT33B0CeUsBLtUOnpIN1i0FxKmC0ahFE9qpMAgmOiY5w3uDYTQ0YYzbCzf1IJyfSge kXqpJMepa4lDmq6fZvo/0bShsbIjOf46nxH/2j2jiyFHUvBeqGdu6k1dNS9zc2XkNQ875CwAXn J7BAAAA X-Change-ID: 20260710-b4-sched-ttwu-wake-flags-e87ada0e65e1 To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak Cc: Christoph Lameter , Shubhang Kaushik , linux-kernel@vger.kernel.org, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1783971975; l=4977; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=qCNUoqcUcw7avE7Mbd/YBIuA9CB7KU8dqhvUBBQvXm8=; b=ajvQtQuhfkvEPbC1iNnkf2lGY+vmsXub5L1wRbV3izXAi5aJjws09qAXyEAjO4LTRaOs/Hbj7 33nR0lUBsinA3AVCSH369e/ApuUz2ltpVCAhKUcgJR41jiY3uCVewrs X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= The queued wakeup path only remembers whether the wakee migrated. When the target CPU drains the wakelist, sched_ttwu_pending() turns that back into either WF_MIGRATED or 0 and passes it to ttwu_do_activate(). That drops the rest of the wake flags seen by the direct wakeup path. Some of those flags can still matter when the queued wakeup is drained on a non-idle target CPU, so queued and direct wakeups can make different preemption decisions. Store the relevant wake_flags bits instead of a single boolean and pass them through when the wakelist is drained. Keep WF_CURRENT_CPU out of the saved mask as it is a placement hint for CPU selection, not an activation flag. Signed-off-by: Shubhang Kaushik (Ampere) --- This aims to preserve activation-relevant wake flags across queued remote wakeups, keeping the queued and direct wakeup paths consistent when calling ttwu_do_activate(). Tested on an Ampere Altra 80 CPU system with perf bench sched messaging and hackbench. Results were within run-to-run noise of the unpatched mainline baseline. Baseline: mainline origin/master at d96fcfe1b7f9. --- Changes in v2: - Move sched_remote_wakeup_flags to a standalone u8 outside the scheduler bitfields. - Drop the unnecessary reset in sched_ttwu_pending(). - Add WF_TTWU_QUEUE_MASK for the wake flags preserved across the wakelist. - Tighten the changelog around direct-vs-queued wakeup consistency. Link to v1: https://lore.kernel.org/r/20260710-b4-sched-ttwu-wake-flags-v1-= 1-23182a4c5367@gentwo.org --- include/linux/sched.h | 17 +---------------- kernel/sched/core.c | 7 +++++-- kernel/sched/sched.h | 1 + 3 files changed, 7 insertions(+), 18 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d10b4256a11f8c8373ece78fa5e0e5..a2064476a6bf84a2513c9aad364= 4f88cbc09a298 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -983,6 +983,7 @@ struct task_struct { =20 /* Used for emulating ABI behavior of previous Linux versions: */ unsigned int personality; + u8 sched_remote_wakeup_flags; =20 /* Scheduler bits, serialized by scheduler locks: */ unsigned sched_reset_on_fork:1; @@ -993,22 +994,6 @@ struct task_struct { /* Force alignment to the next boundary: */ unsigned :0; =20 - /* Unserialized, strictly 'current' */ - - /* - * This field must not be in the scheduler word above due to wakelist - * queueing no longer being serialized by p->on_cpu. However: - * - * p->XXX =3D X; ttwu() - * schedule() if (p->on_rq && ..) // false - * smp_mb__after_spinlock(); if (smp_load_acquire(&p->on_cpu) && //tr= ue - * deactivate_task() ttwu_queue_wakelist()) - * p->on_rq =3D 0; p->sched_remote_wakeup =3D Y; - * - * guarantees all stores of 'current' are visible before - * ->sched_remote_wakeup gets used, so it can be in this word. - */ - unsigned sched_remote_wakeup:1; #ifdef CONFIG_RT_MUTEXES unsigned sched_rt_mutex:1; #endif diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f6135341aa779b8262f113e103d8ad..62708634259f0cf870897fd5bfe= 5c15ff88e40d0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3902,13 +3902,16 @@ void sched_ttwu_pending(void *arg) update_rq_clock(rq); =20 llist_for_each_entry_safe(p, t, llist, wake_entry.llist) { + int wake_flags; + if (WARN_ON_ONCE(p->on_cpu)) smp_cond_load_acquire(&p->on_cpu, !VAL); =20 if (WARN_ON_ONCE(task_cpu(p) !=3D cpu_of(rq))) set_task_cpu(p, cpu_of(rq)); =20 - ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); + wake_flags =3D p->sched_remote_wakeup_flags; + ttwu_do_activate(rq, p, wake_flags, &rf); } =20 /* @@ -3951,7 +3954,7 @@ static void __ttwu_queue_wakelist(struct task_struct = *p, int cpu, int wake_flags { struct rq *rq =3D cpu_rq(cpu); =20 - p->sched_remote_wakeup =3D !!(wake_flags & WF_MIGRATED); + p->sched_remote_wakeup_flags =3D wake_flags & WF_TTWU_QUEUE_MASK; =20 WRITE_ONCE(rq->ttwu_pending, 1); #ifdef CONFIG_SMP diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56acf502ba260ab18bacd7a4c2efdec612d50125..50cd0141aeb49af4beb1af51862= e319f096ec5f2 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2494,6 +2494,7 @@ static inline int task_on_rq_migrating(struct task_st= ruct *p) #define WF_MIGRATED 0x20 /* Internal use, task got migrated */ #define WF_CURRENT_CPU 0x40 /* Prefer to move the wakee to the current CP= U. */ #define WF_RQ_SELECTED 0x80 /* ->select_task_rq() was called */ +#define WF_TTWU_QUEUE_MASK (WF_TTWU | WF_SYNC | WF_MIGRATED | WF_RQ_SELECT= ED) =20 static_assert(WF_EXEC =3D=3D SD_BALANCE_EXEC); static_assert(WF_FORK =3D=3D SD_BALANCE_FORK); --- base-commit: d96fcfe1b7f94ac742984ae7986b94a116abff1b change-id: 20260710-b4-sched-ttwu-wake-flags-e87ada0e65e1 Best regards, --=20 Shubhang Kaushik (Ampere)