From nobody Fri Jul 24 23:32:20 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5F083624DB for ; Wed, 22 Jul 2026 07:06:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784704007; cv=none; b=FWhHnD1Knq8MkFd+wV3wbbMmcCg3XWQohjUuJQhf42RghoZgafUro9JR70+4tW36ZiJh6wlr55WvwCUWm3Is2pMWrpnyB90r6RoY2mpy7ogEAr0U7uf7RINsy+qnnhvk0dU8xo6BIQ3HAJyATSyVDcwRGv5G4y2wWohDnxzjBko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784704007; c=relaxed/simple; bh=dVdhBhJEBTgrzuPmKjrigRRdyAtJ7V/Z/Mip0Ypuywo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=QacuQtd4zecmWQd72j1719qejm18xZMX1AknXlzDL2g4SMGadg4TWiigGzrjGMoj21ExDeL90LPRtm2yFxkG6W3Iw9CujWR0jVkex8l35ATW6ETl3QH8TE+oZ06WEPaPKKPu5iKxwK/qNIQENCMD1SgnK/aqKIMoQueVIJHlTc0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=f4xItE0j; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="f4xItE0j" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1784703295; bh=dVdhBhJEBTgrzuPmKjrigRRdyAtJ7V/Z/Mip0Ypuywo=; h=From:Date:Subject:To:Cc:From; b=f4xItE0jmYBmyAvXrgFsGwtEF8w/3H+nqYrGCC2IEa7rRohZsUyHDyzMHkwdFQMX2 z6AIn1ifAGI3CzEzUh0IfDRa4qjd5eLGnsHAQvCBJ5IYcs0JNpkQCkZaE2KsxE/lal F6INs9RI3g4oqXNLB9Ir6m/tmV4HbfJRV8vbh5SY= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id F195B40287; Tue, 21 Jul 2026 23:54:54 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Tue, 21 Jul 2026 23:54:52 -0700 Subject: [PATCH v3] sched/core: Preserve wake flags across queued wakeups Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260721-b4-sched-ttwu-wake-flags-v3-1-07b92122bf28@gentwo.org> X-B4-Tracking: v=1; b=H4sIADtpYGoC/43NTQ7CIBCG4as0rB3DTwuNK+9hXCCdUqIpBpBqm t5d2pVujMv3S+aZmUQMDiM5VDMJmF10fiwhdhUxgx4tgutKE065pIpRuNQQzYAdpDQ9YNJXhP6 mbQRsle40RdkgI+X8HrB3z40+nUsPLiYfXtunzNb1DzQzYMAFa7muTSOkOloc0+T3Pliyqpl/S uKHxIukJHKhW2MEE1/Ssixv+HTJVgsBAAA= X-Change-ID: 20260710-b4-sched-ttwu-wake-flags-e87ada0e65e1 To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak Cc: Christoph Lameter , Shubhang Kaushik , linux-kernel@vger.kernel.org, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1784703294; l=5337; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=dVdhBhJEBTgrzuPmKjrigRRdyAtJ7V/Z/Mip0Ypuywo=; b=hJXdBP3Zso3NDtuG8jaJQPr6lSFtdi4kkIDHlUrgj5t71mrdFB59y89KM1KPSkKP9h+bmQy0F 4ECjr8N9MsfATpl/JgygSI0bWFbIMbmrtAha82h2eMvy0N6DHCZ1+Wu X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= Queued wakeups currently save only whether the wakee migrated. When the target CPU drains the wakelist, sched_ttwu_pending() therefore passes only WF_MIGRATED or 0 to ttwu_do_activate(). That loses flags which are still used after CPU selection. For example, WF_RQ_SELECTED is set after select_task_rq() chooses a runqueue, then used by ttwu_do_activate() for ENQUEUE_RQ_SELECTED and by wakeup_preempt_fair() for the preemption threshold: direct wakeup: select_task_rq() -> WF_RQ_SELECTED ttwu_do_activate(WF_RQ_SELECTED) wakeup_preempt_fair(WF_RQ_SELECTED) queued wakeup: select_task_rq() -> WF_RQ_SELECTED ttwu_queue_wakelist() -> save WF_MIGRATED only sched_ttwu_pending() -> ttwu_do_activate(WF_MIGRATED or 0) Preserve the wake flags that still matter after queueing: WF_TTWU, WF_SYNC, WF_MIGRATED and WF_RQ_SELECTED. Do not save WF_CURRENT_CPU, which is only a CPU-selection hint. Signed-off-by: Shubhang Kaushik (Ampere) Tested-by: K Prateek Nayak --- Tested on an 80 CPU Ampere Altra system with perf bench sched messaging, perf bench sched pipe, hackbench and SPECjBB. No material regression was observed. Baseline: mainline origin/master at v7.2-rc4 (248951ddc14d) --- Changes in v3: - Use READ_ONCE()/WRITE_ONCE() for sched_remote_wakeup_flags. - Anchor the changelog on the WF_RQ_SELECTED preemption behavior. - Keep the cross-CPU ordering comment for the task field. Link to v2: https://lore.kernel.org/r/20260713-b4-sched-ttwu-wake-flags-v2-= 1-76e23a8cc313@gentwo.org Changes in v2: - Move sched_remote_wakeup_flags to a standalone u8 outside the scheduler bitfields. - Drop the unnecessary reset in sched_ttwu_pending(). - Add WF_TTWU_QUEUE_MASK for the wake flags preserved across the wakelist. - Tighten the changelog around direct-vs-queued wakeup consistency. Link to v1: https://lore.kernel.org/r/20260710-b4-sched-ttwu-wake-flags-v1-= 1-23182a4c5367@gentwo.org --- include/linux/sched.h | 26 ++++++++++---------------- kernel/sched/core.c | 10 ++++++++-- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d10b4256a11f8c8373ece78fa5e0e5..c0de2ea02eeeb3ad9ca1a090260= b00e1b35cbc39 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -983,6 +983,16 @@ struct task_struct { =20 /* Used for emulating ABI behavior of previous Linux versions: */ unsigned int personality; + /* + * Must not share the scheduler bitfield word because wakelist + * queueing is not serialized by p->on_cpu. + * + * smp_load_acquire(&p->on_cpu) in ttwu_queue_wakelist() pairs + * with smp_mb__after_spinlock() in schedule(), making current's + * stores visible before the target CPU uses these flags from + * sched_ttwu_pending(). + */ + u8 sched_remote_wakeup_flags; =20 /* Scheduler bits, serialized by scheduler locks: */ unsigned sched_reset_on_fork:1; @@ -993,22 +1003,6 @@ struct task_struct { /* Force alignment to the next boundary: */ unsigned :0; =20 - /* Unserialized, strictly 'current' */ - - /* - * This field must not be in the scheduler word above due to wakelist - * queueing no longer being serialized by p->on_cpu. However: - * - * p->XXX =3D X; ttwu() - * schedule() if (p->on_rq && ..) // false - * smp_mb__after_spinlock(); if (smp_load_acquire(&p->on_cpu) && //tr= ue - * deactivate_task() ttwu_queue_wakelist()) - * p->on_rq =3D 0; p->sched_remote_wakeup =3D Y; - * - * guarantees all stores of 'current' are visible before - * ->sched_remote_wakeup gets used, so it can be in this word. - */ - unsigned sched_remote_wakeup:1; #ifdef CONFIG_RT_MUTEXES unsigned sched_rt_mutex:1; #endif diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f6135341aa779b8262f113e103d8ad..7ddcc6a7f1c06675ce8084cd304= 475d61f869eb7 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3902,13 +3902,16 @@ void sched_ttwu_pending(void *arg) update_rq_clock(rq); =20 llist_for_each_entry_safe(p, t, llist, wake_entry.llist) { + int wake_flags; + if (WARN_ON_ONCE(p->on_cpu)) smp_cond_load_acquire(&p->on_cpu, !VAL); =20 if (WARN_ON_ONCE(task_cpu(p) !=3D cpu_of(rq))) set_task_cpu(p, cpu_of(rq)); =20 - ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); + wake_flags =3D READ_ONCE(p->sched_remote_wakeup_flags); + ttwu_do_activate(rq, p, wake_flags, &rf); } =20 /* @@ -3947,11 +3950,14 @@ bool call_function_single_prep_ipi(int cpu) * via sched_ttwu_wakeup() for activation so the wakee incurs the cost * of the wakeup instead of the waker. */ +#define WF_TTWU_QUEUE_MASK (WF_TTWU | WF_SYNC | WF_MIGRATED | \ + WF_RQ_SELECTED) static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake= _flags) { struct rq *rq =3D cpu_rq(cpu); =20 - p->sched_remote_wakeup =3D !!(wake_flags & WF_MIGRATED); + WRITE_ONCE(p->sched_remote_wakeup_flags, + wake_flags & WF_TTWU_QUEUE_MASK); =20 WRITE_ONCE(rq->ttwu_pending, 1); #ifdef CONFIG_SMP --- base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df change-id: 20260710-b4-sched-ttwu-wake-flags-e87ada0e65e1 Best regards, --=20 Shubhang Kaushik (Ampere)