From nobody Fri Jul 24 22:57:05 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E11D2641CA for ; Wed, 22 Jul 2026 19:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784748026; cv=none; b=G6cwgSc1cQAOu+/wiU6xMbmUBlgI0eGaddR1BpUiPgBHSoKQysElHK6j4VbpmY5P5DYCg4DqqNei65A/xyyzPwlDEkPaYmxOF7gCIjMXXmKwLMyvdRCOUjjMe83z87MlvPw0ch57s6Wp/bupXBGorftJh7WKwQCX5qfqjBJjxmw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784748026; c=relaxed/simple; bh=TkcvuOC0OV2FZGYqb/q+n4RAzCeC2ucpBu+ITVFNY7k=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=B6TGOdtFZXS5xNr3jxWqDIkSKqVkxzpnPtHsU1xg6v1h3UVrEdHImo9OXbfYCFVcxzplshRjeBeSiBzpsJuSQwVki5H4i2yjWmvpdpsBWQBLl+wfhNcICMkSG9oaLW+o6ULxSHNLM3Kfv8ckoXRQadPvJ2Ch+2lhhFYNAdtCIBw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=S6LhsLP+; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="S6LhsLP+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1784748024; bh=TkcvuOC0OV2FZGYqb/q+n4RAzCeC2ucpBu+ITVFNY7k=; h=From:Date:Subject:To:Cc:From; b=S6LhsLP+cCtomXubMKiBpof2hsWOYO/J2kTOGFJCbC4ZeU5hNzFK3GncLIQPN02uB oIJhR4kUy2JsXASYR6HPv7JU1qDwjSDzefOx/tv7cLfgy71oecy6t5SycKsknglIc/ vhS0lfYq8WETZ+Ujl7Ar4A3dxIWB7XxcExi+Ug6E= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id 7B519401E1; Wed, 22 Jul 2026 12:20:24 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Wed, 22 Jul 2026 12:20:15 -0700 Subject: [PATCH v4] sched/core: Preserve wake flags across queued wakeups Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260722-b4-sched-ttwu-wake-flags-v4-1-8b151b1692cd@gentwo.org> X-B4-Tracking: v=1; b=H4sIAO4XYWoC/43NQQ6CMBCF4auYrh3TmQJFV97DuChlgEZDTYugI dzd6kpjYlz+L5lvZhE5OI5it5pF4NFF5/sU2XolbGf6lsHVqQVJKqRGCVUG0XZcwzBMV5jMiaE 5mzYCl9rURnKRM4p0fgncuNuLPhxTdy4OPtxfn0Z8rn+gIwICKSzJZDZXhd633A+T3/jQiqc60 rukfkiUJF0wKVNaq1B9SepNIvwhqSRJXW0JiaqGyg9pWZYHICcaVFUBAAA= X-Change-ID: 20260710-b4-sched-ttwu-wake-flags-e87ada0e65e1 To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak Cc: Christoph Lameter , Shubhang Kaushik , linux-kernel@vger.kernel.org, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1784748024; l=5556; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=TkcvuOC0OV2FZGYqb/q+n4RAzCeC2ucpBu+ITVFNY7k=; b=QBsGYv16x4lCzPuf4DhvNJwy5hCTbIlwdNbP0PsZn0S44CmeBlMgChfuBtXIyMgw82qh1iMTF nxed8SRRNCdCNBXf241wfjOW7r2/HUS1qDQfo7mmDZzC3NrXzVAYfXe X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= Queued wakeups currently save only whether the wakee migrated. When the target CPU drains the wakelist, sched_ttwu_pending() therefore passes only WF_MIGRATED or 0 to ttwu_do_activate(). That loses flags which may still be used when the target CPU drains the wakelist. One example is the queued path after select_task_rq(), where wake_flags may already include WF_RQ_SELECTED to record that a runqueue was selected: direct wakeup: select_task_rq() -> wake_flags includes WF_RQ_SELECTED ttwu_do_activate(wake_flags) wakeup_preempt_fair(wake_flags) queued wakeup after select_task_rq(): select_task_rq() -> wake_flags includes WF_RQ_SELECTED ttwu_queue_wakelist() -> save WF_MIGRATED only sched_ttwu_pending() -> ttwu_do_activate(WF_MIGRATED or 0) WF_RQ_SELECTED is used by ttwu_do_activate() for ENQUEUE_RQ_SELECTED and by wakeup_preempt_fair() for the preemption threshold. Preserve the wake flags that still matter after queueing: WF_TTWU, WF_SYNC, WF_MIGRATED and WF_RQ_SELECTED. Do not save WF_CURRENT_CPU, which is only a CPU-selection hint. Tested-by: K Prateek Nayak Signed-off-by: Shubhang Kaushik (Ampere) --- Tested on an 80 CPU Ampere Altra system with perf bench sched messaging, perf bench sched pipe, hackbench and SPECjBB. No material regression was observed. Temporary tracing confirmed that the queued wakeups reach sched_ttwu_pending() with WF_RQ_SELECTED preserved. Baseline: mainline origin/master at v7.2-rc4 (248951ddc14d) --- Changes in v4: - Drop READ_ONCE()/WRITE_ONCE() for sched_remote_wakeup_flags. - Remove the extra wake_flags local in sched_ttwu_pending(). - Fix the task_struct ordering comment. Link to v3: https://lore.kernel.org/r/20260721-b4-sched-ttwu-wake-flags-v3-= 1-07b92122bf28@gentwo.org Changes in v3: - Use READ_ONCE()/WRITE_ONCE() for sched_remote_wakeup_flags. - Anchor the changelog on the WF_RQ_SELECTED preemption behavior. - Keep the cross-CPU ordering comment for the task field. Link to v2: https://lore.kernel.org/r/20260713-b4-sched-ttwu-wake-flags-v2-= 1-76e23a8cc313@gentwo.org Changes in v2: - Move sched_remote_wakeup_flags to a standalone u8 outside the scheduler bitfields. - Drop the unnecessary reset in sched_ttwu_pending(). - Add WF_TTWU_QUEUE_MASK for the wake flags preserved across the wakelist. Link to v1: https://lore.kernel.org/r/20260710-b4-sched-ttwu-wake-flags-v1-= 1-23182a4c5367@gentwo.org --- include/linux/sched.h | 25 +++++++++---------------- kernel/sched/core.c | 5 +++-- 2 files changed, 12 insertions(+), 18 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d10b4256a11f8c8373ece78fa5e0e5..4a9d2b44389497bc2d1c65192e3= 0cfc93f08149c 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -983,6 +983,15 @@ struct task_struct { =20 /* Used for emulating ABI behavior of previous Linux versions: */ unsigned int personality; + /* + * Must not share the scheduler bitfield word because wakelist + * queueing is not serialized by p->on_cpu. + * + * smp_load_acquire(&p->on_cpu) before ttwu_queue_wakelist() pairs + * with smp_store_release(&p->on_cpu) in finish_task(). The flags are + * published before the task is added to the wakelist. + */ + u8 sched_remote_wakeup_flags; =20 /* Scheduler bits, serialized by scheduler locks: */ unsigned sched_reset_on_fork:1; @@ -993,22 +1002,6 @@ struct task_struct { /* Force alignment to the next boundary: */ unsigned :0; =20 - /* Unserialized, strictly 'current' */ - - /* - * This field must not be in the scheduler word above due to wakelist - * queueing no longer being serialized by p->on_cpu. However: - * - * p->XXX =3D X; ttwu() - * schedule() if (p->on_rq && ..) // false - * smp_mb__after_spinlock(); if (smp_load_acquire(&p->on_cpu) && //tr= ue - * deactivate_task() ttwu_queue_wakelist()) - * p->on_rq =3D 0; p->sched_remote_wakeup =3D Y; - * - * guarantees all stores of 'current' are visible before - * ->sched_remote_wakeup gets used, so it can be in this word. - */ - unsigned sched_remote_wakeup:1; #ifdef CONFIG_RT_MUTEXES unsigned sched_rt_mutex:1; #endif diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f6135341aa779b8262f113e103d8ad..5a908a0b3c9924c47ae99e96789= d1b16378650d4 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3908,7 +3908,7 @@ void sched_ttwu_pending(void *arg) if (WARN_ON_ONCE(task_cpu(p) !=3D cpu_of(rq))) set_task_cpu(p, cpu_of(rq)); =20 - ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); + ttwu_do_activate(rq, p, p->sched_remote_wakeup_flags, &rf); } =20 /* @@ -3947,11 +3947,12 @@ bool call_function_single_prep_ipi(int cpu) * via sched_ttwu_wakeup() for activation so the wakee incurs the cost * of the wakeup instead of the waker. */ +#define WF_TTWU_QUEUE_MASK (WF_TTWU | WF_SYNC | WF_MIGRATED | WF_RQ_SELECT= ED) static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake= _flags) { struct rq *rq =3D cpu_rq(cpu); =20 - p->sched_remote_wakeup =3D !!(wake_flags & WF_MIGRATED); + p->sched_remote_wakeup_flags =3D wake_flags & WF_TTWU_QUEUE_MASK; =20 WRITE_ONCE(rq->ttwu_pending, 1); #ifdef CONFIG_SMP --- base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df change-id: 20260710-b4-sched-ttwu-wake-flags-e87ada0e65e1 Best regards, --=20 Shubhang Kaushik (Ampere)