From nobody Fri Oct 2 07:47:19 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52B4232AAA8 for ; Tue, 4 Aug 2026 01:45:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785807960; cv=none; b=U4dylNrwDgAZJUlFg0OwKbmtBTSgSMKWYrDaSWDUTyOZSPG7y0tWPIojy0r3q/XXlbRhF4PngI2bvnmjeKy3XHrtyWYINMbJEdwIYS3t/2YaOcaBIhHCKmjo/gzQQjeeFRjm5Ecib8KgkUutjiJzPlQ1Vf/n07FtM4fey4fZarU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785807960; c=relaxed/simple; bh=eaY9J7x8fOmdBtTooesX90thxixCNNqe4GsuoenB1rg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=I02JgYJcG+PzqdFY6QGf/KNRf+yxACLnbeU/q83MGeqGk+pbvC+wbh6X1nSHGh9H6VLL6ZiXShrCH2I1HDAbtxdCBlmWSfK2LrKN28NAnrSEWvbCLvwhEYGDbfRj675M34lujLtl4wedr0cdnpSB7cmC5St+hKi2XdpdRVPg850= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=PbtQqZtP; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="PbtQqZtP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1785807958; bh=eaY9J7x8fOmdBtTooesX90thxixCNNqe4GsuoenB1rg=; h=From:Date:Subject:To:Cc:From; b=PbtQqZtPVdhV5132V4JwXfcIYTpyMrrP8lVbS1QepY7ZeRqwLu6LmCNqtgv8CEHLW XXa8U2BwJiyH0x+aEFfQWYYsmEUP7W3AYgKAfdBiFmfXD6fMfdnO1L7Fm+SczMTXFe O0lUjWZ9JIrSYSUCZGt4ahVqwMqoZPuhEEXlHKC8= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id 3844D4010C; Mon, 03 Aug 2026 18:45:58 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Mon, 03 Aug 2026 18:45:49 -0700 Subject: [PATCH v4] sched/fair: Preserve wake-affine CPU for non-SMT reciprocal sync wakeups Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@gentwo.org> X-B4-Tracking: v=1; b=H4sIAExEcWoC/4XNQQ7CIBCF4asY1mIYHNvqynsYFwWGlphAAxVtm t5ddKUxxuX/kvlmZomio8QOq5lFyi654EvgesV03/qOuDOlmRSyErUErpAn3ZPhafKa39oLXQc u0KDQihSYlpXTIZJ19xd7OpfuXRpDnF5fMjzXP2AGDtzoPVpokGRtjx358RY2IXbsKWb5rsgfi iyKBahwVwklUX0p23el/qFsi7IX2mIDRplm96Esy/IAmm4ZQkUBAAA= X-Change-ID: 20260721-b4-sched-sync-wakeup-04d40cbeb1da To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Madadi Vineeth Reddy , Shrikanth Hegde Cc: "Christoph Lameter (Ampere)" , Shubhang Kaushik , linux-kernel@vger.kernel.org, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1785807958; l=4891; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=eaY9J7x8fOmdBtTooesX90thxixCNNqe4GsuoenB1rg=; b=hViGPH5JH6R9hiVBMm21s5+f80P5kQl9aQA8zsj5p0+vYlGJaxz4BR9G0j+PmdQwP6SkPi7Ku Ysao+95XU58D5MnOy2Wd7T/iUol/lsHEcDkrsmsWWnXo14NdI5RLWaW X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= For WF_SYNC wakeups, wake_affine() may select the waker CPU, but the CFS wakeup path still passes that target to select_idle_sibling(). The idle CPU search can then move the wakee away from the wake-affine target. Pipe-style ping-pong workloads expose this because the wakee is handed back and forth between two tasks. In that case, moving the wakee to another idle CPU can cost more than preserving the wake-affine waker CPU. Use the existing last_wakee and wake_wide() state to identify narrow reciprocal WF_SYNC wakeups: A wakes B B wakes A A wakes B ... Handle only this narrow reciprocal case on non-SMT systems. Once the wake-affine path has selected or kept the waker CPU, preserve that target when the waker rq has no other runnable fair task. Return the waker CPU before select_idle_sibling() so the idle CPU search does not move this handoff away from the wake-affine target. This does not define a generic WF_SYNC placement rule. Generic WF_SYNC wakeups continue through the existing wake_affine() and select_idle_sibling() behavior. SMT systems also continue through select_idle_sibling(), where idle sibling/core placement can be handled with SMT topology visible. On asymmetric-capacity systems, still require the wakee to fit on the waker CPU. Signed-off-by: Shubhang Kaushik (Ampere) --- Tested on 80-core non-SMT Ampere Altra, tip:sched/core baseline. perf bench sched pipe -l 1000000, 20 runs: default: 3.985 -> 3.187 usec/op mean, about 20.0% improvement 4.026 -> 3.181 usec/op median, about 21.0% improvement taskset -c 78,79: 3.851 -> 3.144 usec/op mean, about 18.4% improvement 3.804 -> 3.140 usec/op median, about 17.4% improvement taskset -c 79: 3.055 -> 3.113 usec/op mean, about 1.9% slower 3.045 -> 3.109 usec/op median, about 2.1% slower Hackbench process/thread pipe cases with 1/2/4/8 groups were within noise, with mean deltas from -1.8% to +3.7% over 10 runs. Schbench normal mode at 8/40/80/240 workers and schbench pipe mode at 1/2/4/8 workers showed no material regression. Baseline: tip/sched/core at 5186ef36909c --- Changes in v4: - Preserve the waker CPU only after the wake-affine path selected or kept it. - Clarify that WF_SYNC remains a hint, not a generic placement rule. - Leave SMT systems on the existing select_idle_sibling() path. - Refresh testing on tip:sched/core. Link to v3: https://lore.kernel.org/r/20260727-b4-sched-sync-wakeup-v3-1-90= cf481dbd85@gentwo.org Changes in v3: - Limit the direct waker-CPU preference to !sched_smt_active(); SMT systems continue through the existing wake_affine() and select_idle_sibling() path. - Drop the redundant affinity check; want_affine already verifies the waker CPU is allowed. - Use a plain p->last_wakee read instead of READ_ONCE(). - Rebase and refresh testing on v7.2-rc5. Link to v2: https://lore.kernel.org/r/20260722-b4-sched-sync-wakeup-v2-1-f1= 164560b24b@gentwo.org Changes in v2: - Move the reciprocal handoff preference under the existing SD_WAKE_AFFINE domain check. - Drop futex from the changelog motivation. - Refresh perf bench sched pipe results after rebasing. Link to v1: https://lore.kernel.org/r/20260721-b4-sched-sync-wakeup-v1-1-dc= 94f184e27f@gentwo.org --- kernel/sched/fair.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1343050fcc2794dafb38ade3599e5..e377dc43460a691315d895e5cde= 5208987791ce8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8794,6 +8794,26 @@ static inline bool asym_fits_cpu(unsigned long util, return true; } =20 +/* + * For reciprocal WF_SYNC handoffs, prefer the waker CPU when it has no + * other runnable fair task. + */ +static bool prefer_sync_pair_cpu(struct task_struct *p, int cpu) +{ + struct rq *rq =3D cpu_rq(cpu); + + if ((rq->nr_running - cfs_h_nr_delayed(rq)) !=3D 1) + return false; + + if (sched_asym_cpucap_active()) { + sync_entity_load_avg(&p->se); + if (!task_fits_cpu(p, cpu)) + return false; + } + + return true; +} + /* * Try and locate an idle core/thread in the LLC cache domain. */ @@ -9582,6 +9602,12 @@ select_task_rq_fair(struct task_struct *p, int prev_= cpu, int wake_flags) if (cpu !=3D prev_cpu) new_cpu =3D wake_affine(tmp, p, cpu, prev_cpu, sync); =20 + if (sync && !sched_smt_active() && + new_cpu =3D=3D cpu && + p->last_wakee =3D=3D current && + prefer_sync_pair_cpu(p, cpu)) + return cpu; + sd =3D NULL; /* Prefer wake_affine over balance flags */ break; } --- base-commit: 075b74841bd0065a3bda3440873c747938e69b68 change-id: 20260721-b4-sched-sync-wakeup-04d40cbeb1da Best regards, --=20 Shubhang Kaushik (Ampere)