From nobody Sat Jul 25 00:16:38 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B08D12E7657 for ; Wed, 22 Jul 2026 00:36:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784680608; cv=none; b=cm8ca7yLoOMniit9RHv0+kergH5OJfzykQWUKXgpXOr5frELm/P/XmgIaFcK0JRUX/56lnLgj3oZ7bDyrnT6tVlsZubAGElFBIMXIJu7LCe53Yq8HHfmIFJD340RGQjIuwmRkVquFa9fa9jNLoRLZADMw0hMNH66HBW/PUdUlE8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784680608; c=relaxed/simple; bh=cvLOeRHZP6FTJ62N9Zne0zqFm545UEId9uzU+3vU/X0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=XhOoYdytC1HEZ0jy5h1yZrXUV8lHhJN7iX8jf4Z880zRGmN/PuS2r/CNmhdjnWUR75wGBo4rFrVC1yik+g+rtkNWPDYI0yTZcFpisIb82N6tH9VYv17iLv5s4UiihagNXwrk4Oajs6g8pFseOU+inLDWY5ZlZLpLe0R9gcII8z8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=ThvhmOwx; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="ThvhmOwx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1784678675; bh=cvLOeRHZP6FTJ62N9Zne0zqFm545UEId9uzU+3vU/X0=; h=From:Date:Subject:To:Cc:From; b=ThvhmOwxc8r07V6qQxkcdZz6n21qWiWBb2747C5kEU6pS+JMDjP58Kz9BRzNv8ohw rmc03L/3HzlHCiIplhBayyVSebIVcBiRshFWOiNZjlQksxgoUvfRe8LrLnSTcWnkxB ysNrJPBagBwyaJ08+K+XcTJODwBdbUjh53RQmM68= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id CCB6140287; Tue, 21 Jul 2026 17:04:34 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Tue, 21 Jul 2026 17:04:24 -0700 Subject: [PATCH] sched/fair: Prefer waker CPU for reciprocal sync wakeups Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260721-b4-sched-sync-wakeup-v1-1-dc94f184e27f@gentwo.org> X-B4-Tracking: v=1; b=H4sIAAcJYGoC/x3MSQqAMAxA0atI1gbaUhS8irjoEDUIVRqcEO9uc fkW/z8glJkEuuqBTAcLr6lA1xWE2aWJkGMxGGUa1RqN3qKEmSLKnQKebqF9Q2WjVcGT19FBSbd MI1//th/e9wOK0QRSZgAAAA== X-Change-ID: 20260721-b4-sched-sync-wakeup-04d40cbeb1da To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak Cc: "Christoph Lameter (Ampere)" , Shubhang Kaushik , linux-kernel@vger.kernel.org, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1784678674; l=3309; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=cvLOeRHZP6FTJ62N9Zne0zqFm545UEId9uzU+3vU/X0=; b=kxMu7cD/LEjZtmfOlPhroXtkrcWdrlzyiQUSg8aBpBLLEcBdx5eFi3ysKhIKn45i0OYi+Onh0 KUNitBz6nwECSrFg88FR5j1NOdTwTKx46bYDFNvqCnEKq4hj/4gEWeh X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= Pipe and futex ping-pong workloads can be dominated by handoff cost. In such cases, placing the wakee on an idle CPU can be slower than keeping the pair on the same runqueue. Use the existing last_wakee and wake_wide() state to identify narrow reciprocal WF_SYNC wakeups: A wakes B B wakes A A wakes B ... For these wakeups, keep the wakee on the waker CPU only when this looks like a narrow ping-pong: the wakee recently woke the current task, wake_wide() does not classify the relationship as wide, and the waker CPU has no other runnable CFS task. The wakee must also be allowed to run on that CPU, and on asymmetric-capacity systems it must fit there. Wakeups that do not match this pattern continue through the existing wake_affine() and select_idle_sibling() path. Signed-off-by: Shubhang Kaushik (Ampere) --- Tested on 80-core Ampere Altra: perf bench sched pipe -l 1000000 improved about 21%, averaged over 20 runs. Hackbench, schbench and SPECjBB showed no material regression. Baseline ~ v7.2-rc4 (mainline origin/master at b95f03f04d47) --- kernel/sched/fair.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1343050fcc2794dafb38ade3599e5..d9afc89bc7a45b0b05ebad536f9= a31d5757a2753 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8794,6 +8794,28 @@ static inline bool asym_fits_cpu(unsigned long util, return true; } =20 +/* For reciprocal WF_SYNC handoffs, prefer the waker CPU when it has no + * other runnable fair task. + */ +static bool prefer_sync_pair_cpu(struct task_struct *p, int cpu) +{ + struct rq *rq =3D cpu_rq(cpu); + + if ((rq->nr_running - cfs_h_nr_delayed(rq)) !=3D 1) + return false; + + if (!cpumask_test_cpu(cpu, p->cpus_ptr)) + return false; + + if (sched_asym_cpucap_active()) { + sync_entity_load_avg(&p->se); + if (!task_fits_cpu(p, cpu)) + return false; + } + + return true; +} + /* * Try and locate an idle core/thread in the LLC cache domain. */ @@ -9548,6 +9570,7 @@ select_task_rq_fair(struct task_struct *p, int prev_c= pu, int wake_flags) int cpu =3D smp_processor_id(); int new_cpu =3D prev_cpu; int want_affine =3D 0; + int wide =3D 0; /* SD_flags and WF_flags share the first nibble */ int sd_flag =3D wake_flags & 0xF; =20 @@ -9557,6 +9580,7 @@ select_task_rq_fair(struct task_struct *p, int prev_c= pu, int wake_flags) lockdep_assert_held(&p->pi_lock); if (wake_flags & WF_TTWU) { record_wakee(p); + wide =3D wake_wide(p); =20 if ((wake_flags & WF_CURRENT_CPU) && cpumask_test_cpu(cpu, p->cpus_ptr)) @@ -9569,7 +9593,12 @@ select_task_rq_fair(struct task_struct *p, int prev_= cpu, int wake_flags) new_cpu =3D prev_cpu; } =20 - want_affine =3D !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr); + if (sync && !wide && + READ_ONCE(p->last_wakee) =3D=3D current && + prefer_sync_pair_cpu(p, cpu)) + return cpu; + + want_affine =3D !wide && cpumask_test_cpu(cpu, p->cpus_ptr); } =20 for_each_domain(cpu, tmp) { --- base-commit: b95f03f04d475aa6719d15a636ddf32222d55657 change-id: 20260721-b4-sched-sync-wakeup-04d40cbeb1da Best regards, --=20 Shubhang Kaushik (Ampere)