From nobody Fri Jul 24 22:20:16 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 983C02288D5 for ; Wed, 22 Jul 2026 22:50:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784760649; cv=none; b=Z6QiNVnJplv1AQKoeMWFQ7ZBEA+Anu+cA1PZmWv7TtAzAbThCku209ogt7FGMrvCJ5lF+LUZD9jzOFNpweJfws10+oh7WKmrMaySSwJmXwla12xXgJFBtWkR/udqUwURcsMdokwmUiqW+NN6AiYrHN7xOhlojTZ80NDl14MkV7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784760649; c=relaxed/simple; bh=W6l23OegMLifZdj/DW8eYYRhOQ+jtFXsEHjE26OkKlc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=IwnQoZ/W0ssqL8Uho5SrdsPurKlz3mGl0b+MpbVOeLdMQS6J5XXSi3w7JEzminueCikHttBSNOamOpCoNnxkTikDEr//uBfqee7ISUqbVdext1Sb39JmetlUYHdf0BDOVrZysjnxg/lmqqTAFmS4oZP8j9iPkRniI4F6MsCc24w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=UcKShj5D; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="UcKShj5D" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1784760646; bh=W6l23OegMLifZdj/DW8eYYRhOQ+jtFXsEHjE26OkKlc=; h=From:Date:Subject:To:Cc:From; b=UcKShj5DdWMqEtmKgOImkEN8zcCQ61oWsSy1QwzLtqmyJm8OTrJ6OrQc/zWI5kGpS L/pGSLpABbBQgB/iEcUEHSWR1CIxznHYZQPyA15JiouyVtvFkA6MfVocZNvtxBbRcq NcijOGH+uAAmL5GXsLM0t9fVsl4/9l+kvE8Q/xpc= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id 9A327401E1; Wed, 22 Jul 2026 15:50:46 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Wed, 22 Jul 2026 15:50:45 -0700 Subject: [PATCH v2] sched/fair: Prefer waker CPU for reciprocal sync wakeups Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260722-b4-sched-sync-wakeup-v2-1-f1164560b24b@gentwo.org> X-B4-Tracking: v=1; b=H4sIAERJYWoC/4WNSw7CIBRFt9K8sc8AIVYduQ/TAZ9HS0yggUptG vYudgMOz0nuuTtkSp4y3LsdEhWffQwNxKkDM6kwEnrbGAQTF9YLjlpiNhNZzFswuKoXvWdk0kp mNGluFbTpnMj5z5F9Do0nn5eYtuOl8J/9EywcOVpzk45fJYnePUYKyxrPMY0w1Fq/MeVG9LkAA AA= X-Change-ID: 20260721-b4-sched-sync-wakeup-04d40cbeb1da To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak Cc: "Christoph Lameter (Ampere)" , Shubhang Kaushik , linux-kernel@vger.kernel.org, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1784760646; l=2946; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=W6l23OegMLifZdj/DW8eYYRhOQ+jtFXsEHjE26OkKlc=; b=zCrLpnCUHSvbYB1ODKViCitTkoTfB5GFoEHTYkD+veF8UZS2XNhQGvEg1NT3Y3LEuK+sPKYos OmgYk4LfcDyAgOMDEgC7JSwEYlR8z699WLxo3w0Mmx3V/I4N0N1kAp7 X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= Pipe-style ping-pong workloads can be dominated by handoff cost. In such cases, placing the wakee on an idle CPU can be slower than keeping the pair on the same runqueue. Use the existing last_wakee and wake_wide() state to identify narrow reciprocal WF_SYNC wakeups: A wakes B B wakes A A wakes B ... When the wake-affine domain allows SD_WAKE_AFFINE, prefer the waker CPU for these narrow reciprocal handoffs. Do so only when the waker CPU has no other runnable fair task, the wakee is allowed on that CPU, and the wakee fits there on asymmetric capacity systems. Wakeups that do not match this pattern continue through the existing wake_affine() and select_idle_sibling() path. Signed-off-by: Shubhang Kaushik (Ampere) --- Tested on 80-core Ampere Altra: perf bench sched pipe -l 1000000 improved by about 30%, averaged over 20 runs. Hackbench, schbench and SPECjBB showed no material regression. Baseline ~ v7.2-rc4 (mainline origin/master at 248951ddc14d) --- Changes in v2: - Move the reciprocal handoff preference under the existing SD_WAKE_AFFINE domain check. - Drop futex from the changelog motivation. - Refresh perf bench sched pipe results after rebasing. Link to v1: https://lore.kernel.org/r/20260721-b4-sched-sync-wakeup-v1-1-dc= 94f184e27f@gentwo.org --- kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1343050fcc2794dafb38ade3599e5..d188e91b85d74dd3ed3c8b99f17= 1bd142e72da5e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8794,6 +8794,29 @@ static inline bool asym_fits_cpu(unsigned long util, return true; } =20 +/* + * For reciprocal WF_SYNC handoffs, prefer the waker CPU when it has no + * other runnable fair task. + */ +static bool prefer_sync_pair_cpu(struct task_struct *p, int cpu) +{ + struct rq *rq =3D cpu_rq(cpu); + + if ((rq->nr_running - cfs_h_nr_delayed(rq)) !=3D 1) + return false; + + if (!cpumask_test_cpu(cpu, p->cpus_ptr)) + return false; + + if (sched_asym_cpucap_active()) { + sync_entity_load_avg(&p->se); + if (!task_fits_cpu(p, cpu)) + return false; + } + + return true; +} + /* * Try and locate an idle core/thread in the LLC cache domain. */ @@ -9579,6 +9602,11 @@ select_task_rq_fair(struct task_struct *p, int prev_= cpu, int wake_flags) */ if (want_affine && (tmp->flags & SD_WAKE_AFFINE) && cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) { + if (sync && + READ_ONCE(p->last_wakee) =3D=3D current && + prefer_sync_pair_cpu(p, cpu)) + return cpu; + if (cpu !=3D prev_cpu) new_cpu =3D wake_affine(tmp, p, cpu, prev_cpu, sync); =20 --- base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df change-id: 20260721-b4-sched-sync-wakeup-04d40cbeb1da Best regards, --=20 Shubhang Kaushik (Ampere)