include/net/sock.h | 1 + net/core/sock.c | 31 +++++++++++++++++++------ net/sctp/socket.c | 10 +++++++-- net/smc/af_smc.c | 4 ++-- net/smc/smc_rx.c | 10 +++++++-- net/socket.c | 56 +++++++++++++++++++++++++++++++++++++++------- net/tipc/socket.c | 22 +++++++++++++----- net/unix/af_unix.c | 26 ++++++++++++++------- 8 files changed, 126 insertions(+), 34 deletions(-)
The scheduler assumes in several wakeup paths that a task using WF_SYNC
is likely to yield the CPU shortly. However several networking wakeup
paths unconditionally use synchronous wakeups even when the waking task
continues execution.
During wakeup, with WF_SYNC flag set, because of the assumption that
current thread is ready to give up, the wakee thread will be migrated
from any core within the chip to the current LLC. If these operations
are frequent, and wakers are actually not going away, then it will lead
to load imbalance and hurt performance. This is especially true in
architectures where LLCs are small and number of LLCs per chip are more.
Running vllm workload was run on Power10 system
No patch with patch %diff
Inference Time (sec) 17.84 16.35 -8.35%
llm query bandwidth (tokens/sec) 14.78 16.21 +9.68%
Lower inference time and higher tokens/sec is better.
Srikar Dronamraju (2):
net/socket: Record preference for synchronous wakeups
net/sock: Propagate WF_SYNC only when requested
include/net/sock.h | 1 +
net/core/sock.c | 31 +++++++++++++++++++------
net/sctp/socket.c | 10 +++++++--
net/smc/af_smc.c | 4 ++--
net/smc/smc_rx.c | 10 +++++++--
net/socket.c | 56 +++++++++++++++++++++++++++++++++++++++-------
net/tipc/socket.c | 22 +++++++++++++-----
net/unix/af_unix.c | 26 ++++++++++++++-------
8 files changed, 126 insertions(+), 34 deletions(-)
--
2.51.0
On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote: > The scheduler assumes in several wakeup paths that a task using WF_SYNC > is likely to yield the CPU shortly. However several networking wakeup > paths unconditionally use synchronous wakeups even when the waking task > continues execution. > > During wakeup, with WF_SYNC flag set, because of the assumption that > current thread is ready to give up, the wakee thread will be migrated > from any core within the chip to the current LLC. If these operations > are frequent, and wakers are actually not going away, then it will lead > to load imbalance and hurt performance. This is especially true in > architectures where LLCs are small and number of LLCs per chip are more. > > Running vllm workload was run on Power10 system > No patch with patch %diff > Inference Time (sec) 17.84 16.35 -8.35% > llm query bandwidth (tokens/sec) 14.78 16.21 +9.68% > > Lower inference time and higher tokens/sec is better. Somewhat related recent patch: https://lore.kernel.org/all/20260708133815.3419465-1-usama.arif@linux.dev/ Maybe other maintainers will chime in, but I'm not convinced by the heuristics you're adding. The behavior will be workload specific. The extent of networking involvement should be letting the scheduler know that we're waking from IO, not leaking scheduler heuristics into networking. -- pw-bot: cr
* Jakub Kicinski <kuba@kernel.org> [2026-07-22 10:08:42]: Hi Jakub, Thanks for taking a look. > On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote: > > The scheduler assumes in several wakeup paths that a task using WF_SYNC > > is likely to yield the CPU shortly. However several networking wakeup > > paths unconditionally use synchronous wakeups even when the waking task > > continues execution. > > > > During wakeup, with WF_SYNC flag set, because of the assumption that > > current thread is ready to give up, the wakee thread will be migrated > > from any core within the chip to the current LLC. If these operations > > are frequent, and wakers are actually not going away, then it will lead > > to load imbalance and hurt performance. This is especially true in > > architectures where LLCs are small and number of LLCs per chip are more. > > > > Running vllm workload was run on Power10 system > > No patch with patch %diff > > Inference Time (sec) 17.84 16.35 -8.35% > > llm query bandwidth (tokens/sec) 14.78 16.21 +9.68% > > > > Lower inference time and higher tokens/sec is better. > > Somewhat related recent patch: > > https://lore.kernel.org/all/20260708133815.3419465-1-usama.arif@linux.dev/ > > Maybe other maintainers will chime in, but I'm not convinced > by the heuristics you're adding. The behavior will be workload > specific. Currently scheduler provides 2 APIs because it supports 2 subtle behaviours. 1st API being wake_up_interruptible_poll and 2nd API being wake_up_interruptible_sync_poll. The reason scheduler provides 2 APIs is because the scheduler cant decide which behaviour the callee wants. If the scheduler could have figured out which behaviour then there would have been only 1 API. So the onus of using the right API is on the callee. Now the behaviour between the 2 APIs depends on fact that the callee is going to give up the CPU now or if the callee will continue to run. If the callee is going to continue to run, its better to use the more generic API of the two which is wake_up_interruptible_poll. However if the callee is giving up the CPU, its better to use wake_up_interruptible_sync_poll. @peterz @ingom can you please confirm this behaviour is inline with scheduler maintainers view. Currently the networking code as I see is unconditionally (or blindly) asking for sync behaviour i.e the socket API is saying the callees are going to give up CPU soon. However the problem I am seeing is the callees are actually not giving up CPU. > The extent of networking involvement should be letting > the scheduler know that we're waking from IO, not leaking scheduler > heuristics into networking. In this change, what we are depending on is file's flag that the socket request is blocking or non-blocking. So why do we say we are leaking scheduler hueristics? Would using an API or another API mean leaking scheduler hueristics? If we are blocking, this change will use wake_up_interruptible_sync_poll and if non-blocking, it will use wake_up_interruptible_poll. -- Thanks and Regards Srikar Dronamraju
On Wed, Jul 22, 2026 at 7:08 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote: > > The scheduler assumes in several wakeup paths that a task using WF_SYNC > > is likely to yield the CPU shortly. However several networking wakeup > > paths unconditionally use synchronous wakeups even when the waking task > > continues execution. > > > > During wakeup, with WF_SYNC flag set, because of the assumption that > > current thread is ready to give up, the wakee thread will be migrated > > from any core within the chip to the current LLC. If these operations > > are frequent, and wakers are actually not going away, then it will lead > > to load imbalance and hurt performance. This is especially true in > > architectures where LLCs are small and number of LLCs per chip are more. > > > > Running vllm workload was run on Power10 system > > No patch with patch %diff > > Inference Time (sec) 17.84 16.35 -8.35% > > llm query bandwidth (tokens/sec) 14.78 16.21 +9.68% > > > > Lower inference time and higher tokens/sec is better. > > Somewhat related recent patch: > > https://lore.kernel.org/all/20260708133815.3419465-1-usama.arif@linux.dev/ > > Maybe other maintainers will chime in, but I'm not convinced > by the heuristics you're adding. The behavior will be workload > specific. The extent of networking involvement should be letting > the scheduler know that we're waking from IO, not leaking scheduler > heuristics into networking. > -- > pw-bot: cr In any case, adding code to the fast path in every network system call is undesirable.
* Eric Dumazet <edumazet@google.com> [2026-07-23 05:17:08]: > On Wed, Jul 22, 2026 at 7:08 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > > On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote: > > > The scheduler assumes in several wakeup paths that a task using WF_SYNC > > > is likely to yield the CPU shortly. However several networking wakeup > > > paths unconditionally use synchronous wakeups even when the waking task > > > continues execution. > > > > Somewhat related recent patch: > > > > https://lore.kernel.org/all/20260708133815.3419465-1-usama.arif@linux.dev/ > > > > Maybe other maintainers will chime in, but I'm not convinced > > by the heuristics you're adding. The behavior will be workload > > specific. The extent of networking involvement should be letting > > the scheduler know that we're waking from IO, not leaking scheduler > > heuristics into networking. > > -- > > pw-bot: cr > > In any case, adding code to the fast path in every network system call > is undesirable. Yes, I understand adding code in fast path is undesirable. However as mentioned previous reply, we need to differentiate between blocking and non-blocking API. Doing unconditional nonblocking, is probably even worse. -- Thanks and Regards Srikar Dronamraju
On Thu, Jul 23, 2026 at 10:50 AM Srikar Dronamraju <srikar@linux.ibm.com> wrote: > > * Eric Dumazet <edumazet@google.com> [2026-07-23 05:17:08]: > > > On Wed, Jul 22, 2026 at 7:08 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > > > > On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote: > > > > The scheduler assumes in several wakeup paths that a task using WF_SYNC > > > > is likely to yield the CPU shortly. However several networking wakeup > > > > paths unconditionally use synchronous wakeups even when the waking task > > > > continues execution. > > > > > > Somewhat related recent patch: > > > > > > https://lore.kernel.org/all/20260708133815.3419465-1-usama.arif@linux.dev/ > > > > > > Maybe other maintainers will chime in, but I'm not convinced > > > by the heuristics you're adding. The behavior will be workload > > > specific. The extent of networking involvement should be letting > > > the scheduler know that we're waking from IO, not leaking scheduler > > > heuristics into networking. > > > -- > > > pw-bot: cr > > > > In any case, adding code to the fast path in every network system call > > is undesirable. > > Yes, I understand adding code in fast path is undesirable. However as > mentioned previous reply, we need to differentiate between blocking and > non-blocking API. Doing unconditional nonblocking, is probably even worse. I suggested that applications choose what they prefer for each socket, once and for all. (This can also be a cgroup/BPF setting) Flipping a socket bit at each recvmsg()/sendmsg() is a bad idea, you might break old application expectations.
* Eric Dumazet <edumazet@google.com> [2026-07-23 10:57:24]: > On Thu, Jul 23, 2026 at 10:50 AM Srikar Dronamraju <srikar@linux.ibm.com> wrote: > > > > * Eric Dumazet <edumazet@google.com> [2026-07-23 05:17:08]: > > > > > On Wed, Jul 22, 2026 at 7:08 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > > > > > > On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote: > > > > > The scheduler assumes in several wakeup paths that a task using WF_SYNC > > > > > is likely to yield the CPU shortly. However several networking wakeup > > > > > paths unconditionally use synchronous wakeups even when the waking task > > > > > continues execution. > > > > > > In any case, adding code to the fast path in every network system call > > > is undesirable. > > > > Yes, I understand adding code in fast path is undesirable. However as > > mentioned previous reply, we need to differentiate between blocking and > > non-blocking API. Doing unconditional nonblocking, is probably even worse. > > I suggested that applications choose what they prefer for each socket, > once and for all. > (This can also be a cgroup/BPF setting) > > Flipping a socket bit at each recvmsg()/sendmsg() is a bad idea, you > might break old application expectations. Set once per socket is fine. However at the time of wake, we still have to check the socket option right? I hope that overhead is fine. Also do you have any precedent for once per socket option that I could look at? Thanks for your inputs. -- Thanks and Regards Srikar Dronamraju
© 2016 - 2026 Red Hat, Inc.