A subflow that is slow relative to the others can soak up connection
resources and cause head-of-line blocking of the aggregate stream. This
series lets the packet scheduler send less than such a subflow's
congestion window by halving it, and leaves the congestion control to
raise it again. It implements issue #345 [1]. This is a deliberately
simple first form.

- Patch 1: a prerequisite fix. The scheduler truncates the pacing rate to a
  u32, which skews subflow selection on fast paths (Fixes).
- Patch 2: penalise a slow subflow by halving its cwnd, triggered on its
  pacing rate and gated on the fast path being cwnd-limited.
- Patch 3: skip the penalty when the transfer is receive-window-limited,
  where it would otherwise slow the transfer.
- Patch 4: MPTcpExt counters and a dedicated mptcp_subflow_penalise
  tracepoint for observability.

Validation, baseline versus this series (test helper, its run_matrix.sh
base-vs-patch driver, and run steps are in [2]):
- No regression in the mptcp selftests.
- No regression versus baseline across the buffer-limited transfers
  (send-buffer, receive-window, both, sndbuf>rcvbuf): the two-path completion
  time with the penalty enabled matches the unpatched kernel. Where the
  slow subflow is beneficial (send-buffer-limited), the two-path transfer
  stays faster than the fast path alone.
- Patch 3's gate keeps the penalty from firing when receive-window-limited,
  where halving cwnd would slow the transfer.
- With a bufferbloated slow path, the penalty roughly halves the slow path's
  queueing latency (subflow srtt), with the transfer time unchanged. It does
  not reduce the aggregate out-of-order volume or move data off the slow
  path; that is the role of issue #332 [3], and the two are complementary.
- The conditions above do not catch one case: with the buffers left to
  autotuning and a subflow that is slow because of latency rather than limited
  bandwidth, penalising it costs throughput, and the cost grows as the added
  delay grows.

Used LLM for design, review, coding assistance.

[1] https://github.com/multipath-tcp/mptcp_net-next/issues/345
[2] https://github.com/shardulsdk-mpiric/linux/blob/e0f78406d921/tools/testing/selftests/net/mptcp/mptcp_sched_penalise.sh
    (run_matrix.sh + runs.conf are alongside it in the same directory)
[3] https://github.com/multipath-tcp/mptcp_net-next/issues/332

Changes in v5:
- Rebased on the current export; no code change.
- Dropped the Assisted-by tags, which the kernel docs no longer describe;
  tool use is noted in this cover letter instead.
- Collected Abdul Wasey's Tested-by, thanks:
  https://lore.kernel.org/all/20261025192320.1831707-1-w453y.me@gmail.com/
- Link to v4: https://patch.msgid.link/20260911-mptcp_penalise_send_v2-v4-0-820f2be4a8fe@mpiricsoftware.com

Changes in v4:
- Patch 1: compute the pacing-rate average with 128-bit intermediates
  (mul_u64_u64_div_u64); a single 64-bit numerator overflows at a high
  pacing rate with a large send queue. Dividing before multiplying loses
  too much precision on slow paths, details in the commit message.
- Patch 1: add a second Fixes tag. The u32 truncation predates the
  averaging formula; it dates to d5f49190def6.
- Patch 2: evaluate the conditions on the subflow itself where the penalty is
  applied, under that subflow's socket lock; the scheduler only marks a
  candidate. Annotate the cross-subflow tcp_is_cwnd_limited() read with
  data_race().
- Patch 2: exclude backup subflows from the fastest-path comparison. A
  backup carries no data while an active subflow exists, so its rate and
  cwnd-limited state go stale, and using one as the reference penalised
  the subflow that was actually sending.
- Patch 3: drop a throughput figure taken from an early debug-kernel run;
  the mechanism argument stands without it.
- Patch 4: add the send burst to mptcp_subflow_penalise, and emit the
  verdict once the burst is known.
- Longer explanations moved from code comments into the commit messages.
- Retested base versus this series on one kernel configuration, congestion
  control pinned to reno, three runs of the full matrix on each side. No
  behavioural difference against v3.
- Link to v3: https://patch.msgid.link/20260817-mptcp_penalise_send_v2-v3-0-e6a2ad2f1b82@mpiricsoftware.com

Changes in v3:
- Dropped the RFC tag (Matthieu Baerts (NGI0)). No changes to the patches themselves.
- Rebased on current export.
- Link to v2: https://patch.msgid.link/20260815-mptcp_penalise_send_v2-v2-0-3e5049a73681@mpiricsoftware.com

Changes in v2
- Patch 1 (new, not in v1): fix the scheduler's pre-existing u32 pacing-rate
  truncation (Fixes: 3ce0852c86b9).
- Patch 2 (v1's 1/3): also skip a subflow already at the cwnd floor
  (MPTCP_PENALISE_MIN_CWND); clear the flag when returning without send-window
  room; re-check TCP_CA_Open in the apply path; express the rate trigger as a
  division (avg_pacing_rate < max_pace / ratio, using the widened pace from
  patch 1) instead of a u64-cast multiply.
- Patch 3 (v1's 2/3): make the receive-window test wrap-safe
  (after64, like tcp_snd_wnd_test()); same condition otherwise.
- Patch 4 (v1's 3/3): real MPTcpExt counters (dropped DO-NOT-MERGE) plus
  a dedicated mptcp_subflow_penalise tracepoint (not a field on
  mptcp_subflow_get_send).
- Rebased on current export.
- Link to v1:
  https://lore.kernel.org/mptcp/20260726-mptcp_penalise_send-v1-0-84485e0e995b@mpiricsoftware.com

Signed-off-by: Shardul Bankar <shardul.b@mpiricsoftware.com>
---
Shardul Bankar (4):
      mptcp: sched: avoid truncating the pacing rate in the scheduler
      mptcp: sched: penalise a slow subflow by halving its cwnd
      mptcp: sched: do not penalise when receive-window-limited
      mptcp: sched: add penalise counters and tracepoint

 include/trace/events/mptcp.h | 33 ++++++++++++++++
 net/mptcp/mib.c              |  2 +
 net/mptcp/mib.h              |  2 +
 net/mptcp/protocol.c         | 89 ++++++++++++++++++++++++++++++++++++++++----
 net/mptcp/protocol.h         |  2 +
 5 files changed, 121 insertions(+), 7 deletions(-)
---
base-commit: d367c3b57a16822766c0f0c9b039a3d29d9dca56
change-id: 20260815-mptcp_penalise_send_v2-ed6206f70a84

Best regards,
--  
Shardul Bankar <shardul.b@mpiricsoftware.com>

Hi Shardul,

On 28/10/2026 12:15, Shardul Bankar wrote:
> A subflow that is slow relative to the others can soak up connection
> resources and cause head-of-line blocking of the aggregate stream. This
> series lets the packet scheduler send less than such a subflow's
> congestion window by halving it, and leaves the congestion control to
> raise it again. It implements issue #345 [1]. This is a deliberately
> simple first form.
> 
> - Patch 1: a prerequisite fix. The scheduler truncates the pacing rate to a
>   u32, which skews subflow selection on fast paths (Fixes).
> - Patch 2: penalise a slow subflow by halving its cwnd, triggered on its
>   pacing rate and gated on the fast path being cwnd-limited.
> - Patch 3: skip the penalty when the transfer is receive-window-limited,
>   where it would otherwise slow the transfer.
> - Patch 4: MPTcpExt counters and a dedicated mptcp_subflow_penalise
>   tracepoint for observability.
> 
> Validation, baseline versus this series (test helper, its run_matrix.sh
> base-vs-patch driver, and run steps are in [2]):
> - No regression in the mptcp selftests.
> - No regression versus baseline across the buffer-limited transfers
>   (send-buffer, receive-window, both, sndbuf>rcvbuf): the two-path completion
>   time with the penalty enabled matches the unpatched kernel. Where the
>   slow subflow is beneficial (send-buffer-limited), the two-path transfer
>   stays faster than the fast path alone.
> - Patch 3's gate keeps the penalty from firing when receive-window-limited,
>   where halving cwnd would slow the transfer.
> - With a bufferbloated slow path, the penalty roughly halves the slow path's
>   queueing latency (subflow srtt), with the transfer time unchanged. It does
>   not reduce the aggregate out-of-order volume or move data off the slow
>   path; that is the role of issue #332 [3], and the two are complementary.
> - The conditions above do not catch one case: with the buffers left to
>   autotuning and a subflow that is slow because of latency rather than limited
>   bandwidth, penalising it costs throughput, and the cost grows as the added
>   delay grows.
> 
> Used LLM for design, review, coding assistance.
> 
> [1] https://github.com/multipath-tcp/mptcp_net-next/issues/345
> [2] https://github.com/shardulsdk-mpiric/linux/blob/e0f78406d921/tools/testing/selftests/net/mptcp/mptcp_sched_penalise.sh
>     (run_matrix.sh + runs.conf are alongside it in the same directory)
> [3] https://github.com/multipath-tcp/mptcp_net-next/issues/332
> 
> Changes in v5:
> - Rebased on the current export; no code change.
> - Dropped the Assisted-by tags, which the kernel docs no longer describe;
>   tool use is noted in this cover letter instead.
> - Collected Abdul Wasey's Tested-by, thanks:
>   https://lore.kernel.org/all/20261025192320.1831707-1-w453y.me@gmail.com/
> - Link to v4: https://patch.msgid.link/20260911-mptcp_penalise_send_v2-v4-0-820f2be4a8fe@mpiricsoftware.com

Thank you for the new version!

I just noticed that Abdul's v2 series that I recently sent [1] didn't
get reviewed by Clashiko because it doesn't support dependences yet
(Sashiko does) [2]. Just to avoid both working on the same thing: are
you already working on a new version addressing / rejecting the last
review comments sent here? No rush at all, but if you didn't start yet,
and you prefer me to look at it, I can try to do that this week. I would
then send a version including your patches with fixes (if I can) and the
ones I recently sent. Otherwise, there is no urgency, and I can wait.

[1]
https://lore.kernel.org/20261031-mptcp-sched-opp-retrans-v2-0-e9c4a30f6841@kernel.org
[2]
https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261031-mptcp-sched-opp-retrans-v2-0-e9c4a30f6841@kernel.org

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

On Mon, 2026-11-02 at 19:37 +0100, Matthieu Baerts wrote:
> Hi Shardul,
> 
> On 28/10/2026 12:15, Shardul Bankar wrote:
> > A subflow that is slow relative to the others can soak up
> > connection
> > resources and cause head-of-line blocking of the aggregate stream.
> > This
> > series lets the packet scheduler send less than such a subflow's
> > congestion window by halving it, and leaves the congestion control
> > to
> > raise it again. It implements issue #345 [1]. This is a
> > deliberately
> > simple first form.
> > 
> > - Patch 1: a prerequisite fix. The scheduler truncates the pacing
> > rate to a
> >   u32, which skews subflow selection on fast paths (Fixes).
> > - Patch 2: penalise a slow subflow by halving its cwnd, triggered
> > on its
> >   pacing rate and gated on the fast path being cwnd-limited.
> > - Patch 3: skip the penalty when the transfer is receive-window-
> > limited,
> >   where it would otherwise slow the transfer.
> > - Patch 4: MPTcpExt counters and a dedicated mptcp_subflow_penalise
> >   tracepoint for observability.
> > 
> > Validation, baseline versus this series (test helper, its
> > run_matrix.sh
> > base-vs-patch driver, and run steps are in [2]):
> > - No regression in the mptcp selftests.
> > - No regression versus baseline across the buffer-limited transfers
> >   (send-buffer, receive-window, both, sndbuf>rcvbuf): the two-path
> > completion
> >   time with the penalty enabled matches the unpatched kernel. Where
> > the
> >   slow subflow is beneficial (send-buffer-limited), the two-path
> > transfer
> >   stays faster than the fast path alone.
> > - Patch 3's gate keeps the penalty from firing when receive-window-
> > limited,
> >   where halving cwnd would slow the transfer.
> > - With a bufferbloated slow path, the penalty roughly halves the
> > slow path's
> >   queueing latency (subflow srtt), with the transfer time
> > unchanged. It does
> >   not reduce the aggregate out-of-order volume or move data off the
> > slow
> >   path; that is the role of issue #332 [3], and the two are
> > complementary.
> > - The conditions above do not catch one case: with the buffers left
> > to
> >   autotuning and a subflow that is slow because of latency rather
> > than limited
> >   bandwidth, penalising it costs throughput, and the cost grows as
> > the added
> >   delay grows.
> > 
> > Used LLM for design, review, coding assistance.
> > 
> > [1] https://github.com/multipath-tcp/mptcp_net-next/issues/345
> > [2]
> > https://github.com/shardulsdk-mpiric/linux/blob/e0f78406d921/tools/testing/selftests/net/mptcp/mptcp_sched_penalise.sh
> >     (run_matrix.sh + runs.conf are alongside it in the same
> > directory)
> > [3] https://github.com/multipath-tcp/mptcp_net-next/issues/332
> > 
> > Changes in v5:
> > - Rebased on the current export; no code change.
> > - Dropped the Assisted-by tags, which the kernel docs no longer
> > describe;
> >   tool use is noted in this cover letter instead.
> > - Collected Abdul Wasey's Tested-by, thanks:
> >  
> > https://lore.kernel.org/all/20261025192320.1831707-1-w453y.me@gmail.com/
> > - Link to v4:
> > https://patch.msgid.link/20260911-mptcp_penalise_send_v2-v4-0-820f2be4a8fe@mpiricsoftware.com
> 
> Thank you for the new version!
> 
> I just noticed that Abdul's v2 series that I recently sent [1] didn't
> get reviewed by Clashiko because it doesn't support dependences yet
> (Sashiko does) [2]. Just to avoid both working on the same thing: are
> you already working on a new version addressing / rejecting the last
> review comments sent here? No rush at all, but if you didn't start
> yet,
> and you prefer me to look at it, I can try to do that this week. I
> would
> then send a version including your patches with fixes (if I can) and
> the
> ones I recently sent. Otherwise, there is no urgency, and I can wait.
> 

Hi Matt,

Thank you for the offer. I have not started on a new version yet, so
please go ahead and include my patches in your series, with the fixes
you think are needed. Feel free to reach out if I can help with
anything.

Cheers,
Shardul

> [1]
> https://lore.kernel.org/20261031-mptcp-sched-opp-retrans-v2-0-e9c4a30f6841@kernel.org
> [2]
> https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261031-mptcp-sched-opp-retrans-v2-0-e9c4a30f6841@kernel.org
> 
> Cheers,
> Matt

Hi Shardul,

Thank you for your reply!

On 04/11/2026 11:08, Shardul Bankar wrote:
> On Mon, 2026-11-02 at 19:37 +0100, Matthieu Baerts wrote:
>> Hi Shardul,
>>
>> On 28/10/2026 12:15, Shardul Bankar wrote:
>>> A subflow that is slow relative to the others can soak up
>>> connection
>>> resources and cause head-of-line blocking of the aggregate stream.
>>> This
>>> series lets the packet scheduler send less than such a subflow's
>>> congestion window by halving it, and leaves the congestion control
>>> to
>>> raise it again. It implements issue #345 [1]. This is a
>>> deliberately
>>> simple first form.
>>>
>>> - Patch 1: a prerequisite fix. The scheduler truncates the pacing
>>> rate to a
>>>   u32, which skews subflow selection on fast paths (Fixes).
>>> - Patch 2: penalise a slow subflow by halving its cwnd, triggered
>>> on its
>>>   pacing rate and gated on the fast path being cwnd-limited.
>>> - Patch 3: skip the penalty when the transfer is receive-window-
>>> limited,
>>>   where it would otherwise slow the transfer.
>>> - Patch 4: MPTcpExt counters and a dedicated mptcp_subflow_penalise
>>>   tracepoint for observability.
>>>
>>> Validation, baseline versus this series (test helper, its
>>> run_matrix.sh
>>> base-vs-patch driver, and run steps are in [2]):
>>> - No regression in the mptcp selftests.
>>> - No regression versus baseline across the buffer-limited transfers
>>>   (send-buffer, receive-window, both, sndbuf>rcvbuf): the two-path
>>> completion
>>>   time with the penalty enabled matches the unpatched kernel. Where
>>> the
>>>   slow subflow is beneficial (send-buffer-limited), the two-path
>>> transfer
>>>   stays faster than the fast path alone.
>>> - Patch 3's gate keeps the penalty from firing when receive-window-
>>> limited,
>>>   where halving cwnd would slow the transfer.
>>> - With a bufferbloated slow path, the penalty roughly halves the
>>> slow path's
>>>   queueing latency (subflow srtt), with the transfer time
>>> unchanged. It does
>>>   not reduce the aggregate out-of-order volume or move data off the
>>> slow
>>>   path; that is the role of issue #332 [3], and the two are
>>> complementary.
>>> - The conditions above do not catch one case: with the buffers left
>>> to
>>>   autotuning and a subflow that is slow because of latency rather
>>> than limited
>>>   bandwidth, penalising it costs throughput, and the cost grows as
>>> the added
>>>   delay grows.
>>>
>>> Used LLM for design, review, coding assistance.
>>>
>>> [1] https://github.com/multipath-tcp/mptcp_net-next/issues/345
>>> [2]
>>> https://github.com/shardulsdk-mpiric/linux/blob/e0f78406d921/tools/testing/selftests/net/mptcp/mptcp_sched_penalise.sh
>>>     (run_matrix.sh + runs.conf are alongside it in the same
>>> directory)
>>> [3] https://github.com/multipath-tcp/mptcp_net-next/issues/332
>>>
>>> Changes in v5:
>>> - Rebased on the current export; no code change.
>>> - Dropped the Assisted-by tags, which the kernel docs no longer
>>> describe;
>>>   tool use is noted in this cover letter instead.
>>> - Collected Abdul Wasey's Tested-by, thanks:
>>>  
>>> https://lore.kernel.org/all/20261025192320.1831707-1-w453y.me@gmail.com/
>>> - Link to v4:
>>> https://patch.msgid.link/20260911-mptcp_penalise_send_v2-v4-0-820f2be4a8fe@mpiricsoftware.com
>>
>> Thank you for the new version!
>>
>> I just noticed that Abdul's v2 series that I recently sent [1] didn't
>> get reviewed by Clashiko because it doesn't support dependences yet
>> (Sashiko does) [2]. Just to avoid both working on the same thing: are
>> you already working on a new version addressing / rejecting the last
>> review comments sent here? No rush at all, but if you didn't start
>> yet,
>> and you prefer me to look at it, I can try to do that this week. I
>> would
>> then send a version including your patches with fixes (if I can) and
>> the
>> ones I recently sent. Otherwise, there is no urgency, and I can wait.
>>
> 
> Hi Matt,
> 
> Thank you for the offer. I have not started on a new version yet, so
> please go ahead and include my patches in your series, with the fixes
> you think are needed. Feel free to reach out if I can help with
> anything.

OK, I will do that when I can while I still have my lab setup and a
"fresh" context.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Hi Shardul,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Perf: Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/40681247377

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/6f660b55cccf
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1195344


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)