This is an attempt at issue #332, the "opportunistic retransmission" of
section 4.2 of the NSDI 2012 paper, which the out-of-tree kernel had.

I found it from a multi-WAN router: a fast link next to a slow one gets
much less than the fast link alone, because data sent on the slow link
holds back the DATA_ACK until the receive window or the send buffer is
used up. With the default tcp_rmem, a 400 Mbit link next to a
15 Mbit/240 ms one gave 67-82 Mbit in total with BBR, and 30-142 with
CUBIC. With this patch it is 320-345, and uploads gain the same.

When a DATA_ACK shows that no new data can go out and an active subflow
is idle, the data the slower subflows hold is resent on it: the ranges
from the DSS mappings in their queues, not everything from snd_una. It
only does so when the copy can arrive at least one RTT of the idle
subflow sooner, and no more than the idle subflow can send in that time.
Without these limits it cost about 1% on simult_flows.sh's unbalanced
cases, and even waking the worker for nothing showed there, so the
decision is made before scheduling it.

A subflow that had a retransmission timeout and no progress since is
left alone: the copies would keep the MPTCP-level timeout from firing,
and with it the stale detection (mptcp_join.sh's "multiple flows,
signal, link failure" caught that).

Tested in netns on the current export, against the same kernel without
the patch: nine 2-path cases with BBR and CUBIC, 6 and 32 MB tcp_rmem,
uploads and downloads, 3 runs each and 16 for the slow lossy path.
mptcp_connect.sh, mptcp_join.sh (twice) and simult_flows.sh (16 runs)
pass as they do without the patch. The 2-path cases, mptcp_connect.sh
and simult_flows.sh also ran on a KASAN + lockdep kernel, with no
report, and on a KCSAN one, whose reports are the same data races in
mptcp_subflow_get_send() and mptcp_subflow_active() that it shows
without the patch.

Questions, since this is an RFC:
- It is in the core, for every scheduler. Should it rather be a
  scheduler hook (#344), with the default scheduler using it?
- The rules (slower, one RTT of gain, budget) come from my tests. Is
  there a better signal to use here?
- mptcp_oppr_check() runs on each DATA_ACK and reads the other
  subflows' queues and RTTs without their locks: is that acceptable as a
  heuristic, or should it be done differently?
- I could not find a short simult_flows.sh case where the stock kernel
  is clearly slower: the stall needs a few seconds of traffic to build
  up. Would a counter based test (OppReinject moving on a lossy slow
  path, with the data checked) be enough?

Abdul Wasey (1):
  mptcp: opportunistic reinjection on an idle subflow

 net/mptcp/mib.c      |   2 +
 net/mptcp/mib.h      |   2 +
 net/mptcp/options.c  |   1 +
 net/mptcp/protocol.c | 291 +++++++++++++++++++++++++++++++++++++++++++
 net/mptcp/protocol.h |   3 +
 5 files changed, 299 insertions(+)

-- 
2.43.0

Hi Abdul,

(+cc Shardul)

On 25/10/2026 14:46, Abdul Wasey wrote:
> This is an attempt at issue #332, the "opportunistic retransmission" of
> section 4.2 of the NSDI 2012 paper, which the out-of-tree kernel had.
> 
> I found it from a multi-WAN router: a fast link next to a slow one gets
> much less than the fast link alone, because data sent on the slow link
> holds back the DATA_ACK until the receive window or the send buffer is
> used up. With the default tcp_rmem, a 400 Mbit link next to a
> 15 Mbit/240 ms one gave 67-82 Mbit in total with BBR, and 30-142 with
> CUBIC. With this patch it is 320-345, and uploads gain the same.
> 
> When a DATA_ACK shows that no new data can go out and an active subflow
> is idle, the data the slower subflows hold is resent on it: the ranges
> from the DSS mappings in their queues, not everything from snd_una. It
> only does so when the copy can arrive at least one RTT of the idle
> subflow sooner, and no more than the idle subflow can send in that time.
> Without these limits it cost about 1% on simult_flows.sh's unbalanced
> cases, and even waking the worker for nothing showed there, so the
> decision is made before scheduling it.
> 
> A subflow that had a retransmission timeout and no progress since is
> left alone: the copies would keep the MPTCP-level timeout from firing,
> and with it the stale detection (mptcp_join.sh's "multiple flows,
> signal, link failure" caught that).
> 
> Tested in netns on the current export, against the same kernel without
> the patch: nine 2-path cases with BBR and CUBIC, 6 and 32 MB tcp_rmem,
> uploads and downloads, 3 runs each and 16 for the slow lossy path.
> mptcp_connect.sh, mptcp_join.sh (twice) and simult_flows.sh (16 runs)
> pass as they do without the patch. The 2-path cases, mptcp_connect.sh
> and simult_flows.sh also ran on a KASAN + lockdep kernel, with no
> report, and on a KCSAN one, whose reports are the same data races in
> mptcp_subflow_get_send() and mptcp_subflow_active() that it shows
> without the patch.

Thank you for working on this.

That's indeed something we need to improve.

Before replying to the questions below, I think it would be good if you
rebase your work on top of Shardul's one:


https://lore.kernel.org/20260911-mptcp_penalise_send_v2-v4-0-820f2be4a8fe@mpiricsoftware.com

Note: this work looks good to me, but I was waiting for external reviews
(because I helped a bit) before applying it (and other urgent fixes/work
kept us busy...)

> Questions, since this is an RFC:
> - It is in the core, for every scheduler. Should it rather be a
>   scheduler hook (#344), with the default scheduler using it?

Yes, I think it should (and moved to sched.c), but having a wait for
others schedulers to re-use that. At the end, that's a decision a
scheduler can take depending on the use case. But we want that by default.

When discussing #332 with Shardul, it was clear that the current MPTCP
packet scheduler API is not flexible enough to stop sending on one
subflow and force a reinjection on another subflow. I think it would be
best to take over the following old series, rebase it, address the
comments, and validate it:


https://lore.kernel.org/all/20240527-sched_per_packet-v1-0-09a41d405f7c@gmail.com/t/#u

Once it is there, I think it should be easier to modify the scheduler to
force a reinjection, e.g. set a flag to stop pushing, then find a way to
force using another subflows (e.g. the stale flag or another one). WDYT?
Also, if this work is in place, it would help to add BPF support for the
MPTCP packet scheduler.

Note that I don't know if Shardul is still looking at that. (BTW, sorry
for delaying the work you did with #345 ... note that, in between, I
managed to get Clashiko (Netdev Sashiko) doing reviews not to be blocked
when upstreaming them: so probably we should also check what Clashiko
thinks about this series by resending the last version, to "be prepared"
or to fix issues if any we missed during dev/review).

> - The rules (slower, one RTT of gain, budget) come from my tests. Is
>   there a better signal to use here?

I guess imitating what the old fork was doing is a good starting point.

> - mptcp_oppr_check() runs on each DATA_ACK and reads the other
>   subflows' queues and RTTs without their locks: is that acceptable as a
>   heuristic, or should it be done differently?

Was the fork not limiting this to once per RTT? In any cases, it will be
needed to measure the impact on high perf transfers.

> - I could not find a short simult_flows.sh case where the stock kernel
>   is clearly slower: the stall needs a few seconds of traffic to build
>   up. Would a counter based test (OppReinject moving on a lossy slow
>   path, with the data checked) be enough?

I'm (still :( ) working on a perf lab that might help to measure the
behaviour in different conditions. Would that help here? I don't think
we can have such tests in the selftests: stability needs to be measured
on long runs and in "stable" environments I guess.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Hi Matt,

I'll rebase on Shardul's series. I'll also test his v4 on my setup and
reply on that thread, since it's still waiting for reviews.

Moving it to the scheduler makes sense. The CI also caught it on v1: the
BPF "first" test saw bytes on its second subflow because of the
reinjection.

Gregory, are you still planning to continue the scheduler API series? If
not, I can pick it up. Shardul, same if you already started on it.

About once per RTT: in the old fork that limit was on the cwnd halving.
The reinjection itself only took the head of the rtx queue, when the
holder's srtt was more than 4x the idle subflow's. I'll check the cost on
fast transfers.

The perf lab would help, yes. I can share my netns cases if that's
useful.

I've fixed what Sashiko found locally, and will fold it into the
scheduler version instead of sending a v2 of this one.

Thanks,
Abdul Wasey

Hi Abdul,

Thank you for your reply.

(Do you mind quoting the text you are replying to, please: it helps to
remember the context, especially when switching between many topics per
day :) )

https://subspace.kernel.org/etiquette.html#do-not-top-post-when-replying

While at it, it usually helps to send one version per day, and ideally
wait for Netdev's Sashiko review that can take ~48h, then reply to the
comments on the list before sending a new version. Except if there is an
urgency to fix something of course. By doing that, it's easier for human
reviewers to follow and give guidance. Without that, people might only
look at the last version, and likely after a while or waiting for a
comment to say "Sashiko is happy, please review now". Note that AI
reviews can still miss things or focus on details instead of the big
picture: they might guide a dev to a wrong path.

On 25/10/2026 16:58, Abdul Wasey wrote:
> I'll rebase on Shardul's series. I'll also test his v4 on my setup and
> reply on that thread, since it's still waiting for reviews.

Thank you, that's very helpful!

> Moving it to the scheduler makes sense. The CI also caught it on v1: the
> BPF "first" test saw bytes on its second subflow because of the
> reinjection.
> 
> Gregory, are you still planning to continue the scheduler API series? If
> not, I can pick it up.

For Gregory, I'm sure he is not planning to continue working on that.

> Shardul, same if you already started on it.
> 
> About once per RTT: in the old fork that limit was on the cwnd halving.
> The reinjection itself only took the head of the rtx queue, when the
> holder's srtt was more than 4x the idle subflow's. I'll check the cost on
> fast transfers.

Thanks!

Please also note that for optimisation purposes, the current
implementation tries to push as much as possible on the selected
subflow: it can be quickly costly to stop the delegation and alternate
between subflows.

(Of course, this cost might not be a problem in some cases, e.g. a
"redundant" scheduler is not designed for fast transfers, but for low
latency.)

> The perf lab would help, yes. I can share my netns cases if that's
> useful.

That could indeed. But I'm going to try to push what I have today: it
might be better to work on improving the same thing.

> I've fixed what Sashiko found locally, and will fold it into the
> scheduler version instead of sending a v2 of this one.
Thank you!

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Hi Abdul,

On 26/10/2026 11:13, Matthieu Baerts wrote:

(...)

> On 25/10/2026 16:58, Abdul Wasey wrote:
>> I'll rebase on Shardul's series. I'll also test his v4 on my setup and
>> reply on that thread, since it's still waiting for reviews.
> 
> Thank you, that's very helpful!

Do not hesitate to send what you have without waiting for the last
details from Shardul's series to be fixed. You can simply mention the
base series in your cover letter as explained here:

  https://github.com/multipath-tcp/mptcp_net-next/wiki/CI

e.g.

  Based-on: <20261028-mptcp_penalise_send_v2-v5-0-32fb379ce117@mpiricsoftware.com>

(Sashiko also supports b4's dependencies tags, but not Patchew -- it is
 fine to add both.)

(...)

>> The perf lab would help, yes. I can share my netns cases if that's
>> useful.
> 
> That could indeed. But I'm going to try to push what I have today: it
> might be better to work on improving the same thing.

It took me a bit more time to fix the last details, but it should be
usable now:

  https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/PERF.md#tracking-regressions-locally

There are many use-cases (based on previous work). The idea is that
the regressions are found automatically by looking at the distribution
over multiple runs. So it works best if you execute the tests a lot of
time on a base branch, then check on top of your modifications (always
under the same conditions).

The whole suite might take a lot of time (1 to 2h), but it is possible
to execute only some parts. Do not hesitate to suggest modifications or
send PRs ;)
Or tell me if that doesn't work for what you are doing.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Hi Abdul, Matt,

On Mon, 2026-10-26 at 11:13 +0100, Matthieu Baerts wrote:
> 
> 
> > Moving it to the scheduler makes sense. The CI also caught it on
> > v1: the
> > BPF "first" test saw bytes on its second subflow because of the
> > reinjection.
> > 
> > Gregory, are you still planning to continue the scheduler API
> > series? If
> > not, I can pick it up.
> 
> For Gregory, I'm sure he is not planning to continue working on that.
> 
> > Shardul, same if you already started on it.
> > 

I have not started on it, so please go ahead. I am happy to review the
series as it comes.

Matt, sorry for going quiet on this after we discussed #332.

As you suggested, I am resending the penalize subflow series rebased on
the current export so that Clashiko can have a look.

Abdul, one point on the RFC that should still apply once it moves to
the scheduler, since you asked whether there is a better signal:

mptcp_oppr_blocked() treats an empty send queue (mptcp_send_head() ==
NULL) as "blocked". first_pending is NULL once everything the
application has written so far has been handed to a subflow, which
happens when the send buffer is full, but also when the application
simply has nothing more to write at that moment. In the second case
nothing is stalled, yet the data still held by the slower subflow gets
resent on the idle one at the next DATA_ACK. The commit message
describes the trigger as the receive window or the send buffer being
used up, and the window case is covered by the mptcp_wnd_end() check;
the send buffer case is not checked as such.

Requiring the msk send buffer to be (nearly) full on that branch, i.e.
sk_wmem_queued >= sk_sndbuf as __sk_stream_memory_free() tests, would
separate the two. Note that mptcp_stream_memory_free() alone would not
help here: it only counts the bytes not yet sent, which are zero in
exactly this case.

One more observation: #345 and this work both act on "this subflow is
slow", one by halving its cwnd and the other by copying its data
elsewhere. Once both are in the scheduler it may be worth looking at
how they interact, e.g. whether a subflow being penalised should also
be a source for reinjection. Happy to help with that part.

> 
Thanks,
Shardul

Hi Abdul,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Unstable: 1 failed test(s): bpftest_test_progs_mptcp ⚠️ 
- Perf: Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/40200388982

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/41031047963c
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1193018


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)