[v1] mptcp: ensure snd_nxt is properly initialized on connect

[PATCH 5.10.y] mptcp: ensure snd_nxt is properly initialized on connect

Posted by Matthieu Baerts (NGI0) 1 year, 6 months ago

From: Paolo Abeni <pabeni@redhat.com>

commit fb7a0d334894206ae35f023a82cad5a290fd7386 upstream.

Christoph reported a splat hinting at a corrupted snd_una:

  WARNING: CPU: 1 PID: 38 at net/mptcp/protocol.c:1005 __mptcp_clean_una+0x4b3/0x620 net/mptcp/protocol.c:1005
  Modules linked in:
  CPU: 1 PID: 38 Comm: kworker/1:1 Not tainted 6.9.0-rc1-gbbeac67456c9 #59
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
  Workqueue: events mptcp_worker
  RIP: 0010:__mptcp_clean_una+0x4b3/0x620 net/mptcp/protocol.c:1005
  Code: be 06 01 00 00 bf 06 01 00 00 e8 a8 12 e7 fe e9 00 fe ff ff e8
  	8e 1a e7 fe 0f b7 ab 3e 02 00 00 e9 d3 fd ff ff e8 7d 1a e7 fe
  	<0f> 0b 4c 8b bb e0 05 00 00 e9 74 fc ff ff e8 6a 1a e7 fe 0f 0b e9
  RSP: 0018:ffffc9000013fd48 EFLAGS: 00010293
  RAX: 0000000000000000 RBX: ffff8881029bd280 RCX: ffffffff82382fe4
  RDX: ffff8881003cbd00 RSI: ffffffff823833c3 RDI: 0000000000000001
  RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
  R10: 0000000000000000 R11: fefefefefefefeff R12: ffff888138ba8000
  R13: 0000000000000106 R14: ffff8881029bd908 R15: ffff888126560000
  FS:  0000000000000000(0000) GS:ffff88813bd00000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007f604a5dae38 CR3: 0000000101dac002 CR4: 0000000000170ef0
  Call Trace:
   <TASK>
   __mptcp_clean_una_wakeup net/mptcp/protocol.c:1055 [inline]
   mptcp_clean_una_wakeup net/mptcp/protocol.c:1062 [inline]
   __mptcp_retrans+0x7f/0x7e0 net/mptcp/protocol.c:2615
   mptcp_worker+0x434/0x740 net/mptcp/protocol.c:2767
   process_one_work+0x1e0/0x560 kernel/workqueue.c:3254
   process_scheduled_works kernel/workqueue.c:3335 [inline]
   worker_thread+0x3c7/0x640 kernel/workqueue.c:3416
   kthread+0x121/0x170 kernel/kthread.c:388
   ret_from_fork+0x44/0x50 arch/x86/kernel/process.c:147
   ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243
   </TASK>

When fallback to TCP happens early on a client socket, snd_nxt
is not yet initialized and any incoming ack will copy such value
into snd_una. If the mptcp worker (dumbly) tries mptcp-level
re-injection after such ack, that would unconditionally trigger a send
buffer cleanup using 'bad' snd_una values.

We could easily disable re-injection for fallback sockets, but such
dumb behavior already helped catching a few subtle issues and a very
low to zero impact in practice.

Instead address the issue always initializing snd_nxt (and write_seq,
for consistency) at connect time.

Fixes: 8fd738049ac3 ("mptcp: fallback in case of simultaneous connect")
Cc: stable@vger.kernel.org
Reported-by: Christoph Paasch <cpaasch@apple.com>
Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/485
Tested-by: Christoph Paasch <cpaasch@apple.com>
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240429-upstream-net-20240429-mptcp-snd_nxt-init-connect-v1-1-59ceac0a7dcb@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
[ snd_nxt field is not available in v5.10.y: before, only write_seq was
  used, see commit eaa2ffabfc35 ("mptcp: introduce MPTCP snd_nxt") for
  more details about that. ]
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
---
 net/mptcp/protocol.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index 6be7e7592291..36fa456f42ba 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -2645,6 +2645,8 @@ static int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
 	if (subflow->request_mptcp && mptcp_token_new_connect(ssock->sk))
 		mptcp_subflow_early_fallback(msk, subflow);
 
+	WRITE_ONCE(msk->write_seq, subflow->idsn);
+
 do_connect:
 	err = ssock->ops->connect(ssock, uaddr, addr_len, flags);
 	sock->state = ssock->state;
-- 
2.43.0

Re: [PATCH 5.10.y] mptcp: ensure snd_nxt is properly initialized on connect

Posted by Greg KH 1 year, 6 months ago

On Mon, May 13, 2024 at 05:17:17PM +0200, Matthieu Baerts (NGI0) wrote:
> From: Paolo Abeni <pabeni@redhat.com>
> 
> commit fb7a0d334894206ae35f023a82cad5a290fd7386 upstream.
> 
> Christoph reported a splat hinting at a corrupted snd_una:
> 
>   WARNING: CPU: 1 PID: 38 at net/mptcp/protocol.c:1005 __mptcp_clean_una+0x4b3/0x620 net/mptcp/protocol.c:1005
>   Modules linked in:
>   CPU: 1 PID: 38 Comm: kworker/1:1 Not tainted 6.9.0-rc1-gbbeac67456c9 #59
>   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
>   Workqueue: events mptcp_worker
>   RIP: 0010:__mptcp_clean_una+0x4b3/0x620 net/mptcp/protocol.c:1005
>   Code: be 06 01 00 00 bf 06 01 00 00 e8 a8 12 e7 fe e9 00 fe ff ff e8
>   	8e 1a e7 fe 0f b7 ab 3e 02 00 00 e9 d3 fd ff ff e8 7d 1a e7 fe
>   	<0f> 0b 4c 8b bb e0 05 00 00 e9 74 fc ff ff e8 6a 1a e7 fe 0f 0b e9
>   RSP: 0018:ffffc9000013fd48 EFLAGS: 00010293
>   RAX: 0000000000000000 RBX: ffff8881029bd280 RCX: ffffffff82382fe4
>   RDX: ffff8881003cbd00 RSI: ffffffff823833c3 RDI: 0000000000000001
>   RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
>   R10: 0000000000000000 R11: fefefefefefefeff R12: ffff888138ba8000
>   R13: 0000000000000106 R14: ffff8881029bd908 R15: ffff888126560000
>   FS:  0000000000000000(0000) GS:ffff88813bd00000(0000) knlGS:0000000000000000
>   CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>   CR2: 00007f604a5dae38 CR3: 0000000101dac002 CR4: 0000000000170ef0
>   Call Trace:
>    <TASK>
>    __mptcp_clean_una_wakeup net/mptcp/protocol.c:1055 [inline]
>    mptcp_clean_una_wakeup net/mptcp/protocol.c:1062 [inline]
>    __mptcp_retrans+0x7f/0x7e0 net/mptcp/protocol.c:2615
>    mptcp_worker+0x434/0x740 net/mptcp/protocol.c:2767
>    process_one_work+0x1e0/0x560 kernel/workqueue.c:3254
>    process_scheduled_works kernel/workqueue.c:3335 [inline]
>    worker_thread+0x3c7/0x640 kernel/workqueue.c:3416
>    kthread+0x121/0x170 kernel/kthread.c:388
>    ret_from_fork+0x44/0x50 arch/x86/kernel/process.c:147
>    ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243
>    </TASK>
> 
> When fallback to TCP happens early on a client socket, snd_nxt
> is not yet initialized and any incoming ack will copy such value
> into snd_una. If the mptcp worker (dumbly) tries mptcp-level
> re-injection after such ack, that would unconditionally trigger a send
> buffer cleanup using 'bad' snd_una values.
> 
> We could easily disable re-injection for fallback sockets, but such
> dumb behavior already helped catching a few subtle issues and a very
> low to zero impact in practice.
> 
> Instead address the issue always initializing snd_nxt (and write_seq,
> for consistency) at connect time.
> 
> Fixes: 8fd738049ac3 ("mptcp: fallback in case of simultaneous connect")
> Cc: stable@vger.kernel.org
> Reported-by: Christoph Paasch <cpaasch@apple.com>
> Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/485
> Tested-by: Christoph Paasch <cpaasch@apple.com>
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> Reviewed-by: Mat Martineau <martineau@kernel.org>
> Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
> Link: https://lore.kernel.org/r/20240429-upstream-net-20240429-mptcp-snd_nxt-init-connect-v1-1-59ceac0a7dcb@kernel.org
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> [ snd_nxt field is not available in v5.10.y: before, only write_seq was
>   used, see commit eaa2ffabfc35 ("mptcp: introduce MPTCP snd_nxt") for
>   more details about that. ]
> Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
> ---
>  net/mptcp/protocol.c | 2 ++
>  1 file changed, 2 insertions(+)

Now queued up, thanks.

greg k-h