From nobody Mon May 6 22:01:16 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:a05:6a06:1624:b0:4ae:a3e3:8373 with SMTP id t4csp333419pip; Mon, 11 Apr 2022 03:40:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzcGdRvMKAy7Y2GfnweYwGOQmy5cSMosqvqACZ/mHPfL9bW7qrjjckFmvGJVB5FRs7Dm36D X-Received: by 2002:a63:ad45:0:b0:382:2459:5bc6 with SMTP id y5-20020a63ad45000000b0038224595bc6mr26573668pgo.474.1649673629677; Mon, 11 Apr 2022 03:40:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1649673629; cv=none; d=google.com; s=arc-20160816; b=Kb25I2592OJ9YAqHTP2hk0SQ9R6lsFOCTvWefzGE63ybjCchXRzLR9tNJIsd2uGQVF lRed8rGVxi5IT//bnujwWVy60WO6LZzN7cwYGMXmkFfdyw3RYS47c9MEXdKG0S/6CM4l dQvZUWPM+RETwPTe1laW9CM9OUY52yqheqaKjXUOj5lCgR9td/nUANf34/aG6E/5hilR qOsOMFQaJj5TieraGiUE09rUEw846+8G/YVL7OnabnY7jaC7gj03+HJiFc5ysFawo46W NkfKp0rWpSqx+/UNrgAR6ymg4C9CU/ZoyW+6+IWOc5Afe1rFHr/GuLCuWtjHD84ZfibZ 6V3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=BteYi2BtqALS/ADX+Zx6fkxtikDFCys0tph+cLiN04g=; b=09nMBWibmTShZEZta01Wc4JW/YwX2oJMywHUKYEvcE9Z2gYgq26+EP3RLkpAXb2ssi kwawwzL+t5px/gbynPiqih2ra8QZlk44aMba2cNEcv9fnjIwCyGbG0lfjn0QI5nCKn3N f83Of7Z5y6ZD4jZQvmjZxpcz+4/j+AAz2o7njrxOwhJ/Vdu4JwTZV+VCOAOfWCP6qF8r 6LAf8vA0efzTxbwjHW2AnIkAJrPmWbybzGnX2+/UcczSF5x8H/XQDIFYUvcANOoD6I86 +EHbXw/7u5/TqkuSXLXa1Ue0zC1HJb9KHLwWFJK6HFR8eOapn+CFvgpSsjevbu5bni61 dyZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=S89tNg92; spf=pass (google.com: domain of mptcp+bounces-4715-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4715-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from sjc.edge.kernel.org (sjc.edge.kernel.org. [2604:1380:1000:8100::1]) by mx.google.com with ESMTPS id x13-20020a1709027c0d00b00154319ca2ddsi7573585pll.397.2022.04.11.03.40.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Apr 2022 03:40:29 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-4715-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) client-ip=2604:1380:1000:8100::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=S89tNg92; spf=pass (google.com: domain of mptcp+bounces-4715-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4715-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sjc.edge.kernel.org (Postfix) with ESMTPS id 20C6F3E0EC7 for ; Mon, 11 Apr 2022 10:40:29 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 566E0139A; Mon, 11 Apr 2022 10:40:27 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7D601393 for ; Mon, 11 Apr 2022 10:40:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649673624; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BteYi2BtqALS/ADX+Zx6fkxtikDFCys0tph+cLiN04g=; b=S89tNg92Z+IqJe1HWHkCUM9XvNt38S7sPlxxA6HGLVSXUqWdUjg1mNgri+BjBR9ycz1y35 4YBtwxszBQXF5cRtagJAAQ7pZvuVIiHjemRx+bRSNu/BmI1Pah4W/iB9FNu/zCJd3Ugded YIv3mlx3njZoH/fJz6QUaxmJJOwfVZ8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-312-u4CP9Kb6NzKgO6_CcUJQdA-1; Mon, 11 Apr 2022 06:40:23 -0400 X-MC-Unique: u4CP9Kb6NzKgO6_CcUJQdA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 75CB31C05AE7 for ; Mon, 11 Apr 2022 10:40:23 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.193.187]) by smtp.corp.redhat.com (Postfix) with ESMTP id D0D5B40E80E0 for ; Mon, 11 Apr 2022 10:40:22 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [RFC PATCH 1/4] mptcp: really share subflow snd_wnd Date: Mon, 11 Apr 2022 12:40:02 +0200 Message-Id: <603102b10020153dcfcece067ecc009ef022fd4a.1649672265.git.pabeni@redhat.com> In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" As per RFC, mptcp subflows use a "shared" snd_wnd: the effective window is the maximum among the current values received on all subflows. Without such feature a data transfer using multiple subflows could block. Window sharing is currently implemented in the RX side: __tcp_select_window uses the mptcp-level receive buffer to compute the announced window. That is not enough: the TCP stack will stick to the window size received on the given subflow; we need to propagate the msk window value on each subflow at xmit time. Change the packet scheduler to ignore the subflow level window and and use instead the msk level one Signed-off-by: Paolo Abeni --- net/mptcp/protocol.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index b2c654992de0..9eb33681d00d 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -1141,19 +1141,20 @@ struct mptcp_sendmsg_info { bool data_lock_held; }; =20 -static int mptcp_check_allowed_size(struct mptcp_sock *msk, u64 data_seq, - int avail_size) +static int mptcp_check_allowed_size(const struct mptcp_sock *msk, struct s= ock *ssk, + u64 data_seq, int avail_size) { u64 window_end =3D mptcp_wnd_end(msk); + u64 mptcp_snd_wnd; =20 if (__mptcp_check_fallback(msk)) return avail_size; =20 - if (!before64(data_seq + avail_size, window_end)) { - u64 allowed_size =3D window_end - data_seq; + mptcp_snd_wnd =3D window_end - data_seq; + avail_size =3D min_t(unsigned int, mptcp_snd_wnd, avail_size); =20 - return min_t(unsigned int, allowed_size, avail_size); - } + if (unlikely(tcp_sk(ssk)->snd_wnd < mptcp_snd_wnd)) + tcp_sk(ssk)->snd_wnd =3D min_t(u64, U32_MAX, mptcp_snd_wnd); =20 return avail_size; } @@ -1305,7 +1306,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct= sock *ssk, } =20 /* Zero window and all data acked? Probe. */ - copy =3D mptcp_check_allowed_size(msk, data_seq, copy); + copy =3D mptcp_check_allowed_size(msk, ssk, data_seq, copy); if (copy =3D=3D 0) { u64 snd_una =3D READ_ONCE(msk->snd_una); =20 @@ -1498,11 +1499,16 @@ static struct sock *mptcp_subflow_get_send(struct m= ptcp_sock *msk) * to check that subflow has a non empty cwin. */ ssk =3D send_info[SSK_MODE_ACTIVE].ssk; - if (!ssk || !sk_stream_memory_free(ssk) || !tcp_sk(ssk)->snd_wnd) + if (!ssk || !sk_stream_memory_free(ssk)) return NULL; =20 - burst =3D min_t(int, MPTCP_SEND_BURST_SIZE, tcp_sk(ssk)->snd_wnd); + burst =3D min_t(int, MPTCP_SEND_BURST_SIZE, mptcp_wnd_end(msk) - msk->snd= _nxt); wmem =3D READ_ONCE(ssk->sk_wmem_queued); + if (!burst) { + msk->last_snd =3D NULL; + return ssk; + } + subflow =3D mptcp_subflow_ctx(ssk); subflow->avg_pacing_rate =3D div_u64((u64)subflow->avg_pacing_rate * wmem= + READ_ONCE(ssk->sk_pacing_rate) * burst, --=20 2.35.1 From nobody Mon May 6 22:01:16 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:a05:6a06:1624:b0:4ae:a3e3:8373 with SMTP id t4csp333433pip; Mon, 11 Apr 2022 03:40:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxdANwZypAldGpvcrTWR4Zaiqi3m3BkRAUHXZ62bC9ova89J1t/iTQewGFlfPsxHAn53bc2 X-Received: by 2002:a17:902:b213:b0:156:5a00:325f with SMTP id t19-20020a170902b21300b001565a00325fmr31442034plr.127.1649673630995; Mon, 11 Apr 2022 03:40:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1649673630; cv=none; d=google.com; s=arc-20160816; b=oZaHfekdG3UwHS5ZZV8YtigwPzX3/bTP4m145/M5eyPCK/zM4v7M1PUFPCHyuRsWne sv8gCYEwyrNzQ/qxXpoIJ6Q0quIn4ML62DnZrg99UMRIejx0A0UIlluD7u4VJcBnVaQa sDWYOM/uT5wbqZLy/zF+AbLtl4hpJ5CuzK+3zMKXWCrEq5+VxiHpkURF545DYBzqSwf2 QFQnzoSy1bJlHz82GlVZS1zaupbi2+6+gY+bRBgDI/B/8X/hf8exTdJj+9t6S1/AbobM iMjS6DAYo9OR0Kil0kOVk/ctw1OGffHW/L466EZvh1IrT8soQMD/BCA8bbjPRpdI94MP MT4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=80PASlXeRmpxqDljRTLcZIVvgKZmieZXIsbb6r5EQ+k=; b=TvAGplbpmNMEyCbkleb/dnzcp60ZbV7Pk61o3EdgOnVQ99WDcOehzoCUokM8iH9Y/B 6CPrMELxcyco1pQC1A25iK+xPtCIpfurTiAqbWKh4p2S9TLOVly3TlfOKBzve9aczMW/ YP+Xt71TDzB1ej2XCUTkq37249NdVGiTzYySu8fbldz6rP862wnePg9Hltn4BjZqpbkH zkbPfMi1TqG5fMQyZbdx3TMxu1pkWlQLvF2KPngBYcRlb6i3eaGNHi3W6l4VeqqJQAgm fInExHKBuCD8qyw1I5p1qu+QCsnHT1oUVaVgVihpEY1S3f9kt6W/zSg6QhYuKp7Wi7ZP Gkeg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hgGwhMDz; spf=pass (google.com: domain of mptcp+bounces-4716-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4716-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from sjc.edge.kernel.org (sjc.edge.kernel.org. [2604:1380:1000:8100::1]) by mx.google.com with ESMTPS id x13-20020a1709027c0d00b00154319ca2ddsi7573585pll.397.2022.04.11.03.40.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Apr 2022 03:40:30 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-4716-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) client-ip=2604:1380:1000:8100::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hgGwhMDz; spf=pass (google.com: domain of mptcp+bounces-4716-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4716-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sjc.edge.kernel.org (Postfix) with ESMTPS id 6E2F03E0F4B for ; Mon, 11 Apr 2022 10:40:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 566461846; Mon, 11 Apr 2022 10:40:28 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A6C6138A for ; Mon, 11 Apr 2022 10:40:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649673625; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=80PASlXeRmpxqDljRTLcZIVvgKZmieZXIsbb6r5EQ+k=; b=hgGwhMDzhqhHZioVmDBHjqBth0tZ8K2MXg5IOvhcmEwqxucqKU2PVYnrOm20y6ODJc+M0o 3nNI9y2kjGtqy4UZtNKyPoJ4BRDYuA4IAOdPOrYQSvKINK+PhLcdj8UO2o5sSbN/bmV/u4 DbSHg1e25okCDJd7LZKZJj1w+oYiKUw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-108-8xIKtJXOPTyDxgtFFTpPKw-1; Mon, 11 Apr 2022 06:40:24 -0400 X-MC-Unique: 8xIKtJXOPTyDxgtFFTpPKw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3DA1E803D67 for ; Mon, 11 Apr 2022 10:40:24 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.193.187]) by smtp.corp.redhat.com (Postfix) with ESMTP id BED7740E80E0 for ; Mon, 11 Apr 2022 10:40:23 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [RFC PATCH 2/4] mptcp: add mib for xmit window sharing Date: Mon, 11 Apr 2022 12:40:03 +0200 Message-Id: In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" Bump a counter for counter when snd_wnd is shared among subflow, for observability's sake. Signed-off-by: Paolo Abeni --- net/mptcp/mib.c | 1 + net/mptcp/mib.h | 1 + net/mptcp/protocol.c | 4 +++- 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/net/mptcp/mib.c b/net/mptcp/mib.c index d93a8c9996fd..6a6f8151375a 100644 --- a/net/mptcp/mib.c +++ b/net/mptcp/mib.c @@ -56,6 +56,7 @@ static const struct snmp_mib mptcp_snmp_list[] =3D { SNMP_MIB_ITEM("RcvPruned", MPTCP_MIB_RCVPRUNED), SNMP_MIB_ITEM("SubflowStale", MPTCP_MIB_SUBFLOWSTALE), SNMP_MIB_ITEM("SubflowRecover", MPTCP_MIB_SUBFLOWRECOVER), + SNMP_MIB_ITEM("SndWndShared", MPTCP_MIB_SNDWNDSHARED), SNMP_MIB_SENTINEL }; =20 diff --git a/net/mptcp/mib.h b/net/mptcp/mib.h index 529d07af9e14..2411510bef66 100644 --- a/net/mptcp/mib.h +++ b/net/mptcp/mib.h @@ -49,6 +49,7 @@ enum linux_mptcp_mib_field { MPTCP_MIB_RCVPRUNED, /* Incoming packet dropped due to memory limit */ MPTCP_MIB_SUBFLOWSTALE, /* Subflows entered 'stale' status */ MPTCP_MIB_SUBFLOWRECOVER, /* Subflows returned to active status after bei= ng stale */ + MPTCP_MIB_SNDWNDSHARED, /* Subflow snd wnd is overridden by msk's one */ __MPTCP_MIB_MAX }; =20 diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 9eb33681d00d..6ef530271845 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -1153,8 +1153,10 @@ static int mptcp_check_allowed_size(const struct mpt= cp_sock *msk, struct sock *s mptcp_snd_wnd =3D window_end - data_seq; avail_size =3D min_t(unsigned int, mptcp_snd_wnd, avail_size); =20 - if (unlikely(tcp_sk(ssk)->snd_wnd < mptcp_snd_wnd)) + if (unlikely(tcp_sk(ssk)->snd_wnd < mptcp_snd_wnd)) { tcp_sk(ssk)->snd_wnd =3D min_t(u64, U32_MAX, mptcp_snd_wnd); + MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_SNDWNDSHARED); + } =20 return avail_size; } --=20 2.35.1 From nobody Mon May 6 22:01:16 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:a05:6a06:1624:b0:4ae:a3e3:8373 with SMTP id t4csp333444pip; Mon, 11 Apr 2022 03:40:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzzpB5up99WJ0TbsgqGhMH4ZMCwcHM/VW/MqBfMv0yHC1eWrBGY4F8wSrZLi9m5VxsbXQ55 X-Received: by 2002:a17:90a:4604:b0:1bc:8bdd:4a63 with SMTP id w4-20020a17090a460400b001bc8bdd4a63mr35970142pjg.147.1649673631934; Mon, 11 Apr 2022 03:40:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1649673631; cv=none; d=google.com; s=arc-20160816; b=E1lp93QDoRD68xSgBlSJGEbcE2Bl2obozXu4tDIBwF56vHDq8aOb0/6II5GuoZkO3x IDiuy/lahzmc/5zODo9aEfvbQnSNVxy8PRw5+DbS2sXSRSfBNmbtu/8XiLOZmhKokL9w PSTjL3+hDsujaJGFhMs4Y4l8RfDQbVqAYxDol0wr4C5zRPs/ueqlDt3daBx1shiz4BFO eujV7fRw9Qcp9OxaeZJVQp+TUKrWJaFrlB4Pncq471BpXdk22AfU+3tSmaXqXEjWAUKQ eX0SNq5AqUqq+8deJt3hwCbRIOgoPTnhS/9akmMOTEOgDks+sSl+Mud9nDQ4vHoZK31Q AWrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=uv2BSJlwNpSLP1os0v73gsDuNbUqAMg/sfTV5Q1HBdU=; b=biukboBvrv2Uypu3X2JhvglRROCc4N4j5ECF1J+M3yOCU/0+DvIiyIWeGkH0mIr+ns 0OlUImWbQCXE5bR/AWBgeI7++V+CQl8PMm/t+zra/4lWnGaJTz3a+IIPBFaz+4esi8KU F05s5OQZg0NUDksp77J3s1kG5+jBY7s4IOLsbC5mlTzCuV5o8Rb2SFtF7NRcfeo2uCHr OrBDxzm7hjKeveZv665RZNapEeCYwHBcNSvWWuArvszPKV2pGeZ14yDg0JWpNIqLjYYj vGoebffr7CdnPREVKKfHrmLhZsVUKLNxrcOW/xxzuE1W8mrm+JdQFOdkqGM4IwzfOIeC fsuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PhSmnzqU; spf=pass (google.com: domain of mptcp+bounces-4717-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4717-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from sjc.edge.kernel.org (sjc.edge.kernel.org. [2604:1380:1000:8100::1]) by mx.google.com with ESMTPS id r22-20020a635d16000000b0039d430ff61dsi3953692pgb.184.2022.04.11.03.40.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Apr 2022 03:40:31 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-4717-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) client-ip=2604:1380:1000:8100::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PhSmnzqU; spf=pass (google.com: domain of mptcp+bounces-4717-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4717-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sjc.edge.kernel.org (Postfix) with ESMTPS id 9E9333E0F53 for ; Mon, 11 Apr 2022 10:40:31 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 27B331847; Mon, 11 Apr 2022 10:40:29 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B1D71840 for ; Mon, 11 Apr 2022 10:40:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649673626; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uv2BSJlwNpSLP1os0v73gsDuNbUqAMg/sfTV5Q1HBdU=; b=PhSmnzqU/NVxwANQlShWgRvsWcs+pzVxY0izVMNyk4xfTPT5zA8/D5Og4CRu01w0wb8SE7 MhDThAMEIIcdhWeEK7eQKIkYRoPUgU4UaExtfz/ahUaPudb/Dm5z4CJs7Y2Bt2fkRxzxVg 20oWSuaQGbqUyCCDtLLGPIQDe9Es/KQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-117-a4Ubyb2APKWrBi3FvCS4Dg-1; Mon, 11 Apr 2022 06:40:25 -0400 X-MC-Unique: a4Ubyb2APKWrBi3FvCS4Dg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 06B6686B8A3 for ; Mon, 11 Apr 2022 10:40:25 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.193.187]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8425140E80E0 for ; Mon, 11 Apr 2022 10:40:24 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [RFC PATCH 3/4] mptcp: never shrink offered window Date: Mon, 11 Apr 2022 12:40:04 +0200 Message-Id: In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" As per RFC, the offered MPTCP-level window should never shrink. While we currently track the right edge, we don't enforce the above constraint on the wire. Additionally, concurrent xmit on different subflows can end-up in erroneous right edge update. Address the above explicitly updating the announced window and protecting the update with an additional atomic operation (sic) Signed-off-by: Paolo Abeni --- include/net/mptcp.h | 2 +- net/ipv4/tcp_output.c | 2 +- net/mptcp/options.c | 57 +++++++++++++++++++++++++++++++++++++------ 3 files changed, 51 insertions(+), 10 deletions(-) diff --git a/include/net/mptcp.h b/include/net/mptcp.h index 877077b53200..5706d132713e 100644 --- a/include/net/mptcp.h +++ b/include/net/mptcp.h @@ -125,7 +125,7 @@ bool mptcp_established_options(struct sock *sk, struct = sk_buff *skb, struct mptcp_out_options *opts); bool mptcp_incoming_options(struct sock *sk, struct sk_buff *skb); =20 -void mptcp_write_options(__be32 *ptr, const struct tcp_sock *tp, +void mptcp_write_options(__be32 *ptr, struct tcp_sock *tp, struct mptcp_out_options *opts); =20 void mptcp_diag_fill_info(struct mptcp_sock *msk, struct mptcp_info *info); diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index c221f3bce975..edad33b9db11 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -444,7 +444,7 @@ struct tcp_out_options { struct mptcp_out_options mptcp; }; =20 -static void mptcp_options_write(__be32 *ptr, const struct tcp_sock *tp, +static void mptcp_options_write(__be32 *ptr, struct tcp_sock *tp, struct tcp_out_options *opts) { #if IS_ENABLED(CONFIG_MPTCP) diff --git a/net/mptcp/options.c b/net/mptcp/options.c index e05d9458a025..ffc0292da041 100644 --- a/net/mptcp/options.c +++ b/net/mptcp/options.c @@ -1224,20 +1224,61 @@ bool mptcp_incoming_options(struct sock *sk, struct= sk_buff *skb) return true; } =20 -static void mptcp_set_rwin(const struct tcp_sock *tp) +static void mptcp_set_rwin(struct tcp_sock *tp, void *opt) { const struct sock *ssk =3D (const struct sock *)tp; - const struct mptcp_subflow_context *subflow; + struct mptcp_subflow_context *subflow; + u64 ack_seq, rcv_wnd_old, rcv_wnd_new; struct mptcp_sock *msk; - u64 ack_seq; + struct tcphdr *th; + u32 new_win; + u64 win; =20 subflow =3D mptcp_subflow_ctx(ssk); msk =3D mptcp_sk(subflow->conn); =20 - ack_seq =3D READ_ONCE(msk->ack_seq) + tp->rcv_wnd; + ack_seq =3D READ_ONCE(msk->ack_seq); + rcv_wnd_new =3D ack_seq + tp->rcv_wnd; + + rcv_wnd_old =3D READ_ONCE(msk->rcv_wnd_sent); + if (after64(rcv_wnd_new, rcv_wnd_old)) { + u64 rcv_wnd; =20 - if (after64(ack_seq, READ_ONCE(msk->rcv_wnd_sent))) - WRITE_ONCE(msk->rcv_wnd_sent, ack_seq); + for (;;) { + rcv_wnd =3D cmpxchg64(&msk->rcv_wnd_sent, rcv_wnd_old, rcv_wnd_new); + + if (rcv_wnd =3D=3D rcv_wnd_old) + break; + if (before64(rcv_wnd_new, rcv_wnd)) + goto raise_win; + rcv_wnd_old =3D rcv_wnd; + }; + return; + } + + if (rcv_wnd_new !=3D rcv_wnd_old) { + +raise_win: + th =3D opt - sizeof(struct tcphdr); + win =3D rcv_wnd_old - ack_seq; + tp->rcv_wnd =3D min_t(u64, win, U32_MAX); + new_win =3D tp->rcv_wnd; + + /* Make sure we do not exceed the maximum possible + * scaled window. + */ + if (unlikely(th->syn)) + new_win =3D min(new_win, 65535U) << tp->rx_opt.rcv_wscale; + if (!tp->rx_opt.rcv_wscale && + sock_net(ssk)->ipv4.sysctl_tcp_workaround_signed_windows) + new_win =3D min(new_win, MAX_TCP_WINDOW); + else + new_win =3D min(new_win, (65535U << tp->rx_opt.rcv_wscale)); + + /* RFC1323 scaling applied */ + new_win >>=3D tp->rx_opt.rcv_wscale; + th->window =3D htons(new_win); + } } =20 u16 __mptcp_make_csum(u64 data_seq, u32 subflow_seq, u16 data_len, __wsum = sum) @@ -1265,7 +1306,7 @@ static u16 mptcp_make_csum(const struct mptcp_ext *mp= ext) ~csum_unfold(mpext->csum)); } =20 -void mptcp_write_options(__be32 *ptr, const struct tcp_sock *tp, +void mptcp_write_options(__be32 *ptr, struct tcp_sock *tp, struct mptcp_out_options *opts) { const struct sock *ssk =3D (const struct sock *)tp; @@ -1554,7 +1595,7 @@ void mptcp_write_options(__be32 *ptr, const struct tc= p_sock *tp, } =20 if (tp) - mptcp_set_rwin(tp); + mptcp_set_rwin(tp, opts); } =20 __be32 mptcp_get_reset_option(const struct sk_buff *skb) --=20 2.35.1 From nobody Mon May 6 22:01:16 2024 Delivered-To: wpasupplicant.patchew@gmail.com Received: by 2002:a05:6a06:1624:b0:4ae:a3e3:8373 with SMTP id t4csp333450pip; Mon, 11 Apr 2022 03:40:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJztxHztSP5sjLoLXMU1RePuyXaOoi588XzIzF0wTox9zHrisBUhenx/7DIVxD9M/VN6aEdx X-Received: by 2002:a17:90b:3b8c:b0:1c6:ed78:67e2 with SMTP id pc12-20020a17090b3b8c00b001c6ed7867e2mr36246113pjb.163.1649673633366; Mon, 11 Apr 2022 03:40:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1649673633; cv=none; d=google.com; s=arc-20160816; b=V2XUtzk1ifFcvswq34rODXZ8Gp7tykUs2iQNSmUKcLfCRmgFqbSp/GqDoq7FxhIqAj g3fhGEcNVuYIRaCHDEucP7ed3wchIQsA6vjHqmTAjHkw9Ni53GCQJWzE4n/xjTvdl7DG CUDi4RFQW1jvVEzukNJTHEZ8ZeMjCg/GEybwp5MPqlw+Qce7rEz9ccn1fDdPGIQEpNEp az/WY1C7BWgjriQeQJJSZONTPWCy8ejI49UuLxnjnJSlhQiVyymMITP2aIUnMd5s+BdA 14BCoxind6mxqeaI9od0r3O/3cg0ImIhgzHcD9ukkyVvxYmeAB+u0X/o/JGQDWcIo/GD ypbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:to:from:dkim-signature; bh=PpDc4gVMBDRc+/UUbohG0qL7+IGhwbwPzENfYTvKhmo=; b=lMhLmOLx1jGahK7YTTfZXx/lf5mt14xbxUwoJkmjXDZHbhW0kj4mPyQbMAzYvY394r idiL2tcyy/7P6wmN3cveSXFI6AkoZoJnjxg9P679st73B9R9vdSy/yUEvtemsMdKdejK j/7MNBYCvqPxoitmj2tOlfcJ0oKkepiWPPzA8+jxG9LGEiQGvAvy6o8oxoOhbkzlNv+i m807ExAHe4ANi9x50Ndnd8GYmif9M3h2hde4xmwt8Uo75ViXLyvkD45IDvcoIdcLg6Ib Q1/m6qpz/hUFfNRxCBSWW8QD4OTIpNYpSds+Z4K0hsDiwcLZZUJCUneIbgbt815gmcdI uD+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PTO18LU5; spf=pass (google.com: domain of mptcp+bounces-4718-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4718-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from sjc.edge.kernel.org (sjc.edge.kernel.org. [2604:1380:1000:8100::1]) by mx.google.com with ESMTPS id l16-20020a17090a599000b001cb91d2108fsi3255791pji.117.2022.04.11.03.40.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Apr 2022 03:40:33 -0700 (PDT) Received-SPF: pass (google.com: domain of mptcp+bounces-4718-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) client-ip=2604:1380:1000:8100::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PTO18LU5; spf=pass (google.com: domain of mptcp+bounces-4718-wpasupplicant.patchew=gmail.com@lists.linux.dev designates 2604:1380:1000:8100::1 as permitted sender) smtp.mailfrom="mptcp+bounces-4718-wpasupplicant.patchew=gmail.com@lists.linux.dev"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sjc.edge.kernel.org (Postfix) with ESMTPS id D18593E0F29 for ; Mon, 11 Apr 2022 10:40:32 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7EF9B1840; Mon, 11 Apr 2022 10:40:29 +0000 (UTC) X-Original-To: mptcp@lists.linux.dev Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08BE51844 for ; Mon, 11 Apr 2022 10:40:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649673626; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PpDc4gVMBDRc+/UUbohG0qL7+IGhwbwPzENfYTvKhmo=; b=PTO18LU50Jo6CKKceOLEq1K+TYKZb+qWjo/0xryr6Dwbkn7PiTdZCfqZ8KcUBioPb048aN MeaigL0FoyamAtzcABXKmhm0rzQLHMIfmLXM2fbsBIUcNCCw/nNdRIqRXqYhsEF4F5RLgJ 98WGXngZJyUhrSys3aemAK0sAF+vKxo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-424-cqfo8of6Oiipd-U1vfwPbw-1; Mon, 11 Apr 2022 06:40:26 -0400 X-MC-Unique: cqfo8of6Oiipd-U1vfwPbw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C5454805F7C for ; Mon, 11 Apr 2022 10:40:25 +0000 (UTC) Received: from gerbillo.redhat.com (unknown [10.39.193.187]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5529040E80E0 for ; Mon, 11 Apr 2022 10:40:25 +0000 (UTC) From: Paolo Abeni To: mptcp@lists.linux.dev Subject: [RFC PATCH 4/4] mptcp: add more offered MIBs counter. Date: Mon, 11 Apr 2022 12:40:05 +0200 Message-Id: <66a6c06373c3905d5b0ecf4fc082cc3750f5479b.1649672265.git.pabeni@redhat.com> In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pabeni@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8"; x-default="true" Track the exceptional handling of MPTCP-level offered window with a few more counters for observability. Signed-off-by: Paolo Abeni --- net/mptcp/mib.c | 3 +++ net/mptcp/mib.h | 5 +++++ net/mptcp/options.c | 6 +++++- 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/net/mptcp/mib.c b/net/mptcp/mib.c index 6a6f8151375a..0dac2863c6e1 100644 --- a/net/mptcp/mib.c +++ b/net/mptcp/mib.c @@ -57,6 +57,9 @@ static const struct snmp_mib mptcp_snmp_list[] =3D { SNMP_MIB_ITEM("SubflowStale", MPTCP_MIB_SUBFLOWSTALE), SNMP_MIB_ITEM("SubflowRecover", MPTCP_MIB_SUBFLOWRECOVER), SNMP_MIB_ITEM("SndWndShared", MPTCP_MIB_SNDWNDSHARED), + SNMP_MIB_ITEM("RcvWndShared", MPTCP_MIB_RCVWNDSHARED), + SNMP_MIB_ITEM("RcvWndConflictUpdate", MPTCP_MIB_RCVWNDCONFLICTUPDATE), + SNMP_MIB_ITEM("RcvWndConflict", MPTCP_MIB_RCVWNDCONFLICT), SNMP_MIB_SENTINEL }; =20 diff --git a/net/mptcp/mib.h b/net/mptcp/mib.h index 2411510bef66..e05428838005 100644 --- a/net/mptcp/mib.h +++ b/net/mptcp/mib.h @@ -50,6 +50,11 @@ enum linux_mptcp_mib_field { MPTCP_MIB_SUBFLOWSTALE, /* Subflows entered 'stale' status */ MPTCP_MIB_SUBFLOWRECOVER, /* Subflows returned to active status after bei= ng stale */ MPTCP_MIB_SNDWNDSHARED, /* Subflow snd wnd is overridden by msk's one */ + MPTCP_MIB_RCVWNDSHARED, /* Subflow rcv wnd is overridden by msk's one */ + MPTCP_MIB_RCVWNDCONFLICTUPDATE, /* subflow rcv wnd is overridden by msk's= one due to + * conflict with another subflow while updating msk rcv wnd + */ + MPTCP_MIB_RCVWNDCONFLICT, /* Conflict with another subflow while updating= msk rcv wnd */ __MPTCP_MIB_MAX }; =20 diff --git a/net/mptcp/options.c b/net/mptcp/options.c index ffc0292da041..dc84f39ff9f9 100644 --- a/net/mptcp/options.c +++ b/net/mptcp/options.c @@ -1249,8 +1249,11 @@ static void mptcp_set_rwin(struct tcp_sock *tp, void= *opt) =20 if (rcv_wnd =3D=3D rcv_wnd_old) break; - if (before64(rcv_wnd_new, rcv_wnd)) + if (before64(rcv_wnd_new, rcv_wnd)) { + MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_RCVWNDCONFLICTUPDATE); goto raise_win; + } + MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_RCVWNDCONFLICT); rcv_wnd_old =3D rcv_wnd; }; return; @@ -1278,6 +1281,7 @@ static void mptcp_set_rwin(struct tcp_sock *tp, void = *opt) /* RFC1323 scaling applied */ new_win >>=3D tp->rx_opt.rcv_wscale; th->window =3D htons(new_win); + MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_RCVWNDSHARED); } } =20 --=20 2.35.1