From nobody Fri May 3 06:06:38 2024 Received: from sender11-of-o51.zoho.eu (sender11-of-o51.zoho.eu [31.186.226.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE91D5CBB for ; Mon, 10 Oct 2022 22:18:36 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; t=1665440308; cv=none; d=zohomail.eu; s=zohoarc; b=b9ZIwlCyRlWueCJd3GdKnixePW3j0M5fjkkplD1CtU/70SFAcpBng9Su6nDyVWVrC+gdObF7r74cI7BOQTkdupttuq/eRZhY8TgQqxUGfAN5bRpLepauozcSdL8eMyjFRjSwpLAEQxQEHJ04ZS7S8vicYIaXhBEQzbSCrv0W9SM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.eu; s=zohoarc; t=1665440308; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=cvQq9gdDu5XKGiQ44zR2uokoRkYA4OAqfh2u37svOLU=; b=J1+SROhkIdJff5TJP7Kx1kcH2mXhDXXsI35BKxdUij61WupwnmfdFzQSf2Zd5j+SfjyDbLPp/foe84HnsOpGkTSBKz7u23WCOhpaqga1EafHEtKrWJXwY5LTrpQAHsdmhXVaWSnwNqv2f8NrpVdZXfJ1yk/bE+UGfSjN2WCZTZY= ARC-Authentication-Results: i=1; mx.zohomail.eu; dkim=pass header.i=shytyi.net; spf=pass smtp.mailfrom=dmytro@shytyi.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1665440308; s=hs; d=shytyi.net; i=dmytro@shytyi.net; h=From:From:To:To:Cc:Cc:Message-ID:Subject:Subject:Date:Date:In-Reply-To:References:MIME-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Reply-To; bh=cvQq9gdDu5XKGiQ44zR2uokoRkYA4OAqfh2u37svOLU=; b=DXJsfMJ3V6SaYGXvA03iLRW97VwXekOgR+MbT78vDD4mphg3jE3keTCySB+e72V0 +C/AZ0/0HfQ3uTNPNQfMDN436fNU++ZEA0xoK3/5ojHEiXnJM7rQlE4ze87RrsfX6AW PwgEAFpB4h0kv6SjQxxdphRad/JuEWF+o4To/6gY= Received: from localhost.localdomain (vps-940813d4.vps.ovh.net [198.244.188.50]) by mx.zoho.eu with SMTPS id 1665440306478135.64254988044843; Tue, 11 Oct 2022 00:18:26 +0200 (CEST) From: Dmytro Shytyi To: mptcp@lists.linux.dev Cc: Dmytro Shytyi , Benjamin Hesmans Message-ID: <20221010221809.1792-2-dmytro@shytyi.net> Subject: [RFC PATCH mptcp-next v14 1/5] mptcp: introduce MSG_FASTOPEN flag. Date: Mon, 10 Oct 2022 22:18:05 +0000 X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221010221809.1792-1-dmytro@shytyi.net> References: <20221010221809.1792-1-dmytro@shytyi.net> Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Content-Type: text/plain; charset="utf-8" In the following patches we will analyse the MSG_FASTOPEN flag in the mptcp_sendmsg() and invoke the MPTFO. Signed-of-by: Benjamin Hesmans Signed-off-by: Dmytro Shytyi --- net/mptcp/protocol.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 1aa940928b4f..b06f9bbe0e0a 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -1673,17 +1673,14 @@ static int mptcp_sendmsg(struct sock *sk, struct ms= ghdr *msg, size_t len) int ret =3D 0; long timeo; =20 - /* we don't support FASTOPEN yet */ - if (msg->msg_flags & MSG_FASTOPEN) - return -EOPNOTSUPP; - /* silently ignore everything else */ - msg->msg_flags &=3D MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL; + msg->msg_flags &=3D MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_FASTOPEN; =20 lock_sock(sk); =20 ssock =3D __mptcp_nmpc_socket(msk); - if (unlikely(ssock && inet_sk(ssock->sk)->defer_connect)) { + if (unlikely(ssock && (inet_sk(ssock->sk)->defer_connect || + msg->msg_flags & MSG_FASTOPEN))) { struct sock *ssk =3D ssock->sk; int copied_syn =3D 0; =20 --=20 2.34.1 From nobody Fri May 3 06:06:38 2024 Received: from sender11-of-o51.zoho.eu (sender11-of-o51.zoho.eu [31.186.226.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD8A25CB0 for ; Mon, 10 Oct 2022 22:18:36 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; t=1665440308; cv=none; d=zohomail.eu; s=zohoarc; b=kn0f4d8etNJDGbFQ464clUsqmCTnlKm/R1PKQUSvs2HezwrXswbrS8rYVwksk+JLnZbs6Pu+Jdf2sqxFQ4oks5gOpI52cjfrGiJrjrXEpYVPgcSiFtTeyt0FoiYhgCDdK5BD0pFf1bARo+UGooNIV3X3JDW0cyz/xowoTbVkvxk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.eu; s=zohoarc; t=1665440308; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=Xv76KnkqTZVuO5RScDnsFJ5wzxmYYgBVV1S/QyukRwM=; b=XUKGkKTmEg8NnpAKVGrV59uIRTUAnG/h6xhv/39xiQve+wJpkAHL2ZEjdhk8B+rELxrOT58+CJFovqPJi9D3a6KTullae4mBoYei71KbeMGtTO9ax1n3ogJ1SWy8TmpVtiZnrUxjnG6f/ybp64joKN0L+/tMF3sWvPtFjFtaDLo= ARC-Authentication-Results: i=1; mx.zohomail.eu; dkim=pass header.i=shytyi.net; spf=pass smtp.mailfrom=dmytro@shytyi.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1665440308; s=hs; d=shytyi.net; i=dmytro@shytyi.net; h=From:From:To:To:Cc:Cc:Message-ID:Subject:Subject:Date:Date:In-Reply-To:References:MIME-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Reply-To; bh=Xv76KnkqTZVuO5RScDnsFJ5wzxmYYgBVV1S/QyukRwM=; b=DvXfbwSe2BB+v92uG+4QaCMm+6qD2ZGeLmmM5uve1FjUlIpFVfgOwCuzntuvd+FC 6uMX2Amq2+h/fiPpABuH6+QEB0fGjL9iuLC5IpDVIAAsz6slgrALc1+Vp4V/oLvwtx1 ksjzc+pq+qS6Y/wv9rzM+fOTKVFE9uGJkzjv3qT8= Received: from localhost.localdomain (vps-940813d4.vps.ovh.net [198.244.188.50]) by mx.zoho.eu with SMTPS id 1665440306886890.3400640647226; Tue, 11 Oct 2022 00:18:26 +0200 (CEST) From: Dmytro Shytyi To: mptcp@lists.linux.dev Cc: Dmytro Shytyi Message-ID: <20221010221809.1792-3-dmytro@shytyi.net> Subject: [RFC PATCH mptcp-next v14 2/5] mptcp: implement delayed seq generation for passive fastopen Date: Mon, 10 Oct 2022 22:18:06 +0000 X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221010221809.1792-1-dmytro@shytyi.net> References: <20221010221809.1792-1-dmytro@shytyi.net> Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Content-Type: text/plain; charset="utf-8" With fastopen in place the first subflow socket is created before the MPC handshake completes, and we need to properly initialize the sequence numbers at MPC ACK reception. Signed-off-by: Dmytro Shytyi --- net/mptcp/Makefile | 2 +- net/mptcp/fastopen.c | 19 +++++++++++++++++++ net/mptcp/options.c | 5 +++++ net/mptcp/protocol.h | 6 ++++++ 4 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 net/mptcp/fastopen.c diff --git a/net/mptcp/Makefile b/net/mptcp/Makefile index 8a7f68efa35f..c42ad8609876 100644 --- a/net/mptcp/Makefile +++ b/net/mptcp/Makefile @@ -2,7 +2,7 @@ obj-$(CONFIG_MPTCP) +=3D mptcp.o =20 mptcp-y :=3D protocol.o subflow.o options.o token.o crypto.o ctrl.o pm.o d= iag.o \ - mib.o pm_netlink.o sockopt.o pm_userspace.o sched.o + mib.o pm_netlink.o sockopt.o pm_userspace.o sched.o fastopen.o =20 obj-$(CONFIG_SYN_COOKIES) +=3D syncookies.o obj-$(CONFIG_INET_MPTCP_DIAG) +=3D mptcp_diag.o diff --git a/net/mptcp/fastopen.c b/net/mptcp/fastopen.c new file mode 100644 index 000000000000..d6fb45e6be4f --- /dev/null +++ b/net/mptcp/fastopen.c @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 + * MPTCP Fast Open Mechanism. Copyright (c) 2021-2022, Dmytro SHYTYI + */ + +#include "protocol.h" + +void mptcp_gen_msk_ackseq_fastopen(struct mptcp_sock *msk, struct mptcp_su= bflow_context *subflow, + struct mptcp_options_received mp_opt) +{ + u64 ack_seq; + + WRITE_ONCE(msk->can_ack, true); + WRITE_ONCE(msk->remote_key, mp_opt.sndr_key); + mptcp_crypto_key_sha(msk->remote_key, NULL, &ack_seq); + ack_seq++; + WRITE_ONCE(msk->ack_seq, ack_seq); + pr_debug("ack_seq=3D%llu sndr_key=3D%llu", msk->ack_seq, mp_opt.sndr_key); + atomic64_set(&msk->rcv_wnd_sent, ack_seq); +} diff --git a/net/mptcp/options.c b/net/mptcp/options.c index 30d289044e71..0b6c4535750c 100644 --- a/net/mptcp/options.c +++ b/net/mptcp/options.c @@ -1208,6 +1208,11 @@ bool mptcp_incoming_options(struct sock *sk, struct = sk_buff *skb) mpext->dsn64 =3D 1; mpext->mpc_map =3D 1; mpext->data_fin =3D 0; + + if (msk->is_mptfo) { + mptcp_gen_msk_ackseq_fastopen(msk, subflow, mp_opt); + mpext->data_seq =3D READ_ONCE(msk->ack_seq); + } } else { mpext->data_seq =3D mp_opt.data_seq; mpext->subflow_seq =3D mp_opt.subflow_seq; diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 18f866b1afda..41276788626b 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -282,6 +282,7 @@ struct mptcp_sock { bool use_64bit_ack; /* Set when we received a 64-bit DSN */ bool csum_enabled; bool allow_infinite_fallback; + bool is_mptfo; u8 mpc_endpoint_id; u8 recvmsg_inq:1, cork:1, @@ -840,6 +841,11 @@ void mptcp_event_addr_announced(const struct sock *ssk= , const struct mptcp_addr_ void mptcp_event_addr_removed(const struct mptcp_sock *msk, u8 id); bool mptcp_userspace_pm_active(const struct mptcp_sock *msk); =20 +// Fast Open Mechanism functions begin +void mptcp_gen_msk_ackseq_fastopen(struct mptcp_sock *msk, struct mptcp_su= bflow_context *subflow, + struct mptcp_options_received mp_opt); +// Fast Open Mechanism functions end + static inline bool mptcp_pm_should_add_signal(struct mptcp_sock *msk) { return READ_ONCE(msk->pm.addr_signal) & --=20 2.34.1 From nobody Fri May 3 06:06:38 2024 Received: from sender11-of-o51.zoho.eu (sender11-of-o51.zoho.eu [31.186.226.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E0DB5CBD for ; Mon, 10 Oct 2022 22:18:38 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; t=1665440308; cv=none; d=zohomail.eu; s=zohoarc; b=Z72lMRtB88ZId2jYOb4JWMsie5Ne7XY5trl2Dkj29NuAGOlsOJ85BioEo7cl5bFuN1sqO94ZShgFaj7VDcrvu7wBiz4ECTG9WrFrz5fQUvk13pLxvJPdBkWFoOGr4J82n/Z7bzoKQ7btFzveVKU5+8/moo2kRPEq/vyeZuV/2fE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.eu; s=zohoarc; t=1665440308; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=0STQSYL3rOD1U1464f8CPMLuxkL/9A1MVWkaQQQuUQE=; b=hEQ98g0rS7SmmTT3WJFmKLEpgPGYn32yQvULTfPfyQuXeH8jfuo6RVh2lo8xhtlXaxkLoEQbcMiVcDTgt6cGNqf2w82ZYQZqhxEsCFwsz0rXInwnhnuutryr1k64qL5S2yIpMjYHqO2/GQj+VOYg79mX2Yg9RfvROrmBZj3NLC8= ARC-Authentication-Results: i=1; mx.zohomail.eu; dkim=pass header.i=shytyi.net; spf=pass smtp.mailfrom=dmytro@shytyi.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1665440308; s=hs; d=shytyi.net; i=dmytro@shytyi.net; h=From:From:To:To:Cc:Cc:Message-ID:Subject:Subject:Date:Date:In-Reply-To:References:MIME-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Reply-To; bh=0STQSYL3rOD1U1464f8CPMLuxkL/9A1MVWkaQQQuUQE=; b=c5+HKBsrqtmh9Vojy4h5fbMDH4wIj7RrR0n7xdtTJHXEfiZreGuYajsRGNQ/oW34 mgb5WwkPsCE97X9pbZvCEv1/W2Z7fekmjdZT+ytCZSst/5J44OgvtzjBeW1p0vQ+gU1 oDB8uFEmPQdmWncunHwvpJePfN8r9QDAyc/lpyIo= Received: from localhost.localdomain (vps-940813d4.vps.ovh.net [198.244.188.50]) by mx.zoho.eu with SMTPS id 1665440307208502.7425994858363; Tue, 11 Oct 2022 00:18:27 +0200 (CEST) From: Dmytro Shytyi To: mptcp@lists.linux.dev Cc: Dmytro Shytyi Message-ID: <20221010221809.1792-4-dmytro@shytyi.net> Subject: [RFC PATCH mptcp-next v14 3/5] mptcp: add subflow_v(4,6)_send_synack() Date: Mon, 10 Oct 2022 22:18:07 +0000 X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221010221809.1792-1-dmytro@shytyi.net> References: <20221010221809.1792-1-dmytro@shytyi.net> Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Content-Type: text/plain; charset="utf-8" In this patch we add skb to the msk, dequeue it from sk, remove TSs and do skb mapping. Signed-off-by: Dmytro Shytyi --- net/mptcp/fastopen.c | 78 ++++++++++++++++++++++++++++++++++++++++++++ net/mptcp/protocol.c | 2 +- net/mptcp/protocol.h | 3 ++ net/mptcp/subflow.c | 42 ++++++++++++++++++++++++ 4 files changed, 124 insertions(+), 1 deletion(-) diff --git a/net/mptcp/fastopen.c b/net/mptcp/fastopen.c index d6fb45e6be4f..f9ed409fdfb7 100644 --- a/net/mptcp/fastopen.c +++ b/net/mptcp/fastopen.c @@ -4,6 +4,84 @@ =20 #include "protocol.h" =20 +struct mptcp_skb_cb { + u64 map_seq; + u64 end_seq; + u32 offset; + u8 has_rxtstamp:1; +}; + +#define MPTCP_SKB_CB(__skb) ((struct mptcp_skb_cb *)&((__skb)->cb[0])) + +void subflow_fastopen_send_synack_set_params(struct mptcp_subflow_context = *subflow, + struct request_sock *req) +{ + struct tcp_request_sock *treq =3D tcp_rsk(req); + struct sock *ssk =3D subflow->tcp_sock; + struct sock *sk =3D subflow->conn; + struct mptcp_sock *msk; + struct sk_buff *skb; + struct tcp_sock *tp; + u32 offset; + + msk =3D mptcp_sk(sk); + tp =3D tcp_sk(ssk); + + /* mark subflow/msk as "mptfo" */ + msk->is_mptfo =3D 1; + + skb =3D skb_peek(&ssk->sk_receive_queue); + + /* dequeue the skb from sk receive queue */ + __skb_unlink(skb, &ssk->sk_receive_queue); + skb_ext_reset(skb); + skb_orphan(skb); + + /* set the skb mapping */ + tp->copied_seq +=3D tp->rcv_nxt - treq->rcv_isn - 1; + subflow->map_seq =3D mptcp_subflow_get_mapped_dsn(subflow); + subflow->ssn_offset =3D tp->copied_seq - 1; + + /* innitialize MPTCP_CB */ + offset =3D tp->copied_seq - TCP_SKB_CB(skb)->seq; + MPTCP_SKB_CB(skb)->map_seq =3D mptcp_subflow_get_mapped_dsn(subflow); + MPTCP_SKB_CB(skb)->end_seq =3D MPTCP_SKB_CB(skb)->map_seq + + (skb->len - offset); + MPTCP_SKB_CB(skb)->offset =3D offset; + MPTCP_SKB_CB(skb)->has_rxtstamp =3D TCP_SKB_CB(skb)->has_rxtstamp; + + mptcp_data_lock(sk); + + mptcp_set_owner_r(skb, sk); + __skb_queue_tail(&msk->receive_queue, skb); + + (sk)->sk_data_ready(sk); + + mptcp_data_unlock(sk); +} + +void __mptcp_pre_connect(struct mptcp_sock *msk, struct sock *ssk, + struct msghdr *msg, size_t size) +{ + struct tcp_sock *tp; + struct sk_buff *skb; + struct ubuf_info *uarg; + + lock_sock(ssk); + + tp =3D tcp_sk(ssk); + + skb =3D tcp_write_queue_tail(ssk); + uarg =3D msg_zerocopy_realloc(ssk, size, skb_zcopy(skb)); + tp->fastopen_req =3D kzalloc(sizeof(*tp->fastopen_req), + ssk->sk_allocation); + tp->fastopen_req->data =3D msg; + tp->fastopen_req->size =3D size; + tp->fastopen_req->uarg =3D uarg; + + release_sock(ssk); +} + void mptcp_gen_msk_ackseq_fastopen(struct mptcp_sock *msk, struct mptcp_su= bflow_context *subflow, struct mptcp_options_received mp_opt) { diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index b06f9bbe0e0a..a7571cae2807 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -200,7 +200,7 @@ static void mptcp_rfree(struct sk_buff *skb) mptcp_rmem_uncharge(sk, len); } =20 -static void mptcp_set_owner_r(struct sk_buff *skb, struct sock *sk) +void mptcp_set_owner_r(struct sk_buff *skb, struct sock *sk) { skb_orphan(skb); skb->sk =3D sk; diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 41276788626b..257a3a62a71b 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -844,6 +844,9 @@ bool mptcp_userspace_pm_active(const struct mptcp_sock = *msk); // Fast Open Mechanism functions begin void mptcp_gen_msk_ackseq_fastopen(struct mptcp_sock *msk, struct mptcp_su= bflow_context *subflow, struct mptcp_options_received mp_opt); +void mptcp_set_owner_r(struct sk_buff *skb, struct sock *sk); +void subflow_fastopen_send_synack_set_params(struct mptcp_subflow_context = *subflow, + struct request_sock *req); // Fast Open Mechanism functions end =20 static inline bool mptcp_pm_should_add_signal(struct mptcp_sock *msk) diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c index 07dd23d0fe04..0fbc45eb7bec 100644 --- a/net/mptcp/subflow.c +++ b/net/mptcp/subflow.c @@ -307,7 +307,47 @@ static struct dst_entry *subflow_v4_route_req(const st= ruct sock *sk, return NULL; } =20 +static int subflow_v4_send_synack(const struct sock *sk, struct dst_entry = *dst, + struct flowi *fl, + struct request_sock *req, + struct tcp_fastopen_cookie *foc, + enum tcp_synack_type synack_type, + struct sk_buff *syn_skb) +{ + struct mptcp_subflow_context *subflow =3D mptcp_subflow_ctx(sk); + struct inet_request_sock *ireq =3D inet_rsk(req); + + /* clear tstamp_ok, as needed depending on cookie */ + if (foc && foc->len > -1) + ireq->tstamp_ok =3D 0; + + if (synack_type =3D=3D TCP_SYNACK_FASTOPEN) + subflow_fastopen_send_synack_set_params(subflow, req); + + return tcp_request_sock_ipv4_ops.send_synack(sk, dst, fl, req, foc, synac= k_type, syn_skb); +} + #if IS_ENABLED(CONFIG_MPTCP_IPV6) +static int subflow_v6_send_synack(const struct sock *sk, struct dst_entry = *dst, + struct flowi *fl, + struct request_sock *req, + struct tcp_fastopen_cookie *foc, + enum tcp_synack_type synack_type, + struct sk_buff *syn_skb) +{ + struct mptcp_subflow_context *subflow =3D mptcp_subflow_ctx(sk); + struct inet_request_sock *ireq =3D inet_rsk(req); + + /* clear tstamp_ok, as needed depending on cookie */ + if (foc && foc->len > -1) + ireq->tstamp_ok =3D 0; + + if (synack_type =3D=3D TCP_SYNACK_FASTOPEN) + subflow_fastopen_send_synack_set_params(subflow, req); + + return tcp_request_sock_ipv6_ops.send_synack(sk, dst, fl, req, foc, synac= k_type, syn_skb); +} + static struct dst_entry *subflow_v6_route_req(const struct sock *sk, struct sk_buff *skb, struct flowi *fl, @@ -1920,6 +1960,7 @@ void __init mptcp_subflow_init(void) =20 subflow_request_sock_ipv4_ops =3D tcp_request_sock_ipv4_ops; subflow_request_sock_ipv4_ops.route_req =3D subflow_v4_route_req; + subflow_request_sock_ipv4_ops.send_synack =3D subflow_v4_send_synack; =20 subflow_specific =3D ipv4_specific; subflow_specific.conn_request =3D subflow_v4_conn_request; @@ -1933,6 +1974,7 @@ void __init mptcp_subflow_init(void) #if IS_ENABLED(CONFIG_MPTCP_IPV6) subflow_request_sock_ipv6_ops =3D tcp_request_sock_ipv6_ops; subflow_request_sock_ipv6_ops.route_req =3D subflow_v6_route_req; + subflow_request_sock_ipv6_ops.send_synack =3D subflow_v6_send_synack; =20 subflow_v6_specific =3D ipv6_specific; subflow_v6_specific.conn_request =3D subflow_v6_conn_request; --=20 2.34.1 From nobody Fri May 3 06:06:38 2024 Received: from sender11-of-o51.zoho.eu (sender11-of-o51.zoho.eu [31.186.226.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C6105CBC for ; Mon, 10 Oct 2022 22:18:38 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; t=1665440308; cv=none; d=zohomail.eu; s=zohoarc; b=Vc6erHVIv5oQZJrth1fWiV74kkyPC5jc40i32EfclHI0L7w7EUTr5j8WVjL2Lb3g9tMw8BghtZ4+GrzwWVorP9RiUVd8o+eb9F6rQ5R+uTu8rcOW9rbbW3sJOyxwpusXXEZO/Vt0vCQP/Rb//UO5b7207g15tD7pZ1OSdzTgUVw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.eu; s=zohoarc; t=1665440308; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=pNu48RZme7ExKQiNWIIrnxNBbaEyThENQeRocKx8Nq8=; b=j0lCBro+XoCoDGQ8+/MBm2CirovFgQlh3IHA22sAW/xEvZ4trHGiEl+TJWhQDHddgFe1SjtZNrNMeFDTMpcsnpqIXpnbk33o4B4ClMgyQ7b0SKQrgVBsFMbau4wqvyyRaCKCafTGBMFt3FUPlZri0gg0fly4/ktGRCwD8yd5NZI= ARC-Authentication-Results: i=1; mx.zohomail.eu; dkim=pass header.i=shytyi.net; spf=pass smtp.mailfrom=dmytro@shytyi.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1665440308; s=hs; d=shytyi.net; i=dmytro@shytyi.net; h=From:From:To:To:Cc:Cc:Message-ID:Subject:Subject:Date:Date:In-Reply-To:References:MIME-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Reply-To; bh=pNu48RZme7ExKQiNWIIrnxNBbaEyThENQeRocKx8Nq8=; b=WuhQC5iEpMrBbuarmfdOebAntouuqVIPJy7v/wDEa9+eKGZ0rQpfmAnrhhFzAOWi ks9uN79MHYifrp5k09gGPqgKXCX+WbXE4HSGM88LIx3bhBdOqGD/q8L3Uz8xt0TUEif +NZj3AIZqVo50cStHhNw06s8toh7OGFsBSiyAEfQ= Received: from localhost.localdomain (vps-940813d4.vps.ovh.net [198.244.188.50]) by mx.zoho.eu with SMTPS id 166544030751339.32648651547288; Tue, 11 Oct 2022 00:18:27 +0200 (CEST) From: Dmytro Shytyi To: mptcp@lists.linux.dev Cc: Dmytro Shytyi Message-ID: <20221010221809.1792-5-dmytro@shytyi.net> Subject: [RFC PATCH mptcp-next v14 4/5] mptcp: add TCP_FASTOPEN sock option Date: Mon, 10 Oct 2022 22:18:08 +0000 X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221010221809.1792-1-dmytro@shytyi.net> References: <20221010221809.1792-1-dmytro@shytyi.net> Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Content-Type: text/plain; charset="utf-8" We add the TCP_FASTOPEN socket option in this patch that is going to be set by the listener side. Signed-off-by: Dmytro Shytyi --- net/mptcp/sockopt.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/net/mptcp/sockopt.c b/net/mptcp/sockopt.c index f85e9bbfe86f..7f6977295256 100644 --- a/net/mptcp/sockopt.c +++ b/net/mptcp/sockopt.c @@ -559,6 +559,7 @@ static bool mptcp_supported_sockopt(int level, int optn= ame) case TCP_NOTSENT_LOWAT: case TCP_TX_DELAY: case TCP_INQ: + case TCP_FASTOPEN: case TCP_FASTOPEN_CONNECT: case TCP_FASTOPEN_NO_COOKIE: return true; @@ -569,7 +570,7 @@ static bool mptcp_supported_sockopt(int level, int optn= ame) /* TCP_REPAIR, TCP_REPAIR_QUEUE, TCP_QUEUE_SEQ, TCP_REPAIR_OPTIONS, * TCP_REPAIR_WINDOW are not supported, better avoid this mess */ - /* TCP_FASTOPEN_KEY, TCP_FASTOPEN are not supported because + /* TCP_FASTOPEN_KEY is not supported because * fastopen for the listener side is currently unsupported */ } @@ -801,6 +802,7 @@ static int mptcp_setsockopt_sol_tcp(struct mptcp_sock *= msk, int optname, /* See tcp.c: TCP_DEFER_ACCEPT does not fail */ mptcp_setsockopt_first_sf_only(msk, SOL_TCP, optname, optval, optlen); return 0; + case TCP_FASTOPEN: case TCP_FASTOPEN_CONNECT: case TCP_FASTOPEN_NO_COOKIE: return mptcp_setsockopt_first_sf_only(msk, SOL_TCP, optname, @@ -1166,6 +1168,7 @@ static int mptcp_getsockopt_sol_tcp(struct mptcp_sock= *msk, int optname, case TCP_INFO: case TCP_CC_INFO: case TCP_DEFER_ACCEPT: + case TCP_FASTOPEN: case TCP_FASTOPEN_CONNECT: case TCP_FASTOPEN_NO_COOKIE: return mptcp_getsockopt_first_sf_only(msk, SOL_TCP, optname, --=20 2.34.1 From nobody Fri May 3 06:06:38 2024 Received: from sender11-of-o51.zoho.eu (sender11-of-o51.zoho.eu [31.186.226.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0397E5CBE for ; Mon, 10 Oct 2022 22:33:33 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; t=1665441210; cv=none; d=zohomail.eu; s=zohoarc; b=Fcs3FMyJm4S7SYi+NM5gfLlda97ufEXFLtg8mnYGCg4jnUD+SEqVYDE08AG9kmZRwZUyzs7QCvXxxcq9LAw4zyeYDaOC+lJuN04bO3lT+mm/VdV5gkYf5HUrqfN1TxqrLxiSdgt06mWfW+N2dbgyxBbK9b0h5D2YwjwIePuV3W0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.eu; s=zohoarc; t=1665441210; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=7m8spKlSqCiXNxuznskJITxqVk7gqo7bOtAWvoeCBDU=; b=dOrlEUnY0iapFOxvb/gRHYU0gHjw2mqs2sfW3bbQm9Ms0SZJJa5tdbA1GCpPAHE/tXPakBI9aVouQ6MpuZe2idIX4mKxf7eFoJfT1d31oUvydD/MYDH6IDAOWvpNzM/eDLaIrmc8TfqBBEjcZEEr1PrgO0t27XiTUfGVak04Kqg= ARC-Authentication-Results: i=1; mx.zohomail.eu; dkim=pass header.i=shytyi.net; spf=pass smtp.mailfrom=dmytro@shytyi.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1665441210; s=hs; d=shytyi.net; i=dmytro@shytyi.net; h=From:From:To:To:Cc:Cc:Message-ID:Subject:Subject:Date:Date:In-Reply-To:References:MIME-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Reply-To; bh=7m8spKlSqCiXNxuznskJITxqVk7gqo7bOtAWvoeCBDU=; b=RcMfdyM6/wZjk1U5xpqW/91hTEKdXfe8dolHjvz0OPP/gqaMqjsCKWkBicJYePkI QnSGJW8a2LpLUAIOqILiaYJADMu1IIHpXJcH1aZQIVncTQCj4LsXyqMPe5WfKhZTBn1 cfrSpJYew0FaBJfSuwe8vN0rxa3By6TqIbDFEZ+M= Received: from localhost.localdomain (vps-940813d4.vps.ovh.net [198.244.188.50]) by mx.zoho.eu with SMTPS id 1665440307828843.5495417235318; Tue, 11 Oct 2022 00:18:27 +0200 (CEST) From: Dmytro Shytyi To: mptcp@lists.linux.dev Cc: Dmytro Shytyi Message-ID: <20221010221809.1792-6-dmytro@shytyi.net> Subject: [RFC PATCH mptcp-next v14 5/5] selftests: mptcp: mptfo Initiator/Listener Date: Mon, 10 Oct 2022 22:18:09 +0000 X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221010221809.1792-1-dmytro@shytyi.net> References: <20221010221809.1792-1-dmytro@shytyi.net> Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Content-Type: text/plain; charset="utf-8" This patch adds the selftests support for mptfo in mptcp_connect.c,.sh We introduce mptfo option, that sets "TCP_FASTOPEN" + "MSG_FASTOPEN" We call sendto() instead of connect(). Signed-off-by: Dmytro Shytyi --- .../selftests/net/mptcp/mptcp_connect.c | 140 +++++++++++++----- .../selftests/net/mptcp/mptcp_connect.sh | 77 ++++++++++ 2 files changed, 184 insertions(+), 33 deletions(-) diff --git a/tools/testing/selftests/net/mptcp/mptcp_connect.c b/tools/test= ing/selftests/net/mptcp/mptcp_connect.c index e54653ea2ed4..fdb934eb97ec 100644 --- a/tools/testing/selftests/net/mptcp/mptcp_connect.c +++ b/tools/testing/selftests/net/mptcp/mptcp_connect.c @@ -83,6 +83,7 @@ struct cfg_cmsg_types { =20 struct cfg_sockopt_types { unsigned int transparent:1; + unsigned int mptfo:1; }; =20 struct tcp_inq_state { @@ -90,6 +91,7 @@ struct tcp_inq_state { bool expect_eof; }; =20 +static size_t do_write(const int fd, char *buf, const size_t len); static struct tcp_inq_state tcp_inq; =20 static struct cfg_cmsg_types cfg_cmsg_types; @@ -232,6 +234,22 @@ static void set_transparent(int fd, int pf) } } =20 +static void set_mptfo(int fd, int pf) +{ + int qlen =3D 25; + + switch (pf) { + case AF_INET: + if (-1 =3D=3D setsockopt(fd, 6, TCP_FASTOPEN, &qlen, sizeof(qlen))) + perror("TCP_FASTOPEN"); + break; + case AF_INET6: + if (-1 =3D=3D setsockopt(fd, 6, TCP_FASTOPEN, &qlen, sizeof(qlen))) + perror("TCP_FASTOPEN"); + break; + } +} + static int do_ulp_so(int sock, const char *name) { return setsockopt(sock, IPPROTO_TCP, TCP_ULP, name, strlen(name)); @@ -300,6 +318,9 @@ static int sock_listen_mptcp(const char * const listena= ddr, if (cfg_sockopt_types.transparent) set_transparent(sock, pf); =20 + if (cfg_sockopt_types.mptfo) + set_mptfo(sock, pf); + if (bind(sock, a->ai_addr, a->ai_addrlen) =3D=3D 0) break; /* success */ =20 @@ -330,13 +351,18 @@ static int sock_listen_mptcp(const char * const liste= naddr, =20 static int sock_connect_mptcp(const char * const remoteaddr, const char * const port, int proto, - struct addrinfo **peer) + struct addrinfo **peer, + int infd, + unsigned int *woff) { struct addrinfo hints =3D { .ai_protocol =3D IPPROTO_TCP, .ai_socktype =3D SOCK_STREAM, }; struct addrinfo *a, *addr; + unsigned int wlen =3D 0; + int syn_copied =3D 0; + char wbuf[8192]; int sock =3D -1; =20 hints.ai_family =3D pf; @@ -354,14 +380,33 @@ static int sock_connect_mptcp(const char * const remo= teaddr, if (cfg_mark) set_mark(sock, cfg_mark); =20 - if (connect(sock, a->ai_addr, a->ai_addrlen) =3D=3D 0) { - *peer =3D a; - break; /* success */ - } + if (cfg_sockopt_types.mptfo) { + if (wlen =3D=3D 0) + wlen =3D read(infd, wbuf, sizeof(wbuf)); =20 - perror("connect()"); - close(sock); - sock =3D -1; + syn_copied =3D sendto(sock, wbuf, wlen, MSG_FASTOPEN, + a->ai_addr, a->ai_addrlen); + if (syn_copied) { + *woff =3D syn_copied; + *peer =3D a; + break; /* success */ + } + } else { + if (connect(sock, a->ai_addr, a->ai_addrlen) =3D=3D 0) { + *peer =3D a; + break; /* success */ + } + } + if (cfg_sockopt_types.mptfo) { + perror("sendto()"); + close(sock); + sock =3D -1; + } else { + + perror("connect()"); + close(sock); + sock =3D -1; + } } =20 freeaddrinfo(addr); @@ -571,13 +616,14 @@ static void shut_wr(int fd) shutdown(fd, SHUT_WR); } =20 -static int copyfd_io_poll(int infd, int peerfd, int outfd, bool *in_closed= _after_out) +static int copyfd_io_poll(int infd, int peerfd, int outfd, + bool *in_closed_after_out, unsigned int woff) { struct pollfd fds =3D { .fd =3D peerfd, .events =3D POLLIN | POLLOUT, }; - unsigned int woff =3D 0, wlen =3D 0, total_wlen =3D 0, total_rlen =3D 0; + unsigned int wlen =3D 0, total_wlen =3D 0, total_rlen =3D 0; char wbuf[8192]; =20 set_nonblock(peerfd, true); @@ -717,9 +763,9 @@ static int do_recvfile(int infd, int outfd) return (int)r; } =20 -static int do_mmap(int infd, int outfd, unsigned int size) +static int do_mmap(int infd, int outfd, unsigned int size, unsigned int sy= n_off) { - char *inbuf =3D mmap(NULL, size, PROT_READ, MAP_SHARED, infd, 0); + char *inbuf =3D mmap(NULL, size, PROT_READ, MAP_SHARED, infd, syn_off); ssize_t ret =3D 0, off =3D 0; size_t rem; =20 @@ -790,7 +836,7 @@ static int do_sendfile(int infd, int outfd, unsigned in= t count) } =20 static int copyfd_io_mmap(int infd, int peerfd, int outfd, - unsigned int size, bool *in_closed_after_out) + unsigned int size, bool *in_closed_after_out, unsigned int woff) { int err; =20 @@ -799,9 +845,9 @@ static int copyfd_io_mmap(int infd, int peerfd, int out= fd, if (err) return err; =20 - err =3D do_mmap(infd, peerfd, size); + err =3D do_mmap(infd, peerfd, size, 0); } else { - err =3D do_mmap(infd, peerfd, size); + err =3D do_mmap(infd, peerfd, size - woff, woff); if (err) return err; =20 @@ -839,7 +885,7 @@ static int copyfd_io_sendfile(int infd, int peerfd, int= outfd, return err; } =20 -static int copyfd_io(int infd, int peerfd, int outfd, bool close_peerfd) +static int copyfd_io(int infd, int peerfd, int outfd, bool close_peerfd, u= nsigned int woff) { bool in_closed_after_out =3D false; struct timespec start, end; @@ -851,14 +897,14 @@ static int copyfd_io(int infd, int peerfd, int outfd,= bool close_peerfd) =20 switch (cfg_mode) { case CFG_MODE_POLL: - ret =3D copyfd_io_poll(infd, peerfd, outfd, &in_closed_after_out); + ret =3D copyfd_io_poll(infd, peerfd, outfd, &in_closed_after_out, woff); break; =20 case CFG_MODE_MMAP: file_size =3D get_infd_size(infd); if (file_size < 0) return file_size; - ret =3D copyfd_io_mmap(infd, peerfd, outfd, file_size, &in_closed_after_= out); + ret =3D copyfd_io_mmap(infd, peerfd, outfd, file_size, &in_closed_after_= out, woff); break; =20 case CFG_MODE_SENDFILE: @@ -1033,7 +1079,7 @@ int main_loop_s(int listensock) =20 SOCK_TEST_TCPULP(remotesock, 0); =20 - copyfd_io(fd, remotesock, 1, true); + copyfd_io(fd, remotesock, 1, true, 0); } else { perror("accept"); return 1; @@ -1130,6 +1176,11 @@ static void parse_setsock_options(const char *name) return; } =20 + if (strncmp(name, "MPTFO", len) =3D=3D 0) { + cfg_sockopt_types.mptfo =3D 1; + return; + } + fprintf(stderr, "Unrecognized setsockopt option %s\n", name); exit(1); } @@ -1168,24 +1219,28 @@ int main_loop(void) { int fd, ret, fd_in =3D 0; struct addrinfo *peer; + unsigned int woff =3D 0; =20 - /* listener is ready. */ - fd =3D sock_connect_mptcp(cfg_host, cfg_port, cfg_sock_proto, &peer); - if (fd < 0) - return 2; + if (!cfg_sockopt_types.mptfo) { + /* listener is ready. */ + fd =3D sock_connect_mptcp(cfg_host, cfg_port, cfg_sock_proto, &peer, NUL= L, NULL); + if (fd < 0) + return 2; + } =20 again: - check_getpeername_connect(fd); + if (!cfg_sockopt_types.mptfo) { + check_getpeername_connect(fd); =20 - SOCK_TEST_TCPULP(fd, cfg_sock_proto); - - if (cfg_rcvbuf) - set_rcvbuf(fd, cfg_rcvbuf); - if (cfg_sndbuf) - set_sndbuf(fd, cfg_sndbuf); - if (cfg_cmsg_types.cmsg_enabled) - apply_cmsg_types(fd, &cfg_cmsg_types); + SOCK_TEST_TCPULP(fd, cfg_sock_proto); =20 + if (cfg_rcvbuf) + set_rcvbuf(fd, cfg_rcvbuf); + if (cfg_sndbuf) + set_sndbuf(fd, cfg_sndbuf); + if (cfg_cmsg_types.cmsg_enabled) + apply_cmsg_types(fd, &cfg_cmsg_types); + } if (cfg_input) { fd_in =3D open(cfg_input, O_RDONLY); if (fd < 0) @@ -1193,7 +1248,26 @@ int main_loop(void) } =20 /* close the client socket open only if we are not going to reconnect */ - ret =3D copyfd_io(fd_in, fd, 1, 0); + + if (cfg_sockopt_types.mptfo) { + /* sendto() instead of connect */ + fd =3D sock_connect_mptcp(cfg_host, cfg_port, cfg_sock_proto, &peer, fd_= in, &woff); + if (fd < 0) + return 2; + } + if (cfg_sockopt_types.mptfo) { + check_getpeername_connect(fd); + + SOCK_TEST_TCPULP(fd, cfg_sock_proto); + + if (cfg_rcvbuf) + set_rcvbuf(fd, cfg_rcvbuf); + if (cfg_sndbuf) + set_sndbuf(fd, cfg_sndbuf); + if (cfg_cmsg_types.cmsg_enabled) + apply_cmsg_types(fd, &cfg_cmsg_types); + } + ret =3D copyfd_io(fd_in, fd, 1, 0, woff); if (ret) return ret; =20 diff --git a/tools/testing/selftests/net/mptcp/mptcp_connect.sh b/tools/tes= ting/selftests/net/mptcp/mptcp_connect.sh index 621af6895f4d..d2328f94d1df 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_connect.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_connect.sh @@ -677,6 +677,58 @@ run_tests() run_tests_lo $1 $2 $3 0 } =20 +run_test_mptfo(){ + + local listener_ns=3D"$1" + local connector_ns=3D"$2" + local connect_addr=3D"$3" + local loopback=3D"$4" + local extra_args=3D"$5" + local msg=3D"$6" + local lret=3D0 + + ip netns exec "$listener_ns" sysctl -q net.ipv4.tcp_fastopen=3D2 + ip netns exec "$connector_ns" sysctl -q net.ipv4.tcp_fastopen=3D1 + + # skip if test programs are running inside same netns for subsequent runs. + if [ $loopback -eq 0 ] && [ ${listener_ns} =3D ${connector_ns} ]; then + return 0 + fi + + # skip if we don't want v6 + if ! $ipv6 && is_v6 "${connect_addr}"; then + return 0 + fi + + local local_addr + if is_v6 "${connect_addr}"; then + local_addr=3D"::" + else + local_addr=3D"0.0.0.0" + fi + + + + TEST_COUNT=3D10000 + do_transfer ${listener_ns} ${connector_ns} MPTCP MPTCP \ + ${connect_addr} ${local_addr} "${extra_args}" + lret=3D$? + + if [ $lret -ne 0 ]; then + echo "FAIL: $msg, mptcp connection error" 1>&2 + ret=3D$lret + return 1 + fi + + echo "PASS: $msg" + + + ip netns exec "$listener_ns" sysctl -q net.ipv4.tcp_fastopen=3D0 + ip netns exec "$connector_ns" sysctl -q net.ipv4.tcp_fastopen=3D0 + return 0 + +} + run_test_transparent() { local connect_addr=3D"$1" @@ -906,6 +958,31 @@ run_test_transparent 10.0.3.1 "tproxy ipv4" run_test_transparent dead:beef:3::1 "tproxy ipv6" stop_if_error "Tests with tproxy have failed" =20 +# MPTFO (MultiPath TCP Fatopen tests) + +for sender in $ns1 $ns2 $ns3 $ns4;do + run_test_mptfo "$ns1" $sender 10.0.1.1 0 "-o MPTFO" "mptfoIPv4" + run_test_mptfo "$ns1" $sender dead:beef:1::1 0 "-o MPTFO" "mptfoIPv6" + + run_test_mptfo "$ns2" $sender 10.0.1.2 0 "-o MPTFO" "mptfoIPv4" + run_test_mptfo "$ns2" $sender dead:beef:1::2 0 "-o MPTFO" "mptfoIPv6" + run_test_mptfo "$ns2" $sender 10.0.2.1 0 "-o MPTFO" "mptfoIPv4" + run_test_mptfo "$ns2" $sender dead:beef:2::1 0 "-o MPTFO" "mptfoIPv6" + + run_test_mptfo "$ns3" $sender 10.0.2.2 0 "-o MPTFO" "mptfoIPv4" + run_test_mptfo "$ns3" $sender dead:beef:2::2 0 "-o MPTFO" "mptfoIPv6" + run_test_mptfo "$ns3" $sender 10.0.3.2 0 "-o MPTFO" "mptfoIPv4" + run_test_mptfo "$ns3" $sender dead:beef:3::2 0 "-o MPTFO" "mptfoIPv6" + + run_test_mptfo "$ns4" $sender 10.0.3.1 0 "-o MPTFO" "mptfoIPv4" + run_test_mptfo "$ns4" $sender dead:beef:3::1 0 "-o MPTFO" "mptfoIPv6" + + stop_if_error "Tests with $sender as a sender have failed" +done +#run_test_mptfo 10.0.3.1 "mptfoIPv4" +#run_test_mptfo dead:beef:3::1 "mptfoIPv6" +#stop_if_error "Tests with MPTFO have failed" + run_tests_disconnect =20 display_time --=20 2.34.1