From nobody Thu Sep 19 01:01:47 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0AE9310953; Mon, 5 Aug 2024 09:53:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722851614; cv=none; b=siH9HmTlUik4K+uQWCygXnoEE53EmvM+EPCC5tPODTwr5lsi4xWIP7nSKp3Qm8Nuqq1zZkdaE0lzNjnSIrMs0/K4CC8gFDH6IjF9z57z5FsZ1D8iBSuvgfHFV1xRlJ9uu7qG2kMqXu6GmnecQJBnzwrK31jmJkoVbIs3Q+wxK6E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722851614; c=relaxed/simple; bh=/K688bWxCbz/40YsdYV732cI2lnwaSfpa2N/5rT1qss=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Thz3jKah83cFBqwT8NBqyoX8OoS8qS+f16jqNGZkb1vxLlJ2JBVRMgJXlfzzF39yWilGU50bQ/TjjIwBuFv69pRPE6LlR04b50Bxe++I9JjBfOjsP4FRo+cNnMoqFZs/rnlidUTZvn+L7xJPE0MqQSfJpZQo7cI4fyDxmk6iHKk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eRBDtTKL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eRBDtTKL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0FBE5C4AF0F; Mon, 5 Aug 2024 09:53:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722851613; bh=/K688bWxCbz/40YsdYV732cI2lnwaSfpa2N/5rT1qss=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=eRBDtTKLG82p49x4r3rED3SMVZtc7UiqMQe5Z44HhWdodZz0eA+OEK9nEzkpgpf0z YQnH7vs0vj/ch1ozFg9SRm0E2RfrZvdRXXlDY/y96tgEVQeC7feK9VOmdlPTGEU8tr AKijqVjhyCanf7mQAj3XJnG8opE6V/XJyi5fJqBcRbkw/C6KxNJMUXcFYPPPA8aV8N hEVuELZvMUBLw/U69mhWp3ZHwaBjL4eDdjGhLuXtK/W/rkFJoB5CtRsVxLSKnmuXEL RUPZor+rBOwnK2x259C6Hj/k76PsHWO04g1n/uQcDNmGNNM+ckiOY57axKIqn/+Us8 Ful83wKq0X+vA== From: "Matthieu Baerts (NGI0)" Date: Mon, 05 Aug 2024 11:52:56 +0200 Subject: [PATCH bpf-next v4 1/2] selftests/bpf: Add mptcp subflow example Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240805-upstream-bpf-next-20240506-mptcp-subflow-test-v4-1-2b4ca6994993@kernel.org> References: <20240805-upstream-bpf-next-20240506-mptcp-subflow-test-v4-0-2b4ca6994993@kernel.org> In-Reply-To: <20240805-upstream-bpf-next-20240506-mptcp-subflow-test-v4-0-2b4ca6994993@kernel.org> To: mptcp@lists.linux.dev, Mat Martineau , Geliang Tang , Andrii Nakryiko , Eduard Zingerman , Mykola Lysenko , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Shuah Khan Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, "Matthieu Baerts (NGI0)" , Nicolas Rybowski , Geliang Tang X-Mailer: b4 0.14.1 X-Developer-Signature: v=1; a=openpgp-sha256; l=4209; i=matttbe@kernel.org; h=from:subject:message-id; bh=8LIsOEqtSsrfu9qHTzMNH2/0Gf2ZWWs2EYf0FocaPOw=; b=owEBbQKS/ZANAwAIAfa3gk9CaaBzAcsmYgBmsKETYnbtnagggycrBRFfUzpqkuYKa2v26Lq5b uqgkE4ywpqJAjMEAAEIAB0WIQToy4X3aHcFem4n93r2t4JPQmmgcwUCZrChEwAKCRD2t4JPQmmg cyzHD/9YnNPETh1sMJ+T4Wo5R3EiEKdjsr9vv3vffuSN8zU07ShtuaWwFxWCpRzQ+h0ZJ/XyWKg omz5eGlqL/HXFbQX23/31RwHouZXCaarWK+JMaeTBXU7HuW96mI7yavRni4c2Kz73RvLD7+YRA+ r8rdpRzL7w/JNWWtayRalp9/xAsTL52sTnvm+KPWS1Sw7c/2dzW7KyebOTanOy99xsnILidVBBp Da8v5Xdb8218qf8OfDJZrb1GQYYefFpK1pP1KkBnXYkwN7J5rZddr2LanZrxD02hm7awMVJnrrR P33y4JxHs5Q5qtQp06WdagHnU7OsN/bhjniD77FGccS1kYWz1F+FxmrLPkFK4zGKN+KdzJeqlov RN7vL4Q7Ix6OBU+tlTrkAuh1lEKJt4SujsJeIn7Xma4tZ2G729TOqGbmCGOEb/JvZlNSEIyCirS ao6kgtewVC0NE4myPQeOU031SjRz9OZkmfVDwdXpmkaDihGqrWXqjTfI8EgcWfkc6P4ySvK1wRA LVosVdhhZFltvoZs9ggFcYd/ziwL+/0jOJxLR7tr00AIHKEytzQyj9SldhJ1yrfy+zRTV55r7Qf FewFtrWM+qxD8vZ5K7FpmuEOOI44YJxBQWeqO6IIgIX9ElbGoBGTAY2cAyZOesEBZg4B0dMS1Lm PFhAxTUplZw2fCw== X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 From: Nicolas Rybowski Move Nicolas' patch into bpf selftests directory. This example adds a different mark (SO_MARK) on each subflow, and changes the TCP CC only on the first subflow. From the userspace, an application can do a setsockopt() on an MPTCP socket, and typically the same value will be propagated to all subflows (paths). If someone wants to have different values per subflow, the recommended way is to use BPF. So it is good to add such example here, and make sure there is no regressions. This example shows how it is possible to: Identify the parent msk of an MPTCP subflow. Put different sockopt for each subflow of a same MPTCP connection. Here especially, two different behaviours are implemented: A socket mark (SOL_SOCKET SO_MARK) is put on each subflow of a same MPTCP connection. The order of creation of the current subflow defines its mark. The TCP CC algorithm of the very first subflow of an MPTCP connection is set to "reno". This is just to show it is possible to identify an MPTCP connection, and set socket options, from different SOL levels, per subflow. "reno" has been picked because it is built-in and usually not set as default one. It is easy to verify with 'ss' that these modifications have been applied correctly. That's what the next patch is going to do. Nicolas' code comes from: commit 4d120186e4d6 ("bpf:examples: update mptcp_set_mark_kern.c") from the MPTCP repo https://github.com/multipath-tcp/mptcp_net-next (the "scripts" branch), and it has been adapted by Geliang. Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/76 Co-developed-by: Geliang Tang Signed-off-by: Geliang Tang Signed-off-by: Nicolas Rybowski Reviewed-by: Mat Martineau Signed-off-by: Matthieu Baerts (NGI0) --- Notes: - v1 -> v2: - The commit message has been updated: why setting multiple socket options, why reno, the verification is done in a later patch (different author). (Alexei) - v2 -> v3: - Only #include "bpf_tracing_net.h", linked to: https://lore.kernel.org/20240509175026.3423614-1-martin.lau@linux.dev --- tools/testing/selftests/bpf/progs/mptcp_subflow.c | 59 +++++++++++++++++++= ++++ 1 file changed, 59 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/mptcp_subflow.c b/tools/test= ing/selftests/bpf/progs/mptcp_subflow.c new file mode 100644 index 000000000000..bc572e1d6df8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/mptcp_subflow.c @@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020, Tessares SA. */ +/* Copyright (c) 2024, Kylin Software */ + +/* vmlinux.h, bpf_helpers.h and other 'define' */ +#include "bpf_tracing_net.h" + +char _license[] SEC("license") =3D "GPL"; + +char cc[TCP_CA_NAME_MAX] =3D "reno"; + +/* Associate a subflow counter to each token */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); + __uint(max_entries, 100); +} mptcp_sf SEC(".maps"); + +SEC("sockops") +int mptcp_subflow(struct bpf_sock_ops *skops) +{ + __u32 init =3D 1, key, mark, *cnt; + struct mptcp_sock *msk; + struct bpf_sock *sk; + int err; + + if (skops->op !=3D BPF_SOCK_OPS_TCP_CONNECT_CB) + return 1; + + sk =3D skops->sk; + if (!sk) + return 1; + + msk =3D bpf_skc_to_mptcp_sock(sk); + if (!msk) + return 1; + + key =3D msk->token; + cnt =3D bpf_map_lookup_elem(&mptcp_sf, &key); + if (cnt) { + /* A new subflow is added to an existing MPTCP connection */ + __sync_fetch_and_add(cnt, 1); + mark =3D *cnt; + } else { + /* A new MPTCP connection is just initiated and this is its primary subf= low */ + bpf_map_update_elem(&mptcp_sf, &key, &init, BPF_ANY); + mark =3D init; + } + + /* Set the mark of the subflow's socket based on appearance order */ + err =3D bpf_setsockopt(skops, SOL_SOCKET, SO_MARK, &mark, sizeof(mark)); + if (err < 0) + return 1; + if (mark =3D=3D 1) + err =3D bpf_setsockopt(skops, SOL_TCP, TCP_CONGESTION, cc, TCP_CA_NAME_M= AX); + + return 1; +} --=20 2.45.2 From nobody Thu Sep 19 01:01:47 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB135155353; Mon, 5 Aug 2024 09:53:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722851618; cv=none; b=JPXGxo8VLjeGaGfoRCTP0fvIv0QqrV2/9JWVnQ1XGhFOuTD7cmDNKy+OawxY85+OGIa7D7PfEW9ScCwZUfNfyVwFcn5SLts6MSmfLVoWO5DeyxpOyJzBy1hGTFbmG5KsdXE1woXLGVAwrBNOdqkw+Vz2mnibQ26vZiLfGn7EPYw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722851618; c=relaxed/simple; bh=v8V8NsYqx5U0o5pTyv3XffdmocZLP0sMUWIWr5UKJPM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=V1QvQ/H9qV/G++o4v0inG5/RgvnRKnMaeev7dBiHRlo5/zSi2QV08jneLzJb3nGDOxFplt70N6c3b/BjnY2mqEcBFP1a93TWF1qCt36TBzdt2zSoOCzrjc+HJs1EZTgKZrHAxNSozEhxA1f5ouZtk/cxdGhTv0JiWHWi+e7QQCU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ewYqvmRB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ewYqvmRB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08298C4AF0D; Mon, 5 Aug 2024 09:53:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722851618; bh=v8V8NsYqx5U0o5pTyv3XffdmocZLP0sMUWIWr5UKJPM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=ewYqvmRB8ArH3+CYzIX1ly4M/jHVUnS9iqEZWQTA8/aqTs0HbxAFJXKrrvGIlUDje ohOXVKUrdaB2jbMivPbEySD7/xMyQb6zJI1jsr/7S4zCuzMOdKCAW4KNmFOoHDZdSt VMGCIyWPd6bToOEOsDiG43nX+QU9Im4PisnNq9YTQS35p3Rn1rbYWgJ+oGT+VDU2Ct PYfd+fJfah5Lw5O9QunNRApweoiOLVHRoPuKwiaLAanMM6TpRngmX/KwPi5hLYaG8k /tHZ/D6kYX/BatgY5q2vt9I2NsxLm6wsg+0c2iOa6VHXMauXGnAKMw7HnBweThtyHR 84SGvFOLEsa2w== From: "Matthieu Baerts (NGI0)" Date: Mon, 05 Aug 2024 11:52:57 +0200 Subject: [PATCH bpf-next v4 2/2] selftests/bpf: Add mptcp subflow subtest Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240805-upstream-bpf-next-20240506-mptcp-subflow-test-v4-2-2b4ca6994993@kernel.org> References: <20240805-upstream-bpf-next-20240506-mptcp-subflow-test-v4-0-2b4ca6994993@kernel.org> In-Reply-To: <20240805-upstream-bpf-next-20240506-mptcp-subflow-test-v4-0-2b4ca6994993@kernel.org> To: mptcp@lists.linux.dev, Mat Martineau , Geliang Tang , Andrii Nakryiko , Eduard Zingerman , Mykola Lysenko , Alexei Starovoitov , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Shuah Khan Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, "Matthieu Baerts (NGI0)" , Geliang Tang X-Mailer: b4 0.14.1 X-Developer-Signature: v=1; a=openpgp-sha256; l=4710; i=matttbe@kernel.org; h=from:subject:message-id; bh=2hmbi+FkyvyFNw1rfk0tf+u5UMK3sb/cD/zwQAToiXI=; b=owEBbQKS/ZANAwAIAfa3gk9CaaBzAcsmYgBmsKETTOzGznQXdDJm+VGh6aU5ocfU8uVAzVdjG O9BUcf6rfaJAjMEAAEIAB0WIQToy4X3aHcFem4n93r2t4JPQmmgcwUCZrChEwAKCRD2t4JPQmmg c2ZCEADKCMA3p85IK3NP7i/Nay+1tjmxstFtFsczsMZDfXfhDd2Ox7ZuElX3q86F0tppDUs0tHK ce7lb2GfpHbiD+lzkdDHSCDWM82M+UzLx4L8rQTc+RW5qhPTYrraGXikEfjxdGhH4SOZmg1aVyj EIcVDbVFW+sWN2LdbSPRYVpxxWbMnE7pKeAir2YjhoMzTdpVLQsVMKmStUZoVC5u4cOY55F5EED M3c1B1zfhWb98xHiJFN/j8oJPzt7yW5E36d1xtgTpP7BKtoT6n7y2mQ2ianymOAzo6ZxoQ7NTjj uyleEmKblp0MY7tSncNhhIoJELfvAh97mYGUJmvvzaM2vMIaX1YmkVwaXDcWB9g/WfwXxxP49Ub HhLfPvraLKUcivtz0JQrZ2yu9puNCEp70u87Qaxw2HQb8C3EB/MAwyYHkMqWvTVyeBaoFEDdwzU U6iYglj8Ao5piBYRBQVcEhHKPBC+wNcFV7bB/IsnX06XhUc7+GKEQWTCtcwJGWJrBskhEDiKrJx ZnpnUP8Arupu6hApglxxG3CiC4ZJ/maheZi+o58RP04spfg01Wn8T5AFyvldK6kgCQmnnqYEja1 wuOf+ZzryHq2cYWjVSXWYfTG2W1PnfnW1punVdmuXrdNlTB1g/O9xUDhUmWBUmOMvTwq3lDq5Iw WdPPWJCO9O8e4bw== X-Developer-Key: i=matttbe@kernel.org; a=openpgp; fpr=E8CB85F76877057A6E27F77AF6B7824F4269A073 From: Geliang Tang This patch adds a subtest named test_subflow to load and verify the newly added mptcp subflow example in test_mptcp. Add a helper endpoint_init() to add a new subflow endpoint. Add another helper ss_search() to verify the fwmark and congestion values set by mptcp_subflow prog using setsockopts. Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/76 Signed-off-by: Geliang Tang Reviewed-by: Mat Martineau Signed-off-by: Matthieu Baerts (NGI0) --- Notes: - v2 -> v3: - Use './mptcp_pm_nl_ctl' instead of 'ip mptcp', not supported by the BPF CI running IPRoute 5.5.0. - Use SYS_NOFAIL() in _ss_search() instead of calling system() - v3 -> v4: - Drop './mptcp_pm_nl_ctl', but skip this new test if 'ip mptcp' is not supported. --- tools/testing/selftests/bpf/prog_tests/mptcp.c | 105 +++++++++++++++++++++= ++++ 1 file changed, 105 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/mptcp.c b/tools/testing= /selftests/bpf/prog_tests/mptcp.c index d2ca32fa3b21..d06be03cc0f0 100644 --- a/tools/testing/selftests/bpf/prog_tests/mptcp.c +++ b/tools/testing/selftests/bpf/prog_tests/mptcp.c @@ -9,8 +9,12 @@ #include "network_helpers.h" #include "mptcp_sock.skel.h" #include "mptcpify.skel.h" +#include "mptcp_subflow.skel.h" =20 #define NS_TEST "mptcp_ns" +#define ADDR_1 "10.0.1.1" +#define ADDR_2 "10.0.1.2" +#define PORT_1 10001 =20 #ifndef IPPROTO_MPTCP #define IPPROTO_MPTCP 262 @@ -335,10 +339,111 @@ static void test_mptcpify(void) close(cgroup_fd); } =20 +static int endpoint_init(char *flags) +{ + SYS(fail, "ip -net %s link add veth1 type veth peer name veth2", NS_TEST); + SYS(fail, "ip -net %s addr add %s/24 dev veth1", NS_TEST, ADDR_1); + SYS(fail, "ip -net %s link set dev veth1 up", NS_TEST); + SYS(fail, "ip -net %s addr add %s/24 dev veth2", NS_TEST, ADDR_2); + SYS(fail, "ip -net %s link set dev veth2 up", NS_TEST); + if (SYS_NOFAIL("ip -net %s mptcp endpoint add %s %s", NS_TEST, ADDR_2, fl= ags)) { + printf("'ip mptcp' not supported, skip this test.\n"); + test__skip(); + goto fail; + } + + return 0; +fail: + return -1; +} + +static int _ss_search(char *src, char *dst, char *port, char *keyword) +{ + return SYS_NOFAIL("ip netns exec %s ss -enita src %s dst %s %s %d | grep = -q '%s'", + NS_TEST, src, dst, port, PORT_1, keyword); +} + +static int ss_search(char *src, char *keyword) +{ + return _ss_search(src, ADDR_1, "dport", keyword); +} + +static void run_subflow(char *new) +{ + int server_fd, client_fd, err; + char cc[TCP_CA_NAME_MAX]; + socklen_t len =3D sizeof(cc); + + server_fd =3D start_mptcp_server(AF_INET, ADDR_1, PORT_1, 0); + if (!ASSERT_GE(server_fd, 0, "start_mptcp_server")) + return; + + client_fd =3D connect_to_fd(server_fd, 0); + if (!ASSERT_GE(client_fd, 0, "connect to fd")) + goto fail; + + err =3D getsockopt(server_fd, SOL_TCP, TCP_CONGESTION, cc, &len); + if (!ASSERT_OK(err, "getsockopt(srv_fd, TCP_CONGESTION)")) + goto fail; + + send_byte(client_fd); + + ASSERT_OK(ss_search(ADDR_1, "fwmark:0x1"), "ss_search fwmark:0x1"); + ASSERT_OK(ss_search(ADDR_2, "fwmark:0x2"), "ss_search fwmark:0x2"); + ASSERT_OK(ss_search(ADDR_1, new), "ss_search new cc"); + ASSERT_OK(ss_search(ADDR_2, cc), "ss_search default cc"); + + close(client_fd); +fail: + close(server_fd); +} + +static void test_subflow(void) +{ + int cgroup_fd, prog_fd, err; + struct mptcp_subflow *skel; + struct nstoken *nstoken; + + cgroup_fd =3D test__join_cgroup("/mptcp_subflow"); + if (!ASSERT_GE(cgroup_fd, 0, "join_cgroup: mptcp_subflow")) + return; + + skel =3D mptcp_subflow__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open_load: mptcp_subflow")) + goto close_cgroup; + + err =3D mptcp_subflow__attach(skel); + if (!ASSERT_OK(err, "skel_attach: mptcp_subflow")) + goto skel_destroy; + + prog_fd =3D bpf_program__fd(skel->progs.mptcp_subflow); + err =3D bpf_prog_attach(prog_fd, cgroup_fd, BPF_CGROUP_SOCK_OPS, 0); + if (!ASSERT_OK(err, "prog_attach")) + goto skel_destroy; + + nstoken =3D create_netns(); + if (!ASSERT_OK_PTR(nstoken, "create_netns: mptcp_subflow")) + goto skel_destroy; + + if (endpoint_init("subflow") < 0) + goto close_netns; + + run_subflow(skel->data->cc); + +close_netns: + cleanup_netns(nstoken); +skel_destroy: + mptcp_subflow__destroy(skel); +close_cgroup: + close(cgroup_fd); +} + void test_mptcp(void) { if (test__start_subtest("base")) test_base(); if (test__start_subtest("mptcpify")) test_mptcpify(); + if (test__start_subtest("subflow")) + test_subflow(); } --=20 2.45.2