From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4601543D516 for ; Fri, 15 May 2026 09:46:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838397; cv=none; b=NLvSXxgBGdhe7XYwFFB8NaKrA7QHPHfdICP85vL+2/57kboj/XKss8RQVZXZoCM2UlhMo37IveHjHKt4+Sp0EdTHdII5fEpV4yeCbmSEYE/lBCT90L7zrjNVDu2mf7pkAXAfL/5kMLJ0BFNoaBwLnpGwPm0EsaJm2gTs7ip6FI8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838397; c=relaxed/simple; bh=2qSXmmdRji29Z589AQWpwOeekelWKlF0zayOHrm0fxo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=apoYnDXj6MLKcHEI/XxRt1AJQQ5CEFlXvPAQouxcDUSjkX/GzFLDojdSoNJO3OTZrPOhRtxeI/ZlQ/FdTFNNnWT8BAm6GACZ260XM1uX71s9rC/4At+Ps1MxUgGBYe9rirr9/HRl9dHFi8N0EEdkUA0SrR39jJHYrCpICpDCYjc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=B5nhsAxM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="B5nhsAxM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0DE4AC2BCF7; Fri, 15 May 2026 09:46:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838397; bh=2qSXmmdRji29Z589AQWpwOeekelWKlF0zayOHrm0fxo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B5nhsAxMDoGmWtGcX8h4LGVf6nlN+IYMSnqJ844A0xm/Ak+X1PiZSpB0XASfW5Su4 Q8WH1wtrtPIh2fNmsgOf4eKbPSn9ktCLNBXognAQUhQB8sOXArvI81ZBmv9u0c1JsF 4ge4i/8T0xRps1VmQjyxYCCaJFh/vFB+WNmhhhfbcm9AAhzOLea3JQGiYME0MTLj/T 5P4BUrjOkuG1RSYZTKK+Oho7hV6i75CJW968CB5Tvh7AJ8SYu+5qLom6rOjmh7QFcE xXYFYhBJ+2lwGxclovKXYQRD7NWjcjplU1FQd89gol/+8wuKo9nenzGTgcKG/HM9mN Erbg1OFfKsHPA== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 1/8] nvmet-tcp: define target tcp_proto struct Date: Fri, 15 May 2026 17:46:15 +0800 Message-ID: <77d7c48df2e6b531f3817a4434aa2625dd8993e1.1778837549.git.tanggeliang@kylinos.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang To add MPTCP support in "NVMe over TCP", the target side needs to pass IPPROTO_MPTCP to sock_create() instead of IPPROTO_TCP to create an MPTCP socket. Additionally, the setsockopt operations for this socket need to be switched to a set of MPTCP-specific functions. This patch defines the nvmet_tcp_proto structure, which contains the protocol of the socket and a set of function pointers for these socket operations. A "proto" field is also added to struct nvmet_tcp_port. A TCP-specific version of struct nvmet_tcp_proto is defined. In nvmet_tcp_add_port(), port->proto is set to nvmet_tcp_proto based on whether trtype is TCP. All locations that previously called TCP setsockopt functions are updated to call the corresponding function pointers in the nvmet_tcp_proto structure. This new nvmet_fabrics_ops is selected in nvmet_tcp_done_recv_pdu() based on the protocol type. RCU protection is added when accessing queue->port in the I/O path (nvmet_tcp_done_recv_pdu and nvmet_tcp_set_queue_sock) to prevent use-after-free when a port is removed while asynchronous operations (e.g., TLS handshake) are pending. The port structure is released using kfree_rcu() in nvmet_tcp_remove_port(). The queue->port pointer is cleared using rcu_assign_pointer() and synchronized with synchronize_rcu() in nvmet_tcp_destroy_port_queues(). Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- drivers/nvme/target/tcp.c | 74 +++++++++++++++++++++++++++++++++------ 1 file changed, 63 insertions(+), 11 deletions(-) diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 164a564ba3b4..3ed3767dceaf 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -18,6 +18,7 @@ #include #include #include +#include #include =20 #include "nvmet.h" @@ -198,12 +199,24 @@ struct nvmet_tcp_queue { void (*write_space)(struct sock *); }; =20 +struct nvmet_tcp_proto { + int protocol; + void (*set_reuseaddr)(struct sock *sk); + void (*set_nodelay)(struct sock *sk); + void (*set_priority)(struct sock *sk, u32 priority); + void (*no_linger)(struct sock *sk); + void (*set_tos)(struct sock *sk, int val); + const struct nvmet_fabrics_ops *ops; +}; + struct nvmet_tcp_port { + struct rcu_head rcu; struct socket *sock; struct work_struct accept_work; struct nvmet_port *nport; struct sockaddr_storage addr; void (*data_ready)(struct sock *); + const struct nvmet_tcp_proto *proto; }; =20 static DEFINE_IDA(nvmet_tcp_queue_ida); @@ -1044,6 +1057,7 @@ static int nvmet_tcp_done_recv_pdu(struct nvmet_tcp_q= ueue *queue) { struct nvme_tcp_hdr *hdr =3D &queue->pdu.cmd.hdr; struct nvme_command *nvme_cmd =3D &queue->pdu.cmd.cmd; + struct nvmet_tcp_port *port; struct nvmet_req *req; int ret; =20 @@ -1081,7 +1095,14 @@ static int nvmet_tcp_done_recv_pdu(struct nvmet_tcp_= queue *queue) req =3D &queue->cmd->req; memcpy(req->cmd, nvme_cmd, sizeof(*nvme_cmd)); =20 - if (unlikely(!nvmet_req_init(req, &queue->nvme_sq, &nvmet_tcp_ops))) { + rcu_read_lock(); + port =3D rcu_dereference(queue->port); + if (!port || !port->proto) { + rcu_read_unlock(); + return -EINVAL; + } + rcu_read_unlock(); + if (unlikely(!nvmet_req_init(req, &queue->nvme_sq, port->proto->ops))) { pr_err("failed cmd %p id %d opcode %d, data_len: %d, status: %04x\n", req->cmd, req->cmd->common.command_id, req->cmd->common.opcode, @@ -1697,6 +1718,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_= queue *queue) { struct socket *sock =3D queue->sock; struct inet_sock *inet =3D inet_sk(sock->sk); + struct nvmet_tcp_port *port; int ret; =20 ret =3D kernel_getsockname(sock, @@ -1709,19 +1731,28 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tc= p_queue *queue) if (ret < 0) return ret; =20 + rcu_read_lock(); + port =3D rcu_dereference(queue->port); + if (!port || !port->proto || + port->proto->protocol !=3D sock->sk->sk_protocol) { + rcu_read_unlock(); + return -EINVAL; + } + rcu_read_unlock(); + /* * Cleanup whatever is sitting in the TCP transmit queue on socket * close. This is done to prevent stale data from being sent should * the network connection be restored before TCP times out. */ - sock_no_linger(sock->sk); + port->proto->no_linger(sock->sk); =20 if (so_priority > 0) - sock_set_priority(sock->sk, so_priority); + port->proto->set_priority(sock->sk, so_priority); =20 /* Set socket type of service */ if (inet->rcv_tos > 0) - ip_sock_set_tos(sock->sk, inet->rcv_tos); + port->proto->set_tos(sock->sk, inet->rcv_tos); =20 ret =3D 0; write_lock_bh(&sock->sk->sk_callback_lock); @@ -2041,6 +2072,16 @@ static void nvmet_tcp_listen_data_ready(struct sock = *sk) read_unlock_bh(&sk->sk_callback_lock); } =20 +static const struct nvmet_tcp_proto nvmet_tcp_proto =3D { + .protocol =3D IPPROTO_TCP, + .set_reuseaddr =3D sock_set_reuseaddr, + .set_nodelay =3D tcp_sock_set_nodelay, + .set_priority =3D sock_set_priority, + .no_linger =3D sock_no_linger, + .set_tos =3D ip_sock_set_tos, + .ops =3D &nvmet_tcp_ops, +}; + static int nvmet_tcp_add_port(struct nvmet_port *nport) { struct nvmet_tcp_port *port; @@ -2065,6 +2106,13 @@ static int nvmet_tcp_add_port(struct nvmet_port *npo= rt) goto err_port; } =20 + if (nport->disc_addr.trtype =3D=3D NVMF_TRTYPE_TCP) { + port->proto =3D &nvmet_tcp_proto; + } else { + ret =3D -EINVAL; + goto err_port; + } + ret =3D inet_pton_with_scope(&init_net, af, nport->disc_addr.traddr, nport->disc_addr.trsvcid, &port->addr); if (ret) { @@ -2079,7 +2127,7 @@ static int nvmet_tcp_add_port(struct nvmet_port *npor= t) port->nport->inline_data_size =3D NVMET_TCP_DEF_INLINE_DATA_SIZE; =20 ret =3D sock_create(port->addr.ss_family, SOCK_STREAM, - IPPROTO_TCP, &port->sock); + port->proto->protocol, &port->sock); if (ret) { pr_err("failed to create a socket\n"); goto err_port; @@ -2088,10 +2136,10 @@ static int nvmet_tcp_add_port(struct nvmet_port *np= ort) port->sock->sk->sk_user_data =3D port; port->data_ready =3D port->sock->sk->sk_data_ready; port->sock->sk->sk_data_ready =3D nvmet_tcp_listen_data_ready; - sock_set_reuseaddr(port->sock->sk); - tcp_sock_set_nodelay(port->sock->sk); + port->proto->set_reuseaddr(port->sock->sk); + port->proto->set_nodelay(port->sock->sk); if (so_priority > 0) - sock_set_priority(port->sock->sk, so_priority); + port->proto->set_priority(port->sock->sk, so_priority); =20 ret =3D kernel_bind(port->sock, (struct sockaddr_unsized *)&port->addr, sizeof(port->addr)); @@ -2124,10 +2172,14 @@ static void nvmet_tcp_destroy_port_queues(struct nv= met_tcp_port *port) struct nvmet_tcp_queue *queue; =20 mutex_lock(&nvmet_tcp_queue_mutex); - list_for_each_entry(queue, &nvmet_tcp_queue_list, queue_list) - if (queue->port =3D=3D port) + list_for_each_entry(queue, &nvmet_tcp_queue_list, queue_list) { + if (rcu_dereference(queue->port) =3D=3D port) { + rcu_assign_pointer(queue->port, NULL); kernel_sock_shutdown(queue->sock, SHUT_RDWR); + } + } mutex_unlock(&nvmet_tcp_queue_mutex); + synchronize_rcu(); } =20 static void nvmet_tcp_remove_port(struct nvmet_port *nport) @@ -2146,7 +2198,7 @@ static void nvmet_tcp_remove_port(struct nvmet_port *= nport) nvmet_tcp_destroy_port_queues(port); =20 sock_release(port->sock); - kfree(port); + kfree_rcu(port, rcu); } =20 static void nvmet_tcp_delete_ctrl(struct nvmet_ctrl *ctrl) --=20 2.53.0 From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCB243C8C65 for ; Fri, 15 May 2026 09:46:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838399; cv=none; b=QfXAao+jtaXrDjEzcFTLveVUq0iiFIT31u5EWYKqeBsMvKzRPwefmqCViOG4Ixr5FaonqO1EuA7rCGU/WSUB4nWm+mAy2EGT7VEMq3+OiVtQG6Evj5Jta8Bl5moF8FjvCW6SEIRNSYb86OWz0Szz+3/HfN+B7Jlp8GC4ISc62KE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838399; c=relaxed/simple; bh=Oa36ved91tYaSuZRQu9+Bn9OeF64Iql7ChqZwtKQ4MI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=g/oOCFzQ4DfkDDAdMK4tkRZMwYYS3OnsxPjpQowSicuxGYgHAz/ALe8fCx+DbhvafCESjLO0S6XwG46oDhihRlkw+fFuSV8ebCFNVz8TpGGBVP8ejMwa/VsSTaLfTViI0tYCg0u3L+QmEBjQ8GmesXykfIP3zptJ2ZHji1CtdOY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Arp0C8/T; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Arp0C8/T" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 974F9C2BCF5; Fri, 15 May 2026 09:46:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838399; bh=Oa36ved91tYaSuZRQu9+Bn9OeF64Iql7ChqZwtKQ4MI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Arp0C8/TNvvJQZMNt71LMzt5z8O2ACJT/HqmMHKsBhxxkRbyhi3n/Q5aqc6uAmKeJ +gHTDwPzyA3H444sHjruEVucc/99wKC7yBL+/4d4XrzuhiLq1j1SLE6bVeqv6jW5S5 4MZbtF7Lj3SSVrd/+6W5ELWnZxkU5VGDF8C+56p9sML7kn5h4Ah+3rcpYGuvrOH4Fh EqyCBTYNWm4QZFoc3w17p908+vhJZ9oNvxNcx85jWz1VTexfy7kJcpMq2e+9IswjC8 O64opBnzYlThW/Nw6iXuXvSzxK1adsZ7RxvsTMOMEWFInvUMtmI9jN3Jq5lGvkHolz Dq16gBswM+cFg== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 2/8] nvmet-tcp: register target mptcp transport Date: Fri, 15 May 2026 17:46:16 +0800 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang This patch adds a new nvme target transport type NVMF_TRTYPE_MPTCP for MPTCP. And defines a new nvmet_fabrics_ops named nvmet_mptcp_ops, which is almost the same as nvmet_tcp_ops except .type. It is registered in nvmet_tcp_init() and unregistered in nvmet_tcp_exit(). A MODULE_ALIAS for "nvmet-transport-4" is also added. v2: - use trtype instead of tsas (Hannes). v3: - check mptcp protocol from disc_addr.trtype instead of passing a parameter (Hannes). v4: - check CONFIG_MPTCP. Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- drivers/nvme/target/configfs.c | 1 + drivers/nvme/target/tcp.c | 27 +++++++++++++++++++++++++++ include/linux/nvme.h | 1 + 3 files changed, 29 insertions(+) diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c index b88f897f06e2..51fc0f4d0c32 100644 --- a/drivers/nvme/target/configfs.c +++ b/drivers/nvme/target/configfs.c @@ -37,6 +37,7 @@ static struct nvmet_type_name_map nvmet_transport[] =3D { { NVMF_TRTYPE_RDMA, "rdma" }, { NVMF_TRTYPE_FC, "fc" }, { NVMF_TRTYPE_TCP, "tcp" }, + { NVMF_TRTYPE_MPTCP, "mptcp" }, { NVMF_TRTYPE_PCI, "pci" }, { NVMF_TRTYPE_LOOP, "loop" }, }; diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 3ed3767dceaf..d42bcd809c54 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -2283,6 +2283,21 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = =3D { .host_traddr =3D nvmet_tcp_host_port_addr, }; =20 +#ifdef CONFIG_MPTCP +static const struct nvmet_fabrics_ops nvmet_mptcp_ops =3D { + .owner =3D THIS_MODULE, + .type =3D NVMF_TRTYPE_MPTCP, + .msdbd =3D 1, + .add_port =3D nvmet_tcp_add_port, + .remove_port =3D nvmet_tcp_remove_port, + .queue_response =3D nvmet_tcp_queue_response, + .delete_ctrl =3D nvmet_tcp_delete_ctrl, + .install_queue =3D nvmet_tcp_install_queue, + .disc_traddr =3D nvmet_tcp_disc_port_addr, + .host_traddr =3D nvmet_tcp_host_port_addr, +}; +#endif + static int __init nvmet_tcp_init(void) { int ret; @@ -2296,6 +2311,14 @@ static int __init nvmet_tcp_init(void) if (ret) goto err; =20 +#ifdef CONFIG_MPTCP + ret =3D nvmet_register_transport(&nvmet_mptcp_ops); + if (ret) { + nvmet_unregister_transport(&nvmet_tcp_ops); + goto err; + } +#endif + return 0; err: destroy_workqueue(nvmet_tcp_wq); @@ -2306,6 +2329,9 @@ static void __exit nvmet_tcp_exit(void) { struct nvmet_tcp_queue *queue; =20 +#ifdef CONFIG_MPTCP + nvmet_unregister_transport(&nvmet_mptcp_ops); +#endif nvmet_unregister_transport(&nvmet_tcp_ops); =20 flush_workqueue(nvmet_wq); @@ -2325,3 +2351,4 @@ module_exit(nvmet_tcp_exit); MODULE_DESCRIPTION("NVMe target TCP transport driver"); MODULE_LICENSE("GPL v2"); MODULE_ALIAS("nvmet-transport-3"); /* 3 =3D=3D NVMF_TRTYPE_TCP */ +MODULE_ALIAS("nvmet-transport-4"); /* 4 =3D=3D NVMF_TRTYPE_MPTCP */ diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 041f30931a90..0eada1e0c652 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -68,6 +68,7 @@ enum { NVMF_TRTYPE_RDMA =3D 1, /* RDMA */ NVMF_TRTYPE_FC =3D 2, /* Fibre Channel */ NVMF_TRTYPE_TCP =3D 3, /* TCP/IP */ + NVMF_TRTYPE_MPTCP =3D 4, /* Multipath TCP */ NVMF_TRTYPE_LOOP =3D 254, /* Reserved for host usage */ NVMF_TRTYPE_MAX, }; --=20 2.53.0 From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 818DB3C8C65 for ; Fri, 15 May 2026 09:46:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838402; cv=none; b=rJdupBb3KZg2AwbZSll/KQdfkC5UNBR/sttS3PoN4iZexY8oalpEO9qnrOkg6fJ6Er6tynurTBDEsX/HkT+M1QVX9ViS0uoVjYc2XKeX8qOlG+5kPnzNwmGaLMLapSLltgMTPSnJzFeV+a3+N3neD6s5KZrrvhaSOlZIiN3MHLI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838402; c=relaxed/simple; bh=NmRaFh84UgzZ9RhchAsyttQe0D3cwa8GlD2z69Yn5P4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OxZ7EgtGL+CCkjCnP9IftR8yVEPrty7PKk6wN+b/XA+u+uY98ANKLfEBLMb5Poa8x/KKN4KqVIOCGxWFnUNHYMzucOGEp9SBo7uyt9iXUJv0XWFRr+bmfsF6Fb2Tpt9ack7OovXHFqV3bH5H9/UzRcoZARooO4yOvw/acLF9UDc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=K4n+9/0c; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="K4n+9/0c" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 11758C2BCB0; Fri, 15 May 2026 09:46:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838402; bh=NmRaFh84UgzZ9RhchAsyttQe0D3cwa8GlD2z69Yn5P4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K4n+9/0cQsVtgV9hR8tl1lt1376A0sNlf/9vRcVYHxrbzS2itGDaY3FKfjPnz46cZ BnnXI+nzj208zX8PMe50jZLpmNS8DMh4sHUiIQ/oAFM0FnwO+bNFe4SexxLpevKBLp Qb13srhFbCpa4RVNJHvpb6CBaRnhDQl2/DeW1QDAdn0vgT3ZzTIJ0hYWJJWTv4tiA9 1Fzv6yxQVhdfzTn4+4RAm8zD0DRV9/Gej6ynFZHMSpoJHbfkC6FNCtFRslr9zP8hdb nQMmerlD6lH5pEn6gswRE4l/UTLelIMw4NWGaCeqap+QBh9mgn72KxvxhQ2KhECduC LJzRir0Yp143w== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 3/8] nvmet-tcp: implement target mptcp proto Date: Fri, 15 May 2026 17:46:17 +0800 Message-ID: <4bbbcfc5282f000f79508d1b7cb1ec5135148d50.1778837549.git.tanggeliang@kylinos.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang This patch introduces a new NVMe target transport type NVMF_TRTYPE_MPTCP to support MPTCP. An MPTCP-specific version of struct nvmet_tcp_proto is implemented, and it is assigned to port->proto when the transport type is MPTCP. Dedicated MPTCP helpers are introduced for setting socket options. These helpers set the values on the first subflow socket of an MPTCP connection. The values are then synchronized to other newly created subflows in sync_socket_options(). Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- drivers/nvme/target/tcp.c | 19 ++++++++ include/net/mptcp.h | 20 ++++++++ net/mptcp/sockopt.c | 99 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 138 insertions(+) diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index d42bcd809c54..3cb0e60fb0d7 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -225,6 +225,9 @@ static DEFINE_MUTEX(nvmet_tcp_queue_mutex); =20 static struct workqueue_struct *nvmet_tcp_wq; static const struct nvmet_fabrics_ops nvmet_tcp_ops; +#ifdef CONFIG_MPTCP +static const struct nvmet_fabrics_ops nvmet_mptcp_ops; +#endif static void nvmet_tcp_free_cmd(struct nvmet_tcp_cmd *c); static void nvmet_tcp_free_cmd_buffers(struct nvmet_tcp_cmd *cmd); =20 @@ -2082,6 +2085,18 @@ static const struct nvmet_tcp_proto nvmet_tcp_proto = =3D { .ops =3D &nvmet_tcp_ops, }; =20 +#ifdef CONFIG_MPTCP +static const struct nvmet_tcp_proto nvmet_mptcp_proto =3D { + .protocol =3D IPPROTO_MPTCP, + .set_reuseaddr =3D mptcp_sock_set_reuseaddr, + .set_nodelay =3D mptcp_sock_set_nodelay, + .set_priority =3D mptcp_sock_set_priority, + .no_linger =3D mptcp_sock_no_linger, + .set_tos =3D mptcp_sock_set_tos, + .ops =3D &nvmet_mptcp_ops, +}; +#endif + static int nvmet_tcp_add_port(struct nvmet_port *nport) { struct nvmet_tcp_port *port; @@ -2108,6 +2123,10 @@ static int nvmet_tcp_add_port(struct nvmet_port *npo= rt) =20 if (nport->disc_addr.trtype =3D=3D NVMF_TRTYPE_TCP) { port->proto =3D &nvmet_tcp_proto; +#ifdef CONFIG_MPTCP + } else if (nport->disc_addr.trtype =3D=3D NVMF_TRTYPE_MPTCP) { + port->proto =3D &nvmet_mptcp_proto; +#endif } else { ret =3D -EINVAL; goto err_port; diff --git a/include/net/mptcp.h b/include/net/mptcp.h index 4cf59e83c1c5..91ce7b9b639d 100644 --- a/include/net/mptcp.h +++ b/include/net/mptcp.h @@ -237,6 +237,16 @@ static inline __be32 mptcp_reset_option(const struct s= k_buff *skb) } =20 void mptcp_active_detect_blackhole(struct sock *sk, bool expired); + +void mptcp_sock_set_reuseaddr(struct sock *sk); + +void mptcp_sock_set_nodelay(struct sock *sk); + +void mptcp_sock_set_priority(struct sock *sk, u32 priority); + +void mptcp_sock_no_linger(struct sock *sk); + +void mptcp_sock_set_tos(struct sock *sk, int val); #else =20 static inline void mptcp_init(void) @@ -323,6 +333,16 @@ static inline struct request_sock *mptcp_subflow_reqsk= _alloc(const struct reques static inline __be32 mptcp_reset_option(const struct sk_buff *skb) { retu= rn htonl(0u); } =20 static inline void mptcp_active_detect_blackhole(struct sock *sk, bool exp= ired) { } + +static inline void mptcp_sock_set_reuseaddr(struct sock *sk) { } + +static inline void mptcp_sock_set_nodelay(struct sock *sk) { } + +static inline void mptcp_sock_set_priority(struct sock *sk, u32 priority) = { } + +static inline void mptcp_sock_no_linger(struct sock *sk) { } + +static inline void mptcp_sock_set_tos(struct sock *sk, int val) { } #endif /* CONFIG_MPTCP */ =20 #if IS_ENABLED(CONFIG_MPTCP_IPV6) diff --git a/net/mptcp/sockopt.c b/net/mptcp/sockopt.c index 87b5796d0135..fbcde31ef668 100644 --- a/net/mptcp/sockopt.c +++ b/net/mptcp/sockopt.c @@ -1547,6 +1547,7 @@ static void sync_socket_options(struct mptcp_sock *ms= k, struct sock *ssk) static const unsigned int tx_rx_locks =3D SOCK_RCVBUF_LOCK | SOCK_SNDBUF_= LOCK; struct sock *sk =3D (struct sock *)msk; bool keep_open; + u32 priority; =20 keep_open =3D sock_flag(sk, SOCK_KEEPOPEN); if (ssk->sk_prot->keepalive) @@ -1596,6 +1597,11 @@ static void sync_socket_options(struct mptcp_sock *m= sk, struct sock *ssk) inet_assign_bit(FREEBIND, ssk, inet_test_bit(FREEBIND, sk)); inet_assign_bit(BIND_ADDRESS_NO_PORT, ssk, inet_test_bit(BIND_ADDRESS_NO_= PORT, sk)); WRITE_ONCE(inet_sk(ssk)->local_port_range, READ_ONCE(inet_sk(sk)->local_p= ort_range)); + + ssk->sk_reuse =3D sk->sk_reuse; + priority =3D READ_ONCE(sk->sk_priority); + if (priority > 0) + sock_set_priority(ssk, priority); } =20 void mptcp_sockopt_sync_locked(struct mptcp_sock *msk, struct sock *ssk) @@ -1662,3 +1668,96 @@ int mptcp_set_rcvlowat(struct sock *sk, int val) } return 0; } + +void mptcp_sock_set_reuseaddr(struct sock *sk) +{ + struct mptcp_sock *msk =3D mptcp_sk(sk); + struct sock *ssk; + + lock_sock(sk); + sockopt_seq_inc(msk); + sk->sk_reuse =3D SK_CAN_REUSE; + ssk =3D __mptcp_nmpc_sk(msk); + if (IS_ERR(ssk)) + goto unlock; + sock_set_reuseaddr(ssk); +unlock: + release_sock(sk); +} +EXPORT_SYMBOL(mptcp_sock_set_reuseaddr); + +void mptcp_sock_set_nodelay(struct sock *sk) +{ + struct mptcp_sock *msk =3D mptcp_sk(sk); + struct sock *ssk; + + lock_sock(sk); + sockopt_seq_inc(msk); + msk->nodelay =3D true; + ssk =3D __mptcp_nmpc_sk(msk); + if (IS_ERR(ssk)) + goto unlock; + lock_sock(ssk); + __tcp_sock_set_nodelay(ssk, true); + release_sock(ssk); +unlock: + release_sock(sk); +} +EXPORT_SYMBOL(mptcp_sock_set_nodelay); + +void mptcp_sock_set_priority(struct sock *sk, u32 priority) +{ + struct mptcp_sock *msk =3D mptcp_sk(sk); + struct sock *ssk; + + lock_sock(sk); + sockopt_seq_inc(msk); + sock_set_priority(sk, priority); + ssk =3D READ_ONCE(msk->first); + if (ssk) { + sock_hold(ssk); + lock_sock(ssk); + sock_set_priority(ssk, priority); + release_sock(ssk); + sock_put(ssk); + } + release_sock(sk); +} +EXPORT_SYMBOL(mptcp_sock_set_priority); + +void mptcp_sock_no_linger(struct sock *sk) +{ + struct mptcp_sock *msk =3D mptcp_sk(sk); + struct sock *ssk; + + lock_sock(sk); + sockopt_seq_inc(msk); + WRITE_ONCE(sk->sk_lingertime, 0); + sock_set_flag(sk, SOCK_LINGER); + ssk =3D READ_ONCE(msk->first); + if (ssk) { + sock_hold(ssk); + sock_no_linger(ssk); + sock_put(ssk); + } + release_sock(sk); +} +EXPORT_SYMBOL(mptcp_sock_no_linger); + +void mptcp_sock_set_tos(struct sock *sk, int val) +{ + struct mptcp_sock *msk =3D mptcp_sk(sk); + struct sock *ssk; + + lock_sock(sk); + sockopt_seq_inc(msk); + __ip_sock_set_tos(sk, val); + ssk =3D READ_ONCE(msk->first); + if (ssk) { + sock_hold(ssk); + ip_sock_set_tos(ssk, val); + sock_put(ssk); + } + release_sock(sk); +} +EXPORT_SYMBOL(mptcp_sock_set_tos); --=20 2.53.0 From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83F2B43C06B for ; Fri, 15 May 2026 09:46:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838404; cv=none; b=YYSfbkfHFG3ERxm14XNF/w5dJFvj+uTTHpLPDQKFN8/xBhoIJr5wPvX90e+9uRLsbh0eBp4mLEN1C4hspyK2nn1QOZwAu1iYtL/R2GuBwooiN2L/hduBQTUXPJwJkIvxNeW49nmr8gK9m3XfYZlM6H9m+DGdhDfLIopTiEszRCU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838404; c=relaxed/simple; bh=8HC3fDaEofGerA5yJHiBos5fwxdcuEiJ5kjHH/ADavY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QCMUF1MML1YIaJ09/cXLy0VtJE4cGsx1Yh24lBO5uwXErO7ZXKAieHrjDuRfYt96OOdqiwjj+CmvG0tBN1bP+IAZyxDn3fc3bWvONBmUnXUoYy+OK0c8jQuC+R5Sf7dT+XgnG5aPtLsxZxShm0k2lodb70VaEGD0aCdXwEdjrn0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bmmvc///; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bmmvc///" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B3A46C2BCB0; Fri, 15 May 2026 09:46:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838404; bh=8HC3fDaEofGerA5yJHiBos5fwxdcuEiJ5kjHH/ADavY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bmmvc///bDzMynIXIpQ3cUNMQXwsXQDLmbzKTap9J29JIXLrGDZoRi2OuFmeFiFnl d/5UvNnQfhVstqP8/u4Xx7OSwTqQeUe5GeeTboq+jjskHj3XJYvhOkAhG47rZawk7Z mPT2A3H8n+JsulUiv5zCeoQEdz0qzyW6Ct5hrpBNbnq5x207UpxB0a+qyBnW1z443D XCv++fwKt00BUnL1U1EcA38HcT5lagBRncNES1YRxPexXWE4kiWeQrAg6Meuz7IlKx chK7fSAm8+VfljUARsoHV5Jbmcc230z3BYrx5cX+8XcIUleOSdR9XGX1xs1Nl3ycak 12S3Xzv5gZE8Q== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 4/8] nvme-tcp: define host tcp_proto struct Date: Fri, 15 May 2026 17:46:18 +0800 Message-ID: <78a9cac7438ad44ad2c670345279fb55529d1441.1778837549.git.tanggeliang@kylinos.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang To add MPTCP support in "NVMe over TCP", the host side needs to pass IPPROTO_MPTCP to sock_create_kern() instead of IPPROTO_TCP to create an MPTCP socket. Similar to the target-side nvmet_tcp_proto, this patch defines the host-side nvme_tcp_proto structure, which contains the protocol of the socket and a set of function pointers for socket operations. The only difference is that it defines .set_syncnt instead of .set_reuseaddr. A TCP-specific version of this structure is defined, and a proto field is added to nvme_tcp_ctrl. When the transport string is "tcp", it is assigned to ctrl->proto. All locations that previously called TCP setsockopt functions are updated to call the corresponding function pointers in the nvme_tcp_proto structure. The proto field points to a statically allocated nvme_tcp_proto structure that is never freed, so no RCU protection is needed. The controller's proto pointer is set during initialization and remains valid throughout the controller's lifetime. Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- drivers/nvme/host/tcp.c | 44 ++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 15d36d6a728e..f54b1eb86940 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -182,6 +183,16 @@ struct nvme_tcp_queue { void (*write_space)(struct sock *); }; =20 +struct nvme_tcp_proto { + int protocol; + int (*set_syncnt)(struct sock *sk, int val); + void (*set_nodelay)(struct sock *sk); + void (*no_linger)(struct sock *sk); + void (*set_priority)(struct sock *sk, u32 priority); + void (*set_tos)(struct sock *sk, int val); + const struct nvme_ctrl_ops *ops; +}; + struct nvme_tcp_ctrl { /* read only in the hot path */ struct nvme_tcp_queue *queues; @@ -198,6 +209,8 @@ struct nvme_tcp_ctrl { struct delayed_work connect_work; struct nvme_tcp_request async_req; u32 io_queues[HCTX_MAX_TYPES]; + + const struct nvme_tcp_proto *proto; }; =20 static LIST_HEAD(nvme_tcp_ctrl_list); @@ -1799,7 +1812,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nct= rl, int qid, =20 ret =3D sock_create_kern(current->nsproxy->net_ns, ctrl->addr.ss_family, SOCK_STREAM, - IPPROTO_TCP, &queue->sock); + ctrl->proto->protocol, &queue->sock); if (ret) { dev_err(nctrl->device, "failed to create socket: %d\n", ret); @@ -1816,24 +1829,24 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *n= ctrl, int qid, nvme_tcp_reclassify_socket(queue->sock); =20 /* Single syn retry */ - tcp_sock_set_syncnt(queue->sock->sk, 1); + ctrl->proto->set_syncnt(queue->sock->sk, 1); =20 /* Set TCP no delay */ - tcp_sock_set_nodelay(queue->sock->sk); + ctrl->proto->set_nodelay(queue->sock->sk); =20 /* * Cleanup whatever is sitting in the TCP transmit queue on socket * close. This is done to prevent stale data from being sent should * the network connection be restored before TCP times out. */ - sock_no_linger(queue->sock->sk); + ctrl->proto->no_linger(queue->sock->sk); =20 if (so_priority > 0) - sock_set_priority(queue->sock->sk, so_priority); + ctrl->proto->set_priority(queue->sock->sk, so_priority); =20 /* Set socket type of service */ if (nctrl->opts->tos >=3D 0) - ip_sock_set_tos(queue->sock->sk, nctrl->opts->tos); + ctrl->proto->set_tos(queue->sock->sk, nctrl->opts->tos); =20 /* Set 10 seconds timeout for icresp recvmsg */ queue->sock->sk->sk_rcvtimeo =3D 10 * HZ; @@ -2900,6 +2913,16 @@ nvme_tcp_existing_controller(struct nvmf_ctrl_option= s *opts) return found; } =20 +static const struct nvme_tcp_proto nvme_tcp_proto =3D { + .protocol =3D IPPROTO_TCP, + .set_syncnt =3D tcp_sock_set_syncnt, + .set_nodelay =3D tcp_sock_set_nodelay, + .no_linger =3D sock_no_linger, + .set_priority =3D sock_set_priority, + .set_tos =3D ip_sock_set_tos, + .ops =3D &nvme_tcp_ctrl_ops, +}; + static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, struct nvmf_ctrl_options *opts) { @@ -2964,13 +2987,20 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(st= ruct device *dev, goto out_free_ctrl; } =20 + if (!strcmp(ctrl->ctrl.opts->transport, "tcp")) { + ctrl->proto =3D &nvme_tcp_proto; + } else { + ret =3D -EINVAL; + goto out_free_ctrl; + } + ctrl->queues =3D kzalloc_objs(*ctrl->queues, ctrl->ctrl.queue_count); if (!ctrl->queues) { ret =3D -ENOMEM; goto out_free_ctrl; } =20 - ret =3D nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_tcp_ctrl_ops, 0); + ret =3D nvme_init_ctrl(&ctrl->ctrl, dev, ctrl->proto->ops, 0); if (ret) goto out_kfree_queues; =20 --=20 2.53.0 From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE14F43CEEB for ; Fri, 15 May 2026 09:46:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838406; cv=none; b=X9XS6rVmYWhpK0318EzUcbeGL7zKiFWa7lfX5KyhrXF6JiBuQ3Uz3ldRQkqbWdGgtMugfmVh4AnU2nBE8o4q6u0wQ6UV9jVY+Xln3elouinXj/Ik9aiH9iR8SzNnNvchLIyGK6PIYz7crrowPoZ5bcomJQPstorsCSgbyNfanb4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838406; c=relaxed/simple; bh=MOIKwVaXmEwicmclLIjAD1s6inLmJTXKkQjJ9N8GyQg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o83Oytv+BgTSl+g/c3dJJsWAML3hdKHcFWnqQwg9c5TeQezVlQF/piU9TGPDzFnSRF3tNnWlTpKLARWZZL9hOxSOWFV5dq03sjAyi4R6Kbjrzt4zYL6sNHheYfONG0WBM//t0JFwx9z7VbR4R6PAjB9RohIq2VC666uHaSXXaK8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pB5Q755A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pB5Q755A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CD383C2BCC7; Fri, 15 May 2026 09:46:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838406; bh=MOIKwVaXmEwicmclLIjAD1s6inLmJTXKkQjJ9N8GyQg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pB5Q755AY0ACH+Cs7MINT1GHT9fEHJSgPxUvhFaLYGoat/RQ7YNjYWnpiNWNa4wzV 27Jpcuflelleej3pM9vJwnTTC45feJkVlJAxekIxFzpytIth/ATM0X3gUBJqWnT7Y6 BjAgvlrTI6U4DfE/LYMnI3iYcOPRkQhf+O615tDESrjGYV4iTS5hBfuY8Rgl6zrm0E amsyB3JroBQ0elTcvMXp0SCRKC1aJD+g9U7MYFnDReh3xBQlCwM+0LFC7vZgzoHvrq /GGJ9BL+3iVwqPtsfWdvUhMMqMYgWhJg1Ate9hP7wbuCFE4ibwv+KMFA9krRv+kD2n 1f7iE3NnJWXhQ== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 5/8] nvme-tcp: register host mptcp transport Date: Fri, 15 May 2026 17:46:19 +0800 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang This patch defines a new nvmf_transport_ops named nvme_mptcp_transport, which is almost the same as nvme_tcp_transport except .type and .allowed_opts. MPTCP currently does not support TLS. The four TLS-related options (NVMF_OPT_TLS, NVMF_OPT_KEYRING, NVMF_OPT_TLS_KEY, and NVMF_OPT_CONCAT) have been removed from allowed_opts. They will be added back once MPTCP TLS is supported. It is registered in nvme_tcp_init_module() and unregistered in nvme_tcp_cleanup_module(). A separate nvme_mptcp_ctrl_ops structure with .name =3D "mptcp" is defined and used for MPTCP controllers. A MODULE_ALIAS("nvme-mptcp") declaration alongside the other module metadata is added at the end of the file. v2: - use 'trtype' instead of '--mptcp' (Hannes) v3: - check mptcp protocol from opts->transport instead of passing a parameter (Hannes). v4: - check CONFIG_MPTCP. Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- drivers/nvme/host/tcp.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index f54b1eb86940..bad18d7c323e 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -3067,6 +3067,20 @@ static struct nvmf_transport_ops nvme_tcp_transport = =3D { .create_ctrl =3D nvme_tcp_create_ctrl, }; =20 +#ifdef CONFIG_MPTCP +static struct nvmf_transport_ops nvme_mptcp_transport =3D { + .name =3D "mptcp", + .module =3D THIS_MODULE, + .required_opts =3D NVMF_OPT_TRADDR, + .allowed_opts =3D NVMF_OPT_TRSVCID | NVMF_OPT_RECONNECT_DELAY | + NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO | + NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST | + NVMF_OPT_NR_WRITE_QUEUES | NVMF_OPT_NR_POLL_QUEUES | + NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE, + .create_ctrl =3D nvme_tcp_create_ctrl, +}; +#endif + static int __init nvme_tcp_init_module(void) { unsigned int wq_flags =3D WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_SYSFS; @@ -3092,6 +3106,9 @@ static int __init nvme_tcp_init_module(void) atomic_set(&nvme_tcp_cpu_queues[cpu], 0); =20 nvmf_register_transport(&nvme_tcp_transport); +#ifdef CONFIG_MPTCP + nvmf_register_transport(&nvme_mptcp_transport); +#endif return 0; } =20 @@ -3099,6 +3116,9 @@ static void __exit nvme_tcp_cleanup_module(void) { struct nvme_tcp_ctrl *ctrl; =20 +#ifdef CONFIG_MPTCP + nvmf_unregister_transport(&nvme_mptcp_transport); +#endif nvmf_unregister_transport(&nvme_tcp_transport); =20 mutex_lock(&nvme_tcp_ctrl_mutex); @@ -3116,3 +3136,4 @@ module_exit(nvme_tcp_cleanup_module); MODULE_DESCRIPTION("NVMe host TCP transport driver"); MODULE_LICENSE("GPL v2"); MODULE_ALIAS("nvme-tcp"); +MODULE_ALIAS("nvme-mptcp"); --=20 2.53.0 From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C372B3C8C65 for ; Fri, 15 May 2026 09:46:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838408; cv=none; b=HgqNe1bl0EihYjtyvkYcrm6t1/kHub6JShCL6hLj+x1G+Afo8GowtUyJNxZgakF5TEiCiB+DvftyNmShXJMKrDaDbxIcShBW2Jhz9PXnX9zGJ7wIxGtUcAipR1USjJQ4QbF1hbJiB6j6Dv4etFzQaP+yLFsuiAPUblYpCQb5bCI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838408; c=relaxed/simple; bh=QgCwzCi8B2Pv7FZ73U0LNavUlAREM/zpNh7R4wtAgNk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e+DGt7U+jzJMr5TlQUpnfSGVmPzJs0ROKN3C0CJawwA9gvInrX5N/zPgDG2QrCms46a08WsKVZaOfnN/g7lQfFr56LeMQgD+St7R1heI4DSWcQT6cGvLPkPeNja3oavPvl/L7oVIMjRKbAKurhlBmSQN7y7MXTt4bho02EiQyDw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jkTmxm6D; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jkTmxm6D" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 00617C2BCB0; Fri, 15 May 2026 09:46:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838408; bh=QgCwzCi8B2Pv7FZ73U0LNavUlAREM/zpNh7R4wtAgNk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jkTmxm6Dcr89Ty2zylo0GsBL1Cv0JlWg7eG08c6iGLVofIY17urUo2MR6DW4hjJB3 96/C40pWzLmNO+HNtc4et3XLTHuhazBggdQ/vqeajp3wWYLhKbAQeHJwWyMZdU96JU s1Rk0Ata4Axzh4wRyQ+CMhlU6CrHqJO9B8gOpp4PvXYd3T0VxUP05yxM3GtbqAa9Cl 9TVhRAEUK2SQjsqCbF7JN3GH6MmRKiyMINdQavtNfgPb3HAF2QcnO/t826U+DVhSUU TAzDZxLUwImM6Wiaynwtioue1H0BPp1Jkor6k8SGUSt+yPbnXhtMMElHxTJBBK6EYk hpJaQkzkPgfSQ== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 6/8] nvme-tcp: implement host mptcp proto Date: Fri, 15 May 2026 17:46:20 +0800 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang An MPTCP-specific version of struct nvme_tcp_proto is implemented, and it is assigned to ctrl->proto when the transport string is "mptcp". The socket option setting logic is similar to the target side, except that mptcp_sock_set_syncnt is newly defined for the host side. It sets the value on the first subflow socket of an MPTCP connection. The value is then synchronized to other newly created subflows in sync_socket_options(). A separate nvme_mptcp_ctrl_ops structure with .name =3D "mptcp" is defined and used for MPTCP controllers. Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- drivers/nvme/host/tcp.c | 34 ++++++++++++++++++++++++++++++++++ include/net/mptcp.h | 7 +++++++ net/mptcp/protocol.h | 1 + net/mptcp/sockopt.c | 21 +++++++++++++++++++++ 4 files changed, 63 insertions(+) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index bad18d7c323e..22fcfc3b5c2a 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2896,6 +2896,24 @@ static const struct nvme_ctrl_ops nvme_tcp_ctrl_ops = =3D { .get_virt_boundary =3D nvmf_get_virt_boundary, }; =20 +#ifdef CONFIG_MPTCP +static const struct nvme_ctrl_ops nvme_mptcp_ctrl_ops =3D { + .name =3D "mptcp", + .module =3D THIS_MODULE, + .flags =3D NVME_F_FABRICS | NVME_F_BLOCKING, + .reg_read32 =3D nvmf_reg_read32, + .reg_read64 =3D nvmf_reg_read64, + .reg_write32 =3D nvmf_reg_write32, + .subsystem_reset =3D nvmf_subsystem_reset, + .free_ctrl =3D nvme_tcp_free_ctrl, + .submit_async_event =3D nvme_tcp_submit_async_event, + .delete_ctrl =3D nvme_tcp_delete_ctrl, + .get_address =3D nvme_tcp_get_address, + .stop_ctrl =3D nvme_tcp_stop_ctrl, + .get_virt_boundary =3D nvmf_get_virt_boundary, +}; +#endif + static bool nvme_tcp_existing_controller(struct nvmf_ctrl_options *opts) { @@ -2923,6 +2941,18 @@ static const struct nvme_tcp_proto nvme_tcp_proto = =3D { .ops =3D &nvme_tcp_ctrl_ops, }; =20 +#ifdef CONFIG_MPTCP +static const struct nvme_tcp_proto nvme_mptcp_proto =3D { + .protocol =3D IPPROTO_MPTCP, + .set_syncnt =3D mptcp_sock_set_syncnt, + .set_nodelay =3D mptcp_sock_set_nodelay, + .no_linger =3D mptcp_sock_no_linger, + .set_priority =3D mptcp_sock_set_priority, + .set_tos =3D mptcp_sock_set_tos, + .ops =3D &nvme_mptcp_ctrl_ops, +}; +#endif + static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev, struct nvmf_ctrl_options *opts) { @@ -2989,6 +3019,10 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(str= uct device *dev, =20 if (!strcmp(ctrl->ctrl.opts->transport, "tcp")) { ctrl->proto =3D &nvme_tcp_proto; +#ifdef CONFIG_MPTCP + } else if (!strcmp(ctrl->ctrl.opts->transport, "mptcp")) { + ctrl->proto =3D &nvme_mptcp_proto; +#endif } else { ret =3D -EINVAL; goto out_free_ctrl; diff --git a/include/net/mptcp.h b/include/net/mptcp.h index 91ce7b9b639d..49031a111e69 100644 --- a/include/net/mptcp.h +++ b/include/net/mptcp.h @@ -247,6 +247,8 @@ void mptcp_sock_set_priority(struct sock *sk, u32 prior= ity); void mptcp_sock_no_linger(struct sock *sk); =20 void mptcp_sock_set_tos(struct sock *sk, int val); + +int mptcp_sock_set_syncnt(struct sock *sk, int val); #else =20 static inline void mptcp_init(void) @@ -343,6 +345,11 @@ static inline void mptcp_sock_set_priority(struct sock= *sk, u32 priority) { } static inline void mptcp_sock_no_linger(struct sock *sk) { } =20 static inline void mptcp_sock_set_tos(struct sock *sk, int val) { } + +static inline int mptcp_sock_set_syncnt(struct sock *sk, int val) +{ + return 0; +} #endif /* CONFIG_MPTCP */ =20 #if IS_ENABLED(CONFIG_MPTCP_IPV6) diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 661600f8b573..0096cabdccd2 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -336,6 +336,7 @@ struct mptcp_sock { int keepalive_idle; int keepalive_intvl; int maxseg; + int icsk_syn_retries; struct work_struct work; struct sk_buff *ooo_last_skb; struct rb_root out_of_order_queue; diff --git a/net/mptcp/sockopt.c b/net/mptcp/sockopt.c index fbcde31ef668..ee89c5d0c8de 100644 --- a/net/mptcp/sockopt.c +++ b/net/mptcp/sockopt.c @@ -1602,6 +1602,8 @@ static void sync_socket_options(struct mptcp_sock *ms= k, struct sock *ssk) priority =3D READ_ONCE(sk->sk_priority); if (priority > 0) sock_set_priority(ssk, priority); + if (msk->icsk_syn_retries > 0) + tcp_sock_set_syncnt(ssk, msk->icsk_syn_retries); } =20 void mptcp_sockopt_sync_locked(struct mptcp_sock *msk, struct sock *ssk) @@ -1761,3 +1763,22 @@ void mptcp_sock_set_tos(struct sock *sk, int val) release_sock(sk); } EXPORT_SYMBOL(mptcp_sock_set_tos); + +int mptcp_sock_set_syncnt(struct sock *sk, int val) +{ + struct mptcp_sock *msk =3D mptcp_sk(sk); + struct sock *ssk; + + if (val < 1 || val > MAX_TCP_SYNCNT) + return -EINVAL; + + lock_sock(sk); + sockopt_seq_inc(msk); + msk->icsk_syn_retries =3D val; + ssk =3D __mptcp_nmpc_sk(msk); + if (!IS_ERR(ssk)) + tcp_sock_set_syncnt(ssk, val); + release_sock(sk); + return 0; +} +EXPORT_SYMBOL(mptcp_sock_set_syncnt); --=20 2.53.0 From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6148402B83 for ; Fri, 15 May 2026 09:46:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838410; cv=none; b=aRo8kBVpGYpmSl5Jzygsi+W461Rkmjnub2cZMl8yKGoTSBmV5dHIAjnNfflcg+MMrQC7il668zV0jXwY1yHdzmFHuWAxjQ0i9KEi7/3zKuijVmDx8M4reBczh8lY/GkNoMOE3pyYawDkic20+Xs722fD0pRHMzELvDFnwXWEkDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838410; c=relaxed/simple; bh=LwZnNnls+BjVmCLfrNCXGJRG/desAp4dBblJ7nTUihI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eCHjwJRQ7DOSCmIfCrvtOYQmevauRVMA3hi/emMFJCPHGiMuXgIeMLXFrhmOyvOA3PAKSZxWUAkX+slKIqrRs8IF0uyVI9uIy64AznpztONFDsqQqSEMP2eU6CP4YeLYNFVgVh3Fokv2maei4qWbHcz8jRkM0wVJES7jBvXHEDk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=guPrNRvm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="guPrNRvm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23901C2BCB8; Fri, 15 May 2026 09:46:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838410; bh=LwZnNnls+BjVmCLfrNCXGJRG/desAp4dBblJ7nTUihI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=guPrNRvmjnw8RrwNTFBMM390DUAxWHs4zAYuGVmL1vT1j62d+sKhw5OGUUXhxUd6M Z1TMsIApwiG/HsE7bqsMDZEupeRQlzIoN35t79jHnb01wK7c3Pht4Ae2ZrKzmBD/mY oTr/2LlnB7mQiW/8RMxTNFy9YBkLjPt4ho4YKoGX/DRQZwAbGiMXc5KxNsn9C0F58E PczH6LqCIW3H2A0rcThHPDZb71zIesrBXoujH+Fw1jmjWqyB7YnFE6OXUtbdUO283d Av8XquDm7aY2dx58c8k4HVJF/IQmZ7jGbHL+htN9pRcHAA9mk1QLjgP8JLAtroOOr1 rvR+U0bywPYOw== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 7/8] selftests: mptcp: add nvme over mptcp test Date: Fri, 15 May 2026 17:46:21 +0800 Message-ID: <54e210e67f760869bd3f9a784d9b491cff1c4254.1778837549.git.tanggeliang@kylinos.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang A test case for NVMe over MPTCP has been implemented. It verifies the proper functionality of nvme list, discover, connect, and disconnect commands. Additionally, read/write performance has been evaluated using fio. This script accepts two positional parameters: trtype - Transport type (mptcp|tcp). Default: mptcp path - Number of multipath (1-4). Default: 1 This test simulates four NICs on both target and host sides, each limited to 125MB/s. It shows that 'NVMe over MPTCP' delivered bandwidth up to four times that of standard TCP with a single NVMe multipath configuration: # ./mptcp_nvme.sh tcp READ: bw=3D112MiB/s (118MB/s), 112MiB/s-112MiB/s (118MB/s-118MB/s), io=3D1123MiB (1177MB), run=3D10018-10018msec WRITE: bw=3D112MiB/s (117MB/s), 112MiB/s-112MiB/s (117MB/s-117MB/s), io=3D1118MiB (1173MB), run=3D10018-10018msec # ./mptcp_nvme.sh mptcp READ: bw=3D427MiB/s (448MB/s), 427MiB/s-427MiB/s (448MB/s-448MB/s), io=3D4286MiB (4494MB), run=3D10039-10039msec WRITE: bw=3D387MiB/s (406MB/s), 387MiB/s-387MiB/s (406MB/s-406MB/s), io=3D3885MiB (4073MB), run=3D10043-10043msec It reflects that MPTCP has the same multi-interface bandwidth aggregation capability as NVMe multipath. Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- tools/testing/selftests/net/mptcp/Makefile | 1 + tools/testing/selftests/net/mptcp/config | 7 + .../testing/selftests/net/mptcp/mptcp_lib.sh | 12 + .../testing/selftests/net/mptcp/mptcp_nvme.sh | 311 ++++++++++++++++++ 4 files changed, 331 insertions(+) create mode 100755 tools/testing/selftests/net/mptcp/mptcp_nvme.sh diff --git a/tools/testing/selftests/net/mptcp/Makefile b/tools/testing/sel= ftests/net/mptcp/Makefile index 22ba0da2adb8..7b308447a58b 100644 --- a/tools/testing/selftests/net/mptcp/Makefile +++ b/tools/testing/selftests/net/mptcp/Makefile @@ -13,6 +13,7 @@ TEST_PROGS :=3D \ mptcp_connect_sendfile.sh \ mptcp_connect_splice.sh \ mptcp_join.sh \ + mptcp_nvme.sh \ mptcp_sockopt.sh \ pm_netlink.sh \ simult_flows.sh \ diff --git a/tools/testing/selftests/net/mptcp/config b/tools/testing/selft= ests/net/mptcp/config index 59051ee2a986..0eee348eff8b 100644 --- a/tools/testing/selftests/net/mptcp/config +++ b/tools/testing/selftests/net/mptcp/config @@ -34,3 +34,10 @@ CONFIG_NFT_SOCKET=3Dm CONFIG_NFT_TPROXY=3Dm CONFIG_SYN_COOKIES=3Dy CONFIG_VETH=3Dy +CONFIG_CONFIGFS_FS=3Dy +CONFIG_NVME_CORE=3Dy +CONFIG_NVME_FABRICS=3Dy +CONFIG_NVME_TCP=3Dy +CONFIG_NVME_TARGET=3Dy +CONFIG_NVME_TARGET_TCP=3Dy +CONFIG_NVME_MULTIPATH=3Dy diff --git a/tools/testing/selftests/net/mptcp/mptcp_lib.sh b/tools/testing= /selftests/net/mptcp/mptcp_lib.sh index 989a5975dcea..3d04baa698f5 100644 --- a/tools/testing/selftests/net/mptcp/mptcp_lib.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_lib.sh @@ -530,6 +530,18 @@ mptcp_lib_check_tools() { exit ${KSFT_SKIP} fi ;; + "nvme") + if ! nvme --version &> /dev/null; then + mptcp_lib_pr_skip "Could not run test without nvme tool" + exit ${KSFT_SKIP} + fi + ;; + "fio") + if ! fio -h &> /dev/null; then + mptcp_lib_pr_skip "Could not run test without fio tool" + exit ${KSFT_SKIP} + fi + ;; *) mptcp_lib_pr_fail "Internal error: unsupported tool: ${tool}" exit ${KSFT_FAIL} diff --git a/tools/testing/selftests/net/mptcp/mptcp_nvme.sh b/tools/testin= g/selftests/net/mptcp/mptcp_nvme.sh new file mode 100755 index 000000000000..2dc948cbcb25 --- /dev/null +++ b/tools/testing/selftests/net/mptcp/mptcp_nvme.sh @@ -0,0 +1,311 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(dirname "$0")/mptcp_lib.sh" + +ret=3D0 +trtype=3D"${1:-mptcp}" +path=3D"${2:-1}" +nqn=3D"nqn.2014-08.org.nvmexpress.${trtype}dev.$$.${RANDOM}" +ns=3D1 +port=3D$((RANDOM % 10000 + 20000)) +trsvcid=3D$((RANDOM % 64512 + 1024)) +ns1=3D"" +ns2=3D"" +temp_file=3D"" +loop_dev=3D"" + +usage() +{ + cat << EOF + +Usage: + + $(basename "$0") [trtype] [path] + + trtype Transport type (mptcp|tcp) - default: mptcp + path Number of multipath (1-4) - default: 4 + +EOF +exit 0 +} + +validate_params() +{ + if [[ ! "${trtype}" =3D~ ^(mptcp|tcp)$ ]]; then + echo "Error: Invalid trtype '${trtype}'." + usage + fi + + if [[ ! "${path}" =3D~ ^[0-9]+$ ]] || [ "${path}" -lt 1 ]; then + echo "Error: Invalid path count '${path}'." + usage + fi + + if [ "${path}" -gt 4 ]; then + echo "Warning: path count ${path} > 4, limiting to 4" + path=3D4 + fi +} + +ns1_cleanup() +{ + pushd /sys/kernel/config/nvmet || exit 1 + + for i in $(seq 1 ${path}); do + local portdir=3D$((port + i)) + + rm -rf "ports/${portdir}/subsystems/${nqn}" + rmdir "ports/${portdir}" + done + + echo 0 > "subsystems/${nqn}/namespaces/${ns}/enable" + echo -n 0 > "subsystems/${nqn}/namespaces/${ns}/device_path" + rmdir "subsystems/${nqn}/namespaces/${ns}" + rmdir "subsystems/${nqn}" + + popd || exit 1 +} + +ns2_cleanup() +{ + nvme disconnect -n "${nqn}" || true +} + +# This function is used in the cleanup trap +#shellcheck disable=3DSC2317,SC2329 +cleanup() +{ + ip netns exec "$ns2" bash <<- EOF + $(declare -f ns2_cleanup) + ns2_cleanup + EOF + + sleep 1 + + ip netns exec "$ns1" unshare -m bash <<- EOF + mount -t configfs none /sys/kernel/config + $(declare -f ns1_cleanup) + ns1_cleanup + EOF + + if [ -n "${loop_dev}" ] && [ -b "${loop_dev}" ]; then + losetup -d "${loop_dev}" 2>/dev/null || true + fi + rm -rf "${temp_file}" + + mptcp_lib_ns_exit "$ns1" "$ns2" + + kill "$monitor_pid_ns1" 2>/dev/null + wait "$monitor_pid_ns1" 2>/dev/null + + kill "$monitor_pid_ns2" 2>/dev/null + wait "$monitor_pid_ns2" 2>/dev/null + + unset -v trtype path nqn ns port trsvcid +} + +init() +{ + mptcp_lib_ns_init ns1 ns2 + + # ns1 ns2 + # 10.1.1.1 10.1.1.2 + # 10.1.2.1 10.1.2.2 + # 10.1.3.1 10.1.3.2 + # 10.1.4.1 10.1.4.2 + for i in {1..4}; do + ip link add ns1eth"$i" netns "$ns1" type veth peer \ + name ns2eth"$i" netns "$ns2" + ip -net "$ns1" addr add 10.1."$i".1/24 dev ns1eth"$i" + ip -net "$ns1" addr add dead:beef:"$i"::1/64 \ + dev ns1eth"$i" nodad + ip -net "$ns1" link set ns1eth"$i" up + ip -net "$ns2" addr add 10.1."$i".2/24 dev ns2eth"$i" + ip -net "$ns2" addr add dead:beef:"$i"::2/64 \ + dev ns2eth"$i" nodad + ip -net "$ns2" link set ns2eth"$i" up + ip -net "$ns2" route add default via 10.1."$i".1 \ + dev ns2eth"$i" metric 10"$i" + ip -net "$ns2" route add default via dead:beef:"$i"::1 \ + dev ns2eth"$i" metric 10"$i" + + # Add tc qdisc to both namespaces for bandwidth limiting + tc -n "$ns1" qdisc add dev ns1eth"$i" root netem rate 1000mbit + tc -n "$ns2" qdisc add dev ns2eth"$i" root netem rate 1000mbit + + tc -n "$ns1" qdisc show dev ns1eth"$i" + tc -n "$ns2" qdisc show dev ns2eth"$i" + done + + mptcp_lib_pm_nl_set_limits "${ns1}" 8 8 + + mptcp_lib_pm_nl_add_endpoint "$ns1" 10.1.1.1 flags signal + mptcp_lib_pm_nl_add_endpoint "$ns1" 10.1.2.1 flags signal + mptcp_lib_pm_nl_add_endpoint "$ns1" 10.1.3.1 flags signal + mptcp_lib_pm_nl_add_endpoint "$ns1" 10.1.4.1 flags signal + + mptcp_lib_pm_nl_set_limits "${ns2}" 8 8 + + mptcp_lib_pm_nl_add_endpoint "$ns2" 10.1.1.2 flags subflow + mptcp_lib_pm_nl_add_endpoint "$ns2" 10.1.2.2 flags subflow + mptcp_lib_pm_nl_add_endpoint "$ns2" 10.1.3.2 flags subflow + mptcp_lib_pm_nl_add_endpoint "$ns2" 10.1.4.2 flags subflow + + ip -n "${ns1}" mptcp monitor & + monitor_pid_ns1=3D$! + ip -n "${ns2}" mptcp monitor & + monitor_pid_ns2=3D$! +} + +run_target() +{ + cd /sys/kernel/config/nvmet/subsystems || exit + mkdir -p "${nqn}" + cd "${nqn}" || exit + echo 1 > attr_allow_any_host + mkdir -p namespaces/"${ns}" + echo "${loop_dev}" > namespaces/"${ns}"/device_path + echo 1 > namespaces/"${ns}"/enable + + # Create 4 ports, each on a different IP address + for i in $(seq 1 ${path}); do + local portdir=3D$((port + i)) + + cd /sys/kernel/config/nvmet/ports || exit + mkdir -p "${portdir}" + cd "${portdir}" || exit 1 + echo "${trtype}" > addr_trtype + echo ipv4 > addr_adrfam + if [ "${path}" -eq 1 ]; then + echo "0.0.0.0" > addr_traddr + else + echo "10.1.${i}.1" > addr_traddr + fi + echo "${trsvcid}" > addr_trsvcid + + mkdir -p subsystems + ln -sf "../../subsystems/${nqn}" "subsystems/${nqn}" + cd - >/dev/null + done +} + +run_host() +{ + local traddr=3D10.1.1.1 + local devname + + echo "nvme discover -a ${traddr}" + nvme discover -t "${trtype}" -a "${traddr}" -s "${trsvcid}" + if [ $? -ne 0 ]; then + return 1 + fi + + for i in $(seq 1 ${path}); do + echo "Connecting to 10.1.${i}.1:${trsvcid}" + + nvme connect -t "${trtype}" -a "10.1.${i}.1" \ + -s "${trsvcid}" -n "${nqn}" + if [ $? -ne 0 ]; then + echo "Failed to connect to 10.1.${i}.1" + return 1 + fi + done + + sleep 1 + + # Scan all NVMe block devices + for dev in /dev/nvme*n1 /dev/nvme*cn1; do + if [ -b "$dev" ] 2>/dev/null; then + # Check if this device's controller matches our NQN + if nvme id-ctrl "$dev" 2>/dev/null | grep -q "${nqn}"; then + devname=3D$(basename "$dev") + break 2 + fi + fi + done 2>/dev/null + + if [ -z "$devname" ]; then + echo "No block device found for NQN ${nqn}" >&2 + return 1 + fi + + sleep 1 + + echo "nvme list" + nvme list + + sleep 1 + + echo "fio randread /dev/${devname}" + fio --name=3Dglobal --direct=3D1 --norandommap --randrepeat=3D0 \ + --ioengine=3Dlibaio --thread=3D1 --blocksize=3D128k --runtime=3D10 \ + --time_based --rw=3Drandread --numjobs=3D4 --iodepth=3D256 \ + --group_reporting --size=3D100% --name=3Dlibaio_4_256_4k_randread \ + --filename=3D"/dev/${devname}" + if [ $? -ne 0 ]; then + return 1 + fi + + sleep 1 + + echo "fio randwrite /dev/${devname}" + fio --name=3Dglobal --direct=3D1 --norandommap --randrepeat=3D0 \ + --ioengine=3Dlibaio --thread=3D1 --blocksize=3D128k --runtime=3D10 \ + --time_based --rw=3Drandwrite --numjobs=3D4 --iodepth=3D256 \ + --group_reporting --size=3D100% --name=3Dlibaio_4_256_4k_randwrite \ + --filename=3D"/dev/${devname}" + if [ $? -ne 0 ]; then + return 1 + fi + + nvme flush "/dev/${devname}" +} + +trap cleanup EXIT + +mptcp_lib_check_tools nvme fio +validate_params + +temp_file=3D$(mktemp /tmp/nvme_test.XXXXXX.raw) +if [ $? -ne 0 ]; then + echo "Failed to create temp file" + exit 1 +fi + +dd if=3D/dev/zero of=3D"${temp_file}" bs=3D1M count=3D0 seek=3D512 +loop_dev=3D$(losetup -f --show "${temp_file}") + +init + +run_test() +{ + export trtype path nqn ns port trsvcid + export loop_dev temp_file + + if ! ip netns exec "$ns1" unshare -m bash <<- EOF + mount -t configfs none /sys/kernel/config + $(declare -f run_target) + run_target + exit \$? + EOF + then + ret=3D"${KSFT_FAIL}" + fi + + if ! ip netns exec "$ns2" bash <<- EOF + $(declare -f run_host) + run_host + exit \$? + EOF + then + ret=3D"${KSFT_FAIL}" + fi + + sleep 1 +} + +run_test "$@" + +mptcp_lib_result_print_all_tap +exit "$ret" --=20 2.53.0 From nobody Mon May 25 18:05:07 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C1D1168BD for ; Fri, 15 May 2026 09:46:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838413; cv=none; b=htvl5fM0n11GT4x4hjqKRAR+l37qaTo8cLzn1p7UtMh6dh/WGtIvAeQIxlDG33Wt2BgV5y0yNx5tK+hEX2fyfFWM/bH8waLPd21KFT0/fJSp4fVBjt6HKgijALT3KRdodZ8PHBMxQdhleOGdpE5xZy42MjyGMEli4TnaWdKUkyE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778838413; c=relaxed/simple; bh=5tlh6EF+C7J8EL+q9G5aYaaVediSwaQMs541K+xQ9WE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Il0+pFXsXmoxBo7JqGO/F7E9OyybJih9HHLPbtqwiPXSGljZDjdyKWTvJZvhbngPniOpzFEFnycuV31Rl4peNkcYWBOkXrtkBlhl34F36zAPFt6mOAwLYk9mXe0Qnd1137ijHGbBW/ItSwPmfBaBItaz5YtBCDCmDiqlP3THfco= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Sw96XSmM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Sw96XSmM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3DE80C2BCB3; Fri, 15 May 2026 09:46:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778838412; bh=5tlh6EF+C7J8EL+q9G5aYaaVediSwaQMs541K+xQ9WE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Sw96XSmMExgHGlHvgNKn3cuMiExkgHYavn9+Okwg+M/DMkrpui+HN0U6iD8YPbaET ynx+b3JNNDbvF0PoRbN/pYJBuTJZ5atOr+HX99ln3T/GuRNDckgLnAsc3vhL3Z3vhj kleUX8wSv3Kz3Fwmep5qx7J/IQr4mHfcUoAcWAV345xGG7ZQsJgpbt7wNlsn3s1dxu HELQfOMRZewHLvylFbQN1cvAx4D6Ac5F81n0HTh6TpjcqGgSLVitV6IB0wyJCbi6qo SDIi9ECUVoqsDeS08hRjXxnWmDjKlIOdt5bgPngacSmM0Jew1oSsh+w61nyO0tGJTl YnqJJAydLWtnw== From: Geliang Tang To: mptcp@lists.linux.dev Cc: Geliang Tang , Hannes Reinecke , zhenwei pi , Hui Zhu , Gang Yan Subject: [RFC mptcp-next v9 8/8] selftests: mptcp: nvme: add iopolicy tests Date: Fri, 15 May 2026 17:46:22 +0800 Message-ID: <80d0ca5057c03c24e5320c8194c1a94d48709e23.1778837549.git.tanggeliang@kylinos.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: mptcp@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Geliang Tang Add NVMe iopolicy testing to mptcp_nvme.sh, with the default set to "numa". It can be set to "round-robin" or "queue-depth". Test results with 4 NVMe multipath paths and round-robin iopolicy show that TCP and MPTCP achieve similar bandwidth: # ./mptcp_nvme.sh tcp 4 round-robin READ: bw=3D455MiB/s (478MB/s), 455MiB/s-455MiB/s (478MB/s-478MB/s), io=3D4665MiB (4891MB), run=3D10242-10242msec WRITE: bw=3D455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=3D4633MiB (4858MB), run=3D10184-10184msec # ./mptcp_nvme.sh mptcp 4 round-robin READ: bw=3D445MiB/s (466MB/s), 445MiB/s-445MiB/s (466MB/s-466MB/s), io=3D4575MiB (4797MB), run=3D10287-10287msec WRITE: bw=3D445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s), io=3D4572MiB (4794MB), run=3D10267-10267msec A "loss" argument is added to simulate network packet loss. When loss=3D1, = each veth interface is configured with "delay 5ms loss 0.5%" using tc qdisc. Und= er this scenario, TCP performance is reduced by multiples compared to MPTCP: # ./mptcp_nvme.sh tcp 4 round-robin 1 READ: bw=3D144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=3D1909MiB (2001MB), run=3D13231-13231msec WRITE: bw=3D100.0MiB/s (105MB/s), 100.0MiB/s-100.0MiB/s (105MB/s-105MB/s), io=3D1397MiB (1465MB), run=3D13980-13980msec # ./mptcp_nvme.sh mptcp 4 round-robin 1 READ: bw=3D428MiB/s (449MB/s), 428MiB/s-428MiB/s (449MB/s-449MB/s), io=3D4524MiB (4743MB), run=3D10564-10564msec WRITE: bw=3D431MiB/s (452MB/s), 431MiB/s-431MiB/s (452MB/s-452MB/s), io=3D4513MiB (4732MB), run=3D10481-10481msec These results demonstrate that MPTCP has better resilience against packet loss compared to TCP, as it can leverage multiple subflows to mitigate network degradation. Cc: Hannes Reinecke Co-developed-by: zhenwei pi Signed-off-by: zhenwei pi Co-developed-by: Hui Zhu Signed-off-by: Hui Zhu Co-developed-by: Gang Yan Signed-off-by: Gang Yan Signed-off-by: Geliang Tang --- .../testing/selftests/net/mptcp/mptcp_nvme.sh | 65 ++++++++++++++++++- 1 file changed, 62 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/net/mptcp/mptcp_nvme.sh b/tools/testin= g/selftests/net/mptcp/mptcp_nvme.sh index 2dc948cbcb25..1d1bf3f0780b 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_nvme.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_nvme.sh @@ -6,6 +6,8 @@ ret=3D0 trtype=3D"${1:-mptcp}" path=3D"${2:-1}" +iopolicy=3D${3:-"numa"} # round-robin, queue-depth +loss=3D${4:-0} nqn=3D"nqn.2014-08.org.nvmexpress.${trtype}dev.$$.${RANDOM}" ns=3D1 port=3D$((RANDOM % 10000 + 20000)) @@ -21,10 +23,12 @@ usage() =20 Usage: =20 - $(basename "$0") [trtype] [path] + $(basename "$0") [trtype] [path] [iopolicy] [loss] =20 trtype Transport type (mptcp|tcp) - default: mptcp path Number of multipath (1-4) - default: 4 + iopolicy I/O policy (numa|round-robin|queue-depth) - default: numa + loss Enable packet loss (0|1) - default: 0 =20 EOF exit 0 @@ -46,6 +50,16 @@ validate_params() echo "Warning: path count ${path} > 4, limiting to 4" path=3D4 fi + + if [[ ! "${iopolicy}" =3D~ ^(numa|round-robin|queue-depth)$ ]]; then + echo "Error: Invalid iopolicy '${iopolicy}'." + usage + fi + + if [[ ! "${loss}" =3D~ ^[01]$ ]]; then + echo "Error: Invalid loss value '${loss}'. Must be 0 or 1" + usage + fi } =20 ns1_cleanup() @@ -103,6 +117,7 @@ cleanup() wait "$monitor_pid_ns2" 2>/dev/null =20 unset -v trtype path nqn ns port trsvcid + unset iopolicy loss } =20 init() @@ -131,8 +146,10 @@ init() dev ns2eth"$i" metric 10"$i" =20 # Add tc qdisc to both namespaces for bandwidth limiting - tc -n "$ns1" qdisc add dev ns1eth"$i" root netem rate 1000mbit - tc -n "$ns2" qdisc add dev ns2eth"$i" root netem rate 1000mbit + tc -n "$ns1" qdisc add dev ns1eth"$i" root netem rate 1000mbit \ + $([ ${loss} -eq 1 ] && echo "delay 5ms loss 0.5%") + tc -n "$ns2" qdisc add dev ns2eth"$i" root netem rate 1000mbit \ + $([ ${loss} -eq 1 ] && echo "delay 5ms loss 0.5%") =20 tc -n "$ns1" qdisc show dev ns1eth"$i" tc -n "$ns2" qdisc show dev ns2eth"$i" @@ -190,6 +207,41 @@ run_target() done } =20 +set_io_policy() +{ + local nqn=3D"$1" + local iopolicy=3D"$2" + local subname + local policy + local current + + subname=3D$(nvme list-subsys 2>/dev/null | grep -B 5 "${nqn}" | + grep -o 'nvme-subsys[0-9]*' | head -1) + if [ -z "$subname" ]; then + return 1 + fi + + policy=3D"/sys/class/nvme-subsystem/${subname}/iopolicy" + if [ ! -w "$policy" ]; then + return 1 + fi + + if ! echo "${iopolicy}" > "$policy" 2>/dev/null; then + return 1 + fi + + current=3D$(cat "$policy" 2>/dev/null) + if [ -z "$current" ]; then + return 1 + fi + + if [[ "$current" !=3D *"${iopolicy}"* ]]; then + return 1 + fi + + return 0 +} + run_host() { local traddr=3D10.1.1.1 @@ -235,6 +287,11 @@ run_host() echo "nvme list" nvme list =20 + if ! set_io_policy "${nqn}" "${iopolicy}"; then + echo "Failed to set I/O policy to ${iopolicy}" + return 1 + fi + sleep 1 =20 echo "fio randread /dev/${devname}" @@ -282,6 +339,7 @@ run_test() { export trtype path nqn ns port trsvcid export loop_dev temp_file + export iopolicy loss =20 if ! ip netns exec "$ns1" unshare -m bash <<- EOF mount -t configfs none /sys/kernel/config @@ -294,6 +352,7 @@ run_test() fi =20 if ! ip netns exec "$ns2" bash <<- EOF + $(declare -f set_io_policy) $(declare -f run_host) run_host exit \$? --=20 2.53.0