From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B61F1C5D79 for ; Thu, 20 Feb 2025 02:09:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017365; cv=none; b=NyCI9xwF0lrxSwZGDGDKrQS0ijVtCvQ5q0SFtZCd6Z/m2C6fSW+6vNdNhXa/L52xHZp+FKLs5dAzNQldgguvpfcCzIlW3kSFNMMN6rP0pwJuVPJJ/6KLxdR5c+ucsJeuam0e8fDsvUDTq6uUVqBKVa6Xdl5tNYaUIQ+tGW3QqHo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017365; c=relaxed/simple; bh=4doCwut28ro//QQVzhoD6EMQLMF4i01K6NWMTT4x+N0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=m3Q0iv1QF4sbvHj/PuDroFWNMz9CwNZ4re8EWwN+7qTv5+8bmkYdkUsdBY0U4Bao/q/eUkPPwkh4VSnyUEJSh6bQ0Yzvhw5mWV/3T4Wo+jcls4QqUpzI6MMgoQwesvaHoLNXSHIYiZgkIOXq2okmlPd19Vw7WanfKcdU1uhuNIA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=o0E9fBUg; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="o0E9fBUg" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-220d9d98ea6so12466015ad.3 for ; Wed, 19 Feb 2025 18:09:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017362; x=1740622162; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Z/FNs3iAWmeTptPkn9hf3A4cOseGwFtFcVodnRfhY0M=; b=o0E9fBUgNc0x/SxhuSgkNWq8yMP1OS9MYfDhPqXOA1L9XrHuDoxCcAVO9KTtPCeoQa hfS/W5llRDRBKwlS2g85RyWhNmtmDdQj2WXwyWuesrDqm5X3SGxNuI42+XwdKxnuRDRC U7eyyI+4siJsAC6m8ZbsbkxeEPCw7ywGiGXj2EgCay13o58X2jkh9uf6ZyYttaTRffMj Ykn09w6Tt7VTXIX+xQfLsxDBON58uYfp5l1MVy9tqyADkbdRc5zFPUn5lvYeW8ACKZKF sc7rkxC410Tjj+V3v9iHZyc5Bgs3y5sTU5cde7NQOYGqGO81tF5UGfVdc5NiEkokVllq dG3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017362; x=1740622162; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Z/FNs3iAWmeTptPkn9hf3A4cOseGwFtFcVodnRfhY0M=; b=QwkQ/Q3wc9u5jtI/W5BrDTWHerBPKWanp8ihFjXMD2dInpOYTpKYPdNPoq6eg3k4QU j9kzFGxT5BNKibzv17izm1J4/UWVd7jibN0tUlc6o0Yp40QzqBAjSrcjZAHH84lDvJz5 Ff2EKD7NkpWy/JwMVUmAGDObNmcZeY2s0KabITgQp+nA5qzu3r/viVfAJ45pAW2KnOJr MEpXpGcieNLofCFNBiqTnkUgMgepy3ZVxgYlXmJF5uhArPipbtR0bJvri8r8bweUEaXf eN6hFsL4aAAKmTbF+B7acc+NwEn/2qB8g0C8spg8I3onN+UTcAI2uzDrDtyhpsU7mqkn B5EA== X-Forwarded-Encrypted: i=1; AJvYcCXsP5P8gw6SZcqARORUVUW/RVAwenqMld/9UD5e6UVFAQQBvhpU7nLq5qn/aqyJ68BKsHogTu9xm219BGM=@vger.kernel.org X-Gm-Message-State: AOJu0YwqWAzyiKUs2fEUjBxlKRRx2K88ONlfrGkpg59VI7giPIcPmQi1 pGqRKCDzCo1pWVZ4iAhieiPu9T4dg0J+yO06jQWHaUvfW0yaTn0rKFuqYAahPQDle0nqQJ5feR/ uzPb09+FqwBJ9HY9dYeCBlw== X-Google-Smtp-Source: AGHT+IHnp77Jh1mBSUaA2ZEy8wL/Nxt64Y/xFYB3vMEe/E5gGG12hJqt04zktd5AF5D8ZhowgOsYCQ25S19S9voI+Q== X-Received: from plry23.prod.google.com ([2002:a17:902:b497:b0:212:4557:e89b]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ecc6:b0:21f:617a:f1b2 with SMTP id d9443c01a7336-221040bf7e5mr296400685ad.46.1740017362565; Wed, 19 Feb 2025 18:09:22 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:06 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-2-almasrymina@google.com> Subject: [PATCH net-next v4 1/9] net: add get_netmem/put_netmem support From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently net_iovs support only pp ref counts, and do not support a page ref equivalent. This is fine for the RX path as net_iovs are used exclusively with the pp and only pp refcounting is needed there. The TX path however does not use pp ref counts, thus, support for get_page/put_page equivalent is needed for netmem. Support get_netmem/put_netmem. Check the type of the netmem before passing it to page or net_iov specific code to obtain a page ref equivalent. For dmabuf net_iovs, we obtain a ref on the underlying binding. This ensures the entire binding doesn't disappear until all the net_iovs have been put_netmem'ed. We do not need to track the refcount of individual dmabuf net_iovs as we don't allocate/free them from a pool similar to what the buddy allocator does for pages. This code is written to be extensible by other net_iov implementers. get_netmem/put_netmem will check the type of the netmem and route it to the correct helper: pages -> [get|put]_page() dmabuf net_iovs -> net_devmem_[get|put]_net_iov() new net_iovs -> new helpers Signed-off-by: Mina Almasry Acked-by: Stanislav Fomichev --- v2: - Add comment on top of refcount_t ref explaining the usage in the XT path. - Fix missing definition of net_devmem_dmabuf_binding_put in this patch. --- include/linux/skbuff_ref.h | 4 ++-- include/net/netmem.h | 3 +++ net/core/devmem.c | 10 ++++++++++ net/core/devmem.h | 20 ++++++++++++++++++++ net/core/skbuff.c | 30 ++++++++++++++++++++++++++++++ 5 files changed, 65 insertions(+), 2 deletions(-) diff --git a/include/linux/skbuff_ref.h b/include/linux/skbuff_ref.h index 0f3c58007488..9e49372ef1a0 100644 --- a/include/linux/skbuff_ref.h +++ b/include/linux/skbuff_ref.h @@ -17,7 +17,7 @@ */ static inline void __skb_frag_ref(skb_frag_t *frag) { - get_page(skb_frag_page(frag)); + get_netmem(skb_frag_netmem(frag)); } =20 /** @@ -40,7 +40,7 @@ static inline void skb_page_unref(netmem_ref netmem, bool= recycle) if (recycle && napi_pp_put_page(netmem)) return; #endif - put_page(netmem_to_page(netmem)); + put_netmem(netmem); } =20 /** diff --git a/include/net/netmem.h b/include/net/netmem.h index c61d5b21e7b4..a2148ffb203d 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -264,4 +264,7 @@ static inline unsigned long netmem_get_dma_addr(netmem_= ref netmem) return __netmem_clear_lsb(netmem)->dma_addr; } =20 +void get_netmem(netmem_ref netmem); +void put_netmem(netmem_ref netmem); + #endif /* _NET_NETMEM_H */ diff --git a/net/core/devmem.c b/net/core/devmem.c index 7c6e0b5b6acb..b1aafc66ebb7 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -325,6 +325,16 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigne= d int dmabuf_fd, return ERR_PTR(err); } =20 +void net_devmem_get_net_iov(struct net_iov *niov) +{ + net_devmem_dmabuf_binding_get(net_devmem_iov_binding(niov)); +} + +void net_devmem_put_net_iov(struct net_iov *niov) +{ + net_devmem_dmabuf_binding_put(net_devmem_iov_binding(niov)); +} + /*** "Dmabuf devmem memory provider" ***/ =20 int mp_dmabuf_devmem_init(struct page_pool *pool) diff --git a/net/core/devmem.h b/net/core/devmem.h index 7fc158d52729..946f2e015746 100644 --- a/net/core/devmem.h +++ b/net/core/devmem.h @@ -29,6 +29,10 @@ struct net_devmem_dmabuf_binding { * The binding undos itself and unmaps the underlying dmabuf once all * those refs are dropped and the binding is no longer desired or in * use. + * + * net_devmem_get_net_iov() on dmabuf net_iovs will increment this + * reference, making sure that the binding remains alive until all the + * net_iovs are no longer used. */ refcount_t ref; =20 @@ -111,6 +115,9 @@ net_devmem_dmabuf_binding_put(struct net_devmem_dmabuf_= binding *binding) __net_devmem_dmabuf_binding_free(binding); } =20 +void net_devmem_get_net_iov(struct net_iov *niov); +void net_devmem_put_net_iov(struct net_iov *niov); + struct net_iov * net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding); void net_devmem_free_dmabuf(struct net_iov *ppiov); @@ -120,6 +127,19 @@ bool net_is_devmem_iov(struct net_iov *niov); #else struct net_devmem_dmabuf_binding; =20 +static inline void +net_devmem_dmabuf_binding_put(struct net_devmem_dmabuf_binding *binding) +{ +} + +static inline void net_devmem_get_net_iov(struct net_iov *niov) +{ +} + +static inline void net_devmem_put_net_iov(struct net_iov *niov) +{ +} + static inline void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *binding) { diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a441613a1e6c..815245d5c36b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -88,6 +88,7 @@ #include =20 #include "dev.h" +#include "devmem.h" #include "netmem_priv.h" #include "sock_destructor.h" =20 @@ -7290,3 +7291,32 @@ bool csum_and_copy_from_iter_full(void *addr, size_t= bytes, return false; } EXPORT_SYMBOL(csum_and_copy_from_iter_full); + +void get_netmem(netmem_ref netmem) +{ + if (netmem_is_net_iov(netmem)) { + /* Assume any net_iov is devmem and route it to + * net_devmem_get_net_iov. As new net_iov types are added they + * need to be checked here. + */ + net_devmem_get_net_iov(netmem_to_net_iov(netmem)); + return; + } + get_page(netmem_to_page(netmem)); +} +EXPORT_SYMBOL(get_netmem); + +void put_netmem(netmem_ref netmem) +{ + if (netmem_is_net_iov(netmem)) { + /* Assume any net_iov is devmem and route it to + * net_devmem_put_net_iov. As new net_iov types are added they + * need to be checked here. + */ + net_devmem_put_net_iov(netmem_to_net_iov(netmem)); + return; + } + + put_page(netmem_to_page(netmem)); +} +EXPORT_SYMBOL(put_netmem); --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F00751D5145 for ; Thu, 20 Feb 2025 02:09:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017366; cv=none; b=Sw83qwn3t5edK60CIVf9sdQCyzWFLIg8USRcbArMgnXX3ii2ZtrXE3JUyXJAmJwvQyO6flCKNEwnywsKpZGhxHdYZaGJXVBbZZz7XSX+nBub8yh4LkGrrMuNPBrkIpjvrLCtacVh7YLSD0NBb6eYVJAfDWD/LlYEkRS+DHytrBY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017366; c=relaxed/simple; bh=wv/Nl/0TTuCuSzX9uge87xPhddp3iWORdXEDTuKUoaE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=O8LPoezwYA25xl6cz+Q0ZUKPMVP+blNrmJJJGT2xgBrB3vm4qaa7I4mgYpngS63il3hQ2uX63dusdwfcnLl3SSX0IoeVY6NDOUhYKzkwPBI8pUNzMX5dNbH6P5dxBJ0QyQ5x9K0FszEeIk5MWK09PyCh3542ML4p8btrpqH7D5M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hEXq0IyW; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hEXq0IyW" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-220e04e67e2so10982735ad.2 for ; Wed, 19 Feb 2025 18:09:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017364; x=1740622164; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Q07ZPpGlxJD57O2WL3mxnzOyUUhqTVa6xALP0zRrY7w=; b=hEXq0IyWvIl1FjfDQaTIw1gJwYUcxfI/MMl3tBBB21XEwyYJmCA5uSz8+HT7kpLYsO csu6jA2j6phUM2xL1sTZxlJgdi/u6v+ObLCT2M7VuKxJxT7QO7gqv7p0n9hL68dH5AAs F9ErIdb0VlfHh3PmEad3+uyqC/kfowajdZff4kczjoB+ZxpOdlK16LjosllQCeeEo1P8 N2JeTOtn38QCsw6IeMSg73hN4vYZt2fKZtC+mrkUzoDj1pCcedzkJ6NazTtC/Xo9QvDq 5Pg/HYqgLlEq6pX8i1WKYCVdCNlk81xlV8n+eYaChLQlRQ3KndmDxkWREzxBJmW3/gCe beLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017364; x=1740622164; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q07ZPpGlxJD57O2WL3mxnzOyUUhqTVa6xALP0zRrY7w=; b=pGQDAdWrbjVzodhHBHVTd2O5KLNN/zK6pqrQhRa7C2yA9OoSYQCwlDZ+wvKjBYrzEy B/EPcXdQiMCF1dHnV+ak5T/0lJtZclVcg4qMdXbwwx2ysJaiFbSTnbHqKZA7/KTXriGG MoxMPaiIFP6ugisJQbd5r9qpEhKzXDbadTaBgeP5Lwzl+hHhq8yTucezYqkL5O2LemSH Zz2mUkKzQC8DHTJRZ5IbgqgXT7G2gIi1F68FDZG8TZL8nQvItE9DZyRGQDx5rIStXefd yw/Xp4FQ08MD6QZOPyn3Z9uetE9LnYW8/739A6kmJkr6kKfbCfzx+BHk6CXaggDTrA/f 0QDg== X-Forwarded-Encrypted: i=1; AJvYcCUglPJoz6QKjGjlzApsHFnLBjf1t4MV+q+6DrdYodoJvEeRasMoVRzNA/jPmWYdNhfXcvPMX+LAiRQOncA=@vger.kernel.org X-Gm-Message-State: AOJu0YyxWD4C4jwzAKRew1gUJOXx52RCrTVLAHkM6kQiJ0xbrXdn7ljp 6zkYyiXBPJvjsVmcasyRkwaSQp1lZeJTjkkWB/WqmGP+ZHb/FR37Q7k6avEUrvWXIhtmmeWzgwC xGE44o63QzCkCWwwzXQndkQ== X-Google-Smtp-Source: AGHT+IGor0R9LeqETB1vktgkLfZglF28he79PoEeU9oNKftoaSUt6owPZhDmQ7APNsMa2/gZBH6jr+0UQACvRDYBbA== X-Received: from plblq5.prod.google.com ([2002:a17:903:1445:b0:216:25a2:2ecf]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:db10:b0:220:e7d8:a8a6 with SMTP id d9443c01a7336-221709892e6mr88623765ad.27.1740017364391; Wed, 19 Feb 2025 18:09:24 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:07 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-3-almasrymina@google.com> Subject: [PATCH net-next v4 2/9] net: devmem: TCP tx netlink api From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Stanislav Fomichev Add bind-tx netlink call to attach dmabuf for TX; queue is not required, only ifindex and dmabuf fd for attachment. Signed-off-by: Stanislav Fomichev Signed-off-by: Mina Almasry --- v3: - Fix ynl-regen.sh error (Simon). --- Documentation/netlink/specs/netdev.yaml | 12 ++++++++++++ include/uapi/linux/netdev.h | 1 + net/core/netdev-genl-gen.c | 13 +++++++++++++ net/core/netdev-genl-gen.h | 1 + net/core/netdev-genl.c | 6 ++++++ tools/include/uapi/linux/netdev.h | 1 + 6 files changed, 34 insertions(+) diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlin= k/specs/netdev.yaml index 766b82005d18..aa8c289c254e 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -739,6 +739,18 @@ operations: - defer-hard-irqs - gro-flush-timeout - irq-suspend-timeout + - + name: bind-tx + doc: Bind dmabuf to netdev for TX + attribute-set: dmabuf + do: + request: + attributes: + - ifindex + - fd + reply: + attributes: + - id =20 kernel-family: headers: [ "linux/list.h"] diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h index 4e82f3871473..a3ce02b55b5f 100644 --- a/include/uapi/linux/netdev.h +++ b/include/uapi/linux/netdev.h @@ -216,6 +216,7 @@ enum { NETDEV_CMD_QSTATS_GET, NETDEV_CMD_BIND_RX, NETDEV_CMD_NAPI_SET, + NETDEV_CMD_BIND_TX, =20 __NETDEV_CMD_MAX, NETDEV_CMD_MAX =3D (__NETDEV_CMD_MAX - 1) diff --git a/net/core/netdev-genl-gen.c b/net/core/netdev-genl-gen.c index 996ac6a449eb..f27608d6301c 100644 --- a/net/core/netdev-genl-gen.c +++ b/net/core/netdev-genl-gen.c @@ -99,6 +99,12 @@ static const struct nla_policy netdev_napi_set_nl_policy= [NETDEV_A_NAPI_IRQ_SUSPE [NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT] =3D { .type =3D NLA_UINT, }, }; =20 +/* NETDEV_CMD_BIND_TX - do */ +static const struct nla_policy netdev_bind_tx_nl_policy[NETDEV_A_DMABUF_FD= + 1] =3D { + [NETDEV_A_DMABUF_IFINDEX] =3D NLA_POLICY_MIN(NLA_U32, 1), + [NETDEV_A_DMABUF_FD] =3D { .type =3D NLA_U32, }, +}; + /* Ops table for netdev */ static const struct genl_split_ops netdev_nl_ops[] =3D { { @@ -190,6 +196,13 @@ static const struct genl_split_ops netdev_nl_ops[] =3D= { .maxattr =3D NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, .flags =3D GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, + { + .cmd =3D NETDEV_CMD_BIND_TX, + .doit =3D netdev_nl_bind_tx_doit, + .policy =3D netdev_bind_tx_nl_policy, + .maxattr =3D NETDEV_A_DMABUF_FD, + .flags =3D GENL_CMD_CAP_DO, + }, }; =20 static const struct genl_multicast_group netdev_nl_mcgrps[] =3D { diff --git a/net/core/netdev-genl-gen.h b/net/core/netdev-genl-gen.h index e09dd7539ff2..c1fed66e92b9 100644 --- a/net/core/netdev-genl-gen.h +++ b/net/core/netdev-genl-gen.h @@ -34,6 +34,7 @@ int netdev_nl_qstats_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb); int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info); int netdev_nl_napi_set_doit(struct sk_buff *skb, struct genl_info *info); +int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info); =20 enum { NETDEV_NLGRP_MGMT, diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index c92fba65b20d..73420edd3f1f 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -929,6 +929,12 @@ int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct= genl_info *info) return err; } =20 +/* stub */ +int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info) +{ + return 0; +} + void netdev_nl_sock_priv_init(struct list_head *priv) { INIT_LIST_HEAD(priv); diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/n= etdev.h index 4e82f3871473..a3ce02b55b5f 100644 --- a/tools/include/uapi/linux/netdev.h +++ b/tools/include/uapi/linux/netdev.h @@ -216,6 +216,7 @@ enum { NETDEV_CMD_QSTATS_GET, NETDEV_CMD_BIND_RX, NETDEV_CMD_NAPI_SET, + NETDEV_CMD_BIND_TX, =20 __NETDEV_CMD_MAX, NETDEV_CMD_MAX =3D (__NETDEV_CMD_MAX - 1) --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0E191D5CF9 for ; Thu, 20 Feb 2025 02:09:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017370; cv=none; b=SaZCXIrvmZFkNZL9ms2Nngds41sskdRKkl+Ihc5sAT7UoOPQTQNJGMCldXaBah73prMEnD4V7L4OPGX+DTe41RNTIA+F7/VUDhoH0BMeCvpTdf9cIR+c5BGVmNyt4H92Cs5ocic8HzGipETNjmMTHJwuqGhiV+az9axZ39z4HD0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017370; c=relaxed/simple; bh=e5hPF12eEJLftTNryZbJDs4ubF/USuqVahDAV/kwMiM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=n1svM5FNnQE4Q4fBNuG2gJsoM3N42DCsbeeQlzyMQp2sBq1nNlm+RCTR3JRtX5qk1yv7uwCJpcZAiR041o3GqPAfZNb2sw0ajfV1D8IGKcNARGNB2CJg3T8r9eoVZi7xH394rO80rinw7P9jXuCB6VADujpebNW6QaRsyBZsTlg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=nlBnPRdM; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="nlBnPRdM" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-220e04e67e2so10983415ad.2 for ; Wed, 19 Feb 2025 18:09:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017366; x=1740622166; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=quOt6cIc6toB/KE2mpDVuKSMVPSUKODHhLxEz5QkXPo=; b=nlBnPRdMd3jx1UyaHkggJC7HIVxbIymUSliVa4YCifyOegQeYtM6fpDSOJRl3RwZ0e bbeWPW4WdEBHpMyn8fr1Bm0W8X9eSo3/nzUJSnQ4qHuGErbYcN+GkgyolQnx1YKM/1iG gTDYfMblQTwCzLl2WdbBRSCX46/wrQs5Iu5ZMo3pxDC7tvqqSN/IelGpY24iHdVfItAV zZbs+FTb/+nrgBNj4uGBPl+WGjJOE3LCh89PujnrcxdEJhwZkaqCmiz8cEHnVKZJ5iZ2 t8AbDd9hjOVWaQ0EAGtCxPu4z3a48N5/GbDx5aJ+lhmP6tpO86ZAtGQsXtnm1Sur++1o /orA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017366; x=1740622166; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=quOt6cIc6toB/KE2mpDVuKSMVPSUKODHhLxEz5QkXPo=; b=gY4HaWJe6MMOJsgIkNGhz7OWc6XcAKmglA6NyUfmtaQ13yWiChOHEbXEUj5kW6SwiL 8YbZJE4LLAvjquPFH9DsGIu7w7Nh+/7NZJiMQ0WXnF8S47obLKxXxsnb8M7kMcs9Qr3Y 4jmagVbTwID/cMIFQcxtl36jbq8r06rLdMDk+/O3a8+UL97CfZHEiVG+QajGb5uSFEJF W0S/aq9DRepcYflje8yBMau0bcnjnaz6G1LLVhoAmG9wMXEBui1L7ezfE67QJXLQxtKG EeSA4UkKGF8Q2dZMeXLLTMUzwIl+4BZlXTUy9EW2QdfdnL0RkqjOHI67ENY66wFfBngK UCAg== X-Forwarded-Encrypted: i=1; AJvYcCVFKiTUwCUAgH4K591gDprlKc59vKKCUvmE/lGFatrwZfnQ+8qUYaeF+sdaH1WlAVeQVMpEzNfRR1zlEHU=@vger.kernel.org X-Gm-Message-State: AOJu0Yxq4XlsmDeCGFTgwPwpg18GQ5ZVGcUD3MRAi2RLJSLdzBbJVSVL Vb4qt87sJOn/+vHs8grJWnLH5Qf4nUDob/Lwx4GunGiFdhOZhK3jQuB6ShSqEdUyxOscL2mM68e zzSbTBn+9efF9iG17fbzXVg== X-Google-Smtp-Source: AGHT+IGSoO0GGIzikoGJKWqXOqz6rpD+nq1kteh9bCYGon+EO4UAEWFbDa90cbaW8OVDKqDl0BcrGHEJEkWIbDITeg== X-Received: from plbkh8.prod.google.com ([2002:a17:903:648:b0:221:751f:dafa]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ecca:b0:220:e9ef:ec98 with SMTP id d9443c01a7336-221709622b7mr104598725ad.19.1740017365815; Wed, 19 Feb 2025 18:09:25 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:08 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-4-almasrymina@google.com> Subject: [PATCH net-next v4 3/9] net: devmem: Implement TX path From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja , Kaiyuan Zhang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Augment dmabuf binding to be able to handle TX. Additional to all the RX binding, we also create tx_vec needed for the TX path. Provide API for sendmsg to be able to send dmabufs bound to this device: - Provide a new dmabuf_tx_cmsg which includes the dmabuf to send from. - MSG_ZEROCOPY with SCM_DEVMEM_DMABUF cmsg indicates send from dma-buf. Devmem is uncopyable, so piggyback off the existing MSG_ZEROCOPY implementation, while disabling instances where MSG_ZEROCOPY falls back to copying. We additionally pipe the binding down to the new zerocopy_fill_skb_from_devmem which fills a TX skb with net_iov netmems instead of the traditional page netmems. We also special case skb_frag_dma_map to return the dma-address of these dmabuf net_iovs instead of attempting to map pages. Based on work by Stanislav Fomichev . A lot of the meat of the implementation came from devmem TCP RFC v1[1], which included the TX path, but Stan did all the rebasing on top of netmem/net_iov. Cc: Stanislav Fomichev Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry --- v4: - Remove dmabuf_tx_cmsg definition and just use __u32 for the dma-buf id (Willem). - Check that iov_iter_type() is ITER_IOVEC in zerocopy_fill_skb_from_iter() (Pavel). - Fix binding->tx_vec not being freed on error paths (Paolo). - Make devmem patch mutually exclusive with msg->ubuf_info path (Pavel). - Check that MSG_ZEROCOPY and SOCK_ZEROCOPY are provided when sockc.dmabuf_id is provided. - Don't mm_account_pinned_pages() on devmem TX (Pavel). v3: - Use kvmalloc_array instead of kcalloc (Stan). - Fix unreachable code warning (Simon). v2: - Remove dmabuf_offset from the dmabuf cmsg. - Update zerocopy_fill_skb_from_devmem to interpret the iov_base/iter_iov_addr as the offset into the dmabuf to send from (Stan). - Remove the confusing binding->tx_iter which is not needed if we interpret the iov_base/iter_iov_addr as offset into the dmabuf (Stan). - Remove check for binding->sgt and binding->sgt->nents in dmabuf binding. - Simplify the calculation of binding->tx_vec. - Check in net_devmem_get_binding that the binding we're returning has ifindex matching the sending socket (Willem). --- include/linux/skbuff.h | 17 +++- include/net/sock.h | 1 + net/core/datagram.c | 48 +++++++++++- net/core/devmem.c | 100 ++++++++++++++++++++++-- net/core/devmem.h | 41 +++++++++- net/core/netdev-genl.c | 64 ++++++++++++++- net/core/skbuff.c | 18 +++-- net/core/sock.c | 6 ++ net/ipv4/ip_output.c | 3 +- net/ipv4/tcp.c | 46 ++++++++--- net/ipv6/ip6_output.c | 3 +- net/vmw_vsock/virtio_transport_common.c | 5 +- 12 files changed, 309 insertions(+), 43 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index bb2b751d274a..b8783aa94c1e 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1707,13 +1707,16 @@ static inline void skb_set_end_offset(struct sk_buf= f *skb, unsigned int offset) extern const struct ubuf_info_ops msg_zerocopy_ubuf_ops; =20 struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, - struct ubuf_info *uarg); + struct ubuf_info *uarg, bool devmem); =20 void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref); =20 +struct net_devmem_dmabuf_binding; + int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, struct sk_buff *skb, struct iov_iter *from, - size_t length); + size_t length, + struct net_devmem_dmabuf_binding *binding); =20 int zerocopy_fill_skb_from_iter(struct sk_buff *skb, struct iov_iter *from, size_t length); @@ -1721,12 +1724,14 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb, static inline int skb_zerocopy_iter_dgram(struct sk_buff *skb, struct msghdr *msg, int len) { - return __zerocopy_sg_from_iter(msg, skb->sk, skb, &msg->msg_iter, len); + return __zerocopy_sg_from_iter(msg, skb->sk, skb, &msg->msg_iter, len, + NULL); } =20 int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, struct msghdr *msg, int len, - struct ubuf_info *uarg); + struct ubuf_info *uarg, + struct net_devmem_dmabuf_binding *binding); =20 /* Internal */ #define skb_shinfo(SKB) ((struct skb_shared_info *)(skb_end_pointer(SKB))) @@ -3697,6 +3702,10 @@ static inline dma_addr_t __skb_frag_dma_map(struct d= evice *dev, size_t offset, size_t size, enum dma_data_direction dir) { + if (skb_frag_is_net_iov(frag)) { + return netmem_to_net_iov(frag->netmem)->dma_addr + offset + + frag->offset; + } return dma_map_page(dev, skb_frag_page(frag), skb_frag_off(frag) + offset, size, dir); } diff --git a/include/net/sock.h b/include/net/sock.h index fac65ed30983..aac7e9d92ba5 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1823,6 +1823,7 @@ struct sockcm_cookie { u32 tsflags; u32 ts_opt_id; u32 priority; + u32 dmabuf_id; }; =20 static inline void sockcm_init(struct sockcm_cookie *sockc, diff --git a/net/core/datagram.c b/net/core/datagram.c index f0693707aece..3e60c83305cc 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -63,6 +63,8 @@ #include #include =20 +#include "devmem.h" + /* * Is a socket 'connection oriented' ? */ @@ -692,9 +694,49 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb, return 0; } =20 +static int +zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from, + int length, + struct net_devmem_dmabuf_binding *binding) +{ + int i =3D skb_shinfo(skb)->nr_frags; + size_t virt_addr, size, off; + struct net_iov *niov; + + /* Devmem filling works by taking an IOVEC from the user where the + * iov_addrs are interpreted as an offset in bytes into the dma-buf to + * send from. We do not support other iter types. + */ + if (iov_iter_type(from) !=3D ITER_IOVEC) + return -EINVAL; + + while (length && iov_iter_count(from)) { + if (i =3D=3D MAX_SKB_FRAGS) + return -EMSGSIZE; + + virt_addr =3D (size_t)iter_iov_addr(from); + niov =3D net_devmem_get_niov_at(binding, virt_addr, &off, &size); + if (!niov) + return -EFAULT; + + size =3D min_t(size_t, size, length); + size =3D min_t(size_t, size, iter_iov_len(from)); + + get_netmem(net_iov_to_netmem(niov)); + skb_add_rx_frag_netmem(skb, i, net_iov_to_netmem(niov), off, + size, PAGE_SIZE); + iov_iter_advance(from, size); + length -=3D size; + i++; + } + + return 0; +} + int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk, struct sk_buff *skb, struct iov_iter *from, - size_t length) + size_t length, + struct net_devmem_dmabuf_binding *binding) { unsigned long orig_size =3D skb->truesize; unsigned long truesize; @@ -702,6 +744,8 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct = sock *sk, =20 if (msg && msg->msg_ubuf && msg->sg_from_iter) ret =3D msg->sg_from_iter(skb, from, length); + else if (unlikely(binding)) + ret =3D zerocopy_fill_skb_from_devmem(skb, from, length, binding); else ret =3D zerocopy_fill_skb_from_iter(skb, from, length); =20 @@ -735,7 +779,7 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct i= ov_iter *from) if (skb_copy_datagram_from_iter(skb, 0, from, copy)) return -EFAULT; =20 - return __zerocopy_sg_from_iter(NULL, NULL, skb, from, ~0U); + return __zerocopy_sg_from_iter(NULL, NULL, skb, from, ~0U, NULL); } EXPORT_SYMBOL(zerocopy_sg_from_iter); =20 diff --git a/net/core/devmem.c b/net/core/devmem.c index b1aafc66ebb7..2e576f80b1d8 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -17,6 +17,7 @@ #include #include #include +#include #include =20 #include "devmem.h" @@ -73,8 +74,10 @@ void __net_devmem_dmabuf_binding_free(struct net_devmem_= dmabuf_binding *binding) dma_buf_detach(binding->dmabuf, binding->attachment); dma_buf_put(binding->dmabuf); xa_destroy(&binding->bound_rxqs); + kvfree(binding->tx_vec); kfree(binding); } +EXPORT_SYMBOL(__net_devmem_dmabuf_binding_free); =20 struct net_iov * net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding) @@ -119,6 +122,13 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf= _binding *binding) unsigned long xa_idx; unsigned int rxq_idx; =20 + xa_erase(&net_devmem_dmabuf_bindings, binding->id); + + /* Ensure no tx net_devmem_lookup_dmabuf() are in flight after the + * erase. + */ + synchronize_net(); + if (binding->list.next) list_del(&binding->list); =20 @@ -133,8 +143,6 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_= binding *binding) WARN_ON(netdev_rx_queue_restart(binding->dev, rxq_idx)); } =20 - xa_erase(&net_devmem_dmabuf_bindings, binding->id); - net_devmem_dmabuf_binding_put(binding); } =20 @@ -197,8 +205,9 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *= dev, u32 rxq_idx, } =20 struct net_devmem_dmabuf_binding * -net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, - struct netlink_ext_ack *extack) +net_devmem_bind_dmabuf(struct net_device *dev, + enum dma_data_direction direction, + unsigned int dmabuf_fd, struct netlink_ext_ack *extack) { struct net_devmem_dmabuf_binding *binding; static u32 id_alloc_next; @@ -241,7 +250,7 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned= int dmabuf_fd, } =20 binding->sgt =3D dma_buf_map_attachment_unlocked(binding->attachment, - DMA_FROM_DEVICE); + direction); if (IS_ERR(binding->sgt)) { err =3D PTR_ERR(binding->sgt); NL_SET_ERR_MSG(extack, "Failed to map dmabuf attachment"); @@ -252,13 +261,23 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsign= ed int dmabuf_fd, * binding can be much more flexible than that. We may be able to * allocate MTU sized chunks here. Leave that for future work... */ - binding->chunk_pool =3D - gen_pool_create(PAGE_SHIFT, dev_to_node(&dev->dev)); + binding->chunk_pool =3D gen_pool_create(PAGE_SHIFT, + dev_to_node(&dev->dev)); if (!binding->chunk_pool) { err =3D -ENOMEM; goto err_unmap; } =20 + if (direction =3D=3D DMA_TO_DEVICE) { + binding->tx_vec =3D kvmalloc_array(dmabuf->size / PAGE_SIZE, + sizeof(struct net_iov *), + GFP_KERNEL); + if (!binding->tx_vec) { + err =3D -ENOMEM; + goto err_free_chunks; + } + } + virtual =3D 0; for_each_sgtable_dma_sg(binding->sgt, sg, sg_idx) { dma_addr_t dma_addr =3D sg_dma_address(sg); @@ -300,6 +319,8 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned= int dmabuf_fd, niov->owner =3D &owner->area; page_pool_set_dma_addr_netmem(net_iov_to_netmem(niov), net_devmem_get_dma_addr(niov)); + if (direction =3D=3D DMA_TO_DEVICE) + binding->tx_vec[owner->area.base_virtual / PAGE_SIZE + i] =3D niov; } =20 virtual +=3D len; @@ -311,6 +332,9 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned= int dmabuf_fd, gen_pool_for_each_chunk(binding->chunk_pool, net_devmem_dmabuf_free_chunk_owner, NULL); gen_pool_destroy(binding->chunk_pool); + + if (direction =3D=3D DMA_TO_DEVICE) + kvfree(binding->tx_vec); err_unmap: dma_buf_unmap_attachment_unlocked(binding->attachment, binding->sgt, DMA_FROM_DEVICE); @@ -325,6 +349,21 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigne= d int dmabuf_fd, return ERR_PTR(err); } =20 +struct net_devmem_dmabuf_binding *net_devmem_lookup_dmabuf(u32 id) +{ + struct net_devmem_dmabuf_binding *binding; + + rcu_read_lock(); + binding =3D xa_load(&net_devmem_dmabuf_bindings, id); + if (binding) { + if (!net_devmem_dmabuf_binding_get(binding)) + binding =3D NULL; + } + rcu_read_unlock(); + + return binding; +} + void net_devmem_get_net_iov(struct net_iov *niov) { net_devmem_dmabuf_binding_get(net_devmem_iov_binding(niov)); @@ -335,6 +374,53 @@ void net_devmem_put_net_iov(struct net_iov *niov) net_devmem_dmabuf_binding_put(net_devmem_iov_binding(niov)); } =20 +struct net_devmem_dmabuf_binding *net_devmem_get_binding(struct sock *sk, + unsigned int dmabuf_id) +{ + struct net_devmem_dmabuf_binding *binding; + struct dst_entry *dst =3D __sk_dst_get(sk); + int err =3D 0; + + binding =3D net_devmem_lookup_dmabuf(dmabuf_id); + if (!binding || !binding->tx_vec) { + err =3D -EINVAL; + goto out_err; + } + + /* The dma-addrs in this binding are only reachable to the corresponding + * net_device. + */ + if (!dst || !dst->dev || dst->dev->ifindex !=3D binding->dev->ifindex) { + err =3D -ENODEV; + goto out_err; + } + + return binding; + +out_err: + if (binding) + net_devmem_dmabuf_binding_put(binding); + + return ERR_PTR(err); +} + +struct net_iov * +net_devmem_get_niov_at(struct net_devmem_dmabuf_binding *binding, + size_t virt_addr, size_t *off, size_t *size) +{ + size_t idx; + + if (virt_addr >=3D binding->dmabuf->size) + return NULL; + + idx =3D virt_addr / PAGE_SIZE; + + *off =3D virt_addr % PAGE_SIZE; + *size =3D PAGE_SIZE - *off; + + return binding->tx_vec[idx]; +} + /*** "Dmabuf devmem memory provider" ***/ =20 int mp_dmabuf_devmem_init(struct page_pool *pool) diff --git a/net/core/devmem.h b/net/core/devmem.h index 946f2e015746..a8b79c0e01b3 100644 --- a/net/core/devmem.h +++ b/net/core/devmem.h @@ -48,6 +48,12 @@ struct net_devmem_dmabuf_binding { * active. */ u32 id; + + /* Array of net_iov pointers for this binding, sorted by virtual + * address. This array is convenient to map the virtual addresses to + * net_iovs in the TX path. + */ + struct net_iov **tx_vec; }; =20 #if defined(CONFIG_NET_DEVMEM) @@ -66,12 +72,15 @@ struct dmabuf_genpool_chunk_owner { =20 void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *bi= nding); struct net_devmem_dmabuf_binding * -net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, - struct netlink_ext_ack *extack); +net_devmem_bind_dmabuf(struct net_device *dev, + enum dma_data_direction direction, + unsigned int dmabuf_fd, struct netlink_ext_ack *extack); +struct net_devmem_dmabuf_binding *net_devmem_lookup_dmabuf(u32 id); void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding); int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, struct net_devmem_dmabuf_binding *binding, struct netlink_ext_ack *extack); +void net_devmem_bind_tx_release(struct sock *sk); =20 static inline struct dmabuf_genpool_chunk_owner * net_devmem_iov_to_chunk_owner(const struct net_iov *niov) @@ -100,10 +109,10 @@ static inline unsigned long net_iov_virtual_addr(cons= t struct net_iov *niov) ((unsigned long)net_iov_idx(niov) << PAGE_SHIFT); } =20 -static inline void +static inline bool net_devmem_dmabuf_binding_get(struct net_devmem_dmabuf_binding *binding) { - refcount_inc(&binding->ref); + return refcount_inc_not_zero(&binding->ref); } =20 static inline void @@ -123,6 +132,11 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_bindi= ng *binding); void net_devmem_free_dmabuf(struct net_iov *ppiov); =20 bool net_is_devmem_iov(struct net_iov *niov); +struct net_devmem_dmabuf_binding * +net_devmem_get_binding(struct sock *sk, unsigned int dmabuf_id); +struct net_iov * +net_devmem_get_niov_at(struct net_devmem_dmabuf_binding *binding, size_t a= ddr, + size_t *off, size_t *size); =20 #else struct net_devmem_dmabuf_binding; @@ -147,11 +161,17 @@ __net_devmem_dmabuf_binding_free(struct net_devmem_dm= abuf_binding *binding) =20 static inline struct net_devmem_dmabuf_binding * net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, + enum dma_data_direction direction, struct netlink_ext_ack *extack) { return ERR_PTR(-EOPNOTSUPP); } =20 +static inline struct net_devmem_dmabuf_binding *net_devmem_lookup_dmabuf(u= 32 id) +{ + return NULL; +} + static inline void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) { @@ -190,6 +210,19 @@ static inline bool net_is_devmem_iov(struct net_iov *n= iov) { return false; } + +static inline struct net_devmem_dmabuf_binding * +net_devmem_get_binding(struct sock *sk, unsigned int dmabuf_id) +{ + return ERR_PTR(-EOPNOTSUPP); +} + +static inline struct net_iov * +net_devmem_get_niov_at(struct net_devmem_dmabuf_binding *binding, size_t a= ddr, + size_t *off, size_t *size) +{ + return NULL; +} #endif =20 #endif /* _NET_DEVMEM_H */ diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 73420edd3f1f..1faa2cf4057f 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -872,7 +872,8 @@ int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct = genl_info *info) goto err_unlock; } =20 - binding =3D net_devmem_bind_dmabuf(netdev, dmabuf_fd, info->extack); + binding =3D net_devmem_bind_dmabuf(netdev, DMA_FROM_DEVICE, dmabuf_fd, + info->extack); if (IS_ERR(binding)) { err =3D PTR_ERR(binding); goto err_unlock; @@ -929,10 +930,67 @@ int netdev_nl_bind_rx_doit(struct sk_buff *skb, struc= t genl_info *info) return err; } =20 -/* stub */ int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info) { - return 0; + struct net_devmem_dmabuf_binding *binding; + struct list_head *sock_binding_list; + struct net_device *netdev; + u32 ifindex, dmabuf_fd; + struct sk_buff *rsp; + int err =3D 0; + void *hdr; + + if (GENL_REQ_ATTR_CHECK(info, NETDEV_A_DEV_IFINDEX) || + GENL_REQ_ATTR_CHECK(info, NETDEV_A_DMABUF_FD)) + return -EINVAL; + + ifindex =3D nla_get_u32(info->attrs[NETDEV_A_DEV_IFINDEX]); + dmabuf_fd =3D nla_get_u32(info->attrs[NETDEV_A_DMABUF_FD]); + + sock_binding_list =3D genl_sk_priv_get(&netdev_nl_family, + NETLINK_CB(skb).sk); + if (IS_ERR(sock_binding_list)) + return PTR_ERR(sock_binding_list); + + rsp =3D genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!rsp) + return -ENOMEM; + + hdr =3D genlmsg_iput(rsp, info); + if (!hdr) { + err =3D -EMSGSIZE; + goto err_genlmsg_free; + } + + rtnl_lock(); + + netdev =3D __dev_get_by_index(genl_info_net(info), ifindex); + if (!netdev || !netif_device_present(netdev)) { + err =3D -ENODEV; + goto err_unlock; + } + + binding =3D net_devmem_bind_dmabuf(netdev, DMA_TO_DEVICE, dmabuf_fd, + info->extack); + if (IS_ERR(binding)) { + err =3D PTR_ERR(binding); + goto err_unlock; + } + + list_add(&binding->list, sock_binding_list); + + nla_put_u32(rsp, NETDEV_A_DMABUF_ID, binding->id); + genlmsg_end(rsp, hdr); + + rtnl_unlock(); + + return genlmsg_reply(rsp, info); + +err_unlock: + rtnl_unlock(); +err_genlmsg_free: + nlmsg_free(rsp); + return err; } =20 void netdev_nl_sock_priv_init(struct list_head *priv) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 815245d5c36b..7ff68502e752 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1695,7 +1695,8 @@ void mm_unaccount_pinned_pages(struct mmpin *mmp) } EXPORT_SYMBOL_GPL(mm_unaccount_pinned_pages); =20 -static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) +static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size, + bool devmem) { struct ubuf_info_msgzc *uarg; struct sk_buff *skb; @@ -1710,7 +1711,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct so= ck *sk, size_t size) uarg =3D (void *)skb->cb; uarg->mmp.user =3D NULL; =20 - if (mm_account_pinned_pages(&uarg->mmp, size)) { + if (likely(!devmem) && mm_account_pinned_pages(&uarg->mmp, size)) { kfree_skb(skb); return NULL; } @@ -1733,7 +1734,7 @@ static inline struct sk_buff *skb_from_uarg(struct ub= uf_info_msgzc *uarg) } =20 struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, - struct ubuf_info *uarg) + struct ubuf_info *uarg, bool devmem) { if (uarg) { struct ubuf_info_msgzc *uarg_zc; @@ -1763,7 +1764,8 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *s= k, size_t size, =20 next =3D (u32)atomic_read(&sk->sk_zckey); if ((u32)(uarg_zc->id + uarg_zc->len) =3D=3D next) { - if (mm_account_pinned_pages(&uarg_zc->mmp, size)) + if (likely(!devmem) && + mm_account_pinned_pages(&uarg_zc->mmp, size)) return NULL; uarg_zc->len++; uarg_zc->bytelen =3D bytelen; @@ -1778,7 +1780,7 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *s= k, size_t size, } =20 new_alloc: - return msg_zerocopy_alloc(sk, size); + return msg_zerocopy_alloc(sk, size, devmem); } EXPORT_SYMBOL_GPL(msg_zerocopy_realloc); =20 @@ -1882,7 +1884,8 @@ EXPORT_SYMBOL_GPL(msg_zerocopy_ubuf_ops); =20 int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, struct msghdr *msg, int len, - struct ubuf_info *uarg) + struct ubuf_info *uarg, + struct net_devmem_dmabuf_binding *binding) { int err, orig_len =3D skb->len; =20 @@ -1901,7 +1904,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct = sk_buff *skb, return -EEXIST; } =20 - err =3D __zerocopy_sg_from_iter(msg, sk, skb, &msg->msg_iter, len); + err =3D __zerocopy_sg_from_iter(msg, sk, skb, &msg->msg_iter, len, + binding); if (err =3D=3D -EFAULT || (err =3D=3D -EMSGSIZE && skb->len =3D=3D orig_l= en)) { struct sock *save_sk =3D skb->sk; =20 diff --git a/net/core/sock.c b/net/core/sock.c index 53c7af0038c4..3f9502269b1c 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2965,6 +2965,12 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr= *cmsg, if (!sk_set_prio_allowed(sk, *(u32 *)CMSG_DATA(cmsg))) return -EPERM; sockc->priority =3D *(u32 *)CMSG_DATA(cmsg); + break; + case SCM_DEVMEM_DMABUF: + if (cmsg->cmsg_len !=3D CMSG_LEN(sizeof(u32))) + return -EINVAL; + sockc->dmabuf_id =3D *(u32 *)CMSG_DATA(cmsg); + break; default: return -EINVAL; diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index ea7a260bec8a..7d8a5f3fae9b 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1015,7 +1015,8 @@ static int __ip_append_data(struct sock *sk, uarg =3D msg->msg_ubuf; } } else if (sock_flag(sk, SOCK_ZEROCOPY)) { - uarg =3D msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); + uarg =3D msg_zerocopy_realloc(sk, length, skb_zcopy(skb), + false); if (!uarg) return -ENOBUFS; extra_uref =3D !skb_zcopy(skb); /* only ref on new uarg */ diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 6a8f19a10911..374c275a83b4 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1055,6 +1055,7 @@ int tcp_sendmsg_fastopen(struct sock *sk, struct msgh= dr *msg, int *copied, =20 int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) { + struct net_devmem_dmabuf_binding *binding =3D NULL; struct tcp_sock *tp =3D tcp_sk(sk); struct ubuf_info *uarg =3D NULL; struct sk_buff *skb; @@ -1067,6 +1068,16 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghd= r *msg, size_t size) =20 flags =3D msg->msg_flags; =20 + sockc =3D (struct sockcm_cookie){ .tsflags =3D READ_ONCE(sk->sk_tsflags), + .dmabuf_id =3D 0 }; + if (msg->msg_controllen) { + err =3D sock_cmsg_send(sk, msg, &sockc); + if (unlikely(err)) { + err =3D -EINVAL; + goto out_err; + } + } + if ((flags & MSG_ZEROCOPY) && size) { if (msg->msg_ubuf) { uarg =3D msg->msg_ubuf; @@ -1074,7 +1085,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr= *msg, size_t size) zc =3D MSG_ZEROCOPY; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { skb =3D tcp_write_queue_tail(sk); - uarg =3D msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); + uarg =3D msg_zerocopy_realloc(sk, size, skb_zcopy(skb), + !!sockc.dmabuf_id); if (!uarg) { err =3D -ENOBUFS; goto out_err; @@ -1083,12 +1095,27 @@ int tcp_sendmsg_locked(struct sock *sk, struct msgh= dr *msg, size_t size) zc =3D MSG_ZEROCOPY; else uarg_to_msgzc(uarg)->zerocopy =3D 0; + + if (sockc.dmabuf_id) { + binding =3D net_devmem_get_binding(sk, sockc.dmabuf_id); + if (IS_ERR(binding)) { + err =3D PTR_ERR(binding); + binding =3D NULL; + goto out_err; + } + } } } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { if (sk->sk_route_caps & NETIF_F_SG) zc =3D MSG_SPLICE_PAGES; } =20 + if (sockc.dmabuf_id && + (!(flags & MSG_ZEROCOPY) || !sock_flag(sk, SOCK_ZEROCOPY))) { + err =3D -EINVAL; + goto out_err; + } + if (unlikely(flags & MSG_FASTOPEN || inet_test_bit(DEFER_CONNECT, sk)) && !tp->repair) { @@ -1127,15 +1154,6 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghd= r *msg, size_t size) /* 'common' sending to sendq */ } =20 - sockc =3D (struct sockcm_cookie) { .tsflags =3D READ_ONCE(sk->sk_tsflags)= }; - if (msg->msg_controllen) { - err =3D sock_cmsg_send(sk, msg, &sockc); - if (unlikely(err)) { - err =3D -EINVAL; - goto out_err; - } - } - /* This should be in poll */ sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); =20 @@ -1252,7 +1270,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr= *msg, size_t size) goto wait_for_space; } =20 - err =3D skb_zerocopy_iter_stream(sk, skb, msg, copy, uarg); + err =3D skb_zerocopy_iter_stream(sk, skb, msg, copy, uarg, + binding); if (err =3D=3D -EMSGSIZE || err =3D=3D -EEXIST) { tcp_mark_push(tp, skb); goto new_segment; @@ -1333,6 +1352,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr= *msg, size_t size) /* msg->msg_ubuf is pinned by the caller so we don't take extra refs */ if (uarg && !msg->msg_ubuf) net_zcopy_put(uarg); + if (binding) + net_devmem_dmabuf_binding_put(binding); return copied + copied_syn; =20 do_error: @@ -1350,6 +1371,9 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr= *msg, size_t size) sk->sk_write_space(sk); tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); } + if (binding) + net_devmem_dmabuf_binding_put(binding); + return err; } EXPORT_SYMBOL_GPL(tcp_sendmsg_locked); diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index d577bf2f3053..e9e752f08f87 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1523,7 +1523,8 @@ static int __ip6_append_data(struct sock *sk, uarg =3D msg->msg_ubuf; } } else if (sock_flag(sk, SOCK_ZEROCOPY)) { - uarg =3D msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); + uarg =3D msg_zerocopy_realloc(sk, length, skb_zcopy(skb), + false); if (!uarg) return -ENOBUFS; extra_uref =3D !skb_zcopy(skb); /* only ref on new uarg */ diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio= _transport_common.c index 7f7de6d88096..6e7b727c781c 100644 --- a/net/vmw_vsock/virtio_transport_common.c +++ b/net/vmw_vsock/virtio_transport_common.c @@ -87,7 +87,7 @@ static int virtio_transport_init_zcopy_skb(struct vsock_s= ock *vsk, =20 uarg =3D msg_zerocopy_realloc(sk_vsock(vsk), iter->count, - NULL); + NULL, false); if (!uarg) return -1; =20 @@ -107,8 +107,7 @@ static int virtio_transport_fill_skb(struct sk_buff *sk= b, { if (zcopy) return __zerocopy_sg_from_iter(info->msg, NULL, skb, - &info->msg->msg_iter, - len); + &info->msg->msg_iter, len, NULL); =20 return memcpy_from_msg(skb_put(skb, len), info->msg, len); } --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2D001D7E5B for ; Thu, 20 Feb 2025 02:09:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017371; cv=none; b=Wt6wxya83Wnk2GbmXV3A+BYXHjqymjpM/v2Rjf5kpYNlyc5QdxVpv5kgeLaMjT2/DENWZiI8QYJpVe6YNtkZZi9+Pg6jNYWkqYxzktdVhhGkwV6sXk9Fqb0EMHIb5ZQWFj0Bu/oZ1oKK5+E3bFgzoQPiHPmOVTZPNfaZoQOq2rE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017371; c=relaxed/simple; bh=8qdnNdfQ/uj2uGtVeB+5gVRxg+q3R/Oiilzxr61Biws=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eMQvS0n6hfFFunRghG6kfn10dPWm6+F8gmAdiAyqD0gWQx84mRvBPs5jpsP0pkaFhEkiuHY+9tvYO8mGjAgAKtJksS0zdcFmYyoU1PiiEoHP/LZv4tJf4Z4cATSpcvzF5fwLBwRdDPalNZiK9m4AEzug982yAsvzyMF98YsKwEY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TELiJNu4; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TELiJNu4" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220d8599659so7952105ad.0 for ; Wed, 19 Feb 2025 18:09:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017368; x=1740622168; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=F6BSxKQ00TlgrRbtfolLBJFy1zEtBu3OhJw4muzoQDk=; b=TELiJNu4cQ31sOMhLpFyVmDrgC6Innm7xZrrkQRGqOpQwY9dzlchgsA++yqPajDzQg 5PjMSy2rObL7m/8JPdEvxGJe0omxTJiW2H8Tl4JwVi/5wrDtJu9Y4CupNWr20BqqZZiJ AoUZmHOCgrnOXrGuJxDsZjb0AbGHwFxwYGhu87jKCyfbmHe/1e7uHU6Gt9+lHxucRM6q jfLqUlMuga5aJqgBBYLVpGJf2jEYseASP2JxHfUIRBm4f/8UL+XQlVFaUlQN2V/SKFRT h7v7pK9d6yykP8pFYdkcFxznJuSC6G7TjECyYFvBJhT58o4IEWkNlQwavzBLyvAXK9Rm WsZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017368; x=1740622168; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F6BSxKQ00TlgrRbtfolLBJFy1zEtBu3OhJw4muzoQDk=; b=iwCCiQ5iyk5S3OgyAL3kl4jCS3qCWvPM9Gx/pkhM3J1CYnVqoJOpv0HrsjU11QEwNJ iOkneutX4yrmaneq5psvLZ9S3472RYPyE8Ew1Ja3ogUZnzUgaZnHBg/eyJljiWBt8CMZ bPiapM/1SxdqveFRS06zL8JEITnhPbskbbVHJulDEJe7RVllnyYipP9576czAv/ilIeH B5MBhx7XXdNc4mjg1EEMXGlFJZzzjcvG8KUatS52jW3MgZIlLTReoKWL4DZmcrJNX0wj HJGBHlVl8t0FLCKf9bTHwb//pUFEbhm5ygf6SIE5h3yken4s9fHVh8JgVTo0I5JCrRWX TJ8Q== X-Forwarded-Encrypted: i=1; AJvYcCUZp19lt1vlxcbwF5RpvDvydcLVkCVgLa0iYGYVHOJkK9Mv1MqnMUkc803wHlgcE8LvgzPO+rYRFw0lojs=@vger.kernel.org X-Gm-Message-State: AOJu0YyB4L+opckpyqR5k6KAlh9Bg5aOijWCYtwl5XFgmLl4TVNhjoqf GQ59q+nkUVh8BbC0aCOt+8lCX7KEFVrnynw+dZ8gpl73bias3JICmCiqMMtrUWSpZe26ePoEAad V8mFnbPWRYgG9SbjXShc7Fw== X-Google-Smtp-Source: AGHT+IFc+58FEfMgCkmY33XyrttMcD7+XYuckVAKhUGjEipxl0LjYlIOjsWpCf1+0tMH6PT8yDHYnCMGdo2Gw894FQ== X-Received: from pfbca6.prod.google.com ([2002:a05:6a00:4186:b0:730:848d:a5a3]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:3d8a:b0:1ee:7d91:59bd with SMTP id adf61e73a8af0-1eed4ff4811mr11266821637.31.1740017367599; Wed, 19 Feb 2025 18:09:27 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:09 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-5-almasrymina@google.com> Subject: [PATCH net-next v4 4/9] net: devmem: make dmabuf unbinding scheduled work From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The TX path may release the dmabuf in a context where we cannot wait. This happens when the user unbinds a TX dmabuf while there are still references to its netmems in the TX path. In that case, the netmems will be put_netmem'd from a context where we can't unmap the dmabuf, resulting in a BUG like seen by Stan: [ 1.548495] BUG: sleeping function called from invalid context at driver= s/dma-buf/dma-buf.c:1255 [ 1.548741] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 149, = name: ncdevmem [ 1.548926] preempt_count: 201, expected: 0 [ 1.549026] RCU nest depth: 0, expected: 0 [ 1.549197] [ 1.549237] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D [ 1.549331] [ BUG: Invalid wait context ] [ 1.549425] 6.13.0-rc3-00770-gbc9ef9606dc9-dirty #15 Tainted: G W [ 1.549609] ----------------------------- [ 1.549704] ncdevmem/149 is trying to lock: [ 1.549801] ffff8880066701c0 (reservation_ww_class_mutex){+.+.}-{4:4}, a= t: dma_buf_unmap_attachment_unlocked+0x4b/0x90 [ 1.550051] other info that might help us debug this: [ 1.550167] context-{5:5} [ 1.550229] 3 locks held by ncdevmem/149: [ 1.550322] #0: ffff888005730208 (&sb->s_type->i_mutex_key#11){+.+.}-{4= :4}, at: sock_close+0x40/0xf0 [ 1.550530] #1: ffff88800b148f98 (sk_lock-AF_INET6){+.+.}-{0:0}, at: tc= p_close+0x19/0x80 [ 1.550731] #2: ffff88800b148f18 (slock-AF_INET6){+.-.}-{3:3}, at: __tc= p_close+0x185/0x4b0 [ 1.550921] stack backtrace: [ 1.550990] CPU: 0 UID: 0 PID: 149 Comm: ncdevmem Tainted: G W = 6.13.0-rc3-00770-gbc9ef9606dc9-dirty #15 [ 1.551233] Tainted: [W]=3DWARN [ 1.551304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS = Arch Linux 1.16.3-1-1 04/01/2014 [ 1.551518] Call Trace: [ 1.551584] [ 1.551636] dump_stack_lvl+0x86/0xc0 [ 1.551723] __lock_acquire+0xb0f/0xc30 [ 1.551814] ? dma_buf_unmap_attachment_unlocked+0x4b/0x90 [ 1.551941] lock_acquire+0xf1/0x2a0 [ 1.552026] ? dma_buf_unmap_attachment_unlocked+0x4b/0x90 [ 1.552152] ? dma_buf_unmap_attachment_unlocked+0x4b/0x90 [ 1.552281] ? dma_buf_unmap_attachment_unlocked+0x4b/0x90 [ 1.552408] __ww_mutex_lock+0x121/0x1060 [ 1.552503] ? dma_buf_unmap_attachment_unlocked+0x4b/0x90 [ 1.552648] ww_mutex_lock+0x3d/0xa0 [ 1.552733] dma_buf_unmap_attachment_unlocked+0x4b/0x90 [ 1.552857] __net_devmem_dmabuf_binding_free+0x56/0xb0 [ 1.552979] skb_release_data+0x120/0x1f0 [ 1.553074] __kfree_skb+0x29/0xa0 [ 1.553156] tcp_write_queue_purge+0x41/0x310 [ 1.553259] tcp_v4_destroy_sock+0x127/0x320 [ 1.553363] ? __tcp_close+0x169/0x4b0 [ 1.553452] inet_csk_destroy_sock+0x53/0x130 [ 1.553560] __tcp_close+0x421/0x4b0 [ 1.553646] tcp_close+0x24/0x80 [ 1.553724] inet_release+0x5d/0x90 [ 1.553806] sock_close+0x4a/0xf0 [ 1.553886] __fput+0x9c/0x2b0 [ 1.553960] task_work_run+0x89/0xc0 [ 1.554046] do_exit+0x27f/0x980 [ 1.554125] do_group_exit+0xa4/0xb0 [ 1.554211] __x64_sys_exit_group+0x17/0x20 [ 1.554309] x64_sys_call+0x21a0/0x21a0 [ 1.554400] do_syscall_64+0xec/0x1d0 [ 1.554487] ? exc_page_fault+0x8a/0xf0 [ 1.554585] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.554703] RIP: 0033:0x7f2f8a27abcd Resolve this by making __net_devmem_dmabuf_binding_free schedule_work'd. Suggested-by: Stanislav Fomichev Signed-off-by: Mina Almasry Acked-by: Stanislav Fomichev --- net/core/devmem.c | 4 +++- net/core/devmem.h | 10 ++++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/net/core/devmem.c b/net/core/devmem.c index 2e576f80b1d8..4af9f754f6ff 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -55,8 +55,10 @@ static dma_addr_t net_devmem_get_dma_addr(const struct n= et_iov *niov) ((dma_addr_t)net_iov_idx(niov) << PAGE_SHIFT); } =20 -void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *bi= nding) +void __net_devmem_dmabuf_binding_free(struct work_struct *wq) { + struct net_devmem_dmabuf_binding *binding =3D container_of(wq, typeof(*bi= nding), unbind_w); + size_t size, avail; =20 gen_pool_for_each_chunk(binding->chunk_pool, diff --git a/net/core/devmem.h b/net/core/devmem.h index a8b79c0e01b3..861150349825 100644 --- a/net/core/devmem.h +++ b/net/core/devmem.h @@ -54,6 +54,8 @@ struct net_devmem_dmabuf_binding { * net_iovs in the TX path. */ struct net_iov **tx_vec; + + struct work_struct unbind_w; }; =20 #if defined(CONFIG_NET_DEVMEM) @@ -70,7 +72,7 @@ struct dmabuf_genpool_chunk_owner { dma_addr_t base_dma_addr; }; =20 -void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *bi= nding); +void __net_devmem_dmabuf_binding_free(struct work_struct *wq); struct net_devmem_dmabuf_binding * net_devmem_bind_dmabuf(struct net_device *dev, enum dma_data_direction direction, @@ -121,7 +123,8 @@ net_devmem_dmabuf_binding_put(struct net_devmem_dmabuf_= binding *binding) if (!refcount_dec_and_test(&binding->ref)) return; =20 - __net_devmem_dmabuf_binding_free(binding); + INIT_WORK(&binding->unbind_w, __net_devmem_dmabuf_binding_free); + schedule_work(&binding->unbind_w); } =20 void net_devmem_get_net_iov(struct net_iov *niov); @@ -154,8 +157,7 @@ static inline void net_devmem_put_net_iov(struct net_io= v *niov) { } =20 -static inline void -__net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *binding) +static inline void __net_devmem_dmabuf_binding_free(struct work_struct *wq) { } =20 --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FB3C1DDC14 for ; Thu, 20 Feb 2025 02:09:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017372; cv=none; b=ZvpdYd2HYrJSJhWybMUgIHv5soajCjqp8Q6APXOun4Sda2KJdLatRi6yDpkHrzJH5KPhuYuyOlvLjEUEhHMl9bGQnQWmd3CjLu1LniJxLuDT//Ab4ssF9kwPM96EghodRcKL1odyKFuRpkcDsyHtieopcA75Do8uQqzCdgmTs8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017372; c=relaxed/simple; bh=q5XsmN0+qNQcTTcfaJ4MRPPi7x8kiGcH8m4E8CxyZWo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FWJ0Dd/HCu0FcxukMYMIP9tcB5ZHRSUWHJ6VLiidMFQY9JMgchxm+vQA+tapJ/5Jc6lvJQZRjAmkUol3/ikIwPI/hXwsp5gYXuzfmqSAoM0VpFmFL5SxSWeLaTIKuJNfm61+n65LyAlIuDSAFjTArF7h+1/w0jmcv9Wjg0MRLGI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=H+Pe0RO9; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="H+Pe0RO9" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220ec5c16e9so11024345ad.1 for ; Wed, 19 Feb 2025 18:09:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017369; x=1740622169; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9i5ITI5hsYXNkk95ANyYz00PWaNA3wxaXmmfqorxWmI=; b=H+Pe0RO9SXEt2yoQXi3IeYHdYopwcJsr3ByJ3xKi76c/eFSovzJjpsO4Jbi1l+BInl b4nPPck/EI3cCo7HQ88Ml8niXEWjoS9t1WABh7EITKGnTXJqAXiVDB89KY+hX+MVZ9Ar yxp37z+l9hLOlSv547GKIftyWeXfX28IcZ2jiPb51Sn6tVuUQN/ptV8ZeRdogPJiH4E0 brkEGnaR/+ZFejIfxhUP/F9yCms9UxPXVqECTovw75kbwo6b82hGFfT6S+T8YIXP7J3p 6LSFkVERGgLEo4Srk8QkCoPVAVk64Nd0/dAiTCnnlDXn75zk2nafzunnkGMOQwX+AmQm 05+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017369; x=1740622169; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9i5ITI5hsYXNkk95ANyYz00PWaNA3wxaXmmfqorxWmI=; b=Y3BsbIzr55w5T1SL+doLVaLFBpbi9ekY+74mIbIowKmu4ofaTx7sPe4my/Ujfx7oyg SghsQ/PZtGKcq2ojaEP0GEH4S5rh7HgYQCtrb+ghLIJmAWUt053+qLXiZlRdwK100gyN T3sTVd5I/GDI79drJS7gI+6S5YHrz5QhW3oCxXPtZAAQqZL3CL/gX5xk3wmxJQB0mxbG bryBO+zYBC0ru9uIhK4gzl7ofT7JDI5LjkBL5J+I2kW3ze/QYIvmXbxByeTK6NXK6Gpd 4jWE9Wn8LRYaHSFeFWH3X8RMoKyqf0zTlx5rT0Cb9mVBz7D3Q8Aq1VoUWaLORUiAd82d wy/g== X-Forwarded-Encrypted: i=1; AJvYcCVXSinGAyJC/O50bb1uOWDW7Q453dmiwPi7/GLLD8vIzh062YugRDw7g1hltxZ63TVcYqPKIE/FKx3z1KQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yz3ZFK2GxXGWP4qFQUlCvPRwkXxJ9yMKKmewZ3i04AvwB8Uz1lE NvnDWNnm3B/g7cFj9r2Rg/jU9s9fzId66U1i0wzAxRB+Svptf/8FO+Udbwld9NCkfaIYLQ8jkGr w8Os9dOEeDHRdQ5t7wHJLiQ== X-Google-Smtp-Source: AGHT+IFWUpLngVjg4ZRvdMcFQlYPunbjfg+CrCWS1N/fYD0oezyFMNiTwHerTLo6whLbQg+u/NAp0xpJYUvUoObtoQ== X-Received: from plqw12.prod.google.com ([2002:a17:902:a70c:b0:21f:4d14:8384]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e54e:b0:220:ff3f:6cc5 with SMTP id d9443c01a7336-221040bf806mr344999995ad.35.1740017369127; Wed, 19 Feb 2025 18:09:29 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:10 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-6-almasrymina@google.com> Subject: [PATCH net-next v4 5/9] net: add devmem TCP TX documentation From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation outlining the usage and details of the devmem TCP TX API. Signed-off-by: Mina Almasry Acked-by: Stanislav Fomichev --- v4: - Mention SO_BINDTODEVICE is recommended (me/Pavel). v2: - Update documentation for iov_base is the dmabuf offset (Stan) --- Documentation/networking/devmem.rst | 150 +++++++++++++++++++++++++++- 1 file changed, 146 insertions(+), 4 deletions(-) diff --git a/Documentation/networking/devmem.rst b/Documentation/networking= /devmem.rst index d95363645331..10928a5f912f 100644 --- a/Documentation/networking/devmem.rst +++ b/Documentation/networking/devmem.rst @@ -62,15 +62,15 @@ More Info https://lore.kernel.org/netdev/20240831004313.3713467-1-almasrymina@go= ogle.com/ =20 =20 -Interface -=3D=3D=3D=3D=3D=3D=3D=3D=3D +RX Interface +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 =20 Example ------- =20 -tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setti= ng up -the RX path of this API. +./tools/testing/selftests/drivers/net/hw/ncdevmem:do_server shows an examp= le of +setting up the RX path of this API. =20 =20 NIC Setup @@ -235,6 +235,148 @@ can be less than the tokens provided by the user in c= ase of: (a) an internal kernel leak bug. (b) the user passed more than 1024 frags. =20 +TX Interface +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + +Example +------- + +./tools/testing/selftests/drivers/net/hw/ncdevmem:do_client shows an examp= le of +setting up the TX path of this API. + + +NIC Setup +--------- + +The user must bind a TX dmabuf to a given NIC using the netlink API:: + + struct netdev_bind_tx_req *req =3D NULL; + struct netdev_bind_tx_rsp *rsp =3D NULL; + struct ynl_error yerr; + + *ys =3D ynl_sock_create(&ynl_netdev_family, &yerr); + + req =3D netdev_bind_tx_req_alloc(); + netdev_bind_tx_req_set_ifindex(req, ifindex); + netdev_bind_tx_req_set_fd(req, dmabuf_fd); + + rsp =3D netdev_bind_tx(*ys, req); + + tx_dmabuf_id =3D rsp->id; + + +The netlink API returns a dmabuf_id: a unique ID that refers to this dmabuf +that has been bound. + +The user can unbind the dmabuf from the netdevice by closing the netlink s= ocket +that established the binding. We do this so that the binding is automatica= lly +unbound even if the userspace process crashes. + +Note that any reasonably well-behaved dmabuf from any exporter should work= with +devmem TCP, even if the dmabuf is not actually backed by devmem. An exampl= e of +this is udmabuf, which wraps user memory (non-devmem) in a dmabuf. + +Socket Setup +------------ + +The user application must use MSG_ZEROCOPY flag when sending devmem TCP. D= evmem +cannot be copied by the kernel, so the semantics of the devmem TX are simi= lar +to the semantics of MSG_ZEROCOPY. + + setsockopt(socket_fd, SOL_SOCKET, SO_ZEROCOPY, &opt, sizeof(opt)); + +It is also recommended that the user binds the TX socket to the same inter= face +the dma-buf has been bound to via SO_BINDTODEVICE. + + setsockopt(socket_fd, SOL_SOCKET, SO_BINDTODEVICE, ifname, strlen(ifname)= + 1); + + +Sending data +-------------- + +Devmem data is sent using the SCM_DEVMEM_DMABUF cmsg. + +The user should create a msghdr where, + +iov_base is set to the offset into the dmabuf to start sending from. +iov_len is set to the number of bytes to be sent from the dmabuf. + +The user passes the dma-buf id to send from via the dmabuf_tx_cmsg.dmabuf_= id. + +The example below sends 1024 bytes from offset 100 into the dmabuf, and 20= 48 +from offset 2000 into the dmabuf. The dmabuf to send from is tx_dmabuf_id:: + + char ctrl_data[CMSG_SPACE(sizeof(struct dmabuf_tx_cmsg))]; + struct dmabuf_tx_cmsg ddmabuf; + struct msghdr msg =3D {}; + struct cmsghdr *cmsg; + struct iovec iov[2]; + + iov[0].iov_base =3D (void*)100; + iov[0].iov_len =3D 1024; + iov[1].iov_base =3D (void*)2000; + iov[1].iov_len =3D 2048; + + msg.msg_iov =3D iov; + msg.msg_iovlen =3D 2; + + msg.msg_control =3D ctrl_data; + msg.msg_controllen =3D sizeof(ctrl_data); + + cmsg =3D CMSG_FIRSTHDR(&msg); + cmsg->cmsg_level =3D SOL_SOCKET; + cmsg->cmsg_type =3D SCM_DEVMEM_DMABUF; + cmsg->cmsg_len =3D CMSG_LEN(sizeof(struct dmabuf_tx_cmsg)); + + ddmabuf.dmabuf_id =3D tx_dmabuf_id; + + *((struct dmabuf_tx_cmsg *)CMSG_DATA(cmsg)) =3D ddmabuf; + + sendmsg(socket_fd, &msg, MSG_ZEROCOPY); + + +Reusing TX dmabufs +------------------ + +Similar to MSG_ZEROCOPY with regular memory, the user should not modify the +contents of the dma-buf while a send operation is in progress. This is bec= ause +the kernel does not keep a copy of the dmabuf contents. Instead, the kernel +will pin and send data from the buffer available to the userspace. + +Just as in MSG_ZEROCOPY, the kernel notifies the userspace of send complet= ions +using MSG_ERRQUEUE:: + + int64_t tstop =3D gettimeofday_ms() + waittime_ms; + char control[CMSG_SPACE(100)] =3D {}; + struct sock_extended_err *serr; + struct msghdr msg =3D {}; + struct cmsghdr *cm; + int retries =3D 10; + __u32 hi, lo; + + msg.msg_control =3D control; + msg.msg_controllen =3D sizeof(control); + + while (gettimeofday_ms() < tstop) { + if (!do_poll(fd)) continue; + + ret =3D recvmsg(fd, &msg, MSG_ERRQUEUE); + + for (cm =3D CMSG_FIRSTHDR(&msg); cm; cm =3D CMSG_NXTHDR(&m= sg, cm)) { + serr =3D (void *)CMSG_DATA(cm); + + hi =3D serr->ee_data; + lo =3D serr->ee_info; + + fprintf(stdout, "tx complete [%d,%d]\n", lo, hi); + } + } + +After the associated sendmsg has been completed, the dmabuf can be reused = by +the userspace. + + Implementation & Caveats =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B98BE1DEFFC for ; Thu, 20 Feb 2025 02:09:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017373; cv=none; b=NIKE855PJ3grzhpEeBQRL9JqtqEUMzu68fIBEJLXbpX8iaDkEIshcDmJ5ZMkzYXvCEuBRihYWNSOoamEeyh1rL2FQUz8DBmAAciN0J/Hnvnr0b6GSb4B53n2/WLCxDDU+07VRo+NoeyEsS77Pvqx7jA1NuXSkXxVWNDx85e64mw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017373; c=relaxed/simple; bh=yrZLNnExJxovnUdP+p9+TWBfjC0FJjXK8+7R08yWg8A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Z6XHwuVLFEWTq5MtBteLDQChPM9WBQDGAeo+G8f3xqX2JiifnydZ2qZXq3qQlTN0h8vK3QVsvpacvY66+gMbJ0iy+MlI5xzp0zHMycojlYuqUz4UOHw/gcXTrsczHzMtwPQxaXdGa1ECJ2oDbZqPtCVtp8ld1tOkL6w6mGcWbKA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MF3Z2Vbx; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MF3Z2Vbx" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-221063a808dso8107295ad.1 for ; Wed, 19 Feb 2025 18:09:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017371; x=1740622171; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JqtBs71AQ1fXytwcggDtxu27G/HnTgPlNN4begvh5Pw=; b=MF3Z2VbxSO2nxZfrhNLHe8uLD7Kj+2mvwd5GhbxRG4acgLPf73fcpOhL4GMxhr84sS eGtyHprEQolJDYUG/Z3dYwLX4Iiyp3mk+DCe2eC5H+eEEr4wCdH8jcroNfvr1rvw/8oq 5tO8sbA7FtBT0ycd771ujTp2ytbCHtS5FieeTz6x6e3ZjNQVE+LPO1aDUMaorLk1edlb YmRC3ov1O3ZqRcxu3ndxguw+7P4Tz1b0WwpNG+H0OuSNnvn8g0HGYvgPzZYy4nm1rTnH e6kxdIar8V1wKeSlATSCclddWfdDYpybZSPk08yelQPk22XzOGMtAcZ61ykdCDVQCN2m 6h3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017371; x=1740622171; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JqtBs71AQ1fXytwcggDtxu27G/HnTgPlNN4begvh5Pw=; b=b4HsqI49NLmmDeMw9opnlqCez3i6NeTZ4t3tXg66c2SXJnKnJG97APZCOjUS2ObnED OhB884NKtr6TdplccRHNZSAt4pPJAOj+nh0Z7Qzr5uKCi2iCEcjF4E2MUTaKvOy5G+Sb KTp9D4RL6nQIO+qkbFuWYu4ELFluksQ4Rcq4CV4MemgrvqLXoPfPy0c+w8P5T5J6+95I I/1cK0Bm0MVwAjq9nosNg8XW/aJKo29uMA0Rlri49LGaeppgsThAFd88WrkW0ow1rqpe svPdnGlhiUobJhAs6rhG8zvVtsTWh0COQjL1d1htULLObQTbRWFl2ruDN7o6FHWLmFxw 5xAw== X-Forwarded-Encrypted: i=1; AJvYcCU9QeHBBgPlaI4iaup78YybmZ1b4CDltjAi8DR3tgoSa/+mkgArgcGR3dcB/7KZ9WTaF2d4dVijS0vQmuY=@vger.kernel.org X-Gm-Message-State: AOJu0Yx5nDuUfBQEpexZB9w8RckW0y+E84ZdihqoGmGZGVw4KDNWeyNO dnA7eUbfD8Q24TpI2QHa2bWmTRj34peF0j95+X8/Vu4NaV07Qi7jZBa6TdZyzlSA5+dLt4aVEJ6 qxgbgGVdBBMdIsVjcwG7ACA== X-Google-Smtp-Source: AGHT+IFn1g88SI+wrdiAJ0YpCgRsTA/V0ytuOY7Zv6OeUzLOT3etpr7SPdmkn/51N/sBBJUmhnJLyO9fltZLHVilIg== X-Received: from pfblc1.prod.google.com ([2002:a05:6a00:4f41:b0:731:43c2:88e3]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:62c1:b0:1ee:a914:1d64 with SMTP id adf61e73a8af0-1eee5d694aamr1202693637.28.1740017370945; Wed, 19 Feb 2025 18:09:30 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:11 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-7-almasrymina@google.com> Subject: [PATCH net-next v4 6/9] net: enable driver support for netmem TX From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drivers need to make sure not to pass netmem dma-addrs to the dma-mapping API in order to support netmem TX. Add helpers and netmem_dma_*() helpers that enables special handling of netmem dma-addrs that drivers can use. Document in netmem.rst what drivers need to do to support netmem TX. Signed-off-by: Mina Almasry --- v4: - New patch --- .../networking/net_cachelines/net_device.rst | 1 + Documentation/networking/netdev-features.rst | 5 +++++ Documentation/networking/netmem.rst | 14 +++++++++++-- include/linux/netdevice.h | 2 ++ include/net/netmem.h | 20 +++++++++++++++++++ 5 files changed, 40 insertions(+), 2 deletions(-) diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Docum= entation/networking/net_cachelines/net_device.rst index 15e31ece675f..e3043b033647 100644 --- a/Documentation/networking/net_cachelines/net_device.rst +++ b/Documentation/networking/net_cachelines/net_device.rst @@ -10,6 +10,7 @@ Type Name = fastpath_tx_acce =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D unsigned_long:32 priv_flags read_mostl= y __dev_queue_xmit(tx) unsigned_long:1 lltx read_mostl= y HARD_TX_LOCK,HARD_TX_TRYLOCK,HARD_TX_UNLOCK(t= x) +unsigned long:1 netmem_tx:1; read_mostly char name[16] struct netdev_name_node* name_node struct dev_ifalias* ifalias diff --git a/Documentation/networking/netdev-features.rst b/Documentation/n= etworking/netdev-features.rst index 5014f7cc1398..02bd7536fc0c 100644 --- a/Documentation/networking/netdev-features.rst +++ b/Documentation/networking/netdev-features.rst @@ -188,3 +188,8 @@ Redundancy) frames from one port to another in hardware. This should be set for devices which duplicate outgoing HSR (High-availabi= lity Seamless Redundancy) or PRP (Parallel Redundancy Protocol) tags automatica= lly frames in hardware. + +* netmem-tx + +This should be set for devices which support netmem TX. See +Documentation/networking/netmem.rst diff --git a/Documentation/networking/netmem.rst b/Documentation/networking= /netmem.rst index 7de21ddb5412..43054d44c407 100644 --- a/Documentation/networking/netmem.rst +++ b/Documentation/networking/netmem.rst @@ -19,8 +19,8 @@ Benefits of Netmem : * Simplified Development: Drivers interact with a consistent API, regardless of the underlying memory implementation. =20 -Driver Requirements -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Driver RX Requirements +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 1. The driver must support page_pool. =20 @@ -77,3 +77,13 @@ Driver Requirements that purpose, but be mindful that some netmem types might have longer circulation times, such as when userspace holds a reference in zerocopy scenarios. + +Driver TX Requirements +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +1. Driver should use netmem_dma_unmap_page_attrs() in lieu of + dma_unmap_page[_attrs](), and netmem_dma_unmap_addr_set() in lieu of + dma_unmap_addr_set(). The netmem variants will handle netmems that shou= ld + not be dma-unmapped by the driver, such as dma-buf netmems. + +2. Driver should declare support by setting `netdev->netmem_tx =3D true` diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index fccc03cd2164..d8cfd5d69ddf 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1753,6 +1753,7 @@ enum netdev_reg_state { * @lltx: device supports lockless Tx. Deprecated for real HW * drivers. Mainly used by logical interfaces, such as * bonding and tunnels + * @netmem_tx: device support netmem_tx. * * @name: This is the first field of the "visible" part of this structure * (i.e. as seen by users in the "Space.c" file). It is the name @@ -2061,6 +2062,7 @@ struct net_device { struct_group(priv_flags_fast, unsigned long priv_flags:32; unsigned long lltx:1; + unsigned long netmem_tx:1; ); const struct net_device_ops *netdev_ops; const struct header_ops *header_ops; diff --git a/include/net/netmem.h b/include/net/netmem.h index a2148ffb203d..1fb39ad63290 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -8,6 +8,7 @@ #ifndef _NET_NETMEM_H #define _NET_NETMEM_H =20 +#include #include #include =20 @@ -267,4 +268,23 @@ static inline unsigned long netmem_get_dma_addr(netmem= _ref netmem) void get_netmem(netmem_ref netmem); void put_netmem(netmem_ref netmem); =20 +#define netmem_dma_unmap_addr_set(NETMEM, PTR, ADDR_NAME, VAL) \ + do { \ + if (!netmem_is_net_iov(NETMEM)) \ + dma_unmap_addr_set(PTR, ADDR_NAME, VAL); \ + else \ + dma_unmap_addr_set(PTR, ADDR_NAME, 0); \ + } while (0) + +static inline void netmem_dma_unmap_page_attrs(struct device *dev, + dma_addr_t addr, size_t size, + enum dma_data_direction dir, + unsigned long attrs) +{ + if (!addr) + return; + + dma_unmap_page_attrs(dev, addr, size, dir, attrs); +} + #endif /* _NET_NETMEM_H */ --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 734561E0DD8 for ; Thu, 20 Feb 2025 02:09:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017376; cv=none; b=C8S28UTNYe5rlIVp2YMTXiwt4YsbvPdbZub+E5PabVUPvCfBIXX1ifHoBhobAESMJaS81jYMOXT40/3w4AvFdZWtCPCTqzd+Qtw0WQ3bFo3FDZJ6Z6+7CDp82Hc86EtbyCs0sEj6pYm0eA30f/mRmxWT8/ah+AYSbQiXvUkSWfo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017376; c=relaxed/simple; bh=4a9/ASNRGrjKbqFRJBcsMKUl4XJqLT98iBtNZ9Vd0eg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Q+XHCMUaavtTumuZ20HkOjbThun0OIYDGRakYgrxi8RFj6LMpxyOkC9gv2HitQvBbUfEh7L42T07BtO3aWWBsFMBTTirQA9JXXM21gB/LHaPoNiGHcuvN/HD2DyPsgKp6VTZx2oC1B0ITD6hSf7JtCQjIaD6pgTfJTSxwlLREek= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LYWUIpWO; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LYWUIpWO" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2fc318bd470so927985a91.0 for ; Wed, 19 Feb 2025 18:09:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017373; x=1740622173; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LRh16gD1ltrznZeRyUXa85QXYB+P/DDrWPKAZGQNy08=; b=LYWUIpWOA+GSjcfhNWosq6YsKO6xn94nUNB2qNY/Bf2MEjcpJqZQalEpBJjP6JTUZz /fkD7IcNz8Pz/0L1X7kY96dXmkuAT3sWlQfFd1SFycyQ83huN31iEUh3WD99wVl3WFba kI3k6h0W3Jgu7Z8mrUYc6ty493HCVTXz2YE/huK72Rv5oh6Z0J11OwTAeeWXI6suK4DW DE95YxZqgHfvSqbEfhNeDN5uImEhYlKh1hVe28G46NDrfky6LiNcQcioZYbC2M7mHJ+0 E8rAu4xR9d/v9xH3ODM2RL1rAO3OBcpeaRv+0X4cPwKiHE+zb93Kb27+9EHPbQtbZIam Q6/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017373; x=1740622173; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LRh16gD1ltrznZeRyUXa85QXYB+P/DDrWPKAZGQNy08=; b=rDYDS/ZUYwewo5IpOhaITWaZY+RkISi3YgzfFBewiLROyQVbyAyV5OMAE7ntGIQaLD IhhPlIKQ9h0/T2iC/2dGk7zPU0Gr2/lt6Ud4TwClHFlYdAJrid18zkzrETAYRLYphvhw NtxymKHwV+jZzpVdyg4L3RbQ+hyPE2uzeyw/bC9aJVj6P0SFpUQJ7iNrj0rRnB9KOeAP b/QC/lJgekO8ej0OPWuM75XrGvekLULdsASqUhntmmoq/YI8quMmMaBMhB8cE279vGPJ LNUgOTSUxBT58nKjqsFBwzk8aelBGkKydTWnQSTRMrNnKTWqa2VgSplMtCsIyCllGvZc +opQ== X-Forwarded-Encrypted: i=1; AJvYcCX78C7TGbEjOl4b3KQ78WvRB8adFOEjvQyhEoFrKZULHTDwlsA144ngKfvke/COnDUnPe5r0/HtzX+nCYk=@vger.kernel.org X-Gm-Message-State: AOJu0Yx7D+OHC+uf8/Pk0cetU10id2lnS/ImEHiwe2ZSXAM4OlK5LTK+ 4rvAxzxb3drGkBAFUZ+ScOXuAsRGJemfXScWlCCifEygR89W6ejOFl+M+i6AjGrc2W+Esaq/Xph 1gy6w5p+7RhLRQI6o5e/jWg== X-Google-Smtp-Source: AGHT+IFv7gdFT+Y+75bNV/cPnBGsSoo+TGhAr3nWoydKdBJ94MrpNFek3G2h7TMlX0xSNhdiJ1pw2YWAResKG+nBaQ== X-Received: from pjtd9.prod.google.com ([2002:a17:90b:49:b0:2ef:78ff:bc3b]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2710:b0:2ee:d371:3227 with SMTP id 98e67ed59e1d1-2fcd0ca3ad6mr1322765a91.17.1740017372717; Wed, 19 Feb 2025 18:09:32 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:12 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-8-almasrymina@google.com> Subject: [PATCH net-next v4 7/9] gve: add netmem TX support to GVE DQO-RDA mode From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use netmem_dma_*() helpers in gve_tx_dqo.c DQO-RDA paths to enable netmem TX support in that mode. Declare support for netmem TX in GVE DQO-RDA mode. Signed-off-by: Mina Almasry --- v4: - New patch --- drivers/net/ethernet/google/gve/gve_main.c | 4 ++++ drivers/net/ethernet/google/gve/gve_tx_dqo.c | 8 +++++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ether= net/google/gve/gve_main.c index 533e659b15b3..5047d4e260fb 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -2731,6 +2731,10 @@ static int gve_probe(struct pci_dev *pdev, const str= uct pci_device_id *ent) =20 dev_info(&pdev->dev, "GVE version %s\n", gve_version_str); dev_info(&pdev->dev, "GVE queue format %d\n", (int)priv->queue_format); + + if (!gve_is_gqi(priv) && !gve_is_qpl(priv)) + dev->netmem_tx =3D true; + gve_clear_probe_in_progress(priv); queue_work(priv->gve_wq, &priv->service_task); return 0; diff --git a/drivers/net/ethernet/google/gve/gve_tx_dqo.c b/drivers/net/eth= ernet/google/gve/gve_tx_dqo.c index 394debc62268..e74580dc7ebe 100644 --- a/drivers/net/ethernet/google/gve/gve_tx_dqo.c +++ b/drivers/net/ethernet/google/gve/gve_tx_dqo.c @@ -667,7 +667,8 @@ static int gve_tx_add_skb_no_copy_dqo(struct gve_tx_rin= g *tx, goto err; =20 dma_unmap_len_set(pkt, len[pkt->num_bufs], len); - dma_unmap_addr_set(pkt, dma[pkt->num_bufs], addr); + netmem_dma_unmap_addr_set(skb_frag_netmem(frag), pkt, + dma[pkt->num_bufs], addr); ++pkt->num_bufs; =20 gve_tx_fill_pkt_desc_dqo(tx, desc_idx, skb, len, addr, @@ -1045,8 +1046,9 @@ static void gve_unmap_packet(struct device *dev, dma_unmap_single(dev, dma_unmap_addr(pkt, dma[0]), dma_unmap_len(pkt, len[0]), DMA_TO_DEVICE); for (i =3D 1; i < pkt->num_bufs; i++) { - dma_unmap_page(dev, dma_unmap_addr(pkt, dma[i]), - dma_unmap_len(pkt, len[i]), DMA_TO_DEVICE); + netmem_dma_unmap_page_attrs(dev, dma_unmap_addr(pkt, dma[i]), + dma_unmap_len(pkt, len[i]), + DMA_TO_DEVICE, 0); } pkt->num_bufs =3D 0; } --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA2611E3774 for ; Thu, 20 Feb 2025 02:09:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017377; cv=none; b=dqEH532R0f8YZk+omuuJtG9Pu493Ek8dSQIvMXc3aUQBH0gLDBwwYaDrG6oulSx80Q250gkjj0KwUY6WTZ+nfHUun6dvx3mvK3QMKkRoKOWHYUem/0DAGQBemB0PI0yK3A7YtjMa8IASJpQIkzlwXupinP79bopADxlW1DRIwDU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017377; c=relaxed/simple; bh=xktwRXpJMJv986Qo4eDPxj726JXR+3eKG4eS+/XdbOE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IZhOHkoXcdi8eV86ekBVIB6nfIHuS4tj1u3yZAHyvSVHN1Nt4rhMS/olol2tLl+VQuB9vSRLpfQEhzJpbkSPv0TIgPr3H2fdAYorpZTBLWqnkeeeW3ATPTj/jEMFmepcXisaAbY5WFEAUkIx/IB6ZhsPQPQUlcRtjPiGKaKyFQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=O8lEYp+B; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="O8lEYp+B" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f83e54432dso1464268a91.2 for ; Wed, 19 Feb 2025 18:09:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017374; x=1740622174; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ouoTtMMhd3rb00vY+DSWvls3bj0wfOCkxx6qaQ7XOwM=; b=O8lEYp+B+GVf5CBtc+j9zd8esl5qp9PbFOIlFXkgRUQrAGZZrELqla2NTPg2aK3emc sl3cy3BV5y91hr5LK3qG+VwHZwZ2d4PjgEuHdJ1NGrU5WZ07DfNEmLdF6izUbgNRSiH+ 0YEb/8yk+ZX8vKRG5Dimu7PlTeE6MPmgMxqvqWlhwum6FXBiPgS0DFvTK9raF9wguRHu 1KbRNK0uKflT++GpxkxUaRep2pIYBQbr6A9L6xvl1dI7LngmEsmYYE2Z8hbWmR6xEp9m WtCI4gzjWX14ITRR8T/v0SDRP5fH/YhfbxFwKkPGmqMrS8cTNR9F5y16RwsDTHJqYXV1 r3wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017374; x=1740622174; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ouoTtMMhd3rb00vY+DSWvls3bj0wfOCkxx6qaQ7XOwM=; b=ujsLlPfXCkp4oO6iKddYXDhFnBN0SrX8330Q1FNkKmid+URiVQZQTRWFHmM4eVSETU 8LKmqb7FxOGZqJyg6SnTPUvbC3Uq6dLMhB/kOV89SWWtlCs/WNAW31BAVeY3eNnMX558 7IznP+Ib4BRjj7oNAjIjCWiBt/9wLDYKgT/zva0GaGC3v2mcWsXBATGOTpqByR8PPRqE 8uTM4I5Q/TBXeS8MeVbhJ35mWtBy+s5ppL/6iC/NLdBwm8kIe/wbjxTl/Jet7hbMSjY6 FRV1ZOLbi+XwFNq7mJNQfXWfckocOiE2Q2kYk6xrz3dHnGSCDDhlKZtRUr3jsGkwUT+w BzLA== X-Forwarded-Encrypted: i=1; AJvYcCXQTTVxFjHo06iGW6Pi+LuP2R6mIyZZ/Kfai227HObxFYMcoWhKckLwq6J4kOEMzfXj54FUoe8trqz4e4Q=@vger.kernel.org X-Gm-Message-State: AOJu0YxE9vY4x9gdKvrxELuqWGBfD0wztVXKjf3RtpDfX9K/fmtQwm71 Qf2SAaoJBcDA80JVJMnKUonuYKFH2XUnQQm59HMdowidH8N7KxWsO2u9RSFBWcREjnHFV78dcue UMOsycyDbdpqw5kRERmacug== X-Google-Smtp-Source: AGHT+IE63Y1j485C5r+SG910yHgU7D0EniRUvxRKp0z0ZOdOvF2csyUI5aFQVIvnQkQW3zk2gmV0Mv7QAnOs+6jCLQ== X-Received: from pjc8.prod.google.com ([2002:a17:90b:2f48:b0:2eb:12d7:fedd]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5202:b0:2fc:3264:3657 with SMTP id 98e67ed59e1d1-2fcb56f6d59mr11204857a91.0.1740017374316; Wed, 19 Feb 2025 18:09:34 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:13 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-9-almasrymina@google.com> Subject: [PATCH net-next v4 8/9] net: check for driver support in netmem TX From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We should not enable netmem TX for drivers that don't declare support. Check for driver netmem TX support during devmem TX binding and fail if the driver does not have the functionality. Check for driver support in validate_xmit_skb as well. Signed-off-by: Mina Almasry Acked-by: Stanislav Fomichev --- v4: - New patch --- net/core/dev.c | 3 +++ net/core/netdev-genl.c | 7 +++++++ 2 files changed, 10 insertions(+) diff --git a/net/core/dev.c b/net/core/dev.c index ebc000b56828..f65a2b41a2c3 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3850,6 +3850,9 @@ static struct sk_buff *validate_xmit_skb(struct sk_bu= ff *skb, struct net_device =20 skb =3D validate_xmit_xfrm(skb, features, again); =20 + if (!skb_frags_readable(skb) && !dev->netmem_tx) + goto out_kfree_skb; + return skb; =20 out_kfree_skb: diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 1faa2cf4057f..a154480a6e22 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -970,6 +970,13 @@ int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct= genl_info *info) goto err_unlock; } =20 + if (!netdev->netmem_tx) { + err =3D -EOPNOTSUPP; + NL_SET_ERR_MSG(info->extack, + "Driver does not support netmem TX"); + goto err_unlock; + } + binding =3D net_devmem_bind_dmabuf(netdev, DMA_TO_DEVICE, dmabuf_fd, info->extack); if (IS_ERR(binding)) { --=20 2.48.1.601.g30ceb7b040-goog From nobody Thu Dec 18 16:17:44 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACDF01E5B68 for ; Thu, 20 Feb 2025 02:09:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017379; cv=none; b=C21R7QZ5jb8gV6MYBv0LriVqQrgQTPd7PdZCgPeQScRQRsC4W+PoctNMSGsaTOoOtpiBz3i63HIEcpEdSDOkKyfGIRKgeMUJSH5k9aSv8Nfv/E2KnzdNW7zGu2w2V+tQVkg1q5FNZb03Du7l5dFqbTz4oIDWGmzSExXdP5PwT70= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740017379; c=relaxed/simple; bh=KW7ywZIovk8KK3anzRCr7OWEkROJ21Oe0C4npKLi61Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WXnc8r5dKAZarHnunF8xbaROCR0VLV+53Z9qbdhct77e2loiZx8Zl3PVPvGqCcBBgC9wleApEycfRQoYt2uHx2KW1a9wChV22Ri5NLoFel2uTz+dWNH7thAFNfPMRI1rKlgGs1kGqkga7a+4zhj8SE+fdxkqjKn4ThskI8hXySA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PnaTIvg7; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PnaTIvg7" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f83e54432dso1464351a91.2 for ; Wed, 19 Feb 2025 18:09:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740017376; x=1740622176; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OwTNnDuIp7+dZm+s90yId9LeUwn8LrPW6TNoPj9JUgY=; b=PnaTIvg7cNqokoHPp1nHHTiZQBrHXldzvaiS9IjZ8O580lDujNyPkeC+CXq0WwwRH0 W6l7DWA4TJPZORGncYKP7is+eTSi5hwYTLOS4lynRcciERuDH4A6jTK8vJWIicd8qoZB eyyggx2LdF1/OEmlUgxnimjR9LyvlXuYFniUdNrG1T1tO2RJUOxcfRO9tDGr1Ph6gQzM OkRiFqnd/uTBN6THeqfURM50flmm7Lfv10TlE9O1aYIvdFSoXw03DvYTtjqWJJqjjYnf 2hl/+sj3DbnlaERca8lNUUDIz6UZzWEiigxM/D/08nSSe+JeiUgMcajF7B54u9rQfz4F 45OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740017376; x=1740622176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OwTNnDuIp7+dZm+s90yId9LeUwn8LrPW6TNoPj9JUgY=; b=o+i7J52P5MfFcUttJD/2Mas8JXNPuX6hchSTeAvZtg5TR2U6IeYbiSmMxuM+ygsrG5 SmhSTwesPr4KFs4tDbBlmz0jMitBFgvhc2leHppWRyAQvHColFL0kIuODF6z8ukhsegG Htm8zgKoY4XD7USli1kG2ch9H/Q9+Srv+PgcY6q3BlkWsZDTI/SOIbfK4UVZVh2ptRcB hlqSYH5Nd+xrr5XZvVBLV2tp3RWiLuxpruoUDxrrqJGHL+oy0q5a4pdQQt+GdB1fI+W+ fD37aASdN1B9Loac0jR7VGY5XXlgyn4ZhyTEILu0kCDbRtdfy7jD0kQREq5lVrZyGKrB mv2g== X-Forwarded-Encrypted: i=1; AJvYcCWAxvF9xMrsRlJHelTUg73LLiWq+4nlsuy8XdzLydJQIJWEIeYDqPMXVpY9LigeZ77DcH63RO5MnUKiYVQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyUCnGc6orzRvT6yrKWkqToMPVQ7g7oKyiT7PyqWpeNVAfgeUaZ /TWd5cweJz7GejcbNSEANMHHDzgtCiFRS5o8zfDZmCEoEKl0FnAl/LJy+U/z6AsBtCUi1ajeJw+ Qbylxmdc1iyg8dcETmeIpfA== X-Google-Smtp-Source: AGHT+IFCRLz4Z8eiD34P83PX1+Xt49H/cyNBLFPk8ItuTE5m22IqP+udpBDJ7l1MK7g6R9HpPc/2flvEMovSnZF60g== X-Received: from pjbsy7.prod.google.com ([2002:a17:90b:2d07:b0:2ea:4a74:ac2]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3b88:b0:2ee:dcf6:1c8f with SMTP id 98e67ed59e1d1-2fcb5a3a23amr10206822a91.16.1740017376114; Wed, 19 Feb 2025 18:09:36 -0800 (PST) Date: Thu, 20 Feb 2025 02:09:14 +0000 In-Reply-To: <20250220020914.895431-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250220020914.895431-1-almasrymina@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250220020914.895431-10-almasrymina@google.com> Subject: [PATCH net-next v4 9/9] selftests: ncdevmem: Implement devmem TCP TX From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Donald Hunter , Jonathan Corbet , Andrew Lunn , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Kuniyuki Iwashima , Willem de Bruijn , David Ahern , Neal Cardwell , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Stefan Hajnoczi , Stefano Garzarella , Shuah Khan , sdf@fomichev.me, asml.silence@gmail.com, dw@davidwei.uk, Jamal Hadi Salim , Victor Nogueira , Pedro Tammela , Samiullah Khawaja Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add support for devmem TX in ncdevmem. This is a combination of the ncdevmem from the devmem TCP series RFCv1 which included the TX path, and work by Stan to include the netlink API and refactored on top of his generic memory_provider support. Signed-off-by: Mina Almasry Signed-off-by: Stanislav Fomichev --- v4: - Add TX test to devmem.py (Paolo). v3: - Update ncdevmem docs to run validation with RX-only and RX-with-TX. - Fix build warnings (Stan). - Make the validation expect new lines in the pattern so we can have the TX path behave like netcat (Stan). - Change ret to errno in error() calls (Stan). - Handle the case where client_ip is not provided (Stan). - Don't assume mid is <=3D 2000 (Stan). v2: - make errors a static variable so that we catch instances where there are less than 20 errors across different buffers. - Fix the issue where the seed is reset to 0 instead of its starting value 1. - Use 1000ULL instead of 1000 to guard against overflow (Willem). - Do not set POLLERR (Willem). - Update the test to use the new interface where iov_base is the dmabuf_offset. - Update the test to send 2 iov instead of 1, so we get some test coverage over sending multiple iovs at once. - Print the ifindex the test is using, useful for debugging issues where maybe the test may fail because the ifindex of the socket is different from the dmabuf binding. --- .../selftests/drivers/net/hw/devmem.py | 28 +- .../selftests/drivers/net/hw/ncdevmem.c | 300 +++++++++++++++++- 2 files changed, 312 insertions(+), 16 deletions(-) diff --git a/tools/testing/selftests/drivers/net/hw/devmem.py b/tools/testi= ng/selftests/drivers/net/hw/devmem.py index 1223f0f5c10c..3d4f7fc5e63f 100755 --- a/tools/testing/selftests/drivers/net/hw/devmem.py +++ b/tools/testing/selftests/drivers/net/hw/devmem.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 =20 +from os import path from lib.py import ksft_run, ksft_exit from lib.py import ksft_eq, KsftSkipEx from lib.py import NetDrvEpEnv @@ -10,8 +11,7 @@ from lib.py import ksft_disruptive =20 def require_devmem(cfg): if not hasattr(cfg, "_devmem_probed"): - port =3D rand_port() - probe_command =3D f"./ncdevmem -f {cfg.ifname}" + probe_command =3D f"{cfg.bin_local} -f {cfg.ifname}" cfg._devmem_supported =3D cmd(probe_command, fail=3DFalse, shell= =3DTrue).ret =3D=3D 0 cfg._devmem_probed =3D True =20 @@ -25,18 +25,36 @@ def check_rx(cfg) -> None: require_devmem(cfg) =20 port =3D rand_port() - listen_cmd =3D f"./ncdevmem -l -f {cfg.ifname} -s {cfg.v6} -p {port}" + listen_cmd =3D f"{cfg.bin_local} -l -f {cfg.ifname} -s {cfg.v6} -p {po= rt}" =20 with bkg(listen_cmd) as socat: wait_port_listen(port) - cmd(f"echo -e \"hello\\nworld\"| socat -u - TCP6:[{cfg.v6}]:{port}= ", host=3Dcfg.remote, shell=3DTrue) + cmd(f"echo -e \"hello\\nworld\"| socat -u - TCP6:{cfg.v6}:{port},b= ind=3D{cfg.remote_v6}:{port}", host=3Dcfg.remote, shell=3DTrue) + + ksft_eq(ncdevmem.stdout.strip(), "hello\nworld") + + +@ksft_disruptive +def check_tx(cfg) -> None: + cfg.require_v6() + require_devmem(cfg) + + port =3D rand_port() + listen_cmd =3D f"socat -U - TCP6-LISTEN:{port},bind=3D{cfg.v6}" + + with bkg(listen_cmd) as socat: + wait_port_listen(port) + cmd(f"echo -e \"hello\\nworld\"| {cfg.bin_remote} -f {cfg.ifname} = -s {cfg.v6} -c {cfg.remote_v6} -p {port}", host=3Dcfg.remote, shell=3DTrue) =20 ksft_eq(socat.stdout.strip(), "hello\nworld") =20 =20 def main() -> None: with NetDrvEpEnv(__file__) as cfg: - ksft_run([check_rx], + cfg.bin_local =3D path.abspath(path.dirname(__file__) + "/ncdevmem= ") + cfg.bin_remote =3D cfg.remote.deploy(cfg.bin_local) + + ksft_run([check_rx, check_tx], args=3D(cfg, )) ksft_exit() =20 diff --git a/tools/testing/selftests/drivers/net/hw/ncdevmem.c b/tools/test= ing/selftests/drivers/net/hw/ncdevmem.c index 2bf14ac2b8c6..f801a1b3545f 100644 --- a/tools/testing/selftests/drivers/net/hw/ncdevmem.c +++ b/tools/testing/selftests/drivers/net/hw/ncdevmem.c @@ -9,22 +9,31 @@ * ncdevmem -s [-c ] -f eth1 -l -p 5201 * * On client: - * echo -n "hello\nworld" | nc -s 5201 -p 5201 + * echo -n "hello\nworld" | \ + * ncdevmem -s [-c ] -p 5201 -f eth1 * - * Test data validation: + * Note this is compatible with regular netcat. i.e. the sender or receive= r can + * be replaced with regular netcat to test the RX or TX path in isolation. + * + * Test data validation (devmem TCP on RX only): * * On server: * ncdevmem -s [-c ] -f eth1 -l -p 5201 -v 7 * * On client: * yes $(echo -e \\x01\\x02\\x03\\x04\\x05\\x06) | \ - * tr \\n \\0 | \ - * head -c 5G | \ + * head -c 1G | \ * nc 5201 -p 5201 * + * Test data validation (devmem TCP on RX and TX, validation happens on RX= ): * - * Note this is compatible with regular netcat. i.e. the sender or receive= r can - * be replaced with regular netcat to test the RX or TX path in isolation. + * On server: + * ncdevmem -s [-c ] -l -p 5201 -v 8 -f eth1 + * + * On client: + * yes $(echo -e \\x01\\x02\\x03\\x04\\x05\\x06\\x07) | \ + * head -c 1M | \ + * ncdevmem -s [-c ] -p 5201 -f eth1 */ #define _GNU_SOURCE #define __EXPORTED_HEADERS__ @@ -40,15 +49,18 @@ #include #include #include +#include =20 #include #include #include #include #include +#include =20 #include #include +#include #include #include #include @@ -79,6 +91,8 @@ static int num_queues =3D -1; static char *ifname; static unsigned int ifindex; static unsigned int dmabuf_id; +static uint32_t tx_dmabuf_id; +static int waittime_ms =3D 500; =20 struct memory_buffer { int fd; @@ -92,6 +106,8 @@ struct memory_buffer { struct memory_provider { struct memory_buffer *(*alloc)(size_t size); void (*free)(struct memory_buffer *ctx); + void (*memcpy_to_device)(struct memory_buffer *dst, size_t off, + void *src, int n); void (*memcpy_from_device)(void *dst, struct memory_buffer *src, size_t off, int n); }; @@ -152,6 +168,20 @@ static void udmabuf_free(struct memory_buffer *ctx) free(ctx); } =20 +static void udmabuf_memcpy_to_device(struct memory_buffer *dst, size_t off, + void *src, int n) +{ + struct dma_buf_sync sync =3D {}; + + sync.flags =3D DMA_BUF_SYNC_START | DMA_BUF_SYNC_WRITE; + ioctl(dst->fd, DMA_BUF_IOCTL_SYNC, &sync); + + memcpy(dst->buf_mem + off, src, n); + + sync.flags =3D DMA_BUF_SYNC_END | DMA_BUF_SYNC_WRITE; + ioctl(dst->fd, DMA_BUF_IOCTL_SYNC, &sync); +} + static void udmabuf_memcpy_from_device(void *dst, struct memory_buffer *sr= c, size_t off, int n) { @@ -169,6 +199,7 @@ static void udmabuf_memcpy_from_device(void *dst, struc= t memory_buffer *src, static struct memory_provider udmabuf_memory_provider =3D { .alloc =3D udmabuf_alloc, .free =3D udmabuf_free, + .memcpy_to_device =3D udmabuf_memcpy_to_device, .memcpy_from_device =3D udmabuf_memcpy_from_device, }; =20 @@ -187,14 +218,16 @@ void validate_buffer(void *line, size_t size) { static unsigned char seed =3D 1; unsigned char *ptr =3D line; - int errors =3D 0; + unsigned char expected; + static int errors; size_t i; =20 for (i =3D 0; i < size; i++) { - if (ptr[i] !=3D seed) { + expected =3D seed ? seed : '\n'; + if (ptr[i] !=3D expected) { fprintf(stderr, "Failed validation: expected=3D%u, actual=3D%u, index=3D%lu\n", - seed, ptr[i], i); + expected, ptr[i], i); errors++; if (errors > 20) error(1, 0, "validation failed."); @@ -393,6 +426,49 @@ static int bind_rx_queue(unsigned int ifindex, unsigne= d int dmabuf_fd, return -1; } =20 +static int bind_tx_queue(unsigned int ifindex, unsigned int dmabuf_fd, + struct ynl_sock **ys) +{ + struct netdev_bind_tx_req *req =3D NULL; + struct netdev_bind_tx_rsp *rsp =3D NULL; + struct ynl_error yerr; + + *ys =3D ynl_sock_create(&ynl_netdev_family, &yerr); + if (!*ys) { + fprintf(stderr, "YNL: %s\n", yerr.msg); + return -1; + } + + req =3D netdev_bind_tx_req_alloc(); + netdev_bind_tx_req_set_ifindex(req, ifindex); + netdev_bind_tx_req_set_fd(req, dmabuf_fd); + + rsp =3D netdev_bind_tx(*ys, req); + if (!rsp) { + perror("netdev_bind_tx"); + goto err_close; + } + + if (!rsp->_present.id) { + perror("id not present"); + goto err_close; + } + + fprintf(stderr, "got tx dmabuf id=3D%d\n", rsp->id); + tx_dmabuf_id =3D rsp->id; + + netdev_bind_tx_req_free(req); + netdev_bind_tx_rsp_free(rsp); + + return 0; + +err_close: + fprintf(stderr, "YNL failed: %s\n", (*ys)->err.msg); + netdev_bind_tx_req_free(req); + ynl_sock_destroy(*ys); + return -1; +} + static void enable_reuseaddr(int fd) { int opt =3D 1; @@ -431,7 +507,7 @@ static int parse_address(const char *str, int port, str= uct sockaddr_in6 *sin6) return 0; } =20 -int do_server(struct memory_buffer *mem) +static int do_server(struct memory_buffer *mem) { char ctrl_data[sizeof(int) * 20000]; struct netdev_queue_id *queues; @@ -685,6 +761,206 @@ void run_devmem_tests(void) provider->free(mem); } =20 +static uint64_t gettimeofday_ms(void) +{ + struct timeval tv; + + gettimeofday(&tv, NULL); + return (tv.tv_sec * 1000ULL) + (tv.tv_usec / 1000ULL); +} + +static int do_poll(int fd) +{ + struct pollfd pfd; + int ret; + + pfd.revents =3D 0; + pfd.fd =3D fd; + + ret =3D poll(&pfd, 1, waittime_ms); + if (ret =3D=3D -1) + error(1, errno, "poll"); + + return ret && (pfd.revents & POLLERR); +} + +static void wait_compl(int fd) +{ + int64_t tstop =3D gettimeofday_ms() + waittime_ms; + char control[CMSG_SPACE(100)] =3D {}; + struct sock_extended_err *serr; + struct msghdr msg =3D {}; + struct cmsghdr *cm; + __u32 hi, lo; + int ret; + + msg.msg_control =3D control; + msg.msg_controllen =3D sizeof(control); + + while (gettimeofday_ms() < tstop) { + if (!do_poll(fd)) + continue; + + ret =3D recvmsg(fd, &msg, MSG_ERRQUEUE); + if (ret < 0) { + if (errno =3D=3D EAGAIN) + continue; + error(1, errno, "recvmsg(MSG_ERRQUEUE)"); + return; + } + if (msg.msg_flags & MSG_CTRUNC) + error(1, 0, "MSG_CTRUNC\n"); + + for (cm =3D CMSG_FIRSTHDR(&msg); cm; cm =3D CMSG_NXTHDR(&msg, cm)) { + if (cm->cmsg_level !=3D SOL_IP && + cm->cmsg_level !=3D SOL_IPV6) + continue; + if (cm->cmsg_level =3D=3D SOL_IP && + cm->cmsg_type !=3D IP_RECVERR) + continue; + if (cm->cmsg_level =3D=3D SOL_IPV6 && + cm->cmsg_type !=3D IPV6_RECVERR) + continue; + + serr =3D (void *)CMSG_DATA(cm); + if (serr->ee_origin !=3D SO_EE_ORIGIN_ZEROCOPY) + error(1, 0, "wrong origin %u", serr->ee_origin); + if (serr->ee_errno !=3D 0) + error(1, 0, "wrong errno %d", serr->ee_errno); + + hi =3D serr->ee_data; + lo =3D serr->ee_info; + + fprintf(stderr, "tx complete [%d,%d]\n", lo, hi); + return; + } + } + + error(1, 0, "did not receive tx completion"); +} + +static int do_client(struct memory_buffer *mem) +{ + char ctrl_data[CMSG_SPACE(sizeof(__u32))]; + struct sockaddr_in6 server_sin; + struct sockaddr_in6 client_sin; + struct ynl_sock *ys =3D NULL; + struct msghdr msg =3D {}; + ssize_t line_size =3D 0; + struct cmsghdr *cmsg; + struct iovec iov[2]; + char *line =3D NULL; + unsigned long mid; + size_t len =3D 0; + int socket_fd; + __u32 ddmabuf; + int opt =3D 1; + int ret; + + ret =3D parse_address(server_ip, atoi(port), &server_sin); + if (ret < 0) + error(1, 0, "parse server address"); + + socket_fd =3D socket(AF_INET6, SOCK_STREAM, 0); + if (socket_fd < 0) + error(1, socket_fd, "create socket"); + + enable_reuseaddr(socket_fd); + + ret =3D setsockopt(socket_fd, SOL_SOCKET, SO_BINDTODEVICE, ifname, + strlen(ifname) + 1); + if (ret) + error(1, errno, "bindtodevice"); + + if (bind_tx_queue(ifindex, mem->fd, &ys)) + error(1, 0, "Failed to bind\n"); + + if (client_ip) { + ret =3D parse_address(client_ip, atoi(port), &client_sin); + if (ret < 0) + error(1, 0, "parse client address"); + + ret =3D bind(socket_fd, &client_sin, sizeof(client_sin)); + if (ret) + error(1, errno, "bind"); + } + + ret =3D setsockopt(socket_fd, SOL_SOCKET, SO_ZEROCOPY, &opt, sizeof(opt)); + if (ret) + error(1, errno, "set sock opt"); + + fprintf(stderr, "Connect to %s %d (via %s)\n", server_ip, + ntohs(server_sin.sin6_port), ifname); + + ret =3D connect(socket_fd, &server_sin, sizeof(server_sin)); + if (ret) + error(1, errno, "connect"); + + while (1) { + free(line); + line =3D NULL; + line_size =3D getline(&line, &len, stdin); + + if (line_size < 0) + break; + + mid =3D (line_size / 2) + 1; + + iov[0].iov_base =3D (void *)1; + iov[0].iov_len =3D mid; + iov[1].iov_base =3D (void *)(mid + 2); + iov[1].iov_len =3D line_size - mid; + + provider->memcpy_to_device(mem, (size_t)iov[0].iov_base, line, + iov[0].iov_len); + provider->memcpy_to_device(mem, (size_t)iov[1].iov_base, + line + iov[0].iov_len, + iov[1].iov_len); + + fprintf(stderr, + "read line_size=3D%ld iov[0].iov_base=3D%lu, iov[0].iov_len=3D%lu, iov[= 1].iov_base=3D%lu, iov[1].iov_len=3D%lu\n", + line_size, (unsigned long)iov[0].iov_base, + iov[0].iov_len, (unsigned long)iov[1].iov_base, + iov[1].iov_len); + + msg.msg_iov =3D iov; + msg.msg_iovlen =3D 2; + + msg.msg_control =3D ctrl_data; + msg.msg_controllen =3D sizeof(ctrl_data); + + cmsg =3D CMSG_FIRSTHDR(&msg); + cmsg->cmsg_level =3D SOL_SOCKET; + cmsg->cmsg_type =3D SCM_DEVMEM_DMABUF; + cmsg->cmsg_len =3D CMSG_LEN(sizeof(__u32)); + + ddmabuf =3D tx_dmabuf_id; + + *((__u32 *)CMSG_DATA(cmsg)) =3D ddmabuf; + + ret =3D sendmsg(socket_fd, &msg, MSG_ZEROCOPY); + if (ret < 0) + error(1, errno, "Failed sendmsg"); + + fprintf(stderr, "sendmsg_ret=3D%d\n", ret); + + if (ret !=3D line_size) + error(1, errno, "Did not send all bytes"); + + wait_compl(socket_fd); + } + + fprintf(stderr, "%s: tx ok\n", TEST_PREFIX); + + free(line); + close(socket_fd); + + if (ys) + ynl_sock_destroy(ys); + + return 0; +} + int main(int argc, char *argv[]) { struct memory_buffer *mem; @@ -728,6 +1004,8 @@ int main(int argc, char *argv[]) =20 ifindex =3D if_nametoindex(ifname); =20 + fprintf(stderr, "using ifindex=3D%u\n", ifindex); + if (!server_ip && !client_ip) { if (start_queue < 0 && num_queues < 0) { num_queues =3D rxq_num(ifindex); @@ -778,7 +1056,7 @@ int main(int argc, char *argv[]) error(1, 0, "Missing -p argument\n"); =20 mem =3D provider->alloc(getpagesize() * NUM_PAGES); - ret =3D is_server ? do_server(mem) : 1; + ret =3D is_server ? do_server(mem) : do_client(mem); provider->free(mem); =20 return ret; --=20 2.48.1.601.g30ceb7b040-goog