From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90BEB1AD3F7 for ; Mon, 9 Sep 2024 05:43:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860609; cv=none; b=XmhiADCel6N8BAyxMTWKP58rC/0oHHYEmET35uZ5cZl9nPhIzn5+Ysp0B0JCsSDHQe+W+ShPP/KF7jKNND7pdoShPghzuwMfKYwrmK6c9I2Hpit+8STeTeOfWssSEx4+WUd4MlrLdujQ1BLTMU6kuhOqeaNKtA2DXVIAzcSqtHA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860609; c=relaxed/simple; bh=DNA3F85kfwiW9bgsfvLXXO875oSihrq+DpGt530m/B8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IoYSQWnGht5CLYLayP4IqInr5xNCT3zHS520klQjNEppA9rvjbztuvErfbSYCElEFFn8vZh8Rmntfxcn0bHIbMoX0MUUbNuE1wzVPiYM5xUeiVabZc0VnMd59NTdXiWjbRRbbetLfIxgJo5NESjiXFYJKu+0UAiN1gTpqpH4NWI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bhjNMIen; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bhjNMIen" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e1a7fd2eb36so8206641276.0 for ; Sun, 08 Sep 2024 22:43:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860604; x=1726465404; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=i/PEygQn081YuovQ6b+Ov4J7AW7lnL8H+wq29MS093A=; b=bhjNMIenuk14C2l5V7q6HmUTPR/6a5fI1mLuNOuKhTPL4QFbZJ2hAKOng7I4m73L02 dHo+Zen81V06Aijil2ADBN+FROdjz+IeuA3aQGt1ncU0D0V1emng0fAt2+NaiRpEheEo p8kRRkFfRif7SNooNw9eofva7REUSIZrKXRym/sJ5Ko3jToKwvJsM8gFwWXH1uirzcgT mShe2gTJRgNWiabbBO9zBKkzmVgXiZUmgomCqZDYJOsyZ3kbN/vPlb74iaSAddbXRsmV 08KuEaD+WIiPe6JeWkXqBgu2GBMEYmI5pCWsFZD3BO7dQnqu8g72LK+qf0OQCG4GMBgX oxNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860604; x=1726465404; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=i/PEygQn081YuovQ6b+Ov4J7AW7lnL8H+wq29MS093A=; b=BmivArnvMcAGQL9rUi3d7eezHegLSBPwGKfEbmFz9KhtXUsof94QZ4GgwTzDZM38LQ 709Br2eZbJR6PauIImhN6ywYyydwNTdDzinlJxxRnRiTsfj8D+bLBmoGpilvZ1fod9/U hR1E+vOwIX/UGmo90dWuLEqNSU/56uBsxGanWs5DirBGtq0DFsQw+jrFJjd+YEM7QG20 JZ/3RZAqcjXCyCqf7OqfuZd7zLDfHpRGtSRLuzopZQBKQTqcu+IIrFzgzBvsL5ZIJ5Pk ScBInV4hW8LOv1rr5YeXf8iyokCzEEoV6ni+HBZufJu4lhW98rlT9fWxHE+IxI3Z43ST hSAw== X-Forwarded-Encrypted: i=1; AJvYcCUHMV8ZHmYNZV/v/338a0azrW7oJCpzCRjlt7jY0hTGlK6VTorr5Uh19sVhqQzhk0oQP+dfve9nYrcOCp0=@vger.kernel.org X-Gm-Message-State: AOJu0YySxxKaf/+uFdk5yUau/JC0Q4MpKrMgMfNipMKFoA+8rk6fFS0h aBwyiwymUwfknpkRMPLKwuYK0FI1ANuQu8j+DXl84+RWVZmk8S9Ppu7KIfVBDK4/C7T5eciF9wY ozhSA52uIJamGQ2/LYpUoag== X-Google-Smtp-Source: AGHT+IHY3ZWZM5m6iLmr25lpRxzPgba3UWyYzEPhtrKYdPsv3KEg5Ew5Cd8AVaehz177/KHSxTj8z97y5iwx9jcZ0A== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a05:6902:4d1:b0:e0b:cce3:45c7 with SMTP id 3f1490d57ef6-e1d349e38d5mr18201276.9.1725860604167; Sun, 08 Sep 2024 22:43:24 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:06 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-2-almasrymina@google.com> Subject: [PATCH net-next v25 01/13] netdev: add netdev_rx_queue_restart() From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add netdev_rx_queue_restart(), which resets an rx queue using the queue API recently merged[1]. The queue API was merged to enable the core net stack to reset individual rx queues to actuate changes in the rx queue's configuration. In later patches in this series, we will use netdev_rx_queue_restart() to reset rx queues after binding or unbinding dmabuf configuration, which will cause reallocation of the page_pool to repopulate its memory using the new configuration. [1] https://lore.kernel.org/netdev/20240430231420.699177-1-shailend@google.= com/T/ Signed-off-by: David Wei Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v18: - Add more color to commit message (Xuan Zhuo). v17: - Use ASSERT_RTNL() (Jakub). v13: - Add reviewed-by from Pavel (thanks!) - Fixed comment (Pavel) v11: - Fix not checking dev->queue_mgmt_ops (Pavel). - Fix ndo_queue_mem_free call that passed the wrong pointer (David). v9: https://lore.kernel.org/all/20240502045410.3524155-4-dw@davidwei.uk/ (submitted by David). - fixed SPDX license identifier (Simon). - Rebased on top of merged queue API definition, and changed implementation to match that. - Replace rtnl_lock() with rtnl_is_locked() to make it useable from my netlink code where rtnl is already locked. --- include/net/netdev_rx_queue.h | 3 ++ net/core/Makefile | 1 + net/core/netdev_rx_queue.c | 74 +++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100644 net/core/netdev_rx_queue.c diff --git a/include/net/netdev_rx_queue.h b/include/net/netdev_rx_queue.h index aa1716fb0e53..e78ca52d67fb 100644 --- a/include/net/netdev_rx_queue.h +++ b/include/net/netdev_rx_queue.h @@ -54,4 +54,7 @@ get_netdev_rx_queue_index(struct netdev_rx_queue *queue) return index; } #endif + +int netdev_rx_queue_restart(struct net_device *dev, unsigned int rxq); + #endif diff --git a/net/core/Makefile b/net/core/Makefile index 62be9aef2528..f82232b358a2 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -19,6 +19,7 @@ obj-$(CONFIG_NETDEV_ADDR_LIST_TEST) +=3D dev_addr_lists_t= est.o =20 obj-y +=3D net-sysfs.o obj-y +=3D hotdata.o +obj-y +=3D netdev_rx_queue.o obj-$(CONFIG_PAGE_POOL) +=3D page_pool.o page_pool_user.o obj-$(CONFIG_PROC_FS) +=3D net-procfs.o obj-$(CONFIG_NET_PKTGEN) +=3D pktgen.o diff --git a/net/core/netdev_rx_queue.c b/net/core/netdev_rx_queue.c new file mode 100644 index 000000000000..da11720a5983 --- /dev/null +++ b/net/core/netdev_rx_queue.c @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include +#include +#include + +int netdev_rx_queue_restart(struct net_device *dev, unsigned int rxq_idx) +{ + void *new_mem, *old_mem; + int err; + + if (!dev->queue_mgmt_ops || !dev->queue_mgmt_ops->ndo_queue_stop || + !dev->queue_mgmt_ops->ndo_queue_mem_free || + !dev->queue_mgmt_ops->ndo_queue_mem_alloc || + !dev->queue_mgmt_ops->ndo_queue_start) + return -EOPNOTSUPP; + + ASSERT_RTNL(); + + new_mem =3D kvzalloc(dev->queue_mgmt_ops->ndo_queue_mem_size, GFP_KERNEL); + if (!new_mem) + return -ENOMEM; + + old_mem =3D kvzalloc(dev->queue_mgmt_ops->ndo_queue_mem_size, GFP_KERNEL); + if (!old_mem) { + err =3D -ENOMEM; + goto err_free_new_mem; + } + + err =3D dev->queue_mgmt_ops->ndo_queue_mem_alloc(dev, new_mem, rxq_idx); + if (err) + goto err_free_old_mem; + + err =3D dev->queue_mgmt_ops->ndo_queue_stop(dev, old_mem, rxq_idx); + if (err) + goto err_free_new_queue_mem; + + err =3D dev->queue_mgmt_ops->ndo_queue_start(dev, new_mem, rxq_idx); + if (err) + goto err_start_queue; + + dev->queue_mgmt_ops->ndo_queue_mem_free(dev, old_mem); + + kvfree(old_mem); + kvfree(new_mem); + + return 0; + +err_start_queue: + /* Restarting the queue with old_mem should be successful as we haven't + * changed any of the queue configuration, and there is not much we can + * do to recover from a failure here. + * + * WARN if we fail to recover the old rx queue, and at least free + * old_mem so we don't also leak that. + */ + if (dev->queue_mgmt_ops->ndo_queue_start(dev, old_mem, rxq_idx)) { + WARN(1, + "Failed to restart old queue in error path. RX queue %d may be unhe= althy.", + rxq_idx); + dev->queue_mgmt_ops->ndo_queue_mem_free(dev, old_mem); + } + +err_free_new_queue_mem: + dev->queue_mgmt_ops->ndo_queue_mem_free(dev, new_mem); + +err_free_old_mem: + kvfree(old_mem); + +err_free_new_mem: + kvfree(new_mem); + + return err; +} --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C72401AD404 for ; Mon, 9 Sep 2024 05:43:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860611; cv=none; b=khUH0M+3nEg7ldCm1pYza3SYdSnUPFg9vBweVn+FFjR/jsplnlk8+ThFXAfxHdIgZdaPxCtrgI5n1vcCcsHzBQVggpuWjiUUyzEnGFaXZPFIMBdPyoXjhJt2x6Xt8LHVtVDuxQPPU4B1M8lu1zKJN7VPtxYFvHB6kjTjGl7lH+0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860611; c=relaxed/simple; bh=ifkk5EzOQnvX7N0pj7xFyfgeQQygVm1rE0XeuABHrqg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tGcoxJG4yYc5HTO6lAFosCX3LaO4ssKKoLFTFRNJU1j0EhT0toEy7BOLtdZSIyMKYSoPJXRrEn9h7kmMpAaF0IcP5f49XHccoE4MQ3TWZGNG7i5IXqVa43q28TNPG2esyG8m2EXKIi+AzJ/QtTHsJdOYNIygkxV/ZNgmpKFiHwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=a5d9kbNw; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a5d9kbNw" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6b41e02c255so131979317b3.3 for ; Sun, 08 Sep 2024 22:43:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860606; x=1726465406; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6CrK96qJP4z3b0FDzKM+/eICeW01UPDJxn1zoMHr9yY=; b=a5d9kbNwdXiCQbq7DjyOkRduY4lwRlWMHxQCjsVOXTYIv8VgDQsn3pbzMIeVKUj7Zh GeYKL8wFoWwy2BJk9w4jf6ShaUOJLbQYpCusSL6ltur9jECYJbm6rMdVPG/AzneKWpb1 XdYsqqMwL456RWqcGXnoAeIIC6EQ7TEy829ODkxhVBsjH7YVaIyCwQJ/C+16ZEYkOnnm PixTvYmiWIGfLJNNPwCMMzJawU7wstHxl0joH0TI8uELhdwGYcDhF3hX2M+sOo1Oekbg A8mrRQypKgDay9idu4MHqYGX1BtYuo+pdmYR9qYHsRs1jAkJf0QFp9sb4lX8Q1gkOBlS yrpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860606; x=1726465406; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6CrK96qJP4z3b0FDzKM+/eICeW01UPDJxn1zoMHr9yY=; b=mNlW2DhpbT6OcZb1OLkMuBvCi29AjpeHQKo1K8J/5HYHXlfbZZ2dBe7jlGGMjMeEjg pkfWZtq3GplbjCzJyAaL4w4CU5ujzCnXaM2ImmUssP+cD0+bbf/NTPYx4S39Zsbvn6uz vofMQcLXbb1ql6Znjgj5ER/VQFhrLdv0crG0BpV3iojf/CnYNHY6g7B4gCg4k3lFd0+7 o1bNR6LOURt3Kjn/PnZwOleY3Pr3afaiEYafwkIfKGUSdCG4y2gE8kVdF/O9CRv5EQVg E736soQfrybV6O9MjgK/C56bQ+4UuStTXP7Ylxn6mavzIOuvwoVw9K5+NldL4L42s48o zDWQ== X-Forwarded-Encrypted: i=1; AJvYcCXbwLDfo/bYAcN5Cz+tUwc++qc8ZLATcyRe8+oZKyLIqPDrtpNo5Df5mhWLQBFy20gzH8g0LRmKk2VBE5M=@vger.kernel.org X-Gm-Message-State: AOJu0YzXj4ckHFngNuo9bWDzWFDqc+jYBN5xQ6mvaCqbyO93rnruYolD eZnvASisrHTeBqBaU1OUg8NL/RqtIv42uqO66LJx9OwBLfAgGYgUlxEByZPumGC+FasLDwSrBj6 qBBmIGbPtRFQvrSW0nkjs1A== X-Google-Smtp-Source: AGHT+IGwQgjuvNyD+Z4vlmdws3yK/iqd/mq/O3+Dp4Tm5KrdXVTdk9Si6mlTifCHDFeMeeeKzuzLtTC/snqQIS+9nw== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a0d:e4c2:0:b0:68d:cbd6:e28 with SMTP id 00721157ae682-6db4516321bmr3046067b3.6.1725860605736; Sun, 08 Sep 2024 22:43:25 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:07 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-3-almasrymina@google.com> Subject: [PATCH net-next v25 02/13] net: netdev netlink api to bind dma-buf to a net device From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Stanislav Fomichev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" API takes the dma-buf fd as input, and binds it to the netdevice. The user can specify the rx queues to bind the dma-buf to. Suggested-by: Stanislav Fomichev Signed-off-by: Mina Almasry Reviewed-by: Donald Hunter Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v16: - Use subset-of: queue queue-id instead of queue-dmabuf (Jakub). - Rename attribute 'bind-dmabuf' to more generic 'dmabuf' (Jakub). - Use 'dmabuf' everywhere instead of mix of 'dma-buf' and 'dmabuf' (Donald). - Remove repetitive 'dmabuf' naming that appeared in some places (Jakub). - Reordered where the new operations went so I don't break the enum UAPI (Jakub). v7: - Use flags: [ admin-perm ] instead of a CAP_NET_ADMIN check. Changes in v1: - Add rx-queue-type to distingish rx from tx (Jakub) - Return dma-buf ID from netlink API (David, Stan) Changes in RFC-v3: - Support binding multiple rx rx-queues --- Documentation/netlink/specs/netdev.yaml | 47 +++++++++++++++++++++++++ include/uapi/linux/netdev.h | 11 ++++++ net/core/netdev-genl-gen.c | 19 ++++++++++ net/core/netdev-genl-gen.h | 2 ++ net/core/netdev-genl.c | 6 ++++ tools/include/uapi/linux/netdev.h | 11 ++++++ 6 files changed, 96 insertions(+) diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlin= k/specs/netdev.yaml index 959755be4d7f..4930e8142aa6 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -457,6 +457,39 @@ attribute-sets: Number of times driver re-started accepting send requests to this queue from the stack. type: uint + - + name: queue-id + subset-of: queue + attributes: + - + name: id + - + name: type + - + name: dmabuf + attributes: + - + name: ifindex + doc: netdev ifindex to bind the dmabuf to. + type: u32 + checks: + min: 1 + - + name: queues + doc: receive queues to bind the dmabuf to. + type: nest + nested-attributes: queue-id + multi-attr: true + - + name: fd + doc: dmabuf file descriptor to bind. + type: u32 + - + name: id + doc: id of the dmabuf binding + type: u32 + checks: + min: 1 =20 operations: list: @@ -619,6 +652,20 @@ operations: - rx-bytes - tx-packets - tx-bytes + - + name: bind-rx + doc: Bind dmabuf to netdev + attribute-set: dmabuf + flags: [ admin-perm ] + do: + request: + attributes: + - ifindex + - fd + - queues + reply: + attributes: + - id =20 mcast-groups: list: diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h index 43742ac5b00d..91bf3ecc5f1d 100644 --- a/include/uapi/linux/netdev.h +++ b/include/uapi/linux/netdev.h @@ -173,6 +173,16 @@ enum { NETDEV_A_QSTATS_MAX =3D (__NETDEV_A_QSTATS_MAX - 1) }; =20 +enum { + NETDEV_A_DMABUF_IFINDEX =3D 1, + NETDEV_A_DMABUF_QUEUES, + NETDEV_A_DMABUF_FD, + NETDEV_A_DMABUF_ID, + + __NETDEV_A_DMABUF_MAX, + NETDEV_A_DMABUF_MAX =3D (__NETDEV_A_DMABUF_MAX - 1) +}; + enum { NETDEV_CMD_DEV_GET =3D 1, NETDEV_CMD_DEV_ADD_NTF, @@ -186,6 +196,7 @@ enum { NETDEV_CMD_QUEUE_GET, NETDEV_CMD_NAPI_GET, NETDEV_CMD_QSTATS_GET, + NETDEV_CMD_BIND_RX, =20 __NETDEV_CMD_MAX, NETDEV_CMD_MAX =3D (__NETDEV_CMD_MAX - 1) diff --git a/net/core/netdev-genl-gen.c b/net/core/netdev-genl-gen.c index 8350a0afa9ec..6b7fe6035067 100644 --- a/net/core/netdev-genl-gen.c +++ b/net/core/netdev-genl-gen.c @@ -27,6 +27,11 @@ const struct nla_policy netdev_page_pool_info_nl_policy[= NETDEV_A_PAGE_POOL_IFIND [NETDEV_A_PAGE_POOL_IFINDEX] =3D NLA_POLICY_FULL_RANGE(NLA_U32, &netdev_a= _page_pool_ifindex_range), }; =20 +const struct nla_policy netdev_queue_id_nl_policy[NETDEV_A_QUEUE_TYPE + 1]= =3D { + [NETDEV_A_QUEUE_ID] =3D { .type =3D NLA_U32, }, + [NETDEV_A_QUEUE_TYPE] =3D NLA_POLICY_MAX(NLA_U32, 1), +}; + /* NETDEV_CMD_DEV_GET - do */ static const struct nla_policy netdev_dev_get_nl_policy[NETDEV_A_DEV_IFIND= EX + 1] =3D { [NETDEV_A_DEV_IFINDEX] =3D NLA_POLICY_MIN(NLA_U32, 1), @@ -74,6 +79,13 @@ static const struct nla_policy netdev_qstats_get_nl_poli= cy[NETDEV_A_QSTATS_SCOPE [NETDEV_A_QSTATS_SCOPE] =3D NLA_POLICY_MASK(NLA_UINT, 0x1), }; =20 +/* NETDEV_CMD_BIND_RX - do */ +static const struct nla_policy netdev_bind_rx_nl_policy[NETDEV_A_DMABUF_FD= + 1] =3D { + [NETDEV_A_DMABUF_IFINDEX] =3D NLA_POLICY_MIN(NLA_U32, 1), + [NETDEV_A_DMABUF_FD] =3D { .type =3D NLA_U32, }, + [NETDEV_A_DMABUF_QUEUES] =3D NLA_POLICY_NESTED(netdev_queue_id_nl_policy), +}; + /* Ops table for netdev */ static const struct genl_split_ops netdev_nl_ops[] =3D { { @@ -151,6 +163,13 @@ static const struct genl_split_ops netdev_nl_ops[] =3D= { .maxattr =3D NETDEV_A_QSTATS_SCOPE, .flags =3D GENL_CMD_CAP_DUMP, }, + { + .cmd =3D NETDEV_CMD_BIND_RX, + .doit =3D netdev_nl_bind_rx_doit, + .policy =3D netdev_bind_rx_nl_policy, + .maxattr =3D NETDEV_A_DMABUF_FD, + .flags =3D GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, }; =20 static const struct genl_multicast_group netdev_nl_mcgrps[] =3D { diff --git a/net/core/netdev-genl-gen.h b/net/core/netdev-genl-gen.h index 4db40fd5b4a9..67c34005750c 100644 --- a/net/core/netdev-genl-gen.h +++ b/net/core/netdev-genl-gen.h @@ -13,6 +13,7 @@ =20 /* Common nested types */ extern const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PA= GE_POOL_IFINDEX + 1]; +extern const struct nla_policy netdev_queue_id_nl_policy[NETDEV_A_QUEUE_TY= PE + 1]; =20 int netdev_nl_dev_get_doit(struct sk_buff *skb, struct genl_info *info); int netdev_nl_dev_get_dumpit(struct sk_buff *skb, struct netlink_callback = *cb); @@ -30,6 +31,7 @@ int netdev_nl_napi_get_doit(struct sk_buff *skb, struct g= enl_info *info); int netdev_nl_napi_get_dumpit(struct sk_buff *skb, struct netlink_callback= *cb); int netdev_nl_qstats_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb); +int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info); =20 enum { NETDEV_NLGRP_MGMT, diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index a17d7eaeb001..699c34b9b03c 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -723,6 +723,12 @@ int netdev_nl_qstats_get_dumpit(struct sk_buff *skb, return err; } =20 +/* Stub */ +int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info) +{ + return 0; +} + static int netdev_genl_netdevice_event(struct notifier_block *nb, unsigned long event, void *ptr) { diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/n= etdev.h index 43742ac5b00d..91bf3ecc5f1d 100644 --- a/tools/include/uapi/linux/netdev.h +++ b/tools/include/uapi/linux/netdev.h @@ -173,6 +173,16 @@ enum { NETDEV_A_QSTATS_MAX =3D (__NETDEV_A_QSTATS_MAX - 1) }; =20 +enum { + NETDEV_A_DMABUF_IFINDEX =3D 1, + NETDEV_A_DMABUF_QUEUES, + NETDEV_A_DMABUF_FD, + NETDEV_A_DMABUF_ID, + + __NETDEV_A_DMABUF_MAX, + NETDEV_A_DMABUF_MAX =3D (__NETDEV_A_DMABUF_MAX - 1) +}; + enum { NETDEV_CMD_DEV_GET =3D 1, NETDEV_CMD_DEV_ADD_NTF, @@ -186,6 +196,7 @@ enum { NETDEV_CMD_QUEUE_GET, NETDEV_CMD_NAPI_GET, NETDEV_CMD_QSTATS_GET, + NETDEV_CMD_BIND_RX, =20 __NETDEV_CMD_MAX, NETDEV_CMD_MAX =3D (__NETDEV_CMD_MAX - 1) --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 127701AD41B for ; Mon, 9 Sep 2024 05:43:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860614; cv=none; b=tZPBxeT5ZTxNEwuG/u5IxqlxwpBylqeJbuq0Wo60/eMB/djVTR/kFehm7O11nx8qNGa50zeOzBzNOSk/c4lakfInCACYMwzeyy+tE06yqTtvt2Y/qj9uxxwg4/NcQXuA4b4yYt+bwzR0jzbkh3YcOYXOvgNn8wp37aOQWvh+dmE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860614; c=relaxed/simple; bh=YThCjVefsi6IAUul05R1hSkYRX10WGwXp9CM5JFxoBU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UcpBIfKMKQ+UHYvUNVqKqnaKjevjgShMci20uCu6Cvx/wUrN2ai2YrZpL6U31iRIen6Qfyf3urOtffSYlNxxOXDcHJdYEapZABkAbBIj01BeoxghV2av3FsScoXqvSSjNCrlIwT8LwjJqes3JmhiaXzKlNR3p+WMxRG7YPVC5LI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RRylbBqh; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RRylbBqh" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1ce191f74fso8229745276.2 for ; Sun, 08 Sep 2024 22:43:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860608; x=1726465408; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4Kxo90iZIgMuTSIU5YNDMy1A1yixSpqC3oKgCU2lYcA=; b=RRylbBqhDlbsBdnkpx9T6PHGV/HhiwV5uHUQspuh/8W+h8WTKNLJAbVaTZnD7AFRZM bHN5Gj1gzZSihLfgzoINjsSUn7JQ1qrzvDVomeKmVlZxylaOpJzXU+RFbx3XWYyZ/VaL vXB5R4QyUmXFqaZiWJy1OwoyEJsTACHfYeyKh4zMv14mJRc9BUKLY19ym2MvykAfu3o4 BvgrYYOymWyj8CYExVhEh8e684RZiW0RQvcd1jmpAyxJ8C6Jg6Ai+GAmjsHHX9QY/NpA sUK5kpl4gXMFgA4/UIbWrnxPB+AW1pSnESfsuinLoiusabJLhhQ0eXLggB9gzWzrw5Fb tpaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860608; x=1726465408; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4Kxo90iZIgMuTSIU5YNDMy1A1yixSpqC3oKgCU2lYcA=; b=SJtKcz8WXIYuiioPVgMz4Il6zJmtXHo0TtG4H254l9nFyhGmTjwuynb++zW4lWpdYA o/uBeh7UQtKDD38ukKwxE7r7qlr5HgLUKwMV8HH95+lIRift5vOJo03R61ZLWUXJ4sii N6jKcXpAApfxPUHw3WCwq6kR9X2jBgoLlPJ/yCHubY4s9UhuVwI03Q+kecP61bmy5fcp Qx2UMAk0scAt5y5YM1z+XXZgSpKjqcsBAYYLOhJ6B5AsJdHk+UeFuSvzUg6i5nGyHjJ7 sO5JbsL5j/BAtEOe8xadkBhrkb7nfdttbJ7U0EHSHRe3TEibJScmC9dhbGlj9LVPfPWe F9Sg== X-Forwarded-Encrypted: i=1; AJvYcCXRszusS/O7fil4bp1GkR1JZalqkf7I5FZY3SrHtrhVAmqJWV+Q5pjrvS2HzAMkNMlmggJVr1cgT0Q35ic=@vger.kernel.org X-Gm-Message-State: AOJu0Ywmf95O9bt25nioV/k9T0jupYCoz74Y7LrfSnVI/v6zZwpqOAUo eqaz1GStsy3ADaoaEpa5WN/WbH+HaFEBIH8e5RCKcbVN2eciBnhTLbTT+YGbCpJgsb+HtyDcMaC BE/iOwti2aSU7Z/k1m3UVSw== X-Google-Smtp-Source: AGHT+IGwu6xLZdN1A9EcPYwq+tgHKt/okVw2EehcgL7FBwO38gz8vq3ImBQouq4fcGDNJz94s4s22QDgQvqSv3NTVA== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a5b:b50:0:b0:e0e:445b:606 with SMTP id 3f1490d57ef6-e1d346b001fmr18938276.0.1725860607547; Sun, 08 Sep 2024 22:43:27 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:08 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-4-almasrymina@google.com> Subject: [PATCH net-next v25 03/13] netdev: support binding dma-buf to netdevice From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Willem de Bruijn , Kaiyuan Zhang , Daniel Vetter Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a netdev_dmabuf_binding struct which represents the dma-buf-to-netdevice binding. The netlink API will bind the dma-buf to rx queues on the netdevice. On the binding, the dma_buf_attach & dma_buf_map_attachment will occur. The entries in the sg_table from mapping will be inserted into a genpool to make it ready for allocation. The chunks in the genpool are owned by a dmabuf_chunk_owner struct which holds the dma-buf offset of the base of the chunk and the dma_addr of the chunk. Both are needed to use allocations that come from this chunk. We create a new type that represents an allocation from the genpool: net_iov. We setup the net_iov allocation size in the genpool to PAGE_SIZE for simplicity: to match the PAGE_SIZE normally allocated by the page pool and given to the drivers. The user can unbind the dmabuf from the netdevice by closing the netlink socket that established the binding. We do this so that the binding is automatically unbound even if the userspace process crashes. The binding and unbinding leaves an indicator in struct netdev_rx_queue that the given queue is bound, and the binding is actuated by resetting the rx queue using the queue API. The netdev_dmabuf_binding struct is refcounted, and releases its resources only when all the refs are released. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov # excluding netlink Acked-by: Daniel Vetter Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v25: - Move include/net/devmem.h to net/core/devmem.h. - netmem.h no longer includes devmem.h (which enables moving devmem.h to internal include file). v24: - Fix a couple of problematic goto err_ on error paths (Jakub & Vadim) - Add NL_SET_BAD_ATTR (Jakub). - Fix wrong include order causing ynl check to fail (NIPA) - Add extack error messages (Jakub). v23: - Create CONFIG_NET_DEVMEM instead of checking for CONFIG_DMA_SHARED_BUFFER && CONFIG_GENERIC_ALLOCATOR (Jakub). - Rework loop in dev_get_min_mp_channel_count (Jakub). - Sort includes (Jakub). - WARN_ON if queue in bound_rxqs doesn't match the memory provider being unbound (Jakub). - Add some extack error messages (Jakub). - Move dev_xdp_prog_count check to the device bending rather than the queue binding (Jakub). - goto err_unbind on genlmsg_reply failure (Jakub). v22: - Disable binding xdp to mp bound netdevs, and propagating xdp configuration to mp bound netdevs - Add extack error messages. - Prevent binding dmabuf to a queue with xsk_buff_pool (Jakub) - Prevent xp_assign_dev to a device using memory provider (Jakub) v21: - Move definition of net_devmem_dmabuf_bindings to inside the #ifdef to prevent unused static variable warning. v20: - rename dev_get_max_mp_channel to dev_get_min_mp_channel_count (Jakub) - Removed unnecessary includes from dev.c - Fixed bug with return value of dev_get_min_mp_channel_count getting implicitly cast to unsigned int (Jakub) - Combine netlink attr checks into one statement with || (Jakub) - Removed unnecessary include from ethtool/common.c v19: - Prevent deactivating queues bound to mp (Jakub). - disable attaching xdp to memory provider netdev (Jakub). - Address various nits from Jakub. - In the netlink API, check for presence of queue_id, queue_type and that the queue_type is RX (Jakub). v17: - Add missing kfree(owner) (Jakub) - Fix issue found by Taehee, where may access an rxq that has already been freed if the driver has been unloaded in the meantime (thanks!) v16: - Fix rtnl_lock() not being acquired on unbind path (Reported by Taehee). - Use unlocked versions of dma_buf_[un]map_attachment (Reported by Taehee). - Use real_num_rx_queues instead of num_rx_queues (Taehee). - bound_rxq_list -> bound_rxqs (Jakub). - Removed READ_ONCE/WRITE_ONCE infavor of rtnl_lock() sync. (Jakub). - Use ERR_CAST instead of out param (Jakub). - Add NULL Check for kzalloc_node() call (Paolo). - Move genl_sk_priv_get, genlmsg_new, genlmsg_input outside of the lock acquisition (Jakub). - Add netif_device_present() check (Jakub). - Use nla_for_each_attr_type(Jakub). v13: - Fixed a couple of places that still listed DMA_BIDIRECTIONAL (Pavel). - Added reviewed-by from Pavel. v11: - Fix build error with CONFIG_DMA_SHARED_BUFFER && !CONFIG_GENERIC_ALLOCATOR - Rebased on top of no memory provider ops. v10: - Moved net_iov_dma_addr() to devmem.h and made it devmem specific helper (David). v9: https://lore.kernel.org/all/20240403002053.2376017-5-almasrymina@google= .com/ - Removed net_devmem_restart_rx_queues and put it in its own patch (David). v8: - move dmabuf_devmem_ops usage to later patch to avoid patch-by-patch build error. v7: - Use IS_ERR() instead of IS_ERR_OR_NULL() for the dma_buf_get() return value. - Changes netdev_* naming in devmem.c to net_devmem_* (Yunsheng). - DMA_BIDIRECTIONAL -> DMA_FROM_DEVICE (Yunsheng). - Added a comment around recovering of the old rx queue in net_devmem_restart_rx_queue(), and added freeing of old_mem if the restart of the old queue fails. (Yunsheng). - Use kernel-family sock-priv (Jakub). - Put pp_memory_provider_params in netdev_rx_queue instead of the dma-buf specific binding (Pavel & David). - Move queue management ops to queue_mgmt_ops instead of netdev_ops (Jakub). - Remove excess whitespaces (Jakub). - Use genlmsg_iput (Jakub). v6: - Validate rx queue index - Refactor new functions into devmem.c (Pavel) v5: - Renamed page_pool_iov to net_iov, and moved that support to devmem.h or netmem.h. v1: - Introduce devmem.h instead of bloating netdevice.h (Jakub) - ENOTSUPP -> EOPNOTSUPP (checkpatch.pl I think) - Remove unneeded rcu protection for binding->list (rtnl protected) - Removed extraneous err_binding_put: label. - Removed dma_addr +=3D len (Paolo). - Don't override err on netdev_bind_dmabuf_to_queue failure. - Rename devmem -> dmabuf (David). - Add id to dmabuf binding (David/Stan). - Fix missing xa_destroy bound_rq_list. - Use queue api to reset bound RX queues (Jakub). - Update netlink API for rx-queue type (tx/re) (Jakub). RFC v3: - Support multi rx-queue binding --- Documentation/netlink/specs/netdev.yaml | 4 + include/linux/netdevice.h | 2 + include/net/netdev_rx_queue.h | 2 + include/net/netmem.h | 8 + include/net/page_pool/types.h | 6 + net/Kconfig | 5 + net/core/Makefile | 1 + net/core/dev.c | 26 +++ net/core/devmem.c | 274 ++++++++++++++++++++++++ net/core/devmem.h | 124 +++++++++++ net/core/netdev-genl-gen.c | 4 + net/core/netdev-genl-gen.h | 4 + net/core/netdev-genl.c | 128 ++++++++++- net/ethtool/common.c | 8 + net/xdp/xsk_buff_pool.c | 5 + 15 files changed, 596 insertions(+), 5 deletions(-) create mode 100644 net/core/devmem.c create mode 100644 net/core/devmem.h diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlin= k/specs/netdev.yaml index 4930e8142aa6..0c747530c275 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -667,6 +667,10 @@ operations: attributes: - id =20 +kernel-family: + headers: [ "linux/list.h"] + sock-priv: struct list_head + mcast-groups: list: - diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index b47c00657bd0..bc167a1288fe 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3944,6 +3944,8 @@ u8 dev_xdp_prog_count(struct net_device *dev); int dev_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf); u32 dev_xdp_prog_id(struct net_device *dev, enum bpf_xdp_mode mode); =20 +u32 dev_get_min_mp_channel_count(const struct net_device *dev); + int __dev_forward_skb(struct net_device *dev, struct sk_buff *skb); int dev_forward_skb(struct net_device *dev, struct sk_buff *skb); int dev_forward_skb_nomtu(struct net_device *dev, struct sk_buff *skb); diff --git a/include/net/netdev_rx_queue.h b/include/net/netdev_rx_queue.h index e78ca52d67fb..ac34f5fb4f71 100644 --- a/include/net/netdev_rx_queue.h +++ b/include/net/netdev_rx_queue.h @@ -6,6 +6,7 @@ #include #include #include +#include =20 /* This structure contains an instance of an RX queue. */ struct netdev_rx_queue { @@ -25,6 +26,7 @@ struct netdev_rx_queue { * Readers and writers must hold RTNL */ struct napi_struct *napi; + struct pp_memory_provider_params mp_params; } ____cacheline_aligned_in_smp; =20 /* diff --git a/include/net/netmem.h b/include/net/netmem.h index 46cc9b89ac79..c23e224dd6a0 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -8,6 +8,14 @@ #ifndef _NET_NETMEM_H #define _NET_NETMEM_H =20 +/* net_iov */ + +struct net_iov { + struct dmabuf_genpool_chunk_owner *owner; +}; + +/* netmem */ + /** * typedef netmem_ref - a nonexistent type marking a reference to generic * network memory. diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 50569fed7868..4afd6dd56351 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -139,6 +139,10 @@ struct page_pool_stats { */ #define PAGE_POOL_FRAG_GROUP_ALIGN (4 * sizeof(long)) =20 +struct pp_memory_provider_params { + void *mp_priv; +}; + struct page_pool { struct page_pool_params_fast p; =20 @@ -197,6 +201,8 @@ struct page_pool { */ struct ptr_ring ring; =20 + void *mp_priv; + #ifdef CONFIG_PAGE_POOL_STATS /* recycle stats are per-cpu to avoid locking */ struct page_pool_recycle_stats __percpu *recycle_stats; diff --git a/net/Kconfig b/net/Kconfig index d27d0deac0bf..7574b066d7cd 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -66,6 +66,11 @@ config SKB_DECRYPTED config SKB_EXTENSIONS bool =20 +config NET_DEVMEM + def_bool y + depends on DMA_SHARED_BUFFER + depends on GENERIC_ALLOCATOR + menu "Networking options" =20 source "net/packet/Kconfig" diff --git a/net/core/Makefile b/net/core/Makefile index f82232b358a2..c3ebbaf9c81e 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -44,3 +44,4 @@ obj-$(CONFIG_BPF_SYSCALL) +=3D sock_map.o obj-$(CONFIG_BPF_SYSCALL) +=3D bpf_sk_storage.o obj-$(CONFIG_OF) +=3D of_net.o obj-$(CONFIG_NET_TEST) +=3D net_test.o +obj-$(CONFIG_NET_DEVMEM) +=3D devmem.o diff --git a/net/core/dev.c b/net/core/dev.c index 22c3f14d9287..34d73d152f19 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -161,6 +161,7 @@ #include =20 #include "dev.h" +#include "devmem.h" #include "net-sysfs.h" =20 static DEFINE_SPINLOCK(ptype_lock); @@ -9374,6 +9375,11 @@ int dev_xdp_propagate(struct net_device *dev, struct= netdev_bpf *bpf) if (!dev->netdev_ops->ndo_bpf) return -EOPNOTSUPP; =20 + if (dev_get_min_mp_channel_count(dev)) { + NL_SET_ERR_MSG(bpf->extack, "unable to propagate XDP to device using mem= ory provider"); + return -EBUSY; + } + return dev->netdev_ops->ndo_bpf(dev, bpf); } EXPORT_SYMBOL_GPL(dev_xdp_propagate); @@ -9406,6 +9412,11 @@ static int dev_xdp_install(struct net_device *dev, e= num bpf_xdp_mode mode, struct netdev_bpf xdp; int err; =20 + if (dev_get_min_mp_channel_count(dev)) { + NL_SET_ERR_MSG(extack, "unable to install XDP to device using memory pro= vider"); + return -EBUSY; + } + memset(&xdp, 0, sizeof(xdp)); xdp.command =3D mode =3D=3D XDP_MODE_HW ? XDP_SETUP_PROG_HW : XDP_SETUP_P= ROG; xdp.extack =3D extack; @@ -9830,6 +9841,20 @@ int dev_change_xdp_fd(struct net_device *dev, struct= netlink_ext_ack *extack, return err; } =20 +u32 dev_get_min_mp_channel_count(const struct net_device *dev) +{ + int i; + + ASSERT_RTNL(); + + for (i =3D dev->real_num_rx_queues - 1; i >=3D 0; i--) + if (dev->_rx[i].mp_params.mp_priv) + /* The channel count is the idx plus 1. */ + return i + 1; + + return 0; +} + /** * dev_index_reserve() - allocate an ifindex in a namespace * @net: the applicable net namespace @@ -11366,6 +11391,7 @@ void unregister_netdevice_many_notify(struct list_h= ead *head, dev_tcx_uninstall(dev); dev_xdp_uninstall(dev); bpf_dev_bound_netdev_unregister(dev); + dev_dmabuf_uninstall(dev); =20 netdev_offload_xstats_disable_all(dev); =20 diff --git a/net/core/devmem.c b/net/core/devmem.c new file mode 100644 index 000000000000..8dd7beb080d2 --- /dev/null +++ b/net/core/devmem.c @@ -0,0 +1,274 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Devmem TCP + * + * Authors: Mina Almasry + * Willem de Bruijn + * Kaiyuan Zhang +#include +#include +#include +#include +#include +#include +#include +#include + +#include "devmem.h" + +/* Device memory support */ + +/* Protected by rtnl_lock() */ +static DEFINE_XARRAY_FLAGS(net_devmem_dmabuf_bindings, XA_FLAGS_ALLOC1); + +static void net_devmem_dmabuf_free_chunk_owner(struct gen_pool *genpool, + struct gen_pool_chunk *chunk, + void *not_used) +{ + struct dmabuf_genpool_chunk_owner *owner =3D chunk->owner; + + kvfree(owner->niovs); + kfree(owner); +} + +void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *bi= nding) +{ + size_t size, avail; + + gen_pool_for_each_chunk(binding->chunk_pool, + net_devmem_dmabuf_free_chunk_owner, NULL); + + size =3D gen_pool_size(binding->chunk_pool); + avail =3D gen_pool_avail(binding->chunk_pool); + + if (!WARN(size !=3D avail, "can't destroy genpool. size=3D%zu, avail=3D%z= u", + size, avail)) + gen_pool_destroy(binding->chunk_pool); + + dma_buf_unmap_attachment_unlocked(binding->attachment, binding->sgt, + DMA_FROM_DEVICE); + dma_buf_detach(binding->dmabuf, binding->attachment); + dma_buf_put(binding->dmabuf); + xa_destroy(&binding->bound_rxqs); + kfree(binding); +} + +void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) +{ + struct netdev_rx_queue *rxq; + unsigned long xa_idx; + unsigned int rxq_idx; + + if (binding->list.next) + list_del(&binding->list); + + xa_for_each(&binding->bound_rxqs, xa_idx, rxq) { + WARN_ON(rxq->mp_params.mp_priv !=3D binding); + + rxq->mp_params.mp_priv =3D NULL; + + rxq_idx =3D get_netdev_rx_queue_index(rxq); + + WARN_ON(netdev_rx_queue_restart(binding->dev, rxq_idx)); + } + + xa_erase(&net_devmem_dmabuf_bindings, binding->id); + + net_devmem_dmabuf_binding_put(binding); +} + +int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, + struct net_devmem_dmabuf_binding *binding, + struct netlink_ext_ack *extack) +{ + struct netdev_rx_queue *rxq; + u32 xa_idx; + int err; + + if (rxq_idx >=3D dev->real_num_rx_queues) { + NL_SET_ERR_MSG(extack, "rx queue index out of range"); + return -ERANGE; + } + + rxq =3D __netif_get_rx_queue(dev, rxq_idx); + if (rxq->mp_params.mp_priv) { + NL_SET_ERR_MSG(extack, "designated queue already memory provider bound"); + return -EEXIST; + } + +#ifdef CONFIG_XDP_SOCKETS + if (rxq->pool) { + NL_SET_ERR_MSG(extack, "designated queue already in use by AF_XDP"); + return -EBUSY; + } +#endif + + err =3D xa_alloc(&binding->bound_rxqs, &xa_idx, rxq, xa_limit_32b, + GFP_KERNEL); + if (err) + return err; + + rxq->mp_params.mp_priv =3D binding; + + err =3D netdev_rx_queue_restart(dev, rxq_idx); + if (err) + goto err_xa_erase; + + return 0; + +err_xa_erase: + rxq->mp_params.mp_priv =3D NULL; + xa_erase(&binding->bound_rxqs, xa_idx); + + return err; +} + +struct net_devmem_dmabuf_binding * +net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, + struct netlink_ext_ack *extack) +{ + struct net_devmem_dmabuf_binding *binding; + static u32 id_alloc_next; + struct scatterlist *sg; + struct dma_buf *dmabuf; + unsigned int sg_idx, i; + unsigned long virtual; + int err; + + dmabuf =3D dma_buf_get(dmabuf_fd); + if (IS_ERR(dmabuf)) + return ERR_CAST(dmabuf); + + binding =3D kzalloc_node(sizeof(*binding), GFP_KERNEL, + dev_to_node(&dev->dev)); + if (!binding) { + err =3D -ENOMEM; + goto err_put_dmabuf; + } + + binding->dev =3D dev; + + err =3D xa_alloc_cyclic(&net_devmem_dmabuf_bindings, &binding->id, + binding, xa_limit_32b, &id_alloc_next, + GFP_KERNEL); + if (err < 0) + goto err_free_binding; + + xa_init_flags(&binding->bound_rxqs, XA_FLAGS_ALLOC); + + refcount_set(&binding->ref, 1); + + binding->dmabuf =3D dmabuf; + + binding->attachment =3D dma_buf_attach(binding->dmabuf, dev->dev.parent); + if (IS_ERR(binding->attachment)) { + err =3D PTR_ERR(binding->attachment); + NL_SET_ERR_MSG(extack, "Failed to bind dmabuf to device"); + goto err_free_id; + } + + binding->sgt =3D dma_buf_map_attachment_unlocked(binding->attachment, + DMA_FROM_DEVICE); + if (IS_ERR(binding->sgt)) { + err =3D PTR_ERR(binding->sgt); + NL_SET_ERR_MSG(extack, "Failed to map dmabuf attachment"); + goto err_detach; + } + + /* For simplicity we expect to make PAGE_SIZE allocations, but the + * binding can be much more flexible than that. We may be able to + * allocate MTU sized chunks here. Leave that for future work... + */ + binding->chunk_pool =3D + gen_pool_create(PAGE_SHIFT, dev_to_node(&dev->dev)); + if (!binding->chunk_pool) { + err =3D -ENOMEM; + goto err_unmap; + } + + virtual =3D 0; + for_each_sgtable_dma_sg(binding->sgt, sg, sg_idx) { + dma_addr_t dma_addr =3D sg_dma_address(sg); + struct dmabuf_genpool_chunk_owner *owner; + size_t len =3D sg_dma_len(sg); + struct net_iov *niov; + + owner =3D kzalloc_node(sizeof(*owner), GFP_KERNEL, + dev_to_node(&dev->dev)); + if (!owner) { + err =3D -ENOMEM; + goto err_free_chunks; + } + + owner->base_virtual =3D virtual; + owner->base_dma_addr =3D dma_addr; + owner->num_niovs =3D len / PAGE_SIZE; + owner->binding =3D binding; + + err =3D gen_pool_add_owner(binding->chunk_pool, dma_addr, + dma_addr, len, dev_to_node(&dev->dev), + owner); + if (err) { + kfree(owner); + err =3D -EINVAL; + goto err_free_chunks; + } + + owner->niovs =3D kvmalloc_array(owner->num_niovs, + sizeof(*owner->niovs), + GFP_KERNEL); + if (!owner->niovs) { + err =3D -ENOMEM; + goto err_free_chunks; + } + + for (i =3D 0; i < owner->num_niovs; i++) { + niov =3D &owner->niovs[i]; + niov->owner =3D owner; + } + + virtual +=3D len; + } + + return binding; + +err_free_chunks: + gen_pool_for_each_chunk(binding->chunk_pool, + net_devmem_dmabuf_free_chunk_owner, NULL); + gen_pool_destroy(binding->chunk_pool); +err_unmap: + dma_buf_unmap_attachment_unlocked(binding->attachment, binding->sgt, + DMA_FROM_DEVICE); +err_detach: + dma_buf_detach(dmabuf, binding->attachment); +err_free_id: + xa_erase(&net_devmem_dmabuf_bindings, binding->id); +err_free_binding: + kfree(binding); +err_put_dmabuf: + dma_buf_put(dmabuf); + return ERR_PTR(err); +} + +void dev_dmabuf_uninstall(struct net_device *dev) +{ + struct net_devmem_dmabuf_binding *binding; + struct netdev_rx_queue *rxq; + unsigned long xa_idx; + unsigned int i; + + for (i =3D 0; i < dev->real_num_rx_queues; i++) { + binding =3D dev->_rx[i].mp_params.mp_priv; + if (!binding) + continue; + + xa_for_each(&binding->bound_rxqs, xa_idx, rxq) + if (rxq =3D=3D &dev->_rx[i]) { + xa_erase(&binding->bound_rxqs, xa_idx); + break; + } + } +} diff --git a/net/core/devmem.h b/net/core/devmem.h new file mode 100644 index 000000000000..c50f91d858dd --- /dev/null +++ b/net/core/devmem.h @@ -0,0 +1,124 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Device memory TCP support + * + * Authors: Mina Almasry + * Willem de Bruijn + * Kaiyuan Zhang + * + */ +#ifndef _NET_DEVMEM_H +#define _NET_DEVMEM_H + +struct netlink_ext_ack; + +struct net_devmem_dmabuf_binding { + struct dma_buf *dmabuf; + struct dma_buf_attachment *attachment; + struct sg_table *sgt; + struct net_device *dev; + struct gen_pool *chunk_pool; + + /* The user holds a ref (via the netlink API) for as long as they want + * the binding to remain alive. Each page pool using this binding holds + * a ref to keep the binding alive. Each allocated net_iov holds a + * ref. + * + * The binding undos itself and unmaps the underlying dmabuf once all + * those refs are dropped and the binding is no longer desired or in + * use. + */ + refcount_t ref; + + /* The list of bindings currently active. Used for netlink to notify us + * of the user dropping the bind. + */ + struct list_head list; + + /* rxq's this binding is active on. */ + struct xarray bound_rxqs; + + /* ID of this binding. Globally unique to all bindings currently + * active. + */ + u32 id; +}; + +#if defined(CONFIG_NET_DEVMEM) +/* Owner of the dma-buf chunks inserted into the gen pool. Each scatterlist + * entry from the dmabuf is inserted into the genpool as a chunk, and needs + * this owner struct to keep track of some metadata necessary to create + * allocations from this chunk. + */ +struct dmabuf_genpool_chunk_owner { + /* Offset into the dma-buf where this chunk starts. */ + unsigned long base_virtual; + + /* dma_addr of the start of the chunk. */ + dma_addr_t base_dma_addr; + + /* Array of net_iovs for this chunk. */ + struct net_iov *niovs; + size_t num_niovs; + + struct net_devmem_dmabuf_binding *binding; +}; + +void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *bi= nding); +struct net_devmem_dmabuf_binding * +net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, + struct netlink_ext_ack *extack); +void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding); +int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, + struct net_devmem_dmabuf_binding *binding, + struct netlink_ext_ack *extack); +void dev_dmabuf_uninstall(struct net_device *dev); + +static inline void +net_devmem_dmabuf_binding_get(struct net_devmem_dmabuf_binding *binding) +{ + refcount_inc(&binding->ref); +} + +static inline void +net_devmem_dmabuf_binding_put(struct net_devmem_dmabuf_binding *binding) +{ + if (!refcount_dec_and_test(&binding->ref)) + return; + + __net_devmem_dmabuf_binding_free(binding); +} + +#else +static inline void +__net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *binding) +{ +} + +static inline struct net_devmem_dmabuf_binding * +net_devmem_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd, + struct netlink_ext_ack *extack) +{ + return ERR_PTR(-EOPNOTSUPP); +} + +static inline void +net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) +{ +} + +static inline int +net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, + struct net_devmem_dmabuf_binding *binding, + struct netlink_ext_ack *extack) + +{ + return -EOPNOTSUPP; +} + +static inline void dev_dmabuf_uninstall(struct net_device *dev) +{ +} +#endif + +#endif /* _NET_DEVMEM_H */ diff --git a/net/core/netdev-genl-gen.c b/net/core/netdev-genl-gen.c index 6b7fe6035067..b28424ae06d5 100644 --- a/net/core/netdev-genl-gen.c +++ b/net/core/netdev-genl-gen.c @@ -9,6 +9,7 @@ #include "netdev-genl-gen.h" =20 #include +#include =20 /* Integer value ranges */ static const struct netlink_range_validation netdev_a_page_pool_id_range = =3D { @@ -187,4 +188,7 @@ struct genl_family netdev_nl_family __ro_after_init =3D= { .n_split_ops =3D ARRAY_SIZE(netdev_nl_ops), .mcgrps =3D netdev_nl_mcgrps, .n_mcgrps =3D ARRAY_SIZE(netdev_nl_mcgrps), + .sock_priv_size =3D sizeof(struct list_head), + .sock_priv_init =3D (void *)netdev_nl_sock_priv_init, + .sock_priv_destroy =3D (void *)netdev_nl_sock_priv_destroy, }; diff --git a/net/core/netdev-genl-gen.h b/net/core/netdev-genl-gen.h index 67c34005750c..8cda334fd042 100644 --- a/net/core/netdev-genl-gen.h +++ b/net/core/netdev-genl-gen.h @@ -10,6 +10,7 @@ #include =20 #include +#include =20 /* Common nested types */ extern const struct nla_policy netdev_page_pool_info_nl_policy[NETDEV_A_PA= GE_POOL_IFINDEX + 1]; @@ -40,4 +41,7 @@ enum { =20 extern struct genl_family netdev_nl_family; =20 +void netdev_nl_sock_priv_init(struct list_head *priv); +void netdev_nl_sock_priv_destroy(struct list_head *priv); + #endif /* _LINUX_NETDEV_GEN_H */ diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 699c34b9b03c..9153a8ab0cf8 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -3,16 +3,17 @@ #include #include #include +#include #include +#include +#include #include #include #include -#include -#include -#include =20 -#include "netdev-genl-gen.h" #include "dev.h" +#include "devmem.h" +#include "netdev-genl-gen.h" =20 struct netdev_nl_dump_ctx { unsigned long ifindex; @@ -723,10 +724,127 @@ int netdev_nl_qstats_get_dumpit(struct sk_buff *skb, return err; } =20 -/* Stub */ int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info) { + struct nlattr *tb[ARRAY_SIZE(netdev_queue_id_nl_policy)]; + struct net_devmem_dmabuf_binding *binding; + struct list_head *sock_binding_list; + u32 ifindex, dmabuf_fd, rxq_idx; + struct net_device *netdev; + struct sk_buff *rsp; + struct nlattr *attr; + int rem, err =3D 0; + void *hdr; + + if (GENL_REQ_ATTR_CHECK(info, NETDEV_A_DEV_IFINDEX) || + GENL_REQ_ATTR_CHECK(info, NETDEV_A_DMABUF_FD) || + GENL_REQ_ATTR_CHECK(info, NETDEV_A_DMABUF_QUEUES)) + return -EINVAL; + + ifindex =3D nla_get_u32(info->attrs[NETDEV_A_DEV_IFINDEX]); + dmabuf_fd =3D nla_get_u32(info->attrs[NETDEV_A_DMABUF_FD]); + + sock_binding_list =3D genl_sk_priv_get(&netdev_nl_family, + NETLINK_CB(skb).sk); + if (IS_ERR(sock_binding_list)) + return PTR_ERR(sock_binding_list); + + rsp =3D genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!rsp) + return -ENOMEM; + + hdr =3D genlmsg_iput(rsp, info); + if (!hdr) { + err =3D -EMSGSIZE; + goto err_genlmsg_free; + } + + rtnl_lock(); + + netdev =3D __dev_get_by_index(genl_info_net(info), ifindex); + if (!netdev || !netif_device_present(netdev)) { + err =3D -ENODEV; + goto err_unlock; + } + + if (dev_xdp_prog_count(netdev)) { + NL_SET_ERR_MSG(info->extack, "unable to bind dmabuf to device with XDP p= rogram attached"); + err =3D -EEXIST; + goto err_unlock; + } + + binding =3D net_devmem_bind_dmabuf(netdev, dmabuf_fd, info->extack); + if (IS_ERR(binding)) { + err =3D PTR_ERR(binding); + goto err_unlock; + } + + nla_for_each_attr_type(attr, NETDEV_A_DMABUF_QUEUES, + genlmsg_data(info->genlhdr), + genlmsg_len(info->genlhdr), rem) { + err =3D nla_parse_nested( + tb, ARRAY_SIZE(netdev_queue_id_nl_policy) - 1, attr, + netdev_queue_id_nl_policy, info->extack); + if (err < 0) + goto err_unbind; + + if (NL_REQ_ATTR_CHECK(info->extack, attr, tb, NETDEV_A_QUEUE_ID) || + NL_REQ_ATTR_CHECK(info->extack, attr, tb, NETDEV_A_QUEUE_TYPE)) { + err =3D -EINVAL; + goto err_unbind; + } + + if (nla_get_u32(tb[NETDEV_A_QUEUE_TYPE]) !=3D NETDEV_QUEUE_TYPE_RX) { + NL_SET_BAD_ATTR(info->extack, tb[NETDEV_A_QUEUE_TYPE]); + err =3D -EINVAL; + goto err_unbind; + } + + rxq_idx =3D nla_get_u32(tb[NETDEV_A_QUEUE_ID]); + + err =3D net_devmem_bind_dmabuf_to_queue(netdev, rxq_idx, binding, + info->extack); + if (err) + goto err_unbind; + } + + list_add(&binding->list, sock_binding_list); + + nla_put_u32(rsp, NETDEV_A_DMABUF_ID, binding->id); + genlmsg_end(rsp, hdr); + + err =3D genlmsg_reply(rsp, info); + if (err) + goto err_unbind; + + rtnl_unlock(); + return 0; + +err_unbind: + net_devmem_unbind_dmabuf(binding); +err_unlock: + rtnl_unlock(); +err_genlmsg_free: + nlmsg_free(rsp); + return err; +} + +void netdev_nl_sock_priv_init(struct list_head *priv) +{ + INIT_LIST_HEAD(priv); +} + +void netdev_nl_sock_priv_destroy(struct list_head *priv) +{ + struct net_devmem_dmabuf_binding *binding; + struct net_devmem_dmabuf_binding *temp; + + list_for_each_entry_safe(binding, temp, priv, list) { + rtnl_lock(); + net_devmem_unbind_dmabuf(binding); + rtnl_unlock(); + } } =20 static int netdev_genl_netdevice_event(struct notifier_block *nb, diff --git a/net/ethtool/common.c b/net/ethtool/common.c index 781834ef57c3..91203851a476 100644 --- a/net/ethtool/common.c +++ b/net/ethtool/common.c @@ -654,6 +654,7 @@ int ethtool_check_max_channel(struct net_device *dev, { u64 max_rxnfc_in_use; u32 max_rxfh_in_use; + int max_mp_in_use; =20 /* ensure the new Rx count fits within the configured Rx flow * indirection table/rxnfc settings @@ -672,6 +673,13 @@ int ethtool_check_max_channel(struct net_device *dev, return -EINVAL; } =20 + max_mp_in_use =3D dev_get_min_mp_channel_count(dev); + if (channels.combined_count + channels.rx_count <=3D max_mp_in_use) { + if (info) + GENL_SET_ERR_MSG_FMT(info, "requested channel counts are too low for ex= isting memory provider setting (%d)", max_mp_in_use); + return -EINVAL; + } + return 0; } =20 diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c index c0e0204b9630..6b2756f95629 100644 --- a/net/xdp/xsk_buff_pool.c +++ b/net/xdp/xsk_buff_pool.c @@ -211,6 +211,11 @@ int xp_assign_dev(struct xsk_buff_pool *pool, goto err_unreg_pool; } =20 + if (dev_get_min_mp_channel_count(netdev)) { + err =3D -EBUSY; + goto err_unreg_pool; + } + bpf.command =3D XDP_SETUP_XSK_POOL; bpf.xsk.pool =3D pool; bpf.xsk.queue_id =3D queue_id; --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C424E1AD9E2 for ; Mon, 9 Sep 2024 05:43:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860617; cv=none; b=ZXuM1YEo6PwRGV1BGDcWwXdoHeR5EoftJnYutj8GERTpRv4+js1bHNXekjUB06fRa+afo5DMohilk/1jxH239xvDu/HEvwgglROejHILRStp5KDNfy3ByE9/StHO2iDdbXTIVgesn8GJnfyto8z0UM1sf34zZCYLEYour4KLT00= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860617; c=relaxed/simple; bh=WeTDJ7ZZ1u9c8OCH/DRwuCCCr2tmlYwQobBUfuuwPY8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FQANaRe7oTFPlGGnbK3yaidxk2igUJlgMgxJ2YEX2zsdCUsLl9Gm1OSFPH5vnshcMnd0eYYl4o1uklW6m2gc5AnF5jZgaNw7FZuNmbLSW5camcTuf1KtzjWWMSKaOBAKwti6+nzckUsOW1Vf569R4ci4/m37AfbwhG4ySiA5MDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=t5K3PGdP; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="t5K3PGdP" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e165fc5d94fso9023642276.2 for ; Sun, 08 Sep 2024 22:43:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860609; x=1726465409; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ECIeZR9RzpNIy7i73AGnhxjov/QI8UJt0jpUvT+70vI=; b=t5K3PGdPt/z6TH4R0diqGEYCEOpL5gAnad9t38DA/ZTVH3fcr8BzajKSnp5t6r4NnK XAjpRUEEtxUQWl+c5qmYv6vqhZzjjKwiJYL2+tFe8bXpP5XA9/V0Z9Ozwxr051bOJ1Cc L8dmvNNHoCxm4fkTC92qzl3BNFB/5HQ4gv3nJSF1n5hvNdS2krQaBAm4lCDA5/QFJTQL JgXXIwHUcU7+zta7BXSX/inf3p/42n2ehbB6jhjGuULH3zy6Rmg7jKfrhagLIGB8j0dv 8Alf+zTJfz6xH8crT2Tw9figfu4tl/ZIbQ4i2MU3vx70UEDoCorLG9/k4kdSrKkdcJXG Ej7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860609; x=1726465409; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ECIeZR9RzpNIy7i73AGnhxjov/QI8UJt0jpUvT+70vI=; b=e9Fn0C2Hd+sbBOtM/+GDkT13IIydYcPOzn8MrI6ju9HjZqr/Na7WHJ5A9zEzxuFqk7 QcrGD7fHyjeV1nDlFcNVBGBw5B7W+U7Ls8XQ1pivmd2igakgjHNuerPDUIH5+TUElug2 9IOfB2qLuxZhSj9JjSZUze9h23VXBPKanvg98ht2wuMUPuCFg55hK0ASV//TGfJpKraf FK5cgU5Z9Ji0qXy0xtN5CP+ZXFVyvfyRwYZgkJTcQuaEBmAWSrYo5f0uiXEd0CJlftVS Yc/1CbWFg54lWGnbQI1bz85xF0Fms/HyVsN/XYeZvBL0eFjm2zoSa+Tw7rAyGBd92bH7 J/Yg== X-Forwarded-Encrypted: i=1; AJvYcCXbYShXhIdb5woafQCiIOBvPJ/2vozhlejX3WP6g8eT8kGSnyClu41oZxsgvZHZEsD6QyRlZgenvMFSyQs=@vger.kernel.org X-Gm-Message-State: AOJu0YynMF/tvmmyNEsRAkT+OkQlRXeAQUAHx4Px4So2FourXvYbJU9C Pw/h7vTmkWHfIH4bvVAGHuOf9btjFuKvqwZcMn4/LLeNm8M6zgdQZhV3CXfzTzInmYGqENjjTtV J6fHX04temdfB3mUvGWfgsg== X-Google-Smtp-Source: AGHT+IH6BBD54EtEYwpncKGXuiMb46yGAk99MKkGAmBV/dBkcwIDJg6JBjgLXyLNhVw8tz5j31vVfWXujpW14zoBJw== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a25:abb3:0:b0:e1d:21ae:3b95 with SMTP id 3f1490d57ef6-e1d34a7232cmr22362276.10.1725860609193; Sun, 08 Sep 2024 22:43:29 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:09 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-5-almasrymina@google.com> Subject: [PATCH net-next v25 04/13] netdev: netdevice devmem allocator From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Willem de Bruijn , Kaiyuan Zhang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement netdev devmem allocator. The allocator takes a given struct netdev_dmabuf_binding as input and allocates net_iov from that binding. The allocation simply delegates to the binding's genpool for the allocation logic and wraps the returned memory region in a net_iov struct. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v23: - WARN_ON when we don't see the dma_addr in the gen_pool (Jakub) v20: - Removed dma_addr field in dmabuf_genpool_chunk_owner not used in this patch (moved to later patch where it's used). v19: - Don't reset dma_addr on allocation/free (Jakub) v17: - Don't acquire a binding ref for every allocation (Jakub). v11: - Fix extraneous inline directive (Paolo) v8: - Rename netdev_dmabuf_binding -> net_devmem_dmabuf_binding to avoid patch-by-patch build error. - Move niov->pp_magic/pp/pp_ref_counter usage to later patch to avoid patch-by-patch build error. v7: - netdev_ -> net_devmem_* naming (Yunsheng). v6: - Add comment on net_iov_dma_addr to explain why we don't use niov->dma_addr (Pavel) - Refactor new functions into net/core/devmem.c (Pavel) v1: - Rename devmem -> dmabuf (David). fix allocator --- net/core/devmem.c | 41 +++++++++++++++++++++++++++++++++++++++++ net/core/devmem.h | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) diff --git a/net/core/devmem.c b/net/core/devmem.c index 8dd7beb080d2..9beb03763dc9 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -34,6 +34,14 @@ static void net_devmem_dmabuf_free_chunk_owner(struct ge= n_pool *genpool, kfree(owner); } =20 +static dma_addr_t net_devmem_get_dma_addr(const struct net_iov *niov) +{ + struct dmabuf_genpool_chunk_owner *owner =3D net_iov_owner(niov); + + return owner->base_dma_addr + + ((dma_addr_t)net_iov_idx(niov) << PAGE_SHIFT); +} + void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *bi= nding) { size_t size, avail; @@ -56,6 +64,39 @@ void __net_devmem_dmabuf_binding_free(struct net_devmem_= dmabuf_binding *binding) kfree(binding); } =20 +struct net_iov * +net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding) +{ + struct dmabuf_genpool_chunk_owner *owner; + unsigned long dma_addr; + struct net_iov *niov; + ssize_t offset; + ssize_t index; + + dma_addr =3D gen_pool_alloc_owner(binding->chunk_pool, PAGE_SIZE, + (void **)&owner); + if (!dma_addr) + return NULL; + + offset =3D dma_addr - owner->base_dma_addr; + index =3D offset / PAGE_SIZE; + niov =3D &owner->niovs[index]; + + return niov; +} + +void net_devmem_free_dmabuf(struct net_iov *niov) +{ + struct net_devmem_dmabuf_binding *binding =3D net_iov_binding(niov); + unsigned long dma_addr =3D net_devmem_get_dma_addr(niov); + + if (WARN_ON(!gen_pool_has_addr(binding->chunk_pool, dma_addr, + PAGE_SIZE))) + return; + + gen_pool_free(binding->chunk_pool, dma_addr, PAGE_SIZE); +} + void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) { struct netdev_rx_queue *rxq; diff --git a/net/core/devmem.h b/net/core/devmem.h index c50f91d858dd..b1db4877cff9 100644 --- a/net/core/devmem.h +++ b/net/core/devmem.h @@ -74,6 +74,23 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *d= ev, u32 rxq_idx, struct netlink_ext_ack *extack); void dev_dmabuf_uninstall(struct net_device *dev); =20 +static inline struct dmabuf_genpool_chunk_owner * +net_iov_owner(const struct net_iov *niov) +{ + return niov->owner; +} + +static inline unsigned int net_iov_idx(const struct net_iov *niov) +{ + return niov - net_iov_owner(niov)->niovs; +} + +static inline struct net_devmem_dmabuf_binding * +net_iov_binding(const struct net_iov *niov) +{ + return net_iov_owner(niov)->binding; +} + static inline void net_devmem_dmabuf_binding_get(struct net_devmem_dmabuf_binding *binding) { @@ -89,7 +106,13 @@ net_devmem_dmabuf_binding_put(struct net_devmem_dmabuf_= binding *binding) __net_devmem_dmabuf_binding_free(binding); } =20 +struct net_iov * +net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding); +void net_devmem_free_dmabuf(struct net_iov *ppiov); + #else +struct net_devmem_dmabuf_binding; + static inline void __net_devmem_dmabuf_binding_free(struct net_devmem_dmabuf_binding *binding) { @@ -119,6 +142,17 @@ net_devmem_bind_dmabuf_to_queue(struct net_device *dev= , u32 rxq_idx, static inline void dev_dmabuf_uninstall(struct net_device *dev) { } + +static inline struct net_iov * +net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding) +{ + return NULL; +} + +static inline void net_devmem_free_dmabuf(struct net_iov *ppiov) +{ +} + #endif =20 #endif /* _NET_DEVMEM_H */ --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81CE41AE03B for ; Mon, 9 Sep 2024 05:43:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860619; cv=none; b=Vv51GjCbEO3NcFmaAmaudb4sPJW+vPgK/kiWUSj9MD1GzJIZGMzTNd6RQx+uPtrfQ6SipQMsWENenAlDTxUFR9z0W+lYmvrNVIRtzG2pjcfIWJwaKd1zm9PMVfr1pQbO/kzLveIUlZvwr07ktiWlKPu4NlpBe+Q9bkwuWuBi1F4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860619; c=relaxed/simple; bh=92Mg9ao8pgMq0lV+quFozv+Bagc20nKsuvj4GXxMu6g=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j72nYUUbHr3VYZ7EmgJfuYJ9jVn4ZR+jyiQ/UaJCc1USjdu1qpAASiM8a+y+0S2YBZGlIkrzy7Vz9LoS+W7K77v1AU1VHg7YuZKgyvs/+tPomIuVF1a8lnMLA13VVS+SrWqLJ6RXHax7fZvQr54zjsfaNVRH+JIveGMS7Mpdh1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=KrGGs0Zp; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="KrGGs0Zp" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6d7124939beso122415477b3.2 for ; Sun, 08 Sep 2024 22:43:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860611; x=1726465411; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=r3md9XYWNacDnNgHnHWg+BlEot4za/lTSUrQPQIgvmI=; b=KrGGs0ZpS7OPxEdUOleslFQIwWZ/8DgrUQuOeiyF2eGgzLmN3I+Qio2j07QrKrfmDk 5FkOY2Im0V7XOQr1Myshh+YT7r22D9YN+9GAVSX2a/R0UCTf8EKUJthQ9WT5wV8d/SDv SciCTDVvr6ugN2WKmOnqIYu1qM/RlCprNhj3VuRZDgaHJkFRr9yOMxgC5GQOXVI+Jmt3 c/j9SDsYVzHE8IounTmhI8uro8ZgJN4iRSY0KYwIKGYDSsHOcHklV4ceInwi4c5VJBCd 4l7Kjv+C5O/fTX2C0vZ68IYyeaw0CmK+zCi2VXLmfDxw/pZXb4ipr7nwjY6cKnIuM8l1 +3Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860611; x=1726465411; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r3md9XYWNacDnNgHnHWg+BlEot4za/lTSUrQPQIgvmI=; b=auqsodkxEyhGedclj3W4ioiWbkWgcMquTABsLeoDaYZj4GtAmyJLOy+oUpPJytso92 myFMqm1R5LvdBpIIZloRFTGI6y2/vKMagUrgumNYXcQ13Ew5DKY4MxPqTrvHlFH3qLtj +xyaJg+F5N4+qoWj6IRXEC0dFoAbC3yUeuGqjz0MOk2fsLIehfVJfUaXhDnSyTqc56by bP/jxYwoqr1/vnVFmiSbkJyKbDY0czfIV0CGxT09DJCQF0sEZZsH6oPhCXLEuf9DZCho OzhXGgLPCWne0eAfU26meVMq+lEeUPOzaJ+s9G3QLnYPIc95IZt8MHtKiW7+gd5ZP9KE Tjkw== X-Forwarded-Encrypted: i=1; AJvYcCX5aYu4PyJVctZcJVclJQeBFsl0FTr+7YDymsxhz6J5dERJ0gJjkZYcRomERIrJNxLgMIfvfQGhGxRmedg=@vger.kernel.org X-Gm-Message-State: AOJu0YzdQbpyGzlYRH3jpYhFXUYfigir6XXbtrhQE0yVVeAAODBco8+A GC9O7v5YoSI3s2TVXhLnNRSZ1N20kzVIJcRgeGiHIUAyXBgE9HQrZ3KGmLi1r4s1Lk9dNMcw2zJ cuvmzimpX+F2GQn3cSWC/YQ== X-Google-Smtp-Source: AGHT+IGT3FoRP4BxN1iRsAHuXW+gOVzfhhajdtzzP/nM4j8MYO02dIom0GLvQH2gPajcdGdIv1NJugsCPSpL5L05tg== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a05:690c:6c81:b0:69b:c01:82a5 with SMTP id 00721157ae682-6db4526c2c0mr4910817b3.7.1725860611273; Sun, 08 Sep 2024 22:43:31 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:10 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-6-almasrymina@google.com> Subject: [PATCH net-next v25 05/13] page_pool: devmem support From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , linux-mm@kvack.org, Matthew Wilcox Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert netmem to be a union of struct page and struct netmem. Overload the LSB of struct netmem* to indicate that it's a net_iov, otherwise it's a page. Currently these entries in struct page are rented by the page_pool and used exclusively by the net stack: struct { unsigned long pp_magic; struct page_pool *pp; unsigned long _pp_mapping_pad; unsigned long dma_addr; atomic_long_t pp_ref_count; }; Mirror these (and only these) entries into struct net_iov and implement netmem helpers that can access these common fields regardless of whether the underlying type is page or net_iov. Implement checks for net_iov in netmem helpers which delegate to mm APIs, to ensure net_iov are never passed to the mm stack. Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov Acked-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v23: - Fix comment on netmem_is_perf_nid (Jakub) v19: - Move page_pool_set_dma_addr(_netmem) to page_pool_priv.h - Don't reset niov dma_addr on allocation/free. Instead, it's set once when the binding happens and it never changes (Jakub) v17: - Rename netmem_to_pfn to netmem_pfn_trace (Jakub) - Move some low level netmem helpers to netmem_priv.h (Jakub). v13: - Move NET_IOV dependent changes to this patch. - Fixed comment (Pavel) - Applied Reviewed-by from Pavel. v9: https://lore.kernel.org/netdev/20240403002053.2376017-8-almasrymina@goo= gle.com/ - Remove CONFIG checks in netmem_is_net_iov() (Pavel/David/Jens) v7: - Remove static_branch_unlikely from netmem_to_net_iov(). We're getting better results from the fast path in bench_page_pool_simple tests without the static_branch_unlikely, and the addition of static_branch_unlikely doesn't improve performance of devmem TCP. Additionally only check netmem_to_net_iov() if CONFIG_DMA_SHARED_BUFFER is enabled, otherwise dmabuf net_iovs cannot exist anyway. net-next base: 8 cycle fast path. with static_branch_unlikely: 10 cycle fast path. without static_branch_unlikely: 9 cycle fast path. CONFIG_DMA_SHARED_BUFFER disabled: 8 cycle fast path as baseline. Performance of devmem TCP is at 95% line rate is regardless of static_branch_unlikely or not. v6: - Rebased on top of the merged netmem_ref type. - Rebased on top of the merged skb_pp_frag_ref() changes. v5: - Use netmem instead of page* with LSB set. - Use pp_ref_count for refcounting net_iov. - Removed many of the custom checks for netmem. v1: - Disable fragmentation support for iov properly. - fix napi_pp_put_page() path (Yunsheng). - Use pp_frag_count for devmem refcounting. Cc: linux-mm@kvack.org Cc: Matthew Wilcox --- include/net/netmem.h | 123 +++++++++++++++++++++++++++++-- include/net/page_pool/helpers.h | 39 ++-------- include/trace/events/page_pool.h | 12 +-- net/core/devmem.c | 7 ++ net/core/netmem_priv.h | 31 ++++++++ net/core/page_pool.c | 25 ++++--- net/core/page_pool_priv.h | 26 +++++++ net/core/skbuff.c | 23 +++--- 8 files changed, 217 insertions(+), 69 deletions(-) create mode 100644 net/core/netmem_priv.h diff --git a/include/net/netmem.h b/include/net/netmem.h index c23e224dd6a0..5eccc40df92d 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -8,12 +8,51 @@ #ifndef _NET_NETMEM_H #define _NET_NETMEM_H =20 +#include + /* net_iov */ =20 +DECLARE_STATIC_KEY_FALSE(page_pool_mem_providers); + +/* We overload the LSB of the struct page pointer to indicate whether it's + * a page or net_iov. + */ +#define NET_IOV 0x01UL + struct net_iov { + unsigned long __unused_padding; + unsigned long pp_magic; + struct page_pool *pp; struct dmabuf_genpool_chunk_owner *owner; + unsigned long dma_addr; + atomic_long_t pp_ref_count; }; =20 +/* These fields in struct page are used by the page_pool and net stack: + * + * struct { + * unsigned long pp_magic; + * struct page_pool *pp; + * unsigned long _pp_mapping_pad; + * unsigned long dma_addr; + * atomic_long_t pp_ref_count; + * }; + * + * We mirror the page_pool fields here so the page_pool can access these f= ields + * without worrying whether the underlying fields belong to a page or net_= iov. + * + * The non-net stack fields of struct page are private to the mm stack and= must + * never be mirrored to net_iov. + */ +#define NET_IOV_ASSERT_OFFSET(pg, iov) \ + static_assert(offsetof(struct page, pg) =3D=3D \ + offsetof(struct net_iov, iov)) +NET_IOV_ASSERT_OFFSET(pp_magic, pp_magic); +NET_IOV_ASSERT_OFFSET(pp, pp); +NET_IOV_ASSERT_OFFSET(dma_addr, dma_addr); +NET_IOV_ASSERT_OFFSET(pp_ref_count, pp_ref_count); +#undef NET_IOV_ASSERT_OFFSET + /* netmem */ =20 /** @@ -27,20 +66,37 @@ struct net_iov { */ typedef unsigned long __bitwise netmem_ref; =20 +static inline bool netmem_is_net_iov(const netmem_ref netmem) +{ + return (__force unsigned long)netmem & NET_IOV; +} + /* This conversion fails (returns NULL) if the netmem_ref is not struct pa= ge * backed. - * - * Currently struct page is the only possible netmem, and this helper never - * fails. */ static inline struct page *netmem_to_page(netmem_ref netmem) { + if (WARN_ON_ONCE(netmem_is_net_iov(netmem))) + return NULL; + return (__force struct page *)netmem; } =20 -/* Converting from page to netmem is always safe, because a page can alway= s be - * a netmem. - */ +static inline struct net_iov *netmem_to_net_iov(netmem_ref netmem) +{ + if (netmem_is_net_iov(netmem)) + return (struct net_iov *)((__force unsigned long)netmem & + ~NET_IOV); + + DEBUG_NET_WARN_ON_ONCE(true); + return NULL; +} + +static inline netmem_ref net_iov_to_netmem(struct net_iov *niov) +{ + return (__force netmem_ref)((unsigned long)niov | NET_IOV); +} + static inline netmem_ref page_to_netmem(struct page *page) { return (__force netmem_ref)page; @@ -48,17 +104,70 @@ static inline netmem_ref page_to_netmem(struct page *p= age) =20 static inline int netmem_ref_count(netmem_ref netmem) { + /* The non-pp refcount of net_iov is always 1. On net_iov, we only + * support pp refcounting which uses the pp_ref_count field. + */ + if (netmem_is_net_iov(netmem)) + return 1; + return page_ref_count(netmem_to_page(netmem)); } =20 -static inline unsigned long netmem_to_pfn(netmem_ref netmem) +static inline unsigned long netmem_pfn_trace(netmem_ref netmem) { + if (netmem_is_net_iov(netmem)) + return 0; + return page_to_pfn(netmem_to_page(netmem)); } =20 +static inline struct net_iov *__netmem_clear_lsb(netmem_ref netmem) +{ + return (struct net_iov *)((__force unsigned long)netmem & ~NET_IOV); +} + +static inline struct page_pool *netmem_get_pp(netmem_ref netmem) +{ + return __netmem_clear_lsb(netmem)->pp; +} + +static inline atomic_long_t *netmem_get_pp_ref_count_ref(netmem_ref netmem) +{ + return &__netmem_clear_lsb(netmem)->pp_ref_count; +} + +static inline bool netmem_is_pref_nid(netmem_ref netmem, int pref_nid) +{ + /* NUMA node preference only makes sense if we're allocating + * system memory. Memory providers (which give us net_iovs) + * choose for us. + */ + if (netmem_is_net_iov(netmem)) + return true; + + return page_to_nid(netmem_to_page(netmem)) =3D=3D pref_nid; +} + static inline netmem_ref netmem_compound_head(netmem_ref netmem) { + /* niov are never compounded */ + if (netmem_is_net_iov(netmem)) + return netmem; + return page_to_netmem(compound_head(netmem_to_page(netmem))); } =20 +static inline void *netmem_address(netmem_ref netmem) +{ + if (netmem_is_net_iov(netmem)) + return NULL; + + return page_address(netmem_to_page(netmem)); +} + +static inline unsigned long netmem_get_dma_addr(netmem_ref netmem) +{ + return __netmem_clear_lsb(netmem)->dma_addr; +} + #endif /* _NET_NETMEM_H */ diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helper= s.h index 2b43a893c619..793e6fd78bc5 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -216,7 +216,7 @@ page_pool_get_dma_dir(const struct page_pool *pool) =20 static inline void page_pool_fragment_netmem(netmem_ref netmem, long nr) { - atomic_long_set(&netmem_to_page(netmem)->pp_ref_count, nr); + atomic_long_set(netmem_get_pp_ref_count_ref(netmem), nr); } =20 /** @@ -244,7 +244,7 @@ static inline void page_pool_fragment_page(struct page = *page, long nr) =20 static inline long page_pool_unref_netmem(netmem_ref netmem, long nr) { - struct page *page =3D netmem_to_page(netmem); + atomic_long_t *pp_ref_count =3D netmem_get_pp_ref_count_ref(netmem); long ret; =20 /* If nr =3D=3D pp_ref_count then we have cleared all remaining @@ -261,19 +261,19 @@ static inline long page_pool_unref_netmem(netmem_ref = netmem, long nr) * initially, and only overwrite it when the page is partitioned into * more than one piece. */ - if (atomic_long_read(&page->pp_ref_count) =3D=3D nr) { + if (atomic_long_read(pp_ref_count) =3D=3D nr) { /* As we have ensured nr is always one for constant case using * the BUILD_BUG_ON(), only need to handle the non-constant case * here for pp_ref_count draining, which is a rare case. */ BUILD_BUG_ON(__builtin_constant_p(nr) && nr !=3D 1); if (!__builtin_constant_p(nr)) - atomic_long_set(&page->pp_ref_count, 1); + atomic_long_set(pp_ref_count, 1); =20 return 0; } =20 - ret =3D atomic_long_sub_return(nr, &page->pp_ref_count); + ret =3D atomic_long_sub_return(nr, pp_ref_count); WARN_ON(ret < 0); =20 /* We are the last user here too, reset pp_ref_count back to 1 to @@ -282,7 +282,7 @@ static inline long page_pool_unref_netmem(netmem_ref ne= tmem, long nr) * page_pool_unref_page() currently. */ if (unlikely(!ret)) - atomic_long_set(&page->pp_ref_count, 1); + atomic_long_set(pp_ref_count, 1); =20 return ret; } @@ -401,9 +401,7 @@ static inline void page_pool_free_va(struct page_pool *= pool, void *va, =20 static inline dma_addr_t page_pool_get_dma_addr_netmem(netmem_ref netmem) { - struct page *page =3D netmem_to_page(netmem); - - dma_addr_t ret =3D page->dma_addr; + dma_addr_t ret =3D netmem_get_dma_addr(netmem); =20 if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) ret <<=3D PAGE_SHIFT; @@ -423,24 +421,6 @@ static inline dma_addr_t page_pool_get_dma_addr(const = struct page *page) return page_pool_get_dma_addr_netmem(page_to_netmem((struct page *)page)); } =20 -static inline bool page_pool_set_dma_addr_netmem(netmem_ref netmem, - dma_addr_t addr) -{ - struct page *page =3D netmem_to_page(netmem); - - if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) { - page->dma_addr =3D addr >> PAGE_SHIFT; - - /* We assume page alignment to shave off bottom bits, - * if this "compression" doesn't work we need to drop. - */ - return addr !=3D (dma_addr_t)page->dma_addr << PAGE_SHIFT; - } - - page->dma_addr =3D addr; - return false; -} - /** * page_pool_dma_sync_for_cpu - sync Rx page for CPU after it's written by= HW * @pool: &page_pool the @page belongs to @@ -463,11 +443,6 @@ static inline void page_pool_dma_sync_for_cpu(const st= ruct page_pool *pool, page_pool_get_dma_dir(pool)); } =20 -static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t ad= dr) -{ - return page_pool_set_dma_addr_netmem(page_to_netmem(page), addr); -} - static inline bool page_pool_put(struct page_pool *pool) { return refcount_dec_and_test(&pool->user_cnt); diff --git a/include/trace/events/page_pool.h b/include/trace/events/page_p= ool.h index 543e54e432a1..31825ed30032 100644 --- a/include/trace/events/page_pool.h +++ b/include/trace/events/page_pool.h @@ -57,12 +57,12 @@ TRACE_EVENT(page_pool_state_release, __entry->pool =3D pool; __entry->netmem =3D (__force unsigned long)netmem; __entry->release =3D release; - __entry->pfn =3D netmem_to_pfn(netmem); + __entry->pfn =3D netmem_pfn_trace(netmem); ), =20 - TP_printk("page_pool=3D%p netmem=3D%p pfn=3D0x%lx release=3D%u", + TP_printk("page_pool=3D%p netmem=3D%p is_net_iov=3D%lu pfn=3D0x%lx releas= e=3D%u", __entry->pool, (void *)__entry->netmem, - __entry->pfn, __entry->release) + __entry->netmem & NET_IOV, __entry->pfn, __entry->release) ); =20 TRACE_EVENT(page_pool_state_hold, @@ -83,12 +83,12 @@ TRACE_EVENT(page_pool_state_hold, __entry->pool =3D pool; __entry->netmem =3D (__force unsigned long)netmem; __entry->hold =3D hold; - __entry->pfn =3D netmem_to_pfn(netmem); + __entry->pfn =3D netmem_pfn_trace(netmem); ), =20 - TP_printk("page_pool=3D%p netmem=3D%p pfn=3D0x%lx hold=3D%u", + TP_printk("page_pool=3D%p netmem=3D%p is_net_iov=3D%lu, pfn=3D0x%lx hold= =3D%u", __entry->pool, (void *)__entry->netmem, - __entry->pfn, __entry->hold) + __entry->netmem & NET_IOV, __entry->pfn, __entry->hold) ); =20 TRACE_EVENT(page_pool_update_nid, diff --git a/net/core/devmem.c b/net/core/devmem.c index 9beb03763dc9..7efeb602cf45 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -18,6 +18,7 @@ #include =20 #include "devmem.h" +#include "page_pool_priv.h" =20 /* Device memory support */ =20 @@ -82,6 +83,10 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding= *binding) index =3D offset / PAGE_SIZE; niov =3D &owner->niovs[index]; =20 + niov->pp_magic =3D 0; + niov->pp =3D NULL; + atomic_long_set(&niov->pp_ref_count, 0); + return niov; } =20 @@ -269,6 +274,8 @@ net_devmem_bind_dmabuf(struct net_device *dev, unsigned= int dmabuf_fd, for (i =3D 0; i < owner->num_niovs; i++) { niov =3D &owner->niovs[i]; niov->owner =3D owner; + page_pool_set_dma_addr_netmem(net_iov_to_netmem(niov), + net_devmem_get_dma_addr(niov)); } =20 virtual +=3D len; diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h new file mode 100644 index 000000000000..7eadb8393e00 --- /dev/null +++ b/net/core/netmem_priv.h @@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __NETMEM_PRIV_H +#define __NETMEM_PRIV_H + +static inline unsigned long netmem_get_pp_magic(netmem_ref netmem) +{ + return __netmem_clear_lsb(netmem)->pp_magic; +} + +static inline void netmem_or_pp_magic(netmem_ref netmem, unsigned long pp_= magic) +{ + __netmem_clear_lsb(netmem)->pp_magic |=3D pp_magic; +} + +static inline void netmem_clear_pp_magic(netmem_ref netmem) +{ + __netmem_clear_lsb(netmem)->pp_magic =3D 0; +} + +static inline void netmem_set_pp(netmem_ref netmem, struct page_pool *pool) +{ + __netmem_clear_lsb(netmem)->pp =3D pool; +} + +static inline void netmem_set_dma_addr(netmem_ref netmem, + unsigned long dma_addr) +{ + __netmem_clear_lsb(netmem)->dma_addr =3D dma_addr; +} +#endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 2abe6e919224..52659db2d765 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -24,8 +24,11 @@ =20 #include =20 +#include "netmem_priv.h" #include "page_pool_priv.h" =20 +DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); + #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) =20 @@ -358,7 +361,7 @@ static noinline netmem_ref page_pool_refill_alloc_cache= (struct page_pool *pool) if (unlikely(!netmem)) break; =20 - if (likely(page_to_nid(netmem_to_page(netmem)) =3D=3D pref_nid)) { + if (likely(netmem_is_pref_nid(netmem, pref_nid))) { pool->alloc.cache[pool->alloc.count++] =3D netmem; } else { /* NUMA mismatch; @@ -454,10 +457,8 @@ static bool page_pool_dma_map(struct page_pool *pool, = netmem_ref netmem) =20 static void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netme= m) { - struct page *page =3D netmem_to_page(netmem); - - page->pp =3D pool; - page->pp_magic |=3D PP_SIGNATURE; + netmem_set_pp(netmem, pool); + netmem_or_pp_magic(netmem, PP_SIGNATURE); =20 /* Ensuring all pages have been split into one fragment initially: * page_pool_set_pp_info() is only called once for every page when it @@ -472,10 +473,8 @@ static void page_pool_set_pp_info(struct page_pool *po= ol, netmem_ref netmem) =20 static void page_pool_clear_pp_info(netmem_ref netmem) { - struct page *page =3D netmem_to_page(netmem); - - page->pp_magic =3D 0; - page->pp =3D NULL; + netmem_clear_pp_magic(netmem); + netmem_set_pp(netmem, NULL); } =20 static struct page *__page_pool_alloc_page_order(struct page_pool *pool, @@ -692,8 +691,9 @@ static bool page_pool_recycle_in_cache(netmem_ref netme= m, =20 static bool __page_pool_page_can_be_recycled(netmem_ref netmem) { - return page_ref_count(netmem_to_page(netmem)) =3D=3D 1 && - !page_is_pfmemalloc(netmem_to_page(netmem)); + return netmem_is_net_iov(netmem) || + (page_ref_count(netmem_to_page(netmem)) =3D=3D 1 && + !page_is_pfmemalloc(netmem_to_page(netmem))); } =20 /* If the page refcnt =3D=3D 1, this will try to recycle the page. @@ -728,6 +728,7 @@ __page_pool_put_page(struct page_pool *pool, netmem_ref= netmem, /* Page found as candidate for recycling */ return netmem; } + /* Fallback/non-XDP mode: API user have elevated refcnt. * * Many drivers split up the page into fragments, and some @@ -949,7 +950,7 @@ static void page_pool_empty_ring(struct page_pool *pool) /* Empty recycle ring */ while ((netmem =3D (__force netmem_ref)ptr_ring_consume_bh(&pool->ring)))= { /* Verify the refcnt invariant of cached pages */ - if (!(page_ref_count(netmem_to_page(netmem)) =3D=3D 1)) + if (!(netmem_ref_count(netmem) =3D=3D 1)) pr_crit("%s() page_pool refcnt %d violation\n", __func__, netmem_ref_count(netmem)); =20 diff --git a/net/core/page_pool_priv.h b/net/core/page_pool_priv.h index 90665d40f1eb..d602c1e728c2 100644 --- a/net/core/page_pool_priv.h +++ b/net/core/page_pool_priv.h @@ -3,10 +3,36 @@ #ifndef __PAGE_POOL_PRIV_H #define __PAGE_POOL_PRIV_H =20 +#include + +#include "netmem_priv.h" + s32 page_pool_inflight(const struct page_pool *pool, bool strict); =20 int page_pool_list(struct page_pool *pool); void page_pool_detached(struct page_pool *pool); void page_pool_unlist(struct page_pool *pool); =20 +static inline bool +page_pool_set_dma_addr_netmem(netmem_ref netmem, dma_addr_t addr) +{ + if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA) { + netmem_set_dma_addr(netmem, addr >> PAGE_SHIFT); + + /* We assume page alignment to shave off bottom bits, + * if this "compression" doesn't work we need to drop. + */ + return addr !=3D (dma_addr_t)netmem_get_dma_addr(netmem) + << PAGE_SHIFT; + } + + netmem_set_dma_addr(netmem, addr); + return false; +} + +static inline bool page_pool_set_dma_addr(struct page *page, dma_addr_t ad= dr) +{ + return page_pool_set_dma_addr_netmem(page_to_netmem(page), addr); +} + #endif diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a52638363ea5..d9634ab342cc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -88,6 +88,7 @@ #include =20 #include "dev.h" +#include "netmem_priv.h" #include "sock_destructor.h" =20 #ifdef CONFIG_SKB_EXTENSIONS @@ -920,9 +921,9 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } =20 -static bool is_pp_page(struct page *page) +static bool is_pp_netmem(netmem_ref netmem) { - return (page->pp_magic & ~0x3UL) =3D=3D PP_SIGNATURE; + return (netmem_get_pp_magic(netmem) & ~0x3UL) =3D=3D PP_SIGNATURE; } =20 int skb_pp_cow_data(struct page_pool *pool, struct sk_buff **pskb, @@ -1020,9 +1021,7 @@ EXPORT_SYMBOL(skb_cow_data_for_xdp); #if IS_ENABLED(CONFIG_PAGE_POOL) bool napi_pp_put_page(netmem_ref netmem) { - struct page *page =3D netmem_to_page(netmem); - - page =3D compound_head(page); + netmem =3D netmem_compound_head(netmem); =20 /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation * in order to preserve any existing bits, such as bit 0 for the @@ -1031,10 +1030,10 @@ bool napi_pp_put_page(netmem_ref netmem) * and page_is_pfmemalloc() is checked in __page_pool_put_page() * to avoid recycling the pfmemalloc page. */ - if (unlikely(!is_pp_page(page))) + if (unlikely(!is_pp_netmem(netmem))) return false; =20 - page_pool_put_full_netmem(page->pp, page_to_netmem(page), false); + page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, false); =20 return true; } @@ -1061,7 +1060,7 @@ static bool skb_pp_recycle(struct sk_buff *skb, void = *data) static int skb_pp_frag_ref(struct sk_buff *skb) { struct skb_shared_info *shinfo; - struct page *head_page; + netmem_ref head_netmem; int i; =20 if (!skb->pp_recycle) @@ -1070,11 +1069,11 @@ static int skb_pp_frag_ref(struct sk_buff *skb) shinfo =3D skb_shinfo(skb); =20 for (i =3D 0; i < shinfo->nr_frags; i++) { - head_page =3D compound_head(skb_frag_page(&shinfo->frags[i])); - if (likely(is_pp_page(head_page))) - page_pool_ref_page(head_page); + head_netmem =3D netmem_compound_head(shinfo->frags[i].netmem); + if (likely(is_pp_netmem(head_netmem))) + page_pool_ref_netmem(head_netmem); else - page_ref_inc(head_page); + page_ref_inc(netmem_to_page(head_netmem)); } return 0; } --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 524061AD5C1 for ; Mon, 9 Sep 2024 05:43:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860622; cv=none; b=UUg3v7ltt/WlnTnWfClSy57IgAHN9JoNbxLhBGOexbY7jsgc+8VQZMvXIzY9mIHSJvgeWTd+k/AwcwqECoCIXnD2mbAv4bBhLxZUoPO2aqRFc6yY14J0ckV71lYQPDBq/zuPWQ9BBZJwH5+lftGj7vdXjDUCDCgJTzSD5SdHqw4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860622; c=relaxed/simple; bh=Wl44tIj63+fUhmECOSdYw72q021zG+rG1uQBl4hLvvU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=N6vBG4rylMxSwTTqC6DvwfxE8865gKpHSU410v/m2MWYjrLz2hhlV78JrTNzKYOCJla94Hi+EPHTjKv3KHumPGTULyYfKKN86YG7t4i8cYWiy7jKWLh27osROF9QiYvjczu99e+eEycmkYwLZNBBJi5REOgxNhXacAB7aDYXD+o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gwMQy1t+; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gwMQy1t+" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1159fb161fso9273565276.1 for ; Sun, 08 Sep 2024 22:43:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860613; x=1726465413; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Z5my9lIBS8Kt8aaeKl6dbNNkzL2f+kvxRUOGslDtYFg=; b=gwMQy1t+7/nM7OwBE8qZgXTAe66YmnoNZsLKqzRyAKhhsPfYmVYPSVaSxKSu4Dj7Pa FJ2ZBFffqwdbEttDqWvtOO/EyFDMqlwVkh9+PINwQy/ozPXF+9mrKFkLGW7rz2/D/9o0 2YYNbg4P8Qih+ng1bru/74uPZ63BAsVELRgPuV6NNJqq7s0DdlILgcHjrmxOJc6Q4pq2 jOSJw/hYerLn7qLtB06QwF4Ql98LTurOXxNm7PQ5/fgawjQOn4H261ZCXrfK5cjHWE0Y vbm9hEJzWDD+X/cUfjOVauD00QxLZJRcaAdgb1llrm9gYYabFIlXaI0ISKGTQcLZi48C id5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860613; x=1726465413; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Z5my9lIBS8Kt8aaeKl6dbNNkzL2f+kvxRUOGslDtYFg=; b=JbCjXQc5dgLBJWjMGGdSeQKdpWrgNmUXr3EgqjpV1vWhN1sdIsBR8JrycwHdJapnIX djWv8SLAmb/XdiQPih4wEgjstbIZsd9C/a5D8ppziRLuPJWfIXBP+DGAeD16sXef3YzH ycM2ftdKynoii/4CyB4EJ50JWr7Ppi19yYccc7ljyoSdzmoZnGaX0PXhs0kgTKIezeVZ n7U2uG5FxAhAFRKISkp+GXwj6t4CEum8ezBorf27x4gzScN6fcQ50BWz39hmCTiaxfMG fAqdYpRfk2t7jto+mavmpCj3JxDvHUz2C/aJXlHKWvPLE9B0ZZDxIfZHvVbQ8GlB4MNS H67Q== X-Forwarded-Encrypted: i=1; AJvYcCWPEaZfR72zZTsPrL9P56FoOG2WKZoWwtcYsug519jdlzrGwyzl8u0Xs5rJWtR1kNpMNiK6M3h61dEq6Gk=@vger.kernel.org X-Gm-Message-State: AOJu0YzmeT/kazIbxkQCj12CYyQjc6g6nVsCx0713fuYar7La0c1j5ND oEFYz9Y1yaA0+uT/XlyTe2mIp0im6MDUkY3AeNZivB5tra+wrfxvqjeizgJLhEeaDrC6UzDXPjC l/bYYkTJADmvsousHyLyw5Q== X-Google-Smtp-Source: AGHT+IG6Opvhjlb6+C6OdNH39NhbTND/s060Br2RN0O6b6IdgeCsfaCjK4AvOPTEVXT00sLdSZzrIYqlDKJBteQ0iw== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a25:86c1:0:b0:e0e:4350:d7de with SMTP id 3f1490d57ef6-e1d349dae5dmr16707276.9.1725860613267; Sun, 08 Sep 2024 22:43:33 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:11 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-7-almasrymina@google.com> Subject: [PATCH net-next v25 06/13] memory-provider: dmabuf devmem memory provider From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Willem de Bruijn , Kaiyuan Zhang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement a memory provider that allocates dmabuf devmem in the form of net_iov. The provider receives a reference to the struct netdev_dmabuf_binding via the pool->mp_priv pointer. The driver needs to set this pointer for the provider in the net_iov. The provider obtains a reference on the netdev_dmabuf_binding which guarantees the binding and the underlying mapping remains alive until the provider is destroyed. Usage of PP_FLAG_DMA_MAP is required for this memory provide such that the page_pool can provide the driver with the dma-addrs of the devmem. Support for PP_FLAG_DMA_SYNC_DEV is omitted for simplicity & p.order !=3D 0. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v25: - Change page_pool_param netdev_rx_queue struct to queue_idx (Jakub) - Address nits (Jakub) - Move mp_dmabuf_devmem.h to net/core (jakub). v23: - Sort includes (Jakub) - Add missing linux/mm.h include found after sorting. v21: - Provide empty definitions of functions moved to page_pool_priv.h, so that the build still succeeds when CONFIG_PAGE_POOL is not set. v20: - Moved queue pp_params field from fast path entries to slow path entries. - Moved page_pool_check_memory_provider() call to inside netdev_rx_queue_restart (Pavel). - Removed binding arg to page_pool_check_memory_provider() (Pavel). - Removed unnecessary includes from page_pool.c - Removed EXPORT_SYMBOL(page_pool_mem_providers) (Jakub) - Check pool->slow.queue instead of walking binding xarray (Pavel & Jakub). v19: - Add PP_FLAG_ALLOW_UNREADABLE_NETMEM flag. It serves 2 purposes, (a) it guards drivers that don't support unreadable netmem (net_iov backed) from accidentally getting exposed to it, and (b) drivers that wish to create header pools can unset it for that pool to force readable netmem. - Add page_pool_check_memory_provider, which verifies that the driver has created a page_pool with the expected configuration. This is used to report to the user if the mp configuration succeeded, and also verify that the driver is doing the right thing. - Don't reset niov->dma_addr on allocation/free. v17: - Use ASSERT_RTNL (Jakub) v16: - Add DEBUG_NET_WARN_ON_ONCE(!rtnl_is_locked()), to catch cases if page_pool_init without rtnl_locking when the queue is provided. In this case, the queue configuration may be changed while we're initing the page_pool, which could be a race. v13: - Return on warning (Pavel). - Fixed pool->recycle_stats not being freed on error (Pavel). - Applied reviewed-by from Pavel. v11: - Rebase to not use the ops. (Christoph) v8: - Use skb_frag_size instead of frag->bv_len to fix patch-by-patch build error v6: - refactor new memory provider functions into net/core/devmem.c (Pavel) v2: - Disable devmem for p.order !=3D 0 v1: - static_branch check in page_is_page_pool_iov() (Willem & Paolo). - PP_DEVMEM -> PP_IOV (David). - Require PP_FLAG_DMA_MAP (Jakub). --- include/net/netmem.h | 1 + include/net/page_pool/types.h | 17 +++++- net/core/devmem.c | 67 ++++++++++++++++++++++ net/core/mp_dmabuf_devmem.h | 44 +++++++++++++++ net/core/netdev_rx_queue.c | 7 +++ net/core/page_pool.c | 103 +++++++++++++++++++++++++--------- net/core/page_pool_priv.h | 20 +++++++ net/core/page_pool_user.c | 27 ++++++++- 8 files changed, 257 insertions(+), 29 deletions(-) create mode 100644 net/core/mp_dmabuf_devmem.h diff --git a/include/net/netmem.h b/include/net/netmem.h index 5eccc40df92d..8a6e20be4b9d 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -8,6 +8,7 @@ #ifndef _NET_NETMEM_H #define _NET_NETMEM_H =20 +#include #include =20 /* net_iov */ diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index 4afd6dd56351..c022c410abe3 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -20,8 +20,18 @@ * device driver responsibility */ #define PP_FLAG_SYSTEM_POOL BIT(2) /* Global system page_pool */ + +/* Allow unreadable (net_iov backed) netmem in this page_pool. Drivers set= ting + * this must be able to support unreadable netmem, where netmem_address() = would + * return NULL. This flag should not be set for header page_pools. + * + * If the driver sets PP_FLAG_ALLOW_UNREADABLE_NETMEM, it should also set + * page_pool_params.slow.queue_idx. + */ +#define PP_FLAG_ALLOW_UNREADABLE_NETMEM BIT(3) + #define PP_FLAG_ALL (PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV | \ - PP_FLAG_SYSTEM_POOL) + PP_FLAG_SYSTEM_POOL | PP_FLAG_ALLOW_UNREADABLE_NETMEM) =20 /* * Fast allocation side cache array/stack @@ -57,7 +67,9 @@ struct pp_alloc_cache { * @offset: DMA sync address offset for PP_FLAG_DMA_SYNC_DEV * @slow: params with slowpath access only (initialization and Netlink) * @netdev: netdev this pool will serve (leave as NULL if none or multiple) - * @flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV, PP_FLAG_SYSTEM_POOL + * @queue_idx: queue idx this page_pool is being created for. + * @flags: PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV, PP_FLAG_SYSTEM_POOL, + * PP_FLAG_ALLOW_UNREADABLE_NETMEM. */ struct page_pool_params { struct_group_tagged(page_pool_params_fast, fast, @@ -72,6 +84,7 @@ struct page_pool_params { ); struct_group_tagged(page_pool_params_slow, slow, struct net_device *netdev; + unsigned int queue_idx; unsigned int flags; /* private: used by test code only */ void (*init_callback)(netmem_ref netmem, void *arg); diff --git a/net/core/devmem.c b/net/core/devmem.c index 7efeb602cf45..11b91c12ee11 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -18,6 +18,7 @@ #include =20 #include "devmem.h" +#include "mp_dmabuf_devmem.h" #include "page_pool_priv.h" =20 /* Device memory support */ @@ -320,3 +321,69 @@ void dev_dmabuf_uninstall(struct net_device *dev) } } } + +/*** "Dmabuf devmem memory provider" ***/ + +int mp_dmabuf_devmem_init(struct page_pool *pool) +{ + struct net_devmem_dmabuf_binding *binding =3D pool->mp_priv; + + if (!binding) + return -EINVAL; + + if (!pool->dma_map) + return -EOPNOTSUPP; + + if (pool->dma_sync) + return -EOPNOTSUPP; + + if (pool->p.order !=3D 0) + return -E2BIG; + + net_devmem_dmabuf_binding_get(binding); + return 0; +} + +netmem_ref mp_dmabuf_devmem_alloc_netmems(struct page_pool *pool, gfp_t gf= p) +{ + struct net_devmem_dmabuf_binding *binding =3D pool->mp_priv; + struct net_iov *niov; + netmem_ref netmem; + + niov =3D net_devmem_alloc_dmabuf(binding); + if (!niov) + return 0; + + netmem =3D net_iov_to_netmem(niov); + + page_pool_set_pp_info(pool, netmem); + + pool->pages_state_hold_cnt++; + trace_page_pool_state_hold(pool, netmem, pool->pages_state_hold_cnt); + return netmem; +} + +void mp_dmabuf_devmem_destroy(struct page_pool *pool) +{ + struct net_devmem_dmabuf_binding *binding =3D pool->mp_priv; + + net_devmem_dmabuf_binding_put(binding); +} + +bool mp_dmabuf_devmem_release_page(struct page_pool *pool, netmem_ref netm= em) +{ + long refcount =3D atomic_long_read(netmem_get_pp_ref_count_ref(netmem)); + + if (WARN_ON_ONCE(!netmem_is_net_iov(netmem))) + return false; + + if (WARN_ON_ONCE(refcount !=3D 1)) + return false; + + page_pool_clear_pp_info(netmem); + + net_devmem_free_dmabuf(netmem_to_net_iov(netmem)); + + /* We don't want the page pool put_page()ing our net_iovs. */ + return false; +} diff --git a/net/core/mp_dmabuf_devmem.h b/net/core/mp_dmabuf_devmem.h new file mode 100644 index 000000000000..67cd0dd7319c --- /dev/null +++ b/net/core/mp_dmabuf_devmem.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Dmabuf device memory provider. + * + * Authors: Mina Almasry + * + */ +#ifndef _NET_MP_DMABUF_DEVMEM_H +#define _NET_MP_DMABUF_DEVMEM_H + +#include + +#if defined(CONFIG_NET_DEVMEM) +int mp_dmabuf_devmem_init(struct page_pool *pool); + +netmem_ref mp_dmabuf_devmem_alloc_netmems(struct page_pool *pool, gfp_t gf= p); + +void mp_dmabuf_devmem_destroy(struct page_pool *pool); + +bool mp_dmabuf_devmem_release_page(struct page_pool *pool, netmem_ref netm= em); +#else +static inline int mp_dmabuf_devmem_init(struct page_pool *pool) +{ + return -EOPNOTSUPP; +} + +static inline netmem_ref +mp_dmabuf_devmem_alloc_netmems(struct page_pool *pool, gfp_t gfp) +{ + return 0; +} + +static inline void mp_dmabuf_devmem_destroy(struct page_pool *pool) +{ +} + +static inline bool +mp_dmabuf_devmem_release_page(struct page_pool *pool, netmem_ref netmem) +{ + return false; +} +#endif + +#endif /* _NET_MP_DMABUF_DEVMEM_H */ diff --git a/net/core/netdev_rx_queue.c b/net/core/netdev_rx_queue.c index da11720a5983..e217a5838c87 100644 --- a/net/core/netdev_rx_queue.c +++ b/net/core/netdev_rx_queue.c @@ -4,8 +4,11 @@ #include #include =20 +#include "page_pool_priv.h" + int netdev_rx_queue_restart(struct net_device *dev, unsigned int rxq_idx) { + struct netdev_rx_queue *rxq =3D __netif_get_rx_queue(dev, rxq_idx); void *new_mem, *old_mem; int err; =20 @@ -31,6 +34,10 @@ int netdev_rx_queue_restart(struct net_device *dev, unsi= gned int rxq_idx) if (err) goto err_free_old_mem; =20 + err =3D page_pool_check_memory_provider(dev, rxq); + if (err) + goto err_free_new_queue_mem; + err =3D dev->queue_mgmt_ops->ndo_queue_stop(dev, old_mem, rxq_idx); if (err) goto err_free_new_queue_mem; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 52659db2d765..c737200f4fac 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -11,6 +11,7 @@ #include #include =20 +#include #include #include =20 @@ -24,8 +25,10 @@ =20 #include =20 +#include "mp_dmabuf_devmem.h" #include "netmem_priv.h" #include "page_pool_priv.h" +#include "mp_dmabuf_devmem.h" =20 DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers); =20 @@ -190,6 +193,8 @@ static int page_pool_init(struct page_pool *pool, int cpuid) { unsigned int ring_qsize =3D 1024; /* Default */ + struct netdev_rx_queue *rxq; + int err; =20 page_pool_struct_check(); =20 @@ -271,7 +276,37 @@ static int page_pool_init(struct page_pool *pool, if (pool->dma_map) get_device(pool->p.dev); =20 + if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) { + /* We rely on rtnl_lock()ing to make sure netdev_rx_queue + * configuration doesn't change while we're initializing + * the page_pool. + */ + ASSERT_RTNL(); + rxq =3D __netif_get_rx_queue(pool->slow.netdev, + pool->slow.queue_idx); + pool->mp_priv =3D rxq->mp_params.mp_priv; + } + + if (pool->mp_priv) { + err =3D mp_dmabuf_devmem_init(pool); + if (err) { + pr_warn("%s() mem-provider init failed %d\n", __func__, + err); + goto free_ptr_ring; + } + + static_branch_inc(&page_pool_mem_providers); + } + return 0; + +free_ptr_ring: + ptr_ring_cleanup(&pool->ring, NULL); +#ifdef CONFIG_PAGE_POOL_STATS + if (!pool->system) + free_percpu(pool->recycle_stats); +#endif + return err; } =20 static void page_pool_uninit(struct page_pool *pool) @@ -455,28 +490,6 @@ static bool page_pool_dma_map(struct page_pool *pool, = netmem_ref netmem) return false; } =20 -static void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netme= m) -{ - netmem_set_pp(netmem, pool); - netmem_or_pp_magic(netmem, PP_SIGNATURE); - - /* Ensuring all pages have been split into one fragment initially: - * page_pool_set_pp_info() is only called once for every page when it - * is allocated from the page allocator and page_pool_fragment_page() - * is dirtying the same cache line as the page->pp_magic above, so - * the overhead is negligible. - */ - page_pool_fragment_netmem(netmem, 1); - if (pool->has_init_callback) - pool->slow.init_callback(netmem, pool->slow.init_arg); -} - -static void page_pool_clear_pp_info(netmem_ref netmem) -{ - netmem_clear_pp_magic(netmem); - netmem_set_pp(netmem, NULL); -} - static struct page *__page_pool_alloc_page_order(struct page_pool *pool, gfp_t gfp) { @@ -572,7 +585,10 @@ netmem_ref page_pool_alloc_netmem(struct page_pool *po= ol, gfp_t gfp) return netmem; =20 /* Slow-path: cache empty, do real allocation */ - netmem =3D __page_pool_alloc_pages_slow(pool, gfp); + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_priv) + netmem =3D mp_dmabuf_devmem_alloc_netmems(pool, gfp); + else + netmem =3D __page_pool_alloc_pages_slow(pool, gfp); return netmem; } EXPORT_SYMBOL(page_pool_alloc_netmem); @@ -608,6 +624,28 @@ s32 page_pool_inflight(const struct page_pool *pool, b= ool strict) return inflight; } =20 +void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem) +{ + netmem_set_pp(netmem, pool); + netmem_or_pp_magic(netmem, PP_SIGNATURE); + + /* Ensuring all pages have been split into one fragment initially: + * page_pool_set_pp_info() is only called once for every page when it + * is allocated from the page allocator and page_pool_fragment_page() + * is dirtying the same cache line as the page->pp_magic above, so + * the overhead is negligible. + */ + page_pool_fragment_netmem(netmem, 1); + if (pool->has_init_callback) + pool->slow.init_callback(netmem, pool->slow.init_arg); +} + +void page_pool_clear_pp_info(netmem_ref netmem) +{ + netmem_clear_pp_magic(netmem); + netmem_set_pp(netmem, NULL); +} + static __always_inline void __page_pool_release_page_dma(struct page_pool = *pool, netmem_ref netmem) { @@ -636,8 +674,13 @@ static __always_inline void __page_pool_release_page_d= ma(struct page_pool *pool, void page_pool_return_page(struct page_pool *pool, netmem_ref netmem) { int count; + bool put; =20 - __page_pool_release_page_dma(pool, netmem); + put =3D true; + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_priv) + put =3D mp_dmabuf_devmem_release_page(pool, netmem); + else + __page_pool_release_page_dma(pool, netmem); =20 /* This may be the last page returned, releasing the pool, so * it is not safe to reference pool afterwards. @@ -645,8 +688,10 @@ void page_pool_return_page(struct page_pool *pool, net= mem_ref netmem) count =3D atomic_inc_return_relaxed(&pool->pages_state_release_cnt); trace_page_pool_state_release(pool, netmem, count); =20 - page_pool_clear_pp_info(netmem); - put_page(netmem_to_page(netmem)); + if (put) { + page_pool_clear_pp_info(netmem); + put_page(netmem_to_page(netmem)); + } /* An optimization would be to call __free_pages(page, pool->p.order) * knowing page is not part of page-cache (thus avoiding a * __page_cache_release() call). @@ -965,6 +1010,12 @@ static void __page_pool_destroy(struct page_pool *poo= l) =20 page_pool_unlist(pool); page_pool_uninit(pool); + + if (pool->mp_priv) { + mp_dmabuf_devmem_destroy(pool); + static_branch_dec(&page_pool_mem_providers); + } + kfree(pool); } =20 diff --git a/net/core/page_pool_priv.h b/net/core/page_pool_priv.h index d602c1e728c2..57439787b9c2 100644 --- a/net/core/page_pool_priv.h +++ b/net/core/page_pool_priv.h @@ -35,4 +35,24 @@ static inline bool page_pool_set_dma_addr(struct page *p= age, dma_addr_t addr) return page_pool_set_dma_addr_netmem(page_to_netmem(page), addr); } =20 +#if defined(CONFIG_PAGE_POOL) +void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem); +void page_pool_clear_pp_info(netmem_ref netmem); +int page_pool_check_memory_provider(struct net_device *dev, + struct netdev_rx_queue *rxq); +#else +static inline void page_pool_set_pp_info(struct page_pool *pool, + netmem_ref netmem) +{ +} +static inline void page_pool_clear_pp_info(netmem_ref netmem) +{ +} +static inline int page_pool_check_memory_provider(struct net_device *dev, + struct netdev_rx_queue *rxq) +{ + return 0; +} +#endif + #endif diff --git a/net/core/page_pool_user.c b/net/core/page_pool_user.c index 3a3277ba167b..cd6267ba6fa3 100644 --- a/net/core/page_pool_user.c +++ b/net/core/page_pool_user.c @@ -4,8 +4,9 @@ #include #include #include -#include +#include #include +#include #include =20 #include "page_pool_priv.h" @@ -344,6 +345,30 @@ void page_pool_unlist(struct page_pool *pool) mutex_unlock(&page_pools_lock); } =20 +int page_pool_check_memory_provider(struct net_device *dev, + struct netdev_rx_queue *rxq) +{ + struct net_devmem_dmabuf_binding *binding =3D rxq->mp_params.mp_priv; + struct page_pool *pool; + struct hlist_node *n; + + if (!binding) + return 0; + + mutex_lock(&page_pools_lock); + hlist_for_each_entry_safe(pool, n, &dev->page_pools, user.list) { + if (pool->mp_priv !=3D binding) + continue; + + if (pool->slow.queue_idx =3D=3D get_netdev_rx_queue_index(rxq)) { + mutex_unlock(&page_pools_lock); + return 0; + } + } + mutex_unlock(&page_pools_lock); + return -ENODATA; +} + static void page_pool_unreg_netdev_wipe(struct net_device *netdev) { struct page_pool *pool; --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CADF71AE861 for ; Mon, 9 Sep 2024 05:43:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860622; cv=none; b=ccCPkjQNRmEVfKOAE3yS9mAHMI9YyOprQ5Vp7VF0fN/PRF66cGZpPEpJ44u0dmTLk3c+LPjlyOGXGm4+Z4jtg8IUUYAv1wPOMkpp1Ms4Q8dwPybojbAOEXn2wpqmAJ+dFU2Qcy+iBEB95nkXFrvBRWgxlfU7z/FPMk5/p0U9k20= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860622; c=relaxed/simple; bh=XFfNukANvJSPmHUcUIwhzPvEuUN7J7SPq61BqAQhxLg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dupp0p+OAUXtabqW0wmI2NGVSUF9B4ZevhjjR1CfrtPGVZfbFNd4LYr02S/eTAMfAw95gH6JBb13LpCDyDn+WX8jhhCFnVh3XO8TZlkzUyu2vAcJ48goLAi0+4oC8s08fnrCDf85FYKl2Nas8QduAUXRf8oArFX9O7VSzTpMeC4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gvBxcCMG; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gvBxcCMG" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e1cfb9d655eso8020690276.0 for ; Sun, 08 Sep 2024 22:43:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860615; x=1726465415; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aEmxx1cn5az+ypfNNMkpKP05ANn3vL38LjNETozCE+I=; b=gvBxcCMGg1vrUQRlG5F97vCT/nN/2zWLr8d8nVFmLpFQTysDtV5G0BV/PbvaKu3zrH hHF2a4sQMTrwc1siCwMg7u3sRWXFqvTrMhWiqbQ3BS0KZlGICnrvm6/KMOUKhOxP1L/H Q6Sn9vhaNcuz3aZ6NAbJokBUMqvEQ41ipUwzrPvam8SltJiMVs+B+JWsi9ZGH5Re6eHE hxcy3S81OeZglsndTDJcgTezL+Zaau92gojIpGCJnjRDrojQiIxv6/t/m66NYjODm59k +R9j6hUJoKrqoeCifUDuqtJViR58gX0/PuggtSLfqjSDi/9jtWpau72gtTuFduvQ9lfr XJig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860615; x=1726465415; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aEmxx1cn5az+ypfNNMkpKP05ANn3vL38LjNETozCE+I=; b=eiosi2XDNC0CWSKpDgXNCVp096fUa75+kktk3gpDV51BRb244gP+N42H/aevO5PZTk begUkDx++qEhaGR374kuQQ48+GBNcyGU3B7/zcWNFYxvZdbm3sux72N1DQ1rql6/ajyq R2Ue434W8jmwUp2jwVnc+EfLX4jITzoidZh/2YGXspZ3klKHFCu4Hd5tGBUxr0ZgwfMY /QSDmaaG2D8xjpoHI3/VKVMMkxGIldeMyXoluazlgrgZW2Ohyh2RqtzMu5AKSwvrqZLz dLIViA12fvG8aDmv1bpa7KYIDv22HMxvz+F3ooT0DD8VhsM26k5raYWBSbx9MsylpWNH zEog== X-Forwarded-Encrypted: i=1; AJvYcCU2NY9lBrz6IkR/SowU5AnfCtByA60aYqkuBjkd21UjV4+4rlCVaWNJVtFzCirJ5WYUBx7w9HZRO8c06to=@vger.kernel.org X-Gm-Message-State: AOJu0YyOjgz82BbnYgtCEvF6EFufDKVZIwV3hsvYe8a3/HhH5Dlvvjut ltFA0xu/WlkVya2SiSGAY1x+h+JyMcGRcmoq1SUXvpC+IzYBl7FueMSNrP26E0oV6yMQG4oICm1 MvP4l+efHynAK8Z5eDQkuSQ== X-Google-Smtp-Source: AGHT+IE1ZU85gvwVEa6z3iJxnEeDzUc2Sx0D+3OnEAAsyX5B/ICZiJxnerXjkEjl5jS30uvXa/lvWTASw8uTpGNQgg== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a25:df4c:0:b0:e0b:f93:fe8c with SMTP id 3f1490d57ef6-e1d346b2ca3mr29412276.0.1725860615155; Sun, 08 Sep 2024 22:43:35 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:12 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-8-almasrymina@google.com> Subject: [PATCH net-next v25 07/13] net: support non paged skb frags From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make skb_frag_page() fail in the case where the frag is not backed by a page, and fix its relevant callers to handle this case. Signed-off-by: Mina Almasry Reviewed-by: Eric Dumazet Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v10: - Fixed newly generated kdoc warnings found by patchwork. While we're at it, fix the Return section of the functions I touched. v6: - Rebased on top of the merged netmem changes. Changes in v1: - Fix illegal_highdma() (Yunsheng). - Rework napi_pp_put_page() slightly to reduce code churn (Willem). --- include/linux/skbuff.h | 42 +++++++++++++++++++++++++++++++++++++- include/linux/skbuff_ref.h | 9 ++++---- net/core/dev.c | 3 ++- net/core/gro.c | 3 ++- net/core/skbuff.c | 11 ++++++++++ net/ipv4/esp4.c | 3 ++- net/ipv4/tcp.c | 3 +++ net/ipv6/esp6.c | 3 ++- 8 files changed, 67 insertions(+), 10 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index cf8f6ce06742..dbadf2dd6b35 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3523,21 +3523,58 @@ static inline void skb_frag_off_copy(skb_frag_t *fr= agto, fragto->offset =3D fragfrom->offset; } =20 +/* Return: true if the skb_frag contains a net_iov. */ +static inline bool skb_frag_is_net_iov(const skb_frag_t *frag) +{ + return netmem_is_net_iov(frag->netmem); +} + +/** + * skb_frag_net_iov - retrieve the net_iov referred to by fragment + * @frag: the fragment + * + * Return: the &struct net_iov associated with @frag. Returns NULL if this + * frag has no associated net_iov. + */ +static inline struct net_iov *skb_frag_net_iov(const skb_frag_t *frag) +{ + if (!skb_frag_is_net_iov(frag)) + return NULL; + + return netmem_to_net_iov(frag->netmem); +} + /** * skb_frag_page - retrieve the page referred to by a paged fragment * @frag: the paged fragment * - * Returns the &struct page associated with @frag. + * Return: the &struct page associated with @frag. Returns NULL if this fr= ag + * has no associated page. */ static inline struct page *skb_frag_page(const skb_frag_t *frag) { + if (skb_frag_is_net_iov(frag)) + return NULL; + return netmem_to_page(frag->netmem); } =20 +/** + * skb_frag_netmem - retrieve the netmem referred to by a fragment + * @frag: the fragment + * + * Return: the &netmem_ref associated with @frag. + */ +static inline netmem_ref skb_frag_netmem(const skb_frag_t *frag) +{ + return frag->netmem; +} + int skb_pp_cow_data(struct page_pool *pool, struct sk_buff **pskb, unsigned int headroom); int skb_cow_data_for_xdp(struct page_pool *pool, struct sk_buff **pskb, struct bpf_prog *prog); + /** * skb_frag_address - gets the address of the data contained in a paged fr= agment * @frag: the paged fragment buffer @@ -3547,6 +3584,9 @@ int skb_cow_data_for_xdp(struct page_pool *pool, stru= ct sk_buff **pskb, */ static inline void *skb_frag_address(const skb_frag_t *frag) { + if (!skb_frag_page(frag)) + return NULL; + return page_address(skb_frag_page(frag)) + skb_frag_off(frag); } =20 diff --git a/include/linux/skbuff_ref.h b/include/linux/skbuff_ref.h index 16c241a23472..0f3c58007488 100644 --- a/include/linux/skbuff_ref.h +++ b/include/linux/skbuff_ref.h @@ -34,14 +34,13 @@ static inline void skb_frag_ref(struct sk_buff *skb, in= t f) =20 bool napi_pp_put_page(netmem_ref netmem); =20 -static inline void -skb_page_unref(struct page *page, bool recycle) +static inline void skb_page_unref(netmem_ref netmem, bool recycle) { #ifdef CONFIG_PAGE_POOL - if (recycle && napi_pp_put_page(page_to_netmem(page))) + if (recycle && napi_pp_put_page(netmem)) return; #endif - put_page(page); + put_page(netmem_to_page(netmem)); } =20 /** @@ -54,7 +53,7 @@ skb_page_unref(struct page *page, bool recycle) */ static inline void __skb_frag_unref(skb_frag_t *frag, bool recycle) { - skb_page_unref(skb_frag_page(frag), recycle); + skb_page_unref(skb_frag_netmem(frag), recycle); } =20 /** diff --git a/net/core/dev.c b/net/core/dev.c index 34d73d152f19..b4ef6578f31f 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3434,8 +3434,9 @@ static int illegal_highdma(struct net_device *dev, st= ruct sk_buff *skb) if (!(dev->features & NETIF_F_HIGHDMA)) { for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { skb_frag_t *frag =3D &skb_shinfo(skb)->frags[i]; + struct page *page =3D skb_frag_page(frag); =20 - if (PageHighMem(skb_frag_page(frag))) + if (page && PageHighMem(page)) return 1; } } diff --git a/net/core/gro.c b/net/core/gro.c index 3abad1b567dd..802b4a062400 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -408,7 +408,8 @@ static inline void skb_gro_reset_offset(struct sk_buff = *skb, u32 nhoff) pinfo =3D skb_shinfo(skb); frag0 =3D &pinfo->frags[0]; =20 - if (pinfo->nr_frags && !PageHighMem(skb_frag_page(frag0)) && + if (pinfo->nr_frags && skb_frag_page(frag0) && + !PageHighMem(skb_frag_page(frag0)) && (!NET_IP_ALIGN || !((skb_frag_off(frag0) + nhoff) & 3))) { NAPI_GRO_CB(skb)->frag0 =3D skb_frag_address(frag0); NAPI_GRO_CB(skb)->frag0_len =3D min_t(unsigned int, diff --git a/net/core/skbuff.c b/net/core/skbuff.c index d9634ab342cc..7ea1508a1176 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1371,6 +1371,14 @@ void skb_dump(const char *level, const struct sk_buf= f *skb, bool full_pkt) struct page *p; u8 *vaddr; =20 + if (skb_frag_is_net_iov(frag)) { + printk("%sskb frag %d: not readable\n", level, i); + len -=3D skb_frag_size(frag); + if (!len) + break; + continue; + } + skb_frag_foreach_page(frag, skb_frag_off(frag), skb_frag_size(frag), p, p_off, p_len, copied) { @@ -3163,6 +3171,9 @@ static bool __skb_splice_bits(struct sk_buff *skb, st= ruct pipe_inode_info *pipe, for (seg =3D 0; seg < skb_shinfo(skb)->nr_frags; seg++) { const skb_frag_t *f =3D &skb_shinfo(skb)->frags[seg]; =20 + if (WARN_ON_ONCE(!skb_frag_page(f))) + return false; + if (__splice_segment(skb_frag_page(f), skb_frag_off(f), skb_frag_size(f), offset, len, spd, false, sk, pipe)) diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c index 47378ca41904..f3281312eb5e 100644 --- a/net/ipv4/esp4.c +++ b/net/ipv4/esp4.c @@ -115,7 +115,8 @@ static void esp_ssg_unref(struct xfrm_state *x, void *t= mp, struct sk_buff *skb) */ if (req->src !=3D req->dst) for (sg =3D sg_next(req->src); sg; sg =3D sg_next(sg)) - skb_page_unref(sg_page(sg), skb->pp_recycle); + skb_page_unref(page_to_netmem(sg_page(sg)), + skb->pp_recycle); } =20 #ifdef CONFIG_INET_ESPINTCP diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 8a5680b4e786..05844a36ffeb 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2177,6 +2177,9 @@ static int tcp_zerocopy_receive(struct sock *sk, break; } page =3D skb_frag_page(frags); + if (WARN_ON_ONCE(!page)) + break; + prefetchw(page); pages[pages_to_map++] =3D page; length +=3D PAGE_SIZE; diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c index 3920e8aa1031..b2400c226a32 100644 --- a/net/ipv6/esp6.c +++ b/net/ipv6/esp6.c @@ -132,7 +132,8 @@ static void esp_ssg_unref(struct xfrm_state *x, void *t= mp, struct sk_buff *skb) */ if (req->src !=3D req->dst) for (sg =3D sg_next(req->src); sg; sg =3D sg_next(sg)) - skb_page_unref(sg_page(sg), skb->pp_recycle); + skb_page_unref(page_to_netmem(sg_page(sg)), + skb->pp_recycle); } =20 #ifdef CONFIG_INET6_ESPINTCP --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A893A1AE038 for ; Mon, 9 Sep 2024 05:43:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860625; cv=none; b=tzvhhPeBpBRe06VZJ/Lo4kjd9vL4rNUXTT62KXiyMwxfh4+84fx5uZgYf3SXOseohT2hJAfmZ6+KRu3bzDWy3aEPMfn1UE2hQM1wV3iO0Ms0tWQ61xAiWK4tGRFTnTNc0iYIktFlyqvHtDoLEY5sbjaeJIcgKcpWR2iVfp7cuDQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860625; c=relaxed/simple; bh=8yTKyPXR/JJ8W41/h1ILEYUwp0ILfE7ny7htLf2eC3k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mx5UxvBFcl+3M4HEx2CzaRzGTi2+JDj6/aAFsqrX1peTB6p15V/vFtR/6gnwJbUcavm4WQuHs/dy3dYhUSlWCkwrbGBDxJhlhMjRXGZzxTVP+f1bOQNUra/jR5lUfMVJ5hMcPN+E76wWFRAB9XZx+mA2fLdJeaVpj6Qa9YOTPM0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xShQO+Mc; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xShQO+Mc" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e1a7ef9eb78so7829715276.3 for ; Sun, 08 Sep 2024 22:43:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860617; x=1726465417; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hJWTzgzMpuO58f7zhEkczq8k8Bg4k7akg7ko/9LTHJU=; b=xShQO+MciIJ2af+8V8eulpMa3XmdjeVWz397F/qUSG7Jb/tTXuh+npCMjooVskuBew aQ2XktqYayEUOa7akG9bB+fQLqqKXZtyOZDVc5Fi/t4ygLNPjzFqMT08zyVZoXLNjLnQ p0v74cFz5RmABL48u/S5DsAX3Q2BJYlslCIs5RAXl5HBoCGSVZMNZXvmZXgo/kgwgu0C hBP7XuqjDklQEfcdB2noTOZ5ieyUlqGnfkuL17OSaCUhRViZegx2SmHbNFHsDlsM36c+ KH/KAsUo3UC8fJ91v4MB/D2s3lBhvJxsQGAFzL/2faH0rHuuMwxwb7TPdRKyJoJtev7r PXbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860617; x=1726465417; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hJWTzgzMpuO58f7zhEkczq8k8Bg4k7akg7ko/9LTHJU=; b=vXgUIRbdWz7z79xGJgKQGlIWBpzbyosalzxRiuvmf98Upg3z5jCH7DkzNWSx4hNxnp dqSVzboiqf8IQhnlAdqMd7rmUngOa2hYOLV28CQpxK7C+s5SnNz3ynsYqq684XdlPhs6 K84LACIk+ehWlGAr0W0Oz/uiC1wof8O09nPhsMu/evvRr+d3QcOwXm8lQSFsO20q+l8j ciS8J3NIwOJ8TQa4kqOe/56PxjYfT9XB4yEasqto52PZwFzwykKZQsHTvEDgBPrOaq9o Ndg/l6Kk2QrOxwgIF2epd677NtWqu6zHYCw2Sr6U8+kyCYr2l9d7AducyCyNfVBB9/T2 KZ7Q== X-Forwarded-Encrypted: i=1; AJvYcCXM9tF0TqSdfGrWyQ1Clo4ODnRychkB9ThAA5Of/W1gz0TE98+oWdEZaKf25wH7yhN/vIJIPzdGOhc74OE=@vger.kernel.org X-Gm-Message-State: AOJu0YzWROLyEljBrSL5+yxTocxIMS8AJhmRAQm83oWibuQwe9Ude9hS QcKPxpiKaAJ73DJ5JnLtcEwjmMu2mFAwMtp1/KtkIiMlLOqdgsmN5ts+VMbqb6+5sQBouvDfIqI u2n9AkrJ735RaYxjAcSKrUQ== X-Google-Smtp-Source: AGHT+IHaG4QItTDUAJsf/nvBwk/4JNxRN6QfYUiZZnCQudbcE3ZJunvmcRT3sNrhU3Nm1q9ZUqIwOogeDp7HTE8Qow== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a25:3628:0:b0:e03:2f90:e81d with SMTP id 3f1490d57ef6-e1d34a40ed1mr22965276.11.1725860617082; Sun, 08 Sep 2024 22:43:37 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:13 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-9-almasrymina@google.com> Subject: [PATCH net-next v25 08/13] net: add support for skbs with unreadable frags From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Willem de Bruijn , Kaiyuan Zhang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For device memory TCP, we expect the skb headers to be available in host memory for access, and we expect the skb frags to be in device memory and unaccessible to the host. We expect there to be no mixing and matching of device memory frags (unaccessible) with host memory frags (accessible) in the same skb. Add a skb->devmem flag which indicates whether the frags in this skb are device memory frags or not. __skb_fill_netmem_desc() now checks frags added to skbs for net_iov, and marks the skb as skb->devmem accordingly. Add checks through the network stack to avoid accessing the frags of devmem skbs and avoid coalescing devmem skbs with non devmem skbs. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry Reviewed-by: Eric Dumazet Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v25: - Remove readable check in tcp_skb_can_collapse_to(). - Add check in skb_checksum_help (Jakub). - Add WARN_ON_ONCE around __skb_checksum readable check (Jakub). v16: - Fix unreadable handling in skb_split_no_header() (Eric). v11: - drop excessive checks for frag 0 pull (Paolo) v9: https://lore.kernel.org/netdev/20240403002053.2376017-11-almasrymina@go= ogle.com/ - change skb->readable to skb->unreadable (Pavel/David). skb->readable was very complicated, because by default skbs are readable so the flag needed to be set to true in all code paths where new skbs were created or cloned. Forgetting to set skb->readable=3Dtrue in some paths caused crashes. Flip it to skb->unreadable so that the default 0 value works well, and we only need to set it to true when we add unreadable frags. v6 - skb->dmabuf -> skb->readable (Pavel). Pavel's original suggestion was to remove the skb->dmabuf flag entirely, but when I looked into it closely, I found the issue that if we remove the flag we have to dereference the shinfo(skb) pointer to obtain the first frag, which can cause a performance regression if it dirties the cache line when the shinfo(skb) was not really needed. Instead, I converted the skb->dmabuf flag into a generic skb->readable flag which can be re-used by io_uring. Changes in v1: - Rename devmem -> dmabuf (David). - Flip skb_frags_not_readable (Jakub). --- include/linux/skbuff.h | 19 +++++++++++++++++-- include/net/tcp.h | 3 ++- net/core/datagram.c | 6 ++++++ net/core/dev.c | 4 ++++ net/core/skbuff.c | 43 ++++++++++++++++++++++++++++++++++++++++-- net/ipv4/tcp.c | 3 +++ net/ipv4/tcp_input.c | 13 ++++++++++--- net/ipv4/tcp_output.c | 5 ++++- net/packet/af_packet.c | 4 ++-- 9 files changed, 89 insertions(+), 11 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index dbadf2dd6b35..d02a88bad953 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -827,6 +827,8 @@ enum skb_tstamp_type { * @csum_level: indicates the number of consecutive checksums found in * the packet minus one that have been verified as * CHECKSUM_UNNECESSARY (max 3) + * @unreadable: indicates that at least 1 of the fragments in this skb is + * unreadable. * @dst_pending_confirm: need to confirm neighbour * @decrypted: Decrypted SKB * @slow_gro: state present at GRO time, slower prepare step required @@ -1008,7 +1010,7 @@ struct sk_buff { #if IS_ENABLED(CONFIG_IP_SCTP) __u8 csum_not_inet:1; #endif - + __u8 unreadable:1; #if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS) __u16 tc_index; /* traffic control index */ #endif @@ -1823,6 +1825,12 @@ static inline void skb_zcopy_downgrade_managed(struc= t sk_buff *skb) __skb_zcopy_downgrade_managed(skb); } =20 +/* Return true if frags in this skb are readable by the host. */ +static inline bool skb_frags_readable(const struct sk_buff *skb) +{ + return !skb->unreadable; +} + static inline void skb_mark_not_on_list(struct sk_buff *skb) { skb->next =3D NULL; @@ -2539,10 +2547,17 @@ static inline void skb_len_add(struct sk_buff *skb,= int delta) static inline void __skb_fill_netmem_desc(struct sk_buff *skb, int i, netmem_ref netmem, int off, int size) { - struct page *page =3D netmem_to_page(netmem); + struct page *page; =20 __skb_fill_netmem_desc_noacc(skb_shinfo(skb), i, netmem, off, size); =20 + if (netmem_is_net_iov(netmem)) { + skb->unreadable =3D true; + return; + } + + page =3D netmem_to_page(netmem); + /* Propagate page pfmemalloc to the skb if we can. The problem is * that not all callers have unique ownership of the page but rely * on page_is_pfmemalloc doing the right thing(tm). diff --git a/include/net/tcp.h b/include/net/tcp.h index 2aac11e7e1cc..f77f812bfbe7 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1069,7 +1069,8 @@ static inline bool tcp_skb_can_collapse(const struct = sk_buff *to, /* skb_cmp_decrypted() not needed, use tcp_write_collapse_fence() */ return likely(tcp_skb_can_collapse_to(to) && mptcp_skb_can_collapse(to, from) && - skb_pure_zcopy_same(to, from)); + skb_pure_zcopy_same(to, from) && + skb_frags_readable(to) =3D=3D skb_frags_readable(from)); } =20 static inline bool tcp_skb_can_collapse_rx(const struct sk_buff *to, diff --git a/net/core/datagram.c b/net/core/datagram.c index a40f733b37d7..f0693707aece 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -407,6 +407,9 @@ static int __skb_datagram_iter(const struct sk_buff *sk= b, int offset, return 0; } =20 + if (!skb_frags_readable(skb)) + goto short_copy; + /* Copy paged appendix. Hmm... why does this look so complicated? */ for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; @@ -623,6 +626,9 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb, { int frag =3D skb_shinfo(skb)->nr_frags; =20 + if (!skb_frags_readable(skb)) + return -EFAULT; + while (length && iov_iter_count(from)) { struct page *head, *last_head =3D NULL; struct page *pages[MAX_SKB_FRAGS]; diff --git a/net/core/dev.c b/net/core/dev.c index b4ef6578f31f..c7d5319596c8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3312,6 +3312,10 @@ int skb_checksum_help(struct sk_buff *skb) return -EINVAL; } =20 + if (!skb_frags_readable(skb)) { + return -EFAULT; + } + /* Before computing a checksum, we should make sure no frag could * be modified by an external entity : checksum could be wrong. */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7ea1508a1176..51a6e9570808 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1972,6 +1972,9 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mas= k) if (skb_shared(skb) || skb_unclone(skb, gfp_mask)) return -EINVAL; =20 + if (!skb_frags_readable(skb)) + return -EFAULT; + if (!num_frags) goto release; =20 @@ -2145,6 +2148,9 @@ struct sk_buff *skb_copy(const struct sk_buff *skb, g= fp_t gfp_mask) unsigned int size; int headerlen; =20 + if (!skb_frags_readable(skb)) + return NULL; + if (WARN_ON_ONCE(skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST)) return NULL; =20 @@ -2483,6 +2489,9 @@ struct sk_buff *skb_copy_expand(const struct sk_buff = *skb, struct sk_buff *n; int oldheadroom; =20 + if (!skb_frags_readable(skb)) + return NULL; + if (WARN_ON_ONCE(skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST)) return NULL; =20 @@ -2827,6 +2836,9 @@ void *__pskb_pull_tail(struct sk_buff *skb, int delta) */ int i, k, eat =3D (skb->tail + delta) - skb->end; =20 + if (!skb_frags_readable(skb)) + return NULL; + if (eat > 0 || skb_cloned(skb)) { if (pskb_expand_head(skb, 0, eat > 0 ? eat + 128 : 0, GFP_ATOMIC)) @@ -2980,6 +2992,9 @@ int skb_copy_bits(const struct sk_buff *skb, int offs= et, void *to, int len) to +=3D copy; } =20 + if (!skb_frags_readable(skb)) + goto fault; + for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; skb_frag_t *f =3D &skb_shinfo(skb)->frags[i]; @@ -3168,6 +3183,9 @@ static bool __skb_splice_bits(struct sk_buff *skb, st= ruct pipe_inode_info *pipe, /* * then map the fragments */ + if (!skb_frags_readable(skb)) + return false; + for (seg =3D 0; seg < skb_shinfo(skb)->nr_frags; seg++) { const skb_frag_t *f =3D &skb_shinfo(skb)->frags[seg]; =20 @@ -3391,6 +3409,9 @@ int skb_store_bits(struct sk_buff *skb, int offset, c= onst void *from, int len) from +=3D copy; } =20 + if (!skb_frags_readable(skb)) + goto fault; + for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { skb_frag_t *frag =3D &skb_shinfo(skb)->frags[i]; int end; @@ -3470,6 +3491,9 @@ __wsum __skb_checksum(const struct sk_buff *skb, int = offset, int len, pos =3D copy; } =20 + if (WARN_ON_ONCE(!skb_frags_readable(skb))) + return 0; + for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; skb_frag_t *frag =3D &skb_shinfo(skb)->frags[i]; @@ -3570,6 +3594,9 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *s= kb, int offset, pos =3D copy; } =20 + if (!skb_frags_readable(skb)) + return 0; + for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { int end; =20 @@ -4061,6 +4088,7 @@ static inline void skb_split_inside_header(struct sk_= buff *skb, skb_shinfo(skb1)->frags[i] =3D skb_shinfo(skb)->frags[i]; =20 skb_shinfo(skb1)->nr_frags =3D skb_shinfo(skb)->nr_frags; + skb1->unreadable =3D skb->unreadable; skb_shinfo(skb)->nr_frags =3D 0; skb1->data_len =3D skb->data_len; skb1->len +=3D skb1->data_len; @@ -4108,6 +4136,8 @@ static inline void skb_split_no_header(struct sk_buff= *skb, pos +=3D size; } skb_shinfo(skb1)->nr_frags =3D k; + + skb1->unreadable =3D skb->unreadable; } =20 /** @@ -4345,6 +4375,9 @@ unsigned int skb_seq_read(unsigned int consumed, cons= t u8 **data, return block_limit - abs_offset; } =20 + if (!skb_frags_readable(st->cur_skb)) + return 0; + if (st->frag_idx =3D=3D 0 && !st->frag_data) st->stepped_offset +=3D skb_headlen(st->cur_skb); =20 @@ -5957,7 +5990,10 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_= buff *from, if (to->pp_recycle !=3D from->pp_recycle) return false; =20 - if (len <=3D skb_tailroom(to)) { + if (skb_frags_readable(from) !=3D skb_frags_readable(to)) + return false; + + if (len <=3D skb_tailroom(to) && skb_frags_readable(from)) { if (len) BUG_ON(skb_copy_bits(from, 0, skb_put(to, len), len)); *delta_truesize =3D 0; @@ -6134,6 +6170,9 @@ int skb_ensure_writable(struct sk_buff *skb, unsigned= int write_len) if (!pskb_may_pull(skb, write_len)) return -ENOMEM; =20 + if (!skb_frags_readable(skb)) + return -EFAULT; + if (!skb_cloned(skb) || skb_clone_writable(skb, write_len)) return 0; =20 @@ -6813,7 +6852,7 @@ void skb_condense(struct sk_buff *skb) { if (skb->data_len) { if (skb->data_len > skb->end - skb->tail || - skb_cloned(skb)) + skb_cloned(skb) || !skb_frags_readable(skb)) return; =20 /* Nice, we can free page frag(s) right now */ diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 05844a36ffeb..30238963fe99 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2160,6 +2160,9 @@ static int tcp_zerocopy_receive(struct sock *sk, skb =3D tcp_recv_skb(sk, seq, &offset); } =20 + if (!skb_frags_readable(skb)) + break; + if (TCP_SKB_CB(skb)->has_rxtstamp) { tcp_update_recv_tstamps(skb, tss); zc->msg_flags |=3D TCP_CMSG_TS; diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index e37488d3453f..9f314dfa1490 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5391,6 +5391,9 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *li= st, struct rb_root *root, for (end_of_skbs =3D true; skb !=3D NULL && skb !=3D tail; skb =3D n) { n =3D tcp_skb_next(skb, list); =20 + if (!skb_frags_readable(skb)) + goto skip_this; + /* No new bits? It is possible on ofo queue. */ if (!before(start, TCP_SKB_CB(skb)->end_seq)) { skb =3D tcp_collapse_one(sk, skb, list, root); @@ -5411,17 +5414,20 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *= list, struct rb_root *root, break; } =20 - if (n && n !=3D tail && tcp_skb_can_collapse_rx(skb, n) && + if (n && n !=3D tail && skb_frags_readable(n) && + tcp_skb_can_collapse_rx(skb, n) && TCP_SKB_CB(skb)->end_seq !=3D TCP_SKB_CB(n)->seq) { end_of_skbs =3D false; break; } =20 +skip_this: /* Decided to skip this, advance start seq. */ start =3D TCP_SKB_CB(skb)->end_seq; } if (end_of_skbs || - (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN))) + (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)) || + !skb_frags_readable(skb)) return; =20 __skb_queue_head_init(&tmp); @@ -5463,7 +5469,8 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *li= st, struct rb_root *root, if (!skb || skb =3D=3D tail || !tcp_skb_can_collapse_rx(nskb, skb) || - (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN))) + (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)) || + !skb_frags_readable(skb)) goto end; } } diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index cdd0def14427..4fd746bd4d54 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -2344,7 +2344,8 @@ static bool tcp_can_coalesce_send_queue_head(struct s= ock *sk, int len) =20 if (unlikely(TCP_SKB_CB(skb)->eor) || tcp_has_tx_tstamp(skb) || - !skb_pure_zcopy_same(skb, next)) + !skb_pure_zcopy_same(skb, next) || + skb_frags_readable(skb) !=3D skb_frags_readable(next)) return false; =20 len -=3D skb->len; @@ -3264,6 +3265,8 @@ static bool tcp_can_collapse(const struct sock *sk, c= onst struct sk_buff *skb) return false; if (skb_cloned(skb)) return false; + if (!skb_frags_readable(skb)) + return false; /* Some heuristics for collapsing over SACK'd could be invented */ if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) return false; diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index 4a364cdd445e..a705ec214254 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -2216,7 +2216,7 @@ static int packet_rcv(struct sk_buff *skb, struct net= _device *dev, } } =20 - snaplen =3D skb->len; + snaplen =3D skb_frags_readable(skb) ? skb->len : skb_headlen(skb); =20 res =3D run_filter(skb, sk, snaplen); if (!res) @@ -2336,7 +2336,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct ne= t_device *dev, } } =20 - snaplen =3D skb->len; + snaplen =3D skb_frags_readable(skb) ? skb->len : skb_headlen(skb); =20 res =3D run_filter(skb, sk, snaplen); if (!res) --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 056391741F0 for ; Mon, 9 Sep 2024 05:43:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860629; cv=none; b=fdsJXRI+w9t81oCOMBnNQgcydV8SYtIKFOtm9Sk3yEFdfCr8GNbo0FpyYQ9Aj/Mf5ah8fEPPxfGPSSkOpzEJ0L8t/88W5X4skYqJviw1j/eliUR7YTQO871APuXWvg7jezMCYRDEdXtuFNVshEOBW9o5VpIuq6UkptwAKnyVBEM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860629; c=relaxed/simple; bh=UFllwkNQQtkcH1abIF5EY1ZDWMqdmxvaE8kWhFzMI2E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=r+rLP7MnYp0M10o+yjHtgjC2FL1YfP7L0ZOa3UhbWNiaGLLlo6dI6P4Wze26vk3B0g97xXEYGcWXs2hZduenQCPlboSXYKTKSCKX3tV7EcfVFNuqrFDcvB9/Dqs0eZHC8Hmuiee/BTnsUvkhjQCd2B3H61L43fHjK7XuUa3+vT4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=B9gIwaTi; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="B9gIwaTi" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1a8de19f5fso8278126276.3 for ; Sun, 08 Sep 2024 22:43:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860619; x=1726465419; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PDLjaR97cQcRKtjEUdyE+L+eCzw+FaOWvgNnimLOpjw=; b=B9gIwaTi8Z0BWYcBnj4dMvaIBBqRggOp/VCehQzTqdl+8ps12pUdKcjfGkURCd3es5 0eaqbhlFvtYdoyH7Iyr2xT4IenF2OJaTlPHgrNXNJwt3c33jJw/hpSpW2avUM6Q3a1fr LHrpMX0P6rjkwnLjlAzF+tElwz9jy+9MqlWJ7q1Xv1VH3dZ3UjSNk8KSqAxkpe2ABYgM o4QDrcXZjhkcVZTK3iEzl3jrwSx9Ur5bue194Fts3Y1Jpsg/RuoY1vt6HH0452NaBVND k24q5H85x+3fd5Fq7Knn0KerumGZxCsLzAKkp9LgKtzcgvBJLhpdqRQ4tshkQPfav/05 dVPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860619; x=1726465419; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PDLjaR97cQcRKtjEUdyE+L+eCzw+FaOWvgNnimLOpjw=; b=CMSw5HUMxlqFlRGZ0S0HBIk3cVkGsOKLfWn6oW77nzljNBXz2VHrOZA2Ci6Xr0Mieq 3mTFOIMLLKP/W/t46olp0wige0xhxpGihsVVAN4y6eaQ1PkIcRUDorjNva7QhDKlMU3v VzgZOQvkwyO4n+f/eSA5RohOoePIRmNUMJ2bfmoeLUM1QlB4W4rNG3PnqjaebSsUNdo5 MCMh3AK0oeRLQhbt9gbPKR+3jIyDJZu2B/su67SeYGaXIa4cdAIg9CHyIZSLg/jTDoko ljeG2RmqQnwp1rP+BMuJWBou0J2Czp9avhEnaZAJREkj8ZnjleapY5g1HcofG404fZ/W 9AoQ== X-Forwarded-Encrypted: i=1; AJvYcCUUHeEvfc62O6a+rIo0CH2fQewnfoQV1BtOIPulG9hd6iXt5+J9gleF2M+sVhF2oAx+s5jsabA11xmU6O0=@vger.kernel.org X-Gm-Message-State: AOJu0YwwnkA4byCkNrpQCubMtdNPNvrIXjkw8az6V1hZvgB0if8Trnf0 NoL/SIo7k6JbkNNFxxIakOBS7QFW4dDl5x8SN8PtMpX4q8GRyvJSdu82ckO4+u8rDeMIxdaxT6E HkGb/WLuGu8bYg7tJ/SHpMQ== X-Google-Smtp-Source: AGHT+IHytPeeq/wCgLKKF34D5d7zbDHkL56ROpr4IHUb78uvJDEZ4rcMJsu5BHGU+/tklg6vMy7wuC2XaM/Obv/VvA== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a05:6902:4e7:b0:e16:6e0a:bafe with SMTP id 3f1490d57ef6-e1d346b0014mr192096276.0.1725860619340; Sun, 08 Sep 2024 22:43:39 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:14 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-10-almasrymina@google.com> Subject: [PATCH net-next v25 09/13] tcp: RX path for devmem TCP From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Willem de Bruijn , Kaiyuan Zhang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In tcp_recvmsg_locked(), detect if the skb being received by the user is a devmem skb. In this case - if the user provided the MSG_SOCK_DEVMEM flag - pass it to tcp_recvmsg_devmem() for custom handling. tcp_recvmsg_devmem() copies any data in the skb header to the linear buffer, and returns a cmsg to the user indicating the number of bytes returned in the linear buffer. tcp_recvmsg_devmem() then loops over the unaccessible devmem skb frags, and returns to the user a cmsg_devmem indicating the location of the data in the dmabuf device memory. cmsg_devmem contains this information: 1. the offset into the dmabuf where the payload starts. 'frag_offset'. 2. the size of the frag. 'frag_size'. 3. an opaque token 'frag_token' to return to the kernel when the buffer is to be released. The pages awaiting freeing are stored in the newly added sk->sk_user_frags, and each page passed to userspace is get_page()'d. This reference is dropped once the userspace indicates that it is done reading this page. All pages are released when the socket is destroyed. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov Reviewed-by: Eric Dumazet reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v20: - Change `offset =3D 0` to `offset =3D offset - start` to resolve issue reported by Taehee. v16: - Fix number assignement (Arnd). v13: - Refactored user frags cleanup into a common function to avoid __maybe_unused. (Pavel) - change to offset =3D 0 for some improved clarity. v11: - Refactor to common function te remove conditional lock sparse warning (Paolo) v7: - Updated the SO_DEVMEM_* uapi to use the next available entries (Arnd). - Updated dmabuf_cmsg struct to be __u64 padded (Arnd). - Squashed fix from Eric to initialize sk_user_frags for passive sockets (Eric). v6 - skb->dmabuf -> skb->readable (Pavel) - Fixed asm definitions of SO_DEVMEM_LINEAR/SO_DEVMEM_DMABUF not found on some archs. - Squashed in locking optimizations from edumazet@google.com. With this change we lock the xarray once per per tcp_recvmsg_dmabuf() rather than once per frag in xa_alloc(). Changes in v1: - Added dmabuf_id to dmabuf_cmsg (David/Stan). - Devmem -> dmabuf (David). - Change tcp_recvmsg_dmabuf() check to skb->dmabuf (Paolo). - Use __skb_frag_ref() & napi_pp_put_page() for refcounting (Yunsheng). RFC v3: - Fixed issue with put_cmsg() failing silently. --- arch/alpha/include/uapi/asm/socket.h | 5 + arch/mips/include/uapi/asm/socket.h | 5 + arch/parisc/include/uapi/asm/socket.h | 5 + arch/sparc/include/uapi/asm/socket.h | 5 + include/linux/socket.h | 1 + include/net/sock.h | 2 + include/uapi/asm-generic/socket.h | 5 + include/uapi/linux/uio.h | 13 ++ net/core/devmem.h | 22 +++ net/ipv4/tcp.c | 257 +++++++++++++++++++++++++- net/ipv4/tcp_ipv4.c | 16 ++ net/ipv4/tcp_minisocks.c | 2 + 12 files changed, 333 insertions(+), 5 deletions(-) diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi= /asm/socket.h index e94f621903fe..ef4656a41058 100644 --- a/arch/alpha/include/uapi/asm/socket.h +++ b/arch/alpha/include/uapi/asm/socket.h @@ -140,6 +140,11 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 =20 +#define SO_DEVMEM_LINEAR 78 +#define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR +#define SO_DEVMEM_DMABUF 79 +#define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF + #if !defined(__KERNEL__) =20 #if __BITS_PER_LONG =3D=3D 64 diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/a= sm/socket.h index 60ebaed28a4c..414807d55e33 100644 --- a/arch/mips/include/uapi/asm/socket.h +++ b/arch/mips/include/uapi/asm/socket.h @@ -151,6 +151,11 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 =20 +#define SO_DEVMEM_LINEAR 78 +#define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR +#define SO_DEVMEM_DMABUF 79 +#define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF + #if !defined(__KERNEL__) =20 #if __BITS_PER_LONG =3D=3D 64 diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/ua= pi/asm/socket.h index be264c2b1a11..2b817efd4544 100644 --- a/arch/parisc/include/uapi/asm/socket.h +++ b/arch/parisc/include/uapi/asm/socket.h @@ -132,6 +132,11 @@ #define SO_PASSPIDFD 0x404A #define SO_PEERPIDFD 0x404B =20 +#define SO_DEVMEM_LINEAR 78 +#define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR +#define SO_DEVMEM_DMABUF 79 +#define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF + #if !defined(__KERNEL__) =20 #if __BITS_PER_LONG =3D=3D 64 diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi= /asm/socket.h index 682da3714686..00248fc68977 100644 --- a/arch/sparc/include/uapi/asm/socket.h +++ b/arch/sparc/include/uapi/asm/socket.h @@ -133,6 +133,11 @@ #define SO_PASSPIDFD 0x0055 #define SO_PEERPIDFD 0x0056 =20 +#define SO_DEVMEM_LINEAR 0x0057 +#define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR +#define SO_DEVMEM_DMABUF 0x0058 +#define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF + #if !defined(__KERNEL__) =20 =20 diff --git a/include/linux/socket.h b/include/linux/socket.h index df9cdb8bbfb8..d18cc47e89bd 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { * plain text and require encryption */ =20 +#define MSG_SOCK_DEVMEM 0x2000000 /* Receive devmem skbs as cmsg */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ #define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator i= n sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ diff --git a/include/net/sock.h b/include/net/sock.h index f51d61fab059..c58ca8dd561b 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -337,6 +337,7 @@ struct sk_filter; * @sk_txtime_report_errors: set report errors mode for SO_TXTIME * @sk_txtime_unused: unused txtime flags * @ns_tracker: tracker for netns reference + * @sk_user_frags: xarray of pages the user is holding a reference on. */ struct sock { /* @@ -542,6 +543,7 @@ struct sock { #endif struct rcu_head sk_rcu; netns_tracker ns_tracker; + struct xarray sk_user_frags; }; =20 struct sock_bh_locked { diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/s= ocket.h index 8ce8a39a1e5f..e993edc9c0ee 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,11 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 =20 +#define SO_DEVMEM_LINEAR 78 +#define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR +#define SO_DEVMEM_DMABUF 79 +#define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF + #if !defined(__KERNEL__) =20 #if __BITS_PER_LONG =3D=3D 64 || (defined(__x86_64__) && defined(__ILP32__= )) diff --git a/include/uapi/linux/uio.h b/include/uapi/linux/uio.h index 059b1a9147f4..3a22ddae376a 100644 --- a/include/uapi/linux/uio.h +++ b/include/uapi/linux/uio.h @@ -20,6 +20,19 @@ struct iovec __kernel_size_t iov_len; /* Must be size_t (1003.1g) */ }; =20 +struct dmabuf_cmsg { + __u64 frag_offset; /* offset into the dmabuf where the frag starts. + */ + __u32 frag_size; /* size of the frag. */ + __u32 frag_token; /* token representing this frag for + * DEVMEM_DONTNEED. + */ + __u32 dmabuf_id; /* dmabuf id this frag belongs to. */ + __u32 flags; /* Currently unused. Reserved for future + * uses. + */ +}; + /* * UIO_MAXIOV shall be at least 16 1003.1g (5.4.1.1) */ diff --git a/net/core/devmem.h b/net/core/devmem.h index b1db4877cff9..76099ef9c482 100644 --- a/net/core/devmem.h +++ b/net/core/devmem.h @@ -91,6 +91,19 @@ net_iov_binding(const struct net_iov *niov) return net_iov_owner(niov)->binding; } =20 +static inline unsigned long net_iov_virtual_addr(const struct net_iov *nio= v) +{ + struct dmabuf_genpool_chunk_owner *owner =3D net_iov_owner(niov); + + return owner->base_virtual + + ((unsigned long)net_iov_idx(niov) << PAGE_SHIFT); +} + +static inline u32 net_iov_binding_id(const struct net_iov *niov) +{ + return net_iov_owner(niov)->binding->id; +} + static inline void net_devmem_dmabuf_binding_get(struct net_devmem_dmabuf_binding *binding) { @@ -153,6 +166,15 @@ static inline void net_devmem_free_dmabuf(struct net_i= ov *ppiov) { } =20 +static inline unsigned long net_iov_virtual_addr(const struct net_iov *nio= v) +{ + return 0; +} + +static inline u32 net_iov_binding_id(const struct net_iov *niov) +{ + return 0; +} #endif =20 #endif /* _NET_DEVMEM_H */ diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 30238963fe99..445d17bd74e5 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -285,6 +285,8 @@ #include #include =20 +#include "../core/devmem.h" + /* Track pending CMSGs. */ enum { TCP_CMSG_INQ =3D 1, @@ -471,6 +473,7 @@ void tcp_init_sock(struct sock *sk) =20 set_bit(SOCK_SUPPORT_ZC, &sk->sk_socket->flags); sk_sockets_allocated_inc(sk); + xa_init_flags(&sk->sk_user_frags, XA_FLAGS_ALLOC1); } EXPORT_SYMBOL(tcp_init_sock); =20 @@ -2323,6 +2326,220 @@ static int tcp_inq_hint(struct sock *sk) return inq; } =20 +/* batch __xa_alloc() calls and reduce xa_lock()/xa_unlock() overhead. */ +struct tcp_xa_pool { + u8 max; /* max <=3D MAX_SKB_FRAGS */ + u8 idx; /* idx <=3D max */ + __u32 tokens[MAX_SKB_FRAGS]; + netmem_ref netmems[MAX_SKB_FRAGS]; +}; + +static void tcp_xa_pool_commit_locked(struct sock *sk, struct tcp_xa_pool = *p) +{ + int i; + + /* Commit part that has been copied to user space. */ + for (i =3D 0; i < p->idx; i++) + __xa_cmpxchg(&sk->sk_user_frags, p->tokens[i], XA_ZERO_ENTRY, + (__force void *)p->netmems[i], GFP_KERNEL); + /* Rollback what has been pre-allocated and is no longer needed. */ + for (; i < p->max; i++) + __xa_erase(&sk->sk_user_frags, p->tokens[i]); + + p->max =3D 0; + p->idx =3D 0; +} + +static void tcp_xa_pool_commit(struct sock *sk, struct tcp_xa_pool *p) +{ + if (!p->max) + return; + + xa_lock_bh(&sk->sk_user_frags); + + tcp_xa_pool_commit_locked(sk, p); + + xa_unlock_bh(&sk->sk_user_frags); +} + +static int tcp_xa_pool_refill(struct sock *sk, struct tcp_xa_pool *p, + unsigned int max_frags) +{ + int err, k; + + if (p->idx < p->max) + return 0; + + xa_lock_bh(&sk->sk_user_frags); + + tcp_xa_pool_commit_locked(sk, p); + + for (k =3D 0; k < max_frags; k++) { + err =3D __xa_alloc(&sk->sk_user_frags, &p->tokens[k], + XA_ZERO_ENTRY, xa_limit_31b, GFP_KERNEL); + if (err) + break; + } + + xa_unlock_bh(&sk->sk_user_frags); + + p->max =3D k; + p->idx =3D 0; + return k ? 0 : err; +} + +/* On error, returns the -errno. On success, returns number of bytes sent = to the + * user. May not consume all of @remaining_len. + */ +static int tcp_recvmsg_dmabuf(struct sock *sk, const struct sk_buff *skb, + unsigned int offset, struct msghdr *msg, + int remaining_len) +{ + struct dmabuf_cmsg dmabuf_cmsg =3D { 0 }; + struct tcp_xa_pool tcp_xa_pool; + unsigned int start; + int i, copy, n; + int sent =3D 0; + int err =3D 0; + + tcp_xa_pool.max =3D 0; + tcp_xa_pool.idx =3D 0; + do { + start =3D skb_headlen(skb); + + if (skb_frags_readable(skb)) { + err =3D -ENODEV; + goto out; + } + + /* Copy header. */ + copy =3D start - offset; + if (copy > 0) { + copy =3D min(copy, remaining_len); + + n =3D copy_to_iter(skb->data + offset, copy, + &msg->msg_iter); + if (n !=3D copy) { + err =3D -EFAULT; + goto out; + } + + offset +=3D copy; + remaining_len -=3D copy; + + /* First a dmabuf_cmsg for # bytes copied to user + * buffer. + */ + memset(&dmabuf_cmsg, 0, sizeof(dmabuf_cmsg)); + dmabuf_cmsg.frag_size =3D copy; + err =3D put_cmsg(msg, SOL_SOCKET, SO_DEVMEM_LINEAR, + sizeof(dmabuf_cmsg), &dmabuf_cmsg); + if (err || msg->msg_flags & MSG_CTRUNC) { + msg->msg_flags &=3D ~MSG_CTRUNC; + if (!err) + err =3D -ETOOSMALL; + goto out; + } + + sent +=3D copy; + + if (remaining_len =3D=3D 0) + goto out; + } + + /* after that, send information of dmabuf pages through a + * sequence of cmsg + */ + for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { + skb_frag_t *frag =3D &skb_shinfo(skb)->frags[i]; + struct net_iov *niov; + u64 frag_offset; + int end; + + /* !skb_frags_readable() should indicate that ALL the + * frags in this skb are dmabuf net_iovs. We're checking + * for that flag above, but also check individual frags + * here. If the tcp stack is not setting + * skb_frags_readable() correctly, we still don't want + * to crash here. + */ + if (!skb_frag_net_iov(frag)) { + net_err_ratelimited("Found non-dmabuf skb with net_iov"); + err =3D -ENODEV; + goto out; + } + + niov =3D skb_frag_net_iov(frag); + end =3D start + skb_frag_size(frag); + copy =3D end - offset; + + if (copy > 0) { + copy =3D min(copy, remaining_len); + + frag_offset =3D net_iov_virtual_addr(niov) + + skb_frag_off(frag) + offset - + start; + dmabuf_cmsg.frag_offset =3D frag_offset; + dmabuf_cmsg.frag_size =3D copy; + err =3D tcp_xa_pool_refill(sk, &tcp_xa_pool, + skb_shinfo(skb)->nr_frags - i); + if (err) + goto out; + + /* Will perform the exchange later */ + dmabuf_cmsg.frag_token =3D tcp_xa_pool.tokens[tcp_xa_pool.idx]; + dmabuf_cmsg.dmabuf_id =3D net_iov_binding_id(niov); + + offset +=3D copy; + remaining_len -=3D copy; + + err =3D put_cmsg(msg, SOL_SOCKET, + SO_DEVMEM_DMABUF, + sizeof(dmabuf_cmsg), + &dmabuf_cmsg); + if (err || msg->msg_flags & MSG_CTRUNC) { + msg->msg_flags &=3D ~MSG_CTRUNC; + if (!err) + err =3D -ETOOSMALL; + goto out; + } + + atomic_long_inc(&niov->pp_ref_count); + tcp_xa_pool.netmems[tcp_xa_pool.idx++] =3D skb_frag_netmem(frag); + + sent +=3D copy; + + if (remaining_len =3D=3D 0) + goto out; + } + start =3D end; + } + + tcp_xa_pool_commit(sk, &tcp_xa_pool); + if (!remaining_len) + goto out; + + /* if remaining_len is not satisfied yet, we need to go to the + * next frag in the frag_list to satisfy remaining_len. + */ + skb =3D skb_shinfo(skb)->frag_list ?: skb->next; + + offset =3D offset - start; + } while (skb); + + if (remaining_len) { + err =3D -EFAULT; + goto out; + } + +out: + tcp_xa_pool_commit(sk, &tcp_xa_pool); + if (!sent) + sent =3D err; + + return sent; +} + /* * This routine copies from a sock struct into the user buffer. * @@ -2336,6 +2553,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct= msghdr *msg, size_t len, int *cmsg_flags) { struct tcp_sock *tp =3D tcp_sk(sk); + int last_copied_dmabuf =3D -1; /* uninitialized */ int copied =3D 0; u32 peek_seq; u32 *seq; @@ -2515,15 +2733,44 @@ static int tcp_recvmsg_locked(struct sock *sk, stru= ct msghdr *msg, size_t len, } =20 if (!(flags & MSG_TRUNC)) { - err =3D skb_copy_datagram_msg(skb, offset, msg, used); - if (err) { - /* Exception. Bailout! */ - if (!copied) - copied =3D -EFAULT; + if (last_copied_dmabuf !=3D -1 && + last_copied_dmabuf !=3D !skb_frags_readable(skb)) break; + + if (skb_frags_readable(skb)) { + err =3D skb_copy_datagram_msg(skb, offset, msg, + used); + if (err) { + /* Exception. Bailout! */ + if (!copied) + copied =3D -EFAULT; + break; + } + } else { + if (!(flags & MSG_SOCK_DEVMEM)) { + /* dmabuf skbs can only be received + * with the MSG_SOCK_DEVMEM flag. + */ + if (!copied) + copied =3D -EFAULT; + + break; + } + + err =3D tcp_recvmsg_dmabuf(sk, skb, offset, msg, + used); + if (err <=3D 0) { + if (!copied) + copied =3D -EFAULT; + + break; + } + used =3D err; } } =20 + last_copied_dmabuf =3D !skb_frags_readable(skb); + WRITE_ONCE(*seq, *seq + used); copied +=3D used; len -=3D used; diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index eb631e66ee03..5afe5e57c89b 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -79,6 +79,7 @@ #include #include #include +#include =20 #include #include @@ -2512,10 +2513,25 @@ static void tcp_md5sig_info_free_rcu(struct rcu_hea= d *head) } #endif =20 +static void tcp_release_user_frags(struct sock *sk) +{ +#ifdef CONFIG_PAGE_POOL + unsigned long index; + void *netmem; + + xa_for_each(&sk->sk_user_frags, index, netmem) + WARN_ON_ONCE(!napi_pp_put_page((__force netmem_ref)netmem)); +#endif +} + void tcp_v4_destroy_sock(struct sock *sk) { struct tcp_sock *tp =3D tcp_sk(sk); =20 + tcp_release_user_frags(sk); + + xa_destroy(&sk->sk_user_frags); + trace_tcp_destroy_sock(sk); =20 tcp_clear_xmit_timers(sk); diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c index ad562272db2e..bb1fe1ba867a 100644 --- a/net/ipv4/tcp_minisocks.c +++ b/net/ipv4/tcp_minisocks.c @@ -628,6 +628,8 @@ struct sock *tcp_create_openreq_child(const struct sock= *sk, =20 __TCP_INC_STATS(sock_net(sk), TCP_MIB_PASSIVEOPENS); =20 + xa_init_flags(&newsk->sk_user_frags, XA_FLAGS_ALLOC1); + return newsk; } EXPORT_SYMBOL(tcp_create_openreq_child); --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8D621AF4F6 for ; Mon, 9 Sep 2024 05:43:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860632; cv=none; b=ChgfISGlidS3D9NwTOLYtoGU2uU5oKbTXhtsUAgNDVq1PXQ75p49psXH37H/f3ulaOOYIUZorbjlrdhPo0AD6z7fuPseVxNKxX6odC81Nsoo/45YY7D2bQEdzhbblPbiABscAwctJzRbYbTX3tPlqNEdEiD6hwTIdPWRxRbfWz8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860632; c=relaxed/simple; bh=y493pBuMuXtKSS1nHc/d3DLVMXqmDAj6y5DRsbVmyGQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fip0MHxHXTSWEg06Q4vqcMAkjD4ItvQJdMJ09s84IXlOzlGlXf1FLyKm0eRKO8nb0HGTGbBvy29WLR/9CqyysaeHodIoCJLINHneX427bhFhUWmJkqoB+BTVg/CrNWf37Tb7VLtSZkKg8/pJgE+2XpQavg5OfBGZJo3carA8NA0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pxip9lAM; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pxip9lAM" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1a8de19f7aso8245880276.3 for ; Sun, 08 Sep 2024 22:43:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860621; x=1726465421; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=39H1hvAt4oUXephuqAsRMNiSS8rVQOUnNeObWUoRUdg=; b=pxip9lAMdYiMHfexF+TjF9EMjtLm3e/pDGK9lmuQHcs/zqfJO6JbdfFddiaXtijtXi 5xEcZ7HlgD3L124+GWYfVbbjjiw1kYyP1dw0ROE3Aa7dFtedAlRK2Dib7+8r8Zx8VzkU pDb0sGNDu1cFqKslpMvXbBxP3nSzUxb0SowGRQYlR4Y1eSO6rHRImkDSce4X43D//tAU 0VczJSzHWTSKj2wwzM7ZHsoA5WVBvGEmBMaKyGwr2GfLLgci3ZZ8bsIfSEX87PdA5nw1 5rX8sb5laIaS36IBaz/oEp2qfM4z4eWqEbFnXRbaf7p+qJ8n8Irh7Y8RI/piUDW1VUMe y0+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860621; x=1726465421; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=39H1hvAt4oUXephuqAsRMNiSS8rVQOUnNeObWUoRUdg=; b=HtR4aRysZ9bVXfos/fcvXcV9SQDFthMafVVhNM18OpTu3OVICn2G8xP7tmhq759w/X YaLl0ZbaGhzRhMj8clTbtr/29W0WmjQNY7TgBUmPhya+GI2TPQVPtJiKV8CF4TsLo6M3 OXR4Ar9tlD3kOMGeabY1J12zvLhtTPWBgYOxpyAbKE6cpPfgzht6lJ/PQHgtMEAd8Ga2 ooA84utGtPGLy/IMiRH4p2qFtB+iD3fXzSybM4R5O9ZgPYIkv49oF73Z+3wsXnJ8T+KQ LQzl2F+05r0w1KTF6zH9kIfu3Xa+o5fw/XrK3Va7n+3W2/WUT87A+vZcMYC1vlJaVVoJ kA3Q== X-Forwarded-Encrypted: i=1; AJvYcCX9Yh8Jy1MN/SXoplCaVghJgMgdE7gkb86PChf7FhYUSeyEwKwonVvlAtQNUtqU58oVlL18U2tRiyRvOo0=@vger.kernel.org X-Gm-Message-State: AOJu0YwGpeTbzapSI9n4gcp+128Ml4PJ4YIXSV7YhYUAnCKyYp++ZdAd pVeStlxKHPqPfciLXAxHz6z08XwdijxqOz3TxNX3wM3KEHaMEZGffqfJg2O9tgENDW6rt/3sKzi R6i+CWgacOR0bPttf7K7anA== X-Google-Smtp-Source: AGHT+IEwmsbqC92+DjushWmcPnvMXZEnQwr6ZXYZAUSZ5yEdh5w1D2StqyVObxPpmpAyWdBFMpepDqvTsTcdgy7ohw== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a25:2e08:0:b0:e0b:ea2e:7b00 with SMTP id 3f1490d57ef6-e1d3489ec34mr23893276.5.1725860621103; Sun, 08 Sep 2024 22:43:41 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:15 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-11-almasrymina@google.com> Subject: [PATCH net-next v25 10/13] net: add SO_DEVMEM_DONTNEED setsockopt to release RX frags From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Willem de Bruijn , Kaiyuan Zhang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add an interface for the user to notify the kernel that it is done reading the devmem dmabuf frags returned as cmsg. The kernel will drop the reference on the frags to make them available for reuse. Signed-off-by: Willem de Bruijn Signed-off-by: Kaiyuan Zhang Signed-off-by: Mina Almasry Reviewed-by: Pavel Begunkov Reviewed-by: Eric Dumazet reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v16: - Use sk_is_tcp(). - Fix unnamed 128 DONTNEED limit (David). - Fix kernel allocating for 128 tokens even if the user didn't ask for that much (Eric). - Fix number assignement (Arnd). v10: - Fix leak of tokens (Nikolay). v7: - Updated SO_DEVMEM_* uapi to use the next available entry (Arnd). v6: - Squash in locking optimizations from edumazet@google.com. With his changes we lock the xarray once per sock_devmem_dontneed operation rather than once per frag. Changes in v1: - devmemtoken -> dmabuf_token (David). - Use napi_pp_put_page() for refcounting (Yunsheng). - Fix build error with missing socket options on other asms. --- arch/alpha/include/uapi/asm/socket.h | 1 + arch/mips/include/uapi/asm/socket.h | 1 + arch/parisc/include/uapi/asm/socket.h | 1 + arch/sparc/include/uapi/asm/socket.h | 1 + include/uapi/asm-generic/socket.h | 1 + include/uapi/linux/uio.h | 4 ++ net/core/sock.c | 68 +++++++++++++++++++++++++++ 7 files changed, 77 insertions(+) diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi= /asm/socket.h index ef4656a41058..251b73c5481e 100644 --- a/arch/alpha/include/uapi/asm/socket.h +++ b/arch/alpha/include/uapi/asm/socket.h @@ -144,6 +144,7 @@ #define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR #define SO_DEVMEM_DMABUF 79 #define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF +#define SO_DEVMEM_DONTNEED 80 =20 #if !defined(__KERNEL__) =20 diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/a= sm/socket.h index 414807d55e33..8ab7582291ab 100644 --- a/arch/mips/include/uapi/asm/socket.h +++ b/arch/mips/include/uapi/asm/socket.h @@ -155,6 +155,7 @@ #define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR #define SO_DEVMEM_DMABUF 79 #define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF +#define SO_DEVMEM_DONTNEED 80 =20 #if !defined(__KERNEL__) =20 diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/ua= pi/asm/socket.h index 2b817efd4544..38fc0b188e08 100644 --- a/arch/parisc/include/uapi/asm/socket.h +++ b/arch/parisc/include/uapi/asm/socket.h @@ -136,6 +136,7 @@ #define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR #define SO_DEVMEM_DMABUF 79 #define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF +#define SO_DEVMEM_DONTNEED 80 =20 #if !defined(__KERNEL__) =20 diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi= /asm/socket.h index 00248fc68977..57084ed2f3c4 100644 --- a/arch/sparc/include/uapi/asm/socket.h +++ b/arch/sparc/include/uapi/asm/socket.h @@ -137,6 +137,7 @@ #define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR #define SO_DEVMEM_DMABUF 0x0058 #define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF +#define SO_DEVMEM_DONTNEED 0x0059 =20 #if !defined(__KERNEL__) =20 diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/s= ocket.h index e993edc9c0ee..3b4e3e815602 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -139,6 +139,7 @@ #define SCM_DEVMEM_LINEAR SO_DEVMEM_LINEAR #define SO_DEVMEM_DMABUF 79 #define SCM_DEVMEM_DMABUF SO_DEVMEM_DMABUF +#define SO_DEVMEM_DONTNEED 80 =20 #if !defined(__KERNEL__) =20 diff --git a/include/uapi/linux/uio.h b/include/uapi/linux/uio.h index 3a22ddae376a..d17f8fcd93ec 100644 --- a/include/uapi/linux/uio.h +++ b/include/uapi/linux/uio.h @@ -33,6 +33,10 @@ struct dmabuf_cmsg { */ }; =20 +struct dmabuf_token { + __u32 token_start; + __u32 token_count; +}; /* * UIO_MAXIOV shall be at least 16 1003.1g (5.4.1.1) */ diff --git a/net/core/sock.c b/net/core/sock.c index 468b1239606c..bbb57b5af0b1 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -124,6 +124,7 @@ #include #include #include +#include #include #include #include @@ -1049,6 +1050,69 @@ static int sock_reserve_memory(struct sock *sk, int = bytes) return 0; } =20 +#ifdef CONFIG_PAGE_POOL + +/* This is the number of tokens that the user can SO_DEVMEM_DONTNEED in + * 1 syscall. The limit exists to limit the amount of memory the kernel + * allocates to copy these tokens. + */ +#define MAX_DONTNEED_TOKENS 128 + +static noinline_for_stack int +sock_devmem_dontneed(struct sock *sk, sockptr_t optval, unsigned int optle= n) +{ + unsigned int num_tokens, i, j, k, netmem_num =3D 0; + struct dmabuf_token *tokens; + netmem_ref netmems[16]; + int ret =3D 0; + + if (!sk_is_tcp(sk)) + return -EBADF; + + if (optlen % sizeof(struct dmabuf_token) || + optlen > sizeof(*tokens) * MAX_DONTNEED_TOKENS) + return -EINVAL; + + tokens =3D kvmalloc_array(optlen, sizeof(*tokens), GFP_KERNEL); + if (!tokens) + return -ENOMEM; + + num_tokens =3D optlen / sizeof(struct dmabuf_token); + if (copy_from_sockptr(tokens, optval, optlen)) { + kvfree(tokens); + return -EFAULT; + } + + xa_lock_bh(&sk->sk_user_frags); + for (i =3D 0; i < num_tokens; i++) { + for (j =3D 0; j < tokens[i].token_count; j++) { + netmem_ref netmem =3D (__force netmem_ref)__xa_erase( + &sk->sk_user_frags, tokens[i].token_start + j); + + if (netmem && + !WARN_ON_ONCE(!netmem_is_net_iov(netmem))) { + netmems[netmem_num++] =3D netmem; + if (netmem_num =3D=3D ARRAY_SIZE(netmems)) { + xa_unlock_bh(&sk->sk_user_frags); + for (k =3D 0; k < netmem_num; k++) + WARN_ON_ONCE(!napi_pp_put_page(netmems[k])); + netmem_num =3D 0; + xa_lock_bh(&sk->sk_user_frags); + } + ret++; + } + } + } + + xa_unlock_bh(&sk->sk_user_frags); + for (k =3D 0; k < netmem_num; k++) + WARN_ON_ONCE(!napi_pp_put_page(netmems[k])); + + kvfree(tokens); + return ret; +} +#endif + void sockopt_lock_sock(struct sock *sk) { /* When current->bpf_ctx is set, the setsockopt is called from @@ -1211,6 +1275,10 @@ int sk_setsockopt(struct sock *sk, int level, int op= tname, ret =3D -EOPNOTSUPP; return ret; } +#ifdef CONFIG_PAGE_POOL + case SO_DEVMEM_DONTNEED: + return sock_devmem_dontneed(sk, optval, optlen); +#endif } =20 sockopt_lock_sock(sk); --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE9751AF4F7 for ; Mon, 9 Sep 2024 05:43:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860632; cv=none; b=eKau5/dshi1AYSJvxAZyj4D7XnrXZKWgh/2cikaKoGf8G0CQmTT3Wobk3KwoGdFDK2usegghJUMmHroWCtxMH+dZB3gNMkvel6wlaOBwD7+Q+bW5XWmIGuKffUruULngRslKz/tUSZGqDalGjltWmuOZwIl4RXHZBiBzg8udQOI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860632; c=relaxed/simple; bh=YrClYxYvo9s9591djOBEnMbInkh8iKBZCYFX7TXZP6E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Qc3CvAJjIhCC4Z5RKnafJxRhidZbMmBDw2AmeITA5OCxgxB9DBeH+/oBweUThfyb4EfSCpz3xwx8h3bCzmwPBAO1ekZbtr4TgR5M5wJldG0wV+nEivyTBS4tWnp00zJSFD605cSXZoql+H9piH6OmkfQljuTGIy1Q7odgckevVM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fMY3AwF7; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fMY3AwF7" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7163489149fso4454347a12.3 for ; Sun, 08 Sep 2024 22:43:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860623; x=1726465423; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KoR6gAJPpozxkKyv/MlGxzM53ghqWNZzd2/Sd/e59V0=; b=fMY3AwF7fISA6Jw1dlHt/RTIa3zuN6RKiRxKBlw7ycsdSvi6xjRahIK++XC4Z4nHPq hkTjiNAHEuG27Pww2cEO4fYCu+TnRwWh4enVIDJ4ZWLc4emRL2H4LcOsfuWdtB9vQNUm RCwzICjAPVsjaXoVxKksJcTNlJvajI5W3EJ75lkgWRe6xLNHfVffOfZLctpt3BieBX98 +fggh2E1KnhYlIGLa3AeChONfu9ggqe76AiQqMWXFm5AA5Ps+4eLmmWWBb70nvZvQjFy UyFwH3ire7h9DsnVmaOricG9ZsYCzJ3+IM6GWgoH2rSfSyic25IBxZNYmRLVmvj4gcuu 90Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860623; x=1726465423; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KoR6gAJPpozxkKyv/MlGxzM53ghqWNZzd2/Sd/e59V0=; b=FIt9+vEBXGMroq/Po785GloUaZXCxXYx2Oy7RuR+h4tF+RgwqXP9UZBg8jfxUIvJE8 Zzhk1QSQ0Bn7clMaZii3Z4Ms6wVQgnontkYiAmkeumANcruQVi1lSpTCGFPwRhihvxsO bkD0BZBbUpa5O+/BUn2plhwCFmJM/RF6za3+gLdWXg7utEiSf6+Bk5rP8iezYPQLLD9q LaBV8LTQHxI7jF3940MyQVtA8ZaX+C7inO0+GO1HWqfWnp4/Un0xS4BT4hM6vtcq33gR M0o0vxWmtTFNpHcsUEBEmz+D1N54D5L7KU4bHlKJMLYImueiQJ7issEjwUqFZJnDSsgl lVXA== X-Forwarded-Encrypted: i=1; AJvYcCX2LF9fBsBPblUZiR+PyhAreJQeSbjIPBlu8Ughq0ZfyZe8LDMEfxYAHvDz1xglwpin5cUedbYnzG2eYaM=@vger.kernel.org X-Gm-Message-State: AOJu0YwMyqUM8AT+nEFDmB/AVSi2y09EhLyIhWjqbtRSQ24y38Qe2d7Y O77NUqlwX6AypJ2f914TTtmJPReKVeXZvjq0kXLu2FrAOsISN7GZzgV1IkHhazazLG0V/JxAaYE uLe9JLqQo7xW290hzK5GmZw== X-Google-Smtp-Source: AGHT+IF94RhlWGC5pcaNjtYx531a269LJoV3F2JoEIoZ8o1wKMmLeT6x5oWGmtlHPy5Kqp2S8SnuH9DkmpY2q6itJg== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a63:3dc7:0:b0:7d6:66fb:3da7 with SMTP id 41be03b00d2f7-7d79f5b50a2mr68880a12.3.1725860623166; Sun, 08 Sep 2024 22:43:43 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:16 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-12-almasrymina@google.com> Subject: [PATCH net-next v25 11/13] net: add devmem TCP documentation From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation outlining the usage and details of devmem TCP. Signed-off-by: Mina Almasry Reviewed-by: Bagas Sanjaya Reviewed-by: Donald Hunter reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v25: - Doc cleanups (Jakub) v16: - Add documentation on unbinding the NIC from dmabuf (Donald). - Add note that any dmabuf should work (Donald). v9: https://lore.kernel.org/netdev/20240403002053.2376017-14-almasrymina@go= ogle.com/ - Bagas doc suggestions. v8: - Applied docs suggestions (Randy). Thanks! v7: - Applied docs suggestions (Jakub). v2: - Missing spdx (simon) - add to index.rst (simon) --- Documentation/networking/devmem.rst | 269 ++++++++++++++++++++++++++++ Documentation/networking/index.rst | 1 + 2 files changed, 270 insertions(+) create mode 100644 Documentation/networking/devmem.rst diff --git a/Documentation/networking/devmem.rst b/Documentation/networking= /devmem.rst new file mode 100644 index 000000000000..a55bf21f671c --- /dev/null +++ b/Documentation/networking/devmem.rst @@ -0,0 +1,269 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Device Memory TCP +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + +Intro +=3D=3D=3D=3D=3D + +Device memory TCP (devmem TCP) enables receiving data directly into device +memory (dmabuf). The feature is currently implemented for TCP sockets. + + +Opportunity +----------- + +A large number of data transfers have device memory as the source and/or +destination. Accelerators drastically increased the prevalence of such +transfers. Some examples include: + +- Distributed training, where ML accelerators, such as GPUs on different h= osts, + exchange data. + +- Distributed raw block storage applications transfer large amounts of dat= a with + remote SSDs. Much of this data does not require host processing. + +Typically the Device-to-Device data transfers in the network are implement= ed as +the following low-level operations: Device-to-Host copy, Host-to-Host netw= ork +transfer, and Host-to-Device copy. + +The flow involving host copies is suboptimal, especially for bulk data tra= nsfers, +and can put significant strains on system resources such as host memory +bandwidth and PCIe bandwidth. + +Devmem TCP optimizes this use case by implementing socket APIs that enable +the user to receive incoming network packets directly into device memory. + +Packet payloads go directly from the NIC to device memory. + +Packet headers go to host memory and are processed by the TCP/IP stack +normally. The NIC must support header split to achieve this. + +Advantages: + +- Alleviate host memory bandwidth pressure, compared to existing + network-transfer + device-copy semantics. + +- Alleviate PCIe bandwidth pressure, by limiting data transfer to the lowe= st + level of the PCIe tree, compared to the traditional path which sends data + through the root complex. + + +More Info +--------- + + slides, video + https://netdevconf.org/0x17/sessions/talk/device-memory-tcp.html + + patchset + [PATCH net-next v24 00/13] Device Memory TCP + https://lore.kernel.org/netdev/20240831004313.3713467-1-almasrymina@go= ogle.com/ + + +Interface +=3D=3D=3D=3D=3D=3D=3D=3D=3D + + +Example +------- + +tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setti= ng up +the RX path of this API. + + +NIC Setup +--------- + +Header split, flow steering, & RSS are required features for devmem TCP. + +Header split is used to split incoming packets into a header buffer in host +memory, and a payload buffer in device memory. + +Flow steering & RSS are used to ensure that only flows targeting devmem la= nd on +an RX queue bound to devmem. + +Enable header split & flow steering:: + + # enable header split + ethtool -G eth1 tcp-data-split on + + + # enable flow steering + ethtool -K eth1 ntuple on + +Configure RSS to steer all traffic away from the target RX queue (queue 15= in +this example):: + + ethtool --set-rxfh-indir eth1 equal 15 + + +The user must bind a dmabuf to any number of RX queues on a given NIC using +the netlink API:: + + /* Bind dmabuf to NIC RX queue 15 */ + struct netdev_queue *queues; + queues =3D malloc(sizeof(*queues) * 1); + + queues[0]._present.type =3D 1; + queues[0]._present.idx =3D 1; + queues[0].type =3D NETDEV_RX_QUEUE_TYPE_RX; + queues[0].idx =3D 15; + + *ys =3D ynl_sock_create(&ynl_netdev_family, &yerr); + + req =3D netdev_bind_rx_req_alloc(); + netdev_bind_rx_req_set_ifindex(req, 1 /* ifindex */); + netdev_bind_rx_req_set_dmabuf_fd(req, dmabuf_fd); + __netdev_bind_rx_req_set_queues(req, queues, n_queue_index); + + rsp =3D netdev_bind_rx(*ys, req); + + dmabuf_id =3D rsp->dmabuf_id; + + +The netlink API returns a dmabuf_id: a unique ID that refers to this dmabuf +that has been bound. + +The user can unbind the dmabuf from the netdevice by closing the netlink s= ocket +that established the binding. We do this so that the binding is automatica= lly +unbound even if the userspace process crashes. + +Note that any reasonably well-behaved dmabuf from any exporter should work= with +devmem TCP, even if the dmabuf is not actually backed by devmem. An exampl= e of +this is udmabuf, which wraps user memory (non-devmem) in a dmabuf. + + +Socket Setup +------------ + +The socket must be flow steered to the dmabuf bound RX queue:: + + ethtool -N eth1 flow-type tcp4 ... queue 15 + + +Receiving data +-------------- + +The user application must signal to the kernel that it is capable of recei= ving +devmem data by passing the MSG_SOCK_DEVMEM flag to recvmsg:: + + ret =3D recvmsg(fd, &msg, MSG_SOCK_DEVMEM); + +Applications that do not specify the MSG_SOCK_DEVMEM flag will receive an = EFAULT +on devmem data. + +Devmem data is received directly into the dmabuf bound to the NIC in 'NIC +Setup', and the kernel signals such to the user via the SCM_DEVMEM_* cmsgs= :: + + for (cm =3D CMSG_FIRSTHDR(&msg); cm; cm =3D CMSG_NXTHDR(&msg, cm)) { + if (cm->cmsg_level !=3D SOL_SOCKET || + (cm->cmsg_type !=3D SCM_DEVMEM_DMABUF && + cm->cmsg_type !=3D SCM_DEVMEM_LINEAR)) + continue; + + dmabuf_cmsg =3D (struct dmabuf_cmsg *)CMSG_DATA(cm); + + if (cm->cmsg_type =3D=3D SCM_DEVMEM_DMABUF) { + /* Frag landed in dmabuf. + * + * dmabuf_cmsg->dmabuf_id is the dmabuf the + * frag landed on. + * + * dmabuf_cmsg->frag_offset is the offset into + * the dmabuf where the frag starts. + * + * dmabuf_cmsg->frag_size is the size of the + * frag. + * + * dmabuf_cmsg->frag_token is a token used to + * refer to this frag for later freeing. + */ + + struct dmabuf_token token; + token.token_start =3D dmabuf_cmsg->frag_token; + token.token_count =3D 1; + continue; + } + + if (cm->cmsg_type =3D=3D SCM_DEVMEM_LINEAR) + /* Frag landed in linear buffer. + * + * dmabuf_cmsg->frag_size is the size of the + * frag. + */ + continue; + + } + +Applications may receive 2 cmsgs: + +- SCM_DEVMEM_DMABUF: this indicates the fragment landed in the dmabuf indi= cated + by dmabuf_id. + +- SCM_DEVMEM_LINEAR: this indicates the fragment landed in the linear buff= er. + This typically happens when the NIC is unable to split the packet at the + header boundary, such that part (or all) of the payload landed in host + memory. + +Applications may receive no SO_DEVMEM_* cmsgs. That indicates non-devmem, +regular TCP data that landed on an RX queue not bound to a dmabuf. + + +Freeing frags +------------- + +Frags received via SCM_DEVMEM_DMABUF are pinned by the kernel while the us= er +processes the frag. The user must return the frag to the kernel via +SO_DEVMEM_DONTNEED:: + + ret =3D setsockopt(client_fd, SOL_SOCKET, SO_DEVMEM_DONTNEED, &token, + sizeof(token)); + +The user must ensure the tokens are returned to the kernel in a timely man= ner. +Failure to do so will exhaust the limited dmabuf that is bound to the RX q= ueue +and will lead to packet drops. + + +Implementation & Caveats +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Unreadable skbs +--------------- + +Devmem payloads are inaccessible to the kernel processing the packets. This +results in a few quirks for payloads of devmem skbs: + +- Loopback is not functional. Loopback relies on copying the payload, whic= h is + not possible with devmem skbs. + +- Software checksum calculation fails. + +- TCP Dump and bpf can't access devmem packet payloads. + + +Testing +=3D=3D=3D=3D=3D=3D=3D + +More realistic example code can be found in the kernel source under +``tools/testing/selftests/net/ncdevmem.c`` + +ncdevmem is a devmem TCP netcat. It works very similarly to netcat, but +receives data directly into a udmabuf. + +To run ncdevmem, you need to run it on a server on the machine under test,= and +you need to run netcat on a peer to provide the TX data. + +ncdevmem has a validation mode as well that expects a repeating pattern of +incoming data and validates it as such. For example, you can launch +ncdevmem on the server by:: + + ncdevmem -s -c -f eth1 -d 3 -n 0000:06:00.0 -l \ + -p 5201 -v 7 + +On client side, use regular netcat to send TX data to ncdevmem process +on the server:: + + yes $(echo -e \\x01\\x02\\x03\\x04\\x05\\x06) | \ + tr \\n \\0 | head -c 5G | nc 5201 -p 5201 diff --git a/Documentation/networking/index.rst b/Documentation/networking/= index.rst index c71b87346178..08f437c326ab 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -49,6 +49,7 @@ Contents: cdc_mbim dccp dctcp + devmem dns_resolver driver eql --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0ACE31B013F for ; Mon, 9 Sep 2024 05:43:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860634; cv=none; b=YfwEh3Yyr0z/LHSoiZsEfyAT/YxttFoWrpNliilij1ebUmTB02Zgv5nbvqYrcL1hRZQC/oYpFyKb3nf/ug3t1A0hblkTsxHLy/C5AkbLGRnx+976pm3tNH41InZjJsxNH3aoj1ng7MaFOk07B/Hm2/PMVl4xwAd7fjSD58Poasw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860634; c=relaxed/simple; bh=2t0/+0TBLvt24U/Cay2GykrKaGxoygrhy1NVq2NzoCQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qEcel5s3TmH8/WreFM/W8ZTbP9J3by2RdbnRiKttE0Mh1zNtMz7mUQOeMPpOP8BJ+cG3KHyjYvkrV0GPWa/Ik5LM5b6MXPPXt/ZzuKeu9BToionS/BSiI1spzYHH8BqqozALxECJkofi8XfR3u15fGYZr/NsmtNeHdMdr/ojS3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SWRYyWmq; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SWRYyWmq" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e1ce191f74fso8229981276.2 for ; Sun, 08 Sep 2024 22:43:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860625; x=1726465425; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4UCmDs+YyvPenlA61IqWCFYTu9251Izdm+WiM3HHOus=; b=SWRYyWmqOuaQk0wOJMlulBFRvpSlFghWilrbZvUjuIKX1b2y0uGq+TQun6QLa/Jgay I8zyvKifZh4fCb2W4M8cCGTkVaMicSomawscetljNsmhHD+fqLjkzheHOP59LxLMum0Y +ENR8yiHp9a9hI/NvgZG0MS1oFhOfq0Q9cMB4EfcNENNwzFKrqoWAns/+VjMxpwKQg6W jh80+2ylPDmqvd9TNC/jxeQU3BWDAOJ6WDbrnAlDmvyT3EUHCFT3hw53BpCChVq3dLDQ ub9Usz5aF1Rv4Ps8lf9iYrlN9mReDPC5AdfEMW9o3H8EfgNwOkuqTn6xL3ps+lBIFkFi +Eow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860625; x=1726465425; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4UCmDs+YyvPenlA61IqWCFYTu9251Izdm+WiM3HHOus=; b=B/CE/HdY7MxZsCOZ82QRrcT5iM7tVTRV3pLYFe7po4Aa1Bd09/gNFZBBsQl3uFHgLC taCS4nQz6iu3N3+TfOsaEJMtMCmNnCIVT1HRNMZl4wXR/4QQXU37+oWtG/wXHIRck4pj OQGQA2RWRxHIUJ2j49TpSaVqsM2iJxaEDhUhOrKHdWsIYwX3gacuKpwXJDHtMqzidwas 0uLEhVhiuaRn4q9h3UwI25g4IGd/81PUm3gYZCkVQR+8xZINiznGP8zJ1SslcmBdbt8R Ol4DXQ0WIAfbZilqahQaifF0c75z5510YFLuWkg8S+ulrIoEsjnJmVD1gTHivNJGBG97 mMBw== X-Forwarded-Encrypted: i=1; AJvYcCW7bQlHM37fJOzHQQAXl3Ifi4JHFZ+iy0iBZlAb06RlrPEJe9xKH31HSdVVCFKG8YS5dHy13MDloOPyF5Y=@vger.kernel.org X-Gm-Message-State: AOJu0Yy/Rz/xPQRc9MvQ+VVYdtYSBxqr+5HqgQYSWgZFaLxhFiKMM+da VR8IAW7UNToykLolIIb9uiSzPwocST+b1sVWcMCmbKqi/mrOjlBwcTXaD1a4nhUJARvgaVYTN6o kgorls8ic3/Q2eOSQ2RyIrA== X-Google-Smtp-Source: AGHT+IE2dzlbqeOCkIJo0ES06/D7J34uo+HoGhvXFeIDFEeJtb7QCm38enDVong8KvW/miT6qomHWpfJwUgdnX9wkQ== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a5b:b50:0:b0:e0e:445b:606 with SMTP id 3f1490d57ef6-e1d346b001fmr18940276.0.1725860625148; Sun, 08 Sep 2024 22:43:45 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:17 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-13-almasrymina@google.com> Subject: [PATCH net-next v25 12/13] selftests: add ncdevmem, netcat for devmem TCP From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo , Stanislav Fomichev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ncdevmem is a devmem TCP netcat. It works similarly to netcat, but it sends and receives data using the devmem TCP APIs. It uses udmabuf as the dmabuf provider. It is compatible with a regular netcat running on a peer, or a ncdevmem running on a peer. In addition to normal netcat support, ncdevmem has a validation mode, where it sends a specific pattern and validates this pattern on the receiver side to ensure data integrity. Suggested-by: Stanislav Fomichev Signed-off-by: Mina Almasry reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v24: - Add *.d to .gitignore, to stop tracking this file the build generates: ``` git status ... ?? tools/net/ynl/lib/ynl.d ``` v22: - Add run_command helper. It reduces boiler plate and prints the commands it is running. v20: - Remove unnecessary sleep(1) - Add test to ensure dmabuf binding fails if header split is disabled. v19: - Check return code of ethtool commands. - Add test for deactivating mp bound rx queues - Add test for attempting to bind with missing netlink attributes. v16: - Remove outdated -n option (Taehee). - Use 'ifname' instead of accidentally hardcoded 'eth1'. (Taehee) - Remove dead code 'iterations' (Taehee). - Use if_nametoindex() instead of passing device index (Taehee). v15: - Fix linking against libynl. (Jakub) v9: https://lore.kernel.org/netdev/20240403002053.2376017-15-almasrymina@go= ogle.com/ - Remove unused nic_pci_addr entry (Cong). v6: - Updated to bind 8 queues. - Added RSS configuration. - Added some more tests for the netlink API. Changes in v1: - Many more general cleanups (Willem). - Removed driver reset (Jakub). - Removed hardcoded if index (Paolo). RFC v2: - General cleanups (Willem). --- tools/net/ynl/lib/.gitignore | 1 + tools/testing/selftests/net/.gitignore | 1 + tools/testing/selftests/net/Makefile | 9 + tools/testing/selftests/net/ncdevmem.c | 570 +++++++++++++++++++++++++ 4 files changed, 581 insertions(+) create mode 100644 tools/testing/selftests/net/ncdevmem.c diff --git a/tools/net/ynl/lib/.gitignore b/tools/net/ynl/lib/.gitignore index c18dd8d83cee..296c4035dbf2 100644 --- a/tools/net/ynl/lib/.gitignore +++ b/tools/net/ynl/lib/.gitignore @@ -1 +1,2 @@ __pycache__/ +*.d diff --git a/tools/testing/selftests/net/.gitignore b/tools/testing/selftes= ts/net/.gitignore index 923bf098e2eb..1c04c780db66 100644 --- a/tools/testing/selftests/net/.gitignore +++ b/tools/testing/selftests/net/.gitignore @@ -17,6 +17,7 @@ ipv6_flowlabel ipv6_flowlabel_mgr log.txt msg_zerocopy +ncdevmem nettest psock_fanout psock_snd diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests= /net/Makefile index 27362e40eb37..fc1c3a64de2d 100644 --- a/tools/testing/selftests/net/Makefile +++ b/tools/testing/selftests/net/Makefile @@ -97,6 +97,11 @@ TEST_PROGS +=3D fq_band_pktlimit.sh TEST_PROGS +=3D vlan_hw_filter.sh TEST_PROGS +=3D bpf_offload.py =20 +# YNL files, must be before "include ..lib.mk" +EXTRA_CLEAN +=3D $(OUTPUT)/libynl.a +YNL_GEN_FILES :=3D ncdevmem +TEST_GEN_FILES +=3D $(YNL_GEN_FILES) + TEST_FILES :=3D settings TEST_FILES +=3D in_netns.sh lib.sh net_helper.sh setup_loopback.sh setup_v= eth.sh =20 @@ -106,6 +111,10 @@ TEST_INCLUDES :=3D forwarding/lib.sh =20 include ../lib.mk =20 +# YNL build +YNL_GENS :=3D netdev +include ynl.mk + $(OUTPUT)/epoll_busy_poll: LDLIBS +=3D -lcap $(OUTPUT)/reuseport_bpf_numa: LDLIBS +=3D -lnuma $(OUTPUT)/tcp_mmap: LDLIBS +=3D -lpthread -lcrypto diff --git a/tools/testing/selftests/net/ncdevmem.c b/tools/testing/selftes= ts/net/ncdevmem.c new file mode 100644 index 000000000000..64d6805381c5 --- /dev/null +++ b/tools/testing/selftests/net/ncdevmem.c @@ -0,0 +1,570 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#define __EXPORTED_HEADERS__ + +#include +#include +#include +#include +#include +#include +#include +#define __iovec_defined +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "netdev-user.h" +#include + +#define PAGE_SHIFT 12 +#define TEST_PREFIX "ncdevmem" +#define NUM_PAGES 16000 + +#ifndef MSG_SOCK_DEVMEM +#define MSG_SOCK_DEVMEM 0x2000000 +#endif + +/* + * tcpdevmem netcat. Works similarly to netcat but does device memory TCP + * instead of regular TCP. Uses udmabuf to mock a dmabuf provider. + * + * Usage: + * + * On server: + * ncdevmem -s -c -f eth1 -l -p 5201 -v 7 + * + * On client: + * yes $(echo -e \\x01\\x02\\x03\\x04\\x05\\x06) | \ + * tr \\n \\0 | \ + * head -c 5G | \ + * nc 5201 -p 5201 + * + * Note this is compatible with regular netcat. i.e. the sender or receive= r can + * be replaced with regular netcat to test the RX or TX path in isolation. + */ + +static char *server_ip =3D "192.168.1.4"; +static char *client_ip =3D "192.168.1.2"; +static char *port =3D "5201"; +static size_t do_validation; +static int start_queue =3D 8; +static int num_queues =3D 8; +static char *ifname =3D "eth1"; +static unsigned int ifindex; +static unsigned int dmabuf_id; + +void print_bytes(void *ptr, size_t size) +{ + unsigned char *p =3D ptr; + int i; + + for (i =3D 0; i < size; i++) + printf("%02hhX ", p[i]); + printf("\n"); +} + +void print_nonzero_bytes(void *ptr, size_t size) +{ + unsigned char *p =3D ptr; + unsigned int i; + + for (i =3D 0; i < size; i++) + putchar(p[i]); + printf("\n"); +} + +void validate_buffer(void *line, size_t size) +{ + static unsigned char seed =3D 1; + unsigned char *ptr =3D line; + int errors =3D 0; + size_t i; + + for (i =3D 0; i < size; i++) { + if (ptr[i] !=3D seed) { + fprintf(stderr, + "Failed validation: expected=3D%u, actual=3D%u, index=3D%lu\n", + seed, ptr[i], i); + errors++; + if (errors > 20) + error(1, 0, "validation failed."); + } + seed++; + if (seed =3D=3D do_validation) + seed =3D 0; + } + + fprintf(stdout, "Validated buffer\n"); +} + +#define run_command(cmd, ...) \ + ({ \ + char command[256]; \ + memset(command, 0, sizeof(command)); \ + snprintf(command, sizeof(command), cmd, ##__VA_ARGS__); \ + printf("Running: %s\n", command); \ + system(command); \ + }) + +static int reset_flow_steering(void) +{ + int ret =3D 0; + + ret =3D run_command("sudo ethtool -K %s ntuple off", ifname); + if (ret) + return ret; + + return run_command("sudo ethtool -K %s ntuple on", ifname); +} + +static int configure_headersplit(bool on) +{ + return run_command("sudo ethtool -G %s tcp-data-split %s", ifname, + on ? "on" : "off"); +} + +static int configure_rss(void) +{ + return run_command("sudo ethtool -X %s equal %d", ifname, start_queue); +} + +static int configure_channels(unsigned int rx, unsigned int tx) +{ + return run_command("sudo ethtool -L %s rx %u tx %u", ifname, rx, tx); +} + +static int configure_flow_steering(void) +{ + return run_command("sudo ethtool -N %s flow-type tcp4 src-ip %s dst-ip %s= src-port %s dst-port %s queue %d", + ifname, client_ip, server_ip, port, port, start_queue); +} + +static int bind_rx_queue(unsigned int ifindex, unsigned int dmabuf_fd, + struct netdev_queue_id *queues, + unsigned int n_queue_index, struct ynl_sock **ys) +{ + struct netdev_bind_rx_req *req =3D NULL; + struct netdev_bind_rx_rsp *rsp =3D NULL; + struct ynl_error yerr; + + *ys =3D ynl_sock_create(&ynl_netdev_family, &yerr); + if (!*ys) { + fprintf(stderr, "YNL: %s\n", yerr.msg); + return -1; + } + + req =3D netdev_bind_rx_req_alloc(); + netdev_bind_rx_req_set_ifindex(req, ifindex); + netdev_bind_rx_req_set_fd(req, dmabuf_fd); + __netdev_bind_rx_req_set_queues(req, queues, n_queue_index); + + rsp =3D netdev_bind_rx(*ys, req); + if (!rsp) { + perror("netdev_bind_rx"); + goto err_close; + } + + if (!rsp->_present.id) { + perror("id not present"); + goto err_close; + } + + printf("got dmabuf id=3D%d\n", rsp->id); + dmabuf_id =3D rsp->id; + + netdev_bind_rx_req_free(req); + netdev_bind_rx_rsp_free(rsp); + + return 0; + +err_close: + fprintf(stderr, "YNL failed: %s\n", (*ys)->err.msg); + netdev_bind_rx_req_free(req); + ynl_sock_destroy(*ys); + return -1; +} + +static void create_udmabuf(int *devfd, int *memfd, int *buf, size_t dmabuf= _size) +{ + struct udmabuf_create create; + int ret; + + *devfd =3D open("/dev/udmabuf", O_RDWR); + if (*devfd < 0) { + error(70, 0, + "%s: [skip,no-udmabuf: Unable to access DMA buffer device file]\n", + TEST_PREFIX); + } + + *memfd =3D memfd_create("udmabuf-test", MFD_ALLOW_SEALING); + if (*memfd < 0) + error(70, 0, "%s: [skip,no-memfd]\n", TEST_PREFIX); + + /* Required for udmabuf */ + ret =3D fcntl(*memfd, F_ADD_SEALS, F_SEAL_SHRINK); + if (ret < 0) + error(73, 0, "%s: [skip,fcntl-add-seals]\n", TEST_PREFIX); + + ret =3D ftruncate(*memfd, dmabuf_size); + if (ret =3D=3D -1) + error(74, 0, "%s: [FAIL,memfd-truncate]\n", TEST_PREFIX); + + memset(&create, 0, sizeof(create)); + + create.memfd =3D *memfd; + create.offset =3D 0; + create.size =3D dmabuf_size; + *buf =3D ioctl(*devfd, UDMABUF_CREATE, &create); + if (*buf < 0) + error(75, 0, "%s: [FAIL, create udmabuf]\n", TEST_PREFIX); +} + +int do_server(void) +{ + char ctrl_data[sizeof(int) * 20000]; + struct netdev_queue_id *queues; + size_t non_page_aligned_frags =3D 0; + struct sockaddr_in client_addr; + struct sockaddr_in server_sin; + size_t page_aligned_frags =3D 0; + int devfd, memfd, buf, ret; + size_t total_received =3D 0; + socklen_t client_addr_len; + bool is_devmem =3D false; + char *buf_mem =3D NULL; + struct ynl_sock *ys; + size_t dmabuf_size; + char iobuf[819200]; + char buffer[256]; + int socket_fd; + int client_fd; + size_t i =3D 0; + int opt =3D 1; + + dmabuf_size =3D getpagesize() * NUM_PAGES; + + create_udmabuf(&devfd, &memfd, &buf, dmabuf_size); + + if (reset_flow_steering()) + error(1, 0, "Failed to reset flow steering\n"); + + /* Configure RSS to divert all traffic from our devmem queues */ + if (configure_rss()) + error(1, 0, "Failed to configure rss\n"); + + /* Flow steer our devmem flows to start_queue */ + if (configure_flow_steering()) + error(1, 0, "Failed to configure flow steering\n"); + + sleep(1); + + queues =3D malloc(sizeof(*queues) * num_queues); + + for (i =3D 0; i < num_queues; i++) { + queues[i]._present.type =3D 1; + queues[i]._present.id =3D 1; + queues[i].type =3D NETDEV_QUEUE_TYPE_RX; + queues[i].id =3D start_queue + i; + } + + if (bind_rx_queue(ifindex, buf, queues, num_queues, &ys)) + error(1, 0, "Failed to bind\n"); + + buf_mem =3D mmap(NULL, dmabuf_size, PROT_READ | PROT_WRITE, MAP_SHARED, + buf, 0); + if (buf_mem =3D=3D MAP_FAILED) + error(1, 0, "mmap()"); + + server_sin.sin_family =3D AF_INET; + server_sin.sin_port =3D htons(atoi(port)); + + ret =3D inet_pton(server_sin.sin_family, server_ip, &server_sin.sin_addr); + if (socket < 0) + error(79, 0, "%s: [FAIL, create socket]\n", TEST_PREFIX); + + socket_fd =3D socket(server_sin.sin_family, SOCK_STREAM, 0); + if (socket < 0) + error(errno, errno, "%s: [FAIL, create socket]\n", TEST_PREFIX); + + ret =3D setsockopt(socket_fd, SOL_SOCKET, SO_REUSEPORT, &opt, + sizeof(opt)); + if (ret) + error(errno, errno, "%s: [FAIL, set sock opt]\n", TEST_PREFIX); + + ret =3D setsockopt(socket_fd, SOL_SOCKET, SO_REUSEADDR, &opt, + sizeof(opt)); + if (ret) + error(errno, errno, "%s: [FAIL, set sock opt]\n", TEST_PREFIX); + + printf("binding to address %s:%d\n", server_ip, + ntohs(server_sin.sin_port)); + + ret =3D bind(socket_fd, &server_sin, sizeof(server_sin)); + if (ret) + error(errno, errno, "%s: [FAIL, bind]\n", TEST_PREFIX); + + ret =3D listen(socket_fd, 1); + if (ret) + error(errno, errno, "%s: [FAIL, listen]\n", TEST_PREFIX); + + client_addr_len =3D sizeof(client_addr); + + inet_ntop(server_sin.sin_family, &server_sin.sin_addr, buffer, + sizeof(buffer)); + printf("Waiting or connection on %s:%d\n", buffer, + ntohs(server_sin.sin_port)); + client_fd =3D accept(socket_fd, &client_addr, &client_addr_len); + + inet_ntop(client_addr.sin_family, &client_addr.sin_addr, buffer, + sizeof(buffer)); + printf("Got connection from %s:%d\n", buffer, + ntohs(client_addr.sin_port)); + + while (1) { + struct iovec iov =3D { .iov_base =3D iobuf, + .iov_len =3D sizeof(iobuf) }; + struct dmabuf_cmsg *dmabuf_cmsg =3D NULL; + struct dma_buf_sync sync =3D { 0 }; + struct cmsghdr *cm =3D NULL; + struct msghdr msg =3D { 0 }; + struct dmabuf_token token; + ssize_t ret; + + is_devmem =3D false; + printf("\n\n"); + + msg.msg_iov =3D &iov; + msg.msg_iovlen =3D 1; + msg.msg_control =3D ctrl_data; + msg.msg_controllen =3D sizeof(ctrl_data); + ret =3D recvmsg(client_fd, &msg, MSG_SOCK_DEVMEM); + printf("recvmsg ret=3D%ld\n", ret); + if (ret < 0 && (errno =3D=3D EAGAIN || errno =3D=3D EWOULDBLOCK)) + continue; + if (ret < 0) { + perror("recvmsg"); + continue; + } + if (ret =3D=3D 0) { + printf("client exited\n"); + goto cleanup; + } + + i++; + for (cm =3D CMSG_FIRSTHDR(&msg); cm; cm =3D CMSG_NXTHDR(&msg, cm)) { + if (cm->cmsg_level !=3D SOL_SOCKET || + (cm->cmsg_type !=3D SCM_DEVMEM_DMABUF && + cm->cmsg_type !=3D SCM_DEVMEM_LINEAR)) { + fprintf(stdout, "skipping non-devmem cmsg\n"); + continue; + } + + dmabuf_cmsg =3D (struct dmabuf_cmsg *)CMSG_DATA(cm); + is_devmem =3D true; + + if (cm->cmsg_type =3D=3D SCM_DEVMEM_LINEAR) { + /* TODO: process data copied from skb's linear + * buffer. + */ + fprintf(stdout, + "SCM_DEVMEM_LINEAR. dmabuf_cmsg->frag_size=3D%u\n", + dmabuf_cmsg->frag_size); + + continue; + } + + token.token_start =3D dmabuf_cmsg->frag_token; + token.token_count =3D 1; + + total_received +=3D dmabuf_cmsg->frag_size; + printf("received frag_page=3D%llu, in_page_offset=3D%llu, frag_offset= =3D%llu, frag_size=3D%u, token=3D%u, total_received=3D%lu, dmabuf_id=3D%u\n= ", + dmabuf_cmsg->frag_offset >> PAGE_SHIFT, + dmabuf_cmsg->frag_offset % getpagesize(), + dmabuf_cmsg->frag_offset, dmabuf_cmsg->frag_size, + dmabuf_cmsg->frag_token, total_received, + dmabuf_cmsg->dmabuf_id); + + if (dmabuf_cmsg->dmabuf_id !=3D dmabuf_id) + error(1, 0, + "received on wrong dmabuf_id: flow steering error\n"); + + if (dmabuf_cmsg->frag_size % getpagesize()) + non_page_aligned_frags++; + else + page_aligned_frags++; + + sync.flags =3D DMA_BUF_SYNC_READ | DMA_BUF_SYNC_START; + ioctl(buf, DMA_BUF_IOCTL_SYNC, &sync); + + if (do_validation) + validate_buffer( + ((unsigned char *)buf_mem) + + dmabuf_cmsg->frag_offset, + dmabuf_cmsg->frag_size); + else + print_nonzero_bytes( + ((unsigned char *)buf_mem) + + dmabuf_cmsg->frag_offset, + dmabuf_cmsg->frag_size); + + sync.flags =3D DMA_BUF_SYNC_READ | DMA_BUF_SYNC_END; + ioctl(buf, DMA_BUF_IOCTL_SYNC, &sync); + + ret =3D setsockopt(client_fd, SOL_SOCKET, + SO_DEVMEM_DONTNEED, &token, + sizeof(token)); + if (ret !=3D 1) + error(1, 0, + "SO_DEVMEM_DONTNEED not enough tokens"); + } + if (!is_devmem) + error(1, 0, "flow steering error\n"); + + printf("total_received=3D%lu\n", total_received); + } + + fprintf(stdout, "%s: ok\n", TEST_PREFIX); + + fprintf(stdout, "page_aligned_frags=3D%lu, non_page_aligned_frags=3D%lu\n= ", + page_aligned_frags, non_page_aligned_frags); + + fprintf(stdout, "page_aligned_frags=3D%lu, non_page_aligned_frags=3D%lu\n= ", + page_aligned_frags, non_page_aligned_frags); + +cleanup: + + munmap(buf_mem, dmabuf_size); + close(client_fd); + close(socket_fd); + close(buf); + close(memfd); + close(devfd); + ynl_sock_destroy(ys); + + return 0; +} + +void run_devmem_tests(void) +{ + struct netdev_queue_id *queues; + int devfd, memfd, buf; + struct ynl_sock *ys; + size_t dmabuf_size; + size_t i =3D 0; + + dmabuf_size =3D getpagesize() * NUM_PAGES; + + create_udmabuf(&devfd, &memfd, &buf, dmabuf_size); + + /* Configure RSS to divert all traffic from our devmem queues */ + if (configure_rss()) + error(1, 0, "rss error\n"); + + queues =3D calloc(num_queues, sizeof(*queues)); + + if (configure_headersplit(1)) + error(1, 0, "Failed to configure header split\n"); + + if (!bind_rx_queue(ifindex, buf, queues, num_queues, &ys)) + error(1, 0, "Binding empty queues array should have failed\n"); + + for (i =3D 0; i < num_queues; i++) { + queues[i]._present.type =3D 1; + queues[i]._present.id =3D 1; + queues[i].type =3D NETDEV_QUEUE_TYPE_RX; + queues[i].id =3D start_queue + i; + } + + if (configure_headersplit(0)) + error(1, 0, "Failed to configure header split\n"); + + if (!bind_rx_queue(ifindex, buf, queues, num_queues, &ys)) + error(1, 0, "Configure dmabuf with header split off should have failed\n= "); + + if (configure_headersplit(1)) + error(1, 0, "Failed to configure header split\n"); + + for (i =3D 0; i < num_queues; i++) { + queues[i]._present.type =3D 1; + queues[i]._present.id =3D 1; + queues[i].type =3D NETDEV_QUEUE_TYPE_RX; + queues[i].id =3D start_queue + i; + } + + if (bind_rx_queue(ifindex, buf, queues, num_queues, &ys)) + error(1, 0, "Failed to bind\n"); + + /* Deactivating a bound queue should not be legal */ + if (!configure_channels(num_queues, num_queues - 1)) + error(1, 0, "Deactivating a bound queue should be illegal.\n"); + + /* Closing the netlink socket does an implicit unbind */ + ynl_sock_destroy(ys); +} + +int main(int argc, char *argv[]) +{ + int is_server =3D 0, opt; + + while ((opt =3D getopt(argc, argv, "ls:c:p:v:q:t:f:")) !=3D -1) { + switch (opt) { + case 'l': + is_server =3D 1; + break; + case 's': + server_ip =3D optarg; + break; + case 'c': + client_ip =3D optarg; + break; + case 'p': + port =3D optarg; + break; + case 'v': + do_validation =3D atoll(optarg); + break; + case 'q': + num_queues =3D atoi(optarg); + break; + case 't': + start_queue =3D atoi(optarg); + break; + case 'f': + ifname =3D optarg; + break; + case '?': + printf("unknown option: %c\n", optopt); + break; + } + } + + ifindex =3D if_nametoindex(ifname); + + for (; optind < argc; optind++) + printf("extra arguments: %s\n", argv[optind]); + + run_devmem_tests(); + + if (is_server) + return do_server(); + + return 0; +} --=20 2.46.0.469.g59c65b2a67-goog From nobody Sat Nov 30 16:21:13 2024 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 707E11B1420 for ; Mon, 9 Sep 2024 05:43:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860635; cv=none; b=PthlGdRVUnv6v1ft99lhC+QZskG0iJBrFuOxUJg11ofuQQK0G+q5Qaf96GHWzqK2ql0xqQxJdG0XDe9h0uB7y7pFzjnnrYtNdyE4XLGumkdoPgsvLZeKv58JikCJkLsawGjDxKiRMw2sEhK7HxSOxrUFxVqoBXtztMgalf5Ng+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725860635; c=relaxed/simple; bh=PJB8MnDkY64lJweDrjdNdgxrCAth9X9p/HaOyMbjVAA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=P2eJIcnhJNh9G4CsDPWc4EPqrmqbqwY8d0FjSg6f6LOwHY1bqE/vzyuGUm0kn3Dyl8AtRtMAK++pKSC6ehOGpoW9ByzVrYA/z3mK8gtz6RdBYeczvRuWKzCcaR4HUisUudw4lgtzfXVTXaDDxQXqwrv/eS643t6bYSrcqERlDLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FjgbLPZK; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FjgbLPZK" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7d4fc4652f6so4906493a12.0 for ; Sun, 08 Sep 2024 22:43:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725860628; x=1726465428; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WTSeEQvvLz7JMkJB8rY34SUA6KPOOIDr3cyazlDccxw=; b=FjgbLPZKVhfU2DWXWw/vaBN4AiU7a72MxExlOzESVkOU4SjQ9DN4jKTq+y/GpwILhE nGkj8H+cuSH6jVIkLN8egUsm0Ted/iBL4R08oRUo7yVzWPIO4Z304awPqdekLEA18g5p Mw2cj319aYt9sIDSryz+A8vDTOhW2WY4e8zO2nXzKjODkOlUeOIGpSFnozo3jY+7GebC UOFFCYusTlVhkTPnuft1hv7hrnnMsVVv75SrYgwGiVajRxz4c5tsFD3NUVQgV6y8qNGU xx9YJrfsBb9XlKQzwPt4xUgoSt1/NOK2N8dTS/MFWihI9RboOGtW/8fxmVoFlm5lUbNL 8p7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725860628; x=1726465428; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WTSeEQvvLz7JMkJB8rY34SUA6KPOOIDr3cyazlDccxw=; b=DmU4BRvLHidPV0i+4cIQUNASGUwiMeXHatXxY4iDc1AW8gjPAw14v1SgiDpwJFwQB4 Yha+UXeTduan57EG/MQ8DhKMvJGFGW7oV2lpIrFoDtlS4JUnjmbp19NxJIOgC/AwpGhI bI1FDkAAmNoW+7AUwzgPY/piIPu63ZqEbQ7+SpvS84jVsd6MRCJAhH1WqLzpiCbWp4xH 0Q2WuF7DR9Sr3FPu1xDptU1YsMu7IdOGw2P3rTKkkX8mP3xGJHyxmk6J2TjSBfs/e8sv 3/vVPm6xqaxMOx70azmAZP+LhedRUQe5Dn5pob5GDvcNJlVI6qt5iMDV6xfK1kqEgyWR j96A== X-Forwarded-Encrypted: i=1; AJvYcCX8F7ObQgS+6rQ9rIzWg26NH088Gx56I3bQSs2qwAmeB40FlJHJoT0GWU2+5EUABe+ABTN+8vX9IQhePiA=@vger.kernel.org X-Gm-Message-State: AOJu0YywALK/vP2DMijpPqEq4zY+3Z850+xg9FHa4FdD1xV3OVJ8MhhV UpIU3Fj/sPmj3f12LkQwxAbnrj5CE5AWM73cdOv6W2RzguBfTygLYjKfW024qk2WzZ9Pux0dhuN PMAOCSZaTSZuQFnJneaYb0w== X-Google-Smtp-Source: AGHT+IFlawdC4KCQfLhqI0Xy7yk4ED2go+LFfUpiqb3BRrNWwO5I+1bNuk03iJCYZid8NNj/xbpv1Yi7TsuSmuqPKg== X-Received: from almasrymina.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:4bc5]) (user=almasrymina job=sendgmr) by 2002:a63:3755:0:b0:6f3:b24:6c27 with SMTP id 41be03b00d2f7-7d79f6e5f59mr68139a12.5.1725860627260; Sun, 08 Sep 2024 22:43:47 -0700 (PDT) Date: Mon, 9 Sep 2024 05:43:18 +0000 In-Reply-To: <20240909054318.1809580-1-almasrymina@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240909054318.1809580-1-almasrymina@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240909054318.1809580-14-almasrymina@google.com> Subject: [PATCH net-next v25 13/13] netdev: add dmabuf introspection From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , Donald Hunter , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , "=?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?=" , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Shuah Khan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sumit Semwal , "=?UTF-8?q?Christian=20K=C3=B6nig?=" , Pavel Begunkov , David Wei , Jason Gunthorpe , Yunsheng Lin , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , Bagas Sanjaya , Christoph Hellwig , Nikolay Aleksandrov , Taehee Yoo Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add dmabuf information to page_pool stats: $ ./cli.py --spec ../netlink/specs/netdev.yaml --dump page-pool-get ... {'dmabuf': 10, 'id': 456, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, {'dmabuf': 10, 'id': 455, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, {'dmabuf': 10, 'id': 454, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, {'dmabuf': 10, 'id': 453, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, {'dmabuf': 10, 'id': 452, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, {'dmabuf': 10, 'id': 451, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, {'dmabuf': 10, 'id': 450, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, {'dmabuf': 10, 'id': 449, 'ifindex': 3, 'inflight': 1023, 'inflight-mem': 4190208}, And queue stats: $ ./cli.py --spec ../netlink/specs/netdev.yaml --dump queue-get ... {'dmabuf': 10, 'id': 8, 'ifindex': 3, 'type': 'rx'}, {'dmabuf': 10, 'id': 9, 'ifindex': 3, 'type': 'rx'}, {'dmabuf': 10, 'id': 10, 'ifindex': 3, 'type': 'rx'}, {'dmabuf': 10, 'id': 11, 'ifindex': 3, 'type': 'rx'}, {'dmabuf': 10, 'id': 12, 'ifindex': 3, 'type': 'rx'}, {'dmabuf': 10, 'id': 13, 'ifindex': 3, 'type': 'rx'}, {'dmabuf': 10, 'id': 14, 'ifindex': 3, 'type': 'rx'}, {'dmabuf': 10, 'id': 15, 'ifindex': 3, 'type': 'rx'}, Suggested-by: Jakub Kicinski Signed-off-by: Mina Almasry Reviewed-by: Jakub Kicinski reviewed-by's, and seeks to fix the patchwork build error (sorry!). --- v24: - Code cleanup, no cast and use 1 if statement (Jakub) --- Documentation/netlink/specs/netdev.yaml | 10 ++++++++++ include/uapi/linux/netdev.h | 2 ++ net/core/netdev-genl.c | 7 +++++++ net/core/page_pool_user.c | 5 +++++ tools/include/uapi/linux/netdev.h | 2 ++ 5 files changed, 26 insertions(+) diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlin= k/specs/netdev.yaml index 0c747530c275..08412c279297 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -167,6 +167,10 @@ attribute-sets: "re-attached", they are just waiting to disappear. Attribute is absent if Page Pool has not been detached, and can still be used to allocate new memory. + - + name: dmabuf + doc: ID of the dmabuf this page-pool is attached to. + type: u32 - name: page-pool-info subset-of: page-pool @@ -268,6 +272,10 @@ attribute-sets: name: napi-id doc: ID of the NAPI instance which services this queue. type: u32 + - + name: dmabuf + doc: ID of the dmabuf attached to this queue, if any. + type: u32 =20 - name: qstats @@ -543,6 +551,7 @@ operations: - inflight - inflight-mem - detach-time + - dmabuf dump: reply: *pp-reply config-cond: page-pool @@ -607,6 +616,7 @@ operations: - type - napi-id - ifindex + - dmabuf dump: request: attributes: diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h index 91bf3ecc5f1d..7c308f04e7a0 100644 --- a/include/uapi/linux/netdev.h +++ b/include/uapi/linux/netdev.h @@ -93,6 +93,7 @@ enum { NETDEV_A_PAGE_POOL_INFLIGHT, NETDEV_A_PAGE_POOL_INFLIGHT_MEM, NETDEV_A_PAGE_POOL_DETACH_TIME, + NETDEV_A_PAGE_POOL_DMABUF, =20 __NETDEV_A_PAGE_POOL_MAX, NETDEV_A_PAGE_POOL_MAX =3D (__NETDEV_A_PAGE_POOL_MAX - 1) @@ -131,6 +132,7 @@ enum { NETDEV_A_QUEUE_IFINDEX, NETDEV_A_QUEUE_TYPE, NETDEV_A_QUEUE_NAPI_ID, + NETDEV_A_QUEUE_DMABUF, =20 __NETDEV_A_QUEUE_MAX, NETDEV_A_QUEUE_MAX =3D (__NETDEV_A_QUEUE_MAX - 1) diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 9153a8ab0cf8..1cb954f2d39e 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -295,6 +295,7 @@ static int netdev_nl_queue_fill_one(struct sk_buff *rsp, struct net_device *netdev, u32 q_idx, u32 q_type, const struct genl_info *info) { + struct net_devmem_dmabuf_binding *binding; struct netdev_rx_queue *rxq; struct netdev_queue *txq; void *hdr; @@ -314,6 +315,12 @@ netdev_nl_queue_fill_one(struct sk_buff *rsp, struct n= et_device *netdev, if (rxq->napi && nla_put_u32(rsp, NETDEV_A_QUEUE_NAPI_ID, rxq->napi->napi_id)) goto nla_put_failure; + + binding =3D rxq->mp_params.mp_priv; + if (binding && + nla_put_u32(rsp, NETDEV_A_QUEUE_DMABUF, binding->id)) + goto nla_put_failure; + break; case NETDEV_QUEUE_TYPE_TX: txq =3D netdev_get_tx_queue(netdev, q_idx); diff --git a/net/core/page_pool_user.c b/net/core/page_pool_user.c index cd6267ba6fa3..48335766c1bf 100644 --- a/net/core/page_pool_user.c +++ b/net/core/page_pool_user.c @@ -9,6 +9,7 @@ #include #include =20 +#include "devmem.h" #include "page_pool_priv.h" #include "netdev-genl-gen.h" =20 @@ -213,6 +214,7 @@ static int page_pool_nl_fill(struct sk_buff *rsp, const struct page_pool *pool, const struct genl_info *info) { + struct net_devmem_dmabuf_binding *binding =3D pool->mp_priv; size_t inflight, refsz; void *hdr; =20 @@ -242,6 +244,9 @@ page_pool_nl_fill(struct sk_buff *rsp, const struct pag= e_pool *pool, pool->user.detach_time)) goto err_cancel; =20 + if (binding && nla_put_u32(rsp, NETDEV_A_PAGE_POOL_DMABUF, binding->id)) + goto err_cancel; + genlmsg_end(rsp, hdr); =20 return 0; diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/n= etdev.h index 91bf3ecc5f1d..7c308f04e7a0 100644 --- a/tools/include/uapi/linux/netdev.h +++ b/tools/include/uapi/linux/netdev.h @@ -93,6 +93,7 @@ enum { NETDEV_A_PAGE_POOL_INFLIGHT, NETDEV_A_PAGE_POOL_INFLIGHT_MEM, NETDEV_A_PAGE_POOL_DETACH_TIME, + NETDEV_A_PAGE_POOL_DMABUF, =20 __NETDEV_A_PAGE_POOL_MAX, NETDEV_A_PAGE_POOL_MAX =3D (__NETDEV_A_PAGE_POOL_MAX - 1) @@ -131,6 +132,7 @@ enum { NETDEV_A_QUEUE_IFINDEX, NETDEV_A_QUEUE_TYPE, NETDEV_A_QUEUE_NAPI_ID, + NETDEV_A_QUEUE_DMABUF, =20 __NETDEV_A_QUEUE_MAX, NETDEV_A_QUEUE_MAX =3D (__NETDEV_A_QUEUE_MAX - 1) --=20 2.46.0.469.g59c65b2a67-goog