From nobody Fri Jul 24 21:53:01 2026 Received: from out203-205-221-191.mail.qq.com (out203-205-221-191.mail.qq.com [203.205.221.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABF0F36EAB2; Fri, 24 Jul 2026 09:02:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.205.221.191 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784883730; cv=none; b=qaL6pFV6OPR8xllyd7nZU7LHT5j5/K5vJsK1tuxas/2T26HkhXUpfsfsxpmKW9584Y3YAVnjTMIhdl7dwt5STI7+mfnX5y0J0cHkfi3Rw5p4J+AlzRS627PrmLCkwyFmxg8RRngyf++7DeoI+gEocelp33WGikgtdGogHVjzThk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784883730; c=relaxed/simple; bh=FxCLtBwiJVVCdNZejj8faQeOyMki65SyMA0mwYfgPkQ=; h=Message-ID:From:To:Cc:Subject:Date:In-Reply-To:References: MIME-Version; b=KZrgP74oIAMgQmdQGioUEWfhOrHb3yHC3ERNA/o80q5qEHBmWm1cHoUBVDcuPVRkhZG16XB/XXNHU1/X84gRhFDT1DeeE9iehPjSiBJqp48Xh7LLa9NlhWRvc+qCqJlOUowoKXD9+bosjUJ6FU5vzaDqvLLy5mamVG/+9cHuQpg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name; spf=pass smtp.mailfrom=cyyself.name; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b=Kx3D2E/B; arc=none smtp.client-ip=203.205.221.191 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cyyself.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b="Kx3D2E/B" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1784883715; bh=QlOJ6gN12LoffzLDHiimPrvvRRM4vY38LnNZ+7hdgQE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Kx3D2E/Bc/Fke/dqNfW7rH0zBq811CaC+mowLqPpcMXEP6JD6tI8WcAxfBBKBD4F7 6qo58MbPplFymH8pyatlMx1/HvzhxI8WhwNQi0xwHrLErLLzST3X9YzoJb2JTMJvpE DZw4BwbkI/8uZ+ImvNTULAt8iwn1MO75/DNqORvQ= Received: from localhost.localdomain ([240e:37c:2242:cf00:265e:beff:fe6a:4da1]) by newxmesmtplogicsvrszc50-0.qq.com (NewEsmtp) with SMTP id 73252E3; Fri, 24 Jul 2026 17:01:51 +0800 X-QQ-mid: xmsmtpt1784883711toansbuh3 Message-ID: X-QQ-XMAILINFO: NDkKyiSMIrpLusbFu0mBP53bBayE3A8GXSvR2M83GLMlGkNntW9D3Fm7pko9r0 hKzFz/xhEoRDDhLW0JGTClcK8gRvLQEQqTU8WcPGtZ5GyYZHynQHutJyV0rtF8b7naRLyCa3XfOo azWa2wQ0SqTjyaqufI9KllwOVWBH6igIrRhsE5LiCeIFpaxwXPWwyJERhiV8SSXCJuyhs7Kj83yJ 4JYZKUySWI/g13Ls23pL9+aJ8xLPh8GOM6LQXW1qFp55eUv+pQ2LfnX1+vkv1JjkeKvY/1ZySu+C WzHspGZnqms9wK8WBW9467j3Jcpx43a3ldfvlA1RgSmPwQjC1rr+EhkdDJws5qB/IAcWZxLNfY5h kwl4Z5JrUt1ixfBfZmK0hwhgEpXJ2j5+h85nf7wtKEQ+2nqDDoPNr9EBz9xqPZIl9JWP5EmpUVtN NzyFWkEAbF6rQI6brLlurCVvuGL+oR9y/pr5GLotUfakbBsRx52rDsI0FZaI/lgk6IOWd1ZCwP/4 FwRucXfodOKzKWkv0nc7FrFsPG9os0UVBgh2Tixv1HOJr5hunfbwHwUCVmzjtABfQPvfEGj8aBMp GnScdS4ll3OOwq7O63Axu66fUX9Mxt6DUl/lfqS7DmycbFQqeJKKVbfc+JfNZa2NuzLzDzWzlTbP RlI7OurOz92rElClSI/stzmVLTByiMxLU2h6P+dR6284Y5mtxISCBc4+dnAcmNUnD0kXnfhKXB4H 8Dej+mVdWxK+2+TJ4Yzwx8oAj+1lzttLNL4xIDYHYJkf0HLO2+/SrLx54qTnjU0xMYRyIXhtc39E zcXFWYjQjLRfUrNhkJ5Ckb4AAaAGYIsG3f0llimyEoakd9eYKo67vg+uU7GxtrVpZpdIKNFTOUzi hBpIYKpxWEgeCbAuFlhDUliSRWaEW8BKlizYpgodIjOKOtOGm/m+4g+mraZ6DqIe34zy7dGtgXKH 5wuXwwnzSgs7+dQlHHbxEApWIQvuFXZ3I5ow7wNQnrrDD5YezJkX/laiBFsRjsToBm6OX9TALPzt RYf9WYvV432MBltn2C3LbkpanE1v65l1plR78ZElg4YkV5iPQ7pAyx2rSwzfelTpJn6uvPQljkeN ZyxFF2wFa4q2K25B6Shlv7JTZq+9hgne7Q7ssu8gYsJihwELur54bTyePqjhf2gntLVKav X-QQ-XMRINFO: NI4Ajvh11aEjEMj13RCX7UuhPEoou2bs1g== From: Yangyu Chen To: Sukhdeep Singh , Andrew Lunn , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Mina Almasry , Jesper Dangaard Brouer , Richard Cochran , Lino Sanfilippo , Igor Russkikh , Simon Horman , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Yangyu Chen Subject: [PATCH net-next v2 1/3] net: atlantic: free stranded TX buffers on ring deinit Date: Fri, 24 Jul 2026 17:01:50 +0800 X-OQ-MSGID: <20260724090150.10505-1-cyy@cyyself.name> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" aq_vec_deinit() drains the TX rings with a single aq_ring_tx_clean() call, which frees at most AQ_CFG_TX_CLEAN_BUDGET (256) descriptors and stops at hw_head, which no longer moves once aq_vec_stop() has stopped the hardware and NAPI. Completed descriptors beyond the budget and everything still posted in [hw_head, sw_tail) keep their skb or xdp_frame when the interface goes down: aq_vec_ring_free() then frees the buffer ring and the references are lost for good. Today this is a silent memory leak on every interface down under TX/XDP_TX load. With the following conversion of the RX path to page_pool it becomes much more visible: XDP_TX frames carry fragment references on the RX ring's page_pool, so a single stranded frame keeps the pool's inflight count above zero forever. page_pool_destroy() then never completes, the pool is leaked together with its pages, and "page_pool_release_retry() stalled pool shutdown" is warned every 60 seconds from that point on, on every ifdown, XDP detach or ring resize under XDP_TX load. Bring back aq_ring_tx_deinit() as it was before the removal and use it for teardown again, with one extension: TX rings can hold xdp_frames nowadays, so release those too. They are returned with xdp_return_frame() since this runs in process context. Fixes: eb36bedf28be ("net: aquantia: remove function aq_ring_tx_deinit") Cc: stable@vger.kernel.org # v4.11+ Assisted-by: Claude:claude-fable-5 Signed-off-by: Yangyu Chen --- Notes: Without this fix, converting the RX path to page_pool (last patch of this series) turns the stranded XDP_TX frames into leaked page_pool fragments, so page_pool_destroy() can never drain and the shutdown stalls forever. =20 Reproduced on an AQC100 with this patch dropped from the series (i.e. page_pool applied without the tx-deinit fix): =20 # reflect received frames back out through XDP_TX xdp-bench tx enp99s0 # or any trivial XDP_TX prog # from a peer on the same link, flood RX so frames are in flight, then ip link set enp99s0 down =20 The pool is destroyed with frames still stranded on the TX ring, and page_pool_release_retry() warns every 60s with the same id and inflight count and a growing age, indefinitely: =20 [161110.753385] page_pool_release_retry() stalled pool shutdown: id 3= 61, 12 inflight 60 sec [161171.170756] page_pool_release_retry() stalled pool shutdown: id 3= 61, 12 inflight 120 sec [161231.588685] page_pool_release_retry() stalled pool shutdown: id 3= 61, 12 inflight 181 sec [161292.005886] page_pool_release_retry() stalled pool shutdown: id 3= 61, 12 inflight 241 sec =20 With this patch the stranded buffers are freed at deinit, inflight drops to zero and the pool drains cleanly. =20 v1 -> v2: - return early instead of goto to an end-of-function label (Mina) - drop the likely()/unlikely() annotations, this is a reconfig path (Mina) - add Cc: stable with the affected range .../net/ethernet/aquantia/atlantic/aq_ring.c | 29 +++++++++++++++++++ .../net/ethernet/aquantia/atlantic/aq_ring.h | 1 + .../net/ethernet/aquantia/atlantic/aq_vec.c | 2 +- 3 files changed, 31 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c b/drivers/net= /ethernet/aquantia/atlantic/aq_ring.c index 8ff07de2bd52..81685a4dc5a6 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c @@ -360,6 +360,35 @@ bool aq_ring_tx_clean(struct aq_ring_s *self) return !!budget; } =20 +void aq_ring_tx_deinit(struct aq_ring_s *self) +{ + if (!self) + return; + + for (; self->sw_head !=3D self->sw_tail; + self->sw_head =3D aq_ring_next_dx(self, self->sw_head)) { + struct aq_ring_buff_s *buff =3D &self->buff_ring[self->sw_head]; + struct device *ndev =3D aq_nic_get_dev(self->aq_nic); + + if (buff->is_mapped) { + if (buff->is_sop) { + dma_unmap_single(ndev, buff->pa, buff->len, + DMA_TO_DEVICE); + } else { + dma_unmap_page(ndev, buff->pa, buff->len, + DMA_TO_DEVICE); + } + } + + if (buff->is_eop) { + if (buff->skb) + dev_kfree_skb_any(buff->skb); + else if (buff->xdpf) + xdp_return_frame(buff->xdpf); + } + } +} + static void aq_rx_checksum(struct aq_ring_s *self, struct aq_ring_buff_s *buff, struct sk_buff *skb) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h b/drivers/net= /ethernet/aquantia/atlantic/aq_ring.h index a70b880ada67..6431cc62962f 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h @@ -202,6 +202,7 @@ void aq_ring_update_queue_state(struct aq_ring_s *ring); void aq_ring_queue_wake(struct aq_ring_s *ring); void aq_ring_queue_stop(struct aq_ring_s *ring); bool aq_ring_tx_clean(struct aq_ring_s *self); +void aq_ring_tx_deinit(struct aq_ring_s *self); int aq_xdp_xmit(struct net_device *dev, int num_frames, struct xdp_frame **frames, u32 flags); int aq_ring_rx_clean(struct aq_ring_s *self, diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c b/drivers/net/= ethernet/aquantia/atlantic/aq_vec.c index 2f9033ceed8c..05814fea0f5f 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_vec.c @@ -275,7 +275,7 @@ void aq_vec_deinit(struct aq_vec_s *self) =20 for (i =3D 0U; self->tx_rings > i; ++i) { ring =3D self->ring[i]; - aq_ring_tx_clean(&ring[AQ_VEC_TX_ID]); + aq_ring_tx_deinit(&ring[AQ_VEC_TX_ID]); aq_ring_rx_deinit(&ring[AQ_VEC_RX_ID]); } =20 --=20 2.47.3 From nobody Fri Jul 24 21:53:01 2026 Received: from out203-205-221-210.mail.qq.com (out203-205-221-210.mail.qq.com [203.205.221.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85FAF332913; Fri, 24 Jul 2026 09:02:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.205.221.210 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784883747; cv=none; b=FUvh/pBWy97cOA5r+PQR6zavYC/TEIXMoEFIi2JX6D+lXGAwsPXEOIAP74XIG8i080rxPF1pjwdhpa2Timu/EcSRSvAUfH9qLnYSEo5eDU7stE2NA+XfsqgYsryrJ15T8dSUNmaQjf8AuqhbTvJHbRkb0HcMjYL6sG+arGAjWD8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784883747; c=relaxed/simple; bh=EvtcL2NUiEJOTUWUDoNmjN8xHYQ/OKx/SJ93wkS2qAY=; h=Message-ID:From:To:Cc:Subject:Date:In-Reply-To:References: MIME-Version; b=LS2kPyqEXeYoDZKg975tzkj0KWR0FMWOS54H4UIIHhefpCjtJ5sLk7yiB94bK9PGYo4dtzG/YdNFvkLgIN4O7aG9N4Ctc+7fWT+GJHo21RU/VfO92CEwLb09bq9ojDdXFZKMTaKBXH5GZJlUAbIh6KgpEavrhP4Fyy9M4XvLH5g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name; spf=pass smtp.mailfrom=cyyself.name; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b=f5xtTtQm; arc=none smtp.client-ip=203.205.221.210 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cyyself.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b="f5xtTtQm" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1784883734; bh=QyFlgoeYRjeI4DQuiSC8DugoBorAfJKSqJNj4EACvo4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=f5xtTtQmX+vDRNR9h7fLDo1hLJjqTQA6FywNlLENXCxXih5+cPXfKtFS0To8Nfw/K 6xGG7WvLTejVDs2KixTBpOU12DUqkebMYYGgzp+exI6FKOH6GwKQ6FYElm1ZmqyBVk l5TpvYzW3pxiDses+BhteXJ7Kzme4Z94GsHe90yI= Received: from localhost.localdomain ([240e:37c:2242:cf00:265e:beff:fe6a:4da1]) by newxmesmtplogicsvrsza53-0.qq.com (NewEsmtp) with SMTP id 89B18AE; Fri, 24 Jul 2026 17:02:09 +0800 X-QQ-mid: xmsmtpt1784883729txebgdhma Message-ID: X-QQ-XMAILINFO: NR25d0ihy8j11ORAqCNTWcE1G8ujvGKpWwYBDZ7qk/ajgZRzlvdwwnU3FcopPC TfYeTxqwOJLZNxbcTFl0uQ/5ZbsYrQ+E634088YNdcc37oGKGYqtK3HX9dZikGmUVqqInysS2pmP EodK0XQXZ6KfQRfKqvqsJ7xMKRToMaorWzehwW5QLrmjIQzN55EYXm5QDHY5Om9W/tRISdWQCMmC ydIQxVtaVfroVGuxv/1nJuh6paQ5ajj0nGrlWC3S9p3yzzuABL8BiFaEzoy3kFWQWHWWVqYbTDmt RvEjftp06MDpG7KhTBx9qpVisvib+NO5x5Ty7+aobfTi6A1zha1z1rp4Y6WzBwjIduR+GiXl9fVh PHu4yHYtDbH8idTFwDSh7hLLASdZm4NZgSMhmDGPeRYC7o2jXCUgs9kbUGcBWw8aw5f1Asp96CUh AunVkv5Sn33IwCYXiCM6L5xZTKR+Dj0GcFn63gznuAPY1NpSnOQ6bXeNa2b3KSw5HgmwcAOm9xGn RwUQBMlfbxqqaMaJCY7cIVYQCOtpaxSUgM0xCoXH94tStqm9mkNBghVlNxcQQp7l66ujoBPJXzIB kw4H8FVDekT319ZpqNdrtQ9DO/cSoY7Wg3nQsH6u+nswqFJzlswucNAfBr2w1sS7l7uzwBWPNKU9 Z60XtAJ0OU3of9UM4Dz9I0QZDWoA16SRZYX2ORbtCijJy4KOe+ARgW0zxftk+LptVknDj2/ZetDx mwVA0Wzk8NMh4wI3pj3g/0f2FjGYOjYImJtzO67DfjphLTaY1IhSZoUfOgcIaepk/5WbpyNXUqVH 6XNpZK2icgse9qvMZ427JucfNkF/RWEjbwOONp8oRQh0+avKuoCmuuWTau+D6wy+TOxTD9craxss udLeB1JvRaonBJsMOeaef3vHIAFcxhKxj2aZSuzEV5PntRjTan/vYUxoeMA0igRFw0eEBdNLodud U7DlHLfNOE+BR7MJJxCPRKx0kIZMXFVNtsq/Ww0VAexTw29Zm1U4b3PLUnOIbZjTeQKS5MqTDlRm BbYg7gz1cGj2DuBL8h7ZwO498Zf7hBUxXtZ60kPl6zH/1xdMveYJrV0tm+0dkae77musPhA5EE2a 4tHUpLg0DBslKOxoOn+RYlykslkpO1Pk9O2bkn X-QQ-XMRINFO: MSVp+SPm3vtSI1QTLgDHQqIV1w2oNKDqfg== From: Yangyu Chen To: Sukhdeep Singh , Andrew Lunn , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Mina Almasry , Jesper Dangaard Brouer , Richard Cochran , Lino Sanfilippo , Igor Russkikh , Simon Horman , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Yangyu Chen Subject: [PATCH net-next v2 2/3] net: atlantic: free RX pages of consumed but not refilled buffers Date: Fri, 24 Jul 2026 17:02:08 +0800 X-OQ-MSGID: <20260724090208.10556-1-cyy@cyyself.name> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" aq_ring_rx_deinit() only walks [sw_head, sw_tail), the region posted to hardware. Since the page reuse strategy was added, a cleaned RX buffer keeps its page (and its DMA mapping) in the ring for reuse, and refill is batched: aq_ring_rx_fill() returns early until AQ_CFG_RX_REFILL_THRES slots are free. Slots that were consumed but not yet reposted therefore sit in the complementary [sw_tail, sw_head) gap with a live page, and the deinit walk never visits them: up to a refill batch worth of pages and DMA mappings leak on every interface down. Walk the whole ring instead and release whatever is still there. Also bail out if the buffer ring is already gone: a partial aq_ptp_ring_alloc() failure frees the ring but leaves aq_nic set, so aq_ptp_ring_deinit() still gets here on the unwind path. Fixes: 46f4c29d9de6 ("net: aquantia: optimize rx performance by page reuse = strategy") Cc: stable@vger.kernel.org # v5.2+ Assisted-by: Claude:claude-fable-5 Signed-off-by: Yangyu Chen --- Notes: Without this fix, the following page_pool conversion turns the missed pages into fragments that page_pool_destroy() waits for forever. Reproduced on an AQC100 with the conversion applied and this fix reverted -- ordinary small received frames (<=3D 256 byte header-only packets, e.g. ping replies or pure TCP ACKs) are enough to populate the [sw_tail, sw_head) gap: =20 ping -c 200 -i 0.005 %enp99s0 ip link set enp99s0 down =20 One short ping flow left three of the eight RX rings' pools with stranded fragments, and page_pool_release_retry() warns for each of them every 60 seconds, indefinitely: =20 [278084.929092] page_pool_release_retry() stalled pool shutdown: id 1= 23, 1 inflight 60 sec [278084.961064] page_pool_release_retry() stalled pool shutdown: id 1= 26, 1 inflight 60 sec [278084.961087] page_pool_release_retry() stalled pool shutdown: id 1= 25, 6 inflight 60 sec [278145.346737] page_pool_release_retry() stalled pool shutdown: id 1= 23, 1 inflight 120 sec [278145.378745] page_pool_release_retry() stalled pool shutdown: id 1= 25, 6 inflight 120 sec [278145.378759] page_pool_release_retry() stalled pool shutdown: id 1= 26, 1 inflight 120 sec =20 With this patch the whole ring is walked at deinit, the pages are released, and the pools drain immediately. On the current code the same gap leaks the pages and their DMA mappings silently. =20 Applies and was build- and runtime-tested independently of the rest of this series, against the current page scheme. =20 New in v2: split out of the page_pool conversion and made a standalone fix for the existing page reuse scheme (Mina); Cc stable with the affected range. .../net/ethernet/aquantia/atlantic/aq_ring.c | 22 +++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c b/drivers/net= /ethernet/aquantia/atlantic/aq_ring.c index 81685a4dc5a6..e1193c6719d9 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c @@ -950,15 +950,29 @@ int aq_ring_rx_fill(struct aq_ring_s *self) =20 void aq_ring_rx_deinit(struct aq_ring_s *self) { - if (!self) + unsigned int i; + + if (!self || !self->buff_ring) return; =20 - for (; self->sw_head !=3D self->sw_tail; - self->sw_head =3D aq_ring_next_dx(self, self->sw_head)) { - struct aq_ring_buff_s *buff =3D &self->buff_ring[self->sw_head]; + /* Release every page still owned by the ring. + * + * Walking [sw_head, sw_tail) is not enough: refill is batched + * (aq_ring_rx_fill() waits for AQ_CFG_RX_REFILL_THRES free slots), + * so slots that were cleaned but not yet reposted accumulate in the + * [sw_tail, sw_head) gap, and they keep their page for reuse. Walk + * the whole ring and release whatever is left. + */ + for (i =3D 0; i < self->size; i++) { + struct aq_ring_buff_s *buff =3D &self->buff_ring[i]; + + if (!buff->rxdata.page) + continue; =20 aq_free_rxpage(&buff->rxdata, aq_nic_get_dev(self->aq_nic)); } + + self->sw_head =3D self->sw_tail; } =20 void aq_ring_free(struct aq_ring_s *self) --=20 2.47.3 From nobody Fri Jul 24 21:53:01 2026 Received: from out162-62-57-210.mail.qq.com (out162-62-57-210.mail.qq.com [162.62.57.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 449D83EB0EC; Fri, 24 Jul 2026 09:03:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=162.62.57.210 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784883830; cv=none; b=LA05sOgf0ctRGYwO7k4gX5t6ZHQ5oO9OIYsTJZgnN33k2ynLDw+jCV1o/p2aumecEWnxclDTMQ+h84kHWmILURD/P+WaWCEj0n+A+G+a5xx5VlczwEMNiN9wpehSklDnI8VjVUmztR7HwN0jFoVBnHijBgnxc5W8DM3TxZBoFIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784883830; c=relaxed/simple; bh=sX5ZknnC8DOzh74QXm4D4I48/9s13BCU5Io/oIcw+mU=; h=Message-ID:From:To:Cc:Subject:Date:In-Reply-To:References: MIME-Version; b=vGcD5RBfmJCBJALWqchONfKt7IpRKQZRfNGuiBJZxE3i2see5et7EhVAis6GERE6VXbMv+T4pnoN22OetB1NH0iRhpgUb07ZrutZUYY3tToEPpo6ZKlr9ZscaOhzmKwasajFPXyrtJjjUpOz8mr4PkUYJFPxv2xll6d71AZW8bs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name; spf=pass smtp.mailfrom=cyyself.name; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b=N/KJXHxH; arc=none smtp.client-ip=162.62.57.210 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=cyyself.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cyyself.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=qq.com header.i=@qq.com header.b="N/KJXHxH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1784883814; bh=ZwyGq5cRmq/NMsKR4eDuQZQnKH2G4PZnh5PLXATSmVw=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=N/KJXHxH5rQuqHhcBfgpulJ+dn/v2MHL0bA0pHCO0AIOtdc/mD3ToxJH5IOBMQTHu Gli3KhqEo/Vg2RSfJjtkd37SerIC+Hb+XrfMAFRCZt9ZtL+lMuTY+TSnFN+Ki4J8/C bOQGR/GNozMVMtIVDmyGeHjvFd3ngsE5tpJ38gdU= Received: from localhost.localdomain ([240e:37c:2242:cf00:265e:beff:fe6a:4da1]) by newxmesmtplogicsvrszc56-0.qq.com (NewEsmtp) with SMTP id 95BD6BB; Fri, 24 Jul 2026 17:02:21 +0800 X-QQ-mid: xmsmtpt1784883741t3cxpnb1l Message-ID: X-QQ-XMAILINFO: MllZffuBkEb5huDOy6SpieBhngfTA7TahADWJYpllmwJCFniaBJFy6sh5k7dF2 lT/oHxO/9AqSR/d5KlU9GfpREAUfuC5w1nS6NX1eX2U5AxzPY0OdX07jT+i9wJ1f0wCmIT0wetmn 0a/Xguo0LQ9y3BXBsFrbO/hWy4BDvGBd5KnqpLb1AuGvEVioboSI0mvIpdRvHpLlIWy1oUt/xamP Zp16Qa7kDepTzkms0BtlX4I6LmX4thjjcQ8lNrGo8/jfsiex+0+KfJ7ugVxunGjcCt317Iiq72uI QhjyV0sKtGaofQViS9ElqVTXJoiETG0AlBHjCGDLb6MhPlSR5mhdoEqFj37JvE7uUrT/pf55Tw7R ZIXMXtwJVJ1iRoki+UmwbXDgvjpvo80wBPf+vsw6NcGYIlAwd/C9t+chjcZyiqKsW6ODu/xbRTpG PBBhs+T+nHKiJuRG3J8xSM7mE2OINsRSDeb6xCQYRsy3+8H25nAkMlw0I7icfaRTXT62nRkhuQVA GU3yY1wv7KMnSCCO2D4Si8a8MQVJN3U81oVH1Xux16bVwOKAvK36BMrhz9nJ85VN9myRpiHKhcap 5p8H1Na6xSGdBn/AOSmTwxDJKXWpc3IWPbIDCK0h6M9YaRwpFf3hlBtls+UZDu2ZfD3X3aWB7zbq rBNrbamh9G5ON/ZS/1QE5IHBgg1fKrZ8o4UH/N+rN1c3fjpjAmVdwoxAEo8pKjs4d4DhvqcHmRVp ooQriIvNlWe7VWgJ0zmLB+v//PrqqpQPAgav4pD30p2cNa49OkrzVat9iQ//Y5GKTb8/Guo+yYHM wjCj8khT94RhgQE2ObF/+L2KYPygwcq6qrLZTqFw2DY9VWEi3L3SqNFhhrVSApCfr6O5Twx8RJyM TWrl1ajDNyK0ixLAjn9EkpJx5w9y3H2yRyuE415n3O9ABaOKDxmC+Cl65TCz3C4wNkfMLcyPz9d4 ncLWogsxnGGEPcJqTZXetLl2+HChTDi8EWmxMIMHgUIEYusRYDL4aZRwlZBHkucOO2ohLnfFtF25 WEoF4B5D6MhLzbniiclR4UgU9i5DXQBhqvAm11Mp9U1j99fNJwDgbDkiJQSMB8TCrs1hDEx0PV8d gEAT5jjD5RPk0PtQG2EyblZ7ULpJAFZTp6uN9T X-QQ-XMRINFO: NI4Ajvh11aEjEMj13RCX7UuhPEoou2bs1g== From: Yangyu Chen To: Sukhdeep Singh , Andrew Lunn , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Mina Almasry , Jesper Dangaard Brouer , Richard Cochran , Lino Sanfilippo , Igor Russkikh , Simon Horman , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Yangyu Chen Subject: [PATCH net-next v2 3/3] net: atlantic: convert RX path to page_pool Date: Fri, 24 Jul 2026 17:02:20 +0800 X-OQ-MSGID: <20260724090220.10602-1-cyy@cyyself.name> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The driver currently allocates RX buffers with dev_alloc_pages(), maps them with dma_map_page(), and uses a hand-rolled page-flip scheme to subdivide high-order pages. Behind an IOMMU, the map/unmap churn is a major RX cost: on a Thunderbolt-attached QNAP QNA-T310G1S, iperf3 -R over IPv6 tops out at about 2.2 Gbit/s over MTU 1500. Convert RX buffers to page_pool fragments. Pages are DMA-mapped once when entering the pool and recycled through the stack or XDP via the MEM_TYPE_PAGE_POOL memory model. This removes the custom page-flip accounting, lets page_pool handle fragment reuse, and ensures every RX path either keeps the ring's fragment reference for reposting or hands it to the skb/xdp_buff for later recycling. Register the PTP RX ring's xdp_rxq as well, since it shares the RX clean paths. Drop the ethtool PageFlips/PageReuses/PageFrees counters which only described the old scheme. On the QNA-T310G1S, MTU 1500, TCP over IPv6, iperf3 -R improves from 2.24 Gbit/s to 9.14 Gbit/s. The module was also smoke-tested with native XDP PASS, DROP, and ABORTED actions; carrier recovered after each attach/detach cycle and dmesg showed no page_pool/DMA warnings. Assisted-by: Claude:claude-fable-5 Signed-off-by: Yangyu Chen --- Notes: v1 -> v2: - split the RX deinit gap fix into the previous patch and generate this diff with --histogram so it reads as whole-function removals plus an in-place aq_get_rxpages() rewrite (Mina) - comment the ring-before-rxq-registration ordering and its error unwind in aq_vec_ring_alloc() - use the err_exit_xdp_rxq label instead of open-coding the unregister in aq_ptp_ring_alloc()'s memory model error path (Mina) drivers/net/ethernet/aquantia/Kconfig | 1 + .../ethernet/aquantia/atlantic/aq_ethtool.c | 3 - .../net/ethernet/aquantia/atlantic/aq_ptp.c | 18 +- .../net/ethernet/aquantia/atlantic/aq_ring.c | 210 +++++++----------- .../net/ethernet/aquantia/atlantic/aq_ring.h | 6 +- .../net/ethernet/aquantia/atlantic/aq_vec.c | 21 +- 6 files changed, 120 insertions(+), 139 deletions(-) diff --git a/drivers/net/ethernet/aquantia/Kconfig b/drivers/net/ethernet/a= quantia/Kconfig index cec2018c84a9..c8fb7b33e5b7 100644 --- a/drivers/net/ethernet/aquantia/Kconfig +++ b/drivers/net/ethernet/aquantia/Kconfig @@ -20,6 +20,7 @@ config AQTION tristate "aQuantia AQtion(tm) Support" depends on PCI depends on MACSEC || MACSEC=3Dn + select PAGE_POOL help This enables the support for the aQuantia AQtion(tm) Ethernet card. =20 diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c b/drivers/= net/ethernet/aquantia/atlantic/aq_ethtool.c index 420af958d486..0f5125bd2315 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c @@ -100,9 +100,6 @@ static const char * const aq_ethtool_queue_rx_stat_name= s[] =3D { "%sQueue[%d] AllocFails", "%sQueue[%d] SkbAllocFails", "%sQueue[%d] Polls", - "%sQueue[%d] PageFlips", - "%sQueue[%d] PageReuses", - "%sQueue[%d] PageFrees", "%sQueue[%d] XdpAbort", "%sQueue[%d] XdpDrop", "%sQueue[%d] XdpPass", diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ptp.c b/drivers/net/= ethernet/aquantia/atlantic/aq_ptp.c index 558ac9237f75..3a40d986cd67 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ptp.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ptp.c @@ -13,6 +13,7 @@ #include #include #include +#include =20 #include "aq_nic.h" #include "aq_ptp.h" @@ -1192,12 +1193,23 @@ int aq_ptp_ring_alloc(struct aq_nic_s *aq_nic) if (err) goto err_exit_ptp_tx; =20 + err =3D xdp_rxq_info_reg(&aq_ptp->ptp_rx.xdp_rxq, aq_nic->ndev, + rx_ring_idx, aq_ptp->napi.napi_id); + if (err < 0) + goto err_exit_ptp_rx; + + err =3D xdp_rxq_info_reg_mem_model(&aq_ptp->ptp_rx.xdp_rxq, + MEM_TYPE_PAGE_POOL, + aq_ptp->ptp_rx.pg_pool); + if (err < 0) + goto err_exit_xdp_rxq; + if (aq_ptp->a1_ptp) { err =3D aq_ring_hwts_rx_alloc(&aq_ptp->hwts_rx, aq_nic, PTP_HWST_RING_ID= X, aq_nic->aq_nic_cfg.rxds, aq_nic->aq_nic_cfg.aq_hw_caps->rxd_size); if (err) - goto err_exit_ptp_rx; + goto err_exit_xdp_rxq; } =20 err =3D aq_ptp_skb_ring_init(&aq_ptp->skb_ring, aq_nic->aq_nic_cfg.rxds); @@ -1217,6 +1229,8 @@ int aq_ptp_ring_alloc(struct aq_nic_s *aq_nic) err_exit_hwts_rx: if (aq_ptp->a1_ptp) aq_ring_hwts_rx_free(&aq_ptp->hwts_rx); +err_exit_xdp_rxq: + xdp_rxq_info_unreg(&aq_ptp->ptp_rx.xdp_rxq); err_exit_ptp_rx: aq_ring_free(&aq_ptp->ptp_rx); err_exit_ptp_tx: @@ -1233,6 +1247,8 @@ void aq_ptp_ring_free(struct aq_nic_s *aq_nic) return; =20 aq_ring_free(&aq_ptp->ptp_tx); + if (xdp_rxq_info_is_reg(&aq_ptp->ptp_rx.xdp_rxq)) + xdp_rxq_info_unreg(&aq_ptp->ptp_rx.xdp_rxq); aq_ring_free(&aq_ptp->ptp_rx); if (aq_ptp->a1_ptp) aq_ring_hwts_rx_free(&aq_ptp->hwts_rx); diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c b/drivers/net= /ethernet/aquantia/atlantic/aq_ring.c index e1193c6719d9..9dd881710594 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c @@ -14,120 +14,37 @@ #include "aq_vec.h" #include "aq_main.h" =20 +#include #include #include #include #include #include =20 -static void aq_get_rxpages_xdp(struct aq_ring_buff_s *buff, - struct xdp_buff *xdp) -{ - struct skb_shared_info *sinfo; - int i; - - if (xdp_buff_has_frags(xdp)) { - sinfo =3D xdp_get_shared_info_from_buff(xdp); - - for (i =3D 0; i < sinfo->nr_frags; i++) { - skb_frag_t *frag =3D &sinfo->frags[i]; - - page_ref_inc(skb_frag_page(frag)); - } - } - page_ref_inc(buff->rxdata.page); -} - -static inline void aq_free_rxpage(struct aq_rxpage *rxpage, struct device = *dev) -{ - unsigned int len =3D PAGE_SIZE << rxpage->order; - - dma_unmap_page(dev, rxpage->daddr, len, DMA_FROM_DEVICE); - - /* Drop the ref for being in the ring. */ - __free_pages(rxpage->page, rxpage->order); - rxpage->page =3D NULL; -} - -static int aq_alloc_rxpages(struct aq_rxpage *rxpage, struct aq_ring_s *rx= _ring) -{ - struct device *dev =3D aq_nic_get_dev(rx_ring->aq_nic); - unsigned int order =3D rx_ring->page_order; - struct page *page; - int ret =3D -ENOMEM; - dma_addr_t daddr; - - page =3D dev_alloc_pages(order); - if (unlikely(!page)) - goto err_exit; - - daddr =3D dma_map_page(dev, page, 0, PAGE_SIZE << order, - DMA_FROM_DEVICE); - - if (unlikely(dma_mapping_error(dev, daddr))) - goto free_page; - - rxpage->page =3D page; - rxpage->daddr =3D daddr; - rxpage->order =3D order; - rxpage->pg_off =3D rx_ring->page_offset; - - return 0; - -free_page: - __free_pages(page, order); - -err_exit: - return ret; -} - static int aq_get_rxpages(struct aq_ring_s *self, struct aq_ring_buff_s *r= xbuf) { - unsigned int order =3D self->page_order; - u16 page_offset =3D self->page_offset; - u16 frame_max =3D self->frame_max; - u16 tail_size =3D self->tail_size; - int ret; + unsigned int size =3D self->page_offset + self->frame_max + + self->tail_size; + unsigned int pg_off; + struct page *page; =20 - if (rxbuf->rxdata.page) { - /* One means ring is the only user and can reuse */ - if (page_ref_count(rxbuf->rxdata.page) > 1) { - /* Try reuse buffer */ - rxbuf->rxdata.pg_off +=3D frame_max + page_offset + - tail_size; - if (rxbuf->rxdata.pg_off + frame_max + tail_size <=3D - (PAGE_SIZE << order)) { - u64_stats_update_begin(&self->stats.rx.syncp); - self->stats.rx.pg_flips++; - u64_stats_update_end(&self->stats.rx.syncp); + /* Buffers whose page was not passed up the stack are reposted + * with the data they already carry discarded. + */ + if (rxbuf->rxdata.page) + return 0; =20 - } else { - /* Buffer exhausted. We have other users and - * should release this page and realloc - */ - aq_free_rxpage(&rxbuf->rxdata, - aq_nic_get_dev(self->aq_nic)); - u64_stats_update_begin(&self->stats.rx.syncp); - self->stats.rx.pg_losts++; - u64_stats_update_end(&self->stats.rx.syncp); - } - } else { - rxbuf->rxdata.pg_off =3D page_offset; - u64_stats_update_begin(&self->stats.rx.syncp); - self->stats.rx.pg_reuses++; - u64_stats_update_end(&self->stats.rx.syncp); - } + page =3D page_pool_dev_alloc_frag(self->pg_pool, &pg_off, size); + if (unlikely(!page)) { + u64_stats_update_begin(&self->stats.rx.syncp); + self->stats.rx.alloc_fails++; + u64_stats_update_end(&self->stats.rx.syncp); + return -ENOMEM; } =20 - if (!rxbuf->rxdata.page) { - ret =3D aq_alloc_rxpages(&rxbuf->rxdata, self); - if (ret) { - u64_stats_update_begin(&self->stats.rx.syncp); - self->stats.rx.alloc_fails++; - u64_stats_update_end(&self->stats.rx.syncp); - } - return ret; - } + rxbuf->rxdata.page =3D page; + rxbuf->rxdata.daddr =3D page_pool_get_dma_addr(page); + rxbuf->rxdata.pg_off =3D pg_off + self->page_offset; =20 return 0; } @@ -179,6 +96,15 @@ int aq_ring_rx_alloc(struct aq_ring_s *self, unsigned int idx, struct aq_nic_cfg_s *aq_nic_cfg) { + struct page_pool_params pp_params =3D { + .flags =3D PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV, + .pool_size =3D aq_nic_cfg->rxds, + .nid =3D NUMA_NO_NODE, + .dev =3D aq_nic_get_dev(aq_nic), + .dma_dir =3D DMA_FROM_DEVICE, + }; + struct page_pool *pool; + self->aq_nic =3D aq_nic; self->idx =3D idx; self->size =3D aq_nic_cfg->rxds; @@ -200,6 +126,18 @@ int aq_ring_rx_alloc(struct aq_ring_s *self, self->tail_size =3D 0; } =20 + pp_params.order =3D self->page_order; + pp_params.max_len =3D PAGE_SIZE << self->page_order; + + pool =3D page_pool_create(&pp_params); + if (IS_ERR(pool)) + return PTR_ERR(pool); + + self->pg_pool =3D pool; + + /* On failure aq_ring_alloc() calls aq_ring_free(), which also + * destroys the page pool. + */ return aq_ring_alloc(self, aq_nic); } =20 @@ -346,7 +284,11 @@ bool aq_ring_tx_clean(struct aq_ring_s *self) ++self->stats.tx.packets; self->stats.tx.bytes +=3D xdp_get_frame_len(buff->xdpf); u64_stats_update_end(&self->stats.tx.syncp); - xdp_return_frame_rx_napi(buff->xdpf); + /* Frames queued via ndo_xdp_xmit() may come from a + * page pool owned by another NAPI context: no direct + * recycling. + */ + xdp_return_frame(buff->xdpf); } =20 out: @@ -437,22 +379,15 @@ int aq_xdp_xmit(struct net_device *dev, int num_frame= s, } =20 static struct sk_buff *aq_xdp_build_skb(struct xdp_buff *xdp, - struct net_device *dev, - struct aq_ring_buff_s *buff) + struct net_device *dev) { struct xdp_frame *xdpf; - struct sk_buff *skb; =20 xdpf =3D xdp_convert_buff_to_frame(xdp); if (unlikely(!xdpf)) return NULL; =20 - skb =3D xdp_build_skb_from_frame(xdpf, dev); - if (!skb) - return NULL; - - aq_get_rxpages_xdp(buff, xdp); - return skb; + return xdp_build_skb_from_frame(xdpf, dev); } =20 static struct sk_buff *aq_xdp_run_prog(struct aq_nic_s *aq_nic, @@ -473,8 +408,16 @@ static struct sk_buff *aq_xdp_run_prog(struct aq_nic_s= *aq_nic, u64_stats_update_end(&rx_ring->stats.rx.syncp); =20 prog =3D READ_ONCE(rx_ring->xdp_prog); - if (!prog) - return aq_xdp_build_skb(xdp, aq_nic->ndev, buff); + if (!prog) { + skb =3D aq_xdp_build_skb(xdp, aq_nic->ndev); + /* The ring has already handed its page pool reference to the + * xdp_buff, so if the skb could not be built the buffer must + * be returned to the pool here or its fragments would leak. + */ + if (!skb) + xdp_return_buff(xdp); + return skb; + } =20 prefetchw(xdp->data_hard_start); /* xdp_frame write */ =20 @@ -485,7 +428,7 @@ static struct sk_buff *aq_xdp_run_prog(struct aq_nic_s = *aq_nic, act =3D bpf_prog_run_xdp(prog, xdp); switch (act) { case XDP_PASS: - skb =3D aq_xdp_build_skb(xdp, aq_nic->ndev, buff); + skb =3D aq_xdp_build_skb(xdp, aq_nic->ndev); if (!skb) goto out_aborted; u64_stats_update_begin(&rx_ring->stats.rx.syncp); @@ -503,7 +446,6 @@ static struct sk_buff *aq_xdp_run_prog(struct aq_nic_s = *aq_nic, u64_stats_update_begin(&rx_ring->stats.rx.syncp); ++rx_ring->stats.rx.xdp_tx; u64_stats_update_end(&rx_ring->stats.rx.syncp); - aq_get_rxpages_xdp(buff, xdp); break; case XDP_REDIRECT: if (xdp_do_redirect(aq_nic->ndev, xdp, prog) < 0) @@ -512,7 +454,6 @@ static struct sk_buff *aq_xdp_run_prog(struct aq_nic_s = *aq_nic, u64_stats_update_begin(&rx_ring->stats.rx.syncp); ++rx_ring->stats.rx.xdp_redirect; u64_stats_update_end(&rx_ring->stats.rx.syncp); - aq_get_rxpages_xdp(buff, xdp); break; default: fallthrough; @@ -523,11 +464,13 @@ static struct sk_buff *aq_xdp_run_prog(struct aq_nic_= s *aq_nic, u64_stats_update_end(&rx_ring->stats.rx.syncp); trace_xdp_exception(aq_nic->ndev, prog, act); bpf_warn_invalid_xdp_action(aq_nic->ndev, prog, act); + xdp_return_buff(xdp); break; case XDP_DROP: u64_stats_update_begin(&rx_ring->stats.rx.syncp); ++rx_ring->stats.rx.xdp_drop; u64_stats_update_end(&rx_ring->stats.rx.syncp); + xdp_return_buff(xdp); break; } =20 @@ -546,8 +489,11 @@ static bool aq_add_rx_fragment(struct device *dev, do { skb_frag_t *frag; =20 - if (unlikely(sinfo->nr_frags >=3D MAX_SKB_FRAGS)) + if (unlikely(sinfo->nr_frags >=3D MAX_SKB_FRAGS)) { + /* Attached frags must reach xdp_return_buff() */ + xdp_buff_set_frags_flag(xdp); return true; + } =20 frag =3D &sinfo->frags[sinfo->nr_frags++]; buff_ =3D &ring->buff_ring[buff_->next]; @@ -571,6 +517,11 @@ static bool aq_add_rx_fragment(struct device *dev, if (page_is_pfmemalloc(buff_->rxdata.page)) xdp_buff_set_frag_pfmemalloc(xdp); =20 + /* The frag's page pool reference is owned by the xdp_buff + * from now on. + */ + buff_->rxdata.page =3D NULL; + } while (!buff_->is_eop); =20 xdp_buff_set_frags_flag(xdp); @@ -674,6 +625,7 @@ static int __aq_ring_rx_clean(struct aq_ring_s *self, s= truct napi_struct *napi, err =3D -ENOMEM; goto err_exit; } + skb_mark_for_recycle(skb); if (is_ptp_ring) buff->len -=3D aq_ptp_extract_ts(self->aq_nic, skb_hwtstamps(skb), @@ -694,7 +646,7 @@ static int __aq_ring_rx_clean(struct aq_ring_s *self, s= truct napi_struct *napi, buff->rxdata.pg_off + hdr_len, buff->len - hdr_len, self->frame_max); - page_ref_inc(buff->rxdata.page); + buff->rxdata.page =3D NULL; } =20 if (!buff->is_eop) { @@ -713,7 +665,7 @@ static int __aq_ring_rx_clean(struct aq_ring_s *self, s= truct napi_struct *napi, buff_->rxdata.pg_off, buff_->len, self->frame_max); - page_ref_inc(buff_->rxdata.page); + buff_->rxdata.page =3D NULL; buff_->is_cleaned =3D 1; =20 buff->is_ip_cso &=3D buff_->is_ip_cso; @@ -851,6 +803,11 @@ static int __aq_ring_xdp_clean(struct aq_ring_s *rx_ri= ng, xdp_init_buff(&xdp, frame_sz, &rx_ring->xdp_rxq); xdp_prepare_buff(&xdp, hard_start, rx_ring->page_offset, buff->len, false); + /* The xdp_buff owns the buffer's page pool reference from + * here on; it comes back through the MEM_TYPE_PAGE_POOL + * memory model on every XDP verdict. + */ + buff->rxdata.page =3D NULL; if (!buff->is_eop) { if (aq_add_rx_fragment(dev, rx_ring, buff, &xdp)) { u64_stats_update_begin(&rx_ring->stats.rx.syncp); @@ -858,6 +815,7 @@ static int __aq_ring_xdp_clean(struct aq_ring_s *rx_rin= g, rx_ring->stats.rx.bytes +=3D xdp_get_buff_len(&xdp); ++rx_ring->stats.rx.xdp_aborted; u64_stats_update_end(&rx_ring->stats.rx.syncp); + xdp_return_buff(&xdp); continue; } } @@ -969,7 +927,9 @@ void aq_ring_rx_deinit(struct aq_ring_s *self) if (!buff->rxdata.page) continue; =20 - aq_free_rxpage(&buff->rxdata, aq_nic_get_dev(self->aq_nic)); + page_pool_put_full_page(self->pg_pool, buff->rxdata.page, + false); + buff->rxdata.page =3D NULL; } =20 self->sw_head =3D self->sw_tail; @@ -983,6 +943,11 @@ void aq_ring_free(struct aq_ring_s *self) kfree(self->buff_ring); self->buff_ring =3D NULL; =20 + if (self->pg_pool) { + page_pool_destroy(self->pg_pool); + self->pg_pool =3D NULL; + } + if (self->dx_ring) { dma_free_coherent(aq_nic_get_dev(self->aq_nic), self->size * self->dx_size, self->dx_ring, @@ -1021,9 +986,6 @@ unsigned int aq_ring_fill_stats_data(struct aq_ring_s = *self, u64 *data) data[++count] =3D self->stats.rx.alloc_fails; data[++count] =3D self->stats.rx.skb_alloc_fails; data[++count] =3D self->stats.rx.polls; - data[++count] =3D self->stats.rx.pg_flips; - data[++count] =3D self->stats.rx.pg_reuses; - data[++count] =3D self->stats.rx.pg_losts; data[++count] =3D self->stats.rx.xdp_aborted; data[++count] =3D self->stats.rx.xdp_drop; data[++count] =3D self->stats.rx.xdp_pass; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h b/drivers/net= /ethernet/aquantia/atlantic/aq_ring.h index 6431cc62962f..58bcadb3e3cc 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h @@ -17,12 +17,12 @@ #define AQ_XDP_TAILROOM SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) =20 struct page; +struct page_pool; struct aq_nic_cfg_s; =20 struct aq_rxpage { struct page *page; dma_addr_t daddr; - unsigned int order; unsigned int pg_off; }; =20 @@ -105,9 +105,6 @@ struct aq_ring_stats_rx_s { u64 alloc_fails; u64 skb_alloc_fails; u64 polls; - u64 pg_losts; - u64 pg_flips; - u64 pg_reuses; u64 xdp_aborted; u64 xdp_drop; u64 xdp_pass; @@ -151,6 +148,7 @@ struct aq_ring_s { u16 tail_size; union aq_ring_stats_s stats; dma_addr_t dx_ring_pa; + struct page_pool *pg_pool; struct bpf_prog *xdp_prog; enum atl_ring_type ring_type; struct xdp_rxq_info xdp_rxq; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c b/drivers/net/= ethernet/aquantia/atlantic/aq_vec.c index 05814fea0f5f..8e15405d4941 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_vec.c @@ -146,25 +146,32 @@ int aq_vec_ring_alloc(struct aq_vec_s *self, struct a= q_nic_s *aq_nic, aq_nic_set_tx_ring(aq_nic, idx_ring, ring); =20 ring =3D &self->ring[i][AQ_VEC_RX_ID]; + /* Registering the MEM_TYPE_PAGE_POOL memory model below needs + * the page pool created by aq_ring_rx_alloc(), so the ring is + * allocated first. If a registration fails, the ring has to be + * freed explicitly: rx_rings is not incremented yet, so the + * unwind through aq_vec_ring_free() would not cover it. + */ + err =3D aq_ring_rx_alloc(ring, aq_nic, idx_ring, aq_nic_cfg); + if (err) + goto err_exit; + if (xdp_rxq_info_reg(&ring->xdp_rxq, aq_nic->ndev, idx, self->napi.napi_id) < 0) { + aq_ring_free(ring); err =3D -ENOMEM; goto err_exit; } if (xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, - MEM_TYPE_PAGE_SHARED, NULL) < 0) { + MEM_TYPE_PAGE_POOL, + ring->pg_pool) < 0) { xdp_rxq_info_unreg(&ring->xdp_rxq); + aq_ring_free(ring); err =3D -ENOMEM; goto err_exit; } =20 - err =3D aq_ring_rx_alloc(ring, aq_nic, idx_ring, aq_nic_cfg); - if (err) { - xdp_rxq_info_unreg(&ring->xdp_rxq); - goto err_exit; - } - ++self->rx_rings; } =20 --=20 2.47.3