From nobody Sun Apr 12 10:20:06 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2265B371CEE; Wed, 4 Mar 2026 16:36:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772642171; cv=none; b=mzEtBEJzf+zLQ472Wd/M3E4GOg7+UN0r1vx2s+eY8vQPqxhG6d4CAglcQsQaYhXP6h2iVe4vGpk//BVMpLyyeIM3uIpiWkjpmDzQWzMqWwSMa5qox+jwRlO1CucsES3IT/m5EGNXDvgslmpIXz06kmtt9VQYkAR0LfrYYsG7yto= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772642171; c=relaxed/simple; bh=XmXgIlPS/QT1PKsgToHXtE34poJzGGSWVGBe5XGheoM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KHykaO/w/2S8Dj7nQvO6z6IjH8MjqqsehJz95tV1wmbfeliSJKpbIDQ5NKM3dL5ZI+ClIQ7wDAPXcie8GpxoxwoDAEfeDEEvKErnaACERk1KgMjAFqftIm3BM8uRKtB72owMx2sJKtvOvM2xtXBfEeQzv+vIIFVkd3mv6TtZ++4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=QRcfkHw/; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="QRcfkHw/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1772642164; x=1804178164; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XmXgIlPS/QT1PKsgToHXtE34poJzGGSWVGBe5XGheoM=; b=QRcfkHw/YPhVhBI1IDBlaht03teS+EDARz+Uw5cejFyMfUGGnI0ZU5FL BikQk5gSEwHocjxhxIeXxZQcu+iflu/W6JONBjwgg+cfRhXOhAXrNrdra pLVZvAocU33R7402qTodkK6qBUnJ+mSn3PebNubZpiK9X43CPPCE3nFcj 2XPG1mSHclr/z5xEPj8eiZtiK5eLJ4b3RpDQDdajxs0O4WSQ+IwVhSPlM thkIQadOx6zYzjhL5XzzXRJ/0F0h3CqqSsaddcev+vki+ywL6rzX9ixcD gQoHQ4UI6oSMDZnyjXuV0bxdHw3+k6fRbEmBvofy2ItUyHp8DQ/B+fm6f Q==; X-CSE-ConnectionGUID: CSmnOoy5QVKvS9p1E87dXQ== X-CSE-MsgGUID: saF14aeLT6yezapYovToNQ== X-IronPort-AV: E=McAfee;i="6800,10657,11719"; a="77580029" X-IronPort-AV: E=Sophos;i="6.21,324,1763452800"; d="scan'208";a="77580029" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Mar 2026 08:36:03 -0800 X-CSE-ConnectionGUID: CzrczvLoTVG4MwFyK7iUKQ== X-CSE-MsgGUID: wvDWb5MFSNSmpczv3biUgg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,324,1763452800"; d="scan'208";a="222895569" Received: from irvmail002.ir.intel.com ([10.43.11.120]) by orviesa004.jf.intel.com with ESMTP; 04 Mar 2026 08:35:57 -0800 Received: from lincoln.igk.intel.com (lincoln.igk.intel.com [10.102.21.235]) by irvmail002.ir.intel.com (Postfix) with ESMTP id C7AE3312CA; Wed, 4 Mar 2026 16:35:55 +0000 (GMT) From: Larysa Zaremba To: Tony Nguyen , intel-wired-lan@lists.osuosl.org Cc: Larysa Zaremba , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexander Lobakin , Simon Horman , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Stanislav Fomichev , Aleksandr Loktionov , Natalia Wochtman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH iwl-next v3 03/10] ixgbevf: use libeth in Rx processing Date: Wed, 4 Mar 2026 17:03:35 +0100 Message-ID: <20260304160345.1340940-4-larysa.zaremba@intel.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260304160345.1340940-1-larysa.zaremba@intel.com> References: <20260304160345.1340940-1-larysa.zaremba@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use page_pool buffers by the means of libeth in the Rx queues, this significantly reduces code complexity of the driver itself. Suggested-by: Alexander Lobakin Reviewed-by: Alexander Lobakin Reviewed-by: Aleksandr Loktionov Signed-off-by: Larysa Zaremba --- drivers/net/ethernet/intel/Kconfig | 1 + drivers/net/ethernet/intel/ixgbevf/defines.h | 2 +- drivers/net/ethernet/intel/ixgbevf/ixgbevf.h | 21 +- .../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 313 ++++++------------ 4 files changed, 120 insertions(+), 217 deletions(-) diff --git a/drivers/net/ethernet/intel/Kconfig b/drivers/net/ethernet/inte= l/Kconfig index 780f113986ea..42637514e804 100644 --- a/drivers/net/ethernet/intel/Kconfig +++ b/drivers/net/ethernet/intel/Kconfig @@ -203,6 +203,7 @@ config IXGBE_IPSEC config IXGBEVF tristate "Intel(R) 10GbE PCI Express Virtual Function Ethernet support" depends on PCI_MSI + select LIBETH_XDP help This driver supports Intel(R) PCI Express virtual functions for the Intel(R) ixgbe driver. For more information on how to identify your diff --git a/drivers/net/ethernet/intel/ixgbevf/defines.h b/drivers/net/eth= ernet/intel/ixgbevf/defines.h index e177d1d58696..afc927dd1438 100644 --- a/drivers/net/ethernet/intel/ixgbevf/defines.h +++ b/drivers/net/ethernet/intel/ixgbevf/defines.h @@ -71,7 +71,7 @@ typedef u32 ixgbe_link_speed; #define IXGBE_PSRTYPE_L2HDR 0x00001000 =20 /* SRRCTL bit definitions */ -#define IXGBE_SRRCTL_BSIZEPKT_SHIFT 10 /* so many KBs */ +#define IXGBE_SRRCTL_BSIZEPKT_STEP 1024 #define IXGBE_SRRCTL_RDMTS_SHIFT 22 #define IXGBE_SRRCTL_RDMTS_MASK 0x01C00000 #define IXGBE_SRRCTL_DROP_EN 0x10000000 diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h b/drivers/net/eth= ernet/intel/ixgbevf/ixgbevf.h index 2d7ca3f86868..ebf771f0caa4 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h @@ -42,12 +42,6 @@ struct ixgbevf_tx_buffer { u32 tx_flags; }; =20 -struct ixgbevf_rx_buffer { - dma_addr_t dma; - struct page *page; - __u32 page_offset; -}; - struct ixgbevf_stats { u64 packets; u64 bytes; @@ -84,19 +78,22 @@ struct ixgbevf_ring { struct ixgbevf_ring *next; struct ixgbevf_q_vector *q_vector; /* backpointer to q_vector */ struct net_device *netdev; - struct bpf_prog *xdp_prog; - struct device *dev; + struct bpf_prog __rcu *xdp_prog; + union { + struct page_pool *pp; /* Rx ring */ + struct device *dev; /* Tx ring */ + }; void *desc; /* descriptor ring memory */ dma_addr_t dma; /* phys. address of descriptor ring */ unsigned int size; /* length in bytes */ + u32 truesize; /* Rx buffer full size */ u16 count; /* amount of descriptors */ u16 next_to_use; u16 next_to_clean; - u16 next_to_alloc; =20 union { + struct libeth_fqe *rx_fqes; struct ixgbevf_tx_buffer *tx_buffer_info; - struct ixgbevf_rx_buffer *rx_buffer_info; }; unsigned long state; struct ixgbevf_stats stats; @@ -115,6 +112,7 @@ struct ixgbevf_ring { */ u16 reg_idx; int queue_index; /* needed for multiqueue queue management */ + u32 rx_buf_len; } ____cacheline_internodealigned_in_smp; =20 /* How many Rx Buffers do we bundle into one write to the hardware ? */ @@ -144,7 +142,8 @@ struct ixgbevf_ring { =20 #define MAXIMUM_ETHERNET_VLAN_SIZE (VLAN_ETH_FRAME_LEN + ETH_FCS_LEN) =20 -#define IXGBEVF_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN) +#define IXGBEVF_RX_PAGE_LEN(hr) (ALIGN_DOWN(LIBETH_RX_PAGE_LEN(hr), \ + IXGBE_SRRCTL_BSIZEPKT_STEP)) =20 #define IXGBE_TX_FLAGS_CSUM BIT(0) #define IXGBE_TX_FLAGS_VLAN BIT(1) diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/ne= t/ethernet/intel/ixgbevf/ixgbevf_main.c index f5a7dd37084f..5a270dd2c7aa 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c @@ -30,6 +30,7 @@ #include #include #include +#include #include =20 #include "ixgbevf.h" @@ -82,6 +83,7 @@ static const struct pci_device_id ixgbevf_pci_tbl[] =3D { MODULE_DEVICE_TABLE(pci, ixgbevf_pci_tbl); =20 MODULE_DESCRIPTION("Intel(R) 10 Gigabit Virtual Function Network Driver"); +MODULE_IMPORT_NS("LIBETH"); MODULE_LICENSE("GPL v2"); =20 #define DEFAULT_MSG_ENABLE (NETIF_MSG_DRV|NETIF_MSG_PROBE|NETIF_MSG_LINK) @@ -304,7 +306,7 @@ static bool ixgbevf_clean_tx_irq(struct ixgbevf_q_vecto= r *q_vector, =20 /* free the skb */ if (ring_is_xdp(tx_ring)) - page_frag_free(tx_buffer->data); + libeth_xdp_return_va(tx_buffer->data, true); else napi_consume_skb(tx_buffer->skb, napi_budget); =20 @@ -521,35 +523,6 @@ static void ixgbevf_process_skb_fields(struct ixgbevf_= ring *rx_ring, =20 if (ixgbevf_test_staterr(rx_desc, IXGBE_RXDADV_STAT_SECP)) ixgbevf_ipsec_rx(rx_ring, rx_desc, skb); - - skb->protocol =3D eth_type_trans(skb, rx_ring->netdev); -} - -static -struct ixgbevf_rx_buffer *ixgbevf_get_rx_buffer(struct ixgbevf_ring *rx_ri= ng, - const unsigned int size) -{ - struct ixgbevf_rx_buffer *rx_buffer; - - rx_buffer =3D &rx_ring->rx_buffer_info[rx_ring->next_to_clean]; - prefetchw(rx_buffer->page); - - /* we are reusing so sync this buffer for CPU use */ - dma_sync_single_range_for_cpu(rx_ring->dev, - rx_buffer->dma, - rx_buffer->page_offset, - size, - DMA_FROM_DEVICE); - - return rx_buffer; -} - -static void ixgbevf_put_rx_buffer(struct ixgbevf_ring *rx_ring, - struct ixgbevf_rx_buffer *rx_buffer) -{ - dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma, PAGE_SIZE, - DMA_FROM_DEVICE, IXGBEVF_RX_DMA_ATTR); - rx_buffer->page =3D NULL; } =20 /** @@ -579,41 +552,6 @@ static bool ixgbevf_is_non_eop(struct ixgbevf_ring *rx= _ring, return true; } =20 -static bool ixgbevf_alloc_mapped_page(struct ixgbevf_ring *rx_ring, - struct ixgbevf_rx_buffer *bi) -{ - struct page *page =3D bi->page; - dma_addr_t dma; - - /* alloc new page for storage */ - page =3D dev_alloc_page(); - if (unlikely(!page)) { - rx_ring->rx_stats.alloc_rx_page_failed++; - return false; - } - - /* map page for use */ - dma =3D dma_map_page_attrs(rx_ring->dev, page, 0, PAGE_SIZE, - DMA_FROM_DEVICE, IXGBEVF_RX_DMA_ATTR); - - /* if mapping failed free memory back to system since - * there isn't much point in holding memory we can't use - */ - if (dma_mapping_error(rx_ring->dev, dma)) { - __free_page(page); - - rx_ring->rx_stats.alloc_rx_page_failed++; - return false; - } - - bi->dma =3D dma; - bi->page =3D page; - bi->page_offset =3D IXGBEVF_SKB_PAD; - rx_ring->rx_stats.alloc_rx_page++; - - return true; -} - /** * ixgbevf_alloc_rx_buffers - Replace used receive buffers; packet split * @rx_ring: rx descriptor ring (for a specific queue) to setup buffers on @@ -623,39 +561,34 @@ static void ixgbevf_alloc_rx_buffers(struct ixgbevf_r= ing *rx_ring, u16 cleaned_count) { union ixgbe_adv_rx_desc *rx_desc; - struct ixgbevf_rx_buffer *bi; - unsigned int i =3D rx_ring->next_to_use; + const struct libeth_fq_fp fq =3D { + .pp =3D rx_ring->pp, + .fqes =3D rx_ring->rx_fqes, + .truesize =3D rx_ring->truesize, + .count =3D rx_ring->count, + }; + u16 ntu =3D rx_ring->next_to_use; =20 /* nothing to do or no valid netdev defined */ if (!cleaned_count || !rx_ring->netdev) return; =20 - rx_desc =3D IXGBEVF_RX_DESC(rx_ring, i); - bi =3D &rx_ring->rx_buffer_info[i]; - i -=3D rx_ring->count; + rx_desc =3D IXGBEVF_RX_DESC(rx_ring, ntu); =20 do { - if (!ixgbevf_alloc_mapped_page(rx_ring, bi)) - break; + dma_addr_t addr; =20 - /* sync the buffer for use by the device */ - dma_sync_single_range_for_device(rx_ring->dev, bi->dma, - bi->page_offset, - IXGBEVF_RXBUFFER_3072, - DMA_FROM_DEVICE); + addr =3D libeth_rx_alloc(&fq, ntu); + if (addr =3D=3D DMA_MAPPING_ERROR) + break; =20 - /* Refresh the desc even if pkt_addr didn't change - * because each write-back erases this info. - */ - rx_desc->read.pkt_addr =3D cpu_to_le64(bi->dma + bi->page_offset); + rx_desc->read.pkt_addr =3D cpu_to_le64(addr); =20 rx_desc++; - bi++; - i++; - if (unlikely(!i)) { + ntu++; + if (unlikely(ntu =3D=3D rx_ring->count)) { rx_desc =3D IXGBEVF_RX_DESC(rx_ring, 0); - bi =3D rx_ring->rx_buffer_info; - i -=3D rx_ring->count; + ntu =3D 0; } =20 /* clear the length for the next_to_use descriptor */ @@ -664,14 +597,9 @@ static void ixgbevf_alloc_rx_buffers(struct ixgbevf_ri= ng *rx_ring, cleaned_count--; } while (cleaned_count); =20 - i +=3D rx_ring->count; - - if (rx_ring->next_to_use !=3D i) { + if (likely(rx_ring->next_to_use !=3D ntu)) { /* record the next descriptor to use */ - rx_ring->next_to_use =3D i; - - /* update next to alloc since we have filled the ring */ - rx_ring->next_to_alloc =3D i; + rx_ring->next_to_use =3D ntu; =20 /* Force memory writes to complete before letting h/w * know there are new descriptors to fetch. (Only @@ -679,7 +607,7 @@ static void ixgbevf_alloc_rx_buffers(struct ixgbevf_rin= g *rx_ring, * such as IA-64). */ wmb(); - ixgbevf_write_tail(rx_ring, i); + ixgbevf_write_tail(rx_ring, ntu); } } =20 @@ -716,10 +644,6 @@ static bool ixgbevf_cleanup_headers(struct ixgbevf_rin= g *rx_ring, } } =20 - /* if eth_skb_pad returns an error the skb was freed */ - if (eth_skb_pad(skb)) - return true; - return false; } =20 @@ -732,15 +656,15 @@ static bool ixgbevf_cleanup_headers(struct ixgbevf_ri= ng *rx_ring, * * This function will add the data contained in rx_buffer->page to the skb. **/ -static void ixgbevf_add_rx_frag(struct ixgbevf_ring *rx_ring, - struct ixgbevf_rx_buffer *rx_buffer, +static void ixgbevf_add_rx_frag(const struct libeth_fqe *rx_buffer, struct sk_buff *skb, unsigned int size) { - unsigned int truesize =3D SKB_DATA_ALIGN(IXGBEVF_SKB_PAD + size); + u32 hr =3D netmem_get_pp(rx_buffer->netmem)->p.offset; =20 - skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page, - rx_buffer->page_offset, size, truesize); + skb_add_rx_frag_netmem(skb, skb_shinfo(skb)->nr_frags, + rx_buffer->netmem, rx_buffer->offset + hr, + size, rx_buffer->truesize); } =20 static inline void ixgbevf_irq_enable_queues(struct ixgbevf_adapter *adapt= er, @@ -751,38 +675,6 @@ static inline void ixgbevf_irq_enable_queues(struct ix= gbevf_adapter *adapter, IXGBE_WRITE_REG(hw, IXGBE_VTEIMS, qmask); } =20 -static struct sk_buff *ixgbevf_build_skb(struct ixgbevf_ring *rx_ring, - struct ixgbevf_rx_buffer *rx_buffer, - struct xdp_buff *xdp, - union ixgbe_adv_rx_desc *rx_desc) -{ - unsigned int metasize =3D xdp->data - xdp->data_meta; - unsigned int truesize =3D SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) + - SKB_DATA_ALIGN(xdp->data_end - - xdp->data_hard_start); - struct sk_buff *skb; - - /* Prefetch first cache line of first page. If xdp->data_meta - * is unused, this points to xdp->data, otherwise, we likely - * have a consumer accessing first few bytes of meta data, - * and then actual data. - */ - net_prefetch(xdp->data_meta); - - /* build an skb around the page buffer */ - skb =3D napi_build_skb(xdp->data_hard_start, truesize); - if (unlikely(!skb)) - return NULL; - - /* update pointers within the skb to store the data */ - skb_reserve(skb, xdp->data - xdp->data_hard_start); - __skb_put(skb, xdp->data_end - xdp->data); - if (metasize) - skb_metadata_set(skb, metasize); - - return skb; -} - #define IXGBEVF_XDP_PASS 0 #define IXGBEVF_XDP_CONSUMED 1 #define IXGBEVF_XDP_TX 2 @@ -866,7 +758,7 @@ static int ixgbevf_xmit_xdp_ring(struct ixgbevf_ring *r= ing, =20 static int ixgbevf_run_xdp(struct ixgbevf_adapter *adapter, struct ixgbevf_ring *rx_ring, - struct xdp_buff *xdp) + struct libeth_xdp_buff *xdp) { int result =3D IXGBEVF_XDP_PASS; struct ixgbevf_ring *xdp_ring; @@ -878,13 +770,13 @@ static int ixgbevf_run_xdp(struct ixgbevf_adapter *ad= apter, if (!xdp_prog) goto xdp_out; =20 - act =3D bpf_prog_run_xdp(xdp_prog, xdp); + act =3D bpf_prog_run_xdp(xdp_prog, &xdp->base); switch (act) { case XDP_PASS: break; case XDP_TX: xdp_ring =3D adapter->xdp_ring[rx_ring->queue_index]; - result =3D ixgbevf_xmit_xdp_ring(xdp_ring, xdp); + result =3D ixgbevf_xmit_xdp_ring(xdp_ring, &xdp->base); if (result =3D=3D IXGBEVF_XDP_CONSUMED) goto out_failure; break; @@ -897,6 +789,7 @@ static int ixgbevf_run_xdp(struct ixgbevf_adapter *adap= ter, fallthrough; /* handle aborts by dropping packet */ case XDP_DROP: result =3D IXGBEVF_XDP_CONSUMED; + libeth_xdp_return_buff(xdp); break; } xdp_out: @@ -911,16 +804,15 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vect= or *q_vector, struct ixgbevf_adapter *adapter =3D q_vector->adapter; u16 cleaned_count =3D ixgbevf_desc_unused(rx_ring); struct sk_buff *skb =3D rx_ring->skb; + LIBETH_XDP_ONSTACK_BUFF(xdp); bool xdp_xmit =3D false; - struct xdp_buff xdp; int xdp_res =3D 0; =20 - /* Frame size depend on rx_ring setup when PAGE_SIZE=3D4K */ - xdp_init_buff(&xdp, IXGBEVF_RXBUFFER_3072, &rx_ring->xdp_rxq); + xdp->base.rxq =3D &rx_ring->xdp_rxq; =20 while (likely(total_rx_packets < budget)) { - struct ixgbevf_rx_buffer *rx_buffer; union ixgbe_adv_rx_desc *rx_desc; + struct libeth_fqe *rx_buffer; unsigned int size; =20 /* return some buffers to hardware, one at a time is too slow */ @@ -940,18 +832,14 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vect= or *q_vector, */ rmb(); =20 - rx_buffer =3D - ixgbevf_get_rx_buffer(rx_ring, IXGBEVF_RXBUFFER_3072); + rx_buffer =3D &rx_ring->rx_fqes[rx_ring->next_to_clean]; + libeth_rx_sync_for_cpu(rx_buffer, size); =20 /* retrieve a buffer from the ring */ if (!skb) { - unsigned int offset =3D rx_buffer->page_offset; - unsigned char *hard_start; - - hard_start =3D page_address(rx_buffer->page) + - rx_buffer->page_offset - offset; - xdp_prepare_buff(&xdp, hard_start, offset, size, true); - xdp_res =3D ixgbevf_run_xdp(adapter, rx_ring, &xdp); + libeth_xdp_prepare_buff(xdp, rx_buffer, size); + prefetch(xdp->data); + xdp_res =3D ixgbevf_run_xdp(adapter, rx_ring, xdp); } =20 if (xdp_res) { @@ -961,10 +849,9 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vecto= r *q_vector, total_rx_packets++; total_rx_bytes +=3D size; } else if (skb) { - ixgbevf_add_rx_frag(rx_ring, rx_buffer, skb, size); + ixgbevf_add_rx_frag(rx_buffer, skb, size); } else { - skb =3D ixgbevf_build_skb(rx_ring, rx_buffer, - &xdp, rx_desc); + skb =3D xdp_build_skb_from_buff(&xdp->base); } =20 /* exit if we failed to retrieve a buffer */ @@ -973,7 +860,6 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector= *q_vector, break; } =20 - ixgbevf_put_rx_buffer(rx_ring, rx_buffer); cleaned_count++; =20 /* fetch next buffer in frame if non-eop */ @@ -1542,7 +1428,8 @@ static void ixgbevf_configure_tx(struct ixgbevf_adapt= er *adapter) #define IXGBE_SRRCTL_BSIZEHDRSIZE_SHIFT 2 =20 static void ixgbevf_configure_srrctl(struct ixgbevf_adapter *adapter, - struct ixgbevf_ring *ring, int index) + struct ixgbevf_ring *ring, int index, + bool rlpml_valid) { struct ixgbe_hw *hw =3D &adapter->hw; u32 srrctl; @@ -1550,7 +1437,11 @@ static void ixgbevf_configure_srrctl(struct ixgbevf_= adapter *adapter, srrctl =3D IXGBE_SRRCTL_DROP_EN; =20 srrctl |=3D IXGBEVF_RX_HDR_SIZE << IXGBE_SRRCTL_BSIZEHDRSIZE_SHIFT; - srrctl |=3D IXGBEVF_RXBUFFER_3072 >> IXGBE_SRRCTL_BSIZEPKT_SHIFT; + if (rlpml_valid) + srrctl |=3D DIV_ROUND_UP(ring->rx_buf_len, + IXGBE_SRRCTL_BSIZEPKT_STEP); + else + srrctl |=3D ring->rx_buf_len / IXGBE_SRRCTL_BSIZEPKT_STEP; srrctl |=3D IXGBE_SRRCTL_DESCTYPE_ADV_ONEBUF; =20 IXGBE_WRITE_REG(hw, IXGBE_VFSRRCTL(index), srrctl); @@ -1681,9 +1572,10 @@ static void ixgbevf_configure_rx_ring(struct ixgbevf= _adapter *adapter, { struct ixgbe_hw *hw =3D &adapter->hw; union ixgbe_adv_rx_desc *rx_desc; + u8 reg_idx =3D ring->reg_idx; + bool rlpml_valid =3D false; u64 rdba =3D ring->dma; u32 rxdctl; - u8 reg_idx =3D ring->reg_idx; =20 /* disable queue to avoid issues while updating state */ rxdctl =3D IXGBE_READ_REG(hw, IXGBE_VFRXDCTL(reg_idx)); @@ -1709,10 +1601,6 @@ static void ixgbevf_configure_rx_ring(struct ixgbevf= _adapter *adapter, IXGBE_WRITE_REG(hw, IXGBE_VFRDT(reg_idx), 0); ring->tail =3D adapter->io_addr + IXGBE_VFRDT(reg_idx); =20 - /* initialize rx_buffer_info */ - memset(ring->rx_buffer_info, 0, - sizeof(struct ixgbevf_rx_buffer) * ring->count); - /* initialize Rx descriptor 0 */ rx_desc =3D IXGBEVF_RX_DESC(ring, 0); rx_desc->wb.upper.length =3D 0; @@ -1720,16 +1608,21 @@ static void ixgbevf_configure_rx_ring(struct ixgbev= f_adapter *adapter, /* reset ntu and ntc to place SW in sync with hardwdare */ ring->next_to_clean =3D 0; ring->next_to_use =3D 0; - ring->next_to_alloc =3D 0; - - ixgbevf_configure_srrctl(adapter, ring, reg_idx); =20 /* RXDCTL.RLPML does not work on 82599 */ if (adapter->hw.mac.type !=3D ixgbe_mac_82599_vf) { - rxdctl &=3D ~(IXGBE_RXDCTL_RLPMLMASK | - IXGBE_RXDCTL_RLPML_EN); + u32 pkt_len =3D + READ_ONCE(adapter->netdev->mtu) + LIBETH_RX_LL_LEN; + + rxdctl &=3D ~(IXGBE_RXDCTL_RLPMLMASK | IXGBE_RXDCTL_RLPML_EN); + if (pkt_len <=3D IXGBE_RXDCTL_RLPMLMASK) { + rxdctl |=3D pkt_len | IXGBE_RXDCTL_RLPML_EN; + rlpml_valid =3D true; + } } =20 + ixgbevf_configure_srrctl(adapter, ring, reg_idx, rlpml_valid); + rxdctl |=3D IXGBE_RXDCTL_ENABLE | IXGBE_RXDCTL_VME; IXGBE_WRITE_REG(hw, IXGBE_VFRXDCTL(reg_idx), rxdctl); =20 @@ -2125,8 +2018,6 @@ void ixgbevf_up(struct ixgbevf_adapter *adapter) **/ static void ixgbevf_clean_rx_ring(struct ixgbevf_ring *rx_ring) { - u16 i =3D rx_ring->next_to_clean; - /* Free Rx ring sk_buff */ if (rx_ring->skb) { dev_kfree_skb(rx_ring->skb); @@ -2134,29 +2025,14 @@ static void ixgbevf_clean_rx_ring(struct ixgbevf_ri= ng *rx_ring) } =20 /* Free all the Rx ring pages */ - while (i !=3D rx_ring->next_to_alloc) { - struct ixgbevf_rx_buffer *rx_buffer; + for (u32 i =3D rx_ring->next_to_clean; i !=3D rx_ring->next_to_use; ) { + const struct libeth_fqe *rx_fqe =3D &rx_ring->rx_fqes[i]; =20 - rx_buffer =3D &rx_ring->rx_buffer_info[i]; - - /* Invalidate cache lines that may have been written to by - * device so that we avoid corrupting memory. - */ - dma_sync_single_range_for_cpu(rx_ring->dev, - rx_buffer->dma, - rx_buffer->page_offset, - IXGBEVF_RXBUFFER_3072, - DMA_FROM_DEVICE); - - /* free resources associated with mapping */ - ixgbevf_put_rx_buffer(rx_ring, rx_buffer); - __free_page(rx_buffer->page); - i++; - if (i =3D=3D rx_ring->count) + libeth_rx_recycle_slow(rx_fqe->netmem); + if (unlikely(++i =3D=3D rx_ring->count)) i =3D 0; } =20 - rx_ring->next_to_alloc =3D 0; rx_ring->next_to_clean =3D 0; rx_ring->next_to_use =3D 0; } @@ -2175,7 +2051,7 @@ static void ixgbevf_clean_tx_ring(struct ixgbevf_ring= *tx_ring) =20 /* Free all the Tx ring sk_buffs */ if (ring_is_xdp(tx_ring)) - page_frag_free(tx_buffer->data); + libeth_xdp_return_va(tx_buffer->data, false); else dev_kfree_skb_any(tx_buffer->skb); =20 @@ -3257,12 +3133,26 @@ static int ixgbevf_setup_all_tx_resources(struct ix= gbevf_adapter *adapter) int ixgbevf_setup_rx_resources(struct ixgbevf_adapter *adapter, struct ixgbevf_ring *rx_ring) { - int size; + struct libeth_fq fq =3D { + .count =3D rx_ring->count, + .nid =3D NUMA_NO_NODE, + .type =3D LIBETH_FQE_MTU, + .xdp =3D !!rx_ring->xdp_prog, + .idx =3D rx_ring->queue_index, + .buf_len =3D IXGBEVF_RX_PAGE_LEN(rx_ring->xdp_prog ? + LIBETH_XDP_HEADROOM : + LIBETH_SKB_HEADROOM), + }; + int ret; =20 - size =3D sizeof(struct ixgbevf_rx_buffer) * rx_ring->count; - rx_ring->rx_buffer_info =3D vmalloc(size); - if (!rx_ring->rx_buffer_info) - goto err; + ret =3D libeth_rx_fq_create(&fq, &rx_ring->q_vector->napi); + if (ret) + return ret; + + rx_ring->pp =3D fq.pp; + rx_ring->rx_fqes =3D fq.fqes; + rx_ring->truesize =3D fq.truesize; + rx_ring->rx_buf_len =3D fq.buf_len; =20 u64_stats_init(&rx_ring->syncp); =20 @@ -3270,25 +3160,31 @@ int ixgbevf_setup_rx_resources(struct ixgbevf_adapt= er *adapter, rx_ring->size =3D rx_ring->count * sizeof(union ixgbe_adv_rx_desc); rx_ring->size =3D ALIGN(rx_ring->size, 4096); =20 - rx_ring->desc =3D dma_alloc_coherent(rx_ring->dev, rx_ring->size, + rx_ring->desc =3D dma_alloc_coherent(fq.pp->p.dev, rx_ring->size, &rx_ring->dma, GFP_KERNEL); =20 - if (!rx_ring->desc) + if (!rx_ring->desc) { + ret =3D -ENOMEM; goto err; + } =20 /* XDP RX-queue info */ - if (xdp_rxq_info_reg(&rx_ring->xdp_rxq, adapter->netdev, - rx_ring->queue_index, 0) < 0) + ret =3D __xdp_rxq_info_reg(&rx_ring->xdp_rxq, adapter->netdev, + rx_ring->queue_index, 0, rx_ring->truesize); + if (ret) goto err; =20 + xdp_rxq_info_attach_page_pool(&rx_ring->xdp_rxq, fq.pp); + rx_ring->xdp_prog =3D adapter->xdp_prog; =20 return 0; err: - vfree(rx_ring->rx_buffer_info); - rx_ring->rx_buffer_info =3D NULL; + libeth_rx_fq_destroy(&fq); + rx_ring->rx_fqes =3D NULL; + rx_ring->pp =3D NULL; dev_err(rx_ring->dev, "Unable to allocate memory for the Rx descriptor ri= ng\n"); - return -ENOMEM; + return ret; } =20 /** @@ -3329,17 +3225,24 @@ static int ixgbevf_setup_all_rx_resources(struct ix= gbevf_adapter *adapter) **/ void ixgbevf_free_rx_resources(struct ixgbevf_ring *rx_ring) { + struct libeth_fq fq =3D { + .fqes =3D rx_ring->rx_fqes, + .pp =3D rx_ring->pp, + }; + ixgbevf_clean_rx_ring(rx_ring); =20 rx_ring->xdp_prog =3D NULL; + xdp_rxq_info_detach_mem_model(&rx_ring->xdp_rxq); xdp_rxq_info_unreg(&rx_ring->xdp_rxq); - vfree(rx_ring->rx_buffer_info); - rx_ring->rx_buffer_info =3D NULL; =20 - dma_free_coherent(rx_ring->dev, rx_ring->size, rx_ring->desc, + dma_free_coherent(fq.pp->p.dev, rx_ring->size, rx_ring->desc, rx_ring->dma); - rx_ring->desc =3D NULL; + + libeth_rx_fq_destroy(&fq); + rx_ring->rx_fqes =3D NULL; + rx_ring->pp =3D NULL; } =20 /** --=20 2.52.0