From nobody Fri Apr 10 02:37:16 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E46A351C19; Wed, 4 Mar 2026 16:36:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772642176; cv=none; b=jx3tW1BK94rAXRkZQZ+IkEMVVmyewP6X4EEOAAv4j+1AA1UKwg72uLRNFLIcWV4pqeDPMrZm/MIOSlzpwFOysHU8mbnsH5tSFJ9SOPiIi/FwvMdv6GE9EEFUVfi+nZNQV/ewC5LVYmxnJXfc6VT09JoZNghgn8VUuyehp3GNyXY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772642176; c=relaxed/simple; bh=xsT59N4NUbcB5J49JMTpLAAeLxkARBASaG8PLm+l12g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BeuKkpsloSigFvIXFSGB4NvStYmg4OuS6YSjX+5XbCvlr65YLGCHbV5ykYftyr4J09zpUX88z9ctknzYRtxGQNXgkI8JnGST8Dgcr4t9hVyRigRCVaNdGC7k85uZ0JvWPd3Ib+3JaqXYqj2CXTP/inNG+BxF9mciMUJ3OIFAXE0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=B4hbOs4w; arc=none smtp.client-ip=192.198.163.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="B4hbOs4w" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1772642171; x=1804178171; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xsT59N4NUbcB5J49JMTpLAAeLxkARBASaG8PLm+l12g=; b=B4hbOs4w9VkNBrDY5ZEiFHg5tXOozoOGHhKF1P740O0caGU2bE0Ppa+/ rY0rJar8OkljKeUwCbYPMjGwfvWrBQ173cEIwRaAVUZ4fx+sd37Z5vPAF DlcPvDm5o2KFzyBGRARwx3YEkLiYLlBgpovPArlIQL1O64HJwE9Lla5oI tnO7leRBaYGRDNSdYp8QYHwwOfFzQ+7xPsOtlMMR9PQ/99QWFUtEzervM lVTn7KL+GPt/lzfdlaagtJby/bRBXre/GrS6N1+s4JVcgL9ivlbOpizNU CxfkFCpAbdfuunmHTI7pu/ROL2DeyWnTUpg9NBtAjLa2tGBWfKYiQiEil Q==; X-CSE-ConnectionGUID: qDp7QXNLRLOkBtC8FWluNQ== X-CSE-MsgGUID: FgpoDs3GQ0us6TwwxdSuPg== X-IronPort-AV: E=McAfee;i="6800,10657,11719"; a="72906399" X-IronPort-AV: E=Sophos;i="6.21,324,1763452800"; d="scan'208";a="72906399" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa112.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Mar 2026 08:36:10 -0800 X-CSE-ConnectionGUID: jTaVhzu5QByvoLnfA4dngg== X-CSE-MsgGUID: YLdtvP/RTkGRyJ1x6Z0wXw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.21,324,1763452800"; d="scan'208";a="241404985" Received: from irvmail002.ir.intel.com ([10.43.11.120]) by fmviesa002.fm.intel.com with ESMTP; 04 Mar 2026 08:36:05 -0800 Received: from lincoln.igk.intel.com (lincoln.igk.intel.com [10.102.21.235]) by irvmail002.ir.intel.com (Postfix) with ESMTP id 98269312C8; Wed, 4 Mar 2026 16:36:03 +0000 (GMT) From: Larysa Zaremba To: Tony Nguyen , intel-wired-lan@lists.osuosl.org Cc: Larysa Zaremba , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexander Lobakin , Simon Horman , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Stanislav Fomichev , Aleksandr Loktionov , Natalia Wochtman , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH iwl-next v3 07/10] ixgbevf: support XDP_REDIRECT and .ndo_xdp_xmit Date: Wed, 4 Mar 2026 17:03:39 +0100 Message-ID: <20260304160345.1340940-8-larysa.zaremba@intel.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260304160345.1340940-1-larysa.zaremba@intel.com> References: <20260304160345.1340940-1-larysa.zaremba@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To fully support XDP_REDIRECT, utilize more libeth helpers in XDP Rx path, hence save cached_ntu in the ring structure instead of stack. ixgbevf-supported VFs usually have few queues, so use libeth_xdpsq_lock functionality for XDP queue sharing. Adjust filling-in of XDP Tx descriptors to use data from xdp frame. Otherwise, simply use libeth helpers to implement .ndo_xdp_xmit(). While at it, fix a typo in libeth docs. Reviewed-by: Aleksandr Loktionov Signed-off-by: Larysa Zaremba --- drivers/net/ethernet/intel/ixgbevf/ixgbevf.h | 2 + .../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 142 ++++++++---------- include/net/libeth/xdp.h | 2 +- 3 files changed, 64 insertions(+), 82 deletions(-) diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h b/drivers/net/eth= ernet/intel/ixgbevf/ixgbevf.h index a27081ee764b..ea86679e4f81 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h @@ -98,6 +98,8 @@ struct ixgbevf_ring { struct ixgbevf_tx_buffer *tx_buffer_info; struct libeth_sqe *xdp_sqes; }; + struct libeth_xdpsq_lock xdpq_lock; + u32 cached_ntu; unsigned long state; struct ixgbevf_stats stats; struct u64_stats_sync syncp; diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/ne= t/ethernet/intel/ixgbevf/ixgbevf_main.c index 177eb141e22d..2f3b4954ded8 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c @@ -649,10 +649,6 @@ static inline void ixgbevf_irq_enable_queues(struct ix= gbevf_adapter *adapter, IXGBE_WRITE_REG(hw, IXGBE_VTEIMS, qmask); } =20 -#define IXGBEVF_XDP_PASS 0 -#define IXGBEVF_XDP_CONSUMED 1 -#define IXGBEVF_XDP_TX 2 - static void ixgbevf_clean_xdp_num(struct ixgbevf_ring *xdp_ring, bool in_n= api, u16 to_clean) { @@ -710,12 +706,14 @@ static u16 ixgbevf_tx_get_num_sent(struct ixgbevf_rin= g *xdp_ring) static void ixgbevf_clean_xdp_ring(struct ixgbevf_ring *xdp_ring) { ixgbevf_clean_xdp_num(xdp_ring, false, xdp_ring->pending); + libeth_xdpsq_put(&xdp_ring->xdpq_lock, xdp_ring->netdev); } =20 static u32 ixgbevf_prep_xdp_sq(void *xdpsq, struct libeth_xdpsq *sq) { struct ixgbevf_ring *xdp_ring =3D xdpsq; =20 + libeth_xdpsq_lock(&xdp_ring->xdpq_lock); if (unlikely(ixgbevf_desc_unused(xdp_ring) < LIBETH_XDP_TX_BULK)) { u16 to_clean =3D ixgbevf_tx_get_num_sent(xdp_ring); =20 @@ -749,7 +747,7 @@ static u32 ixgbevf_prep_xdp_sq(void *xdpsq, struct libe= th_xdpsq *sq) *sq =3D (struct libeth_xdpsq) { .count =3D xdp_ring->count, .descs =3D xdp_ring->desc, - .lock =3D NULL, + .lock =3D &xdp_ring->xdpq_lock, .ntu =3D &xdp_ring->next_to_use, .pending =3D &xdp_ring->pending, .pool =3D NULL, @@ -775,9 +773,13 @@ static void ixgbevf_xdp_xmit_desc(struct libeth_xdp_tx= _desc desc, u32 i, cmd_type |=3D IXGBE_TXD_CMD_EOP; =20 if (desc.flags & LIBETH_XDP_TX_FIRST) { - struct skb_shared_info *sinfo =3D sq->sqes[i].sinfo; - u16 full_len =3D desc.len + sinfo->xdp_frags_size; + struct libeth_sqe *sqe =3D &sq->sqes[i]; + struct skb_shared_info *sinfo; + u16 full_len; =20 + sinfo =3D sqe->type =3D=3D LIBETH_SQE_XDP_TX ? sqe->sinfo : + xdp_get_shared_info_from_frame(sqe->xdpf); + full_len =3D desc.len + sinfo->xdp_frags_size; tx_desc->read.olinfo_status =3D cpu_to_le32((full_len << IXGBE_ADVTXD_PAYLEN_SHIFT) | IXGBE_ADVTXD_CC); @@ -787,76 +789,36 @@ static void ixgbevf_xdp_xmit_desc(struct libeth_xdp_t= x_desc desc, u32 i, tx_desc->read.cmd_type_len =3D cpu_to_le32(cmd_type); } =20 -LIBETH_XDP_DEFINE_START(); -LIBETH_XDP_DEFINE_FLUSH_TX(static ixgbevf_xdp_flush_tx, ixgbevf_prep_xdp_s= q, - ixgbevf_xdp_xmit_desc); -LIBETH_XDP_DEFINE_END(); - -static void ixgbevf_xdp_set_rs(struct ixgbevf_ring *xdp_ring, u32 cached_n= tu) +static void ixgbevf_xdp_rs_and_bump(void *xdpsq, bool sent, bool flush) { - u32 ltu =3D (xdp_ring->next_to_use ? : xdp_ring->count) - 1; + struct ixgbevf_ring *xdp_ring =3D xdpsq; union ixgbe_adv_tx_desc *desc; + u32 ltu; + + if ((!flush && xdp_ring->pending < xdp_ring->count - 1) || + xdp_ring->cached_ntu =3D=3D xdp_ring->next_to_use) + return; =20 + ltu =3D (xdp_ring->next_to_use ? : xdp_ring->count) - 1; desc =3D IXGBEVF_TX_DESC(xdp_ring, ltu); - xdp_ring->xdp_sqes[cached_ntu].rs_idx =3D ltu + 1; + xdp_ring->xdp_sqes[xdp_ring->cached_ntu].rs_idx =3D ltu + 1; desc->read.cmd_type_len |=3D cpu_to_le32(IXGBE_TXD_CMD); -} - -static void ixgbevf_rx_finalize_xdp(struct libeth_xdp_tx_bulk *tx_bulk, - bool xdp_xmit, u32 cached_ntu) -{ - struct ixgbevf_ring *xdp_ring =3D tx_bulk->xdpsq; - - if (!xdp_xmit) - goto unlock; - - if (tx_bulk->count) - ixgbevf_xdp_flush_tx(tx_bulk, LIBETH_XDP_TX_DROP); - - ixgbevf_xdp_set_rs(xdp_ring, cached_ntu); + xdp_ring->cached_ntu =3D xdp_ring->next_to_use; =20 /* Finish descriptor writes before bumping tail */ wmb(); ixgbevf_write_tail(xdp_ring, xdp_ring->next_to_use); -unlock: - rcu_read_unlock(); } =20 -static int ixgbevf_run_xdp(struct libeth_xdp_tx_bulk *tx_bulk, - struct libeth_xdp_buff *xdp) -{ - int result =3D IXGBEVF_XDP_PASS; - const struct bpf_prog *xdp_prog; - u32 act; - - xdp_prog =3D tx_bulk->prog; - if (!xdp_prog) - goto xdp_out; - - act =3D bpf_prog_run_xdp(xdp_prog, &xdp->base); - switch (act) { - case XDP_PASS: - break; - case XDP_TX: - result =3D IXGBEVF_XDP_TX; - if (!libeth_xdp_tx_queue_bulk(tx_bulk, xdp, - ixgbevf_xdp_flush_tx)) - result =3D IXGBEVF_XDP_CONSUMED; - break; - default: - bpf_warn_invalid_xdp_action(tx_bulk->dev, xdp_prog, act); - fallthrough; - case XDP_ABORTED: - trace_xdp_exception(tx_bulk->dev, xdp_prog, act); - fallthrough; /* handle aborts by dropping packet */ - case XDP_DROP: - result =3D IXGBEVF_XDP_CONSUMED; - libeth_xdp_return_buff(xdp); - break; - } -xdp_out: - return result; -} +LIBETH_XDP_DEFINE_START(); +LIBETH_XDP_DEFINE_FLUSH_TX(static ixgbevf_xdp_flush_tx, ixgbevf_prep_xdp_s= q, + ixgbevf_xdp_xmit_desc); +LIBETH_XDP_DEFINE_FLUSH_XMIT(static ixgbevf_xdp_flush_xmit, ixgbevf_prep_x= dp_sq, + ixgbevf_xdp_xmit_desc); +LIBETH_XDP_DEFINE_RUN_PROG(static ixgbevf_xdp_run_prog, ixgbevf_xdp_flush_= tx); +LIBETH_XDP_DEFINE_FINALIZE(static ixgbevf_xdp_finalize_xdp_napi, + ixgbevf_xdp_flush_tx, ixgbevf_xdp_rs_and_bump); +LIBETH_XDP_DEFINE_END(); =20 static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector, struct ixgbevf_ring *rx_ring, @@ -867,17 +829,11 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vect= or *q_vector, u16 cleaned_count =3D ixgbevf_desc_unused(rx_ring); LIBETH_XDP_ONSTACK_BULK(xdp_tx_bulk); LIBETH_XDP_ONSTACK_BUFF(xdp); - u32 cached_ntu; - bool xdp_xmit =3D false; - int xdp_res =3D 0; =20 libeth_xdp_init_buff(xdp, &rx_ring->xdp_stash, &rx_ring->xdp_rxq); libeth_xdp_tx_init_bulk(&xdp_tx_bulk, rx_ring->xdp_prog, adapter->netdev, adapter->xdp_ring, adapter->num_xdp_queues); - if (xdp_tx_bulk.prog) - cached_ntu =3D - ((struct ixgbevf_ring *)xdp_tx_bulk.xdpsq)->next_to_use; =20 while (likely(total_rx_packets < budget)) { union ixgbe_adv_rx_desc *rx_desc; @@ -910,11 +866,8 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vecto= r *q_vector, if (ixgbevf_is_non_eop(rx_ring, rx_desc)) continue; =20 - xdp_res =3D ixgbevf_run_xdp(&xdp_tx_bulk, xdp); - if (xdp_res) { - if (xdp_res =3D=3D IXGBEVF_XDP_TX) - xdp_xmit =3D true; - + if (xdp_tx_bulk.prog && + !ixgbevf_xdp_run_prog(xdp, &xdp_tx_bulk)) { xdp->data =3D NULL; total_rx_packets++; total_rx_bytes +=3D xdp_get_buff_len(&xdp->base); @@ -960,7 +913,7 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vector= *q_vector, /* place incomplete frames back on ring for completion */ libeth_xdp_save_buff(&rx_ring->xdp_stash, xdp); =20 - ixgbevf_rx_finalize_xdp(&xdp_tx_bulk, xdp_xmit, cached_ntu); + ixgbevf_xdp_finalize_xdp_napi(&xdp_tx_bulk); =20 u64_stats_update_begin(&rx_ring->syncp); rx_ring->stats.packets +=3D total_rx_packets; @@ -972,6 +925,23 @@ static int ixgbevf_clean_rx_irq(struct ixgbevf_q_vecto= r *q_vector, return total_rx_packets; } =20 +static int ixgbevf_xdp_xmit(struct net_device *dev, int n, + struct xdp_frame **frames, u32 flags) +{ + struct ixgbevf_adapter *adapter =3D netdev_priv(dev); + + if (unlikely(test_bit(__IXGBEVF_DOWN, &adapter->state))) + return -ENETDOWN; + + if (unlikely(!adapter->num_xdp_queues)) + return -ENXIO; + + return libeth_xdp_xmit_do_bulk(dev, n, frames, flags, adapter->xdp_ring, + adapter->num_xdp_queues, + ixgbevf_xdp_flush_xmit, + ixgbevf_xdp_rs_and_bump); +} + /** * ixgbevf_poll - NAPI polling calback * @napi: napi struct with our devices info in it @@ -1432,6 +1402,7 @@ static void ixgbevf_configure_tx_ring(struct ixgbevf_= adapter *adapter, ring->next_to_clean =3D 0; ring->next_to_use =3D 0; ring->pending =3D 0; + ring->cached_ntu =3D 0; =20 /* In order to avoid issues WTHRESH + PTHRESH should always be equal * to or less than the number of on chip descriptors, which is @@ -1444,12 +1415,15 @@ static void ixgbevf_configure_tx_ring(struct ixgbev= f_adapter *adapter, 32; /* PTHRESH =3D 32 */ =20 /* reinitialize tx_buffer_info */ - if (!ring_is_xdp(ring)) + if (!ring_is_xdp(ring)) { memset(ring->tx_buffer_info, 0, sizeof(struct ixgbevf_tx_buffer) * ring->count); - else + } else { memset(ring->xdp_sqes, 0, sizeof(struct libeth_sqe) * ring->count); + libeth_xdpsq_get(&ring->xdpq_lock, ring->netdev, + num_possible_cpus() > adapter->num_xdp_queues); + } =20 clear_bit(__IXGBEVF_HANG_CHECK_ARMED, &ring->state); clear_bit(__IXGBEVF_TX_XDP_RING_PRIMED, &ring->state); @@ -4177,6 +4151,8 @@ static int ixgbevf_xdp_setup(struct net_device *dev, = struct bpf_prog *prog, =20 /* If transitioning XDP modes reconfigure rings */ if (!!prog !=3D !!old_prog) { + xdp_features_clear_redirect_target(dev); + /* Hardware has to reinitialize queues and interrupts to * match packet buffer alignment. Unfortunately, the * hardware is not flexible enough to do this dynamically. @@ -4194,6 +4170,9 @@ static int ixgbevf_xdp_setup(struct net_device *dev, = struct bpf_prog *prog, xchg(&adapter->rx_ring[i]->xdp_prog, adapter->xdp_prog); } =20 + if (prog) + xdp_features_set_redirect_target(dev, true); + if (old_prog) bpf_prog_put(old_prog); =20 @@ -4224,6 +4203,7 @@ static const struct net_device_ops ixgbevf_netdev_ops= =3D { .ndo_vlan_rx_kill_vid =3D ixgbevf_vlan_rx_kill_vid, .ndo_features_check =3D ixgbevf_features_check, .ndo_bpf =3D ixgbevf_xdp, + .ndo_xdp_xmit =3D ixgbevf_xdp_xmit, }; =20 static void ixgbevf_assign_netdev_ops(struct net_device *dev) @@ -4356,7 +4336,7 @@ static int ixgbevf_probe(struct pci_dev *pdev, const = struct pci_device_id *ent) NETIF_F_HW_VLAN_CTAG_TX; =20 netdev->priv_flags |=3D IFF_UNICAST_FLT; - netdev->xdp_features =3D NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_RX_SG; + libeth_xdp_set_features_noredir(netdev, NULL, 0, NULL); =20 /* MTU range: 68 - 1504 or 9710 */ netdev->min_mtu =3D ETH_MIN_MTU; diff --git a/include/net/libeth/xdp.h b/include/net/libeth/xdp.h index 898723ab62e8..2e2154ccecae 100644 --- a/include/net/libeth/xdp.h +++ b/include/net/libeth/xdp.h @@ -1094,7 +1094,7 @@ __libeth_xdp_xmit_do_bulk(struct libeth_xdp_tx_bulk *= bq, * @xqs: array of XDPSQs driver structs * @nqs: number of active XDPSQs, the above array length * @fl: driver callback to flush an XDP xmit bulk - * @fin: driver cabback to finalize the queue + * @fin: driver callback to finalize the queue * * If the driver has active XDPSQs, perform common checks and send the fra= mes. * Finalize the queue, if requested. --=20 2.52.0