From nobody Sun Feb 8 09:32:47 2026 Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9552514B835 for ; Fri, 23 Feb 2024 11:02:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708686138; cv=none; b=RNotcMTsWq1a1gfDejrjgysq871GN/Lm8D5ZB8HE4JOt78iITShVjciblBRxuR7fqzSQlF36CO57dQ3LPTYv5mi/63Lqb4hRKAoZOJFFfNyiA19K9zO5nl5TZof7FJdnteoc31uMHdY0C4R3rHoQjR0RqaYSQdL6kTH3bpHvT54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708686138; c=relaxed/simple; bh=PVvzGTB+DdAhYFzi2A9SFrTtfoZLsYbD95ro9qa4eAw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=Vl/xh39TN/FySEPH0L6um8dpkfJ/pzuT3CvSxY2kF40DQefcz0vhyZR24YXLJnbYbmgzE/eGac3Lsz3QHTAjwsYEqKCObRo7fPYda+6IA7zr5M4Q/64MKo9z5q/XWIJfJgyqMTrLjLWlCtyZF5vkMyMgGW5Km29JN/kjVZwomzw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=baylibre.com; spf=pass smtp.mailfrom=baylibre.com; dkim=pass (2048-bit key) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.b=m56ycVOd; arc=none smtp.client-ip=209.85.208.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=baylibre.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.b="m56ycVOd" Received: by mail-lj1-f172.google.com with SMTP id 38308e7fff4ca-2d2531294faso11193261fa.2 for ; Fri, 23 Feb 2024 03:02:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1708686133; x=1709290933; darn=vger.kernel.org; h=cc:to:message-id:content-transfer-encoding:mime-version:subject :date:from:from:to:cc:subject:date:message-id:reply-to; bh=T4/Oils4dMi6DnXoCiXpKiFP66KSJjo2vzfJqhdN6wQ=; b=m56ycVOdMMgYuX0sgPnJO0VhSM4otMCRCWpXy4VlMJnmw3pDuKNkbMG9niicnsjoMo UrJC2a0s0hemDYXfyrPwAOLP+bXYr0xOFgJkjBXCkw6WsucHCO4QM/EV72UA+Wr0nsfD lpXINTah+v8OSUdPLzm01B/qjzRhb0hQbsFHbnYjDd78yIWPY5HT9Wjju/N1T4mwyBJZ H9tz7gzVbW/HvmLqnQem6cwGLrXyr6SrPP9+nig/aSt3bNme7r81o4G5QdxoElWDjnDL Uebdus8VqYt0DG3kz0TFmGx+HgSfGcdeVSvVw5MKd5MEBDqmaQ3R11Ggy7Z2sYCLYSgw IHDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708686133; x=1709290933; h=cc:to:message-id:content-transfer-encoding:mime-version:subject :date:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=T4/Oils4dMi6DnXoCiXpKiFP66KSJjo2vzfJqhdN6wQ=; b=p6APQBvVlUVJ8Nl07WqNHS8GI3g71aJCfIYAae0XZlA85OZI43CxW7UHZTHi+HE0JV +02Dbvvzf5fwxQj3qXDdqjWhgKC2pV9eUw1IczIYE/bAQ8Jhdw64iHH7MKI7wcrVEQJY NrITKf+X/qWaI5ZotAT0twboWxNz0iaStNnK3Bp14JvYe0Ab6tednyA46+7TACWad1jn gwKongjTWnzZHyPml3eg7KTesYovdgznxBHu/aKxn2OtZaXPaECWRecXmT8GLfZygXDI p0Lf3B/G0se42vrbY3ZddafElPUM/gXJdBaE4Xhz44g8KkCBECHCRquNMC/nOAHN81d9 UFIQ== X-Forwarded-Encrypted: i=1; AJvYcCV6xEF1VJ+5R10ONflr+OFrsbdcW2JL16QOxthD/IQc4aH+nnBfbjL3UCVhdTN6g1wtB/uz1MyUcUz8Wso1ZregTz69/dRKhogkPGOk X-Gm-Message-State: AOJu0Yy9d2dHqtrKhToQwAl6YudO5k7IpQrTLRmy2N0GMEK4zK4o2iFH zLx8pTv/TRCx1Ntgq1yj+QKm+/+EU1+Pm4azvAYB3Rdk/DH0/1KbuAodllUU+8c= X-Google-Smtp-Source: AGHT+IGe6NrMVx3QNu4DVZNW1I++0CUx85oCa3jxIVqAXxqnQ14o/Qf+tFQI1wrRjTDItkH2MTyacw== X-Received: by 2002:a2e:95c6:0:b0:2d2:5430:605a with SMTP id y6-20020a2e95c6000000b002d25430605amr1114617ljh.7.1708686132266; Fri, 23 Feb 2024 03:02:12 -0800 (PST) Received: from [127.0.1.1] ([84.102.31.43]) by smtp.gmail.com with ESMTPSA id e17-20020a056000195100b0033b60bad2fcsm2363745wry.113.2024.02.23.03.02.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 03:02:11 -0800 (PST) From: Julien Panis Date: Fri, 23 Feb 2024 12:01:37 +0100 Subject: [PATCH] net: ethernet: ti: am65-cpsw: Add minimal XDP support Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240223-am65-cpsw-xdp-basic-v1-1-9f0b6cbda310@baylibre.com> X-B4-Tracking: v=1; b=H4sIABB72GUC/x2NUQrCMBAFr1L228UYUwleRaRsktWu2BiyUAuld zf1cx4zvBWUq7DCtVuh8iwqn9zgdOggjpSfjJIagzXWGWvPSNOlx1j0i0sqGEglokvBW98bH5y HVraVMVTKcdzbV6Esemz+MJHkt2QeZrOLpfJDlv/97b5tP87IniyOAAAA To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Russell King , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Sumit Semwal , =?utf-8?q?Christian_K=C3=B6nig?= Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, Julien Panis X-Mailer: b4 0.12.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1708686130; l=20210; i=jpanis@baylibre.com; s=20230526; h=from:subject:message-id; bh=PVvzGTB+DdAhYFzi2A9SFrTtfoZLsYbD95ro9qa4eAw=; b=R668PkKPyqv/S344bX8/DdFYAiqKX9ex8ag8vg+0O3atU5fxK/DGdluiecOeyhezejlUqAQN2 O5TAoUadAOXBNdElvx5/NinLyC7z6PlGbftfTQs+x1F046v8irvMyMy X-Developer-Key: i=jpanis@baylibre.com; a=ed25519; pk=8eSM4/xkiHWz2M1Cw1U3m2/YfPbsUdEJPCWY3Mh9ekQ= This patch adds XDP (eXpress Data Path) support to TI AM65 CPSW Ethernet driver. The following features are implemented: - NETDEV_XDP_ACT_BASIC (XDP_PASS, XDP_TX, XDP_DROP, XDP_ABORTED) - NETDEV_XDP_ACT_REDIRECT (XDP_REDIRECT) - NETDEV_XDP_ACT_NDO_XMIT (ndo_xdp_xmit callback) Signed-off-by: Julien Panis --- This patch adds XDP support to TI AM65 CPSW Ethernet driver. The following features are implemented: NETDEV_XDP_ACT_BASIC, NETDEV_XDP_ACT_REDIRECT, and NETDEV_XDP_ACT_NDO_XMIT. Zero-copy and non-linear XDP buffer supports are NOT implemented. --- drivers/net/ethernet/ti/am65-cpsw-nuss.c | 366 +++++++++++++++++++++++++++= ++-- drivers/net/ethernet/ti/am65-cpsw-nuss.h | 9 + 2 files changed, 351 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/ti/am65-cpsw-nuss.c b/drivers/net/etherne= t/ti/am65-cpsw-nuss.c index 9d2f4ac783e4..080910f45629 100644 --- a/drivers/net/ethernet/ti/am65-cpsw-nuss.c +++ b/drivers/net/ethernet/ti/am65-cpsw-nuss.c @@ -5,6 +5,7 @@ * */ =20 +#include #include #include #include @@ -138,6 +139,17 @@ =20 #define AM65_CPSW_DEFAULT_TX_CHNS 8 =20 +/* CPPI streaming packet interface */ +#define AM65_CPSW_CPPI_TX_FLOW_ID 0x3FFF +#define AM65_CPSW_CPPI_TX_PKT_TYPE 0x7 + +/* XDP */ +#define AM65_CPSW_XDP_CONSUMED 1 +#define AM65_CPSW_XDP_PASS 0 + +/* Include headroom compatible with both skb and xdpf */ +#define AM65_CPSW_HEADROOM max(NET_SKB_PAD, XDP_PACKET_HEADROOM) + static void am65_cpsw_port_set_sl_mac(struct am65_cpsw_port *slave, const u8 *dev_addr) { @@ -369,6 +381,30 @@ static void am65_cpsw_init_host_port_emac(struct am65_= cpsw_common *common); static void am65_cpsw_init_port_switch_ale(struct am65_cpsw_port *port); static void am65_cpsw_init_port_emac_ale(struct am65_cpsw_port *port); =20 +static void am65_cpsw_destroy_xdp_rxq(struct am65_cpsw_port *port) +{ + struct xdp_rxq_info *rxq =3D &port->xdp_rxq; + + if (xdp_rxq_info_is_reg(rxq)) + xdp_rxq_info_unreg(rxq); +} + +static int am65_cpsw_create_xdp_rxq(struct am65_cpsw_port *port) +{ + struct xdp_rxq_info *rxq =3D &port->xdp_rxq; + int ret; + + ret =3D xdp_rxq_info_reg(rxq, port->ndev, port->port_id - 1, 0); + if (ret) + return ret; + + ret =3D xdp_rxq_info_reg_mem_model(rxq, MEM_TYPE_PAGE_ORDER0, NULL); + if (ret) + xdp_rxq_info_unreg(rxq); + + return ret; +} + static void am65_cpsw_nuss_rx_cleanup(void *data, dma_addr_t desc_dma) { struct am65_cpsw_rx_chn *rx_chn =3D data; @@ -440,6 +476,27 @@ static void am65_cpsw_nuss_tx_cleanup(void *data, dma_= addr_t desc_dma) dev_kfree_skb_any(skb); } =20 +static struct sk_buff *am65_cpsw_alloc_skb(struct net_device *ndev, unsign= ed int len) +{ + struct page *page; + struct sk_buff *skb; + + page =3D dev_alloc_pages(0); + if (unlikely(!page)) + return NULL; + + len +=3D AM65_CPSW_HEADROOM; + + skb =3D build_skb(page_address(page), len); + if (unlikely(!skb)) + return NULL; + + skb_reserve(skb, AM65_CPSW_HEADROOM + NET_IP_ALIGN); + skb->dev =3D ndev; + + return skb; +} + static int am65_cpsw_nuss_common_open(struct am65_cpsw_common *common) { struct am65_cpsw_host *host_p =3D am65_common_get_host(common); @@ -506,9 +563,7 @@ static int am65_cpsw_nuss_common_open(struct am65_cpsw_= common *common) am65_cpsw_qos_tx_p0_rate_init(common); =20 for (i =3D 0; i < common->rx_chns.descs_num; i++) { - skb =3D __netdev_alloc_skb_ip_align(NULL, - AM65_CPSW_MAX_PACKET_SIZE, - GFP_KERNEL); + skb =3D am65_cpsw_alloc_skb(NULL, AM65_CPSW_MAX_PACKET_SIZE); if (!skb) { ret =3D -ENOMEM; dev_err(common->dev, "cannot allocate skb\n"); @@ -648,6 +703,8 @@ static int am65_cpsw_nuss_ndo_slave_stop(struct net_dev= ice *ndev) =20 phylink_disconnect_phy(port->slave.phylink); =20 + am65_cpsw_destroy_xdp_rxq(port); + ret =3D am65_cpsw_nuss_common_stop(common); if (ret) return ret; @@ -719,6 +776,10 @@ static int am65_cpsw_nuss_ndo_slave_open(struct net_de= vice *ndev) =20 common->usage_count++; =20 + ret =3D am65_cpsw_create_xdp_rxq(port); + if (ret) + goto error_cleanup; + am65_cpsw_port_set_sl_mac(port, ndev->dev_addr); =20 if (common->is_emac_mode) @@ -749,6 +810,138 @@ static int am65_cpsw_nuss_ndo_slave_open(struct net_d= evice *ndev) return ret; } =20 +static int am65_cpsw_xdp_tx_frame(struct net_device *ndev, + struct am65_cpsw_tx_chn *tx_chn, + struct xdp_frame *xdpf) +{ + struct am65_cpsw_common *common =3D am65_ndev_to_common(ndev); + struct am65_cpsw_port *port =3D am65_ndev_to_port(ndev); + struct netdev_queue *netif_txq; + struct cppi5_host_desc_t *host_desc; + dma_addr_t dma_desc, dma_buf; + u32 pkt_len =3D xdpf->len; + void **swdata; + int ret; + + host_desc =3D k3_cppi_desc_pool_alloc(tx_chn->desc_pool); + if (unlikely(!host_desc)) { + ndev->stats.tx_dropped++; + return -ENOMEM; + } + + dma_buf =3D dma_map_single(tx_chn->dma_dev, xdpf->data, pkt_len, DMA_TO_D= EVICE); + if (unlikely(dma_mapping_error(tx_chn->dma_dev, dma_buf))) { + ndev->stats.tx_dropped++; + ret =3D -ENOMEM; + goto pool_free; + } + + cppi5_hdesc_init(host_desc, CPPI5_INFO0_HDESC_EPIB_PRESENT, AM65_CPSW_NAV= _PS_DATA_SIZE); + cppi5_hdesc_set_pkttype(host_desc, AM65_CPSW_CPPI_TX_PKT_TYPE); + cppi5_hdesc_set_pktlen(host_desc, pkt_len); + cppi5_desc_set_pktids(&host_desc->hdr, 0, AM65_CPSW_CPPI_TX_FLOW_ID); + cppi5_desc_set_tags_ids(&host_desc->hdr, 0, port->port_id); + + k3_udma_glue_tx_dma_to_cppi5_addr(tx_chn->tx_chn, &dma_buf); + cppi5_hdesc_attach_buf(host_desc, dma_buf, pkt_len, dma_buf, pkt_len); + + swdata =3D cppi5_hdesc_get_swdata(host_desc); + *(swdata) =3D xdpf; + + /* Report BQL before sending the packet */ + netif_txq =3D netdev_get_tx_queue(ndev, tx_chn->id); + netdev_tx_sent_queue(netif_txq, pkt_len); + + dma_desc =3D k3_cppi_desc_pool_virt2dma(tx_chn->desc_pool, host_desc); + if (AM65_CPSW_IS_CPSW2G(common)) { + ret =3D k3_udma_glue_push_tx_chn(tx_chn->tx_chn, host_desc, dma_desc); + } else { + spin_lock_bh(&tx_chn->lock); + ret =3D k3_udma_glue_push_tx_chn(tx_chn->tx_chn, host_desc, dma_desc); + spin_unlock_bh(&tx_chn->lock); + } + if (ret) { + /* Inform BQL */ + netdev_tx_completed_queue(netif_txq, 1, pkt_len); + ndev->stats.tx_errors++; + goto dma_unmap; + } + + return 0; + +dma_unmap: + k3_udma_glue_tx_cppi5_to_dma_addr(tx_chn->tx_chn, &dma_buf); + dma_unmap_single(tx_chn->dma_dev, dma_buf, pkt_len, DMA_TO_DEVICE); +pool_free: + k3_cppi_desc_pool_free(tx_chn->desc_pool, host_desc); + return ret; +} + +static int am65_cpsw_run_xdp(struct am65_cpsw_port *port, struct xdp_buff = *xdp, int cpu, int *len) +{ + struct net_device *ndev =3D port->ndev; + struct am65_cpsw_tx_chn *tx_chn; + struct netdev_queue *netif_txq; + struct bpf_prog *prog; + struct xdp_frame *xdpf; + struct page *page; + u32 act; + int ret =3D AM65_CPSW_XDP_CONSUMED; + + prog =3D READ_ONCE(port->xdp_prog); + if (!prog) + return AM65_CPSW_XDP_PASS; + + act =3D bpf_prog_run_xdp(prog, xdp); + /* XDP prog might have changed packet data and boundaries */ + *len =3D xdp->data_end - xdp->data; + + switch (act) { + case XDP_PASS: + ret =3D AM65_CPSW_XDP_PASS; + goto out; + case XDP_TX: + tx_chn =3D &am65_ndev_to_common(ndev)->tx_chns[cpu % AM65_CPSW_MAX_TX_QU= EUES]; + netif_txq =3D netdev_get_tx_queue(ndev, tx_chn->id); + + xdpf =3D xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + break; + + __netif_tx_lock(netif_txq, cpu); + tx_chn->buf_type =3D AM65_CPSW_TX_BUF_TYPE_XDP; + ret =3D am65_cpsw_xdp_tx_frame(ndev, tx_chn, xdpf); + __netif_tx_unlock(netif_txq); + if (ret) + break; + + ndev->stats.rx_bytes +=3D *len; + ndev->stats.rx_packets++; + ret =3D AM65_CPSW_XDP_CONSUMED; + goto out; + case XDP_REDIRECT: + if (unlikely(xdp_do_redirect(ndev, xdp, prog))) + break; + + ndev->stats.rx_bytes +=3D *len; + ndev->stats.rx_packets++; + goto out; + default: + bpf_warn_invalid_xdp_action(ndev, prog, act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(ndev, prog, act); + fallthrough; + case XDP_DROP: + ndev->stats.rx_dropped++; + } + + page =3D virt_to_page(xdp->data); + put_page(page); +out: + return ret; +} + static void am65_cpsw_nuss_rx_ts(struct sk_buff *skb, u32 *psdata) { struct skb_shared_hwtstamps *ssh; @@ -795,7 +988,7 @@ static void am65_cpsw_nuss_rx_csum(struct sk_buff *skb,= u32 csum_info) } =20 static int am65_cpsw_nuss_rx_packets(struct am65_cpsw_common *common, - u32 flow_idx) + u32 flow_idx, int cpu) { struct am65_cpsw_rx_chn *rx_chn =3D &common->rx_chns; u32 buf_dma_len, pkt_len, port_id =3D 0, csum_info; @@ -804,11 +997,14 @@ static int am65_cpsw_nuss_rx_packets(struct am65_cpsw= _common *common, struct cppi5_host_desc_t *desc_rx; struct device *dev =3D common->dev; struct sk_buff *skb, *new_skb; + struct xdp_buff xdp; dma_addr_t desc_dma, buf_dma; struct am65_cpsw_port *port; struct net_device *ndev; + struct page *page; void **swdata; u32 *psdata; + int headroom; int ret =3D 0; =20 ret =3D k3_udma_glue_pop_rx_chn(rx_chn->rx_chn, flow_idx, &desc_dma); @@ -851,7 +1047,23 @@ static int am65_cpsw_nuss_rx_packets(struct am65_cpsw= _common *common, =20 k3_cppi_desc_pool_free(rx_chn->desc_pool, desc_rx); =20 - new_skb =3D netdev_alloc_skb_ip_align(ndev, AM65_CPSW_MAX_PACKET_SIZE); + if (port->xdp_prog) { + xdp_init_buff(&xdp, AM65_CPSW_MAX_PACKET_SIZE, &port->xdp_rxq); + + page =3D virt_to_page(skb->data); + xdp_prepare_buff(&xdp, page_address(page), skb_headroom(skb), pkt_len, f= alse); + + ret =3D am65_cpsw_run_xdp(port, &xdp, cpu, &pkt_len); + if (ret !=3D AM65_CPSW_XDP_PASS) + return ret; + + /* Compute additional headroom to be reserved */ + headroom =3D (xdp.data - xdp.data_hard_start) - skb_headroom(skb); + skb_reserve(skb, headroom); + } + + /* Pass skb to netstack if no XDP prog or returned XDP_PASS */ + new_skb =3D am65_cpsw_alloc_skb(ndev, AM65_CPSW_MAX_PACKET_SIZE); if (new_skb) { ndev_priv =3D netdev_priv(ndev); am65_cpsw_nuss_set_offload_fwd_mark(skb, ndev_priv->offload_fwd_mark); @@ -901,6 +1113,7 @@ static int am65_cpsw_nuss_rx_poll(struct napi_struct *= napi_rx, int budget) { struct am65_cpsw_common *common =3D am65_cpsw_napi_to_common(napi_rx); int flow =3D AM65_CPSW_MAX_RX_FLOWS; + int cpu =3D smp_processor_id(); int cur_budget, ret; int num_rx =3D 0; =20 @@ -909,7 +1122,7 @@ static int am65_cpsw_nuss_rx_poll(struct napi_struct *= napi_rx, int budget) cur_budget =3D budget - num_rx; =20 while (cur_budget--) { - ret =3D am65_cpsw_nuss_rx_packets(common, flow); + ret =3D am65_cpsw_nuss_rx_packets(common, flow, cpu); if (ret) break; num_rx++; @@ -938,8 +1151,8 @@ static int am65_cpsw_nuss_rx_poll(struct napi_struct *= napi_rx, int budget) } =20 static struct sk_buff * -am65_cpsw_nuss_tx_compl_packet(struct am65_cpsw_tx_chn *tx_chn, - dma_addr_t desc_dma) +am65_cpsw_nuss_tx_compl_packet_skb(struct am65_cpsw_tx_chn *tx_chn, + dma_addr_t desc_dma) { struct am65_cpsw_ndev_priv *ndev_priv; struct am65_cpsw_ndev_stats *stats; @@ -968,6 +1181,39 @@ am65_cpsw_nuss_tx_compl_packet(struct am65_cpsw_tx_ch= n *tx_chn, return skb; } =20 +static struct xdp_frame * +am65_cpsw_nuss_tx_compl_packet_xdp(struct am65_cpsw_common *common, + struct am65_cpsw_tx_chn *tx_chn, + dma_addr_t desc_dma, + struct net_device **ndev) +{ + struct am65_cpsw_port *port; + struct am65_cpsw_ndev_priv *ndev_priv; + struct am65_cpsw_ndev_stats *stats; + struct cppi5_host_desc_t *desc_tx; + struct xdp_frame *xdpf; + void **swdata; + u32 port_id =3D 0; + + desc_tx =3D k3_cppi_desc_pool_dma2virt(tx_chn->desc_pool, desc_dma); + cppi5_desc_get_tags_ids(&desc_tx->hdr, NULL, &port_id); + swdata =3D cppi5_hdesc_get_swdata(desc_tx); + xdpf =3D *(swdata); + am65_cpsw_nuss_xmit_free(tx_chn, desc_tx); + + port =3D am65_common_get_port(common, port_id); + *ndev =3D port->ndev; + + ndev_priv =3D netdev_priv(*ndev); + stats =3D this_cpu_ptr(ndev_priv->stats); + u64_stats_update_begin(&stats->syncp); + stats->tx_packets++; + stats->tx_bytes +=3D xdpf->len; + u64_stats_update_end(&stats->syncp); + + return xdpf; +} + static void am65_cpsw_nuss_tx_wake(struct am65_cpsw_tx_chn *tx_chn, struct= net_device *ndev, struct netdev_queue *netif_txq) { @@ -994,6 +1240,7 @@ static int am65_cpsw_nuss_tx_compl_packets(struct am65= _cpsw_common *common, unsigned int total_bytes =3D 0; struct net_device *ndev; struct sk_buff *skb; + struct xdp_frame *xdpf; dma_addr_t desc_dma; int res, num_tx =3D 0; =20 @@ -1013,10 +1260,16 @@ static int am65_cpsw_nuss_tx_compl_packets(struct a= m65_cpsw_common *common, break; } =20 - skb =3D am65_cpsw_nuss_tx_compl_packet(tx_chn, desc_dma); - total_bytes =3D skb->len; - ndev =3D skb->dev; - napi_consume_skb(skb, budget); + if (tx_chn->buf_type =3D=3D AM65_CPSW_TX_BUF_TYPE_SKB) { + skb =3D am65_cpsw_nuss_tx_compl_packet_skb(tx_chn, desc_dma); + ndev =3D skb->dev; + total_bytes =3D skb->len; + napi_consume_skb(skb, budget); + } else { + xdpf =3D am65_cpsw_nuss_tx_compl_packet_xdp(common, tx_chn, desc_dma, &= ndev); + total_bytes =3D xdpf->len; + xdp_return_frame(xdpf); + } num_tx++; =20 netif_txq =3D netdev_get_tx_queue(ndev, chn); @@ -1040,6 +1293,7 @@ static int am65_cpsw_nuss_tx_compl_packets_2g(struct = am65_cpsw_common *common, unsigned int total_bytes =3D 0; struct net_device *ndev; struct sk_buff *skb; + struct xdp_frame *xdpf; dma_addr_t desc_dma; int res, num_tx =3D 0; =20 @@ -1057,11 +1311,16 @@ static int am65_cpsw_nuss_tx_compl_packets_2g(struc= t am65_cpsw_common *common, break; } =20 - skb =3D am65_cpsw_nuss_tx_compl_packet(tx_chn, desc_dma); - - ndev =3D skb->dev; - total_bytes +=3D skb->len; - napi_consume_skb(skb, budget); + if (tx_chn->buf_type =3D=3D AM65_CPSW_TX_BUF_TYPE_SKB) { + skb =3D am65_cpsw_nuss_tx_compl_packet_skb(tx_chn, desc_dma); + ndev =3D skb->dev; + total_bytes +=3D skb->len; + napi_consume_skb(skb, budget); + } else { + xdpf =3D am65_cpsw_nuss_tx_compl_packet_xdp(common, tx_chn, desc_dma, &= ndev); + total_bytes +=3D xdpf->len; + xdp_return_frame(xdpf); + } num_tx++; } =20 @@ -1166,6 +1425,8 @@ static netdev_tx_t am65_cpsw_nuss_ndo_slave_xmit(stru= ct sk_buff *skb, tx_chn =3D &common->tx_chns[q_idx]; netif_txq =3D netdev_get_tx_queue(ndev, q_idx); =20 + tx_chn->buf_type =3D AM65_CPSW_TX_BUF_TYPE_SKB; + /* Map the linear buffer */ buf_dma =3D dma_map_single(tx_chn->dma_dev, skb->data, pkt_len, DMA_TO_DEVICE); @@ -1185,8 +1446,8 @@ static netdev_tx_t am65_cpsw_nuss_ndo_slave_xmit(stru= ct sk_buff *skb, =20 cppi5_hdesc_init(first_desc, CPPI5_INFO0_HDESC_EPIB_PRESENT, AM65_CPSW_NAV_PS_DATA_SIZE); - cppi5_desc_set_pktids(&first_desc->hdr, 0, 0x3FFF); - cppi5_hdesc_set_pkttype(first_desc, 0x7); + cppi5_desc_set_pktids(&first_desc->hdr, 0, AM65_CPSW_CPPI_TX_FLOW_ID); + cppi5_hdesc_set_pkttype(first_desc, AM65_CPSW_CPPI_TX_PKT_TYPE); cppi5_desc_set_tags_ids(&first_desc->hdr, 0, port->port_id); =20 k3_udma_glue_tx_dma_to_cppi5_addr(tx_chn->tx_chn, &buf_dma); @@ -1488,6 +1749,58 @@ static void am65_cpsw_nuss_ndo_get_stats(struct net_= device *dev, stats->tx_dropped =3D dev->stats.tx_dropped; } =20 +static int am65_cpsw_xdp_prog_setup(struct net_device *ndev, struct bpf_pr= og *prog) +{ + struct am65_cpsw_port *port =3D am65_ndev_to_port(ndev); + struct bpf_prog *old_prog; + bool running =3D netif_running(ndev); + + if (running) + am65_cpsw_nuss_ndo_slave_stop(ndev); + + old_prog =3D xchg(&port->xdp_prog, prog); + if (old_prog) + bpf_prog_put(old_prog); + + if (running) + return am65_cpsw_nuss_ndo_slave_open(ndev); + + return 0; +} + +static int am65_cpsw_ndo_bpf(struct net_device *ndev, struct netdev_bpf *b= pf) +{ + switch (bpf->command) { + case XDP_SETUP_PROG: + return am65_cpsw_xdp_prog_setup(ndev, bpf->prog); + default: + return -EINVAL; + } +} + +static int am65_cpsw_ndo_xdp_xmit(struct net_device *ndev, int n, + struct xdp_frame **frames, u32 flags) +{ + struct am65_cpsw_tx_chn *tx_chn; + struct netdev_queue *netif_txq; + int cpu =3D smp_processor_id(); + int i, nxmit =3D 0; + + tx_chn =3D &am65_ndev_to_common(ndev)->tx_chns[cpu % AM65_CPSW_MAX_TX_QUE= UES]; + netif_txq =3D netdev_get_tx_queue(ndev, tx_chn->id); + + __netif_tx_lock(netif_txq, cpu); + tx_chn->buf_type =3D AM65_CPSW_TX_BUF_TYPE_XDP; + for (i =3D 0; i < n; i++) { + if (am65_cpsw_xdp_tx_frame(ndev, tx_chn, frames[i])) + break; + nxmit++; + } + __netif_tx_unlock(netif_txq); + + return nxmit; +} + static const struct net_device_ops am65_cpsw_nuss_netdev_ops =3D { .ndo_open =3D am65_cpsw_nuss_ndo_slave_open, .ndo_stop =3D am65_cpsw_nuss_ndo_slave_stop, @@ -1502,6 +1815,8 @@ static const struct net_device_ops am65_cpsw_nuss_net= dev_ops =3D { .ndo_eth_ioctl =3D am65_cpsw_nuss_ndo_slave_ioctl, .ndo_setup_tc =3D am65_cpsw_qos_ndo_setup_tc, .ndo_set_tx_maxrate =3D am65_cpsw_qos_ndo_tx_p0_set_maxrate, + .ndo_bpf =3D am65_cpsw_ndo_bpf, + .ndo_xdp_xmit =3D am65_cpsw_ndo_xdp_xmit, }; =20 static void am65_cpsw_disable_phy(struct phy *phy) @@ -1816,6 +2131,8 @@ static int am65_cpsw_nuss_init_tx_chns(struct am65_cp= sw_common *common) goto err; } =20 + tx_chn->buf_type =3D AM65_CPSW_TX_BUF_TYPE_SKB; + tx_chn->irq =3D k3_udma_glue_tx_get_irq(tx_chn->tx_chn); if (tx_chn->irq < 0) { dev_err(dev, "Failed to get tx dma irq %d\n", @@ -1873,11 +2190,7 @@ static void am65_cpsw_nuss_remove_rx_chns(void *data) =20 netif_napi_del(&common->napi_rx); =20 - if (!IS_ERR_OR_NULL(rx_chn->desc_pool)) - k3_cppi_desc_pool_destroy(rx_chn->desc_pool); - - if (!IS_ERR_OR_NULL(rx_chn->rx_chn)) - k3_udma_glue_release_rx_chn(rx_chn->rx_chn); + am65_cpsw_nuss_free_rx_chns(common); =20 common->rx_flow_id_base =3D -1; } @@ -2252,6 +2565,9 @@ am65_cpsw_nuss_init_port_ndev(struct am65_cpsw_common= *common, u32 port_idx) NETIF_F_HW_TC; port->ndev->features =3D port->ndev->hw_features | NETIF_F_HW_VLAN_CTAG_FILTER; + port->ndev->xdp_features =3D NETDEV_XDP_ACT_BASIC | + NETDEV_XDP_ACT_REDIRECT | + NETDEV_XDP_ACT_NDO_XMIT; port->ndev->vlan_features |=3D NETIF_F_SG; port->ndev->netdev_ops =3D &am65_cpsw_nuss_netdev_ops; port->ndev->ethtool_ops =3D &am65_cpsw_ethtool_ops_slave; @@ -2315,6 +2631,8 @@ am65_cpsw_nuss_init_port_ndev(struct am65_cpsw_common= *common, u32 port_idx) if (ret) dev_err(dev, "failed to add percpu stat free action %d\n", ret); =20 + port->xdp_prog =3D NULL; + if (!common->dma_ndev) common->dma_ndev =3D port->ndev; =20 diff --git a/drivers/net/ethernet/ti/am65-cpsw-nuss.h b/drivers/net/etherne= t/ti/am65-cpsw-nuss.h index 7da0492dc091..6fbb975427df 100644 --- a/drivers/net/ethernet/ti/am65-cpsw-nuss.h +++ b/drivers/net/ethernet/ti/am65-cpsw-nuss.h @@ -14,6 +14,7 @@ #include #include #include +#include #include "am65-cpsw-qos.h" =20 struct am65_cpts; @@ -56,10 +57,17 @@ struct am65_cpsw_port { bool rx_ts_enabled; struct am65_cpsw_qos qos; struct devlink_port devlink_port; + struct bpf_prog *xdp_prog; + struct xdp_rxq_info xdp_rxq; /* Only for suspend resume context */ u32 vid_context; }; =20 +enum am65_cpsw_tx_buf_type { + AM65_CPSW_TX_BUF_TYPE_SKB, + AM65_CPSW_TX_BUF_TYPE_XDP, +}; + struct am65_cpsw_host { struct am65_cpsw_common *common; void __iomem *port_base; @@ -74,6 +82,7 @@ struct am65_cpsw_tx_chn { struct am65_cpsw_common *common; struct k3_cppi_desc_pool *desc_pool; struct k3_udma_glue_tx_channel *tx_chn; + enum am65_cpsw_tx_buf_type buf_type; spinlock_t lock; /* protect TX rings in multi-port mode */ struct hrtimer tx_hrtimer; unsigned long tx_pace_timeout; --- base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d change-id: 20240223-am65-cpsw-xdp-basic-4db828508b48 Best regards, --=20 Julien Panis