From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8660E313E10; Tue, 26 Aug 2025 16:13:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224813; cv=none; b=fbtIpqlqMuF5wGUNF20KhiTFolKuRSUxOVbGO5ULCUEzkfaquxjyhS0nQgbw+s7YgQ1U8wZMkjr3fLDHH8RGpMrDxC9OFU++O2VQ9oFLid7qsZCvmvIDuFTNWeR165sDuLhKW3JLFlIlDnSrKSe8Mo9KetyIga2GpgqDofVih08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224813; c=relaxed/simple; bh=3azXjeLlMdczMc3oGM/koHv9jY0rlImJEFzdohW82Ig=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fD4o0rz6WjfOomGxOHFg6+F/anbauk6yRmKyWVWjfDVHJ2j1qzLCWwB3tu4qWmebI43W9cghBspjYzAx9OLFporRk0kbniFj0JGgcFoHS6dNXC/YToeBZ19l4utnj52B8h8b9/sL8hVPIBV0ZYAli0+DIh65wNMflGyisaJLnPs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=S9Zdm4s0; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="S9Zdm4s0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224812; x=1787760812; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3azXjeLlMdczMc3oGM/koHv9jY0rlImJEFzdohW82Ig=; b=S9Zdm4s0Fhj2pB/83ARCsT6c8CWEOIBY1jIustSHi2VyCB6+bMjk6uGm XIk8fJZQS4QzOfLQUhVOmhevt2jlEjV2jUnCy0Nu/Rp+GChz1QyWyoW+W zpsXdAQ0sqKBLcqnmo25zIg1/q6PRrgPYs4rpndkU4sh6x/4uyjgUB4K9 ARrzA8BX73wohOIQCfbHacCKw37k1uMjj6hRNADzQwPw7LnzwHvYEtPPJ djUweut6zDrpqvUi1IishVSAXaYEg1cMVqz6GjTqp4rQ6VGt3clY+YL/m 7LsTdaJU1EmG1sT02541ilgvpZRw1pVFhF+pbhlufEu9iTNJEQl0bHqlq g==; X-CSE-ConnectionGUID: aQycQlvhTRae5OM+7YenUg== X-CSE-MsgGUID: ChzgUkBNQeqthO/td0EKAQ== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46044867" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46044867" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:13:31 -0700 X-CSE-ConnectionGUID: R7x4PETyTROq9x4CHM+oiw== X-CSE-MsgGUID: fBqtNADmRz6vnUp9R/eIyw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562037" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:27 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 01/13] xdp, libeth: make the xdp_init_buff() micro-optimization generic Date: Tue, 26 Aug 2025 17:54:55 +0200 Message-ID: <20250826155507.2138401-2-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Often times the compilers are not able to expand two consecutive 32-bit writes into one 64-bit on the corresponding architectures. This applies to xdp_init_buff() called for every received frame (or at least once per each 64 frames when the frag size is fixed). Move the not-so-pretty hack from libeth_xdp straight to xdp_init_buff(), but using a proper union around ::frame_sz and ::flags. The optimization is limited to LE architectures due to the structure layout. One simple example from idpf with the XDP series applied (Clang 22-git, CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE =3D> -O2): add/remove: 0/0 grow/shrink: 0/1 up/down: 0/-27 (-27) Function old new delta idpf_vport_splitq_napi_poll 5076 5049 -27 The perf difference with XDP_DROP is around +0.8-1% which I see as more than satisfying. Suggested-by: Simon Horman Signed-off-by: Alexander Lobakin --- include/net/libeth/xdp.h | 11 +---------- include/net/xdp.h | 28 +++++++++++++++++++++++++--- 2 files changed, 26 insertions(+), 13 deletions(-) diff --git a/include/net/libeth/xdp.h b/include/net/libeth/xdp.h index f4880b50e804..bc3507edd589 100644 --- a/include/net/libeth/xdp.h +++ b/include/net/libeth/xdp.h @@ -1274,7 +1274,6 @@ bool libeth_xdp_buff_add_frag(struct libeth_xdp_buff = *xdp, * Internal, use libeth_xdp_process_buff() instead. Initializes XDP buffer * head with the Rx buffer data: data pointer, length, headroom, and * truesize/tailroom. Zeroes the flags. - * Uses faster single u64 write instead of per-field access. */ static inline void libeth_xdp_prepare_buff(struct libeth_xdp_buff *xdp, const struct libeth_fqe *fqe, @@ -1282,17 +1281,9 @@ static inline void libeth_xdp_prepare_buff(struct li= beth_xdp_buff *xdp, { const struct page *page =3D __netmem_to_page(fqe->netmem); =20 -#ifdef __LIBETH_WORD_ACCESS - static_assert(offsetofend(typeof(xdp->base), flags) - - offsetof(typeof(xdp->base), frame_sz) =3D=3D - sizeof(u64)); - - *(u64 *)&xdp->base.frame_sz =3D fqe->truesize; -#else - xdp_init_buff(&xdp->base, fqe->truesize, xdp->base.rxq); -#endif xdp_prepare_buff(&xdp->base, page_address(page) + fqe->offset, pp_page_to_nmdesc(page)->pp->p.offset, len, true); + xdp_init_buff(&xdp->base, fqe->truesize, xdp->base.rxq); } =20 /** diff --git a/include/net/xdp.h b/include/net/xdp.h index b40f1f96cb11..af60e11b336c 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -85,8 +85,20 @@ struct xdp_buff { void *data_hard_start; struct xdp_rxq_info *rxq; struct xdp_txq_info *txq; - u32 frame_sz; /* frame size to deduce data_hard_end/reserved tailroom*/ - u32 flags; /* supported values defined in xdp_buff_flags */ + + union { + struct { + /* frame size to deduce data_hard_end/tailroom */ + u32 frame_sz; + /* supported values defined in xdp_buff_flags */ + u32 flags; + }; + +#ifdef __LITTLE_ENDIAN + /* Used to micro-optimize xdp_init_buff(), don't use directly */ + u64 frame_sz_flags_init; +#endif + }; }; =20 static __always_inline bool xdp_buff_has_frags(const struct xdp_buff *xdp) @@ -118,9 +130,19 @@ static __always_inline void xdp_buff_set_frag_pfmemall= oc(struct xdp_buff *xdp) static __always_inline void xdp_init_buff(struct xdp_buff *xdp, u32 frame_sz, struct xdp_rxq_info *rxq) { - xdp->frame_sz =3D frame_sz; xdp->rxq =3D rxq; + +#ifdef __LITTLE_ENDIAN + /* + * Force the compilers to initialize ::flags and assign ::frame_sz with + * one write on 64-bit LE architectures as they're often unable to do + * it themselves. + */ + xdp->frame_sz_flags_init =3D frame_sz; +#else + xdp->frame_sz =3D frame_sz; xdp->flags =3D 0; +#endif } =20 static __always_inline void --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B600A341AA5; Tue, 26 Aug 2025 16:13:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224817; cv=none; b=Dr4Po913BVxdnRapI1j3TG9KvA8kKBjQK4Oz2lDh7LKlTRj7S3DlYQWEf71fX+/nNbwAGPVhORgOPt8LmNV5v+uuovCT+WQ+lKeqacjivaChNRvDngj4wDukOyyMcy5S0AHCTNG7dq8X+MfHjg2bHKIONyGFxKyndk1Fk0N+OmQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224817; c=relaxed/simple; bh=bUNbTnj1hQFmwxgQyqBf0WIlVzWXySVKeBksI4xda0g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o7DCutVE0wTiSLdsBsBW4sGJ6RzD72M3SJzmiAzxWyRad473YBnOUfQGfNwna/kOicgbPO2kzeUurnadEwtJ+2V+AL8iHCGvMulPrb836ZFLFjHEBaT4fyTJEcCekx3HziQ6o44v6b4TDV7K18ZCHXmHvwkdNo9FkgzoeQwEkew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=eDcewxUm; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="eDcewxUm" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224816; x=1787760816; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bUNbTnj1hQFmwxgQyqBf0WIlVzWXySVKeBksI4xda0g=; b=eDcewxUmdnivN7YH95A8zdQJgsKesSsZWbjdWQqvCRhjvZ//oIki4STs I7DJJ6EggukZ38LvtwKWABBQaMHKftWhgsX9d8eRY0ZClhd5192ey1z3J FqY6rf10bu4tqQtvh/b2wx7ooiLpZ0hMk8ubztDiK99DbSROi/JQCIdy6 rglvMJBbkJp3vHOoZG5OBKdnX4uv7LuaLef75IlQsu4X5Dwa6xuktjIxx TEoQc59HsldIpghkqtkJkqEclY1jQ149iWTKjAjnfTbKpIOXVI9D49ADi 26seDWcqAWCmTSo9gt8q1duvT1XFYM1A7vXAz95GZ0Vj8Z7onM1B/BmoB A==; X-CSE-ConnectionGUID: uLoATYd7Re2wY1qdAD1pMA== X-CSE-MsgGUID: oH1yM/MTQbWGThCulubkhg== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46044888" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46044888" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:13:35 -0700 X-CSE-ConnectionGUID: 1ED7XHirReKYL0P1+ufTxw== X-CSE-MsgGUID: cGM/nvP0RFqi7q8kd0aGbg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562083" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:31 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 02/13] idpf: fix Rx descriptor ready check barrier in splitq Date: Tue, 26 Aug 2025 17:54:56 +0200 Message-ID: <20250826155507.2138401-3-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No idea what the current barrier position was meant for. At that point, nothing is read from the descriptor, only the pointer to the actual one is fetched. The correct barrier usage here is after the generation check, so that only the first qword is read if the descriptor is not yet ready and we need to stop polling. Debatable on coherent DMA as the Rx descriptor size is <=3D cacheline size, but anyway, the current barrier position only makes the codegen worse. Fixes: 3a8845af66ed ("idpf: add RX splitq napi poll support") Reviewed-by: Maciej Fijalkowski Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index 600bd83ae3fa..bbbbcd885fcf 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -3101,18 +3101,14 @@ static int idpf_rx_splitq_clean(struct idpf_rx_queu= e *rxq, int budget) /* get the Rx desc from Rx queue based on 'next_to_clean' */ rx_desc =3D &rxq->rx[ntc].flex_adv_nic_3_wb; =20 - /* This memory barrier is needed to keep us from reading - * any other fields out of the rx_desc - */ - dma_rmb(); - /* if the descriptor isn't done, no work yet to do */ gen_id =3D le16_get_bits(rx_desc->pktlen_gen_bufq_id, VIRTCHNL2_RX_FLEX_DESC_ADV_GEN_M); - if (idpf_queue_has(GEN_CHK, rxq) !=3D gen_id) break; =20 + dma_rmb(); + rxdid =3D FIELD_GET(VIRTCHNL2_RX_FLEX_DESC_ADV_RXDID_M, rx_desc->rxdid_ucast); if (rxdid !=3D VIRTCHNL2_RXDID_2_FLEX_SPLITQ) { --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3607E30FC19; Tue, 26 Aug 2025 16:13:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224822; cv=none; b=Dh/Ozy4+cuOm1Dt0FCqr195Ip7dlx9VGS9aK5Iqi87NqluaqI0/UmgiwVq2tmp3YqthLYI2RA9L7OItTxWojzRAyeUCh0KuWZY53nu47vil+1bSzD9MDwbESM8BghX8b/gw9yjX48doKnIyhq81tF9gfPLudx3pBbVitQ+I7Ats= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224822; c=relaxed/simple; bh=Ax5V+VbZ9Mzo4Z+LR7bLywE/lpkNFmaZCu9N513RCRg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O3gEBQEaAkHGsx1MRV+rCjR3mtVB+MzFiulQtH/Jxxz/Ezsz90seZfOJB/tg0nh+6w805xwniBGluIj8754fftItyuHDu8D+adrEaAY4bjDHctl8ZV76vuNwu9vWzjgIl6cds5ySGnxuvGa+wpGEdTl66WY4Xdrhekir7W8Zc2c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Q+B+PCd3; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Q+B+PCd3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224820; x=1787760820; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ax5V+VbZ9Mzo4Z+LR7bLywE/lpkNFmaZCu9N513RCRg=; b=Q+B+PCd3EvO+VJYCRU9iRUj3Fd6I3j7Igur6/mbfT5Aci2ebcDP9L+Ka jzewYX9y9/QgGUbP6NFOkvsCNDYTjkyJanPcN9rgyFDwFpHZKqyxmARPo fwpi4xL9qn4NBlbipFIWul0N7Z4Tyw8ai8Eg7ktYTozpOCylHwWM/w49n wQkNG58E8yHN/uTr2s6a3gb/Gp4cjAWTit74vXXnwf8EVfW5V75c/5FPd mNiI86i5I9WIUu1fcAGHpzrRRA8EwWWrzxfbSqQOQDNmqqfkmO7m+5SJx es4SVJnkoP+stH8cpD1eN98WQu480RmpvvAUgWMyGabxEkOdolEE5lFDC w==; X-CSE-ConnectionGUID: 3p9lumLDSmG0DLI5yJvrpw== X-CSE-MsgGUID: AtHbUUqEQ9yvE1D2QO2ENQ== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46044912" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46044912" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:13:40 -0700 X-CSE-ConnectionGUID: FZTdYC/SQ7ytmMqxd0fNRg== X-CSE-MsgGUID: 4OjN21gCStmhXHqLtUPsgw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562125" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:35 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 03/13] idpf: use a saner limit for default number of queues to allocate Date: Tue, 26 Aug 2025 17:54:57 +0200 Message-ID: <20250826155507.2138401-4-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the maximum number of queues available for one vport is 16. This is hardcoded, but then the function calculating the optimal number of queues takes min(16, num_online_cpus()). In order to be able to allocate more queues, which will be then used for XDP, stop hardcoding 16 and rely on what the device gives us[*]. Instead of num_online_cpus(), which is considered suboptimal since at least 2013, use netif_get_num_default_rss_queues() to still have free queues in the pool. [*] With the note: Currently, idpf always allocates `IDPF_MAX_BUFQS_PER_RXQ_GRP` (=3D=3D 2) buffer queues for each Rx queue and one completion queue for each Tx for best performance. But there was no check whether such number is availabe, IOW the assumption was not backed by any "harmonizing" / actual checks. Fix this while at it. nr_cpu_ids number of Tx queues are needed only for lockless XDP sending, the regular stack doesn't benefit from that anyhow. On a 128-thread Xeon, this now gives me 32 regular Tx queues and leaves 224 free for XDP (128 of which will handle XDP_TX, .ndo_xdp_xmit(), and XSk xmit when enabled). Note 2: Unfortunately, some CP/FW versions are not able to reconfigure/enable/disable large amount of queues within the minimum timeout (2 seconds). For now, fall back to the default timeout for every operation until this is resolved. Signed-off-by: Alexander Lobakin --- .../net/ethernet/intel/idpf/idpf_virtchnl.h | 1 - drivers/net/ethernet/intel/idpf/idpf_txrx.c | 8 +-- .../net/ethernet/intel/idpf/idpf_virtchnl.c | 62 +++++++++++-------- 3 files changed, 38 insertions(+), 33 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h b/drivers/net/= ethernet/intel/idpf/idpf_virtchnl.h index 86f30f0db07a..d714ff0eaca0 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h +++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h @@ -4,7 +4,6 @@ #ifndef _IDPF_VIRTCHNL_H_ #define _IDPF_VIRTCHNL_H_ =20 -#define IDPF_VC_XN_MIN_TIMEOUT_MSEC 2000 #define IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC (60 * 1000) #define IDPF_VC_XN_IDX_M GENMASK(7, 0) #define IDPF_VC_XN_SALT_M GENMASK(15, 8) diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index bbbbcd885fcf..f49791eab07d 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -1175,13 +1175,7 @@ int idpf_vport_calc_total_qs(struct idpf_adapter *ad= apter, u16 vport_idx, num_req_tx_qs =3D vport_config->user_config.num_req_tx_qs; num_req_rx_qs =3D vport_config->user_config.num_req_rx_qs; } else { - int num_cpus; - - /* Restrict num of queues to cpus online as a default - * configuration to give best performance. User can always - * override to a max number of queues via ethtool. - */ - num_cpus =3D num_online_cpus(); + u32 num_cpus =3D netif_get_num_default_rss_queues(); =20 dflt_splitq_txq_grps =3D min_t(int, max_q->max_txq, num_cpus); dflt_singleq_txqs =3D min_t(int, max_q->max_txq, num_cpus); diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c b/drivers/net/= ethernet/intel/idpf/idpf_virtchnl.c index e60438633cc4..6164094c6ae5 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c @@ -1061,21 +1061,35 @@ int idpf_vport_alloc_max_qs(struct idpf_adapter *ad= apter, struct idpf_avail_queue_info *avail_queues =3D &adapter->avail_queues; struct virtchnl2_get_capabilities *caps =3D &adapter->caps; u16 default_vports =3D idpf_get_default_vports(adapter); - int max_rx_q, max_tx_q; + u32 max_rx_q, max_tx_q, max_buf_q, max_compl_q; =20 mutex_lock(&adapter->queue_lock); =20 + /* Caps are device-wide. Give each vport an equal piece */ max_rx_q =3D le16_to_cpu(caps->max_rx_q) / default_vports; max_tx_q =3D le16_to_cpu(caps->max_tx_q) / default_vports; - if (adapter->num_alloc_vports < default_vports) { - max_q->max_rxq =3D min_t(u16, max_rx_q, IDPF_MAX_Q); - max_q->max_txq =3D min_t(u16, max_tx_q, IDPF_MAX_Q); - } else { - max_q->max_rxq =3D IDPF_MIN_Q; - max_q->max_txq =3D IDPF_MIN_Q; + max_buf_q =3D le16_to_cpu(caps->max_rx_bufq) / default_vports; + max_compl_q =3D le16_to_cpu(caps->max_tx_complq) / default_vports; + + if (adapter->num_alloc_vports >=3D default_vports) { + max_rx_q =3D IDPF_MIN_Q; + max_tx_q =3D IDPF_MIN_Q; } - max_q->max_bufq =3D max_q->max_rxq * IDPF_MAX_BUFQS_PER_RXQ_GRP; - max_q->max_complq =3D max_q->max_txq; + + /* + * Harmonize the numbers. The current implementation always creates + * `IDPF_MAX_BUFQS_PER_RXQ_GRP` buffer queues for each Rx queue and + * one completion queue for each Tx queue for best performance. + * If less buffer or completion queues is available, cap the number + * of the corresponding Rx/Tx queues. + */ + max_rx_q =3D min(max_rx_q, max_buf_q / IDPF_MAX_BUFQS_PER_RXQ_GRP); + max_tx_q =3D min(max_tx_q, max_compl_q); + + max_q->max_rxq =3D max_rx_q; + max_q->max_txq =3D max_tx_q; + max_q->max_bufq =3D max_rx_q * IDPF_MAX_BUFQS_PER_RXQ_GRP; + max_q->max_complq =3D max_tx_q; =20 if (avail_queues->avail_rxq < max_q->max_rxq || avail_queues->avail_txq < max_q->max_txq || @@ -1506,7 +1520,7 @@ int idpf_send_destroy_vport_msg(struct idpf_vport *vp= ort) xn_params.vc_op =3D VIRTCHNL2_OP_DESTROY_VPORT; xn_params.send_buf.iov_base =3D &v_id; xn_params.send_buf.iov_len =3D sizeof(v_id); - xn_params.timeout_ms =3D IDPF_VC_XN_MIN_TIMEOUT_MSEC; + xn_params.timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC; reply_sz =3D idpf_vc_xn_exec(vport->adapter, &xn_params); =20 return reply_sz < 0 ? reply_sz : 0; @@ -1554,7 +1568,7 @@ int idpf_send_disable_vport_msg(struct idpf_vport *vp= ort) xn_params.vc_op =3D VIRTCHNL2_OP_DISABLE_VPORT; xn_params.send_buf.iov_base =3D &v_id; xn_params.send_buf.iov_len =3D sizeof(v_id); - xn_params.timeout_ms =3D IDPF_VC_XN_MIN_TIMEOUT_MSEC; + xn_params.timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC; reply_sz =3D idpf_vc_xn_exec(vport->adapter, &xn_params); =20 return reply_sz < 0 ? reply_sz : 0; @@ -1845,7 +1859,9 @@ static int idpf_send_ena_dis_queues_msg(struct idpf_v= port *vport, bool ena) struct virtchnl2_del_ena_dis_queues *eq __free(kfree) =3D NULL; struct virtchnl2_queue_chunk *qc __free(kfree) =3D NULL; u32 num_msgs, num_chunks, num_txq, num_rxq, num_q; - struct idpf_vc_xn_params xn_params =3D {}; + struct idpf_vc_xn_params xn_params =3D { + .timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC, + }; struct virtchnl2_queue_chunks *qcs; u32 config_sz, chunk_sz, buf_sz; ssize_t reply_sz; @@ -1946,13 +1962,10 @@ static int idpf_send_ena_dis_queues_msg(struct idpf= _vport *vport, bool ena) if (!eq) return -ENOMEM; =20 - if (ena) { + if (ena) xn_params.vc_op =3D VIRTCHNL2_OP_ENABLE_QUEUES; - xn_params.timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC; - } else { + else xn_params.vc_op =3D VIRTCHNL2_OP_DISABLE_QUEUES; - xn_params.timeout_ms =3D IDPF_VC_XN_MIN_TIMEOUT_MSEC; - } =20 for (i =3D 0, k =3D 0; i < num_msgs; i++) { memset(eq, 0, buf_sz); @@ -1990,7 +2003,9 @@ int idpf_send_map_unmap_queue_vector_msg(struct idpf_= vport *vport, bool map) { struct virtchnl2_queue_vector_maps *vqvm __free(kfree) =3D NULL; struct virtchnl2_queue_vector *vqv __free(kfree) =3D NULL; - struct idpf_vc_xn_params xn_params =3D {}; + struct idpf_vc_xn_params xn_params =3D { + .timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC, + }; u32 config_sz, chunk_sz, buf_sz; u32 num_msgs, num_chunks, num_q; ssize_t reply_sz; @@ -2074,13 +2089,10 @@ int idpf_send_map_unmap_queue_vector_msg(struct idp= f_vport *vport, bool map) if (!vqvm) return -ENOMEM; =20 - if (map) { + if (map) xn_params.vc_op =3D VIRTCHNL2_OP_MAP_QUEUE_VECTOR; - xn_params.timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC; - } else { + else xn_params.vc_op =3D VIRTCHNL2_OP_UNMAP_QUEUE_VECTOR; - xn_params.timeout_ms =3D IDPF_VC_XN_MIN_TIMEOUT_MSEC; - } =20 for (i =3D 0, k =3D 0; i < num_msgs; i++) { memset(vqvm, 0, buf_sz); @@ -2207,7 +2219,7 @@ int idpf_send_delete_queues_msg(struct idpf_vport *vp= ort) num_chunks); =20 xn_params.vc_op =3D VIRTCHNL2_OP_DEL_QUEUES; - xn_params.timeout_ms =3D IDPF_VC_XN_MIN_TIMEOUT_MSEC; + xn_params.timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC; xn_params.send_buf.iov_base =3D eq; xn_params.send_buf.iov_len =3D buf_size; reply_sz =3D idpf_vc_xn_exec(vport->adapter, &xn_params); @@ -2371,7 +2383,7 @@ int idpf_send_dealloc_vectors_msg(struct idpf_adapter= *adapter) xn_params.vc_op =3D VIRTCHNL2_OP_DEALLOC_VECTORS; xn_params.send_buf.iov_base =3D vcs; xn_params.send_buf.iov_len =3D buf_size; - xn_params.timeout_ms =3D IDPF_VC_XN_MIN_TIMEOUT_MSEC; + xn_params.timeout_ms =3D IDPF_VC_XN_DEFAULT_TIMEOUT_MSEC; reply_sz =3D idpf_vc_xn_exec(adapter, &xn_params); if (reply_sz < 0) return reply_sz; --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93DD931A54D; Tue, 26 Aug 2025 16:13:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224826; cv=none; b=BnJyQAbsIQz0cS61p8dGob5X05xH+SYpE+JJo2UN+sY+nQnXBY2GdXLhSy7JmtwESOG9UQk9EGqX90jaZjyZwYQAXdR1XcRcQWCwY801fAmMs+Psk9i999WfIWGbxoITnJAHrVoIoS1Dsi9nVOO1XQk37wbEwfUT4EeixdJBOAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224826; c=relaxed/simple; bh=o+1HV0j8Kaso9cXniUzmaJsDKPTGfWFDT3rQDUMmpio=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qF6aPYP0p58w0Kk5ENqjw2G9X4hiVLYGx3247NUH5xEoXg8t6kXluDuQSPQFtaWUHaNfMsG4JlqKpxjWeDDRP0ihSbzvcPcUJVA/66Uo+pYpVzUZ2FyyFBTwNYj4tHiiLpeiS3MaYKlkq9UNW0KTs5zEFA05ZOSbgcRJicchcdI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=bpvkgPHJ; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="bpvkgPHJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224825; x=1787760825; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=o+1HV0j8Kaso9cXniUzmaJsDKPTGfWFDT3rQDUMmpio=; b=bpvkgPHJftRxzSS9YWwzXM+HHTwPm6/xhXkbG6ErJ9MAGxFgjy4M5RvN hJ4yZA/mrQBOoi6ML2rivPAgscer8vndFazXbO3n0SIvSFPE4aKQi34y0 k2g7HZb7OLcBl9hBGkdTZM7loFcYYiGCgObXwd+EeLD2bhNcQ66AzK64I BxRTOQL3KbS+fhGbafx0NZ4FWkWw2WnWi3bczSWow2+CdwIphJzzWgehU Q7OXPiI7hx78tjWSoEb97V9cel/wezuXskdsC7Hul5Pdc4zg3u+RnUTAQ D1s+W0EAtVm4l/zxLKvTjX++JUSyQFaC/HyToO5Y8V3sXYU2PnX6x7yVD w==; X-CSE-ConnectionGUID: vI4hS26WSuaM0h8DZ5Bcag== X-CSE-MsgGUID: 8GkoBUDoT/2X3PGoOTyGTg== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46044923" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46044923" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:13:44 -0700 X-CSE-ConnectionGUID: bB5BXYTDSuq5F86T+eWFcQ== X-CSE-MsgGUID: EOPxBbbAS4a01YcMlJX5Yg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562149" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:40 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 04/13] idpf: link NAPIs to queues Date: Tue, 26 Aug 2025 17:54:58 +0200 Message-ID: <20250826155507.2138401-5-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the missing linking of NAPIs to netdev queues when enabling interrupt vectors in order to support NAPI configuration and interfaces requiring get_rx_queue()->napi to be set (like XSk busy polling). As currently, idpf_vport_{start,stop}() is called from several flows with inconsistent RTNL locking, we need to synchronize them to avoid runtime assertions. Notably: * idpf_{open,stop}() -- regular NDOs, RTNL is always taken; * idpf_initiate_soft_reset() -- usually called under RTNL; * idpf_init_task -- called from the init work, needs RTNL; * idpf_vport_dealloc -- called without RTNL taken, needs it. Expand common idpf_vport_{start,stop}() to take an additional bool telling whether we need to manually take the RTNL lock. Suggested-by: Maciej Fijalkowski # helper Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/idpf_lib.c | 38 +++++++++++++++------ drivers/net/ethernet/intel/idpf/idpf_txrx.c | 17 +++++++++ 2 files changed, 45 insertions(+), 10 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ether= net/intel/idpf/idpf_lib.c index 67236a68f6be..b5a7215488b9 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -884,14 +884,18 @@ static void idpf_remove_features(struct idpf_vport *v= port) /** * idpf_vport_stop - Disable a vport * @vport: vport to disable + * @rtnl: whether to take RTNL lock */ -static void idpf_vport_stop(struct idpf_vport *vport) +static void idpf_vport_stop(struct idpf_vport *vport, bool rtnl) { struct idpf_netdev_priv *np =3D netdev_priv(vport->netdev); =20 if (np->state <=3D __IDPF_VPORT_DOWN) return; =20 + if (rtnl) + rtnl_lock(); + netif_carrier_off(vport->netdev); netif_tx_disable(vport->netdev); =20 @@ -913,6 +917,9 @@ static void idpf_vport_stop(struct idpf_vport *vport) idpf_vport_queues_rel(vport); idpf_vport_intr_rel(vport); np->state =3D __IDPF_VPORT_DOWN; + + if (rtnl) + rtnl_unlock(); } =20 /** @@ -936,7 +943,7 @@ static int idpf_stop(struct net_device *netdev) idpf_vport_ctrl_lock(netdev); vport =3D idpf_netdev_to_vport(netdev); =20 - idpf_vport_stop(vport); + idpf_vport_stop(vport, false); =20 idpf_vport_ctrl_unlock(netdev); =20 @@ -1029,7 +1036,7 @@ static void idpf_vport_dealloc(struct idpf_vport *vpo= rt) idpf_idc_deinit_vport_aux_device(vport->vdev_info); =20 idpf_deinit_mac_addr(vport); - idpf_vport_stop(vport); + idpf_vport_stop(vport, true); =20 if (!test_bit(IDPF_HR_RESET_IN_PROG, adapter->flags)) idpf_decfg_netdev(vport); @@ -1370,8 +1377,9 @@ static void idpf_rx_init_buf_tail(struct idpf_vport *= vport) /** * idpf_vport_open - Bring up a vport * @vport: vport to bring up + * @rtnl: whether to take RTNL lock */ -static int idpf_vport_open(struct idpf_vport *vport) +static int idpf_vport_open(struct idpf_vport *vport, bool rtnl) { struct idpf_netdev_priv *np =3D netdev_priv(vport->netdev); struct idpf_adapter *adapter =3D vport->adapter; @@ -1381,6 +1389,9 @@ static int idpf_vport_open(struct idpf_vport *vport) if (np->state !=3D __IDPF_VPORT_DOWN) return -EBUSY; =20 + if (rtnl) + rtnl_lock(); + /* we do not allow interface up just yet */ netif_carrier_off(vport->netdev); =20 @@ -1388,7 +1399,7 @@ static int idpf_vport_open(struct idpf_vport *vport) if (err) { dev_err(&adapter->pdev->dev, "Failed to allocate interrupts for vport %u= : %d\n", vport->vport_id, err); - return err; + goto err_rtnl_unlock; } =20 err =3D idpf_vport_queues_alloc(vport); @@ -1475,6 +1486,9 @@ static int idpf_vport_open(struct idpf_vport *vport) goto deinit_rss; } =20 + if (rtnl) + rtnl_unlock(); + return 0; =20 deinit_rss: @@ -1492,6 +1506,10 @@ static int idpf_vport_open(struct idpf_vport *vport) intr_rel: idpf_vport_intr_rel(vport); =20 +err_rtnl_unlock: + if (rtnl) + rtnl_unlock(); + return err; } =20 @@ -1572,7 +1590,7 @@ void idpf_init_task(struct work_struct *work) np =3D netdev_priv(vport->netdev); np->state =3D __IDPF_VPORT_DOWN; if (test_and_clear_bit(IDPF_VPORT_UP_REQUESTED, vport_config->flags)) - idpf_vport_open(vport); + idpf_vport_open(vport, true); =20 /* Spawn and return 'idpf_init_task' work queue until all the * default vports are created @@ -1962,7 +1980,7 @@ int idpf_initiate_soft_reset(struct idpf_vport *vport, idpf_send_delete_queues_msg(vport); } else { set_bit(IDPF_VPORT_DEL_QUEUES, vport->flags); - idpf_vport_stop(vport); + idpf_vport_stop(vport, false); } =20 idpf_deinit_rss(vport); @@ -1992,7 +2010,7 @@ int idpf_initiate_soft_reset(struct idpf_vport *vport, goto err_open; =20 if (current_state =3D=3D __IDPF_VPORT_UP) - err =3D idpf_vport_open(vport); + err =3D idpf_vport_open(vport, false); =20 goto free_vport; =20 @@ -2002,7 +2020,7 @@ int idpf_initiate_soft_reset(struct idpf_vport *vport, =20 err_open: if (current_state =3D=3D __IDPF_VPORT_UP) - idpf_vport_open(vport); + idpf_vport_open(vport, false); =20 free_vport: kfree(new_vport); @@ -2240,7 +2258,7 @@ static int idpf_open(struct net_device *netdev) if (err) goto unlock; =20 - err =3D idpf_vport_open(vport); + err =3D idpf_vport_open(vport, false); =20 unlock: idpf_vport_ctrl_unlock(netdev); diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index f49791eab07d..0a2a2b21d1ef 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -3423,6 +3423,20 @@ void idpf_vport_intr_rel(struct idpf_vport *vport) vport->q_vectors =3D NULL; } =20 +static void idpf_q_vector_set_napi(struct idpf_q_vector *q_vector, bool li= nk) +{ + struct napi_struct *napi =3D link ? &q_vector->napi : NULL; + struct net_device *dev =3D q_vector->vport->netdev; + + for (u32 i =3D 0; i < q_vector->num_rxq; i++) + netif_queue_set_napi(dev, q_vector->rx[i]->idx, + NETDEV_QUEUE_TYPE_RX, napi); + + for (u32 i =3D 0; i < q_vector->num_txq; i++) + netif_queue_set_napi(dev, q_vector->tx[i]->idx, + NETDEV_QUEUE_TYPE_TX, napi); +} + /** * idpf_vport_intr_rel_irq - Free the IRQ association with the OS * @vport: main vport structure @@ -3443,6 +3457,7 @@ static void idpf_vport_intr_rel_irq(struct idpf_vport= *vport) vidx =3D vport->q_vector_idxs[vector]; irq_num =3D adapter->msix_entries[vidx].vector; =20 + idpf_q_vector_set_napi(q_vector, false); kfree(free_irq(irq_num, q_vector)); } } @@ -3630,6 +3645,8 @@ static int idpf_vport_intr_req_irq(struct idpf_vport = *vport) "Request_irq failed, error: %d\n", err); goto free_q_irqs; } + + idpf_q_vector_set_napi(q_vector, true); } =20 return 0; --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23BC53680AF; Tue, 26 Aug 2025 16:13:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224830; cv=none; b=FOGR3yuKM2pS8zVsU/DI9WrsgYvJoaiO7FGh41x0ROqJZ56Vt3HyDh+0q5jr9ypNooYQVGB4+ehsKLtAPpqEp6rYH/QcZ/HFI3McKu0N5U8nwk44XomYLYC/u9GGsMDGlo7INwvJisupjdoa64KzuuqZYku2kTVqwXv/AcvYNro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224830; c=relaxed/simple; bh=xJtOSi5aIWMYLH0p0ok+yb+IaaOhf6kfqMcp711y9r0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cBJ2Z35AUlLJKAbp6N/OqrfOgB0Yzw04RwCQiDsFWQM0MZrI5QUInnN1aDZH0HsmkeGhKnEgcuGT6tIHP7vKC5kZxy5tKFLYHtLZID26GfreJeGr2o0e52We5GTNHh99IE2Cmqh3t57dHuZKAMdTsCzI73RIn/B66XX78390SBI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=LuFU1M7z; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="LuFU1M7z" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224829; x=1787760829; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xJtOSi5aIWMYLH0p0ok+yb+IaaOhf6kfqMcp711y9r0=; b=LuFU1M7zRRFymEUEXiZAo3bu6MuNDIwOpa77t72g3usn0BnJz/MdUKaI MJ6h0FiWBR8hvBHd419GLDikgCshkOpmrhe7G0vAYKb7+n30g08xz+UUa fqGT2kaFK3rcKG94enaAcxnf/iJLhuD1Mx/ZTE2zv6ItucIadGSJIp1Se Qd/R80Crt5A3hQKJvLlB3JUeh6nBTFY5DEWM1X+APLxP5MDHy/TPWMxTO guGvzmsb/Wf1d7oASpzb+q0FuTSTsV5woikdmVZoAZhgAF3ZrsgdALwU2 9cEnormJO8szHXCgJY9WK9th3NcA21jG979xAyIMWwVMbXrvwKDEr3bfI g==; X-CSE-ConnectionGUID: tM+c36XTT7O17mTYpuGnDw== X-CSE-MsgGUID: mcyY6AZQRJqnY3f9c2+rbQ== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46044946" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46044946" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:13:49 -0700 X-CSE-ConnectionGUID: A4iekAqhQquAxOmQHjJXNQ== X-CSE-MsgGUID: duhTR3DRQSeeOOSj0xXs7A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562197" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:44 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 05/13] idpf: add 4-byte completion descriptor definition Date: Tue, 26 Aug 2025 17:54:59 +0200 Message-ID: <20250826155507.2138401-6-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Michal Kubiak In the queue-based scheduling mode, Tx completion descriptor is 4 bytes comparing to 8 bytes in flow-based. Add definition for it and allocate the corresponding amount of memory for the descriptors during the completion queue creation. This does not include handling 4-byte completions during Tx polling, as for now, the only user of QB will be XDP, which has its own routines. Signed-off-by: Michal Kubiak Signed-off-by: Alexander Lobakin --- .../net/ethernet/intel/idpf/idpf_lan_txrx.h | 6 +++- drivers/net/ethernet/intel/idpf/idpf_txrx.h | 11 ++++-- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 34 +++++++++++-------- 3 files changed, 33 insertions(+), 18 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_lan_txrx.h b/drivers/net/= ethernet/intel/idpf/idpf_lan_txrx.h index 7492d1713243..20d5af64e750 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lan_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_lan_txrx.h @@ -186,13 +186,17 @@ struct idpf_base_tx_desc { __le64 qw1; /* type_cmd_offset_bsz_l2tag1 */ }; /* read used with buffer queues */ =20 -struct idpf_splitq_tx_compl_desc { +struct idpf_splitq_4b_tx_compl_desc { /* qid=3D[10:0] comptype=3D[13:11] rsvd=3D[14] gen=3D[15] */ __le16 qid_comptype_gen; union { __le16 q_head; /* Queue head */ __le16 compl_tag; /* Completion tag */ } q_head_compl_tag; +}; /* writeback used with completion queues */ + +struct idpf_splitq_tx_compl_desc { + struct idpf_splitq_4b_tx_compl_desc common; u8 ts[3]; u8 rsvd; /* Reserved */ }; /* writeback used with completion queues */ diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.h index 52753dff381c..11a318fd48d4 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -728,7 +728,9 @@ libeth_cacheline_set_assert(struct idpf_buf_queue, 64, = 24, 32); =20 /** * struct idpf_compl_queue - software structure representing a completion = queue - * @comp: completion descriptor array + * @comp: 8-byte completion descriptor array + * @comp_4b: 4-byte completion descriptor array + * @desc_ring: virtual descriptor ring address * @txq_grp: See struct idpf_txq_group * @flags: See enum idpf_queue_flags_t * @desc_count: Number of descriptors @@ -748,7 +750,12 @@ libeth_cacheline_set_assert(struct idpf_buf_queue, 64,= 24, 32); */ struct idpf_compl_queue { __cacheline_group_begin_aligned(read_mostly); - struct idpf_splitq_tx_compl_desc *comp; + union { + struct idpf_splitq_tx_compl_desc *comp; + struct idpf_splitq_4b_tx_compl_desc *comp_4b; + + void *desc_ring; + }; struct idpf_txq_group *txq_grp; =20 DECLARE_BITMAP(flags, __IDPF_Q_FLAGS_NBITS); diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index 0a2a2b21d1ef..15bdbb2c62a9 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -115,8 +115,8 @@ static void idpf_compl_desc_rel(struct idpf_compl_queue= *complq) return; =20 dma_free_coherent(complq->netdev->dev.parent, complq->size, - complq->comp, complq->dma); - complq->comp =3D NULL; + complq->desc_ring, complq->dma); + complq->desc_ring =3D NULL; complq->next_to_use =3D 0; complq->next_to_clean =3D 0; } @@ -245,12 +245,16 @@ static int idpf_tx_desc_alloc(const struct idpf_vport= *vport, static int idpf_compl_desc_alloc(const struct idpf_vport *vport, struct idpf_compl_queue *complq) { - complq->size =3D array_size(complq->desc_count, sizeof(*complq->comp)); + u32 desc_size; =20 - complq->comp =3D dma_alloc_coherent(complq->netdev->dev.parent, - complq->size, &complq->dma, - GFP_KERNEL); - if (!complq->comp) + desc_size =3D idpf_queue_has(FLOW_SCH_EN, complq) ? + sizeof(*complq->comp) : sizeof(*complq->comp_4b); + complq->size =3D array_size(complq->desc_count, desc_size); + + complq->desc_ring =3D dma_alloc_coherent(complq->netdev->dev.parent, + complq->size, &complq->dma, + GFP_KERNEL); + if (!complq->desc_ring) return -ENOMEM; =20 complq->next_to_use =3D 0; @@ -1758,7 +1762,7 @@ static void idpf_tx_handle_rs_completion(struct idpf_= tx_queue *txq, /* RS completion contains queue head for queue based scheduling or * completion tag for flow based scheduling. */ - u16 rs_compl_val =3D le16_to_cpu(desc->q_head_compl_tag.q_head); + u16 rs_compl_val =3D le16_to_cpu(desc->common.q_head_compl_tag.q_head); =20 if (!idpf_queue_has(FLOW_SCH_EN, txq)) { idpf_tx_splitq_clean(txq, rs_compl_val, budget, cleaned, false); @@ -1793,19 +1797,19 @@ static bool idpf_tx_clean_complq(struct idpf_compl_= queue *complq, int budget, do { struct libeth_sq_napi_stats cleaned_stats =3D { }; struct idpf_tx_queue *tx_q; + __le16 hw_head; int rel_tx_qid; - u16 hw_head; u8 ctype; /* completion type */ u16 gen; =20 /* if the descriptor isn't done, no work yet to do */ - gen =3D le16_get_bits(tx_desc->qid_comptype_gen, + gen =3D le16_get_bits(tx_desc->common.qid_comptype_gen, IDPF_TXD_COMPLQ_GEN_M); if (idpf_queue_has(GEN_CHK, complq) !=3D gen) break; =20 /* Find necessary info of TX queue to clean buffers */ - rel_tx_qid =3D le16_get_bits(tx_desc->qid_comptype_gen, + rel_tx_qid =3D le16_get_bits(tx_desc->common.qid_comptype_gen, IDPF_TXD_COMPLQ_QID_M); if (rel_tx_qid >=3D complq->txq_grp->num_txq || !complq->txq_grp->txqs[rel_tx_qid]) { @@ -1815,14 +1819,14 @@ static bool idpf_tx_clean_complq(struct idpf_compl_= queue *complq, int budget, tx_q =3D complq->txq_grp->txqs[rel_tx_qid]; =20 /* Determine completion type */ - ctype =3D le16_get_bits(tx_desc->qid_comptype_gen, + ctype =3D le16_get_bits(tx_desc->common.qid_comptype_gen, IDPF_TXD_COMPLQ_COMPL_TYPE_M); switch (ctype) { case IDPF_TXD_COMPLT_RE: - hw_head =3D le16_to_cpu(tx_desc->q_head_compl_tag.q_head); + hw_head =3D tx_desc->common.q_head_compl_tag.q_head; =20 - idpf_tx_splitq_clean(tx_q, hw_head, budget, - &cleaned_stats, true); + idpf_tx_splitq_clean(tx_q, le16_to_cpu(hw_head), + budget, &cleaned_stats, true); break; case IDPF_TXD_COMPLT_RS: idpf_tx_handle_rs_completion(tx_q, tx_desc, --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 908C93164A6; Tue, 26 Aug 2025 16:13:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224835; cv=none; b=SJvgnWVxf3isoSmUUjs24ic3KZFk/jt3YakwlYjmXNLo/mJ9w2Caz+nAByfV6KX1ZXw9dsdAKsl4LZ+bR23K/EGFPwl90IdIlXLMXRAMhl4Bv2dtq7Z4kWSv+maVcjH/FjkBxTNKYreWStva3m4GRNAK91RV0R3lygeQXmTHdic= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224835; c=relaxed/simple; bh=UXVuXw6RAqpzgo5vhw5ugiGAw//S00Dh2E6Sjf/Lokg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M8DW2UnGy/QBtd0fRNn3/oRSmSwRp/WZIM0JqB07sHIyjmbz+RlpraemSrSldBjQv5W4X8i9JibpSGsDdj0FoIwyPEmlH3fZEZvFzIf/7gL3ZadQbuuxpUDpWD1TmFIaeNX7fm9WnLxKD40KtIKKotefzjxFXl5RhETbDsui5rI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CPKmoE8Z; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CPKmoE8Z" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224834; x=1787760834; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UXVuXw6RAqpzgo5vhw5ugiGAw//S00Dh2E6Sjf/Lokg=; b=CPKmoE8ZEcdGUyIcSO5WJeLcgn2rkMClHJ0tFH7I6Q1qtJz/cQziYO67 9voMaIwF8GkZUhSbKTilaDZhm8jySgwscamajg0nd/mLabVrWfWB+MvnA Y2hDfOtL2oaBOMOH2E7Zyz2JPavhd6cqN8pY96lmZ9pUK/GQITI/gWviE KvBl0Lpj8nbjvfsuXN1Sgmhckuk7ZfSyV7aT45HC7a623x1bLglYPom9x JhENdOIaPzQW1STHnG5ha69QCKj2Y+STeB2CtJ/PRi6mSAzweyCGjbOgK LWADepw1VDIlBQ3A41YAZr8BbobDq03knpNqmCrzptHicfSnsuZpfcLm/ g==; X-CSE-ConnectionGUID: 64AGYhaLTKGJVlkSIXZTzQ== X-CSE-MsgGUID: 0Bd3b32zS4WShYozyZiMeA== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46044962" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46044962" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:13:53 -0700 X-CSE-ConnectionGUID: PSpNFn4tQTa+yxTIwxjeEg== X-CSE-MsgGUID: O7NOo+NfT++K/opnoXQYhg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562223" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:49 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 06/13] idpf: remove SW marker handling from NAPI Date: Tue, 26 Aug 2025 17:55:00 +0200 Message-ID: <20250826155507.2138401-7-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Michal Kubiak SW marker descriptors on completion queues are used only when a queue is about to be destroyed. It's far from hotpath and handling it in the hotpath NAPI poll makes no sense. Instead, run a simple poller after a virtchnl message for destroying the queue is sent and wait for the replies. If replies for all of the queues are received, this means the synchronization is done correctly and we can go forth with stopping the link. Signed-off-by: Michal Kubiak Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/idpf.h | 7 +- drivers/net/ethernet/intel/idpf/idpf_txrx.h | 4 +- drivers/net/ethernet/intel/idpf/idpf_lib.c | 2 - drivers/net/ethernet/intel/idpf/idpf_txrx.c | 97 ++++++++++++------- .../net/ethernet/intel/idpf/idpf_virtchnl.c | 34 ++----- 5 files changed, 72 insertions(+), 72 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf.h b/drivers/net/ethernet/= intel/idpf/idpf.h index aafbb280c2e7..269e9b41645a 100644 --- a/drivers/net/ethernet/intel/idpf/idpf.h +++ b/drivers/net/ethernet/intel/idpf/idpf.h @@ -40,6 +40,7 @@ struct idpf_vport_max_q; #define IDPF_NUM_CHUNKS_PER_MSG(struct_sz, chunk_sz) \ ((IDPF_CTLQ_MAX_BUF_LEN - (struct_sz)) / (chunk_sz)) =20 +#define IDPF_WAIT_FOR_MARKER_TIMEO 500 #define IDPF_MAX_WAIT 500 =20 /* available message levels */ @@ -248,13 +249,10 @@ enum idpf_vport_reset_cause { /** * enum idpf_vport_flags - Vport flags * @IDPF_VPORT_DEL_QUEUES: To send delete queues message - * @IDPF_VPORT_SW_MARKER: Indicate TX pipe drain software marker packets - * processing is done * @IDPF_VPORT_FLAGS_NBITS: Must be last */ enum idpf_vport_flags { IDPF_VPORT_DEL_QUEUES, - IDPF_VPORT_SW_MARKER, IDPF_VPORT_FLAGS_NBITS, }; =20 @@ -320,7 +318,6 @@ struct idpf_fsteer_fltr { * @tx_itr_profile: TX profiles for Dynamic Interrupt Moderation * @port_stats: per port csum, header split, and other offload stats * @link_up: True if link is up - * @sw_marker_wq: workqueue for marker packets * @tx_tstamp_caps: Capabilities negotiated for Tx timestamping * @tstamp_config: The Tx tstamp config * @tstamp_task: Tx timestamping task @@ -369,8 +366,6 @@ struct idpf_vport { =20 bool link_up; =20 - wait_queue_head_t sw_marker_wq; - struct idpf_ptp_vport_tx_tstamp_caps *tx_tstamp_caps; struct kernel_hwtstamp_config tstamp_config; struct work_struct tstamp_task; diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.h index 11a318fd48d4..1c570794e5bc 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -275,7 +275,6 @@ struct idpf_ptype_state { * bit and Q_RFL_GEN is the SW bit. * @__IDPF_Q_FLOW_SCH_EN: Enable flow scheduling * @__IDPF_Q_SW_MARKER: Used to indicate TX queue marker completions - * @__IDPF_Q_POLL_MODE: Enable poll mode * @__IDPF_Q_CRC_EN: enable CRC offload in singleq mode * @__IDPF_Q_HSPLIT_EN: enable header split on Rx (splitq) * @__IDPF_Q_PTP: indicates whether the Rx timestamping is enabled for the @@ -287,7 +286,6 @@ enum idpf_queue_flags_t { __IDPF_Q_RFL_GEN_CHK, __IDPF_Q_FLOW_SCH_EN, __IDPF_Q_SW_MARKER, - __IDPF_Q_POLL_MODE, __IDPF_Q_CRC_EN, __IDPF_Q_HSPLIT_EN, __IDPF_Q_PTP, @@ -1036,4 +1034,6 @@ bool idpf_rx_singleq_buf_hw_alloc_all(struct idpf_rx_= queue *rxq, u16 cleaned_count); int idpf_tso(struct sk_buff *skb, struct idpf_tx_offload_params *off); =20 +void idpf_wait_for_sw_marker_completion(const struct idpf_tx_queue *txq); + #endif /* !_IDPF_TXRX_H_ */ diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ether= net/intel/idpf/idpf_lib.c index b5a7215488b9..ef9f2b0cef67 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -1566,8 +1566,6 @@ void idpf_init_task(struct work_struct *work) index =3D vport->idx; vport_config =3D adapter->vport_config[index]; =20 - init_waitqueue_head(&vport->sw_marker_wq); - spin_lock_init(&vport_config->mac_filter_list_lock); =20 INIT_LIST_HEAD(&vport_config->user_config.mac_filter_list); diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index 15bdbb2c62a9..d362dab4d5a2 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -1559,32 +1559,6 @@ int idpf_vport_queues_alloc(struct idpf_vport *vport) return err; } =20 -/** - * idpf_tx_handle_sw_marker - Handle queue marker packet - * @tx_q: tx queue to handle software marker - */ -static void idpf_tx_handle_sw_marker(struct idpf_tx_queue *tx_q) -{ - struct idpf_netdev_priv *priv =3D netdev_priv(tx_q->netdev); - struct idpf_vport *vport =3D priv->vport; - int i; - - idpf_queue_clear(SW_MARKER, tx_q); - /* Hardware must write marker packets to all queues associated with - * completion queues. So check if all queues received marker packets - */ - for (i =3D 0; i < vport->num_txq; i++) - /* If we're still waiting on any other TXQ marker completions, - * just return now since we cannot wake up the marker_wq yet. - */ - if (idpf_queue_has(SW_MARKER, vport->txqs[i])) - return; - - /* Drain complete */ - set_bit(IDPF_VPORT_SW_MARKER, vport->flags); - wake_up(&vport->sw_marker_wq); -} - /** * idpf_tx_read_tstamp - schedule a work to read Tx timestamp value * @txq: queue to read the timestamp from @@ -1832,9 +1806,6 @@ static bool idpf_tx_clean_complq(struct idpf_compl_qu= eue *complq, int budget, idpf_tx_handle_rs_completion(tx_q, tx_desc, &cleaned_stats, budget); break; - case IDPF_TXD_COMPLT_SW_MARKER: - idpf_tx_handle_sw_marker(tx_q); - break; default: netdev_err(tx_q->netdev, "Unknown TX completion type: %d\n", ctype); @@ -1906,6 +1877,66 @@ static bool idpf_tx_clean_complq(struct idpf_compl_q= ueue *complq, int budget, return !!complq_budget; } =20 +/** + * idpf_wait_for_sw_marker_completion - wait for SW marker of disabled Tx = queue + * @txq: disabled Tx queue + * + * When Tx queue is requested for disabling, the CP sends a special comple= tion + * descriptor called "SW marker", meaning the queue is ready to be destroy= ed. + * If, for some reason, the marker is not received within 500 ms, break the + * polling to not hang the driver. + */ +void idpf_wait_for_sw_marker_completion(const struct idpf_tx_queue *txq) +{ + struct idpf_compl_queue *complq =3D txq->txq_grp->complq; + u32 ntc =3D complq->next_to_clean; + unsigned long timeout; + bool flow, gen_flag; + + if (!idpf_queue_has(SW_MARKER, txq)) + return; + + flow =3D idpf_queue_has(FLOW_SCH_EN, complq); + gen_flag =3D idpf_queue_has(GEN_CHK, complq); + + timeout =3D jiffies + msecs_to_jiffies(IDPF_WAIT_FOR_MARKER_TIMEO); + + do { + struct idpf_splitq_4b_tx_compl_desc *tx_desc; + struct idpf_tx_queue *target; + u32 ctype_gen, id; + + tx_desc =3D flow ? &complq->comp[ntc].common : + &complq->comp_4b[ntc]; + ctype_gen =3D le16_to_cpu(tx_desc->qid_comptype_gen); + + if (!!(ctype_gen & IDPF_TXD_COMPLQ_GEN_M) !=3D gen_flag) { + usleep_range(500, 1000); + continue; + } + + if (FIELD_GET(IDPF_TXD_COMPLQ_COMPL_TYPE_M, ctype_gen) !=3D + IDPF_TXD_COMPLT_SW_MARKER) + goto next; + + id =3D FIELD_GET(IDPF_TXD_COMPLQ_QID_M, ctype_gen); + target =3D complq->txq_grp->txqs[id]; + + idpf_queue_clear(SW_MARKER, target); + if (target =3D=3D txq) + break; + +next: + if (unlikely(++ntc =3D=3D complq->desc_count)) { + ntc =3D 0; + gen_flag =3D !gen_flag; + } + } while (time_before(jiffies, timeout)); + + idpf_queue_assign(GEN_CHK, complq, gen_flag); + complq->next_to_clean =3D ntc; +} + /** * idpf_tx_splitq_build_ctb - populate command tag and size for queue * based scheduling descriptors @@ -3911,14 +3942,6 @@ static int idpf_vport_splitq_napi_poll(struct napi_s= truct *napi, int budget) return budget; } =20 - /* Switch to poll mode in the tear-down path after sending disable - * queues virtchnl message, as the interrupts will be disabled after - * that. - */ - if (unlikely(q_vector->num_txq && idpf_queue_has(POLL_MODE, - q_vector->tx[0]))) - return budget; - work_done =3D min_t(int, work_done, budget - 1); =20 /* Exit the polling mode, but don't re-enable interrupts if stack might diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c b/drivers/net/= ethernet/intel/idpf/idpf_virtchnl.c index 6164094c6ae5..d3289b3e6602 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c @@ -724,21 +724,17 @@ int idpf_recv_mb_msg(struct idpf_adapter *adapter) **/ static int idpf_wait_for_marker_event(struct idpf_vport *vport) { - int event; - int i; - - for (i =3D 0; i < vport->num_txq; i++) - idpf_queue_set(SW_MARKER, vport->txqs[i]); + bool markers_rcvd =3D true; =20 - event =3D wait_event_timeout(vport->sw_marker_wq, - test_and_clear_bit(IDPF_VPORT_SW_MARKER, - vport->flags), - msecs_to_jiffies(500)); + for (u32 i =3D 0; i < vport->num_txq; i++) { + struct idpf_tx_queue *txq =3D vport->txqs[i]; =20 - for (i =3D 0; i < vport->num_txq; i++) - idpf_queue_clear(POLL_MODE, vport->txqs[i]); + idpf_queue_set(SW_MARKER, txq); + idpf_wait_for_sw_marker_completion(txq); + markers_rcvd &=3D !idpf_queue_has(SW_MARKER, txq); + } =20 - if (event) + if (markers_rcvd) return 0; =20 dev_warn(&vport->adapter->pdev->dev, "Failed to receive marker packets\n"= ); @@ -2137,24 +2133,12 @@ int idpf_send_enable_queues_msg(struct idpf_vport *= vport) */ int idpf_send_disable_queues_msg(struct idpf_vport *vport) { - int err, i; + int err; =20 err =3D idpf_send_ena_dis_queues_msg(vport, false); if (err) return err; =20 - /* switch to poll mode as interrupts will be disabled after disable - * queues virtchnl message is sent - */ - for (i =3D 0; i < vport->num_txq; i++) - idpf_queue_set(POLL_MODE, vport->txqs[i]); - - /* schedule the napi to receive all the marker packets */ - local_bh_disable(); - for (i =3D 0; i < vport->num_q_vectors; i++) - napi_schedule(&vport->q_vectors[i].napi); - local_bh_enable(); - return idpf_wait_for_marker_event(vport); } =20 --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F1A436CC68; Tue, 26 Aug 2025 16:13:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224840; cv=none; b=RUaWvaNe+a29mddsAaJSb5P3p+NQo6YAdcHfq93mOPvzB827J983AcIEJOZhihham1newQXAyijgeEoyTIQ7le7i+lBsOJ2pGrVpCkvLMG5h81deSXKN3W9zKyGpyUpyS5+S6v2Rw476cODoZr7E1hoLQZvWDZgw0JzLY+8Wx8s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224840; c=relaxed/simple; bh=Vhy7Nailub8C+q8nS3Gw3ogR5SUVPzBVym0uhTYVSB0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H95Wf+FJ7Icmkchcb7UvnmfeDuH3S8oV6gSxYkDKoiwS+GqRW19HtN9mMxzIHOKUyEZMI3OJyntLpQTtFpzb+NJ3SmVaNTeS+4o0pD2wm5RA+g+98mT9ozElhlGGOY7tNpBSXEmonY/9UghR786Q0Bg481/PrLbBigxOuGXLdQo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=WQunnjal; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="WQunnjal" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224838; x=1787760838; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Vhy7Nailub8C+q8nS3Gw3ogR5SUVPzBVym0uhTYVSB0=; b=WQunnjallAxiD7INRKQDNwoQFgMHq45ivUEuIgxaG9CpovAwiBxE9j2h fwNnxInZ8FqTGBYmRGpRKNlksE6qHeJeSmnSJwJTTqSqTn1wO1fVxgUT3 G0piXzbJM+F6qdaqHHakGvrxqauH9OPuRru7Q1ji0jKuIqjBDcaCAyyDJ r8cCjYkgA3d7pY3TRaDYhC26rYqt+sRDQhW9PjQJVDyAAgrz+N8jFEl22 HAziLnpb2moNW8HeW/y0pg8XN2SInvAcTC1zy1NklJ14tEciWYql7+9ZI WoTwyh90mkd3qKsIu6y7DhtrUcl24x6jwt0nYf7ZQpMrrGZBpQsik08eE w==; X-CSE-ConnectionGUID: zwKORrOeTZyY8+fayxhB8Q== X-CSE-MsgGUID: stzMVLMGSqKvx7x0mB3wFw== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46044991" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46044991" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:13:58 -0700 X-CSE-ConnectionGUID: DBSP2MCIQBe5+gEuLacdHg== X-CSE-MsgGUID: lfkfDkUqRXSw49hf098Nlw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562277" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:53 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 07/13] idpf: add support for nointerrupt queues Date: Tue, 26 Aug 2025 17:55:01 +0200 Message-ID: <20250826155507.2138401-8-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, queues are associated 1:1 with interrupt vectors as it's assumed queues are always interrupt-driven. For XDP, we want to use Tx queues without interrupts and only do "lazy" cleaning when the number of free elements is <=3D threshold (closest pow-2 to 1/4 of the ring). In order to use a queue without an interrupt, idpf still needs to have a vector assigned to it to flush descriptors. This vector can be global and only one for the whole vport to handle all its noirq queues. Always request one excessive vector and configure it in non-interrupt mode right away when creating vport, so that it can be used later by queues when needed (not only XDP ones). Co-developed-by: Michal Kubiak Signed-off-by: Michal Kubiak Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/idpf.h | 8 +++ drivers/net/ethernet/intel/idpf/idpf_txrx.h | 4 ++ drivers/net/ethernet/intel/idpf/idpf_dev.c | 11 +++- drivers/net/ethernet/intel/idpf/idpf_lib.c | 2 +- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 8 +++ drivers/net/ethernet/intel/idpf/idpf_vf_dev.c | 11 +++- .../net/ethernet/intel/idpf/idpf_virtchnl.c | 54 ++++++++++++++----- 7 files changed, 81 insertions(+), 17 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf.h b/drivers/net/ethernet/= intel/idpf/idpf.h index 269e9b41645a..2bfdf0ae24cf 100644 --- a/drivers/net/ethernet/intel/idpf/idpf.h +++ b/drivers/net/ethernet/intel/idpf/idpf.h @@ -312,6 +312,9 @@ struct idpf_fsteer_fltr { * @num_q_vectors: Number of IRQ vectors allocated * @q_vectors: Array of queue vectors * @q_vector_idxs: Starting index of queue vectors + * @noirq_dyn_ctl: register to enable/disable the vector for NOIRQ queues + * @noirq_dyn_ctl_ena: value to write to the above to enable it + * @noirq_v_idx: ID of the NOIRQ vector * @max_mtu: device given max possible MTU * @default_mac_addr: device will give a default MAC to use * @rx_itr_profile: RX profiles for Dynamic Interrupt Moderation @@ -358,6 +361,11 @@ struct idpf_vport { u16 num_q_vectors; struct idpf_q_vector *q_vectors; u16 *q_vector_idxs; + + void __iomem *noirq_dyn_ctl; + u32 noirq_dyn_ctl_ena; + u16 noirq_v_idx; + u16 max_mtu; u8 default_mac_addr[ETH_ALEN]; u16 rx_itr_profile[IDPF_DIM_PROFILE_SLOTS]; diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.h index 1c570794e5bc..f8e579dab21a 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -58,6 +58,8 @@ #define IDPF_MBX_Q_VEC 1 #define IDPF_MIN_Q_VEC 1 #define IDPF_MIN_RDMA_VEC 2 +/* Data vector for NOIRQ queues */ +#define IDPF_RESERVED_VECS 1 =20 #define IDPF_DFLT_TX_Q_DESC_COUNT 512 #define IDPF_DFLT_TX_COMPLQ_DESC_COUNT 512 @@ -279,6 +281,7 @@ struct idpf_ptype_state { * @__IDPF_Q_HSPLIT_EN: enable header split on Rx (splitq) * @__IDPF_Q_PTP: indicates whether the Rx timestamping is enabled for the * queue + * @__IDPF_Q_NOIRQ: queue is polling-driven and has no interrupt * @__IDPF_Q_FLAGS_NBITS: Must be last */ enum idpf_queue_flags_t { @@ -289,6 +292,7 @@ enum idpf_queue_flags_t { __IDPF_Q_CRC_EN, __IDPF_Q_HSPLIT_EN, __IDPF_Q_PTP, + __IDPF_Q_NOIRQ, =20 __IDPF_Q_FLAGS_NBITS, }; diff --git a/drivers/net/ethernet/intel/idpf/idpf_dev.c b/drivers/net/ether= net/intel/idpf/idpf_dev.c index bfa60f7d43de..3a04a6bd0d7c 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_dev.c +++ b/drivers/net/ethernet/intel/idpf/idpf_dev.c @@ -77,7 +77,7 @@ static int idpf_intr_reg_init(struct idpf_vport *vport) int num_vecs =3D vport->num_q_vectors; struct idpf_vec_regs *reg_vals; int num_regs, i, err =3D 0; - u32 rx_itr, tx_itr; + u32 rx_itr, tx_itr, val; u16 total_vecs; =20 total_vecs =3D idpf_get_reserved_vecs(vport->adapter); @@ -121,6 +121,15 @@ static int idpf_intr_reg_init(struct idpf_vport *vport) intr->tx_itr =3D idpf_get_reg_addr(adapter, tx_itr); } =20 + /* Data vector for NOIRQ queues */ + + val =3D reg_vals[vport->q_vector_idxs[i] - IDPF_MBX_Q_VEC].dyn_ctl_reg; + vport->noirq_dyn_ctl =3D idpf_get_reg_addr(adapter, val); + + val =3D PF_GLINT_DYN_CTL_WB_ON_ITR_M | PF_GLINT_DYN_CTL_INTENA_MSK_M | + FIELD_PREP(PF_GLINT_DYN_CTL_ITR_INDX_M, IDPF_NO_ITR_UPDATE_IDX); + vport->noirq_dyn_ctl_ena =3D val; + free_reg_vals: kfree(reg_vals); =20 diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ether= net/intel/idpf/idpf_lib.c index ef9f2b0cef67..baf1f9b196d5 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -1142,7 +1142,7 @@ static struct idpf_vport *idpf_vport_alloc(struct idp= f_adapter *adapter, if (!vport) return vport; =20 - num_max_q =3D max(max_q->max_txq, max_q->max_rxq); + num_max_q =3D max(max_q->max_txq, max_q->max_rxq) + IDPF_RESERVED_VECS; if (!adapter->vport_config[idx]) { struct idpf_vport_config *vport_config; struct idpf_q_coalesce *q_coal; diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index d362dab4d5a2..ceab25ee1aad 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -3506,6 +3506,8 @@ static void idpf_vport_intr_dis_irq_all(struct idpf_v= port *vport) struct idpf_q_vector *q_vector =3D vport->q_vectors; int q_idx; =20 + writel(0, vport->noirq_dyn_ctl); + for (q_idx =3D 0; q_idx < vport->num_q_vectors; q_idx++) writel(0, q_vector[q_idx].intr_reg.dyn_ctl); } @@ -3749,6 +3751,8 @@ static void idpf_vport_intr_ena_irq_all(struct idpf_v= port *vport) if (qv->num_txq || qv->num_rxq) idpf_vport_intr_update_itr_ena_irq(qv); } + + writel(vport->noirq_dyn_ctl_ena, vport->noirq_dyn_ctl); } =20 /** @@ -4060,6 +4064,8 @@ static int idpf_vport_intr_init_vec_idx(struct idpf_v= port *vport) for (i =3D 0; i < vport->num_q_vectors; i++) vport->q_vectors[i].v_idx =3D vport->q_vector_idxs[i]; =20 + vport->noirq_v_idx =3D vport->q_vector_idxs[i]; + return 0; } =20 @@ -4073,6 +4079,8 @@ static int idpf_vport_intr_init_vec_idx(struct idpf_v= port *vport) for (i =3D 0; i < vport->num_q_vectors; i++) vport->q_vectors[i].v_idx =3D vecids[vport->q_vector_idxs[i]]; =20 + vport->noirq_v_idx =3D vecids[vport->q_vector_idxs[i]]; + kfree(vecids); =20 return 0; diff --git a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c b/drivers/net/et= hernet/intel/idpf/idpf_vf_dev.c index 259d50fded67..4cc58c83688c 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c +++ b/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c @@ -76,7 +76,7 @@ static int idpf_vf_intr_reg_init(struct idpf_vport *vport) int num_vecs =3D vport->num_q_vectors; struct idpf_vec_regs *reg_vals; int num_regs, i, err =3D 0; - u32 rx_itr, tx_itr; + u32 rx_itr, tx_itr, val; u16 total_vecs; =20 total_vecs =3D idpf_get_reserved_vecs(vport->adapter); @@ -120,6 +120,15 @@ static int idpf_vf_intr_reg_init(struct idpf_vport *vp= ort) intr->tx_itr =3D idpf_get_reg_addr(adapter, tx_itr); } =20 + /* Data vector for NOIRQ queues */ + + val =3D reg_vals[vport->q_vector_idxs[i] - IDPF_MBX_Q_VEC].dyn_ctl_reg; + vport->noirq_dyn_ctl =3D idpf_get_reg_addr(adapter, val); + + val =3D VF_INT_DYN_CTLN_WB_ON_ITR_M | VF_INT_DYN_CTLN_INTENA_MSK_M | + FIELD_PREP(VF_INT_DYN_CTLN_ITR_INDX_M, IDPF_NO_ITR_UPDATE_IDX); + vport->noirq_dyn_ctl_ena =3D val; + free_reg_vals: kfree(reg_vals); =20 diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c b/drivers/net/= ethernet/intel/idpf/idpf_virtchnl.c index d3289b3e6602..ff4ea49c0957 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c @@ -2018,21 +2018,31 @@ int idpf_send_map_unmap_queue_vector_msg(struct idp= f_vport *vport, bool map) struct idpf_txq_group *tx_qgrp =3D &vport->txq_grps[i]; =20 for (j =3D 0; j < tx_qgrp->num_txq; j++, k++) { + const struct idpf_tx_queue *txq =3D tx_qgrp->txqs[j]; + const struct idpf_q_vector *vec; + u32 v_idx, tx_itr_idx; + vqv[k].queue_type =3D cpu_to_le32(VIRTCHNL2_QUEUE_TYPE_TX); - vqv[k].queue_id =3D cpu_to_le32(tx_qgrp->txqs[j]->q_id); + vqv[k].queue_id =3D cpu_to_le32(txq->q_id); =20 - if (idpf_is_queue_model_split(vport->txq_model)) { - vqv[k].vector_id =3D - cpu_to_le16(tx_qgrp->complq->q_vector->v_idx); - vqv[k].itr_idx =3D - cpu_to_le32(tx_qgrp->complq->q_vector->tx_itr_idx); + if (idpf_queue_has(NOIRQ, txq)) + vec =3D NULL; + else if (idpf_is_queue_model_split(vport->txq_model)) + vec =3D txq->txq_grp->complq->q_vector; + else + vec =3D txq->q_vector; + + if (vec) { + v_idx =3D vec->v_idx; + tx_itr_idx =3D vec->tx_itr_idx; } else { - vqv[k].vector_id =3D - cpu_to_le16(tx_qgrp->txqs[j]->q_vector->v_idx); - vqv[k].itr_idx =3D - cpu_to_le32(tx_qgrp->txqs[j]->q_vector->tx_itr_idx); + v_idx =3D vport->noirq_v_idx; + tx_itr_idx =3D VIRTCHNL2_ITR_IDX_1; } + + vqv[k].vector_id =3D cpu_to_le16(v_idx); + vqv[k].itr_idx =3D cpu_to_le32(tx_itr_idx); } } =20 @@ -2050,6 +2060,7 @@ int idpf_send_map_unmap_queue_vector_msg(struct idpf_= vport *vport, bool map) =20 for (j =3D 0; j < num_rxq; j++, k++) { struct idpf_rx_queue *rxq; + u32 v_idx, rx_itr_idx; =20 if (idpf_is_queue_model_split(vport->rxq_model)) rxq =3D &rx_qgrp->splitq.rxq_sets[j]->rxq; @@ -2059,8 +2070,17 @@ int idpf_send_map_unmap_queue_vector_msg(struct idpf= _vport *vport, bool map) vqv[k].queue_type =3D cpu_to_le32(VIRTCHNL2_QUEUE_TYPE_RX); vqv[k].queue_id =3D cpu_to_le32(rxq->q_id); - vqv[k].vector_id =3D cpu_to_le16(rxq->q_vector->v_idx); - vqv[k].itr_idx =3D cpu_to_le32(rxq->q_vector->rx_itr_idx); + + if (idpf_queue_has(NOIRQ, rxq)) { + v_idx =3D vport->noirq_v_idx; + rx_itr_idx =3D VIRTCHNL2_ITR_IDX_0; + } else { + v_idx =3D rxq->q_vector->v_idx; + rx_itr_idx =3D rxq->q_vector->rx_itr_idx; + } + + vqv[k].vector_id =3D cpu_to_le16(v_idx); + vqv[k].itr_idx =3D cpu_to_le32(rx_itr_idx); } } =20 @@ -3281,9 +3301,15 @@ int idpf_vport_alloc_vec_indexes(struct idpf_vport *= vport) { struct idpf_vector_info vec_info; int num_alloc_vecs; + u32 req; =20 vec_info.num_curr_vecs =3D vport->num_q_vectors; - vec_info.num_req_vecs =3D max(vport->num_txq, vport->num_rxq); + if (vec_info.num_curr_vecs) + vec_info.num_curr_vecs +=3D IDPF_RESERVED_VECS; + + req =3D max(vport->num_txq, vport->num_rxq) + IDPF_RESERVED_VECS; + vec_info.num_req_vecs =3D req; + vec_info.default_vport =3D vport->default_vport; vec_info.index =3D vport->idx; =20 @@ -3296,7 +3322,7 @@ int idpf_vport_alloc_vec_indexes(struct idpf_vport *v= port) return -EINVAL; } =20 - vport->num_q_vectors =3D num_alloc_vecs; + vport->num_q_vectors =3D num_alloc_vecs - IDPF_RESERVED_VECS; =20 return 0; } --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5666336CC9C; Tue, 26 Aug 2025 16:14:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224847; cv=none; b=EhxorUpVtbV+zyDeUGOCeOCHOw8nb0GI/kzk3XrJ2EZX36/LVIcrHt2qVTt2ieuT6PKLNRy26w32ZKIXYcHNA+SdlPfjxwwRDuhChrBqLexx0n/XmE2vOGFEeiwsU0+tFQHRmQfoHNzWj8IuhTXWsuVgzOaDKd3ZfqziHzySo8Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224847; c=relaxed/simple; bh=sm+tJkmg8Gsu0MQDUj8hYeRO5XjkVkxbExF9YLCeCBw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z8Qzr5P0q3bXwYGkwGmrGxGsAs3bU+Rnz98u52W1fTY61SC2yXbzZvzS2F26kBymRRxyDAVNhAcMMiaPYpLOqL91Gczm4DvVhcvMCnkieFH26EeecAAvy8opFOVbkfZZ0Ok9uWiHFOz/v5LAzTLX9MehxaSxyzdw0qCVlxnIBgk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Ldl7U3Ur; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Ldl7U3Ur" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224843; x=1787760843; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sm+tJkmg8Gsu0MQDUj8hYeRO5XjkVkxbExF9YLCeCBw=; b=Ldl7U3Urpz7f8iZC46859XGlgB967UmnBHloPhhT8HiLcmctDW3GuVpv SGiUQuTWXA5arigxFMTW/APvfzPDSBoaFxhHysU5oGFFaY3VG58sEYta7 i/BKzdS4pSEnvE1hw9bIVkj+Wed5QCq2CSSaHb5cKlUdgtP26LNUyy8s8 +eEVPrc/gle58NqeUZSh1kPw6R726MAgBwdEv7iDqKt2e0ERuWAwF6oVc 1x13MQffBFGKhIysztie69721QDb/yZ1PwPbq8FwXWJzdmzqM7nRwBsB+ k3tscOHAwQMY+2yv6Q1yg/eMfkQG0MqrFhzPsY8VI3YS8eP2cX46T05kU A==; X-CSE-ConnectionGUID: ArfckOU2TKqVtPwEVFLmxg== X-CSE-MsgGUID: IeipJ/+xReyrAR2txCBjYw== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46045017" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46045017" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:14:03 -0700 X-CSE-ConnectionGUID: gPUcz4OIQgqMtgyr5s9AQg== X-CSE-MsgGUID: QXERqCi5Q3KpQA46t+V47g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562369" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:13:58 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 08/13] idpf: prepare structures to support XDP Date: Tue, 26 Aug 2025 17:55:02 +0200 Message-ID: <20250826155507.2138401-9-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Michal Kubiak Extend basic structures of the driver (e.g. 'idpf_vport', 'idpf_*_queue', 'idpf_vport_user_config_data') by adding members necessary to support XDP. Add extra XDP Tx queues needed to support XDP_TX and XDP_REDIRECT actions without interfering with regular Tx traffic. Also add functions dedicated to support XDP initialization for Rx and Tx queues and call those functions from the existing algorithms of queues configuration. Signed-off-by: Michal Kubiak Co-developed-by: Alexander Lobakin Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/Kconfig | 2 +- drivers/net/ethernet/intel/idpf/Makefile | 2 + drivers/net/ethernet/intel/idpf/idpf.h | 16 ++ drivers/net/ethernet/intel/idpf/idpf_txrx.h | 100 +++++++---- drivers/net/ethernet/intel/idpf/xdp.h | 17 ++ drivers/net/ethernet/intel/idpf/idpf_lib.c | 21 ++- drivers/net/ethernet/intel/idpf/idpf_main.c | 1 + .../ethernet/intel/idpf/idpf_singleq_txrx.c | 8 +- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 106 ++++++++--- .../net/ethernet/intel/idpf/idpf_virtchnl.c | 25 +-- drivers/net/ethernet/intel/idpf/xdp.c | 168 ++++++++++++++++++ 11 files changed, 381 insertions(+), 85 deletions(-) create mode 100644 drivers/net/ethernet/intel/idpf/xdp.h create mode 100644 drivers/net/ethernet/intel/idpf/xdp.c diff --git a/drivers/net/ethernet/intel/idpf/Kconfig b/drivers/net/ethernet= /intel/idpf/Kconfig index 2c359a8551c7..adab2154125b 100644 --- a/drivers/net/ethernet/intel/idpf/Kconfig +++ b/drivers/net/ethernet/intel/idpf/Kconfig @@ -6,7 +6,7 @@ config IDPF depends on PCI_MSI depends on PTP_1588_CLOCK_OPTIONAL select DIMLIB - select LIBETH + select LIBETH_XDP help This driver supports Intel(R) Infrastructure Data Path Function devices. diff --git a/drivers/net/ethernet/intel/idpf/Makefile b/drivers/net/etherne= t/intel/idpf/Makefile index 4ef4b2b5e37a..0840c3bef371 100644 --- a/drivers/net/ethernet/intel/idpf/Makefile +++ b/drivers/net/ethernet/intel/idpf/Makefile @@ -21,3 +21,5 @@ idpf-$(CONFIG_IDPF_SINGLEQ) +=3D idpf_singleq_txrx.o =20 idpf-$(CONFIG_PTP_1588_CLOCK) +=3D idpf_ptp.o idpf-$(CONFIG_PTP_1588_CLOCK) +=3D idpf_virtchnl_ptp.o + +idpf-y +=3D xdp.o diff --git a/drivers/net/ethernet/intel/idpf/idpf.h b/drivers/net/ethernet/= intel/idpf/idpf.h index 2bfdf0ae24cf..6e79fa8556e9 100644 --- a/drivers/net/ethernet/intel/idpf/idpf.h +++ b/drivers/net/ethernet/intel/idpf/idpf.h @@ -287,6 +287,10 @@ struct idpf_fsteer_fltr { * @txq_model: Split queue or single queue queuing model * @txqs: Used only in hotpath to get to the right queue very fast * @crc_enable: Enable CRC insertion offload + * @xdpsq_share: whether XDPSQ sharing is enabled + * @num_xdp_txq: number of XDPSQs + * @xdp_txq_offset: index of the first XDPSQ (=3D=3D number of regular SQs) + * @xdp_prog: installed XDP program * @num_rxq: Number of allocated RX queues * @num_bufq: Number of allocated buffer queues * @rxq_desc_count: RX queue descriptor count. *MUST* have enough descript= ors @@ -337,6 +341,11 @@ struct idpf_vport { struct idpf_tx_queue **txqs; bool crc_enable; =20 + bool xdpsq_share; + u16 num_xdp_txq; + u16 xdp_txq_offset; + struct bpf_prog *xdp_prog; + u16 num_rxq; u16 num_bufq; u32 rxq_desc_count; @@ -438,6 +447,7 @@ struct idpf_q_coalesce { * ethtool * @num_req_rxq_desc: Number of user requested RX queue descriptors through * ethtool + * @xdp_prog: requested XDP program to install * @user_flags: User toggled config flags * @mac_filter_list: List of MAC filters * @num_fsteer_fltrs: number of flow steering filters @@ -452,6 +462,7 @@ struct idpf_vport_user_config_data { u16 num_req_rx_qs; u32 num_req_txq_desc; u32 num_req_rxq_desc; + struct bpf_prog *xdp_prog; DECLARE_BITMAP(user_flags, __IDPF_USER_FLAGS_NBITS); struct list_head mac_filter_list; u32 num_fsteer_fltrs; @@ -681,6 +692,11 @@ static inline int idpf_is_queue_model_split(u16 q_mode= l) q_model =3D=3D VIRTCHNL2_QUEUE_MODEL_SPLIT; } =20 +static inline bool idpf_xdp_enabled(const struct idpf_vport *vport) +{ + return vport->adapter && vport->xdp_prog; +} + #define idpf_is_cap_ena(adapter, field, flag) \ idpf_is_capability_ena(adapter, false, field, flag) #define idpf_is_cap_ena_all(adapter, field, flag) \ diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.h index f8e579dab21a..6bc204b68d9e 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -7,8 +7,10 @@ #include =20 #include -#include +#include #include +#include +#include =20 #include "idpf_lan_txrx.h" #include "virtchnl2_lan_desc.h" @@ -282,6 +284,7 @@ struct idpf_ptype_state { * @__IDPF_Q_PTP: indicates whether the Rx timestamping is enabled for the * queue * @__IDPF_Q_NOIRQ: queue is polling-driven and has no interrupt + * @__IDPF_Q_XDP: this is an XDP queue * @__IDPF_Q_FLAGS_NBITS: Must be last */ enum idpf_queue_flags_t { @@ -293,6 +296,7 @@ enum idpf_queue_flags_t { __IDPF_Q_HSPLIT_EN, __IDPF_Q_PTP, __IDPF_Q_NOIRQ, + __IDPF_Q_XDP, =20 __IDPF_Q_FLAGS_NBITS, }; @@ -465,19 +469,21 @@ struct idpf_tx_queue_stats { * @napi: NAPI instance corresponding to this queue (splitq) * @rx_buf: See struct &libeth_fqe * @pp: Page pool pointer in singleq mode - * @netdev: &net_device corresponding to this queue * @tail: Tail offset. Used for both queue models single and split. * @flags: See enum idpf_queue_flags_t * @idx: For RX queue, it is used to index to total RX queue across groups= and * used for skb reporting. * @desc_count: Number of descriptors + * @num_xdp_txq: total number of XDP Tx queues + * @xdpsqs: shortcut for XDP Tx queues array * @rxdids: Supported RX descriptor ids + * @truesize: data buffer truesize in singleq * @rx_ptype_lkup: LUT of Rx ptypes + * @xdp_rxq: XDP queue info * @next_to_use: Next descriptor to use * @next_to_clean: Next descriptor to clean * @next_to_alloc: RX buffer to allocate at * @skb: Pointer to the skb - * @truesize: data buffer truesize in singleq * @cached_phc_time: Cached PHC time for the Rx queue * @stats_sync: See struct u64_stats_sync * @q_stats: See union idpf_rx_queue_stats @@ -508,15 +514,23 @@ struct idpf_rx_queue { struct page_pool *pp; }; }; - struct net_device *netdev; void __iomem *tail; =20 DECLARE_BITMAP(flags, __IDPF_Q_FLAGS_NBITS); u16 idx; u16 desc_count; =20 - u32 rxdids; + u32 num_xdp_txq; + union { + struct idpf_tx_queue **xdpsqs; + struct { + u32 rxdids; + u32 truesize; + }; + }; const struct libeth_rx_pt *rx_ptype_lkup; + + struct xdp_rxq_info xdp_rxq; __cacheline_group_end_aligned(read_mostly); =20 __cacheline_group_begin_aligned(read_write); @@ -525,7 +539,6 @@ struct idpf_rx_queue { u16 next_to_alloc; =20 struct sk_buff *skb; - u32 truesize; u64 cached_phc_time; =20 struct u64_stats_sync stats_sync; @@ -545,8 +558,11 @@ struct idpf_rx_queue { u16 rx_max_pkt_size; __cacheline_group_end_aligned(cold); }; -libeth_cacheline_set_assert(struct idpf_rx_queue, 64, - 88 + sizeof(struct u64_stats_sync), +libeth_cacheline_set_assert(struct idpf_rx_queue, + ALIGN(64, __alignof(struct xdp_rxq_info)) + + sizeof(struct xdp_rxq_info), + 72 + offsetof(struct idpf_rx_queue, q_stats) - + offsetofend(struct idpf_rx_queue, skb), 32); =20 /** @@ -558,6 +574,7 @@ libeth_cacheline_set_assert(struct idpf_rx_queue, 64, * @desc_ring: virtual descriptor ring address * @tx_buf: See struct idpf_tx_buf * @txq_grp: See struct idpf_txq_group + * @complq: corresponding completion queue in XDP mode * @dev: Device back pointer for DMA mapping * @tail: Tail offset. Used for both queue models single and split * @flags: See enum idpf_queue_flags_t @@ -565,26 +582,7 @@ libeth_cacheline_set_assert(struct idpf_rx_queue, 64, * hot path TX pointers stored in vport. Used in both singleq/splitq. * @desc_count: Number of descriptors * @tx_min_pkt_len: Min supported packet length - * @compl_tag_gen_s: Completion tag generation bit - * The format of the completion tag will change based on the TXQ - * descriptor ring size so that we can maintain roughly the same level - * of "uniqueness" across all descriptor sizes. For example, if the - * TXQ descriptor ring size is 64 (the minimum size supported), the - * completion tag will be formatted as below: - * 15 6 5 0 - * -------------------------------- - * | GEN=3D0-1023 |IDX =3D 0-63| - * -------------------------------- - * - * This gives us 64*1024 =3D 65536 possible unique values. Similarly, if - * the TXQ descriptor ring size is 8160 (the maximum size supported), - * the completion tag will be formatted as below: - * 15 13 12 0 - * -------------------------------- - * |GEN | IDX =3D 0-8159 | - * -------------------------------- - * - * This gives us 8*8160 =3D 65280 possible unique values. + * @thresh: XDP queue cleaning threshold * @netdev: &net_device corresponding to this queue * @next_to_use: Next descriptor to use * @next_to_clean: Next descriptor to clean @@ -601,6 +599,10 @@ libeth_cacheline_set_assert(struct idpf_rx_queue, 64, * @clean_budget: singleq only, queue cleaning budget * @cleaned_pkts: Number of packets cleaned for the above said case * @refillq: Pointer to refill queue + * @pending: number of pending descriptors to send in QB + * @xdp_tx: number of pending &xdp_buff or &xdp_frame buffers + * @timer: timer for XDP Tx queue cleanup + * @xdp_lock: lock for XDP Tx queues sharing * @cached_tstamp_caps: Tx timestamp capabilities negotiated with the CP * @tstamp_task: Work that handles Tx timestamp read * @stats_sync: See struct u64_stats_sync @@ -622,7 +624,10 @@ struct idpf_tx_queue { void *desc_ring; }; struct libeth_sqe *tx_buf; - struct idpf_txq_group *txq_grp; + union { + struct idpf_txq_group *txq_grp; + struct idpf_compl_queue *complq; + }; struct device *dev; void __iomem *tail; =20 @@ -630,7 +635,10 @@ struct idpf_tx_queue { u16 idx; u16 desc_count; =20 - u16 tx_min_pkt_len; + union { + u16 tx_min_pkt_len; + u32 thresh; + }; =20 struct net_device *netdev; __cacheline_group_end_aligned(read_mostly); @@ -638,16 +646,28 @@ struct idpf_tx_queue { __cacheline_group_begin_aligned(read_write); u16 next_to_use; u16 next_to_clean; - u16 last_re; - u16 tx_max_bufs; =20 union { - u32 cleaned_bytes; - u32 clean_budget; - }; - u16 cleaned_pkts; + struct { + u16 last_re; + u16 tx_max_bufs; =20 - struct idpf_sw_queue *refillq; + union { + u32 cleaned_bytes; + u32 clean_budget; + }; + u16 cleaned_pkts; + + struct idpf_sw_queue *refillq; + }; + struct { + u32 pending; + u32 xdp_tx; + + struct libeth_xdpsq_timer *timer; + struct libeth_xdpsq_lock xdp_lock; + }; + }; =20 struct idpf_ptp_vport_tx_tstamp_caps *cached_tstamp_caps; struct work_struct *tstamp_task; @@ -666,7 +686,11 @@ struct idpf_tx_queue { __cacheline_group_end_aligned(cold); }; libeth_cacheline_set_assert(struct idpf_tx_queue, 64, - 104 + sizeof(struct u64_stats_sync), + 104 + + offsetof(struct idpf_tx_queue, cached_tstamp_caps) - + offsetofend(struct idpf_tx_queue, timer) + + offsetof(struct idpf_tx_queue, q_stats) - + offsetofend(struct idpf_tx_queue, tstamp_task), 32); =20 /** diff --git a/drivers/net/ethernet/intel/idpf/xdp.h b/drivers/net/ethernet/i= ntel/idpf/xdp.h new file mode 100644 index 000000000000..cf6823b24ba5 --- /dev/null +++ b/drivers/net/ethernet/intel/idpf/xdp.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (C) 2025 Intel Corporation */ + +#ifndef _IDPF_XDP_H_ +#define _IDPF_XDP_H_ + +#include + +struct idpf_vport; + +int idpf_xdp_rxq_info_init_all(const struct idpf_vport *vport); +void idpf_xdp_rxq_info_deinit_all(const struct idpf_vport *vport); + +int idpf_xdpsqs_get(const struct idpf_vport *vport); +void idpf_xdpsqs_put(const struct idpf_vport *vport); + +#endif /* _IDPF_XDP_H_ */ diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ether= net/intel/idpf/idpf_lib.c index baf1f9b196d5..0427f2e86fb8 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -4,6 +4,7 @@ #include "idpf.h" #include "idpf_virtchnl.h" #include "idpf_ptp.h" +#include "xdp.h" =20 static const struct net_device_ops idpf_netdev_ops; =20 @@ -914,6 +915,7 @@ static void idpf_vport_stop(struct idpf_vport *vport, b= ool rtnl) =20 vport->link_up =3D false; idpf_vport_intr_deinit(vport); + idpf_xdp_rxq_info_deinit_all(vport); idpf_vport_queues_rel(vport); idpf_vport_intr_rel(vport); np->state =3D __IDPF_VPORT_DOWN; @@ -1316,13 +1318,13 @@ static void idpf_restore_features(struct idpf_vport= *vport) */ static int idpf_set_real_num_queues(struct idpf_vport *vport) { - int err; + int err, txq =3D vport->num_txq - vport->num_xdp_txq; =20 err =3D netif_set_real_num_rx_queues(vport->netdev, vport->num_rxq); if (err) return err; =20 - return netif_set_real_num_tx_queues(vport->netdev, vport->num_txq); + return netif_set_real_num_tx_queues(vport->netdev, txq); } =20 /** @@ -1435,20 +1437,29 @@ static int idpf_vport_open(struct idpf_vport *vport= , bool rtnl) } =20 idpf_rx_init_buf_tail(vport); + + err =3D idpf_xdp_rxq_info_init_all(vport); + if (err) { + netdev_err(vport->netdev, + "Failed to initialize XDP RxQ info for vport %u: %pe\n", + vport->vport_id, ERR_PTR(err)); + goto intr_deinit; + } + idpf_vport_intr_ena(vport); =20 err =3D idpf_send_config_queues_msg(vport); if (err) { dev_err(&adapter->pdev->dev, "Failed to configure queues for vport %u, %= d\n", vport->vport_id, err); - goto intr_deinit; + goto rxq_deinit; } =20 err =3D idpf_send_map_unmap_queue_vector_msg(vport, true); if (err) { dev_err(&adapter->pdev->dev, "Failed to map queue vectors for vport %u: = %d\n", vport->vport_id, err); - goto intr_deinit; + goto rxq_deinit; } =20 err =3D idpf_send_enable_queues_msg(vport); @@ -1499,6 +1510,8 @@ static int idpf_vport_open(struct idpf_vport *vport, = bool rtnl) idpf_send_disable_queues_msg(vport); unmap_queue_vectors: idpf_send_map_unmap_queue_vector_msg(vport, false); +rxq_deinit: + idpf_xdp_rxq_info_deinit_all(vport); intr_deinit: idpf_vport_intr_deinit(vport); queues_rel: diff --git a/drivers/net/ethernet/intel/idpf/idpf_main.c b/drivers/net/ethe= rnet/intel/idpf/idpf_main.c index dfe9126f1f4a..8c46481d2e1f 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_main.c +++ b/drivers/net/ethernet/intel/idpf/idpf_main.c @@ -9,6 +9,7 @@ =20 MODULE_DESCRIPTION(DRV_SUMMARY); MODULE_IMPORT_NS("LIBETH"); +MODULE_IMPORT_NS("LIBETH_XDP"); MODULE_LICENSE("GPL"); =20 /** diff --git a/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c b/drivers/= net/ethernet/intel/idpf/idpf_singleq_txrx.c index b19b462e0bb6..178c2f3825e3 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c @@ -655,7 +655,7 @@ static void idpf_rx_singleq_csum(struct idpf_rx_queue *= rxq, bool ipv4, ipv6; =20 /* check if Rx checksum is enabled */ - if (!libeth_rx_pt_has_checksum(rxq->netdev, decoded)) + if (!libeth_rx_pt_has_checksum(rxq->xdp_rxq.dev, decoded)) return; =20 /* check if HW has decoded the packet and checksum */ @@ -794,7 +794,7 @@ static void idpf_rx_singleq_base_hash(struct idpf_rx_qu= eue *rx_q, { u64 mask, qw1; =20 - if (!libeth_rx_pt_has_hash(rx_q->netdev, decoded)) + if (!libeth_rx_pt_has_hash(rx_q->xdp_rxq.dev, decoded)) return; =20 mask =3D VIRTCHNL2_RX_BASE_DESC_FLTSTAT_RSS_HASH_M; @@ -822,7 +822,7 @@ static void idpf_rx_singleq_flex_hash(struct idpf_rx_qu= eue *rx_q, const union virtchnl2_rx_desc *rx_desc, struct libeth_rx_pt decoded) { - if (!libeth_rx_pt_has_hash(rx_q->netdev, decoded)) + if (!libeth_rx_pt_has_hash(rx_q->xdp_rxq.dev, decoded)) return; =20 if (FIELD_GET(VIRTCHNL2_RX_FLEX_DESC_STATUS0_RSS_VALID_M, @@ -855,7 +855,7 @@ idpf_rx_singleq_process_skb_fields(struct idpf_rx_queue= *rx_q, struct libeth_rx_csum csum_bits; =20 /* modifies the skb - consumes the enet header */ - skb->protocol =3D eth_type_trans(skb, rx_q->netdev); + skb->protocol =3D eth_type_trans(skb, rx_q->xdp_rxq.dev); =20 /* Check if we're using base mode descriptor IDs */ if (rx_q->rxdids =3D=3D VIRTCHNL2_RXDID_1_32B_BASE_M) { diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index ceab25ee1aad..ecdd9d88f4dc 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -1,12 +1,12 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright (C) 2023 Intel Corporation */ =20 -#include -#include +#include =20 #include "idpf.h" #include "idpf_ptp.h" #include "idpf_virtchnl.h" +#include "xdp.h" =20 #define idpf_tx_buf_next(buf) (*(u32 *)&(buf)->priv) LIBETH_SQE_CHECK_PRIV(u32); @@ -62,8 +62,10 @@ void idpf_tx_timeout(struct net_device *netdev, unsigned= int txqueue) static void idpf_tx_buf_rel_all(struct idpf_tx_queue *txq) { struct libeth_sq_napi_stats ss =3D { }; + struct xdp_frame_bulk bq; struct libeth_cq_pp cp =3D { .dev =3D txq->dev, + .bq =3D &bq, .ss =3D &ss, }; u32 i; @@ -72,9 +74,13 @@ static void idpf_tx_buf_rel_all(struct idpf_tx_queue *tx= q) if (!txq->tx_buf) return; =20 + xdp_frame_bulk_init(&bq); + /* Free all the Tx buffer sk_buffs */ for (i =3D 0; i < txq->buf_pool_size; i++) - libeth_tx_complete(&txq->tx_buf[i], &cp); + libeth_tx_complete_any(&txq->tx_buf[i], &cp); + + xdp_flush_frame_bulk(&bq); =20 kfree(txq->tx_buf); txq->tx_buf =3D NULL; @@ -88,13 +94,20 @@ static void idpf_tx_buf_rel_all(struct idpf_tx_queue *t= xq) */ static void idpf_tx_desc_rel(struct idpf_tx_queue *txq) { + bool xdp =3D idpf_queue_has(XDP, txq); + + if (xdp) + libeth_xdpsq_deinit_timer(txq->timer); + idpf_tx_buf_rel_all(txq); - netdev_tx_reset_subqueue(txq->netdev, txq->idx); + + if (!xdp) + netdev_tx_reset_subqueue(txq->netdev, txq->idx); =20 if (!txq->desc_ring) return; =20 - if (txq->refillq) + if (!xdp && txq->refillq) kfree(txq->refillq->ring); =20 dmam_free_coherent(txq->dev, txq->size, txq->desc_ring, txq->dma); @@ -520,6 +533,7 @@ static int idpf_rx_hdr_buf_alloc_all(struct idpf_buf_qu= eue *bufq) struct libeth_fq fq =3D { .count =3D bufq->desc_count, .type =3D LIBETH_FQE_HDR, + .xdp =3D idpf_xdp_enabled(bufq->q_vector->vport), .nid =3D idpf_q_vector_to_mem(bufq->q_vector), }; int ret; @@ -719,6 +733,7 @@ static int idpf_rx_bufs_init(struct idpf_buf_queue *buf= q, .count =3D bufq->desc_count, .type =3D type, .hsplit =3D idpf_queue_has(HSPLIT_EN, bufq), + .xdp =3D idpf_xdp_enabled(bufq->q_vector->vport), .nid =3D idpf_q_vector_to_mem(bufq->q_vector), }; int ret; @@ -1026,6 +1041,8 @@ void idpf_vport_queues_rel(struct idpf_vport *vport) { idpf_tx_desc_rel_all(vport); idpf_rx_desc_rel_all(vport); + + idpf_xdpsqs_put(vport); idpf_vport_queue_grp_rel_all(vport); =20 kfree(vport->txqs); @@ -1099,6 +1116,18 @@ void idpf_vport_init_num_qs(struct idpf_vport *vport, if (idpf_is_queue_model_split(vport->rxq_model)) vport->num_bufq =3D le16_to_cpu(vport_msg->num_rx_bufq); =20 + vport->xdp_prog =3D config_data->xdp_prog; + if (idpf_xdp_enabled(vport)) { + vport->xdp_txq_offset =3D config_data->num_req_tx_qs; + vport->num_xdp_txq =3D le16_to_cpu(vport_msg->num_tx_q) - + vport->xdp_txq_offset; + vport->xdpsq_share =3D libeth_xdpsq_shared(vport->num_xdp_txq); + } else { + vport->xdp_txq_offset =3D 0; + vport->num_xdp_txq =3D 0; + vport->xdpsq_share =3D false; + } + /* Adjust number of buffer queues per Rx queue group. */ if (!idpf_is_queue_model_split(vport->rxq_model)) { vport->num_bufqs_per_qgrp =3D 0; @@ -1170,9 +1199,10 @@ int idpf_vport_calc_total_qs(struct idpf_adapter *ad= apter, u16 vport_idx, int dflt_splitq_txq_grps =3D 0, dflt_singleq_txqs =3D 0; int dflt_splitq_rxq_grps =3D 0, dflt_singleq_rxqs =3D 0; u16 num_req_tx_qs =3D 0, num_req_rx_qs =3D 0; + struct idpf_vport_user_config_data *user; struct idpf_vport_config *vport_config; u16 num_txq_grps, num_rxq_grps; - u32 num_qs; + u32 num_qs, num_xdpsq; =20 vport_config =3D adapter->vport_config[vport_idx]; if (vport_config) { @@ -1214,6 +1244,24 @@ int idpf_vport_calc_total_qs(struct idpf_adapter *ad= apter, u16 vport_idx, vport_msg->num_rx_bufq =3D 0; } =20 + if (!vport_config) + return 0; + + user =3D &vport_config->user_config; + user->num_req_rx_qs =3D le16_to_cpu(vport_msg->num_rx_q); + user->num_req_tx_qs =3D le16_to_cpu(vport_msg->num_tx_q); + + if (vport_config->user_config.xdp_prog) + num_xdpsq =3D libeth_xdpsq_num(user->num_req_rx_qs, + user->num_req_tx_qs, + vport_config->max_q.max_txq); + else + num_xdpsq =3D 0; + + vport_msg->num_tx_q =3D cpu_to_le16(user->num_req_tx_qs + num_xdpsq); + if (idpf_is_queue_model_split(le16_to_cpu(vport_msg->txq_model))) + vport_msg->num_tx_complq =3D vport_msg->num_tx_q; + return 0; } =20 @@ -1263,14 +1311,13 @@ static void idpf_vport_calc_numq_per_grp(struct idp= f_vport *vport, static void idpf_rxq_set_descids(const struct idpf_vport *vport, struct idpf_rx_queue *q) { - if (idpf_is_queue_model_split(vport->rxq_model)) { - q->rxdids =3D VIRTCHNL2_RXDID_2_FLEX_SPLITQ_M; - } else { - if (vport->base_rxd) - q->rxdids =3D VIRTCHNL2_RXDID_1_32B_BASE_M; - else - q->rxdids =3D VIRTCHNL2_RXDID_2_FLEX_SQ_NIC_M; - } + if (idpf_is_queue_model_split(vport->rxq_model)) + return; + + if (vport->base_rxd) + q->rxdids =3D VIRTCHNL2_RXDID_1_32B_BASE_M; + else + q->rxdids =3D VIRTCHNL2_RXDID_2_FLEX_SQ_NIC_M; } =20 /** @@ -1478,7 +1525,6 @@ static int idpf_rxq_group_alloc(struct idpf_vport *vp= ort, u16 num_rxq) setup_rxq: q->desc_count =3D vport->rxq_desc_count; q->rx_ptype_lkup =3D vport->rx_ptype_lkup; - q->netdev =3D vport->netdev; q->bufq_sets =3D rx_qgrp->splitq.bufq_sets; q->idx =3D (i * num_rxq) + j; q->rx_buffer_low_watermark =3D IDPF_LOW_WATERMARK; @@ -1539,15 +1585,19 @@ int idpf_vport_queues_alloc(struct idpf_vport *vpor= t) if (err) goto err_out; =20 - err =3D idpf_tx_desc_alloc_all(vport); + err =3D idpf_vport_init_fast_path_txqs(vport); if (err) goto err_out; =20 - err =3D idpf_rx_desc_alloc_all(vport); + err =3D idpf_xdpsqs_get(vport); if (err) goto err_out; =20 - err =3D idpf_vport_init_fast_path_txqs(vport); + err =3D idpf_tx_desc_alloc_all(vport); + if (err) + goto err_out; + + err =3D idpf_rx_desc_alloc_all(vport); if (err) goto err_out; =20 @@ -1888,14 +1938,17 @@ static bool idpf_tx_clean_complq(struct idpf_compl_= queue *complq, int budget, */ void idpf_wait_for_sw_marker_completion(const struct idpf_tx_queue *txq) { - struct idpf_compl_queue *complq =3D txq->txq_grp->complq; - u32 ntc =3D complq->next_to_clean; + struct idpf_compl_queue *complq; unsigned long timeout; bool flow, gen_flag; + u32 ntc; =20 if (!idpf_queue_has(SW_MARKER, txq)) return; =20 + complq =3D idpf_queue_has(XDP, txq) ? txq->complq : txq->txq_grp->complq; + ntc =3D complq->next_to_clean; + flow =3D idpf_queue_has(FLOW_SCH_EN, complq); gen_flag =3D idpf_queue_has(GEN_CHK, complq); =20 @@ -2701,10 +2754,11 @@ static netdev_tx_t idpf_tx_splitq_frame(struct sk_b= uff *skb, */ netdev_tx_t idpf_tx_start(struct sk_buff *skb, struct net_device *netdev) { - struct idpf_vport *vport =3D idpf_netdev_to_vport(netdev); + const struct idpf_vport *vport =3D idpf_netdev_to_vport(netdev); struct idpf_tx_queue *tx_q; =20 - if (unlikely(skb_get_queue_mapping(skb) >=3D vport->num_txq)) { + if (unlikely(skb_get_queue_mapping(skb) >=3D + vport->num_txq - vport->num_xdp_txq)) { dev_kfree_skb_any(skb); =20 return NETDEV_TX_OK; @@ -2741,7 +2795,7 @@ idpf_rx_hash(const struct idpf_rx_queue *rxq, struct = sk_buff *skb, { u32 hash; =20 - if (!libeth_rx_pt_has_hash(rxq->netdev, decoded)) + if (!libeth_rx_pt_has_hash(rxq->xdp_rxq.dev, decoded)) return; =20 hash =3D le16_to_cpu(rx_desc->hash1) | @@ -2767,7 +2821,7 @@ static void idpf_rx_csum(struct idpf_rx_queue *rxq, s= truct sk_buff *skb, bool ipv4, ipv6; =20 /* check if Rx checksum is enabled */ - if (!libeth_rx_pt_has_checksum(rxq->netdev, decoded)) + if (!libeth_rx_pt_has_checksum(rxq->xdp_rxq.dev, decoded)) return; =20 /* check if HW has decoded the packet and checksum */ @@ -2966,7 +3020,7 @@ idpf_rx_process_skb_fields(struct idpf_rx_queue *rxq,= struct sk_buff *skb, if (idpf_queue_has(PTP, rxq)) idpf_rx_hwtstamp(rxq, rx_desc, skb); =20 - skb->protocol =3D eth_type_trans(skb, rxq->netdev); + skb->protocol =3D eth_type_trans(skb, rxq->xdp_rxq.dev); skb_record_rx_queue(skb, rxq->idx); =20 if (le16_get_bits(rx_desc->hdrlen_flags, @@ -3967,8 +4021,8 @@ static int idpf_vport_splitq_napi_poll(struct napi_st= ruct *napi, int budget) */ static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport) { + u16 num_txq_grp =3D vport->num_txq_grp - vport->num_xdp_txq; bool split =3D idpf_is_queue_model_split(vport->rxq_model); - u16 num_txq_grp =3D vport->num_txq_grp; struct idpf_rxq_group *rx_qgrp; struct idpf_txq_group *tx_qgrp; u32 i, qv_idx, q_index; diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c b/drivers/net/= ethernet/intel/idpf/idpf_virtchnl.c index ff4ea49c0957..22f08c7682b5 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c +++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c @@ -1748,9 +1748,12 @@ static int idpf_send_config_rx_queues_msg(struct idp= f_vport *vport) for (j =3D 0; j < num_rxq; j++, k++) { const struct idpf_bufq_set *sets; struct idpf_rx_queue *rxq; + u32 rxdids; =20 if (!idpf_is_queue_model_split(vport->rxq_model)) { rxq =3D rx_qgrp->singleq.rxqs[j]; + rxdids =3D rxq->rxdids; + goto common_qi_fields; } =20 @@ -1783,6 +1786,8 @@ static int idpf_send_config_rx_queues_msg(struct idpf= _vport *vport) cpu_to_le16(rxq->rx_hbuf_size); } =20 + rxdids =3D VIRTCHNL2_RXDID_2_FLEX_SPLITQ_M; + common_qi_fields: qi[k].queue_id =3D cpu_to_le32(rxq->q_id); qi[k].model =3D cpu_to_le16(vport->rxq_model); @@ -1793,7 +1798,7 @@ static int idpf_send_config_rx_queues_msg(struct idpf= _vport *vport) qi[k].data_buffer_size =3D cpu_to_le32(rxq->rx_buf_size); qi[k].qflags |=3D cpu_to_le16(VIRTCHNL2_RX_DESC_SIZE_32BYTE); - qi[k].desc_ids =3D cpu_to_le64(rxq->rxdids); + qi[k].desc_ids =3D cpu_to_le64(rxdids); } } =20 @@ -2028,6 +2033,8 @@ int idpf_send_map_unmap_queue_vector_msg(struct idpf_= vport *vport, bool map) =20 if (idpf_queue_has(NOIRQ, txq)) vec =3D NULL; + else if (idpf_queue_has(XDP, txq)) + vec =3D txq->complq->q_vector; else if (idpf_is_queue_model_split(vport->txq_model)) vec =3D txq->txq_grp->complq->q_vector; else @@ -2046,9 +2053,6 @@ int idpf_send_map_unmap_queue_vector_msg(struct idpf_= vport *vport, bool map) } } =20 - if (vport->num_txq !=3D k) - return -EINVAL; - for (i =3D 0; i < vport->num_rxq_grp; i++) { struct idpf_rxq_group *rx_qgrp =3D &vport->rxq_grps[i]; u16 num_rxq; @@ -2084,13 +2088,8 @@ int idpf_send_map_unmap_queue_vector_msg(struct idpf= _vport *vport, bool map) } } =20 - if (idpf_is_queue_model_split(vport->txq_model)) { - if (vport->num_rxq !=3D k - vport->num_complq) - return -EINVAL; - } else { - if (vport->num_rxq !=3D k - vport->num_txq) - return -EINVAL; - } + if (k !=3D num_q) + return -EINVAL; =20 /* Chunk up the vector info into multiple messages */ config_sz =3D sizeof(struct virtchnl2_queue_vector_maps); @@ -3307,7 +3306,9 @@ int idpf_vport_alloc_vec_indexes(struct idpf_vport *v= port) if (vec_info.num_curr_vecs) vec_info.num_curr_vecs +=3D IDPF_RESERVED_VECS; =20 - req =3D max(vport->num_txq, vport->num_rxq) + IDPF_RESERVED_VECS; + /* XDPSQs are all bound to the NOIRQ vector from IDPF_RESERVED_VECS */ + req =3D max(vport->num_txq - vport->num_xdp_txq, vport->num_rxq) + + IDPF_RESERVED_VECS; vec_info.num_req_vecs =3D req; =20 vec_info.default_vport =3D vport->default_vport; diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/i= ntel/idpf/xdp.c new file mode 100644 index 000000000000..98bdccc0c957 --- /dev/null +++ b/drivers/net/ethernet/intel/idpf/xdp.c @@ -0,0 +1,168 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (C) 2025 Intel Corporation */ + +#include + +#include "idpf.h" +#include "xdp.h" + +static int idpf_rxq_for_each(const struct idpf_vport *vport, + int (*fn)(struct idpf_rx_queue *rxq, void *arg), + void *arg) +{ + bool splitq =3D idpf_is_queue_model_split(vport->rxq_model); + + if (!vport->rxq_grps) + return -ENETDOWN; + + for (u32 i =3D 0; i < vport->num_rxq_grp; i++) { + const struct idpf_rxq_group *rx_qgrp =3D &vport->rxq_grps[i]; + u32 num_rxq; + + if (splitq) + num_rxq =3D rx_qgrp->splitq.num_rxq_sets; + else + num_rxq =3D rx_qgrp->singleq.num_rxq; + + for (u32 j =3D 0; j < num_rxq; j++) { + struct idpf_rx_queue *q; + int err; + + if (splitq) + q =3D &rx_qgrp->splitq.rxq_sets[j]->rxq; + else + q =3D rx_qgrp->singleq.rxqs[j]; + + err =3D fn(q, arg); + if (err) + return err; + } + } + + return 0; +} + +static int __idpf_xdp_rxq_info_init(struct idpf_rx_queue *rxq, void *arg) +{ + const struct idpf_vport *vport =3D rxq->q_vector->vport; + bool split =3D idpf_is_queue_model_split(vport->rxq_model); + const struct page_pool *pp; + int err; + + err =3D __xdp_rxq_info_reg(&rxq->xdp_rxq, vport->netdev, rxq->idx, + rxq->q_vector->napi.napi_id, + rxq->rx_buf_size); + if (err) + return err; + + pp =3D split ? rxq->bufq_sets[0].bufq.pp : rxq->pp; + xdp_rxq_info_attach_page_pool(&rxq->xdp_rxq, pp); + + if (!split) + return 0; + + rxq->xdpsqs =3D &vport->txqs[vport->xdp_txq_offset]; + rxq->num_xdp_txq =3D vport->num_xdp_txq; + + return 0; +} + +int idpf_xdp_rxq_info_init_all(const struct idpf_vport *vport) +{ + return idpf_rxq_for_each(vport, __idpf_xdp_rxq_info_init, NULL); +} + +static int __idpf_xdp_rxq_info_deinit(struct idpf_rx_queue *rxq, void *arg) +{ + if (idpf_is_queue_model_split((size_t)arg)) { + rxq->xdpsqs =3D NULL; + rxq->num_xdp_txq =3D 0; + } + + xdp_rxq_info_detach_mem_model(&rxq->xdp_rxq); + xdp_rxq_info_unreg(&rxq->xdp_rxq); + + return 0; +} + +void idpf_xdp_rxq_info_deinit_all(const struct idpf_vport *vport) +{ + idpf_rxq_for_each(vport, __idpf_xdp_rxq_info_deinit, + (void *)(size_t)vport->rxq_model); +} + +int idpf_xdpsqs_get(const struct idpf_vport *vport) +{ + struct libeth_xdpsq_timer **timers __free(kvfree) =3D NULL; + struct net_device *dev; + u32 sqs; + + if (!idpf_xdp_enabled(vport)) + return 0; + + timers =3D kvcalloc(vport->num_xdp_txq, sizeof(*timers), GFP_KERNEL); + if (!timers) + return -ENOMEM; + + for (u32 i =3D 0; i < vport->num_xdp_txq; i++) { + timers[i] =3D kzalloc_node(sizeof(*timers[i]), GFP_KERNEL, + cpu_to_mem(i)); + if (!timers[i]) { + for (int j =3D i - 1; j >=3D 0; j--) + kfree(timers[j]); + + return -ENOMEM; + } + } + + dev =3D vport->netdev; + sqs =3D vport->xdp_txq_offset; + + for (u32 i =3D sqs; i < vport->num_txq; i++) { + struct idpf_tx_queue *xdpsq =3D vport->txqs[i]; + + xdpsq->complq =3D xdpsq->txq_grp->complq; + kfree(xdpsq->refillq); + xdpsq->refillq =3D NULL; + + idpf_queue_clear(FLOW_SCH_EN, xdpsq); + idpf_queue_clear(FLOW_SCH_EN, xdpsq->complq); + idpf_queue_set(NOIRQ, xdpsq); + idpf_queue_set(XDP, xdpsq); + idpf_queue_set(XDP, xdpsq->complq); + + xdpsq->timer =3D timers[i - sqs]; + libeth_xdpsq_get(&xdpsq->xdp_lock, dev, vport->xdpsq_share); + + xdpsq->pending =3D 0; + xdpsq->xdp_tx =3D 0; + xdpsq->thresh =3D libeth_xdp_queue_threshold(xdpsq->desc_count); + } + + return 0; +} + +void idpf_xdpsqs_put(const struct idpf_vport *vport) +{ + struct net_device *dev; + u32 sqs; + + if (!idpf_xdp_enabled(vport)) + return; + + dev =3D vport->netdev; + sqs =3D vport->xdp_txq_offset; + + for (u32 i =3D sqs; i < vport->num_txq; i++) { + struct idpf_tx_queue *xdpsq =3D vport->txqs[i]; + + if (!idpf_queue_has_clear(XDP, xdpsq)) + continue; + + libeth_xdpsq_put(&xdpsq->xdp_lock, dev); + + kfree(xdpsq->timer); + xdpsq->refillq =3D NULL; + idpf_queue_clear(NOIRQ, xdpsq); + } +} --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2EB4314B98; Tue, 26 Aug 2025 16:14:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224849; cv=none; b=CThZ3LylIVP5ikzF/lw4CTuuDW6XYp6ycOf21oKOr27OQCaLn5NN8i1jXc6qdM/P1ytw6KdMqGsNJvOqQTmx3GuS3po4XSGsH7GYXR7nSFYwM9OK64Y/oPtPS8kdoTT5pzZq2ClDk3zBjqGn3gJuNn0ozIBrR54crQa4aI6Drq4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224849; c=relaxed/simple; bh=FByDu3fwk48hL4vjTUA5aayxoLqy+QrYRAj5aKjJU0Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LMZcnGT1QtkFV3LWDDwrjDbNzFBdTLil9++suHnAvoDynajrVHflbokCRDN/yMsOewg+Bx3jVAKz8zG3g4Q0t3WK+HS5NVWXetCtWywArzbijYHHEN+gi8Nb6FDG0SjimVWttAy7TT469WFM63Piqpo1ZAyqWDagzgjtQq/FMNk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=mOzAH3sc; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="mOzAH3sc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224848; x=1787760848; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FByDu3fwk48hL4vjTUA5aayxoLqy+QrYRAj5aKjJU0Y=; b=mOzAH3scU7OtUuSGYtBVRF2vVB1SSymxAVELz+r9b9gDB38KZ7DiF3Ww dJJgjbcDfyvBVtuDhF6d8/EAZ0QgPMRnqcwd+19vo6fktOgPd4y2WY07l 7UG+bWfPvT/YWbJ+tcvq1p5h8Vnu4/UUuXesW3x1LVejabpIEBiQPszXl KYNQN+01gZR3mOe3Qcu3cyDDeQA2KEXrLFuN11lf2vIKAWeYIhGtLe/Dn EhxgZ2CafqPLpe3BnQtir5E0fJp0mh33GQm5XdKHExFOym1M9dDemflbI DQqmCofm0NHxekKpFRQ/yfTN5RTUMAb5/OywnfLn2X/dcVN/R+eqJ0/RV g==; X-CSE-ConnectionGUID: XptqlmPrSGW4fauhYrGalA== X-CSE-MsgGUID: duGgsq0+QbiXrIZqSxgPlw== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46045039" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46045039" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:14:07 -0700 X-CSE-ConnectionGUID: RxFXdrvITSyeQm7A1OjNww== X-CSE-MsgGUID: /8vddwJ3Rmq3+c8lOeqOGw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562489" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:14:03 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 09/13] idpf: implement XDP_SETUP_PROG in ndo_bpf for splitq Date: Tue, 26 Aug 2025 17:55:03 +0200 Message-ID: <20250826155507.2138401-10-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Michal Kubiak Implement loading/removing XDP program using .ndo_bpf callback in the split queue mode. Reconfigure and restart the queues if needed (!!old_prog !=3D !!new_prog), otherwise, just update the pointers. Signed-off-by: Michal Kubiak Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/idpf_txrx.h | 4 +- drivers/net/ethernet/intel/idpf/xdp.h | 7 ++ drivers/net/ethernet/intel/idpf/idpf_lib.c | 1 + drivers/net/ethernet/intel/idpf/idpf_txrx.c | 4 + drivers/net/ethernet/intel/idpf/xdp.c | 97 +++++++++++++++++++++ 5 files changed, 112 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.h index 6bc204b68d9e..f898a9c8de1d 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -467,6 +467,7 @@ struct idpf_tx_queue_stats { * @desc_ring: virtual descriptor ring address * @bufq_sets: Pointer to the array of buffer queues in splitq mode * @napi: NAPI instance corresponding to this queue (splitq) + * @xdp_prog: attached XDP program * @rx_buf: See struct &libeth_fqe * @pp: Page pool pointer in singleq mode * @tail: Tail offset. Used for both queue models single and split. @@ -508,13 +509,14 @@ struct idpf_rx_queue { struct { struct idpf_bufq_set *bufq_sets; struct napi_struct *napi; + struct bpf_prog __rcu *xdp_prog; }; struct { struct libeth_fqe *rx_buf; struct page_pool *pp; + void __iomem *tail; }; }; - void __iomem *tail; =20 DECLARE_BITMAP(flags, __IDPF_Q_FLAGS_NBITS); u16 idx; diff --git a/drivers/net/ethernet/intel/idpf/xdp.h b/drivers/net/ethernet/i= ntel/idpf/xdp.h index cf6823b24ba5..47553ce5f81a 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.h +++ b/drivers/net/ethernet/intel/idpf/xdp.h @@ -6,12 +6,19 @@ =20 #include =20 +struct bpf_prog; struct idpf_vport; +struct net_device; +struct netdev_bpf; =20 int idpf_xdp_rxq_info_init_all(const struct idpf_vport *vport); void idpf_xdp_rxq_info_deinit_all(const struct idpf_vport *vport); +void idpf_xdp_copy_prog_to_rqs(const struct idpf_vport *vport, + struct bpf_prog *xdp_prog); =20 int idpf_xdpsqs_get(const struct idpf_vport *vport); void idpf_xdpsqs_put(const struct idpf_vport *vport); =20 +int idpf_xdp(struct net_device *dev, struct netdev_bpf *xdp); + #endif /* _IDPF_XDP_H_ */ diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ether= net/intel/idpf/idpf_lib.c index 0427f2e86fb8..d645dcc2deda 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -2609,4 +2609,5 @@ static const struct net_device_ops idpf_netdev_ops = =3D { .ndo_tx_timeout =3D idpf_tx_timeout, .ndo_hwtstamp_get =3D idpf_hwtstamp_get, .ndo_hwtstamp_set =3D idpf_hwtstamp_set, + .ndo_bpf =3D idpf_xdp, }; diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index ecdd9d88f4dc..00bf949d60b5 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -761,6 +761,8 @@ int idpf_rx_bufs_init_all(struct idpf_vport *vport) bool split =3D idpf_is_queue_model_split(vport->rxq_model); int i, j, err; =20 + idpf_xdp_copy_prog_to_rqs(vport, vport->xdp_prog); + for (i =3D 0; i < vport->num_rxq_grp; i++) { struct idpf_rxq_group *rx_qgrp =3D &vport->rxq_grps[i]; u32 truesize =3D 0; @@ -1039,6 +1041,8 @@ static void idpf_vport_queue_grp_rel_all(struct idpf_= vport *vport) */ void idpf_vport_queues_rel(struct idpf_vport *vport) { + idpf_xdp_copy_prog_to_rqs(vport, NULL); + idpf_tx_desc_rel_all(vport); idpf_rx_desc_rel_all(vport); =20 diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/i= ntel/idpf/xdp.c index 98bdccc0c957..02f63810632f 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.c +++ b/drivers/net/ethernet/intel/idpf/xdp.c @@ -4,6 +4,7 @@ #include =20 #include "idpf.h" +#include "idpf_virtchnl.h" #include "xdp.h" =20 static int idpf_rxq_for_each(const struct idpf_vport *vport, @@ -91,6 +92,27 @@ void idpf_xdp_rxq_info_deinit_all(const struct idpf_vpor= t *vport) (void *)(size_t)vport->rxq_model); } =20 +static int idpf_xdp_rxq_assign_prog(struct idpf_rx_queue *rxq, void *arg) +{ + struct bpf_prog *prog =3D arg; + struct bpf_prog *old; + + if (prog) + bpf_prog_inc(prog); + + old =3D rcu_replace_pointer(rxq->xdp_prog, prog, lockdep_rtnl_is_held()); + if (old) + bpf_prog_put(old); + + return 0; +} + +void idpf_xdp_copy_prog_to_rqs(const struct idpf_vport *vport, + struct bpf_prog *xdp_prog) +{ + idpf_rxq_for_each(vport, idpf_xdp_rxq_assign_prog, xdp_prog); +} + int idpf_xdpsqs_get(const struct idpf_vport *vport) { struct libeth_xdpsq_timer **timers __free(kvfree) =3D NULL; @@ -166,3 +188,78 @@ void idpf_xdpsqs_put(const struct idpf_vport *vport) idpf_queue_clear(NOIRQ, xdpsq); } } + +static int idpf_xdp_setup_prog(struct idpf_vport *vport, + const struct netdev_bpf *xdp) +{ + const struct idpf_netdev_priv *np =3D netdev_priv(vport->netdev); + struct bpf_prog *old, *prog =3D xdp->prog; + struct idpf_vport_config *cfg; + int ret; + + cfg =3D vport->adapter->vport_config[vport->idx]; + + if (test_bit(IDPF_REMOVE_IN_PROG, vport->adapter->flags) || + !test_bit(IDPF_VPORT_REG_NETDEV, cfg->flags) || + !!vport->xdp_prog =3D=3D !!prog) { + if (np->state =3D=3D __IDPF_VPORT_UP) + idpf_xdp_copy_prog_to_rqs(vport, prog); + + old =3D xchg(&vport->xdp_prog, prog); + if (old) + bpf_prog_put(old); + + cfg->user_config.xdp_prog =3D prog; + + return 0; + } + + if (!vport->num_xdp_txq && vport->num_txq =3D=3D cfg->max_q.max_txq) { + NL_SET_ERR_MSG_MOD(xdp->extack, + "No Tx queues available for XDP, please decrease the number of regu= lar SQs"); + return -ENOSPC; + } + + old =3D cfg->user_config.xdp_prog; + cfg->user_config.xdp_prog =3D prog; + + ret =3D idpf_initiate_soft_reset(vport, IDPF_SR_Q_CHANGE); + if (ret) { + NL_SET_ERR_MSG_MOD(xdp->extack, + "Could not reopen the vport after XDP setup"); + + cfg->user_config.xdp_prog =3D old; + old =3D prog; + } + + if (old) + bpf_prog_put(old); + + return ret; +} + +int idpf_xdp(struct net_device *dev, struct netdev_bpf *xdp) +{ + struct idpf_vport *vport; + int ret; + + idpf_vport_ctrl_lock(dev); + vport =3D idpf_netdev_to_vport(dev); + + if (!idpf_is_queue_model_split(vport->txq_model)) + goto notsupp; + + switch (xdp->command) { + case XDP_SETUP_PROG: + ret =3D idpf_xdp_setup_prog(vport, xdp); + break; + default: +notsupp: + ret =3D -EOPNOTSUPP; + break; + } + + idpf_vport_ctrl_unlock(dev); + + return ret; +} --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68EB2371E84; Tue, 26 Aug 2025 16:14:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224854; cv=none; b=hCoDR7KKDmK/8F2CwXxtCif24vMkreO6fW2Zmetm5+t+b59Z/Gzvju5xQRTftRK7tAuA5tJUWUqOhOxtRkW/slgt6f14ZOILKCwHQGOsJQBE10jjKjNIKNu88VNiR6hcqy6mwxF5KjetRvwsSqF4Z/ki5mquttS3sbYat8CfMyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224854; c=relaxed/simple; bh=cmz2plKi/n6PtlNdlPmb0np0R4ycj8cwkZUfgvCqqkM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c1juMgh/rQjeHgUFIg8g0n5ayVTWGMz+c7KrKYA1cYosaowP8Mff7IulmDxMAvUqQuD1kgKIU6KkvPZ8alNRd/ztuZ1YNWaSluLsohees1z4UFdNLf9N6cUdvBnrRyY6UrUrTpRjVPXARuEYEmFKfBnuAx27qJx+mnRN1FqCdEs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=bejxEifE; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="bejxEifE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224852; x=1787760852; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cmz2plKi/n6PtlNdlPmb0np0R4ycj8cwkZUfgvCqqkM=; b=bejxEifE9RKmPmUpisTM24vSeL/ee6JxPjw+Z6717/jKZQttG6i33+2X gKQefO6+sTK/Kgqz1e0N0LAYawcR/lU2gw3PasqjVNSNvhKagR2Rqa8SU 0wbQjkCQg7Zp7ajIPSHmEy85UTiSkN+KYbnbrGbmArJP4yFiZ9lYEvDaf YbGA2ggUY4WwWwoBd8ZwE4GGINv0t1lo/BYgSp2Un+Td8xCFDE3kMeWQs XO8/bR9ssfsn/9c3GHDjJUHn/irSreOpAr/KDU8TicK1jXvv+4rNh+wLM swO+kymcFf+Tbu1qnEf3okYWJM2ZlIR8LV1dq2VH/WhsGpLE20JQDhzMr Q==; X-CSE-ConnectionGUID: 8eiLLvTzQnKngmjISZPSMQ== X-CSE-MsgGUID: bVxa9T1VRjenXlaEpDiUWg== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46045061" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46045061" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:14:12 -0700 X-CSE-ConnectionGUID: NaPT3GvlT/qH9JY77nARbA== X-CSE-MsgGUID: MYvlFSugQXi/Fg+KdvE0OA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562547" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:14:07 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 10/13] idpf: use generic functions to build xdp_buff and skb Date: Tue, 26 Aug 2025 17:55:04 +0200 Message-ID: <20250826155507.2138401-11-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation of XDP support, move from having skb as the main frame container during the Rx polling to &xdp_buff. This allows to use generic and libeth helpers for building an XDP buffer and changes the logics: now we try to allocate an skb only when we processed all the descriptors related to the frame. Store &libeth_xdp_stash instead of the skb pointer on the Rx queue. It's only 8 bytes wider, but contains everything we may need. Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/idpf_txrx.h | 17 +- .../ethernet/intel/idpf/idpf_singleq_txrx.c | 104 ++++++------- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 145 +++++------------- 3 files changed, 90 insertions(+), 176 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.h index f898a9c8de1d..5039feafdee9 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -484,7 +484,7 @@ struct idpf_tx_queue_stats { * @next_to_use: Next descriptor to use * @next_to_clean: Next descriptor to clean * @next_to_alloc: RX buffer to allocate at - * @skb: Pointer to the skb + * @xdp: XDP buffer with the current frame * @cached_phc_time: Cached PHC time for the Rx queue * @stats_sync: See struct u64_stats_sync * @q_stats: See union idpf_rx_queue_stats @@ -536,11 +536,11 @@ struct idpf_rx_queue { __cacheline_group_end_aligned(read_mostly); =20 __cacheline_group_begin_aligned(read_write); - u16 next_to_use; - u16 next_to_clean; - u16 next_to_alloc; + u32 next_to_use; + u32 next_to_clean; + u32 next_to_alloc; =20 - struct sk_buff *skb; + struct libeth_xdp_buff_stash xdp; u64 cached_phc_time; =20 struct u64_stats_sync stats_sync; @@ -563,8 +563,8 @@ struct idpf_rx_queue { libeth_cacheline_set_assert(struct idpf_rx_queue, ALIGN(64, __alignof(struct xdp_rxq_info)) + sizeof(struct xdp_rxq_info), - 72 + offsetof(struct idpf_rx_queue, q_stats) - - offsetofend(struct idpf_rx_queue, skb), + 96 + offsetof(struct idpf_rx_queue, q_stats) - + offsetofend(struct idpf_rx_queue, cached_phc_time), 32); =20 /** @@ -1047,9 +1047,6 @@ int idpf_config_rss(struct idpf_vport *vport); int idpf_init_rss(struct idpf_vport *vport); void idpf_deinit_rss(struct idpf_vport *vport); int idpf_rx_bufs_init_all(struct idpf_vport *vport); -void idpf_rx_add_frag(struct idpf_rx_buf *rx_buf, struct sk_buff *skb, - unsigned int size); -struct sk_buff *idpf_rx_build_skb(const struct libeth_fqe *buf, u32 size); void idpf_tx_buf_hw_update(struct idpf_tx_queue *tx_q, u32 val, bool xmit_more); unsigned int idpf_size_to_txd_count(unsigned int size); diff --git a/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c b/drivers/= net/ethernet/intel/idpf/idpf_singleq_txrx.c index 178c2f3825e3..61e613066140 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_singleq_txrx.c @@ -1,8 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright (C) 2023 Intel Corporation */ =20 -#include -#include +#include =20 #include "idpf.h" =20 @@ -834,7 +833,7 @@ static void idpf_rx_singleq_flex_hash(struct idpf_rx_qu= eue *rx_q, } =20 /** - * idpf_rx_singleq_process_skb_fields - Populate skb header fields from Rx + * __idpf_rx_singleq_process_skb_fields - Populate skb header fields from = Rx * descriptor * @rx_q: Rx ring being processed * @skb: pointer to current skb being populated @@ -846,17 +845,14 @@ static void idpf_rx_singleq_flex_hash(struct idpf_rx_= queue *rx_q, * other fields within the skb. */ static void -idpf_rx_singleq_process_skb_fields(struct idpf_rx_queue *rx_q, - struct sk_buff *skb, - const union virtchnl2_rx_desc *rx_desc, - u16 ptype) +__idpf_rx_singleq_process_skb_fields(struct idpf_rx_queue *rx_q, + struct sk_buff *skb, + const union virtchnl2_rx_desc *rx_desc, + u16 ptype) { struct libeth_rx_pt decoded =3D rx_q->rx_ptype_lkup[ptype]; struct libeth_rx_csum csum_bits; =20 - /* modifies the skb - consumes the enet header */ - skb->protocol =3D eth_type_trans(skb, rx_q->xdp_rxq.dev); - /* Check if we're using base mode descriptor IDs */ if (rx_q->rxdids =3D=3D VIRTCHNL2_RXDID_1_32B_BASE_M) { idpf_rx_singleq_base_hash(rx_q, skb, rx_desc, decoded); @@ -867,7 +863,6 @@ idpf_rx_singleq_process_skb_fields(struct idpf_rx_queue= *rx_q, } =20 idpf_rx_singleq_csum(rx_q, skb, csum_bits, decoded); - skb_record_rx_queue(skb, rx_q->idx); } =20 /** @@ -1003,6 +998,32 @@ idpf_rx_singleq_extract_fields(const struct idpf_rx_q= ueue *rx_q, idpf_rx_singleq_extract_flex_fields(rx_desc, fields); } =20 +static bool +idpf_rx_singleq_process_skb_fields(struct sk_buff *skb, + const struct libeth_xdp_buff *xdp, + struct libeth_rq_napi_stats *rs) +{ + struct libeth_rqe_info fields; + struct idpf_rx_queue *rxq; + + rxq =3D libeth_xdp_buff_to_rq(xdp, typeof(*rxq), xdp_rxq); + + idpf_rx_singleq_extract_fields(rxq, xdp->desc, &fields); + __idpf_rx_singleq_process_skb_fields(rxq, skb, xdp->desc, + fields.ptype); + + return true; +} + +static void idpf_xdp_run_pass(struct libeth_xdp_buff *xdp, + struct napi_struct *napi, + struct libeth_rq_napi_stats *rs, + const union virtchnl2_rx_desc *desc) +{ + libeth_xdp_run_pass(xdp, NULL, napi, rs, desc, NULL, + idpf_rx_singleq_process_skb_fields); +} + /** * idpf_rx_singleq_clean - Reclaim resources after receive completes * @rx_q: rx queue to clean @@ -1012,14 +1033,15 @@ idpf_rx_singleq_extract_fields(const struct idpf_rx= _queue *rx_q, */ static int idpf_rx_singleq_clean(struct idpf_rx_queue *rx_q, int budget) { - unsigned int total_rx_bytes =3D 0, total_rx_pkts =3D 0; - struct sk_buff *skb =3D rx_q->skb; + struct libeth_rq_napi_stats rs =3D { }; u16 ntc =3D rx_q->next_to_clean; + LIBETH_XDP_ONSTACK_BUFF(xdp); u16 cleaned_count =3D 0; - bool failure =3D false; + + libeth_xdp_init_buff(xdp, &rx_q->xdp, &rx_q->xdp_rxq); =20 /* Process Rx packets bounded by budget */ - while (likely(total_rx_pkts < (unsigned int)budget)) { + while (likely(rs.packets < budget)) { struct libeth_rqe_info fields =3D { }; union virtchnl2_rx_desc *rx_desc; struct idpf_rx_buf *rx_buf; @@ -1046,73 +1068,41 @@ static int idpf_rx_singleq_clean(struct idpf_rx_que= ue *rx_q, int budget) idpf_rx_singleq_extract_fields(rx_q, rx_desc, &fields); =20 rx_buf =3D &rx_q->rx_buf[ntc]; - if (!libeth_rx_sync_for_cpu(rx_buf, fields.len)) - goto skip_data; - - if (skb) - idpf_rx_add_frag(rx_buf, skb, fields.len); - else - skb =3D idpf_rx_build_skb(rx_buf, fields.len); - - /* exit if we failed to retrieve a buffer */ - if (!skb) - break; - -skip_data: + libeth_xdp_process_buff(xdp, rx_buf, fields.len); rx_buf->netmem =3D 0; =20 IDPF_SINGLEQ_BUMP_RING_IDX(rx_q, ntc); cleaned_count++; =20 /* skip if it is non EOP desc */ - if (idpf_rx_singleq_is_non_eop(rx_desc) || unlikely(!skb)) + if (idpf_rx_singleq_is_non_eop(rx_desc) || + unlikely(!xdp->data)) continue; =20 #define IDPF_RXD_ERR_S FIELD_PREP(VIRTCHNL2_RX_BASE_DESC_QW1_ERROR_M, \ VIRTCHNL2_RX_BASE_DESC_ERROR_RXE_M) if (unlikely(idpf_rx_singleq_test_staterr(rx_desc, IDPF_RXD_ERR_S))) { - dev_kfree_skb_any(skb); - skb =3D NULL; - continue; - } - - /* pad skb if needed (to make valid ethernet frame) */ - if (eth_skb_pad(skb)) { - skb =3D NULL; + libeth_xdp_return_buff_slow(xdp); continue; } =20 - /* probably a little skewed due to removing CRC */ - total_rx_bytes +=3D skb->len; - - /* protocol */ - idpf_rx_singleq_process_skb_fields(rx_q, skb, rx_desc, - fields.ptype); - - /* send completed skb up the stack */ - napi_gro_receive(rx_q->pp->p.napi, skb); - skb =3D NULL; - - /* update budget accounting */ - total_rx_pkts++; + idpf_xdp_run_pass(xdp, rx_q->pp->p.napi, &rs, rx_desc); } =20 - rx_q->skb =3D skb; - rx_q->next_to_clean =3D ntc; + libeth_xdp_save_buff(&rx_q->xdp, xdp); =20 page_pool_nid_changed(rx_q->pp, numa_mem_id()); if (cleaned_count) - failure =3D idpf_rx_singleq_buf_hw_alloc_all(rx_q, cleaned_count); + idpf_rx_singleq_buf_hw_alloc_all(rx_q, cleaned_count); =20 u64_stats_update_begin(&rx_q->stats_sync); - u64_stats_add(&rx_q->q_stats.packets, total_rx_pkts); - u64_stats_add(&rx_q->q_stats.bytes, total_rx_bytes); + u64_stats_add(&rx_q->q_stats.packets, rs.packets); + u64_stats_add(&rx_q->q_stats.bytes, rs.bytes); u64_stats_update_end(&rx_q->stats_sync); =20 - /* guarantee a trip back through this routine if there was a failure */ - return failure ? budget : (int)total_rx_pkts; + return rs.packets; } =20 /** diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index 00bf949d60b5..6f484c7672f3 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -421,10 +421,7 @@ static void idpf_rx_desc_rel(struct idpf_rx_queue *rxq= , struct device *dev, if (!rxq) return; =20 - if (rxq->skb) { - dev_kfree_skb_any(rxq->skb); - rxq->skb =3D NULL; - } + libeth_xdp_return_stash(&rxq->xdp); =20 if (!idpf_is_queue_model_split(model)) idpf_rx_buf_rel_all(rxq); @@ -2997,7 +2994,7 @@ idpf_rx_hwtstamp(const struct idpf_rx_queue *rxq, } =20 /** - * idpf_rx_process_skb_fields - Populate skb header fields from Rx descrip= tor + * __idpf_rx_process_skb_fields - Populate skb header fields from Rx descr= iptor * @rxq: Rx descriptor ring packet is being transacted on * @skb: pointer to current skb being populated * @rx_desc: Receive descriptor @@ -3007,8 +3004,8 @@ idpf_rx_hwtstamp(const struct idpf_rx_queue *rxq, * other fields within the skb. */ static int -idpf_rx_process_skb_fields(struct idpf_rx_queue *rxq, struct sk_buff *skb, - const struct virtchnl2_rx_flex_desc_adv_nic_3 *rx_desc) +__idpf_rx_process_skb_fields(struct idpf_rx_queue *rxq, struct sk_buff *sk= b, + const struct virtchnl2_rx_flex_desc_adv_nic_3 *rx_desc) { struct libeth_rx_csum csum_bits; struct libeth_rx_pt decoded; @@ -3024,9 +3021,6 @@ idpf_rx_process_skb_fields(struct idpf_rx_queue *rxq,= struct sk_buff *skb, if (idpf_queue_has(PTP, rxq)) idpf_rx_hwtstamp(rxq, rx_desc, skb); =20 - skb->protocol =3D eth_type_trans(skb, rxq->xdp_rxq.dev); - skb_record_rx_queue(skb, rxq->idx); - if (le16_get_bits(rx_desc->hdrlen_flags, VIRTCHNL2_RX_FLEX_DESC_ADV_RSC_M)) return idpf_rx_rsc(rxq, skb, rx_desc, decoded); @@ -3037,23 +3031,24 @@ idpf_rx_process_skb_fields(struct idpf_rx_queue *rx= q, struct sk_buff *skb, return 0; } =20 -/** - * idpf_rx_add_frag - Add contents of Rx buffer to sk_buff as a frag - * @rx_buf: buffer containing page to add - * @skb: sk_buff to place the data into - * @size: packet length from rx_desc - * - * This function will add the data contained in rx_buf->page to the skb. - * It will just attach the page as a frag to the skb. - * The function will then update the page offset. - */ -void idpf_rx_add_frag(struct idpf_rx_buf *rx_buf, struct sk_buff *skb, - unsigned int size) +static bool idpf_rx_process_skb_fields(struct sk_buff *skb, + const struct libeth_xdp_buff *xdp, + struct libeth_rq_napi_stats *rs) { - u32 hr =3D netmem_get_pp(rx_buf->netmem)->p.offset; + struct idpf_rx_queue *rxq; + + rxq =3D libeth_xdp_buff_to_rq(xdp, typeof(*rxq), xdp_rxq); =20 - skb_add_rx_frag_netmem(skb, skb_shinfo(skb)->nr_frags, rx_buf->netmem, - rx_buf->offset + hr, size, rx_buf->truesize); + return !__idpf_rx_process_skb_fields(rxq, skb, xdp->desc); +} + +static void +idpf_xdp_run_pass(struct libeth_xdp_buff *xdp, struct napi_struct *napi, + struct libeth_rq_napi_stats *ss, + const struct virtchnl2_rx_flex_desc_adv_nic_3 *desc) +{ + libeth_xdp_run_pass(xdp, NULL, napi, ss, desc, NULL, + idpf_rx_process_skb_fields); } =20 /** @@ -3097,36 +3092,6 @@ static u32 idpf_rx_hsplit_wa(const struct libeth_fqe= *hdr, return copy; } =20 -/** - * idpf_rx_build_skb - Allocate skb and populate it from header buffer - * @buf: Rx buffer to pull data from - * @size: the length of the packet - * - * This function allocates an skb. It then populates it with the page data= from - * the current receive descriptor, taking care to set up the skb correctly. - */ -struct sk_buff *idpf_rx_build_skb(const struct libeth_fqe *buf, u32 size) -{ - struct page *buf_page =3D __netmem_to_page(buf->netmem); - u32 hr =3D pp_page_to_nmdesc(buf_page)->pp->p.offset; - struct sk_buff *skb; - void *va; - - va =3D page_address(buf_page) + buf->offset; - prefetch(va + hr); - - skb =3D napi_build_skb(va, buf->truesize); - if (unlikely(!skb)) - return NULL; - - skb_mark_for_recycle(skb); - - skb_reserve(skb, hr); - __skb_put(skb, size); - - return skb; -} - /** * idpf_rx_splitq_test_staterr - tests bits in Rx descriptor * status and error fields @@ -3168,13 +3133,15 @@ static bool idpf_rx_splitq_is_eop(struct virtchnl2_= rx_flex_desc_adv_nic_3 *rx_de */ static int idpf_rx_splitq_clean(struct idpf_rx_queue *rxq, int budget) { - int total_rx_bytes =3D 0, total_rx_pkts =3D 0; struct idpf_buf_queue *rx_bufq =3D NULL; - struct sk_buff *skb =3D rxq->skb; + struct libeth_rq_napi_stats rs =3D { }; u16 ntc =3D rxq->next_to_clean; + LIBETH_XDP_ONSTACK_BUFF(xdp); + + libeth_xdp_init_buff(xdp, &rxq->xdp, &rxq->xdp_rxq); =20 /* Process Rx packets bounded by budget */ - while (likely(total_rx_pkts < budget)) { + while (likely(rs.packets < budget)) { struct virtchnl2_rx_flex_desc_adv_nic_3 *rx_desc; struct libeth_fqe *hdr, *rx_buf =3D NULL; struct idpf_sw_queue *refillq =3D NULL; @@ -3240,7 +3207,7 @@ static int idpf_rx_splitq_clean(struct idpf_rx_queue = *rxq, int budget) =20 hdr =3D &rx_bufq->hdr_buf[buf_id]; =20 - if (unlikely(!hdr_len && !skb)) { + if (unlikely(!hdr_len && !xdp->data)) { hdr_len =3D idpf_rx_hsplit_wa(hdr, rx_buf, pkt_len); /* If failed, drop both buffers by setting len to 0 */ pkt_len -=3D hdr_len ? : pkt_len; @@ -3250,75 +3217,35 @@ static int idpf_rx_splitq_clean(struct idpf_rx_queu= e *rxq, int budget) u64_stats_update_end(&rxq->stats_sync); } =20 - if (libeth_rx_sync_for_cpu(hdr, hdr_len)) { - skb =3D idpf_rx_build_skb(hdr, hdr_len); - if (!skb) - break; - - u64_stats_update_begin(&rxq->stats_sync); - u64_stats_inc(&rxq->q_stats.hsplit_pkts); - u64_stats_update_end(&rxq->stats_sync); - } + if (libeth_xdp_process_buff(xdp, hdr, hdr_len)) + rs.hsplit++; =20 hdr->netmem =3D 0; =20 payload: - if (!libeth_rx_sync_for_cpu(rx_buf, pkt_len)) - goto skip_data; - - if (skb) - idpf_rx_add_frag(rx_buf, skb, pkt_len); - else - skb =3D idpf_rx_build_skb(rx_buf, pkt_len); - - /* exit if we failed to retrieve a buffer */ - if (!skb) - break; - -skip_data: + libeth_xdp_process_buff(xdp, rx_buf, pkt_len); rx_buf->netmem =3D 0; =20 idpf_post_buf_refill(refillq, buf_id); IDPF_RX_BUMP_NTC(rxq, ntc); =20 /* skip if it is non EOP desc */ - if (!idpf_rx_splitq_is_eop(rx_desc) || unlikely(!skb)) - continue; - - /* pad skb if needed (to make valid ethernet frame) */ - if (eth_skb_pad(skb)) { - skb =3D NULL; - continue; - } - - /* probably a little skewed due to removing CRC */ - total_rx_bytes +=3D skb->len; - - /* protocol */ - if (unlikely(idpf_rx_process_skb_fields(rxq, skb, rx_desc))) { - dev_kfree_skb_any(skb); - skb =3D NULL; + if (!idpf_rx_splitq_is_eop(rx_desc) || unlikely(!xdp->data)) continue; - } =20 - /* send completed skb up the stack */ - napi_gro_receive(rxq->napi, skb); - skb =3D NULL; - - /* update budget accounting */ - total_rx_pkts++; + idpf_xdp_run_pass(xdp, rxq->napi, &rs, rx_desc); } =20 rxq->next_to_clean =3D ntc; + libeth_xdp_save_buff(&rxq->xdp, xdp); =20 - rxq->skb =3D skb; u64_stats_update_begin(&rxq->stats_sync); - u64_stats_add(&rxq->q_stats.packets, total_rx_pkts); - u64_stats_add(&rxq->q_stats.bytes, total_rx_bytes); + u64_stats_add(&rxq->q_stats.packets, rs.packets); + u64_stats_add(&rxq->q_stats.bytes, rs.bytes); + u64_stats_add(&rxq->q_stats.hsplit_pkts, rs.hsplit); u64_stats_update_end(&rxq->stats_sync); =20 - /* guarantee a trip back through this routine if there was a failure */ - return total_rx_pkts; + return rs.packets; } =20 /** --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18759246333; Tue, 26 Aug 2025 16:14:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224859; cv=none; b=H1B9lP7qCVu/AQTXjqDnp3TDGoxU7SEkk2qnRZXlVA54aE5zGUdmtC7Terlti4T/FPbwDuDmzR2N28siUeJyT9Oi9jsfFfGZ2gUEZQ4xp8/wZvz+DOSiHIaAuxmzEcscGx9FgeC2AsLoTSX/+Ex3rGqQ3FO7KaHsp+dOps4XQV0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224859; c=relaxed/simple; bh=LRg9aKOqgRLlC/e6mfCI2QplW2EeuXYCYnp43rdOpKY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qreg/WPCS3Ffuyep2dURmq3sixG+KPn9NZgNEprBfoXT7u/h9P6n+jtHtyrvGihSTbCJm1qoG6jb/8j4aSARl1qTfOY71+3rxjge2krbCg69AuJbG+KvndOjotYzDkFhlkPIJgL/Qntt3FX9/ihpBmRPc4HAYvkyEeZVgMD91eA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ThVT+d9j; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ThVT+d9j" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224857; x=1787760857; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LRg9aKOqgRLlC/e6mfCI2QplW2EeuXYCYnp43rdOpKY=; b=ThVT+d9jwmpW+UGDYQ2TCb2YPbL8QU84EdvAjZR23I5YQAIXur3IUQWF lhBoURKR1qCh9+R0GY6I+EGF9wR/lTqFc4HgZyy8dW9rEDeiUPySAQu2t gCW6msDPDh+cdZEQsgmgGxq1qecFQMNi3RblqrZ6qdMVyO9GOjjIzQTAc +WmcE5p+cpc6GOfLcbRvysE9palyjXhcEZZjbv4bySgx3HTTTbJfGK9MX svtJWX7dupgUwp+1IUpjGcBkNVwQQv6j0Q3atSdsdoT98kEXGV0hTs0XP 5acxdVbdbKdPbszrQhpRsvUOFUVdkiZI5v1iqOIdosKTsnTZxRWqhq5BO A==; X-CSE-ConnectionGUID: eV/8Ek+cQSejIyton8a3fQ== X-CSE-MsgGUID: 7RY69edlTDKq01ECGAkHiw== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46045083" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46045083" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:14:17 -0700 X-CSE-ConnectionGUID: B277b0SXQJ2K3ReX46Ex7A== X-CSE-MsgGUID: kxlNfLbbQyC1Zxg6My8gng== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562608" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:14:12 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 11/13] idpf: add support for XDP on Rx Date: Tue, 26 Aug 2025 17:55:05 +0200 Message-ID: <20250826155507.2138401-12-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use libeth XDP infra to support running XDP program on Rx polling. This includes all of the possible verdicts/actions. XDP Tx queues are cleaned only in "lazy" mode when there are less than 1/4 free descriptors left on the ring. libeth helper macros to define driver-specific XDP functions make sure the compiler could uninline them when needed. Use __LIBETH_WORD_ACCESS to parse descriptors more efficiently when applicable. It really gives some good boosts and code size reduction on x86_64: XDP only: add/remove: 0/0 grow/shrink: 3/3 up/down: 5/-59 (-54) with XSk: add/remove: 0/0 grow/shrink: 5/6 up/down: 23/-124 (-101) with the most demanding workloads like XSk xmit differing in up to 5-8%. Co-developed-by: Michal Kubiak Signed-off-by: Michal Kubiak Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/idpf_txrx.h | 4 +- drivers/net/ethernet/intel/idpf/xdp.h | 92 +++++++++++- drivers/net/ethernet/intel/idpf/idpf_lib.c | 2 + drivers/net/ethernet/intel/idpf/idpf_txrx.c | 23 +-- drivers/net/ethernet/intel/idpf/xdp.c | 147 +++++++++++++++++++- 5 files changed, 248 insertions(+), 20 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.h index 5039feafdee9..39a9c6bd6055 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -646,8 +646,8 @@ struct idpf_tx_queue { __cacheline_group_end_aligned(read_mostly); =20 __cacheline_group_begin_aligned(read_write); - u16 next_to_use; - u16 next_to_clean; + u32 next_to_use; + u32 next_to_clean; =20 union { struct { diff --git a/drivers/net/ethernet/intel/idpf/xdp.h b/drivers/net/ethernet/i= ntel/idpf/xdp.h index 47553ce5f81a..986156162e2d 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.h +++ b/drivers/net/ethernet/intel/idpf/xdp.h @@ -4,12 +4,9 @@ #ifndef _IDPF_XDP_H_ #define _IDPF_XDP_H_ =20 -#include +#include =20 -struct bpf_prog; -struct idpf_vport; -struct net_device; -struct netdev_bpf; +#include "idpf_txrx.h" =20 int idpf_xdp_rxq_info_init_all(const struct idpf_vport *vport); void idpf_xdp_rxq_info_deinit_all(const struct idpf_vport *vport); @@ -19,6 +16,91 @@ void idpf_xdp_copy_prog_to_rqs(const struct idpf_vport *= vport, int idpf_xdpsqs_get(const struct idpf_vport *vport); void idpf_xdpsqs_put(const struct idpf_vport *vport); =20 +bool idpf_xdp_tx_flush_bulk(struct libeth_xdp_tx_bulk *bq, u32 flags); + +/** + * idpf_xdp_tx_xmit - produce a single HW Tx descriptor out of XDP desc + * @desc: XDP descriptor to pull the DMA address and length from + * @i: descriptor index on the queue to fill + * @sq: XDP queue to produce the HW Tx descriptor on + * @priv: &xsk_tx_metadata_ops on XSk xmit or %NULL + */ +static inline void idpf_xdp_tx_xmit(struct libeth_xdp_tx_desc desc, u32 i, + const struct libeth_xdpsq *sq, u64 priv) +{ + struct idpf_flex_tx_desc *tx_desc =3D sq->descs; + u32 cmd; + + cmd =3D FIELD_PREP(IDPF_FLEX_TXD_QW1_DTYPE_M, + IDPF_TX_DESC_DTYPE_FLEX_L2TAG1_L2TAG2); + if (desc.flags & LIBETH_XDP_TX_LAST) + cmd |=3D FIELD_PREP(IDPF_FLEX_TXD_QW1_CMD_M, + IDPF_TX_DESC_CMD_EOP); + if (priv && (desc.flags & LIBETH_XDP_TX_CSUM)) + cmd |=3D FIELD_PREP(IDPF_FLEX_TXD_QW1_CMD_M, + IDPF_TX_FLEX_DESC_CMD_CS_EN); + + tx_desc =3D &tx_desc[i]; + tx_desc->buf_addr =3D cpu_to_le64(desc.addr); +#ifdef __LIBETH_WORD_ACCESS + *(u64 *)&tx_desc->qw1 =3D ((u64)desc.len << 48) | cmd; +#else + tx_desc->qw1.buf_size =3D cpu_to_le16(desc.len); + tx_desc->qw1.cmd_dtype =3D cpu_to_le16(cmd); +#endif +} + +static inline void idpf_xdpsq_set_rs(const struct idpf_tx_queue *xdpsq) +{ + u32 ntu, cmd; + + ntu =3D xdpsq->next_to_use; + if (unlikely(!ntu)) + ntu =3D xdpsq->desc_count; + + cmd =3D FIELD_PREP(IDPF_FLEX_TXD_QW1_CMD_M, IDPF_TX_DESC_CMD_RS); +#ifdef __LIBETH_WORD_ACCESS + *(u64 *)&xdpsq->flex_tx[ntu - 1].q.qw1 |=3D cmd; +#else + xdpsq->flex_tx[ntu - 1].q.qw1.cmd_dtype |=3D cpu_to_le16(cmd); +#endif +} + +static inline void idpf_xdpsq_update_tail(const struct idpf_tx_queue *xdps= q) +{ + dma_wmb(); + writel_relaxed(xdpsq->next_to_use, xdpsq->tail); +} + +/** + * idpf_xdp_tx_finalize - finalize sending over XDPSQ + * @_xdpsq: XDP Tx queue + * @sent: whether any frames were sent + * @flush: whether to update RS bit and the tail register + * + * Set the RS bit ("end of batch"), bump the tail, and queue the cleanup t= imer. + * To be called after a NAPI polling loop, at the end of .ndo_xdp_xmit() e= tc. + */ +static inline void idpf_xdp_tx_finalize(void *_xdpsq, bool sent, bool flus= h) +{ + struct idpf_tx_queue *xdpsq =3D _xdpsq; + + if ((!flush || unlikely(!sent)) && + likely(xdpsq->desc_count - 1 !=3D xdpsq->pending)) + return; + + libeth_xdpsq_lock(&xdpsq->xdp_lock); + + idpf_xdpsq_set_rs(xdpsq); + idpf_xdpsq_update_tail(xdpsq); + + libeth_xdpsq_queue_timer(xdpsq->timer); + + libeth_xdpsq_unlock(&xdpsq->xdp_lock); +} + +void idpf_xdp_set_features(const struct idpf_vport *vport); + int idpf_xdp(struct net_device *dev, struct netdev_bpf *xdp); =20 #endif /* _IDPF_XDP_H_ */ diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ether= net/intel/idpf/idpf_lib.c index d645dcc2deda..1c62998cc0e7 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -836,6 +836,8 @@ static int idpf_cfg_netdev(struct idpf_vport *vport) netdev->hw_features |=3D netdev->features | other_offloads; netdev->vlan_features |=3D netdev->features | other_offloads; netdev->hw_enc_features |=3D dflt_features | other_offloads; + idpf_xdp_set_features(vport); + idpf_set_ethtool_ops(netdev); netif_set_affinity_auto(netdev); SET_NETDEV_DEV(netdev, &adapter->pdev->dev); diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index 6f484c7672f3..2d042dd91630 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -1,8 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright (C) 2023 Intel Corporation */ =20 -#include - #include "idpf.h" #include "idpf_ptp.h" #include "idpf_virtchnl.h" @@ -3042,14 +3040,12 @@ static bool idpf_rx_process_skb_fields(struct sk_bu= ff *skb, return !__idpf_rx_process_skb_fields(rxq, skb, xdp->desc); } =20 -static void -idpf_xdp_run_pass(struct libeth_xdp_buff *xdp, struct napi_struct *napi, - struct libeth_rq_napi_stats *ss, - const struct virtchnl2_rx_flex_desc_adv_nic_3 *desc) -{ - libeth_xdp_run_pass(xdp, NULL, napi, ss, desc, NULL, - idpf_rx_process_skb_fields); -} +LIBETH_XDP_DEFINE_START(); +LIBETH_XDP_DEFINE_RUN(static idpf_xdp_run_pass, idpf_xdp_run_prog, + idpf_xdp_tx_flush_bulk, idpf_rx_process_skb_fields); +LIBETH_XDP_DEFINE_FINALIZE(static idpf_xdp_finalize_rx, idpf_xdp_tx_flush_= bulk, + idpf_xdp_tx_finalize); +LIBETH_XDP_DEFINE_END(); =20 /** * idpf_rx_hsplit_wa - handle header buffer overflows and split errors @@ -3137,7 +3133,10 @@ static int idpf_rx_splitq_clean(struct idpf_rx_queue= *rxq, int budget) struct libeth_rq_napi_stats rs =3D { }; u16 ntc =3D rxq->next_to_clean; LIBETH_XDP_ONSTACK_BUFF(xdp); + LIBETH_XDP_ONSTACK_BULK(bq); =20 + libeth_xdp_tx_init_bulk(&bq, rxq->xdp_prog, rxq->xdp_rxq.dev, + rxq->xdpsqs, rxq->num_xdp_txq); libeth_xdp_init_buff(xdp, &rxq->xdp, &rxq->xdp_rxq); =20 /* Process Rx packets bounded by budget */ @@ -3233,9 +3232,11 @@ static int idpf_rx_splitq_clean(struct idpf_rx_queue= *rxq, int budget) if (!idpf_rx_splitq_is_eop(rx_desc) || unlikely(!xdp->data)) continue; =20 - idpf_xdp_run_pass(xdp, rxq->napi, &rs, rx_desc); + idpf_xdp_run_pass(xdp, &bq, rxq->napi, &rs, rx_desc); } =20 + idpf_xdp_finalize_rx(&bq); + rxq->next_to_clean =3D ntc; libeth_xdp_save_buff(&rxq->xdp, xdp); =20 diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/i= ntel/idpf/xdp.c index 02f63810632f..e6b45df95cd3 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.c +++ b/drivers/net/ethernet/intel/idpf/xdp.c @@ -1,8 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright (C) 2025 Intel Corporation */ =20 -#include - #include "idpf.h" #include "idpf_virtchnl.h" #include "xdp.h" @@ -113,6 +111,8 @@ void idpf_xdp_copy_prog_to_rqs(const struct idpf_vport = *vport, idpf_rxq_for_each(vport, idpf_xdp_rxq_assign_prog, xdp_prog); } =20 +static void idpf_xdp_tx_timer(struct work_struct *work); + int idpf_xdpsqs_get(const struct idpf_vport *vport) { struct libeth_xdpsq_timer **timers __free(kvfree) =3D NULL; @@ -155,6 +155,8 @@ int idpf_xdpsqs_get(const struct idpf_vport *vport) =20 xdpsq->timer =3D timers[i - sqs]; libeth_xdpsq_get(&xdpsq->xdp_lock, dev, vport->xdpsq_share); + libeth_xdpsq_init_timer(xdpsq->timer, xdpsq, &xdpsq->xdp_lock, + idpf_xdp_tx_timer); =20 xdpsq->pending =3D 0; xdpsq->xdp_tx =3D 0; @@ -181,6 +183,7 @@ void idpf_xdpsqs_put(const struct idpf_vport *vport) if (!idpf_queue_has_clear(XDP, xdpsq)) continue; =20 + libeth_xdpsq_deinit_timer(xdpsq->timer); libeth_xdpsq_put(&xdpsq->xdp_lock, dev); =20 kfree(xdpsq->timer); @@ -189,6 +192,146 @@ void idpf_xdpsqs_put(const struct idpf_vport *vport) } } =20 +static int idpf_xdp_parse_cqe(const struct idpf_splitq_4b_tx_compl_desc *d= esc, + bool gen) +{ + u32 val; + +#ifdef __LIBETH_WORD_ACCESS + val =3D *(const u32 *)desc; +#else + val =3D ((u32)le16_to_cpu(desc->q_head_compl_tag.q_head) << 16) | + le16_to_cpu(desc->qid_comptype_gen); +#endif + if (!!(val & IDPF_TXD_COMPLQ_GEN_M) !=3D gen) + return -ENODATA; + + if (unlikely((val & GENMASK(IDPF_TXD_COMPLQ_GEN_S - 1, 0)) !=3D + FIELD_PREP(IDPF_TXD_COMPLQ_COMPL_TYPE_M, + IDPF_TXD_COMPLT_RS))) + return -EINVAL; + + return upper_16_bits(val); +} + +static u32 idpf_xdpsq_poll(struct idpf_tx_queue *xdpsq, u32 budget) +{ + struct idpf_compl_queue *cq =3D xdpsq->complq; + u32 tx_ntc =3D xdpsq->next_to_clean; + u32 tx_cnt =3D xdpsq->desc_count; + u32 ntc =3D cq->next_to_clean; + u32 cnt =3D cq->desc_count; + u32 done_frames; + bool gen; + + gen =3D idpf_queue_has(GEN_CHK, cq); + + for (done_frames =3D 0; done_frames < budget; ) { + int ret; + + ret =3D idpf_xdp_parse_cqe(&cq->comp_4b[ntc], gen); + if (ret >=3D 0) { + done_frames =3D ret > tx_ntc ? ret - tx_ntc : + ret + tx_cnt - tx_ntc; + goto next; + } + + switch (ret) { + case -ENODATA: + goto out; + case -EINVAL: + break; + } + +next: + if (unlikely(++ntc =3D=3D cnt)) { + ntc =3D 0; + gen =3D !gen; + idpf_queue_change(GEN_CHK, cq); + } + } + +out: + cq->next_to_clean =3D ntc; + + return done_frames; +} + +static u32 idpf_xdpsq_complete(void *_xdpsq, u32 budget) +{ + struct libeth_xdpsq_napi_stats ss =3D { }; + struct idpf_tx_queue *xdpsq =3D _xdpsq; + u32 tx_ntc =3D xdpsq->next_to_clean; + u32 tx_cnt =3D xdpsq->desc_count; + struct xdp_frame_bulk bq; + struct libeth_cq_pp cp =3D { + .dev =3D xdpsq->dev, + .bq =3D &bq, + .xss =3D &ss, + .napi =3D true, + }; + u32 done_frames; + + done_frames =3D idpf_xdpsq_poll(xdpsq, budget); + if (unlikely(!done_frames)) + return 0; + + xdp_frame_bulk_init(&bq); + + for (u32 i =3D 0; likely(i < done_frames); i++) { + libeth_xdp_complete_tx(&xdpsq->tx_buf[tx_ntc], &cp); + + if (unlikely(++tx_ntc =3D=3D tx_cnt)) + tx_ntc =3D 0; + } + + xdp_flush_frame_bulk(&bq); + + xdpsq->next_to_clean =3D tx_ntc; + xdpsq->pending -=3D done_frames; + xdpsq->xdp_tx -=3D cp.xdp_tx; + + return done_frames; +} + +static u32 idpf_xdp_tx_prep(void *_xdpsq, struct libeth_xdpsq *sq) +{ + struct idpf_tx_queue *xdpsq =3D _xdpsq; + u32 free; + + libeth_xdpsq_lock(&xdpsq->xdp_lock); + + free =3D xdpsq->desc_count - xdpsq->pending; + if (free < xdpsq->thresh) + free +=3D idpf_xdpsq_complete(xdpsq, xdpsq->thresh); + + *sq =3D (struct libeth_xdpsq){ + .sqes =3D xdpsq->tx_buf, + .descs =3D xdpsq->desc_ring, + .count =3D xdpsq->desc_count, + .lock =3D &xdpsq->xdp_lock, + .ntu =3D &xdpsq->next_to_use, + .pending =3D &xdpsq->pending, + .xdp_tx =3D &xdpsq->xdp_tx, + }; + + return free; +} + +LIBETH_XDP_DEFINE_START(); +LIBETH_XDP_DEFINE_TIMER(static idpf_xdp_tx_timer, idpf_xdpsq_complete); +LIBETH_XDP_DEFINE_FLUSH_TX(idpf_xdp_tx_flush_bulk, idpf_xdp_tx_prep, + idpf_xdp_tx_xmit); +LIBETH_XDP_DEFINE_END(); + +void idpf_xdp_set_features(const struct idpf_vport *vport) +{ + if (!idpf_is_queue_model_split(vport->rxq_model)) + return; + + libeth_xdp_set_features_noredir(vport->netdev); +} + static int idpf_xdp_setup_prog(struct idpf_vport *vport, const struct netdev_bpf *xdp) { --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7310F3728AD; Tue, 26 Aug 2025 16:14:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224863; cv=none; b=mPEoeGaJj7By41aPhCbJpmPFAAMUdrpkQwjPorBQPx/gvSt/+dWzAS8UfM6A+h57NGUtb6a4RWIrPPQSfHyEIRh+Dl17n8v6O1sirgItdM+LPso/OGkB+1b5H9r5w7tEkri6xquv59aTbYkGRKs7qbkNh+n4dfCxjeY1DW01Zqs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224863; c=relaxed/simple; bh=SxnuiHVydamTUKCJfnvkSyB7La/z3S5jqAnbJC+dZf8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UM6CTZlCt0KumINMYN1IYCQ6+Lg1ZpIFCIx9Zo6A2qtWTyudwp7yOH2V0WhR5OcnNKy+Pjv9OwA7xsDh2SU4Fee8qAUJrNZUsd5fPIhSZ1p2PI7nYtpAowy0IXf16NK94OfLaFNMDUb6X+WDuN+G+u3uUUagoJAFUNnZJcGKmtQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=PNoErz6E; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="PNoErz6E" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224861; x=1787760861; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SxnuiHVydamTUKCJfnvkSyB7La/z3S5jqAnbJC+dZf8=; b=PNoErz6EuOAg94/dUnnPOzqzBmrcJVbZumH1/ItNI9zegJddmqNWc72b 2WCQnOTYdUAuppC+aUz9ajTl7FNk+RcDxaofr/9mpKky1x3NG6kwdFB+W cEKAzdKjxdKPE24FmOw5jtdVPcxR9XXv/1NhPkRyizJ2ChOc1TiO9sspB fCFzWBlk7PY0JrChsh9dP9bBFvkuM+AU80OPz6ON4WODC2kVZRzZedbJl 5Wuw1vmRTS9SHY0Vbz5wnW8nDX6IStIPFwpBBF7+X7Ii170haonlcQokT c9PcoSfTpRJ6KSfNmDmYvh60l/uUvj63fWTXS9zW64rdzbPjFG9rASXch g==; X-CSE-ConnectionGUID: bev/enV0RE2L+0qf7/dXBg== X-CSE-MsgGUID: tO1ePug/S8qBegItcwQg6Q== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46045104" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46045104" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:14:21 -0700 X-CSE-ConnectionGUID: 7j1vDRS3RT6fRUXwBhs8SQ== X-CSE-MsgGUID: t3HatVNrQpGpwf9BR0E/Qw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562651" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:14:17 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 12/13] idpf: add support for .ndo_xdp_xmit() Date: Tue, 26 Aug 2025 17:55:06 +0200 Message-ID: <20250826155507.2138401-13-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use libeth XDP infra to implement .ndo_xdp_xmit() in idpf. The Tx callbacks are reused from XDP_TX code. XDP redirect target feature is set/cleared depending on the XDP prog presence, as for now we still don't allocate XDP Tx queues when there's no program. Reviewed-by: Maciej Fijalkowski Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/xdp.h | 2 ++ drivers/net/ethernet/intel/idpf/idpf_lib.c | 1 + drivers/net/ethernet/intel/idpf/xdp.c | 20 ++++++++++++++++++++ 3 files changed, 23 insertions(+) diff --git a/drivers/net/ethernet/intel/idpf/xdp.h b/drivers/net/ethernet/i= ntel/idpf/xdp.h index 986156162e2d..db8ecc1843fe 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.h +++ b/drivers/net/ethernet/intel/idpf/xdp.h @@ -102,5 +102,7 @@ static inline void idpf_xdp_tx_finalize(void *_xdpsq, b= ool sent, bool flush) void idpf_xdp_set_features(const struct idpf_vport *vport); =20 int idpf_xdp(struct net_device *dev, struct netdev_bpf *xdp); +int idpf_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames, + u32 flags); =20 #endif /* _IDPF_XDP_H_ */ diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ether= net/intel/idpf/idpf_lib.c index 1c62998cc0e7..6bf563fa0966 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -2612,4 +2612,5 @@ static const struct net_device_ops idpf_netdev_ops = =3D { .ndo_hwtstamp_get =3D idpf_hwtstamp_get, .ndo_hwtstamp_set =3D idpf_hwtstamp_set, .ndo_bpf =3D idpf_xdp, + .ndo_xdp_xmit =3D idpf_xdp_xmit, }; diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/i= ntel/idpf/xdp.c index e6b45df95cd3..b6a8304d61f9 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.c +++ b/drivers/net/ethernet/intel/idpf/xdp.c @@ -322,8 +322,26 @@ LIBETH_XDP_DEFINE_START(); LIBETH_XDP_DEFINE_TIMER(static idpf_xdp_tx_timer, idpf_xdpsq_complete); LIBETH_XDP_DEFINE_FLUSH_TX(idpf_xdp_tx_flush_bulk, idpf_xdp_tx_prep, idpf_xdp_tx_xmit); +LIBETH_XDP_DEFINE_FLUSH_XMIT(static idpf_xdp_xmit_flush_bulk, idpf_xdp_tx_= prep, + idpf_xdp_tx_xmit); LIBETH_XDP_DEFINE_END(); =20 +int idpf_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames, + u32 flags) +{ + const struct idpf_netdev_priv *np =3D netdev_priv(dev); + const struct idpf_vport *vport =3D np->vport; + + if (unlikely(!netif_carrier_ok(dev) || !vport->link_up)) + return -ENETDOWN; + + return libeth_xdp_xmit_do_bulk(dev, n, frames, flags, + &vport->txqs[vport->xdp_txq_offset], + vport->num_xdp_txq, + idpf_xdp_xmit_flush_bulk, + idpf_xdp_tx_finalize); +} + void idpf_xdp_set_features(const struct idpf_vport *vport) { if (!idpf_is_queue_model_split(vport->rxq_model)) @@ -378,6 +396,8 @@ static int idpf_xdp_setup_prog(struct idpf_vport *vport, if (old) bpf_prog_put(old); =20 + libeth_xdp_set_redirect(vport->netdev, vport->xdp_prog); + return ret; } =20 --=20 2.51.0 From nobody Fri Oct 3 20:30:00 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1531374274; Tue, 26 Aug 2025 16:14:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224867; cv=none; b=EDkICv8ojQPua82egvghVt2p+jh1xpTfjnr5FjPqDjUXRdLj8B0BieepmRtvqqbDoANtkraUHJUfq4hfjYSZXeEki3M2yq6v800BgXBzqORBu25JUu88zVEDrEzsRNQgDgZys/HpYiaJYb2bKQsC9xCLoEtopnpBviQUXwuYrgY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756224867; c=relaxed/simple; bh=nZtTZePcJLv/YRyNX7vGsbCe1aj4pTMRnNPvHJnXDj4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aZgBPBvn1YEjyp2d5uN9N5IuGdlWhWptMs/I3EBeMEof5ycj+nGj/mf+bP1ihtwwC6kiGw7G7E99n20Je5E0w3l7B31uFm79DK4CgDQv/xvF0pkylNld/2K/3MVcvggeYvmdXpVq0+3TGBFFG9wQpsbwA5ho5jz5bYvrSSDMDxQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=LXHtXl7J; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="LXHtXl7J" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1756224866; x=1787760866; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nZtTZePcJLv/YRyNX7vGsbCe1aj4pTMRnNPvHJnXDj4=; b=LXHtXl7JoRMoIWtk2VUXET1Z3JtkZ/k0k6fVMdK+TzupJx6uPPqH8Pxb N1u98cFAR8ewQxIka1gAKkoIMYQdEEq1Tfz6JFjF+mGCJJxNS2UapZ9nN 5f4kXS436qcP4059jRGamDIWFSuKQsdINK3bOHrR6ewdmvFTlDuaSCtKD JcWM4H+2ISybwlOMWMoNzDP/PRqolQ/UhuvMUWVyMCnTmjnb/PpCu0Oa8 Q5WKJgkE4jmF2o2v2CVHflKjzz7BwnG3sik8Xd9G/ycMSJaG63y25EwIe JbIPhbzfBacvqqRyYx13apg8+fbThfuYsG7H0ubHFg/yPAoa5/7CtVx90 g==; X-CSE-ConnectionGUID: ISij/YqZQKiDyTtRbM074g== X-CSE-MsgGUID: sEAdLbDETd6OOF6Fcoci8Q== X-IronPort-AV: E=McAfee;i="6800,10657,11534"; a="46045127" X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="46045127" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2025 09:14:25 -0700 X-CSE-ConnectionGUID: /XPE5bhdRJC8jFb8LkKBjQ== X-CSE-MsgGUID: lsP1l7wTSfucNU1ErUoNew== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,214,1751266800"; d="scan'208";a="200562710" Received: from newjersey.igk.intel.com ([10.102.20.203]) by orviesa002.jf.intel.com with ESMTP; 26 Aug 2025 09:14:21 -0700 From: Alexander Lobakin To: intel-wired-lan@lists.osuosl.org Cc: Alexander Lobakin , Michal Kubiak , Maciej Fijalkowski , Tony Nguyen , Przemek Kitszel , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Simon Horman , nxne.cnse.osdt.itp.upstreaming@intel.com, bpf@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH iwl-next v5 13/13] idpf: add XDP RSS hash hint Date: Tue, 26 Aug 2025 17:55:07 +0200 Message-ID: <20250826155507.2138401-14-aleksander.lobakin@intel.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250826155507.2138401-1-aleksander.lobakin@intel.com> References: <20250826155507.2138401-1-aleksander.lobakin@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add &xdp_metadata_ops with a callback to get RSS hash hint from the descriptor. Declare the splitq 32-byte descriptor as 4 u64s to parse them more efficiently when possible. Signed-off-by: Alexander Lobakin --- drivers/net/ethernet/intel/idpf/xdp.h | 64 +++++++++++++++++++++++++++ drivers/net/ethernet/intel/idpf/xdp.c | 28 +++++++++++- 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/intel/idpf/xdp.h b/drivers/net/ethernet/i= ntel/idpf/xdp.h index db8ecc1843fe..66ad83a0e85e 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.h +++ b/drivers/net/ethernet/intel/idpf/xdp.h @@ -99,6 +99,70 @@ static inline void idpf_xdp_tx_finalize(void *_xdpsq, bo= ol sent, bool flush) libeth_xdpsq_unlock(&xdpsq->xdp_lock); } =20 +struct idpf_xdp_rx_desc { + aligned_u64 qw0; +#define IDPF_XDP_RX_BUFQ BIT_ULL(47) +#define IDPF_XDP_RX_GEN BIT_ULL(46) +#define IDPF_XDP_RX_LEN GENMASK_ULL(45, 32) +#define IDPF_XDP_RX_PT GENMASK_ULL(25, 16) + + aligned_u64 qw1; +#define IDPF_XDP_RX_BUF GENMASK_ULL(47, 32) +#define IDPF_XDP_RX_EOP BIT_ULL(1) + + aligned_u64 qw2; +#define IDPF_XDP_RX_HASH GENMASK_ULL(31, 0) + + aligned_u64 qw3; +} __aligned(4 * sizeof(u64)); +static_assert(sizeof(struct idpf_xdp_rx_desc) =3D=3D + sizeof(struct virtchnl2_rx_flex_desc_adv_nic_3)); + +#define idpf_xdp_rx_bufq(desc) !!((desc)->qw0 & IDPF_XDP_RX_BUFQ) +#define idpf_xdp_rx_gen(desc) !!((desc)->qw0 & IDPF_XDP_RX_GEN) +#define idpf_xdp_rx_len(desc) FIELD_GET(IDPF_XDP_RX_LEN, (desc)->qw0) +#define idpf_xdp_rx_pt(desc) FIELD_GET(IDPF_XDP_RX_PT, (desc)->qw0) +#define idpf_xdp_rx_buf(desc) FIELD_GET(IDPF_XDP_RX_BUF, (desc)->qw1) +#define idpf_xdp_rx_eop(desc) !!((desc)->qw1 & IDPF_XDP_RX_EOP) +#define idpf_xdp_rx_hash(desc) FIELD_GET(IDPF_XDP_RX_HASH, (desc)->qw2) + +static inline void +idpf_xdp_get_qw0(struct idpf_xdp_rx_desc *desc, + const struct virtchnl2_rx_flex_desc_adv_nic_3 *rxd) +{ +#ifdef __LIBETH_WORD_ACCESS + desc->qw0 =3D ((const typeof(desc))rxd)->qw0; +#else + desc->qw0 =3D ((u64)le16_to_cpu(rxd->pktlen_gen_bufq_id) << 32) | + ((u64)le16_to_cpu(rxd->ptype_err_fflags0) << 16); +#endif +} + +static inline void +idpf_xdp_get_qw1(struct idpf_xdp_rx_desc *desc, + const struct virtchnl2_rx_flex_desc_adv_nic_3 *rxd) +{ +#ifdef __LIBETH_WORD_ACCESS + desc->qw1 =3D ((const typeof(desc))rxd)->qw1; +#else + desc->qw1 =3D ((u64)le16_to_cpu(rxd->buf_id) << 32) | + rxd->status_err0_qw1; +#endif +} + +static inline void +idpf_xdp_get_qw2(struct idpf_xdp_rx_desc *desc, + const struct virtchnl2_rx_flex_desc_adv_nic_3 *rxd) +{ +#ifdef __LIBETH_WORD_ACCESS + desc->qw2 =3D ((const typeof(desc))rxd)->qw2; +#else + desc->qw2 =3D ((u64)rxd->hash3 << 24) | + ((u64)rxd->ff2_mirrid_hash2.hash2 << 16) | + le16_to_cpu(rxd->hash1); +#endif +} + void idpf_xdp_set_features(const struct idpf_vport *vport); =20 int idpf_xdp(struct net_device *dev, struct netdev_bpf *xdp); diff --git a/drivers/net/ethernet/intel/idpf/xdp.c b/drivers/net/ethernet/i= ntel/idpf/xdp.c index b6a8304d61f9..89d5735f42f2 100644 --- a/drivers/net/ethernet/intel/idpf/xdp.c +++ b/drivers/net/ethernet/intel/idpf/xdp.c @@ -342,12 +342,38 @@ int idpf_xdp_xmit(struct net_device *dev, int n, stru= ct xdp_frame **frames, idpf_xdp_tx_finalize); } =20 +static int idpf_xdpmo_rx_hash(const struct xdp_md *ctx, u32 *hash, + enum xdp_rss_hash_type *rss_type) +{ + const struct libeth_xdp_buff *xdp =3D (typeof(xdp))ctx; + struct idpf_xdp_rx_desc desc __uninitialized; + const struct idpf_rx_queue *rxq; + struct libeth_rx_pt pt; + + rxq =3D libeth_xdp_buff_to_rq(xdp, typeof(*rxq), xdp_rxq); + + idpf_xdp_get_qw0(&desc, xdp->desc); + + pt =3D rxq->rx_ptype_lkup[idpf_xdp_rx_pt(&desc)]; + if (!libeth_rx_pt_has_hash(rxq->xdp_rxq.dev, pt)) + return -ENODATA; + + idpf_xdp_get_qw2(&desc, xdp->desc); + + return libeth_xdpmo_rx_hash(hash, rss_type, idpf_xdp_rx_hash(&desc), + pt); +} + +static const struct xdp_metadata_ops idpf_xdpmo =3D { + .xmo_rx_hash =3D idpf_xdpmo_rx_hash, +}; + void idpf_xdp_set_features(const struct idpf_vport *vport) { if (!idpf_is_queue_model_split(vport->rxq_model)) return; =20 - libeth_xdp_set_features_noredir(vport->netdev); + libeth_xdp_set_features_noredir(vport->netdev, &idpf_xdpmo); } =20 static int idpf_xdp_setup_prog(struct idpf_vport *vport, --=20 2.51.0