From nobody Mon Dec 1 22:07:41 2025 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6648427703C; Thu, 27 Nov 2025 13:58:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764251932; cv=none; b=WGxlP20GROO9dIKmXx7eeRwA3WtVLxUqI3+fExkFUo9JfDiMqYXskEw4WhAe+d2OHNTQrd9ldkOvlrEWQEm6G0bluaaVVKH4DA5HPxuu5XpRBH4QmhhP/yE/GO7U780eYbvCwQSO4M6SSesvfEBHCXMjY2HX92QxxG1BhziQwOI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764251932; c=relaxed/simple; bh=UkN77h0WycF5RA0dcdHcAnnPNVo9IeOdBPJn2N+k1cw=; h=Date:From:To:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition; b=sLCoqCesytM45vYZiL+6UCploQ0T1QuzaJkMbHSVXsfhnxWlvNeA4OFEyqvh18gYPxbMOgEI0RYpDwAxuiHvfZ/3LRos2dPvfqzc0ZrnHU145+semOG/w0oqugpjNW4jPSIUjlHKScF6NrPvQoYXxyiQeDfZd0AtDwFTreJMkL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=OzSATa2/; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="OzSATa2/" Received: by linux.microsoft.com (Postfix, from userid 1204) id E4ABF2126F7A; Thu, 27 Nov 2025 05:58:49 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com E4ABF2126F7A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1764251929; bh=IouluG9PMtOZ6OngX5FzdYCbFa+r2tI8hRcBEe5WM4E=; h=Date:From:To:Subject:From; b=OzSATa2/31Sv+qjYgjHc4tAt7gVuQxujEFSMqKaaXqiUmv++ZkqKoMaG+1azas4S2 q4bq+BJw5eN2a7msojLBs/zln6aZPObarT4f8HSz7pVCp19rhtYx38b4tKUYd9VU2o HjIYBzimLALQF6ffh+npWKlkrOq91s2oUCugaPWw= Date: Thu, 27 Nov 2025 05:58:49 -0800 From: Dipayaan Roy To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, longli@microsoft.com, kotaranov@microsoft.com, horms@kernel.org, shradhagupta@linux.microsoft.com, ssengar@linux.microsoft.com, ernis@linux.microsoft.com, shirazsaleem@microsoft.com, linux-hyperv@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, dipayanroy@microsoft.com Subject: [PATCH net-next, v5] net: mana: Implement ndo_tx_timeout and serialize queue resets per port. Message-ID: <20251127135849.GA8411@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement .ndo_tx_timeout for MANA so any stalled TX queue can be detected and a device-controlled port reset for all queues can be scheduled to a ordered workqueue. The reset for all queues on stall detection is recommended by hardware team. Reviewed-by: Pavan Chebbi Reviewed-by: Haiyang Zhang Signed-off-by: Dipayaan Roy --- Changes in v5: -Fixed commit message, used 'create_singlethread_workqueue' and fixed cleanup part. Changes in v4: -Fixed commit message, work initialization before registering netdev, fixed potential null pointer de-reference bug. Changes in v3: -Fixed commit meesage, removed rtnl_trylock and added disable_work_sync, fixed mana_queue_reset_work, and few cosmetics. Changes in v2: -Fixed cosmetic changes. --- --- drivers/net/ethernet/microsoft/mana/mana_en.c | 77 ++++++++++++++++++- include/net/mana/gdma.h | 7 +- include/net/mana/mana.h | 8 +- 3 files changed, 89 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/et= hernet/microsoft/mana/mana_en.c index 1ad154f9db1a..d8451f550db4 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -299,6 +299,42 @@ static int mana_get_gso_hs(struct sk_buff *skb) return gso_hs; } =20 +static void mana_per_port_queue_reset_work_handler(struct work_struct *wor= k) +{ + struct mana_queue_reset_work *reset_queue_work =3D + container_of(work, struct mana_queue_reset_work, work); + + struct mana_port_context *apc =3D container_of(reset_queue_work, + struct mana_port_context, + queue_reset_work); + struct net_device *ndev =3D apc->ndev; + int err; + + rtnl_lock(); + + /* Pre-allocate buffers to prevent failure in mana_attach later */ + err =3D mana_pre_alloc_rxbufs(apc, ndev->mtu, apc->num_queues); + if (err) { + netdev_err(ndev, "Insufficient memory for reset post tx stall detection\= n"); + goto out; + } + + err =3D mana_detach(ndev, false); + if (err) { + netdev_err(ndev, "mana_detach failed: %d\n", err); + goto dealloc_pre_rxbufs; + } + + err =3D mana_attach(ndev); + if (err) + netdev_err(ndev, "mana_attach failed: %d\n", err); + +dealloc_pre_rxbufs: + mana_pre_dealloc_rxbufs(apc); +out: + rtnl_unlock(); +} + netdev_tx_t mana_start_xmit(struct sk_buff *skb, struct net_device *ndev) { enum mana_tx_pkt_format pkt_fmt =3D MANA_SHORT_PKT_FMT; @@ -839,6 +875,23 @@ static int mana_change_mtu(struct net_device *ndev, in= t new_mtu) return err; } =20 +static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueu= e) +{ + struct mana_port_context *apc =3D netdev_priv(netdev); + struct mana_context *ac =3D apc->ac; + struct gdma_context *gc =3D ac->gdma_dev->gdma_context; + + /* Already in service, hence tx queue reset is not required.*/ + if (gc->in_service) + return; + + /* Note: If there are pending queue reset work for this port(apc), + * subsequent request queued up from here are ignored. This is because + * we are using the same work instance per port(apc). + */ + queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work.work); +} + static int mana_shaper_set(struct net_shaper_binding *binding, const struct net_shaper *shaper, struct netlink_ext_ack *extack) @@ -924,6 +977,7 @@ static const struct net_device_ops mana_devops =3D { .ndo_bpf =3D mana_bpf, .ndo_xdp_xmit =3D mana_xdp_xmit, .ndo_change_mtu =3D mana_change_mtu, + .ndo_tx_timeout =3D mana_tx_timeout, .net_shaper_ops =3D &mana_shaper_ops, }; =20 @@ -3287,6 +3341,8 @@ static int mana_probe_port(struct mana_context *ac, i= nt port_idx, ndev->min_mtu =3D ETH_MIN_MTU; ndev->needed_headroom =3D MANA_HEADROOM; ndev->dev_port =3D port_idx; + /* Recommended timeout based on HW FPGA re-config scenario. */ + ndev->watchdog_timeo =3D 15 * HZ; SET_NETDEV_DEV(ndev, gc->dev); =20 netif_set_tso_max_size(ndev, GSO_MAX_SIZE); @@ -3303,6 +3359,10 @@ static int mana_probe_port(struct mana_context *ac, = int port_idx, if (err) goto reset_apc; =20 + /* Initialize the per port queue reset work.*/ + INIT_WORK(&apc->queue_reset_work.work, + mana_per_port_queue_reset_work_handler); + netdev_lockdep_set_classes(ndev); =20 ndev->hw_features =3D NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM; @@ -3549,6 +3609,14 @@ int mana_probe(struct gdma_dev *gd, bool resuming) if (ac->num_ports > MAX_PORTS_IN_MANA_DEV) ac->num_ports =3D MAX_PORTS_IN_MANA_DEV; =20 + ac->per_port_queue_reset_wq =3D + create_singlethread_workqueue("mana_per_port_queue_reset_wq"); + if (!ac->per_port_queue_reset_wq) { + dev_err(dev, "Failed to allocate per port queue reset workqueue\n"); + err =3D -ENOMEM; + goto out; + } + if (!resuming) { for (i =3D 0; i < ac->num_ports; i++) { err =3D mana_probe_port(ac, i, &ac->ports[i]); @@ -3616,13 +3684,15 @@ void mana_remove(struct gdma_dev *gd, bool suspendi= ng) =20 for (i =3D 0; i < ac->num_ports; i++) { ndev =3D ac->ports[i]; - apc =3D netdev_priv(ndev); if (!ndev) { if (i =3D=3D 0) dev_err(dev, "No net device to remove\n"); goto out; } =20 + apc =3D netdev_priv(ndev); + disable_work_sync(&apc->queue_reset_work.work); + /* All cleanup actions should stay after rtnl_lock(), otherwise * other functions may access partially cleaned up data. */ @@ -3649,6 +3719,11 @@ void mana_remove(struct gdma_dev *gd, bool suspendin= g) =20 mana_destroy_eq(ac); out: + if (ac->per_port_queue_reset_wq) { + destroy_workqueue(ac->per_port_queue_reset_wq); + ac->per_port_queue_reset_wq =3D NULL; + } + mana_gd_deregister_device(gd); =20 if (suspending) diff --git a/include/net/mana/gdma.h b/include/net/mana/gdma.h index a4cf307859f8..808622ae5ccc 100644 --- a/include/net/mana/gdma.h +++ b/include/net/mana/gdma.h @@ -592,6 +592,10 @@ enum { =20 /* Driver can self reset on FPGA Reconfig EQE notification */ #define GDMA_DRV_CAP_FLAG_1_HANDLE_RECONFIG_EQE BIT(17) + +/* Driver detects stalled send queues and recovers them */ +#define GDMA_DRV_CAP_FLAG_1_HANDLE_STALL_SQ_RECOVERY BIT(18) + #define GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE BIT(6) =20 /* Driver supports linearizing the skb when num_sge exceeds hardware limit= */ @@ -611,7 +615,8 @@ enum { GDMA_DRV_CAP_FLAG_1_HANDLE_RECONFIG_EQE | \ GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE | \ GDMA_DRV_CAP_FLAG_1_PERIODIC_STATS_QUERY | \ - GDMA_DRV_CAP_FLAG_1_SKB_LINEARIZE) + GDMA_DRV_CAP_FLAG_1_SKB_LINEARIZE | \ + GDMA_DRV_CAP_FLAG_1_HANDLE_STALL_SQ_RECOVERY) =20 #define GDMA_DRV_CAP_FLAGS2 0 =20 diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h index d7e089c6b694..cef78a871c7c 100644 --- a/include/net/mana/mana.h +++ b/include/net/mana/mana.h @@ -480,7 +480,7 @@ struct mana_context { struct mana_ethtool_hc_stats hc_stats; struct mana_eq *eqs; struct dentry *mana_eqs_debugfs; - + struct workqueue_struct *per_port_queue_reset_wq; /* Workqueue for querying hardware stats */ struct delayed_work gf_stats_work; bool hwc_timeout_occurred; @@ -492,9 +492,15 @@ struct mana_context { u32 link_event; }; =20 +struct mana_queue_reset_work { + /* Work structure */ + struct work_struct work; +}; + struct mana_port_context { struct mana_context *ac; struct net_device *ndev; + struct mana_queue_reset_work queue_reset_work; =20 u8 mac_addr[ETH_ALEN]; =20 --=20 2.43.0