Frames flagged with IEEE80211_TX_CTL_REQ_TX_STATUS mean the driver has to
report to mac80211 stack whether AP sent ACK for the null frame/probe
request or not.  It's not implemented in USB part of the driver yet.

PCIe HCI has its own way of getting TX status incorporated into RPP
feature, and it's always enabled there.  Other HCIs need a different
scheme based on processing C2H messages.

Thus define a .tx_rpt_enabled flag defining which HCIs need to enable a TX
report feature.  Currently it is USB only.

Toggle a bit in the TX descriptor and place flagged skbs in a queue to
wait for a message from the firmware.  The main part is quite similar to
what rtw88 does.  The difference is that rtw89 has a new feature providing
a TX report for each transmission attempt.  Ignore a failed TX status
reported by the firmware until retry limit is reached or successful status
appears.  When there is no success and the retry limit is reached, report
the frame up to the wireless stack as failed eventually.

Forcefully flush the queue on HCI reset.

Found by Linux Verification Center (linuxtesting.org).

Signed-off-by: Fedor Pchelkin <pchelkin@ispras.ru>
---

v3: - don't mix TX RPT queue with TX wait list processing (Ping-Ke)
    - add struct rtw89_tx_rpt for convenience, that's also like in rtw88
    - update changelog to reflect TX RPT retry logic and add some
      comments inside rtw89_mac_c2h_tx_rpt()

v2: - update TX rpt description in rtw89_core_tx_update_desc_info()
    - add a flag in rtw89_hci_info to determine if we should enable TX report (Ping-Ke)
    - refine rtw89_tx_rpt_queue_purge() - it's called on USB HCI reset
      and at rtw89_tx_wait_work.  Queueing delayed rtw89_tx_wait_work may be
      suboptimal but basically it's what rtw88 does with timer stuff.

 drivers/net/wireless/realtek/rtw89/core.c |  4 ++
 drivers/net/wireless/realtek/rtw89/core.h | 10 ++++
 drivers/net/wireless/realtek/rtw89/mac.c  | 24 ++++++++++
 drivers/net/wireless/realtek/rtw89/mac.h  | 56 +++++++++++++++++++++++
 drivers/net/wireless/realtek/rtw89/usb.c  | 15 +++++-
 5 files changed, 108 insertions(+), 1 deletion(-)

diff --git a/drivers/net/wireless/realtek/rtw89/core.c b/drivers/net/wireless/realtek/rtw89/core.c
index abe8eec1d0f5..3aa9a9a28118 100644
--- a/drivers/net/wireless/realtek/rtw89/core.c
+++ b/drivers/net/wireless/realtek/rtw89/core.c
@@ -1112,6 +1112,9 @@ rtw89_core_tx_update_desc_info(struct rtw89_dev *rtwdev,
 	if (addr_cam->valid && desc_info->mlo)
 		upd_wlan_hdr = true;
 
+	if (rtw89_is_tx_rpt_skb(tx_req->skb))
+		rtw89_tx_rpt_init(rtwdev, tx_req);
+
 	is_bmc = (is_broadcast_ether_addr(hdr->addr1) ||
 		  is_multicast_ether_addr(hdr->addr1));
 
@@ -5849,6 +5852,7 @@ int rtw89_core_init(struct rtw89_dev *rtwdev)
 	wiphy_work_init(&rtwdev->cancel_6ghz_probe_work, rtw89_cancel_6ghz_probe_work);
 	INIT_WORK(&rtwdev->load_firmware_work, rtw89_load_firmware_work);
 
+	skb_queue_head_init(&rtwdev->tx_rpt.queue);
 	skb_queue_head_init(&rtwdev->c2h_queue);
 	rtw89_core_ppdu_sts_init(rtwdev);
 	rtw89_traffic_stats_init(rtwdev, &rtwdev->stats);
diff --git a/drivers/net/wireless/realtek/rtw89/core.h b/drivers/net/wireless/realtek/rtw89/core.h
index 66b7bfa5902e..8641e3a8d36d 100644
--- a/drivers/net/wireless/realtek/rtw89/core.h
+++ b/drivers/net/wireless/realtek/rtw89/core.h
@@ -3516,6 +3516,11 @@ struct rtw89_phy_rate_pattern {
 #define RTW89_TX_LIFE_TIME		0x2
 #define RTW89_TX_MACID_DROP		0x3
 
+struct rtw89_tx_rpt {
+	struct sk_buff_head queue;
+	atomic_t sn;
+};
+
 #define RTW89_TX_WAIT_WORK_TIMEOUT msecs_to_jiffies(500)
 struct rtw89_tx_wait_info {
 	struct rcu_head rcu_head;
@@ -3527,6 +3532,8 @@ struct rtw89_tx_wait_info {
 
 struct rtw89_tx_skb_data {
 	struct rtw89_tx_wait_info __rcu *wait;
+	u8 tx_rpt_sn;
+	u8 tx_pkt_cnt_lmt;
 	u8 hci_priv[];
 };
 
@@ -3696,6 +3703,7 @@ struct rtw89_hci_info {
 	u32 rpwm_addr;
 	u32 cpwm_addr;
 	bool paused;
+	bool tx_rpt_enabled;
 };
 
 struct rtw89_chip_ops {
@@ -6015,6 +6023,8 @@ struct rtw89_dev {
 	struct list_head tx_waits;
 	struct wiphy_delayed_work tx_wait_work;
 
+	struct rtw89_tx_rpt tx_rpt;
+
 	struct rtw89_cam_info cam_info;
 
 	struct sk_buff_head c2h_queue;
diff --git a/drivers/net/wireless/realtek/rtw89/mac.c b/drivers/net/wireless/realtek/rtw89/mac.c
index 2fe239f18534..26c7476afdec 100644
--- a/drivers/net/wireless/realtek/rtw89/mac.c
+++ b/drivers/net/wireless/realtek/rtw89/mac.c
@@ -5460,7 +5460,11 @@ rtw89_mac_c2h_mcc_status_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32
 static void
 rtw89_mac_c2h_tx_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len)
 {
+	struct rtw89_tx_rpt *tx_rpt = &rtwdev->tx_rpt;
 	u8 sw_define, tx_status, data_txcnt;
+	struct rtw89_tx_skb_data *skb_data;
+	struct sk_buff *skb, *tmp;
+	unsigned long flags;
 
 	if (rtwdev->chip->chip_id == RTL8922A) {
 		const struct rtw89_c2h_mac_tx_rpt_v2 *rpt_v2;
@@ -5484,6 +5488,26 @@ rtw89_mac_c2h_tx_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len)
 	rtw89_debug(rtwdev, RTW89_DBG_TXRX,
 		    "C2H TX RPT: sn %d, tx_status %d, data_txcnt %d\n",
 		    sw_define, tx_status, data_txcnt);
+
+	spin_lock_irqsave(&tx_rpt->queue.lock, flags);
+	skb_queue_walk_safe(&tx_rpt->queue, skb, tmp) {
+		skb_data = RTW89_TX_SKB_CB(skb);
+
+		/* skip if sequence number doesn't match */
+		if (sw_define != skb_data->tx_rpt_sn)
+			continue;
+		/* skip if TX attempt has failed and retry limit has not been
+		 * reached yet
+		 */
+		if (tx_status != RTW89_TX_DONE &&
+		    data_txcnt != skb_data->tx_pkt_cnt_lmt)
+			continue;
+
+		__skb_unlink(skb, &tx_rpt->queue);
+		rtw89_tx_rpt_tx_status(rtwdev, skb, tx_status);
+		break;
+	}
+	spin_unlock_irqrestore(&tx_rpt->queue.lock, flags);
 }
 
 static void
diff --git a/drivers/net/wireless/realtek/rtw89/mac.h b/drivers/net/wireless/realtek/rtw89/mac.h
index 15c5c7e4033c..e8bd92223497 100644
--- a/drivers/net/wireless/realtek/rtw89/mac.h
+++ b/drivers/net/wireless/realtek/rtw89/mac.h
@@ -1616,4 +1616,60 @@ int rtw89_mac_scan_offload(struct rtw89_dev *rtwdev,
 
 	return ret;
 }
+
+static inline
+void rtw89_tx_rpt_init(struct rtw89_dev *rtwdev,
+		       struct rtw89_core_tx_request *tx_req)
+{
+	struct rtw89_tx_rpt *tx_rpt = &rtwdev->tx_rpt;
+
+	if (!rtwdev->hci.tx_rpt_enabled)
+		return;
+
+	tx_req->desc_info.report = true;
+	/* firmware maintains a 4-bit sequence number */
+	tx_req->desc_info.sn = atomic_inc_return(&tx_rpt->sn) & 0xF;
+	tx_req->desc_info.tx_cnt_lmt_en = true;
+	tx_req->desc_info.tx_cnt_lmt = 8;
+}
+
+static inline
+bool rtw89_is_tx_rpt_skb(struct sk_buff *skb)
+{
+	struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+
+	return info->flags & IEEE80211_TX_CTL_REQ_TX_STATUS;
+}
+
+static inline
+void rtw89_tx_rpt_tx_status(struct rtw89_dev *rtwdev, struct sk_buff *skb, u8 tx_status)
+{
+	struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+
+	ieee80211_tx_info_clear_status(info);
+	if (tx_status == RTW89_TX_DONE)
+		info->flags |= IEEE80211_TX_STAT_ACK;
+	else
+		info->flags &= ~IEEE80211_TX_STAT_ACK;
+
+	ieee80211_tx_status_irqsafe(rtwdev->hw, skb);
+}
+
+static inline
+void rtw89_tx_rpt_queue_purge(struct rtw89_dev *rtwdev)
+{
+	struct rtw89_tx_rpt *tx_rpt = &rtwdev->tx_rpt;
+	struct sk_buff_head q;
+	struct sk_buff *skb;
+	unsigned long flags;
+
+	__skb_queue_head_init(&q);
+
+	spin_lock_irqsave(&tx_rpt->queue.lock, flags);
+	skb_queue_splice_init(&tx_rpt->queue, &q);
+	spin_unlock_irqrestore(&tx_rpt->queue.lock, flags);
+
+	while ((skb = __skb_dequeue(&q)))
+		rtw89_tx_rpt_tx_status(rtwdev, skb, RTW89_TX_MACID_DROP);
+}
 #endif
diff --git a/drivers/net/wireless/realtek/rtw89/usb.c b/drivers/net/wireless/realtek/rtw89/usb.c
index 655e8437d62e..22994c3501f8 100644
--- a/drivers/net/wireless/realtek/rtw89/usb.c
+++ b/drivers/net/wireless/realtek/rtw89/usb.c
@@ -216,6 +216,14 @@ static void rtw89_usb_write_port_complete(struct urb *urb)
 		skb_pull(skb, txdesc_size);
 
 		info = IEEE80211_SKB_CB(skb);
+		if (rtw89_is_tx_rpt_skb(skb)) {
+			/* sequence number is passed to rtw89_mac_c2h_tx_rpt() via
+			 * driver data
+			 */
+			skb_queue_tail(&rtwdev->tx_rpt.queue, skb);
+			continue;
+		}
+
 		ieee80211_tx_info_clear_status(info);
 
 		if (urb->status == 0) {
@@ -372,6 +380,7 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
 {
 	struct rtw89_tx_desc_info *desc_info = &tx_req->desc_info;
 	struct rtw89_usb *rtwusb = rtw89_usb_priv(rtwdev);
+	struct rtw89_tx_skb_data *skb_data;
 	struct sk_buff *skb = tx_req->skb;
 	struct rtw89_txwd_body *txdesc;
 	u32 txdesc_size;
@@ -398,6 +407,9 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
 
 	le32p_replace_bits(&txdesc->dword0, 1, RTW89_TXWD_BODY0_STF_MODE);
 
+	skb_data = RTW89_TX_SKB_CB(skb);
+	skb_data->tx_rpt_sn = tx_req->desc_info.sn;
+
 	skb_queue_tail(&rtwusb->tx_queue[desc_info->ch_dma], skb);
 
 	return 0;
@@ -678,7 +690,7 @@ static void rtw89_usb_deinit_tx(struct rtw89_dev *rtwdev)
 
 static void rtw89_usb_ops_reset(struct rtw89_dev *rtwdev)
 {
-	/* TODO: anything to do here? */
+	rtw89_tx_rpt_queue_purge(rtwdev);
 }
 
 static int rtw89_usb_ops_start(struct rtw89_dev *rtwdev)
@@ -962,6 +974,7 @@ int rtw89_usb_probe(struct usb_interface *intf,
 
 	rtwdev->hci.ops = &rtw89_usb_ops;
 	rtwdev->hci.type = RTW89_HCI_TYPE_USB;
+	rtwdev->hci.tx_rpt_enabled = true;
 
 	ret = rtw89_usb_intf_init(rtwdev, intf);
 	if (ret) {
-- 
2.51.0

Ping-Ke Shih <pkshih@realtek.com> wrote:
> 
> Also, all are dropped, can't we just call ieee80211_purge_tx_queue()?
> 

Please ignore this comment. Since you want to complete wait, then
rtw89_tx_wait_list_clear() can delete wait marked as completed from list.

Fedor Pchelkin <pchelkin@ispras.ru> wrote:

[...]

> diff --git a/drivers/net/wireless/realtek/rtw89/core.c b/drivers/net/wireless/realtek/rtw89/core.c
> index abe8eec1d0f5..3aa9a9a28118 100644
> --- a/drivers/net/wireless/realtek/rtw89/core.c
> +++ b/drivers/net/wireless/realtek/rtw89/core.c
> @@ -1112,6 +1112,9 @@ rtw89_core_tx_update_desc_info(struct rtw89_dev *rtwdev,
>         if (addr_cam->valid && desc_info->mlo)
>                 upd_wlan_hdr = true;
> 
> +       if (rtw89_is_tx_rpt_skb(tx_req->skb))
> +               rtw89_tx_rpt_init(rtwdev, tx_req);
> +
>         is_bmc = (is_broadcast_ether_addr(hdr->addr1) ||
>                   is_multicast_ether_addr(hdr->addr1));
> 
> @@ -5849,6 +5852,7 @@ int rtw89_core_init(struct rtw89_dev *rtwdev)
>         wiphy_work_init(&rtwdev->cancel_6ghz_probe_work, rtw89_cancel_6ghz_probe_work);
>         INIT_WORK(&rtwdev->load_firmware_work, rtw89_load_firmware_work);
> 
> +       skb_queue_head_init(&rtwdev->tx_rpt.queue);

not sure if it's worth to initialize tx_rpt.sn to zero?

>         skb_queue_head_init(&rtwdev->c2h_queue);
>         rtw89_core_ppdu_sts_init(rtwdev);
>         rtw89_traffic_stats_init(rtwdev, &rtwdev->stats);
> diff --git a/drivers/net/wireless/realtek/rtw89/core.h b/drivers/net/wireless/realtek/rtw89/core.h
> index 66b7bfa5902e..8641e3a8d36d 100644
> --- a/drivers/net/wireless/realtek/rtw89/core.h
> +++ b/drivers/net/wireless/realtek/rtw89/core.h
> @@ -3516,6 +3516,11 @@ struct rtw89_phy_rate_pattern {
>  #define RTW89_TX_LIFE_TIME             0x2
>  #define RTW89_TX_MACID_DROP            0x3
> 
> +struct rtw89_tx_rpt {
> +       struct sk_buff_head queue;
> +       atomic_t sn;
> +};
> +
>  #define RTW89_TX_WAIT_WORK_TIMEOUT msecs_to_jiffies(500)
>  struct rtw89_tx_wait_info {
>         struct rcu_head rcu_head;
> @@ -3527,6 +3532,8 @@ struct rtw89_tx_wait_info {
> 
>  struct rtw89_tx_skb_data {
>         struct rtw89_tx_wait_info __rcu *wait;
> +       u8 tx_rpt_sn;
> +       u8 tx_pkt_cnt_lmt;
>         u8 hci_priv[];
>  };
> 
> @@ -3696,6 +3703,7 @@ struct rtw89_hci_info {
>         u32 rpwm_addr;
>         u32 cpwm_addr;
>         bool paused;
> +       bool tx_rpt_enabled;
>  };
> 
>  struct rtw89_chip_ops {
> @@ -6015,6 +6023,8 @@ struct rtw89_dev {
>         struct list_head tx_waits;
>         struct wiphy_delayed_work tx_wait_work;
> 
> +       struct rtw89_tx_rpt tx_rpt;
> +
>         struct rtw89_cam_info cam_info;
> 
>         struct sk_buff_head c2h_queue;
> diff --git a/drivers/net/wireless/realtek/rtw89/mac.c b/drivers/net/wireless/realtek/rtw89/mac.c
> index 2fe239f18534..26c7476afdec 100644
> --- a/drivers/net/wireless/realtek/rtw89/mac.c
> +++ b/drivers/net/wireless/realtek/rtw89/mac.c
> @@ -5460,7 +5460,11 @@ rtw89_mac_c2h_mcc_status_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32
>  static void
>  rtw89_mac_c2h_tx_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len)
>  {
> +       struct rtw89_tx_rpt *tx_rpt = &rtwdev->tx_rpt;
>         u8 sw_define, tx_status, data_txcnt;
> +       struct rtw89_tx_skb_data *skb_data;
> +       struct sk_buff *skb, *tmp;
> +       unsigned long flags;
> 
>         if (rtwdev->chip->chip_id == RTL8922A) {
>                 const struct rtw89_c2h_mac_tx_rpt_v2 *rpt_v2;
> @@ -5484,6 +5488,26 @@ rtw89_mac_c2h_tx_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len)
>         rtw89_debug(rtwdev, RTW89_DBG_TXRX,
>                     "C2H TX RPT: sn %d, tx_status %d, data_txcnt %d\n",
>                     sw_define, tx_status, data_txcnt);
> +
> +       spin_lock_irqsave(&tx_rpt->queue.lock, flags);
> +       skb_queue_walk_safe(&tx_rpt->queue, skb, tmp) {
> +               skb_data = RTW89_TX_SKB_CB(skb);
> +
> +               /* skip if sequence number doesn't match */
> +               if (sw_define != skb_data->tx_rpt_sn)
> +                       continue;
> +               /* skip if TX attempt has failed and retry limit has not been
> +                * reached yet
> +                */
> +               if (tx_status != RTW89_TX_DONE &&
> +                   data_txcnt != skb_data->tx_pkt_cnt_lmt)
> +                       continue;
> +
> +               __skb_unlink(skb, &tx_rpt->queue);
> +               rtw89_tx_rpt_tx_status(rtwdev, skb, tx_status);

Would it be better to run rtw89_tx_rpt_tx_status() after this loop outside
spin_lock()?

> +               break;
> +       }
> +       spin_unlock_irqrestore(&tx_rpt->queue.lock, flags);
>  }
> 
>  static void
> diff --git a/drivers/net/wireless/realtek/rtw89/mac.h b/drivers/net/wireless/realtek/rtw89/mac.h
> index 15c5c7e4033c..e8bd92223497 100644
> --- a/drivers/net/wireless/realtek/rtw89/mac.h
> +++ b/drivers/net/wireless/realtek/rtw89/mac.h
> @@ -1616,4 +1616,60 @@ int rtw89_mac_scan_offload(struct rtw89_dev *rtwdev,
> 
>         return ret;
>  }
> +
> +static inline
> +void rtw89_tx_rpt_init(struct rtw89_dev *rtwdev,
> +                      struct rtw89_core_tx_request *tx_req)
> +{
> +       struct rtw89_tx_rpt *tx_rpt = &rtwdev->tx_rpt;
> +
> +       if (!rtwdev->hci.tx_rpt_enabled)
> +               return;
> +
> +       tx_req->desc_info.report = true;
> +       /* firmware maintains a 4-bit sequence number */
> +       tx_req->desc_info.sn = atomic_inc_return(&tx_rpt->sn) & 0xF;
> +       tx_req->desc_info.tx_cnt_lmt_en = true;
> +       tx_req->desc_info.tx_cnt_lmt = 8;
> +}
> +
> +static inline
> +bool rtw89_is_tx_rpt_skb(struct sk_buff *skb)
> +{
> +       struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
> +
> +       return info->flags & IEEE80211_TX_CTL_REQ_TX_STATUS;
> +}
> +
> +static inline
> +void rtw89_tx_rpt_tx_status(struct rtw89_dev *rtwdev, struct sk_buff *skb, u8 tx_status)
> +{
> +       struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
> +
> +       ieee80211_tx_info_clear_status(info);
> +       if (tx_status == RTW89_TX_DONE)
> +               info->flags |= IEEE80211_TX_STAT_ACK;
> +       else
> +               info->flags &= ~IEEE80211_TX_STAT_ACK;
> +
> +       ieee80211_tx_status_irqsafe(rtwdev->hw, skb);
> +}
> +
> +static inline
> +void rtw89_tx_rpt_queue_purge(struct rtw89_dev *rtwdev)
> +{
> +       struct rtw89_tx_rpt *tx_rpt = &rtwdev->tx_rpt;
> +       struct sk_buff_head q;
> +       struct sk_buff *skb;
> +       unsigned long flags;
> +
> +       __skb_queue_head_init(&q);
> +
> +       spin_lock_irqsave(&tx_rpt->queue.lock, flags);
> +       skb_queue_splice_init(&tx_rpt->queue, &q);
> +       spin_unlock_irqrestore(&tx_rpt->queue.lock, flags);
> +
> +       while ((skb = __skb_dequeue(&q)))
> +               rtw89_tx_rpt_tx_status(rtwdev, skb, RTW89_TX_MACID_DROP);
> +}
>  #endif
> diff --git a/drivers/net/wireless/realtek/rtw89/usb.c b/drivers/net/wireless/realtek/rtw89/usb.c
> index 655e8437d62e..22994c3501f8 100644
> --- a/drivers/net/wireless/realtek/rtw89/usb.c
> +++ b/drivers/net/wireless/realtek/rtw89/usb.c
> @@ -216,6 +216,14 @@ static void rtw89_usb_write_port_complete(struct urb *urb)
>                 skb_pull(skb, txdesc_size);
> 
>                 info = IEEE80211_SKB_CB(skb);
> +               if (rtw89_is_tx_rpt_skb(skb)) {
> +                       /* sequence number is passed to rtw89_mac_c2h_tx_rpt() via

nit: The 'via' is over 80 characters a little bit. Move to next line.

> +                        * driver data
> +                        */
> +                       skb_queue_tail(&rtwdev->tx_rpt.queue, skb);
> +                       continue;
> +               }
> +
>                 ieee80211_tx_info_clear_status(info);
> 
>                 if (urb->status == 0) {

Should we move this checking upward? Enqueue skb into tx_rpt_skb only if
urb->status == 0?

> @@ -372,6 +380,7 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
>  {
>         struct rtw89_tx_desc_info *desc_info = &tx_req->desc_info;
>         struct rtw89_usb *rtwusb = rtw89_usb_priv(rtwdev);
> +       struct rtw89_tx_skb_data *skb_data;
>         struct sk_buff *skb = tx_req->skb;
>         struct rtw89_txwd_body *txdesc;
>         u32 txdesc_size;
> @@ -398,6 +407,9 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
> 
>         le32p_replace_bits(&txdesc->dword0, 1, RTW89_TXWD_BODY0_STF_MODE);
> 
> +       skb_data = RTW89_TX_SKB_CB(skb);
> +       skb_data->tx_rpt_sn = tx_req->desc_info.sn;

Shouldn't set skb_data->tx_pkt_cnt_lmt? 

skb_data->tx_pkt_cnt_lmt = tx_req->desc_info.tx_cnt_lmt;

Also, should we check desc_info.{report, tx_cnt_lmt_en} individually before 
setting?


> +
>         skb_queue_tail(&rtwusb->tx_queue[desc_info->ch_dma], skb);
> 
>         return 0;
> @@ -678,7 +690,7 @@ static void rtw89_usb_deinit_tx(struct rtw89_dev *rtwdev)
> 
>  static void rtw89_usb_ops_reset(struct rtw89_dev *rtwdev)
>  {
> -       /* TODO: anything to do here? */
> +       rtw89_tx_rpt_queue_purge(rtwdev);

Have you consider the SKB that has been rtw89_usb_write_port() but
has not yet rtw89_usb_write_port_complete()?

Since we call rtw89_mac_pwr_off() before rtw89_hci_reset() in 
rtw89_core_stop(), it should be not more C2H at rtw89_hci_reset().
It seems to be safe, right?

Also, all are dropped, can't we just call ieee80211_purge_tx_queue()?

>  }
> 
>  static int rtw89_usb_ops_start(struct rtw89_dev *rtwdev)
> @@ -962,6 +974,7 @@ int rtw89_usb_probe(struct usb_interface *intf,
> 
>         rtwdev->hci.ops = &rtw89_usb_ops;
>         rtwdev->hci.type = RTW89_HCI_TYPE_USB;
> +       rtwdev->hci.tx_rpt_enabled = true;
> 
>         ret = rtw89_usb_intf_init(rtwdev, intf);
>         if (ret) {
> --
> 2.51.0
> 

On Wed, 22. Oct 07:16, Ping-Ke Shih wrote:
> Fedor Pchelkin <pchelkin@ispras.ru> wrote:
> > @@ -5849,6 +5852,7 @@ int rtw89_core_init(struct rtw89_dev *rtwdev)
> >         wiphy_work_init(&rtwdev->cancel_6ghz_probe_work, rtw89_cancel_6ghz_probe_work);
> >         INIT_WORK(&rtwdev->load_firmware_work, rtw89_load_firmware_work);
> > 
> > +       skb_queue_head_init(&rtwdev->tx_rpt.queue);
> 
> not sure if it's worth to initialize tx_rpt.sn to zero?

That shouldn't be needed because rtwdev is zero initialized in
rtw89_alloc_ieee80211_hw().  ieee80211_alloc_hw() fills the private
driver part with zeroes.

> > @@ -5484,6 +5488,26 @@ rtw89_mac_c2h_tx_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len)
> >         rtw89_debug(rtwdev, RTW89_DBG_TXRX,
> >                     "C2H TX RPT: sn %d, tx_status %d, data_txcnt %d\n",
> >                     sw_define, tx_status, data_txcnt);
> > +
> > +       spin_lock_irqsave(&tx_rpt->queue.lock, flags);
> > +       skb_queue_walk_safe(&tx_rpt->queue, skb, tmp) {
> > +               skb_data = RTW89_TX_SKB_CB(skb);
> > +
> > +               /* skip if sequence number doesn't match */
> > +               if (sw_define != skb_data->tx_rpt_sn)
> > +                       continue;
> > +               /* skip if TX attempt has failed and retry limit has not been
> > +                * reached yet
> > +                */
> > +               if (tx_status != RTW89_TX_DONE &&
> > +                   data_txcnt != skb_data->tx_pkt_cnt_lmt)
> > +                       continue;
> > +
> > +               __skb_unlink(skb, &tx_rpt->queue);
> > +               rtw89_tx_rpt_tx_status(rtwdev, skb, tx_status);
> 
> Would it be better to run rtw89_tx_rpt_tx_status() after this loop outside
> spin_lock()?

I don't have a strong opinion here: PCIe side implements the release
like

rtw89_pci_poll_rpq_dma()
 spin_lock_bh(&rtwpci->trx_lock)
 rtw89_pci_release_tx()
 ...
   rtw89_pci_release_txwd_skb()
	skb_queue_walk_safe(&txwd->queue, skb, tmp) {
		skb_unlink(skb, &txwd->queue);

		tx_data = RTW89_PCI_TX_SKB_CB(skb);
		dma_unmap_single(&rtwpci->pdev->dev, tx_data->dma, skb->len,
				 DMA_TO_DEVICE);

		rtw89_pci_tx_status(rtwdev, tx_ring, skb, tx_status);
	}
 ...
 spin_unlock_bh(&rtwpci->trx_lock)


Apart from bh/irqsave part the iteration over skbs looks visually similar
at the moment.

> > --- a/drivers/net/wireless/realtek/rtw89/usb.c
> > +++ b/drivers/net/wireless/realtek/rtw89/usb.c
> > @@ -216,6 +216,14 @@ static void rtw89_usb_write_port_complete(struct urb *urb)
> >                 skb_pull(skb, txdesc_size);
> > 
> >                 info = IEEE80211_SKB_CB(skb);
> > +               if (rtw89_is_tx_rpt_skb(skb)) {
> > +                       /* sequence number is passed to rtw89_mac_c2h_tx_rpt() via
> 
> nit: The 'via' is over 80 characters a little bit. Move to next line.
> 
> > +                        * driver data
> > +                        */
> > +                       skb_queue_tail(&rtwdev->tx_rpt.queue, skb);
> > +                       continue;
> > +               }
> > +
> >                 ieee80211_tx_info_clear_status(info);
> > 
> >                 if (urb->status == 0) {
> 
> Should we move this checking upward? Enqueue skb into tx_rpt_skb only if
> urb->status == 0?

Yep, I agree we can do it.  Just need to report it immediately with
RTW89_TX_MACID_DROP status.

As it currently stands, we should not receive notification from the
firmware for such skb so it would remain inside tx_rpt.queue until HCI
reset occurs.

However, I've not considered the case when the queue is full and we start
queueing skbs with duplicate sequence numbers - the overall range we have
is just 0xF, theoretically the situation is possible if the firmware
fatally crashes and doesn't provide notifications in time.  IMHO the TX
status will be the last thing we're going to be interested in when this
happens.  In the end, HCI reset during SER activity will purge the queue.

But the possibility of having skbs with duplicate seq numbers is not good.
Though I'm not sure if we can ever hit such a situation... Generally it
indicates that the firmware doesn't respond or performs very badly, and
we'd better reset it or something.  What do you think on this?

> 
> > @@ -372,6 +380,7 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
> >  {
> >         struct rtw89_tx_desc_info *desc_info = &tx_req->desc_info;
> >         struct rtw89_usb *rtwusb = rtw89_usb_priv(rtwdev);
> > +       struct rtw89_tx_skb_data *skb_data;
> >         struct sk_buff *skb = tx_req->skb;
> >         struct rtw89_txwd_body *txdesc;
> >         u32 txdesc_size;
> > @@ -398,6 +407,9 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
> > 
> >         le32p_replace_bits(&txdesc->dword0, 1, RTW89_TXWD_BODY0_STF_MODE);
> > 
> > +       skb_data = RTW89_TX_SKB_CB(skb);
> > +       skb_data->tx_rpt_sn = tx_req->desc_info.sn;
> 
> Shouldn't set skb_data->tx_pkt_cnt_lmt? 
> 
> skb_data->tx_pkt_cnt_lmt = tx_req->desc_info.tx_cnt_lmt;
> 
> Also, should we check desc_info.{report, tx_cnt_lmt_en} individually before 
> setting?

Right, this all makes sense.  Will fix it, thanks!

> 
> 
> > +
> >         skb_queue_tail(&rtwusb->tx_queue[desc_info->ch_dma], skb);
> > 
> >         return 0;
> > @@ -678,7 +690,7 @@ static void rtw89_usb_deinit_tx(struct rtw89_dev *rtwdev)
> > 
> >  static void rtw89_usb_ops_reset(struct rtw89_dev *rtwdev)
> >  {
> > -       /* TODO: anything to do here? */
> > +       rtw89_tx_rpt_queue_purge(rtwdev);
> 
> Have you consider the SKB that has been rtw89_usb_write_port() but
> has not yet rtw89_usb_write_port_complete()?
> 
> Since we call rtw89_mac_pwr_off() before rtw89_hci_reset() in 
> rtw89_core_stop(), it should be not more C2H at rtw89_hci_reset().
> It seems to be safe, right?

Well, rtw89_usb_write_port_complete() is asynchronous URB callback managed
by USB subsystem and it's mainly independent of rtw89.  We're guaranteed
all pending URB completions will complete when the device is being
disconnected and rtw89_usb_disconnect() method is called.  That's all, I
think.

In other call sites like SER we risk URB completion be called after
purging the queue, the only consequence will be the obsolete skb still
added to the queue.

We can implement an anchor for TX URBs and explicitly wait with
usb_kill_anchored_urbs() in ->reset() until all pending URB completions
are done, and then purge the queue.

If nothing else, I'll add it in the next respin of the series.

Fedor Pchelkin <pchelkin@ispras.ru> wrote:
> On Wed, 22. Oct 07:16, Ping-Ke Shih wrote:
> > Fedor Pchelkin <pchelkin@ispras.ru> wrote:
> > > @@ -5849,6 +5852,7 @@ int rtw89_core_init(struct rtw89_dev *rtwdev)
> > >         wiphy_work_init(&rtwdev->cancel_6ghz_probe_work, rtw89_cancel_6ghz_probe_work);
> > >         INIT_WORK(&rtwdev->load_firmware_work, rtw89_load_firmware_work);
> > >
> > > +       skb_queue_head_init(&rtwdev->tx_rpt.queue);
> >
> > not sure if it's worth to initialize tx_rpt.sn to zero?
> 
> That shouldn't be needed because rtwdev is zero initialized in
> rtw89_alloc_ieee80211_hw().  ieee80211_alloc_hw() fills the private
> driver part with zeroes.

Ah. I mentioned this in wrong place. I meant that we can initialize tx_rpt.sn
in rtw89_core_start() or do it right after downloading firmware in
__rtw89_fw_download() like ' fw_info->h2c_seq = 0;'.

> 
> > > @@ -5484,6 +5488,26 @@ rtw89_mac_c2h_tx_rpt(struct rtw89_dev *rtwdev, struct sk_buff *c2h, u32 len)
> > >         rtw89_debug(rtwdev, RTW89_DBG_TXRX,
> > >                     "C2H TX RPT: sn %d, tx_status %d, data_txcnt %d\n",
> > >                     sw_define, tx_status, data_txcnt);
> > > +
> > > +       spin_lock_irqsave(&tx_rpt->queue.lock, flags);
> > > +       skb_queue_walk_safe(&tx_rpt->queue, skb, tmp) {
> > > +               skb_data = RTW89_TX_SKB_CB(skb);
> > > +
> > > +               /* skip if sequence number doesn't match */
> > > +               if (sw_define != skb_data->tx_rpt_sn)
> > > +                       continue;
> > > +               /* skip if TX attempt has failed and retry limit has not been
> > > +                * reached yet
> > > +                */
> > > +               if (tx_status != RTW89_TX_DONE &&
> > > +                   data_txcnt != skb_data->tx_pkt_cnt_lmt)
> > > +                       continue;
> > > +
> > > +               __skb_unlink(skb, &tx_rpt->queue);
> > > +               rtw89_tx_rpt_tx_status(rtwdev, skb, tx_status);
> >
> > Would it be better to run rtw89_tx_rpt_tx_status() after this loop outside
> > spin_lock()?
> 
> I don't have a strong opinion here: PCIe side implements the release
> like
> 
> rtw89_pci_poll_rpq_dma()
>  spin_lock_bh(&rtwpci->trx_lock)
>  rtw89_pci_release_tx()
>  ...
>    rtw89_pci_release_txwd_skb()
>         skb_queue_walk_safe(&txwd->queue, skb, tmp) {
>                 skb_unlink(skb, &txwd->queue);
> 
>                 tx_data = RTW89_PCI_TX_SKB_CB(skb);
>                 dma_unmap_single(&rtwpci->pdev->dev, tx_data->dma, skb->len,
>                                  DMA_TO_DEVICE);
> 
>                 rtw89_pci_tx_status(rtwdev, tx_ring, skb, tx_status);
>         }
>  ...
>  spin_unlock_bh(&rtwpci->trx_lock)
> 
> 
> Apart from bh/irqsave part the iteration over skbs looks visually similar
> at the moment.
> 
> > > --- a/drivers/net/wireless/realtek/rtw89/usb.c
> > > +++ b/drivers/net/wireless/realtek/rtw89/usb.c
> > > @@ -216,6 +216,14 @@ static void rtw89_usb_write_port_complete(struct urb *urb)
> > >                 skb_pull(skb, txdesc_size);
> > >
> > >                 info = IEEE80211_SKB_CB(skb);
> > > +               if (rtw89_is_tx_rpt_skb(skb)) {
> > > +                       /* sequence number is passed to rtw89_mac_c2h_tx_rpt() via
> >
> > nit: The 'via' is over 80 characters a little bit. Move to next line.
> >
> > > +                        * driver data
> > > +                        */
> > > +                       skb_queue_tail(&rtwdev->tx_rpt.queue, skb);
> > > +                       continue;
> > > +               }
> > > +
> > >                 ieee80211_tx_info_clear_status(info);
> > >
> > >                 if (urb->status == 0) {
> >
> > Should we move this checking upward? Enqueue skb into tx_rpt_skb only if
> > urb->status == 0?
> 
> Yep, I agree we can do it.  Just need to report it immediately with
> RTW89_TX_MACID_DROP status.
> 
> As it currently stands, we should not receive notification from the
> firmware for such skb so it would remain inside tx_rpt.queue until HCI
> reset occurs.
> 
> However, I've not considered the case when the queue is full and we start
> queueing skbs with duplicate sequence numbers - the overall range we have
> is just 0xF, theoretically the situation is possible if the firmware
> fatally crashes and doesn't provide notifications in time.  IMHO the TX
> status will be the last thing we're going to be interested in when this
> happens.  In the end, HCI reset during SER activity will purge the queue.
> 
> But the possibility of having skbs with duplicate seq numbers is not good.
> Though I'm not sure if we can ever hit such a situation... Generally it
> indicates that the firmware doesn't respond or performs very badly, and
> we'd better reset it or something.  What do you think on this?

Maybe we can maintain a bitmap to detect duplicate seq numbers. Also, we can
search for a SKB according to tx_report.sn in constant time instead of doing
sequential search in a list. Then we can detect and remove duplicate seq
numbers at TX side, if a tx_report.sn has been existing in the bitmap.

> 
> >
> > > @@ -372,6 +380,7 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
> > >  {
> > >         struct rtw89_tx_desc_info *desc_info = &tx_req->desc_info;
> > >         struct rtw89_usb *rtwusb = rtw89_usb_priv(rtwdev);
> > > +       struct rtw89_tx_skb_data *skb_data;
> > >         struct sk_buff *skb = tx_req->skb;
> > >         struct rtw89_txwd_body *txdesc;
> > >         u32 txdesc_size;
> > > @@ -398,6 +407,9 @@ static int rtw89_usb_ops_tx_write(struct rtw89_dev *rtwdev,
> > >
> > >         le32p_replace_bits(&txdesc->dword0, 1, RTW89_TXWD_BODY0_STF_MODE);
> > >
> > > +       skb_data = RTW89_TX_SKB_CB(skb);
> > > +       skb_data->tx_rpt_sn = tx_req->desc_info.sn;
> >
> > Shouldn't set skb_data->tx_pkt_cnt_lmt?
> >
> > skb_data->tx_pkt_cnt_lmt = tx_req->desc_info.tx_cnt_lmt;
> >
> > Also, should we check desc_info.{report, tx_cnt_lmt_en} individually before
> > setting?
> 
> Right, this all makes sense.  Will fix it, thanks!
> 
> >
> >
> > > +
> > >         skb_queue_tail(&rtwusb->tx_queue[desc_info->ch_dma], skb);
> > >
> > >         return 0;
> > > @@ -678,7 +690,7 @@ static void rtw89_usb_deinit_tx(struct rtw89_dev *rtwdev)
> > >
> > >  static void rtw89_usb_ops_reset(struct rtw89_dev *rtwdev)
> > >  {
> > > -       /* TODO: anything to do here? */
> > > +       rtw89_tx_rpt_queue_purge(rtwdev);
> >
> > Have you consider the SKB that has been rtw89_usb_write_port() but
> > has not yet rtw89_usb_write_port_complete()?
> >
> > Since we call rtw89_mac_pwr_off() before rtw89_hci_reset() in
> > rtw89_core_stop(), it should be not more C2H at rtw89_hci_reset().
> > It seems to be safe, right?
> 
> Well, rtw89_usb_write_port_complete() is asynchronous URB callback managed
> by USB subsystem and it's mainly independent of rtw89.  We're guaranteed
> all pending URB completions will complete when the device is being
> disconnected and rtw89_usb_disconnect() method is called.  That's all, I
> think.
> 
> In other call sites like SER we risk URB completion be called after
> purging the queue, the only consequence will be the obsolete skb still
> added to the queue.
> 
> We can implement an anchor for TX URBs and explicitly wait with
> usb_kill_anchored_urbs() in ->reset() until all pending URB completions
> are done, and then purge the queue.
> 
> If nothing else, I'll add it in the next respin of the series.

Good to me. Additionally, I'd like to add a comment right after hci->reset
to explicitly point out we should complete all tx_wait, such as

static inline void rtw89_hci_reset(struct rtw89_dev *rtwdev)
{
	rtwdev->hci.ops->reset(rtwdev);
    /* hci.ops->reset must complete tx_wait of all pending TX SKB */
	rtw89_tx_wait_list_clear(rtwdev);
}

On Mon, 27. Oct 01:14, Ping-Ke Shih wrote:
> Fedor Pchelkin <pchelkin@ispras.ru> wrote:
> > On Wed, 22. Oct 07:16, Ping-Ke Shih wrote:
> > > Fedor Pchelkin <pchelkin@ispras.ru> wrote:
> > > > @@ -5849,6 +5852,7 @@ int rtw89_core_init(struct rtw89_dev *rtwdev)
> > > >         wiphy_work_init(&rtwdev->cancel_6ghz_probe_work, rtw89_cancel_6ghz_probe_work);
> > > >         INIT_WORK(&rtwdev->load_firmware_work, rtw89_load_firmware_work);
> > > >
> > > > +       skb_queue_head_init(&rtwdev->tx_rpt.queue);
> > >
> > > not sure if it's worth to initialize tx_rpt.sn to zero?
> > 
> > That shouldn't be needed because rtwdev is zero initialized in
> > rtw89_alloc_ieee80211_hw().  ieee80211_alloc_hw() fills the private
> > driver part with zeroes.
> 
> Ah. I mentioned this in wrong place. I meant that we can initialize tx_rpt.sn
> in rtw89_core_start() or do it right after downloading firmware in
> __rtw89_fw_download() like ' fw_info->h2c_seq = 0;'.

To my mind, it's not worth adding extra code to initialize tx_rpt.sn to
zero at some point as it's just a sequential number in [0x0, 0xF] range,
which is replayed to firmware and used to synchronize with it.  Actually
we can start counting from 0x1 or 0xA, it doesn't really matter to care
about counter initialization.