All three target net. v1 split them across net/net-next; with the ring
size now shown to be load bearing, they belong together.

Answers to the v1 review:

- "fill_rx_queue() overwrites the DONE bit written by hw" - no. I
  implemented that check anyway (bail out in fill_rx_queue(), clear the
  bit in rx_process()) and it never fired once, while the ring was
  demonstrably stalled: the descriptor at q->head never had
  QDMA_DESC_DONE_MASK set, so there was nothing to catch. Worse,
  clearing desc->ctrl outright also wipes QDMA_DESC_LEN_MASK, which
  fill_rx_queue() writes as the buffer size and hw reads back - RX then
  delivers poisoned pages. That version is dropped.

- "have you tried to just increase the queue size" - yes, and that is
  the fix. Ring 4 at 16 stalls roughly every 35 s under repeated PPPoE
  dial-up and negotiation never completes; at 128 nothing triggered
  across 500 forced reconnects over 20 h. Patch 3.

- "is QDMA_DESC_DROP_MASK set when the issue occurs" - no, never, for
  the whole duration of the stall.

- default RX_DSCP_NUM raised 16 -> 32 per the vendor SDK, as asked.

- recovery pause measured at 986-1131 us over 13 events, against the
  50 ms read_poll_timeout() ceiling.

- style: RCT, verbose comments gone, and the !q->ndesc check dropped -
  the bit is only set from rx_process(), which cannot run on a ring
  without descriptors.

One cost worth flagging: the detector adds an uncached REG_RX_DMA_IDX
read to every rx_process() call that ends on a non-DONE descriptor. airoha_qdma_rx_napi_poll() loops while the last
pass reaped anything, so that is once or twice per NAPI poll, on every
ring, not just the one that can stall. Gating it on the previous poll
having reaped nothing would keep it off busy rings and only delays
detection by about one poll, since q->tail is frozen from the moment
the stall begins. I left it ungated as I have no profiling either way -
happy to add the gate if you prefer it.

A question for the airoha folks: during the stall REG_RX_DMA_IDX keeps advancing while
REG_RX_CPU_IDX stays put and the consumer never sees another DONE. Is
there a documented condition under which hw stops writing completions
back to a ring, or a constraint on RX_CPU_IDX that the driver is
violating by leaving one descriptor unposted?

The rx_stall_recover ethtool counter from v1 is dropped here - new ABI
does not belong in a fix - and will follow for net-next.

Tested on Nokia XG-040G-MF (AN7583) on a live PPPoE line, in both
configurations: as sent, and with stock ring sizes so the recovery path
actually executes.

Vitaliy Sochnev (3):
  net: airoha: handle RX_NO_CPU_DSCP interrupt, not just RX_DONE
  net: airoha: recover RX ring after hw completion stall
  net: airoha: grow the small RX rings

 drivers/net/ethernet/airoha/airoha_eth.c  | 114 ++++++++++++++++++++--
 drivers/net/ethernet/airoha/airoha_eth.h  |  11 ++-
 drivers/net/ethernet/airoha/airoha_regs.h |   2 +
 3 files changed, 120 insertions(+), 7 deletions(-)

-- 
2.55.0

v3: dropped the RX ring stall recovery (v2 2/3) as asked [1]. At 128 the
stall does not occur - 500 forced PPPoE reconnects over 20 h with the
detector compiled in and armed, zero triggers - and it did not fix the
bug on its own anyway. I will resend it if the stall turns up at the
larger ring.

2/2 keeps its code and its Acked-by; the commit message changed. It now
cites the register capture taken with no recovery in the tree instead of
numbers from builds carrying it, states that 1/2 does not cover this
failure, and quantifies what the bigger rings cost in memory.

v2 [2] answered the v1 review: DONE-bit overwrite hypothesis disproven,
QDMA_DESC_DROP_MASK never set, RX_DSCP_NUM default raised to the vendor
SDK's 32.

Still open for the airoha folks: in 2/2 hw set DONE on descriptor 15
while 0-14 were untouched and the driver's consumer sat at 0. Is
out-of-order completion within an RX ring expected, or is the driver
violating a constraint on RX_CPU_IDX by leaving one descriptor unposted?
Growing the ring avoids the symptom; the rule behind it is still unknown.

Tested on Nokia XG-040G-MF (AN7583) on a live PPPoE line. These two
patches without the recovery are what ran longest here: 508 forced
reconnects over 20 h 32 min, zero rx_dropped/rx_errors across 39158
samples.

[1] https://lore.kernel.org/netdev/apaA5jDYH71F0JaS@lore-desk/
[2] https://lore.kernel.org/netdev/20260831234701.206021-1-sochnev.v.74@gmail.com/

Vitaliy Sochnev (2):
  net: airoha: handle RX_NO_CPU_DSCP interrupt, not just RX_DONE
  net: airoha: grow the small RX rings

 drivers/net/ethernet/airoha/airoha_eth.c  | 16 +++++++++++-----
 drivers/net/ethernet/airoha/airoha_eth.h  |  3 ++-
 drivers/net/ethernet/airoha/airoha_regs.h |  2 ++
 3 files changed, 15 insertions(+), 6 deletions(-)

-- 
2.55.0

e84b89f17a12 ("net: airoha: grow the small RX rings") is queued for
net-next as an RX loss fix. It also stops hw from writing descriptors past
the end of the ring into unrelated kernel memory, which its commit message
does not mention. Given that, it may be worth stable.

Nokia XG-040G-MF (AN7583), 512 MiB, 6.18.44, OpenWrt snapshot. Images
below differ only in RX_DSCP_NUM(); no other patches, no instrumentation,
verified against vmlinux. RX_DSCP_NUM() is shared, so other variants with
a small ring are presumably affected too - not tested here.

Reproducer: raw UDP frames with source port 67, so REG_FE_VIP_PATN(8)
forces them to ring 4, ~900 pps to the board's MAC. With ring 4 at 16 the
box panics within a minute.

Mechanism
---------

At 16 descriptors ring 4 occupies 512 bytes; dma_alloc_coherent() rounds
to a page. After the ring stopped advancing, the remaining 3584 bytes of
that page contained 560 non-zero words repeating with a 32-byte period -
sizeof(struct airoha_qdma_desc):

  +4  ctrl  0xC0000000   QDMA_DESC_DONE_MASK | QDMA_DESC_DROP_MASK, len 0
  +16 msg0  0x00008000
  +20 msg1  0x2A5E0000
  +24 msg2  0x007F000E
  +28 msg3  0x0000FFFF

msg1 equals the value in the last descriptor the driver did see, so these
come from the same engine. REG_RX_RING_SIZE(4) reads 0x00020010 - the size
field is programmed correctly as 16. Writes staying inside the page are
invisible; past it they hit whatever follows.

Panics
------

Three on the 16-descriptor build, all garbage pointers in subsystems
unrelated to networking:

  __queue_work+0xa4 <- dbs_irq_work (cpufreq), x21 = 2d9ce5fd003cae80
  sched_balance_rq+0x84 <- sched_balance_domains, addr 0040000034124819
  Kernel panic - not syncing: corrupted stack end detected inside scheduler

The third occurred with no synthetic load: ordinary DHCP traffic after a
network restart, 4 minutes in.

Ring size is the only variable
------------------------------

  ring   frames fed   page tail after run   panic
  16     27 000       560 words, signature  yes, 3x
  32     271 909      0 of 768              no
  128    269 845      0 of 1024             no

Controls: the same scan on idle 32-descriptor rings reads 0 of 768, so the
560 words are not pre-existing content; 273 292 frames of identical traffic
on a non-VIP source port (ring 0) caused no panic.

On the shipped configuration (ring 4 = 128, default 32) the board also
completed 1623 consecutive PPPoE dial-ups, 16 VIP frames each, and ran
9 h 52 min of DHCP with renewals every minute - 1178 samples, no lease
loss, no rx errors, no panic.

The boundary is between 16 and 32. The vendor SDK default is 32, which
looks like a hardware minimum rather than a tuning choice.

Question for airoha
-------------------

Is 32 the minimum RX ring size the QDMA accepts? If so the driver should
clamp or reject smaller values rather than depend on RX_DSCP_NUM() being
large enough.

Withdrawing the NO_CPU_DSCP patch
---------------------------------

Please drop

  [PATCH net v3 1/2] net: airoha: handle RX_NO_CPU_DSCP interrupt

acked by Lorenzo, not applied. Its rationale - ring drains to zero, the
dropped NO_CPU_DSCP interrupt leaves it dead - is contradicted by
measurement:

  - NO_CPU_DSCP never fires. Instrumented builds logged zero events across
    every run; REG_INT_ENABLE(bank0,1) reads 0x839F839F, so the bit is
    unmasked for ring 4.
  - page_pool_dev_alloc_frag() never failed and q->queued never reached
    zero, so the described path is unreachable.
  - A/B images with and without the patch failed identically.

airoha_irq_handler() does drop the interrupt, so handling it may still be
correct, but not for the reason I gave.

Correction
----------

In the v3 cover letter I asked whether out-of-order completion pointed at
a constraint on RX_CPU_IDX, having seen QDMA_DESC_DONE_MASK set in the
slot fill_rx_queue() leaves unposted. airoha_qdma_rx_process() never
clears ctrl, so that bit is the residue of the last consumed frame.
Disregard it.

Correction to the platform description, and a stronger dump.

I wrote that the images "differ only in RX_DSCP_NUM(); no other patches".
That was about the difference between my own images and is misleading as a
description of the driver. The tree is OpenWrt's, and it carries an
out-of-tree HW GRO patch that touches exactly the ring under test:

  AIROHA_RXQ_LRO_EN_MASK = GENMASK(7, 0)   -> rings 0-7, including ring 4

For an LRO ring that patch sets buf_size to 16 KiB instead of PAGE_SIZE/2
and clears RX_RING_SG_EN_MASK, which mainline always sets. Both are
plausibly relevant to a DMA overrun, so the result needed rechecking with
that removed.

To be precise about the base: it is 6.18.44 with the airoha RX path
backported from mainline, including 269389ba5398 ("Set REG_RX_CPU_IDX()
once in airoha_qdma_fill_rx_queue()") and bbfb1983944f ("Reserve RX
headroom to avoid skb reallocation"), both in net today. The GRO patch is
the only out-of-tree piece touching this path.

Rechecked with its LRO mask zeroed, which restores page order 0,
buf_size = PAGE_SIZE/2 and RX_RING_SG_EN - confirmed on the board by the
posted buffer length dropping from 0x3E80 to 0x680. Everything
reproduces:

  ring   LRO   panics                        overrun signature
  16     on    3 (one with no load at all)   present, 560 words
  16     off   2 (one with no load at all)   present, see below
  32     on    none                          0 of 768
  32     off   none, 268 018 frames          0 of 768
  128    on    none                          0 of 1024
  128    off   none, 273 921 frames          0 of 1024

The clean dump, ring 4 = 16, LRO off, taken while the ring was stalled.
Descriptor 16 does not exist in a 16-entry ring:

  desc 15 (last real)      desc 16 (past the end)
  +4  ctrl 0x80000156      +4  ctrl 0xC0000000   DONE|DROP, len 0
      DONE, len 342        +8  addr 0x00000000   no buffer posted
  +20 msg1 0x2A5E0000      +16 msg0 0x00008000
                           +20 msg1 0x2A5E0000
                           +24 msg2 0x007F000E
                           +28 msg3 0x0000FFFF

msg0-msg3 are bit-identical to the dump taken with LRO on, so it is the
same engine either way. addr = 0 explains DROP: hw ran past the posted
descriptors, found no buffer in the next slot - a slot that is not part of
the ring - and marked the completion dropped, but wrote the structure
anyway.

The panic in the ring 4 = 16, LRO off run landed in yet another place:

  nf_conntrack_hash_check_insert+0x480 [nf_conntrack]
  nf_conntrack_in / nf_hook_slow / __ip_local_out / udp_send_skb
  Comm: ntpd

Five panics so far, in four distinct places, none of them networking:
cpufreq's deferred work (__queue_work), the scheduler's load balancer
(sched_balance_rq), the scheduler's stack-end check (twice), and
conntrack's hash insert.

Nothing else in the original mail changes.

airoha_qdma_hw_init() unmasks the per-ring NO_CPU_DSCP interrupt, which
fires when an RX ring runs out of free CPU descriptors, but
airoha_irq_handler() only extracts the RX_DONE bits from the same status
word. The NO_CPU_DSCP bits are acknowledged and dropped.

Once a ring is drained to zero posted descriptors no further RX_DONE can
fire for it - nothing is left for hw to receive into - so NAPI is never
rescheduled, airoha_qdma_fill_rx_queue() is never called again, and the
ring stays dead until the interface is reconfigured.

This is reachable on RX ring 4, which airoha_fe_vip_setup() force-routes
~15 VIP-classified protocols onto (BOOTP, PPPoE Discovery, ISAKMP,
DHCPv6, SIP, LLDP, PPP LCP/IPCP/CHAP/PAP/IPv6CP, ...) while it sits on
the 16-descriptor RX_DSCP_NUM() default. A DHCP renewal or a PPPoE
negotiation burst drains it faster than the CPU reposts, after which all
of those protocols silently stop being received.

Treat NO_CPU_DSCP like RX_DONE for scheduling NAPI:
airoha_qdma_rx_process() already calls airoha_qdma_fill_rx_queue()
unconditionally at the end of every poll, so scheduling NAPI is enough
to make an emptied ring refill itself. Re-enable the bit alongside
RX_DONE in airoha_qdma_rx_napi_poll() so it is not left masked after
first use.

Whether NO_CPU_DSCP re-latches while the ring is still empty is not
documented. If it does, a ring that cannot be refilled (page_pool
returning NULL) keeps reasserting it. Masking it until a refill succeeds
is worse: a starved ring never fires RX_DONE either, so it would stay
dead after the memory pressure clears.

Fixes: f252493e1835 ("net: airoha: Enable multiple IRQ lines support in airoha_eth driver.")
Link: https://github.com/openwrt/openwrt/issues/24715
Acked-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Vitaliy Sochnev <sochnev.v.74@gmail.com>
---
 drivers/net/ethernet/airoha/airoha_eth.c  | 16 +++++++++++-----
 drivers/net/ethernet/airoha/airoha_regs.h |  2 ++
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c
index 64619e9a704d..c59201aded26 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.c
+++ b/drivers/net/ethernet/airoha/airoha_eth.c
@@ -784,13 +784,16 @@ static int airoha_qdma_rx_napi_poll(struct napi_struct *napi, int budget)
 		int i, qid = q - &qdma->q_rx[0];
 		int intr_reg = qid < RX_DONE_HIGH_OFFSET ? QDMA_INT_REG_IDX1
 							 : QDMA_INT_REG_IDX2;
+		u32 irq_id = qid % RX_DONE_HIGH_OFFSET;
+		u32 intr_mask = BIT(irq_id) |
+				BIT(irq_id + RX_NO_CPU_DSCP_LOW_OFFSET);
 
 		for (i = 0; i < ARRAY_SIZE(qdma->irq_banks); i++) {
 			if (!(BIT(qid) & RX_IRQ_BANK_PIN_MASK(i)))
 				continue;
 
 			airoha_qdma_irq_enable(&qdma->irq_banks[i], intr_reg,
-					       BIT(qid % RX_DONE_HIGH_OFFSET));
+					       intr_mask);
 		}
 	}
 
@@ -1468,16 +1471,19 @@ static irqreturn_t airoha_irq_handler(int irq, void *dev_instance)
 	if (!test_bit(DEV_STATE_INITIALIZED, &qdma->eth->state))
 		return IRQ_NONE;
 
-	rx_intr1 = intr[1] & RX_DONE_LOW_INT_MASK;
+	rx_intr1 = intr[1] & (RX_DONE_LOW_INT_MASK | RX_NO_CPU_DSCP_LOW_INT_MASK);
 	if (rx_intr1) {
 		airoha_qdma_irq_disable(irq_bank, QDMA_INT_REG_IDX1, rx_intr1);
-		rx_intr_mask |= rx_intr1;
+		rx_intr_mask |= (rx_intr1 & RX_DONE_LOW_INT_MASK) |
+				RX_NO_CPU_DSCP_INT_RX1_MASK(rx_intr1);
 	}
 
-	rx_intr2 = intr[2] & RX_DONE_HIGH_INT_MASK;
+	rx_intr2 = intr[2] & (RX_DONE_HIGH_INT_MASK | RX_NO_CPU_DSCP_HIGH_INT_MASK);
 	if (rx_intr2) {
 		airoha_qdma_irq_disable(irq_bank, QDMA_INT_REG_IDX2, rx_intr2);
-		rx_intr_mask |= (rx_intr2 << 16);
+		rx_intr_mask |= ((rx_intr2 & RX_DONE_HIGH_INT_MASK) <<
+				 RX_DONE_HIGH_OFFSET) |
+				(rx_intr2 & RX_NO_CPU_DSCP_HIGH_INT_MASK);
 	}
 
 	for (i = 0; rx_intr_mask && i < ARRAY_SIZE(qdma->q_rx); i++) {
diff --git a/drivers/net/ethernet/airoha/airoha_regs.h b/drivers/net/ethernet/airoha/airoha_regs.h
index 442b48c9b991..bde3d599d8b7 100644
--- a/drivers/net/ethernet/airoha/airoha_regs.h
+++ b/drivers/net/ethernet/airoha/airoha_regs.h
@@ -549,6 +549,8 @@
 #define INT_RX1_MASK(_n)							\
 	((((_n) << RX_NO_CPU_DSCP_LOW_OFFSET) & RX_NO_CPU_DSCP_LOW_INT_MASK) |	\
 	 (RX_DONE_LOW_INT_MASK & (_n)))
+#define RX_NO_CPU_DSCP_INT_RX1_MASK(_n)					\
+	(((_n) & RX_NO_CPU_DSCP_LOW_INT_MASK) >> RX_NO_CPU_DSCP_LOW_OFFSET)
 
 /* QDMA_CSR_INT_ENABLE3 */
 #define RX31_NO_CPU_DSCP_INT_MASK	BIT(31)
-- 
2.55.0

On Tue,  1 Sep 2026 19:32:53 +0100 Vitaliy Sochnev wrote:
> Whether NO_CPU_DSCP re-latches while the ring is still empty is not
> documented. If it does, a ring that cannot be refilled (page_pool
> returning NULL) keeps reasserting it. Masking it until a refill succeeds
> is worse: a starved ring never fires RX_DONE either, so it would stay
> dead after the memory pressure clears.

Since you're not sure you should probably add a timer to retry
later? Maybe the IRQ is not needed at all, you could just check 
on allocation failure if the right is completely drained or have 
a periodic task a'la fbnic_napi_depletion_check()

> Since you're not sure you should probably add a timer to retry
> later? Maybe the IRQ is not needed at all, you could just check 
> on allocation failure if the right is completely drained or have 
> a periodic task a'la fbnic_napi_depletion_check()

No timer either: the depletion path is not the one being hit. Please drop
this patch.

Instrumented builds logged zero NO_CPU_DSCP events across every run, with
REG_INT_ENABLE(bank0, 1) reading 0x839F839F, so the bit was unmasked for
ring 4. page_pool_dev_alloc_frag() never failed and q->queued never
reached zero, so the ring never enters the state this patch handles.
Images with and without it stalled identically.

The stall was RX rings smaller than 32 descriptors, fixed by 2/2 of this
series, e84b89f17a12 ("net: airoha: grow the small RX rings"). Details in
my report on the cover thread.

airoha_irq_handler() does discard the NO_CPU_DSCP bits, so handling them
may still be right, but I have no measurement showing it matters and will
not claim one.

Raise the RX_DSCP_NUM() fallback from 16 to 32, the vendor SDK default,
and give ring 4 the 128 descriptors already used for rings 2/11/15.

Ring 4 is the shared "force to CPU" ring that airoha_fe_vip_setup()
routes BOOTP, PPPoE Discovery, PPP LCP/IPCP/CHAP/PAP/IPv6CP, ISAKMP,
DHCPv6, SIP and LLDP onto, so it sees the same bursty non-hashed traffic
as the rings already on 128 while sitting on the smallest allowance.
Both CDMs forward to it: REG_CDM_FWD_CFG(1) and (2) each set
CDM_VIP_QSEL_MASK to 4.

The NO_CPU_DSCP fix does not cover this. With it in place, at 16
descriptors the ring still dies during PPPoE negotiation on a Nokia
XG-040G-MF: over 6455 devmem samples in 60 s, REG_RX_DMA_IDX advanced
0x2c -> 0x52 while REG_RX_CPU_IDX stayed at 15, descriptor 15 had DONE
set on a 60-byte frame, descriptors 0-14 were empty, and q->tail was 0.
Hw had completed a descriptor the strictly sequential consumer in
airoha_qdma_rx_process() cannot reach, and negotiation never completes.
At 16 DHCP still works while PPPoE does not: a dial-up bursts Discovery,
LCP, IPCP and CHAP where DHCP sends one packet every ~150 s. With this
series, 508 forced reconnects over 20 h saw no recurrence.

RX_DONE_INT_MASK covers all 32 rings on both QDMA instances, so this
costs 544 more descriptors per instance, 1088 in total: 2.1 MiB of RX
buffers at PAGE_SIZE/2 each, plus 34 KiB of coherent descriptor memory.

Ring 4 and the VIP classification feeding it are shared driver-wide, with
no DT or hardware property distinguishing one variant's ring 4 from
another's, so this is not scoped to a particular SoC.

Fixes: 23020f049327 ("net: airoha: Introduce ethernet support for EN7581 SoC")
Acked-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Vitaliy Sochnev <sochnev.v.74@gmail.com>
---
 drivers/net/ethernet/airoha/airoha_eth.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/airoha/airoha_eth.h b/drivers/net/ethernet/airoha/airoha_eth.h
index fa9a8edce22f..7fc37f159d2b 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.h
+++ b/drivers/net/ethernet/airoha/airoha_eth.h
@@ -41,9 +41,10 @@
 #define TX_DSCP_NUM			1024
 #define RX_DSCP_NUM(_n)			\
 	((_n) ==  2 ? 128 :		\
+	 (_n) ==  4 ? 128 :		\
 	 (_n) == 11 ? 128 :		\
 	 (_n) == 15 ? 128 :		\
-	 (_n) ==  0 ? 1024 : 16)
+	 (_n) ==  0 ? 1024 : 32)
 
 #define PSE_RSV_PAGES			128
 #define PSE_QUEUE_RSV_PAGES		64
-- 
2.55.0