drivers/net/wireless/ath/ath11k/hal.c | 4 ++++ 1 file changed, 4 insertions(+)
In a number of cases we see kernel panics on resume due
to ath11k kernel page fault, which happens under the
following circumstances:
1) First ath11k_hal_dump_srng_stats() call
Last interrupt received for each group:
ath11k_pci 0000:01:00.0: group_id 0 22511ms before
ath11k_pci 0000:01:00.0: group_id 1 14440788ms before
[..]
ath11k_pci 0000:01:00.0: failed to receive control response completion, polling..
ath11k_pci 0000:01:00.0: Service connect timeout
ath11k_pci 0000:01:00.0: failed to connect to HTT: -110
ath11k_pci 0000:01:00.0: failed to start core: -110
ath11k_pci 0000:01:00.0: firmware crashed: MHI_CB_EE_RDDM
ath11k_pci 0000:01:00.0: already resetting count 2
ath11k_pci 0000:01:00.0: failed to wait wlan mode request (mode 4): -110
ath11k_pci 0000:01:00.0: qmi failed to send wlan mode off: -110
ath11k_pci 0000:01:00.0: failed to reconfigure driver on crash recovery
[..]
2) At this point reconfiguration fails (we have 2 resets) and
ath11k_core_reconfigure_on_crash() calls ath11k_hal_srng_deinit()
which destroys srng lists. However, it does not reset per-list
->initialized flag.
3) Second ath11k_hal_dump_srng_stats() call sees stale ->initialized
flag and attempts to dump srng stats:
Last interrupt received for each group:
ath11k_pci 0000:01:00.0: group_id 0 66785ms before
ath11k_pci 0000:01:00.0: group_id 1 14485062ms before
ath11k_pci 0000:01:00.0: group_id 2 14485062ms before
ath11k_pci 0000:01:00.0: group_id 3 14485062ms before
ath11k_pci 0000:01:00.0: group_id 4 14780845ms before
ath11k_pci 0000:01:00.0: group_id 5 14780845ms before
ath11k_pci 0000:01:00.0: group_id 6 14485062ms before
ath11k_pci 0000:01:00.0: group_id 7 66814ms before
ath11k_pci 0000:01:00.0: group_id 8 68997ms before
ath11k_pci 0000:01:00.0: group_id 9 67588ms before
ath11k_pci 0000:01:00.0: group_id 10 69511ms before
BUG: unable to handle page fault for address: ffffa007404eb010
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 100000067 P4D 100000067 PUD 10022d067 PMD 100b01067 PTE 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
RIP: 0010:ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k]
Call Trace:
<TASK>
? __die_body+0xae/0xb0
? page_fault_oops+0x381/0x3e0
? exc_page_fault+0x69/0xa0
? asm_exc_page_fault+0x22/0x30
? ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k (HASH:6cea 4)]
ath11k_qmi_driver_event_work+0xbd/0x1050 [ath11k (HASH:6cea 4)]
worker_thread+0x389/0x930
kthread+0x149/0x170
Clear per-list ->initialized flag in ath11k_hal_srng_deinit().
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
v3:
- updated commit message and subject line (Baochen Qiang)
drivers/net/wireless/ath/ath11k/hal.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/net/wireless/ath/ath11k/hal.c b/drivers/net/wireless/ath/ath11k/hal.c
index 8cb1505a5a0c..cab11a35f911 100644
--- a/drivers/net/wireless/ath/ath11k/hal.c
+++ b/drivers/net/wireless/ath/ath11k/hal.c
@@ -1346,6 +1346,10 @@ EXPORT_SYMBOL(ath11k_hal_srng_init);
void ath11k_hal_srng_deinit(struct ath11k_base *ab)
{
struct ath11k_hal *hal = &ab->hal;
+ int i;
+
+ for (i = 0; i < HAL_SRNG_RING_ID_MAX; i++)
+ ab->hal.srng_list[i].initialized = 0;
ath11k_hal_unregister_srng_key(ab);
ath11k_hal_free_cont_rdp(ab);
--
2.50.0.rc1.591.g9c95f17f64-goog
On Thu, 12 Jun 2025 17:45:06 +0900, Sergey Senozhatsky wrote: > In a number of cases we see kernel panics on resume due > to ath11k kernel page fault, which happens under the > following circumstances: > > 1) First ath11k_hal_dump_srng_stats() call > > Last interrupt received for each group: > ath11k_pci 0000:01:00.0: group_id 0 22511ms before > ath11k_pci 0000:01:00.0: group_id 1 14440788ms before > [..] > ath11k_pci 0000:01:00.0: failed to receive control response completion, polling.. > ath11k_pci 0000:01:00.0: Service connect timeout > ath11k_pci 0000:01:00.0: failed to connect to HTT: -110 > ath11k_pci 0000:01:00.0: failed to start core: -110 > ath11k_pci 0000:01:00.0: firmware crashed: MHI_CB_EE_RDDM > ath11k_pci 0000:01:00.0: already resetting count 2 > ath11k_pci 0000:01:00.0: failed to wait wlan mode request (mode 4): -110 > ath11k_pci 0000:01:00.0: qmi failed to send wlan mode off: -110 > ath11k_pci 0000:01:00.0: failed to reconfigure driver on crash recovery > [..] > > [...] Applied, thanks! [1/1] wifi: ath11k: clear initialized flag for deinit-ed srng lists commit: a5b46aa7cf5f05c213316a018e49a8e086efd98e Best regards, -- Jeff Johnson <jeff.johnson@oss.qualcomm.com>
On (25/06/12 17:45), Sergey Senozhatsky wrote: > In a number of cases we see kernel panics on resume due > to ath11k kernel page fault, which happens under the > following circumstances: > > 1) First ath11k_hal_dump_srng_stats() call > > Last interrupt received for each group: > ath11k_pci 0000:01:00.0: group_id 0 22511ms before > ath11k_pci 0000:01:00.0: group_id 1 14440788ms before > [..] > ath11k_pci 0000:01:00.0: failed to receive control response completion, polling.. > ath11k_pci 0000:01:00.0: Service connect timeout > ath11k_pci 0000:01:00.0: failed to connect to HTT: -110 > ath11k_pci 0000:01:00.0: failed to start core: -110 > ath11k_pci 0000:01:00.0: firmware crashed: MHI_CB_EE_RDDM > ath11k_pci 0000:01:00.0: already resetting count 2 > ath11k_pci 0000:01:00.0: failed to wait wlan mode request (mode 4): -110 > ath11k_pci 0000:01:00.0: qmi failed to send wlan mode off: -110 > ath11k_pci 0000:01:00.0: failed to reconfigure driver on crash recovery > [..] > > 2) At this point reconfiguration fails (we have 2 resets) and > ath11k_core_reconfigure_on_crash() calls ath11k_hal_srng_deinit() > which destroys srng lists. However, it does not reset per-list > ->initialized flag. > > 3) Second ath11k_hal_dump_srng_stats() call sees stale ->initialized > flag and attempts to dump srng stats: > > Last interrupt received for each group: > ath11k_pci 0000:01:00.0: group_id 0 66785ms before > ath11k_pci 0000:01:00.0: group_id 1 14485062ms before > ath11k_pci 0000:01:00.0: group_id 2 14485062ms before > ath11k_pci 0000:01:00.0: group_id 3 14485062ms before > ath11k_pci 0000:01:00.0: group_id 4 14780845ms before > ath11k_pci 0000:01:00.0: group_id 5 14780845ms before > ath11k_pci 0000:01:00.0: group_id 6 14485062ms before > ath11k_pci 0000:01:00.0: group_id 7 66814ms before > ath11k_pci 0000:01:00.0: group_id 8 68997ms before > ath11k_pci 0000:01:00.0: group_id 9 67588ms before > ath11k_pci 0000:01:00.0: group_id 10 69511ms before > BUG: unable to handle page fault for address: ffffa007404eb010 > #PF: supervisor read access in kernel mode > #PF: error_code(0x0000) - not-present page > PGD 100000067 P4D 100000067 PUD 10022d067 PMD 100b01067 PTE 0 > Oops: 0000 [#1] PREEMPT SMP NOPTI > RIP: 0010:ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k] > Call Trace: > <TASK> > ? __die_body+0xae/0xb0 > ? page_fault_oops+0x381/0x3e0 > ? exc_page_fault+0x69/0xa0 > ? asm_exc_page_fault+0x22/0x30 > ? ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k (HASH:6cea 4)] > ath11k_qmi_driver_event_work+0xbd/0x1050 [ath11k (HASH:6cea 4)] > worker_thread+0x389/0x930 > kthread+0x149/0x170 > > Clear per-list ->initialized flag in ath11k_hal_srng_deinit(). If you are with this, can you please pick it up?
On (25/06/19 11:53), Sergey Senozhatsky wrote: > > If you are with this, can you please pick it up? ^^^ fine
On 6/12/2025 4:45 PM, Sergey Senozhatsky wrote: > In a number of cases we see kernel panics on resume due > to ath11k kernel page fault, which happens under the > following circumstances: > > 1) First ath11k_hal_dump_srng_stats() call > > Last interrupt received for each group: > ath11k_pci 0000:01:00.0: group_id 0 22511ms before > ath11k_pci 0000:01:00.0: group_id 1 14440788ms before > [..] > ath11k_pci 0000:01:00.0: failed to receive control response completion, polling.. > ath11k_pci 0000:01:00.0: Service connect timeout > ath11k_pci 0000:01:00.0: failed to connect to HTT: -110 > ath11k_pci 0000:01:00.0: failed to start core: -110 > ath11k_pci 0000:01:00.0: firmware crashed: MHI_CB_EE_RDDM > ath11k_pci 0000:01:00.0: already resetting count 2 > ath11k_pci 0000:01:00.0: failed to wait wlan mode request (mode 4): -110 > ath11k_pci 0000:01:00.0: qmi failed to send wlan mode off: -110 > ath11k_pci 0000:01:00.0: failed to reconfigure driver on crash recovery > [..] > > 2) At this point reconfiguration fails (we have 2 resets) and > ath11k_core_reconfigure_on_crash() calls ath11k_hal_srng_deinit() > which destroys srng lists. However, it does not reset per-list > ->initialized flag. > > 3) Second ath11k_hal_dump_srng_stats() call sees stale ->initialized > flag and attempts to dump srng stats: > > Last interrupt received for each group: > ath11k_pci 0000:01:00.0: group_id 0 66785ms before > ath11k_pci 0000:01:00.0: group_id 1 14485062ms before > ath11k_pci 0000:01:00.0: group_id 2 14485062ms before > ath11k_pci 0000:01:00.0: group_id 3 14485062ms before > ath11k_pci 0000:01:00.0: group_id 4 14780845ms before > ath11k_pci 0000:01:00.0: group_id 5 14780845ms before > ath11k_pci 0000:01:00.0: group_id 6 14485062ms before > ath11k_pci 0000:01:00.0: group_id 7 66814ms before > ath11k_pci 0000:01:00.0: group_id 8 68997ms before > ath11k_pci 0000:01:00.0: group_id 9 67588ms before > ath11k_pci 0000:01:00.0: group_id 10 69511ms before > BUG: unable to handle page fault for address: ffffa007404eb010 > #PF: supervisor read access in kernel mode > #PF: error_code(0x0000) - not-present page > PGD 100000067 P4D 100000067 PUD 10022d067 PMD 100b01067 PTE 0 > Oops: 0000 [#1] PREEMPT SMP NOPTI > RIP: 0010:ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k] > Call Trace: > <TASK> > ? __die_body+0xae/0xb0 > ? page_fault_oops+0x381/0x3e0 > ? exc_page_fault+0x69/0xa0 > ? asm_exc_page_fault+0x22/0x30 > ? ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k (HASH:6cea 4)] > ath11k_qmi_driver_event_work+0xbd/0x1050 [ath11k (HASH:6cea 4)] > worker_thread+0x389/0x930 > kthread+0x149/0x170 > > Clear per-list ->initialized flag in ath11k_hal_srng_deinit(). > > Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org> > --- > > v3: > - updated commit message and subject line (Baochen Qiang) > > drivers/net/wireless/ath/ath11k/hal.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/drivers/net/wireless/ath/ath11k/hal.c b/drivers/net/wireless/ath/ath11k/hal.c > index 8cb1505a5a0c..cab11a35f911 100644 > --- a/drivers/net/wireless/ath/ath11k/hal.c > +++ b/drivers/net/wireless/ath/ath11k/hal.c > @@ -1346,6 +1346,10 @@ EXPORT_SYMBOL(ath11k_hal_srng_init); > void ath11k_hal_srng_deinit(struct ath11k_base *ab) > { > struct ath11k_hal *hal = &ab->hal; > + int i; > + > + for (i = 0; i < HAL_SRNG_RING_ID_MAX; i++) > + ab->hal.srng_list[i].initialized = 0; > > ath11k_hal_unregister_srng_key(ab); > ath11k_hal_free_cont_rdp(ab); Reviewed-by: Baochen Qiang <quic_bqiang@quicinc.com>
On 6/12/2025 3:49 AM, Baochen Qiang wrote: > > > On 6/12/2025 4:45 PM, Sergey Senozhatsky wrote: >> In a number of cases we see kernel panics on resume due >> to ath11k kernel page fault, which happens under the >> following circumstances: >> >> 1) First ath11k_hal_dump_srng_stats() call >> >> Last interrupt received for each group: >> ath11k_pci 0000:01:00.0: group_id 0 22511ms before >> ath11k_pci 0000:01:00.0: group_id 1 14440788ms before >> [..] >> ath11k_pci 0000:01:00.0: failed to receive control response completion, polling.. >> ath11k_pci 0000:01:00.0: Service connect timeout >> ath11k_pci 0000:01:00.0: failed to connect to HTT: -110 >> ath11k_pci 0000:01:00.0: failed to start core: -110 >> ath11k_pci 0000:01:00.0: firmware crashed: MHI_CB_EE_RDDM >> ath11k_pci 0000:01:00.0: already resetting count 2 >> ath11k_pci 0000:01:00.0: failed to wait wlan mode request (mode 4): -110 >> ath11k_pci 0000:01:00.0: qmi failed to send wlan mode off: -110 >> ath11k_pci 0000:01:00.0: failed to reconfigure driver on crash recovery >> [..] >> >> 2) At this point reconfiguration fails (we have 2 resets) and >> ath11k_core_reconfigure_on_crash() calls ath11k_hal_srng_deinit() >> which destroys srng lists. However, it does not reset per-list >> ->initialized flag. >> >> 3) Second ath11k_hal_dump_srng_stats() call sees stale ->initialized >> flag and attempts to dump srng stats: >> >> Last interrupt received for each group: >> ath11k_pci 0000:01:00.0: group_id 0 66785ms before >> ath11k_pci 0000:01:00.0: group_id 1 14485062ms before >> ath11k_pci 0000:01:00.0: group_id 2 14485062ms before >> ath11k_pci 0000:01:00.0: group_id 3 14485062ms before >> ath11k_pci 0000:01:00.0: group_id 4 14780845ms before >> ath11k_pci 0000:01:00.0: group_id 5 14780845ms before >> ath11k_pci 0000:01:00.0: group_id 6 14485062ms before >> ath11k_pci 0000:01:00.0: group_id 7 66814ms before >> ath11k_pci 0000:01:00.0: group_id 8 68997ms before >> ath11k_pci 0000:01:00.0: group_id 9 67588ms before >> ath11k_pci 0000:01:00.0: group_id 10 69511ms before >> BUG: unable to handle page fault for address: ffffa007404eb010 >> #PF: supervisor read access in kernel mode >> #PF: error_code(0x0000) - not-present page >> PGD 100000067 P4D 100000067 PUD 10022d067 PMD 100b01067 PTE 0 >> Oops: 0000 [#1] PREEMPT SMP NOPTI >> RIP: 0010:ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k] >> Call Trace: >> <TASK> >> ? __die_body+0xae/0xb0 >> ? page_fault_oops+0x381/0x3e0 >> ? exc_page_fault+0x69/0xa0 >> ? asm_exc_page_fault+0x22/0x30 >> ? ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k (HASH:6cea 4)] >> ath11k_qmi_driver_event_work+0xbd/0x1050 [ath11k (HASH:6cea 4)] >> worker_thread+0x389/0x930 >> kthread+0x149/0x170 >> >> Clear per-list ->initialized flag in ath11k_hal_srng_deinit(). >> >> Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org> >> --- >> >> v3: >> - updated commit message and subject line (Baochen Qiang) >> >> drivers/net/wireless/ath/ath11k/hal.c | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/drivers/net/wireless/ath/ath11k/hal.c b/drivers/net/wireless/ath/ath11k/hal.c >> index 8cb1505a5a0c..cab11a35f911 100644 >> --- a/drivers/net/wireless/ath/ath11k/hal.c >> +++ b/drivers/net/wireless/ath/ath11k/hal.c >> @@ -1346,6 +1346,10 @@ EXPORT_SYMBOL(ath11k_hal_srng_init); >> void ath11k_hal_srng_deinit(struct ath11k_base *ab) >> { >> struct ath11k_hal *hal = &ab->hal; >> + int i; >> + >> + for (i = 0; i < HAL_SRNG_RING_ID_MAX; i++) >> + ab->hal.srng_list[i].initialized = 0; >> >> ath11k_hal_unregister_srng_key(ab); >> ath11k_hal_free_cont_rdp(ab); > > Reviewed-by: Baochen Qiang <quic_bqiang@quicinc.com> Baochen, should there be a Fixes: tag? If so, can you provide it? /jeff
On 6/19/2025 1:27 PM, Jeff Johnson wrote: > On 6/12/2025 3:49 AM, Baochen Qiang wrote: >> >> >> On 6/12/2025 4:45 PM, Sergey Senozhatsky wrote: >>> In a number of cases we see kernel panics on resume due >>> to ath11k kernel page fault, which happens under the >>> following circumstances: >>> >>> 1) First ath11k_hal_dump_srng_stats() call >>> >>> Last interrupt received for each group: >>> ath11k_pci 0000:01:00.0: group_id 0 22511ms before >>> ath11k_pci 0000:01:00.0: group_id 1 14440788ms before >>> [..] >>> ath11k_pci 0000:01:00.0: failed to receive control response completion, polling.. >>> ath11k_pci 0000:01:00.0: Service connect timeout >>> ath11k_pci 0000:01:00.0: failed to connect to HTT: -110 >>> ath11k_pci 0000:01:00.0: failed to start core: -110 >>> ath11k_pci 0000:01:00.0: firmware crashed: MHI_CB_EE_RDDM >>> ath11k_pci 0000:01:00.0: already resetting count 2 >>> ath11k_pci 0000:01:00.0: failed to wait wlan mode request (mode 4): -110 >>> ath11k_pci 0000:01:00.0: qmi failed to send wlan mode off: -110 >>> ath11k_pci 0000:01:00.0: failed to reconfigure driver on crash recovery >>> [..] >>> >>> 2) At this point reconfiguration fails (we have 2 resets) and >>> ath11k_core_reconfigure_on_crash() calls ath11k_hal_srng_deinit() >>> which destroys srng lists. However, it does not reset per-list >>> ->initialized flag. >>> >>> 3) Second ath11k_hal_dump_srng_stats() call sees stale ->initialized >>> flag and attempts to dump srng stats: >>> >>> Last interrupt received for each group: >>> ath11k_pci 0000:01:00.0: group_id 0 66785ms before >>> ath11k_pci 0000:01:00.0: group_id 1 14485062ms before >>> ath11k_pci 0000:01:00.0: group_id 2 14485062ms before >>> ath11k_pci 0000:01:00.0: group_id 3 14485062ms before >>> ath11k_pci 0000:01:00.0: group_id 4 14780845ms before >>> ath11k_pci 0000:01:00.0: group_id 5 14780845ms before >>> ath11k_pci 0000:01:00.0: group_id 6 14485062ms before >>> ath11k_pci 0000:01:00.0: group_id 7 66814ms before >>> ath11k_pci 0000:01:00.0: group_id 8 68997ms before >>> ath11k_pci 0000:01:00.0: group_id 9 67588ms before >>> ath11k_pci 0000:01:00.0: group_id 10 69511ms before >>> BUG: unable to handle page fault for address: ffffa007404eb010 >>> #PF: supervisor read access in kernel mode >>> #PF: error_code(0x0000) - not-present page >>> PGD 100000067 P4D 100000067 PUD 10022d067 PMD 100b01067 PTE 0 >>> Oops: 0000 [#1] PREEMPT SMP NOPTI >>> RIP: 0010:ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k] >>> Call Trace: >>> <TASK> >>> ? __die_body+0xae/0xb0 >>> ? page_fault_oops+0x381/0x3e0 >>> ? exc_page_fault+0x69/0xa0 >>> ? asm_exc_page_fault+0x22/0x30 >>> ? ath11k_hal_dump_srng_stats+0x2b4/0x3b0 [ath11k (HASH:6cea 4)] >>> ath11k_qmi_driver_event_work+0xbd/0x1050 [ath11k (HASH:6cea 4)] >>> worker_thread+0x389/0x930 >>> kthread+0x149/0x170 >>> >>> Clear per-list ->initialized flag in ath11k_hal_srng_deinit(). >>> >>> Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org> >>> --- >>> >>> v3: >>> - updated commit message and subject line (Baochen Qiang) >>> >>> drivers/net/wireless/ath/ath11k/hal.c | 4 ++++ >>> 1 file changed, 4 insertions(+) >>> >>> diff --git a/drivers/net/wireless/ath/ath11k/hal.c b/drivers/net/wireless/ath/ath11k/hal.c >>> index 8cb1505a5a0c..cab11a35f911 100644 >>> --- a/drivers/net/wireless/ath/ath11k/hal.c >>> +++ b/drivers/net/wireless/ath/ath11k/hal.c >>> @@ -1346,6 +1346,10 @@ EXPORT_SYMBOL(ath11k_hal_srng_init); >>> void ath11k_hal_srng_deinit(struct ath11k_base *ab) >>> { >>> struct ath11k_hal *hal = &ab->hal; >>> + int i; >>> + >>> + for (i = 0; i < HAL_SRNG_RING_ID_MAX; i++) >>> + ab->hal.srng_list[i].initialized = 0; >>> >>> ath11k_hal_unregister_srng_key(ab); >>> ath11k_hal_free_cont_rdp(ab); >> >> Reviewed-by: Baochen Qiang <quic_bqiang@quicinc.com> > > Baochen, should there be a Fixes: tag? If so, can you provide it? Yeah: Fixes: 5118935b1bc2 ("ath11k: dump SRNG stats during FW assert") > > /jeff
© 2016 - 2025 Red Hat, Inc.