[PATCH v2 0/3] locking/csd-lock: Report how long a CSD stall lasted

Breno Leitao posted 3 patches 1 month, 2 weeks ago
kernel/smp.c        |  77 +++++++++++++++--------
lib/Kconfig.debug   |  12 ++++
lib/Makefile        |   1 +
lib/test_csd_lock.c | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 237 insertions(+), 26 deletions(-)
[PATCH v2 0/3] locking/csd-lock: Report how long a CSD stall lasted
Posted by Breno Leitao 1 month, 2 weeks ago
The CSD lock debug code says when a CPU stops answering an IPI and when it
finally answers, but the recovery line leaves out how long the stall
lasted, even though csd_lock_wait_toolong() has the timestamp the wait
started from.  On Meta's fleet that line fired 211K times in the last 24
hours, none of them with a duration.

Patch 1 packs the state that csd_lock_wait_toolong() carries across the
polling loop into a struct, as suggested on v1, which gives the timestamps
names and leaves room for another one.  Patch 2 prints how long the lock
was stuck and, when the IPI was re-sent, how long after the last re-send
the target released it.

I've created a simple selftest to test both patches, and I am including
it here, given it might be helpful for whoever is hacking this code.

---
Changes in v2:
- Add patch 1, packing csd_lock_wait_toolong() state into a struct, as
  suggested by Dmitry Ilvokhin.
- Add patch 3, a module that stalls a CPU so the reporting can be exercised.
- Link to v1: https://patch.msgid.link/20260805-csd-stall-duration-v1-1-71134fafe150@debian.org

---
Breno Leitao (3):
      locking/csd-lock: Pack csd_lock_wait_toolong() state into a struct
      locking/csd-lock: Report how long a stuck CSD lock took to recover
      lib/test_csd_lock: Add a module to stall a CPU on a CSD lock

 kernel/smp.c        |  77 +++++++++++++++--------
 lib/Kconfig.debug   |  12 ++++
 lib/Makefile        |   1 +
 lib/test_csd_lock.c | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 237 insertions(+), 26 deletions(-)
---
base-commit: f9a2394a23482bfd330911e9c8295b71724feacd
change-id: 20260805-csd-stall-duration-3385343d7a08

Best regards,
--  
Breno Leitao <leitao@debian.org>
Re: [PATCH v2 0/3] locking/csd-lock: Report how long a CSD stall lasted
Posted by Paul E. McKenney 1 month, 2 weeks ago
On Mon, Aug 10, 2026 at 04:29:23AM -0700, Breno Leitao wrote:
> The CSD lock debug code says when a CPU stops answering an IPI and when it
> finally answers, but the recovery line leaves out how long the stall
> lasted, even though csd_lock_wait_toolong() has the timestamp the wait
> started from.  On Meta's fleet that line fired 211K times in the last 24
> hours, none of them with a duration.
> 
> Patch 1 packs the state that csd_lock_wait_toolong() carries across the
> polling loop into a struct, as suggested on v1, which gives the timestamps
> names and leaves room for another one.  Patch 2 prints how long the lock
> was stuck and, when the IPI was re-sent, how long after the last re-send
> the target released it.
> 
> I've created a simple selftest to test both patches, and I am including
> it here, given it might be helpful for whoever is hacking this code.

Queued for testing.  If you would like to send this up some other path:

Acked-by: Paul E. McKenney <paulmck@kernel.org>

> ---
> Changes in v2:
> - Add patch 1, packing csd_lock_wait_toolong() state into a struct, as
>   suggested by Dmitry Ilvokhin.
> - Add patch 3, a module that stalls a CPU so the reporting can be exercised.
> - Link to v1: https://patch.msgid.link/20260805-csd-stall-duration-v1-1-71134fafe150@debian.org
> 
> ---
> Breno Leitao (3):
>       locking/csd-lock: Pack csd_lock_wait_toolong() state into a struct
>       locking/csd-lock: Report how long a stuck CSD lock took to recover
>       lib/test_csd_lock: Add a module to stall a CPU on a CSD lock
> 
>  kernel/smp.c        |  77 +++++++++++++++--------
>  lib/Kconfig.debug   |  12 ++++
>  lib/Makefile        |   1 +
>  lib/test_csd_lock.c | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 237 insertions(+), 26 deletions(-)
> ---
> base-commit: f9a2394a23482bfd330911e9c8295b71724feacd
> change-id: 20260805-csd-stall-duration-3385343d7a08
> 
> Best regards,
> --  
> Breno Leitao <leitao@debian.org>
>