[PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis

wang.yaxin@zte.com.cn posted 3 patches 1 week, 5 days ago
There is a newer version of this series
Documentation/accounting/delay-accounting.rst |  43 ++++
tools/accounting/delaytop.c                   | 232 +++++++++++++++---
2 files changed, 247 insertions(+), 28 deletions(-)
[PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis
Posted by wang.yaxin@zte.com.cn 1 week, 5 days ago
From: Wang Yaxin <wang.yaxin@zte.com.cn>

Previously delaytop only showed average delays. This patch adds:

1. delay_max fields to track the maximum delay value for each delay type
   (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
   per task.

2. The -t/--type option displays only the specified delay type with avg/max
   values side by side, allowing focused analysis:
     delaytop -t cpu    # Show only CPU delay with avg/max
     delaytop -t wpcopy # Show Copy-on-Write delay with avg/max

3. Wall-clock timestamp when each maximum delay occurred, displayed in the
   MAX_TIMESTAMP column when using -t/--type option. This enables:
   - Identifying the time when a process experienced an abnormal delay max
   - Correlating delay max across multiple processes at the same timestamp
   - Cross-referencing with logs, traces, or other metrics at that time

4. When using -t/--type option, tasks are sorted by maximum delay value in
   descending order (largest delay first), enabling quick identification of
   top N processes with highest delay spikes.

Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>

Wang Yaxin (3):
  delaytop: add delay max for delaytop
  delaytop: add timestamp of delay max
  delaytop: sort by max delay to highlight top latency processes

 Documentation/accounting/delay-accounting.rst |  43 ++++
 tools/accounting/delaytop.c                   | 232 +++++++++++++++---
 2 files changed, 247 insertions(+), 28 deletions(-)

-- 
2.25.1
Re: [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis
Posted by Andrew Morton 1 week, 4 days ago
On Wed, 27 May 2026 21:55:24 +0800 (CST) <wang.yaxin@zte.com.cn> wrote:

> From: Wang Yaxin <wang.yaxin@zte.com.cn>
> 
> Previously delaytop only showed average delays. This patch adds:
> 
> 1. delay_max fields to track the maximum delay value for each delay type
>    (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
>    per task.
> 
> 2. The -t/--type option displays only the specified delay type with avg/max
>    values side by side, allowing focused analysis:
>      delaytop -t cpu    # Show only CPU delay with avg/max
>      delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
> 
> 3. Wall-clock timestamp when each maximum delay occurred, displayed in the
>    MAX_TIMESTAMP column when using -t/--type option. This enables:
>    - Identifying the time when a process experienced an abnormal delay max
>    - Correlating delay max across multiple processes at the same timestamp
>    - Cross-referencing with logs, traces, or other metrics at that time
> 
> 4. When using -t/--type option, tasks are sorted by maximum delay value in
>    descending order (largest delay first), enabling quick identification of
>    top N processes with highest delay spikes.

Sounds useful.  Am I correct in assuming that you're a regular user of
delaytop and that these changes are based on your experience with it?

AI review claims to have found a number of things:
	https://sashiko.dev/#/patchset/20260527215524044fG7XSpgveHiaFhraq0yAi@zte.com.cn
Re: [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis
Posted by wang.yaxin@zte.com.cn 1 week, 4 days ago
>> From: Wang Yaxin <wang.yaxin@zte.com.cn>
>> 
>> Previously delaytop only showed average delays. This patch adds:
>> 
>> 1. delay_max fields to track the maximum delay value for each delay type
>>    (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
>>    per task.
>> 
>> 2. The -t/--type option displays only the specified delay type with avg/max
>>    values side by side, allowing focused analysis:
>>      delaytop -t cpu    # Show only CPU delay with avg/max
>>      delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
>> 
>> 3. Wall-clock timestamp when each maximum delay occurred, displayed in the
>>    MAX_TIMESTAMP column when using -t/--type option. This enables:
>>    - Identifying the time when a process experienced an abnormal delay max
>>    - Correlating delay max across multiple processes at the same timestamp
>>    - Cross-referencing with logs, traces, or other metrics at that time
>> 
>> 4. When using -t/--type option, tasks are sorted by maximum delay value in
>>    descending order (largest delay first), enabling quick identification of
>>    top N processes with highest delay spikes.
>
>Sounds useful.  Am I correct in assuming that you're a regular user of
>delaytop and that these changes are based on your experience with it?

This indeed comes from latency observation requirements in business
scenarios. Some of our tasks are highly latency-sensitive, so we
need to monitor the "delay max" of specific tasks and detect anomalies
in a timely manner. delaytop is very useful for observing task-level
latency.

Thanks for the review. I will submit the v2 patch to fix the potential
issues as suggested

Thanks
Yaxin