From: Wang Yaxin <wang.yaxin@zte.com.cn>
Add delay max, timestamp of delay max and max-delay sorting to delaytop,
which can help quickly detect potential abnormal delays from a global
perspective and locate the top-N processes with the highest latency
spikes.
Problem
=======
Delay accounting can track the average delay of processes to show
system workload. delaytop provides a global view of average delays
across all tasks, which differs from getdelays's single-task
perspective. However, when a process experiences a significant delay,
maybe a delay spike, which adversely affects performance, delaytop can
only display the average delay over a period of time. Yet, average
delay is unhelpful for diagnosing delay peak. A brief but severe
latency spike is averaged out and hidden by the average-only metric,
and the top-N list sorted by average cannot identify the processes that
actually contributed most to the delay peak, nor provide any time
context to correlate the spike with other system events.
Solution
========
Extend delaytop with the delay max, timestamp and sorting ability:
1. delay_max fields track the maximum delay value for each delay type
(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
per task, recording every delay peak since task start.
2. Record the timestamp when each maximum delay occurred, which is the
same capability as getdelays but provided from a global perspective.
3. The -t/--type option displays only the specified delay type with
avg/max/timestamp side by side, and sorts tasks by the maximum delay
in descending order. This highlights the top-N processes with the
highest delay spikes, so abnormal delays can be located from a
global perspective and root causes pinpointed by correlating the
timestamps with logs, traces or other metrics.
Use case
========
bash# ./delaytop -t cpu
System Pressure Information: (avg10/avg60/avg300/total)
CPU some: 0.4%/ 0.2%/ 0.1%/ 220(ms)
CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms)
IO full: 0.0%/ 0.0%/ 0.0%/ 12(ms)
IO some: 0.0%/ 0.0%/ 0.0%/ 13(ms)
IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
[q]quit
Top 20 processes (sorted by cpu MAX delay, largest first):
PID TGID COMMAND AVG(ms) MAX(ms) MAX_TIMESTAMP
----------------------------------------------------------------
9 9 kworker/0:0-eve 0.59 16.87 2026-05-27T13:32:39
30 30 kworker/2:0H-kb 2.87 11.36 2026-05-27T13:32:36
27 27 migration/2 1.05 9.51 2026-05-27T13:32:37
50 50 kworker/2:1-eve 0.50 9.13 2026-05-27T13:32:37
15 15 rcu_preempt 0.11 8.98 2026-05-27T13:32:37
1 1 init 0.17 7.12 2026-05-27T13:32:38
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
v4->v5:
https://sashiko.dev/#/patchset/20260908225056266qjyBORsOReFvHLEQ3NKtR@zte.com.cn
[patch 0/3]
1. update cover letter to state problem, solution and use cases for
delay max, timestamp and sorting series
[patch 1/3]
2. fix -t/--type output column alignment by matching the row AVG/MAX
columns to the header spacing.
[patch 2/3]
3. fix Y2038 timestamp truncation check on 32-bit to handle pre-1970
timestamps by comparing the max time_t value in the signed domain.
[patch 3/3]
4. fix delaytop ReST literal block indentation in delay-accounting.rst
to keep all example lines consistently tab-indented.
v3->v4:
[patch 3/3]
1. Fix ReST syntax errors (indentation, list formatting) that triggered
warnings when running 'make htmldocs'. No content changes are made.
v2->v3:
[patch 1/3]
1. fix get_field_by_name() MODE_TYPE filter breaking --sort for non-type fields
2. zero-init stats struct and cap memcpy size in netlink response parsing
3. fix compare_tasks() count field read as unsigned long long for 32-bit safety
[patch 2/3]
1. fix 32-bit time_t overflow check to use max representable value instead of (time_t)-1
2. fix gmtime() pointer aliasing by assigning timespec64 tv_sec to local time_t
[patch 3/3]
1. fixed Y2038 overflow false negative in format_kernel_timespec()
2. switched to localtime_r() with stack `struct tm` for thread safety.
Wang Yaxin (3):
delaytop: add delay max for delaytop
delaytop: add timestamp of delay max
delaytop: sort by max delay to highlight top latency processes
Documentation/accounting/delay-accounting.rst | 87 ++++--
tools/accounting/delaytop.c | 289 +++++++++++++++---
tools/accounting/getdelays.c | 8 +
3 files changed, 324 insertions(+), 60 deletions(-)
--
2.25.1