This patch series improves 'perf sched latency' by suppressing misleading
empty table output, extending pipe mode stream processing, introducing
dynamic unit auto-scaling for latency and runtime statistics, and adding
latency histogram visualisation alongside time-span filtering.

Patch 1 addresses an issue where 'perf sched latency' fell through and
returned success (0) when perf_session__has_traces() failed due to missing
tracepoint events in a perf.data file. This caused empty header tables and
zeroed summary statistics to be rendered. In addition,
thread__get_runtime() in map_switch_event() is guarded against potential
NULL pointer dereferences under memory allocation failures.

Patch 2 extends pipe mode stream support. Because event attributes in pipe
mode are received dynamically during event processing, session->evlist is
not populated prior to event processing.
To handle pipe input correctly:
    - Missing callbacks (.attr, .tracing_data, .build_id, and .feature) are
      registered in cmd_sched() so header events are parsed correctly

    - The handlers array is promoted to file-scope (latency_handlers[]) and
      evlist__set_tracepoints_handlers() is invoked dynamically inside
      perf_sched__process_tracepoint_sample() when evsel->handler is NULL

    - Trace presence checks are performed post-processing when handling
      pipe data, while non-pipe files continue to abort early upfront

Patch 3 introduces dynamic auto-scaling for latency and runtime display
columns (Runtime, Avg delay, Max delay). Previously, all values were
unconditionally formatted in milliseconds (ms), making microsecond- or
second-scale latencies difficult to read. Columns are now dynamically
scaled to the most appropriate unit (ns, us, ms, s), column headers are
updated, and format specifiers are aligned character-for-character with
table headers.

Patch 4 adds three new command-line options to 'perf sched latency':
    --histogram (-H):
            Displays an ASCII bar chart of CPU wait latencies between
            snapshots
    --hist-mode:
            Configures the bucketing scheme to either logarithmic (log) or
            100 us equal-width linear (linear) mode
    --time:
            Filters trace event processing to a specified [start,stop] time
            span

Changes since v6:

 - Fixed O(N) hot-path traversal overhead in
   perf_sched__process_tracepoint_sample() for unhandled tracepoints by
   assigning a dummy ignore handler (process_sched_ignore)

 - Replaced evlist__set_tracepoints_handlers() in sample processing with a
   per-evsel lookup to prevent -EEXIST early aborts and avoid dropping
   late-arriving pipe events

 - Updated commit message for the pipe mode patch to reflect the
   single-pass handler assignment logic

 - Link to v6: https://lore.kernel.org/lkml/20260801234008.176724-1-atomlin@atomlin.com/

Changes since v5:

 - Split pipe mode trace sample handling from Patch 1 into a standalone
   patch (Namhyung Kim)

 - Added a Fixes: tag to the pipe mode patch referencing commit
   27295592c22e ("perf session: Share the common trace sample_check routine
   as perf_session__has_traces")

 - Updated commit log with before and after illustrations of table header
   formatting

 - Link to v5: https://lore.kernel.org/lkml/20260730185416.97166-1-atomlin@atomlin.com/

Changes since v4:

 - Added the missing .feature callback to sched.tool in cmd_sched()

 - Fixed memory leak by reusing uncompleted work atoms when sched_in events
   are skipped or lost

 - Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
   added a dynamic lookup via evlist__set_tracepoints_handlers() inside
   perf_sched__process_tracepoint_sample() whenever a sample arrives with
   an unattached handler (i.e., evsel->handler == NULL)

 - Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
   in pipe mode and non-pipe modes

 - Added missing trailing pipe in output_lat_thread's new auto-scaling
   format string

 - Stripped redundant prefix strings from each row's format string to
   produce a clean, tabular output

 - Link to v4: https://lore.kernel.org/lkml/20260729144451.38286-1-atomlin@atomlin.com/

Changes since v3:

 - Registered Missing Callbacks. In perf_tool__init configuration inside
   cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
   Without these callbacks, pipe mode drops header attributes entirely,
   preventing tracepoints from being populated in session->evlist

 - Introduced an explicit post-processing pipe check. This ensures that
   pipe mode aborts correctly and does not produce superfluous empty
   latency tables when no trace samples are available

 - Prevented potential NULL pointer dereference. Added a NULL check for
   thread__get_runtime() in map_switch_event() under memory allocation
   failures

 - Fixed column alignment. Modified format specifiers to align
   character-for-character with header column widths across all rows

 - Adjusted header label padding. Updated header label padding and column
   width delimiters in perf_sched__lat() to match the exact field format
   specifiers printed in output_lat_thread()

 - Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
   CPU-specific idle thread) from global_hist buckets

 - Preserved task state machine across --time bounds. Moved time-window
   filtering inside add_sched_in_event() to gate latency statistics
   recording without breaking wakeup state tracking across time interval
   boundaries

 - Link to v3: https://lore.kernel.org/lkml/20260726032533.712462-1-atomlin@atomlin.com/

Changes since v2:

 - Ensured vertical pipe separators align across all latency table columns

 - Excluded "swapper" (idle thread) latency samples from global_hist

 - Preserved task state machine transitions across --time boundaries

 - Suppressed empty table headers, total lines, and histogram graphs when
   no matching trace samples exist (e.g., when --time, --CPU, or --pids
   exclude all samples), outputting "No matching trace samples found."
   instead

 - Linked to v2: https://lore.kernel.org/lkml/20260725173341.679782-1-atomlin@atomlin.com/

Changes since v1:

 - Fixed integer overflow in latency_bucket() (Ian Rogers)

 - Linked to v1: https://lore.kernel.org/lkml/20260724142901.634761-1-atomlin@atomlin.com/

Aaron Tomlin (4):
  perf sched: Suppress latency table output when trace samples are
    missing
  perf sched: Handle missing trace samples in pipe mode
  perf sched latency: Auto-scale latency and runtime display units
  perf sched latency: Add histogram and time interval options

 tools/perf/Documentation/perf-sched.txt |   6 +
 tools/perf/builtin-sched.c              | 338 +++++++++++++++++++++---
 2 files changed, 301 insertions(+), 43 deletions(-)

-- 
2.55.0

On Sun, Aug 02, 2026 at 05:09:10PM -0400, Aaron Tomlin wrote:
> This patch series improves 'perf sched latency' by suppressing misleading
> empty table output, extending pipe mode stream processing, introducing
> dynamic unit auto-scaling for latency and runtime statistics, and adding
> latency histogram visualisation alongside time-span filtering.
> 
> Patch 1 addresses an issue where 'perf sched latency' fell through and
> returned success (0) when perf_session__has_traces() failed due to missing
> tracepoint events in a perf.data file. This caused empty header tables and
> zeroed summary statistics to be rendered. In addition,
> thread__get_runtime() in map_switch_event() is guarded against potential
> NULL pointer dereferences under memory allocation failures.
> 
> Patch 2 extends pipe mode stream support. Because event attributes in pipe
> mode are received dynamically during event processing, session->evlist is
> not populated prior to event processing.
> To handle pipe input correctly:
>     - Missing callbacks (.attr, .tracing_data, .build_id, and .feature) are
>       registered in cmd_sched() so header events are parsed correctly
> 
>     - The handlers array is promoted to file-scope (latency_handlers[]) and
>       evlist__set_tracepoints_handlers() is invoked dynamically inside
>       perf_sched__process_tracepoint_sample() when evsel->handler is NULL
> 
>     - Trace presence checks are performed post-processing when handling
>       pipe data, while non-pipe files continue to abort early upfront
> 
> Patch 3 introduces dynamic auto-scaling for latency and runtime display
> columns (Runtime, Avg delay, Max delay). Previously, all values were
> unconditionally formatted in milliseconds (ms), making microsecond- or
> second-scale latencies difficult to read. Columns are now dynamically
> scaled to the most appropriate unit (ns, us, ms, s), column headers are
> updated, and format specifiers are aligned character-for-character with
> table headers.
> 
> Patch 4 adds three new command-line options to 'perf sched latency':
>     --histogram (-H):
>             Displays an ASCII bar chart of CPU wait latencies between
>             snapshots
>     --hist-mode:
>             Configures the bucketing scheme to either logarithmic (log) or
>             100 us equal-width linear (linear) mode
>     --time:
>             Filters trace event processing to a specified [start,stop] time
>             span

Can you please also add a test case to check the basic functionality?

Thanks,
Namhyung

> 
> Changes since v6:
> 
>  - Fixed O(N) hot-path traversal overhead in
>    perf_sched__process_tracepoint_sample() for unhandled tracepoints by
>    assigning a dummy ignore handler (process_sched_ignore)
> 
>  - Replaced evlist__set_tracepoints_handlers() in sample processing with a
>    per-evsel lookup to prevent -EEXIST early aborts and avoid dropping
>    late-arriving pipe events
> 
>  - Updated commit message for the pipe mode patch to reflect the
>    single-pass handler assignment logic
> 
>  - Link to v6: https://lore.kernel.org/lkml/20260801234008.176724-1-atomlin@atomlin.com/
> 
> Changes since v5:
> 
>  - Split pipe mode trace sample handling from Patch 1 into a standalone
>    patch (Namhyung Kim)
> 
>  - Added a Fixes: tag to the pipe mode patch referencing commit
>    27295592c22e ("perf session: Share the common trace sample_check routine
>    as perf_session__has_traces")
> 
>  - Updated commit log with before and after illustrations of table header
>    formatting
> 
>  - Link to v5: https://lore.kernel.org/lkml/20260730185416.97166-1-atomlin@atomlin.com/
> 
> Changes since v4:
> 
>  - Added the missing .feature callback to sched.tool in cmd_sched()
> 
>  - Fixed memory leak by reusing uncompleted work atoms when sched_in events
>    are skipped or lost
> 
>  - Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
>    added a dynamic lookup via evlist__set_tracepoints_handlers() inside
>    perf_sched__process_tracepoint_sample() whenever a sample arrives with
>    an unattached handler (i.e., evsel->handler == NULL)
> 
>  - Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
>    in pipe mode and non-pipe modes
> 
>  - Added missing trailing pipe in output_lat_thread's new auto-scaling
>    format string
> 
>  - Stripped redundant prefix strings from each row's format string to
>    produce a clean, tabular output
> 
>  - Link to v4: https://lore.kernel.org/lkml/20260729144451.38286-1-atomlin@atomlin.com/
> 
> Changes since v3:
> 
>  - Registered Missing Callbacks. In perf_tool__init configuration inside
>    cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
>    Without these callbacks, pipe mode drops header attributes entirely,
>    preventing tracepoints from being populated in session->evlist
> 
>  - Introduced an explicit post-processing pipe check. This ensures that
>    pipe mode aborts correctly and does not produce superfluous empty
>    latency tables when no trace samples are available
> 
>  - Prevented potential NULL pointer dereference. Added a NULL check for
>    thread__get_runtime() in map_switch_event() under memory allocation
>    failures
> 
>  - Fixed column alignment. Modified format specifiers to align
>    character-for-character with header column widths across all rows
> 
>  - Adjusted header label padding. Updated header label padding and column
>    width delimiters in perf_sched__lat() to match the exact field format
>    specifiers printed in output_lat_thread()
> 
>  - Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
>    CPU-specific idle thread) from global_hist buckets
> 
>  - Preserved task state machine across --time bounds. Moved time-window
>    filtering inside add_sched_in_event() to gate latency statistics
>    recording without breaking wakeup state tracking across time interval
>    boundaries
> 
>  - Link to v3: https://lore.kernel.org/lkml/20260726032533.712462-1-atomlin@atomlin.com/
> 
> Changes since v2:
> 
>  - Ensured vertical pipe separators align across all latency table columns
> 
>  - Excluded "swapper" (idle thread) latency samples from global_hist
> 
>  - Preserved task state machine transitions across --time boundaries
> 
>  - Suppressed empty table headers, total lines, and histogram graphs when
>    no matching trace samples exist (e.g., when --time, --CPU, or --pids
>    exclude all samples), outputting "No matching trace samples found."
>    instead
> 
>  - Linked to v2: https://lore.kernel.org/lkml/20260725173341.679782-1-atomlin@atomlin.com/
> 
> Changes since v1:
> 
>  - Fixed integer overflow in latency_bucket() (Ian Rogers)
> 
>  - Linked to v1: https://lore.kernel.org/lkml/20260724142901.634761-1-atomlin@atomlin.com/
> 
> Aaron Tomlin (4):
>   perf sched: Suppress latency table output when trace samples are
>     missing
>   perf sched: Handle missing trace samples in pipe mode
>   perf sched latency: Auto-scale latency and runtime display units
>   perf sched latency: Add histogram and time interval options
> 
>  tools/perf/Documentation/perf-sched.txt |   6 +
>  tools/perf/builtin-sched.c              | 338 +++++++++++++++++++++---
>  2 files changed, 301 insertions(+), 43 deletions(-)
> 
> -- 
> 2.55.0
>

On Tue, Aug 04, 2026 at 10:11:23AM -0700, Namhyung Kim wrote:
> > Patch 4 adds three new command-line options to 'perf sched latency':
> >     --histogram (-H):
> >             Displays an ASCII bar chart of CPU wait latencies between
> >             snapshots
> >     --hist-mode:
> >             Configures the bucketing scheme to either logarithmic (log) or
> >             100 us equal-width linear (linear) mode
> >     --time:
> >             Filters trace event processing to a specified [start,stop] time
> >             span
> 
> Can you please also add a test case to check the basic functionality?
> 
Hi Namhyung,

Absolutely!


Kind regards,
-- 
Aaron Tomlin