Hi Namhyung, Ian, Arnaldo,

This patch series improves 'perf sched latency' by suppressing misleading
empty table output, extending pipe mode stream processing, introducing
dynamic unit auto-scaling for latency and runtime statistics, and adding
latency histogram visualisation alongside time-span filtering.

Patch 1 addresses an issue where 'perf sched latency' fell through and
returned success (0) when perf_session__has_traces() failed due to missing
tracepoint events in a perf.data file. This caused empty header tables and
zeroed summary statistics to be rendered. In addition,
thread__get_runtime() in map_switch_event() is guarded against potential
NULL pointer dereferences under memory allocation failures.

Patch 2 extends pipe mode stream support. Because event attributes in pipe
mode are received dynamically during event processing, session->evlist is
not populated prior to event processing.
To handle pipe input correctly:
    - Register the missing .attr, .tracing_data, .build_id, and .feature
      callbacks in cmd_sched()
    
    - Promote the handlers array to file-scope (latency_handlers[]) and
      dynamically assign matching tracepoint handlers (or
      process_sched_ignore) inside perf_sched__process_tracepoint_sample()
      when evsel->handler is NULL; replace process_sched_wakeup_ignore()
      with process_sched_ignore()
    
    - Perform the trace check post-processing when handling pipe data

Patch 3 introduces dynamic auto-scaling for latency and runtime display
columns (Runtime, Avg delay, Max delay). Previously, all values were
unconditionally formatted in milliseconds (ms), making microsecond- or
second-scale latencies difficult to read. Columns are now dynamically
scaled to the most appropriate unit (ns, us, ms, s), column headers are
updated, and format specifiers are aligned character-for-character with
table headers.

Patch 4 adds three new command-line options to 'perf sched latency':
    --histogram (-H):
            Displays an ASCII bar chart of CPU wait latencies between
            snapshots
    --hist-mode:
            Configures the bucketing scheme to either logarithmic (log) or
            100 us equal-width linear (linear) mode
    --time:
            Filters trace event processing to a specified [start,stop] time
            span

Changes since v7:

 - Added a guard clause in perf_sched__process_tracepoint_sample() to
   reduce loop indentation (Namhyung Kim)

 - Converted dummy process_sched_wakeup_ignore() references to
   process_sched_ignore and eliminated redundant code (Namhyung Kim)

 - Replaced string comparisons with faster integer checks (i.e.,
   thread__tid(...) != 0) for swapper (idle thread) handling (Namhyung Kim)

 - Added automated test cases for --histogram, --hist-mode, and --time into
   tools/perf/tests/shell/sched.sh (Namhyung Kim)

 - Link to v7: https://lore.kernel.org/lkml/20260802210914.199941-1-atomlin@atomlin.com/

Changes since v6:

 - Fixed O(N) hot-path traversal overhead in
   perf_sched__process_tracepoint_sample() for unhandled tracepoints by
   assigning a dummy ignore handler (process_sched_ignore)

 - Replaced evlist__set_tracepoints_handlers() in sample processing with a
   per-evsel lookup to prevent -EEXIST early aborts and avoid dropping
   late-arriving pipe events

 - Updated commit message for the pipe mode patch to reflect the
   single-pass handler assignment logic

 - Link to v6: https://lore.kernel.org/lkml/20260801234008.176724-1-atomlin@atomlin.com/

Changes since v5:

 - Split pipe mode trace sample handling from Patch 1 into a standalone
   patch (Namhyung Kim)

 - Added a Fixes: tag to the pipe mode patch referencing commit
   27295592c22e ("perf session: Share the common trace sample_check routine
   as perf_session__has_traces")

 - Updated commit log with before and after illustrations of table header
   formatting

 - Link to v5: https://lore.kernel.org/lkml/20260730185416.97166-1-atomlin@atomlin.com/

Changes since v4:

 - Added the missing .feature callback to sched.tool in cmd_sched()

 - Fixed memory leak by reusing uncompleted work atoms when sched_in events
   are skipped or lost

 - Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
   added a dynamic lookup via evlist__set_tracepoints_handlers() inside
   perf_sched__process_tracepoint_sample() whenever a sample arrives with
   an unattached handler (i.e., evsel->handler == NULL)

 - Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
   in pipe mode and non-pipe modes

 - Added missing trailing pipe in output_lat_thread's new auto-scaling
   format string

 - Stripped redundant prefix strings from each row's format string to
   produce a clean, tabular output

 - Link to v4: https://lore.kernel.org/lkml/20260729144451.38286-1-atomlin@atomlin.com/

Changes since v3:

 - Registered Missing Callbacks. In perf_tool__init configuration inside
   cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
   Without these callbacks, pipe mode drops header attributes entirely,
   preventing tracepoints from being populated in session->evlist

 - Introduced an explicit post-processing pipe check. This ensures that
   pipe mode aborts correctly and does not produce superfluous empty
   latency tables when no trace samples are available

 - Prevented potential NULL pointer dereference. Added a NULL check for
   thread__get_runtime() in map_switch_event() under memory allocation
   failures

 - Fixed column alignment. Modified format specifiers to align
   character-for-character with header column widths across all rows

 - Adjusted header label padding. Updated header label padding and column
   width delimiters in perf_sched__lat() to match the exact field format
   specifiers printed in output_lat_thread()

 - Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
   CPU-specific idle thread) from global_hist buckets

 - Preserved task state machine across --time bounds. Moved time-window
   filtering inside add_sched_in_event() to gate latency statistics
   recording without breaking wakeup state tracking across time interval
   boundaries

 - Link to v3: https://lore.kernel.org/lkml/20260726032533.712462-1-atomlin@atomlin.com/

Changes since v2:

 - Ensured vertical pipe separators align across all latency table columns

 - Excluded "swapper" (idle thread) latency samples from global_hist

 - Preserved task state machine transitions across --time boundaries

 - Suppressed empty table headers, total lines, and histogram graphs when
   no matching trace samples exist (e.g., when --time, --CPU, or --pids
   exclude all samples), outputting "No matching trace samples found."
   instead

 - Linked to v2: https://lore.kernel.org/lkml/20260725173341.679782-1-atomlin@atomlin.com/

Changes since v1:

 - Fixed integer overflow in latency_bucket() (Ian Rogers)

 - Linked to v1: https://lore.kernel.org/lkml/20260724142901.634761-1-atomlin@atomlin.com/

Aaron Tomlin (4):
  perf sched: Suppress latency table output when trace samples are
    missing
  perf sched: Handle missing trace samples in pipe mode
  perf sched latency: Auto-scale latency and runtime display units
  perf sched latency: Add histogram and time interval options

 tools/perf/Documentation/perf-sched.txt |   6 +
 tools/perf/builtin-sched.c              | 347 ++++++++++++++++++++----
 tools/perf/tests/shell/sched.sh         |  29 ++
 3 files changed, 332 insertions(+), 50 deletions(-)

-- 
2.55.0

Hello,

On Wed, Aug 05, 2026 at 05:07:47PM -0400, Aaron Tomlin wrote:
> Hi Namhyung, Ian, Arnaldo,
> 
> This patch series improves 'perf sched latency' by suppressing misleading
> empty table output, extending pipe mode stream processing, introducing
> dynamic unit auto-scaling for latency and runtime statistics, and adding
> latency histogram visualisation alongside time-span filtering.

Thanks for working on this.  But it doesn't apply unfortunately.
Can you please rebase on to the current perf-tools-next?

Thanks,
Namhyung

On Wed, Aug 5, 2026 at 2:08 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
>
> Hi Namhyung, Ian, Arnaldo,
>
> This patch series improves 'perf sched latency' by suppressing misleading
> empty table output, extending pipe mode stream processing, introducing
> dynamic unit auto-scaling for latency and runtime statistics, and adding
> latency histogram visualisation alongside time-span filtering.
>
> Patch 1 addresses an issue where 'perf sched latency' fell through and
> returned success (0) when perf_session__has_traces() failed due to missing
> tracepoint events in a perf.data file. This caused empty header tables and
> zeroed summary statistics to be rendered. In addition,
> thread__get_runtime() in map_switch_event() is guarded against potential
> NULL pointer dereferences under memory allocation failures.
>
> Patch 2 extends pipe mode stream support. Because event attributes in pipe
> mode are received dynamically during event processing, session->evlist is
> not populated prior to event processing.
> To handle pipe input correctly:
>     - Register the missing .attr, .tracing_data, .build_id, and .feature
>       callbacks in cmd_sched()
>
>     - Promote the handlers array to file-scope (latency_handlers[]) and
>       dynamically assign matching tracepoint handlers (or
>       process_sched_ignore) inside perf_sched__process_tracepoint_sample()
>       when evsel->handler is NULL; replace process_sched_wakeup_ignore()
>       with process_sched_ignore()
>
>     - Perform the trace check post-processing when handling pipe data
>
> Patch 3 introduces dynamic auto-scaling for latency and runtime display
> columns (Runtime, Avg delay, Max delay). Previously, all values were
> unconditionally formatted in milliseconds (ms), making microsecond- or
> second-scale latencies difficult to read. Columns are now dynamically
> scaled to the most appropriate unit (ns, us, ms, s), column headers are
> updated, and format specifiers are aligned character-for-character with
> table headers.
>
> Patch 4 adds three new command-line options to 'perf sched latency':
>     --histogram (-H):
>             Displays an ASCII bar chart of CPU wait latencies between
>             snapshots
>     --hist-mode:
>             Configures the bucketing scheme to either logarithmic (log) or
>             100 us equal-width linear (linear) mode
>     --time:
>             Filters trace event processing to a specified [start,stop] time
>             span

Thanks for these improvements and the tests for coverage! For the series:

Reviewed-by: Ian Rogers <irogers@google.com>

Thanks,
Ian

> Changes since v7:
>
>  - Added a guard clause in perf_sched__process_tracepoint_sample() to
>    reduce loop indentation (Namhyung Kim)
>
>  - Converted dummy process_sched_wakeup_ignore() references to
>    process_sched_ignore and eliminated redundant code (Namhyung Kim)
>
>  - Replaced string comparisons with faster integer checks (i.e.,
>    thread__tid(...) != 0) for swapper (idle thread) handling (Namhyung Kim)
>
>  - Added automated test cases for --histogram, --hist-mode, and --time into
>    tools/perf/tests/shell/sched.sh (Namhyung Kim)
>
>  - Link to v7: https://lore.kernel.org/lkml/20260802210914.199941-1-atomlin@atomlin.com/
>
> Changes since v6:
>
>  - Fixed O(N) hot-path traversal overhead in
>    perf_sched__process_tracepoint_sample() for unhandled tracepoints by
>    assigning a dummy ignore handler (process_sched_ignore)
>
>  - Replaced evlist__set_tracepoints_handlers() in sample processing with a
>    per-evsel lookup to prevent -EEXIST early aborts and avoid dropping
>    late-arriving pipe events
>
>  - Updated commit message for the pipe mode patch to reflect the
>    single-pass handler assignment logic
>
>  - Link to v6: https://lore.kernel.org/lkml/20260801234008.176724-1-atomlin@atomlin.com/
>
> Changes since v5:
>
>  - Split pipe mode trace sample handling from Patch 1 into a standalone
>    patch (Namhyung Kim)
>
>  - Added a Fixes: tag to the pipe mode patch referencing commit
>    27295592c22e ("perf session: Share the common trace sample_check routine
>    as perf_session__has_traces")
>
>  - Updated commit log with before and after illustrations of table header
>    formatting
>
>  - Link to v5: https://lore.kernel.org/lkml/20260730185416.97166-1-atomlin@atomlin.com/
>
> Changes since v4:
>
>  - Added the missing .feature callback to sched.tool in cmd_sched()
>
>  - Fixed memory leak by reusing uncompleted work atoms when sched_in events
>    are skipped or lost
>
>  - Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
>    added a dynamic lookup via evlist__set_tracepoints_handlers() inside
>    perf_sched__process_tracepoint_sample() whenever a sample arrives with
>    an unattached handler (i.e., evsel->handler == NULL)
>
>  - Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
>    in pipe mode and non-pipe modes
>
>  - Added missing trailing pipe in output_lat_thread's new auto-scaling
>    format string
>
>  - Stripped redundant prefix strings from each row's format string to
>    produce a clean, tabular output
>
>  - Link to v4: https://lore.kernel.org/lkml/20260729144451.38286-1-atomlin@atomlin.com/
>
> Changes since v3:
>
>  - Registered Missing Callbacks. In perf_tool__init configuration inside
>    cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
>    Without these callbacks, pipe mode drops header attributes entirely,
>    preventing tracepoints from being populated in session->evlist
>
>  - Introduced an explicit post-processing pipe check. This ensures that
>    pipe mode aborts correctly and does not produce superfluous empty
>    latency tables when no trace samples are available
>
>  - Prevented potential NULL pointer dereference. Added a NULL check for
>    thread__get_runtime() in map_switch_event() under memory allocation
>    failures
>
>  - Fixed column alignment. Modified format specifiers to align
>    character-for-character with header column widths across all rows
>
>  - Adjusted header label padding. Updated header label padding and column
>    width delimiters in perf_sched__lat() to match the exact field format
>    specifiers printed in output_lat_thread()
>
>  - Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
>    CPU-specific idle thread) from global_hist buckets
>
>  - Preserved task state machine across --time bounds. Moved time-window
>    filtering inside add_sched_in_event() to gate latency statistics
>    recording without breaking wakeup state tracking across time interval
>    boundaries
>
>  - Link to v3: https://lore.kernel.org/lkml/20260726032533.712462-1-atomlin@atomlin.com/
>
> Changes since v2:
>
>  - Ensured vertical pipe separators align across all latency table columns
>
>  - Excluded "swapper" (idle thread) latency samples from global_hist
>
>  - Preserved task state machine transitions across --time boundaries
>
>  - Suppressed empty table headers, total lines, and histogram graphs when
>    no matching trace samples exist (e.g., when --time, --CPU, or --pids
>    exclude all samples), outputting "No matching trace samples found."
>    instead
>
>  - Linked to v2: https://lore.kernel.org/lkml/20260725173341.679782-1-atomlin@atomlin.com/
>
> Changes since v1:
>
>  - Fixed integer overflow in latency_bucket() (Ian Rogers)
>
>  - Linked to v1: https://lore.kernel.org/lkml/20260724142901.634761-1-atomlin@atomlin.com/
>
> Aaron Tomlin (4):
>   perf sched: Suppress latency table output when trace samples are
>     missing
>   perf sched: Handle missing trace samples in pipe mode
>   perf sched latency: Auto-scale latency and runtime display units
>   perf sched latency: Add histogram and time interval options
>
>  tools/perf/Documentation/perf-sched.txt |   6 +
>  tools/perf/builtin-sched.c              | 347 ++++++++++++++++++++----
>  tools/perf/tests/shell/sched.sh         |  29 ++
>  3 files changed, 332 insertions(+), 50 deletions(-)
>
> --
> 2.55.0
>

On Wed, Aug 05, 2026 at 03:02:06PM -0700, Ian Rogers wrote:
> On Wed, Aug 5, 2026 at 2:08 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
> >
> > Hi Namhyung, Ian, Arnaldo,
> >
> > This patch series improves 'perf sched latency' by suppressing misleading
> > empty table output, extending pipe mode stream processing, introducing
> > dynamic unit auto-scaling for latency and runtime statistics, and adding
> > latency histogram visualisation alongside time-span filtering.
> >
> > Patch 1 addresses an issue where 'perf sched latency' fell through and
> > returned success (0) when perf_session__has_traces() failed due to missing
> > tracepoint events in a perf.data file. This caused empty header tables and
> > zeroed summary statistics to be rendered. In addition,
> > thread__get_runtime() in map_switch_event() is guarded against potential
> > NULL pointer dereferences under memory allocation failures.
> >
> > Patch 2 extends pipe mode stream support. Because event attributes in pipe
> > mode are received dynamically during event processing, session->evlist is
> > not populated prior to event processing.
> > To handle pipe input correctly:
> >     - Register the missing .attr, .tracing_data, .build_id, and .feature
> >       callbacks in cmd_sched()
> >
> >     - Promote the handlers array to file-scope (latency_handlers[]) and
> >       dynamically assign matching tracepoint handlers (or
> >       process_sched_ignore) inside perf_sched__process_tracepoint_sample()
> >       when evsel->handler is NULL; replace process_sched_wakeup_ignore()
> >       with process_sched_ignore()
> >
> >     - Perform the trace check post-processing when handling pipe data
> >
> > Patch 3 introduces dynamic auto-scaling for latency and runtime display
> > columns (Runtime, Avg delay, Max delay). Previously, all values were
> > unconditionally formatted in milliseconds (ms), making microsecond- or
> > second-scale latencies difficult to read. Columns are now dynamically
> > scaled to the most appropriate unit (ns, us, ms, s), column headers are
> > updated, and format specifiers are aligned character-for-character with
> > table headers.
> >
> > Patch 4 adds three new command-line options to 'perf sched latency':
> >     --histogram (-H):
> >             Displays an ASCII bar chart of CPU wait latencies between
> >             snapshots
> >     --hist-mode:
> >             Configures the bucketing scheme to either logarithmic (log) or
> >             100 us equal-width linear (linear) mode
> >     --time:
> >             Filters trace event processing to a specified [start,stop] time
> >             span
> 
> Thanks for these improvements and the tests for coverage! For the series:
> 
> Reviewed-by: Ian Rogers <irogers@google.com>
> 
> Thanks,
> Ian

Thank you Ian!

-- 
Aaron Tomlin