Documentation/trace/ftrace.rst | 9 +++++++++ include/linux/trace_seq.h | 4 ++-- kernel/trace/trace.h | 1 + kernel/trace/trace_output.c | 6 +++++- kernel/trace/trace_seq.c | 11 +++++++---- 5 files changed, 24 insertions(+), 7 deletions(-)
Add support for displaying bitmasks in human-readable list format (e.g.,
0,2-5,7) in addition to the default hexadecimal bitmap representation.
This is particularly useful when tracing CPU masks and other large
bitmasks where individual bit positions are more meaningful than their
hexadecimal encoding.
When the "bitmask-list" option is enabled, the printk "%*pbl" format
specifier is used to render bitmasks as comma-separated ranges, making
trace output easier to interpret for complex CPU configurations and
large bitmask values.
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
Documentation/trace/ftrace.rst | 9 +++++++++
include/linux/trace_seq.h | 4 ++--
kernel/trace/trace.h | 1 +
kernel/trace/trace_output.c | 6 +++++-
kernel/trace/trace_seq.c | 11 +++++++----
5 files changed, 24 insertions(+), 7 deletions(-)
diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst
index d1f313a5f4ad..639f4d95732f 100644
--- a/Documentation/trace/ftrace.rst
+++ b/Documentation/trace/ftrace.rst
@@ -1290,6 +1290,15 @@ Here are the available options:
This will be useful if you want to find out which hashed
value is corresponding to the real value in trace log.
+ bitmask-list
+ When enabled, bitmasks are displayed as a human-readable list of
+ ranges (e.g., 0,2-5,7) using the printk "%*pbl" format specifier.
+ When disabled (the default), bitmasks are displayed in the
+ traditional hexadecimal bitmap representation. The list format is
+ particularly useful for tracing CPU masks and other large bitmasks
+ where individual bit positions are more meaningful than their
+ hexadecimal encoding.
+
record-cmd
When any event or tracer is enabled, a hook is enabled
in the sched_switch trace point to fill comm cache
diff --git a/include/linux/trace_seq.h b/include/linux/trace_seq.h
index 4a0b8c172d27..5e5b4331f8f1 100644
--- a/include/linux/trace_seq.h
+++ b/include/linux/trace_seq.h
@@ -114,7 +114,7 @@ extern void trace_seq_putmem_hex(struct trace_seq *s, const void *mem,
extern int trace_seq_path(struct trace_seq *s, const struct path *path);
extern void trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp,
- int nmaskbits);
+ int nmaskbits, bool show_list);
extern int trace_seq_hex_dump(struct trace_seq *s, const char *prefix_str,
int prefix_type, int rowsize, int groupsize,
@@ -133,7 +133,7 @@ void trace_seq_bprintf(struct trace_seq *s, const char *fmt, const u32 *binary)
static inline void
trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp,
- int nmaskbits)
+ int nmaskbits, bool show_list)
{
}
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index b6d42fe06115..8888fc9335b6 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -1411,6 +1411,7 @@ extern int trace_get_user(struct trace_parser *parser, const char __user *ubuf,
C(COPY_MARKER, "copy_trace_marker"), \
C(PAUSE_ON_TRACE, "pause-on-trace"), \
C(HASH_PTR, "hash-ptr"), /* Print hashed pointer */ \
+ C(BITMASK_LIST, "bitmask-list"), \
FUNCTION_FLAGS \
FGRAPH_FLAGS \
STACK_FLAGS \
diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c
index cc2d3306bb60..ede1a1b6d5a6 100644
--- a/kernel/trace/trace_output.c
+++ b/kernel/trace/trace_output.c
@@ -199,8 +199,12 @@ trace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr,
unsigned int bitmask_size)
{
const char *ret = trace_seq_buffer_ptr(p);
+ const struct trace_array *tr = trace_get_global_array();
+ bool show_bitmask_list = tr->trace_flags &
+ TRACE_ITER(BITMASK_LIST);
- trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8);
+ trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8,
+ show_bitmask_list);
trace_seq_putc(p, 0);
return ret;
diff --git a/kernel/trace/trace_seq.c b/kernel/trace/trace_seq.c
index 32684ef4fb9d..5445fecf5f95 100644
--- a/kernel/trace/trace_seq.c
+++ b/kernel/trace/trace_seq.c
@@ -98,24 +98,27 @@ void trace_seq_printf(struct trace_seq *s, const char *fmt, ...)
EXPORT_SYMBOL_GPL(trace_seq_printf);
/**
- * trace_seq_bitmask - write a bitmask array in its ASCII representation
+ * trace_seq_bitmask - write a bitmask array in its ASCII or list representation
* @s: trace sequence descriptor
* @maskp: points to an array of unsigned longs that represent a bitmask
* @nmaskbits: The number of bits that are valid in @maskp
+ * @show_list: True for comma-separated list of ranges, false for hex bitmap
*
- * Writes a ASCII representation of a bitmask string into @s.
+ * Writes a ASCII or list (e.g., 0-3,5-7) representation of a bitmask
+ * string into @s.
*/
void trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp,
- int nmaskbits)
+ int nmaskbits, bool show_list)
{
unsigned int save_len = s->seq.len;
+ const char *fmt = show_list ? "%*pbl" : "%*pb";
if (s->full)
return;
__trace_seq_init(s);
- seq_buf_printf(&s->seq, "%*pb", nmaskbits, maskp);
+ seq_buf_printf(&s->seq, fmt, nmaskbits, maskp);
if (unlikely(seq_buf_has_overflowed(&s->seq))) {
s->seq.len = save_len;
--
2.51.0Hi Aaron,
kernel test robot noticed the following build errors:
[auto build test ERROR on trace/for-next]
[also build test ERROR on linus/master v6.19-rc2 next-20251219]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Aaron-Tomlin/tracing-Add-bitmask-list-option-for-human-readable-bitmask-display/20251223-120923
base: https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link: https://lore.kernel.org/r/20251223035622.2084081-1-atomlin%40atomlin.com
patch subject: [PATCH] tracing: Add bitmask-list option for human-readable bitmask display
config: i386-randconfig-141-20251224 (https://download.01.org/0day-ci/archive/20251224/202512240709.cPAuuxFj-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251224/202512240709.cPAuuxFj-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202512240709.cPAuuxFj-lkp@intel.com/
All errors (new ones prefixed by >>):
kernel/trace/trace_output.c: In function 'trace_print_bitmask_seq':
kernel/trace/trace_output.c:202:40: error: implicit declaration of function 'trace_get_global_array'; did you mean 'ftrace_init_global_array_ops'? [-Wimplicit-function-declaration]
202 | const struct trace_array *tr = trace_get_global_array();
| ^~~~~~~~~~~~~~~~~~~~~~
| ftrace_init_global_array_ops
>> kernel/trace/trace_output.c:202:40: error: initialization of 'const struct trace_array *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
vim +202 kernel/trace/trace_output.c
196
197 const char *
198 trace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr,
199 unsigned int bitmask_size)
200 {
201 const char *ret = trace_seq_buffer_ptr(p);
> 202 const struct trace_array *tr = trace_get_global_array();
203 bool show_bitmask_list = tr->trace_flags &
204 TRACE_ITER(BITMASK_LIST);
205
206 trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8,
207 show_bitmask_list);
208 trace_seq_putc(p, 0);
209
210 return ret;
211 }
212 EXPORT_SYMBOL_GPL(trace_print_bitmask_seq);
213
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Hi Aaron,
kernel test robot noticed the following build errors:
[auto build test ERROR on trace/for-next]
[also build test ERROR on linus/master v6.19-rc2 next-20251219]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Aaron-Tomlin/tracing-Add-bitmask-list-option-for-human-readable-bitmask-display/20251223-120923
base: https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
patch link: https://lore.kernel.org/r/20251223035622.2084081-1-atomlin%40atomlin.com
patch subject: [PATCH] tracing: Add bitmask-list option for human-readable bitmask display
config: s390-randconfig-r071-20251224 (https://download.01.org/0day-ci/archive/20251224/202512240638.SumtF5C4-lkp@intel.com/config)
compiler: s390-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251224/202512240638.SumtF5C4-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202512240638.SumtF5C4-lkp@intel.com/
All error/warnings (new ones prefixed by >>):
kernel/trace/trace_output.c: In function 'trace_print_bitmask_seq':
>> kernel/trace/trace_output.c:202:33: error: implicit declaration of function 'trace_get_global_array'; did you mean 'ftrace_init_global_array_ops'? [-Werror=implicit-function-declaration]
const struct trace_array *tr = trace_get_global_array();
^~~~~~~~~~~~~~~~~~~~~~
ftrace_init_global_array_ops
>> kernel/trace/trace_output.c:202:33: warning: initialization of 'const struct trace_array *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
cc1: some warnings being treated as errors
vim +202 kernel/trace/trace_output.c
196
197 const char *
198 trace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr,
199 unsigned int bitmask_size)
200 {
201 const char *ret = trace_seq_buffer_ptr(p);
> 202 const struct trace_array *tr = trace_get_global_array();
203 bool show_bitmask_list = tr->trace_flags &
204 TRACE_ITER(BITMASK_LIST);
205
206 trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8,
207 show_bitmask_list);
208 trace_seq_putc(p, 0);
209
210 return ret;
211 }
212 EXPORT_SYMBOL_GPL(trace_print_bitmask_seq);
213
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
On Mon, 22 Dec 2025 22:56:22 -0500 Aaron Tomlin <atomlin@atomlin.com> wrote: > Add support for displaying bitmasks in human-readable list format (e.g., > 0,2-5,7) in addition to the default hexadecimal bitmap representation. > This is particularly useful when tracing CPU masks and other large > bitmasks where individual bit positions are more meaningful than their > hexadecimal encoding. > > When the "bitmask-list" option is enabled, the printk "%*pbl" format > specifier is used to render bitmasks as comma-separated ranges, making > trace output easier to interpret for complex CPU configurations and > large bitmask values. Hmm, I have a couple of issues with this change. One, this is global. It affects all instances. The other is that if this is going to be done, then instead of adding a parameter to trace_seq_bitmask(), another trace_seq_* API should be created. Perhaps trace_seq_bitmask_cnt()? And have trace_print_bitmask_seq() call them separately. I'm still not convinced that this is needed. What examples do you see? Should it be only for CPU bitmasks? I think a bit more thought needs to be made on a change like this. There's other options that were added that I now regret. I don't want to add another one I wish we didn't have. -- Steve
On Tue, Dec 23, 2025 at 01:34:52PM -0500, Steven Rostedt wrote:
> On Mon, 22 Dec 2025 22:56:22 -0500
> Aaron Tomlin <atomlin@atomlin.com> wrote:
>
> > Add support for displaying bitmasks in human-readable list format (e.g.,
> > 0,2-5,7) in addition to the default hexadecimal bitmap representation.
> > This is particularly useful when tracing CPU masks and other large
> > bitmasks where individual bit positions are more meaningful than their
> > hexadecimal encoding.
> >
> > When the "bitmask-list" option is enabled, the printk "%*pbl" format
> > specifier is used to render bitmasks as comma-separated ranges, making
> > trace output easier to interpret for complex CPU configurations and
> > large bitmask values.
>
> Hmm, I have a couple of issues with this change. One, this is global. It
> affects all instances. The other is that if this is going to be done, then
> instead of adding a parameter to trace_seq_bitmask(), another trace_seq_*
> API should be created. Perhaps trace_seq_bitmask_cnt()? And have
> trace_print_bitmask_seq() call them separately.
>
> I'm still not convinced that this is needed. What examples do you see?
> Should it be only for CPU bitmasks?
>
> I think a bit more thought needs to be made on a change like this. There's
> other options that were added that I now regret. I don't want to add
> another one I wish we didn't have.
>
Hi Steven,
Regarding the scope, I take your point that a global flag is perhaps too
blunt an instrument. I can certainly refactor this to be instance-aware by
passing the trace_iterator through the call chain.
As for the use case, I find this particularly invaluable when debugging IPI
latency or affinity issues on high-core-count systems. I typically run this
with the "nop" tracer and enable only the events/ipi/ipi_send_cpumask/
event.
When dealing with 128+ logical cores, interpreting a raw hexadecimal bitmap
to identify targeted CPUs is taxing and prone to error. For example, if I
am investigating why CPU 6 is being interrupted, I might use a filter like
"cpumask & CPU{6}". Seeing the resulting output as a range list (e.g., 0-7)
rather than a hexadecimal bitmask allows one to deduce almost instantly
which cluster of CPUs is involved in the IPI broadcast.
To avoid duplication, I am of the opinion that we should maintain a single
trace_seq_bitmask() and simply extend it. However, I am happy to extend the
API if you prefer. Given that struct trace_iterator *iter is available in
the output path, how about the following (not yet fully tested):
diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h
index 3690221ba3d8..0a2b8229b999 100644
--- a/include/linux/trace_events.h
+++ b/include/linux/trace_events.h
@@ -38,7 +38,10 @@ const char *trace_print_symbols_seq_u64(struct trace_seq *p,
*symbol_array);
#endif
-const char *trace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr,
+struct trace_iterator;
+struct trace_event;
+
+const char *trace_print_bitmask_seq(struct trace_iterator *iter, void *bitmask_ptr,
unsigned int bitmask_size);
const char *trace_print_hex_seq(struct trace_seq *p,
@@ -54,9 +57,6 @@ trace_print_hex_dump_seq(struct trace_seq *p, const char *prefix_str,
int prefix_type, int rowsize, int groupsize,
const void *buf, size_t len, bool ascii);
-struct trace_iterator;
-struct trace_event;
-
int trace_raw_output_prep(struct trace_iterator *iter,
struct trace_event *event);
extern __printf(2, 3)
diff --git a/include/trace/stages/stage3_trace_output.h b/include/trace/stages/stage3_trace_output.h
index 1e7b0bef95f5..fce85ea2df1c 100644
--- a/include/trace/stages/stage3_trace_output.h
+++ b/include/trace/stages/stage3_trace_output.h
@@ -39,7 +39,7 @@
void *__bitmask = __get_dynamic_array(field); \
unsigned int __bitmask_size; \
__bitmask_size = __get_dynamic_array_len(field); \
- trace_print_bitmask_seq(p, __bitmask, __bitmask_size); \
+ trace_print_bitmask_seq(iter, __bitmask, __bitmask_size); \
})
#undef __get_cpumask
@@ -51,7 +51,7 @@
void *__bitmask = __get_rel_dynamic_array(field); \
unsigned int __bitmask_size; \
__bitmask_size = __get_rel_dynamic_array_len(field); \
- trace_print_bitmask_seq(p, __bitmask, __bitmask_size); \
+ trace_print_bitmask_seq(iter, __bitmask, __bitmask_size); \
})
#undef __get_rel_cpumask
diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c
index cc2d3306bb60..b312d67815fa 100644
--- a/kernel/trace/trace_output.c
+++ b/kernel/trace/trace_output.c
@@ -195,13 +195,18 @@ EXPORT_SYMBOL(trace_print_symbols_seq_u64);
#endif
const char *
-trace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr,
+trace_print_bitmask_seq(struct trace_iterator *iter, void *bitmask_ptr,
unsigned int bitmask_size)
{
- const char *ret = trace_seq_buffer_ptr(p);
-
- trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8);
- trace_seq_putc(p, 0);
+ struct trace_seq *s = &iter->seq;
+ const struct trace_array *tr = iter->tr;
+ const char *ret = trace_seq_buffer_ptr(s);
+ bool show_bitmask_list = tr->trace_flags &
+ TRACE_ITER(BITMASK_LIST);
+
+ trace_seq_bitmask(s, bitmask_ptr, bitmask_size * 8,
+ show_bitmask_list);
+ trace_seq_putc(s, 0);
return ret;
}
Kind regards,
--
Aaron Tomlin
On Tue, Dec 23, 2025 at 01:34:52PM -0500, Steven Rostedt wrote:
> On Mon, 22 Dec 2025 22:56:22 -0500
> Aaron Tomlin <atomlin@atomlin.com> wrote:
>
> > Add support for displaying bitmasks in human-readable list format (e.g.,
> > 0,2-5,7) in addition to the default hexadecimal bitmap representation.
> > This is particularly useful when tracing CPU masks and other large
> > bitmasks where individual bit positions are more meaningful than their
> > hexadecimal encoding.
> >
> > When the "bitmask-list" option is enabled, the printk "%*pbl" format
> > specifier is used to render bitmasks as comma-separated ranges, making
> > trace output easier to interpret for complex CPU configurations and
> > large bitmask values.
>
> Hmm, I have a couple of issues with this change. One, this is global. It
> affects all instances. The other is that if this is going to be done, then
> instead of adding a parameter to trace_seq_bitmask(), another trace_seq_*
> API should be created. Perhaps trace_seq_bitmask_cnt()? And have
> trace_print_bitmask_seq() call them separately.
>
> I'm still not convinced that this is needed. What examples do you see?
> Should it be only for CPU bitmasks?
>
> I think a bit more thought needs to be made on a change like this. There's
> other options that were added that I now regret. I don't want to add
> another one I wish we didn't have.
>
Hi Steve,
Thank you for the feedback.
Regarding the scope, I take your point that a global flag is perhaps too
blunt an instrument. I can see how making this instance-aware would be
preferable. I can look into refactoring this to ensure we pull the flags
from the specific trace_array associated with the current buffer instance
rather than the global state.
I agree with your suggestion to avoid polluting the existing
trace_seq_bitmask() signature. This would allow trace_print_bitmask_seq()
to branch between the two formats cleanly, as you suggested
As for the use case, I find this particularly invaluable when debugging IPI
latency or affinity issues on high-core-count systems. I typically run this
with the "nop" tracer enabled, focusing specifically on the
events/ipi/ipi_send_cpumask/ event.
When dealing with 128+ logical cores, interpreting a raw hexadecimal bitmap
to identify targeted CPUs is mentally taxing and prone to error. For
example, if I am investigating why CPU 6 is being interrupted, I might use
a filter such as "cpumask & CPU{6}". Seeing the resulting output as a range
list (e.g., 0-7) rather than a hexadecimal bitmask allows one to deduce
almost instantly which cluster of CPUs is involved in the IPI broadcast.
Kind regards,
--
Aaron Tomlin
On Tue, 23 Dec 2025 17:14:35 -0500
Aaron Tomlin <atomlin@atomlin.com> wrote:
> When dealing with 128+ logical cores, interpreting a raw hexadecimal bitmap
> to identify targeted CPUs is mentally taxing and prone to error. For
> example, if I am investigating why CPU 6 is being interrupted, I might use
> a filter such as "cpumask & CPU{6}". Seeing the resulting output as a range
> list (e.g., 0-7) rather than a hexadecimal bitmask allows one to deduce
> almost instantly which cluster of CPUs is involved in the IPI broadcast.
Should we just make all cpu bitmask range lists instead?
-- Steve
On Wed, Dec 24, 2025 at 08:58:48AM -0500, Steven Rostedt wrote: > Should we just make all cpu bitmask range lists instead? Hi Steve, I am somewhat hesitant to adopt that suggestion as I would prefer to avoid breaking any existing tooling that relies upon the default hexadecimal bitmask format. Whilst range lists are undoubtedly superior for human interpretation, the hexadecimal output is a well-established standard throughout the kernel. For instance, the hexadecimal format is still strictly adhered to for "Cpus_allowed:" within /proc/[pid]/status. Introducing a global change to ftrace defaults could disrupt parsers and scripts that expect this consistency across the system. By leveraging the existing bitmask-list trace option via trace_print_bitmask_seq(), we offer users the requisite flexibility for high-core-count systems whilst preserving backward compatibility for the wider ecosystem. I shall send a new version of the patch shortly. This version incorporates the use of iter->tmp_seq to ensure the implementation is robust, instance-aware, and free from buffer contention or duplication issues. Kind regards, -- Aaron Tomlin
On Thu, 25 Dec 2025 02:38:03 -0500 Aaron Tomlin <atomlin@atomlin.com> wrote: > On Wed, Dec 24, 2025 at 08:58:48AM -0500, Steven Rostedt wrote: > > Should we just make all cpu bitmask range lists instead? > > Hi Steve, > > I am somewhat hesitant to adopt that suggestion as I would prefer to avoid > breaking any existing tooling that relies upon the default hexadecimal > bitmask format. I am too. But the "do not break user space" rule is basically, "it's only broken if user space notices". If people complain about the change, we can always revert it ;-) > > Whilst range lists are undoubtedly superior for human interpretation, the > hexadecimal output is a well-established standard throughout the kernel. > For instance, the hexadecimal format is still strictly adhered to for > "Cpus_allowed:" within /proc/[pid]/status. Introducing a global change to > ftrace defaults could disrupt parsers and scripts that expect this > consistency across the system. Really, any scripts that parse the ASCII output is broken by design, as things change there all the time, and it can be really slow to read. There's a binary interface for such things. Heck, I bet this change would probably make the scripts simpler, as searching ranges is easier to parse than a hex number. > > By leveraging the existing bitmask-list trace option via > trace_print_bitmask_seq(), we offer users the requisite flexibility for > high-core-count systems whilst preserving backward compatibility for the > wider ecosystem. Perhaps it should only be cpumask-list, and only touch bitmasks that are CPU lists. Although, right now I only see one user of the bitmask code, and it's using it on a cpumask. Perhaps we should change it to use the cpumask. There's not many users of the bitmask in tracepoints, and it is mostly with the new IPI tracepoints (Cc'ing Valentin to get his throughts). > > I shall send a new version of the patch shortly. This version incorporates > the use of iter->tmp_seq to ensure the implementation is robust, > instance-aware, and free from buffer contention or duplication issues. Thanks, -- Steve
© 2016 - 2026 Red Hat, Inc.