Documentation/trace/index.rst | 1 Documentation/trace/wprobetrace.rst | 129 ++ arch/Kconfig | 10 arch/x86/Kconfig | 1 arch/x86/include/asm/hw_breakpoint.h | 3 arch/x86/kernel/hw_breakpoint.c | 61 + include/linux/hw_breakpoint.h | 6 include/linux/trace_events.h | 3 kernel/events/hw_breakpoint.c | 36 + kernel/trace/Kconfig | 24 kernel/trace/Makefile | 1 kernel/trace/trace.c | 9 kernel/trace/trace.h | 5 kernel/trace/trace_probe.c | 20 kernel/trace/trace_probe.h | 8 kernel/trace/trace_wprobe.c | 1111 ++++++++++++++++++++ .../ftrace/test.d/dynevent/add_remove_wprobe.tc | 68 + .../test.d/dynevent/wprobes_syntax_errors.tc | 20 18 files changed, 1513 insertions(+), 3 deletions(-) create mode 100644 Documentation/trace/wprobetrace.rst create mode 100644 kernel/trace/trace_wprobe.c create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc
Hi,
Here is an RFC series for adding new wprobe (watch probe) which
provides memory access tracing event. Moreover, this can be used via
event trigger. Thus it can trace memory access on a dynamically
allocated objects too.
In this version, I reuse Jinchao's arch_reinstall_hw_breakpoint()
patch[1].
[1] https://lore.kernel.org/all/20250828073311.1116593-6-wangjinchao600@gmail.com/
The basic usage of this wprobe is similar to other probes;
w:[GRP/][EVENT] [r|w|rw]@<ADDRESS|SYMBOL[+OFFS]> [FETCHARGS]
This defines a new wprobe event. For example, to trace jiffies update,
you can do;
echo 'w:my_jiffies w@jiffies:8 value=+0($addr)' >> dynamic_events
echo 1 > events/wprobes/my_jiffies/enable
Moreover, this can be combined with event trigger to trace the memory
accecss on slab objects. The trigger syntax is;
set_wprobe:WPROBE_EVENT:FIELD[+ADJUST]
clear_wprobe:WPROBE_EVENT
For example, trace the first 8 byte of the dentry data structure passed
to do_truncate() until it is deleted by __dentry_kill().
(Note: all tracefs setup uses '>>' so that it does not kick do_truncate())
# echo 'w:watch rw@0:8 address=$addr value=+0($addr)' > dynamic_events
# echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
# echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
# echo 'f:dentry_kill __dentry_kill dentry=$arg1' >> dynamic_events
# echo 'clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger
# echo 1 >> events/fprobes/truncate/enable
# echo 1 >> events/fprobes/dentry_kill/enable
# echo aaa > /tmp/hoge
# echo bbb > /tmp/hoge
# echo ccc > /tmp/hoge
# rm /tmp/hoge
Then, the trace data will show;
# tracer: nop
#
# entries-in-buffer/entries-written: 16/16 #P:8
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# [ 7.026136] sh (113) used greatest stack depth: 12912 bytes left
| | | ||||| | |
sh-113 [002] ..... 7.024402: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8
sh-113 [002] ..Zff 7.024822: watch: (lookup_fast+0xaa/0x150) address=0xffff8880069194b8 value=0x200008
sh-113 [002] ..Zff 7.024830: watch: (step_into+0x82/0x360) address=0xffff8880069194b8 value=0x200008
sh-113 [002] ..Zff 7.024834: watch: (step_into+0x9f/0x360) address=0xffff8880069194b8 value=0x200008
sh-113 [002] ..Zff 7.024839: watch: (path_openat+0xb3a/0xe70) address=0xffff8880069194b8 value=0x200008
sh-113 [002] ..Zff 7.024843: watch: (path_openat+0xb9a/0xe70) address=0xffff8880069194b8 value=0x200008
sh-113 [002] ..... 7.024847: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8
sh-113 [002] ...1. 7.025364: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888006919380
sh-113 [002] ...1. 7.025511: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069195f0
rm-118 [003] ...1. 7.027543: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069194b8
sh-113 [002] ...2. 7.027825: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880044429c0
sh-113 [002] ...2. 7.027833: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888004442270
Thank you,
---
Jinchao Wang (1):
x86/HWBP: introduce arch_reinstall_hw_breakpoint() for atomic context
Masami Hiramatsu (Google) (5):
tracing: wprobe: Add watchpoint probe event based on hardware breakpoint
HWBP: Add modify_wide_hw_breakpoint_local() API
tracing: wprobe: Add wprobe event trigger
selftests: tracing: Add a basic testcase for wprobe
selftests: tracing: Add syntax testcase for wprobe
Documentation/trace/index.rst | 1
Documentation/trace/wprobetrace.rst | 129 ++
arch/Kconfig | 10
arch/x86/Kconfig | 1
arch/x86/include/asm/hw_breakpoint.h | 3
arch/x86/kernel/hw_breakpoint.c | 61 +
include/linux/hw_breakpoint.h | 6
include/linux/trace_events.h | 3
kernel/events/hw_breakpoint.c | 36 +
kernel/trace/Kconfig | 24
kernel/trace/Makefile | 1
kernel/trace/trace.c | 9
kernel/trace/trace.h | 5
kernel/trace/trace_probe.c | 20
kernel/trace/trace_probe.h | 8
kernel/trace/trace_wprobe.c | 1111 ++++++++++++++++++++
.../ftrace/test.d/dynevent/add_remove_wprobe.tc | 68 +
.../test.d/dynevent/wprobes_syntax_errors.tc | 20
18 files changed, 1513 insertions(+), 3 deletions(-)
create mode 100644 Documentation/trace/wprobetrace.rst
create mode 100644 kernel/trace/trace_wprobe.c
create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
(Adding Jinchao) On Mon, 1 Sep 2025 23:44:35 +0900 "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: > Hi, > > Here is an RFC series for adding new wprobe (watch probe) which > provides memory access tracing event. Moreover, this can be used via > event trigger. Thus it can trace memory access on a dynamically > allocated objects too. BTW, this series is on the top of probes/for-next branch in the linux-trace tree. Thanks, > > In this version, I reuse Jinchao's arch_reinstall_hw_breakpoint() > patch[1]. > > [1] https://lore.kernel.org/all/20250828073311.1116593-6-wangjinchao600@gmail.com/ > > The basic usage of this wprobe is similar to other probes; > > w:[GRP/][EVENT] [r|w|rw]@<ADDRESS|SYMBOL[+OFFS]> [FETCHARGS] > > This defines a new wprobe event. For example, to trace jiffies update, > you can do; > > echo 'w:my_jiffies w@jiffies:8 value=+0($addr)' >> dynamic_events > echo 1 > events/wprobes/my_jiffies/enable > > Moreover, this can be combined with event trigger to trace the memory > accecss on slab objects. The trigger syntax is; > > set_wprobe:WPROBE_EVENT:FIELD[+ADJUST] > clear_wprobe:WPROBE_EVENT > > For example, trace the first 8 byte of the dentry data structure passed > to do_truncate() until it is deleted by __dentry_kill(). > (Note: all tracefs setup uses '>>' so that it does not kick do_truncate()) > > # echo 'w:watch rw@0:8 address=$addr value=+0($addr)' > dynamic_events > > # echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events > # echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger > > # echo 'f:dentry_kill __dentry_kill dentry=$arg1' >> dynamic_events > # echo 'clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger > > # echo 1 >> events/fprobes/truncate/enable > # echo 1 >> events/fprobes/dentry_kill/enable > > # echo aaa > /tmp/hoge > # echo bbb > /tmp/hoge > # echo ccc > /tmp/hoge > # rm /tmp/hoge > > Then, the trace data will show; > > # tracer: nop > # > # entries-in-buffer/entries-written: 16/16 #P:8 > # > # _-----=> irqs-off/BH-disabled > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / _-=> migrate-disable > # |||| / delay > # TASK-PID CPU# ||||| TIMESTAMP FUNCTION > # [ 7.026136] sh (113) used greatest stack depth: 12912 bytes left > | | | ||||| | | > sh-113 [002] ..... 7.024402: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8 > sh-113 [002] ..Zff 7.024822: watch: (lookup_fast+0xaa/0x150) address=0xffff8880069194b8 value=0x200008 > sh-113 [002] ..Zff 7.024830: watch: (step_into+0x82/0x360) address=0xffff8880069194b8 value=0x200008 > sh-113 [002] ..Zff 7.024834: watch: (step_into+0x9f/0x360) address=0xffff8880069194b8 value=0x200008 > sh-113 [002] ..Zff 7.024839: watch: (path_openat+0xb3a/0xe70) address=0xffff8880069194b8 value=0x200008 > sh-113 [002] ..Zff 7.024843: watch: (path_openat+0xb9a/0xe70) address=0xffff8880069194b8 value=0x200008 > sh-113 [002] ..... 7.024847: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8 > sh-113 [002] ...1. 7.025364: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888006919380 > sh-113 [002] ...1. 7.025511: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069195f0 > rm-118 [003] ...1. 7.027543: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069194b8 > sh-113 [002] ...2. 7.027825: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880044429c0 > sh-113 [002] ...2. 7.027833: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888004442270 > > > Thank you, > > --- > > Jinchao Wang (1): > x86/HWBP: introduce arch_reinstall_hw_breakpoint() for atomic context > > Masami Hiramatsu (Google) (5): > tracing: wprobe: Add watchpoint probe event based on hardware breakpoint > HWBP: Add modify_wide_hw_breakpoint_local() API > tracing: wprobe: Add wprobe event trigger > selftests: tracing: Add a basic testcase for wprobe > selftests: tracing: Add syntax testcase for wprobe > > > Documentation/trace/index.rst | 1 > Documentation/trace/wprobetrace.rst | 129 ++ > arch/Kconfig | 10 > arch/x86/Kconfig | 1 > arch/x86/include/asm/hw_breakpoint.h | 3 > arch/x86/kernel/hw_breakpoint.c | 61 + > include/linux/hw_breakpoint.h | 6 > include/linux/trace_events.h | 3 > kernel/events/hw_breakpoint.c | 36 + > kernel/trace/Kconfig | 24 > kernel/trace/Makefile | 1 > kernel/trace/trace.c | 9 > kernel/trace/trace.h | 5 > kernel/trace/trace_probe.c | 20 > kernel/trace/trace_probe.h | 8 > kernel/trace/trace_wprobe.c | 1111 ++++++++++++++++++++ > .../ftrace/test.d/dynevent/add_remove_wprobe.tc | 68 + > .../test.d/dynevent/wprobes_syntax_errors.tc | 20 > 18 files changed, 1513 insertions(+), 3 deletions(-) > create mode 100644 Documentation/trace/wprobetrace.rst > create mode 100644 kernel/trace/trace_wprobe.c > create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc > create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc > > -- > Masami Hiramatsu (Google) <mhiramat@kernel.org> -- Masami Hiramatsu (Google) <mhiramat@kernel.org>
On 9/2/25 22:02, Masami Hiramatsu (Google) wrote:
> (Adding Jinchao)
>
> On Mon, 1 Sep 2025 23:44:35 +0900
> "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote:
>
>> Hi,
>>
>> Here is an RFC series for adding new wprobe (watch probe) which
>> provides memory access tracing event. Moreover, this can be used via
>> event trigger. Thus it can trace memory access on a dynamically
>> allocated objects too.
>
> BTW, this series is on the top of probes/for-next branch in the
> linux-trace tree.
>
Hi, Masami
Thanks for including me. I only received the cover letter, so I'm
providing my feedback here:
- trigger_data issue:
it appears that the instance being removed is not the same as
the one that was originally set.
- the function call issue:
`trace_wprobe_update_local()` is called twice, once in the trigger
callback and again in `wprobe_work_func`.
I also noticed that the Watchpoint probe and KStackWatch implementations
share very similar logic for managing hardware breakpoints/watchpoints
(`hwbp/watch`):
- `watch_init(unsigned long &place_holder)`
- `watch_on(struct perf_event_attr *attr)`
- `watch_off()` (or reset to the `place_holder` value)
- `watch_uninit()`
Their primary difference lies in their handler functions, specifically
the `perf_overflow_handler_t triggered` callback.
I believe we could work together to unify this logic. I am open to
either approach: I can refactor my watch.c, or you can introduce new
helpers. This would help us save duplicated work and review time.
> Thanks,
>
>>
>> In this version, I reuse Jinchao's arch_reinstall_hw_breakpoint()
>> patch[1].
>>
>> [1] https://lore.kernel.org/all/20250828073311.1116593-6-wangjinchao600@gmail.com/
>>
>> The basic usage of this wprobe is similar to other probes;
>>
>> w:[GRP/][EVENT] [r|w|rw]@<ADDRESS|SYMBOL[+OFFS]> [FETCHARGS]
>>
>> This defines a new wprobe event. For example, to trace jiffies update,
>> you can do;
>>
>> echo 'w:my_jiffies w@jiffies:8 value=+0($addr)' >> dynamic_events
>> echo 1 > events/wprobes/my_jiffies/enable
>>
>> Moreover, this can be combined with event trigger to trace the memory
>> accecss on slab objects. The trigger syntax is;
>>
>> set_wprobe:WPROBE_EVENT:FIELD[+ADJUST]
>> clear_wprobe:WPROBE_EVENT
>>
>> For example, trace the first 8 byte of the dentry data structure passed
>> to do_truncate() until it is deleted by __dentry_kill().
>> (Note: all tracefs setup uses '>>' so that it does not kick do_truncate())
>>
>> # echo 'w:watch rw@0:8 address=$addr value=+0($addr)' > dynamic_events
>>
>> # echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
>> # echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
>>
>> # echo 'f:dentry_kill __dentry_kill dentry=$arg1' >> dynamic_events
>> # echo 'clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger
>>
>> # echo 1 >> events/fprobes/truncate/enable
>> # echo 1 >> events/fprobes/dentry_kill/enable
>>
>> # echo aaa > /tmp/hoge
>> # echo bbb > /tmp/hoge
>> # echo ccc > /tmp/hoge
>> # rm /tmp/hoge
>>
>> Then, the trace data will show;
>>
>> # tracer: nop
>> #
>> # entries-in-buffer/entries-written: 16/16 #P:8
>> #
>> # _-----=> irqs-off/BH-disabled
>> # / _----=> need-resched
>> # | / _---=> hardirq/softirq
>> # || / _--=> preempt-depth
>> # ||| / _-=> migrate-disable
>> # |||| / delay
>> # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
>> # [ 7.026136] sh (113) used greatest stack depth: 12912 bytes left
>> | | | ||||| | |
>> sh-113 [002] ..... 7.024402: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8
>> sh-113 [002] ..Zff 7.024822: watch: (lookup_fast+0xaa/0x150) address=0xffff8880069194b8 value=0x200008
>> sh-113 [002] ..Zff 7.024830: watch: (step_into+0x82/0x360) address=0xffff8880069194b8 value=0x200008
>> sh-113 [002] ..Zff 7.024834: watch: (step_into+0x9f/0x360) address=0xffff8880069194b8 value=0x200008
>> sh-113 [002] ..Zff 7.024839: watch: (path_openat+0xb3a/0xe70) address=0xffff8880069194b8 value=0x200008
>> sh-113 [002] ..Zff 7.024843: watch: (path_openat+0xb9a/0xe70) address=0xffff8880069194b8 value=0x200008
>> sh-113 [002] ..... 7.024847: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8
>> sh-113 [002] ...1. 7.025364: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888006919380
>> sh-113 [002] ...1. 7.025511: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069195f0
>> rm-118 [003] ...1. 7.027543: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069194b8
>> sh-113 [002] ...2. 7.027825: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880044429c0
>> sh-113 [002] ...2. 7.027833: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888004442270
>>
>>
>> Thank you,
>>
>> ---
>>
>> Jinchao Wang (1):
>> x86/HWBP: introduce arch_reinstall_hw_breakpoint() for atomic context
>>
>> Masami Hiramatsu (Google) (5):
>> tracing: wprobe: Add watchpoint probe event based on hardware breakpoint
>> HWBP: Add modify_wide_hw_breakpoint_local() API
>> tracing: wprobe: Add wprobe event trigger
>> selftests: tracing: Add a basic testcase for wprobe
>> selftests: tracing: Add syntax testcase for wprobe
>>
>>
>> Documentation/trace/index.rst | 1
>> Documentation/trace/wprobetrace.rst | 129 ++
>> arch/Kconfig | 10
>> arch/x86/Kconfig | 1
>> arch/x86/include/asm/hw_breakpoint.h | 3
>> arch/x86/kernel/hw_breakpoint.c | 61 +
>> include/linux/hw_breakpoint.h | 6
>> include/linux/trace_events.h | 3
>> kernel/events/hw_breakpoint.c | 36 +
>> kernel/trace/Kconfig | 24
>> kernel/trace/Makefile | 1
>> kernel/trace/trace.c | 9
>> kernel/trace/trace.h | 5
>> kernel/trace/trace_probe.c | 20
>> kernel/trace/trace_probe.h | 8
>> kernel/trace/trace_wprobe.c | 1111 ++++++++++++++++++++
>> .../ftrace/test.d/dynevent/add_remove_wprobe.tc | 68 +
>> .../test.d/dynevent/wprobes_syntax_errors.tc | 20
>> 18 files changed, 1513 insertions(+), 3 deletions(-)
>> create mode 100644 Documentation/trace/wprobetrace.rst
>> create mode 100644 kernel/trace/trace_wprobe.c
>> create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
>> create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc
>>
>> --
>> Masami Hiramatsu (Google) <mhiramat@kernel.org>
>
>
--
Best regards,
Jinchao
On Thu, 4 Sep 2025 14:35:33 +0800 Jinchao Wang <wangjinchao600@gmail.com> wrote: > On 9/2/25 22:02, Masami Hiramatsu (Google) wrote: > > (Adding Jinchao) > > > > On Mon, 1 Sep 2025 23:44:35 +0900 > > "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: > > > >> Hi, > >> > >> Here is an RFC series for adding new wprobe (watch probe) which > >> provides memory access tracing event. Moreover, this can be used via > >> event trigger. Thus it can trace memory access on a dynamically > >> allocated objects too. > > > > BTW, this series is on the top of probes/for-next branch in the > > linux-trace tree. > > > > Hi, Masami > > Thanks for including me. I only received the cover letter, so I'm > providing my feedback here: Thanks for your feedback! I'll send v2. > > - trigger_data issue: > it appears that the instance being removed is not the same as > the one that was originally set. Did you mean it will free invalid struct event_trigger_data by wprobe_trigger_free(), or hw_breakpoint instance? When trace_event_enable_disable(wprobe_data->file, 1, 1) at wprobe_trigger_cmd_parse(), the trace_wprobe is enabled == call enable_trace_wprobe(), which eventually call register_wide_hw_breakpoint() with dummy address. IOW, the (wide)hw_breakpoint is allocated here. When the trigger is hit, modify_wide_hw_breakpoint_local() is called for updating local target address on the same hw_breakpoint. So latter instance should be the same. > > - the function call issue: > `trace_wprobe_update_local()` is called twice, once in the trigger > callback and again in `wprobe_work_func`. Yes, I found it can happen. Maybe I need a cpumask to skip the first one. > I also noticed that the Watchpoint probe and KStackWatch implementations > share very similar logic for managing hardware breakpoints/watchpoints > (`hwbp/watch`): > - `watch_init(unsigned long &place_holder)` > - `watch_on(struct perf_event_attr *attr)` > - `watch_off()` (or reset to the `place_holder` value) > - `watch_uninit()` Hmm, I agree that we are using hwbp in a similar way but I think current interface is enough. > > Their primary difference lies in their handler functions, specifically > the `perf_overflow_handler_t triggered` callback. > > I believe we could work together to unify this logic. I am open to > either approach: I can refactor my watch.c, or you can introduce new > helpers. This would help us save duplicated work and review time. I'm not sure what you mean. Is there any concern to use the same (current hwbp) interfaces? Thank you, > > > Thanks, > > > >> > >> In this version, I reuse Jinchao's arch_reinstall_hw_breakpoint() > >> patch[1]. > >> > >> [1] https://lore.kernel.org/all/20250828073311.1116593-6-wangjinchao600@gmail.com/ > >> > >> The basic usage of this wprobe is similar to other probes; > >> > >> w:[GRP/][EVENT] [r|w|rw]@<ADDRESS|SYMBOL[+OFFS]> [FETCHARGS] > >> > >> This defines a new wprobe event. For example, to trace jiffies update, > >> you can do; > >> > >> echo 'w:my_jiffies w@jiffies:8 value=+0($addr)' >> dynamic_events > >> echo 1 > events/wprobes/my_jiffies/enable > >> > >> Moreover, this can be combined with event trigger to trace the memory > >> accecss on slab objects. The trigger syntax is; > >> > >> set_wprobe:WPROBE_EVENT:FIELD[+ADJUST] > >> clear_wprobe:WPROBE_EVENT > >> > >> For example, trace the first 8 byte of the dentry data structure passed > >> to do_truncate() until it is deleted by __dentry_kill(). > >> (Note: all tracefs setup uses '>>' so that it does not kick do_truncate()) > >> > >> # echo 'w:watch rw@0:8 address=$addr value=+0($addr)' > dynamic_events > >> > >> # echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events > >> # echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger > >> > >> # echo 'f:dentry_kill __dentry_kill dentry=$arg1' >> dynamic_events > >> # echo 'clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger > >> > >> # echo 1 >> events/fprobes/truncate/enable > >> # echo 1 >> events/fprobes/dentry_kill/enable > >> > >> # echo aaa > /tmp/hoge > >> # echo bbb > /tmp/hoge > >> # echo ccc > /tmp/hoge > >> # rm /tmp/hoge > >> > >> Then, the trace data will show; > >> > >> # tracer: nop > >> # > >> # entries-in-buffer/entries-written: 16/16 #P:8 > >> # > >> # _-----=> irqs-off/BH-disabled > >> # / _----=> need-resched > >> # | / _---=> hardirq/softirq > >> # || / _--=> preempt-depth > >> # ||| / _-=> migrate-disable > >> # |||| / delay > >> # TASK-PID CPU# ||||| TIMESTAMP FUNCTION > >> # [ 7.026136] sh (113) used greatest stack depth: 12912 bytes left > >> | | | ||||| | | > >> sh-113 [002] ..... 7.024402: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8 > >> sh-113 [002] ..Zff 7.024822: watch: (lookup_fast+0xaa/0x150) address=0xffff8880069194b8 value=0x200008 > >> sh-113 [002] ..Zff 7.024830: watch: (step_into+0x82/0x360) address=0xffff8880069194b8 value=0x200008 > >> sh-113 [002] ..Zff 7.024834: watch: (step_into+0x9f/0x360) address=0xffff8880069194b8 value=0x200008 > >> sh-113 [002] ..Zff 7.024839: watch: (path_openat+0xb3a/0xe70) address=0xffff8880069194b8 value=0x200008 > >> sh-113 [002] ..Zff 7.024843: watch: (path_openat+0xb9a/0xe70) address=0xffff8880069194b8 value=0x200008 > >> sh-113 [002] ..... 7.024847: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8 > >> sh-113 [002] ...1. 7.025364: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888006919380 > >> sh-113 [002] ...1. 7.025511: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069195f0 > >> rm-118 [003] ...1. 7.027543: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069194b8 > >> sh-113 [002] ...2. 7.027825: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880044429c0 > >> sh-113 [002] ...2. 7.027833: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888004442270 > >> > >> > >> Thank you, > >> > >> --- > >> > >> Jinchao Wang (1): > >> x86/HWBP: introduce arch_reinstall_hw_breakpoint() for atomic context > >> > >> Masami Hiramatsu (Google) (5): > >> tracing: wprobe: Add watchpoint probe event based on hardware breakpoint > >> HWBP: Add modify_wide_hw_breakpoint_local() API > >> tracing: wprobe: Add wprobe event trigger > >> selftests: tracing: Add a basic testcase for wprobe > >> selftests: tracing: Add syntax testcase for wprobe > >> > >> > >> Documentation/trace/index.rst | 1 > >> Documentation/trace/wprobetrace.rst | 129 ++ > >> arch/Kconfig | 10 > >> arch/x86/Kconfig | 1 > >> arch/x86/include/asm/hw_breakpoint.h | 3 > >> arch/x86/kernel/hw_breakpoint.c | 61 + > >> include/linux/hw_breakpoint.h | 6 > >> include/linux/trace_events.h | 3 > >> kernel/events/hw_breakpoint.c | 36 + > >> kernel/trace/Kconfig | 24 > >> kernel/trace/Makefile | 1 > >> kernel/trace/trace.c | 9 > >> kernel/trace/trace.h | 5 > >> kernel/trace/trace_probe.c | 20 > >> kernel/trace/trace_probe.h | 8 > >> kernel/trace/trace_wprobe.c | 1111 ++++++++++++++++++++ > >> .../ftrace/test.d/dynevent/add_remove_wprobe.tc | 68 + > >> .../test.d/dynevent/wprobes_syntax_errors.tc | 20 > >> 18 files changed, 1513 insertions(+), 3 deletions(-) > >> create mode 100644 Documentation/trace/wprobetrace.rst > >> create mode 100644 kernel/trace/trace_wprobe.c > >> create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc > >> create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc > >> > >> -- > >> Masami Hiramatsu (Google) <mhiramat@kernel.org> > > > > > > > -- > Best regards, > Jinchao -- Masami Hiramatsu (Google) <mhiramat@kernel.org>
On 9/5/25 10:08, Masami Hiramatsu (Google) wrote: > On Thu, 4 Sep 2025 14:35:33 +0800 > Jinchao Wang <wangjinchao600@gmail.com> wrote: > >> On 9/2/25 22:02, Masami Hiramatsu (Google) wrote: >>> (Adding Jinchao) >>> >>> On Mon, 1 Sep 2025 23:44:35 +0900 >>> "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: >>> >>>> Hi, >>>> >>>> Here is an RFC series for adding new wprobe (watch probe) which >>>> provides memory access tracing event. Moreover, this can be used via >>>> event trigger. Thus it can trace memory access on a dynamically >>>> allocated objects too. >>> >>> BTW, this series is on the top of probes/for-next branch in the >>> linux-trace tree. >>> >> >> Hi, Masami >> >> Thanks for including me. I only received the cover letter, so I'm >> providing my feedback here: > > Thanks for your feedback! I'll send v2. > >> >> - trigger_data issue: >> it appears that the instance being removed is not the same as >> the one that was originally set. > > Did you mean it will free invalid struct event_trigger_data by > wprobe_trigger_free(), or hw_breakpoint instance? > > When trace_event_enable_disable(wprobe_data->file, 1, 1) at > wprobe_trigger_cmd_parse(), the trace_wprobe is enabled == call > enable_trace_wprobe(), which eventually call register_wide_hw_breakpoint() > with dummy address. IOW, the (wide)hw_breakpoint is allocated here. > > When the trigger is hit, modify_wide_hw_breakpoint_local() is called > for updating local target address on the same hw_breakpoint. > So latter instance should be the same. In wprobe_trigger_cmd_parse(), consider removing: echo '!set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger The trigger_data passed to event_trigger_unregister() is freshly allocated. > >> >> - the function call issue: >> `trace_wprobe_update_local()` is called twice, once in the trigger >> callback and again in `wprobe_work_func`. > > Yes, I found it can happen. Maybe I need a cpumask to skip the first one. > >> I also noticed that the Watchpoint probe and KStackWatch implementations >> share very similar logic for managing hardware breakpoints/watchpoints >> (`hwbp/watch`): >> - `watch_init(unsigned long &place_holder)` >> - `watch_on(struct perf_event_attr *attr)` >> - `watch_off()` (or reset to the `place_holder` value) >> - `watch_uninit()` > > Hmm, I agree that we are using hwbp in a similar way but > I think current interface is enough. > >> >> Their primary difference lies in their handler functions, specifically >> the `perf_overflow_handler_t triggered` callback. >> >> I believe we could work together to unify this logic. I am open to >> either approach: I can refactor my watch.c, or you can introduce new >> helpers. This would help us save duplicated work and review time. > > I'm not sure what you mean. Is there any concern to use the same > (current hwbp) interfaces? > It is not the _current_ HWBP interface, it needs review. We will write the same logic except for the HWBP interface regarding atomic context watch switch (addr/type/len) and the process to sync with other CPUs. The only difference is the HWBP handler. I am not familiar with the process of how we usually do roughly the same thing. Maybe I am thinking about it the wrong way; I will look at it later. > Thank you, > >> >>> Thanks, >>> >>>> >>>> In this version, I reuse Jinchao's arch_reinstall_hw_breakpoint() >>>> patch[1]. >>>> >>>> [1] https://lore.kernel.org/all/20250828073311.1116593-6-wangjinchao600@gmail.com/ >>>> >>>> The basic usage of this wprobe is similar to other probes; >>>> >>>> w:[GRP/][EVENT] [r|w|rw]@<ADDRESS|SYMBOL[+OFFS]> [FETCHARGS] >>>> >>>> This defines a new wprobe event. For example, to trace jiffies update, >>>> you can do; >>>> >>>> echo 'w:my_jiffies w@jiffies:8 value=+0($addr)' >> dynamic_events >>>> echo 1 > events/wprobes/my_jiffies/enable >>>> >>>> Moreover, this can be combined with event trigger to trace the memory >>>> accecss on slab objects. The trigger syntax is; >>>> >>>> set_wprobe:WPROBE_EVENT:FIELD[+ADJUST] >>>> clear_wprobe:WPROBE_EVENT >>>> >>>> For example, trace the first 8 byte of the dentry data structure passed >>>> to do_truncate() until it is deleted by __dentry_kill(). >>>> (Note: all tracefs setup uses '>>' so that it does not kick do_truncate()) >>>> >>>> # echo 'w:watch rw@0:8 address=$addr value=+0($addr)' > dynamic_events >>>> >>>> # echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events >>>> # echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger >>>> >>>> # echo 'f:dentry_kill __dentry_kill dentry=$arg1' >> dynamic_events >>>> # echo 'clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger >>>> >>>> # echo 1 >> events/fprobes/truncate/enable >>>> # echo 1 >> events/fprobes/dentry_kill/enable >>>> >>>> # echo aaa > /tmp/hoge >>>> # echo bbb > /tmp/hoge >>>> # echo ccc > /tmp/hoge >>>> # rm /tmp/hoge >>>> >>>> Then, the trace data will show; >>>> >>>> # tracer: nop >>>> # >>>> # entries-in-buffer/entries-written: 16/16 #P:8 >>>> # >>>> # _-----=> irqs-off/BH-disabled >>>> # / _----=> need-resched >>>> # | / _---=> hardirq/softirq >>>> # || / _--=> preempt-depth >>>> # ||| / _-=> migrate-disable >>>> # |||| / delay >>>> # TASK-PID CPU# ||||| TIMESTAMP FUNCTION >>>> # [ 7.026136] sh (113) used greatest stack depth: 12912 bytes left >>>> | | | ||||| | | >>>> sh-113 [002] ..... 7.024402: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8 >>>> sh-113 [002] ..Zff 7.024822: watch: (lookup_fast+0xaa/0x150) address=0xffff8880069194b8 value=0x200008 >>>> sh-113 [002] ..Zff 7.024830: watch: (step_into+0x82/0x360) address=0xffff8880069194b8 value=0x200008 >>>> sh-113 [002] ..Zff 7.024834: watch: (step_into+0x9f/0x360) address=0xffff8880069194b8 value=0x200008 >>>> sh-113 [002] ..Zff 7.024839: watch: (path_openat+0xb3a/0xe70) address=0xffff8880069194b8 value=0x200008 >>>> sh-113 [002] ..Zff 7.024843: watch: (path_openat+0xb9a/0xe70) address=0xffff8880069194b8 value=0x200008 >>>> sh-113 [002] ..... 7.024847: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880069194b8 >>>> sh-113 [002] ...1. 7.025364: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888006919380 >>>> sh-113 [002] ...1. 7.025511: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069195f0 >>>> rm-118 [003] ...1. 7.027543: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880069194b8 >>>> sh-113 [002] ...2. 7.027825: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff8880044429c0 >>>> sh-113 [002] ...2. 7.027833: dentry_kill: (__dentry_kill+0x0/0x220) dentry=0xffff888004442270 >>>> >>>> >>>> Thank you, >>>> >>>> --- >>>> >>>> Jinchao Wang (1): >>>> x86/HWBP: introduce arch_reinstall_hw_breakpoint() for atomic context >>>> >>>> Masami Hiramatsu (Google) (5): >>>> tracing: wprobe: Add watchpoint probe event based on hardware breakpoint >>>> HWBP: Add modify_wide_hw_breakpoint_local() API >>>> tracing: wprobe: Add wprobe event trigger >>>> selftests: tracing: Add a basic testcase for wprobe >>>> selftests: tracing: Add syntax testcase for wprobe >>>> >>>> >>>> Documentation/trace/index.rst | 1 >>>> Documentation/trace/wprobetrace.rst | 129 ++ >>>> arch/Kconfig | 10 >>>> arch/x86/Kconfig | 1 >>>> arch/x86/include/asm/hw_breakpoint.h | 3 >>>> arch/x86/kernel/hw_breakpoint.c | 61 + >>>> include/linux/hw_breakpoint.h | 6 >>>> include/linux/trace_events.h | 3 >>>> kernel/events/hw_breakpoint.c | 36 + >>>> kernel/trace/Kconfig | 24 >>>> kernel/trace/Makefile | 1 >>>> kernel/trace/trace.c | 9 >>>> kernel/trace/trace.h | 5 >>>> kernel/trace/trace_probe.c | 20 >>>> kernel/trace/trace_probe.h | 8 >>>> kernel/trace/trace_wprobe.c | 1111 ++++++++++++++++++++ >>>> .../ftrace/test.d/dynevent/add_remove_wprobe.tc | 68 + >>>> .../test.d/dynevent/wprobes_syntax_errors.tc | 20 >>>> 18 files changed, 1513 insertions(+), 3 deletions(-) >>>> create mode 100644 Documentation/trace/wprobetrace.rst >>>> create mode 100644 kernel/trace/trace_wprobe.c >>>> create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc >>>> create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc >>>> >>>> -- >>>> Masami Hiramatsu (Google) <mhiramat@kernel.org> >>> >>> >> >> >> -- >> Best regards, >> Jinchao > > -- Best regards, Jinchao
> >> - trigger_data issue: > >> it appears that the instance being removed is not the same as > >> the one that was originally set. > > > > Did you mean it will free invalid struct event_trigger_data by > > wprobe_trigger_free(), or hw_breakpoint instance? > > > > When trace_event_enable_disable(wprobe_data->file, 1, 1) at > > wprobe_trigger_cmd_parse(), the trace_wprobe is enabled == call > > enable_trace_wprobe(), which eventually call register_wide_hw_breakpoint() > > with dummy address. IOW, the (wide)hw_breakpoint is allocated here. > > > > When the trigger is hit, modify_wide_hw_breakpoint_local() is called > > for updating local target address on the same hw_breakpoint. > > So latter instance should be the same. > In wprobe_trigger_cmd_parse(), consider removing: > echo '!set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger > The trigger_data passed to event_trigger_unregister() is freshly allocated. Ah, thanks for pointing it out. I confused to free newly allocated trigger_data instead of existing one on the list. Let me fix that. Thank you, -- Masami Hiramatsu (Google) <mhiramat@kernel.org>
© 2016 - 2025 Red Hat, Inc.