Hi,
Arnaldo reported that "perf test sigtrap" fails on PREEMPT_RT. Sending
the signal gets delayed until event_sched_out() which then uses
task_work_add() for its delivery. This breaks on PREEMPT_RT because the
signal is delivered with disabled preemption.
While looking at this, I also stumbled upon __perf_pending_irq() which
requires disabled interrupts but this is not the case on PREEMPT_RT.
This series aim to address both issues while not introducing a new issue
at the same time ;)
Any testing is appreciated.
v4…v5: https://lore.kernel.org/all/20240624152732.1231678-1-bigeasy@linutronix.de/
- Add TWA_NMI_CURRENT as notify mode for task_work_add() and use it.
PeterZ pointed out that the current version is not NMI safe.
v3…v4: https://lore.kernel.org/all/20240322065208.60456-1-bigeasy@linutronix.de/
- Rebased on top of Frederic's series
(https://lore.kernel.org/all/20240621091601.18227-1-frederic@kernel.org)
- Frederick pointed out that perf_pending_task() needs to
perf_swevent_get_recursion_context() in order not to recurse if
something within perf_swevent_.*_recursion_context() triggers a
software event. To address this, the counters have been moved to
the task_struct (#3 + #4) and preemt_disable() has been replaced
with a RCU-read lock (#5).
- The remaning logic same that means the event is pushed to task-work
instead of delivering from IRQ-work. The series was tested with
remove_on_exec as suggested by Marco Elver: On PREEMPT_RT a single
invocation passes, 100 parallel invocations report (for some)
unexpected SIGTRAPs and timeouts. This also observed on !RT
(without the series) with a higher task-count.
v2…v3: https://lore.kernel.org/all/20240312180814.3373778-1-bigeasy@linutronix.de/
- Marco suggested to add a few comments
- Added a comment to __perf_event_overflow() to explain why irq_work
is raised in the in_nmi() case.
- Added a comment to perf_event_exit_event() to explain why the
pending event is deleted.
v1…v2: https://lore.kernel.org/all/20240308175810.2894694-1-bigeasy@linutronix.de/
- Marco pointed me to the testsuite that showed two problems:
- Delayed task_work from NMI / missing events.
Fixed by triggering dummy irq_work to enforce an interrupt for
the exit-to-userland path which checks task_work
- Increased ref-count on clean up/ during exec.
Mostly addressed by the former change. There is still a window
if the NMI occurs during execve(). This is addressed by removing
the task_work before free_event().
The testsuite (remove_on_exec) fails sometimes if the event/
SIGTRAP is sent before the sighandler is installed.
Sebastian
Thanks, I'll go queue it for perf/core once the robot gets back to me on the current pile.
On 2024-07-05 09:55:11 [+0200], Peter Zijlstra wrote: > > > Thanks, I'll go queue it for perf/core once the robot gets back to me on > the current pile. Thank you. Sebastian
On Thu, Jul 04, 2024 at 07:03:34PM +0200, Sebastian Andrzej Siewior wrote: > Hi, > > Arnaldo reported that "perf test sigtrap" fails on PREEMPT_RT. Sending > the signal gets delayed until event_sched_out() which then uses > task_work_add() for its delivery. This breaks on PREEMPT_RT because the > signal is delivered with disabled preemption. > > While looking at this, I also stumbled upon __perf_pending_irq() which > requires disabled interrupts but this is not the case on PREEMPT_RT. > > This series aim to address both issues while not introducing a new issue > at the same time ;) > Any testing is appreciated. Were should I apply this patch? The v4 series was applied to linux-rt-devel/linux-6.10.y-rt IIRC - Arnaldo > v4…v5: https://lore.kernel.org/all/20240624152732.1231678-1-bigeasy@linutronix.de/ > - Add TWA_NMI_CURRENT as notify mode for task_work_add() and use it. > PeterZ pointed out that the current version is not NMI safe. > > v3…v4: https://lore.kernel.org/all/20240322065208.60456-1-bigeasy@linutronix.de/ > - Rebased on top of Frederic's series > (https://lore.kernel.org/all/20240621091601.18227-1-frederic@kernel.org) > - Frederick pointed out that perf_pending_task() needs to > perf_swevent_get_recursion_context() in order not to recurse if > something within perf_swevent_.*_recursion_context() triggers a > software event. To address this, the counters have been moved to > the task_struct (#3 + #4) and preemt_disable() has been replaced > with a RCU-read lock (#5). > - The remaning logic same that means the event is pushed to task-work > instead of delivering from IRQ-work. The series was tested with > remove_on_exec as suggested by Marco Elver: On PREEMPT_RT a single > invocation passes, 100 parallel invocations report (for some) > unexpected SIGTRAPs and timeouts. This also observed on !RT > (without the series) with a higher task-count. > > v2…v3: https://lore.kernel.org/all/20240312180814.3373778-1-bigeasy@linutronix.de/ > - Marco suggested to add a few comments > - Added a comment to __perf_event_overflow() to explain why irq_work > is raised in the in_nmi() case. > - Added a comment to perf_event_exit_event() to explain why the > pending event is deleted. > > v1…v2: https://lore.kernel.org/all/20240308175810.2894694-1-bigeasy@linutronix.de/ > - Marco pointed me to the testsuite that showed two problems: > - Delayed task_work from NMI / missing events. > Fixed by triggering dummy irq_work to enforce an interrupt for > the exit-to-userland path which checks task_work > - Increased ref-count on clean up/ during exec. > Mostly addressed by the former change. There is still a window > if the NMI occurs during execve(). This is addressed by removing > the task_work before free_event(). > The testsuite (remove_on_exec) fails sometimes if the event/ > SIGTRAP is sent before the sighandler is installed. > > Sebastian >
On 2024-07-04 16:45:18 [-0300], Arnaldo Carvalho de Melo wrote: > Were should I apply this patch? The v4 series was applied to > linux-rt-devel/linux-6.10.y-rt IIRC Revert the v4 series, apply this one. Wait a few hours and there should be a new RT release with this done ;) > - Arnaldo Sebastian
On Thu, Jul 04, 2024 at 04:45:22PM -0300, Arnaldo Carvalho de Melo wrote:
> On Thu, Jul 04, 2024 at 07:03:34PM +0200, Sebastian Andrzej Siewior wrote:
> > Hi,
> >
> > Arnaldo reported that "perf test sigtrap" fails on PREEMPT_RT. Sending
> > the signal gets delayed until event_sched_out() which then uses
> > task_work_add() for its delivery. This breaks on PREEMPT_RT because the
> > signal is delivered with disabled preemption.
> >
> > While looking at this, I also stumbled upon __perf_pending_irq() which
> > requires disabled interrupts but this is not the case on PREEMPT_RT.
> >
> > This series aim to address both issues while not introducing a new issue
> > at the same time ;)
> > Any testing is appreciated.
>
> Were should I apply this patch? The v4 series was applied to
> linux-rt-devel/linux-6.10.y-rt IIRC
Looking at linux-rt-devel/linux-6.10.y-rt I see:
commit ca8b27c51f0962f8fb59e5acb23e0af791fb5c04
Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Date: Tue Jun 25 10:56:32 2024 +0200
perf: Update the perf series
This is an all-in-one patch integrating the following changes:
- Merging Frederick's "Fix leaked sigtrap events" series as of v4 which
is a dependency.
- Update the "perf test sigtrap" fixup to v4 as posted.
Link: https://lore.kernel.org/20240621091601.18227-1-frederic@kernel.org
Link: https://lore.kernel.org/20240624152732.1231678-1-bigeasy@linutronix.de
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
But I think that is v4, right?
- Arnaldo
>
> > v4…v5: https://lore.kernel.org/all/20240624152732.1231678-1-bigeasy@linutronix.de/
> > - Add TWA_NMI_CURRENT as notify mode for task_work_add() and use it.
> > PeterZ pointed out that the current version is not NMI safe.
> >
> > v3…v4: https://lore.kernel.org/all/20240322065208.60456-1-bigeasy@linutronix.de/
> > - Rebased on top of Frederic's series
> > (https://lore.kernel.org/all/20240621091601.18227-1-frederic@kernel.org)
> > - Frederick pointed out that perf_pending_task() needs to
> > perf_swevent_get_recursion_context() in order not to recurse if
> > something within perf_swevent_.*_recursion_context() triggers a
> > software event. To address this, the counters have been moved to
> > the task_struct (#3 + #4) and preemt_disable() has been replaced
> > with a RCU-read lock (#5).
> > - The remaning logic same that means the event is pushed to task-work
> > instead of delivering from IRQ-work. The series was tested with
> > remove_on_exec as suggested by Marco Elver: On PREEMPT_RT a single
> > invocation passes, 100 parallel invocations report (for some)
> > unexpected SIGTRAPs and timeouts. This also observed on !RT
> > (without the series) with a higher task-count.
> >
> > v2…v3: https://lore.kernel.org/all/20240312180814.3373778-1-bigeasy@linutronix.de/
> > - Marco suggested to add a few comments
> > - Added a comment to __perf_event_overflow() to explain why irq_work
> > is raised in the in_nmi() case.
> > - Added a comment to perf_event_exit_event() to explain why the
> > pending event is deleted.
> >
> > v1…v2: https://lore.kernel.org/all/20240308175810.2894694-1-bigeasy@linutronix.de/
> > - Marco pointed me to the testsuite that showed two problems:
> > - Delayed task_work from NMI / missing events.
> > Fixed by triggering dummy irq_work to enforce an interrupt for
> > the exit-to-userland path which checks task_work
> > - Increased ref-count on clean up/ during exec.
> > Mostly addressed by the former change. There is still a window
> > if the NMI occurs during execve(). This is addressed by removing
> > the task_work before free_event().
> > The testsuite (remove_on_exec) fails sometimes if the event/
> > SIGTRAP is sent before the sighandler is installed.
> >
> > Sebastian
> >
© 2016 - 2025 Red Hat, Inc.