Tasks RCU only treats a voluntary context switch, usermode or idle as a
quiescent state, because a preempted task may be sitting in a trampoline
that is about to be freed. That was a fine trade when PREEMPT_NONE
servers compiled Tasks RCU away and PREEMPT desktops rarely ran
long-lived in-kernel loops. PREEMPT_LAZY changes both halves at once:
Tasks RCU is now real on server configs, and cond_resched() is a no-op,
so a CPU-bound kthread or kworker only ever loses the CPU by being
preempted, which is exactly the event Tasks RCU refuses to count.

The way this showed up for us was a cgroup writeback worker draining a
very large cgwb for around eleven minutes on an arm64 box. Nothing wrong
with that on its own, but a BPF program detach on another CPU went
bpf_trampoline_update() -> ftrace_shutdown() -> synchronize_rcu_tasks()
while holding trampoline_mutex, forty-odd tasks piled up behind the
mutex, and the hung task detector panicked the machine. The kprobe jump
optimizer is worse in principle: it does synchronize_rcu_tasks() under
kprobe_mutex, text_mutex and cpus_read_lock(), so one long-running
kthread can stall static key updates and CPU hotplug for its whole run.
The current answer is to find each such loop and add
cond_resched_tasks_rcu_qs() to it, which is the kind of annotation
PREEMPT_LAZY was supposed to let us stop writing.

This series tries the other direction: have the trampolines say when a
task is inside them, so that a preemption anywhere else can be a
quiescent state.

 - task_struct grows an int, rcu_tramp_nesting. Every trampoline whose
   lifetime Tasks RCU guards increments it before calling out and
   decrements it before returning: ftrace_caller and its dynamic copies,
   the BPF trampoline (which drops it again around the call to the
   original function, since im->pcref covers that), the x86 optprobe
   template, and out-of-line register_ftrace_direct() trampolines. Only
   current writes it and nested users are balanced, so it is a plain
   non-atomic inc/dec, one load of current plus one RMW per entry/exit.

 - The inc/dec are inside the trampoline, so there is a window of a few
   instructions on each side where the count is zero but the task is in
   (or on its way into) trampoline text. Nothing there can be preempted
   synchronously, only from an interrupt, so the irq-exit preemption path
   looks at regs->ip and holds the count across preempt_schedule_irq()
   when the IP is somewhere the counter cannot cover: outside core and
   module text (all the dynamically allocated trampolines and slots), in
   the static ftrace stubs or the x86 return thunks that still hold a
   direct-call target, in a module that hosts its own direct trampoline,
   or inside the bytes after a kprobe that the jump optimizer may be
   about to rewrite (the one synchronize_rcu_tasks() user that is not
   about trampolines at all).

 - With those in place, rcu_tasks_classic_qs() also clears the holdout
   flag on a preemption when the count is zero, on architectures that
   opt in. x86-64 and arm64 do so here. Everyone else keeps the
   voluntary-only rule and is untouched apart from the (unused) field.

A running holdout already gets poked via rcu_request_urgent_qs_task(),
which makes the next tick set NEED_RESCHED, so with this the resulting
preemption retires it and a Tasks RCU grace period is bounded by roughly
a tick plus the longest preempt-off section rather than by the longest
stretch without a voluntary schedule().

Patches 1-12 are scaffolding and change no behaviour on their own; patch
13 flips the rule and selects the option for the two architectures.

Testing so far is QEMU only: x86-64, PREEMPT_LAZY with PREEMPT_RCU=n,
PROVE_RCU and lockdep, with and without PREEMPT_DYNAMIC. A kthread
spinning in-kernel for 30s with the function tracer, an ftrace kprobe,
an optimized kprobe and fentry/fexit programs attached:
synchronize_rcu_tasks() goes from 29.7s to 0.1-0.3s, tearing down a
DYNAMIC ftrace_ops (tracefs instance function -> nop) from 27s to
0.2-0.8s, and the ftrace-direct sample modules load, fire and unload in
about 2.5s each while the spinner runs, with no warnings and the new
return-to-user assertion quiet. arm64 is build-tested only at this
point; real hardware numbers for both are the obvious next step and I
did not want to sit on the idea waiting for them.

Things I would particularly like opinions on:

 - Whether hooking rcu_tasks_classic_qs() is the right place, or whether
   Paul would rather see this expressed differently inside Tasks RCU.
 - return_to_handler and the rethook/kretprobe trampolines are not
   instrumented. Their C callees take the ftrace recursion lock before
   touching any ops and the trampolines themselves are static text, so I
   believe they do not need it, but I would like Steven and Masami to
   confirm.
 - The register_ftrace_direct() contract change: out-of-line direct
   trampolines now have to maintain the count themselves (the samples
   are converted). I do not know of out-of-tree users beyond BPF, but
   this is the one place an existing user could be silently weakened.
 - Whether arm64 folks are comfortable with the ldr/add/str in
   ftrace_caller and the BPF trampoline, and with treating all of
   ftrace_caller as trampoline text for the IP check.
 - If this holds up, cond_resched_tasks_rcu_qs() and
   rcu_softirq_qs_periodic() become unnecessary on the opted-in
   architectures; I have not touched them here.

Based on v7.3-rc2+ (893e11787f78).

---
Josef Bacik (13):
      rcu-tasks: Add per-task trampoline nesting count
      entry: Pass pt_regs to irqentry_exit_cond_resched()
      rcu-tasks: Hold trampoline nesting across irq-exit preemption in trampoline text
      kprobes: Let Tasks RCU recognise tasks preempted in an optprobe jump window
      ftrace: Mark modules hosting direct-call trampolines for Tasks RCU
      x86/ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller
      x86/kprobes: Maintain Tasks RCU trampoline nesting in the optprobe template
      bpf, x86: Maintain Tasks RCU trampoline nesting in the BPF trampoline
      arm64: ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller
      bpf, arm64: Maintain Tasks RCU trampoline nesting in the BPF trampoline
      samples: ftrace: Maintain Tasks RCU trampoline nesting in direct-call trampolines
      rcutorture: Bracket Tasks RCU readers with trampoline nesting
      rcu-tasks: Treat preemption outside trampolines as a quiescent state

 arch/arm64/Kconfig                          |  1 +
 arch/arm64/kernel/asm-offsets.c             |  3 ++
 arch/arm64/kernel/entry-ftrace.S            | 35 +++++++++++++
 arch/arm64/kernel/ftrace.c                  | 16 ++++++
 arch/arm64/net/bpf_jit_comp.c               | 46 ++++++++++++++++
 arch/x86/Kconfig                            |  1 +
 arch/x86/kernel/asm-offsets.c               |  3 ++
 arch/x86/kernel/ftrace.c                    | 37 +++++++++++++
 arch/x86/kernel/ftrace_64.S                 | 43 +++++++++++++++
 arch/x86/kernel/kprobes/opt.c               | 20 +++++++
 arch/x86/kernel/vmlinux.lds.S               |  4 ++
 arch/x86/net/bpf_jit_comp.c                 | 43 +++++++++++++++
 arch/x86/xen/enlighten_pv.c                 |  2 +-
 include/linux/irq-entry-common.h            | 14 ++---
 include/linux/kprobes.h                     |  8 ++-
 include/linux/module.h                      |  7 +++
 include/linux/rcupdate.h                    | 70 ++++++++++++++++++++++++-
 include/linux/sched.h                       |  1 +
 kernel/entry/common.c                       | 29 +++++++++--
 kernel/fork.c                               |  1 +
 kernel/kprobes.c                            | 24 +++++++++
 kernel/rcu/Kconfig                          | 17 ++++--
 kernel/rcu/rcutorture.c                     |  6 +++
 kernel/rcu/tasks.h                          | 81 +++++++++++++++++++++++++++--
 kernel/rcu/update.c                         |  2 +
 kernel/trace/ftrace.c                       | 39 ++++++++++++++
 samples/ftrace/ftrace-direct-modify.c       |  9 ++++
 samples/ftrace/ftrace-direct-multi-modify.c |  9 ++++
 samples/ftrace/ftrace-direct-multi.c        |  5 ++
 samples/ftrace/ftrace-direct-too.c          |  5 ++
 samples/ftrace/ftrace-direct.c              |  5 ++
 samples/ftrace/ftrace-direct.h              | 64 +++++++++++++++++++++++
 32 files changed, 629 insertions(+), 21 deletions(-)
---
base-commit: 893e11787f78e43b534e252249ac3fff4d1333f8
change-id: 20260910-b4-rcu-tasks-preempt-qs-401ff45465c7

Best regards,
--  
Josef Bacik <josef@toxicpanda.com>

On Thu, 10 Sep 2026 18:50:23 +0000
Josef Bacik <josef@toxicpanda.com> wrote:

> Tasks RCU only treats a voluntary context switch, usermode or idle as a
> quiescent state, because a preempted task may be sitting in a trampoline
> that is about to be freed. That was a fine trade when PREEMPT_NONE
> servers compiled Tasks RCU away and PREEMPT desktops rarely ran
> long-lived in-kernel loops. PREEMPT_LAZY changes both halves at once:
> Tasks RCU is now real on server configs, and cond_resched() is a no-op,
> so a CPU-bound kthread or kworker only ever loses the CPU by being
> preempted, which is exactly the event Tasks RCU refuses to count.
> 
> The way this showed up for us was a cgroup writeback worker draining a
> very large cgwb for around eleven minutes on an arm64 box. Nothing wrong

So you have a kernel thread running for 11 minutes without a schedule?

You could still put in a cond_resched_tasks_rcu_qs() in that loop. But I
guess you are trying to get rid of doing that too.

> with that on its own, but a BPF program detach on another CPU went
> bpf_trampoline_update() -> ftrace_shutdown() -> synchronize_rcu_tasks()
> while holding trampoline_mutex, forty-odd tasks piled up behind the
> mutex, and the hung task detector panicked the machine. The kprobe jump
> optimizer is worse in principle: it does synchronize_rcu_tasks() under
> kprobe_mutex, text_mutex and cpus_read_lock(), so one long-running
> kthread can stall static key updates and CPU hotplug for its whole run.
> The current answer is to find each such loop and add
> cond_resched_tasks_rcu_qs() to it, which is the kind of annotation
> PREEMPT_LAZY was supposed to let us stop writing.
> 
> This series tries the other direction: have the trampolines say when a
> task is inside them, so that a preemption anywhere else can be a
> quiescent state.
> 
>  - task_struct grows an int, rcu_tramp_nesting. Every trampoline whose
>    lifetime Tasks RCU guards increments it before calling out and
>    decrements it before returning: ftrace_caller and its dynamic copies,
>    the BPF trampoline (which drops it again around the call to the
>    original function, since im->pcref covers that), the x86 optprobe
>    template, and out-of-line register_ftrace_direct() trampolines. Only
>    current writes it and nested users are balanced, so it is a plain
>    non-atomic inc/dec, one load of current plus one RMW per entry/exit.
> 
>  - The inc/dec are inside the trampoline, so there is a window of a few
>    instructions on each side where the count is zero but the task is in
>    (or on its way into) trampoline text. Nothing there can be preempted
>    synchronously, only from an interrupt, so the irq-exit preemption path
>    looks at regs->ip and holds the count across preempt_schedule_irq()
>    when the IP is somewhere the counter cannot cover: outside core and
>    module text (all the dynamically allocated trampolines and slots), in
>    the static ftrace stubs or the x86 return thunks that still hold a
>    direct-call target, in a module that hosts its own direct trampoline,
>    or inside the bytes after a kprobe that the jump optimizer may be
>    about to rewrite (the one synchronize_rcu_tasks() user that is not
>    about trampolines at all).

So basically if the preemption happens outside of core or module text
(which should be the case of any dynamically allocated trampoline), the
task is marked to be in the grace period across its schedule, so that the
RCU_TASK cannot move forward?

> 
>  - With those in place, rcu_tasks_classic_qs() also clears the holdout
>    flag on a preemption when the count is zero, on architectures that
>    opt in. x86-64 and arm64 do so here. Everyone else keeps the
>    voluntary-only rule and is untouched apart from the (unused) field.
> 
> A running holdout already gets poked via rcu_request_urgent_qs_task(),
> which makes the next tick set NEED_RESCHED, so with this the resulting
> preemption retires it and a Tasks RCU grace period is bounded by roughly
> a tick plus the longest preempt-off section rather than by the longest
> stretch without a voluntary schedule().
> 
> Patches 1-12 are scaffolding and change no behaviour on their own; patch
> 13 flips the rule and selects the option for the two architectures.
> 
> Testing so far is QEMU only: x86-64, PREEMPT_LAZY with PREEMPT_RCU=n,
> PROVE_RCU and lockdep, with and without PREEMPT_DYNAMIC. A kthread
> spinning in-kernel for 30s with the function tracer, an ftrace kprobe,
> an optimized kprobe and fentry/fexit programs attached:
> synchronize_rcu_tasks() goes from 29.7s to 0.1-0.3s, tearing down a
> DYNAMIC ftrace_ops (tracefs instance function -> nop) from 27s to
> 0.2-0.8s, and the ftrace-direct sample modules load, fire and unload in
> about 2.5s each while the spinner runs, with no warnings and the new
> return-to-user assertion quiet. arm64 is build-tested only at this
> point; real hardware numbers for both are the obvious next step and I
> did not want to sit on the idea waiting for them.
> 
> Things I would particularly like opinions on:
> 
>  - Whether hooking rcu_tasks_classic_qs() is the right place, or whether
>    Paul would rather see this expressed differently inside Tasks RCU.
>  - return_to_handler and the rethook/kretprobe trampolines are not
>    instrumented. Their C callees take the ftrace recursion lock before
>    touching any ops and the trampolines themselves are static text, so I
>    believe they do not need it, but I would like Steven and Masami to
>    confirm.

Note, there has been some work in the past (and may happen again in the
future) that will remove the preempt_disable() from the trace_recursion
locking. If that happens, then I believe the trace_recursion would need to
increment (and decrement) your counter. Probably need a comment there to
let whomever know about it if they decide to remove the preempt_disable().

>  - The register_ftrace_direct() contract change: out-of-line direct
>    trampolines now have to maintain the count themselves (the samples
>    are converted). I do not know of out-of-tree users beyond BPF, but
>    this is the one place an existing user could be silently weakened.
>  - Whether arm64 folks are comfortable with the ldr/add/str in
>    ftrace_caller and the BPF trampoline, and with treating all of
>    ftrace_caller as trampoline text for the IP check.
>  - If this holds up, cond_resched_tasks_rcu_qs() and
>    rcu_softirq_qs_periodic() become unnecessary on the opted-in
>    architectures; I have not touched them here.

I don't know. It may work, but I have a feeling there's a devil in the
details here that is waiting to bite us in the underside when we are not
(RCU) watching.

-- Steve

On Thu, Sep 10, 2026 at 03:44:50PM -0400, Steven Rostedt wrote:
> On Thu, 10 Sep 2026 18:50:23 +0000
> Josef Bacik <josef@toxicpanda.com> wrote:
> 
> > Tasks RCU only treats a voluntary context switch, usermode or idle as a
> > quiescent state, because a preempted task may be sitting in a trampoline
> > that is about to be freed. That was a fine trade when PREEMPT_NONE
> > servers compiled Tasks RCU away and PREEMPT desktops rarely ran
> > long-lived in-kernel loops. PREEMPT_LAZY changes both halves at once:
> > Tasks RCU is now real on server configs, and cond_resched() is a no-op,
> > so a CPU-bound kthread or kworker only ever loses the CPU by being
> > preempted, which is exactly the event Tasks RCU refuses to count.
> > 
> > The way this showed up for us was a cgroup writeback worker draining a
> > very large cgwb for around eleven minutes on an arm64 box. Nothing wrong
> 
> So you have a kernel thread running for 11 minutes without a schedule?

We have seen this from time to time here as well.

> You could still put in a cond_resched_tasks_rcu_qs() in that loop. But I
> guess you are trying to get rid of doing that too.

And we have done this a few times, but if this proves to be an acceptable
alternative, that would be wonderful.  ;-)

> > with that on its own, but a BPF program detach on another CPU went
> > bpf_trampoline_update() -> ftrace_shutdown() -> synchronize_rcu_tasks()
> > while holding trampoline_mutex, forty-odd tasks piled up behind the
> > mutex, and the hung task detector panicked the machine. The kprobe jump
> > optimizer is worse in principle: it does synchronize_rcu_tasks() under
> > kprobe_mutex, text_mutex and cpus_read_lock(), so one long-running
> > kthread can stall static key updates and CPU hotplug for its whole run.
> > The current answer is to find each such loop and add
> > cond_resched_tasks_rcu_qs() to it, which is the kind of annotation
> > PREEMPT_LAZY was supposed to let us stop writing.
> > 
> > This series tries the other direction: have the trampolines say when a
> > task is inside them, so that a preemption anywhere else can be a
> > quiescent state.
> > 
> >  - task_struct grows an int, rcu_tramp_nesting. Every trampoline whose
> >    lifetime Tasks RCU guards increments it before calling out and
> >    decrements it before returning: ftrace_caller and its dynamic copies,
> >    the BPF trampoline (which drops it again around the call to the
> >    original function, since im->pcref covers that), the x86 optprobe
> >    template, and out-of-line register_ftrace_direct() trampolines. Only
> >    current writes it and nested users are balanced, so it is a plain
> >    non-atomic inc/dec, one load of current plus one RMW per entry/exit.
> > 
> >  - The inc/dec are inside the trampoline, so there is a window of a few
> >    instructions on each side where the count is zero but the task is in
> >    (or on its way into) trampoline text. Nothing there can be preempted
> >    synchronously, only from an interrupt, so the irq-exit preemption path
> >    looks at regs->ip and holds the count across preempt_schedule_irq()
> >    when the IP is somewhere the counter cannot cover: outside core and
> >    module text (all the dynamically allocated trampolines and slots), in
> >    the static ftrace stubs or the x86 return thunks that still hold a
> >    direct-call target, in a module that hosts its own direct trampoline,
> >    or inside the bytes after a kprobe that the jump optimizer may be
> >    about to rewrite (the one synchronize_rcu_tasks() user that is not
> >    about trampolines at all).
> 
> So basically if the preemption happens outside of core or module text
> (which should be the case of any dynamically allocated trampoline), the
> task is marked to be in the grace period across its schedule, so that the
> RCU_TASK cannot move forward?

If I understand correctly, the difference with Josef's patch is
that rcu_tasks_classic_qs(current, true), when called without the
direct or indirect aid of a trampoline, will provide an RCU Tasks
quiescent state.  In contrast, without Josef's patch, no call to
rcu_tasks_classic_qs(current, true) will provide such a quiescent state.

More to the point, because rcu_tasks_classic_qs(current, false) is
invoked from rcu_note_context_switch(), any preemption to kernel code not
within or called from a trampoline will now provide a quiescent state.
Keeping in mind that cond_resched() is treated as a preemption, this
change could potentially greatly reduce the need for sprinkling calls
to cond_resched_tasks_rcu_qs() throughout the kernel.  (Except that
the call to cond_resched() has to actually invoke the scheduler for
anything to happen.)

Which, if it works, would of course be a good thing.  ;-)

> >  - With those in place, rcu_tasks_classic_qs() also clears the holdout
> >    flag on a preemption when the count is zero, on architectures that
> >    opt in. x86-64 and arm64 do so here. Everyone else keeps the
> >    voluntary-only rule and is untouched apart from the (unused) field.
> > 
> > A running holdout already gets poked via rcu_request_urgent_qs_task(),
> > which makes the next tick set NEED_RESCHED, so with this the resulting
> > preemption retires it and a Tasks RCU grace period is bounded by roughly
> > a tick plus the longest preempt-off section rather than by the longest
> > stretch without a voluntary schedule().
> > 
> > Patches 1-12 are scaffolding and change no behaviour on their own; patch
> > 13 flips the rule and selects the option for the two architectures.
> > 
> > Testing so far is QEMU only: x86-64, PREEMPT_LAZY with PREEMPT_RCU=n,
> > PROVE_RCU and lockdep, with and without PREEMPT_DYNAMIC. A kthread
> > spinning in-kernel for 30s with the function tracer, an ftrace kprobe,
> > an optimized kprobe and fentry/fexit programs attached:
> > synchronize_rcu_tasks() goes from 29.7s to 0.1-0.3s, tearing down a
> > DYNAMIC ftrace_ops (tracefs instance function -> nop) from 27s to
> > 0.2-0.8s, and the ftrace-direct sample modules load, fire and unload in
> > about 2.5s each while the spinner runs, with no warnings and the new
> > return-to-user assertion quiet. arm64 is build-tested only at this
> > point; real hardware numbers for both are the obvious next step and I
> > did not want to sit on the idea waiting for them.
> > 
> > Things I would particularly like opinions on:
> > 
> >  - Whether hooking rcu_tasks_classic_qs() is the right place, or whether
> >    Paul would rather see this expressed differently inside Tasks RCU.

We might well need more:

o	The rcu_tasks_pertask() might need to check to see if task "t"
	is in a quiescent state.  The task_call_func() function might
	be helpful in safely accessing that task's state remotely in
	the common case where the task is blocked or preempted.

o	Given a task that runs for a very long time on a CPU that has
	nothing else to do (so that cond_resched() does nothing and
	there are no preemptions), and does so in a code path that never
	invokes cond_resched_tasks_rcu_qs(), it might be necessary to IPI
	to CPU that this task is running on.  Or to invoke resched_cpu()
	in order to force a context switch on that CPU, whether it needs
	one or not.  (Which makes the scheduler do the IPI for us.)

o	PREEMPT_RT kernels might want memory ordering on the nesting count
	increments and decrements, along with READ_ONCE() and WRITE_ONCE()
	or similar, in order to avoid the aforementioned IPIs.  But this
	increases overhead, so !PREEMPT_RT kernels would *not* want this.

And probably other things that I am not yet seeing.  ;-)

> >  - return_to_handler and the rethook/kretprobe trampolines are not
> >    instrumented. Their C callees take the ftrace recursion lock before
> >    touching any ops and the trampolines themselves are static text, so I
> >    believe they do not need it, but I would like Steven and Masami to
> >    confirm.
> 
> Note, there has been some work in the past (and may happen again in the
> future) that will remove the preempt_disable() from the trace_recursion
> locking. If that happens, then I believe the trace_recursion would need to
> increment (and decrement) your counter. Probably need a comment there to
> let whomever know about it if they decide to remove the preempt_disable().
> 
> >  - The register_ftrace_direct() contract change: out-of-line direct
> >    trampolines now have to maintain the count themselves (the samples
> >    are converted). I do not know of out-of-tree users beyond BPF, but
> >    this is the one place an existing user could be silently weakened.
> >  - Whether arm64 folks are comfortable with the ldr/add/str in
> >    ftrace_caller and the BPF trampoline, and with treating all of
> >    ftrace_caller as trampoline text for the IP check.
> >  - If this holds up, cond_resched_tasks_rcu_qs() and
> >    rcu_softirq_qs_periodic() become unnecessary on the opted-in
> >    architectures; I have not touched them here.
> 
> I don't know. It may work, but I have a feeling there's a devil in the
> details here that is waiting to bite us in the underside when we are not
> (RCU) watching.

Well, that is RCU for you!  ;-)

							Thanx, Paul