There was an unrelated thread which made me look into how
!IRQ_WORK_HARD_IRQ is processed on PREEMPT_RT. Turns out the queue of
callbacks gets never flushed on CPU shutdown. !PREEMPT_RT has a flush
but on PREEMPT_RT the thread context is required.
This almost never happens but if it happens it is a pain.

A small fix is for the smpboot thread to invoke the thread-function
before parking. This is an improvement. There is still a small window if
callbacks are added after the irq_work thread has been shutdown.
As a fix I added an explicit flush on the control CPU once the CPU is
dead.
From what I've seen there are two users that don't behave as expected if
invoked from the "wrong" CPU:
- cgrp_dead_tasks_iwork()
  This shouldn't be a problem because no task terminates after the
  smpboot thread parked. Everythng that was queued up before will be
  flushed during parking.
  Regardless posted
     https://lore.kernel.org/all/20260911101900.984420-1-bigeasy@linutronix.de/

- wake_up_klogd_work()
  This will not cause data curruption but console printing will be
  delayed until the following printk. Not pretty, posted
     https://lore.kernel.org/all/20260911103832.w6C8cT4L@linutronix.de/

Sebastian Andrzej Siewior (3):
  irq_work: Update a comment regarding CPU hotplug invocation
  irq_work: Flush lazy work CPU down on PREEMPT_RT
  smpboot: Don't park the thread if work is pending

 include/linux/irq_work.h |  2 ++
 kernel/irq_work.c        | 16 ++++++++++++----
 kernel/smp.c             |  1 +
 kernel/smpboot.c         |  6 ++++--
 4 files changed, 19 insertions(+), 6 deletions(-)

-- 
2.55.0

On 2026-09-11 16:38:11 [+0200], To linux-kernel@vger.kernel.org wrote:
> There was an unrelated thread which made me look into how
> !IRQ_WORK_HARD_IRQ is processed on PREEMPT_RT. Turns out the queue of
> callbacks gets never flushed on CPU shutdown. !PREEMPT_RT has a flush
> but on PREEMPT_RT the thread context is required.
> This almost never happens but if it happens it is a pain.
 
Something else that almost never happens: printk() has this
__printk_percpu_data_ready() thingy. I interpreted this wrongly last
Friday: If someone does prinkt() before per-CPU pages are setup then the
work item has the IRQ_WORK_CLAIMED bit set and this work item is copied
to every CPU's view of the per-CPU data. It is not protecting itself
from setting bits but the irq_work API.
irq_work has its own per-CPU llist_head which gets copied during per-CPU
setup. As a result, once interrupts are enabled the IRQ-work will be
invoked on each CPU passing the "original" irq_work pointer (from the
per-CPU-data-init sample). The actual per-CPU data (after setup) never
gets cleared.

We should either disallow irq_work_queue() until per-CPU data is setup
or flush the irq-work queue before the setup (so it has no items while
the data is copied).
While the printk workaround feels wrong, it gets the job done for now. I
am a bit worried that if its usage spreads around and we get other early
users…

Sebastian