There are reports that console kthreads prevented printing
messages during panic() or shutdown(), see
BugLink: https://lore.kernel.org/r/20220610205038.GA3050413@paulmck-ThinkPad-P17-Gen-1
BugLink: https://lore.kernel.org/r/CAMdYzYpF4FNTBPZsEFeWRuEwSies36QM_As8osPWZSr2q-viEA@mail.gmail.com

In both situations, printk() correctly tries to flush the consoles
directly but it fails to get the global console_lock(). Both
problems went away with these patches:

1st patch blocks console kthreads so that they do not start
handling new messages when the direct printing is preferred
by the system state. It is probably enough on its own. It
solves the problem when the kthreads actively did
the wrong thing.

2nd patch allows to wait for the console kthreads to release
the lock in any context. It should make it more reliable.
It would have been useful even for the legacy code.

More possible improvements:

  +  the waiting might be necessary also in the suspend code paths

  + convert con->mutex to con->spinlock to avoid blocking
    the global console_lock() when sleeping with con->lock

  + at least disable preemption around console_emit_next_record()
    in console kthread to avoid sleeping in the console driver
    code

  + somehow change the priority of the kthread so that it gets
    scheduled immediately when the scheduler works

Petr Mladek (2):
  printk: Block console kthreads when direct printing will be required
  printk: Wait for the global console lock when the system is going down

 include/linux/printk.h      |  5 +++++
 kernel/panic.c              |  2 ++
 kernel/printk/internal.h    |  2 ++
 kernel/printk/printk.c      |  8 +++++++-
 kernel/printk/printk_safe.c | 32 ++++++++++++++++++++++++++++++++
 kernel/reboot.c             |  2 ++
 6 files changed, 50 insertions(+), 1 deletion(-)

-- 
2.35.3

On Wed, Jun 15, 2022 at 06:28:03PM +0200, Petr Mladek wrote:
> There are reports that console kthreads prevented printing
> messages during panic() or shutdown(), see
> BugLink: https://lore.kernel.org/r/20220610205038.GA3050413@paulmck-ThinkPad-P17-Gen-1
> BugLink: https://lore.kernel.org/r/CAMdYzYpF4FNTBPZsEFeWRuEwSies36QM_As8osPWZSr2q-viEA@mail.gmail.com
> 
> In both situations, printk() correctly tries to flush the consoles
> directly but it fails to get the global console_lock(). Both
> problems went away with these patches:
> 
> 1st patch blocks console kthreads so that they do not start
> handling new messages when the direct printing is preferred
> by the system state. It is probably enough on its own. It
> solves the problem when the kthreads actively did
> the wrong thing.
> 
> 2nd patch allows to wait for the console kthreads to release
> the lock in any context. It should make it more reliable.
> It would have been useful even for the legacy code.

Thank you!

For the series:

Tested-by: Paul E. McKenney <paulmck@kernel.org>

> More possible improvements:
> 
>   +  the waiting might be necessary also in the suspend code paths
> 
>   + convert con->mutex to con->spinlock to avoid blocking
>     the global console_lock() when sleeping with con->lock
> 
>   + at least disable preemption around console_emit_next_record()
>     in console kthread to avoid sleeping in the console driver
>     code
> 
>   + somehow change the priority of the kthread so that it gets
>     scheduled immediately when the scheduler works
> 
> Petr Mladek (2):
>   printk: Block console kthreads when direct printing will be required
>   printk: Wait for the global console lock when the system is going down
> 
>  include/linux/printk.h      |  5 +++++
>  kernel/panic.c              |  2 ++
>  kernel/printk/internal.h    |  2 ++
>  kernel/printk/printk.c      |  8 +++++++-
>  kernel/printk/printk_safe.c | 32 ++++++++++++++++++++++++++++++++
>  kernel/reboot.c             |  2 ++
>  6 files changed, 50 insertions(+), 1 deletion(-)
> 
> -- 
> 2.35.3
>

On Wed 2022-06-15 10:10:42, Paul E. McKenney wrote:
> On Wed, Jun 15, 2022 at 06:28:03PM +0200, Petr Mladek wrote:
> > There are reports that console kthreads prevented printing
> > messages during panic() or shutdown(), see
> > BugLink: https://lore.kernel.org/r/20220610205038.GA3050413@paulmck-ThinkPad-P17-Gen-1
> > BugLink: https://lore.kernel.org/r/CAMdYzYpF4FNTBPZsEFeWRuEwSies36QM_As8osPWZSr2q-viEA@mail.gmail.com
> > 
> > In both situations, printk() correctly tries to flush the consoles
> > directly but it fails to get the global console_lock(). Both
> > problems went away with these patches:
> > 
> > 1st patch blocks console kthreads so that they do not start
> > handling new messages when the direct printing is preferred
> > by the system state. It is probably enough on its own. It
> > solves the problem when the kthreads actively did
> > the wrong thing.
> > 
> > 2nd patch allows to wait for the console kthreads to release
> > the lock in any context. It should make it more reliable.
> > It would have been useful even for the legacy code.
> 
> Thank you!
> 
> For the series:
> 
> Tested-by: Paul E. McKenney <paulmck@kernel.org>

Thanks a lot for testing.

I have pushed it into printk/linux.git, branch rework/kthreads
to give it a spin in linux-next.

Best Regards,
Petr