printk: Yield console_sem to blocking waiters

[PATCH 0/1] printk: Yield console_sem to blocking waiters

Posted by Aruna Ramakrishna 2 weeks ago

Problem context:

After a reboot, NetworkManager failed to bring up interfaces because
systemd was stalled, due to a flood of printk warnings. Systemd was
blocked waiting on the console lock for over 122 seconds:

  __schedule+0x224/0x65b
  schedule+0x5f/0xdf
  schedule_timeout+0x110/0x145
  __down_common+0x14d/0x18c
  ? srso_alias_return_thunk+0x5/0x7
  down+0x43/0x54
  console_lock+0x16/0x3d
  show_cons_active+0x3d/0x1b6
  dev_attr_show+0x16/0x50
  sysfs_kf_seq_show+0x9c/0x124
  seq_read_iter+0x12b/0x4e3
  new_sync_read+0x140/0x1db
  vfs_read+0x106/0x199
  ksys_read+0x67/0xe8
  do_syscall_64+0x35/0x87
  entry_SYSCALL_64_after_hwframe+0x6e/0x0

The loglevel was set to 7, due to which there were a flood of messages
to the console, which in turn starved systemd due to the chain of direct
printk handoffs. Direct printk handoffs transfer printing ownership
between printk callers without releasing console_sem. A continuous
chain of these handoffs can therefore leave blocking console_lock()
callers waiting on the semaphore.

Proposed solution:

If there is a blocked console_sem waiter, then:
1. Prevent new printk-to-printk handoffs.
2. Stop the current owner from indefinitely flushing all records until
its done.

This unblocks console_sem waiters who are not doing printk.

Aruna Ramakrishna (1):
  printk: Yield console_sem to blocking waiters

 kernel/printk/printk.c | 39 ++++++++++++++++++++++++++++++++-------
 1 file changed, 32 insertions(+), 7 deletions(-)

-- 
2.43.7

Re: [PATCH 0/1] printk: Yield console_sem to blocking waiters

Posted by John Ogness 2 weeks ago

On 2026-09-10, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
> Problem context:
>
> After a reboot, NetworkManager failed to bring up interfaces because
> systemd was stalled, due to a flood of printk warnings.

Note that kernel warnings are considered serious and will put the CPU
into emergency state.

> Systemd was
> blocked waiting on the console lock for over 122 seconds:

Also note that the console lock is related to legacy consoles. Rather
than trying to make the legacy console trainwreck more reliable and
non-interfering, efforts should be spent converting legacy consoles to
NBCON.

What console driver are you using?

For v7.3 the 8250 driver will be NBCON.

John

Re: [PATCH 0/1] printk: Yield console_sem to blocking waiters

Posted by Aruna Ramakrishna 1 week, 6 days ago

> On Sep 11, 2026, at 1:08 AM, John Ogness <john.ogness@linutronix.de> wrote:
> 
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> On 2026-09-10, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
> > Problem context:
> >
> > After a reboot, NetworkManager failed to bring up interfaces because
> > systemd was stalled, due to a flood of printk warnings.
> 
> Note that kernel warnings are considered serious and will put the CPU
> into emergency state.

I apologize - I misspoke. These were not KERN_WARNs, rather KERN_INFOs
being used by RDS to warn of connection failures etc. A flood of these caused
the console_lock starvation.

> 
> > Systemd was
> > blocked waiting on the console lock for over 122 seconds:
> 
> Also note that the console lock is related to legacy consoles. Rather
> than trying to make the legacy console trainwreck more reliable and
> non-interfering, efforts should be spent converting legacy consoles to
> NBCON.
> 
> What console driver are you using?
> 
> For v7.3 the 8250 driver will be NBCON.
> 
> John
> 

This is for the legacy 8250 serial console.

Thanks,
Aruna

Re: [PATCH 0/1] printk: Yield console_sem to blocking waiters

Posted by Aruna Ramakrishna 1 week, 1 day ago

> On Sep 11, 2026, at 11:24 AM, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
> 
>> 
>> On Sep 11, 2026, at 1:08 AM, John Ogness <john.ogness@linutronix.de> wrote:
>> 
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious
>> On 2026-09-10, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
>>> Problem context:
>>> 
>>> After a reboot, NetworkManager failed to bring up interfaces because
>>> systemd was stalled, due to a flood of printk warnings.
>> 
>> Note that kernel warnings are considered serious and will put the CPU
>> into emergency state.
> 
> I apologize - I misspoke. These were not KERN_WARNs, rather KERN_INFOs
> being used by RDS to warn of connection failures etc. A flood of these caused
> the console_lock starvation.
> 
>> 
>>> Systemd was
>>> blocked waiting on the console lock for over 122 seconds:
>> 
>> Also note that the console lock is related to legacy consoles. Rather
>> than trying to make the legacy console trainwreck more reliable and
>> non-interfering, efforts should be spent converting legacy consoles to
>> NBCON.
>> 
>> What console driver are you using?
>> 
>> For v7.3 the 8250 driver will be NBCON.
>> 
>> John
>> 
> 
> This is for the legacy 8250 serial console.
> 
> Thanks,
> Aruna

Hi John,

Even if this patch is not applicable to v7.3, it is still applicable to the older kernel
versions, right? Will this be considered for the stable tree, without being merged
upstream first? If yes, would you mind reviewing the patch?

Thanks,
Aruna

Re: [PATCH 0/1] printk: Yield console_sem to blocking waiters

Posted by Petr Mladek 3 days, 1 hour ago

On Wed 2026-09-16 18:22:12, Aruna Ramakrishna wrote:
> 
> > On Sep 11, 2026, at 11:24 AM, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
> > 
> >> 
> >> On Sep 11, 2026, at 1:08 AM, John Ogness <john.ogness@linutronix.de> wrote:
> >> 
> >> This Message Is From an External Sender
> >> This message came from outside your organization.
> >> Report Suspicious
> >> On 2026-09-10, Aruna Ramakrishna <aruna.ramakrishna@oracle.com> wrote:
> >>> Problem context:
> >>> 
> >>> After a reboot, NetworkManager failed to bring up interfaces because
> >>> systemd was stalled, due to a flood of printk warnings.
> >> 
> >> Note that kernel warnings are considered serious and will put the CPU
> >> into emergency state.
> > 
> > I apologize - I misspoke. These were not KERN_WARNs, rather KERN_INFOs
> > being used by RDS to warn of connection failures etc. A flood of these caused
> > the console_lock starvation.
> > 
> >> 
> >>> Systemd was
> >>> blocked waiting on the console lock for over 122 seconds:
> >> 
> >> Also note that the console lock is related to legacy consoles. Rather
> >> than trying to make the legacy console trainwreck more reliable and
> >> non-interfering, efforts should be spent converting legacy consoles to
> >> NBCON.
> >> 
> >> What console driver are you using?
> >> 
> >> For v7.3 the 8250 driver will be NBCON.
> >> 
> >> John
> >> 
> > 
> > This is for the legacy 8250 serial console.
> > 
> > Thanks,
> > Aruna
> 
> Hi John,
> 
> Even if this patch is not applicable to v7.3, it is still applicable to the older kernel
> versions, right? Will this be considered for the stable tree, without being merged
> upstream first? If yes, would you mind reviewing the patch?

The patch is not acceptable as is. In particular, it does not handle
panic, suspend, various emergency situations where the waiters likely
won't be waken or we do not want to risk the handoff.

I agree with John, let's focus on converting console drivers
to the NBCON API.

I am sorry but I personally do not have time and motivation to review
hacks which would be suitable only for stable.

Best Regards,
Petr