Hi,

This patch series enables one to dump the messages in printk ring
buffer unless all CPUs are locked up. This is useful to view the
kernel messages when terminal is unresponsive to enter commands
like dmesg and syslog services are also disabled, especially on
embedded targets. Although debug features like kdb/kgdb already
allow this, these debug configs should be enabled which is often
not the case.

Till the last version, kmsg_dump* interface was being used to
retrieve the messages in printk buffer before dumping them to
consoles. However, John Ogness pointed out the issue with
kmsg_dump* interface that it doesn't work well with extended
consoles. He suggested a new method to reuse the code in
console_flush_on_panic() but without disabling scheduling.

In the first commit, code under CONSOLE_REPLAY_ALL mode in
console_flush_on_panic() is taken out to a helper function
console_rewind_all() to set the console sequence numbder to
oldest record in the printk buffer. And the new function to
dump the buffer called dump_printk_buffer() calls this function
after taking the console lock and then releases the lock which
flushes out the contents of printk buffer to console.

In the second commit, code is added to call dump_printk_buffer()
function when sysrq+D is pressed. As the function may sleep,
it cannot be called from interrupt context. A work is queued
in the system unbound workqueue to call the function when
the key is pressed.

Links to previous discussion:
- https://lore.kernel.org/all/cover.1705331453.git.sreenath.vijayan@sony.com/T/#t
- https://lore.kernel.org/linux-serial/20231221133953.1507021-1-sreenath.vijayan@sony.com/

Changelog:
V3 -> V4:
- refactored code in console_flush_on_panic() under CONSOLE_REPLAY_ALL mode
- added helper function console_rewind_all()
- used console_rewind_all() instead of ksmg_dump*() in dump_printk_buffer()

V2 -> V3:
- split the implementation into two commits
- added function in printk.c to dump printk buffer to consoles
- added Suggested-by tag
- removed code to dump printk buffer from sysrq.c and called
new function

V1 -> V2:
- modified kernel ring buffer to printk ring buffer
- allocated buf dynamically to prevent stack frame size warnings
- used buf of size 2048 to match PRINTK_MESSAGE_MAX and added comment

-- Sreenath

Sreenath Vijayan (2):
  printk: Add function to dump printk buffer directly to consoles
  tty/sysrq: Dump printk ring buffer messages via sysrq

 Documentation/admin-guide/sysrq.rst |  2 +
 drivers/tty/sysrq.c                 | 20 +++++++++-
 include/linux/printk.h              |  4 ++
 kernel/printk/printk.c              | 61 +++++++++++++++++------------
 4 files changed, 62 insertions(+), 25 deletions(-)

-- 
2.43.0

Hi,

first, I am sorry for joining the game so late. I was sick
and have had a lot of pending tasks after Christmas's holidays
and the sickness.

On Thu 2024-02-01 13:12:39, Sreenath Vijayan wrote:
> Hi,
> 
> This patch series enables one to dump the messages in printk ring
> buffer unless all CPUs are locked up. This is useful to view the
> kernel messages when terminal is unresponsive to enter commands
> like dmesg and syslog services are also disabled, especially on
> embedded targets.

What is the exact scenario for this feature, please?

IMHO, rewinding the entire log on an unresponsive terminal
has a questionable value. Most messages would scroll down
quickly and only the last messages would stay visible.

Also this code would rewind all consoles, including
(slow) serial ones. I wonder if rewind on these consoles
would be useful as well.

That said, I am not completely against this feature.
I just want to be sure that it does what you expect.

Best Regards,
Petr

On Wed, Feb 07, 2024 at 03:12:09PM +0100, Petr Mladek wrote:
> Hi,
> 
> first, I am sorry for joining the game so late. I was sick
> and have had a lot of pending tasks after Christmas's holidays
> and the sickness.
> 
> On Thu 2024-02-01 13:12:39, Sreenath Vijayan wrote:
> > Hi,
> > 
> > This patch series enables one to dump the messages in printk ring
> > buffer unless all CPUs are locked up. This is useful to view the
> > kernel messages when terminal is unresponsive to enter commands
> > like dmesg and syslog services are also disabled, especially on
> > embedded targets.
> 
> What is the exact scenario for this feature, please?
> 

Sometimes, we do some testing without connecting display or consoles
and when we connect back to console, we find the system in a hung
state and unable to enter any commands. The system maynot be
completely locked up. In this scenario, we use this feature to view
the printk buffer to get some idea about why the system is hung.

Basically, this can be used in any scenario where you are unable
to enter commands due to some reason, but atleast one cpu is in
a condition to process the unbound workqueue. We have found this
useful during nfs hangs also.

> IMHO, rewinding the entire log on an unresponsive terminal
> has a questionable value. Most messages would scroll down
> quickly and only the last messages would stay visible.
> 
> Also this code would rewind all consoles, including
> (slow) serial ones. I wonder if rewind on these consoles
> would be useful as well.
> 
> That said, I am not completely against this feature.
> I just want to be sure that it does what you expect.
> 
> Best Regards,
> Petr

We mostly use this to display the kernel messages on serial
consoles and have found it to be working fine. In this case,
we use some terminal emulator and can scroll through the output.

Regards,
Sreenath