The per-console loglevel feature documentation could use some practical
guidance. This commit adds:

- Examples section covering runtime configuration, effective loglevel
  checking, and boot-time configuration
- Common use case demonstrating high-performance netconsole with quiet
  serial console fallback
- Performance impact section explaining how per-console loglevels reduce
  latency by filtering messages before slow console writes
- Troubleshooting section addressing common issues like messages not
  appearing, loglevel constraints, and minimum console loglevel
- Edge cases section documenting behavior with concurrent writes,
  console unregistration, and global loglevel changes

The guidance interleaves advice about many parts of this patchset, so
let's have it in a distinct commit.

This documentation will help users understand how to effectively use and
debug per-console loglevels.

Signed-off-by: Chris Down <chris@chrisdown.name>
---
 .../admin-guide/per-console-loglevel.rst      | 152 ++++++++++++++++++
 1 file changed, 152 insertions(+)

diff --git a/Documentation/admin-guide/per-console-loglevel.rst b/Documentation/admin-guide/per-console-loglevel.rst
index 4908d5d8ed4f..69eede12e20f 100644
--- a/Documentation/admin-guide/per-console-loglevel.rst
+++ b/Documentation/admin-guide/per-console-loglevel.rst
@@ -97,6 +97,158 @@ are using ``ttyS0``, the console backing it can be viewed at
   This will be in effect if no other global control overrides it. Look at
   ``effective_loglevel`` and ``effective_loglevel_source`` to verify that.
 
+Examples
+--------
+
+Setting per-console loglevel at runtime
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Set serial console to only show warnings and above (level 4)::
+
+    echo 4 > /sys/class/console/ttyS0/loglevel
+
+Set netconsole to show info and above (level 6)::
+
+    echo 6 > /sys/class/console/netcon0/loglevel
+
+Reset a console to use the global loglevel::
+
+    echo -1 > /sys/class/console/ttyS0/loglevel
+
+Checking effective loglevel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Check what loglevel is actually in effect for a console::
+
+    $ cat /sys/class/console/ttyS0/effective_loglevel
+    4
+    $ cat /sys/class/console/ttyS0/effective_loglevel_source
+    local
+
+If the source shows ``global``, the console is using the global loglevel.
+If it shows ``local``, the console is using its per-console loglevel.
+If it shows ``ignore_loglevel``, all loglevel controls are being ignored.
+
+Boot-time configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Set different loglevels for different consoles at boot::
+
+    console=ttyS0,115200n8,loglevel:3 console=tty0,loglevel:5
+
+This sets the serial console (ttyS0) to level 3 (KERN_ERR) and the VGA
+console (tty0) to level 5 (KERN_NOTICE).
+
+For netconsole::
+
+    netconsole=@/,@192.168.1.1/ console=netcon0,loglevel:6
+
+Common use case - high performance with serial fallback
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A common configuration is to set netconsole to a verbose level for normal
+debugging, while keeping the serial console quiet to avoid performance impact,
+but still available for emergencies::
+
+    # Netconsole gets INFO and above (verbose)
+    echo 6 > /sys/class/console/netcon0/loglevel
+
+    # Serial console gets only WARN and above (quiet, for emergencies)
+    echo 4 > /sys/class/console/ttyS0/loglevel
+
+This allows you to see informational messages on the fast netconsole without
+the latency impact of writing them to the slow serial port.
+
+Performance Impact
+------------------
+
+When a console has a higher (less verbose) loglevel than the global level,
+messages that would normally be sent to that console are filtered out before
+the console write callback is invoked. This eliminates the latency that would
+be incurred by writing those messages to slow consoles (e.g., serial ports).
+
+For example, setting a serial console to WARN level (4) while keeping
+netconsole at INFO level (6) prevents INFO and NOTICE messages from being
+written to the slow serial port, reducing application stalls during verbose
+logging periods.
+
+Serial console writes can take tens of milliseconds per message. During
+periods of heavy logging (e.g., during network debugging or block I/O tracing),
+this can cause significant application-level stalls. By setting a higher
+per-console loglevel for the serial console, you can avoid these stalls while
+still capturing all messages on faster consoles like netconsole.
+
+Troubleshooting
+---------------
+
+Messages not appearing on console despite setting loglevel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Check effective loglevel source::
+
+       cat /sys/class/console/<name>/effective_loglevel_source
+
+   If it shows ``ignore_loglevel``, you have the ``printk.ignore_loglevel``
+   kernel parameter set, which overrides all level controls. Remove it from
+   your kernel command line or set it to N in sysfs::
+
+       echo N > /sys/module/printk/parameters/ignore_loglevel
+
+2. Check if per-console loglevels are being ignored::
+
+       cat /sys/module/printk/parameters/ignore_per_console_loglevel
+
+   If it shows ``Y``, per-console settings are disabled. Set it to N::
+
+       echo N > /sys/module/printk/parameters/ignore_per_console_loglevel
+
+3. Verify the message priority is high enough::
+
+       cat /sys/class/console/<name>/effective_loglevel
+
+   Messages must have priority less than this value to appear. For example,
+   if effective_loglevel is 4, only messages with priority 0-3 (EMERG, ALERT,
+   CRIT, ERR) will be printed. If you want to see WARN messages (priority 4),
+   you need to increase the effective_loglevel to at least 5.
+
+Cannot set loglevel to 0
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Per-console loglevels cannot be set to 0 (KERN_EMERG). This is by design, as
+level 0 is reserved for the most critical system messages that should always
+go to all consoles. To use the global loglevel, set the per-console loglevel
+to -1::
+
+    echo -1 > /sys/class/console/<name>/loglevel
+
+Setting below minimum_console_loglevel fails
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you get an error when trying to set a loglevel, check the system-wide
+minimum::
+
+    cat /proc/sys/kernel/console_loglevel
+
+Per-console loglevels cannot be set below this minimum. This is a safety
+feature to ensure critical messages are always visible.
+
+Edge cases
+~~~~~~~~~~
+
+**Setting all consoles to high loglevels**: If you set all consoles to
+very high loglevels (e.g., 1 or 2), most messages won't appear anywhere.
+This is valid but probably not what you want. Keep at least one console
+at a reasonable level for monitoring.
+
+**Console unregistration while sysfs file is open**: If a console is
+unregistered (e.g., module unloaded) while you have its sysfs files open,
+the files will become stale. Close and reopen them, or they will eventually
+return errors.
+
+**Global loglevel changes**: If you change the global console_loglevel
+via sysctl, consoles set to -1 (use global) will immediately reflect the
+new level. Consoles with explicit per-console levels are unaffected.
+
 Deprecated
 ~~~~~~~~~~
 
-- 
2.51.2

On 2025-11-19, Chris Down <chris@chrisdown.name> wrote:
> diff --git a/Documentation/admin-guide/per-console-loglevel.rst b/Documentation/admin-guide/per-console-loglevel.rst
> index 4908d5d8ed4f..69eede12e20f 100644
> --- a/Documentation/admin-guide/per-console-loglevel.rst
> +++ b/Documentation/admin-guide/per-console-loglevel.rst
> @@ -97,6 +97,158 @@ are using ``ttyS0``, the console backing it can be viewed at
>    This will be in effect if no other global control overrides it. Look at
>    ``effective_loglevel`` and ``effective_loglevel_source`` to verify that.
>  
> +Examples
> +--------
> +
> +Setting per-console loglevel at runtime
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Set serial console to only show warnings and above (level 4)::
> +
> +    echo 4 > /sys/class/console/ttyS0/loglevel

Setting it to 4 would only show "above warnings" ... you will not see
warning messages.

> +
> +Set netconsole to show info and above (level 6)::
> +
> +    echo 6 > /sys/class/console/netcon0/loglevel

Setting it to 6 would only show "above info" ... you will not see info
messages.

> +Common use case - high performance with serial fallback
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +A common configuration is to set netconsole to a verbose level for normal
> +debugging, while keeping the serial console quiet to avoid performance impact,
> +but still available for emergencies::
> +
> +    # Netconsole gets INFO and above (verbose)
> +    echo 6 > /sys/class/console/netcon0/loglevel

6 only shows "above INFO" ... you will not see INFO.

> +
> +    # Serial console gets only WARN and above (quiet, for emergencies)
> +    echo 4 > /sys/class/console/ttyS0/loglevel

4 only shows "above WARN" ... you will not see WARN.

> +
> +This allows you to see informational messages on the fast netconsole without
> +the latency impact of writing them to the slow serial port.
> +
> +Performance Impact
> +------------------
> +
> +When a console has a higher (less verbose) loglevel than the global level,
> +messages that would normally be sent to that console are filtered out before
> +the console write callback is invoked. This eliminates the latency that would
> +be incurred by writing those messages to slow consoles (e.g., serial ports).
> +
> +For example, setting a serial console to WARN level (4) while keeping
> +netconsole at INFO level (6) prevents INFO and NOTICE messages from being
> +written to the slow serial port, reducing application stalls during verbose
> +logging periods.

Setting serial to 4 will also prevent WARN messages.

> +
> +Serial console writes can take tens of milliseconds per message. During
> +periods of heavy logging (e.g., during network debugging or block I/O tracing),
> +this can cause significant application-level stalls. By setting a higher
> +per-console loglevel for the serial console, you can avoid these stalls while
> +still capturing all messages on faster consoles like netconsole.
> +
> +Troubleshooting
> +---------------
> +
> +Messages not appearing on console despite setting loglevel
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +1. Check effective loglevel source::
> +
> +       cat /sys/class/console/<name>/effective_loglevel_source
> +
> +   If it shows ``ignore_loglevel``, you have the ``printk.ignore_loglevel``
> +   kernel parameter set, which overrides all level controls. Remove it from
> +   your kernel command line or set it to N in sysfs::
> +
> +       echo N > /sys/module/printk/parameters/ignore_loglevel
> +
> +2. Check if per-console loglevels are being ignored::
> +
> +       cat /sys/module/printk/parameters/ignore_per_console_loglevel
> +
> +   If it shows ``Y``, per-console settings are disabled. Set it to N::
> +
> +       echo N > /sys/module/printk/parameters/ignore_per_console_loglevel
> +
> +3. Verify the message priority is high enough::
> +
> +       cat /sys/class/console/<name>/effective_loglevel
> +
> +   Messages must have priority less than this value to appear. For example,
> +   if effective_loglevel is 4, only messages with priority 0-3 (EMERG, ALERT,
> +   CRIT, ERR) will be printed. If you want to see WARN messages (priority 4),
> +   you need to increase the effective_loglevel to at least 5.

Yay! Here it is documented correctly. ;-)

> +
> +Cannot set loglevel to 0
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Per-console loglevels cannot be set to 0 (KERN_EMERG). This is by design, as
> +level 0 is reserved for the most critical system messages that should always
> +go to all consoles. To use the global loglevel, set the per-console loglevel
> +to -1::
> +
> +    echo -1 > /sys/class/console/<name>/loglevel
> +
> +Setting below minimum_console_loglevel fails
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +If you get an error when trying to set a loglevel, check the system-wide
> +minimum::
> +
> +    cat /proc/sys/kernel/console_loglevel
> +
> +Per-console loglevels cannot be set below this minimum. This is a safety
> +feature to ensure critical messages are always visible.
> +
> +Edge cases
> +~~~~~~~~~~
> +
> +**Setting all consoles to high loglevels**: If you set all consoles to
> +very high loglevels (e.g., 1 or 2), most messages won't appear anywhere.
> +This is valid but probably not what you want. Keep at least one console
> +at a reasonable level for monitoring.

Keep in mind that all messages (regardless of loglevel) are always
available to userspace via dmesg or syslogd. So saying "messages won't
appear anywhere" is a bit misleading.

John Ogness

On Wed 2025-11-19 03:08:01, Chris Down wrote:
> The per-console loglevel feature documentation could use some practical
> guidance. This commit adds:
> 
> - Examples section covering runtime configuration, effective loglevel
>   checking, and boot-time configuration
> - Common use case demonstrating high-performance netconsole with quiet
>   serial console fallback
> - Performance impact section explaining how per-console loglevels reduce
>   latency by filtering messages before slow console writes
> - Troubleshooting section addressing common issues like messages not
>   appearing, loglevel constraints, and minimum console loglevel

I would remove the section about the "minimum console loglevel",
see below.

> - Edge cases section documenting behavior with concurrent writes,
>   console unregistration, and global loglevel changes
> 
> The guidance interleaves advice about many parts of this patchset, so
> let's have it in a distinct commit.
> 
> This documentation will help users understand how to effectively use and
> debug per-console loglevels.
> 
> +Performance Impact
> +------------------
> +
> +When a console has a higher (less verbose) loglevel than the global level,
> +messages that would normally be sent to that console are filtered out before
> +the console write callback is invoked. This eliminates the latency that would
> +be incurred by writing those messages to slow consoles (e.g., serial ports).

The above section is either confusing or wrong ;-) A higher loglevel
number means that more loglevel are allowed and the console is more verbose.

> +For example, setting a serial console to WARN level (4) while keeping
> +netconsole at INFO level (6) prevents INFO and NOTICE messages from being
> +written to the slow serial port, reducing application stalls during verbose
> +logging periods.
> +
> +Serial console writes can take tens of milliseconds per message. During
> +periods of heavy logging (e.g., during network debugging or block I/O tracing),
> +this can cause significant application-level stalls. By setting a higher
> +per-console loglevel for the serial console, you can avoid these stalls while
> +still capturing all messages on faster consoles like netconsole.

This is true for legacy consoles. The drivers converted to nbcon
API offload messages to a kthread when the system is working properly.
It prevents the stalls but the console might be far behind and
the oldest messages can get lost when the log buffer is full.
And the stalls are still possible when the system enters an
emergency, e.g. Oops, still, or WARN().

I think how to update this section. I would write something like:

Performance Impact
------------------

Kernel messages used to be flushed to the consoles immediately even
from a context where the scheduling is not possible. It increases
a chance to see the messages even when the system is in a bad state.
But it might cause significant application-level stalls (e.g., during
network debugging or block I/O tracing). Note that serial console
writes can take tens of milliseconds per message.

The console drivers are being converted to nbcon API (the letter 'N'
in /proc/consoles output). These drivers write the messages in a
dedicated kthreads when the system is working properly. It reduces
the risk of stalls. But the messages are still flushed immediately
when the system detects an emergency situation, for example Oops,
stall, or a warning. Also the messages can get lost when the ring
buffer is full and the console driver is far behind with flushing.

For example, setting a serial console to WARN level (4) while keeping
netconsole at INFO level (6) prevents INFO and NOTICE messages from being
written to the slow serial port. It reduces the risk of application
stalls or message loses during verbose logging periods.

> +Setting below minimum_console_loglevel fails
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +If you get an error when trying to set a loglevel, check the system-wide
> +minimum::
> +
> +    cat /proc/sys/kernel/console_loglevel
> +
> +Per-console loglevels cannot be set below this minimum. This is a safety
> +feature to ensure critical messages are always visible.

This section is not valid anymore. I think that it is from an earlier
version of the patchset where the per-console loglevel only allowed
to reduce the verbosity. But this version allows to simply replace
the global setting to any valid value.

Or the section was about the "minimum_console_loglevel" variable.
But it is basically hardcoded to 1. I doubt that anyone
modifies it in practice.

I would just remove this section..

Best Regards,
Petr