A new module parameter (ignore_per_console_loglevel) is added, which can
be set via the kernel command line or at runtime through
/sys/module/printk/parameters/ignore_per_console_loglevel. When set, the
per-console loglevels are ignored, and the global console loglevel
(console_loglevel) is used for all consoles.
During sysrq, we temporarily disable per-console loglevels to ensure all
requisite messages are printed to the console. This is necessary because
sysrq is often used in dire circumstances where access to
/sys/class/console may not be trivially possible.
Additionally, the syslog actions SYSLOG_ACTION_CONSOLE_ON and
SYSLOG_ACTION_CONSOLE_OFF are augmented to save and restore the state of
ignore_per_console_loglevel. This allows administrators to enable or
disable per-console loglevels dynamically using the syslog() system
call, as supported in userspace by things like dmesg.
This is useful when debugging issues with message emission, or when
needing to quickly break glass and revert to global loglevel only.
Signed-off-by: Chris Down <chris@chrisdown.name>
---
Documentation/admin-guide/index.rst | 1 +
.../admin-guide/per-console-loglevel.rst | 70 +++++++++++++++++++
MAINTAINERS | 1 +
drivers/tty/sysrq.c | 5 ++
include/linux/printk.h | 2 +
kernel/printk/printk.c | 39 ++++++++++-
6 files changed, 116 insertions(+), 2 deletions(-)
create mode 100644 Documentation/admin-guide/per-console-loglevel.rst
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index e85b1adf5908..366a08a1eee2 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -119,6 +119,7 @@ configure specific aspects of kernel behavior to your liking.
namespaces/index
numastat
parport
+ per-console-loglevel
perf-security
pm/index
pnp
diff --git a/Documentation/admin-guide/per-console-loglevel.rst b/Documentation/admin-guide/per-console-loglevel.rst
new file mode 100644
index 000000000000..1ec7608f94b0
--- /dev/null
+++ b/Documentation/admin-guide/per-console-loglevel.rst
@@ -0,0 +1,70 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _per_console_loglevel:
+
+Per-console loglevel support
+============================
+
+Motivation
+----------
+
+Consoles can have vastly different latencies and throughputs. For example,
+writing a message to the serial console can take on the order of tens of
+milliseconds to get the UART to successfully write a message. While this might
+be fine for a single, one-off message, this can cause significant
+application-level stalls in situations where the kernel writes large amounts of
+information to the console.
+
+This means that while you might want to send at least INFO level messages to
+(for example) netconsole, which is relatively fast, you may only want to send
+at least WARN level messages to the serial console. This permits debugging
+using the serial console in cases that netconsole doesn't receive messages
+during particularly bad system issues, while still keeping the noise low enough
+to avoid inducing latency in userspace applications.
+
+Loglevel
+--------
+
+Kernel loglevels are defined thus:
+
++---+--------------+-----------------------------------+
+| 0 | KERN_EMERG | system is unusable |
++---+--------------+-----------------------------------+
+| 1 | KERN_ALERT | action must be taken immediately |
++---+--------------+-----------------------------------+
+| 2 | KERN_CRIT | critical conditions |
++---+--------------+-----------------------------------+
+| 3 | KERN_ERR | error conditions |
++---+--------------+-----------------------------------+
+| 4 | KERN_WARNING | warning conditions |
++---+--------------+-----------------------------------+
+| 5 | KERN_NOTICE | normal but significant condition |
++---+--------------+-----------------------------------+
+| 6 | KERN_INFO | informational |
++---+--------------+-----------------------------------+
+| 7 | KERN_DEBUG | debug-level messages |
++---+--------------+-----------------------------------+
+
+Tunables
+--------
+
+In order to allow tuning per-console loglevels, the following controls exist:
+
+Global
+~~~~~~
+
+The global loglevel is set by the ``kernel.console_loglevel`` sysctl, which can
+also be set as ``loglevel=`` on the kernel command line.
+
+The printk module also takes two parameters which modify this behaviour
+further:
+
+* ``ignore_loglevel`` on the kernel command line or set in printk parameters:
+ Emit all messages. All other controls are ignored if this is present.
+
+* ``ignore_per_console_loglevel`` on the kernel command line or set in printk
+ parameters: Ignore all per-console loglevels and use the global loglevel.
+
+The default value for ``kernel.console_loglevel`` comes from
+``CONFIG_CONSOLE_LOGLEVEL_DEFAULT``, or ``CONFIG_CONSOLE_LOGLEVEL_QUIET`` if
+``quiet`` is passed on the kernel command line.
diff --git a/MAINTAINERS b/MAINTAINERS
index a27407950242..36490a1fc721 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18449,6 +18449,7 @@ R: John Ogness <john.ogness@linutronix.de>
R: Sergey Senozhatsky <senozhatsky@chromium.org>
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux.git
+F: Documentation/admin-guide/per-console-loglevel.rst
F: include/linux/printk.h
F: kernel/printk/
diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index daa9fe7dad54..84befb6d8d82 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -598,6 +598,7 @@ static void __sysrq_put_key_op(u8 key, const struct sysrq_key_op *op_p)
void __handle_sysrq(u8 key, bool check_mask)
{
const struct sysrq_key_op *op_p;
+ bool orig_ignore_per_console_loglevel;
int orig_log_level;
int orig_suppress_printk;
int i;
@@ -616,6 +617,9 @@ void __handle_sysrq(u8 key, bool check_mask)
orig_log_level = console_loglevel;
console_loglevel = CONSOLE_LOGLEVEL_DEFAULT;
+ orig_ignore_per_console_loglevel = ignore_per_console_loglevel;
+ ignore_per_console_loglevel = true;
+
op_p = __sysrq_get_key_op(key);
if (op_p) {
/*
@@ -651,6 +655,7 @@ void __handle_sysrq(u8 key, bool check_mask)
rcu_read_unlock();
rcu_sysrq_end();
+ ignore_per_console_loglevel = orig_ignore_per_console_loglevel;
suppress_printk = orig_suppress_printk;
}
diff --git a/include/linux/printk.h b/include/linux/printk.h
index 5fbd6b7f1ab4..0053533dcfec 100644
--- a/include/linux/printk.h
+++ b/include/linux/printk.h
@@ -73,6 +73,8 @@ extern int console_printk[];
#define minimum_console_loglevel (console_printk[2])
#define default_console_loglevel (console_printk[3])
+extern bool ignore_per_console_loglevel;
+
extern void console_verbose(void);
/* strlen("ratelimit") + 1 */
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 2e99b63efb46..055619c5c7e8 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -103,6 +103,9 @@ DEFINE_STATIC_SRCU(console_srcu);
*/
int __read_mostly suppress_printk;
+/* The sysrq infrastructure needs this even on !CONFIG_PRINTK. */
+bool __read_mostly ignore_per_console_loglevel;
+
#ifdef CONFIG_LOCKDEP
static struct lockdep_map console_lock_dep_map = {
.name = "console_lock"
@@ -1287,9 +1290,21 @@ module_param(ignore_loglevel, bool, S_IRUGO | S_IWUSR);
MODULE_PARM_DESC(ignore_loglevel,
"ignore loglevel setting (prints all kernel messages to the console)");
+static int __init ignore_per_console_loglevel_setup(char *str)
+{
+ ignore_per_console_loglevel = true;
+ return 0;
+}
+
+early_param("ignore_per_console_loglevel", ignore_per_console_loglevel_setup);
+module_param(ignore_per_console_loglevel, bool, 0644);
+MODULE_PARM_DESC(
+ ignore_per_console_loglevel,
+ "ignore per-console loglevel setting (only respect global console loglevel)");
+
bool per_console_loglevel_is_set(const struct console *con)
{
- return con && (READ_ONCE(con->level) > 0);
+ return !ignore_per_console_loglevel && con && (READ_ONCE(con->level) > 0);
}
/*
@@ -1298,6 +1313,16 @@ bool per_console_loglevel_is_set(const struct console *con)
* 1. con->level. The locally set, console-specific loglevel. Optional, only
* valid if >0.
* 2. console_loglevel. The default global console loglevel, always present.
+ *
+ * The behaviour can be further changed by the following printk module
+ * parameters:
+ *
+ * 1. ignore_loglevel. Can be set at boot or at runtime with
+ * /sys/module/printk/parameters/ignore_loglevel. Overrides absolutely
+ * everything since it's used to debug.
+ * 2. ignore_per_console_loglevel. Existing per-console loglevel values are left
+ * intact, but are ignored in favour of console_loglevel as long as this is
+ * true. Also manipulated through syslog(SYSLOG_ACTION_CONSOLE_{ON,OFF}).
*/
enum loglevel_source
console_effective_loglevel_source(const struct console *con)
@@ -1796,6 +1821,7 @@ int do_syslog(int type, char __user *buf, int len, int source)
struct printk_info info;
bool clear = false;
static int saved_console_loglevel = LOGLEVEL_DEFAULT;
+ static int saved_ignore_per_console_loglevel;
int error;
error = check_syslog_permissions(type, source);
@@ -1836,19 +1862,28 @@ int do_syslog(int type, char __user *buf, int len, int source)
break;
/* Disable logging to console */
case SYSLOG_ACTION_CONSOLE_OFF:
- if (saved_console_loglevel == LOGLEVEL_DEFAULT)
+ if (saved_console_loglevel == LOGLEVEL_DEFAULT) {
saved_console_loglevel = console_loglevel;
+ saved_ignore_per_console_loglevel =
+ ignore_per_console_loglevel;
+ }
console_loglevel = minimum_console_loglevel;
+ ignore_per_console_loglevel = true;
break;
/* Enable logging to console */
case SYSLOG_ACTION_CONSOLE_ON:
if (saved_console_loglevel != LOGLEVEL_DEFAULT) {
console_loglevel = saved_console_loglevel;
+ ignore_per_console_loglevel =
+ saved_ignore_per_console_loglevel;
saved_console_loglevel = LOGLEVEL_DEFAULT;
}
break;
/* Set level of messages printed to console */
case SYSLOG_ACTION_CONSOLE_LEVEL:
+ if (!ignore_per_console_loglevel)
+ pr_warn_once(
+ "SYSLOG_ACTION_CONSOLE_LEVEL is ignored by consoles with an explicitly set per-console loglevel, see Documentation/admin-guide/per-console-loglevel.rst\n");
if (len < 1 || len > 8)
return -EINVAL;
if (len < minimum_console_loglevel)
--
2.46.0On Mon 2024-10-28 16:45:40, Chris Down wrote:
> A new module parameter (ignore_per_console_loglevel) is added, which can
> be set via the kernel command line or at runtime through
> /sys/module/printk/parameters/ignore_per_console_loglevel. When set, the
> per-console loglevels are ignored, and the global console loglevel
> (console_loglevel) is used for all consoles.
>
> During sysrq, we temporarily disable per-console loglevels to ensure all
> requisite messages are printed to the console. This is necessary because
> sysrq is often used in dire circumstances where access to
> /sys/class/console may not be trivially possible.
>
> Additionally, the syslog actions SYSLOG_ACTION_CONSOLE_ON and
> SYSLOG_ACTION_CONSOLE_OFF are augmented to save and restore the state of
> ignore_per_console_loglevel. This allows administrators to enable or
> disable per-console loglevels dynamically using the syslog() system
> call, as supported in userspace by things like dmesg.
>
> This is useful when debugging issues with message emission, or when
> needing to quickly break glass and revert to global loglevel only.
>
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -1298,6 +1313,16 @@ bool per_console_loglevel_is_set(const struct console *con)
> * 1. con->level. The locally set, console-specific loglevel. Optional, only
> * valid if >0.
> * 2. console_loglevel. The default global console loglevel, always present.
I think that I have suggested to remove the above comment because
it was obvious from the code. I am in doubts now because use
extendended it below ;-)
> + * The behaviour can be further changed by the following printk module
> + * parameters:
> + *
> + * 1. ignore_loglevel. Can be set at boot or at runtime with
> + * /sys/module/printk/parameters/ignore_loglevel. Overrides absolutely
> + * everything since it's used to debug.
> + * 2. ignore_per_console_loglevel. Existing per-console loglevel values are left
> + * intact, but are ignored in favour of console_loglevel as long as this is
> + * true. Also manipulated through syslog(SYSLOG_ACTION_CONSOLE_{ON,OFF}).
I like that it is summarized in one place. I like the comment after all ;-)
That said, it is also nicely summarized in
Documentation/admin-guide/per-console-loglevel.rst
So, it might be enough to mention it here.
> */
> enum loglevel_source
> console_effective_loglevel_source(const struct console *con)
Best Regards,
Petr
On Mon 2024-10-28 16:45:40, Chris Down wrote:
> A new module parameter (ignore_per_console_loglevel) is added, which can
> be set via the kernel command line or at runtime through
> /sys/module/printk/parameters/ignore_per_console_loglevel. When set, the
> per-console loglevels are ignored, and the global console loglevel
> (console_loglevel) is used for all consoles.
>
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -1836,19 +1862,28 @@ int do_syslog(int type, char __user *buf, int len, int source)
> break;
> /* Disable logging to console */
> case SYSLOG_ACTION_CONSOLE_OFF:
> - if (saved_console_loglevel == LOGLEVEL_DEFAULT)
> + if (saved_console_loglevel == LOGLEVEL_DEFAULT) {
> saved_console_loglevel = console_loglevel;
> + saved_ignore_per_console_loglevel =
> + ignore_per_console_loglevel;
> + }
> console_loglevel = minimum_console_loglevel;
> + ignore_per_console_loglevel = true;
> break;
> /* Enable logging to console */
> case SYSLOG_ACTION_CONSOLE_ON:
> if (saved_console_loglevel != LOGLEVEL_DEFAULT) {
> console_loglevel = saved_console_loglevel;
> + ignore_per_console_loglevel =
> + saved_ignore_per_console_loglevel;
> saved_console_loglevel = LOGLEVEL_DEFAULT;
> }
> break;
> /* Set level of messages printed to console */
> case SYSLOG_ACTION_CONSOLE_LEVEL:
> + if (!ignore_per_console_loglevel)
> + pr_warn_once(
> + "SYSLOG_ACTION_CONSOLE_LEVEL is ignored by consoles with an explicitly set per-console loglevel, see Documentation/admin-guide/per-console-loglevel.rst\n");
I see this warning during every boot because rsyslogd() modifies the
global loglevel.
I am afraid that admins might not like it. I might live in dreams but
I guess that everyone would like to reach a clean boot without any
warning.
One the other hand, we should warn when it does not work as expected.
A compromise would be to warn only when there is a console with
the console specific loglevel set.
I am not sure if we have already discussed this in the past.
But I would prefer the compromise after all.
What do you think, please?
Best Regards,
Petr
Petr Mladek writes:
>I see this warning during every boot because rsyslogd() modifies the
>global loglevel.
>
>[...]
>
>I am not sure if we have already discussed this in the past.
>But I would prefer the compromise after all.
I initially implemented that way until v4, but during the v3 review, as I
understood it you recommended changing it to use !ignore_per_console_loglevel
based on the assumption that SYSLOG_ACTION_CONSOLE_{ON,OFF} wasn't widely used.
Maybe I misunderstood what was intended?
Happy to revert to the previous approach with warn_on_local_loglevel(), just
let me know :-)
On Thu 2024-11-14 13:53:40, Chris Down wrote:
> Petr Mladek writes:
> > I see this warning during every boot because rsyslogd() modifies the
> > global loglevel.
> >
> > [...]
> >
> > I am not sure if we have already discussed this in the past.
> > But I would prefer the compromise after all.
>
> I initially implemented that way until v4, but during the v3 review, as I
> understood it you recommended changing it to use
> !ignore_per_console_loglevel based on the assumption that
> SYSLOG_ACTION_CONSOLE_{ON,OFF} wasn't widely used. Maybe I misunderstood
> what was intended?
The current solution of SYSLOG_ACTION_CONSOLE_{ON,OFF} is fine. I
still hope that it is not used much.
My concern is with SYSLOG_ACTION_CONSOLE_LEVEL. It seems to be used
by rsyslogd.
But wait! I see the warning only on SLE15-SP3 which I use for
testing. It is a pretty old system. I see that rsyslogd is not
longer used on never systems. I guess that it has been obsoleted
by systemd journal.
> Happy to revert to the previous approach with warn_on_local_loglevel(), just
> let me know :-)
No, I take it back ;-) Let's keep it simple as it is done in this patch [v6].
We could always add more conditions around the warning when people
complains.
Best Regards,
Petr
On Mon 2024-10-28 16:45:40, Chris Down wrote:
> A new module parameter (ignore_per_console_loglevel) is added, which can
> be set via the kernel command line or at runtime through
> /sys/module/printk/parameters/ignore_per_console_loglevel. When set, the
> per-console loglevels are ignored, and the global console loglevel
> (console_loglevel) is used for all consoles.
>
> During sysrq, we temporarily disable per-console loglevels to ensure all
> requisite messages are printed to the console. This is necessary because
> sysrq is often used in dire circumstances where access to
> /sys/class/console may not be trivially possible.
I have just pushed a patchset which removed the console_loglevel
manipulation from sysrq, see
https://lore.kernel.org/r/20241105-printk-loud-con-v2-0-bd3ecdf7b0e4@suse.com
As a result, the change in drivers/tty/sysrq.c is not needed anymore.
Note that the other patchset causes some conflict with this patchset.
But they does not seem to be hard to resolved:
1st conflict is in boot_delay_msec(). But the affected logic
has actually been moved to printk_delay(). As a result,
boot_delay_msec() might stay as it is:
static void boot_delay_msec(void)
{
unsigned long long k;
unsigned long timeout;
if (boot_delay == 0 || system_state >= SYSTEM_RUNNING)
return;
k = (unsigned long long)loops_per_msec * boot_delay;
timeout = jiffies + msecs_to_jiffies(boot_delay);
while (k) {
k--;
cpu_relax();
/*
* use (volatile) jiffies to prevent
* compiler reduction; loop termination via jiffies
* is secondary and may or may not happen.
*/
if (time_after(jiffies, timeout))
break;
touch_nmi_watchdog();
}
}
Instead, we should add check of is_printk_force_console() into
printk_delay(). I suggest to do it the following way:
static bool suppress_message_printing_everywhere(int level)
{
bool suppress_everywhere = true;
struct console *con;
int cookie;
cookie = console_srcu_read_lock();
for_each_console_srcu(con) {
if (!suppress_message_printing(level, con)) {
suppress_everywhere = false;
break;
}
}
console_srcu_read_unlock(cookie);
return suppress_everywhere;
}
static inline void printk_delay(int level)
{
if (!is_printk_force_console() &&
suppress_message_printing_everywhere(level))
return;
boot_delay_msec();
if (unlikely(printk_delay_msec)) {
int m = printk_delay_msec;
while (m--) {
mdelay(1);
touch_nmi_watchdog();
}
}
}
2nd conflict is in printk_get_next_message(). I suggest to do
something like:
force_con = r.info->flags & LOG_FORCE_CON;
/*
* Skip records that are not forced to be printed on consoles and that
* has level above the console loglevel. Never suppress when used in
* devkmsg_read().
*/
if (!force_con && con && suppress_message_printing(r.info->level, con))
goto out;
Actually, I suggested to pass @con_level instead of @con here.
In which case, we might need something like:
if (con) {
is_extended = console_srcu_read_flags(con) & CON_EXTENDED;
con_level = console_srcu_read_loglevel(con);
} else {
/* Used only by devkmsg_read(). Show all messages there. */
is_extended = true;
con_level = CONSOLE_LOGLEVEL_MOTORMOUTH;
}
[...]
force_con = r.info->flags & LOG_FORCE_CON;
/*
* Skip records that are not forced to be printed on consoles and that
* has level above the console loglevel.
*/
if (!force_con && suppress_message_printing(r.info->level, con_level))
goto out;
I hope that you find there code snippets useful. I provide them
because I feel a bit guilty that I have already merged the other
patchset... ;-)
Best Regards,
Petr
Petr Mladek writes: >On Mon 2024-10-28 16:45:40, Chris Down wrote: >> A new module parameter (ignore_per_console_loglevel) is added, which can >> be set via the kernel command line or at runtime through >> /sys/module/printk/parameters/ignore_per_console_loglevel. When set, the >> per-console loglevels are ignored, and the global console loglevel >> (console_loglevel) is used for all consoles. >> >> During sysrq, we temporarily disable per-console loglevels to ensure all >> requisite messages are printed to the console. This is necessary because >> sysrq is often used in dire circumstances where access to >> /sys/class/console may not be trivially possible. > >I have just pushed a patchset which removed the console_loglevel >manipulation from sysrq, see >https://lore.kernel.org/r/20241105-printk-loud-con-v2-0-bd3ecdf7b0e4@suse.com Noted, so would you like me to change this to be based on current printk/for-next, or is another branch preferred?
On Thu 2024-11-14 14:28:22, Chris Down wrote: > Petr Mladek writes: > > On Mon 2024-10-28 16:45:40, Chris Down wrote: > > > A new module parameter (ignore_per_console_loglevel) is added, which can > > > be set via the kernel command line or at runtime through > > > /sys/module/printk/parameters/ignore_per_console_loglevel. When set, the > > > per-console loglevels are ignored, and the global console loglevel > > > (console_loglevel) is used for all consoles. > > > > > > During sysrq, we temporarily disable per-console loglevels to ensure all > > > requisite messages are printed to the console. This is necessary because > > > sysrq is often used in dire circumstances where access to > > > /sys/class/console may not be trivially possible. > > > > I have just pushed a patchset which removed the console_loglevel > > manipulation from sysrq, see > > https://lore.kernel.org/r/20241105-printk-loud-con-v2-0-bd3ecdf7b0e4@suse.com > Noted, so would you like me to change this to be based on current > printk/for-next, or is another branch preferred? Please, make v7 based on linux-next. It seems that the merge window for 6.13 opens the following week. So that the other patchset should reach the mainline in 6.13. The per-console loglevels would need to wait for 6.14. Best Regards, Petr
© 2016 - 2024 Red Hat, Inc.