This patch series introduces a new configfs attribute that enables sending
messages directly through netconsole without going through the kernel's logging
infrastructure.

This feature allows users to send custom messages, alerts, or status updates
directly to netconsole receivers by writing to
/sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
buffers, and sending msgs to the serial, which could be slow.

At Meta this is currently used in two cases right now (through printk by
now):

  a) When a new workload enters or leave the machine.
  b) From time to time, as a "ping" to make sure the netconsole/machine
  is alive.

The implementation reuses the existing message transmission functions
(send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
message formats.

Regarding code organization, this version uses forward declarations for
send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
within the file. While forward declarations do add a small amount of
redundancy, they avoid the larger churn that would result from moving entire
function definitions.

---
Breno Leitao (4):
      netconsole: extract message fragmentation into send_msg_udp()
      netconsole: Add configfs attribute for direct message sending
      selftests/netconsole: Switch to configfs send_msg interface
      Documentation: netconsole: Document send_msg configfs attribute

 Documentation/networking/netconsole.rst            | 40 +++++++++++++++
 drivers/net/netconsole.c                           | 59 ++++++++++++++++++----
 .../selftests/drivers/net/netcons_sysdata.sh       |  2 +-
 3 files changed, 91 insertions(+), 10 deletions(-)
---
base-commit: ab084f0b8d6d2ee4b1c6a28f39a2a7430bdfa7f0
change-id: 20251127-netconsole_send_msg-89813956dc23

Best regards,
--  
Breno Leitao <leitao@debian.org>

On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
> This patch series introduces a new configfs attribute that enables sending
> messages directly through netconsole without going through the kernel's logging
> infrastructure.
> 
> This feature allows users to send custom messages, alerts, or status updates
> directly to netconsole receivers by writing to
> /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> buffers, and sending msgs to the serial, which could be slow.
> 
> At Meta this is currently used in two cases right now (through printk by
> now):
> 
>   a) When a new workload enters or leave the machine.
>   b) From time to time, as a "ping" to make sure the netconsole/machine
>   is alive.
> 
> The implementation reuses the existing message transmission functions
> (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> message formats.
> 
> Regarding code organization, this version uses forward declarations for
> send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
> within the file. While forward declarations do add a small amount of
> redundancy, they avoid the larger churn that would result from moving entire
> function definitions.

The two questions we need to address here are :
 - why is the message important in the off-host message stream but not
   important in local dmesg stream. You mention "serial, which could be
   slow" - we need more details here.
 - why do we need the kernel API, netcons is just a UDP message, which
   is easy enough to send from user space. A little bit more detail
   about the advantages would be good to have.
The 2nd point is trivial, the first one is what really gives me pause.
Why do we not care about the logs on host? If the serial is very slow
presumably it impacts a lot of things, certainly boot speed, so...
perhaps it should be configured to only log messages at a high level?

Hello Jakub,

On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
> > This patch series introduces a new configfs attribute that enables sending
> > messages directly through netconsole without going through the kernel's logging
> > infrastructure.
> > 
> > This feature allows users to send custom messages, alerts, or status updates
> > directly to netconsole receivers by writing to
> > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > buffers, and sending msgs to the serial, which could be slow.
> > 
> > At Meta this is currently used in two cases right now (through printk by
> > now):
> > 
> >   a) When a new workload enters or leave the machine.
> >   b) From time to time, as a "ping" to make sure the netconsole/machine
> >   is alive.
> > 
> > The implementation reuses the existing message transmission functions
> > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > message formats.
> > 
> > Regarding code organization, this version uses forward declarations for
> > send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
> > within the file. While forward declarations do add a small amount of
> > redundancy, they avoid the larger churn that would result from moving entire
> > function definitions.
> 
> The two questions we need to address here are :
>  - why is the message important in the off-host message stream but not
>    important in local dmesg stream. You mention "serial, which could be
>    slow" - we need more details here.

Thanks for the questions, and I would like to share my view of the world. The
way I see and use netconsole at my company (Meta) is a "kernel message"
on steroids, where it provides more information about the system than
what is available in kernel log buffers (dmesg)

These netconsole messages already have extra data, which provides
information to each message, such as:

 * scheduler configuration (for sched_ext contenxt)
 * THP memory configuration
 * Job/workload running
 * CPU id
 * task->curr name
 * etc
 
So, netconsole already sends extra information today that is not visible
on kernel console (dmesg), and this has proved to be super useful, so
useful that 16 entries are not enough and Gustavo need to do a dynamic
allocation instead of limiting it to 16.

On top of that, printk() has a similar mechanism where extra data is not
printed to the console. printk buffers has a dictionary of structured
data attached to the message that is not printed to the screen, but,
sent through netconsole.

This feature (in this patchset) is just one step ahead, giving some more
power to netconsole, where extra information could be sent beyond what
is in dmesg.

>  - why do we need the kernel API, netcons is just a UDP message, which
>    is easy enough to send from user space. A little bit more detail
>    about the advantages would be good to have.

The primary advantage is leveraging the existing configured netconsole
infrastructure. At Meta, for example, we have a "continuous ping"
mechanism configured by our Configuration Management software that
simply runs 'echo "ping" > /dev/kmsg'.

A userspace solution would require deploying a binary to millons of
machines,  parsing /sys/kernel/configfs/netconsole/cmdline0/configs
and sends packets directly.

While certainly feasible, it's less convenient than using the
existing infrastructure (though I may just be looking for the easier
path here).

> The 2nd point is trivial, the first one is what really gives me pause.
> Why do we not care about the logs on host? If the serial is very slow
> presumably it impacts a lot of things, certainly boot speed, so...

This is spot-on - slow serial definitely impacts things like boot speed.

See my constant complains here, about slow boot

	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/

And the something similar in reboot/kexec path:

	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/

> perhaps it should be configured to only log messages at a high level?

Chris is actually working on per-console log levels to solve exactly
this problem, so we could filter serial console messages while keeping
everything in other consoles (aka netconsole):

	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/

That work has been in progress for years though, and I'm not sure
when/if it'll land upstream. But if it does, we'd be able to have
different log levels per console and then use your suggested approach.

Thanks for the review, and feel free to yell at me if I am missing the
point,
--breno

On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
> On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> > On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:  
> > > This patch series introduces a new configfs attribute that enables sending
> > > messages directly through netconsole without going through the kernel's logging
> > > infrastructure.
> > > 
> > > This feature allows users to send custom messages, alerts, or status updates
> > > directly to netconsole receivers by writing to
> > > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > > buffers, and sending msgs to the serial, which could be slow.
> > > 
> > > At Meta this is currently used in two cases right now (through printk by
> > > now):
> > > 
> > >   a) When a new workload enters or leave the machine.
> > >   b) From time to time, as a "ping" to make sure the netconsole/machine
> > >   is alive.
> > > 
> > > The implementation reuses the existing message transmission functions
> > > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > > message formats.
> > > 
> > > Regarding code organization, this version uses forward declarations for
> > > send_msg_udp() and send_ext_msg_udp() functions rather than relocating them
> > > within the file. While forward declarations do add a small amount of
> > > redundancy, they avoid the larger churn that would result from moving entire
> > > function definitions.  
> > 
> > The two questions we need to address here are :
> >  - why is the message important in the off-host message stream but not
> >    important in local dmesg stream. You mention "serial, which could be
> >    slow" - we need more details here.  
> 
> Thanks for the questions, and I would like to share my view of the world. The
> way I see and use netconsole at my company (Meta) is a "kernel message"
> on steroids, where it provides more information about the system than
> what is available in kernel log buffers (dmesg)
> 
> These netconsole messages already have extra data, which provides
> information to each message, such as:
> 
>  * scheduler configuration (for sched_ext contenxt)
>  * THP memory configuration
>  * Job/workload running
>  * CPU id
>  * task->curr name
>  * etc
>  
> So, netconsole already sends extra information today that is not visible
> on kernel console (dmesg), and this has proved to be super useful, so
> useful that 16 entries are not enough and Gustavo need to do a dynamic
> allocation instead of limiting it to 16.
> 
> On top of that, printk() has a similar mechanism where extra data is not
> printed to the console. printk buffers has a dictionary of structured
> data attached to the message that is not printed to the screen, but,
> sent through netconsole.
> 
> This feature (in this patchset) is just one step ahead, giving some more
> power to netconsole, where extra information could be sent beyond what
> is in dmesg.

Having extra metadata makes sense, since the interpretation happens in
a different environment. But here we're talking about having extra
messages, not extra metadata.

> >  - why do we need the kernel API, netcons is just a UDP message, which
> >    is easy enough to send from user space. A little bit more detail
> >    about the advantages would be good to have.  
> 
> The primary advantage is leveraging the existing configured netconsole
> infrastructure. At Meta, for example, we have a "continuous ping"
> mechanism configured by our Configuration Management software that
> simply runs 'echo "ping" > /dev/kmsg'.
> 
> A userspace solution would require deploying a binary to millons of
> machines,  parsing /sys/kernel/configfs/netconsole/cmdline0/configs
> and sends packets directly.
> 
> While certainly feasible, it's less convenient than using the
> existing infrastructure (though I may just be looking for the easier
> path here).

If this was your objective, instead of having a uAPI for sending
arbitrary message you should be adding some "keepalive" timer / empty
message sender... With the patches are posted you still need something
to run the echo.

> > The 2nd point is trivial, the first one is what really gives me pause.
> > Why do we not care about the logs on host? If the serial is very slow
> > presumably it impacts a lot of things, certainly boot speed, so...  
> 
> This is spot-on - slow serial definitely impacts things like boot speed.
> 
> See my constant complains here, about slow boot
> 
> 	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
> 
> And the something similar in reboot/kexec path:
> 
> 	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/
> 
> > perhaps it should be configured to only log messages at a high level?  
> 
> Chris is actually working on per-console log levels to solve exactly
> this problem, so we could filter serial console messages while keeping
> everything in other consoles (aka netconsole):
> 
> 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/

Excellent! Unless I'm missing more context Chris does seem to be
attacking the problem at a more suitable layer.

> That work has been in progress for years though, and I'm not sure
> when/if it'll land upstream. But if it does, we'd be able to have
> different log levels per console and then use your suggested approach.
> 
> Thanks for the review, and feel free to yell at me if I am missing the
> point,
> --breno
>

On Tue 2025-12-02 10:24:42, Jakub Kicinski wrote:
> On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
> > On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> > > On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:  
> > > > This patch series introduces a new configfs attribute that enables sending
> > > > messages directly through netconsole without going through the kernel's logging
> > > > infrastructure.
> > > > 
> > > > This feature allows users to send custom messages, alerts, or status updates
> > > > directly to netconsole receivers by writing to
> > > > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > > > buffers, and sending msgs to the serial, which could be slow.
> > > > 
> > > > At Meta this is currently used in two cases right now (through printk by
> > > > now):
> > > > 
> > > >   a) When a new workload enters or leave the machine.
> > > >   b) From time to time, as a "ping" to make sure the netconsole/machine
> > > >   is alive.
> > > > 
> > > > The implementation reuses the existing message transmission functions
> > > > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > > > message formats.
> > > > 
> > This feature (in this patchset) is just one step ahead, giving some more
> > power to netconsole, where extra information could be sent beyond what
> > is in dmesg.
> 
> Having extra metadata makes sense, since the interpretation happens in
> a different environment. But here we're talking about having extra
> messages, not extra metadata.
> 
> > > The 2nd point is trivial, the first one is what really gives me pause.
> > > Why do we not care about the logs on host? If the serial is very slow
> > > presumably it impacts a lot of things, certainly boot speed, so...  
> > 
> > This is spot-on - slow serial definitely impacts things like boot speed.
> > 
> > See my constant complains here, about slow boot
> > 
> > 	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
> > 
> > And the something similar in reboot/kexec path:
> > 
> > 	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/
> > 
> > > perhaps it should be configured to only log messages at a high level?  
> > 
> > Chris is actually working on per-console log levels to solve exactly
> > this problem, so we could filter serial console messages while keeping
> > everything in other consoles (aka netconsole):
> > 
> > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> 
> Excellent! Unless I'm missing more context Chris does seem to be
> attacking the problem at a more suitable layer.

This would help to bypass slow serial consoles. But the extra messages
would still get stored into the kernel ring buffer and passed back
to user space logs, for example journalctl.

I do not have strong opinion whether adding the
/sys/kernel/config/netconsole/<target>/send_msg is a good idea or not.

Ah, I have sent it prematurely.

On Thu 2025-12-04 11:46:21, Petr Mladek wrote:
> On Tue 2025-12-02 10:24:42, Jakub Kicinski wrote:
> > On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
> > > On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
> > > > On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:  
> > > > > This patch series introduces a new configfs attribute that enables sending
> > > > > messages directly through netconsole without going through the kernel's logging
> > > > > infrastructure.
> > > > > 
> > > > > This feature allows users to send custom messages, alerts, or status updates
> > > > > directly to netconsole receivers by writing to
> > > > > /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel
> > > > > buffers, and sending msgs to the serial, which could be slow.
> > > > > 
> > > > > At Meta this is currently used in two cases right now (through printk by
> > > > > now):
> > > > > 
> > > > >   a) When a new workload enters or leave the machine.
> > > > >   b) From time to time, as a "ping" to make sure the netconsole/machine
> > > > >   is alive.
> > > > > 
> > > > > The implementation reuses the existing message transmission functions
> > > > > (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended
> > > > > message formats.
> > > > > 
> > > This feature (in this patchset) is just one step ahead, giving some more
> > > power to netconsole, where extra information could be sent beyond what
> > > is in dmesg.
> > 
> > Having extra metadata makes sense, since the interpretation happens in
> > a different environment. But here we're talking about having extra
> > messages, not extra metadata.
> > 
> > > > The 2nd point is trivial, the first one is what really gives me pause.
> > > > Why do we not care about the logs on host? If the serial is very slow
> > > > presumably it impacts a lot of things, certainly boot speed, so...  
> > > 
> > > This is spot-on - slow serial definitely impacts things like boot speed.
> > > 
> > > See my constant complains here, about slow boot
> > > 
> > > 	https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
> > > 
> > > And the something similar in reboot/kexec path:
> > > 
> > > 	https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu/
> > > 
> > > > perhaps it should be configured to only log messages at a high level?  
> > > 
> > > Chris is actually working on per-console log levels to solve exactly
> > > this problem, so we could filter serial console messages while keeping
> > > everything in other consoles (aka netconsole):
> > > 
> > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > 
> > Excellent! Unless I'm missing more context Chris does seem to be
> > attacking the problem at a more suitable layer.
> 
> This would help to bypass slow serial consoles. But the extra messages
> would still get stored into the kernel ring buffer and passed back
> to user space logs, for example journalctl.

It might actually make sense for the "workload enters or leaves" messages.
But I am not sure about the "ping" messages.

> I do not have strong opinion whether adding the
> /sys/kernel/config/netconsole/<target>/send_msg is a good idea or not.

I just wanted to point out that it is not only about slow serial
consoles.

Best Regards,
Petr

On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
> > > > > perhaps it should be configured to only log messages at a high level?  
> > > > 
> > > > Chris is actually working on per-console log levels to solve exactly
> > > > this problem, so we could filter serial console messages while keeping
> > > > everything in other consoles (aka netconsole):
> > > > 
> > > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > > 
> > > Excellent! Unless I'm missing more context Chris does seem to be
> > > attacking the problem at a more suitable layer.
> > 
> > This would help to bypass slow serial consoles. But the extra messages
> > would still get stored into the kernel ring buffer and passed back
> > to user space logs, for example journalctl.
> 
> It might actually make sense for the "workload enters or leaves" messages.
> But I am not sure about the "ping" messages.

Agree. Let me back up and explain my "ping" messages better, which
I think might add more information about this topic.

Meta has millions of servers, and all of them must have netconsole
running 100% of the time.

Of course that this is not reality, and problems happen for different
reasons, the ones that interest me here are:

1) Top of the rack switch MAC address changes (mostly associated with
   network hardware (top of the rack switches and gateway) replacement)
    a) Keep in mind that netconsole target has the destination MAC as
       part of its configuration.

2) Netconsole got associated with the wrong network port, which comes in
   two different flavors.
   a) The machine got provisioned wrongly since day one (Most common
      case)
   b) The machine NIC changed and: 
      i) The target doesn't bind correctly anymore (if netconsole
         target is bound by mac address)
      	   * This is easier to detect, given the target will never be
	     enabled.

3) Netconsd (the daemon that listen to netconsole packets) is buggy or
   dead

4) Network failures across the route


Possible Solutions
==================

In order to detect those issues above, I think the best (or only) way is
to send messages from the host, and check if they got received. If not,
raise an alarm (in the common distributed way).

This could be done in very different ways, tho. Such as:

1) Have a binary in each machine:
	a) This binary reads the netconsole target that is configured,
	   and mimics "ping" UDP netconsole packet.

	Pro: 
	     * It doesn't need any kernel change
	Cons:
	     * It needs to reimplement the netconsole logic in userspace
	     * This needs also a widely distributed binary on all
	       machines

2) Send a ping directly to the console
	a) Something as 'echo ping from $hostname" > /dev/kmsg')

	Pro:
		* No kernel changes
	Cons:
		* These debug messages will be sent to journalctl and to
		  the console, polluting both

3) Using per-loglevel patchset.
	a) Same as above, but, setting netconsole loglevel to DEBUG, while
	   all other consoles to INFO.

	Pro:
		* No changes on netconsole
		* Netconsole "buffers" continues to be synchronized with
		  kernel buffers. Everything in the same page, but,
		  netconsole data has one loglevel higher.
		* Sending a message to netconsole-only message is not
		  special at all. It uses the same workflow we have
		  today, through `/dev/kmsg'
	Cons:
		* Needs to change printk/console code (Chris' patch)
		  that is on review for years now. Will it ever get
		  accepted?
		* These "ping" message will be in kernel buffers and
		  journalctl, and are useless in there (!?)
		* It is not possible to send a message to a single
		  netconsole target.

4) send messages only to netconsole (this patchset)
	Pro:
		* It is easy to test netconsole connective (problem above),
		  without kernel buffers/journal pollution
		* It doesn't depend on the per-loglevel patchset
		* Adds flexibility to netconsole targets.
			- only certain netconsole targets receive
			  certain messages
	Cons:
		* Messages sent to netconsole is a superset of messages in the
		  kernel buffer. In other words, "dmesg" and machine
		  logs/journal will not be able to see messages that
		  were sent directly to netconsole.
			- It might be seen as a back channel (!?)
		* Different netconsole targets may receive different
		  messages. Too much flexibility might be bad (!?)

Anyway, options 1 and 2 are available today. In order to make the
problem easier to solve, we are deciding between approach 3) and 4),
which will require kernel changes, either in printk/console or
netconsole.

Sorry for the long email, I just wanted to do the brain dump about my
view of the world, so, we can decide it from a community perspective in
the open.

Thanks for the discussion,
--breno

On Fri, 5 Dec 2025 02:21:08 -0800 Breno Leitao wrote:
> 1) Have a binary in each machine:

> 2) Send a ping directly to the console

> 3) Using per-loglevel patchset.

> 4) send messages only to netconsole (this patchset)

I think I was alluding that another option (not saying that it's the
best but IIUC your requirements it'd be the best fit)):

5) Add a keepalive configfs knob, if set to a non-zero value netconsole
will send an empty (?) message at given interval

  Pros:
   - truly does not require a user binary to run periodically, netcons
     would set a timer in the kernel
  Cons:
   - does not provide the arbitrary "console bypass" message
     functionality

Hello Jakub,

On Tue, Dec 09, 2025 at 04:37:45PM +0900, Jakub Kicinski wrote:
> On Fri, 5 Dec 2025 02:21:08 -0800 Breno Leitao wrote:
> > 1) Have a binary in each machine:
> 
> > 2) Send a ping directly to the console
> 
> > 3) Using per-loglevel patchset.
> 
> > 4) send messages only to netconsole (this patchset)
> 
> I think I was alluding that another option (not saying that it's the
> best but IIUC your requirements it'd be the best fit)):
> 
> 5) Add a keepalive configfs knob, if set to a non-zero value netconsole
> will send an empty (?) message at given interval
> 
>   Pros:
>    - truly does not require a user binary to run periodically, netcons
>      would set a timer in the kernel
>   Cons:
>    - does not provide the arbitrary "console bypass" message
>      functionality

This is a good idea if we change it slightly. What about a "ping"
configfs item that send sit when I touch it?

Something as:

	# echo 1 > /sys/kernel/configs/<target>/ping
	
And it would ping the host with a predefined "ping" message, and nothing
else.

That would work, for my current problem, honestly.

One drawback compared to a more flexible "send_msg" is that I don't have
complete flexibility on the message format. Thus, if I want to pass
extra information such as a Nonce, timestamp, host state, interface
name, health state, it will not be possible, which is fine for now,
given I am NOT planning to use it at this stage.

Thanks for the idea and discussion,
--breno

On Tue, 9 Dec 2025 09:46:51 -0800 Breno Leitao wrote:
> > I think I was alluding that another option (not saying that it's the
> > best but IIUC your requirements it'd be the best fit)):
> > 
> > 5) Add a keepalive configfs knob, if set to a non-zero value netconsole
> > will send an empty (?) message at given interval
> > 
> >   Pros:
> >    - truly does not require a user binary to run periodically, netcons
> >      would set a timer in the kernel
> >   Cons:
> >    - does not provide the arbitrary "console bypass" message
> >      functionality  
> 
> This is a good idea if we change it slightly. What about a "ping"
> configfs item that send sit when I touch it?
> 
> Something as:
> 
> 	# echo 1 > /sys/kernel/configs/<target>/ping
>
> And it would ping the host with a predefined "ping" message, and nothing
> else.
> 
> That would work, for my current problem, honestly.
> 
> One drawback compared to a more flexible "send_msg" is that I don't have
> complete flexibility on the message format. Thus, if I want to pass
> extra information such as a Nonce, timestamp, host state, interface
> name, health state, it will not be possible, which is fine for now,
> given I am NOT planning to use it at this stage.

If you still want to tickle it from user space periodically, I guess
send_msg is more flexible. I think the main advantage of keepalive
would be to remove the need for periodic userspace work.

On Fri 2025-12-05 02:21:08, Breno Leitao wrote:
> On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
> > > > > > perhaps it should be configured to only log messages at a high level?  
> > > > > 
> > > > > Chris is actually working on per-console log levels to solve exactly
> > > > > this problem, so we could filter serial console messages while keeping
> > > > > everything in other consoles (aka netconsole):
> > > > > 
> > > > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > > > 
> > > > Excellent! Unless I'm missing more context Chris does seem to be
> > > > attacking the problem at a more suitable layer.
> > > 
> > > This would help to bypass slow serial consoles. But the extra messages
> > > would still get stored into the kernel ring buffer and passed back
> > > to user space logs, for example journalctl.
> > 
> > It might actually make sense for the "workload enters or leaves" messages.
> > But I am not sure about the "ping" messages.
> 
> Agree. Let me back up and explain my "ping" messages better, which
> I think might add more information about this topic.
> 
> Meta has millions of servers, and all of them must have netconsole
> running 100% of the time.
> 
> Of course that this is not reality, and problems happen for different
> reasons, the ones that interest me here are:
> 
> 1) Top of the rack switch MAC address changes (mostly associated with
>    network hardware (top of the rack switches and gateway) replacement)
>     a) Keep in mind that netconsole target has the destination MAC as
>        part of its configuration.
> 
> 2) Netconsole got associated with the wrong network port, which comes in
>    two different flavors.
>    a) The machine got provisioned wrongly since day one (Most common
>       case)
>    b) The machine NIC changed and: 
>       i) The target doesn't bind correctly anymore (if netconsole
>          target is bound by mac address)
>       	   * This is easier to detect, given the target will never be
> 	     enabled.
> 
> 3) Netconsd (the daemon that listen to netconsole packets) is buggy or
>    dead
> 
> 4) Network failures across the route
> 
> 
> Possible Solutions
> ==================
> 
> In order to detect those issues above, I think the best (or only) way is
> to send messages from the host, and check if they got received. If not,
> raise an alarm (in the common distributed way).
> 
> This could be done in very different ways, tho. Such as:
> 
> 1) Have a binary in each machine:
> 	a) This binary reads the netconsole target that is configured,
> 	   and mimics "ping" UDP netconsole packet.
> 
> 	Pro: 
> 	     * It doesn't need any kernel change
> 	Cons:
> 	     * It needs to reimplement the netconsole logic in userspace
> 	     * This needs also a widely distributed binary on all
> 	       machines
> 
> 2) Send a ping directly to the console
> 	a) Something as 'echo ping from $hostname" > /dev/kmsg')
> 
> 	Pro:
> 		* No kernel changes
> 	Cons:
> 		* These debug messages will be sent to journalctl and to
> 		  the console, polluting both
>
> 3) Using per-loglevel patchset.
> 	a) Same as above, but, setting netconsole loglevel to DEBUG, while
> 	   all other consoles to INFO.
> 
> 	Pro:
> 		* No changes on netconsole
> 		* Netconsole "buffers" continues to be synchronized with
> 		  kernel buffers. Everything in the same page, but,
> 		  netconsole data has one loglevel higher.
> 		* Sending a message to netconsole-only message is not
> 		  special at all. It uses the same workflow we have
> 		  today, through `/dev/kmsg'
> 	Cons:
> 		* Needs to change printk/console code (Chris' patch)
> 		  that is on review for years now. Will it ever get
> 		  accepted?
> 		* These "ping" message will be in kernel buffers and
> 		  journalctl, and are useless in there (!?)
> 		* It is not possible to send a message to a single
> 		  netconsole target.

JFYI, I am going to review the last version of the per-console
loglevel patchset later this week. IMHO, we are very close to
get it merged.

BTW: How often do you ping the netconsole, please?

     IMHO, adding a short message once-per-hour might be bearable,
     once-per-minute might be questionable for the kernel buffer
     but still fine for journalctl.

     Also it depends on the size of the kernel buffer and whether
     you use a crash dump. I mean that it might be handy to have
     some useful messages in the kernel buffer when the crash dump
     is generated and used for debugging. Otherwise, the only
     important thing is whether they get stored externally either
     via console or journalctl.


> 4) send messages only to netconsole (this patchset)
> 	Pro:
> 		* It is easy to test netconsole connective (problem above),
> 		  without kernel buffers/journal pollution
> 		* It doesn't depend on the per-loglevel patchset
> 		* Adds flexibility to netconsole targets.
> 			- only certain netconsole targets receive
> 			  certain messages
> 	Cons:
> 		* Messages sent to netconsole is a superset of messages in the
> 		  kernel buffer. In other words, "dmesg" and machine
> 		  logs/journal will not be able to see messages that
> 		  were sent directly to netconsole.
> 			- It might be seen as a back channel (!?)
> 		* Different netconsole targets may receive different
> 		  messages. Too much flexibility might be bad (!?)

I do not have strong opinion about this.

That said, the location /sys/kernel/config/netconsole/<target>/send_msg
looks a bit weird to me. I would rather expect /dev/netconsole_msg
or so. But I do not have strong opinion. It might be an overkill.

How important is it to trigger the ping from userspace, please?
It might be sent by an existing watchdog.

Best Regards,
Petr

Hello Petr,

On Mon, Dec 08, 2025 at 03:52:37PM +0100, Petr Mladek wrote:
> On Fri 2025-12-05 02:21:08, Breno Leitao wrote:

> JFYI, I am going to review the last version of the per-console
> loglevel patchset later this week. IMHO, we are very close to
> get it merged.
> 
> BTW: How often do you ping the netconsole, please?
>      IMHO, adding a short message once-per-hour might be bearable,
>      once-per-minute might be questionable for the kernel buffer
>      but still fine for journalctl.

It is not very often today, about once a week. This is mostly due to the
pollution of kernel buffers.

Ideally we can set it to multiple times a day, but less than hourly.

> > 4) send messages only to netconsole (this patchset)
> 
> I do not have strong opinion about this.
> 
> That said, the location /sys/kernel/config/netconsole/<target>/send_msg
> looks a bit weird to me. I would rather expect /dev/netconsole_msg
> or so. But I do not have strong opinion. It might be an overkill.
> 
> How important is it to trigger the ping from userspace, please?
> It might be sent by an existing watchdog.

Medium importance, I would say. I am inclined to distributed design,
where each machine detects broken netconsole and report itself. Instead
of something else (a service) finding broken-netconsole hosts in the
"fleet".

Something as:

	1. Machine boots
	2. userspace sends the ping (or knows that the ping was already
	   sent)
	3. the configuration management (chef, ansible, etc) makes sure
	   that the message got received.


So, step number 3 needs to be executed only after step 2. Initiating the
ping (step 2) from userspace is the easiest way, to control when the
machine can go to step 3.

Thanks
--breno

On Fri, Nov 28, 2025 at 06:20:45AM -0800, Breno Leitao wrote:
> This patch series introduces a new configfs attribute that enables sending
> messages directly through netconsole without going through the kernel's logging
> infrastructure.

Minor nit:

Hi Breno,

The patchset looks good to me.
But the cover-letter (this email) has no subject.