nodedev: Increase the netlink socket buffer size to the one used by udev

[libvirt] [PATCH v2] nodedev: Increase the netlink socket buffer size to the one used by udev

Posted by Erik Skultety 6 years, 11 months ago

From: "ning.bo" <ning.bo9@zte.com.cn>

When a number of SRIOV VFs (up to 128 on Intel XL710) is created:
for i in `seq 0 1`; do
  echo 63 > /sys/class/net/<interface>/device/sriov_numvfs
done

libvirtd will then report "udev_monitor_receive_device returned NULL"
error because the netlink socket buffer is not big enough (using GDB on
libudev confirmed this with ENOBUFFS) and thus some udev events were
dropped. This results in some devices being missing in the nodedev-list
output. This patch overrides the system's rmem_max limit but for that,
we need to make sure we've got root privileges.

https://bugzilla.redhat.com/show_bug.cgi?id=1450960

Signed-off-by: ning.bo <ning.bo9@zte.com.cn>
Signed-off-by: Erik Skultety <eskultet@redhat.com>
---
Additionally, we might want to check for the errno, providing a specific
message if such issue occurs again in a further non-specified point in time and
return the generic, yet cryptic one for all other cases.

 src/node_device/node_device_udev.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c
index 4ecb0b18f..0b3717397 100644
--- a/src/node_device/node_device_udev.c
+++ b/src/node_device/node_device_udev.c
@@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged,

     udev_monitor_enable_receiving(priv->udev_monitor);

+    /* mimic udevd's behaviour and override the systems rmem_max limit in case
+     * there's a significant number of device 'add' events
+     */
+    if (geteuid() == 0)
+        udev_monitor_set_receive_buffer_size(priv->udev_monitor,
+                                             128 * 1024 * 1024);
+
     /* We register the monitor with the event callback so we are
      * notified by udev of device changes before we enumerate existing
      * devices because libvirt will simply recreate the device if we
--
2.13.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2] nodedev: Increase the netlink socket buffer size to the one used by udev

Posted by Erik Skultety 6 years, 10 months ago

On Fri, May 19, 2017 at 03:16:26PM +0200, Erik Skultety wrote:
> From: "ning.bo" <ning.bo9@zte.com.cn>
>

Ping?

> When a number of SRIOV VFs (up to 128 on Intel XL710) is created:
> for i in `seq 0 1`; do
>   echo 63 > /sys/class/net/<interface>/device/sriov_numvfs
> done
>
> libvirtd will then report "udev_monitor_receive_device returned NULL"
> error because the netlink socket buffer is not big enough (using GDB on
> libudev confirmed this with ENOBUFFS) and thus some udev events were
> dropped. This results in some devices being missing in the nodedev-list
> output. This patch overrides the system's rmem_max limit but for that,
> we need to make sure we've got root privileges.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1450960
>
> Signed-off-by: ning.bo <ning.bo9@zte.com.cn>
> Signed-off-by: Erik Skultety <eskultet@redhat.com>
> ---
> Additionally, we might want to check for the errno, providing a specific
> message if such issue occurs again in a further non-specified point in time and
> return the generic, yet cryptic one for all other cases.
>
>  src/node_device/node_device_udev.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c
> index 4ecb0b18f..0b3717397 100644
> --- a/src/node_device/node_device_udev.c
> +++ b/src/node_device/node_device_udev.c
> @@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged,
>
>      udev_monitor_enable_receiving(priv->udev_monitor);
>
> +    /* mimic udevd's behaviour and override the systems rmem_max limit in case
> +     * there's a significant number of device 'add' events
> +     */
> +    if (geteuid() == 0)
> +        udev_monitor_set_receive_buffer_size(priv->udev_monitor,
> +                                             128 * 1024 * 1024);
> +
>      /* We register the monitor with the event callback so we are
>       * notified by udev of device changes before we enumerate existing
>       * devices because libvirt will simply recreate the device if we
> --
> 2.13.0
>
> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2] nodedev: Increase the netlink socket buffer size to the one used by udev

Posted by Michal Privoznik 6 years, 10 months ago

On 05/19/2017 03:16 PM, Erik Skultety wrote:
> From: "ning.bo" <ning.bo9@zte.com.cn>
> 
> When a number of SRIOV VFs (up to 128 on Intel XL710) is created:
> for i in `seq 0 1`; do
>   echo 63 > /sys/class/net/<interface>/device/sriov_numvfs
> done
> 
> libvirtd will then report "udev_monitor_receive_device returned NULL"
> error because the netlink socket buffer is not big enough (using GDB on
> libudev confirmed this with ENOBUFFS) and thus some udev events were
> dropped. This results in some devices being missing in the nodedev-list
> output. This patch overrides the system's rmem_max limit but for that,
> we need to make sure we've got root privileges.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1450960
> 
> Signed-off-by: ning.bo <ning.bo9@zte.com.cn>
> Signed-off-by: Erik Skultety <eskultet@redhat.com>
> ---
> Additionally, we might want to check for the errno, providing a specific
> message if such issue occurs again in a further non-specified point in time and
> return the generic, yet cryptic one for all other cases.
> 
>  src/node_device/node_device_udev.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c
> index 4ecb0b18f..0b3717397 100644
> --- a/src/node_device/node_device_udev.c
> +++ b/src/node_device/node_device_udev.c
> @@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged,
> 
>      udev_monitor_enable_receiving(priv->udev_monitor);
> 
> +    /* mimic udevd's behaviour and override the systems rmem_max limit in case
> +     * there's a significant number of device 'add' events
> +     */
> +    if (geteuid() == 0)
> +        udev_monitor_set_receive_buffer_size(priv->udev_monitor,
> +                                             128 * 1024 * 1024);
> +
>      /* We register the monitor with the event callback so we are
>       * notified by udev of device changes before we enumerate existing
>       * devices because libvirt will simply recreate the device if we

ACK and safe for freeze. Although on my system it works even without the
check for euid.

Michal

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2] nodedev: Increase the netlink socket buffer size to the one used by udev

Posted by Erik Skultety 6 years, 10 months ago

On Mon, May 29, 2017 at 02:16:17PM +0200, Michal Privoznik wrote:
> On 05/19/2017 03:16 PM, Erik Skultety wrote:
> > From: "ning.bo" <ning.bo9@zte.com.cn>
> >
> > When a number of SRIOV VFs (up to 128 on Intel XL710) is created:
> > for i in `seq 0 1`; do
> >   echo 63 > /sys/class/net/<interface>/device/sriov_numvfs
> > done
> >
> > libvirtd will then report "udev_monitor_receive_device returned NULL"
> > error because the netlink socket buffer is not big enough (using GDB on
> > libudev confirmed this with ENOBUFFS) and thus some udev events were
> > dropped. This results in some devices being missing in the nodedev-list
> > output. This patch overrides the system's rmem_max limit but for that,
> > we need to make sure we've got root privileges.
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1450960
> >
> > Signed-off-by: ning.bo <ning.bo9@zte.com.cn>
> > Signed-off-by: Erik Skultety <eskultet@redhat.com>
> > ---
> > Additionally, we might want to check for the errno, providing a specific
> > message if such issue occurs again in a further non-specified point in time and
> > return the generic, yet cryptic one for all other cases.
> >
> >  src/node_device/node_device_udev.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c
> > index 4ecb0b18f..0b3717397 100644
> > --- a/src/node_device/node_device_udev.c
> > +++ b/src/node_device/node_device_udev.c
> > @@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged,
> >
> >      udev_monitor_enable_receiving(priv->udev_monitor);
> >
> > +    /* mimic udevd's behaviour and override the systems rmem_max limit in case
> > +     * there's a significant number of device 'add' events
> > +     */
> > +    if (geteuid() == 0)
> > +        udev_monitor_set_receive_buffer_size(priv->udev_monitor,
> > +                                             128 * 1024 * 1024);
> > +
> >      /* We register the monitor with the event callback so we are
> >       * notified by udev of device changes before we enumerate existing
> >       * devices because libvirt will simply recreate the device if we
>
> ACK and safe for freeze. Although on my system it works even without the
> check for euid.

Are you sure? Because the session daemon should not have the CAP_NET_ADMIN
capability. Though it's true that since we don't care about the return value
the check is unnecessary I give you that. Still, the check is harmless and we
know that it will fail for non-privileged user, so I'm going to keep it here.

Thanks for review,
Erik

>
> Michal
>
> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2] nodedev: Increase the netlink socket buffer size to the one used by udev

Posted by Daniel P. Berrange 6 years, 10 months ago

On Fri, May 19, 2017 at 03:16:26PM +0200, Erik Skultety wrote:
> From: "ning.bo" <ning.bo9@zte.com.cn>
> 
> When a number of SRIOV VFs (up to 128 on Intel XL710) is created:
> for i in `seq 0 1`; do
>   echo 63 > /sys/class/net/<interface>/device/sriov_numvfs
> done
> 
> libvirtd will then report "udev_monitor_receive_device returned NULL"
> error because the netlink socket buffer is not big enough (using GDB on
> libudev confirmed this with ENOBUFFS) and thus some udev events were
> dropped. This results in some devices being missing in the nodedev-list
> output. This patch overrides the system's rmem_max limit but for that,
> we need to make sure we've got root privileges.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1450960
> 
> Signed-off-by: ning.bo <ning.bo9@zte.com.cn>
> Signed-off-by: Erik Skultety <eskultet@redhat.com>
> ---
> Additionally, we might want to check for the errno, providing a specific
> message if such issue occurs again in a further non-specified point in time and
> return the generic, yet cryptic one for all other cases.
> 
>  src/node_device/node_device_udev.c | 7 +++++++
>  1 file changed, 7 insertions(+)

This has broken the build on older systems  as udev doesn't have this
method.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list