[v3] eBPF RSS through QMP support.

[PATCH v3 0/6] eBPF RSS through QMP support.

Posted by Andrew Melnychenko 11 months, 1 week ago

This series of patches provides the ability to retrieve eBPF program
through qmp, so management application may load bpf blob with proper capabilities.
Now, virtio-net devices can accept eBPF programs and maps through properties
as external file descriptors. Access to the eBPF map is direct through mmap()
call, so it should not require additional capabilities to bpf* calls.
eBPF file descriptors can be passed to QEMU from parent process or by unix
socket with sendfd() qmp command.

Possible solution for libvirt may look like this: https://github.com/daynix/libvirt/tree/RSS_eBPF (WIP)

Changes since v2:
 * moved/refactored QMP command
 * refactored virtio-net

Changes since v1:
 * refactored virtio-net
 * moved hunks for ebpf mmap()
 * added qmp enum for eBPF id.

Andrew Melnychenko (6):
  ebpf: Added eBPF map update through mmap.
  ebpf: Added eBPF initialization by fds.
  virtio-net: Added property to load eBPF RSS with fds.
  ebpf: Added declaration/initialization routines.
  qmp: Added new command to retrieve eBPF blob.
  ebpf: Updated eBPF program and skeleton.

 ebpf/ebpf.c                    |   70 ++
 ebpf/ebpf.h                    |   31 +
 ebpf/ebpf_rss-stub.c           |    6 +
 ebpf/ebpf_rss.c                |  150 +++-
 ebpf/ebpf_rss.h                |   10 +
 ebpf/meson.build               |    2 +-
 ebpf/rss.bpf.skeleton.h        | 1469 ++++++++++++++++----------------
 hw/net/virtio-net.c            |   55 +-
 include/hw/virtio/virtio-net.h |    1 +
 qapi/ebpf.json                 |   55 ++
 qapi/meson.build               |    1 +
 qapi/qapi-schema.json          |    1 +
 tools/ebpf/rss.bpf.c           |    2 +-
 13 files changed, 1093 insertions(+), 760 deletions(-)
 create mode 100644 ebpf/ebpf.c
 create mode 100644 ebpf/ebpf.h
 create mode 100644 qapi/ebpf.json

-- 
2.39.1

Re: [PATCH v3 0/6] eBPF RSS through QMP support.

Posted by Jason Wang 10 months, 3 weeks ago

On Thu, Jun 15, 2023 at 6:29 AM Andrew Melnychenko <andrew@daynix.com> wrote:
>
> This series of patches provides the ability to retrieve eBPF program
> through qmp, so management application may load bpf blob with proper capabilities.
> Now, virtio-net devices can accept eBPF programs and maps through properties
> as external file descriptors. Access to the eBPF map is direct through mmap()
> call, so it should not require additional capabilities to bpf* calls.
> eBPF file descriptors can be passed to QEMU from parent process or by unix
> socket with sendfd() qmp command.
>
> Possible solution for libvirt may look like this: https://github.com/daynix/libvirt/tree/RSS_eBPF (WIP)

I got a build failure:

[18/2192] Compiling C object
libqemuutil.a.p/meson-generated_.._qapi_qapi-commands-ebpf.c.o
FAILED: libqemuutil.a.p/meson-generated_.._qapi_qapi-commands-ebpf.c.o
cc -m64 -mcx16 -Ilibqemuutil.a.p -I. -I.. -Isubprojects/libvhost-user
-I../subprojects/libvhost-user -Iqapi -Itrace -Iui -Iui/shader
-I/usr/include/libmount -I/usr/include/blkid -I/usr/include/glib-2.0
-I/usr/lib/x86_64-linux-gnu/glib-2.0/include
-I/usr/include/gio-unix-2.0 -I/usr/include/pixman-1
-fdiagnostics-color=auto -Wall -Winvalid-pch -Werror -std=gnu11 -O2 -g
-fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wundef
-Wwrite-strings -Wmissing-prototypes -Wstrict-prototypes
-Wredundant-decls -Wold-style-declaration -Wold-style-definition
-Wtype-limits -Wformat-security -Wformat-y2k -Winit-self
-Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels
-Wexpansion-to-defined -Wimplicit-fallthrough=2
-Wmissing-format-attribute -Wno-missing-include-dirs
-Wno-shift-negative-value -Wno-psabi -isystem
/home/devel/git/qemu/linux-headers -isystem linux-headers -iquote .
-iquote /home/devel/git/qemu -iquote /home/devel/git/qemu/include
-iquote /home/devel/git/qemu/host/include/x86_64 -iquote
/home/devel/git/qemu/host/include/generic -iquote
/home/devel/git/qemu/tcg/i386 -pthread -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing
-fno-common -fwrapv -fPIE -MD -MQ
libqemuutil.a.p/meson-generated_.._qapi_qapi-commands-ebpf.c.o -MF
libqemuutil.a.p/meson-generated_.._qapi_qapi-commands-ebpf.c.o.d -o
libqemuutil.a.p/meson-generated_.._qapi_qapi-commands-ebpf.c.o -c
qapi/qapi-commands-ebpf.c
qapi/qapi-commands-ebpf.c:24:13: error:
‘qmp_marshal_output_EbpfObject’ defined but not used
[-Werror=unused-function]
   24 | static void qmp_marshal_output_EbpfObject(EbpfObject *ret_in,
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors

Please fix this.

Thanks

>
> Changes since v2:
>  * moved/refactored QMP command
>  * refactored virtio-net
>
> Changes since v1:
>  * refactored virtio-net
>  * moved hunks for ebpf mmap()
>  * added qmp enum for eBPF id.
>
> Andrew Melnychenko (6):
>   ebpf: Added eBPF map update through mmap.
>   ebpf: Added eBPF initialization by fds.
>   virtio-net: Added property to load eBPF RSS with fds.
>   ebpf: Added declaration/initialization routines.
>   qmp: Added new command to retrieve eBPF blob.
>   ebpf: Updated eBPF program and skeleton.
>
>  ebpf/ebpf.c                    |   70 ++
>  ebpf/ebpf.h                    |   31 +
>  ebpf/ebpf_rss-stub.c           |    6 +
>  ebpf/ebpf_rss.c                |  150 +++-
>  ebpf/ebpf_rss.h                |   10 +
>  ebpf/meson.build               |    2 +-
>  ebpf/rss.bpf.skeleton.h        | 1469 ++++++++++++++++----------------
>  hw/net/virtio-net.c            |   55 +-
>  include/hw/virtio/virtio-net.h |    1 +
>  qapi/ebpf.json                 |   55 ++
>  qapi/meson.build               |    1 +
>  qapi/qapi-schema.json          |    1 +
>  tools/ebpf/rss.bpf.c           |    2 +-
>  13 files changed, 1093 insertions(+), 760 deletions(-)
>  create mode 100644 ebpf/ebpf.c
>  create mode 100644 ebpf/ebpf.h
>  create mode 100644 qapi/ebpf.json
>
> --
> 2.39.1
>

Re: [PATCH v3 0/6] eBPF RSS through QMP support.

Posted by Jason Wang 10 months, 3 weeks ago

On Thu, Jun 15, 2023 at 6:29 AM Andrew Melnychenko <andrew@daynix.com> wrote:
>
> This series of patches provides the ability to retrieve eBPF program
> through qmp, so management application may load bpf blob with proper capabilities.
> Now, virtio-net devices can accept eBPF programs and maps through properties
> as external file descriptors. Access to the eBPF map is direct through mmap()
> call, so it should not require additional capabilities to bpf* calls.
> eBPF file descriptors can be passed to QEMU from parent process or by unix
> socket with sendfd() qmp command.
>
> Possible solution for libvirt may look like this: https://github.com/daynix/libvirt/tree/RSS_eBPF (WIP)
>
> Changes since v2:
>  * moved/refactored QMP command
>  * refactored virtio-net

I've queued this series, but a question left:

mmap() support for eBPF maps is not supported from day0, should we
fallback to syscall for the OS that doesn't support that?

Thanks

>
> Changes since v1:
>  * refactored virtio-net
>  * moved hunks for ebpf mmap()
>  * added qmp enum for eBPF id.
>
> Andrew Melnychenko (6):
>   ebpf: Added eBPF map update through mmap.
>   ebpf: Added eBPF initialization by fds.
>   virtio-net: Added property to load eBPF RSS with fds.
>   ebpf: Added declaration/initialization routines.
>   qmp: Added new command to retrieve eBPF blob.
>   ebpf: Updated eBPF program and skeleton.
>
>  ebpf/ebpf.c                    |   70 ++
>  ebpf/ebpf.h                    |   31 +
>  ebpf/ebpf_rss-stub.c           |    6 +
>  ebpf/ebpf_rss.c                |  150 +++-
>  ebpf/ebpf_rss.h                |   10 +
>  ebpf/meson.build               |    2 +-
>  ebpf/rss.bpf.skeleton.h        | 1469 ++++++++++++++++----------------
>  hw/net/virtio-net.c            |   55 +-
>  include/hw/virtio/virtio-net.h |    1 +
>  qapi/ebpf.json                 |   55 ++
>  qapi/meson.build               |    1 +
>  qapi/qapi-schema.json          |    1 +
>  tools/ebpf/rss.bpf.c           |    2 +-
>  13 files changed, 1093 insertions(+), 760 deletions(-)
>  create mode 100644 ebpf/ebpf.c
>  create mode 100644 ebpf/ebpf.h
>  create mode 100644 qapi/ebpf.json
>
> --
> 2.39.1
>

Re: [PATCH v3 0/6] eBPF RSS through QMP support.

Posted by Daniel P. Berrangé 10 months, 3 weeks ago

On Fri, Jun 30, 2023 at 01:06:22PM +0800, Jason Wang wrote:
> On Thu, Jun 15, 2023 at 6:29 AM Andrew Melnychenko <andrew@daynix.com> wrote:
> >
> > This series of patches provides the ability to retrieve eBPF program
> > through qmp, so management application may load bpf blob with proper capabilities.
> > Now, virtio-net devices can accept eBPF programs and maps through properties
> > as external file descriptors. Access to the eBPF map is direct through mmap()
> > call, so it should not require additional capabilities to bpf* calls.
> > eBPF file descriptors can be passed to QEMU from parent process or by unix
> > socket with sendfd() qmp command.
> >
> > Possible solution for libvirt may look like this: https://github.com/daynix/libvirt/tree/RSS_eBPF (WIP)
> >
> > Changes since v2:
> >  * moved/refactored QMP command
> >  * refactored virtio-net
> 
> I've queued this series, but a question left:
> 
> mmap() support for eBPF maps is not supported from day0, should we
> fallback to syscall for the OS that doesn't support that?

How recent is mmap() support ? Is it difficult to do a fallback ?

As since is a new feature, there's no inherant expectation of support
for arbitrary old platforms. So only worth investing in a fallback if
it is easy, or there's a very compelling reason to support certain
old platforms.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 0/6] eBPF RSS through QMP support.

Posted by Jason Wang 10 months, 3 weeks ago

On Fri, Jun 30, 2023 at 4:04 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jun 30, 2023 at 01:06:22PM +0800, Jason Wang wrote:
> > On Thu, Jun 15, 2023 at 6:29 AM Andrew Melnychenko <andrew@daynix.com> wrote:
> > >
> > > This series of patches provides the ability to retrieve eBPF program
> > > through qmp, so management application may load bpf blob with proper capabilities.
> > > Now, virtio-net devices can accept eBPF programs and maps through properties
> > > as external file descriptors. Access to the eBPF map is direct through mmap()
> > > call, so it should not require additional capabilities to bpf* calls.
> > > eBPF file descriptors can be passed to QEMU from parent process or by unix
> > > socket with sendfd() qmp command.
> > >
> > > Possible solution for libvirt may look like this: https://github.com/daynix/libvirt/tree/RSS_eBPF (WIP)
> > >
> > > Changes since v2:
> > >  * moved/refactored QMP command
> > >  * refactored virtio-net
> >
> > I've queued this series, but a question left:
> >
> > mmap() support for eBPF maps is not supported from day0, should we
> > fallback to syscall for the OS that doesn't support that?
>
> How recent is mmap() support ?

I don't check.

> Is it difficult to do a fallback ?

Nope, but it requires privilege if we go with a syscall.

>
> As since is a new feature,

But it modifies the old rss loading code, no?

-    for (; i < len; ++i) {
-        if (bpf_map_update_elem(ctx->map_indirections_table, &i,
-                                indirections_table + i, 0) < 0) {
-            return false;
-        }
-    }
+    memcpy(ctx->mmap_indirections_table, indirections_table,
+            sizeof(*indirections_table) * len);

> there's no inherant expectation of support
> for arbitrary old platforms. So only worth investing in a fallback if
> it is easy, or there's a very compelling reason to support certain
> old platforms.

The reason is that we support eBPF RSS with syscall based map updating
in the past if Qemu was running with privilege. With this series, it
won't work if the kernel doesn't support mmap.

Thanks

>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: [PATCH v3 0/6] eBPF RSS through QMP support.

Posted by Daniel P. Berrangé 10 months, 3 weeks ago

On Fri, Jun 30, 2023 at 04:21:02PM +0800, Jason Wang wrote:
> On Fri, Jun 30, 2023 at 4:04 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Fri, Jun 30, 2023 at 01:06:22PM +0800, Jason Wang wrote:
> > > On Thu, Jun 15, 2023 at 6:29 AM Andrew Melnychenko <andrew@daynix.com> wrote:
> > > >
> > > > This series of patches provides the ability to retrieve eBPF program
> > > > through qmp, so management application may load bpf blob with proper capabilities.
> > > > Now, virtio-net devices can accept eBPF programs and maps through properties
> > > > as external file descriptors. Access to the eBPF map is direct through mmap()
> > > > call, so it should not require additional capabilities to bpf* calls.
> > > > eBPF file descriptors can be passed to QEMU from parent process or by unix
> > > > socket with sendfd() qmp command.
> > > >
> > > > Possible solution for libvirt may look like this: https://github.com/daynix/libvirt/tree/RSS_eBPF (WIP)
> > > >
> > > > Changes since v2:
> > > >  * moved/refactored QMP command
> > > >  * refactored virtio-net
> > >
> > > I've queued this series, but a question left:
> > >
> > > mmap() support for eBPF maps is not supported from day0, should we
> > > fallback to syscall for the OS that doesn't support that?
> >
> > How recent is mmap() support ?
> 
> I don't check.
> 
> > Is it difficult to do a fallback ?
> 
> Nope, but it requires privilege if we go with a syscall.
> 
> >
> > As since is a new feature,
> 
> But it modifies the old rss loading code, no?
> 
> -    for (; i < len; ++i) {
> -        if (bpf_map_update_elem(ctx->map_indirections_table, &i,
> -                                indirections_table + i, 0) < 0) {
> -            return false;
> -        }
> -    }
> +    memcpy(ctx->mmap_indirections_table, indirections_table,
> +            sizeof(*indirections_table) * len);
> 
> > there's no inherant expectation of support
> > for arbitrary old platforms. So only worth investing in a fallback if
> > it is easy, or there's a very compelling reason to support certain
> > old platforms.
> 
> The reason is that we support eBPF RSS with syscall based map updating
> in the past if Qemu was running with privilege. With this series, it
> won't work if the kernel doesn't support mmap.

Oh right, yes, I missed that aspect. So yeah, we do need fallback.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 0/6] eBPF RSS through QMP support.

Posted by Andrew Melnichenko 10 months, 3 weeks ago

Hi all,
Thank you for your comments. I'll check the error/warning.

On Fri, Jun 30, 2023 at 11:55 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jun 30, 2023 at 04:21:02PM +0800, Jason Wang wrote:
> > On Fri, Jun 30, 2023 at 4:04 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Fri, Jun 30, 2023 at 01:06:22PM +0800, Jason Wang wrote:
> > > > On Thu, Jun 15, 2023 at 6:29 AM Andrew Melnychenko <andrew@daynix.com> wrote:
> > > > >
> > > > > This series of patches provides the ability to retrieve eBPF program
> > > > > through qmp, so management application may load bpf blob with proper capabilities.
> > > > > Now, virtio-net devices can accept eBPF programs and maps through properties
> > > > > as external file descriptors. Access to the eBPF map is direct through mmap()
> > > > > call, so it should not require additional capabilities to bpf* calls.
> > > > > eBPF file descriptors can be passed to QEMU from parent process or by unix
> > > > > socket with sendfd() qmp command.
> > > > >
> > > > > Possible solution for libvirt may look like this: https://github.com/daynix/libvirt/tree/RSS_eBPF (WIP)
> > > > >
> > > > > Changes since v2:
> > > > >  * moved/refactored QMP command
> > > > >  * refactored virtio-net
> > > >
> > > > I've queued this series, but a question left:
> > > >
> > > > mmap() support for eBPF maps is not supported from day0, should we
> > > > fallback to syscall for the OS that doesn't support that?
> > >
> > > How recent is mmap() support ?
> >
> > I don't check.
> >
> > > Is it difficult to do a fallback ?
> >
> > Nope, but it requires privilege if we go with a syscall.
> >
> > >
> > > As since is a new feature,
> >
> > But it modifies the old rss loading code, no?
> >
> > -    for (; i < len; ++i) {
> > -        if (bpf_map_update_elem(ctx->map_indirections_table, &i,
> > -                                indirections_table + i, 0) < 0) {
> > -            return false;
> > -        }
> > -    }
> > +    memcpy(ctx->mmap_indirections_table, indirections_table,
> > +            sizeof(*indirections_table) * len);
> >
> > > there's no inherant expectation of support
> > > for arbitrary old platforms. So only worth investing in a fallback if
> > > it is easy, or there's a very compelling reason to support certain
> > > old platforms.
> >
> > The reason is that we support eBPF RSS with syscall based map updating
> > in the past if Qemu was running with privilege. With this series, it
> > won't work if the kernel doesn't support mmap.
>
> Oh right, yes, I missed that aspect. So yeah, we do need fallback.

Need to check if it is necessary to support the eBPF on an old kernel
that doesn't support mmap for eBPF.
eBPF RSS program itself would require kernel 5.5+.
If I am not mistaken, the mmap for eBPF was added in 5.5
I'll recheck it.

>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>