On 6/22/22 23:26, Jonathon Jongsma wrote:
> Hi guys,
>
> I've been working on adding support for nbdkit to libvirt for network
> storage sources like http and ssh. See
> https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more
> information, but the summary is that RHEL does not want to ship the qemu
> storage plugins for curl and ssh. Handling them outside of the qemu
> process provides several advantages such as reduced attack surface and
> stability.
>
> I have something that works for me, but as I have not dealt with the
> storage stuff much before, I have a feeling that I'm missing some
> things.
>
> A quick summary of the code:
> - at startup I query to see whether nbdkit exists on the host and if
> so, I query which plugins/filters are installed. This is stored as
> qemuNbdkitCaps on the qemu driver
> - When the driver prepares the domain, we go through each disk source
> and determine whether the nbdkit capabilities allow us to support
> this disk via nbdkit, and if so, we allocate a qemuNbdkitProcess
> object and stash it in the private data of the virStorageSource.
> - The presence or absence of this qemuNbdkitProcess data then indicates
> whether this disk will be served to qemu indirectly via nbdkit or not
> - When we launch the qemuProcess, as part of the "external device
> start" step, I launch a ndkit process for each disk that is supported
> by nbdkit. I also optionally fork a child process to communicate
> authentication details and cookies to the nbdkit process via a unix
> socket.
> - for devices which are served by the ndkit process, I change the qemu
> commandline in the following ways:
> - I no longer pass auth/cookie secrets to qemu (those are handled by
> nbdkit)
> - I replace the actual network URL of the remote disk source with the
> path to the nbdkit unix socket
>
> Known shortcomings
> - I don't yet re-query for nbdkit / nbdkit caps, so need to restart libvirt to
> pick up newly-installed nbdkit or additional capabilities
There are couple of ways to solve this:
1) mimic what we do for dnsmasq: basically, when initializing the bridge
driver we query dnsmasq for caps networkStateInitialize() ->
dnsmasqCapsNewFromBinary() and when needing the caps for real (i.e.
spawning dnsmasq when starting a network) we forcefully refresh the
caps: networkStartDhcpDaemon() -> networkDnsmasqCapsRefresh().
2) use full blown virFileCache() machinery which reacts to mtime of
given binary.
An argument against going with 1) is that while spawning an extra
process (dnsmasq) when starting a network is acceptable (a network is
not started that often), spawning an extra process for each disk (or
even an image in backingchain) might be too expensive.
> - testing is pretty limited at the moment
> - selinux not working yet
> - creating disks isn't supported, though Rich has added some support
> for that upstream in the nbdkit ssh plugin.
>
> I'd appreciate feedback on what i've got so far.
>
> Jonathon Jongsma (3):
> docs: clarify 'readahead' and 'timeout' for disks
> schema: Be more flexible for diskSourceNetworkProtocolPropsCommon
> WIP: use nbdkit for remote disk sources
>
> docs/formatdomain.rst | 10 +-
> include/libvirt/virterror.h | 1 +
> po/POTFILES | 1 +
> src/conf/schemas/domaincommon.rng | 34 +-
> src/qemu/meson.build | 1 +
> src/qemu/qemu_block.c | 64 +-
> src/qemu/qemu_block.h | 1 +
> src/qemu/qemu_command.c | 26 +-
> src/qemu/qemu_conf.c | 19 +
> src/qemu/qemu_conf.h | 5 +
> src/qemu/qemu_domain.c | 110 ++-
> src/qemu/qemu_domain.h | 5 +
> src/qemu/qemu_driver.c | 4 +-
> src/qemu/qemu_extdevice.c | 25 +
> src/qemu/qemu_nbdkit.c | 629 ++++++++++++++++++
> src/qemu/qemu_nbdkit.h | 89 +++
> src/qemu/qemu_validate.c | 22 +-
> src/qemu/qemu_validate.h | 4 +-
> src/util/virerror.c | 1 +
> tests/qemublocktest.c | 8 +-
> tests/qemustatusxml2xmldata/modern-in.xml | 1 -
> ...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 ++
> .../disk-cdrom-network-nbdkit.xml | 1 +
> ...isk-network-http-nbdkit.x86_64-latest.args | 45 ++
> .../disk-network-http-nbdkit.xml | 1 +
> ...work-source-curl-nbdkit.x86_64-latest.args | 49 ++
> .../disk-network-source-curl-nbdkit.xml | 1 +
> ...isk-network-source-curl.x86_64-latest.args | 53 ++
> .../disk-network-source-curl.xml | 71 ++
> tests/qemuxml2argvtest.c | 12 +
> tests/testutilsqemu.c | 16 +
> tests/testutilsqemu.h | 4 +
> 32 files changed, 1302 insertions(+), 53 deletions(-)
> create mode 100644 src/qemu/qemu_nbdkit.c
> create mode 100644 src/qemu/qemu_nbdkit.h
> create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args
> create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml
> create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args
> create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml
> create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args
> create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml
> create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
> create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
>