This is the third version of this patch series. See
https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information about
the goal, but the summary is that RHEL does not want to ship the qemu storage
plugins for curl and ssh. Handling them outside of the qemu process provides
several advantages such as reduced attack surface and stability.
A quick summary of the code:
- at startup I query to see whether nbdkit exists on the host and if
so, I query which plugins/filters are installed. These capabilities
are cached and stored in the qemu driver
- When the driver prepares the domain, we go through each disk source
and determine whether the nbdkit capabilities allow us to support
this disk via nbdkit, and if so, we allocate a qemuNbdkitProcess
object and stash it in the private data of the virStorageSource.
- The presence or absence of this qemuNbdkitProcess data then indicates
whether this disk will be served to qemu indirectly via nbdkit or
directly
- When we launch the qemuProcess, as part of the "external device
start" step, I launch a ndkit process for each disk that is supported
by nbdkit.
- for devices which are served by an intermediate ndkit process, I
change the qemu commandline in the following ways:
- I no longer pass auth/cookie secrets to qemu (those are handled by
nbdkit)
- I replace the actual network URL of the remote disk source with the
path to the nbdkit unix socket
- We create a 'monitor' for the nbdkit process that watches to see whether the
process exits. If it does, we pause the domain, attempt to restart nbdkit,
and then resume the domain.
Open questions
- I think selinux will work once we add a policy for the /usr/sbin/nbdkit
binary to allow it to be executed by libvirt, but for now it fails to
execute nbdkit in enforcing mode. The current context for nbdkit (on fedora)
is "system_u:object_r:bin_t:s0". When I temporarily change the context to
something like qemu_exec_t, I am able to start nbdkit and the domain
launches.
Known shortcomings
- creating disks (in ssh) still isn't supported.
Changes in v3:
- Various formatting fixes
- Don't kill process in qemuNbdkitProcessFree() since we want the nbdkit
daemon to continue running even if libvirt restarts.
- Better detection and error reporting when starting the nbdkit process
- Add monitoring for nbdkit process so that it can be restarted if if ever
exits unexpectedly.
Changes in v2:
- split into multiple patches
- added a build option for nbdkit_moddir
- don't instantiate any secret / cookie props for disks that are being served
by nbdkit since we don't send secrets to qemu anymore
- ensure that nbdkit processes are started/stopped for the entire backing
chain
- switch to virFileCache-based capabilities for nbdkit so that we don't need
to requery every time
- switch to using pipes for communicating sensitive data to nbdkit
- use pidfile support built into virCommand rather than nbdkit's --pidfile
argument
- added significantly more tests
Jonathon Jongsma (18):
schema: allow 'ssh' as a protocol for network disks
qemu: Add functions for determining nbdkit availability
qemu: expand nbdkit capabilities
util: Allow virFileCache data to be any GObject
qemu: implement basic virFileCache for nbdkit caps
qemu: implement persistent file cache for nbdkit caps
qemu: use file cache for nbdkit caps
qemu: Add qemuNbdkitProcess
qemu: add functions to start and stop nbdkit
tests: add ability to test various nbdkit capabilities
qemu: split qemuDomainSecretStorageSourcePrepare
qemu: include nbdkit state in private xml
qemu: use nbdkit to serve network disks if available
tests: add tests for nbdkit invocation
util: make virCommandSetSendBuffer testable
qemu: pass sensitive data to nbdkit via pipe
qemu: add test for authenticating a https network disk
qemu: Monitor nbdkit process for exit
build-aux/syntax-check.mk | 4 +-
meson.build | 9 +
meson_options.txt | 1 +
po/POTFILES | 1 +
src/conf/schemas/domaincommon.rng | 1 +
src/libvirt_private.syms | 1 +
src/qemu/meson.build | 1 +
src/qemu/qemu_block.c | 162 ++-
src/qemu/qemu_conf.c | 23 +
src/qemu/qemu_conf.h | 7 +
src/qemu/qemu_domain.c | 180 ++-
src/qemu/qemu_domain.h | 4 +
src/qemu/qemu_driver.c | 4 +
src/qemu/qemu_extdevice.c | 48 +
src/qemu/qemu_nbdkit.c | 1243 +++++++++++++++++
src/qemu/qemu_nbdkit.h | 120 ++
src/qemu/qemu_nbdkitpriv.h | 31 +
src/qemu/qemu_process.c | 13 +
src/util/vircommand.c | 19 +-
src/util/vircommand.h | 8 +
src/util/vircommandpriv.h | 4 +
src/util/virfilecache.c | 14 +-
src/util/virfilecache.h | 2 +-
src/util/virutil.h | 2 +-
tests/meson.build | 1 +
.../disk-cdrom-network.args.disk0 | 7 +
.../disk-cdrom-network.args.disk1 | 9 +
.../disk-cdrom-network.args.disk1.pipe.1778 | 1 +
.../disk-cdrom-network.args.disk2 | 9 +
.../disk-cdrom-network.args.disk2.pipe.1780 | 1 +
.../disk-network-http.args.disk0 | 7 +
.../disk-network-http.args.disk1 | 6 +
.../disk-network-http.args.disk2 | 7 +
.../disk-network-http.args.disk2.pipe.1778 | 1 +
.../disk-network-http.args.disk3 | 8 +
.../disk-network-http.args.disk3.pipe.1780 | 1 +
...work-source-curl-nbdkit-backing.args.disk0 | 8 +
...e-curl-nbdkit-backing.args.disk0.pipe.1778 | 1 +
.../disk-network-source-curl.args.1.pipe.1 | 1 +
.../disk-network-source-curl.args.disk0 | 8 +
...k-network-source-curl.args.disk0.pipe.1778 | 1 +
.../disk-network-source-curl.args.disk1 | 10 +
...k-network-source-curl.args.disk1.pipe.1780 | 1 +
...k-network-source-curl.args.disk1.pipe.1782 | 1 +
...isk-network-source-curl.args.disk1.pipe.49 | 1 +
.../disk-network-source-curl.args.disk2 | 8 +
...k-network-source-curl.args.disk2.pipe.1782 | 1 +
...k-network-source-curl.args.disk2.pipe.1784 | 1 +
...isk-network-source-curl.args.disk2.pipe.51 | 1 +
.../disk-network-source-curl.args.disk3 | 7 +
.../disk-network-source-curl.args.disk4 | 7 +
.../disk-network-ssh.args.disk0 | 7 +
tests/qemunbdkittest.c | 294 ++++
...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 +
.../disk-cdrom-network-nbdkit.xml | 1 +
...isk-network-http-nbdkit.x86_64-latest.args | 45 +
.../disk-network-http-nbdkit.xml | 1 +
...rce-curl-nbdkit-backing.x86_64-latest.args | 38 +
...isk-network-source-curl-nbdkit-backing.xml | 45 +
...work-source-curl-nbdkit.x86_64-latest.args | 50 +
.../disk-network-source-curl-nbdkit.xml | 1 +
...isk-network-source-curl.x86_64-latest.args | 54 +
.../disk-network-source-curl.xml | 74 +
...disk-network-ssh-nbdkit.x86_64-latest.args | 36 +
.../disk-network-ssh-nbdkit.xml | 1 +
.../disk-network-ssh.x86_64-latest.args | 36 +
tests/qemuxml2argvdata/disk-network-ssh.xml | 31 +
tests/qemuxml2argvtest.c | 18 +
tests/testutilsqemu.c | 27 +
tests/testutilsqemu.h | 5 +
70 files changed, 2707 insertions(+), 116 deletions(-)
create mode 100644 src/qemu/qemu_nbdkit.c
create mode 100644 src/qemu/qemu_nbdkit.h
create mode 100644 src/qemu/qemu_nbdkitpriv.h
create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk0
create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1
create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1.pipe.1778
create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2
create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2.pipe.1780
create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk0
create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk1
create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2
create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2.pipe.1778
create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3
create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3.pipe.1780
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0.pipe.1778
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.1.pipe.1
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk0
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk0.pipe.1778
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.1780
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.1782
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.49
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.1782
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.1784
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.51
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk3
create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk4
create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk0
create mode 100644 tests/qemunbdkittest.c
create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args
create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args
create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args
create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-nbdkit.x86_64-latest.args
create mode 120000 tests/qemuxml2argvdata/disk-network-ssh-nbdkit.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.xml
--
2.37.3