hw/net/Makefile.objs | 2 +- hw/net/vhost_net.c | 37 +++ hw/net/vhost_pci_net.c | 137 +++++++++++ hw/virtio/Makefile.objs | 1 + hw/virtio/vhost-pci-slave.c | 310 +++++++++++++++++++++++++ hw/virtio/vhost-user.c | 117 ++-------- hw/virtio/vhost.c | 63 +++-- hw/virtio/virtio-pci.c | 55 +++++ hw/virtio/virtio-pci.h | 14 ++ include/hw/pci/pci.h | 1 + include/hw/virtio/vhost-backend.h | 2 + include/hw/virtio/vhost-pci-net.h | 42 ++++ include/hw/virtio/vhost-pci-slave.h | 12 + include/hw/virtio/vhost-user.h | 108 +++++++++ include/hw/virtio/vhost.h | 2 + include/net/vhost_net.h | 2 + include/standard-headers/linux/vhost_pci_net.h | 65 ++++++ include/standard-headers/linux/virtio_ids.h | 1 + 18 files changed, 851 insertions(+), 120 deletions(-) create mode 100644 hw/net/vhost_pci_net.c create mode 100644 hw/virtio/vhost-pci-slave.c create mode 100644 include/hw/virtio/vhost-pci-net.h create mode 100644 include/hw/virtio/vhost-pci-slave.h create mode 100644 include/hw/virtio/vhost-user.h create mode 100644 include/standard-headers/linux/vhost_pci_net.h
Vhost-pci is a point-to-point based inter-VM communication solution. This
patch series implements the vhost-pci-net device setup and emulation. The
device is implemented as a virtio device, and it is set up via the
vhost-user protocol to get the neessary info (e.g the memory info of the
remote VM, vring info).
Currently, only the fundamental functions are implemented. More features,
such as MQ and live migration, will be updated in the future.
The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here:
http://dpdk.org/ml/archives/dev/2017-November/082615.html
v2->v3 changes:
1) static device creation: instead of creating and hot-plugging the
device when receiving a vhost-user msg, the device is not created
via the qemu booting command line.
2) remove vqs: rq and ctrlq are removed in this version.
- receive vq: the receive vq is not needed anymore. The PMD directly
shares the remote txq and rxq - grab from remote txq to
receive packets, and put to rxq to send packets.
- ctrlq: the ctrlq is replaced by the first 4KB metadata area of the
device Bar-2.
3) simpler implementation: the entire implementation has been tailored
from ~1800 LOC to ~850 LOC.
Wei Wang (7):
vhost-user: share the vhost-user protocol related structures
vhost-pci-net: add vhost-pci-net
virtio/virtio-pci.c: add vhost-pci-net-pci
vhost-pci-slave: add vhost-pci slave implementation
vhost-user: VHOST_USER_SET_VHOST_PCI msg
vhost-pci-slave: handle VHOST_USER_SET_VHOST_PCI
virtio/vhost.c: vhost-pci needs remote gpa
hw/net/Makefile.objs | 2 +-
hw/net/vhost_net.c | 37 +++
hw/net/vhost_pci_net.c | 137 +++++++++++
hw/virtio/Makefile.objs | 1 +
hw/virtio/vhost-pci-slave.c | 310 +++++++++++++++++++++++++
hw/virtio/vhost-user.c | 117 ++--------
hw/virtio/vhost.c | 63 +++--
hw/virtio/virtio-pci.c | 55 +++++
hw/virtio/virtio-pci.h | 14 ++
include/hw/pci/pci.h | 1 +
include/hw/virtio/vhost-backend.h | 2 +
include/hw/virtio/vhost-pci-net.h | 42 ++++
include/hw/virtio/vhost-pci-slave.h | 12 +
include/hw/virtio/vhost-user.h | 108 +++++++++
include/hw/virtio/vhost.h | 2 +
include/net/vhost_net.h | 2 +
include/standard-headers/linux/vhost_pci_net.h | 65 ++++++
include/standard-headers/linux/virtio_ids.h | 1 +
18 files changed, 851 insertions(+), 120 deletions(-)
create mode 100644 hw/net/vhost_pci_net.c
create mode 100644 hw/virtio/vhost-pci-slave.c
create mode 100644 include/hw/virtio/vhost-pci-net.h
create mode 100644 include/hw/virtio/vhost-pci-slave.h
create mode 100644 include/hw/virtio/vhost-user.h
create mode 100644 include/standard-headers/linux/vhost_pci_net.h
--
2.7.4
On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > Vhost-pci is a point-to-point based inter-VM communication solution. This > patch series implements the vhost-pci-net device setup and emulation. The > device is implemented as a virtio device, and it is set up via the > vhost-user protocol to get the neessary info (e.g the memory info of the > remote VM, vring info). > > Currently, only the fundamental functions are implemented. More features, > such as MQ and live migration, will be updated in the future. > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > http://dpdk.org/ml/archives/dev/2017-November/082615.html This thread has become long so I've summarized outstanding issues: * Whether to define a virtio device for each device type or to have a single virtio device that speaks the vhost-user protocol. Discussion ongoing. * Please structure the code so there is a common vhost-pci component that other device types (scsi, blk) could use. This will show that the code is designed with other device types (scsi, blk) in mind. * Please validate all inputs from vhost-user protocol messages to protect against buffer overflows and other bugs. * Please handle short reads/writes and EAGAIN with the UNIX domain socket. Do not use read/write_all() functions because they hang QEMU until I/O completes. * What happens when the vhost-user master disconnects while we're running? * How does the guest know when the QEMU vhost-user slave has finished initializing everything? It seems like a guest driver could access the device before things are initialized. * How can the vhost-user slave inside the guest participate in feature negotiation? It must be able to participate, otherwise slaves cannot disable features that QEMU supports but they don't want to support. It's not feasible to pass in host_features as a QEMU parameter because that would require libvirt, OpenStack, cloud providers, etc to add support so users can manually set the bits for their slave implementation. * How can the the guest limit the number of virtqueues? * Please include tests. See tests/virtio-net-test.c and tests/vhost-user-test.c for examples.
> * Please handle short reads/writes and EAGAIN with the UNIX domain socket. Do > not use read/write_all() functions because they hang QEMU until I/O > completes. I'm not sure I agree with this one. vhost-user uses this extensively right now. It might be a worth-while goal to drop this limitation but I don't see why we should start with vhost-pci. And in particular, VCPU can't make progress unless a slave is present. > * What happens when the vhost-user master disconnects while we're running? > > * How does the guest know when the QEMU vhost-user slave has finished > initializing everything? It seems like a guest driver could access the > device before things are initialized. > > * How can the vhost-user slave inside the guest participate in feature > negotiation? It must be able to participate, otherwise slaves cannot > disable features that QEMU supports but they don't want to support. > > It's not feasible to pass in host_features as a QEMU parameter because > that would require libvirt, OpenStack, cloud providers, etc to add > support so users can manually set the bits for their slave > implementation. > > * How can the the guest limit the number of virtqueues? I think it is feasible to pass in host features, # of vqs etc. Assuming compatibility with existing guests, I don't think you can do anything else really if you assume that vhost guest might boot after the virtio guest. So either you give up on compatibility, or you allow the vhost guest to block the virtio guest. I think compatibility is more important. We can later think about ways to add non-blocking behaviour as a feature. > > * Please include tests. See tests/virtio-net-test.c and > tests/vhost-user-test.c for examples. Unit tests are nice but an actual way to test without running a full blown dpdk stack would be nicer. Something along the lines of a port of vhost user bridge to the guest. -- MST
On Tue, Dec 19, 2017 at 2:56 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> * Please handle short reads/writes and EAGAIN with the UNIX domain socket. Do >> not use read/write_all() functions because they hang QEMU until I/O >> completes. > > I'm not sure I agree with this one. vhost-user uses this extensively > right now. It might be a worth-while goal to drop this limitation > but I don't see why we should start with vhost-pci. > > And in particular, VCPU can't make progress unless a slave is present. Hang on, we're talking about different things: The QEMU vhost-user master blocks because vhost_*() functions are synchronous (they don't use callbacks or yield). Fixing that is beyond the scope of this patch series and I'm not asking for it. This patch series adds a from-scratch vhost-user slave implementation which has no good reason to be blocking. A single malicious or broken guest must not be able to hang a vhost-pci network switch. >> * How can the the guest limit the number of virtqueues? > > I think it is feasible to pass in host features, # of vqs etc. Assuming > compatibility with existing guests, I don't think you can do anything > else really if you assume that vhost guest might boot after the > virtio guest. > > So either you give up on compatibility, or you allow the vhost > guest to block the virtio guest. > > I think compatibility is more important. > > We can later think about ways to add non-blocking behaviour > as a feature. I agree it's a separate feature because it will require non-vhost-pci related changes. I have posted a separate email thread to discuss a solution. >> >> * Please include tests. See tests/virtio-net-test.c and >> tests/vhost-user-test.c for examples. > > Unit tests are nice but an actual way to test without > running a full blown dpdk stack would be nicer. > Something along the lines of a port of vhost user bridge > to the guest. Yes!
On Tue, Dec 19, 2017 at 05:05:59PM +0000, Stefan Hajnoczi wrote: > On Tue, Dec 19, 2017 at 2:56 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> * Please handle short reads/writes and EAGAIN with the UNIX domain socket. Do > >> not use read/write_all() functions because they hang QEMU until I/O > >> completes. > > > > I'm not sure I agree with this one. vhost-user uses this extensively > > right now. It might be a worth-while goal to drop this limitation > > but I don't see why we should start with vhost-pci. > > > > And in particular, VCPU can't make progress unless a slave is present. > > Hang on, we're talking about different things: > > The QEMU vhost-user master blocks because vhost_*() functions are > synchronous (they don't use callbacks or yield). Fixing that is > beyond the scope of this patch series and I'm not asking for it. > > This patch series adds a from-scratch vhost-user slave implementation > which has no good reason to be blocking. A single malicious or broken > guest must not be able to hang a vhost-pci network switch. Hmm that's not an easy change. But I agree, it's more important for the switch. > >> * How can the the guest limit the number of virtqueues? > > > > I think it is feasible to pass in host features, # of vqs etc. Assuming > > compatibility with existing guests, I don't think you can do anything > > else really if you assume that vhost guest might boot after the > > virtio guest. > > > > So either you give up on compatibility, or you allow the vhost > > guest to block the virtio guest. > > > > I think compatibility is more important. > > > > We can later think about ways to add non-blocking behaviour > > as a feature. > > I agree it's a separate feature because it will require non-vhost-pci > related changes. > > I have posted a separate email thread to discuss a solution. > > >> > >> * Please include tests. See tests/virtio-net-test.c and > >> tests/vhost-user-test.c for examples. > > > > Unit tests are nice but an actual way to test without > > running a full blown dpdk stack would be nicer. > > Something along the lines of a port of vhost user bridge > > to the guest. > > Yes!
On Wed, Dec 20, 2017 at 4:06 AM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Tue, Dec 19, 2017 at 05:05:59PM +0000, Stefan Hajnoczi wrote: >> On Tue, Dec 19, 2017 at 2:56 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> * Please handle short reads/writes and EAGAIN with the UNIX domain socket. Do >> >> not use read/write_all() functions because they hang QEMU until I/O >> >> completes. >> > >> > I'm not sure I agree with this one. vhost-user uses this extensively >> > right now. It might be a worth-while goal to drop this limitation >> > but I don't see why we should start with vhost-pci. >> > >> > And in particular, VCPU can't make progress unless a slave is present. >> >> Hang on, we're talking about different things: >> >> The QEMU vhost-user master blocks because vhost_*() functions are >> synchronous (they don't use callbacks or yield). Fixing that is >> beyond the scope of this patch series and I'm not asking for it. >> >> This patch series adds a from-scratch vhost-user slave implementation >> which has no good reason to be blocking. A single malicious or broken >> guest must not be able to hang a vhost-pci network switch. > > Hmm that's not an easy change. But I agree, it's more important for > the switch. It's easy, See "[PATCH v3 4/7] vhost-pci-slave: add vhost-pci slave implementation". There is only one qemu_chr_fe_read_all() call and one qemu_chr_fe_write_all() call! The read is the first thing that happens when the socket becomes readable. The write is the last thing that happens when a vhost-user protocol message has been handled. Moreover, the socket is only used in half-duplex with 1 request/response at any given time. This means most of the code will be unchanged. Only the beginning of vp_slave_read() and vp_slave_write() need to be adapted. I'm not asking for a lot. Stefan
Hi,
This series failed automatic build test. Please find the testing commands and
their output below. If you have docker installed, you can probably reproduce it
locally.
Subject: [Qemu-devel] [PATCH v3 0/7] Vhost-pci for inter-VM communication
Type: series
Message-id: 1512444796-30615-1-git-send-email-wei.w.wang@intel.com
=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-quick@centos6
time make docker-test-build@min-glib
time make docker-test-mingw@fedora
# iotests is broken now, skip
# time make docker-test-block@fedora
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
64719e0dfc virtio/vhost.c: vhost-pci needs remote gpa
80b0909098 vhost-pci-slave: handle VHOST_USER_SET_VHOST_PCI
f9fc141d85 vhost-user: VHOST_USER_SET_VHOST_PCI msg
5bde570154 vhost-pci-slave: add vhost-pci slave implementation
ea59537930 virtio/virtio-pci.c: add vhost-pci-net-pci
563e308b5a vhost-pci-net: add vhost-pci-net
f307bb5850 vhost-user: share the vhost-user protocol related structures
=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/dtc'...
Submodule path 'dtc': checked out '558cd81bdd432769b59bff01240c44f82cfb1a9d'
BUILD centos6
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-r1w5vyee/src'
GEN /var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-22.59.39.2059/qemu.tar
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-22.59.39.2059/qemu.tar.vroot'...
done.
Checking out files: 44% (2518/5667)
Checking out files: 45% (2551/5667)
Checking out files: 46% (2607/5667)
Checking out files: 47% (2664/5667)
Checking out files: 48% (2721/5667)
Checking out files: 49% (2777/5667)
Checking out files: 50% (2834/5667)
Checking out files: 51% (2891/5667)
Checking out files: 52% (2947/5667)
Checking out files: 53% (3004/5667)
Checking out files: 54% (3061/5667)
Checking out files: 55% (3117/5667)
Checking out files: 56% (3174/5667)
Checking out files: 57% (3231/5667)
Checking out files: 58% (3287/5667)
Checking out files: 59% (3344/5667)
Checking out files: 60% (3401/5667)
Checking out files: 61% (3457/5667)
Checking out files: 62% (3514/5667)
Checking out files: 63% (3571/5667)
Checking out files: 64% (3627/5667)
Checking out files: 65% (3684/5667)
Checking out files: 66% (3741/5667)
Checking out files: 67% (3797/5667)
Checking out files: 68% (3854/5667)
Checking out files: 69% (3911/5667)
Checking out files: 70% (3967/5667)
Checking out files: 71% (4024/5667)
Checking out files: 72% (4081/5667)
Checking out files: 73% (4137/5667)
Checking out files: 74% (4194/5667)
Checking out files: 75% (4251/5667)
Checking out files: 76% (4307/5667)
Checking out files: 77% (4364/5667)
Checking out files: 78% (4421/5667)
Checking out files: 79% (4477/5667)
Checking out files: 80% (4534/5667)
Checking out files: 81% (4591/5667)
Checking out files: 82% (4647/5667)
Checking out files: 83% (4704/5667)
Checking out files: 84% (4761/5667)
Checking out files: 85% (4817/5667)
Checking out files: 86% (4874/5667)
Checking out files: 87% (4931/5667)
Checking out files: 88% (4987/5667)
Checking out files: 89% (5044/5667)
Checking out files: 90% (5101/5667)
Checking out files: 91% (5157/5667)
Checking out files: 92% (5214/5667)
Checking out files: 93% (5271/5667)
Checking out files: 94% (5327/5667)
Checking out files: 94% (5378/5667)
Checking out files: 95% (5384/5667)
Checking out files: 96% (5441/5667)
Checking out files: 97% (5497/5667)
Checking out files: 98% (5554/5667)
Checking out files: 99% (5611/5667)
Checking out files: 100% (5667/5667)
Checking out files: 100% (5667/5667), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-22.59.39.2059/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out '558cd81bdd432769b59bff01240c44f82cfb1a9d'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-22.59.39.2059/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '10739aa26051a5d49d88132604539d3ed085e72e'
COPY RUNNER
RUN test-quick in qemu:centos6
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
bison-2.4.1-5.el6.x86_64
bzip2-devel-1.0.5-7.el6_0.x86_64
ccache-3.1.6-2.el6.x86_64
csnappy-devel-0-6.20150729gitd7bc683.el6.x86_64
flex-2.5.35-9.el6.x86_64
gcc-4.4.7-18.el6.x86_64
gettext-0.17-18.el6.x86_64
git-1.7.1-9.el6_9.x86_64
glib2-devel-2.28.8-9.el6.x86_64
libepoxy-devel-1.2-3.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
librdmacm-devel-1.0.21-0.el6.x86_64
lzo-devel-2.03-3.1.el6_5.1.x86_64
make-3.81-23.el6.x86_64
mesa-libEGL-devel-11.0.7-4.el6.x86_64
mesa-libgbm-devel-11.0.7-4.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
spice-glib-devel-0.26-8.el6.x86_64
spice-server-devel-0.12.4-16.el6.x86_64
tar-1.23-15.el6_8.x86_64
vte-devel-0.25.1-9.el6.x86_64
xen-devel-4.6.6-2.el6.x86_64
zlib-devel-1.2.3-29.el6.x86_64
Environment variables:
PACKAGES=bison bzip2-devel ccache csnappy-devel flex g++ gcc gettext git glib2-devel libepoxy-devel libfdt-devel librdmacm-devel lzo-devel make mesa-libEGL-devel mesa-libgbm-devel pixman-devel SDL-devel spice-glib-devel spice-server-devel tar vte-devel xen-devel zlib-devel
HOSTNAME=11d98b15808f
MAKEFLAGS= -j8
J=8
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
FEATURES= dtc
DEBUG=
_=/usr/bin/env
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install
No C++ compiler available; disabling C++ specific optional code
Install prefix /tmp/qemu-test/install
BIOS directory /tmp/qemu-test/install/share/qemu
firmware path /tmp/qemu-test/install/share/qemu-firmware
binary directory /tmp/qemu-test/install/bin
library directory /tmp/qemu-test/install/lib
module directory /tmp/qemu-test/install/lib/qemu
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory /tmp/qemu-test/install/etc
local state directory /tmp/qemu-test/install/var
Manual directory /tmp/qemu-test/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path /tmp/qemu-test/src
GIT binary git
GIT submodules
C compiler cc
Host C compiler cc
C++ compiler
Objective-C compiler cc
ARFLAGS rv
CFLAGS -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g
QEMU_CFLAGS -I/usr/include/pixman-1 -I$(SRC_PATH)/dtc/libfdt -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DNCURSES_WIDECHAR -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -Wendif-labels -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all -Wno-missing-braces -I/usr/include/libpng12 -I/usr/include/libdrm -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/nss3 -I/usr/include/nspr4 -I/usr/include/spice-1
LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g
make make
install install
python python -B
smbd /usr/sbin/smbd
module support no
host CPU x86_64
host big endian no
target list x86_64-softmmu aarch64-softmmu
gprof enabled no
sparse enabled no
strip binaries yes
profiler no
static build no
SDL support yes (1.2.14)
GTK support yes (2.24.23)
GTK GL support no
VTE support yes (0.25.1)
TLS priority NORMAL
GNUTLS support no
GNUTLS rnd no
libgcrypt no
libgcrypt kdf no
nettle no
nettle kdf no
libtasn1 no
curses support yes
virgl support no
curl support no
mingw32 support no
Audio drivers oss
Block whitelist (rw)
Block whitelist (ro)
VirtFS support no
Multipath support no
VNC support yes
VNC SASL support no
VNC JPEG support yes
VNC PNG support yes
xen support yes
xen ctrl version 40600
pv dom build no
brlapi support no
bluez support no
Documentation no
PIE yes
vde support no
netmap support no
Linux AIO support no
ATTR/XATTR support yes
Install blobs yes
KVM support yes
HAX support no
TCG support yes
TCG debug enabled no
TCG interpreter no
RDMA support yes
fdt support yes
preadv support yes
fdatasync yes
madvise yes
posix_madvise yes
libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends log
spice support yes (0.12.6/0.12.4)
rbd support no
xfsctl support no
smartcard support yes
libusb no
usb net redir no
OpenGL support yes
OpenGL dmabufs no
libiscsi support no
libnfs support no
build guest agent yes
QGA VSS support no
QGA w32 disk info no
QGA MSI support no
seccomp support no
coroutine backend ucontext
coroutine pool yes
debug stack usage no
crypto afalg no
GlusterFS support no
gcov gcov
gcov enabled no
TPM support yes
libssh2 support no
TPM passthrough yes
TPM emulator yes
QOM debugging yes
Live block migration yes
lzo support yes
snappy support no
bzip2 support yes
NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization no
replication support yes
VxHS block device no
capstone no
mkdir -p dtc/libfdt
mkdir -p dtc/tests
GEN aarch64-softmmu/config-devices.mak.tmp
GEN x86_64-softmmu/config-devices.mak.tmp
GEN qemu-options.def
GEN qmp-commands.h
GEN config-host.h
GEN qapi-visit.h
GEN qapi-types.h
GEN qapi-event.h
GEN x86_64-softmmu/config-devices.mak
GEN qmp-marshal.c
GEN aarch64-softmmu/config-devices.mak
GEN qapi-types.c
GEN qapi-visit.c
GEN qapi-event.c
GEN qmp-introspect.h
GEN qmp-introspect.c
GEN trace/generated-tcg-tracers.h
GEN trace/generated-helpers-wrappers.h
GEN trace/generated-helpers.h
GEN trace/generated-helpers.c
GEN module_block.h
GEN ui/input-keymap-linux-to-qcode.c
GEN ui/input-keymap-qcode-to-qnum.c
GEN ui/input-keymap-qnum-to-qcode.c
GEN tests/test-qapi-types.h
GEN tests/test-qapi-visit.h
GEN tests/test-qmp-commands.h
GEN tests/test-qapi-event.h
GEN tests/test-qmp-introspect.h
GEN trace-root.h
GEN util/trace.h
GEN crypto/trace.h
GEN io/trace.h
GEN migration/trace.h
GEN block/trace.h
GEN chardev/trace.h
GEN hw/block/trace.h
GEN hw/block/dataplane/trace.h
GEN hw/char/trace.h
GEN hw/intc/trace.h
GEN hw/net/trace.h
GEN hw/virtio/trace.h
GEN hw/audio/trace.h
GEN hw/misc/trace.h
GEN hw/usb/trace.h
GEN hw/scsi/trace.h
GEN hw/nvram/trace.h
GEN hw/display/trace.h
GEN hw/input/trace.h
GEN hw/timer/trace.h
GEN hw/dma/trace.h
GEN hw/sparc/trace.h
GEN hw/sd/trace.h
GEN hw/isa/trace.h
GEN hw/mem/trace.h
GEN hw/i386/trace.h
GEN hw/i386/xen/trace.h
GEN hw/9pfs/trace.h
GEN hw/ppc/trace.h
GEN hw/pci/trace.h
GEN hw/s390x/trace.h
GEN hw/vfio/trace.h
GEN hw/acpi/trace.h
GEN hw/arm/trace.h
GEN hw/alpha/trace.h
GEN hw/xen/trace.h
GEN hw/ide/trace.h
GEN ui/trace.h
GEN audio/trace.h
GEN net/trace.h
GEN target/arm/trace.h
GEN target/i386/trace.h
GEN target/mips/trace.h
GEN target/sparc/trace.h
GEN target/s390x/trace.h
GEN target/ppc/trace.h
GEN qom/trace.h
GEN linux-user/trace.h
GEN qapi/trace.h
GEN accel/tcg/trace.h
GEN accel/kvm/trace.h
GEN nbd/trace.h
GEN scsi/trace.h
GEN trace-root.c
GEN util/trace.c
GEN crypto/trace.c
GEN io/trace.c
GEN migration/trace.c
GEN block/trace.c
GEN chardev/trace.c
GEN hw/block/trace.c
GEN hw/block/dataplane/trace.c
GEN hw/char/trace.c
GEN hw/intc/trace.c
GEN hw/net/trace.c
GEN hw/virtio/trace.c
GEN hw/audio/trace.c
GEN hw/misc/trace.c
GEN hw/usb/trace.c
GEN hw/scsi/trace.c
GEN hw/nvram/trace.c
GEN hw/display/trace.c
GEN hw/input/trace.c
GEN hw/timer/trace.c
GEN hw/dma/trace.c
GEN hw/sparc/trace.c
GEN hw/sd/trace.c
GEN hw/isa/trace.c
GEN hw/mem/trace.c
GEN hw/i386/trace.c
GEN hw/i386/xen/trace.c
GEN hw/9pfs/trace.c
GEN hw/ppc/trace.c
GEN hw/pci/trace.c
GEN hw/s390x/trace.c
GEN hw/vfio/trace.c
GEN hw/acpi/trace.c
GEN hw/arm/trace.c
GEN hw/alpha/trace.c
GEN hw/xen/trace.c
GEN hw/ide/trace.c
GEN ui/trace.c
GEN audio/trace.c
GEN net/trace.c
GEN target/arm/trace.c
GEN target/i386/trace.c
GEN target/mips/trace.c
GEN target/sparc/trace.c
GEN target/s390x/trace.c
GEN target/ppc/trace.c
GEN qom/trace.c
GEN linux-user/trace.c
GEN qapi/trace.c
GEN accel/tcg/trace.c
GEN accel/kvm/trace.c
GEN nbd/trace.c
GEN scsi/trace.c
GEN config-all-devices.mak
DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
DEP /tmp/qemu-test/src/dtc/tests/trees.S
DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
DEP /tmp/qemu-test/src/dtc/tests/testutils.c
DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
DEP /tmp/qemu-test/src/dtc/tests/check_path.c
DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
DEP /tmp/qemu-test/src/dtc/tests/overlay.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
DEP /tmp/qemu-test/src/dtc/tests/incbin.c
DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
DEP /tmp/qemu-test/src/dtc/tests/path-references.c
DEP /tmp/qemu-test/src/dtc/tests/references.c
DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
DEP /tmp/qemu-test/src/dtc/tests/del_node.c
DEP /tmp/qemu-test/src/dtc/tests/del_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop.c
DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/set_name.c
DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
DEP /tmp/qemu-test/src/dtc/tests/notfound.c
DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
DEP /tmp/qemu-test/src/dtc/tests/get_path.c
DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/getprop.c
DEP /tmp/qemu-test/src/dtc/tests/get_name.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
DEP /tmp/qemu-test/src/dtc/tests/find_property.c
DEP /tmp/qemu-test/src/dtc/tests/root_node.c
DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
DEP /tmp/qemu-test/src/dtc/util.c
DEP /tmp/qemu-test/src/dtc/fdtput.c
DEP /tmp/qemu-test/src/dtc/fdtget.c
DEP /tmp/qemu-test/src/dtc/fdtdump.c
LEX convert-dtsv0-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/srcpos.c
BISON dtc-parser.tab.c
LEX dtc-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/treesource.c
DEP /tmp/qemu-test/src/dtc/livetree.c
DEP /tmp/qemu-test/src/dtc/fstree.c
DEP /tmp/qemu-test/src/dtc/flattree.c
DEP /tmp/qemu-test/src/dtc/dtc.c
DEP /tmp/qemu-test/src/dtc/data.c
DEP /tmp/qemu-test/src/dtc/checks.c
DEP convert-dtsv0-lexer.lex.c
DEP dtc-parser.tab.c
DEP dtc-lexer.lex.c
CHK version_gen.h
UPD version_gen.h
DEP /tmp/qemu-test/src/dtc/util.c
CC libfdt/fdt.o
CC libfdt/fdt_ro.o
CC libfdt/fdt_wip.o
CC libfdt/fdt_sw.o
CC libfdt/fdt_strerror.o
CC libfdt/fdt_addresses.o
CC libfdt/fdt_rw.o
CC libfdt/fdt_empty_tree.o
CC libfdt/fdt_overlay.o
AR libfdt/libfdt.a
ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
mkdir -p dtc/libfdt
mkdir -p dtc/tests
GEN qga/qapi-generated/qga-qapi-visit.h
CC tests/qemu-iotests/socket_scm_helper.o
GEN qga/qapi-generated/qga-qapi-types.h
GEN qga/qapi-generated/qga-qapi-types.c
GEN qga/qapi-generated/qga-qmp-commands.h
CC qmp-introspect.o
GEN qga/qapi-generated/qga-qapi-visit.c
GEN qga/qapi-generated/qga-qmp-marshal.c
CC qapi-types.o
CC qapi-visit.o
CC qapi-event.o
CC qapi/qapi-visit-core.o
CC qapi/qapi-dealloc-visitor.o
CC qapi/qobject-input-visitor.o
CC qapi/qobject-output-visitor.o
CC qapi/qmp-registry.o
CC qapi/qmp-dispatch.o
CC qapi/string-input-visitor.o
CC qapi/string-output-visitor.o
CC qapi/opts-visitor.o
CC qapi/qapi-clone-visitor.o
CC qapi/qapi-util.o
CC qapi/qmp-event.o
CC qobject/qnull.o
CC qobject/qnum.o
CC qobject/qdict.o
CC qobject/qlist.o
CC qobject/qlit.o
CC qobject/qbool.o
CC qobject/qstring.o
CC qobject/qjson.o
CC qobject/qobject.o
CC qobject/json-lexer.o
CC qobject/json-streamer.o
CC qobject/json-parser.o
CC trace/qmp.o
CC trace/control.o
CC util/osdep.o
CC util/cutils.o
CC util/unicode.o
CC util/qemu-timer-common.o
CC util/bufferiszero.o
CC util/lockcnt.o
CC util/async.o
CC util/aiocb.o
CC util/thread-pool.o
CC util/qemu-timer.o
CC util/main-loop.o
CC util/iohandler.o
CC util/aio-posix.o
CC util/compatfd.o
CC util/event_notifier-posix.o
CC util/mmap-alloc.o
CC util/oslib-posix.o
CC util/qemu-openpty.o
CC util/qemu-thread-posix.o
CC util/memfd.o
CC util/envlist.o
CC util/path.o
CC util/module.o
CC util/host-utils.o
CC util/bitmap.o
CC util/bitops.o
CC util/hbitmap.o
CC util/fifo8.o
CC util/acl.o
CC util/cacheinfo.o
CC util/qemu-error.o
CC util/error.o
CC util/id.o
CC util/iov.o
CC util/qemu-config.o
CC util/qemu-sockets.o
CC util/uri.o
CC util/notify.o
CC util/qemu-option.o
CC util/qemu-progress.o
CC util/keyval.o
CC util/hexdump.o
CC util/crc32c.o
CC util/uuid.o
CC util/throttle.o
CC util/getauxval.o
CC util/readline.o
CC util/rcu.o
CC util/qemu-coroutine.o
CC util/qemu-coroutine-lock.o
CC util/qemu-coroutine-io.o
CC util/qemu-coroutine-sleep.o
CC util/coroutine-ucontext.o
CC util/buffer.o
CC util/timed-average.o
CC util/base64.o
CC util/log.o
CC util/pagesize.o
CC util/qdist.o
CC util/qht.o
CC util/range.o
CC util/stats64.o
CC util/systemd.o
CC trace-root.o
CC util/trace.o
CC crypto/trace.o
CC io/trace.o
CC migration/trace.o
CC block/trace.o
CC chardev/trace.o
CC hw/block/trace.o
CC hw/block/dataplane/trace.o
CC hw/char/trace.o
CC hw/intc/trace.o
CC hw/net/trace.o
CC hw/audio/trace.o
CC hw/virtio/trace.o
CC hw/misc/trace.o
CC hw/usb/trace.o
CC hw/scsi/trace.o
CC hw/nvram/trace.o
CC hw/display/trace.o
CC hw/timer/trace.o
CC hw/input/trace.o
CC hw/dma/trace.o
CC hw/sparc/trace.o
CC hw/sd/trace.o
CC hw/isa/trace.o
CC hw/mem/trace.o
CC hw/i386/trace.o
CC hw/i386/xen/trace.o
CC hw/9pfs/trace.o
CC hw/ppc/trace.o
CC hw/pci/trace.o
CC hw/s390x/trace.o
CC hw/vfio/trace.o
CC hw/acpi/trace.o
CC hw/arm/trace.o
CC hw/alpha/trace.o
CC hw/xen/trace.o
CC hw/ide/trace.o
CC ui/trace.o
CC audio/trace.o
CC net/trace.o
CC target/arm/trace.o
CC target/i386/trace.o
CC target/mips/trace.o
CC target/sparc/trace.o
CC target/ppc/trace.o
CC target/s390x/trace.o
CC qom/trace.o
CC linux-user/trace.o
CC qapi/trace.o
CC accel/tcg/trace.o
CC accel/kvm/trace.o
CC nbd/trace.o
CC scsi/trace.o
CC crypto/pbkdf-stub.o
CC stubs/arch-query-cpu-def.o
CC stubs/arch-query-cpu-model-expansion.o
CC stubs/arch-query-cpu-model-comparison.o
CC stubs/arch-query-cpu-model-baseline.o
CC stubs/bdrv-next-monitor-owned.o
CC stubs/blk-commit-all.o
CC stubs/blockdev-close-all-bdrv-states.o
CC stubs/clock-warp.o
CC stubs/cpu-get-clock.o
CC stubs/cpu-get-icount.o
CC stubs/dump.o
CC stubs/error-printf.o
CC stubs/fdset.o
CC stubs/gdbstub.o
CC stubs/get-vm-name.o
CC stubs/iothread.o
CC stubs/iothread-lock.o
CC stubs/is-daemonized.o
CC stubs/machine-init-done.o
CC stubs/migr-blocker.o
CC stubs/change-state-handler.o
CC stubs/monitor.o
CC stubs/notify-event.o
CC stubs/qtest.o
CC stubs/replay.o
CC stubs/set-fd-handler.o
CC stubs/runstate-check.o
CC stubs/slirp.o
CC stubs/sysbus.o
CC stubs/tpm.o
CC stubs/uuid.o
CC stubs/vm-stop.o
CC stubs/trace-control.o
CC stubs/vmstate.o
CC stubs/qmp_pc_dimm.o
CC stubs/target-monitor-defs.o
CC stubs/target-get-monitor-def.o
CC stubs/pc_madt_cpu_entry.o
CC stubs/vmgenid.o
CC stubs/xen-common.o
CC stubs/xen-hvm.o
CC stubs/pci-host-piix.o
CC contrib/ivshmem-client/ivshmem-client.o
CC contrib/ivshmem-client/main.o
CC contrib/ivshmem-server/ivshmem-server.o
CC qemu-nbd.o
CC contrib/ivshmem-server/main.o
CC block.o
CC blockjob.o
CC qemu-io-cmds.o
CC replication.o
CC block/raw-format.o
CC block/qcow.o
CC block/vdi.o
CC block/vmdk.o
CC block/cloop.o
CC block/bochs.o
CC block/vvfat.o
CC block/vpc.o
CC block/dmg.o
CC block/qcow2.o
CC block/qcow2-refcount.o
CC block/qcow2-cluster.o
CC block/qcow2-snapshot.o
CC block/qcow2-cache.o
CC block/qcow2-bitmap.o
CC block/qed.o
CC block/qed-l2-cache.o
CC block/qed-cluster.o
CC block/vhdx.o
CC block/qed-check.o
CC block/qed-table.o
CC block/vhdx-endian.o
CC block/vhdx-log.o
CC block/quorum.o
CC block/parallels.o
CC block/blkdebug.o
CC block/blkverify.o
CC block/blkreplay.o
CC block/block-backend.o
CC block/snapshot.o
CC block/qapi.o
CC block/file-posix.o
CC block/null.o
CC block/mirror.o
CC block/commit.o
CC block/io.o
CC block/throttle-groups.o
CC block/nbd.o
CC block/nbd-client.o
CC block/sheepdog.o
CC block/accounting.o
CC block/dirty-bitmap.o
CC block/write-threshold.o
CC block/backup.o
CC block/replication.o
CC block/throttle.o
CC block/crypto.o
CC nbd/server.o
CC nbd/client.o
CC nbd/common.o
CC scsi/utils.o
CC scsi/pr-manager.o
CC scsi/pr-manager-helper.o
CC block/dmg-bz2.o
CC crypto/init.o
CC crypto/hash.o
CC crypto/hash-glib.o
CC crypto/hmac.o
CC crypto/hmac-glib.o
CC crypto/aes.o
CC crypto/desrfb.o
CC crypto/cipher.o
CC crypto/tlscreds.o
CC crypto/tlscredsanon.o
CC crypto/tlscredsx509.o
CC crypto/tlssession.o
CC crypto/secret.o
CC crypto/random-platform.o
CC crypto/pbkdf.o
CC crypto/ivgen.o
CC crypto/ivgen-essiv.o
CC crypto/ivgen-plain.o
CC crypto/ivgen-plain64.o
CC crypto/afsplit.o
CC crypto/xts.o
CC crypto/block.o
CC crypto/block-qcow.o
CC crypto/block-luks.o
CC io/channel-buffer.o
CC io/channel.o
CC io/channel-command.o
CC io/channel-file.o
CC io/channel-socket.o
CC io/channel-tls.o
CC io/channel-watch.o
CC io/channel-websock.o
CC io/channel-util.o
CC io/dns-resolver.o
CC io/task.o
CC qom/object.o
CC qom/container.o
CC qom/qom-qobject.o
CC qom/object_interfaces.o
GEN qemu-img-cmds.h
CC qemu-io.o
CC scsi/qemu-pr-helper.o
CC qemu-bridge-helper.o
CC blockdev.o
CC blockdev-nbd.o
CC bootdevice.o
CC iothread.o
CC qdev-monitor.o
CC device-hotplug.o
CC os-posix.o
CC bt-host.o
CC bt-vhci.o
CC dma-helpers.o
CC vl.o
CC tpm.o
CC qmp-marshal.o
CC device_tree.o
CC qmp.o
CC hmp.o
CC cpus-common.o
CC audio/audio.o
CC audio/noaudio.o
CC audio/wavaudio.o
CC audio/mixeng.o
CC audio/sdlaudio.o
CC audio/ossaudio.o
CC audio/spiceaudio.o
CC audio/wavcapture.o
CC backends/rng.o
CC backends/rng-egd.o
CC backends/rng-random.o
CC backends/tpm.o
CC backends/hostmem.o
CC backends/hostmem-ram.o
CC backends/hostmem-file.o
CC backends/cryptodev.o
CC backends/cryptodev-builtin.o
CC block/stream.o
CC chardev/msmouse.o
CC chardev/wctablet.o
CC chardev/testdev.o
CC chardev/spice.o
CC disas/arm.o
CC disas/i386.o
CC fsdev/qemu-fsdev-dummy.o
CC fsdev/qemu-fsdev-opts.o
CC fsdev/qemu-fsdev-throttle.o
CC hw/acpi/core.o
CC hw/acpi/piix4.o
CC hw/acpi/pcihp.o
CC hw/acpi/ich9.o
CC hw/acpi/tco.o
CC hw/acpi/cpu_hotplug.o
CC hw/acpi/memory_hotplug.o
CC hw/acpi/cpu.o
CC hw/acpi/nvdimm.o
CC hw/acpi/vmgenid.o
CC hw/acpi/acpi_interface.o
CC hw/acpi/bios-linker-loader.o
CC hw/acpi/aml-build.o
CC hw/acpi/ipmi.o
CC hw/acpi/acpi-stub.o
CC hw/acpi/ipmi-stub.o
CC hw/audio/sb16.o
CC hw/audio/es1370.o
CC hw/audio/ac97.o
CC hw/audio/fmopl.o
CC hw/audio/adlib.o
CC hw/audio/gus.o
CC hw/audio/gusemu_hal.o
CC hw/audio/gusemu_mixer.o
CC hw/audio/cs4231a.o
CC hw/audio/intel-hda.o
CC hw/audio/hda-codec.o
CC hw/audio/pcspk.o
CC hw/audio/wm8750.o
CC hw/audio/pl041.o
CC hw/audio/lm4549.o
CC hw/audio/marvell_88w8618.o
CC hw/audio/soundhw.o
CC hw/block/block.o
CC hw/block/cdrom.o
CC hw/block/hd-geometry.o
CC hw/block/fdc.o
CC hw/block/m25p80.o
CC hw/block/nand.o
CC hw/block/pflash_cfi01.o
CC hw/block/pflash_cfi02.o
CC hw/block/xen_disk.o
CC hw/block/ecc.o
CC hw/block/onenand.o
CC hw/block/nvme.o
CC hw/bt/core.o
CC hw/bt/l2cap.o
CC hw/bt/sdp.o
CC hw/bt/hci.o
CC hw/bt/hid.o
CC hw/bt/hci-csr.o
CC hw/char/ipoctal232.o
CC hw/char/parallel.o
CC hw/char/pl011.o
CC hw/char/serial.o
CC hw/char/serial-isa.o
CC hw/char/serial-pci.o
CC hw/char/virtio-console.o
CC hw/char/xen_console.o
CC hw/char/cadence_uart.o
CC hw/char/cmsdk-apb-uart.o
CC hw/char/debugcon.o
CC hw/char/imx_serial.o
CC hw/core/qdev.o
CC hw/core/qdev-properties.o
CC hw/core/bus.o
CC hw/core/reset.o
CC hw/core/fw-path-provider.o
CC hw/core/irq.o
CC hw/core/hotplug.o
CC hw/core/nmi.o
CC hw/core/ptimer.o
CC hw/core/machine.o
CC hw/core/sysbus.o
CC hw/core/loader.o
CC hw/core/qdev-properties-system.o
CC hw/core/register.o
CC hw/core/or-irq.o
CC hw/core/platform-bus.o
CC hw/cpu/core.o
CC hw/display/ads7846.o
CC hw/display/cirrus_vga.o
CC hw/display/pl110.o
CC hw/display/ssd0303.o
CC hw/display/ssd0323.o
CC hw/display/xenfb.o
CC hw/display/vga-pci.o
CC hw/display/vga-isa.o
CC hw/display/vmware_vga.o
CC hw/display/blizzard.o
CC hw/display/exynos4210_fimd.o
CC hw/display/framebuffer.o
CC hw/display/tc6393xb.o
CC hw/display/qxl.o
CC hw/display/qxl-logger.o
CC hw/display/qxl-render.o
CC hw/dma/pl080.o
CC hw/dma/pl330.o
CC hw/dma/i8257.o
CC hw/dma/xlnx-zynq-devcfg.o
CC hw/gpio/max7310.o
CC hw/gpio/pl061.o
CC hw/gpio/zaurus.o
CC hw/gpio/gpio_key.o
CC hw/i2c/core.o
CC hw/i2c/smbus.o
CC hw/i2c/smbus_eeprom.o
CC hw/i2c/i2c-ddc.o
CC hw/i2c/versatile_i2c.o
CC hw/i2c/smbus_ich9.o
CC hw/i2c/pm_smbus.o
CC hw/i2c/bitbang_i2c.o
CC hw/i2c/exynos4210_i2c.o
CC hw/i2c/imx_i2c.o
CC hw/i2c/aspeed_i2c.o
CC hw/ide/core.o
CC hw/ide/atapi.o
CC hw/ide/qdev.o
CC hw/ide/pci.o
CC hw/ide/isa.o
CC hw/ide/piix.o
CC hw/ide/microdrive.o
CC hw/ide/ahci.o
CC hw/ide/ich.o
CC hw/ide/ahci-allwinner.o
CC hw/input/hid.o
CC hw/input/lm832x.o
CC hw/input/pckbd.o
CC hw/input/pl050.o
CC hw/input/ps2.o
CC hw/input/stellaris_input.o
CC hw/input/tsc2005.o
CC hw/input/vmmouse.o
CC hw/input/virtio-input.o
CC hw/input/virtio-input-hid.o
CC hw/input/virtio-input-host.o
CC hw/intc/i8259_common.o
CC hw/intc/i8259.o
CC hw/intc/pl190.o
CC hw/intc/imx_avic.o
CC hw/intc/realview_gic.o
CC hw/intc/ioapic_common.o
CC hw/intc/arm_gic_common.o
CC hw/intc/arm_gic.o
CC hw/intc/arm_gicv2m.o
CC hw/intc/arm_gicv3_common.o
CC hw/intc/arm_gicv3.o
CC hw/intc/arm_gicv3_dist.o
CC hw/intc/arm_gicv3_redist.o
CC hw/intc/arm_gicv3_its_common.o
CC hw/intc/intc.o
CC hw/ipack/ipack.o
CC hw/ipack/tpci200.o
CC hw/ipmi/ipmi_bmc_sim.o
CC hw/ipmi/ipmi.o
CC hw/ipmi/ipmi_bmc_extern.o
CC hw/ipmi/isa_ipmi_kcs.o
CC hw/ipmi/isa_ipmi_bt.o
CC hw/isa/isa-bus.o
CC hw/isa/apm.o
CC hw/mem/pc-dimm.o
CC hw/mem/nvdimm.o
CC hw/misc/applesmc.o
CC hw/misc/max111x.o
CC hw/misc/tmp105.o
CC hw/misc/tmp421.o
CC hw/misc/debugexit.o
CC hw/misc/sga.o
CC hw/misc/pc-testdev.o
CC hw/misc/pci-testdev.o
CC hw/misc/edu.o
CC hw/misc/unimp.o
CC hw/misc/vmcoreinfo.o
CC hw/misc/arm_l2x0.o
CC hw/misc/arm_integrator_debug.o
CC hw/misc/a9scu.o
CC hw/misc/arm11scu.o
CC hw/net/xen_nic.o
CC hw/net/ne2000.o
CC hw/net/eepro100.o
CC hw/net/pcnet-pci.o
CC hw/net/pcnet.o
CC hw/net/e1000.o
CC hw/net/e1000x_common.o
CC hw/net/net_tx_pkt.o
CC hw/net/net_rx_pkt.o
CC hw/net/e1000e.o
CC hw/net/e1000e_core.o
CC hw/net/rtl8139.o
CC hw/net/vmxnet3.o
CC hw/net/smc91c111.o
CC hw/net/lan9118.o
CC hw/net/ne2000-isa.o
CC hw/net/xgmac.o
CC hw/net/allwinner_emac.o
CC hw/net/imx_fec.o
CC hw/net/cadence_gem.o
CC hw/net/stellaris_enet.o
CC hw/net/ftgmac100.o
CC hw/net/rocker/rocker.o
CC hw/net/rocker/rocker_fp.o
CC hw/net/rocker/rocker_desc.o
CC hw/net/rocker/rocker_world.o
CC hw/net/rocker/rocker_of_dpa.o
CC hw/nvram/eeprom93xx.o
CC hw/nvram/fw_cfg.o
CC hw/nvram/chrp_nvram.o
CC hw/pci-bridge/pci_bridge_dev.o
CC hw/pci-bridge/pcie_root_port.o
CC hw/pci-bridge/gen_pcie_root_port.o
CC hw/pci-bridge/pcie_pci_bridge.o
CC hw/pci-bridge/pci_expander_bridge.o
CC hw/pci-bridge/xio3130_upstream.o
CC hw/pci-bridge/xio3130_downstream.o
CC hw/pci-bridge/ioh3420.o
CC hw/pci-bridge/i82801b11.o
CC hw/pci-host/pam.o
CC hw/pci-host/versatile.o
CC hw/pci-host/piix.o
CC hw/pci-host/q35.o
CC hw/pci-host/gpex.o
CC hw/pci/pci.o
CC hw/pci/pci_bridge.o
CC hw/pci/msix.o
CC hw/pci/msi.o
CC hw/pci/shpc.o
CC hw/pci/slotid_cap.o
CC hw/pci/pci_host.o
CC hw/pci/pcie_host.o
CC hw/pci/pcie.o
CC hw/pci/pcie_aer.o
CC hw/pci/pcie_port.o
CC hw/pci/pci-stub.o
CC hw/pcmcia/pcmcia.o
CC hw/scsi/scsi-disk.o
CC hw/scsi/scsi-bus.o
CC hw/scsi/scsi-generic.o
CC hw/scsi/lsi53c895a.o
CC hw/scsi/mptsas.o
CC hw/scsi/mptconfig.o
CC hw/scsi/mptendian.o
CC hw/scsi/megasas.o
CC hw/scsi/vmw_pvscsi.o
CC hw/scsi/esp.o
CC hw/scsi/esp-pci.o
CC hw/sd/pl181.o
CC hw/sd/ssi-sd.o
CC hw/sd/sd.o
CC hw/sd/core.o
CC hw/sd/sdhci.o
CC hw/smbios/smbios.o
CC hw/smbios/smbios_type_38.o
CC hw/smbios/smbios-stub.o
CC hw/smbios/smbios_type_38-stub.o
CC hw/ssi/pl022.o
CC hw/ssi/ssi.o
CC hw/ssi/xilinx_spips.o
CC hw/ssi/aspeed_smc.o
CC hw/ssi/stm32f2xx_spi.o
CC hw/ssi/mss-spi.o
CC hw/timer/arm_timer.o
CC hw/timer/arm_mptimer.o
CC hw/timer/armv7m_systick.o
CC hw/timer/a9gtimer.o
CC hw/timer/cadence_ttc.o
CC hw/timer/ds1338.o
CC hw/timer/hpet.o
CC hw/timer/i8254_common.o
CC hw/timer/i8254.o
CC hw/timer/pl031.o
CC hw/timer/twl92230.o
CC hw/timer/imx_epit.o
CC hw/timer/imx_gpt.o
CC hw/timer/stm32f2xx_timer.o
CC hw/timer/aspeed_timer.o
CC hw/timer/cmsdk-apb-timer.o
CC hw/timer/mss-timer.o
CC hw/tpm/tpm_tis.o
CC hw/tpm/tpm_passthrough.o
CC hw/tpm/tpm_util.o
CC hw/tpm/tpm_emulator.o
CC hw/usb/core.o
CC hw/usb/combined-packet.o
CC hw/usb/bus.o
CC hw/usb/libhw.o
CC hw/usb/desc.o
CC hw/usb/desc-msos.o
CC hw/usb/hcd-uhci.o
CC hw/usb/hcd-ohci.o
CC hw/usb/hcd-ehci.o
CC hw/usb/hcd-ehci-pci.o
CC hw/usb/hcd-ehci-sysbus.o
CC hw/usb/hcd-xhci.o
CC hw/usb/hcd-xhci-nec.o
CC hw/usb/hcd-musb.o
CC hw/usb/dev-hub.o
CC hw/usb/dev-hid.o
CC hw/usb/dev-wacom.o
CC hw/usb/dev-storage.o
CC hw/usb/dev-uas.o
CC hw/usb/dev-audio.o
CC hw/usb/dev-serial.o
CC hw/usb/dev-network.o
CC hw/usb/dev-bluetooth.o
CC hw/usb/dev-smartcard-reader.o
CC hw/usb/ccid-card-passthru.o
CC hw/usb/ccid-card-emulated.o
CC hw/usb/dev-mtp.o
CC hw/usb/host-stub.o
CC hw/virtio/virtio-rng.o
CC hw/virtio/virtio-pci.o
CC hw/virtio/virtio-bus.o
CC hw/virtio/virtio-mmio.o
CC hw/virtio/vhost-pci-slave.o
CC hw/virtio/vhost-stub.o
CC hw/watchdog/watchdog.o
CC hw/watchdog/wdt_i6300esb.o
CC hw/watchdog/wdt_ib700.o
CC hw/watchdog/wdt_aspeed.o
CC hw/xen/xen_backend.o
CC hw/xen/xen_devconfig.o
CC hw/xen/xen_pvdev.o
CC hw/xen/xen-common.o
CC migration/migration.o
CC migration/socket.o
CC migration/fd.o
CC migration/exec.o
CC migration/tls.o
CC migration/channel.o
CC migration/savevm.o
CC migration/colo-comm.o
CC migration/colo.o
CC migration/colo-failover.o
CC migration/vmstate.o
CC migration/vmstate-types.o
CC migration/page_cache.o
CC migration/qemu-file.o
CC migration/global_state.o
CC migration/qemu-file-channel.o
CC migration/xbzrle.o
CC migration/postcopy-ram.o
CC migration/qjson.o
CC migration/rdma.o
CC migration/block.o
CC net/net.o
CC net/queue.o
CC net/checksum.o
CC net/util.o
CC net/hub.o
CC net/socket.o
CC net/dump.o
CC net/eth.o
CC net/l2tpv3.o
CC net/vhost-user.o
CC net/slirp.o
CC net/filter.o
CC net/filter-buffer.o
CC net/filter-mirror.o
CC net/colo-compare.o
CC net/colo.o
CC net/filter-rewriter.o
CC net/filter-replay.o
CC net/tap.o
CC net/tap-linux.o
CC qom/cpu.o
CC replay/replay.o
CC replay/replay-internal.o
/tmp/qemu-test/src/replay/replay-internal.c: In function 'replay_put_array':
/tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
CC replay/replay-events.o
CC replay/replay-time.o
CC replay/replay-input.o
CC replay/replay-char.o
CC replay/replay-snapshot.o
CC replay/replay-net.o
CC replay/replay-audio.o
CC slirp/cksum.o
CC slirp/if.o
CC slirp/ip_icmp.o
CC slirp/ip6_icmp.o
CC slirp/ip6_input.o
CC slirp/ip6_output.o
CC slirp/ip_input.o
CC slirp/ip_output.o
CC slirp/dnssearch.o
CC slirp/dhcpv6.o
CC slirp/slirp.o
CC slirp/mbuf.o
CC slirp/misc.o
CC slirp/sbuf.o
CC slirp/socket.o
CC slirp/tcp_input.o
CC slirp/tcp_output.o
CC slirp/tcp_subr.o
CC slirp/tcp_timer.o
CC slirp/udp.o
CC slirp/udp6.o
/tmp/qemu-test/src/slirp/tcp_input.c: In function 'tcp_input':
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_p' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_len' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_tos' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_id' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_off' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_ttl' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_sum' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_src.s_addr' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_dst.s_addr' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:220: warning: 'save_ip6.ip_nh' may be used uninitialized in this function
CC slirp/bootp.o
CC slirp/tftp.o
CC slirp/arp_table.o
CC slirp/ndp_table.o
CC slirp/ncsi.o
CC ui/keymaps.o
CC ui/console.o
CC ui/qemu-pixman.o
CC ui/cursor.o
CC ui/input.o
CC ui/input-keymap.o
CC ui/input-legacy.o
CC ui/input-linux.o
CC ui/spice-core.o
CC ui/spice-input.o
CC ui/spice-display.o
CC ui/sdl.o
CC ui/sdl_zoom.o
CC ui/x_keymap.o
CC ui/curses.o
CC ui/vnc-enc-zlib.o
CC ui/vnc.o
CC ui/vnc-enc-hextile.o
CC ui/vnc-enc-tight.o
CC ui/vnc-palette.o
CC ui/vnc-enc-zrle.o
CC ui/vnc-auth-vencrypt.o
CC ui/vnc-ws.o
CC ui/vnc-jobs.o
CC ui/gtk.o
VERT ui/shader/texture-blit-vert.h
VERT ui/shader/texture-blit-flip-vert.h
FRAG ui/shader/texture-blit-frag.h
CC ui/console-gl.o
CC ui/egl-helpers.o
CC ui/egl-context.o
CC ui/gtk-egl.o
CC chardev/char.o
CC chardev/char-fe.o
CC chardev/char-fd.o
CC chardev/char-file.o
In file included from /usr/include/gtk-2.0/gtk/gtk.h:235,
from /tmp/qemu-test/src/include/ui/gtk.h:10,
from /tmp/qemu-test/src/ui/gtk-egl.c:21:
/usr/include/gtk-2.0/gtk/gtkitemfactory.h:47: warning: function declaration isn't a prototype
In file included from /usr/include/gtk-2.0/gtk/gtk.h:235,
from /tmp/qemu-test/src/include/ui/gtk.h:10,
from /tmp/qemu-test/src/ui/gtk.c:43:
/usr/include/gtk-2.0/gtk/gtkitemfactory.h:47: warning: function declaration isn't a prototype
CC chardev/char-io.o
CC chardev/char-mux.o
CC chardev/char-null.o
CC chardev/char-parallel.o
CC chardev/char-pipe.o
CC chardev/char-pty.o
CC chardev/char-ringbuf.o
CC chardev/char-serial.o
CC chardev/char-stdio.o
CC chardev/char-socket.o
CC chardev/char-udp.o
LINK tests/qemu-iotests/socket_scm_helper
CC qga/commands.o
CC qga/guest-agent-command-state.o
CC qga/main.o
CC qga/commands-posix.o
CC qga/channel-posix.o
CC qga/qapi-generated/qga-qapi-types.o
CC qga/qapi-generated/qga-qapi-visit.o
CC qga/qapi-generated/qga-qmp-marshal.o
CC qemu-img.o
AR libqemuutil.a
CC ui/shader.o
AS optionrom/linuxboot.o
CC optionrom/linuxboot_dma.o
AS optionrom/multiboot.o
AS optionrom/kvmvapic.o
cc: unrecognized option '-no-integrated-as'
cc: unrecognized option '-no-integrated-as'
BUILD optionrom/linuxboot_dma.img
BUILD optionrom/linuxboot.img
BUILD optionrom/multiboot.img
BUILD optionrom/linuxboot_dma.raw
BUILD optionrom/kvmvapic.img
SIGN optionrom/linuxboot_dma.bin
BUILD optionrom/multiboot.raw
BUILD optionrom/linuxboot.raw
BUILD optionrom/kvmvapic.raw
SIGN optionrom/multiboot.bin
SIGN optionrom/linuxboot.bin
SIGN optionrom/kvmvapic.bin
LINK qemu-ga
LINK ivshmem-client
LINK ivshmem-server
LINK qemu-nbd
LINK qemu-img
LINK qemu-io
LINK scsi/qemu-pr-helper
LINK qemu-bridge-helper
GEN x86_64-softmmu/config-target.h
GEN x86_64-softmmu/hmp-commands.h
GEN x86_64-softmmu/hmp-commands-info.h
CC x86_64-softmmu/exec.o
CC x86_64-softmmu/tcg/optimize.o
CC x86_64-softmmu/tcg/tcg-op.o
CC x86_64-softmmu/tcg/tcg-common.o
CC x86_64-softmmu/tcg/tcg.o
CC x86_64-softmmu/fpu/softfloat.o
CC x86_64-softmmu/disas.o
GEN x86_64-softmmu/gdbstub-xml.c
CC x86_64-softmmu/cpus.o
CC x86_64-softmmu/arch_init.o
CC x86_64-softmmu/monitor.o
CC x86_64-softmmu/gdbstub.o
CC x86_64-softmmu/balloon.o
GEN aarch64-softmmu/hmp-commands.h
CC x86_64-softmmu/ioport.o
CC x86_64-softmmu/numa.o
CC x86_64-softmmu/qtest.o
CC x86_64-softmmu/memory.o
CC x86_64-softmmu/memory_mapping.o
GEN aarch64-softmmu/hmp-commands-info.h
CC x86_64-softmmu/dump.o
CC x86_64-softmmu/migration/ram.o
GEN aarch64-softmmu/config-target.h
CC x86_64-softmmu/accel/accel.o
CC x86_64-softmmu/accel/stubs/hax-stub.o
CC x86_64-softmmu/accel/tcg/tcg-all.o
CC x86_64-softmmu/accel/kvm/kvm-all.o
CC x86_64-softmmu/accel/tcg/cputlb.o
CC aarch64-softmmu/exec.o
CC x86_64-softmmu/accel/tcg/tcg-runtime.o
CC x86_64-softmmu/accel/tcg/cpu-exec.o
CC x86_64-softmmu/accel/tcg/cpu-exec-common.o
CC aarch64-softmmu/tcg/tcg.o
CC aarch64-softmmu/tcg/tcg-op.o
CC x86_64-softmmu/accel/tcg/translate-all.o
CC x86_64-softmmu/accel/tcg/translator.o
CC x86_64-softmmu/hw/block/virtio-blk.o
CC x86_64-softmmu/hw/block/dataplane/virtio-blk.o
CC aarch64-softmmu/tcg/optimize.o
CC x86_64-softmmu/hw/char/virtio-serial-bus.o
CC aarch64-softmmu/tcg/tcg-common.o
CC aarch64-softmmu/fpu/softfloat.o
CC aarch64-softmmu/disas.o
GEN aarch64-softmmu/gdbstub-xml.c
CC x86_64-softmmu/hw/core/generic-loader.o
CC x86_64-softmmu/hw/core/null-machine.o
CC x86_64-softmmu/hw/display/vga.o
CC aarch64-softmmu/arch_init.o
CC x86_64-softmmu/hw/display/virtio-gpu.o
CC aarch64-softmmu/cpus.o
CC aarch64-softmmu/monitor.o
CC x86_64-softmmu/hw/display/virtio-gpu-3d.o
CC x86_64-softmmu/hw/display/virtio-gpu-pci.o
CC aarch64-softmmu/gdbstub.o
CC aarch64-softmmu/balloon.o
CC aarch64-softmmu/ioport.o
CC aarch64-softmmu/numa.o
CC x86_64-softmmu/hw/display/virtio-vga.o
CC aarch64-softmmu/qtest.o
CC x86_64-softmmu/hw/intc/apic.o
CC x86_64-softmmu/hw/intc/apic_common.o
CC aarch64-softmmu/memory.o
CC x86_64-softmmu/hw/intc/ioapic.o
CC x86_64-softmmu/hw/isa/lpc_ich9.o
CC x86_64-softmmu/hw/misc/vmport.o
CC aarch64-softmmu/memory_mapping.o
CC x86_64-softmmu/hw/misc/ivshmem.o
CC aarch64-softmmu/dump.o
CC aarch64-softmmu/migration/ram.o
CC aarch64-softmmu/accel/accel.o
CC aarch64-softmmu/accel/stubs/hax-stub.o
CC x86_64-softmmu/hw/misc/pvpanic.o
CC x86_64-softmmu/hw/misc/hyperv_testdev.o
CC aarch64-softmmu/accel/stubs/kvm-stub.o
CC aarch64-softmmu/accel/tcg/tcg-all.o
CC aarch64-softmmu/accel/tcg/cputlb.o
CC x86_64-softmmu/hw/misc/mmio_interface.o
CC x86_64-softmmu/hw/net/virtio-net.o
CC x86_64-softmmu/hw/net/vhost_pci_net.o
CC x86_64-softmmu/hw/net/vhost_net.o
CC x86_64-softmmu/hw/scsi/virtio-scsi.o
CC x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
CC x86_64-softmmu/hw/scsi/vhost-scsi-common.o
CC aarch64-softmmu/accel/tcg/tcg-runtime.o
CC aarch64-softmmu/accel/tcg/cpu-exec.o
CC x86_64-softmmu/hw/scsi/vhost-scsi.o
CC aarch64-softmmu/accel/tcg/cpu-exec-common.o
CC aarch64-softmmu/accel/tcg/translate-all.o
CC aarch64-softmmu/accel/tcg/translator.o
CC aarch64-softmmu/hw/adc/stm32f2xx_adc.o
CC aarch64-softmmu/hw/block/virtio-blk.o
CC aarch64-softmmu/hw/block/dataplane/virtio-blk.o
CC aarch64-softmmu/hw/char/exynos4210_uart.o
CC aarch64-softmmu/hw/char/omap_uart.o
CC aarch64-softmmu/hw/char/digic-uart.o
CC x86_64-softmmu/hw/scsi/vhost-user-scsi.o
CC x86_64-softmmu/hw/timer/mc146818rtc.o
CC x86_64-softmmu/hw/vfio/common.o
CC x86_64-softmmu/hw/vfio/pci.o
CC aarch64-softmmu/hw/char/stm32f2xx_usart.o
CC x86_64-softmmu/hw/vfio/pci-quirks.o
CC x86_64-softmmu/hw/vfio/platform.o
CC aarch64-softmmu/hw/char/bcm2835_aux.o
CC x86_64-softmmu/hw/vfio/spapr.o
CC aarch64-softmmu/hw/char/virtio-serial-bus.o
CC aarch64-softmmu/hw/core/generic-loader.o
CC x86_64-softmmu/hw/virtio/virtio.o
CC x86_64-softmmu/hw/virtio/virtio-balloon.o
CC aarch64-softmmu/hw/core/null-machine.o
CC aarch64-softmmu/hw/cpu/arm11mpcore.o
CC x86_64-softmmu/hw/virtio/vhost.o
CC aarch64-softmmu/hw/cpu/realview_mpcore.o
CC aarch64-softmmu/hw/cpu/a9mpcore.o
CC aarch64-softmmu/hw/display/omap_dss.o
CC aarch64-softmmu/hw/cpu/a15mpcore.o
CC x86_64-softmmu/hw/virtio/vhost-backend.o
CC x86_64-softmmu/hw/virtio/vhost-user.o
CC aarch64-softmmu/hw/display/omap_lcdc.o
CC aarch64-softmmu/hw/display/pxa2xx_lcd.o
CC x86_64-softmmu/hw/virtio/vhost-vsock.o
CC aarch64-softmmu/hw/display/bcm2835_fb.o
CC aarch64-softmmu/hw/display/vga.o
CC x86_64-softmmu/hw/virtio/virtio-crypto.o
CC aarch64-softmmu/hw/display/virtio-gpu.o
CC aarch64-softmmu/hw/display/virtio-gpu-3d.o
CC aarch64-softmmu/hw/display/virtio-gpu-pci.o
CC aarch64-softmmu/hw/display/dpcd.o
CC x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
CC aarch64-softmmu/hw/display/xlnx_dp.o
CC x86_64-softmmu/hw/xen/xen-host-pci-device.o
CC aarch64-softmmu/hw/dma/xlnx_dpdma.o
CC aarch64-softmmu/hw/dma/omap_dma.o
CC aarch64-softmmu/hw/dma/soc_dma.o
CC aarch64-softmmu/hw/dma/pxa2xx_dma.o
CC aarch64-softmmu/hw/dma/bcm2835_dma.o
CC aarch64-softmmu/hw/gpio/omap_gpio.o
CC x86_64-softmmu/hw/xen/xen_pt.o
CC aarch64-softmmu/hw/gpio/imx_gpio.o
CC aarch64-softmmu/hw/gpio/bcm2835_gpio.o
CC x86_64-softmmu/hw/xen/xen_pt_config_init.o
CC aarch64-softmmu/hw/i2c/omap_i2c.o
CC x86_64-softmmu/hw/xen/xen_pt_graphics.o
CC aarch64-softmmu/hw/input/pxa2xx_keypad.o
CC aarch64-softmmu/hw/input/tsc210x.o
CC x86_64-softmmu/hw/xen/xen_pt_msi.o
CC x86_64-softmmu/hw/xen/xen_pt_load_rom.o
CC aarch64-softmmu/hw/intc/armv7m_nvic.o
CC x86_64-softmmu/hw/i386/multiboot.o
CC x86_64-softmmu/hw/i386/pc.o
CC x86_64-softmmu/hw/i386/pc_piix.o
CC aarch64-softmmu/hw/intc/exynos4210_gic.o
CC aarch64-softmmu/hw/intc/exynos4210_combiner.o
CC aarch64-softmmu/hw/intc/omap_intc.o
CC x86_64-softmmu/hw/i386/pc_q35.o
CC aarch64-softmmu/hw/intc/bcm2835_ic.o
CC x86_64-softmmu/hw/i386/pc_sysfw.o
CC aarch64-softmmu/hw/intc/bcm2836_control.o
CC aarch64-softmmu/hw/intc/allwinner-a10-pic.o
CC aarch64-softmmu/hw/intc/aspeed_vic.o
CC x86_64-softmmu/hw/i386/x86-iommu.o
CC x86_64-softmmu/hw/i386/intel_iommu.o
CC aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
CC x86_64-softmmu/hw/i386/amd_iommu.o
CC aarch64-softmmu/hw/misc/ivshmem.o
CC aarch64-softmmu/hw/misc/arm_sysctl.o
CC x86_64-softmmu/hw/i386/kvmvapic.o
CC aarch64-softmmu/hw/misc/cbus.o
CC aarch64-softmmu/hw/misc/exynos4210_pmu.o
CC aarch64-softmmu/hw/misc/exynos4210_clk.o
CC aarch64-softmmu/hw/misc/exynos4210_rng.o
CC aarch64-softmmu/hw/misc/imx_ccm.o
/tmp/qemu-test/src/hw/i386/pc_piix.c: In function 'igd_passthrough_isa_bridge_create':
/tmp/qemu-test/src/hw/i386/pc_piix.c:1073: warning: 'pch_rev_id' may be used uninitialized in this function
CC x86_64-softmmu/hw/i386/acpi-build.o
CC aarch64-softmmu/hw/misc/imx31_ccm.o
CC aarch64-softmmu/hw/misc/imx25_ccm.o
CC x86_64-softmmu/hw/i386/../xenpv/xen_machine_pv.o
CC x86_64-softmmu/hw/i386/kvm/clock.o
CC x86_64-softmmu/hw/i386/kvm/apic.o
CC x86_64-softmmu/hw/i386/kvm/i8259.o
CC aarch64-softmmu/hw/misc/imx6_ccm.o
CC x86_64-softmmu/hw/i386/kvm/ioapic.o
CC aarch64-softmmu/hw/misc/imx6_src.o
CC aarch64-softmmu/hw/misc/mst_fpga.o
CC aarch64-softmmu/hw/misc/omap_clk.o
CC aarch64-softmmu/hw/misc/omap_gpmc.o
CC aarch64-softmmu/hw/misc/omap_l4.o
CC x86_64-softmmu/hw/i386/kvm/i8254.o
CC aarch64-softmmu/hw/misc/omap_sdrc.o
CC x86_64-softmmu/hw/i386/xen/xen_platform.o
CC aarch64-softmmu/hw/misc/omap_tap.o
CC aarch64-softmmu/hw/misc/bcm2835_mbox.o
CC x86_64-softmmu/hw/i386/xen/xen_apic.o
CC x86_64-softmmu/hw/i386/xen/xen_pvdevice.o
CC aarch64-softmmu/hw/misc/bcm2835_property.o
CC x86_64-softmmu/hw/i386/xen/xen-hvm.o
CC aarch64-softmmu/hw/misc/bcm2835_rng.o
CC aarch64-softmmu/hw/misc/zynq_slcr.o
CC aarch64-softmmu/hw/misc/zynq-xadc.o
CC aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
CC x86_64-softmmu/hw/i386/xen/xen-mapcache.o
CC aarch64-softmmu/hw/misc/mps2-scc.o
CC x86_64-softmmu/target/i386/helper.o
CC aarch64-softmmu/hw/misc/auxbus.o
CC x86_64-softmmu/target/i386/cpu.o
CC x86_64-softmmu/target/i386/gdbstub.o
CC aarch64-softmmu/hw/misc/aspeed_scu.o
CC x86_64-softmmu/target/i386/xsave_helper.o
CC x86_64-softmmu/target/i386/translate.o
CC aarch64-softmmu/hw/misc/aspeed_sdmc.o
CC x86_64-softmmu/target/i386/bpt_helper.o
CC aarch64-softmmu/hw/misc/mmio_interface.o
CC x86_64-softmmu/target/i386/cc_helper.o
CC aarch64-softmmu/hw/misc/msf2-sysreg.o
CC x86_64-softmmu/target/i386/excp_helper.o
CC aarch64-softmmu/hw/net/virtio-net.o
/tmp/qemu-test/src/hw/i386/acpi-build.c: In function 'build_append_pci_bus_devices':
/tmp/qemu-test/src/hw/i386/acpi-build.c:509: warning: 'notify_method' may be used uninitialized in this function
CC x86_64-softmmu/target/i386/fpu_helper.o
CC aarch64-softmmu/hw/net/vhost_pci_net.o
CC aarch64-softmmu/hw/net/vhost_net.o
CC x86_64-softmmu/target/i386/int_helper.o
CC x86_64-softmmu/target/i386/mem_helper.o
CC aarch64-softmmu/hw/pcmcia/pxa2xx.o
CC x86_64-softmmu/target/i386/misc_helper.o
CC aarch64-softmmu/hw/scsi/virtio-scsi.o
CC aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
CC x86_64-softmmu/target/i386/mpx_helper.o
CC x86_64-softmmu/target/i386/seg_helper.o
CC x86_64-softmmu/target/i386/smm_helper.o
CC x86_64-softmmu/target/i386/svm_helper.o
CC aarch64-softmmu/hw/scsi/vhost-scsi-common.o
CC aarch64-softmmu/hw/scsi/vhost-scsi.o
CC x86_64-softmmu/target/i386/machine.o
CC aarch64-softmmu/hw/scsi/vhost-user-scsi.o
CC aarch64-softmmu/hw/sd/omap_mmc.o
CC aarch64-softmmu/hw/sd/pxa2xx_mmci.o
CC aarch64-softmmu/hw/sd/bcm2835_sdhost.o
CC aarch64-softmmu/hw/ssi/omap_spi.o
CC aarch64-softmmu/hw/ssi/imx_spi.o
CC aarch64-softmmu/hw/timer/exynos4210_mct.o
CC aarch64-softmmu/hw/timer/exynos4210_pwm.o
CC aarch64-softmmu/hw/timer/exynos4210_rtc.o
CC aarch64-softmmu/hw/timer/omap_gptimer.o
CC aarch64-softmmu/hw/timer/omap_synctimer.o
CC aarch64-softmmu/hw/timer/pxa2xx_timer.o
CC aarch64-softmmu/hw/timer/digic-timer.o
CC aarch64-softmmu/hw/timer/allwinner-a10-pit.o
CC aarch64-softmmu/hw/usb/tusb6010.o
CC x86_64-softmmu/target/i386/arch_memory_mapping.o
CC x86_64-softmmu/target/i386/arch_dump.o
CC x86_64-softmmu/target/i386/monitor.o
CC x86_64-softmmu/target/i386/kvm.o
CC aarch64-softmmu/hw/vfio/common.o
CC aarch64-softmmu/hw/vfio/pci.o
CC x86_64-softmmu/target/i386/hyperv.o
CC aarch64-softmmu/hw/vfio/pci-quirks.o
CC aarch64-softmmu/hw/vfio/platform.o
CC aarch64-softmmu/hw/vfio/calxeda-xgmac.o
CC aarch64-softmmu/hw/vfio/amd-xgbe.o
CC aarch64-softmmu/hw/vfio/spapr.o
GEN trace/generated-helpers.c
CC aarch64-softmmu/hw/virtio/virtio.o
CC aarch64-softmmu/hw/virtio/virtio-balloon.o
CC aarch64-softmmu/hw/virtio/vhost.o
CC aarch64-softmmu/hw/virtio/vhost-backend.o
CC x86_64-softmmu/trace/control-target.o
CC aarch64-softmmu/hw/virtio/vhost-user.o
CC x86_64-softmmu/gdbstub-xml.o
CC aarch64-softmmu/hw/virtio/vhost-vsock.o
CC aarch64-softmmu/hw/virtio/virtio-crypto.o
CC aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
CC aarch64-softmmu/hw/arm/boot.o
CC aarch64-softmmu/hw/arm/collie.o
CC x86_64-softmmu/trace/generated-helpers.o
CC aarch64-softmmu/hw/arm/exynos4_boards.o
CC aarch64-softmmu/hw/arm/gumstix.o
CC aarch64-softmmu/hw/arm/highbank.o
CC aarch64-softmmu/hw/arm/digic_boards.o
CC aarch64-softmmu/hw/arm/integratorcp.o
CC aarch64-softmmu/hw/arm/mainstone.o
CC aarch64-softmmu/hw/arm/musicpal.o
CC aarch64-softmmu/hw/arm/nseries.o
CC aarch64-softmmu/hw/arm/omap_sx1.o
CC aarch64-softmmu/hw/arm/palm.o
CC aarch64-softmmu/hw/arm/realview.o
CC aarch64-softmmu/hw/arm/spitz.o
CC aarch64-softmmu/hw/arm/stellaris.o
CC aarch64-softmmu/hw/arm/tosa.o
CC aarch64-softmmu/hw/arm/vexpress.o
CC aarch64-softmmu/hw/arm/versatilepb.o
CC aarch64-softmmu/hw/arm/virt.o
CC aarch64-softmmu/hw/arm/xilinx_zynq.o
CC aarch64-softmmu/hw/arm/z2.o
CC aarch64-softmmu/hw/arm/virt-acpi-build.o
CC aarch64-softmmu/hw/arm/netduino2.o
CC aarch64-softmmu/hw/arm/sysbus-fdt.o
CC aarch64-softmmu/hw/arm/armv7m.o
LINK x86_64-softmmu/qemu-system-x86_64
CC aarch64-softmmu/hw/arm/exynos4210.o
CC aarch64-softmmu/hw/arm/pxa2xx.o
CC aarch64-softmmu/hw/arm/pxa2xx_gpio.o
CC aarch64-softmmu/hw/arm/pxa2xx_pic.o
CC aarch64-softmmu/hw/arm/digic.o
CC aarch64-softmmu/hw/arm/omap1.o
CC aarch64-softmmu/hw/arm/omap2.o
CC aarch64-softmmu/hw/arm/strongarm.o
CC aarch64-softmmu/hw/arm/allwinner-a10.o
CC aarch64-softmmu/hw/arm/cubieboard.o
CC aarch64-softmmu/hw/arm/bcm2835_peripherals.o
CC aarch64-softmmu/hw/arm/bcm2836.o
CC aarch64-softmmu/hw/arm/raspi.o
CC aarch64-softmmu/hw/arm/stm32f205_soc.o
CC aarch64-softmmu/hw/arm/xlnx-zynqmp.o
CC aarch64-softmmu/hw/arm/xlnx-zcu102.o
CC aarch64-softmmu/hw/arm/fsl-imx25.o
CC aarch64-softmmu/hw/arm/imx25_pdk.o
CC aarch64-softmmu/hw/arm/fsl-imx31.o
CC aarch64-softmmu/hw/arm/kzm.o
CC aarch64-softmmu/hw/arm/fsl-imx6.o
CC aarch64-softmmu/hw/arm/sabrelite.o
CC aarch64-softmmu/hw/arm/aspeed_soc.o
CC aarch64-softmmu/hw/arm/aspeed.o
CC aarch64-softmmu/hw/arm/mps2.o
CC aarch64-softmmu/hw/arm/msf2-soc.o
CC aarch64-softmmu/hw/arm/msf2-som.o
CC aarch64-softmmu/target/arm/arm-semi.o
CC aarch64-softmmu/target/arm/machine.o
CC aarch64-softmmu/target/arm/psci.o
CC aarch64-softmmu/target/arm/arch_dump.o
CC aarch64-softmmu/target/arm/monitor.o
CC aarch64-softmmu/target/arm/translate.o
CC aarch64-softmmu/target/arm/kvm-stub.o
CC aarch64-softmmu/target/arm/op_helper.o
CC aarch64-softmmu/target/arm/helper.o
CC aarch64-softmmu/target/arm/cpu.o
CC aarch64-softmmu/target/arm/neon_helper.o
CC aarch64-softmmu/target/arm/iwmmxt_helper.o
CC aarch64-softmmu/target/arm/gdbstub.o
CC aarch64-softmmu/target/arm/cpu64.o
CC aarch64-softmmu/target/arm/translate-a64.o
CC aarch64-softmmu/target/arm/helper-a64.o
CC aarch64-softmmu/target/arm/gdbstub64.o
CC aarch64-softmmu/target/arm/crypto_helper.o
CC aarch64-softmmu/target/arm/arm-powerctl.o
GEN trace/generated-helpers.c
CC aarch64-softmmu/trace/control-target.o
CC aarch64-softmmu/gdbstub-xml.o
CC aarch64-softmmu/trace/generated-helpers.o
/tmp/qemu-test/src/target/arm/translate-a64.c: In function 'handle_shri_with_rndacc':
/tmp/qemu-test/src/target/arm/translate-a64.c:6392: warning: 'tcg_src_hi' may be used uninitialized in this function
/tmp/qemu-test/src/target/arm/translate-a64.c: In function 'disas_simd_scalar_two_reg_misc':
/tmp/qemu-test/src/target/arm/translate-a64.c:8119: warning: 'rmode' may be used uninitialized in this function
LINK aarch64-softmmu/qemu-system-aarch64
mkdir -p dtc/libfdt
mkdir -p dtc/tests
TEST tests/qapi-schema/alternate-any.out
TEST tests/qapi-schema/alternate-clash.out
TEST tests/qapi-schema/alternate-base.out
TEST tests/qapi-schema/alternate-conflict-enum-bool.out
TEST tests/qapi-schema/alternate-conflict-dict.out
TEST tests/qapi-schema/alternate-conflict-enum-int.out
TEST tests/qapi-schema/alternate-array.out
TEST tests/qapi-schema/alternate-conflict-string.out
TEST tests/qapi-schema/alternate-conflict-bool-string.out
TEST tests/qapi-schema/alternate-empty.out
TEST tests/qapi-schema/alternate-conflict-num-string.out
TEST tests/qapi-schema/alternate-nested.out
TEST tests/qapi-schema/alternate-unknown.out
TEST tests/qapi-schema/args-alternate.out
TEST tests/qapi-schema/args-any.out
TEST tests/qapi-schema/args-array-empty.out
TEST tests/qapi-schema/args-bad-boxed.out
TEST tests/qapi-schema/args-array-unknown.out
TEST tests/qapi-schema/args-boxed-anon.out
TEST tests/qapi-schema/args-boxed-empty.out
TEST tests/qapi-schema/args-boxed-string.out
TEST tests/qapi-schema/args-int.out
TEST tests/qapi-schema/args-invalid.out
TEST tests/qapi-schema/args-member-array-bad.out
TEST tests/qapi-schema/args-member-case.out
TEST tests/qapi-schema/args-member-unknown.out
TEST tests/qapi-schema/args-name-clash.out
TEST tests/qapi-schema/args-union.out
TEST tests/qapi-schema/args-unknown.out
TEST tests/qapi-schema/bad-base.out
TEST tests/qapi-schema/bad-data.out
TEST tests/qapi-schema/bad-ident.out
TEST tests/qapi-schema/bad-type-bool.out
TEST tests/qapi-schema/bad-type-dict.out
TEST tests/qapi-schema/bad-type-int.out
TEST tests/qapi-schema/base-cycle-direct.out
TEST tests/qapi-schema/base-cycle-indirect.out
TEST tests/qapi-schema/command-int.out
TEST tests/qapi-schema/comments.out
TEST tests/qapi-schema/doc-bad-alternate-member.out
TEST tests/qapi-schema/doc-bad-command-arg.out
TEST tests/qapi-schema/doc-bad-symbol.out
TEST tests/qapi-schema/doc-bad-union-member.out
TEST tests/qapi-schema/doc-before-include.out
TEST tests/qapi-schema/doc-before-pragma.out
TEST tests/qapi-schema/doc-duplicated-arg.out
TEST tests/qapi-schema/doc-duplicated-return.out
TEST tests/qapi-schema/doc-duplicated-since.out
TEST tests/qapi-schema/doc-empty-arg.out
TEST tests/qapi-schema/doc-empty-symbol.out
TEST tests/qapi-schema/doc-good.out
TEST tests/qapi-schema/doc-empty-section.out
TEST tests/qapi-schema/doc-interleaved-section.out
TEST tests/qapi-schema/doc-invalid-end.out
TEST tests/qapi-schema/doc-invalid-end2.out
TEST tests/qapi-schema/doc-invalid-return.out
TEST tests/qapi-schema/doc-invalid-start.out
TEST tests/qapi-schema/doc-invalid-section.out
TEST tests/qapi-schema/doc-missing-colon.out
TEST tests/qapi-schema/doc-missing.out
TEST tests/qapi-schema/doc-missing-expr.out
TEST tests/qapi-schema/doc-missing-space.out
TEST tests/qapi-schema/doc-no-symbol.out
TEST tests/qapi-schema/double-data.out
TEST tests/qapi-schema/double-type.out
TEST tests/qapi-schema/duplicate-key.out
TEST tests/qapi-schema/empty.out
TEST tests/qapi-schema/enum-bad-name.out
TEST tests/qapi-schema/enum-bad-prefix.out
TEST tests/qapi-schema/enum-clash-member.out
TEST tests/qapi-schema/enum-dict-member.out
TEST tests/qapi-schema/enum-int-member.out
TEST tests/qapi-schema/enum-missing-data.out
TEST tests/qapi-schema/enum-wrong-data.out
TEST tests/qapi-schema/enum-member-case.out
TEST tests/qapi-schema/escape-outside-string.out
TEST tests/qapi-schema/escape-too-big.out
TEST tests/qapi-schema/escape-too-short.out
TEST tests/qapi-schema/event-boxed-empty.out
TEST tests/qapi-schema/event-case.out
TEST tests/qapi-schema/event-nest-struct.out
TEST tests/qapi-schema/flat-union-array-branch.out
TEST tests/qapi-schema/flat-union-bad-base.out
TEST tests/qapi-schema/flat-union-bad-discriminator.out
TEST tests/qapi-schema/flat-union-base-any.out
TEST tests/qapi-schema/flat-union-base-union.out
TEST tests/qapi-schema/flat-union-clash-member.out
TEST tests/qapi-schema/flat-union-empty.out
TEST tests/qapi-schema/flat-union-incomplete-branch.out
TEST tests/qapi-schema/flat-union-inline.out
TEST tests/qapi-schema/flat-union-int-branch.out
TEST tests/qapi-schema/flat-union-invalid-branch-key.out
TEST tests/qapi-schema/flat-union-no-base.out
TEST tests/qapi-schema/flat-union-invalid-discriminator.out
TEST tests/qapi-schema/flat-union-optional-discriminator.out
TEST tests/qapi-schema/funny-char.out
TEST tests/qapi-schema/flat-union-string-discriminator.out
TEST tests/qapi-schema/ident-with-escape.out
TEST tests/qapi-schema/include-before-err.out
TEST tests/qapi-schema/include-cycle.out
TEST tests/qapi-schema/include-extra-junk.out
TEST tests/qapi-schema/include-format-err.out
TEST tests/qapi-schema/include-nested-err.out
TEST tests/qapi-schema/include-no-file.out
TEST tests/qapi-schema/include-non-file.out
TEST tests/qapi-schema/include-relpath.out
TEST tests/qapi-schema/include-repetition.out
TEST tests/qapi-schema/include-self-cycle.out
TEST tests/qapi-schema/include-simple.out
TEST tests/qapi-schema/indented-expr.out
TEST tests/qapi-schema/leading-comma-list.out
TEST tests/qapi-schema/leading-comma-object.out
TEST tests/qapi-schema/missing-colon.out
TEST tests/qapi-schema/missing-comma-list.out
TEST tests/qapi-schema/missing-comma-object.out
TEST tests/qapi-schema/missing-type.out
TEST tests/qapi-schema/nested-struct-data.out
TEST tests/qapi-schema/non-objects.out
TEST tests/qapi-schema/pragma-doc-required-crap.out
TEST tests/qapi-schema/pragma-extra-junk.out
TEST tests/qapi-schema/pragma-name-case-whitelist-crap.out
TEST tests/qapi-schema/pragma-non-dict.out
TEST tests/qapi-schema/qapi-schema-test.out
TEST tests/qapi-schema/pragma-returns-whitelist-crap.out
TEST tests/qapi-schema/quoted-structural-chars.out
TEST tests/qapi-schema/redefined-builtin.out
TEST tests/qapi-schema/redefined-command.out
TEST tests/qapi-schema/redefined-event.out
TEST tests/qapi-schema/redefined-type.out
TEST tests/qapi-schema/reserved-command-q.out
TEST tests/qapi-schema/reserved-enum-q.out
TEST tests/qapi-schema/reserved-member-has.out
TEST tests/qapi-schema/reserved-member-q.out
TEST tests/qapi-schema/reserved-member-u.out
TEST tests/qapi-schema/reserved-member-underscore.out
TEST tests/qapi-schema/reserved-type-kind.out
TEST tests/qapi-schema/reserved-type-list.out
TEST tests/qapi-schema/returns-alternate.out
TEST tests/qapi-schema/returns-array-bad.out
TEST tests/qapi-schema/returns-dict.out
TEST tests/qapi-schema/returns-unknown.out
TEST tests/qapi-schema/returns-whitelist.out
TEST tests/qapi-schema/struct-base-clash-deep.out
TEST tests/qapi-schema/struct-base-clash.out
TEST tests/qapi-schema/struct-data-invalid.out
TEST tests/qapi-schema/struct-member-invalid.out
TEST tests/qapi-schema/trailing-comma-list.out
TEST tests/qapi-schema/trailing-comma-object.out
TEST tests/qapi-schema/type-bypass-bad-gen.out
TEST tests/qapi-schema/unclosed-list.out
TEST tests/qapi-schema/unclosed-object.out
TEST tests/qapi-schema/unclosed-string.out
TEST tests/qapi-schema/unicode-str.out
TEST tests/qapi-schema/union-base-empty.out
TEST tests/qapi-schema/union-base-no-discriminator.out
TEST tests/qapi-schema/union-branch-case.out
TEST tests/qapi-schema/union-clash-branches.out
TEST tests/qapi-schema/union-empty.out
TEST tests/qapi-schema/union-invalid-base.out
TEST tests/qapi-schema/union-optional-branch.out
TEST tests/qapi-schema/union-unknown.out
TEST tests/qapi-schema/unknown-escape.out
TEST tests/qapi-schema/unknown-expr-key.out
GEN tests/qapi-schema/doc-good.test.texi
CC tests/check-qdict.o
CC tests/test-char.o
CC tests/check-qnum.o
CC tests/check-qstring.o
CC tests/check-qlist.o
CC tests/check-qnull.o
CC tests/check-qobject.o
CC tests/check-qlit.o
CC tests/check-qjson.o
CC tests/test-qobject-output-visitor.o
GEN tests/test-qapi-types.c
GEN tests/test-qapi-visit.c
GEN tests/test-qapi-event.c
GEN tests/test-qmp-introspect.c
CC tests/test-clone-visitor.o
CC tests/test-qobject-input-visitor.o
GEN tests/test-qmp-marshal.c
CC tests/test-qmp-commands.o
CC tests/test-string-input-visitor.o
CC tests/test-string-output-visitor.o
CC tests/test-qmp-event.o
CC tests/test-opts-visitor.o
CC tests/test-coroutine.o
CC tests/iothread.o
CC tests/test-visitor-serialization.o
CC tests/test-iov.o
CC tests/test-aio.o
CC tests/test-aio-multithread.o
CC tests/test-throttle.o
CC tests/test-thread-pool.o
CC tests/test-hbitmap.o
CC tests/test-blockjob.o
CC tests/test-blockjob-txn.o
CC tests/test-x86-cpuid.o
CC tests/test-xbzrle.o
CC tests/test-vmstate.o
CC tests/test-cutils.o
CC tests/test-shift128.o
CC tests/test-mul64.o
CC tests/test-int128.o
CC tests/rcutorture.o
CC tests/test-rcu-list.o
CC tests/test-qdist.o
/tmp/qemu-test/src/tests/test-int128.c:180: warning: '__noclone__' attribute directive ignored
CC tests/test-qht.o
CC tests/test-qht-par.o
CC tests/qht-bench.o
CC tests/test-bitops.o
CC tests/test-bitcnt.o
CC tests/check-qom-proplist.o
CC tests/test-qemu-opts.o
CC tests/check-qom-interface.o
CC tests/test-keyval.o
CC tests/test-crypto-hash.o
CC tests/test-write-threshold.o
CC tests/test-crypto-hmac.o
CC tests/test-crypto-cipher.o
CC tests/test-crypto-secret.o
CC tests/test-qga.o
CC tests/libqtest.o
CC tests/test-timed-average.o
CC tests/test-io-task.o
CC tests/test-io-channel-socket.o
CC tests/io-channel-helpers.o
CC tests/test-io-channel-file.o
CC tests/test-io-channel-command.o
CC tests/test-io-channel-buffer.o
CC tests/test-base64.o
CC tests/test-crypto-ivgen.o
CC tests/test-crypto-afsplit.o
CC tests/test-crypto-xts.o
CC tests/test-crypto-block.o
CC tests/test-replication.o
CC tests/test-logging.o
CC tests/test-bufferiszero.o
CC tests/test-uuid.o
CC tests/ptimer-test.o
CC tests/ptimer-test-stubs.o
CC tests/test-qapi-util.o
CC tests/vhost-user-test.o
CC tests/libqos/pci.o
CC tests/libqos/fw_cfg.o
CC tests/libqos/malloc.o
CC tests/libqos/i2c.o
CC tests/libqos/libqos.o
CC tests/libqos/malloc-spapr.o
CC tests/libqos/libqos-spapr.o
CC tests/libqos/rtas.o
CC tests/libqos/pci-spapr.o
CC tests/libqos/pci-pc.o
CC tests/libqos/malloc-pc.o
CC tests/libqos/libqos-pc.o
CC tests/libqos/ahci.o
CC tests/libqos/virtio.o
CC tests/libqos/virtio-pci.o
CC tests/libqos/malloc-generic.o
CC tests/libqos/virtio-mmio.o
CC tests/endianness-test.o
CC tests/fdc-test.o
CC tests/ide-test.o
CC tests/ahci-test.o
CC tests/hd-geo-test.o
CC tests/boot-order-test.o
CC tests/bios-tables-test.o
CC tests/boot-sector.o
CC tests/boot-serial-test.o
CC tests/acpi-utils.o
CC tests/pxe-test.o
CC tests/rtc-test.o
CC tests/ipmi-kcs-test.o
CC tests/ipmi-bt-test.o
CC tests/i440fx-test.o
CC tests/drive_del-test.o
CC tests/fw_cfg-test.o
CC tests/wdt_ib700-test.o
CC tests/tco-test.o
CC tests/e1000-test.o
CC tests/e1000e-test.o
CC tests/pcnet-test.o
CC tests/rtl8139-test.o
CC tests/eepro100-test.o
CC tests/ne2000-test.o
CC tests/nvme-test.o
CC tests/ac97-test.o
CC tests/es1370-test.o
CC tests/virtio-net-test.o
CC tests/virtio-balloon-test.o
CC tests/virtio-blk-test.o
CC tests/virtio-rng-test.o
CC tests/virtio-scsi-test.o
CC tests/virtio-serial-test.o
CC tests/tpci200-test.o
CC tests/virtio-console-test.o
CC tests/ipoctal232-test.o
CC tests/display-vga-test.o
CC tests/intel-hda-test.o
CC tests/ivshmem-test.o
CC tests/megasas-test.o
CC tests/pvpanic-test.o
CC tests/i82801b11-test.o
CC tests/ioh3420-test.o
CC tests/vmxnet3-test.o
CC tests/usb-hcd-ohci-test.o
CC tests/libqos/usb.o
CC tests/usb-hcd-uhci-test.o
CC tests/usb-hcd-ehci-test.o
CC tests/usb-hcd-xhci-test.o
CC tests/pc-cpu-test.o
CC tests/q35-test.o
CC tests/vmgenid-test.o
CC tests/test-netfilter.o
CC tests/test-filter-mirror.o
CC tests/test-filter-redirector.o
CC tests/migration-test.o
CC tests/test-x86-cpuid-compat.o
CC tests/numa-test.o
CC tests/qmp-test.o
CC tests/device-introspect-test.o
CC tests/qom-test.o
CC tests/test-hmp.o
LINK tests/check-qdict
LINK tests/test-char
LINK tests/check-qnum
LINK tests/check-qstring
LINK tests/check-qlist
LINK tests/check-qnull
LINK tests/check-qobject
LINK tests/check-qjson
CC tests/test-qapi-visit.o
CC tests/test-qapi-event.o
CC tests/test-qapi-types.o
LINK tests/check-qlit
CC tests/test-qmp-introspect.o
CC tests/test-qmp-marshal.o
LINK tests/test-coroutine
LINK tests/test-visitor-serialization
LINK tests/test-iov
LINK tests/test-aio
LINK tests/test-aio-multithread
LINK tests/test-throttle
LINK tests/test-thread-pool
LINK tests/test-hbitmap
LINK tests/test-blockjob
LINK tests/test-blockjob-txn
LINK tests/test-x86-cpuid
LINK tests/test-xbzrle
LINK tests/test-vmstate
LINK tests/test-cutils
LINK tests/test-shift128
LINK tests/test-mul64
LINK tests/test-int128
LINK tests/rcutorture
LINK tests/test-rcu-list
LINK tests/test-qdist
LINK tests/test-qht
LINK tests/qht-bench
LINK tests/test-bitops
LINK tests/test-bitcnt
LINK tests/check-qom-interface
LINK tests/check-qom-proplist
LINK tests/test-qemu-opts
LINK tests/test-keyval
LINK tests/test-write-threshold
LINK tests/test-crypto-hash
LINK tests/test-crypto-hmac
LINK tests/test-crypto-cipher
LINK tests/test-crypto-secret
LINK tests/test-qga
LINK tests/test-timed-average
LINK tests/test-io-task
LINK tests/test-io-channel-socket
LINK tests/test-io-channel-file
LINK tests/test-io-channel-command
LINK tests/test-io-channel-buffer
LINK tests/test-base64
LINK tests/test-crypto-ivgen
LINK tests/test-crypto-afsplit
LINK tests/test-crypto-xts
LINK tests/test-crypto-block
LINK tests/test-logging
LINK tests/test-replication
LINK tests/test-bufferiszero
LINK tests/test-uuid
LINK tests/ptimer-test
LINK tests/test-qapi-util
LINK tests/vhost-user-test
LINK tests/endianness-test
LINK tests/fdc-test
LINK tests/ide-test
LINK tests/ahci-test
LINK tests/hd-geo-test
LINK tests/boot-order-test
LINK tests/bios-tables-test
LINK tests/boot-serial-test
LINK tests/pxe-test
LINK tests/rtc-test
LINK tests/ipmi-kcs-test
LINK tests/ipmi-bt-test
LINK tests/i440fx-test
LINK tests/fw_cfg-test
LINK tests/drive_del-test
LINK tests/wdt_ib700-test
LINK tests/tco-test
LINK tests/e1000-test
LINK tests/e1000e-test
LINK tests/rtl8139-test
LINK tests/pcnet-test
LINK tests/eepro100-test
LINK tests/ne2000-test
LINK tests/nvme-test
LINK tests/ac97-test
LINK tests/es1370-test
LINK tests/virtio-net-test
LINK tests/virtio-balloon-test
LINK tests/virtio-blk-test
LINK tests/virtio-rng-test
LINK tests/virtio-scsi-test
LINK tests/virtio-serial-test
LINK tests/virtio-console-test
LINK tests/tpci200-test
LINK tests/ipoctal232-test
LINK tests/display-vga-test
LINK tests/intel-hda-test
LINK tests/ivshmem-test
LINK tests/megasas-test
LINK tests/vmxnet3-test
LINK tests/pvpanic-test
LINK tests/i82801b11-test
LINK tests/ioh3420-test
LINK tests/usb-hcd-ohci-test
LINK tests/usb-hcd-uhci-test
LINK tests/usb-hcd-ehci-test
LINK tests/usb-hcd-xhci-test
LINK tests/pc-cpu-test
LINK tests/q35-test
LINK tests/vmgenid-test
LINK tests/test-netfilter
LINK tests/test-filter-mirror
LINK tests/test-filter-redirector
LINK tests/migration-test
LINK tests/test-x86-cpuid-compat
LINK tests/numa-test
LINK tests/qmp-test
LINK tests/device-introspect-test
LINK tests/qom-test
LINK tests/test-hmp
GTESTER tests/check-qdict
GTESTER tests/test-char
GTESTER tests/check-qnum
GTESTER tests/check-qstring
GTESTER tests/check-qlist
GTESTER tests/check-qobject
GTESTER tests/check-qnull
GTESTER tests/check-qlit
GTESTER tests/check-qjson
LINK tests/test-qobject-output-visitor
LINK tests/test-clone-visitor
LINK tests/test-qobject-input-visitor
LINK tests/test-qmp-commands
LINK tests/test-string-input-visitor
LINK tests/test-string-output-visitor
LINK tests/test-qmp-event
LINK tests/test-opts-visitor
GTESTER tests/test-iov
GTESTER tests/test-coroutine
GTESTER tests/test-visitor-serialization
GTESTER tests/test-aio-multithread
GTESTER tests/test-aio
GTESTER tests/test-throttle
GTESTER tests/test-thread-pool
GTESTER tests/test-hbitmap
GTESTER tests/test-blockjob
GTESTER tests/test-blockjob-txn
GTESTER tests/test-x86-cpuid
GTESTER tests/test-xbzrle
GTESTER tests/test-vmstate
Failed to load simple/primitive:b_1
Failed to load simple/primitive:i64_2
Failed to load simple/primitive:i32_1
Failed to load simple/primitive:i32_1
Failed to load test/with_tmp:a
Failed to load test/tmp_child_parent:f
Failed to load test/tmp_child:parent
Failed to load test/with_tmp:tmp
Failed to load test/tmp_child:diff
Failed to load test/with_tmp:tmp
Failed to load test/tmp_child:diff
Failed to load test/with_tmp:tmp
GTESTER tests/test-cutils
GTESTER tests/test-shift128
GTESTER tests/test-mul64
GTESTER tests/test-int128
GTESTER tests/rcutorture
GTESTER tests/test-rcu-list
GTESTER tests/test-qdist
GTESTER tests/test-qht
LINK tests/test-qht-par
GTESTER tests/test-bitops
GTESTER tests/test-bitcnt
GTESTER tests/check-qom-interface
GTESTER tests/check-qom-proplist
GTESTER tests/test-qemu-opts
GTESTER tests/test-keyval
GTESTER tests/test-write-threshold
GTESTER tests/test-crypto-hash
GTESTER tests/test-crypto-hmac
GTESTER tests/test-crypto-cipher
GTESTER tests/test-crypto-secret
GTESTER tests/test-qga
GTESTER tests/test-timed-average
GTESTER tests/test-io-task
GTESTER tests/test-io-channel-socket
GTESTER tests/test-io-channel-file
GTESTER tests/test-io-channel-command
GTESTER tests/test-io-channel-buffer
GTESTER tests/test-base64
GTESTER tests/test-crypto-ivgen
GTESTER tests/test-crypto-afsplit
GTESTER tests/test-crypto-xts
GTESTER tests/test-crypto-block
GTESTER tests/test-logging
GTESTER tests/test-replication
GTESTER tests/test-bufferiszero
GTESTER tests/test-uuid
GTESTER tests/ptimer-test
GTESTER tests/test-qapi-util
GTESTER check-qtest-x86_64
GTESTER check-qtest-aarch64
GTESTER tests/test-qobject-output-visitor
GTESTER tests/test-clone-visitor
GTESTER tests/test-qobject-input-visitor
GTESTER tests/test-qmp-commands
GTESTER tests/test-string-input-visitor
GTESTER tests/test-string-output-visitor
GTESTER tests/test-qmp-event
GTESTER tests/test-opts-visitor
GTESTER tests/test-qht-par
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
mkdir -p dtc/libfdt
mkdir -p dtc/tests
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu"
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/var"/run
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
install -c -m 0755 qemu-ga ivshmem-client ivshmem-server qemu-nbd qemu-img qemu-io scsi/qemu-pr-helper "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-ga" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/ivshmem-client" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/ivshmem-server" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-nbd" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-img" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-io" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-pr-helper"
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/libexec"
install -c -m 0755 qemu-bridge-helper "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/libexec"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/libexec/qemu-bridge-helper"
set -e; for x in bios.bin bios-256k.bin sgabios.bin vgabios.bin vgabios-cirrus.bin vgabios-stdvga.bin vgabios-vmware.bin vgabios-qxl.bin vgabios-virtio.bin acpi-dsdt.aml ppc_rom.bin openbios-sparc32 openbios-sparc64 openbios-ppc QEMU,tcx.bin QEMU,cgthree.bin pxe-e1000.rom pxe-eepro100.rom pxe-ne2k_pci.rom pxe-pcnet.rom pxe-rtl8139.rom pxe-virtio.rom efi-e1000.rom efi-eepro100.rom efi-ne2k_pci.rom efi-pcnet.rom efi-rtl8139.rom efi-virtio.rom efi-e1000e.rom efi-vmxnet3.rom qemu-icon.bmp qemu_logo_no_text.svg bamboo.dtb petalogix-s3adsp1800.dtb petalogix-ml605.dtb multiboot.bin linuxboot.bin linuxboot_dma.bin kvmvapic.bin s390-ccw.img s390-netboot.img spapr-rtas.bin slof.bin skiboot.lid palcode-clipper u-boot.e500 qemu_vga.ndrv; do \
install -c -m 0644 /tmp/qemu-test/src/pc-bios/$x "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu"; \
done
make -C po install
make[1]: Entering directory `/tmp/qemu-test/build/po'
GEN /tmp/qemu-test/src/po/messages.po
GEN /tmp/qemu-test/src/po/bg.po
GEN /tmp/qemu-test/src/po/fr_FR.po
GEN /tmp/qemu-test/src/po/de_DE.po
GEN /tmp/qemu-test/src/po/it.po
GEN /tmp/qemu-test/src/po/hu.po
GEN /tmp/qemu-test/src/po/tr.po
GEN /tmp/qemu-test/src/po/zh_CN.po
GEN de_DE.mo
GEN fr_FR.mo
GEN tr.mo
GEN zh_CN.mo
GEN it.mo
GEN bg.mo
GEN hu.mo
for obj in bg.mo de_DE.mo fr_FR.mo hu.mo it.mo tr.mo zh_CN.mo; do \
base=`basename $obj .mo`; \
install -d /tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/locale/$base/LC_MESSAGES; \
install -m644 $obj /tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/locale/$base/LC_MESSAGES/qemu.mo; \
done
make[1]: Leaving directory `/tmp/qemu-test/build/po'
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu/keymaps"
set -e; for x in da en-gb et fr fr-ch is lt modifiers no pt-br sv ar de en-us fi fr-be hr it lv nl pl ru th common de-ch es fo fr-ca hu ja mk nl-be pt sl tr bepo cz; do \
install -c -m 0644 /tmp/qemu-test/src/pc-bios/keymaps/$x "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu/keymaps"; \
done
install -c -m 0644 /tmp/qemu-test/build/trace-events-all "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu/trace-events-all"
for d in x86_64-softmmu aarch64-softmmu; do \
make --no-print-directory BUILD_DIR=/tmp/qemu-test/build TARGET_DIR=$d/ -C $d install || exit 1 ; \
done
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
install -c -m 0755 qemu-system-x86_64 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-system-x86_64"
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
install -c -m 0755 qemu-system-aarch64 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-system-aarch64"
CLEANUP /var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-22.59.39.2059
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-r1w5vyee/src'
real 8m8.503s
user 0m4.527s
sys 0m4.878s
BUILD min-glib
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-r1w5vyee/src'
GEN /var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.07.49.15568/qemu.tar
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.07.49.15568/qemu.tar.vroot'...
done.
Checking out files: 12% (722/5667)
Checking out files: 13% (737/5667)
Checking out files: 14% (794/5667)
Checking out files: 15% (851/5667)
Checking out files: 16% (907/5667)
Checking out files: 17% (964/5667)
Checking out files: 18% (1021/5667)
Checking out files: 19% (1077/5667)
Checking out files: 20% (1134/5667)
Checking out files: 21% (1191/5667)
Checking out files: 22% (1247/5667)
Checking out files: 23% (1304/5667)
Checking out files: 24% (1361/5667)
Checking out files: 25% (1417/5667)
Checking out files: 26% (1474/5667)
Checking out files: 26% (1498/5667)
Checking out files: 27% (1531/5667)
Checking out files: 28% (1587/5667)
Checking out files: 29% (1644/5667)
Checking out files: 30% (1701/5667)
Checking out files: 31% (1757/5667)
Checking out files: 32% (1814/5667)
Checking out files: 33% (1871/5667)
Checking out files: 34% (1927/5667)
Checking out files: 35% (1984/5667)
Checking out files: 35% (1988/5667)
Checking out files: 36% (2041/5667)
Checking out files: 37% (2097/5667)
Checking out files: 38% (2154/5667)
Checking out files: 39% (2211/5667)
Checking out files: 40% (2267/5667)
Checking out files: 41% (2324/5667)
Checking out files: 42% (2381/5667)
Checking out files: 43% (2437/5667)
Checking out files: 44% (2494/5667)
Checking out files: 44% (2547/5667)
Checking out files: 45% (2551/5667)
Checking out files: 46% (2607/5667)
Checking out files: 47% (2664/5667)
Checking out files: 48% (2721/5667)
Checking out files: 49% (2777/5667)
Checking out files: 50% (2834/5667)
Checking out files: 51% (2891/5667)
Checking out files: 52% (2947/5667)
Checking out files: 53% (3004/5667)
Checking out files: 54% (3061/5667)
Checking out files: 55% (3117/5667)
Checking out files: 56% (3174/5667)
Checking out files: 56% (3200/5667)
Checking out files: 57% (3231/5667)
Checking out files: 58% (3287/5667)
Checking out files: 59% (3344/5667)
Checking out files: 60% (3401/5667)
Checking out files: 61% (3457/5667)
Checking out files: 62% (3514/5667)
Checking out files: 63% (3571/5667)
Checking out files: 64% (3627/5667)
Checking out files: 65% (3684/5667)
Checking out files: 66% (3741/5667)
Checking out files: 67% (3797/5667)
Checking out files: 68% (3854/5667)
Checking out files: 69% (3911/5667)
Checking out files: 70% (3967/5667)
Checking out files: 71% (4024/5667)
Checking out files: 72% (4081/5667)
Checking out files: 73% (4137/5667)
Checking out files: 74% (4194/5667)
Checking out files: 74% (4231/5667)
Checking out files: 75% (4251/5667)
Checking out files: 76% (4307/5667)
Checking out files: 77% (4364/5667)
Checking out files: 78% (4421/5667)
Checking out files: 79% (4477/5667)
Checking out files: 80% (4534/5667)
Checking out files: 81% (4591/5667)
Checking out files: 81% (4598/5667)
Checking out files: 82% (4647/5667)
Checking out files: 83% (4704/5667)
Checking out files: 84% (4761/5667)
Checking out files: 85% (4817/5667)
Checking out files: 86% (4874/5667)
Checking out files: 87% (4931/5667)
Checking out files: 88% (4987/5667)
Checking out files: 89% (5044/5667)
Checking out files: 90% (5101/5667)
Checking out files: 91% (5157/5667)
Checking out files: 92% (5214/5667)
Checking out files: 93% (5271/5667)
Checking out files: 94% (5327/5667)
Checking out files: 95% (5384/5667)
Checking out files: 96% (5441/5667)
Checking out files: 97% (5497/5667)
Checking out files: 98% (5554/5667)
Checking out files: 98% (5591/5667)
Checking out files: 99% (5611/5667)
Checking out files: 100% (5667/5667)
Checking out files: 100% (5667/5667), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.07.49.15568/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out '558cd81bdd432769b59bff01240c44f82cfb1a9d'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.07.49.15568/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '10739aa26051a5d49d88132604539d3ed085e72e'
COPY RUNNER
RUN test-build in qemu:min-glib
Environment variables:
HOSTNAME=c51fba4a3231
MAKEFLAGS= -j8
J=8
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
FEATURES= dtc
DEBUG=
_=/usr/bin/env
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install
No C++ compiler available; disabling C++ specific optional code
Install prefix /tmp/qemu-test/install
BIOS directory /tmp/qemu-test/install/share/qemu
firmware path /tmp/qemu-test/install/share/qemu-firmware
binary directory /tmp/qemu-test/install/bin
library directory /tmp/qemu-test/install/lib
module directory /tmp/qemu-test/install/lib/qemu
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory /tmp/qemu-test/install/etc
local state directory /tmp/qemu-test/install/var
Manual directory /tmp/qemu-test/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path /tmp/qemu-test/src
GIT binary git
GIT submodules
C compiler cc
Host C compiler cc
C++ compiler
Objective-C compiler cc
ARFLAGS rv
CFLAGS -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g
QEMU_CFLAGS -I/usr/include/pixman-1 -I$(SRC_PATH)/dtc/libfdt -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -Wendif-labels -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all -Wno-missing-braces
LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g
make make
install install
python python -B
smbd /usr/sbin/smbd
module support no
host CPU x86_64
host big endian no
target list x86_64-softmmu aarch64-softmmu
gprof enabled no
sparse enabled no
strip binaries yes
profiler no
static build no
SDL support yes (1.2.14)
GTK support no
GTK GL support no
VTE support no
TLS priority NORMAL
GNUTLS support no
GNUTLS rnd no
libgcrypt no
libgcrypt kdf no
nettle no
nettle kdf no
libtasn1 no
curses support no
virgl support no
curl support no
mingw32 support no
Audio drivers oss
Block whitelist (rw)
Block whitelist (ro)
VirtFS support no
Multipath support no
VNC support yes
VNC SASL support no
VNC JPEG support no
VNC PNG support no
xen support no
brlapi support no
bluez support no
Documentation no
PIE yes
vde support no
netmap support no
Linux AIO support no
ATTR/XATTR support yes
Install blobs yes
KVM support yes
HAX support no
TCG support yes
TCG debug enabled no
TCG interpreter no
RDMA support no
fdt support yes
preadv support yes
fdatasync yes
madvise yes
posix_madvise yes
libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends log
spice support no
rbd support no
xfsctl support no
smartcard support no
libusb no
usb net redir no
OpenGL support no
OpenGL dmabufs no
libiscsi support no
libnfs support no
build guest agent yes
QGA VSS support no
QGA w32 disk info no
QGA MSI support no
seccomp support no
coroutine backend ucontext
coroutine pool yes
debug stack usage no
crypto afalg no
GlusterFS support no
gcov gcov
gcov enabled no
TPM support yes
libssh2 support no
TPM passthrough yes
TPM emulator yes
QOM debugging yes
Live block migration yes
lzo support no
snappy support no
bzip2 support no
NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization no
replication support yes
VxHS block device no
capstone no
GEN x86_64-softmmu/config-devices.mak.tmp
GEN aarch64-softmmu/config-devices.mak.tmp
mkdir -p dtc/libfdt
mkdir -p dtc/tests
GEN config-host.h
GEN qemu-options.def
GEN qmp-commands.h
GEN qapi-types.h
GEN qapi-event.h
GEN qapi-visit.h
GEN x86_64-softmmu/config-devices.mak
GEN qmp-marshal.c
GEN aarch64-softmmu/config-devices.mak
GEN qapi-types.c
GEN qapi-visit.c
GEN qapi-event.c
GEN qmp-introspect.h
GEN qmp-introspect.c
GEN trace/generated-tcg-tracers.h
GEN trace/generated-helpers-wrappers.h
GEN trace/generated-helpers.h
GEN trace/generated-helpers.c
GEN module_block.h
GEN ui/input-keymap-linux-to-qcode.c
GEN ui/input-keymap-qcode-to-qnum.c
GEN ui/input-keymap-qnum-to-qcode.c
GEN tests/test-qapi-types.h
GEN tests/test-qapi-visit.h
GEN tests/test-qmp-commands.h
GEN tests/test-qapi-event.h
GEN tests/test-qmp-introspect.h
GEN trace-root.h
GEN util/trace.h
GEN crypto/trace.h
GEN io/trace.h
GEN migration/trace.h
GEN block/trace.h
GEN chardev/trace.h
GEN hw/block/trace.h
GEN hw/block/dataplane/trace.h
GEN hw/char/trace.h
GEN hw/intc/trace.h
GEN hw/net/trace.h
GEN hw/virtio/trace.h
GEN hw/audio/trace.h
GEN hw/misc/trace.h
GEN hw/usb/trace.h
GEN hw/scsi/trace.h
GEN hw/nvram/trace.h
GEN hw/display/trace.h
GEN hw/input/trace.h
GEN hw/timer/trace.h
GEN hw/dma/trace.h
GEN hw/sparc/trace.h
GEN hw/sd/trace.h
GEN hw/isa/trace.h
GEN hw/mem/trace.h
GEN hw/i386/trace.h
GEN hw/i386/xen/trace.h
GEN hw/9pfs/trace.h
GEN hw/ppc/trace.h
GEN hw/pci/trace.h
GEN hw/s390x/trace.h
GEN hw/vfio/trace.h
GEN hw/acpi/trace.h
GEN hw/arm/trace.h
GEN hw/alpha/trace.h
GEN hw/xen/trace.h
GEN hw/ide/trace.h
GEN ui/trace.h
GEN audio/trace.h
GEN net/trace.h
GEN target/arm/trace.h
GEN target/i386/trace.h
GEN target/mips/trace.h
GEN target/sparc/trace.h
GEN target/s390x/trace.h
GEN target/ppc/trace.h
GEN qom/trace.h
GEN linux-user/trace.h
GEN qapi/trace.h
GEN accel/tcg/trace.h
GEN accel/kvm/trace.h
GEN nbd/trace.h
GEN scsi/trace.h
GEN trace-root.c
GEN util/trace.c
GEN crypto/trace.c
GEN io/trace.c
GEN migration/trace.c
GEN block/trace.c
GEN chardev/trace.c
GEN hw/block/trace.c
GEN hw/block/dataplane/trace.c
GEN hw/char/trace.c
GEN hw/intc/trace.c
GEN hw/net/trace.c
GEN hw/virtio/trace.c
GEN hw/audio/trace.c
GEN hw/misc/trace.c
GEN hw/usb/trace.c
GEN hw/scsi/trace.c
GEN hw/nvram/trace.c
GEN hw/display/trace.c
GEN hw/input/trace.c
GEN hw/timer/trace.c
GEN hw/dma/trace.c
GEN hw/sparc/trace.c
GEN hw/sd/trace.c
GEN hw/isa/trace.c
GEN hw/mem/trace.c
GEN hw/i386/trace.c
GEN hw/i386/xen/trace.c
GEN hw/9pfs/trace.c
GEN hw/ppc/trace.c
GEN hw/pci/trace.c
GEN hw/s390x/trace.c
GEN hw/vfio/trace.c
GEN hw/acpi/trace.c
GEN hw/arm/trace.c
GEN hw/alpha/trace.c
GEN hw/xen/trace.c
GEN hw/ide/trace.c
GEN ui/trace.c
GEN audio/trace.c
GEN net/trace.c
GEN target/arm/trace.c
GEN target/i386/trace.c
GEN target/mips/trace.c
GEN target/sparc/trace.c
GEN target/s390x/trace.c
GEN target/ppc/trace.c
GEN qom/trace.c
GEN linux-user/trace.c
GEN qapi/trace.c
GEN accel/tcg/trace.c
GEN accel/kvm/trace.c
GEN nbd/trace.c
GEN scsi/trace.c
GEN config-all-devices.mak
DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
DEP /tmp/qemu-test/src/dtc/tests/trees.S
DEP /tmp/qemu-test/src/dtc/tests/testutils.c
DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
DEP /tmp/qemu-test/src/dtc/tests/check_path.c
DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
DEP /tmp/qemu-test/src/dtc/tests/overlay.c
DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
DEP /tmp/qemu-test/src/dtc/tests/incbin.c
DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
DEP /tmp/qemu-test/src/dtc/tests/path-references.c
DEP /tmp/qemu-test/src/dtc/tests/references.c
DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
DEP /tmp/qemu-test/src/dtc/tests/del_node.c
DEP /tmp/qemu-test/src/dtc/tests/del_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop.c
DEP /tmp/qemu-test/src/dtc/tests/set_name.c
DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
DEP /tmp/qemu-test/src/dtc/tests/notfound.c
DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
DEP /tmp/qemu-test/src/dtc/tests/get_path.c
DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/getprop.c
DEP /tmp/qemu-test/src/dtc/tests/get_name.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
DEP /tmp/qemu-test/src/dtc/tests/root_node.c
DEP /tmp/qemu-test/src/dtc/tests/find_property.c
DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
DEP /tmp/qemu-test/src/dtc/util.c
DEP /tmp/qemu-test/src/dtc/fdtput.c
DEP /tmp/qemu-test/src/dtc/fdtget.c
DEP /tmp/qemu-test/src/dtc/fdtdump.c
LEX convert-dtsv0-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/srcpos.c
make[1]: flex: Command not found
BISON dtc-parser.tab.c
DEP /tmp/qemu-test/src/dtc/treesource.c
LEX dtc-lexer.lex.c
make[1]: bison: Command not found
make[1]: flex: Command not found
DEP /tmp/qemu-test/src/dtc/livetree.c
DEP /tmp/qemu-test/src/dtc/fstree.c
DEP /tmp/qemu-test/src/dtc/dtc.c
DEP /tmp/qemu-test/src/dtc/flattree.c
DEP /tmp/qemu-test/src/dtc/data.c
DEP /tmp/qemu-test/src/dtc/checks.c
CHK version_gen.h
LEX convert-dtsv0-lexer.lex.c
make[1]: bison: Command not found
make[1]: flex: Command not found
BISON dtc-parser.tab.c
LEX dtc-lexer.lex.c
make[1]: flex: Command not found
UPD version_gen.h
DEP /tmp/qemu-test/src/dtc/util.c
LEX convert-dtsv0-lexer.lex.c
make[1]: flex: Command not found
BISON dtc-parser.tab.c
make[1]: bison: Command not found
LEX dtc-lexer.lex.c
make[1]: flex: Command not found
CC libfdt/fdt.o
CC libfdt/fdt_ro.o
CC libfdt/fdt_wip.o
CC libfdt/fdt_rw.o
CC libfdt/fdt_sw.o
CC libfdt/fdt_strerror.o
CC libfdt/fdt_empty_tree.o
CC libfdt/fdt_addresses.o
CC libfdt/fdt_overlay.o
AR libfdt/libfdt.a
ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
mkdir -p dtc/libfdt
mkdir -p dtc/tests
LEX dtc-lexer.lex.c
make[1]: flex: Command not found
make[1]: flex: Command not found
LEX convert-dtsv0-lexer.lex.c
BISON dtc-parser.tab.c
make[1]: bison: Command not found
CC tests/qemu-iotests/socket_scm_helper.o
GEN qga/qapi-generated/qga-qapi-types.h
GEN qga/qapi-generated/qga-qmp-commands.h
GEN qga/qapi-generated/qga-qapi-types.c
GEN qga/qapi-generated/qga-qapi-visit.h
CC qmp-introspect.o
GEN qga/qapi-generated/qga-qapi-visit.c
GEN qga/qapi-generated/qga-qmp-marshal.c
CC qapi-types.o
CC qapi-visit.o
CC qapi-event.o
CC qapi/qapi-visit-core.o
CC qapi/qapi-dealloc-visitor.o
CC qapi/qobject-input-visitor.o
CC qapi/qobject-output-visitor.o
CC qapi/qmp-registry.o
CC qapi/qmp-dispatch.o
CC qapi/string-output-visitor.o
CC qapi/string-input-visitor.o
CC qapi/opts-visitor.o
CC qapi/qapi-clone-visitor.o
CC qapi/qmp-event.o
CC qapi/qapi-util.o
CC qobject/qnull.o
CC qobject/qnum.o
CC qobject/qstring.o
CC qobject/qdict.o
CC qobject/qlist.o
CC qobject/qbool.o
CC qobject/qlit.o
CC qobject/qjson.o
CC qobject/qobject.o
CC qobject/json-lexer.o
CC qobject/json-streamer.o
CC trace/control.o
CC qobject/json-parser.o
CC trace/qmp.o
CC util/osdep.o
CC util/cutils.o
CC util/unicode.o
CC util/qemu-timer-common.o
CC util/bufferiszero.o
CC util/lockcnt.o
CC util/aiocb.o
CC util/async.o
CC util/thread-pool.o
CC util/qemu-timer.o
CC util/main-loop.o
CC util/iohandler.o
CC util/compatfd.o
CC util/aio-posix.o
CC util/event_notifier-posix.o
CC util/mmap-alloc.o
CC util/oslib-posix.o
CC util/qemu-openpty.o
CC util/qemu-thread-posix.o
CC util/memfd.o
CC util/envlist.o
CC util/module.o
CC util/path.o
CC util/host-utils.o
CC util/bitmap.o
CC util/bitops.o
CC util/hbitmap.o
CC util/fifo8.o
CC util/acl.o
CC util/cacheinfo.o
CC util/error.o
CC util/qemu-error.o
CC util/id.o
CC util/iov.o
CC util/qemu-config.o
CC util/qemu-sockets.o
CC util/uri.o
CC util/notify.o
CC util/qemu-progress.o
CC util/qemu-option.o
CC util/keyval.o
CC util/hexdump.o
CC util/crc32c.o
CC util/uuid.o
CC util/throttle.o
CC util/readline.o
CC util/rcu.o
CC util/getauxval.o
CC util/qemu-coroutine.o
CC util/qemu-coroutine-lock.o
CC util/qemu-coroutine-io.o
CC util/coroutine-ucontext.o
CC util/qemu-coroutine-sleep.o
CC util/buffer.o
CC util/timed-average.o
CC util/base64.o
CC util/log.o
CC util/pagesize.o
CC util/qdist.o
CC util/qht.o
CC util/range.o
CC util/stats64.o
CC util/systemd.o
CC util/trace.o
CC trace-root.o
CC crypto/trace.o
CC io/trace.o
CC migration/trace.o
CC block/trace.o
CC chardev/trace.o
CC hw/block/trace.o
CC hw/block/dataplane/trace.o
CC hw/char/trace.o
CC hw/intc/trace.o
CC hw/net/trace.o
CC hw/virtio/trace.o
CC hw/audio/trace.o
CC hw/misc/trace.o
CC hw/usb/trace.o
CC hw/scsi/trace.o
CC hw/nvram/trace.o
CC hw/display/trace.o
CC hw/input/trace.o
CC hw/timer/trace.o
CC hw/dma/trace.o
CC hw/sparc/trace.o
CC hw/sd/trace.o
CC hw/isa/trace.o
CC hw/i386/trace.o
CC hw/mem/trace.o
CC hw/i386/xen/trace.o
CC hw/9pfs/trace.o
CC hw/ppc/trace.o
CC hw/pci/trace.o
CC hw/s390x/trace.o
CC hw/vfio/trace.o
CC hw/acpi/trace.o
CC hw/arm/trace.o
CC hw/alpha/trace.o
CC hw/xen/trace.o
CC hw/ide/trace.o
CC ui/trace.o
CC audio/trace.o
CC net/trace.o
CC target/arm/trace.o
CC target/mips/trace.o
CC target/i386/trace.o
CC target/sparc/trace.o
CC target/s390x/trace.o
CC target/ppc/trace.o
CC qom/trace.o
CC linux-user/trace.o
CC qapi/trace.o
CC accel/tcg/trace.o
CC accel/kvm/trace.o
CC nbd/trace.o
CC scsi/trace.o
CC stubs/arch-query-cpu-def.o
CC crypto/pbkdf-stub.o
CC stubs/arch-query-cpu-model-expansion.o
CC stubs/arch-query-cpu-model-comparison.o
CC stubs/arch-query-cpu-model-baseline.o
CC stubs/bdrv-next-monitor-owned.o
CC stubs/blk-commit-all.o
CC stubs/clock-warp.o
CC stubs/blockdev-close-all-bdrv-states.o
CC stubs/cpu-get-clock.o
CC stubs/cpu-get-icount.o
CC stubs/dump.o
CC stubs/fdset.o
CC stubs/gdbstub.o
CC stubs/error-printf.o
CC stubs/get-vm-name.o
CC stubs/iothread-lock.o
CC stubs/iothread.o
CC stubs/is-daemonized.o
CC stubs/machine-init-done.o
CC stubs/migr-blocker.o
CC stubs/change-state-handler.o
CC stubs/monitor.o
CC stubs/notify-event.o
CC stubs/qtest.o
CC stubs/replay.o
CC stubs/runstate-check.o
CC stubs/set-fd-handler.o
CC stubs/slirp.o
CC stubs/sysbus.o
CC stubs/tpm.o
CC stubs/trace-control.o
CC stubs/uuid.o
CC stubs/vm-stop.o
CC stubs/vmstate.o
CC stubs/qmp_pc_dimm.o
CC stubs/target-monitor-defs.o
CC stubs/target-get-monitor-def.o
CC stubs/pc_madt_cpu_entry.o
CC stubs/vmgenid.o
CC stubs/xen-common.o
CC stubs/pci-host-piix.o
CC stubs/xen-hvm.o
CC contrib/ivshmem-client/ivshmem-client.o
CC contrib/ivshmem-client/main.o
CC contrib/ivshmem-server/ivshmem-server.o
CC contrib/ivshmem-server/main.o
CC qemu-nbd.o
CC block.o
CC blockjob.o
CC qemu-io-cmds.o
CC replication.o
CC block/raw-format.o
CC block/qcow.o
CC block/vdi.o
CC block/vmdk.o
CC block/cloop.o
CC block/bochs.o
CC block/vpc.o
CC block/vvfat.o
CC block/dmg.o
CC block/qcow2.o
CC block/qcow2-refcount.o
CC block/qcow2-cluster.o
CC block/qcow2-snapshot.o
CC block/qcow2-cache.o
CC block/qcow2-bitmap.o
CC block/qed.o
CC block/qed-l2-cache.o
CC block/qed-table.o
CC block/qed-cluster.o
CC block/qed-check.o
CC block/vhdx.o
CC block/vhdx-endian.o
CC block/vhdx-log.o
CC block/quorum.o
CC block/blkverify.o
CC block/blkdebug.o
CC block/parallels.o
CC block/blkreplay.o
CC block/block-backend.o
CC block/snapshot.o
CC block/qapi.o
CC block/file-posix.o
CC block/null.o
CC block/mirror.o
CC block/commit.o
CC block/throttle-groups.o
CC block/io.o
CC block/nbd.o
CC block/nbd-client.o
CC block/sheepdog.o
CC block/accounting.o
CC block/dirty-bitmap.o
CC block/write-threshold.o
CC block/backup.o
CC block/replication.o
CC block/throttle.o
CC block/crypto.o
CC nbd/server.o
CC nbd/client.o
CC nbd/common.o
CC scsi/utils.o
CC scsi/pr-manager-helper.o
CC crypto/init.o
CC crypto/hash.o
CC scsi/pr-manager.o
CC crypto/hash-glib.o
CC crypto/hmac.o
CC crypto/hmac-glib.o
CC crypto/aes.o
CC crypto/desrfb.o
CC crypto/cipher.o
CC crypto/tlscreds.o
CC crypto/tlscredsanon.o
CC crypto/tlscredsx509.o
CC crypto/tlssession.o
CC crypto/secret.o
CC crypto/random-platform.o
CC crypto/pbkdf.o
CC crypto/ivgen.o
CC crypto/ivgen-essiv.o
CC crypto/ivgen-plain.o
CC crypto/ivgen-plain64.o
CC crypto/afsplit.o
CC crypto/xts.o
CC crypto/block.o
CC crypto/block-luks.o
CC io/channel.o
CC crypto/block-qcow.o
CC io/channel-buffer.o
CC io/channel-command.o
CC io/channel-file.o
CC io/channel-socket.o
CC io/channel-tls.o
CC io/channel-watch.o
CC io/channel-websock.o
CC io/channel-util.o
CC io/dns-resolver.o
CC io/task.o
CC qom/object.o
CC qom/container.o
CC qom/qom-qobject.o
CC qom/object_interfaces.o
GEN qemu-img-cmds.h
CC qemu-io.o
CC scsi/qemu-pr-helper.o
CC qemu-bridge-helper.o
CC blockdev.o
CC blockdev-nbd.o
CC bootdevice.o
CC iothread.o
CC qdev-monitor.o
CC device-hotplug.o
CC os-posix.o
CC bt-host.o
CC bt-vhci.o
CC dma-helpers.o
CC vl.o
CC tpm.o
CC device_tree.o
CC qmp-marshal.o
CC qmp.o
CC cpus-common.o
CC hmp.o
CC audio/audio.o
CC audio/noaudio.o
CC audio/wavaudio.o
CC audio/mixeng.o
CC audio/sdlaudio.o
CC audio/ossaudio.o
CC backends/rng.o
CC audio/wavcapture.o
CC backends/rng-egd.o
CC backends/rng-random.o
CC backends/tpm.o
CC backends/hostmem.o
CC backends/hostmem-ram.o
CC backends/hostmem-file.o
CC backends/cryptodev.o
CC block/stream.o
CC backends/cryptodev-builtin.o
CC chardev/msmouse.o
CC chardev/testdev.o
CC chardev/wctablet.o
CC disas/arm.o
CC disas/i386.o
CC fsdev/qemu-fsdev-dummy.o
CC fsdev/qemu-fsdev-opts.o
CC hw/acpi/core.o
CC fsdev/qemu-fsdev-throttle.o
CC hw/acpi/pcihp.o
CC hw/acpi/piix4.o
CC hw/acpi/ich9.o
CC hw/acpi/tco.o
CC hw/acpi/cpu_hotplug.o
CC hw/acpi/memory_hotplug.o
CC hw/acpi/cpu.o
CC hw/acpi/nvdimm.o
CC hw/acpi/vmgenid.o
CC hw/acpi/acpi_interface.o
CC hw/acpi/bios-linker-loader.o
CC hw/acpi/aml-build.o
CC hw/acpi/ipmi.o
CC hw/acpi/acpi-stub.o
CC hw/audio/sb16.o
CC hw/audio/es1370.o
CC hw/acpi/ipmi-stub.o
CC hw/audio/ac97.o
CC hw/audio/adlib.o
CC hw/audio/fmopl.o
CC hw/audio/gusemu_hal.o
CC hw/audio/gus.o
CC hw/audio/gusemu_mixer.o
CC hw/audio/cs4231a.o
CC hw/audio/intel-hda.o
CC hw/audio/hda-codec.o
CC hw/audio/pcspk.o
CC hw/audio/wm8750.o
CC hw/audio/pl041.o
CC hw/audio/lm4549.o
CC hw/audio/marvell_88w8618.o
CC hw/audio/soundhw.o
CC hw/block/block.o
CC hw/block/cdrom.o
CC hw/block/hd-geometry.o
CC hw/block/fdc.o
CC hw/block/m25p80.o
CC hw/block/nand.o
CC hw/block/pflash_cfi01.o
CC hw/block/pflash_cfi02.o
CC hw/block/ecc.o
CC hw/block/onenand.o
CC hw/block/nvme.o
CC hw/bt/core.o
CC hw/bt/l2cap.o
CC hw/bt/sdp.o
CC hw/bt/hci.o
CC hw/bt/hid.o
CC hw/bt/hci-csr.o
CC hw/char/ipoctal232.o
CC hw/char/parallel.o
CC hw/char/pl011.o
CC hw/char/serial.o
CC hw/char/serial-isa.o
CC hw/char/serial-pci.o
CC hw/char/cadence_uart.o
CC hw/char/virtio-console.o
CC hw/char/cmsdk-apb-uart.o
CC hw/char/imx_serial.o
CC hw/char/debugcon.o
CC hw/core/qdev.o
CC hw/core/qdev-properties.o
CC hw/core/reset.o
CC hw/core/bus.o
CC hw/core/fw-path-provider.o
CC hw/core/irq.o
CC hw/core/hotplug.o
CC hw/core/ptimer.o
CC hw/core/nmi.o
CC hw/core/sysbus.o
CC hw/core/machine.o
CC hw/core/loader.o
CC hw/core/qdev-properties-system.o
CC hw/core/register.o
CC hw/core/or-irq.o
CC hw/core/platform-bus.o
CC hw/cpu/core.o
CC hw/display/ads7846.o
CC hw/display/cirrus_vga.o
CC hw/display/pl110.o
CC hw/display/ssd0303.o
CC hw/display/ssd0323.o
CC hw/display/vga-pci.o
CC hw/display/vga-isa.o
CC hw/display/vmware_vga.o
CC hw/display/blizzard.o
CC hw/display/exynos4210_fimd.o
CC hw/display/framebuffer.o
CC hw/display/tc6393xb.o
CC hw/dma/pl080.o
CC hw/dma/pl330.o
CC hw/dma/i8257.o
CC hw/dma/xlnx-zynq-devcfg.o
CC hw/gpio/max7310.o
CC hw/gpio/pl061.o
CC hw/gpio/zaurus.o
CC hw/gpio/gpio_key.o
CC hw/i2c/core.o
CC hw/i2c/smbus_eeprom.o
CC hw/i2c/smbus.o
CC hw/i2c/i2c-ddc.o
CC hw/i2c/versatile_i2c.o
CC hw/i2c/smbus_ich9.o
CC hw/i2c/pm_smbus.o
CC hw/i2c/bitbang_i2c.o
CC hw/i2c/exynos4210_i2c.o
CC hw/i2c/imx_i2c.o
CC hw/i2c/aspeed_i2c.o
CC hw/ide/core.o
CC hw/ide/atapi.o
CC hw/ide/qdev.o
CC hw/ide/pci.o
CC hw/ide/isa.o
CC hw/ide/piix.o
CC hw/ide/microdrive.o
CC hw/ide/ahci.o
CC hw/ide/ich.o
CC hw/ide/ahci-allwinner.o
CC hw/input/hid.o
CC hw/input/lm832x.o
CC hw/input/pckbd.o
CC hw/input/pl050.o
CC hw/input/ps2.o
CC hw/input/stellaris_input.o
CC hw/input/tsc2005.o
CC hw/input/vmmouse.o
CC hw/input/virtio-input.o
CC hw/input/virtio-input-hid.o
CC hw/input/virtio-input-host.o
CC hw/intc/i8259_common.o
CC hw/intc/i8259.o
CC hw/intc/pl190.o
CC hw/intc/imx_avic.o
CC hw/intc/realview_gic.o
CC hw/intc/ioapic_common.o
CC hw/intc/arm_gic_common.o
CC hw/intc/arm_gic.o
CC hw/intc/arm_gicv2m.o
CC hw/intc/arm_gicv3_common.o
CC hw/intc/arm_gicv3.o
CC hw/intc/arm_gicv3_dist.o
CC hw/intc/arm_gicv3_redist.o
CC hw/intc/arm_gicv3_its_common.o
CC hw/intc/intc.o
CC hw/ipack/ipack.o
CC hw/ipack/tpci200.o
CC hw/ipmi/ipmi.o
CC hw/ipmi/ipmi_bmc_sim.o
CC hw/ipmi/ipmi_bmc_extern.o
CC hw/ipmi/isa_ipmi_bt.o
CC hw/ipmi/isa_ipmi_kcs.o
CC hw/isa/isa-bus.o
CC hw/isa/apm.o
CC hw/mem/pc-dimm.o
CC hw/mem/nvdimm.o
CC hw/misc/applesmc.o
CC hw/misc/max111x.o
CC hw/misc/tmp105.o
CC hw/misc/tmp421.o
CC hw/misc/debugexit.o
CC hw/misc/sga.o
CC hw/misc/pc-testdev.o
CC hw/misc/pci-testdev.o
CC hw/misc/edu.o
CC hw/misc/unimp.o
CC hw/misc/vmcoreinfo.o
CC hw/misc/arm_l2x0.o
CC hw/misc/arm_integrator_debug.o
CC hw/misc/a9scu.o
CC hw/misc/arm11scu.o
CC hw/net/ne2000.o
CC hw/net/eepro100.o
CC hw/net/pcnet-pci.o
CC hw/net/pcnet.o
CC hw/net/e1000.o
CC hw/net/e1000x_common.o
CC hw/net/net_tx_pkt.o
CC hw/net/net_rx_pkt.o
CC hw/net/e1000e.o
CC hw/net/e1000e_core.o
CC hw/net/rtl8139.o
CC hw/net/vmxnet3.o
CC hw/net/smc91c111.o
CC hw/net/lan9118.o
CC hw/net/xgmac.o
CC hw/net/ne2000-isa.o
CC hw/net/allwinner_emac.o
CC hw/net/imx_fec.o
CC hw/net/cadence_gem.o
CC hw/net/stellaris_enet.o
CC hw/net/ftgmac100.o
CC hw/net/rocker/rocker.o
CC hw/net/rocker/rocker_fp.o
CC hw/net/rocker/rocker_desc.o
CC hw/net/rocker/rocker_world.o
CC hw/net/rocker/rocker_of_dpa.o
CC hw/nvram/eeprom93xx.o
CC hw/nvram/fw_cfg.o
CC hw/nvram/chrp_nvram.o
CC hw/pci-bridge/pci_bridge_dev.o
CC hw/pci-bridge/pcie_root_port.o
CC hw/pci-bridge/gen_pcie_root_port.o
CC hw/pci-bridge/pcie_pci_bridge.o
CC hw/pci-bridge/pci_expander_bridge.o
CC hw/pci-bridge/xio3130_upstream.o
CC hw/pci-bridge/xio3130_downstream.o
CC hw/pci-bridge/ioh3420.o
CC hw/pci-bridge/i82801b11.o
CC hw/pci-host/pam.o
CC hw/pci-host/versatile.o
CC hw/pci-host/piix.o
CC hw/pci-host/q35.o
CC hw/pci-host/gpex.o
CC hw/pci/pci.o
CC hw/pci/pci_bridge.o
CC hw/pci/msix.o
CC hw/pci/msi.o
CC hw/pci/shpc.o
CC hw/pci/slotid_cap.o
CC hw/pci/pci_host.o
CC hw/pci/pcie_host.o
CC hw/pci/pcie.o
CC hw/pci/pcie_aer.o
CC hw/pci/pcie_port.o
CC hw/pci/pci-stub.o
CC hw/pcmcia/pcmcia.o
CC hw/scsi/scsi-disk.o
CC hw/scsi/scsi-generic.o
CC hw/scsi/scsi-bus.o
CC hw/scsi/lsi53c895a.o
CC hw/scsi/mptsas.o
CC hw/scsi/mptconfig.o
CC hw/scsi/mptendian.o
CC hw/scsi/megasas.o
CC hw/scsi/vmw_pvscsi.o
CC hw/scsi/esp.o
CC hw/scsi/esp-pci.o
CC hw/sd/pl181.o
CC hw/sd/ssi-sd.o
CC hw/sd/sd.o
CC hw/sd/core.o
CC hw/sd/sdhci.o
CC hw/smbios/smbios.o
CC hw/smbios/smbios_type_38.o
CC hw/smbios/smbios-stub.o
CC hw/smbios/smbios_type_38-stub.o
CC hw/ssi/pl022.o
CC hw/ssi/ssi.o
CC hw/ssi/xilinx_spips.o
CC hw/ssi/aspeed_smc.o
CC hw/ssi/stm32f2xx_spi.o
CC hw/ssi/mss-spi.o
CC hw/timer/arm_timer.o
CC hw/timer/arm_mptimer.o
CC hw/timer/armv7m_systick.o
CC hw/timer/a9gtimer.o
CC hw/timer/cadence_ttc.o
CC hw/timer/ds1338.o
CC hw/timer/hpet.o
CC hw/timer/i8254_common.o
CC hw/timer/i8254.o
CC hw/timer/pl031.o
CC hw/timer/twl92230.o
CC hw/timer/imx_epit.o
CC hw/timer/imx_gpt.o
CC hw/timer/stm32f2xx_timer.o
CC hw/timer/aspeed_timer.o
CC hw/timer/cmsdk-apb-timer.o
CC hw/timer/mss-timer.o
CC hw/tpm/tpm_tis.o
CC hw/tpm/tpm_passthrough.o
CC hw/tpm/tpm_util.o
CC hw/tpm/tpm_emulator.o
CC hw/usb/core.o
CC hw/usb/combined-packet.o
CC hw/usb/bus.o
CC hw/usb/libhw.o
CC hw/usb/desc.o
CC hw/usb/desc-msos.o
CC hw/usb/hcd-uhci.o
CC hw/usb/hcd-ohci.o
CC hw/usb/hcd-ehci.o
CC hw/usb/hcd-ehci-pci.o
CC hw/usb/hcd-ehci-sysbus.o
CC hw/usb/hcd-xhci.o
CC hw/usb/hcd-xhci-nec.o
CC hw/usb/hcd-musb.o
CC hw/usb/dev-hub.o
CC hw/usb/dev-hid.o
CC hw/usb/dev-wacom.o
CC hw/usb/dev-storage.o
CC hw/usb/dev-uas.o
CC hw/usb/dev-audio.o
CC hw/usb/dev-serial.o
CC hw/usb/dev-network.o
CC hw/usb/dev-bluetooth.o
CC hw/usb/dev-smartcard-reader.o
CC hw/usb/dev-mtp.o
CC hw/usb/host-stub.o
CC hw/virtio/virtio-rng.o
CC hw/virtio/virtio-pci.o
CC hw/virtio/virtio-bus.o
CC hw/virtio/virtio-mmio.o
CC hw/virtio/vhost-pci-slave.o
CC hw/virtio/vhost-stub.o
CC hw/watchdog/watchdog.o
CC hw/watchdog/wdt_i6300esb.o
CC hw/watchdog/wdt_ib700.o
CC hw/watchdog/wdt_aspeed.o
CC migration/migration.o
CC migration/socket.o
CC migration/fd.o
CC migration/exec.o
CC migration/tls.o
CC migration/channel.o
CC migration/savevm.o
CC migration/colo-comm.o
CC migration/colo.o
CC migration/colo-failover.o
CC migration/vmstate-types.o
CC migration/vmstate.o
CC migration/page_cache.o
CC migration/qemu-file.o
CC migration/global_state.o
CC migration/qemu-file-channel.o
CC migration/xbzrle.o
CC migration/postcopy-ram.o
CC migration/qjson.o
CC migration/block.o
CC net/net.o
CC net/queue.o
CC net/checksum.o
CC net/util.o
CC net/hub.o
CC net/socket.o
CC net/dump.o
CC net/eth.o
CC net/l2tpv3.o
CC net/vhost-user.o
CC net/slirp.o
CC net/filter.o
CC net/filter-buffer.o
CC net/filter-mirror.o
CC net/colo-compare.o
CC net/colo.o
CC net/filter-rewriter.o
CC net/filter-replay.o
CC net/tap.o
CC net/tap-linux.o
CC qom/cpu.o
CC replay/replay.o
CC replay/replay-internal.o
CC replay/replay-events.o
/tmp/qemu-test/src/replay/replay-internal.c: In function 'replay_put_array':
/tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
CC replay/replay-time.o
CC replay/replay-input.o
CC replay/replay-char.o
CC replay/replay-snapshot.o
CC replay/replay-net.o
CC replay/replay-audio.o
CC slirp/cksum.o
CC slirp/if.o
CC slirp/ip_icmp.o
CC slirp/ip6_icmp.o
CC slirp/ip6_input.o
CC slirp/ip6_output.o
CC slirp/ip_input.o
CC slirp/ip_output.o
CC slirp/dnssearch.o
CC slirp/dhcpv6.o
CC slirp/slirp.o
CC slirp/mbuf.o
CC slirp/misc.o
CC slirp/sbuf.o
CC slirp/socket.o
CC slirp/tcp_input.o
CC slirp/tcp_output.o
CC slirp/tcp_subr.o
CC slirp/tcp_timer.o
CC slirp/udp.o
CC slirp/udp6.o
CC slirp/bootp.o
/tmp/qemu-test/src/slirp/tcp_input.c: In function 'tcp_input':
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_p' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_len' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_tos' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_id' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_off' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_ttl' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_sum' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_src.s_addr' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: 'save_ip.ip_dst.s_addr' may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:220: warning: 'save_ip6.ip_nh' may be used uninitialized in this function
CC slirp/tftp.o
CC slirp/arp_table.o
CC slirp/ndp_table.o
CC slirp/ncsi.o
CC ui/keymaps.o
CC ui/console.o
CC ui/cursor.o
CC ui/qemu-pixman.o
CC ui/input.o
CC ui/input-keymap.o
CC ui/input-legacy.o
CC ui/sdl_zoom.o
CC ui/sdl.o
CC ui/x_keymap.o
CC ui/input-linux.o
CC ui/vnc.o
CC ui/vnc-enc-zlib.o
CC ui/vnc-enc-hextile.o
CC ui/vnc-enc-tight.o
CC ui/vnc-palette.o
CC ui/vnc-enc-zrle.o
CC ui/vnc-auth-vencrypt.o
CC ui/vnc-ws.o
CC ui/vnc-jobs.o
CC chardev/char-fd.o
CC chardev/char.o
CC chardev/char-fe.o
CC chardev/char-file.o
CC chardev/char-io.o
CC chardev/char-mux.o
CC chardev/char-null.o
CC chardev/char-pipe.o
CC chardev/char-pty.o
CC chardev/char-parallel.o
CC chardev/char-ringbuf.o
CC chardev/char-serial.o
CC chardev/char-socket.o
CC chardev/char-stdio.o
CC chardev/char-udp.o
LINK tests/qemu-iotests/socket_scm_helper
CC qga/commands.o
CC qga/guest-agent-command-state.o
CC qga/main.o
CC qga/commands-posix.o
CC qga/channel-posix.o
CC qga/qapi-generated/qga-qapi-types.o
CC qga/qapi-generated/qga-qmp-marshal.o
CC qga/qapi-generated/qga-qapi-visit.o
AR libqemuutil.a
CC qemu-img.o
AS optionrom/multiboot.o
AS optionrom/linuxboot.o
CC optionrom/linuxboot_dma.o
AS optionrom/kvmvapic.o
BUILD optionrom/multiboot.img
cc: unrecognized option '-no-integrated-as'
cc: unrecognized option '-no-integrated-as'
BUILD optionrom/linuxboot_dma.img
BUILD optionrom/multiboot.raw
BUILD optionrom/linuxboot.img
BUILD optionrom/linuxboot_dma.raw
BUILD optionrom/kvmvapic.img
SIGN optionrom/multiboot.bin
BUILD optionrom/linuxboot.raw
SIGN optionrom/linuxboot_dma.bin
BUILD optionrom/kvmvapic.raw
SIGN optionrom/linuxboot.bin
SIGN optionrom/kvmvapic.bin
LINK qemu-ga
LINK ivshmem-client
LINK ivshmem-server
LINK qemu-nbd
LINK qemu-img
LINK qemu-io
LINK scsi/qemu-pr-helper
LINK qemu-bridge-helper
GEN x86_64-softmmu/hmp-commands.h
GEN x86_64-softmmu/hmp-commands-info.h
GEN x86_64-softmmu/config-target.h
GEN aarch64-softmmu/config-target.h
GEN aarch64-softmmu/hmp-commands.h
GEN aarch64-softmmu/hmp-commands-info.h
CC x86_64-softmmu/tcg/tcg.o
CC x86_64-softmmu/tcg/tcg-op.o
CC x86_64-softmmu/exec.o
CC x86_64-softmmu/tcg/tcg-common.o
CC x86_64-softmmu/tcg/optimize.o
CC x86_64-softmmu/fpu/softfloat.o
CC x86_64-softmmu/disas.o
GEN x86_64-softmmu/gdbstub-xml.c
CC x86_64-softmmu/arch_init.o
CC x86_64-softmmu/cpus.o
CC aarch64-softmmu/exec.o
CC x86_64-softmmu/monitor.o
CC x86_64-softmmu/gdbstub.o
CC x86_64-softmmu/balloon.o
CC aarch64-softmmu/tcg/tcg.o
CC x86_64-softmmu/ioport.o
CC aarch64-softmmu/tcg/tcg-op.o
CC aarch64-softmmu/tcg/optimize.o
CC x86_64-softmmu/numa.o
CC x86_64-softmmu/qtest.o
CC x86_64-softmmu/memory.o
CC x86_64-softmmu/memory_mapping.o
CC x86_64-softmmu/dump.o
CC x86_64-softmmu/migration/ram.o
CC aarch64-softmmu/tcg/tcg-common.o
CC aarch64-softmmu/fpu/softfloat.o
CC x86_64-softmmu/accel/accel.o
CC x86_64-softmmu/accel/kvm/kvm-all.o
CC aarch64-softmmu/disas.o
CC x86_64-softmmu/accel/stubs/hax-stub.o
CC x86_64-softmmu/accel/tcg/tcg-all.o
CC x86_64-softmmu/accel/tcg/cputlb.o
GEN aarch64-softmmu/gdbstub-xml.c
CC x86_64-softmmu/accel/tcg/tcg-runtime.o
CC x86_64-softmmu/accel/tcg/cpu-exec.o
CC aarch64-softmmu/arch_init.o
CC aarch64-softmmu/cpus.o
CC aarch64-softmmu/monitor.o
CC aarch64-softmmu/gdbstub.o
CC aarch64-softmmu/balloon.o
CC aarch64-softmmu/ioport.o
CC aarch64-softmmu/numa.o
CC aarch64-softmmu/qtest.o
CC aarch64-softmmu/memory.o
CC aarch64-softmmu/memory_mapping.o
CC aarch64-softmmu/dump.o
CC aarch64-softmmu/migration/ram.o
CC aarch64-softmmu/accel/accel.o
CC aarch64-softmmu/accel/stubs/hax-stub.o
CC aarch64-softmmu/accel/stubs/kvm-stub.o
CC aarch64-softmmu/accel/tcg/tcg-all.o
CC aarch64-softmmu/accel/tcg/cputlb.o
CC aarch64-softmmu/accel/tcg/tcg-runtime.o
CC aarch64-softmmu/accel/tcg/cpu-exec.o
CC aarch64-softmmu/accel/tcg/cpu-exec-common.o
CC x86_64-softmmu/accel/tcg/cpu-exec-common.o
CC aarch64-softmmu/accel/tcg/translate-all.o
CC x86_64-softmmu/accel/tcg/translate-all.o
CC x86_64-softmmu/accel/tcg/translator.o
CC aarch64-softmmu/accel/tcg/translator.o
CC x86_64-softmmu/hw/block/virtio-blk.o
CC aarch64-softmmu/hw/adc/stm32f2xx_adc.o
CC aarch64-softmmu/hw/block/virtio-blk.o
CC x86_64-softmmu/hw/block/dataplane/virtio-blk.o
CC x86_64-softmmu/hw/char/virtio-serial-bus.o
CC aarch64-softmmu/hw/block/dataplane/virtio-blk.o
CC aarch64-softmmu/hw/char/exynos4210_uart.o
CC aarch64-softmmu/hw/char/omap_uart.o
CC x86_64-softmmu/hw/core/generic-loader.o
CC aarch64-softmmu/hw/char/digic-uart.o
CC x86_64-softmmu/hw/core/null-machine.o
CC aarch64-softmmu/hw/char/stm32f2xx_usart.o
CC aarch64-softmmu/hw/char/bcm2835_aux.o
CC aarch64-softmmu/hw/char/virtio-serial-bus.o
CC aarch64-softmmu/hw/core/generic-loader.o
CC aarch64-softmmu/hw/core/null-machine.o
CC aarch64-softmmu/hw/cpu/arm11mpcore.o
CC x86_64-softmmu/hw/display/vga.o
CC aarch64-softmmu/hw/cpu/realview_mpcore.o
CC aarch64-softmmu/hw/cpu/a9mpcore.o
CC aarch64-softmmu/hw/cpu/a15mpcore.o
CC aarch64-softmmu/hw/display/omap_dss.o
CC aarch64-softmmu/hw/display/omap_lcdc.o
CC aarch64-softmmu/hw/display/pxa2xx_lcd.o
CC aarch64-softmmu/hw/display/bcm2835_fb.o
CC aarch64-softmmu/hw/display/vga.o
CC x86_64-softmmu/hw/display/virtio-gpu.o
CC aarch64-softmmu/hw/display/virtio-gpu.o
CC x86_64-softmmu/hw/display/virtio-gpu-3d.o
CC aarch64-softmmu/hw/display/virtio-gpu-3d.o
CC x86_64-softmmu/hw/display/virtio-gpu-pci.o
CC x86_64-softmmu/hw/display/virtio-vga.o
CC aarch64-softmmu/hw/display/virtio-gpu-pci.o
CC x86_64-softmmu/hw/intc/apic.o
CC x86_64-softmmu/hw/intc/apic_common.o
CC x86_64-softmmu/hw/intc/ioapic.o
CC x86_64-softmmu/hw/isa/lpc_ich9.o
CC x86_64-softmmu/hw/misc/vmport.o
CC x86_64-softmmu/hw/misc/ivshmem.o
CC x86_64-softmmu/hw/misc/pvpanic.o
CC aarch64-softmmu/hw/display/dpcd.o
CC x86_64-softmmu/hw/misc/hyperv_testdev.o
CC aarch64-softmmu/hw/display/xlnx_dp.o
CC aarch64-softmmu/hw/dma/xlnx_dpdma.o
CC aarch64-softmmu/hw/dma/omap_dma.o
CC aarch64-softmmu/hw/dma/soc_dma.o
CC x86_64-softmmu/hw/misc/mmio_interface.o
CC x86_64-softmmu/hw/net/virtio-net.o
CC aarch64-softmmu/hw/dma/pxa2xx_dma.o
CC x86_64-softmmu/hw/net/vhost_pci_net.o
CC aarch64-softmmu/hw/dma/bcm2835_dma.o
CC x86_64-softmmu/hw/net/vhost_net.o
CC x86_64-softmmu/hw/scsi/virtio-scsi.o
CC aarch64-softmmu/hw/gpio/omap_gpio.o
CC x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
CC aarch64-softmmu/hw/gpio/imx_gpio.o
CC x86_64-softmmu/hw/scsi/vhost-scsi-common.o
CC aarch64-softmmu/hw/gpio/bcm2835_gpio.o
CC aarch64-softmmu/hw/i2c/omap_i2c.o
CC aarch64-softmmu/hw/input/pxa2xx_keypad.o
CC x86_64-softmmu/hw/scsi/vhost-scsi.o
CC aarch64-softmmu/hw/input/tsc210x.o
CC aarch64-softmmu/hw/intc/armv7m_nvic.o
CC aarch64-softmmu/hw/intc/exynos4210_gic.o
CC x86_64-softmmu/hw/scsi/vhost-user-scsi.o
CC aarch64-softmmu/hw/intc/exynos4210_combiner.o
CC x86_64-softmmu/hw/timer/mc146818rtc.o
CC aarch64-softmmu/hw/intc/omap_intc.o
CC aarch64-softmmu/hw/intc/bcm2835_ic.o
CC aarch64-softmmu/hw/intc/bcm2836_control.o
CC x86_64-softmmu/hw/vfio/common.o
CC aarch64-softmmu/hw/intc/allwinner-a10-pic.o
CC aarch64-softmmu/hw/intc/aspeed_vic.o
CC aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
CC aarch64-softmmu/hw/misc/ivshmem.o
CC aarch64-softmmu/hw/misc/arm_sysctl.o
CC aarch64-softmmu/hw/misc/cbus.o
CC aarch64-softmmu/hw/misc/exynos4210_pmu.o
CC aarch64-softmmu/hw/misc/exynos4210_clk.o
CC aarch64-softmmu/hw/misc/exynos4210_rng.o
CC x86_64-softmmu/hw/vfio/pci.o
CC x86_64-softmmu/hw/vfio/pci-quirks.o
CC x86_64-softmmu/hw/vfio/platform.o
CC aarch64-softmmu/hw/misc/imx_ccm.o
CC aarch64-softmmu/hw/misc/imx31_ccm.o
CC x86_64-softmmu/hw/vfio/spapr.o
CC x86_64-softmmu/hw/virtio/virtio.o
CC x86_64-softmmu/hw/virtio/virtio-balloon.o
CC aarch64-softmmu/hw/misc/imx25_ccm.o
CC aarch64-softmmu/hw/misc/imx6_ccm.o
CC aarch64-softmmu/hw/misc/imx6_src.o
CC aarch64-softmmu/hw/misc/mst_fpga.o
CC aarch64-softmmu/hw/misc/omap_clk.o
CC aarch64-softmmu/hw/misc/omap_gpmc.o
CC aarch64-softmmu/hw/misc/omap_l4.o
CC x86_64-softmmu/hw/virtio/vhost.o
CC x86_64-softmmu/hw/virtio/vhost-backend.o
CC x86_64-softmmu/hw/virtio/vhost-user.o
CC x86_64-softmmu/hw/virtio/vhost-vsock.o
CC aarch64-softmmu/hw/misc/omap_sdrc.o
CC aarch64-softmmu/hw/misc/omap_tap.o
CC aarch64-softmmu/hw/misc/bcm2835_mbox.o
CC x86_64-softmmu/hw/virtio/virtio-crypto.o
CC x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
CC aarch64-softmmu/hw/misc/bcm2835_property.o
CC x86_64-softmmu/hw/i386/multiboot.o
CC aarch64-softmmu/hw/misc/bcm2835_rng.o
CC aarch64-softmmu/hw/misc/zynq_slcr.o
CC x86_64-softmmu/hw/i386/pc.o
CC x86_64-softmmu/hw/i386/pc_piix.o
CC x86_64-softmmu/hw/i386/pc_q35.o
CC aarch64-softmmu/hw/misc/zynq-xadc.o
CC x86_64-softmmu/hw/i386/pc_sysfw.o
CC aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
CC aarch64-softmmu/hw/misc/mps2-scc.o
CC aarch64-softmmu/hw/misc/auxbus.o
CC aarch64-softmmu/hw/misc/aspeed_scu.o
CC x86_64-softmmu/hw/i386/x86-iommu.o
CC aarch64-softmmu/hw/misc/aspeed_sdmc.o
CC aarch64-softmmu/hw/misc/mmio_interface.o
CC aarch64-softmmu/hw/misc/msf2-sysreg.o
CC x86_64-softmmu/hw/i386/intel_iommu.o
CC aarch64-softmmu/hw/net/virtio-net.o
CC aarch64-softmmu/hw/net/vhost_pci_net.o
CC x86_64-softmmu/hw/i386/amd_iommu.o
CC aarch64-softmmu/hw/net/vhost_net.o
CC x86_64-softmmu/hw/i386/kvmvapic.o
CC aarch64-softmmu/hw/pcmcia/pxa2xx.o
CC aarch64-softmmu/hw/scsi/virtio-scsi.o
/tmp/qemu-test/src/hw/i386/pc_piix.c: In function 'igd_passthrough_isa_bridge_create':
/tmp/qemu-test/src/hw/i386/pc_piix.c:1073: warning: 'pch_rev_id' may be used uninitialized in this function
CC x86_64-softmmu/hw/i386/acpi-build.o
CC x86_64-softmmu/hw/i386/kvm/clock.o
CC x86_64-softmmu/hw/i386/kvm/apic.o
CC x86_64-softmmu/hw/i386/kvm/i8259.o
CC x86_64-softmmu/hw/i386/kvm/ioapic.o
CC x86_64-softmmu/hw/i386/kvm/i8254.o
CC aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
CC aarch64-softmmu/hw/scsi/vhost-scsi-common.o
CC x86_64-softmmu/target/i386/helper.o
CC x86_64-softmmu/target/i386/cpu.o
CC x86_64-softmmu/target/i386/gdbstub.o
CC x86_64-softmmu/target/i386/xsave_helper.o
CC x86_64-softmmu/target/i386/translate.o
CC x86_64-softmmu/target/i386/bpt_helper.o
CC x86_64-softmmu/target/i386/cc_helper.o
CC x86_64-softmmu/target/i386/excp_helper.o
CC x86_64-softmmu/target/i386/fpu_helper.o
CC aarch64-softmmu/hw/scsi/vhost-scsi.o
CC aarch64-softmmu/hw/scsi/vhost-user-scsi.o
/tmp/qemu-test/src/hw/i386/acpi-build.c: In function 'build_append_pci_bus_devices':
/tmp/qemu-test/src/hw/i386/acpi-build.c:509: warning: 'notify_method' may be used uninitialized in this function
CC x86_64-softmmu/target/i386/int_helper.o
CC x86_64-softmmu/target/i386/mem_helper.o
CC aarch64-softmmu/hw/sd/omap_mmc.o
CC aarch64-softmmu/hw/sd/pxa2xx_mmci.o
CC x86_64-softmmu/target/i386/misc_helper.o
CC aarch64-softmmu/hw/sd/bcm2835_sdhost.o
CC aarch64-softmmu/hw/ssi/omap_spi.o
CC aarch64-softmmu/hw/ssi/imx_spi.o
CC x86_64-softmmu/target/i386/mpx_helper.o
CC aarch64-softmmu/hw/timer/exynos4210_mct.o
CC aarch64-softmmu/hw/timer/exynos4210_pwm.o
CC aarch64-softmmu/hw/timer/exynos4210_rtc.o
CC x86_64-softmmu/target/i386/seg_helper.o
CC aarch64-softmmu/hw/timer/omap_gptimer.o
CC aarch64-softmmu/hw/timer/omap_synctimer.o
CC aarch64-softmmu/hw/timer/pxa2xx_timer.o
CC aarch64-softmmu/hw/timer/digic-timer.o
CC aarch64-softmmu/hw/usb/tusb6010.o
CC aarch64-softmmu/hw/timer/allwinner-a10-pit.o
CC aarch64-softmmu/hw/vfio/common.o
CC x86_64-softmmu/target/i386/smm_helper.o
CC aarch64-softmmu/hw/vfio/pci.o
CC x86_64-softmmu/target/i386/svm_helper.o
CC aarch64-softmmu/hw/vfio/pci-quirks.o
CC x86_64-softmmu/target/i386/machine.o
CC aarch64-softmmu/hw/vfio/platform.o
CC aarch64-softmmu/hw/vfio/calxeda-xgmac.o
CC x86_64-softmmu/target/i386/arch_memory_mapping.o
CC aarch64-softmmu/hw/vfio/amd-xgbe.o
CC aarch64-softmmu/hw/vfio/spapr.o
CC x86_64-softmmu/target/i386/arch_dump.o
CC x86_64-softmmu/target/i386/monitor.o
CC x86_64-softmmu/target/i386/kvm.o
CC aarch64-softmmu/hw/virtio/virtio.o
CC x86_64-softmmu/target/i386/hyperv.o
CC aarch64-softmmu/hw/virtio/virtio-balloon.o
CC aarch64-softmmu/hw/virtio/vhost.o
CC aarch64-softmmu/hw/virtio/vhost-backend.o
CC aarch64-softmmu/hw/virtio/vhost-user.o
CC aarch64-softmmu/hw/virtio/vhost-vsock.o
CC aarch64-softmmu/hw/virtio/virtio-crypto.o
CC aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
CC aarch64-softmmu/hw/arm/boot.o
CC aarch64-softmmu/hw/arm/collie.o
CC aarch64-softmmu/hw/arm/exynos4_boards.o
CC aarch64-softmmu/hw/arm/gumstix.o
CC aarch64-softmmu/hw/arm/highbank.o
GEN trace/generated-helpers.c
CC aarch64-softmmu/hw/arm/digic_boards.o
CC aarch64-softmmu/hw/arm/integratorcp.o
CC aarch64-softmmu/hw/arm/mainstone.o
CC aarch64-softmmu/hw/arm/musicpal.o
CC aarch64-softmmu/hw/arm/nseries.o
CC x86_64-softmmu/trace/control-target.o
CC aarch64-softmmu/hw/arm/omap_sx1.o
CC aarch64-softmmu/hw/arm/palm.o
CC aarch64-softmmu/hw/arm/realview.o
CC x86_64-softmmu/gdbstub-xml.o
CC aarch64-softmmu/hw/arm/spitz.o
CC aarch64-softmmu/hw/arm/stellaris.o
CC aarch64-softmmu/hw/arm/tosa.o
CC aarch64-softmmu/hw/arm/versatilepb.o
CC aarch64-softmmu/hw/arm/vexpress.o
CC aarch64-softmmu/hw/arm/virt.o
CC aarch64-softmmu/hw/arm/xilinx_zynq.o
CC aarch64-softmmu/hw/arm/z2.o
CC aarch64-softmmu/hw/arm/virt-acpi-build.o
CC x86_64-softmmu/trace/generated-helpers.o
CC aarch64-softmmu/hw/arm/netduino2.o
CC aarch64-softmmu/hw/arm/sysbus-fdt.o
CC aarch64-softmmu/hw/arm/armv7m.o
CC aarch64-softmmu/hw/arm/exynos4210.o
CC aarch64-softmmu/hw/arm/pxa2xx.o
CC aarch64-softmmu/hw/arm/pxa2xx_gpio.o
CC aarch64-softmmu/hw/arm/pxa2xx_pic.o
LINK x86_64-softmmu/qemu-system-x86_64
CC aarch64-softmmu/hw/arm/digic.o
CC aarch64-softmmu/hw/arm/omap1.o
CC aarch64-softmmu/hw/arm/omap2.o
CC aarch64-softmmu/hw/arm/strongarm.o
CC aarch64-softmmu/hw/arm/allwinner-a10.o
CC aarch64-softmmu/hw/arm/cubieboard.o
CC aarch64-softmmu/hw/arm/bcm2835_peripherals.o
CC aarch64-softmmu/hw/arm/bcm2836.o
CC aarch64-softmmu/hw/arm/raspi.o
CC aarch64-softmmu/hw/arm/stm32f205_soc.o
CC aarch64-softmmu/hw/arm/xlnx-zynqmp.o
CC aarch64-softmmu/hw/arm/xlnx-zcu102.o
CC aarch64-softmmu/hw/arm/fsl-imx25.o
CC aarch64-softmmu/hw/arm/imx25_pdk.o
CC aarch64-softmmu/hw/arm/fsl-imx31.o
CC aarch64-softmmu/hw/arm/kzm.o
CC aarch64-softmmu/hw/arm/fsl-imx6.o
CC aarch64-softmmu/hw/arm/sabrelite.o
CC aarch64-softmmu/hw/arm/aspeed_soc.o
CC aarch64-softmmu/hw/arm/aspeed.o
CC aarch64-softmmu/hw/arm/mps2.o
CC aarch64-softmmu/hw/arm/msf2-soc.o
CC aarch64-softmmu/hw/arm/msf2-som.o
CC aarch64-softmmu/target/arm/arm-semi.o
CC aarch64-softmmu/target/arm/machine.o
CC aarch64-softmmu/target/arm/psci.o
CC aarch64-softmmu/target/arm/arch_dump.o
CC aarch64-softmmu/target/arm/monitor.o
CC aarch64-softmmu/target/arm/kvm-stub.o
CC aarch64-softmmu/target/arm/translate.o
CC aarch64-softmmu/target/arm/op_helper.o
CC aarch64-softmmu/target/arm/helper.o
CC aarch64-softmmu/target/arm/cpu.o
CC aarch64-softmmu/target/arm/neon_helper.o
CC aarch64-softmmu/target/arm/iwmmxt_helper.o
CC aarch64-softmmu/target/arm/gdbstub.o
CC aarch64-softmmu/target/arm/cpu64.o
CC aarch64-softmmu/target/arm/translate-a64.o
CC aarch64-softmmu/target/arm/helper-a64.o
CC aarch64-softmmu/target/arm/gdbstub64.o
CC aarch64-softmmu/target/arm/crypto_helper.o
CC aarch64-softmmu/target/arm/arm-powerctl.o
/tmp/qemu-test/src/target/arm/translate-a64.c: In function 'handle_shri_with_rndacc':
/tmp/qemu-test/src/target/arm/translate-a64.c:6392: warning: 'tcg_src_hi' may be used uninitialized in this function
/tmp/qemu-test/src/target/arm/translate-a64.c: In function 'disas_simd_scalar_two_reg_misc':
/tmp/qemu-test/src/target/arm/translate-a64.c:8119: warning: 'rmode' may be used uninitialized in this function
GEN trace/generated-helpers.c
CC aarch64-softmmu/trace/control-target.o
CC aarch64-softmmu/gdbstub-xml.o
CC aarch64-softmmu/trace/generated-helpers.o
LINK aarch64-softmmu/qemu-system-aarch64
mkdir -p dtc/libfdt
mkdir -p dtc/tests
LEX convert-dtsv0-lexer.lex.c
make[1]: flex: Command not found
BISON dtc-parser.tab.c
make[1]: bison: Command not found
LEX dtc-lexer.lex.c
make[1]: flex: Command not found
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu"
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/var"/run
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
install -c -m 0755 qemu-ga ivshmem-client ivshmem-server qemu-nbd qemu-img qemu-io scsi/qemu-pr-helper "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-ga" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/ivshmem-client" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/ivshmem-server" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-nbd" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-img" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-io" "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-pr-helper"
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/libexec"
install -c -m 0755 qemu-bridge-helper "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/libexec"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/libexec/qemu-bridge-helper"
set -e; for x in bios.bin bios-256k.bin sgabios.bin vgabios.bin vgabios-cirrus.bin vgabios-stdvga.bin vgabios-vmware.bin vgabios-qxl.bin vgabios-virtio.bin acpi-dsdt.aml ppc_rom.bin openbios-sparc32 openbios-sparc64 openbios-ppc QEMU,tcx.bin QEMU,cgthree.bin pxe-e1000.rom pxe-eepro100.rom pxe-ne2k_pci.rom pxe-pcnet.rom pxe-rtl8139.rom pxe-virtio.rom efi-e1000.rom efi-eepro100.rom efi-ne2k_pci.rom efi-pcnet.rom efi-rtl8139.rom efi-virtio.rom efi-e1000e.rom efi-vmxnet3.rom qemu-icon.bmp qemu_logo_no_text.svg bamboo.dtb petalogix-s3adsp1800.dtb petalogix-ml605.dtb multiboot.bin linuxboot.bin linuxboot_dma.bin kvmvapic.bin s390-ccw.img s390-netboot.img spapr-rtas.bin slof.bin skiboot.lid palcode-clipper u-boot.e500 qemu_vga.ndrv; do \
install -c -m 0644 /tmp/qemu-test/src/pc-bios/$x "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu"; \
done
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu/keymaps"
set -e; for x in da en-gb et fr fr-ch is lt modifiers no pt-br sv ar de en-us fi fr-be hr it lv nl pl ru th common de-ch es fo fr-ca hu ja mk nl-be pt sl tr bepo cz; do \
install -c -m 0644 /tmp/qemu-test/src/pc-bios/keymaps/$x "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu/keymaps"; \
done
install -c -m 0644 /tmp/qemu-test/build/trace-events-all "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/share/qemu/trace-events-all"
for d in x86_64-softmmu aarch64-softmmu; do \
make --no-print-directory BUILD_DIR=/tmp/qemu-test/build TARGET_DIR=$d/ -C $d install || exit 1 ; \
done
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
install -c -m 0755 qemu-system-x86_64 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-system-x86_64"
install -d -m 0755 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
install -c -m 0755 qemu-system-aarch64 "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin"
strip "/tmp/qemu-test/build/=destdir/tmp/qemu-test/install/bin/qemu-system-aarch64"
CLEANUP /var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.07.49.15568
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-r1w5vyee/src'
real 3m9.799s
user 0m5.600s
sys 0m6.051s
BUILD fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-r1w5vyee/src'
GEN /var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.10.59.23249/qemu.tar
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.10.59.23249/qemu.tar.vroot'...
done.
Checking out files: 37% (2151/5667)
Checking out files: 38% (2154/5667)
Checking out files: 39% (2211/5667)
Checking out files: 40% (2267/5667)
Checking out files: 41% (2324/5667)
Checking out files: 42% (2381/5667)
Checking out files: 43% (2437/5667)
Checking out files: 44% (2494/5667)
Checking out files: 45% (2551/5667)
Checking out files: 46% (2607/5667)
Checking out files: 47% (2664/5667)
Checking out files: 48% (2721/5667)
Checking out files: 49% (2777/5667)
Checking out files: 50% (2834/5667)
Checking out files: 51% (2891/5667)
Checking out files: 52% (2947/5667)
Checking out files: 53% (3004/5667)
Checking out files: 54% (3061/5667)
Checking out files: 55% (3117/5667)
Checking out files: 56% (3174/5667)
Checking out files: 57% (3231/5667)
Checking out files: 58% (3287/5667)
Checking out files: 59% (3344/5667)
Checking out files: 60% (3401/5667)
Checking out files: 61% (3457/5667)
Checking out files: 62% (3514/5667)
Checking out files: 63% (3571/5667)
Checking out files: 64% (3627/5667)
Checking out files: 65% (3684/5667)
Checking out files: 66% (3741/5667)
Checking out files: 67% (3797/5667)
Checking out files: 68% (3854/5667)
Checking out files: 69% (3911/5667)
Checking out files: 70% (3967/5667)
Checking out files: 71% (4024/5667)
Checking out files: 72% (4081/5667)
Checking out files: 73% (4137/5667)
Checking out files: 74% (4194/5667)
Checking out files: 75% (4251/5667)
Checking out files: 76% (4307/5667)
Checking out files: 77% (4364/5667)
Checking out files: 78% (4421/5667)
Checking out files: 79% (4477/5667)
Checking out files: 80% (4534/5667)
Checking out files: 81% (4591/5667)
Checking out files: 82% (4647/5667)
Checking out files: 83% (4704/5667)
Checking out files: 84% (4761/5667)
Checking out files: 85% (4817/5667)
Checking out files: 86% (4874/5667)
Checking out files: 87% (4931/5667)
Checking out files: 88% (4987/5667)
Checking out files: 89% (5044/5667)
Checking out files: 90% (5101/5667)
Checking out files: 91% (5157/5667)
Checking out files: 92% (5214/5667)
Checking out files: 93% (5271/5667)
Checking out files: 94% (5327/5667)
Checking out files: 95% (5384/5667)
Checking out files: 95% (5427/5667)
Checking out files: 96% (5441/5667)
Checking out files: 97% (5497/5667)
Checking out files: 98% (5554/5667)
Checking out files: 99% (5611/5667)
Checking out files: 100% (5667/5667)
Checking out files: 100% (5667/5667), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.10.59.23249/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out '558cd81bdd432769b59bff01240c44f82cfb1a9d'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.10.59.23249/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '10739aa26051a5d49d88132604539d3ed085e72e'
COPY RUNNER
RUN test-mingw in qemu:fedora
Packages installed:
PyYAML-3.11-13.fc25.x86_64
SDL-devel-1.2.15-21.fc24.x86_64
bc-1.06.95-16.fc24.x86_64
bison-3.0.4-4.fc24.x86_64
bzip2-1.0.6-21.fc25.x86_64
ccache-3.3.4-1.fc25.x86_64
clang-3.9.1-2.fc25.x86_64
findutils-4.6.0-8.fc25.x86_64
flex-2.6.0-3.fc25.x86_64
gcc-6.4.1-1.fc25.x86_64
gcc-c++-6.4.1-1.fc25.x86_64
gettext-0.19.8.1-3.fc25.x86_64
git-2.9.5-1.fc25.x86_64
glib2-devel-2.50.3-1.fc25.x86_64
hostname-3.15-8.fc25.x86_64
libaio-devel-0.3.110-6.fc24.x86_64
libfdt-devel-1.4.2-1.fc25.x86_64
make-4.1-6.fc25.x86_64
mingw32-SDL-1.2.15-7.fc24.noarch
mingw32-bzip2-1.0.6-7.fc24.noarch
mingw32-curl-7.47.0-1.fc24.noarch
mingw32-glib2-2.50.3-1.fc25.noarch
mingw32-gmp-6.1.1-1.fc25.noarch
mingw32-gnutls-3.5.5-2.fc25.noarch
mingw32-gtk2-2.24.31-2.fc25.noarch
mingw32-gtk3-3.22.17-1.fc25.noarch
mingw32-libjpeg-turbo-1.5.1-1.fc25.noarch
mingw32-libpng-1.6.27-1.fc25.noarch
mingw32-libssh2-1.4.3-5.fc24.noarch
mingw32-libtasn1-4.9-1.fc25.noarch
mingw32-nettle-3.3-1.fc25.noarch
mingw32-pixman-0.34.0-1.fc25.noarch
mingw32-pkg-config-0.28-6.fc24.x86_64
mingw64-SDL-1.2.15-7.fc24.noarch
mingw64-bzip2-1.0.6-7.fc24.noarch
mingw64-curl-7.47.0-1.fc24.noarch
mingw64-glib2-2.50.3-1.fc25.noarch
mingw64-gmp-6.1.1-1.fc25.noarch
mingw64-gnutls-3.5.5-2.fc25.noarch
mingw64-gtk2-2.24.31-2.fc25.noarch
mingw64-gtk3-3.22.17-1.fc25.noarch
mingw64-libjpeg-turbo-1.5.1-1.fc25.noarch
mingw64-libpng-1.6.27-1.fc25.noarch
mingw64-libssh2-1.4.3-5.fc24.noarch
mingw64-libtasn1-4.9-1.fc25.noarch
mingw64-nettle-3.3-1.fc25.noarch
mingw64-pixman-0.34.0-1.fc25.noarch
mingw64-pkg-config-0.28-6.fc24.x86_64
nettle-devel-3.3-1.fc25.x86_64
package python2 is not installed
perl-5.24.2-387.fc25.x86_64
pixman-devel-0.34.0-2.fc24.x86_64
sparse-0.5.0-10.fc25.x86_64
tar-1.29-3.fc25.x86_64
which-2.21-1.fc25.x86_64
zlib-devel-1.2.8-10.fc24.x86_64
Environment variables:
PACKAGES=ccache gettext git tar PyYAML sparse flex bison python2 bzip2 hostname glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel gcc gcc-c++ clang make perl which bc findutils libaio-devel nettle-devel mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1 mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2 mingw32-bzip2 mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1 mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2 mingw64-bzip2
HOSTNAME=58c3c2a0543d
MAKEFLAGS= -j8
J=8
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
FGC=f25
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
DISTTAG=f25container
FEATURES=mingw clang pyyaml dtc
DEBUG=
_=/usr/bin/env
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-debug --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0
Install prefix /tmp/qemu-test/install
BIOS directory /tmp/qemu-test/install
firmware path /tmp/qemu-test/install/share/qemu-firmware
binary directory /tmp/qemu-test/install
library directory /tmp/qemu-test/install/lib
module directory /tmp/qemu-test/install/lib
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory /tmp/qemu-test/install
local state directory queried at runtime
Windows SDK no
Source path /tmp/qemu-test/src
GIT binary git
GIT submodules
C compiler x86_64-w64-mingw32-gcc
Host C compiler cc
C++ compiler x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS rv
CFLAGS -g
QEMU_CFLAGS -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1 -I$(SRC_PATH)/dtc/libfdt -Werror -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -m64 -mcx16 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16
LDFLAGS -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g
make make
install install
python python -B
smbd /usr/sbin/smbd
module support no
host CPU x86_64
host big endian no
target list x86_64-softmmu aarch64-softmmu
gprof enabled no
sparse enabled no
strip binaries no
profiler no
static build no
SDL support yes (1.2.15)
GTK support yes (2.24.31)
GTK GL support no
VTE support no
TLS priority NORMAL
GNUTLS support yes
GNUTLS rnd yes
libgcrypt no
libgcrypt kdf no
nettle yes (3.3)
nettle kdf yes
libtasn1 yes
curses support no
virgl support no
curl support yes
mingw32 support yes
Audio drivers dsound
Block whitelist (rw)
Block whitelist (ro)
VirtFS support no
Multipath support no
VNC support yes
VNC SASL support no
VNC JPEG support yes
VNC PNG support yes
xen support no
brlapi support no
bluez support no
Documentation no
PIE no
vde support no
netmap support no
Linux AIO support no
ATTR/XATTR support no
Install blobs yes
KVM support no
HAX support yes
TCG support yes
TCG debug enabled yes
TCG interpreter no
RDMA support no
fdt support yes
preadv support no
fdatasync no
madvise no
posix_madvise no
libcap-ng support no
vhost-net support no
vhost-scsi support no
vhost-vsock support no
vhost-user support no
Trace backends simple
Trace output file trace-<pid>
spice support no
rbd support no
xfsctl support no
smartcard support no
libusb no
usb net redir no
OpenGL support no
OpenGL dmabufs no
libiscsi support no
libnfs support no
build guest agent yes
QGA VSS support no
QGA w32 disk info yes
QGA MSI support no
seccomp support no
coroutine backend win32
coroutine pool yes
debug stack usage no
crypto afalg no
GlusterFS support no
gcov gcov
gcov enabled no
TPM support yes
libssh2 support yes
TPM passthrough no
TPM emulator no
QOM debugging yes
Live block migration yes
lzo support no
snappy support no
bzip2 support yes
NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization yes
replication support yes
VxHS block device no
capstone no
mkdir -p dtc/libfdt
mkdir -p dtc/tests
GEN x86_64-softmmu/config-devices.mak.tmp
GEN aarch64-softmmu/config-devices.mak.tmp
GEN config-host.h
GEN qemu-options.def
GEN qmp-commands.h
GEN qapi-types.h
GEN qapi-visit.h
GEN qapi-event.h
GEN x86_64-softmmu/config-devices.mak
GEN qmp-marshal.c
GEN aarch64-softmmu/config-devices.mak
GEN qapi-types.c
GEN qapi-visit.c
GEN qapi-event.c
GEN qmp-introspect.h
GEN qmp-introspect.c
GEN trace/generated-tcg-tracers.h
GEN trace/generated-helpers-wrappers.h
GEN trace/generated-helpers.h
GEN trace/generated-helpers.c
GEN module_block.h
GEN ui/input-keymap-linux-to-qcode.c
GEN ui/input-keymap-qcode-to-qnum.c
GEN ui/input-keymap-qnum-to-qcode.c
GEN tests/test-qapi-types.h
GEN tests/test-qapi-visit.h
GEN tests/test-qmp-commands.h
GEN tests/test-qapi-event.h
GEN tests/test-qmp-introspect.h
GEN trace-root.h
GEN util/trace.h
GEN crypto/trace.h
GEN io/trace.h
GEN migration/trace.h
GEN block/trace.h
GEN chardev/trace.h
GEN hw/block/trace.h
GEN hw/block/dataplane/trace.h
GEN hw/char/trace.h
GEN hw/intc/trace.h
GEN hw/net/trace.h
GEN hw/virtio/trace.h
GEN hw/audio/trace.h
GEN hw/misc/trace.h
GEN hw/usb/trace.h
GEN hw/scsi/trace.h
GEN hw/nvram/trace.h
GEN hw/display/trace.h
GEN hw/input/trace.h
GEN hw/timer/trace.h
GEN hw/dma/trace.h
GEN hw/sparc/trace.h
GEN hw/sd/trace.h
GEN hw/isa/trace.h
GEN hw/mem/trace.h
GEN hw/i386/trace.h
GEN hw/i386/xen/trace.h
GEN hw/9pfs/trace.h
GEN hw/ppc/trace.h
GEN hw/pci/trace.h
GEN hw/s390x/trace.h
GEN hw/vfio/trace.h
GEN hw/acpi/trace.h
GEN hw/arm/trace.h
GEN hw/alpha/trace.h
GEN hw/xen/trace.h
GEN hw/ide/trace.h
GEN ui/trace.h
GEN audio/trace.h
GEN net/trace.h
GEN target/arm/trace.h
GEN target/i386/trace.h
GEN target/mips/trace.h
GEN target/sparc/trace.h
GEN target/s390x/trace.h
GEN target/ppc/trace.h
GEN qom/trace.h
GEN linux-user/trace.h
GEN qapi/trace.h
GEN accel/tcg/trace.h
GEN accel/kvm/trace.h
GEN nbd/trace.h
GEN scsi/trace.h
GEN trace-root.c
GEN util/trace.c
GEN crypto/trace.c
GEN io/trace.c
GEN migration/trace.c
GEN block/trace.c
GEN chardev/trace.c
GEN hw/block/trace.c
GEN hw/block/dataplane/trace.c
GEN hw/char/trace.c
GEN hw/intc/trace.c
GEN hw/net/trace.c
GEN hw/virtio/trace.c
GEN hw/audio/trace.c
GEN hw/misc/trace.c
GEN hw/usb/trace.c
GEN hw/scsi/trace.c
GEN hw/nvram/trace.c
GEN hw/display/trace.c
GEN hw/input/trace.c
GEN hw/timer/trace.c
GEN hw/dma/trace.c
GEN hw/sparc/trace.c
GEN hw/sd/trace.c
GEN hw/isa/trace.c
GEN hw/mem/trace.c
GEN hw/i386/trace.c
GEN hw/i386/xen/trace.c
GEN hw/9pfs/trace.c
GEN hw/ppc/trace.c
GEN hw/pci/trace.c
GEN hw/s390x/trace.c
GEN hw/vfio/trace.c
GEN hw/acpi/trace.c
GEN hw/arm/trace.c
GEN hw/alpha/trace.c
GEN hw/ide/trace.c
GEN hw/xen/trace.c
GEN ui/trace.c
GEN audio/trace.c
GEN net/trace.c
GEN target/arm/trace.c
GEN target/i386/trace.c
GEN target/mips/trace.c
GEN target/sparc/trace.c
GEN target/s390x/trace.c
GEN target/ppc/trace.c
GEN qom/trace.c
GEN linux-user/trace.c
GEN qapi/trace.c
GEN accel/tcg/trace.c
GEN accel/kvm/trace.c
GEN nbd/trace.c
GEN scsi/trace.c
GEN config-all-devices.mak
DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
DEP /tmp/qemu-test/src/dtc/tests/trees.S
DEP /tmp/qemu-test/src/dtc/tests/testutils.c
DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
DEP /tmp/qemu-test/src/dtc/tests/check_path.c
DEP /tmp/qemu-test/src/dtc/tests/overlay.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
DEP /tmp/qemu-test/src/dtc/tests/incbin.c
DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
DEP /tmp/qemu-test/src/dtc/tests/path-references.c
DEP /tmp/qemu-test/src/dtc/tests/references.c
DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
DEP /tmp/qemu-test/src/dtc/tests/del_node.c
DEP /tmp/qemu-test/src/dtc/tests/del_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop.c
DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/set_name.c
DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
DEP /tmp/qemu-test/src/dtc/tests/notfound.c
DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
DEP /tmp/qemu-test/src/dtc/tests/get_path.c
DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/getprop.c
DEP /tmp/qemu-test/src/dtc/tests/get_name.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
DEP /tmp/qemu-test/src/dtc/tests/find_property.c
DEP /tmp/qemu-test/src/dtc/tests/root_node.c
DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
DEP /tmp/qemu-test/src/dtc/util.c
DEP /tmp/qemu-test/src/dtc/fdtput.c
DEP /tmp/qemu-test/src/dtc/fdtget.c
DEP /tmp/qemu-test/src/dtc/fdtdump.c
LEX convert-dtsv0-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/srcpos.c
BISON dtc-parser.tab.c
LEX dtc-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/treesource.c
DEP /tmp/qemu-test/src/dtc/livetree.c
DEP /tmp/qemu-test/src/dtc/fstree.c
DEP /tmp/qemu-test/src/dtc/flattree.c
DEP /tmp/qemu-test/src/dtc/dtc.c
DEP /tmp/qemu-test/src/dtc/data.c
DEP /tmp/qemu-test/src/dtc/checks.c
DEP convert-dtsv0-lexer.lex.c
DEP dtc-parser.tab.c
DEP dtc-lexer.lex.c
CHK version_gen.h
UPD version_gen.h
DEP /tmp/qemu-test/src/dtc/util.c
CC libfdt/fdt.o
CC libfdt/fdt_ro.o
CC libfdt/fdt_sw.o
CC libfdt/fdt_rw.o
CC libfdt/fdt_wip.o
CC libfdt/fdt_strerror.o
CC libfdt/fdt_empty_tree.o
CC libfdt/fdt_overlay.o
CC libfdt/fdt_addresses.o
AR libfdt/libfdt.a
x86_64-w64-mingw32-ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
RC version.o
mkdir -p dtc/libfdt
mkdir -p dtc/tests
GEN qga/qapi-generated/qga-qapi-types.h
GEN qga/qapi-generated/qga-qapi-visit.h
GEN qga/qapi-generated/qga-qapi-visit.c
GEN qga/qapi-generated/qga-qmp-commands.h
CC qmp-introspect.o
CC qapi-types.o
GEN qga/qapi-generated/qga-qapi-types.c
GEN qga/qapi-generated/qga-qmp-marshal.c
CC qapi-visit.o
CC qapi-event.o
CC qapi/qapi-visit-core.o
CC qapi/qapi-dealloc-visitor.o
CC qapi/qobject-input-visitor.o
CC qapi/qobject-output-visitor.o
CC qapi/qmp-registry.o
CC qapi/qmp-dispatch.o
CC qapi/string-input-visitor.o
CC qapi/string-output-visitor.o
CC qapi/opts-visitor.o
CC qapi/qapi-clone-visitor.o
CC qapi/qapi-util.o
CC qobject/qnull.o
CC qapi/qmp-event.o
CC qobject/qstring.o
CC qobject/qdict.o
CC qobject/qnum.o
CC qobject/qlist.o
CC qobject/qbool.o
CC qobject/qlit.o
CC qobject/qjson.o
CC qobject/qobject.o
CC qobject/json-lexer.o
CC qobject/json-streamer.o
CC qobject/json-parser.o
CC trace/simple.o
CC trace/control.o
CC trace/qmp.o
CC util/osdep.o
CC util/cutils.o
CC util/unicode.o
CC util/qemu-timer-common.o
CC util/bufferiszero.o
CC util/lockcnt.o
CC util/aiocb.o
CC util/async.o
CC util/thread-pool.o
CC util/qemu-timer.o
CC util/iohandler.o
CC util/main-loop.o
CC util/aio-win32.o
CC util/event_notifier-win32.o
CC util/oslib-win32.o
CC util/qemu-thread-win32.o
CC util/envlist.o
CC util/path.o
CC util/host-utils.o
CC util/bitmap.o
CC util/module.o
CC util/bitops.o
CC util/hbitmap.o
CC util/fifo8.o
CC util/acl.o
CC util/cacheinfo.o
CC util/error.o
CC util/qemu-error.o
CC util/iov.o
CC util/id.o
CC util/qemu-config.o
CC util/qemu-sockets.o
CC util/uri.o
CC util/notify.o
CC util/qemu-option.o
CC util/qemu-progress.o
CC util/keyval.o
CC util/hexdump.o
CC util/crc32c.o
CC util/uuid.o
CC util/throttle.o
CC util/getauxval.o
CC util/readline.o
CC util/rcu.o
CC util/qemu-coroutine.o
CC util/qemu-coroutine-lock.o
CC util/qemu-coroutine-io.o
CC util/qemu-coroutine-sleep.o
CC util/coroutine-win32.o
CC util/buffer.o
CC util/timed-average.o
CC util/base64.o
CC util/log.o
CC util/pagesize.o
CC util/qdist.o
CC util/qht.o
CC util/range.o
CC util/stats64.o
CC util/systemd.o
CC trace-root.o
CC util/trace.o
CC crypto/trace.o
CC io/trace.o
CC migration/trace.o
CC block/trace.o
CC chardev/trace.o
CC hw/block/trace.o
CC hw/block/dataplane/trace.o
CC hw/char/trace.o
CC hw/intc/trace.o
CC hw/virtio/trace.o
CC hw/net/trace.o
CC hw/audio/trace.o
CC hw/misc/trace.o
CC hw/usb/trace.o
CC hw/scsi/trace.o
CC hw/nvram/trace.o
CC hw/display/trace.o
CC hw/input/trace.o
CC hw/timer/trace.o
CC hw/sparc/trace.o
CC hw/dma/trace.o
CC hw/sd/trace.o
CC hw/isa/trace.o
CC hw/mem/trace.o
CC hw/i386/trace.o
CC hw/i386/xen/trace.o
CC hw/9pfs/trace.o
CC hw/ppc/trace.o
CC hw/pci/trace.o
CC hw/s390x/trace.o
CC hw/vfio/trace.o
CC hw/acpi/trace.o
CC hw/arm/trace.o
CC hw/alpha/trace.o
CC hw/xen/trace.o
CC hw/ide/trace.o
CC ui/trace.o
CC audio/trace.o
CC net/trace.o
CC target/arm/trace.o
CC target/i386/trace.o
CC target/mips/trace.o
CC target/sparc/trace.o
CC target/s390x/trace.o
CC target/ppc/trace.o
CC qom/trace.o
CC linux-user/trace.o
CC qapi/trace.o
CC accel/tcg/trace.o
CC accel/kvm/trace.o
CC nbd/trace.o
CC scsi/trace.o
CC crypto/pbkdf-stub.o
CC stubs/arch-query-cpu-def.o
CC stubs/arch-query-cpu-model-expansion.o
CC stubs/arch-query-cpu-model-comparison.o
CC stubs/bdrv-next-monitor-owned.o
CC stubs/arch-query-cpu-model-baseline.o
CC stubs/blk-commit-all.o
CC stubs/blockdev-close-all-bdrv-states.o
CC stubs/clock-warp.o
CC stubs/cpu-get-clock.o
CC stubs/cpu-get-icount.o
CC stubs/dump.o
CC stubs/error-printf.o
CC stubs/gdbstub.o
CC stubs/fdset.o
CC stubs/get-vm-name.o
CC stubs/iothread.o
CC stubs/iothread-lock.o
CC stubs/is-daemonized.o
CC stubs/machine-init-done.o
CC stubs/migr-blocker.o
CC stubs/change-state-handler.o
CC stubs/monitor.o
CC stubs/notify-event.o
CC stubs/qtest.o
CC stubs/replay.o
CC stubs/runstate-check.o
CC stubs/set-fd-handler.o
CC stubs/slirp.o
CC stubs/sysbus.o
CC stubs/tpm.o
CC stubs/trace-control.o
CC stubs/uuid.o
CC stubs/vm-stop.o
CC stubs/vmstate.o
CC stubs/qmp_pc_dimm.o
CC stubs/fd-register.o
CC stubs/target-get-monitor-def.o
CC stubs/target-monitor-defs.o
CC stubs/pc_madt_cpu_entry.o
CC stubs/vmgenid.o
CC stubs/xen-common.o
CC stubs/xen-hvm.o
CC stubs/pci-host-piix.o
GEN qemu-img-cmds.h
CC block.o
CC blockjob.o
CC qemu-io-cmds.o
CC replication.o
CC block/raw-format.o
CC block/qcow.o
CC block/vdi.o
CC block/cloop.o
CC block/vmdk.o
CC block/bochs.o
CC block/vpc.o
CC block/vvfat.o
CC block/dmg.o
CC block/qcow2.o
CC block/qcow2-refcount.o
CC block/qcow2-cluster.o
CC block/qcow2-snapshot.o
CC block/qcow2-cache.o
CC block/qcow2-bitmap.o
CC block/qed.o
CC block/qed-l2-cache.o
CC block/qed-table.o
CC block/qed-cluster.o
CC block/qed-check.o
CC block/vhdx.o
CC block/vhdx-log.o
CC block/vhdx-endian.o
CC block/quorum.o
CC block/parallels.o
CC block/blkdebug.o
CC block/blkverify.o
CC block/blkreplay.o
CC block/block-backend.o
CC block/snapshot.o
CC block/qapi.o
CC block/file-win32.o
CC block/win32-aio.o
CC block/null.o
CC block/io.o
CC block/throttle-groups.o
CC block/commit.o
CC block/mirror.o
CC block/nbd.o
CC block/nbd-client.o
CC block/sheepdog.o
CC block/dirty-bitmap.o
CC block/accounting.o
CC block/write-threshold.o
CC block/backup.o
CC block/replication.o
CC block/throttle.o
CC block/crypto.o
CC nbd/server.o
CC nbd/client.o
CC nbd/common.o
CC scsi/utils.o
CC block/curl.o
CC block/ssh.o
CC block/dmg-bz2.o
CC crypto/init.o
CC crypto/hash.o
CC crypto/hash-nettle.o
CC crypto/hmac.o
CC crypto/hmac-nettle.o
CC crypto/aes.o
CC crypto/desrfb.o
CC crypto/cipher.o
CC crypto/tlscreds.o
CC crypto/tlscredsanon.o
CC crypto/tlscredsx509.o
CC crypto/tlssession.o
CC crypto/secret.o
CC crypto/random-gnutls.o
CC crypto/pbkdf.o
CC crypto/pbkdf-nettle.o
CC crypto/ivgen.o
CC crypto/ivgen-essiv.o
CC crypto/ivgen-plain.o
CC crypto/ivgen-plain64.o
CC crypto/xts.o
CC crypto/afsplit.o
CC crypto/block.o
CC crypto/block-qcow.o
CC crypto/block-luks.o
CC io/channel.o
CC io/channel-buffer.o
CC io/channel-command.o
CC io/channel-file.o
CC io/channel-tls.o
CC io/channel-socket.o
CC io/channel-watch.o
CC io/channel-websock.o
CC io/channel-util.o
CC io/dns-resolver.o
CC io/task.o
CC qom/object.o
CC qom/container.o
CC qom/object_interfaces.o
CC qom/qom-qobject.o
CC qemu-io.o
CC blockdev.o
CC blockdev-nbd.o
CC bootdevice.o
CC iothread.o
CC qdev-monitor.o
CC device-hotplug.o
CC os-win32.o
CC bt-host.o
CC dma-helpers.o
CC bt-vhci.o
CC vl.o
CC tpm.o
CC device_tree.o
CC qmp-marshal.o
CC qmp.o
CC cpus-common.o
CC hmp.o
CC audio/audio.o
CC audio/wavaudio.o
CC audio/noaudio.o
CC audio/mixeng.o
CC audio/sdlaudio.o
CC audio/dsoundaudio.o
CC audio/audio_win_int.o
CC audio/wavcapture.o
CC backends/rng.o
CC backends/rng-egd.o
CC backends/tpm.o
CC backends/hostmem.o
CC backends/hostmem-ram.o
CC backends/cryptodev.o
CC backends/cryptodev-builtin.o
CC block/stream.o
CC chardev/msmouse.o
CC chardev/wctablet.o
CC chardev/testdev.o
CC disas/arm.o
CXX disas/arm-a64.o
CC disas/i386.o
CXX disas/libvixl/vixl/utils.o
CXX disas/libvixl/vixl/compiler-intrinsics.o
CXX disas/libvixl/vixl/a64/instructions-a64.o
CXX disas/libvixl/vixl/a64/decoder-a64.o
CXX disas/libvixl/vixl/a64/disasm-a64.o
CC hw/acpi/core.o
CC hw/acpi/pcihp.o
CC hw/acpi/piix4.o
CC hw/acpi/ich9.o
CC hw/acpi/tco.o
CC hw/acpi/cpu_hotplug.o
CC hw/acpi/memory_hotplug.o
CC hw/acpi/cpu.o
CC hw/acpi/nvdimm.o
CC hw/acpi/vmgenid.o
CC hw/acpi/acpi_interface.o
CC hw/acpi/bios-linker-loader.o
CC hw/acpi/aml-build.o
CC hw/acpi/ipmi.o
CC hw/acpi/acpi-stub.o
CC hw/acpi/ipmi-stub.o
CC hw/audio/sb16.o
CC hw/audio/es1370.o
CC hw/audio/ac97.o
CC hw/audio/fmopl.o
CC hw/audio/adlib.o
CC hw/audio/gus.o
CC hw/audio/gusemu_mixer.o
CC hw/audio/gusemu_hal.o
CC hw/audio/cs4231a.o
CC hw/audio/intel-hda.o
CC hw/audio/hda-codec.o
CC hw/audio/wm8750.o
CC hw/audio/pcspk.o
CC hw/audio/pl041.o
CC hw/audio/lm4549.o
CC hw/audio/marvell_88w8618.o
CC hw/audio/soundhw.o
CC hw/block/cdrom.o
CC hw/block/block.o
CC hw/block/hd-geometry.o
CC hw/block/fdc.o
CC hw/block/m25p80.o
CC hw/block/nand.o
CC hw/block/pflash_cfi01.o
CC hw/block/pflash_cfi02.o
CC hw/block/ecc.o
CC hw/block/onenand.o
CC hw/block/nvme.o
CC hw/bt/core.o
CC hw/bt/l2cap.o
CC hw/bt/sdp.o
CC hw/bt/hci.o
CC hw/bt/hid.o
CC hw/bt/hci-csr.o
CC hw/char/ipoctal232.o
CC hw/char/parallel.o
CC hw/char/pl011.o
CC hw/char/serial.o
CC hw/char/serial-isa.o
CC hw/char/serial-pci.o
CC hw/char/virtio-console.o
CC hw/char/cadence_uart.o
CC hw/char/debugcon.o
CC hw/char/cmsdk-apb-uart.o
CC hw/char/imx_serial.o
CC hw/core/qdev.o
CC hw/core/qdev-properties.o
CC hw/core/bus.o
CC hw/core/reset.o
CC hw/core/fw-path-provider.o
CC hw/core/irq.o
CC hw/core/hotplug.o
CC hw/core/nmi.o
CC hw/core/ptimer.o
CC hw/core/sysbus.o
CC hw/core/machine.o
CC hw/core/loader.o
CC hw/core/qdev-properties-system.o
CC hw/core/register.o
CC hw/core/platform-bus.o
CC hw/core/or-irq.o
CC hw/cpu/core.o
CC hw/display/ads7846.o
CC hw/display/cirrus_vga.o
CC hw/display/pl110.o
CC hw/display/ssd0303.o
CC hw/display/ssd0323.o
CC hw/display/vga-isa.o
CC hw/display/vga-pci.o
CC hw/display/vmware_vga.o
CC hw/display/blizzard.o
CC hw/display/exynos4210_fimd.o
CC hw/display/framebuffer.o
CC hw/display/tc6393xb.o
CC hw/dma/pl080.o
CC hw/dma/pl330.o
CC hw/dma/i8257.o
CC hw/dma/xlnx-zynq-devcfg.o
CC hw/gpio/max7310.o
CC hw/gpio/pl061.o
CC hw/gpio/zaurus.o
CC hw/gpio/gpio_key.o
CC hw/i2c/core.o
CC hw/i2c/smbus.o
CC hw/i2c/smbus_eeprom.o
CC hw/i2c/i2c-ddc.o
CC hw/i2c/versatile_i2c.o
CC hw/i2c/smbus_ich9.o
CC hw/i2c/pm_smbus.o
CC hw/i2c/bitbang_i2c.o
CC hw/i2c/exynos4210_i2c.o
CC hw/i2c/imx_i2c.o
CC hw/i2c/aspeed_i2c.o
CC hw/ide/atapi.o
CC hw/ide/core.o
CC hw/ide/qdev.o
CC hw/ide/pci.o
CC hw/ide/isa.o
CC hw/ide/piix.o
CC hw/ide/microdrive.o
CC hw/ide/ahci.o
CC hw/ide/ich.o
CC hw/ide/ahci-allwinner.o
CC hw/input/hid.o
CC hw/input/lm832x.o
CC hw/input/pckbd.o
CC hw/input/pl050.o
CC hw/input/ps2.o
CC hw/input/stellaris_input.o
CC hw/input/vmmouse.o
CC hw/input/tsc2005.o
CC hw/input/virtio-input.o
CC hw/input/virtio-input-hid.o
CC hw/intc/i8259_common.o
CC hw/intc/i8259.o
CC hw/intc/pl190.o
CC hw/intc/imx_avic.o
CC hw/intc/realview_gic.o
CC hw/intc/ioapic_common.o
CC hw/intc/arm_gic_common.o
CC hw/intc/arm_gic.o
CC hw/intc/arm_gicv2m.o
CC hw/intc/arm_gicv3_common.o
CC hw/intc/arm_gicv3.o
CC hw/intc/arm_gicv3_dist.o
CC hw/intc/arm_gicv3_its_common.o
CC hw/intc/arm_gicv3_redist.o
CC hw/intc/intc.o
CC hw/ipack/ipack.o
CC hw/ipack/tpci200.o
CC hw/ipmi/ipmi.o
CC hw/ipmi/ipmi_bmc_sim.o
CC hw/ipmi/ipmi_bmc_extern.o
CC hw/ipmi/isa_ipmi_kcs.o
CC hw/ipmi/isa_ipmi_bt.o
CC hw/isa/isa-bus.o
CC hw/isa/apm.o
CC hw/mem/pc-dimm.o
CC hw/mem/nvdimm.o
CC hw/misc/applesmc.o
CC hw/misc/max111x.o
CC hw/misc/tmp105.o
CC hw/misc/tmp421.o
CC hw/misc/debugexit.o
CC hw/misc/sga.o
CC hw/misc/pc-testdev.o
CC hw/misc/pci-testdev.o
CC hw/misc/edu.o
CC hw/misc/unimp.o
CC hw/misc/vmcoreinfo.o
CC hw/misc/arm_l2x0.o
CC hw/misc/arm_integrator_debug.o
CC hw/misc/a9scu.o
CC hw/misc/arm11scu.o
CC hw/net/ne2000.o
CC hw/net/eepro100.o
CC hw/net/pcnet-pci.o
CC hw/net/pcnet.o
CC hw/net/e1000.o
CC hw/net/e1000x_common.o
CC hw/net/net_tx_pkt.o
CC hw/net/net_rx_pkt.o
CC hw/net/e1000e.o
CC hw/net/e1000e_core.o
CC hw/net/rtl8139.o
CC hw/net/vmxnet3.o
CC hw/net/smc91c111.o
CC hw/net/lan9118.o
CC hw/net/ne2000-isa.o
CC hw/net/xgmac.o
CC hw/net/allwinner_emac.o
CC hw/net/imx_fec.o
CC hw/net/cadence_gem.o
CC hw/net/stellaris_enet.o
CC hw/net/ftgmac100.o
CC hw/net/rocker/rocker.o
CC hw/net/rocker/rocker_fp.o
CC hw/net/rocker/rocker_desc.o
CC hw/net/rocker/rocker_world.o
CC hw/net/rocker/rocker_of_dpa.o
CC hw/nvram/eeprom93xx.o
CC hw/nvram/fw_cfg.o
CC hw/nvram/chrp_nvram.o
CC hw/pci-bridge/pci_bridge_dev.o
CC hw/pci-bridge/pcie_root_port.o
CC hw/pci-bridge/gen_pcie_root_port.o
CC hw/pci-bridge/pcie_pci_bridge.o
CC hw/pci-bridge/pci_expander_bridge.o
CC hw/pci-bridge/xio3130_upstream.o
CC hw/pci-bridge/xio3130_downstream.o
CC hw/pci-bridge/ioh3420.o
CC hw/pci-bridge/i82801b11.o
CC hw/pci-host/pam.o
CC hw/pci-host/versatile.o
CC hw/pci-host/piix.o
CC hw/pci-host/q35.o
CC hw/pci-host/gpex.o
CC hw/pci/pci.o
CC hw/pci/pci_bridge.o
CC hw/pci/msix.o
CC hw/pci/msi.o
CC hw/pci/shpc.o
CC hw/pci/slotid_cap.o
CC hw/pci/pci_host.o
CC hw/pci/pcie_host.o
CC hw/pci/pcie.o
CC hw/pci/pcie_aer.o
CC hw/pci/pcie_port.o
CC hw/pcmcia/pcmcia.o
CC hw/pci/pci-stub.o
CC hw/scsi/scsi-disk.o
CC hw/scsi/scsi-generic.o
CC hw/scsi/scsi-bus.o
CC hw/scsi/lsi53c895a.o
CC hw/scsi/mptsas.o
CC hw/scsi/mptconfig.o
CC hw/scsi/mptendian.o
CC hw/scsi/megasas.o
CC hw/scsi/vmw_pvscsi.o
CC hw/scsi/esp.o
CC hw/scsi/esp-pci.o
CC hw/sd/pl181.o
CC hw/sd/ssi-sd.o
CC hw/sd/sd.o
CC hw/sd/core.o
CC hw/sd/sdhci.o
CC hw/smbios/smbios.o
CC hw/smbios/smbios_type_38.o
CC hw/smbios/smbios-stub.o
CC hw/smbios/smbios_type_38-stub.o
CC hw/ssi/pl022.o
CC hw/ssi/ssi.o
CC hw/ssi/xilinx_spips.o
CC hw/ssi/aspeed_smc.o
CC hw/ssi/stm32f2xx_spi.o
CC hw/ssi/mss-spi.o
CC hw/timer/arm_timer.o
CC hw/timer/arm_mptimer.o
CC hw/timer/armv7m_systick.o
CC hw/timer/a9gtimer.o
CC hw/timer/cadence_ttc.o
CC hw/timer/ds1338.o
CC hw/timer/hpet.o
CC hw/timer/i8254_common.o
CC hw/timer/i8254.o
CC hw/timer/pl031.o
CC hw/timer/twl92230.o
CC hw/timer/imx_epit.o
CC hw/timer/imx_gpt.o
CC hw/timer/stm32f2xx_timer.o
CC hw/timer/aspeed_timer.o
CC hw/timer/cmsdk-apb-timer.o
CC hw/timer/mss-timer.o
CC hw/tpm/tpm_tis.o
CC hw/usb/core.o
CC hw/usb/combined-packet.o
CC hw/usb/bus.o
CC hw/usb/libhw.o
CC hw/usb/desc.o
CC hw/usb/desc-msos.o
CC hw/usb/hcd-uhci.o
CC hw/usb/hcd-ohci.o
CC hw/usb/hcd-ehci.o
CC hw/usb/hcd-ehci-pci.o
CC hw/usb/hcd-ehci-sysbus.o
CC hw/usb/hcd-xhci.o
CC hw/usb/hcd-xhci-nec.o
CC hw/usb/hcd-musb.o
CC hw/usb/dev-hub.o
CC hw/usb/dev-hid.o
CC hw/usb/dev-wacom.o
CC hw/usb/dev-storage.o
CC hw/usb/dev-uas.o
CC hw/usb/dev-audio.o
CC hw/usb/dev-serial.o
CC hw/usb/dev-network.o
CC hw/usb/dev-bluetooth.o
CC hw/usb/dev-smartcard-reader.o
CC hw/usb/host-stub.o
CC hw/virtio/virtio-rng.o
CC hw/virtio/virtio-pci.o
CC hw/virtio/virtio-bus.o
CC hw/virtio/vhost-pci-slave.o
CC hw/virtio/virtio-mmio.o
CC hw/virtio/vhost-stub.o
CC hw/watchdog/watchdog.o
CC hw/watchdog/wdt_i6300esb.o
CC hw/watchdog/wdt_ib700.o
CC hw/watchdog/wdt_aspeed.o
CC migration/migration.o
CC migration/socket.o
CC migration/fd.o
CC migration/exec.o
CC migration/tls.o
CC migration/channel.o
CC migration/savevm.o
CC migration/colo-comm.o
CC migration/colo-failover.o
CC migration/colo.o
CC migration/vmstate.o
CC migration/vmstate-types.o
CC migration/page_cache.o
CC migration/qemu-file.o
CC migration/global_state.o
CC migration/xbzrle.o
CC migration/qemu-file-channel.o
CC migration/postcopy-ram.o
CC migration/qjson.o
CC migration/block.o
CC net/net.o
CC net/queue.o
CC net/checksum.o
CC net/hub.o
CC net/util.o
CC net/socket.o
CC net/dump.o
CC net/eth.o
CC net/slirp.o
CC net/filter.o
CC net/filter-buffer.o
CC net/filter-mirror.o
CC net/colo-compare.o
CC net/colo.o
CC net/filter-rewriter.o
CC net/filter-replay.o
CC net/tap-win32.o
CC qom/cpu.o
In file included from /tmp/qemu-test/src/include/hw/virtio/vhost-pci-slave.h:4:0,
from /tmp/qemu-test/src/hw/virtio/vhost-pci-slave.c:22:
/tmp/qemu-test/src/linux-headers/linux/vhost.h:13:25: fatal error: linux/types.h: No such file or directory
#include <linux/types.h>
^
compilation terminated.
make: *** [hw/virtio/vhost-pci-slave.o] Error 1
make: *** Waiting for unfinished jobs....
/tmp/qemu-test/src/rules.mak:66: recipe for target 'hw/virtio/vhost-pci-slave.o' failed
CC replay/replay.o
Traceback (most recent call last):
File "./tests/docker/docker.py", line 407, in <module>
sys.exit(main())
File "./tests/docker/docker.py", line 404, in main
return args.cmdobj.run(args, argv)
File "./tests/docker/docker.py", line 261, in run
return Docker().run(argv, args.keep, quiet=args.quiet)
File "./tests/docker/docker.py", line 229, in run
quiet=quiet)
File "./tests/docker/docker.py", line 147, in _do_check
return subprocess.check_call(self._command + cmd, **kwargs)
File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=53487d16d97211e7a09a52540069c830', '-u', '0', '--security-opt', 'seccomp=unconfined', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-r1w5vyee/src/docker-src.2017-12-04-23.10.59.23249:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2
make[1]: *** [tests/docker/Makefile.include:129: docker-run] Error 1
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-r1w5vyee/src'
make: *** [tests/docker/Makefile.include:163: docker-run-test-mingw@fedora] Error 2
real 2m41.899s
user 0m4.831s
sys 0m4.790s
=== OUTPUT END ===
Test command exited with code: 2
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
On 2017年12月05日 11:33, Wei Wang wrote: > Vhost-pci is a point-to-point based inter-VM communication solution. This > patch series implements the vhost-pci-net device setup and emulation. The > device is implemented as a virtio device, and it is set up via the > vhost-user protocol to get the neessary info (e.g the memory info of the > remote VM, vring info). > > Currently, only the fundamental functions are implemented. More features, > such as MQ and live migration, will be updated in the future. > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > http://dpdk.org/ml/archives/dev/2017-November/082615.html > > v2->v3 changes: > 1) static device creation: instead of creating and hot-plugging the > device when receiving a vhost-user msg, the device is not created > via the qemu booting command line. > 2) remove vqs: rq and ctrlq are removed in this version. > - receive vq: the receive vq is not needed anymore. The PMD directly > shares the remote txq and rxq - grab from remote txq to > receive packets, and put to rxq to send packets. > - ctrlq: the ctrlq is replaced by the first 4KB metadata area of the > device Bar-2. > 3) simpler implementation: the entire implementation has been tailored > from ~1800 LOC to ~850 LOC. Hi: Any performance numbers you can share? Thanks
On 12/05/2017 03:01 PM, Jason Wang wrote: > > > On 2017年12月05日 11:33, Wei Wang wrote: >> Vhost-pci is a point-to-point based inter-VM communication solution. >> This >> patch series implements the vhost-pci-net device setup and emulation. >> The >> device is implemented as a virtio device, and it is set up via the >> vhost-user protocol to get the neessary info (e.g the memory info of the >> remote VM, vring info). >> >> Currently, only the fundamental functions are implemented. More >> features, >> such as MQ and live migration, will be updated in the future. >> >> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >> http://dpdk.org/ml/archives/dev/2017-November/082615.html >> >> v2->v3 changes: >> 1) static device creation: instead of creating and hot-plugging the >> device when receiving a vhost-user msg, the device is not created >> via the qemu booting command line. >> 2) remove vqs: rq and ctrlq are removed in this version. >> - receive vq: the receive vq is not needed anymore. The PMD >> directly >> shares the remote txq and rxq - grab from remote >> txq to >> receive packets, and put to rxq to send packets. >> - ctrlq: the ctrlq is replaced by the first 4KB metadata area of >> the >> device Bar-2. >> 3) simpler implementation: the entire implementation has been tailored >> from ~1800 LOC to ~850 LOC. > > Hi: > > Any performance numbers you can share? > Hi Jason, Performance testing and tuning on the data plane is in progress (btw, that wouldn't affect the device part patches). If possible, could we start the device part patch review in the meantime? Best, Wei
On 2017年12月05日 15:15, Wei Wang wrote: > On 12/05/2017 03:01 PM, Jason Wang wrote: >> >> >> On 2017年12月05日 11:33, Wei Wang wrote: >>> Vhost-pci is a point-to-point based inter-VM communication solution. >>> This >>> patch series implements the vhost-pci-net device setup and >>> emulation. The >>> device is implemented as a virtio device, and it is set up via the >>> vhost-user protocol to get the neessary info (e.g the memory info of >>> the >>> remote VM, vring info). >>> >>> Currently, only the fundamental functions are implemented. More >>> features, >>> such as MQ and live migration, will be updated in the future. >>> >>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >>> http://dpdk.org/ml/archives/dev/2017-November/082615.html >>> >>> v2->v3 changes: >>> 1) static device creation: instead of creating and hot-plugging the >>> device when receiving a vhost-user msg, the device is not created >>> via the qemu booting command line. >>> 2) remove vqs: rq and ctrlq are removed in this version. >>> - receive vq: the receive vq is not needed anymore. The PMD >>> directly >>> shares the remote txq and rxq - grab from remote >>> txq to >>> receive packets, and put to rxq to send packets. >>> - ctrlq: the ctrlq is replaced by the first 4KB metadata area >>> of the >>> device Bar-2. >>> 3) simpler implementation: the entire implementation has been tailored >>> from ~1800 LOC to ~850 LOC. >> >> Hi: >> >> Any performance numbers you can share? >> > > Hi Jason, > > Performance testing and tuning on the data plane is in progress (btw, > that wouldn't affect the device part patches). > If possible, could we start the device part patch review in the meantime? > > Best, > Wei > Hi Wei: Will do, but basically, the cover lacks of the motivation for vhost-pci and I want to see some numbers first since I suspect it can over-perform exist data-path. Thanks
> -----Original Message----- > From: Jason Wang [mailto:jasowang@redhat.com] > Sent: Tuesday, 05 December, 2017 9:19 AM > To: Wei Wang; virtio-dev@lists.oasis-open.org; qemu-devel@nongnu.org; > mst@redhat.com; marcandre.lureau@redhat.com; stefanha@redhat.com; > pbonzini@redhat.com > Cc: jan.kiszka@siemens.com; Avi Cohen (A); zhiyong.yang@intel.com > Subject: Re: [virtio-dev] [PATCH v3 0/7] Vhost-pci for inter-VM communication > > > > On 2017年12月05日 15:15, Wei Wang wrote: > > On 12/05/2017 03:01 PM, Jason Wang wrote: > >> > >> > >> On 2017年12月05日 11:33, Wei Wang wrote: > >>> Vhost-pci is a point-to-point based inter-VM communication solution. > >>> This > >>> patch series implements the vhost-pci-net device setup and > >>> emulation. The device is implemented as a virtio device, and it is > >>> set up via the vhost-user protocol to get the neessary info (e.g the > >>> memory info of the remote VM, vring info). > >>> > >>> Currently, only the fundamental functions are implemented. More > >>> features, such as MQ and live migration, will be updated in the > >>> future. > >>> > >>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > >>> http://dpdk.org/ml/archives/dev/2017-November/082615.html > >>> > >>> v2->v3 changes: > >>> 1) static device creation: instead of creating and hot-plugging the > >>> device when receiving a vhost-user msg, the device is not > >>> created > >>> via the qemu booting command line. > >>> 2) remove vqs: rq and ctrlq are removed in this version. > >>> - receive vq: the receive vq is not needed anymore. The PMD > >>> directly > >>> shares the remote txq and rxq - grab from remote > >>> txq to > >>> receive packets, and put to rxq to send packets. > >>> - ctrlq: the ctrlq is replaced by the first 4KB metadata area > >>> of the > >>> device Bar-2. > >>> 3) simpler implementation: the entire implementation has been > >>> tailored > >>> from ~1800 LOC to ~850 LOC. > >> > >> Hi: > >> > >> Any performance numbers you can share? > >> > > > > Hi Jason, > > > > Performance testing and tuning on the data plane is in progress (btw, > > that wouldn't affect the device part patches). > > If possible, could we start the device part patch review in the meantime? > > > > Best, > > Wei > > > > Hi Wei: > > Will do, but basically, the cover lacks of the motivation for vhost-pci and I want > to see some numbers first since I suspect it can over-perform exist data-path. > > Thanks [Avi Cohen (A)] Hi Wei I can try testing to get **numbers** - I can do it now , need a little help from you I've started with downloading/building and installing of > driver: https://github.com/wei-w-wang/vhost-pci-driver to the kernel guest, **without** downloading the 2nd patch > device: https://github.com/wei-w-wang/vhost-pci-device But my guest kernel was corrupted after reboot (kernel panic/out of mem ..) - can you tell me the steps to apply these patches ? Best Regards Avi
On 12/05/2017 04:49 PM, Avi Cohen (A) wrote: > >> -----Original Message----- >> From: Jason Wang [mailto:jasowang@redhat.com] >> Sent: Tuesday, 05 December, 2017 9:19 AM >> To: Wei Wang; virtio-dev@lists.oasis-open.org; qemu-devel@nongnu.org; >> mst@redhat.com; marcandre.lureau@redhat.com; stefanha@redhat.com; >> pbonzini@redhat.com >> Cc: jan.kiszka@siemens.com; Avi Cohen (A); zhiyong.yang@intel.com >> Subject: Re: [virtio-dev] [PATCH v3 0/7] Vhost-pci for inter-VM communication >> >> >> >> On 2017年12月05日 15:15, Wei Wang wrote: >>> On 12/05/2017 03:01 PM, Jason Wang wrote: >>>> >>>> On 2017年12月05日 11:33, Wei Wang wrote: >>>>> Vhost-pci is a point-to-point based inter-VM communication solution. >>>>> This >>>>> patch series implements the vhost-pci-net device setup and >>>>> emulation. The device is implemented as a virtio device, and it is >>>>> set up via the vhost-user protocol to get the neessary info (e.g the >>>>> memory info of the remote VM, vring info). >>>>> >>>>> Currently, only the fundamental functions are implemented. More >>>>> features, such as MQ and live migration, will be updated in the >>>>> future. >>>>> >>>>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >>>>> http://dpdk.org/ml/archives/dev/2017-November/082615.html >>>>> >>>>> v2->v3 changes: >>>>> 1) static device creation: instead of creating and hot-plugging the >>>>> device when receiving a vhost-user msg, the device is not >>>>> created >>>>> via the qemu booting command line. >>>>> 2) remove vqs: rq and ctrlq are removed in this version. >>>>> - receive vq: the receive vq is not needed anymore. The PMD >>>>> directly >>>>> shares the remote txq and rxq - grab from remote >>>>> txq to >>>>> receive packets, and put to rxq to send packets. >>>>> - ctrlq: the ctrlq is replaced by the first 4KB metadata area >>>>> of the >>>>> device Bar-2. >>>>> 3) simpler implementation: the entire implementation has been >>>>> tailored >>>>> from ~1800 LOC to ~850 LOC. >>>> Hi: >>>> >>>> Any performance numbers you can share? >>>> >>> Hi Jason, >>> >>> Performance testing and tuning on the data plane is in progress (btw, >>> that wouldn't affect the device part patches). >>> If possible, could we start the device part patch review in the meantime? >>> >>> Best, >>> Wei >>> >> Hi Wei: >> >> Will do, but basically, the cover lacks of the motivation for vhost-pci and I want >> to see some numbers first since I suspect it can over-perform exist data-path. >> >> Thanks > [Avi Cohen (A)] > Hi Wei > I can try testing to get **numbers** - I can do it now , need a little help from you > I've started with downloading/building and installing of > driver: https://github.com/wei-w-wang/vhost-pci-driver to the kernel guest, > **without** downloading the 2nd patch > device: https://github.com/wei-w-wang/vhost-pci-device > But my guest kernel was corrupted after reboot (kernel panic/out of mem ..) - can you tell me the steps to apply these patches ? > Best Regards > Avi The kernel driver do have some bugs in some environment, so it might be a good source to get feeling about how it works, but I wouldn't recommend you to test it by yourself at this point. We are currently focusing on the dpdk pmd, and wouldn't get back to the kernel driver until all are merged. Sorry about that. Best, Wei
On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > Vhost-pci is a point-to-point based inter-VM communication solution. This > patch series implements the vhost-pci-net device setup and emulation. The > device is implemented as a virtio device, and it is set up via the > vhost-user protocol to get the neessary info (e.g the memory info of the > remote VM, vring info). > > Currently, only the fundamental functions are implemented. More features, > such as MQ and live migration, will be updated in the future. > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > http://dpdk.org/ml/archives/dev/2017-November/082615.html Please be explicit about "vhost-pci" vs "vhost-pci-net" in future emails. In this email you use "vhost-pci" to mean both things and it's likely to cause confusion. For example, MQ and DPDK PMD are related to vhost-pci-net. This is a general problem with all of vhost-user, not just vhost-pci. People sometimes focus solely on virtio-net. It causes layering violations and protocol limitations because they are not thinking about the virtio device model as a whole. Eventually virtio-gpu, virtio-scsi, etc should all work over vhost-pci so it's important to keep the virtio architecture in mind with a separation between the transport and devices.
On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > Vhost-pci is a point-to-point based inter-VM communication solution. This > patch series implements the vhost-pci-net device setup and emulation. The > device is implemented as a virtio device, and it is set up via the > vhost-user protocol to get the neessary info (e.g the memory info of the > remote VM, vring info). > > Currently, only the fundamental functions are implemented. More features, > such as MQ and live migration, will be updated in the future. > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > http://dpdk.org/ml/archives/dev/2017-November/082615.html > > v2->v3 changes: > 1) static device creation: instead of creating and hot-plugging the > device when receiving a vhost-user msg, the device is not created > via the qemu booting command line. > 2) remove vqs: rq and ctrlq are removed in this version. > - receive vq: the receive vq is not needed anymore. The PMD directly > shares the remote txq and rxq - grab from remote txq to > receive packets, and put to rxq to send packets. > - ctrlq: the ctrlq is replaced by the first 4KB metadata area of the > device Bar-2. > 3) simpler implementation: the entire implementation has been tailored > from ~1800 LOC to ~850 LOC. Pls always include full change log, so pls add v1->v2 as well. Also pls include the link to the spec you posted previously (not required to be 100% in sync, but pls list main differences). > Wei Wang (7): > vhost-user: share the vhost-user protocol related structures > vhost-pci-net: add vhost-pci-net > virtio/virtio-pci.c: add vhost-pci-net-pci > vhost-pci-slave: add vhost-pci slave implementation > vhost-user: VHOST_USER_SET_VHOST_PCI msg > vhost-pci-slave: handle VHOST_USER_SET_VHOST_PCI > virtio/vhost.c: vhost-pci needs remote gpa > > hw/net/Makefile.objs | 2 +- > hw/net/vhost_net.c | 37 +++ > hw/net/vhost_pci_net.c | 137 +++++++++++ > hw/virtio/Makefile.objs | 1 + > hw/virtio/vhost-pci-slave.c | 310 +++++++++++++++++++++++++ > hw/virtio/vhost-user.c | 117 ++-------- > hw/virtio/vhost.c | 63 +++-- > hw/virtio/virtio-pci.c | 55 +++++ > hw/virtio/virtio-pci.h | 14 ++ > include/hw/pci/pci.h | 1 + > include/hw/virtio/vhost-backend.h | 2 + > include/hw/virtio/vhost-pci-net.h | 42 ++++ > include/hw/virtio/vhost-pci-slave.h | 12 + > include/hw/virtio/vhost-user.h | 108 +++++++++ > include/hw/virtio/vhost.h | 2 + > include/net/vhost_net.h | 2 + > include/standard-headers/linux/vhost_pci_net.h | 65 ++++++ > include/standard-headers/linux/virtio_ids.h | 1 + > 18 files changed, 851 insertions(+), 120 deletions(-) > create mode 100644 hw/net/vhost_pci_net.c > create mode 100644 hw/virtio/vhost-pci-slave.c > create mode 100644 include/hw/virtio/vhost-pci-net.h > create mode 100644 include/hw/virtio/vhost-pci-slave.h > create mode 100644 include/hw/virtio/vhost-user.h > create mode 100644 include/standard-headers/linux/vhost_pci_net.h > > -- > 2.7.4
On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > Vhost-pci is a point-to-point based inter-VM communication solution. This > patch series implements the vhost-pci-net device setup and emulation. The > device is implemented as a virtio device, and it is set up via the > vhost-user protocol to get the neessary info (e.g the memory info of the > remote VM, vring info). > > Currently, only the fundamental functions are implemented. More features, > such as MQ and live migration, will be updated in the future. > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > http://dpdk.org/ml/archives/dev/2017-November/082615.html > > v2->v3 changes: > 1) static device creation: instead of creating and hot-plugging the > device when receiving a vhost-user msg, the device is not created > via the qemu booting command line. > 2) remove vqs: rq and ctrlq are removed in this version. > - receive vq: the receive vq is not needed anymore. The PMD directly > shares the remote txq and rxq - grab from remote txq to > receive packets, and put to rxq to send packets. > - ctrlq: the ctrlq is replaced by the first 4KB metadata area of the > device Bar-2. > 3) simpler implementation: the entire implementation has been tailored > from ~1800 LOC to ~850 LOC. Please write qtest test cases for vhost-pci. See tests/virtio-net-test.c and tests/vhost-user-test.c for examples. Stefan
On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > Vhost-pci is a point-to-point based inter-VM communication solution. This > patch series implements the vhost-pci-net device setup and emulation. The > device is implemented as a virtio device, and it is set up via the > vhost-user protocol to get the neessary info (e.g the memory info of the > remote VM, vring info). > > Currently, only the fundamental functions are implemented. More features, > such as MQ and live migration, will be updated in the future. > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > http://dpdk.org/ml/archives/dev/2017-November/082615.html I have asked questions about the scope of this feature. In particular, I think it's best to support all device types rather than just virtio-net. Here is a design document that shows how this can be achieved. What I'm proposing is different from the current approach: 1. It's a PCI adapter (see below for justification) 2. The vhost-user protocol is exposed by the device (not handled 100% in QEMU). Ultimately I think your approach would also need to do this. I'm not implementing this and not asking you to implement it. Let's just use this for discussion so we can figure out what the final vhost-pci will look like. Please let me know what you think, Wei, Michael, and others. --- vhost-pci device specification ------------------------------- The vhost-pci device allows guests to act as vhost-user slaves. This enables appliance VMs like network switches or storage targets to back devices in other VMs. VM-to-VM communication is possible without vmexits using polling mode drivers. The vhost-user protocol has been used to implement virtio devices in userspace processes on the host. vhost-pci maps the vhost-user protocol to a PCI adapter so guest software can perform virtio device emulation. This is useful in environments where high-performance VM-to-VM communication is necessary or where it is preferrable to deploy device emulation as VMs instead of host userspace processes. The vhost-user protocol involves file descriptor passing and shared memory. This precludes vhost-user slave implementations over virtio-vsock, virtio-serial, or TCP/IP. Therefore a new device type is needed to expose the vhost-user protocol to guests. The vhost-pci PCI adapter has the following resources: Queues (used for vhost-user protocol communication): 1. Master-to-slave messages 2. Slave-to-master messages Doorbells (used for slave->guest/master events): 1. Vring call (one doorbell per virtqueue) 2. Vring err (one doorbell per virtqueue) 3. Log changed Interrupts (used for guest->slave events): 1. Vring kick (one MSI per virtqueue) Shared Memory BARs: 1. Guest memory 2. Log Master-to-slave queue: The following vhost-user protocol messages are relayed from the vhost-user master. Each message follows the vhost-user protocol VhostUserMsg layout. Messages that include file descriptor passing are relayed but do not carry file descriptors. The relevant resources (doorbells, interrupts, or shared memory BARs) are initialized from the file descriptors prior to the message becoming available on the Master-to-Slave queue. Resources must only be used after the corresponding vhost-user message has been received. For example, the Vring call doorbell can only be used after VHOST_USER_SET_VRING_CALL becomes available on the Master-to-Slave queue. Messages must be processed in order. The following vhost-user protocol messages are relayed: * VHOST_USER_GET_FEATURES * VHOST_USER_SET_FEATURES * VHOST_USER_GET_PROTOCOL_FEATURES * VHOST_USER_SET_PROTOCOL_FEATURES * VHOST_USER_SET_OWNER * VHOST_USER_SET_MEM_TABLE The shared memory is available in the corresponding BAR. * VHOST_USER_SET_LOG_BASE The shared memory is available in the corresponding BAR. * VHOST_USER_SET_LOG_FD The logging file descriptor can be signalled through the logging virtqueue. * VHOST_USER_SET_VRING_NUM * VHOST_USER_SET_VRING_ADDR * VHOST_USER_SET_VRING_BASE * VHOST_USER_GET_VRING_BASE * VHOST_USER_SET_VRING_KICK This message is still needed because it may indicate only polling mode is supported. * VHOST_USER_SET_VRING_CALL This message is still needed because it may indicate only polling mode is supported. * VHOST_USER_SET_VRING_ERR * VHOST_USER_GET_QUEUE_NUM * VHOST_USER_SET_VRING_ENABLE * VHOST_USER_SEND_RARP * VHOST_USER_NET_SET_MTU * VHOST_USER_SET_SLAVE_REQ_FD * VHOST_USER_IOTLB_MSG * VHOST_USER_SET_VRING_ENDIAN Slave-to-Master queue: Messages added to the Slave-to-Master queue are sent to the vhost-user master. Each message follows the vhost-user protocol VhostUserMsg layout. The following vhost-user protocol messages are relayed: * VHOST_USER_SLAVE_IOTLB_MSG Theory of Operation: When the vhost-pci adapter is detected the queues must be set up by the driver. Once the driver is ready the vhost-pci device begins relaying vhost-user protocol messages over the Master-to-Slave queue. The driver must follow the vhost-user protocol specification to implement virtio device initialization and virtqueue processing. Notes: The vhost-user UNIX domain socket connects two host processes. The slave process interprets messages and initializes vhost-pci resources (doorbells, interrupts, shared memory BARs) based on them before relaying via the Master-to-Slave queue. All messages are relayed, even if they only pass a file descriptor, because the message itself may act as a signal (e.g. virtqueue is now enabled). vhost-pci is a PCI adapter instead of a virtio device to allow doorbells and interrupts to be connected to the virtio device in the master VM in the most efficient way possible. This means the Vring call doorbell can be an ioeventfd that signals an irqfd inside the host kernel without host userspace involvement. The Vring kick interrupt can be an irqfd that is signalled by the master VM's virtqueue ioeventfd. It may be possible to write a Linux vhost-pci driver that implements the drivers/vhost/ API. That way existing vhost drivers could work with vhost-pci in the kernel. Guest userspace vhost-pci drivers will be similar to QEMU's contrib/libvhost-user/ except they will probably use vfio to access the vhost-pci device directly from userspace. TODO: * Queue memory layout and hardware registers * vhost-pci-level negotiation and configuration so the hardware interface can be extended in the future. * vhost-pci <-> driver initialization procedure * Master<->Slave disconnected & reconnect
On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: > On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > > Vhost-pci is a point-to-point based inter-VM communication solution. > > This patch series implements the vhost-pci-net device setup and > > emulation. The device is implemented as a virtio device, and it is set > > up via the vhost-user protocol to get the neessary info (e.g the > > memory info of the remote VM, vring info). > > > > Currently, only the fundamental functions are implemented. More > > features, such as MQ and live migration, will be updated in the future. > > > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > > http://dpdk.org/ml/archives/dev/2017-November/082615.html > > I have asked questions about the scope of this feature. In particular, I think > it's best to support all device types rather than just virtio-net. Here is a > design document that shows how this can be achieved. > > What I'm proposing is different from the current approach: > 1. It's a PCI adapter (see below for justification) 2. The vhost-user protocol is > exposed by the device (not handled 100% in > QEMU). Ultimately I think your approach would also need to do this. > > I'm not implementing this and not asking you to implement it. Let's just use > this for discussion so we can figure out what the final vhost-pci will look like. > > Please let me know what you think, Wei, Michael, and others. > Thanks for sharing the thoughts. If I understand it correctly, the key difference is that this approach tries to relay every vhost-user msg to the guest. I'm not sure about the benefits of doing this. To make data plane (i.e. driver to send/receive packets) work, I think, mostly, the memory info and vring info are enough. Other things like callfd, kickfd don't need to be sent to the guest, they are needed by QEMU only for the eventfd and irqfd setup. > --- > vhost-pci device specification > ------------------------------- > The vhost-pci device allows guests to act as vhost-user slaves. This enables > appliance VMs like network switches or storage targets to back devices in > other VMs. VM-to-VM communication is possible without vmexits using > polling mode drivers. > > The vhost-user protocol has been used to implement virtio devices in > userspace processes on the host. vhost-pci maps the vhost-user protocol to > a PCI adapter so guest software can perform virtio device emulation. > This is useful in environments where high-performance VM-to-VM > communication is necessary or where it is preferrable to deploy device > emulation as VMs instead of host userspace processes. > > The vhost-user protocol involves file descriptor passing and shared memory. > This precludes vhost-user slave implementations over virtio-vsock, virtio- > serial, or TCP/IP. Therefore a new device type is needed to expose the > vhost-user protocol to guests. > > The vhost-pci PCI adapter has the following resources: > > Queues (used for vhost-user protocol communication): > 1. Master-to-slave messages > 2. Slave-to-master messages > > Doorbells (used for slave->guest/master events): > 1. Vring call (one doorbell per virtqueue) 2. Vring err (one doorbell per > virtqueue) 3. Log changed > > Interrupts (used for guest->slave events): > 1. Vring kick (one MSI per virtqueue) > > Shared Memory BARs: > 1. Guest memory > 2. Log > > Master-to-slave queue: > The following vhost-user protocol messages are relayed from the vhost-user > master. Each message follows the vhost-user protocol VhostUserMsg layout. > > Messages that include file descriptor passing are relayed but do not carry file > descriptors. The relevant resources (doorbells, interrupts, or shared memory > BARs) are initialized from the file descriptors prior to the message becoming > available on the Master-to-Slave queue. > > Resources must only be used after the corresponding vhost-user message > has been received. For example, the Vring call doorbell can only be used > after VHOST_USER_SET_VRING_CALL becomes available on the Master-to- > Slave queue. > > Messages must be processed in order. > > The following vhost-user protocol messages are relayed: > * VHOST_USER_GET_FEATURES > * VHOST_USER_SET_FEATURES > * VHOST_USER_GET_PROTOCOL_FEATURES > * VHOST_USER_SET_PROTOCOL_FEATURES > * VHOST_USER_SET_OWNER > * VHOST_USER_SET_MEM_TABLE > The shared memory is available in the corresponding BAR. > * VHOST_USER_SET_LOG_BASE > The shared memory is available in the corresponding BAR. > * VHOST_USER_SET_LOG_FD > The logging file descriptor can be signalled through the logging > virtqueue. > * VHOST_USER_SET_VRING_NUM > * VHOST_USER_SET_VRING_ADDR > * VHOST_USER_SET_VRING_BASE > * VHOST_USER_GET_VRING_BASE > * VHOST_USER_SET_VRING_KICK > This message is still needed because it may indicate only polling > mode is supported. > * VHOST_USER_SET_VRING_CALL > This message is still needed because it may indicate only polling > mode is supported. > * VHOST_USER_SET_VRING_ERR > * VHOST_USER_GET_QUEUE_NUM > * VHOST_USER_SET_VRING_ENABLE > * VHOST_USER_SEND_RARP > * VHOST_USER_NET_SET_MTU > * VHOST_USER_SET_SLAVE_REQ_FD > * VHOST_USER_IOTLB_MSG > * VHOST_USER_SET_VRING_ENDIAN > > Slave-to-Master queue: > Messages added to the Slave-to-Master queue are sent to the vhost-user > master. Each message follows the vhost-user protocol VhostUserMsg layout. > > The following vhost-user protocol messages are relayed: > > * VHOST_USER_SLAVE_IOTLB_MSG > > Theory of Operation: > When the vhost-pci adapter is detected the queues must be set up by the > driver. Once the driver is ready the vhost-pci device begins relaying vhost- > user protocol messages over the Master-to-Slave queue. The driver must > follow the vhost-user protocol specification to implement virtio device > initialization and virtqueue processing. > > Notes: > The vhost-user UNIX domain socket connects two host processes. The slave > process interprets messages and initializes vhost-pci resources (doorbells, > interrupts, shared memory BARs) based on them before relaying via the > Master-to-Slave queue. All messages are relayed, even if they only pass a > file descriptor, because the message itself may act as a signal (e.g. virtqueue > is now enabled). > > vhost-pci is a PCI adapter instead of a virtio device to allow doorbells and > interrupts to be connected to the virtio device in the master VM in the most > efficient way possible. This means the Vring call doorbell can be an > ioeventfd that signals an irqfd inside the host kernel without host userspace > involvement. The Vring kick interrupt can be an irqfd that is signalled by the > master VM's virtqueue ioeventfd. > This looks the same as the implementation of inter-VM notification in v2: https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html which is fig. 4 here: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost-pci-rfc2.0.pdf When the vhost-pci driver kicks its tx, the host signals the irqfd of virtio-net's rx. I think this has already bypassed the host userspace (thanks to the fast mmio implementation) Best, Wei
On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> wrote: > On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: >> On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: >> > Vhost-pci is a point-to-point based inter-VM communication solution. >> > This patch series implements the vhost-pci-net device setup and >> > emulation. The device is implemented as a virtio device, and it is set >> > up via the vhost-user protocol to get the neessary info (e.g the >> > memory info of the remote VM, vring info). >> > >> > Currently, only the fundamental functions are implemented. More >> > features, such as MQ and live migration, will be updated in the future. >> > >> > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >> > http://dpdk.org/ml/archives/dev/2017-November/082615.html >> >> I have asked questions about the scope of this feature. In particular, I think >> it's best to support all device types rather than just virtio-net. Here is a >> design document that shows how this can be achieved. >> >> What I'm proposing is different from the current approach: >> 1. It's a PCI adapter (see below for justification) 2. The vhost-user protocol is >> exposed by the device (not handled 100% in >> QEMU). Ultimately I think your approach would also need to do this. >> >> I'm not implementing this and not asking you to implement it. Let's just use >> this for discussion so we can figure out what the final vhost-pci will look like. >> >> Please let me know what you think, Wei, Michael, and others. >> > > Thanks for sharing the thoughts. If I understand it correctly, the key difference is that this approach tries to relay every vhost-user msg to the guest. I'm not sure about the benefits of doing this. > To make data plane (i.e. driver to send/receive packets) work, I think, mostly, the memory info and vring info are enough. Other things like callfd, kickfd don't need to be sent to the guest, they are needed by QEMU only for the eventfd and irqfd setup. Handling the vhost-user protocol inside QEMU and exposing a different interface to the guest makes the interface device-specific. This will cause extra work to support new devices (vhost-user-scsi, vhost-user-blk). It also makes development harder because you might have to learn 3 separate specifications to debug the system (virtio, vhost-user, vhost-pci-net). If vhost-user is mapped to a PCI device then these issues are solved. >> vhost-pci is a PCI adapter instead of a virtio device to allow doorbells and >> interrupts to be connected to the virtio device in the master VM in the most >> efficient way possible. This means the Vring call doorbell can be an >> ioeventfd that signals an irqfd inside the host kernel without host userspace >> involvement. The Vring kick interrupt can be an irqfd that is signalled by the >> master VM's virtqueue ioeventfd. >> > > > This looks the same as the implementation of inter-VM notification in v2: > https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html > which is fig. 4 here: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost-pci-rfc2.0.pdf > > When the vhost-pci driver kicks its tx, the host signals the irqfd of virtio-net's rx. I think this has already bypassed the host userspace (thanks to the fast mmio implementation) Yes, I think the irqfd <-> ioeventfd mapping is good. Perhaps it even makes sense to implement a special fused_irq_ioevent_fd in the host kernel to bypass the need for a kernel thread to read the eventfd so that an interrupt can be injected (i.e. to make the operation synchronous). Is the tx virtqueue in your inter-VM notification v2 series a real virtqueue that gets used? Or is it just a dummy virtqueue that you're using for the ioeventfd doorbell? It looks like vpnet_handle_vq() is empty so it's really just a dummy. The actual virtqueue is in the vhost-user master guest memory.
On 12/07/2017 12:27 AM, Stefan Hajnoczi wrote: > On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> wrote: >> On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: >>> On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: >>>> Vhost-pci is a point-to-point based inter-VM communication solution. >>>> This patch series implements the vhost-pci-net device setup and >>>> emulation. The device is implemented as a virtio device, and it is set >>>> up via the vhost-user protocol to get the neessary info (e.g the >>>> memory info of the remote VM, vring info). >>>> >>>> Currently, only the fundamental functions are implemented. More >>>> features, such as MQ and live migration, will be updated in the future. >>>> >>>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >>>> http://dpdk.org/ml/archives/dev/2017-November/082615.html >>> I have asked questions about the scope of this feature. In particular, I think >>> it's best to support all device types rather than just virtio-net. Here is a >>> design document that shows how this can be achieved. >>> >>> What I'm proposing is different from the current approach: >>> 1. It's a PCI adapter (see below for justification) 2. The vhost-user protocol is >>> exposed by the device (not handled 100% in >>> QEMU). Ultimately I think your approach would also need to do this. >>> >>> I'm not implementing this and not asking you to implement it. Let's just use >>> this for discussion so we can figure out what the final vhost-pci will look like. >>> >>> Please let me know what you think, Wei, Michael, and others. >>> >> Thanks for sharing the thoughts. If I understand it correctly, the key difference is that this approach tries to relay every vhost-user msg to the guest. I'm not sure about the benefits of doing this. >> To make data plane (i.e. driver to send/receive packets) work, I think, mostly, the memory info and vring info are enough. Other things like callfd, kickfd don't need to be sent to the guest, they are needed by QEMU only for the eventfd and irqfd setup. > Handling the vhost-user protocol inside QEMU and exposing a different > interface to the guest makes the interface device-specific. This will > cause extra work to support new devices (vhost-user-scsi, > vhost-user-blk). It also makes development harder because you might > have to learn 3 separate specifications to debug the system (virtio, > vhost-user, vhost-pci-net). > > If vhost-user is mapped to a PCI device then these issues are solved. I intend to have a different opinion about this: 1) Even relaying the msgs to the guest, QEMU still need to handle the msg first, for example, it needs to decode the msg to see if it is the ones (e.g. SET_MEM_TABLE, SET_VRING_KICK, SET_VRING_CALL) that should be used for the device setup (e.g. mmap the memory given via SET_MEM_TABLE). In this case, we will be likely to have 2 slave handlers - one in the guest, another in QEMU device. 2) If people already understand the vhost-user protocol, it would be natural for them to understand the vhost-pci metadata - just the obtained memory and vring info are put to the metadata area (no new things). Inspired from your sharing, how about the following: we can actually factor out a common vhost-pci layer, which handles all the features that are common to all the vhost-pci series of devices (vhost-pci-net, vhost-pci-blk,...) Coming to the implementation, we can have a VhostpciDeviceClass (similar to VirtioDeviceClass), the device realize sequence will be virtio_device_realize()-->vhost_pci_device_realize()-->vhost_pci_net_device_realize() > >>> vhost-pci is a PCI adapter instead of a virtio device to allow doorbells and >>> interrupts to be connected to the virtio device in the master VM in the most >>> efficient way possible. This means the Vring call doorbell can be an >>> ioeventfd that signals an irqfd inside the host kernel without host userspace >>> involvement. The Vring kick interrupt can be an irqfd that is signalled by the >>> master VM's virtqueue ioeventfd. >>> >> >> This looks the same as the implementation of inter-VM notification in v2: >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html >> which is fig. 4 here: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost-pci-rfc2.0.pdf >> >> When the vhost-pci driver kicks its tx, the host signals the irqfd of virtio-net's rx. I think this has already bypassed the host userspace (thanks to the fast mmio implementation) > Yes, I think the irqfd <-> ioeventfd mapping is good. Perhaps it even > makes sense to implement a special fused_irq_ioevent_fd in the host > kernel to bypass the need for a kernel thread to read the eventfd so > that an interrupt can be injected (i.e. to make the operation > synchronous). > > Is the tx virtqueue in your inter-VM notification v2 series a real > virtqueue that gets used? Or is it just a dummy virtqueue that you're > using for the ioeventfd doorbell? It looks like vpnet_handle_vq() is > empty so it's really just a dummy. The actual virtqueue is in the > vhost-user master guest memory. Yes, that tx is a dummy actually, just created to use its doorbell. Currently, with virtio_device, I think ioeventfd comes with virtqueue only. Actually, I think we could have the issues solved by vhost-pci. For example, reserve a piece of the BAR area for ioeventfd. The bar layout can be: BAR 2: 0~4k: vhost-pci device specific usages (ioeventfd etc) 4k~8k: metadata (memory info and vring info) 8k~64GB: remote guest memory (we can make the bar size (64GB is the default value used) configurable via qemu cmdline) Best, Wei
On Thu, Dec 07, 2017 at 11:57:33AM +0800, Wei Wang wrote: > On 12/07/2017 12:27 AM, Stefan Hajnoczi wrote: > > On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> wrote: > > > On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: > > > > On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > > > > > Vhost-pci is a point-to-point based inter-VM communication solution. > > > > > This patch series implements the vhost-pci-net device setup and > > > > > emulation. The device is implemented as a virtio device, and it is set > > > > > up via the vhost-user protocol to get the neessary info (e.g the > > > > > memory info of the remote VM, vring info). > > > > > > > > > > Currently, only the fundamental functions are implemented. More > > > > > features, such as MQ and live migration, will be updated in the future. > > > > > > > > > > The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > > > > > http://dpdk.org/ml/archives/dev/2017-November/082615.html > > > > I have asked questions about the scope of this feature. In particular, I think > > > > it's best to support all device types rather than just virtio-net. Here is a > > > > design document that shows how this can be achieved. > > > > > > > > What I'm proposing is different from the current approach: > > > > 1. It's a PCI adapter (see below for justification) 2. The vhost-user protocol is > > > > exposed by the device (not handled 100% in > > > > QEMU). Ultimately I think your approach would also need to do this. > > > > > > > > I'm not implementing this and not asking you to implement it. Let's just use > > > > this for discussion so we can figure out what the final vhost-pci will look like. > > > > > > > > Please let me know what you think, Wei, Michael, and others. > > > > > > > Thanks for sharing the thoughts. If I understand it correctly, the key difference is that this approach tries to relay every vhost-user msg to the guest. I'm not sure about the benefits of doing this. > > > To make data plane (i.e. driver to send/receive packets) work, I think, mostly, the memory info and vring info are enough. Other things like callfd, kickfd don't need to be sent to the guest, they are needed by QEMU only for the eventfd and irqfd setup. > > Handling the vhost-user protocol inside QEMU and exposing a different > > interface to the guest makes the interface device-specific. This will > > cause extra work to support new devices (vhost-user-scsi, > > vhost-user-blk). It also makes development harder because you might > > have to learn 3 separate specifications to debug the system (virtio, > > vhost-user, vhost-pci-net). > > > > If vhost-user is mapped to a PCI device then these issues are solved. > > I intend to have a different opinion about this: > > 1) Even relaying the msgs to the guest, QEMU still need to handle the msg > first, for example, it needs to decode the msg to see if it is the ones > (e.g. SET_MEM_TABLE, SET_VRING_KICK, SET_VRING_CALL) that should be used for > the device setup (e.g. mmap the memory given via SET_MEM_TABLE). In this > case, we will be likely to have 2 slave handlers - one in the guest, another > in QEMU device. > > 2) If people already understand the vhost-user protocol, it would be natural > for them to understand the vhost-pci metadata - just the obtained memory and > vring info are put to the metadata area (no new things). I see a bigger problem with passthrough. If qemu can't fully decode all messages, it can not operate in a disconected mode - guest will have to stop on disconnect until we re-connect a backend. > > Inspired from your sharing, how about the following: > we can actually factor out a common vhost-pci layer, which handles all the > features that are common to all the vhost-pci series of devices > (vhost-pci-net, vhost-pci-blk,...) > Coming to the implementation, we can have a VhostpciDeviceClass (similar to > VirtioDeviceClass), the device realize sequence will be virtio_device_realize()-->vhost_pci_device_realize()-->vhost_pci_net_device_realize() > > > > > > > > > vhost-pci is a PCI adapter instead of a virtio device to allow doorbells and > > > > interrupts to be connected to the virtio device in the master VM in the most > > > > efficient way possible. This means the Vring call doorbell can be an > > > > ioeventfd that signals an irqfd inside the host kernel without host userspace > > > > involvement. The Vring kick interrupt can be an irqfd that is signalled by the > > > > master VM's virtqueue ioeventfd. > > > > > > > > > > This looks the same as the implementation of inter-VM notification in v2: > > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html > > > which is fig. 4 here: https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost-pci-rfc2.0.pdf > > > > > > When the vhost-pci driver kicks its tx, the host signals the irqfd of virtio-net's rx. I think this has already bypassed the host userspace (thanks to the fast mmio implementation) > > Yes, I think the irqfd <-> ioeventfd mapping is good. Perhaps it even > > makes sense to implement a special fused_irq_ioevent_fd in the host > > kernel to bypass the need for a kernel thread to read the eventfd so > > that an interrupt can be injected (i.e. to make the operation > > synchronous). > > > > Is the tx virtqueue in your inter-VM notification v2 series a real > > virtqueue that gets used? Or is it just a dummy virtqueue that you're > > using for the ioeventfd doorbell? It looks like vpnet_handle_vq() is > > empty so it's really just a dummy. The actual virtqueue is in the > > vhost-user master guest memory. > > > Yes, that tx is a dummy actually, just created to use its doorbell. > Currently, with virtio_device, I think ioeventfd comes with virtqueue only. > Actually, I think we could have the issues solved by vhost-pci. For example, > reserve a piece of the BAR area for ioeventfd. The bar layout can be: > BAR 2: > 0~4k: vhost-pci device specific usages (ioeventfd etc) > 4k~8k: metadata (memory info and vring info) > 8k~64GB: remote guest memory > (we can make the bar size (64GB is the default value used) configurable via > qemu cmdline) > > > Best, > Wei > > > > >
On 12/07/2017 01:11 PM, Michael S. Tsirkin wrote: > On Thu, Dec 07, 2017 at 11:57:33AM +0800, Wei Wang wrote: >> On 12/07/2017 12:27 AM, Stefan Hajnoczi wrote: >>> On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> wrote: >>>> On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: >>>>> On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: >>>>>> Vhost-pci is a point-to-point based inter-VM communication solution. >>>>>> This patch series implements the vhost-pci-net device setup and >>>>>> emulation. The device is implemented as a virtio device, and it is set >>>>>> up via the vhost-user protocol to get the neessary info (e.g the >>>>>> memory info of the remote VM, vring info). >>>>>> >>>>>> Currently, only the fundamental functions are implemented. More >>>>>> features, such as MQ and live migration, will be updated in the future. >>>>>> >>>>>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >>>>>> http://dpdk.org/ml/archives/dev/2017-November/082615.html >>>>> I have asked questions about the scope of this feature. In particular, I think >>>>> it's best to support all device types rather than just virtio-net. Here is a >>>>> design document that shows how this can be achieved. >>>>> >>>>> What I'm proposing is different from the current approach: >>>>> 1. It's a PCI adapter (see below for justification) 2. The vhost-user protocol is >>>>> exposed by the device (not handled 100% in >>>>> QEMU). Ultimately I think your approach would also need to do this. >>>>> >>>>> I'm not implementing this and not asking you to implement it. Let's just use >>>>> this for discussion so we can figure out what the final vhost-pci will look like. >>>>> >>>>> Please let me know what you think, Wei, Michael, and others. >>>>> >>>> Thanks for sharing the thoughts. If I understand it correctly, the key difference is that this approach tries to relay every vhost-user msg to the guest. I'm not sure about the benefits of doing this. >>>> To make data plane (i.e. driver to send/receive packets) work, I think, mostly, the memory info and vring info are enough. Other things like callfd, kickfd don't need to be sent to the guest, they are needed by QEMU only for the eventfd and irqfd setup. >>> Handling the vhost-user protocol inside QEMU and exposing a different >>> interface to the guest makes the interface device-specific. This will >>> cause extra work to support new devices (vhost-user-scsi, >>> vhost-user-blk). It also makes development harder because you might >>> have to learn 3 separate specifications to debug the system (virtio, >>> vhost-user, vhost-pci-net). >>> >>> If vhost-user is mapped to a PCI device then these issues are solved. >> I intend to have a different opinion about this: >> >> 1) Even relaying the msgs to the guest, QEMU still need to handle the msg >> first, for example, it needs to decode the msg to see if it is the ones >> (e.g. SET_MEM_TABLE, SET_VRING_KICK, SET_VRING_CALL) that should be used for >> the device setup (e.g. mmap the memory given via SET_MEM_TABLE). In this >> case, we will be likely to have 2 slave handlers - one in the guest, another >> in QEMU device. >> >> 2) If people already understand the vhost-user protocol, it would be natural >> for them to understand the vhost-pci metadata - just the obtained memory and >> vring info are put to the metadata area (no new things). > I see a bigger problem with passthrough. If qemu can't fully decode all > messages, it can not operate in a disconected mode - guest will have to > stop on disconnect until we re-connect a backend. > What do you mean by "passthrough" in this case? Why qemu can't fully decode all the messages (probably I haven't got the point) Best, Wei
On Thu, Dec 7, 2017 at 3:57 AM, Wei Wang <wei.w.wang@intel.com> wrote: > On 12/07/2017 12:27 AM, Stefan Hajnoczi wrote: >> >> On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> wrote: >>> >>> On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: >>>> >>>> On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: >>>>> >>>>> Vhost-pci is a point-to-point based inter-VM communication solution. >>>>> This patch series implements the vhost-pci-net device setup and >>>>> emulation. The device is implemented as a virtio device, and it is set >>>>> up via the vhost-user protocol to get the neessary info (e.g the >>>>> memory info of the remote VM, vring info). >>>>> >>>>> Currently, only the fundamental functions are implemented. More >>>>> features, such as MQ and live migration, will be updated in the future. >>>>> >>>>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >>>>> http://dpdk.org/ml/archives/dev/2017-November/082615.html >>>> >>>> I have asked questions about the scope of this feature. In particular, >>>> I think >>>> it's best to support all device types rather than just virtio-net. Here >>>> is a >>>> design document that shows how this can be achieved. >>>> >>>> What I'm proposing is different from the current approach: >>>> 1. It's a PCI adapter (see below for justification) 2. The vhost-user >>>> protocol is >>>> exposed by the device (not handled 100% in >>>> QEMU). Ultimately I think your approach would also need to do this. >>>> >>>> I'm not implementing this and not asking you to implement it. Let's >>>> just use >>>> this for discussion so we can figure out what the final vhost-pci will >>>> look like. >>>> >>>> Please let me know what you think, Wei, Michael, and others. >>>> >>> Thanks for sharing the thoughts. If I understand it correctly, the key >>> difference is that this approach tries to relay every vhost-user msg to the >>> guest. I'm not sure about the benefits of doing this. >>> To make data plane (i.e. driver to send/receive packets) work, I think, >>> mostly, the memory info and vring info are enough. Other things like callfd, >>> kickfd don't need to be sent to the guest, they are needed by QEMU only for >>> the eventfd and irqfd setup. >> >> Handling the vhost-user protocol inside QEMU and exposing a different >> interface to the guest makes the interface device-specific. This will >> cause extra work to support new devices (vhost-user-scsi, >> vhost-user-blk). It also makes development harder because you might >> have to learn 3 separate specifications to debug the system (virtio, >> vhost-user, vhost-pci-net). >> >> If vhost-user is mapped to a PCI device then these issues are solved. > > > I intend to have a different opinion about this: > > 1) Even relaying the msgs to the guest, QEMU still need to handle the msg > first, for example, it needs to decode the msg to see if it is the ones > (e.g. SET_MEM_TABLE, SET_VRING_KICK, SET_VRING_CALL) that should be used for > the device setup (e.g. mmap the memory given via SET_MEM_TABLE). In this > case, we will be likely to have 2 slave handlers - one in the guest, another > in QEMU device. In theory the vhost-pci PCI adapter could decide not to relay certain messages. As explained in the document, I think it's better to relay everything because some messages that only carry an fd still have a meaning. They are a signal that the master has entered a new state. The approach in this patch series doesn't really solve the 2 handler problem, it still needs to notify the guest when certain vhost-user messages are received from the master. The difference is just that it's non-trivial in this patch series because each message is handled on a case-by-case basis and has a custom interface (does not simply relay a vhost-user protocol message). A 1:1 model is simple and consistent. I think it will avoid bugs and design mistakes. > 2) If people already understand the vhost-user protocol, it would be natural > for them to understand the vhost-pci metadata - just the obtained memory and > vring info are put to the metadata area (no new things). This is debatable. It's like saying if you understand QEMU command-line options you will understand libvirt domain XML. They map to each other but how obvious that mapping is depends on the details. I'm saying a 1:1 mapping (reusing the vhost-user protocol message layout) is the cleanest option. > Inspired from your sharing, how about the following: > we can actually factor out a common vhost-pci layer, which handles all the > features that are common to all the vhost-pci series of devices > (vhost-pci-net, vhost-pci-blk,...) > Coming to the implementation, we can have a VhostpciDeviceClass (similar to > VirtioDeviceClass), the device realize sequence will be > virtio_device_realize()-->vhost_pci_device_realize()-->vhost_pci_net_device_realize() Why have individual device types (vhost-pci-net, vhost-pci-blk, etc) instead of just a vhost-pci device? >>>> vhost-pci is a PCI adapter instead of a virtio device to allow doorbells >>>> and >>>> interrupts to be connected to the virtio device in the master VM in the >>>> most >>>> efficient way possible. This means the Vring call doorbell can be an >>>> ioeventfd that signals an irqfd inside the host kernel without host >>>> userspace >>>> involvement. The Vring kick interrupt can be an irqfd that is signalled >>>> by the >>>> master VM's virtqueue ioeventfd. >>>> >>> >>> This looks the same as the implementation of inter-VM notification in v2: >>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html >>> which is fig. 4 here: >>> https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost-pci-rfc2.0.pdf >>> >>> When the vhost-pci driver kicks its tx, the host signals the irqfd of >>> virtio-net's rx. I think this has already bypassed the host userspace >>> (thanks to the fast mmio implementation) >> >> Yes, I think the irqfd <-> ioeventfd mapping is good. Perhaps it even >> makes sense to implement a special fused_irq_ioevent_fd in the host >> kernel to bypass the need for a kernel thread to read the eventfd so >> that an interrupt can be injected (i.e. to make the operation >> synchronous). >> >> Is the tx virtqueue in your inter-VM notification v2 series a real >> virtqueue that gets used? Or is it just a dummy virtqueue that you're >> using for the ioeventfd doorbell? It looks like vpnet_handle_vq() is >> empty so it's really just a dummy. The actual virtqueue is in the >> vhost-user master guest memory. > > > > Yes, that tx is a dummy actually, just created to use its doorbell. > Currently, with virtio_device, I think ioeventfd comes with virtqueue only. > Actually, I think we could have the issues solved by vhost-pci. For example, > reserve a piece of the BAR area for ioeventfd. The bar layout can be: > BAR 2: > 0~4k: vhost-pci device specific usages (ioeventfd etc) > 4k~8k: metadata (memory info and vring info) > 8k~64GB: remote guest memory > (we can make the bar size (64GB is the default value used) configurable via > qemu cmdline) Why use a virtio device? The doorbell and shared memory don't fit the virtio architecture. There are no real virtqueues. This makes it a strange virtio device. Stefan
There is already a virtio mechanism in which 2 VMs assigned a virtio device , are communicating via a veth pair in the host . KVM just passes a pointer of the page of the writer VM to the reader VM - resulting in excellent performance (no vSwitch in the middle) **Question**: What is the advantage of vhost-pci compared to this ? Best Regards Avi > -----Original Message----- > From: Stefan Hajnoczi [mailto:stefanha@gmail.com] > Sent: Thursday, 07 December, 2017 8:31 AM > To: Wei Wang > Cc: Stefan Hajnoczi; virtio-dev@lists.oasis-open.org; mst@redhat.com; Yang, > Zhiyong; jan.kiszka@siemens.com; jasowang@redhat.com; Avi Cohen (A); > qemu-devel@nongnu.org; marcandre.lureau@redhat.com; > pbonzini@redhat.com > Subject: Re: [Qemu-devel] [virtio-dev] [PATCH v3 0/7] Vhost-pci for inter-VM > communication > > On Thu, Dec 7, 2017 at 3:57 AM, Wei Wang <wei.w.wang@intel.com> wrote: > > On 12/07/2017 12:27 AM, Stefan Hajnoczi wrote: > >> > >> On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> > wrote: > >>> > >>> On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: > >>>> > >>>> On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: > >>>>> > >>>>> Vhost-pci is a point-to-point based inter-VM communication solution. > >>>>> This patch series implements the vhost-pci-net device setup and > >>>>> emulation. The device is implemented as a virtio device, and it is > >>>>> set up via the vhost-user protocol to get the neessary info (e.g > >>>>> the memory info of the remote VM, vring info). > >>>>> > >>>>> Currently, only the fundamental functions are implemented. More > >>>>> features, such as MQ and live migration, will be updated in the future. > >>>>> > >>>>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: > >>>>> http://dpdk.org/ml/archives/dev/2017-November/082615.html > >>>> > >>>> I have asked questions about the scope of this feature. In > >>>> particular, I think it's best to support all device types rather > >>>> than just virtio-net. Here is a design document that shows how > >>>> this can be achieved. > >>>> > >>>> What I'm proposing is different from the current approach: > >>>> 1. It's a PCI adapter (see below for justification) 2. The > >>>> vhost-user protocol is exposed by the device (not handled 100% in > >>>> QEMU). Ultimately I think your approach would also need to do this. > >>>> > >>>> I'm not implementing this and not asking you to implement it. > >>>> Let's just use this for discussion so we can figure out what the > >>>> final vhost-pci will look like. > >>>> > >>>> Please let me know what you think, Wei, Michael, and others. > >>>> > >>> Thanks for sharing the thoughts. If I understand it correctly, the > >>> key difference is that this approach tries to relay every vhost-user > >>> msg to the guest. I'm not sure about the benefits of doing this. > >>> To make data plane (i.e. driver to send/receive packets) work, I > >>> think, mostly, the memory info and vring info are enough. Other > >>> things like callfd, kickfd don't need to be sent to the guest, they > >>> are needed by QEMU only for the eventfd and irqfd setup. > >> > >> Handling the vhost-user protocol inside QEMU and exposing a different > >> interface to the guest makes the interface device-specific. This > >> will cause extra work to support new devices (vhost-user-scsi, > >> vhost-user-blk). It also makes development harder because you might > >> have to learn 3 separate specifications to debug the system (virtio, > >> vhost-user, vhost-pci-net). > >> > >> If vhost-user is mapped to a PCI device then these issues are solved. > > > > > > I intend to have a different opinion about this: > > > > 1) Even relaying the msgs to the guest, QEMU still need to handle the > > msg first, for example, it needs to decode the msg to see if it is the > > ones (e.g. SET_MEM_TABLE, SET_VRING_KICK, SET_VRING_CALL) that should > > be used for the device setup (e.g. mmap the memory given via > > SET_MEM_TABLE). In this case, we will be likely to have 2 slave > > handlers - one in the guest, another in QEMU device. > > In theory the vhost-pci PCI adapter could decide not to relay certain messages. > As explained in the document, I think it's better to relay everything because > some messages that only carry an fd still have a meaning. They are a signal > that the master has entered a new state. > > The approach in this patch series doesn't really solve the 2 handler problem, it > still needs to notify the guest when certain vhost-user messages are received > from the master. The difference is just that it's non-trivial in this patch series > because each message is handled on a case-by-case basis and has a custom > interface (does not simply relay a vhost-user protocol message). > > A 1:1 model is simple and consistent. I think it will avoid bugs and design > mistakes. > > > 2) If people already understand the vhost-user protocol, it would be > > natural for them to understand the vhost-pci metadata - just the > > obtained memory and vring info are put to the metadata area (no new > things). > > This is debatable. It's like saying if you understand QEMU command-line > options you will understand libvirt domain XML. They map to each other but > how obvious that mapping is depends on the details. > I'm saying a 1:1 mapping (reusing the vhost-user protocol message > layout) is the cleanest option. > > > Inspired from your sharing, how about the following: > > we can actually factor out a common vhost-pci layer, which handles all > > the features that are common to all the vhost-pci series of devices > > (vhost-pci-net, vhost-pci-blk,...) Coming to the implementation, we > > can have a VhostpciDeviceClass (similar to VirtioDeviceClass), the > > device realize sequence will be > > virtio_device_realize()-->vhost_pci_device_realize()-->vhost_pci_net_d > > evice_realize() > > Why have individual device types (vhost-pci-net, vhost-pci-blk, etc) instead of > just a vhost-pci device? > > >>>> vhost-pci is a PCI adapter instead of a virtio device to allow > >>>> doorbells and interrupts to be connected to the virtio device in > >>>> the master VM in the most efficient way possible. This means the > >>>> Vring call doorbell can be an ioeventfd that signals an irqfd > >>>> inside the host kernel without host userspace involvement. The > >>>> Vring kick interrupt can be an irqfd that is signalled by the > >>>> master VM's virtqueue ioeventfd. > >>>> > >>> > >>> This looks the same as the implementation of inter-VM notification in v2: > >>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html > >>> which is fig. 4 here: > >>> https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost > >>> -pci-rfc2.0.pdf > >>> > >>> When the vhost-pci driver kicks its tx, the host signals the irqfd > >>> of virtio-net's rx. I think this has already bypassed the host > >>> userspace (thanks to the fast mmio implementation) > >> > >> Yes, I think the irqfd <-> ioeventfd mapping is good. Perhaps it > >> even makes sense to implement a special fused_irq_ioevent_fd in the > >> host kernel to bypass the need for a kernel thread to read the > >> eventfd so that an interrupt can be injected (i.e. to make the > >> operation synchronous). > >> > >> Is the tx virtqueue in your inter-VM notification v2 series a real > >> virtqueue that gets used? Or is it just a dummy virtqueue that > >> you're using for the ioeventfd doorbell? It looks like > >> vpnet_handle_vq() is empty so it's really just a dummy. The actual > >> virtqueue is in the vhost-user master guest memory. > > > > > > > > Yes, that tx is a dummy actually, just created to use its doorbell. > > Currently, with virtio_device, I think ioeventfd comes with virtqueue only. > > Actually, I think we could have the issues solved by vhost-pci. For > > example, reserve a piece of the BAR area for ioeventfd. The bar layout can > be: > > BAR 2: > > 0~4k: vhost-pci device specific usages (ioeventfd etc) > > 4k~8k: metadata (memory info and vring info) > > 8k~64GB: remote guest memory > > (we can make the bar size (64GB is the default value used) > > configurable via qemu cmdline) > > Why use a virtio device? The doorbell and shared memory don't fit the virtio > architecture. There are no real virtqueues. This makes it a strange virtio > device. > > Stefan
On Thu, Dec 7, 2017 at 7:54 AM, Avi Cohen (A) <avi.cohen@huawei.com> wrote: > There is already a virtio mechanism in which 2 VMs assigned a virtio device , are communicating via a veth pair in the host . > KVM just passes a pointer of the page of the writer VM to the reader VM - resulting in excellent performance (no vSwitch in the middle) > **Question**: What is the advantage of vhost-pci compared to this ? Which mechanism do you mean? vhost-pci will allow VM-to-VM communication without vmexits when polling mode is used. Does the mechanism you are thinking about require vmexits? Stefan
On 2017年12月07日 16:04, Stefan Hajnoczi wrote: > On Thu, Dec 7, 2017 at 7:54 AM, Avi Cohen (A) <avi.cohen@huawei.com> wrote: >> There is already a virtio mechanism in which 2 VMs assigned a virtio device , are communicating via a veth pair in the host . >> KVM just passes a pointer of the page of the writer VM to the reader VM - resulting in excellent performance (no vSwitch in the middle) >> **Question**: What is the advantage of vhost-pci compared to this ? > Which mechanism do you mean? > > vhost-pci will allow VM-to-VM communication without vmexits when > polling mode is used. Does the mechanism you are thinking about > require vmexits? > > Stefan I guess what Avi means is probably veth tap support (RFC) here: https://www.spinics.net/lists/netdev/msg454040.html But in fact, we don't need veth at all, by using rx handler trick, we can easily implement pair mode on TUN/TAP. Thanks
On Thu, Dec 07, 2017 at 04:31:27PM +0800, Jason Wang wrote: > > > On 2017年12月07日 16:04, Stefan Hajnoczi wrote: > > On Thu, Dec 7, 2017 at 7:54 AM, Avi Cohen (A) <avi.cohen@huawei.com> wrote: > > > There is already a virtio mechanism in which 2 VMs assigned a virtio device , are communicating via a veth pair in the host . > > > KVM just passes a pointer of the page of the writer VM to the reader VM - resulting in excellent performance (no vSwitch in the middle) > > > **Question**: What is the advantage of vhost-pci compared to this ? > > Which mechanism do you mean? > > > > vhost-pci will allow VM-to-VM communication without vmexits when > > polling mode is used. Does the mechanism you are thinking about > > require vmexits? > > > > Stefan > > I guess what Avi means is probably veth tap support (RFC) here: > > https://www.spinics.net/lists/netdev/msg454040.html > > But in fact, we don't need veth at all, by using rx handler trick, we can > easily implement pair mode on TUN/TAP. Okay, that is a different use case: 1. VM<->physical NIC: macvtap 2. VM<->host userspace: vhost-user 3. VM<->host/container: vethtap or pair mode 4. VM<->VM: vhost-pci Stefan
On Thu, Dec 07, 2017 at 07:54:48AM +0000, Avi Cohen (A) wrote: > There is already a virtio mechanism in which 2 VMs assigned a virtio device , are communicating via a veth pair in the host . > KVM just passes a pointer of the page of the writer VM to the reader VM - resulting in excellent performance (no vSwitch in the middle) What exactly do you refer to? > **Question**: What is the advantage of vhost-pci compared to this ? > Best Regards > Avi I suspect what's missing in this thread is the original motivation rfc: http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication
On 12/07/2017 02:31 PM, Stefan Hajnoczi wrote: > On Thu, Dec 7, 2017 at 3:57 AM, Wei Wang <wei.w.wang@intel.com> wrote: >> On 12/07/2017 12:27 AM, Stefan Hajnoczi wrote: >>> On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> wrote: >>>> On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: >>>>> On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: >>>>>> Vhost-pci is a point-to-point based inter-VM communication solution. >>>>>> This patch series implements the vhost-pci-net device setup and >>>>>> emulation. The device is implemented as a virtio device, and it is set >>>>>> up via the vhost-user protocol to get the neessary info (e.g the >>>>>> memory info of the remote VM, vring info). >>>>>> >>>>>> Currently, only the fundamental functions are implemented. More >>>>>> features, such as MQ and live migration, will be updated in the future. >>>>>> >>>>>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >>>>>> http://dpdk.org/ml/archives/dev/2017-November/082615.html >>>>> I have asked questions about the scope of this feature. In particular, >>>>> I think >>>>> it's best to support all device types rather than just virtio-net. Here >>>>> is a >>>>> design document that shows how this can be achieved. >>>>> >>>>> What I'm proposing is different from the current approach: >>>>> 1. It's a PCI adapter (see below for justification) 2. The vhost-user >>>>> protocol is >>>>> exposed by the device (not handled 100% in >>>>> QEMU). Ultimately I think your approach would also need to do this. >>>>> >>>>> I'm not implementing this and not asking you to implement it. Let's >>>>> just use >>>>> this for discussion so we can figure out what the final vhost-pci will >>>>> look like. >>>>> >>>>> Please let me know what you think, Wei, Michael, and others. >>>>> >>>> Thanks for sharing the thoughts. If I understand it correctly, the key >>>> difference is that this approach tries to relay every vhost-user msg to the >>>> guest. I'm not sure about the benefits of doing this. >>>> To make data plane (i.e. driver to send/receive packets) work, I think, >>>> mostly, the memory info and vring info are enough. Other things like callfd, >>>> kickfd don't need to be sent to the guest, they are needed by QEMU only for >>>> the eventfd and irqfd setup. >>> Handling the vhost-user protocol inside QEMU and exposing a different >>> interface to the guest makes the interface device-specific. This will >>> cause extra work to support new devices (vhost-user-scsi, >>> vhost-user-blk). It also makes development harder because you might >>> have to learn 3 separate specifications to debug the system (virtio, >>> vhost-user, vhost-pci-net). >>> >>> If vhost-user is mapped to a PCI device then these issues are solved. >> >> I intend to have a different opinion about this: >> >> 1) Even relaying the msgs to the guest, QEMU still need to handle the msg >> first, for example, it needs to decode the msg to see if it is the ones >> (e.g. SET_MEM_TABLE, SET_VRING_KICK, SET_VRING_CALL) that should be used for >> the device setup (e.g. mmap the memory given via SET_MEM_TABLE). In this >> case, we will be likely to have 2 slave handlers - one in the guest, another >> in QEMU device. > In theory the vhost-pci PCI adapter could decide not to relay certain > messages. As explained in the document, I think it's better to relay > everything because some messages that only carry an fd still have a > meaning. It has its meaning, which is useful for the device setup, but that's not useful for the guest. I think the point is most of the mgs are not useful for the guest. IMHO, the relay mechanism is useful when 1) the QEMU slave handler doesn't need to process the messages at all (receive and directly pass on to the guest) 2) most of the msgs are useful for the guest (say we have more than 20 msgs, only 2 or 3 of them are useful for the guest, why let the device pass all of them to the guest) Also the relay mechanism complicates the vhost-user protocol interaction: normally, only master<->QemuSlave. With the relay mechanism, it will be master<->QemuSlave<->GuestSlave. For example, when the master sends VHOST_USER_GET_QUEUE_NUM, normally it can be answered by QemuSlave directly. Why complicate it by passing the msg the GuestSlave, and then get the same answer from GuestSlave. > They are a signal that the master has entered a new state. Actually vhost-user isn't state-machine based. > Why have individual device types (vhost-pci-net, vhost-pci-blk, etc) > instead of just a vhost-pci device? This is the same as virtio - we don't have a single virtio device, we have virtio-net, virtio-blk etc. So, the same way, we can have a common TYPE_VHOST_PCI_DEVICE parent device (like TYPE_VIRTIO_DEVICE), but net may have its own special features like MRG_RXBUF, and own config registers like mac[6] etc, so we can have TYPE_VHOST_PCI_NET under TYPE_VHOST_PCI_DEVICE. > >>>>> vhost-pci is a PCI adapter instead of a virtio device to allow doorbells >>>>> and >>>>> interrupts to be connected to the virtio device in the master VM in the >>>>> most >>>>> efficient way possible. This means the Vring call doorbell can be an >>>>> ioeventfd that signals an irqfd inside the host kernel without host >>>>> userspace >>>>> involvement. The Vring kick interrupt can be an irqfd that is signalled >>>>> by the >>>>> master VM's virtqueue ioeventfd. >>>>> >>>> This looks the same as the implementation of inter-VM notification in v2: >>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html >>>> which is fig. 4 here: >>>> https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost-pci-rfc2.0.pdf >>>> >>>> When the vhost-pci driver kicks its tx, the host signals the irqfd of >>>> virtio-net's rx. I think this has already bypassed the host userspace >>>> (thanks to the fast mmio implementation) >>> Yes, I think the irqfd <-> ioeventfd mapping is good. Perhaps it even >>> makes sense to implement a special fused_irq_ioevent_fd in the host >>> kernel to bypass the need for a kernel thread to read the eventfd so >>> that an interrupt can be injected (i.e. to make the operation >>> synchronous). >>> >>> Is the tx virtqueue in your inter-VM notification v2 series a real >>> virtqueue that gets used? Or is it just a dummy virtqueue that you're >>> using for the ioeventfd doorbell? It looks like vpnet_handle_vq() is >>> empty so it's really just a dummy. The actual virtqueue is in the >>> vhost-user master guest memory. >> >> >> Yes, that tx is a dummy actually, just created to use its doorbell. >> Currently, with virtio_device, I think ioeventfd comes with virtqueue only. >> Actually, I think we could have the issues solved by vhost-pci. For example, >> reserve a piece of the BAR area for ioeventfd. The bar layout can be: >> BAR 2: >> 0~4k: vhost-pci device specific usages (ioeventfd etc) >> 4k~8k: metadata (memory info and vring info) >> 8k~64GB: remote guest memory >> (we can make the bar size (64GB is the default value used) configurable via >> qemu cmdline) > Why use a virtio device? The doorbell and shared memory don't fit the > virtio architecture. There are no real virtqueues. This makes it a > strange virtio device. The virtio spec doesn't seem to require the device to have at lease one virtqueue. It doesn't make a huge difference to me whether it is a virtio device or a regular PCI device. We use it as a virtio device because it acts as a backend of virtio devices, not sure if it could be used by other devices (I guess virtio would be the main paravirtualized-like device here)
On Thu, Dec 7, 2017 at 9:02 AM, Wei Wang <wei.w.wang@intel.com> wrote: > On 12/07/2017 02:31 PM, Stefan Hajnoczi wrote: >> >> On Thu, Dec 7, 2017 at 3:57 AM, Wei Wang <wei.w.wang@intel.com> wrote: >>> >>> On 12/07/2017 12:27 AM, Stefan Hajnoczi wrote: >>>> >>>> On Wed, Dec 6, 2017 at 4:09 PM, Wang, Wei W <wei.w.wang@intel.com> >>>> wrote: >>>>> >>>>> On Wednesday, December 6, 2017 9:50 PM, Stefan Hajnoczi wrote: >>>>>> >>>>>> On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: >>>>>>> >>>>>>> Vhost-pci is a point-to-point based inter-VM communication solution. >>>>>>> This patch series implements the vhost-pci-net device setup and >>>>>>> emulation. The device is implemented as a virtio device, and it is >>>>>>> set >>>>>>> up via the vhost-user protocol to get the neessary info (e.g the >>>>>>> memory info of the remote VM, vring info). >>>>>>> >>>>>>> Currently, only the fundamental functions are implemented. More >>>>>>> features, such as MQ and live migration, will be updated in the >>>>>>> future. >>>>>>> >>>>>>> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist >>>>>>> here: >>>>>>> http://dpdk.org/ml/archives/dev/2017-November/082615.html >>>>>> >>>>>> I have asked questions about the scope of this feature. In >>>>>> particular, >>>>>> I think >>>>>> it's best to support all device types rather than just virtio-net. >>>>>> Here >>>>>> is a >>>>>> design document that shows how this can be achieved. >>>>>> >>>>>> What I'm proposing is different from the current approach: >>>>>> 1. It's a PCI adapter (see below for justification) 2. The vhost-user >>>>>> protocol is >>>>>> exposed by the device (not handled 100% in >>>>>> QEMU). Ultimately I think your approach would also need to do >>>>>> this. >>>>>> >>>>>> I'm not implementing this and not asking you to implement it. Let's >>>>>> just use >>>>>> this for discussion so we can figure out what the final vhost-pci will >>>>>> look like. >>>>>> >>>>>> Please let me know what you think, Wei, Michael, and others. >>>>>> >>>>> Thanks for sharing the thoughts. If I understand it correctly, the key >>>>> difference is that this approach tries to relay every vhost-user msg to >>>>> the >>>>> guest. I'm not sure about the benefits of doing this. >>>>> To make data plane (i.e. driver to send/receive packets) work, I think, >>>>> mostly, the memory info and vring info are enough. Other things like >>>>> callfd, >>>>> kickfd don't need to be sent to the guest, they are needed by QEMU only >>>>> for >>>>> the eventfd and irqfd setup. >>>> >>>> Handling the vhost-user protocol inside QEMU and exposing a different >>>> interface to the guest makes the interface device-specific. This will >>>> cause extra work to support new devices (vhost-user-scsi, >>>> vhost-user-blk). It also makes development harder because you might >>>> have to learn 3 separate specifications to debug the system (virtio, >>>> vhost-user, vhost-pci-net). >>>> >>>> If vhost-user is mapped to a PCI device then these issues are solved. >>> >>> >>> I intend to have a different opinion about this: >>> >>> 1) Even relaying the msgs to the guest, QEMU still need to handle the msg >>> first, for example, it needs to decode the msg to see if it is the ones >>> (e.g. SET_MEM_TABLE, SET_VRING_KICK, SET_VRING_CALL) that should be used >>> for >>> the device setup (e.g. mmap the memory given via SET_MEM_TABLE). In this >>> case, we will be likely to have 2 slave handlers - one in the guest, >>> another >>> in QEMU device. >> >> In theory the vhost-pci PCI adapter could decide not to relay certain >> messages. As explained in the document, I think it's better to relay >> everything because some messages that only carry an fd still have a >> meaning. > > > It has its meaning, which is useful for the device setup, but that's not > useful for the guest. I think the point is most of the mgs are not useful > for the guest. > > IMHO, the relay mechanism is useful when > > 1) the QEMU slave handler doesn't need to process the messages at all > (receive and directly pass on to the guest) > > 2) most of the msgs are useful for the guest (say we have more than 20 msgs, > only 2 or 3 of them are useful for the guest, why let the device pass all of > them to the guest) > > Also the relay mechanism complicates the vhost-user protocol interaction: > normally, only master<->QemuSlave. With the relay mechanism, it will be > master<->QemuSlave<->GuestSlave. For example, when the master sends > VHOST_USER_GET_QUEUE_NUM, normally it can be answered by QemuSlave directly. > Why complicate it by passing the msg the GuestSlave, and then get the same > answer from GuestSlave. [...] >> They are a signal that the master has entered a new state. > > > Actually vhost-user isn't state-machine based. [...] >> Why have individual device types (vhost-pci-net, vhost-pci-blk, etc) >> instead of just a vhost-pci device? > > > This is the same as virtio - we don't have a single virtio device, we have > virtio-net, virtio-blk etc. > > So, the same way, we can have a common TYPE_VHOST_PCI_DEVICE parent device > (like TYPE_VIRTIO_DEVICE), but net may have its own special features like > MRG_RXBUF, and own config registers like mac[6] etc, so we can have > TYPE_VHOST_PCI_NET under TYPE_VHOST_PCI_DEVICE. [...] Instead of responding individually to these points, I hope this will explain my perspective. Let me know if you do want individual responses, I'm happy to talk more about the points above but I think the biggest difference is our perspective on this: Existing vhost-user slave code should be able to run on top of vhost-pci. For example, QEMU's contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest with only minimal changes to the source file (i.e. today it explicitly opens a UNIX domain socket and that should be done by libvhost-user instead). It shouldn't be hard to add vhost-pci vfio support to contrib/libvhost-user/ alongside the existing UNIX domain socket code. This seems pretty easy to achieve with the vhost-pci PCI adapter that I've described but I'm not sure how to implement libvhost-user on top of vhost-pci vfio if the device doesn't expose the vhost-user protocol. I think this is a really important goal. Let's use a single vhost-user software stack instead of creating a separate one for guest code only. Do you agree that the vhost-user software stack should be shared between host userspace and guest code as much as possible? >> >>>>>> vhost-pci is a PCI adapter instead of a virtio device to allow >>>>>> doorbells >>>>>> and >>>>>> interrupts to be connected to the virtio device in the master VM in >>>>>> the >>>>>> most >>>>>> efficient way possible. This means the Vring call doorbell can be an >>>>>> ioeventfd that signals an irqfd inside the host kernel without host >>>>>> userspace >>>>>> involvement. The Vring kick interrupt can be an irqfd that is >>>>>> signalled >>>>>> by the >>>>>> master VM's virtqueue ioeventfd. >>>>>> >>>>> This looks the same as the implementation of inter-VM notification in >>>>> v2: >>>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg450005.html >>>>> which is fig. 4 here: >>>>> >>>>> https://github.com/wei-w-wang/vhost-pci-discussion/blob/master/vhost-pci-rfc2.0.pdf >>>>> >>>>> When the vhost-pci driver kicks its tx, the host signals the irqfd of >>>>> virtio-net's rx. I think this has already bypassed the host userspace >>>>> (thanks to the fast mmio implementation) >>>> >>>> Yes, I think the irqfd <-> ioeventfd mapping is good. Perhaps it even >>>> makes sense to implement a special fused_irq_ioevent_fd in the host >>>> kernel to bypass the need for a kernel thread to read the eventfd so >>>> that an interrupt can be injected (i.e. to make the operation >>>> synchronous). >>>> >>>> Is the tx virtqueue in your inter-VM notification v2 series a real >>>> virtqueue that gets used? Or is it just a dummy virtqueue that you're >>>> using for the ioeventfd doorbell? It looks like vpnet_handle_vq() is >>>> empty so it's really just a dummy. The actual virtqueue is in the >>>> vhost-user master guest memory. >>> >>> >>> >>> Yes, that tx is a dummy actually, just created to use its doorbell. >>> Currently, with virtio_device, I think ioeventfd comes with virtqueue >>> only. >>> Actually, I think we could have the issues solved by vhost-pci. For >>> example, >>> reserve a piece of the BAR area for ioeventfd. The bar layout can be: >>> BAR 2: >>> 0~4k: vhost-pci device specific usages (ioeventfd etc) >>> 4k~8k: metadata (memory info and vring info) >>> 8k~64GB: remote guest memory >>> (we can make the bar size (64GB is the default value used) configurable >>> via >>> qemu cmdline) >> >> Why use a virtio device? The doorbell and shared memory don't fit the >> virtio architecture. There are no real virtqueues. This makes it a >> strange virtio device. > > > > The virtio spec doesn't seem to require the device to have at lease one > virtqueue. It doesn't make a huge difference to me whether it is a virtio > device or a regular PCI device. We use it as a virtio device because it acts > as a backend of virtio devices, not sure if it could be used by other > devices (I guess virtio would be the main paravirtualized-like device here) If virtio was symmetric then I would agree that the device backend should be a virtio device too. Unfortunately the virtio device model is assymmetric - the driver and the device play unique roles and you cannot invert them. Two examples of assymmetry: 1. Virtqueue memory ownership (we've already discussed this). Normally the driver allocates vrings but this doesn't work for the vhost-pci device. But the vhost-pci device still needs a doorbell to signal. 2. Configuration space is accessed using reads/writes by the driver and updates are signalled using the configuration space change interrupt by the device. The vhost-pci driver cannot get the same semantics by reading/writing virtio configuration space so a separate doorbell is necessary. What I'm getting at is that vhost-pci doesn't fit the virtio device model. It is possible to abuse virtqueues as doorbells, add BARs that are not part of the virtio device model, etc but it's cleaner to use a PCI adapter. No hacks are necessary with a PCI adapter. Stefan
On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: > Instead of responding individually to these points, I hope this will > explain my perspective. Let me know if you do want individual > responses, I'm happy to talk more about the points above but I think > the biggest difference is our perspective on this: > > Existing vhost-user slave code should be able to run on top of > vhost-pci. For example, QEMU's > contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest > with only minimal changes to the source file (i.e. today it explicitly > opens a UNIX domain socket and that should be done by libvhost-user > instead). It shouldn't be hard to add vhost-pci vfio support to > contrib/libvhost-user/ alongside the existing UNIX domain socket code. > > This seems pretty easy to achieve with the vhost-pci PCI adapter that > I've described but I'm not sure how to implement libvhost-user on top > of vhost-pci vfio if the device doesn't expose the vhost-user > protocol. > > I think this is a really important goal. Let's use a single > vhost-user software stack instead of creating a separate one for guest > code only. > > Do you agree that the vhost-user software stack should be shared > between host userspace and guest code as much as possible? The sharing you propose is not necessarily practical because the security goals of the two are different. It seems that the best motivation presentation is still the original rfc http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication So comparing with vhost-user iotlb handling is different: With vhost-user guest trusts the vhost-user backend on the host. With vhost-pci we can strive to limit the trust to qemu only. The switch running within a VM does not have to be trusted. -- MST
On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: >> Instead of responding individually to these points, I hope this will >> explain my perspective. Let me know if you do want individual >> responses, I'm happy to talk more about the points above but I think >> the biggest difference is our perspective on this: >> >> Existing vhost-user slave code should be able to run on top of >> vhost-pci. For example, QEMU's >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest >> with only minimal changes to the source file (i.e. today it explicitly >> opens a UNIX domain socket and that should be done by libvhost-user >> instead). It shouldn't be hard to add vhost-pci vfio support to >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. >> >> This seems pretty easy to achieve with the vhost-pci PCI adapter that >> I've described but I'm not sure how to implement libvhost-user on top >> of vhost-pci vfio if the device doesn't expose the vhost-user >> protocol. >> >> I think this is a really important goal. Let's use a single >> vhost-user software stack instead of creating a separate one for guest >> code only. >> >> Do you agree that the vhost-user software stack should be shared >> between host userspace and guest code as much as possible? > > > > The sharing you propose is not necessarily practical because the security goals > of the two are different. > > It seems that the best motivation presentation is still the original rfc > > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication > > So comparing with vhost-user iotlb handling is different: > > With vhost-user guest trusts the vhost-user backend on the host. > > With vhost-pci we can strive to limit the trust to qemu only. > The switch running within a VM does not have to be trusted. Can you give a concrete example? I have an idea about what you're saying but it may be wrong: Today the iotlb mechanism in vhost-user does not actually enforce memory permissions. The vhost-user slave has full access to mmapped memory regions even when iotlb is enabled. Currently the iotlb just adds an indirection layer but no real security. (Is this correct?) Are you saying the vhost-pci device code in QEMU should enforce iotlb permissions so the vhost-user slave guest only has access to memory regions that are allowed by the iotlb? This is a weak way to enforce memory permissions. If the guest is able to exploit a bug in QEMU then it has full memory access. It's a security problem waiting to happen and QEMU generally doesn't implement things this way. A stronger solution is for the vhost-user master to control memory protection and to disallow the vhost-user slave from changing memory protection. I think the kernel mechanism to support this does not exist today. Such a mechanism would also make the vhost-user host userspace use case secure. The kernel mechanism to do this would definitely be useful outside of virtualization too. Stefan
On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: > On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: > >> Instead of responding individually to these points, I hope this will > >> explain my perspective. Let me know if you do want individual > >> responses, I'm happy to talk more about the points above but I think > >> the biggest difference is our perspective on this: > >> > >> Existing vhost-user slave code should be able to run on top of > >> vhost-pci. For example, QEMU's > >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest > >> with only minimal changes to the source file (i.e. today it explicitly > >> opens a UNIX domain socket and that should be done by libvhost-user > >> instead). It shouldn't be hard to add vhost-pci vfio support to > >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. > >> > >> This seems pretty easy to achieve with the vhost-pci PCI adapter that > >> I've described but I'm not sure how to implement libvhost-user on top > >> of vhost-pci vfio if the device doesn't expose the vhost-user > >> protocol. > >> > >> I think this is a really important goal. Let's use a single > >> vhost-user software stack instead of creating a separate one for guest > >> code only. > >> > >> Do you agree that the vhost-user software stack should be shared > >> between host userspace and guest code as much as possible? > > > > > > > > The sharing you propose is not necessarily practical because the security goals > > of the two are different. > > > > It seems that the best motivation presentation is still the original rfc > > > > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication > > > > So comparing with vhost-user iotlb handling is different: > > > > With vhost-user guest trusts the vhost-user backend on the host. > > > > With vhost-pci we can strive to limit the trust to qemu only. > > The switch running within a VM does not have to be trusted. > > Can you give a concrete example? > > I have an idea about what you're saying but it may be wrong: > > Today the iotlb mechanism in vhost-user does not actually enforce > memory permissions. The vhost-user slave has full access to mmapped > memory regions even when iotlb is enabled. Currently the iotlb just > adds an indirection layer but no real security. (Is this correct?) Not exactly. iotlb protects against malicious drivers within guest. But yes, not against a vhost-user driver on the host. > Are you saying the vhost-pci device code in QEMU should enforce iotlb > permissions so the vhost-user slave guest only has access to memory > regions that are allowed by the iotlb? Yes. > This is a weak way to enforce memory permissions. If the guest is > able to exploit a bug in QEMU then it has full memory access. That's par for the course though. We don't have many of these. If you assume qemu is insecure, using a theoretical kernel-based mechanism does not add much since kernel exploits are pretty common too :). > It's a > security problem waiting to happen It's better security than running the switch on host though. > and QEMU generally doesn't > implement things this way. Not sure what does this mean. > A stronger solution is for the vhost-user master to control memory > protection and to disallow the vhost-user slave from changing memory > protection. I think the kernel mechanism to support this does not > exist today. Such a mechanism would also make the vhost-user host > userspace use case secure. The kernel mechanism to do this would > definitely be useful outside of virtualization too. > > Stefan In theory, maybe. But I'm not up to implementing this, it is very far from trivial. We can do a QEMU based one and then add the kernel based one on top when it surfaces. Also I forgot - this has some performance advantages too. -- MST
On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: >> >> Instead of responding individually to these points, I hope this will >> >> explain my perspective. Let me know if you do want individual >> >> responses, I'm happy to talk more about the points above but I think >> >> the biggest difference is our perspective on this: >> >> >> >> Existing vhost-user slave code should be able to run on top of >> >> vhost-pci. For example, QEMU's >> >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest >> >> with only minimal changes to the source file (i.e. today it explicitly >> >> opens a UNIX domain socket and that should be done by libvhost-user >> >> instead). It shouldn't be hard to add vhost-pci vfio support to >> >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. >> >> >> >> This seems pretty easy to achieve with the vhost-pci PCI adapter that >> >> I've described but I'm not sure how to implement libvhost-user on top >> >> of vhost-pci vfio if the device doesn't expose the vhost-user >> >> protocol. >> >> >> >> I think this is a really important goal. Let's use a single >> >> vhost-user software stack instead of creating a separate one for guest >> >> code only. >> >> >> >> Do you agree that the vhost-user software stack should be shared >> >> between host userspace and guest code as much as possible? >> > >> > >> > >> > The sharing you propose is not necessarily practical because the security goals >> > of the two are different. >> > >> > It seems that the best motivation presentation is still the original rfc >> > >> > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication >> > >> > So comparing with vhost-user iotlb handling is different: >> > >> > With vhost-user guest trusts the vhost-user backend on the host. >> > >> > With vhost-pci we can strive to limit the trust to qemu only. >> > The switch running within a VM does not have to be trusted. >> >> Can you give a concrete example? >> >> I have an idea about what you're saying but it may be wrong: >> >> Today the iotlb mechanism in vhost-user does not actually enforce >> memory permissions. The vhost-user slave has full access to mmapped >> memory regions even when iotlb is enabled. Currently the iotlb just >> adds an indirection layer but no real security. (Is this correct?) > > Not exactly. iotlb protects against malicious drivers within guest. > But yes, not against a vhost-user driver on the host. > >> Are you saying the vhost-pci device code in QEMU should enforce iotlb >> permissions so the vhost-user slave guest only has access to memory >> regions that are allowed by the iotlb? > > Yes. Okay, thanks for confirming. This can be supported by the approach I've described. The vhost-pci QEMU code has control over the BAR memory so it can prevent the guest from accessing regions that are not allowed by the iotlb. Inside the guest the vhost-user slave still has the memory region descriptions and sends iotlb messages. This is completely compatible with the libvirt-user APIs and existing vhost-user slave code can run fine. The only unique thing is that guest accesses to memory regions not allowed by the iotlb do not work because QEMU has prevented it. If better performance is needed then it might be possible to optimize this interface by handling most or even all of the iotlb stuff in QEMU vhost-pci code and not exposing it to the vhost-user slave in the guest. But it doesn't change the fact that the vhost-user protocol can be used and the same software stack works. Do you have a concrete example of why sharing the same vhost-user software stack inside the guest can't work? >> and QEMU generally doesn't >> implement things this way. > > Not sure what does this mean. It's the reason why virtio-9p has a separate virtfs-proxy-helper program. Root is needed to set file uid/gids. Instead of running QEMU as root, there is a separate helper process that handles the privileged operations. It slows things down and makes the codebase larger but it prevents the guest from getting root in case of QEMU bugs. The reason why VMs are considered more secure than containers is because of the extra level of isolation provided by running device emulation in an unprivileged userspace process. If you change this model then QEMU loses the "security in depth" advantage. Stefan
On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: > On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: > >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: > >> >> Instead of responding individually to these points, I hope this will > >> >> explain my perspective. Let me know if you do want individual > >> >> responses, I'm happy to talk more about the points above but I think > >> >> the biggest difference is our perspective on this: > >> >> > >> >> Existing vhost-user slave code should be able to run on top of > >> >> vhost-pci. For example, QEMU's > >> >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest > >> >> with only minimal changes to the source file (i.e. today it explicitly > >> >> opens a UNIX domain socket and that should be done by libvhost-user > >> >> instead). It shouldn't be hard to add vhost-pci vfio support to > >> >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. > >> >> > >> >> This seems pretty easy to achieve with the vhost-pci PCI adapter that > >> >> I've described but I'm not sure how to implement libvhost-user on top > >> >> of vhost-pci vfio if the device doesn't expose the vhost-user > >> >> protocol. > >> >> > >> >> I think this is a really important goal. Let's use a single > >> >> vhost-user software stack instead of creating a separate one for guest > >> >> code only. > >> >> > >> >> Do you agree that the vhost-user software stack should be shared > >> >> between host userspace and guest code as much as possible? > >> > > >> > > >> > > >> > The sharing you propose is not necessarily practical because the security goals > >> > of the two are different. > >> > > >> > It seems that the best motivation presentation is still the original rfc > >> > > >> > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication > >> > > >> > So comparing with vhost-user iotlb handling is different: > >> > > >> > With vhost-user guest trusts the vhost-user backend on the host. > >> > > >> > With vhost-pci we can strive to limit the trust to qemu only. > >> > The switch running within a VM does not have to be trusted. > >> > >> Can you give a concrete example? > >> > >> I have an idea about what you're saying but it may be wrong: > >> > >> Today the iotlb mechanism in vhost-user does not actually enforce > >> memory permissions. The vhost-user slave has full access to mmapped > >> memory regions even when iotlb is enabled. Currently the iotlb just > >> adds an indirection layer but no real security. (Is this correct?) > > > > Not exactly. iotlb protects against malicious drivers within guest. > > But yes, not against a vhost-user driver on the host. > > > >> Are you saying the vhost-pci device code in QEMU should enforce iotlb > >> permissions so the vhost-user slave guest only has access to memory > >> regions that are allowed by the iotlb? > > > > Yes. > > Okay, thanks for confirming. > > This can be supported by the approach I've described. The vhost-pci > QEMU code has control over the BAR memory so it can prevent the guest > from accessing regions that are not allowed by the iotlb. > > Inside the guest the vhost-user slave still has the memory region > descriptions and sends iotlb messages. This is completely compatible > with the libvirt-user APIs and existing vhost-user slave code can run > fine. The only unique thing is that guest accesses to memory regions > not allowed by the iotlb do not work because QEMU has prevented it. I don't think this can work since suddenly you need to map full IOMMU address space into BAR. Besides, this means implementing iotlb in both qemu and guest. > If better performance is needed then it might be possible to optimize > this interface by handling most or even all of the iotlb stuff in QEMU > vhost-pci code and not exposing it to the vhost-user slave in the > guest. But it doesn't change the fact that the vhost-user protocol > can be used and the same software stack works. For one, the iotlb part would be out of scope then. Instead you would have code to offset from BAR. > Do you have a concrete example of why sharing the same vhost-user > software stack inside the guest can't work? With enough dedication some code might be shared. OTOH reusing virtio gains you a ready feature negotiation and discovery protocol. I'm not convinced which has more value, and the second proposal has been implemented already. > >> and QEMU generally doesn't > >> implement things this way. > > > > Not sure what does this mean. > > It's the reason why virtio-9p has a separate virtfs-proxy-helper > program. Root is needed to set file uid/gids. Instead of running > QEMU as root, there is a separate helper process that handles the > privileged operations. It slows things down and makes the codebase > larger but it prevents the guest from getting root in case of QEMU > bugs. > > The reason why VMs are considered more secure than containers is > because of the extra level of isolation provided by running device > emulation in an unprivileged userspace process. If you change this > model then QEMU loses the "security in depth" advantage. > > Stefan I don't see where vhost-pci needs QEMU to run as root though.
On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: >> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: >> >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: >> >> >> Instead of responding individually to these points, I hope this will >> >> >> explain my perspective. Let me know if you do want individual >> >> >> responses, I'm happy to talk more about the points above but I think >> >> >> the biggest difference is our perspective on this: >> >> >> >> >> >> Existing vhost-user slave code should be able to run on top of >> >> >> vhost-pci. For example, QEMU's >> >> >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest >> >> >> with only minimal changes to the source file (i.e. today it explicitly >> >> >> opens a UNIX domain socket and that should be done by libvhost-user >> >> >> instead). It shouldn't be hard to add vhost-pci vfio support to >> >> >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. >> >> >> >> >> >> This seems pretty easy to achieve with the vhost-pci PCI adapter that >> >> >> I've described but I'm not sure how to implement libvhost-user on top >> >> >> of vhost-pci vfio if the device doesn't expose the vhost-user >> >> >> protocol. >> >> >> >> >> >> I think this is a really important goal. Let's use a single >> >> >> vhost-user software stack instead of creating a separate one for guest >> >> >> code only. >> >> >> >> >> >> Do you agree that the vhost-user software stack should be shared >> >> >> between host userspace and guest code as much as possible? >> >> > >> >> > >> >> > >> >> > The sharing you propose is not necessarily practical because the security goals >> >> > of the two are different. >> >> > >> >> > It seems that the best motivation presentation is still the original rfc >> >> > >> >> > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication >> >> > >> >> > So comparing with vhost-user iotlb handling is different: >> >> > >> >> > With vhost-user guest trusts the vhost-user backend on the host. >> >> > >> >> > With vhost-pci we can strive to limit the trust to qemu only. >> >> > The switch running within a VM does not have to be trusted. >> >> >> >> Can you give a concrete example? >> >> >> >> I have an idea about what you're saying but it may be wrong: >> >> >> >> Today the iotlb mechanism in vhost-user does not actually enforce >> >> memory permissions. The vhost-user slave has full access to mmapped >> >> memory regions even when iotlb is enabled. Currently the iotlb just >> >> adds an indirection layer but no real security. (Is this correct?) >> > >> > Not exactly. iotlb protects against malicious drivers within guest. >> > But yes, not against a vhost-user driver on the host. >> > >> >> Are you saying the vhost-pci device code in QEMU should enforce iotlb >> >> permissions so the vhost-user slave guest only has access to memory >> >> regions that are allowed by the iotlb? >> > >> > Yes. >> >> Okay, thanks for confirming. >> >> This can be supported by the approach I've described. The vhost-pci >> QEMU code has control over the BAR memory so it can prevent the guest >> from accessing regions that are not allowed by the iotlb. >> >> Inside the guest the vhost-user slave still has the memory region >> descriptions and sends iotlb messages. This is completely compatible >> with the libvirt-user APIs and existing vhost-user slave code can run >> fine. The only unique thing is that guest accesses to memory regions >> not allowed by the iotlb do not work because QEMU has prevented it. > > I don't think this can work since suddenly you need > to map full IOMMU address space into BAR. The BAR covers all guest RAM but QEMU can set up MemoryRegions that hide parts from the guest (e.g. reads produce 0xff). I'm not sure how expensive that is but implementing a strict IOMMU is hard to do without performance overhead. > Besides, this means implementing iotlb in both qemu and guest. It's free in the guest, the libvhost-user stack already has it. >> If better performance is needed then it might be possible to optimize >> this interface by handling most or even all of the iotlb stuff in QEMU >> vhost-pci code and not exposing it to the vhost-user slave in the >> guest. But it doesn't change the fact that the vhost-user protocol >> can be used and the same software stack works. > > For one, the iotlb part would be out of scope then. > Instead you would have code to offset from BAR. > >> Do you have a concrete example of why sharing the same vhost-user >> software stack inside the guest can't work? > > With enough dedication some code might be shared. OTOH reusing virtio > gains you a ready feature negotiation and discovery protocol. > > I'm not convinced which has more value, and the second proposal > has been implemented already. Thanks to you and Wei for the discussion. I've learnt a lot about vhost-user. If you have questions about what I've posted, please let me know and we can discuss further. The decision is not up to me so I'll just summarize what the vhost-pci PCI adapter approach achieves: 1. Just one device and driver 2. Support for all device types (net, scsi, blk, etc) 3. Reuse of software stack so vhost-user slaves can run in both host userspace and the guest 4. Simpler to debug because the vhost-user protocol used by QEMU is also used by the guest Stefan
On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: > >> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: > >> >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: > >> >> >> Instead of responding individually to these points, I hope this will > >> >> >> explain my perspective. Let me know if you do want individual > >> >> >> responses, I'm happy to talk more about the points above but I think > >> >> >> the biggest difference is our perspective on this: > >> >> >> > >> >> >> Existing vhost-user slave code should be able to run on top of > >> >> >> vhost-pci. For example, QEMU's > >> >> >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest > >> >> >> with only minimal changes to the source file (i.e. today it explicitly > >> >> >> opens a UNIX domain socket and that should be done by libvhost-user > >> >> >> instead). It shouldn't be hard to add vhost-pci vfio support to > >> >> >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. > >> >> >> > >> >> >> This seems pretty easy to achieve with the vhost-pci PCI adapter that > >> >> >> I've described but I'm not sure how to implement libvhost-user on top > >> >> >> of vhost-pci vfio if the device doesn't expose the vhost-user > >> >> >> protocol. > >> >> >> > >> >> >> I think this is a really important goal. Let's use a single > >> >> >> vhost-user software stack instead of creating a separate one for guest > >> >> >> code only. > >> >> >> > >> >> >> Do you agree that the vhost-user software stack should be shared > >> >> >> between host userspace and guest code as much as possible? > >> >> > > >> >> > > >> >> > > >> >> > The sharing you propose is not necessarily practical because the security goals > >> >> > of the two are different. > >> >> > > >> >> > It seems that the best motivation presentation is still the original rfc > >> >> > > >> >> > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication > >> >> > > >> >> > So comparing with vhost-user iotlb handling is different: > >> >> > > >> >> > With vhost-user guest trusts the vhost-user backend on the host. > >> >> > > >> >> > With vhost-pci we can strive to limit the trust to qemu only. > >> >> > The switch running within a VM does not have to be trusted. > >> >> > >> >> Can you give a concrete example? > >> >> > >> >> I have an idea about what you're saying but it may be wrong: > >> >> > >> >> Today the iotlb mechanism in vhost-user does not actually enforce > >> >> memory permissions. The vhost-user slave has full access to mmapped > >> >> memory regions even when iotlb is enabled. Currently the iotlb just > >> >> adds an indirection layer but no real security. (Is this correct?) > >> > > >> > Not exactly. iotlb protects against malicious drivers within guest. > >> > But yes, not against a vhost-user driver on the host. > >> > > >> >> Are you saying the vhost-pci device code in QEMU should enforce iotlb > >> >> permissions so the vhost-user slave guest only has access to memory > >> >> regions that are allowed by the iotlb? > >> > > >> > Yes. > >> > >> Okay, thanks for confirming. > >> > >> This can be supported by the approach I've described. The vhost-pci > >> QEMU code has control over the BAR memory so it can prevent the guest > >> from accessing regions that are not allowed by the iotlb. > >> > >> Inside the guest the vhost-user slave still has the memory region > >> descriptions and sends iotlb messages. This is completely compatible > >> with the libvirt-user APIs and existing vhost-user slave code can run > >> fine. The only unique thing is that guest accesses to memory regions > >> not allowed by the iotlb do not work because QEMU has prevented it. > > > > I don't think this can work since suddenly you need > > to map full IOMMU address space into BAR. > > The BAR covers all guest RAM > but QEMU can set up MemoryRegions that > hide parts from the guest (e.g. reads produce 0xff). I'm not sure how > expensive that is but implementing a strict IOMMU is hard to do > without performance overhead. I'm worried about leaking PAs. fundamentally if you want proper protection you need your device driver to use VA for addressing, On the one hand BAR only needs to be as large as guest PA then. On the other hand it must cover all of guest PA, not just what is accessible to the device. > > > Besides, this means implementing iotlb in both qemu and guest. > > It's free in the guest, the libvhost-user stack already has it. That library is designed to work with a unix domain socket though. We'll need extra kernel code to make a device pretend it's a socket. > >> If better performance is needed then it might be possible to optimize > >> this interface by handling most or even all of the iotlb stuff in QEMU > >> vhost-pci code and not exposing it to the vhost-user slave in the > >> guest. But it doesn't change the fact that the vhost-user protocol > >> can be used and the same software stack works. > > > > For one, the iotlb part would be out of scope then. > > Instead you would have code to offset from BAR. > > > >> Do you have a concrete example of why sharing the same vhost-user > >> software stack inside the guest can't work? > > > > With enough dedication some code might be shared. OTOH reusing virtio > > gains you a ready feature negotiation and discovery protocol. > > > > I'm not convinced which has more value, and the second proposal > > has been implemented already. > > Thanks to you and Wei for the discussion. I've learnt a lot about > vhost-user. If you have questions about what I've posted, please let > me know and we can discuss further. > > The decision is not up to me so I'll just summarize what the vhost-pci > PCI adapter approach achieves: > 1. Just one device and driver > 2. Support for all device types (net, scsi, blk, etc) > 3. Reuse of software stack so vhost-user slaves can run in both host > userspace and the guest > 4. Simpler to debug because the vhost-user protocol used by QEMU is > also used by the guest > > Stefan
On Thu, Dec 7, 2017 at 11:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: >> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> > On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: >> >> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: >> >> >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: >> >> >> >> Instead of responding individually to these points, I hope this will >> >> >> >> explain my perspective. Let me know if you do want individual >> >> >> >> responses, I'm happy to talk more about the points above but I think >> >> >> >> the biggest difference is our perspective on this: >> >> >> >> >> >> >> >> Existing vhost-user slave code should be able to run on top of >> >> >> >> vhost-pci. For example, QEMU's >> >> >> >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest >> >> >> >> with only minimal changes to the source file (i.e. today it explicitly >> >> >> >> opens a UNIX domain socket and that should be done by libvhost-user >> >> >> >> instead). It shouldn't be hard to add vhost-pci vfio support to >> >> >> >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. >> >> >> >> >> >> >> >> This seems pretty easy to achieve with the vhost-pci PCI adapter that >> >> >> >> I've described but I'm not sure how to implement libvhost-user on top >> >> >> >> of vhost-pci vfio if the device doesn't expose the vhost-user >> >> >> >> protocol. >> >> >> >> >> >> >> >> I think this is a really important goal. Let's use a single >> >> >> >> vhost-user software stack instead of creating a separate one for guest >> >> >> >> code only. >> >> >> >> >> >> >> >> Do you agree that the vhost-user software stack should be shared >> >> >> >> between host userspace and guest code as much as possible? >> >> >> > >> >> >> > >> >> >> > >> >> >> > The sharing you propose is not necessarily practical because the security goals >> >> >> > of the two are different. >> >> >> > >> >> >> > It seems that the best motivation presentation is still the original rfc >> >> >> > >> >> >> > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication >> >> >> > >> >> >> > So comparing with vhost-user iotlb handling is different: >> >> >> > >> >> >> > With vhost-user guest trusts the vhost-user backend on the host. >> >> >> > >> >> >> > With vhost-pci we can strive to limit the trust to qemu only. >> >> >> > The switch running within a VM does not have to be trusted. >> >> >> >> >> >> Can you give a concrete example? >> >> >> >> >> >> I have an idea about what you're saying but it may be wrong: >> >> >> >> >> >> Today the iotlb mechanism in vhost-user does not actually enforce >> >> >> memory permissions. The vhost-user slave has full access to mmapped >> >> >> memory regions even when iotlb is enabled. Currently the iotlb just >> >> >> adds an indirection layer but no real security. (Is this correct?) >> >> > >> >> > Not exactly. iotlb protects against malicious drivers within guest. >> >> > But yes, not against a vhost-user driver on the host. >> >> > >> >> >> Are you saying the vhost-pci device code in QEMU should enforce iotlb >> >> >> permissions so the vhost-user slave guest only has access to memory >> >> >> regions that are allowed by the iotlb? >> >> > >> >> > Yes. >> >> >> >> Okay, thanks for confirming. >> >> >> >> This can be supported by the approach I've described. The vhost-pci >> >> QEMU code has control over the BAR memory so it can prevent the guest >> >> from accessing regions that are not allowed by the iotlb. >> >> >> >> Inside the guest the vhost-user slave still has the memory region >> >> descriptions and sends iotlb messages. This is completely compatible >> >> with the libvirt-user APIs and existing vhost-user slave code can run >> >> fine. The only unique thing is that guest accesses to memory regions >> >> not allowed by the iotlb do not work because QEMU has prevented it. >> > >> > I don't think this can work since suddenly you need >> > to map full IOMMU address space into BAR. >> >> The BAR covers all guest RAM >> but QEMU can set up MemoryRegions that >> hide parts from the guest (e.g. reads produce 0xff). I'm not sure how >> expensive that is but implementing a strict IOMMU is hard to do >> without performance overhead. > > I'm worried about leaking PAs. > fundamentally if you want proper protection you > need your device driver to use VA for addressing, > > On the one hand BAR only needs to be as large as guest PA then. > On the other hand it must cover all of guest PA, > not just what is accessible to the device. A more heavyweight iotlb implementation in QEMU's vhost-pci device could present only VAs to the vhost-pci driver. It would use MemoryRegions to map pieces of shared guest memory dynamically. The only information leak would be the overall guest RAM size because we still need to set the correct BAR size. >> >> > Besides, this means implementing iotlb in both qemu and guest. >> >> It's free in the guest, the libvhost-user stack already has it. > > That library is designed to work with a unix domain socket > though. We'll need extra kernel code to make a device > pretend it's a socket. A kernel vhost-pci driver isn't necessary because I don't think there are in-kernel users. A vfio vhost-pci backend can go alongside the UNIX domain socket backend that exists today in libvhost-user. If we want to expose kernel vhost devices via vhost-pci then a libvhost-user program can translate the vhost-user protocol into kernel ioctls. For example: $ vhost-pci-proxy --vhost-pci-addr 00:04.0 --vhost-fd 3 3<>/dev/vhost-net The vhost-pci-proxy implements the vhost-user protocol callbacks and submits ioctls on the vhost kernel device fd. I haven't compared the kernel ioctl interface vs the vhost-user protocol to see if everything maps cleanly though. Stefan
On Fri, Dec 08, 2017 at 06:08:05AM +0000, Stefan Hajnoczi wrote: > On Thu, Dec 7, 2017 at 11:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > >> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> > On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: > >> >> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> >> > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: > >> >> >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> >> >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: > >> >> >> >> Instead of responding individually to these points, I hope this will > >> >> >> >> explain my perspective. Let me know if you do want individual > >> >> >> >> responses, I'm happy to talk more about the points above but I think > >> >> >> >> the biggest difference is our perspective on this: > >> >> >> >> > >> >> >> >> Existing vhost-user slave code should be able to run on top of > >> >> >> >> vhost-pci. For example, QEMU's > >> >> >> >> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest > >> >> >> >> with only minimal changes to the source file (i.e. today it explicitly > >> >> >> >> opens a UNIX domain socket and that should be done by libvhost-user > >> >> >> >> instead). It shouldn't be hard to add vhost-pci vfio support to > >> >> >> >> contrib/libvhost-user/ alongside the existing UNIX domain socket code. > >> >> >> >> > >> >> >> >> This seems pretty easy to achieve with the vhost-pci PCI adapter that > >> >> >> >> I've described but I'm not sure how to implement libvhost-user on top > >> >> >> >> of vhost-pci vfio if the device doesn't expose the vhost-user > >> >> >> >> protocol. > >> >> >> >> > >> >> >> >> I think this is a really important goal. Let's use a single > >> >> >> >> vhost-user software stack instead of creating a separate one for guest > >> >> >> >> code only. > >> >> >> >> > >> >> >> >> Do you agree that the vhost-user software stack should be shared > >> >> >> >> between host userspace and guest code as much as possible? > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > The sharing you propose is not necessarily practical because the security goals > >> >> >> > of the two are different. > >> >> >> > > >> >> >> > It seems that the best motivation presentation is still the original rfc > >> >> >> > > >> >> >> > http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication > >> >> >> > > >> >> >> > So comparing with vhost-user iotlb handling is different: > >> >> >> > > >> >> >> > With vhost-user guest trusts the vhost-user backend on the host. > >> >> >> > > >> >> >> > With vhost-pci we can strive to limit the trust to qemu only. > >> >> >> > The switch running within a VM does not have to be trusted. > >> >> >> > >> >> >> Can you give a concrete example? > >> >> >> > >> >> >> I have an idea about what you're saying but it may be wrong: > >> >> >> > >> >> >> Today the iotlb mechanism in vhost-user does not actually enforce > >> >> >> memory permissions. The vhost-user slave has full access to mmapped > >> >> >> memory regions even when iotlb is enabled. Currently the iotlb just > >> >> >> adds an indirection layer but no real security. (Is this correct?) > >> >> > > >> >> > Not exactly. iotlb protects against malicious drivers within guest. > >> >> > But yes, not against a vhost-user driver on the host. > >> >> > > >> >> >> Are you saying the vhost-pci device code in QEMU should enforce iotlb > >> >> >> permissions so the vhost-user slave guest only has access to memory > >> >> >> regions that are allowed by the iotlb? > >> >> > > >> >> > Yes. > >> >> > >> >> Okay, thanks for confirming. > >> >> > >> >> This can be supported by the approach I've described. The vhost-pci > >> >> QEMU code has control over the BAR memory so it can prevent the guest > >> >> from accessing regions that are not allowed by the iotlb. > >> >> > >> >> Inside the guest the vhost-user slave still has the memory region > >> >> descriptions and sends iotlb messages. This is completely compatible > >> >> with the libvirt-user APIs and existing vhost-user slave code can run > >> >> fine. The only unique thing is that guest accesses to memory regions > >> >> not allowed by the iotlb do not work because QEMU has prevented it. > >> > > >> > I don't think this can work since suddenly you need > >> > to map full IOMMU address space into BAR. > >> > >> The BAR covers all guest RAM > >> but QEMU can set up MemoryRegions that > >> hide parts from the guest (e.g. reads produce 0xff). I'm not sure how > >> expensive that is but implementing a strict IOMMU is hard to do > >> without performance overhead. > > > > I'm worried about leaking PAs. > > fundamentally if you want proper protection you > > need your device driver to use VA for addressing, > > > > On the one hand BAR only needs to be as large as guest PA then. > > On the other hand it must cover all of guest PA, > > not just what is accessible to the device. > > A more heavyweight iotlb implementation in QEMU's vhost-pci device > could present only VAs to the vhost-pci driver. It would use > MemoryRegions to map pieces of shared guest memory dynamically. The > only information leak would be the overall guest RAM size because we > still need to set the correct BAR size. I'm not sure this will work. KVM simply isn't designed with a huge number of fragmented regions in mind. Wei, just what is the plan for the IOMMU? How will all virtual addresses fit in a BAR? Maybe we really do want a non-translating IOMMU (leaking PA to userspace but oh well)? > >> > >> > Besides, this means implementing iotlb in both qemu and guest. > >> > >> It's free in the guest, the libvhost-user stack already has it. > > > > That library is designed to work with a unix domain socket > > though. We'll need extra kernel code to make a device > > pretend it's a socket. > > A kernel vhost-pci driver isn't necessary because I don't think there > are in-kernel users. > > A vfio vhost-pci backend can go alongside the UNIX domain socket > backend that exists today in libvhost-user. > > If we want to expose kernel vhost devices via vhost-pci then a > libvhost-user program can translate the vhost-user protocol into > kernel ioctls. For example: > $ vhost-pci-proxy --vhost-pci-addr 00:04.0 --vhost-fd 3 3<>/dev/vhost-net > > The vhost-pci-proxy implements the vhost-user protocol callbacks and > submits ioctls on the vhost kernel device fd. I haven't compared the > kernel ioctl interface vs the vhost-user protocol to see if everything > maps cleanly though. > > Stefan I don't really like this, it's yet another package to install, yet another process to complicate debugging and yet another service that can go down. Maybe vsock can do the trick though? -- MST
On Fri, Dec 8, 2017 at 2:27 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Dec 08, 2017 at 06:08:05AM +0000, Stefan Hajnoczi wrote: >> On Thu, Dec 7, 2017 at 11:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> > On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: >> >> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> > On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: >> >> >> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> >> > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: >> >> >> >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> >> >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: >> >> >> >> > Besides, this means implementing iotlb in both qemu and guest. >> >> >> >> It's free in the guest, the libvhost-user stack already has it. >> > >> > That library is designed to work with a unix domain socket >> > though. We'll need extra kernel code to make a device >> > pretend it's a socket. >> >> A kernel vhost-pci driver isn't necessary because I don't think there >> are in-kernel users. >> >> A vfio vhost-pci backend can go alongside the UNIX domain socket >> backend that exists today in libvhost-user. >> >> If we want to expose kernel vhost devices via vhost-pci then a >> libvhost-user program can translate the vhost-user protocol into >> kernel ioctls. For example: >> $ vhost-pci-proxy --vhost-pci-addr 00:04.0 --vhost-fd 3 3<>/dev/vhost-net >> >> The vhost-pci-proxy implements the vhost-user protocol callbacks and >> submits ioctls on the vhost kernel device fd. I haven't compared the >> kernel ioctl interface vs the vhost-user protocol to see if everything >> maps cleanly though. >> >> Stefan > > I don't really like this, it's yet another package to install, yet > another process to complicate debugging and yet another service that can > go down. > > Maybe vsock can do the trick though? Not sure what you have in mind. An in-kernel vhost-pci driver is possible too. The neatest integration would be alongside drivers/vhost/vhost.c so that existing net, scsi, vsock vhost drivers can work with vhost-pci. Userspace still needs to configure the devices and associate them with a vhost-pci instance (e.g. which vhost-pci device should be a vhost_scsi target and the SCSI target configuration). But I think this approach is more work than the vhost-pci-proxy program I've described. Stefan
On Friday, December 8, 2017 10:28 PM, Michael S. Tsirkin wrote: > On Fri, Dec 08, 2017 at 06:08:05AM +0000, Stefan Hajnoczi wrote: > > On Thu, Dec 7, 2017 at 11:54 PM, Michael S. Tsirkin <mst@redhat.com> > wrote: > > > On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > > >> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: > > >> >> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> >> > On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: > > >> >> >> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin > <mst@redhat.com> wrote: > > >> >> >> > On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi > wrote: > > >> >> >> >> Instead of responding individually to these points, I hope > > >> >> >> >> this will explain my perspective. Let me know if you do > > >> >> >> >> want individual responses, I'm happy to talk more about > > >> >> >> >> the points above but I think the biggest difference is our > perspective on this: > > >> >> >> >> > > >> >> >> >> Existing vhost-user slave code should be able to run on > > >> >> >> >> top of vhost-pci. For example, QEMU's > > >> >> >> >> contrib/vhost-user-scsi/vhost-user-scsi.c should work > > >> >> >> >> inside the guest with only minimal changes to the source > > >> >> >> >> file (i.e. today it explicitly opens a UNIX domain socket > > >> >> >> >> and that should be done by libvhost-user instead). It > > >> >> >> >> shouldn't be hard to add vhost-pci vfio support to > contrib/libvhost-user/ alongside the existing UNIX domain socket code. > > >> >> >> >> > > >> >> >> >> This seems pretty easy to achieve with the vhost-pci PCI > > >> >> >> >> adapter that I've described but I'm not sure how to > > >> >> >> >> implement libvhost-user on top of vhost-pci vfio if the > > >> >> >> >> device doesn't expose the vhost-user protocol. > > >> >> >> >> > > >> >> >> >> I think this is a really important goal. Let's use a > > >> >> >> >> single vhost-user software stack instead of creating a > > >> >> >> >> separate one for guest code only. > > >> >> >> >> > > >> >> >> >> Do you agree that the vhost-user software stack should be > > >> >> >> >> shared between host userspace and guest code as much as > possible? > > >> >> >> > > > >> >> >> > > > >> >> >> > > > >> >> >> > The sharing you propose is not necessarily practical > > >> >> >> > because the security goals of the two are different. > > >> >> >> > > > >> >> >> > It seems that the best motivation presentation is still the > > >> >> >> > original rfc > > >> >> >> > > > >> >> >> > http://virtualization.linux-foundation.narkive.com/A7FkzAgp > > >> >> >> > /rfc-vhost-user-enhancements-for-vm2vm-communication > > >> >> >> > > > >> >> >> > So comparing with vhost-user iotlb handling is different: > > >> >> >> > > > >> >> >> > With vhost-user guest trusts the vhost-user backend on the host. > > >> >> >> > > > >> >> >> > With vhost-pci we can strive to limit the trust to qemu only. > > >> >> >> > The switch running within a VM does not have to be trusted. > > >> >> >> > > >> >> >> Can you give a concrete example? > > >> >> >> > > >> >> >> I have an idea about what you're saying but it may be wrong: > > >> >> >> > > >> >> >> Today the iotlb mechanism in vhost-user does not actually > > >> >> >> enforce memory permissions. The vhost-user slave has full > > >> >> >> access to mmapped memory regions even when iotlb is enabled. > > >> >> >> Currently the iotlb just adds an indirection layer but no > > >> >> >> real security. (Is this correct?) > > >> >> > > > >> >> > Not exactly. iotlb protects against malicious drivers within guest. > > >> >> > But yes, not against a vhost-user driver on the host. > > >> >> > > > >> >> >> Are you saying the vhost-pci device code in QEMU should > > >> >> >> enforce iotlb permissions so the vhost-user slave guest only > > >> >> >> has access to memory regions that are allowed by the iotlb? > > >> >> > > > >> >> > Yes. > > >> >> > > >> >> Okay, thanks for confirming. > > >> >> > > >> >> This can be supported by the approach I've described. The > > >> >> vhost-pci QEMU code has control over the BAR memory so it can > > >> >> prevent the guest from accessing regions that are not allowed by the > iotlb. > > >> >> > > >> >> Inside the guest the vhost-user slave still has the memory > > >> >> region descriptions and sends iotlb messages. This is > > >> >> completely compatible with the libvirt-user APIs and existing > > >> >> vhost-user slave code can run fine. The only unique thing is > > >> >> that guest accesses to memory regions not allowed by the iotlb do > not work because QEMU has prevented it. > > >> > > > >> > I don't think this can work since suddenly you need to map full > > >> > IOMMU address space into BAR. > > >> > > >> The BAR covers all guest RAM > > >> but QEMU can set up MemoryRegions that hide parts from the guest > > >> (e.g. reads produce 0xff). I'm not sure how expensive that is but > > >> implementing a strict IOMMU is hard to do without performance > > >> overhead. > > > > > > I'm worried about leaking PAs. > > > fundamentally if you want proper protection you need your device > > > driver to use VA for addressing, > > > > > > On the one hand BAR only needs to be as large as guest PA then. > > > On the other hand it must cover all of guest PA, not just what is > > > accessible to the device. > > > > A more heavyweight iotlb implementation in QEMU's vhost-pci device > > could present only VAs to the vhost-pci driver. It would use > > MemoryRegions to map pieces of shared guest memory dynamically. The > > only information leak would be the overall guest RAM size because we > > still need to set the correct BAR size. > > I'm not sure this will work. KVM simply > isn't designed with a huge number of fragmented regions in mind. > > Wei, just what is the plan for the IOMMU? How will all virtual addresses fit > in a BAR? > > Maybe we really do want a non-translating IOMMU (leaking PA to userspace > but oh well)? Yes, I have 2 ways of implementation in mind. Basically, I think we will leverage the slave side EPT to provide the accessible master memory to the slave guest. Before getting into that, please let me introduce some background first: here, VM1 is the slave VM, and VM2 is the master VM The VM2's memory, which is exposed to the vhost-pci driver, does not have to be mapped and added to the bar MemoryRegion when the device is plugged to VM1. They can be added later even after the driver does ioremap of the bar. Here is what this patch series has done: i. when VM1 boots, the bar MemoryRegion is initialized, and registered via pci_register_bar (in the vpnet_pci_realize function), the bar MemoryRegion is like a "skeleton" without any real memory added to it, except the 4KB metadata memory which is added to the top of bar MemoryRegion (in vpnet_device_realize) ii. when the vhost-pci driver probes, it does ioremap(bar, 64GB), which sets up the guest kernel mapping of the 64GB bar, but till now only the top 4KB memory has qva->gpa mapping, which will be referenced to create the EPT mapping when accessed. iii. when VM2 boots, the vhost-user master sends the VM2's memory info to VM1, then those memory are mapped by QEMU1 and updated to the vhost-pci bar MemoryRegion(in the vp_slave_set_mem_table function). After this, VM2's memory can appear in VM1's EPT when accessed. With the above understanding, here are the 2 possible options (which are similar actually) of IOMMU support: Option1: 1) In the above phase iii., when QEMU1 receives the memory info, it does not map and expose the entire memory to the guest, instead, it just stores the memory info there, e.g., info1 (fd1, base1, len1), info2 (fd2, base2, len2), info3(fd3, base3, len3) etc. 2) When VM2's virtio-net driver wants to set up dma remapping table entries, the dma_map_page function will trap to QEMU2 with iova and gpa (is this correct? I'll double checked this part) 3) then QEMU2 sends an IOTLB_MSG(iova1, gpa1, size1) to QEMU1 - iova and uaddr seem not useful for vhost-pci, so maybe we will need to use gpa instead of uaddr 4) When QEMU1 receives the msg, it compares gpa1 with the memory info, for example, it may find base1 < gpa1 < base1+len1, and offset1=gpa1-base1, then do mmap(.., size1, fd1, offset1), and add the sub MemoryRegion (gpa1, size1) to the bar memory region, which will finally get the memory added to ept. Option 2: Similar to Option1, but in 1), QEMU1 can map and exposes the entire memory to the guest as non-accessible (instead of not mapping), and changes that pieces of memory (from IOTLB_MSG) to accessible in 4) Best, Wei
On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: > On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: >> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >>> On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: >>>> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >>>>> On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: >>>>>> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >>>>>>> On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: >>>>>>>> Instead of responding individually to these points, I hope this will >>>>>>>> explain my perspective. Let me know if you do want individual >>>>>>>> responses, I'm happy to talk more about the points above but I think >>>>>>>> the biggest difference is our perspective on this: >>>>>>>> >>>>>>>> Existing vhost-user slave code should be able to run on top of >>>>>>>> vhost-pci. For example, QEMU's >>>>>>>> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the guest >>>>>>>> with only minimal changes to the source file (i.e. today it explicitly >>>>>>>> opens a UNIX domain socket and that should be done by libvhost-user >>>>>>>> instead). It shouldn't be hard to add vhost-pci vfio support to >>>>>>>> contrib/libvhost-user/ alongside the existing UNIX domain socket code. >>>>>>>> >>>>>>>> This seems pretty easy to achieve with the vhost-pci PCI adapter that >>>>>>>> I've described but I'm not sure how to implement libvhost-user on top >>>>>>>> of vhost-pci vfio if the device doesn't expose the vhost-user >>>>>>>> protocol. >>>>>>>> >>>>>>>> I think this is a really important goal. Let's use a single >>>>>>>> vhost-user software stack instead of creating a separate one for guest >>>>>>>> code only. >>>>>>>> >>>>>>>> Do you agree that the vhost-user software stack should be shared >>>>>>>> between host userspace and guest code as much as possible? >>>>>>> >>>>>>> >>>>>>> The sharing you propose is not necessarily practical because the security goals >>>>>>> of the two are different. >>>>>>> >>>>>>> It seems that the best motivation presentation is still the original rfc >>>>>>> >>>>>>> http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication >>>>>>> >>>>>>> So comparing with vhost-user iotlb handling is different: >>>>>>> >>>>>>> With vhost-user guest trusts the vhost-user backend on the host. >>>>>>> >>>>>>> With vhost-pci we can strive to limit the trust to qemu only. >>>>>>> The switch running within a VM does not have to be trusted. >>>>>> Can you give a concrete example? >>>>>> >>>>>> I have an idea about what you're saying but it may be wrong: >>>>>> >>>>>> Today the iotlb mechanism in vhost-user does not actually enforce >>>>>> memory permissions. The vhost-user slave has full access to mmapped >>>>>> memory regions even when iotlb is enabled. Currently the iotlb just >>>>>> adds an indirection layer but no real security. (Is this correct?) >>>>> Not exactly. iotlb protects against malicious drivers within guest. >>>>> But yes, not against a vhost-user driver on the host. >>>>> >>>>>> Are you saying the vhost-pci device code in QEMU should enforce iotlb >>>>>> permissions so the vhost-user slave guest only has access to memory >>>>>> regions that are allowed by the iotlb? >>>>> Yes. >>>> Okay, thanks for confirming. >>>> >>>> This can be supported by the approach I've described. The vhost-pci >>>> QEMU code has control over the BAR memory so it can prevent the guest >>>> from accessing regions that are not allowed by the iotlb. >>>> >>>> Inside the guest the vhost-user slave still has the memory region >>>> descriptions and sends iotlb messages. This is completely compatible >>>> with the libvirt-user APIs and existing vhost-user slave code can run >>>> fine. The only unique thing is that guest accesses to memory regions >>>> not allowed by the iotlb do not work because QEMU has prevented it. >>> I don't think this can work since suddenly you need >>> to map full IOMMU address space into BAR. >> The BAR covers all guest RAM >> but QEMU can set up MemoryRegions that >> hide parts from the guest (e.g. reads produce 0xff). I'm not sure how >> expensive that is but implementing a strict IOMMU is hard to do >> without performance overhead. > I'm worried about leaking PAs. > fundamentally if you want proper protection you > need your device driver to use VA for addressing, > > On the one hand BAR only needs to be as large as guest PA then. > On the other hand it must cover all of guest PA, > not just what is accessible to the device. > > >>> Besides, this means implementing iotlb in both qemu and guest. >> It's free in the guest, the libvhost-user stack already has it. > That library is designed to work with a unix domain socket > though. We'll need extra kernel code to make a device > pretend it's a socket. > >>>> If better performance is needed then it might be possible to optimize >>>> this interface by handling most or even all of the iotlb stuff in QEMU >>>> vhost-pci code and not exposing it to the vhost-user slave in the >>>> guest. But it doesn't change the fact that the vhost-user protocol >>>> can be used and the same software stack works. >>> For one, the iotlb part would be out of scope then. >>> Instead you would have code to offset from BAR. >>> >>>> Do you have a concrete example of why sharing the same vhost-user >>>> software stack inside the guest can't work? >>> With enough dedication some code might be shared. OTOH reusing virtio >>> gains you a ready feature negotiation and discovery protocol. >>> >>> I'm not convinced which has more value, and the second proposal >>> has been implemented already. >> Thanks to you and Wei for the discussion. I've learnt a lot about >> vhost-user. If you have questions about what I've posted, please let >> me know and we can discuss further. >> >> The decision is not up to me so I'll just summarize what the vhost-pci >> PCI adapter approach achieves: >> 1. Just one device and driver >> 2. Support for all device types (net, scsi, blk, etc) >> 3. Reuse of software stack so vhost-user slaves can run in both host >> userspace and the guest >> 4. Simpler to debug because the vhost-user protocol used by QEMU is >> also used by the guest >> >> Stefan Thanks Stefan and Michael for the sharing and discussion. I think above 3 and 4 are debatable (e.g. whether it is simpler really depends). 1 and 2 are implementations, I think both approaches could implement the device that way. We originally thought about one device and driver to support all types (called it transformer sometimes :-) ), that would look interesting from research point of view, but from real usage point of view, I think it would be better to have them separated, because: - different device types have different driver logic, mixing them together would cause the driver to look messy. Imagine that a networking driver developer has to go over the block related code to debug, that also increases the difficulty. - For the kernel driver (looks like some people from Huawei are interested in that), I think users may want to see a standard network device and driver. If we mix all the types together, not sure what type of device will it be (misc?). Please let me know if you have a different viewpoint. Btw, from your perspective, what would be the practical usage of vhost-pci-blk? Best, Wei
On Fri, Dec 8, 2017 at 6:43 AM, Wei Wang <wei.w.wang@intel.com> wrote: > On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: >> >> On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: >>> >>> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> >>> wrote: >>>> >>>> On Thu, Dec 07, 2017 at 05:29:14PM +0000, Stefan Hajnoczi wrote: >>>>> >>>>> On Thu, Dec 7, 2017 at 4:47 PM, Michael S. Tsirkin <mst@redhat.com> >>>>> wrote: >>>>>> >>>>>> On Thu, Dec 07, 2017 at 04:29:45PM +0000, Stefan Hajnoczi wrote: >>>>>>> >>>>>>> On Thu, Dec 7, 2017 at 2:02 PM, Michael S. Tsirkin <mst@redhat.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> On Thu, Dec 07, 2017 at 01:08:04PM +0000, Stefan Hajnoczi wrote: >>>>>>>>> >>>>>>>>> Instead of responding individually to these points, I hope this >>>>>>>>> will >>>>>>>>> explain my perspective. Let me know if you do want individual >>>>>>>>> responses, I'm happy to talk more about the points above but I >>>>>>>>> think >>>>>>>>> the biggest difference is our perspective on this: >>>>>>>>> >>>>>>>>> Existing vhost-user slave code should be able to run on top of >>>>>>>>> vhost-pci. For example, QEMU's >>>>>>>>> contrib/vhost-user-scsi/vhost-user-scsi.c should work inside the >>>>>>>>> guest >>>>>>>>> with only minimal changes to the source file (i.e. today it >>>>>>>>> explicitly >>>>>>>>> opens a UNIX domain socket and that should be done by libvhost-user >>>>>>>>> instead). It shouldn't be hard to add vhost-pci vfio support to >>>>>>>>> contrib/libvhost-user/ alongside the existing UNIX domain socket >>>>>>>>> code. >>>>>>>>> >>>>>>>>> This seems pretty easy to achieve with the vhost-pci PCI adapter >>>>>>>>> that >>>>>>>>> I've described but I'm not sure how to implement libvhost-user on >>>>>>>>> top >>>>>>>>> of vhost-pci vfio if the device doesn't expose the vhost-user >>>>>>>>> protocol. >>>>>>>>> >>>>>>>>> I think this is a really important goal. Let's use a single >>>>>>>>> vhost-user software stack instead of creating a separate one for >>>>>>>>> guest >>>>>>>>> code only. >>>>>>>>> >>>>>>>>> Do you agree that the vhost-user software stack should be shared >>>>>>>>> between host userspace and guest code as much as possible? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> The sharing you propose is not necessarily practical because the >>>>>>>> security goals >>>>>>>> of the two are different. >>>>>>>> >>>>>>>> It seems that the best motivation presentation is still the original >>>>>>>> rfc >>>>>>>> >>>>>>>> >>>>>>>> http://virtualization.linux-foundation.narkive.com/A7FkzAgp/rfc-vhost-user-enhancements-for-vm2vm-communication >>>>>>>> >>>>>>>> So comparing with vhost-user iotlb handling is different: >>>>>>>> >>>>>>>> With vhost-user guest trusts the vhost-user backend on the host. >>>>>>>> >>>>>>>> With vhost-pci we can strive to limit the trust to qemu only. >>>>>>>> The switch running within a VM does not have to be trusted. >>>>>>> >>>>>>> Can you give a concrete example? >>>>>>> >>>>>>> I have an idea about what you're saying but it may be wrong: >>>>>>> >>>>>>> Today the iotlb mechanism in vhost-user does not actually enforce >>>>>>> memory permissions. The vhost-user slave has full access to mmapped >>>>>>> memory regions even when iotlb is enabled. Currently the iotlb just >>>>>>> adds an indirection layer but no real security. (Is this correct?) >>>>>> >>>>>> Not exactly. iotlb protects against malicious drivers within guest. >>>>>> But yes, not against a vhost-user driver on the host. >>>>>> >>>>>>> Are you saying the vhost-pci device code in QEMU should enforce iotlb >>>>>>> permissions so the vhost-user slave guest only has access to memory >>>>>>> regions that are allowed by the iotlb? >>>>>> >>>>>> Yes. >>>>> >>>>> Okay, thanks for confirming. >>>>> >>>>> This can be supported by the approach I've described. The vhost-pci >>>>> QEMU code has control over the BAR memory so it can prevent the guest >>>>> from accessing regions that are not allowed by the iotlb. >>>>> >>>>> Inside the guest the vhost-user slave still has the memory region >>>>> descriptions and sends iotlb messages. This is completely compatible >>>>> with the libvirt-user APIs and existing vhost-user slave code can run >>>>> fine. The only unique thing is that guest accesses to memory regions >>>>> not allowed by the iotlb do not work because QEMU has prevented it. >>>> >>>> I don't think this can work since suddenly you need >>>> to map full IOMMU address space into BAR. >>> >>> The BAR covers all guest RAM >>> but QEMU can set up MemoryRegions that >>> hide parts from the guest (e.g. reads produce 0xff). I'm not sure how >>> expensive that is but implementing a strict IOMMU is hard to do >>> without performance overhead. >> >> I'm worried about leaking PAs. >> fundamentally if you want proper protection you >> need your device driver to use VA for addressing, >> >> On the one hand BAR only needs to be as large as guest PA then. >> On the other hand it must cover all of guest PA, >> not just what is accessible to the device. >> >> >>>> Besides, this means implementing iotlb in both qemu and guest. >>> >>> It's free in the guest, the libvhost-user stack already has it. >> >> That library is designed to work with a unix domain socket >> though. We'll need extra kernel code to make a device >> pretend it's a socket. >> >>>>> If better performance is needed then it might be possible to optimize >>>>> this interface by handling most or even all of the iotlb stuff in QEMU >>>>> vhost-pci code and not exposing it to the vhost-user slave in the >>>>> guest. But it doesn't change the fact that the vhost-user protocol >>>>> can be used and the same software stack works. >>>> >>>> For one, the iotlb part would be out of scope then. >>>> Instead you would have code to offset from BAR. >>>> >>>>> Do you have a concrete example of why sharing the same vhost-user >>>>> software stack inside the guest can't work? >>>> >>>> With enough dedication some code might be shared. OTOH reusing virtio >>>> gains you a ready feature negotiation and discovery protocol. >>>> >>>> I'm not convinced which has more value, and the second proposal >>>> has been implemented already. >>> >>> Thanks to you and Wei for the discussion. I've learnt a lot about >>> vhost-user. If you have questions about what I've posted, please let >>> me know and we can discuss further. >>> >>> The decision is not up to me so I'll just summarize what the vhost-pci >>> PCI adapter approach achieves: >>> 1. Just one device and driver >>> 2. Support for all device types (net, scsi, blk, etc) >>> 3. Reuse of software stack so vhost-user slaves can run in both host >>> userspace and the guest >>> 4. Simpler to debug because the vhost-user protocol used by QEMU is >>> also used by the guest >>> >>> Stefan > > > Thanks Stefan and Michael for the sharing and discussion. I think above 3 > and 4 are debatable (e.g. whether it is simpler really depends). 1 and 2 are > implementations, I think both approaches could implement the device that > way. We originally thought about one device and driver to support all types > (called it transformer sometimes :-) ), that would look interesting from > research point of view, but from real usage point of view, I think it would > be better to have them separated, because: > - different device types have different driver logic, mixing them together > would cause the driver to look messy. Imagine that a networking driver > developer has to go over the block related code to debug, that also > increases the difficulty. I'm not sure I understand where things get messy because: 1. The vhost-pci device implementation in QEMU relays messages but has no device logic, so device-specific messages like VHOST_USER_NET_SET_MTU are trivial at this layer. 2. vhost-user slaves only handle certain vhost-user protocol messages. They handle device-specific messages for their device type only. This is like vhost drivers today where the ioctl() function returns an error if the ioctl is not supported by the device. It's not messy. Where are you worried about messy driver logic? > - For the kernel driver (looks like some people from Huawei are interested > in that), I think users may want to see a standard network device and > driver. If we mix all the types together, not sure what type of device will > it be (misc?). From my reply to Michael: "If we want to expose kernel vhost devices via vhost-pci then a libvhost-user program can translate the vhost-user protocol into kernel ioctls. For example: $ vhost-pci-proxy --vhost-pci-addr 00:04.0 --vhost-fd 3 3<>/dev/vhost-net The vhost-pci-proxy implements the vhost-user protocol callbacks and submits ioctls on the vhost kernel device fd. I haven't compared the kernel ioctl interface vs the vhost-user protocol to see if everything maps cleanly though." If an in-kernel implementation is really needed then a vhost-pci PCI driver would work like this: 1. The vhost-pci driver starts in a "unbound" state where it doesn't respond to vhost-user protocol messages. 2. The vhost-pci driver instance must be explicitly bound to an in-kernel vhost driver (net, scsi, vsock). 3. The vhost-pci driver code integrates with drivers/vhost/vhost.c so that in-kernel vhost drivers work over vhost-pci (i.e. vhost-user protocol messages become ioctl calls). The binding step is necessary because there is not enough information to start a vhost device instance automatically. For example, a vhost_scsi driver instance needs to be associated with a SCSI target and have a unique target ID. But I think the in-kernel implementation is unnecessary, at least in the beginning. > Please let me know if you have a different viewpoint. This patch series' approach of defining virtio devices for vhost-user slave devices is hard to do well because: 1. Virtio device backends cannot be mapped to virtio devices because virtio is asymmetric. See my previous replies for more info. 2. The vhost-user protocol does not cover the full virtio device model. For example, the vhost-user protocol doesn't have device types, the status register, etc. This patch series is trying to take something that is not a virtio device and expose it as a virtio device. I'm suggesting that we don't try to make it look like a virtio device because it's not possible to do that cleanly. > Btw, from your perspective, what would be the practical usage of > vhost-pci-blk? The SPDK community is working on vhost-user-blk: http://www.spdk.io/roadmap/ Stefan
On Friday, December 8, 2017 4:34 PM, Stefan Hajnoczi wrote: > On Fri, Dec 8, 2017 at 6:43 AM, Wei Wang <wei.w.wang@intel.com> wrote: > > On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: > >> > >> On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > >>> > >>> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> > > Thanks Stefan and Michael for the sharing and discussion. I think > > above 3 and 4 are debatable (e.g. whether it is simpler really > > depends). 1 and 2 are implementations, I think both approaches could > > implement the device that way. We originally thought about one device > > and driver to support all types (called it transformer sometimes :-) > > ), that would look interesting from research point of view, but from > > real usage point of view, I think it would be better to have them separated, > because: > > - different device types have different driver logic, mixing them > > together would cause the driver to look messy. Imagine that a > > networking driver developer has to go over the block related code to > > debug, that also increases the difficulty. > > I'm not sure I understand where things get messy because: > 1. The vhost-pci device implementation in QEMU relays messages but has no > device logic, so device-specific messages like VHOST_USER_NET_SET_MTU are > trivial at this layer. > 2. vhost-user slaves only handle certain vhost-user protocol messages. > They handle device-specific messages for their device type only. This is like > vhost drivers today where the ioctl() function returns an error if the ioctl is > not supported by the device. It's not messy. > > Where are you worried about messy driver logic? Probably I didn’t explain well, please let me summarize my thought a little bit, from the perspective of the control path and data path. Control path: the vhost-user messages - I would prefer just have the interaction between QEMUs, instead of relaying to the GuestSlave, because 1) I think the claimed advantage (easier to debug and develop) doesn’t seem very convincing 2) some messages can be directly answered by QemuSlave , and some messages are not useful to give to the GuestSlave (inside the VM), e.g. fds, VhostUserMemoryRegion from SET_MEM_TABLE msg (the device first maps the master memory and gives the offset (in terms of the bar, i.e., where does it sit in the bar) of the mapped gpa to the guest. if we give the raw VhostUserMemoryRegion to the guest, that wouldn’t be usable). Data path: that's the discussion we had about one driver or separate driver for different device types, and this is not related to the control path. I meant if we have one driver for all the types, that driver would look messy, because each type has its own data sending/receiving logic. For example, net type deals with a pair of tx and rx, and transmission is skb based (e.g. xmit_skb), while block type deals with a request queue. If we have one driver, then the driver will include all the things together. The last part is whether to make it a virtio device or a regular pci device I don’t have a strong preference. I think virtio device works fine (e.g. use some bar area to create ioevenfds to solve the "no virtqueue no fds" issue if you and Michael think that's acceptable), and we can reuse some other things like feature negotiation from virtio. But if Michael and you have a decision to make it a regular PCI device, I think that would also work though. Best, Wei
On Sat, Dec 09, 2017 at 04:23:17PM +0000, Wang, Wei W wrote: > On Friday, December 8, 2017 4:34 PM, Stefan Hajnoczi wrote: > > On Fri, Dec 8, 2017 at 6:43 AM, Wei Wang <wei.w.wang@intel.com> wrote: > > > On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: > > >> > > >> On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > > >>> > > >>> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin <mst@redhat.com> > > > Thanks Stefan and Michael for the sharing and discussion. I think > > > above 3 and 4 are debatable (e.g. whether it is simpler really > > > depends). 1 and 2 are implementations, I think both approaches could > > > implement the device that way. We originally thought about one device > > > and driver to support all types (called it transformer sometimes :-) > > > ), that would look interesting from research point of view, but from > > > real usage point of view, I think it would be better to have them separated, > > because: > > > - different device types have different driver logic, mixing them > > > together would cause the driver to look messy. Imagine that a > > > networking driver developer has to go over the block related code to > > > debug, that also increases the difficulty. > > > > I'm not sure I understand where things get messy because: > > 1. The vhost-pci device implementation in QEMU relays messages but has no > > device logic, so device-specific messages like VHOST_USER_NET_SET_MTU are > > trivial at this layer. > > 2. vhost-user slaves only handle certain vhost-user protocol messages. > > They handle device-specific messages for their device type only. This is like > > vhost drivers today where the ioctl() function returns an error if the ioctl is > > not supported by the device. It's not messy. > > > > Where are you worried about messy driver logic? > > Probably I didn’t explain well, please let me summarize my thought a little bit, from the perspective of the control path and data path. > > Control path: the vhost-user messages - I would prefer just have the interaction between QEMUs, instead of relaying to the GuestSlave, because > 1) I think the claimed advantage (easier to debug and develop) doesn’t seem very convincing You are defining a mapping from the vhost-user protocol to a custom virtio device interface. Every time the vhost-user protocol (feature bits, messages, etc) is extended it will be necessary to map this new extension to the virtio device interface. That's non-trivial. Mistakes are possible when designing the mapping. Using the vhost-user protocol as the device interface minimizes the effort and risk of mistakes because most messages are relayed 1:1. > 2) some messages can be directly answered by QemuSlave , and some messages are not useful to give to the GuestSlave (inside the VM), e.g. fds, VhostUserMemoryRegion from SET_MEM_TABLE msg (the device first maps the master memory and gives the offset (in terms of the bar, i.e., where does it sit in the bar) of the mapped gpa to the guest. if we give the raw VhostUserMemoryRegion to the guest, that wouldn’t be usable). I agree that QEMU has to handle some of messages, but it should still relay all (possibly modified) messages to the guest. The point of using the vhost-user protocol is not just to use a familiar binary encoding, it's to match the semantics of vhost-user 100%. That way the vhost-user software stack can work either in host userspace or with vhost-pci without significant changes. Using the vhost-user protocol as the device interface doesn't seem any harder than defining a completely new virtio device interface. It has the advantages that I've pointed out: 1. Simple 1:1 mapping for most that is easy to maintain as the vhost-user protocol grows. 2. Compatible with vhost-user so slaves can run in host userspace or the guest. I don't see why it makes sense to define new device interfaces for each device type and create a software stack that is incompatible with vhost-user. > > > Data path: that's the discussion we had about one driver or separate driver for different device types, and this is not related to the control path. > I meant if we have one driver for all the types, that driver would look messy, because each type has its own data sending/receiving logic. For example, net type deals with a pair of tx and rx, and transmission is skb based (e.g. xmit_skb), while block type deals with a request queue. If we have one driver, then the driver will include all the things together. I don't understand this. Why would we have to put all devices (net, scsi, etc) into just one driver? The device drivers sit on top of the vhost-pci driver. For example, imagine a libvhost-user application that handles the net device. The vhost-pci vfio driver would be part of libvhost-user and the application would only emulate the net device (RX and TX queues). Stefan
On Monday, December 11, 2017 7:12 PM, Stefan Hajnoczi wrote: > On Sat, Dec 09, 2017 at 04:23:17PM +0000, Wang, Wei W wrote: > > On Friday, December 8, 2017 4:34 PM, Stefan Hajnoczi wrote: > > > On Fri, Dec 8, 2017 at 6:43 AM, Wei Wang <wei.w.wang@intel.com> > wrote: > > > > On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: > > > >> > > > >> On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > > > >>> > > > >>> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin > > > >>> <mst@redhat.com> > > > > Thanks Stefan and Michael for the sharing and discussion. I > > > > think above 3 and 4 are debatable (e.g. whether it is simpler > > > > really depends). 1 and 2 are implementations, I think both > > > > approaches could implement the device that way. We originally > > > > thought about one device and driver to support all types (called > > > > it transformer sometimes :-) ), that would look interesting from > > > > research point of view, but from real usage point of view, I > > > > think it would be better to have them separated, > > > because: > > > > - different device types have different driver logic, mixing > > > > them together would cause the driver to look messy. Imagine that > > > > a networking driver developer has to go over the block related > > > > code to debug, that also increases the difficulty. > > > > > > I'm not sure I understand where things get messy because: > > > 1. The vhost-pci device implementation in QEMU relays messages but > > > has no device logic, so device-specific messages like > > > VHOST_USER_NET_SET_MTU are trivial at this layer. > > > 2. vhost-user slaves only handle certain vhost-user protocol messages. > > > They handle device-specific messages for their device type only. > > > This is like vhost drivers today where the ioctl() function > > > returns an error if the ioctl is not supported by the device. It's not messy. > > > > > > Where are you worried about messy driver logic? > > > > Probably I didn’t explain well, please let me summarize my thought a > > little > bit, from the perspective of the control path and data path. > > > > Control path: the vhost-user messages - I would prefer just have the > > interaction between QEMUs, instead of relaying to the GuestSlave, > > because > > 1) I think the claimed advantage (easier to debug and develop) > > doesn’t seem very convincing > > You are defining a mapping from the vhost-user protocol to a custom > virtio device interface. Every time the vhost-user protocol (feature > bits, messages, > etc) is extended it will be necessary to map this new extension to the > virtio device interface. > > That's non-trivial. Mistakes are possible when designing the mapping. > Using the vhost-user protocol as the device interface minimizes the > effort and risk of mistakes because most messages are relayed 1:1. > > > 2) some messages can be directly answered by QemuSlave , and some > messages are not useful to give to the GuestSlave (inside the VM), > e.g. fds, VhostUserMemoryRegion from SET_MEM_TABLE msg (the device > first maps the master memory and gives the offset (in terms of the > bar, i.e., where does it sit in the bar) of the mapped gpa to the > guest. if we give the raw VhostUserMemoryRegion to the guest, that wouldn’t be usable). > > I agree that QEMU has to handle some of messages, but it should still > relay all (possibly modified) messages to the guest. > > The point of using the vhost-user protocol is not just to use a > familiar binary encoding, it's to match the semantics of vhost-user > 100%. That way the vhost-user software stack can work either in host > userspace or with vhost-pci without significant changes. > > Using the vhost-user protocol as the device interface doesn't seem any > harder than defining a completely new virtio device interface. It has > the advantages that I've pointed out: > > 1. Simple 1:1 mapping for most that is easy to maintain as the > vhost-user protocol grows. > > 2. Compatible with vhost-user so slaves can run in host userspace > or the guest. > > I don't see why it makes sense to define new device interfaces for > each device type and create a software stack that is incompatible with vhost-user. I think this 1:1 mapping wouldn't be easy: 1) We will have 2 Qemu side slaves to achieve this bidirectional relaying, that is, the working model will be - master to slave: Master->QemuSlave1->GuestSlave; and - slave to master: GuestSlave->QemuSlave2->Master QemuSlave1 and QemuSlave2 can't be the same piece of code, because QemuSlave1 needs to do some setup with some messages, and QemuSlave2 is more likely to be a true "relayer" (receive and directly pass on) 2) poor re-usability of the QemuSlave and GuestSlave We couldn’t reuse much of the QemuSlave handling code for GuestSlave. For example, for the VHOST_USER_SET_MEM_TABLE msg, all the QemuSlave handling code (please see the vp_slave_set_mem_table function), won't be used by GuestSlave. On the other hand, GuestSlave needs an implementation to reply back to the QEMU device, and this implementation isn't needed by QemuSlave. If we want to run the same piece of the slave code in both QEMU and guest, then we may need "if (QemuSlave) else" in each msg handling entry to choose the code path for QemuSlave and GuestSlave separately. So, ideally we wish to run (reuse) one slave implementation in both QEMU and guest. In practice, we will still need to handle them each case by case, which is no different than maintaining two separate slaves for QEMU and guest, and I'm afraid this would be much more complex. Best, Wei
On Mon, Dec 11, 2017 at 01:53:40PM +0000, Wang, Wei W wrote: > On Monday, December 11, 2017 7:12 PM, Stefan Hajnoczi wrote: > > On Sat, Dec 09, 2017 at 04:23:17PM +0000, Wang, Wei W wrote: > > > On Friday, December 8, 2017 4:34 PM, Stefan Hajnoczi wrote: > > > > On Fri, Dec 8, 2017 at 6:43 AM, Wei Wang <wei.w.wang@intel.com> > > wrote: > > > > > On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: > > > > >> > > > > >> On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > > > > >>> > > > > >>> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin > > > > >>> <mst@redhat.com> > > > > > Thanks Stefan and Michael for the sharing and discussion. I > > > > > think above 3 and 4 are debatable (e.g. whether it is simpler > > > > > really depends). 1 and 2 are implementations, I think both > > > > > approaches could implement the device that way. We originally > > > > > thought about one device and driver to support all types (called > > > > > it transformer sometimes :-) ), that would look interesting from > > > > > research point of view, but from real usage point of view, I > > > > > think it would be better to have them separated, > > > > because: > > > > > - different device types have different driver logic, mixing > > > > > them together would cause the driver to look messy. Imagine that > > > > > a networking driver developer has to go over the block related > > > > > code to debug, that also increases the difficulty. > > > > > > > > I'm not sure I understand where things get messy because: > > > > 1. The vhost-pci device implementation in QEMU relays messages but > > > > has no device logic, so device-specific messages like > > > > VHOST_USER_NET_SET_MTU are trivial at this layer. > > > > 2. vhost-user slaves only handle certain vhost-user protocol messages. > > > > They handle device-specific messages for their device type only. > > > > This is like vhost drivers today where the ioctl() function > > > > returns an error if the ioctl is not supported by the device. It's not messy. > > > > > > > > Where are you worried about messy driver logic? > > > > > > Probably I didn’t explain well, please let me summarize my thought a > > > little > > bit, from the perspective of the control path and data path. > > > > > > Control path: the vhost-user messages - I would prefer just have the > > > interaction between QEMUs, instead of relaying to the GuestSlave, > > > because > > > 1) I think the claimed advantage (easier to debug and develop) > > > doesn’t seem very convincing > > > > You are defining a mapping from the vhost-user protocol to a custom > > virtio device interface. Every time the vhost-user protocol (feature > > bits, messages, > > etc) is extended it will be necessary to map this new extension to the > > virtio device interface. > > > > That's non-trivial. Mistakes are possible when designing the mapping. > > Using the vhost-user protocol as the device interface minimizes the > > effort and risk of mistakes because most messages are relayed 1:1. > > > > > 2) some messages can be directly answered by QemuSlave , and some > > messages are not useful to give to the GuestSlave (inside the VM), > > e.g. fds, VhostUserMemoryRegion from SET_MEM_TABLE msg (the device > > first maps the master memory and gives the offset (in terms of the > > bar, i.e., where does it sit in the bar) of the mapped gpa to the > > guest. if we give the raw VhostUserMemoryRegion to the guest, that wouldn’t be usable). > > > > I agree that QEMU has to handle some of messages, but it should still > > relay all (possibly modified) messages to the guest. > > > > The point of using the vhost-user protocol is not just to use a > > familiar binary encoding, it's to match the semantics of vhost-user > > 100%. That way the vhost-user software stack can work either in host > > userspace or with vhost-pci without significant changes. > > > > Using the vhost-user protocol as the device interface doesn't seem any > > harder than defining a completely new virtio device interface. It has > > the advantages that I've pointed out: > > > > 1. Simple 1:1 mapping for most that is easy to maintain as the > > vhost-user protocol grows. > > > > 2. Compatible with vhost-user so slaves can run in host userspace > > or the guest. > > > > I don't see why it makes sense to define new device interfaces for > > each device type and create a software stack that is incompatible with vhost-user. > > > I think this 1:1 mapping wouldn't be easy: > > 1) We will have 2 Qemu side slaves to achieve this bidirectional relaying, that is, the working model will be > - master to slave: Master->QemuSlave1->GuestSlave; and > - slave to master: GuestSlave->QemuSlave2->Master > QemuSlave1 and QemuSlave2 can't be the same piece of code, because QemuSlave1 needs to do some setup with some messages, and QemuSlave2 is more likely to be a true "relayer" (receive and directly pass on) I mostly agree with this. Some messages cannot be passed through. QEMU needs to process some messages so that makes it both a slave (on the host) and a master (to the guest). > 2) poor re-usability of the QemuSlave and GuestSlave > We couldn’t reuse much of the QemuSlave handling code for GuestSlave. > For example, for the VHOST_USER_SET_MEM_TABLE msg, all the QemuSlave handling code (please see the vp_slave_set_mem_table function), won't be used by GuestSlave. On the other hand, GuestSlave needs an implementation to reply back to the QEMU device, and this implementation isn't needed by QemuSlave. > If we want to run the same piece of the slave code in both QEMU and guest, then we may need "if (QemuSlave) else" in each msg handling entry to choose the code path for QemuSlave and GuestSlave separately. > So, ideally we wish to run (reuse) one slave implementation in both QEMU and guest. In practice, we will still need to handle them each case by case, which is no different than maintaining two separate slaves for QEMU and guest, and I'm afraid this would be much more complex. Are you saying QEMU's vhost-pci code cannot be reused by guest slaves? If so, I agree and it was not my intention to run the same slave code in QEMU and the guest. When I referred to reusing the vhost-user software stack I meant something else: 1. contrib/libvhost-user/ is a vhost-user slave library. QEMU itself does not use it but external programs may use it to avoid reimplementing vhost-user and vrings. Currently this code handles the vhost-user protocol over UNIX domain sockets, but it's possible to add vfio vhost-pci support. Programs using libvhost-user would be able to take advantage of vhost-pci easily (no big changes required). 2. DPDK and other codebases that implement custom vhost-user slaves are also easy to update for vhost-pci since the same protocol is used. Only the lowest layer of vhost-user slave code needs to be touched. Stefan
On 12/12/2017 06:14 PM, Stefan Hajnoczi wrote: > On Mon, Dec 11, 2017 at 01:53:40PM +0000, Wang, Wei W wrote: >> On Monday, December 11, 2017 7:12 PM, Stefan Hajnoczi wrote: >>> On Sat, Dec 09, 2017 at 04:23:17PM +0000, Wang, Wei W wrote: >>>> On Friday, December 8, 2017 4:34 PM, Stefan Hajnoczi wrote: >>>>> On Fri, Dec 8, 2017 at 6:43 AM, Wei Wang <wei.w.wang@intel.com> >>> wrote: >>>>>> On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: >>>>>>> On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: >>>>>>>> On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin >>>>>>>> <mst@redhat.com> >>>>>> Thanks Stefan and Michael for the sharing and discussion. I >>>>>> think above 3 and 4 are debatable (e.g. whether it is simpler >>>>>> really depends). 1 and 2 are implementations, I think both >>>>>> approaches could implement the device that way. We originally >>>>>> thought about one device and driver to support all types (called >>>>>> it transformer sometimes :-) ), that would look interesting from >>>>>> research point of view, but from real usage point of view, I >>>>>> think it would be better to have them separated, >>>>> because: >>>>>> - different device types have different driver logic, mixing >>>>>> them together would cause the driver to look messy. Imagine that >>>>>> a networking driver developer has to go over the block related >>>>>> code to debug, that also increases the difficulty. >>>>> I'm not sure I understand where things get messy because: >>>>> 1. The vhost-pci device implementation in QEMU relays messages but >>>>> has no device logic, so device-specific messages like >>>>> VHOST_USER_NET_SET_MTU are trivial at this layer. >>>>> 2. vhost-user slaves only handle certain vhost-user protocol messages. >>>>> They handle device-specific messages for their device type only. >>>>> This is like vhost drivers today where the ioctl() function >>>>> returns an error if the ioctl is not supported by the device. It's not messy. >>>>> >>>>> Where are you worried about messy driver logic? >>>> Probably I didn’t explain well, please let me summarize my thought a >>>> little >>> bit, from the perspective of the control path and data path. >>>> Control path: the vhost-user messages - I would prefer just have the >>>> interaction between QEMUs, instead of relaying to the GuestSlave, >>>> because >>>> 1) I think the claimed advantage (easier to debug and develop) >>>> doesn’t seem very convincing >>> You are defining a mapping from the vhost-user protocol to a custom >>> virtio device interface. Every time the vhost-user protocol (feature >>> bits, messages, >>> etc) is extended it will be necessary to map this new extension to the >>> virtio device interface. >>> >>> That's non-trivial. Mistakes are possible when designing the mapping. >>> Using the vhost-user protocol as the device interface minimizes the >>> effort and risk of mistakes because most messages are relayed 1:1. >>> >>>> 2) some messages can be directly answered by QemuSlave , and some >>> messages are not useful to give to the GuestSlave (inside the VM), >>> e.g. fds, VhostUserMemoryRegion from SET_MEM_TABLE msg (the device >>> first maps the master memory and gives the offset (in terms of the >>> bar, i.e., where does it sit in the bar) of the mapped gpa to the >>> guest. if we give the raw VhostUserMemoryRegion to the guest, that wouldn’t be usable). >>> >>> I agree that QEMU has to handle some of messages, but it should still >>> relay all (possibly modified) messages to the guest. >>> >>> The point of using the vhost-user protocol is not just to use a >>> familiar binary encoding, it's to match the semantics of vhost-user >>> 100%. That way the vhost-user software stack can work either in host >>> userspace or with vhost-pci without significant changes. >>> >>> Using the vhost-user protocol as the device interface doesn't seem any >>> harder than defining a completely new virtio device interface. It has >>> the advantages that I've pointed out: >>> >>> 1. Simple 1:1 mapping for most that is easy to maintain as the >>> vhost-user protocol grows. >>> >>> 2. Compatible with vhost-user so slaves can run in host userspace >>> or the guest. >>> >>> I don't see why it makes sense to define new device interfaces for >>> each device type and create a software stack that is incompatible with vhost-user. >> >> I think this 1:1 mapping wouldn't be easy: >> >> 1) We will have 2 Qemu side slaves to achieve this bidirectional relaying, that is, the working model will be >> - master to slave: Master->QemuSlave1->GuestSlave; and >> - slave to master: GuestSlave->QemuSlave2->Master >> QemuSlave1 and QemuSlave2 can't be the same piece of code, because QemuSlave1 needs to do some setup with some messages, and QemuSlave2 is more likely to be a true "relayer" (receive and directly pass on) > I mostly agree with this. Some messages cannot be passed through. QEMU > needs to process some messages so that makes it both a slave (on the > host) and a master (to the guest). > >> 2) poor re-usability of the QemuSlave and GuestSlave >> We couldn’t reuse much of the QemuSlave handling code for GuestSlave. >> For example, for the VHOST_USER_SET_MEM_TABLE msg, all the QemuSlave handling code (please see the vp_slave_set_mem_table function), won't be used by GuestSlave. On the other hand, GuestSlave needs an implementation to reply back to the QEMU device, and this implementation isn't needed by QemuSlave. >> If we want to run the same piece of the slave code in both QEMU and guest, then we may need "if (QemuSlave) else" in each msg handling entry to choose the code path for QemuSlave and GuestSlave separately. >> So, ideally we wish to run (reuse) one slave implementation in both QEMU and guest. In practice, we will still need to handle them each case by case, which is no different than maintaining two separate slaves for QEMU and guest, and I'm afraid this would be much more complex. > Are you saying QEMU's vhost-pci code cannot be reused by guest slaves? > If so, I agree and it was not my intention to run the same slave code in > QEMU and the guest. Yes, it is too difficult to reuse in practice. > > When I referred to reusing the vhost-user software stack I meant > something else: > > 1. contrib/libvhost-user/ is a vhost-user slave library. QEMU itself > does not use it but external programs may use it to avoid reimplementing > vhost-user and vrings. Currently this code handles the vhost-user > protocol over UNIX domain sockets, but it's possible to add vfio > vhost-pci support. Programs using libvhost-user would be able to take > advantage of vhost-pci easily (no big changes required). > > 2. DPDK and other codebases that implement custom vhost-user slaves are > also easy to update for vhost-pci since the same protocol is used. Only > the lowest layer of vhost-user slave code needs to be touched. I'm not sure if libvhost-user would be limited to be used by QEMU only in practice. For example, DPDK currently implements its own vhost-user slave, and changing to use libvhost-user may require dpdk to be bound with QEMU, that is, applications like OVS-DPDK will have a dependency on QEMU. Probably people wouldn't want it this way. On the other side, vhost-pci is more coupled with the QEMU implementation, because some of the msg handling will need to do some device setup (e.g. mmap memory and add sub MemoryRegion to the bar). This device emulation related code is specific to QEMU, so I think vhost-pci slave may not be reused by applications other than QEMU. Would it be acceptable to use the vhost-pci slave from this patch series as the initial solution? It is already implemented, and we can investigate the possibility of integrating it into the libvhost-user as the next step. Best, Wei
On Wed, Dec 13, 2017 at 04:11:45PM +0800, Wei Wang wrote: > On 12/12/2017 06:14 PM, Stefan Hajnoczi wrote: > > On Mon, Dec 11, 2017 at 01:53:40PM +0000, Wang, Wei W wrote: > > > On Monday, December 11, 2017 7:12 PM, Stefan Hajnoczi wrote: > > > > On Sat, Dec 09, 2017 at 04:23:17PM +0000, Wang, Wei W wrote: > > > > > On Friday, December 8, 2017 4:34 PM, Stefan Hajnoczi wrote: > > > > > > On Fri, Dec 8, 2017 at 6:43 AM, Wei Wang <wei.w.wang@intel.com> > > > > wrote: > > > > > > > On 12/08/2017 07:54 AM, Michael S. Tsirkin wrote: > > > > > > > > On Thu, Dec 07, 2017 at 06:28:19PM +0000, Stefan Hajnoczi wrote: > > > > > > > > > On Thu, Dec 7, 2017 at 5:38 PM, Michael S. Tsirkin > > > > > > > > > <mst@redhat.com> > > > > > > > Thanks Stefan and Michael for the sharing and discussion. I > > > > > > > think above 3 and 4 are debatable (e.g. whether it is simpler > > > > > > > really depends). 1 and 2 are implementations, I think both > > > > > > > approaches could implement the device that way. We originally > > > > > > > thought about one device and driver to support all types (called > > > > > > > it transformer sometimes :-) ), that would look interesting from > > > > > > > research point of view, but from real usage point of view, I > > > > > > > think it would be better to have them separated, > > > > > > because: > > > > > > > - different device types have different driver logic, mixing > > > > > > > them together would cause the driver to look messy. Imagine that > > > > > > > a networking driver developer has to go over the block related > > > > > > > code to debug, that also increases the difficulty. > > > > > > I'm not sure I understand where things get messy because: > > > > > > 1. The vhost-pci device implementation in QEMU relays messages but > > > > > > has no device logic, so device-specific messages like > > > > > > VHOST_USER_NET_SET_MTU are trivial at this layer. > > > > > > 2. vhost-user slaves only handle certain vhost-user protocol messages. > > > > > > They handle device-specific messages for their device type only. > > > > > > This is like vhost drivers today where the ioctl() function > > > > > > returns an error if the ioctl is not supported by the device. It's not messy. > > > > > > > > > > > > Where are you worried about messy driver logic? > > > > > Probably I didn’t explain well, please let me summarize my thought a > > > > > little > > > > bit, from the perspective of the control path and data path. > > > > > Control path: the vhost-user messages - I would prefer just have the > > > > > interaction between QEMUs, instead of relaying to the GuestSlave, > > > > > because > > > > > 1) I think the claimed advantage (easier to debug and develop) > > > > > doesn’t seem very convincing > > > > You are defining a mapping from the vhost-user protocol to a custom > > > > virtio device interface. Every time the vhost-user protocol (feature > > > > bits, messages, > > > > etc) is extended it will be necessary to map this new extension to the > > > > virtio device interface. > > > > > > > > That's non-trivial. Mistakes are possible when designing the mapping. > > > > Using the vhost-user protocol as the device interface minimizes the > > > > effort and risk of mistakes because most messages are relayed 1:1. > > > > > > > > > 2) some messages can be directly answered by QemuSlave , and some > > > > messages are not useful to give to the GuestSlave (inside the VM), > > > > e.g. fds, VhostUserMemoryRegion from SET_MEM_TABLE msg (the device > > > > first maps the master memory and gives the offset (in terms of the > > > > bar, i.e., where does it sit in the bar) of the mapped gpa to the > > > > guest. if we give the raw VhostUserMemoryRegion to the guest, that wouldn’t be usable). > > > > > > > > I agree that QEMU has to handle some of messages, but it should still > > > > relay all (possibly modified) messages to the guest. > > > > > > > > The point of using the vhost-user protocol is not just to use a > > > > familiar binary encoding, it's to match the semantics of vhost-user > > > > 100%. That way the vhost-user software stack can work either in host > > > > userspace or with vhost-pci without significant changes. > > > > > > > > Using the vhost-user protocol as the device interface doesn't seem any > > > > harder than defining a completely new virtio device interface. It has > > > > the advantages that I've pointed out: > > > > > > > > 1. Simple 1:1 mapping for most that is easy to maintain as the > > > > vhost-user protocol grows. > > > > > > > > 2. Compatible with vhost-user so slaves can run in host userspace > > > > or the guest. > > > > > > > > I don't see why it makes sense to define new device interfaces for > > > > each device type and create a software stack that is incompatible with vhost-user. > > > > > > I think this 1:1 mapping wouldn't be easy: > > > > > > 1) We will have 2 Qemu side slaves to achieve this bidirectional relaying, that is, the working model will be > > > - master to slave: Master->QemuSlave1->GuestSlave; and > > > - slave to master: GuestSlave->QemuSlave2->Master > > > QemuSlave1 and QemuSlave2 can't be the same piece of code, because QemuSlave1 needs to do some setup with some messages, and QemuSlave2 is more likely to be a true "relayer" (receive and directly pass on) > > I mostly agree with this. Some messages cannot be passed through. QEMU > > needs to process some messages so that makes it both a slave (on the > > host) and a master (to the guest). > > > > > 2) poor re-usability of the QemuSlave and GuestSlave > > > We couldn’t reuse much of the QemuSlave handling code for GuestSlave. > > > For example, for the VHOST_USER_SET_MEM_TABLE msg, all the QemuSlave handling code (please see the vp_slave_set_mem_table function), won't be used by GuestSlave. On the other hand, GuestSlave needs an implementation to reply back to the QEMU device, and this implementation isn't needed by QemuSlave. > > > If we want to run the same piece of the slave code in both QEMU and guest, then we may need "if (QemuSlave) else" in each msg handling entry to choose the code path for QemuSlave and GuestSlave separately. > > > So, ideally we wish to run (reuse) one slave implementation in both QEMU and guest. In practice, we will still need to handle them each case by case, which is no different than maintaining two separate slaves for QEMU and guest, and I'm afraid this would be much more complex. > > Are you saying QEMU's vhost-pci code cannot be reused by guest slaves? > > If so, I agree and it was not my intention to run the same slave code in > > QEMU and the guest. > > Yes, it is too difficult to reuse in practice. > > > > > When I referred to reusing the vhost-user software stack I meant > > something else: > > > > 1. contrib/libvhost-user/ is a vhost-user slave library. QEMU itself > > does not use it but external programs may use it to avoid reimplementing > > vhost-user and vrings. Currently this code handles the vhost-user > > protocol over UNIX domain sockets, but it's possible to add vfio > > vhost-pci support. Programs using libvhost-user would be able to take > > advantage of vhost-pci easily (no big changes required). > > > > 2. DPDK and other codebases that implement custom vhost-user slaves are > > also easy to update for vhost-pci since the same protocol is used. Only > > the lowest layer of vhost-user slave code needs to be touched. > > I'm not sure if libvhost-user would be limited to be used by QEMU only in > practice. For example, DPDK currently implements its own vhost-user slave, > and changing to use libvhost-user may require dpdk to be bound with QEMU, > that is, applications like OVS-DPDK will have a dependency on QEMU. Probably > people wouldn't want it this way. I'm not saying that DPDK should use libvhost-user. I'm saying that it's easy to add vfio vhost-pci support (for the PCI adapter I described) to DPDK. This patch series would require writing a completely new slave for vhost-pci because the device interface is so different from vhost-user. > On the other side, vhost-pci is more coupled with the QEMU implementation, > because some of the msg handling will need to do some device setup (e.g. > mmap memory and add sub MemoryRegion to the bar). This device emulation > related code is specific to QEMU, so I think vhost-pci slave may not be > reused by applications other than QEMU. As mentioned previously, I did not propose reusing QEMU vhost-pci code in vhost-user slaves. It wouldn't make sense to do that. > Would it be acceptable to use the vhost-pci slave from this patch series as > the initial solution? It is already implemented, and we can investigate the > possibility of integrating it into the libvhost-user as the next step. I think the current approach is fine for a prototype but is not suitable for wider use by the community because it: 1. Does not scale to multiple device types (net, scsi, blk, etc) 2. Does not scale as the vhost-user protocol changes 3. It is hard to make slaves run in both host userspace and the guest It would be good to solve these problems so that vhost-pci can become successful. It's very hard to fix these things after the code is merged because guests will depend on the device interface. Here are the points in detail (in order of importance): 1. Does not scale to multiple device types (net, scsi, blk, etc) vhost-user is being applied to new device types beyond virtio-net. There will be demand for supporting other device types besides virtio-net with vhost-pci. This patch series requires defining a new virtio device type for each vhost-user device type. It is a lot of work to design a new virtio device. Additionally, the new virtio device type should become part of the VIRTIO standard, which can also take some time and requires writing a standards document. 2. Does not scale as the vhost-user protocol changes When the vhost-user protocol changes it will be necessary to update the vhost-pci device interface to reflect those changes. Each protocol change requires thinking how the virtio devices need to look in order to support the new behavior. Changes to the vhost-user protocol will result in changes to the VIRTIO specification for the vhost-pci virtio devices. 3. It is hard to make slaves run in both host userspace and the guest If a vhost-user slave wishes to support running in host userspace and the guest then not much code can be shared between these two modes since the interfaces are so different. How would you solve these issues? Stefan
On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: > I'm not saying that DPDK should use libvhost-user. I'm saying that it's > easy to add vfio vhost-pci support (for the PCI adapter I described) to > DPDK. This patch series would require writing a completely new slave > for vhost-pci because the device interface is so different from > vhost-user. The main question is how appropriate is the vhost user protocol for passing to guests. And I am not sure at this point. Someone should go over vhost user messages and see whether they are safe to pass to guest. If most are then we can try the transparent approach. If most aren't then we can't and might as well use the proposed protocol which at least has code behind it. I took a quick look and I doubt we can do something that is both compatible with the existing vhost-user and will make it possible to extend the protocol without qemu changes. Let's assume I pass a new message over the vhost-user channel. How do we know it's safe to pass it to the guest? That's why we gate any protocol change on a feature bit and must parse all messages. Cc Maxime as well.
On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: >> I'm not saying that DPDK should use libvhost-user. I'm saying that it's >> easy to add vfio vhost-pci support (for the PCI adapter I described) to >> DPDK. This patch series would require writing a completely new slave >> for vhost-pci because the device interface is so different from >> vhost-user. > > The main question is how appropriate is the vhost user protocol > for passing to guests. And I am not sure at this point. > > Someone should go over vhost user messages and see whether they are safe > to pass to guest. If most are then we can try the transparent approach. > If most aren't then we can't and might as well use the proposed protocol > which at least has code behind it. I have done that: Master message types -------------------- * VHOST_USER_GET_FEATURES Safe to pass through. * VHOST_USER_SET_FEATURES Safe to pass through. * VHOST_USER_GET_PROTOCOL_FEATURES QEMU must mask unsupported VHOST_USER_PROTOCOL_F_* bits that the guest set. * VHOST_USER_SET_PROTOCOL_FEATURES Safe to pass through. * VHOST_USER_SET_OWNER Safe to pass through. * VHOST_USER_RESET_OWNER Unused message, ignore and do not pass through. * VHOST_USER_SET_MEM_TABLE Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. * VHOST_USER_SET_LOG_BASE Set up log shared memory region before sending VHOST_USER_SET_LOG_BASE to the guest. * VHOST_USER_SET_LOG_FD Set up log doorbell before sending VHOST_USER_SET_LOG_FD to the guest. * VHOST_USER_SET_VRING_NUM Safe to pass through. * VHOST_USER_SET_VRING_ADDR Passthrough if QEMU isn't doing IOMMU. Otherwise remap the addresses before passing them to the guest. * VHOST_USER_SET_VRING_BASE Safe to pass through. * VHOST_USER_GET_VRING_BASE Safe to pass through. * VHOST_USER_SET_VRING_KICK Set up vring kick doorbell (unless bit 8 is set) before sending VHOST_USER_SET_VRING_KICK to the guest. * VHOST_USER_SET_VRING_CALL Set up the vring call doorbell (unless bit 8 is set) before sending VHOST_USER_SET_VRING_CALL to the guest. * VHOST_USER_SET_VRING_ERR Set up the vring err doorbell (unless bit 8 is set) before sending VHOST_USER_SET_VRING_ERR to the guest. * VHOST_USER_GET_QUEUE_NUM Safe to pass through. * VHOST_USER_SET_VRING_ENABLE Safe to pass through. * VHOST_USER_SEND_RARP Safe to pass through. * VHOST_USER_NET_SET_MTU Safe to pass through. * VHOST_USER_SET_SLAVE_REQ_FD Stash slave req fd so that Slave-to-Master queue messages can be relayed before sending VHOST_USER_SET_SLAVE_REQ_FD to guest. * VHOST_USER_IOTLB_MSG Translate addresses if QEMU implements IOMMU before sending VHOST_USER_IOTLB_MSG to the guest. * VHOST_USER_SET_VRING_ENDIAN Safe to pass through. Slave message types ------------------- * VHOST_USER_SLAVE_IOTLB_MSG Handled by QEMU if it implements the IOMMU. > I took a quick look and I doubt we can do something that is both > compatible with the existing vhost-user and will make it possible to > extend the protocol without qemu changes. Let's assume I pass a new > message over the vhost-user channel. How do we know it's safe to pass > it to the guest? > > That's why we gate any protocol change on a feature bit and must parse > all messages. QEMU must parse all messages and cannot pass through unknown messages. Think of QEMU vhost-pci as both a vhost-user slave to the other VM and a vhost-user master to the guest. QEMU changes are necessary when the vhost protocol is extended. Device interface changes are only necessary if doorbells or shared memory regions are added, any other protocol changes do not change the device interface. Stefan
> * VHOST_USER_SET_VRING_KICK > > Set up vring kick doorbell (unless bit 8 is set) before sending > VHOST_USER_SET_VRING_KICK to the guest. But guest can't use it, now can it? What guest needs is a mapping to interrupts. > * VHOST_USER_SET_VRING_CALL > > Set up the vring call doorbell (unless bit 8 is set) before sending > VHOST_USER_SET_VRING_CALL to the guest. Same here. what guest needs is mapping from io to notifications, right? --- > > I took a quick look and I doubt we can do something that is both > > compatible with the existing vhost-user and will make it possible to > > extend the protocol without qemu changes. Let's assume I pass a new > > message over the vhost-user channel. How do we know it's safe to pass > > it to the guest? > > > > That's why we gate any protocol change on a feature bit and must parse > > all messages. > > QEMU must parse all messages and cannot pass through unknown messages. > Think of QEMU vhost-pci as both a vhost-user slave to the other VM and > a vhost-user master to the guest. > > QEMU changes are necessary when the vhost protocol is extended. > Device interface changes are only necessary if doorbells or shared > memory regions are added, any other protocol changes do not change the > device interface. > > Stefan I guess you have a different definition of a device interface than myself - I consider it an interface change if a feature bit changes :) -- MST
On Wed, Dec 13, 2017 at 10:59:10PM +0200, Michael S. Tsirkin wrote: > > * VHOST_USER_SET_VRING_KICK > > > > Set up vring kick doorbell (unless bit 8 is set) before sending > > VHOST_USER_SET_VRING_KICK to the guest. > > But guest can't use it, now can it? > > What guest needs is a mapping to interrupts. ... > > * VHOST_USER_SET_VRING_CALL > > > > Set up the vring call doorbell (unless bit 8 is set) before sending > > VHOST_USER_SET_VRING_CALL to the guest. > > Same here. what guest needs is mapping from io to notifications, > right? The PCI device should contain a BAR with doorbell registers. I don't think a fancy mapping is necessary, instead the device spec should define the BAR layout. When the guest vhost-user slave receives this message it knows it can now begin using the doorbell register. > --- > > > > I took a quick look and I doubt we can do something that is both > > > compatible with the existing vhost-user and will make it possible to > > > extend the protocol without qemu changes. Let's assume I pass a new > > > message over the vhost-user channel. How do we know it's safe to pass > > > it to the guest? > > > > > > That's why we gate any protocol change on a feature bit and must parse > > > all messages. > > > > QEMU must parse all messages and cannot pass through unknown messages. > > Think of QEMU vhost-pci as both a vhost-user slave to the other VM and > > a vhost-user master to the guest. > > > > QEMU changes are necessary when the vhost protocol is extended. > > Device interface changes are only necessary if doorbells or shared > > memory regions are added, any other protocol changes do not change the > > device interface. > > > > Stefan > > I guess you have a different definition of a device interface than > myself - I consider it an interface change if a feature bit changes :) The feature bits are defined in the vhost-user protocol specification, not in the vhost-pci PCI device specification. For example, imagine we are adding the virtio-net MTU feature to the protocol: 1. The VHOST_USER_PROTOCOL_F_MTU feature bit is added to the vhost-user protocol specification. 2. The VHOST_USER_NET_SET_MTU message is added to the vhost-user protocol specification. As a result of this: 1. No PCI adapter resources (BARs, register layout, etc) change. This is why I say the device interface is unchanged. The vhost-pci specification does not change. 2. QEMU vhost-pci code needs to unmask VHOST_USER_PROTOCOL_F_MTU and pass through VHOST_USER_NET_SET_MTU. Stefan
On 12/14/2017 11:06 PM, Stefan Hajnoczi wrote: > On Wed, Dec 13, 2017 at 10:59:10PM +0200, Michael S. Tsirkin wrote: >>> * VHOST_USER_SET_VRING_KICK >>> >>> Set up vring kick doorbell (unless bit 8 is set) before sending >>> VHOST_USER_SET_VRING_KICK to the guest. >> But guest can't use it, now can it? >> >> What guest needs is a mapping to interrupts. > ... >>> * VHOST_USER_SET_VRING_CALL >>> >>> Set up the vring call doorbell (unless bit 8 is set) before sending >>> VHOST_USER_SET_VRING_CALL to the guest. >> Same here. what guest needs is mapping from io to notifications, >> right? > The PCI device should contain a BAR with doorbell registers. I don't > think a fancy mapping is necessary, instead the device spec should > define the BAR layout. > > When the guest vhost-user slave receives this message it knows it can > now begin using the doorbell register. Not really. A doorbell will cause an interrupt to be injected to the master device driver, which is not ready to work at that time. The slave driver isn't expected to use the doorbell until the master is ready by sending the last message VHOST_USER_SET_VHOST_PCI to link UP the slave device. So I think passing the fd msg to guest doesn't have a value in terms of functionality. Best, Wei
On Fri, Dec 15, 2017 at 06:33:14PM +0800, Wei Wang wrote: > On 12/14/2017 11:06 PM, Stefan Hajnoczi wrote: > > On Wed, Dec 13, 2017 at 10:59:10PM +0200, Michael S. Tsirkin wrote: > > > > * VHOST_USER_SET_VRING_KICK > > > > > > > > Set up vring kick doorbell (unless bit 8 is set) before sending > > > > VHOST_USER_SET_VRING_KICK to the guest. > > > But guest can't use it, now can it? > > > > > > What guest needs is a mapping to interrupts. > > ... > > > > * VHOST_USER_SET_VRING_CALL > > > > > > > > Set up the vring call doorbell (unless bit 8 is set) before sending > > > > VHOST_USER_SET_VRING_CALL to the guest. > > > Same here. what guest needs is mapping from io to notifications, > > > right? > > The PCI device should contain a BAR with doorbell registers. I don't > > think a fancy mapping is necessary, instead the device spec should > > define the BAR layout. > > > > When the guest vhost-user slave receives this message it knows it can > > now begin using the doorbell register. > > Not really. A doorbell will cause an interrupt to be injected to the master > device driver, which is not ready to work at that time. The slave driver > isn't expected to use the doorbell until the master is ready by sending the > last message VHOST_USER_SET_VHOST_PCI to link UP the slave device. > > So I think passing the fd msg to guest doesn't have a value in terms of > functionality. The new VHOST_USER_SET_VHOST_PCI device message is something you defined in this patch series. This email sub-thread is about the vhost-user protocol specification as it exist today. The VHOST_USER_SET_VRING_CALL is the message that allows the guest to begin using the doorbell when VHOST_USER_F_PROTOCOL_FEATURES wasn't negotiated (i.e. the vring is enabled right away). Stefan
Hi Stefan, On 12/13/2017 09:08 PM, Stefan Hajnoczi wrote: > On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: >>> I'm not saying that DPDK should use libvhost-user. I'm saying that it's >>> easy to add vfio vhost-pci support (for the PCI adapter I described) to >>> DPDK. This patch series would require writing a completely new slave >>> for vhost-pci because the device interface is so different from >>> vhost-user. >> >> The main question is how appropriate is the vhost user protocol >> for passing to guests. And I am not sure at this point. >> >> Someone should go over vhost user messages and see whether they are safe >> to pass to guest. If most are then we can try the transparent approach. >> If most aren't then we can't and might as well use the proposed protocol >> which at least has code behind it. > > I have done that: > ... > * VHOST_USER_SET_MEM_TABLE > > Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. It would require to filter out userspace_addr from the payload not to leak other QEMU process VAs to the guest. Maxime
On Wed, Dec 13, 2017 at 10:50:11PM +0100, Maxime Coquelin wrote: > On 12/13/2017 09:08 PM, Stefan Hajnoczi wrote: > > On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > > On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: > > > > I'm not saying that DPDK should use libvhost-user. I'm saying that it's > > > > easy to add vfio vhost-pci support (for the PCI adapter I described) to > > > > DPDK. This patch series would require writing a completely new slave > > > > for vhost-pci because the device interface is so different from > > > > vhost-user. > > > > > > The main question is how appropriate is the vhost user protocol > > > for passing to guests. And I am not sure at this point. > > > > > > Someone should go over vhost user messages and see whether they are safe > > > to pass to guest. If most are then we can try the transparent approach. > > > If most aren't then we can't and might as well use the proposed protocol > > > which at least has code behind it. > > > > I have done that: > > > ... > > * VHOST_USER_SET_MEM_TABLE > > > > Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. > > It would require to filter out userspace_addr from the payload not to > leak other QEMU process VAs to the guest. QEMU's vhost-user master implementation is insecure because it leaks QEMU process VAs. This also affects vhost-user host processes, not just vhost-pci. The QEMU vhost-user master could send an post-IOMMU guest physical addresses whereever the vhost-user protocol specification says "user address". That way no address space information is leaked although it does leak IOMMU mappings. If we want to hide the IOMMU mappings too then we need another logical address space (kind a randomized ramaddr_t). Anyway, my point is that the current vhost-user master implementation is insecure and should be fixed. vhost-pci doesn't need to worry about this issue. Stefan
On Thu, Dec 14, 2017 at 03:46:56PM +0000, Stefan Hajnoczi wrote: > On Wed, Dec 13, 2017 at 10:50:11PM +0100, Maxime Coquelin wrote: > > On 12/13/2017 09:08 PM, Stefan Hajnoczi wrote: > > > On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: > > > > > I'm not saying that DPDK should use libvhost-user. I'm saying that it's > > > > > easy to add vfio vhost-pci support (for the PCI adapter I described) to > > > > > DPDK. This patch series would require writing a completely new slave > > > > > for vhost-pci because the device interface is so different from > > > > > vhost-user. > > > > > > > > The main question is how appropriate is the vhost user protocol > > > > for passing to guests. And I am not sure at this point. > > > > > > > > Someone should go over vhost user messages and see whether they are safe > > > > to pass to guest. If most are then we can try the transparent approach. > > > > If most aren't then we can't and might as well use the proposed protocol > > > > which at least has code behind it. > > > > > > I have done that: > > > > > ... > > > * VHOST_USER_SET_MEM_TABLE > > > > > > Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. > > > > It would require to filter out userspace_addr from the payload not to > > leak other QEMU process VAs to the guest. > > QEMU's vhost-user master implementation is insecure because it leaks > QEMU process VAs. This also affects vhost-user host processes, not just > vhost-pci. > > The QEMU vhost-user master could send an post-IOMMU guest physical > addresses whereever the vhost-user protocol specification says "user > address". That way no address space information is leaked although it > does leak IOMMU mappings. > > If we want to hide the IOMMU mappings too then we need another logical > address space (kind a randomized ramaddr_t). > > Anyway, my point is that the current vhost-user master implementation is > insecure and should be fixed. vhost-pci doesn't need to worry about > this issue. > > Stefan I was going to make this point too. It does not look like anyone uses userspace_addr. It might have been a mistake to put it there - maybe we should have reused it for map offset. It does not look like anyone uses this for anything. How about we put zero, or a copy of the GPA there? -- MST
On 12/14/2017 05:27 PM, Michael S. Tsirkin wrote: > On Thu, Dec 14, 2017 at 03:46:56PM +0000, Stefan Hajnoczi wrote: >> On Wed, Dec 13, 2017 at 10:50:11PM +0100, Maxime Coquelin wrote: >>> On 12/13/2017 09:08 PM, Stefan Hajnoczi wrote: >>>> On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >>>>> On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: >>>>>> I'm not saying that DPDK should use libvhost-user. I'm saying that it's >>>>>> easy to add vfio vhost-pci support (for the PCI adapter I described) to >>>>>> DPDK. This patch series would require writing a completely new slave >>>>>> for vhost-pci because the device interface is so different from >>>>>> vhost-user. >>>>> >>>>> The main question is how appropriate is the vhost user protocol >>>>> for passing to guests. And I am not sure at this point. >>>>> >>>>> Someone should go over vhost user messages and see whether they are safe >>>>> to pass to guest. If most are then we can try the transparent approach. >>>>> If most aren't then we can't and might as well use the proposed protocol >>>>> which at least has code behind it. >>>> >>>> I have done that: >>>> >>> ... >>>> * VHOST_USER_SET_MEM_TABLE >>>> >>>> Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. >>> >>> It would require to filter out userspace_addr from the payload not to >>> leak other QEMU process VAs to the guest. >> >> QEMU's vhost-user master implementation is insecure because it leaks >> QEMU process VAs. This also affects vhost-user host processes, not just >> vhost-pci. >> >> The QEMU vhost-user master could send an post-IOMMU guest physical >> addresses whereever the vhost-user protocol specification says "user >> address". That way no address space information is leaked although it >> does leak IOMMU mappings. >> >> If we want to hide the IOMMU mappings too then we need another logical >> address space (kind a randomized ramaddr_t). >> >> Anyway, my point is that the current vhost-user master implementation is >> insecure and should be fixed. vhost-pci doesn't need to worry about >> this issue. >> >> Stefan > > I was going to make this point too. It does not look like anyone uses > userspace_addr. It might have been a mistake to put it there - > maybe we should have reused it for map offset. > > It does not look like anyone uses this for anything. > > How about we put zero, or a copy of the GPA there? > > It is used when no iommu for the ring addresses, and when iommu is used for the IOTLB update messages. Maxime
On Thu, Dec 14, 2017 at 05:39:19PM +0100, Maxime Coquelin wrote: > > > On 12/14/2017 05:27 PM, Michael S. Tsirkin wrote: > > On Thu, Dec 14, 2017 at 03:46:56PM +0000, Stefan Hajnoczi wrote: > > > On Wed, Dec 13, 2017 at 10:50:11PM +0100, Maxime Coquelin wrote: > > > > On 12/13/2017 09:08 PM, Stefan Hajnoczi wrote: > > > > > On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: > > > > > > > I'm not saying that DPDK should use libvhost-user. I'm saying that it's > > > > > > > easy to add vfio vhost-pci support (for the PCI adapter I described) to > > > > > > > DPDK. This patch series would require writing a completely new slave > > > > > > > for vhost-pci because the device interface is so different from > > > > > > > vhost-user. > > > > > > > > > > > > The main question is how appropriate is the vhost user protocol > > > > > > for passing to guests. And I am not sure at this point. > > > > > > > > > > > > Someone should go over vhost user messages and see whether they are safe > > > > > > to pass to guest. If most are then we can try the transparent approach. > > > > > > If most aren't then we can't and might as well use the proposed protocol > > > > > > which at least has code behind it. > > > > > > > > > > I have done that: > > > > > > > > > ... > > > > > * VHOST_USER_SET_MEM_TABLE > > > > > > > > > > Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. > > > > > > > > It would require to filter out userspace_addr from the payload not to > > > > leak other QEMU process VAs to the guest. > > > > > > QEMU's vhost-user master implementation is insecure because it leaks > > > QEMU process VAs. This also affects vhost-user host processes, not just > > > vhost-pci. > > > > > > The QEMU vhost-user master could send an post-IOMMU guest physical > > > addresses whereever the vhost-user protocol specification says "user > > > address". That way no address space information is leaked although it > > > does leak IOMMU mappings. > > > > > > If we want to hide the IOMMU mappings too then we need another logical > > > address space (kind a randomized ramaddr_t). > > > > > > Anyway, my point is that the current vhost-user master implementation is > > > insecure and should be fixed. vhost-pci doesn't need to worry about > > > this issue. > > > > > > Stefan > > > > I was going to make this point too. It does not look like anyone uses > > userspace_addr. It might have been a mistake to put it there - > > maybe we should have reused it for map offset. > > > > It does not look like anyone uses this for anything. > > > > How about we put zero, or a copy of the GPA there? > > > > > > It is used when no iommu for the ring addresses, and when iommu is used > for the IOTLB update messages. > > Maxime How do clients use it? Why won't GPA do just as well? -- MST
On 12/14/2017 05:40 PM, Michael S. Tsirkin wrote: > On Thu, Dec 14, 2017 at 05:39:19PM +0100, Maxime Coquelin wrote: >> >> >> On 12/14/2017 05:27 PM, Michael S. Tsirkin wrote: >>> On Thu, Dec 14, 2017 at 03:46:56PM +0000, Stefan Hajnoczi wrote: >>>> On Wed, Dec 13, 2017 at 10:50:11PM +0100, Maxime Coquelin wrote: >>>>> On 12/13/2017 09:08 PM, Stefan Hajnoczi wrote: >>>>>> On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >>>>>>> On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: >>>>>>>> I'm not saying that DPDK should use libvhost-user. I'm saying that it's >>>>>>>> easy to add vfio vhost-pci support (for the PCI adapter I described) to >>>>>>>> DPDK. This patch series would require writing a completely new slave >>>>>>>> for vhost-pci because the device interface is so different from >>>>>>>> vhost-user. >>>>>>> >>>>>>> The main question is how appropriate is the vhost user protocol >>>>>>> for passing to guests. And I am not sure at this point. >>>>>>> >>>>>>> Someone should go over vhost user messages and see whether they are safe >>>>>>> to pass to guest. If most are then we can try the transparent approach. >>>>>>> If most aren't then we can't and might as well use the proposed protocol >>>>>>> which at least has code behind it. >>>>>> >>>>>> I have done that: >>>>>> >>>>> ... >>>>>> * VHOST_USER_SET_MEM_TABLE >>>>>> >>>>>> Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. >>>>> >>>>> It would require to filter out userspace_addr from the payload not to >>>>> leak other QEMU process VAs to the guest. >>>> >>>> QEMU's vhost-user master implementation is insecure because it leaks >>>> QEMU process VAs. This also affects vhost-user host processes, not just >>>> vhost-pci. >>>> >>>> The QEMU vhost-user master could send an post-IOMMU guest physical >>>> addresses whereever the vhost-user protocol specification says "user >>>> address". That way no address space information is leaked although it >>>> does leak IOMMU mappings. >>>> >>>> If we want to hide the IOMMU mappings too then we need another logical >>>> address space (kind a randomized ramaddr_t). >>>> >>>> Anyway, my point is that the current vhost-user master implementation is >>>> insecure and should be fixed. vhost-pci doesn't need to worry about >>>> this issue. >>>> >>>> Stefan >>> >>> I was going to make this point too. It does not look like anyone uses >>> userspace_addr. It might have been a mistake to put it there - >>> maybe we should have reused it for map offset. >>> >>> It does not look like anyone uses this for anything. >>> >>> How about we put zero, or a copy of the GPA there? >>> >>> >> >> It is used when no iommu for the ring addresses, and when iommu is used >> for the IOTLB update messages. >> >> Maxime > > How do clients use it? Why won't GPA do just as well? It is used to calculate the offset in the regions, so if we change all to use GPA, it may work without backend change. Maxime
On Thu, Dec 14, 2017 at 05:50:22PM +0100, Maxime Coquelin wrote: > > > On 12/14/2017 05:40 PM, Michael S. Tsirkin wrote: > > On Thu, Dec 14, 2017 at 05:39:19PM +0100, Maxime Coquelin wrote: > > > > > > > > > On 12/14/2017 05:27 PM, Michael S. Tsirkin wrote: > > > > On Thu, Dec 14, 2017 at 03:46:56PM +0000, Stefan Hajnoczi wrote: > > > > > On Wed, Dec 13, 2017 at 10:50:11PM +0100, Maxime Coquelin wrote: > > > > > > On 12/13/2017 09:08 PM, Stefan Hajnoczi wrote: > > > > > > > On Wed, Dec 13, 2017 at 3:01 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Wed, Dec 13, 2017 at 12:35:21PM +0000, Stefan Hajnoczi wrote: > > > > > > > > > I'm not saying that DPDK should use libvhost-user. I'm saying that it's > > > > > > > > > easy to add vfio vhost-pci support (for the PCI adapter I described) to > > > > > > > > > DPDK. This patch series would require writing a completely new slave > > > > > > > > > for vhost-pci because the device interface is so different from > > > > > > > > > vhost-user. > > > > > > > > > > > > > > > > The main question is how appropriate is the vhost user protocol > > > > > > > > for passing to guests. And I am not sure at this point. > > > > > > > > > > > > > > > > Someone should go over vhost user messages and see whether they are safe > > > > > > > > to pass to guest. If most are then we can try the transparent approach. > > > > > > > > If most aren't then we can't and might as well use the proposed protocol > > > > > > > > which at least has code behind it. > > > > > > > > > > > > > > I have done that: > > > > > > > > > > > > > ... > > > > > > > * VHOST_USER_SET_MEM_TABLE > > > > > > > > > > > > > > Set up BARs before sending a VHOST_USER_SET_MEM_TABLE to the guest. > > > > > > > > > > > > It would require to filter out userspace_addr from the payload not to > > > > > > leak other QEMU process VAs to the guest. > > > > > > > > > > QEMU's vhost-user master implementation is insecure because it leaks > > > > > QEMU process VAs. This also affects vhost-user host processes, not just > > > > > vhost-pci. > > > > > > > > > > The QEMU vhost-user master could send an post-IOMMU guest physical > > > > > addresses whereever the vhost-user protocol specification says "user > > > > > address". That way no address space information is leaked although it > > > > > does leak IOMMU mappings. > > > > > > > > > > If we want to hide the IOMMU mappings too then we need another logical > > > > > address space (kind a randomized ramaddr_t). > > > > > > > > > > Anyway, my point is that the current vhost-user master implementation is > > > > > insecure and should be fixed. vhost-pci doesn't need to worry about > > > > > this issue. > > > > > > > > > > Stefan > > > > > > > > I was going to make this point too. It does not look like anyone uses > > > > userspace_addr. It might have been a mistake to put it there - > > > > maybe we should have reused it for map offset. > > > > > > > > It does not look like anyone uses this for anything. > > > > > > > > How about we put zero, or a copy of the GPA there? > > > > > > > > > > > > > > It is used when no iommu for the ring addresses, and when iommu is used > > > for the IOTLB update messages. > > > > > > Maxime > > > > How do clients use it? Why won't GPA do just as well? > > It is used to calculate the offset in the regions, so if we change all > to use GPA, it may work without backend change. Great. Stefan
On 12/13/2017 08:35 PM, Stefan Hajnoczi wrote:
> On Wed, Dec 13, 2017 at 04:11:45PM +0800, Wei Wang wrote:
>
> I think the current approach is fine for a prototype but is not suitable
> for wider use by the community because it:
> 1. Does not scale to multiple device types (net, scsi, blk, etc)
> 2. Does not scale as the vhost-user protocol changes
> 3. It is hard to make slaves run in both host userspace and the guest
>
> It would be good to solve these problems so that vhost-pci can become
> successful. It's very hard to fix these things after the code is merged
> because guests will depend on the device interface.
>
> Here are the points in detail (in order of importance):
>
> 1. Does not scale to multiple device types (net, scsi, blk, etc)
>
> vhost-user is being applied to new device types beyond virtio-net.
> There will be demand for supporting other device types besides
> virtio-net with vhost-pci.
>
> This patch series requires defining a new virtio device type for each
> vhost-user device type. It is a lot of work to design a new virtio
> device. Additionally, the new virtio device type should become part of
> the VIRTIO standard, which can also take some time and requires writing
> a standards document.
>
> 2. Does not scale as the vhost-user protocol changes
>
> When the vhost-user protocol changes it will be necessary to update the
> vhost-pci device interface to reflect those changes. Each protocol
> change requires thinking how the virtio devices need to look in order to
> support the new behavior. Changes to the vhost-user protocol will
> result in changes to the VIRTIO specification for the vhost-pci virtio
> devices.
>
> 3. It is hard to make slaves run in both host userspace and the guest
>
> If a vhost-user slave wishes to support running in host userspace and
> the guest then not much code can be shared between these two modes since
> the interfaces are so different.
>
> How would you solve these issues?
1st one: I think we can factor out a common vhost-pci device layer in
QEMU. Specific devices (net, scsi etc) emulation comes on top of it. The
vhost-user protocol sets up VhostPCIDev only. So we will have something
like this:
struct VhostPCINet {
struct VhostPCIDev vp_dev;
u8 mac[8];
....
}
2nd one: I think we need to view it the other way around: If there is a
demand to change the protocol, then where is the demand from? I think
mostly it is because there is some new features from the device/driver.
That is, we first have already thought about how the virtio device looks
like with the new feature, then we add the support to the protocol. I'm
not sure how would it cause not scaling well, and how using another
GuestSlave-to-QemuMaster changes the story (we will also need to patch
the GuestSlave inside the VM to support the vhost-user negotiation of
the new feature), in comparison to the standard virtio feature negotiation.
3rd one: I'm not able to solve this one, as discussed, there are too
many differences and it's too complex. I prefer the direction of simply
gating the vhost-user protocol and deliver to the guest what it should
see (just what this patch series shows). You would need to solve this
issue to show this direction is simpler :)
Best,
Wei
On Thu, Dec 14, 2017 at 01:53:16PM +0800, Wei Wang wrote:
> On 12/13/2017 08:35 PM, Stefan Hajnoczi wrote:
> > On Wed, Dec 13, 2017 at 04:11:45PM +0800, Wei Wang wrote:
> >
> > I think the current approach is fine for a prototype but is not suitable
> > for wider use by the community because it:
> > 1. Does not scale to multiple device types (net, scsi, blk, etc)
> > 2. Does not scale as the vhost-user protocol changes
> > 3. It is hard to make slaves run in both host userspace and the guest
> >
> > It would be good to solve these problems so that vhost-pci can become
> > successful. It's very hard to fix these things after the code is merged
> > because guests will depend on the device interface.
> >
> > Here are the points in detail (in order of importance):
> >
> > 1. Does not scale to multiple device types (net, scsi, blk, etc)
> >
> > vhost-user is being applied to new device types beyond virtio-net.
> > There will be demand for supporting other device types besides
> > virtio-net with vhost-pci.
> >
> > This patch series requires defining a new virtio device type for each
> > vhost-user device type. It is a lot of work to design a new virtio
> > device. Additionally, the new virtio device type should become part of
> > the VIRTIO standard, which can also take some time and requires writing
> > a standards document.
> >
> > 2. Does not scale as the vhost-user protocol changes
> >
> > When the vhost-user protocol changes it will be necessary to update the
> > vhost-pci device interface to reflect those changes. Each protocol
> > change requires thinking how the virtio devices need to look in order to
> > support the new behavior. Changes to the vhost-user protocol will
> > result in changes to the VIRTIO specification for the vhost-pci virtio
> > devices.
> >
> > 3. It is hard to make slaves run in both host userspace and the guest
> >
> > If a vhost-user slave wishes to support running in host userspace and
> > the guest then not much code can be shared between these two modes since
> > the interfaces are so different.
> >
> > How would you solve these issues?
>
> 1st one: I think we can factor out a common vhost-pci device layer in QEMU.
> Specific devices (net, scsi etc) emulation comes on top of it. The
> vhost-user protocol sets up VhostPCIDev only. So we will have something like
> this:
>
> struct VhostPCINet {
> struct VhostPCIDev vp_dev;
> u8 mac[8];
> ....
> }
Defining VhostPCIDev is an important step to making it easy to implement
other device types. I'm interested is seeing how this would look either
in code or in a more detailed outline.
I wonder what the device-specific parts will be. This patch series does
not implement a fully functional vhost-user-net device so I'm not sure.
> 2nd one: I think we need to view it the other way around: If there is a
> demand to change the protocol, then where is the demand from? I think mostly
> it is because there is some new features from the device/driver. That is, we
> first have already thought about how the virtio device looks like with the
> new feature, then we add the support to the protocol.
The vhost-user protocol will change when people using host userspace
slaves decide to change it. They may not know or care about vhost-pci,
so the virtio changes will be an afterthought that falls on whoever
wants to support vhost-pci.
This is why I think it makes a lot more sense to stick to the vhost-user
protocol as the vhost-pci slave interface instead of inventing a new
interface on top of it.
> I'm not sure how would
> it cause not scaling well, and how using another GuestSlave-to-QemuMaster
> changes the story (we will also need to patch the GuestSlave inside the VM
> to support the vhost-user negotiation of the new feature), in comparison to
> the standard virtio feature negotiation.
Plus the VIRTIO specification needs to be updated.
And if the vhost-user protocol change affects all device types then it
may be necessary to change multiple virtio devices! This is O(1) vs
O(N).
> 3rd one: I'm not able to solve this one, as discussed, there are too many
> differences and it's too complex. I prefer the direction of simply gating
> the vhost-user protocol and deliver to the guest what it should see (just
> what this patch series shows). You would need to solve this issue to show
> this direction is simpler :)
#3 is nice to have but not critical.
In the approach I suggested it would be done by implementing vfio
vhost-pci for libvhost-user or DPDK.
Stefan
On 12/15/2017 01:32 AM, Stefan Hajnoczi wrote:
> On Thu, Dec 14, 2017 at 01:53:16PM +0800, Wei Wang wrote:
>> On 12/13/2017 08:35 PM, Stefan Hajnoczi wrote:
>>> On Wed, Dec 13, 2017 at 04:11:45PM +0800, Wei Wang wrote:
>>>
>>> I think the current approach is fine for a prototype but is not suitable
>>> for wider use by the community because it:
>>> 1. Does not scale to multiple device types (net, scsi, blk, etc)
>>> 2. Does not scale as the vhost-user protocol changes
>>> 3. It is hard to make slaves run in both host userspace and the guest
>>>
>>> It would be good to solve these problems so that vhost-pci can become
>>> successful. It's very hard to fix these things after the code is merged
>>> because guests will depend on the device interface.
>>>
>>> Here are the points in detail (in order of importance):
>>>
>>> 1. Does not scale to multiple device types (net, scsi, blk, etc)
>>>
>>> vhost-user is being applied to new device types beyond virtio-net.
>>> There will be demand for supporting other device types besides
>>> virtio-net with vhost-pci.
>>>
>>> This patch series requires defining a new virtio device type for each
>>> vhost-user device type. It is a lot of work to design a new virtio
>>> device. Additionally, the new virtio device type should become part of
>>> the VIRTIO standard, which can also take some time and requires writing
>>> a standards document.
>>>
>>> 2. Does not scale as the vhost-user protocol changes
>>>
>>> When the vhost-user protocol changes it will be necessary to update the
>>> vhost-pci device interface to reflect those changes. Each protocol
>>> change requires thinking how the virtio devices need to look in order to
>>> support the new behavior. Changes to the vhost-user protocol will
>>> result in changes to the VIRTIO specification for the vhost-pci virtio
>>> devices.
>>>
>>> 3. It is hard to make slaves run in both host userspace and the guest
>>>
>>> If a vhost-user slave wishes to support running in host userspace and
>>> the guest then not much code can be shared between these two modes since
>>> the interfaces are so different.
>>>
>>> How would you solve these issues?
>> 1st one: I think we can factor out a common vhost-pci device layer in QEMU.
>> Specific devices (net, scsi etc) emulation comes on top of it. The
>> vhost-user protocol sets up VhostPCIDev only. So we will have something like
>> this:
>>
>> struct VhostPCINet {
>> struct VhostPCIDev vp_dev;
>> u8 mac[8];
>> ....
>> }
> Defining VhostPCIDev is an important step to making it easy to implement
> other device types. I'm interested is seeing how this would look either
> in code or in a more detailed outline.
>
> I wonder what the device-specific parts will be. This patch series does
> not implement a fully functional vhost-user-net device so I'm not sure.
I think we can move most of the fields from this series' VhostPCINet to
VhostPCIDev:
struct VhostPCIDev {
VirtIODevice parent_obj;
MemoryRegion bar_region;
MemoryRegion metadata_region;
struct vhost_pci_metadata *metadata;
void *remote_mem_base[MAX_REMOTE_REGION];
uint64_t remote_mem_map_size[MAX_REMOTE_REGION];
CharBackend chr_be;
};
struct VhostPCINet {
struct VhostPCIDev vp_dev;
uint32_t host_features;
struct vpnet_config config;
size_t config_size;
uint16_t status;
}
>
>> 2nd one: I think we need to view it the other way around: If there is a
>> demand to change the protocol, then where is the demand from? I think mostly
>> it is because there is some new features from the device/driver. That is, we
>> first have already thought about how the virtio device looks like with the
>> new feature, then we add the support to the protocol.
> The vhost-user protocol will change when people using host userspace
> slaves decide to change it. They may not know or care about vhost-pci,
> so the virtio changes will be an afterthought that falls on whoever
> wants to support vhost-pci.
>
> This is why I think it makes a lot more sense to stick to the vhost-user
> protocol as the vhost-pci slave interface instead of inventing a new
> interface on top of it.
I don't think it is different in practice. If the added protocol msg
needs some setup on the vhost-pci device side, then we will also need to
think about how to use it for the device setup explicitly, with the
possibility of delivering the modified msg to the guest (like
SET_MEM_TABLE) if using the relaying method.
Vhost-pci takes "advantage" of the vhost-user protocol for the inter-VM
data path setup. If the added vhost-user message isn't useful for
vhost-pci setup, I think the slave don't need to handle it even.
>
>> I'm not sure how would
>> it cause not scaling well, and how using another GuestSlave-to-QemuMaster
>> changes the story (we will also need to patch the GuestSlave inside the VM
>> to support the vhost-user negotiation of the new feature), in comparison to
>> the standard virtio feature negotiation.
> Plus the VIRTIO specification needs to be updated.
>
> And if the vhost-user protocol change affects all device types then it
> may be necessary to change multiple virtio devices! This is O(1) vs
> O(N).
>
If this change is common to all the vhost-pci series devices, the change
will be made to the vhost-pci layer (i.e. VhostPCIDev) only.
Best,
Wei
On Fri, Dec 15, 2017 at 05:10:37PM +0800, Wei Wang wrote:
> On 12/15/2017 01:32 AM, Stefan Hajnoczi wrote:
> > On Thu, Dec 14, 2017 at 01:53:16PM +0800, Wei Wang wrote:
> > > On 12/13/2017 08:35 PM, Stefan Hajnoczi wrote:
> > > > On Wed, Dec 13, 2017 at 04:11:45PM +0800, Wei Wang wrote:
> > > >
> > > > I think the current approach is fine for a prototype but is not suitable
> > > > for wider use by the community because it:
> > > > 1. Does not scale to multiple device types (net, scsi, blk, etc)
> > > > 2. Does not scale as the vhost-user protocol changes
> > > > 3. It is hard to make slaves run in both host userspace and the guest
> > > >
> > > > It would be good to solve these problems so that vhost-pci can become
> > > > successful. It's very hard to fix these things after the code is merged
> > > > because guests will depend on the device interface.
> > > >
> > > > Here are the points in detail (in order of importance):
> > > >
> > > > 1. Does not scale to multiple device types (net, scsi, blk, etc)
> > > >
> > > > vhost-user is being applied to new device types beyond virtio-net.
> > > > There will be demand for supporting other device types besides
> > > > virtio-net with vhost-pci.
> > > >
> > > > This patch series requires defining a new virtio device type for each
> > > > vhost-user device type. It is a lot of work to design a new virtio
> > > > device. Additionally, the new virtio device type should become part of
> > > > the VIRTIO standard, which can also take some time and requires writing
> > > > a standards document.
> > > >
> > > > 2. Does not scale as the vhost-user protocol changes
> > > >
> > > > When the vhost-user protocol changes it will be necessary to update the
> > > > vhost-pci device interface to reflect those changes. Each protocol
> > > > change requires thinking how the virtio devices need to look in order to
> > > > support the new behavior. Changes to the vhost-user protocol will
> > > > result in changes to the VIRTIO specification for the vhost-pci virtio
> > > > devices.
> > > >
> > > > 3. It is hard to make slaves run in both host userspace and the guest
> > > >
> > > > If a vhost-user slave wishes to support running in host userspace and
> > > > the guest then not much code can be shared between these two modes since
> > > > the interfaces are so different.
> > > >
> > > > How would you solve these issues?
> > > 1st one: I think we can factor out a common vhost-pci device layer in QEMU.
> > > Specific devices (net, scsi etc) emulation comes on top of it. The
> > > vhost-user protocol sets up VhostPCIDev only. So we will have something like
> > > this:
> > >
> > > struct VhostPCINet {
> > > struct VhostPCIDev vp_dev;
> > > u8 mac[8];
> > > ....
> > > }
> > Defining VhostPCIDev is an important step to making it easy to implement
> > other device types. I'm interested is seeing how this would look either
> > in code or in a more detailed outline.
> >
> > I wonder what the device-specific parts will be. This patch series does
> > not implement a fully functional vhost-user-net device so I'm not sure.
>
> I think we can move most of the fields from this series' VhostPCINet to
> VhostPCIDev:
>
> struct VhostPCIDev {
> VirtIODevice parent_obj;
> MemoryRegion bar_region;
> MemoryRegion metadata_region;
> struct vhost_pci_metadata *metadata;
> void *remote_mem_base[MAX_REMOTE_REGION];
> uint64_t remote_mem_map_size[MAX_REMOTE_REGION];
> CharBackend chr_be;
> };
>
> struct VhostPCINet {
> struct VhostPCIDev vp_dev;
> uint32_t host_features;
> struct vpnet_config config;
> size_t config_size;
> uint16_t status;
> }
>
>
>
>
> >
> > > 2nd one: I think we need to view it the other way around: If there is a
> > > demand to change the protocol, then where is the demand from? I think mostly
> > > it is because there is some new features from the device/driver. That is, we
> > > first have already thought about how the virtio device looks like with the
> > > new feature, then we add the support to the protocol.
> > The vhost-user protocol will change when people using host userspace
> > slaves decide to change it. They may not know or care about vhost-pci,
> > so the virtio changes will be an afterthought that falls on whoever
> > wants to support vhost-pci.
> >
> > This is why I think it makes a lot more sense to stick to the vhost-user
> > protocol as the vhost-pci slave interface instead of inventing a new
> > interface on top of it.
>
> I don't think it is different in practice. If the added protocol msg needs
> some setup on the vhost-pci device side, then we will also need to think
> about how to use it for the device setup explicitly, with the possibility of
> delivering the modified msg to the guest (like SET_MEM_TABLE) if using the
> relaying method.
>
> Vhost-pci takes "advantage" of the vhost-user protocol for the inter-VM data
> path setup. If the added vhost-user message isn't useful for vhost-pci
> setup, I think the slave don't need to handle it even.
vhost-pci setup is incomplete in this patch series. Currently the guest
slave is unable to participate in feature negotiation or limit the
number of queues. Reconnection is also not supported yet.
Guest slaves must be able to use a vhost-pci device without requiring
QEMU -device options (like max queues, feature bits, etc) for that
specific slave implementation. Why? In a cloud environment it is
incovenient or impossible for users to add QEMU -device options for the
slave implementation they are using.
Addressing these issues will require more vhost-pci<->guest slave
communication. The number of messages that can be hidden will shrink.
Stefan
On Thu, Dec 14, 2017 at 01:53:16PM +0800, Wei Wang wrote: > 3rd one: I'm not able to solve this one, as discussed, there are too many > differences and it's too complex. I prefer the direction of simply gating > the vhost-user protocol and deliver to the guest what it should see (just > what this patch series shows). You would need to solve this issue to show > this direction is simpler :) At the moment the main issue deciding what to do seems to be that we have no patches for the approach I've described. I'll begin working on a patch series. If there is something else you think I could help with, please let me know. I'd like to contribute to vhost-pci. Stefan
On 12/15/2017 02:04 AM, Stefan Hajnoczi wrote: > On Thu, Dec 14, 2017 at 01:53:16PM +0800, Wei Wang wrote: >> 3rd one: I'm not able to solve this one, as discussed, there are too many >> differences and it's too complex. I prefer the direction of simply gating >> the vhost-user protocol and deliver to the guest what it should see (just >> what this patch series shows). You would need to solve this issue to show >> this direction is simpler :) > At the moment the main issue deciding what to do seems to be that we > have no patches for the approach I've described. I'll begin working on > a patch series. Are you planning to implement the multiple masters and slaves? Master-->QemuSlave1-->GuestSlave Master<--QemuSlave2<--GuestSlave > If there is something else you think I could help with, please let me > know. I'd like to contribute to vhost-pci. > Thanks. Best, Wei
On Fri, Dec 15, 2017 at 06:33:45PM +0800, Wei Wang wrote:
> On 12/15/2017 02:04 AM, Stefan Hajnoczi wrote:
> > On Thu, Dec 14, 2017 at 01:53:16PM +0800, Wei Wang wrote:
> > > 3rd one: I'm not able to solve this one, as discussed, there are too many
> > > differences and it's too complex. I prefer the direction of simply gating
> > > the vhost-user protocol and deliver to the guest what it should see (just
> > > what this patch series shows). You would need to solve this issue to show
> > > this direction is simpler :)
> > At the moment the main issue deciding what to do seems to be that we
> > have no patches for the approach I've described. I'll begin working on
> > a patch series.
>
>
> Are you planning to implement the multiple masters and slaves?
> Master-->QemuSlave1-->GuestSlave
> Master<--QemuSlave2<--GuestSlave
Not sure if we are thinking of the same model. Here's what I have in
mind:
Master <-> QEMU virtio-vhost-user-slave <-> GuestSlave
The virtio-vhost-user-slave device implements a vhost-user slave on the
host side and a virtio device that acts as a vhost-user master to the
guest. This is a single virtio device that speaks the vhost-user
protocol, not a family of different devices.
("virtio-vhost-user-slave" is the name to prevent confusion with your
vhost-pci work.)
I think your idea of using virtio is good since it avoids duplicating
low-level PCI device initialization and queues, so I've decided to try
that.
I will send a draft device specification later today so you can review
it. I think the virtio PCI transport mapping that I'm documenting will
be useful whether we choose vhost-pci or virtio-vhost-user-slave.
Stefan
On Wed, Dec 6, 2017 at 1:49 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote: > On Tue, Dec 05, 2017 at 11:33:09AM +0800, Wei Wang wrote: >> Vhost-pci is a point-to-point based inter-VM communication solution. This >> patch series implements the vhost-pci-net device setup and emulation. The >> device is implemented as a virtio device, and it is set up via the >> vhost-user protocol to get the neessary info (e.g the memory info of the >> remote VM, vring info). >> >> Currently, only the fundamental functions are implemented. More features, >> such as MQ and live migration, will be updated in the future. >> >> The DPDK PMD of vhost-pci has been posted to the dpdk mailinglist here: >> http://dpdk.org/ml/archives/dev/2017-November/082615.html > > I have asked questions about the scope of this feature. In particular, > I think it's best to support all device types rather than just > virtio-net. Here is a design document that shows how this can be > achieved. > > What I'm proposing is different from the current approach: > 1. It's a PCI adapter (see below for justification) > 2. The vhost-user protocol is exposed by the device (not handled 100% in > QEMU). Ultimately I think your approach would also need to do this. Michael asked me to provide more information on the differences between this patch series and my proposal: My understanding of this patch series is: it adds a new virtio device type called vhost-pci-net. The QEMU vhost-pci-net code implements the vhost-user protocol and then exposes virtio-net-specific functionality to the guest. This means the vhost-pci-net driver inside the guest doesn't speak vhost-user, it speaks vhost-pci-net. Currently no virtqueues are defined so this is a very unusual virtio device. It also relies on a PCI BAR for shared memory access. Some vhost-user features like multiple virtqueues, logging (migration), etc are not supported. This proposal takes a different approach. Instead of create a new virtio device type (e.g. vhost-pci-net) for each device type (e.g. virtio-net, virtio-scsi, virtio-blk), it defines a vhost-pci PCI adapter that allows the guest to speak the vhost-user protocol. The vhost-pci device maps the vhost-user protocol to a PCI adapter so that software running inside the guest can basically speak the vhost-user protocol. It requires less logic inside QEMU except to handle vhost-user file descriptor passing. It allows guests to decide whether logging (migration) and other features are supported. It allows optimized irqfd <-> ioeventfd signalling which cannot be done with regular virtio devices. > I'm not implementing this and not asking you to implement it. Let's > just use this for discussion so we can figure out what the final > vhost-pci will look like. > > Please let me know what you think, Wei, Michael, and others. > > --- > vhost-pci device specification > ------------------------------- > The vhost-pci device allows guests to act as vhost-user slaves. This > enables appliance VMs like network switches or storage targets to back > devices in other VMs. VM-to-VM communication is possible without > vmexits using polling mode drivers. > > The vhost-user protocol has been used to implement virtio devices in > userspace processes on the host. vhost-pci maps the vhost-user protocol > to a PCI adapter so guest software can perform virtio device emulation. > This is useful in environments where high-performance VM-to-VM > communication is necessary or where it is preferrable to deploy device > emulation as VMs instead of host userspace processes. > > The vhost-user protocol involves file descriptor passing and shared > memory. This precludes vhost-user slave implementations over > virtio-vsock, virtio-serial, or TCP/IP. Therefore a new device type is > needed to expose the vhost-user protocol to guests. > > The vhost-pci PCI adapter has the following resources: > > Queues (used for vhost-user protocol communication): > 1. Master-to-slave messages > 2. Slave-to-master messages > > Doorbells (used for slave->guest/master events): > 1. Vring call (one doorbell per virtqueue) > 2. Vring err (one doorbell per virtqueue) > 3. Log changed > > Interrupts (used for guest->slave events): > 1. Vring kick (one MSI per virtqueue) > > Shared Memory BARs: > 1. Guest memory > 2. Log > > Master-to-slave queue: > The following vhost-user protocol messages are relayed from the > vhost-user master. Each message follows the vhost-user protocol > VhostUserMsg layout. > > Messages that include file descriptor passing are relayed but do not > carry file descriptors. The relevant resources (doorbells, interrupts, > or shared memory BARs) are initialized from the file descriptors prior > to the message becoming available on the Master-to-Slave queue. > > Resources must only be used after the corresponding vhost-user message > has been received. For example, the Vring call doorbell can only be > used after VHOST_USER_SET_VRING_CALL becomes available on the > Master-to-Slave queue. > > Messages must be processed in order. > > The following vhost-user protocol messages are relayed: > * VHOST_USER_GET_FEATURES > * VHOST_USER_SET_FEATURES > * VHOST_USER_GET_PROTOCOL_FEATURES > * VHOST_USER_SET_PROTOCOL_FEATURES > * VHOST_USER_SET_OWNER > * VHOST_USER_SET_MEM_TABLE > The shared memory is available in the corresponding BAR. > * VHOST_USER_SET_LOG_BASE > The shared memory is available in the corresponding BAR. > * VHOST_USER_SET_LOG_FD > The logging file descriptor can be signalled through the logging > virtqueue. > * VHOST_USER_SET_VRING_NUM > * VHOST_USER_SET_VRING_ADDR > * VHOST_USER_SET_VRING_BASE > * VHOST_USER_GET_VRING_BASE > * VHOST_USER_SET_VRING_KICK > This message is still needed because it may indicate only polling > mode is supported. > * VHOST_USER_SET_VRING_CALL > This message is still needed because it may indicate only polling > mode is supported. > * VHOST_USER_SET_VRING_ERR > * VHOST_USER_GET_QUEUE_NUM > * VHOST_USER_SET_VRING_ENABLE > * VHOST_USER_SEND_RARP > * VHOST_USER_NET_SET_MTU > * VHOST_USER_SET_SLAVE_REQ_FD > * VHOST_USER_IOTLB_MSG > * VHOST_USER_SET_VRING_ENDIAN > > Slave-to-Master queue: > Messages added to the Slave-to-Master queue are sent to the vhost-user > master. Each message follows the vhost-user protocol VhostUserMsg > layout. > > The following vhost-user protocol messages are relayed: > > * VHOST_USER_SLAVE_IOTLB_MSG > > Theory of Operation: > When the vhost-pci adapter is detected the queues must be set up by the > driver. Once the driver is ready the vhost-pci device begins relaying > vhost-user protocol messages over the Master-to-Slave queue. The driver > must follow the vhost-user protocol specification to implement > virtio device initialization and virtqueue processing. > > Notes: > The vhost-user UNIX domain socket connects two host processes. The > slave process interprets messages and initializes vhost-pci resources > (doorbells, interrupts, shared memory BARs) based on them before > relaying via the Master-to-Slave queue. All messages are relayed, even > if they only pass a file descriptor, because the message itself may act > as a signal (e.g. virtqueue is now enabled). > > vhost-pci is a PCI adapter instead of a virtio device to allow doorbells > and interrupts to be connected to the virtio device in the master VM in > the most efficient way possible. This means the Vring call doorbell can > be an ioeventfd that signals an irqfd inside the host kernel without > host userspace involvement. The Vring kick interrupt can be an irqfd > that is signalled by the master VM's virtqueue ioeventfd. > > It may be possible to write a Linux vhost-pci driver that implements the > drivers/vhost/ API. That way existing vhost drivers could work with > vhost-pci in the kernel. > > Guest userspace vhost-pci drivers will be similar to QEMU's > contrib/libvhost-user/ except they will probably use vfio to access the > vhost-pci device directly from userspace. > > TODO: > * Queue memory layout and hardware registers > * vhost-pci-level negotiation and configuration so the hardware > interface can be extended in the future. > * vhost-pci <-> driver initialization procedure > * Master<->Slave disconnected & reconnect
© 2016 - 2024 Red Hat, Inc.