Series comparison

-[Qemu-devel] [PULL 00/12] Block patches
+[PULL 00/18] Block patches
-The following changes since commit bfec359afba088aaacc7d316f43302f28c6e642a:
+The following changes since commit 8e6c70b9d4a1b1f3011805947925cfdb31642f7f:
-  Merge remote-tracking branch 'remotes/armbru/tags/pull-qdev-2017-04-21' into staging (2017-04-21 11:42:03 +0100)
+  Merge tag 'kraxel-20220614-pull-request' of git://git.kraxel.org/qemu into staging (2022-06-14 06:21:46 -0700)
-are available in the git repository at:
+are available in the Git repository at:
-  git://github.com/codyprime/qemu-kvm-jtc.git tags/block-pull-request
+  https://gitlab.com/stefanha/qemu.git tags/block-pull-request
-for you to fetch changes up to 1507631e438930bc07f776f303af127a9cdb4d41:
+for you to fetch changes up to 99b969fbe105117f5af6060d3afef40ca39cc9c1:
-  qemu-iotests: _cleanup_qemu must be called on exit (2017-04-21 08:32:44 -0400)
+  linux-aio: explain why max batch is checked in laio_io_unplug() (2022-06-15 16:43:42 +0100)
 ----------------------------------------------------------------
 Pull request
 This pull request includes an important aio=native I/O stall fix, the
 experimental vifo-user server, the io_uring_register_ring_fd() optimization for
 aio=io_uring, and an update to Vladimir Sementsov-Ogievskiy's maintainership
 details.
 ----------------------------------------------------------------
-Block patches for 2.10
+Jagannathan Raman (14):
   qdev: unplug blocker for devices
   remote/machine: add HotplugHandler for remote machine
   remote/machine: add vfio-user property
   vfio-user: build library
   vfio-user: define vfio-user-server object
   vfio-user: instantiate vfio-user context
   vfio-user: find and init PCI device
   vfio-user: run vfio-user context
   vfio-user: handle PCI config space accesses
   vfio-user: IOMMU support for remote device
   vfio-user: handle DMA mappings
   vfio-user: handle PCI BAR accesses
   vfio-user: handle device interrupts
   vfio-user: handle reset of remote device
-----------------------------------------------------------------
+Sam Li (1):
   Use io_uring_register_ring_fd() to skip fd operations
-Ashish Mittal (2):
+Stefan Hajnoczi (2):
-  block/vxhs.c: Add support for a new block device type called "vxhs"
+  linux-aio: fix unbalanced plugged counter in laio_io_unplug()
-  block/vxhs.c: Add qemu-iotests for new block device type "vxhs"
+  linux-aio: explain why max batch is checked in laio_io_unplug()
-Jeff Cody (10):
+Vladimir Sementsov-Ogievskiy (1):
-  qemu-iotests: exclude vxhs from image creation via protocol
+  MAINTAINERS: update Vladimir's address and repositories
   block: add bdrv_set_read_only() helper function
   block: do not set BDS read_only if copy_on_read enabled
   block: honor BDRV_O_ALLOW_RDWR when clearing bs->read_only
   block: code movement
   block: introduce bdrv_can_set_read_only()
   block: use bdrv_can_set_read_only() during reopen
   block/rbd - update variable names to more apt names
   block/rbd: Add support for reopen()
   qemu-iotests: _cleanup_qemu must be called on exit
- block.c                          |  56 +++-
+ MAINTAINERS                             |  27 +-
- block/Makefile.objs              |   2 +
+ meson_options.txt                       |   2 +
- block/bochs.c                    |   5 +-
+ qapi/misc.json                          |  31 +
- block/cloop.c                    |   5 +-
+ qapi/qom.json                           |  20 +-
- block/dmg.c                      |   6 +-
+ configure                               |  17 +
- block/rbd.c                      |  65 +++--
+ meson.build                             |  24 +-
- block/trace-events               |  17 ++
+ include/exec/memory.h                   |   3 +
- block/vvfat.c                    |  19 +-
+ include/hw/pci/msi.h                    |   1 +
- block/vxhs.c                     | 575 +++++++++++++++++++++++++++++++++++++++
+ include/hw/pci/msix.h                   |   1 +
- configure                        |  39 +++
+ include/hw/pci/pci.h                    |  13 +
- include/block/block.h            |   2 +
+ include/hw/qdev-core.h                  |  29 +
- qapi/block-core.json             |  23 +-
+ include/hw/remote/iommu.h               |  40 +
- tests/qemu-iotests/017           |   1 +
+ include/hw/remote/machine.h             |   4 +
- tests/qemu-iotests/020           |   1 +
+ include/hw/remote/vfio-user-obj.h       |   6 +
- tests/qemu-iotests/028           |   1 +
+ block/io_uring.c                        |  12 +-
- tests/qemu-iotests/029           |   1 +
+ block/linux-aio.c                       |  10 +-
- tests/qemu-iotests/073           |   1 +
+ hw/core/qdev.c                          |  24 +
- tests/qemu-iotests/094           |  11 +-
+ hw/pci/msi.c                            |  49 +-
- tests/qemu-iotests/102           |   5 +-
+ hw/pci/msix.c                           |  35 +-
- tests/qemu-iotests/109           |   1 +
+ hw/pci/pci.c                            |  13 +
- tests/qemu-iotests/114           |   1 +
+ hw/remote/iommu.c                       | 131 ++++
- tests/qemu-iotests/117           |   1 +
+ hw/remote/machine.c                     |  88 ++-
- tests/qemu-iotests/130           |   2 +
+ hw/remote/vfio-user-obj.c               | 958 ++++++++++++++++++++++++
- tests/qemu-iotests/134           |   1 +
+ softmmu/physmem.c                       |   4 +-
- tests/qemu-iotests/140           |   1 +
+ softmmu/qdev-monitor.c                  |   4 +
- tests/qemu-iotests/141           |   1 +
+ stubs/vfio-user-obj.c                   |   6 +
- tests/qemu-iotests/143           |   1 +
+ tests/qtest/fuzz/generic_fuzz.c         |   9 +-
- tests/qemu-iotests/156           |   2 +
+ .gitlab-ci.d/buildtest.yml              |   1 +
- tests/qemu-iotests/158           |   1 +
+ .gitmodules                             |   3 +
- tests/qemu-iotests/common        |   6 +
+ Kconfig.host                            |   4 +
- tests/qemu-iotests/common.config |  13 +
+ hw/remote/Kconfig                       |   4 +
- tests/qemu-iotests/common.filter |   1 +
+ hw/remote/meson.build                   |   4 +
- tests/qemu-iotests/common.rc     |  19 ++
+ hw/remote/trace-events                  |  11 +
-files changed, 844 insertions(+), 42 deletions(-)
+ scripts/meson-buildoptions.sh           |   4 +
- create mode 100644 block/vxhs.c
+ stubs/meson.build                       |   1 +
  subprojects/libvfio-user                |   1 +
  tests/docker/dockerfiles/centos8.docker |   2 +
 files changed, 1565 insertions(+), 31 deletions(-)
  create mode 100644 include/hw/remote/iommu.h
  create mode 100644 include/hw/remote/vfio-user-obj.h
  create mode 100644 hw/remote/iommu.c
  create mode 100644 hw/remote/vfio-user-obj.c
  create mode 100644 stubs/vfio-user-obj.c
  create mode 160000 subprojects/libvfio-user
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 03/12] qemu-iotests: exclude vxhs from image creation via protocol
+[PULL 01/18] MAINTAINERS: update Vladimir's address and repositories
-The protocol VXHS does not support image creation.  Some tests expect
+From: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 to be able to create images through the protocol.  Exclude VXHS from
 these tests.
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 Message-id: 20220526115432.138384-1-vsementsov@yandex-team.ru
 Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
- tests/qemu-iotests/017 | 1 +
+ MAINTAINERS | 22 ++++++++++++----------
- tests/qemu-iotests/020 | 1 +
+file changed, 12 insertions(+), 10 deletions(-)
  tests/qemu-iotests/029 | 1 +
  tests/qemu-iotests/073 | 1 +
  tests/qemu-iotests/114 | 1 +
  tests/qemu-iotests/130 | 1 +
  tests/qemu-iotests/134 | 1 +
  tests/qemu-iotests/156 | 1 +
  tests/qemu-iotests/158 | 1 +
 files changed, 9 insertions(+)
-diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
+diff --git a/MAINTAINERS b/MAINTAINERS
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/017
+--- a/MAINTAINERS
-+++ b/tests/qemu-iotests/017
++++ b/MAINTAINERS
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ F: scsi/*
- # Any format supporting backing files
- _supported_fmt qcow qcow2 vmdk qed
+ Block Jobs
- _supported_proto generic
+ M: John Snow <jsnow@redhat.com>
-+_unsupported_proto vxhs
+-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
- _supported_os Linux
++M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
- _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+ L: qemu-block@nongnu.org
+ S: Supported
-diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
+ F: blockjob.c
-index XXXXXXX..XXXXXXX 100755
+@@ -XXX,XX +XXX,XX @@ F: block/aio_task.c
---- a/tests/qemu-iotests/020
+ F: util/qemu-co-shared-resource.c
-+++ b/tests/qemu-iotests/020
+ F: include/qemu/co-shared-resource.h
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+ T: git https://gitlab.com/jsnow/qemu.git jobs
- # Any format supporting backing files
+-T: git https://src.openvz.org/scm/~vsementsov/qemu.git jobs
- _supported_fmt qcow qcow2 vmdk qed
++T: git https://gitlab.com/vsementsov/qemu.git block
- _supported_proto generic
-+_unsupported_proto vxhs
+ Block QAPI, monitor, command line
- _supported_os Linux
+ M: Markus Armbruster <armbru@redhat.com>
- _unsupported_imgopts "subformat=monolithicFlat" \
+@@ -XXX,XX +XXX,XX @@ F: include/hw/cxl/
-                      "subformat=twoGbMaxExtentFlat" \
-diff --git a/tests/qemu-iotests/029 b/tests/qemu-iotests/029
+ Dirty Bitmaps
-index XXXXXXX..XXXXXXX 100755
+ M: Eric Blake <eblake@redhat.com>
---- a/tests/qemu-iotests/029
+-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
-+++ b/tests/qemu-iotests/029
++M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+ R: John Snow <jsnow@redhat.com>
- # Any format supporting intenal snapshots
+ L: qemu-block@nongnu.org
- _supported_fmt qcow2
+ S: Supported
- _supported_proto generic
+@@ -XXX,XX +XXX,XX @@ F: util/hbitmap.c
-+_unsupported_proto vxhs
+ F: tests/unit/test-hbitmap.c
- _supported_os Linux
+ F: docs/interop/bitmaps.rst
- # Internal snapshots are (currently) impossible with refcount_bits=1
+ T: git https://repo.or.cz/qemu/ericb.git bitmaps
- _unsupported_imgopts 'refcount_bits=1[^0-9]'
++T: git https://gitlab.com/vsementsov/qemu.git block
-diff --git a/tests/qemu-iotests/073 b/tests/qemu-iotests/073
-index XXXXXXX..XXXXXXX 100755
+ Character device backends
---- a/tests/qemu-iotests/073
+ M: Marc-André Lureau <marcandre.lureau@redhat.com>
-+++ b/tests/qemu-iotests/073
+@@ -XXX,XX +XXX,XX @@ F: scripts/*.py
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+ F: tests/*.py
- _supported_fmt qcow2
+ Benchmark util
- _supported_proto generic
+-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
-+_unsupported_proto vxhs
++M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
- _supported_os Linux
+ S: Maintained
+ F: scripts/simplebench/
- CLUSTER_SIZE=64k
+-T: git https://src.openvz.org/scm/~vsementsov/qemu.git simplebench
-diff --git a/tests/qemu-iotests/114 b/tests/qemu-iotests/114
++T: git https://gitlab.com/vsementsov/qemu.git simplebench
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/114
+ Transactions helper
-+++ b/tests/qemu-iotests/114
+-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
++M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
+ S: Maintained
- _supported_fmt qcow2
+ F: include/qemu/transactions.h
- _supported_proto generic
+ F: util/transactions.c
-+_unsupported_proto vxhs
++T: git https://gitlab.com/vsementsov/qemu.git block
- _supported_os Linux
+ QAPI
+ M: Markus Armbruster <armbru@redhat.com>
-diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130
+@@ -XXX,XX +XXX,XX @@ F: block/iscsi-opts.c
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/130
+ Network Block Device (NBD)
-+++ b/tests/qemu-iotests/130
+ M: Eric Blake <eblake@redhat.com>
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
++M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
- _supported_fmt qcow2
+ L: qemu-block@nongnu.org
- _supported_proto generic
+ S: Maintained
-+_unsupported_proto vxhs
+ F: block/nbd*
- _supported_os Linux
+@@ -XXX,XX +XXX,XX @@ F: docs/interop/nbd.txt
+ F: docs/tools/qemu-nbd.rst
- qemu_comm_method="monitor"
+ F: tests/qemu-iotests/tests/*nbd*
-diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
+ T: git https://repo.or.cz/qemu/ericb.git nbd
-index XXXXXXX..XXXXXXX 100755
+-T: git https://src.openvz.org/scm/~vsementsov/qemu.git nbd
---- a/tests/qemu-iotests/134
++T: git https://gitlab.com/vsementsov/qemu.git block
-+++ b/tests/qemu-iotests/134
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+ NFS
+ M: Peter Lieven <pl@kamp.de>
- _supported_fmt qcow2
+@@ -XXX,XX +XXX,XX @@ F: block/dmg.c
- _supported_proto generic
+ parallels
-+_unsupported_proto vxhs
+ M: Stefan Hajnoczi <stefanha@redhat.com>
- _supported_os Linux
+ M: Denis V. Lunev <den@openvz.org>
+-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
++M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
-diff --git a/tests/qemu-iotests/156 b/tests/qemu-iotests/156
+ L: qemu-block@nongnu.org
-index XXXXXXX..XXXXXXX 100755
+ S: Supported
---- a/tests/qemu-iotests/156
+ F: block/parallels.c
-+++ b/tests/qemu-iotests/156
+ F: block/parallels-ext.c
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+ F: docs/interop/parallels.txt
+-T: git https://src.openvz.org/scm/~vsementsov/qemu.git parallels
- _supported_fmt qcow2 qed
++T: git https://gitlab.com/vsementsov/qemu.git block
- _supported_proto generic
-+_unsupported_proto vxhs
+ qed
- _supported_os Linux
+ M: Stefan Hajnoczi <stefanha@redhat.com>
  # Create source disk
 diff --git a/tests/qemu-iotests/158 b/tests/qemu-iotests/158
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/158
 +++ b/tests/qemu-iotests/158
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto generic
 +_unsupported_proto vxhs
  _supported_os Linux
 --
-.9.3
+.36.1

-New patch
+[PULL 02/18] Use io_uring_register_ring_fd() to skip fd operations
+From: Sam Li <faithilikerun@gmail.com>
+Linux recently added a new io_uring(7) optimization API that QEMU
+doesn't take advantage of yet. The liburing library that QEMU uses
+has added a corresponding new API calling io_uring_register_ring_fd().
+When this API is called after creating the ring, the io_uring_submit()
+library function passes a flag to the io_uring_enter(2) syscall
+allowing it to skip the ring file descriptor fdget()/fdput()
+operations. This saves some CPU cycles.
+Signed-off-by: Sam Li <faithilikerun@gmail.com>
+Message-id: 20220531105011.111082-1-faithilikerun@gmail.com
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+---
+ meson.build      |  1 +
+ block/io_uring.c | 12 +++++++++++-
+files changed, 12 insertions(+), 1 deletion(-)
+diff --git a/meson.build b/meson.build
+index XXXXXXX..XXXXXXX 100644
+--- a/meson.build
++++ b/meson.build
+@@ -XXX,XX +XXX,XX @@ config_host_data.set('CONFIG_LIBNFS', libnfs.found())
+ config_host_data.set('CONFIG_LIBSSH', libssh.found())
+ config_host_data.set('CONFIG_LINUX_AIO', libaio.found())
+ config_host_data.set('CONFIG_LINUX_IO_URING', linux_io_uring.found())
++config_host_data.set('CONFIG_LIBURING_REGISTER_RING_FD', cc.has_function('io_uring_register_ring_fd', prefix: '#include <liburing.h>', dependencies:linux_io_uring))
+ config_host_data.set('CONFIG_LIBPMEM', libpmem.found())
+ config_host_data.set('CONFIG_NUMA', numa.found())
+ config_host_data.set('CONFIG_OPENGL', opengl.found())
+diff --git a/block/io_uring.c b/block/io_uring.c
+index XXXXXXX..XXXXXXX 100644
+--- a/block/io_uring.c
++++ b/block/io_uring.c
+@@ -XXX,XX +XXX,XX @@
+ #include "qapi/error.h"
+ #include "trace.h"
++
+ /* io_uring ring size */
+ #define MAX_ENTRIES 128
+@@ -XXX,XX +XXX,XX @@ LuringState *luring_init(Error **errp)
+     }
+     ioq_init(&s->io_q);
++#ifdef CONFIG_LIBURING_REGISTER_RING_FD
++    if (io_uring_register_ring_fd(&s->ring) < 0) {
++        /*
++         * Only warn about this error: we will fallback to the non-optimized
++         * io_uring operations.
++         */
++        warn_report("failed to register linux io_uring ring file descriptor");
++    }
++#endif
++
+     return s;
+-
+ }
+ void luring_cleanup(LuringState *s)
+--
+.36.1

-[Qemu-devel] [PULL 12/12] qemu-iotests: _cleanup_qemu must be called on exit
+[PULL 03/18] qdev: unplug blocker for devices
-For the tests that use the common.qemu functions for running a QEMU
+From: Jagannathan Raman <jag.raman@oracle.com>
 process, _cleanup_qemu must be called in the exit function.
-If it is not, if the qemu process aborts, then not all of the droppings
+Add blocker to prevent hot-unplug of devices
 are cleaned up (e.g. pidfile, fifos).
-This updates those tests that did not have a cleanup in qemu-iotests.
+TYPE_VFIO_USER_SERVER, which is introduced shortly, attaches itself to a
 PCIDevice on which it depends. If the attached PCIDevice gets removed
 while the server in use, it could cause it crash. To prevent this,
 TYPE_VFIO_USER_SERVER adds an unplug blocker for the PCIDevice.
-(I swapped spaces for tabs in test 102 as well)
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
 Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: c41ef80b7cc063314d629737bed2159e5713f2e0.1655151679.git.jag.raman@oracle.com
 Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
  include/hw/qdev-core.h | 29 +++++++++++++++++++++++++++++
  hw/core/qdev.c         | 24 ++++++++++++++++++++++++
  softmmu/qdev-monitor.c |  4 ++++
 files changed, 57 insertions(+)
-Reported-by: Eric Blake <eblake@redhat.com>
+diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
-Reviewed-by: Eric Blake <eblake@redhat.com>
+index XXXXXXX..XXXXXXX 100644
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+--- a/include/hw/qdev-core.h
-Message-id: d59c2f6ad6c1da8b9b3c7f357c94a7122ccfc55a.1492544096.git.jcody@redhat.com
++++ b/include/hw/qdev-core.h
----
+@@ -XXX,XX +XXX,XX @@ struct DeviceState {
- tests/qemu-iotests/028 |  1 +
+     int instance_id_alias;
- tests/qemu-iotests/094 | 11 ++++++++---
+     int alias_required_for_version;
- tests/qemu-iotests/102 |  5 +++--
+     ResettableState reset;
- tests/qemu-iotests/109 |  1 +
++    GSList *unplug_blockers;
- tests/qemu-iotests/117 |  1 +
+ };
- tests/qemu-iotests/130 |  1 +
- tests/qemu-iotests/140 |  1 +
+ struct DeviceListener {
- tests/qemu-iotests/141 |  1 +
+@@ -XXX,XX +XXX,XX @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,
- tests/qemu-iotests/143 |  1 +
+ void qdev_machine_creation_done(void);
- tests/qemu-iotests/156 |  1 +
+ bool qdev_machine_modified(void);
-files changed, 19 insertions(+), 5 deletions(-)
++/**
-diff --git a/tests/qemu-iotests/028 b/tests/qemu-iotests/028
++ * qdev_add_unplug_blocker: Add an unplug blocker to a device
-index XXXXXXX..XXXXXXX 100755
++ *
---- a/tests/qemu-iotests/028
++ * @dev: Device to be blocked from unplug
-+++ b/tests/qemu-iotests/028
++ * @reason: Reason for blocking
-@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
++ */
++void qdev_add_unplug_blocker(DeviceState *dev, Error *reason);
- _cleanup()
++
- {
++/**
-+    _cleanup_qemu
++ * qdev_del_unplug_blocker: Remove an unplug blocker from a device
-     rm -f "${TEST_IMG}.copy"
++ *
-     _cleanup_test_img
++ * @dev: Device to be unblocked
 + * @reason: Pointer to the Error used with qdev_add_unplug_blocker.
 + *          Used as a handle to lookup the blocker for deletion.
 + */
 +void qdev_del_unplug_blocker(DeviceState *dev, Error *reason);
 +
 +/**
 + * qdev_unplug_blocked: Confirm if a device is blocked from unplug
 + *
 + * @dev: Device to be tested
 + * @reason: Returns one of the reasons why the device is blocked,
 + *          if any
 + *
 + * Returns: true if device is blocked from unplug, false otherwise
 + */
 +bool qdev_unplug_blocked(DeviceState *dev, Error **errp);
 +
  /**
   * GpioPolarity: Polarity of a GPIO line
   *
 diff --git a/hw/core/qdev.c b/hw/core/qdev.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/core/qdev.c
 +++ b/hw/core/qdev.c
@@ -XXX,XX +XXX,XX @@ char *qdev_get_dev_path(DeviceState *dev)
      return NULL;
  }
-diff --git a/tests/qemu-iotests/094 b/tests/qemu-iotests/094
-index XXXXXXX..XXXXXXX 100755
++void qdev_add_unplug_blocker(DeviceState *dev, Error *reason)
 --- a/tests/qemu-iotests/094
 +++ b/tests/qemu-iotests/094
@@ -XXX,XX +XXX,XX @@ echo "QA output created by $seq"
  here="$PWD"
  status=1    # failure is the default!
 -trap "exit \$status" 0 1 2 3 15
 +_cleanup()
 +{
-+    _cleanup_qemu
++    dev->unplug_blockers = g_slist_prepend(dev->unplug_blockers, reason);
 +    _cleanup_test_img
 +    rm -f "$TEST_DIR/source.$IMGFMT"
 +}
 +
-+trap "_cleanup; exit \$status" 0 1 2 3 15
++void qdev_del_unplug_blocker(DeviceState *dev, Error *reason)
++{
- # get standard environment, filters and checks
++    dev->unplug_blockers = g_slist_remove(dev->unplug_blockers, reason);
- . ./common.rc
++}
-@@ -XXX,XX +XXX,XX @@ _send_qemu_cmd $QEMU_HANDLE \
++
++bool qdev_unplug_blocked(DeviceState *dev, Error **errp)
- wait=1 _cleanup_qemu
++{
++    ERRP_GUARD();
--_cleanup_test_img
++
--rm -f "$TEST_DIR/source.$IMGFMT"
++    if (dev->unplug_blockers) {
++        error_propagate(errp, error_copy(dev->unplug_blockers->data));
- # success, all done
++        return true;
- echo '*** done'
++    }
-diff --git a/tests/qemu-iotests/102 b/tests/qemu-iotests/102
++
-index XXXXXXX..XXXXXXX 100755
++    return false;
---- a/tests/qemu-iotests/102
++}
-+++ b/tests/qemu-iotests/102
++
-@@ -XXX,XX +XXX,XX @@ seq=$(basename $0)
+ static bool device_get_realized(Object *obj, Error **errp)
  echo "QA output created by $seq"
  here=$PWD
 -status=1    # failure is the default!
 +status=1    # failure is the default!
  _cleanup()
  {
--    _cleanup_test_img
+     DeviceState *dev = DEVICE(obj);
-+    _cleanup_qemu
+@@ -XXX,XX +XXX,XX @@ static void device_finalize(Object *obj)
-+    _cleanup_test_img
- }
+     DeviceState *dev = DEVICE(obj);
- trap "_cleanup; exit \$status" 0 1 2 3 15
++    g_assert(!dev->unplug_blockers);
-diff --git a/tests/qemu-iotests/109 b/tests/qemu-iotests/109
++
-index XXXXXXX..XXXXXXX 100755
+     QLIST_FOREACH_SAFE(ngl, &dev->gpios, node, next) {
---- a/tests/qemu-iotests/109
+         QLIST_REMOVE(ngl, node);
-+++ b/tests/qemu-iotests/109
+         qemu_free_irqs(ngl->in, ngl->num_in);
-@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
+diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c
+index XXXXXXX..XXXXXXX 100644
- _cleanup()
+--- a/softmmu/qdev-monitor.c
- {
++++ b/softmmu/qdev-monitor.c
-+    _cleanup_qemu
+@@ -XXX,XX +XXX,XX @@ void qdev_unplug(DeviceState *dev, Error **errp)
-     rm -f $TEST_IMG.src
+     HotplugHandlerClass *hdc;
-     _cleanup_test_img
+     Error *local_err = NULL;
- }
-diff --git a/tests/qemu-iotests/117 b/tests/qemu-iotests/117
++    if (qdev_unplug_blocked(dev, errp)) {
-index XXXXXXX..XXXXXXX 100755
++        return;
---- a/tests/qemu-iotests/117
++    }
-+++ b/tests/qemu-iotests/117
++
-@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
+     if (dev->parent_bus && !qbus_is_hotpluggable(dev->parent_bus)) {
+         error_setg(errp, QERR_BUS_NO_HOTPLUG, dev->parent_bus->name);
- _cleanup()
+         return;
  {
 +    _cleanup_qemu
      _cleanup_test_img
  }
  trap "_cleanup; exit \$status" 0 1 2 3 15
 diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/130
 +++ b/tests/qemu-iotests/130
@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
  _cleanup()
  {
 +    _cleanup_qemu
      _cleanup_test_img
  }
  trap "_cleanup; exit \$status" 0 1 2 3 15
 diff --git a/tests/qemu-iotests/140 b/tests/qemu-iotests/140
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/140
 +++ b/tests/qemu-iotests/140
@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
  _cleanup()
  {
 +    _cleanup_qemu
      _cleanup_test_img
      rm -f "$TEST_DIR/nbd"
  }
 diff --git a/tests/qemu-iotests/141 b/tests/qemu-iotests/141
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/141
 +++ b/tests/qemu-iotests/141
@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
  _cleanup()
  {
 +    _cleanup_qemu
      _cleanup_test_img
      rm -f "$TEST_DIR/{b,m,o}.$IMGFMT"
  }
 diff --git a/tests/qemu-iotests/143 b/tests/qemu-iotests/143
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/143
 +++ b/tests/qemu-iotests/143
@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
  _cleanup()
  {
 +    _cleanup_qemu
      rm -f "$TEST_DIR/nbd"
  }
  trap "_cleanup; exit \$status" 0 1 2 3 15
 diff --git a/tests/qemu-iotests/156 b/tests/qemu-iotests/156
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/156
 +++ b/tests/qemu-iotests/156
@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
  _cleanup()
  {
 +    _cleanup_qemu
      rm -f "$TEST_IMG{,.target}{,.backing,.overlay}"
  }
  trap "_cleanup; exit \$status" 0 1 2 3 15
 --
-.9.3
+.36.1

-New patch
+[PULL 04/18] remote/machine: add HotplugHandler for remote machine
+From: Jagannathan Raman <jag.raman@oracle.com>
+Allow hotplugging of PCI(e) devices to remote machine
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
+Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
+Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
+Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
+Message-id: d1e6cfa0afb528ad343758f9b1d918be0175c5e5.1655151679.git.jag.raman@oracle.com
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+---
+ hw/remote/machine.c | 10 ++++++++++
+file changed, 10 insertions(+)
+diff --git a/hw/remote/machine.c b/hw/remote/machine.c
+index XXXXXXX..XXXXXXX 100644
+--- a/hw/remote/machine.c
++++ b/hw/remote/machine.c
+@@ -XXX,XX +XXX,XX @@
+ #include "qapi/error.h"
+ #include "hw/pci/pci_host.h"
+ #include "hw/remote/iohub.h"
++#include "hw/qdev-core.h"
+ static void remote_machine_init(MachineState *machine)
+ {
+@@ -XXX,XX +XXX,XX @@ static void remote_machine_init(MachineState *machine)
+     pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                  &s->iohub, REMOTE_IOHUB_NB_PIRQS);
++
++    qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
+ }
+ static void remote_machine_class_init(ObjectClass *oc, void *data)
+ {
+     MachineClass *mc = MACHINE_CLASS(oc);
++    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
+     mc->init = remote_machine_init;
+     mc->desc = "Experimental remote machine";
++
++    hc->unplug = qdev_simple_device_unplug_cb;
+ }
+ static const TypeInfo remote_machine = {
+@@ -XXX,XX +XXX,XX @@ static const TypeInfo remote_machine = {
+     .parent = TYPE_MACHINE,
+     .instance_size = sizeof(RemoteMachineState),
+     .class_init = remote_machine_class_init,
++    .interfaces = (InterfaceInfo[]) {
++        { TYPE_HOTPLUG_HANDLER },
++        { }
++    }
+ };
+ static void remote_machine_register_types(void)
+--
+.36.1

-[Qemu-devel] [PULL 07/12] block: code movement
+[PULL 05/18] remote/machine: add vfio-user property
-Move bdrv_is_read_only() up with its friends.
+From: Jagannathan Raman <jag.raman@oracle.com>
+Add vfio-user to x-remote machine. It is a boolean, which indicates if
+the machine supports vfio-user protocol. The machine configures the bus
+differently vfio-user and multiprocess protocols, so this property
+informs it on how to configure the bus.
+This property should be short lived. Once vfio-user fully replaces
+multiprocess, this property could be removed.
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
+Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
+Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Reviewed-by: John Snow <jsnow@redhat.com>
+Message-id: 5d51a152a419cbda35d070b8e49b772b60a7230a.1655151679.git.jag.raman@oracle.com
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: 73b2399459760c32506f9407efb9dddb3a2789de.1491597120.git.jcody@redhat.com
 ---
- block.c | 10 +++++-----
+ include/hw/remote/machine.h |  2 ++
-file changed, 5 insertions(+), 5 deletions(-)
+ hw/remote/machine.c         | 23 +++++++++++++++++++++++
 files changed, 25 insertions(+)
-diff --git a/block.c b/block.c
+diff --git a/include/hw/remote/machine.h b/include/hw/remote/machine.h
 index XXXXXXX..XXXXXXX 100644
---- a/block.c
+--- a/include/hw/remote/machine.h
-+++ b/block.c
++++ b/include/hw/remote/machine.h
-@@ -XXX,XX +XXX,XX @@ void path_combine(char *dest, int dest_size,
+@@ -XXX,XX +XXX,XX @@ struct RemoteMachineState {
-     }
      RemotePCIHost *host;
      RemoteIOHubState iohub;
 +
 +    bool vfio_user;
  };
  /* Used to pass to co-routine device and ioc. */
 diff --git a/hw/remote/machine.c b/hw/remote/machine.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/machine.c
 +++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@ static void remote_machine_init(MachineState *machine)
      qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
  }
-+bool bdrv_is_read_only(BlockDriverState *bs)
++static bool remote_machine_get_vfio_user(Object *obj, Error **errp)
 +{
-+    return bs->read_only;
++    RemoteMachineState *s = REMOTE_MACHINE(obj);
 +
 +    return s->vfio_user;
 +}
 +
- int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
++static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp)
 +{
 +    RemoteMachineState *s = REMOTE_MACHINE(obj);
 +
 +    if (phase_check(PHASE_MACHINE_CREATED)) {
 +        error_setg(errp, "Error enabling vfio-user - machine already created");
 +        return;
 +    }
 +
 +    s->vfio_user = value;
 +}
 +
  static void remote_machine_class_init(ObjectClass *oc, void *data)
  {
-     /* Do not set read_only if copy_on_read is enabled */
+     MachineClass *mc = MACHINE_CLASS(oc);
-@@ -XXX,XX +XXX,XX @@ void bdrv_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr)
+@@ -XXX,XX +XXX,XX @@ static void remote_machine_class_init(ObjectClass *oc, void *data)
-     *nb_sectors_ptr = nb_sectors < 0 ? 0 : nb_sectors;
+     mc->desc = "Experimental remote machine";
      hc->unplug = qdev_simple_device_unplug_cb;
 +
 +    object_class_property_add_bool(oc, "vfio-user",
 +                                   remote_machine_get_vfio_user,
 +                                   remote_machine_set_vfio_user);
  }
--bool bdrv_is_read_only(BlockDriverState *bs)
+ static const TypeInfo remote_machine = {
 -{
 -    return bs->read_only;
 -}
 -
  bool bdrv_is_sg(BlockDriverState *bs)
  {
      return bs->sg;
 --
-.9.3
+.36.1

-New patch
+[PULL 06/18] vfio-user: build library
+From: Jagannathan Raman <jag.raman@oracle.com>
 add the libvfio-user library as a submodule. build it as a meson
 subproject.
 libvfio-user is distributed with BSD 3-Clause license and
 json-c with MIT (Expat) license
 Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
 Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: c2adec87958b081d1dc8775d4aa05c897912f025.1655151679.git.jag.raman@oracle.com
 [Changed submodule URL to QEMU's libvfio-user mirror on GitLab. The QEMU
 project mirrors its dependencies so that it can provide full source code
 even in the event that its dependencies become unavailable. Note that
 the mirror repo is manually updated, so please contact me to make newer
 libvfio-user commits available. If I become a bottleneck we can set up a
 cronjob.
 Updated scripts/meson-buildoptions.sh to match the meson_options.txt
 change. Failure to do so can result in scripts/meson-buildoptions.sh
 being modified by the build system later on and you end up with a dirty
 working tree.
 --Stefan]
 Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
  MAINTAINERS                             |  1 +
  meson_options.txt                       |  2 ++
  configure                               | 17 +++++++++++++++++
  meson.build                             | 23 ++++++++++++++++++++++-
  .gitlab-ci.d/buildtest.yml              |  1 +
  .gitmodules                             |  3 +++
  Kconfig.host                            |  4 ++++
  hw/remote/Kconfig                       |  4 ++++
  hw/remote/meson.build                   |  2 ++
  scripts/meson-buildoptions.sh           |  4 ++++
  subprojects/libvfio-user                |  1 +
  tests/docker/dockerfiles/centos8.docker |  2 ++
 files changed, 63 insertions(+), 1 deletion(-)
  create mode 160000 subprojects/libvfio-user
 diff --git a/MAINTAINERS b/MAINTAINERS
 index XXXXXXX..XXXXXXX 100644
 --- a/MAINTAINERS
 +++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: hw/remote/proxy-memory-listener.c
  F: include/hw/remote/proxy-memory-listener.h
  F: hw/remote/iohub.c
  F: include/hw/remote/iohub.h
 +F: subprojects/libvfio-user
  EBPF:
  M: Jason Wang <jasowang@redhat.com>
 diff --git a/meson_options.txt b/meson_options.txt
 index XXXXXXX..XXXXXXX 100644
 --- a/meson_options.txt
 +++ b/meson_options.txt
@@ -XXX,XX +XXX,XX @@ option('cfi_debug', type: 'boolean', value: 'false',
         description: 'Verbose errors in case of CFI violation')
  option('multiprocess', type: 'feature', value: 'auto',
         description: 'Out of process device emulation support')
 +option('vfio_user_server', type: 'feature', value: 'disabled',
 +       description: 'vfio-user server support')
  option('dbus_display', type: 'feature', value: 'auto',
         description: '-display dbus support')
  option('tpm', type : 'feature', value : 'auto',
 diff --git a/configure b/configure
 index XXXXXXX..XXXXXXX 100755
 --- a/configure
 +++ b/configure
@@ -XXX,XX +XXX,XX @@ meson_args=""
  ninja=""
  bindir="bin"
  skip_meson=no
 +vfio_user_server="disabled"
  # The following Meson options are handled manually (still they
  # are included in the automatically generated help message)
@@ -XXX,XX +XXX,XX @@ for opt do
    ;;
    --disable-blobs) meson_option_parse --disable-install-blobs ""
    ;;
 +  --enable-vfio-user-server) vfio_user_server="enabled"
 +  ;;
 +  --disable-vfio-user-server) vfio_user_server="disabled"
 +  ;;
    --enable-tcmalloc) meson_option_parse --enable-malloc=tcmalloc tcmalloc
    ;;
    --enable-jemalloc) meson_option_parse --enable-malloc=jemalloc jemalloc
@@ -XXX,XX +XXX,XX @@ write_container_target_makefile() {
 +##########################################
 +# check for vfio_user_server
 +
 +case "$vfio_user_server" in
 +  enabled )
 +    if test "$git_submodules_action" != "ignore"; then
 +      git_submodules="${git_submodules} subprojects/libvfio-user"
 +    fi
 +    ;;
 +esac
 +
  ##########################################
  # End of CC checks
  # After here, no more $cc or $ld runs
@@ -XXX,XX +XXX,XX @@ if test "$skip_meson" = no; then
    test "$slirp" != auto && meson_option_add "-Dslirp=$slirp"
    test "$smbd" != '' && meson_option_add "-Dsmbd=$smbd"
    test "$tcg" != enabled && meson_option_add "-Dtcg=$tcg"
 +  test "$vfio_user_server" != auto && meson_option_add "-Dvfio_user_server=$vfio_user_server"
    run_meson() {
      NINJA=$ninja $meson setup --prefix "$prefix" "$@" $cross_arg "$PWD" "$source_path"
    }
 diff --git a/meson.build b/meson.build
 index XXXXXXX..XXXXXXX 100644
 --- a/meson.build
 +++ b/meson.build
@@ -XXX,XX +XXX,XX @@ multiprocess_allowed = get_option('multiprocess') \
    .require(targetos == 'linux', error_message: 'Multiprocess QEMU is supported only on Linux') \
    .allowed()
 +vfio_user_server_allowed = get_option('vfio_user_server') \
 +  .require(targetos == 'linux', error_message: 'vfio-user server is supported only on Linux') \
 +  .allowed()
 +
  have_tpm = get_option('tpm') \
    .require(targetos != 'windows', error_message: 'TPM emulation only available on POSIX systems') \
    .allowed()
@@ -XXX,XX +XXX,XX @@ host_kconfig = \
    (have_virtfs ? ['CONFIG_VIRTFS=y'] : []) + \
    ('CONFIG_LINUX' in config_host ? ['CONFIG_LINUX=y'] : []) + \
    (have_pvrdma ? ['CONFIG_PVRDMA=y'] : []) + \
 -  (multiprocess_allowed ? ['CONFIG_MULTIPROCESS_ALLOWED=y'] : [])
 +  (multiprocess_allowed ? ['CONFIG_MULTIPROCESS_ALLOWED=y'] : []) + \
 +  (vfio_user_server_allowed ? ['CONFIG_VFIO_USER_SERVER_ALLOWED=y'] : [])
  ignored = [ 'TARGET_XML_FILES', 'TARGET_ABI_DIR', 'TARGET_ARCH' ]
@@ -XXX,XX +XXX,XX @@ if have_system
    endif
  endif
 +libvfio_user_dep = not_found
 +if have_system and vfio_user_server_allowed
 +  have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/meson.build')
 +
 +  if not have_internal
 +    error('libvfio-user source not found - please pull git submodule')
 +  endif
 +
 +  libvfio_user_proj = subproject('libvfio-user')
 +
 +  libvfio_user_lib = libvfio_user_proj.get_variable('libvfio_user_dep')
 +
 +  libvfio_user_dep = declare_dependency(dependencies: [libvfio_user_lib])
 +endif
 +
  fdt = not_found
  if have_system
    fdt_opt = get_option('fdt')
@@ -XXX,XX +XXX,XX @@ summary_info += {'target list':       ' '.join(target_dirs)}
  if have_system
    summary_info += {'default devices':   get_option('default_devices')}
    summary_info += {'out of process emulation': multiprocess_allowed}
 +  summary_info += {'vfio-user server': vfio_user_server_allowed}
  endif
  summary(summary_info, bool_yn: true, section: 'Targets and accelerators')
 diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
 index XXXXXXX..XXXXXXX 100644
 --- a/.gitlab-ci.d/buildtest.yml
 +++ b/.gitlab-ci.d/buildtest.yml
@@ -XXX,XX +XXX,XX @@ build-system-centos:
      IMAGE: centos8
      CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
        --enable-modules --enable-trace-backends=dtrace --enable-docs
 +      --enable-vfio-user-server
      TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
        x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
      MAKE_CHECK_ARGS: check-build
 diff --git a/.gitmodules b/.gitmodules
 index XXXXXXX..XXXXXXX 100644
 --- a/.gitmodules
 +++ b/.gitmodules
@@ -XXX,XX +XXX,XX @@
  [submodule "tests/lcitool/libvirt-ci"]
      path = tests/lcitool/libvirt-ci
      url = https://gitlab.com/libvirt/libvirt-ci.git
 +[submodule "subprojects/libvfio-user"]
 +    path = subprojects/libvfio-user
 +    url = https://gitlab.com/qemu-project/libvfio-user.git
 diff --git a/Kconfig.host b/Kconfig.host
 index XXXXXXX..XXXXXXX 100644
 --- a/Kconfig.host
 +++ b/Kconfig.host
@@ -XXX,XX +XXX,XX @@ config MULTIPROCESS_ALLOWED
  config FUZZ
      bool
      select SPARSE_MEM
 +
 +config VFIO_USER_SERVER_ALLOWED
 +    bool
 +    imply VFIO_USER_SERVER
 diff --git a/hw/remote/Kconfig b/hw/remote/Kconfig
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/Kconfig
 +++ b/hw/remote/Kconfig
@@ -XXX,XX +XXX,XX @@ config MULTIPROCESS
      bool
      depends on PCI && PCI_EXPRESS && KVM
      select REMOTE_PCIHOST
 +
 +config VFIO_USER_SERVER
 +    bool
 +    depends on MULTIPROCESS
 diff --git a/hw/remote/meson.build b/hw/remote/meson.build
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/meson.build
 +++ b/hw/remote/meson.build
@@ -XXX,XX +XXX,XX @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
  remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
  remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
 +remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: libvfio_user_dep)
 +
  specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c'))
  specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
 diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
 index XXXXXXX..XXXXXXX 100644
 --- a/scripts/meson-buildoptions.sh
 +++ b/scripts/meson-buildoptions.sh
@@ -XXX,XX +XXX,XX @@ meson_options_help() {
    printf "%s\n" '  usb-redir       libusbredir support'
    printf "%s\n" '  vde             vde network backend support'
    printf "%s\n" '  vdi             vdi image format support'
 +  printf "%s\n" '  vfio-user-server'
 +  printf "%s\n" '                  vfio-user server support'
    printf "%s\n" '  vhost-crypto    vhost-user crypto backend support'
    printf "%s\n" '  vhost-kernel    vhost kernel backend support'
    printf "%s\n" '  vhost-net       vhost-net kernel acceleration support'
@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
      --disable-vde) printf "%s" -Dvde=disabled ;;
      --enable-vdi) printf "%s" -Dvdi=enabled ;;
      --disable-vdi) printf "%s" -Dvdi=disabled ;;
 +    --enable-vfio-user-server) printf "%s" -Dvfio_user_server=enabled ;;
 +    --disable-vfio-user-server) printf "%s" -Dvfio_user_server=disabled ;;
      --enable-vhost-crypto) printf "%s" -Dvhost_crypto=enabled ;;
      --disable-vhost-crypto) printf "%s" -Dvhost_crypto=disabled ;;
      --enable-vhost-kernel) printf "%s" -Dvhost_kernel=enabled ;;
 diff --git a/subprojects/libvfio-user b/subprojects/libvfio-user
 new file mode 160000
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/subprojects/libvfio-user
@@ -0,0 +1 @@
 +Subproject commit 0b28d205572c80b568a1003db2c8f37ca333e4d7
 diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/centos8.docker
 +++ b/tests/docker/dockerfiles/centos8.docker
@@ -XXX,XX +XXX,XX @@ RUN dnf update -y && \
          libbpf-devel \
          libcacard-devel \
          libcap-ng-devel \
 +        libcmocka-devel \
          libcurl-devel \
          libdrm-devel \
          libepoxy-devel \
@@ -XXX,XX +XXX,XX @@ RUN dnf update -y && \
          libgcrypt-devel \
          libiscsi-devel \
          libjpeg-devel \
 +        json-c-devel \
          libnfs-devel \
          libpmem-devel \
          libpng-devel \
 --
 .36.1

-[Qemu-devel] [PULL 01/12] block/vxhs.c: Add support for a new block device type called "vxhs"
+[PULL 07/18] vfio-user: define vfio-user-server object
-From: Ashish Mittal <ashmit602@gmail.com>
+From: Jagannathan Raman <jag.raman@oracle.com>
-Source code for the qnio library that this code loads can be downloaded from:
+Define vfio-user object which is remote process server for QEMU. Setup
-https://github.com/VeritasHyperScale/libqnio.git
+object initialization functions and properties necessary to instantiate
+the object
-Sample command line using JSON syntax:
-./x86_64-softmmu/qemu-system-x86_64 -name instance-00000008 -S -vnc 0.0.0.0:0
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
--k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
+Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
--msg timestamp=on
+Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 'json:{"driver":"vxhs","vdisk-id":"c3e9095a-a5ee-4dce-afeb-2a59fb387410",
 "server":{"host":"172.172.17.4","port":"9999"}}'
 Sample command line using URI syntax:
 qemu-img convert -f raw -O raw -n
 /var/lib/nova/instances/_base/0c5eacd5ebea5ed914b6a3e7b18f1ce734c386ad
 vxhs://192.168.0.1:9999/c6718f6b-0401-441d-a8c3-1f0064d75ee0
 Sample command line using TLS credentials (run in secure mode):
 ./qemu-io --object
 tls-creds-x509,id=tls0,dir=/etc/pki/qemu/vxhs,endpoint=client -c 'read
 -v 66000 2.5k' 'json:{"server.host": "127.0.0.1", "server.port": "9999",
 "vdisk-id": "/test.raw", "driver": "vxhs", "tls-creds":"tls0"}'
 Signed-off-by: Ashish Mittal <Ashish.Mittal@veritas.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Reviewed-by: Jeff Cody <jcody@redhat.com>
+Message-id: e45a17001e9b38f451543a664ababdf860e5f2f2.1655151679.git.jag.raman@oracle.com
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: 1491277689-24949-2-git-send-email-Ashish.Mittal@veritas.com
 ---
- block/Makefile.objs  |   2 +
+ MAINTAINERS                 |   1 +
- block/trace-events   |  17 ++
+ qapi/qom.json               |  20 +++-
- block/vxhs.c         | 575 +++++++++++++++++++++++++++++++++++++++++++++++++++
+ include/hw/remote/machine.h |   2 +
- configure            |  39 ++++
+ hw/remote/machine.c         |  27 +++++
- qapi/block-core.json |  23 ++-
+ hw/remote/vfio-user-obj.c   | 210 ++++++++++++++++++++++++++++++++++++
-files changed, 654 insertions(+), 2 deletions(-)
+ hw/remote/meson.build       |   1 +
- create mode 100644 block/vxhs.c
+ hw/remote/trace-events      |   3 +
+files changed, 262 insertions(+), 2 deletions(-)
-diff --git a/block/Makefile.objs b/block/Makefile.objs
+ create mode 100644 hw/remote/vfio-user-obj.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/Makefile.objs
+diff --git a/MAINTAINERS b/MAINTAINERS
-+++ b/block/Makefile.objs
+index XXXXXXX..XXXXXXX 100644
-@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_LIBNFS) += nfs.o
+--- a/MAINTAINERS
- block-obj-$(CONFIG_CURL) += curl.o
++++ b/MAINTAINERS
- block-obj-$(CONFIG_RBD) += rbd.o
+@@ -XXX,XX +XXX,XX @@ F: include/hw/remote/proxy-memory-listener.h
- block-obj-$(CONFIG_GLUSTERFS) += gluster.o
+ F: hw/remote/iohub.c
-+block-obj-$(CONFIG_VXHS) += vxhs.o
+ F: include/hw/remote/iohub.h
- block-obj-$(CONFIG_LIBSSH2) += ssh.o
+ F: subprojects/libvfio-user
- block-obj-y += accounting.o dirty-bitmap.o
++F: hw/remote/vfio-user-obj.c
- block-obj-y += write-threshold.o
-@@ -XXX,XX +XXX,XX @@ rbd.o-cflags       := $(RBD_CFLAGS)
+ EBPF:
- rbd.o-libs         := $(RBD_LIBS)
+ M: Jason Wang <jasowang@redhat.com>
- gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
+diff --git a/qapi/qom.json b/qapi/qom.json
- gluster.o-libs     := $(GLUSTERFS_LIBS)
+index XXXXXXX..XXXXXXX 100644
-+vxhs.o-libs        := $(VXHS_LIBS)
+--- a/qapi/qom.json
- ssh.o-cflags       := $(LIBSSH2_CFLAGS)
++++ b/qapi/qom.json
- ssh.o-libs         := $(LIBSSH2_LIBS)
+@@ -XXX,XX +XXX,XX @@
- block-obj-$(if $(CONFIG_BZIP2),m,n) += dmg-bz2.o
+ { 'struct': 'RemoteObjectProperties',
-diff --git a/block/trace-events b/block/trace-events
+   'data': { 'fd': 'str', 'devid': 'str' } }
-index XXXXXXX..XXXXXXX 100644
---- a/block/trace-events
++##
-+++ b/block/trace-events
++# @VfioUserServerProperties:
-@@ -XXX,XX +XXX,XX @@ qed_aio_write_data(void *s, void *acb, int ret, uint64_t offset, size_t len) "s
++#
- qed_aio_write_prefill(void *s, void *acb, uint64_t start, size_t len, uint64_t offset) "s %p acb %p start %"PRIu64" len %zu offset %"PRIu64
++# Properties for x-vfio-user-server objects.
- qed_aio_write_postfill(void *s, void *acb, uint64_t start, size_t len, uint64_t offset) "s %p acb %p start %"PRIu64" len %zu offset %"PRIu64
++#
- qed_aio_write_main(void *s, void *acb, int ret, uint64_t offset, size_t len) "s %p acb %p ret %d offset %"PRIu64" len %zu"
++# @socket: socket to be used by the libvfio-user library
-+
++#
-+# block/vxhs.c
++# @device: the ID of the device to be emulated at the server
-+vxhs_iio_callback(int error) "ctx is NULL: error %d"
++#
-+vxhs_iio_callback_chnfail(int err, int error) "QNIO channel failed, no i/o %d, %d"
++# Since: 7.1
-+vxhs_iio_callback_unknwn(int opcode, int err) "unexpected opcode %d, errno %d"
++##
-+vxhs_aio_rw_invalid(int req) "Invalid I/O request iodir %d"
++{ 'struct': 'VfioUserServerProperties',
-+vxhs_aio_rw_ioerr(char *guid, int iodir, uint64_t size, uint64_t off, void *acb, int ret, int err) "IO ERROR (vDisk %s) FOR : Read/Write = %d size = %lu offset = %lu ACB = %p. Error = %d, errno = %d"
++  'data': { 'socket': 'SocketAddress', 'device': 'str' } }
-+vxhs_get_vdisk_stat_err(char *guid, int ret, int err) "vDisk (%s) stat ioctl failed, ret = %d, errno = %d"
++
-+vxhs_get_vdisk_stat(char *vdisk_guid, uint64_t vdisk_size) "vDisk %s stat ioctl returned size %lu"
+ ##
-+vxhs_complete_aio(void *acb, uint64_t ret) "aio failed acb %p ret %ld"
+ # @RngProperties:
-+vxhs_parse_uri_filename(const char *filename) "URI passed via bdrv_parse_filename %s"
+ #
-+vxhs_open_vdiskid(const char *vdisk_id) "Opening vdisk-id %s"
+@@ -XXX,XX +XXX,XX @@
-+vxhs_open_hostinfo(char *of_vsa_addr, int port) "Adding host %s:%d to BDRVVXHSState"
+     'tls-creds-psk',
-+vxhs_open_iio_open(const char *host) "Failed to connect to storage agent on host %s"
+     'tls-creds-x509',
-+vxhs_parse_uri_hostinfo(char *host, int port) "Host: IP %s, Port %d"
+     'tls-cipher-suites',
-+vxhs_close(char *vdisk_guid) "Closing vdisk %s"
+-    { 'name': 'x-remote-object', 'features': [ 'unstable' ] }
-+vxhs_get_creds(const char *cacert, const char *client_key, const char *client_cert) "cacert %s, client_key %s, client_cert %s"
++    { 'name': 'x-remote-object', 'features': [ 'unstable' ] },
-diff --git a/block/vxhs.c b/block/vxhs.c
++    { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] }
    ] }
  ##
@@ -XXX,XX +XXX,XX @@
        'tls-creds-psk':              'TlsCredsPskProperties',
        'tls-creds-x509':             'TlsCredsX509Properties',
        'tls-cipher-suites':          'TlsCredsProperties',
 -      'x-remote-object':            'RemoteObjectProperties'
 +      'x-remote-object':            'RemoteObjectProperties',
 +      'x-vfio-user-server':         'VfioUserServerProperties'
    } }
  ##
 diff --git a/include/hw/remote/machine.h b/include/hw/remote/machine.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/hw/remote/machine.h
 +++ b/include/hw/remote/machine.h
@@ -XXX,XX +XXX,XX @@ struct RemoteMachineState {
      RemoteIOHubState iohub;
      bool vfio_user;
 +
 +    bool auto_shutdown;
  };
  /* Used to pass to co-routine device and ioc. */
 diff --git a/hw/remote/machine.c b/hw/remote/machine.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/machine.c
 +++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@ static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp)
      s->vfio_user = value;
  }
 +static bool remote_machine_get_auto_shutdown(Object *obj, Error **errp)
 +{
 +    RemoteMachineState *s = REMOTE_MACHINE(obj);
 +
 +    return s->auto_shutdown;
 +}
 +
 +static void remote_machine_set_auto_shutdown(Object *obj, bool value,
 +                                             Error **errp)
 +{
 +    RemoteMachineState *s = REMOTE_MACHINE(obj);
 +
 +    s->auto_shutdown = value;
 +}
 +
 +static void remote_machine_instance_init(Object *obj)
 +{
 +    RemoteMachineState *s = REMOTE_MACHINE(obj);
 +
 +    s->auto_shutdown = true;
 +}
 +
  static void remote_machine_class_init(ObjectClass *oc, void *data)
  {
      MachineClass *mc = MACHINE_CLASS(oc);
@@ -XXX,XX +XXX,XX @@ static void remote_machine_class_init(ObjectClass *oc, void *data)
      object_class_property_add_bool(oc, "vfio-user",
                                     remote_machine_get_vfio_user,
                                     remote_machine_set_vfio_user);
 +
 +    object_class_property_add_bool(oc, "auto-shutdown",
 +                                   remote_machine_get_auto_shutdown,
 +                                   remote_machine_set_auto_shutdown);
  }
  static const TypeInfo remote_machine = {
      .name = TYPE_REMOTE_MACHINE,
      .parent = TYPE_MACHINE,
      .instance_size = sizeof(RemoteMachineState),
 +    .instance_init = remote_machine_instance_init,
      .class_init = remote_machine_class_init,
      .interfaces = (InterfaceInfo[]) {
          { TYPE_HOTPLUG_HANDLER },
 diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
-+++ b/block/vxhs.c
++++ b/hw/remote/vfio-user-obj.c
 @@ -XXX,XX +XXX,XX @@
-+/*
++/**
-+ * QEMU Block driver for Veritas HyperScale (VxHS)
++ * QEMU vfio-user-server server object
 + *
-+ * Copyright (c) 2017 Veritas Technologies LLC.
++ * Copyright © 2022 Oracle and/or its affiliates.
 + *
-+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
++ * This work is licensed under the terms of the GNU GPL-v2, version 2 or later.
 + *
 + * See the COPYING file in the top-level directory.
 + *
 + */
 +
++/**
++ * Usage: add options:
++ *     -machine x-remote,vfio-user=on,auto-shutdown=on
++ *     -device <PCI-device>,id=<pci-dev-id>
++ *     -object x-vfio-user-server,id=<id>,type=unix,path=<socket-path>,
++ *             device=<pci-dev-id>
++ *
++ * Note that x-vfio-user-server object must be used with x-remote machine only.
++ * This server could only support PCI devices for now.
++ *
++ * type - SocketAddress type - presently "unix" alone is supported. Required
++ *        option
++ *
++ * path - named unix socket, it will be created by the server. It is
++ *        a required option
++ *
++ * device - id of a device on the server, a required option. PCI devices
++ *          alone are supported presently.
++ */
++
 +#include "qemu/osdep.h"
-+#include <qnio/qnio_api.h>
++
-+#include <sys/param.h>
++#include "qom/object.h"
-+#include "block/block_int.h"
++#include "qom/object_interfaces.h"
-+#include "qapi/qmp/qerror.h"
++#include "qemu/error-report.h"
 +#include "qapi/qmp/qdict.h"
 +#include "qapi/qmp/qstring.h"
 +#include "trace.h"
-+#include "qemu/uri.h"
++#include "sysemu/runstate.h"
 +#include "hw/boards.h"
 +#include "hw/remote/machine.h"
 +#include "qapi/error.h"
-+#include "qemu/uuid.h"
++#include "qapi/qapi-visit-sockets.h"
-+#include "crypto/tlscredsx509.h"
++
-+
++#define TYPE_VFU_OBJECT "x-vfio-user-server"
-+#define VXHS_OPT_FILENAME           "filename"
++OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
-+#define VXHS_OPT_VDISK_ID           "vdisk-id"
++
-+#define VXHS_OPT_SERVER             "server"
++/**
-+#define VXHS_OPT_HOST               "host"
++ * VFU_OBJECT_ERROR - reports an error message. If auto_shutdown
-+#define VXHS_OPT_PORT               "port"
++ * is set, it aborts the machine on error. Otherwise, it logs an
-+
++ * error message without aborting.
 +/* Only accessed under QEMU global mutex */
 +static uint32_t vxhs_ref;
 +
 +typedef enum {
 +    VDISK_AIO_READ,
 +    VDISK_AIO_WRITE,
 +} VDISKAIOCmd;
 +
 +/*
 + * HyperScale AIO callbacks structure
 + */
-+typedef struct VXHSAIOCB {
++#define VFU_OBJECT_ERROR(o, fmt, ...)                                     \
-+    BlockAIOCB common;
++    {                                                                     \
-+    int err;
++        if (vfu_object_auto_shutdown()) {                                 \
-+} VXHSAIOCB;
++            error_setg(&error_abort, (fmt), ## __VA_ARGS__);              \
-+
++        } else {                                                          \
-+typedef struct VXHSvDiskHostsInfo {
++            error_report((fmt), ## __VA_ARGS__);                          \
-+    void *dev_handle; /* Device handle */
++        }                                                                 \
-+    char *host; /* Host name or IP */
++    }                                                                     \
-+    int port; /* Host's port number */
++
-+} VXHSvDiskHostsInfo;
++struct VfuObjectClass {
-+
++    ObjectClass parent_class;
-+/*
++
-+ * Structure per vDisk maintained for state
++    unsigned int nr_devs;
 + */
 +typedef struct BDRVVXHSState {
 +    VXHSvDiskHostsInfo vdisk_hostinfo; /* Per host info */
 +    char *vdisk_guid;
 +    char *tlscredsid; /* tlscredsid */
 +} BDRVVXHSState;
 +
 +static void vxhs_complete_aio_bh(void *opaque)
 +{
 +    VXHSAIOCB *acb = opaque;
 +    BlockCompletionFunc *cb = acb->common.cb;
 +    void *cb_opaque = acb->common.opaque;
 +    int ret = 0;
 +
 +    if (acb->err != 0) {
 +        trace_vxhs_complete_aio(acb, acb->err);
 +        ret = (-EIO);
 +    }
 +
 +    qemu_aio_unref(acb);
 +    cb(cb_opaque, ret);
 +}
 +
 +/*
 + * Called from a libqnio thread
 + */
 +static void vxhs_iio_callback(void *ctx, uint32_t opcode, uint32_t error)
 +{
 +    VXHSAIOCB *acb = NULL;
 +
 +    switch (opcode) {
 +    case IRP_READ_REQUEST:
 +    case IRP_WRITE_REQUEST:
 +
 +        /*
 +         * ctx is VXHSAIOCB*
 +         * ctx is NULL if error is QNIOERROR_CHANNEL_HUP
 +         */
 +        if (ctx) {
 +            acb = ctx;
 +        } else {
 +            trace_vxhs_iio_callback(error);
 +            goto out;
 +        }
 +
 +        if (error) {
 +            if (!acb->err) {
 +                acb->err = error;
 +            }
 +            trace_vxhs_iio_callback(error);
 +        }
 +
 +        aio_bh_schedule_oneshot(bdrv_get_aio_context(acb->common.bs),
 +                                vxhs_complete_aio_bh, acb);
 +        break;
 +
 +    default:
 +        if (error == QNIOERROR_HUP) {
 +            /*
 +             * Channel failed, spontaneous notification,
 +             * not in response to I/O
 +             */
 +            trace_vxhs_iio_callback_chnfail(error, errno);
 +        } else {
 +            trace_vxhs_iio_callback_unknwn(opcode, error);
 +        }
 +        break;
 +    }
 +out:
 +    return;
 +}
 +
 +static QemuOptsList runtime_opts = {
 +    .name = "vxhs",
 +    .head = QTAILQ_HEAD_INITIALIZER(runtime_opts.head),
 +    .desc = {
 +        {
 +            .name = VXHS_OPT_FILENAME,
 +            .type = QEMU_OPT_STRING,
 +            .help = "URI to the Veritas HyperScale image",
 +        },
 +        {
 +            .name = VXHS_OPT_VDISK_ID,
 +            .type = QEMU_OPT_STRING,
 +            .help = "UUID of the VxHS vdisk",
 +        },
 +        {
 +            .name = "tls-creds",
 +            .type = QEMU_OPT_STRING,
 +            .help = "ID of the TLS/SSL credentials to use",
 +        },
 +        { /* end of list */ }
 +    },
 +};
 +
-+static QemuOptsList runtime_tcp_opts = {
++struct VfuObject {
-+    .name = "vxhs_tcp",
++    /* private */
-+    .head = QTAILQ_HEAD_INITIALIZER(runtime_tcp_opts.head),
++    Object parent;
-+    .desc = {
++
-+        {
++    SocketAddress *socket;
-+            .name = VXHS_OPT_HOST,
++
-+            .type = QEMU_OPT_STRING,
++    char *device;
-+            .help = "host address (ipv4 addresses)",
++
-+        },
++    Error *err;
 +        {
 +            .name = VXHS_OPT_PORT,
 +            .type = QEMU_OPT_NUMBER,
 +            .help = "port number on which VxHSD is listening (default 9999)",
 +            .def_value_str = "9999"
 +        },
 +        { /* end of list */ }
 +    },
 +};
 +
-+/*
++static bool vfu_object_auto_shutdown(void)
-+ * Parse incoming URI and populate *options with the host
++{
-+ * and device information
++    bool auto_shutdown = true;
-+ */
++    Error *local_err = NULL;
-+static int vxhs_parse_uri(const char *filename, QDict *options)
++
-+{
++    if (!current_machine) {
-+    URI *uri = NULL;
++        return auto_shutdown;
-+    char *port;
++    }
-+    int ret = 0;
++
-+
++    auto_shutdown = object_property_get_bool(OBJECT(current_machine),
-+    trace_vxhs_parse_uri_filename(filename);
++                                             "auto-shutdown",
-+    uri = uri_parse(filename);
++                                             &local_err);
-+    if (!uri || !uri->server || !uri->path) {
++
-+        uri_free(uri);
++    /*
-+        return -EINVAL;
++     * local_err would be set if no such property exists - safe to ignore.
-+    }
++     * Unlikely scenario as auto-shutdown is always defined for
-+
++     * TYPE_REMOTE_MACHINE, and  TYPE_VFU_OBJECT only works with
-+    qdict_put(options, VXHS_OPT_SERVER".host", qstring_from_str(uri->server));
++     * TYPE_REMOTE_MACHINE
-+
++     */
-+    if (uri->port) {
++    if (local_err) {
-+        port = g_strdup_printf("%d", uri->port);
++        auto_shutdown = true;
-+        qdict_put(options, VXHS_OPT_SERVER".port", qstring_from_str(port));
++        error_free(local_err);
-+        g_free(port);
++    }
-+    }
++
-+
++    return auto_shutdown;
-+    qdict_put(options, "vdisk-id", qstring_from_str(uri->path));
++}
 +
-+    trace_vxhs_parse_uri_hostinfo(uri->server, uri->port);
++static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
-+    uri_free(uri);
++                                  void *opaque, Error **errp)
-+
++{
-+    return ret;
++    VfuObject *o = VFU_OBJECT(obj);
-+}
++
-+
++    qapi_free_SocketAddress(o->socket);
-+static void vxhs_parse_filename(const char *filename, QDict *options,
++
-+                                Error **errp)
++    o->socket = NULL;
-+{
++
-+    if (qdict_haskey(options, "vdisk-id") || qdict_haskey(options, "server")) {
++    visit_type_SocketAddress(v, name, &o->socket, errp);
-+        error_setg(errp, "vdisk-id/server and a file name may not be specified "
++
-+                         "at the same time");
++    if (o->socket->type != SOCKET_ADDRESS_TYPE_UNIX) {
 +        error_setg(errp, "vfu: Unsupported socket type - %s",
 +                   SocketAddressType_str(o->socket->type));
 +        qapi_free_SocketAddress(o->socket);
 +        o->socket = NULL;
 +        return;
 +    }
 +
-+    if (strstr(filename, "://")) {
++    trace_vfu_prop("socket", o->socket->u.q_unix.path);
-+        int ret = vxhs_parse_uri(filename, options);
++}
-+        if (ret < 0) {
++
-+            error_setg(errp, "Invalid URI. URI should be of the form "
++static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
-+                       "  vxhs://<host_ip>:<port>/<vdisk-id>");
++{
-+        }
++    VfuObject *o = VFU_OBJECT(obj);
-+    }
++
-+}
++    g_free(o->device);
 +
-+static int vxhs_init_and_ref(void)
++    o->device = g_strdup(str);
-+{
++
-+    if (vxhs_ref++ == 0) {
++    trace_vfu_prop("device", str);
-+        if (iio_init(QNIO_VERSION, vxhs_iio_callback)) {
++}
-+            return -ENODEV;
++
-+        }
++static void vfu_object_init(Object *obj)
-+    }
++{
-+    return 0;
++    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
-+}
++    VfuObject *o = VFU_OBJECT(obj);
 +
-+static void vxhs_unref(void)
++    k->nr_devs++;
-+{
++
-+    if (--vxhs_ref == 0) {
++    if (!object_dynamic_cast(OBJECT(current_machine), TYPE_REMOTE_MACHINE)) {
-+        iio_fini();
++        error_setg(&o->err, "vfu: %s only compatible with %s machine",
-+    }
++                   TYPE_VFU_OBJECT, TYPE_REMOTE_MACHINE);
 +}
 +
 +static void vxhs_get_tls_creds(const char *id, char **cacert,
 +                               char **key, char **cert, Error **errp)
 +{
 +    Object *obj;
 +    QCryptoTLSCreds *creds;
 +    QCryptoTLSCredsX509 *creds_x509;
 +
 +    obj = object_resolve_path_component(
 +        object_get_objects_root(), id);
 +
 +    if (!obj) {
 +        error_setg(errp, "No TLS credentials with id '%s'",
 +                   id);
 +        return;
 +    }
-+
++}
-+    creds_x509 = (QCryptoTLSCredsX509 *)
++
-+        object_dynamic_cast(obj, TYPE_QCRYPTO_TLS_CREDS_X509);
++static void vfu_object_finalize(Object *obj)
-+
++{
-+    if (!creds_x509) {
++    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
-+        error_setg(errp, "Object with id '%s' is not TLS credentials",
++    VfuObject *o = VFU_OBJECT(obj);
-+                   id);
++
-+        return;
++    k->nr_devs--;
-+    }
++
-+
++    qapi_free_SocketAddress(o->socket);
-+    creds = &creds_x509->parent_obj;
++
-+
++    o->socket = NULL;
-+    if (creds->endpoint != QCRYPTO_TLS_CREDS_ENDPOINT_CLIENT) {
++
-+        error_setg(errp,
++    g_free(o->device);
-+                   "Expecting TLS credentials with a client endpoint");
++
-+        return;
++    o->device = NULL;
-+    }
++
-+
++    if (!k->nr_devs && vfu_object_auto_shutdown()) {
-+    /*
++        qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
-+     * Get the cacert, client_cert and client_key file names.
++    }
-+     */
++}
-+    if (!creds->dir) {
++
-+        error_setg(errp, "TLS object missing 'dir' property value");
++static void vfu_object_class_init(ObjectClass *klass, void *data)
-+        return;
++{
-+    }
++    VfuObjectClass *k = VFU_OBJECT_CLASS(klass);
 +
-+    *cacert = g_strdup_printf("%s/%s", creds->dir,
++    k->nr_devs = 0;
-+                              QCRYPTO_TLS_CREDS_X509_CA_CERT);
++
-+    *cert = g_strdup_printf("%s/%s", creds->dir,
++    object_class_property_add(klass, "socket", "SocketAddress", NULL,
-+                            QCRYPTO_TLS_CREDS_X509_CLIENT_CERT);
++                              vfu_object_set_socket, NULL, NULL);
-+    *key = g_strdup_printf("%s/%s", creds->dir,
++    object_class_property_set_description(klass, "socket",
-+                           QCRYPTO_TLS_CREDS_X509_CLIENT_KEY);
++                                          "SocketAddress "
-+}
++                                          "(ex: type=unix,path=/tmp/sock). "
-+
++                                          "Only UNIX is presently supported");
-+static int vxhs_open(BlockDriverState *bs, QDict *options,
++    object_class_property_add_str(klass, "device", NULL,
-+                     int bdrv_flags, Error **errp)
++                                  vfu_object_set_device);
-+{
++    object_class_property_set_description(klass, "device",
-+    BDRVVXHSState *s = bs->opaque;
++                                          "device ID - only PCI devices "
-+    void *dev_handlep;
++                                          "are presently supported");
-+    QDict *backing_options = NULL;
++}
-+    QemuOpts *opts = NULL;
++
-+    QemuOpts *tcp_opts = NULL;
++static const TypeInfo vfu_object_info = {
-+    char *of_vsa_addr = NULL;
++    .name = TYPE_VFU_OBJECT,
-+    Error *local_err = NULL;
++    .parent = TYPE_OBJECT,
-+    const char *vdisk_id_opt;
++    .instance_size = sizeof(VfuObject),
-+    const char *server_host_opt;
++    .instance_init = vfu_object_init,
-+    int ret = 0;
++    .instance_finalize = vfu_object_finalize,
-+    char *cacert = NULL;
++    .class_size = sizeof(VfuObjectClass),
-+    char *client_key = NULL;
++    .class_init = vfu_object_class_init,
-+    char *client_cert = NULL;
++    .interfaces = (InterfaceInfo[]) {
-+
++        { TYPE_USER_CREATABLE },
-+    ret = vxhs_init_and_ref();
++        { }
-+    if (ret < 0) {
++    }
 +        ret = -EINVAL;
 +        goto out;
 +    }
 +
 +    /* Create opts info from runtime_opts and runtime_tcp_opts list */
 +    opts = qemu_opts_create(&runtime_opts, NULL, 0, &error_abort);
 +    tcp_opts = qemu_opts_create(&runtime_tcp_opts, NULL, 0, &error_abort);
 +
 +    qemu_opts_absorb_qdict(opts, options, &local_err);
 +    if (local_err) {
 +        ret = -EINVAL;
 +        goto out;
 +    }
 +
 +    /* vdisk-id is the disk UUID */
 +    vdisk_id_opt = qemu_opt_get(opts, VXHS_OPT_VDISK_ID);
 +    if (!vdisk_id_opt) {
 +        error_setg(&local_err, QERR_MISSING_PARAMETER, VXHS_OPT_VDISK_ID);
 +        ret = -EINVAL;
 +        goto out;
 +    }
 +
 +    /* vdisk-id may contain a leading '/' */
 +    if (strlen(vdisk_id_opt) > UUID_FMT_LEN + 1) {
 +        error_setg(&local_err, "vdisk-id cannot be more than %d characters",
 +                   UUID_FMT_LEN);
 +        ret = -EINVAL;
 +        goto out;
 +    }
 +
 +    s->vdisk_guid = g_strdup(vdisk_id_opt);
 +    trace_vxhs_open_vdiskid(vdisk_id_opt);
 +
 +    /* get the 'server.' arguments */
 +    qdict_extract_subqdict(options, &backing_options, VXHS_OPT_SERVER".");
 +
 +    qemu_opts_absorb_qdict(tcp_opts, backing_options, &local_err);
 +    if (local_err != NULL) {
 +        ret = -EINVAL;
 +        goto out;
 +    }
 +
 +    server_host_opt = qemu_opt_get(tcp_opts, VXHS_OPT_HOST);
 +    if (!server_host_opt) {
 +        error_setg(&local_err, QERR_MISSING_PARAMETER,
 +                   VXHS_OPT_SERVER"."VXHS_OPT_HOST);
 +        ret = -EINVAL;
 +        goto out;
 +    }
 +
 +    if (strlen(server_host_opt) > MAXHOSTNAMELEN) {
 +        error_setg(&local_err, "server.host cannot be more than %d characters",
 +                   MAXHOSTNAMELEN);
 +        ret = -EINVAL;
 +        goto out;
 +    }
 +
 +    /* check if we got tls-creds via the --object argument */
 +    s->tlscredsid = g_strdup(qemu_opt_get(opts, "tls-creds"));
 +    if (s->tlscredsid) {
 +        vxhs_get_tls_creds(s->tlscredsid, &cacert, &client_key,
 +                           &client_cert, &local_err);
 +        if (local_err != NULL) {
 +            ret = -EINVAL;
 +            goto out;
 +        }
 +        trace_vxhs_get_creds(cacert, client_key, client_cert);
 +    }
 +
 +    s->vdisk_hostinfo.host = g_strdup(server_host_opt);
 +    s->vdisk_hostinfo.port = g_ascii_strtoll(qemu_opt_get(tcp_opts,
 +                                                          VXHS_OPT_PORT),
 +                                                          NULL, 0);
 +
 +    trace_vxhs_open_hostinfo(s->vdisk_hostinfo.host,
 +                             s->vdisk_hostinfo.port);
 +
 +    of_vsa_addr = g_strdup_printf("of://%s:%d",
 +                                  s->vdisk_hostinfo.host,
 +                                  s->vdisk_hostinfo.port);
 +
 +    /*
 +     * Open qnio channel to storage agent if not opened before
 +     */
 +    dev_handlep = iio_open(of_vsa_addr, s->vdisk_guid, 0,
 +                           cacert, client_key, client_cert);
 +    if (dev_handlep == NULL) {
 +        trace_vxhs_open_iio_open(of_vsa_addr);
 +        ret = -ENODEV;
 +        goto out;
 +    }
 +    s->vdisk_hostinfo.dev_handle = dev_handlep;
 +
 +out:
 +    g_free(of_vsa_addr);
 +    QDECREF(backing_options);
 +    qemu_opts_del(tcp_opts);
 +    qemu_opts_del(opts);
 +    g_free(cacert);
 +    g_free(client_key);
 +    g_free(client_cert);
 +
 +    if (ret < 0) {
 +        vxhs_unref();
 +        error_propagate(errp, local_err);
 +        g_free(s->vdisk_hostinfo.host);
 +        g_free(s->vdisk_guid);
 +        g_free(s->tlscredsid);
 +        s->vdisk_guid = NULL;
 +    }
 +
 +    return ret;
 +}
 +
 +static const AIOCBInfo vxhs_aiocb_info = {
 +    .aiocb_size = sizeof(VXHSAIOCB)
 +};
 +
-+/*
++static void vfu_register_types(void)
-+ * This allocates QEMU-VXHS callback for each IO
++{
-+ * and is passed to QNIO. When QNIO completes the work,
++    type_register_static(&vfu_object_info);
-+ * it will be passed back through the callback.
++}
-+ */
++
-+static BlockAIOCB *vxhs_aio_rw(BlockDriverState *bs, int64_t sector_num,
++type_init(vfu_register_types);
-+                               QEMUIOVector *qiov, int nb_sectors,
+diff --git a/hw/remote/meson.build b/hw/remote/meson.build
-+                               BlockCompletionFunc *cb, void *opaque,
+index XXXXXXX..XXXXXXX 100644
-+                               VDISKAIOCmd iodir)
+--- a/hw/remote/meson.build
-+{
++++ b/hw/remote/meson.build
-+    VXHSAIOCB *acb = NULL;
+@@ -XXX,XX +XXX,XX @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
-+    BDRVVXHSState *s = bs->opaque;
+ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
-+    size_t size;
+ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
-+    uint64_t offset;
+ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
-+    int iio_flags = 0;
++remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: files('vfio-user-obj.c'))
-+    int ret = 0;
-+    void *dev_handle = s->vdisk_hostinfo.dev_handle;
+ remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: libvfio_user_dep)
-+
-+    offset = sector_num * BDRV_SECTOR_SIZE;
+diff --git a/hw/remote/trace-events b/hw/remote/trace-events
-+    size = nb_sectors * BDRV_SECTOR_SIZE;
+index XXXXXXX..XXXXXXX 100644
-+    acb = qemu_aio_get(&vxhs_aiocb_info, bs, cb, opaque);
+--- a/hw/remote/trace-events
-+
++++ b/hw/remote/trace-events
-+    /*
+@@ -XXX,XX +XXX,XX @@
-+     * Initialize VXHSAIOCB.
-+     */
+ mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process"
-+    acb->err = 0;
+ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process"
 +
-+    iio_flags = IIO_FLAG_ASYNC;
++# vfio-user-obj.c
-+
++vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
 +    switch (iodir) {
 +    case VDISK_AIO_WRITE:
 +            ret = iio_writev(dev_handle, acb, qiov->iov, qiov->niov,
 +                             offset, (uint64_t)size, iio_flags);
 +            break;
 +    case VDISK_AIO_READ:
 +            ret = iio_readv(dev_handle, acb, qiov->iov, qiov->niov,
 +                            offset, (uint64_t)size, iio_flags);
 +            break;
 +    default:
 +            trace_vxhs_aio_rw_invalid(iodir);
 +            goto errout;
 +    }
 +
 +    if (ret != 0) {
 +        trace_vxhs_aio_rw_ioerr(s->vdisk_guid, iodir, size, offset,
 +                                acb, ret, errno);
 +        goto errout;
 +    }
 +    return &acb->common;
 +
 +errout:
 +    qemu_aio_unref(acb);
 +    return NULL;
 +}
 +
 +static BlockAIOCB *vxhs_aio_readv(BlockDriverState *bs,
 +                                   int64_t sector_num, QEMUIOVector *qiov,
 +                                   int nb_sectors,
 +                                   BlockCompletionFunc *cb, void *opaque)
 +{
 +    return vxhs_aio_rw(bs, sector_num, qiov, nb_sectors, cb,
 +                       opaque, VDISK_AIO_READ);
 +}
 +
 +static BlockAIOCB *vxhs_aio_writev(BlockDriverState *bs,
 +                                   int64_t sector_num, QEMUIOVector *qiov,
 +                                   int nb_sectors,
 +                                   BlockCompletionFunc *cb, void *opaque)
 +{
 +    return vxhs_aio_rw(bs, sector_num, qiov, nb_sectors,
 +                       cb, opaque, VDISK_AIO_WRITE);
 +}
 +
 +static void vxhs_close(BlockDriverState *bs)
 +{
 +    BDRVVXHSState *s = bs->opaque;
 +
 +    trace_vxhs_close(s->vdisk_guid);
 +
 +    g_free(s->vdisk_guid);
 +    s->vdisk_guid = NULL;
 +
 +    /*
 +     * Close vDisk device
 +     */
 +    if (s->vdisk_hostinfo.dev_handle) {
 +        iio_close(s->vdisk_hostinfo.dev_handle);
 +        s->vdisk_hostinfo.dev_handle = NULL;
 +    }
 +
 +    vxhs_unref();
 +
 +    /*
 +     * Free the dynamically allocated host string etc
 +     */
 +    g_free(s->vdisk_hostinfo.host);
 +    g_free(s->tlscredsid);
 +    s->tlscredsid = NULL;
 +    s->vdisk_hostinfo.host = NULL;
 +    s->vdisk_hostinfo.port = 0;
 +}
 +
 +static int64_t vxhs_get_vdisk_stat(BDRVVXHSState *s)
 +{
 +    int64_t vdisk_size = -1;
 +    int ret = 0;
 +    void *dev_handle = s->vdisk_hostinfo.dev_handle;
 +
 +    ret = iio_ioctl(dev_handle, IOR_VDISK_STAT, &vdisk_size, 0);
 +    if (ret < 0) {
 +        trace_vxhs_get_vdisk_stat_err(s->vdisk_guid, ret, errno);
 +        return -EIO;
 +    }
 +
 +    trace_vxhs_get_vdisk_stat(s->vdisk_guid, vdisk_size);
 +    return vdisk_size;
 +}
 +
 +/*
 + * Returns the size of vDisk in bytes. This is required
 + * by QEMU block upper block layer so that it is visible
 + * to guest.
 + */
 +static int64_t vxhs_getlength(BlockDriverState *bs)
 +{
 +    BDRVVXHSState *s = bs->opaque;
 +    int64_t vdisk_size;
 +
 +    vdisk_size = vxhs_get_vdisk_stat(s);
 +    if (vdisk_size < 0) {
 +        return -EIO;
 +    }
 +
 +    return vdisk_size;
 +}
 +
 +static BlockDriver bdrv_vxhs = {
 +    .format_name                  = "vxhs",
 +    .protocol_name                = "vxhs",
 +    .instance_size                = sizeof(BDRVVXHSState),
 +    .bdrv_file_open               = vxhs_open,
 +    .bdrv_parse_filename          = vxhs_parse_filename,
 +    .bdrv_close                   = vxhs_close,
 +    .bdrv_getlength               = vxhs_getlength,
 +    .bdrv_aio_readv               = vxhs_aio_readv,
 +    .bdrv_aio_writev              = vxhs_aio_writev,
 +};
 +
 +static void bdrv_vxhs_init(void)
 +{
 +    bdrv_register(&bdrv_vxhs);
 +}
 +
 +block_init(bdrv_vxhs_init);
 diff --git a/configure b/configure
 index XXXXXXX..XXXXXXX 100755
 --- a/configure
 +++ b/configure
@@ -XXX,XX +XXX,XX @@ numa=""
  tcmalloc="no"
  jemalloc="no"
  replication="yes"
 +vxhs=""
  supported_cpu="no"
  supported_os="no"
@@ -XXX,XX +XXX,XX @@ for opt do
    ;;
    --enable-replication) replication="yes"
    ;;
 +  --disable-vxhs) vxhs="no"
 +  ;;
 +  --enable-vxhs) vxhs="yes"
 +  ;;
    *)
        echo "ERROR: unknown option $opt"
        echo "Try '$0 --help' for more information"
@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
    xfsctl          xfsctl support
    qom-cast-debug  cast debugging support
    tools           build qemu-io, qemu-nbd and qemu-image tools
 +  vxhs            Veritas HyperScale vDisk backend support
  NOTE: The object files are built at the place where configure is launched
  EOF
@@ -XXX,XX +XXX,XX @@ if compile_prog "" "" ; then
  fi
  ##########################################
 +# Veritas HyperScale block driver VxHS
 +# Check if libvxhs is installed
 +
 +if test "$vxhs" != "no" ; then
 +  cat > $TMPC <<EOF
 +#include <stdint.h>
 +#include <qnio/qnio_api.h>
 +
 +void *vxhs_callback;
 +
 +int main(void) {
 +    iio_init(QNIO_VERSION, vxhs_callback);
 +    return 0;
 +}
 +EOF
 +  vxhs_libs="-lvxhs -lssl"
 +  if compile_prog "" "$vxhs_libs" ; then
 +    vxhs=yes
 +  else
 +    if test "$vxhs" = "yes" ; then
 +      feature_not_found "vxhs block device" "Install libvxhs See github"
 +    fi
 +    vxhs=no
 +  fi
 +fi
 +
 +##########################################
  # End of CC checks
  # After here, no more $cc or $ld runs
@@ -XXX,XX +XXX,XX @@ echo "tcmalloc support  $tcmalloc"
  echo "jemalloc support  $jemalloc"
  echo "avx2 optimization $avx2_opt"
  echo "replication support $replication"
 +echo "VxHS block device $vxhs"
  if test "$sdl_too_old" = "yes"; then
  echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -XXX,XX +XXX,XX @@ if test "$pthread_setname_np" = "yes" ; then
    echo "CONFIG_PTHREAD_SETNAME_NP=y" >> $config_host_mak
  fi
 +if test "$vxhs" = "yes" ; then
 +  echo "CONFIG_VXHS=y" >> $config_host_mak
 +  echo "VXHS_LIBS=$vxhs_libs" >> $config_host_mak
 +fi
 +
  if test "$tcg_interpreter" = "yes"; then
    QEMU_INCLUDES="-I\$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
  elif test "$ARCH" = "sparc64" ; then
 diff --git a/qapi/block-core.json b/qapi/block-core.json
 index XXXXXXX..XXXXXXX 100644
 --- a/qapi/block-core.json
 +++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
  #
  # Drivers that are supported in block device operations.
  #
 +# @vxhs: Since 2.10
 +#
  # Since: 2.9
  ##
  { 'enum': 'BlockdevDriver',
@@ -XXX,XX +XXX,XX @@
              'host_device', 'http', 'https', 'iscsi', 'luks', 'nbd', 'nfs',
              'null-aio', 'null-co', 'parallels', 'qcow', 'qcow2', 'qed',
              'quorum', 'raw', 'rbd', 'replication', 'sheepdog', 'ssh',
 -            'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat' ] }
 +            'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] }
  ##
  # @BlockdevOptionsFile:
@@ -XXX,XX +XXX,XX @@
    'data': { '*offset': 'int', '*size': 'int' } }
  ##
 +# @BlockdevOptionsVxHS:
 +#
 +# Driver specific block device options for VxHS
 +#
 +# @vdisk-id:    UUID of VxHS volume
 +# @server:      vxhs server IP, port
 +# @tls-creds:   TLS credentials ID
 +#
 +# Since: 2.10
 +##
 +{ 'struct': 'BlockdevOptionsVxHS',
 +  'data': { 'vdisk-id': 'str',
 +            'server': 'InetSocketAddressBase',
 +            '*tls-creds': 'str' } }
 +
 +##
  # @BlockdevOptions:
  #
  # Options for creating a block device.  Many options are available for all
@@ -XXX,XX +XXX,XX @@
        'vhdx':       'BlockdevOptionsGenericFormat',
        'vmdk':       'BlockdevOptionsGenericCOWFormat',
        'vpc':        'BlockdevOptionsGenericFormat',
 -      'vvfat':      'BlockdevOptionsVVFAT'
 +      'vvfat':      'BlockdevOptionsVVFAT',
 +      'vxhs':       'BlockdevOptionsVxHS'
    } }
  ##
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 10/12] block/rbd - update variable names to more apt names
+[PULL 08/18] vfio-user: instantiate vfio-user context
-Update 'clientname' to be 'user', which tracks better with both
+From: Jagannathan Raman <jag.raman@oracle.com>
 the QAPI and rados variable naming.
-Update 'name' to be 'image_name', as it indicates the rbd image.
+create a context with the vfio-user library to run a PCI device
 Naming it 'image' would have been ideal, but we are using that for
 the rados_image_t value returned by rbd_open().
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
+Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
+Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Message-id: a452871ac8c812ff96fc4f0ce6037f4769953fab.1655151679.git.jag.raman@oracle.com
-Reviewed-by: John Snow <jsnow@redhat.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: b7ec1fb2e1cf36f9b6911631447a5b0422590b7d.1491597120.git.jcody@redhat.com
 ---
- block/rbd.c | 33 +++++++++++++++++----------------
+ hw/remote/vfio-user-obj.c | 82 +++++++++++++++++++++++++++++++++++++++
-file changed, 17 insertions(+), 16 deletions(-)
+file changed, 82 insertions(+)
-diff --git a/block/rbd.c b/block/rbd.c
+diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/rbd.c
+--- a/hw/remote/vfio-user-obj.c
-+++ b/block/rbd.c
++++ b/hw/remote/vfio-user-obj.c
-@@ -XXX,XX +XXX,XX @@ typedef struct BDRVRBDState {
+@@ -XXX,XX +XXX,XX @@
-     rados_t cluster;
+ #include "hw/remote/machine.h"
-     rados_ioctx_t io_ctx;
+ #include "qapi/error.h"
-     rbd_image_t image;
+ #include "qapi/qapi-visit-sockets.h"
--    char *name;
++#include "qemu/notify.h"
-+    char *image_name;
++#include "sysemu/sysemu.h"
-     char *snap;
++#include "libvfio-user.h"
- } BDRVRBDState;
+ #define TYPE_VFU_OBJECT "x-vfio-user-server"
-@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_create(const char *filename, QemuOpts *opts, Error **errp)
+ OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
-     int64_t bytes = 0;
+@@ -XXX,XX +XXX,XX @@ struct VfuObject {
-     int64_t objsize;
+     char *device;
-     int obj_order = 0;
--    const char *pool, *name, *conf, *clientname, *keypairs;
+     Error *err;
-+    const char *pool, *image_name, *conf, *user, *keypairs;
++
-     const char *secretid;
++    Notifier machine_done;
-     rados_t cluster;
++
-     rados_ioctx_t io_ctx;
++    vfu_ctx_t *vfu_ctx;
-@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_create(const char *filename, QemuOpts *opts, Error **errp)
+ };
-      */
-     pool       = qdict_get_try_str(options, "pool");
++static void vfu_object_init_ctx(VfuObject *o, Error **errp);
-     conf       = qdict_get_try_str(options, "conf");
++
--    clientname = qdict_get_try_str(options, "user");
+ static bool vfu_object_auto_shutdown(void)
--    name       = qdict_get_try_str(options, "image");
+ {
-+    user       = qdict_get_try_str(options, "user");
+     bool auto_shutdown = true;
-+    image_name = qdict_get_try_str(options, "image");
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
-     keypairs   = qdict_get_try_str(options, "=keyvalue-pairs");
+ {
+     VfuObject *o = VFU_OBJECT(obj);
--    ret = rados_create(&cluster, clientname);
-+    ret = rados_create(&cluster, user);
++    if (o->vfu_ctx) {
-     if (ret < 0) {
++        error_setg(errp, "vfu: Unable to set socket property - server busy");
-         error_setg_errno(errp, -ret, "error initializing");
++        return;
-         goto exit;
++    }
-@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_create(const char *filename, QemuOpts *opts, Error **errp)
++
-         goto shutdown;
+     qapi_free_SocketAddress(o->socket);
      o->socket = NULL;
@@ -XXX,XX +XXX,XX @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
      }
--    ret = rbd_create(io_ctx, name, bytes, &obj_order);
+     trace_vfu_prop("socket", o->socket->u.q_unix.path);
-+    ret = rbd_create(io_ctx, image_name, bytes, &obj_order);
++
-     if (ret < 0) {
++    vfu_object_init_ctx(o, errp);
-         error_setg_errno(errp, -ret, "error rbd create");
+ }
  static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
  {
      VfuObject *o = VFU_OBJECT(obj);
 +    if (o->vfu_ctx) {
 +        error_setg(errp, "vfu: Unable to set device property - server busy");
 +        return;
 +    }
 +
      g_free(o->device);
      o->device = g_strdup(str);
      trace_vfu_prop("device", str);
 +
 +    vfu_object_init_ctx(o, errp);
 +}
 +
 +/*
 + * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
 + * properties. It also depends on devices instantiated in QEMU. These
 + * dependencies are not available during the instance_init phase of this
 + * object's life-cycle. As such, the server is initialized after the
 + * machine is setup. machine_init_done_notifier notifies TYPE_VFU_OBJECT
 + * when the machine is setup, and the dependencies are available.
 + */
 +static void vfu_object_machine_done(Notifier *notifier, void *data)
 +{
 +    VfuObject *o = container_of(notifier, VfuObject, machine_done);
 +    Error *err = NULL;
 +
 +    vfu_object_init_ctx(o, &err);
 +
 +    if (err) {
 +        error_propagate(&error_abort, err);
 +    }
 +}
 +
 +static void vfu_object_init_ctx(VfuObject *o, Error **errp)
 +{
 +    ERRP_GUARD();
 +
 +    if (o->vfu_ctx || !o->socket || !o->device ||
 +            !phase_check(PHASE_MACHINE_READY)) {
 +        return;
 +    }
 +
 +    if (o->err) {
 +        error_propagate(errp, o->err);
 +        o->err = NULL;
 +        return;
 +    }
 +
 +    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path, 0,
 +                                o, VFU_DEV_TYPE_PCI);
 +    if (o->vfu_ctx == NULL) {
 +        error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
 +        return;
 +    }
  }
  static void vfu_object_init(Object *obj)
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init(Object *obj)
                     TYPE_VFU_OBJECT, TYPE_REMOTE_MACHINE);
          return;
      }
-@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
++
-                          Error **errp)
++    if (!phase_check(PHASE_MACHINE_READY)) {
- {
++        o->machine_done.notify = vfu_object_machine_done;
-     BDRVRBDState *s = bs->opaque;
++        qemu_add_machine_init_done_notifier(&o->machine_done);
--    const char *pool, *snap, *conf, *clientname, *name, *keypairs;
++    }
-+    const char *pool, *snap, *conf, *user, *image_name, *keypairs;
++
-     const char *secretid;
+ }
-     QemuOpts *opts;
-     Error *local_err = NULL;
+ static void vfu_object_finalize(Object *obj)
-@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
-     pool           = qemu_opt_get(opts, "pool");
-     conf           = qemu_opt_get(opts, "conf");
+     o->socket = NULL;
-     snap           = qemu_opt_get(opts, "snapshot");
--    clientname     = qemu_opt_get(opts, "user");
++    if (o->vfu_ctx) {
--    name           = qemu_opt_get(opts, "image");
++        vfu_destroy_ctx(o->vfu_ctx);
-+    user           = qemu_opt_get(opts, "user");
++        o->vfu_ctx = NULL;
-+    image_name     = qemu_opt_get(opts, "image");
++    }
-     keypairs       = qemu_opt_get(opts, "=keyvalue-pairs");
++
+     g_free(o->device);
--    if (!pool || !name) {
-+    if (!pool || !image_name) {
+     o->device = NULL;
-         error_setg(errp, "Parameters 'pool' and 'image' are required");
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
-         r = -EINVAL;
+     if (!k->nr_devs && vfu_object_auto_shutdown()) {
-         goto failed_opts;
+         qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
      }
++
--    r = rados_create(&s->cluster, clientname);
++    if (o->machine_done.notify) {
-+    r = rados_create(&s->cluster, user);
++        qemu_remove_machine_init_done_notifier(&o->machine_done);
-     if (r < 0) {
++        o->machine_done.notify = NULL;
-         error_setg_errno(errp, -r, "error initializing");
++    }
          goto failed_opts;
      }
      s->snap = g_strdup(snap);
 -    s->name = g_strdup(name);
 +    s->image_name = g_strdup(image_name);
      /* try default location when conf=NULL, but ignore failure */
      r = rados_conf_read_file(s->cluster, conf);
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
      }
      /* rbd_open is always r/w */
 -    r = rbd_open(s->io_ctx, s->name, &s->image, s->snap);
 +    r = rbd_open(s->io_ctx, s->image_name, &s->image, s->snap);
      if (r < 0) {
 -        error_setg_errno(errp, -r, "error reading header from %s", s->name);
 +        error_setg_errno(errp, -r, "error reading header from %s",
 +                         s->image_name);
          goto failed_open;
      }
@@ -XXX,XX +XXX,XX @@ failed_open:
  failed_shutdown:
      rados_shutdown(s->cluster);
      g_free(s->snap);
 -    g_free(s->name);
 +    g_free(s->image_name);
  failed_opts:
      qemu_opts_del(opts);
      g_free(mon_host);
@@ -XXX,XX +XXX,XX @@ static void qemu_rbd_close(BlockDriverState *bs)
      rbd_close(s->image);
      rados_ioctx_destroy(s->io_ctx);
      g_free(s->snap);
 -    g_free(s->name);
 +    g_free(s->image_name);
      rados_shutdown(s->cluster);
  }
+ static void vfu_object_class_init(ObjectClass *klass, void *data)
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 06/12] block: honor BDRV_O_ALLOW_RDWR when clearing bs->read_only
+[PULL 09/18] vfio-user: find and init PCI device
-The BDRV_O_ALLOW_RDWR flag allows / prohibits the changing of
+From: Jagannathan Raman <jag.raman@oracle.com>
 the BDS 'read_only' state, but there are a few places where it
 is ignored.  In the bdrv_set_read_only() helper, make sure to
 honor the flag.
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Find the PCI device with specified id. Initialize the device context
 with the QEMU PCI device
 Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
 Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Reviewed-by: John Snow <jsnow@redhat.com>
+Message-id: 7798dbd730099b33fdd00c4c202cfe79e5c5c151.1655151679.git.jag.raman@oracle.com
-Message-id: be2e5fb2d285cbece2b6d06bed54a6f56520d251.1491597120.git.jcody@redhat.com
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
- block.c | 7 +++++++
+ hw/remote/vfio-user-obj.c | 67 +++++++++++++++++++++++++++++++++++++++
-file changed, 7 insertions(+)
+file changed, 67 insertions(+)
-diff --git a/block.c b/block.c
+diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 index XXXXXXX..XXXXXXX 100644
---- a/block.c
+--- a/hw/remote/vfio-user-obj.c
-+++ b/block.c
++++ b/hw/remote/vfio-user-obj.c
-@@ -XXX,XX +XXX,XX @@ int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
+@@ -XXX,XX +XXX,XX @@
-         return -EINVAL;
+ #include "qemu/notify.h"
  #include "sysemu/sysemu.h"
  #include "libvfio-user.h"
 +#include "hw/qdev-core.h"
 +#include "hw/pci/pci.h"
  #define TYPE_VFU_OBJECT "x-vfio-user-server"
  OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ struct VfuObject {
      Notifier machine_done;
      vfu_ctx_t *vfu_ctx;
 +
 +    PCIDevice *pci_dev;
 +
 +    Error *unplug_blocker;
  };
  static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -XXX,XX +XXX,XX @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
  static void vfu_object_init_ctx(VfuObject *o, Error **errp)
  {
      ERRP_GUARD();
 +    DeviceState *dev = NULL;
 +    vfu_pci_type_t pci_type = VFU_PCI_TYPE_CONVENTIONAL;
 +    int ret;
      if (o->vfu_ctx || !o->socket || !o->device ||
              !phase_check(PHASE_MACHINE_READY)) {
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
          error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
          return;
      }
++
-+    /* Do not clear read_only if it is prohibited */
++    dev = qdev_find_recursive(sysbus_get_default(), o->device);
-+    if (!read_only && !(bs->open_flags & BDRV_O_ALLOW_RDWR)) {
++    if (dev == NULL) {
-+        error_setg(errp, "Node '%s' is read only",
++        error_setg(errp, "vfu: Device %s not found", o->device);
-+                   bdrv_get_device_or_node_name(bs));
++        goto fail;
 +        return -EPERM;
 +    }
 +
-     bs->read_only = read_only;
++    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
-     return 0;
++        error_setg(errp, "vfu: %s not a PCI device", o->device);
 +        goto fail;
 +    }
 +
 +    o->pci_dev = PCI_DEVICE(dev);
 +
 +    object_ref(OBJECT(o->pci_dev));
 +
 +    if (pci_is_express(o->pci_dev)) {
 +        pci_type = VFU_PCI_TYPE_EXPRESS;
 +    }
 +
 +    ret = vfu_pci_init(o->vfu_ctx, pci_type, PCI_HEADER_TYPE_NORMAL, 0);
 +    if (ret < 0) {
 +        error_setg(errp,
 +                   "vfu: Failed to attach PCI device %s to context - %s",
 +                   o->device, strerror(errno));
 +        goto fail;
 +    }
 +
 +    error_setg(&o->unplug_blocker,
 +               "vfu: %s for %s must be deleted before unplugging",
 +               TYPE_VFU_OBJECT, o->device);
 +    qdev_add_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
 +
 +    return;
 +
 +fail:
 +    vfu_destroy_ctx(o->vfu_ctx);
 +    if (o->unplug_blocker && o->pci_dev) {
 +        qdev_del_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
 +        error_free(o->unplug_blocker);
 +        o->unplug_blocker = NULL;
 +    }
 +    if (o->pci_dev) {
 +        object_unref(OBJECT(o->pci_dev));
 +        o->pci_dev = NULL;
 +    }
 +    o->vfu_ctx = NULL;
  }
+ static void vfu_object_init(Object *obj)
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
+     o->device = NULL;
++    if (o->unplug_blocker && o->pci_dev) {
++        qdev_del_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
++        error_free(o->unplug_blocker);
++        o->unplug_blocker = NULL;
++    }
++
++    if (o->pci_dev) {
++        object_unref(OBJECT(o->pci_dev));
++        o->pci_dev = NULL;
++    }
++
+     if (!k->nr_devs && vfu_object_auto_shutdown()) {
+         qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
+     }
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 04/12] block: add bdrv_set_read_only() helper function
+[PULL 10/18] vfio-user: run vfio-user context
-We have a helper wrapper for checking for the BDS read_only flag,
+From: Jagannathan Raman <jag.raman@oracle.com>
 add a helper wrapper to set the read_only flag as well.
+Setup a handler to run vfio-user context. The context is driven by
+messages to the file descriptor associated with it - get the fd for
+the context and hook up the handler with it
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
+Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
+Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Message-id: e934b0090529d448b6a7972b21dfc3d7421ce494.1655151679.git.jag.raman@oracle.com
-Reviewed-by: John Snow <jsnow@redhat.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: 9b18972d05f5fa2ac16c014f0af98d680553048d.1491597120.git.jcody@redhat.com
 ---
- block.c               | 5 +++++
+ qapi/misc.json            |  31 ++++++++++
- block/bochs.c         | 2 +-
+ hw/remote/vfio-user-obj.c | 118 +++++++++++++++++++++++++++++++++++++-
- block/cloop.c         | 2 +-
+files changed, 148 insertions(+), 1 deletion(-)
  block/dmg.c           | 2 +-
  block/rbd.c           | 2 +-
  block/vvfat.c         | 4 ++--
  include/block/block.h | 1 +
 files changed, 12 insertions(+), 6 deletions(-)
-diff --git a/block.c b/block.c
+diff --git a/qapi/misc.json b/qapi/misc.json
 index XXXXXXX..XXXXXXX 100644
---- a/block.c
+--- a/qapi/misc.json
-+++ b/block.c
++++ b/qapi/misc.json
-@@ -XXX,XX +XXX,XX @@ void path_combine(char *dest, int dest_size,
+@@ -XXX,XX +XXX,XX @@
  ##
  { 'event': 'RTC_CHANGE',
    'data': { 'offset': 'int', 'qom-path': 'str' } }
 +
 +##
 +# @VFU_CLIENT_HANGUP:
 +#
 +# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the
 +# communication channel
 +#
 +# @vfu-id: ID of the TYPE_VFIO_USER_SERVER object. It is the last component
 +#          of @vfu-qom-path referenced below
 +#
 +# @vfu-qom-path: path to the TYPE_VFIO_USER_SERVER object in the QOM tree
 +#
 +# @dev-id: ID of attached PCI device
 +#
 +# @dev-qom-path: path to attached PCI device in the QOM tree
 +#
 +# Since: 7.1
 +#
 +# Example:
 +#
 +# <- { "event": "VFU_CLIENT_HANGUP",
 +#      "data": { "vfu-id": "vfu1",
 +#                "vfu-qom-path": "/objects/vfu1",
 +#                "dev-id": "sas1",
 +#                "dev-qom-path": "/machine/peripheral/sas1" },
 +#      "timestamp": { "seconds": 1265044230, "microseconds": 450486 } }
 +#
 +##
 +{ 'event': 'VFU_CLIENT_HANGUP',
 +  'data': { 'vfu-id': 'str', 'vfu-qom-path': 'str',
 +            'dev-id': 'str', 'dev-qom-path': 'str' } }
 diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/vfio-user-obj.c
 +++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
   *
   * device - id of a device on the server, a required option. PCI devices
   *          alone are supported presently.
 + *
 + * notes - x-vfio-user-server could block IO and monitor during the
 + *         initialization phase.
   */
  #include "qemu/osdep.h"
@@ -XXX,XX +XXX,XX @@
  #include "hw/remote/machine.h"
  #include "qapi/error.h"
  #include "qapi/qapi-visit-sockets.h"
 +#include "qapi/qapi-events-misc.h"
  #include "qemu/notify.h"
 +#include "qemu/thread.h"
  #include "sysemu/sysemu.h"
  #include "libvfio-user.h"
  #include "hw/qdev-core.h"
  #include "hw/pci/pci.h"
 +#include "qemu/timer.h"
  #define TYPE_VFU_OBJECT "x-vfio-user-server"
  OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ struct VfuObject {
      PCIDevice *pci_dev;
      Error *unplug_blocker;
 +
 +    int vfu_poll_fd;
  };
  static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -XXX,XX +XXX,XX @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
      vfu_object_init_ctx(o, errp);
  }
 +static void vfu_object_ctx_run(void *opaque)
 +{
 +    VfuObject *o = opaque;
 +    const char *vfu_id;
 +    char *vfu_path, *pci_dev_path;
 +    int ret = -1;
 +
 +    while (ret != 0) {
 +        ret = vfu_run_ctx(o->vfu_ctx);
 +        if (ret < 0) {
 +            if (errno == EINTR) {
 +                continue;
 +            } else if (errno == ENOTCONN) {
 +                vfu_id = object_get_canonical_path_component(OBJECT(o));
 +                vfu_path = object_get_canonical_path(OBJECT(o));
 +                g_assert(o->pci_dev);
 +                pci_dev_path = object_get_canonical_path(OBJECT(o->pci_dev));
 +                 /* o->device is a required property and is non-NULL here */
 +                g_assert(o->device);
 +                qapi_event_send_vfu_client_hangup(vfu_id, vfu_path,
 +                                                  o->device, pci_dev_path);
 +                qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
 +                o->vfu_poll_fd = -1;
 +                object_unparent(OBJECT(o));
 +                g_free(vfu_path);
 +                g_free(pci_dev_path);
 +                break;
 +            } else {
 +                VFU_OBJECT_ERROR(o, "vfu: Failed to run device %s - %s",
 +                                 o->device, strerror(errno));
 +                break;
 +            }
 +        }
 +    }
 +}
 +
 +static void vfu_object_attach_ctx(void *opaque)
 +{
 +    VfuObject *o = opaque;
 +    GPollFD pfds[1];
 +    int ret;
 +
 +    qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
 +
 +    pfds[0].fd = o->vfu_poll_fd;
 +    pfds[0].events = G_IO_IN | G_IO_HUP | G_IO_ERR;
 +
 +retry_attach:
 +    ret = vfu_attach_ctx(o->vfu_ctx);
 +    if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
 +        /**
 +         * vfu_object_attach_ctx can block QEMU's main loop
 +         * during attach - the monitor and other IO
 +         * could be unresponsive during this time.
 +         */
 +        (void)qemu_poll_ns(pfds, 1, 500 * (int64_t)SCALE_MS);
 +        goto retry_attach;
 +    } else if (ret < 0) {
 +        VFU_OBJECT_ERROR(o, "vfu: Failed to attach device %s to context - %s",
 +                         o->device, strerror(errno));
 +        return;
 +    }
 +
 +    o->vfu_poll_fd = vfu_get_poll_fd(o->vfu_ctx);
 +    if (o->vfu_poll_fd < 0) {
 +        VFU_OBJECT_ERROR(o, "vfu: Failed to get poll fd %s", o->device);
 +        return;
 +    }
 +
 +    qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_ctx_run, NULL, o);
 +}
 +
  /*
   * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
   * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
      }
  }
-+void bdrv_set_read_only(BlockDriverState *bs, bool read_only)
++/**
-+{
++ * vfu_object_init_ctx: Create and initialize libvfio-user context. Add
-+    bs->read_only = read_only;
++ *     an unplug blocker for the associated PCI device. Setup a FD handler
-+}
++ *     to process incoming messages in the context's socket.
-+
++ *
- void bdrv_get_full_backing_filename_from_filename(const char *backed,
++ *     The socket and device properties are mandatory, and this function
-                                                   const char *backing,
++ *     will not create the context without them - the setters for these
-                                                   char *dest, size_t sz,
++ *     properties should call this function when the property is set. The
-diff --git a/block/bochs.c b/block/bochs.c
++ *     machine should also be ready when this function is invoked - it is
-index XXXXXXX..XXXXXXX 100644
++ *     because QEMU objects are initialized before devices, and the
---- a/block/bochs.c
++ *     associated PCI device wouldn't be available at the object
-+++ b/block/bochs.c
++ *     initialization time. Until these conditions are satisfied, this
-@@ -XXX,XX +XXX,XX @@ static int bochs_open(BlockDriverState *bs, QDict *options, int flags,
++ *     function would return early without performing any task.
-         return -EINVAL;
++ */
  static void vfu_object_init_ctx(VfuObject *o, Error **errp)
  {
      ERRP_GUARD();
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
          return;
      }
--    bs->read_only = true; /* no write support yet */
+-    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path, 0,
-+    bdrv_set_read_only(bs, true); /* no write support yet */
++    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path,
++                                LIBVFIO_USER_FLAG_ATTACH_NB,
-     ret = bdrv_pread(bs->file, 0, &bochs, sizeof(bochs));
+                                 o, VFU_DEV_TYPE_PCI);
-     if (ret < 0) {
+     if (o->vfu_ctx == NULL) {
-diff --git a/block/cloop.c b/block/cloop.c
+         error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
-index XXXXXXX..XXXXXXX 100644
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
---- a/block/cloop.c
+                TYPE_VFU_OBJECT, o->device);
-+++ b/block/cloop.c
+     qdev_add_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
-@@ -XXX,XX +XXX,XX @@ static int cloop_open(BlockDriverState *bs, QDict *options, int flags,
-         return -EINVAL;
++    ret = vfu_realize_ctx(o->vfu_ctx);
 +    if (ret < 0) {
 +        error_setg(errp, "vfu: Failed to realize device %s- %s",
 +                   o->device, strerror(errno));
 +        goto fail;
 +    }
 +
 +    o->vfu_poll_fd = vfu_get_poll_fd(o->vfu_ctx);
 +    if (o->vfu_poll_fd < 0) {
 +        error_setg(errp, "vfu: Failed to get poll fd %s", o->device);
 +        goto fail;
 +    }
 +
 +    qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_attach_ctx, NULL, o);
 +
      return;
  fail:
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init(Object *obj)
          qemu_add_machine_init_done_notifier(&o->machine_done);
      }
--    bs->read_only = true;
++    o->vfu_poll_fd = -1;
-+    bdrv_set_read_only(bs, true);
+ }
-     /* read header */
+ static void vfu_object_finalize(Object *obj)
-     ret = bdrv_pread(bs->file, 128, &s->block_size, 4);
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
-diff --git a/block/dmg.c b/block/dmg.c
-index XXXXXXX..XXXXXXX 100644
+     o->socket = NULL;
---- a/block/dmg.c
-+++ b/block/dmg.c
++    if (o->vfu_poll_fd != -1) {
-@@ -XXX,XX +XXX,XX @@ static int dmg_open(BlockDriverState *bs, QDict *options, int flags,
++        qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
-     }
++        o->vfu_poll_fd = -1;
++    }
-     block_module_load_one("dmg-bz2");
++
--    bs->read_only = true;
+     if (o->vfu_ctx) {
-+    bdrv_set_read_only(bs, true);
+         vfu_destroy_ctx(o->vfu_ctx);
+         o->vfu_ctx = NULL;
      s->n_chunks = 0;
      s->offsets = s->lengths = s->sectors = s->sectorcounts = NULL;
 diff --git a/block/rbd.c b/block/rbd.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/rbd.c
 +++ b/block/rbd.c
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
          goto failed_open;
      }
 -    bs->read_only = (s->snap != NULL);
 +    bdrv_set_read_only(bs, (s->snap != NULL));
      qemu_opts_del(opts);
      return 0;
 diff --git a/block/vvfat.c b/block/vvfat.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/vvfat.c
 +++ b/block/vvfat.c
@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
      s->current_cluster=0xffffffff;
      /* read only is the default for safety */
 -    bs->read_only = true;
 +    bdrv_set_read_only(bs, true);
      s->qcow = NULL;
      s->qcow_filename = NULL;
      s->fat2 = NULL;
@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
          if (ret < 0) {
              goto fail;
          }
 -        bs->read_only = false;
 +        bdrv_set_read_only(bs, false);
      }
      bs->total_sectors = cyls * heads * secs;
 diff --git a/include/block/block.h b/include/block/block.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/block/block.h
 +++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
                              int64_t sector_num, int nb_sectors, int *pnum);
  bool bdrv_is_read_only(BlockDriverState *bs);
 +void bdrv_set_read_only(BlockDriverState *bs, bool read_only);
  bool bdrv_is_sg(BlockDriverState *bs);
  bool bdrv_is_inserted(BlockDriverState *bs);
  int bdrv_media_changed(BlockDriverState *bs);
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 11/12] block/rbd: Add support for reopen()
+[PULL 11/18] vfio-user: handle PCI config space accesses
-This adds support for reopen in rbd, for changing between r/w and r/o.
+From: Jagannathan Raman <jag.raman@oracle.com>
-Note, that this is only a flag change, but we will block a change from
+Define and register handlers for PCI config space accesses
 r/o to r/w if we are using an RBD internal snapshot.
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
+Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
+Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Message-id: be9d2ccf9b1d24e50dcd9c23404dbf284142cec7.1655151679.git.jag.raman@oracle.com
-Reviewed-by: John Snow <jsnow@redhat.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: d4e87539167ec6527d44c97b164eabcccf96e4f3.1491597120.git.jcody@redhat.com
 ---
- block/rbd.c | 21 +++++++++++++++++++++
+ hw/remote/vfio-user-obj.c | 51 +++++++++++++++++++++++++++++++++++++++
-file changed, 21 insertions(+)
+ hw/remote/trace-events    |  2 ++
 files changed, 53 insertions(+)
-diff --git a/block/rbd.c b/block/rbd.c
+diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/rbd.c
+--- a/hw/remote/vfio-user-obj.c
-+++ b/block/rbd.c
++++ b/hw/remote/vfio-user-obj.c
-@@ -XXX,XX +XXX,XX @@ failed_opts:
+@@ -XXX,XX +XXX,XX @@
-     return r;
+ #include "qapi/qapi-events-misc.h"
  #include "qemu/notify.h"
  #include "qemu/thread.h"
 +#include "qemu/main-loop.h"
  #include "sysemu/sysemu.h"
  #include "libvfio-user.h"
  #include "hw/qdev-core.h"
@@ -XXX,XX +XXX,XX @@ retry_attach:
      qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_ctx_run, NULL, o);
  }
++static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
++                                     size_t count, loff_t offset,
++                                     const bool is_write)
++{
++    VfuObject *o = vfu_get_private(vfu_ctx);
++    uint32_t pci_access_width = sizeof(uint32_t);
++    size_t bytes = count;
++    uint32_t val = 0;
++    char *ptr = buf;
++    int len;
 +
-+/* Since RBD is currently always opened R/W via the API,
++    /*
-+ * we just need to check if we are using a snapshot or not, in
++     * Writes to the BAR registers would trigger an update to the
-+ * order to determine if we will allow it to be R/W */
++     * global Memory and IO AddressSpaces. But the remote device
-+static int qemu_rbd_reopen_prepare(BDRVReopenState *state,
++     * never uses the global AddressSpaces, therefore overlapping
-+                                   BlockReopenQueue *queue, Error **errp)
++     * memory regions are not a problem
-+{
++     */
-+    BDRVRBDState *s = state->bs->opaque;
++    while (bytes > 0) {
-+    int ret = 0;
++        len = (bytes > pci_access_width) ? pci_access_width : bytes;
-+
++        if (is_write) {
-+    if (s->snap && state->flags & BDRV_O_RDWR) {
++            memcpy(&val, ptr, len);
-+        error_setg(errp,
++            pci_host_config_write_common(o->pci_dev, offset,
-+                   "Cannot change node '%s' to r/w when using RBD snapshot",
++                                         pci_config_size(o->pci_dev),
-+                   bdrv_get_device_or_node_name(state->bs));
++                                         val, len);
-+        ret = -EINVAL;
++            trace_vfu_cfg_write(offset, val);
 +        } else {
 +            val = pci_host_config_read_common(o->pci_dev, offset,
 +                                              pci_config_size(o->pci_dev), len);
 +            memcpy(ptr, &val, len);
 +            trace_vfu_cfg_read(offset, val);
 +        }
 +        offset += len;
 +        ptr += len;
 +        bytes -= len;
 +    }
 +
-+    return ret;
++    return count;
 +}
 +
- static void qemu_rbd_close(BlockDriverState *bs)
+ /*
- {
+  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
-     BDRVRBDState *s = bs->opaque;
+  * properties. It also depends on devices instantiated in QEMU. These
-@@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_rbd = {
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
-     .bdrv_parse_filename    = qemu_rbd_parse_filename,
+                TYPE_VFU_OBJECT, o->device);
-     .bdrv_file_open         = qemu_rbd_open,
+     qdev_add_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
-     .bdrv_close             = qemu_rbd_close,
-+    .bdrv_reopen_prepare    = qemu_rbd_reopen_prepare,
++    ret = vfu_setup_region(o->vfu_ctx, VFU_PCI_DEV_CFG_REGION_IDX,
-     .bdrv_create            = qemu_rbd_create,
++                           pci_config_size(o->pci_dev), &vfu_object_cfg_access,
-     .bdrv_has_zero_init     = bdrv_has_zero_init_1,
++                           VFU_REGION_FLAG_RW | VFU_REGION_FLAG_ALWAYS_CB,
-     .bdrv_get_info          = qemu_rbd_getinfo,
++                           NULL, 0, -1, 0);
 +    if (ret < 0) {
 +        error_setg(errp,
 +                   "vfu: Failed to setup config space handlers for %s- %s",
 +                   o->device, strerror(errno));
 +        goto fail;
 +    }
 +
      ret = vfu_realize_ctx(o->vfu_ctx);
      if (ret < 0) {
          error_setg(errp, "vfu: Failed to realize device %s- %s",
 diff --git a/hw/remote/trace-events b/hw/remote/trace-events
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/trace-events
 +++ b/hw/remote/trace-events
@@ -XXX,XX +XXX,XX @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
  # vfio-user-obj.c
  vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
 +vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 +vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
 --
-.9.3
+.36.1

-New patch
+[PULL 12/18] vfio-user: IOMMU support for remote device
+From: Jagannathan Raman <jag.raman@oracle.com>
 Assign separate address space for each device in the remote processes.
 Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
 Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: afe0b0a97582cdad42b5b25636a29c523265a10a.1655151679.git.jag.raman@oracle.com
 Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
  MAINTAINERS               |   2 +
  include/hw/remote/iommu.h |  40 ++++++++++++
  hw/remote/iommu.c         | 131 ++++++++++++++++++++++++++++++++++++++
  hw/remote/machine.c       |  13 +++-
  hw/remote/meson.build     |   1 +
 files changed, 186 insertions(+), 1 deletion(-)
  create mode 100644 include/hw/remote/iommu.h
  create mode 100644 hw/remote/iommu.c
 diff --git a/MAINTAINERS b/MAINTAINERS
 index XXXXXXX..XXXXXXX 100644
 --- a/MAINTAINERS
 +++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: hw/remote/iohub.c
  F: include/hw/remote/iohub.h
  F: subprojects/libvfio-user
  F: hw/remote/vfio-user-obj.c
 +F: hw/remote/iommu.c
 +F: include/hw/remote/iommu.h
  EBPF:
  M: Jason Wang <jasowang@redhat.com>
 diff --git a/include/hw/remote/iommu.h b/include/hw/remote/iommu.h
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/include/hw/remote/iommu.h
@@ -XXX,XX +XXX,XX @@
 +/**
 + * Copyright © 2022 Oracle and/or its affiliates.
 + *
 + * This work is licensed under the terms of the GNU GPL, version 2 or later.
 + * See the COPYING file in the top-level directory.
 + *
 + */
 +
 +#ifndef REMOTE_IOMMU_H
 +#define REMOTE_IOMMU_H
 +
 +#include "hw/pci/pci_bus.h"
 +#include "hw/pci/pci.h"
 +
 +#ifndef INT2VOIDP
 +#define INT2VOIDP(i) (void *)(uintptr_t)(i)
 +#endif
 +
 +typedef struct RemoteIommuElem {
 +    MemoryRegion *mr;
 +
 +    AddressSpace as;
 +} RemoteIommuElem;
 +
 +#define TYPE_REMOTE_IOMMU "x-remote-iommu"
 +OBJECT_DECLARE_SIMPLE_TYPE(RemoteIommu, REMOTE_IOMMU)
 +
 +struct RemoteIommu {
 +    Object parent;
 +
 +    GHashTable *elem_by_devfn;
 +
 +    QemuMutex lock;
 +};
 +
 +void remote_iommu_setup(PCIBus *pci_bus);
 +
 +void remote_iommu_unplug_dev(PCIDevice *pci_dev);
 +
 +#endif
 diff --git a/hw/remote/iommu.c b/hw/remote/iommu.c
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/hw/remote/iommu.c
@@ -XXX,XX +XXX,XX @@
 +/**
 + * IOMMU for remote device
 + *
 + * Copyright © 2022 Oracle and/or its affiliates.
 + *
 + * This work is licensed under the terms of the GNU GPL, version 2 or later.
 + * See the COPYING file in the top-level directory.
 + *
 + */
 +
 +#include "qemu/osdep.h"
 +
 +#include "hw/remote/iommu.h"
 +#include "hw/pci/pci_bus.h"
 +#include "hw/pci/pci.h"
 +#include "exec/memory.h"
 +#include "exec/address-spaces.h"
 +#include "trace.h"
 +
 +/**
 + * IOMMU for TYPE_REMOTE_MACHINE - manages DMA address space isolation
 + *     for remote machine. It is used by TYPE_VFIO_USER_SERVER.
 + *
 + * - Each TYPE_VFIO_USER_SERVER instance handles one PCIDevice on a PCIBus.
 + *   There is one RemoteIommu per PCIBus, so the RemoteIommu tracks multiple
 + *   PCIDevices by maintaining a ->elem_by_devfn mapping.
 + *
 + * - memory_region_init_iommu() is not used because vfio-user MemoryRegions
 + *   will be added to the elem->mr container instead. This is more natural
 + *   than implementing the IOMMUMemoryRegionClass APIs since vfio-user
 + *   provides something that is close to a full-fledged MemoryRegion and
 + *   not like an IOMMU mapping.
 + *
 + * - When a device is hot unplugged, the elem->mr reference is dropped so
 + *   all vfio-user MemoryRegions associated with this vfio-user server are
 + *   destroyed.
 + */
 +
 +static AddressSpace *remote_iommu_find_add_as(PCIBus *pci_bus,
 +                                              void *opaque, int devfn)
 +{
 +    RemoteIommu *iommu = opaque;
 +    RemoteIommuElem *elem = NULL;
 +
 +    qemu_mutex_lock(&iommu->lock);
 +
 +    elem = g_hash_table_lookup(iommu->elem_by_devfn, INT2VOIDP(devfn));
 +
 +    if (!elem) {
 +        elem = g_malloc0(sizeof(RemoteIommuElem));
 +        g_hash_table_insert(iommu->elem_by_devfn, INT2VOIDP(devfn), elem);
 +    }
 +
 +    if (!elem->mr) {
 +        elem->mr = MEMORY_REGION(object_new(TYPE_MEMORY_REGION));
 +        memory_region_set_size(elem->mr, UINT64_MAX);
 +        address_space_init(&elem->as, elem->mr, NULL);
 +    }
 +
 +    qemu_mutex_unlock(&iommu->lock);
 +
 +    return &elem->as;
 +}
 +
 +void remote_iommu_unplug_dev(PCIDevice *pci_dev)
 +{
 +    AddressSpace *as = pci_device_iommu_address_space(pci_dev);
 +    RemoteIommuElem *elem = NULL;
 +
 +    if (as == &address_space_memory) {
 +        return;
 +    }
 +
 +    elem = container_of(as, RemoteIommuElem, as);
 +
 +    address_space_destroy(&elem->as);
 +
 +    object_unref(elem->mr);
 +
 +    elem->mr = NULL;
 +}
 +
 +static void remote_iommu_init(Object *obj)
 +{
 +    RemoteIommu *iommu = REMOTE_IOMMU(obj);
 +
 +    iommu->elem_by_devfn = g_hash_table_new_full(NULL, NULL, NULL, g_free);
 +
 +    qemu_mutex_init(&iommu->lock);
 +}
 +
 +static void remote_iommu_finalize(Object *obj)
 +{
 +    RemoteIommu *iommu = REMOTE_IOMMU(obj);
 +
 +    qemu_mutex_destroy(&iommu->lock);
 +
 +    g_hash_table_destroy(iommu->elem_by_devfn);
 +
 +    iommu->elem_by_devfn = NULL;
 +}
 +
 +void remote_iommu_setup(PCIBus *pci_bus)
 +{
 +    RemoteIommu *iommu = NULL;
 +
 +    g_assert(pci_bus);
 +
 +    iommu = REMOTE_IOMMU(object_new(TYPE_REMOTE_IOMMU));
 +
 +    pci_setup_iommu(pci_bus, remote_iommu_find_add_as, iommu);
 +
 +    object_property_add_child(OBJECT(pci_bus), "remote-iommu", OBJECT(iommu));
 +
 +    object_unref(OBJECT(iommu));
 +}
 +
 +static const TypeInfo remote_iommu_info = {
 +    .name = TYPE_REMOTE_IOMMU,
 +    .parent = TYPE_OBJECT,
 +    .instance_size = sizeof(RemoteIommu),
 +    .instance_init = remote_iommu_init,
 +    .instance_finalize = remote_iommu_finalize,
 +};
 +
 +static void remote_iommu_register_types(void)
 +{
 +    type_register_static(&remote_iommu_info);
 +}
 +
 +type_init(remote_iommu_register_types)
 diff --git a/hw/remote/machine.c b/hw/remote/machine.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/machine.c
 +++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@
  #include "qapi/error.h"
  #include "hw/pci/pci_host.h"
  #include "hw/remote/iohub.h"
 +#include "hw/remote/iommu.h"
  #include "hw/qdev-core.h"
  static void remote_machine_init(MachineState *machine)
@@ -XXX,XX +XXX,XX @@ static void remote_machine_instance_init(Object *obj)
      s->auto_shutdown = true;
  }
 +static void remote_machine_dev_unplug_cb(HotplugHandler *hotplug_dev,
 +                                         DeviceState *dev, Error **errp)
 +{
 +    qdev_unrealize(dev);
 +
 +    if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
 +        remote_iommu_unplug_dev(PCI_DEVICE(dev));
 +    }
 +}
 +
  static void remote_machine_class_init(ObjectClass *oc, void *data)
  {
      MachineClass *mc = MACHINE_CLASS(oc);
@@ -XXX,XX +XXX,XX @@ static void remote_machine_class_init(ObjectClass *oc, void *data)
      mc->init = remote_machine_init;
      mc->desc = "Experimental remote machine";
 -    hc->unplug = qdev_simple_device_unplug_cb;
 +    hc->unplug = remote_machine_dev_unplug_cb;
      object_class_property_add_bool(oc, "vfio-user",
                                     remote_machine_get_vfio_user,
 diff --git a/hw/remote/meson.build b/hw/remote/meson.build
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/meson.build
 +++ b/hw/remote/meson.build
@@ -XXX,XX +XXX,XX @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
  remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
  remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
  remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
 +remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iommu.c'))
  remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: files('vfio-user-obj.c'))
  remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: libvfio_user_dep)
 --
 .36.1

-[Qemu-devel] [PULL 09/12] block: use bdrv_can_set_read_only() during reopen
+[PULL 13/18] vfio-user: handle DMA mappings
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+From: Jagannathan Raman <jag.raman@oracle.com>
 Define and register callbacks to manage the RAM regions used for
 device DMA
 Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
 Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Reviewed-by: John Snow <jsnow@redhat.com>
+Message-id: faacbcd45c4d02c591f0dbfdc19041fbb3eae7eb.1655151679.git.jag.raman@oracle.com
-Message-id: 00aed7ffdd7be4b9ed9ce1007d50028a72b34ebe.1491597120.git.jcody@redhat.com
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
- block.c | 14 ++++++++------
+ hw/remote/machine.c       |  5 ++++
-file changed, 8 insertions(+), 6 deletions(-)
+ hw/remote/vfio-user-obj.c | 55 +++++++++++++++++++++++++++++++++++++++
  hw/remote/trace-events    |  2 ++
 files changed, 62 insertions(+)
-diff --git a/block.c b/block.c
+diff --git a/hw/remote/machine.c b/hw/remote/machine.c
 index XXXXXXX..XXXXXXX 100644
---- a/block.c
+--- a/hw/remote/machine.c
-+++ b/block.c
++++ b/hw/remote/machine.c
-@@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
+@@ -XXX,XX +XXX,XX @@
-     BlockDriver *drv;
+ #include "hw/remote/iohub.h"
-     QemuOpts *opts;
+ #include "hw/remote/iommu.h"
-     const char *value;
+ #include "hw/qdev-core.h"
-+    bool read_only;
++#include "hw/remote/iommu.h"
-     assert(reopen_state != NULL);
+ static void remote_machine_init(MachineState *machine)
-     assert(reopen_state->bs->drv != NULL);
+ {
-@@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
+@@ -XXX,XX +XXX,XX @@ static void remote_machine_init(MachineState *machine)
-         qdict_put(reopen_state->options, "driver", qstring_from_str(value));
      pci_host = PCI_HOST_BRIDGE(rem_host);
 +    if (s->vfio_user) {
 +        remote_iommu_setup(pci_host->bus);
 +    }
 +
      remote_iohub_init(&s->iohub);
      pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
 diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/vfio-user-obj.c
 +++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
      return count;
  }
 +static void dma_register(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
 +{
 +    VfuObject *o = vfu_get_private(vfu_ctx);
 +    AddressSpace *dma_as = NULL;
 +    MemoryRegion *subregion = NULL;
 +    g_autofree char *name = NULL;
 +    struct iovec *iov = &info->iova;
 +
 +    if (!info->vaddr) {
 +        return;
 +    }
 +
 +    name = g_strdup_printf("mem-%s-%"PRIx64"", o->device,
 +                           (uint64_t)info->vaddr);
 +
 +    subregion = g_new0(MemoryRegion, 1);
 +
 +    memory_region_init_ram_ptr(subregion, NULL, name,
 +                               iov->iov_len, info->vaddr);
 +
 +    dma_as = pci_device_iommu_address_space(o->pci_dev);
 +
 +    memory_region_add_subregion(dma_as->root, (hwaddr)iov->iov_base, subregion);
 +
 +    trace_vfu_dma_register((uint64_t)iov->iov_base, iov->iov_len);
 +}
 +
 +static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
 +{
 +    VfuObject *o = vfu_get_private(vfu_ctx);
 +    AddressSpace *dma_as = NULL;
 +    MemoryRegion *mr = NULL;
 +    ram_addr_t offset;
 +
 +    mr = memory_region_from_host(info->vaddr, &offset);
 +    if (!mr) {
 +        return;
 +    }
 +
 +    dma_as = pci_device_iommu_address_space(o->pci_dev);
 +
 +    memory_region_del_subregion(dma_as->root, mr);
 +
 +    object_unparent((OBJECT(mr)));
 +
 +    trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
 +}
 +
  /*
   * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
   * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
          goto fail;
      }
--    /* if we are to stay read-only, do not allow permission change
++    ret = vfu_setup_device_dma(o->vfu_ctx, &dma_register, &dma_unregister);
--     * to r/w */
++    if (ret < 0) {
--    if (!(reopen_state->bs->open_flags & BDRV_O_ALLOW_RDWR) &&
++        error_setg(errp, "vfu: Failed to setup DMA handlers for %s",
--        reopen_state->flags & BDRV_O_RDWR) {
++                   o->device);
--        error_setg(errp, "Node '%s' is read only",
++        goto fail;
--                   bdrv_get_device_or_node_name(reopen_state->bs));
++    }
-+    /* If we are to stay read-only, do not allow permission change
++
-+     * to r/w. Attempting to set to r/w may fail if either BDRV_O_ALLOW_RDWR is
+     ret = vfu_realize_ctx(o->vfu_ctx);
-+     * not set, or if the BDS still has copy_on_read enabled */
+     if (ret < 0) {
-+    read_only = !(reopen_state->flags & BDRV_O_RDWR);
+         error_setg(errp, "vfu: Failed to realize device %s- %s",
-+    ret = bdrv_can_set_read_only(reopen_state->bs, read_only, &local_err);
+diff --git a/hw/remote/trace-events b/hw/remote/trace-events
-+    if (local_err) {
+index XXXXXXX..XXXXXXX 100644
-+        error_propagate(errp, local_err);
+--- a/hw/remote/trace-events
-         goto error;
++++ b/hw/remote/trace-events
-     }
+@@ -XXX,XX +XXX,XX @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
+ vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
  vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
  vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
 +vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
 +vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 02/12] block/vxhs.c: Add qemu-iotests for new block device type "vxhs"
+[PULL 14/18] vfio-user: handle PCI BAR accesses
-From: Ashish Mittal <ashmit602@gmail.com>
+From: Jagannathan Raman <jag.raman@oracle.com>
-These changes use a vxhs test server that is a part of the following
+Determine the BARs used by the PCI device and register handlers to
-repository:
+manage the access to the same.
 https://github.com/VeritasHyperScale/libqnio.git
-Signed-off-by: Ashish Mittal <Ashish.Mittal@veritas.com>
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
 Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Reviewed-by: Jeff Cody <jcody@redhat.com>
+Message-id: 3373e10b5be5f42846f0632d4382466e1698c505.1655151679.git.jag.raman@oracle.com
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: 1491277689-24949-3-git-send-email-Ashish.Mittal@veritas.com
 ---
- tests/qemu-iotests/common        |  6 ++++++
+ include/exec/memory.h           |   3 +
- tests/qemu-iotests/common.config | 13 +++++++++++++
+ hw/remote/vfio-user-obj.c       | 190 ++++++++++++++++++++++++++++++++
- tests/qemu-iotests/common.filter |  1 +
+ softmmu/physmem.c               |   4 +-
- tests/qemu-iotests/common.rc     | 19 +++++++++++++++++++
+ tests/qtest/fuzz/generic_fuzz.c |   9 +-
-files changed, 39 insertions(+)
+ hw/remote/trace-events          |   3 +
 files changed, 203 insertions(+), 6 deletions(-)
-diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
+diff --git a/include/exec/memory.h b/include/exec/memory.h
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/common
+--- a/include/exec/memory.h
-+++ b/tests/qemu-iotests/common
++++ b/include/exec/memory.h
-@@ -XXX,XX +XXX,XX @@ check options
+@@ -XXX,XX +XXX,XX @@ MemTxResult address_space_write_cached_slow(MemoryRegionCache *cache,
-     -ssh                test ssh
+                                             hwaddr addr, const void *buf,
-     -nfs                test nfs
+                                             hwaddr len);
-     -luks               test luks
-+    -vxhs               test vxhs
++int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr);
-     -xdiff              graphical mode diff
++bool prepare_mmio_access(MemoryRegion *mr);
-     -nocache            use O_DIRECT on backing file
++
-     -misalign           misalign memory allocations
+ static inline bool memory_access_is_direct(MemoryRegion *mr, bool is_write)
@@ -XXX,XX +XXX,XX @@ testlist options
              xpand=false
              ;;
 +        -vxhs)
 +            IMGPROTO=vxhs
 +            xpand=false
 +            ;;
 +
          -ssh)
              IMGPROTO=ssh
              xpand=false
 diff --git a/tests/qemu-iotests/common.config b/tests/qemu-iotests/common.config
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/common.config
 +++ b/tests/qemu-iotests/common.config
@@ -XXX,XX +XXX,XX @@ if [ -z "$QEMU_NBD_PROG" ]; then
      export QEMU_NBD_PROG="`set_prog_path qemu-nbd`"
  fi
 +if [ -z "$QEMU_VXHS_PROG" ]; then
 +    export QEMU_VXHS_PROG="`set_prog_path qnio_server`"
 +fi
 +
  _qemu_wrapper()
  {
-     (
+     if (is_write) {
-@@ -XXX,XX +XXX,XX @@ _qemu_nbd_wrapper()
+diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
-     )
+index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/vfio-user-obj.c
 +++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
  #include "hw/qdev-core.h"
  #include "hw/pci/pci.h"
  #include "qemu/timer.h"
 +#include "exec/memory.h"
  #define TYPE_VFU_OBJECT "x-vfio-user-server"
  OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
      trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
  }
-+_qemu_vxhs_wrapper()
++static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
 +                            hwaddr size, const bool is_write)
 +{
-+    (
++    uint8_t *ptr = buf;
-+        echo $BASHPID > "${TEST_DIR}/qemu-vxhs.pid"
++    bool release_lock = false;
-+        exec "$QEMU_VXHS_PROG" $QEMU_VXHS_OPTIONS "$@"
++    uint8_t *ram_ptr = NULL;
-+    )
++    MemTxResult result;
 +    int access_size;
 +    uint64_t val;
 +
 +    if (memory_access_is_direct(mr, is_write)) {
 +        /**
 +         * Some devices expose a PCI expansion ROM, which could be buffer
 +         * based as compared to other regions which are primarily based on
 +         * MemoryRegionOps. memory_region_find() would already check
 +         * for buffer overflow, we don't need to repeat it here.
 +         */
 +        ram_ptr = memory_region_get_ram_ptr(mr);
 +
 +        if (is_write) {
 +            memcpy((ram_ptr + offset), buf, size);
 +        } else {
 +            memcpy(buf, (ram_ptr + offset), size);
 +        }
 +
 +        return 0;
 +    }
 +
 +    while (size) {
 +        /**
 +         * The read/write logic used below is similar to the ones in
 +         * flatview_read/write_continue()
 +         */
 +        release_lock = prepare_mmio_access(mr);
 +
 +        access_size = memory_access_size(mr, size, offset);
 +
 +        if (is_write) {
 +            val = ldn_he_p(ptr, access_size);
 +
 +            result = memory_region_dispatch_write(mr, offset, val,
 +                                                  size_memop(access_size),
 +                                                  MEMTXATTRS_UNSPECIFIED);
 +        } else {
 +            result = memory_region_dispatch_read(mr, offset, &val,
 +                                                 size_memop(access_size),
 +                                                 MEMTXATTRS_UNSPECIFIED);
 +
 +            stn_he_p(ptr, access_size, val);
 +        }
 +
 +        if (release_lock) {
 +            qemu_mutex_unlock_iothread();
 +            release_lock = false;
 +        }
 +
 +        if (result != MEMTX_OK) {
 +            return -1;
 +        }
 +
 +        size -= access_size;
 +        ptr += access_size;
 +        offset += access_size;
 +    }
 +
 +    return 0;
 +}
 +
- export QEMU=_qemu_wrapper
++static size_t vfu_object_bar_rw(PCIDevice *pci_dev, int pci_bar,
- export QEMU_IMG=_qemu_img_wrapper
++                                hwaddr bar_offset, char * const buf,
- export QEMU_IO=_qemu_io_wrapper
++                                hwaddr len, const bool is_write)
- export QEMU_NBD=_qemu_nbd_wrapper
++{
-+export QEMU_VXHS=_qemu_vxhs_wrapper
++    MemoryRegionSection section = { 0 };
++    uint8_t *ptr = (uint8_t *)buf;
- QEMU_IMG_EXTRA_ARGS=
++    MemoryRegion *section_mr = NULL;
- if [ "$IMGOPTSSYNTAX" = "true" ]; then
++    uint64_t section_size;
-diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
++    hwaddr section_offset;
-index XXXXXXX..XXXXXXX 100644
++    hwaddr size = 0;
---- a/tests/qemu-iotests/common.filter
++
-+++ b/tests/qemu-iotests/common.filter
++    while (len) {
-@@ -XXX,XX +XXX,XX @@ _filter_img_info()
++        section = memory_region_find(pci_dev->io_regions[pci_bar].memory,
-         -e "s#$TEST_DIR#TEST_DIR#g" \
++                                     bar_offset, len);
-         -e "s#$IMGFMT#IMGFMT#g" \
++
-         -e 's#nbd://127.0.0.1:10810$#TEST_DIR/t.IMGFMT#g' \
++        if (!section.mr) {
-+        -e 's#json.*vdisk-id.*vxhs"}}#TEST_DIR/t.IMGFMT#' \
++            warn_report("vfu: invalid address 0x%"PRIx64"", bar_offset);
-         -e "/encrypted: yes/d" \
++            return size;
-         -e "/cluster_size: [0-9]\\+/d" \
++        }
-         -e "/table_size: [0-9]\\+/d" \
++
-diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
++        section_mr = section.mr;
-index XXXXXXX..XXXXXXX 100644
++        section_offset = section.offset_within_region;
---- a/tests/qemu-iotests/common.rc
++        section_size = int128_get64(section.size);
-+++ b/tests/qemu-iotests/common.rc
++
-@@ -XXX,XX +XXX,XX @@ else
++        if (is_write && section_mr->readonly) {
-     elif [ "$IMGPROTO" = "nfs" ]; then
++            warn_report("vfu: attempting to write to readonly region in "
-         TEST_DIR="nfs://127.0.0.1/$TEST_DIR"
++                        "bar %d - [0x%"PRIx64" - 0x%"PRIx64"]",
-         TEST_IMG=$TEST_DIR/t.$IMGFMT
++                        pci_bar, bar_offset,
-+    elif [ "$IMGPROTO" = "vxhs" ]; then
++                        (bar_offset + section_size));
-+        TEST_IMG_FILE=$TEST_DIR/t.$IMGFMT
++            memory_region_unref(section_mr);
-+        TEST_IMG="vxhs://127.0.0.1:9999/t.$IMGFMT"
++            return size;
-     else
++        }
-         TEST_IMG=$IMGPROTO:$TEST_DIR/t.$IMGFMT
++
-     fi
++        if (vfu_object_mr_rw(section_mr, ptr, section_offset,
-@@ -XXX,XX +XXX,XX @@ _make_test_img()
++                             section_size, is_write)) {
-         eval "$QEMU_NBD -v -t -b 127.0.0.1 -p 10810 -f $IMGFMT  $TEST_IMG_FILE >/dev/null &"
++            warn_report("vfu: failed to %s "
-         sleep 1 # FIXME: qemu-nbd needs to be listening before we continue
++                        "[0x%"PRIx64" - 0x%"PRIx64"] in bar %d",
-     fi
++                        is_write ? "write to" : "read from", bar_offset,
-+
++                        (bar_offset + section_size), pci_bar);
-+    # Start QNIO server on image directory for vxhs protocol
++            memory_region_unref(section_mr);
-+    if [ $IMGPROTO = "vxhs" ]; then
++            return size;
-+        eval "$QEMU_VXHS -d  $TEST_DIR > /dev/null &"
++        }
-+        sleep 1 # Wait for server to come up.
++
-+    fi
++        size += section_size;
 +        bar_offset += section_size;
 +        ptr += section_size;
 +        len -= section_size;
 +
 +        memory_region_unref(section_mr);
 +    }
 +
 +    return size;
 +}
 +
 +/**
 + * VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs.
 + *
 + * To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would
 + * define vfu_object_bar2_handler
 + */
 +#define VFU_OBJECT_BAR_HANDLER(BAR_NO)                                         \
 +    static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx,        \
 +                                        char * const buf, size_t count,        \
 +                                        loff_t offset, const bool is_write)    \
 +    {                                                                          \
 +        VfuObject *o = vfu_get_private(vfu_ctx);                               \
 +        PCIDevice *pci_dev = o->pci_dev;                                       \
 +                                                                               \
 +        return vfu_object_bar_rw(pci_dev, BAR_NO, offset,                      \
 +                                 buf, count, is_write);                        \
 +    }                                                                          \
 +
 +VFU_OBJECT_BAR_HANDLER(0)
 +VFU_OBJECT_BAR_HANDLER(1)
 +VFU_OBJECT_BAR_HANDLER(2)
 +VFU_OBJECT_BAR_HANDLER(3)
 +VFU_OBJECT_BAR_HANDLER(4)
 +VFU_OBJECT_BAR_HANDLER(5)
 +VFU_OBJECT_BAR_HANDLER(6)
 +
 +static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS] = {
 +    &vfu_object_bar0_handler,
 +    &vfu_object_bar1_handler,
 +    &vfu_object_bar2_handler,
 +    &vfu_object_bar3_handler,
 +    &vfu_object_bar4_handler,
 +    &vfu_object_bar5_handler,
 +    &vfu_object_bar6_handler,
 +};
 +
 +/**
 + * vfu_object_register_bars - Identify active BAR regions of pdev and setup
 + *                            callbacks to handle read/write accesses
 + */
 +static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
 +{
 +    int flags = VFU_REGION_FLAG_RW;
 +    int i;
 +
 +    for (i = 0; i < PCI_NUM_REGIONS; i++) {
 +        if (!pdev->io_regions[i].size) {
 +            continue;
 +        }
 +
 +        if ((i == VFU_PCI_DEV_ROM_REGION_IDX) ||
 +            pdev->io_regions[i].memory->readonly) {
 +            flags &= ~VFU_REGION_FLAG_WRITE;
 +        }
 +
 +        vfu_setup_region(vfu_ctx, VFU_PCI_DEV_BAR0_REGION_IDX + i,
 +                         (size_t)pdev->io_regions[i].size,
 +                         vfu_object_bar_handlers[i],
 +                         flags, NULL, 0, -1, 0);
 +
 +        trace_vfu_bar_register(i, pdev->io_regions[i].addr,
 +                               pdev->io_regions[i].size);
 +    }
 +}
 +
  /*
   * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
   * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
          goto fail;
      }
 +    vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
 +
      ret = vfu_realize_ctx(o->vfu_ctx);
      if (ret < 0) {
          error_setg(errp, "vfu: Failed to realize device %s- %s",
 diff --git a/softmmu/physmem.c b/softmmu/physmem.c
 index XXXXXXX..XXXXXXX 100644
 --- a/softmmu/physmem.c
 +++ b/softmmu/physmem.c
@@ -XXX,XX +XXX,XX @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)
      invalidate_and_set_dirty(mr, addr, size);
  }
- _rm_test_img()
+-static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
-@@ -XXX,XX +XXX,XX @@ _cleanup_test_img()
++int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
-             fi
+ {
-             rm -f "$TEST_IMG_FILE"
+     unsigned access_size_max = mr->ops->valid.max_access_size;
-             ;;
-+        vxhs)
+@@ -XXX,XX +XXX,XX @@ static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
-+            if [ -f "${TEST_DIR}/qemu-vxhs.pid" ]; then
+     return l;
-+                local QEMU_VXHS_PID
+ }
-+                read QEMU_VXHS_PID < "${TEST_DIR}/qemu-vxhs.pid"
-+                kill ${QEMU_VXHS_PID} >/dev/null 2>&1
+-static bool prepare_mmio_access(MemoryRegion *mr)
-+                rm -f "${TEST_DIR}/qemu-vxhs.pid"
++bool prepare_mmio_access(MemoryRegion *mr)
-+            fi
+ {
-+            rm -f "$TEST_IMG_FILE"
+     bool release_lock = false;
-+            ;;
-+
+diff --git a/tests/qtest/fuzz/generic_fuzz.c b/tests/qtest/fuzz/generic_fuzz.c
-         file)
+index XXXXXXX..XXXXXXX 100644
-             _rm_test_img "$TEST_DIR/t.$IMGFMT"
+--- a/tests/qtest/fuzz/generic_fuzz.c
-             _rm_test_img "$TEST_DIR/t.$IMGFMT.orig"
++++ b/tests/qtest/fuzz/generic_fuzz.c
@@ -XXX,XX +XXX,XX @@ static void *pattern_alloc(pattern p, size_t len)
      return buf;
  }
 -static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
 +static int fuzz_memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
  {
      unsigned access_size_max = mr->ops->valid.max_access_size;
@@ -XXX,XX +XXX,XX @@ void fuzz_dma_read_cb(size_t addr, size_t len, MemoryRegion *mr)
          /*
           *  If mr1 isn't RAM, address_space_translate doesn't update l. Use
 -         *  memory_access_size to identify the number of bytes that it is safe
 -         *  to write without accidentally writing to another MemoryRegion.
 +         *  fuzz_memory_access_size to identify the number of bytes that it
 +         *  is safe to write without accidentally writing to another
 +         *  MemoryRegion.
           */
          if (!memory_region_is_ram(mr1)) {
 -            l = memory_access_size(mr1, l, addr1);
 +            l = fuzz_memory_access_size(mr1, l, addr1);
          }
          if (memory_region_is_ram(mr1) ||
              memory_region_is_romd(mr1) ||
 diff --git a/hw/remote/trace-events b/hw/remote/trace-events
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/trace-events
 +++ b/hw/remote/trace-events
@@ -XXX,XX +XXX,XX @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
  vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
  vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
  vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
 +vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""
 +vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
 +vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 05/12] block: do not set BDS read_only if copy_on_read enabled
+[PULL 15/18] vfio-user: handle device interrupts
-A few block drivers will set the BDS read_only flag from their
+From: Jagannathan Raman <jag.raman@oracle.com>
 .bdrv_open() function.  This means the bs->read_only flag could
 be set after we enable copy_on_read, as the BDRV_O_COPY_ON_READ
 flag check occurs prior to the call to bdrv->bdrv_open().
-This adds an error return to bdrv_set_read_only(), and an error will be
+Forward remote device's interrupts to the guest
 return if we try to set the BDS to read_only while copy_on_read is
 enabled.
-This patch also changes the behavior of vvfat.  Before, vvfat could
+Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
-override the drive 'readonly' flag with its own, internal 'rw' flag.
+Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Message-id: 9523479eaafe050677f4de2af5dd0df18c27cfd9.1655151679.git.jag.raman@oracle.com
 Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
  MAINTAINERS                       |   1 +
  include/hw/pci/msi.h              |   1 +
  include/hw/pci/msix.h             |   1 +
  include/hw/pci/pci.h              |  13 +++
  include/hw/remote/vfio-user-obj.h |   6 ++
  hw/pci/msi.c                      |  49 +++++++--
  hw/pci/msix.c                     |  35 ++++++-
  hw/pci/pci.c                      |  13 +++
  hw/remote/machine.c               |  16 ++-
  hw/remote/vfio-user-obj.c         | 167 ++++++++++++++++++++++++++++++
  stubs/vfio-user-obj.c             |   6 ++
  hw/remote/trace-events            |   1 +
  stubs/meson.build                 |   1 +
 files changed, 298 insertions(+), 12 deletions(-)
  create mode 100644 include/hw/remote/vfio-user-obj.h
  create mode 100644 stubs/vfio-user-obj.c
-For instance, this -drive parameter would result in a writable image:
+diff --git a/MAINTAINERS b/MAINTAINERS
+index XXXXXXX..XXXXXXX 100644
-"-drive format=vvfat,dir=/tmp/vvfat,rw,if=virtio,readonly=on"
+--- a/MAINTAINERS
++++ b/MAINTAINERS
-This is not correct.  Now, attempting to use the above -drive parameter
+@@ -XXX,XX +XXX,XX @@ F: hw/remote/iohub.c
-will result in an error (i.e., 'rw' is incompatible with 'readonly=on').
+ F: include/hw/remote/iohub.h
+ F: subprojects/libvfio-user
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+ F: hw/remote/vfio-user-obj.c
-Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
++F: include/hw/remote/vfio-user-obj.h
-Reviewed-by: John Snow <jsnow@redhat.com>
+ F: hw/remote/iommu.c
-Message-id: 0c5b4c1cc2c651471b131f21376dfd5ea24d2196.1491597120.git.jcody@redhat.com
+ F: include/hw/remote/iommu.h
----
- block.c               | 10 +++++++++-
+diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
- block/bochs.c         |  5 ++++-
+index XXXXXXX..XXXXXXX 100644
- block/cloop.c         |  5 ++++-
+--- a/include/hw/pci/msi.h
- block/dmg.c           |  6 +++++-
++++ b/include/hw/pci/msi.h
- block/rbd.c           | 11 ++++++++++-
+@@ -XXX,XX +XXX,XX @@ void msi_notify(PCIDevice *dev, unsigned int vector);
- block/vvfat.c         | 19 +++++++++++++++----
+ void msi_send_message(PCIDevice *dev, MSIMessage msg);
- include/block/block.h |  2 +-
+ void msi_write_config(PCIDevice *dev, uint32_t addr, uint32_t val, int len);
-files changed, 48 insertions(+), 10 deletions(-)
+ unsigned int msi_nr_vectors_allocated(const PCIDevice *dev);
++void msi_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp);
-diff --git a/block.c b/block.c
-index XXXXXXX..XXXXXXX 100644
+ static inline bool msi_present(const PCIDevice *dev)
---- a/block.c
+ {
-+++ b/block.c
+diff --git a/include/hw/pci/msix.h b/include/hw/pci/msix.h
-@@ -XXX,XX +XXX,XX @@ void path_combine(char *dest, int dest_size,
+index XXXXXXX..XXXXXXX 100644
-     }
+--- a/include/hw/pci/msix.h
- }
++++ b/include/hw/pci/msix.h
+@@ -XXX,XX +XXX,XX @@ void msix_clr_pending(PCIDevice *dev, int vector);
--void bdrv_set_read_only(BlockDriverState *bs, bool read_only)
+ int msix_vector_use(PCIDevice *dev, unsigned vector);
-+int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
+ void msix_vector_unuse(PCIDevice *dev, unsigned vector);
- {
+ void msix_unuse_all_vectors(PCIDevice *dev);
-+    /* Do not set read_only if copy_on_read is enabled */
++void msix_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp);
-+    if (bs->copy_on_read && read_only) {
-+        error_setg(errp, "Can't set node '%s' to r/o with copy-on-read enabled",
+ void msix_notify(PCIDevice *dev, unsigned vector);
-+                   bdrv_get_device_or_node_name(bs));
-+        return -EINVAL;
+diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
-+    }
+index XXXXXXX..XXXXXXX 100644
-+
+--- a/include/hw/pci/pci.h
-     bs->read_only = read_only;
++++ b/include/hw/pci/pci.h
-+    return 0;
+@@ -XXX,XX +XXX,XX @@ extern bool pci_available;
- }
+ #define PCI_SLOT(devfn)         (((devfn) >> 3) & 0x1f)
+ #define PCI_FUNC(devfn)         ((devfn) & 0x07)
- void bdrv_get_full_backing_filename_from_filename(const char *backed,
+ #define PCI_BUILD_BDF(bus, devfn)     ((bus << 8) | (devfn))
-diff --git a/block/bochs.c b/block/bochs.c
++#define PCI_BDF_TO_DEVFN(x)     ((x) & 0xff)
-index XXXXXXX..XXXXXXX 100644
+ #define PCI_BUS_MAX             256
---- a/block/bochs.c
+ #define PCI_DEVFN_MAX           256
-+++ b/block/bochs.c
+ #define PCI_SLOT_MAX            32
-@@ -XXX,XX +XXX,XX @@ static int bochs_open(BlockDriverState *bs, QDict *options, int flags,
+@@ -XXX,XX +XXX,XX @@ typedef void PCIMapIORegionFunc(PCIDevice *pci_dev, int region_num,
-         return -EINVAL;
+                                 pcibus_t addr, pcibus_t size, int type);
-     }
+ typedef void PCIUnregisterFunc(PCIDevice *pci_dev);
--    bdrv_set_read_only(bs, true); /* no write support yet */
++typedef void MSITriggerFunc(PCIDevice *dev, MSIMessage msg);
-+    ret = bdrv_set_read_only(bs, true, errp); /* no write support yet */
++typedef MSIMessage MSIPrepareMessageFunc(PCIDevice *dev, unsigned vector);
 +typedef MSIMessage MSIxPrepareMessageFunc(PCIDevice *dev, unsigned vector);
 +
  typedef struct PCIIORegion {
      pcibus_t addr; /* current PCI mapping address. -1 means not mapped */
  #define PCI_BAR_UNMAPPED (~(pcibus_t)0)
@@ -XXX,XX +XXX,XX @@ struct PCIDevice {
      /* Space to store MSIX table & pending bit array */
      uint8_t *msix_table;
      uint8_t *msix_pba;
 +
 +    /* May be used by INTx or MSI during interrupt notification */
 +    void *irq_opaque;
 +
 +    MSITriggerFunc *msi_trigger;
 +    MSIPrepareMessageFunc *msi_prepare_message;
 +    MSIxPrepareMessageFunc *msix_prepare_message;
 +
      /* MemoryRegion container for msix exclusive BAR setup */
      MemoryRegion msix_exclusive_bar;
      /* Memory Regions for MSIX table and pending bit entries. */
 diff --git a/include/hw/remote/vfio-user-obj.h b/include/hw/remote/vfio-user-obj.h
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/include/hw/remote/vfio-user-obj.h
@@ -XXX,XX +XXX,XX @@
 +#ifndef VFIO_USER_OBJ_H
 +#define VFIO_USER_OBJ_H
 +
 +void vfu_object_set_bus_irq(PCIBus *pci_bus);
 +
 +#endif
 diff --git a/hw/pci/msi.c b/hw/pci/msi.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/pci/msi.c
 +++ b/hw/pci/msi.c
@@ -XXX,XX +XXX,XX @@ void msi_set_message(PCIDevice *dev, MSIMessage msg)
      pci_set_word(dev->config + msi_data_off(dev, msi64bit), msg.data);
  }
 -MSIMessage msi_get_message(PCIDevice *dev, unsigned int vector)
 +static MSIMessage msi_prepare_message(PCIDevice *dev, unsigned int vector)
  {
      uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
      bool msi64bit = flags & PCI_MSI_FLAGS_64BIT;
@@ -XXX,XX +XXX,XX @@ MSIMessage msi_get_message(PCIDevice *dev, unsigned int vector)
      return msg;
  }
 +MSIMessage msi_get_message(PCIDevice *dev, unsigned int vector)
 +{
 +    return dev->msi_prepare_message(dev, vector);
 +}
 +
  bool msi_enabled(const PCIDevice *dev)
  {
      return msi_present(dev) &&
@@ -XXX,XX +XXX,XX @@ int msi_init(struct PCIDevice *dev, uint8_t offset,
 xffffffff >> (PCI_MSI_VECTORS_MAX - nr_vectors));
      }
 +    dev->msi_prepare_message = msi_prepare_message;
 +
      return 0;
  }
@@ -XXX,XX +XXX,XX @@ void msi_uninit(struct PCIDevice *dev)
      cap_size = msi_cap_sizeof(flags);
      pci_del_capability(dev, PCI_CAP_ID_MSI, cap_size);
      dev->cap_present &= ~QEMU_PCI_CAP_MSI;
 +    dev->msi_prepare_message = NULL;
      MSI_DEV_PRINTF(dev, "uninit\n");
  }
@@ -XXX,XX +XXX,XX @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)
      return mask & (1U << vector);
  }
 +void msi_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp)
 +{
 +    ERRP_GUARD();
 +    uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
 +    bool msi64bit = flags & PCI_MSI_FLAGS_64BIT;
 +    uint32_t irq_state, vector_mask, pending;
 +
 +    if (vector > PCI_MSI_VECTORS_MAX) {
 +        error_setg(errp, "msi: vector %d not allocated. max vector is %d",
 +                   vector, PCI_MSI_VECTORS_MAX);
 +        return;
 +    }
 +
 +    vector_mask = (1U << vector);
 +
 +    irq_state = pci_get_long(dev->config + msi_mask_off(dev, msi64bit));
 +
 +    if (mask) {
 +        irq_state |= vector_mask;
 +    } else {
 +        irq_state &= ~vector_mask;
 +    }
 +
 +    pci_set_long(dev->config + msi_mask_off(dev, msi64bit), irq_state);
 +
 +    pending = pci_get_long(dev->config + msi_pending_off(dev, msi64bit));
 +    if (!mask && (pending & vector_mask)) {
 +        pending &= ~vector_mask;
 +        pci_set_long(dev->config + msi_pending_off(dev, msi64bit), pending);
 +        msi_notify(dev, vector);
 +    }
 +}
 +
  void msi_notify(PCIDevice *dev, unsigned int vector)
  {
      uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
@@ -XXX,XX +XXX,XX @@ void msi_notify(PCIDevice *dev, unsigned int vector)
  void msi_send_message(PCIDevice *dev, MSIMessage msg)
  {
 -    MemTxAttrs attrs = {};
 -
 -    attrs.requester_id = pci_requester_id(dev);
 -    address_space_stl_le(&dev->bus_master_as, msg.address, msg.data,
 -                         attrs, NULL);
 +    dev->msi_trigger(dev, msg);
  }
  /* Normally called by pci_default_write_config(). */
 diff --git a/hw/pci/msix.c b/hw/pci/msix.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/pci/msix.c
 +++ b/hw/pci/msix.c
@@ -XXX,XX +XXX,XX @@
  #define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)
  #define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8)
 -MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
 +static MSIMessage msix_prepare_message(PCIDevice *dev, unsigned vector)
  {
      uint8_t *table_entry = dev->msix_table + vector * PCI_MSIX_ENTRY_SIZE;
      MSIMessage msg;
@@ -XXX,XX +XXX,XX @@ MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
      return msg;
  }
 +MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
 +{
 +    return dev->msix_prepare_message(dev, vector);
 +}
 +
  /*
   * Special API for POWER to configure the vectors through
   * a side channel. Should never be used by devices.
@@ -XXX,XX +XXX,XX @@ static void msix_handle_mask_update(PCIDevice *dev, int vector, bool was_masked)
      }
  }
 +void msix_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp)
 +{
 +    ERRP_GUARD();
 +    unsigned offset;
 +    bool was_masked;
 +
 +    if (vector > dev->msix_entries_nr) {
 +        error_setg(errp, "msix: vector %d not allocated. max vector is %d",
 +                   vector, dev->msix_entries_nr);
 +        return;
 +    }
 +
 +    offset = vector * PCI_MSIX_ENTRY_SIZE + PCI_MSIX_ENTRY_VECTOR_CTRL;
 +
 +    was_masked = msix_is_masked(dev, vector);
 +
 +    if (mask) {
 +        dev->msix_table[offset] |= PCI_MSIX_ENTRY_CTRL_MASKBIT;
 +    } else {
 +        dev->msix_table[offset] &= ~PCI_MSIX_ENTRY_CTRL_MASKBIT;
 +    }
 +
 +    msix_handle_mask_update(dev, vector, was_masked);
 +}
 +
  static bool msix_masked(PCIDevice *dev)
  {
      return dev->config[dev->msix_cap + MSIX_CONTROL_OFFSET] & MSIX_MASKALL_MASK;
@@ -XXX,XX +XXX,XX @@ int msix_init(struct PCIDevice *dev, unsigned short nentries,
                            "msix-pba", pba_size);
      memory_region_add_subregion(pba_bar, pba_offset, &dev->msix_pba_mmio);
 +    dev->msix_prepare_message = msix_prepare_message;
 +
      return 0;
  }
@@ -XXX,XX +XXX,XX @@ void msix_uninit(PCIDevice *dev, MemoryRegion *table_bar, MemoryRegion *pba_bar)
      g_free(dev->msix_entry_used);
      dev->msix_entry_used = NULL;
      dev->cap_present &= ~QEMU_PCI_CAP_MSIX;
 +    dev->msix_prepare_message = NULL;
  }
  void msix_uninit_exclusive_bar(PCIDevice *dev)
 diff --git a/hw/pci/pci.c b/hw/pci/pci.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/pci/pci.c
 +++ b/hw/pci/pci.c
@@ -XXX,XX +XXX,XX @@ void pci_device_deassert_intx(PCIDevice *dev)
      }
  }
 +static void pci_msi_trigger(PCIDevice *dev, MSIMessage msg)
 +{
 +    MemTxAttrs attrs = {};
 +
 +    attrs.requester_id = pci_requester_id(dev);
 +    address_space_stl_le(&dev->bus_master_as, msg.address, msg.data,
 +                         attrs, NULL);
 +}
 +
  static void pci_reset_regions(PCIDevice *dev)
  {
      int r;
@@ -XXX,XX +XXX,XX @@ static void pci_qdev_unrealize(DeviceState *dev)
      pci_device_deassert_intx(pci_dev);
      do_pci_unregister_device(pci_dev);
 +
 +    pci_dev->msi_trigger = NULL;
  }
  void pci_register_bar(PCIDevice *pci_dev, int region_num,
@@ -XXX,XX +XXX,XX @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp)
      }
      pci_set_power(pci_dev, true);
 +
 +    pci_dev->msi_trigger = pci_msi_trigger;
  }
  PCIDevice *pci_new_multifunction(int devfn, bool multifunction,
 diff --git a/hw/remote/machine.c b/hw/remote/machine.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/machine.c
 +++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@
  #include "hw/remote/iommu.h"
  #include "hw/qdev-core.h"
  #include "hw/remote/iommu.h"
 +#include "hw/remote/vfio-user-obj.h"
 +#include "hw/pci/msi.h"
  static void remote_machine_init(MachineState *machine)
  {
@@ -XXX,XX +XXX,XX @@ static void remote_machine_init(MachineState *machine)
      if (s->vfio_user) {
          remote_iommu_setup(pci_host->bus);
 +
 +        msi_nonbroken = true;
 +
 +        vfu_object_set_bus_irq(pci_host->bus);
 +    } else {
 +        remote_iohub_init(&s->iohub);
 +
 +        pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
 +                     &s->iohub, REMOTE_IOHUB_NB_PIRQS);
      }
 -    remote_iohub_init(&s->iohub);
 -
 -    pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
 -                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
 -
      qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
  }
 diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 index XXXXXXX..XXXXXXX 100644
 --- a/hw/remote/vfio-user-obj.c
 +++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
  #include "hw/pci/pci.h"
  #include "qemu/timer.h"
  #include "exec/memory.h"
 +#include "hw/pci/msi.h"
 +#include "hw/pci/msix.h"
 +#include "hw/remote/vfio-user-obj.h"
  #define TYPE_VFU_OBJECT "x-vfio-user-server"
  OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ struct VfuObject {
      Error *unplug_blocker;
      int vfu_poll_fd;
 +
 +    MSITriggerFunc *default_msi_trigger;
 +    MSIPrepareMessageFunc *default_msi_prepare_message;
 +    MSIxPrepareMessageFunc *default_msix_prepare_message;
  };
  static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -XXX,XX +XXX,XX @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
      }
  }
 +static int vfu_object_map_irq(PCIDevice *pci_dev, int intx)
 +{
 +    int pci_bdf = PCI_BUILD_BDF(pci_bus_num(pci_get_bus(pci_dev)),
 +                                pci_dev->devfn);
 +
 +    return pci_bdf;
 +}
 +
 +static void vfu_object_set_irq(void *opaque, int pirq, int level)
 +{
 +    PCIBus *pci_bus = opaque;
 +    PCIDevice *pci_dev = NULL;
 +    vfu_ctx_t *vfu_ctx = NULL;
 +    int pci_bus_num, devfn;
 +
 +    if (level) {
 +        pci_bus_num = PCI_BUS_NUM(pirq);
 +        devfn = PCI_BDF_TO_DEVFN(pirq);
 +
 +        /*
 +         * pci_find_device() performs at O(1) if the device is attached
 +         * to the root PCI bus. Whereas, if the device is attached to a
 +         * secondary PCI bus (such as when a root port is involved),
 +         * finding the parent PCI bus could take O(n)
 +         */
 +        pci_dev = pci_find_device(pci_bus, pci_bus_num, devfn);
 +
 +        vfu_ctx = pci_dev->irq_opaque;
 +
 +        g_assert(vfu_ctx);
 +
 +        vfu_irq_trigger(vfu_ctx, 0);
 +    }
 +}
 +
 +static MSIMessage vfu_object_msi_prepare_msg(PCIDevice *pci_dev,
 +                                             unsigned int vector)
 +{
 +    MSIMessage msg;
 +
 +    msg.address = 0;
 +    msg.data = vector;
 +
 +    return msg;
 +}
 +
 +static void vfu_object_msi_trigger(PCIDevice *pci_dev, MSIMessage msg)
 +{
 +    vfu_ctx_t *vfu_ctx = pci_dev->irq_opaque;
 +
 +    vfu_irq_trigger(vfu_ctx, msg.data);
 +}
 +
 +static void vfu_object_setup_msi_cbs(VfuObject *o)
 +{
 +    o->default_msi_trigger = o->pci_dev->msi_trigger;
 +    o->default_msi_prepare_message = o->pci_dev->msi_prepare_message;
 +    o->default_msix_prepare_message = o->pci_dev->msix_prepare_message;
 +
 +    o->pci_dev->msi_trigger = vfu_object_msi_trigger;
 +    o->pci_dev->msi_prepare_message = vfu_object_msi_prepare_msg;
 +    o->pci_dev->msix_prepare_message = vfu_object_msi_prepare_msg;
 +}
 +
 +static void vfu_object_restore_msi_cbs(VfuObject *o)
 +{
 +    o->pci_dev->msi_trigger = o->default_msi_trigger;
 +    o->pci_dev->msi_prepare_message = o->default_msi_prepare_message;
 +    o->pci_dev->msix_prepare_message = o->default_msix_prepare_message;
 +}
 +
 +static void vfu_msix_irq_state(vfu_ctx_t *vfu_ctx, uint32_t start,
 +                               uint32_t count, bool mask)
 +{
 +    VfuObject *o = vfu_get_private(vfu_ctx);
 +    Error *err = NULL;
 +    uint32_t vector;
 +
 +    for (vector = start; vector < count; vector++) {
 +        msix_set_mask(o->pci_dev, vector, mask, &err);
 +        if (err) {
 +            VFU_OBJECT_ERROR(o, "vfu: %s: %s", o->device,
 +                             error_get_pretty(err));
 +            error_free(err);
 +            err = NULL;
 +        }
 +    }
 +}
 +
 +static void vfu_msi_irq_state(vfu_ctx_t *vfu_ctx, uint32_t start,
 +                              uint32_t count, bool mask)
 +{
 +    VfuObject *o = vfu_get_private(vfu_ctx);
 +    Error *err = NULL;
 +    uint32_t vector;
 +
 +    for (vector = start; vector < count; vector++) {
 +        msi_set_mask(o->pci_dev, vector, mask, &err);
 +        if (err) {
 +            VFU_OBJECT_ERROR(o, "vfu: %s: %s", o->device,
 +                             error_get_pretty(err));
 +            error_free(err);
 +            err = NULL;
 +        }
 +    }
 +}
 +
 +static int vfu_object_setup_irqs(VfuObject *o, PCIDevice *pci_dev)
 +{
 +    vfu_ctx_t *vfu_ctx = o->vfu_ctx;
 +    int ret;
 +
 +    ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_INTX_IRQ, 1);
 +    if (ret < 0) {
 +        return ret;
 +    }
++
-     ret = bdrv_pread(bs->file, 0, &bochs, sizeof(bochs));
++    if (msix_nr_vectors_allocated(pci_dev)) {
-     if (ret < 0) {
++        ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_MSIX_IRQ,
-diff --git a/block/cloop.c b/block/cloop.c
++                                       msix_nr_vectors_allocated(pci_dev));
-index XXXXXXX..XXXXXXX 100644
++        vfu_setup_irq_state_callback(vfu_ctx, VFU_DEV_MSIX_IRQ,
---- a/block/cloop.c
++                                     &vfu_msix_irq_state);
-+++ b/block/cloop.c
++    } else if (msi_nr_vectors_allocated(pci_dev)) {
-@@ -XXX,XX +XXX,XX @@ static int cloop_open(BlockDriverState *bs, QDict *options, int flags,
++        ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_MSI_IRQ,
-         return -EINVAL;
++                                       msi_nr_vectors_allocated(pci_dev));
-     }
++        vfu_setup_irq_state_callback(vfu_ctx, VFU_DEV_MSI_IRQ,
++                                     &vfu_msi_irq_state);
--    bdrv_set_read_only(bs, true);
++    }
-+    ret = bdrv_set_read_only(bs, true, errp);
++
 +    if (ret < 0) {
 +        return ret;
 +    }
++
-     /* read header */
++    vfu_object_setup_msi_cbs(o);
-     ret = bdrv_pread(bs->file, 128, &s->block_size, 4);
++
-diff --git a/block/dmg.c b/block/dmg.c
++    pci_dev->irq_opaque = vfu_ctx;
-index XXXXXXX..XXXXXXX 100644
++
---- a/block/dmg.c
++    return 0;
-+++ b/block/dmg.c
++}
-@@ -XXX,XX +XXX,XX @@ static int dmg_open(BlockDriverState *bs, QDict *options, int flags,
++
-         return -EINVAL;
++void vfu_object_set_bus_irq(PCIBus *pci_bus)
-     }
++{
++    int bus_num = pci_bus_num(pci_bus);
-+    ret = bdrv_set_read_only(bs, true, errp);
++    int max_bdf = PCI_BUILD_BDF(bus_num, PCI_DEVFN_MAX - 1);
 +
 +    pci_bus_irqs(pci_bus, vfu_object_set_irq, vfu_object_map_irq, pci_bus,
 +                 max_bdf);
 +}
 +
  /*
   * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
   * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
      vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
 +    ret = vfu_object_setup_irqs(o, o->pci_dev);
 +    if (ret < 0) {
-+        return ret;
++        error_setg(errp, "vfu: Failed to setup interrupts for %s",
-+    }
++                   o->device);
-+
++        goto fail;
-     block_module_load_one("dmg-bz2");
++    }
--    bdrv_set_read_only(bs, true);
++
+     ret = vfu_realize_ctx(o->vfu_ctx);
-     s->n_chunks = 0;
+     if (ret < 0) {
-     s->offsets = s->lengths = s->sectors = s->sectorcounts = NULL;
+         error_setg(errp, "vfu: Failed to realize device %s- %s",
-diff --git a/block/rbd.c b/block/rbd.c
+@@ -XXX,XX +XXX,XX @@ fail:
-index XXXXXXX..XXXXXXX 100644
+         o->unplug_blocker = NULL;
---- a/block/rbd.c
+     }
-+++ b/block/rbd.c
+     if (o->pci_dev) {
-@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
++        vfu_object_restore_msi_cbs(o);
-         goto failed_shutdown;
++        o->pci_dev->irq_opaque = NULL;
-     }
+         object_unref(OBJECT(o->pci_dev));
+         o->pci_dev = NULL;
-+    /* rbd_open is always r/w */
+     }
-     r = rbd_open(s->io_ctx, s->name, &s->image, s->snap);
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
-     if (r < 0) {
+     }
-         error_setg_errno(errp, -r, "error reading header from %s", s->name);
-         goto failed_open;
+     if (o->pci_dev) {
-     }
++        vfu_object_restore_msi_cbs(o);
++        o->pci_dev->irq_opaque = NULL;
--    bdrv_set_read_only(bs, (s->snap != NULL));
+         object_unref(OBJECT(o->pci_dev));
-+    /* If we are using an rbd snapshot, we must be r/o, otherwise
+         o->pci_dev = NULL;
-+     * leave as-is */
+     }
-+    if (s->snap != NULL) {
+diff --git a/stubs/vfio-user-obj.c b/stubs/vfio-user-obj.c
-+        r = bdrv_set_read_only(bs, true, &local_err);
+new file mode 100644
-+        if (r < 0) {
+index XXXXXXX..XXXXXXX
-+            error_propagate(errp, local_err);
+--- /dev/null
-+            goto failed_open;
++++ b/stubs/vfio-user-obj.c
-+        }
+@@ -XXX,XX +XXX,XX @@
-+    }
++#include "qemu/osdep.h"
++#include "hw/remote/vfio-user-obj.h"
-     qemu_opts_del(opts);
++
-     return 0;
++void vfu_object_set_bus_irq(PCIBus *pci_bus)
-diff --git a/block/vvfat.c b/block/vvfat.c
++{
-index XXXXXXX..XXXXXXX 100644
++}
---- a/block/vvfat.c
+diff --git a/hw/remote/trace-events b/hw/remote/trace-events
-+++ b/block/vvfat.c
+index XXXXXXX..XXXXXXX 100644
-@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
+--- a/hw/remote/trace-events
++++ b/hw/remote/trace-events
-     s->current_cluster=0xffffffff;
+@@ -XXX,XX +XXX,XX @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
+ vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""
--    /* read only is the default for safety */
+ vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
--    bdrv_set_read_only(bs, true);
+ vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
-     s->qcow = NULL;
++vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
-     s->qcow_filename = NULL;
+diff --git a/stubs/meson.build b/stubs/meson.build
-     s->fat2 = NULL;
+index XXXXXXX..XXXXXXX 100644
-@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
+--- a/stubs/meson.build
-     s->sector_count = cyls * heads * secs - (s->first_sectors_number - 1);
++++ b/stubs/meson.build
+@@ -XXX,XX +XXX,XX @@ if have_system
-     if (qemu_opt_get_bool(opts, "rw", false)) {
+ else
--        ret = enable_write_target(bs, errp);
+   stub_ss.add(files('qdev.c'))
-+        if (!bdrv_is_read_only(bs)) {
+ endif
-+            ret = enable_write_target(bs, errp);
++stub_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_false: files('vfio-user-obj.c'))
 +            if (ret < 0) {
 +                goto fail;
 +            }
 +        } else {
 +            ret = -EPERM;
 +            error_setg(errp,
 +                       "Unable to set VVFAT to 'rw' when drive is read-only");
 +            goto fail;
 +        }
 +    } else  {
 +        /* read only is the default for safety */
 +        ret = bdrv_set_read_only(bs, true, &local_err);
          if (ret < 0) {
 +            error_propagate(errp, local_err);
              goto fail;
          }
 -        bdrv_set_read_only(bs, false);
      }
      bs->total_sectors = cyls * heads * secs;
 diff --git a/include/block/block.h b/include/block/block.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/block/block.h
 +++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
                              int64_t sector_num, int nb_sectors, int *pnum);
  bool bdrv_is_read_only(BlockDriverState *bs);
 -void bdrv_set_read_only(BlockDriverState *bs, bool read_only);
 +int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp);
  bool bdrv_is_sg(BlockDriverState *bs);
  bool bdrv_is_inserted(BlockDriverState *bs);
  int bdrv_media_changed(BlockDriverState *bs);
 --
-.9.3
+.36.1

-[Qemu-devel] [PULL 08/12] block: introduce bdrv_can_set_read_only()
+[PULL 16/18] vfio-user: handle reset of remote device
-Introduce check function for setting read_only flags.  Will return < 0 on
+From: Jagannathan Raman <jag.raman@oracle.com>
 error, with appropriate Error value set.  Does not alter any flags.
-Signed-off-by: Jeff Cody <jcody@redhat.com>
+Adds handler to reset a remote device
 Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
 Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
 Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Reviewed-by: John Snow <jsnow@redhat.com>
+Message-id: 112eeadf3bc4c6cdb100bc3f9a6fcfc20b467c1b.1655151679.git.jag.raman@oracle.com
-Message-id: e2bba34ac3bc76a0c42adc390413f358ae0566e8.1491597120.git.jcody@redhat.com
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 ---
- block.c               | 14 +++++++++++++-
+ hw/remote/vfio-user-obj.c | 20 ++++++++++++++++++++
- include/block/block.h |  1 +
+file changed, 20 insertions(+)
 files changed, 14 insertions(+), 1 deletion(-)
-diff --git a/block.c b/block.c
+diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
 index XXXXXXX..XXXXXXX 100644
---- a/block.c
+--- a/hw/remote/vfio-user-obj.c
-+++ b/block.c
++++ b/hw/remote/vfio-user-obj.c
-@@ -XXX,XX +XXX,XX @@ bool bdrv_is_read_only(BlockDriverState *bs)
+@@ -XXX,XX +XXX,XX @@ void vfu_object_set_bus_irq(PCIBus *pci_bus)
-     return bs->read_only;
+                  max_bdf);
  }
--int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
++static int vfu_object_device_reset(vfu_ctx_t *vfu_ctx, vfu_reset_type_t type)
-+int bdrv_can_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
++{
- {
++    VfuObject *o = vfu_get_private(vfu_ctx);
-     /* Do not set read_only if copy_on_read is enabled */
++
-     if (bs->copy_on_read && read_only) {
++    /* vfu_object_ctx_run() handles lost connection */
-@@ -XXX,XX +XXX,XX @@ int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
++    if (type == VFU_RESET_LOST_CONN) {
-         return -EPERM;
++        return 0;
-     }
++    }
++
 +    qdev_reset_all(DEVICE(o->pci_dev));
 +
 +    return 0;
 +}
 +
-+int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
+ /*
-+{
+  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
-+    int ret = 0;
+  * properties. It also depends on devices instantiated in QEMU. These
-+
+@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
-+    ret = bdrv_can_set_read_only(bs, read_only, errp);
+         goto fail;
      }
 +    ret = vfu_setup_device_reset_cb(o->vfu_ctx, &vfu_object_device_reset);
 +    if (ret < 0) {
-+        return ret;
++        error_setg(errp, "vfu: Failed to setup reset callback");
 +        goto fail;
 +    }
 +
-     bs->read_only = read_only;
+     ret = vfu_realize_ctx(o->vfu_ctx);
-     return 0;
+     if (ret < 0) {
- }
+         error_setg(errp, "vfu: Failed to realize device %s- %s",
 diff --git a/include/block/block.h b/include/block/block.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/block/block.h
 +++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
                              int64_t sector_num, int nb_sectors, int *pnum);
  bool bdrv_is_read_only(BlockDriverState *bs);
 +int bdrv_can_set_read_only(BlockDriverState *bs, bool read_only, Error **errp);
  int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp);
  bool bdrv_is_sg(BlockDriverState *bs);
  bool bdrv_is_inserted(BlockDriverState *bs);
 --
-.9.3
+.36.1

-New patch
+[PULL 17/18] linux-aio: fix unbalanced plugged counter in laio_io_unplug()
+Every laio_io_plug() call has a matching laio_io_unplug() call. There is
+a plugged counter that tracks the number of levels of plugging and
+allows for nesting.
+The plugged counter must reflect the balance between laio_io_plug() and
+laio_io_unplug() calls accurately. Otherwise I/O stalls occur since
+io_submit(2) calls are skipped while plugged.
+Reported-by: Nikolay Tenev <nt@storpool.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
+Message-id: 20220609164712.1539045-2-stefanha@redhat.com
+Cc: Stefano Garzarella <sgarzare@redhat.com>
+Fixes: 68d7946648 ("linux-aio: add `dev_max_batch` parameter to laio_io_unplug()")
+[Stefano Garzarella suggested adding a Fixes tag.
+--Stefan]
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+---
+ block/linux-aio.c | 4 +++-
+file changed, 3 insertions(+), 1 deletion(-)
+diff --git a/block/linux-aio.c b/block/linux-aio.c
+index XXXXXXX..XXXXXXX 100644
+--- a/block/linux-aio.c
++++ b/block/linux-aio.c
+@@ -XXX,XX +XXX,XX @@ void laio_io_unplug(BlockDriverState *bs, LinuxAioState *s,
+                     uint64_t dev_max_batch)
+ {
+     assert(s->io_q.plugged);
++    s->io_q.plugged--;
++
+     if (s->io_q.in_queue >= laio_max_batch(s, dev_max_batch) ||
+-        (--s->io_q.plugged == 0 &&
++        (!s->io_q.plugged &&
+          !s->io_q.blocked && !QSIMPLEQ_EMPTY(&s->io_q.pending))) {
+         ioq_submit(s);
+     }
+--
+.36.1

-New patch
+[PULL 18/18] linux-aio: explain why max batch is checked in laio_io_unplug()
+It may not be obvious why laio_io_unplug() checks max batch. I discussed
+this with Stefano and have added a comment summarizing the reason.
+Cc: Stefano Garzarella <sgarzare@redhat.com>
+Cc: Kevin Wolf <kwolf@redhat.com>
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
+Message-id: 20220609164712.1539045-3-stefanha@redhat.com
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+---
+ block/linux-aio.c | 6 ++++++
+file changed, 6 insertions(+)
+diff --git a/block/linux-aio.c b/block/linux-aio.c
+index XXXXXXX..XXXXXXX 100644
+--- a/block/linux-aio.c
++++ b/block/linux-aio.c
+@@ -XXX,XX +XXX,XX @@ void laio_io_unplug(BlockDriverState *bs, LinuxAioState *s,
+     assert(s->io_q.plugged);
+     s->io_q.plugged--;
++    /*
++     * Why max batch checking is performed here:
++     * Another BDS may have queued requests with a higher dev_max_batch and
++     * therefore in_queue could now exceed our dev_max_batch. Re-check the max
++     * batch so we can honor our device's dev_max_batch.
++     */
+     if (s->io_q.in_queue >= laio_max_batch(s, dev_max_batch) ||
+         (!s->io_q.plugged &&
+          !s->io_q.blocked && !QSIMPLEQ_EMPTY(&s->io_q.pending))) {
+--
+.36.1

The following changes since commit bfec359afba088aaacc7d316f43302f28c6e642a:

Merge remote-tracking branch 'remotes/armbru/tags/pull-qdev-2017-04-21' into staging (2017-04-21 11:42:03 +0100)

are available in the git repository at:

git://github.com/codyprime/qemu-kvm-jtc.git tags/block-pull-request

for you to fetch changes up to 1507631e438930bc07f776f303af127a9cdb4d41:

qemu-iotests: _cleanup_qemu must be called on exit (2017-04-21 08:32:44 -0400)

----------------------------------------------------------------

Block patches for 2.10

----------------------------------------------------------------

Ashish Mittal (2):
  block/vxhs.c: Add support for a new block device type called "vxhs"
  block/vxhs.c: Add qemu-iotests for new block device type "vxhs"

Jeff Cody (10):
  qemu-iotests: exclude vxhs from image creation via protocol
  block: add bdrv_set_read_only() helper function
  block: do not set BDS read_only if copy_on_read enabled
  block: honor BDRV_O_ALLOW_RDWR when clearing bs->read_only
  block: code movement
  block: introduce bdrv_can_set_read_only()
  block: use bdrv_can_set_read_only() during reopen
  block/rbd - update variable names to more apt names
  block/rbd: Add support for reopen()
  qemu-iotests: _cleanup_qemu must be called on exit

-- 
2.9.3

From: Ashish Mittal <ashmit602@gmail.com>

Source code for the qnio library that this code loads can be downloaded from:
https://github.com/VeritasHyperScale/libqnio.git

Sample command line using JSON syntax:
./x86_64-softmmu/qemu-system-x86_64 -name instance-00000008 -S -vnc 0.0.0.0:0
-k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
-msg timestamp=on
'json:{"driver":"vxhs","vdisk-id":"c3e9095a-a5ee-4dce-afeb-2a59fb387410",
"server":{"host":"172.172.17.4","port":"9999"}}'

Sample command line using URI syntax:
qemu-img convert -f raw -O raw -n
/var/lib/nova/instances/_base/0c5eacd5ebea5ed914b6a3e7b18f1ce734c386ad
vxhs://192.168.0.1:9999/c6718f6b-0401-441d-a8c3-1f0064d75ee0

Sample command line using TLS credentials (run in secure mode):
./qemu-io --object
tls-creds-x509,id=tls0,dir=/etc/pki/qemu/vxhs,endpoint=client -c 'read
-v 66000 2.5k' 'json:{"server.host": "127.0.0.1", "server.port": "9999",
"vdisk-id": "/test.raw", "driver": "vxhs", "tls-creds":"tls0"}'

Signed-off-by: Ashish Mittal <Ashish.Mittal@veritas.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Jeff Cody <jcody@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Message-id: 1491277689-24949-2-git-send-email-Ashish.Mittal@veritas.com
---
 block/Makefile.objs  |   2 +
 block/trace-events   |  17 ++
 block/vxhs.c         | 575 +++++++++++++++++++++++++++++++++++++++++++++++++++
 configure            |  39 ++++
 qapi/block-core.json |  23 ++-
 5 files changed, 654 insertions(+), 2 deletions(-)
 create mode 100644 block/vxhs.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index XXXXXXX..XXXXXXX 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_LIBNFS) += nfs.o
 block-obj-$(CONFIG_CURL) += curl.o
 block-obj-$(CONFIG_RBD) += rbd.o
 block-obj-$(CONFIG_GLUSTERFS) += gluster.o
+block-obj-$(CONFIG_VXHS) += vxhs.o
 block-obj-$(CONFIG_LIBSSH2) += ssh.o
 block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
@@ -XXX,XX +XXX,XX @@ rbd.o-cflags       := $(RBD_CFLAGS)
 rbd.o-libs         := $(RBD_LIBS)
 gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
 gluster.o-libs     := $(GLUSTERFS_LIBS)
+vxhs.o-libs        := $(VXHS_LIBS)
 ssh.o-cflags       := $(LIBSSH2_CFLAGS)
 ssh.o-libs         := $(LIBSSH2_LIBS)
 block-obj-$(if $(CONFIG_BZIP2),m,n) += dmg-bz2.o
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ qed_aio_write_data(void *s, void *acb, int ret, uint64_t offset, size_t len) "s
 qed_aio_write_prefill(void *s, void *acb, uint64_t start, size_t len, uint64_t offset) "s %p acb %p start %"PRIu64" len %zu offset %"PRIu64
 qed_aio_write_postfill(void *s, void *acb, uint64_t start, size_t len, uint64_t offset) "s %p acb %p start %"PRIu64" len %zu offset %"PRIu64
 qed_aio_write_main(void *s, void *acb, int ret, uint64_t offset, size_t len) "s %p acb %p ret %d offset %"PRIu64" len %zu"
+
+# block/vxhs.c
+vxhs_iio_callback(int error) "ctx is NULL: error %d"
+vxhs_iio_callback_chnfail(int err, int error) "QNIO channel failed, no i/o %d, %d"
+vxhs_iio_callback_unknwn(int opcode, int err) "unexpected opcode %d, errno %d"
+vxhs_aio_rw_invalid(int req) "Invalid I/O request iodir %d"
+vxhs_aio_rw_ioerr(char *guid, int iodir, uint64_t size, uint64_t off, void *acb, int ret, int err) "IO ERROR (vDisk %s) FOR : Read/Write = %d size = %lu offset = %lu ACB = %p. Error = %d, errno = %d"
+vxhs_get_vdisk_stat_err(char *guid, int ret, int err) "vDisk (%s) stat ioctl failed, ret = %d, errno = %d"
+vxhs_get_vdisk_stat(char *vdisk_guid, uint64_t vdisk_size) "vDisk %s stat ioctl returned size %lu"
+vxhs_complete_aio(void *acb, uint64_t ret) "aio failed acb %p ret %ld"
+vxhs_parse_uri_filename(const char *filename) "URI passed via bdrv_parse_filename %s"
+vxhs_open_vdiskid(const char *vdisk_id) "Opening vdisk-id %s"
+vxhs_open_hostinfo(char *of_vsa_addr, int port) "Adding host %s:%d to BDRVVXHSState"
+vxhs_open_iio_open(const char *host) "Failed to connect to storage agent on host %s"
+vxhs_parse_uri_hostinfo(char *host, int port) "Host: IP %s, Port %d"
+vxhs_close(char *vdisk_guid) "Closing vdisk %s"
+vxhs_get_creds(const char *cacert, const char *client_key, const char *client_cert) "cacert %s, client_key %s, client_cert %s"
diff --git a/block/vxhs.c b/block/vxhs.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/block/vxhs.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * QEMU Block driver for Veritas HyperScale (VxHS)
+ *
+ * Copyright (c) 2017 Veritas Technologies LLC.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include <qnio/qnio_api.h>
+#include <sys/param.h>
+#include "block/block_int.h"
+#include "qapi/qmp/qerror.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qstring.h"
+#include "trace.h"
+#include "qemu/uri.h"
+#include "qapi/error.h"
+#include "qemu/uuid.h"
+#include "crypto/tlscredsx509.h"
+
+#define VXHS_OPT_FILENAME           "filename"
+#define VXHS_OPT_VDISK_ID           "vdisk-id"
+#define VXHS_OPT_SERVER             "server"
+#define VXHS_OPT_HOST               "host"
+#define VXHS_OPT_PORT               "port"
+
+/* Only accessed under QEMU global mutex */
+static uint32_t vxhs_ref;
+
+typedef enum {
+    VDISK_AIO_READ,
+    VDISK_AIO_WRITE,
+} VDISKAIOCmd;
+
+/*
+ * HyperScale AIO callbacks structure
+ */
+typedef struct VXHSAIOCB {
+    BlockAIOCB common;
+    int err;
+} VXHSAIOCB;
+
+typedef struct VXHSvDiskHostsInfo {
+    void *dev_handle; /* Device handle */
+    char *host; /* Host name or IP */
+    int port; /* Host's port number */
+} VXHSvDiskHostsInfo;
+
+/*
+ * Structure per vDisk maintained for state
+ */
+typedef struct BDRVVXHSState {
+    VXHSvDiskHostsInfo vdisk_hostinfo; /* Per host info */
+    char *vdisk_guid;
+    char *tlscredsid; /* tlscredsid */
+} BDRVVXHSState;
+
+static void vxhs_complete_aio_bh(void *opaque)
+{
+    VXHSAIOCB *acb = opaque;
+    BlockCompletionFunc *cb = acb->common.cb;
+    void *cb_opaque = acb->common.opaque;
+    int ret = 0;
+
+    if (acb->err != 0) {
+        trace_vxhs_complete_aio(acb, acb->err);
+        ret = (-EIO);
+    }
+
+    qemu_aio_unref(acb);
+    cb(cb_opaque, ret);
+}
+
+/*
+ * Called from a libqnio thread
+ */
+static void vxhs_iio_callback(void *ctx, uint32_t opcode, uint32_t error)
+{
+    VXHSAIOCB *acb = NULL;
+
+    switch (opcode) {
+    case IRP_READ_REQUEST:
+    case IRP_WRITE_REQUEST:
+
+        /*
+         * ctx is VXHSAIOCB*
+         * ctx is NULL if error is QNIOERROR_CHANNEL_HUP
+         */
+        if (ctx) {
+            acb = ctx;
+        } else {
+            trace_vxhs_iio_callback(error);
+            goto out;
+        }
+
+        if (error) {
+            if (!acb->err) {
+                acb->err = error;
+            }
+            trace_vxhs_iio_callback(error);
+        }
+
+        aio_bh_schedule_oneshot(bdrv_get_aio_context(acb->common.bs),
+                                vxhs_complete_aio_bh, acb);
+        break;
+
+    default:
+        if (error == QNIOERROR_HUP) {
+            /*
+             * Channel failed, spontaneous notification,
+             * not in response to I/O
+             */
+            trace_vxhs_iio_callback_chnfail(error, errno);
+        } else {
+            trace_vxhs_iio_callback_unknwn(opcode, error);
+        }
+        break;
+    }
+out:
+    return;
+}
+
+static QemuOptsList runtime_opts = {
+    .name = "vxhs",
+    .head = QTAILQ_HEAD_INITIALIZER(runtime_opts.head),
+    .desc = {
+        {
+            .name = VXHS_OPT_FILENAME,
+            .type = QEMU_OPT_STRING,
+            .help = "URI to the Veritas HyperScale image",
+        },
+        {
+            .name = VXHS_OPT_VDISK_ID,
+            .type = QEMU_OPT_STRING,
+            .help = "UUID of the VxHS vdisk",
+        },
+        {
+            .name = "tls-creds",
+            .type = QEMU_OPT_STRING,
+            .help = "ID of the TLS/SSL credentials to use",
+        },
+        { /* end of list */ }
+    },
+};
+
+static QemuOptsList runtime_tcp_opts = {
+    .name = "vxhs_tcp",
+    .head = QTAILQ_HEAD_INITIALIZER(runtime_tcp_opts.head),
+    .desc = {
+        {
+            .name = VXHS_OPT_HOST,
+            .type = QEMU_OPT_STRING,
+            .help = "host address (ipv4 addresses)",
+        },
+        {
+            .name = VXHS_OPT_PORT,
+            .type = QEMU_OPT_NUMBER,
+            .help = "port number on which VxHSD is listening (default 9999)",
+            .def_value_str = "9999"
+        },
+        { /* end of list */ }
+    },
+};
+
+/*
+ * Parse incoming URI and populate *options with the host
+ * and device information
+ */
+static int vxhs_parse_uri(const char *filename, QDict *options)
+{
+    URI *uri = NULL;
+    char *port;
+    int ret = 0;
+
+    trace_vxhs_parse_uri_filename(filename);
+    uri = uri_parse(filename);
+    if (!uri || !uri->server || !uri->path) {
+        uri_free(uri);
+        return -EINVAL;
+    }
+
+    qdict_put(options, VXHS_OPT_SERVER".host", qstring_from_str(uri->server));
+
+    if (uri->port) {
+        port = g_strdup_printf("%d", uri->port);
+        qdict_put(options, VXHS_OPT_SERVER".port", qstring_from_str(port));
+        g_free(port);
+    }
+
+    qdict_put(options, "vdisk-id", qstring_from_str(uri->path));
+
+    trace_vxhs_parse_uri_hostinfo(uri->server, uri->port);
+    uri_free(uri);
+
+    return ret;
+}
+
+static void vxhs_parse_filename(const char *filename, QDict *options,
+                                Error **errp)
+{
+    if (qdict_haskey(options, "vdisk-id") || qdict_haskey(options, "server")) {
+        error_setg(errp, "vdisk-id/server and a file name may not be specified "
+                         "at the same time");
+        return;
+    }
+
+    if (strstr(filename, "://")) {
+        int ret = vxhs_parse_uri(filename, options);
+        if (ret < 0) {
+            error_setg(errp, "Invalid URI. URI should be of the form "
+                       "  vxhs://<host_ip>:<port>/<vdisk-id>");
+        }
+    }
+}
+
+static int vxhs_init_and_ref(void)
+{
+    if (vxhs_ref++ == 0) {
+        if (iio_init(QNIO_VERSION, vxhs_iio_callback)) {
+            return -ENODEV;
+        }
+    }
+    return 0;
+}
+
+static void vxhs_unref(void)
+{
+    if (--vxhs_ref == 0) {
+        iio_fini();
+    }
+}
+
+static void vxhs_get_tls_creds(const char *id, char **cacert,
+                               char **key, char **cert, Error **errp)
+{
+    Object *obj;
+    QCryptoTLSCreds *creds;
+    QCryptoTLSCredsX509 *creds_x509;
+
+    obj = object_resolve_path_component(
+        object_get_objects_root(), id);
+
+    if (!obj) {
+        error_setg(errp, "No TLS credentials with id '%s'",
+                   id);
+        return;
+    }
+
+    creds_x509 = (QCryptoTLSCredsX509 *)
+        object_dynamic_cast(obj, TYPE_QCRYPTO_TLS_CREDS_X509);
+
+    if (!creds_x509) {
+        error_setg(errp, "Object with id '%s' is not TLS credentials",
+                   id);
+        return;
+    }
+
+    creds = &creds_x509->parent_obj;
+
+    if (creds->endpoint != QCRYPTO_TLS_CREDS_ENDPOINT_CLIENT) {
+        error_setg(errp,
+                   "Expecting TLS credentials with a client endpoint");
+        return;
+    }
+
+    /*
+     * Get the cacert, client_cert and client_key file names.
+     */
+    if (!creds->dir) {
+        error_setg(errp, "TLS object missing 'dir' property value");
+        return;
+    }
+
+    *cacert = g_strdup_printf("%s/%s", creds->dir,
+                              QCRYPTO_TLS_CREDS_X509_CA_CERT);
+    *cert = g_strdup_printf("%s/%s", creds->dir,
+                            QCRYPTO_TLS_CREDS_X509_CLIENT_CERT);
+    *key = g_strdup_printf("%s/%s", creds->dir,
+                           QCRYPTO_TLS_CREDS_X509_CLIENT_KEY);
+}
+
+static int vxhs_open(BlockDriverState *bs, QDict *options,
+                     int bdrv_flags, Error **errp)
+{
+    BDRVVXHSState *s = bs->opaque;
+    void *dev_handlep;
+    QDict *backing_options = NULL;
+    QemuOpts *opts = NULL;
+    QemuOpts *tcp_opts = NULL;
+    char *of_vsa_addr = NULL;
+    Error *local_err = NULL;
+    const char *vdisk_id_opt;
+    const char *server_host_opt;
+    int ret = 0;
+    char *cacert = NULL;
+    char *client_key = NULL;
+    char *client_cert = NULL;
+
+    ret = vxhs_init_and_ref();
+    if (ret < 0) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    /* Create opts info from runtime_opts and runtime_tcp_opts list */
+    opts = qemu_opts_create(&runtime_opts, NULL, 0, &error_abort);
+    tcp_opts = qemu_opts_create(&runtime_tcp_opts, NULL, 0, &error_abort);
+
+    qemu_opts_absorb_qdict(opts, options, &local_err);
+    if (local_err) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    /* vdisk-id is the disk UUID */
+    vdisk_id_opt = qemu_opt_get(opts, VXHS_OPT_VDISK_ID);
+    if (!vdisk_id_opt) {
+        error_setg(&local_err, QERR_MISSING_PARAMETER, VXHS_OPT_VDISK_ID);
+        ret = -EINVAL;
+        goto out;
+    }
+
+    /* vdisk-id may contain a leading '/' */
+    if (strlen(vdisk_id_opt) > UUID_FMT_LEN + 1) {
+        error_setg(&local_err, "vdisk-id cannot be more than %d characters",
+                   UUID_FMT_LEN);
+        ret = -EINVAL;
+        goto out;
+    }
+
+    s->vdisk_guid = g_strdup(vdisk_id_opt);
+    trace_vxhs_open_vdiskid(vdisk_id_opt);
+
+    /* get the 'server.' arguments */
+    qdict_extract_subqdict(options, &backing_options, VXHS_OPT_SERVER".");
+
+    qemu_opts_absorb_qdict(tcp_opts, backing_options, &local_err);
+    if (local_err != NULL) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    server_host_opt = qemu_opt_get(tcp_opts, VXHS_OPT_HOST);
+    if (!server_host_opt) {
+        error_setg(&local_err, QERR_MISSING_PARAMETER,
+                   VXHS_OPT_SERVER"."VXHS_OPT_HOST);
+        ret = -EINVAL;
+        goto out;
+    }
+
+    if (strlen(server_host_opt) > MAXHOSTNAMELEN) {
+        error_setg(&local_err, "server.host cannot be more than %d characters",
+                   MAXHOSTNAMELEN);
+        ret = -EINVAL;
+        goto out;
+    }
+
+    /* check if we got tls-creds via the --object argument */
+    s->tlscredsid = g_strdup(qemu_opt_get(opts, "tls-creds"));
+    if (s->tlscredsid) {
+        vxhs_get_tls_creds(s->tlscredsid, &cacert, &client_key,
+                           &client_cert, &local_err);
+        if (local_err != NULL) {
+            ret = -EINVAL;
+            goto out;
+        }
+        trace_vxhs_get_creds(cacert, client_key, client_cert);
+    }
+
+    s->vdisk_hostinfo.host = g_strdup(server_host_opt);
+    s->vdisk_hostinfo.port = g_ascii_strtoll(qemu_opt_get(tcp_opts,
+                                                          VXHS_OPT_PORT),
+                                                          NULL, 0);
+
+    trace_vxhs_open_hostinfo(s->vdisk_hostinfo.host,
+                             s->vdisk_hostinfo.port);
+
+    of_vsa_addr = g_strdup_printf("of://%s:%d",
+                                  s->vdisk_hostinfo.host,
+                                  s->vdisk_hostinfo.port);
+
+    /*
+     * Open qnio channel to storage agent if not opened before
+     */
+    dev_handlep = iio_open(of_vsa_addr, s->vdisk_guid, 0,
+                           cacert, client_key, client_cert);
+    if (dev_handlep == NULL) {
+        trace_vxhs_open_iio_open(of_vsa_addr);
+        ret = -ENODEV;
+        goto out;
+    }
+    s->vdisk_hostinfo.dev_handle = dev_handlep;
+
+out:
+    g_free(of_vsa_addr);
+    QDECREF(backing_options);
+    qemu_opts_del(tcp_opts);
+    qemu_opts_del(opts);
+    g_free(cacert);
+    g_free(client_key);
+    g_free(client_cert);
+
+    if (ret < 0) {
+        vxhs_unref();
+        error_propagate(errp, local_err);
+        g_free(s->vdisk_hostinfo.host);
+        g_free(s->vdisk_guid);
+        g_free(s->tlscredsid);
+        s->vdisk_guid = NULL;
+    }
+
+    return ret;
+}
+
+static const AIOCBInfo vxhs_aiocb_info = {
+    .aiocb_size = sizeof(VXHSAIOCB)
+};
+
+/*
+ * This allocates QEMU-VXHS callback for each IO
+ * and is passed to QNIO. When QNIO completes the work,
+ * it will be passed back through the callback.
+ */
+static BlockAIOCB *vxhs_aio_rw(BlockDriverState *bs, int64_t sector_num,
+                               QEMUIOVector *qiov, int nb_sectors,
+                               BlockCompletionFunc *cb, void *opaque,
+                               VDISKAIOCmd iodir)
+{
+    VXHSAIOCB *acb = NULL;
+    BDRVVXHSState *s = bs->opaque;
+    size_t size;
+    uint64_t offset;
+    int iio_flags = 0;
+    int ret = 0;
+    void *dev_handle = s->vdisk_hostinfo.dev_handle;
+
+    offset = sector_num * BDRV_SECTOR_SIZE;
+    size = nb_sectors * BDRV_SECTOR_SIZE;
+    acb = qemu_aio_get(&vxhs_aiocb_info, bs, cb, opaque);
+
+    /*
+     * Initialize VXHSAIOCB.
+     */
+    acb->err = 0;
+
+    iio_flags = IIO_FLAG_ASYNC;
+
+    switch (iodir) {
+    case VDISK_AIO_WRITE:
+            ret = iio_writev(dev_handle, acb, qiov->iov, qiov->niov,
+                             offset, (uint64_t)size, iio_flags);
+            break;
+    case VDISK_AIO_READ:
+            ret = iio_readv(dev_handle, acb, qiov->iov, qiov->niov,
+                            offset, (uint64_t)size, iio_flags);
+            break;
+    default:
+            trace_vxhs_aio_rw_invalid(iodir);
+            goto errout;
+    }
+
+    if (ret != 0) {
+        trace_vxhs_aio_rw_ioerr(s->vdisk_guid, iodir, size, offset,
+                                acb, ret, errno);
+        goto errout;
+    }
+    return &acb->common;
+
+errout:
+    qemu_aio_unref(acb);
+    return NULL;
+}
+
+static BlockAIOCB *vxhs_aio_readv(BlockDriverState *bs,
+                                   int64_t sector_num, QEMUIOVector *qiov,
+                                   int nb_sectors,
+                                   BlockCompletionFunc *cb, void *opaque)
+{
+    return vxhs_aio_rw(bs, sector_num, qiov, nb_sectors, cb,
+                       opaque, VDISK_AIO_READ);
+}
+
+static BlockAIOCB *vxhs_aio_writev(BlockDriverState *bs,
+                                   int64_t sector_num, QEMUIOVector *qiov,
+                                   int nb_sectors,
+                                   BlockCompletionFunc *cb, void *opaque)
+{
+    return vxhs_aio_rw(bs, sector_num, qiov, nb_sectors,
+                       cb, opaque, VDISK_AIO_WRITE);
+}
+
+static void vxhs_close(BlockDriverState *bs)
+{
+    BDRVVXHSState *s = bs->opaque;
+
+    trace_vxhs_close(s->vdisk_guid);
+
+    g_free(s->vdisk_guid);
+    s->vdisk_guid = NULL;
+
+    /*
+     * Close vDisk device
+     */
+    if (s->vdisk_hostinfo.dev_handle) {
+        iio_close(s->vdisk_hostinfo.dev_handle);
+        s->vdisk_hostinfo.dev_handle = NULL;
+    }
+
+    vxhs_unref();
+
+    /*
+     * Free the dynamically allocated host string etc
+     */
+    g_free(s->vdisk_hostinfo.host);
+    g_free(s->tlscredsid);
+    s->tlscredsid = NULL;
+    s->vdisk_hostinfo.host = NULL;
+    s->vdisk_hostinfo.port = 0;
+}
+
+static int64_t vxhs_get_vdisk_stat(BDRVVXHSState *s)
+{
+    int64_t vdisk_size = -1;
+    int ret = 0;
+    void *dev_handle = s->vdisk_hostinfo.dev_handle;
+
+    ret = iio_ioctl(dev_handle, IOR_VDISK_STAT, &vdisk_size, 0);
+    if (ret < 0) {
+        trace_vxhs_get_vdisk_stat_err(s->vdisk_guid, ret, errno);
+        return -EIO;
+    }
+
+    trace_vxhs_get_vdisk_stat(s->vdisk_guid, vdisk_size);
+    return vdisk_size;
+}
+
+/*
+ * Returns the size of vDisk in bytes. This is required
+ * by QEMU block upper block layer so that it is visible
+ * to guest.
+ */
+static int64_t vxhs_getlength(BlockDriverState *bs)
+{
+    BDRVVXHSState *s = bs->opaque;
+    int64_t vdisk_size;
+
+    vdisk_size = vxhs_get_vdisk_stat(s);
+    if (vdisk_size < 0) {
+        return -EIO;
+    }
+
+    return vdisk_size;
+}
+
+static BlockDriver bdrv_vxhs = {
+    .format_name                  = "vxhs",
+    .protocol_name                = "vxhs",
+    .instance_size                = sizeof(BDRVVXHSState),
+    .bdrv_file_open               = vxhs_open,
+    .bdrv_parse_filename          = vxhs_parse_filename,
+    .bdrv_close                   = vxhs_close,
+    .bdrv_getlength               = vxhs_getlength,
+    .bdrv_aio_readv               = vxhs_aio_readv,
+    .bdrv_aio_writev              = vxhs_aio_writev,
+};
+
+static void bdrv_vxhs_init(void)
+{
+    bdrv_register(&bdrv_vxhs);
+}
+
+block_init(bdrv_vxhs_init);
diff --git a/configure b/configure
index XXXXXXX..XXXXXXX 100755
--- a/configure
+++ b/configure
@@ -XXX,XX +XXX,XX @@ numa=""
 tcmalloc="no"
 jemalloc="no"
 replication="yes"
+vxhs=""
 
 supported_cpu="no"
 supported_os="no"
@@ -XXX,XX +XXX,XX @@ for opt do
   ;;
   --enable-replication) replication="yes"
   ;;
+  --disable-vxhs) vxhs="no"
+  ;;
+  --enable-vxhs) vxhs="yes"
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
   xfsctl          xfsctl support
   qom-cast-debug  cast debugging support
   tools           build qemu-io, qemu-nbd and qemu-image tools
+  vxhs            Veritas HyperScale vDisk backend support
 
 NOTE: The object files are built at the place where configure is launched
 EOF
@@ -XXX,XX +XXX,XX @@ if compile_prog "" "" ; then
 fi
 
 ##########################################
+# Veritas HyperScale block driver VxHS
+# Check if libvxhs is installed
+
+if test "$vxhs" != "no" ; then
+  cat > $TMPC <<EOF
+#include <stdint.h>
+#include <qnio/qnio_api.h>
+
+void *vxhs_callback;
+
+int main(void) {
+    iio_init(QNIO_VERSION, vxhs_callback);
+    return 0;
+}
+EOF
+  vxhs_libs="-lvxhs -lssl"
+  if compile_prog "" "$vxhs_libs" ; then
+    vxhs=yes
+  else
+    if test "$vxhs" = "yes" ; then
+      feature_not_found "vxhs block device" "Install libvxhs See github"
+    fi
+    vxhs=no
+  fi
+fi
+
+##########################################
 # End of CC checks
 # After here, no more $cc or $ld runs
 
@@ -XXX,XX +XXX,XX @@ echo "tcmalloc support  $tcmalloc"
 echo "jemalloc support  $jemalloc"
 echo "avx2 optimization $avx2_opt"
 echo "replication support $replication"
+echo "VxHS block device $vxhs"
 
 if test "$sdl_too_old" = "yes"; then
 echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -XXX,XX +XXX,XX @@ if test "$pthread_setname_np" = "yes" ; then
   echo "CONFIG_PTHREAD_SETNAME_NP=y" >> $config_host_mak
 fi
 
+if test "$vxhs" = "yes" ; then
+  echo "CONFIG_VXHS=y" >> $config_host_mak
+  echo "VXHS_LIBS=$vxhs_libs" >> $config_host_mak
+fi
+
 if test "$tcg_interpreter" = "yes"; then
   QEMU_INCLUDES="-I\$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
 elif test "$ARCH" = "sparc64" ; then
diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # Drivers that are supported in block device operations.
 #
+# @vxhs: Since 2.10
+#
 # Since: 2.9
 ##
 { 'enum': 'BlockdevDriver',
@@ -XXX,XX +XXX,XX @@
             'host_device', 'http', 'https', 'iscsi', 'luks', 'nbd', 'nfs',
             'null-aio', 'null-co', 'parallels', 'qcow', 'qcow2', 'qed',
             'quorum', 'raw', 'rbd', 'replication', 'sheepdog', 'ssh',
-            'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat' ] }
+            'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] }
 
 ##
 # @BlockdevOptionsFile:
@@ -XXX,XX +XXX,XX @@
   'data': { '*offset': 'int', '*size': 'int' } }
 
 ##
+# @BlockdevOptionsVxHS:
+#
+# Driver specific block device options for VxHS
+#
+# @vdisk-id:    UUID of VxHS volume
+# @server:      vxhs server IP, port
+# @tls-creds:   TLS credentials ID
+#
+# Since: 2.10
+##
+{ 'struct': 'BlockdevOptionsVxHS',
+  'data': { 'vdisk-id': 'str',
+            'server': 'InetSocketAddressBase',
+            '*tls-creds': 'str' } }
+
+##
 # @BlockdevOptions:
 #
 # Options for creating a block device.  Many options are available for all
@@ -XXX,XX +XXX,XX @@
       'vhdx':       'BlockdevOptionsGenericFormat',
       'vmdk':       'BlockdevOptionsGenericCOWFormat',
       'vpc':        'BlockdevOptionsGenericFormat',
-      'vvfat':      'BlockdevOptionsVVFAT'
+      'vvfat':      'BlockdevOptionsVVFAT',
+      'vxhs':       'BlockdevOptionsVxHS'
   } }
 
 ##
-- 
2.9.3

From: Ashish Mittal <ashmit602@gmail.com>

These changes use a vxhs test server that is a part of the following
repository:
https://github.com/VeritasHyperScale/libqnio.git

Signed-off-by: Ashish Mittal <Ashish.Mittal@veritas.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Jeff Cody <jcody@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Message-id: 1491277689-24949-3-git-send-email-Ashish.Mittal@veritas.com
---
 tests/qemu-iotests/common        |  6 ++++++
 tests/qemu-iotests/common.config | 13 +++++++++++++
 tests/qemu-iotests/common.filter |  1 +
 tests/qemu-iotests/common.rc     | 19 +++++++++++++++++++
 4 files changed, 39 insertions(+)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -XXX,XX +XXX,XX @@ check options
     -ssh                test ssh
     -nfs                test nfs
     -luks               test luks
+    -vxhs               test vxhs
     -xdiff              graphical mode diff
     -nocache            use O_DIRECT on backing file
     -misalign           misalign memory allocations
@@ -XXX,XX +XXX,XX @@ testlist options
             xpand=false
             ;;
 
+        -vxhs)
+            IMGPROTO=vxhs
+            xpand=false
+            ;;
+
         -ssh)
             IMGPROTO=ssh
             xpand=false
diff --git a/tests/qemu-iotests/common.config b/tests/qemu-iotests/common.config
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.config
+++ b/tests/qemu-iotests/common.config
@@ -XXX,XX +XXX,XX @@ if [ -z "$QEMU_NBD_PROG" ]; then
     export QEMU_NBD_PROG="`set_prog_path qemu-nbd`"
 fi
 
+if [ -z "$QEMU_VXHS_PROG" ]; then
+    export QEMU_VXHS_PROG="`set_prog_path qnio_server`"
+fi
+
 _qemu_wrapper()
 {
     (
@@ -XXX,XX +XXX,XX @@ _qemu_nbd_wrapper()
     )
 }
 
+_qemu_vxhs_wrapper()
+{
+    (
+        echo $BASHPID > "${TEST_DIR}/qemu-vxhs.pid"
+        exec "$QEMU_VXHS_PROG" $QEMU_VXHS_OPTIONS "$@"
+    )
+}
+
 export QEMU=_qemu_wrapper
 export QEMU_IMG=_qemu_img_wrapper
 export QEMU_IO=_qemu_io_wrapper
 export QEMU_NBD=_qemu_nbd_wrapper
+export QEMU_VXHS=_qemu_vxhs_wrapper
 
 QEMU_IMG_EXTRA_ARGS=
 if [ "$IMGOPTSSYNTAX" = "true" ]; then
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_info()
         -e "s#$TEST_DIR#TEST_DIR#g" \
         -e "s#$IMGFMT#IMGFMT#g" \
         -e 's#nbd://127.0.0.1:10810$#TEST_DIR/t.IMGFMT#g' \
+        -e 's#json.*vdisk-id.*vxhs"}}#TEST_DIR/t.IMGFMT#' \
         -e "/encrypted: yes/d" \
         -e "/cluster_size: [0-9]\\+/d" \
         -e "/table_size: [0-9]\\+/d" \
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -XXX,XX +XXX,XX @@ else
     elif [ "$IMGPROTO" = "nfs" ]; then
         TEST_DIR="nfs://127.0.0.1/$TEST_DIR"
         TEST_IMG=$TEST_DIR/t.$IMGFMT
+    elif [ "$IMGPROTO" = "vxhs" ]; then
+        TEST_IMG_FILE=$TEST_DIR/t.$IMGFMT
+        TEST_IMG="vxhs://127.0.0.1:9999/t.$IMGFMT"
     else
         TEST_IMG=$IMGPROTO:$TEST_DIR/t.$IMGFMT
     fi
@@ -XXX,XX +XXX,XX @@ _make_test_img()
         eval "$QEMU_NBD -v -t -b 127.0.0.1 -p 10810 -f $IMGFMT  $TEST_IMG_FILE >/dev/null &"
         sleep 1 # FIXME: qemu-nbd needs to be listening before we continue
     fi
+
+    # Start QNIO server on image directory for vxhs protocol
+    if [ $IMGPROTO = "vxhs" ]; then
+        eval "$QEMU_VXHS -d  $TEST_DIR > /dev/null &"
+        sleep 1 # Wait for server to come up.
+    fi
 }
 
 _rm_test_img()
@@ -XXX,XX +XXX,XX @@ _cleanup_test_img()
             fi
             rm -f "$TEST_IMG_FILE"
             ;;
+        vxhs)
+            if [ -f "${TEST_DIR}/qemu-vxhs.pid" ]; then
+                local QEMU_VXHS_PID
+                read QEMU_VXHS_PID < "${TEST_DIR}/qemu-vxhs.pid"
+                kill ${QEMU_VXHS_PID} >/dev/null 2>&1
+                rm -f "${TEST_DIR}/qemu-vxhs.pid"
+            fi
+            rm -f "$TEST_IMG_FILE"
+            ;;
+
         file)
             _rm_test_img "$TEST_DIR/t.$IMGFMT"
             _rm_test_img "$TEST_DIR/t.$IMGFMT.orig"
-- 
2.9.3

The protocol VXHS does not support image creation.  Some tests expect
to be able to create images through the protocol.  Exclude VXHS from
these tests.

diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Any format supporting backing files
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
 
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Any format supporting backing files
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
diff --git a/tests/qemu-iotests/029 b/tests/qemu-iotests/029
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/029
+++ b/tests/qemu-iotests/029
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Any format supporting intenal snapshots
 _supported_fmt qcow2
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 # Internal snapshots are (currently) impossible with refcount_bits=1
 _unsupported_imgopts 'refcount_bits=1[^0-9]'
diff --git a/tests/qemu-iotests/073 b/tests/qemu-iotests/073
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/073
+++ b/tests/qemu-iotests/073
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 
 CLUSTER_SIZE=64k
diff --git a/tests/qemu-iotests/114 b/tests/qemu-iotests/114
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/114
+++ b/tests/qemu-iotests/114
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 
 
diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/130
+++ b/tests/qemu-iotests/130
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 
 qemu_comm_method="monitor"
diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/134
+++ b/tests/qemu-iotests/134
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 
 
diff --git a/tests/qemu-iotests/156 b/tests/qemu-iotests/156
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/156
+++ b/tests/qemu-iotests/156
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2 qed
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 
 # Create source disk
diff --git a/tests/qemu-iotests/158 b/tests/qemu-iotests/158
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/158
+++ b/tests/qemu-iotests/158
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto generic
+_unsupported_proto vxhs
 _supported_os Linux
 
 
-- 
2.9.3

We have a helper wrapper for checking for the BDS read_only flag,
add a helper wrapper to set the read_only flag as well.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 9b18972d05f5fa2ac16c014f0af98d680553048d.1491597120.git.jcody@redhat.com
---
 block.c               | 5 +++++
 block/bochs.c         | 2 +-
 block/cloop.c         | 2 +-
 block/dmg.c           | 2 +-
 block/rbd.c           | 2 +-
 block/vvfat.c         | 4 ++--
 include/block/block.h | 1 +
 7 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ void path_combine(char *dest, int dest_size,
     }
 }
 
+void bdrv_set_read_only(BlockDriverState *bs, bool read_only)
+{
+    bs->read_only = read_only;
+}
+
 void bdrv_get_full_backing_filename_from_filename(const char *backed,
                                                   const char *backing,
                                                   char *dest, size_t sz,
diff --git a/block/bochs.c b/block/bochs.c
index XXXXXXX..XXXXXXX 100644
--- a/block/bochs.c
+++ b/block/bochs.c
@@ -XXX,XX +XXX,XX @@ static int bochs_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
-    bs->read_only = true; /* no write support yet */
+    bdrv_set_read_only(bs, true); /* no write support yet */
 
     ret = bdrv_pread(bs->file, 0, &bochs, sizeof(bochs));
     if (ret < 0) {
diff --git a/block/cloop.c b/block/cloop.c
index XXXXXXX..XXXXXXX 100644
--- a/block/cloop.c
+++ b/block/cloop.c
@@ -XXX,XX +XXX,XX @@ static int cloop_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
-    bs->read_only = true;
+    bdrv_set_read_only(bs, true);
 
     /* read header */
     ret = bdrv_pread(bs->file, 128, &s->block_size, 4);
diff --git a/block/dmg.c b/block/dmg.c
index XXXXXXX..XXXXXXX 100644
--- a/block/dmg.c
+++ b/block/dmg.c
@@ -XXX,XX +XXX,XX @@ static int dmg_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     block_module_load_one("dmg-bz2");
-    bs->read_only = true;
+    bdrv_set_read_only(bs, true);
 
     s->n_chunks = 0;
     s->offsets = s->lengths = s->sectors = s->sectorcounts = NULL;
diff --git a/block/rbd.c b/block/rbd.c
index XXXXXXX..XXXXXXX 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
         goto failed_open;
     }
 
-    bs->read_only = (s->snap != NULL);
+    bdrv_set_read_only(bs, (s->snap != NULL));
 
     qemu_opts_del(opts);
     return 0;
diff --git a/block/vvfat.c b/block/vvfat.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vvfat.c
+++ b/block/vvfat.c
@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
     s->current_cluster=0xffffffff;
 
     /* read only is the default for safety */
-    bs->read_only = true;
+    bdrv_set_read_only(bs, true);
     s->qcow = NULL;
     s->qcow_filename = NULL;
     s->fat2 = NULL;
@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
         if (ret < 0) {
             goto fail;
         }
-        bs->read_only = false;
+        bdrv_set_read_only(bs, false);
     }
 
     bs->total_sectors = cyls * heads * secs;
diff --git a/include/block/block.h b/include/block/block.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
                             int64_t sector_num, int nb_sectors, int *pnum);
 
 bool bdrv_is_read_only(BlockDriverState *bs);
+void bdrv_set_read_only(BlockDriverState *bs, bool read_only);
 bool bdrv_is_sg(BlockDriverState *bs);
 bool bdrv_is_inserted(BlockDriverState *bs);
 int bdrv_media_changed(BlockDriverState *bs);
-- 
2.9.3

A few block drivers will set the BDS read_only flag from their
.bdrv_open() function.  This means the bs->read_only flag could
be set after we enable copy_on_read, as the BDRV_O_COPY_ON_READ
flag check occurs prior to the call to bdrv->bdrv_open().

This adds an error return to bdrv_set_read_only(), and an error will be
return if we try to set the BDS to read_only while copy_on_read is
enabled.

This patch also changes the behavior of vvfat.  Before, vvfat could
override the drive 'readonly' flag with its own, internal 'rw' flag.

For instance, this -drive parameter would result in a writable image:

"-drive format=vvfat,dir=/tmp/vvfat,rw,if=virtio,readonly=on"

This is not correct.  Now, attempting to use the above -drive parameter
will result in an error (i.e., 'rw' is incompatible with 'readonly=on').

Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 0c5b4c1cc2c651471b131f21376dfd5ea24d2196.1491597120.git.jcody@redhat.com
---
 block.c               | 10 +++++++++-
 block/bochs.c         |  5 ++++-
 block/cloop.c         |  5 ++++-
 block/dmg.c           |  6 +++++-
 block/rbd.c           | 11 ++++++++++-
 block/vvfat.c         | 19 +++++++++++++++----
 include/block/block.h |  2 +-
 7 files changed, 48 insertions(+), 10 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ void path_combine(char *dest, int dest_size,
     }
 }
 
-void bdrv_set_read_only(BlockDriverState *bs, bool read_only)
+int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
 {
+    /* Do not set read_only if copy_on_read is enabled */
+    if (bs->copy_on_read && read_only) {
+        error_setg(errp, "Can't set node '%s' to r/o with copy-on-read enabled",
+                   bdrv_get_device_or_node_name(bs));
+        return -EINVAL;
+    }
+
     bs->read_only = read_only;
+    return 0;
 }
 
 void bdrv_get_full_backing_filename_from_filename(const char *backed,
diff --git a/block/bochs.c b/block/bochs.c
index XXXXXXX..XXXXXXX 100644
--- a/block/bochs.c
+++ b/block/bochs.c
@@ -XXX,XX +XXX,XX @@ static int bochs_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
-    bdrv_set_read_only(bs, true); /* no write support yet */
+    ret = bdrv_set_read_only(bs, true, errp); /* no write support yet */
+    if (ret < 0) {
+        return ret;
+    }
 
     ret = bdrv_pread(bs->file, 0, &bochs, sizeof(bochs));
     if (ret < 0) {
diff --git a/block/cloop.c b/block/cloop.c
index XXXXXXX..XXXXXXX 100644
--- a/block/cloop.c
+++ b/block/cloop.c
@@ -XXX,XX +XXX,XX @@ static int cloop_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
-    bdrv_set_read_only(bs, true);
+    ret = bdrv_set_read_only(bs, true, errp);
+    if (ret < 0) {
+        return ret;
+    }
 
     /* read header */
     ret = bdrv_pread(bs->file, 128, &s->block_size, 4);
diff --git a/block/dmg.c b/block/dmg.c
index XXXXXXX..XXXXXXX 100644
--- a/block/dmg.c
+++ b/block/dmg.c
@@ -XXX,XX +XXX,XX @@ static int dmg_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
+    ret = bdrv_set_read_only(bs, true, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
     block_module_load_one("dmg-bz2");
-    bdrv_set_read_only(bs, true);
 
     s->n_chunks = 0;
     s->offsets = s->lengths = s->sectors = s->sectorcounts = NULL;
diff --git a/block/rbd.c b/block/rbd.c
index XXXXXXX..XXXXXXX 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
         goto failed_shutdown;
     }
 
+    /* rbd_open is always r/w */
     r = rbd_open(s->io_ctx, s->name, &s->image, s->snap);
     if (r < 0) {
         error_setg_errno(errp, -r, "error reading header from %s", s->name);
         goto failed_open;
     }
 
-    bdrv_set_read_only(bs, (s->snap != NULL));
+    /* If we are using an rbd snapshot, we must be r/o, otherwise
+     * leave as-is */
+    if (s->snap != NULL) {
+        r = bdrv_set_read_only(bs, true, &local_err);
+        if (r < 0) {
+            error_propagate(errp, local_err);
+            goto failed_open;
+        }
+    }
 
     qemu_opts_del(opts);
     return 0;
diff --git a/block/vvfat.c b/block/vvfat.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vvfat.c
+++ b/block/vvfat.c
@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
 
     s->current_cluster=0xffffffff;
 
-    /* read only is the default for safety */
-    bdrv_set_read_only(bs, true);
     s->qcow = NULL;
     s->qcow_filename = NULL;
     s->fat2 = NULL;
@@ -XXX,XX +XXX,XX @@ static int vvfat_open(BlockDriverState *bs, QDict *options, int flags,
     s->sector_count = cyls * heads * secs - (s->first_sectors_number - 1);
 
     if (qemu_opt_get_bool(opts, "rw", false)) {
-        ret = enable_write_target(bs, errp);
+        if (!bdrv_is_read_only(bs)) {
+            ret = enable_write_target(bs, errp);
+            if (ret < 0) {
+                goto fail;
+            }
+        } else {
+            ret = -EPERM;
+            error_setg(errp,
+                       "Unable to set VVFAT to 'rw' when drive is read-only");
+            goto fail;
+        }
+    } else  {
+        /* read only is the default for safety */
+        ret = bdrv_set_read_only(bs, true, &local_err);
         if (ret < 0) {
+            error_propagate(errp, local_err);
             goto fail;
         }
-        bdrv_set_read_only(bs, false);
     }
 
     bs->total_sectors = cyls * heads * secs;
diff --git a/include/block/block.h b/include/block/block.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
                             int64_t sector_num, int nb_sectors, int *pnum);
 
 bool bdrv_is_read_only(BlockDriverState *bs);
-void bdrv_set_read_only(BlockDriverState *bs, bool read_only);
+int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp);
 bool bdrv_is_sg(BlockDriverState *bs);
 bool bdrv_is_inserted(BlockDriverState *bs);
 int bdrv_media_changed(BlockDriverState *bs);
-- 
2.9.3

The BDRV_O_ALLOW_RDWR flag allows / prohibits the changing of
the BDS 'read_only' state, but there are a few places where it
is ignored.  In the bdrv_set_read_only() helper, make sure to
honor the flag.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: be2e5fb2d285cbece2b6d06bed54a6f56520d251.1491597120.git.jcody@redhat.com
---
 block.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
         return -EINVAL;
     }
 
+    /* Do not clear read_only if it is prohibited */
+    if (!read_only && !(bs->open_flags & BDRV_O_ALLOW_RDWR)) {
+        error_setg(errp, "Node '%s' is read only",
+                   bdrv_get_device_or_node_name(bs));
+        return -EPERM;
+    }
+
     bs->read_only = read_only;
     return 0;
 }
-- 
2.9.3

Move bdrv_is_read_only() up with its friends.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Message-id: 73b2399459760c32506f9407efb9dddb3a2789de.1491597120.git.jcody@redhat.com
---
 block.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ void path_combine(char *dest, int dest_size,
     }
 }
 
+bool bdrv_is_read_only(BlockDriverState *bs)
+{
+    return bs->read_only;
+}
+
 int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
 {
     /* Do not set read_only if copy_on_read is enabled */
@@ -XXX,XX +XXX,XX @@ void bdrv_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr)
     *nb_sectors_ptr = nb_sectors < 0 ? 0 : nb_sectors;
 }
 
-bool bdrv_is_read_only(BlockDriverState *bs)
-{
-    return bs->read_only;
-}
-
 bool bdrv_is_sg(BlockDriverState *bs)
 {
     return bs->sg;
-- 
2.9.3

Introduce check function for setting read_only flags.  Will return < 0 on
error, with appropriate Error value set.  Does not alter any flags.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: e2bba34ac3bc76a0c42adc390413f358ae0566e8.1491597120.git.jcody@redhat.com
---
 block.c               | 14 +++++++++++++-
 include/block/block.h |  1 +
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ bool bdrv_is_read_only(BlockDriverState *bs)
     return bs->read_only;
 }
 
-int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
+int bdrv_can_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
 {
     /* Do not set read_only if copy_on_read is enabled */
     if (bs->copy_on_read && read_only) {
@@ -XXX,XX +XXX,XX @@ int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
         return -EPERM;
     }
 
+    return 0;
+}
+
+int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp)
+{
+    int ret = 0;
+
+    ret = bdrv_can_set_read_only(bs, read_only, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
     bs->read_only = read_only;
     return 0;
 }
diff --git a/include/block/block.h b/include/block/block.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
                             int64_t sector_num, int nb_sectors, int *pnum);
 
 bool bdrv_is_read_only(BlockDriverState *bs);
+int bdrv_can_set_read_only(BlockDriverState *bs, bool read_only, Error **errp);
 int bdrv_set_read_only(BlockDriverState *bs, bool read_only, Error **errp);
 bool bdrv_is_sg(BlockDriverState *bs);
 bool bdrv_is_inserted(BlockDriverState *bs);
-- 
2.9.3

Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 00aed7ffdd7be4b9ed9ce1007d50028a72b34ebe.1491597120.git.jcody@redhat.com
---
 block.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
     BlockDriver *drv;
     QemuOpts *opts;
     const char *value;
+    bool read_only;
 
     assert(reopen_state != NULL);
     assert(reopen_state->bs->drv != NULL);
@@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
         qdict_put(reopen_state->options, "driver", qstring_from_str(value));
     }
 
-    /* if we are to stay read-only, do not allow permission change
-     * to r/w */
-    if (!(reopen_state->bs->open_flags & BDRV_O_ALLOW_RDWR) &&
-        reopen_state->flags & BDRV_O_RDWR) {
-        error_setg(errp, "Node '%s' is read only",
-                   bdrv_get_device_or_node_name(reopen_state->bs));
+    /* If we are to stay read-only, do not allow permission change
+     * to r/w. Attempting to set to r/w may fail if either BDRV_O_ALLOW_RDWR is
+     * not set, or if the BDS still has copy_on_read enabled */
+    read_only = !(reopen_state->flags & BDRV_O_RDWR);
+    ret = bdrv_can_set_read_only(reopen_state->bs, read_only, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
         goto error;
     }
 
-- 
2.9.3

Update 'clientname' to be 'user', which tracks better with both
the QAPI and rados variable naming.

Update 'name' to be 'image_name', as it indicates the rbd image.
Naming it 'image' would have been ideal, but we are using that for
the rados_image_t value returned by rbd_open().

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: b7ec1fb2e1cf36f9b6911631447a5b0422590b7d.1491597120.git.jcody@redhat.com
---
 block/rbd.c | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index XXXXXXX..XXXXXXX 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVRBDState {
     rados_t cluster;
     rados_ioctx_t io_ctx;
     rbd_image_t image;
-    char *name;
+    char *image_name;
     char *snap;
 } BDRVRBDState;
 
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_create(const char *filename, QemuOpts *opts, Error **errp)
     int64_t bytes = 0;
     int64_t objsize;
     int obj_order = 0;
-    const char *pool, *name, *conf, *clientname, *keypairs;
+    const char *pool, *image_name, *conf, *user, *keypairs;
     const char *secretid;
     rados_t cluster;
     rados_ioctx_t io_ctx;
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_create(const char *filename, QemuOpts *opts, Error **errp)
      */
     pool       = qdict_get_try_str(options, "pool");
     conf       = qdict_get_try_str(options, "conf");
-    clientname = qdict_get_try_str(options, "user");
-    name       = qdict_get_try_str(options, "image");
+    user       = qdict_get_try_str(options, "user");
+    image_name = qdict_get_try_str(options, "image");
     keypairs   = qdict_get_try_str(options, "=keyvalue-pairs");
 
-    ret = rados_create(&cluster, clientname);
+    ret = rados_create(&cluster, user);
     if (ret < 0) {
         error_setg_errno(errp, -ret, "error initializing");
         goto exit;
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_create(const char *filename, QemuOpts *opts, Error **errp)
         goto shutdown;
     }
 
-    ret = rbd_create(io_ctx, name, bytes, &obj_order);
+    ret = rbd_create(io_ctx, image_name, bytes, &obj_order);
     if (ret < 0) {
         error_setg_errno(errp, -ret, "error rbd create");
     }
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
                          Error **errp)
 {
     BDRVRBDState *s = bs->opaque;
-    const char *pool, *snap, *conf, *clientname, *name, *keypairs;
+    const char *pool, *snap, *conf, *user, *image_name, *keypairs;
     const char *secretid;
     QemuOpts *opts;
     Error *local_err = NULL;
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
     pool           = qemu_opt_get(opts, "pool");
     conf           = qemu_opt_get(opts, "conf");
     snap           = qemu_opt_get(opts, "snapshot");
-    clientname     = qemu_opt_get(opts, "user");
-    name           = qemu_opt_get(opts, "image");
+    user           = qemu_opt_get(opts, "user");
+    image_name     = qemu_opt_get(opts, "image");
     keypairs       = qemu_opt_get(opts, "=keyvalue-pairs");
 
-    if (!pool || !name) {
+    if (!pool || !image_name) {
         error_setg(errp, "Parameters 'pool' and 'image' are required");
         r = -EINVAL;
         goto failed_opts;
     }
 
-    r = rados_create(&s->cluster, clientname);
+    r = rados_create(&s->cluster, user);
     if (r < 0) {
         error_setg_errno(errp, -r, "error initializing");
         goto failed_opts;
     }
 
     s->snap = g_strdup(snap);
-    s->name = g_strdup(name);
+    s->image_name = g_strdup(image_name);
 
     /* try default location when conf=NULL, but ignore failure */
     r = rados_conf_read_file(s->cluster, conf);
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     /* rbd_open is always r/w */
-    r = rbd_open(s->io_ctx, s->name, &s->image, s->snap);
+    r = rbd_open(s->io_ctx, s->image_name, &s->image, s->snap);
     if (r < 0) {
-        error_setg_errno(errp, -r, "error reading header from %s", s->name);
+        error_setg_errno(errp, -r, "error reading header from %s",
+                         s->image_name);
         goto failed_open;
     }
 
@@ -XXX,XX +XXX,XX @@ failed_open:
 failed_shutdown:
     rados_shutdown(s->cluster);
     g_free(s->snap);
-    g_free(s->name);
+    g_free(s->image_name);
 failed_opts:
     qemu_opts_del(opts);
     g_free(mon_host);
@@ -XXX,XX +XXX,XX @@ static void qemu_rbd_close(BlockDriverState *bs)
     rbd_close(s->image);
     rados_ioctx_destroy(s->io_ctx);
     g_free(s->snap);
-    g_free(s->name);
+    g_free(s->image_name);
     rados_shutdown(s->cluster);
 }
 
-- 
2.9.3

This adds support for reopen in rbd, for changing between r/w and r/o.

Note, that this is only a flag change, but we will block a change from
r/o to r/w if we are using an RBD internal snapshot.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: d4e87539167ec6527d44c97b164eabcccf96e4f3.1491597120.git.jcody@redhat.com
---
 block/rbd.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/block/rbd.c b/block/rbd.c
index XXXXXXX..XXXXXXX 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -XXX,XX +XXX,XX @@ failed_opts:
     return r;
 }
 
+
+/* Since RBD is currently always opened R/W via the API,
+ * we just need to check if we are using a snapshot or not, in
+ * order to determine if we will allow it to be R/W */
+static int qemu_rbd_reopen_prepare(BDRVReopenState *state,
+                                   BlockReopenQueue *queue, Error **errp)
+{
+    BDRVRBDState *s = state->bs->opaque;
+    int ret = 0;
+
+    if (s->snap && state->flags & BDRV_O_RDWR) {
+        error_setg(errp,
+                   "Cannot change node '%s' to r/w when using RBD snapshot",
+                   bdrv_get_device_or_node_name(state->bs));
+        ret = -EINVAL;
+    }
+
+    return ret;
+}
+
 static void qemu_rbd_close(BlockDriverState *bs)
 {
     BDRVRBDState *s = bs->opaque;
@@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_rbd = {
     .bdrv_parse_filename    = qemu_rbd_parse_filename,
     .bdrv_file_open         = qemu_rbd_open,
     .bdrv_close             = qemu_rbd_close,
+    .bdrv_reopen_prepare    = qemu_rbd_reopen_prepare,
     .bdrv_create            = qemu_rbd_create,
     .bdrv_has_zero_init     = bdrv_has_zero_init_1,
     .bdrv_get_info          = qemu_rbd_getinfo,
-- 
2.9.3

For the tests that use the common.qemu functions for running a QEMU
process, _cleanup_qemu must be called in the exit function.

If it is not, if the qemu process aborts, then not all of the droppings
are cleaned up (e.g. pidfile, fifos).

This updates those tests that did not have a cleanup in qemu-iotests.

(I swapped spaces for tabs in test 102 as well)

Reported-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Message-id: d59c2f6ad6c1da8b9b3c7f357c94a7122ccfc55a.1492544096.git.jcody@redhat.com
---
 tests/qemu-iotests/028 |  1 +
 tests/qemu-iotests/094 | 11 ++++++++---
 tests/qemu-iotests/102 |  5 +++--
 tests/qemu-iotests/109 |  1 +
 tests/qemu-iotests/117 |  1 +
 tests/qemu-iotests/130 |  1 +
 tests/qemu-iotests/140 |  1 +
 tests/qemu-iotests/141 |  1 +
 tests/qemu-iotests/143 |  1 +
 tests/qemu-iotests/156 |  1 +
 10 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/028 b/tests/qemu-iotests/028
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/028
+++ b/tests/qemu-iotests/028
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
     rm -f "${TEST_IMG}.copy"
     _cleanup_test_img
 }
diff --git a/tests/qemu-iotests/094 b/tests/qemu-iotests/094
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/094
+++ b/tests/qemu-iotests/094
@@ -XXX,XX +XXX,XX @@ echo "QA output created by $seq"
 here="$PWD"
 status=1	# failure is the default!
 
-trap "exit \$status" 0 1 2 3 15
+_cleanup()
+{
+    _cleanup_qemu
+    _cleanup_test_img
+    rm -f "$TEST_DIR/source.$IMGFMT"
+}
+
+trap "_cleanup; exit \$status" 0 1 2 3 15
 
 # get standard environment, filters and checks
 . ./common.rc
@@ -XXX,XX +XXX,XX @@ _send_qemu_cmd $QEMU_HANDLE \
 
 wait=1 _cleanup_qemu
 
-_cleanup_test_img
-rm -f "$TEST_DIR/source.$IMGFMT"
 
 # success, all done
 echo '*** done'
diff --git a/tests/qemu-iotests/102 b/tests/qemu-iotests/102
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/102
+++ b/tests/qemu-iotests/102
@@ -XXX,XX +XXX,XX @@ seq=$(basename $0)
 echo "QA output created by $seq"
 
 here=$PWD
-status=1	# failure is the default!
+status=1    # failure is the default!
 
 _cleanup()
 {
-	_cleanup_test_img
+    _cleanup_qemu
+    _cleanup_test_img
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
diff --git a/tests/qemu-iotests/109 b/tests/qemu-iotests/109
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/109
+++ b/tests/qemu-iotests/109
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
     rm -f $TEST_IMG.src
 	_cleanup_test_img
 }
diff --git a/tests/qemu-iotests/117 b/tests/qemu-iotests/117
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/117
+++ b/tests/qemu-iotests/117
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
 	_cleanup_test_img
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/130
+++ b/tests/qemu-iotests/130
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
     _cleanup_test_img
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
diff --git a/tests/qemu-iotests/140 b/tests/qemu-iotests/140
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/140
+++ b/tests/qemu-iotests/140
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
     _cleanup_test_img
     rm -f "$TEST_DIR/nbd"
 }
diff --git a/tests/qemu-iotests/141 b/tests/qemu-iotests/141
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/141
+++ b/tests/qemu-iotests/141
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
     _cleanup_test_img
     rm -f "$TEST_DIR/{b,m,o}.$IMGFMT"
 }
diff --git a/tests/qemu-iotests/143 b/tests/qemu-iotests/143
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/143
+++ b/tests/qemu-iotests/143
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
     rm -f "$TEST_DIR/nbd"
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
diff --git a/tests/qemu-iotests/156 b/tests/qemu-iotests/156
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/156
+++ b/tests/qemu-iotests/156
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
+    _cleanup_qemu
     rm -f "$TEST_IMG{,.target}{,.backing,.overlay}"
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
-- 
2.9.3

The following changes since commit 8e6c70b9d4a1b1f3011805947925cfdb31642f7f:

Merge tag 'kraxel-20220614-pull-request' of git://git.kraxel.org/qemu into staging (2022-06-14 06:21:46 -0700)

are available in the Git repository at:

https://gitlab.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 99b969fbe105117f5af6060d3afef40ca39cc9c1:

linux-aio: explain why max batch is checked in laio_io_unplug() (2022-06-15 16:43:42 +0100)

----------------------------------------------------------------
Pull request

This pull request includes an important aio=native I/O stall fix, the
experimental vifo-user server, the io_uring_register_ring_fd() optimization for
aio=io_uring, and an update to Vladimir Sementsov-Ogievskiy's maintainership
details.

----------------------------------------------------------------

Jagannathan Raman (14):
  qdev: unplug blocker for devices
  remote/machine: add HotplugHandler for remote machine
  remote/machine: add vfio-user property
  vfio-user: build library
  vfio-user: define vfio-user-server object
  vfio-user: instantiate vfio-user context
  vfio-user: find and init PCI device
  vfio-user: run vfio-user context
  vfio-user: handle PCI config space accesses
  vfio-user: IOMMU support for remote device
  vfio-user: handle DMA mappings
  vfio-user: handle PCI BAR accesses
  vfio-user: handle device interrupts
  vfio-user: handle reset of remote device

Sam Li (1):
  Use io_uring_register_ring_fd() to skip fd operations

Stefan Hajnoczi (2):
  linux-aio: fix unbalanced plugged counter in laio_io_unplug()
  linux-aio: explain why max batch is checked in laio_io_unplug()

Vladimir Sementsov-Ogievskiy (1):
  MAINTAINERS: update Vladimir's address and repositories

-- 
2.36.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-id: 20220526115432.138384-1-vsementsov@yandex-team.ru
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 MAINTAINERS | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index XXXXXXX..XXXXXXX 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: scsi/*
 
 Block Jobs
 M: John Snow <jsnow@redhat.com>
-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
+M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 L: qemu-block@nongnu.org
 S: Supported
 F: blockjob.c
@@ -XXX,XX +XXX,XX @@ F: block/aio_task.c
 F: util/qemu-co-shared-resource.c
 F: include/qemu/co-shared-resource.h
 T: git https://gitlab.com/jsnow/qemu.git jobs
-T: git https://src.openvz.org/scm/~vsementsov/qemu.git jobs
+T: git https://gitlab.com/vsementsov/qemu.git block
 
 Block QAPI, monitor, command line
 M: Markus Armbruster <armbru@redhat.com>
@@ -XXX,XX +XXX,XX @@ F: include/hw/cxl/
 
 Dirty Bitmaps
 M: Eric Blake <eblake@redhat.com>
-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
+M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 R: John Snow <jsnow@redhat.com>
 L: qemu-block@nongnu.org
 S: Supported
@@ -XXX,XX +XXX,XX @@ F: util/hbitmap.c
 F: tests/unit/test-hbitmap.c
 F: docs/interop/bitmaps.rst
 T: git https://repo.or.cz/qemu/ericb.git bitmaps
+T: git https://gitlab.com/vsementsov/qemu.git block
 
 Character device backends
 M: Marc-André Lureau <marcandre.lureau@redhat.com>
@@ -XXX,XX +XXX,XX @@ F: scripts/*.py
 F: tests/*.py
 
 Benchmark util
-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
+M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 S: Maintained
 F: scripts/simplebench/
-T: git https://src.openvz.org/scm/~vsementsov/qemu.git simplebench
+T: git https://gitlab.com/vsementsov/qemu.git simplebench
 
 Transactions helper
-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
+M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 S: Maintained
 F: include/qemu/transactions.h
 F: util/transactions.c
+T: git https://gitlab.com/vsementsov/qemu.git block
 
 QAPI
 M: Markus Armbruster <armbru@redhat.com>
@@ -XXX,XX +XXX,XX @@ F: block/iscsi-opts.c
 
 Network Block Device (NBD)
 M: Eric Blake <eblake@redhat.com>
-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
+M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 L: qemu-block@nongnu.org
 S: Maintained
 F: block/nbd*
@@ -XXX,XX +XXX,XX @@ F: docs/interop/nbd.txt
 F: docs/tools/qemu-nbd.rst
 F: tests/qemu-iotests/tests/*nbd*
 T: git https://repo.or.cz/qemu/ericb.git nbd
-T: git https://src.openvz.org/scm/~vsementsov/qemu.git nbd
+T: git https://gitlab.com/vsementsov/qemu.git block
 
 NFS
 M: Peter Lieven <pl@kamp.de>
@@ -XXX,XX +XXX,XX @@ F: block/dmg.c
 parallels
 M: Stefan Hajnoczi <stefanha@redhat.com>
 M: Denis V. Lunev <den@openvz.org>
-M: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
+M: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
 L: qemu-block@nongnu.org
 S: Supported
 F: block/parallels.c
 F: block/parallels-ext.c
 F: docs/interop/parallels.txt
-T: git https://src.openvz.org/scm/~vsementsov/qemu.git parallels
+T: git https://gitlab.com/vsementsov/qemu.git block
 
 qed
 M: Stefan Hajnoczi <stefanha@redhat.com>
-- 
2.36.1

From: Sam Li <faithilikerun@gmail.com>

Linux recently added a new io_uring(7) optimization API that QEMU
doesn't take advantage of yet. The liburing library that QEMU uses
has added a corresponding new API calling io_uring_register_ring_fd().
When this API is called after creating the ring, the io_uring_submit()
library function passes a flag to the io_uring_enter(2) syscall
allowing it to skip the ring file descriptor fdget()/fdput()
operations. This saves some CPU cycles.

Signed-off-by: Sam Li <faithilikerun@gmail.com>
Message-id: 20220531105011.111082-1-faithilikerun@gmail.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 meson.build      |  1 +
 block/io_uring.c | 12 +++++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/meson.build b/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/meson.build
+++ b/meson.build
@@ -XXX,XX +XXX,XX @@ config_host_data.set('CONFIG_LIBNFS', libnfs.found())
 config_host_data.set('CONFIG_LIBSSH', libssh.found())
 config_host_data.set('CONFIG_LINUX_AIO', libaio.found())
 config_host_data.set('CONFIG_LINUX_IO_URING', linux_io_uring.found())
+config_host_data.set('CONFIG_LIBURING_REGISTER_RING_FD', cc.has_function('io_uring_register_ring_fd', prefix: '#include <liburing.h>', dependencies:linux_io_uring))
 config_host_data.set('CONFIG_LIBPMEM', libpmem.found())
 config_host_data.set('CONFIG_NUMA', numa.found())
 config_host_data.set('CONFIG_OPENGL', opengl.found())
diff --git a/block/io_uring.c b/block/io_uring.c
index XXXXXXX..XXXXXXX 100644
--- a/block/io_uring.c
+++ b/block/io_uring.c
@@ -XXX,XX +XXX,XX @@
 #include "qapi/error.h"
 #include "trace.h"
 
+
 /* io_uring ring size */
 #define MAX_ENTRIES 128
 
@@ -XXX,XX +XXX,XX @@ LuringState *luring_init(Error **errp)
     }
 
     ioq_init(&s->io_q);
+#ifdef CONFIG_LIBURING_REGISTER_RING_FD
+    if (io_uring_register_ring_fd(&s->ring) < 0) {
+        /*
+         * Only warn about this error: we will fallback to the non-optimized
+         * io_uring operations.
+         */
+        warn_report("failed to register linux io_uring ring file descriptor");
+    }
+#endif
+
     return s;
-
 }
 
 void luring_cleanup(LuringState *s)
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Add blocker to prevent hot-unplug of devices

TYPE_VFIO_USER_SERVER, which is introduced shortly, attaches itself to a
PCIDevice on which it depends. If the attached PCIDevice gets removed
while the server in use, it could cause it crash. To prevent this,
TYPE_VFIO_USER_SERVER adds an unplug blocker for the PCIDevice.

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: c41ef80b7cc063314d629737bed2159e5713f2e0.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 include/hw/qdev-core.h | 29 +++++++++++++++++++++++++++++
 hw/core/qdev.c         | 24 ++++++++++++++++++++++++
 softmmu/qdev-monitor.c |  4 ++++
 3 files changed, 57 insertions(+)

diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
index XXXXXXX..XXXXXXX 100644
--- a/include/hw/qdev-core.h
+++ b/include/hw/qdev-core.h
@@ -XXX,XX +XXX,XX @@ struct DeviceState {
     int instance_id_alias;
     int alias_required_for_version;
     ResettableState reset;
+    GSList *unplug_blockers;
 };
 
 struct DeviceListener {
@@ -XXX,XX +XXX,XX @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,
 void qdev_machine_creation_done(void);
 bool qdev_machine_modified(void);
 
+/**
+ * qdev_add_unplug_blocker: Add an unplug blocker to a device
+ *
+ * @dev: Device to be blocked from unplug
+ * @reason: Reason for blocking
+ */
+void qdev_add_unplug_blocker(DeviceState *dev, Error *reason);
+
+/**
+ * qdev_del_unplug_blocker: Remove an unplug blocker from a device
+ *
+ * @dev: Device to be unblocked
+ * @reason: Pointer to the Error used with qdev_add_unplug_blocker.
+ *          Used as a handle to lookup the blocker for deletion.
+ */
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason);
+
+/**
+ * qdev_unplug_blocked: Confirm if a device is blocked from unplug
+ *
+ * @dev: Device to be tested
+ * @reason: Returns one of the reasons why the device is blocked,
+ *          if any
+ *
+ * Returns: true if device is blocked from unplug, false otherwise
+ */
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp);
+
 /**
  * GpioPolarity: Polarity of a GPIO line
  *
diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -XXX,XX +XXX,XX @@ char *qdev_get_dev_path(DeviceState *dev)
     return NULL;
 }
 
+void qdev_add_unplug_blocker(DeviceState *dev, Error *reason)
+{
+    dev->unplug_blockers = g_slist_prepend(dev->unplug_blockers, reason);
+}
+
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason)
+{
+    dev->unplug_blockers = g_slist_remove(dev->unplug_blockers, reason);
+}
+
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (dev->unplug_blockers) {
+        error_propagate(errp, error_copy(dev->unplug_blockers->data));
+        return true;
+    }
+
+    return false;
+}
+
 static bool device_get_realized(Object *obj, Error **errp)
 {
     DeviceState *dev = DEVICE(obj);
@@ -XXX,XX +XXX,XX @@ static void device_finalize(Object *obj)
 
     DeviceState *dev = DEVICE(obj);
 
+    g_assert(!dev->unplug_blockers);
+
     QLIST_FOREACH_SAFE(ngl, &dev->gpios, node, next) {
         QLIST_REMOVE(ngl, node);
         qemu_free_irqs(ngl->in, ngl->num_in);
diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c
index XXXXXXX..XXXXXXX 100644
--- a/softmmu/qdev-monitor.c
+++ b/softmmu/qdev-monitor.c
@@ -XXX,XX +XXX,XX @@ void qdev_unplug(DeviceState *dev, Error **errp)
     HotplugHandlerClass *hdc;
     Error *local_err = NULL;
 
+    if (qdev_unplug_blocked(dev, errp)) {
+        return;
+    }
+
     if (dev->parent_bus && !qbus_is_hotpluggable(dev->parent_bus)) {
         error_setg(errp, QERR_BUS_NO_HOTPLUG, dev->parent_bus->name);
         return;
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Allow hotplugging of PCI(e) devices to remote machine

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: d1e6cfa0afb528ad343758f9b1d918be0175c5e5.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/remote/machine.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

From: Jagannathan Raman <jag.raman@oracle.com>

Add vfio-user to x-remote machine. It is a boolean, which indicates if
the machine supports vfio-user protocol. The machine configures the bus
differently vfio-user and multiprocess protocols, so this property
informs it on how to configure the bus.

This property should be short lived. Once vfio-user fully replaces
multiprocess, this property could be removed.

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 5d51a152a419cbda35d070b8e49b772b60a7230a.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 include/hw/remote/machine.h |  2 ++
 hw/remote/machine.c         | 23 +++++++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/include/hw/remote/machine.h b/include/hw/remote/machine.h
index XXXXXXX..XXXXXXX 100644
--- a/include/hw/remote/machine.h
+++ b/include/hw/remote/machine.h
@@ -XXX,XX +XXX,XX @@ struct RemoteMachineState {
 
     RemotePCIHost *host;
     RemoteIOHubState iohub;
+
+    bool vfio_user;
 };
 
 /* Used to pass to co-routine device and ioc. */
diff --git a/hw/remote/machine.c b/hw/remote/machine.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/machine.c
+++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@ static void remote_machine_init(MachineState *machine)
     qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
 }
 
+static bool remote_machine_get_vfio_user(Object *obj, Error **errp)
+{
+    RemoteMachineState *s = REMOTE_MACHINE(obj);
+
+    return s->vfio_user;
+}
+
+static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp)
+{
+    RemoteMachineState *s = REMOTE_MACHINE(obj);
+
+    if (phase_check(PHASE_MACHINE_CREATED)) {
+        error_setg(errp, "Error enabling vfio-user - machine already created");
+        return;
+    }
+
+    s->vfio_user = value;
+}
+
 static void remote_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -XXX,XX +XXX,XX @@ static void remote_machine_class_init(ObjectClass *oc, void *data)
     mc->desc = "Experimental remote machine";
 
     hc->unplug = qdev_simple_device_unplug_cb;
+
+    object_class_property_add_bool(oc, "vfio-user",
+                                   remote_machine_get_vfio_user,
+                                   remote_machine_set_vfio_user);
 }
 
 static const TypeInfo remote_machine = {
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

add the libvfio-user library as a submodule. build it as a meson
subproject.

libvfio-user is distributed with BSD 3-Clause license and
json-c with MIT (Expat) license

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: c2adec87958b081d1dc8775d4aa05c897912f025.1655151679.git.jag.raman@oracle.com

[Changed submodule URL to QEMU's libvfio-user mirror on GitLab. The QEMU
project mirrors its dependencies so that it can provide full source code
even in the event that its dependencies become unavailable. Note that
the mirror repo is manually updated, so please contact me to make newer
libvfio-user commits available. If I become a bottleneck we can set up a
cronjob.

Updated scripts/meson-buildoptions.sh to match the meson_options.txt
change. Failure to do so can result in scripts/meson-buildoptions.sh
being modified by the build system later on and you end up with a dirty
working tree.
--Stefan]

diff --git a/MAINTAINERS b/MAINTAINERS
index XXXXXXX..XXXXXXX 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: hw/remote/proxy-memory-listener.c
 F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
+F: subprojects/libvfio-user
 
 EBPF:
 M: Jason Wang <jasowang@redhat.com>
diff --git a/meson_options.txt b/meson_options.txt
index XXXXXXX..XXXXXXX 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -XXX,XX +XXX,XX @@ option('cfi_debug', type: 'boolean', value: 'false',
        description: 'Verbose errors in case of CFI violation')
 option('multiprocess', type: 'feature', value: 'auto',
        description: 'Out of process device emulation support')
+option('vfio_user_server', type: 'feature', value: 'disabled',
+       description: 'vfio-user server support')
 option('dbus_display', type: 'feature', value: 'auto',
        description: '-display dbus support')
 option('tpm', type : 'feature', value : 'auto',
diff --git a/configure b/configure
index XXXXXXX..XXXXXXX 100755
--- a/configure
+++ b/configure
@@ -XXX,XX +XXX,XX @@ meson_args=""
 ninja=""
 bindir="bin"
 skip_meson=no
+vfio_user_server="disabled"
 
 # The following Meson options are handled manually (still they
 # are included in the automatically generated help message)
@@ -XXX,XX +XXX,XX @@ for opt do
   ;;
   --disable-blobs) meson_option_parse --disable-install-blobs ""
   ;;
+  --enable-vfio-user-server) vfio_user_server="enabled"
+  ;;
+  --disable-vfio-user-server) vfio_user_server="disabled"
+  ;;
   --enable-tcmalloc) meson_option_parse --enable-malloc=tcmalloc tcmalloc
   ;;
   --enable-jemalloc) meson_option_parse --enable-malloc=jemalloc jemalloc
@@ -XXX,XX +XXX,XX @@ write_container_target_makefile() {
 
 
 
+##########################################
+# check for vfio_user_server
+
+case "$vfio_user_server" in
+  enabled )
+    if test "$git_submodules_action" != "ignore"; then
+      git_submodules="${git_submodules} subprojects/libvfio-user"
+    fi
+    ;;
+esac
+
 ##########################################
 # End of CC checks
 # After here, no more $cc or $ld runs
@@ -XXX,XX +XXX,XX @@ if test "$skip_meson" = no; then
   test "$slirp" != auto && meson_option_add "-Dslirp=$slirp"
   test "$smbd" != '' && meson_option_add "-Dsmbd=$smbd"
   test "$tcg" != enabled && meson_option_add "-Dtcg=$tcg"
+  test "$vfio_user_server" != auto && meson_option_add "-Dvfio_user_server=$vfio_user_server"
   run_meson() {
     NINJA=$ninja $meson setup --prefix "$prefix" "$@" $cross_arg "$PWD" "$source_path"
   }
diff --git a/meson.build b/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/meson.build
+++ b/meson.build
@@ -XXX,XX +XXX,XX @@ multiprocess_allowed = get_option('multiprocess') \
   .require(targetos == 'linux', error_message: 'Multiprocess QEMU is supported only on Linux') \
   .allowed()
 
+vfio_user_server_allowed = get_option('vfio_user_server') \
+  .require(targetos == 'linux', error_message: 'vfio-user server is supported only on Linux') \
+  .allowed()
+
 have_tpm = get_option('tpm') \
   .require(targetos != 'windows', error_message: 'TPM emulation only available on POSIX systems') \
   .allowed()
@@ -XXX,XX +XXX,XX @@ host_kconfig = \
   (have_virtfs ? ['CONFIG_VIRTFS=y'] : []) + \
   ('CONFIG_LINUX' in config_host ? ['CONFIG_LINUX=y'] : []) + \
   (have_pvrdma ? ['CONFIG_PVRDMA=y'] : []) + \
-  (multiprocess_allowed ? ['CONFIG_MULTIPROCESS_ALLOWED=y'] : [])
+  (multiprocess_allowed ? ['CONFIG_MULTIPROCESS_ALLOWED=y'] : []) + \
+  (vfio_user_server_allowed ? ['CONFIG_VFIO_USER_SERVER_ALLOWED=y'] : [])
 
 ignored = [ 'TARGET_XML_FILES', 'TARGET_ABI_DIR', 'TARGET_ARCH' ]
 
@@ -XXX,XX +XXX,XX @@ if have_system
   endif
 endif
 
+libvfio_user_dep = not_found
+if have_system and vfio_user_server_allowed
+  have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/meson.build')
+
+  if not have_internal
+    error('libvfio-user source not found - please pull git submodule')
+  endif
+
+  libvfio_user_proj = subproject('libvfio-user')
+
+  libvfio_user_lib = libvfio_user_proj.get_variable('libvfio_user_dep')
+
+  libvfio_user_dep = declare_dependency(dependencies: [libvfio_user_lib])
+endif
+
 fdt = not_found
 if have_system
   fdt_opt = get_option('fdt')
@@ -XXX,XX +XXX,XX @@ summary_info += {'target list':       ' '.join(target_dirs)}
 if have_system
   summary_info += {'default devices':   get_option('default_devices')}
   summary_info += {'out of process emulation': multiprocess_allowed}
+  summary_info += {'vfio-user server': vfio_user_server_allowed}
 endif
 summary(summary_info, bool_yn: true, section: 'Targets and accelerators')
 
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index XXXXXXX..XXXXXXX 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -XXX,XX +XXX,XX @@ build-system-centos:
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
       --enable-modules --enable-trace-backends=dtrace --enable-docs
+      --enable-vfio-user-server
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
diff --git a/.gitmodules b/.gitmodules
index XXXXXXX..XXXXXXX 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -XXX,XX +XXX,XX @@
 [submodule "tests/lcitool/libvirt-ci"]
 	path = tests/lcitool/libvirt-ci
 	url = https://gitlab.com/libvirt/libvirt-ci.git
+[submodule "subprojects/libvfio-user"]
+	path = subprojects/libvfio-user
+	url = https://gitlab.com/qemu-project/libvfio-user.git
diff --git a/Kconfig.host b/Kconfig.host
index XXXXXXX..XXXXXXX 100644
--- a/Kconfig.host
+++ b/Kconfig.host
@@ -XXX,XX +XXX,XX @@ config MULTIPROCESS_ALLOWED
 config FUZZ
     bool
     select SPARSE_MEM
+
+config VFIO_USER_SERVER_ALLOWED
+    bool
+    imply VFIO_USER_SERVER
diff --git a/hw/remote/Kconfig b/hw/remote/Kconfig
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/Kconfig
+++ b/hw/remote/Kconfig
@@ -XXX,XX +XXX,XX @@ config MULTIPROCESS
     bool
     depends on PCI && PCI_EXPRESS && KVM
     select REMOTE_PCIHOST
+
+config VFIO_USER_SERVER
+    bool
+    depends on MULTIPROCESS
diff --git a/hw/remote/meson.build b/hw/remote/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/meson.build
+++ b/hw/remote/meson.build
@@ -XXX,XX +XXX,XX @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
 
+remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: libvfio_user_dep)
+
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c'))
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
 
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index XXXXXXX..XXXXXXX 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -XXX,XX +XXX,XX @@ meson_options_help() {
   printf "%s\n" '  usb-redir       libusbredir support'
   printf "%s\n" '  vde             vde network backend support'
   printf "%s\n" '  vdi             vdi image format support'
+  printf "%s\n" '  vfio-user-server'
+  printf "%s\n" '                  vfio-user server support'
   printf "%s\n" '  vhost-crypto    vhost-user crypto backend support'
   printf "%s\n" '  vhost-kernel    vhost kernel backend support'
   printf "%s\n" '  vhost-net       vhost-net kernel acceleration support'
@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
     --disable-vde) printf "%s" -Dvde=disabled ;;
     --enable-vdi) printf "%s" -Dvdi=enabled ;;
     --disable-vdi) printf "%s" -Dvdi=disabled ;;
+    --enable-vfio-user-server) printf "%s" -Dvfio_user_server=enabled ;;
+    --disable-vfio-user-server) printf "%s" -Dvfio_user_server=disabled ;;
     --enable-vhost-crypto) printf "%s" -Dvhost_crypto=enabled ;;
     --disable-vhost-crypto) printf "%s" -Dvhost_crypto=disabled ;;
     --enable-vhost-kernel) printf "%s" -Dvhost_kernel=enabled ;;
diff --git a/subprojects/libvfio-user b/subprojects/libvfio-user
new file mode 160000
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/subprojects/libvfio-user
@@ -0,0 +1 @@
+Subproject commit 0b28d205572c80b568a1003db2c8f37ca333e4d7
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -XXX,XX +XXX,XX @@ RUN dnf update -y && \
         libbpf-devel \
         libcacard-devel \
         libcap-ng-devel \
+        libcmocka-devel \
         libcurl-devel \
         libdrm-devel \
         libepoxy-devel \
@@ -XXX,XX +XXX,XX @@ RUN dnf update -y && \
         libgcrypt-devel \
         libiscsi-devel \
         libjpeg-devel \
+        json-c-devel \
         libnfs-devel \
         libpmem-devel \
         libpng-devel \
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Define vfio-user object which is remote process server for QEMU. Setup
object initialization functions and properties necessary to instantiate
the object

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: e45a17001e9b38f451543a664ababdf860e5f2f2.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 MAINTAINERS                 |   1 +
 qapi/qom.json               |  20 +++-
 include/hw/remote/machine.h |   2 +
 hw/remote/machine.c         |  27 +++++
 hw/remote/vfio-user-obj.c   | 210 ++++++++++++++++++++++++++++++++++++
 hw/remote/meson.build       |   1 +
 hw/remote/trace-events      |   3 +
 7 files changed, 262 insertions(+), 2 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c

diff --git a/MAINTAINERS b/MAINTAINERS
index XXXXXXX..XXXXXXX 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: subprojects/libvfio-user
+F: hw/remote/vfio-user-obj.c
 
 EBPF:
 M: Jason Wang <jasowang@redhat.com>
diff --git a/qapi/qom.json b/qapi/qom.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -XXX,XX +XXX,XX @@
 { 'struct': 'RemoteObjectProperties',
   'data': { 'fd': 'str', 'devid': 'str' } }
 
+##
+# @VfioUserServerProperties:
+#
+# Properties for x-vfio-user-server objects.
+#
+# @socket: socket to be used by the libvfio-user library
+#
+# @device: the ID of the device to be emulated at the server
+#
+# Since: 7.1
+##
+{ 'struct': 'VfioUserServerProperties',
+  'data': { 'socket': 'SocketAddress', 'device': 'str' } }
+
 ##
 # @RngProperties:
 #
@@ -XXX,XX +XXX,XX @@
     'tls-creds-psk',
     'tls-creds-x509',
     'tls-cipher-suites',
-    { 'name': 'x-remote-object', 'features': [ 'unstable' ] }
+    { 'name': 'x-remote-object', 'features': [ 'unstable' ] },
+    { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] }
   ] }
 
 ##
@@ -XXX,XX +XXX,XX @@
       'tls-creds-psk':              'TlsCredsPskProperties',
       'tls-creds-x509':             'TlsCredsX509Properties',
       'tls-cipher-suites':          'TlsCredsProperties',
-      'x-remote-object':            'RemoteObjectProperties'
+      'x-remote-object':            'RemoteObjectProperties',
+      'x-vfio-user-server':         'VfioUserServerProperties'
   } }
 
 ##
diff --git a/include/hw/remote/machine.h b/include/hw/remote/machine.h
index XXXXXXX..XXXXXXX 100644
--- a/include/hw/remote/machine.h
+++ b/include/hw/remote/machine.h
@@ -XXX,XX +XXX,XX @@ struct RemoteMachineState {
     RemoteIOHubState iohub;
 
     bool vfio_user;
+
+    bool auto_shutdown;
 };
 
 /* Used to pass to co-routine device and ioc. */
diff --git a/hw/remote/machine.c b/hw/remote/machine.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/machine.c
+++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@ static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp)
     s->vfio_user = value;
 }
 
+static bool remote_machine_get_auto_shutdown(Object *obj, Error **errp)
+{
+    RemoteMachineState *s = REMOTE_MACHINE(obj);
+
+    return s->auto_shutdown;
+}
+
+static void remote_machine_set_auto_shutdown(Object *obj, bool value,
+                                             Error **errp)
+{
+    RemoteMachineState *s = REMOTE_MACHINE(obj);
+
+    s->auto_shutdown = value;
+}
+
+static void remote_machine_instance_init(Object *obj)
+{
+    RemoteMachineState *s = REMOTE_MACHINE(obj);
+
+    s->auto_shutdown = true;
+}
+
 static void remote_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -XXX,XX +XXX,XX @@ static void remote_machine_class_init(ObjectClass *oc, void *data)
     object_class_property_add_bool(oc, "vfio-user",
                                    remote_machine_get_vfio_user,
                                    remote_machine_set_vfio_user);
+
+    object_class_property_add_bool(oc, "auto-shutdown",
+                                   remote_machine_get_auto_shutdown,
+                                   remote_machine_set_auto_shutdown);
 }
 
 static const TypeInfo remote_machine = {
     .name = TYPE_REMOTE_MACHINE,
     .parent = TYPE_MACHINE,
     .instance_size = sizeof(RemoteMachineState),
+    .instance_init = remote_machine_instance_init,
     .class_init = remote_machine_class_init,
     .interfaces = (InterfaceInfo[]) {
         { TYPE_HOTPLUG_HANDLER },
diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
+/**
+ * QEMU vfio-user-server server object
+ *
+ * Copyright © 2022 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL-v2, version 2 or later.
+ *
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+/**
+ * Usage: add options:
+ *     -machine x-remote,vfio-user=on,auto-shutdown=on
+ *     -device <PCI-device>,id=<pci-dev-id>
+ *     -object x-vfio-user-server,id=<id>,type=unix,path=<socket-path>,
+ *             device=<pci-dev-id>
+ *
+ * Note that x-vfio-user-server object must be used with x-remote machine only.
+ * This server could only support PCI devices for now.
+ *
+ * type - SocketAddress type - presently "unix" alone is supported. Required
+ *        option
+ *
+ * path - named unix socket, it will be created by the server. It is
+ *        a required option
+ *
+ * device - id of a device on the server, a required option. PCI devices
+ *          alone are supported presently.
+ */
+
+#include "qemu/osdep.h"
+
+#include "qom/object.h"
+#include "qom/object_interfaces.h"
+#include "qemu/error-report.h"
+#include "trace.h"
+#include "sysemu/runstate.h"
+#include "hw/boards.h"
+#include "hw/remote/machine.h"
+#include "qapi/error.h"
+#include "qapi/qapi-visit-sockets.h"
+
+#define TYPE_VFU_OBJECT "x-vfio-user-server"
+OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
+
+/**
+ * VFU_OBJECT_ERROR - reports an error message. If auto_shutdown
+ * is set, it aborts the machine on error. Otherwise, it logs an
+ * error message without aborting.
+ */
+#define VFU_OBJECT_ERROR(o, fmt, ...)                                     \
+    {                                                                     \
+        if (vfu_object_auto_shutdown()) {                                 \
+            error_setg(&error_abort, (fmt), ## __VA_ARGS__);              \
+        } else {                                                          \
+            error_report((fmt), ## __VA_ARGS__);                          \
+        }                                                                 \
+    }                                                                     \
+
+struct VfuObjectClass {
+    ObjectClass parent_class;
+
+    unsigned int nr_devs;
+};
+
+struct VfuObject {
+    /* private */
+    Object parent;
+
+    SocketAddress *socket;
+
+    char *device;
+
+    Error *err;
+};
+
+static bool vfu_object_auto_shutdown(void)
+{
+    bool auto_shutdown = true;
+    Error *local_err = NULL;
+
+    if (!current_machine) {
+        return auto_shutdown;
+    }
+
+    auto_shutdown = object_property_get_bool(OBJECT(current_machine),
+                                             "auto-shutdown",
+                                             &local_err);
+
+    /*
+     * local_err would be set if no such property exists - safe to ignore.
+     * Unlikely scenario as auto-shutdown is always defined for
+     * TYPE_REMOTE_MACHINE, and  TYPE_VFU_OBJECT only works with
+     * TYPE_REMOTE_MACHINE
+     */
+    if (local_err) {
+        auto_shutdown = true;
+        error_free(local_err);
+    }
+
+    return auto_shutdown;
+}
+
+static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
+                                  void *opaque, Error **errp)
+{
+    VfuObject *o = VFU_OBJECT(obj);
+
+    qapi_free_SocketAddress(o->socket);
+
+    o->socket = NULL;
+
+    visit_type_SocketAddress(v, name, &o->socket, errp);
+
+    if (o->socket->type != SOCKET_ADDRESS_TYPE_UNIX) {
+        error_setg(errp, "vfu: Unsupported socket type - %s",
+                   SocketAddressType_str(o->socket->type));
+        qapi_free_SocketAddress(o->socket);
+        o->socket = NULL;
+        return;
+    }
+
+    trace_vfu_prop("socket", o->socket->u.q_unix.path);
+}
+
+static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
+{
+    VfuObject *o = VFU_OBJECT(obj);
+
+    g_free(o->device);
+
+    o->device = g_strdup(str);
+
+    trace_vfu_prop("device", str);
+}
+
+static void vfu_object_init(Object *obj)
+{
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
+    VfuObject *o = VFU_OBJECT(obj);
+
+    k->nr_devs++;
+
+    if (!object_dynamic_cast(OBJECT(current_machine), TYPE_REMOTE_MACHINE)) {
+        error_setg(&o->err, "vfu: %s only compatible with %s machine",
+                   TYPE_VFU_OBJECT, TYPE_REMOTE_MACHINE);
+        return;
+    }
+}
+
+static void vfu_object_finalize(Object *obj)
+{
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(obj);
+    VfuObject *o = VFU_OBJECT(obj);
+
+    k->nr_devs--;
+
+    qapi_free_SocketAddress(o->socket);
+
+    o->socket = NULL;
+
+    g_free(o->device);
+
+    o->device = NULL;
+
+    if (!k->nr_devs && vfu_object_auto_shutdown()) {
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
+    }
+}
+
+static void vfu_object_class_init(ObjectClass *klass, void *data)
+{
+    VfuObjectClass *k = VFU_OBJECT_CLASS(klass);
+
+    k->nr_devs = 0;
+
+    object_class_property_add(klass, "socket", "SocketAddress", NULL,
+                              vfu_object_set_socket, NULL, NULL);
+    object_class_property_set_description(klass, "socket",
+                                          "SocketAddress "
+                                          "(ex: type=unix,path=/tmp/sock). "
+                                          "Only UNIX is presently supported");
+    object_class_property_add_str(klass, "device", NULL,
+                                  vfu_object_set_device);
+    object_class_property_set_description(klass, "device",
+                                          "device ID - only PCI devices "
+                                          "are presently supported");
+}
+
+static const TypeInfo vfu_object_info = {
+    .name = TYPE_VFU_OBJECT,
+    .parent = TYPE_OBJECT,
+    .instance_size = sizeof(VfuObject),
+    .instance_init = vfu_object_init,
+    .instance_finalize = vfu_object_finalize,
+    .class_size = sizeof(VfuObjectClass),
+    .class_init = vfu_object_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_USER_CREATABLE },
+        { }
+    }
+};
+
+static void vfu_register_types(void)
+{
+    type_register_static(&vfu_object_info);
+}
+
+type_init(vfu_register_types);
diff --git a/hw/remote/meson.build b/hw/remote/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/meson.build
+++ b/hw/remote/meson.build
@@ -XXX,XX +XXX,XX @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
+remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: files('vfio-user-obj.c'))
 
 remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: libvfio_user_dep)
 
diff --git a/hw/remote/trace-events b/hw/remote/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/trace-events
+++ b/hw/remote/trace-events
@@ -XXX,XX +XXX,XX @@
 
 mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process"
 mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process"
+
+# vfio-user-obj.c
+vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

create a context with the vfio-user library to run a PCI device

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: a452871ac8c812ff96fc4f0ce6037f4769953fab.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/remote/vfio-user-obj.c | 82 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)

diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
 #include "hw/remote/machine.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-sockets.h"
+#include "qemu/notify.h"
+#include "sysemu/sysemu.h"
+#include "libvfio-user.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ struct VfuObject {
     char *device;
 
     Error *err;
+
+    Notifier machine_done;
+
+    vfu_ctx_t *vfu_ctx;
 };
 
+static void vfu_object_init_ctx(VfuObject *o, Error **errp);
+
 static bool vfu_object_auto_shutdown(void)
 {
     bool auto_shutdown = true;
@@ -XXX,XX +XXX,XX @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
 {
     VfuObject *o = VFU_OBJECT(obj);
 
+    if (o->vfu_ctx) {
+        error_setg(errp, "vfu: Unable to set socket property - server busy");
+        return;
+    }
+
     qapi_free_SocketAddress(o->socket);
 
     o->socket = NULL;
@@ -XXX,XX +XXX,XX @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
     }
 
     trace_vfu_prop("socket", o->socket->u.q_unix.path);
+
+    vfu_object_init_ctx(o, errp);
 }
 
 static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
 {
     VfuObject *o = VFU_OBJECT(obj);
 
+    if (o->vfu_ctx) {
+        error_setg(errp, "vfu: Unable to set device property - server busy");
+        return;
+    }
+
     g_free(o->device);
 
     o->device = g_strdup(str);
 
     trace_vfu_prop("device", str);
+
+    vfu_object_init_ctx(o, errp);
+}
+
+/*
+ * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
+ * properties. It also depends on devices instantiated in QEMU. These
+ * dependencies are not available during the instance_init phase of this
+ * object's life-cycle. As such, the server is initialized after the
+ * machine is setup. machine_init_done_notifier notifies TYPE_VFU_OBJECT
+ * when the machine is setup, and the dependencies are available.
+ */
+static void vfu_object_machine_done(Notifier *notifier, void *data)
+{
+    VfuObject *o = container_of(notifier, VfuObject, machine_done);
+    Error *err = NULL;
+
+    vfu_object_init_ctx(o, &err);
+
+    if (err) {
+        error_propagate(&error_abort, err);
+    }
+}
+
+static void vfu_object_init_ctx(VfuObject *o, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (o->vfu_ctx || !o->socket || !o->device ||
+            !phase_check(PHASE_MACHINE_READY)) {
+        return;
+    }
+
+    if (o->err) {
+        error_propagate(errp, o->err);
+        o->err = NULL;
+        return;
+    }
+
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path, 0,
+                                o, VFU_DEV_TYPE_PCI);
+    if (o->vfu_ctx == NULL) {
+        error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
+        return;
+    }
 }
 
 static void vfu_object_init(Object *obj)
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init(Object *obj)
                    TYPE_VFU_OBJECT, TYPE_REMOTE_MACHINE);
         return;
     }
+
+    if (!phase_check(PHASE_MACHINE_READY)) {
+        o->machine_done.notify = vfu_object_machine_done;
+        qemu_add_machine_init_done_notifier(&o->machine_done);
+    }
+
 }
 
 static void vfu_object_finalize(Object *obj)
@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
 
     o->socket = NULL;
 
+    if (o->vfu_ctx) {
+        vfu_destroy_ctx(o->vfu_ctx);
+        o->vfu_ctx = NULL;
+    }
+
     g_free(o->device);
 
     o->device = NULL;
@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
     if (!k->nr_devs && vfu_object_auto_shutdown()) {
         qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
     }
+
+    if (o->machine_done.notify) {
+        qemu_remove_machine_init_done_notifier(&o->machine_done);
+        o->machine_done.notify = NULL;
+    }
 }
 
 static void vfu_object_class_init(ObjectClass *klass, void *data)
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Find the PCI device with specified id. Initialize the device context
with the QEMU PCI device

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 7798dbd730099b33fdd00c4c202cfe79e5c5c151.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/remote/vfio-user-obj.c | 67 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)

diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
 #include "qemu/notify.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
+#include "hw/qdev-core.h"
+#include "hw/pci/pci.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ struct VfuObject {
     Notifier machine_done;
 
     vfu_ctx_t *vfu_ctx;
+
+    PCIDevice *pci_dev;
+
+    Error *unplug_blocker;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -XXX,XX +XXX,XX @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
 static void vfu_object_init_ctx(VfuObject *o, Error **errp)
 {
     ERRP_GUARD();
+    DeviceState *dev = NULL;
+    vfu_pci_type_t pci_type = VFU_PCI_TYPE_CONVENTIONAL;
+    int ret;
 
     if (o->vfu_ctx || !o->socket || !o->device ||
             !phase_check(PHASE_MACHINE_READY)) {
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
         return;
     }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->device);
+    if (dev == NULL) {
+        error_setg(errp, "vfu: Device %s not found", o->device);
+        goto fail;
+    }
+
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "vfu: %s not a PCI device", o->device);
+        goto fail;
+    }
+
+    o->pci_dev = PCI_DEVICE(dev);
+
+    object_ref(OBJECT(o->pci_dev));
+
+    if (pci_is_express(o->pci_dev)) {
+        pci_type = VFU_PCI_TYPE_EXPRESS;
+    }
+
+    ret = vfu_pci_init(o->vfu_ctx, pci_type, PCI_HEADER_TYPE_NORMAL, 0);
+    if (ret < 0) {
+        error_setg(errp,
+                   "vfu: Failed to attach PCI device %s to context - %s",
+                   o->device, strerror(errno));
+        goto fail;
+    }
+
+    error_setg(&o->unplug_blocker,
+               "vfu: %s for %s must be deleted before unplugging",
+               TYPE_VFU_OBJECT, o->device);
+    qdev_add_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
+
+    return;
+
+fail:
+    vfu_destroy_ctx(o->vfu_ctx);
+    if (o->unplug_blocker && o->pci_dev) {
+        qdev_del_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
+        error_free(o->unplug_blocker);
+        o->unplug_blocker = NULL;
+    }
+    if (o->pci_dev) {
+        object_unref(OBJECT(o->pci_dev));
+        o->pci_dev = NULL;
+    }
+    o->vfu_ctx = NULL;
 }
 
 static void vfu_object_init(Object *obj)
@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
 
     o->device = NULL;
 
+    if (o->unplug_blocker && o->pci_dev) {
+        qdev_del_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
+        error_free(o->unplug_blocker);
+        o->unplug_blocker = NULL;
+    }
+
+    if (o->pci_dev) {
+        object_unref(OBJECT(o->pci_dev));
+        o->pci_dev = NULL;
+    }
+
     if (!k->nr_devs && vfu_object_auto_shutdown()) {
         qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
     }
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Setup a handler to run vfio-user context. The context is driven by
messages to the file descriptor associated with it - get the fd for
the context and hook up the handler with it

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: e934b0090529d448b6a7972b21dfc3d7421ce494.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 qapi/misc.json            |  31 ++++++++++
 hw/remote/vfio-user-obj.c | 118 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 148 insertions(+), 1 deletion(-)

diff --git a/qapi/misc.json b/qapi/misc.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -XXX,XX +XXX,XX @@
 ##
 { 'event': 'RTC_CHANGE',
   'data': { 'offset': 'int', 'qom-path': 'str' } }
+
+##
+# @VFU_CLIENT_HANGUP:
+#
+# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the
+# communication channel
+#
+# @vfu-id: ID of the TYPE_VFIO_USER_SERVER object. It is the last component
+#          of @vfu-qom-path referenced below
+#
+# @vfu-qom-path: path to the TYPE_VFIO_USER_SERVER object in the QOM tree
+#
+# @dev-id: ID of attached PCI device
+#
+# @dev-qom-path: path to attached PCI device in the QOM tree
+#
+# Since: 7.1
+#
+# Example:
+#
+# <- { "event": "VFU_CLIENT_HANGUP",
+#      "data": { "vfu-id": "vfu1",
+#                "vfu-qom-path": "/objects/vfu1",
+#                "dev-id": "sas1",
+#                "dev-qom-path": "/machine/peripheral/sas1" },
+#      "timestamp": { "seconds": 1265044230, "microseconds": 450486 } }
+#
+##
+{ 'event': 'VFU_CLIENT_HANGUP',
+  'data': { 'vfu-id': 'str', 'vfu-qom-path': 'str',
+            'dev-id': 'str', 'dev-qom-path': 'str' } }
diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
  *
  * device - id of a device on the server, a required option. PCI devices
  *          alone are supported presently.
+ *
+ * notes - x-vfio-user-server could block IO and monitor during the
+ *         initialization phase.
  */
 
 #include "qemu/osdep.h"
@@ -XXX,XX +XXX,XX @@
 #include "hw/remote/machine.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-sockets.h"
+#include "qapi/qapi-events-misc.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
 #include "hw/qdev-core.h"
 #include "hw/pci/pci.h"
+#include "qemu/timer.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ struct VfuObject {
     PCIDevice *pci_dev;
 
     Error *unplug_blocker;
+
+    int vfu_poll_fd;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -XXX,XX +XXX,XX @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
     vfu_object_init_ctx(o, errp);
 }
 
+static void vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    const char *vfu_id;
+    char *vfu_path, *pci_dev_path;
+    int ret = -1;
+
+    while (ret != 0) {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
+                continue;
+            } else if (errno == ENOTCONN) {
+                vfu_id = object_get_canonical_path_component(OBJECT(o));
+                vfu_path = object_get_canonical_path(OBJECT(o));
+                g_assert(o->pci_dev);
+                pci_dev_path = object_get_canonical_path(OBJECT(o->pci_dev));
+                 /* o->device is a required property and is non-NULL here */
+                g_assert(o->device);
+                qapi_event_send_vfu_client_hangup(vfu_id, vfu_path,
+                                                  o->device, pci_dev_path);
+                qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
+                o->vfu_poll_fd = -1;
+                object_unparent(OBJECT(o));
+                g_free(vfu_path);
+                g_free(pci_dev_path);
+                break;
+            } else {
+                VFU_OBJECT_ERROR(o, "vfu: Failed to run device %s - %s",
+                                 o->device, strerror(errno));
+                break;
+            }
+        }
+    }
+}
+
+static void vfu_object_attach_ctx(void *opaque)
+{
+    VfuObject *o = opaque;
+    GPollFD pfds[1];
+    int ret;
+
+    qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
+
+    pfds[0].fd = o->vfu_poll_fd;
+    pfds[0].events = G_IO_IN | G_IO_HUP | G_IO_ERR;
+
+retry_attach:
+    ret = vfu_attach_ctx(o->vfu_ctx);
+    if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
+        /**
+         * vfu_object_attach_ctx can block QEMU's main loop
+         * during attach - the monitor and other IO
+         * could be unresponsive during this time.
+         */
+        (void)qemu_poll_ns(pfds, 1, 500 * (int64_t)SCALE_MS);
+        goto retry_attach;
+    } else if (ret < 0) {
+        VFU_OBJECT_ERROR(o, "vfu: Failed to attach device %s to context - %s",
+                         o->device, strerror(errno));
+        return;
+    }
+
+    o->vfu_poll_fd = vfu_get_poll_fd(o->vfu_ctx);
+    if (o->vfu_poll_fd < 0) {
+        VFU_OBJECT_ERROR(o, "vfu: Failed to get poll fd %s", o->device);
+        return;
+    }
+
+    qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_ctx_run, NULL, o);
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
     }
 }
 
+/**
+ * vfu_object_init_ctx: Create and initialize libvfio-user context. Add
+ *     an unplug blocker for the associated PCI device. Setup a FD handler
+ *     to process incoming messages in the context's socket.
+ *
+ *     The socket and device properties are mandatory, and this function
+ *     will not create the context without them - the setters for these
+ *     properties should call this function when the property is set. The
+ *     machine should also be ready when this function is invoked - it is
+ *     because QEMU objects are initialized before devices, and the
+ *     associated PCI device wouldn't be available at the object
+ *     initialization time. Until these conditions are satisfied, this
+ *     function would return early without performing any task.
+ */
 static void vfu_object_init_ctx(VfuObject *o, Error **errp)
 {
     ERRP_GUARD();
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         return;
     }
 
-    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path, 0,
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path,
+                                LIBVFIO_USER_FLAG_ATTACH_NB,
                                 o, VFU_DEV_TYPE_PCI);
     if (o->vfu_ctx == NULL) {
         error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
                TYPE_VFU_OBJECT, o->device);
     qdev_add_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
 
+    ret = vfu_realize_ctx(o->vfu_ctx);
+    if (ret < 0) {
+        error_setg(errp, "vfu: Failed to realize device %s- %s",
+                   o->device, strerror(errno));
+        goto fail;
+    }
+
+    o->vfu_poll_fd = vfu_get_poll_fd(o->vfu_ctx);
+    if (o->vfu_poll_fd < 0) {
+        error_setg(errp, "vfu: Failed to get poll fd %s", o->device);
+        goto fail;
+    }
+
+    qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_attach_ctx, NULL, o);
+
     return;
 
 fail:
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init(Object *obj)
         qemu_add_machine_init_done_notifier(&o->machine_done);
     }
 
+    o->vfu_poll_fd = -1;
 }
 
 static void vfu_object_finalize(Object *obj)
@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
 
     o->socket = NULL;
 
+    if (o->vfu_poll_fd != -1) {
+        qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);
+        o->vfu_poll_fd = -1;
+    }
+
     if (o->vfu_ctx) {
         vfu_destroy_ctx(o->vfu_ctx);
         o->vfu_ctx = NULL;
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Define and register handlers for PCI config space accesses

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: be9d2ccf9b1d24e50dcd9c23404dbf284142cec7.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/remote/vfio-user-obj.c | 51 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 53 insertions(+)

diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
 #include "qapi/qapi-events-misc.h"
 #include "qemu/notify.h"
 #include "qemu/thread.h"
+#include "qemu/main-loop.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
 #include "hw/qdev-core.h"
@@ -XXX,XX +XXX,XX @@ retry_attach:
     qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_ctx_run, NULL, o);
 }
 
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t pci_access_width = sizeof(uint32_t);
+    size_t bytes = count;
+    uint32_t val = 0;
+    char *ptr = buf;
+    int len;
+
+    /*
+     * Writes to the BAR registers would trigger an update to the
+     * global Memory and IO AddressSpaces. But the remote device
+     * never uses the global AddressSpaces, therefore overlapping
+     * memory regions are not a problem
+     */
+    while (bytes > 0) {
+        len = (bytes > pci_access_width) ? pci_access_width : bytes;
+        if (is_write) {
+            memcpy(&val, ptr, len);
+            pci_host_config_write_common(o->pci_dev, offset,
+                                         pci_config_size(o->pci_dev),
+                                         val, len);
+            trace_vfu_cfg_write(offset, val);
+        } else {
+            val = pci_host_config_read_common(o->pci_dev, offset,
+                                              pci_config_size(o->pci_dev), len);
+            memcpy(ptr, &val, len);
+            trace_vfu_cfg_read(offset, val);
+        }
+        offset += len;
+        ptr += len;
+        bytes -= len;
+    }
+
+    return count;
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
                TYPE_VFU_OBJECT, o->device);
     qdev_add_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
 
+    ret = vfu_setup_region(o->vfu_ctx, VFU_PCI_DEV_CFG_REGION_IDX,
+                           pci_config_size(o->pci_dev), &vfu_object_cfg_access,
+                           VFU_REGION_FLAG_RW | VFU_REGION_FLAG_ALWAYS_CB,
+                           NULL, 0, -1, 0);
+    if (ret < 0) {
+        error_setg(errp,
+                   "vfu: Failed to setup config space handlers for %s- %s",
+                   o->device, strerror(errno));
+        goto fail;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
diff --git a/hw/remote/trace-events b/hw/remote/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/trace-events
+++ b/hw/remote/trace-events
@@ -XXX,XX +XXX,XX @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 
 # vfio-user-obj.c
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
+vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
+vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Assign separate address space for each device in the remote processes.

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: afe0b0a97582cdad42b5b25636a29c523265a10a.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 MAINTAINERS               |   2 +
 include/hw/remote/iommu.h |  40 ++++++++++++
 hw/remote/iommu.c         | 131 ++++++++++++++++++++++++++++++++++++++
 hw/remote/machine.c       |  13 +++-
 hw/remote/meson.build     |   1 +
 5 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 include/hw/remote/iommu.h
 create mode 100644 hw/remote/iommu.c

From: Jagannathan Raman <jag.raman@oracle.com>

Define and register callbacks to manage the RAM regions used for
device DMA

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: faacbcd45c4d02c591f0dbfdc19041fbb3eae7eb.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/remote/machine.c       |  5 ++++
 hw/remote/vfio-user-obj.c | 55 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 3 files changed, 62 insertions(+)

diff --git a/hw/remote/machine.c b/hw/remote/machine.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/machine.c
+++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@
 #include "hw/remote/iohub.h"
 #include "hw/remote/iommu.h"
 #include "hw/qdev-core.h"
+#include "hw/remote/iommu.h"
 
 static void remote_machine_init(MachineState *machine)
 {
@@ -XXX,XX +XXX,XX @@ static void remote_machine_init(MachineState *machine)
 
     pci_host = PCI_HOST_BRIDGE(rem_host);
 
+    if (s->vfio_user) {
+        remote_iommu_setup(pci_host->bus);
+    }
+
     remote_iohub_init(&s->iohub);
 
     pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
     return count;
 }
 
+static void dma_register(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    AddressSpace *dma_as = NULL;
+    MemoryRegion *subregion = NULL;
+    g_autofree char *name = NULL;
+    struct iovec *iov = &info->iova;
+
+    if (!info->vaddr) {
+        return;
+    }
+
+    name = g_strdup_printf("mem-%s-%"PRIx64"", o->device,
+                           (uint64_t)info->vaddr);
+
+    subregion = g_new0(MemoryRegion, 1);
+
+    memory_region_init_ram_ptr(subregion, NULL, name,
+                               iov->iov_len, info->vaddr);
+
+    dma_as = pci_device_iommu_address_space(o->pci_dev);
+
+    memory_region_add_subregion(dma_as->root, (hwaddr)iov->iov_base, subregion);
+
+    trace_vfu_dma_register((uint64_t)iov->iov_base, iov->iov_len);
+}
+
+static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    AddressSpace *dma_as = NULL;
+    MemoryRegion *mr = NULL;
+    ram_addr_t offset;
+
+    mr = memory_region_from_host(info->vaddr, &offset);
+    if (!mr) {
+        return;
+    }
+
+    dma_as = pci_device_iommu_address_space(o->pci_dev);
+
+    memory_region_del_subregion(dma_as->root, mr);
+
+    object_unparent((OBJECT(mr)));
+
+    trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         goto fail;
     }
 
+    ret = vfu_setup_device_dma(o->vfu_ctx, &dma_register, &dma_unregister);
+    if (ret < 0) {
+        error_setg(errp, "vfu: Failed to setup DMA handlers for %s",
+                   o->device);
+        goto fail;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
diff --git a/hw/remote/trace-events b/hw/remote/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/trace-events
+++ b/hw/remote/trace-events
@@ -XXX,XX +XXX,XX @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
 vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
+vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
+vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Determine the BARs used by the PCI device and register handlers to
manage the access to the same.

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 3373e10b5be5f42846f0632d4382466e1698c505.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 include/exec/memory.h           |   3 +
 hw/remote/vfio-user-obj.c       | 190 ++++++++++++++++++++++++++++++++
 softmmu/physmem.c               |   4 +-
 tests/qtest/fuzz/generic_fuzz.c |   9 +-
 hw/remote/trace-events          |   3 +
 5 files changed, 203 insertions(+), 6 deletions(-)

diff --git a/include/exec/memory.h b/include/exec/memory.h
index XXXXXXX..XXXXXXX 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -XXX,XX +XXX,XX @@ MemTxResult address_space_write_cached_slow(MemoryRegionCache *cache,
                                             hwaddr addr, const void *buf,
                                             hwaddr len);
 
+int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr);
+bool prepare_mmio_access(MemoryRegion *mr);
+
 static inline bool memory_access_is_direct(MemoryRegion *mr, bool is_write)
 {
     if (is_write) {
diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
 #include "hw/qdev-core.h"
 #include "hw/pci/pci.h"
 #include "qemu/timer.h"
+#include "exec/memory.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
     trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
 }
 
+static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
+                            hwaddr size, const bool is_write)
+{
+    uint8_t *ptr = buf;
+    bool release_lock = false;
+    uint8_t *ram_ptr = NULL;
+    MemTxResult result;
+    int access_size;
+    uint64_t val;
+
+    if (memory_access_is_direct(mr, is_write)) {
+        /**
+         * Some devices expose a PCI expansion ROM, which could be buffer
+         * based as compared to other regions which are primarily based on
+         * MemoryRegionOps. memory_region_find() would already check
+         * for buffer overflow, we don't need to repeat it here.
+         */
+        ram_ptr = memory_region_get_ram_ptr(mr);
+
+        if (is_write) {
+            memcpy((ram_ptr + offset), buf, size);
+        } else {
+            memcpy(buf, (ram_ptr + offset), size);
+        }
+
+        return 0;
+    }
+
+    while (size) {
+        /**
+         * The read/write logic used below is similar to the ones in
+         * flatview_read/write_continue()
+         */
+        release_lock = prepare_mmio_access(mr);
+
+        access_size = memory_access_size(mr, size, offset);
+
+        if (is_write) {
+            val = ldn_he_p(ptr, access_size);
+
+            result = memory_region_dispatch_write(mr, offset, val,
+                                                  size_memop(access_size),
+                                                  MEMTXATTRS_UNSPECIFIED);
+        } else {
+            result = memory_region_dispatch_read(mr, offset, &val,
+                                                 size_memop(access_size),
+                                                 MEMTXATTRS_UNSPECIFIED);
+
+            stn_he_p(ptr, access_size, val);
+        }
+
+        if (release_lock) {
+            qemu_mutex_unlock_iothread();
+            release_lock = false;
+        }
+
+        if (result != MEMTX_OK) {
+            return -1;
+        }
+
+        size -= access_size;
+        ptr += access_size;
+        offset += access_size;
+    }
+
+    return 0;
+}
+
+static size_t vfu_object_bar_rw(PCIDevice *pci_dev, int pci_bar,
+                                hwaddr bar_offset, char * const buf,
+                                hwaddr len, const bool is_write)
+{
+    MemoryRegionSection section = { 0 };
+    uint8_t *ptr = (uint8_t *)buf;
+    MemoryRegion *section_mr = NULL;
+    uint64_t section_size;
+    hwaddr section_offset;
+    hwaddr size = 0;
+
+    while (len) {
+        section = memory_region_find(pci_dev->io_regions[pci_bar].memory,
+                                     bar_offset, len);
+
+        if (!section.mr) {
+            warn_report("vfu: invalid address 0x%"PRIx64"", bar_offset);
+            return size;
+        }
+
+        section_mr = section.mr;
+        section_offset = section.offset_within_region;
+        section_size = int128_get64(section.size);
+
+        if (is_write && section_mr->readonly) {
+            warn_report("vfu: attempting to write to readonly region in "
+                        "bar %d - [0x%"PRIx64" - 0x%"PRIx64"]",
+                        pci_bar, bar_offset,
+                        (bar_offset + section_size));
+            memory_region_unref(section_mr);
+            return size;
+        }
+
+        if (vfu_object_mr_rw(section_mr, ptr, section_offset,
+                             section_size, is_write)) {
+            warn_report("vfu: failed to %s "
+                        "[0x%"PRIx64" - 0x%"PRIx64"] in bar %d",
+                        is_write ? "write to" : "read from", bar_offset,
+                        (bar_offset + section_size), pci_bar);
+            memory_region_unref(section_mr);
+            return size;
+        }
+
+        size += section_size;
+        bar_offset += section_size;
+        ptr += section_size;
+        len -= section_size;
+
+        memory_region_unref(section_mr);
+    }
+
+    return size;
+}
+
+/**
+ * VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs.
+ *
+ * To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would
+ * define vfu_object_bar2_handler
+ */
+#define VFU_OBJECT_BAR_HANDLER(BAR_NO)                                         \
+    static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx,        \
+                                        char * const buf, size_t count,        \
+                                        loff_t offset, const bool is_write)    \
+    {                                                                          \
+        VfuObject *o = vfu_get_private(vfu_ctx);                               \
+        PCIDevice *pci_dev = o->pci_dev;                                       \
+                                                                               \
+        return vfu_object_bar_rw(pci_dev, BAR_NO, offset,                      \
+                                 buf, count, is_write);                        \
+    }                                                                          \
+
+VFU_OBJECT_BAR_HANDLER(0)
+VFU_OBJECT_BAR_HANDLER(1)
+VFU_OBJECT_BAR_HANDLER(2)
+VFU_OBJECT_BAR_HANDLER(3)
+VFU_OBJECT_BAR_HANDLER(4)
+VFU_OBJECT_BAR_HANDLER(5)
+VFU_OBJECT_BAR_HANDLER(6)
+
+static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS] = {
+    &vfu_object_bar0_handler,
+    &vfu_object_bar1_handler,
+    &vfu_object_bar2_handler,
+    &vfu_object_bar3_handler,
+    &vfu_object_bar4_handler,
+    &vfu_object_bar5_handler,
+    &vfu_object_bar6_handler,
+};
+
+/**
+ * vfu_object_register_bars - Identify active BAR regions of pdev and setup
+ *                            callbacks to handle read/write accesses
+ */
+static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
+{
+    int flags = VFU_REGION_FLAG_RW;
+    int i;
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        if (!pdev->io_regions[i].size) {
+            continue;
+        }
+
+        if ((i == VFU_PCI_DEV_ROM_REGION_IDX) ||
+            pdev->io_regions[i].memory->readonly) {
+            flags &= ~VFU_REGION_FLAG_WRITE;
+        }
+
+        vfu_setup_region(vfu_ctx, VFU_PCI_DEV_BAR0_REGION_IDX + i,
+                         (size_t)pdev->io_regions[i].size,
+                         vfu_object_bar_handlers[i],
+                         flags, NULL, 0, -1, 0);
+
+        trace_vfu_bar_register(i, pdev->io_regions[i].addr,
+                               pdev->io_regions[i].size);
+    }
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         goto fail;
     }
 
+    vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
diff --git a/softmmu/physmem.c b/softmmu/physmem.c
index XXXXXXX..XXXXXXX 100644
--- a/softmmu/physmem.c
+++ b/softmmu/physmem.c
@@ -XXX,XX +XXX,XX @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)
     invalidate_and_set_dirty(mr, addr, size);
 }
 
-static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
+int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
 {
     unsigned access_size_max = mr->ops->valid.max_access_size;
 
@@ -XXX,XX +XXX,XX @@ static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
     return l;
 }
 
-static bool prepare_mmio_access(MemoryRegion *mr)
+bool prepare_mmio_access(MemoryRegion *mr)
 {
     bool release_lock = false;
 
diff --git a/tests/qtest/fuzz/generic_fuzz.c b/tests/qtest/fuzz/generic_fuzz.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/fuzz/generic_fuzz.c
+++ b/tests/qtest/fuzz/generic_fuzz.c
@@ -XXX,XX +XXX,XX @@ static void *pattern_alloc(pattern p, size_t len)
     return buf;
 }
 
-static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
+static int fuzz_memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)
 {
     unsigned access_size_max = mr->ops->valid.max_access_size;
 
@@ -XXX,XX +XXX,XX @@ void fuzz_dma_read_cb(size_t addr, size_t len, MemoryRegion *mr)
 
         /*
          *  If mr1 isn't RAM, address_space_translate doesn't update l. Use
-         *  memory_access_size to identify the number of bytes that it is safe
-         *  to write without accidentally writing to another MemoryRegion.
+         *  fuzz_memory_access_size to identify the number of bytes that it
+         *  is safe to write without accidentally writing to another
+         *  MemoryRegion.
          */
         if (!memory_region_is_ram(mr1)) {
-            l = memory_access_size(mr1, l, addr1);
+            l = fuzz_memory_access_size(mr1, l, addr1);
         }
         if (memory_region_is_ram(mr1) ||
             memory_region_is_romd(mr1) ||
diff --git a/hw/remote/trace-events b/hw/remote/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/trace-events
+++ b/hw/remote/trace-events
@@ -XXX,XX +XXX,XX @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
 vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
 vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
+vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""
+vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
+vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Forward remote device's interrupts to the guest

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Message-id: 9523479eaafe050677f4de2af5dd0df18c27cfd9.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 MAINTAINERS                       |   1 +
 include/hw/pci/msi.h              |   1 +
 include/hw/pci/msix.h             |   1 +
 include/hw/pci/pci.h              |  13 +++
 include/hw/remote/vfio-user-obj.h |   6 ++
 hw/pci/msi.c                      |  49 +++++++--
 hw/pci/msix.c                     |  35 ++++++-
 hw/pci/pci.c                      |  13 +++
 hw/remote/machine.c               |  16 ++-
 hw/remote/vfio-user-obj.c         | 167 ++++++++++++++++++++++++++++++
 stubs/vfio-user-obj.c             |   6 ++
 hw/remote/trace-events            |   1 +
 stubs/meson.build                 |   1 +
 13 files changed, 298 insertions(+), 12 deletions(-)
 create mode 100644 include/hw/remote/vfio-user-obj.h
 create mode 100644 stubs/vfio-user-obj.c

diff --git a/MAINTAINERS b/MAINTAINERS
index XXXXXXX..XXXXXXX 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: subprojects/libvfio-user
 F: hw/remote/vfio-user-obj.c
+F: include/hw/remote/vfio-user-obj.h
 F: hw/remote/iommu.c
 F: include/hw/remote/iommu.h
 
diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
index XXXXXXX..XXXXXXX 100644
--- a/include/hw/pci/msi.h
+++ b/include/hw/pci/msi.h
@@ -XXX,XX +XXX,XX @@ void msi_notify(PCIDevice *dev, unsigned int vector);
 void msi_send_message(PCIDevice *dev, MSIMessage msg);
 void msi_write_config(PCIDevice *dev, uint32_t addr, uint32_t val, int len);
 unsigned int msi_nr_vectors_allocated(const PCIDevice *dev);
+void msi_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp);
 
 static inline bool msi_present(const PCIDevice *dev)
 {
diff --git a/include/hw/pci/msix.h b/include/hw/pci/msix.h
index XXXXXXX..XXXXXXX 100644
--- a/include/hw/pci/msix.h
+++ b/include/hw/pci/msix.h
@@ -XXX,XX +XXX,XX @@ void msix_clr_pending(PCIDevice *dev, int vector);
 int msix_vector_use(PCIDevice *dev, unsigned vector);
 void msix_vector_unuse(PCIDevice *dev, unsigned vector);
 void msix_unuse_all_vectors(PCIDevice *dev);
+void msix_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp);
 
 void msix_notify(PCIDevice *dev, unsigned vector);
 
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index XXXXXXX..XXXXXXX 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -XXX,XX +XXX,XX @@ extern bool pci_available;
 #define PCI_SLOT(devfn)         (((devfn) >> 3) & 0x1f)
 #define PCI_FUNC(devfn)         ((devfn) & 0x07)
 #define PCI_BUILD_BDF(bus, devfn)     ((bus << 8) | (devfn))
+#define PCI_BDF_TO_DEVFN(x)     ((x) & 0xff)
 #define PCI_BUS_MAX             256
 #define PCI_DEVFN_MAX           256
 #define PCI_SLOT_MAX            32
@@ -XXX,XX +XXX,XX @@ typedef void PCIMapIORegionFunc(PCIDevice *pci_dev, int region_num,
                                 pcibus_t addr, pcibus_t size, int type);
 typedef void PCIUnregisterFunc(PCIDevice *pci_dev);
 
+typedef void MSITriggerFunc(PCIDevice *dev, MSIMessage msg);
+typedef MSIMessage MSIPrepareMessageFunc(PCIDevice *dev, unsigned vector);
+typedef MSIMessage MSIxPrepareMessageFunc(PCIDevice *dev, unsigned vector);
+
 typedef struct PCIIORegion {
     pcibus_t addr; /* current PCI mapping address. -1 means not mapped */
 #define PCI_BAR_UNMAPPED (~(pcibus_t)0)
@@ -XXX,XX +XXX,XX @@ struct PCIDevice {
     /* Space to store MSIX table & pending bit array */
     uint8_t *msix_table;
     uint8_t *msix_pba;
+
+    /* May be used by INTx or MSI during interrupt notification */
+    void *irq_opaque;
+
+    MSITriggerFunc *msi_trigger;
+    MSIPrepareMessageFunc *msi_prepare_message;
+    MSIxPrepareMessageFunc *msix_prepare_message;
+
     /* MemoryRegion container for msix exclusive BAR setup */
     MemoryRegion msix_exclusive_bar;
     /* Memory Regions for MSIX table and pending bit entries. */
diff --git a/include/hw/remote/vfio-user-obj.h b/include/hw/remote/vfio-user-obj.h
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/include/hw/remote/vfio-user-obj.h
@@ -XXX,XX +XXX,XX @@
+#ifndef VFIO_USER_OBJ_H
+#define VFIO_USER_OBJ_H
+
+void vfu_object_set_bus_irq(PCIBus *pci_bus);
+
+#endif
diff --git a/hw/pci/msi.c b/hw/pci/msi.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/pci/msi.c
+++ b/hw/pci/msi.c
@@ -XXX,XX +XXX,XX @@ void msi_set_message(PCIDevice *dev, MSIMessage msg)
     pci_set_word(dev->config + msi_data_off(dev, msi64bit), msg.data);
 }
 
-MSIMessage msi_get_message(PCIDevice *dev, unsigned int vector)
+static MSIMessage msi_prepare_message(PCIDevice *dev, unsigned int vector)
 {
     uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
     bool msi64bit = flags & PCI_MSI_FLAGS_64BIT;
@@ -XXX,XX +XXX,XX @@ MSIMessage msi_get_message(PCIDevice *dev, unsigned int vector)
     return msg;
 }
 
+MSIMessage msi_get_message(PCIDevice *dev, unsigned int vector)
+{
+    return dev->msi_prepare_message(dev, vector);
+}
+
 bool msi_enabled(const PCIDevice *dev)
 {
     return msi_present(dev) &&
@@ -XXX,XX +XXX,XX @@ int msi_init(struct PCIDevice *dev, uint8_t offset,
                      0xffffffff >> (PCI_MSI_VECTORS_MAX - nr_vectors));
     }
 
+    dev->msi_prepare_message = msi_prepare_message;
+
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ void msi_uninit(struct PCIDevice *dev)
     cap_size = msi_cap_sizeof(flags);
     pci_del_capability(dev, PCI_CAP_ID_MSI, cap_size);
     dev->cap_present &= ~QEMU_PCI_CAP_MSI;
+    dev->msi_prepare_message = NULL;
 
     MSI_DEV_PRINTF(dev, "uninit\n");
 }
@@ -XXX,XX +XXX,XX @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)
     return mask & (1U << vector);
 }
 
+void msi_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp)
+{
+    ERRP_GUARD();
+    uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
+    bool msi64bit = flags & PCI_MSI_FLAGS_64BIT;
+    uint32_t irq_state, vector_mask, pending;
+
+    if (vector > PCI_MSI_VECTORS_MAX) {
+        error_setg(errp, "msi: vector %d not allocated. max vector is %d",
+                   vector, PCI_MSI_VECTORS_MAX);
+        return;
+    }
+
+    vector_mask = (1U << vector);
+
+    irq_state = pci_get_long(dev->config + msi_mask_off(dev, msi64bit));
+
+    if (mask) {
+        irq_state |= vector_mask;
+    } else {
+        irq_state &= ~vector_mask;
+    }
+
+    pci_set_long(dev->config + msi_mask_off(dev, msi64bit), irq_state);
+
+    pending = pci_get_long(dev->config + msi_pending_off(dev, msi64bit));
+    if (!mask && (pending & vector_mask)) {
+        pending &= ~vector_mask;
+        pci_set_long(dev->config + msi_pending_off(dev, msi64bit), pending);
+        msi_notify(dev, vector);
+    }
+}
+
 void msi_notify(PCIDevice *dev, unsigned int vector)
 {
     uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
@@ -XXX,XX +XXX,XX @@ void msi_notify(PCIDevice *dev, unsigned int vector)
 
 void msi_send_message(PCIDevice *dev, MSIMessage msg)
 {
-    MemTxAttrs attrs = {};
-
-    attrs.requester_id = pci_requester_id(dev);
-    address_space_stl_le(&dev->bus_master_as, msg.address, msg.data,
-                         attrs, NULL);
+    dev->msi_trigger(dev, msg);
 }
 
 /* Normally called by pci_default_write_config(). */
diff --git a/hw/pci/msix.c b/hw/pci/msix.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/pci/msix.c
+++ b/hw/pci/msix.c
@@ -XXX,XX +XXX,XX @@
 #define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)
 #define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8)
 
-MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
+static MSIMessage msix_prepare_message(PCIDevice *dev, unsigned vector)
 {
     uint8_t *table_entry = dev->msix_table + vector * PCI_MSIX_ENTRY_SIZE;
     MSIMessage msg;
@@ -XXX,XX +XXX,XX @@ MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
     return msg;
 }
 
+MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
+{
+    return dev->msix_prepare_message(dev, vector);
+}
+
 /*
  * Special API for POWER to configure the vectors through
  * a side channel. Should never be used by devices.
@@ -XXX,XX +XXX,XX @@ static void msix_handle_mask_update(PCIDevice *dev, int vector, bool was_masked)
     }
 }
 
+void msix_set_mask(PCIDevice *dev, int vector, bool mask, Error **errp)
+{
+    ERRP_GUARD();
+    unsigned offset;
+    bool was_masked;
+
+    if (vector > dev->msix_entries_nr) {
+        error_setg(errp, "msix: vector %d not allocated. max vector is %d",
+                   vector, dev->msix_entries_nr);
+        return;
+    }
+
+    offset = vector * PCI_MSIX_ENTRY_SIZE + PCI_MSIX_ENTRY_VECTOR_CTRL;
+
+    was_masked = msix_is_masked(dev, vector);
+
+    if (mask) {
+        dev->msix_table[offset] |= PCI_MSIX_ENTRY_CTRL_MASKBIT;
+    } else {
+        dev->msix_table[offset] &= ~PCI_MSIX_ENTRY_CTRL_MASKBIT;
+    }
+
+    msix_handle_mask_update(dev, vector, was_masked);
+}
+
 static bool msix_masked(PCIDevice *dev)
 {
     return dev->config[dev->msix_cap + MSIX_CONTROL_OFFSET] & MSIX_MASKALL_MASK;
@@ -XXX,XX +XXX,XX @@ int msix_init(struct PCIDevice *dev, unsigned short nentries,
                           "msix-pba", pba_size);
     memory_region_add_subregion(pba_bar, pba_offset, &dev->msix_pba_mmio);
 
+    dev->msix_prepare_message = msix_prepare_message;
+
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ void msix_uninit(PCIDevice *dev, MemoryRegion *table_bar, MemoryRegion *pba_bar)
     g_free(dev->msix_entry_used);
     dev->msix_entry_used = NULL;
     dev->cap_present &= ~QEMU_PCI_CAP_MSIX;
+    dev->msix_prepare_message = NULL;
 }
 
 void msix_uninit_exclusive_bar(PCIDevice *dev)
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -XXX,XX +XXX,XX @@ void pci_device_deassert_intx(PCIDevice *dev)
     }
 }
 
+static void pci_msi_trigger(PCIDevice *dev, MSIMessage msg)
+{
+    MemTxAttrs attrs = {};
+
+    attrs.requester_id = pci_requester_id(dev);
+    address_space_stl_le(&dev->bus_master_as, msg.address, msg.data,
+                         attrs, NULL);
+}
+
 static void pci_reset_regions(PCIDevice *dev)
 {
     int r;
@@ -XXX,XX +XXX,XX @@ static void pci_qdev_unrealize(DeviceState *dev)
 
     pci_device_deassert_intx(pci_dev);
     do_pci_unregister_device(pci_dev);
+
+    pci_dev->msi_trigger = NULL;
 }
 
 void pci_register_bar(PCIDevice *pci_dev, int region_num,
@@ -XXX,XX +XXX,XX @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp)
     }
 
     pci_set_power(pci_dev, true);
+
+    pci_dev->msi_trigger = pci_msi_trigger;
 }
 
 PCIDevice *pci_new_multifunction(int devfn, bool multifunction,
diff --git a/hw/remote/machine.c b/hw/remote/machine.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/machine.c
+++ b/hw/remote/machine.c
@@ -XXX,XX +XXX,XX @@
 #include "hw/remote/iommu.h"
 #include "hw/qdev-core.h"
 #include "hw/remote/iommu.h"
+#include "hw/remote/vfio-user-obj.h"
+#include "hw/pci/msi.h"
 
 static void remote_machine_init(MachineState *machine)
 {
@@ -XXX,XX +XXX,XX @@ static void remote_machine_init(MachineState *machine)
 
     if (s->vfio_user) {
         remote_iommu_setup(pci_host->bus);
+
+        msi_nonbroken = true;
+
+        vfu_object_set_bus_irq(pci_host->bus);
+    } else {
+        remote_iohub_init(&s->iohub);
+
+        pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                     &s->iohub, REMOTE_IOHUB_NB_PIRQS);
     }
 
-    remote_iohub_init(&s->iohub);
-
-    pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
-                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
-
     qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
 }
 
diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
 #include "hw/pci/pci.h"
 #include "qemu/timer.h"
 #include "exec/memory.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/msix.h"
+#include "hw/remote/vfio-user-obj.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -XXX,XX +XXX,XX @@ struct VfuObject {
     Error *unplug_blocker;
 
     int vfu_poll_fd;
+
+    MSITriggerFunc *default_msi_trigger;
+    MSIPrepareMessageFunc *default_msi_prepare_message;
+    MSIxPrepareMessageFunc *default_msix_prepare_message;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -XXX,XX +XXX,XX @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
     }
 }
 
+static int vfu_object_map_irq(PCIDevice *pci_dev, int intx)
+{
+    int pci_bdf = PCI_BUILD_BDF(pci_bus_num(pci_get_bus(pci_dev)),
+                                pci_dev->devfn);
+
+    return pci_bdf;
+}
+
+static void vfu_object_set_irq(void *opaque, int pirq, int level)
+{
+    PCIBus *pci_bus = opaque;
+    PCIDevice *pci_dev = NULL;
+    vfu_ctx_t *vfu_ctx = NULL;
+    int pci_bus_num, devfn;
+
+    if (level) {
+        pci_bus_num = PCI_BUS_NUM(pirq);
+        devfn = PCI_BDF_TO_DEVFN(pirq);
+
+        /*
+         * pci_find_device() performs at O(1) if the device is attached
+         * to the root PCI bus. Whereas, if the device is attached to a
+         * secondary PCI bus (such as when a root port is involved),
+         * finding the parent PCI bus could take O(n)
+         */
+        pci_dev = pci_find_device(pci_bus, pci_bus_num, devfn);
+
+        vfu_ctx = pci_dev->irq_opaque;
+
+        g_assert(vfu_ctx);
+
+        vfu_irq_trigger(vfu_ctx, 0);
+    }
+}
+
+static MSIMessage vfu_object_msi_prepare_msg(PCIDevice *pci_dev,
+                                             unsigned int vector)
+{
+    MSIMessage msg;
+
+    msg.address = 0;
+    msg.data = vector;
+
+    return msg;
+}
+
+static void vfu_object_msi_trigger(PCIDevice *pci_dev, MSIMessage msg)
+{
+    vfu_ctx_t *vfu_ctx = pci_dev->irq_opaque;
+
+    vfu_irq_trigger(vfu_ctx, msg.data);
+}
+
+static void vfu_object_setup_msi_cbs(VfuObject *o)
+{
+    o->default_msi_trigger = o->pci_dev->msi_trigger;
+    o->default_msi_prepare_message = o->pci_dev->msi_prepare_message;
+    o->default_msix_prepare_message = o->pci_dev->msix_prepare_message;
+
+    o->pci_dev->msi_trigger = vfu_object_msi_trigger;
+    o->pci_dev->msi_prepare_message = vfu_object_msi_prepare_msg;
+    o->pci_dev->msix_prepare_message = vfu_object_msi_prepare_msg;
+}
+
+static void vfu_object_restore_msi_cbs(VfuObject *o)
+{
+    o->pci_dev->msi_trigger = o->default_msi_trigger;
+    o->pci_dev->msi_prepare_message = o->default_msi_prepare_message;
+    o->pci_dev->msix_prepare_message = o->default_msix_prepare_message;
+}
+
+static void vfu_msix_irq_state(vfu_ctx_t *vfu_ctx, uint32_t start,
+                               uint32_t count, bool mask)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    Error *err = NULL;
+    uint32_t vector;
+
+    for (vector = start; vector < count; vector++) {
+        msix_set_mask(o->pci_dev, vector, mask, &err);
+        if (err) {
+            VFU_OBJECT_ERROR(o, "vfu: %s: %s", o->device,
+                             error_get_pretty(err));
+            error_free(err);
+            err = NULL;
+        }
+    }
+}
+
+static void vfu_msi_irq_state(vfu_ctx_t *vfu_ctx, uint32_t start,
+                              uint32_t count, bool mask)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    Error *err = NULL;
+    uint32_t vector;
+
+    for (vector = start; vector < count; vector++) {
+        msi_set_mask(o->pci_dev, vector, mask, &err);
+        if (err) {
+            VFU_OBJECT_ERROR(o, "vfu: %s: %s", o->device,
+                             error_get_pretty(err));
+            error_free(err);
+            err = NULL;
+        }
+    }
+}
+
+static int vfu_object_setup_irqs(VfuObject *o, PCIDevice *pci_dev)
+{
+    vfu_ctx_t *vfu_ctx = o->vfu_ctx;
+    int ret;
+
+    ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_INTX_IRQ, 1);
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (msix_nr_vectors_allocated(pci_dev)) {
+        ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_MSIX_IRQ,
+                                       msix_nr_vectors_allocated(pci_dev));
+        vfu_setup_irq_state_callback(vfu_ctx, VFU_DEV_MSIX_IRQ,
+                                     &vfu_msix_irq_state);
+    } else if (msi_nr_vectors_allocated(pci_dev)) {
+        ret = vfu_setup_device_nr_irqs(vfu_ctx, VFU_DEV_MSI_IRQ,
+                                       msi_nr_vectors_allocated(pci_dev));
+        vfu_setup_irq_state_callback(vfu_ctx, VFU_DEV_MSI_IRQ,
+                                     &vfu_msi_irq_state);
+    }
+
+    if (ret < 0) {
+        return ret;
+    }
+
+    vfu_object_setup_msi_cbs(o);
+
+    pci_dev->irq_opaque = vfu_ctx;
+
+    return 0;
+}
+
+void vfu_object_set_bus_irq(PCIBus *pci_bus)
+{
+    int bus_num = pci_bus_num(pci_bus);
+    int max_bdf = PCI_BUILD_BDF(bus_num, PCI_DEVFN_MAX - 1);
+
+    pci_bus_irqs(pci_bus, vfu_object_set_irq, vfu_object_map_irq, pci_bus,
+                 max_bdf);
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
 
     vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
 
+    ret = vfu_object_setup_irqs(o, o->pci_dev);
+    if (ret < 0) {
+        error_setg(errp, "vfu: Failed to setup interrupts for %s",
+                   o->device);
+        goto fail;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
@@ -XXX,XX +XXX,XX @@ fail:
         o->unplug_blocker = NULL;
     }
     if (o->pci_dev) {
+        vfu_object_restore_msi_cbs(o);
+        o->pci_dev->irq_opaque = NULL;
         object_unref(OBJECT(o->pci_dev));
         o->pci_dev = NULL;
     }
@@ -XXX,XX +XXX,XX @@ static void vfu_object_finalize(Object *obj)
     }
 
     if (o->pci_dev) {
+        vfu_object_restore_msi_cbs(o);
+        o->pci_dev->irq_opaque = NULL;
         object_unref(OBJECT(o->pci_dev));
         o->pci_dev = NULL;
     }
diff --git a/stubs/vfio-user-obj.c b/stubs/vfio-user-obj.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/stubs/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@
+#include "qemu/osdep.h"
+#include "hw/remote/vfio-user-obj.h"
+
+void vfu_object_set_bus_irq(PCIBus *pci_bus)
+{
+}
diff --git a/hw/remote/trace-events b/hw/remote/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/trace-events
+++ b/hw/remote/trace-events
@@ -XXX,XX +XXX,XX @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
 vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""
 vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
 vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
+vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
diff --git a/stubs/meson.build b/stubs/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/stubs/meson.build
+++ b/stubs/meson.build
@@ -XXX,XX +XXX,XX @@ if have_system
 else
   stub_ss.add(files('qdev.c'))
 endif
+stub_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_false: files('vfio-user-obj.c'))
-- 
2.36.1

From: Jagannathan Raman <jag.raman@oracle.com>

Adds handler to reset a remote device

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 112eeadf3bc4c6cdb100bc3f9a6fcfc20b467c1b.1655151679.git.jag.raman@oracle.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/remote/vfio-user-obj.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/hw/remote/vfio-user-obj.c b/hw/remote/vfio-user-obj.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/remote/vfio-user-obj.c
+++ b/hw/remote/vfio-user-obj.c
@@ -XXX,XX +XXX,XX @@ void vfu_object_set_bus_irq(PCIBus *pci_bus)
                  max_bdf);
 }
 
+static int vfu_object_device_reset(vfu_ctx_t *vfu_ctx, vfu_reset_type_t type)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    /* vfu_object_ctx_run() handles lost connection */
+    if (type == VFU_RESET_LOST_CONN) {
+        return 0;
+    }
+
+    qdev_reset_all(DEVICE(o->pci_dev));
+
+    return 0;
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -XXX,XX +XXX,XX @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         goto fail;
     }
 
+    ret = vfu_setup_device_reset_cb(o->vfu_ctx, &vfu_object_device_reset);
+    if (ret < 0) {
+        error_setg(errp, "vfu: Failed to setup reset callback");
+        goto fail;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
-- 
2.36.1

Every laio_io_plug() call has a matching laio_io_unplug() call. There is
a plugged counter that tracks the number of levels of plugging and
allows for nesting.

The plugged counter must reflect the balance between laio_io_plug() and
laio_io_unplug() calls accurately. Otherwise I/O stalls occur since
io_submit(2) calls are skipped while plugged.

Reported-by: Nikolay Tenev <nt@storpool.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Message-id: 20220609164712.1539045-2-stefanha@redhat.com
Cc: Stefano Garzarella <sgarzare@redhat.com>
Fixes: 68d7946648 ("linux-aio: add `dev_max_batch` parameter to laio_io_unplug()")
[Stefano Garzarella suggested adding a Fixes tag.
--Stefan]
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/linux-aio.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/block/linux-aio.c b/block/linux-aio.c
index XXXXXXX..XXXXXXX 100644
--- a/block/linux-aio.c
+++ b/block/linux-aio.c
@@ -XXX,XX +XXX,XX @@ void laio_io_unplug(BlockDriverState *bs, LinuxAioState *s,
                     uint64_t dev_max_batch)
 {
     assert(s->io_q.plugged);
+    s->io_q.plugged--;
+
     if (s->io_q.in_queue >= laio_max_batch(s, dev_max_batch) ||
-        (--s->io_q.plugged == 0 &&
+        (!s->io_q.plugged &&
          !s->io_q.blocked && !QSIMPLEQ_EMPTY(&s->io_q.pending))) {
         ioq_submit(s);
     }
-- 
2.36.1

It may not be obvious why laio_io_unplug() checks max batch. I discussed
this with Stefano and have added a comment summarizing the reason.

Cc: Stefano Garzarella <sgarzare@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Message-id: 20220609164712.1539045-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block/linux-aio.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/block/linux-aio.c b/block/linux-aio.c
index XXXXXXX..XXXXXXX 100644
--- a/block/linux-aio.c
+++ b/block/linux-aio.c
@@ -XXX,XX +XXX,XX @@ void laio_io_unplug(BlockDriverState *bs, LinuxAioState *s,
     assert(s->io_q.plugged);
     s->io_q.plugged--;
 
+    /*
+     * Why max batch checking is performed here:
+     * Another BDS may have queued requests with a higher dev_max_batch and
+     * therefore in_queue could now exceed our dev_max_batch. Re-check the max
+     * batch so we can honor our device's dev_max_batch.
+     */
     if (s->io_q.in_queue >= laio_max_batch(s, dev_max_batch) ||
         (!s->io_q.plugged &&
          !s->io_q.blocked && !QSIMPLEQ_EMPTY(&s->io_q.pending))) {
-- 
2.36.1