This patch introduces the VFIO-over-socket protocol specification, which
is designed to allow devices to be emulated outside QEMU, in a separate
process. VFIO-over-socket reuses the existing VFIO defines, structs and
concepts.

It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"

Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>

---

Changed since v1:
  * fix coding style issues
  * update MAINTAINERS for VFIO-over-socket
  * add vfio-over-socket to ToC

Changed since v2:
  * fix whitespace

Regarding the build failure, I have not been able to reproduce it locally
using the docker image on my Debian 10.4 machine.
---
 MAINTAINERS                     |    6 +
 docs/devel/index.rst            |    1 +
 docs/devel/vfio-over-socket.rst | 1135 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 1142 insertions(+)
 create mode 100644 docs/devel/vfio-over-socket.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 030faf0..bb81590 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1732,6 +1732,12 @@ F: hw/vfio/ap.c
 F: docs/system/s390x/vfio-ap.rst
 L: qemu-s390x@nongnu.org
 
+VFIO-over-socket
+M: John G Johnson <john.g.johnson@oracle.com>
+M: Thanos Makatos <thanos.makatos@nutanix.com>
+S: Supported
+F: docs/devel/vfio-over-socket.rst
+
 vhost
 M: Michael S. Tsirkin <mst@redhat.com>
 S: Supported
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index ae6eac7..0439460 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -30,3 +30,4 @@ Contents:
    reset
    s390-dasd-ipl
    clocks
+   vfio-over-socket
diff --git a/docs/devel/vfio-over-socket.rst b/docs/devel/vfio-over-socket.rst
new file mode 100644
index 0000000..b474f23
--- /dev/null
+++ b/docs/devel/vfio-over-socket.rst
@@ -0,0 +1,1135 @@
+***************************************
+VFIO-over-socket Protocol Specification
+***************************************
+
+Version 0.1
+
+Introduction
+============
+VFIO-over-socket, also known as vfio-user, is a protocol that allows a device
+to be virtualized in a separate process outside of QEMU. VFIO-over-socket
+devices consist of a generic VFIO device type, living inside QEMU, which we
+call the client, and the core device implementation, living outside QEMU, which
+we call the server. VFIO-over-socket can be the main transport mechanism for
+multi-process QEMU, however it can be used by other applications offering
+device virtualization. Explaining the advantages of a
+disaggregated/multi-process QEMU, and device virtualization outside QEMU in
+general, is beyond the scope of this document.
+
+This document focuses on specifying the VFIO-over-socket protocol. VFIO has
+been chosen for the following reasons:
+
+1) It is a mature and stable API, backed by an extensively used framework.
+2) The existing VFIO client implementation (qemu/hw/vfio/) can be largely
+   reused.
+
+In a proof of concept implementation it has been demonstrated that using VFIO
+over a UNIX domain socket is a viable option. VFIO-over-socket is designed with
+QEMU in mind, however it could be used by other client applications. The
+VFIO-over-socket protocol does not require that QEMU's VFIO client
+implementation is used in QEMU. None of the VFIO kernel modules are required
+for supporting the protocol, neither in the client nor the server, only the
+source header files are used.
+
+The main idea is to allow a virtual device to function in a separate process in
+the same host over a UNIX domain socket. A UNIX domain socket (AF_UNIX) is
+chosen because we can trivially send file descriptors over it, which in turn
+allows:
+
+* Sharing of guest memory for DMA with the virtual device process.
+* Sharing of virtual device memory with the guest for fast MMIO.
+* Efficient sharing of eventfd's for triggering interrupts.
+
+However, other socket types could be used which allows the virtual device
+process to run in a separate guest in the same host (AF_VSOCK) or remotely
+(AF_INET). Theoretically the underlying transport doesn't necessarily have to
+be a socket, however we don't examine such alternatives. In this document we
+focus on using a UNIX domain socket and introduce basic support for the other
+two types of sockets without considering performance implications.
+
+This document does not yet describe any internal details of the server-side
+implementation, however QEMU's VFIO client implementation will have to be
+adapted according to this protocol in order to support VFIO-over-socket virtual
+devices.
+
+VFIO
+====
+VFIO is a framework that allows a physical device to be securely passed through
+to a user space process; the kernel does not drive the device at all.
+Typically, the user space process is a VM and the device is passed through to
+it in order to achieve high performance. VFIO provides an API and the required
+functionality in the kernel. QEMU has adopted VFIO to allow a guest virtual
+machine to directly access physical devices, instead of emulating them in
+software
+
+VFIO-over-socket reuses the core VFIO concepts defined in its API, but
+implements them as messages to be sent over a UNIX-domain socket. It does not
+change the kernel-based VFIO in any way, in fact none of the VFIO kernel
+modules need to be loaded to use VFIO-over-socket. It is also possible for QEMU
+to concurrently use the current kernel-based VFIO for one guest device, and use
+VFIO-over-socket for another device in the same guest.
+
+VFIO Device Model
+-----------------
+A device under VFIO presents a standard VFIO model to the user process. Many
+of the VFIO operations in the existing kernel model use the ioctl() system
+call, and references to the existing model are called the ioctl()
+implementation in this document.
+
+The following sections describe the set of messages that implement the VFIO
+device model over a UNIX domain socket. In many cases, the messages are direct
+translations of data structures used in the ioctl() implementation. Messages
+derived from ioctl()s will have a name derived from the ioctl() command name.
+E.g., the VFIO_GET_INFO ioctl() command becomes a VFIO_USER_GET_INFO message.
+The purpose for this reuse is to share as much code as feasible with the
+ioctl() implementation.
+
+Client and Server
+^^^^^^^^^^^^^^^^^
+The socket connects two processes together: a client process and a server
+process. In the context of this document, the client process is the process
+emulating a guest virtual machine, such as QEMU. The server process is a
+process that provides device emulation.
+
+Connection Initiation
+^^^^^^^^^^^^^^^^^^^^^
+After the client connects to the server, the initial server message is
+VFIO_USER_VERSION to propose a protocol version and set of capabilities to
+apply to the session. The client replies with a compatible version and set of
+capabilities it will support, or closes the connection if it cannot support the
+advertised version.
+
+Guest Memory Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+The client uses VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP messages to inform
+the server of the valid guest DMA ranges that the server can access on behalf
+of a device. Guest memory may be accessed by the server via VFIO_USER_DMA_READ
+and VFIO_USER_DMA_WRITE messages over the socket.
+
+An optimization for server access to guest memory is for the client to provide
+file descriptors the server can mmap() to directly access guest memory. Note
+that mmap() privileges cannot be revoked by the client, therefore file
+descriptors should only be exported in environments where the client trusts the
+server not to corrupt guest memory.
+
+Device Information
+^^^^^^^^^^^^^^^^^^
+The client uses a VFIO_USER_DEVICE_GET_INFO message to query the server for
+information about the device. This information includes:
+
+* The device type and capabilities,
+* the number of memory regions, and
+* the device presents to the guest the number of interrupt types the device
+  supports.
+
+Region Information
+^^^^^^^^^^^^^^^^^^
+The client uses VFIO_USER_DEVICE_GET_REGION_INFO messages to query the server
+for information about the device's memory regions. This information describes:
+
+* Read and write permissions, whether it can be memory mapped, and whether it
+  supports additional capabilities.
+* Region index, size, and offset.
+
+When a region can be mapped by the client, the server provides a file
+descriptor which the client can mmap(). The server is responsible for polling
+for client updates to memory mapped regions.
+
+Region Capabilities
+"""""""""""""""""""
+Some regions have additional capabilities that cannot be described adequately
+by the region info data structure. These capabilities are returned in the
+region info reply in a list similar to PCI capabilities in a PCI device's
+configuration space.
+
+Sparse Regions
+""""""""""""""
+A region can be memory-mappable in whole or in part. When only a subset of a
+region can be mapped by the client, a VFIO_REGION_INFO_CAP_SPARSE_MMAP
+capability is included in the region info reply. This capability describes
+which portions can be mapped by the client.
+
+For example, in a virtual NVMe controller, sparse regions can be used so that
+accesses to the NVMe registers (found in the beginning of BAR0) are trapped (an
+infrequent an event), while allowing direct access to the doorbells (an
+extremely frequent event as every I/O submission requires a write to BAR0),
+found right after the NVMe registers in BAR0.
+
+Interrupts
+^^^^^^^^^^
+The client uses VFIO_USER_DEVICE_GET_IRQ_INFO messages to query the server for
+the device's interrupt types. The interrupt types are specific to the bus the
+device is attached to, and the client is expected to know the capabilities of
+each interrupt type. The server can signal an interrupt either with
+VFIO_USER_VM_INTERRUPT messages over the socket, or can directly inject
+interrupts into the guest via an event file descriptor. The client configures
+how the server signals an interrupt with VFIO_USER_SET_IRQS messages.
+
+Device Read and Write
+^^^^^^^^^^^^^^^^^^^^^
+When the guest executes load or store operations to device memory, the client
+forwards these operations to the server with VFIO_USER_REGION_READ or
+VFIO_USER_REGION_WRITE messages. The server will reply with data from the
+device on read operations or an acknowledgement on write operations.
+
+DMA
+^^^
+When a device performs DMA accesses to guest memory, the server will forward
+them to the client with VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages.
+These messages can only be used to access guest memory the client has
+configured into the server.
+
+Protocol Specification
+======================
+To distinguish from the base VFIO symbols, all VFIO-over-socket symbols are
+prefixed with vfio_user or VFIO_USER. In revision 0.1, all data is in the
+little-endian format, although this may be relaxed in future revision in cases
+where the client and server are both big-endian. The messages are formatted
+for seamless reuse of the native VFIO structs. A server can serve:
+
+1) multiple clients, and/or
+2) multiple virtual devices, belonging to one or more clients.
+
+Therefore each message requires a header that uniquely identifies the virtual
+device. It is a server-side implementation detail whether a single server
+handles multiple virtual devices from the same or multiple guests.
+
+Socket
+------
+A single UNIX domain socket is assumed to be used for each device. The location
+of the socket is implementation-specific. Multiplexing clients, devices, and
+servers over the same socket is not supported in this version of the protocol,
+but a device ID field exists in the message header so that a future support can
+be added without a major version change.
+
+Authentication
+--------------
+For AF_UNIX, we rely on OS mandatory access controls on the socket files,
+therefore it is up to the management layer to set up the socket as required.
+Socket types than span guests or hosts will require a proper authentication
+mechanism. Defining that mechanism is deferred to a future version of the
+protocol.
+
+Request Concurrency
+-------------------
+There can be multiple outstanding requests per virtual device, e.g. a
+frame buffer where the guest does multiple stores to the virtual device. The
+server can execute and reorder non-conflicting requests in parallel, depending
+on the device semantics.
+
+Socket Disconnection Behavior
+-----------------------------
+The server and the client can disconnect from each other, either intentionally
+or unexpectedly. Both the client and the server need to know how to handle such
+events.
+
+Server Disconnection
+^^^^^^^^^^^^^^^^^^^^
+A server disconnecting from the client may indicate that:
+
+1) A virtual device has been restarted, either intentionally (e.g. because of a
+device update) or unintentionally (e.g. because of a crash). In any case, the
+virtual device will come back so the client should not do anything (e.g. simply
+reconnect and retry failed operations).
+
+2) A virtual device has been shut down with no intention to be restarted.
+
+It is impossible for the client to know whether or not a failure is
+intermittent or innocuous and should be retried, therefore the client should
+attempt to reconnect to the socket. Since an intentional server restart (e.g.
+due to an upgrade) might take some time, a reasonable timeout should be used.
+In cases where the disconnection is expected (e.g. the guest shutting down), no
+new requests will be sent anyway so this situation doesn't pose a problem. The
+control stack will clean up accordingly.
+
+Parametrizing this behaviour by having the virtual device advertise a
+reasonable reconnect is deferred to a future version of the protocol.
+
+Client Disconnection
+^^^^^^^^^^^^^^^^^^^^
+The client disconnecting from the server primarily means that the QEMU process
+has exited. Currently this means that the guest is shut down so the device is
+no longer needed therefore the server can automatically exit. However, there
+can be cases where a client disconnect should not result in a server exit:
+
+1) A single server serving multiple clients.
+2) A multi-process QEMU upgrading itself step by step, which isn't yet
+   implemented.
+
+Therefore in order for the protocol to be forward compatible the server should
+take no action when the client disconnects. If anything happens to the client
+process the control stack will know about it and can clean up resources
+accordingly.
+
+Request Retry and Response Timeout
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+QEMU's VFIO retries certain operations if they fail. While this makes sense for
+real HW, we don't know for sure whether it makes sense for virtual devices. A
+failed request is a request that has been successfully sent and has been
+responded to with an error code. Failure to send the request in the first place
+(e.g. because the socket is disconnected) is a different type of error examined
+earlier in the disconnect section.
+
+Defining a retry and timeout scheme if deferred to a future version of the
+protocol.
+
+Commands
+--------
+The following table lists the VFIO message command IDs, and whether the
+message request is sent from the client or the server.
+
++----------------------------------+---------+-------------------+
+| Name                             | Command | Request Direction |
++==================================+=========+===================+
+| VFIO_USER_VERSION                | 1       | server → client   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_MAP                | 2       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_UNMAP              | 3       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_INFO        | 4       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_REGION_INFO | 5       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_IRQ_INFO    | 6       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_SET_IRQS        | 7       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_REGION_READ            | 8       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_REGION_WRITE           | 9       | client → server   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_READ               | 10      | server → client   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_READ               | 11      | server → client   |
++----------------------------------+---------+-------------------+
+| VFIO_USER_VM_INTERRUPT           | 12      | server → client   |
++----------------------------------+---------+-------------------+
+| VFIO_DEVICE_RESET                | 13      | client → server   |
++----------------------------------+---------+-------------------+
+
+Header
+------
+All messages are preceded by a 16 byte header that contains basic information
+about the message. The header is followed by message-specific data described
+in the sections below.
+
++----------------+--------+-------------+
+| Name           | Offset | Size        |
++================+========+=============+
+| Device ID      | 0      | 2           |
++----------------+--------+-------------+
+| Message ID     | 2      | 2           |
++----------------+--------+-------------+
+| Command        | 4      | 4           |
++----------------+--------+-------------+
+| Message size   | 8      | 4           |
++----------------+--------+-------------+
+| Flags          | 12     | 4           |
++----------------+--------+-------------+
+|                | +-----+------------+ |
+|                | | Bit | Definition | |
+|                | +=====+============+ |
+|                | | 0   | Reply      | |
+|                | +-----+------------+ |
+|                | | 1   | No_reply   | |
+|                | +-----+------------+ |
++----------------+--------+-------------+
+| <message data> | 16     | variable    |
++----------------+--------+-------------+
+
+* Device ID identifies the destination device of the message. This field is
+  reserved when the server only supports one device per socket.
+* Message ID identifies the message, and is used in the message acknowledgement.
+* Command specifies the command to be executed, listed in the Command Table.
+* Message size contains the size of the entire message, including the header.
+* Flags contains attributes of the message:
+
+  * The reply bit differentiates request messages from reply messages. A reply
+    message acknowledges a previous request with the same message ID.
+  * No_reply indicates that no reply is needed for this request. This is
+    commonly used when multiple requests are sent, and only the last needs
+    acknowledgement.
+
+VFIO_USER_VERSION
+-----------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Device ID    | 0                      |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 1                      |
++--------------+------------------------+
+| Message size | 16 + version length    |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Version      | JSON byte array        |
++--------------+------------------------+
+
+This is the initial message sent by the server after the socket connection is
+established. The version is in JSON format, and the following objects must be
+included:
+
++--------------+--------+---------------------------------------------------+
+| Name         | Type   | Description                                       |
++==============+========+===================================================+
+| version      | object | {“major”: <number>, “minor”: <number>}            |
+|              |        | Version supported by the sender, e.g. “0.1”.      |
++--------------+--------+---------------------------------------------------+
+| type         | string | Fixed to “vfio-user”.                             |
++--------------+--------+---------------------------------------------------+
+| capabilities | array  | Reserved. Can be omitted for v0.1, otherwise must |
+|              |        | be empty.                                         |
++--------------+--------+---------------------------------------------------+
+
+Versioning and Feature Support
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Upon accepting a connection, the server must send a VFIO_USER_VERSION message
+proposing a protocol version and a set of capabilities. The client compares
+these with the versions and capabilities it supports and sends a
+VFIO_USER_VERSION reply according to the following rules.
+
+* The major version in the reply must be the same as proposed. If the client
+  does not support the proposed major, it closes the connection.
+* The minor version in the reply must be equal to or less than the minor
+  version proposed.
+* The capability list must be a subset of those proposed. If the client
+  requires a capability the server did not include, it closes the connection.
+* If type is not “vfio-user”, the client closes the connection.
+
+The protocol major version will only change when incompatible protocol changes
+are made, such as changing the message format. The minor version may change
+when compatible changes are made, such as adding new messages or capabilities,
+Both the client and server must support all minor versions less than the
+maximum minor version it supports. E.g., an implementation that supports
+version 1.3 must also support 1.0 through 1.2.
+
+VFIO_USER_DMA_MAP
+-----------------
+
+VFIO_USER_DMA_UNMAP
+-------------------
+
+Message Format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Device ID    | 0                      |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | MAP=2, UNMAP=3         |
++--------------+------------------------+
+| Message size | 16 + table size        |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Table        | array of table entries |
++--------------+------------------------+
+
+This message is sent by the client to the server to inform it of the guest
+memory regions the device can access. It must be sent before the device can
+perform any DMA to the guest. It is normally sent directly after the version
+handshake is completed, but may also occur when memory is added or subtracted
+in the guest.
+
+The table is an array of the following structure. This structure is 32 bytes
+in size, so the message size will be 16 + (# of table entries * 32). If a
+region being added can be directly mapped by the server, an array of file
+descriptors will be sent as part of the message meta-data. Each region entry
+will have a corresponding file descriptor. On AF_UNIX sockets, the file
+descriptors will be passed as SCM_RIGHTS type ancillary data.
+
+Table entry format
+^^^^^^^^^^^^^^^^^^
+
++-------------+--------+-------------+
+| Name        | Offset | Size        |
++=============+========+=============+
+| Address     | 0      | 8           |
++-------------+--------+-------------+
+| Size        | 8      | 8           |
++-------------+--------+-------------+
+| Offset      | 16     | 8           |
++-------------+--------+-------------+
+| Protections | 24     | 4           |
++-------------+--------+-------------+
+| Flags       | 28     | 4           |
++-------------+--------+-------------+
+|             | +-----+------------+ |
+|             | | Bit | Definition | |
+|             | +=====+============+ |
+|             | | 0   | Mappable   | |
+|             | +-----+------------+ |
++-------------+--------+-------------+
+
+* Address is the base DMA address of the region.
+* Size is the size of the region.
+* Offset is the file offset of the region with respect to the associated file
+  descriptor.
+* Protections are the region's protection attributes as encoded in
+  ``<sys/mman.h>``.
+* Flags contain the following region attributes:
+
+  * Mappable indicate the region can be mapped via the mmap() system call using
+    the file descriptor provided in the message meta-data.
+
+VFIO_USER_DEVICE_GET_INFO
+-------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+----------------------------+
+| Name         | Value                      |
++==============+============================+
+| Device ID    | <ID>                       |
++--------------+----------------------------+
+| Message ID   | <ID>                       |
++--------------+----------------------------+
+| Command      | 4                          |
++--------------+----------------------------+
+| Message size | 16 in request, 32 in reply |
++--------------+----------------------------+
+| Flags        | Reply bit set in reply     |
++--------------+----------------------------+
+| Device info  | VFIO device info           |
++--------------+----------------------------+
+
+This message is sent by the client to the server to query for basic information
+about the device. Only the message header is needed in the request message.
+The VFIO device info structure is defined in ``<sys/vfio.h>`` (``struct
+vfio_device_info``).
+
+VFIO device info format
+^^^^^^^^^^^^^^^^^^^^^^^
+
++-------------+--------+--------------------------+
+| Name        | Offset | Size                     |
++=============+========+==========================+
+| argsz       | 16     | 4                        |
++-------------+--------+--------------------------+
+| flags       | 20     | 4                        |
++-------------+--------+--------------------------+
+|             | +-----+-------------------------+ |
+|             | | Bit | Definition              | |
+|             | +=====+=========================+ |
+|             | | 0   | VFIO_DEVICE_FLAGS_RESET | |
+|             | +-----+-------------------------+ |
+|             | | 1   | VFIO_DEVICE_FLAGS_PCI   | |
+|             | +-----+-------------------------+ |
++-------------+--------+--------------------------+
+| num_regions | 24     | 4                        |
++-------------+--------+--------------------------+
+| num_irqs    | 28     | 4                        |
++-------------+--------+--------------------------+
+
+* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
+  implementation.
+* flags contains the following device attributes.
+
+  * VFIO_DEVICE_FLAGS_RESET indicates the device supports the
+    VFIO_USER_DEVICE_RESET message.
+  * VFIO_DEVICE_FLAGS_PCI indicates the device is a PCI device.
+
+* num_regions is the number of memory regions the device exposes.
+* num_irqs is the number of distinct interrupt types the device supports.
+
+This version of the protocol only supports PCI devices. Additional devices may
+be supported in future versions.
+
+VFIO_USER_DEVICE_GET_REGION_INFO
+--------------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------+
+| Name         | Value            |
++==============+==================+
+| Device ID    | <ID>             |
++--------------+------------------+
+| Message ID   | <ID>             |
++--------------+------------------+
+| Command      | 5                |
++--------------+------------------+
+| Message size | 48 + any caps    |
++--------------+------------------+
+| Flags Reply  | bit set in reply |
++--------------+------------------+
+| Region info  | VFIO region info |
++--------------+------------------+
+
+This message is sent by the client to the server to query for information about
+device memory regions. The VFIO region info structure is defined in
+``<sys/vfio.h>`` (``struct vfio_region_info``).
+
+VFIO region info format
+^^^^^^^^^^^^^^^^^^^^^^^
+
++------------+--------+------------------------------+
+| Name       | Offset | Size                         |
++============+========+==============================+
+| argsz      | 16     | 4                            |
++------------+--------+------------------------------+
+| flags      | 20     | 4                            |
++------------+--------+------------------------------+
+|            | +-----+-----------------------------+ |
+|            | | Bit | Definition                  | |
+|            | +=====+=============================+ |
+|            | | 0   | VFIO_REGION_INFO_FLAG_READ  | |
+|            | +-----+-----------------------------+ |
+|            | | 1   | VFIO_REGION_INFO_FLAG_WRITE | |
+|            | +-----+-----------------------------+ |
+|            | | 2   | VFIO_REGION_INFO_FLAG_MMAP  | |
+|            | +-----+-----------------------------+ |
+|            | | 3   | VFIO_REGION_INFO_FLAG_CAPS  | |
+|            | +-----+-----------------------------+ |
++------------+--------+------------------------------+
+| index      | 24     | 4                            |
++------------+--------+------------------------------+
+| cap_offset | 28     | 4                            |
++------------+--------+------------------------------+
+| size       | 32     | 8                            |
++------------+--------+------------------------------+
+| offset     | 40     | 8                            |
++------------+--------+------------------------------+
+
+* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
+  implementation.
+* flags are attributes of the region:
+
+  * VFIO_REGION_INFO_FLAG_READ allows client read access to the region.
+  * VFIO_REGION_INFO_FLAG_WRITE allows client write access region.
+  * VFIO_REGION_INFO_FLAG_MMAP specifies the client can mmap() the region. When
+    this flag is set, the reply will include a file descriptor in its meta-data.
+    On AF_UNIX sockets, the file descriptors will be passed as SCM_RIGHTS type
+    ancillary data.
+  * VFIO_REGION_INFO_FLAG_CAPS indicates additional capabilities found in the
+    reply.
+
+* index is the index of memory region being queried, it is the only field that
+  is required to be set in the request message.
+* cap_offset describes where additional region capabilities can be found.
+  cap_offset is relative to the beginning of the VFIO region info structure.
+  The data structure it points is a VFIO cap header defined in ``<sys/vfio.h>``.
+* size is the size of the region.
+* offset is the offset given to the mmap() system call for regions with the
+  MMAP attribute. It is also used as the base offset when mapping a VFIO
+  sparse mmap area, described below.
+
+VFIO Region capabilities
+^^^^^^^^^^^^^^^^^^^^^^^^
+The VFIO region information can also include a capabilities list. This list is
+similar to a PCI capability list - each entry has a common header that
+identifies a capability and where the next capability in the list can be found.
+The VFIO capability header format is defined in ``<sys/vfio.h>`` (``struct
+vfio_info_cap_header``).
+
+VFIO cap header format
+^^^^^^^^^^^^^^^^^^^^^^
+
++---------+--------+------+
+| Name    | Offset | Size |
++=========+========+======+
+| id      | 0      | 2    |
++---------+--------+------+
+| version | 2      | 2    |
++---------+--------+------+
+| next    | 4      | 4    |
++---------+--------+------+
+
+* id is the capability identity.
+* version is a capability-specific version number.
+* next specifies the offset of the next capability in the capability list. It
+  is relative to the beginning of the VFIO region info structure.
+
+VFIO sparse mmap
+^^^^^^^^^^^^^^^^
+
++------------------+----------------------------------+
+| Name             | Value                            |
++==================+==================================+
+| id               | VFIO_REGION_INFO_CAP_SPARSE_MMAP |
++------------------+----------------------------------+
+| version          | 0x1                              |
++------------------+----------------------------------+
+| next             | <next>                           |
++------------------+----------------------------------+
+| sparse mmap info | VFIO region info sparse mmap     |
++------------------+----------------------------------+
+
+The only capability supported in this version of the protocol is for sparse
+mmap. This capability is defined when only a subrange of the region supports
+direct access by the client via mmap(). The VFIO sparse mmap area is defined in
+``<sys/vfio.h>`` (``struct vfio_region_sparse_mmap_area``).
+
+VFIO region info cap sparse mmap
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++----------+--------+------+
+| Name     | Offset | Size |
++==========+========+======+
+| nr_areas | 0      | 4    |
++----------+--------+------+
+| reserved | 4      | 4    |
++----------+--------+------+
+| offset   | 8      | 8    |
++----------+--------+------+
+| size     | 16     | 9    |
++----------+--------+------+
+| ...      |        |      |
++----------+--------+------+
+
+* nr_areas is the number of sparse mmap areas in the region.
+* offset and size describe a single area that can be mapped by the client.
+  There will be nr_areas pairs of offset and size. The offset will be added to
+  the base offset given in the VFIO_USER_DEVICE_GET_REGION_INFO to form the
+  offset argument of the subsequent mmap() call.
+
+The VFIO sparse mmap area is defined in ``<sys/vfio.h>`` (``struct
+vfio_region_info_cap_sparse_mmap``).
+
+VFIO_USER_DEVICE_GET_IRQ_INFO
+-----------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Device ID    | <ID>                   |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 6                      |
++--------------+------------------------+
+| Message size | 32                     |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| IRQ info     | VFIO IRQ info          |
++--------------+------------------------+
+
+This message is sent by the client to the server to query for information about
+device interrupt types. The VFIO IRQ info structure is defined in
+``<sys/vfio.h>`` (``struct vfio_irq_info``).
+
+VFIO IRQ info format
+^^^^^^^^^^^^^^^^^^^^
+
++-------+--------+---------------------------+
+| Name  | Offset | Size                      |
++=======+========+===========================+
+| argsz | 16     | 4                         |
++-------+--------+---------------------------+
+| flags | 20     | 4                         |
++-------+--------+---------------------------+
+|       | +-----+--------------------------+ |
+|       | | Bit | Definition               | |
+|       | +=====+==========================+ |
+|       | | 0   | VFIO_IRQ_INFO_EVENTFD    | |
+|       | +-----+--------------------------+ |
+|       | | 1   | VFIO_IRQ_INFO_MASKABLE   | |
+|       | +-----+--------------------------+ |
+|       | | 2   | VFIO_IRQ_INFO_AUTOMASKED | |
+|       | +-----+--------------------------+ |
+|       | | 3   | VFIO_IRQ_INFO_NORESIZE   | |
+|       | +-----+--------------------------+ |
++-------+--------+---------------------------+
+| index | 24     | 4                         |
++-------+--------+---------------------------+
+| count | 28     | 4                         |
++-------+--------+---------------------------+
+
+* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
+  implementation.
+* flags defines IRQ attributes:
+
+  * VFIO_IRQ_INFO_EVENTFD indicates the IRQ type can support server eventfd
+    signalling.
+  * VFIO_IRQ_INFO_MASKABLE indicates that the IRQ type supports the MASK and
+    UNMASK actions in a VFIO_USER_DEVICE_SET_IRQS message.
+  * VFIO_IRQ_INFO_AUTOMASKED indicates the IRQ type masks itself after being
+    triggered, and the client must send an UNMASK action to receive new
+    interrupts.
+  * VFIO_IRQ_INFO_NORESIZE indicates VFIO_USER_SET_IRQS operations setup
+    interrupts as a set, and new subindexes cannot be enabled without disabling
+    the entire type.
+
+* index is the index of IRQ type being queried, it is the only field that is
+  required to be set in the request message.
+* count describes the number of interrupts of the queried type.
+
+VFIO_USER_DEVICE_SET_IRQS
+-------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
+| Device ID    | <ID>                   |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 7                      |
++--------------+------------------------+
+| Message size | 36 + any data          |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| IRQ set      | VFIO IRQ set           |
++--------------+------------------------+
+
+This message is sent by the client to the server to set actions for device
+interrupt types. The VFIO IRQ set structure is defined in ``<sys/vfio.h>``
+(``struct vfio_irq_set``).
+
+VFIO IRQ info format
+^^^^^^^^^^^^^^^^^^^^
+
++-------+--------+------------------------------+
+| Name  | Offset | Size                         |
++=======+========+==============================+
+| argsz | 6      | 4                            |
++-------+--------+------------------------------+
+| flags | 20     | 4                            |
++-------+--------+------------------------------+
+|       | +-----+-----------------------------+ |
+|       | | Bit | Definition                  | |
+|       | +=====+=============================+ |
+|       | | 0   | VFIO_IRQ_SET_DATA_NONE      | |
+|       | +-----+-----------------------------+ |
+|       | | 1   | VFIO_IRQ_SET_DATA_BOOL      | |
+|       | +-----+-----------------------------+ |
+|       | | 2   | VFIO_IRQ_SET_DATA_EVENTFD   | |
+|       | +-----+-----------------------------+ |
+|       | | 3   | VFIO_IRQ_SET_ACTION_MASK    | |
+|       | +-----+-----------------------------+ |
+|       | | 4   | VFIO_IRQ_SET_ACTION_UNMASK  | |
+|       | +-----+-----------------------------+ |
+|       | | 5   | VFIO_IRQ_SET_ACTION_TRIGGER | |
+|       | +-----+-----------------------------+ |
++-------+--------+------------------------------+
+| index | 24     | 4                            |
++-------+--------+------------------------------+
+| start | 28     | 4                            |
++-------+--------+------------------------------+
+| count | 32     | 4                            |
++-------+--------+------------------------------+
+| data  | 36     | variable                     |
++-------+--------+------------------------------+
+
+* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
+  implementation.
+* flags defines the action performed on the interrupt range. The DATA flags
+  describe the data field sent in the message; the ACTION flags describe the
+  action to be performed. The flags are mutually exclusive for both sets.
+
+  * VFIO_IRQ_SET_DATA_NONE indicates there is no data field in the request. The
+    action is performed unconditionally.
+  * VFIO_IRQ_SET_DATA_BOOL indicates the data field is an array of boolean
+    bytes. The action is performed if the corresponding boolean is true.
+  * VFIO_IRQ_SET_DATA_EVENTFD indicates an array of event file descriptors was
+    sent in the message meta-data. These descriptors will be signalled when the
+    action defined by the action flags occurs. In AF_UNIX sockets, the
+    descriptors are sent as SCM_RIGHTS type ancillary data.
+  * VFIO_IRQ_SET_ACTION_MASK indicates a masking event. It can be used with
+    VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to mask an interrupt, or
+    with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the guest masks
+    the interrupt.
+  * VFIO_IRQ_SET_ACTION_UNMASK indicates an unmasking event. It can be used
+    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to unmask an
+    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
+    guest unmasks the interrupt.
+  * VFIO_IRQ_SET_ACTION_TRIGGER indicates a triggering event. It can be used
+    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to trigger an
+    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
+    guest triggers the interrupt.
+
+* index is the index of IRQ type being setup.
+* start is the start of the subindex being set.
+* count describes the number of sub-indexes being set. As a special case, a
+  count of 0 with data flags of VFIO_IRQ_SET_DATA_NONE disables all interrupts
+  of the index data is an optional field included when the
+  VFIO_IRQ_SET_DATA_BOOL flag is present. It contains an array of booleans
+  that specify whether the action is to be performed on the corresponding
+  index. It's used when the action is only performed on a subset of the range
+  specified.
+
+Not all interrupt types support every combination of data and action flags.
+The client must know the capabilities of the device and IRQ index before it
+sends a VFIO_USER_DEVICE_SET_IRQ message.
+
+Read and Write Operations
+-------------------------
+
+Not all I/O operations between the client and server can be done via direct
+access of memory mapped with an mmap() call. In these cases, the client and
+server use messages sent over the socket. It is expected that these operations
+will have lower performance than direct access.
+
+The client can access device memory with VFIO_USER_REGION_READ and
+VFIO_USER_REGION_WRITE requests. These share a common data structure that
+appears after the 16 byte message header.
+
+REGION Read/Write Data
+^^^^^^^^^^^^^^^^^^^^^^
+
++--------+--------+----------+
+| Name   | Offset | Size     |
++========+========+==========+
+| Offset | 16     | 8        |
++--------+--------+----------+
+| Region | 24     | 4        |
++--------+--------+----------+
+| Count  | 28     | 4        |
++--------+--------+----------+
+| Data   | 32     | variable |
++--------+--------+----------+
+
+* Offset into the region being accessed.
+* Region is the index of the region being accessed.
+* Count is the size of the data to be transferred.
+* Data is the data to be read or written.
+
+The server can access guest memory with VFIO_USER_DMA_READ and
+VFIO_USER_DMA_WRITE messages. These also share a common data structure that
+appears after the 16 byte message header.
+
+DMA Read/Write Data
+^^^^^^^^^^^^^^^^^^^
+
++---------+--------+----------+
+| Name    | Offset | Size     |
++=========+========+==========+
+| Address | 16     | 8        |
++---------+--------+----------+
+| Count   | 24     | 4        |
++---------+--------+----------+
+| Data    | 28     | variable |
++---------+--------+----------+
+
+* Address is the area of guest memory being accessed. This address must have
+  been exported to the server with a VFIO_USER_DMA_MAP message.
+* Count is the size of the data to be transferred.
+* Data is the data to be read or written.
+
+Address and count can also be accessed as ``struct iovec`` from ``<sys/uio.h>``.
+
+VFIO_USER_REGION_READ
+---------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Device ID    | <ID>                   |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 8                      |
++--------------+------------------------+
+| Message size | 32 + data size         |
++--------------+------------------------+
+| Flags Reply  | bit set in reply       |
++--------------+------------------------+
+| Read info    | REGION read/write data |
++--------------+------------------------+
+
+This request is sent from the client to the server to read from device memory.
+In the request messages, there will be no data, and the count field will be the
+amount of data to be read. The reply will include the data read, and its count
+field will be the amount of data read.
+
+VFIO_USER_REGION_WRITE
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Device ID    | <ID>                   |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 9                      |
++--------------+------------------------+
+| Message size | 32 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Write info   | REGION read write data |
++--------------+------------------------+
+
+This request is sent from the client to the server to write to device memory.
+The request message will contain the data to be written, and its count field
+will contain the amount of write data. The count field in the reply will be
+zero.
+
+VFIO_USER_DMA_READ
+------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+---------------------+
+| Name         | Value               |
++==============+=====================+
+| Device ID    | <ID>                |
++--------------+---------------------+
+| Message ID   | <ID>                |
++--------------+---------------------+
+| Command      | 10                  |
++--------------+---------------------+
+| Message size | 28 + data size      |
++--------------+---------------------+
+| Flags Reply  | bit set in reply    |
++--------------+---------------------+
+| DMA info     | DMA read/write data |
++--------------+---------------------+
+
+This request is sent from the server to the client to read from guest memory.
+In the request messages, there will be no data, and the count field will be the
+amount of data to be read. The reply will include the data read, and its count
+field will be the amount of data read.
+
+VFIO_USER_DMA_WRITE
+-------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Device ID    | <ID>                   |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 11                     |
++--------------+------------------------+
+| Message size | 28 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| DMA info     | DMA read/write data    |
++--------------+------------------------+
+
+This request is sent from the server to the client to write to guest memory.
+The request message will contain the data to be written, and its count field
+will contain the amount of write data. The count field in the reply will be
+zero.
+
+VFIO_USER_VM_INTERRUPT
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++----------------+------------------------+
+| Name           | Value                  |
++================+========================+
+| Device ID      | <ID>                   |
++----------------+------------------------+
+| Message ID     | <ID>                   |
++----------------+------------------------+
+| Command        | 12                     |
++----------------+------------------------+
+| Message size   | 24                     |
++----------------+------------------------+
+| Flags          | Reply bit set in reply |
++----------------+------------------------+
+| Interrupt info | <interrupt>            |
++----------------+------------------------+
+
+This request is sent from the server to the client to signal the device has
+raised an interrupt.
+
+Interrupt info format
+^^^^^^^^^^^^^^^^^^^^^
+
++----------+--------+------+
+| Name     | Offset | Size |
++==========+========+======+
+| Index    | 16     | 4    |
++----------+--------+------+
+| Subindex | 20     | 4    |
++----------+--------+------+
+
+* Index is the interrupt index; it is the same value used in VFIO_USER_SET_IRQS.
+* Subindex is relative to the index, e.g., the vector number used in PCI MSI/X
+  type interrupts.
+
+VFIO_USER_DEVICE_RESET
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Device ID    | <ID>                   |
++--------------+------------------------+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 13                     |
++--------------+------------------------+
+| Message size | 16                     |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+
+This request is sent from the client to the server to reset the device.
+
+Appendices
+==========
+
+Unused VFIO ioctl() commands
+----------------------------
+
+The following commands must be handled by the client and not sent to the server:
+
+* VFIO_GET_API_VERSION
+* VFIO_CHECK_EXTENSION
+* VFIO_SET_IOMMU
+* VFIO_GROUP_GET_STATUS
+* VFIO_GROUP_SET_CONTAINER
+* VFIO_GROUP_UNSET_CONTAINER
+* VFIO_GROUP_GET_DEVICE_FD
+* VFIO_IOMMU_GET_INFO
+
+However, once support for live migration for VFIO devices is finalized some
+of the above commands might have to be handled by the client. This will be
+addressed in a future protocol version.
+
+Live Migration
+--------------
+Currently live migration is not supported for devices passed through via VFIO,
+therefore it is not supported for VFIO-over-socket, either. This is being
+actively worked on in the "Add migration support for VFIO devices" (v25) patch
+series.
+
+VFIO groups and containers
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current VFIO implementation includes group and container idioms that
+describe how a device relates to the host IOMMU. In the VFIO over socket
+implementation, the IOMMU is implemented in SW by the client, and isn't visible
+to the server. The simplest idea is for the client is to put each device into
+its own group and container.
-- 
2.5.5

Hi Thanos,

I had a quick look on the spec. Leaving some comments inline.

On 17/7/20 2:20 μ.μ., Thanos Makatos wrote:

> This patch introduces the VFIO-over-socket protocol specification, which
> is designed to allow devices to be emulated outside QEMU, in a separate
> process. VFIO-over-socket reuses the existing VFIO defines, structs and
> concepts.
>
> It has been earlier discussed as an RFC in:
> "RFC: use VFIO over a UNIX domain socket to implement device offloading"
>
> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
>
> ---
>
> Changed since v1:
>    * fix coding style issues
>    * update MAINTAINERS for VFIO-over-socket
>    * add vfio-over-socket to ToC
>
> Changed since v2:
>    * fix whitespace
>
> Regarding the build failure, I have not been able to reproduce it locally
> using the docker image on my Debian 10.4 machine.
> ---
>   MAINTAINERS                     |    6 +
>   docs/devel/index.rst            |    1 +
>   docs/devel/vfio-over-socket.rst | 1135 +++++++++++++++++++++++++++++++++++++++
>   3 files changed, 1142 insertions(+)
>   create mode 100644 docs/devel/vfio-over-socket.rst
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 030faf0..bb81590 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1732,6 +1732,12 @@ F: hw/vfio/ap.c
>   F: docs/system/s390x/vfio-ap.rst
>   L: qemu-s390x@nongnu.org
>   
> +VFIO-over-socket
> +M: John G Johnson <john.g.johnson@oracle.com>
> +M: Thanos Makatos <thanos.makatos@nutanix.com>
> +S: Supported
> +F: docs/devel/vfio-over-socket.rst
> +
>   vhost
>   M: Michael S. Tsirkin <mst@redhat.com>
>   S: Supported
> diff --git a/docs/devel/index.rst b/docs/devel/index.rst
> index ae6eac7..0439460 100644
> --- a/docs/devel/index.rst
> +++ b/docs/devel/index.rst
> @@ -30,3 +30,4 @@ Contents:
>      reset
>      s390-dasd-ipl
>      clocks
> +   vfio-over-socket
> diff --git a/docs/devel/vfio-over-socket.rst b/docs/devel/vfio-over-socket.rst
> new file mode 100644
> index 0000000..b474f23
> --- /dev/null
> +++ b/docs/devel/vfio-over-socket.rst
> @@ -0,0 +1,1135 @@
> +***************************************
> +VFIO-over-socket Protocol Specification
> +***************************************
> +
> +Version 0.1
> +
> +Introduction
> +============
> +VFIO-over-socket, also known as vfio-user, is a protocol that allows a device

I think there is no point in having two names for the same protocol,
"vfio-over-socket" and "vfio-user".

> +to be virtualized in a separate process outside of QEMU. VFIO-over-socket
> +devices consist of a generic VFIO device type, living inside QEMU, which we
> +call the client, and the core device implementation, living outside QEMU, which
> +we call the server. VFIO-over-socket can be the main transport mechanism for
> +multi-process QEMU, however it can be used by other applications offering
> +device virtualization. Explaining the advantages of a
> +disaggregated/multi-process QEMU, and device virtualization outside QEMU in
> +general, is beyond the scope of this document.
> +
> +This document focuses on specifying the VFIO-over-socket protocol. VFIO has
> +been chosen for the following reasons:
> +
> +1) It is a mature and stable API, backed by an extensively used framework.
> +2) The existing VFIO client implementation (qemu/hw/vfio/) can be largely
> +   reused.
> +
> +In a proof of concept implementation it has been demonstrated that using VFIO
> +over a UNIX domain socket is a viable option. VFIO-over-socket is designed with
> +QEMU in mind, however it could be used by other client applications. The
> +VFIO-over-socket protocol does not require that QEMU's VFIO client
> +implementation is used in QEMU. None of the VFIO kernel modules are required
> +for supporting the protocol, neither in the client nor the server, only the
> +source header files are used.
> +
> +The main idea is to allow a virtual device to function in a separate process in
> +the same host over a UNIX domain socket. A UNIX domain socket (AF_UNIX) is
> +chosen because we can trivially send file descriptors over it, which in turn
> +allows:
> +
> +* Sharing of guest memory for DMA with the virtual device process.
> +* Sharing of virtual device memory with the guest for fast MMIO.
> +* Efficient sharing of eventfd's for triggering interrupts.
> +
> +However, other socket types could be used which allows the virtual device
> +process to run in a separate guest in the same host (AF_VSOCK) or remotely
> +(AF_INET). Theoretically the underlying transport doesn't necessarily have to
> +be a socket, however we don't examine such alternatives. In this document we
> +focus on using a UNIX domain socket and introduce basic support for the other
> +two types of sockets without considering performance implications.
> +
> +This document does not yet describe any internal details of the server-side
> +implementation, however QEMU's VFIO client implementation will have to be
> +adapted according to this protocol in order to support VFIO-over-socket virtual
> +devices.
> +
> +VFIO
> +====
> +VFIO is a framework that allows a physical device to be securely passed through
> +to a user space process; the kernel does not drive the device at all.

I would remove the last part: "the kernel does not drive the device at
all". Isn't that quite inaccurate? The kernel does drive the device with
the vfio driver. The user space driver needs the vfio driver in order to
do certain things like, for example, write on a port or receive
notifications for device interrupts.

> +Typically, the user space process is a VM and the device is passed through to
> +it in order to achieve high performance. VFIO provides an API and the required
> +functionality in the kernel. QEMU has adopted VFIO to allow a guest virtual
> +machine to directly access physical devices, instead of emulating them in

Maybe s/guest virtual machine/guest ?

> +software

Missing dot here.

> +
> +VFIO-over-socket reuses the core VFIO concepts defined in its API, but
> +implements them as messages to be sent over a UNIX-domain socket. It does not

s/UNIX-domain/UNIX domain (just to have the same name everywhere)

> +change the kernel-based VFIO in any way, in fact none of the VFIO kernel
> +modules need to be loaded to use VFIO-over-socket. It is also possible for QEMU
> +to concurrently use the current kernel-based VFIO for one guest device, and use
> +VFIO-over-socket for another device in the same guest.
> +
> +VFIO Device Model
> +-----------------
> +A device under VFIO presents a standard VFIO model to the user process. Many
> +of the VFIO operations in the existing kernel model use the ioctl() system
> +call, and references to the existing model are called the ioctl()
> +implementation in this document.
> +
> +The following sections describe the set of messages that implement the VFIO
> +device model over a UNIX domain socket. In many cases, the messages are direct
> +translations of data structures used in the ioctl() implementation. Messages
> +derived from ioctl()s will have a name derived from the ioctl() command name.
> +E.g., the VFIO_GET_INFO ioctl() command becomes a VFIO_USER_GET_INFO message.
> +The purpose for this reuse is to share as much code as feasible with the

s/for/of

> +ioctl() implementation.
> +
> +Client and Server
> +^^^^^^^^^^^^^^^^^
> +The socket connects two processes together: a client process and a server
> +process. In the context of this document, the client process is the process
> +emulating a guest virtual machine, such as QEMU. The server process is a
> +process that provides device emulation.
> +
> +Connection Initiation
> +^^^^^^^^^^^^^^^^^^^^^
> +After the client connects to the server, the initial server message is
> +VFIO_USER_VERSION to propose a protocol version and set of capabilities to
> +apply to the session. The client replies with a compatible version and set of
> +capabilities it will support, or closes the connection if it cannot support the
> +advertised version.
> +
> +Guest Memory Configuration
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +The client uses VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP messages to inform
> +the server of the valid guest DMA ranges that the server can access on behalf
> +of a device. Guest memory may be accessed by the server via VFIO_USER_DMA_READ
> +and VFIO_USER_DMA_WRITE messages over the socket.
> +
> +An optimization for server access to guest memory is for the client to provide
> +file descriptors the server can mmap() to directly access guest memory. Note
> +that mmap() privileges cannot be revoked by the client, therefore file
> +descriptors should only be exported in environments where the client trusts the
> +server not to corrupt guest memory.
> +
> +Device Information
> +^^^^^^^^^^^^^^^^^^
> +The client uses a VFIO_USER_DEVICE_GET_INFO message to query the server for
> +information about the device. This information includes:
> +
> +* The device type and capabilities,
> +* the number of memory regions, and
> +* the device presents to the guest the number of interrupt types the device
> +  supports.
> +
> +Region Information
> +^^^^^^^^^^^^^^^^^^
> +The client uses VFIO_USER_DEVICE_GET_REGION_INFO messages to query the server
> +for information about the device's memory regions. This information describes:
> +
> +* Read and write permissions, whether it can be memory mapped, and whether it
> +  supports additional capabilities.
> +* Region index, size, and offset.
> +
> +When a region can be mapped by the client, the server provides a file
> +descriptor which the client can mmap(). The server is responsible for polling
> +for client updates to memory mapped regions.
> +
> +Region Capabilities
> +"""""""""""""""""""
> +Some regions have additional capabilities that cannot be described adequately
> +by the region info data structure. These capabilities are returned in the
> +region info reply in a list similar to PCI capabilities in a PCI device's
> +configuration space.
> +
> +Sparse Regions
> +""""""""""""""
> +A region can be memory-mappable in whole or in part. When only a subset of a
> +region can be mapped by the client, a VFIO_REGION_INFO_CAP_SPARSE_MMAP
> +capability is included in the region info reply. This capability describes
> +which portions can be mapped by the client.
> +
> +For example, in a virtual NVMe controller, sparse regions can be used so that
> +accesses to the NVMe registers (found in the beginning of BAR0) are trapped (an
> +infrequent an event), while allowing direct access to the doorbells (an

s/an event/event

> +extremely frequent event as every I/O submission requires a write to BAR0),
> +found right after the NVMe registers in BAR0.
> +
> +Interrupts
> +^^^^^^^^^^
> +The client uses VFIO_USER_DEVICE_GET_IRQ_INFO messages to query the server for
> +the device's interrupt types. The interrupt types are specific to the bus the
> +device is attached to, and the client is expected to know the capabilities of
> +each interrupt type. The server can signal an interrupt either with
> +VFIO_USER_VM_INTERRUPT messages over the socket, or can directly inject
> +interrupts into the guest via an event file descriptor. The client configures
> +how the server signals an interrupt with VFIO_USER_SET_IRQS messages.
> +
> +Device Read and Write
> +^^^^^^^^^^^^^^^^^^^^^
> +When the guest executes load or store operations to device memory, the client
> +forwards these operations to the server with VFIO_USER_REGION_READ or
> +VFIO_USER_REGION_WRITE messages. The server will reply with data from the
> +device on read operations or an acknowledgement on write operations.
> +
> +DMA
> +^^^
> +When a device performs DMA accesses to guest memory, the server will forward
> +them to the client with VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages.
> +These messages can only be used to access guest memory the client has
> +configured into the server.
> +
> +Protocol Specification
> +======================
> +To distinguish from the base VFIO symbols, all VFIO-over-socket symbols are
> +prefixed with vfio_user or VFIO_USER. In revision 0.1, all data is in the
> +little-endian format, although this may be relaxed in future revision in cases
> +where the client and server are both big-endian. The messages are formatted
> +for seamless reuse of the native VFIO structs. A server can serve:
> +
> +1) multiple clients, and/or
> +2) multiple virtual devices, belonging to one or more clients.
> +
> +Therefore each message requires a header that uniquely identifies the virtual
> +device. It is a server-side implementation detail whether a single server
> +handles multiple virtual devices from the same or multiple guests.
> +
> +Socket
> +------
> +A single UNIX domain socket is assumed to be used for each device. The location

Is it correct for a spec to assume things?

> +of the socket is implementation-specific. Multiplexing clients, devices, and
> +servers over the same socket is not supported in this version of the protocol,
> +but a device ID field exists in the message header so that a future support can
> +be added without a major version change.
> +
> +Authentication
> +--------------
> +For AF_UNIX, we rely on OS mandatory access controls on the socket files,
> +therefore it is up to the management layer to set up the socket as required.
> +Socket types than span guests or hosts will require a proper authentication
> +mechanism. Defining that mechanism is deferred to a future version of the
> +protocol.
> +
> +Request Concurrency
> +-------------------
> +There can be multiple outstanding requests per virtual device, e.g. a
> +frame buffer where the guest does multiple stores to the virtual device. The
> +server can execute and reorder non-conflicting requests in parallel, depending
> +on the device semantics.
> +
> +Socket Disconnection Behavior
> +-----------------------------
> +The server and the client can disconnect from each other, either intentionally
> +or unexpectedly. Both the client and the server need to know how to handle such
> +events.
> +
> +Server Disconnection
> +^^^^^^^^^^^^^^^^^^^^
> +A server disconnecting from the client may indicate that:
> +
> +1) A virtual device has been restarted, either intentionally (e.g. because of a
> +device update) or unintentionally (e.g. because of a crash). In any case, the
> +virtual device will come back so the client should not do anything (e.g. simply
> +reconnect and retry failed operations).
> +

Indentation issue ^^ (also remove the space, there are no spaces between the
elements in the other numbered lists).

> +2) A virtual device has been shut down with no intention to be restarted.
> +
> +It is impossible for the client to know whether or not a failure is
> +intermittent or innocuous and should be retried, therefore the client should
> +attempt to reconnect to the socket. Since an intentional server restart (e.g.
> +due to an upgrade) might take some time, a reasonable timeout should be used.
> +In cases where the disconnection is expected (e.g. the guest shutting down), no
> +new requests will be sent anyway so this situation doesn't pose a problem. The
> +control stack will clean up accordingly.
> +
> +Parametrizing this behaviour by having the virtual device advertise a
> +reasonable reconnect is deferred to a future version of the protocol.
> +
> +Client Disconnection
> +^^^^^^^^^^^^^^^^^^^^
> +The client disconnecting from the server primarily means that the QEMU process
> +has exited. Currently this means that the guest is shut down so the device is
> +no longer needed therefore the server can automatically exit. However, there
> +can be cases where a client disconnect should not result in a server exit:

s/disconnect/disconnection

> +
> +1) A single server serving multiple clients.
> +2) A multi-process QEMU upgrading itself step by step, which isn't yet
> +   implemented.
> +
> +Therefore in order for the protocol to be forward compatible the server should
> +take no action when the client disconnects. If anything happens to the client
> +process the control stack will know about it and can clean up resources
> +accordingly.
> +
> +Request Retry and Response Timeout
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +QEMU's VFIO retries certain operations if they fail. While this makes sense for
> +real HW, we don't know for sure whether it makes sense for virtual devices. A
> +failed request is a request that has been successfully sent and has been
> +responded to with an error code. Failure to send the request in the first place
> +(e.g. because the socket is disconnected) is a different type of error examined
> +earlier in the disconnect section.
> +
> +Defining a retry and timeout scheme if deferred to a future version of the

s/if/is

> +protocol.
> +
> +Commands
> +--------
> +The following table lists the VFIO message command IDs, and whether the
> +message request is sent from the client or the server.
> +
> ++----------------------------------+---------+-------------------+
> +| Name                             | Command | Request Direction |
> ++==================================+=========+===================+
> +| VFIO_USER_VERSION                | 1       | server → client   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_MAP                | 2       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_UNMAP              | 3       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_GET_INFO        | 4       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_GET_REGION_INFO | 5       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_GET_IRQ_INFO    | 6       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_SET_IRQS        | 7       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_REGION_READ            | 8       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_REGION_WRITE           | 9       | client → server   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_READ               | 10      | server → client   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_READ               | 11      | server → client   |

Isn't that VFIO_USER_DMA_WRITE?

> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_VM_INTERRUPT           | 12      | server → client   |
> ++----------------------------------+---------+-------------------+
> +| VFIO_DEVICE_RESET                | 13      | client → server   |
> ++----------------------------------+---------+-------------------+
> +
> +Header
> +------
> +All messages are preceded by a 16 byte header that contains basic information
> +about the message. The header is followed by message-specific data described
> +in the sections below.
> +
> ++----------------+--------+-------------+
> +| Name           | Offset | Size        |
> ++================+========+=============+
> +| Device ID      | 0      | 2           |
> ++----------------+--------+-------------+
> +| Message ID     | 2      | 2           |
> ++----------------+--------+-------------+
> +| Command        | 4      | 4           |
> ++----------------+--------+-------------+
> +| Message size   | 8      | 4           |
> ++----------------+--------+-------------+
> +| Flags          | 12     | 4           |
> ++----------------+--------+-------------+
> +|                | +-----+------------+ |
> +|                | | Bit | Definition | |
> +|                | +=====+============+ |
> +|                | | 0   | Reply      | |
> +|                | +-----+------------+ |
> +|                | | 1   | No_reply   | |
> +|                | +-----+------------+ |
> ++----------------+--------+-------------+
> +| <message data> | 16     | variable    |
> ++----------------+--------+-------------+
> +
> +* Device ID identifies the destination device of the message. This field is
> +  reserved when the server only supports one device per socket.
> +* Message ID identifies the message, and is used in the message acknowledgement.
> +* Command specifies the command to be executed, listed in the Command Table.
> +* Message size contains the size of the entire message, including the header.
> +* Flags contains attributes of the message:
> +
> +  * The reply bit differentiates request messages from reply messages. A reply
> +    message acknowledges a previous request with the same message ID.
> +  * No_reply indicates that no reply is needed for this request. This is
> +    commonly used when multiple requests are sent, and only the last needs
> +    acknowledgement.
> +
> +VFIO_USER_VERSION
> +-----------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Device ID    | 0                      |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 1                      |
> ++--------------+------------------------+
> +| Message size | 16 + version length    |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Version      | JSON byte array        |
> ++--------------+------------------------+
> +
> +This is the initial message sent by the server after the socket connection is
> +established. The version is in JSON format, and the following objects must be
> +included:
> +
> ++--------------+--------+---------------------------------------------------+
> +| Name         | Type   | Description                                       |
> ++==============+========+===================================================+
> +| version      | object | {“major”: <number>, “minor”: <number>}            |
> +|              |        | Version supported by the sender, e.g. “0.1”.      |
> ++--------------+--------+---------------------------------------------------+
> +| type         | string | Fixed to “vfio-user”.                             |
> ++--------------+--------+---------------------------------------------------+
> +| capabilities | array  | Reserved. Can be omitted for v0.1, otherwise must |
> +|              |        | be empty.                                         |
> ++--------------+--------+---------------------------------------------------+
> +
> +Versioning and Feature Support
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +Upon accepting a connection, the server must send a VFIO_USER_VERSION message
> +proposing a protocol version and a set of capabilities. The client compares
> +these with the versions and capabilities it supports and sends a
> +VFIO_USER_VERSION reply according to the following rules.
> +
> +* The major version in the reply must be the same as proposed. If the client
> +  does not support the proposed major, it closes the connection.
> +* The minor version in the reply must be equal to or less than the minor
> +  version proposed.
> +* The capability list must be a subset of those proposed. If the client
> +  requires a capability the server did not include, it closes the connection.
> +* If type is not “vfio-user”, the client closes the connection.
> +
> +The protocol major version will only change when incompatible protocol changes
> +are made, such as changing the message format. The minor version may change
> +when compatible changes are made, such as adding new messages or capabilities,
> +Both the client and server must support all minor versions less than the
> +maximum minor version it supports. E.g., an implementation that supports
> +version 1.3 must also support 1.0 through 1.2.
> +
> +VFIO_USER_DMA_MAP
> +-----------------
> +
> +VFIO_USER_DMA_UNMAP
> +-------------------
> +
> +Message Format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Device ID    | 0                      |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | MAP=2, UNMAP=3         |
> ++--------------+------------------------+
> +| Message size | 16 + table size        |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Table        | array of table entries |
> ++--------------+------------------------+
> +
> +This message is sent by the client to the server to inform it of the guest
> +memory regions the device can access. It must be sent before the device can
> +perform any DMA to the guest. It is normally sent directly after the version
> +handshake is completed, but may also occur when memory is added or subtracted
> +in the guest.
> +
> +The table is an array of the following structure. This structure is 32 bytes
> +in size, so the message size will be 16 + (# of table entries * 32). If a
> +region being added can be directly mapped by the server, an array of file
> +descriptors will be sent as part of the message meta-data. Each region entry
> +will have a corresponding file descriptor. On AF_UNIX sockets, the file
> +descriptors will be passed as SCM_RIGHTS type ancillary data.
> +
> +Table entry format
> +^^^^^^^^^^^^^^^^^^
> +
> ++-------------+--------+-------------+
> +| Name        | Offset | Size        |
> ++=============+========+=============+
> +| Address     | 0      | 8           |
> ++-------------+--------+-------------+
> +| Size        | 8      | 8           |
> ++-------------+--------+-------------+
> +| Offset      | 16     | 8           |
> ++-------------+--------+-------------+
> +| Protections | 24     | 4           |
> ++-------------+--------+-------------+
> +| Flags       | 28     | 4           |
> ++-------------+--------+-------------+
> +|             | +-----+------------+ |
> +|             | | Bit | Definition | |
> +|             | +=====+============+ |
> +|             | | 0   | Mappable   | |
> +|             | +-----+------------+ |
> ++-------------+--------+-------------+
> +
> +* Address is the base DMA address of the region.
> +* Size is the size of the region.
> +* Offset is the file offset of the region with respect to the associated file
> +  descriptor.
> +* Protections are the region's protection attributes as encoded in
> +  ``<sys/mman.h>``.
> +* Flags contain the following region attributes:
> +
> +  * Mappable indicate the region can be mapped via the mmap() system call using
> +    the file descriptor provided in the message meta-data.
> +
> +VFIO_USER_DEVICE_GET_INFO
> +-------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+----------------------------+
> +| Name         | Value                      |
> ++==============+============================+
> +| Device ID    | <ID>                       |
> ++--------------+----------------------------+
> +| Message ID   | <ID>                       |
> ++--------------+----------------------------+
> +| Command      | 4                          |
> ++--------------+----------------------------+
> +| Message size | 16 in request, 32 in reply |
> ++--------------+----------------------------+
> +| Flags        | Reply bit set in reply     |
> ++--------------+----------------------------+
> +| Device info  | VFIO device info           |
> ++--------------+----------------------------+
> +
> +This message is sent by the client to the server to query for basic information
> +about the device. Only the message header is needed in the request message.
> +The VFIO device info structure is defined in ``<sys/vfio.h>`` (``struct
> +vfio_device_info``).
> +
> +VFIO device info format
> +^^^^^^^^^^^^^^^^^^^^^^^
> +
> ++-------------+--------+--------------------------+
> +| Name        | Offset | Size                     |
> ++=============+========+==========================+
> +| argsz       | 16     | 4                        |
> ++-------------+--------+--------------------------+
> +| flags       | 20     | 4                        |
> ++-------------+--------+--------------------------+
> +|             | +-----+-------------------------+ |
> +|             | | Bit | Definition              | |
> +|             | +=====+=========================+ |
> +|             | | 0   | VFIO_DEVICE_FLAGS_RESET | |
> +|             | +-----+-------------------------+ |
> +|             | | 1   | VFIO_DEVICE_FLAGS_PCI   | |
> +|             | +-----+-------------------------+ |
> ++-------------+--------+--------------------------+
> +| num_regions | 24     | 4                        |
> ++-------------+--------+--------------------------+
> +| num_irqs    | 28     | 4                        |
> ++-------------+--------+--------------------------+
> +
> +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> +  implementation.
> +* flags contains the following device attributes.
> +
> +  * VFIO_DEVICE_FLAGS_RESET indicates the device supports the
> +    VFIO_USER_DEVICE_RESET message.
> +  * VFIO_DEVICE_FLAGS_PCI indicates the device is a PCI device.
> +
> +* num_regions is the number of memory regions the device exposes.
> +* num_irqs is the number of distinct interrupt types the device supports.
> +
> +This version of the protocol only supports PCI devices. Additional devices may
> +be supported in future versions.
> +
> +VFIO_USER_DEVICE_GET_REGION_INFO
> +--------------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------+
> +| Name         | Value            |
> ++==============+==================+
> +| Device ID    | <ID>             |
> ++--------------+------------------+
> +| Message ID   | <ID>             |
> ++--------------+------------------+
> +| Command      | 5                |
> ++--------------+------------------+
> +| Message size | 48 + any caps    |
> ++--------------+------------------+
> +| Flags Reply  | bit set in reply |
> ++--------------+------------------+
> +| Region info  | VFIO region info |
> ++--------------+------------------+
> +
> +This message is sent by the client to the server to query for information about
> +device memory regions. The VFIO region info structure is defined in
> +``<sys/vfio.h>`` (``struct vfio_region_info``).
> +
> +VFIO region info format
> +^^^^^^^^^^^^^^^^^^^^^^^
> +
> ++------------+--------+------------------------------+
> +| Name       | Offset | Size                         |
> ++============+========+==============================+
> +| argsz      | 16     | 4                            |
> ++------------+--------+------------------------------+
> +| flags      | 20     | 4                            |
> ++------------+--------+------------------------------+
> +|            | +-----+-----------------------------+ |
> +|            | | Bit | Definition                  | |
> +|            | +=====+=============================+ |
> +|            | | 0   | VFIO_REGION_INFO_FLAG_READ  | |
> +|            | +-----+-----------------------------+ |
> +|            | | 1   | VFIO_REGION_INFO_FLAG_WRITE | |
> +|            | +-----+-----------------------------+ |
> +|            | | 2   | VFIO_REGION_INFO_FLAG_MMAP  | |
> +|            | +-----+-----------------------------+ |
> +|            | | 3   | VFIO_REGION_INFO_FLAG_CAPS  | |
> +|            | +-----+-----------------------------+ |
> ++------------+--------+------------------------------+
> +| index      | 24     | 4                            |
> ++------------+--------+------------------------------+
> +| cap_offset | 28     | 4                            |
> ++------------+--------+------------------------------+
> +| size       | 32     | 8                            |
> ++------------+--------+------------------------------+
> +| offset     | 40     | 8                            |
> ++------------+--------+------------------------------+
> +
> +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> +  implementation.
> +* flags are attributes of the region:
> +
> +  * VFIO_REGION_INFO_FLAG_READ allows client read access to the region.
> +  * VFIO_REGION_INFO_FLAG_WRITE allows client write access region.

s/region/to the region

> +  * VFIO_REGION_INFO_FLAG_MMAP specifies the client can mmap() the region. When
> +    this flag is set, the reply will include a file descriptor in its meta-data.
> +    On AF_UNIX sockets, the file descriptors will be passed as SCM_RIGHTS type
> +    ancillary data.
> +  * VFIO_REGION_INFO_FLAG_CAPS indicates additional capabilities found in the
> +    reply.
> +
> +* index is the index of memory region being queried, it is the only field that
> +  is required to be set in the request message.
> +* cap_offset describes where additional region capabilities can be found.
> +  cap_offset is relative to the beginning of the VFIO region info structure.
> +  The data structure it points is a VFIO cap header defined in ``<sys/vfio.h>``.
> +* size is the size of the region.
> +* offset is the offset given to the mmap() system call for regions with the
> +  MMAP attribute. It is also used as the base offset when mapping a VFIO
> +  sparse mmap area, described below.
> +
> +VFIO Region capabilities
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +The VFIO region information can also include a capabilities list. This list is
> +similar to a PCI capability list - each entry has a common header that
> +identifies a capability and where the next capability in the list can be found.
> +The VFIO capability header format is defined in ``<sys/vfio.h>`` (``struct
> +vfio_info_cap_header``).
> +
> +VFIO cap header format
> +^^^^^^^^^^^^^^^^^^^^^^
> +
> ++---------+--------+------+
> +| Name    | Offset | Size |
> ++=========+========+======+
> +| id      | 0      | 2    |
> ++---------+--------+------+
> +| version | 2      | 2    |
> ++---------+--------+------+
> +| next    | 4      | 4    |
> ++---------+--------+------+
> +
> +* id is the capability identity.
> +* version is a capability-specific version number.
> +* next specifies the offset of the next capability in the capability list. It
> +  is relative to the beginning of the VFIO region info structure.
> +
> +VFIO sparse mmap
> +^^^^^^^^^^^^^^^^
> +
> ++------------------+----------------------------------+
> +| Name             | Value                            |
> ++==================+==================================+
> +| id               | VFIO_REGION_INFO_CAP_SPARSE_MMAP |
> ++------------------+----------------------------------+
> +| version          | 0x1                              |
> ++------------------+----------------------------------+
> +| next             | <next>                           |
> ++------------------+----------------------------------+
> +| sparse mmap info | VFIO region info sparse mmap     |
> ++------------------+----------------------------------+
> +
> +The only capability supported in this version of the protocol is for sparse
> +mmap. This capability is defined when only a subrange of the region supports
> +direct access by the client via mmap(). The VFIO sparse mmap area is defined in
> +``<sys/vfio.h>`` (``struct vfio_region_sparse_mmap_area``).
> +
> +VFIO region info cap sparse mmap
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ++----------+--------+------+
> +| Name     | Offset | Size |
> ++==========+========+======+
> +| nr_areas | 0      | 4    |
> ++----------+--------+------+
> +| reserved | 4      | 4    |
> ++----------+--------+------+
> +| offset   | 8      | 8    |
> ++----------+--------+------+
> +| size     | 16     | 9    |
> ++----------+--------+------+
> +| ...      |        |      |
> ++----------+--------+------+
> +
> +* nr_areas is the number of sparse mmap areas in the region.
> +* offset and size describe a single area that can be mapped by the client.
> +  There will be nr_areas pairs of offset and size. The offset will be added to
> +  the base offset given in the VFIO_USER_DEVICE_GET_REGION_INFO to form the
> +  offset argument of the subsequent mmap() call.
> +
> +The VFIO sparse mmap area is defined in ``<sys/vfio.h>`` (``struct
> +vfio_region_info_cap_sparse_mmap``).
> +
> +VFIO_USER_DEVICE_GET_IRQ_INFO
> +-----------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Device ID    | <ID>                   |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 6                      |
> ++--------------+------------------------+
> +| Message size | 32                     |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| IRQ info     | VFIO IRQ info          |
> ++--------------+------------------------+
> +
> +This message is sent by the client to the server to query for information about
> +device interrupt types. The VFIO IRQ info structure is defined in
> +``<sys/vfio.h>`` (``struct vfio_irq_info``).
> +
> +VFIO IRQ info format
> +^^^^^^^^^^^^^^^^^^^^
> +
> ++-------+--------+---------------------------+
> +| Name  | Offset | Size                      |
> ++=======+========+===========================+
> +| argsz | 16     | 4                         |
> ++-------+--------+---------------------------+
> +| flags | 20     | 4                         |
> ++-------+--------+---------------------------+
> +|       | +-----+--------------------------+ |
> +|       | | Bit | Definition               | |
> +|       | +=====+==========================+ |
> +|       | | 0   | VFIO_IRQ_INFO_EVENTFD    | |
> +|       | +-----+--------------------------+ |
> +|       | | 1   | VFIO_IRQ_INFO_MASKABLE   | |
> +|       | +-----+--------------------------+ |
> +|       | | 2   | VFIO_IRQ_INFO_AUTOMASKED | |
> +|       | +-----+--------------------------+ |
> +|       | | 3   | VFIO_IRQ_INFO_NORESIZE   | |
> +|       | +-----+--------------------------+ |
> ++-------+--------+---------------------------+
> +| index | 24     | 4                         |
> ++-------+--------+---------------------------+
> +| count | 28     | 4                         |
> ++-------+--------+---------------------------+
> +
> +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> +  implementation.
> +* flags defines IRQ attributes:
> +
> +  * VFIO_IRQ_INFO_EVENTFD indicates the IRQ type can support server eventfd
> +    signalling.
> +  * VFIO_IRQ_INFO_MASKABLE indicates that the IRQ type supports the MASK and
> +    UNMASK actions in a VFIO_USER_DEVICE_SET_IRQS message.
> +  * VFIO_IRQ_INFO_AUTOMASKED indicates the IRQ type masks itself after being
> +    triggered, and the client must send an UNMASK action to receive new
> +    interrupts.
> +  * VFIO_IRQ_INFO_NORESIZE indicates VFIO_USER_SET_IRQS operations setup
> +    interrupts as a set, and new subindexes cannot be enabled without disabling
> +    the entire type.
> +
> +* index is the index of IRQ type being queried, it is the only field that is
> +  required to be set in the request message.
> +* count describes the number of interrupts of the queried type.
> +
> +VFIO_USER_DEVICE_SET_IRQS
> +-------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> +| Device ID    | <ID>                   |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 7                      |
> ++--------------+------------------------+
> +| Message size | 36 + any data          |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| IRQ set      | VFIO IRQ set           |
> ++--------------+------------------------+
> +
> +This message is sent by the client to the server to set actions for device
> +interrupt types. The VFIO IRQ set structure is defined in ``<sys/vfio.h>``
> +(``struct vfio_irq_set``).
> +
> +VFIO IRQ info format
> +^^^^^^^^^^^^^^^^^^^^
> +
> ++-------+--------+------------------------------+
> +| Name  | Offset | Size                         |
> ++=======+========+==============================+
> +| argsz | 6      | 4                            |
> ++-------+--------+------------------------------+
> +| flags | 20     | 4                            |
> ++-------+--------+------------------------------+
> +|       | +-----+-----------------------------+ |
> +|       | | Bit | Definition                  | |
> +|       | +=====+=============================+ |
> +|       | | 0   | VFIO_IRQ_SET_DATA_NONE      | |
> +|       | +-----+-----------------------------+ |
> +|       | | 1   | VFIO_IRQ_SET_DATA_BOOL      | |
> +|       | +-----+-----------------------------+ |
> +|       | | 2   | VFIO_IRQ_SET_DATA_EVENTFD   | |
> +|       | +-----+-----------------------------+ |
> +|       | | 3   | VFIO_IRQ_SET_ACTION_MASK    | |
> +|       | +-----+-----------------------------+ |
> +|       | | 4   | VFIO_IRQ_SET_ACTION_UNMASK  | |
> +|       | +-----+-----------------------------+ |
> +|       | | 5   | VFIO_IRQ_SET_ACTION_TRIGGER | |
> +|       | +-----+-----------------------------+ |
> ++-------+--------+------------------------------+
> +| index | 24     | 4                            |
> ++-------+--------+------------------------------+
> +| start | 28     | 4                            |
> ++-------+--------+------------------------------+
> +| count | 32     | 4                            |
> ++-------+--------+------------------------------+
> +| data  | 36     | variable                     |
> ++-------+--------+------------------------------+
> +
> +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> +  implementation.
> +* flags defines the action performed on the interrupt range. The DATA flags
> +  describe the data field sent in the message; the ACTION flags describe the
> +  action to be performed. The flags are mutually exclusive for both sets.
> +
> +  * VFIO_IRQ_SET_DATA_NONE indicates there is no data field in the request. The
> +    action is performed unconditionally.
> +  * VFIO_IRQ_SET_DATA_BOOL indicates the data field is an array of boolean
> +    bytes. The action is performed if the corresponding boolean is true.
> +  * VFIO_IRQ_SET_DATA_EVENTFD indicates an array of event file descriptors was
> +    sent in the message meta-data. These descriptors will be signalled when the
> +    action defined by the action flags occurs. In AF_UNIX sockets, the
> +    descriptors are sent as SCM_RIGHTS type ancillary data.
> +  * VFIO_IRQ_SET_ACTION_MASK indicates a masking event. It can be used with
> +    VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to mask an interrupt, or
> +    with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the guest masks
> +    the interrupt.
> +  * VFIO_IRQ_SET_ACTION_UNMASK indicates an unmasking event. It can be used
> +    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to unmask an
> +    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
> +    guest unmasks the interrupt.
> +  * VFIO_IRQ_SET_ACTION_TRIGGER indicates a triggering event. It can be used
> +    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to trigger an
> +    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
> +    guest triggers the interrupt.
> +
> +* index is the index of IRQ type being setup.
> +* start is the start of the subindex being set.
> +* count describes the number of sub-indexes being set. As a special case, a
> +  count of 0 with data flags of VFIO_IRQ_SET_DATA_NONE disables all interrupts
> +  of the index data is an optional field included when the
> +  VFIO_IRQ_SET_DATA_BOOL flag is present. It contains an array of booleans
> +  that specify whether the action is to be performed on the corresponding
> +  index. It's used when the action is only performed on a subset of the range
> +  specified.
> +
> +Not all interrupt types support every combination of data and action flags.
> +The client must know the capabilities of the device and IRQ index before it
> +sends a VFIO_USER_DEVICE_SET_IRQ message.
> +
> +Read and Write Operations
> +-------------------------
> +
> +Not all I/O operations between the client and server can be done via direct
> +access of memory mapped with an mmap() call. In these cases, the client and
> +server use messages sent over the socket. It is expected that these operations
> +will have lower performance than direct access.
> +
> +The client can access device memory with VFIO_USER_REGION_READ and
> +VFIO_USER_REGION_WRITE requests. These share a common data structure that
> +appears after the 16 byte message header.
> +
> +REGION Read/Write Data
> +^^^^^^^^^^^^^^^^^^^^^^
> +
> ++--------+--------+----------+
> +| Name   | Offset | Size     |
> ++========+========+==========+
> +| Offset | 16     | 8        |
> ++--------+--------+----------+
> +| Region | 24     | 4        |
> ++--------+--------+----------+
> +| Count  | 28     | 4        |
> ++--------+--------+----------+
> +| Data   | 32     | variable |
> ++--------+--------+----------+
> +
> +* Offset into the region being accessed.
> +* Region is the index of the region being accessed.
> +* Count is the size of the data to be transferred.
> +* Data is the data to be read or written.
> +
> +The server can access guest memory with VFIO_USER_DMA_READ and
> +VFIO_USER_DMA_WRITE messages. These also share a common data structure that
> +appears after the 16 byte message header.
> +
> +DMA Read/Write Data
> +^^^^^^^^^^^^^^^^^^^
> +
> ++---------+--------+----------+
> +| Name    | Offset | Size     |
> ++=========+========+==========+
> +| Address | 16     | 8        |
> ++---------+--------+----------+
> +| Count   | 24     | 4        |
> ++---------+--------+----------+
> +| Data    | 28     | variable |
> ++---------+--------+----------+
> +
> +* Address is the area of guest memory being accessed. This address must have
> +  been exported to the server with a VFIO_USER_DMA_MAP message.
> +* Count is the size of the data to be transferred.
> +* Data is the data to be read or written.
> +
> +Address and count can also be accessed as ``struct iovec`` from ``<sys/uio.h>``.
> +
> +VFIO_USER_REGION_READ
> +---------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Device ID    | <ID>                   |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 8                      |
> ++--------------+------------------------+
> +| Message size | 32 + data size         |
> ++--------------+------------------------+
> +| Flags Reply  | bit set in reply       |
> ++--------------+------------------------+
> +| Read info    | REGION read/write data |
> ++--------------+------------------------+
> +
> +This request is sent from the client to the server to read from device memory.
> +In the request messages, there will be no data, and the count field will be the
> +amount of data to be read. The reply will include the data read, and its count
> +field will be the amount of data read.
> +
> +VFIO_USER_REGION_WRITE
> +----------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Device ID    | <ID>                   |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 9                      |
> ++--------------+------------------------+
> +| Message size | 32 + data size         |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Write info   | REGION read write data |
> ++--------------+------------------------+
> +
> +This request is sent from the client to the server to write to device memory.
> +The request message will contain the data to be written, and its count field
> +will contain the amount of write data. The count field in the reply will be
> +zero.
> +
> +VFIO_USER_DMA_READ
> +------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+---------------------+
> +| Name         | Value               |
> ++==============+=====================+
> +| Device ID    | <ID>                |
> ++--------------+---------------------+
> +| Message ID   | <ID>                |
> ++--------------+---------------------+
> +| Command      | 10                  |
> ++--------------+---------------------+
> +| Message size | 28 + data size      |
> ++--------------+---------------------+
> +| Flags Reply  | bit set in reply    |
> ++--------------+---------------------+
> +| DMA info     | DMA read/write data |
> ++--------------+---------------------+
> +
> +This request is sent from the server to the client to read from guest memory.
> +In the request messages, there will be no data, and the count field will be the
> +amount of data to be read. The reply will include the data read, and its count
> +field will be the amount of data read.
> +
> +VFIO_USER_DMA_WRITE
> +-------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Device ID    | <ID>                   |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 11                     |
> ++--------------+------------------------+
> +| Message size | 28 + data size         |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| DMA info     | DMA read/write data    |
> ++--------------+------------------------+
> +
> +This request is sent from the server to the client to write to guest memory.
> +The request message will contain the data to be written, and its count field
> +will contain the amount of write data. The count field in the reply will be
> +zero.
> +
> +VFIO_USER_VM_INTERRUPT
> +----------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++----------------+------------------------+
> +| Name           | Value                  |
> ++================+========================+
> +| Device ID      | <ID>                   |
> ++----------------+------------------------+
> +| Message ID     | <ID>                   |
> ++----------------+------------------------+
> +| Command        | 12                     |
> ++----------------+------------------------+
> +| Message size   | 24                     |
> ++----------------+------------------------+
> +| Flags          | Reply bit set in reply |
> ++----------------+------------------------+
> +| Interrupt info | <interrupt>            |
> ++----------------+------------------------+
> +
> +This request is sent from the server to the client to signal the device has
> +raised an interrupt.
> +
> +Interrupt info format
> +^^^^^^^^^^^^^^^^^^^^^
> +
> ++----------+--------+------+
> +| Name     | Offset | Size |
> ++==========+========+======+
> +| Index    | 16     | 4    |
> ++----------+--------+------+
> +| Subindex | 20     | 4    |
> ++----------+--------+------+
> +
> +* Index is the interrupt index; it is the same value used in VFIO_USER_SET_IRQS.
> +* Subindex is relative to the index, e.g., the vector number used in PCI MSI/X
> +  type interrupts.
> +
> +VFIO_USER_DEVICE_RESET
> +----------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Device ID    | <ID>                   |
> ++--------------+------------------------+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 13                     |
> ++--------------+------------------------+
> +| Message size | 16                     |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +
> +This request is sent from the client to the server to reset the device.
> +
> +Appendices
> +==========
> +
> +Unused VFIO ioctl() commands
> +----------------------------
> +
> +The following commands must be handled by the client and not sent to the server:
> +
> +* VFIO_GET_API_VERSION
> +* VFIO_CHECK_EXTENSION
> +* VFIO_SET_IOMMU
> +* VFIO_GROUP_GET_STATUS
> +* VFIO_GROUP_SET_CONTAINER
> +* VFIO_GROUP_UNSET_CONTAINER
> +* VFIO_GROUP_GET_DEVICE_FD
> +* VFIO_IOMMU_GET_INFO
> +
> +However, once support for live migration for VFIO devices is finalized some
> +of the above commands might have to be handled by the client. This will be
> +addressed in a future protocol version.
> +
> +Live Migration
> +--------------
> +Currently live migration is not supported for devices passed through via VFIO,
> +therefore it is not supported for VFIO-over-socket, either. This is being
> +actively worked on in the "Add migration support for VFIO devices" (v25) patch
> +series.
> +
> +VFIO groups and containers
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The current VFIO implementation includes group and container idioms that
> +describe how a device relates to the host IOMMU. In the VFIO over socket
> +implementation, the IOMMU is implemented in SW by the client, and isn't visible
> +to the server. The simplest idea is for the client is to put each device into


s/is for/for

> +its own group and container.

> -----Original Message-----
> From: Nikos Dragazis <ndragazis@arrikto.com>
> Sent: 21 July 2020 17:34
> To: Thanos Makatos <thanos.makatos@nutanix.com>
> Cc: qemu-devel@nongnu.org; benjamin.walker@intel.com;
> elena.ufimtseva@oracle.com; tomassetti.andrea@gmail.com; John G
> Johnson <john.g.johnson@oracle.com>; jag.raman@oracle.com; Swapnil
> Ingle <swapnil.ingle@nutanix.com>; james.r.harris@intel.com;
> konrad.wilk@oracle.com; yuvalkashtan@gmail.com; dgilbert@redhat.com;
> Raphael Norwitz <raphael.norwitz@nutanix.com>; ismael@linux.com;
> alex.williamson@redhat.com; Kanth.Ghatraju@oracle.com;
> stefanha@redhat.com; Felipe Franciosi <felipe@nutanix.com>;
> marcandre.lureau@redhat.com; tina.zhang@intel.com;
> changpeng.liu@intel.com
> Subject: Re: [PATCH v3] introduce VFIO-over-socket protocol specificaion
> 
> Hi Thanos,
> 
> I had a quick look on the spec. Leaving some comments inline.
> 
> On 17/7/20 2:20 μ.μ., Thanos Makatos wrote:
> 
> > This patch introduces the VFIO-over-socket protocol specification, which
> > is designed to allow devices to be emulated outside QEMU, in a separate
> > process. VFIO-over-socket reuses the existing VFIO defines, structs and
> > concepts.
> >
> > It has been earlier discussed as an RFC in:
> > "RFC: use VFIO over a UNIX domain socket to implement device offloading"
> >
> > Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> > Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
> >
> > ---
> >
> > Changed since v1:
> >    * fix coding style issues
> >    * update MAINTAINERS for VFIO-over-socket
> >    * add vfio-over-socket to ToC
> >
> > Changed since v2:
> >    * fix whitespace
> >
> > Regarding the build failure, I have not been able to reproduce it locally
> > using the docker image on my Debian 10.4 machine.
> > ---
> >   MAINTAINERS                     |    6 +
> >   docs/devel/index.rst            |    1 +
> >   docs/devel/vfio-over-socket.rst | 1135
> +++++++++++++++++++++++++++++++++++++++
> >   3 files changed, 1142 insertions(+)
> >   create mode 100644 docs/devel/vfio-over-socket.rst
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 030faf0..bb81590 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -1732,6 +1732,12 @@ F: hw/vfio/ap.c
> >   F: docs/system/s390x/vfio-ap.rst
> >   L: qemu-s390x@nongnu.org
> >
> > +VFIO-over-socket
> > +M: John G Johnson <john.g.johnson@oracle.com>
> > +M: Thanos Makatos <thanos.makatos@nutanix.com>
> > +S: Supported
> > +F: docs/devel/vfio-over-socket.rst
> > +
> >   vhost
> >   M: Michael S. Tsirkin <mst@redhat.com>
> >   S: Supported
> > diff --git a/docs/devel/index.rst b/docs/devel/index.rst
> > index ae6eac7..0439460 100644
> > --- a/docs/devel/index.rst
> > +++ b/docs/devel/index.rst
> > @@ -30,3 +30,4 @@ Contents:
> >      reset
> >      s390-dasd-ipl
> >      clocks
> > +   vfio-over-socket
> > diff --git a/docs/devel/vfio-over-socket.rst b/docs/devel/vfio-over-
> socket.rst
> > new file mode 100644
> > index 0000000..b474f23
> > --- /dev/null
> > +++ b/docs/devel/vfio-over-socket.rst
> > @@ -0,0 +1,1135 @@
> > +***************************************
> > +VFIO-over-socket Protocol Specification
> > +***************************************
> > +
> > +Version 0.1
> > +
> > +Introduction
> > +============
> > +VFIO-over-socket, also known as vfio-user, is a protocol that allows a
> device
> 
> I think there is no point in having two names for the same protocol,
> "vfio-over-socket" and "vfio-user".

Yes, we'll use vfio-user from now on.

> 
> > +to be virtualized in a separate process outside of QEMU. VFIO-over-
> socket
> > +devices consist of a generic VFIO device type, living inside QEMU, which
> we
> > +call the client, and the core device implementation, living outside QEMU,
> which
> > +we call the server. VFIO-over-socket can be the main transport
> mechanism for
> > +multi-process QEMU, however it can be used by other applications
> offering
> > +device virtualization. Explaining the advantages of a
> > +disaggregated/multi-process QEMU, and device virtualization outside
> QEMU in
> > +general, is beyond the scope of this document.
> > +
> > +This document focuses on specifying the VFIO-over-socket protocol. VFIO
> has
> > +been chosen for the following reasons:
> > +
> > +1) It is a mature and stable API, backed by an extensively used
> framework.
> > +2) The existing VFIO client implementation (qemu/hw/vfio/) can be
> largely
> > +   reused.
> > +
> > +In a proof of concept implementation it has been demonstrated that
> using VFIO
> > +over a UNIX domain socket is a viable option. VFIO-over-socket is
> designed with
> > +QEMU in mind, however it could be used by other client applications. The
> > +VFIO-over-socket protocol does not require that QEMU's VFIO client
> > +implementation is used in QEMU. None of the VFIO kernel modules are
> required
> > +for supporting the protocol, neither in the client nor the server, only the
> > +source header files are used.
> > +
> > +The main idea is to allow a virtual device to function in a separate process
> in
> > +the same host over a UNIX domain socket. A UNIX domain socket
> (AF_UNIX) is
> > +chosen because we can trivially send file descriptors over it, which in turn
> > +allows:
> > +
> > +* Sharing of guest memory for DMA with the virtual device process.
> > +* Sharing of virtual device memory with the guest for fast MMIO.
> > +* Efficient sharing of eventfd's for triggering interrupts.
> > +
> > +However, other socket types could be used which allows the virtual
> device
> > +process to run in a separate guest in the same host (AF_VSOCK) or
> remotely
> > +(AF_INET). Theoretically the underlying transport doesn't necessarily have
> to
> > +be a socket, however we don't examine such alternatives. In this
> document we
> > +focus on using a UNIX domain socket and introduce basic support for the
> other
> > +two types of sockets without considering performance implications.
> > +
> > +This document does not yet describe any internal details of the server-
> side
> > +implementation, however QEMU's VFIO client implementation will have
> to be
> > +adapted according to this protocol in order to support VFIO-over-socket
> virtual
> > +devices.
> > +
> > +VFIO
> > +====
> > +VFIO is a framework that allows a physical device to be securely passed
> through
> > +to a user space process; the kernel does not drive the device at all.
> 
> I would remove the last part: "the kernel does not drive the device at
> all". Isn't that quite inaccurate? The kernel does drive the device with
> the vfio driver. The user space driver needs the vfio driver in order to
> do certain things like, for example, write on a port or receive
> notifications for device interrupts.

In that sense you're right, what we meant to say is that the device-specific
driver doesn't drive the device. We'll rephrase it.

> 
> > +Typically, the user space process is a VM and the device is passed through
> to
> > +it in order to achieve high performance. VFIO provides an API and the
> required
> > +functionality in the kernel. QEMU has adopted VFIO to allow a guest
> virtual
> > +machine to directly access physical devices, instead of emulating them in
> 
> Maybe s/guest virtual machine/guest ?

OK.

> 
> > +software
> 
> Missing dot here.

OK.

> 
> > +
> > +VFIO-over-socket reuses the core VFIO concepts defined in its API, but
> > +implements them as messages to be sent over a UNIX-domain socket. It
> does not
> 
> s/UNIX-domain/UNIX domain (just to have the same name everywhere)
> 
> > +change the kernel-based VFIO in any way, in fact none of the VFIO kernel
> > +modules need to be loaded to use VFIO-over-socket. It is also possible for
> QEMU
> > +to concurrently use the current kernel-based VFIO for one guest device,
> and use
> > +VFIO-over-socket for another device in the same guest.
> > +
> > +VFIO Device Model
> > +-----------------
> > +A device under VFIO presents a standard VFIO model to the user process.
> Many
> > +of the VFIO operations in the existing kernel model use the ioctl() system
> > +call, and references to the existing model are called the ioctl()
> > +implementation in this document.
> > +
> > +The following sections describe the set of messages that implement the
> VFIO
> > +device model over a UNIX domain socket. In many cases, the messages
> are direct
> > +translations of data structures used in the ioctl() implementation.
> Messages
> > +derived from ioctl()s will have a name derived from the ioctl() command
> name.
> > +E.g., the VFIO_GET_INFO ioctl() command becomes a
> VFIO_USER_GET_INFO message.
> > +The purpose for this reuse is to share as much code as feasible with the
> 
> s/for/of

OK.

> 
> > +ioctl() implementation.
> > +
> > +Client and Server
> > +^^^^^^^^^^^^^^^^^
> > +The socket connects two processes together: a client process and a server
> > +process. In the context of this document, the client process is the process
> > +emulating a guest virtual machine, such as QEMU. The server process is a
> > +process that provides device emulation.
> > +
> > +Connection Initiation
> > +^^^^^^^^^^^^^^^^^^^^^
> > +After the client connects to the server, the initial server message is
> > +VFIO_USER_VERSION to propose a protocol version and set of capabilities
> to
> > +apply to the session. The client replies with a compatible version and set
> of
> > +capabilities it will support, or closes the connection if it cannot support the
> > +advertised version.
> > +
> > +Guest Memory Configuration
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +The client uses VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP
> messages to inform
> > +the server of the valid guest DMA ranges that the server can access on
> behalf
> > +of a device. Guest memory may be accessed by the server via
> VFIO_USER_DMA_READ
> > +and VFIO_USER_DMA_WRITE messages over the socket.
> > +
> > +An optimization for server access to guest memory is for the client to
> provide
> > +file descriptors the server can mmap() to directly access guest memory.
> Note
> > +that mmap() privileges cannot be revoked by the client, therefore file
> > +descriptors should only be exported in environments where the client
> trusts the
> > +server not to corrupt guest memory.
> > +
> > +Device Information
> > +^^^^^^^^^^^^^^^^^^
> > +The client uses a VFIO_USER_DEVICE_GET_INFO message to query the
> server for
> > +information about the device. This information includes:
> > +
> > +* The device type and capabilities,
> > +* the number of memory regions, and
> > +* the device presents to the guest the number of interrupt types the
> device
> > +  supports.
> > +
> > +Region Information
> > +^^^^^^^^^^^^^^^^^^
> > +The client uses VFIO_USER_DEVICE_GET_REGION_INFO messages to
> query the server
> > +for information about the device's memory regions. This information
> describes:
> > +
> > +* Read and write permissions, whether it can be memory mapped, and
> whether it
> > +  supports additional capabilities.
> > +* Region index, size, and offset.
> > +
> > +When a region can be mapped by the client, the server provides a file
> > +descriptor which the client can mmap(). The server is responsible for
> polling
> > +for client updates to memory mapped regions.
> > +
> > +Region Capabilities
> > +"""""""""""""""""""
> > +Some regions have additional capabilities that cannot be described
> adequately
> > +by the region info data structure. These capabilities are returned in the
> > +region info reply in a list similar to PCI capabilities in a PCI device's
> > +configuration space.
> > +
> > +Sparse Regions
> > +""""""""""""""
> > +A region can be memory-mappable in whole or in part. When only a
> subset of a
> > +region can be mapped by the client, a
> VFIO_REGION_INFO_CAP_SPARSE_MMAP
> > +capability is included in the region info reply. This capability describes
> > +which portions can be mapped by the client.
> > +
> > +For example, in a virtual NVMe controller, sparse regions can be used so
> that
> > +accesses to the NVMe registers (found in the beginning of BAR0) are
> trapped (an
> > +infrequent an event), while allowing direct access to the doorbells (an
> 
> s/an event/event

OK.

> 
> > +extremely frequent event as every I/O submission requires a write to
> BAR0),
> > +found right after the NVMe registers in BAR0.
> > +
> > +Interrupts
> > +^^^^^^^^^^
> > +The client uses VFIO_USER_DEVICE_GET_IRQ_INFO messages to query
> the server for
> > +the device's interrupt types. The interrupt types are specific to the bus
> the
> > +device is attached to, and the client is expected to know the capabilities of
> > +each interrupt type. The server can signal an interrupt either with
> > +VFIO_USER_VM_INTERRUPT messages over the socket, or can directly
> inject
> > +interrupts into the guest via an event file descriptor. The client configures
> > +how the server signals an interrupt with VFIO_USER_SET_IRQS messages.
> > +
> > +Device Read and Write
> > +^^^^^^^^^^^^^^^^^^^^^
> > +When the guest executes load or store operations to device memory, the
> client
> > +forwards these operations to the server with VFIO_USER_REGION_READ
> or
> > +VFIO_USER_REGION_WRITE messages. The server will reply with data
> from the
> > +device on read operations or an acknowledgement on write operations.
> > +
> > +DMA
> > +^^^
> > +When a device performs DMA accesses to guest memory, the server will
> forward
> > +them to the client with VFIO_USER_DMA_READ and
> VFIO_USER_DMA_WRITE messages.
> > +These messages can only be used to access guest memory the client has
> > +configured into the server.
> > +
> > +Protocol Specification
> > +======================
> > +To distinguish from the base VFIO symbols, all VFIO-over-socket symbols
> are
> > +prefixed with vfio_user or VFIO_USER. In revision 0.1, all data is in the
> > +little-endian format, although this may be relaxed in future revision in
> cases
> > +where the client and server are both big-endian. The messages are
> formatted
> > +for seamless reuse of the native VFIO structs. A server can serve:
> > +
> > +1) multiple clients, and/or
> > +2) multiple virtual devices, belonging to one or more clients.
> > +
> > +Therefore each message requires a header that uniquely identifies the
> virtual
> > +device. It is a server-side implementation detail whether a single server
> > +handles multiple virtual devices from the same or multiple guests.
> > +
> > +Socket
> > +------
> > +A single UNIX domain socket is assumed to be used for each device. The
> location
> 
> Is it correct for a spec to assume things?

We actually want to say that "a single UNIX domain socket is used for each
device", we'll correct it.

> 
> > +of the socket is implementation-specific. Multiplexing clients, devices, and
> > +servers over the same socket is not supported in this version of the
> protocol,
> > +but a device ID field exists in the message header so that a future support
> can
> > +be added without a major version change.
> > +
> > +Authentication
> > +--------------
> > +For AF_UNIX, we rely on OS mandatory access controls on the socket
> files,
> > +therefore it is up to the management layer to set up the socket as
> required.
> > +Socket types than span guests or hosts will require a proper
> authentication
> > +mechanism. Defining that mechanism is deferred to a future version of
> the
> > +protocol.
> > +
> > +Request Concurrency
> > +-------------------
> > +There can be multiple outstanding requests per virtual device, e.g. a
> > +frame buffer where the guest does multiple stores to the virtual device.
> The
> > +server can execute and reorder non-conflicting requests in parallel,
> depending
> > +on the device semantics.
> > +
> > +Socket Disconnection Behavior
> > +-----------------------------
> > +The server and the client can disconnect from each other, either
> intentionally
> > +or unexpectedly. Both the client and the server need to know how to
> handle such
> > +events.
> > +
> > +Server Disconnection
> > +^^^^^^^^^^^^^^^^^^^^
> > +A server disconnecting from the client may indicate that:
> > +
> > +1) A virtual device has been restarted, either intentionally (e.g. because of
> a
> > +device update) or unintentionally (e.g. because of a crash). In any case,
> the
> > +virtual device will come back so the client should not do anything (e.g.
> simply
> > +reconnect and retry failed operations).
> > +
> 
> Indentation issue ^^ (also remove the space, there are no spaces between
> the
> elements in the other numbered lists).

OK.

> 
> > +2) A virtual device has been shut down with no intention to be restarted.
> > +
> > +It is impossible for the client to know whether or not a failure is
> > +intermittent or innocuous and should be retried, therefore the client
> should
> > +attempt to reconnect to the socket. Since an intentional server restart
> (e.g.
> > +due to an upgrade) might take some time, a reasonable timeout should
> be used.
> > +In cases where the disconnection is expected (e.g. the guest shutting
> down), no
> > +new requests will be sent anyway so this situation doesn't pose a
> problem. The
> > +control stack will clean up accordingly.
> > +
> > +Parametrizing this behaviour by having the virtual device advertise a
> > +reasonable reconnect is deferred to a future version of the protocol.
> > +
> > +Client Disconnection
> > +^^^^^^^^^^^^^^^^^^^^
> > +The client disconnecting from the server primarily means that the QEMU
> process
> > +has exited. Currently this means that the guest is shut down so the device
> is
> > +no longer needed therefore the server can automatically exit. However,
> there
> > +can be cases where a client disconnect should not result in a server exit:
> 
> s/disconnect/disconnection

They're synonyms.

> 
> > +
> > +1) A single server serving multiple clients.
> > +2) A multi-process QEMU upgrading itself step by step, which isn't yet
> > +   implemented.
> > +
> > +Therefore in order for the protocol to be forward compatible the server
> should
> > +take no action when the client disconnects. If anything happens to the
> client
> > +process the control stack will know about it and can clean up resources
> > +accordingly.
> > +
> > +Request Retry and Response Timeout
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +QEMU's VFIO retries certain operations if they fail. While this makes sense
> for
> > +real HW, we don't know for sure whether it makes sense for virtual
> devices. A
> > +failed request is a request that has been successfully sent and has been
> > +responded to with an error code. Failure to send the request in the first
> place
> > +(e.g. because the socket is disconnected) is a different type of error
> examined
> > +earlier in the disconnect section.
> > +
> > +Defining a retry and timeout scheme if deferred to a future version of the
> 
> s/if/is

OK.

> 
> > +protocol.
> > +
> > +Commands
> > +--------
> > +The following table lists the VFIO message command IDs, and whether
> the
> > +message request is sent from the client or the server.
> > +
> > ++----------------------------------+---------+-------------------+
> > +| Name                             | Command | Request Direction |
> >
> ++==================================+=========+===========
> ========+
> > +| VFIO_USER_VERSION                | 1       | server → client   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DMA_MAP                | 2       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DMA_UNMAP              | 3       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DEVICE_GET_INFO        | 4       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DEVICE_GET_REGION_INFO | 5       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DEVICE_GET_IRQ_INFO    | 6       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DEVICE_SET_IRQS        | 7       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_REGION_READ            | 8       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_REGION_WRITE           | 9       | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DMA_READ               | 10      | server → client   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_DMA_READ               | 11      | server → client   |
> 
> Isn't that VFIO_USER_DMA_WRITE?

Correct.

> 
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_USER_VM_INTERRUPT           | 12      | server → client   |
> > ++----------------------------------+---------+-------------------+
> > +| VFIO_DEVICE_RESET                | 13      | client → server   |
> > ++----------------------------------+---------+-------------------+
> > +
> > +Header
> > +------
> > +All messages are preceded by a 16 byte header that contains basic
> information
> > +about the message. The header is followed by message-specific data
> described
> > +in the sections below.
> > +
> > ++----------------+--------+-------------+
> > +| Name           | Offset | Size        |
> > ++================+========+=============+
> > +| Device ID      | 0      | 2           |
> > ++----------------+--------+-------------+
> > +| Message ID     | 2      | 2           |
> > ++----------------+--------+-------------+
> > +| Command        | 4      | 4           |
> > ++----------------+--------+-------------+
> > +| Message size   | 8      | 4           |
> > ++----------------+--------+-------------+
> > +| Flags          | 12     | 4           |
> > ++----------------+--------+-------------+
> > +|                | +-----+------------+ |
> > +|                | | Bit | Definition | |
> > +|                | +=====+============+ |
> > +|                | | 0   | Reply      | |
> > +|                | +-----+------------+ |
> > +|                | | 1   | No_reply   | |
> > +|                | +-----+------------+ |
> > ++----------------+--------+-------------+
> > +| <message data> | 16     | variable    |
> > ++----------------+--------+-------------+
> > +
> > +* Device ID identifies the destination device of the message. This field is
> > +  reserved when the server only supports one device per socket.
> > +* Message ID identifies the message, and is used in the message
> acknowledgement.
> > +* Command specifies the command to be executed, listed in the
> Command Table.
> > +* Message size contains the size of the entire message, including the
> header.
> > +* Flags contains attributes of the message:
> > +
> > +  * The reply bit differentiates request messages from reply messages. A
> reply
> > +    message acknowledges a previous request with the same message ID.
> > +  * No_reply indicates that no reply is needed for this request. This is
> > +    commonly used when multiple requests are sent, and only the last
> needs
> > +    acknowledgement.
> > +
> > +VFIO_USER_VERSION
> > +-----------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > ++==============+========================+
> > +| Device ID    | 0                      |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | 1                      |
> > ++--------------+------------------------+
> > +| Message size | 16 + version length    |
> > ++--------------+------------------------+
> > +| Flags        | Reply bit set in reply |
> > ++--------------+------------------------+
> > +| Version      | JSON byte array        |
> > ++--------------+------------------------+
> > +
> > +This is the initial message sent by the server after the socket connection is
> > +established. The version is in JSON format, and the following objects must
> be
> > +included:
> > +
> > ++--------------+--------+---------------------------------------------------+
> > +| Name         | Type   | Description                                       |
> >
> ++==============+========+================================
> ===================+
> > +| version      | object | {“major”: <number>, “minor”: <number>}            |
> > +|              |        | Version supported by the sender, e.g. “0.1”.      |
> > ++--------------+--------+---------------------------------------------------+
> > +| type         | string | Fixed to “vfio-user”.                             |
> > ++--------------+--------+---------------------------------------------------+
> > +| capabilities | array  | Reserved. Can be omitted for v0.1, otherwise must
> |
> > +|              |        | be empty.                                         |
> > ++--------------+--------+---------------------------------------------------+
> > +
> > +Versioning and Feature Support
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +Upon accepting a connection, the server must send a
> VFIO_USER_VERSION message
> > +proposing a protocol version and a set of capabilities. The client compares
> > +these with the versions and capabilities it supports and sends a
> > +VFIO_USER_VERSION reply according to the following rules.
> > +
> > +* The major version in the reply must be the same as proposed. If the
> client
> > +  does not support the proposed major, it closes the connection.
> > +* The minor version in the reply must be equal to or less than the minor
> > +  version proposed.
> > +* The capability list must be a subset of those proposed. If the client
> > +  requires a capability the server did not include, it closes the connection.
> > +* If type is not “vfio-user”, the client closes the connection.
> > +
> > +The protocol major version will only change when incompatible protocol
> changes
> > +are made, such as changing the message format. The minor version may
> change
> > +when compatible changes are made, such as adding new messages or
> capabilities,
> > +Both the client and server must support all minor versions less than the
> > +maximum minor version it supports. E.g., an implementation that
> supports
> > +version 1.3 must also support 1.0 through 1.2.
> > +
> > +VFIO_USER_DMA_MAP
> > +-----------------
> > +
> > +VFIO_USER_DMA_UNMAP
> > +-------------------
> > +
> > +Message Format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > ++==============+========================+
> > +| Device ID    | 0                      |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | MAP=2, UNMAP=3         |
> > ++--------------+------------------------+
> > +| Message size | 16 + table size        |
> > ++--------------+------------------------+
> > +| Flags        | Reply bit set in reply |
> > ++--------------+------------------------+
> > +| Table        | array of table entries |
> > ++--------------+------------------------+
> > +
> > +This message is sent by the client to the server to inform it of the guest
> > +memory regions the device can access. It must be sent before the device
> can
> > +perform any DMA to the guest. It is normally sent directly after the
> version
> > +handshake is completed, but may also occur when memory is added or
> subtracted
> > +in the guest.
> > +
> > +The table is an array of the following structure. This structure is 32 bytes
> > +in size, so the message size will be 16 + (# of table entries * 32). If a
> > +region being added can be directly mapped by the server, an array of file
> > +descriptors will be sent as part of the message meta-data. Each region
> entry
> > +will have a corresponding file descriptor. On AF_UNIX sockets, the file
> > +descriptors will be passed as SCM_RIGHTS type ancillary data.
> > +
> > +Table entry format
> > +^^^^^^^^^^^^^^^^^^
> > +
> > ++-------------+--------+-------------+
> > +| Name        | Offset | Size        |
> > ++=============+========+=============+
> > +| Address     | 0      | 8           |
> > ++-------------+--------+-------------+
> > +| Size        | 8      | 8           |
> > ++-------------+--------+-------------+
> > +| Offset      | 16     | 8           |
> > ++-------------+--------+-------------+
> > +| Protections | 24     | 4           |
> > ++-------------+--------+-------------+
> > +| Flags       | 28     | 4           |
> > ++-------------+--------+-------------+
> > +|             | +-----+------------+ |
> > +|             | | Bit | Definition | |
> > +|             | +=====+============+ |
> > +|             | | 0   | Mappable   | |
> > +|             | +-----+------------+ |
> > ++-------------+--------+-------------+
> > +
> > +* Address is the base DMA address of the region.
> > +* Size is the size of the region.
> > +* Offset is the file offset of the region with respect to the associated file
> > +  descriptor.
> > +* Protections are the region's protection attributes as encoded in
> > +  ``<sys/mman.h>``.
> > +* Flags contain the following region attributes:
> > +
> > +  * Mappable indicate the region can be mapped via the mmap() system
> call using
> > +    the file descriptor provided in the message meta-data.
> > +
> > +VFIO_USER_DEVICE_GET_INFO
> > +-------------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+----------------------------+
> > +| Name         | Value                      |
> > ++==============+============================+
> > +| Device ID    | <ID>                       |
> > ++--------------+----------------------------+
> > +| Message ID   | <ID>                       |
> > ++--------------+----------------------------+
> > +| Command      | 4                          |
> > ++--------------+----------------------------+
> > +| Message size | 16 in request, 32 in reply |
> > ++--------------+----------------------------+
> > +| Flags        | Reply bit set in reply     |
> > ++--------------+----------------------------+
> > +| Device info  | VFIO device info           |
> > ++--------------+----------------------------+
> > +
> > +This message is sent by the client to the server to query for basic
> information
> > +about the device. Only the message header is needed in the request
> message.
> > +The VFIO device info structure is defined in ``<sys/vfio.h>`` (``struct
> > +vfio_device_info``).
> > +
> > +VFIO device info format
> > +^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > ++-------------+--------+--------------------------+
> > +| Name        | Offset | Size                     |
> > ++=============+========+==========================+
> > +| argsz       | 16     | 4                        |
> > ++-------------+--------+--------------------------+
> > +| flags       | 20     | 4                        |
> > ++-------------+--------+--------------------------+
> > +|             | +-----+-------------------------+ |
> > +|             | | Bit | Definition              | |
> > +|             | +=====+=========================+ |
> > +|             | | 0   | VFIO_DEVICE_FLAGS_RESET | |
> > +|             | +-----+-------------------------+ |
> > +|             | | 1   | VFIO_DEVICE_FLAGS_PCI   | |
> > +|             | +-----+-------------------------+ |
> > ++-------------+--------+--------------------------+
> > +| num_regions | 24     | 4                        |
> > ++-------------+--------+--------------------------+
> > +| num_irqs    | 28     | 4                        |
> > ++-------------+--------+--------------------------+
> > +
> > +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> > +  implementation.
> > +* flags contains the following device attributes.
> > +
> > +  * VFIO_DEVICE_FLAGS_RESET indicates the device supports the
> > +    VFIO_USER_DEVICE_RESET message.
> > +  * VFIO_DEVICE_FLAGS_PCI indicates the device is a PCI device.
> > +
> > +* num_regions is the number of memory regions the device exposes.
> > +* num_irqs is the number of distinct interrupt types the device supports.
> > +
> > +This version of the protocol only supports PCI devices. Additional devices
> may
> > +be supported in future versions.
> > +
> > +VFIO_USER_DEVICE_GET_REGION_INFO
> > +--------------------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------+
> > +| Name         | Value            |
> > ++==============+==================+
> > +| Device ID    | <ID>             |
> > ++--------------+------------------+
> > +| Message ID   | <ID>             |
> > ++--------------+------------------+
> > +| Command      | 5                |
> > ++--------------+------------------+
> > +| Message size | 48 + any caps    |
> > ++--------------+------------------+
> > +| Flags Reply  | bit set in reply |
> > ++--------------+------------------+
> > +| Region info  | VFIO region info |
> > ++--------------+------------------+
> > +
> > +This message is sent by the client to the server to query for information
> about
> > +device memory regions. The VFIO region info structure is defined in
> > +``<sys/vfio.h>`` (``struct vfio_region_info``).
> > +
> > +VFIO region info format
> > +^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > ++------------+--------+------------------------------+
> > +| Name       | Offset | Size                         |
> > ++============+========+==============================+
> > +| argsz      | 16     | 4                            |
> > ++------------+--------+------------------------------+
> > +| flags      | 20     | 4                            |
> > ++------------+--------+------------------------------+
> > +|            | +-----+-----------------------------+ |
> > +|            | | Bit | Definition                  | |
> > +|            | +=====+=============================+ |
> > +|            | | 0   | VFIO_REGION_INFO_FLAG_READ  | |
> > +|            | +-----+-----------------------------+ |
> > +|            | | 1   | VFIO_REGION_INFO_FLAG_WRITE | |
> > +|            | +-----+-----------------------------+ |
> > +|            | | 2   | VFIO_REGION_INFO_FLAG_MMAP  | |
> > +|            | +-----+-----------------------------+ |
> > +|            | | 3   | VFIO_REGION_INFO_FLAG_CAPS  | |
> > +|            | +-----+-----------------------------+ |
> > ++------------+--------+------------------------------+
> > +| index      | 24     | 4                            |
> > ++------------+--------+------------------------------+
> > +| cap_offset | 28     | 4                            |
> > ++------------+--------+------------------------------+
> > +| size       | 32     | 8                            |
> > ++------------+--------+------------------------------+
> > +| offset     | 40     | 8                            |
> > ++------------+--------+------------------------------+
> > +
> > +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> > +  implementation.
> > +* flags are attributes of the region:
> > +
> > +  * VFIO_REGION_INFO_FLAG_READ allows client read access to the
> region.
> > +  * VFIO_REGION_INFO_FLAG_WRITE allows client write access region.
> 
> s/region/to the region

OK.

> 
> > +  * VFIO_REGION_INFO_FLAG_MMAP specifies the client can mmap() the
> region. When
> > +    this flag is set, the reply will include a file descriptor in its meta-data.
> > +    On AF_UNIX sockets, the file descriptors will be passed as SCM_RIGHTS
> type
> > +    ancillary data.
> > +  * VFIO_REGION_INFO_FLAG_CAPS indicates additional capabilities
> found in the
> > +    reply.
> > +
> > +* index is the index of memory region being queried, it is the only field
> that
> > +  is required to be set in the request message.
> > +* cap_offset describes where additional region capabilities can be found.
> > +  cap_offset is relative to the beginning of the VFIO region info structure.
> > +  The data structure it points is a VFIO cap header defined in
> ``<sys/vfio.h>``.
> > +* size is the size of the region.
> > +* offset is the offset given to the mmap() system call for regions with the
> > +  MMAP attribute. It is also used as the base offset when mapping a VFIO
> > +  sparse mmap area, described below.
> > +
> > +VFIO Region capabilities
> > +^^^^^^^^^^^^^^^^^^^^^^^^
> > +The VFIO region information can also include a capabilities list. This list is
> > +similar to a PCI capability list - each entry has a common header that
> > +identifies a capability and where the next capability in the list can be
> found.
> > +The VFIO capability header format is defined in ``<sys/vfio.h>`` (``struct
> > +vfio_info_cap_header``).
> > +
> > +VFIO cap header format
> > +^^^^^^^^^^^^^^^^^^^^^^
> > +
> > ++---------+--------+------+
> > +| Name    | Offset | Size |
> > ++=========+========+======+
> > +| id      | 0      | 2    |
> > ++---------+--------+------+
> > +| version | 2      | 2    |
> > ++---------+--------+------+
> > +| next    | 4      | 4    |
> > ++---------+--------+------+
> > +
> > +* id is the capability identity.
> > +* version is a capability-specific version number.
> > +* next specifies the offset of the next capability in the capability list. It
> > +  is relative to the beginning of the VFIO region info structure.
> > +
> > +VFIO sparse mmap
> > +^^^^^^^^^^^^^^^^
> > +
> > ++------------------+----------------------------------+
> > +| Name             | Value                            |
> > ++==================+==================================+
> > +| id               | VFIO_REGION_INFO_CAP_SPARSE_MMAP |
> > ++------------------+----------------------------------+
> > +| version          | 0x1                              |
> > ++------------------+----------------------------------+
> > +| next             | <next>                           |
> > ++------------------+----------------------------------+
> > +| sparse mmap info | VFIO region info sparse mmap     |
> > ++------------------+----------------------------------+
> > +
> > +The only capability supported in this version of the protocol is for sparse
> > +mmap. This capability is defined when only a subrange of the region
> supports
> > +direct access by the client via mmap(). The VFIO sparse mmap area is
> defined in
> > +``<sys/vfio.h>`` (``struct vfio_region_sparse_mmap_area``).
> > +
> > +VFIO region info cap sparse mmap
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > ++----------+--------+------+
> > +| Name     | Offset | Size |
> > ++==========+========+======+
> > +| nr_areas | 0      | 4    |
> > ++----------+--------+------+
> > +| reserved | 4      | 4    |
> > ++----------+--------+------+
> > +| offset   | 8      | 8    |
> > ++----------+--------+------+
> > +| size     | 16     | 9    |
> > ++----------+--------+------+
> > +| ...      |        |      |
> > ++----------+--------+------+
> > +
> > +* nr_areas is the number of sparse mmap areas in the region.
> > +* offset and size describe a single area that can be mapped by the client.
> > +  There will be nr_areas pairs of offset and size. The offset will be added to
> > +  the base offset given in the VFIO_USER_DEVICE_GET_REGION_INFO to
> form the
> > +  offset argument of the subsequent mmap() call.
> > +
> > +The VFIO sparse mmap area is defined in ``<sys/vfio.h>`` (``struct
> > +vfio_region_info_cap_sparse_mmap``).
> > +
> > +VFIO_USER_DEVICE_GET_IRQ_INFO
> > +-----------------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > ++==============+========================+
> > +| Device ID    | <ID>                   |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | 6                      |
> > ++--------------+------------------------+
> > +| Message size | 32                     |
> > ++--------------+------------------------+
> > +| Flags        | Reply bit set in reply |
> > ++--------------+------------------------+
> > +| IRQ info     | VFIO IRQ info          |
> > ++--------------+------------------------+
> > +
> > +This message is sent by the client to the server to query for information
> about
> > +device interrupt types. The VFIO IRQ info structure is defined in
> > +``<sys/vfio.h>`` (``struct vfio_irq_info``).
> > +
> > +VFIO IRQ info format
> > +^^^^^^^^^^^^^^^^^^^^
> > +
> > ++-------+--------+---------------------------+
> > +| Name  | Offset | Size                      |
> > ++=======+========+===========================+
> > +| argsz | 16     | 4                         |
> > ++-------+--------+---------------------------+
> > +| flags | 20     | 4                         |
> > ++-------+--------+---------------------------+
> > +|       | +-----+--------------------------+ |
> > +|       | | Bit | Definition               | |
> > +|       | +=====+==========================+ |
> > +|       | | 0   | VFIO_IRQ_INFO_EVENTFD    | |
> > +|       | +-----+--------------------------+ |
> > +|       | | 1   | VFIO_IRQ_INFO_MASKABLE   | |
> > +|       | +-----+--------------------------+ |
> > +|       | | 2   | VFIO_IRQ_INFO_AUTOMASKED | |
> > +|       | +-----+--------------------------+ |
> > +|       | | 3   | VFIO_IRQ_INFO_NORESIZE   | |
> > +|       | +-----+--------------------------+ |
> > ++-------+--------+---------------------------+
> > +| index | 24     | 4                         |
> > ++-------+--------+---------------------------+
> > +| count | 28     | 4                         |
> > ++-------+--------+---------------------------+
> > +
> > +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> > +  implementation.
> > +* flags defines IRQ attributes:
> > +
> > +  * VFIO_IRQ_INFO_EVENTFD indicates the IRQ type can support server
> eventfd
> > +    signalling.
> > +  * VFIO_IRQ_INFO_MASKABLE indicates that the IRQ type supports the
> MASK and
> > +    UNMASK actions in a VFIO_USER_DEVICE_SET_IRQS message.
> > +  * VFIO_IRQ_INFO_AUTOMASKED indicates the IRQ type masks itself
> after being
> > +    triggered, and the client must send an UNMASK action to receive new
> > +    interrupts.
> > +  * VFIO_IRQ_INFO_NORESIZE indicates VFIO_USER_SET_IRQS operations
> setup
> > +    interrupts as a set, and new subindexes cannot be enabled without
> disabling
> > +    the entire type.
> > +
> > +* index is the index of IRQ type being queried, it is the only field that is
> > +  required to be set in the request message.
> > +* count describes the number of interrupts of the queried type.
> > +
> > +VFIO_USER_DEVICE_SET_IRQS
> > +-------------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > +| Device ID    | <ID>                   |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | 7                      |
> > ++--------------+------------------------+
> > +| Message size | 36 + any data          |
> > ++--------------+------------------------+
> > +| Flags        | Reply bit set in reply |
> > ++--------------+------------------------+
> > +| IRQ set      | VFIO IRQ set           |
> > ++--------------+------------------------+
> > +
> > +This message is sent by the client to the server to set actions for device
> > +interrupt types. The VFIO IRQ set structure is defined in ``<sys/vfio.h>``
> > +(``struct vfio_irq_set``).
> > +
> > +VFIO IRQ info format
> > +^^^^^^^^^^^^^^^^^^^^
> > +
> > ++-------+--------+------------------------------+
> > +| Name  | Offset | Size                         |
> > ++=======+========+==============================+
> > +| argsz | 6      | 4                            |
> > ++-------+--------+------------------------------+
> > +| flags | 20     | 4                            |
> > ++-------+--------+------------------------------+
> > +|       | +-----+-----------------------------+ |
> > +|       | | Bit | Definition                  | |
> > +|       | +=====+=============================+ |
> > +|       | | 0   | VFIO_IRQ_SET_DATA_NONE      | |
> > +|       | +-----+-----------------------------+ |
> > +|       | | 1   | VFIO_IRQ_SET_DATA_BOOL      | |
> > +|       | +-----+-----------------------------+ |
> > +|       | | 2   | VFIO_IRQ_SET_DATA_EVENTFD   | |
> > +|       | +-----+-----------------------------+ |
> > +|       | | 3   | VFIO_IRQ_SET_ACTION_MASK    | |
> > +|       | +-----+-----------------------------+ |
> > +|       | | 4   | VFIO_IRQ_SET_ACTION_UNMASK  | |
> > +|       | +-----+-----------------------------+ |
> > +|       | | 5   | VFIO_IRQ_SET_ACTION_TRIGGER | |
> > +|       | +-----+-----------------------------+ |
> > ++-------+--------+------------------------------+
> > +| index | 24     | 4                            |
> > ++-------+--------+------------------------------+
> > +| start | 28     | 4                            |
> > ++-------+--------+------------------------------+
> > +| count | 32     | 4                            |
> > ++-------+--------+------------------------------+
> > +| data  | 36     | variable                     |
> > ++-------+--------+------------------------------+
> > +
> > +* argz is reserved in vfio-user, it is only used in the ioctl() VFIO
> > +  implementation.
> > +* flags defines the action performed on the interrupt range. The DATA
> flags
> > +  describe the data field sent in the message; the ACTION flags describe
> the
> > +  action to be performed. The flags are mutually exclusive for both sets.
> > +
> > +  * VFIO_IRQ_SET_DATA_NONE indicates there is no data field in the
> request. The
> > +    action is performed unconditionally.
> > +  * VFIO_IRQ_SET_DATA_BOOL indicates the data field is an array of
> boolean
> > +    bytes. The action is performed if the corresponding boolean is true.
> > +  * VFIO_IRQ_SET_DATA_EVENTFD indicates an array of event file
> descriptors was
> > +    sent in the message meta-data. These descriptors will be signalled
> when the
> > +    action defined by the action flags occurs. In AF_UNIX sockets, the
> > +    descriptors are sent as SCM_RIGHTS type ancillary data.
> > +  * VFIO_IRQ_SET_ACTION_MASK indicates a masking event. It can be
> used with
> > +    VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to mask an
> interrupt, or
> > +    with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
> guest masks
> > +    the interrupt.
> > +  * VFIO_IRQ_SET_ACTION_UNMASK indicates an unmasking event. It can
> be used
> > +    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to
> unmask an
> > +    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event
> when the
> > +    guest unmasks the interrupt.
> > +  * VFIO_IRQ_SET_ACTION_TRIGGER indicates a triggering event. It can be
> used
> > +    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to
> trigger an
> > +    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event
> when the
> > +    guest triggers the interrupt.
> > +
> > +* index is the index of IRQ type being setup.
> > +* start is the start of the subindex being set.
> > +* count describes the number of sub-indexes being set. As a special case,
> a
> > +  count of 0 with data flags of VFIO_IRQ_SET_DATA_NONE disables all
> interrupts
> > +  of the index data is an optional field included when the
> > +  VFIO_IRQ_SET_DATA_BOOL flag is present. It contains an array of
> booleans
> > +  that specify whether the action is to be performed on the corresponding
> > +  index. It's used when the action is only performed on a subset of the
> range
> > +  specified.
> > +
> > +Not all interrupt types support every combination of data and action flags.
> > +The client must know the capabilities of the device and IRQ index before it
> > +sends a VFIO_USER_DEVICE_SET_IRQ message.
> > +
> > +Read and Write Operations
> > +-------------------------
> > +
> > +Not all I/O operations between the client and server can be done via
> direct
> > +access of memory mapped with an mmap() call. In these cases, the client
> and
> > +server use messages sent over the socket. It is expected that these
> operations
> > +will have lower performance than direct access.
> > +
> > +The client can access device memory with VFIO_USER_REGION_READ and
> > +VFIO_USER_REGION_WRITE requests. These share a common data
> structure that
> > +appears after the 16 byte message header.
> > +
> > +REGION Read/Write Data
> > +^^^^^^^^^^^^^^^^^^^^^^
> > +
> > ++--------+--------+----------+
> > +| Name   | Offset | Size     |
> > ++========+========+==========+
> > +| Offset | 16     | 8        |
> > ++--------+--------+----------+
> > +| Region | 24     | 4        |
> > ++--------+--------+----------+
> > +| Count  | 28     | 4        |
> > ++--------+--------+----------+
> > +| Data   | 32     | variable |
> > ++--------+--------+----------+
> > +
> > +* Offset into the region being accessed.
> > +* Region is the index of the region being accessed.
> > +* Count is the size of the data to be transferred.
> > +* Data is the data to be read or written.
> > +
> > +The server can access guest memory with VFIO_USER_DMA_READ and
> > +VFIO_USER_DMA_WRITE messages. These also share a common data
> structure that
> > +appears after the 16 byte message header.
> > +
> > +DMA Read/Write Data
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > ++---------+--------+----------+
> > +| Name    | Offset | Size     |
> > ++=========+========+==========+
> > +| Address | 16     | 8        |
> > ++---------+--------+----------+
> > +| Count   | 24     | 4        |
> > ++---------+--------+----------+
> > +| Data    | 28     | variable |
> > ++---------+--------+----------+
> > +
> > +* Address is the area of guest memory being accessed. This address must
> have
> > +  been exported to the server with a VFIO_USER_DMA_MAP message.
> > +* Count is the size of the data to be transferred.
> > +* Data is the data to be read or written.
> > +
> > +Address and count can also be accessed as ``struct iovec`` from
> ``<sys/uio.h>``.
> > +
> > +VFIO_USER_REGION_READ
> > +---------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > ++==============+========================+
> > +| Device ID    | <ID>                   |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | 8                      |
> > ++--------------+------------------------+
> > +| Message size | 32 + data size         |
> > ++--------------+------------------------+
> > +| Flags Reply  | bit set in reply       |
> > ++--------------+------------------------+
> > +| Read info    | REGION read/write data |
> > ++--------------+------------------------+
> > +
> > +This request is sent from the client to the server to read from device
> memory.
> > +In the request messages, there will be no data, and the count field will be
> the
> > +amount of data to be read. The reply will include the data read, and its
> count
> > +field will be the amount of data read.
> > +
> > +VFIO_USER_REGION_WRITE
> > +----------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > ++==============+========================+
> > +| Device ID    | <ID>                   |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | 9                      |
> > ++--------------+------------------------+
> > +| Message size | 32 + data size         |
> > ++--------------+------------------------+
> > +| Flags        | Reply bit set in reply |
> > ++--------------+------------------------+
> > +| Write info   | REGION read write data |
> > ++--------------+------------------------+
> > +
> > +This request is sent from the client to the server to write to device
> memory.
> > +The request message will contain the data to be written, and its count
> field
> > +will contain the amount of write data. The count field in the reply will be
> > +zero.
> > +
> > +VFIO_USER_DMA_READ
> > +------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+---------------------+
> > +| Name         | Value               |
> > ++==============+=====================+
> > +| Device ID    | <ID>                |
> > ++--------------+---------------------+
> > +| Message ID   | <ID>                |
> > ++--------------+---------------------+
> > +| Command      | 10                  |
> > ++--------------+---------------------+
> > +| Message size | 28 + data size      |
> > ++--------------+---------------------+
> > +| Flags Reply  | bit set in reply    |
> > ++--------------+---------------------+
> > +| DMA info     | DMA read/write data |
> > ++--------------+---------------------+
> > +
> > +This request is sent from the server to the client to read from guest
> memory.
> > +In the request messages, there will be no data, and the count field will be
> the
> > +amount of data to be read. The reply will include the data read, and its
> count
> > +field will be the amount of data read.
> > +
> > +VFIO_USER_DMA_WRITE
> > +-------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > ++==============+========================+
> > +| Device ID    | <ID>                   |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | 11                     |
> > ++--------------+------------------------+
> > +| Message size | 28 + data size         |
> > ++--------------+------------------------+
> > +| Flags        | Reply bit set in reply |
> > ++--------------+------------------------+
> > +| DMA info     | DMA read/write data    |
> > ++--------------+------------------------+
> > +
> > +This request is sent from the server to the client to write to guest
> memory.
> > +The request message will contain the data to be written, and its count
> field
> > +will contain the amount of write data. The count field in the reply will be
> > +zero.
> > +
> > +VFIO_USER_VM_INTERRUPT
> > +----------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++----------------+------------------------+
> > +| Name           | Value                  |
> > ++================+========================+
> > +| Device ID      | <ID>                   |
> > ++----------------+------------------------+
> > +| Message ID     | <ID>                   |
> > ++----------------+------------------------+
> > +| Command        | 12                     |
> > ++----------------+------------------------+
> > +| Message size   | 24                     |
> > ++----------------+------------------------+
> > +| Flags          | Reply bit set in reply |
> > ++----------------+------------------------+
> > +| Interrupt info | <interrupt>            |
> > ++----------------+------------------------+
> > +
> > +This request is sent from the server to the client to signal the device has
> > +raised an interrupt.
> > +
> > +Interrupt info format
> > +^^^^^^^^^^^^^^^^^^^^^
> > +
> > ++----------+--------+------+
> > +| Name     | Offset | Size |
> > ++==========+========+======+
> > +| Index    | 16     | 4    |
> > ++----------+--------+------+
> > +| Subindex | 20     | 4    |
> > ++----------+--------+------+
> > +
> > +* Index is the interrupt index; it is the same value used in
> VFIO_USER_SET_IRQS.
> > +* Subindex is relative to the index, e.g., the vector number used in PCI
> MSI/X
> > +  type interrupts.
> > +
> > +VFIO_USER_DEVICE_RESET
> > +----------------------
> > +
> > +Message format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name         | Value                  |
> > ++==============+========================+
> > +| Device ID    | <ID>                   |
> > ++--------------+------------------------+
> > +| Message ID   | <ID>                   |
> > ++--------------+------------------------+
> > +| Command      | 13                     |
> > ++--------------+------------------------+
> > +| Message size | 16                     |
> > ++--------------+------------------------+
> > +| Flags        | Reply bit set in reply |
> > ++--------------+------------------------+
> > +
> > +This request is sent from the client to the server to reset the device.
> > +
> > +Appendices
> > +==========
> > +
> > +Unused VFIO ioctl() commands
> > +----------------------------
> > +
> > +The following commands must be handled by the client and not sent to
> the server:
> > +
> > +* VFIO_GET_API_VERSION
> > +* VFIO_CHECK_EXTENSION
> > +* VFIO_SET_IOMMU
> > +* VFIO_GROUP_GET_STATUS
> > +* VFIO_GROUP_SET_CONTAINER
> > +* VFIO_GROUP_UNSET_CONTAINER
> > +* VFIO_GROUP_GET_DEVICE_FD
> > +* VFIO_IOMMU_GET_INFO
> > +
> > +However, once support for live migration for VFIO devices is finalized
> some
> > +of the above commands might have to be handled by the client. This will
> be
> > +addressed in a future protocol version.
> > +
> > +Live Migration
> > +--------------
> > +Currently live migration is not supported for devices passed through via
> VFIO,
> > +therefore it is not supported for VFIO-over-socket, either. This is being
> > +actively worked on in the "Add migration support for VFIO devices" (v25)
> patch
> > +series.
> > +
> > +VFIO groups and containers
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +The current VFIO implementation includes group and container idioms
> that
> > +describe how a device relates to the host IOMMU. In the VFIO over
> socket
> > +implementation, the IOMMU is implemented in SW by the client, and isn't
> visible
> > +to the server. The simplest idea is for the client is to put each device into
> 
> 
> s/is for/for

OK.

> 
> > +its own group and container.


John and Thanos

This patch introduces the vfio-user protocol specification (formerly
known as VFIO-over-socket), which is designed to allow devices to be
emulated outside QEMU, in a separate process. vfio-user reuses the
existing VFIO defines, structs and concepts.

It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"

Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>

---

Changed since v1:
  * fix coding style issues
  * update MAINTAINERS for VFIO-over-socket
  * add vfio-over-socket to ToC

Changed since v2:
  * fix whitespace

Changed since v3:
  * rename protocol to vfio-user
  * add table of contents
  * fix Unicode problems
  * fix typos and various reStructuredText issues
  * various stylistic improvements
  * add backend program conventions
  * rewrite part of intro, drop QEMU-specific stuff
  * drop QEMU-specific paragraph about implementation
  * explain that passing of FDs isn't necessary
  * minor improvements in the VFIO section
  * various text substitutions for the sake of consistency
  * drop paragraph about client and server, already explained in intro
  * drop device ID
  * drop type from version
  * elaborate on request concurrency
  * convert some inessential paragraphs into notes
  * explain why some existing VFIO defines cannot be reused
  * explain how to make changes to the protocol
  * improve text of DMA map
  * reword comment about existing VFIO commands
  * add reference to Version section
  * reset device on disconnection
  * reword live migration section
  * replace sys/vfio.h with linux/vfio.h
  * drop reference to iovec
  * use argz the same way it is used in VFIO
  * add type field in header for clarity

Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
---
 MAINTAINERS              |    6 +
 docs/devel/index.rst     |    1 +
 docs/devel/vfio-user.rst | 1191 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1198 insertions(+)
 create mode 100644 docs/devel/vfio-user.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 030faf0..a7f4b8f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1732,6 +1732,12 @@ F: hw/vfio/ap.c
 F: docs/system/s390x/vfio-ap.rst
 L: qemu-s390x@nongnu.org
 
+vfio-user
+M: John G Johnson <john.g.johnson@oracle.com>
+M: Thanos Makatos <thanos.makatos@nutanix.com>
+S: Supported
+F: docs/devel/vfio-user.rst
+
 vhost
 M: Michael S. Tsirkin <mst@redhat.com>
 S: Supported
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index ae6eac7..e6a89c2 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -30,3 +30,4 @@ Contents:
    reset
    s390-dasd-ipl
    clocks
+   vfio-user
diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst
new file mode 100644
index 0000000..0217270
--- /dev/null
+++ b/docs/devel/vfio-user.rst
@@ -0,0 +1,1191 @@
+********************************
+vfio-user Protocol Specification
+********************************
+
+------------
+Version_ 0.1
+------------
+
+.. contents:: Table of Contents
+
+Introduction
+============
+vfio-user is a protocol that allows a device to be emulated in a separate
+process outside of a Virtual Machine Monitor (VMM). vfio-user devices consist
+of a generic VFIO device type, living inside the VMM, which we call the client,
+and the core device implementation, living outside the VMM, which we call the
+server.
+
+The `Linux VFIO ioctl interface <https://www.kernel.org/doc/html/latest/driver-api/vfio.html>`_
+been chosen as the base for this protocol for the following reasons:
+
+1) It is a mature and stable API, backed by an extensively used framework.
+2) The existing VFIO client implementation in QEMU (qemu/hw/vfio/) can be
+   largely reused.
+
+.. Note::
+   In a proof of concept implementation it has been demonstrated that using VFIO
+   over a UNIX domain socket is a viable option. vfio-user is designed with
+   QEMU in mind, however it could be used by other client applications. The
+   vfio-user protocol does not require that QEMU's VFIO client  implementation
+   is used in QEMU.
+
+None of the VFIO kernel modules are required for supporting the protocol,
+neither in the client nor the server, only the source header files are used.
+
+The main idea is to allow a virtual device to function in a separate process in
+the same host over a UNIX domain socket. A UNIX domain socket (AF_UNIX) is
+chosen because file descriptors can be trivially sent over it, which in turn
+allows:
+
+* Sharing of client memory for DMA with the server.
+* Sharing of server memory with the client for fast MMIO.
+* Efficient sharing of eventfd's for triggering interrupts.
+
+Other socket types could be used which allow the server to run in a separate
+guest in the same host (AF_VSOCK) or remotely (AF_INET). Theoretically the
+underlying transport does not necessarily have to be a socket, however we do
+not examine such alternatives. In this protocol version we focus on using a
+UNIX domain socket and introduce basic support for the other two types of
+sockets without considering performance implications.
+
+While passing of file descriptors is desirable for performance reasons, it is
+not necessary neither for the client nor for the server to support it in order
+to implement the protocol. There is always an in-band, message-passing fall
+back mechanism.
+
+VFIO
+====
+VFIO is a framework that allows a physical device to be securely passed through
+to a user space process; the device-specific kernel driver does not drive the
+device at all.  Typically, the user space process is a VMM and the device is
+passed through to it in order to achieve high performance. VFIO provides an API
+and the required functionality in the kernel. QEMU has adopted VFIO to allow a
+guest to directly access physical devices, instead of emulating them in
+software.
+
+vfio-user reuses the core VFIO concepts defined in its API, but implements them
+as messages to be sent over a socket. It does not change the kernel-based VFIO
+in any way, in fact none of the VFIO kernel modules need to be loaded to use
+vfio-user. It is also possible for the client to concurrently use the current
+kernel-based VFIO for one device, and vfio-user for another device.
+
+VFIO Device Model
+-----------------
+A device under VFIO presents a standard interface to the user process. Many of
+the VFIO operations in the existing interface use the ioctl() system call, and
+references to the existing interface are called the ioctl() implementation in
+this document.
+
+The following sections describe the set of messages that implement the VFIO
+interface over a socket. In many cases, the messages are direct translations of
+data structures used in the ioctl() implementation. Messages derived from
+ioctl()s will have a name derived from the ioctl() command name.  E.g., the
+VFIO_GET_INFO ioctl() command becomes a VFIO_USER_GET_INFO message.  The
+purpose of this reuse is to share as much code as feasible with the ioctl()
+implementation.
+
+Connection Initiation
+^^^^^^^^^^^^^^^^^^^^^
+After the client connects to the server, the initial server message is
+VFIO_USER_VERSION to propose a protocol version and set of capabilities to
+apply to the session. The client replies with a compatible version and set of
+capabilities it supports, or closes the connection if it cannot support the
+advertised version.
+
+DMA Memory Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+The client uses VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP messages to inform
+the server of the valid DMA ranges that the server can access on behalf
+of a device. DMA memory may be accessed by the server via VFIO_USER_DMA_READ
+and VFIO_USER_DMA_WRITE messages over the socket.
+
+An optimization for server access to client memory is for the client to provide
+file descriptors the server can mmap() to directly access client memory. Note
+that mmap() privileges cannot be revoked by the client, therefore file
+descriptors should only be exported in environments where the client trusts the
+server not to corrupt guest memory.
+
+Device Information
+^^^^^^^^^^^^^^^^^^
+The client uses a VFIO_USER_DEVICE_GET_INFO message to query the server for
+information about the device. This information includes:
+
+* The device type and capabilities,
+* the number of device regions, and
+* the device presents to the client the number of interrupt types the device
+  supports.
+
+Region Information
+^^^^^^^^^^^^^^^^^^
+The client uses VFIO_USER_DEVICE_GET_REGION_INFO messages to query the server
+for information about the device's memory regions. This information describes:
+
+* Read and write permissions, whether it can be memory mapped, and whether it
+  supports additional capabilities.
+* Region index, size, and offset.
+
+When a region can be mapped by the client, the server provides a file
+descriptor which the client can mmap(). The server is responsible for polling
+for client updates to memory mapped regions.
+
+Region Capabilities
+"""""""""""""""""""
+Some regions have additional capabilities that cannot be described adequately
+by the region info data structure. These capabilities are returned in the
+region info reply in a list similar to PCI capabilities in a PCI device's
+configuration space.
+
+Sparse Regions
+""""""""""""""
+A region can be memory-mappable in whole or in part. When only a subset of a
+region can be mapped by the client, a VFIO_REGION_INFO_CAP_SPARSE_MMAP
+capability is included in the region info reply. This capability describes
+which portions can be mapped by the client.
+
+.. Note::
+   For example, in a virtual NVMe controller, sparse regions can be used so
+   that accesses to the NVMe registers (found in the beginning of BAR0) are
+   trapped (an infrequent event), while allowing direct access to the doorbells
+   (an extremely frequent event as every I/O submission requires a write to
+   BAR0), found right after the NVMe registers in BAR0.
+
+Interrupts
+^^^^^^^^^^
+The client uses VFIO_USER_DEVICE_GET_IRQ_INFO messages to query the server for
+the device's interrupt types. The interrupt types are specific to the bus the
+device is attached to, and the client is expected to know the capabilities of
+each interrupt type. The server can signal an interrupt either with
+VFIO_USER_VM_INTERRUPT messages over the socket, or can directly inject
+interrupts into the guest via an event file descriptor. The client configures
+how the server signals an interrupt with VFIO_USER_SET_IRQS messages.
+
+Device Read and Write
+^^^^^^^^^^^^^^^^^^^^^
+When the guest executes load or store operations to device memory, the client
+forwards these operations to the server with VFIO_USER_REGION_READ or
+VFIO_USER_REGION_WRITE messages. The server will reply with data from the
+device on read operations or an acknowledgement on write operations.
+
+DMA
+^^^
+When a device performs DMA accesses to guest memory, the server will forward
+them to the client with VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages.
+These messages can only be used to access guest memory the client has
+configured into the server.
+
+Protocol Specification
+======================
+To distinguish from the base VFIO symbols, all vfio-user symbols are prefixed
+with vfio_user or VFIO_USER. In revision 0.1, all data is in the little-endian
+format, although this may be relaxed in future revision in cases where the
+client and server are both big-endian. The messages are formatted for seamless
+reuse of the native VFIO structs.
+
+Socket
+------
+
+A server can serve:
+
+1) one or more clients, and/or
+2) one or more virtual devices, belonging to one or more clients.
+
+The current protocol specification requires a dedicated socket per
+client/server connection. It is a server-side implementation detail whether a
+single server handles multiple virtual devices from the same or multiple
+clients. The location of the socket is implementation-specific. Multiplexing
+clients, devices, and servers over the same socket is not supported in this
+version of the protocol.
+
+Authentication
+--------------
+For AF_UNIX, we rely on OS mandatory access controls on the socket files,
+therefore it is up to the management layer to set up the socket as required.
+Socket types than span guests or hosts will require a proper authentication
+mechanism. Defining that mechanism is deferred to a future version of the
+protocol.
+
+Command Concurrency
+-------------------
+A client may pipeline multiple commands without waiting for previous command
+replies.  The server will process commands in the order they are received.
+A consequence of this is if a client issues a command with the *No_reply* bit,
+then subseqently issues a command without *No_reply*, the older command will
+have been processed before the reply to the younger command is sent by the
+server.  The client must be aware of the device's capability to process concurrent
+commands if pipelining is used.  For example, pipelining allows multiple client
+threads to concurently access device memory; the client must ensure these acceses
+obey device semantics.
+
+An example is a frame buffer device, where the device may allow concurrent access
+to different areas of video memory, but may have indeterminate behavior if concurrent
+acceses are performed to command or status registers.
+
+Socket Disconnection Behavior
+-----------------------------
+The server and the client can disconnect from each other, either intentionally
+or unexpectedly. Both the client and the server need to know how to handle such
+events.
+
+Server Disconnection
+^^^^^^^^^^^^^^^^^^^^
+A server disconnecting from the client may indicate that:
+
+1) A virtual device has been restarted, either intentionally (e.g. because of a
+   device update) or unintentionally (e.g. because of a crash).
+2) A virtual device has been shut down with no intention to be restarted.
+
+It is impossible for the client to know whether or not a failure is
+intermittent or innocuous and should be retried, therefore the client should
+reset the VFIO device when it detects the socket has been disconnected.
+Error recovery will be driven by the guest's device error handling
+behavior.
+
+Client Disconnection
+^^^^^^^^^^^^^^^^^^^^
+The client disconnecting from the server primarily means that the client
+has exited. Currently, this means that the guest is shut down so the device is
+no longer needed therefore the server can automatically exit. However, there
+can be cases where a client disconnection should not result in a server exit:
+
+1) A single server serving multiple clients.
+2) A multi-process QEMU upgrading itself step by step, which is not yet
+   implemented.
+
+Therefore in order for the protocol to be forward compatible the server should
+take no action when the client disconnects. If anything happens to the client
+the control stack will know about it and can clean up resources
+accordingly.
+
+Live Migration
+^^^^^^^^^^^^^^
+A future version of the protocol will support client live migration.  This action
+will require the socket to be quiesced before it is disconnected,  This mechanism
+will be defined when live migration support is added.
+
+Request Retry and Response Timeout
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+A failed command is a command that has been successfully sent and has been
+responded to with an error code. Failure to send the command in the first place
+(e.g. because the socket is disconnected) is a different type of error examined
+earlier in the disconnect section.
+
+.. Note::
+   QEMU's VFIO retries certain operations if they fail. While this makes sense
+   for real HW, we don't know for sure whether it makes sense for virtual
+   devices.
+
+Defining a retry and timeout scheme is deferred to a future version of the
+protocol.
+
+.. _Commands:
+
+Commands
+--------
+The following table lists the VFIO message command IDs, and whether the
+message command is sent from the client or the server.
+
++----------------------------------+---------+-------------------+
+| Name                             | Command | Request Direction |
++==================================+=========+===================+
+| VFIO_USER_VERSION                | 1       | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_MAP                | 2       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_UNMAP              | 3       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_INFO        | 4       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_REGION_INFO | 5       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_IRQ_INFO    | 6       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_SET_IRQS        | 7       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_REGION_READ            | 8       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_REGION_WRITE           | 9       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_READ               | 10      | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_WRITE              | 11      | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_VM_INTERRUPT           | 12      | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_RESET           | 13      | client -> server  |
++----------------------------------+---------+-------------------+
+
+.. Note:: Some VFIO defines cannot be reused since their values are
+   architecture-specific (e.g. VFIO_IOMMU_MAP_DMA).
+
+Header
+------
+All messages, both command messages and reply messages, are preceded by a
+header that contains basic information about the message. The header is
+followed by message-specific data described in the sections below.
+
++----------------+--------+-------------+
+| Name           | Offset | Size        |
++================+========+=============+
+| Message ID     | 0      | 2           |
++----------------+--------+-------------+
+| Command        | 2      | 2           |
++----------------+--------+-------------+
+| Message size   | 4      | 4           |
++----------------+--------+-------------+
+| Flags          | 8      | 4           |
++----------------+--------+-------------+
+|                | +-----+------------+ |
+|                | | Bit | Definition | |
+|                | +=====+============+ |
+|                | | 0-3 | Type       | |
+|                | +-----+------------+ |
+|                | | 4   | No_reply   | |
+|                | +-----+------------+ |
+|                | | 5   | Error      | |
+|                | +-----+------------+ |
++----------------+--------+-------------+
+| Error          | 12     | 4           |
++----------------+--------+-------------+
+| <message data> | 16     | variable    |
++----------------+--------+-------------+
+
+* *Message ID* identifies the message, and is echoed in the command's reply message.
+* *Command* specifies the command to be executed, listed in Commands_.
+* *Message size* contains the size of the entire message, including the header.
+* *Flags* contains attributes of the message:
+
+  * The *Type* bits indicate the message type.
+
+    *  *Command* (value 0x0) indicates a command message.
+    *  *Reply* (value 0x1) indicates a reply message acknowledging a previous
+       command with the same message ID.
+  * *No_reply* in a command message indicates that no reply is needed for this command.
+    This is commonly used when multiple commands are sent, and only the last needs
+    acknowledgement.
+  * *Error* in a reply message indicates the command being acknowledged had
+    an error. In this case, the *Error* field will be valid.
+
+* *Error* in a reply message is a UNIX errno value. It is reserved in a command message.
+
+Each command message in Commands_ must be replied to with a reply message, unless the
+message sets the *No_Reply* bit.  The reply consists of the header with the *Reply*
+bit set, plus any additional data.
+
+VFIO_USER_VERSION
+-----------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 1                      |
++--------------+------------------------+
+| Message size | 16 + version length    |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Version      | JSON byte array        |
++--------------+------------------------+
+
+This is the initial message sent by the server after the socket connection is
+established. The version is in JSON format, and the following objects must be
+included:
+
++--------------+--------+---------------------------------------------------+
+| Name         | Type   | Description                                       |
++==============+========+===================================================+
+| version      | object | ``{"major": <number>, "minor": <number>}``        |
+|              |        |                                                   |
+|              |        | Version supported by the sender, e.g. "0.1".      |
++--------------+--------+---------------------------------------------------+
+| capabilities | array  | Reserved. Can be omitted for v0.1, otherwise must |
+|              |        | be empty.                                         |
++--------------+--------+---------------------------------------------------+
+
+.. _Version:
+
+Versioning and Feature Support
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Upon accepting a connection, the server must send a VFIO_USER_VERSION message
+proposing a protocol version and a set of capabilities. The client compares
+these with the versions and capabilities it supports and sends a
+VFIO_USER_VERSION reply according to the following rules.
+
+* The major version in the reply must be the same as proposed. If the client
+  does not support the proposed major, it closes the connection.
+* The minor version in the reply must be equal to or less than the minor
+  version proposed.
+* The capability list must be a subset of those proposed. If the client
+  requires a capability the server did not include, it closes the connection.
+
+The protocol major version will only change when incompatible protocol changes
+are made, such as changing the message format. The minor version may change
+when compatible changes are made, such as adding new messages or capabilities,
+Both the client and server must support all minor versions less than the
+maximum minor version it supports. E.g., an implementation that supports
+version 1.3 must also support 1.0 through 1.2.
+
+When making a change to this specification, the protocol version number must
+be included in the form "added in version X.Y"
+
+
+VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP
+-----------------------------------------
+
+Message Format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | MAP=2, UNMAP=3         |
++--------------+------------------------+
+| Message size | 16 + table size        |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Table        | array of table entries |
++--------------+------------------------+
+
+This command message is sent by the client to the server to inform it of the
+memory regions the server can access. It must be sent before the server can
+perform any DMA to the client. It is normally sent directly after the version
+handshake is completed, but may also occur when memory is added to or
+subtracted from the client, or if the client uses a vIOMMU. If the client does
+not expect the server to perform DMA then it does not need to send to the
+server VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP commands. If the server does
+not need to perform DMA the then it can ignore such commands but it must still
+reply to them. The table is an array of the following structure.  This
+structure is 32 bytes in size, so the message size is:
+16 + (# of table entries * 32).
+
+Table entry format
+^^^^^^^^^^^^^^^^^^
+
++-------------+--------+-------------+
+| Name        | Offset | Size        |
++=============+========+=============+
+| Address     | 0      | 8           |
++-------------+--------+-------------+
+| Size        | 8      | 8           |
++-------------+--------+-------------+
+| Offset      | 16     | 8           |
++-------------+--------+-------------+
+| Protections | 24     | 4           |
++-------------+--------+-------------+
+| Flags       | 28     | 4           |
++-------------+--------+-------------+
+|             | +-----+------------+ |
+|             | | Bit | Definition | |
+|             | +=====+============+ |
+|             | | 0   | Mappable   | |
+|             | +-----+------------+ |
++-------------+--------+-------------+
+
+* *Address* is the base DMA address of the region.
+* *Size* is the size of the region.
+* *Offset* is the file offset of the region with respect to the associated file
+  descriptor.
+* *Protections* are the region's protection attributes as encoded in
+  ``<sys/mman.h>``.
+* *Flags* contain the following region attributes:
+
+  * *Mappable* indicates that the region can be mapped via the mmap() system call
+    using the file descriptor provided in the message meta-data.
+
+VFIO_USER_DMA_MAP
+"""""""""""""""""
+If a DMA region being added can be directly mapped by the server, an array of
+file descriptors must be sent as part of the message meta-data. Each region
+entry must have a corresponding file descriptor. On AF_UNIX sockets, the file
+descriptors must be passed as SCM_RIGHTS type ancillary data. Otherwise, if a
+DMA region cannot be directly mapped by the server, it can be accessed by the
+server using VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages, explained in
+`Read and Write Operations`_. A command to map over an existing region must be
+failed by the server with ``EEXIST`` set in error field in the reply.
+
+VFIO_USER_DMA_UNMAP
+"""""""""""""""""""
+Upon receiving a VFIO_USER_DMA_UNMAP command, if the file descriptor is mapped
+then the server must release all references to that DMA region before replying,
+which includes potentially in flight DMA transactions. Removing a portion of a
+DMA region is possible.
+
+VFIO_USER_DEVICE_GET_INFO
+-------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+----------------------------+
+| Name         | Value                      |
++==============+============================+
+| Message ID   | <ID>                       |
++--------------+----------------------------+
+| Command      | 4                          |
++--------------+----------------------------+
+| Message size | 16 in command, 32 in reply |
++--------------+----------------------------+
+| Flags        | Reply bit set in reply     |
++--------------+----------------------------+
+| Error        | 0/errno                    |
++--------------+----------------------------+
+| Device info  | VFIO device info           |
++--------------+----------------------------+
+
+This command message is sent by the client to the server to query for basic
+information about the device. Only the message header is needed in the command
+message.  The VFIO device info structure is defined in ``<linux/vfio.h>``
+(``struct vfio_device_info``).
+
+VFIO device info format
+^^^^^^^^^^^^^^^^^^^^^^^
+
++-------------+--------+--------------------------+
+| Name        | Offset | Size                     |
++=============+========+==========================+
+| argsz       | 16     | 4                        |
++-------------+--------+--------------------------+
+| flags       | 20     | 4                        |
++-------------+--------+--------------------------+
+|             | +-----+-------------------------+ |
+|             | | Bit | Definition              | |
+|             | +=====+=========================+ |
+|             | | 0   | VFIO_DEVICE_FLAGS_RESET | |
+|             | +-----+-------------------------+ |
+|             | | 1   | VFIO_DEVICE_FLAGS_PCI   | |
+|             | +-----+-------------------------+ |
++-------------+--------+--------------------------+
+| num_regions | 24     | 4                        |
++-------------+--------+--------------------------+
+| num_irqs    | 28     | 4                        |
++-------------+--------+--------------------------+
+
+* *argsz* is the size of the VFIO device info structure.
+* *flags* contains the following device attributes.
+
+  * VFIO_DEVICE_FLAGS_RESET indicates that the device supports the
+    VFIO_USER_DEVICE_RESET message.
+  * VFIO_DEVICE_FLAGS_PCI indicates that the device is a PCI device.
+
+* *num_regions* is the number of memory regions that the device exposes.
+* *num_irqs* is the number of distinct interrupt types that the device supports.
+
+This version of the protocol only supports PCI devices. Additional devices may
+be supported in future versions.
+
+VFIO_USER_DEVICE_GET_REGION_INFO
+--------------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 5                      |
++--------------+------------------------+
+| Message size | 48 + any caps          |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Region info  | VFIO region info       |
++--------------+------------------------+
+
+This command message is sent by the client to the server to query for
+information about device memory regions. The VFIO region info structure is
+defined in ``<linux/vfio.h>`` (``struct vfio_region_info``). Since the client
+does not know the size of the capabilities, the size of the reply it should
+expect is 48 plus any capabilities whose size is indicated in the size field of
+the reply header.
+
+VFIO region info format
+^^^^^^^^^^^^^^^^^^^^^^^
+
++------------+--------+------------------------------+
+| Name       | Offset | Size                         |
++============+========+==============================+
+| argsz      | 16     | 4                            |
++------------+--------+------------------------------+
+| flags      | 20     | 4                            |
++------------+--------+------------------------------+
+|            | +-----+-----------------------------+ |
+|            | | Bit | Definition                  | |
+|            | +=====+=============================+ |
+|            | | 0   | VFIO_REGION_INFO_FLAG_READ  | |
+|            | +-----+-----------------------------+ |
+|            | | 1   | VFIO_REGION_INFO_FLAG_WRITE | |
+|            | +-----+-----------------------------+ |
+|            | | 2   | VFIO_REGION_INFO_FLAG_MMAP  | |
+|            | +-----+-----------------------------+ |
+|            | | 3   | VFIO_REGION_INFO_FLAG_CAPS  | |
+|            | +-----+-----------------------------+ |
++------------+--------+------------------------------+
+| index      | 24     | 4                            |
++------------+--------+------------------------------+
+| cap_offset | 28     | 4                            |
++------------+--------+------------------------------+
+| size       | 32     | 8                            |
++------------+--------+------------------------------+
+| offset     | 40     | 8                            |
++------------+--------+------------------------------+
+
+* *argsz* is the size of the VFIO region info structure plus the
+  size of any region capabilities returned.
+* *flags* are attributes of the region:
+
+  * *VFIO_REGION_INFO_FLAG_READ* allows client read access to the region.
+  * *VFIO_REGION_INFO_FLAG_WRITE* allows client write access to the region.
+  * *VFIO_REGION_INFO_FLAG_MMAP* specifies the client can mmap() the region.
+    When this flag is set, the reply will include a file descriptor in its
+    meta-data. On AF_UNIX sockets, the file descriptors will be passed as
+    SCM_RIGHTS type ancillary data.
+  * *VFIO_REGION_INFO_FLAG_CAPS* indicates additional capabilities found in the
+    reply.
+
+* *index* is the index of memory region being queried, it is the only field
+  that is required to be set in the command message.
+* *cap_offset* describes where additional region capabilities can be found.
+  cap_offset is relative to the beginning of the VFIO region info structure.
+  The data structure it points is a VFIO cap header defined in
+  ``<linux/vfio.h>``.
+* *size* is the size of the region.
+* *offset* is the offset given to the mmap() system call for regions with the
+  MMAP attribute. It is also used as the base offset when mapping a VFIO
+  sparse mmap area, described below.
+
+VFIO Region capabilities
+^^^^^^^^^^^^^^^^^^^^^^^^
+The VFIO region information can also include a capabilities list. This list is
+similar to a PCI capability list - each entry has a common header that
+identifies a capability and where the next capability in the list can be found.
+The VFIO capability header format is defined in ``<linux/vfio.h>`` (``struct
+vfio_info_cap_header``).
+
+VFIO cap header format
+^^^^^^^^^^^^^^^^^^^^^^
+
++---------+--------+------+
+| Name    | Offset | Size |
++=========+========+======+
+| id      | 0      | 2    |
++---------+--------+------+
+| version | 2      | 2    |
++---------+--------+------+
+| next    | 4      | 4    |
++---------+--------+------+
+
+* *id* is the capability identity.
+* *version* is a capability-specific version number.
+* *next* specifies the offset of the next capability in the capability list. It
+  is relative to the beginning of the VFIO region info structure.
+
+VFIO sparse mmap
+^^^^^^^^^^^^^^^^
+
++------------------+----------------------------------+
+| Name             | Value                            |
++==================+==================================+
+| id               | VFIO_REGION_INFO_CAP_SPARSE_MMAP |
++------------------+----------------------------------+
+| version          | 0x1                              |
++------------------+----------------------------------+
+| next             | <next>                           |
++------------------+----------------------------------+
+| sparse mmap info | VFIO region info sparse mmap     |
++------------------+----------------------------------+
+
+The only capability supported in this version of the protocol is for sparse
+mmap. This capability is defined when only a subrange of the region supports
+direct access by the client via mmap(). The VFIO sparse mmap area is defined in
+``<linux/vfio.h>`` (``struct vfio_region_sparse_mmap_area``).
+
+VFIO region info cap sparse mmap
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++----------+--------+------+
+| Name     | Offset | Size |
++==========+========+======+
+| nr_areas | 0      | 4    |
++----------+--------+------+
+| reserved | 4      | 4    |
++----------+--------+------+
+| offset   | 8      | 8    |
++----------+--------+------+
+| size     | 16     | 9    |
++----------+--------+------+
+| ...      |        |      |
++----------+--------+------+
+
+* *nr_areas* is the number of sparse mmap areas in the region.
+* *offset* and size describe a single area that can be mapped by the client.
+  There will be nr_areas pairs of offset and size. The offset will be added to
+  the base offset given in the VFIO_USER_DEVICE_GET_REGION_INFO to form the
+  offset argument of the subsequent mmap() call.
+
+The VFIO sparse mmap area is defined in ``<linux/vfio.h>`` (``struct
+vfio_region_info_cap_sparse_mmap``).
+
+VFIO_USER_DEVICE_GET_IRQ_INFO
+-----------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 6                      |
++--------------+------------------------+
+| Message size | 32                     |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| IRQ info     | VFIO IRQ info          |
++--------------+------------------------+
+
+This command message is sent by the client to the server to query for
+information about device interrupt types. The VFIO IRQ info structure is
+defined in ``<linux/vfio.h>`` (``struct vfio_irq_info``).
+
+VFIO IRQ info format
+^^^^^^^^^^^^^^^^^^^^
+
++-------+--------+---------------------------+
+| Name  | Offset | Size                      |
++=======+========+===========================+
+| argsz | 16     | 4                         |
++-------+--------+---------------------------+
+| flags | 20     | 4                         |
++-------+--------+---------------------------+
+|       | +-----+--------------------------+ |
+|       | | Bit | Definition               | |
+|       | +=====+==========================+ |
+|       | | 0   | VFIO_IRQ_INFO_EVENTFD    | |
+|       | +-----+--------------------------+ |
+|       | | 1   | VFIO_IRQ_INFO_MASKABLE   | |
+|       | +-----+--------------------------+ |
+|       | | 2   | VFIO_IRQ_INFO_AUTOMASKED | |
+|       | +-----+--------------------------+ |
+|       | | 3   | VFIO_IRQ_INFO_NORESIZE   | |
+|       | +-----+--------------------------+ |
++-------+--------+---------------------------+
+| index | 24     | 4                         |
++-------+--------+---------------------------+
+| count | 28     | 4                         |
++-------+--------+---------------------------+
+
+* *argsz* is the size of the VFIO IRQ info structure.
+* *flags* defines IRQ attributes:
+
+  * *VFIO_IRQ_INFO_EVENTFD* indicates the IRQ type can support server eventfd
+    signalling.
+  * *VFIO_IRQ_INFO_MASKABLE* indicates that the IRQ type supports the MASK and
+    UNMASK actions in a VFIO_USER_DEVICE_SET_IRQS message.
+  * *VFIO_IRQ_INFO_AUTOMASKED* indicates the IRQ type masks itself after being
+    triggered, and the client must send an UNMASK action to receive new
+    interrupts.
+  * *VFIO_IRQ_INFO_NORESIZE* indicates VFIO_USER_SET_IRQS operations setup
+    interrupts as a set, and new sub-indexes cannot be enabled without disabling
+    the entire type.
+
+* index is the index of IRQ type being queried, it is the only field that is
+  required to be set in the command message.
+* count describes the number of interrupts of the queried type.
+
+VFIO_USER_DEVICE_SET_IRQS
+-------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 7                      |
++--------------+------------------------+
+| Message size | 36 + any data          |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| IRQ set      | VFIO IRQ set           |
++--------------+------------------------+
+
+This command message is sent by the client to the server to set actions for
+device interrupt types. The VFIO IRQ set structure is defined in
+``<linux/vfio.h>`` (``struct vfio_irq_set``).
+
+VFIO IRQ set format
+^^^^^^^^^^^^^^^^^^^
+
++-------+--------+------------------------------+
+| Name  | Offset | Size                         |
++=======+========+==============================+
+| argsz | 16     | 4                            |
++-------+--------+------------------------------+
+| flags | 20     | 4                            |
++-------+--------+------------------------------+
+|       | +-----+-----------------------------+ |
+|       | | Bit | Definition                  | |
+|       | +=====+=============================+ |
+|       | | 0   | VFIO_IRQ_SET_DATA_NONE      | |
+|       | +-----+-----------------------------+ |
+|       | | 1   | VFIO_IRQ_SET_DATA_BOOL      | |
+|       | +-----+-----------------------------+ |
+|       | | 2   | VFIO_IRQ_SET_DATA_EVENTFD   | |
+|       | +-----+-----------------------------+ |
+|       | | 3   | VFIO_IRQ_SET_ACTION_MASK    | |
+|       | +-----+-----------------------------+ |
+|       | | 4   | VFIO_IRQ_SET_ACTION_UNMASK  | |
+|       | +-----+-----------------------------+ |
+|       | | 5   | VFIO_IRQ_SET_ACTION_TRIGGER | |
+|       | +-----+-----------------------------+ |
++-------+--------+------------------------------+
+| index | 24     | 4                            |
++-------+--------+------------------------------+
+| start | 28     | 4                            |
++-------+--------+------------------------------+
+| count | 32     | 4                            |
++-------+--------+------------------------------+
+| data  | 36     | variable                     |
++-------+--------+------------------------------+
+
+* *argsz* is the size of the VFIO IRQ set structure, including any *data* field.
+* *flags* defines the action performed on the interrupt range. The DATA flags
+  describe the data field sent in the message; the ACTION flags describe the
+  action to be performed. The flags are mutually exclusive for both sets.
+
+  * *VFIO_IRQ_SET_DATA_NONE* indicates there is no data field in the command.
+    The action is performed unconditionally.
+  * *VFIO_IRQ_SET_DATA_BOOL* indicates the data field is an array of boolean
+    bytes. The action is performed if the corresponding boolean is true.
+  * *VFIO_IRQ_SET_DATA_EVENTFD* indicates an array of event file descriptors
+    was sent in the message meta-data. These descriptors will be signalled when
+    the action defined by the action flags occurs. In AF_UNIX sockets, the
+    descriptors are sent as SCM_RIGHTS type ancillary data.
+  * *VFIO_IRQ_SET_ACTION_MASK* indicates a masking event. It can be used with
+    VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to mask an interrupt, or
+    with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the guest masks
+    the interrupt.
+  * *VFIO_IRQ_SET_ACTION_UNMASK* indicates an unmasking event. It can be used
+    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to unmask an
+    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
+    guest unmasks the interrupt.
+  * *VFIO_IRQ_SET_ACTION_TRIGGER* indicates a triggering event. It can be used
+    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to trigger an
+    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
+    server triggers the interrupt.
+
+* *index* is the index of IRQ type being setup.
+* *start* is the start of the sub-index being set.
+* *count* describes the number of sub-indexes being set. As a special case, a
+  count of 0 with data flags of VFIO_IRQ_SET_DATA_NONE disables all interrupts
+  of the index.
+* *data* is an optional field included when the
+  VFIO_IRQ_SET_DATA_BOOL flag is present. It contains an array of booleans
+  that specify whether the action is to be performed on the corresponding
+  index. It's used when the action is only performed on a subset of the range
+  specified.
+
+Not all interrupt types support every combination of data and action flags.
+The client must know the capabilities of the device and IRQ index before it
+sends a VFIO_USER_DEVICE_SET_IRQ message.
+
+.. _Read and Write Operations:
+
+Read and Write Operations
+-------------------------
+
+Not all I/O operations between the client and server can be done via direct
+access of memory mapped with an mmap() call. In these cases, the client and
+server use messages sent over the socket. It is expected that these operations
+will have lower performance than direct access.
+
+The client can access server memory with VFIO_USER_REGION_READ and
+VFIO_USER_REGION_WRITE commands. These share a common data structure that
+appears after the message header.
+
+REGION Read/Write Data
+^^^^^^^^^^^^^^^^^^^^^^
+
++--------+--------+----------+
+| Name   | Offset | Size     |
++========+========+==========+
+| Offset | 16     | 8        |
++--------+--------+----------+
+| Region | 24     | 4        |
++--------+--------+----------+
+| Count  | 28     | 4        |
++--------+--------+----------+
+| Data   | 32     | variable |
++--------+--------+----------+
+
+* *Offset* into the region being accessed.
+* *Region* is the index of the region being accessed.
+* *Count* is the size of the data to be transferred.
+* *Data* is the data to be read or written.
+
+The server can access client memory with VFIO_USER_DMA_READ and
+VFIO_USER_DMA_WRITE messages. These also share a common data structure that
+appears after the message header.
+
+DMA Read/Write Data
+^^^^^^^^^^^^^^^^^^^
+
++---------+--------+----------+
+| Name    | Offset | Size     |
++=========+========+==========+
+| Address | 16     | 8        |
++---------+--------+----------+
+| Count   | 24     | 4        |
++---------+--------+----------+
+| Data    | 28     | variable |
++---------+--------+----------+
+
+* *Address* is the area of client memory being accessed. This address must have
+  been previously exported to the server with a VFIO_USER_DMA_MAP message.
+* *Count* is the size of the data to be transferred.
+* *Data* is the data to be read or written.
+
+VFIO_USER_REGION_READ
+---------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 8                      |
++--------------+------------------------+
+| Message size | 32 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Read info    | REGION read/write data |
++--------------+------------------------+
+
+This command message is sent from the client to the server to read from server
+memory.  In the command messages, there is no data, and the count is the amount
+of data to be read. The reply message must include the data read, and its count
+field is the amount of data read.
+
+VFIO_USER_REGION_WRITE
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 9                      |
++--------------+------------------------+
+| Message size | 32 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Write info   | REGION read/write data |
++--------------+------------------------+
+
+This command message is sent from the client to the server to write to server
+memory.  The command message must contain the data to be written, and its count
+field must contain the amount of write data. The count field in the reply
+message must be zero.
+
+VFIO_USER_DMA_READ
+------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 10                     |
++--------------+------------------------+
+| Message size | 28 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| DMA info     | DMA read/write data    |
++--------------+------------------------+
+
+This command message is sent from the server to the client to read from client
+memory.  In the command message, there is no data, and the count must will be
+the amount of data to be read. The reply message must include the data read,
+and its count field must be the amount of data read.
+
+VFIO_USER_DMA_WRITE
+-------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 11                     |
++--------------+------------------------+
+| Message size | 28 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| DMA info     | DMA read/write data    |
++--------------+------------------------+
+
+This command message is sent from the server to the client to write to server
+memory.  The command message must contain the data to be written, and its count
+field must contain the amount of write data. The count field in the reply
+message must be zero.
+
+VFIO_USER_VM_INTERRUPT
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++----------------+------------------------+
+| Name           | Value                  |
++================+========================+
+| Message ID     | <ID>                   |
++----------------+------------------------+
+| Command        | 12                     |
++----------------+------------------------+
+| Message size   | 24                     |
++----------------+------------------------+
+| Flags          | Reply bit set in reply |
++----------------+------------------------+
+| Error          | 0/errno                |
++----------------+------------------------+
+| Interrupt info | <interrupt>            |
++----------------+------------------------+
+
+This command message is sent from the server to the client to signal the device
+has raised an interrupt.
+
+Interrupt info format
+^^^^^^^^^^^^^^^^^^^^^
+
++-----------+--------+------+
+| Name      | Offset | Size |
++===========+========+======+
+| Index     | 16     | 4    |
++-----------+--------+------+
+| Sub-index | 20     | 4    |
++-----------+--------+------+
+
+* *Index* is the interrupt index; it is the same value used in
+  VFIO_USER_SET_IRQS.
+* *Sub-index* is relative to the index, e.g., the vector number used in PCI
+  MSI/X type interrupts.
+
+VFIO_USER_DEVICE_RESET
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 13                     |
++--------------+------------------------+
+| Message size | 16                     |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+
+This command message is sent from the client to the server to reset the device.
+
+Appendices
+==========
+
+Unused VFIO ioctl() commands
+----------------------------
+
+The following VFIO commands do not have an equivalent vfio-user command:
+
+* VFIO_GET_API_VERSION
+* VFIO_CHECK_EXTENSION
+* VFIO_SET_IOMMU
+* VFIO_GROUP_GET_STATUS
+* VFIO_GROUP_SET_CONTAINER
+* VFIO_GROUP_UNSET_CONTAINER
+* VFIO_GROUP_GET_DEVICE_FD
+* VFIO_IOMMU_GET_INFO
+
+However, once support for live migration for VFIO devices is finalized some
+of the above commands may have to be handled by the client in their
+corresponding vfio-user form. This will be addressed in a future protocol
+version.
+
+VFIO groups and containers
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current VFIO implementation includes group and container idioms that
+describe how a device relates to the host IOMMU. In the vfio-user
+implementation, the IOMMU is implemented in SW by the client, and is not
+visible to the server. The simplest idea would be that the client put each
+device into its own group and container.
+
+Backend Program Conventions
+---------------------------
+
+vfio-user backend program conventions are based on the vhost-user ones.
+
+* The backend program must not daemonize itself.
+* No assumptions must be made as to what access the backend program has on the
+  system.
+* File descriptors 0, 1 and 2 must exist, must have regular
+  stdin/stdout/stderr semantics, and can be redirected.
+* The backend program must honor the SIGTERM signal.
+* The backend program must accept the following commands line options:
+
+  * ``--socket-path=PATH``: path to UNIX domain socket,
+  * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible with
+    ``--socket-path``
+* The backend program must be accompanied with a JSON file stored under
+  ``/usr/share/vfio-user``.
-- 
2.5.5

On Tue, Sep 15, 2020 at 07:29:17AM -0700, Thanos Makatos wrote:
> This patch introduces the vfio-user protocol specification (formerly
> known as VFIO-over-socket), which is designed to allow devices to be
> emulated outside QEMU, in a separate process. vfio-user reuses the
> existing VFIO defines, structs and concepts.
> 
> It has been earlier discussed as an RFC in:
> "RFC: use VFIO over a UNIX domain socket to implement device offloading"
> 
> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>

The approach looks promising. It's hard to know what changes will be
required when this is implemented, so let's not worry about getting
every detail of the spec right.

Now that there is a spec to start from, the next step is patches
implementing --device vfio-user-pci,chardev=<chardev> in
hw/vfio-user/pci.c (mirroring hw/vfio/).

It should be accompanied by a test in tests/. PCI-level testing APIS for
BARs, configuration space, interrupts, etc are available in
tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
device backend interact with QEMU's vfio-user-pci implementation.

I think this spec can be merged in docs/devel/ now and marked as
"subject to change (not a stable public interface)".

After the details have been proven and any necessary changes have been
made the spec can be promoted to docs/interop/ as a stable public
interface. This gives the freedom to make changes discovered when
figuring out issues like disconnect/reconnect, live migration, etc that
can be hard to get right without a working implementation.

Does this approach sound good?

Also please let us know who is working on what so additional people can
get involved in areas that need work!

Stefan

On Thu, Sep 24, 2020 at 09:21:32AM +0100, Stefan Hajnoczi wrote:
> On Tue, Sep 15, 2020 at 07:29:17AM -0700, Thanos Makatos wrote:
> > This patch introduces the vfio-user protocol specification (formerly
> > known as VFIO-over-socket), which is designed to allow devices to be
> > emulated outside QEMU, in a separate process. vfio-user reuses the
> > existing VFIO defines, structs and concepts.
> > 
> > It has been earlier discussed as an RFC in:
> > "RFC: use VFIO over a UNIX domain socket to implement device offloading"
> > 
> > Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> > Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
> 
> The approach looks promising. It's hard to know what changes will be
> required when this is implemented, so let's not worry about getting
> every detail of the spec right.
> 
> Now that there is a spec to start from, the next step is patches
> implementing --device vfio-user-pci,chardev=<chardev> in
> hw/vfio-user/pci.c (mirroring hw/vfio/).
> 
> It should be accompanied by a test in tests/. PCI-level testing APIS for
> BARs, configuration space, interrupts, etc are available in
> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
> device backend interact with QEMU's vfio-user-pci implementation.
> 
> I think this spec can be merged in docs/devel/ now and marked as
> "subject to change (not a stable public interface)".
> 
> After the details have been proven and any necessary changes have been
> made the spec can be promoted to docs/interop/ as a stable public
> interface. This gives the freedom to make changes discovered when
> figuring out issues like disconnect/reconnect, live migration, etc that
> can be hard to get right without a working implementation.
> 
> Does this approach sound good?
> 
> Also please let us know who is working on what so additional people can
> get involved in areas that need work!
> 
> Stefan

Problem we discovered with e.g. vhost is once you ship a management
interface, people start using it immediately and it does not matter
that you never promised stability.

So I feel a good first step would be to limit this to only allow known
in-tree devices, started/destroyed automatically by qemu when device is
created.  This way lots of reconnect etc issues go away, and we don't
commit to a stable protocol until we have a decent handle on how things
will work in production.

-- 
MST

> -----Original Message-----
> From: Stefan Hajnoczi <stefanha@redhat.com>
> Sent: 24 September 2020 09:22
> To: Thanos Makatos <thanos.makatos@nutanix.com>
> Cc: qemu-devel@nongnu.org; Michael S. Tsirkin <mst@redhat.com>;
> alex.williamson@redhat.com; benjamin.walker@intel.com;
> elena.ufimtseva@oracle.com; jag.raman@oracle.com; Swapnil Ingle
> <swapnil.ingle@nutanix.com>; james.r.harris@intel.com;
> konrad.wilk@oracle.com; Raphael Norwitz <raphael.norwitz@nutanix.com>;
> marcandre.lureau@redhat.com; Kanth.Ghatraju@oracle.com; Felipe
> Franciosi <felipe@nutanix.com>; tina.zhang@intel.com;
> changpeng.liu@intel.com; dgilbert@redhat.com;
> tomassetti.andrea@gmail.com; yuvalkashtan@gmail.com;
> ismael@linux.com; xiuchun.lu@intel.com; John G Johnson
> <john.g.johnson@oracle.com>
> Subject: Re: [PATCH v4] introduce vfio-user protocol specification
> 
> On Tue, Sep 15, 2020 at 07:29:17AM -0700, Thanos Makatos wrote:
> > This patch introduces the vfio-user protocol specification (formerly
> > known as VFIO-over-socket), which is designed to allow devices to be
> > emulated outside QEMU, in a separate process. vfio-user reuses the
> > existing VFIO defines, structs and concepts.
> >
> > It has been earlier discussed as an RFC in:
> > "RFC: use VFIO over a UNIX domain socket to implement device offloading"
> >
> > Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> > Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
> 
> The approach looks promising. It's hard to know what changes will be
> required when this is implemented, so let's not worry about getting
> every detail of the spec right.
> 
> Now that there is a spec to start from, the next step is patches
> implementing --device vfio-user-pci,chardev=<chardev> in
> hw/vfio-user/pci.c (mirroring hw/vfio/).
> 
> It should be accompanied by a test in tests/. PCI-level testing APIS for
> BARs, configuration space, interrupts, etc are available in
> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
> device backend interact with QEMU's vfio-user-pci implementation.

We plan to use a libmuser-based backend for testing. This, I suppose, will
make libmuser a dependency of QEMU (either as a submodule or as a library),
which for now can be disabled in the default configuration. Is this acceptable?

> 
> I think this spec can be merged in docs/devel/ now and marked as
> "subject to change (not a stable public interface

Great!

> 
> After the details have been proven and any necessary changes have been
> made the spec can be promoted to docs/interop/ as a stable public
> interface. This gives the freedom to make changes discovered when
> figuring out issues like disconnect/reconnect, live migration, etc that
> can be hard to get right without a working implementation.
> 
> Does this approach sound good?

Yes.

> 
> Also please let us know who is working on what so additional people can
> get involved in areas that need work!

Swapnil and I will be working on libmuser and the test in QEMU, John and
the mp-qemu folks will be working on the patches for implementing
--device vfio-user-pci.

> 
> Stefan

On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
> > It should be accompanied by a test in tests/. PCI-level testing APIS for
> > BARs, configuration space, interrupts, etc are available in
> > tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
> > device backend interact with QEMU's vfio-user-pci implementation.
> 
> We plan to use a libmuser-based backend for testing. This, I suppose, will
> make libmuser a dependency of QEMU (either as a submodule or as a library),
> which for now can be disabled in the default configuration. Is this acceptable?

If there are no other dependencies and libmuser supports all host
operating systems that QEMU's -device vfio-user supports, then I think
it's a good idea to use libmuser for at least one in-tree test in QEMU.

> > Also please let us know who is working on what so additional people can
> > get involved in areas that need work!
> 
> Swapnil and I will be working on libmuser and the test in QEMU, John and
> the mp-qemu folks will be working on the patches for implementing
> --device vfio-user-pci.

Great!

John: Will mpqemu use libmuser to implement the remote PCI host
controller?

Stefan

> On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
>>> It should be accompanied by a test in tests/. PCI-level testing APIS for
>>> BARs, configuration space, interrupts, etc are available in
>>> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
>>> device backend interact with QEMU's vfio-user-pci implementation.
>> 
>> We plan to use a libmuser-based backend for testing. This, I suppose, will
>> make libmuser a dependency of QEMU (either as a submodule or as a library),
>> which for now can be disabled in the default configuration. Is this acceptable?
> 
> If there are no other dependencies and libmuser supports all host
> operating systems that QEMU's -device vfio-user supports, then I think
> it's a good idea to use libmuser for at least one in-tree test in QEMU.
> 
>>> Also please let us know who is working on what so additional people can
>>> get involved in areas that need work!
>> 
>> Swapnil and I will be working on libmuser and the test in QEMU, John and
>> the mp-qemu folks will be working on the patches for implementing
>> --device vfio-user-pci.
> 
> Great!
> 
> John: Will mpqemu use libmuser to implement the remote PCI host
> controller?
> 


	The vfio-user-pci plan is to use libmuser on the server side.

								JJ

On Tue, Sep 29, 2020 at 09:21:54AM -0700, John G Johnson wrote:
> > On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > 
> > On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
> >>> It should be accompanied by a test in tests/. PCI-level testing APIS for
> >>> BARs, configuration space, interrupts, etc are available in
> >>> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
> >>> device backend interact with QEMU's vfio-user-pci implementation.
> >> 
> >> We plan to use a libmuser-based backend for testing. This, I suppose, will
> >> make libmuser a dependency of QEMU (either as a submodule or as a library),
> >> which for now can be disabled in the default configuration. Is this acceptable?
> > 
> > If there are no other dependencies and libmuser supports all host
> > operating systems that QEMU's -device vfio-user supports, then I think
> > it's a good idea to use libmuser for at least one in-tree test in QEMU.
> > 
> >>> Also please let us know who is working on what so additional people can
> >>> get involved in areas that need work!
> >> 
> >> Swapnil and I will be working on libmuser and the test in QEMU, John and
> >> the mp-qemu folks will be working on the patches for implementing
> >> --device vfio-user-pci.
> > 
> > Great!
> > 
> > John: Will mpqemu use libmuser to implement the remote PCI host
> > controller?
> > 
> 
> 
> 	The vfio-user-pci plan is to use libmuser on the server side.

Okay. Using libmuser in tests seems like a good choice in that case.

We'll need to figure out the details of how to do it because the
traditional shared library dependency approach is not well-suited to
in-development code. It would involve shipping libmuser distro packages
so QEMU's build system can declare a library dependency (with details
provided in a pkg-config file).

Here are approaches that are better for in-development libraries:
1. Keep the libmuser code in qemu.git.
2. A copy of libmuser in qemu.git with changes being sent upstream
   (allows more flexibility in case QEMU-specific issues require
   experimentation).
3. Git submodules.

#1 if you're happy to use the QEMU development process for merging
libmuser code then it's easiest to officially host the code in qemu.git.
libmuser gets a subdirectory in the qemu.git tree and you (the
maintainers) send pull requests. A libmuser library build target
provides installable static and shared libraries so external
applications can link against libmuser too. The big advantage here is
that QEMU can instantly use the latest libmuser code changes.

#2 works best if the library is a small (just a few source files) with
no fancy build system requirements.

#3 is used in QEMU for several other components. Submodules are a pain
to sync (requires sending a qemu.git patch to move to a new commit ID),
so this isn't good for a dependency that moves quickly.

Stefan

> On Sep 30, 2020, at 3:24 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> On Tue, Sep 29, 2020 at 09:21:54AM -0700, John G Johnson wrote:
>>> On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>> 
>>> On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
>>>>> It should be accompanied by a test in tests/. PCI-level testing APIS for
>>>>> BARs, configuration space, interrupts, etc are available in
>>>>> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
>>>>> device backend interact with QEMU's vfio-user-pci implementation.
>>>> 
>>>> We plan to use a libmuser-based backend for testing. This, I suppose, will
>>>> make libmuser a dependency of QEMU (either as a submodule or as a library),
>>>> which for now can be disabled in the default configuration. Is this acceptable?
>>> 
>>> If there are no other dependencies and libmuser supports all host
>>> operating systems that QEMU's -device vfio-user supports, then I think
>>> it's a good idea to use libmuser for at least one in-tree test in QEMU.
>>> 
>>>>> Also please let us know who is working on what so additional people can
>>>>> get involved in areas that need work!
>>>> 
>>>> Swapnil and I will be working on libmuser and the test in QEMU, John and
>>>> the mp-qemu folks will be working on the patches for implementing
>>>> --device vfio-user-pci.
>>> 
>>> Great!
>>> 
>>> John: Will mpqemu use libmuser to implement the remote PCI host
>>> controller?
>>> 
>> 
>> 
>> 	The vfio-user-pci plan is to use libmuser on the server side.
> 
> Okay. Using libmuser in tests seems like a good choice in that case.
> 
> We'll need to figure out the details of how to do it because the
> traditional shared library dependency approach is not well-suited to
> in-development code. It would involve shipping libmuser distro packages
> so QEMU's build system can declare a library dependency (with details
> provided in a pkg-config file).
> 
> Here are approaches that are better for in-development libraries:
> 1. Keep the libmuser code in qemu.git.
> 2. A copy of libmuser in qemu.git with changes being sent upstream
>   (allows more flexibility in case QEMU-specific issues require
>   experimentation).
> 3. Git submodules.
> 
> #1 if you're happy to use the QEMU development process for merging
> libmuser code then it's easiest to officially host the code in qemu.git.
> libmuser gets a subdirectory in the qemu.git tree and you (the
> maintainers) send pull requests. A libmuser library build target
> provides installable static and shared libraries so external
> applications can link against libmuser too. The big advantage here is
> that QEMU can instantly use the latest libmuser code changes.

I think there's a couple of limitations here which we should keep in mind.

1. Does putting it in qemu.git precludes it being BSD-3?
There's been evidence of people using (or at least trying out) muser
from where it currently lives. That doesn't mean we can't move it, but
I'm wondering if it means we have to make it GPL.

2. What about other projects that need libmuser code?
What worries me more is projects like SPDK/DPDK wanting to link
against the library and having to clone the entire QEMU repo as a
submodule. That sounds a lot more expensive than option 3 and probably
have further complications if they aren't GPL.

> 
> #2 works best if the library is a small (just a few source files) with
> no fancy build system requirements.

The risk here is that they go out of sync. There's the same (or even
more) maintenance burden as point 3 below, with the added risk that
someone could patch the files and make cherry-picks non-trivial.

> 
> #3 is used in QEMU for several other components. Submodules are a pain
> to sync (requires sending a qemu.git patch to move to a new commit ID),
> so this isn't good for a dependency that moves quickly.

I argue this is no worse than option 2. It's what I think aligns best,
but let's keep weighing pros/cons and come to a conclusion together.
The list of maintainers for muser.git should be extended to include
more QEMU stakeholders and probably other projects that will use it
(as) heavily. The topic has been raised in SPDK's Slack team on
whether the client library should live in a repo of its own (eg.
libvfio-user.git). Given the reference implementation is in libmuser,
I still think muser.git is accurate (but can easily be persuaded
otherwise).

Cheers,
Felipe

> 
> Stefan

On Fri, Oct 02, 2020 at 10:14:23AM +0000, Felipe Franciosi wrote:
> 
> 
> > On Sep 30, 2020, at 3:24 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > 
> > On Tue, Sep 29, 2020 at 09:21:54AM -0700, John G Johnson wrote:
> >>> On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >>> 
> >>> On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
> >>>>> It should be accompanied by a test in tests/. PCI-level testing APIS for
> >>>>> BARs, configuration space, interrupts, etc are available in
> >>>>> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
> >>>>> device backend interact with QEMU's vfio-user-pci implementation.
> >>>> 
> >>>> We plan to use a libmuser-based backend for testing. This, I suppose, will
> >>>> make libmuser a dependency of QEMU (either as a submodule or as a library),
> >>>> which for now can be disabled in the default configuration. Is this acceptable?
> >>> 
> >>> If there are no other dependencies and libmuser supports all host
> >>> operating systems that QEMU's -device vfio-user supports, then I think
> >>> it's a good idea to use libmuser for at least one in-tree test in QEMU.
> >>> 
> >>>>> Also please let us know who is working on what so additional people can
> >>>>> get involved in areas that need work!
> >>>> 
> >>>> Swapnil and I will be working on libmuser and the test in QEMU, John and
> >>>> the mp-qemu folks will be working on the patches for implementing
> >>>> --device vfio-user-pci.
> >>> 
> >>> Great!
> >>> 
> >>> John: Will mpqemu use libmuser to implement the remote PCI host
> >>> controller?
> >>> 
> >> 
> >> 
> >> 	The vfio-user-pci plan is to use libmuser on the server side.
> > 
> > Okay. Using libmuser in tests seems like a good choice in that case.
> > 
> > We'll need to figure out the details of how to do it because the
> > traditional shared library dependency approach is not well-suited to
> > in-development code. It would involve shipping libmuser distro packages
> > so QEMU's build system can declare a library dependency (with details
> > provided in a pkg-config file).
> > 
> > Here are approaches that are better for in-development libraries:
> > 1. Keep the libmuser code in qemu.git.
> > 2. A copy of libmuser in qemu.git with changes being sent upstream
> >   (allows more flexibility in case QEMU-specific issues require
> >   experimentation).
> > 3. Git submodules.
> > 
> > #1 if you're happy to use the QEMU development process for merging
> > libmuser code then it's easiest to officially host the code in qemu.git.
> > libmuser gets a subdirectory in the qemu.git tree and you (the
> > maintainers) send pull requests. A libmuser library build target
> > provides installable static and shared libraries so external
> > applications can link against libmuser too. The big advantage here is
> > that QEMU can instantly use the latest libmuser code changes.
> 
> I think there's a couple of limitations here which we should keep in mind.
> 
> 1. Does putting it in qemu.git precludes it being BSD-3?
> There's been evidence of people using (or at least trying out) muser
> from where it currently lives. That doesn't mean we can't move it, but
> I'm wondering if it means we have to make it GPL.

The 3-clause BSD license is compatible with the GPL according to
Wikipedia:
https://en.wikipedia.org/wiki/BSD_licenses

> 2. What about other projects that need libmuser code?
> What worries me more is projects like SPDK/DPDK wanting to link
> against the library and having to clone the entire QEMU repo as a
> submodule. That sounds a lot more expensive than option 3 and probably
> have further complications if they aren't GPL.

In the early stages where the vfio-user protocol and library interfaces
might need changes it will be hard to use it from multiple applications
without compatibility issues. If SPDK/DPDK are communicating with QEMU
using a cutting-edge library then they probably need to build QEMU from
source anyway. ISTR they also maintain a QEMU fork? So maybe it's not a
big issue for them.

> > 
> > #2 works best if the library is a small (just a few source files) with
> > no fancy build system requirements.
> 
> The risk here is that they go out of sync. There's the same (or even
> more) maintenance burden as point 3 below, with the added risk that
> someone could patch the files and make cherry-picks non-trivial.
> 
> > 
> > #3 is used in QEMU for several other components. Submodules are a pain
> > to sync (requires sending a qemu.git patch to move to a new commit ID),
> > so this isn't good for a dependency that moves quickly.
> 
> I argue this is no worse than option 2. It's what I think aligns best,
> but let's keep weighing pros/cons and come to a conclusion together.
> The list of maintainers for muser.git should be extended to include
> more QEMU stakeholders and probably other projects that will use it
> (as) heavily. The topic has been raised in SPDK's Slack team on
> whether the client library should live in a repo of its own (eg.
> libvfio-user.git). Given the reference implementation is in libmuser,
> I still think muser.git is accurate (but can easily be persuaded
> otherwise).

Me too, no solution is perfect. My thoughts about developing it within
qemu.git for now is that this will make protocol and library interface
changes easy. It will also encourage applications (DPDK/SPDK) to build
against a matching QEMU so that there are no compatibility problems at
the protocol or library level while the code is still heavily under
development.

Stefan

Hi,

> On Oct 13, 2020, at 10:30 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> On Fri, Oct 02, 2020 at 10:14:23AM +0000, Felipe Franciosi wrote:
>> 
>> 
>>> On Sep 30, 2020, at 3:24 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>> 
>>> On Tue, Sep 29, 2020 at 09:21:54AM -0700, John G Johnson wrote:
>>>>> On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>>> 
>>>>> On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
>>>>>>> It should be accompanied by a test in tests/. PCI-level testing APIS for
>>>>>>> BARs, configuration space, interrupts, etc are available in
>>>>>>> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
>>>>>>> device backend interact with QEMU's vfio-user-pci implementation.
>>>>>> 
>>>>>> We plan to use a libmuser-based backend for testing. This, I suppose, will
>>>>>> make libmuser a dependency of QEMU (either as a submodule or as a library),
>>>>>> which for now can be disabled in the default configuration. Is this acceptable?
>>>>> 
>>>>> If there are no other dependencies and libmuser supports all host
>>>>> operating systems that QEMU's -device vfio-user supports, then I think
>>>>> it's a good idea to use libmuser for at least one in-tree test in QEMU.
>>>>> 
>>>>>>> Also please let us know who is working on what so additional people can
>>>>>>> get involved in areas that need work!
>>>>>> 
>>>>>> Swapnil and I will be working on libmuser and the test in QEMU, John and
>>>>>> the mp-qemu folks will be working on the patches for implementing
>>>>>> --device vfio-user-pci.
>>>>> 
>>>>> Great!
>>>>> 
>>>>> John: Will mpqemu use libmuser to implement the remote PCI host
>>>>> controller?
>>>>> 
>>>> 
>>>> 
>>>> 	The vfio-user-pci plan is to use libmuser on the server side.
>>> 
>>> Okay. Using libmuser in tests seems like a good choice in that case.
>>> 
>>> We'll need to figure out the details of how to do it because the
>>> traditional shared library dependency approach is not well-suited to
>>> in-development code. It would involve shipping libmuser distro packages
>>> so QEMU's build system can declare a library dependency (with details
>>> provided in a pkg-config file).
>>> 
>>> Here are approaches that are better for in-development libraries:
>>> 1. Keep the libmuser code in qemu.git.
>>> 2. A copy of libmuser in qemu.git with changes being sent upstream
>>>  (allows more flexibility in case QEMU-specific issues require
>>>  experimentation).
>>> 3. Git submodules.
>>> 
>>> #1 if you're happy to use the QEMU development process for merging
>>> libmuser code then it's easiest to officially host the code in qemu.git.
>>> libmuser gets a subdirectory in the qemu.git tree and you (the
>>> maintainers) send pull requests. A libmuser library build target
>>> provides installable static and shared libraries so external
>>> applications can link against libmuser too. The big advantage here is
>>> that QEMU can instantly use the latest libmuser code changes.
>> 
>> I think there's a couple of limitations here which we should keep in mind.
>> 
>> 1. Does putting it in qemu.git precludes it being BSD-3?
>> There's been evidence of people using (or at least trying out) muser
>> from where it currently lives. That doesn't mean we can't move it, but
>> I'm wondering if it means we have to make it GPL.
> 
> The 3-clause BSD license is compatible with the GPL according to
> Wikipedia:
> https://en.wikipedia.org/wiki/BSD_licenses

Ah, ok. That's not an issue then.

> 
>> 2. What about other projects that need libmuser code?
>> What worries me more is projects like SPDK/DPDK wanting to link
>> against the library and having to clone the entire QEMU repo as a
>> submodule. That sounds a lot more expensive than option 3 and probably
>> have further complications if they aren't GPL.
> 
> In the early stages where the vfio-user protocol and library interfaces
> might need changes it will be hard to use it from multiple applications
> without compatibility issues. If SPDK/DPDK are communicating with QEMU
> using a cutting-edge library then they probably need to build QEMU from
> source anyway. ISTR they also maintain a QEMU fork? So maybe it's not a
> big issue for them.

A few things on this:

Per Daniel's response on the other fork of the thread, I think we can
develop faster if we're not depending on qemu-devel. With the right
set of maintainers on the project, there's arguably more flexibility
in working at a restricted set of code. QEMU can then update the
submodule when a "checkpoint" is ready.

Also, I don't see why SPDK/DPDK would need to build QEMU from source
for their implementations. That is, they will need an up-to-date QEMU
for enabling their development, but they don't provide a QEMU to their
users; they just have to wait for a release.  I talked to Ben Walker
(SPDK), and he confirmed SPDK doesn't really maintain a fork of QEMU.
Apparently there was one when they were doing vhost-user-nvme, but
that's now abandoned.

If they had to add the entire QEMU as a submodule, it could
potentially be an issue, right? That's a lot of code just to get some
headers.

> 
>>> 
>>> #2 works best if the library is a small (just a few source files) with
>>> no fancy build system requirements.
>> 
>> The risk here is that they go out of sync. There's the same (or even
>> more) maintenance burden as point 3 below, with the added risk that
>> someone could patch the files and make cherry-picks non-trivial.
>> 
>>> 
>>> #3 is used in QEMU for several other components. Submodules are a pain
>>> to sync (requires sending a qemu.git patch to move to a new commit ID),
>>> so this isn't good for a dependency that moves quickly.
>> 
>> I argue this is no worse than option 2. It's what I think aligns best,
>> but let's keep weighing pros/cons and come to a conclusion together.
>> The list of maintainers for muser.git should be extended to include
>> more QEMU stakeholders and probably other projects that will use it
>> (as) heavily. The topic has been raised in SPDK's Slack team on
>> whether the client library should live in a repo of its own (eg.
>> libvfio-user.git). Given the reference implementation is in libmuser,
>> I still think muser.git is accurate (but can easily be persuaded
>> otherwise).
> 
> Me too, no solution is perfect. My thoughts about developing it within
> qemu.git for now is that this will make protocol and library interface
> changes easy. It will also encourage applications (DPDK/SPDK) to build
> against a matching QEMU so that there are no compatibility problems at
> the protocol or library level while the code is still heavily under
> development.

I suppose that challenge is up to the vfio-user/libmuser maintainers
to resolve by tagging rc versions or something along those lines.
Those would be the ones we can add to QEMU master as the work
progresses.

Thoughts?
F.

> 
> Stefan

On Thu, Oct 15, 2020 at 2:39 PM Felipe Franciosi <felipe@nutanix.com> wrote:
> > On Oct 13, 2020, at 10:30 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:> >
> > On Fri, Oct 02, 2020 at 10:14:23AM +0000, Felipe Franciosi wrote:
> >>> On Sep 30, 2020, at 3:24 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >>> On Tue, Sep 29, 2020 at 09:21:54AM -0700, John G Johnson wrote:
> >>>>> On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >>>>>
> >>>>> On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
> Per Daniel's response on the other fork of the thread, I think we can
> develop faster if we're not depending on qemu-devel. With the right
> set of maintainers on the project, there's arguably more flexibility
> in working at a restricted set of code. QEMU can then update the
> submodule when a "checkpoint" is ready.

Sure, if you are happy with the submodule approach that's great.

On the QEMU side the device name should have an "x-" prefix so it's
clear that the feature is experimental and subject to change. This way
there is no stability guarantee during development. Both the protocol
and command-line can be changed without introducing feature bits and
keeping backwards compatibility.

Stefan

On Wed, Sep 30, 2020 at 03:24:08PM +0100, Stefan Hajnoczi wrote:
> On Tue, Sep 29, 2020 at 09:21:54AM -0700, John G Johnson wrote:
> > > On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > 
> > > On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
> > >>> It should be accompanied by a test in tests/. PCI-level testing APIS for
> > >>> BARs, configuration space, interrupts, etc are available in
> > >>> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
> > >>> device backend interact with QEMU's vfio-user-pci implementation.
> > >> 
> > >> We plan to use a libmuser-based backend for testing. This, I suppose, will
> > >> make libmuser a dependency of QEMU (either as a submodule or as a library),
> > >> which for now can be disabled in the default configuration. Is this acceptable?
> > > 
> > > If there are no other dependencies and libmuser supports all host
> > > operating systems that QEMU's -device vfio-user supports, then I think
> > > it's a good idea to use libmuser for at least one in-tree test in QEMU.
> > > 
> > >>> Also please let us know who is working on what so additional people can
> > >>> get involved in areas that need work!
> > >> 
> > >> Swapnil and I will be working on libmuser and the test in QEMU, John and
> > >> the mp-qemu folks will be working on the patches for implementing
> > >> --device vfio-user-pci.
> > > 
> > > Great!
> > > 
> > > John: Will mpqemu use libmuser to implement the remote PCI host
> > > controller?
> > > 
> > 
> > 
> > 	The vfio-user-pci plan is to use libmuser on the server side.
> 
> Okay. Using libmuser in tests seems like a good choice in that case.
> 
> We'll need to figure out the details of how to do it because the
> traditional shared library dependency approach is not well-suited to
> in-development code. It would involve shipping libmuser distro packages
> so QEMU's build system can declare a library dependency (with details
> provided in a pkg-config file).
> 
> Here are approaches that are better for in-development libraries:
> 1. Keep the libmuser code in qemu.git.
> 2. A copy of libmuser in qemu.git with changes being sent upstream
>    (allows more flexibility in case QEMU-specific issues require
>    experimentation).
> 3. Git submodules.
> 
> #1 if you're happy to use the QEMU development process for merging
> libmuser code then it's easiest to officially host the code in qemu.git.
> libmuser gets a subdirectory in the qemu.git tree and you (the
> maintainers) send pull requests. A libmuser library build target
> provides installable static and shared libraries so external
> applications can link against libmuser too. The big advantage here is
> that QEMU can instantly use the latest libmuser code changes.
> 
> #2 works best if the library is a small (just a few source files) with
> no fancy build system requirements.
> 
> #3 is used in QEMU for several other components. Submodules are a pain
> to sync (requires sending a qemu.git patch to move to a new commit ID),
> so this isn't good for a dependency that moves quickly.

I don't think this is actually downside. If anything I think submodules
I think it would actually let libmuser develop faster.

QEMU is such a large project with high volume patches, that changes
take a long time to merge, even when everything is working well.

I think libmuser would have much faster development velocity if
the bit flowing into qemu.git was only as the trivial submodule
hash updates, as opposed to every single libmuser functional patch.

Maintaining libmuser as a standalone git repo avoids need for
libmuser to tie itself to QEMU's release cycle or processes. In
particular it means libmuser could opt to use a modern development
workflow using a GitForge with merge requests, instead of the
legacy email workflow.

Especially if we're considering libmuser to be consumed by non-QEMU
codebases, then long term it almost certainly should be a standalone
project. With this in mind, I think it will be better for libmuser
to start off as a standalone project straight away, so it can build
up its developer community organically, and not have people put off
contributing by the QEMU firehose.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Sep 15, 2020 at 07:29:17AM -0700, Thanos Makatos wrote:
> This patch introduces the vfio-user protocol specification (formerly
> known as VFIO-over-socket), which is designed to allow devices to be
> emulated outside QEMU, in a separate process. vfio-user reuses the
> existing VFIO defines, structs and concepts.
> 
> It has been earlier discussed as an RFC in:
> "RFC: use VFIO over a UNIX domain socket to implement device offloading"
> 
> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>

I reviewed the recently-added kernel VFIO_IOMMU_DIRTY_PAGES ioctl and
VFIO_REGION_TYPE_MIGRATION features. They enable live migration and
device state save/load.

Including them early would be good because it's difficult to retrofit
live migration into existing code later.

Stefan

This patch introduces the vfio-user protocol specification (formerly
known as VFIO-over-socket), which is designed to allow devices to be
emulated outside QEMU, in a separate process. vfio-user reuses the
existing VFIO defines, structs and concepts.

It has been earlier discussed as an RFC in:
"RFC: use VFIO over a UNIX domain socket to implement device offloading"

Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>

---

Changed since v1:
  * fix coding style issues
  * update MAINTAINERS for VFIO-over-socket
  * add vfio-over-socket to ToC

Changed since v2:
  * fix whitespace

Changed since v3:
  * rename protocol to vfio-user
  * add table of contents
  * fix Unicode problems
  * fix typos and various reStructuredText issues
  * various stylistic improvements
  * add backend program conventions
  * rewrite part of intro, drop QEMU-specific stuff
  * drop QEMU-specific paragraph about implementation
  * explain that passing of FDs isn't necessary
  * minor improvements in the VFIO section
  * various text substitutions for the sake of consistency
  * drop paragraph about client and server, already explained in intro
  * drop device ID
  * drop type from version
  * elaborate on request concurrency
  * convert some inessential paragraphs into notes
  * explain why some existing VFIO defines cannot be reused
  * explain how to make changes to the protocol
  * improve text of DMA map
  * reword comment about existing VFIO commands
  * add reference to Version section
  * reset device on disconnection
  * reword live migration section
  * replace sys/vfio.h with linux/vfio.h
  * drop reference to iovec
  * use argz the same way it is used in VFIO
  * add type field in header for clarity

Changed since v4:
  * introduce support for live migration as defined in include/uapi/linux/vfio.h
  * introduce 'max_fds' and 'migration' capabilities:
  * remove 'index' from VFIO_USER_DEVICE_GET_IRQ_INFO
  * fix minor typos and reworded some text for clarity

You can focus on v4 to v5 changes by cloning my fork
(https://github.com/tmakatos/qemu) and doing:

	git diff refs/tags/vfio-user/v4 refs/heads/vfio-user/v5
---
 MAINTAINERS              |    6 +
 docs/devel/index.rst     |    1 +
 docs/devel/vfio-user.rst | 1552 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1559 insertions(+)
 create mode 100644 docs/devel/vfio-user.rst

diff --git a/MAINTAINERS b/MAINTAINERS
index 7e442b5247..3611f9e365 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1754,6 +1754,12 @@ F: hw/vfio/ap.c
 F: docs/system/s390x/vfio-ap.rst
 L: qemu-s390x@nongnu.org
 
+vfio-user
+M: John G Johnson <john.g.johnson@oracle.com>
+M: Thanos Makatos <thanos.makatos@nutanix.com>
+S: Supported
+F: docs/devel/vfio-user.rst
+
 vhost
 M: Michael S. Tsirkin <mst@redhat.com>
 S: Supported
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 77baae5c77..7c7740a096 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -34,3 +34,4 @@ Contents:
    clocks
    qom
    block-coroutine-wrapper
+   vfio-user
diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst
new file mode 100644
index 0000000000..d8664e864f
--- /dev/null
+++ b/docs/devel/vfio-user.rst
@@ -0,0 +1,1552 @@
+.. include:: <isonum.txt>
+
+********************************
+vfio-user Protocol Specification
+********************************
+
+------------
+Version_ 0.1
+------------
+
+.. contents:: Table of Contents
+
+Introduction
+============
+vfio-user is a protocol that allows a device to be emulated in a separate
+process outside of a Virtual Machine Monitor (VMM). vfio-user devices consist
+of a generic VFIO device type, living inside the VMM, which we call the client,
+and the core device implementation, living outside the VMM, which we call the
+server.
+
+The `Linux VFIO ioctl interface <https://www.kernel.org/doc/html/latest/driver-api/vfio.html>`_
+been chosen as the base for this protocol for the following reasons:
+
+1) It is a mature and stable API, backed by an extensively used framework.
+2) The existing VFIO client implementation in QEMU (qemu/hw/vfio/) can be
+   largely reused.
+
+.. Note::
+   In a proof of concept implementation it has been demonstrated that using VFIO
+   over a UNIX domain socket is a viable option. vfio-user is designed with
+   QEMU in mind, however it could be used by other client applications. The
+   vfio-user protocol does not require that QEMU's VFIO client  implementation
+   is used in QEMU.
+
+None of the VFIO kernel modules are required for supporting the protocol,
+neither in the client nor the server, only the source header files are used.
+
+The main idea is to allow a virtual device to function in a separate process in
+the same host over a UNIX domain socket. A UNIX domain socket (AF_UNIX) is
+chosen because file descriptors can be trivially sent over it, which in turn
+allows:
+
+* Sharing of client memory for DMA with the server.
+* Sharing of server memory with the client for fast MMIO.
+* Efficient sharing of eventfd's for triggering interrupts.
+
+Other socket types could be used which allow the server to run in a separate
+guest in the same host (AF_VSOCK) or remotely (AF_INET). Theoretically the
+underlying transport does not necessarily have to be a socket, however we do
+not examine such alternatives. In this protocol version we focus on using a
+UNIX domain socket and introduce basic support for the other two types of
+sockets without considering performance implications.
+
+While passing of file descriptors is desirable for performance reasons, it is
+not necessary neither for the client nor for the server to support it in order
+to implement the protocol. There is always an in-band, message-passing fall
+back mechanism.
+
+VFIO
+====
+VFIO is a framework that allows a physical device to be securely passed through
+to a user space process; the device-specific kernel driver does not drive the
+device at all.  Typically, the user space process is a VMM and the device is
+passed through to it in order to achieve high performance. VFIO provides an API
+and the required functionality in the kernel. QEMU has adopted VFIO to allow a
+guest to directly access physical devices, instead of emulating them in
+software.
+
+vfio-user reuses the core VFIO concepts defined in its API, but implements them
+as messages to be sent over a socket. It does not change the kernel-based VFIO
+in any way, in fact none of the VFIO kernel modules need to be loaded to use
+vfio-user. It is also possible for the client to concurrently use the current
+kernel-based VFIO for one device, and vfio-user for another device.
+
+VFIO Device Model
+-----------------
+A device under VFIO presents a standard interface to the user process. Many of
+the VFIO operations in the existing interface use the ioctl() system call, and
+references to the existing interface are called the ioctl() implementation in
+this document.
+
+The following sections describe the set of messages that implement the VFIO
+interface over a socket. In many cases, the messages are direct translations of
+data structures used in the ioctl() implementation. Messages derived from
+ioctl()s will have a name derived from the ioctl() command name.  E.g., the
+VFIO_GET_INFO ioctl() command becomes a VFIO_USER_GET_INFO message.  The
+purpose of this reuse is to share as much code as feasible with the ioctl()
+implementation.
+
+Connection Initiation
+^^^^^^^^^^^^^^^^^^^^^
+After the client connects to the server, the initial server message is
+VFIO_USER_VERSION to propose a protocol version and set of capabilities to
+apply to the session. The client replies with a compatible version and set of
+capabilities it supports, or closes the connection if it cannot support the
+advertised version.
+
+DMA Memory Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+The client uses VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP messages to inform
+the server of the valid DMA ranges that the server can access on behalf
+of a device. DMA memory may be accessed by the server via VFIO_USER_DMA_READ
+and VFIO_USER_DMA_WRITE messages over the socket.
+
+An optimization for server access to client memory is for the client to provide
+file descriptors the server can mmap() to directly access client memory. Note
+that mmap() privileges cannot be revoked by the client, therefore file
+descriptors should only be exported in environments where the client trusts the
+server not to corrupt guest memory.
+
+Device Information
+^^^^^^^^^^^^^^^^^^
+The client uses a VFIO_USER_DEVICE_GET_INFO message to query the server for
+information about the device. This information includes:
+
+* The device type and whether it supports reset (``VFIO_DEVICE_FLAGS_``),
+* the number of device regions, and
+* the device presents to the client the number of interrupt types the device
+  supports.
+
+Region Information
+^^^^^^^^^^^^^^^^^^
+The client uses VFIO_USER_DEVICE_GET_REGION_INFO messages to query the server
+for information about the device's memory regions. This information describes:
+
+* Read and write permissions, whether it can be memory mapped, and whether it
+  supports additional capabilities (``VFIO_REGION_INFO_CAP_``).
+* Region index, size, and offset.
+
+When a region can be mapped by the client, the server provides a file
+descriptor which the client can mmap(). The server is responsible for polling
+for client updates to memory mapped regions.
+
+Region Capabilities
+"""""""""""""""""""
+Some regions have additional capabilities that cannot be described adequately
+by the region info data structure. These capabilities are returned in the
+region info reply in a list similar to PCI capabilities in a PCI device's
+configuration space.
+
+Sparse Regions
+""""""""""""""
+A region can be memory-mappable in whole or in part. When only a subset of a
+region can be mapped by the client, a VFIO_REGION_INFO_CAP_SPARSE_MMAP
+capability is included in the region info reply. This capability describes
+which portions can be mapped by the client.
+
+.. Note::
+   For example, in a virtual NVMe controller, sparse regions can be used so
+   that accesses to the NVMe registers (found in the beginning of BAR0) are
+   trapped (an infrequent event), while allowing direct access to the doorbells
+   (an extremely frequent event as every I/O submission requires a write to
+   BAR0), found right after the NVMe registers in BAR0.
+
+Device-Specific Regions
+"""""""""""""""""""""""
+
+A device can define regions additional to the standard ones (e.g. PCI indexes
+0-8). This is achieved by including a VFIO_REGION_INFO_CAP_TYPE capability
+in the region info reply of a device-specific region. Such regions are reflected
+in ``struct vfio_device_info.num_regions``. Thus, for PCI devices this value can
+be equal to, or higher than, VFIO_PCI_NUM_REGIONS.
+
+Interrupts
+^^^^^^^^^^
+The client uses VFIO_USER_DEVICE_GET_IRQ_INFO messages to query the server for
+the device's interrupt types. The interrupt types are specific to the bus the
+device is attached to, and the client is expected to know the capabilities of
+each interrupt type. The server can signal an interrupt either with
+VFIO_USER_VM_INTERRUPT messages over the socket, or can directly inject
+interrupts into the guest via an event file descriptor. The client configures
+how the server signals an interrupt with VFIO_USER_SET_IRQS messages.
+
+Device Read and Write
+^^^^^^^^^^^^^^^^^^^^^
+When the guest executes load or store operations to device memory, the client
+forwards these operations to the server with VFIO_USER_REGION_READ or
+VFIO_USER_REGION_WRITE messages. The server will reply with data from the
+device on read operations or an acknowledgement on write operations.
+
+DMA
+^^^
+When a device performs DMA accesses to guest memory, the server will forward
+them to the client with VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages.
+These messages can only be used to access guest memory the client has
+configured into the server.
+
+Protocol Specification
+======================
+To distinguish from the base VFIO symbols, all vfio-user symbols are prefixed
+with vfio_user or VFIO_USER. In revision 0.1, all data is in the little-endian
+format, although this may be relaxed in future revision in cases where the
+client and server are both big-endian. The messages are formatted for seamless
+reuse of the native VFIO structs.
+
+Socket
+------
+
+A server can serve:
+
+1) one or more clients, and/or
+2) one or more virtual devices, belonging to one or more clients.
+
+The current protocol specification requires a dedicated socket per
+client/server connection. It is a server-side implementation detail whether a
+single server handles multiple virtual devices from the same or multiple
+clients. The location of the socket is implementation-specific. Multiplexing
+clients, devices, and servers over the same socket is not supported in this
+version of the protocol.
+
+Authentication
+--------------
+For AF_UNIX, we rely on OS mandatory access controls on the socket files,
+therefore it is up to the management layer to set up the socket as required.
+Socket types than span guests or hosts will require a proper authentication
+mechanism. Defining that mechanism is deferred to a future version of the
+protocol.
+
+Command Concurrency
+-------------------
+A client may pipeline multiple commands without waiting for previous command
+replies.  The server will process commands in the order they are received.
+A consequence of this is if a client issues a command with the *No_reply* bit,
+then subseqently issues a command without *No_reply*, the older command will
+have been processed before the reply to the younger command is sent by the
+server.  The client must be aware of the device's capability to process concurrent
+commands if pipelining is used.  For example, pipelining allows multiple client
+threads to concurently access device memory; the client must ensure these acceses
+obey device semantics.
+
+An example is a frame buffer device, where the device may allow concurrent access
+to different areas of video memory, but may have indeterminate behavior if concurrent
+acceses are performed to command or status registers.
+
+Socket Disconnection Behavior
+-----------------------------
+The server and the client can disconnect from each other, either intentionally
+or unexpectedly. Both the client and the server need to know how to handle such
+events.
+
+Server Disconnection
+^^^^^^^^^^^^^^^^^^^^
+A server disconnecting from the client may indicate that:
+
+1) A virtual device has been restarted, either intentionally (e.g. because of a
+   device update) or unintentionally (e.g. because of a crash).
+2) A virtual device has been shut down with no intention to be restarted.
+
+It is impossible for the client to know whether or not a failure is
+intermittent or innocuous and should be retried, therefore the client should
+reset the VFIO device when it detects the socket has been disconnected.
+Error recovery will be driven by the guest's device error handling
+behavior.
+
+Client Disconnection
+^^^^^^^^^^^^^^^^^^^^
+The client disconnecting from the server primarily means that the client
+has exited. Currently, this means that the guest is shut down so the device is
+no longer needed therefore the server can automatically exit. However, there
+can be cases where a client disconnection should not result in a server exit:
+
+1) A single server serving multiple clients.
+2) A multi-process QEMU upgrading itself step by step, which is not yet
+   implemented.
+
+Therefore in order for the protocol to be forward compatible the server should
+take no action when the client disconnects. If anything happens to the client
+the control stack will know about it and can clean up resources
+accordingly.
+
+Request Retry and Response Timeout
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+A failed command is a command that has been successfully sent and has been
+responded to with an error code. Failure to send the command in the first place
+(e.g. because the socket is disconnected) is a different type of error examined
+earlier in the disconnect section.
+
+.. Note::
+   QEMU's VFIO retries certain operations if they fail. While this makes sense
+   for real HW, we don't know for sure whether it makes sense for virtual
+   devices.
+
+Defining a retry and timeout scheme is deferred to a future version of the
+protocol.
+
+.. _Commands:
+
+Commands
+--------
+The following table lists the VFIO message command IDs, and whether the
+message command is sent from the client or the server.
+
++----------------------------------+---------+-------------------+
+| Name                             | Command | Request Direction |
++==================================+=========+===================+
+| VFIO_USER_VERSION                | 1       | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_MAP                | 2       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_UNMAP              | 3       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_INFO        | 4       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_REGION_INFO | 5       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_GET_IRQ_INFO    | 6       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_SET_IRQS        | 7       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_REGION_READ            | 8       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_REGION_WRITE           | 9       | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_READ               | 10      | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DMA_WRITE              | 11      | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_VM_INTERRUPT           | 12      | server -> client  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DEVICE_RESET           | 13      | client -> server  |
++----------------------------------+---------+-------------------+
+| VFIO_USER_DIRTY_PAGES            | 14      | client -> server  |
++----------------------------------+---------+-------------------+
+
+
+.. Note:: Some VFIO defines cannot be reused since their values are
+   architecture-specific (e.g. VFIO_IOMMU_MAP_DMA).
+
+Header
+------
+All messages, both command messages and reply messages, are preceded by a
+header that contains basic information about the message. The header is
+followed by message-specific data described in the sections below.
+
++----------------+--------+-------------+
+| Name           | Offset | Size        |
++================+========+=============+
+| Message ID     | 0      | 2           |
++----------------+--------+-------------+
+| Command        | 2      | 2           |
++----------------+--------+-------------+
+| Message size   | 4      | 4           |
++----------------+--------+-------------+
+| Flags          | 8      | 4           |
++----------------+--------+-------------+
+|                | +-----+------------+ |
+|                | | Bit | Definition | |
+|                | +=====+============+ |
+|                | | 0-3 | Type       | |
+|                | +-----+------------+ |
+|                | | 4   | No_reply   | |
+|                | +-----+------------+ |
+|                | | 5   | Error      | |
+|                | +-----+------------+ |
++----------------+--------+-------------+
+| Error          | 12     | 4           |
++----------------+--------+-------------+
+| <message data> | 16     | variable    |
++----------------+--------+-------------+
+
+* *Message ID* identifies the message, and is echoed in the command's reply message.
+* *Command* specifies the command to be executed, listed in Commands_.
+* *Message size* contains the size of the entire message, including the header.
+* *Flags* contains attributes of the message:
+
+  * The *Type* bits indicate the message type.
+
+    *  *Command* (value 0x0) indicates a command message.
+    *  *Reply* (value 0x1) indicates a reply message acknowledging a previous
+       command with the same message ID.
+  * *No_reply* in a command message indicates that no reply is needed for this command.
+    This is commonly used when multiple commands are sent, and only the last needs
+    acknowledgement.
+  * *Error* in a reply message indicates the command being acknowledged had
+    an error. In this case, the *Error* field will be valid.
+
+* *Error* in a reply message is a UNIX errno value. It is reserved in a command message.
+
+Each command message in Commands_ must be replied to with a reply message, unless the
+message sets the *No_Reply* bit.  The reply consists of the header with the *Reply*
+bit set, plus any additional data.
+
+VFIO_USER_VERSION
+-----------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 1                      |
++--------------+------------------------+
+| Message size | 16 + version length    |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Version      | JSON byte array        |
++--------------+------------------------+
+
+This is the initial message sent by the server after the socket connection is
+established. The version is in JSON format, and the following objects must be
+included:
+
++--------------+--------+---------------------------------------------------+
+| Name         | Type   | Description                                       |
++==============+========+===================================================+
+| version      | object | ``{"major": <number>, "minor": <number>}``        |
+|              |        |                                                   |
+|              |        | Version supported by the sender, e.g. "0.1".      |
++--------------+--------+---------------------------------------------------+
+| capabilities | array  | Reserved. Can be omitted for v0.1, otherwise must |
+|              |        | be empty.                                         |
++--------------+--------+---------------------------------------------------+
+
+Common capabilities:
+
++---------------+------------------------------------------------------------+
+| Name          | Description                                                |
++===============+============================================================+
+| ``max_fds``   | Maximum number of file descriptors that can be received by |
+|               | the sender. Optional.                                      |
++---------------+------------------------------------------------------------+
+| ``migration`` | Migration capability object with the following format:     |
+|               |                                                            |
+|               | +------------+-------------------------------------------+ |
+|               | | Name       | Description                               | |
+|               | +============+===========================================+ |
+|               | | ``pgsize`` | Page size of dirty pages bitmap. The      | |
+|               | |            | smallest between the client and the       | |
+|               | |            | server is used.                           | |
+|               | +------------+-------------------------------------------+ |
++---------------+------------------------------------------------------------+
+
+.. _Version:
+
+Versioning and Feature Support
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Upon accepting a connection, the server must send a VFIO_USER_VERSION message
+proposing a protocol version and a set of capabilities. The client compares
+these with the versions and capabilities it supports and sends a
+VFIO_USER_VERSION reply according to the following rules.
+
+* The major version in the reply must be the same as proposed. If the client
+  does not support the proposed major, it closes the connection.
+* The minor version in the reply must be equal to or less than the minor
+  version proposed.
+* The capability list must be a subset of those proposed. If the client
+  requires a capability the server did not include, it closes the connection.
+
+The protocol major version will only change when incompatible protocol changes
+are made, such as changing the message format. The minor version may change
+when compatible changes are made, such as adding new messages or capabilities,
+Both the client and server must support all minor versions less than the
+maximum minor version it supports. E.g., an implementation that supports
+version 1.3 must also support 1.0 through 1.2.
+
+When making a change to this specification, the protocol version number must
+be included in the form "added in version X.Y"
+
+
+VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP
+-----------------------------------------
+
+Message Format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | MAP=2, UNMAP=3         |
++--------------+------------------------+
+| Message size | 16 + table size        |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Table        | array of table entries |
++--------------+------------------------+
+
+This command message is sent by the client to the server to inform it of the
+memory regions the server can access. It must be sent before the server can
+perform any DMA to the client. It is normally sent directly after the version
+handshake is completed, but may also occur when memory is added to or
+subtracted from the client, or if the client uses a vIOMMU. If the client does
+not expect the server to perform DMA then it does not need to send to the
+server VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP commands. If the server does
+not need to perform DMA then it can ignore such commands but it must still
+reply to them. The table is an array of the following structure.  This
+structure is 32 bytes in size, so the message size is:
+16 + (# of table entries * 32).
+
+VFIO bitmap format
+^^^^^^^^^^^^^^^^^^^^^^
+
++--------+--------+------+
+| Name   | Offset | Size |
++========+========+======+
+| pgsize | 0      | 8    |
++--------+--------+------+
+| size   | 8      | 8    |
++--------+--------+------+
+| data   | 16     | 8    |
++--------+--------+------+
+
+* *pgsize* is the page size for the bitmap, in bytes.
+* *size* the size for the bitmap, in bytes.
+* *data* This field is unused in vfio-user.
+
+The VFIO bitmap structure is defined in ``<linux/vfio.h>``
+(``struct vfio_bitmap``).
+
+Table entry format
+^^^^^^^^^^^^^^^^^^
+
++-------------+--------+--------------------------------------------------+
+| Name        | Offset | Size                                             |
++=============+========+==================================================+
+| Address     | 0      | 8                                                |
++-------------+--------+--------------------------------------------------+
+| Size        | 8      | 8                                                |
++-------------+--------+--------------------------------------------------+
+| Offset      | 16     | 8                                                |
++-------------+--------+--------------------------------------------------+
+| Protections | 24     | 4                                                |
++-------------+--------+--------------------------------------------------+
+| Flags       | 28     | 4                                                |
++-------------+--------+--------------------------------------------------+
+|             | +-----+-------------------------------------------------+ |
+|             | | Bit | Definition                                      | |
+|             | +=====+=================================================+ |
+|             | | 0   | Mappable/VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP   | |
+|             | +-----+-------------------------------------------------+ |
++-------------+--------+--------------------------------------------------+
+| Data        | 32     | variable                                         |
++-------------+--------+--------------------------------------------------+
+
+
+* *Address* is the base DMA address of the region.
+* *Size* is the size of the region.
+* *Offset* is the file offset of the region with respect to the associated file
+  descriptor.
+* *Protections* are the region's protection attributes as encoded in
+  ``<sys/mman.h>``.
+* *Flags* contain the following region attributes:
+
+  * *Mappable* indicates that the region can be mapped via the mmap() system call
+    using the file descriptor provided in the message meta-data. This flag is
+    only valid for VFIO_USER_DMA_MAP.
+
+  * *VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP* indicates that a dirty page bitmap
+    must be populated before unmapping the the DMA region. This flag is only
+    valid for VFIO_USER_DMA_UNMAP. The client must provide a
+    ``struct vfio_bitmap`` in the data field with the ``vfio_bitmap.pgsize``
+    and ``vfio_bitmap.size`` fields initialized.
+
+VFIO_USER_DMA_MAP
+"""""""""""""""""
+If a DMA region being added can be directly mapped by the server, an array of
+file descriptors must be sent as part of the message meta-data. Each region
+entry must have a corresponding file descriptor. On AF_UNIX sockets, the file
+descriptors must be passed as SCM_RIGHTS type ancillary data. Otherwise, if a
+DMA region cannot be directly mapped by the server, it can be accessed by the
+server using VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages, explained in
+`Read and Write Operations`_. A command to map over an existing region must be
+failed by the server with ``EEXIST`` set in error field in the reply.
+
+VFIO_USER_DMA_UNMAP
+"""""""""""""""""""
+Upon receiving a VFIO_USER_DMA_UNMAP command, if the file descriptor is mapped
+then the server must release all references to that DMA region before replying,
+which includes potentially in flight DMA transactions. Removing a portion of a
+DMA region is possible. If the VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP bit is set
+in the request, the server must append to the header the ``struct vfio_bitmap``
+received in the command, followed by the bitmap. Thus, the message size the
+client should is expect is the size of the header plus the size of
+``struct vfio_bitmap`` plus ``vfio_bitmap.size`` bytes. Each bit in the bitmap
+represents one page of size ``vfio_bitmap.pgsize``.
+
+.. Note::
+ I suppose dirt page logging must have been previously enabled in order for the
+ client to be able to use the VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP flag?
+
+VFIO_USER_DEVICE_GET_INFO
+-------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+----------------------------+
+| Name         | Value                      |
++==============+============================+
+| Message ID   | <ID>                       |
++--------------+----------------------------+
+| Command      | 4                          |
++--------------+----------------------------+
+| Message size | 16 in command, 32 in reply |
++--------------+----------------------------+
+| Flags        | Reply bit set in reply     |
++--------------+----------------------------+
+| Error        | 0/errno                    |
++--------------+----------------------------+
+| Device info  | VFIO device info           |
++--------------+----------------------------+
+
+This command message is sent by the client to the server to query for basic
+information about the device. Only the message header is needed in the command
+message.  The VFIO device info structure is defined in ``<linux/vfio.h>``
+(``struct vfio_device_info``).
+
+VFIO device info format
+^^^^^^^^^^^^^^^^^^^^^^^
+
++-------------+--------+--------------------------+
+| Name        | Offset | Size                     |
++=============+========+==========================+
+| argsz       | 16     | 4                        |
++-------------+--------+--------------------------+
+| flags       | 20     | 4                        |
++-------------+--------+--------------------------+
+|             | +-----+-------------------------+ |
+|             | | Bit | Definition              | |
+|             | +=====+=========================+ |
+|             | | 0   | VFIO_DEVICE_FLAGS_RESET | |
+|             | +-----+-------------------------+ |
+|             | | 1   | VFIO_DEVICE_FLAGS_PCI   | |
+|             | +-----+-------------------------+ |
++-------------+--------+--------------------------+
+| num_regions | 24     | 4                        |
++-------------+--------+--------------------------+
+| num_irqs    | 28     | 4                        |
++-------------+--------+--------------------------+
+
+* *argsz* is the size of the VFIO device info structure.
+* *flags* contains the following device attributes.
+
+  * VFIO_DEVICE_FLAGS_RESET indicates that the device supports the
+    VFIO_USER_DEVICE_RESET message.
+  * VFIO_DEVICE_FLAGS_PCI indicates that the device is a PCI device.
+
+* *num_regions* is the number of memory regions that the device exposes.
+* *num_irqs* is the number of distinct interrupt types that the device supports.
+
+This version of the protocol only supports PCI devices. Additional devices may
+be supported in future versions.
+
+VFIO_USER_DEVICE_GET_REGION_INFO
+--------------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 5                      |
++--------------+------------------------+
+| Message size | 48 + any caps          |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Region info  | VFIO region info       |
++--------------+------------------------+
+
+This command message is sent by the client to the server to query for
+information about device memory regions. The VFIO region info structure is
+defined in ``<linux/vfio.h>`` (``struct vfio_region_info``). Since the client
+does not know the size of the capabilities, the size of the reply it should
+expect is 48 plus any capabilities whose size is indicated in the size field of
+the reply header.
+
+VFIO region info format
+^^^^^^^^^^^^^^^^^^^^^^^
+
++------------+--------+------------------------------+
+| Name       | Offset | Size                         |
++============+========+==============================+
+| argsz      | 16     | 4                            |
++------------+--------+------------------------------+
+| flags      | 20     | 4                            |
++------------+--------+------------------------------+
+|            | +-----+-----------------------------+ |
+|            | | Bit | Definition                  | |
+|            | +=====+=============================+ |
+|            | | 0   | VFIO_REGION_INFO_FLAG_READ  | |
+|            | +-----+-----------------------------+ |
+|            | | 1   | VFIO_REGION_INFO_FLAG_WRITE | |
+|            | +-----+-----------------------------+ |
+|            | | 2   | VFIO_REGION_INFO_FLAG_MMAP  | |
+|            | +-----+-----------------------------+ |
+|            | | 3   | VFIO_REGION_INFO_FLAG_CAPS  | |
+|            | +-----+-----------------------------+ |
++------------+--------+------------------------------+
+| index      | 24     | 4                            |
++------------+--------+------------------------------+
+| cap_offset | 28     | 4                            |
++------------+--------+------------------------------+
+| size       | 32     | 8                            |
++------------+--------+------------------------------+
+| offset     | 40     | 8                            |
++------------+--------+------------------------------+
+
+* *argsz* is the size of the VFIO region info structure plus the
+  size of any region capabilities returned.
+* *flags* are attributes of the region:
+
+  * *VFIO_REGION_INFO_FLAG_READ* allows client read access to the region.
+  * *VFIO_REGION_INFO_FLAG_WRITE* allows client write access to the region.
+  * *VFIO_REGION_INFO_FLAG_MMAP* specifies the client can mmap() the region.
+    When this flag is set, the reply will include a file descriptor in its
+    meta-data. On AF_UNIX sockets, the file descriptors will be passed as
+    SCM_RIGHTS type ancillary data.
+  * *VFIO_REGION_INFO_FLAG_CAPS* indicates additional capabilities found in the
+    reply.
+
+* *index* is the index of memory region being queried, it is the only field
+  that is required to be set in the command message.
+* *cap_offset* describes where additional region capabilities can be found.
+  cap_offset is relative to the beginning of the VFIO region info structure.
+  The data structure it points is a VFIO cap header defined in
+  ``<linux/vfio.h>``.
+* *size* is the size of the region.
+* *offset* is the offset given to the mmap() system call for regions with the
+  MMAP attribute. It is also used as the base offset when mapping a VFIO
+  sparse mmap area, described below.
+
+VFIO Region capabilities
+^^^^^^^^^^^^^^^^^^^^^^^^
+The VFIO region information can also include a capabilities list. This list is
+similar to a PCI capability list - each entry has a common header that
+identifies a capability and where the next capability in the list can be found.
+The VFIO capability header format is defined in ``<linux/vfio.h>`` (``struct
+vfio_info_cap_header``).
+
+VFIO cap header format
+^^^^^^^^^^^^^^^^^^^^^^
+
++---------+--------+------+
+| Name    | Offset | Size |
++=========+========+======+
+| id      | 0      | 2    |
++---------+--------+------+
+| version | 2      | 2    |
++---------+--------+------+
+| next    | 4      | 4    |
++---------+--------+------+
+
+* *id* is the capability identity.
+* *version* is a capability-specific version number.
+* *next* specifies the offset of the next capability in the capability list. It
+  is relative to the beginning of the VFIO region info structure.
+
+VFIO sparse mmap
+^^^^^^^^^^^^^^^^
+
++------------------+----------------------------------+
+| Name             | Value                            |
++==================+==================================+
+| id               | VFIO_REGION_INFO_CAP_SPARSE_MMAP |
++------------------+----------------------------------+
+| version          | 0x1                              |
++------------------+----------------------------------+
+| next             | <next>                           |
++------------------+----------------------------------+
+| sparse mmap info | VFIO region info sparse mmap     |
++------------------+----------------------------------+
+
+This capability is defined when only a subrange of the region supports
+direct access by the client via mmap(). The VFIO sparse mmap area is defined in
+``<linux/vfio.h>`` (``struct vfio_region_sparse_mmap_area``).
+
+VFIO region info cap sparse mmap
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++----------+--------+------+
+| Name     | Offset | Size |
++==========+========+======+
+| nr_areas | 0      | 4    |
++----------+--------+------+
+| reserved | 4      | 4    |
++----------+--------+------+
+| offset   | 8      | 8    |
++----------+--------+------+
+| size     | 16     | 9    |
++----------+--------+------+
+| ...      |        |      |
++----------+--------+------+
+
+* *nr_areas* is the number of sparse mmap areas in the region.
+* *offset* and size describe a single area that can be mapped by the client.
+  There will be nr_areas pairs of offset and size. The offset will be added to
+  the base offset given in the VFIO_USER_DEVICE_GET_REGION_INFO to form the
+  offset argument of the subsequent mmap() call.
+
+The VFIO sparse mmap area is defined in ``<linux/vfio.h>`` (``struct
+vfio_region_info_cap_sparse_mmap``).
+
+VFIO Region Type
+^^^^^^^^^^^^^^^^
+
++------------------+---------------------------+
+| Name             | Value                     |
++==================+===========================+
+| id               | VFIO_REGION_INFO_CAP_TYPE |
++------------------+---------------------------+
+| version          | 0x1                       |
++------------------+---------------------------+
+| next             | <next>                    |
++------------------+---------------------------+
+| region info type | VFIO region info type     |
++------------------+---------------------------+
+
+This capability is defined when a region is specific to the device.
+
+VFIO region info type
+^^^^^^^^^^^^^^^^^^^^^
+
+The VFIO region info type is defined in ``<linux/vfio.h>``
+(``struct vfio_region_info_cap_type``).
+
++---------+--------+------+
+| Name    | Offset | Size |
++=========+========+======+
+| type    | 0      | 4    |
++---------+--------+------+
+| subtype | 4      | 4    |
++---------+--------+------+
+
+The only device-specific region type and subtype supported by vfio-user is
+VFIO_REGION_TYPE_MIGRATION (3) and VFIO_REGION_SUBTYPE_MIGRATION (1).
+
+VFIO Device Migration Info
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The beginning of the subregion must contain
+``struct vfio_device_migration_info``, defined in ``<linux/vfio.h>``. This
+subregion is accessed like any other part of a standard vfio-user PCI region
+using VFIO_USER_REGION_READ/VFIO_USER_REGION_WRITE.
+
++---------------+--------+-----------------------------+
+| Name          | Offset | Size                        |
++===============+========+=============================+
+| device_state  | 0      | 4                           |
++---------------+--------+-----------------------------+
+|               | +-----+----------------------------+ |
+|               | | Bit | Definition                 | |
+|               | +=====+============================+ |
+|               | | 0   | VFIO_DEVICE_STATE_RUNNING  | |
+|               | +-----+----------------------------+ |
+|               | | 1   | VFIO_DEVICE_STATE_SAVING   | |
+|               | +-----+----------------------------+ |
+|               | | 2   | VFIO_DEVICE_STATE_RESUMING | |
+|               | +-----+----------------------------+ |
++---------------+--------+-----------------------------+
+| reserved      | 4      | 4                           |
++---------------+--------+-----------------------------+
+| pending_bytes | 8      | 8                           |
++---------------+--------+-----------------------------+
+| data_offset   | 16     | 8                           |
++---------------+--------+-----------------------------+
+| data_size     | 24     | 8                           |
++---------------+--------+-----------------------------+
+
+* *device_state* defines the state of the device:
+
+  The client initiates device state transition by writing the intended state.
+  The server must respond only after it has succesfully transitioned to the new
+  state. If an error occurs then the server must respond to the
+  VFIO_USER_REGION_WRITE operation with the Error field set accordingly and
+  must remain at the previous state, or in case of internal error it must
+  transtition to the error state, defined as
+  VFIO_DEVICE_STATE_RESUMING | VFIO_DEVICE_STATE_SAVING. The client must
+  re-read the device state in order to determine it afresh.
+
+  The following device states are defined:
+
+  +-----------+---------+----------+-----------------------------------+
+  | _RESUMING | _SAVING | _RUNNING | Description                       |
+  +===========+=========+==========+===================================+
+  | 0         | 0       | 0        | Device is stopped.                |
+  +-----------+---------+----------+-----------------------------------+
+  | 0         | 0       | 1        | Device is running, default state. |
+  +-----------+---------+----------+-----------------------------------+
+  | 0         | 1       | 0        | Stop-and-copy state               |
+  +-----------+---------+----------+-----------------------------------+
+  | 0         | 1       | 1        | Pre-copy state                    |
+  +-----------+---------+----------+-----------------------------------+
+  | 1         | 0       | 0        | Resuming                          |
+  +-----------+---------+----------+-----------------------------------+
+  | 1         | 0       | 1        | Invalid state                     |
+  +-----------+---------+----------+-----------------------------------+
+  | 1         | 1       | 1        | Error state                       |
+  +-----------+---------+----------+-----------------------------------+
+  | 1         | 1       | 1        | Invalid state                     |
+  +-----------+---------+----------+-----------------------------------+
+
+  Valid state transitions are shown in the following table:
+
+  +-------------------------+---------+---------+---------------+----------+----------+
+  | |darr| From / To |rarr| | Stopped | Running | Stop-and-copy | Pre-copy | Resuming |
+  +=========================+=========+=========+===============+==========+==========+
+  | Stopped                 |    \-   |    0    |       0       |    0     |     0    |
+  +-------------------------+---------+---------+---------------+----------+----------+
+  | Running                 |    1    |    \-   |       1       |    1     |     1    |
+  +-------------------------+---------+---------+---------------+----------+----------+
+  | Stop-and-copy           |    1    |    0    |       \-      |    0     |     0    |
+  +-------------------------+---------+---------+---------------+----------+----------+
+  | Pre-copy                |    0    |    0    |       1       |    \-    |     0    |
+  +-------------------------+---------+---------+---------------+----------+----------+
+  | Resuming                |    0    |    1    |       0       |    0     |     \-   |
+  +-------------------------+---------+---------+---------------+----------+----------+
+
+  A device is migrated to the destination as follows:
+
+  * The source client transitions the device state from the running state to
+    the pre-copy state. This transition is optional for the client but must be
+    supported by the server. The souce server starts sending device state data
+    to the source client through the migration region while the device is
+    running.
+
+  * The source client transitions the device state from the running state or the
+    pre-copy state to the stop-and-copy state. The source server stops the
+    device, saves device state and sends it to the source client through the
+    migration region.
+
+  The source client is responsible for sending the migration data to the
+  destination client.
+
+  A device is resumed on the destination as follows:
+
+  * The destination client transitions the device state from the running state
+    to the resuming state. The destination server uses the device state data
+    received through the migration region to resume the device.
+
+  * The destination client provides saved device state to the destination
+    server and then transitions the device to back to the running state.
+
+* *reserved* This field is reserved and any access to it must be ignored by the
+  server.
+
+* *pending_bytes* Remaining bytes to be migrated by the server. This field is
+  read only.
+
+* *data_offset* Offset in the migration region where the client must:
+
+  * read from, during the pre-copy or stop-and-copy state, or
+
+  * write to, during the resuming state.
+
+  This field is read only.
+
+* *data_size* Contains the size, in bytes, of the amount of data copied to:
+
+  * the source migration region by the source server during the pre-copy or
+    stop-and copy state, or
+
+  * the destination migration region by the destination client during the
+    resuming state.
+
+Device-specific data must be stored at any position after
+`struct vfio_device_migration_info`. Note that the migration region can be
+memory mappable, even partially. In practise, only the migration data portion
+can be memory mapped.
+
+The client processes device state data during the pre-copy and the
+stop-and-copy state in the following iterative manner:
+
+  1. The client reads `pending_bytes` to mark a new iteration. Repeated reads
+     of this field is an idempotent operation. If there are no migration data
+     to be consumed then the next step depends on the current device state:
+
+     * pre-copy: the client must try again.
+
+     * stop-and-copy: this procedure can end and the device can now start
+       resuming on the destination.
+
+  2. The client reads `data_offset`; at thich point the server must make
+     available a portion of migration data at this offset to be read by the
+     client, which must happen *before* completing the read operation. The
+     amount of data to be read must be stored in the `data_size` field, which
+     the client reads next.
+
+  3. The client reads `data_size` to determine the amount of migration data
+     available.
+
+  4. The client reads and processes the migration data.
+
+  5. Go to step 1.
+
+Note that the client can transition the device from the pre-copy state to the
+stop-and-copy state at any time; `pending_bytes` does not need to become zero.
+
+The client initializes the device state on the destination by setting the
+device state in the resuming state and writing the migration data to the
+destination migration region at `data_offset` offset. The client can write the
+source migration data in an iterative manner and the server must consume this
+data before completing each write operation, updating the `data_offset` field.
+The server must apply the source migration data on the device resume state. The
+client must write data on the same order and transction size as read.
+
+If an error occurs then the server must fail the read or write operation. It is
+an implementation detail of the client how to handle errors.
+
+VFIO_USER_DEVICE_GET_IRQ_INFO
+-----------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 6                      |
++--------------+------------------------+
+| Message size | 32                     |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| IRQ info     | VFIO IRQ info          |
++--------------+------------------------+
+
+This command message is sent by the client to the server to query for
+information about device interrupt types. The VFIO IRQ info structure is
+defined in ``<linux/vfio.h>`` (``struct vfio_irq_info``).
+
+VFIO IRQ info format
+^^^^^^^^^^^^^^^^^^^^
+
++-------+--------+---------------------------+
+| Name  | Offset | Size                      |
++=======+========+===========================+
+| argsz | 16     | 4                         |
++-------+--------+---------------------------+
+| flags | 20     | 4                         |
++-------+--------+---------------------------+
+|       | +-----+--------------------------+ |
+|       | | Bit | Definition               | |
+|       | +=====+==========================+ |
+|       | | 0   | VFIO_IRQ_INFO_EVENTFD    | |
+|       | +-----+--------------------------+ |
+|       | | 1   | VFIO_IRQ_INFO_MASKABLE   | |
+|       | +-----+--------------------------+ |
+|       | | 2   | VFIO_IRQ_INFO_AUTOMASKED | |
+|       | +-----+--------------------------+ |
+|       | | 3   | VFIO_IRQ_INFO_NORESIZE   | |
+|       | +-----+--------------------------+ |
++-------+--------+---------------------------+
+| index | 24     | 4                         |
++-------+--------+---------------------------+
+| count | 28     | 4                         |
++-------+--------+---------------------------+
+
+* *argsz* is the size of the VFIO IRQ info structure.
+* *flags* defines IRQ attributes:
+
+  * *VFIO_IRQ_INFO_EVENTFD* indicates the IRQ type can support server eventfd
+    signalling.
+  * *VFIO_IRQ_INFO_MASKABLE* indicates that the IRQ type supports the MASK and
+    UNMASK actions in a VFIO_USER_DEVICE_SET_IRQS message.
+  * *VFIO_IRQ_INFO_AUTOMASKED* indicates the IRQ type masks itself after being
+    triggered, and the client must send an UNMASK action to receive new
+    interrupts.
+  * *VFIO_IRQ_INFO_NORESIZE* indicates VFIO_USER_SET_IRQS operations setup
+    interrupts as a set, and new sub-indexes cannot be enabled without disabling
+    the entire type.
+
+* index is the index of IRQ type being queried, it is the only field that is
+  required to be set in the command message.
+* count describes the number of interrupts of the queried type.
+
+VFIO_USER_DEVICE_SET_IRQS
+-------------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 7                      |
++--------------+------------------------+
+| Message size | 36 + any data          |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| IRQ set      | VFIO IRQ set           |
++--------------+------------------------+
+
+This command message is sent by the client to the server to set actions for
+device interrupt types. The VFIO IRQ set structure is defined in
+``<linux/vfio.h>`` (``struct vfio_irq_set``).
+
+VFIO IRQ set format
+^^^^^^^^^^^^^^^^^^^
+
++-------+--------+------------------------------+
+| Name  | Offset | Size                         |
++=======+========+==============================+
+| argsz | 16     | 4                            |
++-------+--------+------------------------------+
+| flags | 20     | 4                            |
++-------+--------+------------------------------+
+|       | +-----+-----------------------------+ |
+|       | | Bit | Definition                  | |
+|       | +=====+=============================+ |
+|       | | 0   | VFIO_IRQ_SET_DATA_NONE      | |
+|       | +-----+-----------------------------+ |
+|       | | 1   | VFIO_IRQ_SET_DATA_BOOL      | |
+|       | +-----+-----------------------------+ |
+|       | | 2   | VFIO_IRQ_SET_DATA_EVENTFD   | |
+|       | +-----+-----------------------------+ |
+|       | | 3   | VFIO_IRQ_SET_ACTION_MASK    | |
+|       | +-----+-----------------------------+ |
+|       | | 4   | VFIO_IRQ_SET_ACTION_UNMASK  | |
+|       | +-----+-----------------------------+ |
+|       | | 5   | VFIO_IRQ_SET_ACTION_TRIGGER | |
+|       | +-----+-----------------------------+ |
++-------+--------+------------------------------+
+| index | 24     | 4                            |
++-------+--------+------------------------------+
+| start | 28     | 4                            |
++-------+--------+------------------------------+
+| count | 32     | 4                            |
++-------+--------+------------------------------+
+| data  | 36     | variable                     |
++-------+--------+------------------------------+
+
+* *argsz* is the size of the VFIO IRQ set structure, including any *data* field.
+* *flags* defines the action performed on the interrupt range. The DATA flags
+  describe the data field sent in the message; the ACTION flags describe the
+  action to be performed. The flags are mutually exclusive for both sets.
+
+  * *VFIO_IRQ_SET_DATA_NONE* indicates there is no data field in the command.
+    The action is performed unconditionally.
+  * *VFIO_IRQ_SET_DATA_BOOL* indicates the data field is an array of boolean
+    bytes. The action is performed if the corresponding boolean is true.
+  * *VFIO_IRQ_SET_DATA_EVENTFD* indicates an array of event file descriptors
+    was sent in the message meta-data. These descriptors will be signalled when
+    the action defined by the action flags occurs. In AF_UNIX sockets, the
+    descriptors are sent as SCM_RIGHTS type ancillary data.
+  * *VFIO_IRQ_SET_ACTION_MASK* indicates a masking event. It can be used with
+    VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to mask an interrupt, or
+    with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the guest masks
+    the interrupt.
+  * *VFIO_IRQ_SET_ACTION_UNMASK* indicates an unmasking event. It can be used
+    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to unmask an
+    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
+    guest unmasks the interrupt.
+  * *VFIO_IRQ_SET_ACTION_TRIGGER* indicates a triggering event. It can be used
+    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to trigger an
+    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
+    server triggers the interrupt.
+
+* *index* is the index of IRQ type being setup.
+* *start* is the start of the sub-index being set.
+* *count* describes the number of sub-indexes being set. As a special case, a
+  count of 0 with data flags of VFIO_IRQ_SET_DATA_NONE disables all interrupts
+  of the index.
+* *data* is an optional field included when the
+  VFIO_IRQ_SET_DATA_BOOL flag is present. It contains an array of booleans
+  that specify whether the action is to be performed on the corresponding
+  index. It's used when the action is only performed on a subset of the range
+  specified.
+
+Not all interrupt types support every combination of data and action flags.
+The client must know the capabilities of the device and IRQ index before it
+sends a VFIO_USER_DEVICE_SET_IRQ message.
+
+.. _Read and Write Operations:
+
+Read and Write Operations
+-------------------------
+
+Not all I/O operations between the client and server can be done via direct
+access of memory mapped with an mmap() call. In these cases, the client and
+server use messages sent over the socket. It is expected that these operations
+will have lower performance than direct access.
+
+The client can access server memory with VFIO_USER_REGION_READ and
+VFIO_USER_REGION_WRITE commands. These share a common data structure that
+appears after the message header.
+
+REGION Read/Write Data
+^^^^^^^^^^^^^^^^^^^^^^
+
++--------+--------+----------+
+| Name   | Offset | Size     |
++========+========+==========+
+| Offset | 16     | 8        |
++--------+--------+----------+
+| Region | 24     | 4        |
++--------+--------+----------+
+| Count  | 28     | 4        |
++--------+--------+----------+
+| Data   | 32     | variable |
++--------+--------+----------+
+
+* *Offset* into the region being accessed.
+* *Region* is the index of the region being accessed.
+* *Count* is the size of the data to be transferred.
+* *Data* is the data to be read or written.
+
+The server can access client memory with VFIO_USER_DMA_READ and
+VFIO_USER_DMA_WRITE messages. These also share a common data structure that
+appears after the message header.
+
+DMA Read/Write Data
+^^^^^^^^^^^^^^^^^^^
+
++---------+--------+----------+
+| Name    | Offset | Size     |
++=========+========+==========+
+| Address | 16     | 8        |
++---------+--------+----------+
+| Count   | 24     | 4        |
++---------+--------+----------+
+| Data    | 28     | variable |
++---------+--------+----------+
+
+* *Address* is the area of client memory being accessed. This address must have
+  been previously exported to the server with a VFIO_USER_DMA_MAP message.
+* *Count* is the size of the data to be transferred.
+* *Data* is the data to be read or written.
+
+VFIO_USER_REGION_READ
+---------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 8                      |
++--------------+------------------------+
+| Message size | 32 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Read info    | REGION read/write data |
++--------------+------------------------+
+
+This command message is sent from the client to the server to read from server
+memory.  In the command messages, there is no data, and the count is the amount
+of data to be read. The reply message must include the data read, and its count
+field is the amount of data read.
+
+VFIO_USER_REGION_WRITE
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 9                      |
++--------------+------------------------+
+| Message size | 32 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| Write info   | REGION read/write data |
++--------------+------------------------+
+
+This command message is sent from the client to the server to write to server
+memory.  The command message must contain the data to be written, and its count
+field must contain the amount of write data. The count field in the reply
+message must be zero.
+
+VFIO_USER_DMA_READ
+------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 10                     |
++--------------+------------------------+
+| Message size | 28 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| DMA info     | DMA read/write data    |
++--------------+------------------------+
+
+This command message is sent from the server to the client to read from client
+memory.  In the command message, there is no data, and the count must will be
+the amount of data to be read. The reply message must include the data read,
+and its count field must be the amount of data read.
+
+VFIO_USER_DMA_WRITE
+-------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 11                     |
++--------------+------------------------+
+| Message size | 28 + data size         |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+| DMA info     | DMA read/write data    |
++--------------+------------------------+
+
+This command message is sent from the server to the client to write to server
+memory.  The command message must contain the data to be written, and its count
+field must contain the amount of write data. The count field in the reply
+message must be zero.
+
+VFIO_USER_VM_INTERRUPT
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++----------------+------------------------+
+| Name           | Value                  |
++================+========================+
+| Message ID     | <ID>                   |
++----------------+------------------------+
+| Command        | 12                     |
++----------------+------------------------+
+| Message size   | 20                     |
++----------------+------------------------+
+| Flags          | Reply bit set in reply |
++----------------+------------------------+
+| Error          | 0/errno                |
++----------------+------------------------+
+| Interrupt info | <interrupt>            |
++----------------+------------------------+
+
+This command message is sent from the server to the client to signal the device
+has raised an interrupt.
+
+Interrupt info format
+^^^^^^^^^^^^^^^^^^^^^
+
++-----------+--------+------+
+| Name      | Offset | Size |
++===========+========+======+
+| Sub-index | 16     | 4    |
++-----------+--------+------+
+
+* *Sub-index* is relative to the IRQ index, e.g., the vector number used in PCI
+  MSI/X type interrupts.
+
+VFIO_USER_DEVICE_RESET
+----------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------+------------------------+
+| Name         | Value                  |
++==============+========================+
+| Message ID   | <ID>                   |
++--------------+------------------------+
+| Command      | 13                     |
++--------------+------------------------+
+| Message size | 16                     |
++--------------+------------------------+
+| Flags        | Reply bit set in reply |
++--------------+------------------------+
+| Error        | 0/errno                |
++--------------+------------------------+
+
+This command message is sent from the client to the server to reset the device.
+
+VFIO_USER_DIRY_PAGES
+--------------------
+
+Message format
+^^^^^^^^^^^^^^
+
++--------------------+------------------------+
+| Name               | Value                  |
++====================+========================+
+| Message ID         | <ID>                   |
++--------------------+------------------------+
+| Command            | 14                     |
++--------------------+------------------------+
+| Message size       | 16                     |
++--------------------+------------------------+
+| Flags              | Reply bit set in reply |
++--------------------+------------------------+
+| Error              | 0/errno                |
++--------------------+------------------------+
+| VFIO Dirty bitmap  | <dirty bitmap>         |
++--------------------+------------------------+
+
+This command is analogous to VFIO_IOMMU_DIRTY_PAGES. It is sent by the client
+to the server in order to control logging of dirty pages, usually during a live
+migration. The VFIO dirty bitmap structure is defined in ``<linux/vfio.h>``
+(``struct vfio_iommu_type1_dirty_bitmap``).
+
+VFIO Dirty Bitmap Format
+^^^^^^^^^^^^^^^^^^^^^^^^
+
++-------+--------+-----------------------------------------+
+| Name  | Offset | Size                                    |
++=======+========+=========================================+
+| argsz | 0      | 4                                       |
++-------+--------+-----------------------------------------+
+| flags | 4      | 4                                       |
++-------+--------+-----------------------------------------+
+|       | +-----+----------------------------------------+ |
+|       | | Bit | Definition                             | |
+|       | +=====+========================================+ |
+|       | | 0   | VFIO_IOMMU_DIRTY_PAGES_FLAG_START      | |
+|       | +-----+----------------------------------------+ |
+|       | | 1   | VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP       | |
+|       | +-----+----------------------------------------+ |
+|       | | 2   | VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP | |
+|       | +-----+----------------------------------------+ |
++-------+--------+-----------------------------------------+
+| data  | 8      | 4                                       |
++-------+--------+-----------------------------------------+
+
+* *argsz* is the size of the VFIO dirty bitmap info structure.
+
+* *flags* defines the action to be performed by the server:
+
+  * *VFIO_IOMMU_DIRTY_PAGES_FLAG_START* instructs the server to start logging
+    pages it dirties. Logging continues until explicitly disabled by
+    VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP.
+
+  * *VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP* instructs the server to stop logging
+    dirty pages.
+
+  * *VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP* requests from the server to return
+    the dirty bitmap for a specific IOVA range. The IOVA range is specified by
+    "VFIO dirty bitmap get" structure, which must immediatelly follow the
+    "VFIO dirty bitmap" structure, explained next. This operation is only valid
+    if logging of dirty pages has been previously started. The server must
+    respond the same way it does for VFIO_USER_DMA_UNMAP (the dirty pages
+    bitmap must follow the response header).
+
+  These flags are mutually exclusive with each other.
+
+* *data* This field is unused in vfio-user.
+
+VFIO Dirty Bitmap Get Format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++--------+--------+------+
+| Name   | Offset | Size |
++========+========+======+
+| iova   | 0      | 8    |
++--------+--------+------+
+| size   | 8      | 8    |
++--------+--------+------+
+| bitmap | 16     | 24   |
++--------+--------+------+
+
+* *iova* is the IOVA offset
+
+* *size* is the size of the IOVA region
+
+* *bitmap* is the VFIO bitmap (``struct vfio_bitmap``), with the same semantics
+  as VFIO_USER_DMA_UNMAP.
+
+
+Appendices
+==========
+
+Unused VFIO ioctl() commands
+----------------------------
+
+The following VFIO commands do not have an equivalent vfio-user command:
+
+* VFIO_GET_API_VERSION
+* VFIO_CHECK_EXTENSION
+* VFIO_SET_IOMMU
+* VFIO_GROUP_GET_STATUS
+* VFIO_GROUP_SET_CONTAINER
+* VFIO_GROUP_UNSET_CONTAINER
+* VFIO_GROUP_GET_DEVICE_FD
+* VFIO_IOMMU_GET_INFO
+
+However, once support for live migration for VFIO devices is finalized some
+of the above commands may have to be handled by the client in their
+corresponding vfio-user form. This will be addressed in a future protocol
+version.
+
+VFIO groups and containers
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current VFIO implementation includes group and container idioms that
+describe how a device relates to the host IOMMU. In the vfio-user
+implementation, the IOMMU is implemented in SW by the client, and is not
+visible to the server. The simplest idea would be that the client put each
+device into its own group and container.
+
+Backend Program Conventions
+---------------------------
+
+vfio-user backend program conventions are based on the vhost-user ones.
+
+* The backend program must not daemonize itself.
+* No assumptions must be made as to what access the backend program has on the
+  system.
+* File descriptors 0, 1 and 2 must exist, must have regular
+  stdin/stdout/stderr semantics, and can be redirected.
+* The backend program must honor the SIGTERM signal.
+* The backend program must accept the following commands line options:
+
+  * ``--socket-path=PATH``: path to UNIX domain socket,
+  * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible with
+    ``--socket-path``
+* The backend program must be accompanied with a JSON file stored under
+  ``/usr/share/vfio-user``.
-- 
2.12.2

FYI here's v5 of the vfio-user protocol, my --cc in git send-email got messed up somehow

> -----Original Message-----
> From: Qemu-devel <qemu-devel-
> bounces+thanos.makatos=nutanix.com@nongnu.org> On Behalf Of Thanos
> Makatos
> Sent: 28 October 2020 16:10
> To: qemu-devel@nongnu.org
> Subject: [PATCH v5] introduce vfio-user protocol specification
> 
> This patch introduces the vfio-user protocol specification (formerly
> known as VFIO-over-socket), which is designed to allow devices to be
> emulated outside QEMU, in a separate process. vfio-user reuses the
> existing VFIO defines, structs and concepts.
> 
> It has been earlier discussed as an RFC in:
> "RFC: use VFIO over a UNIX domain socket to implement device offloading"
> 
> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
> 
> ---
> 
> Changed since v1:
>   * fix coding style issues
>   * update MAINTAINERS for VFIO-over-socket
>   * add vfio-over-socket to ToC
> 
> Changed since v2:
>   * fix whitespace
> 
> Changed since v3:
>   * rename protocol to vfio-user
>   * add table of contents
>   * fix Unicode problems
>   * fix typos and various reStructuredText issues
>   * various stylistic improvements
>   * add backend program conventions
>   * rewrite part of intro, drop QEMU-specific stuff
>   * drop QEMU-specific paragraph about implementation
>   * explain that passing of FDs isn't necessary
>   * minor improvements in the VFIO section
>   * various text substitutions for the sake of consistency
>   * drop paragraph about client and server, already explained in intro
>   * drop device ID
>   * drop type from version
>   * elaborate on request concurrency
>   * convert some inessential paragraphs into notes
>   * explain why some existing VFIO defines cannot be reused
>   * explain how to make changes to the protocol
>   * improve text of DMA map
>   * reword comment about existing VFIO commands
>   * add reference to Version section
>   * reset device on disconnection
>   * reword live migration section
>   * replace sys/vfio.h with linux/vfio.h
>   * drop reference to iovec
>   * use argz the same way it is used in VFIO
>   * add type field in header for clarity
> 
> Changed since v4:
>   * introduce support for live migration as defined in include/uapi/linux/vfio.h
>   * introduce 'max_fds' and 'migration' capabilities:
>   * remove 'index' from VFIO_USER_DEVICE_GET_IRQ_INFO
>   * fix minor typos and reworded some text for clarity
> 
> You can focus on v4 to v5 changes by cloning my fork
> (https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_tmakatos_qemu&d=DwIBAg&c=s883GpUCOChKOHiocYtGc
> g&r=XTpYsh5Ps2zJvtw6ogtti46atk736SI4vgsJiUKIyDE&m=LUZ0P5VWbFFynYq
> 5SPmJD4n-E6Tow26xDuPYIeCKV28&s=IgZugyqjIvKQ3-
> gpftrAm73sKizX51JYUroR-4aaaI0&e= ) and doing:
> 
> 	git diff refs/tags/vfio-user/v4 refs/heads/vfio-user/v5
> ---
>  MAINTAINERS              |    6 +
>  docs/devel/index.rst     |    1 +
>  docs/devel/vfio-user.rst | 1552
> ++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 1559 insertions(+)
>  create mode 100644 docs/devel/vfio-user.rst
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 7e442b5247..3611f9e365 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1754,6 +1754,12 @@ F: hw/vfio/ap.c
>  F: docs/system/s390x/vfio-ap.rst
>  L: qemu-s390x@nongnu.org
> 
> +vfio-user
> +M: John G Johnson <john.g.johnson@oracle.com>
> +M: Thanos Makatos <thanos.makatos@nutanix.com>
> +S: Supported
> +F: docs/devel/vfio-user.rst
> +
>  vhost
>  M: Michael S. Tsirkin <mst@redhat.com>
>  S: Supported
> diff --git a/docs/devel/index.rst b/docs/devel/index.rst
> index 77baae5c77..7c7740a096 100644
> --- a/docs/devel/index.rst
> +++ b/docs/devel/index.rst
> @@ -34,3 +34,4 @@ Contents:
>     clocks
>     qom
>     block-coroutine-wrapper
> +   vfio-user
> diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst
> new file mode 100644
> index 0000000000..d8664e864f
> --- /dev/null
> +++ b/docs/devel/vfio-user.rst
> @@ -0,0 +1,1552 @@
> +.. include:: <isonum.txt>
> +
> +********************************
> +vfio-user Protocol Specification
> +********************************
> +
> +------------
> +Version_ 0.1
> +------------
> +
> +.. contents:: Table of Contents
> +
> +Introduction
> +============
> +vfio-user is a protocol that allows a device to be emulated in a separate
> +process outside of a Virtual Machine Monitor (VMM). vfio-user devices
> consist
> +of a generic VFIO device type, living inside the VMM, which we call the
> client,
> +and the core device implementation, living outside the VMM, which we call
> the
> +server.
> +
> +The `Linux VFIO ioctl interface
> <https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.kernel.org_doc_html_latest_driver-
> 2Dapi_vfio.html&d=DwIBAg&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJ
> vtw6ogtti46atk736SI4vgsJiUKIyDE&m=LUZ0P5VWbFFynYq5SPmJD4n-
> E6Tow26xDuPYIeCKV28&s=pfcbCOMAyP07TTkii2t-vm1l-
> rmnmKQbqygTPjxatsQ&e= >`_
> +been chosen as the base for this protocol for the following reasons:
> +
> +1) It is a mature and stable API, backed by an extensively used framework.
> +2) The existing VFIO client implementation in QEMU (qemu/hw/vfio/) can
> be
> +   largely reused.
> +
> +.. Note::
> +   In a proof of concept implementation it has been demonstrated that using
> VFIO
> +   over a UNIX domain socket is a viable option. vfio-user is designed with
> +   QEMU in mind, however it could be used by other client applications. The
> +   vfio-user protocol does not require that QEMU's VFIO client
> implementation
> +   is used in QEMU.
> +
> +None of the VFIO kernel modules are required for supporting the protocol,
> +neither in the client nor the server, only the source header files are used.
> +
> +The main idea is to allow a virtual device to function in a separate process in
> +the same host over a UNIX domain socket. A UNIX domain socket
> (AF_UNIX) is
> +chosen because file descriptors can be trivially sent over it, which in turn
> +allows:
> +
> +* Sharing of client memory for DMA with the server.
> +* Sharing of server memory with the client for fast MMIO.
> +* Efficient sharing of eventfd's for triggering interrupts.
> +
> +Other socket types could be used which allow the server to run in a
> separate
> +guest in the same host (AF_VSOCK) or remotely (AF_INET). Theoretically
> the
> +underlying transport does not necessarily have to be a socket, however we
> do
> +not examine such alternatives. In this protocol version we focus on using a
> +UNIX domain socket and introduce basic support for the other two types of
> +sockets without considering performance implications.
> +
> +While passing of file descriptors is desirable for performance reasons, it is
> +not necessary neither for the client nor for the server to support it in order
> +to implement the protocol. There is always an in-band, message-passing fall
> +back mechanism.
> +
> +VFIO
> +====
> +VFIO is a framework that allows a physical device to be securely passed
> through
> +to a user space process; the device-specific kernel driver does not drive the
> +device at all.  Typically, the user space process is a VMM and the device is
> +passed through to it in order to achieve high performance. VFIO provides an
> API
> +and the required functionality in the kernel. QEMU has adopted VFIO to
> allow a
> +guest to directly access physical devices, instead of emulating them in
> +software.
> +
> +vfio-user reuses the core VFIO concepts defined in its API, but implements
> them
> +as messages to be sent over a socket. It does not change the kernel-based
> VFIO
> +in any way, in fact none of the VFIO kernel modules need to be loaded to
> use
> +vfio-user. It is also possible for the client to concurrently use the current
> +kernel-based VFIO for one device, and vfio-user for another device.
> +
> +VFIO Device Model
> +-----------------
> +A device under VFIO presents a standard interface to the user process.
> Many of
> +the VFIO operations in the existing interface use the ioctl() system call, and
> +references to the existing interface are called the ioctl() implementation in
> +this document.
> +
> +The following sections describe the set of messages that implement the
> VFIO
> +interface over a socket. In many cases, the messages are direct translations
> of
> +data structures used in the ioctl() implementation. Messages derived from
> +ioctl()s will have a name derived from the ioctl() command name.  E.g., the
> +VFIO_GET_INFO ioctl() command becomes a VFIO_USER_GET_INFO
> message.  The
> +purpose of this reuse is to share as much code as feasible with the ioctl()
> +implementation.
> +
> +Connection Initiation
> +^^^^^^^^^^^^^^^^^^^^^
> +After the client connects to the server, the initial server message is
> +VFIO_USER_VERSION to propose a protocol version and set of capabilities
> to
> +apply to the session. The client replies with a compatible version and set of
> +capabilities it supports, or closes the connection if it cannot support the
> +advertised version.
> +
> +DMA Memory Configuration
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +The client uses VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP
> messages to inform
> +the server of the valid DMA ranges that the server can access on behalf
> +of a device. DMA memory may be accessed by the server via
> VFIO_USER_DMA_READ
> +and VFIO_USER_DMA_WRITE messages over the socket.
> +
> +An optimization for server access to client memory is for the client to
> provide
> +file descriptors the server can mmap() to directly access client memory.
> Note
> +that mmap() privileges cannot be revoked by the client, therefore file
> +descriptors should only be exported in environments where the client
> trusts the
> +server not to corrupt guest memory.
> +
> +Device Information
> +^^^^^^^^^^^^^^^^^^
> +The client uses a VFIO_USER_DEVICE_GET_INFO message to query the
> server for
> +information about the device. This information includes:
> +
> +* The device type and whether it supports reset
> (``VFIO_DEVICE_FLAGS_``),
> +* the number of device regions, and
> +* the device presents to the client the number of interrupt types the device
> +  supports.
> +
> +Region Information
> +^^^^^^^^^^^^^^^^^^
> +The client uses VFIO_USER_DEVICE_GET_REGION_INFO messages to query
> the server
> +for information about the device's memory regions. This information
> describes:
> +
> +* Read and write permissions, whether it can be memory mapped, and
> whether it
> +  supports additional capabilities (``VFIO_REGION_INFO_CAP_``).
> +* Region index, size, and offset.
> +
> +When a region can be mapped by the client, the server provides a file
> +descriptor which the client can mmap(). The server is responsible for polling
> +for client updates to memory mapped regions.
> +
> +Region Capabilities
> +"""""""""""""""""""
> +Some regions have additional capabilities that cannot be described
> adequately
> +by the region info data structure. These capabilities are returned in the
> +region info reply in a list similar to PCI capabilities in a PCI device's
> +configuration space.
> +
> +Sparse Regions
> +""""""""""""""
> +A region can be memory-mappable in whole or in part. When only a subset
> of a
> +region can be mapped by the client, a
> VFIO_REGION_INFO_CAP_SPARSE_MMAP
> +capability is included in the region info reply. This capability describes
> +which portions can be mapped by the client.
> +
> +.. Note::
> +   For example, in a virtual NVMe controller, sparse regions can be used so
> +   that accesses to the NVMe registers (found in the beginning of BAR0) are
> +   trapped (an infrequent event), while allowing direct access to the
> doorbells
> +   (an extremely frequent event as every I/O submission requires a write to
> +   BAR0), found right after the NVMe registers in BAR0.
> +
> +Device-Specific Regions
> +"""""""""""""""""""""""
> +
> +A device can define regions additional to the standard ones (e.g. PCI
> indexes
> +0-8). This is achieved by including a VFIO_REGION_INFO_CAP_TYPE
> capability
> +in the region info reply of a device-specific region. Such regions are
> reflected
> +in ``struct vfio_device_info.num_regions``. Thus, for PCI devices this value
> can
> +be equal to, or higher than, VFIO_PCI_NUM_REGIONS.
> +
> +Interrupts
> +^^^^^^^^^^
> +The client uses VFIO_USER_DEVICE_GET_IRQ_INFO messages to query the
> server for
> +the device's interrupt types. The interrupt types are specific to the bus the
> +device is attached to, and the client is expected to know the capabilities of
> +each interrupt type. The server can signal an interrupt either with
> +VFIO_USER_VM_INTERRUPT messages over the socket, or can directly
> inject
> +interrupts into the guest via an event file descriptor. The client configures
> +how the server signals an interrupt with VFIO_USER_SET_IRQS messages.
> +
> +Device Read and Write
> +^^^^^^^^^^^^^^^^^^^^^
> +When the guest executes load or store operations to device memory, the
> client
> +forwards these operations to the server with VFIO_USER_REGION_READ or
> +VFIO_USER_REGION_WRITE messages. The server will reply with data from
> the
> +device on read operations or an acknowledgement on write operations.
> +
> +DMA
> +^^^
> +When a device performs DMA accesses to guest memory, the server will
> forward
> +them to the client with VFIO_USER_DMA_READ and
> VFIO_USER_DMA_WRITE messages.
> +These messages can only be used to access guest memory the client has
> +configured into the server.
> +
> +Protocol Specification
> +======================
> +To distinguish from the base VFIO symbols, all vfio-user symbols are
> prefixed
> +with vfio_user or VFIO_USER. In revision 0.1, all data is in the little-endian
> +format, although this may be relaxed in future revision in cases where the
> +client and server are both big-endian. The messages are formatted for
> seamless
> +reuse of the native VFIO structs.
> +
> +Socket
> +------
> +
> +A server can serve:
> +
> +1) one or more clients, and/or
> +2) one or more virtual devices, belonging to one or more clients.
> +
> +The current protocol specification requires a dedicated socket per
> +client/server connection. It is a server-side implementation detail whether
> a
> +single server handles multiple virtual devices from the same or multiple
> +clients. The location of the socket is implementation-specific. Multiplexing
> +clients, devices, and servers over the same socket is not supported in this
> +version of the protocol.
> +
> +Authentication
> +--------------
> +For AF_UNIX, we rely on OS mandatory access controls on the socket files,
> +therefore it is up to the management layer to set up the socket as required.
> +Socket types than span guests or hosts will require a proper authentication
> +mechanism. Defining that mechanism is deferred to a future version of the
> +protocol.
> +
> +Command Concurrency
> +-------------------
> +A client may pipeline multiple commands without waiting for previous
> command
> +replies.  The server will process commands in the order they are received.
> +A consequence of this is if a client issues a command with the *No_reply*
> bit,
> +then subseqently issues a command without *No_reply*, the older
> command will
> +have been processed before the reply to the younger command is sent by
> the
> +server.  The client must be aware of the device's capability to process
> concurrent
> +commands if pipelining is used.  For example, pipelining allows multiple
> client
> +threads to concurently access device memory; the client must ensure these
> acceses
> +obey device semantics.
> +
> +An example is a frame buffer device, where the device may allow
> concurrent access
> +to different areas of video memory, but may have indeterminate behavior
> if concurrent
> +acceses are performed to command or status registers.
> +
> +Socket Disconnection Behavior
> +-----------------------------
> +The server and the client can disconnect from each other, either
> intentionally
> +or unexpectedly. Both the client and the server need to know how to
> handle such
> +events.
> +
> +Server Disconnection
> +^^^^^^^^^^^^^^^^^^^^
> +A server disconnecting from the client may indicate that:
> +
> +1) A virtual device has been restarted, either intentionally (e.g. because of a
> +   device update) or unintentionally (e.g. because of a crash).
> +2) A virtual device has been shut down with no intention to be restarted.
> +
> +It is impossible for the client to know whether or not a failure is
> +intermittent or innocuous and should be retried, therefore the client should
> +reset the VFIO device when it detects the socket has been disconnected.
> +Error recovery will be driven by the guest's device error handling
> +behavior.
> +
> +Client Disconnection
> +^^^^^^^^^^^^^^^^^^^^
> +The client disconnecting from the server primarily means that the client
> +has exited. Currently, this means that the guest is shut down so the device
> is
> +no longer needed therefore the server can automatically exit. However,
> there
> +can be cases where a client disconnection should not result in a server exit:
> +
> +1) A single server serving multiple clients.
> +2) A multi-process QEMU upgrading itself step by step, which is not yet
> +   implemented.
> +
> +Therefore in order for the protocol to be forward compatible the server
> should
> +take no action when the client disconnects. If anything happens to the
> client
> +the control stack will know about it and can clean up resources
> +accordingly.
> +
> +Request Retry and Response Timeout
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +A failed command is a command that has been successfully sent and has
> been
> +responded to with an error code. Failure to send the command in the first
> place
> +(e.g. because the socket is disconnected) is a different type of error
> examined
> +earlier in the disconnect section.
> +
> +.. Note::
> +   QEMU's VFIO retries certain operations if they fail. While this makes sense
> +   for real HW, we don't know for sure whether it makes sense for virtual
> +   devices.
> +
> +Defining a retry and timeout scheme is deferred to a future version of the
> +protocol.
> +
> +.. _Commands:
> +
> +Commands
> +--------
> +The following table lists the VFIO message command IDs, and whether the
> +message command is sent from the client or the server.
> +
> ++----------------------------------+---------+-------------------+
> +| Name                             | Command | Request Direction |
> ++==================================+=========+===========
> ========+
> +| VFIO_USER_VERSION                | 1       | server -> client  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_MAP                | 2       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_UNMAP              | 3       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_GET_INFO        | 4       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_GET_REGION_INFO | 5       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_GET_IRQ_INFO    | 6       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_SET_IRQS        | 7       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_REGION_READ            | 8       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_REGION_WRITE           | 9       | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_READ               | 10      | server -> client  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DMA_WRITE              | 11      | server -> client  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_VM_INTERRUPT           | 12      | server -> client  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DEVICE_RESET           | 13      | client -> server  |
> ++----------------------------------+---------+-------------------+
> +| VFIO_USER_DIRTY_PAGES            | 14      | client -> server  |
> ++----------------------------------+---------+-------------------+
> +
> +
> +.. Note:: Some VFIO defines cannot be reused since their values are
> +   architecture-specific (e.g. VFIO_IOMMU_MAP_DMA).
> +
> +Header
> +------
> +All messages, both command messages and reply messages, are preceded
> by a
> +header that contains basic information about the message. The header is
> +followed by message-specific data described in the sections below.
> +
> ++----------------+--------+-------------+
> +| Name           | Offset | Size        |
> ++================+========+=============+
> +| Message ID     | 0      | 2           |
> ++----------------+--------+-------------+
> +| Command        | 2      | 2           |
> ++----------------+--------+-------------+
> +| Message size   | 4      | 4           |
> ++----------------+--------+-------------+
> +| Flags          | 8      | 4           |
> ++----------------+--------+-------------+
> +|                | +-----+------------+ |
> +|                | | Bit | Definition | |
> +|                | +=====+============+ |
> +|                | | 0-3 | Type       | |
> +|                | +-----+------------+ |
> +|                | | 4   | No_reply   | |
> +|                | +-----+------------+ |
> +|                | | 5   | Error      | |
> +|                | +-----+------------+ |
> ++----------------+--------+-------------+
> +| Error          | 12     | 4           |
> ++----------------+--------+-------------+
> +| <message data> | 16     | variable    |
> ++----------------+--------+-------------+
> +
> +* *Message ID* identifies the message, and is echoed in the command's
> reply message.
> +* *Command* specifies the command to be executed, listed in
> Commands_.
> +* *Message size* contains the size of the entire message, including the
> header.
> +* *Flags* contains attributes of the message:
> +
> +  * The *Type* bits indicate the message type.
> +
> +    *  *Command* (value 0x0) indicates a command message.
> +    *  *Reply* (value 0x1) indicates a reply message acknowledging a
> previous
> +       command with the same message ID.
> +  * *No_reply* in a command message indicates that no reply is needed for
> this command.
> +    This is commonly used when multiple commands are sent, and only the
> last needs
> +    acknowledgement.
> +  * *Error* in a reply message indicates the command being acknowledged
> had
> +    an error. In this case, the *Error* field will be valid.
> +
> +* *Error* in a reply message is a UNIX errno value. It is reserved in a
> command message.
> +
> +Each command message in Commands_ must be replied to with a reply
> message, unless the
> +message sets the *No_Reply* bit.  The reply consists of the header with
> the *Reply*
> +bit set, plus any additional data.
> +
> +VFIO_USER_VERSION
> +-----------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 1                      |
> ++--------------+------------------------+
> +| Message size | 16 + version length    |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| Version      | JSON byte array        |
> ++--------------+------------------------+
> +
> +This is the initial message sent by the server after the socket connection is
> +established. The version is in JSON format, and the following objects must
> be
> +included:
> +
> ++--------------+--------+---------------------------------------------------+
> +| Name         | Type   | Description                                       |
> ++==============+========+================================
> ===================+
> +| version      | object | ``{"major": <number>, "minor": <number>}``        |
> +|              |        |                                                   |
> +|              |        | Version supported by the sender, e.g. "0.1".      |
> ++--------------+--------+---------------------------------------------------+
> +| capabilities | array  | Reserved. Can be omitted for v0.1, otherwise must |
> +|              |        | be empty.                                         |
> ++--------------+--------+---------------------------------------------------+
> +
> +Common capabilities:
> +
> ++---------------+------------------------------------------------------------+
> +| Name          | Description                                                |
> ++===============+========================================
> ====================+
> +| ``max_fds``   | Maximum number of file descriptors that can be received
> by |
> +|               | the sender. Optional.                                      |
> ++---------------+------------------------------------------------------------+
> +| ``migration`` | Migration capability object with the following format:     |
> +|               |                                                            |
> +|               | +------------+-------------------------------------------+ |
> +|               | | Name       | Description                               | |
> +|               |
> +============+===========================================+
> |
> +|               | | ``pgsize`` | Page size of dirty pages bitmap. The      | |
> +|               | |            | smallest between the client and the       | |
> +|               | |            | server is used.                           | |
> +|               | +------------+-------------------------------------------+ |
> ++---------------+------------------------------------------------------------+
> +
> +.. _Version:
> +
> +Versioning and Feature Support
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +Upon accepting a connection, the server must send a VFIO_USER_VERSION
> message
> +proposing a protocol version and a set of capabilities. The client compares
> +these with the versions and capabilities it supports and sends a
> +VFIO_USER_VERSION reply according to the following rules.
> +
> +* The major version in the reply must be the same as proposed. If the client
> +  does not support the proposed major, it closes the connection.
> +* The minor version in the reply must be equal to or less than the minor
> +  version proposed.
> +* The capability list must be a subset of those proposed. If the client
> +  requires a capability the server did not include, it closes the connection.
> +
> +The protocol major version will only change when incompatible protocol
> changes
> +are made, such as changing the message format. The minor version may
> change
> +when compatible changes are made, such as adding new messages or
> capabilities,
> +Both the client and server must support all minor versions less than the
> +maximum minor version it supports. E.g., an implementation that supports
> +version 1.3 must also support 1.0 through 1.2.
> +
> +When making a change to this specification, the protocol version number
> must
> +be included in the form "added in version X.Y"
> +
> +
> +VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP
> +-----------------------------------------
> +
> +Message Format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | MAP=2, UNMAP=3         |
> ++--------------+------------------------+
> +| Message size | 16 + table size        |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| Table        | array of table entries |
> ++--------------+------------------------+
> +
> +This command message is sent by the client to the server to inform it of the
> +memory regions the server can access. It must be sent before the server
> can
> +perform any DMA to the client. It is normally sent directly after the version
> +handshake is completed, but may also occur when memory is added to or
> +subtracted from the client, or if the client uses a vIOMMU. If the client does
> +not expect the server to perform DMA then it does not need to send to the
> +server VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP commands.
> If the server does
> +not need to perform DMA then it can ignore such commands but it must still
> +reply to them. The table is an array of the following structure.  This
> +structure is 32 bytes in size, so the message size is:
> +16 + (# of table entries * 32).
> +
> +VFIO bitmap format
> +^^^^^^^^^^^^^^^^^^^^^^
> +
> ++--------+--------+------+
> +| Name   | Offset | Size |
> ++========+========+======+
> +| pgsize | 0      | 8    |
> ++--------+--------+------+
> +| size   | 8      | 8    |
> ++--------+--------+------+
> +| data   | 16     | 8    |
> ++--------+--------+------+
> +
> +* *pgsize* is the page size for the bitmap, in bytes.
> +* *size* the size for the bitmap, in bytes.
> +* *data* This field is unused in vfio-user.
> +
> +The VFIO bitmap structure is defined in ``<linux/vfio.h>``
> +(``struct vfio_bitmap``).
> +
> +Table entry format
> +^^^^^^^^^^^^^^^^^^
> +
> ++-------------+--------+--------------------------------------------------+
> +| Name        | Offset | Size                                             |
> ++=============+========+=================================
> =================+
> +| Address     | 0      | 8                                                |
> ++-------------+--------+--------------------------------------------------+
> +| Size        | 8      | 8                                                |
> ++-------------+--------+--------------------------------------------------+
> +| Offset      | 16     | 8                                                |
> ++-------------+--------+--------------------------------------------------+
> +| Protections | 24     | 4                                                |
> ++-------------+--------+--------------------------------------------------+
> +| Flags       | 28     | 4                                                |
> ++-------------+--------+--------------------------------------------------+
> +|             | +-----+-------------------------------------------------+ |
> +|             | | Bit | Definition                                      | |
> +|             |
> +=====+=================================================+ |
> +|             | | 0   | Mappable/VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP
> | |
> +|             | +-----+-------------------------------------------------+ |
> ++-------------+--------+--------------------------------------------------+
> +| Data        | 32     | variable                                         |
> ++-------------+--------+--------------------------------------------------+
> +
> +
> +* *Address* is the base DMA address of the region.
> +* *Size* is the size of the region.
> +* *Offset* is the file offset of the region with respect to the associated file
> +  descriptor.
> +* *Protections* are the region's protection attributes as encoded in
> +  ``<sys/mman.h>``.
> +* *Flags* contain the following region attributes:
> +
> +  * *Mappable* indicates that the region can be mapped via the mmap()
> system call
> +    using the file descriptor provided in the message meta-data. This flag is
> +    only valid for VFIO_USER_DMA_MAP.
> +
> +  * *VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP* indicates that a dirty
> page bitmap
> +    must be populated before unmapping the the DMA region. This flag is
> only
> +    valid for VFIO_USER_DMA_UNMAP. The client must provide a
> +    ``struct vfio_bitmap`` in the data field with the ``vfio_bitmap.pgsize``
> +    and ``vfio_bitmap.size`` fields initialized.
> +
> +VFIO_USER_DMA_MAP
> +"""""""""""""""""
> +If a DMA region being added can be directly mapped by the server, an array
> of
> +file descriptors must be sent as part of the message meta-data. Each region
> +entry must have a corresponding file descriptor. On AF_UNIX sockets, the
> file
> +descriptors must be passed as SCM_RIGHTS type ancillary data. Otherwise,
> if a
> +DMA region cannot be directly mapped by the server, it can be accessed by
> the
> +server using VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE
> messages, explained in
> +`Read and Write Operations`_. A command to map over an existing region
> must be
> +failed by the server with ``EEXIST`` set in error field in the reply.
> +
> +VFIO_USER_DMA_UNMAP
> +"""""""""""""""""""
> +Upon receiving a VFIO_USER_DMA_UNMAP command, if the file descriptor
> is mapped
> +then the server must release all references to that DMA region before
> replying,
> +which includes potentially in flight DMA transactions. Removing a portion of
> a
> +DMA region is possible. If the
> VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP bit is set
> +in the request, the server must append to the header the ``struct
> vfio_bitmap``
> +received in the command, followed by the bitmap. Thus, the message size
> the
> +client should is expect is the size of the header plus the size of
> +``struct vfio_bitmap`` plus ``vfio_bitmap.size`` bytes. Each bit in the bitmap
> +represents one page of size ``vfio_bitmap.pgsize``.
> +
> +.. Note::
> + I suppose dirt page logging must have been previously enabled in order for
> the
> + client to be able to use the
> VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP flag?
> +
> +VFIO_USER_DEVICE_GET_INFO
> +-------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+----------------------------+
> +| Name         | Value                      |
> ++==============+============================+
> +| Message ID   | <ID>                       |
> ++--------------+----------------------------+
> +| Command      | 4                          |
> ++--------------+----------------------------+
> +| Message size | 16 in command, 32 in reply |
> ++--------------+----------------------------+
> +| Flags        | Reply bit set in reply     |
> ++--------------+----------------------------+
> +| Error        | 0/errno                    |
> ++--------------+----------------------------+
> +| Device info  | VFIO device info           |
> ++--------------+----------------------------+
> +
> +This command message is sent by the client to the server to query for basic
> +information about the device. Only the message header is needed in the
> command
> +message.  The VFIO device info structure is defined in ``<linux/vfio.h>``
> +(``struct vfio_device_info``).
> +
> +VFIO device info format
> +^^^^^^^^^^^^^^^^^^^^^^^
> +
> ++-------------+--------+--------------------------+
> +| Name        | Offset | Size                     |
> ++=============+========+==========================+
> +| argsz       | 16     | 4                        |
> ++-------------+--------+--------------------------+
> +| flags       | 20     | 4                        |
> ++-------------+--------+--------------------------+
> +|             | +-----+-------------------------+ |
> +|             | | Bit | Definition              | |
> +|             | +=====+=========================+ |
> +|             | | 0   | VFIO_DEVICE_FLAGS_RESET | |
> +|             | +-----+-------------------------+ |
> +|             | | 1   | VFIO_DEVICE_FLAGS_PCI   | |
> +|             | +-----+-------------------------+ |
> ++-------------+--------+--------------------------+
> +| num_regions | 24     | 4                        |
> ++-------------+--------+--------------------------+
> +| num_irqs    | 28     | 4                        |
> ++-------------+--------+--------------------------+
> +
> +* *argsz* is the size of the VFIO device info structure.
> +* *flags* contains the following device attributes.
> +
> +  * VFIO_DEVICE_FLAGS_RESET indicates that the device supports the
> +    VFIO_USER_DEVICE_RESET message.
> +  * VFIO_DEVICE_FLAGS_PCI indicates that the device is a PCI device.
> +
> +* *num_regions* is the number of memory regions that the device
> exposes.
> +* *num_irqs* is the number of distinct interrupt types that the device
> supports.
> +
> +This version of the protocol only supports PCI devices. Additional devices
> may
> +be supported in future versions.
> +
> +VFIO_USER_DEVICE_GET_REGION_INFO
> +--------------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 5                      |
> ++--------------+------------------------+
> +| Message size | 48 + any caps          |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| Region info  | VFIO region info       |
> ++--------------+------------------------+
> +
> +This command message is sent by the client to the server to query for
> +information about device memory regions. The VFIO region info structure is
> +defined in ``<linux/vfio.h>`` (``struct vfio_region_info``). Since the client
> +does not know the size of the capabilities, the size of the reply it should
> +expect is 48 plus any capabilities whose size is indicated in the size field of
> +the reply header.
> +
> +VFIO region info format
> +^^^^^^^^^^^^^^^^^^^^^^^
> +
> ++------------+--------+------------------------------+
> +| Name       | Offset | Size                         |
> ++============+========+==============================+
> +| argsz      | 16     | 4                            |
> ++------------+--------+------------------------------+
> +| flags      | 20     | 4                            |
> ++------------+--------+------------------------------+
> +|            | +-----+-----------------------------+ |
> +|            | | Bit | Definition                  | |
> +|            | +=====+=============================+ |
> +|            | | 0   | VFIO_REGION_INFO_FLAG_READ  | |
> +|            | +-----+-----------------------------+ |
> +|            | | 1   | VFIO_REGION_INFO_FLAG_WRITE | |
> +|            | +-----+-----------------------------+ |
> +|            | | 2   | VFIO_REGION_INFO_FLAG_MMAP  | |
> +|            | +-----+-----------------------------+ |
> +|            | | 3   | VFIO_REGION_INFO_FLAG_CAPS  | |
> +|            | +-----+-----------------------------+ |
> ++------------+--------+------------------------------+
> +| index      | 24     | 4                            |
> ++------------+--------+------------------------------+
> +| cap_offset | 28     | 4                            |
> ++------------+--------+------------------------------+
> +| size       | 32     | 8                            |
> ++------------+--------+------------------------------+
> +| offset     | 40     | 8                            |
> ++------------+--------+------------------------------+
> +
> +* *argsz* is the size of the VFIO region info structure plus the
> +  size of any region capabilities returned.
> +* *flags* are attributes of the region:
> +
> +  * *VFIO_REGION_INFO_FLAG_READ* allows client read access to the
> region.
> +  * *VFIO_REGION_INFO_FLAG_WRITE* allows client write access to the
> region.
> +  * *VFIO_REGION_INFO_FLAG_MMAP* specifies the client can mmap()
> the region.
> +    When this flag is set, the reply will include a file descriptor in its
> +    meta-data. On AF_UNIX sockets, the file descriptors will be passed as
> +    SCM_RIGHTS type ancillary data.
> +  * *VFIO_REGION_INFO_FLAG_CAPS* indicates additional capabilities
> found in the
> +    reply.
> +
> +* *index* is the index of memory region being queried, it is the only field
> +  that is required to be set in the command message.
> +* *cap_offset* describes where additional region capabilities can be found.
> +  cap_offset is relative to the beginning of the VFIO region info structure.
> +  The data structure it points is a VFIO cap header defined in
> +  ``<linux/vfio.h>``.
> +* *size* is the size of the region.
> +* *offset* is the offset given to the mmap() system call for regions with the
> +  MMAP attribute. It is also used as the base offset when mapping a VFIO
> +  sparse mmap area, described below.
> +
> +VFIO Region capabilities
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +The VFIO region information can also include a capabilities list. This list is
> +similar to a PCI capability list - each entry has a common header that
> +identifies a capability and where the next capability in the list can be found.
> +The VFIO capability header format is defined in ``<linux/vfio.h>`` (``struct
> +vfio_info_cap_header``).
> +
> +VFIO cap header format
> +^^^^^^^^^^^^^^^^^^^^^^
> +
> ++---------+--------+------+
> +| Name    | Offset | Size |
> ++=========+========+======+
> +| id      | 0      | 2    |
> ++---------+--------+------+
> +| version | 2      | 2    |
> ++---------+--------+------+
> +| next    | 4      | 4    |
> ++---------+--------+------+
> +
> +* *id* is the capability identity.
> +* *version* is a capability-specific version number.
> +* *next* specifies the offset of the next capability in the capability list. It
> +  is relative to the beginning of the VFIO region info structure.
> +
> +VFIO sparse mmap
> +^^^^^^^^^^^^^^^^
> +
> ++------------------+----------------------------------+
> +| Name             | Value                            |
> ++==================+==================================+
> +| id               | VFIO_REGION_INFO_CAP_SPARSE_MMAP |
> ++------------------+----------------------------------+
> +| version          | 0x1                              |
> ++------------------+----------------------------------+
> +| next             | <next>                           |
> ++------------------+----------------------------------+
> +| sparse mmap info | VFIO region info sparse mmap     |
> ++------------------+----------------------------------+
> +
> +This capability is defined when only a subrange of the region supports
> +direct access by the client via mmap(). The VFIO sparse mmap area is
> defined in
> +``<linux/vfio.h>`` (``struct vfio_region_sparse_mmap_area``).
> +
> +VFIO region info cap sparse mmap
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ++----------+--------+------+
> +| Name     | Offset | Size |
> ++==========+========+======+
> +| nr_areas | 0      | 4    |
> ++----------+--------+------+
> +| reserved | 4      | 4    |
> ++----------+--------+------+
> +| offset   | 8      | 8    |
> ++----------+--------+------+
> +| size     | 16     | 9    |
> ++----------+--------+------+
> +| ...      |        |      |
> ++----------+--------+------+
> +
> +* *nr_areas* is the number of sparse mmap areas in the region.
> +* *offset* and size describe a single area that can be mapped by the client.
> +  There will be nr_areas pairs of offset and size. The offset will be added to
> +  the base offset given in the VFIO_USER_DEVICE_GET_REGION_INFO to
> form the
> +  offset argument of the subsequent mmap() call.
> +
> +The VFIO sparse mmap area is defined in ``<linux/vfio.h>`` (``struct
> +vfio_region_info_cap_sparse_mmap``).
> +
> +VFIO Region Type
> +^^^^^^^^^^^^^^^^
> +
> ++------------------+---------------------------+
> +| Name             | Value                     |
> ++==================+===========================+
> +| id               | VFIO_REGION_INFO_CAP_TYPE |
> ++------------------+---------------------------+
> +| version          | 0x1                       |
> ++------------------+---------------------------+
> +| next             | <next>                    |
> ++------------------+---------------------------+
> +| region info type | VFIO region info type     |
> ++------------------+---------------------------+
> +
> +This capability is defined when a region is specific to the device.
> +
> +VFIO region info type
> +^^^^^^^^^^^^^^^^^^^^^
> +
> +The VFIO region info type is defined in ``<linux/vfio.h>``
> +(``struct vfio_region_info_cap_type``).
> +
> ++---------+--------+------+
> +| Name    | Offset | Size |
> ++=========+========+======+
> +| type    | 0      | 4    |
> ++---------+--------+------+
> +| subtype | 4      | 4    |
> ++---------+--------+------+
> +
> +The only device-specific region type and subtype supported by vfio-user is
> +VFIO_REGION_TYPE_MIGRATION (3) and
> VFIO_REGION_SUBTYPE_MIGRATION (1).
> +
> +VFIO Device Migration Info
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The beginning of the subregion must contain
> +``struct vfio_device_migration_info``, defined in ``<linux/vfio.h>``. This
> +subregion is accessed like any other part of a standard vfio-user PCI region
> +using VFIO_USER_REGION_READ/VFIO_USER_REGION_WRITE.
> +
> ++---------------+--------+-----------------------------+
> +| Name          | Offset | Size                        |
> ++===============+========+=============================+
> +| device_state  | 0      | 4                           |
> ++---------------+--------+-----------------------------+
> +|               | +-----+----------------------------+ |
> +|               | | Bit | Definition                 | |
> +|               | +=====+============================+ |
> +|               | | 0   | VFIO_DEVICE_STATE_RUNNING  | |
> +|               | +-----+----------------------------+ |
> +|               | | 1   | VFIO_DEVICE_STATE_SAVING   | |
> +|               | +-----+----------------------------+ |
> +|               | | 2   | VFIO_DEVICE_STATE_RESUMING | |
> +|               | +-----+----------------------------+ |
> ++---------------+--------+-----------------------------+
> +| reserved      | 4      | 4                           |
> ++---------------+--------+-----------------------------+
> +| pending_bytes | 8      | 8                           |
> ++---------------+--------+-----------------------------+
> +| data_offset   | 16     | 8                           |
> ++---------------+--------+-----------------------------+
> +| data_size     | 24     | 8                           |
> ++---------------+--------+-----------------------------+
> +
> +* *device_state* defines the state of the device:
> +
> +  The client initiates device state transition by writing the intended state.
> +  The server must respond only after it has succesfully transitioned to the
> new
> +  state. If an error occurs then the server must respond to the
> +  VFIO_USER_REGION_WRITE operation with the Error field set accordingly
> and
> +  must remain at the previous state, or in case of internal error it must
> +  transtition to the error state, defined as
> +  VFIO_DEVICE_STATE_RESUMING | VFIO_DEVICE_STATE_SAVING. The
> client must
> +  re-read the device state in order to determine it afresh.
> +
> +  The following device states are defined:
> +
> +  +-----------+---------+----------+-----------------------------------+
> +  | _RESUMING | _SAVING | _RUNNING | Description                       |
> +
> +===========+=========+==========+========================
> ===========+
> +  | 0         | 0       | 0        | Device is stopped.                |
> +  +-----------+---------+----------+-----------------------------------+
> +  | 0         | 0       | 1        | Device is running, default state. |
> +  +-----------+---------+----------+-----------------------------------+
> +  | 0         | 1       | 0        | Stop-and-copy state               |
> +  +-----------+---------+----------+-----------------------------------+
> +  | 0         | 1       | 1        | Pre-copy state                    |
> +  +-----------+---------+----------+-----------------------------------+
> +  | 1         | 0       | 0        | Resuming                          |
> +  +-----------+---------+----------+-----------------------------------+
> +  | 1         | 0       | 1        | Invalid state                     |
> +  +-----------+---------+----------+-----------------------------------+
> +  | 1         | 1       | 1        | Error state                       |
> +  +-----------+---------+----------+-----------------------------------+
> +  | 1         | 1       | 1        | Invalid state                     |
> +  +-----------+---------+----------+-----------------------------------+
> +
> +  Valid state transitions are shown in the following table:
> +
> +  +-------------------------+---------+---------+---------------+----------+----------+
> +  | |darr| From / To |rarr| | Stopped | Running | Stop-and-copy | Pre-copy
> | Resuming |
> +
> +=========================+=========+=========+===========
> ====+==========+==========+
> +  | Stopped                 |    \-   |    0    |       0       |    0     |     0    |
> +  +-------------------------+---------+---------+---------------+----------+----------+
> +  | Running                 |    1    |    \-   |       1       |    1     |     1    |
> +  +-------------------------+---------+---------+---------------+----------+----------+
> +  | Stop-and-copy           |    1    |    0    |       \-      |    0     |     0    |
> +  +-------------------------+---------+---------+---------------+----------+----------+
> +  | Pre-copy                |    0    |    0    |       1       |    \-    |     0    |
> +  +-------------------------+---------+---------+---------------+----------+----------+
> +  | Resuming                |    0    |    1    |       0       |    0     |     \-   |
> +  +-------------------------+---------+---------+---------------+----------+----------+
> +
> +  A device is migrated to the destination as follows:
> +
> +  * The source client transitions the device state from the running state to
> +    the pre-copy state. This transition is optional for the client but must be
> +    supported by the server. The souce server starts sending device state
> data
> +    to the source client through the migration region while the device is
> +    running.
> +
> +  * The source client transitions the device state from the running state or
> the
> +    pre-copy state to the stop-and-copy state. The source server stops the
> +    device, saves device state and sends it to the source client through the
> +    migration region.
> +
> +  The source client is responsible for sending the migration data to the
> +  destination client.
> +
> +  A device is resumed on the destination as follows:
> +
> +  * The destination client transitions the device state from the running state
> +    to the resuming state. The destination server uses the device state data
> +    received through the migration region to resume the device.
> +
> +  * The destination client provides saved device state to the destination
> +    server and then transitions the device to back to the running state.
> +
> +* *reserved* This field is reserved and any access to it must be ignored by
> the
> +  server.
> +
> +* *pending_bytes* Remaining bytes to be migrated by the server. This field
> is
> +  read only.
> +
> +* *data_offset* Offset in the migration region where the client must:
> +
> +  * read from, during the pre-copy or stop-and-copy state, or
> +
> +  * write to, during the resuming state.
> +
> +  This field is read only.
> +
> +* *data_size* Contains the size, in bytes, of the amount of data copied to:
> +
> +  * the source migration region by the source server during the pre-copy or
> +    stop-and copy state, or
> +
> +  * the destination migration region by the destination client during the
> +    resuming state.
> +
> +Device-specific data must be stored at any position after
> +`struct vfio_device_migration_info`. Note that the migration region can be
> +memory mappable, even partially. In practise, only the migration data
> portion
> +can be memory mapped.
> +
> +The client processes device state data during the pre-copy and the
> +stop-and-copy state in the following iterative manner:
> +
> +  1. The client reads `pending_bytes` to mark a new iteration. Repeated
> reads
> +     of this field is an idempotent operation. If there are no migration data
> +     to be consumed then the next step depends on the current device state:
> +
> +     * pre-copy: the client must try again.
> +
> +     * stop-and-copy: this procedure can end and the device can now start
> +       resuming on the destination.
> +
> +  2. The client reads `data_offset`; at thich point the server must make
> +     available a portion of migration data at this offset to be read by the
> +     client, which must happen *before* completing the read operation. The
> +     amount of data to be read must be stored in the `data_size` field, which
> +     the client reads next.
> +
> +  3. The client reads `data_size` to determine the amount of migration data
> +     available.
> +
> +  4. The client reads and processes the migration data.
> +
> +  5. Go to step 1.
> +
> +Note that the client can transition the device from the pre-copy state to the
> +stop-and-copy state at any time; `pending_bytes` does not need to become
> zero.
> +
> +The client initializes the device state on the destination by setting the
> +device state in the resuming state and writing the migration data to the
> +destination migration region at `data_offset` offset. The client can write the
> +source migration data in an iterative manner and the server must consume
> this
> +data before completing each write operation, updating the `data_offset`
> field.
> +The server must apply the source migration data on the device resume
> state. The
> +client must write data on the same order and transction size as read.
> +
> +If an error occurs then the server must fail the read or write operation. It is
> +an implementation detail of the client how to handle errors.
> +
> +VFIO_USER_DEVICE_GET_IRQ_INFO
> +-----------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 6                      |
> ++--------------+------------------------+
> +| Message size | 32                     |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| IRQ info     | VFIO IRQ info          |
> ++--------------+------------------------+
> +
> +This command message is sent by the client to the server to query for
> +information about device interrupt types. The VFIO IRQ info structure is
> +defined in ``<linux/vfio.h>`` (``struct vfio_irq_info``).
> +
> +VFIO IRQ info format
> +^^^^^^^^^^^^^^^^^^^^
> +
> ++-------+--------+---------------------------+
> +| Name  | Offset | Size                      |
> ++=======+========+===========================+
> +| argsz | 16     | 4                         |
> ++-------+--------+---------------------------+
> +| flags | 20     | 4                         |
> ++-------+--------+---------------------------+
> +|       | +-----+--------------------------+ |
> +|       | | Bit | Definition               | |
> +|       | +=====+==========================+ |
> +|       | | 0   | VFIO_IRQ_INFO_EVENTFD    | |
> +|       | +-----+--------------------------+ |
> +|       | | 1   | VFIO_IRQ_INFO_MASKABLE   | |
> +|       | +-----+--------------------------+ |
> +|       | | 2   | VFIO_IRQ_INFO_AUTOMASKED | |
> +|       | +-----+--------------------------+ |
> +|       | | 3   | VFIO_IRQ_INFO_NORESIZE   | |
> +|       | +-----+--------------------------+ |
> ++-------+--------+---------------------------+
> +| index | 24     | 4                         |
> ++-------+--------+---------------------------+
> +| count | 28     | 4                         |
> ++-------+--------+---------------------------+
> +
> +* *argsz* is the size of the VFIO IRQ info structure.
> +* *flags* defines IRQ attributes:
> +
> +  * *VFIO_IRQ_INFO_EVENTFD* indicates the IRQ type can support server
> eventfd
> +    signalling.
> +  * *VFIO_IRQ_INFO_MASKABLE* indicates that the IRQ type supports the
> MASK and
> +    UNMASK actions in a VFIO_USER_DEVICE_SET_IRQS message.
> +  * *VFIO_IRQ_INFO_AUTOMASKED* indicates the IRQ type masks itself
> after being
> +    triggered, and the client must send an UNMASK action to receive new
> +    interrupts.
> +  * *VFIO_IRQ_INFO_NORESIZE* indicates VFIO_USER_SET_IRQS
> operations setup
> +    interrupts as a set, and new sub-indexes cannot be enabled without
> disabling
> +    the entire type.
> +
> +* index is the index of IRQ type being queried, it is the only field that is
> +  required to be set in the command message.
> +* count describes the number of interrupts of the queried type.
> +
> +VFIO_USER_DEVICE_SET_IRQS
> +-------------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 7                      |
> ++--------------+------------------------+
> +| Message size | 36 + any data          |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| IRQ set      | VFIO IRQ set           |
> ++--------------+------------------------+
> +
> +This command message is sent by the client to the server to set actions for
> +device interrupt types. The VFIO IRQ set structure is defined in
> +``<linux/vfio.h>`` (``struct vfio_irq_set``).
> +
> +VFIO IRQ set format
> +^^^^^^^^^^^^^^^^^^^
> +
> ++-------+--------+------------------------------+
> +| Name  | Offset | Size                         |
> ++=======+========+==============================+
> +| argsz | 16     | 4                            |
> ++-------+--------+------------------------------+
> +| flags | 20     | 4                            |
> ++-------+--------+------------------------------+
> +|       | +-----+-----------------------------+ |
> +|       | | Bit | Definition                  | |
> +|       | +=====+=============================+ |
> +|       | | 0   | VFIO_IRQ_SET_DATA_NONE      | |
> +|       | +-----+-----------------------------+ |
> +|       | | 1   | VFIO_IRQ_SET_DATA_BOOL      | |
> +|       | +-----+-----------------------------+ |
> +|       | | 2   | VFIO_IRQ_SET_DATA_EVENTFD   | |
> +|       | +-----+-----------------------------+ |
> +|       | | 3   | VFIO_IRQ_SET_ACTION_MASK    | |
> +|       | +-----+-----------------------------+ |
> +|       | | 4   | VFIO_IRQ_SET_ACTION_UNMASK  | |
> +|       | +-----+-----------------------------+ |
> +|       | | 5   | VFIO_IRQ_SET_ACTION_TRIGGER | |
> +|       | +-----+-----------------------------+ |
> ++-------+--------+------------------------------+
> +| index | 24     | 4                            |
> ++-------+--------+------------------------------+
> +| start | 28     | 4                            |
> ++-------+--------+------------------------------+
> +| count | 32     | 4                            |
> ++-------+--------+------------------------------+
> +| data  | 36     | variable                     |
> ++-------+--------+------------------------------+
> +
> +* *argsz* is the size of the VFIO IRQ set structure, including any *data*
> field.
> +* *flags* defines the action performed on the interrupt range. The DATA
> flags
> +  describe the data field sent in the message; the ACTION flags describe the
> +  action to be performed. The flags are mutually exclusive for both sets.
> +
> +  * *VFIO_IRQ_SET_DATA_NONE* indicates there is no data field in the
> command.
> +    The action is performed unconditionally.
> +  * *VFIO_IRQ_SET_DATA_BOOL* indicates the data field is an array of
> boolean
> +    bytes. The action is performed if the corresponding boolean is true.
> +  * *VFIO_IRQ_SET_DATA_EVENTFD* indicates an array of event file
> descriptors
> +    was sent in the message meta-data. These descriptors will be signalled
> when
> +    the action defined by the action flags occurs. In AF_UNIX sockets, the
> +    descriptors are sent as SCM_RIGHTS type ancillary data.
> +  * *VFIO_IRQ_SET_ACTION_MASK* indicates a masking event. It can be
> used with
> +    VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to mask an
> interrupt, or
> +    with VFIO_IRQ_SET_DATA_EVENTFD to generate an event when the
> guest masks
> +    the interrupt.
> +  * *VFIO_IRQ_SET_ACTION_UNMASK* indicates an unmasking event. It
> can be used
> +    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to
> unmask an
> +    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event
> when the
> +    guest unmasks the interrupt.
> +  * *VFIO_IRQ_SET_ACTION_TRIGGER* indicates a triggering event. It can
> be used
> +    with VFIO_IRQ_SET_DATA_BOOL or VFIO_IRQ_SET_DATA_NONE to
> trigger an
> +    interrupt, or with VFIO_IRQ_SET_DATA_EVENTFD to generate an event
> when the
> +    server triggers the interrupt.
> +
> +* *index* is the index of IRQ type being setup.
> +* *start* is the start of the sub-index being set.
> +* *count* describes the number of sub-indexes being set. As a special case,
> a
> +  count of 0 with data flags of VFIO_IRQ_SET_DATA_NONE disables all
> interrupts
> +  of the index.
> +* *data* is an optional field included when the
> +  VFIO_IRQ_SET_DATA_BOOL flag is present. It contains an array of
> booleans
> +  that specify whether the action is to be performed on the corresponding
> +  index. It's used when the action is only performed on a subset of the range
> +  specified.
> +
> +Not all interrupt types support every combination of data and action flags.
> +The client must know the capabilities of the device and IRQ index before it
> +sends a VFIO_USER_DEVICE_SET_IRQ message.
> +
> +.. _Read and Write Operations:
> +
> +Read and Write Operations
> +-------------------------
> +
> +Not all I/O operations between the client and server can be done via direct
> +access of memory mapped with an mmap() call. In these cases, the client
> and
> +server use messages sent over the socket. It is expected that these
> operations
> +will have lower performance than direct access.
> +
> +The client can access server memory with VFIO_USER_REGION_READ and
> +VFIO_USER_REGION_WRITE commands. These share a common data
> structure that
> +appears after the message header.
> +
> +REGION Read/Write Data
> +^^^^^^^^^^^^^^^^^^^^^^
> +
> ++--------+--------+----------+
> +| Name   | Offset | Size     |
> ++========+========+==========+
> +| Offset | 16     | 8        |
> ++--------+--------+----------+
> +| Region | 24     | 4        |
> ++--------+--------+----------+
> +| Count  | 28     | 4        |
> ++--------+--------+----------+
> +| Data   | 32     | variable |
> ++--------+--------+----------+
> +
> +* *Offset* into the region being accessed.
> +* *Region* is the index of the region being accessed.
> +* *Count* is the size of the data to be transferred.
> +* *Data* is the data to be read or written.
> +
> +The server can access client memory with VFIO_USER_DMA_READ and
> +VFIO_USER_DMA_WRITE messages. These also share a common data
> structure that
> +appears after the message header.
> +
> +DMA Read/Write Data
> +^^^^^^^^^^^^^^^^^^^
> +
> ++---------+--------+----------+
> +| Name    | Offset | Size     |
> ++=========+========+==========+
> +| Address | 16     | 8        |
> ++---------+--------+----------+
> +| Count   | 24     | 4        |
> ++---------+--------+----------+
> +| Data    | 28     | variable |
> ++---------+--------+----------+
> +
> +* *Address* is the area of client memory being accessed. This address must
> have
> +  been previously exported to the server with a VFIO_USER_DMA_MAP
> message.
> +* *Count* is the size of the data to be transferred.
> +* *Data* is the data to be read or written.
> +
> +VFIO_USER_REGION_READ
> +---------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 8                      |
> ++--------------+------------------------+
> +| Message size | 32 + data size         |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| Read info    | REGION read/write data |
> ++--------------+------------------------+
> +
> +This command message is sent from the client to the server to read from
> server
> +memory.  In the command messages, there is no data, and the count is the
> amount
> +of data to be read. The reply message must include the data read, and its
> count
> +field is the amount of data read.
> +
> +VFIO_USER_REGION_WRITE
> +----------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 9                      |
> ++--------------+------------------------+
> +| Message size | 32 + data size         |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| Write info   | REGION read/write data |
> ++--------------+------------------------+
> +
> +This command message is sent from the client to the server to write to
> server
> +memory.  The command message must contain the data to be written, and
> its count
> +field must contain the amount of write data. The count field in the reply
> +message must be zero.
> +
> +VFIO_USER_DMA_READ
> +------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 10                     |
> ++--------------+------------------------+
> +| Message size | 28 + data size         |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| DMA info     | DMA read/write data    |
> ++--------------+------------------------+
> +
> +This command message is sent from the server to the client to read from
> client
> +memory.  In the command message, there is no data, and the count must
> will be
> +the amount of data to be read. The reply message must include the data
> read,
> +and its count field must be the amount of data read.
> +
> +VFIO_USER_DMA_WRITE
> +-------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 11                     |
> ++--------------+------------------------+
> +| Message size | 28 + data size         |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +| DMA info     | DMA read/write data    |
> ++--------------+------------------------+
> +
> +This command message is sent from the server to the client to write to
> server
> +memory.  The command message must contain the data to be written, and
> its count
> +field must contain the amount of write data. The count field in the reply
> +message must be zero.
> +
> +VFIO_USER_VM_INTERRUPT
> +----------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++----------------+------------------------+
> +| Name           | Value                  |
> ++================+========================+
> +| Message ID     | <ID>                   |
> ++----------------+------------------------+
> +| Command        | 12                     |
> ++----------------+------------------------+
> +| Message size   | 20                     |
> ++----------------+------------------------+
> +| Flags          | Reply bit set in reply |
> ++----------------+------------------------+
> +| Error          | 0/errno                |
> ++----------------+------------------------+
> +| Interrupt info | <interrupt>            |
> ++----------------+------------------------+
> +
> +This command message is sent from the server to the client to signal the
> device
> +has raised an interrupt.
> +
> +Interrupt info format
> +^^^^^^^^^^^^^^^^^^^^^
> +
> ++-----------+--------+------+
> +| Name      | Offset | Size |
> ++===========+========+======+
> +| Sub-index | 16     | 4    |
> ++-----------+--------+------+
> +
> +* *Sub-index* is relative to the IRQ index, e.g., the vector number used in
> PCI
> +  MSI/X type interrupts.
> +
> +VFIO_USER_DEVICE_RESET
> +----------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------+------------------------+
> +| Name         | Value                  |
> ++==============+========================+
> +| Message ID   | <ID>                   |
> ++--------------+------------------------+
> +| Command      | 13                     |
> ++--------------+------------------------+
> +| Message size | 16                     |
> ++--------------+------------------------+
> +| Flags        | Reply bit set in reply |
> ++--------------+------------------------+
> +| Error        | 0/errno                |
> ++--------------+------------------------+
> +
> +This command message is sent from the client to the server to reset the
> device.
> +
> +VFIO_USER_DIRY_PAGES
> +--------------------
> +
> +Message format
> +^^^^^^^^^^^^^^
> +
> ++--------------------+------------------------+
> +| Name               | Value                  |
> ++====================+========================+
> +| Message ID         | <ID>                   |
> ++--------------------+------------------------+
> +| Command            | 14                     |
> ++--------------------+------------------------+
> +| Message size       | 16                     |
> ++--------------------+------------------------+
> +| Flags              | Reply bit set in reply |
> ++--------------------+------------------------+
> +| Error              | 0/errno                |
> ++--------------------+------------------------+
> +| VFIO Dirty bitmap  | <dirty bitmap>         |
> ++--------------------+------------------------+
> +
> +This command is analogous to VFIO_IOMMU_DIRTY_PAGES. It is sent by
> the client
> +to the server in order to control logging of dirty pages, usually during a live
> +migration. The VFIO dirty bitmap structure is defined in ``<linux/vfio.h>``
> +(``struct vfio_iommu_type1_dirty_bitmap``).
> +
> +VFIO Dirty Bitmap Format
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +
> ++-------+--------+-----------------------------------------+
> +| Name  | Offset | Size                                    |
> ++=======+========+=======================================
> ==+
> +| argsz | 0      | 4                                       |
> ++-------+--------+-----------------------------------------+
> +| flags | 4      | 4                                       |
> ++-------+--------+-----------------------------------------+
> +|       | +-----+----------------------------------------+ |
> +|       | | Bit | Definition                             | |
> +|       | +=====+========================================+ |
> +|       | | 0   | VFIO_IOMMU_DIRTY_PAGES_FLAG_START      | |
> +|       | +-----+----------------------------------------+ |
> +|       | | 1   | VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP       | |
> +|       | +-----+----------------------------------------+ |
> +|       | | 2   | VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP | |
> +|       | +-----+----------------------------------------+ |
> ++-------+--------+-----------------------------------------+
> +| data  | 8      | 4                                       |
> ++-------+--------+-----------------------------------------+
> +
> +* *argsz* is the size of the VFIO dirty bitmap info structure.
> +
> +* *flags* defines the action to be performed by the server:
> +
> +  * *VFIO_IOMMU_DIRTY_PAGES_FLAG_START* instructs the server to
> start logging
> +    pages it dirties. Logging continues until explicitly disabled by
> +    VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP.
> +
> +  * *VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP* instructs the server to stop
> logging
> +    dirty pages.
> +
> +  * *VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP* requests from the
> server to return
> +    the dirty bitmap for a specific IOVA range. The IOVA range is specified by
> +    "VFIO dirty bitmap get" structure, which must immediatelly follow the
> +    "VFIO dirty bitmap" structure, explained next. This operation is only valid
> +    if logging of dirty pages has been previously started. The server must
> +    respond the same way it does for VFIO_USER_DMA_UNMAP (the dirty
> pages
> +    bitmap must follow the response header).
> +
> +  These flags are mutually exclusive with each other.
> +
> +* *data* This field is unused in vfio-user.
> +
> +VFIO Dirty Bitmap Get Format
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> ++--------+--------+------+
> +| Name   | Offset | Size |
> ++========+========+======+
> +| iova   | 0      | 8    |
> ++--------+--------+------+
> +| size   | 8      | 8    |
> ++--------+--------+------+
> +| bitmap | 16     | 24   |
> ++--------+--------+------+
> +
> +* *iova* is the IOVA offset
> +
> +* *size* is the size of the IOVA region
> +
> +* *bitmap* is the VFIO bitmap (``struct vfio_bitmap``), with the same
> semantics
> +  as VFIO_USER_DMA_UNMAP.
> +
> +
> +Appendices
> +==========
> +
> +Unused VFIO ioctl() commands
> +----------------------------
> +
> +The following VFIO commands do not have an equivalent vfio-user
> command:
> +
> +* VFIO_GET_API_VERSION
> +* VFIO_CHECK_EXTENSION
> +* VFIO_SET_IOMMU
> +* VFIO_GROUP_GET_STATUS
> +* VFIO_GROUP_SET_CONTAINER
> +* VFIO_GROUP_UNSET_CONTAINER
> +* VFIO_GROUP_GET_DEVICE_FD
> +* VFIO_IOMMU_GET_INFO
> +
> +However, once support for live migration for VFIO devices is finalized some
> +of the above commands may have to be handled by the client in their
> +corresponding vfio-user form. This will be addressed in a future protocol
> +version.
> +
> +VFIO groups and containers
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The current VFIO implementation includes group and container idioms that
> +describe how a device relates to the host IOMMU. In the vfio-user
> +implementation, the IOMMU is implemented in SW by the client, and is not
> +visible to the server. The simplest idea would be that the client put each
> +device into its own group and container.
> +
> +Backend Program Conventions
> +---------------------------
> +
> +vfio-user backend program conventions are based on the vhost-user ones.
> +
> +* The backend program must not daemonize itself.
> +* No assumptions must be made as to what access the backend program
> has on the
> +  system.
> +* File descriptors 0, 1 and 2 must exist, must have regular
> +  stdin/stdout/stderr semantics, and can be redirected.
> +* The backend program must honor the SIGTERM signal.
> +* The backend program must accept the following commands line options:
> +
> +  * ``--socket-path=PATH``: path to UNIX domain socket,
> +  * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible
> with
> +    ``--socket-path``
> +* The backend program must be accompanied with a JSON file stored under
> +  ``/usr/share/vfio-user``.
> --
> 2.12.2
> 

On Wed, Oct 28, 2020 at 04:41:31PM +0000, Thanos Makatos wrote:

> FYI here's v5 of the vfio-user protocol, my --cc in git send-email got messed up somehow

Hi Thanos, this looks great, I just had some minor questions below.

> Command Concurrency
> -------------------
> A client may pipeline multiple commands without waiting for previous command
> replies.  The server will process commands in the order they are received.
> A consequence of this is if a client issues a command with the *No_reply* bit,
> then subseqently issues a command without *No_reply*, the older command will
> have been processed before the reply to the younger command is sent by the
> server.  The client must be aware of the device's capability to process concurrent
> commands if pipelining is used.  For example, pipelining allows multiple client
> threads to concurently access device memory; the client must ensure these acceses
> obey device semantics.
> 
> An example is a frame buffer device, where the device may allow concurrent access
> to different areas of video memory, but may have indeterminate behavior if concurrent
> acceses are performed to command or status registers.

Is it valid for an unrelated server->client message to appear in between a
client->server request/reply, or not? And vice versa? Either way, seems useful
for the spec to say.

> |                | +-----+------------+ |
> |                | | Bit | Definition | |
> |                | +=====+============+ |
> |                | | 0-3 | Type       | |
> |                | +-----+------------+ |
> |                | | 4   | No_reply   | |
> |                | +-----+------------+ |
> |                | | 5   | Error      | |
> |                | +-----+------------+ |
> +----------------+--------+-------------+
> | Error          | 12     | 4           |
> +----------------+--------+-------------+
> 
> * *Message ID* identifies the message, and is echoed in the command's reply message.

Is it valid to re-use an ID? When/when not?

>   * *Error* in a reply message indicates the command being acknowledged had
>     an error. In this case, the *Error* field will be valid.
> 
> * *Error* in a reply message is a UNIX errno value. It is reserved in a command message.

I'm not quite following why we need a bit flag and an error field. Do you
anticipate a failure, but with errno==0?

> VFIO_USER_VERSION
> -----------------
> 
> +--------------+------------------------+
> | Message size | 16 + version length    |

Terminating NUL included?

> +--------------+--------+---------------------------------------------------+
> | Name         | Type   | Description                                       |
> +==============+========+===================================================+
> | version      | object | ``{"major": <number>, "minor": <number>}``        |
> |              |        |                                                   |
> |              |        | Version supported by the sender, e.g. "0.1".      |

It seems quite unlikely but this should specify it's strings not floating point
values maybe?

Definitely applies to max_fds too.

> Common capabilities:
> 
> +---------------+------------------------------------------------------------+
> | Name          | Description                                                |
> +===============+============================================================+
> | ``max_fds``   | Maximum number of file descriptors that can be received by |
> |               | the sender. Optional.                                      |

Could specify the meaning when absent?

By array I presume you mean associative array i.e. an Object. Does the whole
thing look like this:

 {
   "major": ..
   "minor": ..
   "capabilities": {
      "max_fds": ..,
      "migration
   }
 }

or something else?

> Versioning and Feature Support
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Upon accepting a connection, the server must send a VFIO_USER_VERSION message
> proposing a protocol version and a set of capabilities. The client compares
> these with the versions and capabilities it supports and sends a
> VFIO_USER_VERSION reply according to the following rules.

I'm curious if there was a specific reason it's this way around, when it seems
more natural for the client to propose first, and the server to reply?

> VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP
> -----------------------------------------

Huge nit, but why are these DMA_*MAP when vfio uses *MAP_DMA ?

> VFIO bitmap format
> * *size* the size for the bitmap, in bytes.

Should this clarify it does *not* include the bitmap header in its size, unlike
other size fields?

> VFIO_USER_DMA_MAP
> """""""""""""""""
> If a DMA region being added can be directly mapped by the server, an array of
> file descriptors must be sent as part of the message meta-data. Each region
> entry must have a corresponding file descriptor.

"Each mappable region entry" ?

> descriptors must be passed as SCM_RIGHTS type ancillary data. Otherwise, if a
> DMA region cannot be directly mapped by the server, it can be accessed by the
> server using VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages, explained in
> `Read and Write Operations`_. A command to map over an existing region must be
> failed by the server with ``EEXIST`` set in error field in the reply.
> 
> VFIO_USER_DMA_UNMAP
> """""""""""""""""""
> Upon receiving a VFIO_USER_DMA_UNMAP command, if the file descriptor is mapped
> then the server must release all references to that DMA region before replying,
> which includes potentially in flight DMA transactions. Removing a portion of a
> DMA region is possible. If the VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP bit is set
> in the request, the server must append to the header the ``struct vfio_bitmap``
> received in the command, followed by the bitmap. Thus, the message size the
> client should is expect is the size of the header plus the size of
> ``struct vfio_bitmap`` plus ``vfio_bitmap.size`` bytes. Each bit in the bitmap
> represents one page of size ``vfio_bitmap.pgsize``.

I'm finding this makes the sizing a bit confusing between map and unmap, could
we may be separate them out, and always define a vfio_bitmap slot for unmap?

Also, shouldn't the client expect sizeof (header) + (nr_table_entries_in_request
* (each vfio_bitmap's size))  in the server's response?

Does the reply header size field reflect this?

> VFIO_USER_DMA_WRITE
> -------------------
> 
> This command message is sent from the server to the client to write to server
> memory.

"write to client memory"?

> VFIO_USER_DIRY_PAGES

Nit, "DIRTY"

thanks
john

> -----Original Message-----
> From: John Levon <levon@movementarian.org>
> Sent: 30 October 2020 17:03
> To: Thanos Makatos <thanos.makatos@nutanix.com>
> Cc: qemu-devel@nongnu.org; benjamin.walker@intel.com; Elena Ufimtseva
> <elena.ufimtseva@oracle.com>; tomassetti.andrea@gmail.com;
> john.g.johnson@oracle.com; jag.raman@oracle.com; Swapnil Ingle
> <swapnil.ingle@nutanix.com>; james.r.harris@intel.com;
> konrad.wilk@oracle.com; yuvalkashtan@gmail.com; dgilbert@redhat.com;
> Raphael Norwitz <raphael.norwitz@nutanix.com>; ismael@linux.com;
> alex.williamson@redhat.com; Kanth.Ghatraju@oracle.com; Stefan Hajnoczi
> <stefanha@redhat.com>; Felipe Franciosi <felipe@nutanix.com>;
> xiuchun.lu@intel.com; Marc-André Lureau
> <marcandre.lureau@redhat.com>; tina.zhang@intel.com;
> changpeng.liu@intel.com
> Subject: Re: [PATCH v5] introduce vfio-user protocol specification
> 
> On Wed, Oct 28, 2020 at 04:41:31PM +0000, Thanos Makatos wrote:
> 
> > FYI here's v5 of the vfio-user protocol, my --cc in git send-email got messed
> up somehow
> 
> Hi Thanos, this looks great, I just had some minor questions below.
> 
> > Command Concurrency
> > -------------------
> > A client may pipeline multiple commands without waiting for previous
> command
> > replies.  The server will process commands in the order they are received.
> > A consequence of this is if a client issues a command with the *No_reply*
> bit,
> > then subseqently issues a command without *No_reply*, the older
> command will
> > have been processed before the reply to the younger command is sent by
> the
> > server.  The client must be aware of the device's capability to process
> concurrent
> > commands if pipelining is used.  For example, pipelining allows multiple
> client
> > threads to concurently access device memory; the client must ensure
> these acceses
> > obey device semantics.
> >
> > An example is a frame buffer device, where the device may allow
> concurrent access
> > to different areas of video memory, but may have indeterminate behavior
> if concurrent
> > acceses are performed to command or status registers.
> 
> Is it valid for an unrelated server->client message to appear in between a
> client->server request/reply, or not? And vice versa? Either way, seems
> useful
> for the spec to say.

Yes, it's valid. I don't see a reason why it shouldn't. I'll update the text to
make it explicit.

> 
> > |                | +-----+------------+ |
> > |                | | Bit | Definition | |
> > |                | +=====+============+ |
> > |                | | 0-3 | Type       | |
> > |                | +-----+------------+ |
> > |                | | 4   | No_reply   | |
> > |                | +-----+------------+ |
> > |                | | 5   | Error      | |
> > |                | +-----+------------+ |
> > +----------------+--------+-------------+
> > | Error          | 12     | 4           |
> > +----------------+--------+-------------+
> >
> > * *Message ID* identifies the message, and is echoed in the command's
> reply message.
> 
> Is it valid to re-use an ID? When/when not?

Yes, it's valid to re-use an ID. I suppose it's also valid, though should be
discouraged, to have multiple outstanding requests with the same ID, even though
that probably doesn't make much sense and will most likely break things. The ID
belongs purely to whomever sends the request, the receiver simply echoes it
back in the response and must make no assumptions about its uniqueness. I think
it's simpler to have it this way, otherwise implementations might start abusing
it or rely on it too much. If there's no objection to these semantics I'll
update the text to clarify.


> 
> >   * *Error* in a reply message indicates the command being acknowledged
> had
> >     an error. In this case, the *Error* field will be valid.
> >
> > * *Error* in a reply message is a UNIX errno value. It is reserved in a
> command message.
> 
> I'm not quite following why we need a bit flag and an error field. Do you
> anticipate a failure, but with errno==0?

Indeed, the Error bit seems redundant. John, is there a reason why we need the
error bit?

> 
> > VFIO_USER_VERSION
> > -----------------
> >
> > +--------------+------------------------+
> > | Message size | 16 + version length    |
> 
> Terminating NUL included?

Good point, in the current libmuser implementation we do include the terminating
NUL, however it's not necessary. I don't have a strong opinion on this one, I'll
update the text to include the terminating NUL just to be correct for now,
however if there's a good argument for/against it we should definitely consider
it.

> 
> > +--------------+--------+---------------------------------------------------+
> > | Name         | Type   | Description                                       |
> >
> +==============+========+=================================
> ==================+
> > | version      | object | ``{"major": <number>, "minor": <number>}``        |
> > |              |        |                                                   |
> > |              |        | Version supported by the sender, e.g. "0.1".      |
> 
> It seems quite unlikely but this should specify it's strings not floating point
> values maybe?
> 
> Definitely applies to max_fds too.


major and minor are JSON numbers and specifically integers. The rationale
behind this is to simplify parsing. Is specifying that major/minor/max_fds
should be an interger sufficient to clear any vagueness here?


> 
> > Common capabilities:
> >
> > +---------------+------------------------------------------------------------+
> > | Name          | Description                                                |
> >
> +===============+=========================================
> ===================+
> > | ``max_fds``   | Maximum number of file descriptors that can be received
> by |
> > |               | the sender. Optional.                                      |
> 
> Could specify the meaning when absent?

Good point. I suppose the receiver must assume max_fds=1 if it's missing.

> 
> By array I presume you mean associative array i.e. an Object. Does the whole
> thing look like this:
> 
>  {
>    "major": ..
>    "minor": ..
>    "capabilities": {
>       "max_fds": ..,
>       "migration
>    }
>  }
> 
> or something else?

Yes, it's an associative array, and you're correct in your example. I'll update
the text.

> 
> > Versioning and Feature Support
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > Upon accepting a connection, the server must send a VFIO_USER_VERSION
> message
> > proposing a protocol version and a set of capabilities. The client compares
> > these with the versions and capabilities it supports and sends a
> > VFIO_USER_VERSION reply according to the following rules.
> 
> I'm curious if there was a specific reason it's this way around, when it seems
> more natural for the client to propose first, and the server to reply?

I'm not aware of any specific reason.

> 
> > VFIO_USER_DMA_MAP and VFIO_USER_DMA_UNMAP
> > -----------------------------------------
> 
> Huge nit, but why are these DMA_*MAP when vfio uses *MAP_DMA ?

We discussed this with John privately but I don't have the details, John do
you remember why we decided to name it like that?

> 
> > VFIO bitmap format
> > * *size* the size for the bitmap, in bytes.
> 
> Should this clarify it does *not* include the bitmap header in its size, unlike
> other size fields?

Yes, definitely, I'll update the text.

> 
> > VFIO_USER_DMA_MAP
> > """""""""""""""""
> > If a DMA region being added can be directly mapped by the server, an array
> of
> > file descriptors must be sent as part of the message meta-data. Each region
> > entry must have a corresponding file descriptor.
> 
> "Each mappable region entry" ?

OK.

> 
> > descriptors must be passed as SCM_RIGHTS type ancillary data. Otherwise,
> if a
> > DMA region cannot be directly mapped by the server, it can be accessed by
> the
> > server using VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE
> messages, explained in
> > `Read and Write Operations`_. A command to map over an existing region
> must be
> > failed by the server with ``EEXIST`` set in error field in the reply.
> >
> > VFIO_USER_DMA_UNMAP
> > """""""""""""""""""
> > Upon receiving a VFIO_USER_DMA_UNMAP command, if the file
> descriptor is mapped
> > then the server must release all references to that DMA region before
> replying,
> > which includes potentially in flight DMA transactions. Removing a portion of
> a
> > DMA region is possible. If the
> VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP bit is set
> > in the request, the server must append to the header the ``struct
> vfio_bitmap``
> > received in the command, followed by the bitmap. Thus, the message size
> the
> > client should is expect is the size of the header plus the size of
> > ``struct vfio_bitmap`` plus ``vfio_bitmap.size`` bytes. Each bit in the bitmap
> > represents one page of size ``vfio_bitmap.pgsize``.
> 
> I'm finding this makes the sizing a bit confusing between map and unmap,
> could
> we may be separate them out, and always define a vfio_bitmap slot for
> unmap?

We can separate them.

Regarding always defining a vfio_bitmap slot for unmap, we need to consider the
case when a vIOMMU is used, where map/unmap messages will be extremely frequent.
If VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP is not used then the vfio_bitmap slot
will be unused, so some bytes will be wasted. If this doesn't really affect
performance then I suppose we can always including it.
 
> 
> Also, shouldn't the client expect sizeof (header) +
> (nr_table_entries_in_request
> * (each vfio_bitmap's size))  in the server's response?
> 
> Does the reply header size field reflect this?

Correct, I'll update the text.


> 
> > VFIO_USER_DMA_WRITE
> > -------------------
> >
> > This command message is sent from the server to the client to write to
> server
> > memory.
> 
> "write to client memory"?
> 
> > VFIO_USER_DIRY_PAGES
> 
> Nit, "DIRTY"

Correct in both cases, I'll update the text.

> 
> thanks
> john

On Mon, Nov 02, 2020 at 11:29:23AM +0000, Thanos Makatos wrote:

> > +==============+========+=================================
> > ==================+
> > > | version      | object | ``{"major": <number>, "minor": <number>}``        |
> > > |              |        |                                                   |
> > > |              |        | Version supported by the sender, e.g. "0.1".      |
> > 
> > It seems quite unlikely but this should specify it's strings not floating point
> > values maybe?
> > 
> > Definitely applies to max_fds too.
> 
> major and minor are JSON numbers and specifically integers.

It is debatable as to whether there is such a thing as a JSON integer :)

> The rationale behind this is to simplify parsing. Is specifying that
> major/minor/max_fds should be an interger sufficient to clear any vagueness
> here?

I suppose that's OK as long as we never want a 0.1.1 or whatever. I'm not sure
it simplifies parsing, but maybe it does.

> > > Versioning and Feature Support
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > Upon accepting a connection, the server must send a VFIO_USER_VERSION
> > message
> > > proposing a protocol version and a set of capabilities. The client compares
> > > these with the versions and capabilities it supports and sends a
> > > VFIO_USER_VERSION reply according to the following rules.
> > 
> > I'm curious if there was a specific reason it's this way around, when it seems
> > more natural for the client to propose first, and the server to reply?
> 
> I'm not aware of any specific reason.

So can we switch it now so the initial setup is a send/recv too?

thanks
john

> -----Original Message-----
> From: Qemu-devel <qemu-devel-
> bounces+thanos.makatos=nutanix.com@nongnu.org> On Behalf Of John
> Levon
> Sent: 02 November 2020 11:41
> To: Thanos Makatos <thanos.makatos@nutanix.com>
> Cc: benjamin.walker@intel.com; Elena Ufimtseva
> <elena.ufimtseva@oracle.com>; jag.raman@oracle.com;
> james.r.harris@intel.com; Swapnil Ingle <swapnil.ingle@nutanix.com>;
> john.g.johnson@oracle.com; yuvalkashtan@gmail.com;
> konrad.wilk@oracle.com; tina.zhang@intel.com; qemu-devel@nongnu.org;
> dgilbert@redhat.com; Marc-André Lureau
> <marcandre.lureau@redhat.com>; ismael@linux.com;
> alex.williamson@redhat.com; Stefan Hajnoczi <stefanha@redhat.com>;
> Felipe Franciosi <felipe@nutanix.com>; xiuchun.lu@intel.com;
> tomassetti.andrea@gmail.com; changpeng.liu@intel.com; Raphael Norwitz
> <raphael.norwitz@nutanix.com>; Kanth.Ghatraju@oracle.com
> Subject: Re: [PATCH v5] introduce vfio-user protocol specification
> 
> On Mon, Nov 02, 2020 at 11:29:23AM +0000, Thanos Makatos wrote:
> 
> > >
> +==============+========+=================================
> > > ==================+
> > > > | version      | object | ``{"major": <number>, "minor": <number>}``
> |
> > > > |              |        |                                                   |
> > > > |              |        | Version supported by the sender, e.g. "0.1".      |
> > >
> > > It seems quite unlikely but this should specify it's strings not floating point
> > > values maybe?
> > >
> > > Definitely applies to max_fds too.
> >
> > major and minor are JSON numbers and specifically integers.
> 
> It is debatable as to whether there is such a thing as a JSON integer :)

AFAIK there isn't.

> 
> > The rationale behind this is to simplify parsing. Is specifying that
> > major/minor/max_fds should be an interger sufficient to clear any
> vagueness
> > here?
> 
> I suppose that's OK as long as we never want a 0.1.1 or whatever. I'm not
> sure
> it simplifies parsing, but maybe it does.

Now that you mention it, why preclude 0.1.1? IIUC the whole point of not
stating the version as a float is to simply have this flexibility in the future.
You're right in your earlier suggestion to explicitly state major/minor as
strings.

> 
> > > > Versioning and Feature Support
> > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > Upon accepting a connection, the server must send a
> VFIO_USER_VERSION
> > > message
> > > > proposing a protocol version and a set of capabilities. The client
> compares
> > > > these with the versions and capabilities it supports and sends a
> > > > VFIO_USER_VERSION reply according to the following rules.
> > >
> > > I'm curious if there was a specific reason it's this way around, when it
> seems
> > > more natural for the client to propose first, and the server to reply?
> >
> > I'm not aware of any specific reason.
> 
> So can we switch it now so the initial setup is a send/recv too?

I'm fine with that but would first like to hear back from John in case he objects.

> On Nov 2, 2020, at 3:51 AM, Thanos Makatos <thanos.makatos@nutanix.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Qemu-devel <qemu-devel-
>> bounces+thanos.makatos=nutanix.com@nongnu.org> On Behalf Of John
>> Levon
>> Sent: 02 November 2020 11:41
>> To: Thanos Makatos <thanos.makatos@nutanix.com>
>> Cc: benjamin.walker@intel.com; Elena Ufimtseva
>> <elena.ufimtseva@oracle.com>; jag.raman@oracle.com;
>> james.r.harris@intel.com; Swapnil Ingle <swapnil.ingle@nutanix.com>;
>> john.g.johnson@oracle.com; yuvalkashtan@gmail.com;
>> konrad.wilk@oracle.com; tina.zhang@intel.com; qemu-devel@nongnu.org;
>> dgilbert@redhat.com; Marc-André Lureau
>> <marcandre.lureau@redhat.com>; ismael@linux.com;
>> alex.williamson@redhat.com; Stefan Hajnoczi <stefanha@redhat.com>;
>> Felipe Franciosi <felipe@nutanix.com>; xiuchun.lu@intel.com;
>> tomassetti.andrea@gmail.com; changpeng.liu@intel.com; Raphael Norwitz
>> <raphael.norwitz@nutanix.com>; Kanth.Ghatraju@oracle.com
>> Subject: Re: [PATCH v5] introduce vfio-user protocol specification
>> 
>> On Mon, Nov 02, 2020 at 11:29:23AM +0000, Thanos Makatos wrote:
>> 
>>>> 
>> +==============+========+=================================
>>>> ==================+
>>>>> | version      | object | ``{"major": <number>, "minor": <number>}``
>> |
>>>>> |              |        |                                                   |
>>>>> |              |        | Version supported by the sender, e.g. "0.1".      |
>>>> 
>>>> It seems quite unlikely but this should specify it's strings not floating point
>>>> values maybe?
>>>> 
>>>> Definitely applies to max_fds too.
>>> 
>>> major and minor are JSON numbers and specifically integers.
>> 
>> It is debatable as to whether there is such a thing as a JSON integer :)
> 
> AFAIK there isn't.
> 
>> 
>>> The rationale behind this is to simplify parsing. Is specifying that
>>> major/minor/max_fds should be an interger sufficient to clear any
>> vagueness
>>> here?
>> 
>> I suppose that's OK as long as we never want a 0.1.1 or whatever. I'm not
>> sure
>> it simplifies parsing, but maybe it does.
> 
> Now that you mention it, why preclude 0.1.1? IIUC the whole point of not
> stating the version as a float is to simply have this flexibility in the future.
> You're right in your earlier suggestion to explicitly state major/minor as
> strings.
> 

	The idea behind the version IDs is to identify incompatible protocol
changes as major versions, and compatible changes as minor versions.  What
would be the purpose of the third version type?

	The thing that makes parsing the JSON easier is knowing the version
beforehand so the parser knows what keys to expect, so I’d like to promote
major and minor to separate fields in the packet from being embedded at an
arbitrary points in the JSON string.


>> 
>>>>> Versioning and Feature Support
>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>> Upon accepting a connection, the server must send a
>> VFIO_USER_VERSION
>>>> message
>>>>> proposing a protocol version and a set of capabilities. The client
>> compares
>>>>> these with the versions and capabilities it supports and sends a
>>>>> VFIO_USER_VERSION reply according to the following rules.
>>>> 
>>>> I'm curious if there was a specific reason it's this way around, when it
>> seems
>>>> more natural for the client to propose first, and the server to reply?
>>> 
>>> I'm not aware of any specific reason.
>> 
>> So can we switch it now so the initial setup is a send/recv too?
> 
> I'm fine with that but would first like to hear back from John in case he objects.


	I think I write that section, and just switched client and server.  The code
is written as client proposes, server responds; this is the better model.

									JJ

On Thu, Nov 05, 2020 at 05:50:27PM -0800, John G Johnson wrote:

> 	The idea behind the version IDs is to identify incompatible protocol
> changes as major versions, and compatible changes as minor versions.  What
> would be the purpose of the third version type?

Well, like any patch version, it'd be for identifying versions on the other side
for reporting, debugging purposes. Not imply anything about the protocol
version. But it's not a big deal.

> 	The thing that makes parsing the JSON easier is knowing the version
> beforehand so the parser knows what keys to expect, so I’d like to promote
> major and minor to separate fields in the packet from being embedded at an
> arbitrary points in the JSON string.

I agree that'd be a sensible change (and then I wonder if the little bit of JSON
is actually useful any more).

> >> So can we switch it now so the initial setup is a send/recv too?
> > 
> > I'm fine with that but would first like to hear back from John in case he objects.
> 
> 
> 	I think I write that section, and just switched client and server.  The code
> is written as client proposes, server responds; this is the better model.

Hah, great, thanks.

regards
john

> -----Original Message-----
> From: John Levon <levon@movementarian.org>
> Sent: 07 November 2020 12:26
> To: John G Johnson <john.g.johnson@oracle.com>
> Cc: Thanos Makatos <thanos.makatos@nutanix.com>;
> benjamin.walker@intel.com; Elena Ufimtseva
> <elena.ufimtseva@oracle.com>; tomassetti.andrea@gmail.com;
> jag.raman@oracle.com; james.r.harris@intel.com; Swapnil Ingle
> <swapnil.ingle@nutanix.com>; yuvalkashtan@gmail.com;
> konrad.wilk@oracle.com; Kanth.Ghatraju@oracle.com; qemu-
> devel@nongnu.org; tina.zhang@intel.com; ismael@linux.com;
> alex.williamson@redhat.com; Stefan Hajnoczi <stefanha@redhat.com>;
> Felipe Franciosi <felipe@nutanix.com>; xiuchun.lu@intel.com; Marc-André
> Lureau <marcandre.lureau@redhat.com>; Raphael Norwitz
> <raphael.norwitz@nutanix.com>; changpeng.liu@intel.com;
> dgilbert@redhat.com
> Subject: Re: [PATCH v5] introduce vfio-user protocol specification
> 
> On Thu, Nov 05, 2020 at 05:50:27PM -0800, John G Johnson wrote:
> 
> > 	The idea behind the version IDs is to identify incompatible protocol
> > changes as major versions, and compatible changes as minor versions.
> What
> > would be the purpose of the third version type?
> 
> Well, like any patch version, it'd be for identifying versions on the other side
> for reporting, debugging purposes. Not imply anything about the protocol
> version. But it's not a big deal.
> 
> > 	The thing that makes parsing the JSON easier is knowing the version
> > beforehand so the parser knows what keys to expect, so I’d like to
> promote
> > major and minor to separate fields in the packet from being embedded at
> an
> > arbitrary points in the JSON string.
> 
> I agree that'd be a sensible change (and then I wonder if the little bit of JSON
> is actually useful any more).

The reason why the JSON string exists is that it simplifies adding information to the version, should we ever need to.

> 
> > >> So can we switch it now so the initial setup is a send/recv too?
> > >
> > > I'm fine with that but would first like to hear back from John in case he
> objects.
> >
> >
> > 	I think I write that section, and just switched client and server.  The
> code
> > is written as client proposes, server responds; this is the better model.
> 
> Hah, great, thanks.
> 
> regards
> john

> > >   * *Error* in a reply message indicates the command being
> acknowledged
> > had
> > >     an error. In this case, the *Error* field will be valid.
> > >
> > > * *Error* in a reply message is a UNIX errno value. It is reserved in a
> > command message.
> >
> > I'm not quite following why we need a bit flag and an error field. Do you
> > anticipate a failure, but with errno==0?
> 
> Indeed, the Error bit seems redundant. John, is there a reason why we need
> the
> error bit?

I haven't heard back from John yet; however I think it makes sense to maintain
an error bit and not necessarily require errno to be set, since there can be
errors that no errno value makes sense. I'll update the spec.