The series contains an implement of a vfio-user client in QEMU, along with a few
more preparatory patches.

The vfio-user protocol allows for implementing (PCI) devices in another
userspace process; SPDK is one example, which includes a virtual NVMe
implementation.

The vfio-user framework consists of 3 parts:
 1) The VFIO user protocol specification.
 2) A client - the VFIO device in QEMU that encapsulates VFIO messages
    and sends them to the server.
 3) A server - a remote process that emulates a device.

This patchset implements parts 1 and 2.

It has been tested against libvfio-user test servers as well as SPDK.

A previous version of this series can be found at
https://lore.kernel.org/qemu-devel/20250520150419.2172078-1-john.levon@nutanix.com/

Changes since v2:

 - rebase to current master
 - code review feedback for region fd change

thanks
john

John Levon (22):
  vfio: export PCI helpers needed for vfio-user
  vfio: enable per-IRQ MSI-X masking
  vfio: add per-region fd support
  vfio: mark posted writes in region write callbacks
  vfio-user: add vfio-user class and container
  vfio-user: connect vfio proxy to remote server
  vfio-user: implement message receive infrastructure
  vfio-user: implement message send infrastructure
  vfio-user: implement VFIO_USER_DEVICE_GET_INFO
  vfio-user: implement VFIO_USER_DEVICE_GET_REGION_INFO
  vfio-user: implement VFIO_USER_REGION_READ/WRITE
  vfio-user: set up PCI in vfio_user_pci_realize()
  vfio-user: implement VFIO_USER_DEVICE_GET/SET_IRQ*
  vfio-user: forward MSI-X PBA BAR accesses to server
  vfio-user: set up container access to the proxy
  vfio-user: implement VFIO_USER_DEVICE_RESET
  vfio-user: implement VFIO_USER_DMA_MAP/UNMAP
  vfio-user: implement VFIO_USER_DMA_READ/WRITE
  vfio-user: add 'x-msg-timeout' option
  vfio-user: support posted writes
  vfio-user: add coalesced posted writes
  docs: add vfio-user documentation

Thanos Makatos (1):
  vfio-user: introduce vfio-user protocol specification

 MAINTAINERS                           |   10 +-
 docs/devel/index-internals.rst        |    1 +
 docs/devel/vfio-user.rst              | 1522 +++++++++++++++++++++++++
 docs/system/device-emulation.rst      |    1 +
 docs/system/devices/vfio-user.rst     |   24 +
 meson.build                           |    1 +
 hw/vfio-user/container.h              |   27 +
 hw/vfio-user/device.h                 |   27 +
 hw/vfio-user/protocol.h               |  245 ++++
 hw/vfio-user/proxy.h                  |  137 +++
 hw/vfio-user/trace.h                  |    1 +
 hw/vfio/pci.h                         |   13 +
 include/hw/vfio/vfio-container-base.h |    1 +
 include/hw/vfio/vfio-device.h         |   15 +-
 include/hw/vfio/vfio-region.h         |    1 +
 hw/vfio-user/container.c              |  348 ++++++
 hw/vfio-user/device.c                 |  389 +++++++
 hw/vfio-user/pci.c                    |  438 +++++++
 hw/vfio-user/proxy.c                  | 1314 +++++++++++++++++++++
 hw/vfio/device.c                      |   58 +-
 hw/vfio/pci.c                         |   99 +-
 hw/vfio/region.c                      |   12 +-
 hw/meson.build                        |    1 +
 hw/vfio-user/meson.build              |   11 +
 hw/vfio-user/trace-events             |   18 +
 hw/vfio/trace-events                  |    6 +-
 meson_options.txt                     |    2 +
 scripts/meson-buildoptions.sh         |    4 +
 28 files changed, 4679 insertions(+), 47 deletions(-)
 create mode 100644 docs/devel/vfio-user.rst
 create mode 100644 docs/system/devices/vfio-user.rst
 create mode 100644 hw/vfio-user/container.h
 create mode 100644 hw/vfio-user/device.h
 create mode 100644 hw/vfio-user/protocol.h
 create mode 100644 hw/vfio-user/proxy.h
 create mode 100644 hw/vfio-user/trace.h
 create mode 100644 hw/vfio-user/container.c
 create mode 100644 hw/vfio-user/device.c
 create mode 100644 hw/vfio-user/pci.c
 create mode 100644 hw/vfio-user/proxy.c
 create mode 100644 hw/vfio-user/meson.build
 create mode 100644 hw/vfio-user/trace-events

-- 
2.43.0

John,

On 6/7/25 02:10, John Levon wrote:
> The series contains an implement of a vfio-user client in QEMU, along with a few
> more preparatory patches.
> 
> The vfio-user protocol allows for implementing (PCI) devices in another
> userspace process; SPDK is one example, which includes a virtual NVMe
> implementation.
> 
> The vfio-user framework consists of 3 parts:
>   1) The VFIO user protocol specification.
>   2) A client - the VFIO device in QEMU that encapsulates VFIO messages
>      and sends them to the server.
>   3) A server - a remote process that emulates a device.
> 
> This patchset implements parts 1 and 2.
> 
> It has been tested against libvfio-user test servers as well as SPDK.

Is libvfio-user a server-only library ? It seems a bit odd not to use it
on the client side. Anyhow,


We have now merged upstream all the prerequisite changes for vfio-user.
This gives us about 1 month to fix regressions and merge the model for
the vfio-user-pci device, its documentation and tests.


I *really* would prefer if we had functional tests although I don't how
complex it would be. At least an initial framework would be good to have.
More sophisticated scenarios, such as mappings and interrupts, can come
later.

Thanks,

C.

On Thu, Jun 12, 2025 at 08:57:54AM +0200, Cédric Le Goater wrote:

> On 6/7/25 02:10, John Levon wrote:
> > The series contains an implement of a vfio-user client in QEMU, along with a few
> > more preparatory patches.
> > 
> > The vfio-user protocol allows for implementing (PCI) devices in another
> > userspace process; SPDK is one example, which includes a virtual NVMe
> > implementation.
> > 
> > The vfio-user framework consists of 3 parts:
> >   1) The VFIO user protocol specification.
> >   2) A client - the VFIO device in QEMU that encapsulates VFIO messages
> >      and sends them to the server.
> >   3) A server - a remote process that emulates a device.
> > 
> > This patchset implements parts 1 and 2.
> > 
> > It has been tested against libvfio-user test servers as well as SPDK.
> 
> Is libvfio-user a server-only library ?

Yes.

>It seems a bit odd not to use it on the client side.

I wasn't around for any original discussion here, so don't have context, but
there are two clients I know of today: qemu and cloud-hypervisor. The latter is
in Rust, and the former is opinionated enough that I'm not sure the message
handling part (the only useful bit of the library here) would be a good match.

Also I rather like having more than one implementation of a protocol, anyway.

> I *really* would prefer if we had functional tests although I don't how

I agree it makes sense to have a working functional test before merging.

In the meantime, I would like to continue with review of the main chunk - as you
said, there's a lot of code. I've not made substantial changes since v3 (only
the comments so far), so I'm happy to continue with v3 review, or if you prefer,
I can send out v4 against master. Please let me know.

regards
john

Hi John,

>> I *really* would prefer if we had functional tests although I don't how
> 
> I agree it makes sense to have a working functional test before merging.

Any progress ?
  
> In the meantime, I would like to continue with review of the main chunk - as you
> said, there's a lot of code. I've not made substantial changes since v3 (only
> the comments so far), so I'm happy to continue with v3 review, or if you prefer,
> I can send out v4 against master. Please let me know.

I doubt we will have much more reviews on the vfio-user part. Can you
please send v4 ?

Thanks,

C.

On 19/06/2025 12:56, Cédric Le Goater wrote:

> Hi John,
> 
>>> I *really* would prefer if we had functional tests although I don't how
>>
>> I agree it makes sense to have a working functional test before merging.
> 
> Any progress ?

I've had to look at some other bits and pieces this week, but the basic 
status is sadly still unchanged - I have a test with 2 VMs, a client and 
server, that works fine when run by hand yet hangs when run within the 
functional test framework.

I do now have a VM with lots of extra debugging installed, but figuring 
out why I see these hangs is proving to be quite time intensive. If we 
could point you towards a manual image and some scripts, would that be 
sufficient for now?

>> In the meantime, I would like to continue with review of the main 
>> chunk - as you
>> said, there's a lot of code. I've not made substantial changes since 
>> v3 (only
>> the comments so far), so I'm happy to continue with v3 review, or if 
>> you prefer,
>> I can send out v4 against master. Please let me know.
> 
> I doubt we will have much more reviews on the vfio-user part. Can you
> please send v4 ?


ATB,

Mark.

Hello Mark,

On 6/20/25 10:20, Mark Cave-Ayland wrote:
> On 19/06/2025 12:56, Cédric Le Goater wrote:
> 
>> Hi John,
>>
>>>> I *really* would prefer if we had functional tests although I don't how
>>>
>>> I agree it makes sense to have a working functional test before merging.
>>
>> Any progress ?
> 
> I've had to look at some other bits and pieces this week, but the basic status is sadly still unchanged - I have a test with 2 VMs, a client and server, that works fine when run by hand yet hangs when run within the functional test framework.
> 
> I do now have a VM with lots of extra debugging installed, but figuring out why I see these hangs is proving to be quite time intensive. If we could point you towards a manual image and some scripts, would that be sufficient for now?

Before merging, I would like to be able to experiment a minimum.

Does a dummy device (server side) implementation exist ? and a
GH repo I could pull the code from.

John,

How do you test ?

Thanks,

C.

On Fri, Jun 20, 2025 at 10:32:10AM +0200, Cédric Le Goater wrote:

> Before merging, I would like to be able to experiment a minimum.
> 
> Does a dummy device (server side) implementation exist ? and a
> GH repo I could pull the code from.
> 
> John,
> 
> How do you test ?

I use these scripts for smoke-testing:

https://github.com/jlevon/grot/blob/master/tests/vfio-user/run.sh
is easiest - you just need to compile libvfio-user and qemu
and you should be good if you have a suitable Linux image.

See https://github.com/nutanix/libvfio-user/blob/master/docs/qemu.md
if you need an image that has the relevant kernel module, and for some basic
testing instructions.

https://github.com/jlevon/grot/blob/master/tests/vfio-user/spdk-run.sh
is a more complete test - covers mmaps both ways etc. You can test this by
interacting with the vNVMe device inside your VM.

See https://github.com/nutanix/libvfio-user/blob/master/docs/spdk.md

The qemu and live migration sections there are obviously stale.

Please let me know if you hit problems.

regards
john

Hello,

On 6/21/25 14:22, John Levon wrote:
> On Fri, Jun 20, 2025 at 10:32:10AM +0200, Cédric Le Goater wrote:
> 
>> Before merging, I would like to be able to experiment a minimum.
>>
>> Does a dummy device (server side) implementation exist ? and a
>> GH repo I could pull the code from.
>>
>> John,
>>
>> How do you test ?
> 
> I use these scripts for smoke-testing:
> 
> https://github.com/jlevon/grot/blob/master/tests/vfio-user/run.sh
> is easiest - you just need to compile libvfio-user and qemu
> and you should be good if you have a suitable Linux image.

PCI enumeration work but the kernel module won't load on ubuntu 24.04
and 25.04 :

root@ubuntu:~# lspci -s 00:0d.0 -k -vv
00:0d.0 Non-VGA unclassified device: ACCES I/O Products, Inc. PCI-IDIO-16 Isolated Digital Input / FET Output Card
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 21
	Region 2: I/O ports at 6000 [size=256]
	Kernel modules: gpio_pci_idio_16

root@ubuntu:~# dmesg | grep 0000:00:0d.0
[    0.381550] pci 0000:00:0d.0: [494f:0dc8] type 00 class 0x000000 conventional PCI endpoint
[    0.382696] pci 0000:00:0d.0: BAR 2 [io  0x6000-0x60ff]
[    4.017654] pci-idio-16 0000:00:0d.0: error -EINVAL: Unable to initialize register map
[    4.027063] pci-idio-16 0000:00:0d.0: probe with driver pci-idio-16 failed with error -22

Any idea why ?

> 
> See https://github.com/nutanix/libvfio-user/blob/master/docs/qemu.md
> if you need an image that has the relevant kernel module, and for some basic
> testing instructions.
> 
> https://github.com/jlevon/grot/blob/master/tests/vfio-user/spdk-run.sh
> is a more complete test - covers mmaps both ways etc. You can test this by
> interacting with the vNVMe device inside your VM.
> 
> See https://github.com/nutanix/libvfio-user/blob/master/docs/spdk.md
I didn't try SPDK yet.

Thanks,

C.

On 22/06/2025 13:57, Cédric Le Goater wrote:

> Hello,
> 
> On 6/21/25 14:22, John Levon wrote:
>> On Fri, Jun 20, 2025 at 10:32:10AM +0200, Cédric Le Goater wrote:
>>
>>> Before merging, I would like to be able to experiment a minimum.
>>>
>>> Does a dummy device (server side) implementation exist ? and a
>>> GH repo I could pull the code from.
>>>
>>> John,
>>>
>>> How do you test ?
>>
>> I use these scripts for smoke-testing:
>>
>> https://urldefense.proofpoint.com/v2/url? 
>> u=https-3A__github.com_jlevon_grot_blob_master_tests_vfio-2Duser_run.sh&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=V-sTFYIBuNsRDS6PAkkLHCjd5V2lvtIYfbhCV4BsWHxVWzLB_Dj21kOCZ_00B8wZ&s=ZZtYgH13tGNzKPUfKumHJno3qZjfUTg5z6GMJ_9o5so&e= is easiest - you just need to compile libvfio-user and qemu
>> and you should be good if you have a suitable Linux image.
> 
> PCI enumeration work but the kernel module won't load on ubuntu 24.04
> and 25.04 :
> 
> root@ubuntu:~# lspci -s 00:0d.0 -k -vv
> 00:0d.0 Non-VGA unclassified device: ACCES I/O Products, Inc. PCI- 
> IDIO-16 Isolated Digital Input / FET Output Card
>      Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Interrupt: pin A routed to IRQ 21
>      Region 2: I/O ports at 6000 [size=256]
>      Kernel modules: gpio_pci_idio_16
> 
> root@ubuntu:~# dmesg | grep 0000:00:0d.0
> [    0.381550] pci 0000:00:0d.0: [494f:0dc8] type 00 class 0x000000 
> conventional PCI endpoint
> [    0.382696] pci 0000:00:0d.0: BAR 2 [io  0x6000-0x60ff]
> [    4.017654] pci-idio-16 0000:00:0d.0: error -EINVAL: Unable to 
> initialize register map
> [    4.027063] pci-idio-16 0000:00:0d.0: probe with driver pci-idio-16 
> failed with error -22
> 
> Any idea why ?

Annoyingly this is because the gpio_pci_idio_16 module is currently 
broken upstream :(  The last LTS I was able to able to successfully test 
here was 6.1, and so that's what I've used to generate my functional 
test images.


ATB,

Mark.

On 6/23/25 18:09, Mark Cave-Ayland wrote:
> On 22/06/2025 13:57, Cédric Le Goater wrote:
> 
>> Hello,
>>
>> On 6/21/25 14:22, John Levon wrote:
>>> On Fri, Jun 20, 2025 at 10:32:10AM +0200, Cédric Le Goater wrote:
>>>
>>>> Before merging, I would like to be able to experiment a minimum.
>>>>
>>>> Does a dummy device (server side) implementation exist ? and a
>>>> GH repo I could pull the code from.
>>>>
>>>> John,
>>>>
>>>> How do you test ?
>>>
>>> I use these scripts for smoke-testing:
>>>
>>> https://urldefense.proofpoint.com/v2/url? u=https-3A__github.com_jlevon_grot_blob_master_tests_vfio-2Duser_run.sh&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=V-sTFYIBuNsRDS6PAkkLHCjd5V2lvtIYfbhCV4BsWHxVWzLB_Dj21kOCZ_00B8wZ&s=ZZtYgH13tGNzKPUfKumHJno3qZjfUTg5z6GMJ_9o5so&e= is easiest - you just need to compile libvfio-user and qemu
>>> and you should be good if you have a suitable Linux image.
>>
>> PCI enumeration work but the kernel module won't load on ubuntu 24.04
>> and 25.04 :
>>
>> root@ubuntu:~# lspci -s 00:0d.0 -k -vv
>> 00:0d.0 Non-VGA unclassified device: ACCES I/O Products, Inc. PCI- IDIO-16 Isolated Digital Input / FET Output Card
>>      Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
>>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>      Interrupt: pin A routed to IRQ 21
>>      Region 2: I/O ports at 6000 [size=256]
>>      Kernel modules: gpio_pci_idio_16
>>
>> root@ubuntu:~# dmesg | grep 0000:00:0d.0
>> [    0.381550] pci 0000:00:0d.0: [494f:0dc8] type 00 class 0x000000 conventional PCI endpoint
>> [    0.382696] pci 0000:00:0d.0: BAR 2 [io  0x6000-0x60ff]
>> [    4.017654] pci-idio-16 0000:00:0d.0: error -EINVAL: Unable to initialize register map
>> [    4.027063] pci-idio-16 0000:00:0d.0: probe with driver pci-idio-16 failed with error -22
>>
>> Any idea why ?
> 
> Annoyingly this is because the gpio_pci_idio_16 module is currently 
> broken upstream :(  

Is is a known issue ? the gpio_pci_idio_16 module seems well maintained.

> The last LTS I was able to able to successfully test here was 6.1, 
> and so that's what I've used to generate my functional test images.

All seems to work fine with the debian-12-nocloud-amd64.qcow2 imageand the gpio-pci-idio-16 kmod is even provided by default.


It would be nice to be able generate some interrupts

Thanks,

C.

On Sun, Jun 22, 2025 at 02:57:57PM +0200, Cédric Le Goater wrote:

> PCI enumeration work but the kernel module won't load on ubuntu 24.04

Sorry, I forgot - there's a kernel bug in the Ubuntu LTS kernel for this driver.
Mark has a fix - Mark, can you share?

regards
john

On 23/06/2025 09:36, John Levon wrote:

> On Sun, Jun 22, 2025 at 02:57:57PM +0200, Cédric Le Goater wrote:
> 
>> PCI enumeration work but the kernel module won't load on ubuntu 24.04
> 
> Sorry, I forgot - there's a kernel bug in the Ubuntu LTS kernel for this driver.
> Mark has a fix - Mark, can you share?

I did have a bit of a dig into this a few weeks ago, and I think there 
are at least 2 bugs related to recent changes to the kernel regmap code. 
I was able to persuade the module to load with a couple of hacks, but I 
think there was still something else required to get the basic test up 
and running...


ATB,

Mark.