virtio, vhost: Add VIRTIO_F_NOTIFICATION_DATA support

[PATCH v3 for 9.1 0/6] virtio, vhost: Add VIRTIO_F_NOTIFICATION_DATA support

Posted by Jonah Palmer 1 month, 2 weeks ago

The goal of these patches are to add support to a variety of virtio and
vhost devices for the VIRTIO_F_NOTIFICATION_DATA transport feature. This
feature indicates that a driver will pass extra data (instead of just a
virtqueue's index) when notifying the corresponding device.

The data passed in by the driver when this feature is enabled varies in
format depending on if the device is using a split or packed virtqueue
layout:

 Split VQ
  - Upper 16 bits: shadow_avail_idx
  - Lower 16 bits: virtqueue index

 Packed VQ
  - Upper 16 bits: 1-bit wrap counter & 15-bit shadow_avail_idx
  - Lower 16 bits: virtqueue index

Also, due to the limitations of ioeventfd not being able to carry the
extra provided by the driver, having both VIRTIO_F_NOTIFICATION_DATA
feature and ioeventfd enabled is a functional mismatch. The user must
explicitly disable ioeventfd for the device in the Qemu arguments when
using this feature, else the device will fail to complete realization.

For example, a device must explicitly enable notification_data as well
as disable ioeventfd:

    -device virtio-scsi-pci,...,ioeventfd=off,notification_data=on

A significant aspect of this effort has been to maintain compatibility
across different backends. As such, the feature is offered by backend
devices only when supported, with fallback mechanisms where backend
support is absent.

v3: Validate VQ idx via. virtio_queue_get_num() (pci, mmio, ccw)
    Rename virtio_queue_set_shadow_avail_data
    Only pass in upper 16 bits of 32-bit extra data (was redundant)
    Make notification compatibility check function static
    Drop tags on patches 1/6, 3/6, and 4/6

v2: Don't disable ioeventfd by default, user must disable it
    Drop tags on patch 2/6

Jonah Palmer (6):
  virtio/virtio-pci: Handle extra notification data
  virtio: Prevent creation of device using notification-data with ioeventfd
  virtio-mmio: Handle extra notification data
  virtio-ccw: Handle extra notification data
  vhost/vhost-user: Add VIRTIO_F_NOTIFICATION_DATA to vhost feature bits
  virtio: Add VIRTIO_F_NOTIFICATION_DATA property definition

 hw/block/vhost-user-blk.c    |  1 +
 hw/net/vhost_net.c           |  2 ++
 hw/s390x/s390-virtio-ccw.c   | 17 +++++++++++----
 hw/scsi/vhost-scsi.c         |  1 +
 hw/scsi/vhost-user-scsi.c    |  1 +
 hw/virtio/vhost-user-fs.c    |  2 +-
 hw/virtio/vhost-user-vsock.c |  1 +
 hw/virtio/virtio-mmio.c      | 10 +++++++--
 hw/virtio/virtio-pci.c       | 11 +++++++---
 hw/virtio/virtio.c           | 40 ++++++++++++++++++++++++++++++++++++
 include/hw/virtio/virtio.h   |  6 +++++-
 net/vhost-vdpa.c             |  1 +
 12 files changed, 82 insertions(+), 11 deletions(-)

-- 
2.39.3

Re: [PATCH v3 for 9.1 0/6] virtio, vhost: Add VIRTIO_F_NOTIFICATION_DATA support

Posted by Jiri Pirko 1 month, 2 weeks ago

Fri, Mar 15, 2024 at 05:55:51PM CET, jonah.palmer@oracle.com wrote:
>The goal of these patches are to add support to a variety of virtio and
>vhost devices for the VIRTIO_F_NOTIFICATION_DATA transport feature. This
>feature indicates that a driver will pass extra data (instead of just a
>virtqueue's index) when notifying the corresponding device.
>
>The data passed in by the driver when this feature is enabled varies in
>format depending on if the device is using a split or packed virtqueue
>layout:
>
> Split VQ
>  - Upper 16 bits: shadow_avail_idx
>  - Lower 16 bits: virtqueue index
>
> Packed VQ
>  - Upper 16 bits: 1-bit wrap counter & 15-bit shadow_avail_idx
>  - Lower 16 bits: virtqueue index
>
>Also, due to the limitations of ioeventfd not being able to carry the
>extra provided by the driver, having both VIRTIO_F_NOTIFICATION_DATA
>feature and ioeventfd enabled is a functional mismatch. The user must
>explicitly disable ioeventfd for the device in the Qemu arguments when
>using this feature, else the device will fail to complete realization.
>
>For example, a device must explicitly enable notification_data as well
>as disable ioeventfd:
>
>    -device virtio-scsi-pci,...,ioeventfd=off,notification_data=on
>
>A significant aspect of this effort has been to maintain compatibility
>across different backends. As such, the feature is offered by backend
>devices only when supported, with fallback mechanisms where backend
>support is absent.
>
>v3: Validate VQ idx via. virtio_queue_get_num() (pci, mmio, ccw)
>    Rename virtio_queue_set_shadow_avail_data
>    Only pass in upper 16 bits of 32-bit extra data (was redundant)
>    Make notification compatibility check function static
>    Drop tags on patches 1/6, 3/6, and 4/6
>
>v2: Don't disable ioeventfd by default, user must disable it
>    Drop tags on patch 2/6
>
>Jonah Palmer (6):
>  virtio/virtio-pci: Handle extra notification data
>  virtio: Prevent creation of device using notification-data with ioeventfd
>  virtio-mmio: Handle extra notification data
>  virtio-ccw: Handle extra notification data
>  vhost/vhost-user: Add VIRTIO_F_NOTIFICATION_DATA to vhost feature bits
>  virtio: Add VIRTIO_F_NOTIFICATION_DATA property definition

Jonah, do you have kernel patches to add this feature as well?

Thanks!

Re: [PATCH v3 for 9.1 0/6] virtio, vhost: Add VIRTIO_F_NOTIFICATION_DATA support

Posted by Jonah Palmer 1 month, 2 weeks ago


On 3/16/24 11:45 AM, Jiri Pirko wrote:
> Fri, Mar 15, 2024 at 05:55:51PM CET, jonah.palmer@oracle.com wrote:
>> The goal of these patches are to add support to a variety of virtio and
>> vhost devices for the VIRTIO_F_NOTIFICATION_DATA transport feature. This
>> feature indicates that a driver will pass extra data (instead of just a
>> virtqueue's index) when notifying the corresponding device.
>>
>> The data passed in by the driver when this feature is enabled varies in
>> format depending on if the device is using a split or packed virtqueue
>> layout:
>>
>> Split VQ
>>   - Upper 16 bits: shadow_avail_idx
>>   - Lower 16 bits: virtqueue index
>>
>> Packed VQ
>>   - Upper 16 bits: 1-bit wrap counter & 15-bit shadow_avail_idx
>>   - Lower 16 bits: virtqueue index
>>
>> Also, due to the limitations of ioeventfd not being able to carry the
>> extra provided by the driver, having both VIRTIO_F_NOTIFICATION_DATA
>> feature and ioeventfd enabled is a functional mismatch. The user must
>> explicitly disable ioeventfd for the device in the Qemu arguments when
>> using this feature, else the device will fail to complete realization.
>>
>> For example, a device must explicitly enable notification_data as well
>> as disable ioeventfd:
>>
>>     -device virtio-scsi-pci,...,ioeventfd=off,notification_data=on
>>
>> A significant aspect of this effort has been to maintain compatibility
>> across different backends. As such, the feature is offered by backend
>> devices only when supported, with fallback mechanisms where backend
>> support is absent.
>>
>> v3: Validate VQ idx via. virtio_queue_get_num() (pci, mmio, ccw)
>>     Rename virtio_queue_set_shadow_avail_data
>>     Only pass in upper 16 bits of 32-bit extra data (was redundant)
>>     Make notification compatibility check function static
>>     Drop tags on patches 1/6, 3/6, and 4/6
>>
>> v2: Don't disable ioeventfd by default, user must disable it
>>     Drop tags on patch 2/6
>>
>> Jonah Palmer (6):
>>   virtio/virtio-pci: Handle extra notification data
>>   virtio: Prevent creation of device using notification-data with ioeventfd
>>   virtio-mmio: Handle extra notification data
>>   virtio-ccw: Handle extra notification data
>>   vhost/vhost-user: Add VIRTIO_F_NOTIFICATION_DATA to vhost feature bits
>>   virtio: Add VIRTIO_F_NOTIFICATION_DATA property definition
> 
> Jonah, do you have kernel patches to add this feature as well?
> 
> Thanks!

Hi Jiri! I think there are already kernel patches for 
VIRTIO_F_NOTIFICATION_DATA, unless you're referring to something more 
specific that wasn't included in these patches:

[1]: virtio: add VIRTIO_F_NOTIFICATION_DATA feature support
https://lore.kernel.org/lkml/20230324195029.2410503-1-viktor@daynix.com/

[2]: virtio-vdpa: add VIRTIO_F_NOTIFICATION_DATA feature support
https://lore.kernel.org/lkml/20230413081855.36643-3-alvaro.karsz@solid-run.com/

Jonah

Re: [PATCH v3 for 9.1 0/6] virtio, vhost: Add VIRTIO_F_NOTIFICATION_DATA support

Posted by Jiri Pirko 1 month, 2 weeks ago

Mon, Mar 18, 2024 at 12:22:02PM CET, jonah.palmer@oracle.com wrote:
>
>
>On 3/16/24 11:45 AM, Jiri Pirko wrote:
>> Fri, Mar 15, 2024 at 05:55:51PM CET, jonah.palmer@oracle.com wrote:
>> > The goal of these patches are to add support to a variety of virtio and
>> > vhost devices for the VIRTIO_F_NOTIFICATION_DATA transport feature. This
>> > feature indicates that a driver will pass extra data (instead of just a
>> > virtqueue's index) when notifying the corresponding device.
>> > 
>> > The data passed in by the driver when this feature is enabled varies in
>> > format depending on if the device is using a split or packed virtqueue
>> > layout:
>> > 
>> > Split VQ
>> >   - Upper 16 bits: shadow_avail_idx
>> >   - Lower 16 bits: virtqueue index
>> > 
>> > Packed VQ
>> >   - Upper 16 bits: 1-bit wrap counter & 15-bit shadow_avail_idx
>> >   - Lower 16 bits: virtqueue index
>> > 
>> > Also, due to the limitations of ioeventfd not being able to carry the
>> > extra provided by the driver, having both VIRTIO_F_NOTIFICATION_DATA
>> > feature and ioeventfd enabled is a functional mismatch. The user must
>> > explicitly disable ioeventfd for the device in the Qemu arguments when
>> > using this feature, else the device will fail to complete realization.
>> > 
>> > For example, a device must explicitly enable notification_data as well
>> > as disable ioeventfd:
>> > 
>> >     -device virtio-scsi-pci,...,ioeventfd=off,notification_data=on
>> > 
>> > A significant aspect of this effort has been to maintain compatibility
>> > across different backends. As such, the feature is offered by backend
>> > devices only when supported, with fallback mechanisms where backend
>> > support is absent.
>> > 
>> > v3: Validate VQ idx via. virtio_queue_get_num() (pci, mmio, ccw)
>> >     Rename virtio_queue_set_shadow_avail_data
>> >     Only pass in upper 16 bits of 32-bit extra data (was redundant)
>> >     Make notification compatibility check function static
>> >     Drop tags on patches 1/6, 3/6, and 4/6
>> > 
>> > v2: Don't disable ioeventfd by default, user must disable it
>> >     Drop tags on patch 2/6
>> > 
>> > Jonah Palmer (6):
>> >   virtio/virtio-pci: Handle extra notification data
>> >   virtio: Prevent creation of device using notification-data with ioeventfd
>> >   virtio-mmio: Handle extra notification data
>> >   virtio-ccw: Handle extra notification data
>> >   vhost/vhost-user: Add VIRTIO_F_NOTIFICATION_DATA to vhost feature bits
>> >   virtio: Add VIRTIO_F_NOTIFICATION_DATA property definition
>> 
>> Jonah, do you have kernel patches to add this feature as well?
>> 
>> Thanks!
>
>Hi Jiri! I think there are already kernel patches for
>VIRTIO_F_NOTIFICATION_DATA, unless you're referring to something more
>specific that wasn't included in these patches:
>
>[1]: virtio: add VIRTIO_F_NOTIFICATION_DATA feature support
>https://lore.kernel.org/lkml/20230324195029.2410503-1-viktor@daynix.com/
>
>[2]: virtio-vdpa: add VIRTIO_F_NOTIFICATION_DATA feature support
>https://lore.kernel.org/lkml/20230413081855.36643-3-alvaro.karsz@solid-run.com/

I missed this. Thx!

>
>Jonah