Fixes a couple of issues with irqfd use by config interrupt:
- Rearrange initialization so cleanup happens in the reverse order
- Don't use irqfd for config when not in use for data path
I am not sure this is a complete fix though: I think we
are better off limiting the effect to vdpa devices
with config interrupt support. Or even bypass irqfd
for config completely and inject into KVM using ioctl?
The advantage would be less FDs used.
This would mean mostly reverting the patchset though.

Fixes: d5d24d859c ("virtio-pci: add support for configure interrupt")
Cc: "Cindy Lu" <lulu@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 hw/virtio/virtio-pci.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 98fb5493ae..b77cd69f97 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1130,15 +1130,15 @@ static int virtio_pci_set_guest_notifiers(DeviceState *d, int nvqs, bool assign)
             proxy->vector_irqfd =
                 g_malloc0(sizeof(*proxy->vector_irqfd) *
                           msix_nr_vectors_allocated(&proxy->pci_dev));
+            r = kvm_virtio_pci_vector_config_use(proxy);
+            if (r < 0) {
+                goto config_error;
+            }
             r = kvm_virtio_pci_vector_use(proxy, nvqs);
             if (r < 0) {
                 goto config_assign_error;
             }
         }
-        r = kvm_virtio_pci_vector_config_use(proxy);
-        if (r < 0) {
-            goto config_error;
-        }
         r = msix_set_vector_notifiers(&proxy->pci_dev, virtio_pci_vector_unmask,
                                       virtio_pci_vector_mask,
                                       virtio_pci_vector_poll);
@@ -1155,7 +1155,9 @@ notifiers_error:
         kvm_virtio_pci_vector_release(proxy, nvqs);
     }
 config_error:
-    kvm_virtio_pci_vector_config_release(proxy);
+    if (with_irqfd) {
+        kvm_virtio_pci_vector_config_release(proxy);
+    }
 config_assign_error:
     virtio_pci_set_guest_notifier(d, VIRTIO_CONFIG_IRQ_IDX, !assign,
                                   with_irqfd);
-- 
MST

Hi Michael,

On Sun, 09 Jan 2022 17:49:19 +0000,
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> Fixes a couple of issues with irqfd use by config interrupt:
> - Rearrange initialization so cleanup happens in the reverse order
> - Don't use irqfd for config when not in use for data path
> I am not sure this is a complete fix though: I think we
> are better off limiting the effect to vdpa devices
> with config interrupt support. Or even bypass irqfd
> for config completely and inject into KVM using ioctl?
> The advantage would be less FDs used.
> This would mean mostly reverting the patchset though.
> 
> Fixes: d5d24d859c ("virtio-pci: add support for configure interrupt")
> Cc: "Cindy Lu" <lulu@redhat.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

This doesn't seem to fix the problems I'm seeing here with a KVM/arm64
guest (the issue exists with and without this patch)

On initial boot:

<quote>
Loading Linux 5.7.0-1-arm64 ...
Loading initial ramdisk ...
qemu-system-aarch64: virtio-blk failed to set guest notifier (-16), ensure -accel kvm is set.
qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-system-aarch64: virtio-scsi: Failed to set guest notifiers (-16), ensure -accel kvm is set.
qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
</quote>

The guest is functional though. However, on reboot:

<quote>
Loading Linux 5.7.0-1-arm64 ...
Loading initial ramdisk ...
qemu-system-aarch64: ../hw/pci/msix.c:622: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier' failed.
</quote>

Reverting d5d24d859c fixes the issue. For the record, my qemu command
line:

/home/maz/qemu/build/qemu-system-aarch64 -m 1G -smp 8 -cpu host,aarch64=on -machine virt,accel=kvm,gic-version=host -nographic -drive if=pflash,format=raw,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd -drive if=pflash,format=raw,file=bullseye/bBwcgtDY2UwXklV6.fd -netdev user,id=hostnet0 -device virtio-net-pci,netdev=hostnet0 -drive if=none,format=raw,cache=none,aio=native,file=bullseye/bBwcgtDY2UwXklV6.img,id=disk0 -device virtio-blk-pci,drive=disk0 -drive file=debian-testing-arm64-netinst-preseed.iso,id=cdrom,if=none,media=cdrom -device virtio-scsi-pci -device scsi-cd,drive=cdrom

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

On Mon, Jan 10, 2022 at 08:37:43PM +0000, Marc Zyngier wrote:
> Hi Michael,
> 
> On Sun, 09 Jan 2022 17:49:19 +0000,
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > Fixes a couple of issues with irqfd use by config interrupt:
> > - Rearrange initialization so cleanup happens in the reverse order
> > - Don't use irqfd for config when not in use for data path
> > I am not sure this is a complete fix though: I think we
> > are better off limiting the effect to vdpa devices
> > with config interrupt support. Or even bypass irqfd
> > for config completely and inject into KVM using ioctl?
> > The advantage would be less FDs used.
> > This would mean mostly reverting the patchset though.
> > 
> > Fixes: d5d24d859c ("virtio-pci: add support for configure interrupt")
> > Cc: "Cindy Lu" <lulu@redhat.com>
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> This doesn't seem to fix the problems I'm seeing here with a KVM/arm64
> guest (the issue exists with and without this patch)
> 
> On initial boot:
> 
> <quote>
> Loading Linux 5.7.0-1-arm64 ...
> Loading initial ramdisk ...
> qemu-system-aarch64: virtio-blk failed to set guest notifier (-16), ensure -accel kvm is set.
> qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
> qemu-system-aarch64: virtio-scsi: Failed to set guest notifiers (-16), ensure -accel kvm is set.
> qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
> </quote>
> 
> The guest is functional though. However, on reboot:
> 
> <quote>
> Loading Linux 5.7.0-1-arm64 ...
> Loading initial ramdisk ...
> qemu-system-aarch64: ../hw/pci/msix.c:622: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier' failed.
> </quote>
> 
> Reverting d5d24d859c fixes the issue. For the record, my qemu command
> line:
> 
> /home/maz/qemu/build/qemu-system-aarch64 -m 1G -smp 8 -cpu host,aarch64=on -machine virt,accel=kvm,gic-version=host -nographic -drive if=pflash,format=raw,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd -drive if=pflash,format=raw,file=bullseye/bBwcgtDY2UwXklV6.fd -netdev user,id=hostnet0 -device virtio-net-pci,netdev=hostnet0 -drive if=none,format=raw,cache=none,aio=native,file=bullseye/bBwcgtDY2UwXklV6.img,id=disk0 -device virtio-blk-pci,drive=disk0 -drive file=debian-testing-arm64-netinst-preseed.iso,id=cdrom,if=none,media=cdrom -device virtio-scsi-pci -device scsi-cd,drive=cdrom
> 
> Thanks,
> 
> 	M.


Thanks, I'm reverting these changes for now.
Cindy, once you have your patches ready pinging people who encountered
problems with the previous versions is a very good idea.

> -- 
> Without deviation from the norm, progress is not possible.

On 1/9/22 18:49, Michael S. Tsirkin wrote:
> Fixes a couple of issues with irqfd use by config interrupt:
> - Rearrange initialization so cleanup happens in the reverse order
> - Don't use irqfd for config when not in use for data path
> I am not sure this is a complete fix though: I think we
> are better off limiting the effect to vdpa devices
> with config interrupt support. Or even bypass irqfd
> for config completely and inject into KVM using ioctl?
> The advantage would be less FDs used.
> This would mean mostly reverting the patchset though.
> 
> Fixes: d5d24d859c ("virtio-pci: add support for configure interrupt")
> Cc: "Cindy Lu" <lulu@redhat.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

On a KVM guest with vhost, I am still seeing at reboot an issue :/

../hw/pci/msix.c:622: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier'


C.

> ---
>   hw/virtio/virtio-pci.c | 12 +++++++-----
>   1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index 98fb5493ae..b77cd69f97 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -1130,15 +1130,15 @@ static int virtio_pci_set_guest_notifiers(DeviceState *d, int nvqs, bool assign)
>               proxy->vector_irqfd =
>                   g_malloc0(sizeof(*proxy->vector_irqfd) *
>                             msix_nr_vectors_allocated(&proxy->pci_dev));
> +            r = kvm_virtio_pci_vector_config_use(proxy);
> +            if (r < 0) {
> +                goto config_error;
> +            }
>               r = kvm_virtio_pci_vector_use(proxy, nvqs);
>               if (r < 0) {
>                   goto config_assign_error;
>               }
>           }
> -        r = kvm_virtio_pci_vector_config_use(proxy);
> -        if (r < 0) {
> -            goto config_error;
> -        }
>           r = msix_set_vector_notifiers(&proxy->pci_dev, virtio_pci_vector_unmask,
>                                         virtio_pci_vector_mask,
>                                         virtio_pci_vector_poll);
> @@ -1155,7 +1155,9 @@ notifiers_error:
>           kvm_virtio_pci_vector_release(proxy, nvqs);
>       }
>   config_error:
> -    kvm_virtio_pci_vector_config_release(proxy);
> +    if (with_irqfd) {
> +        kvm_virtio_pci_vector_config_release(proxy);
> +    }
>   config_assign_error:
>       virtio_pci_set_guest_notifier(d, VIRTIO_CONFIG_IRQ_IDX, !assign,
>                                     with_irqfd);
>