During migration restoring, vfio_enable_vectors() is called to restore
enabling MSI-X interrupts for assigned devices. It sets the range from
0 to nr_vectors to kernel to enable MSI-X and the vectors unmasked in
guest. During the MSI-X enabling, all the vectors within the range are
allocated according to the VFIO_DEVICE_SET_IRQS ioctl.

When dynamic MSI-X allocation is supported, we only want the guest
unmasked vectors being allocated and enabled. Use vector 0 with an
invalid fd to get MSI-X enabled, after that, all the vectors can be
allocated in need.

Signed-off-by: Jing Liu <jing2.liu@intel.com>
---
Changes since v1:
- No change.

Changes since RFC v1:
- Revise the comments. (Alex)
- Call the new helper function in previous patch to enable MSI-X. (Alex)
---
 hw/vfio/pci.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 0117f230e934..f5f891dc0792 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -402,6 +402,23 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix)
     int ret = 0, i, argsz;
     int32_t *fds;
 
+    /*
+     * If dynamic MSI-X allocation is supported, the vectors to be allocated
+     * and enabled can be scattered. Before kernel enabling MSI-X, setting
+     * nr_vectors causes all these vectors to be allocated on host.
+     *
+     * To keep allocation as needed, use vector 0 with an invalid fd to get
+     * MSI-X enabled first, then set vectors with a potentially sparse set of
+     * eventfds to enable interrupts only when enabled in guest.
+     */
+    if (msix && !vdev->msix->noresize) {
+        ret = vfio_enable_msix_no_vec(vdev);
+
+        if (ret) {
+            return ret;
+        }
+    }
+
     argsz = sizeof(*irq_set) + (vdev->nr_vectors * sizeof(*fds));
 
     irq_set = g_malloc0(argsz);
-- 
2.27.0

On 9/18/23 11:45, Jing Liu wrote:
> During migration restoring, vfio_enable_vectors() is called to restore
> enabling MSI-X interrupts for assigned devices. It sets the range from
> 0 to nr_vectors to kernel to enable MSI-X and the vectors unmasked in
> guest. During the MSI-X enabling, all the vectors within the range are
> allocated according to the VFIO_DEVICE_SET_IRQS ioctl.
> 
> When dynamic MSI-X allocation is supported, we only want the guest
> unmasked vectors being allocated and enabled. Use vector 0 with an
> invalid fd to get MSI-X enabled, after that, all the vectors can be
> allocated in need.
> 
> Signed-off-by: Jing Liu <jing2.liu@intel.com>


Reviewed-by: Cédric Le Goater <clg@redhat.com>

Thanks,

C.


> ---
> Changes since v1:
> - No change.
> 
> Changes since RFC v1:
> - Revise the comments. (Alex)
> - Call the new helper function in previous patch to enable MSI-X. (Alex)
> ---
>   hw/vfio/pci.c | 17 +++++++++++++++++
>   1 file changed, 17 insertions(+)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index 0117f230e934..f5f891dc0792 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -402,6 +402,23 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix)
>       int ret = 0, i, argsz;
>       int32_t *fds;
>   
> +    /*
> +     * If dynamic MSI-X allocation is supported, the vectors to be allocated
> +     * and enabled can be scattered. Before kernel enabling MSI-X, setting
> +     * nr_vectors causes all these vectors to be allocated on host.
> +     *
> +     * To keep allocation as needed, use vector 0 with an invalid fd to get
> +     * MSI-X enabled first, then set vectors with a potentially sparse set of
> +     * eventfds to enable interrupts only when enabled in guest.
> +     */
> +    if (msix && !vdev->msix->noresize) {
> +        ret = vfio_enable_msix_no_vec(vdev);
> +
> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
>       argsz = sizeof(*irq_set) + (vdev->nr_vectors * sizeof(*fds));
>   
>       irq_set = g_malloc0(argsz);

Hi Cédric,
> On 9/19/2023 11:21 PM, Cédric Le Goater wrote:
> 
> On 9/18/23 11:45, Jing Liu wrote:
> > During migration restoring, vfio_enable_vectors() is called to restore
> > enabling MSI-X interrupts for assigned devices. It sets the range from
> > 0 to nr_vectors to kernel to enable MSI-X and the vectors unmasked in
> > guest. During the MSI-X enabling, all the vectors within the range are
> > allocated according to the VFIO_DEVICE_SET_IRQS ioctl.
> >
> > When dynamic MSI-X allocation is supported, we only want the guest
> > unmasked vectors being allocated and enabled. Use vector 0 with an
> > invalid fd to get MSI-X enabled, after that, all the vectors can be
> > allocated in need.
> >
> > Signed-off-by: Jing Liu <jing2.liu@intel.com>
> 
> 
> Reviewed-by: Cédric Le Goater <clg@redhat.com>

Thanks very much for your feedback.

Jing

> 
> Thanks,
> 
> C.
> 
> 
> > ---
> > Changes since v1:
> > - No change.
> >
> > Changes since RFC v1:
> > - Revise the comments. (Alex)
> > - Call the new helper function in previous patch to enable MSI-X.
> > (Alex)
> > ---
> >   hw/vfio/pci.c | 17 +++++++++++++++++
> >   1 file changed, 17 insertions(+)
> >
> > diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index
> > 0117f230e934..f5f891dc0792 100644
> > --- a/hw/vfio/pci.c
> > +++ b/hw/vfio/pci.c
> > @@ -402,6 +402,23 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev,
> bool msix)
> >       int ret = 0, i, argsz;
> >       int32_t *fds;
> >
> > +    /*
> > +     * If dynamic MSI-X allocation is supported, the vectors to be allocated
> > +     * and enabled can be scattered. Before kernel enabling MSI-X, setting
> > +     * nr_vectors causes all these vectors to be allocated on host.
> > +     *
> > +     * To keep allocation as needed, use vector 0 with an invalid fd to get
> > +     * MSI-X enabled first, then set vectors with a potentially sparse set of
> > +     * eventfds to enable interrupts only when enabled in guest.
> > +     */
> > +    if (msix && !vdev->msix->noresize) {
> > +        ret = vfio_enable_msix_no_vec(vdev);
> > +
> > +        if (ret) {
> > +            return ret;
> > +        }
> > +    }
> > +
> >       argsz = sizeof(*irq_set) + (vdev->nr_vectors * sizeof(*fds));
> >
> >       irq_set = g_malloc0(argsz);