Changes since v2:
- v2: https://www.mail-archive.com/qemu-devel@nongnu.org/msg989852.html
- Use a bool type to test (vdev->nr_vectors < nr + 1). (Alex)
- Revise comments. (Alex)
- Apply Cédric's and Alex's Reviewed-by.

Changes since v1:
- v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html
- Revise Qemu to QEMU. (Cédric)
- Add g_free when failure of getting MSI-X irq info. (Cédric)
- Apply Cédric's Reviewed-by. (Cédric)
- Use g_autofree to automatically release. (Cédric)
- Remove the failure message in vfio_enable_msix_no_vec(). (Cédric)

Changes since RFC v1:
- RFC v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html
- Revise the comments. (Alex)
- Report error of getting irq info and remove the trace of failure
  case. (Alex, Cédric)
- Only store dynamic allocation flag as a bool type and test
  accordingly. (Alex)
- Move dynamic allocation detection to vfio_msix_early_setup(). (Alex)
- Change the condition logic in vfio_msix_vector_do_use() that moving
  the defer_kvm_irq_routing test out and create a common place to update
  nr_vectors. (Alex)
- Consolidate the way of MSI-X enabling during device initialization and
  interrupt restoring that uses fd = -1 trick. Create a function doing
  that. (Alex)

Before kernel v6.5, dynamic allocation of MSI-X interrupts was not
supported. QEMU therefore when allocating a new interrupt, should first
release all previously allocated interrupts (including disable of MSI-X)
and re-allocate all interrupts that includes the new one.

The kernel series [1] adds the support of dynamic MSI-X allocation to
vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide user
space, that when dynamic MSI-X is supported the flag is cleared.

This series makes the behavior for VFIO PCI devices when dynamic MSI-X
allocation is supported. When guest unmasks an interrupt, QEMU can
directly allocate an interrupt on host for this and has nothing to do
with the previously allocated ones. Therefore, host only allocates
interrupts for those unmasked (enabled) interrupts inside guest when
dynamic MSI-X allocation is supported by device.

When guests enable MSI-X with all of the vectors masked, QEMU need match
the state to enable MSI-X with no vector enabled. During migration
restore, QEMU also need enable MSI-X first in dynamic allocation mode,
to avoid the guest unused vectors being allocated on host. To
consolidate them, we use vector 0 with an invalid fd to get MSI-X
enabled and create a common function for this. This is cleaner than
setting userspace triggering and immediately release.

Any feedback is appreciated.

Jing

[1] https://lwn.net/Articles/931679/

Jing Liu (4):
  vfio/pci: detect the support of dynamic MSI-X allocation
  vfio/pci: enable vector on dynamic MSI-X allocation
  vfio/pci: use an invalid fd to enable MSI-X
  vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation

 hw/vfio/pci.c        | 123 +++++++++++++++++++++++++++++++++----------
 hw/vfio/pci.h        |   1 +
 hw/vfio/trace-events |   2 +-
 3 files changed, 97 insertions(+), 29 deletions(-)

-- 
2.27.0

On 9/26/23 04:14, Jing Liu wrote:
> Changes since v2:
> - v2: https://www.mail-archive.com/qemu-devel@nongnu.org/msg989852.html
> - Use a bool type to test (vdev->nr_vectors < nr + 1). (Alex)
> - Revise comments. (Alex)
> - Apply Cédric's and Alex's Reviewed-by.
> 
> Changes since v1:
> - v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html
> - Revise Qemu to QEMU. (Cédric)
> - Add g_free when failure of getting MSI-X irq info. (Cédric)
> - Apply Cédric's Reviewed-by. (Cédric)
> - Use g_autofree to automatically release. (Cédric)
> - Remove the failure message in vfio_enable_msix_no_vec(). (Cédric)
> 
> Changes since RFC v1:
> - RFC v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html
> - Revise the comments. (Alex)
> - Report error of getting irq info and remove the trace of failure
>    case. (Alex, Cédric)
> - Only store dynamic allocation flag as a bool type and test
>    accordingly. (Alex)
> - Move dynamic allocation detection to vfio_msix_early_setup(). (Alex)
> - Change the condition logic in vfio_msix_vector_do_use() that moving
>    the defer_kvm_irq_routing test out and create a common place to update
>    nr_vectors. (Alex)
> - Consolidate the way of MSI-X enabling during device initialization and
>    interrupt restoring that uses fd = -1 trick. Create a function doing
>    that. (Alex)
> 
> Before kernel v6.5, dynamic allocation of MSI-X interrupts was not
> supported. QEMU therefore when allocating a new interrupt, should first
> release all previously allocated interrupts (including disable of MSI-X)
> and re-allocate all interrupts that includes the new one.
> 
> The kernel series [1] adds the support of dynamic MSI-X allocation to
> vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide user
> space, that when dynamic MSI-X is supported the flag is cleared.
> 
> This series makes the behavior for VFIO PCI devices when dynamic MSI-X
> allocation is supported. When guest unmasks an interrupt, QEMU can
> directly allocate an interrupt on host for this and has nothing to do
> with the previously allocated ones. Therefore, host only allocates
> interrupts for those unmasked (enabled) interrupts inside guest when
> dynamic MSI-X allocation is supported by device.
> 
> When guests enable MSI-X with all of the vectors masked, QEMU need match
> the state to enable MSI-X with no vector enabled. During migration
> restore, QEMU also need enable MSI-X first in dynamic allocation mode,
> to avoid the guest unused vectors being allocated on host. To
> consolidate them, we use vector 0 with an invalid fd to get MSI-X
> enabled and create a common function for this. This is cleaner than
> setting userspace triggering and immediately release.
> 
> Any feedback is appreciated.
> 
> Jing
> 
> [1] https://lwn.net/Articles/931679/
> 
> Jing Liu (4):
>    vfio/pci: detect the support of dynamic MSI-X allocation
>    vfio/pci: enable vector on dynamic MSI-X allocation
>    vfio/pci: use an invalid fd to enable MSI-X
>    vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation
> 
>   hw/vfio/pci.c        | 123 +++++++++++++++++++++++++++++++++----------
>   hw/vfio/pci.h        |   1 +
>   hw/vfio/trace-events |   2 +-
>   3 files changed, 97 insertions(+), 29 deletions(-)
> 

Applied to vfio-next.

Thanks,

C.