Hi Alex,

> On Sat, 9/23/2023 4:57AM, Alex Williamson <alex.williamson@redhat.com> wrote:
> 
> On Mon, 18 Sep 2023 05:45:03 -0400
> Jing Liu <jing2.liu@intel.com> wrote:
> 
> > Changes since v1:
> > - v1:
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html
> > - Revise Qemu to QEMU. (Cédric)
> > - Add g_free when failure of getting MSI-X irq info. (Cédric)
> > - Apply Cédric's Reviewed-by. (Cédric)
> > - Use g_autofree to automatically release. (Cédric)
> > - Remove the failure message in vfio_enable_msix_no_vec(). (Cédric)
> >
> > Changes since RFC v1:
> > - RFC v1:
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html
> > - Revise the comments. (Alex)
> > - Report error of getting irq info and remove the trace of failure
> >   case. (Alex, Cédric)
> > - Only store dynamic allocation flag as a bool type and test
> >   accordingly. (Alex)
> > - Move dynamic allocation detection to vfio_msix_early_setup(). (Alex)
> > - Change the condition logic in vfio_msix_vector_do_use() that moving
> >   the defer_kvm_irq_routing test out and create a common place to update
> >   nr_vectors. (Alex)
> > - Consolidate the way of MSI-X enabling during device initialization and
> >   interrupt restoring that uses fd = -1 trick. Create a function doing
> >   that. (Alex)
> >
> > Before kernel v6.5, dynamic allocation of MSI-X interrupts was not
> > supported. QEMU therefore when allocating a new interrupt, should
> > first release all previously allocated interrupts (including disable
> > of MSI-X) and re-allocate all interrupts that includes the new one.
> >
> > The kernel series [1] adds the support of dynamic MSI-X allocation to
> > vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide
> > user space, that when dynamic MSI-X is supported the flag is cleared.
> >
> > This series makes the behavior for VFIO PCI devices when dynamic MSI-X
> > allocation is supported. When guest unmasks an interrupt, QEMU can
> > directly allocate an interrupt on host for this and has nothing to do
> > with the previously allocated ones. Therefore, host only allocates
> > interrupts for those unmasked (enabled) interrupts inside guest when
> > dynamic MSI-X allocation is supported by device.
> >
> > When guests enable MSI-X with all of the vectors masked, QEMU need
> > match the state to enable MSI-X with no vector enabled. During
> > migration restore, QEMU also need enable MSI-X first in dynamic
> > allocation mode, to avoid the guest unused vectors being allocated on
> > host. To consolidate them, we use vector 0 with an invalid fd to get
> > MSI-X enabled and create a common function for this. This is cleaner
> > than setting userspace triggering and immediately release.
> >
> > Any feedback is appreciated.
> >
> > Jing
> >
> > [1] https://lwn.net/Articles/931679/
> >
> > Jing Liu (4):
> >   vfio/pci: detect the support of dynamic MSI-X allocation
> >   vfio/pci: enable vector on dynamic MSI-X allocation
> >   vfio/pci: use an invalid fd to enable MSI-X
> >   vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation
> >
> >  hw/vfio/pci.c        | 121 +++++++++++++++++++++++++++++++++----------
> >  hw/vfio/pci.h        |   1 +
> >  hw/vfio/trace-events |   2 +-
> >  3 files changed, 96 insertions(+), 28 deletions(-)
> >
> 
> Some minor comments on 2/ but otherwise:
> 
> Reviewed-by: Alex Williamson <alex.williamson@redhat.com>

Thank you very much for the feedback. Will apply on v3 with fix for 2/4. 

Jing