[PATCH v2 0/4] Support dynamic MSI-X allocation

Jing Liu posted 4 patches 7 months, 2 weeks ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20230918094507.409050-1-jing2.liu@intel.com
Maintainers: Alex Williamson <alex.williamson@redhat.com>, "Cédric Le Goater" <clg@redhat.com>
There is a newer version of this series
hw/vfio/pci.c        | 121 +++++++++++++++++++++++++++++++++----------
hw/vfio/pci.h        |   1 +
hw/vfio/trace-events |   2 +-
3 files changed, 96 insertions(+), 28 deletions(-)
[PATCH v2 0/4] Support dynamic MSI-X allocation
Posted by Jing Liu 7 months, 2 weeks ago
Changes since v1:
- v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html
- Revise Qemu to QEMU. (Cédric)
- Add g_free when failure of getting MSI-X irq info. (Cédric)
- Apply Cédric's Reviewed-by. (Cédric)
- Use g_autofree to automatically release. (Cédric)
- Remove the failure message in vfio_enable_msix_no_vec(). (Cédric)

Changes since RFC v1:
- RFC v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html
- Revise the comments. (Alex)
- Report error of getting irq info and remove the trace of failure
  case. (Alex, Cédric)
- Only store dynamic allocation flag as a bool type and test
  accordingly. (Alex)
- Move dynamic allocation detection to vfio_msix_early_setup(). (Alex)
- Change the condition logic in vfio_msix_vector_do_use() that moving
  the defer_kvm_irq_routing test out and create a common place to update
  nr_vectors. (Alex)
- Consolidate the way of MSI-X enabling during device initialization and
  interrupt restoring that uses fd = -1 trick. Create a function doing
  that. (Alex)

Before kernel v6.5, dynamic allocation of MSI-X interrupts was not
supported. QEMU therefore when allocating a new interrupt, should first
release all previously allocated interrupts (including disable of MSI-X)
and re-allocate all interrupts that includes the new one.

The kernel series [1] adds the support of dynamic MSI-X allocation to
vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide user
space, that when dynamic MSI-X is supported the flag is cleared.

This series makes the behavior for VFIO PCI devices when dynamic MSI-X
allocation is supported. When guest unmasks an interrupt, QEMU can
directly allocate an interrupt on host for this and has nothing to do
with the previously allocated ones. Therefore, host only allocates
interrupts for those unmasked (enabled) interrupts inside guest when
dynamic MSI-X allocation is supported by device.

When guests enable MSI-X with all of the vectors masked, QEMU need match
the state to enable MSI-X with no vector enabled. During migration
restore, QEMU also need enable MSI-X first in dynamic allocation mode,
to avoid the guest unused vectors being allocated on host. To
consolidate them, we use vector 0 with an invalid fd to get MSI-X
enabled and create a common function for this. This is cleaner than
setting userspace triggering and immediately release.

Any feedback is appreciated.

Jing

[1] https://lwn.net/Articles/931679/

Jing Liu (4):
  vfio/pci: detect the support of dynamic MSI-X allocation
  vfio/pci: enable vector on dynamic MSI-X allocation
  vfio/pci: use an invalid fd to enable MSI-X
  vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation

 hw/vfio/pci.c        | 121 +++++++++++++++++++++++++++++++++----------
 hw/vfio/pci.h        |   1 +
 hw/vfio/trace-events |   2 +-
 3 files changed, 96 insertions(+), 28 deletions(-)

-- 
2.27.0


Re: [PATCH v2 0/4] Support dynamic MSI-X allocation
Posted by Alex Williamson 7 months, 1 week ago
On Mon, 18 Sep 2023 05:45:03 -0400
Jing Liu <jing2.liu@intel.com> wrote:

> Changes since v1:
> - v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html
> - Revise Qemu to QEMU. (Cédric)
> - Add g_free when failure of getting MSI-X irq info. (Cédric)
> - Apply Cédric's Reviewed-by. (Cédric)
> - Use g_autofree to automatically release. (Cédric)
> - Remove the failure message in vfio_enable_msix_no_vec(). (Cédric)
> 
> Changes since RFC v1:
> - RFC v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html
> - Revise the comments. (Alex)
> - Report error of getting irq info and remove the trace of failure
>   case. (Alex, Cédric)
> - Only store dynamic allocation flag as a bool type and test
>   accordingly. (Alex)
> - Move dynamic allocation detection to vfio_msix_early_setup(). (Alex)
> - Change the condition logic in vfio_msix_vector_do_use() that moving
>   the defer_kvm_irq_routing test out and create a common place to update
>   nr_vectors. (Alex)
> - Consolidate the way of MSI-X enabling during device initialization and
>   interrupt restoring that uses fd = -1 trick. Create a function doing
>   that. (Alex)
> 
> Before kernel v6.5, dynamic allocation of MSI-X interrupts was not
> supported. QEMU therefore when allocating a new interrupt, should first
> release all previously allocated interrupts (including disable of MSI-X)
> and re-allocate all interrupts that includes the new one.
> 
> The kernel series [1] adds the support of dynamic MSI-X allocation to
> vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide user
> space, that when dynamic MSI-X is supported the flag is cleared.
> 
> This series makes the behavior for VFIO PCI devices when dynamic MSI-X
> allocation is supported. When guest unmasks an interrupt, QEMU can
> directly allocate an interrupt on host for this and has nothing to do
> with the previously allocated ones. Therefore, host only allocates
> interrupts for those unmasked (enabled) interrupts inside guest when
> dynamic MSI-X allocation is supported by device.
> 
> When guests enable MSI-X with all of the vectors masked, QEMU need match
> the state to enable MSI-X with no vector enabled. During migration
> restore, QEMU also need enable MSI-X first in dynamic allocation mode,
> to avoid the guest unused vectors being allocated on host. To
> consolidate them, we use vector 0 with an invalid fd to get MSI-X
> enabled and create a common function for this. This is cleaner than
> setting userspace triggering and immediately release.
> 
> Any feedback is appreciated.
> 
> Jing
> 
> [1] https://lwn.net/Articles/931679/
> 
> Jing Liu (4):
>   vfio/pci: detect the support of dynamic MSI-X allocation
>   vfio/pci: enable vector on dynamic MSI-X allocation
>   vfio/pci: use an invalid fd to enable MSI-X
>   vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation
> 
>  hw/vfio/pci.c        | 121 +++++++++++++++++++++++++++++++++----------
>  hw/vfio/pci.h        |   1 +
>  hw/vfio/trace-events |   2 +-
>  3 files changed, 96 insertions(+), 28 deletions(-)
> 

Some minor comments on 2/ but otherwise:

Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
RE: [PATCH v2 0/4] Support dynamic MSI-X allocation
Posted by Liu, Jing2 7 months, 1 week ago
Hi Alex,

> On Sat, 9/23/2023 4:57AM, Alex Williamson <alex.williamson@redhat.com> wrote:
> 
> On Mon, 18 Sep 2023 05:45:03 -0400
> Jing Liu <jing2.liu@intel.com> wrote:
> 
> > Changes since v1:
> > - v1:
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html
> > - Revise Qemu to QEMU. (Cédric)
> > - Add g_free when failure of getting MSI-X irq info. (Cédric)
> > - Apply Cédric's Reviewed-by. (Cédric)
> > - Use g_autofree to automatically release. (Cédric)
> > - Remove the failure message in vfio_enable_msix_no_vec(). (Cédric)
> >
> > Changes since RFC v1:
> > - RFC v1:
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html
> > - Revise the comments. (Alex)
> > - Report error of getting irq info and remove the trace of failure
> >   case. (Alex, Cédric)
> > - Only store dynamic allocation flag as a bool type and test
> >   accordingly. (Alex)
> > - Move dynamic allocation detection to vfio_msix_early_setup(). (Alex)
> > - Change the condition logic in vfio_msix_vector_do_use() that moving
> >   the defer_kvm_irq_routing test out and create a common place to update
> >   nr_vectors. (Alex)
> > - Consolidate the way of MSI-X enabling during device initialization and
> >   interrupt restoring that uses fd = -1 trick. Create a function doing
> >   that. (Alex)
> >
> > Before kernel v6.5, dynamic allocation of MSI-X interrupts was not
> > supported. QEMU therefore when allocating a new interrupt, should
> > first release all previously allocated interrupts (including disable
> > of MSI-X) and re-allocate all interrupts that includes the new one.
> >
> > The kernel series [1] adds the support of dynamic MSI-X allocation to
> > vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide
> > user space, that when dynamic MSI-X is supported the flag is cleared.
> >
> > This series makes the behavior for VFIO PCI devices when dynamic MSI-X
> > allocation is supported. When guest unmasks an interrupt, QEMU can
> > directly allocate an interrupt on host for this and has nothing to do
> > with the previously allocated ones. Therefore, host only allocates
> > interrupts for those unmasked (enabled) interrupts inside guest when
> > dynamic MSI-X allocation is supported by device.
> >
> > When guests enable MSI-X with all of the vectors masked, QEMU need
> > match the state to enable MSI-X with no vector enabled. During
> > migration restore, QEMU also need enable MSI-X first in dynamic
> > allocation mode, to avoid the guest unused vectors being allocated on
> > host. To consolidate them, we use vector 0 with an invalid fd to get
> > MSI-X enabled and create a common function for this. This is cleaner
> > than setting userspace triggering and immediately release.
> >
> > Any feedback is appreciated.
> >
> > Jing
> >
> > [1] https://lwn.net/Articles/931679/
> >
> > Jing Liu (4):
> >   vfio/pci: detect the support of dynamic MSI-X allocation
> >   vfio/pci: enable vector on dynamic MSI-X allocation
> >   vfio/pci: use an invalid fd to enable MSI-X
> >   vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation
> >
> >  hw/vfio/pci.c        | 121 +++++++++++++++++++++++++++++++++----------
> >  hw/vfio/pci.h        |   1 +
> >  hw/vfio/trace-events |   2 +-
> >  3 files changed, 96 insertions(+), 28 deletions(-)
> >
> 
> Some minor comments on 2/ but otherwise:
> 
> Reviewed-by: Alex Williamson <alex.williamson@redhat.com>

Thank you very much for the feedback. Will apply on v3 with fix for 2/4. 

Jing