Hi

This series relax the vIOMMU check and allows live migration with vIOMMU
without VFs using device dirty tracking. It's rewritten based on first 4
patches of [1] from Joao.

Currently what block us is the lack of dirty bitmap query with iommufd
before unmap. By adding that query and handle some corner case we can
relax the check.

Based on vfio-next branch:

patch1-2: add dirty bitmap query with iommufd
patch3:   a ranaming cleanup
patch4-5: unmap_bitmap optimization
patch6-7: fixes to avoid losing dirty pages
patch8:   add a blocker if VM memory is really quite large for unmap_bitmap
patch9:   relax vIOMMU check

Qemu code can be found at [2], it's based on vfio-next + base nesting series.

We tested VM live migration (running QAT workload in VM) with QAT device
passthrough, below matrix configs with guest config 'iommu=pt' and 'iommu=nopt':
1.Scalable mode vIOMMU + IOMMUFD cdev mode
2.Scalable mode vIOMMU + legacy VFIO mode
3.legacy mode vIOMMU + IOMMUFD cdev mode
4.legacy mode vIOMMU + legacy VFIO mode

The QAT workload is a user level app that utilizes VFIO to control QAT device.

Thanks
Zhenzhong

[1] https://github.com/jpemartins/qemu/commits/vfio-migration-viommu/
[2] https://github.com/yiliu1765/qemu/tree/liuyi/zhenzhong/relax_viommu_check

Changelog:
v6:
- check global_dirty_tracking in vfio_listener_region_del() (Liuyi)
- add note and update subject of patch7 (Liuyi)
- check global_dirty_tracking in vtd_address_space_unmap()

v5:
- drop the patch checking iommu_dirty_tracking (Avihai, Joao)
- pass iotlb info to unmap_bitmap when switch out of system AS

v4:
- bypass memory size check for device dirty tracking as it's unrelated (Avihai)
- split vfio_device_dirty_pages_disabled() helper out as a separate patch
- add a patch to fix minor error on checking vbasedev->iommu_dirty_tracking

v3:
- return bitmap query failure to fail migration (Avihai)
- refine patch7, set IOMMUFD backend 'dirty_pgsizes' and 'max_dirty_bitmap_size' (Cédric)
- refine patch7, calculate memory limit instead of hardcode 8TB (Liuyi)
- refine commit log (Cédric, Liuyi)

v2:
- add backend_flag parameter to pass DIRTY_BITMAP_NO_CLEAR (Joao, Cédric)
- add a cleanup patch to rename vfio_dma_unmap_bitmap (Cédric)
- add blocker if unmap_bitmap limit check fail (Liuyi)


Joao Martins (1):
  vfio: Add a backend_flag parameter to
    vfio_contianer_query_dirty_bitmap()

Zhenzhong Duan (8):
  vfio/iommufd: Add framework code to support getting dirty bitmap
    before unmap
  vfio/iommufd: Query dirty bitmap before DMA unmap
  vfio/container-legacy: rename vfio_dma_unmap_bitmap() to
    vfio_legacy_dma_unmap_get_dirty_bitmap()
  vfio/iommufd: Add IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR flag support
  intel_iommu: Fix unmap_bitmap failure with legacy VFIO backend
  vfio/listener: Add missing dirty tracking in region_del
  vfio/migration: Add migration blocker if VM memory is too large to
    cause unmap_bitmap failure
  vfio/migration: Allow live migration with vIOMMU without VFs using
    device dirty tracking

 include/hw/vfio/vfio-container.h |  8 +++--
 include/hw/vfio/vfio-device.h    | 10 ++++++
 include/system/iommufd.h         |  2 +-
 backends/iommufd.c               |  5 +--
 hw/i386/intel_iommu.c            | 42 +++++++++++++++++++++++++
 hw/vfio-user/container.c         |  5 +--
 hw/vfio/container-legacy.c       | 15 +++++----
 hw/vfio/container.c              | 20 ++++++------
 hw/vfio/device.c                 |  6 ++++
 hw/vfio/iommufd.c                | 53 +++++++++++++++++++++++++++++---
 hw/vfio/listener.c               | 28 ++++++++++++++---
 hw/vfio/migration.c              | 40 ++++++++++++++++++++++--
 backends/trace-events            |  2 +-
 hw/vfio/trace-events             |  2 +-
 14 files changed, 201 insertions(+), 37 deletions(-)

-- 
2.47.1

Zhenzhong,

On 12/18/25 07:26, Zhenzhong Duan wrote:
> Hi
> 
> This series relax the vIOMMU check and allows live migration with vIOMMU
> without VFs using device dirty tracking. It's rewritten based on first 4
> patches of [1] from Joao.
> 
> Currently what block us is the lack of dirty bitmap query with iommufd
> before unmap. By adding that query and handle some corner case we can
> relax the check.
> 
> Based on vfio-next branch:
> 
> patch1-2: add dirty bitmap query with iommufd
> patch3:   a ranaming cleanup
> patch4-5: unmap_bitmap optimization
> patch6-7: fixes to avoid losing dirty pages
> patch8:   add a blocker if VM memory is really quite large for unmap_bitmap
> patch9:   relax vIOMMU check
> 
> Qemu code can be found at [2], it's based on vfio-next + base nesting series.
> 
> We tested VM live migration (running QAT workload in VM) with QAT device
> passthrough, below matrix configs with guest config 'iommu=pt' and 'iommu=nopt':
> 1.Scalable mode vIOMMU + IOMMUFD cdev mode
> 2.Scalable mode vIOMMU + legacy VFIO mode
> 3.legacy mode vIOMMU + IOMMUFD cdev mode
> 4.legacy mode vIOMMU + legacy VFIO mode
> 
> The QAT workload is a user level app that utilizes VFIO to control QAT device.


FYI,

There is a conflict with :

   [PATCH v9 00/19] intel_iommu: Enable first stage translation for passthrough device

which one should come first ?


Thanks,

C.

> 
> Thanks
> Zhenzhong
> 
> [1] https://github.com/jpemartins/qemu/commits/vfio-migration-viommu/
> [2] https://github.com/yiliu1765/qemu/tree/liuyi/zhenzhong/relax_viommu_check
> 
> Changelog:
> v6:
> - check global_dirty_tracking in vfio_listener_region_del() (Liuyi)
> - add note and update subject of patch7 (Liuyi)
> - check global_dirty_tracking in vtd_address_space_unmap()
> 
> v5:
> - drop the patch checking iommu_dirty_tracking (Avihai, Joao)
> - pass iotlb info to unmap_bitmap when switch out of system AS
> 
> v4:
> - bypass memory size check for device dirty tracking as it's unrelated (Avihai)
> - split vfio_device_dirty_pages_disabled() helper out as a separate patch
> - add a patch to fix minor error on checking vbasedev->iommu_dirty_tracking
> 
> v3:
> - return bitmap query failure to fail migration (Avihai)
> - refine patch7, set IOMMUFD backend 'dirty_pgsizes' and 'max_dirty_bitmap_size' (Cédric)
> - refine patch7, calculate memory limit instead of hardcode 8TB (Liuyi)
> - refine commit log (Cédric, Liuyi)
> 
> v2:
> - add backend_flag parameter to pass DIRTY_BITMAP_NO_CLEAR (Joao, Cédric)
> - add a cleanup patch to rename vfio_dma_unmap_bitmap (Cédric)
> - add blocker if unmap_bitmap limit check fail (Liuyi)
> 
> 
> Joao Martins (1):
>    vfio: Add a backend_flag parameter to
>      vfio_contianer_query_dirty_bitmap()
> 
> Zhenzhong Duan (8):
>    vfio/iommufd: Add framework code to support getting dirty bitmap
>      before unmap
>    vfio/iommufd: Query dirty bitmap before DMA unmap
>    vfio/container-legacy: rename vfio_dma_unmap_bitmap() to
>      vfio_legacy_dma_unmap_get_dirty_bitmap()
>    vfio/iommufd: Add IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR flag support
>    intel_iommu: Fix unmap_bitmap failure with legacy VFIO backend
>    vfio/listener: Add missing dirty tracking in region_del
>    vfio/migration: Add migration blocker if VM memory is too large to
>      cause unmap_bitmap failure
>    vfio/migration: Allow live migration with vIOMMU without VFs using
>      device dirty tracking
> 
>   include/hw/vfio/vfio-container.h |  8 +++--
>   include/hw/vfio/vfio-device.h    | 10 ++++++
>   include/system/iommufd.h         |  2 +-
>   backends/iommufd.c               |  5 +--
>   hw/i386/intel_iommu.c            | 42 +++++++++++++++++++++++++
>   hw/vfio-user/container.c         |  5 +--
>   hw/vfio/container-legacy.c       | 15 +++++----
>   hw/vfio/container.c              | 20 ++++++------
>   hw/vfio/device.c                 |  6 ++++
>   hw/vfio/iommufd.c                | 53 +++++++++++++++++++++++++++++---
>   hw/vfio/listener.c               | 28 ++++++++++++++---
>   hw/vfio/migration.c              | 40 ++++++++++++++++++++++--
>   backends/trace-events            |  2 +-
>   hw/vfio/trace-events             |  2 +-
>   14 files changed, 201 insertions(+), 37 deletions(-)
> 

Hi Cédric,

>-----Original Message-----
>From: Cédric Le Goater <clg@redhat.com>
>Subject: Re: [PATCH v6 0/9] vfio: relax the vIOMMU check
>
>Zhenzhong,
>
>On 12/18/25 07:26, Zhenzhong Duan wrote:
>> Hi
>>
>> This series relax the vIOMMU check and allows live migration with vIOMMU
>> without VFs using device dirty tracking. It's rewritten based on first 4
>> patches of [1] from Joao.
>>
>> Currently what block us is the lack of dirty bitmap query with iommufd
>> before unmap. By adding that query and handle some corner case we can
>> relax the check.
>>
>> Based on vfio-next branch:
>>
>> patch1-2: add dirty bitmap query with iommufd
>> patch3:   a ranaming cleanup
>> patch4-5: unmap_bitmap optimization
>> patch6-7: fixes to avoid losing dirty pages
>> patch8:   add a blocker if VM memory is really quite large for
>unmap_bitmap
>> patch9:   relax vIOMMU check
>>
>> Qemu code can be found at [2], it's based on vfio-next + base nesting series.
>>
>> We tested VM live migration (running QAT workload in VM) with QAT device
>> passthrough, below matrix configs with guest config 'iommu=pt' and
>'iommu=nopt':
>> 1.Scalable mode vIOMMU + IOMMUFD cdev mode
>> 2.Scalable mode vIOMMU + legacy VFIO mode
>> 3.legacy mode vIOMMU + IOMMUFD cdev mode
>> 4.legacy mode vIOMMU + legacy VFIO mode
>>
>> The QAT workload is a user level app that utilizes VFIO to control QAT
>device.
>
>
>FYI,
>
>There is a conflict with :
>
>   [PATCH v9 00/19] intel_iommu: Enable first stage translation for
>passthrough device
>
>which one should come first ?

The order is as below following your suggestion at [1]:

[PATCH v9 00/19] intel_iommu: Enable first stage translation for passthrough device
[PATCH v6 0/9] vfio: relax the vIOMMU check
[PATCH v9 0/4] Implement ERRATA_772415 quirk for VTD

[1] https://lore.kernel.org/qemu-devel/bbc8412b-25c3-4c95-9fde-a1c9c29b54ce@redhat.com/

Thanks
Zhenzhong

On 12/18/25 07:26, Zhenzhong Duan wrote:
> Hi
> 
> This series relax the vIOMMU check and allows live migration with vIOMMU
> without VFs using device dirty tracking. It's rewritten based on first 4
> patches of [1] from Joao.
> 
> Currently what block us is the lack of dirty bitmap query with iommufd
> before unmap. By adding that query and handle some corner case we can
> relax the check.
> 
> Based on vfio-next branch:
> 
> patch1-2: add dirty bitmap query with iommufd
> patch3:   a ranaming cleanup
> patch4-5: unmap_bitmap optimization
> patch6-7: fixes to avoid losing dirty pages
> patch8:   add a blocker if VM memory is really quite large for unmap_bitmap
> patch9:   relax vIOMMU check
> 
> Qemu code can be found at [2], it's based on vfio-next + base nesting series.
> 
> We tested VM live migration (running QAT workload in VM) with QAT device
> passthrough, below matrix configs with guest config 'iommu=pt' and 'iommu=nopt':
> 1.Scalable mode vIOMMU + IOMMUFD cdev mode
> 2.Scalable mode vIOMMU + legacy VFIO mode
> 3.legacy mode vIOMMU + IOMMUFD cdev mode
> 4.legacy mode vIOMMU + legacy VFIO mode
> 
> The QAT workload is a user level app that utilizes VFIO to control QAT device.
> 
> Thanks
> Zhenzhong
> 
> [1] https://github.com/jpemartins/qemu/commits/vfio-migration-viommu/
> [2] https://github.com/yiliu1765/qemu/tree/liuyi/zhenzhong/relax_viommu_check
> 
> Changelog:
> v6:
> - check global_dirty_tracking in vfio_listener_region_del() (Liuyi)
> - add note and update subject of patch7 (Liuyi)
> - check global_dirty_tracking in vtd_address_space_unmap()
> 
> v5:
> - drop the patch checking iommu_dirty_tracking (Avihai, Joao)
> - pass iotlb info to unmap_bitmap when switch out of system AS
> 
> v4:
> - bypass memory size check for device dirty tracking as it's unrelated (Avihai)
> - split vfio_device_dirty_pages_disabled() helper out as a separate patch
> - add a patch to fix minor error on checking vbasedev->iommu_dirty_tracking
> 
> v3:
> - return bitmap query failure to fail migration (Avihai)
> - refine patch7, set IOMMUFD backend 'dirty_pgsizes' and 'max_dirty_bitmap_size' (Cédric)
> - refine patch7, calculate memory limit instead of hardcode 8TB (Liuyi)
> - refine commit log (Cédric, Liuyi)
> 
> v2:
> - add backend_flag parameter to pass DIRTY_BITMAP_NO_CLEAR (Joao, Cédric)
> - add a cleanup patch to rename vfio_dma_unmap_bitmap (Cédric)
> - add blocker if unmap_bitmap limit check fail (Liuyi)
> 
> 
> Joao Martins (1):
>    vfio: Add a backend_flag parameter to
>      vfio_contianer_query_dirty_bitmap()
> 
> Zhenzhong Duan (8):
>    vfio/iommufd: Add framework code to support getting dirty bitmap
>      before unmap
>    vfio/iommufd: Query dirty bitmap before DMA unmap
>    vfio/container-legacy: rename vfio_dma_unmap_bitmap() to
>      vfio_legacy_dma_unmap_get_dirty_bitmap()
>    vfio/iommufd: Add IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR flag support
>    intel_iommu: Fix unmap_bitmap failure with legacy VFIO backend
>    vfio/listener: Add missing dirty tracking in region_del
>    vfio/migration: Add migration blocker if VM memory is too large to
>      cause unmap_bitmap failure
>    vfio/migration: Allow live migration with vIOMMU without VFs using
>      device dirty tracking
> 
>   include/hw/vfio/vfio-container.h |  8 +++--
>   include/hw/vfio/vfio-device.h    | 10 ++++++
>   include/system/iommufd.h         |  2 +-
>   backends/iommufd.c               |  5 +--
>   hw/i386/intel_iommu.c            | 42 +++++++++++++++++++++++++
>   hw/vfio-user/container.c         |  5 +--
>   hw/vfio/container-legacy.c       | 15 +++++----
>   hw/vfio/container.c              | 20 ++++++------
>   hw/vfio/device.c                 |  6 ++++
>   hw/vfio/iommufd.c                | 53 +++++++++++++++++++++++++++++---
>   hw/vfio/listener.c               | 28 ++++++++++++++---
>   hw/vfio/migration.c              | 40 ++++++++++++++++++++++--
>   backends/trace-events            |  2 +-
>   hw/vfio/trace-events             |  2 +-
>   14 files changed, 201 insertions(+), 37 deletions(-)
> 


Applied to vfio-next.

Thanks,

C.