Hello,

Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
growing requirement, but real hardware with migration support is
scarce and hard to debug. An emulated device provides a fully
controlled testbed for developing and validating the entire software
stack - vfio-pci variant drivers, VFIO core migration v2 framework,
QEMU, libvirt - and for tuning complex migration policies such as
downtime convergence. It also serves as an educational reference for
understanding VFIO migration end-to-end, from device state
serialization to dirty page tracking.

This series adds an experimental VF live migration interface to the
emulated igb (82576) device. It enables a vfio-pci variant driver
(igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
protocol with stop-copy and pre-copy support.

The target scenario is nested virtualization:

  L0 QEMU (these patches)
    igb PF with x-vf-migration=on
    └── VFs with migration BAR + vendor cap

  L1 kernel
    igb-vfio-pci variant driver [1]
    translates VFIO migration v2 ioctls → BAR2 MMIO

  L1 QEMU (stock, unmodified)
    vfio-pci device model, standard migration fd

  L2 guest
    standard igbvf driver, unaware of migration

The L1 QEMU is completely unmodified -- it sees a standard VFIO
migratable device and uses the normal migration fd path.

* Design

The migration interface is exposed through a hidden 64KB PCI BAR
(BAR2) on each VF, discovered via a vendor-specific PCI capability
("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
RESUMING, PRE_COPY).

Device state is serialized as a versioned blob of per-VF register
(offset, value) pairs covering control, interrupt, RX/TX queue,
receive address (RA/RA2), etc. plus TX context descriptors and
VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
device, since VFIO owns the VF's IOMMU domain and the variant driver
maps its DMA buffers through the PF.

Dirty page tracking is implemented with per-range bitmaps maintained
in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
descriptor writeback) are instrumented to record touched pages. The
variant driver registers tracked IOVA ranges and queries dirty bitmaps
through a DMA shared buffer, using a single MMIO doorbell
(DIRTY_CTRL=QUERY) per iteration.

* Caveats

Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
BAR flag that keeps the migration BAR mapped even after VFIO's
Function Level Reset clears PCI_COMMAND_MEMORY.

The x-vf-migration property is experimental (x- prefix, default off)
and the migration BAR register interface may change.

The dirty bitmaps are maintained inside the device, which is not
realistic for discrete NICs without on-chip DRAM.

* Testing

The target scenario is nested virtualization: L0 runs QEMU with an
igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
and an unmodified QEMU, and L2 runs a standard igbvf driver.

Migration under iperf3 load works correctly: dirty page tracking
converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
STOP_COPY), and STOP_COPY stays under 250ms.

* Todo/Ideas

  1. Add migration blocker when x-vf-migration=on (no VMState yet) or
     add VMState support for L0 migration (dirty bitmaps, tracking
     engines, migration BAR registers, stats)
     
  2. Add PRE_COPY match data validation (magic, version, caps)
  
  3. Support driver-provided DMA bitmaps per dirty range
     (DIRTY_RANGE_ADDR_LO/HI)
     
  4. Add QMP/HMP knobs for error injection and capability tuning:
  
    . Dirty rate throttling: artificial delay or rate limit on DMA
      dirty tracking to simulate different convergence scenarios
    . State blob size inflation: pad the migration blob to stress
      large-state transfers and test DATA_SIZE limits
    . Migration phase timing: expose per-VF counters for time spent
      in each state (PRE_COPY duration, STOP_COPY latency)
    . Hot page simulation: mark specific page ranges as always-dirty
      to test worst-case convergence
    . Error injection: force STATUS error codes, fail DMA transfers,
      corrupt state blobs, mask CAPS bits, etc.
      
  5. Add qtests for migration state machine transitions, dirty page
     tracking ?

* Credits

Alex Williamson suggested the overall approach: a hidden migration BAR
discovered via a vendor-specific PCI capability, the "vf-migration"
device property to gate the feature. Thanks for the ever ongoing
support and valuable discussions throughout these years.

* AI disclaimer

The lack of a migration-capable device has been a recurring pain point
for VFIO development over the years, and we hope this proposal
demonstrates the value of having one.

Claude was used to analyze the IGB PF and VF internal state and
identify the pain points of a working live migration of such devices.
The generated code served as a starting point but *significant* time
was then spent cleaning up, reworking, and shaping it into a clear,
reviewable proposal. As QEMU does not yet accept AI-assisted
contributions, this series is submitted as an RFC.

Thanks,

C.

[1] https://github.com/legoater/vfio-pci-extras

Cédric Le Goater (11):
  pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
  igb: Add x-vf-migration property and vendor-specific capability for
    IGBVF
  igb: Add migration BAR with state machine
  igb: Add VF state serialization for live migration
  igb: Add VF post-load fixups for live migration
  igb: Add dirty page tracking for IGBVF migration
  igb: Quiesce VFs on STOP and include PF enable state in migration blob
  igb: Fix post-migration RX ring deadlock
  igb: Send RARP after VF migration to update bridge FDB
  docs: Add igb VF migration testing setup guide
  igb: Add migration statistics registers to VF migration BAR

 MAINTAINERS                           |    6 +
 docs/system/device-emulation.rst      |    1 +
 docs/system/devices/igb-migration.rst |  313 +++++++
 docs/system/devices/igb.rst           |    6 +
 hw/net/igb_common.h                   |   11 +
 hw/net/igb_core.h                     |    8 +
 hw/net/igb_migration.h                |  201 +++++
 include/hw/pci/pci.h                  |    6 +
 hw/net/igb.c                          |   18 +
 hw/net/igb_core.c                     |  144 ++-
 hw/net/igb_migration.c                | 1194 +++++++++++++++++++++++++
 hw/net/igbvf.c                        |   29 +-
 hw/pci/pci.c                          |    6 +-
 hw/net/meson.build                    |    2 +-
 hw/net/trace-events                   |   20 +
 15 files changed, 1939 insertions(+), 26 deletions(-)
 create mode 100644 docs/system/devices/igb-migration.rst
 create mode 100644 hw/net/igb_migration.h
 create mode 100644 hw/net/igb_migration.c

-- 
2.55.0

Akihiko,

On 7/27/26 07:39, Cédric Le Goater wrote:
> Hello,
> 
> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
> growing requirement, but real hardware with migration support is
> scarce and hard to debug. An emulated device provides a fully
> controlled testbed for developing and validating the entire software
> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
> QEMU, libvirt - and for tuning complex migration policies such as
> downtime convergence. It also serves as an educational reference for
> understanding VFIO migration end-to-end, from device state
> serialization to dirty page tracking.
> 
> This series adds an experimental VF live migration interface to the
> emulated igb (82576) device. It enables a vfio-pci variant driver
> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
> protocol with stop-copy and pre-copy support.
> 
> The target scenario is nested virtualization:
> 
>    L0 QEMU (these patches)
>      igb PF with x-vf-migration=on
>      └── VFs with migration BAR + vendor cap
> 
>    L1 kernel
>      igb-vfio-pci variant driver [1]
>      translates VFIO migration v2 ioctls → BAR2 MMIO
> 
>    L1 QEMU (stock, unmodified)
>      vfio-pci device model, standard migration fd
> 
>    L2 guest
>      standard igbvf driver, unaware of migration
> 
> The L1 QEMU is completely unmodified -- it sees a standard VFIO
> migratable device and uses the normal migration fd path.
> 
> * Design
> 
> The migration interface is exposed through a hidden 64KB PCI BAR
> (BAR2) on each VF, discovered via a vendor-specific PCI capability
> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
> RESUMING, PRE_COPY).
> 
> Device state is serialized as a versioned blob of per-VF register
> (offset, value) pairs covering control, interrupt, RX/TX queue,
> receive address (RA/RA2), etc. plus TX context descriptors and
> VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
> device, since VFIO owns the VF's IOMMU domain and the variant driver
> maps its DMA buffers through the PF.
> 
> Dirty page tracking is implemented with per-range bitmaps maintained
> in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
> descriptor writeback) are instrumented to record touched pages. The
> variant driver registers tracked IOVA ranges and queries dirty bitmaps
> through a DMA shared buffer, using a single MMIO doorbell
> (DIRTY_CTRL=QUERY) per iteration.
> 
> * Caveats
> 
> Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
> BAR flag that keeps the migration BAR mapped even after VFIO's
> Function Level Reset clears PCI_COMMAND_MEMORY.
> 
> The x-vf-migration property is experimental (x- prefix, default off)
> and the migration BAR register interface may change.
> 
> The dirty bitmaps are maintained inside the device, which is not
> realistic for discrete NICs without on-chip DRAM.
> 
> * Testing
> 
> The target scenario is nested virtualization: L0 runs QEMU with an
> igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
> and an unmodified QEMU, and L2 runs a standard igbvf driver.
> 
> Migration under iperf3 load works correctly: dirty page tracking
> converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
> STOP_COPY), and STOP_COPY stays under 250ms.
> 
> * Todo/Ideas
> 
>    1. Add migration blocker when x-vf-migration=on (no VMState yet) or
>       add VMState support for L0 migration (dirty bitmaps, tracking
>       engines, migration BAR registers, stats)
>       
>    2. Add PRE_COPY match data validation (magic, version, caps)
>    
>    3. Support driver-provided DMA bitmaps per dirty range
>       (DIRTY_RANGE_ADDR_LO/HI)
>       
>    4. Add QMP/HMP knobs for error injection and capability tuning:
>    
>      . Dirty rate throttling: artificial delay or rate limit on DMA
>        dirty tracking to simulate different convergence scenarios
>      . State blob size inflation: pad the migration blob to stress
>        large-state transfers and test DATA_SIZE limits
>      . Migration phase timing: expose per-VF counters for time spent
>        in each state (PRE_COPY duration, STOP_COPY latency)
>      . Hot page simulation: mark specific page ranges as always-dirty
>        to test worst-case convergence
>      . Error injection: force STATUS error codes, fail DMA transfers,
>        corrupt state blobs, mask CAPS bits, etc.
>        
>    5. Add qtests for migration state machine transitions, dirty page
>       tracking ?
> 
> * Credits
> 
> Alex Williamson suggested the overall approach: a hidden migration BAR
> discovered via a vendor-specific PCI capability, the "vf-migration"
> device property to gate the feature. Thanks for the ever ongoing
> support and valuable discussions throughout these years.
> 
> * AI disclaimer
> 
> The lack of a migration-capable device has been a recurring pain point
> for VFIO development over the years, and we hope this proposal
> demonstrates the value of having one.
> 
> Claude was used to analyze the IGB PF and VF internal state and
> identify the pain points of a working live migration of such devices.
> The generated code served as a starting point but *significant* time
> was then spent cleaning up, reworking, and shaping it into a clear,
> reviewable proposal. As QEMU does not yet accept AI-assisted
> contributions, this series is submitted as an RFC.
> 
> Thanks,
> 
> C.
> 
> [1] https://github.com/legoater/vfio-pci-extras
> 
> Cédric Le Goater (11):
>    pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
>    igb: Add x-vf-migration property and vendor-specific capability for
>      IGBVF
>    igb: Add migration BAR with state machine
>    igb: Add VF state serialization for live migration
>    igb: Add VF post-load fixups for live migration
>    igb: Add dirty page tracking for IGBVF migration
>    igb: Quiesce VFs on STOP and include PF enable state in migration blob
>    igb: Fix post-migration RX ring deadlock
>    igb: Send RARP after VF migration to update bridge FDB
>    docs: Add igb VF migration testing setup guide
>    igb: Add migration statistics registers to VF migration BAR
> 
>   MAINTAINERS                           |    6 +
>   docs/system/device-emulation.rst      |    1 +
>   docs/system/devices/igb-migration.rst |  313 +++++++
>   docs/system/devices/igb.rst           |    6 +
>   hw/net/igb_common.h                   |   11 +
>   hw/net/igb_core.h                     |    8 +
>   hw/net/igb_migration.h                |  201 +++++
>   include/hw/pci/pci.h                  |    6 +
>   hw/net/igb.c                          |   18 +
>   hw/net/igb_core.c                     |  144 ++-
>   hw/net/igb_migration.c                | 1194 +++++++++++++++++++++++++
>   hw/net/igbvf.c                        |   29 +-
>   hw/pci/pci.c                          |    6 +-
>   hw/net/meson.build                    |    2 +-
>   hw/net/trace-events                   |   20 +
>   15 files changed, 1939 insertions(+), 26 deletions(-)
>   create mode 100644 docs/system/devices/igb-migration.rst
>   create mode 100644 hw/net/igb_migration.h
>   create mode 100644 hw/net/igb_migration.c
> 

First thanks for the quick review ! I will recap here :

* bisectability, will improve the first patches.
* state serialization, the blob management is awful. I was expecting
   flames. Yes. it needs a rework. I lack a qbuf-style put/get API. The
   contents of the blob need a review. Seems feasible.
* relocation between VFs, on my TODO. The code already checks the vfn but
   needs tightening and better support.
* interrupts bits: will look into it. hopefully, the model "only" needs
   to be more precise. This was a difficult aspect of the igb to understand
   and I am still learning.
* dirty tracking, I worked on 2 different interfaces and some of the code
   fell through the cracks. DMA failures, bitmap clearing, size validation,
   all need improvements. Mostly addressed already, remaining items are code
   reorg. should be fine.
* quiesce: looks like a bug in the model. Will check.
* RARP : I got inspiration from the tests. you are right saying it belongs
   to the management layer. Keeping it for now as it simplifies testing, but
   will move it out.
* Stats: the addition is from yesterday. Needs fixes indeed.

Thanks,

C.

On 2026/07/28 2:35, Cédric Le Goater wrote:
> Akihiko,
> 
> On 7/27/26 07:39, Cédric Le Goater wrote:
>> Hello,
>>
>> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
>> growing requirement, but real hardware with migration support is
>> scarce and hard to debug. An emulated device provides a fully
>> controlled testbed for developing and validating the entire software
>> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
>> QEMU, libvirt - and for tuning complex migration policies such as
>> downtime convergence. It also serves as an educational reference for
>> understanding VFIO migration end-to-end, from device state
>> serialization to dirty page tracking.

This is another good example of using QEMU as a SR-IOV testbed.

>>
>> This series adds an experimental VF live migration interface to the
>> emulated igb (82576) device. It enables a vfio-pci variant driver
>> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
>> protocol with stop-copy and pre-copy support.
I suggest looking into virtio-net as an alternative to igb. It would 
allow avoiding lots of complexities of igb.

>>
>> The target scenario is nested virtualization:
>>
>>    L0 QEMU (these patches)
>>      igb PF with x-vf-migration=on
>>      └── VFs with migration BAR + vendor cap
>>
>>    L1 kernel
>>      igb-vfio-pci variant driver [1]
>>      translates VFIO migration v2 ioctls → BAR2 MMIO
>>
>>    L1 QEMU (stock, unmodified)
>>      vfio-pci device model, standard migration fd
>>
>>    L2 guest
>>      standard igbvf driver, unaware of migration
>>
>> The L1 QEMU is completely unmodified -- it sees a standard VFIO
>> migratable device and uses the normal migration fd path.
>>
>> * Design
>>
>> The migration interface is exposed through a hidden 64KB PCI BAR
>> (BAR2) on each VF, discovered via a vendor-specific PCI capability
>> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
>> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
>> RESUMING, PRE_COPY).
>>
>> Device state is serialized as a versioned blob of per-VF register
>> (offset, value) pairs covering control, interrupt, RX/TX queue,
>> receive address (RA/RA2), etc. plus TX context descriptors and
>> VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
>> device, since VFIO owns the VF's IOMMU domain and the variant driver
>> maps its DMA buffers through the PF.
>>
>> Dirty page tracking is implemented with per-range bitmaps maintained
>> in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
>> descriptor writeback) are instrumented to record touched pages. The
>> variant driver registers tracked IOVA ranges and queries dirty bitmaps
>> through a DMA shared buffer, using a single MMIO doorbell
>> (DIRTY_CTRL=QUERY) per iteration.
>>
>> * Caveats
>>
>> Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
>> BAR flag that keeps the migration BAR mapped even after VFIO's
>> Function Level Reset clears PCI_COMMAND_MEMORY.
>>
>> The x-vf-migration property is experimental (x- prefix, default off)
>> and the migration BAR register interface may change.
>>
>> The dirty bitmaps are maintained inside the device, which is not
>> realistic for discrete NICs without on-chip DRAM.
>>
>> * Testing
>>
>> The target scenario is nested virtualization: L0 runs QEMU with an
>> igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
>> and an unmodified QEMU, and L2 runs a standard igbvf driver.
>>
>> Migration under iperf3 load works correctly: dirty page tracking
>> converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
>> STOP_COPY), and STOP_COPY stays under 250ms.
>>
>> * Todo/Ideas
>>
>>    1. Add migration blocker when x-vf-migration=on (no VMState yet) or
>>       add VMState support for L0 migration (dirty bitmaps, tracking
>>       engines, migration BAR registers, stats)

If you have AI assistance, why don't you just add migration blocker? It 
should be just one prompt away and not more troublesome than leaving 
this "Todo". ;)

>>    2. Add PRE_COPY match data validation (magic, version, caps)
>>    3. Support driver-provided DMA bitmaps per dirty range
>>       (DIRTY_RANGE_ADDR_LO/HI)
>>    4. Add QMP/HMP knobs for error injection and capability tuning:
>>      . Dirty rate throttling: artificial delay or rate limit on DMA
>>        dirty tracking to simulate different convergence scenarios
>>      . State blob size inflation: pad the migration blob to stress
>>        large-state transfers and test DATA_SIZE limits
>>      . Migration phase timing: expose per-VF counters for time spent
>>        in each state (PRE_COPY duration, STOP_COPY latency)
>>      . Hot page simulation: mark specific page ranges as always-dirty
>>        to test worst-case convergence
>>      . Error injection: force STATUS error codes, fail DMA transfers,
>>        corrupt state blobs, mask CAPS bits, etc.
>>    5. Add qtests for migration state machine transitions, dirty page
>>       tracking ?
>>
>> * Credits
>>
>> Alex Williamson suggested the overall approach: a hidden migration BAR
>> discovered via a vendor-specific PCI capability, the "vf-migration"
>> device property to gate the feature. Thanks for the ever ongoing
>> support and valuable discussions throughout these years.
>>
>> * AI disclaimer
>>
>> The lack of a migration-capable device has been a recurring pain point
>> for VFIO development over the years, and we hope this proposal
>> demonstrates the value of having one.
>>
>> Claude was used to analyze the IGB PF and VF internal state and
>> identify the pain points of a working live migration of such devices.
>> The generated code served as a starting point but *significant* time
>> was then spent cleaning up, reworking, and shaping it into a clear,
>> reviewable proposal. As QEMU does not yet accept AI-assisted
>> contributions, this series is submitted as an RFC.
>>
>> Thanks,
>>
>> C.
>>
>> [1] https://github.com/legoater/vfio-pci-extras
>>
>> Cédric Le Goater (11):
>>    pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
>>    igb: Add x-vf-migration property and vendor-specific capability for
>>      IGBVF
>>    igb: Add migration BAR with state machine
>>    igb: Add VF state serialization for live migration
>>    igb: Add VF post-load fixups for live migration
>>    igb: Add dirty page tracking for IGBVF migration
>>    igb: Quiesce VFs on STOP and include PF enable state in migration blob
>>    igb: Fix post-migration RX ring deadlock
>>    igb: Send RARP after VF migration to update bridge FDB
>>    docs: Add igb VF migration testing setup guide
>>    igb: Add migration statistics registers to VF migration BAR
>>
>>   MAINTAINERS                           |    6 +
>>   docs/system/device-emulation.rst      |    1 +
>>   docs/system/devices/igb-migration.rst |  313 +++++++
>>   docs/system/devices/igb.rst           |    6 +
>>   hw/net/igb_common.h                   |   11 +
>>   hw/net/igb_core.h                     |    8 +
>>   hw/net/igb_migration.h                |  201 +++++
>>   include/hw/pci/pci.h                  |    6 +
>>   hw/net/igb.c                          |   18 +
>>   hw/net/igb_core.c                     |  144 ++-
>>   hw/net/igb_migration.c                | 1194 +++++++++++++++++++++++++
>>   hw/net/igbvf.c                        |   29 +-
>>   hw/pci/pci.c                          |    6 +-
>>   hw/net/meson.build                    |    2 +-
>>   hw/net/trace-events                   |   20 +
>>   15 files changed, 1939 insertions(+), 26 deletions(-)
>>   create mode 100644 docs/system/devices/igb-migration.rst
>>   create mode 100644 hw/net/igb_migration.h
>>   create mode 100644 hw/net/igb_migration.c
>>
> 
> First thanks for the quick review ! I will recap here :
> 
> * bisectability, will improve the first patches.
> * state serialization, the blob management is awful. I was expecting
>    flames. Yes. it needs a rework. I lack a qbuf-style put/get API. The
>    contents of the blob need a review. Seems feasible.
> * relocation between VFs, on my TODO. The code already checks the vfn but
>    needs tightening and better support.
> * interrupts bits: will look into it. hopefully, the model "only" needs
>    to be more precise. This was a difficult aspect of the igb to understand
>    and I am still learning.
> * dirty tracking, I worked on 2 different interfaces and some of the code
>    fell through the cracks. DMA failures, bitmap clearing, size validation,
>    all need improvements. Mostly addressed already, remaining items are 
> code
>    reorg. should be fine.
> * quiesce: looks like a bug in the model. Will check.
> * RARP : I got inspiration from the tests. you are right saying it belongs
>    to the management layer. Keeping it for now as it simplifies testing, 
> but
>    will move it out.
> * Stats: the addition is from yesterday. Needs fixes indeed.

Thanks for the summary. Well, the "AI disclaimer" states that you spent 
significant time on cleanup, but your recap highlights several critical 
issues.

Ideally, AI assistance should minimize manual labor while keeping 
quality high. Right now, the workflow seems inverted: the AI created the 
basic scaffolding, you spent significant time reworking it, yet critical 
flaws not mentioned in the "Caveats" and "Todo/Ideas" sections still 
remain. I believe we can do better with AI assistance.

In fact, I am part of the Codex for Open Source program and use GPT-5.6 
Sol Ultra extensively for reviews. It is quite effective at raising 
quality while cutting down manual effort. Most of the comments I 
provided came directly from Codex using a simple prompt: "review each 
commit." The model somehow possesses a latent understanding of QEMU 
requirements, allowing it to flag issues like bisectability breaks and 
lack of L0/L1/L2 boundary enforcement.

I expect that iterating with prompts like "review each commit" and "fix 
these issues" would automatically resolve most of these problems, 
reducing error-prone human intervention. I am not sure if Claude has the 
same depth of virtualization-specific knowledge, but even if it doesn't, 
you can feed your recap back into the model to let it handle the heavy 
lifting.

Moving forward, I see two potential directions for this work:

- Focus on high-level design: Explore alternative architectures like 
hacking virtio-net SR-IOV or keeping RARP out of QEMU. You could send 
the results as a fresh RFC while openly noting the rough edges. This 
allows us to discuss the design architecture without spending massive 
effort polishing the implementation details.

- Focus on code quality and advocacy: Burn more tokens to improve the 
code quality. Use this series as a concrete case study to demonstrate 
that AI assistance can successfully implement complex virtualization 
features. This could help shift project policy regarding AI usage and 
pave the way for upstreaming this and future AI-assisted patches.

Whichever direction you choose, reframing the narrative in the cover 
letter will better align the series with the project's goals.

Regards,
Akihiko Odaki

On 7/28/26 07:50, Akihiko Odaki wrote:
> On 2026/07/28 2:35, Cédric Le Goater wrote:
>> Akihiko,
>>
>> On 7/27/26 07:39, Cédric Le Goater wrote:
>>> Hello,
>>>
>>> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
>>> growing requirement, but real hardware with migration support is
>>> scarce and hard to debug. An emulated device provides a fully
>>> controlled testbed for developing and validating the entire software
>>> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
>>> QEMU, libvirt - and for tuning complex migration policies such as
>>> downtime convergence. It also serves as an educational reference for
>>> understanding VFIO migration end-to-end, from device state
>>> serialization to dirty page tracking.
> 
> This is another good example of using QEMU as a SR-IOV testbed.
> 
>>>
>>> This series adds an experimental VF live migration interface to the
>>> emulated igb (82576) device. It enables a vfio-pci variant driver
>>> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
>>> protocol with stop-copy and pre-copy support.
> I suggest looking into virtio-net as an alternative to igb. It would allow avoiding lots of complexities of igb.
> 
>>>
>>> The target scenario is nested virtualization:
>>>
>>>    L0 QEMU (these patches)
>>>      igb PF with x-vf-migration=on
>>>      └── VFs with migration BAR + vendor cap
>>>
>>>    L1 kernel
>>>      igb-vfio-pci variant driver [1]
>>>      translates VFIO migration v2 ioctls → BAR2 MMIO
>>>
>>>    L1 QEMU (stock, unmodified)
>>>      vfio-pci device model, standard migration fd
>>>
>>>    L2 guest
>>>      standard igbvf driver, unaware of migration
>>>
>>> The L1 QEMU is completely unmodified -- it sees a standard VFIO
>>> migratable device and uses the normal migration fd path.
>>>
>>> * Design
>>>
>>> The migration interface is exposed through a hidden 64KB PCI BAR
>>> (BAR2) on each VF, discovered via a vendor-specific PCI capability
>>> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
>>> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
>>> RESUMING, PRE_COPY).
>>>
>>> Device state is serialized as a versioned blob of per-VF register
>>> (offset, value) pairs covering control, interrupt, RX/TX queue,
>>> receive address (RA/RA2), etc. plus TX context descriptors and
>>> VFRE/VFTE enable bits. The blob is transferred via DMA through the PF
>>> device, since VFIO owns the VF's IOMMU domain and the variant driver
>>> maps its DMA buffers through the PF.
>>>
>>> Dirty page tracking is implemented with per-range bitmaps maintained
>>> in IGBCore. All VF DMA paths in igb_core.c (TX data, RX data,
>>> descriptor writeback) are instrumented to record touched pages. The
>>> variant driver registers tracked IOVA ranges and queries dirty bitmaps
>>> through a DMA shared buffer, using a single MMIO doorbell
>>> (DIRTY_CTRL=QUERY) per iteration.
>>>
>>> * Caveats
>>>
>>> Patch 1 introduces PCI_BASE_ADDRESS_MEM_ALWAYS_ON, a QEMU-internal
>>> BAR flag that keeps the migration BAR mapped even after VFIO's
>>> Function Level Reset clears PCI_COMMAND_MEMORY.
>>>
>>> The x-vf-migration property is experimental (x- prefix, default off)
>>> and the migration BAR register interface may change.
>>>
>>> The dirty bitmaps are maintained inside the device, which is not
>>> realistic for discrete NICs without on-chip DRAM.
>>>
>>> * Testing
>>>
>>> The target scenario is nested virtualization: L0 runs QEMU with an
>>> igb PF (x-vf-migration=on), L1 runs the igb-vfio-pci variant driver
>>> and an unmodified QEMU, and L2 runs a standard igbvf driver.
>>>
>>> Migration under iperf3 load works correctly: dirty page tracking
>>> converges (from ~2000 pages per PRE_COPY iteration down to ~280 at
>>> STOP_COPY), and STOP_COPY stays under 250ms.
>>>
>>> * Todo/Ideas
>>>
>>>    1. Add migration blocker when x-vf-migration=on (no VMState yet) or
>>>       add VMState support for L0 migration (dirty bitmaps, tracking
>>>       engines, migration BAR registers, stats)
> 
> If you have AI assistance, why don't you just add migration blocker? It should be just one prompt away and not more troublesome than leaving this "Todo". ;)
> 
>>>    2. Add PRE_COPY match data validation (magic, version, caps)
>>>    3. Support driver-provided DMA bitmaps per dirty range
>>>       (DIRTY_RANGE_ADDR_LO/HI)
>>>    4. Add QMP/HMP knobs for error injection and capability tuning:
>>>      . Dirty rate throttling: artificial delay or rate limit on DMA
>>>        dirty tracking to simulate different convergence scenarios
>>>      . State blob size inflation: pad the migration blob to stress
>>>        large-state transfers and test DATA_SIZE limits
>>>      . Migration phase timing: expose per-VF counters for time spent
>>>        in each state (PRE_COPY duration, STOP_COPY latency)
>>>      . Hot page simulation: mark specific page ranges as always-dirty
>>>        to test worst-case convergence
>>>      . Error injection: force STATUS error codes, fail DMA transfers,
>>>        corrupt state blobs, mask CAPS bits, etc.
>>>    5. Add qtests for migration state machine transitions, dirty page
>>>       tracking ?
>>>
>>> * Credits
>>>
>>> Alex Williamson suggested the overall approach: a hidden migration BAR
>>> discovered via a vendor-specific PCI capability, the "vf-migration"
>>> device property to gate the feature. Thanks for the ever ongoing
>>> support and valuable discussions throughout these years.
>>>
>>> * AI disclaimer
>>>
>>> The lack of a migration-capable device has been a recurring pain point
>>> for VFIO development over the years, and we hope this proposal
>>> demonstrates the value of having one.
>>>
>>> Claude was used to analyze the IGB PF and VF internal state and
>>> identify the pain points of a working live migration of such devices.
>>> The generated code served as a starting point but *significant* time
>>> was then spent cleaning up, reworking, and shaping it into a clear,
>>> reviewable proposal. As QEMU does not yet accept AI-assisted
>>> contributions, this series is submitted as an RFC.
>>>
>>> Thanks,
>>>
>>> C.
>>>
>>> [1] https://github.com/legoater/vfio-pci-extras
>>>
>>> Cédric Le Goater (11):
>>>    pci: Add PCI_BASE_ADDRESS_MEM_ALWAYS_ON BAR flag
>>>    igb: Add x-vf-migration property and vendor-specific capability for
>>>      IGBVF
>>>    igb: Add migration BAR with state machine
>>>    igb: Add VF state serialization for live migration
>>>    igb: Add VF post-load fixups for live migration
>>>    igb: Add dirty page tracking for IGBVF migration
>>>    igb: Quiesce VFs on STOP and include PF enable state in migration blob
>>>    igb: Fix post-migration RX ring deadlock
>>>    igb: Send RARP after VF migration to update bridge FDB
>>>    docs: Add igb VF migration testing setup guide
>>>    igb: Add migration statistics registers to VF migration BAR
>>>
>>>   MAINTAINERS                           |    6 +
>>>   docs/system/device-emulation.rst      |    1 +
>>>   docs/system/devices/igb-migration.rst |  313 +++++++
>>>   docs/system/devices/igb.rst           |    6 +
>>>   hw/net/igb_common.h                   |   11 +
>>>   hw/net/igb_core.h                     |    8 +
>>>   hw/net/igb_migration.h                |  201 +++++
>>>   include/hw/pci/pci.h                  |    6 +
>>>   hw/net/igb.c                          |   18 +
>>>   hw/net/igb_core.c                     |  144 ++-
>>>   hw/net/igb_migration.c                | 1194 +++++++++++++++++++++++++
>>>   hw/net/igbvf.c                        |   29 +-
>>>   hw/pci/pci.c                          |    6 +-
>>>   hw/net/meson.build                    |    2 +-
>>>   hw/net/trace-events                   |   20 +
>>>   15 files changed, 1939 insertions(+), 26 deletions(-)
>>>   create mode 100644 docs/system/devices/igb-migration.rst
>>>   create mode 100644 hw/net/igb_migration.h
>>>   create mode 100644 hw/net/igb_migration.c
>>>
>>
>> First thanks for the quick review ! I will recap here :
>>
>> * bisectability, will improve the first patches.
>> * state serialization, the blob management is awful. I was expecting
>>    flames. Yes. it needs a rework. I lack a qbuf-style put/get API. The
>>    contents of the blob need a review. Seems feasible.
>> * relocation between VFs, on my TODO. The code already checks the vfn but
>>    needs tightening and better support.
>> * interrupts bits: will look into it. hopefully, the model "only" needs
>>    to be more precise. This was a difficult aspect of the igb to understand
>>    and I am still learning.
>> * dirty tracking, I worked on 2 different interfaces and some of the code
>>    fell through the cracks. DMA failures, bitmap clearing, size validation,
>>    all need improvements. Mostly addressed already, remaining items are code
>>    reorg. should be fine.
>> * quiesce: looks like a bug in the model. Will check.
>> * RARP : I got inspiration from the tests. you are right saying it belongs
>>    to the management layer. Keeping it for now as it simplifies testing, but
>>    will move it out.
>> * Stats: the addition is from yesterday. Needs fixes indeed.
> 
> Thanks for the summary. Well, the "AI disclaimer" states that you spent significant time on cleanup, but your recap highlights several critical issues.

The issues are mostly related to the IGB VF state. It was expected.
Dirty tracking issues reflect my various attempts of using MMIOs and
shared buffer. The rest are bugs :)

As for AI, yes, I find it useful to draft code, but really, it
requires a lot of directions and control. At the end, some parts
are not elegant or just wrong for some reasons. I don't trust the
output.

Where it is really good at, and where it saved me a lot of time,
is in analyzing QEMU and kernel logs of the L0/L1/L2 software stack.
Extracting the relevant info, correlating events, doing the timing
analysis, adding/removing logs. I did similar work in the past on
nested PPC emulation, and it was a painful and time-consuming process
without this kind of assistance.

Once the core debug is done, you can use it to produce python
scripts to automate the process and use that for the follow ups.
> 
> Ideally, AI assistance should minimize manual labor while keeping quality high. > Right now, the workflow seems inverted: the AI created the basic scaffolding, you spent significant time reworking it, 
> yet critical flaws not mentioned in the "Caveats" and "Todo/Ideas" sections still remain. I believe we can do better with AI assistance.

Your review uncovered several important issues in the interrupt
handling that I hadn't expected Claude (Opus 4.6) to miss.
  
> In fact, I am part of the Codex for Open Source program and use GPT-5.6 Sol Ultra extensively for reviews. It is quite effective at raising quality while cutting down manual effort. Most of the comments I provided came directly from Codex using a simple prompt: "review each commit." The model somehow possesses a latent understanding of QEMU requirements, allowing it to flag issues like bisectability breaks and lack of L0/L1/L2 boundary enforcement.

All were good findings.

> I expect that iterating with prompts like "review each commit" and "fix these issues" would automatically resolve most of these problems, reducing error-prone human intervention. I am not sure if Claude has the same depth of virtualization-specific knowledge, but even if it doesn't, you can feed your recap back into the model to let it handle the heavy lifting.

yes. I will try to fix most of the issues with the current model
before moving the the next version.

> Moving forward, I see two potential directions for this work:
> 
> - Focus on high-level design: Explore alternative architectures like hacking virtio-net SR-IOV or keeping RARP out of QEMU. You could send the results as a fresh RFC while openly noting the rough edges. This allows us to discuss the design architecture without spending massive effort polishing the implementation details.

Alex proposed good directions for a new HW implementation. I will
keep the igb vf as a target for now. Hopefully, I will have time
to fix the igb vf serialization and the interrupt handling for
better support. All issues seems feasible. Takes time.

virtio-net SR-IOV could be next.

As for RARP, yes. I will add a disclaimer saying that we need it
for tests. It's a extra patch any how.

  
> - Focus on code quality and advocacy: Burn more tokens to improve the code quality. Use this series as a concrete case study to demonstrate that AI assistance can successfully implement complex virtualization features. This could help shift project policy regarding AI usage and pave the way for upstreaming this and future AI-assisted patches.

Yes. that's one aspect.

Regarding the VFIO subsystem, improving testing is necessary and
it won't happen without a complex framework that people can use.

> 
> Whichever direction you choose, reframing the narrative in the cover letter will better align the series with the project's goals.
It's upstream first of course :)

Thanks,

C.

On Mon, 27 Jul 2026 07:39:24 +0200
Cédric Le Goater <clg@redhat.com> wrote:

> Hello,
> 
> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
> growing requirement, but real hardware with migration support is
> scarce and hard to debug. An emulated device provides a fully
> controlled testbed for developing and validating the entire software
> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
> QEMU, libvirt - and for tuning complex migration policies such as
> downtime convergence. It also serves as an educational reference for
> understanding VFIO migration end-to-end, from device state
> serialization to dirty page tracking.
> 
> This series adds an experimental VF live migration interface to the
> emulated igb (82576) device. It enables a vfio-pci variant driver
> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
> protocol with stop-copy and pre-copy support.
> 
> The target scenario is nested virtualization:
> 
>   L0 QEMU (these patches)
>     igb PF with x-vf-migration=on
>     └── VFs with migration BAR + vendor cap
> 
>   L1 kernel
>     igb-vfio-pci variant driver [1]
>     translates VFIO migration v2 ioctls → BAR2 MMIO
> 
>   L1 QEMU (stock, unmodified)
>     vfio-pci device model, standard migration fd
> 
>   L2 guest
>     standard igbvf driver, unaware of migration
> 
> The L1 QEMU is completely unmodified -- it sees a standard VFIO
> migratable device and uses the normal migration fd path.
> 
> * Design
> 
> The migration interface is exposed through a hidden 64KB PCI BAR
> (BAR2) on each VF, discovered via a vendor-specific PCI capability
> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
> RESUMING, PRE_COPY).

I think you're placing the migration BAR on the VF in order to
implement this in a small footprint, QEMU + vfio-pci variant driver,
without PF guest driver changes.  A model that better matches real
world hardware might be to put the migration BAR on the PF, segmented
per VF, and then have the PF driver vend those segments out to the VF
drivers.  That would remove the BAR always mapped problem, but expands
the footprint to include the PF driver.  However, we're not exactly
clean with respect to the PF driver as implemented here when we're
going around the PF driver's back to setup DMA mappings.

Can we take advantage of the fact that this is a virtual device to
avoid all these warts?

For example, do we really need MMIO BAR space for the register set
exposed or can we prune that down to some key registers and doorbells
and move the rest to memory?  We can put the vendor capability in
extended config space to give ourselves more room to work with if
necessary.  We also don't really need to play by the physical rules for
access, the variant driver in the L1 kernel can allocate contiguous
ranges and write GPAs into config space registers.  L0 QEMU can just
write migration data and dirty bitmaps directly to those GPAs,
bypassing any pretense of DMA mapping.

There might be some tricks we can steal from virtio as it seems to
optionally honor things like vIOMMUs as well.  Anyway, if we want to
confine the implementation to the virtual VF, avoiding dependencies on
the PF driver, both at the cross-driver API and device DMA state, I
think we can probably lean harder on QEMU being able to push data into
an arbitrary GPA regardless of the IO topology we're exposing.  Thanks,

Alex

On 7/27/26 22:36, Alex Williamson wrote:
> On Mon, 27 Jul 2026 07:39:24 +0200
> Cédric Le Goater <clg@redhat.com> wrote:
> 
>> Hello,
>>
>> Live migration of VFIO-passthrough devices - SR-IOV VFs, vGPUs - is a
>> growing requirement, but real hardware with migration support is
>> scarce and hard to debug. An emulated device provides a fully
>> controlled testbed for developing and validating the entire software
>> stack - vfio-pci variant drivers, VFIO core migration v2 framework,
>> QEMU, libvirt - and for tuning complex migration policies such as
>> downtime convergence. It also serves as an educational reference for
>> understanding VFIO migration end-to-end, from device state
>> serialization to dirty page tracking.
>>
>> This series adds an experimental VF live migration interface to the
>> emulated igb (82576) device. It enables a vfio-pci variant driver
>> (igb-vfio-pci) to migrate VFs using the standard VFIO migration v2
>> protocol with stop-copy and pre-copy support.
>>
>> The target scenario is nested virtualization:
>>
>>    L0 QEMU (these patches)
>>      igb PF with x-vf-migration=on
>>      └── VFs with migration BAR + vendor cap
>>
>>    L1 kernel
>>      igb-vfio-pci variant driver [1]
>>      translates VFIO migration v2 ioctls → BAR2 MMIO
>>
>>    L1 QEMU (stock, unmodified)
>>      vfio-pci device model, standard migration fd
>>
>>    L2 guest
>>      standard igbvf driver, unaware of migration
>>
>> The L1 QEMU is completely unmodified -- it sees a standard VFIO
>> migratable device and uses the normal migration fd path.
>>
>> * Design
>>
>> The migration interface is exposed through a hidden 64KB PCI BAR
>> (BAR2) on each VF, discovered via a vendor-specific PCI capability
>> ("MIGB", PCI_CAP_ID_VNDR). The BAR exposes a register-based state
>> machine that mirrors VFIO migration states (RUNNING, STOP, STOP_COPY,
>> RESUMING, PRE_COPY).
> 
> I think you're placing the migration BAR on the VF in order to
> implement this in a small footprint, QEMU + vfio-pci variant driver,
> without PF guest driver changes.  

yes.

> A model that better matches real
> world hardware might be to put the migration BAR on the PF, segmented
> per VF, and then have the PF driver vend those segments out to the VF
> drivers.  

True.

On the migration topic, I saw that the SR-IOV specs had "VF Migration
State Array" feature, which was deprecated.

> That would remove the BAR always mapped problem, 

That's the main problem today.

> but expands
> the footprint to include the PF driver.  However, we're not exactly
> clean with respect to the PF driver as implemented here when we're
> going around the PF driver's back to setup DMA mappings.

That's not uncommon today. The PDS vfio-pci variant driver setups DMA
mappings in the PF to migrate the VF. But yes, given that the IGB PF
has no idea that a VF could be migrated, it's a bit of a resource hijack.

> Can we take advantage of the fact that this is a virtual device to
> avoid all these warts?
> 
> For example, do we really need MMIO BAR space for the register set
> exposed or can we prune that down to some key registers and doorbells
> and move the rest to memory?  We can put the vendor capability in
> extended config space to give ourselves more room to work with if
> necessary.  

There is plenty of space in the extended config space. The required
register set is relatively small.

> We also don't really need to play by the physical rules for
> access, the variant driver in the L1 kernel can allocate contiguous
> ranges and write GPAs into config space registers.  

yes.

> L0 QEMU can just
> write migration data and dirty bitmaps directly to those GPAs,
> bypassing any pretense of DMA mapping.

yes that works.

A previous implementation of this proposal was allocating GPAs.
I then preferred a solution that was more PCI friendly. Anyhow,
it's not a problem to change it again, once we agree on the
HW interface.

> There might be some tricks we can steal from virtio as it seems to
> optionally honor things like vIOMMUs as well. 

Yes. That's where the PCI DMAs mapping were interesting. I will
look at it.

> Anyway, if we want to
> confine the implementation to the virtual VF, avoiding dependencies on
> the PF driver, both at the cross-driver API and device DMA state, I
> think we can probably lean harder on QEMU being able to push data into
> an arbitrary GPA regardless of the IO topology we're exposing.  

So, the extra PCI BAR is indeed a problem and we should switch to
extended config space to avoid it. I have been exploring several
ideas and this proposal is an hybrid MMIO/shared buffer solution.


Here are my plans for the next.

Reduce the register set to a minimum and use a shared buffer for
all commands. The driver would allocate a single large enough buffer
for the largest command payload (DIRTY_QUERY with bitmap) and write
its GPA into BUF_ADDR once at init. Each operation is: fill buffer,
kick CTRL, poll STATUS.


Registers in extended config space :

   0x00    Header        Cap ID
   0x04    CAPS          Features, max_ranges, pgsizes
   0x08    CTRL          Doorbell (state transitions, dirty ops)
   0x0C    STATUS        Completion + error code
   0x10    BUF_ADDR_LO   Shared buffer GPA low
   0x14    BUF_ADDR_HI   Shared buffer GPA high

Commands and payload :

   1. SET_STATE
   0x00    target_state    driver      Requested VFIO state

   2. SAVE
   0x00    data_size       device      Bytes written  (32-bit)
   0x04    reserved        -           May be more reserved bytes to align data
                                       on 32bits
   0x08    data[]          device      State blob

   3. LOAD
   0x00    data_size       driver      Bytes to load (32-bit)
   0x04    reserved        -           May be more reserved
   0x08    data[]          driver      State blob

   4. DIRTY_ENABLE
   0x00    iova            driver      Range start (64-bit)
   0x08    size            driver      Range size (64-bit)
   0x10    pgsize          driver      Page granularity
   0x14    reserved        -

   5. DIRTY_DISABLE

   6. DIRTY_QUERY
   Request (driver):
   0x00    iova            driver      Query range start (64-bit)
   0x08    size            driver      Query range size (64-bit)
   0x10    pgsize          driver      Page granularity (for queries)
   0x14    reserved        -

   Response (device):
   0x18    bitmap_size     device      Bytes in bitmap
   0x1C    dirty_pages     device      Set bits count
   0x20    dma_writes      device      DMA writes since enable (64-bit)
   0x28    reserved        -
   0x30    bitmap[]        device      Dirty page bitmap

   7. GET_STATS
   0x00    dma_writes      device      (32-bit)
   0x04    dma_bytes       device      (64-bit)
   0x0C    dirty_pages_set device
   0x10    dirty_pages_clr device
   0x14    dirty_page_count device
   0x18    dirty_query_cnt device


We could add a shared buffer header (not sure this is useful though)

   0x00    command         driver      Operation to perform
   0x04    status          device      Completion + error code
   0x08    payload_size    both        Size of payload following header
   0x0C    reserved        -           Alignment

Do we want 'flags' anywhere ?

Thanks,

C.