[PATCH v5 0/8] hw/nvme: add basic live migration support

Alexander Mikhalitsyn posted 8 patches 2 weeks, 6 days ago
Failed in applying to current master (apply log)
hw/nvme/ctrl.c                                | 874 +++++++++++++++++-
hw/nvme/ns.c                                  | 160 ++++
hw/nvme/nvme.h                                |   8 +
hw/nvme/trace-events                          |  11 +
include/block/nvme.h                          |  12 +
include/migration/vmstate.h                   |  83 +-
migration/savevm.c                            |  26 +
migration/vmstate-types.c                     |  91 ++
migration/vmstate.c                           |  64 +-
tests/functional/migration.py                 |  22 +-
tests/functional/x86_64/meson.build           |   1 +
.../functional/x86_64/test_nvme_migration.py  | 159 ++++
tests/unit/test-vmstate.c                     | 157 ++++
13 files changed, 1618 insertions(+), 50 deletions(-)
create mode 100755 tests/functional/x86_64/test_nvme_migration.py
[PATCH v5 0/8] hw/nvme: add basic live migration support
Posted by Alexander Mikhalitsyn 2 weeks, 6 days ago
From: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>

Dear friends,

This patchset adds basic live migration support for
QEMU emulated NVMe device.

Implementation has some limitations:
- only one NVMe namespace is supported
- SMART counters are not preserved
- CMB is not supported
- PMR is not supported
- SPDM is not supported
- SR-IOV is not supported

I believe this is something I can support in next patchset versions or
separately on-demand (when usecase appears).

Testing.

This patch series was manually tested on:
- Debian 13.3 VM (kernel 6.12.69+deb13-amd64) using fio on *non-root* NVMe disk
  (root disk was virtio-scsi):

time fio --name=nvme-verify \
    --filename=/dev/nvme0n1 \
    --size=5G \
    --rw=randwrite \
    --bs=4k \
    --iodepth=16 \
    --numjobs=1 \
    --direct=0 \
    --ioengine=io_uring \
    --verify=crc32c \
    --verify_fatal=1

- Windows Server 2022 VM (NVMe drive was a *root* disk) with opened browser
  playing video.

No defects were found.

Git tree:
https://github.com/mihalicyn/qemu/commits/nvme-live-migration

Changelog for version 5:
- rebased on top of https://lore.kernel.org/all/20260304212303.667141-1-vsementsov@yandex-team.ru/
  (as Peter has requested)

Changelog for version 4:
- vmstate dynamic array support reworked as suggested by Peter Xu
  VMS_ARRAY_OF_POINTER_ALLOW_NULL flag was introduced
  qtests were added
- NVMe migration blockers were reworked as Klaus has requested earlier
  Now, instead of having "deny list" approach, we have more strict pattern
  of NVMe features filtering and it should be harded to break migration when
  adding new NVMe features.

Changelog for version 3:
- rebased
- simple functional test was added (in accordance with Klaus Jensen's review comment)
  $ meson test 'func-x86_64-nvme_migration' --setup thorough -C build

Changelog for version 2:
- full support for AERs (in-flight requests and queued events too)

Kind regards,
Alex

Alexander Mikhalitsyn (8):
  migration/vmstate: export vmstate_{load, save}_field helpers
  migration: add VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT{8, 32}_ALLOC
  tests/unit/test-vmstate: add tests for VMS_ARRAY_OF_POINTER_ALLOW_NULL
  tests/functional/migration: add VM launch/configure hooks
  hw/nvme: add migration blockers for non-supported cases
  hw/nvme: split nvme_init_sq/nvme_init_cq into helpers
  hw/nvme: add basic live migration support
  tests/functional/x86_64: add migration test for NVMe device

 hw/nvme/ctrl.c                                | 874 +++++++++++++++++-
 hw/nvme/ns.c                                  | 160 ++++
 hw/nvme/nvme.h                                |   8 +
 hw/nvme/trace-events                          |  11 +
 include/block/nvme.h                          |  12 +
 include/migration/vmstate.h                   |  83 +-
 migration/savevm.c                            |  26 +
 migration/vmstate-types.c                     |  91 ++
 migration/vmstate.c                           |  64 +-
 tests/functional/migration.py                 |  22 +-
 tests/functional/x86_64/meson.build           |   1 +
 .../functional/x86_64/test_nvme_migration.py  | 159 ++++
 tests/unit/test-vmstate.c                     | 157 ++++
 13 files changed, 1618 insertions(+), 50 deletions(-)
 create mode 100755 tests/functional/x86_64/test_nvme_migration.py

-- 
2.47.3
Re: [PATCH v5 0/8] hw/nvme: add basic live migration support
Posted by Alexander Mikhalitsyn 1 week ago
Hi Klaus,

sorry for bothering you with this.

I've fixed my patchset in accordance with all your suggestions from
https://lore.kernel.org/qemu-devel/aaFrB7fMMUPfMP_5@AALNPWKJENSEN.aal.scsc.local/
please can you take a look on my patchset once again?

Kind regards,
Alex

Am Di., 17. März 2026 um 11:27 Uhr schrieb Alexander Mikhalitsyn
<alexander@mihalicyn.com>:
>
> From: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
>
> Dear friends,
>
> This patchset adds basic live migration support for
> QEMU emulated NVMe device.
>
> Implementation has some limitations:
> - only one NVMe namespace is supported
> - SMART counters are not preserved
> - CMB is not supported
> - PMR is not supported
> - SPDM is not supported
> - SR-IOV is not supported
>
> I believe this is something I can support in next patchset versions or
> separately on-demand (when usecase appears).
>
> Testing.
>
> This patch series was manually tested on:
> - Debian 13.3 VM (kernel 6.12.69+deb13-amd64) using fio on *non-root* NVMe disk
>   (root disk was virtio-scsi):
>
> time fio --name=nvme-verify \
>     --filename=/dev/nvme0n1 \
>     --size=5G \
>     --rw=randwrite \
>     --bs=4k \
>     --iodepth=16 \
>     --numjobs=1 \
>     --direct=0 \
>     --ioengine=io_uring \
>     --verify=crc32c \
>     --verify_fatal=1
>
> - Windows Server 2022 VM (NVMe drive was a *root* disk) with opened browser
>   playing video.
>
> No defects were found.
>
> Git tree:
> https://github.com/mihalicyn/qemu/commits/nvme-live-migration
>
> Changelog for version 5:
> - rebased on top of https://lore.kernel.org/all/20260304212303.667141-1-vsementsov@yandex-team.ru/
>   (as Peter has requested)
>
> Changelog for version 4:
> - vmstate dynamic array support reworked as suggested by Peter Xu
>   VMS_ARRAY_OF_POINTER_ALLOW_NULL flag was introduced
>   qtests were added
> - NVMe migration blockers were reworked as Klaus has requested earlier
>   Now, instead of having "deny list" approach, we have more strict pattern
>   of NVMe features filtering and it should be harded to break migration when
>   adding new NVMe features.
>
> Changelog for version 3:
> - rebased
> - simple functional test was added (in accordance with Klaus Jensen's review comment)
>   $ meson test 'func-x86_64-nvme_migration' --setup thorough -C build
>
> Changelog for version 2:
> - full support for AERs (in-flight requests and queued events too)
>
> Kind regards,
> Alex
>
> Alexander Mikhalitsyn (8):
>   migration/vmstate: export vmstate_{load, save}_field helpers
>   migration: add VMSTATE_VARRAY_OF_POINTER_TO_STRUCT_UINT{8, 32}_ALLOC
>   tests/unit/test-vmstate: add tests for VMS_ARRAY_OF_POINTER_ALLOW_NULL
>   tests/functional/migration: add VM launch/configure hooks
>   hw/nvme: add migration blockers for non-supported cases
>   hw/nvme: split nvme_init_sq/nvme_init_cq into helpers
>   hw/nvme: add basic live migration support
>   tests/functional/x86_64: add migration test for NVMe device
>
>  hw/nvme/ctrl.c                                | 874 +++++++++++++++++-
>  hw/nvme/ns.c                                  | 160 ++++
>  hw/nvme/nvme.h                                |   8 +
>  hw/nvme/trace-events                          |  11 +
>  include/block/nvme.h                          |  12 +
>  include/migration/vmstate.h                   |  83 +-
>  migration/savevm.c                            |  26 +
>  migration/vmstate-types.c                     |  91 ++
>  migration/vmstate.c                           |  64 +-
>  tests/functional/migration.py                 |  22 +-
>  tests/functional/x86_64/meson.build           |   1 +
>  .../functional/x86_64/test_nvme_migration.py  | 159 ++++
>  tests/unit/test-vmstate.c                     | 157 ++++
>  13 files changed, 1618 insertions(+), 50 deletions(-)
>  create mode 100755 tests/functional/x86_64/test_nvme_migration.py
>
> --
> 2.47.3
>