[libvirt RFCv6 00/27] multifd save restore prototype

Claudio Fontana posted 27 patches 1 week, 4 days ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/20220505125241.13291-1-cfontana@suse.de
docs/formatsnapshot.rst                       |   5 +-
docs/manpages/virsh.rst                       |  34 +-
include/libvirt/libvirt-domain.h              |  49 ++
po/POTFILES.in                                |   1 +
src/driver-hypervisor.h                       |  14 +
src/libvirt-domain.c                          |  99 +++-
src/libvirt_private.syms                      |   1 +
src/libvirt_public.syms                       |   6 +
src/qemu/qemu_capabilities.c                  |   6 +
src/qemu/qemu_capabilities.h                  |   4 +
src/qemu/qemu_driver.c                        | 239 +++++++--
src/qemu/qemu_migration.c                     | 155 ++++--
src/qemu/qemu_migration.h                     |  16 +-
src/qemu/qemu_migration_params.c              |  71 ++-
src/qemu/qemu_migration_params.h              |  15 +
src/qemu/qemu_process.c                       |   3 +-
src/qemu/qemu_process.h                       |   5 +-
src/qemu/qemu_saveimage.c                     | 496 ++++++++++++++----
src/qemu/qemu_saveimage.h                     |  49 +-
src/qemu/qemu_snapshot.c                      |   6 +-
src/remote/remote_driver.c                    |   2 +
src/remote/remote_protocol.x                  |  29 +-
src/remote_protocol-structs                   |  17 +
src/rpc/gendispatch.pl                        |   5 +-
src/util/iohelper.c                           | 162 +-----
src/util/meson.build                          |  19 +
src/util/multifd-helper.c                     | 249 +++++++++
src/util/virfile.c                            | 218 ++++++++
src/util/virfile.h                            |   2 +
src/util/virthread.c                          |   5 +
src/util/virthread.h                          |   1 +
.../caps_4.0.0.aarch64.xml                    |   1 +
.../qemucapabilitiesdata/caps_4.0.0.ppc64.xml |   1 +
.../caps_4.0.0.riscv32.xml                    |   1 +
.../caps_4.0.0.riscv64.xml                    |   1 +
.../qemucapabilitiesdata/caps_4.0.0.s390x.xml |   1 +
.../caps_4.0.0.x86_64.xml                     |   1 +
.../caps_4.1.0.x86_64.xml                     |   1 +
.../caps_4.2.0.aarch64.xml                    |   1 +
.../qemucapabilitiesdata/caps_4.2.0.ppc64.xml |   1 +
.../qemucapabilitiesdata/caps_4.2.0.s390x.xml |   1 +
.../caps_4.2.0.x86_64.xml                     |   1 +
.../caps_5.0.0.aarch64.xml                    |   2 +
.../qemucapabilitiesdata/caps_5.0.0.ppc64.xml |   2 +
.../caps_5.0.0.riscv64.xml                    |   2 +
.../caps_5.0.0.x86_64.xml                     |   2 +
.../qemucapabilitiesdata/caps_5.1.0.sparc.xml |   2 +
.../caps_5.1.0.x86_64.xml                     |   2 +
.../caps_5.2.0.aarch64.xml                    |   2 +
.../qemucapabilitiesdata/caps_5.2.0.ppc64.xml |   2 +
.../caps_5.2.0.riscv64.xml                    |   2 +
.../qemucapabilitiesdata/caps_5.2.0.s390x.xml |   2 +
.../caps_5.2.0.x86_64.xml                     |   2 +
.../caps_6.0.0.aarch64.xml                    |   2 +
.../qemucapabilitiesdata/caps_6.0.0.s390x.xml |   2 +
.../caps_6.0.0.x86_64.xml                     |   2 +
.../caps_6.1.0.x86_64.xml                     |   2 +
.../caps_6.2.0.aarch64.xml                    |   2 +
.../qemucapabilitiesdata/caps_6.2.0.ppc64.xml |   2 +
.../caps_6.2.0.x86_64.xml                     |   2 +
.../caps_7.0.0.aarch64.xml                    |   2 +
.../qemucapabilitiesdata/caps_7.0.0.ppc64.xml |   2 +
.../caps_7.0.0.x86_64.xml                     |   2 +
tools/virsh-domain.c                          |  96 +++-
64 files changed, 1686 insertions(+), 446 deletions(-)
create mode 100644 src/util/multifd-helper.c
[libvirt RFCv6 00/27] multifd save restore prototype
Posted by Claudio Fontana 1 week, 4 days ago
This a respin of the multifd save prototype,
rebased on post 8.3.0 release.

There are still quite a few rough edges:

KNOWN ISSUES:

1) this applies only to virsh save and virsh restore for now
   (no managed save etc).

2) error handling is not good yet, especially during resume,
   errors may leave behind a qemu process and such.
   May need some help find all of these cases

3) the compression part is demonstrative only, there needs
   to be more attention to compression options, and detecting
   the compression used to store the multifd saves.

...

changes from v4:

* runIO renamed to virFileDiskCopy and rethought arguments

* renamed new APIs from ...ParametersFlags to ...Params

* introduce the new virDomainSaveParams and virDomainRestoreParams
  without any additional parameters, so they can be upstreamed first.

* solved the issue in the gendispatch.pl script generating code that
  was missing the conn parameter.

---

changes from v3:

* reordered series to have all helper-related change at the start

* solved all reported issues from ninja test, including documentation

* fixed most broken migration capabilities code (still imperfect likely)

* added G_GNUC_UNUSED as needed

* after multifd restore, added what I think were the missing operations:

  qemuProcessRefreshState(),
  qemuProcessStartCPUs() - most importantly,
  virDomainObjSave()

  The domain now starts running after restore without further encouragement

* removed the sleep(10) from the multifd-helper


changes from v2:

* added ability to restore the VM from disk using multifd

* fixed the multifd-helper to work in both directions,
  assuming the need to listen for save, and connect for restore.

* fixed a large number of bugs, and probably introduced some :-)


Thanks for your thoughts,

Claudio


Claudio Fontana (27):
  iohelper: introduce new struct to carry copy operation parameters
  iohelper: refactor copy operation as a separate function
  iohelper: move runIO function to virfile.c
  virfile: rename runIO to virFileDiskCopy
  virfile: change virFileDiskCopy arguments to extend beyond stdin,
    stdout
  virfile: add comment about the use of SEEK_END in virFileDiskCopy
  multifd-helper: new helper for parallel save/restore
  libvirt: introduce virDomainSaveParams public API
  libvirt: introduce virDomainRestoreParams public API
  remote: Add RPC support for the virDomainSaveParams API
  gendispatch: add DomainRestoreParams as requiring conn argument
  remote: Add RPC support for the virDomainRestoreParams API
  qemu: add implementation for virDomainSaveParams API
  qemu: add implementation for virDomainRestoreParams API
  libvirt: add new VIR_DOMAIN_SAVE_PARALLEL flag and parameter
  qemu: add stub support for VIR_DOMAIN_SAVE_PARALLEL in save
  qemu: add stub support for VIR_DOMAIN_SAVE_PARALLEL in restore
  qemu: saveimage: introduce virQEMUSaveFd
  qemu: wire up saveimage code with the multifd helper
  qemu: capabilities: add multifd to the probed migration capabilities
  qemu: implement qemuMigrationSrcToFilesMultiFd
  qemu: add parameter to qemuMigrationDstRun to skip waiting
  qemu: implement qemuSaveImageLoadMultiFd
  tools: add parallel parameter to virsh save command
  tools: add parallel parameter to virsh restore command
  docs: update refs to virDomainSaveParams and virDomainRestoreParams
  qemu: add migration parameter multifd-compression

 docs/formatsnapshot.rst                       |   5 +-
 docs/manpages/virsh.rst                       |  34 +-
 include/libvirt/libvirt-domain.h              |  49 ++
 po/POTFILES.in                                |   1 +
 src/driver-hypervisor.h                       |  14 +
 src/libvirt-domain.c                          |  99 +++-
 src/libvirt_private.syms                      |   1 +
 src/libvirt_public.syms                       |   6 +
 src/qemu/qemu_capabilities.c                  |   6 +
 src/qemu/qemu_capabilities.h                  |   4 +
 src/qemu/qemu_driver.c                        | 239 +++++++--
 src/qemu/qemu_migration.c                     | 155 ++++--
 src/qemu/qemu_migration.h                     |  16 +-
 src/qemu/qemu_migration_params.c              |  71 ++-
 src/qemu/qemu_migration_params.h              |  15 +
 src/qemu/qemu_process.c                       |   3 +-
 src/qemu/qemu_process.h                       |   5 +-
 src/qemu/qemu_saveimage.c                     | 496 ++++++++++++++----
 src/qemu/qemu_saveimage.h                     |  49 +-
 src/qemu/qemu_snapshot.c                      |   6 +-
 src/remote/remote_driver.c                    |   2 +
 src/remote/remote_protocol.x                  |  29 +-
 src/remote_protocol-structs                   |  17 +
 src/rpc/gendispatch.pl                        |   5 +-
 src/util/iohelper.c                           | 162 +-----
 src/util/meson.build                          |  19 +
 src/util/multifd-helper.c                     | 249 +++++++++
 src/util/virfile.c                            | 218 ++++++++
 src/util/virfile.h                            |   2 +
 src/util/virthread.c                          |   5 +
 src/util/virthread.h                          |   1 +
 .../caps_4.0.0.aarch64.xml                    |   1 +
 .../qemucapabilitiesdata/caps_4.0.0.ppc64.xml |   1 +
 .../caps_4.0.0.riscv32.xml                    |   1 +
 .../caps_4.0.0.riscv64.xml                    |   1 +
 .../qemucapabilitiesdata/caps_4.0.0.s390x.xml |   1 +
 .../caps_4.0.0.x86_64.xml                     |   1 +
 .../caps_4.1.0.x86_64.xml                     |   1 +
 .../caps_4.2.0.aarch64.xml                    |   1 +
 .../qemucapabilitiesdata/caps_4.2.0.ppc64.xml |   1 +
 .../qemucapabilitiesdata/caps_4.2.0.s390x.xml |   1 +
 .../caps_4.2.0.x86_64.xml                     |   1 +
 .../caps_5.0.0.aarch64.xml                    |   2 +
 .../qemucapabilitiesdata/caps_5.0.0.ppc64.xml |   2 +
 .../caps_5.0.0.riscv64.xml                    |   2 +
 .../caps_5.0.0.x86_64.xml                     |   2 +
 .../qemucapabilitiesdata/caps_5.1.0.sparc.xml |   2 +
 .../caps_5.1.0.x86_64.xml                     |   2 +
 .../caps_5.2.0.aarch64.xml                    |   2 +
 .../qemucapabilitiesdata/caps_5.2.0.ppc64.xml |   2 +
 .../caps_5.2.0.riscv64.xml                    |   2 +
 .../qemucapabilitiesdata/caps_5.2.0.s390x.xml |   2 +
 .../caps_5.2.0.x86_64.xml                     |   2 +
 .../caps_6.0.0.aarch64.xml                    |   2 +
 .../qemucapabilitiesdata/caps_6.0.0.s390x.xml |   2 +
 .../caps_6.0.0.x86_64.xml                     |   2 +
 .../caps_6.1.0.x86_64.xml                     |   2 +
 .../caps_6.2.0.aarch64.xml                    |   2 +
 .../qemucapabilitiesdata/caps_6.2.0.ppc64.xml |   2 +
 .../caps_6.2.0.x86_64.xml                     |   2 +
 .../caps_7.0.0.aarch64.xml                    |   2 +
 .../qemucapabilitiesdata/caps_7.0.0.ppc64.xml |   2 +
 .../caps_7.0.0.x86_64.xml                     |   2 +
 tools/virsh-domain.c                          |  96 +++-
 64 files changed, 1686 insertions(+), 446 deletions(-)
 create mode 100644 src/util/multifd-helper.c

-- 
2.35.3
Re: [libvirt RFCv6 00/27] multifd save restore prototype
Posted by Claudio Fontana 1 week, 3 days ago
Hi Daniel,

will repost a v7 today with many more improvements, in terms of split
and in terms of multifd-compression.

Thanks!

Claudio

On 5/5/22 2:52 PM, Claudio Fontana wrote:
> This a respin of the multifd save prototype,
> rebased on post 8.3.0 release.
> 
> There are still quite a few rough edges:
> 
> KNOWN ISSUES:
> 
> 1) this applies only to virsh save and virsh restore for now
>    (no managed save etc).
> 
> 2) error handling is not good yet, especially during resume,
>    errors may leave behind a qemu process and such.
>    May need some help find all of these cases
> 
> 3) the compression part is demonstrative only, there needs
>    to be more attention to compression options, and detecting
>    the compression used to store the multifd saves.
> 
> ...
> 
> changes from v4:
> 
> * runIO renamed to virFileDiskCopy and rethought arguments
> 
> * renamed new APIs from ...ParametersFlags to ...Params
> 
> * introduce the new virDomainSaveParams and virDomainRestoreParams
>   without any additional parameters, so they can be upstreamed first.
> 
> * solved the issue in the gendispatch.pl script generating code that
>   was missing the conn parameter.
> 
> ---
> 
> changes from v3:
> 
> * reordered series to have all helper-related change at the start
> 
> * solved all reported issues from ninja test, including documentation
> 
> * fixed most broken migration capabilities code (still imperfect likely)
> 
> * added G_GNUC_UNUSED as needed
> 
> * after multifd restore, added what I think were the missing operations:
> 
>   qemuProcessRefreshState(),
>   qemuProcessStartCPUs() - most importantly,
>   virDomainObjSave()
> 
>   The domain now starts running after restore without further encouragement
> 
> * removed the sleep(10) from the multifd-helper
> 
> 
> changes from v2:
> 
> * added ability to restore the VM from disk using multifd
> 
> * fixed the multifd-helper to work in both directions,
>   assuming the need to listen for save, and connect for restore.
> 
> * fixed a large number of bugs, and probably introduced some :-)
> 
> 
> Thanks for your thoughts,
> 
> Claudio
> 
> 
> Claudio Fontana (27):
>   iohelper: introduce new struct to carry copy operation parameters
>   iohelper: refactor copy operation as a separate function
>   iohelper: move runIO function to virfile.c
>   virfile: rename runIO to virFileDiskCopy
>   virfile: change virFileDiskCopy arguments to extend beyond stdin,
>     stdout
>   virfile: add comment about the use of SEEK_END in virFileDiskCopy
>   multifd-helper: new helper for parallel save/restore
>   libvirt: introduce virDomainSaveParams public API
>   libvirt: introduce virDomainRestoreParams public API
>   remote: Add RPC support for the virDomainSaveParams API
>   gendispatch: add DomainRestoreParams as requiring conn argument
>   remote: Add RPC support for the virDomainRestoreParams API
>   qemu: add implementation for virDomainSaveParams API
>   qemu: add implementation for virDomainRestoreParams API
>   libvirt: add new VIR_DOMAIN_SAVE_PARALLEL flag and parameter
>   qemu: add stub support for VIR_DOMAIN_SAVE_PARALLEL in save
>   qemu: add stub support for VIR_DOMAIN_SAVE_PARALLEL in restore
>   qemu: saveimage: introduce virQEMUSaveFd
>   qemu: wire up saveimage code with the multifd helper
>   qemu: capabilities: add multifd to the probed migration capabilities
>   qemu: implement qemuMigrationSrcToFilesMultiFd
>   qemu: add parameter to qemuMigrationDstRun to skip waiting
>   qemu: implement qemuSaveImageLoadMultiFd
>   tools: add parallel parameter to virsh save command
>   tools: add parallel parameter to virsh restore command
>   docs: update refs to virDomainSaveParams and virDomainRestoreParams
>   qemu: add migration parameter multifd-compression
> 
>  docs/formatsnapshot.rst                       |   5 +-
>  docs/manpages/virsh.rst                       |  34 +-
>  include/libvirt/libvirt-domain.h              |  49 ++
>  po/POTFILES.in                                |   1 +
>  src/driver-hypervisor.h                       |  14 +
>  src/libvirt-domain.c                          |  99 +++-
>  src/libvirt_private.syms                      |   1 +
>  src/libvirt_public.syms                       |   6 +
>  src/qemu/qemu_capabilities.c                  |   6 +
>  src/qemu/qemu_capabilities.h                  |   4 +
>  src/qemu/qemu_driver.c                        | 239 +++++++--
>  src/qemu/qemu_migration.c                     | 155 ++++--
>  src/qemu/qemu_migration.h                     |  16 +-
>  src/qemu/qemu_migration_params.c              |  71 ++-
>  src/qemu/qemu_migration_params.h              |  15 +
>  src/qemu/qemu_process.c                       |   3 +-
>  src/qemu/qemu_process.h                       |   5 +-
>  src/qemu/qemu_saveimage.c                     | 496 ++++++++++++++----
>  src/qemu/qemu_saveimage.h                     |  49 +-
>  src/qemu/qemu_snapshot.c                      |   6 +-
>  src/remote/remote_driver.c                    |   2 +
>  src/remote/remote_protocol.x                  |  29 +-
>  src/remote_protocol-structs                   |  17 +
>  src/rpc/gendispatch.pl                        |   5 +-
>  src/util/iohelper.c                           | 162 +-----
>  src/util/meson.build                          |  19 +
>  src/util/multifd-helper.c                     | 249 +++++++++
>  src/util/virfile.c                            | 218 ++++++++
>  src/util/virfile.h                            |   2 +
>  src/util/virthread.c                          |   5 +
>  src/util/virthread.h                          |   1 +
>  .../caps_4.0.0.aarch64.xml                    |   1 +
>  .../qemucapabilitiesdata/caps_4.0.0.ppc64.xml |   1 +
>  .../caps_4.0.0.riscv32.xml                    |   1 +
>  .../caps_4.0.0.riscv64.xml                    |   1 +
>  .../qemucapabilitiesdata/caps_4.0.0.s390x.xml |   1 +
>  .../caps_4.0.0.x86_64.xml                     |   1 +
>  .../caps_4.1.0.x86_64.xml                     |   1 +
>  .../caps_4.2.0.aarch64.xml                    |   1 +
>  .../qemucapabilitiesdata/caps_4.2.0.ppc64.xml |   1 +
>  .../qemucapabilitiesdata/caps_4.2.0.s390x.xml |   1 +
>  .../caps_4.2.0.x86_64.xml                     |   1 +
>  .../caps_5.0.0.aarch64.xml                    |   2 +
>  .../qemucapabilitiesdata/caps_5.0.0.ppc64.xml |   2 +
>  .../caps_5.0.0.riscv64.xml                    |   2 +
>  .../caps_5.0.0.x86_64.xml                     |   2 +
>  .../qemucapabilitiesdata/caps_5.1.0.sparc.xml |   2 +
>  .../caps_5.1.0.x86_64.xml                     |   2 +
>  .../caps_5.2.0.aarch64.xml                    |   2 +
>  .../qemucapabilitiesdata/caps_5.2.0.ppc64.xml |   2 +
>  .../caps_5.2.0.riscv64.xml                    |   2 +
>  .../qemucapabilitiesdata/caps_5.2.0.s390x.xml |   2 +
>  .../caps_5.2.0.x86_64.xml                     |   2 +
>  .../caps_6.0.0.aarch64.xml                    |   2 +
>  .../qemucapabilitiesdata/caps_6.0.0.s390x.xml |   2 +
>  .../caps_6.0.0.x86_64.xml                     |   2 +
>  .../caps_6.1.0.x86_64.xml                     |   2 +
>  .../caps_6.2.0.aarch64.xml                    |   2 +
>  .../qemucapabilitiesdata/caps_6.2.0.ppc64.xml |   2 +
>  .../caps_6.2.0.x86_64.xml                     |   2 +
>  .../caps_7.0.0.aarch64.xml                    |   2 +
>  .../qemucapabilitiesdata/caps_7.0.0.ppc64.xml |   2 +
>  .../caps_7.0.0.x86_64.xml                     |   2 +
>  tools/virsh-domain.c                          |  96 +++-
>  64 files changed, 1686 insertions(+), 446 deletions(-)
>  create mode 100644 src/util/multifd-helper.c
>