[PATCH V4 00/18] qemu: support mapped-ram+directio+mulitfd

Jim Fehlig via Devel posted 18 patches 11 months, 1 week ago
Failed in applying to current master (apply log)
docs/manpages/virsh.rst          |  21 +++-
include/libvirt/libvirt-domain.h |  11 ++
src/libvirt-domain.c             |  92 +++++++++++---
src/qemu/qemu.conf.in            |   9 +-
src/qemu/qemu_driver.c           |  78 +++++++++---
src/qemu/qemu_fd.c               |  46 +++++++
src/qemu/qemu_fd.h               |   4 +
src/qemu/qemu_migration.c        | 204 ++++++++++++++++++++++---------
src/qemu/qemu_migration.h        |  10 +-
src/qemu/qemu_migration_params.c |  92 ++++++++++++++
src/qemu/qemu_migration_params.h |  17 +++
src/qemu/qemu_monitor.c          |  37 ++++++
src/qemu/qemu_monitor.h          |   5 +
src/qemu/qemu_process.c          | 100 ++++++++++-----
src/qemu/qemu_process.h          |  19 ++-
src/qemu/qemu_saveimage.c        | 120 ++++++++++++------
src/qemu/qemu_saveimage.h        |   4 +
src/qemu/qemu_snapshot.c         |  15 ++-
tools/virsh-domain.c             |  81 ++++++++++--
19 files changed, 775 insertions(+), 190 deletions(-)
[PATCH V4 00/18] qemu: support mapped-ram+directio+mulitfd
Posted by Jim Fehlig via Devel 11 months, 1 week ago
V4 series adding support for QEMU's mapped-ram stream format [1] and
migration capability. The use of mapped-ram is controlled by extending
save_image_format setting in qemu.conf with a new 'sparse' option. The
'raw' format continues to be the default.

Also included are patches that leverage mapped-ram to add support for
parallel save/restore.

Changes in V4:
* Rebased on latest master, including the series
  "qemu: Support specifying save image format"

V1:
https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/MNBHGQH7PKV4RXQZXLPAGMOTNEVR3JVS/

V2:
https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/GNJNAXSB77PNUKOZ7Q4D5RVF23SAVMDN/

V3:
https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/SK2PSRXO4MPKRZLP74GR4J2EUVI5VVHW/

[1] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/migration/mapped-ram.rst?ref_type=heads

Claudio Fontana (3):
  include: Define constants for parallel save/restore
  tools: add parallel parameter to virsh save command
  tools: add parallel parameter to virsh restore command

Jim Fehlig (15):
  lib: virDomain{Save,Restore}Params: Ensure absolute path
  qemu: Add function to get FDPass object from monitor
  qemu: Add function to check capability in migration params
  qemu: Add function to get bool value from migration params
  qemu: Add mapped-ram migration capability
  qemu: Add function to get migration params for save
  qemu_saveimage: add "sparse" to supported save image formats
  qemu: Add helper function for creating save image fd
  qemu: Add support for mapped-ram on save
  qemu: Move creation of qemuProcessIncomingDef struct
  qemu: Apply migration parameters in qemuMigrationDstRun
  qemu: Add support for mapped-ram on restore
  qemu: Support O_DIRECT with mapped-ram on save
  qemu: Support O_DIRECT with mapped-ram on restore
  qemu: Add support for parallel save and restore

 docs/manpages/virsh.rst          |  21 +++-
 include/libvirt/libvirt-domain.h |  11 ++
 src/libvirt-domain.c             |  92 +++++++++++---
 src/qemu/qemu.conf.in            |   9 +-
 src/qemu/qemu_driver.c           |  78 +++++++++---
 src/qemu/qemu_fd.c               |  46 +++++++
 src/qemu/qemu_fd.h               |   4 +
 src/qemu/qemu_migration.c        | 204 ++++++++++++++++++++++---------
 src/qemu/qemu_migration.h        |  10 +-
 src/qemu/qemu_migration_params.c |  92 ++++++++++++++
 src/qemu/qemu_migration_params.h |  17 +++
 src/qemu/qemu_monitor.c          |  37 ++++++
 src/qemu/qemu_monitor.h          |   5 +
 src/qemu/qemu_process.c          | 100 ++++++++++-----
 src/qemu/qemu_process.h          |  19 ++-
 src/qemu/qemu_saveimage.c        | 120 ++++++++++++------
 src/qemu/qemu_saveimage.h        |   4 +
 src/qemu/qemu_snapshot.c         |  15 ++-
 tools/virsh-domain.c             |  81 ++++++++++--
 19 files changed, 775 insertions(+), 190 deletions(-)

-- 
2.43.0
Re: [PATCH V4 00/18] qemu: support mapped-ram+directio+mulitfd
Posted by Daniel P. Berrangé via Devel 10 months, 3 weeks ago
Hi Jim,

There are a couple of CI failures after this, one on 32-bit and
one with CLang

On Wed, Mar 05, 2025 at 03:48:09PM -0700, Jim Fehlig via Devel wrote:
> V4 series adding support for QEMU's mapped-ram stream format [1] and
> migration capability. The use of mapped-ram is controlled by extending
> save_image_format setting in qemu.conf with a new 'sparse' option. The
> 'raw' format continues to be the default.
> 
> Also included are patches that leverage mapped-ram to add support for
> parallel save/restore.
> 
> Changes in V4:
> * Rebased on latest master, including the series
>   "qemu: Support specifying save image format"
> 
> V1:
> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/MNBHGQH7PKV4RXQZXLPAGMOTNEVR3JVS/
> 
> V2:
> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/GNJNAXSB77PNUKOZ7Q4D5RVF23SAVMDN/
> 
> V3:
> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/SK2PSRXO4MPKRZLP74GR4J2EUVI5VVHW/
> 
> [1] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/migration/mapped-ram.rst?ref_type=heads
> 
> Claudio Fontana (3):
>   include: Define constants for parallel save/restore
>   tools: add parallel parameter to virsh save command
>   tools: add parallel parameter to virsh restore command
> 
> Jim Fehlig (15):
>   lib: virDomain{Save,Restore}Params: Ensure absolute path
>   qemu: Add function to get FDPass object from monitor
>   qemu: Add function to check capability in migration params
>   qemu: Add function to get bool value from migration params
>   qemu: Add mapped-ram migration capability
>   qemu: Add function to get migration params for save
>   qemu_saveimage: add "sparse" to supported save image formats
>   qemu: Add helper function for creating save image fd
>   qemu: Add support for mapped-ram on save
>   qemu: Move creation of qemuProcessIncomingDef struct
>   qemu: Apply migration parameters in qemuMigrationDstRun
>   qemu: Add support for mapped-ram on restore
>   qemu: Support O_DIRECT with mapped-ram on save
>   qemu: Support O_DIRECT with mapped-ram on restore
>   qemu: Add support for parallel save and restore
> 
>  docs/manpages/virsh.rst          |  21 +++-
>  include/libvirt/libvirt-domain.h |  11 ++
>  src/libvirt-domain.c             |  92 +++++++++++---
>  src/qemu/qemu.conf.in            |   9 +-
>  src/qemu/qemu_driver.c           |  78 +++++++++---
>  src/qemu/qemu_fd.c               |  46 +++++++
>  src/qemu/qemu_fd.h               |   4 +
>  src/qemu/qemu_migration.c        | 204 ++++++++++++++++++++++---------
>  src/qemu/qemu_migration.h        |  10 +-
>  src/qemu/qemu_migration_params.c |  92 ++++++++++++++
>  src/qemu/qemu_migration_params.h |  17 +++
>  src/qemu/qemu_monitor.c          |  37 ++++++
>  src/qemu/qemu_monitor.h          |   5 +
>  src/qemu/qemu_process.c          | 100 ++++++++++-----
>  src/qemu/qemu_process.h          |  19 ++-
>  src/qemu/qemu_saveimage.c        | 120 ++++++++++++------
>  src/qemu/qemu_saveimage.h        |   4 +
>  src/qemu/qemu_snapshot.c         |  15 ++-
>  tools/virsh-domain.c             |  81 ++++++++++--
>  19 files changed, 775 insertions(+), 190 deletions(-)
> 
> -- 
> 2.43.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [PATCH V4 00/18] qemu: support mapped-ram+directio+mulitfd
Posted by Jim Fehlig via Devel 10 months, 3 weeks ago
On 3/20/25 11:55, Daniel P. Berrangé wrote:
> Hi Jim,
> 
> There are a couple of CI failures after this, one on 32-bit and
> one with CLang

Yep, received mail from gitlab as well. Will take a look...

Regards,
Jim

> 
> On Wed, Mar 05, 2025 at 03:48:09PM -0700, Jim Fehlig via Devel wrote:
>> V4 series adding support for QEMU's mapped-ram stream format [1] and
>> migration capability. The use of mapped-ram is controlled by extending
>> save_image_format setting in qemu.conf with a new 'sparse' option. The
>> 'raw' format continues to be the default.
>>
>> Also included are patches that leverage mapped-ram to add support for
>> parallel save/restore.
>>
>> Changes in V4:
>> * Rebased on latest master, including the series
>>    "qemu: Support specifying save image format"
>>
>> V1:
>> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/MNBHGQH7PKV4RXQZXLPAGMOTNEVR3JVS/
>>
>> V2:
>> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/GNJNAXSB77PNUKOZ7Q4D5RVF23SAVMDN/
>>
>> V3:
>> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/SK2PSRXO4MPKRZLP74GR4J2EUVI5VVHW/
>>
>> [1] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/migration/mapped-ram.rst?ref_type=heads
>>
>> Claudio Fontana (3):
>>    include: Define constants for parallel save/restore
>>    tools: add parallel parameter to virsh save command
>>    tools: add parallel parameter to virsh restore command
>>
>> Jim Fehlig (15):
>>    lib: virDomain{Save,Restore}Params: Ensure absolute path
>>    qemu: Add function to get FDPass object from monitor
>>    qemu: Add function to check capability in migration params
>>    qemu: Add function to get bool value from migration params
>>    qemu: Add mapped-ram migration capability
>>    qemu: Add function to get migration params for save
>>    qemu_saveimage: add "sparse" to supported save image formats
>>    qemu: Add helper function for creating save image fd
>>    qemu: Add support for mapped-ram on save
>>    qemu: Move creation of qemuProcessIncomingDef struct
>>    qemu: Apply migration parameters in qemuMigrationDstRun
>>    qemu: Add support for mapped-ram on restore
>>    qemu: Support O_DIRECT with mapped-ram on save
>>    qemu: Support O_DIRECT with mapped-ram on restore
>>    qemu: Add support for parallel save and restore
>>
>>   docs/manpages/virsh.rst          |  21 +++-
>>   include/libvirt/libvirt-domain.h |  11 ++
>>   src/libvirt-domain.c             |  92 +++++++++++---
>>   src/qemu/qemu.conf.in            |   9 +-
>>   src/qemu/qemu_driver.c           |  78 +++++++++---
>>   src/qemu/qemu_fd.c               |  46 +++++++
>>   src/qemu/qemu_fd.h               |   4 +
>>   src/qemu/qemu_migration.c        | 204 ++++++++++++++++++++++---------
>>   src/qemu/qemu_migration.h        |  10 +-
>>   src/qemu/qemu_migration_params.c |  92 ++++++++++++++
>>   src/qemu/qemu_migration_params.h |  17 +++
>>   src/qemu/qemu_monitor.c          |  37 ++++++
>>   src/qemu/qemu_monitor.h          |   5 +
>>   src/qemu/qemu_process.c          | 100 ++++++++++-----
>>   src/qemu/qemu_process.h          |  19 ++-
>>   src/qemu/qemu_saveimage.c        | 120 ++++++++++++------
>>   src/qemu/qemu_saveimage.h        |   4 +
>>   src/qemu/qemu_snapshot.c         |  15 ++-
>>   tools/virsh-domain.c             |  81 ++++++++++--
>>   19 files changed, 775 insertions(+), 190 deletions(-)
>>
>> -- 
>> 2.43.0
>>
> 
> With regards,
> Daniel
Re: [PATCH V4 00/18] qemu: support mapped-ram+directio+mulitfd
Posted by Jim Fehlig via Devel 11 months ago
Hi All,

Does anyone have time to review this series? QEMU has supported mapped-ram for a 
few releases now. Adding support in libvirt allows us to build parallel 
save/restore on top. Would be a nice feature for the next release :-).

Regards,
Jim

On 3/5/25 15:48, Jim Fehlig wrote:
> V4 series adding support for QEMU's mapped-ram stream format [1] and
> migration capability. The use of mapped-ram is controlled by extending
> save_image_format setting in qemu.conf with a new 'sparse' option. The
> 'raw' format continues to be the default.
> 
> Also included are patches that leverage mapped-ram to add support for
> parallel save/restore.
> 
> Changes in V4:
> * Rebased on latest master, including the series
>    "qemu: Support specifying save image format"
> 
> V1:
> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/MNBHGQH7PKV4RXQZXLPAGMOTNEVR3JVS/
> 
> V2:
> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/GNJNAXSB77PNUKOZ7Q4D5RVF23SAVMDN/
> 
> V3:
> https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/SK2PSRXO4MPKRZLP74GR4J2EUVI5VVHW/
> 
> [1] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/migration/mapped-ram.rst?ref_type=heads
> 
> Claudio Fontana (3):
>    include: Define constants for parallel save/restore
>    tools: add parallel parameter to virsh save command
>    tools: add parallel parameter to virsh restore command
> 
> Jim Fehlig (15):
>    lib: virDomain{Save,Restore}Params: Ensure absolute path
>    qemu: Add function to get FDPass object from monitor
>    qemu: Add function to check capability in migration params
>    qemu: Add function to get bool value from migration params
>    qemu: Add mapped-ram migration capability
>    qemu: Add function to get migration params for save
>    qemu_saveimage: add "sparse" to supported save image formats
>    qemu: Add helper function for creating save image fd
>    qemu: Add support for mapped-ram on save
>    qemu: Move creation of qemuProcessIncomingDef struct
>    qemu: Apply migration parameters in qemuMigrationDstRun
>    qemu: Add support for mapped-ram on restore
>    qemu: Support O_DIRECT with mapped-ram on save
>    qemu: Support O_DIRECT with mapped-ram on restore
>    qemu: Add support for parallel save and restore
> 
>   docs/manpages/virsh.rst          |  21 +++-
>   include/libvirt/libvirt-domain.h |  11 ++
>   src/libvirt-domain.c             |  92 +++++++++++---
>   src/qemu/qemu.conf.in            |   9 +-
>   src/qemu/qemu_driver.c           |  78 +++++++++---
>   src/qemu/qemu_fd.c               |  46 +++++++
>   src/qemu/qemu_fd.h               |   4 +
>   src/qemu/qemu_migration.c        | 204 ++++++++++++++++++++++---------
>   src/qemu/qemu_migration.h        |  10 +-
>   src/qemu/qemu_migration_params.c |  92 ++++++++++++++
>   src/qemu/qemu_migration_params.h |  17 +++
>   src/qemu/qemu_monitor.c          |  37 ++++++
>   src/qemu/qemu_monitor.h          |   5 +
>   src/qemu/qemu_process.c          | 100 ++++++++++-----
>   src/qemu/qemu_process.h          |  19 ++-
>   src/qemu/qemu_saveimage.c        | 120 ++++++++++++------
>   src/qemu/qemu_saveimage.h        |   4 +
>   src/qemu/qemu_snapshot.c         |  15 ++-
>   tools/virsh-domain.c             |  81 ++++++++++--
>   19 files changed, 775 insertions(+), 190 deletions(-)
>