[libvirt] [PATCH v4 00/20] Incremental Backup API additions

Eric Blake posted 20 patches 2 weeks ago
Test syntax-check passed
Failed in applying to current master (apply log)
include/libvirt/virterror.h                   |    7 +-
src/util/virerror.c                           |   12 +-
include/libvirt/libvirt-domain-checkpoint.h   |  176 ++
include/libvirt/libvirt-domain-snapshot.h     |    2 +-
include/libvirt/libvirt-domain.h              |   17 +-
include/libvirt/libvirt.h                     |    3 +-
src/access/viraccessperm.h                    |    8 +-
src/conf/checkpoint_conf.h                    |  215 +++
src/conf/domain_conf.h                        |   16 +-
src/datatypes.h                               |   31 +-
src/driver-hypervisor.h                       |   74 +-
src/qemu/qemu_block.h                         |    3 +
src/qemu/qemu_blockjob.h                      |    1 +
src/qemu/qemu_capabilities.h                  |    2 +
src/qemu/qemu_conf.h                          |    2 +
src/qemu/qemu_domain.h                        |   29 +-
src/qemu/qemu_monitor.h                       |   21 +-
src/qemu/qemu_monitor_json.h                  |   19 +-
tools/virsh-checkpoint.h                      |   29 +
tools/virsh-completer.h                       |    4 +
tools/virsh-util.h                            |    3 +
tools/virsh.h                                 |    1 +
docs/Makefile.am                              |    3 +
docs/apibuild.py                              |    2 +
docs/docs.html.in                             |    9 +-
docs/domainstatecapture.html.in               |  314 ++++
docs/format.html.in                           |    1 +
docs/formatcheckpoint.html.in                 |  291 +++
docs/formatsnapshot.html.in                   |   33 +-
docs/index.html.in                            |    3 +-
docs/schemas/domainbackup.rng                 |  185 ++
docs/schemas/domaincheckpoint.rng             |   94 +
examples/object-events/event-test.c           |    3 +
libvirt.spec.in                               |    3 +
mingw-libvirt.spec.in                         |    6 +
po/POTFILES                                   |    3 +
src/Makefile.am                               |    2 +
src/access/viraccessperm.c                    |    5 +-
src/conf/Makefile.inc.am                      |    2 +
src/conf/checkpoint_conf.c                    | 1533 ++++++++++++++++
src/conf/domain_conf.c                        |   52 +-
src/conf/snapshot_conf.c                      |    4 +-
src/datatypes.c                               |   62 +-
src/libvirt-domain-checkpoint.c               |  925 ++++++++++
src/libvirt-domain-snapshot.c                 |    7 +-
src/libvirt-domain.c                          |    8 +-
src/libvirt_private.syms                      |   30 +
src/libvirt_public.syms                       |   23 +
src/qemu/qemu_block.c                         |   12 +
src/qemu/qemu_capabilities.c                  |    4 +
src/qemu/qemu_conf.c                          |    5 +
src/qemu/qemu_domain.c                        |  217 ++-
src/qemu/qemu_driver.c                        | 1552 +++++++++++++++-
src/qemu/qemu_migration.c                     |    2 +-
src/qemu/qemu_monitor.c                       |   65 +-
src/qemu/qemu_monitor_json.c                  |  197 +-
src/qemu/qemu_process.c                       |    7 +
src/remote/remote_daemon_dispatch.c           |   22 +-
src/remote/remote_driver.c                    |   33 +-
src/remote/remote_protocol.x                  |  238 ++-
src/remote_protocol-structs                   |  129 ++
src/rpc/gendispatch.pl                        |   32 +-
tests/Makefile.am                             |   15 +-
tests/domainbackupxml2xmlin/backup-pull.xml   |    9 +
tests/domainbackupxml2xmlin/backup-push.xml   |    9 +
tests/domainbackupxml2xmlin/empty.xml         |    1 +
tests/domainbackupxml2xmlout/backup-pull.xml  |    9 +
tests/domainbackupxml2xmlout/backup-push.xml  |    9 +
tests/domainbackupxml2xmlout/empty.xml        |    7 +
tests/domaincheckpointxml2xmlin/empty.xml     |    1 +
tests/domaincheckpointxml2xmlin/sample.xml    |    7 +
tests/domaincheckpointxml2xmlout/empty.xml    |   10 +
tests/domaincheckpointxml2xmlout/sample.xml   |   16 +
tests/domaincheckpointxml2xmltest.c           |  231 +++
.../caps_4.0.0.riscv32.xml                    |    2 +
.../caps_4.0.0.riscv64.xml                    |    2 +
tests/qemumonitorjsontest.c                   |    2 +-
tests/virschematest.c                         |    4 +
tools/Makefile.am                             |    3 +-
tools/virsh-checkpoint.c                      | 1578 +++++++++++++++++
tools/virsh-completer.c                       |   52 +-
tools/virsh-domain.c                          |    8 +-
tools/virsh-snapshot.c                        |    2 +-
tools/virsh-util.c                            |   11 +
tools/virsh.c                                 |    2 +
tools/virsh.pod                               |   14 +-
86 files changed, 8674 insertions(+), 123 deletions(-)
create mode 100644 include/libvirt/libvirt-domain-checkpoint.h
create mode 100644 src/conf/checkpoint_conf.h
create mode 100644 tools/virsh-checkpoint.h
create mode 100644 docs/domainstatecapture.html.in
create mode 100644 docs/formatcheckpoint.html.in
create mode 100644 docs/schemas/domainbackup.rng
create mode 100644 docs/schemas/domaincheckpoint.rng
create mode 100644 src/conf/checkpoint_conf.c
create mode 100644 src/libvirt-domain-checkpoint.c
create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlin/empty.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlout/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlin/sample.xml
create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlout/sample.xml
create mode 100644 tests/domaincheckpointxml2xmltest.c
create mode 100644 tools/virsh-checkpoint.c

[libvirt] [PATCH v4 00/20] Incremental Backup API additions

Posted by Eric Blake 2 weeks ago
The following is the latest version of my API proposal for
incremental backups, and follows along from the demo I presented
as part of my KVM Forum 2018 talk:
https://kvmforum2018.sched.com/event/FzuB/facilitating-incremental-backup-eric-blake-red-hat

The patches are also available via https://repo.or.cz/libvirt/ericb.git
at the tag backup-v4.

Integration with external snapshots is still not included in these
patches, although I have been playing more with that locally, and
I still haven't gotten a working demonstration of a push-mode
incremental backup.

At this point, I'm posting mainly because there have been enough
changes in qemu (the backup APIs are now stable! and in qemu.git
in time for 4.0) and libvirt (several releases and code cleanups
in between, such that the v3 no longer applies easily), while I
continue to work locally on more features, addressing the review
comments I got on v3, and remove the 'wip' tag on several of the
later patches.

Among other things, I still haven't determined how we want to
integrate checkpoints with external snapshots; we could either have:

virDomainSnapshotCreateXML("<domainsnapshot>...") # existing
virDomainBackupBegin("<domainbackup>...", "<domaincheckpoint>...") # this series
virDomainSnapshotCheckpointCreateXML("<domainsnapshot>...", "<domaincheckpoint>...") # new

or to make checkpoint creation part of the snapshot and backup XML, as in:

virDomainSnapshotCreateXML("<domainsnapshot><domaincheckpoint>...</domainsnapshot>") # extension of existing
virDomainBackupBegin("<domainbackup><domaincheckpoint>...</domainbackup>") # tweak to this series

I'm also planning to add an API to query which job ids are currently
running (right now, the code only supports one job at a time, and
always calls it job 1, but that is not future-friendly) and an update
to the redefine XML command to make it easier to redefine multiple
checkpoints in one API call.

Since v3:

001/20:[----] [--] 'snapshots: Avoid term 'checkpoint' for full system snapshot'
002/20:[----] [--] 'domain_conf: Expose virDomainStorageNetworkParseHost'
003/20:[----] [--] 'qemu: Allow optional export name during NBD export'
004/20:[----] [--] 'backup: Document nuances between different state capture APIs'
005/20:[0032] [FC] 'backup: Introduce virDomainCheckpointPtr'
006/20:[----] [--] 'backup: Document new XML for backups'
007/20:[0013] [FC] 'backup: Introduce virDomainCheckpoint APIs'
008/20:[0017] [FC] 'backup: Introduce virDomainBackup APIs'
009/20:[----] [--] 'backup: Add new domain:checkpoint access control'
010/20:[0104] [FC] 'backup: Implement backup APIs for remote driver'
011/20:[0034] [FC] 'wip: backup: Parse and output checkpoint XML'
012/20:[0013] [FC] 'wip: backup: Parse and output backup XML'
013/20:[0006] [FC] 'backup: Implement virsh support for checkpoints'
014/20:[----] [--] 'wip: backup: virsh support for backup'
015/20:[0126] [FC] 'backup: Add new qemu monitor interactions'
016/20:[0072] [FC] 'backup: qemu: Implement metadata tracking for checkpoint APIs'
017/20:[0069] [FC] 'wip: backup: Wire up qemu checkpoint commands over QMP'
018/20:[0018] [FC] 'wip: backup: qemu: Implement framework for backup job APIs'
019/20:[0045] [FC] 'backup: Wire up qemu full pull backup commands over QMP'
020/20:[0051] [FC] 'backup: implement qemu incremental pull backup'

Eric Blake (20):
  snapshots: Avoid term 'checkpoint' for full system snapshot
  domain_conf: Expose virDomainStorageNetworkParseHost
  qemu: Allow optional export name during NBD export
  backup: Document nuances between different state capture APIs
  backup: Introduce virDomainCheckpointPtr
  backup: Document new XML for backups
  backup: Introduce virDomainCheckpoint APIs
  backup: Introduce virDomainBackup APIs
  backup: Add new domain:checkpoint access control
  backup: Implement backup APIs for remote driver
  wip: backup: Parse and output checkpoint XML
  wip: backup: Parse and output backup XML
  backup: Implement virsh support for checkpoints
  wip: backup: virsh support for backup
  backup: Add new qemu monitor interactions
  backup: qemu: Implement metadata tracking for checkpoint APIs
  wip: backup: Wire up qemu checkpoint commands over QMP
  wip: backup: qemu: Implement framework for backup job APIs
  backup: Wire up qemu full pull backup commands over QMP
  backup: implement qemu incremental pull backup

 include/libvirt/virterror.h                   |    7 +-
 src/util/virerror.c                           |   12 +-
 include/libvirt/libvirt-domain-checkpoint.h   |  176 ++
 include/libvirt/libvirt-domain-snapshot.h     |    2 +-
 include/libvirt/libvirt-domain.h              |   17 +-
 include/libvirt/libvirt.h                     |    3 +-
 src/access/viraccessperm.h                    |    8 +-
 src/conf/checkpoint_conf.h                    |  215 +++
 src/conf/domain_conf.h                        |   16 +-
 src/datatypes.h                               |   31 +-
 src/driver-hypervisor.h                       |   74 +-
 src/qemu/qemu_block.h                         |    3 +
 src/qemu/qemu_blockjob.h                      |    1 +
 src/qemu/qemu_capabilities.h                  |    2 +
 src/qemu/qemu_conf.h                          |    2 +
 src/qemu/qemu_domain.h                        |   29 +-
 src/qemu/qemu_monitor.h                       |   21 +-
 src/qemu/qemu_monitor_json.h                  |   19 +-
 tools/virsh-checkpoint.h                      |   29 +
 tools/virsh-completer.h                       |    4 +
 tools/virsh-util.h                            |    3 +
 tools/virsh.h                                 |    1 +
 docs/Makefile.am                              |    3 +
 docs/apibuild.py                              |    2 +
 docs/docs.html.in                             |    9 +-
 docs/domainstatecapture.html.in               |  314 ++++
 docs/format.html.in                           |    1 +
 docs/formatcheckpoint.html.in                 |  291 +++
 docs/formatsnapshot.html.in                   |   33 +-
 docs/index.html.in                            |    3 +-
 docs/schemas/domainbackup.rng                 |  185 ++
 docs/schemas/domaincheckpoint.rng             |   94 +
 examples/object-events/event-test.c           |    3 +
 libvirt.spec.in                               |    3 +
 mingw-libvirt.spec.in                         |    6 +
 po/POTFILES                                   |    3 +
 src/Makefile.am                               |    2 +
 src/access/viraccessperm.c                    |    5 +-
 src/conf/Makefile.inc.am                      |    2 +
 src/conf/checkpoint_conf.c                    | 1533 ++++++++++++++++
 src/conf/domain_conf.c                        |   52 +-
 src/conf/snapshot_conf.c                      |    4 +-
 src/datatypes.c                               |   62 +-
 src/libvirt-domain-checkpoint.c               |  925 ++++++++++
 src/libvirt-domain-snapshot.c                 |    7 +-
 src/libvirt-domain.c                          |    8 +-
 src/libvirt_private.syms                      |   30 +
 src/libvirt_public.syms                       |   23 +
 src/qemu/qemu_block.c                         |   12 +
 src/qemu/qemu_capabilities.c                  |    4 +
 src/qemu/qemu_conf.c                          |    5 +
 src/qemu/qemu_domain.c                        |  217 ++-
 src/qemu/qemu_driver.c                        | 1552 +++++++++++++++-
 src/qemu/qemu_migration.c                     |    2 +-
 src/qemu/qemu_monitor.c                       |   65 +-
 src/qemu/qemu_monitor_json.c                  |  197 +-
 src/qemu/qemu_process.c                       |    7 +
 src/remote/remote_daemon_dispatch.c           |   22 +-
 src/remote/remote_driver.c                    |   33 +-
 src/remote/remote_protocol.x                  |  238 ++-
 src/remote_protocol-structs                   |  129 ++
 src/rpc/gendispatch.pl                        |   32 +-
 tests/Makefile.am                             |   15 +-
 tests/domainbackupxml2xmlin/backup-pull.xml   |    9 +
 tests/domainbackupxml2xmlin/backup-push.xml   |    9 +
 tests/domainbackupxml2xmlin/empty.xml         |    1 +
 tests/domainbackupxml2xmlout/backup-pull.xml  |    9 +
 tests/domainbackupxml2xmlout/backup-push.xml  |    9 +
 tests/domainbackupxml2xmlout/empty.xml        |    7 +
 tests/domaincheckpointxml2xmlin/empty.xml     |    1 +
 tests/domaincheckpointxml2xmlin/sample.xml    |    7 +
 tests/domaincheckpointxml2xmlout/empty.xml    |   10 +
 tests/domaincheckpointxml2xmlout/sample.xml   |   16 +
 tests/domaincheckpointxml2xmltest.c           |  231 +++
 .../caps_4.0.0.riscv32.xml                    |    2 +
 .../caps_4.0.0.riscv64.xml                    |    2 +
 tests/qemumonitorjsontest.c                   |    2 +-
 tests/virschematest.c                         |    4 +
 tools/Makefile.am                             |    3 +-
 tools/virsh-checkpoint.c                      | 1578 +++++++++++++++++
 tools/virsh-completer.c                       |   52 +-
 tools/virsh-domain.c                          |    8 +-
 tools/virsh-snapshot.c                        |    2 +-
 tools/virsh-util.c                            |   11 +
 tools/virsh.c                                 |    2 +
 tools/virsh.pod                               |   14 +-
 86 files changed, 8674 insertions(+), 123 deletions(-)
 create mode 100644 include/libvirt/libvirt-domain-checkpoint.h
 create mode 100644 src/conf/checkpoint_conf.h
 create mode 100644 tools/virsh-checkpoint.h
 create mode 100644 docs/domainstatecapture.html.in
 create mode 100644 docs/formatcheckpoint.html.in
 create mode 100644 docs/schemas/domainbackup.rng
 create mode 100644 docs/schemas/domaincheckpoint.rng
 create mode 100644 src/conf/checkpoint_conf.c
 create mode 100644 src/libvirt-domain-checkpoint.c
 create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlin/empty.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlout/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlin/sample.xml
 create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlout/sample.xml
 create mode 100644 tests/domaincheckpointxml2xmltest.c
 create mode 100644 tools/virsh-checkpoint.c

-- 
2.20.1

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v4 00/20] Incremental Backup API additions

Posted by Kashyap Chamarthy 3 days ago
On Wed, Feb 06, 2019 at 01:17:58PM -0600, Eric Blake wrote:
> The following is the latest version of my API proposal for
> incremental backups, and follows along from the demo I presented
> as part of my KVM Forum 2018 talk:
> https://kvmforum2018.sched.com/event/FzuB/facilitating-incremental-backup-eric-blake-red-hat
> 
> The patches are also available via https://repo.or.cz/libvirt/ericb.git
> at the tag backup-v4.
> 
> Integration with external snapshots is still not included in these
> patches, although I have been playing more with that locally, and
> I still haven't gotten a working demonstration of a push-mode
> incremental backup.
> 
> At this point, I'm posting mainly because there have been enough
> changes in qemu (the backup APIs are now stable! and in qemu.git
> in time for 4.0) and libvirt (several releases and code cleanups
> in between, such that the v3 no longer applies easily), while I
> continue to work locally on more features, addressing the review
> comments I got on v3, and remove the 'wip' tag on several of the
> later patches.
> 
> Among other things, I still haven't determined how we want to
> integrate checkpoints with external snapshots; we could either have:
> 
> virDomainSnapshotCreateXML("<domainsnapshot>...") # existing
> virDomainBackupBegin("<domainbackup>...", "<domaincheckpoint>...") # this series
> virDomainSnapshotCheckpointCreateXML("<domainsnapshot>...", "<domaincheckpoint>...") # new

A slightly related question: when these new APIs (thanks for working on
them!) are merged, am I right in assuming that they should be able to
"replace" the existing (and provide additional):

    virDomainBlockRebase(); and 
    virDomainBlockCopy()

... _provided_ that an application is adjusted to using libvirt that is
new enough to drive QEMU's '-blockdev', QMP `blockdev-add` et al?

Or is that (the new APIs being backward-compatible with blockRebase()
and blockCopy()) an explicit non-goal?

I'm only this out of curiosity.

> or to make checkpoint creation part of the snapshot and backup XML, as in:
> 
> virDomainSnapshotCreateXML("<domainsnapshot><domaincheckpoint>...</domainsnapshot>") # extension of existing
> virDomainBackupBegin("<domainbackup><domaincheckpoint>...</domainbackup>") # tweak to this series
> 
> I'm also planning to add an API to query which job ids are currently
> running (right now, the code only supports one job at a time, and
> always calls it job 1, but that is not future-friendly) and an update
> to the redefine XML command to make it easier to redefine multiple
> checkpoints in one API call.

[...]


-- 
/kashyap

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list