[libvirt] [PATCH v2 0/9] Incremental Backup API additions

Eric Blake posted 9 patches 5 years, 5 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/20181012051011.1398546-1-eblake@redhat.com
Test syntax-check passed
There is a newer version of this series
include/libvirt/virterror.h                  |   6 +-
src/util/virerror.c                          |  21 +-
include/libvirt/libvirt-domain-checkpoint.h  | 177 ++++
include/libvirt/libvirt-domain-snapshot.h    |   2 +-
include/libvirt/libvirt-domain.h             |  14 +-
include/libvirt/libvirt.h                    |   3 +-
src/access/viraccessperm.h                   |   8 +-
src/conf/domain_conf.h                       |   2 +
src/datatypes.h                              |  31 +-
src/driver-hypervisor.h                      |  74 +-
docs/Makefile.am                             |   3 +
docs/apibuild.py                             |   2 +
docs/docs.html.in                            |   9 +-
docs/domainstatecapture.html.in              | 314 +++++++
docs/format.html.in                          |   1 +
docs/formatcheckpoint.html.in                | 285 ++++++
docs/formatsnapshot.html.in                  |  33 +-
docs/index.html.in                           |   3 +-
docs/schemas/domainbackup.rng                | 185 ++++
docs/schemas/domaincheckpoint.rng            |  94 ++
libvirt.spec.in                              |   3 +
mingw-libvirt.spec.in                        |   6 +
po/POTFILES                                  |   1 +
src/Makefile.am                              |   2 +
src/access/viraccessperm.c                   |   5 +-
src/conf/domain_conf.c                       |  43 +-
src/conf/snapshot_conf.c                     |   4 +-
src/datatypes.c                              |  62 +-
src/libvirt-domain-checkpoint.c              | 936 +++++++++++++++++++
src/libvirt-domain-snapshot.c                |   7 +-
src/libvirt-domain.c                         |   8 +-
src/libvirt_private.syms                     |   3 +
src/libvirt_public.syms                      |  23 +
src/qemu/qemu_driver.c                       |  12 +-
src/remote/remote_daemon_dispatch.c          |  15 +
src/remote/remote_driver.c                   |  33 +-
src/remote/remote_protocol.x                 | 240 ++++-
src/remote_protocol-structs                  | 129 +++
src/rpc/gendispatch.pl                       |  32 +-
tests/Makefile.am                            |   6 +-
tests/domainbackupxml2xmlin/backup-pull.xml  |   9 +
tests/domainbackupxml2xmlin/backup-push.xml  |   9 +
tests/domainbackupxml2xmlin/empty.xml        |   1 +
tests/domainbackupxml2xmlout/backup-pull.xml |   9 +
tests/domainbackupxml2xmlout/backup-push.xml |   9 +
tests/domainbackupxml2xmlout/empty.xml       |   7 +
tests/domaincheckpointxml2xmlin/empty.xml    |   1 +
tests/domaincheckpointxml2xmlin/sample.xml   |   7 +
tests/domaincheckpointxml2xmlout/empty.xml   |  10 +
tests/domaincheckpointxml2xmlout/sample.xml  |  16 +
tests/virschematest.c                        |   4 +
tools/virsh-domain.c                         |   3 +-
tools/virsh-snapshot.c                       |   2 +-
tools/virsh.pod                              |  14 +-
54 files changed, 2846 insertions(+), 92 deletions(-)
create mode 100644 include/libvirt/libvirt-domain-checkpoint.h
create mode 100644 docs/domainstatecapture.html.in
create mode 100644 docs/formatcheckpoint.html.in
create mode 100644 docs/schemas/domainbackup.rng
create mode 100644 docs/schemas/domaincheckpoint.rng
create mode 100644 src/libvirt-domain-checkpoint.c
create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlin/empty.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlout/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlin/sample.xml
create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
create mode 100644 tests/domaincheckpointxml2xmlout/sample.xml
[libvirt] [PATCH v2 0/9] Incremental Backup API additions
Posted by Eric Blake 5 years, 5 months ago
The following is the latest version of my API proposal for
incremental backups.

I have even more work-in-progress patches on top of these:
https://repo.or.cz/libvirt/ericb.git
which I am slowly improving to be more in line with my thread
on the overview of the API usage:
https://www.redhat.com/archives/libvir-list/2018-October/msg00217.html

But I am fairly satisfied that the API as presented is sufficient for
everything I have still been implementing in the qemu driver, and
that even when qemu is slightly tweaked (such as dropping the x-
prefix on various commands, or maybe adding a new command to make
it easier to compute the estimated size of the union of several
bitmaps), those changes will be limited to the src/qemu directory
rather than affecting the API.

Since I will be demonstrating the use of this API at the KVM Forum,
I would really like a decision on whether we can commit the API
into libvirt now, even if we have to wait for the qemu implementation
of the API until qemu stabilizes its interfaces (also, having the
libvirt API in place gives qemu an incentive to drop the x- prefix
sooner rather than later).  That in turn implies that I need code
reviews; John Ferlan started reviewing v1, but mostly focused on
the documentation grammar:
https://www.redhat.com/archives/libvir-list/2018-June/msg01066.html

The following gives an idea of how this series has changed since
June (in part due to rebases, in part due to documentation improvements):

The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/9:[0050] [FC] 'snapshots: Avoid term 'checkpoint' for full system snapshot'
002/9:[down] 'domain_conf: Expose virDomainStorageNetworkParseHost'
003/9:[0342] [FC] 'backup: Document nuances between different state capture APIs'
004/9:[0007] [FC] 'backup: Introduce virDomainCheckpointPtr'
005/9:[0100] [FC] 'backup: Document new XML for backups'
006/9:[0042] [FC] 'backup: Introduce virDomainCheckpoint APIs'
007/9:[0035] [FC] 'backup: Introduce virDomainBackup APIs'
008/9:[----] [--] 'backup: Add new domain:checkpoint access control'
009/9:[0039] [FC] 'backup: Implement backup APIs for remote driver'

Eric Blake (9):
  snapshots: Avoid term 'checkpoint' for full system snapshot
  domain_conf: Expose virDomainStorageNetworkParseHost
  backup: Document nuances between different state capture APIs
  backup: Introduce virDomainCheckpointPtr
  backup: Document new XML for backups
  backup: Introduce virDomainCheckpoint APIs
  backup: Introduce virDomainBackup APIs
  backup: Add new domain:checkpoint access control
  backup: Implement backup APIs for remote driver

 include/libvirt/virterror.h                  |   6 +-
 src/util/virerror.c                          |  21 +-
 include/libvirt/libvirt-domain-checkpoint.h  | 177 ++++
 include/libvirt/libvirt-domain-snapshot.h    |   2 +-
 include/libvirt/libvirt-domain.h             |  14 +-
 include/libvirt/libvirt.h                    |   3 +-
 src/access/viraccessperm.h                   |   8 +-
 src/conf/domain_conf.h                       |   2 +
 src/datatypes.h                              |  31 +-
 src/driver-hypervisor.h                      |  74 +-
 docs/Makefile.am                             |   3 +
 docs/apibuild.py                             |   2 +
 docs/docs.html.in                            |   9 +-
 docs/domainstatecapture.html.in              | 314 +++++++
 docs/format.html.in                          |   1 +
 docs/formatcheckpoint.html.in                | 285 ++++++
 docs/formatsnapshot.html.in                  |  33 +-
 docs/index.html.in                           |   3 +-
 docs/schemas/domainbackup.rng                | 185 ++++
 docs/schemas/domaincheckpoint.rng            |  94 ++
 libvirt.spec.in                              |   3 +
 mingw-libvirt.spec.in                        |   6 +
 po/POTFILES                                  |   1 +
 src/Makefile.am                              |   2 +
 src/access/viraccessperm.c                   |   5 +-
 src/conf/domain_conf.c                       |  43 +-
 src/conf/snapshot_conf.c                     |   4 +-
 src/datatypes.c                              |  62 +-
 src/libvirt-domain-checkpoint.c              | 936 +++++++++++++++++++
 src/libvirt-domain-snapshot.c                |   7 +-
 src/libvirt-domain.c                         |   8 +-
 src/libvirt_private.syms                     |   3 +
 src/libvirt_public.syms                      |  23 +
 src/qemu/qemu_driver.c                       |  12 +-
 src/remote/remote_daemon_dispatch.c          |  15 +
 src/remote/remote_driver.c                   |  33 +-
 src/remote/remote_protocol.x                 | 240 ++++-
 src/remote_protocol-structs                  | 129 +++
 src/rpc/gendispatch.pl                       |  32 +-
 tests/Makefile.am                            |   6 +-
 tests/domainbackupxml2xmlin/backup-pull.xml  |   9 +
 tests/domainbackupxml2xmlin/backup-push.xml  |   9 +
 tests/domainbackupxml2xmlin/empty.xml        |   1 +
 tests/domainbackupxml2xmlout/backup-pull.xml |   9 +
 tests/domainbackupxml2xmlout/backup-push.xml |   9 +
 tests/domainbackupxml2xmlout/empty.xml       |   7 +
 tests/domaincheckpointxml2xmlin/empty.xml    |   1 +
 tests/domaincheckpointxml2xmlin/sample.xml   |   7 +
 tests/domaincheckpointxml2xmlout/empty.xml   |  10 +
 tests/domaincheckpointxml2xmlout/sample.xml  |  16 +
 tests/virschematest.c                        |   4 +
 tools/virsh-domain.c                         |   3 +-
 tools/virsh-snapshot.c                       |   2 +-
 tools/virsh.pod                              |  14 +-
 54 files changed, 2846 insertions(+), 92 deletions(-)
 create mode 100644 include/libvirt/libvirt-domain-checkpoint.h
 create mode 100644 docs/domainstatecapture.html.in
 create mode 100644 docs/formatcheckpoint.html.in
 create mode 100644 docs/schemas/domainbackup.rng
 create mode 100644 docs/schemas/domaincheckpoint.rng
 create mode 100644 src/libvirt-domain-checkpoint.c
 create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlin/empty.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
 create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
 create mode 100644 tests/domainbackupxml2xmlout/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlin/sample.xml
 create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml
 create mode 100644 tests/domaincheckpointxml2xmlout/sample.xml

-- 
2.17.1

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v2 0/9] Incremental Backup API additions
Posted by Jiri Denemark 5 years, 5 months ago
On Fri, Oct 12, 2018 at 00:10:02 -0500, Eric Blake wrote:
> The following is the latest version of my API proposal for
> incremental backups.
> 
> I have even more work-in-progress patches on top of these:
> https://repo.or.cz/libvirt/ericb.git
> which I am slowly improving to be more in line with my thread
> on the overview of the API usage:
> https://www.redhat.com/archives/libvir-list/2018-October/msg00217.html
> 
> But I am fairly satisfied that the API as presented is sufficient for
> everything I have still been implementing in the qemu driver, and
> that even when qemu is slightly tweaked (such as dropping the x-
> prefix on various commands, or maybe adding a new command to make
> it easier to compute the estimated size of the union of several
> bitmaps), those changes will be limited to the src/qemu directory
> rather than affecting the API.
> 
> Since I will be demonstrating the use of this API at the KVM Forum,
> I would really like a decision on whether we can commit the API
> into libvirt now, even if we have to wait for the qemu implementation

No we can't. We did it once in the past and although I don't remember
the exact details I remember it turned out to be a bad idea. Also
pushing an API with no implementation doesn't really make sense since
the API is unusable until it is implemented in some hypervisor driver.

> of the API until qemu stabilizes its interfaces (also, having the
> libvirt API in place gives qemu an incentive to drop the x- prefix
> sooner rather than later).

The presence of public API in libvirt gives no clue to QEMU engineers
that their API provides everything needed and x- prefix should be
dropped. Only working patches implementing the libvirt APIs for QEMU can
provide such feedback to QEMU developers.

Jirka

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v2 0/9] Incremental Backup API additions
Posted by Eric Blake 5 years, 5 months ago
On 10/12/18 11:28 AM, Jiri Denemark wrote:

>> Since I will be demonstrating the use of this API at the KVM Forum,
>> I would really like a decision on whether we can commit the API
>> into libvirt now, even if we have to wait for the qemu implementation
> 
> No we can't. We did it once in the past and although I don't remember
> the exact details I remember it turned out to be a bad idea. Also
> pushing an API with no implementation doesn't really make sense since
> the API is unusable until it is implemented in some hypervisor driver.

You wouldn't, by chance, be thinking of the fact that 
virDomainBlockCopy() landed in 1.2.8, but didn't get implemented until 
1.2.9?  (Yes, that one also has my name suspiciously tied to it.  How 
come I always seem to be implementing APIs that downstream wants sooner 
than upstream is ready to commit to?)

> 
>> of the API until qemu stabilizes its interfaces (also, having the
>> libvirt API in place gives qemu an incentive to drop the x- prefix
>> sooner rather than later).
> 
> The presence of public API in libvirt gives no clue to QEMU engineers
> that their API provides everything needed and x- prefix should be
> dropped. Only working patches implementing the libvirt APIs for QEMU can
> provide such feedback to QEMU developers.

My v3 is demonstration that QEMU is on the right track for providing 
everything I needed for a rudimentary implementation; the qemu 
implementation can be further fine-tuned as it waits for qemu to polish 
its interface.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list