qemu: Incremental backup and TLS handling fixes

[PATCH 00/24] qemu: Incremental backup and TLS handling fixes

Posted by Peter Krempa 3 years, 9 months ago

This series consists of multiple parts fixing the following bugs. Some
of them depend on previous so I'm sending it as one to prevent
conflicts.

- Patches 1 - 11:

  https://bugzilla.redhat.com/show_bug.cgi?id=1602328
  [RFE] Add support for encrypted TLS client keys for disks

- Patch 12:

  https://bugzilla.redhat.com/show_bug.cgi?id=1840053
  [incremental_backup] cannot do FULL backup for a READONLY disk

- Patches 13 - 14:

  https://bugzilla.redhat.com/show_bug.cgi?id=1829829
   [incremental backup] Creating incremental backup that includes a new VM disk that requires full backup is impossible

- Patch 15:

  https://bugzilla.redhat.com/show_bug.cgi?id=1799010
  incremental-backup: RFE: Handle backup bitmaps during virDomainBlockPull

- Patches 16 - 24:

  https://bugzilla.redhat.com/show_bug.cgi?id=1822631
  [incremental backup] RFE: Support TLS for NBD connections for pull mode backup


Peter Krempa (24):
  qemu: domain: Introduce helper for always fetching virStorageSource
    private data
  qemuDomainDiskHasEncryptionSecret: unexport
  qemu.conf: Remove misleading mention of 'migrate_tls'
  qemu: conf: Move 'nbd' and 'vxhs' tls config variables together with
    rest of tls setup
  virQEMUDriverConfigLoadSpecificTLSEntry: Move fetching of
    'chardev_tls' above macro
  virQEMUDriverConfigLoadSpecificTLSEntry: Split up fetching of
    server-only config options
  qemu: domain: Add infrastructure passing in TLS key's decryption key
    via 'secret'
  qemu block: Add internals for handling 'secret' corresponding to TLS
    key
  qemu: conf: Add configuration of TLS key encryption for 'vxhs' and
    'nbd' disks
  qemu: domain: Setup secret for TLS key for nbd/vxhs disks
  tests: qemuxml2argv: Test encrypted TLS key for nbd/vxhs disks
  conf: backup: Don't explicitly forbid backup of read-only disk
  docs: backup: Convert XML documentation to RST
  backup: Allow configuring incremental backup per-disk individually
  qemu: backup: integrate with blockpull
  docs: checkpoint: Convert XML documentation to RST
  conf: checkpoint: Add a flag storing whether disk 'size' is valid
  qemu: checkpoint: Implement VIR_DOMAIN_CHECKPOINT_XML_SIZE
  checkpoint: Mention that VIR_DOMAIN_CHECKPOINT_XML_SIZE is expensive
    and stale
  testCompareBackupXML: Add infrastructure for testing internal fields
  conf: backup: Store 'tlsAlias' and 'tlsSecretAlias' as internals of a
    backup
  qemu: conf: Add configuration of TLS environment for NBD transport of
    pull-backups
  conf: backup: Add 'tls' attribute for 'server' element
  qemu: backup: Setup TLS environment for pull-mode backup jobs

 docs/formatbackup.html.in                     | 191 -----------------
 docs/formatbackup.rst                         | 164 +++++++++++++++
 docs/formatcheckpoint.html.in                 | 198 ------------------
 docs/formatcheckpoint.rst                     | 166 +++++++++++++++
 docs/schemas/domainbackup.rng                 |  25 ++-
 src/conf/backup_conf.c                        | 123 ++++++++++-
 src/conf/backup_conf.h                        |  17 ++
 src/conf/checkpoint_conf.c                    |   2 +-
 src/conf/checkpoint_conf.h                    |   1 +
 src/libvirt-domain-checkpoint.c               |   3 +-
 src/qemu/libvirtd_qemu.aug                    |  19 +-
 src/qemu/qemu.conf                            |  63 +++++-
 src/qemu/qemu_backup.c                        |  80 ++++++-
 src/qemu/qemu_block.c                         |  12 ++
 src/qemu/qemu_block.h                         |   2 +
 src/qemu/qemu_blockjob.c                      |  37 ++++
 src/qemu/qemu_checkpoint.c                    | 143 ++++++++++++-
 src/qemu/qemu_command.c                       |  11 +-
 src/qemu/qemu_conf.c                          |  57 +++--
 src/qemu/qemu_conf.h                          |  19 +-
 src/qemu/qemu_domain.c                        |  66 ++++--
 src/qemu/qemu_domain.h                        |   8 +-
 src/qemu/test_libvirtd_qemu.aug.in            |   5 +
 .../backup-pull-encrypted.xml                 |   2 +-
 .../backup-pull-internal-invalid.xml          |  36 ++++
 tests/domainbackupxml2xmlin/backup-pull.xml   |  12 ++
 .../backup-pull-encrypted.xml                 |   2 +-
 .../backup-pull-internal-invalid.xml          |   1 +
 tests/domainbackupxml2xmlout/backup-pull.xml  |  12 ++
 tests/genericxml2xmltest.c                    |  32 ++-
 tests/qemudomaincheckpointxml2xmltest.c       |   1 +
 tests/qemustatusxml2xmldata/modern-in.xml     |   1 +
 .../disk-network-tlsx509.x86_64-2.12.0.args   |  15 +-
 .../disk-network-tlsx509.x86_64-latest.args   |  18 +-
 tests/qemuxml2argvtest.c                      |   2 +
 35 files changed, 1079 insertions(+), 467 deletions(-)
 delete mode 100644 docs/formatbackup.html.in
 create mode 100644 docs/formatbackup.rst
 delete mode 100644 docs/formatcheckpoint.html.in
 create mode 100644 docs/formatcheckpoint.rst
 create mode 100644 tests/domainbackupxml2xmlin/backup-pull-internal-invalid.xml
 create mode 120000 tests/domainbackupxml2xmlout/backup-pull-internal-invalid.xml

-- 
2.26.2

Re: [PATCH 00/24] qemu: Incremental backup and TLS handling fixes

Posted by Andrea Bolognani 3 years, 9 months ago

On Thu, 2020-07-02 at 16:39 +0200, Peter Krempa wrote:
> This series consists of multiple parts fixing the following bugs. Some
> of them depend on previous so I'm sending it as one to prevent
> conflicts.
> 
> - Patches 1 - 11:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1602328
>   [RFE] Add support for encrypted TLS client keys for disks
> 
> - Patch 12:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1840053
>   [incremental_backup] cannot do FULL backup for a READONLY disk
> 
> - Patches 13 - 14:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1829829
>    [incremental backup] Creating incremental backup that includes a new VM disk that requires full backup is impossible
> 
> - Patch 15:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1799010
>   incremental-backup: RFE: Handle backup bitmaps during virDomainBlockPull
> 
> - Patches 16 - 24:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1822631
>   [incremental backup] RFE: Support TLS for NBD connections for pull mode backup

Can you please include updates to the release notes in your series?
Based on the summary above, it sounds like most of the changes will
be user-visible. Thanks!

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH 00/24] qemu: Incremental backup and TLS handling fixes

Posted by Peter Krempa 3 years, 9 months ago

On Thu, Jul 02, 2020 at 18:13:49 +0200, Andrea Bolognani wrote:
> On Thu, 2020-07-02 at 16:39 +0200, Peter Krempa wrote:
> > This series consists of multiple parts fixing the following bugs. Some
> > of them depend on previous so I'm sending it as one to prevent
> > conflicts.
> > 
> > - Patches 1 - 11:
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1602328
> >   [RFE] Add support for encrypted TLS client keys for disks
> > 
> > - Patch 12:
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1840053
> >   [incremental_backup] cannot do FULL backup for a READONLY disk
> > 
> > - Patches 13 - 14:
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1829829
> >    [incremental backup] Creating incremental backup that includes a new VM disk that requires full backup is impossible
> > 
> > - Patch 15:
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1799010
> >   incremental-backup: RFE: Handle backup bitmaps during virDomainBlockPull
> > 
> > - Patches 16 - 24:
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1822631
> >   [incremental backup] RFE: Support TLS for NBD connections for pull mode backup
> 
> Can you please include updates to the release notes in your series?
> Based on the summary above, it sounds like most of the changes will
> be user-visible. Thanks!

Incremental backup is still not enabled so I usually don't include
anything related to it in the news as users can't really use it without
hacking around.

I'll definitely mention the encrypted keys support for NBD.

Also please note that I update news with my changes regularly when I
think it's worth mentioning so I'd prefer to be unsubscribed from these
notifications.

Re: [PATCH 00/24] qemu: Incremental backup and TLS handling fixes

Posted by Andrea Bolognani 3 years, 9 months ago

On Tue, 2020-07-07 at 09:47 +0200, Peter Krempa wrote:
> On Thu, Jul 02, 2020 at 18:13:49 +0200, Andrea Bolognani wrote:
> > Can you please include updates to the release notes in your series?
> > Based on the summary above, it sounds like most of the changes will
> > be user-visible. Thanks!
> 
> Incremental backup is still not enabled so I usually don't include
> anything related to it in the news as users can't really use it without
> hacking around.

Fair enough.

> I'll definitely mention the encrypted keys support for NBD.
> 
> Also please note that I update news with my changes regularly when I
> think it's worth mentioning so I'd prefer to be unsubscribed from these
> notifications.

I can do that, but any reason not to update them at the same time as
you're introducing the corresponding code changes? If you're going to
do the work anyway, to me it makes sense to do all it one chunk and
not have to go back to it later. Makes it harder to accidentally lose
track and forget to do that, too.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH 00/24] qemu: Incremental backup and TLS handling fixes

Posted by Ján Tomko 3 years, 9 months ago

On a Tuesday in 2020, Andrea Bolognani wrote:
>On Tue, 2020-07-07 at 09:47 +0200, Peter Krempa wrote:
>> On Thu, Jul 02, 2020 at 18:13:49 +0200, Andrea Bolognani wrote:
>> > Can you please include updates to the release notes in your series?
>> > Based on the summary above, it sounds like most of the changes will
>> > be user-visible. Thanks!
>>
>> Incremental backup is still not enabled so I usually don't include
>> anything related to it in the news as users can't really use it without
>> hacking around.
>
>Fair enough.
>
>> I'll definitely mention the encrypted keys support for NBD.
>>
>> Also please note that I update news with my changes regularly when I
>> think it's worth mentioning so I'd prefer to be unsubscribed from these
>> notifications.
>
>I can do that, but any reason not to update them at the same time as
>you're introducing the corresponding code changes?

Conflict resolution. Though it should be possible to automate that.

Jano

>If you're going to
>do the work anyway, to me it makes sense to do all it one chunk and
>not have to go back to it later. Makes it harder to accidentally lose
>track and forget to do that, too.
>
>-- 
>Andrea Bolognani / Red Hat / Virtualization
>