From nobody Wed Nov 5 14:39:41 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1535663900361317.88269786218734; Thu, 30 Aug 2018 14:18:20 -0700 (PDT) Received: from localhost ([::1]:50913 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fvUKV-00075C-4T for importer@patchew.org; Thu, 30 Aug 2018 17:18:15 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51631) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fvUIc-0004tt-8Z for qemu-devel@nongnu.org; Thu, 30 Aug 2018 17:16:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fvUIa-0003Kj-2K for qemu-devel@nongnu.org; Thu, 30 Aug 2018 17:16:18 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:58688 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fvUIS-00036E-2P; Thu, 30 Aug 2018 17:16:08 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AA06787A70; Thu, 30 Aug 2018 21:16:07 +0000 (UTC) Received: from probe.bos.redhat.com (dhcp-17-156.bos.redhat.com [10.18.17.156]) by smtp.corp.redhat.com (Postfix) with ESMTP id 69CFD10EE79F; Thu, 30 Aug 2018 21:16:07 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Thu, 30 Aug 2018 17:16:04 -0400 Message-Id: <20180830211605.13683-2-jsnow@redhat.com> In-Reply-To: <20180830211605.13683-1-jsnow@redhat.com> References: <20180830211605.13683-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Thu, 30 Aug 2018 21:16:07 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Thu, 30 Aug 2018 21:16:07 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'jsnow@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 1/2] blockdev-backup: add bitmap argument X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , John Snow , Markus Armbruster , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It is only an oversight that we don't allow incremental backup with blockdev-backup. Add the bitmap argument which enables this. Signed-off-by: John Snow --- blockdev.c | 16 +++++++++++++++- qapi/block-core.json | 7 ++++++- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/blockdev.c b/blockdev.c index 72f5347df5..b9f19afacc 100644 --- a/blockdev.c +++ b/blockdev.c @@ -3491,6 +3491,7 @@ BlockJob *do_blockdev_backup(BlockdevBackup *backup, = JobTxn *txn, BlockDriverState *bs; BlockDriverState *target_bs; Error *local_err =3D NULL; + BdrvDirtyBitmap *bmap =3D NULL; AioContext *aio_context; BlockJob *job =3D NULL; int job_flags =3D JOB_DEFAULT; @@ -3541,6 +3542,19 @@ BlockJob *do_blockdev_backup(BlockdevBackup *backup,= JobTxn *txn, goto out; } } + if (backup->has_bitmap) { + bmap =3D bdrv_find_dirty_bitmap(bs, backup->bitmap); + if (!bmap) { + error_setg(errp, "Bitmap '%s' could not be found", backup->bit= map); + goto out; + } + if (bdrv_dirty_bitmap_qmp_locked(bmap)) { + error_setg(errp, + "Bitmap '%s' is currently locked and cannot be used= for " + "backup", backup->bitmap); + goto out; + } + } if (!backup->auto_finalize) { job_flags |=3D JOB_MANUAL_FINALIZE; } @@ -3548,7 +3562,7 @@ BlockJob *do_blockdev_backup(BlockdevBackup *backup, = JobTxn *txn, job_flags |=3D JOB_MANUAL_DISMISS; } job =3D backup_job_create(backup->job_id, bs, target_bs, backup->speed, - backup->sync, NULL, backup->compress, + backup->sync, bmap, backup->compress, backup->on_source_error, backup->on_target_err= or, job_flags, NULL, NULL, txn, &local_err); if (local_err !=3D NULL) { diff --git a/qapi/block-core.json b/qapi/block-core.json index 4c7a37afdc..d3ea281039 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -1315,6 +1315,10 @@ # @speed: the maximum speed, in bytes per second. The default is 0, # for unlimited. # +# @bitmap: the name of dirty bitmap if sync is "incremental". +# Must be present if sync is "incremental", must NOT be present +# otherwise. (Since 3.1) +# # @compress: true to compress data, if the target format supports it. # (default: false) (since 2.8) # @@ -1346,7 +1350,8 @@ ## { 'struct': 'BlockdevBackup', 'data': { '*job-id': 'str', 'device': 'str', 'target': 'str', - 'sync': 'MirrorSyncMode', '*speed': 'int', '*compress': 'bool', + 'sync': 'MirrorSyncMode', '*speed': 'int', + '*bitmap': 'str', '*compress': 'bool', '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } } --=20 2.14.4 From nobody Wed Nov 5 14:39:41 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1535663902072424.4569065569576; Thu, 30 Aug 2018 14:18:22 -0700 (PDT) Received: from localhost ([::1]:50914 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fvUKV-00075Z-0v for importer@patchew.org; Thu, 30 Aug 2018 17:18:15 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51630) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fvUIc-0004tr-8B for qemu-devel@nongnu.org; Thu, 30 Aug 2018 17:16:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fvUIa-0003KX-0R for qemu-devel@nongnu.org; Thu, 30 Aug 2018 17:16:18 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:58691 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fvUIS-00036s-AF; Thu, 30 Aug 2018 17:16:08 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id ED70587A7C; Thu, 30 Aug 2018 21:16:07 +0000 (UTC) Received: from probe.bos.redhat.com (dhcp-17-156.bos.redhat.com [10.18.17.156]) by smtp.corp.redhat.com (Postfix) with ESMTP id AC204107CCEE; Thu, 30 Aug 2018 21:16:07 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Thu, 30 Aug 2018 17:16:05 -0400 Message-Id: <20180830211605.13683-3-jsnow@redhat.com> In-Reply-To: <20180830211605.13683-1-jsnow@redhat.com> References: <20180830211605.13683-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Thu, 30 Aug 2018 21:16:07 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Thu, 30 Aug 2018 21:16:07 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'jsnow@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 2/2] HACK: test blockdev-backup instead of drive-backup X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , John Snow , Markus Armbruster , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is just a real chainsaw job on 124 to prove that we can indeed use blockdev-backup interchangeably with drive-backup for incremental backups. A nicer test will follow once I refactor this a bit to look a little less like the Texas Chainsaw Massacre. Signed-off-by: John Snow --- tests/qemu-iotests/124 | 99 ++++++++++++++++++++++++++++++++++------------= ---- 1 file changed, 67 insertions(+), 32 deletions(-) diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124 index 3ea4ac53f5..b5a1f5e248 100755 --- a/tests/qemu-iotests/124 +++ b/tests/qemu-iotests/124 @@ -49,7 +49,7 @@ def transaction_bitmap_clear(node, name, **kwargs): =20 =20 def transaction_drive_backup(device, target, **kwargs): - return transaction_action('drive-backup', job_id=3Ddevice, device=3Dde= vice, + return transaction_action('blockdev-backup', job_id=3Ddevice, device= =3Ddevice, target=3Dtarget, **kwargs) =20 =20 @@ -146,10 +146,22 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): =20 =20 def do_qmp_backup(self, error=3D'Input/output error', **kwargs): - res =3D self.vm.qmp('drive-backup', **kwargs) + res =3D self.vm.qmp('blockdev-backup', **kwargs) self.assert_qmp(res, 'return', {}) return self.wait_qmp_backup(kwargs['device'], error) =20 + def _FIXME_new_target(self, name, fmt, filename): + result =3D self.vm.qmp('blockdev-add', + node_name=3Dname, + driver=3Dfmt, + file=3D{"driver": "file", + "filename": filename}) + self.assert_qmp(result, 'return', {}) + + def _FIXME_del_target(self, name): + result =3D self.vm.qmp('blockdev-del', + node_name=3Dname) + self.assert_qmp(result, 'return', {}) =20 def ignore_job_status_change_events(self): while True: @@ -185,11 +197,13 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): def create_anchor_backup(self, drive=3DNone): if drive is None: drive =3D self.drives[-1] + self.img_create(drive['backup'], drive['fmt']) + self._FIXME_new_target('target', drive['fmt'], drive['backup']) res =3D self.do_qmp_backup(job_id=3Ddrive['id'], device=3Ddrive['id'], sync=3D'full', - format=3Ddrive['fmt'], target=3Ddrive['ba= ckup']) + target=3D'target') + self._FIXME_del_target('target') self.assertTrue(res) - self.files.append(drive['backup']) return drive['backup'] =20 =20 @@ -197,9 +211,12 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): if bitmap is None: bitmap =3D self.bitmaps[-1] _, reference =3D bitmap.last_target() + self.img_create(reference, bitmap.drive['fmt']) + self._FIXME_new_target('target', bitmap.drive['fmt'], reference) res =3D self.do_qmp_backup(job_id=3Dbitmap.drive['id'], device=3Dbitmap.drive['id'], sync=3D'full= ', - format=3Dbitmap.drive['fmt'], target=3Dre= ference) + target=3D'target') + self._FIXME_del_target('target') self.assertTrue(res) =20 =20 @@ -223,19 +240,19 @@ class TestIncrementalBackupBase(iotests.QMPTestCase): return target =20 =20 - def create_incremental(self, bitmap=3DNone, parent=3DNone, - parentFormat=3DNone, validate=3DTrue): + def create_incremental(self, bitmap=3DNone, parent=3DNone, validate=3D= True): if bitmap is None: bitmap =3D self.bitmaps[-1] if parent is None: parent, _ =3D bitmap.last_target() =20 target =3D self.prepare_backup(bitmap, parent) + self._FIXME_new_target('target', bitmap.drive['fmt'], target) res =3D self.do_qmp_backup(job_id=3Dbitmap.drive['id'], device=3Dbitmap.drive['id'], sync=3D'incremental', bitmap=3Dbitmap.nam= e, - format=3Dbitmap.drive['fmt'], target=3Dta= rget, - mode=3D'existing') + target=3D'target') + self._FIXME_del_target('target') if not res: bitmap.del_target(); self.assertFalse(validate) @@ -331,10 +348,11 @@ class TestIncrementalBackup(TestIncrementalBackupBase= ): =20 # Create a cluster_size=3D128k full backup / "anchor" backup self.img_create(drive0['backup'], cluster_size=3D'128k') + self.img_create(drive0['backup'], drive0['fmt']) + self._FIXME_new_target('target', drive0['fmt'], drive0['backup']) self.assertTrue(self.do_qmp_backup(device=3Ddrive0['id'], sync=3D'= full', - format=3Ddrive0['fmt'], - target=3Ddrive0['backup'], - mode=3D'existing')) + target=3D'target')) + self._FIXME_del_target('target') =20 # Create bitmap and dirty it with some new writes. # overwrite [32736, 32799] which will dirty bitmap clusters at @@ -356,12 +374,12 @@ class TestIncrementalBackup(TestIncrementalBackupBase= ): self.img_create(target, bitmap0.drive['fmt'], cluster_size=3D'128k= ') =20 # Perform Incremental Backup + self._FIXME_new_target('target', bitmap0.drive['fmt'], target) self.assertTrue(self.do_qmp_backup(device=3Dbitmap0.drive['id'], sync=3D'incremental', bitmap=3Dbitmap0.name, - format=3Dbitmap0.drive['fmt'], - target=3Dtarget, - mode=3D'existing')) + target=3D'target')) + self._FIXME_del_target('target') self.make_reference_backup(bitmap0) =20 # Add the backing file, then compare and exit. @@ -388,15 +406,18 @@ class TestIncrementalBackup(TestIncrementalBackupBase= ): ('0x64', '32736k', '64k'))) bitmap1 =3D self.add_bitmap('bitmap1', drive0) =20 + # Create and mount image + self.img_create(drive0['backup'], drive0['fmt']) + self._FIXME_new_target('target', drive0['fmt'], drive0['backup']) + result =3D self.vm.qmp('transaction', actions=3D[ transaction_bitmap_clear(bitmap0.drive['id'], bitmap0.name), transaction_bitmap_clear(bitmap1.drive['id'], bitmap1.name), - transaction_drive_backup(drive0['id'], drive0['backup'], - sync=3D'full', format=3Ddrive0['fmt']) + transaction_drive_backup(drive0['id'], 'target', sync=3D'full') ]) self.assert_qmp(result, 'return', {}) self.wait_until_completed(drive0['id']) - self.files.append(drive0['backup']) + self._FIXME_del_target('target') =20 self.hmp_io_writes(drive0['id'], (('0x9a', 0, 512), ('0x55', '8M', '352k'), @@ -464,16 +485,18 @@ class TestIncrementalBackup(TestIncrementalBackupBase= ): target0 =3D self.prepare_backup(dr0bm0) target1 =3D self.prepare_backup(dr1bm0) =20 + # Mount targets + self._FIXME_new_target('target0', drive0['fmt'], target0) + self._FIXME_new_target('target1', drive1['fmt'], target1) + # Ask for a new incremental backup per-each drive, # expecting drive1's backup to fail. In the 'race' test, # we expect drive1 to attempt to cancel the empty drive0 job. transaction =3D [ - transaction_drive_backup(drive0['id'], target0, sync=3D'increm= ental', - format=3Ddrive0['fmt'], mode=3D'exist= ing', - bitmap=3Ddr0bm0.name), - transaction_drive_backup(drive1['id'], target1, sync=3D'increm= ental', - format=3Ddrive1['fmt'], mode=3D'exist= ing', - bitmap=3Ddr1bm0.name) + transaction_drive_backup(drive0['id'], 'target0', + sync=3D'incremental', bitmap=3Ddr0bm0= .name), + transaction_drive_backup(drive1['id'], 'target1', + sync=3D'incremental', bitmap=3Ddr1bm0= .name) ] result =3D self.vm.qmp('transaction', actions=3Dtransaction, properties=3D{'completion-mode': 'grouped'} ) @@ -490,6 +513,10 @@ class TestIncrementalBackup(TestIncrementalBackupBase): self.assertFalse(self.vm.get_qmp_events(wait=3DFalse)) self.assert_no_active_block_jobs() =20 + # Unmount targets + self._FIXME_del_target('target0') + self._FIXME_del_target('target1') + # Delete drive0's successful target and eliminate our record of the # unsuccessful drive1 target. dr0bm0.del_target() @@ -499,9 +526,11 @@ class TestIncrementalBackup(TestIncrementalBackupBase): self.vm.shutdown() return =20 - # Re-run the same transaction: + # Re-run the same transaction; create and mount images target0 =3D self.prepare_backup(dr0bm0) target1 =3D self.prepare_backup(dr1bm0) + self._FIXME_new_target('target0', drive0['fmt'], target0) + self._FIXME_new_target('target1', drive1['fmt'], target1) =20 # Re-run the exact same transaction. result =3D self.vm.qmp('transaction', actions=3Dtransaction, @@ -516,6 +545,9 @@ class TestIncrementalBackup(TestIncrementalBackupBase): self.assertFalse(self.vm.get_qmp_events(wait=3DFalse)) self.assert_no_active_block_jobs() =20 + self._FIXME_del_target('target0') + self._FIXME_del_target('target1') + # And the images should of course validate. self.vm.shutdown() self.check_backups() @@ -544,19 +576,22 @@ class TestIncrementalBackup(TestIncrementalBackupBase= ): =20 def test_sync_dirty_bitmap_missing(self): self.assert_no_active_block_jobs() - self.files.append(self.err_img) - result =3D self.vm.qmp('drive-backup', device=3Dself.drives[0]['id= '], - sync=3D'incremental', format=3Dself.drives[0]= ['fmt'], - target=3Dself.err_img) + self.img_create(self.err_img, iotests.imgfmt) + self._FIXME_new_target('err0', iotests.imgfmt, self.err_img) + result =3D self.vm.qmp('blockdev-backup', device=3Dself.drives[0][= 'id'], + sync=3D'incremental', target=3D'err0') + self._FIXME_del_target('err0') self.assert_qmp(result, 'error/class', 'GenericError') =20 =20 def test_sync_dirty_bitmap_not_found(self): self.assert_no_active_block_jobs() - self.files.append(self.err_img) - result =3D self.vm.qmp('drive-backup', device=3Dself.drives[0]['id= '], + self.img_create(self.err_img, iotests.imgfmt) + self._FIXME_new_target('err0', iotests.imgfmt, self.err_img) + result =3D self.vm.qmp('blockdev-backup', device=3Dself.drives[0][= 'id'], sync=3D'incremental', bitmap=3D'unknown', - format=3Dself.drives[0]['fmt'], target=3Dself= .err_img) + target=3D'err0') + self._FIXME_del_target('err0') self.assert_qmp(result, 'error/class', 'GenericError') =20 =20 --=20 2.14.4