From nobody Fri May 3 13:49:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1564672476; cv=none; d=zoho.com; s=zohoarc; b=Xw5G5LVJ3srexqH+lkZJ3NcQTx4kGZ0XgJeNT4rtMnTyD9THR+GapO7gtZqmHmPSGdihTrllV1HzUCaYMlC6qtteV/HWo+Sl6al3a+FfsFnDFXaWl7n/QA/yuXa366EK3mzObfV5XcZlicXmJID6amnysvUu656h1lOxWxqPfuw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564672476; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=sdX5zC2GW1p6l7vxmKOoQalkDFqHsvdU/dmWtV+uA8o=; b=OL3M7uuxt7nkwRFyhb+tth2nn+Pi6B3etpU6nexFsmpiAsdpAjIEcWa0Iaq9f+QimMoQBqlTOpjKIaRVO15/aSYEYaJG1IpTAnhVlGjJDppz2qnec7oU7fFww+oudPQ38F/65eaEoxXE85nEOBuv+sMa6O/25+LBzt8XMqs6d1g= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1564672476484523.4850721329235; Thu, 1 Aug 2019 08:14:36 -0700 (PDT) Received: from localhost ([::1]:56648 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1htCmn-00038k-UT for importer@patchew.org; Thu, 01 Aug 2019 11:14:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35009) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1htClA-0001Os-Sr for qemu-devel@nongnu.org; Thu, 01 Aug 2019 11:12:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1htCl9-0006PJ-Un for qemu-devel@nongnu.org; Thu, 01 Aug 2019 11:12:52 -0400 Received: from mx1.redhat.com ([209.132.183.28]:27411) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1htCl0-0006LJ-VG; Thu, 01 Aug 2019 11:12:45 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DBA18C0B934C; Thu, 1 Aug 2019 15:12:39 +0000 (UTC) Received: from localhost (ovpn-204-187.brq.redhat.com [10.40.204.187]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 72CDA60852; Thu, 1 Aug 2019 15:12:39 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Thu, 1 Aug 2019 17:12:34 +0200 Message-Id: <20190801151235.8434-2-mreitz@redhat.com> In-Reply-To: <20190801151235.8434-1-mreitz@redhat.com> References: <20190801151235.8434-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 01 Aug 2019 15:12:39 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH for-4.1 1/2] backup: Copy only dirty areas X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , John Snow , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" The backup job must only copy areas that the copy_bitmap reports as dirty. This is always the case when using traditional non-offloading backup, because it copies each cluster separately. When offloading the copy operation, we sometimes copy more than one cluster at a time, but we only check whether the first one is dirty. Therefore, whenever copy offloading is possible, the backup job currently produces wrong output when the guest writes to an area of which an inner part has already been backed up, because that inner part will be re-copied. Fixes: 9ded4a0114968e98b41494fc035ba14f84cdf700 Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/backup.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/block/backup.c b/block/backup.c index 715e1d3be8..1ee271f9f1 100644 --- a/block/backup.c +++ b/block/backup.c @@ -202,22 +202,31 @@ static int coroutine_fn backup_do_cow(BackupBlockJob = *job, cow_request_begin(&cow_request, job, start, end); =20 while (start < end) { + int64_t dirty_end; + if (!hbitmap_get(job->copy_bitmap, start)) { trace_backup_do_cow_skip(job, start); start +=3D job->cluster_size; continue; /* already copied */ } =20 + dirty_end =3D hbitmap_next_zero(job->copy_bitmap, start, (end - st= art)); + if (dirty_end < 0) { + dirty_end =3D end; + } + trace_backup_do_cow_process(job, start); =20 if (job->use_copy_range) { - ret =3D backup_cow_with_offload(job, start, end, is_write_noti= fier); + ret =3D backup_cow_with_offload(job, start, dirty_end, + is_write_notifier); if (ret < 0) { job->use_copy_range =3D false; } } if (!job->use_copy_range) { - ret =3D backup_cow_with_bounce_buffer(job, start, end, is_writ= e_notifier, + ret =3D backup_cow_with_bounce_buffer(job, start, dirty_end, + is_write_notifier, error_is_read, &bounce_buf= fer); } if (ret < 0) { --=20 2.21.0 From nobody Fri May 3 13:49:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1564672476; cv=none; d=zoho.com; s=zohoarc; b=SyT3fEpvoueRUFbIBAk6L6UvEORGM7hI0fDjGFM/474NceeqidopkwKPDoyjv9ab0xX55jOwD2IkZKiP8MjWWydCGA1T68ipDE0PMEIiGjeSfhbz7uo6Flk+qnTzqkxnxMkNx6Xk/jmDH3aIt5E9F/5GXcFI4MsPgF9vfvi5L3c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564672476; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=3CxHDSu7yAK41Ba93k9+JJlVfeMgVv+jM0IFtqWiGzU=; b=k8bauMVsTwLaxAIipwEeoLQ66LemMz1iv/5FVOjdFJVuEFwR3uWTPtswXLgMWAZ/Zccfuv0d7pN2KmdDMepTkCiiQb5huMbFldb7vbrX85ZFEBd+PVDdlgTQbWIphOILtYgsu6d3SErfpHAXGrZ4wSXYxN2/MNHNB2zHX4WzoPw= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1564672476450951.716408887564; Thu, 1 Aug 2019 08:14:36 -0700 (PDT) Received: from localhost ([::1]:56646 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1htCmk-00031d-Vo for importer@patchew.org; Thu, 01 Aug 2019 11:14:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35020) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1htClB-0001Q5-Ga for qemu-devel@nongnu.org; Thu, 01 Aug 2019 11:12:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1htClA-0006PR-DZ for qemu-devel@nongnu.org; Thu, 01 Aug 2019 11:12:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:15036) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1htCl7-0006Lo-7T; Thu, 01 Aug 2019 11:12:49 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 32153C0BAA1C; Thu, 1 Aug 2019 15:12:42 +0000 (UTC) Received: from localhost (ovpn-204-187.brq.redhat.com [10.40.204.187]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BB19B60BE0; Thu, 1 Aug 2019 15:12:41 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Thu, 1 Aug 2019 17:12:35 +0200 Message-Id: <20190801151235.8434-3-mreitz@redhat.com> In-Reply-To: <20190801151235.8434-1-mreitz@redhat.com> References: <20190801151235.8434-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 01 Aug 2019 15:12:42 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH for-4.1 2/2] iotests: Test backup job with two guest writes X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , John Snow , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" Perform two guest writes to not yet backed up areas of an image, where the former touches an inner area of the latter. Before HEAD^, copy offloading broke this in two ways: (1) The output differs from the reference output (what the source was before the guest writes). (2) But you will not see that in the failing output, because the job offset is reported as being greater than the job length. This is because one cluster is copied twice, and thus accounted for twice, but of course the job length does not increase. Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Tested-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/056 | 34 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/056.out | 4 ++-- 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056 index f40fc11a09..d7198507f5 100755 --- a/tests/qemu-iotests/056 +++ b/tests/qemu-iotests/056 @@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm =3D iotests.VM() self.test_img =3D img_create('test') self.dest_img =3D img_create('dest') + self.ref_img =3D img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch() =20 @@ -140,6 +141,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.ref_img) =20 def hmp_io_writes(self, drive, patterns): for pattern in patterns: @@ -177,6 +179,38 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(event, 'data/error', qerror) return False =20 + def test_overlapping_writes(self): + # Write something to back up + self.hmp_io_writes('drive0', [('42', '0M', '2M')]) + + # Create a reference backup + self.qmp_backup_and_wait(device=3D'drive0', format=3Diotests.imgfm= t, + sync=3D'full', target=3Dself.ref_img) + + # Now to the test backup: We simulate the following guest + # writes: + # (1) [1M + 64k, 1M + 128k): Afterwards, everything in that + # area should be in the target image, and we must not copy + # it again (because the source image has changed now) + # (64k is the job's cluster size) + # (2) [1M, 2M): The backup job must not get overeager. It + # must copy [1M, 1M + 64k) and [1M + 128k, 2M) separately, + # but not the area in between. + + self.qmp_backup(device=3D'drive0', format=3Diotests.imgfmt, sync= =3D'full', + target=3Dself.dest_img, speed=3D1) + + self.hmp_io_writes('drive0', [('23', '%ik' % (1024 + 64), '64k'), + ('66', '1M', '1M')]) + + # Let the job complete + res =3D self.vm.qmp('block-job-set-speed', device=3D'drive0', spee= d=3D0) + self.assert_qmp(res, 'return', {}) + self.qmp_backup_wait('drive0') + + self.assertTrue(iotests.compare_images(self.ref_img, self.dest_img= ), + 'target image does not match reference image') + def test_dismiss_false(self): res =3D self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return', []) diff --git a/tests/qemu-iotests/056.out b/tests/qemu-iotests/056.out index dae404e278..36376bed87 100644 --- a/tests/qemu-iotests/056.out +++ b/tests/qemu-iotests/056.out @@ -1,5 +1,5 @@ -......... +.......... ---------------------------------------------------------------------- -Ran 9 tests +Ran 10 tests =20 OK --=20 2.21.0