From nobody Thu Dec 18 17:49:44 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1611671179; cv=none; d=zohomail.com; s=zohoarc; b=jPkuuiptPEeeZd1bjwyoTKL0vPYy2bWVDl+LTlUIAVkYk4pPTWSe4l+0x64qZAHhU2BCHsSd0x9O+6lftadn6aI1F/ze7qR2HtUlJLse2gWKVvvZVTjfHriPM/LeSPtnPgG0fGdrN5l14wxj8OEQi2rudqslpHDzhHkPRBol9us= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1611671179; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=7DY26Uwzm/RG90y+FllHzvY8BWsKMqUOb0JEOqyIMu8=; b=iA/bgLnZ3Cs7eeZtxfGdNCr94vfqr+GWMLeIdrQuY5CsZMZQ3ZCuVpnZ0IjbOV9CIoPTHVbhFbf7xKAQN+JXzlP+ji1werOZcUz6Q43BOb4MVxIQUVfuGXhKmCotV6+Vr+m0j+zO8+a8+i7yKEHK9LS2vK3IqtvKEJk2+YsTZ90= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1611671179754716.2870073930368; Tue, 26 Jan 2021 06:26:19 -0800 (PST) Received: from localhost ([::1]:46274 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l4PIQ-000685-FG for importer@patchew.org; Tue, 26 Jan 2021 09:26:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:46508) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l4PDX-0008T4-OW for qemu-devel@nongnu.org; Tue, 26 Jan 2021 09:21:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:35386) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l4PDP-00067G-5K for qemu-devel@nongnu.org; Tue, 26 Jan 2021 09:21:15 -0500 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-28-K8lqBSUQOYW45McMe7XKCg-1; Tue, 26 Jan 2021 09:21:03 -0500 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C406E59; Tue, 26 Jan 2021 14:20:50 +0000 (UTC) Received: from localhost (ovpn-114-175.ams2.redhat.com [10.36.114.175]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 395B110023AB; Tue, 26 Jan 2021 14:20:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611670865; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7DY26Uwzm/RG90y+FllHzvY8BWsKMqUOb0JEOqyIMu8=; b=RGpUD7i+qlCHvxYDne79CBslw/QkCvkq5zt+YFNuZwsgNVhRqS8JHmVVhXZViMHJpOrV6h PGjYLQxwsCVDbJI5BAs5uPOwlHhsx0EKSIqWbyJV9ALx4kUFvY1exY+AjZKmsHhrLxWSwV YJQOHYVLQDM+r+X7rQBBcj8tDGhnAG0= X-MC-Unique: K8lqBSUQOYW45McMe7XKCg-1 From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 14/53] block: apply COR-filter to block-stream jobs Date: Tue, 26 Jan 2021 15:19:37 +0100 Message-Id: <20210126142016.806073-15-mreitz@redhat.com> In-Reply-To: <20210126142016.806073-1-mreitz@redhat.com> References: <20210126142016.806073-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mreitz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=63.128.21.124; envelope-from=mreitz@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.255, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Andrey Shinkevich This patch completes the series with the COR-filter applied to block-stream operations. Adding the filter makes it possible in future implement discarding copied regions in backing files during the block-stream job, to reduce the disk overuse (we need control on permissions). Also, the filter now is smart enough to do copy-on-read with specified base, so we have benefit on guest reads even when doing block-stream of the part of the backing chain. Several iotests are slightly modified due to filter insertion. Signed-off-by: Andrey Shinkevich Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20201216061703.70908-14-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- block/stream.c | 105 ++++++++++++++++++++++--------------- tests/qemu-iotests/030 | 8 +-- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/245 | 20 ++++--- 4 files changed, 80 insertions(+), 55 deletions(-) diff --git a/block/stream.c b/block/stream.c index 626dfa2b22..1fa742b0db 100644 --- a/block/stream.c +++ b/block/stream.c @@ -17,8 +17,10 @@ #include "block/blockjob_int.h" #include "qapi/error.h" #include "qapi/qmp/qerror.h" +#include "qapi/qmp/qdict.h" #include "qemu/ratelimit.h" #include "sysemu/block-backend.h" +#include "block/copy-on-read.h" =20 enum { /* @@ -33,11 +35,11 @@ typedef struct StreamBlockJob { BlockJob common; BlockDriverState *base_overlay; /* COW overlay (stream from this) */ BlockDriverState *above_base; /* Node directly above the base */ + BlockDriverState *cor_filter_bs; BlockDriverState *target_bs; BlockdevOnError on_error; char *backing_file_str; bool bs_read_only; - bool chain_frozen; } StreamBlockJob; =20 static int coroutine_fn stream_populate(BlockBackend *blk, @@ -45,17 +47,7 @@ static int coroutine_fn stream_populate(BlockBackend *bl= k, { assert(bytes < SIZE_MAX); =20 - return blk_co_preadv(blk, offset, bytes, NULL, - BDRV_REQ_COPY_ON_READ | BDRV_REQ_PREFETCH); -} - -static void stream_abort(Job *job) -{ - StreamBlockJob *s =3D container_of(job, StreamBlockJob, common.job); - - if (s->chain_frozen) { - bdrv_unfreeze_backing_chain(s->target_bs, s->above_base); - } + return blk_co_preadv(blk, offset, bytes, NULL, BDRV_REQ_PREFETCH); } =20 static int stream_prepare(Job *job) @@ -67,8 +59,9 @@ static int stream_prepare(Job *job) Error *local_err =3D NULL; int ret =3D 0; =20 - bdrv_unfreeze_backing_chain(s->target_bs, s->above_base); - s->chain_frozen =3D false; + /* We should drop filter at this point, as filter hold the backing cha= in */ + bdrv_cor_filter_drop(s->cor_filter_bs); + s->cor_filter_bs =3D NULL; =20 if (bdrv_cow_child(unfiltered_bs)) { const char *base_id =3D NULL, *base_fmt =3D NULL; @@ -94,6 +87,11 @@ static void stream_clean(Job *job) StreamBlockJob *s =3D container_of(job, StreamBlockJob, common.job); BlockJob *bjob =3D &s->common; =20 + if (s->cor_filter_bs) { + bdrv_cor_filter_drop(s->cor_filter_bs); + s->cor_filter_bs =3D NULL; + } + /* Reopen the image back in read-only mode if necessary */ if (s->bs_read_only) { /* Give up write permissions before making it read-only */ @@ -109,7 +107,6 @@ static int coroutine_fn stream_run(Job *job, Error **er= rp) StreamBlockJob *s =3D container_of(job, StreamBlockJob, common.job); BlockBackend *blk =3D s->common.blk; BlockDriverState *unfiltered_bs =3D bdrv_skip_filters(s->target_bs); - bool enable_cor =3D !bdrv_cow_child(s->base_overlay); int64_t len; int64_t offset =3D 0; uint64_t delay_ns =3D 0; @@ -127,15 +124,6 @@ static int coroutine_fn stream_run(Job *job, Error **e= rrp) } job_progress_set_remaining(&s->common.job, len); =20 - /* Turn on copy-on-read for the whole block device so that guest read - * requests help us make progress. Only do this when copying the enti= re - * backing chain since the copy-on-read operation does not take base i= nto - * account. - */ - if (enable_cor) { - bdrv_enable_copy_on_read(s->target_bs); - } - for ( ; offset < len; offset +=3D n) { bool copy; int ret; @@ -194,10 +182,6 @@ static int coroutine_fn stream_run(Job *job, Error **e= rrp) } } =20 - if (enable_cor) { - bdrv_disable_copy_on_read(s->target_bs); - } - /* Do not remove the backing file if an error was there but ignored. */ return error; } @@ -209,7 +193,6 @@ static const BlockJobDriver stream_job_driver =3D { .free =3D block_job_free, .run =3D stream_run, .prepare =3D stream_prepare, - .abort =3D stream_abort, .clean =3D stream_clean, .user_resume =3D block_job_user_resume, }, @@ -228,7 +211,9 @@ void stream_start(const char *job_id, BlockDriverState = *bs, bool bs_read_only; int basic_flags =3D BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGE= D; BlockDriverState *base_overlay; + BlockDriverState *cor_filter_bs =3D NULL; BlockDriverState *above_base; + QDict *opts; =20 assert(!(base && bottom)); assert(!(backing_file_str && bottom)); @@ -266,30 +251,62 @@ void stream_start(const char *job_id, BlockDriverStat= e *bs, } } =20 - if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) { - return; - } - /* Make sure that the image is opened in read-write mode */ bs_read_only =3D bdrv_is_read_only(bs); if (bs_read_only) { - if (bdrv_reopen_set_read_only(bs, false, errp) !=3D 0) { - bs_read_only =3D false; - goto fail; + int ret; + /* Hold the chain during reopen */ + if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) { + return; + } + + ret =3D bdrv_reopen_set_read_only(bs, false, errp); + + /* failure, or cor-filter will hold the chain */ + bdrv_unfreeze_backing_chain(bs, above_base); + + if (ret < 0) { + return; } } =20 - /* Prevent concurrent jobs trying to modify the graph structure here, = we - * already have our own plans. Also don't allow resize as the image si= ze is - * queried only at the job start and then cached. */ - s =3D block_job_create(job_id, &stream_job_driver, NULL, bs, - basic_flags | BLK_PERM_GRAPH_MOD, + opts =3D qdict_new(); + + qdict_put_str(opts, "driver", "copy-on-read"); + qdict_put_str(opts, "file", bdrv_get_node_name(bs)); + /* Pass the base_overlay node name as 'bottom' to COR driver */ + qdict_put_str(opts, "bottom", base_overlay->node_name); + if (filter_node_name) { + qdict_put_str(opts, "node-name", filter_node_name); + } + + cor_filter_bs =3D bdrv_insert_node(bs, opts, BDRV_O_RDWR, errp); + if (!cor_filter_bs) { + goto fail; + } + + if (!filter_node_name) { + cor_filter_bs->implicit =3D true; + } + + s =3D block_job_create(job_id, &stream_job_driver, NULL, cor_filter_bs, + BLK_PERM_CONSISTENT_READ, basic_flags | BLK_PERM_WRITE, speed, creation_flags, NULL, NULL, errp); if (!s) { goto fail; } =20 + /* + * Prevent concurrent jobs trying to modify the graph structure here, = we + * already have our own plans. Also don't allow resize as the image si= ze is + * queried only at the job start and then cached. + */ + if (block_job_add_bdrv(&s->common, "active node", bs, 0, + basic_flags | BLK_PERM_WRITE, &error_abort)) { + goto fail; + } + /* Block all intermediate nodes between bs and base, because they will * disappear from the chain after this operation. The streaming job re= ads * every block only once, assuming that it doesn't change, so forbid w= rites @@ -310,9 +327,9 @@ void stream_start(const char *job_id, BlockDriverState = *bs, s->base_overlay =3D base_overlay; s->above_base =3D above_base; s->backing_file_str =3D g_strdup(backing_file_str); + s->cor_filter_bs =3D cor_filter_bs; s->target_bs =3D bs; s->bs_read_only =3D bs_read_only; - s->chain_frozen =3D true; =20 s->on_error =3D on_error; trace_stream_start(bs, base, s); @@ -320,8 +337,10 @@ void stream_start(const char *job_id, BlockDriverState= *bs, return; =20 fail: + if (cor_filter_bs) { + bdrv_cor_filter_drop(cor_filter_bs); + } if (bs_read_only) { bdrv_reopen_set_read_only(bs, true, NULL); } - bdrv_unfreeze_backing_chain(bs, above_base); } diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index f8a692432c..832fe4a1e2 100755 --- a/tests/qemu-iotests/030 +++ b/tests/qemu-iotests/030 @@ -279,12 +279,14 @@ class TestParallelOps(iotests.QMPTestCase): self.assert_no_active_block_jobs() =20 # Set a speed limit to make sure that this job blocks the rest - result =3D self.vm.qmp('block-stream', device=3D'node4', job_id=3D= 'stream-node4', base=3Dself.imgs[1], speed=3D1024*1024) + result =3D self.vm.qmp('block-stream', device=3D'node4', + job_id=3D'stream-node4', base=3Dself.imgs[1], + filter_node_name=3D'stream-filter', speed=3D1= 024*1024) self.assert_qmp(result, 'return', {}) =20 result =3D self.vm.qmp('block-stream', device=3D'node5', job_id=3D= 'stream-node5', base=3Dself.imgs[2]) self.assert_qmp(result, 'error/desc', - "Node 'node4' is busy: block device is in use by block job: st= ream") + "Node 'stream-filter' is busy: block device is in use by block= job: stream") =20 result =3D self.vm.qmp('block-stream', device=3D'node3', job_id=3D= 'stream-node3', base=3Dself.imgs[2]) self.assert_qmp(result, 'error/desc', @@ -297,7 +299,7 @@ class TestParallelOps(iotests.QMPTestCase): # block-commit should also fail if it touches nodes used by the st= ream job result =3D self.vm.qmp('block-commit', device=3D'drive0', base=3Ds= elf.imgs[4], job_id=3D'commit-node4') self.assert_qmp(result, 'error/desc', - "Node 'node4' is busy: block device is in use by block job: st= ream") + "Node 'stream-filter' is busy: block device is in use by block= job: stream") =20 result =3D self.vm.qmp('block-commit', device=3D'drive0', base=3Ds= elf.imgs[1], top=3Dself.imgs[3], job_id=3D'commit-node1') self.assert_qmp(result, 'error/desc', diff --git a/tests/qemu-iotests/141.out b/tests/qemu-iotests/141.out index 6d8652e22b..c4c15fb275 100644 --- a/tests/qemu-iotests/141.out +++ b/tests/qemu-iotests/141.out @@ -165,7 +165,7 @@ wrote 1048576/1048576 bytes at offset 0 {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event"= : "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} {'execute': 'blockdev-del', 'arguments': {'node-name': 'drv0'}} -{"error": {"class": "GenericError", "desc": "Node drv0 is in use"}} +{"error": {"class": "GenericError", "desc": "Node 'drv0' is busy: block de= vice is in use by block job: stream"}} {'execute': 'block-job-cancel', 'arguments': {'device': 'job0'}} {"return": {}} diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245 index 86f00f290f..cfdeb902be 100755 --- a/tests/qemu-iotests/245 +++ b/tests/qemu-iotests/245 @@ -893,20 +893,24 @@ class TestBlockdevReopen(iotests.QMPTestCase): =20 # hd1 <- hd0 result =3D self.vm.qmp('block-stream', conv_keys =3D True, job_id = =3D 'stream0', - device =3D 'hd1', auto_finalize =3D False) + device =3D 'hd1', filter_node_name=3D'cor', + auto_finalize =3D False) self.assert_qmp(result, 'return', {}) =20 - # We can't reopen with the original options because that would - # make hd1 read-only and block-stream requires it to be read-write - # (Which error message appears depends on whether the stream job is - # already done with copying at this point.) + # We can't reopen with the original options because there is a fil= ter + # inserted by stream job above hd1. self.reopen(opts, {}, - ["Can't set node 'hd1' to r/o with copy-on-read enabled", - "Cannot make block node read-only, there is a writer on it"]) + "Cannot change the option 'backing.backing.file.node-n= ame'") + + # We can't reopen hd1 to read-only, as block-stream requires it to= be + # read-write + self.reopen(opts['backing'], {'read-only': True}, + "Cannot make block node read-only, there is a writer o= n it") =20 # We can't remove hd2 while the stream job is ongoing opts['backing']['backing'] =3D None - self.reopen(opts, {'backing.read-only': False}, "Cannot change 'ba= cking' link from 'hd1' to 'hd2'") + self.reopen(opts['backing'], {'read-only': False}, + "Cannot change 'backing' link from 'hd1' to 'hd2'") =20 # We can detach hd1 from hd0 because it doesn't affect the stream = job opts['backing'] =3D None --=20 2.29.2