From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559071920; cv=none; d=zoho.com; s=zohoarc; b=XmmPVqk36sWR17RaAFPpAJD8aFUGZnryURpDaYho9ruzedF2KZm8R/jaJQFW5JOp8MLhdCaQgMGg7/pCzP8MCfUpDWm13SAS0iVSrx5wvaPfb+4Jg/DSDpL2cPiIHFsdZv3qDk8+hUXff+UEiUKMLyvctz/rECuagz6T1i139I8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559071920; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=lQD7eqg2KU/kdosvDY3jGbPYXWxoRsiTdgA1YZFTzIo=; b=kExsPcW6CjrjFJPuQlnnCm+WGgwXzAcomLQoU4AUSkm8ByJ+VcbtBpVLB06n8OrZalFhLvOUgqtWrizbSEA7vHSj+Tj8hKBOI6BRhx9onobctqURMKnro8f8dVgyAjdRp0nqXnojrwrrIN0zFNq5zo2FgX2rlfaVJUi1y9vJBQ0= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559071920912697.1907007543299; Tue, 28 May 2019 12:32:00 -0700 (PDT) Received: from localhost ([127.0.0.1]:41520 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhpB-0001yu-T9 for importer@patchew.org; Tue, 28 May 2019 15:31:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36655) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhmy-0000jR-J2 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhmn-0000gB-Vg for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:29 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41830) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhmP-0008Ul-M6; Tue, 28 May 2019 15:29:03 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C7ACC3082134; Tue, 28 May 2019 19:28:51 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5B5DD1972B; Tue, 28 May 2019 19:28:51 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:27 +0200 Message-Id: <20190528192847.2730-2-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Tue, 28 May 2019 19:28:51 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 01/21] qcow2.h: add missing include X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy qcow2.h depends on block_int.h. Compilation isn't broken currently only due to block_int.h always included before qcow2.h. Though, it seems better to directly include block_int.h in qcow2.h. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Max Reitz Message-id: 20190506142741.41731-2-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/qcow2.h | 1 + block/qcow2-bitmap.c | 1 - block/qcow2-cache.c | 1 - block/qcow2-cluster.c | 1 - block/qcow2-refcount.c | 1 - block/qcow2-snapshot.c | 1 - block/qcow2.c | 1 - 7 files changed, 1 insertion(+), 6 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index 8d92ef1fee..c2b0e955eb 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -28,6 +28,7 @@ #include "crypto/block.h" #include "qemu/coroutine.h" #include "qemu/units.h" +#include "block/block_int.h" =20 //#define DEBUG_ALLOC //#define DEBUG_ALLOC2 diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 8a75366c92..640da68ce1 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -29,7 +29,6 @@ #include "qapi/error.h" #include "qemu/cutils.h" =20 -#include "block/block_int.h" #include "qcow2.h" =20 /* NOTICE: BME here means Bitmaps Extension and used as a namespace for diff --git a/block/qcow2-cache.c b/block/qcow2-cache.c index df02e7b20a..b33bcbc984 100644 --- a/block/qcow2-cache.c +++ b/block/qcow2-cache.c @@ -23,7 +23,6 @@ */ =20 #include "qemu/osdep.h" -#include "block/block_int.h" #include "qemu-common.h" #include "qcow2.h" #include "trace.h" diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index b36f4aa84a..a4c878398e 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -27,7 +27,6 @@ =20 #include "qapi/error.h" #include "qemu-common.h" -#include "block/block_int.h" #include "qcow2.h" #include "qemu/bswap.h" #include "trace.h" diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 0b09d6838b..4c1794f9af 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -25,7 +25,6 @@ #include "qemu/osdep.h" #include "qapi/error.h" #include "qemu-common.h" -#include "block/block_int.h" #include "qcow2.h" #include "qemu/range.h" #include "qemu/bswap.h" diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index a6ffae89a6..d0e7fa9311 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -24,7 +24,6 @@ =20 #include "qemu/osdep.h" #include "qapi/error.h" -#include "block/block_int.h" #include "qcow2.h" #include "qemu/bswap.h" #include "qemu/error-report.h" diff --git a/block/qcow2.c b/block/qcow2.c index d39882785d..38bf729a1e 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -27,7 +27,6 @@ #define ZLIB_CONST #include =20 -#include "block/block_int.h" #include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/module.h" --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072141; cv=none; d=zoho.com; s=zohoarc; b=Lqdcxjzqzrno1wLv2JK5cIOEAk2VSMy/T8U90NpAYL8OPXlZ0FbTG5jeMUTBjU9hc14wMAxyWr6EUCLShBhYYdksEn6v9NHVvhtXyNrWTwaEocaINQvcIQyEJrNvWwLDklU9pNniTy/Fyacebkqh7p8CsH1qviY7mZG4RDpdpYE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072141; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=nGVIX0FDIcgtOpoVTpom8GL2loaLO7ydxajLrb8Lq+Q=; b=KSy1txChl1+rbCIZdaAeUxbr0FOH/srgmxWPExPmSMQ85YcX5wYbqEBgt0WB6yTBZHKbLKxbuPyyHVwsNpi67RIDQH91298gMpqoUCgfEDnTE8Voup0PortEBAR4aQKSN5AruficJdy6v4qFD/WNXnpg9h2SWjirySvIhG9Xg4o= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072141813729.1854156460508; Tue, 28 May 2019 12:35:41 -0700 (PDT) Received: from localhost ([127.0.0.1]:41549 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhsk-0004dB-O9 for importer@patchew.org; Tue, 28 May 2019 15:35:34 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36701) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhnA-0000ry-NN for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhmz-0000v2-Bh for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:40 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41846) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhmT-00006X-P2; Tue, 28 May 2019 15:29:08 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9D5EB307E04E; Tue, 28 May 2019 19:28:54 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C9FD11972B; Tue, 28 May 2019 19:28:53 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:28 +0200 Message-Id: <20190528192847.2730-3-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Tue, 28 May 2019 19:28:54 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 02/21] qcow2: add separate file for threaded data processing functions X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Move compression-on-threads to separate file. Encryption will be in it too. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Max Reitz Message-id: 20190506142741.41731-3-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/Makefile.objs | 2 +- block/qcow2.h | 7 ++ block/qcow2-threads.c | 201 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 169 ----------------------------------- 4 files changed, 209 insertions(+), 170 deletions(-) create mode 100644 block/qcow2-threads.c diff --git a/block/Makefile.objs b/block/Makefile.objs index 7a81892a52..ae11605c9f 100644 --- a/block/Makefile.objs +++ b/block/Makefile.objs @@ -6,7 +6,7 @@ block-obj-$(CONFIG_BOCHS) +=3D bochs.o block-obj-$(CONFIG_VVFAT) +=3D vvfat.o block-obj-$(CONFIG_DMG) +=3D dmg.o =20 -block-obj-y +=3D qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o= qcow2-cache.o qcow2-bitmap.o +block-obj-y +=3D qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o= qcow2-cache.o qcow2-bitmap.o qcow2-threads.o block-obj-$(CONFIG_QED) +=3D qed.o qed-l2-cache.o qed-table.o qed-cluster.o block-obj-$(CONFIG_QED) +=3D qed-check.o block-obj-y +=3D vhdx.o vhdx-endian.o vhdx-log.o diff --git a/block/qcow2.h b/block/qcow2.h index c2b0e955eb..8207ed374e 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -738,4 +738,11 @@ void qcow2_remove_persistent_dirty_bitmap(BlockDriverS= tate *bs, const char *name, Error **errp); =20 +ssize_t coroutine_fn +qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, + const void *src, size_t src_size); +ssize_t coroutine_fn +qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, + const void *src, size_t src_size); + #endif diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c new file mode 100644 index 0000000000..85b8034fd2 --- /dev/null +++ b/block/qcow2-threads.c @@ -0,0 +1,201 @@ +/* + * Threaded data processing for Qcow2: compression, encryption + * + * Copyright (c) 2004-2006 Fabrice Bellard + * Copyright (c) 2018 Virtuozzo International GmbH. All rights reserved. + * + * Permission is hereby granted, free of charge, to any person obtaining a= copy + * of this software and associated documentation files (the "Software"), t= o deal + * in the Software without restriction, including without limitation the r= ights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or se= ll + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included= in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS= OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OT= HER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING= FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS = IN + * THE SOFTWARE. + */ + +#include "qemu/osdep.h" + +#define ZLIB_CONST +#include + +#include "qcow2.h" +#include "block/thread-pool.h" + +#define MAX_COMPRESS_THREADS 4 + +typedef ssize_t (*Qcow2CompressFunc)(void *dest, size_t dest_size, + const void *src, size_t src_size); +typedef struct Qcow2CompressData { + void *dest; + size_t dest_size; + const void *src; + size_t src_size; + ssize_t ret; + + Qcow2CompressFunc func; +} Qcow2CompressData; + +/* + * qcow2_compress() + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: compressed size on success + * -ENOMEM destination buffer is not enough to store compressed d= ata + * -EIO on any other error + */ +static ssize_t qcow2_compress(void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + ssize_t ret; + z_stream strm; + + /* best compression, small window, no zlib header */ + memset(&strm, 0, sizeof(strm)); + ret =3D deflateInit2(&strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED, + -12, 9, Z_DEFAULT_STRATEGY); + if (ret !=3D Z_OK) { + return -EIO; + } + + /* + * strm.next_in is not const in old zlib versions, such as those used = on + * OpenBSD/NetBSD, so cast the const away + */ + strm.avail_in =3D src_size; + strm.next_in =3D (void *) src; + strm.avail_out =3D dest_size; + strm.next_out =3D dest; + + ret =3D deflate(&strm, Z_FINISH); + if (ret =3D=3D Z_STREAM_END) { + ret =3D dest_size - strm.avail_out; + } else { + ret =3D (ret =3D=3D Z_OK ? -ENOMEM : -EIO); + } + + deflateEnd(&strm); + + return ret; +} + +/* + * qcow2_decompress() + * + * Decompress some data (not more than @src_size bytes) to produce exactly + * @dest_size bytes. + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: 0 on success + * -1 on fail + */ +static ssize_t qcow2_decompress(void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + int ret =3D 0; + z_stream strm; + + memset(&strm, 0, sizeof(strm)); + strm.avail_in =3D src_size; + strm.next_in =3D (void *) src; + strm.avail_out =3D dest_size; + strm.next_out =3D dest; + + ret =3D inflateInit2(&strm, -12); + if (ret !=3D Z_OK) { + return -1; + } + + ret =3D inflate(&strm, Z_FINISH); + if ((ret !=3D Z_STREAM_END && ret !=3D Z_BUF_ERROR) || strm.avail_out = !=3D 0) { + /* + * We approve Z_BUF_ERROR because we need @dest buffer to be fille= d, but + * @src buffer may be processed partly (because in qcow2 we know s= ize of + * compressed data with precision of one sector) + */ + ret =3D -1; + } + + inflateEnd(&strm); + + return ret; +} + +static int qcow2_compress_pool_func(void *opaque) +{ + Qcow2CompressData *data =3D opaque; + + data->ret =3D data->func(data->dest, data->dest_size, + data->src, data->src_size); + + return 0; +} + +static void qcow2_compress_complete(void *opaque, int ret) +{ + qemu_coroutine_enter(opaque); +} + +static ssize_t coroutine_fn +qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, + const void *src, size_t src_size, Qcow2CompressFunc f= unc) +{ + BDRVQcow2State *s =3D bs->opaque; + BlockAIOCB *acb; + ThreadPool *pool =3D aio_get_thread_pool(bdrv_get_aio_context(bs)); + Qcow2CompressData arg =3D { + .dest =3D dest, + .dest_size =3D dest_size, + .src =3D src, + .src_size =3D src_size, + .func =3D func, + }; + + while (s->nb_compress_threads >=3D MAX_COMPRESS_THREADS) { + qemu_co_queue_wait(&s->compress_wait_queue, NULL); + } + + s->nb_compress_threads++; + acb =3D thread_pool_submit_aio(pool, qcow2_compress_pool_func, &arg, + qcow2_compress_complete, + qemu_coroutine_self()); + + if (!acb) { + s->nb_compress_threads--; + return -EINVAL; + } + qemu_coroutine_yield(); + s->nb_compress_threads--; + qemu_co_queue_next(&s->compress_wait_queue); + + return arg.ret; +} + +ssize_t coroutine_fn +qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, + qcow2_compress); +} + +ssize_t coroutine_fn +qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, + qcow2_decompress); +} diff --git a/block/qcow2.c b/block/qcow2.c index 38bf729a1e..a8c4b11539 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -24,9 +24,6 @@ =20 #include "qemu/osdep.h" =20 -#define ZLIB_CONST -#include - #include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/module.h" @@ -43,7 +40,6 @@ #include "qapi/qobject-input-visitor.h" #include "qapi/qapi-visit-block-core.h" #include "crypto.h" -#include "block/thread-pool.h" =20 /* Differences with QCOW: @@ -3920,171 +3916,6 @@ fail: return ret; } =20 -/* - * qcow2_compress() - * - * @dest - destination buffer, @dest_size bytes - * @src - source buffer, @src_size bytes - * - * Returns: compressed size on success - * -ENOMEM destination buffer is not enough to store compressed d= ata - * -EIO on any other error - */ -static ssize_t qcow2_compress(void *dest, size_t dest_size, - const void *src, size_t src_size) -{ - ssize_t ret; - z_stream strm; - - /* best compression, small window, no zlib header */ - memset(&strm, 0, sizeof(strm)); - ret =3D deflateInit2(&strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED, - -12, 9, Z_DEFAULT_STRATEGY); - if (ret !=3D Z_OK) { - return -EIO; - } - - /* strm.next_in is not const in old zlib versions, such as those used = on - * OpenBSD/NetBSD, so cast the const away */ - strm.avail_in =3D src_size; - strm.next_in =3D (void *) src; - strm.avail_out =3D dest_size; - strm.next_out =3D dest; - - ret =3D deflate(&strm, Z_FINISH); - if (ret =3D=3D Z_STREAM_END) { - ret =3D dest_size - strm.avail_out; - } else { - ret =3D (ret =3D=3D Z_OK ? -ENOMEM : -EIO); - } - - deflateEnd(&strm); - - return ret; -} - -/* - * qcow2_decompress() - * - * Decompress some data (not more than @src_size bytes) to produce exactly - * @dest_size bytes. - * - * @dest - destination buffer, @dest_size bytes - * @src - source buffer, @src_size bytes - * - * Returns: 0 on success - * -1 on fail - */ -static ssize_t qcow2_decompress(void *dest, size_t dest_size, - const void *src, size_t src_size) -{ - int ret =3D 0; - z_stream strm; - - memset(&strm, 0, sizeof(strm)); - strm.avail_in =3D src_size; - strm.next_in =3D (void *) src; - strm.avail_out =3D dest_size; - strm.next_out =3D dest; - - ret =3D inflateInit2(&strm, -12); - if (ret !=3D Z_OK) { - return -1; - } - - ret =3D inflate(&strm, Z_FINISH); - if ((ret !=3D Z_STREAM_END && ret !=3D Z_BUF_ERROR) || strm.avail_out = !=3D 0) { - /* We approve Z_BUF_ERROR because we need @dest buffer to be fille= d, but - * @src buffer may be processed partly (because in qcow2 we know s= ize of - * compressed data with precision of one sector) */ - ret =3D -1; - } - - inflateEnd(&strm); - - return ret; -} - -#define MAX_COMPRESS_THREADS 4 - -typedef ssize_t (*Qcow2CompressFunc)(void *dest, size_t dest_size, - const void *src, size_t src_size); -typedef struct Qcow2CompressData { - void *dest; - size_t dest_size; - const void *src; - size_t src_size; - ssize_t ret; - - Qcow2CompressFunc func; -} Qcow2CompressData; - -static int qcow2_compress_pool_func(void *opaque) -{ - Qcow2CompressData *data =3D opaque; - - data->ret =3D data->func(data->dest, data->dest_size, - data->src, data->src_size); - - return 0; -} - -static void qcow2_compress_complete(void *opaque, int ret) -{ - qemu_coroutine_enter(opaque); -} - -static ssize_t coroutine_fn -qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, - const void *src, size_t src_size, Qcow2CompressFunc f= unc) -{ - BDRVQcow2State *s =3D bs->opaque; - BlockAIOCB *acb; - ThreadPool *pool =3D aio_get_thread_pool(bdrv_get_aio_context(bs)); - Qcow2CompressData arg =3D { - .dest =3D dest, - .dest_size =3D dest_size, - .src =3D src, - .src_size =3D src_size, - .func =3D func, - }; - - while (s->nb_compress_threads >=3D MAX_COMPRESS_THREADS) { - qemu_co_queue_wait(&s->compress_wait_queue, NULL); - } - - s->nb_compress_threads++; - acb =3D thread_pool_submit_aio(pool, qcow2_compress_pool_func, &arg, - qcow2_compress_complete, - qemu_coroutine_self()); - - if (!acb) { - s->nb_compress_threads--; - return -EINVAL; - } - qemu_coroutine_yield(); - s->nb_compress_threads--; - qemu_co_queue_next(&s->compress_wait_queue); - - return arg.ret; -} - -static ssize_t coroutine_fn -qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, - const void *src, size_t src_size) -{ - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, - qcow2_compress); -} - -static ssize_t coroutine_fn -qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, - const void *src, size_t src_size) -{ - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, - qcow2_decompress); -} - /* XXX: put compressed sectors first, then all the cluster aligned tables to avoid losing bytes in alignment */ static coroutine_fn int --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072344; cv=none; d=zoho.com; s=zohoarc; b=QhAcJI5h7gpHqFgSzSq7W3p/+YEDZmJW+skcB8RlyevhDM5D4n15jkPMsc/A7ZZqdFCsbC9AbC7d6Uz2QwmeQbwXdGjJ9xBi4+1mwrFn9A99wXyb97uNcJYznk0vLRuPMqm5Fwe0iGpC7Uq4r96yu0/exWK0LuMeTbrZa0VXePQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072344; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=232Tj5AFt7YO824Xla1fvhAfuK2hXA2lH5zgwDBvTkM=; b=F0Q88XcOMbph2ZfDaQa5GNUHFSUZg7k4D/0vRzAKIE1zYYozogbbSqtjrKP6vhkMfQP2n9TtUmtxuIYSGcXNzxjPWii2r2b0LzVpXTknaMb7/WGoyz8LuXAWOSJ1+8q4RFlBBrKWwi4iPCX/tZA4CWoEEFYfwkFUpnaE41+qvxg= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072344688192.03319791347224; Tue, 28 May 2019 12:39:04 -0700 (PDT) Received: from localhost ([127.0.0.1]:41607 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhvM-0006zI-8f for importer@patchew.org; Tue, 28 May 2019 15:38:16 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36693) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhn9-0000rs-Qs for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhmy-0000u9-JX for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:40 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43160) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhmR-00009I-Ai; Tue, 28 May 2019 15:29:04 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0256430833AF; Tue, 28 May 2019 19:28:57 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8CDDA1F8; Tue, 28 May 2019 19:28:56 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:29 +0200 Message-Id: <20190528192847.2730-4-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Tue, 28 May 2019 19:28:57 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 03/21] qcow2-threads: use thread_pool_submit_co X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Use thread_pool_submit_co, instead of reinventing it here. Note, that thread_pool_submit_aio() never returns NULL, so checking it was an extra thing. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Max Reitz Message-id: 20190506142741.41731-4-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/qcow2-threads.c | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-) diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c index 85b8034fd2..5a39fdac69 100644 --- a/block/qcow2-threads.c +++ b/block/qcow2-threads.c @@ -144,17 +144,11 @@ static int qcow2_compress_pool_func(void *opaque) return 0; } =20 -static void qcow2_compress_complete(void *opaque, int ret) -{ - qemu_coroutine_enter(opaque); -} - static ssize_t coroutine_fn qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, const void *src, size_t src_size, Qcow2CompressFunc f= unc) { BDRVQcow2State *s =3D bs->opaque; - BlockAIOCB *acb; ThreadPool *pool =3D aio_get_thread_pool(bdrv_get_aio_context(bs)); Qcow2CompressData arg =3D { .dest =3D dest, @@ -169,16 +163,9 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,= size_t dest_size, } =20 s->nb_compress_threads++; - acb =3D thread_pool_submit_aio(pool, qcow2_compress_pool_func, &arg, - qcow2_compress_complete, - qemu_coroutine_self()); - - if (!acb) { - s->nb_compress_threads--; - return -EINVAL; - } - qemu_coroutine_yield(); + thread_pool_submit_co(pool, qcow2_compress_pool_func, &arg); s->nb_compress_threads--; + qemu_co_queue_next(&s->compress_wait_queue); =20 return arg.ret; --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072128; cv=none; d=zoho.com; s=zohoarc; b=dLP2ScifzMPZI4U0cPNpbIQCTiIG/v5bXY9kFiv90Y2x4UW6cguJIrMSnsIWmgsg7cm0NRmR65U4GxNJl4vBgHw3bgQaS7sLb0ZZH2uxXkx6piXob1sgGjh6ySAbNgYgIw8E8hQ+XjDlbriL45GeQvgkvNWziRq4hSGo37V5ifY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072128; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=h+lst/GeUa13ZYVzew2t9XjfrQS2mWFIOLvlRqXzbjg=; b=OAKyQtMxxdsZypM6rLWp+sFnl0xWHEeHFkLdms5uoh4QNqSvqmN5JqfzJNcWJfAXUFxzJaITTWFyweSVJvv0pvjW9VhEMZTW/TNZCglwNJ6RniSRp+FHekqMn6lJDIXdOC41gHAgQEVWA1oW1jH+tD7vkXI2GAgF9AyMv7tMHxY= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072128668529.1281369297897; Tue, 28 May 2019 12:35:28 -0700 (PDT) Received: from localhost ([127.0.0.1]:41547 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhsR-0004Pj-Jb for importer@patchew.org; Tue, 28 May 2019 15:35:15 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36678) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhn2-0000mm-BS for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhms-0000nY-UO for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:35 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51430) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhmR-0000Bd-9x; Tue, 28 May 2019 15:29:03 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 68F9981114; Tue, 28 May 2019 19:28:59 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 012F2611B4; Tue, 28 May 2019 19:28:58 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:30 +0200 Message-Id: <20190528192847.2730-5-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Tue, 28 May 2019 19:28:59 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 04/21] qcow2-threads: qcow2_co_do_compress: protect queuing by mutex X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Drop dependence on AioContext lock. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Paolo Bonzini Reviewed-by: Max Reitz Message-id: 20190506142741.41731-5-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/qcow2-threads.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c index 5a39fdac69..50a9fdeec5 100644 --- a/block/qcow2-threads.c +++ b/block/qcow2-threads.c @@ -158,15 +158,19 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest= , size_t dest_size, .func =3D func, }; =20 + qemu_co_mutex_lock(&s->lock); while (s->nb_compress_threads >=3D MAX_COMPRESS_THREADS) { - qemu_co_queue_wait(&s->compress_wait_queue, NULL); + qemu_co_queue_wait(&s->compress_wait_queue, &s->lock); } - s->nb_compress_threads++; + qemu_co_mutex_unlock(&s->lock); + thread_pool_submit_co(pool, qcow2_compress_pool_func, &arg); - s->nb_compress_threads--; =20 + qemu_co_mutex_lock(&s->lock); + s->nb_compress_threads--; qemu_co_queue_next(&s->compress_wait_queue); + qemu_co_mutex_unlock(&s->lock); =20 return arg.ret; } --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559071956; cv=none; d=zoho.com; s=zohoarc; b=nrwk/cC9HQI8o58QY/24+DFjXhJubI/y4ZlqC5V49ky9H4q0WcGNzAqQrQwWAy/Dl/QNFr+wx7CL1bbNN18TuKyz/fTUQUgNBYXWpSx4HbEDHPvTkigdzgGyrFLqPSpqtSzB97vA7hahAnJspuYT+p+/4llgmwaDP7U3cZ5ErR4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559071956; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=B1Z/VmJRQ1/yh4jdB+KOFbh2F4ieiAEY3c4seqmV0b0=; b=A6kBre55ewXAhC/T7vdJtOvZimBGOMcqnkwboKxF41w7rlabR+YgAPyq6PuDVUB63wPDGqmTafGCMtGaZoxsU1vyWjxn16DQGvmJ40AovjkzkI6EnpaoyLzAVd4/HEdHB8Zyycodct/fl9tJ6N+NixsCtJs208OtDYQMXqggOIM= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559071956121994.5204327623267; Tue, 28 May 2019 12:32:36 -0700 (PDT) Received: from localhost ([127.0.0.1]:41522 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhpl-0002OB-2f for importer@patchew.org; Tue, 28 May 2019 15:32:29 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36694) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhnA-0000rt-KJ for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhmy-0000uD-JR for qemu-devel@nongnu.org; Tue, 28 May 2019 15:29:40 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38014) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhmT-0000EQ-3L; Tue, 28 May 2019 15:29:06 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C325F307D986; Tue, 28 May 2019 19:29:01 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5937F1972B; Tue, 28 May 2019 19:29:01 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:31 +0200 Message-Id: <20190528192847.2730-6-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.48]); Tue, 28 May 2019 19:29:01 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 05/21] qcow2-threads: split out generic path X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Move generic part out of qcow2_co_do_compress, to reuse it for encryption and rename things that would be shared with encryption path. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Max Reitz Message-id: 20190506142741.41731-6-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/qcow2.h | 4 ++-- block/qcow2-threads.c | 47 ++++++++++++++++++++++++++++--------------- block/qcow2.c | 2 +- 3 files changed, 34 insertions(+), 19 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index 8207ed374e..637552e137 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -350,8 +350,8 @@ typedef struct BDRVQcow2State { char *image_backing_format; char *image_data_file; =20 - CoQueue compress_wait_queue; - int nb_compress_threads; + CoQueue thread_task_queue; + int nb_threads; =20 BdrvChild *data_file; } BDRVQcow2State; diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c index 50a9fdeec5..650aa2ed19 100644 --- a/block/qcow2-threads.c +++ b/block/qcow2-threads.c @@ -31,7 +31,36 @@ #include "qcow2.h" #include "block/thread-pool.h" =20 -#define MAX_COMPRESS_THREADS 4 +#define QCOW2_MAX_THREADS 4 + +static int coroutine_fn +qcow2_co_process(BlockDriverState *bs, ThreadPoolFunc *func, void *arg) +{ + int ret; + BDRVQcow2State *s =3D bs->opaque; + ThreadPool *pool =3D aio_get_thread_pool(bdrv_get_aio_context(bs)); + + qemu_co_mutex_lock(&s->lock); + while (s->nb_threads >=3D QCOW2_MAX_THREADS) { + qemu_co_queue_wait(&s->thread_task_queue, &s->lock); + } + s->nb_threads++; + qemu_co_mutex_unlock(&s->lock); + + ret =3D thread_pool_submit_co(pool, func, arg); + + qemu_co_mutex_lock(&s->lock); + s->nb_threads--; + qemu_co_queue_next(&s->thread_task_queue); + qemu_co_mutex_unlock(&s->lock); + + return ret; +} + + +/* + * Compression + */ =20 typedef ssize_t (*Qcow2CompressFunc)(void *dest, size_t dest_size, const void *src, size_t src_size); @@ -148,8 +177,6 @@ static ssize_t coroutine_fn qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, const void *src, size_t src_size, Qcow2CompressFunc f= unc) { - BDRVQcow2State *s =3D bs->opaque; - ThreadPool *pool =3D aio_get_thread_pool(bdrv_get_aio_context(bs)); Qcow2CompressData arg =3D { .dest =3D dest, .dest_size =3D dest_size, @@ -158,19 +185,7 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest,= size_t dest_size, .func =3D func, }; =20 - qemu_co_mutex_lock(&s->lock); - while (s->nb_compress_threads >=3D MAX_COMPRESS_THREADS) { - qemu_co_queue_wait(&s->compress_wait_queue, &s->lock); - } - s->nb_compress_threads++; - qemu_co_mutex_unlock(&s->lock); - - thread_pool_submit_co(pool, qcow2_compress_pool_func, &arg); - - qemu_co_mutex_lock(&s->lock); - s->nb_compress_threads--; - qemu_co_queue_next(&s->compress_wait_queue); - qemu_co_mutex_unlock(&s->lock); + qcow2_co_process(bs, qcow2_compress_pool_func, &arg); =20 return arg.ret; } diff --git a/block/qcow2.c b/block/qcow2.c index a8c4b11539..1413df1389 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1693,7 +1693,7 @@ static int coroutine_fn qcow2_do_open(BlockDriverStat= e *bs, QDict *options, } #endif =20 - qemu_co_queue_init(&s->compress_wait_queue); + qemu_co_queue_init(&s->thread_task_queue); =20 return ret; =20 --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559071996; cv=none; d=zoho.com; s=zohoarc; b=IXjKZOxBWat78/hQbp+ErI0cT//9Gv4V413IQP4djZtqpJ/dQqwd6aAOuE/QGUdLUAO5re8xCJ/Pl8kBqQbrHRfmiIHFUrv0FAB08Wuz8RbFoNSF6wkPwF+ZposMoM6Onk/b0Ky+Mni9FcNpq21Uj7EPh/5NYMEeF/9DAQ3U8o8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559071996; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=dMJwS6jYPhj6QRzSNxtaBR3rCQyQKz6hzMxBP3r0ZJk=; b=b3+R8aFtrtaM3sjqcvLLkQmKBL4/XY8dinKTKBdYI7G7qE/ikhqsxwbGmlD3h+6tpX+aDX4goJjdcKPA7FC0E4lHi6JVJSUjhCuouieCy9i95NuFtcHmzhYNOGJyAiTyHlST4kodToJTxsF17aRiHt1Bk1Af8VUfSYQmFqu3S2M= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 15590719965308.405449537395384; Tue, 28 May 2019 12:33:16 -0700 (PDT) Received: from localhost ([127.0.0.1]:41524 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhqS-0002uG-Fr for importer@patchew.org; Tue, 28 May 2019 15:33:12 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36908) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhnr-0001Tw-RV for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhnj-0001g4-9u for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:27 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41994) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhnS-0000Hi-Ro; Tue, 28 May 2019 15:30:08 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 315C73003A4E; Tue, 28 May 2019 19:29:04 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B8DAC611D9; Tue, 28 May 2019 19:29:03 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:32 +0200 Message-Id: <20190528192847.2730-7-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Tue, 28 May 2019 19:29:04 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 06/21] qcow2: qcow2_co_preadv: improve locking X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Background: decryption will be done in threads, to take benefit of it, we should move it out of the lock first. But let's go further: it turns out, that only qcow2_get_cluster_offset() needs locking, so reduce locking to it. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-id: 20190506142741.41731-7-vsementsov@virtuozzo.com Reviewed-by: Alberto Garcia Signed-off-by: Max Reitz --- block/qcow2.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 1413df1389..4a89850389 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1969,8 +1969,6 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverSt= ate *bs, uint64_t offset, =20 qemu_iovec_init(&hd_qiov, qiov->niov); =20 - qemu_co_mutex_lock(&s->lock); - while (bytes !=3D 0) { =20 /* prepare next request */ @@ -1980,7 +1978,9 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverSt= ate *bs, uint64_t offset, QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size); } =20 + qemu_co_mutex_lock(&s->lock); ret =3D qcow2_get_cluster_offset(bs, offset, &cur_bytes, &cluster_= offset); + qemu_co_mutex_unlock(&s->lock); if (ret < 0) { goto fail; } @@ -1995,10 +1995,8 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverS= tate *bs, uint64_t offset, =20 if (bs->backing) { BLKDBG_EVENT(bs->file, BLKDBG_READ_BACKING_AIO); - qemu_co_mutex_unlock(&s->lock); ret =3D bdrv_co_preadv(bs->backing, offset, cur_bytes, &hd_qiov, 0); - qemu_co_mutex_lock(&s->lock); if (ret < 0) { goto fail; } @@ -2014,11 +2012,9 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverS= tate *bs, uint64_t offset, break; =20 case QCOW2_CLUSTER_COMPRESSED: - qemu_co_mutex_unlock(&s->lock); ret =3D qcow2_co_preadv_compressed(bs, cluster_offset, offset, cur_bytes, &hd_qiov); - qemu_co_mutex_lock(&s->lock); if (ret < 0) { goto fail; } @@ -2055,11 +2051,9 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverS= tate *bs, uint64_t offset, } =20 BLKDBG_EVENT(bs->file, BLKDBG_READ_AIO); - qemu_co_mutex_unlock(&s->lock); ret =3D bdrv_co_preadv(s->data_file, cluster_offset + offset_in_cluster, cur_bytes, &hd_qiov, 0); - qemu_co_mutex_lock(&s->lock); if (ret < 0) { goto fail; } @@ -2094,8 +2088,6 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverSt= ate *bs, uint64_t offset, ret =3D 0; =20 fail: - qemu_co_mutex_unlock(&s->lock); - qemu_iovec_destroy(&hd_qiov); qemu_vfree(cluster_data); =20 --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072154; cv=none; d=zoho.com; s=zohoarc; b=IkyOh4tcklofznd7vRI2qwbW8rLd/Q7TP2uwEpQE4fSjb8OW+SzPUMq2ltO19kbr3EPCO9VmV13Mp0i86QGcjC0I7FgY1gPE85YJO33ObvuW7d3dDFuTL583ruzzuEM3V+kNKUAvB2uWBUUDqEmfOwBA3lDWHewdSgu9Zxcz2CM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072154; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=+8ZwIwawVNg/1zVFjQ9u3+tsDpRugMIigRvSs8H7gmc=; b=YIJGuaGE4JsMzfTHvSwG+whsSaH6QY8gADzYZH5/+2mh1GhnsCnJGKYBBn0HCZdRPCs4K67rCMUKIQz9ww0fUpFi7vP8iCcvfXcvzPZt0y4RF0TP805q251NhH8u1MyAmMBeNyUhrOZgDJX6exrnbetl3aGn3gxKfgLS4J9XwYk= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072154899493.0276921817464; Tue, 28 May 2019 12:35:54 -0700 (PDT) Received: from localhost ([127.0.0.1]:41566 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhss-0004mY-Ff for importer@patchew.org; Tue, 28 May 2019 15:35:42 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36899) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhnr-0001TK-5K for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhnf-0001dH-VJ for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:23 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60324) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhnS-0000KF-CD; Tue, 28 May 2019 15:30:06 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8D63530593D8; Tue, 28 May 2019 19:29:06 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 26C405C8A4; Tue, 28 May 2019 19:29:05 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:33 +0200 Message-Id: <20190528192847.2730-8-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Tue, 28 May 2019 19:29:06 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 07/21] qcow2: bdrv_co_pwritev: move encryption code out of the lock X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Encryption will be done in threads, to take benefit of it, we should move it out of the lock first. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Max Reitz Message-id: 20190506142741.41731-8-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/qcow2.c | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 4a89850389..4b4a0ce6e0 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -2168,11 +2168,20 @@ static coroutine_fn int qcow2_co_pwritev(BlockDrive= rState *bs, uint64_t offset, ret =3D qcow2_alloc_cluster_offset(bs, offset, &cur_bytes, &cluster_offset, &l2meta); if (ret < 0) { - goto fail; + goto out_locked; } =20 assert((cluster_offset & 511) =3D=3D 0); =20 + ret =3D qcow2_pre_write_overlap_check(bs, 0, + cluster_offset + offset_in_clu= ster, + cur_bytes, true); + if (ret < 0) { + goto out_locked; + } + + qemu_co_mutex_unlock(&s->lock); + qemu_iovec_reset(&hd_qiov); qemu_iovec_concat(&hd_qiov, qiov, bytes_done, cur_bytes); =20 @@ -2184,7 +2193,7 @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverS= tate *bs, uint64_t offset, * s->cluster_size); if (cluster_data =3D=3D NULL) { ret =3D -ENOMEM; - goto fail; + goto out_unlocked; } } =20 @@ -2199,40 +2208,34 @@ static coroutine_fn int qcow2_co_pwritev(BlockDrive= rState *bs, uint64_t offset, cluster_data, cur_bytes, NULL) < 0) { ret =3D -EIO; - goto fail; + goto out_unlocked; } =20 qemu_iovec_reset(&hd_qiov); qemu_iovec_add(&hd_qiov, cluster_data, cur_bytes); } =20 - ret =3D qcow2_pre_write_overlap_check(bs, 0, - cluster_offset + offset_in_cluster, cur_bytes, true); - if (ret < 0) { - goto fail; - } - /* If we need to do COW, check if it's possible to merge the * writing of the guest data together with that of the COW regions. * If it's not possible (or not necessary) then write the * guest data now. */ if (!merge_cow(offset, cur_bytes, &hd_qiov, l2meta)) { - qemu_co_mutex_unlock(&s->lock); BLKDBG_EVENT(bs->file, BLKDBG_WRITE_AIO); trace_qcow2_writev_data(qemu_coroutine_self(), cluster_offset + offset_in_cluster); ret =3D bdrv_co_pwritev(s->data_file, cluster_offset + offset_in_cluster, cur_bytes, &hd_qiov, 0); - qemu_co_mutex_lock(&s->lock); if (ret < 0) { - goto fail; + goto out_unlocked; } } =20 + qemu_co_mutex_lock(&s->lock); + ret =3D qcow2_handle_l2meta(bs, &l2meta, true); if (ret) { - goto fail; + goto out_locked; } =20 bytes -=3D cur_bytes; @@ -2241,8 +2244,12 @@ static coroutine_fn int qcow2_co_pwritev(BlockDriver= State *bs, uint64_t offset, trace_qcow2_writev_done_part(qemu_coroutine_self(), cur_bytes); } ret =3D 0; + goto out_locked; =20 -fail: +out_unlocked: + qemu_co_mutex_lock(&s->lock); + +out_locked: qcow2_handle_l2meta(bs, &l2meta, false); =20 qemu_co_mutex_unlock(&s->lock); --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072887; cv=none; d=zoho.com; s=zohoarc; b=hpi4SuepoXAGj9AVa1Smh/+grDsMzVqzsMUUHXxjFSZ1Mrwagu3ZrJwBEEZJ22dMIz8kHPwZa95x9JFCVhev8REwRtmZQ4eDbER/8PiSMeiOzTS8g3sPyGYaq2u89/4SFfRd5wOocZqmftG2P4Tcu+/Mzx4ggC/8GX5S0S2APmk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072887; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=PdPI1bCKe6s5Uo6grwcjInPni5Lk4O9RXn/GGEgGX2E=; b=B839OucrzuDqXpCo9vUqLCH8Ys4r+V/v6XuDBm1ootG7nyYbZYsY5oC7oZLtk6CuBJD91SWco350DFOqICF/1zCEDM9kYrL0ixMUhztnuPvYENI1KqUZPfMUnvSo+k7ENTzkwSf4WT4qhDc0030m3tUI/ixcN4L8bkHg7/tynM8= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 155907288764316.52166881913422; Tue, 28 May 2019 12:48:07 -0700 (PDT) Received: from localhost ([127.0.0.1]:41778 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVi4m-0006zS-JY for importer@patchew.org; Tue, 28 May 2019 15:48:00 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37046) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhoB-0001dk-6r for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVho6-00023n-D0 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:4098) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhnX-0000N9-IR; Tue, 28 May 2019 15:30:12 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2AFDAC05D419; Tue, 28 May 2019 19:29:09 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7CD215D6A9; Tue, 28 May 2019 19:29:08 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:34 +0200 Message-Id: <20190528192847.2730-9-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Tue, 28 May 2019 19:29:09 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 08/21] qcow2: do encryption in threads X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Do encryption/decryption in threads, like it is already done for compression. This improves asynchronous encrypted io. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Max Reitz Message-id: 20190506142741.41731-9-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/qcow2.h | 8 ++++++ block/qcow2-cluster.c | 7 ++--- block/qcow2-threads.c | 65 +++++++++++++++++++++++++++++++++++++++++-- block/qcow2.c | 22 +++++---------- 4 files changed, 81 insertions(+), 21 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index 637552e137..e873ae5d12 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -268,6 +268,8 @@ typedef struct Qcow2BitmapHeaderExt { uint64_t bitmap_directory_offset; } QEMU_PACKED Qcow2BitmapHeaderExt; =20 +#define QCOW2_MAX_THREADS 4 + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size; @@ -744,5 +746,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, si= ze_t dest_size, ssize_t coroutine_fn qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, const void *src, size_t src_size); +int coroutine_fn +qcow2_co_encrypt(BlockDriverState *bs, uint64_t file_cluster_offset, + uint64_t offset, void *buf, size_t len); +int coroutine_fn +qcow2_co_decrypt(BlockDriverState *bs, uint64_t file_cluster_offset, + uint64_t offset, void *buf, size_t len); =20 #endif diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index a4c878398e..4a929900cf 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -471,13 +471,12 @@ static bool coroutine_fn do_perform_cow_encrypt(Block= DriverState *bs, { if (bytes && bs->encrypted) { BDRVQcow2State *s =3D bs->opaque; - int64_t offset =3D (s->crypt_physical_offset ? - (cluster_offset + offset_in_cluster) : - (src_cluster_offset + offset_in_cluster)); assert((offset_in_cluster & ~BDRV_SECTOR_MASK) =3D=3D 0); assert((bytes & ~BDRV_SECTOR_MASK) =3D=3D 0); assert(s->crypto); - if (qcrypto_block_encrypt(s->crypto, offset, buffer, bytes, NULL) = < 0) { + if (qcow2_co_encrypt(bs, cluster_offset, + src_cluster_offset + offset_in_cluster, + buffer, bytes) < 0) { return false; } } diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c index 650aa2ed19..3b1e63fe41 100644 --- a/block/qcow2-threads.c +++ b/block/qcow2-threads.c @@ -30,8 +30,7 @@ =20 #include "qcow2.h" #include "block/thread-pool.h" - -#define QCOW2_MAX_THREADS 4 +#include "crypto.h" =20 static int coroutine_fn qcow2_co_process(BlockDriverState *bs, ThreadPoolFunc *func, void *arg) @@ -205,3 +204,65 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, = size_t dest_size, return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, qcow2_decompress); } + + +/* + * Cryptography + */ + +/* + * Qcow2EncDecFunc: common prototype of qcrypto_block_encrypt() and + * qcrypto_block_decrypt() functions. + */ +typedef int (*Qcow2EncDecFunc)(QCryptoBlock *block, uint64_t offset, + uint8_t *buf, size_t len, Error **errp); + +typedef struct Qcow2EncDecData { + QCryptoBlock *block; + uint64_t offset; + uint8_t *buf; + size_t len; + + Qcow2EncDecFunc func; +} Qcow2EncDecData; + +static int qcow2_encdec_pool_func(void *opaque) +{ + Qcow2EncDecData *data =3D opaque; + + return data->func(data->block, data->offset, data->buf, data->len, NUL= L); +} + +static int coroutine_fn +qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset, + uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc = func) +{ + BDRVQcow2State *s =3D bs->opaque; + Qcow2EncDecData arg =3D { + .block =3D s->crypto, + .offset =3D s->crypt_physical_offset ? + file_cluster_offset + offset_into_cluster(s, offset)= : + offset, + .buf =3D buf, + .len =3D len, + .func =3D func, + }; + + return qcow2_co_process(bs, qcow2_encdec_pool_func, &arg); +} + +int coroutine_fn +qcow2_co_encrypt(BlockDriverState *bs, uint64_t file_cluster_offset, + uint64_t offset, void *buf, size_t len) +{ + return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len, + qcrypto_block_encrypt); +} + +int coroutine_fn +qcow2_co_decrypt(BlockDriverState *bs, uint64_t file_cluster_offset, + uint64_t offset, void *buf, size_t len) +{ + return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len, + qcrypto_block_decrypt); +} diff --git a/block/qcow2.c b/block/qcow2.c index 4b4a0ce6e0..dea765b2f4 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -297,7 +297,7 @@ static int qcow2_read_extensions(BlockDriverState *bs, = uint64_t start_offset, } s->crypto =3D qcrypto_block_open(s->crypto_opts, "encrypt.", qcow2_crypto_hdr_read_func, - bs, cflags, 1, errp); + bs, cflags, QCOW2_MAX_THREADS, = errp); if (!s->crypto) { return -EINVAL; } @@ -1538,7 +1538,8 @@ static int coroutine_fn qcow2_do_open(BlockDriverStat= e *bs, QDict *options, cflags |=3D QCRYPTO_BLOCK_OPEN_NO_IO; } s->crypto =3D qcrypto_block_open(s->crypto_opts, "encrypt.", - NULL, NULL, cflags, 1, errp); + NULL, NULL, cflags, + QCOW2_MAX_THREADS, errp); if (!s->crypto) { ret =3D -EINVAL; goto fail; @@ -2061,13 +2062,8 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverS= tate *bs, uint64_t offset, assert(s->crypto); assert((offset & (BDRV_SECTOR_SIZE - 1)) =3D=3D 0); assert((cur_bytes & (BDRV_SECTOR_SIZE - 1)) =3D=3D 0); - if (qcrypto_block_decrypt(s->crypto, - (s->crypt_physical_offset ? - cluster_offset + offset_in_clus= ter : - offset), - cluster_data, - cur_bytes, - NULL) < 0) { + if (qcow2_co_decrypt(bs, cluster_offset, offset, + cluster_data, cur_bytes) < 0) { ret =3D -EIO; goto fail; } @@ -2201,12 +2197,8 @@ static coroutine_fn int qcow2_co_pwritev(BlockDriver= State *bs, uint64_t offset, QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size); qemu_iovec_to_buf(&hd_qiov, 0, cluster_data, hd_qiov.size); =20 - if (qcrypto_block_encrypt(s->crypto, - (s->crypt_physical_offset ? - cluster_offset + offset_in_cluster : - offset), - cluster_data, - cur_bytes, NULL) < 0) { + if (qcow2_co_encrypt(bs, cluster_offset, offset, + cluster_data, cur_bytes) < 0) { ret =3D -EIO; goto out_unlocked; } --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072638; cv=none; d=zoho.com; s=zohoarc; b=Cv7mdBfGU+zp+Tt2aTX/YvDnoNtUt7MvBcEfsVphQOn8LV+YnM/I9qPqgVsdjOPr803JvqeKp0sTdE1e4U2kJxnRTTA72BJPiIbqVPmdb5dICdaj/4Ga9BQicSOZgViszobSyszfsHFSBkFnx+NuywCndPnuF97GioPcGYJqj/A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072638; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=XyHWB4dsfuNVPwM7GipuTm9pf8cviXw5YL9n1spnDJw=; b=V95Rxq2kAQixMdD2XO7YM2cP+1AZp44YNC9DyskeKUNjlRaIzc4AzO6iu0rtHlQA4PZcLrCmOFZmO5oiT1bByfY8dtF7gymdv2V3Am6oZDaajxzykOAy2Y1o0/ubL+NHKzY3813OBcOBCGokgKtSW24nohdpNdManl7N7NjoHBM= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072638789953.3019675604887; Tue, 28 May 2019 12:43:58 -0700 (PDT) Received: from localhost ([127.0.0.1]:41700 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVi0X-0003bj-9U for importer@patchew.org; Tue, 28 May 2019 15:43:37 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37041) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhoA-0001dT-W1 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVho6-00023u-E4 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50982) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhns-0000Po-Ml; Tue, 28 May 2019 15:30:34 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8B8F59FDE7; Tue, 28 May 2019 19:29:11 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 248B560C47; Tue, 28 May 2019 19:29:10 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:35 +0200 Message-Id: <20190528192847.2730-10-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 28 May 2019 19:29:11 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 09/21] block/backup: simplify backup_incremental_init_copy_bitmap X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Simplify backup_incremental_init_copy_bitmap using the function bdrv_dirty_bitmap_next_dirty_area. Note: move to job->len instead of bitmap size: it should not matter but less code. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-id: 20190429090842.57910-2-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/backup.c | 40 ++++++++++++---------------------------- 1 file changed, 12 insertions(+), 28 deletions(-) diff --git a/block/backup.c b/block/backup.c index 916817d8b1..db83b09a0b 100644 --- a/block/backup.c +++ b/block/backup.c @@ -394,43 +394,27 @@ static int coroutine_fn backup_run_incremental(Backup= BlockJob *job) /* init copy_bitmap from sync_bitmap */ static void backup_incremental_init_copy_bitmap(BackupBlockJob *job) { - BdrvDirtyBitmapIter *dbi; - int64_t offset; - int64_t end =3D DIV_ROUND_UP(bdrv_dirty_bitmap_size(job->sync_bitmap), - job->cluster_size); - - dbi =3D bdrv_dirty_iter_new(job->sync_bitmap); - while ((offset =3D bdrv_dirty_iter_next(dbi)) !=3D -1) { - int64_t cluster =3D offset / job->cluster_size; - int64_t next_cluster; - - offset +=3D bdrv_dirty_bitmap_granularity(job->sync_bitmap); - if (offset >=3D bdrv_dirty_bitmap_size(job->sync_bitmap)) { - hbitmap_set(job->copy_bitmap, cluster, end - cluster); - break; - } + uint64_t offset =3D 0; + uint64_t bytes =3D job->len; =20 - offset =3D bdrv_dirty_bitmap_next_zero(job->sync_bitmap, offset, - UINT64_MAX); - if (offset =3D=3D -1) { - hbitmap_set(job->copy_bitmap, cluster, end - cluster); - break; - } + while (bdrv_dirty_bitmap_next_dirty_area(job->sync_bitmap, + &offset, &bytes)) + { + uint64_t cluster =3D offset / job->cluster_size; + uint64_t end_cluster =3D DIV_ROUND_UP(offset + bytes, job->cluster= _size); =20 - next_cluster =3D DIV_ROUND_UP(offset, job->cluster_size); - hbitmap_set(job->copy_bitmap, cluster, next_cluster - cluster); - if (next_cluster >=3D end) { + hbitmap_set(job->copy_bitmap, cluster, end_cluster - cluster); + + offset =3D end_cluster * job->cluster_size; + if (offset >=3D job->len) { break; } - - bdrv_set_dirty_iter(dbi, next_cluster * job->cluster_size); + bytes =3D job->len - offset; } =20 /* TODO job_progress_set_remaining() would make more sense */ job_progress_update(&job->common.job, job->len - hbitmap_count(job->copy_bitmap) * job->cluster_size); - - bdrv_dirty_iter_free(dbi); } =20 static int coroutine_fn backup_run(Job *job, Error **errp) --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072391; cv=none; d=zoho.com; s=zohoarc; b=OrDyOgzcBCJMdJXpiQy3z5F9wwxl2PynduKRHQLUiMlGeJkJecK9tco/CBwSYZukNWugqC5k0buVvBJAm8u7cHgKFG85QrLukPJcLf4ehoTzH0ir5OMIQR6dUrcDjWr4cBYfKRjf02QTqE4euw+5OXRj8UZD0GWndiqGbDBDdsA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072391; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=02TKw/5j5OrgkaWBqbJOvLF5yKNw105AII0Yi65WBvA=; b=glQhV11gkwfqZ57xDFqr8tTxdQJVefeZ3jq8HfDOd+FVRr7lX0EAjqfGJDQnGa1cX+MiZ2tVsgdtTORDPvkPvbSd0qA+pRcvpgl6dOGyDC7oJ8EKxtdIlMZnx/dvEQLfOLNn9WVZSnnHQyEoSx27XcGjuHCIXvvcHURforPL1fI= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 155907239195061.378452300669665; Tue, 28 May 2019 12:39:51 -0700 (PDT) Received: from localhost ([127.0.0.1]:41616 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhwk-00085x-Oh for importer@patchew.org; Tue, 28 May 2019 15:39:42 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37048) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhoB-0001dm-6y for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVho6-000243-E1 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40950) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhnr-0000ZG-Ty; Tue, 28 May 2019 15:30:32 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2ABFD8764B; Tue, 28 May 2019 19:29:14 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 89EBD60C47; Tue, 28 May 2019 19:29:13 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:36 +0200 Message-Id: <20190528192847.2730-11-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Tue, 28 May 2019 19:29:14 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 10/21] block/backup: move to copy_bitmap with granularity X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy We are going to share this bitmap between backup and backup-top filter driver, so let's share something more meaningful. It also simplifies some calculations. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-id: 20190429090842.57910-3-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/backup.c | 48 +++++++++++++++++++++++------------------------- 1 file changed, 23 insertions(+), 25 deletions(-) diff --git a/block/backup.c b/block/backup.c index db83b09a0b..98a2d2b070 100644 --- a/block/backup.c +++ b/block/backup.c @@ -112,7 +112,8 @@ static int coroutine_fn backup_cow_with_bounce_buffer(B= ackupBlockJob *job, int read_flags =3D is_write_notifier ? BDRV_REQ_NO_SERIALISING : 0; int write_flags =3D job->serialize_target_writes ? BDRV_REQ_SERIALISIN= G : 0; =20 - hbitmap_reset(job->copy_bitmap, start / job->cluster_size, 1); + assert(QEMU_IS_ALIGNED(start, job->cluster_size)); + hbitmap_reset(job->copy_bitmap, start, job->cluster_size); nbytes =3D MIN(job->cluster_size, job->len - start); if (!*bounce_buffer) { *bounce_buffer =3D blk_blockalign(blk, job->cluster_size); @@ -145,7 +146,7 @@ static int coroutine_fn backup_cow_with_bounce_buffer(B= ackupBlockJob *job, =20 return nbytes; fail: - hbitmap_set(job->copy_bitmap, start / job->cluster_size, 1); + hbitmap_set(job->copy_bitmap, start, job->cluster_size); return ret; =20 } @@ -165,16 +166,15 @@ static int coroutine_fn backup_cow_with_offload(Backu= pBlockJob *job, int write_flags =3D job->serialize_target_writes ? BDRV_REQ_SERIALISIN= G : 0; =20 assert(QEMU_IS_ALIGNED(job->copy_range_size, job->cluster_size)); + assert(QEMU_IS_ALIGNED(start, job->cluster_size)); nbytes =3D MIN(job->copy_range_size, end - start); nr_clusters =3D DIV_ROUND_UP(nbytes, job->cluster_size); - hbitmap_reset(job->copy_bitmap, start / job->cluster_size, - nr_clusters); + hbitmap_reset(job->copy_bitmap, start, job->cluster_size * nr_clusters= ); ret =3D blk_co_copy_range(blk, start, job->target, start, nbytes, read_flags, write_flags); if (ret < 0) { trace_backup_do_cow_copy_range_fail(job, start, ret); - hbitmap_set(job->copy_bitmap, start / job->cluster_size, - nr_clusters); + hbitmap_set(job->copy_bitmap, start, job->cluster_size * nr_cluste= rs); return ret; } =20 @@ -202,7 +202,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *j= ob, cow_request_begin(&cow_request, job, start, end); =20 while (start < end) { - if (!hbitmap_get(job->copy_bitmap, start / job->cluster_size)) { + if (!hbitmap_get(job->copy_bitmap, start)) { trace_backup_do_cow_skip(job, start); start +=3D job->cluster_size; continue; /* already copied */ @@ -298,12 +298,16 @@ static void backup_clean(Job *job) assert(s->target); blk_unref(s->target); s->target =3D NULL; + + if (s->copy_bitmap) { + hbitmap_free(s->copy_bitmap); + s->copy_bitmap =3D NULL; + } } =20 void backup_do_checkpoint(BlockJob *job, Error **errp) { BackupBlockJob *backup_job =3D container_of(job, BackupBlockJob, commo= n); - int64_t len; =20 assert(block_job_driver(job) =3D=3D &backup_job_driver); =20 @@ -313,8 +317,7 @@ void backup_do_checkpoint(BlockJob *job, Error **errp) return; } =20 - len =3D DIV_ROUND_UP(backup_job->len, backup_job->cluster_size); - hbitmap_set(backup_job->copy_bitmap, 0, len); + hbitmap_set(backup_job->copy_bitmap, 0, backup_job->len); } =20 static void backup_drain(BlockJob *job) @@ -369,16 +372,16 @@ static int coroutine_fn backup_run_incremental(Backup= BlockJob *job) { int ret; bool error_is_read; - int64_t cluster; + int64_t offset; HBitmapIter hbi; =20 hbitmap_iter_init(&hbi, job->copy_bitmap, 0); - while ((cluster =3D hbitmap_iter_next(&hbi)) !=3D -1) { + while ((offset =3D hbitmap_iter_next(&hbi)) !=3D -1) { do { if (yield_and_check(job)) { return 0; } - ret =3D backup_do_cow(job, cluster * job->cluster_size, + ret =3D backup_do_cow(job, offset, job->cluster_size, &error_is_read, false); if (ret < 0 && backup_error_action(job, error_is_read, -ret) = =3D=3D BLOCK_ERROR_ACTION_REPORT) @@ -400,12 +403,9 @@ static void backup_incremental_init_copy_bitmap(Backup= BlockJob *job) while (bdrv_dirty_bitmap_next_dirty_area(job->sync_bitmap, &offset, &bytes)) { - uint64_t cluster =3D offset / job->cluster_size; - uint64_t end_cluster =3D DIV_ROUND_UP(offset + bytes, job->cluster= _size); + hbitmap_set(job->copy_bitmap, offset, bytes); =20 - hbitmap_set(job->copy_bitmap, cluster, end_cluster - cluster); - - offset =3D end_cluster * job->cluster_size; + offset +=3D bytes; if (offset >=3D job->len) { break; } @@ -414,30 +414,27 @@ static void backup_incremental_init_copy_bitmap(Backu= pBlockJob *job) =20 /* TODO job_progress_set_remaining() would make more sense */ job_progress_update(&job->common.job, - job->len - hbitmap_count(job->copy_bitmap) * job->cluster_size); + job->len - hbitmap_count(job->copy_bitmap)); } =20 static int coroutine_fn backup_run(Job *job, Error **errp) { BackupBlockJob *s =3D container_of(job, BackupBlockJob, common.job); BlockDriverState *bs =3D blk_bs(s->common.blk); - int64_t offset, nb_clusters; + int64_t offset; int ret =3D 0; =20 QLIST_INIT(&s->inflight_reqs); qemu_co_rwlock_init(&s->flush_rwlock); =20 - nb_clusters =3D DIV_ROUND_UP(s->len, s->cluster_size); job_progress_set_remaining(job, s->len); =20 - s->copy_bitmap =3D hbitmap_alloc(nb_clusters, 0); if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_INCREMENTAL) { backup_incremental_init_copy_bitmap(s); } else { - hbitmap_set(s->copy_bitmap, 0, nb_clusters); + hbitmap_set(s->copy_bitmap, 0, s->len); } =20 - s->before_write.notify =3D backup_before_write_notify; bdrv_add_before_write_notifier(bs, &s->before_write); =20 @@ -518,7 +515,6 @@ static int coroutine_fn backup_run(Job *job, Error **er= rp) /* wait until pending backup_do_cow() calls have completed */ qemu_co_rwlock_wrlock(&s->flush_rwlock); qemu_co_rwlock_unlock(&s->flush_rwlock); - hbitmap_free(s->copy_bitmap); =20 return ret; } @@ -668,6 +664,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDr= iverState *bs, } else { job->cluster_size =3D MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster= _size); } + + job->copy_bitmap =3D hbitmap_alloc(len, ctz32(job->cluster_size)); job->use_copy_range =3D true; job->copy_range_size =3D MIN_NON_ZERO(blk_get_max_transfer(job->common= .blk), blk_get_max_transfer(job->target)); --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072563; cv=none; d=zoho.com; s=zohoarc; b=aD/aUCtlg2pLfXEPgPCUCdswtCXEL4u4d3UtfTF8avxQO9/bSkp1AZE1yrHVYzJ+iwS56zpwtqG5cSmJkwPDjIePp18aVBqWaC1SwKcfLP9ZTJo/Psbw/nwxrPV3+Jv6k5Yv3aitNZbLABAwuz5awdf48TtuXKFzhd4mkqFaMeg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072563; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=fxo1lh3kSRFOld43puExXE4u9w+yc7vcozyudQqFaxI=; b=oaXgcD7vPOxXDWFblk9T5NDG38R92qypnxulDJs6hRHIYjXAcQHvbr26RMIn3MI/0/aaAM0dltEqPi0SOD3CJd6BsbZlblrA309hi0Rw8MxarfRAgYMQwt5M1Si9699lWSqkf94+tgYNCrCCJdj0M00W19pWvy3ekxcm/6FFZ54= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072563140738.5199934384805; Tue, 28 May 2019 12:42:43 -0700 (PDT) Received: from localhost ([127.0.0.1]:41681 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhzb-0002pm-37 for importer@patchew.org; Tue, 28 May 2019 15:42:39 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37269) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhow-0002JH-OI for qemu-devel@nongnu.org; Tue, 28 May 2019 15:31:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhou-0002Yb-PY for qemu-devel@nongnu.org; Tue, 28 May 2019 15:31:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:42214) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhof-0000py-Tt; Tue, 28 May 2019 15:31:24 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BC7F5307E04E; Tue, 28 May 2019 19:29:17 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 52B855F9A6; Tue, 28 May 2019 19:29:17 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:37 +0200 Message-Id: <20190528192847.2730-12-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Tue, 28 May 2019 19:29:22 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 11/21] block/backup: refactor and tolerate unallocated cluster skipping X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Split allocation checking to separate function and reduce nesting. Consider bdrv_is_allocated() fail as allocated area, as copying more than needed is not wrong (and we do it anyway) and seems better than fail the whole job. And, most probably we will fail on the next read, if there are real problem with source. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-id: 20190429090842.57910-4-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/backup.c | 60 +++++++++++++++++++------------------------------- 1 file changed, 23 insertions(+), 37 deletions(-) diff --git a/block/backup.c b/block/backup.c index 98a2d2b070..78f1b79354 100644 --- a/block/backup.c +++ b/block/backup.c @@ -368,6 +368,22 @@ static bool coroutine_fn yield_and_check(BackupBlockJo= b *job) return false; } =20 +static bool bdrv_is_unallocated_range(BlockDriverState *bs, + int64_t offset, int64_t bytes) +{ + int64_t end =3D offset + bytes; + + while (offset < end && !bdrv_is_allocated(bs, offset, bytes, &bytes)) { + if (bytes =3D=3D 0) { + return true; + } + offset +=3D bytes; + bytes =3D end - offset; + } + + return offset >=3D end; +} + static int coroutine_fn backup_run_incremental(BackupBlockJob *job) { int ret; @@ -453,49 +469,19 @@ static int coroutine_fn backup_run(Job *job, Error **= errp) for (offset =3D 0; offset < s->len; offset +=3D s->cluster_size) { bool error_is_read; - int alloced =3D 0; =20 if (yield_and_check(s)) { break; } =20 - if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_TOP) { - int i; - int64_t n; - - /* Check to see if these blocks are already in the - * backing file. */ - - for (i =3D 0; i < s->cluster_size;) { - /* bdrv_is_allocated() only returns true/false based - * on the first set of sectors it comes across that - * are are all in the same state. - * For that reason we must verify each sector in the - * backup cluster length. We end up copying more than - * needed but at some point that is always the case. */ - alloced =3D - bdrv_is_allocated(bs, offset + i, - s->cluster_size - i, &n); - i +=3D n; - - if (alloced || n =3D=3D 0) { - break; - } - } - - /* If the above loop never found any sectors that are in - * the topmost image, skip this backup. */ - if (alloced =3D=3D 0) { - continue; - } - } - /* FULL sync mode we copy the whole drive. */ - if (alloced < 0) { - ret =3D alloced; - } else { - ret =3D backup_do_cow(s, offset, s->cluster_size, - &error_is_read, false); + if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_TOP && + bdrv_is_unallocated_range(bs, offset, s->cluster_size)) + { + continue; } + + ret =3D backup_do_cow(s, offset, s->cluster_size, + &error_is_read, false); if (ret < 0) { /* Depending on error action, fail now or retry cluster */ BlockErrorAction action =3D --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559073131; cv=none; d=zoho.com; s=zohoarc; b=HOT0bNfGTI2yDrXtnL0ujzdDek/r9fgknSVQuiUogkmvnVfpp7Qd1HLePTTwZoUqYdQWIvoenmf1CNnYdvnr+5NSxAjrmxYX2S2cH+LN3HQNz0sXI+3aZ4lX7x5Vwas2UA3Gt+PfMYS+65dHLe09NMbbBxeZmEt2h9SADfpm1xw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559073131; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=D17AWHu66xpYOHtcWCLd29lCPXQQLzDuvDG5TstlwdA=; b=myFZw0Rsa5OWwuuLawSFfQGebBDFN9JoUzida6aIXdizRy1aZkfnH2j2tePR7y1hXm6HxkcRJ5nkN1E0L7hWs3tbq8VVTZn8ZInHNbxqpcCvN4aiYKyzKM7hBBrRWTq2uVePbjueQ5Ug33tcmLh8uM5d65FJbi0smk+IaXZPJcA= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 155907313143463.86013398695354; Tue, 28 May 2019 12:52:11 -0700 (PDT) Received: from localhost ([127.0.0.1]:41842 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVi8Z-0001Rf-N9 for importer@patchew.org; Tue, 28 May 2019 15:51:55 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37587) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhpN-0002bS-AO for qemu-devel@nongnu.org; Tue, 28 May 2019 15:32:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhpM-0002sp-2Z for qemu-devel@nongnu.org; Tue, 28 May 2019 15:32:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:42154) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhpH-0000ic-1X; Tue, 28 May 2019 15:31:59 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4D8FE307E040; Tue, 28 May 2019 19:29:22 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D825C1001DD7; Tue, 28 May 2019 19:29:19 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:38 +0200 Message-Id: <20190528192847.2730-13-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Tue, 28 May 2019 19:29:27 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 12/21] block/backup: unify different modes code path X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Do full, top and incremental mode copying all in one place. This unifies the code path and helps further improvements. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-id: 20190429090842.57910-5-vsementsov@virtuozzo.com Signed-off-by: Max Reitz --- block/backup.c | 43 ++++++++++--------------------------------- 1 file changed, 10 insertions(+), 33 deletions(-) diff --git a/block/backup.c b/block/backup.c index 78f1b79354..5b3fc9d123 100644 --- a/block/backup.c +++ b/block/backup.c @@ -384,15 +384,23 @@ static bool bdrv_is_unallocated_range(BlockDriverStat= e *bs, return offset >=3D end; } =20 -static int coroutine_fn backup_run_incremental(BackupBlockJob *job) +static int coroutine_fn backup_loop(BackupBlockJob *job) { int ret; bool error_is_read; int64_t offset; HBitmapIter hbi; + BlockDriverState *bs =3D blk_bs(job->common.blk); =20 hbitmap_iter_init(&hbi, job->copy_bitmap, 0); while ((offset =3D hbitmap_iter_next(&hbi)) !=3D -1) { + if (job->sync_mode =3D=3D MIRROR_SYNC_MODE_TOP && + bdrv_is_unallocated_range(bs, offset, job->cluster_size)) + { + hbitmap_reset(job->copy_bitmap, offset, job->cluster_size); + continue; + } + do { if (yield_and_check(job)) { return 0; @@ -437,7 +445,6 @@ static int coroutine_fn backup_run(Job *job, Error **er= rp) { BackupBlockJob *s =3D container_of(job, BackupBlockJob, common.job); BlockDriverState *bs =3D blk_bs(s->common.blk); - int64_t offset; int ret =3D 0; =20 QLIST_INIT(&s->inflight_reqs); @@ -462,38 +469,8 @@ static int coroutine_fn backup_run(Job *job, Error **e= rrp) * notify callback service CoW requests. */ job_yield(job); } - } else if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_INCREMENTAL) { - ret =3D backup_run_incremental(s); } else { - /* Both FULL and TOP SYNC_MODE's require copying.. */ - for (offset =3D 0; offset < s->len; - offset +=3D s->cluster_size) { - bool error_is_read; - - if (yield_and_check(s)) { - break; - } - - if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_TOP && - bdrv_is_unallocated_range(bs, offset, s->cluster_size)) - { - continue; - } - - ret =3D backup_do_cow(s, offset, s->cluster_size, - &error_is_read, false); - if (ret < 0) { - /* Depending on error action, fail now or retry cluster */ - BlockErrorAction action =3D - backup_error_action(s, error_is_read, -ret); - if (action =3D=3D BLOCK_ERROR_ACTION_REPORT) { - break; - } else { - offset -=3D s->cluster_size; - continue; - } - } - } + ret =3D backup_loop(s); } =20 notifier_with_return_remove(&s->before_write); --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072406; cv=none; d=zoho.com; s=zohoarc; b=h9+ZwVhFMUzGHgQ3T1TWT/WUU8kq5VLHNBzGyNY7nI6HIhK8I2bG8QwpBJXHz4fbSo4X+7YBiNWsQVjY/QZNC/cKtLK0xA5oGVHYlP8mTa5XMkeNaKpN0cgsutTMlRYa9NAP9zKX0TVNOYAch/iZ4/mJQ+MmOs2vChh6De2dhuo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072406; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=xV6t+wnBkIqxPashmNPD1BHLT7RmcGiR5TO6blZJzEY=; b=HjM99fWM2iqC9ilf3nxRl/dHUqo5t9D2XaxdXtk88QPBqPpbNPT/0HQWe20V/K+yCr1JwGPTmGLnGslcp+zJeAgt0gLGWkMMXQmAh4oyaNitrRiOtMxIraQEJ/vvH5cOVn8r5MyLZ6PCCu9aiogjrrZn1W3VdFJh4x4PdZynKr4= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072406240319.96957640829055; Tue, 28 May 2019 12:40:06 -0700 (PDT) Received: from localhost ([127.0.0.1]:41622 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhx1-0008L8-4k for importer@patchew.org; Tue, 28 May 2019 15:39:59 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36876) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhnj-0001LL-9L for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhnY-0001Vm-2c for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53586) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhnR-0000vM-Uj; Tue, 28 May 2019 15:30:06 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B1F5981E1D; Tue, 28 May 2019 19:29:24 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4670D5C8A3; Tue, 28 May 2019 19:29:24 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:39 +0200 Message-Id: <20190528192847.2730-14-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 28 May 2019 19:29:27 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 13/21] block/backup: refactor: split out backup_calculate_cluster_size X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Split out cluster_size calculation. Move copy-bitmap creation above block-job creation, as we are going to share it with upcoming backup-top filter, which also should be created before actual block job creation. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-id: 20190429090842.57910-6-vsementsov@virtuozzo.com [mreitz: Dropped a paragraph from the commit message that was left over from a previous version] Signed-off-by: Max Reitz --- block/backup.c | 82 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 52 insertions(+), 30 deletions(-) diff --git a/block/backup.c b/block/backup.c index 5b3fc9d123..00f4f8af53 100644 --- a/block/backup.c +++ b/block/backup.c @@ -497,6 +497,42 @@ static const BlockJobDriver backup_job_driver =3D { .drain =3D backup_drain, }; =20 +static int64_t backup_calculate_cluster_size(BlockDriverState *target, + Error **errp) +{ + int ret; + BlockDriverInfo bdi; + + /* + * If there is no backing file on the target, we cannot rely on COW if= our + * backup cluster size is smaller than the target cluster size. Even f= or + * targets with a backing file, try to avoid COW if possible. + */ + ret =3D bdrv_get_info(target, &bdi); + if (ret =3D=3D -ENOTSUP && !target->backing) { + /* Cluster size is not defined */ + warn_report("The target block device doesn't provide " + "information about the block size and it doesn't have = a " + "backing file. The default block size of %u bytes is " + "used. If the actual block size of the target exceeds " + "this default, the backup may be unusable", + BACKUP_CLUSTER_SIZE_DEFAULT); + return BACKUP_CLUSTER_SIZE_DEFAULT; + } else if (ret < 0 && !target->backing) { + error_setg_errno(errp, -ret, + "Couldn't determine the cluster size of the target image, " + "which has no backing file"); + error_append_hint(errp, + "Aborting, since this may create an unusable destination image= \n"); + return ret; + } else if (ret < 0 && target->backing) { + /* Not fatal; just trudge on ahead. */ + return BACKUP_CLUSTER_SIZE_DEFAULT; + } + + return MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster_size); +} + BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, BlockDriverState *target, int64_t speed, MirrorSyncMode sync_mode, BdrvDirtyBitmap *sync_bitmap, @@ -508,9 +544,10 @@ BlockJob *backup_job_create(const char *job_id, BlockD= riverState *bs, JobTxn *txn, Error **errp) { int64_t len; - BlockDriverInfo bdi; BackupBlockJob *job =3D NULL; int ret; + int64_t cluster_size; + HBitmap *copy_bitmap =3D NULL; =20 assert(bs); assert(target); @@ -572,6 +609,13 @@ BlockJob *backup_job_create(const char *job_id, BlockD= riverState *bs, goto error; } =20 + cluster_size =3D backup_calculate_cluster_size(target, errp); + if (cluster_size < 0) { + goto error; + } + + copy_bitmap =3D hbitmap_alloc(len, ctz32(cluster_size)); + /* job->len is fixed, so we can't allow resize */ job =3D block_job_create(job_id, &backup_job_driver, txn, bs, BLK_PERM_CONSISTENT_READ, @@ -600,35 +644,9 @@ BlockJob *backup_job_create(const char *job_id, BlockD= riverState *bs, =20 /* Detect image-fleecing (and similar) schemes */ job->serialize_target_writes =3D bdrv_chain_contains(target, bs); - - /* If there is no backing file on the target, we cannot rely on COW if= our - * backup cluster size is smaller than the target cluster size. Even f= or - * targets with a backing file, try to avoid COW if possible. */ - ret =3D bdrv_get_info(target, &bdi); - if (ret =3D=3D -ENOTSUP && !target->backing) { - /* Cluster size is not defined */ - warn_report("The target block device doesn't provide " - "information about the block size and it doesn't have = a " - "backing file. The default block size of %u bytes is " - "used. If the actual block size of the target exceeds " - "this default, the backup may be unusable", - BACKUP_CLUSTER_SIZE_DEFAULT); - job->cluster_size =3D BACKUP_CLUSTER_SIZE_DEFAULT; - } else if (ret < 0 && !target->backing) { - error_setg_errno(errp, -ret, - "Couldn't determine the cluster size of the target image, " - "which has no backing file"); - error_append_hint(errp, - "Aborting, since this may create an unusable destination image= \n"); - goto error; - } else if (ret < 0 && target->backing) { - /* Not fatal; just trudge on ahead. */ - job->cluster_size =3D BACKUP_CLUSTER_SIZE_DEFAULT; - } else { - job->cluster_size =3D MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster= _size); - } - - job->copy_bitmap =3D hbitmap_alloc(len, ctz32(job->cluster_size)); + job->cluster_size =3D cluster_size; + job->copy_bitmap =3D copy_bitmap; + copy_bitmap =3D NULL; job->use_copy_range =3D true; job->copy_range_size =3D MIN_NON_ZERO(blk_get_max_transfer(job->common= .blk), blk_get_max_transfer(job->target)); @@ -644,6 +662,10 @@ BlockJob *backup_job_create(const char *job_id, BlockD= riverState *bs, return &job->common; =20 error: + if (copy_bitmap) { + assert(!job || !job->copy_bitmap); + hbitmap_free(copy_bitmap); + } if (sync_bitmap) { bdrv_reclaim_dirty_bitmap(bs, sync_bitmap, NULL); } --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072196; cv=none; d=zoho.com; s=zohoarc; b=gAyURsBMCqBi3XnhM+E1/S7eQBYd3cGZQzUxWnSI1Vc6wL/4bLYQHNLsSo7t+VuhG93Iiojc0QtPv6b6GhdMLpHK6au0ZGH1tHBjsjOAONUpYVvOContwY9tO7+poGdqgLBUIIQAv9h+QmMA67EjAk/KYzxywzLSQtxrfO3nMuo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072196; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=CfIY0frMGy8gxvVIqXeGVwCrD7rRjzom0xaue1xriyg=; b=DJhWQH7VFjOXdWAhcP9OBe+Rv8mbqHSZfJErAHo+wV5ScKImPn46argkg/S5ade/VysWZ9BFw/caNQmv2POVcsixymVkaqybHKcSPFXbJpxVhgsYXwVs1eSeKCZhTxdLlGpVJOjt1hSMNh2K9iURHFS+PIWyNbdJWOT4r/CMvXA= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072196221907.61273654433; Tue, 28 May 2019 12:36:36 -0700 (PDT) Received: from localhost ([127.0.0.1]:41585 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhtd-0005Ps-5L for importer@patchew.org; Tue, 28 May 2019 15:36:29 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36945) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhnz-0001Wm-3m for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhnw-0001v7-02 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40378) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhnY-000126-2G; Tue, 28 May 2019 15:30:19 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1FD4A2E97DA; Tue, 28 May 2019 19:29:28 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id ACAB51972A; Tue, 28 May 2019 19:29:26 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:40 +0200 Message-Id: <20190528192847.2730-15-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 28 May 2019 19:29:38 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 14/21] block: Use bdrv_unref_child() for all children in bdrv_close() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Alberto Garcia bdrv_unref_child() does the following things: - Updates the child->bs->inherits_from pointer. - Calls bdrv_detach_child() to remove the BdrvChild from bs->children. - Calls bdrv_unref() to unref the child BlockDriverState. When bdrv_unref_child() was introduced in commit 33a604075c it was not used in bdrv_close() because the drivers that had additional children (like quorum or blkverify) had already called bdrv_unref() on their children during their own close functions. This was changed later (in 0bd6e91a7e for quorum, in 3e586be0b2 for blkverify) so there's no reason not to use bdrv_unref_child() in bdrv_close() anymore. After this there's also no need to remove bs->backing and bs->file separately from the rest of the children, so bdrv_close() can be simplified. Now bdrv_close() unrefs all children (before this patch it was only bs->file and bs->backing). As a result, none of the callers of brvd_attach_child() should remove their reference to child_bs (because this function effectively steals that reference). This patch updates a couple of tests that were doing their own bdrv_unref(). Signed-off-by: Alberto Garcia Message-id: 6d1d5feaa53aa1ab127adb73d605dc4503e3abd5.1557754872.git.berto@i= galia.com [mreitz: s/where/were/] Signed-off-by: Max Reitz --- block.c | 16 +++------------- tests/test-bdrv-drain.c | 6 ------ tests/test-bdrv-graph-mod.c | 1 - 3 files changed, 3 insertions(+), 20 deletions(-) diff --git a/block.c b/block.c index cb11537029..be37280dc7 100644 --- a/block.c +++ b/block.c @@ -3877,22 +3877,12 @@ static void bdrv_close(BlockDriverState *bs) bs->drv =3D NULL; } =20 - bdrv_set_backing_hd(bs, NULL, &error_abort); - - if (bs->file !=3D NULL) { - bdrv_unref_child(bs, bs->file); - bs->file =3D NULL; - } - QLIST_FOREACH_SAFE(child, &bs->children, next, next) { - /* TODO Remove bdrv_unref() from drivers' close function and use - * bdrv_unref_child() here */ - if (child->bs->inherits_from =3D=3D bs) { - child->bs->inherits_from =3D NULL; - } - bdrv_detach_child(child); + bdrv_unref_child(bs, child); } =20 + bs->backing =3D NULL; + bs->file =3D NULL; g_free(bs->opaque); bs->opaque =3D NULL; atomic_set(&bs->copy_on_read, 0); diff --git a/tests/test-bdrv-drain.c b/tests/test-bdrv-drain.c index eda90750eb..5534c2adf9 100644 --- a/tests/test-bdrv-drain.c +++ b/tests/test-bdrv-drain.c @@ -1436,12 +1436,6 @@ static void test_detach_indirect(bool by_parent_cb) bdrv_unref(parent_b); blk_unref(blk); =20 - /* XXX Once bdrv_close() unref's children instead of just detaching th= em, - * this won't be necessary any more. */ - bdrv_unref(a); - bdrv_unref(a); - bdrv_unref(c); - g_assert_cmpint(a->refcnt, =3D=3D, 1); g_assert_cmpint(b->refcnt, =3D=3D, 1); g_assert_cmpint(c->refcnt, =3D=3D, 1); diff --git a/tests/test-bdrv-graph-mod.c b/tests/test-bdrv-graph-mod.c index 283dc84869..747c0bf8fc 100644 --- a/tests/test-bdrv-graph-mod.c +++ b/tests/test-bdrv-graph-mod.c @@ -116,7 +116,6 @@ static void test_update_perm_tree(void) g_assert_nonnull(local_err); error_free(local_err); =20 - bdrv_unref(bs); blk_unref(root); } =20 --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072704; cv=none; d=zoho.com; s=zohoarc; b=CMYcOi7XAogWAeWS581QYbWLCEiuGCIcZorCPOo6QX2MAtJSg0tuVBcLMQFCNZ5kNM1sgQNwzDrslznmpLezNNrRLeoOQ3V6R6TXraQ5GYfogLca5iUsdZ97YKJBodJ2QwxdeunJ3UEw2pyyFGjwbJl4DSlVkWOFamrPr8djn7Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072704; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=0dmrhltqSVAMQEkEeBSTP5O3cH2v3e37qANMfOCJHAc=; b=j/38y/8NjdqReUMn9n/6LsXQznAQF6OeRZjPuVPuajcoPjzdDt95ylPpFTOdVQvDsWJGDwcawi+CNyFxZy4G993D2zI25+06Nmq5/pMMpgcjrPbfTpeayQW90BtlaSH0JMr1hvuxy6Qq20LquP7JW9H41fuOLgH0FOFOr3c85o0= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072704101937.6277115152963; Tue, 28 May 2019 12:45:04 -0700 (PDT) Received: from localhost ([127.0.0.1]:41716 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVi1m-0004bo-UB for importer@patchew.org; Tue, 28 May 2019 15:44:54 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37047) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhoB-0001dl-7D for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVho6-000244-E4 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36580) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhns-0000vy-0z; Tue, 28 May 2019 15:30:33 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7BC267E424; Tue, 28 May 2019 19:29:31 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1363D10027C2; Tue, 28 May 2019 19:29:29 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:41 +0200 Message-Id: <20190528192847.2730-16-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 28 May 2019 19:29:37 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 15/21] block: Make bdrv_root_attach_child() unref child_bs on failure X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Alberto Garcia A consequence of the previous patch is that bdrv_attach_child() transfers the reference to child_bs from the caller to parent_bs, which will drop it on bdrv_close() or when someone calls bdrv_unref_child(). But this only happens when bdrv_attach_child() succeeds. If it fails then the caller is responsible for dropping the reference to child_bs. This patch makes bdrv_attach_child() take the reference also when there is an error, freeing the caller for having to do it. A similar situation happens with bdrv_root_attach_child(), so the changes on this patch affect both functions. Signed-off-by: Alberto Garcia Message-id: 20dfb3d9ccec559cdd1a9690146abad5d204a186.1557754872.git.berto@i= galia.com [mreitz: Removed now superfluous BdrvChild * variable in bdrv_open_child()] Signed-off-by: Max Reitz --- block.c | 30 ++++++++++++++++++------------ block/block-backend.c | 3 +-- block/quorum.c | 1 - blockjob.c | 2 +- 4 files changed, 20 insertions(+), 16 deletions(-) diff --git a/block.c b/block.c index be37280dc7..1a73e310c1 100644 --- a/block.c +++ b/block.c @@ -2243,6 +2243,13 @@ static void bdrv_replace_child(BdrvChild *child, Blo= ckDriverState *new_bs) } } =20 +/* + * This function steals the reference to child_bs from the caller. + * That reference is later dropped by bdrv_root_unref_child(). + * + * On failure NULL is returned, errp is set and the reference to + * child_bs is also dropped. + */ BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs, const char *child_name, const BdrvChildRole *child_role, @@ -2255,6 +2262,7 @@ BdrvChild *bdrv_root_attach_child(BlockDriverState *c= hild_bs, ret =3D bdrv_check_update_perm(child_bs, NULL, perm, shared_perm, NULL= , errp); if (ret < 0) { bdrv_abort_perm_update(child_bs); + bdrv_unref(child_bs); return NULL; } =20 @@ -2274,6 +2282,14 @@ BdrvChild *bdrv_root_attach_child(BlockDriverState *= child_bs, return child; } =20 +/* + * This function transfers the reference to child_bs from the caller + * to parent_bs. That reference is later dropped by parent_bs on + * bdrv_close() or if someone calls bdrv_unref_child(). + * + * On failure NULL is returned, errp is set and the reference to + * child_bs is also dropped. + */ BdrvChild *bdrv_attach_child(BlockDriverState *parent_bs, BlockDriverState *child_bs, const char *child_name, @@ -2401,12 +2417,9 @@ void bdrv_set_backing_hd(BlockDriverState *bs, Block= DriverState *backing_hd, /* If backing_hd was already part of bs's backing chain, and * inherits_from pointed recursively to bs then let's update it to * point directly to bs (else it will become NULL). */ - if (update_inherits_from) { + if (bs->backing && update_inherits_from) { backing_hd->inherits_from =3D bs; } - if (!bs->backing) { - bdrv_unref(backing_hd); - } =20 out: bdrv_refresh_limits(bs, NULL); @@ -2594,7 +2607,6 @@ BdrvChild *bdrv_open_child(const char *filename, const BdrvChildRole *child_role, bool allow_none, Error **errp) { - BdrvChild *c; BlockDriverState *bs; =20 bs =3D bdrv_open_child_bs(filename, options, bdref_key, parent, child_= role, @@ -2603,13 +2615,7 @@ BdrvChild *bdrv_open_child(const char *filename, return NULL; } =20 - c =3D bdrv_attach_child(parent, bs, bdref_key, child_role, errp); - if (!c) { - bdrv_unref(bs); - return NULL; - } - - return c; + return bdrv_attach_child(parent, bs, bdref_key, child_role, errp); } =20 /* TODO Future callers may need to specify parent/child_role in order for diff --git a/block/block-backend.c b/block/block-backend.c index 4c0a8ef88d..ad3e1c882d 100644 --- a/block/block-backend.c +++ b/block/block-backend.c @@ -392,7 +392,6 @@ BlockBackend *blk_new_open(const char *filename, const = char *reference, blk->root =3D bdrv_root_attach_child(bs, "root", &child_root, perm, BLK_PERM_ALL, blk, errp); if (!blk->root) { - bdrv_unref(bs); blk_unref(blk); return NULL; } @@ -800,12 +799,12 @@ void blk_remove_bs(BlockBackend *blk) int blk_insert_bs(BlockBackend *blk, BlockDriverState *bs, Error **errp) { ThrottleGroupMember *tgm =3D &blk->public.throttle_group_member; + bdrv_ref(bs); blk->root =3D bdrv_root_attach_child(bs, "root", &child_root, blk->perm, blk->shared_perm, blk, e= rrp); if (blk->root =3D=3D NULL) { return -EPERM; } - bdrv_ref(bs); =20 notifier_list_notify(&blk->insert_bs_notifiers, blk); if (tgm->throttle_state) { diff --git a/block/quorum.c b/block/quorum.c index 352f729136..133ee18204 100644 --- a/block/quorum.c +++ b/block/quorum.c @@ -1019,7 +1019,6 @@ static void quorum_add_child(BlockDriverState *bs, Bl= ockDriverState *child_bs, child =3D bdrv_attach_child(bs, child_bs, indexstr, &child_format, err= p); if (child =3D=3D NULL) { s->next_child_index--; - bdrv_unref(child_bs); goto out; } s->children =3D g_renew(BdrvChild *, s->children, s->num_children + 1); diff --git a/blockjob.c b/blockjob.c index 9ca942ba01..cc5f18e7cd 100644 --- a/blockjob.c +++ b/blockjob.c @@ -204,6 +204,7 @@ int block_job_add_bdrv(BlockJob *job, const char *name,= BlockDriverState *bs, { BdrvChild *c; =20 + bdrv_ref(bs); c =3D bdrv_root_attach_child(bs, name, &child_job, perm, shared_perm, job, errp); if (c =3D=3D NULL) { @@ -211,7 +212,6 @@ int block_job_add_bdrv(BlockJob *job, const char *name,= BlockDriverState *bs, } =20 job->nodes =3D g_slist_prepend(job->nodes, c); - bdrv_ref(bs); bdrv_op_block_all(bs, job->blocker); =20 return 0; --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072497; cv=none; d=zoho.com; s=zohoarc; b=DhVnKz0g4BuCi7COmTyLuxJz8MaVx3dgv+J5PUV+622A/hubLR/p/Vu7hgmEILtwMnVoPJEAEREoFJi+9W6iImmM1zK2xZ7PQ5N2mh/4kq0gHf7yy9mnr0BgmrnXe2WipOVvDZ7kSe3jEA5VSLznPQu2Q/AgYv8sLKn7kTYepC4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072497; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=lYOl6DM+S0I1mkZzYcYiiSjEp2QspraXpoabFr5HWkE=; b=bo7PRvIzLti7GQL3DuI8Y37ycIgsSI+1wHotLF7PVp1rZDcDR92UxX96UNnXCToYDeaLAgq2Ec29noVRzVRCtqufKojnCBFOy1qvZe47YdDn0XQkxvoxbKfePdJ7KF+gPC9jCP8uugnfw+vvo5qfOfZ7dGEzjQlk/wlSxrL5mrQ= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072497915764.1325647172309; Tue, 28 May 2019 12:41:37 -0700 (PDT) Received: from localhost ([127.0.0.1]:41667 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhyO-0001bA-Kz for importer@patchew.org; Tue, 28 May 2019 15:41:24 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36878) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhnj-0001LM-9R for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhnY-0001Vo-2S for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41084) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhnN-0000vt-5p; Tue, 28 May 2019 15:30:03 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E45A3883D7; Tue, 28 May 2019 19:29:34 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7844B60565; Tue, 28 May 2019 19:29:33 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:42 +0200 Message-Id: <20190528192847.2730-17-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Tue, 28 May 2019 19:29:34 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 16/21] qemu-img: rebase: Reuse parent BlockDriverState X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Sam Eiderman In safe mode we open the entire chain, including the parent backing file of the rebased file. Do not open a new BlockBackend for the parent backing file, which saves opening the rest of the chain twice, which for long chains saves many "pricy" bdrv_open() calls. Permissions for blk_new() were copied from blk_new_open() when flags =3D 0. Reviewed-by: Karl Heubaum Reviewed-by: Eyal Moscovici Signed-off-by: Sagi Amit Co-developed-by: Sagi Amit Signed-off-by: Sam Eiderman Message-id: 20190523163337.4497-2-shmuel.eiderman@oracle.com Signed-off-by: Max Reitz --- qemu-img.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index 28fba1e7a7..9bd0bb1e47 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3309,29 +3309,18 @@ static int img_rebase(int argc, char **argv) =20 /* For safe rebasing we need to compare old and new backing file */ if (!unsafe) { - char backing_name[PATH_MAX]; QDict *options =3D NULL; + BlockDriverState *base_bs =3D backing_bs(bs); =20 - if (bs->backing) { - if (bs->backing_format[0] !=3D '\0') { - options =3D qdict_new(); - qdict_put_str(options, "driver", bs->backing_format); - } - - if (force_share) { - if (!options) { - options =3D qdict_new(); - } - qdict_put_bool(options, BDRV_OPT_FORCE_SHARE, true); - } - bdrv_get_backing_filename(bs, backing_name, sizeof(backing_nam= e)); - blk_old_backing =3D blk_new_open(backing_name, NULL, - options, src_flags, &local_err); - if (!blk_old_backing) { + if (base_bs) { + blk_old_backing =3D blk_new(BLK_PERM_CONSISTENT_READ, + BLK_PERM_ALL); + ret =3D blk_insert_bs(blk_old_backing, base_bs, + &local_err); + if (ret < 0) { error_reportf_err(local_err, - "Could not open old backing file '%s': ", - backing_name); - ret =3D -1; + "Could not reuse old backing file '%s': = ", + base_bs->filename); goto out; } } else { --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072815; cv=none; d=zoho.com; s=zohoarc; b=RBxI5P3q/34q514izjmjsNil+MP1mQCv/lJJMOHRGk+SY8ebTWMYQdZCOfQDXYnPz8oM4YdUlXrpVG3x6RbcyLI/qqbCpLJ+Bef5iLdi3ntwCxiQUgxOtEOihR9xyghMAfKmKG6QvQE+IUzOXEsogIHBQIY7lrFmF+hQ1jz8mCo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072815; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=GkLwIGQd11F5/eRR9vtDT/HziixTZGwg2XBFjhe9NTs=; b=TNW7pFamd0e8ipQ2jrc9xenigNVQVDDLLcOcOPYoshOBxT3F2GUOQ4zGz3aWbc/T4vLL0zCK17x4vmq47PW8a9KWtz1iMthCr4SSPZtL68XVyIZdgDhbs13jUKjUojdZdAdd8pPyFnQkiP4tuCY52eG5MwJyIC5SFPEm1oXVT24= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072815990450.3956936076776; Tue, 28 May 2019 12:46:55 -0700 (PDT) Received: from localhost ([127.0.0.1]:41762 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVi3c-00064I-Vy for importer@patchew.org; Tue, 28 May 2019 15:46:49 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37271) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhow-0002JI-OQ for qemu-devel@nongnu.org; Tue, 28 May 2019 15:31:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhou-0002Ya-Pe for qemu-devel@nongnu.org; Tue, 28 May 2019 15:31:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36626) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhoe-00017l-N7; Tue, 28 May 2019 15:31:22 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3A7BA7EBD3; Tue, 28 May 2019 19:29:38 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C67DD5C8A3; Tue, 28 May 2019 19:29:36 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:43 +0200 Message-Id: <20190528192847.2730-18-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 28 May 2019 19:29:47 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 17/21] qemu-img: rebase: Reduce reads on in-chain rebase X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Sam Eiderman In the following case: (base) A <- B <- C (tip) when running: qemu-img rebase -b A C QEMU would read all sectors not allocated in the file being rebased (C) and compare them to the new base image (A), regardless of whether they were changed or even allocated anywhere along the chain between the new base and the top image (B). This causes many unneeded reads when rebasing an image which represents a small diff of a large disk, as it would read most of the disk's sectors. Instead, use bdrv_is_allocated_above() to reduce the number of unnecessary reads. Reviewed-by: Karl Heubaum Signed-off-by: Sam Eiderman Signed-off-by: Eyal Moscovici Message-id: 20190523163337.4497-3-shmuel.eiderman@oracle.com Signed-off-by: Max Reitz --- qemu-img.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/qemu-img.c b/qemu-img.c index 9bd0bb1e47..2d96a491e2 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3164,7 +3164,7 @@ static int img_rebase(int argc, char **argv) BlockBackend *blk =3D NULL, *blk_old_backing =3D NULL, *blk_new_backin= g =3D NULL; uint8_t *buf_old =3D NULL; uint8_t *buf_new =3D NULL; - BlockDriverState *bs =3D NULL; + BlockDriverState *bs =3D NULL, *prefix_chain_bs =3D NULL; char *filename; const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg; int c, flags, src_flags, ret; @@ -3353,6 +3353,12 @@ static int img_rebase(int argc, char **argv) goto out; } =20 + /* + * Find out whether we rebase an image on top of a previous im= age + * in its chain. + */ + prefix_chain_bs =3D bdrv_find_backing_image(bs, out_real_path); + blk_new_backing =3D blk_new_open(out_real_path, NULL, options, src_flags, &local_err); g_free(out_real_path); @@ -3437,6 +3443,23 @@ static int img_rebase(int argc, char **argv) continue; } =20 + if (prefix_chain_bs) { + /* + * If cluster wasn't changed since prefix_chain, we don't = need + * to take action + */ + ret =3D bdrv_is_allocated_above(backing_bs(bs), prefix_cha= in_bs, + offset, n, &n); + if (ret < 0) { + error_report("error while reading image metadata: %s", + strerror(-ret)); + goto out; + } + if (!ret) { + continue; + } + } + /* * Read old and new backing file and take into consideration t= hat * backing files may be smaller than the COW image. --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072518; cv=none; d=zoho.com; s=zohoarc; b=jCbPMipVQL0pNNPQGBXd0SKtayoLypdC+tujQmdhYgL7hMSrwr+wN8JM7c8JPeS6bDs9Z7r8LFyzejcHhDTekSdDhh4LA10hRuryxaBfJmqThWJS9iuue4vDbHhHtd3asSuPq8MrwlbEkN7HYHzwh9nb5hvLQtm3DzMycRNUadE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072518; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=qTIWdZbZrn+zulzy55gIkl1Iy2pXqdm4ff7CpGk4qhg=; b=QUhxJnkLcisdyhJWbHIBs8n2CE1hmenkin/aRL/n/mN55PCBFEwzAfAl7hdXF4v8T/LLgcZZCLKOnECokYGA8A+FAFiIAZWyHg8MTcGddFOxJfoabBsTowX3DO/M4UassbKqLWk/+6NgLzU+nJ6p9eaoY64kyIfdnK0iC7zAjWA= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072518991163.68763729166915; Tue, 28 May 2019 12:41:58 -0700 (PDT) Received: from localhost ([127.0.0.1]:41673 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhyt-0001zE-EM for importer@patchew.org; Tue, 28 May 2019 15:41:55 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37105) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhoP-0001sI-3O for qemu-devel@nongnu.org; Tue, 28 May 2019 15:31:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhoH-0002Bq-7h for qemu-devel@nongnu.org; Tue, 28 May 2019 15:31:00 -0400 Received: from mx1.redhat.com ([209.132.183.28]:42314) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVho2-00017U-L5; Tue, 28 May 2019 15:30:44 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9C8CF300CA9A; Tue, 28 May 2019 19:29:42 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3087A60C47; Tue, 28 May 2019 19:29:39 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:44 +0200 Message-Id: <20190528192847.2730-19-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Tue, 28 May 2019 19:29:42 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 18/21] qemu-img: rebase: Reuse in-chain BlockDriverState X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Sam Eiderman If a chain was detected, don't open a new BlockBackend from the target backing file which will create a new BlockDriverState. Instead, create an empty BlockBackend and attach the already open BlockDriverState. Permissions for blk_new() were copied from blk_new_open() when flags =3D 0. Reviewed-by: Karl Heubaum Reviewed-by: Eyal Moscovici Signed-off-by: Sagi Amit Co-developed-by: Sagi Amit Signed-off-by: Sam Eiderman Message-id: 20190523163337.4497-4-shmuel.eiderman@oracle.com Signed-off-by: Max Reitz --- qemu-img.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index 2d96a491e2..b0535919b1 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3358,16 +3358,29 @@ static int img_rebase(int argc, char **argv) * in its chain. */ prefix_chain_bs =3D bdrv_find_backing_image(bs, out_real_path); - - blk_new_backing =3D blk_new_open(out_real_path, NULL, - options, src_flags, &local_err); - g_free(out_real_path); - if (!blk_new_backing) { - error_reportf_err(local_err, - "Could not open new backing file '%s': ", - out_baseimg); - ret =3D -1; - goto out; + if (prefix_chain_bs) { + g_free(out_real_path); + blk_new_backing =3D blk_new(BLK_PERM_CONSISTENT_READ, + BLK_PERM_ALL); + ret =3D blk_insert_bs(blk_new_backing, prefix_chain_bs, + &local_err); + if (ret < 0) { + error_reportf_err(local_err, + "Could not reuse backing file '%s': = ", + out_baseimg); + goto out; + } + } else { + blk_new_backing =3D blk_new_open(out_real_path, NULL, + options, src_flags, &local_= err); + g_free(out_real_path); + if (!blk_new_backing) { + error_reportf_err(local_err, + "Could not open new backing file '%s= ': ", + out_baseimg); + ret =3D -1; + goto out; + } } } } --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072726; cv=none; d=zoho.com; s=zohoarc; b=EqwHWAtDKYSZ4Nfmsq9X3CsICWIPdYnVJLVhXbDxBwDPqAZ1SbPY7AeTtlZGMBW8LL1QwsBueMVigEdg5GQXYCjREEWjx0ucyiq7aQOoA2UPGONxi9MhSvKYzzBGneJN8nddYc3iEXvyK0Lw/YM9oak1i7T+IAgjS+0dpyYzoto= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072726; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=RxFMsmAxeM55SXwwZCX0ZC/L0mbYrZWxeNbeTYr2oOU=; b=QuzqMYqqfztu9TseypeWou/zyO6b1ElOxJKSK0GxOA/a0dyqF4ZonY+uaoAxm8jqMovUs1aPUzTLEVYmJIWv5769890JEUq06SwhUU7nm8/V2w2A7Na7Tw/mBEEiiBBKOIexS+GVN9DXh+cmoO8ZPjDnr8Hrrz5RJoy3fTh6d48= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072726027818.0183822977929; Tue, 28 May 2019 12:45:26 -0700 (PDT) Received: from localhost ([127.0.0.1]:41723 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVi27-0004xV-SK for importer@patchew.org; Tue, 28 May 2019 15:45:15 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37641) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhpP-0002dh-Dg for qemu-devel@nongnu.org; Tue, 28 May 2019 15:32:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhpN-0002ti-8L for qemu-devel@nongnu.org; Tue, 28 May 2019 15:32:07 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49382) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhpE-0001IZ-OW; Tue, 28 May 2019 15:31:57 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 45A853082E44; Tue, 28 May 2019 19:29:46 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9B0925D9CC; Tue, 28 May 2019 19:29:44 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:45 +0200 Message-Id: <20190528192847.2730-20-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.46]); Tue, 28 May 2019 19:29:52 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 19/21] qcow2: skip writing zero buffers to empty COW areas X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Anton Nefedov If COW areas of the newly allocated clusters are zeroes on the backing image, efficient bdrv_write_zeroes(flags=3DBDRV_REQ_NO_FALLBACK) can be used on the whole cluster instead of writing explicit zero buffers later in perform_cow(). iotest 060: write to the discarded cluster does not trigger COW anymore. Use a backing image instead. Signed-off-by: Anton Nefedov Message-id: 20190516142749.81019-2-anton.nefedov@virtuozzo.com Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Signed-off-by: Max Reitz --- qapi/block-core.json | 4 +- block/qcow2.h | 6 +++ block/qcow2-cluster.c | 2 +- block/qcow2.c | 85 ++++++++++++++++++++++++++++++++++++++ block/trace-events | 1 + tests/qemu-iotests/060 | 7 +++- tests/qemu-iotests/060.out | 5 ++- 7 files changed, 106 insertions(+), 4 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 7ccbfff9d0..3e4042be7f 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -3215,6 +3215,8 @@ # # @cor_write: a write due to copy-on-read (since 2.11) # +# @cluster_alloc_space: an allocation of file space for a cluster (since 4= .1) +# # Since: 2.9 ## { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG', @@ -3233,7 +3235,7 @@ 'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev', 'pwritev_zero', 'pwritev_done', 'empty_image_prepare', 'l1_shrink_write_table', 'l1_shrink_free_l2_clusters', - 'cor_write'] } + 'cor_write', 'cluster_alloc_space'] } =20 ## # @BlkdebugInjectErrorOptions: diff --git a/block/qcow2.h b/block/qcow2.h index e873ae5d12..567375e56c 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -405,6 +405,12 @@ typedef struct QCowL2Meta */ Qcow2COWRegion cow_end; =20 + /* + * Indicates that COW regions are already handled and do not require + * any more processing. + */ + bool skip_cow; + /** * The I/O vector with the data from the actual guest write request. * If non-NULL, this is meant to be merged together with the data diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 4a929900cf..cf892f37a8 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -831,7 +831,7 @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta= *m) assert(start->offset + start->nb_bytes <=3D end->offset); assert(!m->data_qiov || m->data_qiov->size =3D=3D data_bytes); =20 - if (start->nb_bytes =3D=3D 0 && end->nb_bytes =3D=3D 0) { + if ((start->nb_bytes =3D=3D 0 && end->nb_bytes =3D=3D 0) || m->skip_co= w) { return 0; } =20 diff --git a/block/qcow2.c b/block/qcow2.c index dea765b2f4..f2cb131048 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -2103,6 +2103,11 @@ static bool merge_cow(uint64_t offset, unsigned byte= s, continue; } =20 + /* If COW regions are handled already, skip this too */ + if (m->skip_cow) { + continue; + } + /* The data (middle) region must be immediately after the * start region */ if (l2meta_cow_start(m) + m->cow_start.nb_bytes !=3D offset) { @@ -2128,6 +2133,80 @@ static bool merge_cow(uint64_t offset, unsigned byte= s, return false; } =20 +static bool is_unallocated(BlockDriverState *bs, int64_t offset, int64_t b= ytes) +{ + int64_t nr; + return !bytes || + (!bdrv_is_allocated_above(bs, NULL, offset, bytes, &nr) && nr =3D= =3D bytes); +} + +static bool is_zero_cow(BlockDriverState *bs, QCowL2Meta *m) +{ + /* + * This check is designed for optimization shortcut so it must be + * efficient. + * Instead of is_zero(), use is_unallocated() as it is faster (but not + * as accurate and can result in false negatives). + */ + return is_unallocated(bs, m->offset + m->cow_start.offset, + m->cow_start.nb_bytes) && + is_unallocated(bs, m->offset + m->cow_end.offset, + m->cow_end.nb_bytes); +} + +static int handle_alloc_space(BlockDriverState *bs, QCowL2Meta *l2meta) +{ + BDRVQcow2State *s =3D bs->opaque; + QCowL2Meta *m; + + if (!(s->data_file->bs->supported_zero_flags & BDRV_REQ_NO_FALLBACK)) { + return 0; + } + + if (bs->encrypted) { + return 0; + } + + for (m =3D l2meta; m !=3D NULL; m =3D m->next) { + int ret; + + if (!m->cow_start.nb_bytes && !m->cow_end.nb_bytes) { + continue; + } + + if (!is_zero_cow(bs, m)) { + continue; + } + + /* + * instead of writing zero COW buffers, + * efficiently zero out the whole clusters + */ + + ret =3D qcow2_pre_write_overlap_check(bs, 0, m->alloc_offset, + m->nb_clusters * s->cluster_si= ze, + true); + if (ret < 0) { + return ret; + } + + BLKDBG_EVENT(bs->file, BLKDBG_CLUSTER_ALLOC_SPACE); + ret =3D bdrv_co_pwrite_zeroes(s->data_file, m->alloc_offset, + m->nb_clusters * s->cluster_size, + BDRV_REQ_NO_FALLBACK); + if (ret < 0) { + if (ret !=3D -ENOTSUP && ret !=3D -EAGAIN) { + return ret; + } + continue; + } + + trace_qcow2_skip_cow(qemu_coroutine_self(), m->offset, m->nb_clust= ers); + m->skip_cow =3D true; + } + return 0; +} + static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t of= fset, uint64_t bytes, QEMUIOVector *qio= v, int flags) @@ -2207,6 +2286,12 @@ static coroutine_fn int qcow2_co_pwritev(BlockDriver= State *bs, uint64_t offset, qemu_iovec_add(&hd_qiov, cluster_data, cur_bytes); } =20 + /* Try to efficiently initialize the physical space with zeroes */ + ret =3D handle_alloc_space(bs, l2meta); + if (ret < 0) { + goto out_unlocked; + } + /* If we need to do COW, check if it's possible to merge the * writing of the guest data together with that of the COW regions. * If it's not possible (or not necessary) then write the diff --git a/block/trace-events b/block/trace-events index 79ccd8d824..1e0653ce6d 100644 --- a/block/trace-events +++ b/block/trace-events @@ -68,6 +68,7 @@ qcow2_writev_done_part(void *co, int cur_bytes) "co %p cu= r_bytes %d" qcow2_writev_data(void *co, uint64_t offset) "co %p offset 0x%" PRIx64 qcow2_pwrite_zeroes_start_req(void *co, int64_t offset, int count) "co %p = offset 0x%" PRIx64 " count %d" qcow2_pwrite_zeroes(void *co, int64_t offset, int count) "co %p offset 0x%= " PRIx64 " count %d" +qcow2_skip_cow(void *co, uint64_t offset, int nb_clusters) "co %p offset 0= x%" PRIx64 " nb_clusters %d" =20 # qcow2-cluster.c qcow2_alloc_clusters_offset(void *co, uint64_t offset, int bytes) "co %p o= ffset 0x%" PRIx64 " bytes %d" diff --git a/tests/qemu-iotests/060 b/tests/qemu-iotests/060 index 89e911400c..b91d8321bb 100755 --- a/tests/qemu-iotests/060 +++ b/tests/qemu-iotests/060 @@ -150,10 +150,15 @@ $QEMU_IO -c "$OPEN_RO" -c "read -P 1 0 512" | _filter= _qemu_io echo echo "=3D=3D=3D Testing overlap while COW is in flight =3D=3D=3D" echo +BACKING_IMG=3D$TEST_IMG.base +TEST_IMG=3D$BACKING_IMG _make_test_img 1G + +$QEMU_IO -c 'write 0k 64k' "$BACKING_IMG" | _filter_qemu_io + # compat=3D0.10 is required in order to make the following discard actually # unallocate the sector rather than make it a zero sector - we want COW, a= fter # all. -IMGOPTS=3D'compat=3D0.10' _make_test_img 1G +IMGOPTS=3D'compat=3D0.10' _make_test_img -b "$BACKING_IMG" 1G # Write two clusters, the second one enforces creation of an L2 table after # the first data cluster. $QEMU_IO -c 'write 0k 64k' -c 'write 512M 64k' "$TEST_IMG" | _filter_qemu_= io diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out index e42bf8c5a9..0f6b0658a1 100644 --- a/tests/qemu-iotests/060.out +++ b/tests/qemu-iotests/060.out @@ -97,7 +97,10 @@ read 512/512 bytes at offset 0 =20 =3D=3D=3D Testing overlap while COW is in flight =3D=3D=3D =20 -Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D1073741824 +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=3DIMGFMT size=3D1073741824 +wrote 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D1073741824 backing_fil= e=3DTEST_DIR/t.IMGFMT.base wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 65536/65536 bytes at offset 536870912 --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072990; cv=none; d=zoho.com; s=zohoarc; b=mBIX1RUr/8mOtjlOfyt5hb1cNyROu7QRj5GNka1vVF0xx/Wi7x99kzPxZ9ogaRNdsji+G6B704GQFFgsknzscrp4N/PZC1OmPyPkPXiigUGrZisgIoJuW7TeSBiOMs9p46nPy0Z5g8El9whFxc3eyZTjgQqAJcT4r9WcTuUlJk8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072990; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=3VbzSwN2tBd1cuHuygf/n9QeWUpdQSR8LXHhmkcIsRE=; b=kzVTf20pa+NLctaL6iCRssoZPftI5dT9jKfKYQJIrO1KroNxYHhL0nKy7tKh2oMb2GVC+Lz0Fv61d7pnOTxvRBosXSQknqqGCsLwmhGrDQCIkomyt/j8wQQWUYI9GjQ0UTpsEPOySKyFQbnh8FlVzSoiabHNtQLy7P9AakXSvwM= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 155907299075847.84086371724095; Tue, 28 May 2019 12:49:50 -0700 (PDT) Received: from localhost ([127.0.0.1]:41791 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVi6O-00082B-PL for importer@patchew.org; Tue, 28 May 2019 15:49:40 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37585) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhpN-0002bR-A1 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:32:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhpM-0002sv-3b for qemu-devel@nongnu.org; Tue, 28 May 2019 15:32:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:23536) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhpH-00018f-5d; Tue, 28 May 2019 15:31:59 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A4ECB81E05; Tue, 28 May 2019 19:29:48 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3BE9F5D71A; Tue, 28 May 2019 19:29:48 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:46 +0200 Message-Id: <20190528192847.2730-21-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 28 May 2019 19:29:48 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 20/21] qcow2-bitmap: initialize bitmap directory alignment X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Andrey Shinkevich Valgrind detects multiple issues in QEMU iotests when the memory is used without being initialized. Valgrind may dump lots of unnecessary reports what makes the memory issue analysis harder. Particularly, that is true for the aligned bitmap directory and can be seen while running the iotest #169. Padding the aligned space with zeros eases the pain. Signed-off-by: Andrey Shinkevich Message-id: 1558961521-131620-1-git-send-email-andrey.shinkevich@virtuozzo.= com Signed-off-by: Max Reitz --- block/qcow2-bitmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 640da68ce1..b2487101ed 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -753,7 +753,7 @@ static int bitmap_list_store(BlockDriverState *bs, Qcow= 2BitmapList *bm_list, dir_offset =3D *offset; } =20 - dir =3D g_try_malloc(dir_size); + dir =3D g_try_malloc0(dir_size); if (dir =3D=3D NULL) { return -ENOMEM; } --=20 2.21.0 From nobody Tue May 7 10:54:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1559072346; cv=none; d=zoho.com; s=zohoarc; b=OwneBfXVeFTcn5+f+kx8Safv65rc3WdhKxszLvZixZl/lU9eGCfWElgtvAZVr1+jJCIMydVUex4dlB+4kT935MjVlDiTpuGJKUUBa3Q7hIxeCAYAyZND7im83iVf6pXgZ0GbBYHs8dRtbdC7Z7TdD2goDqvdNnCZ4bLKdo/MxTI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559072346; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=wON7Kh98/E/DFU++2S9sF64uLgZXwb0hIWoYoZE5z8A=; b=XBoqmTKreYkInuE7RS/tBCD6WeURMWPXMcXMFU0tP3Ngz2YX9Y4QuL7Ra0RibWkUQFdgR6U/abPfkPg5X/L8GSUUUal+4jlyXxX7GFFWXC36Js+qE/7ZBSDaPBnlkJ2bwK0tReHrsJTTjnO7qoEsLhDNwLenJrEkGc8KkqDPuvE= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559072346534202.44437160128496; Tue, 28 May 2019 12:39:06 -0700 (PDT) Received: from localhost ([127.0.0.1]:41613 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhw0-0007WD-G1 for importer@patchew.org; Tue, 28 May 2019 15:38:56 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37078) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVhoH-0001kx-Tr for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVhoA-000272-J6 for qemu-devel@nongnu.org; Tue, 28 May 2019 15:30:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36698) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVhny-0001Lf-OI; Tue, 28 May 2019 15:30:39 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0BBA67E426; Tue, 28 May 2019 19:29:51 +0000 (UTC) Received: from localhost (unknown [10.40.205.223]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 94560189A2; Tue, 28 May 2019 19:29:50 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 28 May 2019 21:28:47 +0200 Message-Id: <20190528192847.2730-22-mreitz@redhat.com> In-Reply-To: <20190528192847.2730-1-mreitz@redhat.com> References: <20190528192847.2730-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 28 May 2019 19:29:57 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 21/21] blockdev: loosen restrictions on drive-backup source node X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: John Snow We mandate that the source node must be a root node; but there's no reason I am aware of that it needs to be restricted to such. In some cases, we need to make sure that there's a medium present, but in the general case we can allow the backup job itself to do the graph checking. This patch helps improve the error message when you try to backup from the same node more than once, which is reflected in the change to test 056. For backups with bitmaps, it will also show a better error message that the bitmap is in use instead of giving you something cryptic like "need a root node." Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=3D1707303 Signed-off-by: John Snow Message-id: 20190521210053.8864-1-jsnow@redhat.com Signed-off-by: Max Reitz --- blockdev.c | 7 ++++++- tests/qemu-iotests/056 | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/blockdev.c b/blockdev.c index 79fbac8450..bc339d7818 100644 --- a/blockdev.c +++ b/blockdev.c @@ -3450,11 +3450,16 @@ static BlockJob *do_drive_backup(DriveBackup *backu= p, JobTxn *txn, backup->compress =3D false; } =20 - bs =3D qmp_get_root_bs(backup->device, errp); + bs =3D bdrv_lookup_bs(backup->device, backup->device, errp); if (!bs) { return NULL; } =20 + if (!bs->drv) { + error_setg(errp, "Device has no medium"); + return NULL; + } + aio_context =3D bdrv_get_aio_context(bs); aio_context_acquire(aio_context); =20 diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056 index 3df323984d..f40fc11a09 100755 --- a/tests/qemu-iotests/056 +++ b/tests/qemu-iotests/056 @@ -214,7 +214,7 @@ class BackupTest(iotests.QMPTestCase): res =3D self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return[0]/status', 'concluded') # Leave zombie job un-dismissed, observe a failure: - res =3D self.qmp_backup_and_wait(serror=3D'Need a root block node', + res =3D self.qmp_backup_and_wait(serror=3D"Node 'drive0' is busy: = block device is in use by block job: backup", device=3D'drive0', format=3Diotests= .imgfmt, sync=3D'full', target=3Dself.dest_i= mg, auto_dismiss=3DFalse) --=20 2.21.0