From nobody Wed Nov 12 15:02:12 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1571604078; cv=none; d=zoho.com; s=zohoarc; b=fVwDbCnA51na8HGzxrhdee30Qju/LXkKygK47uVLmnBd4hGsPx/crjzFYIXcX0YgyuZYOCxcqLd/tH7aRXuJ+/zr2CRkc+lZ0E8YNQ70IV6WegCeTN3n4KD45+Nk3ABnkSiCpkRTevL+Xtufze1mtHVfgp1mi5op53jBkwrjHas= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1571604078; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=fvcR3hoP24tVYraiWZH6H4hkQ5B0+hp0qVrk8xmWypc=; b=FJc2NgZO0k789QrBusEdBUWlVf8a225lMKSKs7KlkR7RXjbzzr8VpsUJN/yICksxzPMqeJlqln5xGD0o0bpW4Q/4znauX5rtrM10Ia0KZKqUc1y2kxtPJ58ZXsaR1y6AA2cEkagK0JriD9oFj2r/ezSZO9na4m1Qf8ZGfiPU0PM= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1571604078190144.55384488372886; Sun, 20 Oct 2019 13:41:18 -0700 (PDT) Received: from localhost ([::1]:47164 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMI0p-0001sK-Dr for importer@patchew.org; Sun, 20 Oct 2019 16:41:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51667) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMHx7-0005vS-Ea for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iMHx5-0006Xs-Kb for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:25 -0400 Received: from relay.sw.ru ([185.231.240.75]:58838) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iMHx5-0006Wv-CV; Sun, 20 Oct 2019 16:37:23 -0400 Received: from [172.16.25.136] (helo=dhcp-172-16-25-136.sw.ru) by relay.sw.ru with esmtp (Exim 4.92.2) (envelope-from ) id 1iMHx1-00089m-6B; Sun, 20 Oct 2019 23:37:19 +0300 From: Andrey Shinkevich To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v5 1/4] block: support compressed write at generic layer Date: Sun, 20 Oct 2019 23:37:05 +0300 Message-Id: <1571603828-185910-2-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 185.231.240.75 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, fam@euphon.net, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" To inform the block layer about writing all the data compressed, we introduce the 'compress' command line option. Based on that option, the written data will be aligned by the cluster size at the generic layer. Suggested-by: Roman Kagan Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block.c | 20 +++++++++++++++++++- block/io.c | 13 +++++++++---- block/qcow2.c | 4 ++++ blockdev.c | 9 ++++++++- include/block/block.h | 1 + include/block/block_int.h | 2 ++ qapi/block-core.json | 5 ++++- qemu-options.hx | 6 ++++-- 8 files changed, 51 insertions(+), 9 deletions(-) diff --git a/block.c b/block.c index 1946fc6..a674920 100644 --- a/block.c +++ b/block.c @@ -1418,6 +1418,11 @@ QemuOptsList bdrv_runtime_opts =3D { .type =3D QEMU_OPT_BOOL, .help =3D "always accept other writers (default: off)", }, + { + .name =3D BDRV_OPT_COMPRESS, + .type =3D QEMU_OPT_BOOL, + .help =3D "compress all writes to the image (default: off)", + }, { /* end of list */ } }, }; @@ -1545,6 +1550,14 @@ static int bdrv_open_common(BlockDriverState *bs, Bl= ockBackend *file, } pstrcpy(bs->exact_filename, sizeof(bs->exact_filename), bs->filename); =20 + if (bs->all_write_compressed && !drv->bdrv_co_pwritev_compressed_part)= { + error_setg(errp, "Compression is not supported for the driver '%s'= ", + drv->format_name); + bs->all_write_compressed =3D false; + ret =3D -ENOTSUP; + goto fail_opts; + } + /* Open the image, either directly or using a protocol */ open_flags =3D bdrv_open_flags(bs, bs->open_flags); node_name =3D qemu_opt_get(opts, "node-name"); @@ -2983,6 +2996,11 @@ static BlockDriverState *bdrv_open_inherit(const cha= r *filename, flags &=3D ~BDRV_O_RDWR; } =20 + if (!g_strcmp0(qdict_get_try_str(options, BDRV_OPT_COMPRESS), "on") || + qdict_get_try_bool(options, BDRV_OPT_COMPRESS, false)) { + bs->all_write_compressed =3D true; + } + if (flags & BDRV_O_SNAPSHOT) { snapshot_options =3D qdict_new(); bdrv_temp_snapshot_options(&snapshot_flags, snapshot_options, @@ -3208,7 +3226,7 @@ static int bdrv_reset_options_allowed(BlockDriverStat= e *bs, * in bdrv_reopen_prepare() so they can be left out of @new_opts */ const char *const common_options[] =3D { "node-name", "discard", "cache.direct", "cache.no-flush", - "read-only", "auto-read-only", "detect-zeroes", NULL + "read-only", "auto-read-only", "detect-zeroes", "compress", NULL }; =20 for (e =3D qdict_first(bs->options); e; e =3D qdict_next(bs->options, = e)) { diff --git a/block/io.c b/block/io.c index f0b86c1..eb2ed36 100644 --- a/block/io.c +++ b/block/io.c @@ -1360,9 +1360,14 @@ static int coroutine_fn bdrv_co_do_copy_on_readv(Bdr= vChild *child, /* This does not change the data on the disk, it is not * necessary to flush even in cache=3Dwritethrough mode. */ - ret =3D bdrv_driver_pwritev(bs, cluster_offset, pnum, - &local_qiov, 0, - BDRV_REQ_WRITE_UNCHANGED); + if (bs->all_write_compressed) { + ret =3D bdrv_driver_pwritev_compressed(bs, cluster_off= set, + pnum, &local_qiov= , 0); + } else { + ret =3D bdrv_driver_pwritev(bs, cluster_offset, pnum, + &local_qiov, 0, + BDRV_REQ_WRITE_UNCHANGED); + } } =20 if (ret < 0) { @@ -1954,7 +1959,7 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChil= d *child, } else if (flags & BDRV_REQ_ZERO_WRITE) { bdrv_debug_event(bs, BLKDBG_PWRITEV_ZERO); ret =3D bdrv_co_do_pwrite_zeroes(bs, offset, bytes, flags); - } else if (flags & BDRV_REQ_WRITE_COMPRESSED) { + } else if (flags & BDRV_REQ_WRITE_COMPRESSED || bs->all_write_compress= ed) { ret =3D bdrv_driver_pwritev_compressed(bs, offset, bytes, qiov, qiov_offset); } else if (bytes <=3D max_transfer) { diff --git a/block/qcow2.c b/block/qcow2.c index 7961c05..6b29e16 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1787,6 +1787,10 @@ static void qcow2_refresh_limits(BlockDriverState *b= s, Error **errp) /* Encryption works on a sector granularity */ bs->bl.request_alignment =3D qcrypto_block_get_sector_size(s->cryp= to); } + if (bs->all_write_compressed) { + bs->bl.request_alignment =3D MAX(bs->bl.request_alignment, + s->cluster_size); + } bs->bl.pwrite_zeroes_alignment =3D s->cluster_size; bs->bl.pdiscard_alignment =3D s->cluster_size; } diff --git a/blockdev.c b/blockdev.c index f89e48f..0c0b398 100644 --- a/blockdev.c +++ b/blockdev.c @@ -471,7 +471,7 @@ static BlockBackend *blockdev_init(const char *file, QD= ict *bs_opts, int bdrv_flags =3D 0; int on_read_error, on_write_error; bool account_invalid, account_failed; - bool writethrough, read_only; + bool writethrough, read_only, compress; BlockBackend *blk; BlockDriverState *bs; ThrottleConfig cfg; @@ -570,6 +570,7 @@ static BlockBackend *blockdev_init(const char *file, QD= ict *bs_opts, } =20 read_only =3D qemu_opt_get_bool(opts, BDRV_OPT_READ_ONLY, false); + compress =3D qemu_opt_get_bool(opts, BDRV_OPT_COMPRESS, false); =20 /* init */ if ((!file || !*file) && !qdict_size(bs_opts)) { @@ -595,6 +596,8 @@ static BlockBackend *blockdev_init(const char *file, QD= ict *bs_opts, qdict_set_default_str(bs_opts, BDRV_OPT_READ_ONLY, read_only ? "on" : "off"); qdict_set_default_str(bs_opts, BDRV_OPT_AUTO_READ_ONLY, "on"); + qdict_set_default_str(bs_opts, BDRV_OPT_COMPRESS, + compress ? "on" : "off"); assert((bdrv_flags & BDRV_O_CACHE_MASK) =3D=3D 0); =20 if (runstate_check(RUN_STATE_INMIGRATE)) { @@ -4683,6 +4686,10 @@ QemuOptsList qemu_common_drive_opts =3D { .name =3D BDRV_OPT_READ_ONLY, .type =3D QEMU_OPT_BOOL, .help =3D "open drive file as read-only", + },{ + .name =3D BDRV_OPT_COMPRESS, + .type =3D QEMU_OPT_BOOL, + .help =3D "compress all writes to image", }, =20 THROTTLE_OPTS, diff --git a/include/block/block.h b/include/block/block.h index 792bb82..7e0a927 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -139,6 +139,7 @@ typedef struct HDGeometry { #define BDRV_OPT_AUTO_READ_ONLY "auto-read-only" #define BDRV_OPT_DISCARD "discard" #define BDRV_OPT_FORCE_SHARE "force-share" +#define BDRV_OPT_COMPRESS "compress" =20 =20 #define BDRV_SECTOR_BITS 9 diff --git a/include/block/block_int.h b/include/block/block_int.h index 05056b3..3fe8923 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -923,6 +923,8 @@ struct BlockDriverState { =20 /* BdrvChild links to this node may never be frozen */ bool never_freeze; + /* Compress all writes to the image */ + bool all_write_compressed; }; =20 struct BlockBackendRootState { diff --git a/qapi/block-core.json b/qapi/block-core.json index f66553a..d57064a 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4013,6 +4013,8 @@ # (default: off) # @force-share: force share all permission on added nodes. # Requires read-only=3Dtrue. (Since 2.10) +# @compress: compress all writes to the image (Since 4.2) +# (default: false) # # Remaining options are determined by the block driver. # @@ -4026,7 +4028,8 @@ '*read-only': 'bool', '*auto-read-only': 'bool', '*force-share': 'bool', - '*detect-zeroes': 'BlockdevDetectZeroesOptions' }, + '*detect-zeroes': 'BlockdevDetectZeroesOptions', + '*compress': 'bool' }, 'discriminator': 'driver', 'data': { 'blkdebug': 'BlockdevOptionsBlkdebug', diff --git a/qemu-options.hx b/qemu-options.hx index 793d70f..1bfbf1a 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -850,7 +850,7 @@ DEF("blockdev", HAS_ARG, QEMU_OPTION_blockdev, "-blockdev [driver=3D]driver[,node-name=3DN][,discard=3Dignore|unmap]\= n" " [,cache.direct=3Don|off][,cache.no-flush=3Don|off]\n" " [,read-only=3Don|off][,detect-zeroes=3Don|off|unmap]\n" - " [,driver specific parameters...]\n" + " [,compress=3Don|off][,driver specific parameters...]\n" " configure a block backend\n", QEMU_ARCH_ALL) STEXI @item -blockdev @var{option}[,@var{option}[,@var{option}[,...]]] @@ -905,6 +905,8 @@ discard requests. conversion of plain zero writes by the OS to driver specific optimized zero write commands. You may even choose "unmap" if @var{discard} is set to "unmap" to allow a zero write to be converted to an @code{unmap} operat= ion. +@item compress +Compress all writes to the image. @end table =20 @item Driver-specific options for @code{file} @@ -1026,7 +1028,7 @@ DEF("drive", HAS_ARG, QEMU_OPTION_drive, " [,cache=3Dwritethrough|writeback|none|directsync|unsafe][,form= at=3Df]\n" " [,snapshot=3Don|off][,rerror=3Dignore|stop|report]\n" " [,werror=3Dignore|stop|report|enospc][,id=3Dname][,aio=3Dthrea= ds|native]\n" - " [,readonly=3Don|off][,copy-on-read=3Don|off]\n" + " [,readonly=3Don|off][,copy-on-read=3Don|off][,compress=3Don|of= f]\n" " [,discard=3Dignore|unmap][,detect-zeroes=3Don|off|unmap]\n" " [[,bps=3Db]|[[,bps_rd=3Dr][,bps_wr=3Dw]]]\n" " [[,iops=3Di]|[[,iops_rd=3Dr][,iops_wr=3Dw]]]\n" --=20 1.8.3.1 From nobody Wed Nov 12 15:02:12 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1571603944; cv=none; d=zoho.com; s=zohoarc; b=OO/r5ChEg5H+OtjOcQ9S8aikwP4127lgFTdH6WCY1sqj8lax2SMLuQ8bdicYK4eikea9wmBYXGB2IxHgFHrMnTqlCL03E8dwB7SRGJ8l3jpm4wtCxnpRyntXyYUZml+78IQzMcIt/TsDBh218eIyIJ0PQyHdRUgNGUO/LiqyhsY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1571603944; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=aMX3+uqSKv1OzqLht1+KohFBYVCkVjW/qU8FNnb5oPY=; b=LfNJAKzk1ZMQjmYKVyCYVUZJUF5aceWO6KPUEFc3nA4rnCPoqSzeTUEbiVVPaiRlb9YMBXF2w/MAMI9exWHKuQaEyaZ7I92VcnGlGtyrwBfagJJhtl3l1doKiZYQYMpLmr1Q94yj3eNvdH3+os3I33J68wls4O7+PNZplW8dnsw= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1571603944859899.1535554726337; Sun, 20 Oct 2019 13:39:04 -0700 (PDT) Received: from localhost ([::1]:47086 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMHyf-00078t-Lm for importer@patchew.org; Sun, 20 Oct 2019 16:39:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51660) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMHx7-0005vR-AU for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iMHx5-0006Y4-Mz for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:24 -0400 Received: from relay.sw.ru ([185.231.240.75]:58846) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iMHx5-0006Wy-Fd; Sun, 20 Oct 2019 16:37:23 -0400 Received: from [172.16.25.136] (helo=dhcp-172-16-25-136.sw.ru) by relay.sw.ru with esmtp (Exim 4.92.2) (envelope-from ) id 1iMHx1-00089m-DA; Sun, 20 Oct 2019 23:37:19 +0300 From: Andrey Shinkevich To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v5 2/4] qcow2: Allow writing compressed data of multiple clusters Date: Sun, 20 Oct 2019 23:37:06 +0300 Message-Id: <1571603828-185910-3-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 185.231.240.75 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, fam@euphon.net, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" QEMU currently supports writing compressed data of the size equal to one cluster. This patch allows writing QCOW2 compressed data that exceed one cluster. Now, we split buffered data into separate clusters and write them compressed using the existing functionality. Suggested-by: Pavel Butsykin Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.c | 102 ++++++++++++++++++++++++++++++++++++++++++------------= ---- 1 file changed, 75 insertions(+), 27 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 6b29e16..a1e7279 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -4156,10 +4156,8 @@ fail: return ret; } =20 -/* XXX: put compressed sectors first, then all the cluster aligned - tables to avoid losing bytes in alignment */ static coroutine_fn int -qcow2_co_pwritev_compressed_part(BlockDriverState *bs, +qcow2_co_pwritev_compressed_task(BlockDriverState *bs, uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, size_t qiov_offset) { @@ -4169,32 +4167,11 @@ qcow2_co_pwritev_compressed_part(BlockDriverState *= bs, uint8_t *buf, *out_buf; uint64_t cluster_offset; =20 - if (has_data_file(bs)) { - return -ENOTSUP; - } - - if (bytes =3D=3D 0) { - /* align end of file to a sector boundary to ease reading with - sector based I/Os */ - int64_t len =3D bdrv_getlength(bs->file->bs); - if (len < 0) { - return len; - } - return bdrv_co_truncate(bs->file, len, PREALLOC_MODE_OFF, NULL); - } - - if (offset_into_cluster(s, offset)) { - return -EINVAL; - } + assert(bytes =3D=3D s->cluster_size || (bytes < s->cluster_size && + (offset + bytes =3D=3D bs->total_sectors << BDRV_SECTOR_BITS))); =20 buf =3D qemu_blockalign(bs, s->cluster_size); - if (bytes !=3D s->cluster_size) { - if (bytes > s->cluster_size || - offset + bytes !=3D bs->total_sectors << BDRV_SECTOR_BITS) - { - qemu_vfree(buf); - return -EINVAL; - } + if (bytes < s->cluster_size) { /* Zero-pad last write if image size is not cluster aligned */ memset(buf + bytes, 0, s->cluster_size - bytes); } @@ -4243,6 +4220,77 @@ fail: return ret; } =20 +static coroutine_fn int qcow2_co_pwritev_compressed_task_entry(AioTask *ta= sk) +{ + Qcow2AioTask *t =3D container_of(task, Qcow2AioTask, task); + + assert(!t->cluster_type && !t->l2meta); + + return qcow2_co_pwritev_compressed_task(t->bs, t->offset, t->bytes, t-= >qiov, + t->qiov_offset); +} + +/* + * XXX: put compressed sectors first, then all the cluster aligned + * tables to avoid losing bytes in alignment + */ +static coroutine_fn int +qcow2_co_pwritev_compressed_part(BlockDriverState *bs, + uint64_t offset, uint64_t bytes, + QEMUIOVector *qiov, size_t qiov_offset) +{ + BDRVQcow2State *s =3D bs->opaque; + AioTaskPool *aio =3D NULL; + int ret =3D 0; + + if (has_data_file(bs)) { + return -ENOTSUP; + } + + if (bytes =3D=3D 0) { + /* + * align end of file to a sector boundary to ease reading with + * sector based I/Os + */ + int64_t len =3D bdrv_getlength(bs->file->bs); + if (len < 0) { + return len; + } + return bdrv_co_truncate(bs->file, len, PREALLOC_MODE_OFF, NULL); + } + + if (offset_into_cluster(s, offset)) { + return -EINVAL; + } + + while (bytes && aio_task_pool_status(aio) =3D=3D 0) { + uint64_t chunk_size =3D MIN(bytes, s->cluster_size); + + if (!aio && chunk_size !=3D bytes) { + aio =3D aio_task_pool_new(QCOW2_MAX_WORKERS); + } + + ret =3D qcow2_add_task(bs, aio, qcow2_co_pwritev_compressed_task_e= ntry, + 0, 0, offset, chunk_size, qiov, qiov_offset, = NULL); + if (ret < 0) { + break; + } + qiov_offset +=3D chunk_size; + offset +=3D chunk_size; + bytes -=3D chunk_size; + } + + if (aio) { + aio_task_pool_wait_all(aio); + if (ret =3D=3D 0) { + ret =3D aio_task_pool_status(aio); + } + g_free(aio); + } + + return ret; +} + static int coroutine_fn qcow2_co_preadv_compressed(BlockDriverState *bs, uint64_t file_cluster_offset, --=20 1.8.3.1 From nobody Wed Nov 12 15:02:12 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1571604020; cv=none; d=zoho.com; s=zohoarc; b=lxrwm3DOZS9k8XBtCLPQyVN9gfqiFquyfCt7TCRxxUb90f6NKKauG0LSHsgulSEDBzcbkuti3H0f7aQn7y7ruZIlX6YTTfAgme31dCTIB25sKlMXcJiZIyjQHbPciTSyYLni0vFyhTPe+VTWlUoCx8wwKAEYyV5gNI7T7yQZSG0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1571604020; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=3vwS1Vwm5zztwGKdlG+SngGQemtrjexfzLz7nNYZcqU=; b=l2BKsqnIv65j2UkYkMfsAN1BtPu0vb8GXJ9rQqqKm2gq2Hdp7w5wf/JHlOz4/DE7CT4ZFJtYtio+FhO+NscOruUzpBU588+e2XaHr36Pz2FLk4T3mGSbcghA10wtnxr7NCjvmYaku8sRl2ff6zyIDWsWpqQIrigeEtvDRdqe6Kw= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1571604020698399.41906837786223; Sun, 20 Oct 2019 13:40:20 -0700 (PDT) Received: from localhost ([::1]:47140 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMHzv-0000uc-NB for importer@patchew.org; Sun, 20 Oct 2019 16:40:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51658) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMHx7-0005vQ-6i for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iMHx5-0006Xy-MC for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:24 -0400 Received: from relay.sw.ru ([185.231.240.75]:58842) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iMHx5-0006Ww-EF; Sun, 20 Oct 2019 16:37:23 -0400 Received: from [172.16.25.136] (helo=dhcp-172-16-25-136.sw.ru) by relay.sw.ru with esmtp (Exim 4.92.2) (envelope-from ) id 1iMHx1-00089m-Hz; Sun, 20 Oct 2019 23:37:19 +0300 From: Andrey Shinkevich To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v5 3/4] tests/qemu-iotests: add case to write compressed data of multiple clusters Date: Sun, 20 Oct 2019 23:37:07 +0300 Message-Id: <1571603828-185910-4-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 185.231.240.75 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, fam@euphon.net, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add the test case to the iotest #214 that checks possibility of writing compressed data of more than one cluster size. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/214 | 45 ++++++++++++++++++++++++++++++++++++++++++= +++ tests/qemu-iotests/214.out | 14 ++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/tests/qemu-iotests/214 b/tests/qemu-iotests/214 index 21ec8a2..ab9b862 100755 --- a/tests/qemu-iotests/214 +++ b/tests/qemu-iotests/214 @@ -89,6 +89,51 @@ _check_test_img -r all $QEMU_IO -c "read -P 0x11 0 4M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _fi= lter_testdir $QEMU_IO -c "read -P 0x22 4M 4M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _fi= lter_testdir =20 +echo +echo "=3D=3D=3D Write compressed data of multiple clusters =3D=3D=3D" +echo +cluster_size=3D0x10000 +_make_test_img 2M -o cluster_size=3D$cluster_size + +echo "Write uncompressed data:" +let data_size=3D"8 * $cluster_size" +$QEMU_IO -c "write -P 0xaa 0 $data_size" "$TEST_IMG" \ + 2>&1 | _filter_qemu_io | _filter_testdir +sizeA=3D$($QEMU_IMG info --output=3Djson "$TEST_IMG" | + sed -n '/"actual-size":/ s/[^0-9]//gp') + +_make_test_img 2M -o cluster_size=3D$cluster_size +echo "Write compressed data:" +let data_size=3D"3 * $cluster_size + ($cluster_size / 2)" +# Set compress=3Don. That will align the written data +# by the cluster size and will write them compressed. +QEMU_IO_OPTIONS=3D$QEMU_IO_OPTIONS_NO_FMT \ +$QEMU_IO -c "write -P 0xbb 0 $data_size" --image-opts \ + driver=3D$IMGFMT,compress=3Don,file.filename=3D$TEST_IMG \ + 2>&1 | _filter_qemu_io | _filter_testdir + +let offset=3D"4 * $cluster_size" +QEMU_IO_OPTIONS=3D$QEMU_IO_OPTIONS_NO_FMT \ +$QEMU_IO -c "write -P 0xcc $offset $data_size" "json:{\ + 'driver': '$IMGFMT', + 'file': { + 'driver': 'file', + 'filename': '$TEST_IMG' + }, + 'compress': true +}" | _filter_qemu_io | _filter_testdir + +sizeB=3D$($QEMU_IMG info --output=3Djson "$TEST_IMG" | + sed -n '/"actual-size":/ s/[^0-9]//gp') + +if [ $sizeA -le $sizeB ] +then + echo "Compression ERROR" +fi + +$QEMU_IMG check --output=3Djson "$TEST_IMG" | + sed -n 's/,$//; /"compressed-clusters":/ s/^ *//p' + # success, all done echo '*** done' rm -f $seq.full diff --git a/tests/qemu-iotests/214.out b/tests/qemu-iotests/214.out index 0fcd8dc..4a2ec33 100644 --- a/tests/qemu-iotests/214.out +++ b/tests/qemu-iotests/214.out @@ -32,4 +32,18 @@ read 4194304/4194304 bytes at offset 0 4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 4194304/4194304 bytes at offset 4194304 4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=3D=3D=3D Write compressed data of multiple clusters =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D2097152 +Write uncompressed data: +wrote 524288/524288 bytes at offset 0 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D2097152 +Write compressed data: +wrote 229376/229376 bytes at offset 0 +224 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 229376/229376 bytes at offset 262144 +224 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +"compressed-clusters": 8 *** done --=20 1.8.3.1 From nobody Wed Nov 12 15:02:12 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1571603943; cv=none; d=zoho.com; s=zohoarc; b=FGR1EkXEPE4OoMoRr51r+E4BWkJKGvKR3VdrIdZxbaiQpVwKK2E4rDKnj6aiWp81GybIyvIp0nAlMMB+vTDr2e0IDQOs6QL4Q3iM5mUj3QFCHmNjEGkIzZeGqXwKtFPDZ/H/XZOTqneBJ0BljDTqHYmAzyuw2lE1GJOeQD10F30= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1571603943; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=+y/NT7jCMcCJ5hHIkmj8f17h3fNoAyHczdUoyMlhL0w=; b=WrDU9HgrfeGH+EDLy0lh78Dqf1w28JvnFFLcmzipQRETRTfQXVrIJmgALiLYiczlHyrvCqiCDuLADB96WBQd0sNTtWR3U/6Z7pBoHoZUzKjdVWwseFliQNFgWhFnwItQ2UuOKhoIN+RHBsAUoQqcugaS8KGErzsPD7qT5D6Lfck= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1571603943643779.3725598086174; Sun, 20 Oct 2019 13:39:03 -0700 (PDT) Received: from localhost ([::1]:47090 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMHyg-0007DH-AA for importer@patchew.org; Sun, 20 Oct 2019 16:39:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51654) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iMHx7-0005vO-5N for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iMHx5-0006Xf-J6 for qemu-devel@nongnu.org; Sun, 20 Oct 2019 16:37:24 -0400 Received: from relay.sw.ru ([185.231.240.75]:58844) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iMHx5-0006Wu-Bn; Sun, 20 Oct 2019 16:37:23 -0400 Received: from [172.16.25.136] (helo=dhcp-172-16-25-136.sw.ru) by relay.sw.ru with esmtp (Exim 4.92.2) (envelope-from ) id 1iMHx1-00089m-Py; Sun, 20 Oct 2019 23:37:19 +0300 From: Andrey Shinkevich To: qemu-devel@nongnu.org, qemu-block@nongnu.org Subject: [PATCH v5 4/4] tests/qemu-iotests: add case for block-stream compress Date: Sun, 20 Oct 2019 23:37:08 +0300 Message-Id: <1571603828-185910-5-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1571603828-185910-1-git-send-email-andrey.shinkevich@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 185.231.240.75 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, fam@euphon.net, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, stefanha@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add a case to the iotest #030 that tests the 'compress' option for a block-stream job. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/030 | 34 +++++++++++++++++++++++++++++++++- tests/qemu-iotests/030.out | 4 ++-- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index f3766f2..f33fd21 100755 --- a/tests/qemu-iotests/030 +++ b/tests/qemu-iotests/030 @@ -21,7 +21,8 @@ import time import os import iotests -from iotests import qemu_img, qemu_io +from iotests import qemu_img, qemu_io, qemu_img_pipe +import json =20 backing_img =3D os.path.join(iotests.test_dir, 'backing.img') mid_img =3D os.path.join(iotests.test_dir, 'mid.img') @@ -956,6 +957,37 @@ class TestSetSpeed(iotests.QMPTestCase): =20 self.cancel_and_wait(resume=3DTrue) =20 +class TestCompressed(iotests.QMPTestCase): + allocated_clusters =3D 8 + + def setUp(self): + qemu_img('create', '-f', iotests.imgfmt, backing_img, '1M') + qemu_img('create', '-f', iotests.imgfmt, '-o', + 'backing_file=3D{}'.format(backing_img), test_img) + cluster_size =3D 0x10000 + data_size =3D self.allocated_clusters * cluster_size + qemu_io('-c', 'write -P 0x1 0 {}'.format(data_size), backing_img) + self.vm =3D iotests.VM().add_drive(test_img, "compress=3Don") + self.vm.launch() + + def tearDown(self): + self.vm.shutdown() + os.remove(test_img) + os.remove(backing_img) + + def test_stream_compress(self): + self.assert_no_active_block_jobs() + + result =3D self.vm.qmp('block-stream', device=3D'drive0') + self.assert_qmp(result, 'return', {}) + + match =3D {'data': {'type': 'stream', 'device': 'drive0'}} + self.vm.event_wait(name=3D'BLOCK_JOB_COMPLETED', match=3Dmatch) + self.vm.shutdown() + + top =3D json.loads(qemu_img_pipe('check', '--output=3Djson', test_= img)) + self.assertEqual(top['compressed-clusters'], self.allocated_cluste= rs) + if __name__ =3D=3D '__main__': iotests.main(supported_fmts=3D['qcow2', 'qed'], supported_protocols=3D['file']) diff --git a/tests/qemu-iotests/030.out b/tests/qemu-iotests/030.out index 6d9bee1..af8dac1 100644 --- a/tests/qemu-iotests/030.out +++ b/tests/qemu-iotests/030.out @@ -1,5 +1,5 @@ -........................... +............................ ---------------------------------------------------------------------- -Ran 27 tests +Ran 28 tests =20 OK --=20 1.8.3.1