From nobody Sun May 12 12:31:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1562246096; cv=none; d=zoho.com; s=zohoarc; b=OPyHoO3hAusiqEXyhAQsmbqseX7ScmcWYP+U0hNc4o52fWcwhW90NsmA7BPpdTmsUhTPkH6s2rzC6Wd7dISVE25YSdieVcGTrFUEn3UYQLz/V+bR+3VaXNVBPYaRmj7FrmyZhqRAcMCpfTHvzuGW0vm+yEYNQApPBDSutW9aP/4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1562246096; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=olfpOGHSBLo+9pJp45Jt8mum4AHhSdtKWsoeg+f2u0g=; b=Ib9I8yeqCM6k+MW/otebnO9Smk3Sx/JbNWUmRluSjvYM2x40p85WPdIGCPdXNT0drDjScxb/Zchj5GdyUDX0vsRZ7SehrOHqKAMi9fqn8q12zAdEDIq1z4DkbT96fD1uc9Bf5BMDVRRhdjBPzUZBhn3cjjqZ10V8hyWZl74GxYw= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1562246096819846.1108964472903; Thu, 4 Jul 2019 06:14:56 -0700 (PDT) Received: from localhost ([::1]:45714 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hj1Zf-0001J2-OY for importer@patchew.org; Thu, 04 Jul 2019 09:14:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46102) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hj1V8-0004Y2-OJ for qemu-devel@nongnu.org; Thu, 04 Jul 2019 09:10:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hj1V3-0008Lu-QA for qemu-devel@nongnu.org; Thu, 04 Jul 2019 09:10:13 -0400 Received: from relay.sw.ru ([185.231.240.75]:54566) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hj1Uv-0008GT-Sv; Thu, 04 Jul 2019 09:10:02 -0400 Received: from [10.94.4.71] (helo=dptest2.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hj1Uq-000226-04; Thu, 04 Jul 2019 16:09:56 +0300 From: Denis Plotnikov To: kwolf@redhat.com, mreitz@redhat.com, eblake@redhat.com, armbru@redhat.com Date: Thu, 4 Jul 2019 16:09:47 +0300 Message-Id: <20190704130949.14017-2-dplotnikov@virtuozzo.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20190704130949.14017-1-dplotnikov@virtuozzo.com> References: <20190704130949.14017-1-dplotnikov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v2 1/3] qcow2: introduce compression type feature X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vsementsov@virtuozzo.com, den@virtuozzo.com, qemu-block@nongnu.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type. It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image. The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLI= B. It works roughly 2x faster than ZLIB providing a comparable compression rat= io and therefore provide a performance advantage in backup scenarios. The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions. Signed-off-by: Denis Plotnikov Acked-by: Markus Armbruster --- block/qcow2.c | 95 +++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 26 ++++++++--- docs/interop/qcow2.txt | 21 ++++++++- include/block/block_int.h | 1 + qapi/block-core.json | 22 ++++++++- 5 files changed, 155 insertions(+), 10 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 3ace3b2209..8fa932a349 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1202,6 +1202,32 @@ static int qcow2_update_options(BlockDriverState *bs= , QDict *options, return ret; } =20 +static int check_compression_type(BDRVQcow2State *s, Error **errp) +{ + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + break; + + default: + error_setg(errp, "qcow2: unknown compression type: %u", + s->compression_type); + return -ENOTSUP; + } + + /* + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB + * the incompatible feature flag must be set + */ + + if (s->compression_type !=3D QCOW2_COMPRESSION_TYPE_ZLIB && + !(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) { + error_setg(errp, "qcow2: Invalid compression type setting"); + return -EINVAL; + } + + return 0; +} + /* Called with s->lock held. */ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, Error **errp) @@ -1318,6 +1344,35 @@ static int coroutine_fn qcow2_do_open(BlockDriverSta= te *bs, QDict *options, s->compatible_features =3D header.compatible_features; s->autoclear_features =3D header.autoclear_features; =20 + /* + * Handle compression type + * Older qcow2 images don't contain the compression type header. + * Distinguish them by the header length and use + * the only valid (default) compression type in that case + */ + if (header.header_length > offsetof(QCowHeader, compression_type)) { + /* sanity check that we can read a compression type */ + size_t min_len =3D offsetof(QCowHeader, compression_type) + + sizeof(header.compression_type); + if (header.header_length < min_len) { + error_setg(errp, + "Could not read compression type." + "qcow2 header is too short"); + ret =3D -EINVAL; + goto fail; + } + + header.compression_type =3D be32_to_cpu(header.compression_type); + s->compression_type =3D header.compression_type; + } else { + s->compression_type =3D QCOW2_COMPRESSION_TYPE_ZLIB; + } + + ret =3D check_compression_type(s, errp); + if (ret) { + goto fail; + } + if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) { void *feature_table =3D NULL; qcow2_read_extensions(bs, header.header_length, ext_end, @@ -2434,6 +2489,13 @@ int qcow2_update_header(BlockDriverState *bs) total_size =3D bs->total_sectors * BDRV_SECTOR_SIZE; refcount_table_clusters =3D s->refcount_table_size >> (s->cluster_bits= - 3); =20 + ret =3D check_compression_type(s, NULL); + + if (ret) { + goto fail; + } + + *header =3D (QCowHeader) { /* Version 2 fields */ .magic =3D cpu_to_be32(QCOW_MAGIC), @@ -2456,6 +2518,7 @@ int qcow2_update_header(BlockDriverState *bs) .autoclear_features =3D cpu_to_be64(s->autoclear_features), .refcount_order =3D cpu_to_be32(s->refcount_order), .header_length =3D cpu_to_be32(header_length), + .compression_type =3D cpu_to_be32(s->compression_type), }; =20 /* For older versions, write a shorter header */ @@ -2553,6 +2616,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit =3D QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name =3D "lazy refcounts", }, + { + .type =3D QCOW2_FEAT_TYPE_INCOMPATIBLE, + .bit =3D QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name =3D "compression type", + }, }; =20 ret =3D header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE, @@ -3107,6 +3175,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options= , Error **errp) .refcount_table_offset =3D cpu_to_be64(cluster_size), .refcount_table_clusters =3D cpu_to_be32(1), .refcount_order =3D cpu_to_be32(refcount_order), + .compression_type =3D cpu_to_be32(QCOW2_COMPRESSION_TYPE= _ZLIB), .header_length =3D cpu_to_be32(sizeof(*header)), }; =20 @@ -3126,6 +3195,24 @@ qcow2_co_create(BlockdevCreateOptions *create_option= s, Error **errp) cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW); } =20 + if (qcow2_opts->has_compression_type && + qcow2_opts->compression_type !=3D QCOW2_COMPRESSION_TYPE_ZLIB) { + + switch(qcow2_opts->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + break; + + default: + error_setg_errno(errp, -EINVAL, "Unknown compression type"); + goto out; + } + + header->compression_type =3D cpu_to_be32(qcow2_opts->compression_t= ype); + + header->incompatible_features |=3D + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE); + } + ret =3D blk_pwrite(blk, 0, header, cluster_size, 0); g_free(header); if (ret < 0) { @@ -3307,6 +3394,7 @@ static int coroutine_fn qcow2_co_create_opts(const ch= ar *filename, QemuOpts *opt { BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT }, { BLOCK_OPT_COMPAT_LEVEL, "version" }, { BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" }, + { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" }, { NULL, NULL }, }; =20 @@ -4675,6 +4763,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(Blo= ckDriverState *bs, .data_file =3D g_strdup(s->image_data_file), .has_data_file_raw =3D has_data_file(bs), .data_file_raw =3D data_file_is_raw(bs), + .compression_type =3D s->compression_type, }; } else { /* if this assertion fails, this probably means a new version was @@ -5239,6 +5328,12 @@ static QemuOptsList qcow2_create_opts =3D { .help =3D "Width of a reference count entry in bits", .def_value_str =3D "16" }, + { + .name =3D BLOCK_OPT_COMPRESSION_TYPE, + .type =3D QEMU_OPT_STRING, + .help =3D "Compression method used for image clusters compress= ion", + .def_value_str =3D "zlib" + }, { /* end of list */ } } }; diff --git a/block/qcow2.h b/block/qcow2.h index fdee297f33..b44344c78e 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -135,6 +135,7 @@ typedef struct QCowHeader { =20 uint32_t refcount_order; uint32_t header_length; + uint32_t compression_type; } QEMU_PACKED QCowHeader; =20 typedef struct QEMU_PACKED QCowSnapshotHeader { @@ -198,16 +199,20 @@ enum { =20 /* Incompatible feature bits */ enum { - QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, - QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, - QCOW2_INCOMPAT_DATA_FILE_BITNR =3D 2, - QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BITNR, - QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_BITNR, - QCOW2_INCOMPAT_DATA_FILE =3D 1 << QCOW2_INCOMPAT_DATA_FILE_BITN= R, + QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, + QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, + QCOW2_INCOMPAT_DATA_FILE_BITNR =3D 2, + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR =3D 3, + QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BI= TNR, + QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_= BITNR, + QCOW2_INCOMPAT_DATA_FILE =3D 1 << QCOW2_INCOMPAT_DATA_FIL= E_BITNR, + QCOW2_INCOMPAT_COMPRESSION_TYPE =3D + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, =20 QCOW2_INCOMPAT_MASK =3D QCOW2_INCOMPAT_DIRTY | QCOW2_INCOMPAT_CORRUPT - | QCOW2_INCOMPAT_DATA_FILE, + | QCOW2_INCOMPAT_DATA_FILE + | QCOW2_INCOMPAT_COMPRESSION_TYPE, }; =20 /* Compatible feature bits */ @@ -350,6 +355,13 @@ typedef struct BDRVQcow2State { int nb_compress_threads; =20 BdrvChild *data_file; + /* + * Compression type used for the image. Default: 0 - ZLIB + * The image compression type is set on image creation. + * The only way to change the compression type is to convert the image + * with the desired compression type set + */ + uint32_t compression_type; } BDRVQcow2State; =20 typedef struct Qcow2COWRegion { diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e533..7cf068f814 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -109,7 +109,12 @@ in the description of a field. An External Data File Name header extensio= n may be present if this bit is set. =20 - Bits 3-63: Reserved (set to 0) + Bit 3: Compression type bit. The bit must be set = if + the compression type differs from default:= zlib. + If the compression type is default the bit= must + be unset. + + Bits 4-63: Reserved (set to 0) =20 80 - 87: compatible_features Bitmask of compatible features. An implementation can @@ -165,6 +170,20 @@ in the description of a field. Length of the header structure in bytes. For version 2 images, the length is always assumed to be 72 bytes. =20 + 104 - 107: compression_type + Defines the compression method used for compressed clu= sters. + A single compression type is applied to all compressed= image + clusters. + The compression type is set on image creation only. + The default compression type is zlib (value: 0). + When the compression type differs from the default + the compression type bit (incompatible feature bit 3) + must be set. + Qemu versions older than 4.1 can use images created wi= th + the default compression type without any additional + preparations and cannot use images created with any ot= her + compression type. + Directly after the image header, optional sections called header extension= s can be stored. Each extension has a structure like the following: =20 diff --git a/include/block/block_int.h b/include/block/block_int.h index 01e855a066..814917baec 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -58,6 +58,7 @@ #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits" #define BLOCK_OPT_DATA_FILE "data_file" #define BLOCK_OPT_DATA_FILE_RAW "data_file_raw" +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type" =20 #define BLOCK_PROBE_BUF_SIZE 512 =20 diff --git a/qapi/block-core.json b/qapi/block-core.json index 7ccbfff9d0..835dd3c37f 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -78,6 +78,8 @@ # # @bitmaps: A list of qcow2 bitmap details (since 4.0) # +# @compression-type: the image cluster compression method (since 4.1) +# # Since: 1.7 ## { 'struct': 'ImageInfoSpecificQCow2', @@ -89,7 +91,8 @@ '*corrupt': 'bool', 'refcount-bits': 'int', '*encrypt': 'ImageInfoSpecificQCow2Encryption', - '*bitmaps': ['Qcow2BitmapInfo'] + '*bitmaps': ['Qcow2BitmapInfo'], + 'compression-type': 'Qcow2CompressionType' } } =20 ## @@ -4206,6 +4209,18 @@ 'data': [ 'v2', 'v3' ] } =20 =20 +## +# @Qcow2CompressionType: +# +# Compression type used in qcow2 image file +# +# @zlib: zlib compression, see +# +# Since: 4.1 +## +{ 'enum': 'Qcow2CompressionType', + 'data': [ 'zlib' ] } + ## # @BlockdevCreateOptionsQcow2: # @@ -4228,6 +4243,8 @@ # @preallocation Preallocation mode for the new image (default: off) # @lazy-refcounts True if refcounts may be updated lazily (default: off) # @refcount-bits Width of reference counts in bits (default: 16) +# @compression-type The image cluster compression method +# (default: zlib, since 4.1) # # Since: 2.12 ## @@ -4243,7 +4260,8 @@ '*cluster-size': 'size', '*preallocation': 'PreallocMode', '*lazy-refcounts': 'bool', - '*refcount-bits': 'int' } } + '*refcount-bits': 'int', + '*compression-type': 'Qcow2CompressionType' } } =20 ## # @BlockdevCreateOptionsQed: --=20 2.17.0 From nobody Sun May 12 12:31:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1562245958; cv=none; d=zoho.com; s=zohoarc; b=G3rboeHKcYsgd5XihojIzmpKINVOtLCalXWwipym1J+YRDLiX+Nl3AVNT6SdCY0dVDZg/ZidI6UAgF9cytpynCRV3rLx+/a2xks98ZV9y4VojEux2O1c6wfJ0YLKodKRkxj/JSv4laM4Yc7QHDxq5xTtbuqhjD9Ye1saP/h/7IY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1562245958; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=33s3dPwReKRsGc5i2rkF9kYkMvib9PQZI5lgtcqgaog=; b=PSI138EwRzDeA+iA1kxj7KgsAYxpXlRNGiVI7ga2EXVW4JLCccqe1uMF7HPvvQ0pH6Wu0UJxTe30muKVOdX9xFshY5SlsgiycijvNfJK9dgsJ45FcqO6IJNgkcehL9tQ4ElYTKjCMxXX0GSniJJ/G8GjEPzX+72dyUD7IuLQ/ZA= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 156224595834788.61871130791826; Thu, 4 Jul 2019 06:12:38 -0700 (PDT) Received: from localhost ([::1]:45688 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hj1XO-0006xU-F1 for importer@patchew.org; Thu, 04 Jul 2019 09:12:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46104) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hj1V8-0004Y4-Oj for qemu-devel@nongnu.org; Thu, 04 Jul 2019 09:10:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hj1V5-0008N2-DJ for qemu-devel@nongnu.org; Thu, 04 Jul 2019 09:10:13 -0400 Received: from relay.sw.ru ([185.231.240.75]:54572) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hj1Uw-0008Gw-0W; Thu, 04 Jul 2019 09:10:02 -0400 Received: from [10.94.4.71] (helo=dptest2.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hj1Uq-000226-4J; Thu, 04 Jul 2019 16:09:56 +0300 From: Denis Plotnikov To: kwolf@redhat.com, mreitz@redhat.com, eblake@redhat.com, armbru@redhat.com Date: Thu, 4 Jul 2019 16:09:48 +0300 Message-Id: <20190704130949.14017-3-dplotnikov@virtuozzo.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20190704130949.14017-1-dplotnikov@virtuozzo.com> References: <20190704130949.14017-1-dplotnikov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v2 2/3] qcow2: rework the cluster compression routine X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vsementsov@virtuozzo.com, den@virtuozzo.com, qemu-block@nongnu.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The patch allow to process image compression type defined in the image header and choose an appropriate method for image clusters (de)compression. Signed-off-by: Denis Plotnikov --- block/qcow2.c | 93 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 73 insertions(+), 20 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 8fa932a349..a107f76e98 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -4005,8 +4005,11 @@ fail: } =20 /* - * qcow2_compress() + * qcow2_zlib_compress() * + * Compress @src_size bytes of data using zlib compression method + * + * @dest_size bytes. * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes * @@ -4014,8 +4017,8 @@ fail: * -1 destination buffer is not enough to store compressed data * -2 on any other error */ -static ssize_t qcow2_compress(void *dest, size_t dest_size, - const void *src, size_t src_size) +static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size, + const void *src, size_t src_size) { ssize_t ret; z_stream strm; @@ -4025,7 +4028,7 @@ static ssize_t qcow2_compress(void *dest, size_t dest= _size, ret =3D deflateInit2(&strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED, -12, 9, Z_DEFAULT_STRATEGY); if (ret !=3D Z_OK) { - return -2; + return -EIO; } =20 /* strm.next_in is not const in old zlib versions, such as those used = on @@ -4039,7 +4042,7 @@ static ssize_t qcow2_compress(void *dest, size_t dest= _size, if (ret =3D=3D Z_STREAM_END) { ret =3D dest_size - strm.avail_out; } else { - ret =3D (ret =3D=3D Z_OK ? -1 : -2); + ret =3D (ret =3D=3D Z_OK ? -ENOMEM : -EIO); } =20 deflateEnd(&strm); @@ -4048,10 +4051,10 @@ static ssize_t qcow2_compress(void *dest, size_t de= st_size, } =20 /* - * qcow2_decompress() + * qcow2_zlib_decompress() * * Decompress some data (not more than @src_size bytes) to produce exactly - * @dest_size bytes. + * @dest_size bytes using zlib compression method * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes @@ -4059,8 +4062,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest= _size, * Returns: 0 on success * -1 on fail */ -static ssize_t qcow2_decompress(void *dest, size_t dest_size, - const void *src, size_t src_size) +static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, + const void *src, size_t src_size) { int ret =3D 0; z_stream strm; @@ -4073,7 +4076,7 @@ static ssize_t qcow2_decompress(void *dest, size_t de= st_size, =20 ret =3D inflateInit2(&strm, -12); if (ret !=3D Z_OK) { - return -1; + return -EIO; } =20 ret =3D inflate(&strm, Z_FINISH); @@ -4081,7 +4084,7 @@ static ssize_t qcow2_decompress(void *dest, size_t de= st_size, /* We approve Z_BUF_ERROR because we need @dest buffer to be fille= d, but * @src buffer may be processed partly (because in qcow2 we know s= ize of * compressed data with precision of one sector) */ - ret =3D -1; + ret =3D -EIO; } =20 inflateEnd(&strm); @@ -4153,20 +4156,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *de= st, size_t dest_size, return arg.ret; } =20 +/* + * qcow2_co_compress() + * + * Compress @src_size bytes of data using the compression + * method defined by the image compression type + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: 0 on success + * a negative error code on fail + */ static ssize_t coroutine_fn qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, const void *src, size_t src_size) { - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, - qcow2_compress); + BDRVQcow2State *s =3D bs->opaque; + Qcow2CompressFunc fn; + + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + fn =3D qcow2_zlib_compress; + break; + + default: + return -ENOTSUP; + } + + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn); } =20 +/* + * qcow2_co_decompress() + * + * Decompress some data (not more than @src_size bytes) to produce exactly + * @dest_size bytes using the compression method defined by the image + * compression type + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: 0 on success + * a negative error code on fail + */ static ssize_t coroutine_fn qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, const void *src, size_t src_size) { - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, - qcow2_decompress); + BDRVQcow2State *s =3D bs->opaque; + Qcow2CompressFunc fn; + + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + fn =3D qcow2_zlib_decompress; + break; + + default: + return -ENOTSUP; + } + + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn); } =20 /* XXX: put compressed sectors first, then all the cluster aligned @@ -4178,7 +4228,7 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uin= t64_t offset, BDRVQcow2State *s =3D bs->opaque; QEMUIOVector hd_qiov; int ret; - size_t out_len; + ssize_t out_len; uint8_t *buf, *out_buf; uint64_t cluster_offset; =20 @@ -4217,16 +4267,19 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, u= int64_t offset, =20 out_len =3D qcow2_co_compress(bs, out_buf, s->cluster_size - 1, buf, s->cluster_size); - if (out_len =3D=3D -2) { - ret =3D -EINVAL; - goto fail; - } else if (out_len =3D=3D -1) { + if (out_len =3D=3D -ENOMEM) { /* could not compress: write normal cluster */ ret =3D qcow2_co_pwritev(bs, offset, bytes, qiov, 0); if (ret < 0) { goto fail; } goto success; + } else if (out_len < 0) { + /* + * encounter other compression issues propagate error to the upper= level + */ + ret =3D out_len; + goto fail; } =20 qemu_co_mutex_lock(&s->lock); --=20 2.17.0 From nobody Sun May 12 12:31:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1562245975; cv=none; d=zoho.com; s=zohoarc; b=GnCci2pv+jUsDLXhNfBK6g7Cx4kTzFMTwYuPurFW/It8M3RptKfkYQWvqBebSJDtzOgMnXzOyf9LZldNQykrbjK4NvMd1aqGhbm7V/FZhewx7rpneqbDcIEBJHqdZ6LM9i8KVzw+m4ZuOLmyU9GqX2toZo/gK8nKE9Kwrj/Nkfo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1562245975; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=98C1+rHZIIK0EE62niMQ6D6BkdkXIArVE+hGpx5a/4k=; b=VRDDOWkO5RRbsHHPJhaPyWc7EgUyvsc3Hip7lz1zpyU0jTI3ZP2L1hPROTTfDmcX5ImEtpoBsg8IYqhvhBHgrUh5ytz2URcQqYgQkyykKg06dWMtYJ0dy305ozq9sP3MIV0z0WUIAgi+fUJFP+nHpGlls1xyqrfnkEjCzr2tnAA= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1562245975725857.564264197822; Thu, 4 Jul 2019 06:12:55 -0700 (PDT) Received: from localhost ([::1]:45690 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hj1Xa-0006zX-Sg for importer@patchew.org; Thu, 04 Jul 2019 09:12:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46083) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hj1V5-0004Wb-1X for qemu-devel@nongnu.org; Thu, 04 Jul 2019 09:10:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hj1V2-0008LF-KL for qemu-devel@nongnu.org; Thu, 04 Jul 2019 09:10:10 -0400 Received: from relay.sw.ru ([185.231.240.75]:54570) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hj1Uw-0008Gs-16; Thu, 04 Jul 2019 09:10:03 -0400 Received: from [10.94.4.71] (helo=dptest2.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.92) (envelope-from ) id 1hj1Uq-000226-7z; Thu, 04 Jul 2019 16:09:56 +0300 From: Denis Plotnikov To: kwolf@redhat.com, mreitz@redhat.com, eblake@redhat.com, armbru@redhat.com Date: Thu, 4 Jul 2019 16:09:49 +0300 Message-Id: <20190704130949.14017-4-dplotnikov@virtuozzo.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20190704130949.14017-1-dplotnikov@virtuozzo.com> References: <20190704130949.14017-1-dplotnikov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v2 3/3] qcow2: add zstd cluster compression X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vsementsov@virtuozzo.com, den@virtuozzo.com, qemu-block@nongnu.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, by the moment, has been the only compression method available. The performance test results: Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest. Image cluster size: 64K. Image on disk size: 2.2G The test was conducted with brd disk to reduce the influence of disk subsystem to the test results. The results is given in seconds. compress cmd: time ./qemu-img convert -O qcow2 -c -o compression_type=3D[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img compression decompression zlib zstd zlib zstd Acked-by: Markus Armbruster ------------------------------------------------------------ real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0 Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G Signed-off-by: Denis Plotnikov --- block/qcow2.c | 99 ++++++++++++++++++++++++++++++++++++++++++ configure | 32 ++++++++++++++ docs/interop/qcow2.txt | 19 ++++++++ qapi/block-core.json | 3 +- 4 files changed, 152 insertions(+), 1 deletion(-) diff --git a/block/qcow2.c b/block/qcow2.c index a107f76e98..252eba636f 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -27,6 +27,11 @@ #define ZLIB_CONST #include =20 +#ifdef CONFIG_ZSTD +#include +#include +#endif + #include "block/block_int.h" #include "block/qdict.h" #include "sysemu/block-backend.h" @@ -1206,6 +1211,9 @@ static int check_compression_type(BDRVQcow2State *s, = Error **errp) { switch (s->compression_type) { case QCOW2_COMPRESSION_TYPE_ZLIB: +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: +#endif break; =20 default: @@ -3200,6 +3208,9 @@ qcow2_co_create(BlockdevCreateOptions *create_options= , Error **errp) =20 switch(qcow2_opts->compression_type) { case QCOW2_COMPRESSION_TYPE_ZLIB: +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: +#endif break; =20 default: @@ -4092,6 +4103,84 @@ static ssize_t qcow2_zlib_decompress(void *dest, siz= e_t dest_size, return ret; } =20 +#ifdef CONFIG_ZSTD +/* + * qcow2_zstd_compress() + * + * Compress @src_size bytes of data using zstd compression method + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: compressed size on success + * a negative error code on fail + */ + +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + ssize_t ret; + uint32_t *c_size =3D dest; + /* steal some bytes to store compressed chunk size */ + char *d_buf =3D ((char *) dest) + sizeof(*c_size); + + if (dest_size < sizeof(*c_size)) { + return -ENOMEM; + } + + dest_size -=3D sizeof(*c_size); + + ret =3D ZSTD_compress(d_buf, dest_size, src, src_size, 5); + + if (ZSTD_isError(ret)) { + if (ret =3D=3D ZSTD_error_dstSize_tooSmall) { + return -ENOMEM; + } else { + return -EIO; + } + } + + /* store the compressed chunk size in the very beginning of the buffer= */ + *c_size =3D ret; + + return ret + sizeof(ret); +} + +/* + * qcow2_zstd_decompress() + * + * Decompress some data (not more than @src_size bytes) to produce exactly + * @dest_size bytes using zstd compression method + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: 0 on success + * -EIO on fail + */ + +static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + ssize_t ret; + /* + * zstd decompress wants to know the exact lenght of the data + * for that purpose, on the compression the length is stored in + * the very beginning of the compressed buffer + */ + const uint32_t *s_size =3D src; + const char *s_buf =3D ((char *) src) + sizeof(*s_size); + + ret =3D ZSTD_decompress(dest, dest_size, s_buf, *s_size); + + if (ZSTD_isError(ret)) { + return -EIO; + } + + return 0; +} +#endif + #define MAX_COMPRESS_THREADS 4 =20 typedef ssize_t (*Qcow2CompressFunc)(void *dest, size_t dest_size, @@ -4180,6 +4269,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, = size_t dest_size, fn =3D qcow2_zlib_compress; break; =20 +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: + fn =3D qcow2_zstd_compress; + break; +#endif default: return -ENOTSUP; } @@ -4212,6 +4306,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest= , size_t dest_size, fn =3D qcow2_zlib_decompress; break; =20 +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: + fn =3D qcow2_zstd_decompress; + break; +#endif default: return -ENOTSUP; } diff --git a/configure b/configure index 1c563a7027..57a80e38e7 100755 --- a/configure +++ b/configure @@ -433,6 +433,7 @@ opengl_dmabuf=3D"no" cpuid_h=3D"no" avx2_opt=3D"" zlib=3D"yes" +zstd=3D"" capstone=3D"" lzo=3D"" snappy=3D"" @@ -1333,6 +1334,10 @@ for opt do ;; --disable-lzfse) lzfse=3D"no" ;; + --enable-zstd) zstd=3D"yes" + ;; + --disable-zstd) zstd=3D"no" + ;; --enable-guest-agent) guest_agent=3D"yes" ;; --disable-guest-agent) guest_agent=3D"no" @@ -1788,6 +1793,7 @@ disabled with --disable-FEATURE, default is enabled i= f available: (for reading bzip2-compressed dmg images) lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) + zstd support of zstd compression library seccomp seccomp support coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backend @@ -2374,6 +2380,29 @@ EOF fi fi =20 +######################################### +# zstd check + +if test "$zstd" !=3D "no" ; then + if $pkg_config --exists libzstd; then + zstd_cflags=3D$($pkg_config --cflags libzstd) + zstd_libs=3D$($pkg_config --libs libzstd) + QEMU_CFLAGS=3D"$zstd_cflags $QEMU_CFLAGS" + LIBS=3D"$zstd_libs $LIBS" + else + cat > $TMPC << EOF +#include +int main(void) { ZSTD_versionNumber(); return 0; } +EOF + if compile_prog "" "-lzstd" ; then + LIBS=3D"$LIBS -lzstd" + else + error_exit "zstd check failed" \ + "Make sure to have the zstd libs and headers installed." + fi + fi +fi + ########################################## # libseccomp check =20 @@ -7253,6 +7282,9 @@ fi if test "$sheepdog" =3D "yes" ; then echo "CONFIG_SHEEPDOG=3Dy" >> $config_host_mak fi +if test "$zstd" =3D "yes" ; then + echo "CONFIG_ZSTD=3Dy" >> $config_host_mak +fi =20 if test "$tcg_interpreter" =3D "yes"; then QEMU_INCLUDES=3D"-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES" diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index 7cf068f814..4344e858cb 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -538,6 +538,9 @@ Compressed Clusters Descriptor (x =3D 62 - (cluster_bit= s - 8)): Another compressed cluster may map to the tail of the = final sector used by this compressed cluster. =20 + The layout of the compressed data depends on the compr= ession + type used for the image (see compressed cluster layout= ). + If a cluster is unallocated, read requests shall read the data from the ba= cking file (except if bit 0 in the Standard Cluster Descriptor is set). If there= is no backing file or the backing file is smaller than the image, they shall = read @@ -790,3 +793,19 @@ In the image file the 'enabled' state is reflected by = the 'auto' flag. If this flag is set, the software must consider the bitmap as 'enabled' and start tracking virtual disk changes to this bitmap from the first write to the virtual disk. If this flag is not set then the bitmap is disabled. + +=3D=3D=3D Compressed cluster layout =3D=3D=3D + +The compressed cluster data may have a different layout depending on the +compression type used for the image, and store specific data for the parti= cular +compression type. + +Compressed data layout for the available compression types: +(x =3D data_space_length - 1) + + zlib: + Byte 0 - x: the compressed data content + all the space provided used for compressed data + zstd: + Byte 0 - 3: the length of compressed data + 4 - x: the compressed data content diff --git a/qapi/block-core.json b/qapi/block-core.json index 835dd3c37f..2021e03a84 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4215,11 +4215,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see +# @zstd: zstd compression, see # # Since: 4.1 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } =20 ## # @BlockdevCreateOptionsQcow2: --=20 2.17.0