From nobody Sat May 4 07:11:32 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1559054400; cv=none; d=zoho.com; s=zohoarc; b=Z5QRdxp6fGPPWFGtFxdyRR9CeXXzF58Z7U6oPLgu4sQE2rAyFSzMl3CEPnRCfbUbXY+ltccXL1iemTy/hVkHaJISHCd+P/4/rSa0FQ1TouAVuC7hNC6SgffXA+ofbLH50YCIAQIotZOscA9xjN9/JxRGIEgUstb+XjuqrOlhRNw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559054400; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=mmvBgS9tMA1+JbAiiUUXbbHsMf08YqKpsE+helsusSE=; b=PRamGyuzA8ijjNYoTWw4lnhX4GvLVpspSKfy9n7XFRmpShIpsDXlJHjoUNHYytw5ErpDMpO1dB2RI/s4lPYLiXYpA/hUR5d45rHsz4VMEAZwXFn+vtB2iVHeTLYDUGxZxSw6IEOGs1RYsH0k1Ndm4NpoW3sezbepZDZfwPlVA20= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559054400014657.3088032968931; Tue, 28 May 2019 07:40:00 -0700 (PDT) Received: from localhost ([127.0.0.1]:35999 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVdGU-0002kq-FB for importer@patchew.org; Tue, 28 May 2019 10:39:46 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50699) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVdFK-0002EK-3j for qemu-devel@nongnu.org; Tue, 28 May 2019 10:38:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVdF8-0007ca-2q for qemu-devel@nongnu.org; Tue, 28 May 2019 10:38:26 -0400 Received: from relay.sw.ru ([185.231.240.75]:46350) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVdEo-0006u7-99; Tue, 28 May 2019 10:38:02 -0400 Received: from [10.94.4.71] (helo=dptest2.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.91) (envelope-from ) id 1hVdEM-0004ye-0f; Tue, 28 May 2019 17:37:34 +0300 From: Denis Plotnikov To: kwolf@redhat.com, mreitz@redhat.com, eblake@redhat.com, armbru@redhat.com, qemu-block@nongnu.org Date: Tue, 28 May 2019 17:37:25 +0300 Message-Id: <20190528143727.10529-2-dplotnikov@virtuozzo.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20190528143727.10529-1-dplotnikov@virtuozzo.com> References: <20190528143727.10529-1-dplotnikov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v0 1/3] qcow2: introduce compression type feature X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vsementsov@virtuozzo.com, qemu-devel@nongnu.org, den@virtuozzo.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type. It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image. The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLI= B. It works roughly x2 faster than ZLIB providing a comparable compression rat= io and therefore provide a performance advantage in backup scenarios. The default compression is ZLIB. Images created with ZLIB compression type is backward compatible with older qemu versions. Signed-off-by: Denis Plotnikov --- block/qcow2.c | 61 +++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 29 ++++++++++++++----- docs/interop/qcow2.txt | 37 +++++++++++++++++++++++- include/block/block_int.h | 1 + qapi/block-core.json | 34 ++++++++++++++++++++-- 5 files changed, 151 insertions(+), 11 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 3ace3b2209..c4b5b93408 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -74,6 +74,7 @@ typedef struct { #define QCOW2_EXT_MAGIC_CRYPTO_HEADER 0x0537be77 #define QCOW2_EXT_MAGIC_BITMAPS 0x23852875 #define QCOW2_EXT_MAGIC_DATA_FILE 0x44415441 +#define QCOW2_EXT_MAGIC_COMPRESSION_TYPE 0x434D5052 =20 static int coroutine_fn qcow2_co_preadv_compressed(BlockDriverState *bs, @@ -398,6 +399,13 @@ static int qcow2_read_extensions(BlockDriverState *bs,= uint64_t start_offset, #endif break; =20 + case QCOW2_EXT_MAGIC_COMPRESSION_TYPE: + /* + * Setting compression type to BDRVQcow2State->compression_type + * from the image header is going to be here + */ + break; + case QCOW2_EXT_MAGIC_DATA_FILE: { s->image_data_file =3D g_malloc0(ext.len + 1); @@ -2553,6 +2561,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit =3D QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name =3D "lazy refcounts", }, + { + .type =3D QCOW2_FEAT_TYPE_INCOMPATIBLE, + .bit =3D QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name =3D "compression type", + }, }; =20 ret =3D header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE, @@ -2583,6 +2596,22 @@ int qcow2_update_header(BlockDriverState *bs) buflen -=3D ret; } =20 + /* Compression type extension */ + if (s->compression_type !=3D 0) { + Qcow2CompressionTypeExt comp_header =3D { + .compression_type =3D cpu_to_be32(s->compression_type), + }; + ret =3D header_ext_add(buf, QCOW2_EXT_MAGIC_COMPRESSION_TYPE, + &comp_header, + sizeof(comp_header), + buflen); + if (ret < 0) { + goto fail; + } + buf +=3D ret; + buflen -=3D ret; + } + /* Keep unknown header extensions */ QLIST_FOREACH(uext, &s->unknown_header_ext, next) { ret =3D header_ext_add(buf, uext->magic, uext->data, uext->len, bu= flen); @@ -3184,6 +3213,29 @@ qcow2_co_create(BlockdevCreateOptions *create_option= s, Error **errp) s->image_data_file =3D g_strdup(data_bs->filename); } =20 + /* + * default compression type is ZLIB: 0 + * to apply it, there is no need for any header modification + */ + if (qcow2_opts->has_compression_type && + qcow2_opts->compression_type !=3D QCOW2_COMPRESSION_TYPE_ZLIB) { + + BDRVQcow2State *s =3D blk_bs(blk)->opaque; + + /* + * check for known compression types + * ZLIB shouldn't be here since it's the default + */ + switch (qcow2_opts->compression_type) { + default: + error_setg_errno(errp, -EINVAL, "Unknown compression type"); + goto out; + } + s->incompatible_features |=3D QCOW2_INCOMPAT_COMPRESSION_TYPE; + s->compression_type =3D qcow2_opts->compression_type; + } + + /* Create a full header (including things like feature table) */ ret =3D qcow2_update_header(blk_bs(blk)); if (ret < 0) { @@ -3307,6 +3359,7 @@ static int coroutine_fn qcow2_co_create_opts(const ch= ar *filename, QemuOpts *opt { BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT }, { BLOCK_OPT_COMPAT_LEVEL, "version" }, { BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" }, + { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" }, { NULL, NULL }, }; =20 @@ -4675,6 +4728,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(Blo= ckDriverState *bs, .data_file =3D g_strdup(s->image_data_file), .has_data_file_raw =3D has_data_file(bs), .data_file_raw =3D data_file_is_raw(bs), + .has_compression_type =3D true, + .compression_type =3D s->compression_type }; } else { /* if this assertion fails, this probably means a new version was @@ -5239,6 +5294,12 @@ static QemuOptsList qcow2_create_opts =3D { .help =3D "Width of a reference count entry in bits", .def_value_str =3D "16" }, + { + .name =3D BLOCK_OPT_COMPRESSION_TYPE, + .type =3D QEMU_OPT_STRING, + .help =3D "Compression method used for image clusters compress= ion", + .def_value_str =3D "0" + }, { /* end of list */ } } }; diff --git a/block/qcow2.h b/block/qcow2.h index fdee297f33..b70da3138d 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -198,16 +198,20 @@ enum { =20 /* Incompatible feature bits */ enum { - QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, - QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, - QCOW2_INCOMPAT_DATA_FILE_BITNR =3D 2, - QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BITNR, - QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_BITNR, - QCOW2_INCOMPAT_DATA_FILE =3D 1 << QCOW2_INCOMPAT_DATA_FILE_BITN= R, + QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, + QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, + QCOW2_INCOMPAT_DATA_FILE_BITNR =3D 2, + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR =3D 3, + QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BI= TNR, + QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_= BITNR, + QCOW2_INCOMPAT_DATA_FILE =3D 1 << QCOW2_INCOMPAT_DATA_FIL= E_BITNR, + QCOW2_INCOMPAT_COMPRESSION_TYPE =3D + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, =20 QCOW2_INCOMPAT_MASK =3D QCOW2_INCOMPAT_DIRTY | QCOW2_INCOMPAT_CORRUPT - | QCOW2_INCOMPAT_DATA_FILE, + | QCOW2_INCOMPAT_DATA_FILE + | QCOW2_INCOMPAT_COMPRESSION_TYPE, }; =20 /* Compatible feature bits */ @@ -263,6 +267,10 @@ typedef struct Qcow2BitmapHeaderExt { uint64_t bitmap_directory_offset; } QEMU_PACKED Qcow2BitmapHeaderExt; =20 +typedef struct Qcow2CompressionTypeExt { + uint32_t compression_type; +} QEMU_PACKED Qcow2CompressionTypeExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size; @@ -350,6 +358,13 @@ typedef struct BDRVQcow2State { int nb_compress_threads; =20 BdrvChild *data_file; + /* + * Compression type used for the image. Default: 0 - ZLIB + * The image compression type is set on image creation. + * The only way to change the compression type is to convert the image + * with the desired compression type set + */ + uint32_t compression_type; } BDRVQcow2State; =20 typedef struct Qcow2COWRegion { diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e533..cebcbc4f2f 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -109,7 +109,14 @@ in the description of a field. An External Data File Name header extensio= n may be present if this bit is set. =20 - Bits 3-63: Reserved (set to 0) + Bit 3: Compression type. If the bit is set, then = the + type of compression the image uses is set = in the + header extension. When the bit is set the + compression type extension header must be = present. + When the bit is not set the compression ty= pe + header must absent. + + Bits 4-63: Reserved (set to 0) =20 80 - 87: compatible_features Bitmask of compatible features. An implementation can @@ -175,6 +182,7 @@ be stored. Each extension has a structure like the foll= owing: 0x23852875 - Bitmaps extension 0x0537be77 - Full disk encryption header pointer 0x44415441 - External data file name string + 0x434D5052 - Compression type extension other - Unknown header extension, can be safe= ly ignored =20 @@ -771,3 +779,30 @@ In the image file the 'enabled' state is reflected by = the 'auto' flag. If this flag is set, the software must consider the bitmap as 'enabled' and start tracking virtual disk changes to this bitmap from the first write to the virtual disk. If this flag is not set then the bitmap is disabled. + + +=3D=3D Compression type extension =3D=3D + +The compression type extension is an optional header extension. It stores = the +compression type used for disk clusters (de)compression. +A single compression type is applied to all compressed disk clusters, +with no way to change compression types per cluster. Two clusters of the i= mage +couldn't be compressed with different compression types. + +The compression type is set on image creation. The only way to change +the compression type is to convert the image explicitly. + +The compression type extension is present if and only if the incompatible +compression type bit is set. When the bit is not set the compression type +header must be absent. + +When the compression type bit is not set and the compression type header +extension is absent, ZLIB compression is used for compressed clusters. +This defines default image compression type: ZLIB. +Qemu < 4.1 can use images created with compression type ZLIB without any +additional preparations and cannot use images created with compression +types !=3D ZLIB. + +Available compression types: + 0: ZLIB + 1: ZSTD diff --git a/include/block/block_int.h b/include/block/block_int.h index 01e855a066..814917baec 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -58,6 +58,7 @@ #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits" #define BLOCK_OPT_DATA_FILE "data_file" #define BLOCK_OPT_DATA_FILE_RAW "data_file_raw" +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type" =20 #define BLOCK_PROBE_BUF_SIZE 512 =20 diff --git a/qapi/block-core.json b/qapi/block-core.json index 7ccbfff9d0..59610153fd 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -78,6 +78,9 @@ # # @bitmaps: A list of qcow2 bitmap details (since 4.0) # +# @compression-type: the compression method used for image clusters +# compression (since 4.1) +# # Since: 1.7 ## { 'struct': 'ImageInfoSpecificQCow2', @@ -89,7 +92,8 @@ '*corrupt': 'bool', 'refcount-bits': 'int', '*encrypt': 'ImageInfoSpecificQCow2Encryption', - '*bitmaps': ['Qcow2BitmapInfo'] + '*bitmaps': ['Qcow2BitmapInfo'], + '*compression-type': 'Qcow2CompressionType' } } =20 ## @@ -3119,6 +3123,10 @@ # an image, the data file name is loaded from the = image # file. (since 4.0) # +# @compression-type: compression method to use for image clusters com= pression +# The comression method is set on image creation a= nd can +# be changed via image converting only. (since 4.1) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsQcow2', @@ -3134,7 +3142,8 @@ '*refcount-cache-size': 'int', '*cache-clean-interval': 'int', '*encrypt': 'BlockdevQcow2Encryption', - '*data-file': 'BlockdevRef' } } + '*data-file': 'BlockdevRef', + '*compression-type': 'Qcow2CompressionType' } } =20 ## # @SshHostKeyCheckMode: @@ -4206,6 +4215,19 @@ 'data': [ 'v2', 'v3' ] } =20 =20 +## +# @Qcow2CompressionType: +# +# Compression type used in qcow2 image file +# +# @zlib : gzip compressor +# @zstd : zstd compression +# +# Since: 4.1 +## +{ 'enum': 'Qcow2CompressionType', + 'data': [ 'zlib', 'zstd' ] } + ## # @BlockdevCreateOptionsQcow2: # @@ -4228,6 +4250,11 @@ # @preallocation Preallocation mode for the new image (default: off) # @lazy-refcounts True if refcounts may be updated lazily (default: off) # @refcount-bits Width of reference counts in bits (default: 16) +# @compression-type Compression method used for image compressed clusters +# (default: zlib(gzip), since 4.1). +# Available types: +# zlib +# zstd # # Since: 2.12 ## @@ -4243,7 +4270,8 @@ '*cluster-size': 'size', '*preallocation': 'PreallocMode', '*lazy-refcounts': 'bool', - '*refcount-bits': 'int' } } + '*refcount-bits': 'int', + '*compression-type': 'Qcow2CompressionType' } } =20 ## # @BlockdevCreateOptionsQed: --=20 2.17.0 From nobody Sat May 4 07:11:32 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1559054494; cv=none; d=zoho.com; s=zohoarc; b=Ly+o68VvbalPLvD92G0qoPMSIH7kMM5EJTf9us++KoLLDwh8gWqFj/VfhI0kUZcT2xNuNotHKkJmvOhbVj+uVklf/hXj/l/VuepE3YLX+CDEvvSiGESq9Yfc4Sq6UKkBzJQfN91dD44P+7liHJ5qRe9DSdnKEecYVMi2UHlsAOE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559054494; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=8gdqjmaM5j4NJM6tctURoN6rOW66hd9TcsNdRCMxcyY=; b=RSeEA+SoIrHrYPiXcsnhcK3+moXaOc5U7vpoAk9U6hKDLUXG/UhIPjzL9Ns714otRBXD4Dxfzbyolnj7/crOmM8NK5OpWyEmDF2vW5PTRdj5yhpzDAELXSoo/Ae+8MBCTAptdTAe0JZ/CWcLQXC4tT6gXEqkDVp5oVwvAE+G+qs= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559054494335728.2806508975065; Tue, 28 May 2019 07:41:34 -0700 (PDT) Received: from localhost ([127.0.0.1]:36048 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVdI7-00040O-Ay for importer@patchew.org; Tue, 28 May 2019 10:41:27 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50816) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVdFj-0002QU-5w for qemu-devel@nongnu.org; Tue, 28 May 2019 10:39:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVdFX-0007tz-0b for qemu-devel@nongnu.org; Tue, 28 May 2019 10:38:49 -0400 Received: from relay.sw.ru ([185.231.240.75]:46362) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVdFR-0006uA-PB; Tue, 28 May 2019 10:38:42 -0400 Received: from [10.94.4.71] (helo=dptest2.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.91) (envelope-from ) id 1hVdEM-0004ye-6p; Tue, 28 May 2019 17:37:34 +0300 From: Denis Plotnikov To: kwolf@redhat.com, mreitz@redhat.com, eblake@redhat.com, armbru@redhat.com, qemu-block@nongnu.org Date: Tue, 28 May 2019 17:37:26 +0300 Message-Id: <20190528143727.10529-3-dplotnikov@virtuozzo.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20190528143727.10529-1-dplotnikov@virtuozzo.com> References: <20190528143727.10529-1-dplotnikov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v0 2/3] qcow2: add compression type processing X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vsementsov@virtuozzo.com, qemu-devel@nongnu.org, den@virtuozzo.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" With the patch, qcow2 is able to process image compression type defined in the image header and choose the corresponding method for clusters compressing. Also, it rework the cluster compression code for adding more compression types. Signed-off-by: Denis Plotnikov --- block/qcow2.c | 103 ++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 92 insertions(+), 11 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index c4b5b93408..90f15cc3c9 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -400,11 +400,39 @@ static int qcow2_read_extensions(BlockDriverState *bs= , uint64_t start_offset, break; =20 case QCOW2_EXT_MAGIC_COMPRESSION_TYPE: + /* Compression type always goes with the compression type bit = set */ + if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TY= PE)) { + error_setg(errp, + "compression_type_ext: " + "expect compression type bit set"); + return -EINVAL; + } + + ret =3D bdrv_pread(bs->file, offset, &s->compression_type, ext= .len); + s->compression_type =3D be32_to_cpu(s->compression_type); + + if (ret < 0) { + error_setg_errno(errp, -ret, + "ERROR: Could not read compression type"); + return ret; + } + /* - * Setting compression type to BDRVQcow2State->compression_type - * from the image header is going to be here + * The default compression type is not allowed when the extens= ion + * is present. ZLIB is used as the default compression type. + * When compression type extension header is present then + * compression_type should have a value different from the def= ault. */ - break; + if (s->compression_type =3D=3D QCOW2_COMPRESSION_TYPE_ZLIB) { + error_setg(errp, + "compression_type_ext:" + "invalid compression type %d", + QCOW2_COMPRESSION_TYPE_ZLIB); + } +#ifdef DEBUG_EXT + printf("Qcow2: image compression type %s\n", s->compression_ty= pe); +#endif + break; =20 case QCOW2_EXT_MAGIC_DATA_FILE: { @@ -1492,6 +1520,9 @@ static int coroutine_fn qcow2_do_open(BlockDriverStat= e *bs, QDict *options, QLIST_INIT(&s->cluster_allocs); QTAILQ_INIT(&s->discards); =20 + /* Set default compression type */ + s->compression_type =3D QCOW2_COMPRESSION_TYPE_ZLIB; + /* read qcow2 extensions */ if (qcow2_read_extensions(bs, header.header_length, ext_end, NULL, flags, &update_header, &local_err)) { @@ -1500,6 +1531,34 @@ static int coroutine_fn qcow2_do_open(BlockDriverSta= te *bs, QDict *options, goto fail; } =20 + /* + * The compression type is set on the extension header processing + * if the compression type extension header is present. + * When the compression type is different from ZLIB (default) there + * should be both the compression type bit and the compression + * type extension header set. When the ZLIB (default) compression + * type is used there should be neither the compression type bit nor + * the compression type extension header set. + */ + + if ((s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE) & + (s->compression_type =3D=3D QCOW2_COMPRESSION_TYPE_ZLIB)) { + error_setg(errp, "Illegal compression type setting"); + ret =3D -EINVAL; + goto fail; + } + + /* Check available compression types */ + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + break; + + default: + error_setg(errp, "Unknown compression type"); + ret =3D -EINVAL; + goto fail; + } + /* Open external data file */ s->data_file =3D bdrv_open_child(NULL, options, "data-file", bs, &chil= d_file, true, &local_err); @@ -3970,7 +4029,7 @@ fail: } =20 /* - * qcow2_compress() + * zlib_compress() * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes @@ -3979,7 +4038,7 @@ fail: * -1 destination buffer is not enough to store compressed data * -2 on any other error */ -static ssize_t qcow2_compress(void *dest, size_t dest_size, +static ssize_t zlib_compress(void *dest, size_t dest_size, const void *src, size_t src_size) { ssize_t ret; @@ -4013,7 +4072,7 @@ static ssize_t qcow2_compress(void *dest, size_t dest= _size, } =20 /* - * qcow2_decompress() + * zlib_decompress() * * Decompress some data (not more than @src_size bytes) to produce exactly * @dest_size bytes. @@ -4024,7 +4083,7 @@ static ssize_t qcow2_compress(void *dest, size_t dest= _size, * Returns: 0 on success * -1 on fail */ -static ssize_t qcow2_decompress(void *dest, size_t dest_size, +static ssize_t zlib_decompress(void *dest, size_t dest_size, const void *src, size_t src_size) { int ret =3D 0; @@ -4122,16 +4181,38 @@ static ssize_t coroutine_fn qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, const void *src, size_t src_size) { - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, - qcow2_compress); + BDRVQcow2State *s =3D bs->opaque; + Qcow2CompressFunc fn; + + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + fn =3D zlib_compress; + break; + + default: + return -ENOTSUP; + } + + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn); } =20 static ssize_t coroutine_fn qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, const void *src, size_t src_size) { - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, - qcow2_decompress); + BDRVQcow2State *s =3D bs->opaque; + Qcow2CompressFunc fn; + + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + fn =3D zlib_decompress; + break; + + default: + return -ENOTSUP; + } + + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn); } =20 /* XXX: put compressed sectors first, then all the cluster aligned --=20 2.17.0 From nobody Sat May 4 07:11:32 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1559054423; cv=none; d=zoho.com; s=zohoarc; b=nRmenMLOdzoAG70D7XlClzwy68VVH50VQO2EqfHAPW+N9BMZrboIcSGKGCbjThe+sik1Tp7T7YnMRJgP+ChL32wvOA6veARN16Ug5B/EvMiV6qL96N1FQRVQkh09uaXM6aKdCBSxhK8beEiS9+k9S8q2q6uJKUizNYSUkrhU1u4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559054423; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=NTEHcqVLv7nv48D5XSMmL4afkmqlaUEdg6X9QKxmbdo=; b=QnnBmzSpI+LfMBNdO9RgYSjzKeCxcE5Nr3VF8Zn+SDrnGmW2MVUa8BBA8zWvjw8NvsCRvkBfwMUkvwMMSAIsDQznQJp2W9+W9eOxDUlDGwNe7nNOH+/D1N7bgER1p1KtIwpWnP3B6r+o9bsV46r5OnJPyEZmwzhPAiUgQrYFFGo= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1559054423555844.9284880029987; Tue, 28 May 2019 07:40:23 -0700 (PDT) Received: from localhost ([127.0.0.1]:36003 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVdGs-00032g-Dx for importer@patchew.org; Tue, 28 May 2019 10:40:10 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50809) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVdFg-0002QG-Kz for qemu-devel@nongnu.org; Tue, 28 May 2019 10:38:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hVdFV-0007t2-Ek for qemu-devel@nongnu.org; Tue, 28 May 2019 10:38:47 -0400 Received: from relay.sw.ru ([185.231.240.75]:46356) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hVdFO-0006u9-LX; Tue, 28 May 2019 10:38:40 -0400 Received: from [10.94.4.71] (helo=dptest2.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.91) (envelope-from ) id 1hVdEM-0004ye-Cx; Tue, 28 May 2019 17:37:34 +0300 From: Denis Plotnikov To: kwolf@redhat.com, mreitz@redhat.com, eblake@redhat.com, armbru@redhat.com, qemu-block@nongnu.org Date: Tue, 28 May 2019 17:37:27 +0300 Message-Id: <20190528143727.10529-4-dplotnikov@virtuozzo.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20190528143727.10529-1-dplotnikov@virtuozzo.com> References: <20190528143727.10529-1-dplotnikov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v0 3/3] qcow2: add zstd cluster compression X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vsementsov@virtuozzo.com, qemu-devel@nongnu.org, den@virtuozzo.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, by the moment, has been the only compression method available. The performance test results: Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest. Image cluster size: 64K. Image on disk size: 2.2G The test was conducted with brd disk to reduce the influence of disk subsystem to the test results. The results is given in seconds. compress cmd: time ./qemu-img convert -O qcow2 -c -o compression_type=3D[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img compression decompression zlib zstd zlib zstd ------------------------------------------------------------ real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0 Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G Signed-off-by: Denis Plotnikov --- block/qcow2.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++ configure | 26 ++++++++++++++++ 2 files changed, 108 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 90f15cc3c9..58901f9f79 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -26,6 +26,7 @@ =20 #define ZLIB_CONST #include +#include =20 #include "block/block_int.h" #include "block/qdict.h" @@ -1553,6 +1554,9 @@ static int coroutine_fn qcow2_do_open(BlockDriverStat= e *bs, QDict *options, case QCOW2_COMPRESSION_TYPE_ZLIB: break; =20 + case QCOW2_COMPRESSION_TYPE_ZSTD: + break; + default: error_setg(errp, "Unknown compression type"); ret =3D -EINVAL; @@ -3286,6 +3290,9 @@ qcow2_co_create(BlockdevCreateOptions *create_options= , Error **errp) * ZLIB shouldn't be here since it's the default */ switch (qcow2_opts->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZSTD: + break; + default: error_setg_errno(errp, -EINVAL, "Unknown compression type"); goto out; @@ -4113,6 +4120,73 @@ static ssize_t zlib_decompress(void *dest, size_t de= st_size, return ret; } =20 +/* + * zstd_compress() + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: compressed size on success + * -1 on any error + */ + +static ssize_t zstd_compress(void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + /* steal some bytes to store compressed chunk size */ + size_t ret; + size_t *c_size =3D dest; + char *d_buf =3D dest; + d_buf +=3D sizeof(ret); + dest_size -=3D sizeof(ret); + + ret =3D ZSTD_compress(d_buf, dest_size, src, src_size, 5); + + if (ZSTD_isError(ret)) { + return -1; + } + + /* store the compressed chunk size in the very beginning of the buffer= */ + *c_size =3D ret; + + return ret + sizeof(ret); +} + +/* + * zstd_decompress() + * + * Decompress some data (not more than @src_size bytes) to produce exactly + * @dest_size bytes. + * + * @dest - destination buffer, @dest_size bytes + * @src - source buffer, @src_size bytes + * + * Returns: 0 on success + * -1 on fail + */ + +static ssize_t zstd_decompress(void *dest, size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret; + /* + * zstd decompress wants to know the exact lenght of the data + * for that purpose, zstd_compress stores the length in the + * very beginning of the compressed buffer + */ + const size_t *s_size =3D src; + const char *s_buf =3D src; + s_buf +=3D sizeof(size_t); + + ret =3D ZSTD_decompress(dest, dest_size, s_buf, *s_size); + + if (ZSTD_isError(ret)) { + return -1; + } + + return 0; +} + #define MAX_COMPRESS_THREADS 4 =20 typedef ssize_t (*Qcow2CompressFunc)(void *dest, size_t dest_size, @@ -4189,6 +4263,10 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, = size_t dest_size, fn =3D zlib_compress; break; =20 + case QCOW2_COMPRESSION_TYPE_ZSTD: + fn =3D zstd_compress; + break; + default: return -ENOTSUP; } @@ -4208,6 +4286,10 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest= , size_t dest_size, fn =3D zlib_decompress; break; =20 + case QCOW2_COMPRESSION_TYPE_ZSTD: + fn =3D zstd_decompress; + break; + default: return -ENOTSUP; } diff --git a/configure b/configure index 1c563a7027..c90716189c 100755 --- a/configure +++ b/configure @@ -433,6 +433,7 @@ opengl_dmabuf=3D"no" cpuid_h=3D"no" avx2_opt=3D"" zlib=3D"yes" +zstd=3D"yes" capstone=3D"" lzo=3D"" snappy=3D"" @@ -1317,6 +1318,8 @@ for opt do ;; --disable-zlib-test) zlib=3D"no" ;; + --disable-zstd-test) zstd=3D"no" + ;; --disable-lzo) lzo=3D"no" ;; --enable-lzo) lzo=3D"yes" @@ -3702,6 +3705,29 @@ EOF fi fi =20 +######################################### +# zstd check + +if test "$zstd" !=3D "no" ; then + if $pkg_config --exists libzstd; then + zstd_cflags=3D$($pkg_config --cflags libzstd) + zstd_libs=3D$($pkg_config --libs libzstd) + QEMU_CFLAGS=3D"$zstd_cflags $QEMU_CFLAGS" + LIBS=3D"$zstd_libs $LIBS" + else + cat > $TMPC << EOF +#include +int main(void) { ZSTD_versionNumber(); return 0; } +EOF + if compile_prog "" "-lzstd" ; then + LIBS=3D"$LIBS -lzstd" + else + error_exit "zstd check failed" \ + "Make sure to have the zstd libs and headers installed." + fi + fi +fi + ########################################## # SHA command probe for modules if test "$modules" =3D yes; then --=20 2.17.0