From nobody Thu May 16 20:42:34 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1558333326; cv=none; d=zoho.com; s=zohoarc; b=ZoxwVChAnGApqfWykpdk0jfY25HGsPI28ishT2/KMTfZNJbGFrWTFGWknhnLGRtdI0EY1leGMYoNrFwJD4wmif7oYQY80YEWyzl1sVuc8O6oztwCBXptl7GYM+kpDGOMg4VUR4AzyMDhu6dwvYflXUqCWLaWEmcV7DeNGi8MdX0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1558333326; h=Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:Sender:Subject:To:ARC-Authentication-Results; bh=+n8XAwO/V7VSyV7uZ53tKPD07sH4rfkfbp4K9t7ZHhE=; b=jPyjLfW9wkiO2/HAa8F17PNV2pFZ44bmbqxgRB5iVCqjRD6w5BE2/aJTj1dXUvzT0Biai51reKC9f7A6NkZtNihfHId21wSvKRhjqr2TsH5Cg9K4qDnT8mvmHMAI7WE/GmtIgjtecdPlOoptnWdhrdEFFfSVmBzlFflGMm4W8HA= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1558333326471793.5937287078633; Sun, 19 May 2019 23:22:06 -0700 (PDT) Received: from localhost ([127.0.0.1]:58659 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hSbgM-0008IU-M7 for importer@patchew.org; Mon, 20 May 2019 02:21:58 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48782) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hSbfT-0007u7-Rt for qemu-devel@nongnu.org; Mon, 20 May 2019 02:21:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hSbfS-0003Xr-Fd for qemu-devel@nongnu.org; Mon, 20 May 2019 02:21:03 -0400 Received: from relay.sw.ru ([185.231.240.75]:36658) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hSbfO-0003VB-VH; Mon, 20 May 2019 02:20:59 -0400 Received: from [10.94.4.71] (helo=dptest2.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.91) (envelope-from ) id 1hSbfJ-0006BF-Pg; Mon, 20 May 2019 09:20:53 +0300 From: Denis Plotnikov To: eblake@redhat.com, kwolf@redhat.com, mreitz@redhat.com Date: Mon, 20 May 2019 09:20:23 +0300 Message-Id: <20190520062023.11599-1-dplotnikov@virtuozzo.com> X-Mailer: git-send-email 2.17.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v2] [RFC] qcow2: add compression type feature X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, den@virtuozzo.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The patch adds some preparation parts for incompatible compression type feature into QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type. It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus the only compression algorithm is used for the image. The plan is to add support for ZSTD and then may be something more effective in the future. ZSTD compression algorithm consumes 3-5 times less CPU power with a comparable compression ratio with zlib. It would be wise to use it for data compression e.g. for backups. The default compression is ZLIB. Signed-off-by: Denis Plotnikov --- block/qcow2.c | 25 +++++++++++++++++++++++++ block/qcow2.h | 29 ++++++++++++++++++++++------- docs/interop/qcow2.txt | 34 +++++++++++++++++++++++++++++++++- qapi/block-core.json | 13 +++++++++++++ 4 files changed, 93 insertions(+), 8 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 3ace3b2209..bca506b80f 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -74,6 +74,7 @@ typedef struct { #define QCOW2_EXT_MAGIC_CRYPTO_HEADER 0x0537be77 #define QCOW2_EXT_MAGIC_BITMAPS 0x23852875 #define QCOW2_EXT_MAGIC_DATA_FILE 0x44415441 +#define QCOW2_EXT_MAGIC_COMPRESSION_TYPE 0x434D5052 =20 static int coroutine_fn qcow2_co_preadv_compressed(BlockDriverState *bs, @@ -398,6 +399,9 @@ static int qcow2_read_extensions(BlockDriverState *bs, = uint64_t start_offset, #endif break; =20 + case QCOW2_EXT_MAGIC_COMPRESSION_TYPE: + /* Setting compression type to BDRVQcow2State->compression_typ= e */ + /* from the image header is going to be here */ case QCOW2_EXT_MAGIC_DATA_FILE: { s->image_data_file =3D g_malloc0(ext.len + 1); @@ -2553,6 +2557,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit =3D QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name =3D "lazy refcounts", }, + { + .type =3D QCOW2_FEAT_TYPE_INCOMPATIBLE, + .bit =3D QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name =3D "compression type", + }, }; =20 ret =3D header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE, @@ -2583,6 +2592,22 @@ int qcow2_update_header(BlockDriverState *bs) buflen -=3D ret; } =20 + /* Compression type extension */ + if (s->compression_type !=3D 0) { + Qcow2CompressionTypeExt comp_header =3D { + .compression_type =3D cpu_to_be32(s->compression_type), + }; + ret =3D header_ext_add(buf, QCOW2_EXT_MAGIC_COMPRESSION_TYPE, + &comp_header, + cpu_to_be64(sizeof(comp_header)), + buflen); + if (ret < 0) { + goto fail; + } + buf +=3D ret; + buflen -=3D ret; + } + /* Keep unknown header extensions */ QLIST_FOREACH(uext, &s->unknown_header_ext, next) { ret =3D header_ext_add(buf, uext->magic, uext->data, uext->len, bu= flen); diff --git a/block/qcow2.h b/block/qcow2.h index fdee297f33..08468ab97d 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -198,16 +198,20 @@ enum { =20 /* Incompatible feature bits */ enum { - QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, - QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, - QCOW2_INCOMPAT_DATA_FILE_BITNR =3D 2, - QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BITNR, - QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_BITNR, - QCOW2_INCOMPAT_DATA_FILE =3D 1 << QCOW2_INCOMPAT_DATA_FILE_BITN= R, + QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, + QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, + QCOW2_INCOMPAT_DATA_FILE_BITNR =3D 2, + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR =3D 3, + QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BI= TNR, + QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_= BITNR, + QCOW2_INCOMPAT_DATA_FILE =3D 1 << QCOW2_INCOMPAT_DATA_FIL= E_BITNR, + QCOW2_INCOMPAT_COMPRESSION_TYPE =3D + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, =20 QCOW2_INCOMPAT_MASK =3D QCOW2_INCOMPAT_DIRTY | QCOW2_INCOMPAT_CORRUPT - | QCOW2_INCOMPAT_DATA_FILE, + | QCOW2_INCOMPAT_DATA_FILE + | QCOW2_INCOMPAT_COMPRESSION_TYPE, }; =20 /* Compatible feature bits */ @@ -263,6 +267,10 @@ typedef struct Qcow2BitmapHeaderExt { uint64_t bitmap_directory_offset; } QEMU_PACKED Qcow2BitmapHeaderExt; =20 +typedef struct Qcow2CompressionTypeExt { + uint32_t compression_type; +} QEMU_PACKED Qcow2CompressionTypeExt; + typedef struct BDRVQcow2State { int cluster_bits; int cluster_size; @@ -350,6 +358,13 @@ typedef struct BDRVQcow2State { int nb_compress_threads; =20 BdrvChild *data_file; + /** + * Compression type used for the image. Default: 0 - ZLIB + * The image compression type is set on image creation. + * The only way to change the compression type is to convert the image + * with the desired compression type set + */ + uint32_t compression_type; } BDRVQcow2State; =20 typedef struct Qcow2COWRegion { diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index af5711e533..a070e82e1a 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -109,7 +109,14 @@ in the description of a field. An External Data File Name header extensio= n may be present if this bit is set. =20 - Bits 3-63: Reserved (set to 0) + Bit 3: Compression type. If the bit is set, then = the + type of compression the image uses is set = in the + header extension. When the bit is set the + compression type extension header must be = present. + When the bit is not set the compression ty= pe + header must absent. + + Bits 4-63: Reserved (set to 0) =20 80 - 87: compatible_features Bitmask of compatible features. An implementation can @@ -175,6 +182,7 @@ be stored. Each extension has a structure like the foll= owing: 0x23852875 - Bitmaps extension 0x0537be77 - Full disk encryption header pointer 0x44415441 - External data file name string + 0x434D5052 - Compression type extension other - Unknown header extension, can be safe= ly ignored =20 @@ -771,3 +779,27 @@ In the image file the 'enabled' state is reflected by = the 'auto' flag. If this flag is set, the software must consider the bitmap as 'enabled' and start tracking virtual disk changes to this bitmap from the first write to the virtual disk. If this flag is not set then the bitmap is disabled. + + +=3D=3D Compression type extension =3D=3D + +The compression type extension is an optional header extension. It stores = the +ID of the compressor which has to be used to compress/decompress disk clus= ters. +A single compression type is applied to all compressed disk clusters, +with no way to change compression types per cluster. Two clusters of the i= mage +couldn't be compressed with different compressors. + +The compression type is set on the image creation. The only way to change +the compression type is to convert the image explicitly. + +The compression type extension is present if and only if the incompatible +compression type bit is set. When the bit is not set the compression type +header must absent. + +When the compression type bit is not set and the compression type header +extension is absent ZLIB compression is used for compressed clusters. +This defines default image compression type: ZLIB. + +Available compression types: + ID 0: Invalid value + 1: ZSTD diff --git a/qapi/block-core.json b/qapi/block-core.json index 7ccbfff9d0..9bde1372ba 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -5257,3 +5257,16 @@ 'data' : { 'node-name': 'str', 'iothread': 'StrOrNull', '*force': 'bool' } } + +## +# @Qcow2CompressionType: +# +# Compression type used in qcow2 image file +# +# @zlib - gzip compressor +# @zstd - zstd compression +# +# Since: 4.1 +## +{ 'enum': 'Qcow2CompressionType', + 'data': [ 'zlib', 'zstd' ] } --=20 2.17.0