From nobody Thu May 2 18:12:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498567131520878.7792554653154; Tue, 27 Jun 2017 05:38:51 -0700 (PDT) Received: from localhost ([::1]:52649 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPplY-0000Rn-1K for importer@patchew.org; Tue, 27 Jun 2017 08:38:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42454) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPphQ-00057W-A3 for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPphN-0005H5-Rw for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:32 -0400 Received: from mx-v6.kamp.de ([2a02:248:0:51::16]:47924 helo=mx01.kamp.de) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPphN-0005FA-IR for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:29 -0400 Received: (qmail 22717 invoked by uid 89); 27 Jun 2017 12:34:24 -0000 Received: from [195.62.97.28] by client-16-kamp (envelope-from , uid 89) with qmail-scanner-2010/03/19-MF (clamdscan: 0.99.2/23511. avast: 1.2.2/17010300. spamassassin: 3.4.1. Clear:RC:1(195.62.97.28):. Processed in 0.364339 secs); 27 Jun 2017 12:34:24 -0000 Received: from smtp.kamp.de (HELO submission.kamp.de) ([195.62.97.28]) by mx01.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted); 27 Jun 2017 12:34:21 -0000 Received: (qmail 29614 invoked from network); 27 Jun 2017 12:34:20 -0000 Received: from lieven-pc.kamp-intra.net (HELO lieven-pc) (relay@kamp.de@::ffff:172.21.12.60) by submission.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted) ESMTPA; 27 Jun 2017 12:34:20 -0000 Received: by lieven-pc (Postfix, from userid 1000) id 7BF2F207E5; Tue, 27 Jun 2017 14:34:20 +0200 (CEST) X-GL_Whitelist: yes From: Peter Lieven To: qemu-block@nongnu.org Date: Tue, 27 Jun 2017 14:34:07 +0200 Message-Id: <1498566850-7934-2-git-send-email-pl@kamp.de> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1498566850-7934-1-git-send-email-pl@kamp.de> References: <1498566850-7934-1-git-send-email-pl@kamp.de> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a02:248:0:51::16 Subject: [Qemu-devel] [PATCH 1/4] block/qcow2: add compression_algorithm create option X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Peter Lieven , qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org, lersek@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" this patch adds a new compression_algorithm option when creating qcow2 imag= es. The current default for the compresison algorithm is zlib and zlib will be used when this option is omitted (like before). If the option is specified e.g. with: qemu-img create -f qcow2 -o compression_algorithm=3Dzlib image.qcow2 1G then a new compression algorithm header extension is added and an incompati= ble feature bit is set. This means that if the header is present it must be par= sed by Qemu on qcow2_open and it must be validated if the specified compression algorithm is supported by the current build of Qemu. This means if the compression_algorithm option is specified Qemu prior to t= his commit will not be able to open the created image. Signed-off-by: Peter Lieven --- block/qcow2.c | 93 +++++++++++++++++++++++++++++++++++++++++++= +--- block/qcow2.h | 20 +++++++--- docs/interop/qcow2.txt | 8 +++- include/block/block_int.h | 35 +++++++++--------- qemu-img.texi | 10 +++++ 5 files changed, 138 insertions(+), 28 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 2f94f03..893b145 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -60,9 +60,11 @@ typedef struct { uint32_t len; } QEMU_PACKED QCowExtension; =20 -#define QCOW2_EXT_MAGIC_END 0 -#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA -#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 +#define QCOW2_EXT_MAGIC_END 0 +#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA +#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 +#define QCOW2_EXT_MAGIC_COMPRESSION_ALGORITHM 0xc0318300 +/* 0xc03183xx reserved for further use of compression algorithm parameters= */ =20 static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filen= ame) { @@ -76,6 +78,15 @@ static int qcow2_probe(const uint8_t *buf, int buf_size,= const char *filename) return 0; } =20 +static uint32_t is_compression_algorithm_supported(char *algorithm) +{ + if (!algorithm[0] || !strcmp(algorithm, "zlib")) { + /* no algorithm means the old default of zlib compression + * with 12 window bits */ + return QCOW2_COMPRESSION_ZLIB; + } + return 0; +} =20 /*=20 * read qcow2 extension and fill bs @@ -148,6 +159,34 @@ static int qcow2_read_extensions(BlockDriverState *bs,= uint64_t start_offset, #endif break; =20 + case QCOW2_EXT_MAGIC_COMPRESSION_ALGORITHM: + if (ext.len >=3D sizeof(s->compression_algorithm)) { + error_setg(errp, "ERROR: ext_compression_algorithm: len=3D= %" + PRIu32 " too large (>=3D%zu)", ext.len, + sizeof(s->compression_algorithm)); + return 2; + } + ret =3D bdrv_pread(bs->file, offset, s->compression_algorithm, + ext.len); + if (ret < 0) { + error_setg_errno(errp, -ret, "ERROR: ext_compression_algor= ithm:" + " Could not read algorithm name"); + return 3; + } + s->compression_algorithm[ext.len] =3D '\0'; + s->compression_algorithm_id =3D + is_compression_algorithm_supported(s->compression_algorith= m); + if (!s->compression_algorithm_id) { + error_setg(errp, "ERROR: compression algorithm '%s' is " + " unsupported", s->compression_algorithm); + return 4; + } +#ifdef DEBUG_EXT + printf("Qcow2: Got compression algorithm %s\n", + s->compression_algorithm); +#endif + break; + case QCOW2_EXT_MAGIC_FEATURE_TABLE: if (p_feature_table !=3D NULL) { void* feature_table =3D g_malloc0(ext.len + 2 * sizeof(Qco= w2Feature)); @@ -1104,6 +1143,7 @@ static int qcow2_do_open(BlockDriverState *bs, QDict = *options, int flags, =20 s->cluster_cache_offset =3D -1; s->flags =3D flags; + s->compression_algorithm_id =3D QCOW2_COMPRESSION_ZLIB; =20 ret =3D qcow2_refcount_init(bs); if (ret !=3D 0) { @@ -1981,6 +2021,21 @@ int qcow2_update_header(BlockDriverState *bs) buflen -=3D ret; } =20 + /* Compression Algorithm header extension */ + if (s->compression_algorithm[0]) { + ret =3D header_ext_add(buf, QCOW2_EXT_MAGIC_COMPRESSION_ALGORITHM, + s->compression_algorithm, + strlen(s->compression_algorithm), + buflen); + if (ret < 0) { + goto fail; + } + buf +=3D ret; + buflen -=3D ret; + header->incompatible_features |=3D + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION); + } + /* Feature table */ if (s->qcow_version >=3D 3) { Qcow2Feature features[] =3D { @@ -1995,6 +2050,11 @@ int qcow2_update_header(BlockDriverState *bs) .name =3D "corrupt bit", }, { + .type =3D QCOW2_FEAT_TYPE_INCOMPATIBLE, + .bit =3D QCOW2_INCOMPAT_COMPRESSION_BITNR, + .name =3D "compression algorithm", + }, + { .type =3D QCOW2_FEAT_TYPE_COMPATIBLE, .bit =3D QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name =3D "lazy refcounts", @@ -2144,7 +2204,7 @@ static int qcow2_create2(const char *filename, int64_= t total_size, const char *backing_file, const char *backing_for= mat, int flags, size_t cluster_size, PreallocMode prea= lloc, QemuOpts *opts, int version, int refcount_order, - Error **errp) + char *compression_algorithm, Error **errp) { int cluster_bits; QDict *options; @@ -2332,6 +2392,12 @@ static int qcow2_create2(const char *filename, int64= _t total_size, abort(); } =20 + if (compression_algorithm[0]) { + BDRVQcow2State *s =3D blk_bs(blk)->opaque; + memcpy(s->compression_algorithm, compression_algorithm, + strlen(compression_algorithm)); + } + /* Create a full header (including things like feature table) */ ret =3D qcow2_update_header(blk_bs(blk)); if (ret < 0) { @@ -2395,6 +2461,7 @@ static int qcow2_create(const char *filename, QemuOpt= s *opts, Error **errp) char *backing_file =3D NULL; char *backing_fmt =3D NULL; char *buf =3D NULL; + char *compression_algorithm =3D NULL; uint64_t size =3D 0; int flags =3D 0; size_t cluster_size =3D DEFAULT_CLUSTER_SIZE; @@ -2475,15 +2542,25 @@ static int qcow2_create(const char *filename, QemuO= pts *opts, Error **errp) =20 refcount_order =3D ctz32(refcount_bits); =20 + compression_algorithm =3D qemu_opt_get_del(opts, + BLOCK_OPT_COMPRESSION_ALGORIT= HM); + if (!is_compression_algorithm_supported(compression_algorithm)) { + error_setg(errp, "Compression algorithm '%s' is not supported", + compression_algorithm); + ret =3D -EINVAL; + goto finish; + } + ret =3D qcow2_create2(filename, size, backing_file, backing_fmt, flags, cluster_size, prealloc, opts, version, refcount_or= der, - &local_err); + compression_algorithm, &local_err); error_propagate(errp, local_err); =20 finish: g_free(backing_file); g_free(backing_fmt); g_free(buf); + g_free(compression_algorithm); return ret; } =20 @@ -3458,6 +3535,12 @@ static QemuOptsList qcow2_create_opts =3D { .help =3D "Width of a reference count entry in bits", .def_value_str =3D "16" }, + { + .name =3D BLOCK_OPT_COMPRESSION_ALGORITHM, + .type =3D QEMU_OPT_STRING, + .help =3D "Compression algorithm used for compressed clusters", + .def_value_str =3D "" + }, { /* end of list */ } } }; diff --git a/block/qcow2.h b/block/qcow2.h index 87b15eb..1c9ba06 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -171,6 +171,10 @@ typedef struct Qcow2UnknownHeaderExtension { } Qcow2UnknownHeaderExtension; =20 enum { + QCOW2_COMPRESSION_ZLIB =3D 0xC0318301, +}; + +enum { QCOW2_FEAT_TYPE_INCOMPATIBLE =3D 0, QCOW2_FEAT_TYPE_COMPATIBLE =3D 1, QCOW2_FEAT_TYPE_AUTOCLEAR =3D 2, @@ -178,13 +182,16 @@ enum { =20 /* Incompatible feature bits */ enum { - QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, - QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, - QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BITNR, - QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_BITNR, + QCOW2_INCOMPAT_DIRTY_BITNR =3D 0, + QCOW2_INCOMPAT_CORRUPT_BITNR =3D 1, + QCOW2_INCOMPAT_COMPRESSION_BITNR =3D 2, + QCOW2_INCOMPAT_DIRTY =3D 1 << QCOW2_INCOMPAT_DIRTY_BITNR, + QCOW2_INCOMPAT_CORRUPT =3D 1 << QCOW2_INCOMPAT_CORRUPT_BITN= R, + QCOW2_INCOMPAT_COMPRESSION =3D 1 << QCOW2_INCOMPAT_COMPRESSION_= BITNR, =20 QCOW2_INCOMPAT_MASK =3D QCOW2_INCOMPAT_DIRTY - | QCOW2_INCOMPAT_CORRUPT, + | QCOW2_INCOMPAT_CORRUPT + | QCOW2_INCOMPAT_COMPRESSION, }; =20 /* Compatible feature bits */ @@ -294,6 +301,9 @@ typedef struct BDRVQcow2State { * override) */ char *image_backing_file; char *image_backing_format; + + char compression_algorithm[16]; + uint32_t compression_algorithm_id; } BDRVQcow2State; =20 typedef struct Qcow2COWRegion { diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index 80cdfd0..1f165d6 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -85,7 +85,11 @@ in the description of a field. be written to (unless for regaining consistency). =20 - Bits 2-63: Reserved (set to 0) + Bit 2: Compress algorithm bit. If this bit is se= t then + the compress algorithm extension must be p= arsed + and checked for compatiblity. + + Bits 3-63: Reserved (set to 0) =20 80 - 87: compatible_features Bitmask of compatible features. An implementation can @@ -135,6 +139,8 @@ be stored. Each extension has a structure like the foll= owing: 0xE2792ACA - Backing file format name 0x6803f857 - Feature name table 0x23852875 - Bitmaps extension + 0xC0318300 - Compression Algorithm + 0xC03183xx - Reserved for compression algorithm pa= rams other - Unknown header extension, can be safe= ly ignored =20 diff --git a/include/block/block_int.h b/include/block/block_int.h index 15fa602..03a4b8f 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -40,23 +40,24 @@ #define BLOCK_FLAG_ENCRYPT 1 #define BLOCK_FLAG_LAZY_REFCOUNTS 8 =20 -#define BLOCK_OPT_SIZE "size" -#define BLOCK_OPT_ENCRYPT "encryption" -#define BLOCK_OPT_COMPAT6 "compat6" -#define BLOCK_OPT_HWVERSION "hwversion" -#define BLOCK_OPT_BACKING_FILE "backing_file" -#define BLOCK_OPT_BACKING_FMT "backing_fmt" -#define BLOCK_OPT_CLUSTER_SIZE "cluster_size" -#define BLOCK_OPT_TABLE_SIZE "table_size" -#define BLOCK_OPT_PREALLOC "preallocation" -#define BLOCK_OPT_SUBFMT "subformat" -#define BLOCK_OPT_COMPAT_LEVEL "compat" -#define BLOCK_OPT_LAZY_REFCOUNTS "lazy_refcounts" -#define BLOCK_OPT_ADAPTER_TYPE "adapter_type" -#define BLOCK_OPT_REDUNDANCY "redundancy" -#define BLOCK_OPT_NOCOW "nocow" -#define BLOCK_OPT_OBJECT_SIZE "object_size" -#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits" +#define BLOCK_OPT_SIZE "size" +#define BLOCK_OPT_ENCRYPT "encryption" +#define BLOCK_OPT_COMPAT6 "compat6" +#define BLOCK_OPT_HWVERSION "hwversion" +#define BLOCK_OPT_BACKING_FILE "backing_file" +#define BLOCK_OPT_BACKING_FMT "backing_fmt" +#define BLOCK_OPT_CLUSTER_SIZE "cluster_size" +#define BLOCK_OPT_TABLE_SIZE "table_size" +#define BLOCK_OPT_PREALLOC "preallocation" +#define BLOCK_OPT_SUBFMT "subformat" +#define BLOCK_OPT_COMPAT_LEVEL "compat" +#define BLOCK_OPT_LAZY_REFCOUNTS "lazy_refcounts" +#define BLOCK_OPT_ADAPTER_TYPE "adapter_type" +#define BLOCK_OPT_REDUNDANCY "redundancy" +#define BLOCK_OPT_NOCOW "nocow" +#define BLOCK_OPT_OBJECT_SIZE "object_size" +#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits" +#define BLOCK_OPT_COMPRESSION_ALGORITHM "compression_algorithm" =20 #define BLOCK_PROBE_BUF_SIZE 512 =20 diff --git a/qemu-img.texi b/qemu-img.texi index 5b925ec..c0d1bec 100644 --- a/qemu-img.texi +++ b/qemu-img.texi @@ -621,6 +621,16 @@ file which is COW and has data blocks already, it coul= dn't be changed to NOCOW by setting @code{nocow=3Don}. One can issue @code{lsattr filename} to chec= k if the NOCOW flag is set or not (Capital 'C' is NOCOW flag). =20 +@item compression_algorithm +Defines which compression algorithm is should be used for compressed clust= ers. +The following options are available if support for the respective libraries +has been enabled at compile time: + + zlib Uses standard zlib compression (default) + +The compression algorithm can only be defined at image create time and can= not +be changed later. + @end table =20 @item Other --=20 1.9.1 From nobody Thu May 2 18:12:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498566981455531.2663936408392; Tue, 27 Jun 2017 05:36:21 -0700 (PDT) Received: from localhost ([::1]:52639 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPpj8-0006Mz-Hs for importer@patchew.org; Tue, 27 Jun 2017 08:36:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42456) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPphQ-00057Y-AJ for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPphN-0005Gu-RV for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:32 -0400 Received: from mx-v6.kamp.de ([2a02:248:0:51::16]:45019 helo=mx01.kamp.de) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPphN-0005F8-IB for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:29 -0400 Received: (qmail 22681 invoked by uid 89); 27 Jun 2017 12:34:24 -0000 Received: from [195.62.97.28] by client-16-kamp (envelope-from , uid 89) with qmail-scanner-2010/03/19-MF (clamdscan: 0.99.2/23511. avast: 1.2.2/17010300. spamassassin: 3.4.1. Clear:RC:1(195.62.97.28):. Processed in 0.241906 secs); 27 Jun 2017 12:34:24 -0000 Received: from smtp.kamp.de (HELO submission.kamp.de) ([195.62.97.28]) by mx01.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted); 27 Jun 2017 12:34:21 -0000 Received: (qmail 29616 invoked from network); 27 Jun 2017 12:34:20 -0000 Received: from lieven-pc.kamp-intra.net (HELO lieven-pc) (relay@kamp.de@::ffff:172.21.12.60) by submission.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted) ESMTPA; 27 Jun 2017 12:34:20 -0000 Received: by lieven-pc (Postfix, from userid 1000) id 7FE4E26C5D; Tue, 27 Jun 2017 14:34:20 +0200 (CEST) X-GL_Whitelist: yes From: Peter Lieven To: qemu-block@nongnu.org Date: Tue, 27 Jun 2017 14:34:08 +0200 Message-Id: <1498566850-7934-3-git-send-email-pl@kamp.de> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1498566850-7934-1-git-send-email-pl@kamp.de> References: <1498566850-7934-1-git-send-email-pl@kamp.de> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a02:248:0:51::16 Subject: [Qemu-devel] [PATCH 2/4] block/qcow2: optimize qcow2_co_pwritev_compressed X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Peter Lieven , qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org, lersek@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" if we specify exactly one iov of s->cluster_size bytes we can avoid the bounce buffer. Signed-off-by: Peter Lieven --- block/qcow2.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 893b145..c91eb1f 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -2716,7 +2716,7 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uin= t64_t offset, struct iovec iov; z_stream strm; int ret, out_len; - uint8_t *buf, *out_buf; + uint8_t *buf, *out_buf, *local_buf =3D NULL; uint64_t cluster_offset; =20 if (bytes =3D=3D 0) { @@ -2726,8 +2726,8 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uin= t64_t offset, return bdrv_truncate(bs->file, cluster_offset, NULL); } =20 - buf =3D qemu_blockalign(bs, s->cluster_size); - if (bytes !=3D s->cluster_size) { + if (bytes !=3D s->cluster_size || qiov->niov !=3D 1) { + buf =3D local_buf =3D qemu_blockalign(bs, s->cluster_size); if (bytes > s->cluster_size || offset + bytes !=3D bs->total_sectors << BDRV_SECTOR_BITS) { @@ -2736,8 +2736,10 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, ui= nt64_t offset, } /* Zero-pad last write if image size is not cluster aligned */ memset(buf + bytes, 0, s->cluster_size - bytes); + qemu_iovec_to_buf(qiov, 0, buf, bytes); + } else { + buf =3D qiov->iov[0].iov_base; } - qemu_iovec_to_buf(qiov, 0, buf, bytes); =20 out_buf =3D g_malloc(s->cluster_size); =20 @@ -2805,7 +2807,7 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uin= t64_t offset, success: ret =3D 0; fail: - qemu_vfree(buf); + qemu_vfree(local_buf); g_free(out_buf); return ret; } --=20 1.9.1 From nobody Thu May 2 18:12:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498566997614137.75568644430166; Tue, 27 Jun 2017 05:36:37 -0700 (PDT) Received: from localhost ([::1]:52640 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPpjP-0006b0-Tr for importer@patchew.org; Tue, 27 Jun 2017 08:36:35 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42475) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPphQ-00057o-Pl for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPphP-0005Hw-3L for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:32 -0400 Received: from mx-v6.kamp.de ([2a02:248:0:51::16]:41082 helo=mx01.kamp.de) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPphO-0005HC-Q9 for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:31 -0400 Received: (qmail 22895 invoked by uid 89); 27 Jun 2017 12:34:29 -0000 Received: from [195.62.97.28] by client-16-kamp (envelope-from , uid 89) with qmail-scanner-2010/03/19-MF (clamdscan: 0.99.2/23511. avast: 1.2.2/17010300. spamassassin: 3.4.1. Clear:RC:1(195.62.97.28):. Processed in 0.29016 secs); 27 Jun 2017 12:34:29 -0000 Received: from smtp.kamp.de (HELO submission.kamp.de) ([195.62.97.28]) by mx01.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted); 27 Jun 2017 12:34:25 -0000 Received: (qmail 29620 invoked from network); 27 Jun 2017 12:34:20 -0000 Received: from lieven-pc.kamp-intra.net (HELO lieven-pc) (relay@kamp.de@::ffff:172.21.12.60) by submission.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted) ESMTPA; 27 Jun 2017 12:34:20 -0000 Received: by lieven-pc (Postfix, from userid 1000) id 81F7F26C5E; Tue, 27 Jun 2017 14:34:20 +0200 (CEST) X-GL_Whitelist: yes From: Peter Lieven To: qemu-block@nongnu.org Date: Tue, 27 Jun 2017 14:34:09 +0200 Message-Id: <1498566850-7934-4-git-send-email-pl@kamp.de> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1498566850-7934-1-git-send-email-pl@kamp.de> References: <1498566850-7934-1-git-send-email-pl@kamp.de> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a02:248:0:51::16 Subject: [Qemu-devel] [PATCH 3/4] block/qcow2: add lzo compression algorithm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Peter Lieven , qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org, lersek@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Peter Lieven --- block/qcow2-cluster.c | 65 +++++++++++++++++++++++++++++++--------------- block/qcow2.c | 72 ++++++++++++++++++++++++++++++++++-------------= ---- block/qcow2.h | 1 + configure | 2 +- qemu-img.texi | 1 + 5 files changed, 95 insertions(+), 46 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 3d341fd..ecb059b 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -24,6 +24,9 @@ =20 #include "qemu/osdep.h" #include +#ifdef CONFIG_LZO +#include +#endif =20 #include "qapi/error.h" #include "qemu-common.h" @@ -1521,30 +1524,49 @@ again: } =20 static int decompress_buffer(uint8_t *out_buf, int out_buf_size, - const uint8_t *buf, int buf_size) + const uint8_t *buf, int buf_size, + uint32_t compression_algorithm_id) { z_stream strm1, *strm =3D &strm1; - int ret, out_len; - - memset(strm, 0, sizeof(*strm)); - - strm->next_in =3D (uint8_t *)buf; - strm->avail_in =3D buf_size; - strm->next_out =3D out_buf; - strm->avail_out =3D out_buf_size; - - ret =3D inflateInit2(strm, -12); - if (ret !=3D Z_OK) - return -1; - ret =3D inflate(strm, Z_FINISH); - out_len =3D strm->next_out - out_buf; - if ((ret !=3D Z_STREAM_END && ret !=3D Z_BUF_ERROR) || - out_len !=3D out_buf_size) { + int ret =3D 0, out_len; + + switch (compression_algorithm_id) { + case QCOW2_COMPRESSION_ZLIB: + memset(strm, 0, sizeof(*strm)); + + strm->next_in =3D (uint8_t *)buf; + strm->avail_in =3D buf_size; + strm->next_out =3D out_buf; + strm->avail_out =3D out_buf_size; + + ret =3D inflateInit2(strm, -12); + if (ret !=3D Z_OK) { + return -1; + } + ret =3D inflate(strm, Z_FINISH); + out_len =3D strm->next_out - out_buf; + ret =3D -(ret !=3D Z_STREAM_END); inflateEnd(strm); - return -1; + break; +#ifdef CONFIG_LZO + case QCOW2_COMPRESSION_LZO: + out_len =3D out_buf_size; + ret =3D lzo1x_decompress_safe(buf, buf_size, out_buf, + (lzo_uint *) &out_len, NULL); + if (ret =3D=3D LZO_E_INPUT_NOT_CONSUMED) { + /* We always read up to the next sector boundary. Thus + * buf_size may be larger than the original compressed size. */ + ret =3D 0; + } + break; +#endif + default: + abort(); /* should never reach this point */ } - inflateEnd(strm); - return 0; + if (out_len !=3D out_buf_size) { + ret =3D -1; + } + return ret; } =20 int qcow2_decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset) @@ -1565,7 +1587,8 @@ int qcow2_decompress_cluster(BlockDriverState *bs, ui= nt64_t cluster_offset) return ret; } if (decompress_buffer(s->cluster_cache, s->cluster_size, - s->cluster_data + sector_offset, csize) < 0)= { + s->cluster_data + sector_offset, csize, + s->compression_algorithm_id) < 0) { return -EIO; } s->cluster_cache_offset =3D coffset; diff --git a/block/qcow2.c b/block/qcow2.c index c91eb1f..bd65582 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -26,6 +26,9 @@ #include "sysemu/block-backend.h" #include "qemu/module.h" #include +#ifdef CONFIG_LZO +#include +#endif #include "block/qcow2.h" #include "qemu/error-report.h" #include "qapi/qmp/qerror.h" @@ -84,6 +87,10 @@ static uint32_t is_compression_algorithm_supported(char = *algorithm) /* no algorithm means the old default of zlib compression * with 12 window bits */ return QCOW2_COMPRESSION_ZLIB; +#ifdef CONFIG_LZO + } else if (!strcmp(algorithm, "lzo")) { + return QCOW2_COMPRESSION_LZO; +#endif } return 0; } @@ -2715,8 +2722,8 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uin= t64_t offset, QEMUIOVector hd_qiov; struct iovec iov; z_stream strm; - int ret, out_len; - uint8_t *buf, *out_buf, *local_buf =3D NULL; + int ret, out_len =3D 0; + uint8_t *buf, *out_buf =3D NULL, *local_buf =3D NULL, *work_buf =3D NU= LL; uint64_t cluster_offset; =20 if (bytes =3D=3D 0) { @@ -2741,34 +2748,50 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, u= int64_t offset, buf =3D qiov->iov[0].iov_base; } =20 - out_buf =3D g_malloc(s->cluster_size); + switch (s->compression_algorithm_id) { + case QCOW2_COMPRESSION_ZLIB: + out_buf =3D g_malloc(s->cluster_size); =20 - /* best compression, small window, no zlib header */ - memset(&strm, 0, sizeof(strm)); - ret =3D deflateInit2(&strm, Z_DEFAULT_COMPRESSION, - Z_DEFLATED, -12, - 9, Z_DEFAULT_STRATEGY); - if (ret !=3D 0) { - ret =3D -EINVAL; - goto fail; - } + /* best compression, small window, no zlib header */ + memset(&strm, 0, sizeof(strm)); + ret =3D deflateInit2(&strm, Z_DEFAULT_COMPRESSION, + Z_DEFLATED, -12, + 9, Z_DEFAULT_STRATEGY); + if (ret !=3D 0) { + ret =3D -EINVAL; + goto fail; + } =20 - strm.avail_in =3D s->cluster_size; - strm.next_in =3D (uint8_t *)buf; - strm.avail_out =3D s->cluster_size; - strm.next_out =3D out_buf; + strm.avail_in =3D s->cluster_size; + strm.next_in =3D (uint8_t *)buf; + strm.avail_out =3D s->cluster_size; + strm.next_out =3D out_buf; + + ret =3D deflate(&strm, Z_FINISH); + if (ret !=3D Z_STREAM_END && ret !=3D Z_OK) { + deflateEnd(&strm); + ret =3D -EINVAL; + goto fail; + } + out_len =3D strm.next_out - out_buf; =20 - ret =3D deflate(&strm, Z_FINISH); - if (ret !=3D Z_STREAM_END && ret !=3D Z_OK) { deflateEnd(&strm); - ret =3D -EINVAL; - goto fail; - } - out_len =3D strm.next_out - out_buf; =20 - deflateEnd(&strm); + ret =3D ret !=3D Z_STREAM_END; + break; +#ifdef CONFIG_LZO + case QCOW2_COMPRESSION_LZO: + out_buf =3D g_malloc(s->cluster_size + s->cluster_size / 64 + 16 += 3); + work_buf =3D g_malloc(LZO1X_1_MEM_COMPRESS); + ret =3D lzo1x_1_compress(buf, s->cluster_size, out_buf, + (lzo_uint *) &out_len, work_buf); + break; +#endif + default: + abort(); /* should never reach this point */ + } =20 - if (ret !=3D Z_STREAM_END || out_len >=3D s->cluster_size) { + if (ret || out_len >=3D s->cluster_size) { /* could not compress: write normal cluster */ ret =3D qcow2_co_pwritev(bs, offset, bytes, qiov, 0); if (ret < 0) { @@ -2809,6 +2832,7 @@ success: fail: qemu_vfree(local_buf); g_free(out_buf); + g_free(work_buf); return ret; } =20 diff --git a/block/qcow2.h b/block/qcow2.h index 1c9ba06..716012c 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -172,6 +172,7 @@ typedef struct Qcow2UnknownHeaderExtension { =20 enum { QCOW2_COMPRESSION_ZLIB =3D 0xC0318301, + QCOW2_COMPRESSION_LZO =3D 0xC0318302, }; =20 enum { diff --git a/configure b/configure index c571ad1..81d3286 100755 --- a/configure +++ b/configure @@ -1890,7 +1890,7 @@ if test "$lzo" !=3D "no" ; then int main(void) { lzo_version(); return 0; } EOF if compile_prog "" "-llzo2" ; then - libs_softmmu=3D"$libs_softmmu -llzo2" + LIBS=3D"$LIBS -llzo2" lzo=3D"yes" else if test "$lzo" =3D "yes"; then diff --git a/qemu-img.texi b/qemu-img.texi index c0d1bec..043c1ba 100644 --- a/qemu-img.texi +++ b/qemu-img.texi @@ -627,6 +627,7 @@ The following options are available if support for the = respective libraries has been enabled at compile time: =20 zlib Uses standard zlib compression (default) + lzo Uses LZO1X compression =20 The compression algorithm can only be defined at image create time and can= not be changed later. --=20 1.9.1 From nobody Thu May 2 18:12:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1498567126228346.3229747381256; Tue, 27 Jun 2017 05:38:46 -0700 (PDT) Received: from localhost ([::1]:52648 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPplT-0000Pj-K1 for importer@patchew.org; Tue, 27 Jun 2017 08:38:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42465) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dPphQ-00057f-Em for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dPphO-0005Hg-RZ for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:32 -0400 Received: from mx-v6.kamp.de ([2a02:248:0:51::16]:53826 helo=mx01.kamp.de) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dPphO-0005Gc-HS for qemu-devel@nongnu.org; Tue, 27 Jun 2017 08:34:30 -0400 Received: (qmail 22880 invoked by uid 89); 27 Jun 2017 12:34:29 -0000 Received: from [195.62.97.28] by client-16-kamp (envelope-from , uid 89) with qmail-scanner-2010/03/19-MF (clamdscan: 0.99.2/23511. avast: 1.2.2/17010300. spamassassin: 3.4.1. Clear:RC:1(195.62.97.28):. Processed in 0.227462 secs); 27 Jun 2017 12:34:29 -0000 Received: from smtp.kamp.de (HELO submission.kamp.de) ([195.62.97.28]) by mx01.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted); 27 Jun 2017 12:34:25 -0000 Received: (qmail 29621 invoked from network); 27 Jun 2017 12:34:20 -0000 Received: from lieven-pc.kamp-intra.net (HELO lieven-pc) (relay@kamp.de@::ffff:172.21.12.60) by submission.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted) ESMTPA; 27 Jun 2017 12:34:20 -0000 Received: by lieven-pc (Postfix, from userid 1000) id 838A826C38; Tue, 27 Jun 2017 14:34:20 +0200 (CEST) X-GL_Whitelist: yes From: Peter Lieven To: qemu-block@nongnu.org Date: Tue, 27 Jun 2017 14:34:10 +0200 Message-Id: <1498566850-7934-5-git-send-email-pl@kamp.de> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1498566850-7934-1-git-send-email-pl@kamp.de> References: <1498566850-7934-1-git-send-email-pl@kamp.de> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a02:248:0:51::16 Subject: [Qemu-devel] [PATCH 4/4] block/qcow2: add zlib-fast compression algorithm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Peter Lieven , qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org, lersek@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" this adds support for optimized zlib settings which almost tripples the compression speed while maintaining about the same compressed size. Signed-off-by: Peter Lieven --- block/qcow2-cluster.c | 3 ++- block/qcow2.c | 11 +++++++++-- block/qcow2.h | 1 + qemu-img.texi | 1 + 4 files changed, 13 insertions(+), 3 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index ecb059b..d8e2378 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -1532,6 +1532,7 @@ static int decompress_buffer(uint8_t *out_buf, int ou= t_buf_size, =20 switch (compression_algorithm_id) { case QCOW2_COMPRESSION_ZLIB: + case QCOW2_COMPRESSION_ZLIB_FAST: memset(strm, 0, sizeof(*strm)); =20 strm->next_in =3D (uint8_t *)buf; @@ -1539,7 +1540,7 @@ static int decompress_buffer(uint8_t *out_buf, int ou= t_buf_size, strm->next_out =3D out_buf; strm->avail_out =3D out_buf_size; =20 - ret =3D inflateInit2(strm, -12); + ret =3D inflateInit2(strm, -15); if (ret !=3D Z_OK) { return -1; } diff --git a/block/qcow2.c b/block/qcow2.c index bd65582..f07d8f0 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -87,6 +87,8 @@ static uint32_t is_compression_algorithm_supported(char *= algorithm) /* no algorithm means the old default of zlib compression * with 12 window bits */ return QCOW2_COMPRESSION_ZLIB; + } else if (!strcmp(algorithm, "zlib-fast")) { + return QCOW2_COMPRESSION_ZLIB_FAST; #ifdef CONFIG_LZO } else if (!strcmp(algorithm, "lzo")) { return QCOW2_COMPRESSION_LZO; @@ -2722,6 +2724,7 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uin= t64_t offset, QEMUIOVector hd_qiov; struct iovec iov; z_stream strm; + int z_level =3D Z_DEFAULT_COMPRESSION, z_windowBits =3D -12; int ret, out_len =3D 0; uint8_t *buf, *out_buf =3D NULL, *local_buf =3D NULL, *work_buf =3D NU= LL; uint64_t cluster_offset; @@ -2749,13 +2752,17 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, u= int64_t offset, } =20 switch (s->compression_algorithm_id) { + case QCOW2_COMPRESSION_ZLIB_FAST: + z_level =3D Z_BEST_SPEED; + z_windowBits =3D -15; + /* fall-through */ case QCOW2_COMPRESSION_ZLIB: out_buf =3D g_malloc(s->cluster_size); =20 /* best compression, small window, no zlib header */ memset(&strm, 0, sizeof(strm)); - ret =3D deflateInit2(&strm, Z_DEFAULT_COMPRESSION, - Z_DEFLATED, -12, + ret =3D deflateInit2(&strm, z_level, + Z_DEFLATED, z_windowBits, 9, Z_DEFAULT_STRATEGY); if (ret !=3D 0) { ret =3D -EINVAL; diff --git a/block/qcow2.h b/block/qcow2.h index 716012c..a89f986 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -173,6 +173,7 @@ typedef struct Qcow2UnknownHeaderExtension { enum { QCOW2_COMPRESSION_ZLIB =3D 0xC0318301, QCOW2_COMPRESSION_LZO =3D 0xC0318302, + QCOW2_COMPRESSION_ZLIB_FAST =3D 0xC0318303, }; =20 enum { diff --git a/qemu-img.texi b/qemu-img.texi index 043c1ba..83a5db2 100644 --- a/qemu-img.texi +++ b/qemu-img.texi @@ -627,6 +627,7 @@ The following options are available if support for the = respective libraries has been enabled at compile time: =20 zlib Uses standard zlib compression (default) + zlib-fast Uses zlib compression with optimized compression parame= ters lzo Uses LZO1X compression =20 The compression algorithm can only be defined at image create time and can= not --=20 1.9.1