From nobody Sat Feb 7 07:31:38 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1662044096; cv=none; d=zohomail.com; s=zohoarc; b=NIs12jfHQcw80CiI6dQirNybUBzakHITi7IC8ALPT7DwbzOMrDvkoWv9o2V0IUwBtvcl5tOokUrEAa9HgAdTIyX/dmSd8LVZ3dYqFhtI0Cy63BCsiTdo3HnMbOldniQgvoTvN4025UXLAN95tkMO4tzVHBHu1ZbWfEilSbdlRTQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1662044096; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=SSJJnmIXmudaMnxuNmMvGw3Z7IavYVzasHgSDRE1bBY=; b=cE2ksjk5R4oTVqn2wIGhHbqKSacmSaNrOVX+sMUxDX/OpqUZuLW4mNr1QiqCKnGh+pRrA4JeXnSraxpJxIrnhkSPpnbWel1MMDyCvXWRkh822s1wWLU60z3Hw6yennGzELXaKtcJwPDn/JarO9dp3K2Na4rDYIbeda3wqKaFGnU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1662044096885421.8943011894263; Thu, 1 Sep 2022 07:54:56 -0700 (PDT) Received: from localhost ([::1]:60432 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oTlap-0002JF-QU for importer@patchew.org; Thu, 01 Sep 2022 10:54:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34116) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oTlTA-0001Bt-Ck for qemu-devel@nongnu.org; Thu, 01 Sep 2022 10:47:00 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:39991) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oTlGb-0006yD-SS for qemu-devel@nongnu.org; Thu, 01 Sep 2022 10:34:04 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-272-eQ17MzSyNTShQU6sEfs8AA-1; Thu, 01 Sep 2022 10:32:28 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D724B296A62D; Thu, 1 Sep 2022 14:32:27 +0000 (UTC) Received: from loop.redhat.com (unknown [10.35.206.127]) by smtp.corp.redhat.com (Postfix) with ESMTP id 936E4492C3B; Thu, 1 Sep 2022 14:32:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662042841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SSJJnmIXmudaMnxuNmMvGw3Z7IavYVzasHgSDRE1bBY=; b=TE2SFIzLGFXu5UnemZUdnwPHP7MuMeGBTv9Q1+KioZnJuX8mcIfmHmfyQmWLfiO4IbPKLV BSK/9azgwh4T5MP5d36vjxdQiFjZQg/x/a7nt9aKqrlFzX4GQVWs5hXQyRGq5dBs5FhI0v nEwERPqApJiXs92oPMUZZld0bAR4ZDk= X-MC-Unique: eQ17MzSyNTShQU6sEfs8AA-1 From: Nir Soffer To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , Nir Soffer Subject: [PATCH 1/3] qemu-img: Add checksum command Date: Thu, 1 Sep 2022 17:32:21 +0300 Message-Id: <20220901143223.201295-2-nsoffer@redhat.com> In-Reply-To: <20220901143223.201295-1-nsoffer@redhat.com> References: <20220901143223.201295-1-nsoffer@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=nsoffer@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1662044099062100001 The checksum command compute a checksum for disk image content using the blkhash library[1]. The blkhash library is not packaged yet, but it is available via copr[2]. Example run: $ ./qemu-img checksum -p fedora-35.qcow2 6e5c00c995056319d52395f8d91c7f84725ae3da69ffcba4de4c7d22cff713a5 fedor= a-35.qcow2 The block checksum is constructed by splitting the image to fixed sized blocks and computing a digest of every block. The image checksum is the digest of the all block digests. The checksum uses internally the "sha256" algorithm but it cannot be compared with checksums created by other tools such as `sha256sum`. The blkhash library supports sparse images, zero detection, and optimizes zero block hashing (they are practically free). The library uses multiple threads to speed up the computation. Comparing to `sha256sum`, `qemu-img checksum` is 3.5-4800[3] times faster, depending on the amount of data in the image: $ ./qemu-img info /scratch/50p.raw file format: raw virtual size: 6 GiB (6442450944 bytes) disk size: 2.91 GiB $ hyperfine -w2 -r5 -p "sleep 1" "./qemu-img checksum /scratch/50p.raw"= \ "sha256sum /scratch/50p.raw" Benchmark 1: ./qemu-img checksum /scratch/50p.raw Time (mean =C2=B1 =CF=83): 1.849 s =C2=B1 0.037 s [User: 7.7= 64 s, System: 0.962 s] Range (min =E2=80=A6 max): 1.813 s =E2=80=A6 1.908 s 5 runs Benchmark 2: sha256sum /scratch/50p.raw Time (mean =C2=B1 =CF=83): 14.585 s =C2=B1 0.072 s [User: 13.= 537 s, System: 1.003 s] Range (min =E2=80=A6 max): 14.501 s =E2=80=A6 14.697 s 5 runs Summary './qemu-img checksum /scratch/50p.raw' ran 7.89 =C2=B1 0.16 times faster than 'sha256sum /scratch/50p.raw' The new command is available only when `blkhash` is available during build. To test the new command please install the `blkhash-devel` package: $ dnf copr enable nsoffer/blkhash $ sudo dnf install blkhash-devel [1] https://gitlab.com/nirs/blkhash [2] https://copr.fedorainfracloud.org/coprs/nsoffer/blkhash/ [3] Computing checksum for 8T empty image: qemu-img checksum: 3.7s, sha256sum (estimate): 17,749s Signed-off-by: Nir Soffer --- docs/tools/qemu-img.rst | 22 +++++ meson.build | 10 ++- meson_options.txt | 2 + qemu-img-cmds.hx | 8 ++ qemu-img.c | 191 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 232 insertions(+), 1 deletion(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index 85a6e05b35..8be9c45cbf 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -347,20 +347,42 @@ Command description: Check completed, image is corrupted 3 Check completed, image has leaked clusters, but is not corrupted 63 Checks are not supported by the image format =20 If ``-r`` is specified, exit codes representing the image state refer to= the state after (the attempt at) repairing it. That is, a successful ``-r al= l`` will yield the exit code 0, independently of the image state before. =20 +.. option:: checksum [--object OBJECTDEF] [--image-opts] [-f FMT] [-T SRC_= CACHE] [-p] FILENAME + + Print a checksum for image *FILENAME* guest visible content. Images with + different format or settings wil have the same checksum. + + The format is probed unless you specify it by ``-f``. + + The checksum is computed for guest visible content. Allocated areas full= of + zeroes, zero clusters, and unallocated areas are read as zeros so they w= ill + have the same checksum. Images with single or multiple files or backing = files + will have the same checksums if the guest will see the same content when + reading the image. + + Image metadata that is not visible to the guest such as dirty bitmaps do= es + not affect the checksum. + + Computing a checksum requires a read-only image. You cannot compute a + checksum of an active image used by a guest, but you can compute a check= sum + of a guest during pull mode incremental backup using NBD URL. + + The checksum is not compatible with other tools such as *sha256sum*. + .. option:: commit [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [-t C= ACHE] [-b BASE] [-r RATE_LIMIT] [-d] [-p] FILENAME =20 Commit the changes recorded in *FILENAME* in its base image or backing f= ile. If the backing file is smaller than the snapshot, then the backing file = will be resized to be the same size as the snapshot. If the snapshot is smaller= than the backing file, the backing file will not be truncated. If you want t= he backing file to match the size of the smaller snapshot, you can safely t= runcate it yourself once the commit operation successfully completes. =20 The image *FILENAME* is emptied after the operation has succeeded. If yo= u do diff --git a/meson.build b/meson.build index 20fddbd707..56b648d8a7 100644 --- a/meson.build +++ b/meson.build @@ -727,20 +727,24 @@ if not get_option('curl').auto() or have_block kwargs: static_kwargs) endif libudev =3D not_found if targetos =3D=3D 'linux' and (have_system or have_tools) libudev =3D dependency('libudev', method: 'pkg-config', required: get_option('libudev'), kwargs: static_kwargs) endif =20 +blkhash =3D dependency('blkhash', version: '>=3D0.4.1', + method: 'pkg-config', + required: get_option('blkhash')) + mpathlibs =3D [libudev] mpathpersist =3D not_found mpathpersist_new_api =3D false if targetos =3D=3D 'linux' and have_tools and get_option('mpath').allowed() mpath_test_source_new =3D ''' #include #include unsigned mpath_mx_alloc_len =3D 1024; int logsink; static struct config *multipath_conf; @@ -1852,20 +1856,22 @@ config_host_data.set('CONFIG_CFI', get_option('cfi'= )) config_host_data.set('CONFIG_SELINUX', selinux.found()) config_host_data.set('CONFIG_XEN_BACKEND', xen.found()) if xen.found() # protect from xen.version() having less than three components xen_version =3D xen.version().split('.') + ['0', '0'] xen_ctrl_version =3D xen_version[0] + \ ('0' + xen_version[1]).substring(-2) + \ ('0' + xen_version[2]).substring(-2) config_host_data.set('CONFIG_XEN_CTRL_INTERFACE_VERSION', xen_ctrl_versi= on) endif +config_host_data.set('CONFIG_BLKHASH', blkhash.found()) + config_host_data.set('QEMU_VERSION', '"@0@"'.format(meson.project_version(= ))) config_host_data.set('QEMU_VERSION_MAJOR', meson.project_version().split('= .')[0]) config_host_data.set('QEMU_VERSION_MINOR', meson.project_version().split('= .')[1]) config_host_data.set('QEMU_VERSION_MICRO', meson.project_version().split('= .')[2]) =20 config_host_data.set_quoted('CONFIG_HOST_DSOSUF', host_dsosuf) config_host_data.set('HAVE_HOST_BLOCK_DEVICE', have_host_block_device) =20 have_coroutine_pool =3D get_option('coroutine_pool') if get_option('debug_stack_usage') and have_coroutine_pool @@ -3594,21 +3600,22 @@ subdir('qga') # Don't build qemu-keymap if xkbcommon is not explicitly enabled # when we don't build tools or system if xkbcommon.found() # used for the update-keymaps target, so include rules even if !have_too= ls qemu_keymap =3D executable('qemu-keymap', files('qemu-keymap.c', 'ui/inp= ut-keymap.c') + genh, dependencies: [qemuutil, xkbcommon], install: h= ave_tools) endif =20 if have_tools qemu_img =3D executable('qemu-img', [files('qemu-img.c'), hxdep], - dependencies: [authz, block, crypto, io, qom, qemuutil], inst= all: true) + dependencies: [authz, block, crypto, io, qom, qemuutil, blkha= sh], + install: true) qemu_io =3D executable('qemu-io', files('qemu-io.c'), dependencies: [block, qemuutil], install: true) qemu_nbd =3D executable('qemu-nbd', files('qemu-nbd.c'), dependencies: [blockdev, qemuutil, gnutls, selinux], install: true) =20 subdir('storage-daemon') subdir('contrib/rdmacm-mux') subdir('contrib/elf2dmp') =20 @@ -3977,20 +3984,21 @@ summary_info +=3D {'bzip2 support': libbzip2} summary_info +=3D {'lzfse support': liblzfse} summary_info +=3D {'zstd support': zstd} summary_info +=3D {'NUMA host support': numa} summary_info +=3D {'capstone': capstone} summary_info +=3D {'libpmem support': libpmem} summary_info +=3D {'libdaxctl support': libdaxctl} summary_info +=3D {'libudev': libudev} # Dummy dependency, keep .found() summary_info +=3D {'FUSE lseek': fuse_lseek.found()} summary_info +=3D {'selinux': selinux} +summary_info +=3D {'blkhash': blkhash} summary(summary_info, bool_yn: true, section: 'Dependencies') =20 if not supported_cpus.contains(cpu) message() warning('SUPPORT FOR THIS HOST CPU WILL GO AWAY IN FUTURE RELEASES!') message() message('CPU host architecture ' + cpu + ' support is not currently main= tained.') message('The QEMU project intends to remove support for this host CPU in= ') message('a future release if nobody volunteers to maintain it and to') message('provide a build host for our continuous integration setup.') diff --git a/meson_options.txt b/meson_options.txt index e58e158396..8bf1dece12 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -304,10 +304,12 @@ option('debug_mutex', type: 'boolean', value: false, option('debug_stack_usage', type: 'boolean', value: false, description: 'measure coroutine stack usage') option('qom_cast_debug', type: 'boolean', value: false, description: 'cast debugging support') option('gprof', type: 'boolean', value: false, description: 'QEMU profiling with gprof') option('profiler', type: 'boolean', value: false, description: 'profiler support') option('slirp_smbd', type : 'feature', value : 'auto', description: 'use smbd (at path --smbd=3D*) in slirp networking') +option('blkhash', type: 'feature', value: 'auto', + description: 'blkhash support for computing image checksum') diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index 1b1dab5b17..c4fc935a37 100644 --- a/qemu-img-cmds.hx +++ b/qemu-img-cmds.hx @@ -26,20 +26,28 @@ DEF("bitmap", img_bitmap, SRST .. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable= | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--objec= t OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP ERST =20 DEF("check", img_check, "check [--object objectdef] [--image-opts] [-q] [-f fmt] [--output=3Do= fmt] [-r [leaks | all]] [-T src_cache] [-U] filename") SRST .. option:: check [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [--out= put=3DOFMT] [-r [leaks | all]] [-T SRC_CACHE] [-U] FILENAME ERST =20 +#ifdef CONFIG_BLKHASH +DEF("checksum", img_checksum, + "checksum [--object objectdef] [--image-opts] [-f fmt] [-T src_cache] = [-p] filename") +SRST +.. option:: checksum [--object OBJECTDEF] [--image-opts] [-f FMT] [-T SRC_= CACHE] [-p] FILENAME +ERST +#endif /* CONFIG_BLKHASH */ + DEF("commit", img_commit, "commit [--object objectdef] [--image-opts] [-q] [-f fmt] [-t cache] [= -b base] [-r rate_limit] [-d] [-p] filename") SRST .. option:: commit [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [-t C= ACHE] [-b BASE] [-r RATE_LIMIT] [-d] [-p] FILENAME ERST =20 DEF("compare", img_compare, "compare [--object objectdef] [--image-opts] [-f fmt] [-F fmt] [-T src= _cache] [-p] [-q] [-s] [-U] filename1 filename2") SRST .. option:: compare [--object OBJECTDEF] [--image-opts] [-f FMT] [-F FMT] = [-T SRC_CACHE] [-p] [-q] [-s] [-U] FILENAME1 FILENAME2 diff --git a/qemu-img.c b/qemu-img.c index 7d4b33b3da..7edcfe4bc8 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -17,20 +17,21 @@ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OT= HER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING= FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS = IN * THE SOFTWARE. */ =20 #include "qemu/osdep.h" #include +#include =20 #include "qemu/help-texts.h" #include "qemu/qemu-progress.h" #include "qemu-version.h" #include "qapi/error.h" #include "qapi/qapi-commands-block-core.h" #include "qapi/qapi-visit-block-core.h" #include "qapi/qobject-output-visitor.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qdict.h" @@ -1611,20 +1612,210 @@ out: qemu_vfree(buf1); qemu_vfree(buf2); blk_unref(blk2); out2: blk_unref(blk1); out3: qemu_progress_end(); return ret; } =20 +#ifdef CONFIG_BLKHASH +/* + * Compute image checksum. + */ +static int img_checksum(int argc, char **argv) +{ + const char *digest_name =3D "sha256"; + const size_t block_size =3D 64 * KiB; + + const char *format =3D NULL; + const char *cache =3D BDRV_DEFAULT_CACHE; + const char *filename; + BlockBackend *blk; + BlockDriverState *bs; + uint8_t *buf =3D NULL; + int64_t pnum; + bool progress =3D false; + int flags =3D 0; + bool writethrough; + int64_t total_size; + int64_t offset =3D 0; + int c; + bool image_opts =3D false; + struct blkhash *h =3D NULL; + unsigned char digest[64]; + unsigned int digest_len; + int ret =3D 1; + int err; + + for (;;) { + static const struct option long_options[] =3D { + {"help", no_argument, 0, 'h'}, + {"object", required_argument, 0, OPTION_OBJECT}, + {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, + {0, 0, 0, 0} + }; + c =3D getopt_long(argc, argv, ":hf:T:p", + long_options, NULL); + if (c =3D=3D -1) { + break; + } + switch (c) { + case ':': + missing_argument(argv[optind - 1]); + break; + case '?': + unrecognized_option(argv[optind - 1]); + break; + case 'h': + help(); + break; + case 'f': + format =3D optarg; + break; + case 'T': + cache =3D optarg; + break; + case 'p': + progress =3D true; + break; + case OPTION_OBJECT: + { + Error *local_err =3D NULL; + + if (!user_creatable_add_from_str(optarg, &local_err)) { + if (local_err) { + error_report_err(local_err); + exit(2); + } else { + /* Help was printed */ + exit(EXIT_SUCCESS); + } + } + break; + } + case OPTION_IMAGE_OPTS: + image_opts =3D true; + break; + } + } + + if (optind !=3D argc - 1) { + error_exit("Expecting image file name"); + } + + filename =3D argv[optind++]; + + err =3D bdrv_parse_cache_mode(cache, &flags, &writethrough); + if (err < 0) { + error_report("Invalid source cache option: %s", cache); + return ret; + } + + blk =3D img_open(image_opts, filename, format, flags, writethrough, fa= lse, + false); + if (!blk) { + return ret; + } + + /* Initialize before using goto out. */ + qemu_progress_init(progress, 2.0); + + bs =3D blk_bs(blk); + buf =3D blk_blockalign(blk, IO_BUF_SIZE); + + total_size =3D blk_getlength(blk); + if (total_size < 0) { + error_report("Can't get size of %s: %s", + filename, strerror(-total_size)); + goto out; + } + + h =3D blkhash_new(block_size, digest_name); + if (!h) { + error_report("Can't create blkhash: %s", strerror(errno)); + goto out; + } + + qemu_progress_print(0, 100); + + while (offset < total_size) { + int status; + int64_t chunk; + + status =3D bdrv_block_status_above(bs, NULL, offset, + total_size - offset, &pnum, NULL, + NULL); + if (status < 0) { + error_report("Error checking status at offset %" PRId64 " for = %s", + offset, filename); + goto out; + } + + assert(pnum); + chunk =3D pnum; + + if (status & BDRV_BLOCK_ZERO) { + chunk =3D MIN(chunk, SIZE_MAX); + err =3D blkhash_zero(h, chunk); + if (err) { + error_report("Error zeroing hash at offset %" PRId64 + " of %s: %s", + offset, filename, strerror(err)); + goto out; + } + } else { + chunk =3D MIN(chunk, IO_BUF_SIZE); + err =3D blk_pread(blk, offset, chunk, buf, 0); + if (err < 0) { + error_report("Error reading at offset %" PRId64 " of %s: %= s", + offset, filename, strerror(-err)); + goto out; + } + err =3D blkhash_update(h, buf, chunk); + if (err) { + error_report("Error updating hash at offset %" PRId64 + " of %s: %s", + offset, filename, strerror(err)); + goto out; + } + } + + offset +=3D chunk; + qemu_progress_print(((float) chunk / total_size) * 100, 100); + } + + err =3D blkhash_final(h, digest, &digest_len); + if (err) { + error_report("Error finalizing hash of %s: %s", + filename, strerror(err)); + goto out; + } + + for (unsigned i =3D 0; i < digest_len; i++) { + printf("%02x", digest[i]); + } + printf(" %s%s", filename, progress ? "" : "\n"); + + ret =3D 0; + +out: + blkhash_free(h); + qemu_vfree(buf); + blk_unref(blk); + qemu_progress_end(); + + return ret; +} +#endif /* CONFIG_BLKHASH */ + /* Convenience wrapper around qmp_block_dirty_bitmap_merge */ static void do_dirty_bitmap_merge(const char *dst_node, const char *dst_na= me, const char *src_node, const char *src_na= me, Error **errp) { BlockDirtyBitmapOrStr *merge_src; BlockDirtyBitmapOrStrList *list =3D NULL; =20 merge_src =3D g_new0(BlockDirtyBitmapOrStr, 1); merge_src->type =3D QTYPE_QDICT; --=20 2.37.2 From nobody Sat Feb 7 07:31:38 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1662043892; cv=none; d=zohomail.com; s=zohoarc; b=Ag/eZtxTYdoGO54CBRfaSpWhi/HD//q5lNQJbhgPYoPFHnNmIjcX5n94dT7pTNNIYrcIT6ZXbZaqZJnPa90+TwS5pQjhizLXhn91bVS3yJ5JCLb2Hjc2petmhl+X+mYwEv9ZbwyVEr38SMdaVRBr4pBxKdeR2lpo+oAp6FfXcSo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1662043892; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=EkRsP9fWWOpk1xsECwKFbrlU5bQB+gkBZQ8RVDIqAPM=; b=YpOlZvcPYIL4EL2aOBbmjJtPMXcFspQRQnonOOoXpPK4jQs3Tbt6Cj0VIn2rMPSQvYgR/Z8phFMYdU7/JuQ4hdlBkO6yX35ODiH7BbXack6SUhVSMob8PMmGEV1K5ae824ddfxbQIhm2KLTePK46jxfS8xuBI4ds8CYebpWvQjE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 166204389253631.634968291696737; Thu, 1 Sep 2022 07:51:32 -0700 (PDT) Received: from localhost ([::1]:56938 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oTlXX-0004YA-6u for importer@patchew.org; Thu, 01 Sep 2022 10:51:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34116) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oTlTA-0001Bt-0H for qemu-devel@nongnu.org; Thu, 01 Sep 2022 10:47:00 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:37530) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oTlGp-0006z9-56 for qemu-devel@nongnu.org; Thu, 01 Sep 2022 10:34:18 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-39-NKrmJcKUM_GT9H4WvtR31w-1; Thu, 01 Sep 2022 10:32:29 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4B2D029324A4; Thu, 1 Sep 2022 14:32:29 +0000 (UTC) Received: from loop.redhat.com (unknown [10.35.206.127]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2EB22492C3B; Thu, 1 Sep 2022 14:32:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662042853; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EkRsP9fWWOpk1xsECwKFbrlU5bQB+gkBZQ8RVDIqAPM=; b=PttQGbZM2nkEqQxT+q1UFJyKDuwkLeYvP9dweeW0R6wMXFAmJApOhh0PCZBD93DxA+s0zH +ZotxK/MuIEjQKkjb0XAM/Evj1Mfb2TCm9gOc0UMvV7ri7HG69Z46PBdRF9/H8verUOKus 4cvaWP8K3lyZorHiZB+mviVPjLvM39A= X-MC-Unique: NKrmJcKUM_GT9H4WvtR31w-1 From: Nir Soffer To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , Nir Soffer Subject: [PATCH 2/3] iotests: Test qemu-img checksum Date: Thu, 1 Sep 2022 17:32:22 +0300 Message-Id: <20220901143223.201295-3-nsoffer@redhat.com> In-Reply-To: <20220901143223.201295-1-nsoffer@redhat.com> References: <20220901143223.201295-1-nsoffer@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=nsoffer@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1662043894742100001 Content-Type: text/plain; charset="utf-8" Add simple tests creating an image with all kinds of extents, different formats, different backing chain, different protocol, and different image options. Since all images have the same guest visible content they must have the same checksum. To help debugging in case of failures, the output includes a json map of every test image. Signed-off-by: Nir Soffer --- tests/qemu-iotests/tests/qemu-img-checksum | 149 ++++++++++++++++++ .../qemu-iotests/tests/qemu-img-checksum.out | 74 +++++++++ 2 files changed, 223 insertions(+) create mode 100755 tests/qemu-iotests/tests/qemu-img-checksum create mode 100644 tests/qemu-iotests/tests/qemu-img-checksum.out diff --git a/tests/qemu-iotests/tests/qemu-img-checksum b/tests/qemu-iotest= s/tests/qemu-img-checksum new file mode 100755 index 0000000000..3a85ba33f2 --- /dev/null +++ b/tests/qemu-iotests/tests/qemu-img-checksum @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# group: rw auto quick +# +# Test cases for qemu-img checksum. +# +# Copyright (C) 2022 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +import re + +import iotests + +from iotests import ( + filter_testfiles, + qemu_img, + qemu_img_log, + qemu_io, + qemu_nbd_popen, +) + + +def checksum_available(): + out =3D qemu_img("--help").stdout + return re.search(r"\bchecksum .+ filename\b", out) is not None + + +if not checksum_available(): + iotests.notrun("checksum command not available") + +iotests.script_initialize( + supported_fmts=3D["raw", "qcow2"], + supported_cache_modes=3D["none", "writeback"], + supported_protocols=3D["file", "nbd"], + required_fmts=3D["raw", "qcow2"], +) + +print() +print("=3D=3D=3D Test images =3D=3D=3D") +print() + +disk_raw =3D iotests.file_path('raw') +qemu_img("create", "-f", "raw", disk_raw, "10m") +qemu_io("-f", "raw", + "-c", "write -P 0x1 0 2m", # data + "-c", "write -P 0x0 2m 2m", # data with zeroes + "-c", "write -z 4m 2m", # zero allocated + "-c", "write -z -u 6m 2m", # zero hole + # unallocated + disk_raw) +print(filter_testfiles(disk_raw)) +qemu_img_log("map", "--output", "json", disk_raw) + +disk_qcow2 =3D iotests.file_path('qcow2') +qemu_img("create", "-f", "qcow2", disk_qcow2, "10m") +qemu_io("-f", "qcow2", + "-c", "write -P 0x1 0 2m", # data + "-c", "write -P 0x0 2m 2m", # data with zeroes + "-c", "write -z 4m 2m", # zero allocated + "-c", "write -z -u 6m 2m", # zero hole + # unallocated + disk_qcow2) +print(filter_testfiles(disk_qcow2)) +qemu_img_log("map", "--output", "json", disk_qcow2) + +disk_compressed =3D iotests.file_path('compressed') +qemu_img("convert", "-f", "qcow2", "-O", "qcow2", "-c", + disk_qcow2, disk_compressed) +print(filter_testfiles(disk_compressed)) +qemu_img_log("map", "--output", "json", disk_compressed) + +disk_base =3D iotests.file_path('base') +qemu_img("create", "-f", "raw", disk_base, "10m") +qemu_io("-f", "raw", + "-c", "write -P 0x1 0 2m", + "-c", "write -P 0x0 2m 2m", + disk_base) +print(filter_testfiles(disk_base)) +qemu_img_log("map", "--output", "json", disk_base) + +disk_top =3D iotests.file_path('top') +qemu_img("create", "-f", "qcow2", "-b", disk_base, "-F", "raw", + disk_top) +qemu_io("-f", "qcow2", + "-c", "write -z 4m 2m", + "-c", "write -z -u 6m 2m", + disk_top) +print(filter_testfiles(disk_top)) +qemu_img_log("map", "--output", "json", disk_top) + +print() +print("=3D=3D=3D Checksums - file =3D=3D=3D") +print() + +qemu_img_log("checksum", disk_raw) +qemu_img_log("checksum", disk_qcow2) +qemu_img_log("checksum", disk_compressed) +qemu_img_log("checksum", disk_top) +qemu_img_log("checksum", disk_base) + +print() +print("=3D=3D=3D Checksums - nbd =3D=3D=3D") +print() + +nbd_sock =3D iotests.file_path("nbd.sock", base_dir=3Diotests.sock_dir) +nbd_uri =3D f"nbd+unix:///{{}}?socket=3D{nbd_sock}" + +with qemu_nbd_popen("-k", nbd_sock, "-f", "raw", "-x", "raw", disk_raw): + qemu_img_log("checksum", nbd_uri.format("raw")) + +with qemu_nbd_popen("-k", nbd_sock, "-f", "qcow2", "-x", "qcow2", disk_qco= w2): + qemu_img_log("checksum", nbd_uri.format("qcow2")) + +with qemu_nbd_popen("-k", nbd_sock, "-f", "qcow2", "-x", "compressed", dis= k_compressed): + qemu_img_log("checksum", nbd_uri.format("compressed")) + +with qemu_nbd_popen("-k", nbd_sock, "-f", "raw", "-x", "base", disk_base): + qemu_img_log("checksum", nbd_uri.format("base")) + +with qemu_nbd_popen("-k", nbd_sock, "-f", "qcow2", "-x", "top", disk_top): + qemu_img_log("checksum", nbd_uri.format("top")) + +print() +print("=3D=3D=3D Command line options =3D=3D=3D") +print() + +qemu_img_log("checksum", "-f", "qcow2", disk_top) +qemu_img_log("checksum", "-T", "none", disk_top) + +out =3D qemu_img("checksum", "-p", disk_top).stdout +last =3D out.splitlines()[-1] # Filter progress lines. +print(filter_testfiles(last)) + +print() +print("=3D=3D=3D Incorrect usage =3D=3D=3D") +print() + +qemu_img_log("checksum", "-f", "qcow2", disk_raw, check=3DFalse) diff --git a/tests/qemu-iotests/tests/qemu-img-checksum.out b/tests/qemu-io= tests/tests/qemu-img-checksum.out new file mode 100644 index 0000000000..2cff03439f --- /dev/null +++ b/tests/qemu-iotests/tests/qemu-img-checksum.out @@ -0,0 +1,74 @@ + +=3D=3D=3D Test images =3D=3D=3D + +TEST_DIR/PID-raw +[{ "start": 0, "length": 4194304, "depth": 0, "present": true, "zero": fal= se, "data": true, "offset": 0}, +{ "start": 4194304, "length": 6291456, "depth": 0, "present": true, "zero"= : true, "data": false, "offset": 4194304}] + +TEST_DIR/PID-qcow2 +[{ "start": 0, "length": 4194304, "depth": 0, "present": true, "zero": fal= se, "data": true, "offset": 327680}, +{ "start": 4194304, "length": 4194304, "depth": 0, "present": true, "zero"= : true, "data": false}, +{ "start": 8388608, "length": 2097152, "depth": 0, "present": false, "zero= ": true, "data": false}] + +TEST_DIR/PID-compressed +[{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": fal= se, "data": true}, +{ "start": 2097152, "length": 8388608, "depth": 0, "present": false, "zero= ": true, "data": false}] + +TEST_DIR/PID-base +[{ "start": 0, "length": 4194304, "depth": 0, "present": true, "zero": fal= se, "data": true, "offset": 0}, +{ "start": 4194304, "length": 6291456, "depth": 0, "present": true, "zero"= : true, "data": false, "offset": 4194304}] + +TEST_DIR/PID-top +[{ "start": 0, "length": 4194304, "depth": 1, "present": true, "zero": fal= se, "data": true, "offset": 0}, +{ "start": 4194304, "length": 4194304, "depth": 0, "present": true, "zero"= : true, "data": false}, +{ "start": 8388608, "length": 2097152, "depth": 1, "present": true, "zero"= : true, "data": false, "offset": 8388608}] + + +=3D=3D=3D Checksums - file =3D=3D=3D + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-raw + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-qcow2 + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-compressed + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-top + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-base + + +=3D=3D=3D Checksums - nbd =3D=3D=3D + +Start NBD server +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 nbd+unix= :///raw?socket=3DSOCK_DIR/PID-nbd.sock + +Kill NBD server +Start NBD server +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 nbd+unix= :///qcow2?socket=3DSOCK_DIR/PID-nbd.sock + +Kill NBD server +Start NBD server +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 nbd+unix= :///compressed?socket=3DSOCK_DIR/PID-nbd.sock + +Kill NBD server +Start NBD server +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 nbd+unix= :///base?socket=3DSOCK_DIR/PID-nbd.sock + +Kill NBD server +Start NBD server +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 nbd+unix= :///top?socket=3DSOCK_DIR/PID-nbd.sock + +Kill NBD server + +=3D=3D=3D Command line options =3D=3D=3D + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-top + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-top + +57cd8ef0cfad106d737f8fb0de3a0306a8a1a41db7bf7c0c36e2dfe75ee9bd26 TEST_DIR= /PID-top + +=3D=3D=3D Incorrect usage =3D=3D=3D + +qemu-img: Could not open 'TEST_DIR/PID-raw': Image is not in qcow2 format + --=20 2.37.2 From nobody Sat Feb 7 07:31:38 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1662043898; cv=none; d=zohomail.com; s=zohoarc; b=Wwtw1FtAk32LUSM6eoMuKqok6zJ+pQ28bZHlHa0S5DP1WN5mR4SCFmzLCAeIHo4tFBtE/F69J03SQI1PpSfoWH1i+sFZmtYVnaJjcLp5ygGxK9W8j8ymAlCOzWB39amwTRiqPyMwSXiasot/uIuadZTMfKfzQfs5q5xFq9Wu2W0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1662043898; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=EItwtgzK73sR40MjoSlHF54DssM+VaeY49YDpn+1pto=; b=Bq2qOAD2sf06laPq7GH6T2UPZOgbxSn7jjBk+CcNLAT2PReJPu5PXMmos5ZkDIrC46N9XTpD2dBUhYP1WgJezSkXKOUNRTMXSgUywlVNVUjW37LjwV5TFLa9TooGt8skUiujrhZbsQ0CmpXSRZnGg9l2fWF4utq+4XYByCjLlHQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1662043898420938.4725423010543; Thu, 1 Sep 2022 07:51:38 -0700 (PDT) Received: from localhost ([::1]:56940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oTlXc-0004dt-8p for importer@patchew.org; Thu, 01 Sep 2022 10:51:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34116) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oTlTD-0001Bt-Dt for qemu-devel@nongnu.org; Thu, 01 Sep 2022 10:47:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:55407) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oTlGH-0006w5-Pl for qemu-devel@nongnu.org; Thu, 01 Sep 2022 10:33:43 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-187-k5W7oY7ZNeaZBtxRyK6o9A-1; Thu, 01 Sep 2022 10:32:31 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DEA8F3C10230; Thu, 1 Sep 2022 14:32:30 +0000 (UTC) Received: from loop.redhat.com (unknown [10.35.206.127]) by smtp.corp.redhat.com (Postfix) with ESMTP id 98823492C3B; Thu, 1 Sep 2022 14:32:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662042819; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EItwtgzK73sR40MjoSlHF54DssM+VaeY49YDpn+1pto=; b=f3NekxpqECMqDw36R+wJ/yCoE3wp6jL9r8OEgr7GWI0sxO2pQX4QlBcsYWl5hFKuGxIprQ BVeCI1i3hNqfOTG+n8YZwu5AdfvbG+RDcdAK7YzQd0vZ23IuPl5tKyMdBqZAXjpUW+IUW6 3obvePU8BQTncvZvdxvz8tQU5WCW080= X-MC-Unique: k5W7oY7ZNeaZBtxRyK6o9A-1 From: Nir Soffer To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , Nir Soffer Subject: [PATCH 3/3] qemu-img: Speed up checksum Date: Thu, 1 Sep 2022 17:32:23 +0300 Message-Id: <20220901143223.201295-4-nsoffer@redhat.com> In-Reply-To: <20220901143223.201295-1-nsoffer@redhat.com> References: <20220901143223.201295-1-nsoffer@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=nsoffer@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1662043898824100001 Add coroutine based loop inspired by `qemu-img convert` design. Changes compared to `qemu-img convert`: - State for the entire image is kept in ImgChecksumState - State for single worker coroutine is kept in ImgChecksumworker. - "Writes" are always in-order, ensured using a queue. - Calling block status once per image extent, when the current extent is consumed by the workers. - Using 1m buffer size - testings shows that this gives best read performance both with buffered and direct I/O. - Number of coroutines is not configurable. Testing does not show improvement when using more than 8 coroutines. - Progress include entire image, not only the allocated state. Comparing to the simple read loop shows that this version is up to 4.67 times faster when computing a checksum for an image full of zeroes. For real images it is 1.59 times faster with direct I/O, and with buffered I/O there is no difference. Test results on Dell PowerEdge R640 in a CentOS Stream 9 container: | image | size | i/o | before | after | change | |----------|------|-----------|----------------|----------------|--------| | zero [1] | 6g | buffered | 1.600s =C2=B10.014s | 0.342s =C2=B10.016s |= x4.67 | | zero | 6g | direct | 4.684s =C2=B10.093s | 2.211s =C2=B10.009s |= x2.12 | | real [2] | 6g | buffered | 1.841s =C2=B10.075s | 1.806s =C2=B10.036s |= x1.02 | | real | 6g | direct | 3.094s =C2=B10.079s | 1.947s =C2=B10.017s |= x1.59 | | nbd [3] | 6g | buffered | 2.455s =C2=B10.183s | 1.808s =C2=B10.016s |= x1.36 | | nbd | 6g | direct | 3.540s =C2=B10.020s | 1.749s =C2=B10.018s |= x2.02 | [1] raw image full of zeroes [2] raw fedora 35 image with additional random data, 50% full [3] image [2] exported by qemu-nbd via unix socket Signed-off-by: Nir Soffer --- qemu-img.c | 343 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 270 insertions(+), 73 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index 7edcfe4bc8..bfa8e2862f 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -1613,48 +1613,288 @@ out: qemu_vfree(buf2); blk_unref(blk2); out2: blk_unref(blk1); out3: qemu_progress_end(); return ret; } =20 #ifdef CONFIG_BLKHASH + +#define CHECKSUM_COROUTINES 8 +#define CHECKSUM_BUF_SIZE (1 * MiB) +#define CHECKSUM_ZERO_SIZE MIN(16 * GiB, SIZE_MAX) + +typedef struct ImgChecksumState ImgChecksumState; + +typedef struct ImgChecksumWorker { + QTAILQ_ENTRY(ImgChecksumWorker) entry; + ImgChecksumState *state; + Coroutine *co; + uint8_t *buf; + + /* The current chunk. */ + int64_t offset; + int64_t length; + bool zero; + + /* Always true for zero extent, false for data extent. Set to true + * when reading the chunk completes. */ + bool ready; +} ImgChecksumWorker; + +struct ImgChecksumState { + const char *filename; + BlockBackend *blk; + BlockDriverState *bs; + int64_t total_size; + + /* Current extent, modified in checksum_co_next. */ + int64_t offset; + int64_t length; + bool zero; + + int running_coroutines; + CoMutex lock; + ImgChecksumWorker workers[CHECKSUM_COROUTINES]; + + /* Ensure in-order updates. Update are scheduled at the tail of the + * queue and processed from the head of the queue when a worker is + * ready. */ + QTAILQ_HEAD(, ImgChecksumWorker) update_queue; + + struct blkhash *hash; + int ret; +}; + +static int checksum_block_status(ImgChecksumState *s) +{ + int64_t length; + int status; + + /* Must be called when current extent is consumed. */ + assert(s->length =3D=3D 0); + + status =3D bdrv_block_status_above(s->bs, NULL, s->offset, + s->total_size - s->offset, &length, N= ULL, + NULL); + if (status < 0) { + error_report("Error checking status at offset %" PRId64 " for %s", + s->offset, s->filename); + s->ret =3D status; + return -1; + } + + assert(length > 0); + + s->length =3D length; + s->zero =3D !!(status & BDRV_BLOCK_ZERO); + + return 0; +} + +/** + * Grab the next chunk from the current extent, getting the next extent if + * needed, and schecule the next update at the end fo the update queue. + * + * Retrun true if the worker has work to do, false if the worker has + * finished or there was an error getting the next extent. + */ +static coroutine_fn bool checksum_co_next(ImgChecksumWorker *w) +{ + ImgChecksumState *s =3D w->state; + + qemu_co_mutex_lock(&s->lock); + + if (s->offset =3D=3D s->total_size || s->ret !=3D -EINPROGRESS) { + qemu_co_mutex_unlock(&s->lock); + return false; + } + + if (s->length =3D=3D 0 && checksum_block_status(s)) { + qemu_co_mutex_unlock(&s->lock); + return false; + } + + /* Grab one chunk from current extent. */ + w->offset =3D s->offset; + w->length =3D MIN(s->length, + s->zero ? CHECKSUM_ZERO_SIZE : CHECKSUM_BUF_SIZE); + w->zero =3D s->zero; + w->ready =3D s->zero; + + /* Advance state to the next chunk. */ + s->offset +=3D w->length; + s->length -=3D w->length; + + /* Schedule this chunk update. */ + QTAILQ_INSERT_TAIL(&s->update_queue, w, entry); + + qemu_co_mutex_unlock(&s->lock); + + return true; +} + +/* Wait until this chunk is in the head of the update queue. */ +static void coroutine_fn checksum_co_wait_for_update(ImgChecksumWorker *w) +{ + ImgChecksumState *s =3D w->state; + + /* Must be called only when we are ready to update the hash. */ + assert(w->ready); + + while (QTAILQ_FIRST(&s->update_queue) !=3D w) { + qemu_coroutine_yield(); + } + + QTAILQ_REMOVE(&s->update_queue, w, entry); +} + +/* Enter the next worker coroutine if the worker is ready. */ +static void coroutine_fn checksum_co_enter_next(ImgChecksumWorker *w) +{ + ImgChecksumState *s =3D w->state; + ImgChecksumWorker *next; + + if (!QTAILQ_EMPTY(&s->update_queue)) { + next =3D QTAILQ_FIRST(&s->update_queue); + if (next->ready) + qemu_coroutine_enter(next->co); + } +} + +static int coroutine_fn checksum_co_read(ImgChecksumWorker *w) +{ + ImgChecksumState *s =3D w->state; + int err; + + err =3D blk_co_pread(s->blk, w->offset, w->length, w->buf, 0); + if (err < 0) { + error_report("Error reading at offset %" PRId64 " of %s: %s", + w->offset, s->filename, strerror(-err)); + /* Unschedule this chunk. */ + QTAILQ_REMOVE(&s->update_queue, w, entry); + s->ret =3D err; + return -1; + } + + w->ready =3D true; + + return 0; +} + +static int checksum_update_hash(ImgChecksumWorker *w) +{ + ImgChecksumState *s =3D w->state; + int err; + + if (w->zero) { + err =3D blkhash_zero(s->hash, w->length); + if (err) { + error_report("Error zeroing hash at offset %" PRId64 " of %s: = %s", + w->offset, s->filename, strerror(err)); + s->ret =3D -err; + return -1; + } + } else { + err =3D blkhash_update(s->hash, w->buf, w->length); + if (err) { + error_report("Error updating hash at offset %" PRId64 " of %s:= %s", + w->offset, s->filename, strerror(err)); + s->ret =3D -err; + return -1; + } + } + + return 0; +} + +static void coroutine_fn checksum_co_loop(void *opaque) +{ + ImgChecksumWorker *w =3D opaque; + ImgChecksumState *s =3D w->state; + + s->running_coroutines++; + w->buf =3D blk_blockalign(s->blk, CHECKSUM_BUF_SIZE); + + while (checksum_co_next(w)) { + if (!w->zero && checksum_co_read(w)) { + break; + } + checksum_co_wait_for_update(w); + if (s->ret =3D=3D -EINPROGRESS) { + if (checksum_update_hash(w)) { + break; + } + qemu_progress_print(((float) w->length / s->total_size) * 100,= 100); + } + checksum_co_enter_next(w); + } + + qemu_vfree(w->buf); + w->co =3D NULL; + s->running_coroutines--; + + if (s->running_coroutines =3D=3D 0 && s->ret =3D=3D -EINPROGRESS) { + /* the checksum job finished successfully */ + s->ret =3D 0; + } +} + +static int checksum_loop(ImgChecksumState *s) +{ + int i; + + s->ret =3D -EINPROGRESS; + + qemu_co_mutex_init(&s->lock); + QTAILQ_INIT(&s->update_queue); + + for (i =3D 0; i < CHECKSUM_COROUTINES; i++) { + ImgChecksumWorker *w =3D &s->workers[i]; + w->state =3D s; + w->co =3D qemu_coroutine_create(checksum_co_loop, w); + qemu_coroutine_enter(w->co); + } + + while (s->running_coroutines) { + main_loop_wait(false); + } + + return s->ret; +} + /* * Compute image checksum. */ static int img_checksum(int argc, char **argv) { const char *digest_name =3D "sha256"; const size_t block_size =3D 64 * KiB; =20 const char *format =3D NULL; const char *cache =3D BDRV_DEFAULT_CACHE; - const char *filename; - BlockBackend *blk; - BlockDriverState *bs; - uint8_t *buf =3D NULL; - int64_t pnum; bool progress =3D false; + bool image_opts =3D false; + int flags =3D 0; - bool writethrough; - int64_t total_size; - int64_t offset =3D 0; + bool writethrough =3D false; int c; - bool image_opts =3D false; - struct blkhash *h =3D NULL; - unsigned char digest[64]; - unsigned int digest_len; + int ret =3D 1; int err; =20 + ImgChecksumState s =3D {0}; + unsigned char digest[64]; + unsigned int digest_len; + for (;;) { static const struct option long_options[] =3D { {"help", no_argument, 0, 'h'}, {"object", required_argument, 0, OPTION_OBJECT}, {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, {0, 0, 0, 0} }; c =3D getopt_long(argc, argv, ":hf:T:p", long_options, NULL); if (c =3D=3D -1) { @@ -1697,119 +1937,76 @@ static int img_checksum(int argc, char **argv) case OPTION_IMAGE_OPTS: image_opts =3D true; break; } } =20 if (optind !=3D argc - 1) { error_exit("Expecting image file name"); } =20 - filename =3D argv[optind++]; + s.filename =3D argv[optind++]; =20 err =3D bdrv_parse_cache_mode(cache, &flags, &writethrough); if (err < 0) { error_report("Invalid source cache option: %s", cache); return ret; } =20 - blk =3D img_open(image_opts, filename, format, flags, writethrough, fa= lse, - false); - if (!blk) { + s.blk =3D img_open(image_opts, s.filename, format, flags, writethrough, + false, false); + if (!s.blk) { return ret; } =20 /* Initialize before using goto out. */ qemu_progress_init(progress, 2.0); =20 - bs =3D blk_bs(blk); - buf =3D blk_blockalign(blk, IO_BUF_SIZE); + s.bs =3D blk_bs(s.blk); =20 - total_size =3D blk_getlength(blk); - if (total_size < 0) { + s.total_size =3D blk_getlength(s.blk); + if (s.total_size < 0) { error_report("Can't get size of %s: %s", - filename, strerror(-total_size)); + s.filename, strerror(-s.total_size)); goto out; } =20 - h =3D blkhash_new(block_size, digest_name); - if (!h) { + s.hash =3D blkhash_new(block_size, digest_name); + if (!s.hash) { error_report("Can't create blkhash: %s", strerror(errno)); goto out; } =20 qemu_progress_print(0, 100); =20 - while (offset < total_size) { - int status; - int64_t chunk; - - status =3D bdrv_block_status_above(bs, NULL, offset, - total_size - offset, &pnum, NULL, - NULL); - if (status < 0) { - error_report("Error checking status at offset %" PRId64 " for = %s", - offset, filename); - goto out; - } - - assert(pnum); - chunk =3D pnum; - - if (status & BDRV_BLOCK_ZERO) { - chunk =3D MIN(chunk, SIZE_MAX); - err =3D blkhash_zero(h, chunk); - if (err) { - error_report("Error zeroing hash at offset %" PRId64 - " of %s: %s", - offset, filename, strerror(err)); - goto out; - } - } else { - chunk =3D MIN(chunk, IO_BUF_SIZE); - err =3D blk_pread(blk, offset, chunk, buf, 0); - if (err < 0) { - error_report("Error reading at offset %" PRId64 " of %s: %= s", - offset, filename, strerror(-err)); - goto out; - } - err =3D blkhash_update(h, buf, chunk); - if (err) { - error_report("Error updating hash at offset %" PRId64 - " of %s: %s", - offset, filename, strerror(err)); - goto out; - } - } - - offset +=3D chunk; - qemu_progress_print(((float) chunk / total_size) * 100, 100); + err =3D checksum_loop(&s); + if (err) { + goto out; } =20 - err =3D blkhash_final(h, digest, &digest_len); + err =3D blkhash_final(s.hash, digest, &digest_len); if (err) { error_report("Error finalizing hash of %s: %s", - filename, strerror(err)); + s.filename, strerror(err)); goto out; } =20 for (unsigned i =3D 0; i < digest_len; i++) { printf("%02x", digest[i]); } - printf(" %s%s", filename, progress ? "" : "\n"); + printf(" %s%s", s.filename, progress ? "" : "\n"); =20 ret =3D 0; =20 out: - blkhash_free(h); - qemu_vfree(buf); - blk_unref(blk); + blkhash_free(s.hash); + blk_unref(s.blk); qemu_progress_end(); =20 return ret; } #endif /* CONFIG_BLKHASH */ =20 /* Convenience wrapper around qmp_block_dirty_bitmap_merge */ static void do_dirty_bitmap_merge(const char *dst_node, const char *dst_na= me, const char *src_node, const char *src_na= me, Error **errp) --=20 2.37.2