From nobody Tue Apr 8 23:37:22 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1598345470; cv=none; d=zohomail.com; s=zohoarc; b=fYjHmTr7qBrQ1IuCjtA2Vs7KhFt+ROr09u2PPSkkNgCuGyVnOdTZCCiept7niGl0C6uDxG8qvZyPu9qRL9sl5Lf68jLKvWbqPpCEGdkG3pThIuuWoZK6HS+jHqGjSQq1KLPtJ3hua8Lp9BlvszmI1Cy8bscqs4eWQZIFei6zHaE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598345470; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=n9E6K5bbj69wBRK9kEo5471ZbhOQQHF9y1l72IRIynA=; b=hsKZZwgpnA9wayAUMplMxWdUeEevq7AEniqMSMlfghgLJymL+jfYEGkfB97a3AUSo9svFkhDwOcK3CmmwlQZ4ZtcONV+9Wdy9JluCFwAXhnorf0smRD0FHss+VdePgOYJ0FQV+4eb4fscrOMbD5ibuSRN6bmzG49gNbp+bxgauw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598345470339328.7497202949136; Tue, 25 Aug 2020 01:51:10 -0700 (PDT) Received: from localhost ([::1]:34774 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kAUfc-0004QL-R4 for importer@patchew.org; Tue, 25 Aug 2020 04:51:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60964) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kAUPk-0005hz-Bm for qemu-devel@nongnu.org; Tue, 25 Aug 2020 04:34:44 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:30466 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kAUPf-0000lh-9V for qemu-devel@nongnu.org; Tue, 25 Aug 2020 04:34:44 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-228-yZhiA864MAiTlEHAj6Kpig-1; Tue, 25 Aug 2020 04:34:33 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E9DAA80732F; Tue, 25 Aug 2020 08:34:32 +0000 (UTC) Received: from localhost (ovpn-113-72.ams2.redhat.com [10.36.113.72]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 01F841A92A; Tue, 25 Aug 2020 08:34:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1598344478; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n9E6K5bbj69wBRK9kEo5471ZbhOQQHF9y1l72IRIynA=; b=QW+sGUvg5EjZyZz21f23oQvwW/tIBl2kFTceFU0V+pvXzSckkHV2VvApb288XaMC2hR6ED c26fda1jE84SKGRX7mKQRvx9B4qM68ewzyk1esJ2SRAjtwDOM1QVGFpZHuctkQqs94zOZW XCxKbBBERdHhSvgwDhYsVT4U/jyqVIE= X-MC-Unique: yZhiA864MAiTlEHAj6Kpig-1 From: Max Reitz To: qemu-block@nongnu.org Subject: [PULL 34/34] iotests: Add tests for qcow2 images with extended L2 entries Date: Tue, 25 Aug 2020 10:33:11 +0200 Message-Id: <20200825083311.1098442-35-mreitz@redhat.com> In-Reply-To: <20200825083311.1098442-1-mreitz@redhat.com> References: <20200825083311.1098442-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mreitz@redhat.com X-Mimecast-Spam-Score: 0.003 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.120; envelope-from=mreitz@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/25 02:05:08 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.956, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Peter Maydell , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Alberto Garcia Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Message-Id: Signed-off-by: Max Reitz --- tests/qemu-iotests/271 | 901 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/271.out | 726 ++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 1628 insertions(+) create mode 100755 tests/qemu-iotests/271 create mode 100644 tests/qemu-iotests/271.out diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271 new file mode 100755 index 0000000000..39ff462328 --- /dev/null +++ b/tests/qemu-iotests/271 @@ -0,0 +1,901 @@ +#!/bin/bash +# +# Test qcow2 images with extended L2 entries +# +# Copyright (C) 2019-2020 Igalia, S.L. +# Author: Alberto Garcia +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# creator +owner=3Dberto@igalia.com + +seq=3D"$(basename $0)" +echo "QA output created by $seq" + +here=3D"$PWD" +status=3D1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img + rm -f "$TEST_IMG.raw" +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file nfs +_supported_os Linux +_unsupported_imgopts extended_l2 compat=3D0.10 cluster_size data_file refc= ount_bits=3D1[^0-9] + +l2_offset=3D$((0x40000)) + +_verify_img() +{ + $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG.raw" | grep -v 'Images are id= entical' + $QEMU_IMG check "$TEST_IMG" | _filter_qemu_img_check | \ + grep -v 'No errors were found on the image' +} + +# Compare the bitmap of an extended L2 entry against an expected value +_verify_l2_bitmap() +{ + entry_no=3D"$1" # L2 entry number, starting from 0 + expected_alloc=3D"$alloc" # Space-separated list of allocated subclus= ter indexes + expected_zero=3D"$zero" # Space-separated list of zero subcluster i= ndexes + + offset=3D$(($l2_offset + $entry_no * 16)) + entry=3D$(peek_file_be "$TEST_IMG" $offset 8) + offset=3D$(($offset + 8)) + bitmap=3D$(peek_file_be "$TEST_IMG" $offset 8) + + expected_bitmap=3D0 + for bit in $expected_alloc; do + expected_bitmap=3D$(($expected_bitmap | (1 << $bit))) + done + for bit in $expected_zero; do + expected_bitmap=3D$(($expected_bitmap | (1 << (32 + $bit)))) + done + printf -v expected_bitmap "%u" $expected_bitmap # Convert to unsigned + + printf "L2 entry #%d: 0x%016x %016x\n" "$entry_no" "$entry" "$bitmap" + if [ "$bitmap" !=3D "$expected_bitmap" ]; then + printf "ERROR: expecting bitmap 0x%016x\n" "$expected_bitmap" + fi +} + +# This should be called as _run_test c=3DXXX sc=3DXXX off=3DXXX len=3DXXX = cmd=3DXXX +# c: cluster number (0 if unset) +# sc: subcluster number inside cluster @c (0 if unset) +# off: offset inside subcluster @sc, in kilobytes (0 if unset) +# len: request length, passed directly to qemu-io (e.g: 256, 4k, 1M, ...) +# cmd: the command to pass to qemu-io, must be one of +# write -> write +# zero -> write -z +# unmap -> write -z -u +# compress -> write -c +# discard -> discard +_run_test() +{ + unset c sc off len cmd + for var in "$@"; do eval "$var"; done + case "${cmd:-write}" in + zero) + cmd=3D"write -q -z";; + unmap) + cmd=3D"write -q -z -u";; + compress) + pat=3D$((${pat:-0} + 1)) + cmd=3D"write -q -c -P ${pat}";; + write) + pat=3D$((${pat:-0} + 1)) + cmd=3D"write -q -P ${pat}";; + discard) + cmd=3D"discard -q";; + *) + echo "Unknown option $cmd" + exit 1;; + esac + c=3D"${c:-0}" + sc=3D"${sc:-0}" + off=3D"${off:-0}" + offset=3D"$(($c * 64 + $sc * 2 + $off))" + [ "$offset" !=3D 0 ] && offset=3D"${offset}k" + cmd=3D"$cmd ${offset} ${len}" + raw_cmd=3D$(echo $cmd | sed s/-c//) # Raw images don't support -c + echo $cmd | sed 's/-P [0-9][0-9]\?/-P PATTERN/' + $QEMU_IO -c "$cmd" "$TEST_IMG" | _filter_qemu_io + $QEMU_IO -c "$raw_cmd" -f raw "$TEST_IMG.raw" | _filter_qemu_io + _verify_img + _verify_l2_bitmap "$c" +} + +_reset_img() +{ + size=3D"$1" + $QEMU_IMG create -f raw "$TEST_IMG.raw" "$size" | _filter_img_create + if [ "$use_backing_file" =3D "yes" ]; then + $QEMU_IMG create -f raw "$TEST_IMG.base" "$size" | _filter_img_cre= ate + $QEMU_IO -c "write -q -P 0xFF 0 $size" -f raw "$TEST_IMG.base" | _= filter_qemu_io + $QEMU_IO -c "write -q -P 0xFF 0 $size" -f raw "$TEST_IMG.raw" | _f= ilter_qemu_io + _make_test_img -o extended_l2=3Don -F raw -b "$TEST_IMG.base" "$si= ze" + else + _make_test_img -o extended_l2=3Don "$size" + fi +} + +############################################################ +############################################################ +############################################################ + +# Test that writing to an image with subclusters produces the expected +# results, in images with and without backing files +for use_backing_file in yes no; do + echo + echo "### Standard write tests (backing file: $use_backing_file) ###" + echo + _reset_img 1M + ### Write subcluster #0 (beginning of subcluster) ### + alloc=3D"0"; zero=3D"" + _run_test sc=3D0 len=3D1k + + ### Write subcluster #1 (middle of subcluster) ### + alloc=3D"0 1"; zero=3D"" + _run_test sc=3D1 off=3D1 len=3D512 + + ### Write subcluster #2 (end of subcluster) ### + alloc=3D"0 1 2"; zero=3D"" + _run_test sc=3D2 off=3D1 len=3D1k + + ### Write subcluster #3 (full subcluster) ### + alloc=3D"0 1 2 3"; zero=3D"" + _run_test sc=3D3 len=3D2k + + ### Write subclusters #4-6 (full subclusters) ### + alloc=3D"$(seq 0 6)"; zero=3D"" + _run_test sc=3D4 len=3D6k + + ### Write subclusters #7-9 (partial subclusters) ### + alloc=3D"$(seq 0 9)"; zero=3D"" + _run_test sc=3D7 off=3D1 len=3D4k + + ### Write subcluster #16 (partial subcluster) ### + alloc=3D"$(seq 0 9) 16"; zero=3D"" + _run_test sc=3D16 len=3D1k + + ### Write subcluster #31-#33 (cluster overlap) ### + alloc=3D"$(seq 0 9) 16 31"; zero=3D"" + _run_test sc=3D31 off=3D1 len=3D4k + alloc=3D"0 1" ; zero=3D"" + _verify_l2_bitmap 1 + + ### Zero subcluster #1 + alloc=3D"0 $(seq 2 9) 16 31"; zero=3D"1" + _run_test sc=3D1 len=3D2k cmd=3Dzero + + ### Zero cluster #0 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test sc=3D0 len=3D64k cmd=3Dzero + + ### Fill cluster #0 with data + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test sc=3D0 len=3D64k + + ### Zero and unmap half of cluster #0 (this won't unmap it) + alloc=3D"$(seq 16 31)"; zero=3D"$(seq 0 15)" + _run_test sc=3D0 len=3D32k cmd=3Dunmap + + ### Zero and unmap cluster #0 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test sc=3D0 len=3D64k cmd=3Dunmap + + ### Write subcluster #1 (middle of subcluster) + alloc=3D"1"; zero=3D"0 $(seq 2 31)" + _run_test sc=3D1 off=3D1 len=3D512 + + ### Fill cluster #0 with data + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test sc=3D0 len=3D64k + + ### Discard cluster #0 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test sc=3D0 len=3D64k cmd=3Ddiscard + + ### Write compressed data to cluster #0 + alloc=3D""; zero=3D"" + _run_test sc=3D0 len=3D64k cmd=3Dcompress + + ### Write subcluster #1 (middle of subcluster) + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test sc=3D1 off=3D1 len=3D512 +done + +############################################################ +############################################################ +############################################################ + +# calculate_l2_meta() checks if none of the clusters affected by a +# write operation need COW or changes to their L2 metadata and simply +# returns when they don't. This is a test for that optimization. +# Here clusters #0-#3 are overwritten but only #1 and #2 need changes. +echo +echo '### Overwriting several clusters without COW ###' +echo +use_backing_file=3D"no" _reset_img 1M +# Write cluster #0, subclusters #12-#31 +alloc=3D"$(seq 12 31)"; zero=3D"" +_run_test sc=3D12 len=3D40k + +# Write cluster #1, subcluster #13 +alloc=3D"13"; zero=3D"" +_run_test c=3D1 sc=3D13 len=3D2k + +# Zeroize cluster #2, subcluster #14 +alloc=3D"14"; zero=3D"" +_run_test c=3D2 sc=3D14 len=3D2k +alloc=3D""; zero=3D"14" +_run_test c=3D2 sc=3D14 len=3D2k cmd=3Dzero + +# Write cluster #3, subclusters #0-#16 +alloc=3D"$(seq 0 16)"; zero=3D"" +_run_test c=3D3 sc=3D0 len=3D34k + +# Write from cluster #0, subcluster #12 to cluster #3, subcluster #11 +alloc=3D"$(seq 12 31)"; zero=3D"" +_run_test sc=3D12 len=3D192k +alloc=3D"$(seq 0 31)"; zero=3D"" +_verify_l2_bitmap 1 +_verify_l2_bitmap 2 + +alloc=3D"$(seq 0 16)"; zero=3D"" +_verify_l2_bitmap 3 + +############################################################ +############################################################ +############################################################ + +# Test different patterns of writing zeroes +for use_backing_file in yes no; do + echo + echo "### Writing zeroes 1: unallocated clusters (backing file: $use_b= acking_file) ###" + echo + # Note that the image size is not a multiple of the cluster size + _reset_img 2083k + + # Cluster-aligned request from clusters #0 to #2 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test c=3D0 sc=3D0 len=3D192k cmd=3Dzero + _verify_l2_bitmap 1 + _verify_l2_bitmap 2 + + # Subcluster-aligned request from clusters #3 to #5 + alloc=3D""; zero=3D"$(seq 16 31)" + _run_test c=3D3 sc=3D16 len=3D128k cmd=3Dzero + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 4 + alloc=3D""; zero=3D"$(seq 0 15)" + _verify_l2_bitmap 5 + + # Unaligned request from clusters #6 to #8 + if [ "$use_backing_file" =3D "yes" ]; then + alloc=3D"15"; zero=3D"$(seq 16 31)" # copy-on-write happening here + else + alloc=3D""; zero=3D"$(seq 15 31)" + fi + _run_test c=3D6 sc=3D15 off=3D1 len=3D128k cmd=3Dzero + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 7 + if [ "$use_backing_file" =3D "yes" ]; then + alloc=3D"15"; zero=3D"$(seq 0 14)" # copy-on-write happening here + else + alloc=3D""; zero=3D"$(seq 0 15)" + fi + _verify_l2_bitmap 8 + + echo + echo "### Writing zeroes 2: allocated clusters (backing file: $use_bac= king_file) ###" + echo + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test c=3D9 sc=3D0 len=3D576k + _verify_l2_bitmap 10 + _verify_l2_bitmap 11 + _verify_l2_bitmap 12 + _verify_l2_bitmap 13 + _verify_l2_bitmap 14 + _verify_l2_bitmap 15 + _verify_l2_bitmap 16 + _verify_l2_bitmap 17 + + # Cluster-aligned request from clusters #9 to #11 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test c=3D9 sc=3D0 len=3D192k cmd=3Dzero + _verify_l2_bitmap 10 + _verify_l2_bitmap 11 + + # Subcluster-aligned request from clusters #12 to #14 + alloc=3D"$(seq 0 15)"; zero=3D"$(seq 16 31)" + _run_test c=3D12 sc=3D16 len=3D128k cmd=3Dzero + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 13 + alloc=3D"$(seq 16 31)"; zero=3D"$(seq 0 15)" + _verify_l2_bitmap 14 + + # Unaligned request from clusters #15 to #17 + alloc=3D"$(seq 0 15)"; zero=3D"$(seq 16 31)" + _run_test c=3D15 sc=3D15 off=3D1 len=3D128k cmd=3Dzero + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 16 + alloc=3D"$(seq 15 31)"; zero=3D"$(seq 0 14)" + _verify_l2_bitmap 17 + + echo + echo "### Writing zeroes 3: compressed clusters (backing file: $use_ba= cking_file) ###" + echo + alloc=3D""; zero=3D"" + for c in $(seq 18 28); do + _run_test c=3D$c sc=3D0 len=3D64k cmd=3Dcompress + done + + # Cluster-aligned request from clusters #18 to #20 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test c=3D18 sc=3D0 len=3D192k cmd=3Dzero + _verify_l2_bitmap 19 + _verify_l2_bitmap 20 + + # Subcluster-aligned request from clusters #21 to #23. + # We cannot partially zero a compressed cluster so the code + # returns -ENOTSUP, which means copy-on-write of the compressed + # data and fill the rest with actual zeroes on disk. + # TODO: cluster #22 should use the 'all zeroes' bits. + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test c=3D21 sc=3D16 len=3D128k cmd=3Dzero + _verify_l2_bitmap 22 + _verify_l2_bitmap 23 + + # Unaligned request from clusters #24 to #26 + # In this case QEMU internally sends a 1k request followed by a + # subcluster-aligned 128k request. The first request decompresses + # cluster #24, but that's not enough to perform the second request + # efficiently because it partially writes to cluster #26 (which is + # compressed) so we hit the same problem as before. + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test c=3D24 sc=3D15 off=3D1 len=3D129k cmd=3Dzero + _verify_l2_bitmap 25 + _verify_l2_bitmap 26 + + # Unaligned request from clusters #27 to #29 + # Similar to the previous case, but this time the tail of the + # request does not correspond to a compressed cluster, so it can + # be zeroed efficiently. + # Note that the very last subcluster is partially written, so if + # there's a backing file we need to perform cow. + alloc=3D"$(seq 0 15)"; zero=3D"$(seq 16 31)" + _run_test c=3D27 sc=3D15 off=3D1 len=3D128k cmd=3Dzero + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 28 + if [ "$use_backing_file" =3D "yes" ]; then + alloc=3D"15"; zero=3D"$(seq 0 14)" # copy-on-write happening here + else + alloc=3D""; zero=3D"$(seq 0 15)" + fi + _verify_l2_bitmap 29 + + echo + echo "### Writing zeroes 4: other tests (backing file: $use_backing_fi= le) ###" + echo + # Unaligned request in the middle of cluster #30. + # If there's a backing file we need to allocate and do + # copy-on-write on the partially zeroed subclusters. + # If not we can set the 'all zeroes' bit on them. + if [ "$use_backing_file" =3D "yes" ]; then + alloc=3D"15 19"; zero=3D"$(seq 16 18)" # copy-on-write happening h= ere + else + alloc=3D""; zero=3D"$(seq 15 19)" + fi + _run_test c=3D30 sc=3D15 off=3D1 len=3D8k cmd=3Dzero + + # Fill the last cluster with zeroes, up to the end of the image + # (the image size is not a multiple of the cluster or subcluster size). + alloc=3D""; zero=3D"$(seq 0 17)" + _run_test c=3D32 sc=3D0 len=3D35k cmd=3Dzero +done + +############################################################ +############################################################ +############################################################ + +# Zero + unmap +for use_backing_file in yes no; do + echo + echo "### Zero + unmap 1: allocated clusters (backing file: $use_backi= ng_file) ###" + echo + # Note that the image size is not a multiple of the cluster size + _reset_img 2083k + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test c=3D9 sc=3D0 len=3D576k + _verify_l2_bitmap 10 + _verify_l2_bitmap 11 + _verify_l2_bitmap 12 + _verify_l2_bitmap 13 + _verify_l2_bitmap 14 + _verify_l2_bitmap 15 + _verify_l2_bitmap 16 + _verify_l2_bitmap 17 + + # Cluster-aligned request from clusters #9 to #11 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test c=3D9 sc=3D0 len=3D192k cmd=3Dunmap + _verify_l2_bitmap 10 + _verify_l2_bitmap 11 + + # Subcluster-aligned request from clusters #12 to #14 + alloc=3D"$(seq 0 15)"; zero=3D"$(seq 16 31)" + _run_test c=3D12 sc=3D16 len=3D128k cmd=3Dunmap + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 13 + alloc=3D"$(seq 16 31)"; zero=3D"$(seq 0 15)" + _verify_l2_bitmap 14 + + # Unaligned request from clusters #15 to #17 + alloc=3D"$(seq 0 15)"; zero=3D"$(seq 16 31)" + _run_test c=3D15 sc=3D15 off=3D1 len=3D128k cmd=3Dunmap + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 16 + alloc=3D"$(seq 15 31)"; zero=3D"$(seq 0 14)" + _verify_l2_bitmap 17 + + echo + echo "### Zero + unmap 2: compressed clusters (backing file: $use_back= ing_file) ###" + echo + alloc=3D""; zero=3D"" + for c in $(seq 18 28); do + _run_test c=3D$c sc=3D0 len=3D64k cmd=3Dcompress + done + + # Cluster-aligned request from clusters #18 to #20 + alloc=3D""; zero=3D"$(seq 0 31)" + _run_test c=3D18 sc=3D0 len=3D192k cmd=3Dunmap + _verify_l2_bitmap 19 + _verify_l2_bitmap 20 + + # Subcluster-aligned request from clusters #21 to #23. + # We cannot partially zero a compressed cluster so the code + # returns -ENOTSUP, which means copy-on-write of the compressed + # data and fill the rest with actual zeroes on disk. + # TODO: cluster #22 should use the 'all zeroes' bits. + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test c=3D21 sc=3D16 len=3D128k cmd=3Dunmap + _verify_l2_bitmap 22 + _verify_l2_bitmap 23 + + # Unaligned request from clusters #24 to #26 + # In this case QEMU internally sends a 1k request followed by a + # subcluster-aligned 128k request. The first request decompresses + # cluster #24, but that's not enough to perform the second request + # efficiently because it partially writes to cluster #26 (which is + # compressed) so we hit the same problem as before. + alloc=3D"$(seq 0 31)"; zero=3D"" + _run_test c=3D24 sc=3D15 off=3D1 len=3D129k cmd=3Dunmap + _verify_l2_bitmap 25 + _verify_l2_bitmap 26 + + # Unaligned request from clusters #27 to #29 + # Similar to the previous case, but this time the tail of the + # request does not correspond to a compressed cluster, so it can + # be zeroed efficiently. + # Note that the very last subcluster is partially written, so if + # there's a backing file we need to perform cow. + alloc=3D"$(seq 0 15)"; zero=3D"$(seq 16 31)" + _run_test c=3D27 sc=3D15 off=3D1 len=3D128k cmd=3Dunmap + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 28 + if [ "$use_backing_file" =3D "yes" ]; then + alloc=3D"15"; zero=3D"$(seq 0 14)" # copy-on-write happening here + else + alloc=3D""; zero=3D"$(seq 0 15)" + fi + _verify_l2_bitmap 29 +done + +############################################################ +############################################################ +############################################################ + +# Test qcow2_cluster_discard() with full and normal discards +for use_backing_file in yes no; do + echo + echo "### Discarding clusters with non-zero bitmaps (backing file: $us= e_backing_file) ###" + echo + if [ "$use_backing_file" =3D "yes" ]; then + _make_test_img -o extended_l2=3Don -F raw -b "$TEST_IMG.base" 1M + else + _make_test_img -o extended_l2=3Don 1M + fi + # Write clusters #0-#2 and then discard them + $QEMU_IO -c 'write -q 0 128k' "$TEST_IMG" + $QEMU_IO -c 'discard -q 0 128k' "$TEST_IMG" + # 'qemu-io discard' doesn't do a full discard, it zeroizes the + # cluster, so both clusters have all zero bits set now + alloc=3D""; zero=3D"$(seq 0 31)" + _verify_l2_bitmap 0 + _verify_l2_bitmap 1 + # Now mark the 2nd half of the subclusters from cluster #0 as unalloca= ted + poke_file "$TEST_IMG" $(($l2_offset+8)) "\x00\x00" + # Discard cluster #0 again to see how the zero bits have changed + $QEMU_IO -c 'discard -q 0 64k' "$TEST_IMG" + # And do a full discard of cluster #1 by shrinking and growing the ima= ge + $QEMU_IMG resize --shrink "$TEST_IMG" 64k + $QEMU_IMG resize "$TEST_IMG" 1M + # A normal discard sets all 'zero' bits only if the image has a + # backing file, otherwise it won't touch them. + if [ "$use_backing_file" =3D "yes" ]; then + alloc=3D""; zero=3D"$(seq 0 31)" + else + alloc=3D""; zero=3D"$(seq 0 15)" + fi + _verify_l2_bitmap 0 + # A full discard should clear the L2 entry completely. However + # when growing an image with a backing file the new clusters are + # zeroized to hide the stale data from the backing file + if [ "$use_backing_file" =3D "yes" ]; then + alloc=3D""; zero=3D"$(seq 0 31)" + else + alloc=3D""; zero=3D"" + fi + _verify_l2_bitmap 1 +done + +############################################################ +############################################################ +############################################################ + +# Test that corrupted L2 entries are detected in both read and write +# operations +for corruption_test_cmd in read write; do + echo + echo "### Corrupted L2 entries - $corruption_test_cmd test (allocated)= ###" + echo + echo "# 'cluster is zero' bit set on the standard cluster descriptor" + echo + # We actually don't consider this a corrupted image. + # The 'cluster is zero' bit is unused in extended L2 entries so + # QEMU ignores it. + # TODO: maybe treat the image as corrupted and make qemu-img check fix= it? + _make_test_img -o extended_l2=3Don 1M + $QEMU_IO -c 'write -q -P 0x11 0 2k' "$TEST_IMG" + poke_file "$TEST_IMG" $(($l2_offset+7)) "\x01" + alloc=3D"0"; zero=3D"" + _verify_l2_bitmap 0 + $QEMU_IO -c "$corruption_test_cmd -q -P 0x11 0 1k" "$TEST_IMG" + if [ "$corruption_test_cmd" =3D "write" ]; then + alloc=3D"0"; zero=3D"" + fi + _verify_l2_bitmap 0 + + echo + echo "# Both 'subcluster is zero' and 'subcluster is allocated' bits s= et" + echo + _make_test_img -o extended_l2=3Don 1M + # Write from the middle of cluster #0 to the middle of cluster #2 + $QEMU_IO -c 'write -q 32k 128k' "$TEST_IMG" + # Corrupt the L2 entry from cluster #1 + poke_file_be "$TEST_IMG" $(($l2_offset+24)) 4 1 + alloc=3D"$(seq 0 31)"; zero=3D"0" + _verify_l2_bitmap 1 + $QEMU_IO -c "$corruption_test_cmd 0 192k" "$TEST_IMG" + + echo + echo "### Corrupted L2 entries - $corruption_test_cmd test (unallocate= d) ###" + echo + echo "# 'cluster is zero' bit set on the standard cluster descriptor" + echo + # We actually don't consider this a corrupted image. + # The 'cluster is zero' bit is unused in extended L2 entries so + # QEMU ignores it. + # TODO: maybe treat the image as corrupted and make qemu-img check fix= it? + _make_test_img -o extended_l2=3Don 1M + # We want to modify the (empty) L2 entry from cluster #0, + # but we write to #4 in order to initialize the L2 table first + $QEMU_IO -c 'write -q 256k 1k' "$TEST_IMG" + poke_file "$TEST_IMG" $(($l2_offset+7)) "\x01" + alloc=3D""; zero=3D"" + _verify_l2_bitmap 0 + $QEMU_IO -c "$corruption_test_cmd -q 0 1k" "$TEST_IMG" + if [ "$corruption_test_cmd" =3D "write" ]; then + alloc=3D"0"; zero=3D"" + fi + _verify_l2_bitmap 0 + + echo + echo "# 'subcluster is allocated' bit set" + echo + _make_test_img -o extended_l2=3Don 1M + # We want to corrupt the (empty) L2 entry from cluster #0, + # but we write to #4 in order to initialize the L2 table first + $QEMU_IO -c 'write -q 256k 1k' "$TEST_IMG" + poke_file "$TEST_IMG" $(($l2_offset+15)) "\x01" + alloc=3D"0"; zero=3D"" + _verify_l2_bitmap 0 + $QEMU_IO -c "$corruption_test_cmd 0 1k" "$TEST_IMG" + + echo + echo "# Both 'subcluster is zero' and 'subcluster is allocated' bits s= et" + echo + _make_test_img -o extended_l2=3Don 1M + # We want to corrupt the (empty) L2 entry from cluster #1, + # but we write to #4 in order to initialize the L2 table first + $QEMU_IO -c 'write -q 256k 1k' "$TEST_IMG" + # Corrupt the L2 entry from cluster #1 + poke_file_be "$TEST_IMG" $(($l2_offset+24)) 8 $(((1 << 32) | 1)) + alloc=3D"0"; zero=3D"0" + _verify_l2_bitmap 1 + $QEMU_IO -c "$corruption_test_cmd 0 192k" "$TEST_IMG" + + echo + echo "### Compressed cluster with subcluster bitmap !=3D 0 - $corrupti= on_test_cmd test ###" + echo + # We actually don't consider this a corrupted image. + # The bitmap in compressed clusters is unused so QEMU should just igno= re it. + _make_test_img -o extended_l2=3Don 1M + $QEMU_IO -c 'write -q -P 11 -c 0 64k' "$TEST_IMG" + # Change the L2 bitmap to allocate subcluster #31 and zeroize subclust= er #0 + poke_file "$TEST_IMG" $(($l2_offset+11)) "\x01\x80" + alloc=3D"31"; zero=3D"0" + _verify_l2_bitmap 0 + $QEMU_IO -c "$corruption_test_cmd -P 11 0 64k" "$TEST_IMG" | _filter_q= emu_io + # Writing allocates a new uncompressed cluster so we get a new bitmap + if [ "$corruption_test_cmd" =3D "write" ]; then + alloc=3D"$(seq 0 31)"; zero=3D"" + fi + _verify_l2_bitmap 0 +done + +############################################################ +############################################################ +############################################################ + +echo +echo "### Detect and repair unaligned clusters ###" +echo +# Create a backing file and fill it with data +$QEMU_IMG create -f raw "$TEST_IMG.base" 128k | _filter_img_create +$QEMU_IO -c "write -q -P 0xff 0 128k" -f raw "$TEST_IMG.base" | _filter_qe= mu_io + +echo "# Corrupted L2 entry, allocated subcluster #" +# Create a new image, allocate a cluster and write some data to it +_make_test_img -o extended_l2=3Don -F raw -b "$TEST_IMG.base" +$QEMU_IO -c 'write -q -P 1 4k 2k' "$TEST_IMG" +# Corrupt the L2 entry by making the offset unaligned +poke_file "$TEST_IMG" "$(($l2_offset+6))" "\x02" +# This cannot be repaired, qemu-img check will fail to fix it +_check_test_img -r all +# Attempting to read the image will still show that it's corrupted +$QEMU_IO -c 'read -q 0 2k' "$TEST_IMG" + +echo "# Corrupted L2 entry, no allocated subclusters #" +# Create a new image, allocate a cluster and zeroize subcluster #2 +_make_test_img -o extended_l2=3Don -F raw -b "$TEST_IMG.base" +$QEMU_IO -c 'write -q -P 1 4k 2k' "$TEST_IMG" +$QEMU_IO -c 'write -q -z 4k 2k' "$TEST_IMG" +# Corrupt the L2 entry by making the offset unaligned +poke_file "$TEST_IMG" "$(($l2_offset+6))" "\x02" +# This time none of the subclusters are allocated so we can repair the ima= ge +_check_test_img -r all +# And the data can be read normally +$QEMU_IO -c 'read -q -P 0xff 0 4k' "$TEST_IMG" +$QEMU_IO -c 'read -q -P 0x00 4k 2k' "$TEST_IMG" +$QEMU_IO -c 'read -q -P 0xff 6k 122k' "$TEST_IMG" + +############################################################ +############################################################ +############################################################ + +echo +echo "### Image creation options ###" +echo +echo "# cluster_size < 16k" +_make_test_img -o extended_l2=3Don,cluster_size=3D8k 1M + +echo "# backing file and preallocation=3Dmetadata" +# For preallocation with backing files, create a backing file first +$QEMU_IMG create -f raw "$TEST_IMG.base" 1M | _filter_img_create +$QEMU_IO -c "write -q -P 0xff 0 1M" -f raw "$TEST_IMG.base" | _filter_qemu= _io + +_make_test_img -o extended_l2=3Don,preallocation=3Dmetadata -F raw -b "$TE= ST_IMG.base" 512k +$QEMU_IMG resize "$TEST_IMG" 1M +$QEMU_IO -c 'read -P 0xff 0 512k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 512k 512k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG map "$TEST_IMG" | _filter_testdir + +echo "# backing file and preallocation=3Dfalloc" +_make_test_img -o extended_l2=3Don,preallocation=3Dfalloc -F raw -b "$TEST= _IMG.base" 512k +$QEMU_IMG resize "$TEST_IMG" 1M +$QEMU_IO -c 'read -P 0xff 0 512k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 512k 512k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG map "$TEST_IMG" | _filter_testdir + +echo "# backing file and preallocation=3Dfull" +_make_test_img -o extended_l2=3Don,preallocation=3Dfull -F raw -b "$TEST_I= MG.base" 512k +$QEMU_IMG resize "$TEST_IMG" 1M +$QEMU_IO -c 'read -P 0xff 0 512k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 512k 512k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG map "$TEST_IMG" | _filter_testdir + +echo +echo "### Image resizing with preallocation and backing files ###" +echo +# In this case the new subclusters must have the 'all zeroes' bit set +echo "# resize --preallocation=3Dmetadata" +_make_test_img -o extended_l2=3Don -F raw -b "$TEST_IMG.base" 503k +$QEMU_IMG resize --preallocation=3Dmetadata "$TEST_IMG" 1013k +$QEMU_IO -c 'read -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 503k 510k' "$TEST_IMG" | _filter_qemu_io + +# In this case and the next one the new subclusters must be allocated +echo "# resize --preallocation=3Dfalloc" +_make_test_img -o extended_l2=3Don -F raw -b "$TEST_IMG.base" 503k +$QEMU_IMG resize --preallocation=3Dfalloc "$TEST_IMG" 1013k +$QEMU_IO -c 'read -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 503k 510k' "$TEST_IMG" | _filter_qemu_io + +echo "# resize --preallocation=3Dfull" +_make_test_img -o extended_l2=3Don -F raw -b "$TEST_IMG.base" 503k +$QEMU_IMG resize --preallocation=3Dfull "$TEST_IMG" 1013k +$QEMU_IO -c 'read -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 503k 510k' "$TEST_IMG" | _filter_qemu_io + +echo +echo "### Image resizing with preallocation without backing files ###" +echo +# In this case the new subclusters must have the 'all zeroes' bit set +echo "# resize --preallocation=3Dmetadata" +_make_test_img -o extended_l2=3Don 503k +$QEMU_IO -c 'write -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG resize --preallocation=3Dmetadata "$TEST_IMG" 1013k +$QEMU_IO -c 'read -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 503k 510k' "$TEST_IMG" | _filter_qemu_io + +# In this case and the next one the new subclusters must be allocated +echo "# resize --preallocation=3Dfalloc" +_make_test_img -o extended_l2=3Don 503k +$QEMU_IO -c 'write -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG resize --preallocation=3Dfalloc "$TEST_IMG" 1013k +$QEMU_IO -c 'read -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 503k 510k' "$TEST_IMG" | _filter_qemu_io + +echo "# resize --preallocation=3Dfull" +_make_test_img -o extended_l2=3Don 503k +$QEMU_IO -c 'write -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG resize --preallocation=3Dfull "$TEST_IMG" 1013k +$QEMU_IO -c 'read -P 0xff 0 503k' "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x00 503k 510k' "$TEST_IMG" | _filter_qemu_io + +echo +echo "### qemu-img measure ###" +echo +echo "# 512MB, extended_l2=3Doff" # This needs one L2 table +$QEMU_IMG measure --size 512M -O qcow2 -o extended_l2=3Doff +echo "# 512MB, extended_l2=3Don" # This needs two L2 tables +$QEMU_IMG measure --size 512M -O qcow2 -o extended_l2=3Don + +echo "# 16K clusters, 64GB, extended_l2=3Doff" # This needs one full L1 ta= ble cluster +$QEMU_IMG measure --size 64G -O qcow2 -o cluster_size=3D16k,extended_l2=3D= off +echo "# 16K clusters, 64GB, extended_l2=3Don" # This needs two full L2 ta= ble clusters +$QEMU_IMG measure --size 64G -O qcow2 -o cluster_size=3D16k,extended_l2=3D= on + +echo "# 8k clusters" # This should fail +$QEMU_IMG measure --size 1M -O qcow2 -o cluster_size=3D8k,extended_l2=3Don + +echo "# 1024 TB" # Maximum allowed size with extended_l2=3Don and 64K clus= ters +$QEMU_IMG measure --size 1024T -O qcow2 -o extended_l2=3Don +echo "# 1025 TB" # This should fail +$QEMU_IMG measure --size 1025T -O qcow2 -o extended_l2=3Don + +echo +echo "### qemu-img amend ###" +echo +_make_test_img -o extended_l2=3Don 1M +$QEMU_IMG amend -o extended_l2=3Doff "$TEST_IMG" && echo "Unexpected pass" + +_make_test_img -o extended_l2=3Doff 1M +$QEMU_IMG amend -o extended_l2=3Don "$TEST_IMG" && echo "Unexpected pass" + +echo +echo "### Test copy-on-write on an image with snapshots ###" +echo +_make_test_img -o extended_l2=3Don 1M + +# For each cluster from #0 to #9 this loop zeroes subcluster #7 +# and allocates subclusters #13 and #18. +alloc=3D"13 18"; zero=3D"7" +for c in $(seq 0 9); do + $QEMU_IO -c "write -q -z $((64*$c+14))k 2k" \ + -c "write -q -P $((0xd0+$c)) $((64*$c+26))k 2k" \ + -c "write -q -P $((0xe0+$c)) $((64*$c+36))k 2k" "$TEST_IMG" + _verify_l2_bitmap "$c" +done + +# Create a snapshot and set l2_offset to the new L2 table +$QEMU_IMG snapshot -c snap1 "$TEST_IMG" +l2_offset=3D$((0x110000)) + +# Write different patterns to each one of the clusters +# in order to see how copy-on-write behaves in each case. +$QEMU_IO -c "write -q -P 0xf0 $((64*0+30))k 1k" \ + -c "write -q -P 0xf1 $((64*1+20))k 1k" \ + -c "write -q -P 0xf2 $((64*2+40))k 1k" \ + -c "write -q -P 0xf3 $((64*3+26))k 1k" \ + -c "write -q -P 0xf4 $((64*4+14))k 1k" \ + -c "write -q -P 0xf5 $((64*5+1))k 1k" \ + -c "write -q -z $((64*6+30))k 3k" \ + -c "write -q -z $((64*7+26))k 2k" \ + -c "write -q -z $((64*8+26))k 1k" \ + -c "write -q -z $((64*9+12))k 1k" \ + "$TEST_IMG" +alloc=3D"$(seq 13 18)"; zero=3D"7" _verify_l2_bitmap 0 +alloc=3D"$(seq 10 18)"; zero=3D"7" _verify_l2_bitmap 1 +alloc=3D"$(seq 13 20)"; zero=3D"7" _verify_l2_bitmap 2 +alloc=3D"$(seq 13 18)"; zero=3D"7" _verify_l2_bitmap 3 +alloc=3D"$(seq 7 18)"; zero=3D"" _verify_l2_bitmap 4 +alloc=3D"$(seq 0 18)"; zero=3D"" _verify_l2_bitmap 5 +alloc=3D"13 18"; zero=3D"7 15 16" _verify_l2_bitmap 6 +alloc=3D"18"; zero=3D"7 13" _verify_l2_bitmap 7 +alloc=3D"$(seq 13 18)"; zero=3D"7" _verify_l2_bitmap 8 +alloc=3D"13 18"; zero=3D"6 7" _verify_l2_bitmap 9 + +echo +echo "### Test concurrent requests ###" +echo + +_concurrent_io() +{ +# Allocate three subclusters in the same cluster. +# This works because handle_dependencies() checks whether the requests +# allocate the same cluster, even if the COW regions don't overlap (in +# this case they don't). +cat <