From nobody Sun May 19 03:54:33 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1618500263; cv=none; d=zohomail.com; s=zohoarc; b=MLda01KMhijfUzi5a0WM+AiTY2E4czoEcJIv+Q9m7ivbBazlgBT69dap0V0xV+5aM+Z/s30anpEvbPl84bI5FkWuss/48nEcHjZiH44sfEcca8a8xcnnVed+7b3wU4Ipe6BxTgPcjBf1raliGY8lMkIAoEovvnY2+VprYG/Ljmg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1618500263; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=EoWyZ2dB1Ly3LvFvZQ+QSc42Tvdp2sayrjC5gsGvwNY=; b=As/WfR0H8QZaWUkyb6pWViR8/S0Hh9gHWie5l2Y4pVPD9OmeKIAieq+G5JxBZ6rYMIBeXR1It/lPNnsFKIaDpU8+L89l97yuIDZeigALaMj8vtL4X5AVTZQXpFzef1eiqbLPbhhS/lEgpETfxl/4Tx++tdp31EyS1O9k49eq3CU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1618500263527850.1252664042449; Thu, 15 Apr 2021 08:24:23 -0700 (PDT) Received: from localhost ([::1]:42394 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lX3qw-0005wM-C9 for importer@patchew.org; Thu, 15 Apr 2021 11:24:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50984) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lX3pO-0004NE-SX for qemu-devel@nongnu.org; Thu, 15 Apr 2021 11:22:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27407) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lX3pH-0008CE-0j for qemu-devel@nongnu.org; Thu, 15 Apr 2021 11:22:44 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-254-RDeLtA0fOcqzU_GvQtpSIQ-1; Thu, 15 Apr 2021 11:22:36 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 59B531020C41; Thu, 15 Apr 2021 15:22:35 +0000 (UTC) Received: from merkur.fritz.box (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 47241369A; Thu, 15 Apr 2021 15:22:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618500157; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EoWyZ2dB1Ly3LvFvZQ+QSc42Tvdp2sayrjC5gsGvwNY=; b=Cjd0klJZny0tdzzw/f8q1jMmb7KdS7DWM2Q8uQmgyE+ZGJsEVt9FTmBFdn+ggv/OeUIGZQ 07gDeg4HfG/i2ovK8jhvHMl8DtoDAK358B9Xny5yG0cGyoNKrEifP4JavrAu6iAocC8mus SAMFcbe3Lkof5RwXsAfFt1P7u/h1qHA= X-MC-Unique: RDeLtA0fOcqzU_GvQtpSIQ-1 From: Kevin Wolf To: qemu-block@nongnu.org Subject: [RFC PATCH 1/2] iotests: Test qemu-img convert of zeroed data cluster Date: Thu, 15 Apr 2021 17:22:13 +0200 Message-Id: <20210415152214.279844-2-kwolf@redhat.com> In-Reply-To: <20210415152214.279844-1-kwolf@redhat.com> References: <20210415152214.279844-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, pl@kamp.de, qemu-devel@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This demonstrates what happens when the block status changes in sub-min_sparse granularity, but all of the parts are zeroed out. The alignment logic in is_allocated_sectors() prevents that the target image remains fully sparse as expected, but turns it into a data cluster of explicit zeros. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/122 | 1 + tests/qemu-iotests/122.out | 10 ++++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122 index 5d550ed13e..7a213a4df9 100755 --- a/tests/qemu-iotests/122 +++ b/tests/qemu-iotests/122 @@ -251,6 +251,7 @@ $QEMU_IO -c "write -P 0 0 64k" "$TEST_IMG" 2>&1 | _filt= er_qemu_io | _filter_test $QEMU_IO -c "write 0 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_test= dir $QEMU_IO -c "write 8k 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_tes= tdir $QEMU_IO -c "write 17k 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_te= stdir +$QEMU_IO -c "write -P 0 65k 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filt= er_testdir =20 for min_sparse in 4k 8k; do echo diff --git a/tests/qemu-iotests/122.out b/tests/qemu-iotests/122.out index 3a3e121d57..dcc44a2304 100644 --- a/tests/qemu-iotests/122.out +++ b/tests/qemu-iotests/122.out @@ -192,6 +192,8 @@ wrote 1024/1024 bytes at offset 8192 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 1024/1024 bytes at offset 17408 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 1024/1024 bytes at offset 66560 +1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) =20 convert -S 4k [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "o= ffset": OFFSET}, @@ -199,7 +201,9 @@ convert -S 4k { "start": 8192, "length": 4096, "depth": 0, "zero": false, "data": true, = "offset": OFFSET}, { "start": 12288, "length": 4096, "depth": 0, "zero": true, "data": false}, { "start": 16384, "length": 4096, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, -{ "start": 20480, "length": 67088384, "depth": 0, "zero": true, "data": fa= lse}] +{ "start": 20480, "length": 46080, "depth": 0, "zero": true, "data": false= }, +{ "start": 66560, "length": 1024, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, +{ "start": 67584, "length": 67041280, "depth": 0, "zero": true, "data": fa= lse}] =20 convert -c -S 4k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, @@ -211,7 +215,9 @@ convert -c -S 4k =20 convert -S 8k [{ "start": 0, "length": 24576, "depth": 0, "zero": false, "data": true, "= offset": OFFSET}, -{ "start": 24576, "length": 67084288, "depth": 0, "zero": true, "data": fa= lse}] +{ "start": 24576, "length": 41984, "depth": 0, "zero": true, "data": false= }, +{ "start": 66560, "length": 1024, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, +{ "start": 67584, "length": 67041280, "depth": 0, "zero": true, "data": fa= lse}] =20 convert -c -S 8k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, --=20 2.30.2 From nobody Sun May 19 03:54:33 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1618500261; cv=none; d=zohomail.com; s=zohoarc; b=fRT76tmgc/QoxtQcVr9PdbM7yVnYSkDHi6dFfFGkbVMwBGZdLGDwQb5slrt37m+dq4oBHy6UJo1XWyjjgeh/aYgYpLbiF+qaCNH0DTZcwOex8cmgvewOta+a4DeLP69wYzVu2AArQ0W6x32XkqQpYjoRV74VaBpkUtrcOxsk67A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1618500261; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=MvIAIffJZkmyy5FYwfK9b2N1pZyUuOTjt4dirhwFjBE=; b=H+douhUKu+1wmvbsB8L4m1oRVwbCgQRv+rXDQX8XnTymYBcnpI6dyNAD7V5Y/moZm12Q7B7PLcq+KBkJfn16n6BKfTjHm19fVgWbzezjSrrxCfHfJwRTY8SgOa49niL9yTpwucLJIchzDmNnyXN3Ett/YXoNsryVmv0TBkXjq9k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1618500261327710.191587425149; Thu, 15 Apr 2021 08:24:21 -0700 (PDT) Received: from localhost ([::1]:42206 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lX3qt-0005rs-Jy for importer@patchew.org; Thu, 15 Apr 2021 11:24:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51042) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lX3pR-0004Ro-Se for qemu-devel@nongnu.org; Thu, 15 Apr 2021 11:22:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:22864) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lX3pO-0008Ed-N2 for qemu-devel@nongnu.org; Thu, 15 Apr 2021 11:22:49 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-512-CJcybRe3NzuHkmqqYxJ1OA-1; Thu, 15 Apr 2021 11:22:44 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1BC50107ACCA; Thu, 15 Apr 2021 15:22:43 +0000 (UTC) Received: from merkur.fritz.box (ovpn-113-169.ams2.redhat.com [10.36.113.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id A3BC32C15B; Thu, 15 Apr 2021 15:22:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618500165; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MvIAIffJZkmyy5FYwfK9b2N1pZyUuOTjt4dirhwFjBE=; b=Sqo52LngtWMVKM+LXm+GdqDIjTxh8stIP8373TQvG2LUa/dtkvY3VvYl7/Rt377hyLq3fy XD3eRwFRbZtfZ+wH0Zk5NoNsQgzH3ps5ry03OKlXq5Jc0f+XdXzMDM1HdlyzTawHLVYBic ROhnqWD/D6hY3Ib0T3Vdb744UGQPf+E= X-MC-Unique: CJcybRe3NzuHkmqqYxJ1OA-1 From: Kevin Wolf To: qemu-block@nongnu.org Subject: [RFC PATCH 2/2] qemu-img convert: Fix sparseness detection Date: Thu, 15 Apr 2021 17:22:14 +0200 Message-Id: <20210415152214.279844-3-kwolf@redhat.com> In-Reply-To: <20210415152214.279844-1-kwolf@redhat.com> References: <20210415152214.279844-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, pl@kamp.de, qemu-devel@nongnu.org, mreitz@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" In order to avoid RMW cycles, is_allocated_sectors() treats zeroed areas like non-zero data if the end of the checked area isn't aligned. This can improve the efficiency of the conversion and was introduced in commit 8dcd3c9b91a. However, it comes with a correctness problem: qemu-img convert is supposed to sparsify areas that contain only zeros, which it doesn't do any more. It turns out that this even happens when not only the unaligned area is zeroed, but also the blocks before and after it. In the bug report, conversion of a fragmented 10G image containing only zeros resulted in an image consuming 2.82 GiB even though the expected size is only 4 KiB. As a tradeoff between both, let's ignore zeroed sectors only after non-zero data to fix the alignment, but if we're only looking at zeros, keep them as such, even if it may mean additional RMW cycles. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=3D1882917 Signed-off-by: Kevin Wolf --- qemu-img.c | 18 ++++-------------- tests/qemu-iotests/122.out | 12 ++++-------- 2 files changed, 8 insertions(+), 22 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index a5993682aa..ca4eba2dd1 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -1168,20 +1168,10 @@ static int is_allocated_sectors(const uint8_t *buf,= int n, int *pnum, } =20 tail =3D (sector_num + i) & (alignment - 1); - if (tail) { - if (is_zero && i <=3D tail) { - /* treat unallocated areas which only consist - * of a small tail as allocated. */ - is_zero =3D false; - } - if (!is_zero) { - /* align up end offset of allocated areas. */ - i +=3D alignment - tail; - i =3D MIN(i, n); - } else { - /* align down end offset of zero areas. */ - i -=3D tail; - } + if (tail && !is_zero) { + /* align up end offset of allocated areas. */ + i +=3D alignment - tail; + i =3D MIN(i, n); } *pnum =3D i; return !is_zero; diff --git a/tests/qemu-iotests/122.out b/tests/qemu-iotests/122.out index dcc44a2304..fe0ea34164 100644 --- a/tests/qemu-iotests/122.out +++ b/tests/qemu-iotests/122.out @@ -199,11 +199,9 @@ convert -S 4k [{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "o= ffset": OFFSET}, { "start": 4096, "length": 4096, "depth": 0, "zero": true, "data": false}, { "start": 8192, "length": 4096, "depth": 0, "zero": false, "data": true, = "offset": OFFSET}, -{ "start": 12288, "length": 4096, "depth": 0, "zero": true, "data": false}, -{ "start": 16384, "length": 4096, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, -{ "start": 20480, "length": 46080, "depth": 0, "zero": true, "data": false= }, -{ "start": 66560, "length": 1024, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, -{ "start": 67584, "length": 67041280, "depth": 0, "zero": true, "data": fa= lse}] +{ "start": 12288, "length": 5120, "depth": 0, "zero": true, "data": false}, +{ "start": 17408, "length": 3072, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, +{ "start": 20480, "length": 67088384, "depth": 0, "zero": true, "data": fa= lse}] =20 convert -c -S 4k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, @@ -215,9 +213,7 @@ convert -c -S 4k =20 convert -S 8k [{ "start": 0, "length": 24576, "depth": 0, "zero": false, "data": true, "= offset": OFFSET}, -{ "start": 24576, "length": 41984, "depth": 0, "zero": true, "data": false= }, -{ "start": 66560, "length": 1024, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, -{ "start": 67584, "length": 67041280, "depth": 0, "zero": true, "data": fa= lse}] +{ "start": 24576, "length": 67084288, "depth": 0, "zero": true, "data": fa= lse}] =20 convert -c -S 8k [{ "start": 0, "length": 1024, "depth": 0, "zero": false, "data": true}, --=20 2.30.2