From nobody Sat Apr 20 04:48:15 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1557893892; cv=none; d=zoho.com; s=zohoarc; b=ElIr3Yo9ML4TdWTPduEISjd2y+tTp8KZMdOFIu6MIKCkSJQhomie8ovSiVePnrTgzgjoU+mU5NSyTrrdU6bAuHXP31cd1yS3Y9ixbeA5JkrTZyDRJzXhYlYz35PgyCcU3oNwjMOsU5kXCPUKVzt8s/vk3Hrz91zmLGTFqMV5cdg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1557893892; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=2XTRZbv7wmlbKqOIgQzfKkprIIlJMrn8f8iWJ2QFj+U=; b=Wocrv7VsX/QFvbdxXfyiX1z5xrDYMbaosqZqIN+aj9We1cO6L4uHQNpnkMcMLITXPpCThGhFJV+kU6woSeJHULbp7WNZw/9mVjBVanh4d9HMk++bm6Ed/h3djyXrRj3WFqBQCNRooAMz/3e3jgyxlF4lYgbHFV0YMRVBw+qsddo= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 155789389248512.530178093688164; Tue, 14 May 2019 21:18:12 -0700 (PDT) Received: from localhost ([127.0.0.1]:59126 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hQlMQ-0008NH-N4 for importer@patchew.org; Wed, 15 May 2019 00:17:46 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56160) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hQlKa-00075V-87 for qemu-devel@nongnu.org; Wed, 15 May 2019 00:15:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hQlKZ-0004oF-5W for qemu-devel@nongnu.org; Wed, 15 May 2019 00:15:52 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43580) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hQlKU-0004e9-U4; Wed, 15 May 2019 00:15:47 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2B3043086211; Wed, 15 May 2019 04:15:46 +0000 (UTC) Received: from localhost (ovpn-204-29.brq.redhat.com [10.40.204.29]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B688B60BE5; Wed, 15 May 2019 04:15:45 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Wed, 15 May 2019 06:15:40 +0200 Message-Id: <20190515041541.12367-2-mreitz@redhat.com> In-Reply-To: <20190515041541.12367-1-mreitz@redhat.com> References: <20190515041541.12367-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Wed, 15 May 2019 04:15:46 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v2 1/2] block/file-posix: Unaligned O_DIRECT block-status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-stable@nongnu.org, qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" Currently, qemu crashes whenever someone queries the block status of an unaligned image tail of an O_DIRECT image: $ echo > foo $ qemu-img map --image-opts driver=3Dfile,filename=3Dfoo,cache.direct=3Don Offset Length Mapped to File qemu-img: block/io.c:2093: bdrv_co_block_status: Assertion `*pnum && QEMU_IS_ALIGNED(*pnum, align) && align > offset - aligned_offset' failed. This is because bdrv_co_block_status() checks that the result returned by the driver's implementation is aligned to the request_alignment, but file-posix can fail to do so, which is actually mentioned in a comment there: "[...] possibly including a partial sector at EOF". Fix this by rounding up those partial sectors. There are two possible alternative fixes: (1) We could refuse to open unaligned image files with O_DIRECT altogether. That sounds reasonable until you realize that qcow2 does necessarily not fill up its metadata clusters, and that nobody runs qemu-img create with O_DIRECT. Therefore, unpreallocated qcow2 files usually have an unaligned image tail. (2) bdrv_co_block_status() could ignore unaligned tails. It actually throws away everything past the EOF already, so that sounds reasonable. Unfortunately, the block layer knows file lengths only with a granularity of BDRV_SECTOR_SIZE, so bdrv_co_block_status() usually would have to guess whether its file length information is inexact or whether the driver is broken. Fixing what raw_co_block_status() returns is the safest thing to do. There seems to be no other block driver that sets request_alignment and does not make sure that it always returns aligned values. Cc: qemu-stable@nongnu.org Signed-off-by: Max Reitz Reviewed-by: Eric Blake --- block/file-posix.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/block/file-posix.c b/block/file-posix.c index e09e15bbf8..d018429672 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -2488,6 +2488,8 @@ static int coroutine_fn raw_co_block_status(BlockDriv= erState *bs, off_t data =3D 0, hole =3D 0; int ret; =20 + assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment)); + ret =3D fd_open(bs); if (ret < 0) { return ret; @@ -2513,6 +2515,20 @@ static int coroutine_fn raw_co_block_status(BlockDri= verState *bs, /* On a data extent, compute bytes to the end of the extent, * possibly including a partial sector at EOF. */ *pnum =3D MIN(bytes, hole - offset); + + /* + * We are not allowed to return partial sectors, though, so + * round up if necessary. + */ + if (!QEMU_IS_ALIGNED(*pnum, bs->bl.request_alignment)) { + int64_t file_length =3D raw_getlength(bs); + if (file_length > 0) { + /* Ignore errors, this is just a safeguard */ + assert(hole =3D=3D file_length); + } + *pnum =3D ROUND_UP(*pnum, bs->bl.request_alignment); + } + ret =3D BDRV_BLOCK_DATA; } else { /* On a hole, compute bytes to the beginning of the next extent. = */ --=20 2.21.0 From nobody Sat Apr 20 04:48:15 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1557894005; cv=none; d=zoho.com; s=zohoarc; b=URKS4e/EW9VVSpFjEE/X6u4TUGndpAHBGT383hcmi2oyh+phjz1BW1D5pUvTn+PFERZIZ8UoCWr3D7pTm19bug6pj7ZuLpfEh+lWySU2miq5/MPNev9A4HliLtc1ONrDhyv3ZVzkW3Cxk8BFG+7LRfI3O6fGQo6QiIRBDjqr8C8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1557894005; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=0i3lh+JogsBNUytEoGVd5dwHQjD1qC9xvZWga2Qhh4w=; b=CCuoz18K5F60q6yOZmKXloG0YwEHpgq/lStz532Nhdu0xQWXQvs6+/cty+Ol+BTtX9GVkr6x+McZno2yxbaI36unatAkwBJl1taSWxqiybkhvpLnSZ7xGWA6FkrcGze4zOnPKTGF2hivV5Dn47UWtG/Yfd0K2iXLzPlFauUKw7o= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1557894005808948.1216381817352; Tue, 14 May 2019 21:20:05 -0700 (PDT) Received: from localhost ([127.0.0.1]:59144 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hQlOe-0001U5-M0 for importer@patchew.org; Wed, 15 May 2019 00:20:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56178) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hQlKb-0007C4-UI for qemu-devel@nongnu.org; Wed, 15 May 2019 00:15:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hQlKa-0004sE-G9 for qemu-devel@nongnu.org; Wed, 15 May 2019 00:15:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55888) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hQlKX-0004hk-7W; Wed, 15 May 2019 00:15:49 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 837093082A8D; Wed, 15 May 2019 04:15:48 +0000 (UTC) Received: from localhost (ovpn-204-29.brq.redhat.com [10.40.204.29]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1AE6F60BE5; Wed, 15 May 2019 04:15:47 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Wed, 15 May 2019 06:15:41 +0200 Message-Id: <20190515041541.12367-3-mreitz@redhat.com> In-Reply-To: <20190515041541.12367-1-mreitz@redhat.com> References: <20190515041541.12367-1-mreitz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Wed, 15 May 2019 04:15:48 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v2 2/2] iotests: Test unaligned raw images with O_DIRECT X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-stable@nongnu.org, qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" We already have 221 for accesses through the page cache, but it is better to create a new file for O_DIRECT instead of integrating those test cases into 221. This way, we can make use of _supported_cache_modes (and _default_cache_mode) so the test is automatically skipped on filesystems that do not support O_DIRECT. As part of the split, add _supported_cache_modes to 221. With that, it no longer fails when run with -c none or -c directsync. Signed-off-by: Max Reitz Reviewed-by: Eric Blake --- tests/qemu-iotests/221 | 4 ++ tests/qemu-iotests/253 | 84 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/253.out | 14 +++++++ tests/qemu-iotests/group | 1 + 4 files changed, 103 insertions(+) create mode 100755 tests/qemu-iotests/253 create mode 100644 tests/qemu-iotests/253.out diff --git a/tests/qemu-iotests/221 b/tests/qemu-iotests/221 index 25dd47bcfe..0e9096fec7 100755 --- a/tests/qemu-iotests/221 +++ b/tests/qemu-iotests/221 @@ -1,6 +1,7 @@ #!/usr/bin/env bash # # Test qemu-img vs. unaligned images +# (See also 253, which is the O_DIRECT version) # # Copyright (C) 2018-2019 Red Hat, Inc. # @@ -37,6 +38,9 @@ _supported_fmt raw _supported_proto file _supported_os Linux =20 +_default_cache_mode writeback +_supported_cache_modes writeback writethrough unsafe + echo echo "=3D=3D=3D Check mapping of unaligned raw image =3D=3D=3D" echo diff --git a/tests/qemu-iotests/253 b/tests/qemu-iotests/253 new file mode 100755 index 0000000000..d88d5afa45 --- /dev/null +++ b/tests/qemu-iotests/253 @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# +# Test qemu-img vs. unaligned images; O_DIRECT version +# (Originates from 221) +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +seq=3D"$(basename $0)" +echo "QA output created by $seq" + +status=3D1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt raw +_supported_proto file +_supported_os Linux + +_default_cache_mode none +_supported_cache_modes none directsync + +echo +echo "=3D=3D=3D Check mapping of unaligned raw image =3D=3D=3D" +echo + +# We do not know how large a physical sector is, but it is certainly +# going to be a factor of 1 MB +size=3D$((1 * 1024 * 1024 - 1)) + +# qemu-img create rounds size up to BDRV_SECTOR_SIZE +_make_test_img $size +$QEMU_IMG map --output=3Djson --image-opts \ + "driver=3D$IMGFMT,file.driver=3Dfile,file.filename=3D$TEST_IMG,cache.d= irect=3Don" \ + | _filter_qemu_img_map + +# so we resize it and check again +truncate --size=3D$size "$TEST_IMG" +$QEMU_IMG map --output=3Djson --image-opts \ + "driver=3D$IMGFMT,file.driver=3Dfile,file.filename=3D$TEST_IMG,cache.d= irect=3Don" \ + | _filter_qemu_img_map + +# qemu-io with O_DIRECT always writes whole physical sectors. Again, +# we do not know how large a physical sector is, so we just start +# writing from a 64 kB boundary, which should always be aligned. +offset=3D$((1 * 1024 * 1024 - 64 * 1024)) +$QEMU_IO -c "w $offset $((size - offset))" "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG map --output=3Djson --image-opts \ + "driver=3D$IMGFMT,file.driver=3Dfile,file.filename=3D$TEST_IMG,cache.d= irect=3Don" \ + | _filter_qemu_img_map + +# Resize it and check again -- contrary to 221, we may not get partial +# sectors here, so there should be only two areas (one zero, one +# data). +truncate --size=3D$size "$TEST_IMG" +$QEMU_IMG map --output=3Djson --image-opts \ + "driver=3D$IMGFMT,file.driver=3Dfile,file.filename=3D$TEST_IMG,cache.d= irect=3Don" \ + | _filter_qemu_img_map + +# success, all done +echo '*** done' +rm -f $seq.full +status=3D0 diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out new file mode 100644 index 0000000000..607c0baa0b --- /dev/null +++ b/tests/qemu-iotests/253.out @@ -0,0 +1,14 @@ +QA output created by 253 + +=3D=3D=3D Check mapping of unaligned raw image =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D1048575 +[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false,= "offset": OFFSET}] +[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false,= "offset": OFFSET}] +wrote 65535/65535 bytes at offset 983040 +63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, = "offset": OFFSET}, +{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": tru= e, "offset": OFFSET}] +[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, = "offset": OFFSET}, +{ "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": tru= e, "offset": OFFSET}] +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 00e474ab0a..52b7c16e15 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -250,3 +250,4 @@ 248 rw auto quick 249 rw auto quick 252 rw auto backing quick +253 rw auto quick --=20 2.21.0