From nobody Sun Oct 5 19:25:47 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1524522974459675.5486721824681; Mon, 23 Apr 2018 15:36:14 -0700 (PDT) Received: from localhost ([::1]:55718 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fAk44-0007io-Pp for importer@patchew.org; Mon, 23 Apr 2018 18:36:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41594) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fAk1z-0006Pi-1m for qemu-devel@nongnu.org; Mon, 23 Apr 2018 18:33:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fAk1x-0005d9-Un for qemu-devel@nongnu.org; Mon, 23 Apr 2018 18:33:55 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:49026 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fAk1r-0005YM-8s; Mon, 23 Apr 2018 18:33:47 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 98E88406C79C; Mon, 23 Apr 2018 22:33:45 +0000 (UTC) Received: from red.redhat.com (ovpn-122-114.rdu2.redhat.com [10.10.122.114]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F5DF7C52; Mon, 23 Apr 2018 22:33:45 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 23 Apr 2018 17:33:34 -0500 Message-Id: <20180423223337.82366-3-eblake@redhat.com> In-Reply-To: <20180423223337.82366-1-eblake@redhat.com> References: <20180423223337.82366-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 23 Apr 2018 22:33:45 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 23 Apr 2018 22:33:45 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'eblake@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 2/5] qcow2: Document some maximum size constraints X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, berto@igalia.com, qemu-block@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Although off_t permits up to 63 bits (8EB) of file offsets, in practice, we're going to hit other limits first. Document some of those limits in the qcow2 spec, and how choice of cluster size can influence some of the limits. While at it, notice that since we cannot map any virtual cluster to any address higher than 64 PB (56 bits) (due to the current L1/L2 field encoding stopping at bit 55), it makes little sense to require the refcount table to access host offsets beyond that point. Mark the upper bits of the refcount table entries as reserved to match the L1/L2 table, with no ill effects, since it is unlikely that there are any existing images larger than 64PB in the first place, and thus all existing images already have those bits as 0. If 64PB proves to be too small in the future, we could enlarge all three uses of bit 55 into the reserved bits at that time. (For reference, ext4 with 4k blocks caps files at 16PB.) However, there is one limit that reserved bits don't help with: for compressed clusters, the L2 layout requires an ever-smaller maximum host offset as cluster size gets larger, down to a 512 TB maximum with 2M clusters. Signed-off-by: Eric Blake -- v5: even more wording tweaks v4: more wording tweaks v3: new patch Reviewed-by: Alberto Garcia --- docs/interop/qcow2.txt | 40 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index feb711fb6a8..f0338f356fe 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -40,7 +40,17 @@ The first cluster of a qcow2 image contains the file hea= der: with larger cluster sizes. 24 - 31: size - Virtual disk size in bytes + Virtual disk size in bytes. + + Note: with a 2 MB cluster size, the maximum + virtual size is 2 EB (61 bits) for a sparse file, + but other sizing limitations in refcount and L1/L2 + tables mean that an image cannot have more than 64 + PB (56 bits) of populated clusters (assuming the + image does not first hit other limits such as a + file system's maximum size). With a 512 byte + cluster size, the maximum virtual size drops to + 128 GB (37 bits). 32 - 35: crypt_method 0 for no encryption @@ -326,6 +336,15 @@ in the image file. It contains pointers to the second level structures which are called refco= unt blocks and are exactly one cluster in size. +All qcow2 metadata, including the refcount table and refcount blocks, +must currently reside at host offsets below 64 PB (56 bits) (if the +underlying protocol can even be sized that large); this limit could be +enlarged by putting reserved bits into use, but only if a similar +limit on L1/L2 tables is revisited at the same time. While a larger +cluster size theoretically allows the refcount table to cover more +host offsets, in practice, other inherent limits will constrain the +maximum image size before the refcount table is full. + Given a offset into the image file, the refcount of its cluster can be obt= ained as follows: @@ -341,7 +360,7 @@ Refcount table entry: Bit 0 - 8: Reserved (set to 0) - 9 - 63: Bits 9-63 of the offset into the image file at which t= he + 9 - 55: Bits 9-55 of the offset into the image file at which t= he refcount block starts. Must be aligned to a cluster boundary. @@ -349,6 +368,8 @@ Refcount table entry: been allocated. All refcounts managed by this refcount= block are 0. + 56 - 63: Reserved (set to 0) + Refcount block entry (x =3D refcount_bits - 1): Bit 0 - x: Reference count of the cluster. If refcount_bits impli= es a @@ -365,6 +386,17 @@ The L1 table has a variable size (stored in the header= ) and may use multiple clusters, however it must be contiguous in the image file. L2 tables are exactly one cluster in size. +The L1 and L2 tables have implications on the maximum virtual file +size; a larger cluster size is required for the guest to have access +to more space. Furthermore, a virtual cluster must currently map to a +host offset below 64 PB (56 bits) (this limit could be enlarged by +putting reserved bits into use, but only if a similar limit on +refcount tables is revisited at the same time). Additionally, with +larger cluster sizes, compressed clusters have a smaller limit on host +cluster mappings (a 2M cluster size requires compressed clusters to +reside below 512 TB (49 bits), where enlarging this would require an +incompatible layout change). + Given a offset into the virtual disk, the offset into the image file can be obtained as follows: @@ -427,7 +459,9 @@ Standard Cluster Descriptor: Compressed Clusters Descriptor (x =3D 62 - (cluster_bits - 8)): Bit 0 - x-1: Host cluster offset. This is usually _not_ aligned to a - cluster or sector boundary! + cluster or sector boundary! If cluster_bits is + small enough that this field includes bits beyond + 55, those upper bits must be set to 0. x - 61: Number of additional 512-byte sectors used for the compressed data, beyond the sector containing the offs= et --=20 2.14.3