[PATCH v8 00/34] Add subcluster allocation to qcow2

Alberto Garcia posted 34 patches 3 years, 10 months ago
Test docker-mingw@fedora passed
Test checkpatch passed
Test asan passed
Test docker-quick@centos7 failed
Test FreeBSD passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/cover.1591801197.git.berto@igalia.com
Maintainers: Kevin Wolf <kwolf@redhat.com>, Markus Armbruster <armbru@redhat.com>, Max Reitz <mreitz@redhat.com>, Eric Blake <eblake@redhat.com>
There is a newer version of this series
docs/interop/qcow2.txt           |  68 ++-
docs/qcow2-cache.txt             |  19 +-
qapi/block-core.json             |   7 +
block/qcow2.h                    | 211 ++++++-
include/block/block_int.h        |   1 +
block/qcow2-cluster.c            | 912 +++++++++++++++++++++----------
block/qcow2-refcount.c           |  38 +-
block/qcow2.c                    | 304 +++++++----
block/trace-events               |   2 +-
tests/qemu-iotests/031.out       |   8 +-
tests/qemu-iotests/036.out       |   4 +-
tests/qemu-iotests/049.out       | 102 ++--
tests/qemu-iotests/060.out       |   1 +
tests/qemu-iotests/061           |   6 +
tests/qemu-iotests/061.out       |  25 +-
tests/qemu-iotests/065           |  12 +-
tests/qemu-iotests/082.out       |  48 +-
tests/qemu-iotests/085.out       |  38 +-
tests/qemu-iotests/144.out       |   4 +-
tests/qemu-iotests/182.out       |   2 +-
tests/qemu-iotests/185.out       |   8 +-
tests/qemu-iotests/198.out       |   2 +
tests/qemu-iotests/206.out       |   6 +-
tests/qemu-iotests/242.out       |   5 +
tests/qemu-iotests/255.out       |   8 +-
tests/qemu-iotests/271           | 801 +++++++++++++++++++++++++++
tests/qemu-iotests/271.out       | 676 +++++++++++++++++++++++
tests/qemu-iotests/274.out       |  49 +-
tests/qemu-iotests/280.out       |   2 +-
tests/qemu-iotests/291.out       |   2 +
tests/qemu-iotests/common.filter |   1 +
tests/qemu-iotests/group         |   1 +
32 files changed, 2807 insertions(+), 566 deletions(-)
create mode 100755 tests/qemu-iotests/271
create mode 100644 tests/qemu-iotests/271.out
[PATCH v8 00/34] Add subcluster allocation to qcow2
Posted by Alberto Garcia 3 years, 10 months ago
Hi,

here's the new version of the patches to add subcluster allocation
support to qcow2.

Please refer to the cover letter of the first version for a full
description of the patches:

   https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

The big change here is that now when an image is preallocated then the
requested clusters are allocated but the L2 bitmap is left untouched.
This makes it possible to preallocate an image that has a backing
file.

If you want to test this series make sure to apply this patch first:

   https://lists.gnu.org/archive/html/qemu-block/2020-06/msg00504.html

Berto

v8:
- Patch 30: New patch
- Patch 31: Update test expectations after commit cf2d1203dc
- Patch 32: New patch
- Patch 34: New tests, fixes and general refactoring of the code

v7: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg01683.html
v6: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg01583.html
v5: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg00251.html
v4: https://lists.gnu.org/archive/html/qemu-block/2020-03/msg00966.html
v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html
v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html
v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

Output of git backport-diff against v7:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/34:[----] [--] 'qcow2: Make Qcow2AioTask store the full host offset'
002/34:[----] [--] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()'
003/34:[----] [--] 'qcow2: Add calculate_l2_meta()'
004/34:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()'
005/34:[----] [--] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()'
006/34:[----] [--] 'qcow2: Add get_l2_entry() and set_l2_entry()'
007/34:[----] [--] 'qcow2: Document the Extended L2 Entries feature'
008/34:[----] [--] 'qcow2: Add dummy has_subclusters() function'
009/34:[----] [--] 'qcow2: Add subcluster-related fields to BDRVQcow2State'
010/34:[----] [--] 'qcow2: Add offset_to_sc_index()'
011/34:[----] [--] 'qcow2: Add offset_into_subcluster() and size_to_subclusters()'
012/34:[----] [--] 'qcow2: Add l2_entry_size()'
013/34:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()'
014/34:[----] [--] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()'
015/34:[----] [--] 'qcow2: Add qcow2_get_subcluster_range_type()'
016/34:[----] [--] 'qcow2: Add qcow2_cluster_is_allocated()'
017/34:[----] [--] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()'
018/34:[----] [--] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*'
019/34:[----] [--] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC'
020/34:[----] [--] 'qcow2: Add subcluster support to calculate_l2_meta()'
021/34:[----] [--] 'qcow2: Add subcluster support to qcow2_get_host_offset()'
022/34:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()'
023/34:[----] [--] 'qcow2: Add subcluster support to discard_in_l2_slice()'
024/34:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()'
025/34:[----] [--] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()'
026/34:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster'
027/34:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()'
028/34:[----] [--] 'qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()'
029/34:[----] [-C] 'qcow2: Add subcluster support to qcow2_measure()'
030/34:[down] 'qcow2: Add prealloc field to QCowL2Meta'
031/34:[0002] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit'
032/34:[down] 'qcow2: Allow preallocation and backing files if extended_l2 is set'
033/34:[----] [--] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters'
034/34:[0669] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries'

Alberto Garcia (34):
  qcow2: Make Qcow2AioTask store the full host offset
  qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()
  qcow2: Add calculate_l2_meta()
  qcow2: Split cluster_needs_cow() out of count_cow_clusters()
  qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()
  qcow2: Add get_l2_entry() and set_l2_entry()
  qcow2: Document the Extended L2 Entries feature
  qcow2: Add dummy has_subclusters() function
  qcow2: Add subcluster-related fields to BDRVQcow2State
  qcow2: Add offset_to_sc_index()
  qcow2: Add offset_into_subcluster() and size_to_subclusters()
  qcow2: Add l2_entry_size()
  qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()
  qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()
  qcow2: Add qcow2_get_subcluster_range_type()
  qcow2: Add qcow2_cluster_is_allocated()
  qcow2: Add cluster type parameter to qcow2_get_host_offset()
  qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
  qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
  qcow2: Add subcluster support to calculate_l2_meta()
  qcow2: Add subcluster support to qcow2_get_host_offset()
  qcow2: Add subcluster support to zero_in_l2_slice()
  qcow2: Add subcluster support to discard_in_l2_slice()
  qcow2: Add subcluster support to check_refcounts_l2()
  qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
  qcow2: Clear the L2 bitmap when allocating a compressed cluster
  qcow2: Add subcluster support to handle_alloc_space()
  qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
  qcow2: Add subcluster support to qcow2_measure()
  qcow2: Add prealloc field to QCowL2Meta
  qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
  qcow2: Allow preallocation and backing files if extended_l2 is set
  qcow2: Assert that expand_zero_clusters_in_l1() does not support
    subclusters
  iotests: Add tests for qcow2 images with extended L2 entries

 docs/interop/qcow2.txt           |  68 ++-
 docs/qcow2-cache.txt             |  19 +-
 qapi/block-core.json             |   7 +
 block/qcow2.h                    | 211 ++++++-
 include/block/block_int.h        |   1 +
 block/qcow2-cluster.c            | 912 +++++++++++++++++++++----------
 block/qcow2-refcount.c           |  38 +-
 block/qcow2.c                    | 304 +++++++----
 block/trace-events               |   2 +-
 tests/qemu-iotests/031.out       |   8 +-
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++--
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061           |   6 +
 tests/qemu-iotests/061.out       |  25 +-
 tests/qemu-iotests/065           |  12 +-
 tests/qemu-iotests/082.out       |  48 +-
 tests/qemu-iotests/085.out       |  38 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/185.out       |   8 +-
 tests/qemu-iotests/198.out       |   2 +
 tests/qemu-iotests/206.out       |   6 +-
 tests/qemu-iotests/242.out       |   5 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/271           | 801 +++++++++++++++++++++++++++
 tests/qemu-iotests/271.out       | 676 +++++++++++++++++++++++
 tests/qemu-iotests/274.out       |  49 +-
 tests/qemu-iotests/280.out       |   2 +-
 tests/qemu-iotests/291.out       |   2 +
 tests/qemu-iotests/common.filter |   1 +
 tests/qemu-iotests/group         |   1 +
 32 files changed, 2807 insertions(+), 566 deletions(-)
 create mode 100755 tests/qemu-iotests/271
 create mode 100644 tests/qemu-iotests/271.out

-- 
2.20.1


Re: [PATCH v8 00/34] Add subcluster allocation to qcow2
Posted by Eric Blake 3 years, 10 months ago
On 6/10/20 10:02 AM, Alberto Garcia wrote:
> Hi,
> 
> here's the new version of the patches to add subcluster allocation
> support to qcow2.
> 
> Please refer to the cover letter of the first version for a full
> description of the patches:
> 
>     https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html
> 
> The big change here is that now when an image is preallocated then the
> requested clusters are allocated but the L2 bitmap is left untouched.
> This makes it possible to preallocate an image that has a backing
> file.
> 
> If you want to test this series make sure to apply this patch first:
> 
>     https://lists.gnu.org/archive/html/qemu-block/2020-06/msg00504.html

Let's spell that the way patchew can recognize:
Based-on: <20200610094600.4029-1-berto@igalia.com>

> 
> Berto
> 
> v8:
> - Patch 30: New patch
> - Patch 31: Update test expectations after commit cf2d1203dc
> - Patch 32: New patch
> - Patch 34: New tests, fixes and general refactoring of the code
> 


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v8 00/34] Add subcluster allocation to qcow2
Posted by no-reply@patchew.org 3 years, 10 months ago
Patchew URL: https://patchew.org/QEMU/cover.1591801197.git.berto@igalia.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/vhdx.o
  CC      block/vhdx-endian.o
/tmp/qemu-test/src/block/qcow2-cluster.c: In function 'qcow2_get_host_offset':
/tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (type != expected_type) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:449:25: note: 'expected_type' was declared here
     QCow2SubclusterType expected_type, type;
                         ^
/tmp/qemu-test/src/block/qcow2-cluster.c:475:19: error: 'check_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (check_offset) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:447:10: note: 'check_offset' was declared here
     bool check_offset;
          ^
/tmp/qemu-test/src/block/qcow2-cluster.c:476:29: error: 'expected_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
             expected_offset += s->cluster_size;
                             ^
/tmp/qemu-test/src/block/qcow2-cluster.c:448:14: note: 'expected_offset' was declared here
     uint64_t expected_offset;
              ^
cc1: all warnings being treated as errors
make: *** [block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 665, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=080447e1604744fb934c6e9a0210ed36', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-yrq9p6br/src/docker-src.2020-06-10-17.42.20.24850:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=080447e1604744fb934c6e9a0210ed36
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-yrq9p6br/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m47.601s
user    0m8.384s


The full log is available at
http://patchew.org/logs/cover.1591801197.git.berto@igalia.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH v8 00/34] Add subcluster allocation to qcow2
Posted by no-reply@patchew.org 3 years, 10 months ago
Patchew URL: https://patchew.org/QEMU/cover.1591801197.git.berto@igalia.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/blkdebug.o
  CC      block/blkverify.o
/tmp/qemu-test/src/block/qcow2-cluster.c: In function 'qcow2_get_host_offset':
/tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (type != expected_type) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:449:25: note: 'expected_type' was declared here
     QCow2SubclusterType expected_type, type;
                         ^
/tmp/qemu-test/src/block/qcow2-cluster.c:475:19: error: 'check_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (check_offset) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:447:10: note: 'check_offset' was declared here
     bool check_offset;
          ^
/tmp/qemu-test/src/block/qcow2-cluster.c:476:29: error: 'expected_offset' may be used uninitialized in this function [-Werror=maybe-uninitialized]
             expected_offset += s->cluster_size;
                             ^
/tmp/qemu-test/src/block/qcow2-cluster.c:448:14: note: 'expected_offset' was declared here
     uint64_t expected_offset;
              ^
cc1: all warnings being treated as errors
make: *** [block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 665, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=c26843e7a1d24a3c860bd5ab2506a33c', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-h2erlgmp/src/docker-src.2020-06-10-18.50.30.7791:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=c26843e7a1d24a3c860bd5ab2506a33c
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-h2erlgmp/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m20.876s
user    0m8.581s


The full log is available at
http://patchew.org/logs/cover.1591801197.git.berto@igalia.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com