[PATCH v11 00/34] Add subcluster allocation to qcow2

Alberto Garcia posted 34 patches 3 years, 9 months ago
Test checkpatch passed
Test docker-mingw@fedora passed
Test FreeBSD passed
Test docker-quick@centos7 failed
Failed in applying to current master (apply log)
docs/interop/qcow2.txt           |  68 ++-
docs/qcow2-cache.txt             |  19 +-
qapi/block-core.json             |   7 +
block/qcow2.h                    | 211 ++++++-
include/block/block_int.h        |   1 +
block/qcow2-cluster.c            | 912 +++++++++++++++++++++----------
block/qcow2-refcount.c           |  47 +-
block/qcow2.c                    | 302 ++++++----
block/trace-events               |   2 +-
tests/qemu-iotests/031.out       |   8 +-
tests/qemu-iotests/036.out       |   4 +-
tests/qemu-iotests/049.out       | 102 ++--
tests/qemu-iotests/060.out       |   3 +-
tests/qemu-iotests/061           |   6 +
tests/qemu-iotests/061.out       |  25 +-
tests/qemu-iotests/065           |  12 +-
tests/qemu-iotests/082.out       |  39 +-
tests/qemu-iotests/085.out       |  38 +-
tests/qemu-iotests/144.out       |   4 +-
tests/qemu-iotests/182.out       |   2 +-
tests/qemu-iotests/185.out       |   8 +-
tests/qemu-iotests/198           |   2 +
tests/qemu-iotests/206.out       |   6 +-
tests/qemu-iotests/242.out       |   5 +
tests/qemu-iotests/255.out       |   8 +-
tests/qemu-iotests/271           | 901 ++++++++++++++++++++++++++++++
tests/qemu-iotests/271.out       | 726 ++++++++++++++++++++++++
tests/qemu-iotests/274.out       |  49 +-
tests/qemu-iotests/280.out       |   2 +-
tests/qemu-iotests/291.out       |   2 +
tests/qemu-iotests/common.filter |   1 +
tests/qemu-iotests/group         |   1 +
32 files changed, 2950 insertions(+), 573 deletions(-)
create mode 100755 tests/qemu-iotests/271
create mode 100644 tests/qemu-iotests/271.out
[PATCH v11 00/34] Add subcluster allocation to qcow2
Posted by Alberto Garcia 3 years, 9 months ago
Hi,

here's the new version of the patches to add subcluster allocation
support to qcow2.

Please refer to the cover letter of the first version for a full
description of the patches:

   https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

This version is rebased on top of the latest master (f2a1cf9180),
fixes the relevant conflicts (particularly after df373fb0a3) and
updates the test expectations.

Berto

v11:
- Patch 31: Fix rebase conflicts after df373fb0a3, update test
            expectations.
- Patch 34: Update test expectations.

v10: https://lists.gnu.org/archive/html/qemu-block/2020-07/msg00328.html
v9: https://lists.gnu.org/archive/html/qemu-block/2020-06/msg01526.html
v8: https://lists.gnu.org/archive/html/qemu-block/2020-06/msg00546.html
v7: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg01683.html
v6: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg01583.html
v5: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg00251.html
v4: https://lists.gnu.org/archive/html/qemu-block/2020-03/msg00966.html
v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html
v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html
v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

Output of git backport-diff against v10:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/34:[----] [--] 'qcow2: Make Qcow2AioTask store the full host offset'
002/34:[----] [--] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()'
003/34:[----] [--] 'qcow2: Add calculate_l2_meta()'
004/34:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()'
005/34:[----] [--] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()'
006/34:[----] [--] 'qcow2: Add get_l2_entry() and set_l2_entry()'
007/34:[----] [--] 'qcow2: Document the Extended L2 Entries feature'
008/34:[----] [--] 'qcow2: Add dummy has_subclusters() function'
009/34:[----] [--] 'qcow2: Add subcluster-related fields to BDRVQcow2State'
010/34:[----] [--] 'qcow2: Add offset_to_sc_index()'
011/34:[----] [--] 'qcow2: Add offset_into_subcluster() and size_to_subclusters()'
012/34:[----] [--] 'qcow2: Add l2_entry_size()'
013/34:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()'
014/34:[----] [--] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()'
015/34:[----] [--] 'qcow2: Add qcow2_get_subcluster_range_type()'
016/34:[----] [--] 'qcow2: Add qcow2_cluster_is_allocated()'
017/34:[----] [--] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()'
018/34:[----] [--] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*'
019/34:[----] [--] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC'
020/34:[----] [--] 'qcow2: Add subcluster support to calculate_l2_meta()'
021/34:[----] [--] 'qcow2: Add subcluster support to qcow2_get_host_offset()'
022/34:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()'
023/34:[----] [--] 'qcow2: Add subcluster support to discard_in_l2_slice()'
024/34:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()'
025/34:[----] [--] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()'
026/34:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster'
027/34:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()'
028/34:[----] [--] 'qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()'
029/34:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()'
030/34:[----] [--] 'qcow2: Add prealloc field to QCowL2Meta'
031/34:[0493] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit'
032/34:[----] [--] 'qcow2: Allow preallocation and backing files if extended_l2 is set'
033/34:[----] [--] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters'
034/34:[0006] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries'

Alberto Garcia (34):
  qcow2: Make Qcow2AioTask store the full host offset
  qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()
  qcow2: Add calculate_l2_meta()
  qcow2: Split cluster_needs_cow() out of count_cow_clusters()
  qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()
  qcow2: Add get_l2_entry() and set_l2_entry()
  qcow2: Document the Extended L2 Entries feature
  qcow2: Add dummy has_subclusters() function
  qcow2: Add subcluster-related fields to BDRVQcow2State
  qcow2: Add offset_to_sc_index()
  qcow2: Add offset_into_subcluster() and size_to_subclusters()
  qcow2: Add l2_entry_size()
  qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()
  qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()
  qcow2: Add qcow2_get_subcluster_range_type()
  qcow2: Add qcow2_cluster_is_allocated()
  qcow2: Add cluster type parameter to qcow2_get_host_offset()
  qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
  qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
  qcow2: Add subcluster support to calculate_l2_meta()
  qcow2: Add subcluster support to qcow2_get_host_offset()
  qcow2: Add subcluster support to zero_in_l2_slice()
  qcow2: Add subcluster support to discard_in_l2_slice()
  qcow2: Add subcluster support to check_refcounts_l2()
  qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
  qcow2: Clear the L2 bitmap when allocating a compressed cluster
  qcow2: Add subcluster support to handle_alloc_space()
  qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
  qcow2: Add subcluster support to qcow2_measure()
  qcow2: Add prealloc field to QCowL2Meta
  qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
  qcow2: Allow preallocation and backing files if extended_l2 is set
  qcow2: Assert that expand_zero_clusters_in_l1() does not support
    subclusters
  iotests: Add tests for qcow2 images with extended L2 entries

 docs/interop/qcow2.txt           |  68 ++-
 docs/qcow2-cache.txt             |  19 +-
 qapi/block-core.json             |   7 +
 block/qcow2.h                    | 211 ++++++-
 include/block/block_int.h        |   1 +
 block/qcow2-cluster.c            | 912 +++++++++++++++++++++----------
 block/qcow2-refcount.c           |  47 +-
 block/qcow2.c                    | 302 ++++++----
 block/trace-events               |   2 +-
 tests/qemu-iotests/031.out       |   8 +-
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++--
 tests/qemu-iotests/060.out       |   3 +-
 tests/qemu-iotests/061           |   6 +
 tests/qemu-iotests/061.out       |  25 +-
 tests/qemu-iotests/065           |  12 +-
 tests/qemu-iotests/082.out       |  39 +-
 tests/qemu-iotests/085.out       |  38 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/185.out       |   8 +-
 tests/qemu-iotests/198           |   2 +
 tests/qemu-iotests/206.out       |   6 +-
 tests/qemu-iotests/242.out       |   5 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/271           | 901 ++++++++++++++++++++++++++++++
 tests/qemu-iotests/271.out       | 726 ++++++++++++++++++++++++
 tests/qemu-iotests/274.out       |  49 +-
 tests/qemu-iotests/280.out       |   2 +-
 tests/qemu-iotests/291.out       |   2 +
 tests/qemu-iotests/common.filter |   1 +
 tests/qemu-iotests/group         |   1 +
 32 files changed, 2950 insertions(+), 573 deletions(-)
 create mode 100755 tests/qemu-iotests/271
 create mode 100644 tests/qemu-iotests/271.out

-- 
2.20.1


Re: [PATCH v11 00/34] Add subcluster allocation to qcow2
Posted by no-reply@patchew.org 3 years, 9 months ago
Patchew URL: https://patchew.org/QEMU/cover.1594396418.git.berto@igalia.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/snapshot.o
  CC      block/qapi.o
/tmp/qemu-test/src/block/qcow2-cluster.c: In function 'qcow2_get_host_offset':
/tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (type != expected_type) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:449:25: note: 'expected_type' was declared here
     QCow2SubclusterType expected_type, type;
                         ^
cc1: all warnings being treated as errors
make: *** [block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 669, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=10623143eef0416b9c3b987efa6621a1', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-zd6p1e8g/src/docker-src.2020-07-10-12.40.45.27056:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=10623143eef0416b9c3b987efa6621a1
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-zd6p1e8g/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    3m13.395s
user    0m8.778s


The full log is available at
http://patchew.org/logs/cover.1594396418.git.berto@igalia.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH v11 00/34] Add subcluster allocation to qcow2
Posted by Alberto Garcia 3 years, 9 months ago
On Fri 10 Jul 2020 06:43:59 PM CEST, no-reply@patchew.org wrote:
> /tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may be used uninitialized in this function [-Werror=maybe-uninitialized]
>          } else if (type != expected_type) {
>                    ^
> /tmp/qemu-test/src/block/qcow2-cluster.c:449:25: note: 'expected_type' was declared here
>      QCow2SubclusterType expected_type, type;
>                          ^

Meh, this is a false positive but I forgot to fix it. I'll do it if
there is a new version, otherwise please someone just initialize
expected_type to 0 when committing.

Berto

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2
Posted by no-reply@patchew.org 3 years, 9 months ago
Patchew URL: https://patchew.org/QEMU/cover.1594396418.git.berto@igalia.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/blkverify.o
  CC      block/blkreplay.o
/tmp/qemu-test/src/block/qcow2-cluster.c: In function 'qcow2_get_host_offset':
/tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         } else if (type != expected_type) {
                   ^
/tmp/qemu-test/src/block/qcow2-cluster.c:449:25: note: 'expected_type' was declared here
     QCow2SubclusterType expected_type, type;
                         ^
cc1: all warnings being treated as errors
make: *** [block/qcow2-cluster.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 669, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=21da7e0fdd084eea8b00971953a3ff2a', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-yoa6xyt4/src/docker-src.2020-07-10-13.26.25.25621:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=21da7e0fdd084eea8b00971953a3ff2a
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-yoa6xyt4/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m27.189s
user    0m8.691s


The full log is available at
http://patchew.org/logs/cover.1594396418.git.berto@igalia.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH v11 00/34] Add subcluster allocation to qcow2
Posted by Max Reitz 3 years, 9 months ago
On 10.07.20 18:12, Alberto Garcia wrote:
> Hi,
> 
> here's the new version of the patches to add subcluster allocation
> support to qcow2.
> 
> Please refer to the cover letter of the first version for a full
> description of the patches:
> 
>    https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html
> 
> This version is rebased on top of the latest master (f2a1cf9180),
> fixes the relevant conflicts (particularly after df373fb0a3) and
> updates the test expectations.
> 
> Berto

Thanks, fixed up patch 21 (by initializing expected_type to just
anything, I chose QCOW2_SUBCLUSTER_NORMAL, which I preferred over just
using 0), resolved some iotest conflicts (I’m sure, more are to come
before the release, but we’ll see), and applied the series to my
block-next branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block-next

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2
Posted by Alberto Garcia 3 years, 9 months ago
On Tue, Jul 14, 2020 at 02:56:57PM +0200, Max Reitz wrote:
> Thanks, fixed up patch 21 (by initializing expected_type to just
> anything, I chose QCOW2_SUBCLUSTER_NORMAL, which I preferred over
> just using 0), resolved some iotest conflicts (I’m sure, more are
> to come before the release, but we’ll see), and applied the series
> to my block-next branch:
> 
> https://git.xanclic.moe/XanClic/qemu/commits/branch/block-next

I just realized that there's also a couple of 'since: 5.1' in
block-core.json that need to be updated. Can you do that please?

Thanks!

Berto

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2
Posted by Max Reitz 3 years, 9 months ago
On 15.07.20 13:51, Alberto Garcia wrote:
> On Tue, Jul 14, 2020 at 02:56:57PM +0200, Max Reitz wrote:
>> Thanks, fixed up patch 21 (by initializing expected_type to just
>> anything, I chose QCOW2_SUBCLUSTER_NORMAL, which I preferred over
>> just using 0), resolved some iotest conflicts (I’m sure, more are
>> to come before the release, but we’ll see), and applied the series
>> to my block-next branch:
>>
>> https://git.xanclic.moe/XanClic/qemu/commits/branch/block-next
> 
> I just realized that there's also a couple of 'since: 5.1' in
> block-core.json that need to be updated. Can you do that please?

Ah, sure.

Max