1. Patchew
  2. QEMU
  3. [PATCH 00/14] iotests: support zstd
  4. View patch

[PATCH 11/14] iotests: bash tests: filter compression type

Vladimir Sementsov-Ogievskiy posted 14 patches 4 years, 7 months ago
Maintainers: Max Reitz <mreitz@redhat.com>, Kevin Wolf <kwolf@redhat.com>
There is a newer version of this series
[PATCH 11/14] iotests: bash tests: filter compression type
Posted by Vladimir Sementsov-Ogievskiy 4 years, 7 months ago 
We want iotests pass with both the default zlib compression and with
IMGOPTS='compression_type=zstd'.

Actually the only test that is interested in real compression type in
test output is 287 (test for qcow2 compression type), so implement
specific option for it.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/060.out       |  2 +-
 tests/qemu-iotests/061.out       | 12 ++++++------
 tests/qemu-iotests/082.out       | 14 +++++++-------
 tests/qemu-iotests/198.out       |  4 ++--
 tests/qemu-iotests/287           |  8 ++++----
 tests/qemu-iotests/common.filter |  7 +++++++
 tests/qemu-iotests/common.rc     | 14 +++++++++++++-
 7 files changed, 40 insertions(+), 21 deletions(-)

diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
index b74540bafb..329977d9b9 100644
--- a/tests/qemu-iotests/060.out
+++ b/tests/qemu-iotests/060.out
@@ -17,7 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     corrupt: true
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index ee30da2665..11b6404186 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -524,7 +524,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -551,7 +551,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     data file: foo
@@ -566,7 +566,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     data file raw: false
@@ -582,7 +582,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -596,7 +596,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -611,7 +611,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index b70c12c139..f8e2e039fc 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -17,7 +17,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 4096
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -31,7 +31,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 8192
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -329,7 +329,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 4096
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -342,7 +342,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 8192
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -639,7 +639,7 @@ virtual size: 128 MiB (134217728 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
@@ -652,7 +652,7 @@ virtual size: 130 MiB (136314880 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: false
     refcount bits: 16
     corrupt: false
@@ -665,7 +665,7 @@ virtual size: 132 MiB (138412032 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     lazy refcounts: true
     refcount bits: 16
     corrupt: false
diff --git a/tests/qemu-iotests/198.out b/tests/qemu-iotests/198.out
index 3952708444..805494916f 100644
--- a/tests/qemu-iotests/198.out
+++ b/tests/qemu-iotests/198.out
@@ -36,7 +36,7 @@ image: json:{ /* filtered */ }
 file format: IMGFMT
 virtual size: 16 MiB (16777216 bytes)
 Format specific information:
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     encrypt:
         ivgen alg: plain64
         hash alg: sha256
@@ -81,7 +81,7 @@ virtual size: 16 MiB (16777216 bytes)
 backing file: TEST_DIR/t.IMGFMT.base
 backing file format: IMGFMT
 Format specific information:
-    compression type: zlib
+    compression type: COMPRESSION_TYPE
     encrypt:
         ivgen alg: plain64
         hash alg: sha256
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
index 6716419da4..aab03fb973 100755
--- a/tests/qemu-iotests/287
+++ b/tests/qemu-iotests/287
@@ -61,13 +61,13 @@ echo
 echo "=== Testing compression type incompatible bit setting for zlib ==="
 echo
 _make_test_img -o compression_type=zlib 64M
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
 
 echo
 echo "=== Testing compression type incompatible bit setting for zstd ==="
 echo
 _make_test_img -o compression_type=zstd 64M
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
 
 echo
 echo "=== Testing zlib with incompatible bit set ==="
@@ -75,7 +75,7 @@ echo
 _make_test_img -o compression_type=zlib 64M
 $PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
 # to make sure the bit was actually set
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
 
 if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
     echo "Error: The image opened successfully. The image must not be opened."
@@ -87,7 +87,7 @@ echo
 _make_test_img -o compression_type=zstd 64M
 $PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
 # to make sure the bit was actually unset
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
 
 if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
     echo "Error: The image opened successfully. The image must not be opened."
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 268b749e2f..78efe3e4dd 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -247,6 +247,7 @@ _filter_img_info()
         -e "/block_state_zero: \\(on\\|off\\)/d" \
         -e "/log_size: [0-9]\\+/d" \
         -e "s/iters: [0-9]\\+/iters: 1024/" \
+        -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
         -e "s/uuid: [-a-f0-9]\\+/uuid: 00000000-0000-0000-0000-000000000000/" | \
     while IFS='' read -r line; do
         if [[ $format_specific == 1 ]]; then
@@ -332,5 +333,11 @@ for fname in fnames:
 sys.stdout.write(result)'
 }
 
+_filter_qcow2_compression_type_bit()
+{
+    $SED -e 's/\(incompatible_features\s\+\)\[3\(, \)\?/\1[/' \
+         -e 's/\(incompatible_features.*\), 3\]/\1]/'
+}
+
 # make sure this script returns success
 true
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index ee4b9d795e..813b51ee03 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -697,6 +697,7 @@ _img_info()
             -e "s#$TEST_DIR#TEST_DIR#g" \
             -e "s#$SOCK_DIR/fuse-#TEST_DIR/#g" \
             -e "s#$IMGFMT#IMGFMT#g" \
+            -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
             -e "/^disk size:/ D" \
             -e "/actual-size/ D" | \
         while IFS='' read -r line; do
@@ -996,12 +997,23 @@ _require_one_device_of()
 
 _qcow2_dump_header()
 {
+    if [[ "$1" == "--no-filter-compression" ]]; then
+        local filter_compression=0
+        shift
+    else
+        local filter_compression=1
+    fi
+
     img="$1"
     if [ -z "$img" ]; then
         img="$TEST_IMG"
     fi
 
-    $PYTHON qcow2.py "$img" dump-header
+    if [[ $filter_compression == 0 ]]; then
+        $PYTHON qcow2.py "$img" dump-header
+    else
+        $PYTHON qcow2.py "$img" dump-header | _filter_qcow2_compression_type_bit
+    fi
 }
 
 # make sure this script returns success
-- 
2.29.2
Re: [PATCH 11/14] iotests: bash tests: filter compression type
Posted by Max Reitz 4 years, 6 months ago 
On 05.07.21 11:15, Vladimir Sementsov-Ogievskiy wrote:
> We want iotests pass with both the default zlib compression and with
> IMGOPTS='compression_type=zstd'.
>
> Actually the only test that is interested in real compression type in
> test output is 287 (test for qcow2 compression type), so implement
> specific option for it.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>   tests/qemu-iotests/060.out       |  2 +-
>   tests/qemu-iotests/061.out       | 12 ++++++------
>   tests/qemu-iotests/082.out       | 14 +++++++-------
>   tests/qemu-iotests/198.out       |  4 ++--
>   tests/qemu-iotests/287           |  8 ++++----
>   tests/qemu-iotests/common.filter |  7 +++++++
>   tests/qemu-iotests/common.rc     | 14 +++++++++++++-
>   7 files changed, 40 insertions(+), 21 deletions(-)

[...]

> diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
> index 268b749e2f..78efe3e4dd 100644
> --- a/tests/qemu-iotests/common.filter
> +++ b/tests/qemu-iotests/common.filter
> @@ -247,6 +247,7 @@ _filter_img_info()
>           -e "/block_state_zero: \\(on\\|off\\)/d" \
>           -e "/log_size: [0-9]\\+/d" \
>           -e "s/iters: [0-9]\\+/iters: 1024/" \
> +        -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
>           -e "s/uuid: [-a-f0-9]\\+/uuid: 00000000-0000-0000-0000-000000000000/" | \
>       while IFS='' read -r line; do
>           if [[ $format_specific == 1 ]]; then
> @@ -332,5 +333,11 @@ for fname in fnames:
>   sys.stdout.write(result)'
>   }
>   
> +_filter_qcow2_compression_type_bit()
> +{
> +    $SED -e 's/\(incompatible_features\s\+\)\[3\(, \)\?/\1[/' \
> +         -e 's/\(incompatible_features.*\), 3\]/\1]/'

What about “incompatble_features   [2, 3, 4]”?

I’d like to propose adding some form of filtering parameter to qcow2.py 
which allows filtering a specific bit from the qcow2_format.Flags64 
representation, but that seems rather difficult, actually...

> +}
> +
>   # make sure this script returns success
>   true
> diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
> index ee4b9d795e..813b51ee03 100644
> --- a/tests/qemu-iotests/common.rc
> +++ b/tests/qemu-iotests/common.rc
> @@ -697,6 +697,7 @@ _img_info()
>               -e "s#$TEST_DIR#TEST_DIR#g" \
>               -e "s#$SOCK_DIR/fuse-#TEST_DIR/#g" \
>               -e "s#$IMGFMT#IMGFMT#g" \
> +            -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
>               -e "/^disk size:/ D" \
>               -e "/actual-size/ D" | \
>           while IFS='' read -r line; do
> @@ -996,12 +997,23 @@ _require_one_device_of()
>   
>   _qcow2_dump_header()
>   {
> +    if [[ "$1" == "--no-filter-compression" ]]; then
> +        local filter_compression=0
> +        shift
> +    else
> +        local filter_compression=1
> +    fi
> +
>       img="$1"
>       if [ -z "$img" ]; then
>           img="$TEST_IMG"
>       fi
>   
> -    $PYTHON qcow2.py "$img" dump-header
> +    if [[ $filter_compression == 0 ]]; then
> +        $PYTHON qcow2.py "$img" dump-header
> +    else
> +        $PYTHON qcow2.py "$img" dump-header | _filter_qcow2_compression_type_bit
> +    fi
>   }
>   
>   # make sure this script returns success

Could have been done more extensibly for the future (i.e. a loop over 
the parameters, and a variable to invoke all applicable filters), but, 
well.  Not much reason to think about a future that we’re not sure will 
ever happen.

Max
Re: [PATCH 11/14] iotests: bash tests: filter compression type
Posted by Vladimir Sementsov-Ogievskiy 4 years, 6 months ago 
16.07.2021 16:17, Max Reitz wrote:
> On 05.07.21 11:15, Vladimir Sementsov-Ogievskiy wrote:
>> We want iotests pass with both the default zlib compression and with
>> IMGOPTS='compression_type=zstd'.
>>
>> Actually the only test that is interested in real compression type in
>> test output is 287 (test for qcow2 compression type), so implement
>> specific option for it.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>>   tests/qemu-iotests/060.out       |  2 +-
>>   tests/qemu-iotests/061.out       | 12 ++++++------
>>   tests/qemu-iotests/082.out       | 14 +++++++-------
>>   tests/qemu-iotests/198.out       |  4 ++--
>>   tests/qemu-iotests/287           |  8 ++++----
>>   tests/qemu-iotests/common.filter |  7 +++++++
>>   tests/qemu-iotests/common.rc     | 14 +++++++++++++-
>>   7 files changed, 40 insertions(+), 21 deletions(-)
> 
> [...]
> 
>> diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
>> index 268b749e2f..78efe3e4dd 100644
>> --- a/tests/qemu-iotests/common.filter
>> +++ b/tests/qemu-iotests/common.filter
>> @@ -247,6 +247,7 @@ _filter_img_info()
>>           -e "/block_state_zero: \\(on\\|off\\)/d" \
>>           -e "/log_size: [0-9]\\+/d" \
>>           -e "s/iters: [0-9]\\+/iters: 1024/" \
>> +        -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
>>           -e "s/uuid: [-a-f0-9]\\+/uuid: 00000000-0000-0000-0000-000000000000/" | \
>>       while IFS='' read -r line; do
>>           if [[ $format_specific == 1 ]]; then
>> @@ -332,5 +333,11 @@ for fname in fnames:
>>   sys.stdout.write(result)'
>>   }
>> +_filter_qcow2_compression_type_bit()
>> +{
>> +    $SED -e 's/\(incompatible_features\s\+\)\[3\(, \)\?/\1[/' \
>> +         -e 's/\(incompatible_features.*\), 3\]/\1]/'
> 
> What about “incompatble_features   [2, 3, 4]”?

Will add.

> 
> I’d like to propose adding some form of filtering parameter to qcow2.py which allows filtering a specific bit from the qcow2_format.Flags64 representation, but that seems rather difficult, actually...
> 
>> +}
>> +
>>   # make sure this script returns success
>>   true
>> diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
>> index ee4b9d795e..813b51ee03 100644
>> --- a/tests/qemu-iotests/common.rc
>> +++ b/tests/qemu-iotests/common.rc
>> @@ -697,6 +697,7 @@ _img_info()
>>               -e "s#$TEST_DIR#TEST_DIR#g" \
>>               -e "s#$SOCK_DIR/fuse-#TEST_DIR/#g" \
>>               -e "s#$IMGFMT#IMGFMT#g" \
>> +            -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
>>               -e "/^disk size:/ D" \
>>               -e "/actual-size/ D" | \
>>           while IFS='' read -r line; do
>> @@ -996,12 +997,23 @@ _require_one_device_of()
>>   _qcow2_dump_header()
>>   {
>> +    if [[ "$1" == "--no-filter-compression" ]]; then
>> +        local filter_compression=0
>> +        shift
>> +    else
>> +        local filter_compression=1
>> +    fi
>> +
>>       img="$1"
>>       if [ -z "$img" ]; then
>>           img="$TEST_IMG"
>>       fi
>> -    $PYTHON qcow2.py "$img" dump-header
>> +    if [[ $filter_compression == 0 ]]; then
>> +        $PYTHON qcow2.py "$img" dump-header
>> +    else
>> +        $PYTHON qcow2.py "$img" dump-header | _filter_qcow2_compression_type_bit
>> +    fi
>>   }
>>   # make sure this script returns success
> 
> Could have been done more extensibly for the future (i.e. a loop over the parameters, and a variable to invoke all applicable filters), but, well.  Not much reason to think about a future that we’re not sure will ever happen.
> 
> Max
> 


-- 
Best regards,
Vladimir

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.