[PATCH v3 00/17] iotests: support zstd

Vladimir Sementsov-Ogievskiy posted 17 patches 1 week, 3 days ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20210914102547.83963-1-vsementsov@virtuozzo.com
Maintainers: Hanna Reitz <hreitz@redhat.com>, Kevin Wolf <kwolf@redhat.com>
block/qcow2.c                    | 58 +++++++++++++++++++++-
tests/qemu-iotests/031           | 11 +++--
tests/qemu-iotests/036           |  6 +--
tests/qemu-iotests/039           | 22 ++++-----
tests/qemu-iotests/044           |  5 +-
tests/qemu-iotests/044.out       |  1 +
tests/qemu-iotests/051           |  5 +-
tests/qemu-iotests/060           | 22 ++++-----
tests/qemu-iotests/060.out       |  2 +-
tests/qemu-iotests/061           | 42 ++++++++--------
tests/qemu-iotests/061.out       | 12 ++---
tests/qemu-iotests/065           | 16 +++---
tests/qemu-iotests/082.out       | 14 +++---
tests/qemu-iotests/112           |  3 +-
tests/qemu-iotests/137           |  2 +-
tests/qemu-iotests/149.out       | 21 --------
tests/qemu-iotests/198.out       |  4 +-
tests/qemu-iotests/206.out       | 10 ++--
tests/qemu-iotests/209           |  7 +--
tests/qemu-iotests/209.out       |  2 +
tests/qemu-iotests/210           |  8 +--
tests/qemu-iotests/214           |  2 +-
tests/qemu-iotests/237.out       |  3 --
tests/qemu-iotests/242.out       | 10 ++--
tests/qemu-iotests/255.out       |  4 --
tests/qemu-iotests/274.out       | 39 ++-------------
tests/qemu-iotests/280.out       |  1 -
tests/qemu-iotests/287           |  8 +--
tests/qemu-iotests/290           |  2 +-
tests/qemu-iotests/296.out       | 10 ++--
tests/qemu-iotests/302           |  4 +-
tests/qemu-iotests/302.out       |  7 ++-
tests/qemu-iotests/303           | 25 ++++++----
tests/qemu-iotests/303.out       | 30 +++++++++++-
tests/qemu-iotests/common.filter |  8 +++
tests/qemu-iotests/common.rc     | 22 +++++++++
tests/qemu-iotests/iotests.py    | 84 ++++++++++++++++++++------------
37 files changed, 313 insertions(+), 219 deletions(-)

[PATCH v3 00/17] iotests: support zstd

Posted by Vladimir Sementsov-Ogievskiy 1 week, 3 days ago
These series makes tests pass with

   IMGOPTS='compression_type=zstd'

Also, python iotests start to support IMGOPTS (they didn't before).

v3:
02-04,06,08,14,17: add Hanna's r-b
07  iotests.py: filter out successful output of qemu-img create
      fix subject
      handle 149, 237 and 296 iotests
         (note, 149 is handled intuitively, as it fails :(

Vladimir Sementsov-Ogievskiy (17):
  iotests.py: img_info_log(): rename imgopts argument
  iotests.py: qemu_img*("create"): support
    IMGOPTS='compression_type=zstd'
  iotests: drop qemu_img_verbose() helper
  iotests.py: rewrite default luks support in qemu_img
  iotest 303: explicit compression type
  iotest 065: explicit compression type
  iotests.py: filter out successful output of qemu-img create
  iotests.py: filter compression type out
  iotest 302: use img_info_log() helper
  qcow2: simple case support for downgrading of qcow2 images with zstd
  iotests/common.rc: introduce _qcow2_dump_header helper
  iotests: massive use _qcow2_dump_header
  iotest 39: use _qcow2_dump_header
  iotests: bash tests: filter compression type
  iotests 60: more accurate set dirty bit in qcow2 header
  iotest 214: explicit compression type
  iotests: declare lack of support for compresion_type in IMGOPTS

 block/qcow2.c                    | 58 +++++++++++++++++++++-
 tests/qemu-iotests/031           | 11 +++--
 tests/qemu-iotests/036           |  6 +--
 tests/qemu-iotests/039           | 22 ++++-----
 tests/qemu-iotests/044           |  5 +-
 tests/qemu-iotests/044.out       |  1 +
 tests/qemu-iotests/051           |  5 +-
 tests/qemu-iotests/060           | 22 ++++-----
 tests/qemu-iotests/060.out       |  2 +-
 tests/qemu-iotests/061           | 42 ++++++++--------
 tests/qemu-iotests/061.out       | 12 ++---
 tests/qemu-iotests/065           | 16 +++---
 tests/qemu-iotests/082.out       | 14 +++---
 tests/qemu-iotests/112           |  3 +-
 tests/qemu-iotests/137           |  2 +-
 tests/qemu-iotests/149.out       | 21 --------
 tests/qemu-iotests/198.out       |  4 +-
 tests/qemu-iotests/206.out       | 10 ++--
 tests/qemu-iotests/209           |  7 +--
 tests/qemu-iotests/209.out       |  2 +
 tests/qemu-iotests/210           |  8 +--
 tests/qemu-iotests/214           |  2 +-
 tests/qemu-iotests/237.out       |  3 --
 tests/qemu-iotests/242.out       | 10 ++--
 tests/qemu-iotests/255.out       |  4 --
 tests/qemu-iotests/274.out       | 39 ++-------------
 tests/qemu-iotests/280.out       |  1 -
 tests/qemu-iotests/287           |  8 +--
 tests/qemu-iotests/290           |  2 +-
 tests/qemu-iotests/296.out       | 10 ++--
 tests/qemu-iotests/302           |  4 +-
 tests/qemu-iotests/302.out       |  7 ++-
 tests/qemu-iotests/303           | 25 ++++++----
 tests/qemu-iotests/303.out       | 30 +++++++++++-
 tests/qemu-iotests/common.filter |  8 +++
 tests/qemu-iotests/common.rc     | 22 +++++++++
 tests/qemu-iotests/iotests.py    | 84 ++++++++++++++++++++------------
 37 files changed, 313 insertions(+), 219 deletions(-)

-- 
2.29.2


Re: [PATCH v3 00/17] iotests: support zstd

Posted by Hanna Reitz 1 week, 3 days ago
On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
> These series makes tests pass with
>
>     IMGOPTS='compression_type=zstd'
>
> Also, python iotests start to support IMGOPTS (they didn't before).
>
> v3:
> 02-04,06,08,14,17: add Hanna's r-b
> 07  iotests.py: filter out successful output of qemu-img create
>        fix subject
>        handle 149, 237 and 296 iotests
>           (note, 149 is handled intuitively, as it fails :(

It was also reviewed intuitively. :)

Thanks, applied to my block branch:

https://github.com/XanClic/qemu/commits/block

Hanna


Re: [PATCH v3 00/17] iotests: support zstd

Posted by Hanna Reitz 1 week, 2 days ago
On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
> These series makes tests pass with
>
>     IMGOPTS='compression_type=zstd'
>
> Also, python iotests start to support IMGOPTS (they didn't before).

Unfortunately, the problem I have now is that it makes the tests fail 
with other IMGOPTS.

My regular test set includes 'refcount_bits=1', 'compat=0.10', and 
'data_file=$TEST_IMG.ext_data_file'.  These fail now, because the Python 
tests don’t have a way to specify which test options they don’t support 
(like _unsupported_imgopts).  Handling data_file of course is extra 
tricky because now every disk image consists of two files, and the 
qemu-img create invocation needs to expand '$TEST_IMG', like 
_make_test_img does (in the line where imgopts_expanded is set).

I think we need an unsupported_imgopts parameter for Python tests, and 
it needs to be filled (perhaps 
https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg00082.html 
can serve as a starting point).  And for the time being, I think Python 
tests should be skipped altogether when IMGOPTS contains data_file.

Hanna


Re: [PATCH v3 00/17] iotests: support zstd

Posted by Hanna Reitz 1 week ago
On 15.09.21 16:45, Hanna Reitz wrote:
> On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
>> These series makes tests pass with
>>
>>     IMGOPTS='compression_type=zstd'
>>
>> Also, python iotests start to support IMGOPTS (they didn't before).
>
> Unfortunately, the problem I have now is that it makes the tests fail 
> with other IMGOPTS.
>
> My regular test set includes 'refcount_bits=1', 'compat=0.10', and 
> 'data_file=$TEST_IMG.ext_data_file'.  These fail now, because the 
> Python tests don’t have a way to specify which test options they don’t 
> support (like _unsupported_imgopts).  Handling data_file of course is 
> extra tricky because now every disk image consists of two files, and 
> the qemu-img create invocation needs to expand '$TEST_IMG', like 
> _make_test_img does (in the line where imgopts_expanded is set).
>
> I think we need an unsupported_imgopts parameter for Python tests, and 
> it needs to be filled (perhaps 
> https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg00082.html 
> can serve as a starting point).  And for the time being, I think 
> Python tests should be skipped altogether when IMGOPTS contains data_file.

(Perhaps I should explicitly say that this means I didn’t include this 
series in my pull request this week, because, well, my pre-pull tests 
were failing :/)

(Also wanted to let you know I’ll be on PTO the next two weeks, so I 
won’t be able to review a v4 or talk about how exactly we want to handle 
IMGOPTS other than compression_type until Oct 4.)

Hanna


Re: [PATCH v3 00/17] iotests: support zstd

Posted by Vladimir Sementsov-Ogievskiy 1 week ago
17.09.2021 17:49, Hanna Reitz wrote:
> On 15.09.21 16:45, Hanna Reitz wrote:
>> On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
>>> These series makes tests pass with
>>>
>>>     IMGOPTS='compression_type=zstd'
>>>
>>> Also, python iotests start to support IMGOPTS (they didn't before).
>>
>> Unfortunately, the problem I have now is that it makes the tests fail with other IMGOPTS.
>>
>> My regular test set includes 'refcount_bits=1', 'compat=0.10', and 'data_file=$TEST_IMG.ext_data_file'.  These fail now, because the Python tests don’t have a way to specify which test options they don’t support (like _unsupported_imgopts).  Handling data_file of course is extra tricky because now every disk image consists of two files, and the qemu-img create invocation needs to expand '$TEST_IMG', like _make_test_img does (in the line where imgopts_expanded is set).
>>
>> I think we need an unsupported_imgopts parameter for Python tests, and it needs to be filled (perhaps https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg00082.html can serve as a starting point).  And for the time being, I think Python tests should be skipped altogether when IMGOPTS contains data_file.
> 
> (Perhaps I should explicitly say that this means I didn’t include this series in my pull request this week, because, well, my pre-pull tests were failing :/)

That was clean :) I'll resend

> 
> (Also wanted to let you know I’ll be on PTO the next two weeks, so I won’t be able to review a v4 or talk about how exactly we want to handle IMGOPTS other than compression_type until Oct 4.)
> 

-- 
Best regards,
Vladimir