These series makes tests pass with

   IMGOPTS='compression_type=zstd'

Also, python iotests start to support IMGOPTS (they didn't before).

v3:
02-04,06,08,14,17: add Hanna's r-b
07  iotests.py: filter out successful output of qemu-img create
      fix subject
      handle 149, 237 and 296 iotests
         (note, 149 is handled intuitively, as it fails :(

Vladimir Sementsov-Ogievskiy (17):
  iotests.py: img_info_log(): rename imgopts argument
  iotests.py: qemu_img*("create"): support
    IMGOPTS='compression_type=zstd'
  iotests: drop qemu_img_verbose() helper
  iotests.py: rewrite default luks support in qemu_img
  iotest 303: explicit compression type
  iotest 065: explicit compression type
  iotests.py: filter out successful output of qemu-img create
  iotests.py: filter compression type out
  iotest 302: use img_info_log() helper
  qcow2: simple case support for downgrading of qcow2 images with zstd
  iotests/common.rc: introduce _qcow2_dump_header helper
  iotests: massive use _qcow2_dump_header
  iotest 39: use _qcow2_dump_header
  iotests: bash tests: filter compression type
  iotests 60: more accurate set dirty bit in qcow2 header
  iotest 214: explicit compression type
  iotests: declare lack of support for compresion_type in IMGOPTS

 block/qcow2.c                    | 58 +++++++++++++++++++++-
 tests/qemu-iotests/031           | 11 +++--
 tests/qemu-iotests/036           |  6 +--
 tests/qemu-iotests/039           | 22 ++++-----
 tests/qemu-iotests/044           |  5 +-
 tests/qemu-iotests/044.out       |  1 +
 tests/qemu-iotests/051           |  5 +-
 tests/qemu-iotests/060           | 22 ++++-----
 tests/qemu-iotests/060.out       |  2 +-
 tests/qemu-iotests/061           | 42 ++++++++--------
 tests/qemu-iotests/061.out       | 12 ++---
 tests/qemu-iotests/065           | 16 +++---
 tests/qemu-iotests/082.out       | 14 +++---
 tests/qemu-iotests/112           |  3 +-
 tests/qemu-iotests/137           |  2 +-
 tests/qemu-iotests/149.out       | 21 --------
 tests/qemu-iotests/198.out       |  4 +-
 tests/qemu-iotests/206.out       | 10 ++--
 tests/qemu-iotests/209           |  7 +--
 tests/qemu-iotests/209.out       |  2 +
 tests/qemu-iotests/210           |  8 +--
 tests/qemu-iotests/214           |  2 +-
 tests/qemu-iotests/237.out       |  3 --
 tests/qemu-iotests/242.out       | 10 ++--
 tests/qemu-iotests/255.out       |  4 --
 tests/qemu-iotests/274.out       | 39 ++-------------
 tests/qemu-iotests/280.out       |  1 -
 tests/qemu-iotests/287           |  8 +--
 tests/qemu-iotests/290           |  2 +-
 tests/qemu-iotests/296.out       | 10 ++--
 tests/qemu-iotests/302           |  4 +-
 tests/qemu-iotests/302.out       |  7 ++-
 tests/qemu-iotests/303           | 25 ++++++----
 tests/qemu-iotests/303.out       | 30 +++++++++++-
 tests/qemu-iotests/common.filter |  8 +++
 tests/qemu-iotests/common.rc     | 22 +++++++++
 tests/qemu-iotests/iotests.py    | 84 ++++++++++++++++++++------------
 37 files changed, 313 insertions(+), 219 deletions(-)

-- 
2.29.2

On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
> These series makes tests pass with
>
>     IMGOPTS='compression_type=zstd'
>
> Also, python iotests start to support IMGOPTS (they didn't before).
>
> v3:
> 02-04,06,08,14,17: add Hanna's r-b
> 07  iotests.py: filter out successful output of qemu-img create
>        fix subject
>        handle 149, 237 and 296 iotests
>           (note, 149 is handled intuitively, as it fails :(

It was also reviewed intuitively. :)

Thanks, applied to my block branch:

https://github.com/XanClic/qemu/commits/block

Hanna

9/14/21 20:08, Hanna Reitz wrote:
> On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
>> These series makes tests pass with
>>
>>     IMGOPTS='compression_type=zstd'
>>
>> Also, python iotests start to support IMGOPTS (they didn't before).
>>
>> v3:
>> 02-04,06,08,14,17: add Hanna's r-b
>> 07  iotests.py: filter out successful output of qemu-img create
>>        fix subject
>>        handle 149, 237 and 296 iotests
>>           (note, 149 is handled intuitively, as it fails :(
> 
> It was also reviewed intuitively. :)
> 
> Thanks, applied to my block branch:
> 
> https://github.com/XanClic/qemu/commits/block
> 

Hmm, patches are not here neither in master. Aren't they missed somehow?



-- 
Best regards,
Vladimir

On 07.10.21 18:29, Vladimir Sementsov-Ogievskiy wrote:
> 9/14/21 20:08, Hanna Reitz wrote:
>> On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
>>> These series makes tests pass with
>>>
>>>     IMGOPTS='compression_type=zstd'
>>>
>>> Also, python iotests start to support IMGOPTS (they didn't before).
>>>
>>> v3:
>>> 02-04,06,08,14,17: add Hanna's r-b
>>> 07  iotests.py: filter out successful output of qemu-img create
>>>        fix subject
>>>        handle 149, 237 and 296 iotests
>>>           (note, 149 is handled intuitively, as it fails :(
>>
>> It was also reviewed intuitively. :)
>>
>> Thanks, applied to my block branch:
>>
>> https://github.com/XanClic/qemu/commits/block
>>
>
> Hmm, patches are not here neither in master. Aren't they missed somehow?

Yes, I had to remove them because they broke other test configurations 
(like `-o refcount_bits=1`): 
https://lists.nongnu.org/archive/html/qemu-block/2021-09/msg00692.html

Hanna

10/8/21 09:39, Hanna Reitz wrote:
> On 07.10.21 18:29, Vladimir Sementsov-Ogievskiy wrote:
>> 9/14/21 20:08, Hanna Reitz wrote:
>>> On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
>>>> These series makes tests pass with
>>>>
>>>>     IMGOPTS='compression_type=zstd'
>>>>
>>>> Also, python iotests start to support IMGOPTS (they didn't before).
>>>>
>>>> v3:
>>>> 02-04,06,08,14,17: add Hanna's r-b
>>>> 07  iotests.py: filter out successful output of qemu-img create
>>>>        fix subject
>>>>        handle 149, 237 and 296 iotests
>>>>           (note, 149 is handled intuitively, as it fails :(
>>>
>>> It was also reviewed intuitively. :)
>>>
>>> Thanks, applied to my block branch:
>>>
>>> https://github.com/XanClic/qemu/commits/block
>>>
>>
>> Hmm, patches are not here neither in master. Aren't they missed somehow?
> 
> Yes, I had to remove them because they broke other test configurations (like `-o refcount_bits=1`): https://lists.nongnu.org/archive/html/qemu-block/2021-09/msg00692.html
> 

Ah, yes, right, sorry. I forget and somehow lost these emails from my thunderberd.


-- 
Best regards,
Vladimir

On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
> These series makes tests pass with
>
>     IMGOPTS='compression_type=zstd'
>
> Also, python iotests start to support IMGOPTS (they didn't before).

Unfortunately, the problem I have now is that it makes the tests fail 
with other IMGOPTS.

My regular test set includes 'refcount_bits=1', 'compat=0.10', and 
'data_file=$TEST_IMG.ext_data_file'.  These fail now, because the Python 
tests don’t have a way to specify which test options they don’t support 
(like _unsupported_imgopts).  Handling data_file of course is extra 
tricky because now every disk image consists of two files, and the 
qemu-img create invocation needs to expand '$TEST_IMG', like 
_make_test_img does (in the line where imgopts_expanded is set).

I think we need an unsupported_imgopts parameter for Python tests, and 
it needs to be filled (perhaps 
https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg00082.html 
can serve as a starting point).  And for the time being, I think Python 
tests should be skipped altogether when IMGOPTS contains data_file.

Hanna

On 15.09.21 16:45, Hanna Reitz wrote:
> On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
>> These series makes tests pass with
>>
>>     IMGOPTS='compression_type=zstd'
>>
>> Also, python iotests start to support IMGOPTS (they didn't before).
>
> Unfortunately, the problem I have now is that it makes the tests fail 
> with other IMGOPTS.
>
> My regular test set includes 'refcount_bits=1', 'compat=0.10', and 
> 'data_file=$TEST_IMG.ext_data_file'.  These fail now, because the 
> Python tests don’t have a way to specify which test options they don’t 
> support (like _unsupported_imgopts).  Handling data_file of course is 
> extra tricky because now every disk image consists of two files, and 
> the qemu-img create invocation needs to expand '$TEST_IMG', like 
> _make_test_img does (in the line where imgopts_expanded is set).
>
> I think we need an unsupported_imgopts parameter for Python tests, and 
> it needs to be filled (perhaps 
> https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg00082.html 
> can serve as a starting point).  And for the time being, I think 
> Python tests should be skipped altogether when IMGOPTS contains data_file.

(Perhaps I should explicitly say that this means I didn’t include this 
series in my pull request this week, because, well, my pre-pull tests 
were failing :/)

(Also wanted to let you know I’ll be on PTO the next two weeks, so I 
won’t be able to review a v4 or talk about how exactly we want to handle 
IMGOPTS other than compression_type until Oct 4.)

Hanna

17.09.2021 17:49, Hanna Reitz wrote:
> On 15.09.21 16:45, Hanna Reitz wrote:
>> On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
>>> These series makes tests pass with
>>>
>>>     IMGOPTS='compression_type=zstd'
>>>
>>> Also, python iotests start to support IMGOPTS (they didn't before).
>>
>> Unfortunately, the problem I have now is that it makes the tests fail with other IMGOPTS.
>>
>> My regular test set includes 'refcount_bits=1', 'compat=0.10', and 'data_file=$TEST_IMG.ext_data_file'.  These fail now, because the Python tests don’t have a way to specify which test options they don’t support (like _unsupported_imgopts).  Handling data_file of course is extra tricky because now every disk image consists of two files, and the qemu-img create invocation needs to expand '$TEST_IMG', like _make_test_img does (in the line where imgopts_expanded is set).
>>
>> I think we need an unsupported_imgopts parameter for Python tests, and it needs to be filled (perhaps https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg00082.html can serve as a starting point).  And for the time being, I think Python tests should be skipped altogether when IMGOPTS contains data_file.
> 
> (Perhaps I should explicitly say that this means I didn’t include this series in my pull request this week, because, well, my pre-pull tests were failing :/)

That was clean :) I'll resend

> 
> (Also wanted to let you know I’ll be on PTO the next two weeks, so I won’t be able to review a v4 or talk about how exactly we want to handle IMGOPTS other than compression_type until Oct 4.)
> 

-- 
Best regards,
Vladimir