Hi all!

When sending iotests to upstream or do patch porting from one branch
to another we very often have to resolve conflicts in group file, as
many absolutely independent features are intersecting by this file.
These conflicts are simple, but imagine how much time we all have
already spent on resolving them? Let's finally get rid of group file.

Next, another thing I don't like about iotests is race for test number
chosing: you should search through mail box, before chosing test number
for new test.

So, I propose to get rid of group file and search for tests another way
[look at patch 02]. Additionally I propose to move to human-readable
names for test files, with notation test-* .

v1 was one patch "[PATCH] iotests: drop group file"

Vladimir Sementsov-Ogievskiy (2):
  iotests: define group in each iotests
  iotests: rework test finding

 docs/devel/testing.rst           |  51 +++++-
 tests/qemu-iotests/001           |   1 +
 tests/qemu-iotests/002           |   1 +
 tests/qemu-iotests/003           |   1 +
 tests/qemu-iotests/004           |   1 +
 tests/qemu-iotests/005           |   1 +
 tests/qemu-iotests/007           |   1 +
 tests/qemu-iotests/008           |   1 +
 tests/qemu-iotests/009           |   1 +
 tests/qemu-iotests/010           |   1 +
 tests/qemu-iotests/011           |   1 +
 tests/qemu-iotests/012           |   1 +
 tests/qemu-iotests/013           |   1 +
 tests/qemu-iotests/014           |   1 +
 tests/qemu-iotests/015           |   1 +
 tests/qemu-iotests/017           |   1 +
 tests/qemu-iotests/018           |   1 +
 tests/qemu-iotests/019           |   1 +
 tests/qemu-iotests/020           |   1 +
 tests/qemu-iotests/021           |   1 +
 tests/qemu-iotests/022           |   1 +
 tests/qemu-iotests/023           |   1 +
 tests/qemu-iotests/024           |   1 +
 tests/qemu-iotests/025           |   1 +
 tests/qemu-iotests/026           |   1 +
 tests/qemu-iotests/027           |   1 +
 tests/qemu-iotests/028           |   1 +
 tests/qemu-iotests/029           |   1 +
 tests/qemu-iotests/030           |   1 +
 tests/qemu-iotests/031           |   1 +
 tests/qemu-iotests/032           |   1 +
 tests/qemu-iotests/033           |   1 +
 tests/qemu-iotests/034           |   1 +
 tests/qemu-iotests/035           |   1 +
 tests/qemu-iotests/036           |   1 +
 tests/qemu-iotests/037           |   1 +
 tests/qemu-iotests/038           |   1 +
 tests/qemu-iotests/039           |   1 +
 tests/qemu-iotests/040           |   1 +
 tests/qemu-iotests/041           |   1 +
 tests/qemu-iotests/042           |   1 +
 tests/qemu-iotests/043           |   1 +
 tests/qemu-iotests/044           |   1 +
 tests/qemu-iotests/045           |   1 +
 tests/qemu-iotests/046           |   1 +
 tests/qemu-iotests/047           |   1 +
 tests/qemu-iotests/048           |   1 +
 tests/qemu-iotests/049           |   1 +
 tests/qemu-iotests/050           |   1 +
 tests/qemu-iotests/051           |   1 +
 tests/qemu-iotests/052           |   1 +
 tests/qemu-iotests/053           |   1 +
 tests/qemu-iotests/054           |   1 +
 tests/qemu-iotests/055           |   1 +
 tests/qemu-iotests/056           |   1 +
 tests/qemu-iotests/057           |   1 +
 tests/qemu-iotests/058           |   1 +
 tests/qemu-iotests/059           |   1 +
 tests/qemu-iotests/060           |   1 +
 tests/qemu-iotests/061           |   1 +
 tests/qemu-iotests/062           |   1 +
 tests/qemu-iotests/063           |   1 +
 tests/qemu-iotests/064           |   1 +
 tests/qemu-iotests/065           |   1 +
 tests/qemu-iotests/066           |   1 +
 tests/qemu-iotests/067           |   1 +
 tests/qemu-iotests/068           |   1 +
 tests/qemu-iotests/069           |   1 +
 tests/qemu-iotests/070           |   1 +
 tests/qemu-iotests/071           |   1 +
 tests/qemu-iotests/072           |   1 +
 tests/qemu-iotests/073           |   1 +
 tests/qemu-iotests/074           |   1 +
 tests/qemu-iotests/075           |   1 +
 tests/qemu-iotests/076           |   1 +
 tests/qemu-iotests/077           |   1 +
 tests/qemu-iotests/078           |   1 +
 tests/qemu-iotests/079           |   1 +
 tests/qemu-iotests/080           |   1 +
 tests/qemu-iotests/081           |   1 +
 tests/qemu-iotests/082           |   1 +
 tests/qemu-iotests/083           |   1 +
 tests/qemu-iotests/084           |   1 +
 tests/qemu-iotests/085           |   1 +
 tests/qemu-iotests/086           |   1 +
 tests/qemu-iotests/087           |   1 +
 tests/qemu-iotests/088           |   1 +
 tests/qemu-iotests/089           |   1 +
 tests/qemu-iotests/090           |   1 +
 tests/qemu-iotests/091           |   1 +
 tests/qemu-iotests/092           |   1 +
 tests/qemu-iotests/093           |   1 +
 tests/qemu-iotests/094           |   1 +
 tests/qemu-iotests/095           |   1 +
 tests/qemu-iotests/096           |   1 +
 tests/qemu-iotests/097           |   1 +
 tests/qemu-iotests/098           |   1 +
 tests/qemu-iotests/099           |   1 +
 tests/qemu-iotests/101           |   1 +
 tests/qemu-iotests/102           |   1 +
 tests/qemu-iotests/103           |   1 +
 tests/qemu-iotests/104           |   1 +
 tests/qemu-iotests/105           |   1 +
 tests/qemu-iotests/106           |   1 +
 tests/qemu-iotests/107           |   1 +
 tests/qemu-iotests/108           |   1 +
 tests/qemu-iotests/109           |   1 +
 tests/qemu-iotests/110           |   1 +
 tests/qemu-iotests/111           |   1 +
 tests/qemu-iotests/112           |   1 +
 tests/qemu-iotests/113           |   1 +
 tests/qemu-iotests/114           |   1 +
 tests/qemu-iotests/115           |   1 +
 tests/qemu-iotests/116           |   1 +
 tests/qemu-iotests/117           |   1 +
 tests/qemu-iotests/118           |   1 +
 tests/qemu-iotests/119           |   1 +
 tests/qemu-iotests/120           |   1 +
 tests/qemu-iotests/121           |   1 +
 tests/qemu-iotests/122           |   1 +
 tests/qemu-iotests/123           |   1 +
 tests/qemu-iotests/124           |   1 +
 tests/qemu-iotests/125           |   1 +
 tests/qemu-iotests/126           |   1 +
 tests/qemu-iotests/127           |   1 +
 tests/qemu-iotests/128           |   1 +
 tests/qemu-iotests/129           |   1 +
 tests/qemu-iotests/130           |   1 +
 tests/qemu-iotests/131           |   1 +
 tests/qemu-iotests/132           |   1 +
 tests/qemu-iotests/133           |   1 +
 tests/qemu-iotests/134           |   1 +
 tests/qemu-iotests/135           |   1 +
 tests/qemu-iotests/136           |   1 +
 tests/qemu-iotests/137           |   1 +
 tests/qemu-iotests/138           |   1 +
 tests/qemu-iotests/139           |   1 +
 tests/qemu-iotests/140           |   1 +
 tests/qemu-iotests/141           |   1 +
 tests/qemu-iotests/143           |   1 +
 tests/qemu-iotests/144           |   1 +
 tests/qemu-iotests/145           |   1 +
 tests/qemu-iotests/146           |   1 +
 tests/qemu-iotests/147           |   1 +
 tests/qemu-iotests/148           |   1 +
 tests/qemu-iotests/149           |   1 +
 tests/qemu-iotests/150           |   1 +
 tests/qemu-iotests/151           |   1 +
 tests/qemu-iotests/152           |   1 +
 tests/qemu-iotests/153           |   1 +
 tests/qemu-iotests/154           |   1 +
 tests/qemu-iotests/155           |   1 +
 tests/qemu-iotests/156           |   1 +
 tests/qemu-iotests/157           |   1 +
 tests/qemu-iotests/158           |   1 +
 tests/qemu-iotests/159           |   1 +
 tests/qemu-iotests/160           |   1 +
 tests/qemu-iotests/161           |   1 +
 tests/qemu-iotests/162           |   1 +
 tests/qemu-iotests/163           |   1 +
 tests/qemu-iotests/165           |   1 +
 tests/qemu-iotests/169           |   1 +
 tests/qemu-iotests/170           |   1 +
 tests/qemu-iotests/171           |   1 +
 tests/qemu-iotests/172           |   1 +
 tests/qemu-iotests/173           |   1 +
 tests/qemu-iotests/174           |   1 +
 tests/qemu-iotests/175           |   1 +
 tests/qemu-iotests/176           |   1 +
 tests/qemu-iotests/177           |   1 +
 tests/qemu-iotests/178           |   1 +
 tests/qemu-iotests/179           |   1 +
 tests/qemu-iotests/181           |   1 +
 tests/qemu-iotests/182           |   1 +
 tests/qemu-iotests/183           |   1 +
 tests/qemu-iotests/184           |   1 +
 tests/qemu-iotests/185           |   1 +
 tests/qemu-iotests/186           |   1 +
 tests/qemu-iotests/187           |   1 +
 tests/qemu-iotests/188           |   1 +
 tests/qemu-iotests/189           |   1 +
 tests/qemu-iotests/190           |   1 +
 tests/qemu-iotests/191           |   1 +
 tests/qemu-iotests/192           |   1 +
 tests/qemu-iotests/194           |   1 +
 tests/qemu-iotests/195           |   1 +
 tests/qemu-iotests/196           |   1 +
 tests/qemu-iotests/197           |   1 +
 tests/qemu-iotests/198           |   1 +
 tests/qemu-iotests/199           |   1 +
 tests/qemu-iotests/200           |   1 +
 tests/qemu-iotests/201           |   1 +
 tests/qemu-iotests/202           |   1 +
 tests/qemu-iotests/203           |   1 +
 tests/qemu-iotests/204           |   1 +
 tests/qemu-iotests/205           |   1 +
 tests/qemu-iotests/206           |   1 +
 tests/qemu-iotests/207           |   1 +
 tests/qemu-iotests/208           |   1 +
 tests/qemu-iotests/209           |   1 +
 tests/qemu-iotests/210           |   1 +
 tests/qemu-iotests/211           |   1 +
 tests/qemu-iotests/212           |   1 +
 tests/qemu-iotests/213           |   1 +
 tests/qemu-iotests/214           |   1 +
 tests/qemu-iotests/215           |   1 +
 tests/qemu-iotests/216           |   1 +
 tests/qemu-iotests/217           |   1 +
 tests/qemu-iotests/218           |   1 +
 tests/qemu-iotests/219           |   1 +
 tests/qemu-iotests/220           |   1 +
 tests/qemu-iotests/221           |   1 +
 tests/qemu-iotests/222           |   1 +
 tests/qemu-iotests/223           |   1 +
 tests/qemu-iotests/224           |   1 +
 tests/qemu-iotests/225           |   1 +
 tests/qemu-iotests/226           |   1 +
 tests/qemu-iotests/227           |   1 +
 tests/qemu-iotests/228           |   1 +
 tests/qemu-iotests/229           |   1 +
 tests/qemu-iotests/231           |   1 +
 tests/qemu-iotests/232           |   1 +
 tests/qemu-iotests/233           |   1 +
 tests/qemu-iotests/234           |   1 +
 tests/qemu-iotests/235           |   1 +
 tests/qemu-iotests/236           |   1 +
 tests/qemu-iotests/237           |   1 +
 tests/qemu-iotests/238           |   1 +
 tests/qemu-iotests/239           |   1 +
 tests/qemu-iotests/240           |   1 +
 tests/qemu-iotests/241           |   1 +
 tests/qemu-iotests/242           |   1 +
 tests/qemu-iotests/243           |   1 +
 tests/qemu-iotests/244           |   1 +
 tests/qemu-iotests/245           |   1 +
 tests/qemu-iotests/246           |   1 +
 tests/qemu-iotests/247           |   1 +
 tests/qemu-iotests/248           |   1 +
 tests/qemu-iotests/249           |   1 +
 tests/qemu-iotests/250           |   1 +
 tests/qemu-iotests/251           |   1 +
 tests/qemu-iotests/252           |   1 +
 tests/qemu-iotests/253           |   1 +
 tests/qemu-iotests/254           |   1 +
 tests/qemu-iotests/255           |   1 +
 tests/qemu-iotests/256           |   1 +
 tests/qemu-iotests/257           |   1 +
 tests/qemu-iotests/258           |   1 +
 tests/qemu-iotests/259           |   1 +
 tests/qemu-iotests/260           |   1 +
 tests/qemu-iotests/261           |   1 +
 tests/qemu-iotests/262           |   1 +
 tests/qemu-iotests/263           |   1 +
 tests/qemu-iotests/264           |   1 +
 tests/qemu-iotests/265           |   1 +
 tests/qemu-iotests/266           |   1 +
 tests/qemu-iotests/267           |   1 +
 tests/qemu-iotests/268           |   1 +
 tests/qemu-iotests/270           |   1 +
 tests/qemu-iotests/272           |   1 +
 tests/qemu-iotests/273           |   1 +
 tests/qemu-iotests/277           |   1 +
 tests/qemu-iotests/279           |   1 +
 tests/qemu-iotests/280           |   1 +
 tests/qemu-iotests/281           |   1 +
 tests/qemu-iotests/282           |   1 +
 tests/qemu-iotests/283           |   1 +
 tests/qemu-iotests/284           |   1 +
 tests/qemu-iotests/286           |   1 +
 tests/qemu-iotests/288           |   1 +
 tests/qemu-iotests/289           |   1 +
 tests/qemu-iotests/check         |  20 +--
 tests/qemu-iotests/find_tests.py |  72 ++++++++
 tests/qemu-iotests/group         | 298 -------------------------------
 274 files changed, 402 insertions(+), 309 deletions(-)
 create mode 100755 tests/qemu-iotests/find_tests.py
 delete mode 100644 tests/qemu-iotests/group

-- 
2.21.0

Patchew URL: https://patchew.org/QEMU/20200325102131.23270-1-vsementsov@virtuozzo.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      s_eq128.o
./check: line 171: ./find_tests.py: No such file or directory
Group "auto" is empty or not defined?
make: *** [check-tests/check-block.sh] Error 1
make: *** Waiting for unfinished jobs....
  CC      s_le128.o
  CC      s_lt128.o
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=ea762c961bed46aea47cd60fda18c0de', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-nf_7yydp/src/docker-src.2020-03-25-07.53.52.20951:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=ea762c961bed46aea47cd60fda18c0de
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-nf_7yydp/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m47.178s
user    0m8.387s


The full log is available at
http://patchew.org/logs/20200325102131.23270-1-vsementsov@virtuozzo.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

25.03.2020 14:56, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20200325102131.23270-1-vsementsov@virtuozzo.com/
> 
> 
> 
> Hi,
> 
> This series failed the docker-quick@centos7 build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> make docker-image-centos7 V=1 NETWORK=1
> time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
> === TEST SCRIPT END ===
> 
>    CC      s_eq128.o
> ./check: line 171: ./find_tests.py: No such file or directory


Hmm but my patches add find_tests.py in same directory as check script. What's wrong with it, who can help?

For me, "make check" works and "cd tests/qemu-iotests; ./check -qcow2" works too.

> Group "auto" is empty or not defined?
> make: *** [check-tests/check-block.sh] Error 1
> make: *** Waiting for unfinished jobs....
>    CC      s_le128.o
>    CC      s_lt128.o
> ---
>      raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=ea762c961bed46aea47cd60fda18c0de', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-nf_7yydp/src/docker-src.2020-03-25-07.53.52.20951:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
> filter=--filter=label=com.qemu.instance.uuid=ea762c961bed46aea47cd60fda18c0de
> make[1]: *** [docker-run] Error 1
> make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-nf_7yydp/src'
> make: *** [docker-run-test-quick@centos7] Error 2
> 
> real    2m47.178s
> user    0m8.387s
> 
> 
> The full log is available at
> http://patchew.org/logs/20200325102131.23270-1-vsementsov@virtuozzo.com/testing.docker-quick@centos7/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
> 


-- 
Best regards,
Vladimir

On 3/25/20 5:21 AM, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> When sending iotests to upstream or do patch porting from one branch
> to another we very often have to resolve conflicts in group file, as
> many absolutely independent features are intersecting by this file.
> These conflicts are simple, but imagine how much time we all have
> already spent on resolving them? Let's finally get rid of group file.
> 
> Next, another thing I don't like about iotests is race for test number
> chosing: you should search through mail box, before chosing test number
> for new test.
> 
> So, I propose to get rid of group file and search for tests another way
> [look at patch 02]. Additionally I propose to move to human-readable
> names for test files, with notation test-* .

I suggest swapping the name: It's easier to write a glob for *-test vs. 
*-test.out than it is to write for test-* but not test-*.out.  (You 
don't want to execute the output files as a test).  That is, I suggest 
that ./check consider all 3-digit files and all files ending in -test as 
tests.

> 
> v1 was one patch "[PATCH] iotests: drop group file"
> 
> Vladimir Sementsov-Ogievskiy (2):
>    iotests: define group in each iotests
>    iotests: rework test finding

Do you plan on an additional patch (or set of patches) to rename some or 
all of the existing 3-digit tests?

Overall, having sensibly named tests makes sense to me.

If we are going to rename files, I'd also suggest that we rename the 
directory: tests/qemu-iotests/ feels redundant, compared to tests/iotests.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

25.03.2020 16:08, Eric Blake wrote:
> On 3/25/20 5:21 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> When sending iotests to upstream or do patch porting from one branch
>> to another we very often have to resolve conflicts in group file, as
>> many absolutely independent features are intersecting by this file.
>> These conflicts are simple, but imagine how much time we all have
>> already spent on resolving them? Let's finally get rid of group file.
>>
>> Next, another thing I don't like about iotests is race for test number
>> chosing: you should search through mail box, before chosing test number
>> for new test.
>>
>> So, I propose to get rid of group file and search for tests another way
>> [look at patch 02]. Additionally I propose to move to human-readable
>> names for test files, with notation test-* .
> 
> I suggest swapping the name: It's easier to write a glob for *-test vs. *-test.out than it is to write for test-* but not test-*.out.  (You don't want to execute the output files as a test).  That is, I suggest that ./check consider all 3-digit files and all files ending in -test as tests.

I like the idea. It also helps typing test name in terminal: stre<TAB> -> stream-test instead of te<TAB>stre<TAB> -> test-stream.

> 
>>
>> v1 was one patch "[PATCH] iotests: drop group file"
>>
>> Vladimir Sementsov-Ogievskiy (2):
>>    iotests: define group in each iotests
>>    iotests: rework test finding
> 
> Do you plan on an additional patch (or set of patches) to rename some or all of the existing 3-digit tests?

I think, I'll rename tests that I've written.

> 
> Overall, having sensibly named tests makes sense to me.
> 
> If we are going to rename files, I'd also suggest that we rename the directory: tests/qemu-iotests/ feels redundant, compared to tests/iotests.
> 

Agree.


-- 
Best regards,
Vladimir