Hi all!

These series has 3 goals:

 - get rid of group file (to forget about rebase and in-list conflicts)
 - introduce human-readable names for tests
 - rewrite check into python

v7:
  - fix wording and grammar
  - satisfy python linters
  - move argv interfaces all into one in new check script
  - support '-n' == '--dry-run' option
  - update check-block to run check with correct PYTHON

 findtests: - stop parsing test file after first '# group: ' line
            - use parse_test_name in add_group_file()
            - make new list instead of modifying parameter exclude_groups

 testenv: - fix machine_map
          - fix self.python

 testrunner: use env.python to run python tests

Vladimir Sementsov-Ogievskiy (11):
  iotests/277: use dot slash for nbd-fault-injector.py running
  iotests/303: use dot slash for qcow2.py running
  iotests: fix some whitespaces in test output files
  iotests: make tests executable
  iotests/294: add shebang line
  iotests: define group in each iotest
  iotests: add findtests.py
  iotests: add testenv.py
  iotests: add testrunner.py
  iotests: rewrite check into python
  iotests: rename and move 169 and 199 tests

 docs/devel/testing.rst                        |   50 +-
 Makefile                                      |    1 -
 tests/check-block.sh                          |    2 +-
 tests/qemu-iotests/001                        |    1 +
 tests/qemu-iotests/002                        |    1 +
 tests/qemu-iotests/003                        |    1 +
 tests/qemu-iotests/004                        |    1 +
 tests/qemu-iotests/005                        |    1 +
 tests/qemu-iotests/007                        |    1 +
 tests/qemu-iotests/008                        |    1 +
 tests/qemu-iotests/009                        |    1 +
 tests/qemu-iotests/010                        |    1 +
 tests/qemu-iotests/011                        |    1 +
 tests/qemu-iotests/012                        |    1 +
 tests/qemu-iotests/013                        |    1 +
 tests/qemu-iotests/014                        |    1 +
 tests/qemu-iotests/015                        |    1 +
 tests/qemu-iotests/017                        |    1 +
 tests/qemu-iotests/018                        |    1 +
 tests/qemu-iotests/019                        |    1 +
 tests/qemu-iotests/020                        |    1 +
 tests/qemu-iotests/021                        |    1 +
 tests/qemu-iotests/022                        |    1 +
 tests/qemu-iotests/023                        |    1 +
 tests/qemu-iotests/024                        |    1 +
 tests/qemu-iotests/025                        |    1 +
 tests/qemu-iotests/026                        |    1 +
 tests/qemu-iotests/027                        |    1 +
 tests/qemu-iotests/028                        |    1 +
 tests/qemu-iotests/029                        |    1 +
 tests/qemu-iotests/030                        |    1 +
 tests/qemu-iotests/031                        |    1 +
 tests/qemu-iotests/032                        |    1 +
 tests/qemu-iotests/033                        |    1 +
 tests/qemu-iotests/034                        |    1 +
 tests/qemu-iotests/035                        |    1 +
 tests/qemu-iotests/036                        |    1 +
 tests/qemu-iotests/037                        |    1 +
 tests/qemu-iotests/038                        |    1 +
 tests/qemu-iotests/039                        |    1 +
 tests/qemu-iotests/040                        |    1 +
 tests/qemu-iotests/041                        |    1 +
 tests/qemu-iotests/042                        |    1 +
 tests/qemu-iotests/043                        |    1 +
 tests/qemu-iotests/044                        |    1 +
 tests/qemu-iotests/045                        |    1 +
 tests/qemu-iotests/046                        |    1 +
 tests/qemu-iotests/047                        |    1 +
 tests/qemu-iotests/048                        |    1 +
 tests/qemu-iotests/049                        |    1 +
 tests/qemu-iotests/050                        |    1 +
 tests/qemu-iotests/051                        |    1 +
 tests/qemu-iotests/052                        |    1 +
 tests/qemu-iotests/053                        |    1 +
 tests/qemu-iotests/054                        |    1 +
 tests/qemu-iotests/055                        |    1 +
 tests/qemu-iotests/056                        |    1 +
 tests/qemu-iotests/057                        |    1 +
 tests/qemu-iotests/058                        |    1 +
 tests/qemu-iotests/059                        |    1 +
 tests/qemu-iotests/060                        |    1 +
 tests/qemu-iotests/061                        |    1 +
 tests/qemu-iotests/062                        |    1 +
 tests/qemu-iotests/063                        |    1 +
 tests/qemu-iotests/064                        |    1 +
 tests/qemu-iotests/065                        |    1 +
 tests/qemu-iotests/066                        |    1 +
 tests/qemu-iotests/068                        |    1 +
 tests/qemu-iotests/069                        |    1 +
 tests/qemu-iotests/070                        |    1 +
 tests/qemu-iotests/071                        |    1 +
 tests/qemu-iotests/072                        |    1 +
 tests/qemu-iotests/073                        |    1 +
 tests/qemu-iotests/074                        |    1 +
 tests/qemu-iotests/075                        |    1 +
 tests/qemu-iotests/076                        |    1 +
 tests/qemu-iotests/077                        |    1 +
 tests/qemu-iotests/078                        |    1 +
 tests/qemu-iotests/079                        |    1 +
 tests/qemu-iotests/080                        |    1 +
 tests/qemu-iotests/081                        |    1 +
 tests/qemu-iotests/082                        |    1 +
 tests/qemu-iotests/083                        |    1 +
 tests/qemu-iotests/084                        |    1 +
 tests/qemu-iotests/085                        |    1 +
 tests/qemu-iotests/086                        |    1 +
 tests/qemu-iotests/087                        |    1 +
 tests/qemu-iotests/088                        |    1 +
 tests/qemu-iotests/089                        |    1 +
 tests/qemu-iotests/090                        |    1 +
 tests/qemu-iotests/091                        |    1 +
 tests/qemu-iotests/092                        |    1 +
 tests/qemu-iotests/093                        |    1 +
 tests/qemu-iotests/094                        |    1 +
 tests/qemu-iotests/095                        |    1 +
 tests/qemu-iotests/096                        |    1 +
 tests/qemu-iotests/097                        |    1 +
 tests/qemu-iotests/098                        |    1 +
 tests/qemu-iotests/099                        |    1 +
 tests/qemu-iotests/101                        |    1 +
 tests/qemu-iotests/102                        |    1 +
 tests/qemu-iotests/103                        |    1 +
 tests/qemu-iotests/104                        |    1 +
 tests/qemu-iotests/105                        |    1 +
 tests/qemu-iotests/106                        |    1 +
 tests/qemu-iotests/107                        |    1 +
 tests/qemu-iotests/108                        |    1 +
 tests/qemu-iotests/109                        |    1 +
 tests/qemu-iotests/110                        |    1 +
 tests/qemu-iotests/111                        |    1 +
 tests/qemu-iotests/112                        |    1 +
 tests/qemu-iotests/113                        |    1 +
 tests/qemu-iotests/114                        |    1 +
 tests/qemu-iotests/115                        |    1 +
 tests/qemu-iotests/116                        |    1 +
 tests/qemu-iotests/117                        |    1 +
 tests/qemu-iotests/118                        |    1 +
 tests/qemu-iotests/119                        |    1 +
 tests/qemu-iotests/120                        |    1 +
 tests/qemu-iotests/121                        |    1 +
 tests/qemu-iotests/122                        |    1 +
 tests/qemu-iotests/123                        |    1 +
 tests/qemu-iotests/124                        |    1 +
 tests/qemu-iotests/125                        |    1 +
 tests/qemu-iotests/126                        |    1 +
 tests/qemu-iotests/127                        |    1 +
 tests/qemu-iotests/128                        |    1 +
 tests/qemu-iotests/129                        |    1 +
 tests/qemu-iotests/130                        |    1 +
 tests/qemu-iotests/131                        |    1 +
 tests/qemu-iotests/132                        |    1 +
 tests/qemu-iotests/133                        |    1 +
 tests/qemu-iotests/134                        |    1 +
 tests/qemu-iotests/135                        |    1 +
 tests/qemu-iotests/136                        |    1 +
 tests/qemu-iotests/137                        |    1 +
 tests/qemu-iotests/138                        |    1 +
 tests/qemu-iotests/139                        |    1 +
 tests/qemu-iotests/140                        |    1 +
 tests/qemu-iotests/141                        |    1 +
 tests/qemu-iotests/143                        |    1 +
 tests/qemu-iotests/144                        |    1 +
 tests/qemu-iotests/145                        |    1 +
 tests/qemu-iotests/146                        |    1 +
 tests/qemu-iotests/147                        |    1 +
 tests/qemu-iotests/148                        |    1 +
 tests/qemu-iotests/149                        |    1 +
 tests/qemu-iotests/150                        |    1 +
 tests/qemu-iotests/151                        |    1 +
 tests/qemu-iotests/152                        |    1 +
 tests/qemu-iotests/153                        |    1 +
 tests/qemu-iotests/154                        |    1 +
 tests/qemu-iotests/155                        |    1 +
 tests/qemu-iotests/156                        |    1 +
 tests/qemu-iotests/157                        |    1 +
 tests/qemu-iotests/158                        |    1 +
 tests/qemu-iotests/159                        |    1 +
 tests/qemu-iotests/160                        |    1 +
 tests/qemu-iotests/161                        |    1 +
 tests/qemu-iotests/162                        |    1 +
 tests/qemu-iotests/163                        |    1 +
 tests/qemu-iotests/165                        |    1 +
 tests/qemu-iotests/170                        |    1 +
 tests/qemu-iotests/171                        |    1 +
 tests/qemu-iotests/172                        |    1 +
 tests/qemu-iotests/173                        |    1 +
 tests/qemu-iotests/174                        |    1 +
 tests/qemu-iotests/175                        |    1 +
 tests/qemu-iotests/175.out                    |    2 +-
 tests/qemu-iotests/176                        |    1 +
 tests/qemu-iotests/177                        |    1 +
 tests/qemu-iotests/178                        |    1 +
 tests/qemu-iotests/179                        |    1 +
 tests/qemu-iotests/181                        |    1 +
 tests/qemu-iotests/182                        |    1 +
 tests/qemu-iotests/183                        |    1 +
 tests/qemu-iotests/184                        |    1 +
 tests/qemu-iotests/185                        |    1 +
 tests/qemu-iotests/186                        |    1 +
 tests/qemu-iotests/187                        |    1 +
 tests/qemu-iotests/188                        |    1 +
 tests/qemu-iotests/189                        |    1 +
 tests/qemu-iotests/190                        |    1 +
 tests/qemu-iotests/191                        |    1 +
 tests/qemu-iotests/192                        |    1 +
 tests/qemu-iotests/194                        |    1 +
 tests/qemu-iotests/195                        |    1 +
 tests/qemu-iotests/196                        |    1 +
 tests/qemu-iotests/197                        |    1 +
 tests/qemu-iotests/198                        |    1 +
 tests/qemu-iotests/200                        |    1 +
 tests/qemu-iotests/201                        |    1 +
 tests/qemu-iotests/202                        |    1 +
 tests/qemu-iotests/203                        |    1 +
 tests/qemu-iotests/204                        |    1 +
 tests/qemu-iotests/205                        |    1 +
 tests/qemu-iotests/206                        |    1 +
 tests/qemu-iotests/207                        |    1 +
 tests/qemu-iotests/208                        |    1 +
 tests/qemu-iotests/209                        |    1 +
 tests/qemu-iotests/210                        |    1 +
 tests/qemu-iotests/211                        |    1 +
 tests/qemu-iotests/212                        |    1 +
 tests/qemu-iotests/213                        |    1 +
 tests/qemu-iotests/214                        |    1 +
 tests/qemu-iotests/215                        |    1 +
 tests/qemu-iotests/216                        |    1 +
 tests/qemu-iotests/217                        |    1 +
 tests/qemu-iotests/218                        |    1 +
 tests/qemu-iotests/219                        |    1 +
 tests/qemu-iotests/220                        |    1 +
 tests/qemu-iotests/221                        |    1 +
 tests/qemu-iotests/222                        |    1 +
 tests/qemu-iotests/223                        |    1 +
 tests/qemu-iotests/224                        |    1 +
 tests/qemu-iotests/225                        |    1 +
 tests/qemu-iotests/226                        |    1 +
 tests/qemu-iotests/227                        |    1 +
 tests/qemu-iotests/228                        |    1 +
 tests/qemu-iotests/229                        |    1 +
 tests/qemu-iotests/231                        |    1 +
 tests/qemu-iotests/232                        |    1 +
 tests/qemu-iotests/233                        |    1 +
 tests/qemu-iotests/234                        |    1 +
 tests/qemu-iotests/235                        |    1 +
 tests/qemu-iotests/236                        |    1 +
 tests/qemu-iotests/237                        |    1 +
 tests/qemu-iotests/238                        |    1 +
 tests/qemu-iotests/239                        |    1 +
 tests/qemu-iotests/240                        |    1 +
 tests/qemu-iotests/241                        |    1 +
 tests/qemu-iotests/242                        |    1 +
 tests/qemu-iotests/243                        |    1 +
 tests/qemu-iotests/244                        |    1 +
 tests/qemu-iotests/245                        |    1 +
 tests/qemu-iotests/246                        |    1 +
 tests/qemu-iotests/247                        |    1 +
 tests/qemu-iotests/248                        |    1 +
 tests/qemu-iotests/249                        |    1 +
 tests/qemu-iotests/250                        |    1 +
 tests/qemu-iotests/251                        |    1 +
 tests/qemu-iotests/252                        |    1 +
 tests/qemu-iotests/253                        |    1 +
 tests/qemu-iotests/254                        |    1 +
 tests/qemu-iotests/255                        |    1 +
 tests/qemu-iotests/256                        |    1 +
 tests/qemu-iotests/257                        |    1 +
 tests/qemu-iotests/258                        |    1 +
 tests/qemu-iotests/259                        |    1 +
 tests/qemu-iotests/260                        |    1 +
 tests/qemu-iotests/261                        |    1 +
 tests/qemu-iotests/262                        |    1 +
 tests/qemu-iotests/263                        |    1 +
 tests/qemu-iotests/264                        |    1 +
 tests/qemu-iotests/265                        |    1 +
 tests/qemu-iotests/266                        |    1 +
 tests/qemu-iotests/267                        |    1 +
 tests/qemu-iotests/268                        |    1 +
 tests/qemu-iotests/270                        |    1 +
 tests/qemu-iotests/271                        |    1 +
 tests/qemu-iotests/271.out                    |   12 +-
 tests/qemu-iotests/272                        |    1 +
 tests/qemu-iotests/273                        |    1 +
 tests/qemu-iotests/274                        |    1 +
 tests/qemu-iotests/277                        |    3 +-
 tests/qemu-iotests/279                        |    1 +
 tests/qemu-iotests/280                        |    1 +
 tests/qemu-iotests/281                        |    1 +
 tests/qemu-iotests/282                        |    1 +
 tests/qemu-iotests/283                        |    1 +
 tests/qemu-iotests/284                        |    1 +
 tests/qemu-iotests/286                        |    1 +
 tests/qemu-iotests/287                        |    1 +
 tests/qemu-iotests/287.out                    |   10 +-
 tests/qemu-iotests/288                        |    1 +
 tests/qemu-iotests/289                        |    1 +
 tests/qemu-iotests/290                        |    1 +
 tests/qemu-iotests/291                        |    1 +
 tests/qemu-iotests/292                        |    1 +
 tests/qemu-iotests/293                        |    1 +
 tests/qemu-iotests/294                        |    2 +
 tests/qemu-iotests/295                        |    1 +
 tests/qemu-iotests/296                        |    1 +
 tests/qemu-iotests/297                        |    1 +
 tests/qemu-iotests/298                        |    0
 tests/qemu-iotests/299                        |    1 +
 tests/qemu-iotests/300                        |    1 +
 tests/qemu-iotests/301                        |    1 +
 tests/qemu-iotests/302                        |    1 +
 tests/qemu-iotests/303                        |    3 +-
 tests/qemu-iotests/304                        |    1 +
 tests/qemu-iotests/305                        |    1 +
 tests/qemu-iotests/307                        |    1 +
 tests/qemu-iotests/308                        |    1 +
 tests/qemu-iotests/309                        |    1 +
 tests/qemu-iotests/312                        |    1 +
 tests/qemu-iotests/check                      | 1085 ++---------------
 tests/qemu-iotests/common.env.in              |    3 -
 tests/qemu-iotests/findtests.py               |  159 +++
 tests/qemu-iotests/group                      |  321 -----
 tests/qemu-iotests/meson.build                |    3 -
 tests/qemu-iotests/testenv.py                 |  255 ++++
 tests/qemu-iotests/testrunner.py              |  344 ++++++
 .../migrate-bitmaps-postcopy-test}            |    1 +
 .../migrate-bitmaps-postcopy-test.out}        |    0
 .../{169 => tests/migrate-bitmaps-test}       |    1 +
 .../migrate-bitmaps-test.out}                 |    0
 307 files changed, 1233 insertions(+), 1310 deletions(-)
 mode change 100644 => 100755 tests/qemu-iotests/283
 mode change 100644 => 100755 tests/qemu-iotests/298
 mode change 100644 => 100755 tests/qemu-iotests/299
 delete mode 100644 tests/qemu-iotests/common.env.in
 create mode 100644 tests/qemu-iotests/findtests.py
 delete mode 100644 tests/qemu-iotests/group
 create mode 100644 tests/qemu-iotests/testenv.py
 create mode 100644 tests/qemu-iotests/testrunner.py
 rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (99%)
 rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%)
 rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (99%)
 rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)

-- 
2.29.2

On 1/16/21 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> These series has 3 goals:
> 
>  - get rid of group file (to forget about rebase and in-list conflicts)
>  - introduce human-readable names for tests
>  - rewrite check into python
> 
> v7:
>   - fix wording and grammar
>   - satisfy python linters
>   - move argv interfaces all into one in new check script
>   - support '-n' == '--dry-run' option
>   - update check-block to run check with correct PYTHON

I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my
NBD tree now.

> 
>  findtests: - stop parsing test file after first '# group: ' line
>             - use parse_test_name in add_group_file()
>             - make new list instead of modifying parameter exclude_groups
> 
>  testenv: - fix machine_map
>           - fix self.python
> 
>  testrunner: use env.python to run python tests
> 
> Vladimir Sementsov-Ogievskiy (11):
>   iotests/277: use dot slash for nbd-fault-injector.py running
>   iotests/303: use dot slash for qcow2.py running
>   iotests: fix some whitespaces in test output files
>   iotests: make tests executable
>   iotests/294: add shebang line
>   iotests: define group in each iotest
>   iotests: add findtests.py
>   iotests: add testenv.py
>   iotests: add testrunner.py
>   iotests: rewrite check into python
>   iotests: rename and move 169 and 199 tests
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:
> On 1/16/21 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
> > Hi all!
> > 
> > These series has 3 goals:
> > 
> >  - get rid of group file (to forget about rebase and in-list conflicts)
> >  - introduce human-readable names for tests
> >  - rewrite check into python
> > 
> > v7:
> >   - fix wording and grammar
> >   - satisfy python linters
> >   - move argv interfaces all into one in new check script
> >   - support '-n' == '--dry-run' option
> >   - update check-block to run check with correct PYTHON
> 
> I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my
> NBD tree now.

Oh, you already sent a pull request? Having 6 in without the rest is not
a good state. We now have the group info duplicated and one of them is
ignored, but will become the meaningful copy later. We need to be
careful to not let them diverge now.

I hope the rest is fine so we can switch over quickly, otherwise I'd
prefer to revert 6 and redo it from the then current state when we merge
the whole series.

Kevin

22.01.2021 14:27, Kevin Wolf wrote:
> Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:
>> On 1/16/21 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> Hi all!
>>>
>>> These series has 3 goals:
>>>
>>>   - get rid of group file (to forget about rebase and in-list conflicts)
>>>   - introduce human-readable names for tests
>>>   - rewrite check into python
>>>
>>> v7:
>>>    - fix wording and grammar
>>>    - satisfy python linters
>>>    - move argv interfaces all into one in new check script
>>>    - support '-n' == '--dry-run' option
>>>    - update check-block to run check with correct PYTHON
>>
>> I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my
>> NBD tree now.
> 
> Oh, you already sent a pull request? Having 6 in without the rest is not
> a good state. We now have the group info duplicated and one of them is
> ignored, but will become the meaningful copy later. We need to be
> careful to not let them diverge now.
> 
> I hope the rest is fine so we can switch over quickly, otherwise I'd
> prefer to revert 6 and redo it from the then current state when we merge
> the whole series.
> 

I can make a new version now with tiny fixes suggested by Eric if it is convenient. (and keep --color suggestion for a separate follow-up)



-- 
Best regards,
Vladimir

On 1/22/21 5:27 AM, Kevin Wolf wrote:
> Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:
>> On 1/16/21 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> Hi all!
>>>
>>> These series has 3 goals:
>>>
>>>  - get rid of group file (to forget about rebase and in-list conflicts)
>>>  - introduce human-readable names for tests
>>>  - rewrite check into python
>>>
>>> v7:
>>>   - fix wording and grammar
>>>   - satisfy python linters
>>>   - move argv interfaces all into one in new check script
>>>   - support '-n' == '--dry-run' option
>>>   - update check-block to run check with correct PYTHON
>>
>> I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my
>> NBD tree now.
> 
> Oh, you already sent a pull request? Having 6 in without the rest is not
> a good state. We now have the group info duplicated and one of them is
> ignored, but will become the meaningful copy later. We need to be
> careful to not let them diverge now.
> 
> I hope the rest is fine so we can switch over quickly, otherwise I'd
> prefer to revert 6 and redo it from the then current state when we merge
> the whole series.

Yeah, I probably jumped the gun there.  If we don't have v8 in good
working shape soon, I'm happy to send a temporary revert pull request
for patch 6 - let's shoot for late Monday as the cut-off point where I
will act if the revert is needed.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Am 22.01.2021 um 17:08 hat Eric Blake geschrieben:
> On 1/22/21 5:27 AM, Kevin Wolf wrote:
> > Am 20.01.2021 um 21:52 hat Eric Blake geschrieben:
> >> On 1/16/21 7:44 AM, Vladimir Sementsov-Ogievskiy wrote:
> >>> Hi all!
> >>>
> >>> These series has 3 goals:
> >>>
> >>>  - get rid of group file (to forget about rebase and in-list conflicts)
> >>>  - introduce human-readable names for tests
> >>>  - rewrite check into python
> >>>
> >>> v7:
> >>>   - fix wording and grammar
> >>>   - satisfy python linters
> >>>   - move argv interfaces all into one in new check script
> >>>   - support '-n' == '--dry-run' option
> >>>   - update check-block to run check with correct PYTHON
> >>
> >> I'd really like a second reviewer on 7-11, but I'm queueing 1-6 on my
> >> NBD tree now.
> > 
> > Oh, you already sent a pull request? Having 6 in without the rest is not
> > a good state. We now have the group info duplicated and one of them is
> > ignored, but will become the meaningful copy later. We need to be
> > careful to not let them diverge now.
> > 
> > I hope the rest is fine so we can switch over quickly, otherwise I'd
> > prefer to revert 6 and redo it from the then current state when we merge
> > the whole series.
> 
> Yeah, I probably jumped the gun there.  If we don't have v8 in good
> working shape soon, I'm happy to send a temporary revert pull request
> for patch 6 - let's shoot for late Monday as the cut-off point where I
> will act if the revert is needed.

I think I'm going to send a pull request anyway, so I could include it
there if necessary.

Kevin

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:
> Hi all!
> 
> These series has 3 goals:
> 
>  - get rid of group file (to forget about rebase and in-list conflicts)
>  - introduce human-readable names for tests
>  - rewrite check into python
> 
> v7:
>   - fix wording and grammar
>   - satisfy python linters
>   - move argv interfaces all into one in new check script
>   - support '-n' == '--dry-run' option
>   - update check-block to run check with correct PYTHON

Okay, I think I'm finished with the review for this version.

I also tried pylint/mypy again and it's mostly clean now (pylint
complains about the TODO comments, I think we should just disable that
warning).

Feel free to include the following as patch 12 in v8.

Kevin


diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index 85bc1c0c85..8aaa3e455c 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -31,13 +31,15 @@ if ! type -p "mypy" > /dev/null; then
     _notrun "mypy not found"
 fi

-pylint-3 --score=n iotests.py
+FILES="findtests.py iotests.py testenv.py testrunner.py check"
+
+pylint-3 --score=n $FILES

 MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
     --disallow-any-generics --disallow-incomplete-defs \
     --disallow-untyped-decorators --no-implicit-optional \
     --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+    --no-implicit-reexport $FILES

 # success, all done
 echo "*** done"
diff --git a/tests/qemu-iotests/297.out b/tests/qemu-iotests/297.out
index 6acc843649..85213ef96e 100644
--- a/tests/qemu-iotests/297.out
+++ b/tests/qemu-iotests/297.out
@@ -1,3 +1,3 @@
 QA output created by 297
-Success: no issues found in 1 source file
+Success: no issues found in 5 source files
 *** done
diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc
index cd3702e23c..980bfa7b20 100644
--- a/tests/qemu-iotests/pylintrc
+++ b/tests/qemu-iotests/pylintrc
@@ -9,7 +9,8 @@
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use "--disable=all --enable=classes
 # --disable=W".
-disable=invalid-name,
+disable=fixme,
+        invalid-name,
         no-else-return,
         too-few-public-methods,
         too-many-arguments,

22.01.2021 19:16, Kevin Wolf wrote:
> Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> Hi all!
>>
>> These series has 3 goals:
>>
>>   - get rid of group file (to forget about rebase and in-list conflicts)
>>   - introduce human-readable names for tests
>>   - rewrite check into python
>>
>> v7:
>>    - fix wording and grammar
>>    - satisfy python linters
>>    - move argv interfaces all into one in new check script
>>    - support '-n' == '--dry-run' option
>>    - update check-block to run check with correct PYTHON
> 
> Okay, I think I'm finished with the review for this version.
> 
> I also tried pylint/mypy again and it's mostly clean now (pylint
> complains about the TODO comments, I think we should just disable that
> warning).
> 
> Feel free to include the following as patch 12 in v8.
> 
> Kevin

I remember Max already queued good changes for 297 in his block branch

> 
> 
> diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
> index 85bc1c0c85..8aaa3e455c 100755
> --- a/tests/qemu-iotests/297
> +++ b/tests/qemu-iotests/297
> @@ -31,13 +31,15 @@ if ! type -p "mypy" > /dev/null; then
>       _notrun "mypy not found"
>   fi
> 
> -pylint-3 --score=n iotests.py
> +FILES="findtests.py iotests.py testenv.py testrunner.py check"
> +
> +pylint-3 --score=n $FILES
> 
>   MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
>       --disallow-any-generics --disallow-incomplete-defs \
>       --disallow-untyped-decorators --no-implicit-optional \
>       --warn-redundant-casts --warn-unused-ignores \
> -    --no-implicit-reexport iotests.py
> +    --no-implicit-reexport $FILES
> 
>   # success, all done
>   echo "*** done"
> diff --git a/tests/qemu-iotests/297.out b/tests/qemu-iotests/297.out
> index 6acc843649..85213ef96e 100644
> --- a/tests/qemu-iotests/297.out
> +++ b/tests/qemu-iotests/297.out
> @@ -1,3 +1,3 @@
>   QA output created by 297
> -Success: no issues found in 1 source file
> +Success: no issues found in 5 source files
>   *** done
> diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc
> index cd3702e23c..980bfa7b20 100644
> --- a/tests/qemu-iotests/pylintrc
> +++ b/tests/qemu-iotests/pylintrc
> @@ -9,7 +9,8 @@
>   # --enable=similarities". If you want to run only the classes checker, but have
>   # no Warning level messages displayed, use "--disable=all --enable=classes
>   # --disable=W".
> -disable=invalid-name,
> +disable=fixme,
> +        invalid-name,

Max just disable it in 297. I think we should not disable it globally, as additional hint doesn't hurt (except for test output).

>           no-else-return,
>           too-few-public-methods,
>           too-many-arguments,
> 


-- 
Best regards,
Vladimir

On 16/01/21 14:44, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> These series has 3 goals:
> 
>   - get rid of group file (to forget about rebase and in-list conflicts)
>   - introduce human-readable names for tests
>   - rewrite check into python

Very nice!

I have wondered for some time if we should rewrite check as a wrapper 
around avocado.  Your effort certainly goes in the right direction, 
since it at least uses the same programming language and introduces 
abstractions that can be easily turned into avocado wrappers.

Paolo

> v7:
>    - fix wording and grammar
>    - satisfy python linters
>    - move argv interfaces all into one in new check script
>    - support '-n' == '--dry-run' option
>    - update check-block to run check with correct PYTHON
> 
>   findtests: - stop parsing test file after first '# group: ' line
>              - use parse_test_name in add_group_file()
>              - make new list instead of modifying parameter exclude_groups
> 
>   testenv: - fix machine_map
>            - fix self.python
> 
>   testrunner: use env.python to run python tests
> 
> Vladimir Sementsov-Ogievskiy (11):
>    iotests/277: use dot slash for nbd-fault-injector.py running
>    iotests/303: use dot slash for qcow2.py running
>    iotests: fix some whitespaces in test output files
>    iotests: make tests executable
>    iotests/294: add shebang line
>    iotests: define group in each iotest
>    iotests: add findtests.py
>    iotests: add testenv.py
>    iotests: add testrunner.py
>    iotests: rewrite check into python
>    iotests: rename and move 169 and 199 tests
> 
>   docs/devel/testing.rst                        |   50 +-
>   Makefile                                      |    1 -
>   tests/check-block.sh                          |    2 +-
>   tests/qemu-iotests/001                        |    1 +
>   tests/qemu-iotests/002                        |    1 +
>   tests/qemu-iotests/003                        |    1 +
>   tests/qemu-iotests/004                        |    1 +
>   tests/qemu-iotests/005                        |    1 +
>   tests/qemu-iotests/007                        |    1 +
>   tests/qemu-iotests/008                        |    1 +
>   tests/qemu-iotests/009                        |    1 +
>   tests/qemu-iotests/010                        |    1 +
>   tests/qemu-iotests/011                        |    1 +
>   tests/qemu-iotests/012                        |    1 +
>   tests/qemu-iotests/013                        |    1 +
>   tests/qemu-iotests/014                        |    1 +
>   tests/qemu-iotests/015                        |    1 +
>   tests/qemu-iotests/017                        |    1 +
>   tests/qemu-iotests/018                        |    1 +
>   tests/qemu-iotests/019                        |    1 +
>   tests/qemu-iotests/020                        |    1 +
>   tests/qemu-iotests/021                        |    1 +
>   tests/qemu-iotests/022                        |    1 +
>   tests/qemu-iotests/023                        |    1 +
>   tests/qemu-iotests/024                        |    1 +
>   tests/qemu-iotests/025                        |    1 +
>   tests/qemu-iotests/026                        |    1 +
>   tests/qemu-iotests/027                        |    1 +
>   tests/qemu-iotests/028                        |    1 +
>   tests/qemu-iotests/029                        |    1 +
>   tests/qemu-iotests/030                        |    1 +
>   tests/qemu-iotests/031                        |    1 +
>   tests/qemu-iotests/032                        |    1 +
>   tests/qemu-iotests/033                        |    1 +
>   tests/qemu-iotests/034                        |    1 +
>   tests/qemu-iotests/035                        |    1 +
>   tests/qemu-iotests/036                        |    1 +
>   tests/qemu-iotests/037                        |    1 +
>   tests/qemu-iotests/038                        |    1 +
>   tests/qemu-iotests/039                        |    1 +
>   tests/qemu-iotests/040                        |    1 +
>   tests/qemu-iotests/041                        |    1 +
>   tests/qemu-iotests/042                        |    1 +
>   tests/qemu-iotests/043                        |    1 +
>   tests/qemu-iotests/044                        |    1 +
>   tests/qemu-iotests/045                        |    1 +
>   tests/qemu-iotests/046                        |    1 +
>   tests/qemu-iotests/047                        |    1 +
>   tests/qemu-iotests/048                        |    1 +
>   tests/qemu-iotests/049                        |    1 +
>   tests/qemu-iotests/050                        |    1 +
>   tests/qemu-iotests/051                        |    1 +
>   tests/qemu-iotests/052                        |    1 +
>   tests/qemu-iotests/053                        |    1 +
>   tests/qemu-iotests/054                        |    1 +
>   tests/qemu-iotests/055                        |    1 +
>   tests/qemu-iotests/056                        |    1 +
>   tests/qemu-iotests/057                        |    1 +
>   tests/qemu-iotests/058                        |    1 +
>   tests/qemu-iotests/059                        |    1 +
>   tests/qemu-iotests/060                        |    1 +
>   tests/qemu-iotests/061                        |    1 +
>   tests/qemu-iotests/062                        |    1 +
>   tests/qemu-iotests/063                        |    1 +
>   tests/qemu-iotests/064                        |    1 +
>   tests/qemu-iotests/065                        |    1 +
>   tests/qemu-iotests/066                        |    1 +
>   tests/qemu-iotests/068                        |    1 +
>   tests/qemu-iotests/069                        |    1 +
>   tests/qemu-iotests/070                        |    1 +
>   tests/qemu-iotests/071                        |    1 +
>   tests/qemu-iotests/072                        |    1 +
>   tests/qemu-iotests/073                        |    1 +
>   tests/qemu-iotests/074                        |    1 +
>   tests/qemu-iotests/075                        |    1 +
>   tests/qemu-iotests/076                        |    1 +
>   tests/qemu-iotests/077                        |    1 +
>   tests/qemu-iotests/078                        |    1 +
>   tests/qemu-iotests/079                        |    1 +
>   tests/qemu-iotests/080                        |    1 +
>   tests/qemu-iotests/081                        |    1 +
>   tests/qemu-iotests/082                        |    1 +
>   tests/qemu-iotests/083                        |    1 +
>   tests/qemu-iotests/084                        |    1 +
>   tests/qemu-iotests/085                        |    1 +
>   tests/qemu-iotests/086                        |    1 +
>   tests/qemu-iotests/087                        |    1 +
>   tests/qemu-iotests/088                        |    1 +
>   tests/qemu-iotests/089                        |    1 +
>   tests/qemu-iotests/090                        |    1 +
>   tests/qemu-iotests/091                        |    1 +
>   tests/qemu-iotests/092                        |    1 +
>   tests/qemu-iotests/093                        |    1 +
>   tests/qemu-iotests/094                        |    1 +
>   tests/qemu-iotests/095                        |    1 +
>   tests/qemu-iotests/096                        |    1 +
>   tests/qemu-iotests/097                        |    1 +
>   tests/qemu-iotests/098                        |    1 +
>   tests/qemu-iotests/099                        |    1 +
>   tests/qemu-iotests/101                        |    1 +
>   tests/qemu-iotests/102                        |    1 +
>   tests/qemu-iotests/103                        |    1 +
>   tests/qemu-iotests/104                        |    1 +
>   tests/qemu-iotests/105                        |    1 +
>   tests/qemu-iotests/106                        |    1 +
>   tests/qemu-iotests/107                        |    1 +
>   tests/qemu-iotests/108                        |    1 +
>   tests/qemu-iotests/109                        |    1 +
>   tests/qemu-iotests/110                        |    1 +
>   tests/qemu-iotests/111                        |    1 +
>   tests/qemu-iotests/112                        |    1 +
>   tests/qemu-iotests/113                        |    1 +
>   tests/qemu-iotests/114                        |    1 +
>   tests/qemu-iotests/115                        |    1 +
>   tests/qemu-iotests/116                        |    1 +
>   tests/qemu-iotests/117                        |    1 +
>   tests/qemu-iotests/118                        |    1 +
>   tests/qemu-iotests/119                        |    1 +
>   tests/qemu-iotests/120                        |    1 +
>   tests/qemu-iotests/121                        |    1 +
>   tests/qemu-iotests/122                        |    1 +
>   tests/qemu-iotests/123                        |    1 +
>   tests/qemu-iotests/124                        |    1 +
>   tests/qemu-iotests/125                        |    1 +
>   tests/qemu-iotests/126                        |    1 +
>   tests/qemu-iotests/127                        |    1 +
>   tests/qemu-iotests/128                        |    1 +
>   tests/qemu-iotests/129                        |    1 +
>   tests/qemu-iotests/130                        |    1 +
>   tests/qemu-iotests/131                        |    1 +
>   tests/qemu-iotests/132                        |    1 +
>   tests/qemu-iotests/133                        |    1 +
>   tests/qemu-iotests/134                        |    1 +
>   tests/qemu-iotests/135                        |    1 +
>   tests/qemu-iotests/136                        |    1 +
>   tests/qemu-iotests/137                        |    1 +
>   tests/qemu-iotests/138                        |    1 +
>   tests/qemu-iotests/139                        |    1 +
>   tests/qemu-iotests/140                        |    1 +
>   tests/qemu-iotests/141                        |    1 +
>   tests/qemu-iotests/143                        |    1 +
>   tests/qemu-iotests/144                        |    1 +
>   tests/qemu-iotests/145                        |    1 +
>   tests/qemu-iotests/146                        |    1 +
>   tests/qemu-iotests/147                        |    1 +
>   tests/qemu-iotests/148                        |    1 +
>   tests/qemu-iotests/149                        |    1 +
>   tests/qemu-iotests/150                        |    1 +
>   tests/qemu-iotests/151                        |    1 +
>   tests/qemu-iotests/152                        |    1 +
>   tests/qemu-iotests/153                        |    1 +
>   tests/qemu-iotests/154                        |    1 +
>   tests/qemu-iotests/155                        |    1 +
>   tests/qemu-iotests/156                        |    1 +
>   tests/qemu-iotests/157                        |    1 +
>   tests/qemu-iotests/158                        |    1 +
>   tests/qemu-iotests/159                        |    1 +
>   tests/qemu-iotests/160                        |    1 +
>   tests/qemu-iotests/161                        |    1 +
>   tests/qemu-iotests/162                        |    1 +
>   tests/qemu-iotests/163                        |    1 +
>   tests/qemu-iotests/165                        |    1 +
>   tests/qemu-iotests/170                        |    1 +
>   tests/qemu-iotests/171                        |    1 +
>   tests/qemu-iotests/172                        |    1 +
>   tests/qemu-iotests/173                        |    1 +
>   tests/qemu-iotests/174                        |    1 +
>   tests/qemu-iotests/175                        |    1 +
>   tests/qemu-iotests/175.out                    |    2 +-
>   tests/qemu-iotests/176                        |    1 +
>   tests/qemu-iotests/177                        |    1 +
>   tests/qemu-iotests/178                        |    1 +
>   tests/qemu-iotests/179                        |    1 +
>   tests/qemu-iotests/181                        |    1 +
>   tests/qemu-iotests/182                        |    1 +
>   tests/qemu-iotests/183                        |    1 +
>   tests/qemu-iotests/184                        |    1 +
>   tests/qemu-iotests/185                        |    1 +
>   tests/qemu-iotests/186                        |    1 +
>   tests/qemu-iotests/187                        |    1 +
>   tests/qemu-iotests/188                        |    1 +
>   tests/qemu-iotests/189                        |    1 +
>   tests/qemu-iotests/190                        |    1 +
>   tests/qemu-iotests/191                        |    1 +
>   tests/qemu-iotests/192                        |    1 +
>   tests/qemu-iotests/194                        |    1 +
>   tests/qemu-iotests/195                        |    1 +
>   tests/qemu-iotests/196                        |    1 +
>   tests/qemu-iotests/197                        |    1 +
>   tests/qemu-iotests/198                        |    1 +
>   tests/qemu-iotests/200                        |    1 +
>   tests/qemu-iotests/201                        |    1 +
>   tests/qemu-iotests/202                        |    1 +
>   tests/qemu-iotests/203                        |    1 +
>   tests/qemu-iotests/204                        |    1 +
>   tests/qemu-iotests/205                        |    1 +
>   tests/qemu-iotests/206                        |    1 +
>   tests/qemu-iotests/207                        |    1 +
>   tests/qemu-iotests/208                        |    1 +
>   tests/qemu-iotests/209                        |    1 +
>   tests/qemu-iotests/210                        |    1 +
>   tests/qemu-iotests/211                        |    1 +
>   tests/qemu-iotests/212                        |    1 +
>   tests/qemu-iotests/213                        |    1 +
>   tests/qemu-iotests/214                        |    1 +
>   tests/qemu-iotests/215                        |    1 +
>   tests/qemu-iotests/216                        |    1 +
>   tests/qemu-iotests/217                        |    1 +
>   tests/qemu-iotests/218                        |    1 +
>   tests/qemu-iotests/219                        |    1 +
>   tests/qemu-iotests/220                        |    1 +
>   tests/qemu-iotests/221                        |    1 +
>   tests/qemu-iotests/222                        |    1 +
>   tests/qemu-iotests/223                        |    1 +
>   tests/qemu-iotests/224                        |    1 +
>   tests/qemu-iotests/225                        |    1 +
>   tests/qemu-iotests/226                        |    1 +
>   tests/qemu-iotests/227                        |    1 +
>   tests/qemu-iotests/228                        |    1 +
>   tests/qemu-iotests/229                        |    1 +
>   tests/qemu-iotests/231                        |    1 +
>   tests/qemu-iotests/232                        |    1 +
>   tests/qemu-iotests/233                        |    1 +
>   tests/qemu-iotests/234                        |    1 +
>   tests/qemu-iotests/235                        |    1 +
>   tests/qemu-iotests/236                        |    1 +
>   tests/qemu-iotests/237                        |    1 +
>   tests/qemu-iotests/238                        |    1 +
>   tests/qemu-iotests/239                        |    1 +
>   tests/qemu-iotests/240                        |    1 +
>   tests/qemu-iotests/241                        |    1 +
>   tests/qemu-iotests/242                        |    1 +
>   tests/qemu-iotests/243                        |    1 +
>   tests/qemu-iotests/244                        |    1 +
>   tests/qemu-iotests/245                        |    1 +
>   tests/qemu-iotests/246                        |    1 +
>   tests/qemu-iotests/247                        |    1 +
>   tests/qemu-iotests/248                        |    1 +
>   tests/qemu-iotests/249                        |    1 +
>   tests/qemu-iotests/250                        |    1 +
>   tests/qemu-iotests/251                        |    1 +
>   tests/qemu-iotests/252                        |    1 +
>   tests/qemu-iotests/253                        |    1 +
>   tests/qemu-iotests/254                        |    1 +
>   tests/qemu-iotests/255                        |    1 +
>   tests/qemu-iotests/256                        |    1 +
>   tests/qemu-iotests/257                        |    1 +
>   tests/qemu-iotests/258                        |    1 +
>   tests/qemu-iotests/259                        |    1 +
>   tests/qemu-iotests/260                        |    1 +
>   tests/qemu-iotests/261                        |    1 +
>   tests/qemu-iotests/262                        |    1 +
>   tests/qemu-iotests/263                        |    1 +
>   tests/qemu-iotests/264                        |    1 +
>   tests/qemu-iotests/265                        |    1 +
>   tests/qemu-iotests/266                        |    1 +
>   tests/qemu-iotests/267                        |    1 +
>   tests/qemu-iotests/268                        |    1 +
>   tests/qemu-iotests/270                        |    1 +
>   tests/qemu-iotests/271                        |    1 +
>   tests/qemu-iotests/271.out                    |   12 +-
>   tests/qemu-iotests/272                        |    1 +
>   tests/qemu-iotests/273                        |    1 +
>   tests/qemu-iotests/274                        |    1 +
>   tests/qemu-iotests/277                        |    3 +-
>   tests/qemu-iotests/279                        |    1 +
>   tests/qemu-iotests/280                        |    1 +
>   tests/qemu-iotests/281                        |    1 +
>   tests/qemu-iotests/282                        |    1 +
>   tests/qemu-iotests/283                        |    1 +
>   tests/qemu-iotests/284                        |    1 +
>   tests/qemu-iotests/286                        |    1 +
>   tests/qemu-iotests/287                        |    1 +
>   tests/qemu-iotests/287.out                    |   10 +-
>   tests/qemu-iotests/288                        |    1 +
>   tests/qemu-iotests/289                        |    1 +
>   tests/qemu-iotests/290                        |    1 +
>   tests/qemu-iotests/291                        |    1 +
>   tests/qemu-iotests/292                        |    1 +
>   tests/qemu-iotests/293                        |    1 +
>   tests/qemu-iotests/294                        |    2 +
>   tests/qemu-iotests/295                        |    1 +
>   tests/qemu-iotests/296                        |    1 +
>   tests/qemu-iotests/297                        |    1 +
>   tests/qemu-iotests/298                        |    0
>   tests/qemu-iotests/299                        |    1 +
>   tests/qemu-iotests/300                        |    1 +
>   tests/qemu-iotests/301                        |    1 +
>   tests/qemu-iotests/302                        |    1 +
>   tests/qemu-iotests/303                        |    3 +-
>   tests/qemu-iotests/304                        |    1 +
>   tests/qemu-iotests/305                        |    1 +
>   tests/qemu-iotests/307                        |    1 +
>   tests/qemu-iotests/308                        |    1 +
>   tests/qemu-iotests/309                        |    1 +
>   tests/qemu-iotests/312                        |    1 +
>   tests/qemu-iotests/check                      | 1085 ++---------------
>   tests/qemu-iotests/common.env.in              |    3 -
>   tests/qemu-iotests/findtests.py               |  159 +++
>   tests/qemu-iotests/group                      |  321 -----
>   tests/qemu-iotests/meson.build                |    3 -
>   tests/qemu-iotests/testenv.py                 |  255 ++++
>   tests/qemu-iotests/testrunner.py              |  344 ++++++
>   .../migrate-bitmaps-postcopy-test}            |    1 +
>   .../migrate-bitmaps-postcopy-test.out}        |    0
>   .../{169 => tests/migrate-bitmaps-test}       |    1 +
>   .../migrate-bitmaps-test.out}                 |    0
>   307 files changed, 1233 insertions(+), 1310 deletions(-)
>   mode change 100644 => 100755 tests/qemu-iotests/283
>   mode change 100644 => 100755 tests/qemu-iotests/298
>   mode change 100644 => 100755 tests/qemu-iotests/299
>   delete mode 100644 tests/qemu-iotests/common.env.in
>   create mode 100644 tests/qemu-iotests/findtests.py
>   delete mode 100644 tests/qemu-iotests/group
>   create mode 100644 tests/qemu-iotests/testenv.py
>   create mode 100644 tests/qemu-iotests/testrunner.py
>   rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (99%)
>   rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%)
>   rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (99%)
>   rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)
>