This series uses python logging to enable output conditionally on
iotests.log(). We unify an initialization call (which also enables
debugging output for those tests with -d) and then make the switch
inside of iotests.

It will help alleviate the need to create logged/unlogged versions
of all the various helpers we have made.

Also, I got lost and accidentally delinted iotests while I was here.
Sorry about that. By version 9, it's now the overwhelming focus of
this series. No good deed, etc.

V9:

001/14:[----] [-C] 'iotests: do a light delinting'
002/14:[----] [--] 'iotests: don't use 'format' for drive_add'
003/14:[----] [-C] 'iotests: ignore import warnings from pylint'
004/14:[----] [--] 'iotests: replace mutable list default args'
005/14:[----] [--] 'iotests: add pylintrc file'
006/14:[down]      'iotests: alphabetize standard imports'
007/14:[down]      'iotests: drop pre-Python 3.4 compatibility code'
008/14:[down]      'iotests: touch up log function signature'
009/14:[----] [--] 'iotests: limit line length to 79 chars'
010/14:[down]      'iotests: add hmp helper with logging'
011/14:[0004] [FC] 'iotests: add script_initialize'
012/14:[----] [--] 'iotest 258: use script_main'
013/14:[----] [--] 'iotests: Mark verify functions as private'
014/14:[0001] [FC] 'iotests: use python logging for iotests.log()'

006: New.
007: Split from old patch.
008: Split from old patch; enhanced a little to justify its own patch.
010: New, pulled in from bitmap-populate series. Helps line length.
011: Reflow columns for long `typing` import list. (Kept RB.)
014: New blank line. (Kept RB.)

V8:
- Split out the little drop of Python 3.4 code. (Phil)
- Change line continuation styles (QEMU Memorial Choir)
- Rebase changes; remove use_log from more places, adjust test output.

V7:
- All delinting patches are now entirely front-loaded.
- Redid delinting to avoid "correcting" no-else-return statements.
- Moved more mutable list corrections into patch 4, to make it standalone.
- Moved pylintrc up to patch 5. Disabled no-else-return.
- Added patch 6 to require line length checks.
  (Some python 3.4 compatibility code is removed as a consequence.)
- Patch 7 changes slightly as a result of patch 4 changes.
- Added some logging explainer into patch 10.
  (Patch changes slightly because of patch 6.)

V6:
 - It's been so long since V5, let's just look at it anew.

John Snow (14):
  iotests: do a light delinting
  iotests: don't use 'format' for drive_add
  iotests: ignore import warnings from pylint
  iotests: replace mutable list default args
  iotests: add pylintrc file
  iotests: alphabetize standard imports
  iotests: drop pre-Python 3.4 compatibility code
  iotests: touch up log function signature
  iotests: limit line length to 79 chars
  iotests: add hmp helper with logging
  iotests: add script_initialize
  iotest 258: use script_main
  iotests: Mark verify functions as private
  iotests: use python logging for iotests.log()

 tests/qemu-iotests/030        |   4 +-
 tests/qemu-iotests/055        |   3 +-
 tests/qemu-iotests/149        |   3 +-
 tests/qemu-iotests/155        |   2 +-
 tests/qemu-iotests/194        |   4 +-
 tests/qemu-iotests/202        |   4 +-
 tests/qemu-iotests/203        |   4 +-
 tests/qemu-iotests/206        |   2 +-
 tests/qemu-iotests/207        |   6 +-
 tests/qemu-iotests/208        |   2 +-
 tests/qemu-iotests/209        |   2 +-
 tests/qemu-iotests/210        |   6 +-
 tests/qemu-iotests/211        |   6 +-
 tests/qemu-iotests/212        |   6 +-
 tests/qemu-iotests/213        |   6 +-
 tests/qemu-iotests/216        |   4 +-
 tests/qemu-iotests/218        |   2 +-
 tests/qemu-iotests/219        |   2 +-
 tests/qemu-iotests/222        |   7 +-
 tests/qemu-iotests/224        |   4 +-
 tests/qemu-iotests/228        |   6 +-
 tests/qemu-iotests/234        |   4 +-
 tests/qemu-iotests/235        |   4 +-
 tests/qemu-iotests/236        |   2 +-
 tests/qemu-iotests/237        |   2 +-
 tests/qemu-iotests/238        |   2 +
 tests/qemu-iotests/242        |   2 +-
 tests/qemu-iotests/245        |   1 +
 tests/qemu-iotests/245.out    |  24 +--
 tests/qemu-iotests/246        |   2 +-
 tests/qemu-iotests/248        |   2 +-
 tests/qemu-iotests/254        |   2 +-
 tests/qemu-iotests/255        |   2 +-
 tests/qemu-iotests/256        |   2 +-
 tests/qemu-iotests/258        |  10 +-
 tests/qemu-iotests/260        |   4 +-
 tests/qemu-iotests/262        |   4 +-
 tests/qemu-iotests/264        |   4 +-
 tests/qemu-iotests/277        |   2 +
 tests/qemu-iotests/280        |   8 +-
 tests/qemu-iotests/283        |   4 +-
 tests/qemu-iotests/iotests.py | 356 ++++++++++++++++++++--------------
 tests/qemu-iotests/pylintrc   |  26 +++
 43 files changed, 333 insertions(+), 221 deletions(-)
 create mode 100644 tests/qemu-iotests/pylintrc

-- 
2.21.1

On 25.03.20 00:20, John Snow wrote:
> This series uses python logging to enable output conditionally on
> iotests.log(). We unify an initialization call (which also enables
> debugging output for those tests with -d) and then make the switch
> inside of iotests.
> 
> It will help alleviate the need to create logged/unlogged versions
> of all the various helpers we have made.
> 
> Also, I got lost and accidentally delinted iotests while I was here.
> Sorry about that. By version 9, it's now the overwhelming focus of
> this series. No good deed, etc.

Generally, test patches are fair game for the freeze period.  However,
this series is quite extensive, so I might prefer block-next here.
OTOH, if I do take it to block-next, patch 11 might grow stale.

Do you have a strong opinion either way?

Max

On 3/30/20 9:00 AM, Max Reitz wrote:
> On 25.03.20 00:20, John Snow wrote:
>> This series uses python logging to enable output conditionally on
>> iotests.log(). We unify an initialization call (which also enables
>> debugging output for those tests with -d) and then make the switch
>> inside of iotests.
>>
>> It will help alleviate the need to create logged/unlogged versions
>> of all the various helpers we have made.
>>
>> Also, I got lost and accidentally delinted iotests while I was here.
>> Sorry about that. By version 9, it's now the overwhelming focus of
>> this series. No good deed, etc.
> 
> Generally, test patches are fair game for the freeze period.  However,
> this series is quite extensive, so I might prefer block-next here.
> OTOH, if I do take it to block-next, patch 11 might grow stale.
> 
> Do you have a strong opinion either way?
> 
> Max
> 

Not really, no.

Might be nice to see it tossed into the RC while everyone is doing
testing to see if it causes problems. On the other hand, it probably
WILL cause problems with various untested version combinations and so forth.

Either or.

--js

On 3/24/20 7:20 PM, John Snow wrote:
> This series uses python logging to enable output conditionally on
> iotests.log(). We unify an initialization call (which also enables
> debugging output for those tests with -d) and then make the switch
> inside of iotests.
> 
> It will help alleviate the need to create logged/unlogged versions
> of all the various helpers we have made.
> 
> Also, I got lost and accidentally delinted iotests while I was here.
> Sorry about that. By version 9, it's now the overwhelming focus of
> this series. No good deed, etc.


Version requirements, as discovered by Kevin's Python Museum:

mypy >= 0.620
pylint >= 2.2.0
astroid == 2.1.0 (or >= 2.2.0 if using pylint >= 2.3.0)


Hm, though ... pylint does not like 'Collection' very much:

iotests.py:1139:41: E1136: Value 'Collection' is unsubscriptable
(unsubscriptable-object)

It works OK for the same pylint versions under 3.7, but it's busted a
bit under 3.6. See https://github.com/PyCQA/pylint/issues/2377

Well. Collection is indeed the actual type we want (we need Iterable and
Container properties; i.e. supports 'for' and 'in'). There's no reason
to require a Sequence (adds Reversible and some notion of a fixed
ordering) -- but it will fix the typing problems in 3.6, so I'm going to
do that.



You can create a "minimum requirements" venv to test this:

> cd ~/src/qemu/tests/qemu-iotests/
> sudo dnf install python36
> pipenv --python 3.6
> pipenv shell
> pipenv install astroid==2.1.0 pylint==2.2.0 mypy==0.620
> pylint iotests.py
> set -x MYPYPATH ~/src/qemu/python/
> mypy --ignore-missing-imports iotests.py


(You can drop the --ignore-missing-imports if you are using mypy >= 0.650.)


--js

Am 30.03.2020 um 21:03 hat John Snow geschrieben:
> 
> 
> On 3/24/20 7:20 PM, John Snow wrote:
> > This series uses python logging to enable output conditionally on
> > iotests.log(). We unify an initialization call (which also enables
> > debugging output for those tests with -d) and then make the switch
> > inside of iotests.
> > 
> > It will help alleviate the need to create logged/unlogged versions
> > of all the various helpers we have made.
> > 
> > Also, I got lost and accidentally delinted iotests while I was here.
> > Sorry about that. By version 9, it's now the overwhelming focus of
> > this series. No good deed, etc.
> 
> 
> Version requirements, as discovered by Kevin's Python Museum:
> 
> mypy >= 0.620
> pylint >= 2.2.0
> astroid == 2.1.0 (or >= 2.2.0 if using pylint >= 2.3.0)
> 
> 
> Hm, though ... pylint does not like 'Collection' very much:
> 
> iotests.py:1139:41: E1136: Value 'Collection' is unsubscriptable
> (unsubscriptable-object)
> 
> It works OK for the same pylint versions under 3.7, but it's busted a
> bit under 3.6. See https://github.com/PyCQA/pylint/issues/2377
> 
> Well. Collection is indeed the actual type we want (we need Iterable and
> Container properties; i.e. supports 'for' and 'in'). There's no reason
> to require a Sequence (adds Reversible and some notion of a fixed
> ordering) -- but it will fix the typing problems in 3.6, so I'm going to
> do that.

I wouldn't actually worry about Python museums much as far as pylint and
mypy are concerned. 3.6 compatibility is important for actually running
the code, but if older mypy/pylint versions get false positives, I would
consider that acceptable.

Kevin