[PATCH 00/37] qapi: static typing conversion, pt1

John Snow posted 37 patches 2 weeks ago
Test docker-quick@centos7 failed
Test docker-mingw@fedora failed
Test checkpatch failed
Test FreeBSD failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200915224027.2529813-1-jsnow@redhat.com
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, Max Reitz <mreitz@redhat.com>, "Philippe Mathieu-Daudé" <philmd@redhat.com>, Kevin Wolf <kwolf@redhat.com>, Michael Roth <mdroth@linux.vnet.ibm.com>, Eduardo Habkost <ehabkost@redhat.com>, Markus Armbruster <armbru@redhat.com>, Fam Zheng <fam@euphon.net>, Cleber Rosa <crosa@redhat.com>, Peter Maydell <peter.maydell@linaro.org>
docs/conf.py                  |   4 +-
configure                     |   6 +-
.readthedocs.yml              |   2 +-
.travis.yml                   |   8 --
scripts/qapi-gen.py           |  59 ++--------
scripts/qapi/.flake8          |   2 +
scripts/qapi/commands.py      |  85 +++++++++++----
scripts/qapi/common.py        | 161 +++++++++++++++------------
scripts/qapi/debug.py         |  78 +++++++++++++
scripts/qapi/doc.py           |   3 +-
scripts/qapi/events.py        |  61 ++++++++---
scripts/qapi/expr.py          |   4 +-
scripts/qapi/gen.py           | 147 +++++++++++++------------
scripts/qapi/introspect.py    | 200 +++++++++++++++++++++++-----------
scripts/qapi/mypy.ini         |  30 +++++
scripts/qapi/params.py        |  40 +++++++
scripts/qapi/parser.py        |   4 +-
scripts/qapi/pylintrc         |  71 ++++++++++++
scripts/qapi/schema.py        |  33 +++---
scripts/qapi/script.py        |  91 ++++++++++++++++
scripts/qapi/source.py        |  34 +++---
scripts/qapi/types.py         | 125 ++++++++++++++-------
scripts/qapi/visit.py         | 104 +++++++++++++-----
tests/qemu-iotests/iotests.py |   2 -
24 files changed, 947 insertions(+), 407 deletions(-)
create mode 100644 scripts/qapi/.flake8
create mode 100644 scripts/qapi/debug.py
create mode 100644 scripts/qapi/mypy.ini
create mode 100644 scripts/qapi/params.py
create mode 100644 scripts/qapi/pylintrc
create mode 100644 scripts/qapi/script.py

[PATCH 00/37] qapi: static typing conversion, pt1

Posted by John Snow 2 weeks ago
Hi, this series starts adding static type hints to the QAPI module. As
you can see, the series started getting quite a bit long, so this is
only a partial conversion that focuses on a handful of the easier files.

The tougher files -- schema.py, expr.py, parser.py -- will each receive
their own series as a follow-up to this one.

Notes:

- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.

- Any patch named "add notational type hints" changes ONLY signatures,
  which have no runtime impact whatsoever. These are big patches,
  but fairly straightforward.

- Most other patches are as bite-sized as possible, generally fixing one
  single warning.

- After patch 6, `flake8 qapi/` should pass 100% on this and every
  future commit.

- After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
  on this and every future commit.

- After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
  100% on this and every future commit.

Preliminary refactoring and prerequisites:

001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools'
003/37: 'qapi-gen: Separate arg-parsing from generation'
004/37: 'qapi: move generator entrypoint into module'
005/37: 'qapi: Remove wildcard includes'
006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'

common.py (and params.py):

008/37: 'qapi/common.py: Remove python compatibility workaround'
009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable'
012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints'
014/37: 'qapi/common.py: Move comments into docstrings'
015/37: 'qapi/common.py: split build_params into new file'
016/37: 'qapi: establish mypy type-checking baseline'

events.py:

017/37: 'qapi/events.py: add notational type hints'
018/37: 'qapi/events.py: Move comments into docstrings'

commands.py:

019/37: 'qapi/commands.py: Don't re-bind to variable of different type'
020/37: 'qapi/commands.py: add notational type hints'
021/37: 'qapi/commands.py: enable checking with mypy'

source.py:

022/37: 'qapi/source.py: add notational type hints'
023/37: 'qapi/source.py: delint with pylint'

gen.py:

024/37: 'qapi/gen.py: Fix edge-case of _is_user_module'
025/37: 'qapi/gen.py: add notational type hints'
026/37: 'qapi/gen.py: Enable checking with mypy'
027/37: 'qapi/gen.py: Remove unused parameter'
028/37: 'qapi/gen.py: update write() to be more idiomatic'
029/37: 'qapi/gen.py: delint with pylint'

introspect.py:

030/37: 'qapi/introspect.py: Add a typed 'extra' structure'
031/37: 'qapi/introspect.py: add _gen_features helper'
032/37: 'qapi/introspect.py: create a typed 'Node' data structure'
033/37: 'qapi/introspect.py: add notational type hints'

types.py:

034/37: 'qapi/types.py: add notational type hints'
035/37: 'qapi/types.py: remove one-letter variables'

visit.py:

036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object'
037/37: 'qapi/visit.py: add notational type hints'

John Snow (37):
  python: Require 3.6+
  [DO-NOT-MERGE] qapi: add debugging tools
  qapi-gen: Separate arg-parsing from generation
  qapi: move generator entrypoint into module
  qapi: Remove wildcard includes
  qapi: delint using flake8
  qapi: add pylintrc
  qapi/common.py: Remove python compatibility workaround
  qapi/common.py: Add indent manager
  qapi/common.py: delint with pylint
  qapi/common.py: Replace one-letter 'c' variable
  qapi/common.py: check with pylint
  qapi/common.py: add notational type hints
  qapi/common.py: Move comments into docstrings
  qapi/common.py: split build_params into new file
  qapi: establish mypy type-checking baseline
  qapi/events.py: add notational type hints
  qapi/events.py: Move comments into docstrings
  qapi/commands.py: Don't re-bind to variable of different type
  qapi/commands.py: add notational type hints
  qapi/commands.py: enable checking with mypy
  qapi/source.py: add notational type hints
  qapi/source.py: delint with pylint
  qapi/gen.py: Fix edge-case of _is_user_module
  qapi/gen.py: add notational type hints
  qapi/gen.py: Enable checking with mypy
  qapi/gen.py: Remove unused parameter
  qapi/gen.py: update write() to be more idiomatic
  qapi/gen.py: delint with pylint
  qapi/introspect.py: Add a typed 'extra' structure
  qapi/introspect.py: add _gen_features helper
  qapi/introspect.py: create a typed 'Node' data structure
  qapi/introspect.py: add notational type hints
  qapi/types.py: add notational type hints
  qapi/types.py: remove one-letter variables
  qapi/visit.py: remove unused parameters from gen_visit_object
  qapi/visit.py: add notational type hints

 docs/conf.py                  |   4 +-
 configure                     |   6 +-
 .readthedocs.yml              |   2 +-
 .travis.yml                   |   8 --
 scripts/qapi-gen.py           |  59 ++--------
 scripts/qapi/.flake8          |   2 +
 scripts/qapi/commands.py      |  85 +++++++++++----
 scripts/qapi/common.py        | 161 +++++++++++++++------------
 scripts/qapi/debug.py         |  78 +++++++++++++
 scripts/qapi/doc.py           |   3 +-
 scripts/qapi/events.py        |  61 ++++++++---
 scripts/qapi/expr.py          |   4 +-
 scripts/qapi/gen.py           | 147 +++++++++++++------------
 scripts/qapi/introspect.py    | 200 +++++++++++++++++++++++-----------
 scripts/qapi/mypy.ini         |  30 +++++
 scripts/qapi/params.py        |  40 +++++++
 scripts/qapi/parser.py        |   4 +-
 scripts/qapi/pylintrc         |  71 ++++++++++++
 scripts/qapi/schema.py        |  33 +++---
 scripts/qapi/script.py        |  91 ++++++++++++++++
 scripts/qapi/source.py        |  34 +++---
 scripts/qapi/types.py         | 125 ++++++++++++++-------
 scripts/qapi/visit.py         | 104 +++++++++++++-----
 tests/qemu-iotests/iotests.py |   2 -
 24 files changed, 947 insertions(+), 407 deletions(-)
 create mode 100644 scripts/qapi/.flake8
 create mode 100644 scripts/qapi/debug.py
 create mode 100644 scripts/qapi/mypy.ini
 create mode 100644 scripts/qapi/params.py
 create mode 100644 scripts/qapi/pylintrc
 create mode 100644 scripts/qapi/script.py

-- 
2.26.2



Re: [PATCH 00/37] qapi: static typing conversion, pt1

Posted by John Snow 1 week ago
On 9/15/20 6:39 PM, John Snow wrote:
> Hi, this series starts adding static type hints to the QAPI module. As
> you can see, the series started getting quite a bit long, so this is
> only a partial conversion that focuses on a handful of the easier files.
> 
> The tougher files -- schema.py, expr.py, parser.py -- will each receive
> their own series as a follow-up to this one.
> 
> Notes:
> 
> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
> 
> - Any patch named "add notational type hints" changes ONLY signatures,
>    which have no runtime impact whatsoever. These are big patches,
>    but fairly straightforward.
> 
> - Most other patches are as bite-sized as possible, generally fixing one
>    single warning.
> 
> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>    future commit.
> 
> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>    on this and every future commit.
> 
> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>    100% on this and every future commit.
> 

Addressed today's feedback and pushed to gitlab.
Let's call it "Version 1.3".

https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/

Waiting on feedback for 16-37 before re-authoring to list.

--js


Re: [PATCH 00/37] qapi: static typing conversion, pt1

Posted by Markus Armbruster 1 week ago
John Snow <jsnow@redhat.com> writes:

> On 9/15/20 6:39 PM, John Snow wrote:
>> Hi, this series starts adding static type hints to the QAPI module. As
>> you can see, the series started getting quite a bit long, so this is
>> only a partial conversion that focuses on a handful of the easier files.
>> The tougher files -- schema.py, expr.py, parser.py -- will each
>> receive
>> their own series as a follow-up to this one.
>> Notes:
>> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
>> - Any patch named "add notational type hints" changes ONLY
>> signatures,
>>    which have no runtime impact whatsoever. These are big patches,
>>    but fairly straightforward.
>> - Most other patches are as bite-sized as possible, generally fixing
>> one
>>    single warning.
>> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>>    future commit.
>> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass
>> 100%
>>    on this and every future commit.
>> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should
>> pass
>>    100% on this and every future commit.
>> 
>
> Addressed today's feedback and pushed to gitlab.
> Let's call it "Version 1.3".
>
> https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/
>
> Waiting on feedback for 16-37 before re-authoring to list.

If 03-12 don't actually depend on Python 3.6, you may want to respin
these for me to queue right away.


Re: [PATCH 00/37] qapi: static typing conversion, pt1

Posted by John Snow 2 weeks ago
On 9/15/20 6:39 PM, John Snow wrote:
> Hi, this series starts adding static type hints to the QAPI module. As
> you can see, the series started getting quite a bit long, so this is
> only a partial conversion that focuses on a handful of the easier files.
> 

Updated series with current feedback and pushed to gitlab. Awaiting 
feedback on patches 10-37 before re-spinning.

https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/

--js


Re: [PATCH 00/37] qapi: static typing conversion, pt1

Posted by Philippe Mathieu-Daudé 1 week ago
Cc'ing Ben who offered to help with having the QEMU python
scripts packaged (and this series is a step toward that).

On 9/16/20 12:39 AM, John Snow wrote:
> Hi, this series starts adding static type hints to the QAPI module. As
> you can see, the series started getting quite a bit long, so this is
> only a partial conversion that focuses on a handful of the easier files.
> 
> The tougher files -- schema.py, expr.py, parser.py -- will each receive
> their own series as a follow-up to this one.
> 
> Notes:
> 
> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
> 
> - Any patch named "add notational type hints" changes ONLY signatures,
>   which have no runtime impact whatsoever. These are big patches,
>   but fairly straightforward.
> 
> - Most other patches are as bite-sized as possible, generally fixing one
>   single warning.
> 
> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>   future commit.
> 
> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>   on this and every future commit.
> 
> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>   100% on this and every future commit.
> 
> Preliminary refactoring and prerequisites:
> 
> 001/37: 'python: Require 3.6+'
> 002/37: '[DO-NOT-MERGE] qapi: add debugging tools'
> 003/37: 'qapi-gen: Separate arg-parsing from generation'
> 004/37: 'qapi: move generator entrypoint into module'
> 005/37: 'qapi: Remove wildcard includes'
> 006/37: 'qapi: delint using flake8'
> 007/37: 'qapi: add pylintrc'
> 
> common.py (and params.py):
> 
> 008/37: 'qapi/common.py: Remove python compatibility workaround'
> 009/37: 'qapi/common.py: Add indent manager'
> 010/37: 'qapi/common.py: delint with pylint'
> 011/37: 'qapi/common.py: Replace one-letter 'c' variable'
> 012/37: 'qapi/common.py: check with pylint'
> 013/37: 'qapi/common.py: add notational type hints'
> 014/37: 'qapi/common.py: Move comments into docstrings'
> 015/37: 'qapi/common.py: split build_params into new file'
> 016/37: 'qapi: establish mypy type-checking baseline'
> 
> events.py:
> 
> 017/37: 'qapi/events.py: add notational type hints'
> 018/37: 'qapi/events.py: Move comments into docstrings'
> 
> commands.py:
> 
> 019/37: 'qapi/commands.py: Don't re-bind to variable of different type'
> 020/37: 'qapi/commands.py: add notational type hints'
> 021/37: 'qapi/commands.py: enable checking with mypy'
> 
> source.py:
> 
> 022/37: 'qapi/source.py: add notational type hints'
> 023/37: 'qapi/source.py: delint with pylint'
> 
> gen.py:
> 
> 024/37: 'qapi/gen.py: Fix edge-case of _is_user_module'
> 025/37: 'qapi/gen.py: add notational type hints'
> 026/37: 'qapi/gen.py: Enable checking with mypy'
> 027/37: 'qapi/gen.py: Remove unused parameter'
> 028/37: 'qapi/gen.py: update write() to be more idiomatic'
> 029/37: 'qapi/gen.py: delint with pylint'
> 
> introspect.py:
> 
> 030/37: 'qapi/introspect.py: Add a typed 'extra' structure'
> 031/37: 'qapi/introspect.py: add _gen_features helper'
> 032/37: 'qapi/introspect.py: create a typed 'Node' data structure'
> 033/37: 'qapi/introspect.py: add notational type hints'
> 
> types.py:
> 
> 034/37: 'qapi/types.py: add notational type hints'
> 035/37: 'qapi/types.py: remove one-letter variables'
> 
> visit.py:
> 
> 036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object'
> 037/37: 'qapi/visit.py: add notational type hints'


Re: [PATCH 00/37] qapi: static typing conversion, pt1

Posted by John Snow 1 week ago
On 9/18/20 9:07 AM, Philippe Mathieu-Daudé wrote:
> Cc'ing Ben who offered to help with having the QEMU python
> scripts packaged (and this series is a step toward that).
> 

FWIW, I want to move ./python/qemu/ into ./python/qemu/core, and then 
move ./scripts/qapi to ./python/qemu/qapi.

I will leave the qapi-gen.py stub script behind for compatibility.

Once I've done that, I can add package setup scripts.

I have a long series adding mypy type hints and so forth to 
./python/qemu and adding packaging glue there too, but I shelved it as 
5.1 released and right now I am focusing on QAPI.

> On 9/16/20 12:39 AM, John Snow wrote:
>> Hi, this series starts adding static type hints to the QAPI module. As
>> you can see, the series started getting quite a bit long, so this is
>> only a partial conversion that focuses on a handful of the easier files.
>>
>> The tougher files -- schema.py, expr.py, parser.py -- will each receive
>> their own series as a follow-up to this one.
>>
>> Notes:
>>
>> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
>>
>> - Any patch named "add notational type hints" changes ONLY signatures,
>>    which have no runtime impact whatsoever. These are big patches,
>>    but fairly straightforward.
>>
>> - Most other patches are as bite-sized as possible, generally fixing one
>>    single warning.
>>
>> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>>    future commit.
>>
>> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>>    on this and every future commit.
>>
>> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>>    100% on this and every future commit.
>>
>> Preliminary refactoring and prerequisites:
>>
>> 001/37: 'python: Require 3.6+'
>> 002/37: '[DO-NOT-MERGE] qapi: add debugging tools'
>> 003/37: 'qapi-gen: Separate arg-parsing from generation'
>> 004/37: 'qapi: move generator entrypoint into module'
>> 005/37: 'qapi: Remove wildcard includes'
>> 006/37: 'qapi: delint using flake8'
>> 007/37: 'qapi: add pylintrc'
>>
>> common.py (and params.py):
>>
>> 008/37: 'qapi/common.py: Remove python compatibility workaround'
>> 009/37: 'qapi/common.py: Add indent manager'
>> 010/37: 'qapi/common.py: delint with pylint'
>> 011/37: 'qapi/common.py: Replace one-letter 'c' variable'
>> 012/37: 'qapi/common.py: check with pylint'
>> 013/37: 'qapi/common.py: add notational type hints'
>> 014/37: 'qapi/common.py: Move comments into docstrings'
>> 015/37: 'qapi/common.py: split build_params into new file'
>> 016/37: 'qapi: establish mypy type-checking baseline'
>>
>> events.py:
>>
>> 017/37: 'qapi/events.py: add notational type hints'
>> 018/37: 'qapi/events.py: Move comments into docstrings'
>>
>> commands.py:
>>
>> 019/37: 'qapi/commands.py: Don't re-bind to variable of different type'
>> 020/37: 'qapi/commands.py: add notational type hints'
>> 021/37: 'qapi/commands.py: enable checking with mypy'
>>
>> source.py:
>>
>> 022/37: 'qapi/source.py: add notational type hints'
>> 023/37: 'qapi/source.py: delint with pylint'
>>
>> gen.py:
>>
>> 024/37: 'qapi/gen.py: Fix edge-case of _is_user_module'
>> 025/37: 'qapi/gen.py: add notational type hints'
>> 026/37: 'qapi/gen.py: Enable checking with mypy'
>> 027/37: 'qapi/gen.py: Remove unused parameter'
>> 028/37: 'qapi/gen.py: update write() to be more idiomatic'
>> 029/37: 'qapi/gen.py: delint with pylint'
>>
>> introspect.py:
>>
>> 030/37: 'qapi/introspect.py: Add a typed 'extra' structure'
>> 031/37: 'qapi/introspect.py: add _gen_features helper'
>> 032/37: 'qapi/introspect.py: create a typed 'Node' data structure'
>> 033/37: 'qapi/introspect.py: add notational type hints'
>>
>> types.py:
>>
>> 034/37: 'qapi/types.py: add notational type hints'
>> 035/37: 'qapi/types.py: remove one-letter variables'
>>
>> visit.py:
>>
>> 036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object'
>> 037/37: 'qapi/visit.py: add notational type hints'
>