[PATCH v2 00/16] python: add mypy support to python/qemu

John Snow posted 16 patches 3 years, 11 months ago
Test docker-mingw@fedora passed
Test checkpatch passed
Test asan passed
Test docker-quick@centos7 passed
Test FreeBSD passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200602214528.12107-1-jsnow@redhat.com
Maintainers: Max Reitz <mreitz@redhat.com>, Cleber Rosa <crosa@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>, Kevin Wolf <kwolf@redhat.com>
There is a newer version of this series
python/qemu/accel.py          |   8 +-
python/qemu/machine.py        | 267 ++++++++++++++++++++--------------
python/qemu/qmp.py            | 111 ++++++++++----
python/qemu/qtest.py          |  54 ++++---
scripts/render_block_graph.py |   7 +-
tests/qemu-iotests/iotests.py |  11 +-
6 files changed, 285 insertions(+), 173 deletions(-)
[PATCH v2 00/16] python: add mypy support to python/qemu
Posted by John Snow 3 years, 11 months ago
Requires: 20200602194844.15258-1-jsnow@redhat.com

This series is extracted from my larger series that attempted to bundle
our python module as an installable module. These fixes don't require that,
so they are being sent first as I think there's less up for debate in here.

This requires my "refactor shutdown" patch as a pre-requisite.

"v2":
- This version supports iotests 297
- Many patches merged by Phil are removed
- Replaces iotests.py type aliases with centralized ones
  (See patch 2)
- Imports etc are reworked to use the non-installable
  package layout instead. (Mostly important for patch 3)

Testing this out:
- You'll need Python3.6+
- I encourage you to use a virtual environment!
- You don't necessarily need these exact versions, but I didn't test the
  lower bounds, use older versions at your peril:
  - pylint==2.5.0
  - mypy=0.770
  - flake8=3.7.8

> cd ~/src/qemu/python/
> flake8 qemu
> mypy --strict qemu
> cd qemu
> pylint *.py

These should all 100% pass.

---

Open RFC: What's the right way to hook this into make check to prevent
regressions?

John Snow (16):
  python/qmp.py: Define common types
  iotests.py: use qemu.qmp type aliases
  python/qmp.py: re-absorb MonitorResponseError
  python/qmp.py: Do not return None from cmd_obj
  python/qmp.py: add casts to JSON deserialization
  python/qmp.py: add QMPProtocolError
  python/machine.py: Fix monitor address typing
  python/machine.py: reorder __init__
  python/machine.py: Don't modify state in _base_args()
  python/machine.py: Handle None events in event_wait
  python/machine.py: use qmp.command
  python/machine.py: Add _qmp access shim
  python/machine.py: fix _popen access
  python/qemu: make 'args' style arguments immutable
  iotests.py: Adjust HMP kwargs typing
  python/qemu: Add mypy type annotations

 python/qemu/accel.py          |   8 +-
 python/qemu/machine.py        | 267 ++++++++++++++++++++--------------
 python/qemu/qmp.py            | 111 ++++++++++----
 python/qemu/qtest.py          |  54 ++++---
 scripts/render_block_graph.py |   7 +-
 tests/qemu-iotests/iotests.py |  11 +-
 6 files changed, 285 insertions(+), 173 deletions(-)

-- 
2.21.3


Re: [PATCH v2 00/16] python: add mypy support to python/qemu
Posted by Eric Blake 3 years, 11 months ago
On 6/2/20 4:45 PM, John Snow wrote:
> Requires: 20200602194844.15258-1-jsnow@redhat.com

I don't know if patchew understand that, or if it requires:

Based-on: 20200602194844.15258-1-jsnow@redhat.com

> 
> This series is extracted from my larger series that attempted to bundle
> our python module as an installable module. These fixes don't require that,
> so they are being sent first as I think there's less up for debate in here.
> 
> This requires my "refactor shutdown" patch as a pre-requisite.
> 
> "v2":
> - This version supports iotests 297
> - Many patches merged by Phil are removed
> - Replaces iotests.py type aliases with centralized ones
>    (See patch 2)
> - Imports etc are reworked to use the non-installable
>    package layout instead. (Mostly important for patch 3)
> 
> Testing this out:
> - You'll need Python3.6+
> - I encourage you to use a virtual environment!
> - You don't necessarily need these exact versions, but I didn't test the
>    lower bounds, use older versions at your peril:
>    - pylint==2.5.0
>    - mypy=0.770
>    - flake8=3.7.8
> 
>> cd ~/src/qemu/python/
>> flake8 qemu
>> mypy --strict qemu
>> cd qemu
>> pylint *.py
> 
> These should all 100% pass.
> 
> ---
> 
> Open RFC: What's the right way to hook this into make check to prevent
> regressions?

We recently added iotest 297 in group meta; is there a way to run 
'./check -g meta' alongside the other iotests that 'make check' already 
triggers?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v2 00/16] python: add mypy support to python/qemu
Posted by John Snow 3 years, 11 months ago

On 6/2/20 5:51 PM, Eric Blake wrote:
> On 6/2/20 4:45 PM, John Snow wrote:
>> Requires: 20200602194844.15258-1-jsnow@redhat.com
> 
> I don't know if patchew understand that, or if it requires:
> 
> Based-on: 20200602194844.15258-1-jsnow@redhat.com
> 
>>
>> This series is extracted from my larger series that attempted to bundle
>> our python module as an installable module. These fixes don't require
>> that,
>> so they are being sent first as I think there's less up for debate in
>> here.
>>
>> This requires my "refactor shutdown" patch as a pre-requisite.
>>
>> "v2":
>> - This version supports iotests 297
>> - Many patches merged by Phil are removed
>> - Replaces iotests.py type aliases with centralized ones
>>    (See patch 2)
>> - Imports etc are reworked to use the non-installable
>>    package layout instead. (Mostly important for patch 3)
>>
>> Testing this out:
>> - You'll need Python3.6+
>> - I encourage you to use a virtual environment!
>> - You don't necessarily need these exact versions, but I didn't test the
>>    lower bounds, use older versions at your peril:
>>    - pylint==2.5.0
>>    - mypy=0.770
>>    - flake8=3.7.8
>>
>>> cd ~/src/qemu/python/
>>> flake8 qemu
>>> mypy --strict qemu
>>> cd qemu
>>> pylint *.py
>>
>> These should all 100% pass.
>>
>> ---
>>
>> Open RFC: What's the right way to hook this into make check to prevent
>> regressions?
> 
> We recently added iotest 297 in group meta; is there a way to run
> './check -g meta' alongside the other iotests that 'make check' already
> triggers?
> 

If we want to distribute any of this code independently of qemu.git (And
I think we do), I think relying on the iotests infrastructure will hurt
more than it will help.

I think I should try to maintain some independence of this folder from
the rest of the QEMU base; so it should be able to run the linting tests
under its own power.

(So, I guess, a Makefile?)

but there's further problems: this infrastructure is 3.6+ only, but the
build system only requires 3.5+. It has to be an optional testing target
that executes only when it's possible to. It also requires additional
dependencies not checked for in configure -- mypy, pylint, and flake8.

I am wondering if there's a nice way to create a check target that
builds a virtual environment with pinned dependencies, and then uses
that to run the lint tests. As long as the machine you're running on has
at least python3.6+ it should be able to run the tests.

I just don't really have a plan yet...

--js


Re: [PATCH v2 00/16] python: add mypy support to python/qemu
Posted by Kevin Wolf 3 years, 11 months ago
Am 02.06.2020 um 23:45 hat John Snow geschrieben:
> Requires: 20200602194844.15258-1-jsnow@redhat.com
> 
> This series is extracted from my larger series that attempted to bundle
> our python module as an installable module. These fixes don't require that,
> so they are being sent first as I think there's less up for debate in here.
> 
> This requires my "refactor shutdown" patch as a pre-requisite.

I had only minor comments, so with or without the suggested changes:

Reviewed-by: Kevin Wolf <kwolf@redhat.com>


Re: [PATCH v2 00/16] python: add mypy support to python/qemu
Posted by Vladimir Sementsov-Ogievskiy 3 years, 11 months ago
03.06.2020 00:45, John Snow wrote:
> Requires: 20200602194844.15258-1-jsnow@redhat.com

Hmm, somehow, I can't find it neither in https://lists.gnu.org/archive/html/qemu-devel/
nor in my thunderbird..

Could you post sequence of your series by subject, or export a git branch?


-- 
Best regards,
Vladimir

Re: [PATCH v2 00/16] python: add mypy support to python/qemu
Posted by John Snow 3 years, 11 months ago

On 6/3/20 4:24 AM, Vladimir Sementsov-Ogievskiy wrote:
> 03.06.2020 00:45, John Snow wrote:
>> Requires: 20200602194844.15258-1-jsnow@redhat.com
> 
> Hmm, somehow, I can't find it neither in
> https://lists.gnu.org/archive/html/qemu-devel/
> nor in my thunderbird..
> 
> Could you post sequence of your series by subject, or export a git branch?
> 
> 

1.
[PATCH v2 0/1] python/machine.py: refactor shutdown​
https://github.com/jnsnow/qemu/tree/python-package-reorder
(Patch that re-writes shutdown and kill)

2.
[PATCH v2 00/16] python: add mypy support to python/qemu​
https://github.com/jnsnow/qemu/tree/python-package-mypy
(This patchset.)

3.
[PATCH 0/7] python: create installable package
https://github.com/jnsnow/qemu/tree/python-package-refactor
(Python package series)