Hi,

With this series we can now write tests using Python rather than C.
For complex tests this can reduce the test development time, we can focus on
the test purposes instead of his implementation details.

- 1,2: we already have Python classes to run Block tests, move all the
  non Block-related methods to qtest.py,
- 3: default TEST_DIR to TMPDIR,
- 4: add a method to restrict tests to a list of supported machines,
- 5: since the Block tests are output sensitive, do not mess with their
  current tuned iotests::main(unittest), add a more generic one in qtest.py,
- 6: finally modify the tests Makefile to run C/Python tests with the same
  rule.

to have a better idea, here is a snippet from the next series:

    class TestSdCardSpecV2(qtest.QMPTestCase):
        [...]
        def test_power_on_spec_v2(self):
            self.bus.do_cmd(GO_IDLE_STATE)
            [...]
            # verify Card ID
            data = self.bus.do_cmd(ALL_SEND_CID)
            oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
            self.assertEqual(oid, "XY") # QEMU default
            self.assertEqual(pnm, "QEMU!") # QEMU default
            self.assertEqual(psn, 0xdeadbeef) # QEMU default

The test suite is ran the same way:

$ make check-qtest-aarch64 
  GTESTER check-qtest-aarch64
  PYTEST  check-qtest-aarch64

Adding the verbose flag we see the SD bus commands:

$ make check-qtest-aarch64 V=1
TEST: tests/test-hmp... (pid=16074)
...
  /aarch64/hmp/xlnx-zcu102:                                            OK
  /aarch64/hmp/lm3s811evb:                                             OK
  /aarch64/hmp/none+2MB:                                               OK
PASS: tests/test-hmp
PYTHONPATH=/source/qemu/scripts QEMU_PROG=aarch64-softmmu/qemu-system-aarch64 TEST_DIR=/tmp python -B /source/qemu/tests/sdcard_tests.py -v
INFO:root:CMD#00 -> (none)
INFO:root:CMD#08 -> 00000100
INFO:root:CMD#08 -> 00000200
INFO:root:CMD#08 -> 00000400
INFO:root:CMD#08 -> 00000800
INFO:root:CMD#08 -> 00001000
INFO:root:CMD#08 -> 00002000
INFO:root:CMD#55 -> 00000120
INFO:root:CMD#41 -> 00ffff00
INFO:root:CMD#55 -> 00000120
INFO:root:CMD#41 -> 80ffff00
INFO:root:CMD#02 -> aa585951454d552101deadbeef006219
INFO:root:CMD#03 -> 45670500
INFO:root:CMD#03 -> 8ace0700
.
----------------------------------------------------------------------
Ran 1 test in 0.070s

OK

Regards,

Phil.

Based-on: 20171213204436.5379-12-f4bug@amsat.org
          (QOM'ify SDBus, housekeeping)

Philippe Mathieu-Daudé (6):
  iotests.py: split BlockQMPTestCase class of QMPTestCase
  iotests.py: move the generic QMPTestCase to qtest.py
  qtest.py: use TMPDIR/TEMP if the TEST_DIR env var is missing
  qtest.py: add verify_machine(supported_machines)
  qtest.py: add a simple main() which calls unittest.main()
  tests: add a Makefile rule to run Python qtests

 scripts/qtest.py              | 152 ++++++++++++++++++++++++++++++++++++++++++
 tests/Makefile.include        |  11 ++-
 tests/qemu-iotests/030        |  14 ++--
 tests/qemu-iotests/040        |   2 +-
 tests/qemu-iotests/041        |  20 +++---
 tests/qemu-iotests/044        |   2 +-
 tests/qemu-iotests/045        |   4 +-
 tests/qemu-iotests/055        |   8 +--
 tests/qemu-iotests/056        |   4 +-
 tests/qemu-iotests/057        |   2 +-
 tests/qemu-iotests/065        |   2 +-
 tests/qemu-iotests/093        |   6 +-
 tests/qemu-iotests/096        |   2 +-
 tests/qemu-iotests/118        |   2 +-
 tests/qemu-iotests/124        |   2 +-
 tests/qemu-iotests/129        |   2 +-
 tests/qemu-iotests/132        |   2 +-
 tests/qemu-iotests/136        |   2 +-
 tests/qemu-iotests/139        |   2 +-
 tests/qemu-iotests/147        |   2 +-
 tests/qemu-iotests/148        |   2 +-
 tests/qemu-iotests/152        |   2 +-
 tests/qemu-iotests/155        |   2 +-
 tests/qemu-iotests/163        |   2 +-
 tests/qemu-iotests/165        |   2 +-
 tests/qemu-iotests/196        |   2 +-
 tests/qemu-iotests/iotests.py |  84 ++---------------------
 27 files changed, 212 insertions(+), 127 deletions(-)

-- 
2.15.1

On Wed, Dec 13, 2017 at 06:35:51PM -0300, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> With this series we can now write tests using Python rather than C.
> For complex tests this can reduce the test development time, we can focus on
> the test purposes instead of his implementation details.
> 
> - 1,2: we already have Python classes to run Block tests, move all the
>   non Block-related methods to qtest.py,
> - 3: default TEST_DIR to TMPDIR,
> - 4: add a method to restrict tests to a list of supported machines,
> - 5: since the Block tests are output sensitive, do not mess with their
>   current tuned iotests::main(unittest), add a more generic one in qtest.py,
> - 6: finally modify the tests Makefile to run C/Python tests with the same
>   rule.

Python tests are great for functional tests of qemu-system-* that
interact via the command-line and QMP monitor.  From this perspective I
think the series is good.

> to have a better idea, here is a snippet from the next series:
> 
>     class TestSdCardSpecV2(qtest.QMPTestCase):
>         [...]
>         def test_power_on_spec_v2(self):
>             self.bus.do_cmd(GO_IDLE_STATE)
>             [...]
>             # verify Card ID
>             data = self.bus.do_cmd(ALL_SEND_CID)
>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>             self.assertEqual(oid, "XY") # QEMU default
>             self.assertEqual(pnm, "QEMU!") # QEMU default
>             self.assertEqual(psn, 0xdeadbeef) # QEMU default

Device qtests are better done in C than Python.  Python is not good at
binary I/O and porting this to Python 3 will be extra work later (Python
2 is set for End-of-Life in 2020, see https://pythonclock.org/).

More importantly, we already have libqos in C with a guest memory
allocator, PCI, and virtio support.  Fragmenting the small amount effort
that goes into device testing will delay libqos reaching critical mass.
Critical mass is where libqos provides all the infrastructure you need
to set up a device and focus on your actual test instead of machine,
bus, or device initialization.  Starting a Python device testing effort
will just lead to duplication and 2 underdeveloped device testing
frameworks.

Is there a specific reason why adding SD Card support to libqos is not
possible in C?

On 14/12/2017 10:39, Stefan Hajnoczi wrote:
>>             # verify Card ID
>>             data = self.bus.do_cmd(ALL_SEND_CID)
>>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>>             self.assertEqual(oid, "XY") # QEMU default
>>             self.assertEqual(pnm, "QEMU!") # QEMU default
>>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
> Device qtests are better done in C than Python.  Python is not good at
> binary I/O and porting this to Python 3 will be extra work later (Python
> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> 
> More importantly, we already have libqos in C with a guest memory
> allocator, PCI, and virtio support.  Fragmenting the small amount effort
> that goes into device testing will delay libqos reaching critical mass.
> Critical mass is where libqos provides all the infrastructure you need
> to set up a device and focus on your actual test instead of machine,
> bus, or device initialization.  Starting a Python device testing effort
> will just lead to duplication and 2 underdeveloped device testing
> frameworks.

I agree that fragmentation is bad.  However, libqos is small (about 4k
lines of code, maybe 3k in Python).

I also agree that any qtest written in Python should be written in
Python 3 from the beginning (in fact we should consider dropping Python
2.x support in 2.12).  Doing so should not make binary I/O much
different than C.

From my point of view, the main advantage of Python is the reflection
mechanisms.  Those would make it possible to automatically discover the
set of runnable tests; for example, based on:

- two machine descriptions (including malloc) for ARM -M virt and x86 -M q35

- a driver for the generic PCIe host bridge

- a driver for virtio-pci

- a driver for virtio-scsi

- a set of SCSI tests

... it should be possible to run the same tests as either ARM or x86
qtests.  This would be a very different framework compared to the C libqos.

Thanks,

Paolo

> Is there a specific reason why adding SD Card support to libqos is not
> possible in C?

On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 14/12/2017 10:39, Stefan Hajnoczi wrote:
> >>             # verify Card ID
> >>             data = self.bus.do_cmd(ALL_SEND_CID)
> >>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
> >>             self.assertEqual(oid, "XY") # QEMU default
> >>             self.assertEqual(pnm, "QEMU!") # QEMU default
> >>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
> > Device qtests are better done in C than Python.  Python is not good at
> > binary I/O and porting this to Python 3 will be extra work later (Python
> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> >
> > More importantly, we already have libqos in C with a guest memory
> > allocator, PCI, and virtio support.  Fragmenting the small amount effort
> > that goes into device testing will delay libqos reaching critical mass.
> > Critical mass is where libqos provides all the infrastructure you need
> > to set up a device and focus on your actual test instead of machine,
> > bus, or device initialization.  Starting a Python device testing effort
> > will just lead to duplication and 2 underdeveloped device testing
> > frameworks.
>
> I agree that fragmentation is bad.  However, libqos is small (about 4k
> lines of code, maybe 3k in Python).
>
> I also agree that any qtest written in Python should be written in
> Python 3 from the beginning (in fact we should consider dropping Python
> 2.x support in 2.12).  Doing so should not make binary I/O much
> different than C.
>

Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
which do not provide python 3 yet.

Doing binary io in python 2 and 3 is the same, there is no need to use
python 3 for this. The only advantage is not having to backport code
later from 2 to 3.

Nir


>
> From my point of view, the main advantage of Python is the reflection
> mechanisms.  Those would make it possible to automatically discover the
> set of runnable tests; for example, based on:
>
> - two machine descriptions (including malloc) for ARM -M virt and x86 -M
> q35
>
> - a driver for the generic PCIe host bridge
>
> - a driver for virtio-pci
>
> - a driver for virtio-scsi
>
> - a set of SCSI tests
>
> ... it should be possible to run the same tests as either ARM or x86
> qtests.  This would be a very different framework compared to the C libqos.
>
> Thanks,
>
> Paolo
>
> > Is there a specific reason why adding SD Card support to libqos is not
> > possible in C?
>
>
>

Hi Nir,

On 12/14/2017 12:39 PM, Nir Soffer wrote:
> On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com
[...]
>     > Device qtests are better done in C than Python.  Python is not good at
>     > binary I/O and porting this to Python 3 will be extra work later
>     (Python
>     > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>     >
>     > More importantly, we already have libqos in C with a guest memory
>     > allocator, PCI, and virtio support.  Fragmenting the small amount
>     effort
>     > that goes into device testing will delay libqos reaching critical
>     mass.
>     > Critical mass is where libqos provides all the infrastructure you need
>     > to set up a device and focus on your actual test instead of machine,
>     > bus, or device initialization.  Starting a Python device testing
>     effort
>     > will just lead to duplication and 2 underdeveloped device testing
>     > frameworks.
> 
>     I agree that fragmentation is bad.  However, libqos is small (about 4k
>     lines of code, maybe 3k in Python).
> 
>     I also agree that any qtest written in Python should be written in
>     Python 3 from the beginning (in fact we should consider dropping Python
>     2.x support in 2.12).  Doing so should not make binary I/O much
>     different than C.
> 
> 
> Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
> which do not provide python 3 yet.

Restricting it to python 2 also make it hard to develop and test QEMU on
Arch Linux which default come with python 3 ;)

> Doing binary io in python 2 and 3 is the same, there is no need to use
> python 3 for this. The only advantage is not having to backport code
> later from 2 to 3.
> 
> Nir

Nir Soffer <nirsof@gmail.com> writes:

> On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> On 14/12/2017 10:39, Stefan Hajnoczi wrote:
>> >>             # verify Card ID
>> >>             data = self.bus.do_cmd(ALL_SEND_CID)
>> >>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>> >>             self.assertEqual(oid, "XY") # QEMU default
>> >>             self.assertEqual(pnm, "QEMU!") # QEMU default
>> >>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
>> > Device qtests are better done in C than Python.  Python is not good at
>> > binary I/O and porting this to Python 3 will be extra work later (Python
>> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>> >
>> > More importantly, we already have libqos in C with a guest memory
>> > allocator, PCI, and virtio support.  Fragmenting the small amount effort
>> > that goes into device testing will delay libqos reaching critical mass.
>> > Critical mass is where libqos provides all the infrastructure you need
>> > to set up a device and focus on your actual test instead of machine,
>> > bus, or device initialization.  Starting a Python device testing effort
>> > will just lead to duplication and 2 underdeveloped device testing
>> > frameworks.
>>
>> I agree that fragmentation is bad.  However, libqos is small (about 4k
>> lines of code, maybe 3k in Python).
>>
>> I also agree that any qtest written in Python should be written in
>> Python 3 from the beginning (in fact we should consider dropping Python
>> 2.x support in 2.12).  Doing so should not make binary I/O much
>> different than C.
>>
>
> Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
> which do not provide python 3 yet.

s/hard/slightly inconvenient/: Python 3 is available in EPEL.  

> Doing binary io in python 2 and 3 is the same, there is no need to use
> python 3 for this. The only advantage is not having to backport code
> later from 2 to 3.

On 14/12/2017 17:05, Markus Armbruster wrote:
> Nir Soffer <nirsof@gmail.com> writes:
> 
>> On Thu, Dec 14, 2017 at 1:36 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>> On 14/12/2017 10:39, Stefan Hajnoczi wrote:
>>>>>             # verify Card ID
>>>>>             data = self.bus.do_cmd(ALL_SEND_CID)
>>>>>             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
>>>>>             self.assertEqual(oid, "XY") # QEMU default
>>>>>             self.assertEqual(pnm, "QEMU!") # QEMU default
>>>>>             self.assertEqual(psn, 0xdeadbeef) # QEMU default
>>>> Device qtests are better done in C than Python.  Python is not good at
>>>> binary I/O and porting this to Python 3 will be extra work later (Python
>>>> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>>>>
>>>> More importantly, we already have libqos in C with a guest memory
>>>> allocator, PCI, and virtio support.  Fragmenting the small amount effort
>>>> that goes into device testing will delay libqos reaching critical mass.
>>>> Critical mass is where libqos provides all the infrastructure you need
>>>> to set up a device and focus on your actual test instead of machine,
>>>> bus, or device initialization.  Starting a Python device testing effort
>>>> will just lead to duplication and 2 underdeveloped device testing
>>>> frameworks.
>>>
>>> I agree that fragmentation is bad.  However, libqos is small (about 4k
>>> lines of code, maybe 3k in Python).
>>>
>>> I also agree that any qtest written in Python should be written in
>>> Python 3 from the beginning (in fact we should consider dropping Python
>>> 2.x support in 2.12).  Doing so should not make binary I/O much
>>> different than C.
>>
>> Using python 3 will make it hard to develop and test qemu on RHEL/CentOS
>> which do not provide python 3 yet.
> 
> s/hard/slightly inconvenient/: Python 3 is available in EPEL.  

And also in software collections.  For 2.12 (as part of the above "drop
Python 2.x support" plan) my idea is to set up a virtual Python
environment as part of configure, so that you can just do

	scl enable rh-python36 ./configure
	make

if you don't want to use EPEL.

Paolo

>> Doing binary io in python 2 and 3 is the same, there is no need to use
>> python 3 for this. The only advantage is not having to backport code
>> later from 2 to 3.

On 14 December 2017 at 17:33, Paolo Bonzini <pbonzini@redhat.com> wrote:
> And also in software collections.  For 2.12 (as part of the above "drop
> Python 2.x support" plan)

I'm reluctant to require a non-system python on OSX to build.

thanks
-- PMM

On 14/12/2017 20:08, Peter Maydell wrote:
> On 14 December 2017 at 17:33, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> And also in software collections.  For 2.12 (as part of the above "drop
>> Python 2.x support" plan)
> 
> I'm reluctant to require a non-system python on OSX to build.

Oh well.  I guess I can post the infrastructure to simplify using a
non-system Python, without actually requiring it.

Paolo

Hi Stefan,

On 12/14/2017 06:39 AM, Stefan Hajnoczi wrote:
[...]
> Device qtests are better done in C than Python.  Python is not good at
> binary I/O and porting this to Python 3 will be extra work later (Python
> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> 
> More importantly, we already have libqos in C with a guest memory
> allocator, PCI, and virtio support.  Fragmenting the small amount effort
> that goes into device testing will delay libqos reaching critical mass.
> Critical mass is where libqos provides all the infrastructure you need
> to set up a device and focus on your actual test instead of machine,
> bus, or device initialization.  Starting a Python device testing effort
> will just lead to duplication and 2 underdeveloped device testing
> frameworks.
> 
> Is there a specific reason why adding SD Card support to libqos is not
> possible in C?

Short (joking) answer: Would you write tests/qemu-iotests/041 in C? ;)

Now thinking about the specific reasons...

What I intend to do is add qtests for the SDHCI implementation.
There are different revisions of the standard specs.

Currently QEMU implements a mixed subset of v1.10 and v2.00, while the
real SDHCI hardware implemented is v2.00 or v3.01. I also saw 2 series
adding SDHCI devices supporting the v4.00 (not yet in /master).

With the current codebase we can plug SDSC/SDHC cards (up to 32 GB) but
we can not plug a SDXC card, so:
- plugging a 64GB card into the ZynqMP machine fails,
- plugging a 64GB card into the SABRE Lite machine fails,
- booting a Win10 guest on a exFAT formatted SD card also fails.

A later test create different cards with [2GB, 16GB, 64GB] and test if
the SDHCI correctly access them.
Machines using the Exynos4210 SoC which only support the v2.00 spec are
expected to fail with 64GB SDXC cards.

The Zynq machine has 2 SD buses, another later test unplug/replug the
same card between the buses. This test is expected to be useful to test
the custom SDHOST controller from the BCM2835 SoC (Raspi2 machine).

It is surely possible to write all those tests in C, but it resulted way
faster to me to write them in Python...

I could think of a bunch of I2C tests too, more generally I found using
Python for bus backend testing is much easier, as with the Block backends.

Regards,

Phil.

On Thu, Dec 14, 2017 at 11:33:03AM -0300, Philippe Mathieu-Daudé wrote:
> Hi Stefan,
> 
> On 12/14/2017 06:39 AM, Stefan Hajnoczi wrote:
> [...]
> > Device qtests are better done in C than Python.  Python is not good at
> > binary I/O and porting this to Python 3 will be extra work later (Python
> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> > 
> > More importantly, we already have libqos in C with a guest memory
> > allocator, PCI, and virtio support.  Fragmenting the small amount effort
> > that goes into device testing will delay libqos reaching critical mass.
> > Critical mass is where libqos provides all the infrastructure you need
> > to set up a device and focus on your actual test instead of machine,
> > bus, or device initialization.  Starting a Python device testing effort
> > will just lead to duplication and 2 underdeveloped device testing
> > frameworks.
> > 
> > Is there a specific reason why adding SD Card support to libqos is not
> > possible in C?
> 
> Short (joking) answer: Would you write tests/qemu-iotests/041 in C? ;)

041 is not a device-level test.  It doesn't poke device registers, it's
a functional test.  In my email I said I support using Python for those
types of tests.

> Now thinking about the specific reasons...
> 
> What I intend to do is add qtests for the SDHCI implementation.
> There are different revisions of the standard specs.

It would be great to have SDHCI support in libqos.  PCI and virtio are
covered today and support for more busses will help reach that critical
mass where tests can be written for most QEMU device models without
first writing a new device driver framework.

Please post your SDHCI test code, so we can discuss the details and
consider whether it's more like 041 or more like tests/ide-test.c.

Stefan

>> > Is there a specific reason why adding SD Card support to libqos is not
>> > possible in C?
>>
>> Short (joking) answer: Would you write tests/qemu-iotests/041 in C? ;)
>
> 041 is not a device-level test.  It doesn't poke device registers, it's
> a functional test.  In my email I said I support using Python for those
> types of tests.
>
>> Now thinking about the specific reasons...
>>
>> What I intend to do is add qtests for the SDHCI implementation.
>> There are different revisions of the standard specs.
>
> It would be great to have SDHCI support in libqos.  PCI and virtio are
> covered today and support for more busses will help reach that critical
> mass where tests can be written for most QEMU device models without
> first writing a new device driver framework.
>
> Please post your SDHCI test code, so we can discuss the details and
> consider whether it's more like 041 or more like tests/ide-test.c.

The SDHCI test is like ide-test, poking register.

Being not a 'generic' series but only SDHCI I didn't Cc'd you,
you can find the test here:

http://lists.nongnu.org/archive/html/qemu-devel/2017-12/msg02391.html

and the SDCard python test here (you got this one):

http://lists.nongnu.org/archive/html/qemu-devel/2017-12/msg02350.html

On Thu, Dec 14, 2017 at 11:39 AM Stefan Hajnoczi <stefanha@redhat.com>
wrote:

> On Wed, Dec 13, 2017 at 06:35:51PM -0300, Philippe Mathieu-Daudé wrote:
> > Hi,
> >
> > With this series we can now write tests using Python rather than C.
> > For complex tests this can reduce the test development time, we can
> focus on
> > the test purposes instead of his implementation details.
> >
> > - 1,2: we already have Python classes to run Block tests, move all the
> >   non Block-related methods to qtest.py,
> > - 3: default TEST_DIR to TMPDIR,
> > - 4: add a method to restrict tests to a list of supported machines,
> > - 5: since the Block tests are output sensitive, do not mess with their
> >   current tuned iotests::main(unittest), add a more generic one in
> qtest.py,
> > - 6: finally modify the tests Makefile to run C/Python tests with the
> same
> >   rule.
>
> Python tests are great for functional tests of qemu-system-* that
> interact via the command-line and QMP monitor.  From this perspective I
> think the series is good.
>
> > to have a better idea, here is a snippet from the next series:
> >
> >     class TestSdCardSpecV2(qtest.QMPTestCase):
> >         [...]
> >         def test_power_on_spec_v2(self):
> >             self.bus.do_cmd(GO_IDLE_STATE)
> >             [...]
> >             # verify Card ID
> >             data = self.bus.do_cmd(ALL_SEND_CID)
> >             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
> >             self.assertEqual(oid, "XY") # QEMU default
> >             self.assertEqual(pnm, "QEMU!") # QEMU default
> >             self.assertEqual(psn, 0xdeadbeef) # QEMU default
>
> Device qtests are better done in C than Python.  Python is not good at
> binary I/O


Why do you think that? what is missing in python 2 for binary io?


> and porting this to Python 3 will be extra work later (Python
> 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
>

You can write today code that work with both python 2 and 3. For binary
io the key is using io.FileIO and io.BytesIO instead of open() and StringIO
and CStringIO.

Nir


>
> More importantly, we already have libqos in C with a guest memory
> allocator, PCI, and virtio support.  Fragmenting the small amount effort
> that goes into device testing will delay libqos reaching critical mass.
> Critical mass is where libqos provides all the infrastructure you need
> to set up a device and focus on your actual test instead of machine,
> bus, or device initialization.  Starting a Python device testing effort
> will just lead to duplication and 2 underdeveloped device testing
> frameworks.
>
> Is there a specific reason why adding SD Card support to libqos is not
> possible in C?
>

On Thu, Dec 14, 2017 at 03:35:07PM +0000, Nir Soffer wrote:
> On Thu, Dec 14, 2017 at 11:39 AM Stefan Hajnoczi <stefanha@redhat.com>
> wrote:
> 
> > On Wed, Dec 13, 2017 at 06:35:51PM -0300, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > >
> > > With this series we can now write tests using Python rather than C.
> > > For complex tests this can reduce the test development time, we can
> > focus on
> > > the test purposes instead of his implementation details.
> > >
> > > - 1,2: we already have Python classes to run Block tests, move all the
> > >   non Block-related methods to qtest.py,
> > > - 3: default TEST_DIR to TMPDIR,
> > > - 4: add a method to restrict tests to a list of supported machines,
> > > - 5: since the Block tests are output sensitive, do not mess with their
> > >   current tuned iotests::main(unittest), add a more generic one in
> > qtest.py,
> > > - 6: finally modify the tests Makefile to run C/Python tests with the
> > same
> > >   rule.
> >
> > Python tests are great for functional tests of qemu-system-* that
> > interact via the command-line and QMP monitor.  From this perspective I
> > think the series is good.
> >
> > > to have a better idea, here is a snippet from the next series:
> > >
> > >     class TestSdCardSpecV2(qtest.QMPTestCase):
> > >         [...]
> > >         def test_power_on_spec_v2(self):
> > >             self.bus.do_cmd(GO_IDLE_STATE)
> > >             [...]
> > >             # verify Card ID
> > >             data = self.bus.do_cmd(ALL_SEND_CID)
> > >             oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
> > >             self.assertEqual(oid, "XY") # QEMU default
> > >             self.assertEqual(pnm, "QEMU!") # QEMU default
> > >             self.assertEqual(psn, 0xdeadbeef) # QEMU default
> >
> > Device qtests are better done in C than Python.  Python is not good at
> > binary I/O
> 
> 
> Why do you think that? what is missing in python 2 for binary io?

C code can reuse existing header files for struct definitions instead
of doing oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data).

In general, Python's model for binary data is a marshalling (copy) model
rather than a type casting (in-place access) model.  It's fine for some
things but programs that deal with a lot of raw data may use third-party
unboxed data primitives like numpy's instead.  I definitely wouldn't say
that binary I/O is a strength of Python.

> > and porting this to Python 3 will be extra work later (Python
> > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> >
> 
> You can write today code that work with both python 2 and 3. For binary
> io the key is using io.FileIO and io.BytesIO instead of open() and StringIO
> and CStringIO.

Yes, it's possible to write the code carefully and test under both
Python 2 & 3.  We have to do that for Python code in QEMU since the
world is currently transitioning and both Python versions are in use.

So in this case, where it's questionable to start a new device-level
testing framework in the first place, the extra trouble of dealing with
Python 2 & 3 makes it even less appealing.

Stefan

> C code can reuse existing header files for struct definitions instead
> of doing oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data).

On the other hand, C code risks having the same bug in tests and
device models, and also endianness is less apparent in C.  Overall
I'd say it's a wash---if I were to start now I'd use Python but I
agree with you that fragmenting the tests is not a good idea.

For the purpose of testing the SD card code, I'd use the SD-over-SSI
device.  Unfortunately there is no board that uses bit-banging SSI,
but STM32F405 is as close as it can get to it.

Paolo

> In general, Python's model for binary data is a marshalling (copy) model
> rather than a type casting (in-place access) model.  It's fine for some
> things but programs that deal with a lot of raw data may use third-party
> unboxed data primitives like numpy's instead.  I definitely wouldn't say
> that binary I/O is a strength of Python.
> 
> > > and porting this to Python 3 will be extra work later (Python
> > > 2 is set for End-of-Life in 2020, see https://pythonclock.org/).
> > >
> > 
> > You can write today code that work with both python 2 and 3. For binary
> > io the key is using io.FileIO and io.BytesIO instead of open() and StringIO
> > and CStringIO.
> 
> Yes, it's possible to write the code carefully and test under both
> Python 2 & 3.  We have to do that for Python code in QEMU since the
> world is currently transitioning and both Python versions are in use.
> 
> So in this case, where it's questionable to start a new device-level
> testing framework in the first place, the extra trouble of dealing with
> Python 2 & 3 makes it even less appealing.
> 
> Stefan
>