[Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface

Balamuruhan S posted 6 patches 4 years, 7 months ago
Test asan failed
Test FreeBSD passed
Test checkpatch failed
Test docker-mingw@fedora passed
Test docker-clang@ubuntu failed
Test s390x failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190807071445.4109-1-bala24@linux.ibm.com
Maintainers: David Gibson <david@gibson.dropbear.id.au>, "Cédric Le Goater" <clg@kaod.org>, Paolo Bonzini <pbonzini@redhat.com>
configure                   |  10 +++
hw/ppc/Makefile.objs        |   2 +-
hw/ppc/pnv.c                |  49 ++++++++++-
hw/ppc/pnv_homer.c          | 205 ++++++++++++++++++++++++++++++++++++++++++++
hw/ppc/pnv_xscom.c          |  59 +++++++++++--
include/hw/ppc/pnv.h        |  15 ++++
include/hw/ppc/pnv_homer.h  |  41 +++++++++
include/sysemu/python_api.h |  30 +++++++
include/sysemu/sysemu.h     |   8 ++
qemu-options.hx             |  14 +++
util/Makefile.objs          |   1 +
util/python_api.c           | 100 +++++++++++++++++++++
vl.c                        |  66 ++++++++++++++
13 files changed, 588 insertions(+), 12 deletions(-)
create mode 100644 hw/ppc/pnv_homer.c
create mode 100644 include/hw/ppc/pnv_homer.h
create mode 100644 include/sysemu/python_api.h
create mode 100644 util/python_api.c
[Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by Balamuruhan S 4 years, 7 months ago
Hi All,

This is a proposal to extend mmio callbacks in Qemu with scripting interface
that is prototyped with python in this implementation. It gives ability to
feed runtime data through callbacks without recompiling Qemu in generic way.
This patchset adds library that provides APIs for Qemu to talk with python
scripts placed in path -module-path and how existing xscom can be extended
with python interface infrastructure.

We have also added an hacky emulation for memory region (OCC common area and HOMER)
which is shared between core and un-core engine (ideally this should be via
sram device) to showcase the effectiveness of having the scripting interface
(uncore engine taken for discussion here is powerpc specificed called OCC).
Having scripting interface helps to emulate/test different uncore-core
interactions including uncore engine failure or hang. It also helps in feeding
randomized data at byte level access. This patchset is primarily to extend mmio
callbacks with scripting interface and to demonstrate effectiveness it.

Some changes are required in PowerPC skiboot tree to test these changes since
the memory region is disabled currently for Qemu emulated PowerNV host,
https://github.com/balamuruhans/skiboot/commit/a655514d2a730e0372a2faee277d1cf01f71a524

Qemu commandline used to test,

```
# qemu/ppc64-softmmu/qemu-system-ppc64 \
-M powernv \
-cpu POWER9 \
-m 16G \
-kernel vmlinux \
-initrd debug_homer.cpio \
-nographic -bios skiboot/skiboot.lid \
-module-path /home/bala/homer/python-modules/,xscom_module=homer,xscom_read=xscom_read,xscom_write=xscom_write,homer_module=homer,homer=homer_read,occ_module=homer,occ=occ_read
```

Script used to feed data can be something like,
https://github.com/balamuruhans/python-modules/blob/master/script.py

It could uncover couple of firmware bugs,
https://github.com/balamuruhans/skiboot/commit/fd3d93d92ec66a7494346d6d24ced7b48264c9a0
https://github.com/balamuruhans/skiboot/commit/165b3829a93bc177c18133945a8cca3a2d701173

Code changes:
Patch 1: adds library to provide python interface APIs
Patch 2: extend existing xscom to adopt this python interface
Patch 3 - 6: emulate uncore/core shared memory region with mmio callbacks and
add support with this infrastructure.

I request for comments, suggestions, ideas on getting a scripting interface
like python added in qemu.

Balamuruhan S (6):
  utils/python_api: add scripting interface for Qemu with python lib
  hw/ppc/pnv_xscom: extend xscom to use python interface
  hw/ppc/pnv_homer: add homer/occ common area emulation for PowerNV
  hw/ppc/pnv: initialize and realize homer/occ common area
  hw/ppc/pnv_xscom: retrieve homer/occ base address from PBA BARs
  hw/ppc/pnv_homer: add python interface support for homer/occ common
    area

 configure                   |  10 +++
 hw/ppc/Makefile.objs        |   2 +-
 hw/ppc/pnv.c                |  49 ++++++++++-
 hw/ppc/pnv_homer.c          | 205 ++++++++++++++++++++++++++++++++++++++++++++
 hw/ppc/pnv_xscom.c          |  59 +++++++++++--
 include/hw/ppc/pnv.h        |  15 ++++
 include/hw/ppc/pnv_homer.h  |  41 +++++++++
 include/sysemu/python_api.h |  30 +++++++
 include/sysemu/sysemu.h     |   8 ++
 qemu-options.hx             |  14 +++
 util/Makefile.objs          |   1 +
 util/python_api.c           | 100 +++++++++++++++++++++
 vl.c                        |  66 ++++++++++++++
 13 files changed, 588 insertions(+), 12 deletions(-)
 create mode 100644 hw/ppc/pnv_homer.c
 create mode 100644 include/hw/ppc/pnv_homer.h
 create mode 100644 include/sysemu/python_api.h
 create mode 100644 util/python_api.c

-- 
2.14.5


Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by no-reply@patchew.org 4 years, 7 months ago
Patchew URL: https://patchew.org/QEMU/20190807071445.4109-1-bala24@linux.ibm.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Subject: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Message-id: 20190807071445.4109-1-bala24@linux.ibm.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/20190807071445.4109-1-bala24@linux.ibm.com -> patchew/20190807071445.4109-1-bala24@linux.ibm.com
Submodule 'capstone' (https://git.qemu.org/git/capstone.git) registered for path 'capstone'
Submodule 'dtc' (https://git.qemu.org/git/dtc.git) registered for path 'dtc'
Submodule 'roms/QemuMacDrivers' (https://git.qemu.org/git/QemuMacDrivers.git) registered for path 'roms/QemuMacDrivers'
Submodule 'roms/SLOF' (https://git.qemu.org/git/SLOF.git) registered for path 'roms/SLOF'
Submodule 'roms/edk2' (https://git.qemu.org/git/edk2.git) registered for path 'roms/edk2'
Submodule 'roms/ipxe' (https://git.qemu.org/git/ipxe.git) registered for path 'roms/ipxe'
Submodule 'roms/openbios' (https://git.qemu.org/git/openbios.git) registered for path 'roms/openbios'
Submodule 'roms/openhackware' (https://git.qemu.org/git/openhackware.git) registered for path 'roms/openhackware'
Submodule 'roms/opensbi' (https://git.qemu.org/git/opensbi.git) registered for path 'roms/opensbi'
Submodule 'roms/qemu-palcode' (https://git.qemu.org/git/qemu-palcode.git) registered for path 'roms/qemu-palcode'
Submodule 'roms/seabios' (https://git.qemu.org/git/seabios.git/) registered for path 'roms/seabios'
Submodule 'roms/seabios-hppa' (https://git.qemu.org/git/seabios-hppa.git) registered for path 'roms/seabios-hppa'
Submodule 'roms/sgabios' (https://git.qemu.org/git/sgabios.git) registered for path 'roms/sgabios'
Submodule 'roms/skiboot' (https://git.qemu.org/git/skiboot.git) registered for path 'roms/skiboot'
Submodule 'roms/u-boot' (https://git.qemu.org/git/u-boot.git) registered for path 'roms/u-boot'
Submodule 'roms/u-boot-sam460ex' (https://git.qemu.org/git/u-boot-sam460ex.git) registered for path 'roms/u-boot-sam460ex'
Submodule 'slirp' (https://git.qemu.org/git/libslirp.git) registered for path 'slirp'
Submodule 'tests/fp/berkeley-softfloat-3' (https://git.qemu.org/git/berkeley-softfloat-3.git) registered for path 'tests/fp/berkeley-softfloat-3'
Submodule 'tests/fp/berkeley-testfloat-3' (https://git.qemu.org/git/berkeley-testfloat-3.git) registered for path 'tests/fp/berkeley-testfloat-3'
Submodule 'ui/keycodemapdb' (https://git.qemu.org/git/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into 'capstone'...
Submodule path 'capstone': checked out '22ead3e0bfdb87516656453336160e0a37b066bf'
Cloning into 'dtc'...
Submodule path 'dtc': checked out '88f18909db731a627456f26d779445f84e449536'
Cloning into 'roms/QemuMacDrivers'...
Submodule path 'roms/QemuMacDrivers': checked out '90c488d5f4a407342247b9ea869df1c2d9c8e266'
Cloning into 'roms/SLOF'...
Submodule path 'roms/SLOF': checked out 'ba1ab360eebe6338bb8d7d83a9220ccf7e213af3'
Cloning into 'roms/edk2'...
Submodule path 'roms/edk2': checked out '20d2e5a125e34fc8501026613a71549b2a1a3e54'
Submodule 'SoftFloat' (https://github.com/ucb-bar/berkeley-softfloat-3.git) registered for path 'ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'
Submodule 'CryptoPkg/Library/OpensslLib/openssl' (https://github.com/openssl/openssl) registered for path 'CryptoPkg/Library/OpensslLib/openssl'
Cloning into 'ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3'...
Submodule path 'roms/edk2/ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3': checked out 'b64af41c3276f97f0e181920400ee056b9c88037'
Cloning into 'CryptoPkg/Library/OpensslLib/openssl'...
Submodule path 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl': checked out '50eaac9f3337667259de725451f201e784599687'
Submodule 'boringssl' (https://boringssl.googlesource.com/boringssl) registered for path 'boringssl'
Submodule 'krb5' (https://github.com/krb5/krb5) registered for path 'krb5'
Submodule 'pyca.cryptography' (https://github.com/pyca/cryptography.git) registered for path 'pyca-cryptography'
Cloning into 'boringssl'...
Submodule path 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl/boringssl': checked out '2070f8ad9151dc8f3a73bffaa146b5e6937a583f'
Cloning into 'krb5'...
Submodule path 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl/krb5': checked out 'b9ad6c49505c96a088326b62a52568e3484f2168'
Cloning into 'pyca-cryptography'...
Submodule path 'roms/edk2/CryptoPkg/Library/OpensslLib/openssl/pyca-cryptography': checked out '09403100de2f6f1cdd0d484dcb8e620f1c335c8f'
Cloning into 'roms/ipxe'...
Submodule path 'roms/ipxe': checked out 'de4565cbe76ea9f7913a01f331be3ee901bb6e17'
Cloning into 'roms/openbios'...
Submodule path 'roms/openbios': checked out 'c79e0ecb84f4f1ee3f73f521622e264edd1bf174'
Cloning into 'roms/openhackware'...
Submodule path 'roms/openhackware': checked out 'c559da7c8eec5e45ef1f67978827af6f0b9546f5'
Cloning into 'roms/opensbi'...
Submodule path 'roms/opensbi': checked out 'ce228ee0919deb9957192d723eecc8aaae2697c6'
Cloning into 'roms/qemu-palcode'...
Submodule path 'roms/qemu-palcode': checked out 'bf0e13698872450164fa7040da36a95d2d4b326f'
Cloning into 'roms/seabios'...
Submodule path 'roms/seabios': checked out 'a5cab58e9a3fb6e168aba919c5669bea406573b4'
Cloning into 'roms/seabios-hppa'...
Submodule path 'roms/seabios-hppa': checked out '0f4fe84658165e96ce35870fd19fc634e182e77b'
Cloning into 'roms/sgabios'...
Submodule path 'roms/sgabios': checked out 'cbaee52287e5f32373181cff50a00b6c4ac9015a'
Cloning into 'roms/skiboot'...
Submodule path 'roms/skiboot': checked out '261ca8e779e5138869a45f174caa49be6a274501'
Cloning into 'roms/u-boot'...
Submodule path 'roms/u-boot': checked out 'd3689267f92c5956e09cc7d1baa4700141662bff'
Cloning into 'roms/u-boot-sam460ex'...
Submodule path 'roms/u-boot-sam460ex': checked out '60b3916f33e617a815973c5a6df77055b2e3a588'
Cloning into 'slirp'...
Submodule path 'slirp': checked out '126c04acbabd7ad32c2b018fe10dfac2a3bc1210'
Cloning into 'tests/fp/berkeley-softfloat-3'...
Submodule path 'tests/fp/berkeley-softfloat-3': checked out 'b64af41c3276f97f0e181920400ee056b9c88037'
Cloning into 'tests/fp/berkeley-testfloat-3'...
Submodule path 'tests/fp/berkeley-testfloat-3': checked out '5a59dcec19327396a011a17fd924aed4fec416b3'
Cloning into 'ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '6b3d716e2b6472eb7189d3220552280ef3d832ce'
Switched to a new branch 'test'
0dabfa2 hw/ppc/pnv_homer: add python interface support for homer/occ common area
aa6063d hw/ppc/pnv_xscom: retrieve homer/occ base address from PBA BARs
43e9902 hw/ppc/pnv: initialize and realize homer/occ common area
a4755fe hw/ppc/pnv_homer: add homer/occ common area emulation for PowerNV
4cd625a hw/ppc/pnv_xscom: extend xscom to use python interface
9ad8c48 utils/python_api: add scripting interface for Qemu with python lib

=== OUTPUT BEGIN ===
1/6 Checking commit 9ad8c482b9ae (utils/python_api: add scripting interface for Qemu with python lib)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#45: 
new file mode 100644

ERROR: braces {} are necessary for all arms of this statement
#109: FILE: util/python_api.c:12:
+    if (!abs_module_path)
[...]

ERROR: else should follow close brace '}'
#141: FILE: util/python_api.c:44:
+    }
+    else {

ERROR: braces {} are necessary for all arms of this statement
#142: FILE: util/python_api.c:45:
+        if (PyErr_Occurred())
[...]

ERROR: spaces required around that '=' (ctx:VxW)
#182: FILE: util/python_api.c:85:
+    args[pos]= malloc(sizeof(int));
              ^

ERROR: spaces required around that '=' (ctx:VxW)
#188: FILE: util/python_api.c:91:
+    args[pos]= g_malloc(sizeof(uint64_t) * 2);
              ^

total: 5 errors, 1 warnings, 159 lines checked

Patch 1/6 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

2/6 Checking commit 4cd625a90b2a (hw/ppc/pnv_xscom: extend xscom to use python interface)
ERROR: else should follow close brace '}'
#48: FILE: hw/ppc/pnv_xscom.c:172:
+    }
+    else {

ERROR: braces {} are necessary for all arms of this statement
#67: FILE: hw/ppc/pnv_xscom.c:209:
+        if (xscom_success)
[...]

ERROR: do not initialise globals to 0 or NULL
#124: FILE: vl.c:143:
+const char *module_path = NULL;

ERROR: do not initialise globals to 0 or NULL
#125: FILE: vl.c:144:
+const char *xscom_module = NULL;

ERROR: do not initialise globals to 0 or NULL
#126: FILE: vl.c:145:
+const char *xscom_readp = NULL;

ERROR: do not initialise globals to 0 or NULL
#127: FILE: vl.c:146:
+const char *xscom_writep = NULL;

WARNING: Block comments use a leading /* on a separate line
#157: FILE: vl.c:498:
+        { /* end of list */ }

total: 6 errors, 1 warnings, 151 lines checked

Patch 2/6 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

3/6 Checking commit a4755feccbee (hw/ppc/pnv_homer: add homer/occ common area emulation for PowerNV)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#36: 
new file mode 100644

ERROR: braces {} are necessary for all arms of this statement
#70: FILE: hw/ppc/pnv_homer.c:30:
+    if (cpu_type)
[...]

ERROR: suspect code indent for conditional statements (4, 7)
#72: FILE: hw/ppc/pnv_homer.c:32:
+    for (int i = 0; i <= ms->smp.cores; i++)
+       if (addr == (core_max_base + i))

ERROR: suspect code indent for conditional statements (7, 11)
#73: FILE: hw/ppc/pnv_homer.c:33:
+       if (addr == (core_max_base + i))
+           return true;

ERROR: braces {} are necessary for all arms of this statement
#73: FILE: hw/ppc/pnv_homer.c:33:
+       if (addr == (core_max_base + i))
[...]

ERROR: switch and case should be at the same indent
#80: FILE: hw/ppc/pnv_homer.c:40:
+    switch (addr) {
+        case 0xe2006:  /* max pstate ultra turbo */
+        case 0xe2018:  /* pstate id for 0 */
+        case 0x1f8001: /* P8 occ pstate version */
+        case 0x1f8003: /* P8 pstate min */
+        case 0x1f8010: /* P8 pstate id for 0 */
[...]
+        case 0xe2000:  /* occ data area */
+        case 0xe2002:  /* occ_role master/slave*/
+        case 0xe2004:  /* pstate nom */
+        case 0xe2005:  /* pstate turbo */
+        case 0xe2020:  /* pstate id for 1 */
+        case 0xe2818:  /* pstate ultra turbo */
+        case 0xe2b85:  /* opal dynamic data (runtime) */
+        case 0x1f8000: /* P8 occ pstate valid */
+        case 0x1f8002: /* P8 throttle */
+        case 0x1f8004: /* P8 pstate nom */
+        case 0x1f8005: /* P8 pstate turbo */
+        case 0x1f8012: /* vdd voltage identifier */
+        case 0x1f8013: /* vcs voltage identifier */
+        case 0x1f8018: /* P8 pstate id for 1 */
[...]
+        case 0xe2003:  /* pstate min (2 as pstate min) */
+        case 0xe2028:  /* pstate id for 2 */
+        case 0x1f8006: /* P8 pstate ultra turbo */
+        case 0x1f8020: /* P8 pstate id for 2 */
[...]
+        case 0xe2001:  /* major version */
[...]
+        case 0xe201c:
+        case 0xe2024:
+        case 0xe202c:
[...]
+        case 0x1f8014:
+        case 0x1f801c:
+        case 0x1f8024:
[...]
+        case 0x0:      /* homer base */
+        case 0xe2008:  /* occ data area + 8 */
+        case 0x1f8008: /* P8 occ data area + 8 */
+        case 0x200008: /* homer base access to get homer image pointer*/

ERROR: braces {} are necessary for all arms of this statement
#125: FILE: hw/ppc/pnv_homer.c:85:
+    if (core_max_array(addr))
[...]

ERROR: switch and case should be at the same indent
#149: FILE: hw/ppc/pnv_homer.c:109:
+    switch (addr) {
[...]
+        case 0x580000: /* occ sensor data block */
+        case 0x580001: /* valid */
+        case 0x580002: /* version */
+        case 0x580004: /* reading_version */
+        case 0x580008: /* nr_sensors */
+        case 0x580010: /* names_offset */
+        case 0x580014: /* reading_ping_offset */
+        case 0x58000c: /* reading_pong_offset */
+        case 0x580023: /* structure_type */
[...]
+        case 0x58000d: /* name length */
[...]
+        case 0x580022: /* occ sensor loc core */
[...]
+        case 0x580003: /* occ sensor type power */
[...]
+        case 0x580005: /* sensor name */
[...]
+        case 0x58001e: /* HWMON_SENSORS_MASK */
+        case 0x580020:
[...]
+        case 0x0:      /* P8 slw base access for slw image size */

WARNING: line over 80 characters
#316: FILE: include/hw/ppc/pnv_homer.h:39:
+    (0x203fff800000ull + ((uint64_t)(chip)->chip_num) * PNV_OCC_COMMON_AREA_SIZE)

total: 7 errors, 2 warnings, 272 lines checked

Patch 3/6 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

4/6 Checking commit 43e990229339 (hw/ppc/pnv: initialize and realize homer/occ common area)
5/6 Checking commit aa6063d8ca2f (hw/ppc/pnv_xscom: retrieve homer/occ base address from PBA BARs)
6/6 Checking commit 0dabfa28124e (hw/ppc/pnv_homer: add python interface support for homer/occ common area)
WARNING: line over 80 characters
#38: FILE: hw/ppc/pnv_homer.c:46:
+        homer_ret = python_callback_int(module_path, homer_module, homer, address, 1);

ERROR: do not initialise globals to 0 or NULL
#108: FILE: vl.c:147:
+const char *homer_module = NULL;

ERROR: do not initialise globals to 0 or NULL
#109: FILE: vl.c:148:
+const char *homer = NULL;

ERROR: do not initialise globals to 0 or NULL
#110: FILE: vl.c:149:
+const char *occ_module = NULL;

ERROR: do not initialise globals to 0 or NULL
#111: FILE: vl.c:150:
+const char *occ = NULL;

total: 4 errors, 1 warnings, 109 lines checked

Patch 6/6 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190807071445.4109-1-bala24@linux.ibm.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by no-reply@patchew.org 4 years, 7 months ago
Patchew URL: https://patchew.org/QEMU/20190807071445.4109-1-bala24@linux.ibm.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      accel/tcg/trace.o
  CC      crypto/trace.o
In file included from /tmp/qemu-test/src/util/python_api.c:1:
/tmp/qemu-test/src/include/sysemu/python_api.h:5:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
  CC      monitor/trace.o


The full log is available at
http://patchew.org/logs/20190807071445.4109-1-bala24@linux.ibm.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by Stefan Hajnoczi 4 years, 7 months ago
On Wed, Aug 07, 2019 at 12:44:39PM +0530, Balamuruhan S wrote:
> This is a proposal to extend mmio callbacks in Qemu with scripting interface
> that is prototyped with python in this implementation. It gives ability to
> feed runtime data through callbacks without recompiling Qemu in generic way.
> This patchset adds library that provides APIs for Qemu to talk with python
> scripts placed in path -module-path and how existing xscom can be extended
> with python interface infrastructure.
> 
> We have also added an hacky emulation for memory region (OCC common area and HOMER)
> which is shared between core and un-core engine (ideally this should be via
> sram device) to showcase the effectiveness of having the scripting interface
> (uncore engine taken for discussion here is powerpc specificed called OCC).
> Having scripting interface helps to emulate/test different uncore-core
> interactions including uncore engine failure or hang. It also helps in feeding
> randomized data at byte level access. This patchset is primarily to extend mmio
> callbacks with scripting interface and to demonstrate effectiveness it.
> 
> Some changes are required in PowerPC skiboot tree to test these changes since
> the memory region is disabled currently for Qemu emulated PowerNV host,
> https://github.com/balamuruhans/skiboot/commit/a655514d2a730e0372a2faee277d1cf01f71a524

Although writing Python is quick and easy, carefully wiring up the
Python C API for it is not.  In practice you lose much of the benefit of
Python if you need to study the Python C API every time you wish to do
some quick scripting :(.

It must be possible to compile out the Python integration code.  If the
Python integration code remains in the device model then the QEMU binary
has a dependency on libpython, which is undesirable when this feature is
not in use.

Assuming this feature can be compiled out, I think it should have a
chance to prove its usefulness and gain users.  Documentation and an
active maintainer are essential.

Stefan
Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by Balamuruhan S 4 years, 7 months ago

On 8/8/19 3:55 PM, Stefan Hajnoczi wrote:
> On Wed, Aug 07, 2019 at 12:44:39PM +0530, Balamuruhan S wrote:
>> This is a proposal to extend mmio callbacks in Qemu with scripting interface
>> that is prototyped with python in this implementation. It gives ability to
>> feed runtime data through callbacks without recompiling Qemu in generic way.
>> This patchset adds library that provides APIs for Qemu to talk with python
>> scripts placed in path -module-path and how existing xscom can be extended
>> with python interface infrastructure.
>>
>> We have also added an hacky emulation for memory region (OCC common area and HOMER)
>> which is shared between core and un-core engine (ideally this should be via
>> sram device) to showcase the effectiveness of having the scripting interface
>> (uncore engine taken for discussion here is powerpc specificed called OCC).
>> Having scripting interface helps to emulate/test different uncore-core
>> interactions including uncore engine failure or hang. It also helps in feeding
>> randomized data at byte level access. This patchset is primarily to extend mmio
>> callbacks with scripting interface and to demonstrate effectiveness it.
>>
>> Some changes are required in PowerPC skiboot tree to test these changes since
>> the memory region is disabled currently for Qemu emulated PowerNV host,
>> https://github.com/balamuruhans/skiboot/commit/a655514d2a730e0372a2faee277d1cf01f71a524
> Although writing Python is quick and easy, carefully wiring up the
> Python C API for it is not.  In practice you lose much of the benefit of
> Python if you need to study the Python C API every time you wish to do
> some quick scripting :(.

Initially as you said to wire Python C API it needs study and care, but
once the framework reach to a stability we can use it for easy scripting
using it seamlessly.

>
> It must be possible to compile out the Python integration code.  If the
> Python integration code remains in the device model then the QEMU binary
> has a dependency on libpython, which is undesirable when this feature is
> not in use.

if we can keep it conditional during configure, so that by default qemu
binary will not depend on libpython. If user needs to use this feature
then python 3 based libpython and python-config are necessary.

>
> Assuming this feature can be compiled out, I think it should have a
> chance to prove its usefulness and gain users.  Documentation and an
> active maintainer are essential.

yes, hope that community will find its usefulness as it helped us to
test, study behavior of firmware/kernel and find bugs that is hard
to find otherwise.

Agreed that it would need proper documentation and active maintainer.

>
> Stefan


Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by no-reply@patchew.org 4 years, 7 months ago
Patchew URL: https://patchew.org/QEMU/20190807071445.4109-1-bala24@linux.ibm.com/



Hi,

This series failed build test on s390x host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e

echo
echo "=== ENV ==="
env

echo
echo "=== PACKAGES ==="
rpm -qa

echo
echo "=== UNAME ==="
uname -a

CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

  CC      util/filemonitor-inotify.o
  CC      util/vfio-helpers.o
/var/tmp/patchew-tester-tmp-j_q3aucf/src/util/python_api.c: In function ‘python_callback_str’:
/var/tmp/patchew-tester-tmp-j_q3aucf/src/util/python_api.c:72:12: error: return discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]
   72 |     return PyUnicode_AsUTF8(result);
      |            ^~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors


The full log is available at
http://patchew.org/logs/20190807071445.4109-1-bala24@linux.ibm.com/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by Cédric Le Goater 4 years, 7 months ago
On 07/08/2019 09:14, Balamuruhan S wrote:
> Hi All,
> 
> This is a proposal to extend mmio callbacks in Qemu with scripting interface
> that is prototyped with python in this implementation. It gives ability to
> feed runtime data through callbacks without recompiling Qemu in generic way.
> This patchset adds library that provides APIs for Qemu to talk with python
> scripts placed in path -module-path and how existing xscom can be extended
> with python interface infrastructure.
> 
> We have also added an hacky emulation for memory region (OCC common area and HOMER)
> which is shared between core and un-core engine (ideally this should be via
> sram device) to showcase the effectiveness of having the scripting interface
> (uncore engine taken for discussion here is powerpc specificed called OCC).

We should try to merge this part first. It is useful as it is after some
cleanups.

> Having scripting interface helps to emulate/test different uncore-core
> interactions including uncore engine failure or hang. It also helps in feeding
> randomized data at byte level access. This patchset is primarily to extend mmio
> callbacks with scripting interface and to demonstrate effectiveness it.

It is already possible to feed device models with external data using QMP or
external agents using a chardev backend transport. What are the benefits
of using the embedded python approach ?  

> Some changes are required in PowerPC skiboot tree to test these changes since
> the memory region is disabled currently for Qemu emulated PowerNV host,
> https://github.com/balamuruhans/skiboot/commit/a655514d2a730e0372a2faee277d1cf01f71a524

You should send that patch.

Thanks,

C. 

> Qemu commandline used to test,
> 
> ```
> # qemu/ppc64-softmmu/qemu-system-ppc64 \
> -M powernv \
> -cpu POWER9 \
> -m 16G \
> -kernel vmlinux \
> -initrd debug_homer.cpio \
> -nographic -bios skiboot/skiboot.lid \
> -module-path /home/bala/homer/python-modules/,xscom_module=homer,xscom_read=xscom_read,xscom_write=xscom_write,homer_module=homer,homer=homer_read,occ_module=homer,occ=occ_read
> ```
> 
> Script used to feed data can be something like,
> https://github.com/balamuruhans/python-modules/blob/master/script.py
> 
> It could uncover couple of firmware bugs,
> https://github.com/balamuruhans/skiboot/commit/fd3d93d92ec66a7494346d6d24ced7b48264c9a0
> https://github.com/balamuruhans/skiboot/commit/165b3829a93bc177c18133945a8cca3a2d701173
> 
> Code changes:
> Patch 1: adds library to provide python interface APIs
> Patch 2: extend existing xscom to adopt this python interface
> Patch 3 - 6: emulate uncore/core shared memory region with mmio callbacks and
> add support with this infrastructure.
> 
> I request for comments, suggestions, ideas on getting a scripting interface
> like python added in qemu.
> 
> Balamuruhan S (6):
>   utils/python_api: add scripting interface for Qemu with python lib
>   hw/ppc/pnv_xscom: extend xscom to use python interface
>   hw/ppc/pnv_homer: add homer/occ common area emulation for PowerNV
>   hw/ppc/pnv: initialize and realize homer/occ common area
>   hw/ppc/pnv_xscom: retrieve homer/occ base address from PBA BARs
>   hw/ppc/pnv_homer: add python interface support for homer/occ common
>     area
> 
>  configure                   |  10 +++
>  hw/ppc/Makefile.objs        |   2 +-
>  hw/ppc/pnv.c                |  49 ++++++++++-
>  hw/ppc/pnv_homer.c          | 205 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/ppc/pnv_xscom.c          |  59 +++++++++++--
>  include/hw/ppc/pnv.h        |  15 ++++
>  include/hw/ppc/pnv_homer.h  |  41 +++++++++
>  include/sysemu/python_api.h |  30 +++++++
>  include/sysemu/sysemu.h     |   8 ++
>  qemu-options.hx             |  14 +++
>  util/Makefile.objs          |   1 +
>  util/python_api.c           | 100 +++++++++++++++++++++
>  vl.c                        |  66 ++++++++++++++
>  13 files changed, 588 insertions(+), 12 deletions(-)
>  create mode 100644 hw/ppc/pnv_homer.c
>  create mode 100644 include/hw/ppc/pnv_homer.h
>  create mode 100644 include/sysemu/python_api.h
>  create mode 100644 util/python_api.c
> 


Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by Balamuruhan S 4 years, 7 months ago
On Wed, Aug 07, 2019 at 10:15:48AM +0200, Cédric Le Goater wrote:
> On 07/08/2019 09:14, Balamuruhan S wrote:
> > Hi All,
> > 
> > This is a proposal to extend mmio callbacks in Qemu with scripting interface
> > that is prototyped with python in this implementation. It gives ability to
> > feed runtime data through callbacks without recompiling Qemu in generic way.
> > This patchset adds library that provides APIs for Qemu to talk with python
> > scripts placed in path -module-path and how existing xscom can be extended
> > with python interface infrastructure.
> > 
> > We have also added an hacky emulation for memory region (OCC common area and HOMER)
> > which is shared between core and un-core engine (ideally this should be via
> > sram device) to showcase the effectiveness of having the scripting interface
> > (uncore engine taken for discussion here is powerpc specificed called OCC).
> 
> We should try to merge this part first. It is useful as it is after some
> cleanups.

okay :+1:

> 
> > Having scripting interface helps to emulate/test different uncore-core
> > interactions including uncore engine failure or hang. It also helps in feeding
> > randomized data at byte level access. This patchset is primarily to extend mmio
> > callbacks with scripting interface and to demonstrate effectiveness it.
> 
> It is already possible to feed device models with external data using QMP or
> external agents using a chardev backend transport. What are the benefits
> of using the embedded python approach ?  
> 
> > Some changes are required in PowerPC skiboot tree to test these changes since
> > the memory region is disabled currently for Qemu emulated PowerNV host,
> > https://github.com/balamuruhans/skiboot/commit/a655514d2a730e0372a2faee277d1cf01f71a524
> 
> You should send that patch.

okay, I will send it.

Thank you Cedric for your review and suggestions.

-- Bala
> 
> Thanks,
> 
> C. 
> 
> > Qemu commandline used to test,
> > 
> > ```
> > # qemu/ppc64-softmmu/qemu-system-ppc64 \
> > -M powernv \
> > -cpu POWER9 \
> > -m 16G \
> > -kernel vmlinux \
> > -initrd debug_homer.cpio \
> > -nographic -bios skiboot/skiboot.lid \
> > -module-path /home/bala/homer/python-modules/,xscom_module=homer,xscom_read=xscom_read,xscom_write=xscom_write,homer_module=homer,homer=homer_read,occ_module=homer,occ=occ_read
> > ```
> > 
> > Script used to feed data can be something like,
> > https://github.com/balamuruhans/python-modules/blob/master/script.py
> > 
> > It could uncover couple of firmware bugs,
> > https://github.com/balamuruhans/skiboot/commit/fd3d93d92ec66a7494346d6d24ced7b48264c9a0
> > https://github.com/balamuruhans/skiboot/commit/165b3829a93bc177c18133945a8cca3a2d701173
> > 
> > Code changes:
> > Patch 1: adds library to provide python interface APIs
> > Patch 2: extend existing xscom to adopt this python interface
> > Patch 3 - 6: emulate uncore/core shared memory region with mmio callbacks and
> > add support with this infrastructure.
> > 
> > I request for comments, suggestions, ideas on getting a scripting interface
> > like python added in qemu.
> > 
> > Balamuruhan S (6):
> >   utils/python_api: add scripting interface for Qemu with python lib
> >   hw/ppc/pnv_xscom: extend xscom to use python interface
> >   hw/ppc/pnv_homer: add homer/occ common area emulation for PowerNV
> >   hw/ppc/pnv: initialize and realize homer/occ common area
> >   hw/ppc/pnv_xscom: retrieve homer/occ base address from PBA BARs
> >   hw/ppc/pnv_homer: add python interface support for homer/occ common
> >     area
> > 
> >  configure                   |  10 +++
> >  hw/ppc/Makefile.objs        |   2 +-
> >  hw/ppc/pnv.c                |  49 ++++++++++-
> >  hw/ppc/pnv_homer.c          | 205 ++++++++++++++++++++++++++++++++++++++++++++
> >  hw/ppc/pnv_xscom.c          |  59 +++++++++++--
> >  include/hw/ppc/pnv.h        |  15 ++++
> >  include/hw/ppc/pnv_homer.h  |  41 +++++++++
> >  include/sysemu/python_api.h |  30 +++++++
> >  include/sysemu/sysemu.h     |   8 ++
> >  qemu-options.hx             |  14 +++
> >  util/Makefile.objs          |   1 +
> >  util/python_api.c           | 100 +++++++++++++++++++++
> >  vl.c                        |  66 ++++++++++++++
> >  13 files changed, 588 insertions(+), 12 deletions(-)
> >  create mode 100644 hw/ppc/pnv_homer.c
> >  create mode 100644 include/hw/ppc/pnv_homer.h
> >  create mode 100644 include/sysemu/python_api.h
> >  create mode 100644 util/python_api.c
> > 
> 


Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by David Gibson 4 years, 7 months ago
On Wed, Aug 07, 2019 at 10:15:48AM +0200, Cédric Le Goater wrote:
> On 07/08/2019 09:14, Balamuruhan S wrote:
> > Hi All,
> > 
> > This is a proposal to extend mmio callbacks in Qemu with scripting interface
> > that is prototyped with python in this implementation. It gives ability to
> > feed runtime data through callbacks without recompiling Qemu in generic way.
> > This patchset adds library that provides APIs for Qemu to talk with python
> > scripts placed in path -module-path and how existing xscom can be extended
> > with python interface infrastructure.
> > 
> > We have also added an hacky emulation for memory region (OCC common area and HOMER)
> > which is shared between core and un-core engine (ideally this should be via
> > sram device) to showcase the effectiveness of having the scripting interface
> > (uncore engine taken for discussion here is powerpc specificed called OCC).
> 
> We should try to merge this part first. It is useful as it is after some
> cleanups.
> 
> > Having scripting interface helps to emulate/test different uncore-core
> > interactions including uncore engine failure or hang. It also helps in feeding
> > randomized data at byte level access. This patchset is primarily to extend mmio
> > callbacks with scripting interface and to demonstrate effectiveness it.
> 
> It is already possible to feed device models with external data using QMP or
> external agents using a chardev backend transport. What are the benefits
> of using the embedded python approach ?  

Yeah, I also think this needs better justification.

In particular what's the case that Python makes this significantly
easier than hacking up experimental interactions with C.  I mean you
already have to understand POWER9 internals to work with this, right,
so I wouldn't expect Python's greater accessibility to be a big
concern here.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
Re: [Qemu-devel] [RFC PATCH 0/6] Enhancing Qemu MMIO emulation with scripting interface
Posted by Balamuruhan S 4 years, 7 months ago
On 8/9/19 10:19 AM, David Gibson wrote:
> On Wed, Aug 07, 2019 at 10:15:48AM +0200, Cédric Le Goater wrote:
>> On 07/08/2019 09:14, Balamuruhan S wrote:
>>> Hi All,
>>>
>>> This is a proposal to extend mmio callbacks in Qemu with scripting interface
>>> that is prototyped with python in this implementation. It gives ability to
>>> feed runtime data through callbacks without recompiling Qemu in generic way.
>>> This patchset adds library that provides APIs for Qemu to talk with python
>>> scripts placed in path -module-path and how existing xscom can be extended
>>> with python interface infrastructure.
>>>
>>> We have also added an hacky emulation for memory region (OCC common area and HOMER)
>>> which is shared between core and un-core engine (ideally this should be via
>>> sram device) to showcase the effectiveness of having the scripting interface
>>> (uncore engine taken for discussion here is powerpc specificed called OCC).
>> We should try to merge this part first. It is useful as it is after some
>> cleanups.
>>
>>> Having scripting interface helps to emulate/test different uncore-core
>>> interactions including uncore engine failure or hang. It also helps in feeding
>>> randomized data at byte level access. This patchset is primarily to extend mmio
>>> callbacks with scripting interface and to demonstrate effectiveness it.
>> It is already possible to feed device models with external data using QMP or
>> external agents using a chardev backend transport. What are the benefits
>> of using the embedded python approach ?  
> Yeah, I also think this needs better justification.
>
> In particular what's the case that Python makes this significantly
> easier than hacking up experimental interactions with C.  I mean you
> already have to understand POWER9 internals to work with this, right,
> so I wouldn't expect Python's greater accessibility to be a big
> concern here.

right, with python interface what I could think of is,

1. we don't have to patch up every experimental interactions and recompile.
2. we can easily feed in invalid data type to see the behavior for negative/error
   scenarios.
3. Similar to qtest and acceptance test we can use this to cover many scenarios.
4. Ease the CI and maintenance to have test/code separately.

>