[Qemu-devel] [PATCH v2 00/29] Modularize generated QAPI code

Markus Armbruster posted 29 patches 6 years, 1 month ago
Failed in applying to current master (apply log)
Test checkpatch passed
Test docker-build@min-glib passed
Test docker-mingw@fedora failed
Test docker-quick@centos6 passed
Test ppcbe passed
Test ppcle passed
Test s390x passed
.gitignore                                         |  80 ++++-
Makefile                                           | 233 ++++++++++----
Makefile.objs                                      |  80 ++++-
arch_init.c                                        |   2 +-
backends/cryptodev.c                               |   1 -
backends/hostmem.c                                 |   3 +-
balloon.c                                          |   2 +-
block.c                                            |   2 -
block/block-backend.c                              |   2 +-
block/crypto.c                                     |   2 +-
block/iscsi.c                                      |   2 +-
block/nbd.c                                        |   2 +-
block/nfs.c                                        |   2 +-
block/qapi.c                                       |   4 +-
block/qcow2.c                                      |   4 +-
block/quorum.c                                     |   2 +-
block/sheepdog.c                                   |   2 +-
block/ssh.c                                        |   2 +-
block/throttle-groups.c                            |   2 +-
block/write-threshold.c                            |   4 +-
blockdev-nbd.c                                     |   2 +-
blockdev.c                                         |   5 +-
blockjob.c                                         |   2 +-
chardev/char-fe.c                                  |   2 +-
chardev/char-ringbuf.c                             |   2 +-
chardev/char-socket.c                              |   1 +
chardev/char.c                                     |   4 +-
configure                                          |   1 -
cpus.c                                             |   4 +-
crypto/cipherpriv.h                                |   2 +-
docs/devel/qapi-code-gen.txt                       | 119 ++++---
docs/devel/writing-qmp-commands.txt                |  39 +--
docs/interop/qmp-intro.txt                         |   3 +-
docs/xen-save-devices-state.txt                    |   3 +-
dump.c                                             |   4 +-
hmp.c                                              |  12 +-
hw/acpi/core.c                                     |   4 +-
hw/acpi/cpu.c                                      |   2 +-
hw/acpi/memory_hotplug.c                           |   2 +-
hw/acpi/vmgenid.c                                  |   2 +-
hw/block/block.c                                   |   1 +
hw/block/hd-geometry.c                             |   1 +
hw/char/virtio-console.c                           |   2 +-
hw/core/machine.c                                  |   2 +-
hw/core/qdev.c                                     |   2 +-
hw/i386/pc.c                                       |   2 +-
hw/i386/xen/xen-hvm.c                              |   2 +-
hw/ipmi/ipmi.c                                     |   2 +-
hw/mem/nvdimm.c                                    |   1 -
hw/net/rocker/qmp-norocker.c                       |   2 +-
hw/net/rocker/rocker.c                             |   2 +-
hw/net/rocker/rocker_fp.c                          |   2 +-
hw/net/rocker/rocker_of_dpa.c                      |   2 +-
hw/net/virtio-net.c                                |   2 +-
hw/pci/pci-stub.c                                  |   2 +-
hw/pci/pci.c                                       |   2 +-
hw/ppc/spapr_rtas.c                                |   1 -
hw/ppc/spapr_rtc.c                                 |   2 +-
hw/s390x/s390-skeys.c                              |   2 +-
hw/timer/mc146818rtc.c                             |   4 +-
hw/tpm/tpm_emulator.c                              |   1 +
hw/tpm/tpm_passthrough.c                           |   1 +
hw/virtio/virtio-balloon.c                         |   2 +-
hw/watchdog/watchdog.c                             |   4 +-
include/block/block.h                              |   2 +-
include/block/dirty-bitmap.h                       |   2 +-
include/block/nbd.h                                |   2 +-
include/chardev/char.h                             |   1 +
include/crypto/cipher.h                            |   2 +-
include/crypto/hash.h                              |   2 +-
include/crypto/hmac.h                              |   2 +-
include/crypto/secret.h                            |   1 +
include/crypto/tlscreds.h                          |   1 +
include/hw/block/block.h                           |   2 +-
include/hw/block/fdc.h                             |   2 +-
include/hw/ppc/spapr_drc.h                         |   1 +
include/hw/qdev-properties.h                       |   2 +
include/io/dns-resolver.h                          |   1 +
include/migration/colo.h                           |   2 +-
include/migration/failover.h                       |   2 +-
include/migration/global_state.h                   |   1 +
include/monitor/monitor.h                          |   1 +
include/net/filter.h                               |   1 +
include/net/net.h                                  |   2 +-
include/qapi/clone-visitor.h                       |   1 -
include/qapi/error.h                               |   2 +-
include/qapi/qmp/qobject.h                         |   2 +-
include/qapi/visitor.h                             |   2 +-
include/qemu/sockets.h                             |   2 +-
include/qemu/throttle.h                            |   2 +-
include/qom/cpu.h                                  |   1 +
include/qom/object.h                               |   2 +-
include/sysemu/arch_init.h                         |   2 +-
include/sysemu/balloon.h                           |   2 +-
include/sysemu/dump.h                              |   2 +
include/sysemu/hostmem.h                           |   1 +
include/sysemu/replay.h                            |   2 +
include/sysemu/sysemu.h                            |   1 +
include/sysemu/tpm.h                               |   1 +
include/sysemu/watchdog.h                          |   2 +-
include/ui/input.h                                 |   2 +-
io/channel-socket.c                                |   1 +
io/dns-resolver.c                                  |   1 +
iothread.c                                         |   2 +-
migration/colo-failover.c                          |   2 +-
migration/colo.c                                   |   2 +-
migration/migration.c                              |   4 +-
migration/migration.h                              |   1 +
migration/ram.c                                    |   2 +-
migration/ram.h                                    |   2 +-
migration/savevm.c                                 |   3 +-
monitor.c                                          |   8 +-
net/colo-compare.c                                 |   1 -
net/filter-buffer.c                                |   2 +-
net/filter-mirror.c                                |   1 -
net/filter-rewriter.c                              |   1 -
net/net.c                                          |   4 +-
net/tap_int.h                                      |   2 +-
net/vhost-user.c                                   |   2 +-
numa.c                                             |   4 +-
qapi-schema.json => qapi/misc.json                 | 105 +-----
qapi/qapi-schema.json                              |  95 ++++++
qapi/run-state.json                                |  10 +
qdev-monitor.c                                     |   2 +-
qemu-img.c                                         |   2 +-
qga/Makefile.objs                                  |   2 +-
qga/commands-posix.c                               |   2 +-
qga/commands-win32.c                               |   2 +-
qga/commands.c                                     |   2 +-
qga/main.c                                         |   2 +-
qmp.c                                              |  18 +-
qom/object.c                                       |   2 +-
qom/object_interfaces.c                            |   2 +-
replay/replay-input.c                              |   1 +
replication.h                                      |   1 +
scripts/qapi-gen.py                                |  57 ++++
scripts/qapi/__init__.py                           |   0
scripts/{qapi-commands.py => qapi/commands.py}     | 155 ++++-----
scripts/{qapi.py => qapi/common.py}                | 355 +++++++++++++--------
scripts/{qapi2texi.py => qapi/doc.py}              |  92 +++---
scripts/{qapi-event.py => qapi/events.py}          | 128 +++-----
scripts/{qapi-introspect.py => qapi/introspect.py} | 123 +++----
scripts/{qapi-types.py => qapi/types.py}           | 185 ++++-------
scripts/{qapi-visit.py => qapi/visit.py}           | 189 ++++-------
stubs/tpm.c                                        |   3 +-
stubs/uuid.c                                       |   2 +-
stubs/vmgenid.c                                    |   2 +-
stubs/xen-hvm.c                                    |   2 +-
target/arm/monitor.c                               |   3 +-
target/i386/cpu.c                                  |   4 +-
target/s390x/kvm.c                                 |   1 -
tests/.gitignore                                   |   9 +-
tests/Makefile.include                             |  70 ++--
tests/qapi-schema/comments.out                     |   3 +-
tests/qapi-schema/doc-bad-section.out              |   5 +-
tests/qapi-schema/doc-good.out                     |  33 +-
tests/qapi-schema/doc-good.texi                    |   3 +-
tests/qapi-schema/empty.out                        |   2 +-
tests/qapi-schema/event-case.out                   |   3 +-
tests/qapi-schema/ident-with-escape.out            |   7 +-
tests/qapi-schema/include-no-file.err              |   2 +-
tests/qapi-schema/include-relpath.out              |   7 +-
tests/qapi-schema/include-repetition.out           |  12 +-
tests/qapi-schema/include-simple.out               |   5 +-
tests/qapi-schema/indented-expr.out                |   3 +-
tests/qapi-schema/qapi-schema-test.out             | 321 +++++++++----------
tests/qapi-schema/test-qapi.py                     |  19 +-
tests/qmp-test.c                                   |   3 +-
tests/test-char.c                                  |   2 +-
tests/{test-qmp-commands.c => test-qmp-cmds.c}     |   2 +-
tests/test-qmp-event.c                             |   3 +-
tests/test-qobject-input-visitor.c                 |   8 +-
tests/test-visitor-serialization.c                 |   1 -
tpm.c                                              |   3 +-
trace/qmp.c                                        |   2 +-
ui/cocoa.m                                         |   2 +-
ui/console.c                                       |   2 +-
ui/gtk.c                                           |   2 +-
ui/input-legacy.c                                  |   2 +-
ui/input.c                                         |   2 +-
ui/spice-core.c                                    |   4 +-
ui/vnc.c                                           |   4 +-
ui/vnc.h                                           |   1 +
util/qemu-config.c                                 |   2 +-
util/qemu-sockets.c                                |   2 +-
vl.c                                               |   8 +-
186 files changed, 1556 insertions(+), 1352 deletions(-)
rename qapi-schema.json => qapi/misc.json (95%)
create mode 100644 qapi/qapi-schema.json
create mode 100755 scripts/qapi-gen.py
create mode 100644 scripts/qapi/__init__.py
rename scripts/{qapi-commands.py => qapi/commands.py} (67%)
rename scripts/{qapi.py => qapi/common.py} (90%)
rename scripts/{qapi2texi.py => qapi/doc.py} (78%)
mode change 100755 => 100644
rename scripts/{qapi-event.py => qapi/events.py} (61%)
rename scripts/{qapi-introspect.py => qapi/introspect.py} (71%)
rename scripts/{qapi-types.py => qapi/types.py} (53%)
rename scripts/{qapi-visit.py => qapi/visit.py} (64%)
rename tests/{test-qmp-commands.c => test-qmp-cmds.c} (99%)
[Qemu-devel] [PATCH v2 00/29] Modularize generated QAPI code
Posted by Markus Armbruster 6 years, 1 month ago
Our qapi-schema.json is composed of modules connected by include
directives, but the generated code is monolithic all the same: one
qapi-types.h with all the types, one qapi-visit.h with all the
visitors, and so forth.  These monolithic headers get included all
over the place.  In my "build everything" tree, adding a QAPI type
recompiles about 4800 out of 5100 objects.

We wouldn't write such monolithic headers by hand.  It stands to
reason that we shouldn't generate them, either.

This series' basic idea is to split up generated headers to mirror the
schema's modular structure: one header per module.  That way, you can
include just what you need.

The compile-time improvements can be massive.  Before this series, any
QAPI schema change recompiles some 4800 out of 5100 objects in my
"build everything" tree.  Afterwards, adding a type to
qapi/migration.json recompiles a bit over 100, and a documentation
change no longer recompiles anything.

Related: Marc-André's 'unit' pragma proposal.  That's a different way
to split off parts of the generated code, motivated by the desire to
use poisoned identifiers such as TARGET_I386.  I noted in my review of
v3 that I "can either accept it, or come up with a better solution."
This is my attempt at a better solution.  It's a bit more ambitious,
and thus more useful (I hope).  The pragma has one theoretical
advantage, though: you can modularize the generated output in
different ways than the input.  The patches using don't do that,
however.

v2:
* Rebased, annoying conflicts due to the Python 2 rigmarole
* PATCH 01: New
* PATCH 02-03,08,10-11,17,19-24: Commit messages improved
  [Eric, Marc-André]
* PATCH 04: Split off the next patch [Eric]
* PATCH 05,07-08,10,19: QAPIGen methods renamed [Marc-André]
* PATCH 08,20,22: fix missing .gitignore updates
* PATCH 08:
  - generated doc renamed to qapi-doc.texi
  - qapi-code-gen.txt updated
  - useless options -c, -h dropped
* PATCH 09: New, replacing old PATCH 07
* PATCH 16: Latent bug in QAPISchema.check() fixed
* PATCH 18: New
* PATCH 19-20,22-24:
  - Common code factored out into QAPISchemaMonolithicCVisitor
    and QAPISchemaModularCVisitor
  - All code-generating visitors covered
  - Consistent parameter order prefix, opt_builtins
  - R-bys dropped
* PATCH 25-29: New

Markus Armbruster (29):
  Include qapi/qmp/qerror.h exactly where needed
  qapi: Streamline boilerplate comment generation
  qapi: Generate up-to-date copyright notice
  qapi: Rename variable holding the QAPISchemaGenFOOVisitor
  qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc
  qapi: Reduce use of global variables in generators some
  qapi: Turn generators into modules
  qapi-gen: New common driver for code and doc generators
  qapi-gen: Convert from getopt to argparse
  qapi: Touch generated files only when they change
  qapi: Improve include file name reporting in error messages
  qapi/common: Eliminate QAPISchema.exprs
  qapi: Lift error reporting from QAPISchema.__init__() to callers
  qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
  qapi: Record 'include' directives in parse tree
  qapi: Generate in source order
  qapi: Record 'include' directives in intermediate representation
  qapi: Rename generated qmp-marshal.c to qmp-commands.c
  qapi: Make code-generating visitors use QAPIGen more
  qapi/types qapi/visit: Generate built-in stuff into separate files
  qapi/common: Fix guardname() for funny filenames
  qapi: Generate separate .h, .c for each module
  Include less of the generated modular QAPI headers
  qapi: Empty out qapi-schema.json
  docs/devel/writing-qmp-commands: Update for modular QAPI
  docs: Correct outdated information on QAPI
  qapi: Move qapi-schema.json to qapi/, rename generated files
  Fix up dangling references to qmp-commands.* in comment and doc
  qapi: Don't create useless directory qapi-generated

 .gitignore                                         |  80 ++++-
 Makefile                                           | 233 ++++++++++----
 Makefile.objs                                      |  80 ++++-
 arch_init.c                                        |   2 +-
 backends/cryptodev.c                               |   1 -
 backends/hostmem.c                                 |   3 +-
 balloon.c                                          |   2 +-
 block.c                                            |   2 -
 block/block-backend.c                              |   2 +-
 block/crypto.c                                     |   2 +-
 block/iscsi.c                                      |   2 +-
 block/nbd.c                                        |   2 +-
 block/nfs.c                                        |   2 +-
 block/qapi.c                                       |   4 +-
 block/qcow2.c                                      |   4 +-
 block/quorum.c                                     |   2 +-
 block/sheepdog.c                                   |   2 +-
 block/ssh.c                                        |   2 +-
 block/throttle-groups.c                            |   2 +-
 block/write-threshold.c                            |   4 +-
 blockdev-nbd.c                                     |   2 +-
 blockdev.c                                         |   5 +-
 blockjob.c                                         |   2 +-
 chardev/char-fe.c                                  |   2 +-
 chardev/char-ringbuf.c                             |   2 +-
 chardev/char-socket.c                              |   1 +
 chardev/char.c                                     |   4 +-
 configure                                          |   1 -
 cpus.c                                             |   4 +-
 crypto/cipherpriv.h                                |   2 +-
 docs/devel/qapi-code-gen.txt                       | 119 ++++---
 docs/devel/writing-qmp-commands.txt                |  39 +--
 docs/interop/qmp-intro.txt                         |   3 +-
 docs/xen-save-devices-state.txt                    |   3 +-
 dump.c                                             |   4 +-
 hmp.c                                              |  12 +-
 hw/acpi/core.c                                     |   4 +-
 hw/acpi/cpu.c                                      |   2 +-
 hw/acpi/memory_hotplug.c                           |   2 +-
 hw/acpi/vmgenid.c                                  |   2 +-
 hw/block/block.c                                   |   1 +
 hw/block/hd-geometry.c                             |   1 +
 hw/char/virtio-console.c                           |   2 +-
 hw/core/machine.c                                  |   2 +-
 hw/core/qdev.c                                     |   2 +-
 hw/i386/pc.c                                       |   2 +-
 hw/i386/xen/xen-hvm.c                              |   2 +-
 hw/ipmi/ipmi.c                                     |   2 +-
 hw/mem/nvdimm.c                                    |   1 -
 hw/net/rocker/qmp-norocker.c                       |   2 +-
 hw/net/rocker/rocker.c                             |   2 +-
 hw/net/rocker/rocker_fp.c                          |   2 +-
 hw/net/rocker/rocker_of_dpa.c                      |   2 +-
 hw/net/virtio-net.c                                |   2 +-
 hw/pci/pci-stub.c                                  |   2 +-
 hw/pci/pci.c                                       |   2 +-
 hw/ppc/spapr_rtas.c                                |   1 -
 hw/ppc/spapr_rtc.c                                 |   2 +-
 hw/s390x/s390-skeys.c                              |   2 +-
 hw/timer/mc146818rtc.c                             |   4 +-
 hw/tpm/tpm_emulator.c                              |   1 +
 hw/tpm/tpm_passthrough.c                           |   1 +
 hw/virtio/virtio-balloon.c                         |   2 +-
 hw/watchdog/watchdog.c                             |   4 +-
 include/block/block.h                              |   2 +-
 include/block/dirty-bitmap.h                       |   2 +-
 include/block/nbd.h                                |   2 +-
 include/chardev/char.h                             |   1 +
 include/crypto/cipher.h                            |   2 +-
 include/crypto/hash.h                              |   2 +-
 include/crypto/hmac.h                              |   2 +-
 include/crypto/secret.h                            |   1 +
 include/crypto/tlscreds.h                          |   1 +
 include/hw/block/block.h                           |   2 +-
 include/hw/block/fdc.h                             |   2 +-
 include/hw/ppc/spapr_drc.h                         |   1 +
 include/hw/qdev-properties.h                       |   2 +
 include/io/dns-resolver.h                          |   1 +
 include/migration/colo.h                           |   2 +-
 include/migration/failover.h                       |   2 +-
 include/migration/global_state.h                   |   1 +
 include/monitor/monitor.h                          |   1 +
 include/net/filter.h                               |   1 +
 include/net/net.h                                  |   2 +-
 include/qapi/clone-visitor.h                       |   1 -
 include/qapi/error.h                               |   2 +-
 include/qapi/qmp/qobject.h                         |   2 +-
 include/qapi/visitor.h                             |   2 +-
 include/qemu/sockets.h                             |   2 +-
 include/qemu/throttle.h                            |   2 +-
 include/qom/cpu.h                                  |   1 +
 include/qom/object.h                               |   2 +-
 include/sysemu/arch_init.h                         |   2 +-
 include/sysemu/balloon.h                           |   2 +-
 include/sysemu/dump.h                              |   2 +
 include/sysemu/hostmem.h                           |   1 +
 include/sysemu/replay.h                            |   2 +
 include/sysemu/sysemu.h                            |   1 +
 include/sysemu/tpm.h                               |   1 +
 include/sysemu/watchdog.h                          |   2 +-
 include/ui/input.h                                 |   2 +-
 io/channel-socket.c                                |   1 +
 io/dns-resolver.c                                  |   1 +
 iothread.c                                         |   2 +-
 migration/colo-failover.c                          |   2 +-
 migration/colo.c                                   |   2 +-
 migration/migration.c                              |   4 +-
 migration/migration.h                              |   1 +
 migration/ram.c                                    |   2 +-
 migration/ram.h                                    |   2 +-
 migration/savevm.c                                 |   3 +-
 monitor.c                                          |   8 +-
 net/colo-compare.c                                 |   1 -
 net/filter-buffer.c                                |   2 +-
 net/filter-mirror.c                                |   1 -
 net/filter-rewriter.c                              |   1 -
 net/net.c                                          |   4 +-
 net/tap_int.h                                      |   2 +-
 net/vhost-user.c                                   |   2 +-
 numa.c                                             |   4 +-
 qapi-schema.json => qapi/misc.json                 | 105 +-----
 qapi/qapi-schema.json                              |  95 ++++++
 qapi/run-state.json                                |  10 +
 qdev-monitor.c                                     |   2 +-
 qemu-img.c                                         |   2 +-
 qga/Makefile.objs                                  |   2 +-
 qga/commands-posix.c                               |   2 +-
 qga/commands-win32.c                               |   2 +-
 qga/commands.c                                     |   2 +-
 qga/main.c                                         |   2 +-
 qmp.c                                              |  18 +-
 qom/object.c                                       |   2 +-
 qom/object_interfaces.c                            |   2 +-
 replay/replay-input.c                              |   1 +
 replication.h                                      |   1 +
 scripts/qapi-gen.py                                |  57 ++++
 scripts/qapi/__init__.py                           |   0
 scripts/{qapi-commands.py => qapi/commands.py}     | 155 ++++-----
 scripts/{qapi.py => qapi/common.py}                | 355 +++++++++++++--------
 scripts/{qapi2texi.py => qapi/doc.py}              |  92 +++---
 scripts/{qapi-event.py => qapi/events.py}          | 128 +++-----
 scripts/{qapi-introspect.py => qapi/introspect.py} | 123 +++----
 scripts/{qapi-types.py => qapi/types.py}           | 185 ++++-------
 scripts/{qapi-visit.py => qapi/visit.py}           | 189 ++++-------
 stubs/tpm.c                                        |   3 +-
 stubs/uuid.c                                       |   2 +-
 stubs/vmgenid.c                                    |   2 +-
 stubs/xen-hvm.c                                    |   2 +-
 target/arm/monitor.c                               |   3 +-
 target/i386/cpu.c                                  |   4 +-
 target/s390x/kvm.c                                 |   1 -
 tests/.gitignore                                   |   9 +-
 tests/Makefile.include                             |  70 ++--
 tests/qapi-schema/comments.out                     |   3 +-
 tests/qapi-schema/doc-bad-section.out              |   5 +-
 tests/qapi-schema/doc-good.out                     |  33 +-
 tests/qapi-schema/doc-good.texi                    |   3 +-
 tests/qapi-schema/empty.out                        |   2 +-
 tests/qapi-schema/event-case.out                   |   3 +-
 tests/qapi-schema/ident-with-escape.out            |   7 +-
 tests/qapi-schema/include-no-file.err              |   2 +-
 tests/qapi-schema/include-relpath.out              |   7 +-
 tests/qapi-schema/include-repetition.out           |  12 +-
 tests/qapi-schema/include-simple.out               |   5 +-
 tests/qapi-schema/indented-expr.out                |   3 +-
 tests/qapi-schema/qapi-schema-test.out             | 321 +++++++++----------
 tests/qapi-schema/test-qapi.py                     |  19 +-
 tests/qmp-test.c                                   |   3 +-
 tests/test-char.c                                  |   2 +-
 tests/{test-qmp-commands.c => test-qmp-cmds.c}     |   2 +-
 tests/test-qmp-event.c                             |   3 +-
 tests/test-qobject-input-visitor.c                 |   8 +-
 tests/test-visitor-serialization.c                 |   1 -
 tpm.c                                              |   3 +-
 trace/qmp.c                                        |   2 +-
 ui/cocoa.m                                         |   2 +-
 ui/console.c                                       |   2 +-
 ui/gtk.c                                           |   2 +-
 ui/input-legacy.c                                  |   2 +-
 ui/input.c                                         |   2 +-
 ui/spice-core.c                                    |   4 +-
 ui/vnc.c                                           |   4 +-
 ui/vnc.h                                           |   1 +
 util/qemu-config.c                                 |   2 +-
 util/qemu-sockets.c                                |   2 +-
 vl.c                                               |   8 +-
 186 files changed, 1556 insertions(+), 1352 deletions(-)
 rename qapi-schema.json => qapi/misc.json (95%)
 create mode 100644 qapi/qapi-schema.json
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (67%)
 rename scripts/{qapi.py => qapi/common.py} (90%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (78%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (61%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (71%)
 rename scripts/{qapi-types.py => qapi/types.py} (53%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (64%)
 rename tests/{test-qmp-commands.c => test-qmp-cmds.c} (99%)

-- 
2.13.6


Re: [Qemu-devel] [PATCH v2 00/29] Modularize generated QAPI code
Posted by no-reply@patchew.org 6 years, 1 month ago
Hi,

This series failed docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180211093607.27351-1-armbru@redhat.com
Subject: [Qemu-devel] [PATCH v2 00/29] Modularize generated QAPI code

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
88c36fed50 qapi: Don't create useless directory qapi-generated
8695c58a23 Fix up dangling references to qmp-commands.* in comment and doc
cb0173d2de qapi: Move qapi-schema.json to qapi/, rename generated files
0163832937 docs: Correct outdated information on QAPI
0e8593c3cc docs/devel/writing-qmp-commands: Update for modular QAPI
610c107fba qapi: Empty out qapi-schema.json
978b56117d Include less of the generated modular QAPI headers
b9fbec1f14 qapi: Generate separate .h, .c for each module
349f60fd78 qapi/common: Fix guardname() for funny filenames
7d0c330c11 qapi/types qapi/visit: Generate built-in stuff into separate files
6be749acd0 qapi: Make code-generating visitors use QAPIGen more
26fb1118b1 qapi: Rename generated qmp-marshal.c to qmp-commands.c
3373ff0b73 qapi: Record 'include' directives in intermediate representation
992e298bed qapi: Generate in source order
27ec87b85b qapi: Record 'include' directives in parse tree
704eb94f85 qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
679e456092 qapi: Lift error reporting from QAPISchema.__init__() to callers
b6747d37d3 qapi/common: Eliminate QAPISchema.exprs
12c8bc8c3c qapi: Improve include file name reporting in error messages
15d87d797d qapi: Touch generated files only when they change
c1fb7b93d8 qapi-gen: Convert from getopt to argparse
8174e1f7f9 qapi-gen: New common driver for code and doc generators
76b229359e qapi: Turn generators into modules
e9687814c8 qapi: Reduce use of global variables in generators some
df03d3d345 qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc
8dcb95864b qapi: Rename variable holding the QAPISchemaGenFOOVisitor
176807c729 qapi: Generate up-to-date copyright notice
57f04f7bcd qapi: Streamline boilerplate comment generation
dc8f4505cb Include qapi/qmp/qerror.h exactly where needed

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-7cnnoaxn/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   fedora
  GEN     /var/tmp/patchew-tester-tmp-7cnnoaxn/src/docker-src.2018-02-11-05.15.21.19660/qemu.tar
Cloning into '/var/tmp/patchew-tester-tmp-7cnnoaxn/src/docker-src.2018-02-11-05.15.21.19660/qemu.tar.vroot'...
done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-7cnnoaxn/src/docker-src.2018-02-11-05.15.21.19660/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/var/tmp/patchew-tester-tmp-7cnnoaxn/src/docker-src.2018-02-11-05.15.21.19660/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPY    RUNNER
    RUN test-mingw in qemu:fedora 
Packages installed:
PyYAML-3.12-5.fc27.x86_64
SDL-devel-1.2.15-29.fc27.x86_64
bc-1.07.1-3.fc27.x86_64
bison-3.0.4-8.fc27.x86_64
bzip2-1.0.6-24.fc27.x86_64
ccache-3.3.5-1.fc27.x86_64
clang-5.0.1-1.fc27.x86_64
findutils-4.6.0-14.fc27.x86_64
flex-2.6.1-5.fc27.x86_64
gcc-7.3.1-2.fc27.x86_64
gcc-c++-7.3.1-2.fc27.x86_64
gettext-0.19.8.1-12.fc27.x86_64
git-2.14.3-2.fc27.x86_64
glib2-devel-2.54.3-2.fc27.x86_64
hostname-3.18-4.fc27.x86_64
libaio-devel-0.3.110-9.fc27.x86_64
libasan-7.3.1-2.fc27.x86_64
libfdt-devel-1.4.6-1.fc27.x86_64
libubsan-7.3.1-2.fc27.x86_64
make-4.2.1-4.fc27.x86_64
mingw32-SDL-1.2.15-9.fc27.noarch
mingw32-bzip2-1.0.6-9.fc27.noarch
mingw32-curl-7.54.1-2.fc27.noarch
mingw32-glib2-2.54.1-1.fc27.noarch
mingw32-gmp-6.1.2-2.fc27.noarch
mingw32-gnutls-3.5.13-2.fc27.noarch
mingw32-gtk2-2.24.31-4.fc27.noarch
mingw32-gtk3-3.22.16-1.fc27.noarch
mingw32-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw32-libpng-1.6.29-2.fc27.noarch
mingw32-libssh2-1.8.0-3.fc27.noarch
mingw32-libtasn1-4.13-1.fc27.noarch
mingw32-nettle-3.3-3.fc27.noarch
mingw32-pixman-0.34.0-3.fc27.noarch
mingw32-pkg-config-0.28-9.fc27.x86_64
mingw64-SDL-1.2.15-9.fc27.noarch
mingw64-bzip2-1.0.6-9.fc27.noarch
mingw64-curl-7.54.1-2.fc27.noarch
mingw64-glib2-2.54.1-1.fc27.noarch
mingw64-gmp-6.1.2-2.fc27.noarch
mingw64-gnutls-3.5.13-2.fc27.noarch
mingw64-gtk2-2.24.31-4.fc27.noarch
mingw64-gtk3-3.22.16-1.fc27.noarch
mingw64-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw64-libpng-1.6.29-2.fc27.noarch
mingw64-libssh2-1.8.0-3.fc27.noarch
mingw64-libtasn1-4.13-1.fc27.noarch
mingw64-nettle-3.3-3.fc27.noarch
mingw64-pixman-0.34.0-3.fc27.noarch
mingw64-pkg-config-0.28-9.fc27.x86_64
nettle-devel-3.4-1.fc27.x86_64
perl-5.26.1-402.fc27.x86_64
pixman-devel-0.34.0-4.fc27.x86_64
python3-3.6.2-13.fc27.x86_64
sparse-0.5.1-2.fc27.x86_64
tar-1.29-7.fc27.x86_64
which-2.21-4.fc27.x86_64
zlib-devel-1.2.11-4.fc27.x86_64

Environment variables:
TARGET_LIST=
PACKAGES=ccache gettext git tar PyYAML sparse flex bison python3 bzip2 hostname     glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel     gcc gcc-c++ clang make perl which bc findutils libaio-devel     nettle-devel libasan libubsan     mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config     mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1     mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2     mingw32-bzip2     mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config     mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1     mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2     mingw64-bzip2
J=8
V=
HOSTNAME=8e24a3c7cddb
DEBUG=
SHOW_ENV=1
PWD=/
HOME=/root
CCACHE_DIR=/var/tmp/ccache
DISTTAG=f27container
QEMU_CONFIGURE_OPTS=--python=/usr/bin/python3
FGC=f27
TEST_DIR=/tmp/qemu-test
SHLVL=1
FEATURES=mingw clang pyyaml asan dtc
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAKEFLAGS= -j8
EXTRA_CONFIGURE_OPTS=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --python=/usr/bin/python3 --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0
Install prefix    /tmp/qemu-test/install
BIOS directory    /tmp/qemu-test/install
firmware path     /tmp/qemu-test/install/share/qemu-firmware
binary directory  /tmp/qemu-test/install
library directory /tmp/qemu-test/install/lib
module directory  /tmp/qemu-test/install/lib
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory  /tmp/qemu-test/install
local state directory   queried at runtime
Windows SDK       no
Source path       /tmp/qemu-test/src
GIT binary        git
GIT submodules    
C compiler        x86_64-w64-mingw32-gcc
Host C compiler   cc
C++ compiler      x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1  -I$(SRC_PATH)/dtc/libfdt -Werror -DHAS_LIBSSH2_SFTP_FSYNC -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -m64 -mcx16 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -I/usr/x86_64-w64-mingw32/sys-root/mingw/include   -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 
LDFLAGS           -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g 
make              make
install           install
python            /usr/bin/python3 -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
SDL support       yes (1.2.15)
GTK support       yes (2.24.31)
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    no
virgl support     no
curl support      yes
mingw32 support   yes
Audio drivers     dsound
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
Multipath support no
VNC support       yes
VNC SASL support  no
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               no
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support no
Install blobs     yes
KVM support       no
HAX support       yes
HVF support       no
WHPX support      no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
malloc trim support no
RDMA support      no
fdt support       yes
preadv support    no
fdatasync         no
madvise           no
posix_madvise     no
libcap-ng support no
vhost-net support no
vhost-scsi support no
vhost-vsock support no
vhost-user support no
Trace backends    simple
Trace output file trace-<pid>
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info yes
QGA MSI support   no
seccomp support   no
coroutine backend win32
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
TPM emulator      no
QOM debugging     yes
Live block migration yes
lzo support       no
snappy support    no
bzip2 support     yes
NUMA host support no
libxml2           no
tcmalloc support  no
jemalloc support  no
avx2 optimization yes
replication support yes
VxHS block device no
capstone          no

WARNING: Use of GTK 2.0 is deprecated and will be removed in
WARNING: future releases. Please switch to using GTK 3.0

WARNING: Use of SDL 1.2 is deprecated and will be removed in
WARNING: future releases. Please switch to using SDL 2.0
  GEN     config-host.h
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     qemu-options.def
  GEN     qapi-gen
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.h
  GEN     x86_64-softmmu/config-devices.mak
  GEN     trace/generated-helpers.c
  GEN     aarch64-softmmu/config-devices.mak
  GEN     module_block.h
  GEN     ui/input-keymap-atset1-to-qcode.c
  GEN     ui/input-keymap-linux-to-qcode.c
  GEN     ui/input-keymap-qcode-to-atset1.c
  GEN     ui/input-keymap-qcode-to-atset2.c
  GEN     ui/input-keymap-qcode-to-atset3.c
Traceback (most recent call last):
  File "/tmp/qemu-test/src/scripts/qapi-gen.py", line 11, in <module>
    from qapi.common import QAPIError, QAPISchema
  File "/tmp/qemu-test/src/scripts/qapi/common.py", line 2002
    fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0666)
                                                      ^
SyntaxError: invalid token
  GEN     ui/input-keymap-qcode-to-linux.c
make: *** [Makefile:665: qapi-gen-timestamp] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 407, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 404, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 261, in run
    return Docker().run(argv, args.keep, quiet=args.quiet)
  File "./tests/docker/docker.py", line 229, in run
    quiet=quiet)
  File "./tests/docker/docker.py", line 147, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=7c5f6f3e0f1411e8af5c52540069c830', '-u', '0', '--security-opt', 'seccomp=unconfined', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-7cnnoaxn/src/docker-src.2018-02-11-05.15.21.19660:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2
make[1]: *** [tests/docker/Makefile.include:129: docker-run] Error 1
make: *** [tests/docker/Makefile.include:163: docker-run-test-mingw@fedora] Error 2

real	1m0.694s
user	0m4.526s
sys	0m3.165s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
Re: [Qemu-devel] [PATCH v2 00/29] Modularize generated QAPI code
Posted by Eric Blake 6 years ago
On 02/11/2018 03:35 AM, Markus Armbruster wrote:
> Our qapi-schema.json is composed of modules connected by include
> directives, but the generated code is monolithic all the same: one
> qapi-types.h with all the types, one qapi-visit.h with all the
> visitors, and so forth.  These monolithic headers get included all
> over the place.  In my "build everything" tree, adding a QAPI type
> recompiles about 4800 out of 5100 objects.
> 
> We wouldn't write such monolithic headers by hand.  It stands to
> reason that we shouldn't generate them, either.
> 
> This series' basic idea is to split up generated headers to mirror the
> schema's modular structure: one header per module.  That way, you can
> include just what you need.
> 
> The compile-time improvements can be massive.  Before this series, any
> QAPI schema change recompiles some 4800 out of 5100 objects in my
> "build everything" tree.  Afterwards, adding a type to
> qapi/migration.json recompiles a bit over 100, and a documentation
> change no longer recompiles anything.
> 
> Related: Marc-André's 'unit' pragma proposal.  That's a different way
> to split off parts of the generated code, motivated by the desire to
> use poisoned identifiers such as TARGET_I386.  I noted in my review of
> v3 that I "can either accept it, or come up with a better solution."
> This is my attempt at a better solution.  It's a bit more ambitious,
> and thus more useful (I hope).  The pragma has one theoretical
> advantage, though: you can modularize the generated output in
> different ways than the input.  The patches using don't do that,
> however.
> 

>   186 files changed, 1556 insertions(+), 1352 deletions(-)

I've posted a staging version of this at:

http://repo.or.cz/qemu/ericb.git qapi

Barring any last-minute comments, or a need to rebase for other branches 
going in, I'll submit a pull request later today.


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org