[v3] QAPI patches for 2018-03-12, 2.12 softfreeze

[Qemu-devel] [PULL v3 00/38] QAPI patches for 2018-03-12, 2.12 softfreeze

Posted by Eric Blake 7 years, 6 months ago

The following changes since commit 2c8cfc0b52b5a4d123c26c0b5fdf941be24805be:

  Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2018-03-19 11:44:26 +0000)

are available in the Git repository at:

  git://repo.or.cz/qemu/ericb.git tags/pull-qapi-2018-03-12-v3

for you to fetch changes up to ebf1105a7086897da6963edefdb247c80f605424:

  qapi: Pass '-u' when doing non-silent diff (2018-03-19 11:32:36 -0500)

in v3:
rebase to master (a few more qobject_to_qdict() uses crept in)
squash in Peter Xu's fixup for test failures
[if tests still fail, then I will send a v4 without OOB from 2.12]
sending only the patches that changed from v2

----------------------------------------------------------------
qapi patches for 2018-03-12, 2.12 softfreeze

- Marc-André Lureau: 0/4 qapi: generate a literal qobject for introspection
- Max Reitz: 0/7 block: Handle null backing link
- Daniel P. Berrange: chardev: tcp: postpone TLS work until machine done
- Peter Xu: 00/23 QMP: out-of-band (OOB) execution support
- Vladimir Sementsov-Ogievskiy: 0/2 block latency histogram
- Eric Blake: qapi: Pass '-u' when doing non-silent diff

----------------------------------------------------------------
Daniel P. Berrangé (1):
      chardev: tcp: postpone TLS work until machine done

Eric Blake (2):
      qmp: add new event "command-dropped"
      qapi: Pass '-u' when doing non-silent diff

Marc-André Lureau (4):
      qapi2texi: minor python code simplification
      qlit: use QType instead of int
      qlit: add qobject_from_qlit()
      qapi: generate a literal qobject for introspection

Max Reitz (7):
      compiler: Add QEMU_BUILD_BUG_MSG() macro
      qapi: Add qobject_to()
      qapi: Replace qobject_to_X(o) by qobject_to(X, o)
      qapi: Remove qobject_to_X() functions
      qapi: Make more of qobject_to()
      block: Handle null backing link
      block: Deprecate "backing": ""

Peter Xu (22):
      docs: update QMP documents for OOB commands
      qobject: introduce qstring_get_try_str()
      qobject: introduce qobject_get_try_str()
      qobject: let object_property_get_str() use new API
      monitor: move skip_flush into monitor_data_init
      monitor: move the cur_mon hack deeper for QMP
      monitor: unify global init
      monitor: let mon_list be tail queue
      monitor: allow using IO thread for parsing
      qmp: introduce QMPCapability
      monitor: introduce monitor_qmp_respond()
      monitor: let suspend_cnt be thread safe
      monitor: let suspend/resume work even with QMPs
      monitor: separate QMP parser and dispatcher
      monitor: send event when command queue full
      qapi: introduce new cmd option "allow-oob"
      qmp: support out-of-band (oob) execution
      qmp: isolate responses into io thread
      monitor: enable IO thread for (qmp & !mux) typed
      qmp: add command "x-oob-test"
      tests: qmp-test: verify command batching
      tests: qmp-test: add oob test

Vladimir Sementsov-Ogievskiy (2):
      block/accounting: introduce latency histogram
      qapi: add block latency histogram interface

 docs/devel/qapi-code-gen.txt        |  87 ++++-
 docs/interop/qmp-spec.txt           |  36 +-
 qemu-doc.texi                       |   7 +
 qapi/block-core.json                | 115 +++++-
 qapi/introspect.json                |   6 +-
 qapi/misc.json                      |  87 ++++-
 scripts/qapi/commands.py            |  18 +-
 scripts/qapi/common.py              |  15 +-
 scripts/qapi/doc.py                 |   9 +-
 scripts/qapi/introspect.py          |  83 +++--
 include/block/accounting.h          |  35 ++
 include/monitor/monitor.h           |   2 +-
 include/qapi/qmp/dispatch.h         |   7 +-
 include/qapi/qmp/qbool.h            |   1 -
 include/qapi/qmp/qdict.h            |   1 -
 include/qapi/qmp/qlist.h            |   1 -
 include/qapi/qmp/qlit.h             |   4 +-
 include/qapi/qmp/qnum.h             |   1 -
 include/qapi/qmp/qobject.h          |  30 ++
 include/qapi/qmp/qstring.h          |   3 +-
 include/qemu/compiler.h             |  12 +-
 vl.c                                |   7 +-
 tests/libqtest.c                    |   6 +-
 block.c                             |  13 +-
 block/accounting.c                  |  91 +++++
 block/parallels.c                   |   2 +-
 block/qapi.c                        |  53 ++-
 block/qcow.c                        |   2 +-
 block/qcow2.c                       |   2 +-
 block/qed.c                         |   2 +-
 block/rbd.c                         |   8 +-
 block/sheepdog.c                    |   2 +-
 block/vhdx.c                        |   2 +-
 block/vpc.c                         |   2 +-
 blockdev.c                          |  64 +++-
 chardev/char-socket.c               |  10 +
 hw/i386/acpi-build.c                |  16 +-
 monitor.c                           | 706 +++++++++++++++++++++++++++++++-----
 qapi/qmp-dispatch.c                 |  35 +-
 qapi/qobject-input-visitor.c        |  24 +-
 qapi/qobject-output-visitor.c       |   4 +-
 qga/main.c                          |   2 +-
 qmp.c                               |  18 +-
 qobject/json-parser.c               |  13 +-
 qobject/qbool.c                     |  15 +-
 qobject/qdict.c                     |  65 ++--
 qobject/qjson.c                     |  10 +-
 qobject/qlist.c                     |  17 +-
 qobject/qlit.c                      |  47 ++-
 qobject/qnum.c                      |  17 +-
 qobject/qstring.c                   |  38 +-
 qom/object.c                        |  15 +-
 target/i386/cpu.c                   |   2 +-
 target/s390x/cpu_models.c           |   2 +-
 tests/check-qdict.c                 |  20 +-
 tests/check-qjson.c                 |  41 ++-
 tests/check-qlist.c                 |   4 +-
 tests/check-qlit.c                  |  30 +-
 tests/check-qnum.c                  |   4 +-
 tests/check-qobject.c               |   2 +-
 tests/check-qstring.c               |   2 +-
 tests/device-introspect-test.c      |  14 +-
 tests/numa-test.c                   |   8 +-
 tests/qmp-test.c                    |  97 ++++-
 tests/qom-test.c                    |   4 +-
 tests/test-char.c                   |   2 +-
 tests/test-keyval.c                 |   8 +-
 tests/test-qga.c                    |  19 +-
 tests/test-qmp-cmds.c               |  12 +-
 tests/test-qmp-event.c              |  16 +-
 tests/test-qobject-input-visitor.c  |  21 +-
 tests/test-qobject-output-visitor.c |  54 +--
 tests/test-x86-cpuid-compat.c       |  17 +-
 util/keyval.c                       |   4 +-
 util/qemu-config.c                  |   2 +-
 util/qemu-option.c                  |   6 +-
 qemu-options.hx                     |   4 +-
 tests/Makefile.include              |   8 +-
 tests/qapi-schema/test-qapi.py      |   2 +-
 tests/qemu-iotests/089              |  20 +
 tests/qemu-iotests/089.out          |   8 +
 trace-events                        |   3 +
 82 files changed, 1810 insertions(+), 494 deletions(-)

-- 
2.14.3

Re: [Qemu-devel] [PULL v3 00/38] QAPI patches for 2018-03-12, 2.12 softfreeze

Posted by Peter Maydell 7 years, 6 months ago

On 19 March 2018 at 16:41, Eric Blake <eblake@redhat.com> wrote:
> The following changes since commit 2c8cfc0b52b5a4d123c26c0b5fdf941be24805be:
>
>   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2018-03-19 11:44:26 +0000)
>
> are available in the Git repository at:
>
>   git://repo.or.cz/qemu/ericb.git tags/pull-qapi-2018-03-12-v3
>
> for you to fetch changes up to ebf1105a7086897da6963edefdb247c80f605424:
>
>   qapi: Pass '-u' when doing non-silent diff (2018-03-19 11:32:36 -0500)
>
> in v3:
> rebase to master (a few more qobject_to_qdict() uses crept in)
> squash in Peter Xu's fixup for test failures
> [if tests still fail, then I will send a v4 without OOB from 2.12]
> sending only the patches that changed from v2
>
> ----------------------------------------------------------------
> qapi patches for 2018-03-12, 2.12 softfreeze
>
> - Marc-André Lureau: 0/4 qapi: generate a literal qobject for introspection
> - Max Reitz: 0/7 block: Handle null backing link
> - Daniel P. Berrange: chardev: tcp: postpone TLS work until machine done
> - Peter Xu: 00/23 QMP: out-of-band (OOB) execution support
> - Vladimir Sementsov-Ogievskiy: 0/2 block latency histogram
> - Eric Blake: qapi: Pass '-u' when doing non-silent diff

The assert seems to have gone away, but I get over 10,000 new
clang sanitizer runtime errors like

  /visitor/input/large_number:                                         OK
  /visitor/input/number_keyval:                                        OK
  /visitor/input/number_str_keyval:                                    OK
  /visitor/input/number_str_fail:
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:550:12:
runtime error: member access within null pointer of type 'QNum' (aka
'struct QNum')
OK
  /visitor/input/size_str_keyval:                                      OK
  /visitor/input/size_str_fail:
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:433:12:
runtime error: member access within null pointer of type 'QNum' (aka
'struct QNum')
OK
  /visitor/input/string:                                               OK
  /visitor/input/enum:                                                 OK
  /visitor/input/struct:                                               OK
  /visitor/input/struct-nested:                                        OK
  /visitor/input/list:                                                 OK
  /visitor/input/any:                                                  OK
  /visitor/input/null:                                                 OK
  /visitor/input/union-flat:                                           OK
  /visitor/input/alternate:                                            OK
  /visitor/input/errors:
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:521:12:
runtime error: member access within null pointer of type 'QString'
(aka 'struct QString')
OK
  /visitor/input/wrong-type:                                           OK
  /visitor/input/alternate-number:                                     OK
  /visitor/input/qapi-introspect:                                      OK


Here's some backtraces for some of those:

/visitor/input/int_keyval:
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:188:12:
runtime error: member access within null pointer of type 'QString'
(aka 'struct QString')
    #0 0x56375e642549 in qobject_input_get_keyval
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:188:12
    #1 0x56375e63f76b in qobject_input_type_int64_keyval
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:409:23
    #2 0x56375e63c13d in visit_type_int
/home/petmay01/linaro/qemu-for-merges/qapi/qapi-visit-core.c:150:5
    #3 0x56375e61a025 in test_visitor_in_int_keyval
/home/petmay01/linaro/qemu-for-merges/tests/test-qobject-input-visitor.c:187:5
    #4 0x7f84d08df7da  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f7da)
    #5 0x7f84d08df9a2  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f9a2)
    #6 0x7f84d08df9a2  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f9a2)
    #7 0x7f84d08dfbad in g_test_run_suite
(/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6fbad)
    #8 0x7f84d08dfbd0 in g_test_run
(/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6fbd0)
    #9 0x56375e619ab6 in main
/home/petmay01/linaro/qemu-for-merges/tests/test-qobject-input-visitor.c:1385:5
    #10 0x7f84cea7482f in __libc_start_main
/build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
    #11 0x56375e5f6f88 in _start
(/home/petmay01/linaro/qemu-for-merges/build/clang/tests/test-qobject-input-visitor+0xabf88)

OK

/visitor/input/int_str_fail:
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:398:12:
runtime error: member access within null pointer of type 'QNum' (aka
'struct QNum')
    #0 0x56375e63ec7e in qobject_input_type_int64
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:398:12
    #1 0x56375e63c13d in visit_type_int
/home/petmay01/linaro/qemu-for-merges/qapi/qapi-visit-core.c:150:5
    #2 0x56375e61a149 in test_visitor_in_int_str_fail
/home/petmay01/linaro/qemu-for-merges/tests/test-qobject-input-visitor.c:212:5
    #3 0x7f84d08df7da  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f7da)
    #4 0x7f84d08df9a2  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f9a2)
    #5 0x7f84d08df9a2  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f9a2)
    #6 0x7f84d08dfbad in g_test_run_suite
(/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6fbad)
    #7 0x7f84d08dfbd0 in g_test_run
(/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6fbd0)
    #8 0x56375e619ab6 in main
/home/petmay01/linaro/qemu-for-merges/tests/test-qobject-input-visitor.c:1385:5
    #9 0x7f84cea7482f in __libc_start_main
/build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
    #10 0x56375e5f6f88 in _start
(/home/petmay01/linaro/qemu-for-merges/build/clang/tests/test-qobject-input-visitor+0xabf88)

OK


/visitor/input/bool_str_fail:
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:480:13:
runtime error: member access within null pointer of type 'QBool' (aka
'struct QBool')
    #0 0x56375e63eeb4 in qobject_input_type_bool
/home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:480:13
    #1 0x56375e63cfdd in visit_type_bool
/home/petmay01/linaro/qemu-for-merges/qapi/qapi-visit-core.c:285:5
    #2 0x56375e61a3a5 in test_visitor_in_bool_str_fail
/home/petmay01/linaro/qemu-for-merges/tests/test-qobject-input-visitor.c:262:5
    #3 0x7f84d08df7da  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f7da)
    #4 0x7f84d08df9a2  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f9a2)
    #5 0x7f84d08df9a2  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6f9a2)
    #6 0x7f84d08dfbad in g_test_run_suite
(/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6fbad)
    #7 0x7f84d08dfbd0 in g_test_run
(/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x6fbd0)
    #8 0x56375e619ab6 in main
/home/petmay01/linaro/qemu-for-merges/tests/test-qobject-input-visitor.c:1385:5
    #9 0x7f84cea7482f in __libc_start_main
/build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
    #10 0x56375e5f6f88 in _start
(/home/petmay01/linaro/qemu-for-merges/build/clang/tests/test-qobject-input-visitor+0xabf88)

OK


thanks
-- PMM

Re: [Qemu-devel] [PULL v3 00/38] QAPI patches for 2018-03-12, 2.12 softfreeze

Posted by Eric Blake 7 years, 6 months ago

On 03/19/2018 01:43 PM, Peter Maydell wrote:
> On 19 March 2018 at 16:41, Eric Blake <eblake@redhat.com> wrote:
>> The following changes since commit 2c8cfc0b52b5a4d123c26c0b5fdf941be24805be:
>>
>>    Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2018-03-19 11:44:26 +0000)
>>
>> are available in the Git repository at:
>>
>>    git://repo.or.cz/qemu/ericb.git tags/pull-qapi-2018-03-12-v3
>>
>> for you to fetch changes up to ebf1105a7086897da6963edefdb247c80f605424:
>>
>>    qapi: Pass '-u' when doing non-silent diff (2018-03-19 11:32:36 -0500)
>>
>> in v3:
>> rebase to master (a few more qobject_to_qdict() uses crept in)
>> squash in Peter Xu's fixup for test failures
>> [if tests still fail, then I will send a v4 without OOB from 2.12]
>> sending only the patches that changed from v2

> 
> The assert seems to have gone away,

Progress! Peter's OOB patches are working; your ubsan errors are from a 
different part of the series.

> but I get over 10,000 new
> clang sanitizer runtime errors like
> 
>    /visitor/input/large_number:                                         OK
>    /visitor/input/number_keyval:                                        OK
>    /visitor/input/number_str_keyval:                                    OK
>    /visitor/input/number_str_fail:
> /home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:550:12:
> runtime error: member access within null pointer of type 'QNum' (aka
> 'struct QNum')
> OK

> Here's some backtraces for some of those:
> 
> /visitor/input/int_keyval:
> /home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:188:12:
> runtime error: member access within null pointer of type 'QString'
> (aka 'struct QString')
>      #0 0x56375e642549 in qobject_input_get_keyval
> /home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:188:12
>      #1 0x56375e63f76b in qobject_input_type_int64_keyval
> /home/petmay01/linaro/qemu-for-merges/qapi/qobject-input-visitor.c:409:23
>      #2 0x56375e63c13d in visit_type_int
> /home/petmay01/linaro/qemu-for-merges/qapi/qapi-visit-core.c:150:5

Max, this is the qobject_to() macro that is making ubsan gripe :(

     qstr = qobject_to(QString, qobj);

which partially expands to

     qstr = container_of(qobject_check_type(qobj,
        QTYPE_CAST_TO_QString)) ?: QOBJECT((QString *)NULL), QString, base);

I'm guessing that container_of(NULL, type, base) is what ubsan is 
griping at?  Can we rewrite qobject_to() to only call container_of() if 
qobject_check_type() returns non-NULL?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org