[Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread

Peter Xu posted 6 patches 6 years, 7 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/1503301464-27886-1-git-send-email-peterx@redhat.com
Test FreeBSD passed
Test checkpatch passed
Test docker failed
Test s390x passed
There is a newer version of this series
docs/devel/qapi-code-gen.txt | 10 +++++-
hmp-commands.hx              |  1 +
include/qapi/qmp/dispatch.h  |  1 +
monitor.c                    | 79 ++++++++++++++++++++++++++++++++++++++++----
qapi-schema.json             |  3 +-
qapi/qmp-dispatch.c          | 26 +++++++++++++++
scripts/qapi-commands.py     | 18 +++++++---
scripts/qapi-introspect.py   |  2 +-
scripts/qapi.py              | 15 ++++++---
scripts/qapi2texi.py         |  2 +-
10 files changed, 137 insertions(+), 20 deletions(-)
[Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Peter Xu 6 years, 7 months ago
This is an extended work for migration postcopy recovery. This series
is tested with the following series to make sure it solves the monitor
hang problem that we have encountered for postcopy recovery:

  [RFC 00/29] Migration: postcopy failure recovery
  [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery

The root problem is that, monitor commands are all handled in main
loop thread now, no matter how many monitors we specify. And, if main
loop thread hangs due to some reason, all monitors will be stuck.
This can be done in reversed order as well: if any of the monitor
hangs, it will hang the main loop, and the rest of the monitors (if
there is any).

That affects postcopy recovery, since the recovery requires user input
on destination side.  If monitors hang, the destination VM dies and
lose hope for even a final recovery.

So, sometimes we need to make sure the monitor be alive, at least one
of them.

The whole idea of this series is that instead if handling monitor
commands all in main loop thread, we do it separately in per-monitor
threads.  Then, even if main loop thread hangs at any point by any
reason, per-monitor thread can still survive.  Further, we add hint in
QMP/HMP to show whether a command can be executed without QMP, if so,
we avoid taking BQL when running that command.  It greatly reduced
contention of BQL.  Now the only user of that new parameter (currently
I call it "without-bql") is "migrate-incoming" command, which is the
only command to rescue a paused postcopy migration.

However, even with the series, it does not mean that per-monitor
threads will never hang.  One example is that we can still run "info
vcpus" in per-monitor threads during a paused postcopy (in that state,
page faults are never handled, and "info cpus" will never return since
it tries to sync every vcpus).  So to make sure it does not hang, we
not only need the per-monitor thread, the user should be careful as
well on how to use it.

For postcopy recovery, we may need dedicated monitor channel for
recovery.  In other words, a destination VM that supports postcopy
recovery would possibly need:

  -qmp MAIN_CHANNEL -qmp RECOVERY_CHANNEL

Here, the MAIN_CHANNEL can be MUXed and shared by other chardev
frontends, while the RECOVERY_CHANNEL should *ONLY* be used to input
the "migrate-incoming" command (similar thing applies to HMP
channels).  As long as we are following this rule, the
RECOVERY_CHANNEL can never hang.

Some details on each patch:

Patch 1: a simple cleanup only

Patch 2: allow monitors to create per-monitor thread to handle monitor
         command requests. Since monitor is only one type of chardev
         frontend, we only do this if the backend is dedicated, say,
         if MUX is not turned on (if MUX is on, it's still using main
         loop thread).

Patch 3: based on patch 2, this patch introduced a new parameter for
         QMP commands called "without-bql", it is a hint that the
         command does not need BQL.

Patch 4: Let QMP command "migrate-incoming" avoid taking BQL.

Patch 5: Introduced sister parameter for HMP "without_bql", which
         works just like QMP "without-bql".

Patch 6: Let HMP command "migrate-incoming" avoid taking BQL.

Please review. Thanks,

Peter Xu (6):
  monitor: move skip_flush into monitor_data_init
  monitor: allow monitor to create thread to poll
  QAPI: new QMP command option "without-bql"
  migration: qmp: migrate_incoming don't need BQL
  hmp: support "without_bql"
  migration: hmp: migrate_incoming don't need BQL

 docs/devel/qapi-code-gen.txt | 10 +++++-
 hmp-commands.hx              |  1 +
 include/qapi/qmp/dispatch.h  |  1 +
 monitor.c                    | 79 ++++++++++++++++++++++++++++++++++++++++----
 qapi-schema.json             |  3 +-
 qapi/qmp-dispatch.c          | 26 +++++++++++++++
 scripts/qapi-commands.py     | 18 +++++++---
 scripts/qapi-introspect.py   |  2 +-
 scripts/qapi.py              | 15 ++++++---
 scripts/qapi2texi.py         |  2 +-
 10 files changed, 137 insertions(+), 20 deletions(-)

-- 
2.7.4


Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Mon, 08/21 15:44, Peter Xu wrote:
> This is an extended work for migration postcopy recovery. This series
> is tested with the following series to make sure it solves the monitor
> hang problem that we have encountered for postcopy recovery:
> 
>   [RFC 00/29] Migration: postcopy failure recovery
>   [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery
> 
> The root problem is that, monitor commands are all handled in main
> loop thread now, no matter how many monitors we specify. And, if main
> loop thread hangs due to some reason, all monitors will be stuck.
> This can be done in reversed order as well: if any of the monitor
> hangs, it will hang the main loop, and the rest of the monitors (if
> there is any).
> 
> That affects postcopy recovery, since the recovery requires user input
> on destination side.  If monitors hang, the destination VM dies and
> lose hope for even a final recovery.
> 
> So, sometimes we need to make sure the monitor be alive, at least one
> of them.
> 
> The whole idea of this series is that instead if handling monitor
> commands all in main loop thread, we do it separately in per-monitor
> threads.  Then, even if main loop thread hangs at any point by any
> reason, per-monitor thread can still survive.  Further, we add hint in
> QMP/HMP to show whether a command can be executed without QMP, if so,
> we avoid taking BQL when running that command.  It greatly reduced
> contention of BQL.  Now the only user of that new parameter (currently
> I call it "without-bql") is "migrate-incoming" command, which is the
> only command to rescue a paused postcopy migration.
> 
> However, even with the series, it does not mean that per-monitor
> threads will never hang.  One example is that we can still run "info
> vcpus" in per-monitor threads during a paused postcopy (in that state,
> page faults are never handled, and "info cpus" will never return since
> it tries to sync every vcpus).  So to make sure it does not hang, we
> not only need the per-monitor thread, the user should be careful as
> well on how to use it.

I think this is like saying we expect the user to understand the internals of
QEMU, unless the "rules" are clearly documented.  Taking this into account,
does it make sense to make the per-monitor thread only allow BQL-free commands?

Fam

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Peter Xu 6 years, 7 months ago
On Mon, Aug 21, 2017 at 04:58:51PM +0800, Fam Zheng wrote:
> On Mon, 08/21 15:44, Peter Xu wrote:
> > This is an extended work for migration postcopy recovery. This series
> > is tested with the following series to make sure it solves the monitor
> > hang problem that we have encountered for postcopy recovery:
> > 
> >   [RFC 00/29] Migration: postcopy failure recovery
> >   [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery
> > 
> > The root problem is that, monitor commands are all handled in main
> > loop thread now, no matter how many monitors we specify. And, if main
> > loop thread hangs due to some reason, all monitors will be stuck.
> > This can be done in reversed order as well: if any of the monitor
> > hangs, it will hang the main loop, and the rest of the monitors (if
> > there is any).
> > 
> > That affects postcopy recovery, since the recovery requires user input
> > on destination side.  If monitors hang, the destination VM dies and
> > lose hope for even a final recovery.
> > 
> > So, sometimes we need to make sure the monitor be alive, at least one
> > of them.
> > 
> > The whole idea of this series is that instead if handling monitor
> > commands all in main loop thread, we do it separately in per-monitor
> > threads.  Then, even if main loop thread hangs at any point by any
> > reason, per-monitor thread can still survive.  Further, we add hint in
> > QMP/HMP to show whether a command can be executed without QMP, if so,
> > we avoid taking BQL when running that command.  It greatly reduced
> > contention of BQL.  Now the only user of that new parameter (currently
> > I call it "without-bql") is "migrate-incoming" command, which is the
> > only command to rescue a paused postcopy migration.
> > 
> > However, even with the series, it does not mean that per-monitor
> > threads will never hang.  One example is that we can still run "info
> > vcpus" in per-monitor threads during a paused postcopy (in that state,
> > page faults are never handled, and "info cpus" will never return since
> > it tries to sync every vcpus).  So to make sure it does not hang, we
> > not only need the per-monitor thread, the user should be careful as
> > well on how to use it.
> 
> I think this is like saying we expect the user to understand the internals of
> QEMU, unless the "rules" are clearly documented.  Taking this into account,
> does it make sense to make the per-monitor thread only allow BQL-free commands?

I don't think users need to know the internals - they just need to be
careful on using them.  Just take the example of "info cpus": during
paused postcopy it will hang, but IMHO it does not mean that it's
illegal for user to send that command.  It's "by-design" that it'll be
stuck if one of the vcpus is stuck somewhere; it's just not the
correct way to use it when the monitor is prepared for postcopy
recovery.

And IMHO we should not treat threaded monitors special - it should be
exactly the same monitor service when used with main loop thread.  It
just has its own thread to handle the requests, so it is less
dependent on main loop thread, and that's all.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Dr. David Alan Gilbert 6 years, 7 months ago
* Peter Xu (peterx@redhat.com) wrote:
> On Mon, Aug 21, 2017 at 04:58:51PM +0800, Fam Zheng wrote:
> > On Mon, 08/21 15:44, Peter Xu wrote:
> > > This is an extended work for migration postcopy recovery. This series
> > > is tested with the following series to make sure it solves the monitor
> > > hang problem that we have encountered for postcopy recovery:
> > > 
> > >   [RFC 00/29] Migration: postcopy failure recovery
> > >   [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery
> > > 
> > > The root problem is that, monitor commands are all handled in main
> > > loop thread now, no matter how many monitors we specify. And, if main
> > > loop thread hangs due to some reason, all monitors will be stuck.
> > > This can be done in reversed order as well: if any of the monitor
> > > hangs, it will hang the main loop, and the rest of the monitors (if
> > > there is any).
> > > 
> > > That affects postcopy recovery, since the recovery requires user input
> > > on destination side.  If monitors hang, the destination VM dies and
> > > lose hope for even a final recovery.
> > > 
> > > So, sometimes we need to make sure the monitor be alive, at least one
> > > of them.
> > > 
> > > The whole idea of this series is that instead if handling monitor
> > > commands all in main loop thread, we do it separately in per-monitor
> > > threads.  Then, even if main loop thread hangs at any point by any
> > > reason, per-monitor thread can still survive.  Further, we add hint in
> > > QMP/HMP to show whether a command can be executed without QMP, if so,
> > > we avoid taking BQL when running that command.  It greatly reduced
> > > contention of BQL.  Now the only user of that new parameter (currently
> > > I call it "without-bql") is "migrate-incoming" command, which is the
> > > only command to rescue a paused postcopy migration.
> > > 
> > > However, even with the series, it does not mean that per-monitor
> > > threads will never hang.  One example is that we can still run "info
> > > vcpus" in per-monitor threads during a paused postcopy (in that state,
> > > page faults are never handled, and "info cpus" will never return since
> > > it tries to sync every vcpus).  So to make sure it does not hang, we
> > > not only need the per-monitor thread, the user should be careful as
> > > well on how to use it.
> > 
> > I think this is like saying we expect the user to understand the internals of
> > QEMU, unless the "rules" are clearly documented.  Taking this into account,
> > does it make sense to make the per-monitor thread only allow BQL-free commands?
> 
> I don't think users need to know the internals - they just need to be
> careful on using them.  Just take the example of "info cpus": during
> paused postcopy it will hang, but IMHO it does not mean that it's
> illegal for user to send that command.  It's "by-design" that it'll be
> stuck if one of the vcpus is stuck somewhere; it's just not the
> correct way to use it when the monitor is prepared for postcopy
> recovery.
> 
> And IMHO we should not treat threaded monitors special - it should be
> exactly the same monitor service when used with main loop thread.  It
> just has its own thread to handle the requests, so it is less
> dependent on main loop thread, and that's all.

From previous discussions we've had, one suggestion was to have some
type of 'safe' command; once issued in a thread, the monitor thread
would only allow other lock-free commands to be issued; it stops any
accidents of them issuing unsafe commands.

Dave

> Thanks,
> 
> -- 
> Peter Xu
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Mon, 08/21 11:17, Dr. David Alan Gilbert wrote:
> From previous discussions we've had, one suggestion was to have some
> type of 'safe' command; once issued in a thread, the monitor thread
> would only allow other lock-free commands to be issued; it stops any
> accidents of them issuing unsafe commands.

I'm not sure I understand. If the 'safe' command is not issued, users are
allowed to do unsafe things? What are the possible consequences of those
'unsafe' commands? Errors/hangs/crashes?

Fam

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Dr. David Alan Gilbert 6 years, 7 months ago
* Fam Zheng (famz@redhat.com) wrote:
> On Mon, 08/21 11:17, Dr. David Alan Gilbert wrote:
> > From previous discussions we've had, one suggestion was to have some
> > type of 'safe' command; once issued in a thread, the monitor thread
> > would only allow other lock-free commands to be issued; it stops any
> > accidents of them issuing unsafe commands.
> 
> I'm not sure I understand. If the 'safe' command is not issued, users are
> allowed to do unsafe things? What are the possible consequences of those
> 'unsafe' commands? Errors/hangs/crashes?

With or without the safe command no command could cause a crash.
However, a command might try and take the bql and block waiting for it.
With the 'safe' command only those commands that were declared as
not-wanting the lock would be allowed.

Dave

> Fam
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Mon, 08/21 18:05, Peter Xu wrote:
> On Mon, Aug 21, 2017 at 04:58:51PM +0800, Fam Zheng wrote:
> > On Mon, 08/21 15:44, Peter Xu wrote:
> > > This is an extended work for migration postcopy recovery. This series
> > > is tested with the following series to make sure it solves the monitor
> > > hang problem that we have encountered for postcopy recovery:
> > > 
> > >   [RFC 00/29] Migration: postcopy failure recovery
> > >   [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery
> > > 
> > > The root problem is that, monitor commands are all handled in main
> > > loop thread now, no matter how many monitors we specify. And, if main
> > > loop thread hangs due to some reason, all monitors will be stuck.
> > > This can be done in reversed order as well: if any of the monitor
> > > hangs, it will hang the main loop, and the rest of the monitors (if
> > > there is any).
> > > 
> > > That affects postcopy recovery, since the recovery requires user input
> > > on destination side.  If monitors hang, the destination VM dies and
> > > lose hope for even a final recovery.
> > > 
> > > So, sometimes we need to make sure the monitor be alive, at least one
> > > of them.
> > > 
> > > The whole idea of this series is that instead if handling monitor
> > > commands all in main loop thread, we do it separately in per-monitor
> > > threads.  Then, even if main loop thread hangs at any point by any
> > > reason, per-monitor thread can still survive.  Further, we add hint in
> > > QMP/HMP to show whether a command can be executed without QMP, if so,
> > > we avoid taking BQL when running that command.  It greatly reduced
> > > contention of BQL.  Now the only user of that new parameter (currently
> > > I call it "without-bql") is "migrate-incoming" command, which is the
> > > only command to rescue a paused postcopy migration.
> > > 
> > > However, even with the series, it does not mean that per-monitor
> > > threads will never hang.  One example is that we can still run "info
> > > vcpus" in per-monitor threads during a paused postcopy (in that state,
> > > page faults are never handled, and "info cpus" will never return since
> > > it tries to sync every vcpus).  So to make sure it does not hang, we
> > > not only need the per-monitor thread, the user should be careful as
> > > well on how to use it.
> > 
> > I think this is like saying we expect the user to understand the internals of
> > QEMU, unless the "rules" are clearly documented.  Taking this into account,
> > does it make sense to make the per-monitor thread only allow BQL-free commands?
> 
> I don't think users need to know the internals - they just need to be
> careful on using them.  Just take the example of "info cpus": during
> paused postcopy it will hang, but IMHO it does not mean that it's
> illegal for user to send that command.  It's "by-design" that it'll be
> stuck if one of the vcpus is stuck somewhere; it's just not the
> correct way to use it when the monitor is prepared for postcopy
> recovery.

They still need to know "what" is the correct way to use the monitor, and what
I'm saying is there doesn't seem to be an easy way for users to know exactly
what is correct. See below.

> 
> And IMHO we should not treat threaded monitors special - it should be
> exactly the same monitor service when used with main loop thread.  It
> just has its own thread to handle the requests, so it is less
> dependent on main loop thread, and that's all.

It's not that simple, I think all non-trivial commands need very careful audit
before assuming they're safe. For example many block related commands
(qmp_trasaction, for example) indirectly calls BDRV_POLL_WHILE(), which, if
called from a per-monitor thread, will enter the else branch then fail the first
assert.

Fam

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Dr. David Alan Gilbert 6 years, 7 months ago
* Fam Zheng (famz@redhat.com) wrote:
> On Mon, 08/21 18:05, Peter Xu wrote:
> > On Mon, Aug 21, 2017 at 04:58:51PM +0800, Fam Zheng wrote:
> > > On Mon, 08/21 15:44, Peter Xu wrote:
> > > > This is an extended work for migration postcopy recovery. This series
> > > > is tested with the following series to make sure it solves the monitor
> > > > hang problem that we have encountered for postcopy recovery:
> > > > 
> > > >   [RFC 00/29] Migration: postcopy failure recovery
> > > >   [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery
> > > > 
> > > > The root problem is that, monitor commands are all handled in main
> > > > loop thread now, no matter how many monitors we specify. And, if main
> > > > loop thread hangs due to some reason, all monitors will be stuck.
> > > > This can be done in reversed order as well: if any of the monitor
> > > > hangs, it will hang the main loop, and the rest of the monitors (if
> > > > there is any).
> > > > 
> > > > That affects postcopy recovery, since the recovery requires user input
> > > > on destination side.  If monitors hang, the destination VM dies and
> > > > lose hope for even a final recovery.
> > > > 
> > > > So, sometimes we need to make sure the monitor be alive, at least one
> > > > of them.
> > > > 
> > > > The whole idea of this series is that instead if handling monitor
> > > > commands all in main loop thread, we do it separately in per-monitor
> > > > threads.  Then, even if main loop thread hangs at any point by any
> > > > reason, per-monitor thread can still survive.  Further, we add hint in
> > > > QMP/HMP to show whether a command can be executed without QMP, if so,
> > > > we avoid taking BQL when running that command.  It greatly reduced
> > > > contention of BQL.  Now the only user of that new parameter (currently
> > > > I call it "without-bql") is "migrate-incoming" command, which is the
> > > > only command to rescue a paused postcopy migration.
> > > > 
> > > > However, even with the series, it does not mean that per-monitor
> > > > threads will never hang.  One example is that we can still run "info
> > > > vcpus" in per-monitor threads during a paused postcopy (in that state,
> > > > page faults are never handled, and "info cpus" will never return since
> > > > it tries to sync every vcpus).  So to make sure it does not hang, we
> > > > not only need the per-monitor thread, the user should be careful as
> > > > well on how to use it.
> > > 
> > > I think this is like saying we expect the user to understand the internals of
> > > QEMU, unless the "rules" are clearly documented.  Taking this into account,
> > > does it make sense to make the per-monitor thread only allow BQL-free commands?
> > 
> > I don't think users need to know the internals - they just need to be
> > careful on using them.  Just take the example of "info cpus": during
> > paused postcopy it will hang, but IMHO it does not mean that it's
> > illegal for user to send that command.  It's "by-design" that it'll be
> > stuck if one of the vcpus is stuck somewhere; it's just not the
> > correct way to use it when the monitor is prepared for postcopy
> > recovery.
> 
> They still need to know "what" is the correct way to use the monitor, and what
> I'm saying is there doesn't seem to be an easy way for users to know exactly
> what is correct. See below.
> 
> > 
> > And IMHO we should not treat threaded monitors special - it should be
> > exactly the same monitor service when used with main loop thread.  It
> > just has its own thread to handle the requests, so it is less
> > dependent on main loop thread, and that's all.
> 
> It's not that simple, I think all non-trivial commands need very careful audit
> before assuming they're safe. For example many block related commands
> (qmp_trasaction, for example) indirectly calls BDRV_POLL_WHILE(), which, if
> called from a per-monitor thread, will enter the else branch then fail the first
> assert.

OK, that's interesting - I'd assumed that as long as we actually held
the bql we were reasonably safe.
Can you explain what that assert is actually asserting?

Dave

> Fam
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Mon, 08/21 16:36, Dr. David Alan Gilbert wrote:
> * Fam Zheng (famz@redhat.com) wrote:
> > On Mon, 08/21 18:05, Peter Xu wrote:
> > > On Mon, Aug 21, 2017 at 04:58:51PM +0800, Fam Zheng wrote:
> > > > On Mon, 08/21 15:44, Peter Xu wrote:
> > > > > This is an extended work for migration postcopy recovery. This series
> > > > > is tested with the following series to make sure it solves the monitor
> > > > > hang problem that we have encountered for postcopy recovery:
> > > > > 
> > > > >   [RFC 00/29] Migration: postcopy failure recovery
> > > > >   [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery
> > > > > 
> > > > > The root problem is that, monitor commands are all handled in main
> > > > > loop thread now, no matter how many monitors we specify. And, if main
> > > > > loop thread hangs due to some reason, all monitors will be stuck.
> > > > > This can be done in reversed order as well: if any of the monitor
> > > > > hangs, it will hang the main loop, and the rest of the monitors (if
> > > > > there is any).
> > > > > 
> > > > > That affects postcopy recovery, since the recovery requires user input
> > > > > on destination side.  If monitors hang, the destination VM dies and
> > > > > lose hope for even a final recovery.
> > > > > 
> > > > > So, sometimes we need to make sure the monitor be alive, at least one
> > > > > of them.
> > > > > 
> > > > > The whole idea of this series is that instead if handling monitor
> > > > > commands all in main loop thread, we do it separately in per-monitor
> > > > > threads.  Then, even if main loop thread hangs at any point by any
> > > > > reason, per-monitor thread can still survive.  Further, we add hint in
> > > > > QMP/HMP to show whether a command can be executed without QMP, if so,
> > > > > we avoid taking BQL when running that command.  It greatly reduced
> > > > > contention of BQL.  Now the only user of that new parameter (currently
> > > > > I call it "without-bql") is "migrate-incoming" command, which is the
> > > > > only command to rescue a paused postcopy migration.
> > > > > 
> > > > > However, even with the series, it does not mean that per-monitor
> > > > > threads will never hang.  One example is that we can still run "info
> > > > > vcpus" in per-monitor threads during a paused postcopy (in that state,
> > > > > page faults are never handled, and "info cpus" will never return since
> > > > > it tries to sync every vcpus).  So to make sure it does not hang, we
> > > > > not only need the per-monitor thread, the user should be careful as
> > > > > well on how to use it.
> > > > 
> > > > I think this is like saying we expect the user to understand the internals of
> > > > QEMU, unless the "rules" are clearly documented.  Taking this into account,
> > > > does it make sense to make the per-monitor thread only allow BQL-free commands?
> > > 
> > > I don't think users need to know the internals - they just need to be
> > > careful on using them.  Just take the example of "info cpus": during
> > > paused postcopy it will hang, but IMHO it does not mean that it's
> > > illegal for user to send that command.  It's "by-design" that it'll be
> > > stuck if one of the vcpus is stuck somewhere; it's just not the
> > > correct way to use it when the monitor is prepared for postcopy
> > > recovery.
> > 
> > They still need to know "what" is the correct way to use the monitor, and what
> > I'm saying is there doesn't seem to be an easy way for users to know exactly
> > what is correct. See below.
> > 
> > > 
> > > And IMHO we should not treat threaded monitors special - it should be
> > > exactly the same monitor service when used with main loop thread.  It
> > > just has its own thread to handle the requests, so it is less
> > > dependent on main loop thread, and that's all.
> > 
> > It's not that simple, I think all non-trivial commands need very careful audit
> > before assuming they're safe. For example many block related commands
> > (qmp_trasaction, for example) indirectly calls BDRV_POLL_WHILE(), which, if
> > called from a per-monitor thread, will enter the else branch then fail the first
> > assert.
> 
> OK, that's interesting - I'd assumed that as long as we actually held
> the bql we were reasonably safe.
> Can you explain what that assert is actually asserting?

It's not much more than asserting qemu_mutex_iothread_locked(), the problem is
the new monitor thread breaks certain assumptions that was true.

What is interesting in this is that block layer's nested aio_poll() now not only
run in the main thread but also in the monitor thread. Bugs may hide there.  :)

That's why I suggested a "safe by default" strategy.


One step back, is it possible to "unblock" main thread even upon network issue?
What is the scenario that causes main thread hang? Is there a backtrace?

Fam

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Dr. David Alan Gilbert 6 years, 7 months ago
* Fam Zheng (famz@redhat.com) wrote:
> On Mon, 08/21 16:36, Dr. David Alan Gilbert wrote:
> > * Fam Zheng (famz@redhat.com) wrote:
> > > On Mon, 08/21 18:05, Peter Xu wrote:
> > > > On Mon, Aug 21, 2017 at 04:58:51PM +0800, Fam Zheng wrote:
> > > > > On Mon, 08/21 15:44, Peter Xu wrote:
> > > > > > This is an extended work for migration postcopy recovery. This series
> > > > > > is tested with the following series to make sure it solves the monitor
> > > > > > hang problem that we have encountered for postcopy recovery:
> > > > > > 
> > > > > >   [RFC 00/29] Migration: postcopy failure recovery
> > > > > >   [RFC 0/6] migration: re-use migrate_incoming for postcopy recovery
> > > > > > 
> > > > > > The root problem is that, monitor commands are all handled in main
> > > > > > loop thread now, no matter how many monitors we specify. And, if main
> > > > > > loop thread hangs due to some reason, all monitors will be stuck.
> > > > > > This can be done in reversed order as well: if any of the monitor
> > > > > > hangs, it will hang the main loop, and the rest of the monitors (if
> > > > > > there is any).
> > > > > > 
> > > > > > That affects postcopy recovery, since the recovery requires user input
> > > > > > on destination side.  If monitors hang, the destination VM dies and
> > > > > > lose hope for even a final recovery.
> > > > > > 
> > > > > > So, sometimes we need to make sure the monitor be alive, at least one
> > > > > > of them.
> > > > > > 
> > > > > > The whole idea of this series is that instead if handling monitor
> > > > > > commands all in main loop thread, we do it separately in per-monitor
> > > > > > threads.  Then, even if main loop thread hangs at any point by any
> > > > > > reason, per-monitor thread can still survive.  Further, we add hint in
> > > > > > QMP/HMP to show whether a command can be executed without QMP, if so,
> > > > > > we avoid taking BQL when running that command.  It greatly reduced
> > > > > > contention of BQL.  Now the only user of that new parameter (currently
> > > > > > I call it "without-bql") is "migrate-incoming" command, which is the
> > > > > > only command to rescue a paused postcopy migration.
> > > > > > 
> > > > > > However, even with the series, it does not mean that per-monitor
> > > > > > threads will never hang.  One example is that we can still run "info
> > > > > > vcpus" in per-monitor threads during a paused postcopy (in that state,
> > > > > > page faults are never handled, and "info cpus" will never return since
> > > > > > it tries to sync every vcpus).  So to make sure it does not hang, we
> > > > > > not only need the per-monitor thread, the user should be careful as
> > > > > > well on how to use it.
> > > > > 
> > > > > I think this is like saying we expect the user to understand the internals of
> > > > > QEMU, unless the "rules" are clearly documented.  Taking this into account,
> > > > > does it make sense to make the per-monitor thread only allow BQL-free commands?
> > > > 
> > > > I don't think users need to know the internals - they just need to be
> > > > careful on using them.  Just take the example of "info cpus": during
> > > > paused postcopy it will hang, but IMHO it does not mean that it's
> > > > illegal for user to send that command.  It's "by-design" that it'll be
> > > > stuck if one of the vcpus is stuck somewhere; it's just not the
> > > > correct way to use it when the monitor is prepared for postcopy
> > > > recovery.
> > > 
> > > They still need to know "what" is the correct way to use the monitor, and what
> > > I'm saying is there doesn't seem to be an easy way for users to know exactly
> > > what is correct. See below.
> > > 
> > > > 
> > > > And IMHO we should not treat threaded monitors special - it should be
> > > > exactly the same monitor service when used with main loop thread.  It
> > > > just has its own thread to handle the requests, so it is less
> > > > dependent on main loop thread, and that's all.
> > > 
> > > It's not that simple, I think all non-trivial commands need very careful audit
> > > before assuming they're safe. For example many block related commands
> > > (qmp_trasaction, for example) indirectly calls BDRV_POLL_WHILE(), which, if
> > > called from a per-monitor thread, will enter the else branch then fail the first
> > > assert.
> > 
> > OK, that's interesting - I'd assumed that as long as we actually held
> > the bql we were reasonably safe.
> > Can you explain what that assert is actually asserting?
> 
> It's not much more than asserting qemu_mutex_iothread_locked(), the problem is
> the new monitor thread breaks certain assumptions that was true.
> 
> What is interesting in this is that block layer's nested aio_poll() now not only
> run in the main thread but also in the monitor thread. Bugs may hide there.  :)
> 
> That's why I suggested a "safe by default" strategy.

OK, that's going to need some more flags somewhere; we've now
effectively got three types of command:
   a) Commands that can only run in the main thread
   b) Commands that can run in other monitor threads, but must have the bql
   c) Commands that can run in other monitor threads but don't take the
   bql

   The class (a) that you point out are a pain; arguably if we have to
split them up then perhaps we should initially only allow (c).

> One step back, is it possible to "unblock" main thread even upon network issue?
> What is the scenario that causes main thread hang? Is there a backtrace?

There are at least 3 scenarious I know of:

  a) Postcopy: An IO operation takes the lock and accesses guest memory;
     the guest memory is missing due to userfault'd memory.
     Unfortunately the network connection to the source happens to fail;
      so we never receive that page and the thread stays stuck in the userfault.
     We can't issue a recovery command to reopen a network connection
     because the monitor is blocked.
  b) Postcopy: A monitor command either accesses guest memory or has
     to wait on another thread that is doing; e.g. info cpu  waits
     for the CPU threads to exit the loop, but they might be blocked
     waiting on userfault.
  c) COLO or migration: The network fails during the critical bit
     at the end of migration when we have the bql held.  You can't
     issue a migration_cancel or a colo-failover via the monitor
     because it's blocked.

There are other advantages of being able to do bql'less commands;
things like an 'info status' or the like should be doable without bql,
so just avoding taking the bql when the management layer is doing
stuff (or alternatively getting faster replies on management)
are both useful.

Dave

> 
> Fam
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Mon, 08/21 18:28, Dr. David Alan Gilbert wrote:
> > It's not much more than asserting qemu_mutex_iothread_locked(), the problem is
> > the new monitor thread breaks certain assumptions that was true.
> > 
> > What is interesting in this is that block layer's nested aio_poll() now not only
> > run in the main thread but also in the monitor thread. Bugs may hide there.  :)
> > 
> > That's why I suggested a "safe by default" strategy.
> 
> OK, that's going to need some more flags somewhere; we've now
> effectively got three types of command:
>    a) Commands that can only run in the main thread
>    b) Commands that can run in other monitor threads, but must have the bql
>    c) Commands that can run in other monitor threads but don't take the
>    bql
> 
>    The class (a) that you point out are a pain; arguably if we have to
> split them up then perhaps we should initially only allow (c).
> 
> > One step back, is it possible to "unblock" main thread even upon network issue?
> > What is the scenario that causes main thread hang? Is there a backtrace?
> 
> There are at least 3 scenarious I know of:
> 
>   a) Postcopy: An IO operation takes the lock and accesses guest memory;
>      the guest memory is missing due to userfault'd memory.
>      Unfortunately the network connection to the source happens to fail;
>       so we never receive that page and the thread stays stuck in the userfault.
>      We can't issue a recovery command to reopen a network connection
>      because the monitor is blocked.
>   b) Postcopy: A monitor command either accesses guest memory or has
>      to wait on another thread that is doing; e.g. info cpu  waits
>      for the CPU threads to exit the loop, but they might be blocked
>      waiting on userfault.
>   c) COLO or migration: The network fails during the critical bit
>      at the end of migration when we have the bql held.  You can't
>      issue a migration_cancel or a colo-failover via the monitor
>      because it's blocked.

Thanks for explainaing!

What commands are in class (c)? From the cover letter it seems migrate-incoming
is the only one in mind, I'm not sure how it resolves any of the three
scenarios?

> 
> There are other advantages of being able to do bql'less commands;
> things like an 'info status' or the like should be doable without bql,
> so just avoding taking the bql when the management layer is doing
> stuff (or alternatively getting faster replies on management)
> are both useful.

Agreed. It is very useful not just for migration.

Fam

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Peter Xu 6 years, 7 months ago
On Tue, Aug 22, 2017 at 10:15:56AM +0800, Fam Zheng wrote:
> On Mon, 08/21 18:28, Dr. David Alan Gilbert wrote:
> > > It's not much more than asserting qemu_mutex_iothread_locked(), the problem is
> > > the new monitor thread breaks certain assumptions that was true.
> > > 
> > > What is interesting in this is that block layer's nested aio_poll() now not only
> > > run in the main thread but also in the monitor thread. Bugs may hide there.  :)
> > > 
> > > That's why I suggested a "safe by default" strategy.
> > 
> > OK, that's going to need some more flags somewhere; we've now
> > effectively got three types of command:
> >    a) Commands that can only run in the main thread
> >    b) Commands that can run in other monitor threads, but must have the bql
> >    c) Commands that can run in other monitor threads but don't take the
> >    bql
> > 
> >    The class (a) that you point out are a pain; arguably if we have to
> > split them up then perhaps we should initially only allow (c).

Good to know that this series may break something; That's really what
I want to know for this RFC post. :-)

I agree with Dave that if we assume (a) unchangable, then the only way
to go is only allow (c) in threaded monitors.

However before that, I am curious about whether we can replace the
assertion with something else like locks?  And how hard would it be?

(Paolo?)

> > 
> > > One step back, is it possible to "unblock" main thread even upon network issue?
> > > What is the scenario that causes main thread hang? Is there a backtrace?
> > 
> > There are at least 3 scenarious I know of:
> > 
> >   a) Postcopy: An IO operation takes the lock and accesses guest memory;
> >      the guest memory is missing due to userfault'd memory.
> >      Unfortunately the network connection to the source happens to fail;
> >       so we never receive that page and the thread stays stuck in the userfault.
> >      We can't issue a recovery command to reopen a network connection
> >      because the monitor is blocked.
> >   b) Postcopy: A monitor command either accesses guest memory or has
> >      to wait on another thread that is doing; e.g. info cpu  waits
> >      for the CPU threads to exit the loop, but they might be blocked
> >      waiting on userfault.
> >   c) COLO or migration: The network fails during the critical bit
> >      at the end of migration when we have the bql held.  You can't
> >      issue a migration_cancel or a colo-failover via the monitor
> >      because it's blocked.
> 
> Thanks for explainaing!
> 
> What commands are in class (c)? From the cover letter it seems migrate-incoming
> is the only one in mind, I'm not sure how it resolves any of the three
> scenarios?

I haven't really encountered (c), but I think it's the migrate_cancel
command that matters, which should not need BQL as well.  For (a) and
(b), they should both need the other migrate_incoming command.

Thanks,

> 
> > 
> > There are other advantages of being able to do bql'less commands;
> > things like an 'info status' or the like should be doable without bql,
> > so just avoding taking the bql when the management layer is doing
> > stuff (or alternatively getting faster replies on management)
> > are both useful.
> 
> Agreed. It is very useful not just for migration.
> 
> Fam

-- 
Peter Xu

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Tue, 08/22 10:56, Peter Xu wrote:
> I haven't really encountered (c), but I think it's the migrate_cancel
> command that matters, which should not need BQL as well.

There is bdrv_invalidate_cache_all() in migrate_cancel which clearly isn't safe.
Is that if block unreachable in this case? If so we should assert, otherwise
this command is not okay to run without BQL.

Generically, what guarantee the thread-safety of a qmp command when you decide
BQL is not needed? In other words, how do you prove commands are safe without
BQL? I think almost every command accesses global state, but lock-free data
structures are rare AFAICT.

Fam

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Peter Xu 6 years, 7 months ago
On Tue, Aug 22, 2017 at 12:15:19PM +0800, Fam Zheng wrote:
> On Tue, 08/22 10:56, Peter Xu wrote:
> > I haven't really encountered (c), but I think it's the migrate_cancel
> > command that matters, which should not need BQL as well.
> 
> There is bdrv_invalidate_cache_all() in migrate_cancel which clearly isn't safe.
> Is that if block unreachable in this case? If so we should assert, otherwise
> this command is not okay to run without BQL.

Ah. I see.  Even if so, if that is the only usage of BQL, IMHO we can
still mark migrate_cancel as "without-bql=true", instead we take the
BQL before calling bdrv_invalidate_cache_all().  Then migrate_cancel
can be BQL-free at least when block migration is not active.

> 
> Generically, what guarantee the thread-safety of a qmp command when you decide
> BQL is not needed? In other words, how do you prove commands are safe without
> BQL? I think almost every command accesses global state, but lock-free data
> structures are rare AFAICT.

I would suggest we split the problem into at least three parts.  IMHO
we need to answer below questions one by one to know what we should do
next:

1. whether we can handle monitor commands outside iothread, or say, in
   an isolated thread?

   This is basically what patch 2 does, the "per-monitor threads".

   IMHO this is the very first question to ask.  So now I know that at
   least current code cannot do it.  We need to at least do something
   to remove/replace the assertion to make this happen.  Can we?  I
   don't really know the answer yet.  If this is undoable, we can skip
   question 2/3 below and may need to rethink on how to solve the
   problem that postcopy recovery encounters.

2. whether there is any monitor commands can run without BQL?

   This is basically what patch 3/5 does, one for QMP, one for HMP.

   If we can settle question 1, then we can possibly start consider
   this question.  This step does not really allow any command to run
   without BQL, but we need to know whether it's possible in general,
   and if possible, we provide a framework to allow QMP/HMP developers
   to specify that.  If you see patch 3/5, the default behavior is
   still taking the BQL for all commands.

   IMHO doing this whole thing is generally good in the sense that
   this is actually forcing ourselves to break the BQL into smaller
   locks.  Take the migration commands for example: migrate_incoming
   do not need BQL, and when we write codes around it we know that we
   don't need to think about thread-safety.  That's not good IMHO.  I
   think it's time we should start consider thread-safety always.
   Again, for migrate_incoming to do this, actually we'll possibly at
   least need a migration management lock (the smaller lock) to make
   sure e.g. the user is not running two migrate_incoming commands in
   parallel (after per-monitor threads, it can happen).  But it's
   better than BQL, because BQL is for sure too big, so even a guest
   page access (as long as it held the BQL) can block migration
   commands.

3. which monitor commands can be run without BQL?

   This is what patch 4/6 was doing.  It tries to move
   migrate_incoming command out as the first candidate BQL-free
   command.

   Yes it's hard to say which command can be run without BQL.  So we
   need to investigate, possibly modify existing codes to make sure
   it's thread-safe, prove validity, then we can add the new ones into
   the BQL-free list.

   If after evaluating the pros and cons, we found that one command
   can be put into BQL-free but not worth the time for working on it,
   we can also keep those commands under BQL.

I assume question 3 is the one you were asking, and I'd say we may
need to solve question 1/2 first.  If we are done with 1/2, we just
need to spend time on each command to prove whether it is doable to
let that command run without BQL, and whether it worths itself to move
the command out of BQL.  Then we decide.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Tue, 08/22 13:59, Peter Xu wrote:
> On Tue, Aug 22, 2017 at 12:15:19PM +0800, Fam Zheng wrote:
> > On Tue, 08/22 10:56, Peter Xu wrote:
> > > I haven't really encountered (c), but I think it's the migrate_cancel
> > > command that matters, which should not need BQL as well.
> > 
> > There is bdrv_invalidate_cache_all() in migrate_cancel which clearly isn't safe.
> > Is that if block unreachable in this case? If so we should assert, otherwise
> > this command is not okay to run without BQL.
> 
> Ah. I see.  Even if so, if that is the only usage of BQL, IMHO we can
> still mark migrate_cancel as "without-bql=true", instead we take the
> BQL before calling bdrv_invalidate_cache_all().  Then migrate_cancel
> can be BQL-free at least when block migration is not active.
> 
> > 
> > Generically, what guarantee the thread-safety of a qmp command when you decide
> > BQL is not needed? In other words, how do you prove commands are safe without
> > BQL? I think almost every command accesses global state, but lock-free data
> > structures are rare AFAICT.
> 
> I would suggest we split the problem into at least three parts.  IMHO
> we need to answer below questions one by one to know what we should do
> next:
> 
> 1. whether we can handle monitor commands outside iothread, or say, in
>    an isolated thread?
> 
>    This is basically what patch 2 does, the "per-monitor threads".
> 
>    IMHO this is the very first question to ask.  So now I know that at
>    least current code cannot do it.  We need to at least do something
>    to remove/replace the assertion to make this happen.  Can we?  I
>    don't really know the answer yet.  If this is undoable, we can skip
>    question 2/3 below and may need to rethink on how to solve the
>    problem that postcopy recovery encounters.
> 
> 2. whether there is any monitor commands can run without BQL?
> 
>    This is basically what patch 3/5 does, one for QMP, one for HMP.
> 
>    If we can settle question 1, then we can possibly start consider
>    this question.  This step does not really allow any command to run
>    without BQL, but we need to know whether it's possible in general,
>    and if possible, we provide a framework to allow QMP/HMP developers
>    to specify that.  If you see patch 3/5, the default behavior is
>    still taking the BQL for all commands.
> 
>    IMHO doing this whole thing is generally good in the sense that
>    this is actually forcing ourselves to break the BQL into smaller
>    locks.  Take the migration commands for example: migrate_incoming
>    do not need BQL, and when we write codes around it we know that we
>    don't need to think about thread-safety.  That's not good IMHO.  I
>    think it's time we should start consider thread-safety always.
>    Again, for migrate_incoming to do this, actually we'll possibly at
>    least need a migration management lock (the smaller lock) to make
>    sure e.g. the user is not running two migrate_incoming commands in
>    parallel (after per-monitor threads, it can happen).  But it's
>    better than BQL, because BQL is for sure too big, so even a guest
>    page access (as long as it held the BQL) can block migration
>    commands.

Yes, this is my point. You cannot just declare a command "BQL-free" without
adding small locks first, and I think this is actually missing in this series.
As you said, two per-monitor threads can race if they do migrate_incoming in
parallel.  This is also the answer to 3.

Fam

> 
> 3. which monitor commands can be run without BQL?
> 
>    This is what patch 4/6 was doing.  It tries to move
>    migrate_incoming command out as the first candidate BQL-free
>    command.
> 
>    Yes it's hard to say which command can be run without BQL.  So we
>    need to investigate, possibly modify existing codes to make sure
>    it's thread-safe, prove validity, then we can add the new ones into
>    the BQL-free list.
> 
>    If after evaluating the pros and cons, we found that one command
>    can be put into BQL-free but not worth the time for working on it,
>    we can also keep those commands under BQL.
> 
> I assume question 3 is the one you were asking, and I'd say we may
> need to solve question 1/2 first.  If we are done with 1/2, we just
> need to spend time on each command to prove whether it is doable to
> let that command run without BQL, and whether it worths itself to move
> the command out of BQL.  Then we decide.  Thanks,
> 
> -- 
> Peter Xu

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Peter Xu 6 years, 7 months ago
On Tue, Aug 22, 2017 at 02:33:48PM +0800, Fam Zheng wrote:
> On Tue, 08/22 13:59, Peter Xu wrote:
> > On Tue, Aug 22, 2017 at 12:15:19PM +0800, Fam Zheng wrote:
> > > On Tue, 08/22 10:56, Peter Xu wrote:
> > > > I haven't really encountered (c), but I think it's the migrate_cancel
> > > > command that matters, which should not need BQL as well.
> > > 
> > > There is bdrv_invalidate_cache_all() in migrate_cancel which clearly isn't safe.
> > > Is that if block unreachable in this case? If so we should assert, otherwise
> > > this command is not okay to run without BQL.
> > 
> > Ah. I see.  Even if so, if that is the only usage of BQL, IMHO we can
> > still mark migrate_cancel as "without-bql=true", instead we take the
> > BQL before calling bdrv_invalidate_cache_all().  Then migrate_cancel
> > can be BQL-free at least when block migration is not active.
> > 
> > > 
> > > Generically, what guarantee the thread-safety of a qmp command when you decide
> > > BQL is not needed? In other words, how do you prove commands are safe without
> > > BQL? I think almost every command accesses global state, but lock-free data
> > > structures are rare AFAICT.
> > 
> > I would suggest we split the problem into at least three parts.  IMHO
> > we need to answer below questions one by one to know what we should do
> > next:
> > 
> > 1. whether we can handle monitor commands outside iothread, or say, in
> >    an isolated thread?
> > 
> >    This is basically what patch 2 does, the "per-monitor threads".
> > 
> >    IMHO this is the very first question to ask.  So now I know that at
> >    least current code cannot do it.  We need to at least do something
> >    to remove/replace the assertion to make this happen.  Can we?  I
> >    don't really know the answer yet.  If this is undoable, we can skip
> >    question 2/3 below and may need to rethink on how to solve the
> >    problem that postcopy recovery encounters.
> > 
> > 2. whether there is any monitor commands can run without BQL?
> > 
> >    This is basically what patch 3/5 does, one for QMP, one for HMP.
> > 
> >    If we can settle question 1, then we can possibly start consider
> >    this question.  This step does not really allow any command to run
> >    without BQL, but we need to know whether it's possible in general,
> >    and if possible, we provide a framework to allow QMP/HMP developers
> >    to specify that.  If you see patch 3/5, the default behavior is
> >    still taking the BQL for all commands.
> > 
> >    IMHO doing this whole thing is generally good in the sense that
> >    this is actually forcing ourselves to break the BQL into smaller
> >    locks.  Take the migration commands for example: migrate_incoming
> >    do not need BQL, and when we write codes around it we know that we
> >    don't need to think about thread-safety.  That's not good IMHO.  I
> >    think it's time we should start consider thread-safety always.
> >    Again, for migrate_incoming to do this, actually we'll possibly at
> >    least need a migration management lock (the smaller lock) to make
> >    sure e.g. the user is not running two migrate_incoming commands in
> >    parallel (after per-monitor threads, it can happen).  But it's
> >    better than BQL, because BQL is for sure too big, so even a guest
> >    page access (as long as it held the BQL) can block migration
> >    commands.
> 
> Yes, this is my point. You cannot just declare a command "BQL-free" without
> adding small locks first, and I think this is actually missing in this series.
> As you said, two per-monitor threads can race if they do migrate_incoming in
> parallel.  This is also the answer to 3.

Ah, I see.  The small lock will be there if there is another post. :)

-- 
Peter Xu

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Dr. David Alan Gilbert 6 years, 7 months ago
* Fam Zheng (famz@redhat.com) wrote:
> On Tue, 08/22 13:59, Peter Xu wrote:
> > On Tue, Aug 22, 2017 at 12:15:19PM +0800, Fam Zheng wrote:
> > > On Tue, 08/22 10:56, Peter Xu wrote:
> > > > I haven't really encountered (c), but I think it's the migrate_cancel
> > > > command that matters, which should not need BQL as well.
> > > 
> > > There is bdrv_invalidate_cache_all() in migrate_cancel which clearly isn't safe.
> > > Is that if block unreachable in this case? If so we should assert, otherwise
> > > this command is not okay to run without BQL.
> > 
> > Ah. I see.  Even if so, if that is the only usage of BQL, IMHO we can
> > still mark migrate_cancel as "without-bql=true", instead we take the
> > BQL before calling bdrv_invalidate_cache_all().  Then migrate_cancel
> > can be BQL-free at least when block migration is not active.
> > 
> > > 
> > > Generically, what guarantee the thread-safety of a qmp command when you decide
> > > BQL is not needed? In other words, how do you prove commands are safe without
> > > BQL? I think almost every command accesses global state, but lock-free data
> > > structures are rare AFAICT.
> > 
> > I would suggest we split the problem into at least three parts.  IMHO
> > we need to answer below questions one by one to know what we should do
> > next:
> > 
> > 1. whether we can handle monitor commands outside iothread, or say, in
> >    an isolated thread?
> > 
> >    This is basically what patch 2 does, the "per-monitor threads".
> > 
> >    IMHO this is the very first question to ask.  So now I know that at
> >    least current code cannot do it.  We need to at least do something
> >    to remove/replace the assertion to make this happen.  Can we?  I
> >    don't really know the answer yet.  If this is undoable, we can skip
> >    question 2/3 below and may need to rethink on how to solve the
> >    problem that postcopy recovery encounters.
> > 
> > 2. whether there is any monitor commands can run without BQL?
> > 
> >    This is basically what patch 3/5 does, one for QMP, one for HMP.
> > 
> >    If we can settle question 1, then we can possibly start consider
> >    this question.  This step does not really allow any command to run
> >    without BQL, but we need to know whether it's possible in general,
> >    and if possible, we provide a framework to allow QMP/HMP developers
> >    to specify that.  If you see patch 3/5, the default behavior is
> >    still taking the BQL for all commands.
> > 
> >    IMHO doing this whole thing is generally good in the sense that
> >    this is actually forcing ourselves to break the BQL into smaller
> >    locks.  Take the migration commands for example: migrate_incoming
> >    do not need BQL, and when we write codes around it we know that we
> >    don't need to think about thread-safety.  That's not good IMHO.  I
> >    think it's time we should start consider thread-safety always.
> >    Again, for migrate_incoming to do this, actually we'll possibly at
> >    least need a migration management lock (the smaller lock) to make
> >    sure e.g. the user is not running two migrate_incoming commands in
> >    parallel (after per-monitor threads, it can happen).  But it's
> >    better than BQL, because BQL is for sure too big, so even a guest
> >    page access (as long as it held the BQL) can block migration
> >    commands.
> 
> Yes, this is my point. You cannot just declare a command "BQL-free" without
> adding small locks first, and I think this is actually missing in this series.
> As you said, two per-monitor threads can race if they do migrate_incoming in
> parallel.  This is also the answer to 3.

Some commands really could be lock-free.  For example an 'info status'
should just be a read without any locking.
It might also be possible to make a version of migrate_cancel lock-free;
in particular the critical part is being able to do a shutdown() on the
socket - that's enough to cause any of the migration threads to 
unblock and take their failure path.

Also in the case of COLO; there should be an equivalent place to
do the shutdown() on the migration thread and then set a flag to cause
failover.
(It's currently qmp_x_colo_lost_heartbeat but I think the eventual
version would be more complex).

Dave

> Fam
> 
> > 
> > 3. which monitor commands can be run without BQL?
> > 
> >    This is what patch 4/6 was doing.  It tries to move
> >    migrate_incoming command out as the first candidate BQL-free
> >    command.
> > 
> >    Yes it's hard to say which command can be run without BQL.  So we
> >    need to investigate, possibly modify existing codes to make sure
> >    it's thread-safe, prove validity, then we can add the new ones into
> >    the BQL-free list.
> > 
> >    If after evaluating the pros and cons, we found that one command
> >    can be put into BQL-free but not worth the time for working on it,
> >    we can also keep those commands under BQL.
> > 
> > I assume question 3 is the one you were asking, and I'd say we may
> > need to solve question 1/2 first.  If we are done with 1/2, we just
> > need to spend time on each command to prove whether it is doable to
> > let that command run without BQL, and whether it worths itself to move
> > the command out of BQL.  Then we decide.  Thanks,
> > 
> > -- 
> > Peter Xu
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Fam Zheng 6 years, 7 months ago
On Tue, 08/22 09:29, Dr. David Alan Gilbert wrote:
> * Fam Zheng (famz@redhat.com) wrote:
> > On Tue, 08/22 13:59, Peter Xu wrote:
> > > On Tue, Aug 22, 2017 at 12:15:19PM +0800, Fam Zheng wrote:
> > > > On Tue, 08/22 10:56, Peter Xu wrote:
> > > > > I haven't really encountered (c), but I think it's the migrate_cancel
> > > > > command that matters, which should not need BQL as well.
> > > > 
> > > > There is bdrv_invalidate_cache_all() in migrate_cancel which clearly isn't safe.
> > > > Is that if block unreachable in this case? If so we should assert, otherwise
> > > > this command is not okay to run without BQL.
> > > 
> > > Ah. I see.  Even if so, if that is the only usage of BQL, IMHO we can
> > > still mark migrate_cancel as "without-bql=true", instead we take the
> > > BQL before calling bdrv_invalidate_cache_all().  Then migrate_cancel
> > > can be BQL-free at least when block migration is not active.
> > > 
> > > > 
> > > > Generically, what guarantee the thread-safety of a qmp command when you decide
> > > > BQL is not needed? In other words, how do you prove commands are safe without
> > > > BQL? I think almost every command accesses global state, but lock-free data
> > > > structures are rare AFAICT.
> > > 
> > > I would suggest we split the problem into at least three parts.  IMHO
> > > we need to answer below questions one by one to know what we should do
> > > next:
> > > 
> > > 1. whether we can handle monitor commands outside iothread, or say, in
> > >    an isolated thread?
> > > 
> > >    This is basically what patch 2 does, the "per-monitor threads".
> > > 
> > >    IMHO this is the very first question to ask.  So now I know that at
> > >    least current code cannot do it.  We need to at least do something
> > >    to remove/replace the assertion to make this happen.  Can we?  I
> > >    don't really know the answer yet.  If this is undoable, we can skip
> > >    question 2/3 below and may need to rethink on how to solve the
> > >    problem that postcopy recovery encounters.
> > > 
> > > 2. whether there is any monitor commands can run without BQL?
> > > 
> > >    This is basically what patch 3/5 does, one for QMP, one for HMP.
> > > 
> > >    If we can settle question 1, then we can possibly start consider
> > >    this question.  This step does not really allow any command to run
> > >    without BQL, but we need to know whether it's possible in general,
> > >    and if possible, we provide a framework to allow QMP/HMP developers
> > >    to specify that.  If you see patch 3/5, the default behavior is
> > >    still taking the BQL for all commands.
> > > 
> > >    IMHO doing this whole thing is generally good in the sense that
> > >    this is actually forcing ourselves to break the BQL into smaller
> > >    locks.  Take the migration commands for example: migrate_incoming
> > >    do not need BQL, and when we write codes around it we know that we
> > >    don't need to think about thread-safety.  That's not good IMHO.  I
> > >    think it's time we should start consider thread-safety always.
> > >    Again, for migrate_incoming to do this, actually we'll possibly at
> > >    least need a migration management lock (the smaller lock) to make
> > >    sure e.g. the user is not running two migrate_incoming commands in
> > >    parallel (after per-monitor threads, it can happen).  But it's
> > >    better than BQL, because BQL is for sure too big, so even a guest
> > >    page access (as long as it held the BQL) can block migration
> > >    commands.
> > 
> > Yes, this is my point. You cannot just declare a command "BQL-free" without
> > adding small locks first, and I think this is actually missing in this series.
> > As you said, two per-monitor threads can race if they do migrate_incoming in
> > parallel.  This is also the answer to 3.
> 
> Some commands really could be lock-free.  For example an 'info status'
> should just be a read without any locking.

Indeed, it makes sense to enable as many info (query-*) commands as possible on
non-mainloop monitors.

> It might also be possible to make a version of migrate_cancel lock-free;
> in particular the critical part is being able to do a shutdown() on the
> socket - that's enough to cause any of the migration threads to 
> unblock and take their failure path.
> 
> Also in the case of COLO; there should be an equivalent place to
> do the shutdown() on the migration thread and then set a flag to cause
> failover.
> (It's currently qmp_x_colo_lost_heartbeat but I think the eventual
> version would be more complex).

Fam

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Dr. David Alan Gilbert 6 years, 7 months ago
* Fam Zheng (famz@redhat.com) wrote:
> On Tue, 08/22 10:56, Peter Xu wrote:
> > I haven't really encountered (c), but I think it's the migrate_cancel
> > command that matters, which should not need BQL as well.
> 
> There is bdrv_invalidate_cache_all() in migrate_cancel which clearly isn't safe.
> Is that if block unreachable in this case? If so we should assert, otherwise
> this command is not okay to run without BQL.

Hmm yes that's a fairly recent addition to migrate_cancel - it used to
be a simplish shutdown();  still I think it must be possible to get
back to a simple shutdown() which is run directly with the cache stuff
perhaps coming along later;  actually the tricky bit is that the fd for
the shutdown() isn't just a simple int, but is in an object which
we have to make sure doesn't get freed while we do it.

> Generically, what guarantee the thread-safety of a qmp command when you decide
> BQL is not needed? In other words, how do you prove commands are safe without
> BQL? I think almost every command accesses global state, but lock-free data
> structures are rare AFAICT.

I'm assuming there would initially be very few commands marked as
lock-free, that were mostly trivially small.

I was thinking that it might be possible to do a debug build with
some checking as well; something like making the code that takes
the bql check a per-thread flag, and assert if the flag is set.
The monitor-threads could set the flag, so that if we fell
down a path that tried to take the BQL we'd fail.

Dave

> Fam
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by no-reply@patchew.org 6 years, 7 months ago
Hi,

This series failed automatic build test. Please find the testing commands and
their output below. If you have docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 1503301464-27886-1-git-send-email-peterx@redhat.com
Subject: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-quick@centos6
time make docker-test-build@min-glib
time make docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/20170822042416.26758-1-david@gibson.dropbear.id.au -> patchew/20170822042416.26758-1-david@gibson.dropbear.id.au
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Switched to a new branch 'test'
6c48e66427 migration: hmp: migrate_incoming don't need BQL
1ffc968231 hmp: support "without_bql"
42b3b3a711 migration: qmp: migrate_incoming don't need BQL
afbb938df1 QAPI: new QMP command option "without-bql"
047188cbc7 monitor: allow monitor to create thread to poll
a89bb1619b monitor: move skip_flush into monitor_data_init

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-vbkqrhlc/src/dtc'...
Submodule path 'dtc': checked out '558cd81bdd432769b59bff01240c44f82cfb1a9d'
  BUILD   centos6
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-vbkqrhlc/src'
  ARCHIVE qemu.tgz
  ARCHIVE dtc.tgz
  COPY    RUNNER
    RUN test-quick in qemu:centos6 
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
bison-2.4.1-5.el6.x86_64
bzip2-devel-1.0.5-7.el6_0.x86_64
ccache-3.1.6-2.el6.x86_64
csnappy-devel-0-6.20150729gitd7bc683.el6.x86_64
flex-2.5.35-9.el6.x86_64
gcc-4.4.7-18.el6.x86_64
git-1.7.1-8.el6.x86_64
glib2-devel-2.28.8-9.el6.x86_64
libepoxy-devel-1.2-3.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
librdmacm-devel-1.0.21-0.el6.x86_64
lzo-devel-2.03-3.1.el6_5.1.x86_64
make-3.81-23.el6.x86_64
mesa-libEGL-devel-11.0.7-4.el6.x86_64
mesa-libgbm-devel-11.0.7-4.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
spice-glib-devel-0.26-8.el6.x86_64
spice-server-devel-0.12.4-16.el6.x86_64
tar-1.23-15.el6_8.x86_64
vte-devel-0.25.1-9.el6.x86_64
xen-devel-4.6.3-15.el6.x86_64
zlib-devel-1.2.3-29.el6.x86_64

Environment variables:
PACKAGES=bison     bzip2-devel     ccache     csnappy-devel     flex     g++     gcc     git     glib2-devel     libepoxy-devel     libfdt-devel     librdmacm-devel     lzo-devel     make     mesa-libEGL-devel     mesa-libgbm-devel     pixman-devel     SDL-devel     spice-glib-devel     spice-server-devel     tar     vte-devel     xen-devel     zlib-devel
HOSTNAME=b29af1b43246
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
LANG=en_US.UTF-8
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES= dtc
DEBUG=
G_BROKEN_FILENAMES=1
CCACHE_HASHDIR=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/var/tmp/qemu-build/install
No C++ compiler available; disabling C++ specific optional code
Install prefix    /var/tmp/qemu-build/install
BIOS directory    /var/tmp/qemu-build/install/share/qemu
binary directory  /var/tmp/qemu-build/install/bin
library directory /var/tmp/qemu-build/install/lib
module directory  /var/tmp/qemu-build/install/lib/qemu
libexec directory /var/tmp/qemu-build/install/libexec
include directory /var/tmp/qemu-build/install/include
config directory  /var/tmp/qemu-build/install/etc
local state directory   /var/tmp/qemu-build/install/var
Manual directory  /var/tmp/qemu-build/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /tmp/qemu-test/src
C compiler        cc
Host C compiler   cc
C++ compiler      
Objective-C compiler cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1   -I$(SRC_PATH)/dtc/libfdt -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -DNCURSES_WIDECHAR   -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all  -I/usr/include/libpng12   -I/usr/include/libdrm     -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/nss3 -I/usr/include/nspr4 -I/usr/include/spice-1   -I/usr/include/cacard -I/usr/include/nss3 -I/usr/include/nspr4  
LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
pixman            system
SDL support       yes (1.2.14)
GTK support       yes (2.24.23)
GTK GL support    no
VTE support       yes (0.25.1)
TLS priority      NORMAL
GNUTLS support    no
GNUTLS rnd        no
libgcrypt         no
libgcrypt kdf     no
nettle            no 
nettle kdf        no
libtasn1          no
curses support    yes
virgl support     no
curl support      no
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
VNC support       yes
VNC SASL support  no
VNC JPEG support  yes
VNC PNG support   yes
xen support       yes
xen ctrl version  40600
pv dom build      no
brlapi support    no
bluez  support    no
Documentation     no
PIE               yes
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
HAX support       no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
RDMA support      yes
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends    log
spice support     yes (0.12.6/0.12.4)
rbd support       no
xfsctl support    no
smartcard support yes
libusb            no
usb net redir     no
OpenGL support    yes
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   no
TPM passthrough   yes
QOM debugging     yes
Live block migration yes
lzo support       yes
snappy support    no
bzip2 support     yes
NUMA host support no
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
VxHS block device no
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak.tmp
mkdir -p dtc/libfdt
mkdir -p dtc/tests
  GEN     config-host.h
  GEN     qemu-options.def
  GEN     qapi-types.h
  GEN     qmp-commands.h
  GEN     qapi-event.h
  GEN     qapi-visit.h
  GEN     x86_64-softmmu/config-devices.mak
  GEN     qmp-marshal.c
  GEN     aarch64-softmmu/config-devices.mak
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     qapi-event.c
  GEN     qmp-introspect.h
  GEN     qmp-introspect.c
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.h
  GEN     trace/generated-helpers.c
  GEN     module_block.h
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qapi-event.h
  GEN     tests/test-qmp-introspect.h
  GEN     trace-root.h
  GEN     util/trace.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/xen/trace.h
  GEN     ui/trace.h
  GEN     audio/trace.h
  GEN     net/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/sparc/trace.h
  GEN     target/s390x/trace.h
  GEN     target/ppc/trace.h
  GEN     qom/trace.h
  GEN     linux-user/trace.h
  GEN     qapi/trace.h
  GEN     accel/tcg/trace.h
  GEN     accel/kvm/trace.h
  GEN     nbd/trace.h
  GEN     trace-root.c
  GEN     util/trace.c
  GEN     crypto/trace.c
  GEN     io/trace.c
  GEN     migration/trace.c
  GEN     block/trace.c
  GEN     chardev/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/net/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/display/trace.c
  GEN     hw/input/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/alpha/trace.c
  GEN     hw/xen/trace.c
  GEN     ui/trace.c
  GEN     audio/trace.c
  GEN     net/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/sparc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/ppc/trace.c
  GEN     qom/trace.c
  GEN     linux-user/trace.c
  GEN     qapi/trace.c
  GEN     accel/tcg/trace.c
  GEN     accel/kvm/trace.c
  GEN     config-all-devices.mak
  GEN     nbd/trace.c
	 DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
	 DEP /tmp/qemu-test/src/dtc/tests/trees.S
	 DEP /tmp/qemu-test/src/dtc/tests/testutils.c
	 DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
	 DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
	 DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/check_path.c
	 DEP /tmp/qemu-test/src/dtc/tests/overlay.c
	 DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
	 DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
	 DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
	 DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
	 DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
	 DEP /tmp/qemu-test/src/dtc/tests/incbin.c
	 DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
	 DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
	 DEP /tmp/qemu-test/src/dtc/tests/path-references.c
	 DEP /tmp/qemu-test/src/dtc/tests/references.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
	 DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/set_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
	 DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
	 DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
	 DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
	 DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
	 DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
	 DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
	 DEP /tmp/qemu-test/src/dtc/tests/notfound.c
	 DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
	 DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
	 DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_path.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/getprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/find_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/root_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
	 DEP /tmp/qemu-test/src/dtc/util.c
	 DEP /tmp/qemu-test/src/dtc/fdtput.c
	 DEP /tmp/qemu-test/src/dtc/fdtget.c
	 DEP /tmp/qemu-test/src/dtc/fdtdump.c
	 LEX convert-dtsv0-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/srcpos.c
	 BISON dtc-parser.tab.c
	 LEX dtc-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/treesource.c
	 DEP /tmp/qemu-test/src/dtc/livetree.c
	 DEP /tmp/qemu-test/src/dtc/fstree.c
	 DEP /tmp/qemu-test/src/dtc/flattree.c
	 DEP /tmp/qemu-test/src/dtc/dtc.c
	 DEP /tmp/qemu-test/src/dtc/data.c
	 DEP /tmp/qemu-test/src/dtc/checks.c
	 DEP convert-dtsv0-lexer.lex.c
	 DEP dtc-lexer.lex.c
	 DEP dtc-parser.tab.c
	CHK version_gen.h
	UPD version_gen.h
	 DEP /tmp/qemu-test/src/dtc/util.c
	 CC libfdt/fdt.o
	 CC libfdt/fdt_ro.o
	 CC libfdt/fdt_wip.o
	 CC libfdt/fdt_rw.o
	 CC libfdt/fdt_sw.o
	 CC libfdt/fdt_empty_tree.o
	 CC libfdt/fdt_strerror.o
	 CC libfdt/fdt_addresses.o
	 CC libfdt/fdt_overlay.o
	 AR libfdt/libfdt.a
ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qmp-commands.h
  CC      tests/qemu-iotests/socket_scm_helper.o
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  GEN     qga/qapi-generated/qga-qapi-visit.c
  CC      qmp-introspect.o
  GEN     qga/qapi-generated/qga-qapi-types.c
  CC      qapi-types.o
  CC      qapi-visit.o
  CC      qapi-event.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qstring.o
  CC      qobject/qnum.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qbool.o
  CC      qobject/qjson.o
  CC      qobject/qobject.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/qemu-timer-common.o
  CC      util/unicode.o
  CC      util/bufferiszero.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/async.o
  CC      util/thread-pool.o
  CC      util/qemu-timer.o
  CC      util/iohandler.o
  CC      util/main-loop.o
  CC      util/aio-posix.o
  CC      util/compatfd.o
  CC      util/event_notifier-posix.o
  CC      util/mmap-alloc.o
  CC      util/oslib-posix.o
  CC      util/qemu-openpty.o
  CC      util/qemu-thread-posix.o
  CC      util/memfd.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/host-utils.o
  CC      util/module.o
  CC      util/bitmap.o
  CC      util/bitops.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/cacheinfo.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/uri.o
  CC      util/qemu-sockets.o
  CC      util/qemu-config.o
  CC      util/notify.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/keyval.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/qemu-coroutine.o
  CC      util/rcu.o
  CC      util/readline.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/coroutine-ucontext.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/log.o
  CC      util/qdist.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      trace-root.o
  CC      util/trace.o
  CC      crypto/trace.o
  CC      io/trace.o
  CC      migration/trace.o
  CC      block/trace.o
  CC      chardev/trace.o
  CC      hw/block/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/intc/trace.o
  CC      hw/net/trace.o
  CC      hw/virtio/trace.o
  CC      hw/audio/trace.o
  CC      hw/misc/trace.o
  CC      hw/usb/trace.o
  CC      hw/nvram/trace.o
  CC      hw/scsi/trace.o
  CC      hw/display/trace.o
  CC      hw/input/trace.o
  CC      hw/timer/trace.o
  CC      hw/dma/trace.o
  CC      hw/sd/trace.o
  CC      hw/sparc/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/i386/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/pci/trace.o
  CC      hw/ppc/trace.o
  CC      hw/s390x/trace.o
  CC      hw/vfio/trace.o
  CC      hw/acpi/trace.o
  CC      hw/arm/trace.o
  CC      hw/alpha/trace.o
  CC      hw/xen/trace.o
  CC      ui/trace.o
  CC      audio/trace.o
  CC      net/trace.o
  CC      target/arm/trace.o
  CC      target/i386/trace.o
  CC      target/mips/trace.o
  CC      target/sparc/trace.o
  CC      target/s390x/trace.o
  CC      qom/trace.o
  CC      target/ppc/trace.o
  CC      linux-user/trace.o
  CC      qapi/trace.o
  CC      accel/tcg/trace.o
  CC      accel/kvm/trace.o
  CC      crypto/pbkdf-stub.o
  CC      nbd/trace.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/blk-commit-all.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/clock-warp.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/iothread.o
  CC      stubs/gdbstub.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/runstate-check.o
  CC      stubs/replay.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/qmp_pc_dimm_device_list.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      contrib/ivshmem-client/ivshmem-client.o
  CC      contrib/ivshmem-client/main.o
  CC      contrib/ivshmem-server/main.o
  CC      contrib/ivshmem-server/ivshmem-server.o
  CC      qemu-nbd.o
  CC      block.o
  CC      blockjob.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/vdi.o
  CC      block/qcow.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/parallels.o
  CC      block/quorum.o
  CC      block/blkdebug.o
  CC      block/blkverify.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-posix.o
  CC      block/null.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/accounting.o
/tmp/qemu-test/src/block/nbd-client.c: In function ‘nbd_read_reply_entry’:
/tmp/qemu-test/src/block/nbd-client.c:110: warning: ‘ret’ may be used uninitialized in this function
/tmp/qemu-test/src/block/nbd-client.c:73: note: ‘ret’ was declared here
  CC      block/write-threshold.o
  CC      block/dirty-bitmap.o
  CC      block/backup.o
  CC      block/crypto.o
  CC      block/replication.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-glib.o
  CC      crypto/hmac.o
  CC      crypto/hmac-glib.o
  CC      crypto/aes.o
  CC      crypto/desrfb.o
  CC      crypto/tlscreds.o
  CC      crypto/cipher.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-platform.o
  CC      crypto/pbkdf.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      crypto/block-luks.o
  CC      io/channel.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-watch.o
  CC      io/channel-websock.o
  CC      io/channel-util.o
  CC      io/dns-resolver.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  GEN     qemu-img-cmds.h
  CC      qemu-io.o
  CC      qemu-bridge-helper.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-posix.o
  CC      bt-host.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  CC      qmp-marshal.o
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/sdlaudio.o
  CC      audio/ossaudio.o
  CC      audio/spiceaudio.o
  CC      audio/wavcapture.o
  CC      backends/rng.o
  CC      backends/rng-egd.o
  CC      backends/rng-random.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/cryptodev.o
  CC      backends/hostmem-file.o
  CC      backends/cryptodev-builtin.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      chardev/spice.o
  CC      disas/arm.o
  CC      disas/i386.o
  CC      fsdev/qemu-fsdev-dummy.o
  CC      fsdev/qemu-fsdev-opts.o
  CC      fsdev/qemu-fsdev-throttle.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/xen_disk.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/core.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/xen_console.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/debugcon.o
  CC      hw/char/imx_serial.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/ptimer.o
  CC      hw/core/machine.o
  CC      hw/core/sysbus.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/loader.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/cpu/core.o
  CC      hw/display/ads7846.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/xenfb.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/blizzard.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/tc6393xb.o
  CC      hw/display/qxl.o
  CC      hw/display/qxl-render.o
  CC      hw/display/qxl-logger.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/gpio_key.o
  CC      hw/gpio/zaurus.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/ahci.o
  CC      hw/ide/piix.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ich.o
  CC      hw/input/hid.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/vmmouse.o
  CC      hw/input/virtio-input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/input/virtio-input-host.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/pl190.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/unimp.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/net/xen_nic.o
  CC      hw/net/ne2000.o
  CC      hw/misc/arm11scu.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/xgmac.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/imx_fec.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci-host/q35.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/shpc.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/pl181.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254.o
  CC      hw/timer/pl031.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/tpm/tpm_passthrough.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/tpm/tpm_util.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/ccid-card-passthru.o
  CC      hw/usb/ccid-card-emulated.o
  CC      hw/usb/dev-mtp.o
  CC      hw/usb/host-stub.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      hw/xen/xen_backend.o
  CC      hw/xen/xen_devconfig.o
  CC      hw/xen/xen_pvdev.o
  CC      hw/xen/xen-common.o
  CC      migration/migration.o
  CC      migration/socket.o
  CC      migration/fd.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/savevm.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate.o
  CC      migration/vmstate-types.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/global_state.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/rdma.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/l2tpv3.o
  CC      net/vhost-user.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/filter-mirror.o
  CC      net/colo-compare.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap.o
  CC      net/tap-linux.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
/tmp/qemu-test/src/replay/replay-internal.c: In function ‘replay_put_array’:
/tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
  CC      replay/replay-char.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/if.o
  CC      slirp/ip_icmp.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_input.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/slirp.o
  CC      slirp/dhcpv6.o
  CC      slirp/mbuf.o
  CC      slirp/misc.o
  CC      slirp/sbuf.o
  CC      slirp/socket.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
/tmp/qemu-test/src/slirp/tcp_input.c: In function ‘tcp_input’:
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_p’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_len’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_tos’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_id’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_off’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_ttl’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_sum’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_src.s_addr’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_dst.s_addr’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:220: warning: ‘save_ip6.ip_nh’ may be used uninitialized in this function
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/input-linux.o
  CC      ui/spice-core.o
  CC      ui/spice-input.o
  CC      ui/spice-display.o
  CC      ui/sdl.o
  CC      ui/sdl_zoom.o
  CC      ui/x_keymap.o
  CC      ui/curses.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  CC      ui/gtk.o
  CC      ui/shader.o
  VERT    ui/shader/texture-blit-vert.h
  FRAG    ui/shader/texture-blit-frag.h
  CC      ui/egl-helpers.o
  CC      ui/egl-context.o
  CC      ui/gtk-egl.o
  CC      chardev/char.o
In file included from /usr/include/gtk-2.0/gtk/gtk.h:235,
                 from /tmp/qemu-test/src/include/ui/gtk.h:10,
                 from /tmp/qemu-test/src/ui/gtk-egl.c:21:
/usr/include/gtk-2.0/gtk/gtkitemfactory.h:47: warning: function declaration isn’t a prototype
  CC      chardev/char-fd.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-parallel.o
  CC      chardev/char-pipe.o
  CC      chardev/char-pty.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  LINK    tests/qemu-iotests/socket_scm_helper
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  AS      optionrom/multiboot.o
  AS      optionrom/linuxboot.o
  CC      optionrom/linuxboot_dma.o
cc: unrecognized option '-no-integrated-as'
cc: unrecognized option '-no-integrated-as'
  AS      optionrom/kvmvapic.o
  BUILD   optionrom/multiboot.img
  BUILD   optionrom/linuxboot.img
  BUILD   optionrom/linuxboot_dma.img
  BUILD   optionrom/kvmvapic.img
  BUILD   optionrom/multiboot.raw
  BUILD   optionrom/linuxboot.raw
  BUILD   optionrom/linuxboot_dma.raw
  CC      qga/main.o
  BUILD   optionrom/kvmvapic.raw
  SIGN    optionrom/multiboot.bin
  SIGN    optionrom/linuxboot.bin
  CC      qga/commands-posix.o
  SIGN    optionrom/linuxboot_dma.bin
  SIGN    optionrom/kvmvapic.bin
  CC      qga/channel-posix.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  CC      qga/qapi-generated/qga-qmp-marshal.o
  AR      libqemuutil.a
  AR      libqemustub.a
  CC      qemu-img.o
  CC      ui/console-gl.o
  LINK    ivshmem-client
  LINK    ivshmem-server
  LINK    qemu-nbd
  LINK    qemu-img
  LINK    qemu-io
  LINK    qemu-bridge-helper
In file included from /usr/include/gtk-2.0/gtk/gtk.h:235,
                 from /tmp/qemu-test/src/include/ui/gtk.h:10,
                 from /tmp/qemu-test/src/ui/gtk.c:43:
/usr/include/gtk-2.0/gtk/gtkitemfactory.h:47: warning: function declaration isn’t a prototype
  LINK    qemu-ga
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/config-target.h
  CC      x86_64-softmmu/tcg/tcg.o
  CC      x86_64-softmmu/exec.o
  CC      x86_64-softmmu/tcg/tcg-runtime.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/disas.o
  GEN     x86_64-softmmu/gdbstub-xml.c
  CC      x86_64-softmmu/hax-stub.o
  CC      x86_64-softmmu/arch_init.o
  CC      x86_64-softmmu/monitor.o
  CC      x86_64-softmmu/balloon.o
  CC      x86_64-softmmu/cpus.o
  CC      x86_64-softmmu/gdbstub.o
  CC      x86_64-softmmu/ioport.o
  CC      aarch64-softmmu/exec.o
  CC      x86_64-softmmu/numa.o
  CC      x86_64-softmmu/qtest.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/tcg/tcg-runtime.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/disas.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/hax-stub.o
  CC      aarch64-softmmu/arch_init.o
  CC      aarch64-softmmu/cpus.o
  CC      aarch64-softmmu/monitor.o
  CC      aarch64-softmmu/gdbstub.o
  CC      aarch64-softmmu/balloon.o
  CC      aarch64-softmmu/ioport.o
  CC      aarch64-softmmu/numa.o
  CC      aarch64-softmmu/qtest.o
  CC      aarch64-softmmu/memory.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      aarch64-softmmu/dump.o
  CC      aarch64-softmmu/migration/ram.o
  CC      aarch64-softmmu/accel/accel.o
  CC      aarch64-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/accel/tcg/translate-all.o
  CC      x86_64-softmmu/memory.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      x86_64-softmmu/dump.o
  CC      x86_64-softmmu/migration/ram.o
  CC      x86_64-softmmu/accel/accel.o
  CC      x86_64-softmmu/accel/kvm/kvm-all.o
  CC      x86_64-softmmu/accel/tcg/tcg-all.o
  CC      x86_64-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      x86_64-softmmu/accel/tcg/translate-all.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CC      x86_64-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      aarch64-softmmu/hw/misc/ivshmem.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      x86_64-softmmu/hw/misc/vmport.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      x86_64-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
  CC      x86_64-softmmu/hw/misc/hyperv_testdev.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      x86_64-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      aarch64-softmmu/hw/misc/mps2-scc.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      aarch64-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      aarch64-softmmu/hw/sd/bcm2835_sdhost.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      x86_64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      x86_64-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/hw/timer/mc146818rtc.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      x86_64-softmmu/hw/vfio/pci.o
  CC      x86_64-softmmu/hw/vfio/pci-quirks.o
  CC      x86_64-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/virtio/virtio.o
  CC      x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      x86_64-softmmu/hw/virtio/vhost.o
  CC      x86_64-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      x86_64-softmmu/hw/virtio/vhost-user.o
  CC      x86_64-softmmu/hw/virtio/vhost-vsock.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      x86_64-softmmu/hw/xen/xen-host-pci-device.o
  CC      x86_64-softmmu/hw/xen/xen_pt.o
  CC      aarch64-softmmu/hw/vfio/pci.o
  CC      x86_64-softmmu/hw/xen/xen_pt_config_init.o
  CC      aarch64-softmmu/hw/vfio/pci-quirks.o
  CC      x86_64-softmmu/hw/xen/xen_pt_graphics.o
  CC      aarch64-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/hw/xen/xen_pt_msi.o
  CC      x86_64-softmmu/hw/i386/multiboot.o
  CC      x86_64-softmmu/hw/i386/pc.o
  CC      x86_64-softmmu/hw/i386/pc_piix.o
  CC      x86_64-softmmu/hw/i386/pc_q35.o
  CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      x86_64-softmmu/hw/i386/pc_sysfw.o
  CC      x86_64-softmmu/hw/i386/x86-iommu.o
  CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC      aarch64-softmmu/hw/vfio/spapr.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      x86_64-softmmu/hw/i386/intel_iommu.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/virtio/vhost.o
  CC      aarch64-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC      x86_64-softmmu/hw/i386/amd_iommu.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      aarch64-softmmu/hw/arm/boot.o
  CC      aarch64-softmmu/hw/arm/collie.o
  CC      aarch64-softmmu/hw/arm/exynos4_boards.o
  CC      x86_64-softmmu/hw/i386/kvmvapic.o
  CC      aarch64-softmmu/hw/arm/gumstix.o
  CC      aarch64-softmmu/hw/arm/highbank.o
  CC      aarch64-softmmu/hw/arm/digic_boards.o
  CC      x86_64-softmmu/hw/i386/acpi-build.o
  CC      aarch64-softmmu/hw/arm/integratorcp.o
  CC      x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC      aarch64-softmmu/hw/arm/mainstone.o
/tmp/qemu-test/src/hw/i386/acpi-build.c: In function ‘build_append_pci_bus_devices’:
/tmp/qemu-test/src/hw/i386/acpi-build.c:539: warning: ‘notify_method’ may be used uninitialized in this function
  CC      aarch64-softmmu/hw/arm/musicpal.o
  CC      aarch64-softmmu/hw/arm/nseries.o
  CC      x86_64-softmmu/hw/i386/../xenpv/xen_machine_pv.o
  CC      aarch64-softmmu/hw/arm/omap_sx1.o
  CC      aarch64-softmmu/hw/arm/palm.o
  CC      x86_64-softmmu/hw/i386/kvm/clock.o
  CC      aarch64-softmmu/hw/arm/realview.o
  CC      x86_64-softmmu/hw/i386/kvm/apic.o
  CC      x86_64-softmmu/hw/i386/kvm/i8259.o
  CC      aarch64-softmmu/hw/arm/spitz.o
  CC      x86_64-softmmu/hw/i386/kvm/ioapic.o
  CC      x86_64-softmmu/hw/i386/kvm/i8254.o
  CC      x86_64-softmmu/hw/i386/kvm/pci-assign.o
  CC      x86_64-softmmu/hw/i386/xen/xen_platform.o
  CC      aarch64-softmmu/hw/arm/stellaris.o
  CC      x86_64-softmmu/hw/i386/xen/xen_apic.o
  CC      aarch64-softmmu/hw/arm/tosa.o
  CC      aarch64-softmmu/hw/arm/versatilepb.o
  CC      aarch64-softmmu/hw/arm/vexpress.o
  CC      x86_64-softmmu/hw/i386/xen/xen_pvdevice.o
  CC      x86_64-softmmu/hw/i386/xen/xen-hvm.o
  CC      aarch64-softmmu/hw/arm/virt.o
  CC      x86_64-softmmu/hw/i386/xen/xen-mapcache.o
  CC      x86_64-softmmu/target/i386/helper.o
  CC      x86_64-softmmu/target/i386/cpu.o
  CC      aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC      x86_64-softmmu/target/i386/gdbstub.o
  CC      x86_64-softmmu/target/i386/xsave_helper.o
  CC      aarch64-softmmu/hw/arm/z2.o
  CC      x86_64-softmmu/target/i386/translate.o
  CC      aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC      x86_64-softmmu/target/i386/bpt_helper.o
  CC      aarch64-softmmu/hw/arm/netduino2.o
  CC      x86_64-softmmu/target/i386/cc_helper.o
  CC      x86_64-softmmu/target/i386/excp_helper.o
  CC      x86_64-softmmu/target/i386/fpu_helper.o
  CC      aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC      aarch64-softmmu/hw/arm/armv7m.o
  CC      x86_64-softmmu/target/i386/int_helper.o
  CC      aarch64-softmmu/hw/arm/exynos4210.o
  CC      aarch64-softmmu/hw/arm/pxa2xx.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC      x86_64-softmmu/target/i386/mem_helper.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC      x86_64-softmmu/target/i386/misc_helper.o
  CC      x86_64-softmmu/target/i386/mpx_helper.o
  CC      x86_64-softmmu/target/i386/seg_helper.o
  CC      aarch64-softmmu/hw/arm/digic.o
  CC      x86_64-softmmu/target/i386/smm_helper.o
  CC      x86_64-softmmu/target/i386/svm_helper.o
  CC      x86_64-softmmu/target/i386/machine.o
  CC      aarch64-softmmu/hw/arm/omap1.o
  CC      aarch64-softmmu/hw/arm/omap2.o
  CC      x86_64-softmmu/target/i386/arch_memory_mapping.o
  CC      x86_64-softmmu/target/i386/monitor.o
  CC      x86_64-softmmu/target/i386/arch_dump.o
  CC      x86_64-softmmu/target/i386/kvm.o
  CC      aarch64-softmmu/hw/arm/strongarm.o
  CC      x86_64-softmmu/target/i386/hyperv.o
  CC      aarch64-softmmu/hw/arm/allwinner-a10.o
  CC      aarch64-softmmu/hw/arm/cubieboard.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC      x86_64-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/hw/arm/bcm2836.o
  CC      aarch64-softmmu/hw/arm/raspi.o
  CC      aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC      aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC      x86_64-softmmu/trace/generated-helpers.o
  CC      aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC      aarch64-softmmu/hw/arm/fsl-imx25.o
  CC      aarch64-softmmu/hw/arm/imx25_pdk.o
  CC      aarch64-softmmu/hw/arm/fsl-imx31.o
  CC      aarch64-softmmu/hw/arm/kzm.o
  CC      aarch64-softmmu/hw/arm/fsl-imx6.o
  CC      aarch64-softmmu/hw/arm/aspeed_soc.o
  CC      aarch64-softmmu/hw/arm/sabrelite.o
  CC      aarch64-softmmu/hw/arm/aspeed.o
  CC      aarch64-softmmu/hw/arm/mps2.o
  CC      aarch64-softmmu/target/arm/arm-semi.o
  CC      aarch64-softmmu/target/arm/machine.o
  CC      aarch64-softmmu/target/arm/psci.o
  CC      aarch64-softmmu/target/arm/arch_dump.o
  CC      aarch64-softmmu/target/arm/monitor.o
  CC      aarch64-softmmu/target/arm/kvm-stub.o
  CC      aarch64-softmmu/target/arm/translate.o
  CC      aarch64-softmmu/target/arm/op_helper.o
  CC      aarch64-softmmu/target/arm/helper.o
/tmp/qemu-test/src/hw/i386/pc_piix.c: In function ‘igd_passthrough_isa_bridge_create’:
/tmp/qemu-test/src/hw/i386/pc_piix.c:1065: warning: ‘pch_rev_id’ may be used uninitialized in this function
  CC      aarch64-softmmu/target/arm/cpu.o
  CC      aarch64-softmmu/target/arm/neon_helper.o
  CC      aarch64-softmmu/target/arm/iwmmxt_helper.o
  CC      aarch64-softmmu/target/arm/gdbstub.o
  CC      aarch64-softmmu/target/arm/cpu64.o
  CC      aarch64-softmmu/target/arm/translate-a64.o
  CC      aarch64-softmmu/target/arm/helper-a64.o
  CC      aarch64-softmmu/target/arm/gdbstub64.o
  CC      aarch64-softmmu/target/arm/crypto_helper.o
/tmp/qemu-test/src/target/arm/translate-a64.c: In function ‘handle_shri_with_rndacc’:
/tmp/qemu-test/src/target/arm/translate-a64.c:6372: warning: ‘tcg_src_hi’ may be used uninitialized in this function
/tmp/qemu-test/src/target/arm/translate-a64.c: In function ‘disas_simd_scalar_two_reg_misc’:
/tmp/qemu-test/src/target/arm/translate-a64.c:8099: warning: ‘rmode’ may be used uninitialized in this function
  CC      aarch64-softmmu/target/arm/arm-powerctl.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/trace/control-target.o
  CC      aarch64-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/trace/generated-helpers.o
  LINK    x86_64-softmmu/qemu-system-x86_64
  LINK    aarch64-softmmu/qemu-system-aarch64
  TEST    tests/qapi-schema/alternate-any.out
  TEST    tests/qapi-schema/alternate-array.out
  TEST    tests/qapi-schema/alternate-base.out
  TEST    tests/qapi-schema/alternate-clash.out
  TEST    tests/qapi-schema/alternate-conflict-dict.out
  TEST    tests/qapi-schema/alternate-conflict-enum-bool.out
  TEST    tests/qapi-schema/alternate-conflict-enum-int.out
  TEST    tests/qapi-schema/alternate-conflict-string.out
  TEST    tests/qapi-schema/alternate-empty.out
  TEST    tests/qapi-schema/alternate-nested.out
  TEST    tests/qapi-schema/alternate-unknown.out
  TEST    tests/qapi-schema/args-alternate.out
  TEST    tests/qapi-schema/args-any.out
  TEST    tests/qapi-schema/args-array-unknown.out
  TEST    tests/qapi-schema/args-array-empty.out
  TEST    tests/qapi-schema/args-bad-boxed.out
  TEST    tests/qapi-schema/args-boxed-anon.out
  TEST    tests/qapi-schema/args-boxed-empty.out
  TEST    tests/qapi-schema/args-boxed-string.out
  TEST    tests/qapi-schema/args-int.out
  TEST    tests/qapi-schema/args-invalid.out
  TEST    tests/qapi-schema/args-member-array-bad.out
  TEST    tests/qapi-schema/args-member-case.out
  TEST    tests/qapi-schema/args-member-unknown.out
  TEST    tests/qapi-schema/args-name-clash.out
  TEST    tests/qapi-schema/args-union.out
  TEST    tests/qapi-schema/args-unknown.out
  TEST    tests/qapi-schema/bad-base.out
  TEST    tests/qapi-schema/bad-data.out
  TEST    tests/qapi-schema/bad-ident.out
  TEST    tests/qapi-schema/bad-type-bool.out
  TEST    tests/qapi-schema/bad-type-dict.out
  TEST    tests/qapi-schema/bad-type-int.out
  TEST    tests/qapi-schema/base-cycle-direct.out
  TEST    tests/qapi-schema/base-cycle-indirect.out
  TEST    tests/qapi-schema/command-int.out
  TEST    tests/qapi-schema/comments.out
  TEST    tests/qapi-schema/doc-bad-alternate-member.out
  TEST    tests/qapi-schema/doc-bad-command-arg.out
  TEST    tests/qapi-schema/doc-bad-symbol.out
  TEST    tests/qapi-schema/doc-bad-union-member.out
  TEST    tests/qapi-schema/doc-before-include.out
  TEST    tests/qapi-schema/doc-before-pragma.out
  TEST    tests/qapi-schema/doc-duplicated-arg.out
  TEST    tests/qapi-schema/doc-duplicated-return.out
  TEST    tests/qapi-schema/doc-duplicated-since.out
  TEST    tests/qapi-schema/doc-empty-arg.out
  TEST    tests/qapi-schema/doc-empty-section.out
  TEST    tests/qapi-schema/doc-empty-symbol.out
  TEST    tests/qapi-schema/doc-good.out
  TEST    tests/qapi-schema/doc-interleaved-section.out
  TEST    tests/qapi-schema/doc-invalid-end.out
  TEST    tests/qapi-schema/doc-invalid-end2.out
  TEST    tests/qapi-schema/doc-invalid-return.out
  TEST    tests/qapi-schema/doc-invalid-section.out
Files /tmp/qemu-test/src/tests/qapi-schema/doc-good.out and tests/qapi-schema/doc-good.test.out differ
make: *** [check-tests/qapi-schema/doc-good.json] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 382, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 379, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 237, in run
    return Docker().run(argv, args.keep, quiet=args.quiet)
  File "./tests/docker/docker.py", line 205, in run
    quiet=quiet)
  File "./tests/docker/docker.py", line 123, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=579f531486f511e7961952540069c830', '-u', '0', '-t', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/var/tmp/patchew-tester-tmp-vbkqrhlc/src/docker-src.2017-08-22-00.49.51.7825:/var/tmp/qemu:z,ro', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', 'qemu:centos6', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2
make[1]: *** [tests/docker/Makefile.include:139: docker-run] Error 1
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-vbkqrhlc/src'
make: *** [tests/docker/Makefile.include:168: docker-run-test-quick@centos6] Error 2

real	1m55.553s
user	0m5.231s
sys	0m1.740s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
Re: [Qemu-devel] [RFC 0/6] monitor: allow per-monitor thread
Posted by Peter Xu 6 years, 7 months ago
On Mon, Aug 21, 2017 at 09:51:47PM -0700, no-reply@patchew.org wrote:

[...]

>   TEST    tests/qapi-schema/doc-duplicated-return.out
>   TEST    tests/qapi-schema/doc-duplicated-since.out
>   TEST    tests/qapi-schema/doc-empty-arg.out
>   TEST    tests/qapi-schema/doc-empty-section.out
>   TEST    tests/qapi-schema/doc-empty-symbol.out
>   TEST    tests/qapi-schema/doc-good.out
>   TEST    tests/qapi-schema/doc-interleaved-section.out
>   TEST    tests/qapi-schema/doc-invalid-end.out
>   TEST    tests/qapi-schema/doc-invalid-end2.out
>   TEST    tests/qapi-schema/doc-invalid-return.out
>   TEST    tests/qapi-schema/doc-invalid-section.out
> Files /tmp/qemu-test/src/tests/qapi-schema/doc-good.out and tests/qapi-schema/doc-good.test.out differ
> make: *** [check-tests/qapi-schema/doc-good.json] Error 1
> make: *** Waiting for unfinished jobs....
> Traceback (most recent call last):
>   File "./tests/docker/docker.py", line 382, in <module>
>     sys.exit(main())
>   File "./tests/docker/docker.py", line 379, in main
>     return args.cmdobj.run(args, argv)
>   File "./tests/docker/docker.py", line 237, in run
>     return Docker().run(argv, args.keep, quiet=args.quiet)
>   File "./tests/docker/docker.py", line 205, in run
>     quiet=quiet)
>   File "./tests/docker/docker.py", line 123, in _do_check
>     return subprocess.check_call(self._command + cmd, **kwargs)
>   File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=579f531486f511e7961952540069c830', '-u', '0', '-t', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/var/tmp/patchew-tester-tmp-vbkqrhlc/src/docker-src.2017-08-22-00.49.51.7825:/var/tmp/qemu:z,ro', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', 'qemu:centos6', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2
> make[1]: *** [tests/docker/Makefile.include:139: docker-run] Error 1
> make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-vbkqrhlc/src'
> make: *** [tests/docker/Makefile.include:168: docker-run-test-quick@centos6] Error 2

This is because I forgot to touch-up tests/qapi-schema/test-qapi.py
for its visit_command() test.  Fixed up.

-- 
Peter Xu