v6->v7:
  * fix up wording in documentation and left-overs of 'reusing' 'cont' command in v5
  * split out preconfig runstate introduction into separate patch to make
    series more bisection friendly and put allow-preconfig patch before
    --preconfig CLI patch
  * merge qapi schema tests patches into one
  * rename allowed_in_preconfig flag to a shorter allow_preconfig
  * rename QCO_ALLOWED_IN_PRECONFIG into shorter QCO_ALLOW_PRECONFIG to match
    allow_preconfig flag
  * reuse an-oob-command for allow_preconfig flag testing and rename it to
    test-flags-command
v5->v6:
  * add exit-preconfig QMP command instead of overloading meaning of 'cont' command
  * add doc text to qemu-tech.texi about -S and --preconfig
  * add numa configuration example into commit message of 10/11
  * limit set-numa-node QMP command to preconfig mode
v4->v5:
  * rebase on top of OOB changes that's now in master
  * s/qobject_to_qdict(/qobject_to(QDict,/
  * s/-preconfig/--preconfig/
  * s/2.12/2.13/
  * s/parse_NumaOptions/set_numa_options/
  * drop if (err) guard around error_propagate()
  * move QCO_ALLOWED_IN_PRECONFIG and do_qmp_dispatch() runstate check
    from the later patch 'qapi: introduce new cmd option  "allowed-in-preconfig"'
    to here for better bissectability
  * add TODO comment to '{ RUN_STATE_PRECONFIG, RUN_STATE_INMIGRATE }' transition
  * check for incoming && preconfig outside of option parsing loop
  * add 'use QMP instead" to error message, suggesting user the right interface to use
  * allow query-command-line-options in preconfig state
  * make sure that allowed-in-preconfig could be set only to True
  * move out QCO_ALLOWED_IN_PRECONFIG check in do_qmp_dispatch() to
    earlier 'cli: add --preconfig option' patch
  * 2 extra test patches
     'tests: let qapi-schema tests detect allowed-in-preconfig'
     'tests: add allowed-in-preconfig-test for qapi-schema'
v3->v4:
  * replace 'runstates' list in  QMP command with a single
    boolean 'ption allowed-in-preconfig' like it's done with
    'allow-oob'. Which allows to simplify intrusive QAPI
    changes quite a lot. (Eric Blake <eblake@redhat.com>)
  * Make sure HMP is disbled for real, v3 was just printing
    error mesage but allowing command to be executed
    ("Dr. David Alan Gilbert" <dgilbert@redhat.com>)
  * Improve "cli: add -preconfig option" commit message,
    explain a bit more on semantics of new state/option.
  * ithe rest of minor fixups suggested at v3 review
    (Eric Blake <eblake@redhat.com>)
    PS:
       havn't impl. test for new option in
         tests/qapi-schema/qapi-schema-test.json yet,
       can do it on top if approach is acceptable.
v1->v3:
  * introduce PRECONFIG runstate with -preconfig option.
    it's cleaner to manage transitions and do checks
    than reusing existing PRELAUNCH state.
  * extend QAPI schema commands with 'runstates' keyword,
    so that it would be possible to specify in command definition
    when it is valid to execute.
    (python changes a bit hackery, since I have little to
     no idea how it should work)
  * add preconfig QMP and set-numa-node tests
  * make mutually exclusive -preconfig and -incoming options,
    for simplicity sake. Shouldn't be problem as target can
    be starter with pure CLI, since mapping on source is
    already known.
  * Drop HMP part and leave only QMP in preconfig state.


Currently it's problematic to configure NUMA mapping for CPUs using "-numa cpu="
option, without restarting QEMU, first is to query board specific CPU layout
with query-hotpluggable-cpus QMP command (which is function of -M and -smp CLI
options). Mgmt side isn't happy with restarting QEMU to query configuration
parameters first and then run actual instance, so it keeps using old cpu_index
based option '-numa cpus' instead of new socket/core/thread-id based one.

Introduce a new '--preconfig' CLI option which allows pausing QEMU before
machine_init() is run (preconfig state). So it would be possible to query
possible CPUs layout, configure NUMA mapping based on query result with
set-numa-node QMP command and continue executing  without restarting QEMU.

Difference between new --preconfig pause point vs -S is that the later pauses
QEMU after machine is constructed and ready to run guest code or in process of
incoming migration (essentially machine is in some running state (with paused
VCPUs) and any action on it is considered as hotplug). At this point it's hard
to configure or reconfigure parameters that affect machine_init() and later
stages. While the new --preconfig option pauses QEMU instance before
machine_init() and would allow to configure parameters as if doing it from CLI
but in interactve manner using QMP interface, which would allow introspecting
and configuring QEMU instance without restarting it. 

Initially only limited set of commands (that are ready to work with non
initialized machine or without it) are allowed in preconfig state:

   #existing commands:
       qmp_capabilities
       query-qmp-schema
       query-commands
       query-command-line-options
       query-status
       query-hotpluggable-cpus
   #new commands
       exit-preconfig
       set-numa-node

Which commands are allowed at preconfig state, is defined by QAPI Schema
with help of optional 'allowed-in-preconfig' flag in command definition,
which allows users to get list of commands it can run with help of
query-qmp-schema command.

Once user done with configuration at preconfig state, it could be exited with
exit-preconfig command, in which case QEMU would proceed with VM intialization
and either start guest execution or pause in prelaunch state if -S CLI option
were specified along with --prelaunch on startup.

PS:
Later we can modify other commands to run early, for example device_add. So
that devices would be added in cold-plug context, instead of getting device
'hotplugged' with followed up fixups and workarounds to make it look like
cold-plugged.

Example of configuration CPU's NUMA mapping with --preconfig:
$QEMU -smp 2 --preconfig ...

QMP:
# get CPUs layout for current target/machine/CLI
-> {'execute': 'query-hotpluggable-cpus' }        
<- {'return': [
       {'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 1}, ... },
       {'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 0}, ... }
   ]}

# configure 1st node
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 0 } }        
<- {'return': {}}
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',         
       'node-id': 0, 'core-id': 0, 'thread-id': 0, 'socket-id': 1, }
   }
<- {'return': {}}

# configure 2nd node
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 1 } }
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',        
       'node-id': 1, 'core-id': 0, 'thread-id': 0, 'socket-id': 0 }
   }
<- {'return': {}}

# [optional] verify configuration
-> {'execute': 'query-hotpluggable-cpus' }        
<- {'return': [
       {'props': {'core-id': 0, 'thread-id': 0, 'node-id': 0, 'socket-id': 1}, ... },
       {'props': {'core-id': 0, 'thread-id': 0, 'node-id': 1, 'socket-id': 0}, ... }
   ]}


Git tree:
    https://github.com/imammedo/qemu.git qmp_preconfig_v7

 
CC: eblake@redhat.com
CC: ehabkost@redhat.com
CC: pkrempa@redhat.com
CC: armbru@redhat.com


Igor Mammedov (11):
  numa: postpone options post-processing till machine_run_board_init()
  numa: split out NumaOptions parsing into set_numa_options()
  qapi: introduce preconfig runstate
  hmp: disable monitor in preconfig state
  qapi: introduce new cmd option "allowed-in-preconfig"
  tests: qapi-schema tests for allow-preconfig
  cli: add --preconfig option
  tests: extend qmp test with preconfig checks
  qmp: permit query-hotpluggable-cpus in preconfig state
  qmp: add set-numa-node command
  tests: functional tests for QMP command set-numa-node

 include/qapi/qmp/dispatch.h                 |  1 +
 include/sysemu/numa.h                       |  2 +
 include/sysemu/sysemu.h                     |  1 +
 tests/libqtest.h                            |  9 ++++
 docs/devel/qapi-code-gen.txt                | 10 ++++-
 hw/core/machine.c                           |  5 ++-
 monitor.c                                   | 11 +++--
 numa.c                                      | 70 +++++++++++++++++++----------
 qapi/introspect.json                        |  5 ++-
 qapi/misc.json                              | 49 ++++++++++++++++++--
 qapi/qmp-dispatch.c                         |  8 ++++
 qapi/run-state.json                         |  8 +++-
 qemu-options.hx                             | 13 ++++++
 qemu-tech.texi                              | 40 +++++++++++++++++
 qmp.c                                       | 10 +++++
 scripts/qapi/commands.py                    | 11 +++--
 scripts/qapi/common.py                      | 18 +++++---
 scripts/qapi/doc.py                         |  4 +-
 scripts/qapi/introspect.py                  |  7 +--
 tests/Makefile.include                      |  1 +
 tests/libqtest.c                            |  7 +++
 tests/numa-test.c                           | 61 +++++++++++++++++++++++++
 tests/qapi-schema/allow-preconfig-test.err  |  1 +
 tests/qapi-schema/allow-preconfig-test.exit |  1 +
 tests/qapi-schema/allow-preconfig-test.json |  2 +
 tests/qapi-schema/allow-preconfig-test.out  |  0
 tests/qapi-schema/doc-good.out              |  4 +-
 tests/qapi-schema/ident-with-escape.out     |  2 +-
 tests/qapi-schema/indented-expr.out         |  4 +-
 tests/qapi-schema/qapi-schema-test.json     |  4 +-
 tests/qapi-schema/qapi-schema-test.out      | 22 ++++-----
 tests/qapi-schema/test-qapi.py              |  8 ++--
 tests/qmp-test.c                            | 37 +++++++++++++++
 tests/test-qmp-cmds.c                       |  2 +-
 vl.c                                        | 35 ++++++++++++++-
 35 files changed, 396 insertions(+), 77 deletions(-)
 create mode 100644 tests/qapi-schema/allow-preconfig-test.err
 create mode 100644 tests/qapi-schema/allow-preconfig-test.exit
 create mode 100644 tests/qapi-schema/allow-preconfig-test.json
 create mode 100644 tests/qapi-schema/allow-preconfig-test.out

-- 
2.7.4

I'm queueing this version (including v8 of patches 5-7) on
numa-next.

Markus, Daniel: you were the people I remember expressing some
concerns about the new preconfig mechanism.  Do you have any
objections to this version?


On Fri, May 04, 2018 at 10:37:38AM +0200, Igor Mammedov wrote:
> 
> 
> v6->v7:
>   * fix up wording in documentation and left-overs of 'reusing' 'cont' command in v5
>   * split out preconfig runstate introduction into separate patch to make
>     series more bisection friendly and put allow-preconfig patch before
>     --preconfig CLI patch
>   * merge qapi schema tests patches into one
>   * rename allowed_in_preconfig flag to a shorter allow_preconfig
>   * rename QCO_ALLOWED_IN_PRECONFIG into shorter QCO_ALLOW_PRECONFIG to match
>     allow_preconfig flag
>   * reuse an-oob-command for allow_preconfig flag testing and rename it to
>     test-flags-command
> v5->v6:
>   * add exit-preconfig QMP command instead of overloading meaning of 'cont' command
>   * add doc text to qemu-tech.texi about -S and --preconfig
>   * add numa configuration example into commit message of 10/11
>   * limit set-numa-node QMP command to preconfig mode
> v4->v5:
>   * rebase on top of OOB changes that's now in master
>   * s/qobject_to_qdict(/qobject_to(QDict,/
>   * s/-preconfig/--preconfig/
>   * s/2.12/2.13/
>   * s/parse_NumaOptions/set_numa_options/
>   * drop if (err) guard around error_propagate()
>   * move QCO_ALLOWED_IN_PRECONFIG and do_qmp_dispatch() runstate check
>     from the later patch 'qapi: introduce new cmd option  "allowed-in-preconfig"'
>     to here for better bissectability
>   * add TODO comment to '{ RUN_STATE_PRECONFIG, RUN_STATE_INMIGRATE }' transition
>   * check for incoming && preconfig outside of option parsing loop
>   * add 'use QMP instead" to error message, suggesting user the right interface to use
>   * allow query-command-line-options in preconfig state
>   * make sure that allowed-in-preconfig could be set only to True
>   * move out QCO_ALLOWED_IN_PRECONFIG check in do_qmp_dispatch() to
>     earlier 'cli: add --preconfig option' patch
>   * 2 extra test patches
>      'tests: let qapi-schema tests detect allowed-in-preconfig'
>      'tests: add allowed-in-preconfig-test for qapi-schema'
> v3->v4:
>   * replace 'runstates' list in  QMP command with a single
>     boolean 'ption allowed-in-preconfig' like it's done with
>     'allow-oob'. Which allows to simplify intrusive QAPI
>     changes quite a lot. (Eric Blake <eblake@redhat.com>)
>   * Make sure HMP is disbled for real, v3 was just printing
>     error mesage but allowing command to be executed
>     ("Dr. David Alan Gilbert" <dgilbert@redhat.com>)
>   * Improve "cli: add -preconfig option" commit message,
>     explain a bit more on semantics of new state/option.
>   * ithe rest of minor fixups suggested at v3 review
>     (Eric Blake <eblake@redhat.com>)
>     PS:
>        havn't impl. test for new option in
>          tests/qapi-schema/qapi-schema-test.json yet,
>        can do it on top if approach is acceptable.
> v1->v3:
>   * introduce PRECONFIG runstate with -preconfig option.
>     it's cleaner to manage transitions and do checks
>     than reusing existing PRELAUNCH state.
>   * extend QAPI schema commands with 'runstates' keyword,
>     so that it would be possible to specify in command definition
>     when it is valid to execute.
>     (python changes a bit hackery, since I have little to
>      no idea how it should work)
>   * add preconfig QMP and set-numa-node tests
>   * make mutually exclusive -preconfig and -incoming options,
>     for simplicity sake. Shouldn't be problem as target can
>     be starter with pure CLI, since mapping on source is
>     already known.
>   * Drop HMP part and leave only QMP in preconfig state.
> 
> 
> Currently it's problematic to configure NUMA mapping for CPUs using "-numa cpu="
> option, without restarting QEMU, first is to query board specific CPU layout
> with query-hotpluggable-cpus QMP command (which is function of -M and -smp CLI
> options). Mgmt side isn't happy with restarting QEMU to query configuration
> parameters first and then run actual instance, so it keeps using old cpu_index
> based option '-numa cpus' instead of new socket/core/thread-id based one.
> 
> Introduce a new '--preconfig' CLI option which allows pausing QEMU before
> machine_init() is run (preconfig state). So it would be possible to query
> possible CPUs layout, configure NUMA mapping based on query result with
> set-numa-node QMP command and continue executing  without restarting QEMU.
> 
> Difference between new --preconfig pause point vs -S is that the later pauses
> QEMU after machine is constructed and ready to run guest code or in process of
> incoming migration (essentially machine is in some running state (with paused
> VCPUs) and any action on it is considered as hotplug). At this point it's hard
> to configure or reconfigure parameters that affect machine_init() and later
> stages. While the new --preconfig option pauses QEMU instance before
> machine_init() and would allow to configure parameters as if doing it from CLI
> but in interactve manner using QMP interface, which would allow introspecting
> and configuring QEMU instance without restarting it. 
> 
> Initially only limited set of commands (that are ready to work with non
> initialized machine or without it) are allowed in preconfig state:
> 
>    #existing commands:
>        qmp_capabilities
>        query-qmp-schema
>        query-commands
>        query-command-line-options
>        query-status
>        query-hotpluggable-cpus
>    #new commands
>        exit-preconfig
>        set-numa-node
> 
> Which commands are allowed at preconfig state, is defined by QAPI Schema
> with help of optional 'allowed-in-preconfig' flag in command definition,
> which allows users to get list of commands it can run with help of
> query-qmp-schema command.
> 
> Once user done with configuration at preconfig state, it could be exited with
> exit-preconfig command, in which case QEMU would proceed with VM intialization
> and either start guest execution or pause in prelaunch state if -S CLI option
> were specified along with --prelaunch on startup.
> 
> PS:
> Later we can modify other commands to run early, for example device_add. So
> that devices would be added in cold-plug context, instead of getting device
> 'hotplugged' with followed up fixups and workarounds to make it look like
> cold-plugged.
> 
> Example of configuration CPU's NUMA mapping with --preconfig:
> $QEMU -smp 2 --preconfig ...
> 
> QMP:
> # get CPUs layout for current target/machine/CLI
> -> {'execute': 'query-hotpluggable-cpus' }        
> <- {'return': [
>        {'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 1}, ... },
>        {'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 0}, ... }
>    ]}
> 
> # configure 1st node
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 0 } }        
> <- {'return': {}}
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',         
>        'node-id': 0, 'core-id': 0, 'thread-id': 0, 'socket-id': 1, }
>    }
> <- {'return': {}}
> 
> # configure 2nd node
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 1 } }
> -> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',        
>        'node-id': 1, 'core-id': 0, 'thread-id': 0, 'socket-id': 0 }
>    }
> <- {'return': {}}
> 
> # [optional] verify configuration
> -> {'execute': 'query-hotpluggable-cpus' }        
> <- {'return': [
>        {'props': {'core-id': 0, 'thread-id': 0, 'node-id': 0, 'socket-id': 1}, ... },
>        {'props': {'core-id': 0, 'thread-id': 0, 'node-id': 1, 'socket-id': 0}, ... }
>    ]}
> 
> 
> Git tree:
>     https://github.com/imammedo/qemu.git qmp_preconfig_v7
> 
>  
> CC: eblake@redhat.com
> CC: ehabkost@redhat.com
> CC: pkrempa@redhat.com
> CC: armbru@redhat.com
> 
> 
> Igor Mammedov (11):
>   numa: postpone options post-processing till machine_run_board_init()
>   numa: split out NumaOptions parsing into set_numa_options()
>   qapi: introduce preconfig runstate
>   hmp: disable monitor in preconfig state
>   qapi: introduce new cmd option "allowed-in-preconfig"
>   tests: qapi-schema tests for allow-preconfig
>   cli: add --preconfig option
>   tests: extend qmp test with preconfig checks
>   qmp: permit query-hotpluggable-cpus in preconfig state
>   qmp: add set-numa-node command
>   tests: functional tests for QMP command set-numa-node
> 
>  include/qapi/qmp/dispatch.h                 |  1 +
>  include/sysemu/numa.h                       |  2 +
>  include/sysemu/sysemu.h                     |  1 +
>  tests/libqtest.h                            |  9 ++++
>  docs/devel/qapi-code-gen.txt                | 10 ++++-
>  hw/core/machine.c                           |  5 ++-
>  monitor.c                                   | 11 +++--
>  numa.c                                      | 70 +++++++++++++++++++----------
>  qapi/introspect.json                        |  5 ++-
>  qapi/misc.json                              | 49 ++++++++++++++++++--
>  qapi/qmp-dispatch.c                         |  8 ++++
>  qapi/run-state.json                         |  8 +++-
>  qemu-options.hx                             | 13 ++++++
>  qemu-tech.texi                              | 40 +++++++++++++++++
>  qmp.c                                       | 10 +++++
>  scripts/qapi/commands.py                    | 11 +++--
>  scripts/qapi/common.py                      | 18 +++++---
>  scripts/qapi/doc.py                         |  4 +-
>  scripts/qapi/introspect.py                  |  7 +--
>  tests/Makefile.include                      |  1 +
>  tests/libqtest.c                            |  7 +++
>  tests/numa-test.c                           | 61 +++++++++++++++++++++++++
>  tests/qapi-schema/allow-preconfig-test.err  |  1 +
>  tests/qapi-schema/allow-preconfig-test.exit |  1 +
>  tests/qapi-schema/allow-preconfig-test.json |  2 +
>  tests/qapi-schema/allow-preconfig-test.out  |  0
>  tests/qapi-schema/doc-good.out              |  4 +-
>  tests/qapi-schema/ident-with-escape.out     |  2 +-
>  tests/qapi-schema/indented-expr.out         |  4 +-
>  tests/qapi-schema/qapi-schema-test.json     |  4 +-
>  tests/qapi-schema/qapi-schema-test.out      | 22 ++++-----
>  tests/qapi-schema/test-qapi.py              |  8 ++--
>  tests/qmp-test.c                            | 37 +++++++++++++++
>  tests/test-qmp-cmds.c                       |  2 +-
>  vl.c                                        | 35 ++++++++++++++-
>  35 files changed, 396 insertions(+), 77 deletions(-)
>  create mode 100644 tests/qapi-schema/allow-preconfig-test.err
>  create mode 100644 tests/qapi-schema/allow-preconfig-test.exit
>  create mode 100644 tests/qapi-schema/allow-preconfig-test.json
>  create mode 100644 tests/qapi-schema/allow-preconfig-test.out
> 
> -- 
> 2.7.4
> 
> 

-- 
Eduardo

Eduardo Habkost <ehabkost@redhat.com> writes:

> I'm queueing this version (including v8 of patches 5-7) on
> numa-next.
>
> Markus, Daniel: you were the people I remember expressing some
> concerns about the new preconfig mechanism.  Do you have any
> objections to this version?

My gut's reaction to preconfig is as nauseous as ever.  It adds
complexity we wouldn't want or need with sanely structured program
startup.  It might still be the best we can do, and something we need to
do.  You have to take on technical debt sometimes.  To know for sure
this is the case here, we'd have to explore alternatives more seriously.
Like repaying some of the existing technical debt around there.  Sadly,
I lack the time to take on such a project.

Since I lack the money to put where my mouth is, I'm going to shut up
and get out of your way.

On Thu, 24 May 2018 17:42:03 +0200
Markus Armbruster <armbru@redhat.com> wrote:

> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > I'm queueing this version (including v8 of patches 5-7) on
> > numa-next.
> >
> > Markus, Daniel: you were the people I remember expressing some
> > concerns about the new preconfig mechanism.  Do you have any
> > objections to this version?  
> 
> My gut's reaction to preconfig is as nauseous as ever.  It adds
> complexity we wouldn't want or need with sanely structured program
> startup.  It might still be the best we can do, and something we need to
> do.  You have to take on technical debt sometimes.  To know for sure
> this is the case here, we'd have to explore alternatives more seriously.
> Like repaying some of the existing technical debt around there.  Sadly,
> I lack the time to take on such a project.
Usually I do quite a bit of re-factoring to reduce our technical debt,
so while working on enabling VM start with libvirt using only --preconfig
(without -S) I most likely will have to improve code around VM creation.

[...]

Eduardo,

I've rebased series on top of current master
the only change in several patches was s/2.13/3.0/
otherwise there weren't any other conflicts.
You can find rebased version at

https://github.com/imammedo/qemu.git qmp_preconfig_v9

On Wed, May 30, 2018 at 12:47:17PM +0200, Igor Mammedov wrote:
> Eduardo,
> 
> I've rebased series on top of current master
> the only change in several patches was s/2.13/3.0/
> otherwise there weren't any other conflicts.
> You can find rebased version at
> 
> https://github.com/imammedo/qemu.git qmp_preconfig_v9

Thanks!  I've rebased my branch on top of it and noticed that the
only differences are the "since 3.0" lines.  I've fixed them and
pushed a new numa-next branch.

Once the travis-ci job finishes, I will send a pull request.

-- 
Eduardo